2024-03-28T23:19:09.709 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-28T23:19:09.710 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} --owner scheduled_teuthology@teuthology --name teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi --verbose --archive /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627196 -- /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627196/orig.config.yaml 2024-03-28T23:19:09.751 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T23:19:09.834 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627196 branch: squid description: fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} email: ceph-qa@ceph.com first_in_suite: false job_id: '7627196' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - Replacing daemon mds - \(MDS_INSUFFICIENT_STANDBY\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MDS_DAMAGE\) - \(FS_DEGRADED\) sha1: 323d23dc228bbcb7f16f188fdc9c0dcb801844be ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: disabled: true install: ceph: flavor: default sha1: 323d23dc228bbcb7f16f188fdc9c0dcb801844be selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 55f5138942aa4866e49329c9103468889cec3f73 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 seed: 7214 sha1: 323d23dc228bbcb7f16f188fdc9c0dcb801844be sleep_before_teardown: 0 subset: 39/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 55f5138942aa4866e49329c9103468889cec3f73 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi161.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBByTCof/EqEz1jIVz78GqGKb+p3QSZbaeon8+Zw2GmOCmoalhHdC1zWHiRPrTT1QZKvq2jFe7DSq61l/GXWj/YU= smithi170.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOcXNoCIPqw31oOW4OMimd9Wdo5l/Jf3E136yIF9WJYmGzcPjP9ayv8TR+bXAGJHUV+4rOjmqVjUj7KgskU0mUk= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - exec: mon.a: - ceph config set mgr mgr/crash/warn_recent_interval 0 - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_failover teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-27_21:24:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-28T23:19:09.834 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa; will attempt to use it 2024-03-28T23:19:09.835 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa/tasks 2024-03-28T23:19:09.835 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-28T23:19:09.837 INFO:teuthology.task.internal:Checking packages... 2024-03-28T23:19:09.863 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '323d23dc228bbcb7f16f188fdc9c0dcb801844be' 2024-03-28T23:19:09.863 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-28T23:19:09.863 INFO:teuthology.packaging:ref: None 2024-03-28T23:19:09.863 INFO:teuthology.packaging:tag: None 2024-03-28T23:19:09.863 INFO:teuthology.packaging:branch: squid 2024-03-28T23:19:09.863 INFO:teuthology.packaging:sha1: 323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T23:19:09.864 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-28T23:19:10.023 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1959.g8da23624 2024-03-28T23:19:10.024 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-28T23:19:10.030 INFO:teuthology.task.internal:no buildpackages task found 2024-03-28T23:19:10.030 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-28T23:19:10.038 INFO:teuthology.task.internal:Saving configuration 2024-03-28T23:19:10.054 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-28T23:19:10.060 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T23:19:10.084 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi161.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627196', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 23:13:52.513898', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBByTCof/EqEz1jIVz78GqGKb+p3QSZbaeon8+Zw2GmOCmoalhHdC1zWHiRPrTT1QZKvq2jFe7DSq61l/GXWj/YU='} 2024-03-28T23:19:10.105 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627196', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 23:13:52.512897', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOcXNoCIPqw31oOW4OMimd9Wdo5l/Jf3E136yIF9WJYmGzcPjP9ayv8TR+bXAGJHUV+4rOjmqVjUj7KgskU0mUk='} 2024-03-28T23:19:10.105 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-28T23:19:10.112 INFO:teuthology.task.internal:roles: ubuntu@smithi161.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-03-28T23:19:10.112 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-03-28T23:19:10.112 INFO:teuthology.run_tasks:Running task console_log... 2024-03-28T23:19:10.182 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f55303c5040>, signals=[15]) 2024-03-28T23:19:10.182 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-28T23:19:10.188 INFO:teuthology.task.internal:Opening connections... 2024-03-28T23:19:10.189 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi161.front.sepia.ceph.com 2024-03-28T23:19:10.190 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T23:19:10.282 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi170.front.sepia.ceph.com 2024-03-28T23:19:10.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T23:19:10.371 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-28T23:19:10.376 DEBUG:teuthology.orchestra.run.smithi161:> uname -m 2024-03-28T23:19:10.393 INFO:teuthology.orchestra.run.smithi161.stdout:x86_64 2024-03-28T23:19:10.393 DEBUG:teuthology.orchestra.run.smithi161:> cat /etc/os-release 2024-03-28T23:19:10.450 INFO:teuthology.orchestra.run.smithi161.stdout:NAME="CentOS Stream" 2024-03-28T23:19:10.451 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION="9" 2024-03-28T23:19:10.451 INFO:teuthology.orchestra.run.smithi161.stdout:ID="centos" 2024-03-28T23:19:10.451 INFO:teuthology.orchestra.run.smithi161.stdout:ID_LIKE="rhel fedora" 2024-03-28T23:19:10.451 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_ID="9" 2024-03-28T23:19:10.451 INFO:teuthology.orchestra.run.smithi161.stdout:PLATFORM_ID="platform:el9" 2024-03-28T23:19:10.451 INFO:teuthology.orchestra.run.smithi161.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T23:19:10.451 INFO:teuthology.orchestra.run.smithi161.stdout:ANSI_COLOR="0;31" 2024-03-28T23:19:10.451 INFO:teuthology.orchestra.run.smithi161.stdout:LOGO="fedora-logo-icon" 2024-03-28T23:19:10.451 INFO:teuthology.orchestra.run.smithi161.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T23:19:10.451 INFO:teuthology.orchestra.run.smithi161.stdout:HOME_URL="https://centos.org/" 2024-03-28T23:19:10.451 INFO:teuthology.orchestra.run.smithi161.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T23:19:10.451 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T23:19:10.451 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T23:19:10.452 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server 2024-03-28T23:19:10.475 DEBUG:teuthology.orchestra.run.smithi170:> uname -m 2024-03-28T23:19:10.492 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2024-03-28T23:19:10.492 DEBUG:teuthology.orchestra.run.smithi170:> cat /etc/os-release 2024-03-28T23:19:10.549 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="CentOS Stream" 2024-03-28T23:19:10.549 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="9" 2024-03-28T23:19:10.549 INFO:teuthology.orchestra.run.smithi170.stdout:ID="centos" 2024-03-28T23:19:10.549 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="rhel fedora" 2024-03-28T23:19:10.549 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="9" 2024-03-28T23:19:10.549 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el9" 2024-03-28T23:19:10.549 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T23:19:10.549 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31" 2024-03-28T23:19:10.549 INFO:teuthology.orchestra.run.smithi170.stdout:LOGO="fedora-logo-icon" 2024-03-28T23:19:10.549 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T23:19:10.549 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://centos.org/" 2024-03-28T23:19:10.549 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T23:19:10.550 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T23:19:10.550 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T23:19:10.550 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2024-03-28T23:19:10.566 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-28T23:19:10.576 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-28T23:19:10.581 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-28T23:19:10.582 DEBUG:teuthology.orchestra.run.smithi161:> test '!' -e /home/ubuntu/cephtest 2024-03-28T23:19:10.584 DEBUG:teuthology.orchestra.run.smithi170:> test '!' -e /home/ubuntu/cephtest 2024-03-28T23:19:10.604 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-28T23:19:10.610 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-28T23:19:10.610 DEBUG:teuthology.orchestra.run.smithi161:> test -z $(ls -A /var/lib/ceph) 2024-03-28T23:19:10.641 DEBUG:teuthology.orchestra.run.smithi170:> test -z $(ls -A /var/lib/ceph) 2024-03-28T23:19:10.739 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-28T23:19:10.794 INFO:teuthology.run_tasks:Running task kernel... 2024-03-28T23:19:10.809 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-28T23:19:10.809 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-28T23:19:10.809 DEBUG:teuthology.orchestra.run.smithi161:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T23:19:10.810 DEBUG:teuthology.orchestra.run.smithi170:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T23:19:10.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:19:10.825 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2024-03-28T23:19:10.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:19:10.827 DEBUG:teuthology.orchestra.run.smithi161:> uname -r 2024-03-28T23:19:10.881 INFO:teuthology.orchestra.run.smithi170.stdout:5.14.0-361.el9.x86_64 2024-03-28T23:19:10.881 INFO:teuthology.task.kernel:Running kernel on smithi170: 5.14.0-361.el9.x86_64 2024-03-28T23:19:10.882 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum install -y kernel 2024-03-28T23:19:10.883 INFO:teuthology.orchestra.run.smithi161.stdout:5.14.0-361.el9.x86_64 2024-03-28T23:19:10.883 INFO:teuthology.task.kernel:Running kernel on smithi161: 5.14.0-361.el9.x86_64 2024-03-28T23:19:10.884 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum install -y kernel 2024-03-28T23:19:12.830 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-28T23:19:13.120 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-28T23:19:14.197 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - BaseOS 7.8 MB/s | 2.0 MB 00:00 2024-03-28T23:19:14.478 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-28T23:19:15.187 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-28T23:19:20.663 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - AppStream 4.3 MB/s | 19 MB 00:04 2024-03-28T23:19:23.776 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - BaseOS 918 kB/s | 8.0 MB 00:08 2024-03-28T23:19:25.846 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-03-28T23:19:26.238 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 6.1 MB 00:02 2024-03-28T23:19:27.689 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 16 kB 00:00 2024-03-28T23:19:29.168 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-28T23:19:29.828 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-28T23:19:31.312 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 16 kB 00:00 2024-03-28T23:19:34.107 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-03-28T23:19:36.322 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-28T23:19:36.322 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-28T23:19:36.409 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-28T23:19:36.417 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-28T23:19:36.417 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-28T23:19:36.417 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-28T23:19:36.417 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-28T23:19:36.417 INFO:teuthology.orchestra.run.smithi170.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-28T23:19:36.417 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-28T23:19:36.418 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-28T23:19:36.418 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-28T23:19:36.418 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-28T23:19:36.418 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:19:36.418 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-28T23:19:36.418 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-28T23:19:36.418 INFO:teuthology.orchestra.run.smithi170.stdout:Install 4 Packages 2024-03-28T23:19:36.418 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:19:36.419 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 76 M 2024-03-28T23:19:36.419 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 123 M 2024-03-28T23:19:36.419 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-28T23:19:36.575 INFO:teuthology.orchestra.run.smithi170.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-28T23:19:37.492 INFO:teuthology.orchestra.run.smithi170.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 14 MB/s | 15 MB 00:01 2024-03-28T23:19:38.033 INFO:teuthology.orchestra.run.smithi170.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 21 MB/s | 34 MB 00:01 2024-03-28T23:19:38.350 INFO:teuthology.orchestra.run.smithi170.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 16 MB/s | 28 MB 00:01 2024-03-28T23:19:38.353 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-28T23:19:38.353 INFO:teuthology.orchestra.run.smithi170.stdout:Total 40 MB/s | 76 MB 00:01 2024-03-28T23:19:38.354 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-28T23:19:38.720 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-28T23:19:38.721 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-28T23:19:39.566 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-28T23:19:39.567 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-28T23:19:41.870 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-28T23:19:42.467 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-28T23:19:42.560 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-28T23:19:45.315 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-28T23:19:45.455 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-28T23:19:50.183 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-28T23:19:50.287 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-28T23:19:51.397 INFO:teuthology.orchestra.run.smithi161.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:19 2024-03-28T23:19:55.054 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-28T23:19:56.286 INFO:teuthology.orchestra.run.smithi161.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-03-28T23:19:58.526 INFO:teuthology.orchestra.run.smithi161.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-28T23:19:58.526 INFO:teuthology.orchestra.run.smithi161.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-28T23:19:58.619 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-28T23:19:58.624 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-28T23:19:58.624 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-03-28T23:19:58.624 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-28T23:19:58.624 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-28T23:19:58.624 INFO:teuthology.orchestra.run.smithi161.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-28T23:19:58.625 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-03-28T23:19:58.625 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-28T23:19:58.625 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-28T23:19:58.625 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-28T23:19:58.625 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:19:58.625 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-28T23:19:58.625 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-28T23:19:58.625 INFO:teuthology.orchestra.run.smithi161.stdout:Install 4 Packages 2024-03-28T23:19:58.625 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:19:58.626 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 76 M 2024-03-28T23:19:58.626 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 123 M 2024-03-28T23:19:58.626 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-28T23:19:58.962 INFO:teuthology.orchestra.run.smithi161.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 570 kB/s | 192 kB 00:00 2024-03-28T23:20:00.054 INFO:teuthology.orchestra.run.smithi161.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 11 MB/s | 15 MB 00:01 2024-03-28T23:20:00.754 INFO:teuthology.orchestra.run.smithi161.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 16 MB/s | 34 MB 00:02 2024-03-28T23:20:01.187 INFO:teuthology.orchestra.run.smithi161.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 12 MB/s | 28 MB 00:02 2024-03-28T23:20:01.190 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-28T23:20:01.190 INFO:teuthology.orchestra.run.smithi161.stdout:Total 30 MB/s | 76 MB 00:02 2024-03-28T23:20:01.191 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-28T23:20:01.565 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-28T23:20:01.565 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-28T23:20:02.507 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-28T23:20:02.507 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-28T23:20:04.827 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-28T23:20:05.433 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-28T23:20:05.513 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-28T23:20:08.272 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-28T23:20:08.417 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-28T23:20:13.164 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-28T23:20:13.273 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-28T23:20:18.073 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-28T23:20:24.507 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-28T23:20:24.508 INFO:teuthology.orchestra.run.smithi170.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-28T23:20:24.508 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:20:24.558 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-28T23:20:24.963 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-28T23:20:24.963 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-28T23:20:24.963 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-28T23:20:24.963 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-28T23:20:25.365 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-28T23:20:25.365 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:20:25.365 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-28T23:20:25.365 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-28T23:20:25.365 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-28T23:20:25.365 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-28T23:20:25.365 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-28T23:20:25.365 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:20:25.365 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-28T23:20:25.688 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T23:20:25.785 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-28T23:20:25.785 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-28T23:20:25.785 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-28T23:20:25.785 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-28T23:20:25.785 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-28T23:20:25.786 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-28T23:20:25.786 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi170.front.sepia.ceph.com, path=None, version=distro) 2024-03-28T23:20:25.786 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T23:20:25.867 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-28T23:20:25.867 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-28T23:20:25.869 DEBUG:teuthology.orchestra.run.smithi170:> sudo rpm -qi grub2-tools 2024-03-28T23:20:25.903 INFO:teuthology.orchestra.run.smithi170.stdout:Name : grub2-tools 2024-03-28T23:20:25.903 INFO:teuthology.orchestra.run.smithi170.stdout:Epoch : 1 2024-03-28T23:20:25.903 INFO:teuthology.orchestra.run.smithi170.stdout:Version : 2.06 2024-03-28T23:20:25.903 INFO:teuthology.orchestra.run.smithi170.stdout:Release : 68.el9 2024-03-28T23:20:25.903 INFO:teuthology.orchestra.run.smithi170.stdout:Architecture: x86_64 2024-03-28T23:20:25.904 INFO:teuthology.orchestra.run.smithi170.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-28T23:20:25.904 INFO:teuthology.orchestra.run.smithi170.stdout:Group : Unspecified 2024-03-28T23:20:25.904 INFO:teuthology.orchestra.run.smithi170.stdout:Size : 8264254 2024-03-28T23:20:25.904 INFO:teuthology.orchestra.run.smithi170.stdout:License : GPLv3+ 2024-03-28T23:20:25.904 INFO:teuthology.orchestra.run.smithi170.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-28T23:20:25.904 INFO:teuthology.orchestra.run.smithi170.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-28T23:20:25.904 INFO:teuthology.orchestra.run.smithi170.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-28T23:20:25.904 INFO:teuthology.orchestra.run.smithi170.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-28T23:20:25.904 INFO:teuthology.orchestra.run.smithi170.stdout:Packager : builder@centos.org 2024-03-28T23:20:25.904 INFO:teuthology.orchestra.run.smithi170.stdout:Vendor : CentOS 2024-03-28T23:20:25.904 INFO:teuthology.orchestra.run.smithi170.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-28T23:20:25.904 INFO:teuthology.orchestra.run.smithi170.stdout:Summary : Support tools for GRUB. 2024-03-28T23:20:25.904 INFO:teuthology.orchestra.run.smithi170.stdout:Description : 2024-03-28T23:20:25.904 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:20:25.904 INFO:teuthology.orchestra.run.smithi170.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-28T23:20:25.905 INFO:teuthology.orchestra.run.smithi170.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-28T23:20:25.905 INFO:teuthology.orchestra.run.smithi170.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-28T23:20:25.905 INFO:teuthology.orchestra.run.smithi170.stdout:hardware devices. 2024-03-28T23:20:25.905 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:20:25.905 INFO:teuthology.orchestra.run.smithi170.stdout:This subpackage provides tools for support of all platforms. 2024-03-28T23:20:25.906 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-28T23:20:25.906 INFO:teuthology.task.kernel:Updating grub on smithi170 to boot 5.14.0-434.el9.x86_64 2024-03-28T23:20:25.906 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-28T23:20:26.530 INFO:teuthology.orchestra.run.smithi170.stderr:Generating grub configuration file ... 2024-03-28T23:20:27.904 INFO:teuthology.orchestra.run.smithi170.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-28T23:20:27.940 INFO:teuthology.orchestra.run.smithi170.stderr:done 2024-03-28T23:20:27.943 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2024-03-28T23:20:27.958 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.Bvw0bPzf45 2024-03-28T23:20:27.958 DEBUG:teuthology.orchestra.run.smithi170:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Bvw0bPzf45 2024-03-28T23:20:28.030 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.Bvw0bPzf45 2024-03-28T23:20:28.171 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.Bvw0bPzf45 is 6KB 2024-03-28T23:20:28.190 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.Bvw0bPzf45 2024-03-28T23:20:28.204 DEBUG:teuthology.orchestra.run.smithi170:> sudo /bin/ls /boot/loader/entries || true 2024-03-28T23:20:28.270 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-28T23:20:28.270 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-28T23:20:28.270 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-28T23:20:28.270 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-28T23:20:28.271 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-28T23:20:28.429 DEBUG:teuthology.orchestra.run.smithi170:> sudo shutdown -r now 2024-03-28T23:20:47.967 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-28T23:20:47.967 INFO:teuthology.orchestra.run.smithi161.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-28T23:20:47.967 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:20:48.009 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-28T23:20:48.408 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-28T23:20:48.409 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-28T23:20:48.409 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-28T23:20:48.409 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-28T23:20:48.835 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-28T23:20:48.835 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:20:48.835 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-28T23:20:48.835 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-28T23:20:48.835 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-28T23:20:48.835 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-28T23:20:48.835 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-28T23:20:48.835 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:20:48.835 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-28T23:20:49.157 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T23:20:49.262 INFO:teuthology.orchestra.run.smithi161.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-28T23:20:49.263 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-28T23:20:49.263 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-28T23:20:49.263 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-28T23:20:49.263 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-28T23:20:49.263 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-28T23:20:49.263 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi161.front.sepia.ceph.com, path=None, version=distro) 2024-03-28T23:20:49.263 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T23:20:49.345 INFO:teuthology.orchestra.run.smithi161.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-28T23:20:49.346 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-28T23:20:49.346 DEBUG:teuthology.orchestra.run.smithi161:> sudo rpm -qi grub2-tools 2024-03-28T23:20:49.389 INFO:teuthology.orchestra.run.smithi161.stdout:Name : grub2-tools 2024-03-28T23:20:49.389 INFO:teuthology.orchestra.run.smithi161.stdout:Epoch : 1 2024-03-28T23:20:49.389 INFO:teuthology.orchestra.run.smithi161.stdout:Version : 2.06 2024-03-28T23:20:49.389 INFO:teuthology.orchestra.run.smithi161.stdout:Release : 68.el9 2024-03-28T23:20:49.389 INFO:teuthology.orchestra.run.smithi161.stdout:Architecture: x86_64 2024-03-28T23:20:49.389 INFO:teuthology.orchestra.run.smithi161.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-28T23:20:49.389 INFO:teuthology.orchestra.run.smithi161.stdout:Group : Unspecified 2024-03-28T23:20:49.389 INFO:teuthology.orchestra.run.smithi161.stdout:Size : 8264254 2024-03-28T23:20:49.389 INFO:teuthology.orchestra.run.smithi161.stdout:License : GPLv3+ 2024-03-28T23:20:49.390 INFO:teuthology.orchestra.run.smithi161.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-28T23:20:49.390 INFO:teuthology.orchestra.run.smithi161.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-28T23:20:49.390 INFO:teuthology.orchestra.run.smithi161.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-28T23:20:49.390 INFO:teuthology.orchestra.run.smithi161.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-28T23:20:49.390 INFO:teuthology.orchestra.run.smithi161.stdout:Packager : builder@centos.org 2024-03-28T23:20:49.390 INFO:teuthology.orchestra.run.smithi161.stdout:Vendor : CentOS 2024-03-28T23:20:49.390 INFO:teuthology.orchestra.run.smithi161.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-28T23:20:49.390 INFO:teuthology.orchestra.run.smithi161.stdout:Summary : Support tools for GRUB. 2024-03-28T23:20:49.390 INFO:teuthology.orchestra.run.smithi161.stdout:Description : 2024-03-28T23:20:49.390 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:20:49.390 INFO:teuthology.orchestra.run.smithi161.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-28T23:20:49.390 INFO:teuthology.orchestra.run.smithi161.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-28T23:20:49.390 INFO:teuthology.orchestra.run.smithi161.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-28T23:20:49.390 INFO:teuthology.orchestra.run.smithi161.stdout:hardware devices. 2024-03-28T23:20:49.391 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:20:49.391 INFO:teuthology.orchestra.run.smithi161.stdout:This subpackage provides tools for support of all platforms. 2024-03-28T23:20:49.392 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-28T23:20:49.392 INFO:teuthology.task.kernel:Updating grub on smithi161 to boot 5.14.0-434.el9.x86_64 2024-03-28T23:20:49.392 DEBUG:teuthology.orchestra.run.smithi161:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-28T23:20:50.015 INFO:teuthology.orchestra.run.smithi161.stderr:Generating grub configuration file ... 2024-03-28T23:20:51.311 INFO:teuthology.orchestra.run.smithi161.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-28T23:20:51.348 INFO:teuthology.orchestra.run.smithi161.stderr:done 2024-03-28T23:20:51.350 DEBUG:teuthology.orchestra.run.smithi161:> mktemp 2024-03-28T23:20:51.366 INFO:teuthology.orchestra.run.smithi161.stdout:/tmp/tmp.UlOeYGfhUS 2024-03-28T23:20:51.366 DEBUG:teuthology.orchestra.run.smithi161:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.UlOeYGfhUS 2024-03-28T23:20:51.435 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0666 /tmp/tmp.UlOeYGfhUS 2024-03-28T23:20:51.574 DEBUG:teuthology.orchestra.remote:smithi161:/tmp/tmp.UlOeYGfhUS is 6KB 2024-03-28T23:20:51.624 DEBUG:teuthology.orchestra.run.smithi161:> rm -fr /tmp/tmp.UlOeYGfhUS 2024-03-28T23:20:51.639 DEBUG:teuthology.orchestra.run.smithi161:> sudo /bin/ls /boot/loader/entries || true 2024-03-28T23:20:51.704 INFO:teuthology.orchestra.run.smithi161.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-28T23:20:51.704 INFO:teuthology.orchestra.run.smithi161.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-28T23:20:51.704 INFO:teuthology.orchestra.run.smithi161.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-28T23:20:51.704 INFO:teuthology.orchestra.run.smithi161.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-28T23:20:51.705 DEBUG:teuthology.orchestra.run.smithi161:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-28T23:20:51.856 DEBUG:teuthology.orchestra.run.smithi161:> sudo shutdown -r now 2024-03-28T23:20:58.439 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-28T23:20:58.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-28T23:20:58.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T23:21:21.872 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-28T23:21:21.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-03-28T23:21:21.874 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T23:21:56.511 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.161 2024-03-28T23:21:58.444 DEBUG:teuthology.orchestra.remote:timed out 2024-03-28T23:22:05.518 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-03-28T23:22:05.519 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T23:22:07.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-28T23:22:07.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T23:22:08.824 DEBUG:teuthology.orchestra.run.smithi170:> true 2024-03-28T23:22:09.368 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-28T23:22:09.368 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-28T23:22:09.368 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2024-03-28T23:22:09.382 INFO:teuthology.orchestra.run.smithi170.stdout:5.14.0-434.el9.x86_64 2024-03-28T23:22:09.383 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-28T23:22:09.383 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-28T23:22:09.383 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-28T23:22:10.383 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-28T23:22:10.384 DEBUG:teuthology.orchestra.run.smithi170:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-28T23:22:10.473 INFO:teuthology.orchestra.run.smithi170.stdout:ttyS1 2024-03-28T23:22:10.512 DEBUG:teuthology.parallel:result is None 2024-03-28T23:22:37.392 DEBUG:teuthology.orchestra.run.smithi161:> true 2024-03-28T23:22:37.846 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-03-28T23:22:37.846 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-28T23:22:37.846 DEBUG:teuthology.orchestra.run.smithi161:> uname -r 2024-03-28T23:22:37.861 INFO:teuthology.orchestra.run.smithi161.stdout:5.14.0-434.el9.x86_64 2024-03-28T23:22:37.861 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-28T23:22:37.861 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-28T23:22:37.861 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-28T23:22:38.862 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-28T23:22:38.862 DEBUG:teuthology.orchestra.run.smithi161:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-28T23:22:38.951 INFO:teuthology.orchestra.run.smithi161.stdout:ttyS1 2024-03-28T23:22:38.994 DEBUG:teuthology.parallel:result is None 2024-03-28T23:22:38.995 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-28T23:22:39.002 INFO:teuthology.task.internal:Creating test directory... 2024-03-28T23:22:39.003 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T23:22:39.005 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T23:22:39.024 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-28T23:22:39.030 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-28T23:22:39.039 INFO:teuthology.task.internal:Creating archive directory... 2024-03-28T23:22:39.040 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T23:22:39.059 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T23:22:39.095 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-28T23:22:39.101 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-28T23:22:39.102 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T23:22:39.116 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T23:22:39.151 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T23:22:39.164 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T23:22:39.173 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T23:22:39.187 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T23:22:39.189 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-28T23:22:39.195 INFO:teuthology.task.internal:Configuring sudo... 2024-03-28T23:22:39.195 DEBUG:teuthology.orchestra.run.smithi161:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T23:22:39.209 DEBUG:teuthology.orchestra.run.smithi170:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T23:22:39.259 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-28T23:22:39.268 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-28T23:22:39.268 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T23:22:39.276 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T23:22:39.316 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T23:22:39.368 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T23:22:39.416 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T23:22:39.497 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T23:22:39.540 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:22:39.540 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T23:22:39.609 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T23:22:39.653 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T23:22:39.730 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T23:22:39.770 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T23:22:39.818 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:22:39.818 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T23:22:39.888 DEBUG:teuthology.orchestra.run.smithi161:> sudo service rsyslog restart 2024-03-28T23:22:39.890 DEBUG:teuthology.orchestra.run.smithi170:> sudo service rsyslog restart 2024-03-28T23:22:39.937 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T23:22:39.982 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T23:22:40.328 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-28T23:22:40.337 INFO:teuthology.task.internal:Starting timer... 2024-03-28T23:22:40.337 INFO:teuthology.run_tasks:Running task pcp... 2024-03-28T23:22:40.350 INFO:teuthology.run_tasks:Running task selinux... 2024-03-28T23:22:40.358 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-28T23:22:40.358 DEBUG:teuthology.orchestra.run.smithi161:> sudo service auditd rotate 2024-03-28T23:22:40.416 INFO:teuthology.orchestra.run.smithi161.stdout:Rotating logs: 2024-03-28T23:22:40.417 DEBUG:teuthology.orchestra.run.smithi170:> sudo service auditd rotate 2024-03-28T23:22:40.470 INFO:teuthology.orchestra.run.smithi170.stdout:Rotating logs: 2024-03-28T23:22:40.471 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-28T23:22:40.472 DEBUG:teuthology.orchestra.run.smithi161:> /usr/sbin/getenforce 2024-03-28T23:22:40.493 INFO:teuthology.orchestra.run.smithi161.stdout:Permissive 2024-03-28T23:22:40.494 DEBUG:teuthology.orchestra.run.smithi170:> /usr/sbin/getenforce 2024-03-28T23:22:40.520 INFO:teuthology.orchestra.run.smithi170.stdout:Permissive 2024-03-28T23:22:40.521 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi161.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi170.front.sepia.ceph.com': 'permissive'} 2024-03-28T23:22:40.521 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T23:22:40.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:22:40.561 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T23:22:40.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:22:40.590 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-28T23:22:40.590 DEBUG:teuthology.orchestra.run.smithi161:> sudo /usr/sbin/setenforce permissive 2024-03-28T23:22:40.635 DEBUG:teuthology.orchestra.run.smithi170:> sudo /usr/sbin/setenforce permissive 2024-03-28T23:22:40.672 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-28T23:22:40.682 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-28T23:22:40.700 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-28T23:22:40.701 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi161.front.sepia.ceph.com,smithi170.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-28T23:27:02.089 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi161.front.sepia.ceph.com'), Remote(name='ubuntu@smithi170.front.sepia.ceph.com')] 2024-03-28T23:27:02.090 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-03-28T23:27:02.091 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T23:27:02.183 DEBUG:teuthology.orchestra.run.smithi161:> true 2024-03-28T23:27:02.260 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-03-28T23:27:02.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-28T23:27:02.261 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T23:27:02.348 DEBUG:teuthology.orchestra.run.smithi170:> true 2024-03-28T23:27:02.429 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-28T23:27:02.430 INFO:teuthology.run_tasks:Running task clock... 2024-03-28T23:27:02.440 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-28T23:27:02.440 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T23:27:02.440 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T23:27:02.443 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T23:27:02.443 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T23:27:02.471 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T23:27:02.487 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T23:27:02.503 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T23:27:02.516 INFO:teuthology.orchestra.run.smithi161.stderr:sudo: ntpd: command not found 2024-03-28T23:27:02.520 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T23:27:02.529 INFO:teuthology.orchestra.run.smithi161.stdout:506 Cannot talk to daemon 2024-03-28T23:27:02.546 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T23:27:02.549 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: ntpd: command not found 2024-03-28T23:27:02.562 INFO:teuthology.orchestra.run.smithi170.stdout:506 Cannot talk to daemon 2024-03-28T23:27:02.563 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T23:27:02.579 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T23:27:02.595 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T23:27:02.619 INFO:teuthology.orchestra.run.smithi161.stderr:bash: line 1: ntpq: command not found 2024-03-28T23:27:02.622 INFO:teuthology.orchestra.run.smithi161.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T23:27:02.623 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================== 2024-03-28T23:27:02.623 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T23:27:02.623 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T23:27:02.623 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T23:27:02.623 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T23:27:02.649 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2024-03-28T23:27:02.653 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T23:27:02.653 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2024-03-28T23:27:02.653 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T23:27:02.653 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T23:27:02.653 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T23:27:02.654 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T23:27:02.655 INFO:teuthology.run_tasks:Running task install... 2024-03-28T23:27:02.663 DEBUG:teuthology.task.install:project ceph 2024-03-28T23:27:02.663 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '323d23dc228bbcb7f16f188fdc9c0dcb801844be'}} 2024-03-28T23:27:02.663 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '323d23dc228bbcb7f16f188fdc9c0dcb801844be'} 2024-03-28T23:27:02.663 INFO:teuthology.task.install:Using flavor: default 2024-03-28T23:27:02.669 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-28T23:27:02.669 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-28T23:27:02.669 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '323d23dc228bbcb7f16f188fdc9c0dcb801844be', 'tag': None, 'wait_for_package': False} 2024-03-28T23:27:02.670 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T23:27:02.675 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '323d23dc228bbcb7f16f188fdc9c0dcb801844be', 'tag': None, 'wait_for_package': False} 2024-03-28T23:27:02.675 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T23:27:02.813 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/ 2024-03-28T23:27:02.813 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1899.g323d23dc 2024-03-28T23:27:02.824 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/ 2024-03-28T23:27:02.824 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1899.g323d23dc 2024-03-28T23:27:02.936 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T23:27:02.936 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:27:02.936 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T23:27:02.964 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T23:27:02.964 DEBUG:teuthology.orchestra.run.smithi170:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/323d23dc228bbcb7f16f188fdc9c0dcb801844be/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T23:27:02.988 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T23:27:02.988 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:27:02.988 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T23:27:03.017 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T23:27:03.017 DEBUG:teuthology.orchestra.run.smithi161:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/323d23dc228bbcb7f16f188fdc9c0dcb801844be/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T23:27:03.031 DEBUG:teuthology.orchestra.run.smithi170:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T23:27:03.087 DEBUG:teuthology.orchestra.run.smithi161:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T23:27:03.112 DEBUG:teuthology.orchestra.run.smithi170:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T23:27:03.168 DEBUG:teuthology.orchestra.run.smithi161:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T23:27:03.178 INFO:teuthology.orchestra.run.smithi170.stdout:check_obsoletes = 1 2024-03-28T23:27:03.180 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum clean all 2024-03-28T23:27:03.234 INFO:teuthology.orchestra.run.smithi161.stdout:check_obsoletes = 1 2024-03-28T23:27:03.235 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum clean all 2024-03-28T23:27:03.432 INFO:teuthology.orchestra.run.smithi170.stdout:74 files removed 2024-03-28T23:27:03.459 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T23:27:03.474 INFO:teuthology.orchestra.run.smithi161.stdout:74 files removed 2024-03-28T23:27:03.502 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T23:27:04.171 INFO:teuthology.orchestra.run.smithi161.stdout:ceph packages for x86_64 227 kB/s | 83 kB 00:00 2024-03-28T23:27:04.173 INFO:teuthology.orchestra.run.smithi170.stdout:ceph packages for x86_64 201 kB/s | 83 kB 00:00 2024-03-28T23:27:04.684 INFO:teuthology.orchestra.run.smithi170.stdout:ceph noarch packages 25 kB/s | 11 kB 00:00 2024-03-28T23:27:04.705 INFO:teuthology.orchestra.run.smithi161.stdout:ceph noarch packages 24 kB/s | 11 kB 00:00 2024-03-28T23:27:04.993 INFO:teuthology.orchestra.run.smithi170.stdout:ceph source packages 7.1 kB/s | 1.9 kB 00:00 2024-03-28T23:27:05.039 INFO:teuthology.orchestra.run.smithi161.stdout:ceph source packages 6.4 kB/s | 1.9 kB 00:00 2024-03-28T23:27:05.412 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-28T23:27:05.466 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-28T23:27:06.887 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-28T23:27:07.175 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-28T23:27:09.299 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - BaseOS 4.5 MB/s | 8.0 MB 00:01 2024-03-28T23:27:11.377 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-03-28T23:27:15.331 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-28T23:27:16.527 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - BaseOS 884 kB/s | 8.0 MB 00:09 2024-03-28T23:27:16.942 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 16 kB 00:00 2024-03-28T23:27:18.219 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - AppStream 32 MB/s | 19 MB 00:00 2024-03-28T23:27:19.069 INFO:teuthology.orchestra.run.smithi161.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-03-28T23:27:24.219 INFO:teuthology.orchestra.run.smithi161.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-28T23:27:24.372 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.1 MB 00:02 2024-03-28T23:27:26.220 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 16 kB 00:00 2024-03-28T23:27:26.728 INFO:teuthology.orchestra.run.smithi161.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T23:27:26.729 INFO:teuthology.orchestra.run.smithi161.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T23:27:26.742 INFO:teuthology.orchestra.run.smithi161.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T23:27:26.743 INFO:teuthology.orchestra.run.smithi161.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T23:27:26.744 INFO:teuthology.orchestra.run.smithi161.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T23:27:26.745 INFO:teuthology.orchestra.run.smithi161.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T23:27:26.747 INFO:teuthology.orchestra.run.smithi161.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T23:27:26.750 INFO:teuthology.orchestra.run.smithi161.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T23:27:26.751 INFO:teuthology.orchestra.run.smithi161.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T23:27:26.832 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-28T23:27:26.850 INFO:teuthology.orchestra.run.smithi161.stdout:============================================================================================ 2024-03-28T23:27:26.850 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-03-28T23:27:26.850 INFO:teuthology.orchestra.run.smithi161.stdout:============================================================================================ 2024-03-28T23:27:26.850 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-28T23:27:26.850 INFO:teuthology.orchestra.run.smithi161.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T23:27:26.850 INFO:teuthology.orchestra.run.smithi161.stdout: ceph x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 6.6 k 2024-03-28T23:27:26.850 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 5.3 M 2024-03-28T23:27:26.850 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 889 k 2024-03-28T23:27:26.850 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 145 k 2024-03-28T23:27:26.850 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 1.5 M 2024-03-28T23:27:26.850 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 138 k 2024-03-28T23:27:26.851 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 3.5 M 2024-03-28T23:27:26.851 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 7.4 M 2024-03-28T23:27:26.851 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-rook noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 49 k 2024-03-28T23:27:26.851 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 9.8 M 2024-03-28T23:27:26.851 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 47 M 2024-03-28T23:27:26.851 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-volume noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 264 k 2024-03-28T23:27:26.851 INFO:teuthology.orchestra.run.smithi161.stdout: cephadm noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 748 k 2024-03-28T23:27:26.851 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-mirror x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 221 k 2024-03-28T23:27:26.851 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-shell x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 27 k 2024-03-28T23:27:26.851 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-top noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 18 k 2024-03-28T23:27:26.851 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T23:27:26.851 INFO:teuthology.orchestra.run.smithi161.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T23:27:26.851 INFO:teuthology.orchestra.run.smithi161.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T23:27:26.852 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-devel x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 32 k 2024-03-28T23:27:26.852 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 746 k 2024-03-28T23:27:26.852 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 127 k 2024-03-28T23:27:26.852 INFO:teuthology.orchestra.run.smithi161.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T23:27:26.852 INFO:teuthology.orchestra.run.smithi161.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T23:27:26.852 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T23:27:26.852 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-03-28T23:27:26.852 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-03-28T23:27:26.852 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-03-28T23:27:26.852 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 162 k 2024-03-28T23:27:26.852 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 321 k 2024-03-28T23:27:26.852 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 299 k 2024-03-28T23:27:26.853 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 100 k 2024-03-28T23:27:26.853 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 86 k 2024-03-28T23:27:26.853 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 3.1 M 2024-03-28T23:27:26.853 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 171 k 2024-03-28T23:27:26.853 INFO:teuthology.orchestra.run.smithi161.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T23:27:26.853 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrading: 2024-03-28T23:27:26.853 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager x86_64 1:1.47.2-1.el9 CentOS-BaseOS 2.3 M 2024-03-28T23:27:26.853 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 CentOS-BaseOS 1.8 M 2024-03-28T23:27:26.853 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 CentOS-BaseOS 41 k 2024-03-28T23:27:26.853 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 CentOS-BaseOS 247 k 2024-03-28T23:27:26.853 INFO:teuthology.orchestra.run.smithi161.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T23:27:26.853 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T23:27:26.854 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T23:27:26.854 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T23:27:26.854 INFO:teuthology.orchestra.run.smithi161.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T23:27:26.854 INFO:teuthology.orchestra.run.smithi161.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T23:27:26.854 INFO:teuthology.orchestra.run.smithi161.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T23:27:26.854 INFO:teuthology.orchestra.run.smithi161.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T23:27:26.854 INFO:teuthology.orchestra.run.smithi161.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T23:27:26.854 INFO:teuthology.orchestra.run.smithi161.stdout: librados2 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 3.4 M 2024-03-28T23:27:26.854 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 3.1 M 2024-03-28T23:27:26.854 INFO:teuthology.orchestra.run.smithi161.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T23:27:26.854 INFO:teuthology.orchestra.run.smithi161.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T23:27:26.854 INFO:teuthology.orchestra.run.smithi161.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T23:27:26.854 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T23:27:26.855 INFO:teuthology.orchestra.run.smithi161.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T23:27:26.855 INFO:teuthology.orchestra.run.smithi161.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T23:27:26.855 INFO:teuthology.orchestra.run.smithi161.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T23:27:26.855 INFO:teuthology.orchestra.run.smithi161.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T23:27:26.855 INFO:teuthology.orchestra.run.smithi161.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T23:27:26.855 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-03-28T23:27:26.855 INFO:teuthology.orchestra.run.smithi161.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T23:27:26.855 INFO:teuthology.orchestra.run.smithi161.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T23:27:26.855 INFO:teuthology.orchestra.run.smithi161.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T23:27:26.855 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 21 M 2024-03-28T23:27:26.855 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 31 k 2024-03-28T23:27:26.856 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 2.4 M 2024-03-28T23:27:26.856 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 248 k 2024-03-28T23:27:26.856 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 4.7 M 2024-03-28T23:27:26.856 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 17 M 2024-03-28T23:27:26.856 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 15 k 2024-03-28T23:27:26.856 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-selinux x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 25 k 2024-03-28T23:27:26.856 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T23:27:26.856 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T23:27:26.856 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T23:27:26.856 INFO:teuthology.orchestra.run.smithi161.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T23:27:26.856 INFO:teuthology.orchestra.run.smithi161.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T23:27:26.856 INFO:teuthology.orchestra.run.smithi161.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T23:27:26.857 INFO:teuthology.orchestra.run.smithi161.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T23:27:26.857 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T23:27:26.857 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T23:27:26.857 INFO:teuthology.orchestra.run.smithi161.stdout: libcephsqlite x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 163 k 2024-03-28T23:27:26.857 INFO:teuthology.orchestra.run.smithi161.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T23:27:26.857 INFO:teuthology.orchestra.run.smithi161.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T23:27:26.857 INFO:teuthology.orchestra.run.smithi161.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T23:27:26.857 INFO:teuthology.orchestra.run.smithi161.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T23:27:26.857 INFO:teuthology.orchestra.run.smithi161.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T23:27:26.857 INFO:teuthology.orchestra.run.smithi161.stdout: libradosstriper1 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 501 k 2024-03-28T23:27:26.857 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T23:27:26.857 INFO:teuthology.orchestra.run.smithi161.stdout: librgw2 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 5.1 M 2024-03-28T23:27:26.857 INFO:teuthology.orchestra.run.smithi161.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T23:27:26.857 INFO:teuthology.orchestra.run.smithi161.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T23:27:26.858 INFO:teuthology.orchestra.run.smithi161.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T23:27:26.858 INFO:teuthology.orchestra.run.smithi161.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T23:27:26.858 INFO:teuthology.orchestra.run.smithi161.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T23:27:26.858 INFO:teuthology.orchestra.run.smithi161.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T23:27:26.858 INFO:teuthology.orchestra.run.smithi161.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T23:27:26.858 INFO:teuthology.orchestra.run.smithi161.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T23:27:26.858 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T23:27:26.858 INFO:teuthology.orchestra.run.smithi161.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T23:27:26.858 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-03-28T23:27:26.858 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T23:27:26.858 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T23:27:26.858 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T23:27:26.858 INFO:teuthology.orchestra.run.smithi161.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T23:27:26.859 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T23:27:26.859 INFO:teuthology.orchestra.run.smithi161.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T23:27:26.859 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T23:27:26.859 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 45 k 2024-03-28T23:27:26.859 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 131 k 2024-03-28T23:27:26.859 INFO:teuthology.orchestra.run.smithi161.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T23:27:26.859 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T23:27:26.859 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T23:27:26.859 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T23:27:26.859 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T23:27:26.859 INFO:teuthology.orchestra.run.smithi161.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T23:27:26.859 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T23:27:26.860 INFO:teuthology.orchestra.run.smithi161.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T23:27:26.860 INFO:teuthology.orchestra.run.smithi161.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T23:27:26.860 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T23:27:26.860 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T23:27:26.860 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T23:27:26.860 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T23:27:26.860 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T23:27:26.860 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T23:27:26.860 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T23:27:26.860 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T23:27:26.860 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T23:27:26.860 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T23:27:26.860 INFO:teuthology.orchestra.run.smithi161.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T23:27:26.861 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T23:27:26.861 INFO:teuthology.orchestra.run.smithi161.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T23:27:26.861 INFO:teuthology.orchestra.run.smithi161.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T23:27:26.861 INFO:teuthology.orchestra.run.smithi161.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T23:27:26.861 INFO:teuthology.orchestra.run.smithi161.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T23:27:26.861 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T23:27:26.861 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T23:27:26.861 INFO:teuthology.orchestra.run.smithi161.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T23:27:26.861 INFO:teuthology.orchestra.run.smithi161.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T23:27:26.861 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T23:27:26.861 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T23:27:26.861 INFO:teuthology.orchestra.run.smithi161.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T23:27:26.861 INFO:teuthology.orchestra.run.smithi161.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T23:27:26.862 INFO:teuthology.orchestra.run.smithi161.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T23:27:26.862 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T23:27:26.862 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T23:27:26.862 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T23:27:26.862 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T23:27:26.862 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T23:27:26.862 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T23:27:26.862 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T23:27:26.862 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T23:27:26.862 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T23:27:26.862 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T23:27:26.862 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T23:27:26.862 INFO:teuthology.orchestra.run.smithi161.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T23:27:26.862 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T23:27:26.863 INFO:teuthology.orchestra.run.smithi161.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T23:27:26.863 INFO:teuthology.orchestra.run.smithi161.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T23:27:26.863 INFO:teuthology.orchestra.run.smithi161.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T23:27:26.863 INFO:teuthology.orchestra.run.smithi161.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T23:27:26.863 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T23:27:26.863 INFO:teuthology.orchestra.run.smithi161.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T23:27:26.863 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T23:27:26.863 INFO:teuthology.orchestra.run.smithi161.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T23:27:26.863 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-28T23:27:26.863 INFO:teuthology.orchestra.run.smithi161.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-28T23:27:26.863 INFO:teuthology.orchestra.run.smithi161.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T23:27:26.863 INFO:teuthology.orchestra.run.smithi161.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T23:27:26.863 INFO:teuthology.orchestra.run.smithi161.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T23:27:26.863 INFO:teuthology.orchestra.run.smithi161.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T23:27:26.864 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T23:27:26.864 INFO:teuthology.orchestra.run.smithi161.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T23:27:26.864 INFO:teuthology.orchestra.run.smithi161.stdout:Installing weak dependencies: 2024-03-28T23:27:26.864 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-28T23:27:26.864 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:27:26.864 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-28T23:27:26.864 INFO:teuthology.orchestra.run.smithi161.stdout:============================================================================================ 2024-03-28T23:27:26.864 INFO:teuthology.orchestra.run.smithi161.stdout:Install 150 Packages 2024-03-28T23:27:26.864 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrade 24 Packages 2024-03-28T23:27:26.864 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:27:26.865 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 226 M 2024-03-28T23:27:26.865 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-28T23:27:27.645 INFO:teuthology.orchestra.run.smithi161.stdout:(1/174): ceph-19.0.0-1899.g323d23dc.el9.x86_64. 17 kB/s | 6.6 kB 00:00 2024-03-28T23:27:28.466 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 9.8 MB/s | 21 MB 00:02 2024-03-28T23:27:29.203 INFO:teuthology.orchestra.run.smithi161.stdout:(2/174): ceph-fuse-19.0.0-1899.g323d23dc.el9.x8 571 kB/s | 889 kB 00:01 2024-03-28T23:27:29.437 INFO:teuthology.orchestra.run.smithi161.stdout:(3/174): ceph-immutable-object-cache-19.0.0-189 621 kB/s | 145 kB 00:00 2024-03-28T23:27:29.845 INFO:teuthology.orchestra.run.smithi161.stdout:(4/174): ceph-base-19.0.0-1899.g323d23dc.el9.x8 2.1 MB/s | 5.3 MB 00:02 2024-03-28T23:27:30.270 INFO:teuthology.orchestra.run.smithi161.stdout:(5/174): ceph-mgr-19.0.0-1899.g323d23dc.el9.x86 3.5 MB/s | 1.5 MB 00:00 2024-03-28T23:27:31.646 INFO:teuthology.orchestra.run.smithi161.stdout:(6/174): ceph-mon-19.0.0-1899.g323d23dc.el9.x86 3.4 MB/s | 4.7 MB 00:01 2024-03-28T23:27:31.913 INFO:teuthology.orchestra.run.smithi161.stdout:(7/174): ceph-common-19.0.0-1899.g323d23dc.el9. 4.4 MB/s | 21 MB 00:04 2024-03-28T23:27:31.963 INFO:teuthology.orchestra.run.smithi161.stdout:(8/174): ceph-mds-19.0.0-1899.g323d23dc.el9.x86 957 kB/s | 2.4 MB 00:02 2024-03-28T23:27:32.063 INFO:teuthology.orchestra.run.smithi161.stdout:(9/174): ceph-selinux-19.0.0-1899.g323d23dc.el9 252 kB/s | 25 kB 00:00 2024-03-28T23:27:33.439 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-28T23:27:34.238 INFO:teuthology.orchestra.run.smithi161.stdout:(10/174): ceph-osd-19.0.0-1899.g323d23dc.el9.x8 6.5 MB/s | 17 MB 00:02 2024-03-28T23:27:34.338 INFO:teuthology.orchestra.run.smithi161.stdout:(11/174): ceph-radosgw-19.0.0-1899.g323d23dc.el 4.0 MB/s | 9.8 MB 00:02 2024-03-28T23:27:34.363 INFO:teuthology.orchestra.run.smithi161.stdout:(12/174): cephfs-mirror-19.0.0-1899.g323d23dc.e 1.7 MB/s | 221 kB 00:00 2024-03-28T23:27:34.389 INFO:teuthology.orchestra.run.smithi161.stdout:(13/174): cephfs-shell-19.0.0-1899.g323d23dc.el 542 kB/s | 27 kB 00:00 2024-03-28T23:27:34.439 INFO:teuthology.orchestra.run.smithi161.stdout:(14/174): libcephfs-devel-19.0.0-1899.g323d23dc 422 kB/s | 32 kB 00:00 2024-03-28T23:27:34.506 INFO:teuthology.orchestra.run.smithi161.stdout:(15/174): libcephsqlite-19.0.0-1899.g323d23dc.e 2.4 MB/s | 163 kB 00:00 2024-03-28T23:27:34.540 INFO:teuthology.orchestra.run.smithi161.stdout:(16/174): libcephfs2-19.0.0-1899.g323d23dc.el9. 4.8 MB/s | 746 kB 00:00 2024-03-28T23:27:34.590 INFO:teuthology.orchestra.run.smithi161.stdout:(17/174): librados-devel-19.0.0-1899.g323d23dc. 1.5 MB/s | 127 kB 00:00 2024-03-28T23:27:34.640 INFO:teuthology.orchestra.run.smithi161.stdout:(18/174): libradosstriper1-19.0.0-1899.g323d23d 4.9 MB/s | 501 kB 00:00 2024-03-28T23:27:34.698 INFO:teuthology.orchestra.run.smithi161.stdout:(19/174): python3-ceph-argparse-19.0.0-1899.g32 776 kB/s | 45 kB 00:00 2024-03-28T23:27:34.774 INFO:teuthology.orchestra.run.smithi161.stdout:(20/174): python3-ceph-common-19.0.0-1899.g323d 1.7 MB/s | 131 kB 00:00 2024-03-28T23:27:34.824 INFO:teuthology.orchestra.run.smithi161.stdout:(21/174): python3-cephfs-19.0.0-1899.g323d23dc. 3.2 MB/s | 162 kB 00:00 2024-03-28T23:27:34.925 INFO:teuthology.orchestra.run.smithi161.stdout:(22/174): python3-rados-19.0.0-1899.g323d23dc.e 3.1 MB/s | 321 kB 00:00 2024-03-28T23:27:35.000 INFO:teuthology.orchestra.run.smithi161.stdout:(23/174): python3-rbd-19.0.0-1899.g323d23dc.el9 3.9 MB/s | 299 kB 00:00 2024-03-28T23:27:35.050 INFO:teuthology.orchestra.run.smithi161.stdout:(24/174): python3-rgw-19.0.0-1899.g323d23dc.el9 1.9 MB/s | 100 kB 00:00 2024-03-28T23:27:35.092 INFO:teuthology.orchestra.run.smithi161.stdout:(25/174): rbd-fuse-19.0.0-1899.g323d23dc.el9.x8 2.0 MB/s | 86 kB 00:00 2024-03-28T23:27:35.176 INFO:teuthology.orchestra.run.smithi161.stdout:(26/174): librgw2-19.0.0-1899.g323d23dc.el9.x86 8.7 MB/s | 5.1 MB 00:00 2024-03-28T23:27:35.242 INFO:teuthology.orchestra.run.smithi161.stdout:(27/174): rbd-nbd-19.0.0-1899.g323d23dc.el9.x86 2.5 MB/s | 171 kB 00:00 2024-03-28T23:27:35.318 INFO:teuthology.orchestra.run.smithi161.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1899.g 419 kB/s | 31 kB 00:00 2024-03-28T23:27:35.385 INFO:teuthology.orchestra.run.smithi161.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1899.g323d23d 2.0 MB/s | 138 kB 00:00 2024-03-28T23:27:35.577 INFO:teuthology.orchestra.run.smithi161.stdout:(30/174): rbd-mirror-19.0.0-1899.g323d23dc.el9. 6.4 MB/s | 3.1 MB 00:00 2024-03-28T23:27:35.846 INFO:teuthology.orchestra.run.smithi161.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1899.g323d2 7.7 MB/s | 3.5 MB 00:00 2024-03-28T23:27:35.911 INFO:teuthology.orchestra.run.smithi161.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1899.g32 3.7 MB/s | 248 kB 00:00 2024-03-28T23:27:35.977 INFO:teuthology.orchestra.run.smithi161.stdout:(33/174): ceph-mgr-rook-19.0.0-1899.g323d23dc.e 742 kB/s | 49 kB 00:00 2024-03-28T23:27:35.995 INFO:teuthology.orchestra.run.smithi170.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T23:27:35.996 INFO:teuthology.orchestra.run.smithi170.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T23:27:36.010 INFO:teuthology.orchestra.run.smithi170.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T23:27:36.011 INFO:teuthology.orchestra.run.smithi170.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T23:27:36.012 INFO:teuthology.orchestra.run.smithi170.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T23:27:36.013 INFO:teuthology.orchestra.run.smithi170.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T23:27:36.015 INFO:teuthology.orchestra.run.smithi170.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T23:27:36.018 INFO:teuthology.orchestra.run.smithi170.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T23:27:36.019 INFO:teuthology.orchestra.run.smithi170.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T23:27:36.044 INFO:teuthology.orchestra.run.smithi161.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1899.g3 230 kB/s | 15 kB 00:00 2024-03-28T23:27:36.100 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-28T23:27:36.112 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2024-03-28T23:27:36.112 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-28T23:27:36.112 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2024-03-28T23:27:36.112 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-28T23:27:36.112 INFO:teuthology.orchestra.run.smithi170.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T23:27:36.112 INFO:teuthology.orchestra.run.smithi170.stdout: ceph x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 6.6 k 2024-03-28T23:27:36.112 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 5.3 M 2024-03-28T23:27:36.113 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 889 k 2024-03-28T23:27:36.113 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 145 k 2024-03-28T23:27:36.113 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 1.5 M 2024-03-28T23:27:36.113 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 138 k 2024-03-28T23:27:36.113 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 3.5 M 2024-03-28T23:27:36.113 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 7.4 M 2024-03-28T23:27:36.113 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 49 k 2024-03-28T23:27:36.113 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 9.8 M 2024-03-28T23:27:36.113 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 47 M 2024-03-28T23:27:36.113 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-volume noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 264 k 2024-03-28T23:27:36.113 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 748 k 2024-03-28T23:27:36.113 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-mirror x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 221 k 2024-03-28T23:27:36.113 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-shell x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 27 k 2024-03-28T23:27:36.114 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-top noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 18 k 2024-03-28T23:27:36.114 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T23:27:36.114 INFO:teuthology.orchestra.run.smithi170.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T23:27:36.114 INFO:teuthology.orchestra.run.smithi170.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T23:27:36.114 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 32 k 2024-03-28T23:27:36.114 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 746 k 2024-03-28T23:27:36.114 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 127 k 2024-03-28T23:27:36.114 INFO:teuthology.orchestra.run.smithi170.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T23:27:36.114 INFO:teuthology.orchestra.run.smithi170.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T23:27:36.114 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T23:27:36.114 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-03-28T23:27:36.114 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-03-28T23:27:36.115 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-03-28T23:27:36.115 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 162 k 2024-03-28T23:27:36.115 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 321 k 2024-03-28T23:27:36.115 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 299 k 2024-03-28T23:27:36.115 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 100 k 2024-03-28T23:27:36.115 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 86 k 2024-03-28T23:27:36.115 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 3.1 M 2024-03-28T23:27:36.115 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 171 k 2024-03-28T23:27:36.115 INFO:teuthology.orchestra.run.smithi170.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T23:27:36.115 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrading: 2024-03-28T23:27:36.115 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager x86_64 1:1.47.2-1.el9 CentOS-BaseOS 2.3 M 2024-03-28T23:27:36.115 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 CentOS-BaseOS 1.8 M 2024-03-28T23:27:36.115 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 CentOS-BaseOS 41 k 2024-03-28T23:27:36.115 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 CentOS-BaseOS 247 k 2024-03-28T23:27:36.115 INFO:teuthology.orchestra.run.smithi170.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T23:27:36.116 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T23:27:36.116 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T23:27:36.116 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T23:27:36.116 INFO:teuthology.orchestra.run.smithi170.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T23:27:36.116 INFO:teuthology.orchestra.run.smithi170.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T23:27:36.116 INFO:teuthology.orchestra.run.smithi170.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T23:27:36.116 INFO:teuthology.orchestra.run.smithi170.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T23:27:36.116 INFO:teuthology.orchestra.run.smithi170.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T23:27:36.116 INFO:teuthology.orchestra.run.smithi170.stdout: librados2 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 3.4 M 2024-03-28T23:27:36.116 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 3.1 M 2024-03-28T23:27:36.116 INFO:teuthology.orchestra.run.smithi170.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T23:27:36.116 INFO:teuthology.orchestra.run.smithi170.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T23:27:36.116 INFO:teuthology.orchestra.run.smithi170.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T23:27:36.116 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T23:27:36.117 INFO:teuthology.orchestra.run.smithi170.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T23:27:36.117 INFO:teuthology.orchestra.run.smithi170.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T23:27:36.117 INFO:teuthology.orchestra.run.smithi170.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T23:27:36.117 INFO:teuthology.orchestra.run.smithi170.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T23:27:36.117 INFO:teuthology.orchestra.run.smithi170.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T23:27:36.117 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-28T23:27:36.117 INFO:teuthology.orchestra.run.smithi170.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T23:27:36.117 INFO:teuthology.orchestra.run.smithi170.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T23:27:36.117 INFO:teuthology.orchestra.run.smithi170.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T23:27:36.117 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 21 M 2024-03-28T23:27:36.117 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 31 k 2024-03-28T23:27:36.117 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 2.4 M 2024-03-28T23:27:36.117 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 248 k 2024-03-28T23:27:36.117 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 4.7 M 2024-03-28T23:27:36.118 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 17 M 2024-03-28T23:27:36.118 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 15 k 2024-03-28T23:27:36.118 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 25 k 2024-03-28T23:27:36.118 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T23:27:36.118 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T23:27:36.118 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T23:27:36.118 INFO:teuthology.orchestra.run.smithi170.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T23:27:36.118 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T23:27:36.118 INFO:teuthology.orchestra.run.smithi170.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T23:27:36.118 INFO:teuthology.orchestra.run.smithi170.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T23:27:36.118 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T23:27:36.118 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T23:27:36.118 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 163 k 2024-03-28T23:27:36.119 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T23:27:36.119 INFO:teuthology.orchestra.run.smithi170.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T23:27:36.119 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T23:27:36.119 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T23:27:36.119 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T23:27:36.119 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 501 k 2024-03-28T23:27:36.119 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T23:27:36.119 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 5.1 M 2024-03-28T23:27:36.119 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T23:27:36.119 INFO:teuthology.orchestra.run.smithi170.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T23:27:36.119 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T23:27:36.119 INFO:teuthology.orchestra.run.smithi170.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T23:27:36.119 INFO:teuthology.orchestra.run.smithi170.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T23:27:36.119 INFO:teuthology.orchestra.run.smithi170.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T23:27:36.120 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T23:27:36.120 INFO:teuthology.orchestra.run.smithi170.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T23:27:36.120 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T23:27:36.120 INFO:teuthology.orchestra.run.smithi170.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T23:27:36.120 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-03-28T23:27:36.120 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T23:27:36.120 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T23:27:36.120 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T23:27:36.120 INFO:teuthology.orchestra.run.smithi170.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T23:27:36.120 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T23:27:36.120 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T23:27:36.120 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T23:27:36.120 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 45 k 2024-03-28T23:27:36.120 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 131 k 2024-03-28T23:27:36.121 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T23:27:36.121 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T23:27:36.121 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T23:27:36.121 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T23:27:36.121 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T23:27:36.121 INFO:teuthology.orchestra.run.smithi170.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T23:27:36.121 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T23:27:36.121 INFO:teuthology.orchestra.run.smithi170.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T23:27:36.121 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T23:27:36.121 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T23:27:36.121 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T23:27:36.121 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T23:27:36.121 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T23:27:36.122 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T23:27:36.122 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T23:27:36.122 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T23:27:36.122 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T23:27:36.122 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T23:27:36.122 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T23:27:36.122 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T23:27:36.122 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T23:27:36.122 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T23:27:36.122 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T23:27:36.122 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T23:27:36.122 INFO:teuthology.orchestra.run.smithi170.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T23:27:36.122 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T23:27:36.122 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T23:27:36.123 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T23:27:36.123 INFO:teuthology.orchestra.run.smithi170.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T23:27:36.123 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T23:27:36.123 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T23:27:36.123 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T23:27:36.123 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T23:27:36.123 INFO:teuthology.orchestra.run.smithi170.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T23:27:36.123 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T23:27:36.123 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T23:27:36.123 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T23:27:36.123 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T23:27:36.123 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T23:27:36.123 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T23:27:36.123 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T23:27:36.124 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T23:27:36.124 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T23:27:36.124 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T23:27:36.124 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T23:27:36.124 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T23:27:36.124 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T23:27:36.124 INFO:teuthology.orchestra.run.smithi170.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T23:27:36.124 INFO:teuthology.orchestra.run.smithi170.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T23:27:36.124 INFO:teuthology.orchestra.run.smithi170.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T23:27:36.124 INFO:teuthology.orchestra.run.smithi170.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T23:27:36.124 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T23:27:36.124 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T23:27:36.124 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T23:27:36.124 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T23:27:36.125 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-28T23:27:36.125 INFO:teuthology.orchestra.run.smithi170.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-28T23:27:36.125 INFO:teuthology.orchestra.run.smithi170.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T23:27:36.125 INFO:teuthology.orchestra.run.smithi170.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T23:27:36.125 INFO:teuthology.orchestra.run.smithi170.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T23:27:36.125 INFO:teuthology.orchestra.run.smithi170.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T23:27:36.125 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T23:27:36.125 INFO:teuthology.orchestra.run.smithi170.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T23:27:36.125 INFO:teuthology.orchestra.run.smithi170.stdout:Installing weak dependencies: 2024-03-28T23:27:36.126 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-28T23:27:36.126 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:27:36.126 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-28T23:27:36.126 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2024-03-28T23:27:36.126 INFO:teuthology.orchestra.run.smithi170.stdout:Install 150 Packages 2024-03-28T23:27:36.126 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrade 24 Packages 2024-03-28T23:27:36.126 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:27:36.127 INFO:teuthology.orchestra.run.smithi161.stdout:(35/174): ceph-volume-19.0.0-1899.g323d23dc.el9 3.4 MB/s | 264 kB 00:00 2024-03-28T23:27:36.127 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 226 M 2024-03-28T23:27:36.127 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-28T23:27:36.255 INFO:teuthology.orchestra.run.smithi161.stdout:(36/174): cephadm-19.0.0-1899.g323d23dc.el9.noa 5.5 MB/s | 748 kB 00:00 2024-03-28T23:27:36.320 INFO:teuthology.orchestra.run.smithi161.stdout:(37/174): cephfs-top-19.0.0-1899.g323d23dc.el9. 274 kB/s | 18 kB 00:00 2024-03-28T23:27:36.521 INFO:teuthology.orchestra.run.smithi161.stdout:(38/174): ceph-mgr-diskprediction-local-19.0.0- 7.8 MB/s | 7.4 MB 00:00 2024-03-28T23:27:37.486 INFO:teuthology.orchestra.run.smithi170.stdout:(1/174): ceph-19.0.0-1899.g323d23dc.el9.x86_64. 40 kB/s | 6.6 kB 00:00 2024-03-28T23:27:37.489 INFO:teuthology.orchestra.run.smithi161.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 662 kB 00:00 2024-03-28T23:27:37.489 INFO:teuthology.orchestra.run.smithi161.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 6.8 MB/s | 920 kB 00:00 2024-03-28T23:27:37.490 INFO:teuthology.orchestra.run.smithi161.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 106 kB 00:00 2024-03-28T23:27:37.490 INFO:teuthology.orchestra.run.smithi161.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 4.5 MB/s | 307 kB 00:00 2024-03-28T23:27:37.490 INFO:teuthology.orchestra.run.smithi161.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 403 kB/s | 32 kB 00:00 2024-03-28T23:27:37.490 INFO:teuthology.orchestra.run.smithi161.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 14 MB/s | 3.0 MB 00:00 2024-03-28T23:27:37.490 INFO:teuthology.orchestra.run.smithi161.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 87 kB/s | 17 kB 00:00 2024-03-28T23:27:37.490 INFO:teuthology.orchestra.run.smithi161.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 200 kB/s | 10 kB 00:00 2024-03-28T23:27:37.490 INFO:teuthology.orchestra.run.smithi161.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-28T23:27:37.490 INFO:teuthology.orchestra.run.smithi161.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 886 kB/s | 44 kB 00:00 2024-03-28T23:27:37.490 INFO:teuthology.orchestra.run.smithi161.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-28T23:27:37.490 INFO:teuthology.orchestra.run.smithi161.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.9 MB/s | 578 kB 00:00 2024-03-28T23:27:37.491 INFO:teuthology.orchestra.run.smithi161.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 704 kB/s | 46 kB 00:00 2024-03-28T23:27:37.491 INFO:teuthology.orchestra.run.smithi161.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-28T23:27:37.491 INFO:teuthology.orchestra.run.smithi161.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-28T23:27:37.491 INFO:teuthology.orchestra.run.smithi161.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-28T23:27:37.491 INFO:teuthology.orchestra.run.smithi161.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 685 kB/s | 34 kB 00:00 2024-03-28T23:27:37.491 INFO:teuthology.orchestra.run.smithi161.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 20 MB/s | 4.7 MB 00:00 2024-03-28T23:27:37.491 INFO:teuthology.orchestra.run.smithi161.stdout:(57/174): postgresql-13.14-1.el9.x86_64.rpm 5.9 MB/s | 1.5 MB 00:00 2024-03-28T23:27:37.498 INFO:teuthology.orchestra.run.smithi170.stdout:(2/174): ceph-base-19.0.0-1899.g323d23dc.el9.x8 8.6 MB/s | 5.3 MB 00:00 2024-03-28T23:27:37.518 INFO:teuthology.orchestra.run.smithi161.stdout:(58/174): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 817 kB 00:00 2024-03-28T23:27:37.523 INFO:teuthology.orchestra.run.smithi170.stdout:(3/174): ceph-fuse-19.0.0-1899.g323d23dc.el9.x8 1.8 MB/s | 889 kB 00:00 2024-03-28T23:27:37.543 INFO:teuthology.orchestra.run.smithi161.stdout:(59/174): postgresql-private-libs-13.14-1.el9.x 2.3 MB/s | 132 kB 00:00 2024-03-28T23:27:37.548 INFO:teuthology.orchestra.run.smithi170.stdout:(4/174): ceph-immutable-object-cache-19.0.0-189 2.8 MB/s | 145 kB 00:00 2024-03-28T23:27:37.626 INFO:teuthology.orchestra.run.smithi161.stdout:(60/174): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-03-28T23:27:37.674 INFO:teuthology.orchestra.run.smithi170.stdout:(5/174): ceph-mgr-19.0.0-1899.g323d23dc.el9.x86 12 MB/s | 1.5 MB 00:00 2024-03-28T23:27:37.768 INFO:teuthology.orchestra.run.smithi161.stdout:(61/174): postgresql-server-13.14-1.el9.x86_64. 23 MB/s | 5.7 MB 00:00 2024-03-28T23:27:37.785 INFO:teuthology.orchestra.run.smithi161.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 497 kB/s | 78 kB 00:00 2024-03-28T23:27:37.919 INFO:teuthology.orchestra.run.smithi161.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 39 MB/s | 5.8 MB 00:00 2024-03-28T23:27:37.944 INFO:teuthology.orchestra.run.smithi161.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 155 kB/s | 24 kB 00:00 2024-03-28T23:27:37.969 INFO:teuthology.orchestra.run.smithi161.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 361 kB/s | 18 kB 00:00 2024-03-28T23:27:37.995 INFO:teuthology.orchestra.run.smithi161.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-28T23:27:38.007 INFO:teuthology.orchestra.run.smithi170.stdout:(6/174): ceph-mon-19.0.0-1899.g323d23dc.el9.x86 14 MB/s | 4.7 MB 00:00 2024-03-28T23:27:38.020 INFO:teuthology.orchestra.run.smithi161.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 632 kB/s | 32 kB 00:00 2024-03-28T23:27:38.094 INFO:teuthology.orchestra.run.smithi170.stdout:(7/174): ceph-mds-19.0.0-1899.g323d23dc.el9.x86 4.2 MB/s | 2.4 MB 00:00 2024-03-28T23:27:38.112 INFO:teuthology.orchestra.run.smithi161.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 41 MB/s | 4.8 MB 00:00 2024-03-28T23:27:38.137 INFO:teuthology.orchestra.run.smithi161.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.9 MB/s | 217 kB 00:00 2024-03-28T23:27:38.162 INFO:teuthology.orchestra.run.smithi161.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-28T23:27:38.187 INFO:teuthology.orchestra.run.smithi161.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.4 MB/s | 69 kB 00:00 2024-03-28T23:27:38.204 INFO:teuthology.orchestra.run.smithi161.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 989 kB/s | 41 kB 00:00 2024-03-28T23:27:38.230 INFO:teuthology.orchestra.run.smithi161.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 5.6 MB/s | 238 kB 00:00 2024-03-28T23:27:38.255 INFO:teuthology.orchestra.run.smithi161.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-28T23:27:38.280 INFO:teuthology.orchestra.run.smithi161.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-28T23:27:38.306 INFO:teuthology.orchestra.run.smithi161.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 948 kB/s | 47 kB 00:00 2024-03-28T23:27:38.331 INFO:teuthology.orchestra.run.smithi161.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 854 kB/s | 43 kB 00:00 2024-03-28T23:27:38.556 INFO:teuthology.orchestra.run.smithi161.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 174 kB/s | 39 kB 00:00 2024-03-28T23:27:38.707 INFO:teuthology.orchestra.run.smithi161.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-03-28T23:27:38.732 INFO:teuthology.orchestra.run.smithi161.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 235 kB/s | 41 kB 00:00 2024-03-28T23:27:38.766 INFO:teuthology.orchestra.run.smithi161.stdout:(81/174): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 220 kB 00:00 2024-03-28T23:27:38.791 INFO:teuthology.orchestra.run.smithi161.stdout:(82/174): qatlib-service-24.02.0-1.el9.x86_64.r 594 kB/s | 35 kB 00:00 2024-03-28T23:27:38.933 INFO:teuthology.orchestra.run.smithi161.stdout:(83/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 277 kB/s | 46 kB 00:00 2024-03-28T23:27:38.966 INFO:teuthology.orchestra.run.smithi161.stdout:(84/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.7 MB/s | 300 kB 00:00 2024-03-28T23:27:38.992 INFO:teuthology.orchestra.run.smithi161.stdout:(85/174): uuid-1.6.2-55.el9.x86_64.rpm 969 kB/s | 56 kB 00:00 2024-03-28T23:27:39.017 INFO:teuthology.orchestra.run.smithi161.stdout:(86/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-28T23:27:39.042 INFO:teuthology.orchestra.run.smithi161.stdout:(87/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-28T23:27:39.059 INFO:teuthology.orchestra.run.smithi161.stdout:(88/174): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-03-28T23:27:39.084 INFO:teuthology.orchestra.run.smithi161.stdout:(89/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-28T23:27:39.118 INFO:teuthology.orchestra.run.smithi161.stdout:(90/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-28T23:27:39.135 INFO:teuthology.orchestra.run.smithi161.stdout:(91/174): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 103 kB 00:00 2024-03-28T23:27:39.160 INFO:teuthology.orchestra.run.smithi161.stdout:(92/174): python3-pycparser-2.20-6.el9.noarch.r 2.9 MB/s | 124 kB 00:00 2024-03-28T23:27:39.186 INFO:teuthology.orchestra.run.smithi161.stdout:(93/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-28T23:27:39.319 INFO:teuthology.orchestra.run.smithi161.stdout:(94/174): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-28T23:27:39.369 INFO:teuthology.orchestra.run.smithi161.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 4.3 MB/s | 803 kB 00:00 2024-03-28T23:27:39.395 INFO:teuthology.orchestra.run.smithi161.stdout:(96/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 195 kB 00:00 2024-03-28T23:27:39.420 INFO:teuthology.orchestra.run.smithi161.stdout:(97/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-28T23:27:39.504 INFO:teuthology.orchestra.run.smithi161.stdout:(98/174): elfutils-libelf-devel-0.190-2.el9.x86 198 kB/s | 21 kB 00:00 2024-03-28T23:27:39.529 INFO:teuthology.orchestra.run.smithi161.stdout:(99/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 234 kB/s | 25 kB 00:00 2024-03-28T23:27:39.637 INFO:teuthology.orchestra.run.smithi161.stdout:(100/174): python3-devel-3.9.17-2.el9.x86_64.rp 2.2 MB/s | 245 kB 00:00 2024-03-28T23:27:39.643 INFO:teuthology.orchestra.run.smithi170.stdout:(8/174): ceph-osd-19.0.0-1899.g323d23dc.el9.x86 10 MB/s | 17 MB 00:01 2024-03-28T23:27:39.683 INFO:teuthology.orchestra.run.smithi170.stdout:(9/174): ceph-selinux-19.0.0-1899.g323d23dc.el9 630 kB/s | 25 kB 00:00 2024-03-28T23:27:39.688 INFO:teuthology.orchestra.run.smithi161.stdout:(101/174): python3-jinja2-2.11.3-5.el9.noarch.r 4.8 MB/s | 248 kB 00:00 2024-03-28T23:27:39.738 INFO:teuthology.orchestra.run.smithi161.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-28T23:27:39.888 INFO:teuthology.orchestra.run.smithi161.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 149 kB/s | 22 kB 00:00 2024-03-28T23:27:39.892 INFO:teuthology.orchestra.run.smithi170.stdout:(10/174): ceph-common-19.0.0-1899.g323d23dc.el9 6.8 MB/s | 21 MB 00:03 2024-03-28T23:27:39.976 INFO:teuthology.orchestra.run.smithi170.stdout:(11/174): cephfs-mirror-19.0.0-1899.g323d23dc.e 2.6 MB/s | 221 kB 00:00 2024-03-28T23:27:40.051 INFO:teuthology.orchestra.run.smithi170.stdout:(12/174): cephfs-shell-19.0.0-1899.g323d23dc.el 361 kB/s | 27 kB 00:00 2024-03-28T23:27:40.122 INFO:teuthology.orchestra.run.smithi161.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.6 MB/s | 863 kB 00:00 2024-03-28T23:27:40.126 INFO:teuthology.orchestra.run.smithi170.stdout:(13/174): libcephfs-devel-19.0.0-1899.g323d23dc 423 kB/s | 32 kB 00:00 2024-03-28T23:27:40.285 INFO:teuthology.orchestra.run.smithi170.stdout:(14/174): libcephfs2-19.0.0-1899.g323d23dc.el9. 4.6 MB/s | 746 kB 00:00 2024-03-28T23:27:40.377 INFO:teuthology.orchestra.run.smithi170.stdout:(15/174): libcephsqlite-19.0.0-1899.g323d23dc.e 1.7 MB/s | 163 kB 00:00 2024-03-28T23:27:40.447 INFO:teuthology.orchestra.run.smithi161.stdout:(105/174): openssl-devel-3.0.7-27.el9.x86_64.rp 4.4 MB/s | 4.1 MB 00:00 2024-03-28T23:27:40.502 INFO:teuthology.orchestra.run.smithi170.stdout:(16/174): librados-devel-19.0.0-1899.g323d23dc. 1.0 MB/s | 127 kB 00:00 2024-03-28T23:27:40.602 INFO:teuthology.orchestra.run.smithi170.stdout:(17/174): libradosstriper1-19.0.0-1899.g323d23d 4.9 MB/s | 501 kB 00:00 2024-03-28T23:27:40.711 INFO:teuthology.orchestra.run.smithi170.stdout:(18/174): ceph-radosgw-19.0.0-1899.g323d23dc.el 3.7 MB/s | 9.8 MB 00:02 2024-03-28T23:27:40.764 INFO:teuthology.orchestra.run.smithi161.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 854 kB/s | 548 kB 00:00 2024-03-28T23:27:40.811 INFO:teuthology.orchestra.run.smithi170.stdout:(19/174): python3-ceph-argparse-19.0.0-1899.g32 452 kB/s | 45 kB 00:00 2024-03-28T23:27:40.864 INFO:teuthology.orchestra.run.smithi161.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 195 kB/s | 19 kB 00:00 2024-03-28T23:27:40.998 INFO:teuthology.orchestra.run.smithi161.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 561 kB/s | 309 kB 00:00 2024-03-28T23:27:41.098 INFO:teuthology.orchestra.run.smithi161.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-03-28T23:27:41.128 INFO:teuthology.orchestra.run.smithi170.stdout:(20/174): librgw2-19.0.0-1899.g323d23dc.el9.x86 9.7 MB/s | 5.1 MB 00:00 2024-03-28T23:27:41.198 INFO:teuthology.orchestra.run.smithi161.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 486 kB/s | 49 kB 00:00 2024-03-28T23:27:41.212 INFO:teuthology.orchestra.run.smithi170.stdout:(21/174): python3-cephfs-19.0.0-1899.g323d23dc. 1.9 MB/s | 162 kB 00:00 2024-03-28T23:27:41.274 INFO:teuthology.orchestra.run.smithi161.stdout:(111/174): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-03-28T23:27:41.295 INFO:teuthology.orchestra.run.smithi170.stdout:(22/174): python3-rados-19.0.0-1899.g323d23dc.e 3.8 MB/s | 321 kB 00:00 2024-03-28T23:27:41.307 INFO:teuthology.orchestra.run.smithi161.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 619 kB/s | 67 kB 00:00 2024-03-28T23:27:41.337 INFO:teuthology.orchestra.run.smithi170.stdout:(23/174): python3-ceph-common-19.0.0-1899.g323d 250 kB/s | 131 kB 00:00 2024-03-28T23:27:41.374 INFO:teuthology.orchestra.run.smithi161.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-03-28T23:27:41.379 INFO:teuthology.orchestra.run.smithi170.stdout:(24/174): python3-rbd-19.0.0-1899.g323d23dc.el9 3.5 MB/s | 299 kB 00:00 2024-03-28T23:27:41.421 INFO:teuthology.orchestra.run.smithi170.stdout:(25/174): python3-rgw-19.0.0-1899.g323d23dc.el9 1.2 MB/s | 100 kB 00:00 2024-03-28T23:27:41.455 INFO:teuthology.orchestra.run.smithi170.stdout:(26/174): rbd-fuse-19.0.0-1899.g323d23dc.el9.x8 1.1 MB/s | 86 kB 00:00 2024-03-28T23:27:41.499 INFO:teuthology.orchestra.run.smithi161.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 4.3 MB/s | 548 kB 00:00 2024-03-28T23:27:41.538 INFO:teuthology.orchestra.run.smithi170.stdout:(27/174): rbd-nbd-19.0.0-1899.g323d23dc.el9.x86 2.0 MB/s | 171 kB 00:00 2024-03-28T23:27:41.550 INFO:teuthology.orchestra.run.smithi161.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.4 MB/s | 838 kB 00:00 2024-03-28T23:27:41.600 INFO:teuthology.orchestra.run.smithi161.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 433 kB/s | 43 kB 00:00 2024-03-28T23:27:41.614 INFO:teuthology.orchestra.run.smithi170.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1899.g 420 kB/s | 31 kB 00:00 2024-03-28T23:27:41.650 INFO:teuthology.orchestra.run.smithi161.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 321 kB/s | 32 kB 00:00 2024-03-28T23:27:41.697 INFO:teuthology.orchestra.run.smithi170.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1899.g323d23d 1.6 MB/s | 138 kB 00:00 2024-03-28T23:27:41.700 INFO:teuthology.orchestra.run.smithi161.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 150 kB/s | 15 kB 00:00 2024-03-28T23:27:41.759 INFO:teuthology.orchestra.run.smithi161.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-03-28T23:27:41.801 INFO:teuthology.orchestra.run.smithi161.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-03-28T23:27:41.876 INFO:teuthology.orchestra.run.smithi161.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-03-28T23:27:41.902 INFO:teuthology.orchestra.run.smithi161.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 352 kB/s | 35 kB 00:00 2024-03-28T23:27:41.985 INFO:teuthology.orchestra.run.smithi161.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 1.8 MB/s | 203 kB 00:00 2024-03-28T23:27:42.052 INFO:teuthology.orchestra.run.smithi161.stdout:(124/174): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-03-28T23:27:42.094 INFO:teuthology.orchestra.run.smithi161.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-03-28T23:27:42.153 INFO:teuthology.orchestra.run.smithi161.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 107 kB/s | 11 kB 00:00 2024-03-28T23:27:42.172 INFO:teuthology.orchestra.run.smithi170.stdout:(30/174): ceph-mgr-dashboard-19.0.0-1899.g323d2 7.4 MB/s | 3.5 MB 00:00 2024-03-28T23:27:42.194 INFO:teuthology.orchestra.run.smithi161.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 177 kB/s | 18 kB 00:00 2024-03-28T23:27:42.278 INFO:teuthology.orchestra.run.smithi161.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 185 kB/s | 23 kB 00:00 2024-03-28T23:27:42.312 INFO:teuthology.orchestra.run.smithi161.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 166 kB/s | 19 kB 00:00 2024-03-28T23:27:42.382 INFO:teuthology.orchestra.run.smithi161.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-03-28T23:27:42.488 INFO:teuthology.orchestra.run.smithi161.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 5.8 MB/s | 1.0 MB 00:00 2024-03-28T23:27:42.513 INFO:teuthology.orchestra.run.smithi161.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 347 kB/s | 46 kB 00:00 2024-03-28T23:27:42.596 INFO:teuthology.orchestra.run.smithi161.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 727 kB/s | 79 kB 00:00 2024-03-28T23:27:42.622 INFO:teuthology.orchestra.run.smithi161.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 532 kB/s | 58 kB 00:00 2024-03-28T23:27:42.631 INFO:teuthology.orchestra.run.smithi170.stdout:(31/174): ceph-test-19.0.0-1899.g323d23dc.el9.x 16 MB/s | 47 MB 00:02 2024-03-28T23:27:42.681 INFO:teuthology.orchestra.run.smithi170.stdout:(32/174): rbd-mirror-19.0.0-1899.g323d23dc.el9. 2.5 MB/s | 3.1 MB 00:01 2024-03-28T23:27:42.706 INFO:teuthology.orchestra.run.smithi170.stdout:(33/174): ceph-mgr-modules-core-19.0.0-1899.g32 3.2 MB/s | 248 kB 00:00 2024-03-28T23:27:42.714 INFO:teuthology.orchestra.run.smithi161.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-03-28T23:27:42.732 INFO:teuthology.orchestra.run.smithi170.stdout:(34/174): ceph-mgr-rook-19.0.0-1899.g323d23dc.e 986 kB/s | 49 kB 00:00 2024-03-28T23:27:42.739 INFO:teuthology.orchestra.run.smithi161.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 140 kB/s | 16 kB 00:00 2024-03-28T23:27:42.757 INFO:teuthology.orchestra.run.smithi170.stdout:(35/174): ceph-prometheus-alerts-19.0.0-1899.g3 306 kB/s | 15 kB 00:00 2024-03-28T23:27:42.816 INFO:teuthology.orchestra.run.smithi170.stdout:(36/174): ceph-volume-19.0.0-1899.g323d23dc.el9 3.1 MB/s | 264 kB 00:00 2024-03-28T23:27:42.831 INFO:teuthology.orchestra.run.smithi161.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 771 kB/s | 90 kB 00:00 2024-03-28T23:27:42.856 INFO:teuthology.orchestra.run.smithi161.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 241 kB/s | 28 kB 00:00 2024-03-28T23:27:42.874 INFO:teuthology.orchestra.run.smithi170.stdout:(37/174): cephadm-19.0.0-1899.g323d23dc.el9.noa 6.3 MB/s | 748 kB 00:00 2024-03-28T23:27:42.891 INFO:teuthology.orchestra.run.smithi170.stdout:(38/174): cephfs-top-19.0.0-1899.g323d23dc.el9. 243 kB/s | 18 kB 00:00 2024-03-28T23:27:42.931 INFO:teuthology.orchestra.run.smithi161.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 307 kB/s | 31 kB 00:00 2024-03-28T23:27:42.965 INFO:teuthology.orchestra.run.smithi161.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-03-28T23:27:43.040 INFO:teuthology.orchestra.run.smithi161.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 544 kB/s | 59 kB 00:00 2024-03-28T23:27:43.050 INFO:teuthology.orchestra.run.smithi170.stdout:(39/174): ceph-mgr-diskprediction-local-19.0.0- 8.4 MB/s | 7.4 MB 00:00 2024-03-28T23:27:43.065 INFO:teuthology.orchestra.run.smithi161.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 356 kB/s | 36 kB 00:00 2024-03-28T23:27:43.141 INFO:teuthology.orchestra.run.smithi161.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 508 kB/s | 51 kB 00:00 2024-03-28T23:27:43.183 INFO:teuthology.orchestra.run.smithi161.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-03-28T23:27:43.242 INFO:teuthology.orchestra.run.smithi170.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 2.6 MB/s | 920 kB 00:00 2024-03-28T23:27:43.250 INFO:teuthology.orchestra.run.smithi161.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 825 kB/s | 90 kB 00:00 2024-03-28T23:27:43.300 INFO:teuthology.orchestra.run.smithi170.stdout:(41/174): automake-1.16.2-8.el9.noarch.rpm 1.5 MB/s | 662 kB 00:00 2024-03-28T23:27:43.308 INFO:teuthology.orchestra.run.smithi161.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-03-28T23:27:43.326 INFO:teuthology.orchestra.run.smithi170.stdout:(42/174): boost-program-options-1.75.0-8.el9.x8 386 kB/s | 106 kB 00:00 2024-03-28T23:27:43.350 INFO:teuthology.orchestra.run.smithi161.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 200 kB/s | 20 kB 00:00 2024-03-28T23:27:43.367 INFO:teuthology.orchestra.run.smithi170.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 2.4 MB/s | 307 kB 00:00 2024-03-28T23:27:43.384 INFO:teuthology.orchestra.run.smithi170.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 380 kB/s | 32 kB 00:00 2024-03-28T23:27:43.417 INFO:teuthology.orchestra.run.smithi161.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-28T23:27:43.542 INFO:teuthology.orchestra.run.smithi161.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 8.2 MB/s | 1.6 MB 00:00 2024-03-28T23:27:43.601 INFO:teuthology.orchestra.run.smithi170.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 73 kB/s | 17 kB 00:00 2024-03-28T23:27:43.618 INFO:teuthology.orchestra.run.smithi170.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 43 kB/s | 10 kB 00:00 2024-03-28T23:27:43.844 INFO:teuthology.orchestra.run.smithi170.stdout:(47/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.8 MB/s | 3.0 MB 00:00 2024-03-28T23:27:43.860 INFO:teuthology.orchestra.run.smithi170.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 183 kB/s | 44 kB 00:00 2024-03-28T23:27:43.893 INFO:teuthology.orchestra.run.smithi161.stdout:(150/174): librados2-19.0.0-1899.g323d23dc.el9. 7.1 MB/s | 3.4 MB 00:00 2024-03-28T23:27:43.903 INFO:teuthology.orchestra.run.smithi170.stdout:(49/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 561 kB/s | 159 kB 00:00 2024-03-28T23:27:43.953 INFO:teuthology.orchestra.run.smithi170.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 5.9 MB/s | 661 kB 00:00 2024-03-28T23:27:44.003 INFO:teuthology.orchestra.run.smithi170.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 4.0 MB/s | 578 kB 00:00 2024-03-28T23:27:44.020 INFO:teuthology.orchestra.run.smithi170.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 398 kB/s | 46 kB 00:00 2024-03-28T23:27:44.026 INFO:teuthology.orchestra.run.smithi161.stdout:(151/174): librbd1-19.0.0-1899.g323d23dc.el9.x8 6.5 MB/s | 3.1 MB 00:00 2024-03-28T23:27:44.045 INFO:teuthology.orchestra.run.smithi170.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-28T23:27:44.071 INFO:teuthology.orchestra.run.smithi170.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2024-03-28T23:27:44.076 INFO:teuthology.orchestra.run.smithi161.stdout:(152/174): NetworkManager-1.47.2-1.el9.x86_64.r 12 MB/s | 2.3 MB 00:00 2024-03-28T23:27:44.096 INFO:teuthology.orchestra.run.smithi170.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-28T23:27:44.112 INFO:teuthology.orchestra.run.smithi170.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 514 kB/s | 34 kB 00:00 2024-03-28T23:27:44.144 INFO:teuthology.orchestra.run.smithi161.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 16 MB/s | 1.8 MB 00:00 2024-03-28T23:27:44.169 INFO:teuthology.orchestra.run.smithi161.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 462 kB/s | 41 kB 00:00 2024-03-28T23:27:44.194 INFO:teuthology.orchestra.run.smithi161.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 4.8 MB/s | 247 kB 00:00 2024-03-28T23:27:44.219 INFO:teuthology.orchestra.run.smithi161.stdout:(156/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-28T23:27:44.244 INFO:teuthology.orchestra.run.smithi161.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-03-28T23:27:44.270 INFO:teuthology.orchestra.run.smithi161.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 728 kB/s | 36 kB 00:00 2024-03-28T23:27:44.295 INFO:teuthology.orchestra.run.smithi161.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.7 MB/s | 192 kB 00:00 2024-03-28T23:27:44.321 INFO:teuthology.orchestra.run.smithi161.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 5.0 MB/s | 256 kB 00:00 2024-03-28T23:27:44.345 INFO:teuthology.orchestra.run.smithi161.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 461 kB/s | 23 kB 00:00 2024-03-28T23:27:44.354 INFO:teuthology.orchestra.run.smithi170.stdout:(57/174): postgresql-contrib-13.14-1.el9.x86_64 3.3 MB/s | 817 kB 00:00 2024-03-28T23:27:44.371 INFO:teuthology.orchestra.run.smithi161.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 108 kB 00:00 2024-03-28T23:27:44.388 INFO:teuthology.orchestra.run.smithi170.stdout:(58/174): postgresql-13.14-1.el9.x86_64.rpm 5.6 MB/s | 1.5 MB 00:00 2024-03-28T23:27:44.396 INFO:teuthology.orchestra.run.smithi161.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 155 kB 00:00 2024-03-28T23:27:44.421 INFO:teuthology.orchestra.run.smithi161.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 136 kB 00:00 2024-03-28T23:27:44.447 INFO:teuthology.orchestra.run.smithi161.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-03-28T23:27:44.455 INFO:teuthology.orchestra.run.smithi170.stdout:(59/174): openblas-openmp-0.3.21-2.el9.x86_64.r 12 MB/s | 4.7 MB 00:00 2024-03-28T23:27:44.472 INFO:teuthology.orchestra.run.smithi170.stdout:(60/174): postgresql-private-libs-13.14-1.el9.x 1.1 MB/s | 132 kB 00:00 2024-03-28T23:27:44.472 INFO:teuthology.orchestra.run.smithi161.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 572 kB/s | 29 kB 00:00 2024-03-28T23:27:44.555 INFO:teuthology.orchestra.run.smithi161.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 11 MB/s | 1.2 MB 00:00 2024-03-28T23:27:44.597 INFO:teuthology.orchestra.run.smithi161.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 17 MB/s | 2.2 MB 00:00 2024-03-28T23:27:44.605 INFO:teuthology.orchestra.run.smithi170.stdout:(61/174): postgresql-server-13.14-1.el9.x86_64. 26 MB/s | 5.7 MB 00:00 2024-03-28T23:27:44.639 INFO:teuthology.orchestra.run.smithi170.stdout:(62/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.1 MB/s | 1.0 MB 00:00 2024-03-28T23:27:44.664 INFO:teuthology.orchestra.run.smithi170.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 410 kB/s | 78 kB 00:00 2024-03-28T23:27:44.673 INFO:teuthology.orchestra.run.smithi161.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 20 MB/s | 2.3 MB 00:00 2024-03-28T23:27:44.698 INFO:teuthology.orchestra.run.smithi161.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 4.5 MB/s | 465 kB 00:00 2024-03-28T23:27:44.723 INFO:teuthology.orchestra.run.smithi161.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 622 kB/s | 31 kB 00:00 2024-03-28T23:27:44.757 INFO:teuthology.orchestra.run.smithi161.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2024-03-28T23:27:44.764 INFO:teuthology.orchestra.run.smithi170.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 36 MB/s | 5.8 MB 00:00 2024-03-28T23:27:44.790 INFO:teuthology.orchestra.run.smithi170.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 147 kB/s | 18 kB 00:00 2024-03-28T23:27:44.815 INFO:teuthology.orchestra.run.smithi170.stdout:(66/174): python3-jsonpatch-1.21-16.el9.noarch. 165 kB/s | 24 kB 00:00 2024-03-28T23:27:44.824 INFO:teuthology.orchestra.run.smithi161.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 741 kB/s | 74 kB 00:00 2024-03-28T23:27:44.840 INFO:teuthology.orchestra.run.smithi170.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-28T23:27:44.865 INFO:teuthology.orchestra.run.smithi170.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 419 kB/s | 32 kB 00:00 2024-03-28T23:27:44.890 INFO:teuthology.orchestra.run.smithi170.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 217 kB 00:00 2024-03-28T23:27:44.966 INFO:teuthology.orchestra.run.smithi170.stdout:(70/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-28T23:27:44.991 INFO:teuthology.orchestra.run.smithi170.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.4 MB/s | 174 kB 00:00 2024-03-28T23:27:45.016 INFO:teuthology.orchestra.run.smithi170.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 559 kB/s | 69 kB 00:00 2024-03-28T23:27:45.042 INFO:teuthology.orchestra.run.smithi170.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 550 kB/s | 41 kB 00:00 2024-03-28T23:27:45.067 INFO:teuthology.orchestra.run.smithi170.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 4.7 MB/s | 238 kB 00:00 2024-03-28T23:27:45.092 INFO:teuthology.orchestra.run.smithi170.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-28T23:27:45.117 INFO:teuthology.orchestra.run.smithi170.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-03-28T23:27:45.134 INFO:teuthology.orchestra.run.smithi170.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 43 kB 00:00 2024-03-28T23:27:45.151 INFO:teuthology.orchestra.run.smithi170.stdout:(78/174): python3-pytz-2021.1-5.el9.noarch.rpm 812 kB/s | 47 kB 00:00 2024-03-28T23:27:45.201 INFO:teuthology.orchestra.run.smithi170.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 589 kB/s | 39 kB 00:00 2024-03-28T23:27:45.435 INFO:teuthology.orchestra.run.smithi170.stdout:(80/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 49 MB/s | 16 MB 00:00 2024-03-28T23:27:45.452 INFO:teuthology.orchestra.run.smithi170.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 137 kB/s | 41 kB 00:00 2024-03-28T23:27:45.477 INFO:teuthology.orchestra.run.smithi170.stdout:(82/174): qatlib-24.02.0-1.el9.x86_64.rpm 804 kB/s | 220 kB 00:00 2024-03-28T23:27:45.494 INFO:teuthology.orchestra.run.smithi170.stdout:(83/174): qatlib-service-24.02.0-1.el9.x86_64.r 597 kB/s | 35 kB 00:00 2024-03-28T23:27:45.511 INFO:teuthology.orchestra.run.smithi170.stdout:(84/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-28T23:27:45.537 INFO:teuthology.orchestra.run.smithi170.stdout:(85/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-28T23:27:45.553 INFO:teuthology.orchestra.run.smithi170.stdout:(86/174): uuid-1.6.2-55.el9.x86_64.rpm 971 kB/s | 56 kB 00:00 2024-03-28T23:27:45.570 INFO:teuthology.orchestra.run.smithi170.stdout:(87/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-28T23:27:45.587 INFO:teuthology.orchestra.run.smithi170.stdout:(88/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 881 kB/s | 44 kB 00:00 2024-03-28T23:27:45.612 INFO:teuthology.orchestra.run.smithi170.stdout:(89/174): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2024-03-28T23:27:45.637 INFO:teuthology.orchestra.run.smithi170.stdout:(90/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-28T23:27:45.663 INFO:teuthology.orchestra.run.smithi170.stdout:(91/174): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.1 MB 00:00 2024-03-28T23:27:45.688 INFO:teuthology.orchestra.run.smithi170.stdout:(92/174): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-28T23:27:45.722 INFO:teuthology.orchestra.run.smithi170.stdout:(93/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.1 MB/s | 187 kB 00:00 2024-03-28T23:27:45.738 INFO:teuthology.orchestra.run.smithi170.stdout:(94/174): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 124 kB 00:00 2024-03-28T23:27:45.847 INFO:teuthology.orchestra.run.smithi170.stdout:(95/174): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-28T23:27:45.872 INFO:teuthology.orchestra.run.smithi170.stdout:(96/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-03-28T23:27:45.898 INFO:teuthology.orchestra.run.smithi170.stdout:(97/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-28T23:27:45.931 INFO:teuthology.orchestra.run.smithi170.stdout:(98/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 803 kB 00:00 2024-03-28T23:27:46.023 INFO:teuthology.orchestra.run.smithi170.stdout:(99/174): elfutils-libelf-devel-0.190-2.el9.x86 143 kB/s | 21 kB 00:00 2024-03-28T23:27:46.048 INFO:teuthology.orchestra.run.smithi170.stdout:(100/174): libuuid-devel-2.37.4-18.el9.x86_64.r 169 kB/s | 25 kB 00:00 2024-03-28T23:27:46.207 INFO:teuthology.orchestra.run.smithi170.stdout:(101/174): python3-devel-3.9.17-2.el9.x86_64.rp 1.5 MB/s | 245 kB 00:00 2024-03-28T23:27:46.232 INFO:teuthology.orchestra.run.smithi170.stdout:(102/174): python3-jinja2-2.11.3-5.el9.noarch.r 1.3 MB/s | 248 kB 00:00 2024-03-28T23:27:46.266 INFO:teuthology.orchestra.run.smithi170.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-03-28T23:27:46.525 INFO:teuthology.orchestra.run.smithi170.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 76 kB/s | 22 kB 00:00 2024-03-28T23:27:46.625 INFO:teuthology.orchestra.run.smithi170.stdout:(105/174): openssl-devel-3.0.7-27.el9.x86_64.rp 6.0 MB/s | 4.1 MB 00:00 2024-03-28T23:27:46.983 INFO:teuthology.orchestra.run.smithi170.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.2 MB/s | 863 kB 00:00 2024-03-28T23:27:47.200 INFO:teuthology.orchestra.run.smithi170.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 537 kB/s | 309 kB 00:00 2024-03-28T23:27:47.284 INFO:teuthology.orchestra.run.smithi170.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 65 kB/s | 19 kB 00:00 2024-03-28T23:27:47.317 INFO:teuthology.orchestra.run.smithi170.stdout:(109/174): abseil-cpp-20211102.0-3.el9.x86_64.r 692 kB/s | 548 kB 00:00 2024-03-28T23:27:47.401 INFO:teuthology.orchestra.run.smithi170.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 222 kB/s | 26 kB 00:00 2024-03-28T23:27:47.434 INFO:teuthology.orchestra.run.smithi170.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 415 kB/s | 49 kB 00:00 2024-03-28T23:27:47.585 INFO:teuthology.orchestra.run.smithi170.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-28T23:27:47.618 INFO:teuthology.orchestra.run.smithi170.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 310 kB/s | 67 kB 00:00 2024-03-28T23:27:47.752 INFO:teuthology.orchestra.run.smithi170.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 8.0 MB/s | 4.4 MB 00:00 2024-03-28T23:27:47.802 INFO:teuthology.orchestra.run.smithi170.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.8 MB/s | 838 kB 00:00 2024-03-28T23:27:47.869 INFO:teuthology.orchestra.run.smithi170.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 370 kB/s | 43 kB 00:00 2024-03-28T23:27:47.920 INFO:teuthology.orchestra.run.smithi170.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 275 kB/s | 32 kB 00:00 2024-03-28T23:27:47.945 INFO:teuthology.orchestra.run.smithi170.stdout:(118/174): python3-asyncssh-2.13.2-5.el9.noarch 1.6 MB/s | 548 kB 00:00 2024-03-28T23:27:47.987 INFO:teuthology.orchestra.run.smithi170.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 128 kB/s | 15 kB 00:00 2024-03-28T23:27:48.037 INFO:teuthology.orchestra.run.smithi170.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-03-28T23:27:48.062 INFO:teuthology.orchestra.run.smithi170.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-03-28T23:27:48.106 INFO:teuthology.orchestra.run.smithi170.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-03-28T23:27:48.146 INFO:teuthology.orchestra.run.smithi170.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 324 kB/s | 35 kB 00:00 2024-03-28T23:27:48.180 INFO:teuthology.orchestra.run.smithi170.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 1.7 MB/s | 203 kB 00:00 2024-03-28T23:27:48.263 INFO:teuthology.orchestra.run.smithi170.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 13 MB/s | 2.0 MB 00:00 2024-03-28T23:27:48.288 INFO:teuthology.orchestra.run.smithi170.stdout:(126/174): python3-grpcio-tools-1.46.7-10.el9.x 1.0 MB/s | 144 kB 00:00 2024-03-28T23:27:48.314 INFO:teuthology.orchestra.run.smithi170.stdout:(127/174): python3-jaraco-8.2.1-3.el9.noarch.rp 80 kB/s | 11 kB 00:00 2024-03-28T23:27:48.381 INFO:teuthology.orchestra.run.smithi170.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 151 kB/s | 18 kB 00:00 2024-03-28T23:27:48.406 INFO:teuthology.orchestra.run.smithi170.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 198 kB/s | 23 kB 00:00 2024-03-28T23:27:48.431 INFO:teuthology.orchestra.run.smithi170.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 166 kB/s | 19 kB 00:00 2024-03-28T23:27:48.498 INFO:teuthology.orchestra.run.smithi170.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 168 kB/s | 20 kB 00:00 2024-03-28T23:27:48.540 INFO:teuthology.orchestra.run.smithi170.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 7.6 MB/s | 1.0 MB 00:00 2024-03-28T23:27:48.566 INFO:teuthology.orchestra.run.smithi170.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 345 kB/s | 46 kB 00:00 2024-03-28T23:27:48.616 INFO:teuthology.orchestra.run.smithi170.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 671 kB/s | 79 kB 00:00 2024-03-28T23:27:48.658 INFO:teuthology.orchestra.run.smithi170.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 493 kB/s | 58 kB 00:00 2024-03-28T23:27:48.683 INFO:teuthology.orchestra.run.smithi170.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-03-28T23:27:48.733 INFO:teuthology.orchestra.run.smithi170.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 140 kB/s | 16 kB 00:00 2024-03-28T23:27:48.775 INFO:teuthology.orchestra.run.smithi170.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 768 kB/s | 90 kB 00:00 2024-03-28T23:27:48.800 INFO:teuthology.orchestra.run.smithi170.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 241 kB/s | 28 kB 00:00 2024-03-28T23:27:48.851 INFO:teuthology.orchestra.run.smithi170.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 262 kB/s | 31 kB 00:00 2024-03-28T23:27:48.892 INFO:teuthology.orchestra.run.smithi170.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-03-28T23:27:48.918 INFO:teuthology.orchestra.run.smithi170.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 504 kB/s | 59 kB 00:00 2024-03-28T23:27:48.968 INFO:teuthology.orchestra.run.smithi170.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 305 kB/s | 36 kB 00:00 2024-03-28T23:27:49.010 INFO:teuthology.orchestra.run.smithi170.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 435 kB/s | 51 kB 00:00 2024-03-28T23:27:49.035 INFO:teuthology.orchestra.run.smithi170.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-03-28T23:27:49.086 INFO:teuthology.orchestra.run.smithi170.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 764 kB/s | 90 kB 00:00 2024-03-28T23:27:49.129 INFO:teuthology.orchestra.run.smithi170.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.5 MB/s | 427 kB 00:00 2024-03-28T23:27:49.153 INFO:teuthology.orchestra.run.smithi170.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 170 kB/s | 20 kB 00:00 2024-03-28T23:27:49.228 INFO:teuthology.orchestra.run.smithi170.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-28T23:27:49.279 INFO:teuthology.orchestra.run.smithi170.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-28T23:27:49.470 INFO:teuthology.orchestra.run.smithi170.stdout:(151/174): NetworkManager-1.47.2-1.el9.x86_64.r 14 MB/s | 2.3 MB 00:00 2024-03-28T23:27:49.538 INFO:teuthology.orchestra.run.smithi170.stdout:(152/174): NetworkManager-libnm-1.47.2-1.el9.x8 27 MB/s | 1.8 MB 00:00 2024-03-28T23:27:49.596 INFO:teuthology.orchestra.run.smithi170.stdout:(153/174): librados2-19.0.0-1899.g323d23dc.el9. 7.6 MB/s | 3.4 MB 00:00 2024-03-28T23:27:49.621 INFO:teuthology.orchestra.run.smithi170.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 492 kB/s | 41 kB 00:00 2024-03-28T23:27:49.646 INFO:teuthology.orchestra.run.smithi170.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 4.8 MB/s | 247 kB 00:00 2024-03-28T23:27:49.672 INFO:teuthology.orchestra.run.smithi170.stdout:(156/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-28T23:27:49.730 INFO:teuthology.orchestra.run.smithi170.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 857 kB/s | 71 kB 00:00 2024-03-28T23:27:49.756 INFO:teuthology.orchestra.run.smithi170.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 436 kB/s | 36 kB 00:00 2024-03-28T23:27:49.797 INFO:teuthology.orchestra.run.smithi170.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.8 MB/s | 192 kB 00:00 2024-03-28T23:27:49.856 INFO:teuthology.orchestra.run.smithi170.stdout:(160/174): librbd1-19.0.0-1899.g323d23dc.el9.x8 5.0 MB/s | 3.1 MB 00:00 2024-03-28T23:27:49.881 INFO:teuthology.orchestra.run.smithi170.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 275 kB/s | 23 kB 00:00 2024-03-28T23:27:49.906 INFO:teuthology.orchestra.run.smithi170.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.7 MB/s | 256 kB 00:00 2024-03-28T23:27:50.015 INFO:teuthology.orchestra.run.smithi170.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 155 kB 00:00 2024-03-28T23:27:50.040 INFO:teuthology.orchestra.run.smithi170.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.0 MB/s | 136 kB 00:00 2024-03-28T23:27:50.066 INFO:teuthology.orchestra.run.smithi170.stdout:(165/174): libblkid-2.37.4-18.el9.x86_64.rpm 517 kB/s | 108 kB 00:00 2024-03-28T23:27:50.091 INFO:teuthology.orchestra.run.smithi170.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 571 kB/s | 29 kB 00:00 2024-03-28T23:27:50.108 INFO:teuthology.orchestra.run.smithi170.stdout:(167/174): libsmartcols-2.37.4-18.el9.x86_64.rp 691 kB/s | 64 kB 00:00 2024-03-28T23:27:50.192 INFO:teuthology.orchestra.run.smithi170.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 9.7 MB/s | 1.2 MB 00:00 2024-03-28T23:27:50.234 INFO:teuthology.orchestra.run.smithi170.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 15 MB/s | 2.2 MB 00:00 2024-03-28T23:27:50.267 INFO:teuthology.orchestra.run.smithi170.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 6.0 MB/s | 465 kB 00:00 2024-03-28T23:27:50.293 INFO:teuthology.orchestra.run.smithi170.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 533 kB/s | 31 kB 00:00 2024-03-28T23:27:50.343 INFO:teuthology.orchestra.run.smithi170.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 9.8 MB/s | 2.3 MB 00:00 2024-03-28T23:27:50.376 INFO:teuthology.orchestra.run.smithi170.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.9 MB/s | 1.1 MB 00:00 2024-03-28T23:27:50.418 INFO:teuthology.orchestra.run.smithi170.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 595 kB/s | 74 kB 00:00 2024-03-28T23:27:50.422 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-28T23:27:50.422 INFO:teuthology.orchestra.run.smithi170.stdout:Total 16 MB/s | 226 MB 00:14 2024-03-28T23:27:50.795 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-28T23:27:51.067 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-28T23:27:51.067 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-28T23:27:51.815 INFO:teuthology.orchestra.run.smithi161.stdout:(174/174): ceph-test-19.0.0-1899.g323d23dc.el9. 2.4 MB/s | 47 MB 00:19 2024-03-28T23:27:51.820 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-28T23:27:51.821 INFO:teuthology.orchestra.run.smithi161.stdout:Total 9.0 MB/s | 226 MB 00:24 2024-03-28T23:27:52.185 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-28T23:27:52.466 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-28T23:27:52.466 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-28T23:27:53.075 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-28T23:27:53.076 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-28T23:27:54.716 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-28T23:27:54.717 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-28T23:28:00.185 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-28T23:28:00.273 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T23:28:00.316 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T23:28:00.345 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T23:28:00.355 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T23:28:00.403 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T23:28:00.457 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T23:28:00.490 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T23:28:00.516 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T23:28:00.570 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T23:28:00.894 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T23:28:00.955 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 10/198 2024-03-28T23:28:01.076 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 10/198 2024-03-28T23:28:01.106 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 11/198 2024-03-28T23:28:01.197 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 11/198 2024-03-28T23:28:01.227 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rados-2:19.0.0-1899.g323d23dc.el9.x86_64 12/198 2024-03-28T23:28:01.265 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T23:28:01.440 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T23:28:01.483 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T23:28:01.512 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 16/198 2024-03-28T23:28:01.617 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 16/198 2024-03-28T23:28:01.637 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86 17/198 2024-03-28T23:28:01.648 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-28T23:28:01.702 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86 17/198 2024-03-28T23:28:01.736 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T23:28:01.750 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T23:28:01.772 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T23:28:01.779 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T23:28:01.787 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T23:28:01.816 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T23:28:01.826 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T23:28:01.868 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T23:28:01.920 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T23:28:01.953 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T23:28:01.987 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T23:28:02.002 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T23:28:02.043 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T23:28:02.046 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T23:28:02.077 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T23:28:02.259 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T23:28:02.378 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T23:28:02.380 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T23:28:02.438 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 10/198 2024-03-28T23:28:02.440 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T23:28:02.533 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T23:28:02.562 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 10/198 2024-03-28T23:28:02.583 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T23:28:02.607 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 11/198 2024-03-28T23:28:02.698 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 11/198 2024-03-28T23:28:02.727 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rados-2:19.0.0-1899.g323d23dc.el9.x86_64 12/198 2024-03-28T23:28:02.764 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T23:28:02.810 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T23:28:02.917 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T23:28:02.976 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T23:28:03.017 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T23:28:03.033 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T23:28:03.055 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T23:28:03.061 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 16/198 2024-03-28T23:28:03.082 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T23:28:03.117 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T23:28:03.153 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 16/198 2024-03-28T23:28:03.178 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86 17/198 2024-03-28T23:28:03.191 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T23:28:03.227 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86 17/198 2024-03-28T23:28:03.244 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T23:28:03.266 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T23:28:03.283 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T23:28:03.288 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T23:28:03.304 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T23:28:03.347 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T23:28:03.382 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T23:28:03.425 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T23:28:03.459 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T23:28:03.486 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T23:28:03.505 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T23:28:03.536 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T23:28:03.555 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T23:28:03.583 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T23:28:03.591 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-common-2:19.0.0-1899.g323d23dc.el9. 39/198 2024-03-28T23:28:03.634 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-argparse-2:19.0.0-1899.g323d23dc.el 40/198 2024-03-28T23:28:03.663 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cephfs-2:19.0.0-1899.g323d23dc.el9.x86_6 41/198 2024-03-28T23:28:03.731 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T23:28:03.777 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T23:28:03.783 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T23:28:03.859 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T23:28:03.913 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T23:28:03.970 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T23:28:03.992 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T23:28:04.047 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T23:28:04.075 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T23:28:04.122 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T23:28:04.336 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T23:28:04.444 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T23:28:04.538 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T23:28:04.568 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T23:28:04.585 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T23:28:04.612 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T23:28:04.650 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T23:28:04.655 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T23:28:04.730 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T23:28:04.734 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T23:28:04.770 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T23:28:04.780 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T23:28:04.785 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T23:28:04.818 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T23:28:04.823 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T23:28:04.868 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T23:28:04.886 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T23:28:04.906 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librados-devel-2:19.0.0-1899.g323d23dc.el9.x86_6 53/198 2024-03-28T23:28:04.931 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T23:28:04.937 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T23:28:04.974 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T23:28:05.007 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T23:28:05.035 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T23:28:05.103 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T23:28:05.139 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ceph-common-2:19.0.0-1899.g323d23dc.el9. 39/198 2024-03-28T23:28:05.183 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ceph-argparse-2:19.0.0-1899.g323d23dc.el 40/198 2024-03-28T23:28:05.212 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cephfs-2:19.0.0-1899.g323d23dc.el9.x86_6 41/198 2024-03-28T23:28:05.214 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T23:28:05.280 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T23:28:05.326 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T23:28:05.399 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T23:28:05.442 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T23:28:05.523 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T23:28:05.532 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T23:28:05.575 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T23:28:05.587 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T23:28:05.720 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T23:28:05.762 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T23:28:05.909 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T23:28:06.099 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T23:28:06.104 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T23:28:06.130 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T23:28:06.166 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T23:28:06.181 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T23:28:06.213 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T23:28:06.257 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T23:28:06.267 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T23:28:06.300 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T23:28:06.318 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T23:28:06.327 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T23:28:06.342 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T23:28:06.373 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T23:28:06.416 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T23:28:06.453 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librados-devel-2:19.0.0-1899.g323d23dc.el9.x86_6 53/198 2024-03-28T23:28:06.474 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T23:28:06.484 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T23:28:06.531 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T23:28:06.562 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T23:28:06.603 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T23:28:06.714 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T23:28:06.745 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T23:28:06.756 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T23:28:06.786 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T23:28:06.820 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T23:28:06.835 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T23:28:06.922 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T23:28:06.948 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T23:28:07.036 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T23:28:07.052 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T23:28:07.080 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T23:28:07.158 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T23:28:07.210 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T23:28:07.339 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T23:28:07.339 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T23:28:07.339 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T23:28:07.339 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T23:28:07.340 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T23:28:07.393 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T23:28:07.424 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T23:28:07.461 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T23:28:07.505 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T23:28:07.525 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T23:28:07.548 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T23:28:07.561 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T23:28:07.604 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T23:28:07.646 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T23:28:07.673 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T23:28:07.713 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T23:28:07.753 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T23:28:07.780 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T23:28:07.784 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T23:28:07.811 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T23:28:07.834 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T23:28:07.865 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T23:28:07.869 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T23:28:07.906 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T23:28:07.910 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T23:28:07.942 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T23:28:07.972 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T23:28:08.010 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T23:28:08.047 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T23:28:08.048 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T23:28:08.083 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T23:28:08.088 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T23:28:08.114 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T23:28:08.116 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T23:28:08.138 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T23:28:08.149 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T23:28:08.175 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T23:28:08.219 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T23:28:08.264 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T23:28:08.340 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T23:28:08.346 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T23:28:08.389 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T23:28:08.421 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T23:28:08.439 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T23:28:08.468 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T23:28:08.488 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T23:28:08.537 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T23:28:08.564 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T23:28:08.592 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T23:28:08.622 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T23:28:08.779 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T23:28:08.826 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T23:28:08.971 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T23:28:09.035 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T23:28:09.208 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T23:28:09.336 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T23:28:09.344 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T23:28:09.386 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T23:28:09.420 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T23:28:09.468 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T23:28:09.489 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T23:28:09.524 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T23:28:09.546 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T23:28:09.563 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T23:28:09.588 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T23:28:09.594 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T23:28:09.624 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T23:28:09.701 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T23:28:09.734 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T23:28:09.768 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T23:28:09.768 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T23:28:09.797 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T23:28:09.810 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T23:28:09.844 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T23:28:09.880 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T23:28:09.934 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T23:28:10.023 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T23:28:10.067 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T23:28:10.097 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T23:28:10.145 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T23:28:10.243 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T23:28:10.291 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T23:28:10.528 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T23:28:10.636 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T23:28:11.023 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T23:28:11.078 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T23:28:11.112 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T23:28:11.161 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T23:28:11.265 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T23:28:11.313 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T23:28:11.491 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T23:28:11.527 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T23:28:11.604 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T23:28:11.853 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T23:28:11.923 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T23:28:11.954 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T23:28:11.986 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T23:28:12.013 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T23:28:12.256 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T23:28:12.322 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T23:28:12.352 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 124/198 2024-03-28T23:28:12.374 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T23:28:12.442 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 124/198 2024-03-28T23:28:12.483 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rbd-2:19.0.0-1899.g323d23dc.el9.x86_64 125/198 2024-03-28T23:28:12.518 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T23:28:12.976 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T23:28:13.148 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 128/198 2024-03-28T23:28:13.419 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 128/198 2024-03-28T23:28:13.456 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rgw-2:19.0.0-1899.g323d23dc.el9.x86_64 129/198 2024-03-28T23:28:13.601 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T23:28:13.702 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T23:28:13.733 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T23:28:13.764 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T23:28:13.792 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T23:28:14.104 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T23:28:14.138 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 124/198 2024-03-28T23:28:14.229 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 124/198 2024-03-28T23:28:14.260 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rbd-2:19.0.0-1899.g323d23dc.el9.x86_64 125/198 2024-03-28T23:28:14.296 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T23:28:14.754 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T23:28:14.784 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 128/198 2024-03-28T23:28:14.854 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 128/198 2024-03-28T23:28:14.876 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rgw-2:19.0.0-1899.g323d23dc.el9.x86_64 129/198 2024-03-28T23:28:15.712 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 130/198 2024-03-28T23:28:15.745 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 130/198 2024-03-28T23:28:16.282 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 130/198 2024-03-28T23:28:16.331 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 131/198 2024-03-28T23:28:16.423 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 131/198 2024-03-28T23:28:16.423 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T23:28:16.424 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T23:28:16.424 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:16.466 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 132/198 2024-03-28T23:28:17.001 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 130/198 2024-03-28T23:28:17.040 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 130/198 2024-03-28T23:28:17.582 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 130/198 2024-03-28T23:28:17.618 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 131/198 2024-03-28T23:28:17.715 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 131/198 2024-03-28T23:28:17.715 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T23:28:17.715 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T23:28:17.715 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:17.752 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 132/198 2024-03-28T23:28:27.967 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 132/198 2024-03-28T23:28:27.967 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /sys 2024-03-28T23:28:27.967 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /proc 2024-03-28T23:28:27.968 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /mnt 2024-03-28T23:28:27.968 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /var/tmp 2024-03-28T23:28:27.968 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /home 2024-03-28T23:28:27.968 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /root 2024-03-28T23:28:27.968 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /tmp 2024-03-28T23:28:27.968 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:28.130 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 133/198 2024-03-28T23:28:28.146 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 133/198 2024-03-28T23:28:28.146 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T23:28:28.146 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T23:28:28.146 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:29.592 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 134/198 2024-03-28T23:28:29.617 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 134/198 2024-03-28T23:28:29.618 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T23:28:29.618 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T23:28:29.618 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T23:28:29.618 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T23:28:29.618 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:29.713 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 132/198 2024-03-28T23:28:29.714 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /sys 2024-03-28T23:28:29.714 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /proc 2024-03-28T23:28:29.714 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /mnt 2024-03-28T23:28:29.714 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /var/tmp 2024-03-28T23:28:29.714 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /home 2024-03-28T23:28:29.714 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /root 2024-03-28T23:28:29.714 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /tmp 2024-03-28T23:28:29.714 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:29.842 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 135/198 2024-03-28T23:28:29.865 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 135/198 2024-03-28T23:28:29.865 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T23:28:29.866 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T23:28:29.866 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T23:28:29.866 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T23:28:29.866 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:29.890 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 133/198 2024-03-28T23:28:29.905 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 133/198 2024-03-28T23:28:29.905 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T23:28:29.905 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T23:28:29.905 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:30.310 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 136/198 2024-03-28T23:28:30.332 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 136/198 2024-03-28T23:28:30.332 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T23:28:30.332 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T23:28:30.333 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T23:28:30.333 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T23:28:30.333 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:30.420 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T23:28:30.888 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 138/198 2024-03-28T23:28:30.910 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 138/198 2024-03-28T23:28:30.941 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1899.g323d23dc.e 139/198 2024-03-28T23:28:31.002 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1899.g323d23dc. 140/198 2024-03-28T23:28:31.025 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noa 141/198 2024-03-28T23:28:31.409 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 134/198 2024-03-28T23:28:31.435 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 134/198 2024-03-28T23:28:31.435 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T23:28:31.435 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T23:28:31.435 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T23:28:31.435 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T23:28:31.435 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:31.660 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 135/198 2024-03-28T23:28:31.682 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 135/198 2024-03-28T23:28:31.682 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T23:28:31.682 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T23:28:31.682 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T23:28:31.682 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T23:28:31.683 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:32.053 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noa 141/198 2024-03-28T23:28:32.094 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.n 142/198 2024-03-28T23:28:32.127 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 136/198 2024-03-28T23:28:32.150 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 136/198 2024-03-28T23:28:32.151 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T23:28:32.151 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T23:28:32.151 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T23:28:32.151 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T23:28:32.151 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:32.245 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T23:28:32.705 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 138/198 2024-03-28T23:28:32.735 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 138/198 2024-03-28T23:28:32.767 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1899.g323d23dc.e 139/198 2024-03-28T23:28:32.828 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1899.g323d23dc. 140/198 2024-03-28T23:28:32.850 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noa 141/198 2024-03-28T23:28:33.054 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.n 142/198 2024-03-28T23:28:33.086 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1899.g323 143/198 2024-03-28T23:28:33.185 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1899.g323 143/198 2024-03-28T23:28:33.356 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1899.g323d23dc.el 144/198 2024-03-28T23:28:33.380 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 145/198 2024-03-28T23:28:33.403 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 145/198 2024-03-28T23:28:33.403 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T23:28:33.403 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T23:28:33.403 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T23:28:33.403 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T23:28:33.403 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:33.455 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 146/198 2024-03-28T23:28:33.466 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 146/198 2024-03-28T23:28:33.519 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-2:19.0.0-1899.g323d23dc.el9.x86_64 147/198 2024-03-28T23:28:33.883 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noa 141/198 2024-03-28T23:28:33.918 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.n 142/198 2024-03-28T23:28:34.372 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T23:28:34.407 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 149/198 2024-03-28T23:28:34.430 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 149/198 2024-03-28T23:28:34.430 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T23:28:34.430 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T23:28:34.430 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T23:28:34.430 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T23:28:34.431 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:34.475 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1899.g323d2 150/198 2024-03-28T23:28:34.497 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1899.g323d2 150/198 2024-03-28T23:28:34.497 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T23:28:34.497 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T23:28:34.497 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:34.542 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 151/198 2024-03-28T23:28:34.564 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 151/198 2024-03-28T23:28:34.564 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T23:28:34.565 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T23:28:34.565 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T23:28:34.565 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T23:28:34.565 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:34.867 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 152/198 2024-03-28T23:28:34.882 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.n 142/198 2024-03-28T23:28:34.890 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 152/198 2024-03-28T23:28:34.890 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T23:28:34.890 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T23:28:34.890 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T23:28:34.890 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T23:28:34.890 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:34.910 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1899.g323 143/198 2024-03-28T23:28:35.012 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1899.g323 143/198 2024-03-28T23:28:35.173 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1899.g323d23dc.el 144/198 2024-03-28T23:28:35.204 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 145/198 2024-03-28T23:28:35.227 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 145/198 2024-03-28T23:28:35.227 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T23:28:35.227 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T23:28:35.227 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T23:28:35.227 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T23:28:35.227 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:35.272 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 146/198 2024-03-28T23:28:35.282 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 146/198 2024-03-28T23:28:35.330 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-2:19.0.0-1899.g323d23dc.el9.x86_64 147/198 2024-03-28T23:28:36.185 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T23:28:36.215 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 149/198 2024-03-28T23:28:36.237 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 149/198 2024-03-28T23:28:36.237 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T23:28:36.238 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T23:28:36.238 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T23:28:36.238 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T23:28:36.238 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:36.283 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1899.g323d2 150/198 2024-03-28T23:28:36.304 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1899.g323d2 150/198 2024-03-28T23:28:36.305 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T23:28:36.305 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T23:28:36.305 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:36.350 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 151/198 2024-03-28T23:28:36.372 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 151/198 2024-03-28T23:28:36.372 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T23:28:36.373 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T23:28:36.373 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T23:28:36.373 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T23:28:36.373 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:36.676 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 152/198 2024-03-28T23:28:36.697 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 152/198 2024-03-28T23:28:36.697 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T23:28:36.698 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T23:28:36.698 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T23:28:36.698 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T23:28:36.698 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:39.355 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-test-2:19.0.0-1899.g323d23dc.el9.x86_64 153/198 2024-03-28T23:28:39.397 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 154/198 2024-03-28T23:28:39.427 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-nbd-2:19.0.0-1899.g323d23dc.el9.x86_64 155/198 2024-03-28T23:28:39.462 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T23:28:39.651 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephfs-shell-2:19.0.0-1899.g323d23dc.el9.x86_64 157/198 2024-03-28T23:28:39.763 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T23:28:39.802 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 159/198 2024-03-28T23:28:39.853 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T23:28:39.911 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T23:28:39.946 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T23:28:39.978 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs-devel-2:19.0.0-1899.g323d23dc.el9.x86_ 163/198 2024-03-28T23:28:40.002 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T23:28:40.032 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T23:28:40.098 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T23:28:40.175 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T23:28:40.214 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T23:28:40.827 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T23:28:40.862 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T23:28:40.883 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T23:28:41.171 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-test-2:19.0.0-1899.g323d23dc.el9.x86_64 153/198 2024-03-28T23:28:41.212 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 154/198 2024-03-28T23:28:41.243 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-nbd-2:19.0.0-1899.g323d23dc.el9.x86_64 155/198 2024-03-28T23:28:41.270 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T23:28:41.460 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephfs-shell-2:19.0.0-1899.g323d23dc.el9.x86_64 157/198 2024-03-28T23:28:41.579 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T23:28:41.617 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 159/198 2024-03-28T23:28:41.672 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T23:28:41.698 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephfs-top-2:19.0.0-1899.g323d23dc.el9.noarch 170/198 2024-03-28T23:28:41.734 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T23:28:41.769 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T23:28:41.774 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T23:28:41.802 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs-devel-2:19.0.0-1899.g323d23dc.el9.x86_ 163/198 2024-03-28T23:28:41.816 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T23:28:41.834 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T23:28:41.863 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T23:28:41.912 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T23:28:41.931 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T23:28:41.954 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T23:28:41.978 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T23:28:42.004 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T23:28:42.007 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T23:28:42.030 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T23:28:42.030 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T23:28:42.045 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T23:28:42.151 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T23:28:42.181 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T23:28:42.181 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T23:28:42.265 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T23:28:42.292 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T23:28:42.325 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T23:28:42.350 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T23:28:42.367 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T23:28:42.402 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T23:28:42.440 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T23:28:42.472 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T23:28:42.505 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T23:28:42.531 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T23:28:42.556 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T23:28:42.582 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T23:28:42.588 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T23:28:42.588 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T23:28:42.676 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T23:28:42.718 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T23:28:42.737 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T23:28:42.768 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T23:28:42.768 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T23:28:42.851 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T23:28:42.875 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T23:28:42.908 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T23:28:42.935 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T23:28:42.968 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T23:28:43.034 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T23:28:43.571 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephfs-top-2:19.0.0-1899.g323d23dc.el9.noarch 170/198 2024-03-28T23:28:43.657 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T23:28:43.698 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T23:28:43.797 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T23:28:43.827 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T23:28:43.852 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T23:28:43.877 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T23:28:43.903 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T23:28:43.904 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T23:28:43.992 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T23:28:44.021 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T23:28:44.021 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T23:28:44.105 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T23:28:44.141 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T23:28:44.173 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T23:28:44.198 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T23:28:44.224 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T23:28:44.258 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T23:28:44.305 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T23:28:44.345 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T23:28:44.370 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T23:28:44.395 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T23:28:44.429 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T23:28:44.454 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T23:28:44.460 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T23:28:44.461 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T23:28:44.641 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T23:28:44.641 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T23:28:44.707 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T23:28:44.731 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T23:28:44.763 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T23:28:44.791 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T23:28:44.816 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T23:28:44.882 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T23:28:48.525 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T23:28:48.525 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-2:19.0.0-1899.g323d23dc.el9.x86_64 1/198 2024-03-28T23:28:48.525 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 2/198 2024-03-28T23:28:48.525 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 3/198 2024-03-28T23:28:48.525 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 4/198 2024-03-28T23:28:48.525 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1899.g323d2 5/198 2024-03-28T23:28:48.525 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 6/198 2024-03-28T23:28:48.525 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 7/198 2024-03-28T23:28:48.525 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 8/198 2024-03-28T23:28:48.525 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 9/198 2024-03-28T23:28:48.526 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 10/198 2024-03-28T23:28:48.526 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 11/198 2024-03-28T23:28:48.526 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-test-2:19.0.0-1899.g323d23dc.el9.x86_64 12/198 2024-03-28T23:28:48.526 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 13/198 2024-03-28T23:28:48.526 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephfs-shell-2:19.0.0-1899.g323d23dc.el9.x86_64 14/198 2024-03-28T23:28:48.526 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs-devel-2:19.0.0-1899.g323d23dc.el9.x86_ 15/198 2024-03-28T23:28:48.526 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 16/198 2024-03-28T23:28:48.526 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 17/198 2024-03-28T23:28:48.526 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados-devel-2:19.0.0-1899.g323d23dc.el9.x86_6 18/198 2024-03-28T23:28:48.526 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86 19/198 2024-03-28T23:28:48.526 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 20/198 2024-03-28T23:28:48.526 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1899.g323d23dc.el 21/198 2024-03-28T23:28:48.526 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-common-2:19.0.0-1899.g323d23dc.el9. 22/198 2024-03-28T23:28:48.527 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cephfs-2:19.0.0-1899.g323d23dc.el9.x86_6 23/198 2024-03-28T23:28:48.527 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rados-2:19.0.0-1899.g323d23dc.el9.x86_64 24/198 2024-03-28T23:28:48.527 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rbd-2:19.0.0-1899.g323d23dc.el9.x86_64 25/198 2024-03-28T23:28:48.527 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rgw-2:19.0.0-1899.g323d23dc.el9.x86_64 26/198 2024-03-28T23:28:48.527 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 27/198 2024-03-28T23:28:48.527 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 28/198 2024-03-28T23:28:48.527 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-nbd-2:19.0.0-1899.g323d23dc.el9.x86_64 29/198 2024-03-28T23:28:48.527 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1899.g323d23dc. 30/198 2024-03-28T23:28:48.542 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noa 31/198 2024-03-28T23:28:48.543 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.n 32/198 2024-03-28T23:28:48.543 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1899.g323 33/198 2024-03-28T23:28:48.543 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1899.g323d23dc.el 34/198 2024-03-28T23:28:48.543 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 35/198 2024-03-28T23:28:48.543 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1899.g323d23dc.e 36/198 2024-03-28T23:28:48.543 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 37/198 2024-03-28T23:28:48.543 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 38/198 2024-03-28T23:28:48.543 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephfs-top-2:19.0.0-1899.g323d23dc.el9.noarch 39/198 2024-03-28T23:28:48.543 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T23:28:48.543 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T23:28:48.543 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T23:28:48.543 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T23:28:48.543 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T23:28:48.544 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T23:28:48.544 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T23:28:48.544 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T23:28:48.544 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T23:28:48.544 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T23:28:48.544 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T23:28:48.544 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T23:28:48.544 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T23:28:48.544 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T23:28:48.544 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T23:28:48.544 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T23:28:48.544 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T23:28:48.544 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T23:28:48.544 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-03-28T23:28:48.545 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-03-28T23:28:48.545 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-03-28T23:28:48.545 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-03-28T23:28:48.545 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-28T23:28:48.545 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-28T23:28:48.545 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-28T23:28:48.545 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-28T23:28:48.545 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-28T23:28:48.545 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-28T23:28:48.545 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-28T23:28:48.545 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-28T23:28:48.545 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-28T23:28:48.545 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-28T23:28:48.546 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-28T23:28:48.546 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-28T23:28:48.546 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-28T23:28:48.546 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-28T23:28:48.546 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-28T23:28:48.546 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-28T23:28:48.546 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-28T23:28:48.546 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-28T23:28:48.546 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-28T23:28:48.546 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-28T23:28:48.546 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-03-28T23:28:48.546 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-03-28T23:28:48.546 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-03-28T23:28:48.547 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-03-28T23:28:48.547 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-03-28T23:28:48.547 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-03-28T23:28:48.547 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-03-28T23:28:48.547 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T23:28:48.547 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-03-28T23:28:48.547 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-03-28T23:28:48.547 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-03-28T23:28:48.547 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-03-28T23:28:48.547 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-03-28T23:28:48.547 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-03-28T23:28:48.547 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-03-28T23:28:48.547 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-03-28T23:28:48.548 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-03-28T23:28:48.548 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-03-28T23:28:48.548 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-03-28T23:28:48.548 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-03-28T23:28:48.548 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-03-28T23:28:48.548 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-03-28T23:28:48.548 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T23:28:48.548 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T23:28:48.548 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T23:28:48.548 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T23:28:48.548 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T23:28:48.548 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T23:28:48.548 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T23:28:48.548 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T23:28:48.549 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T23:28:48.549 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T23:28:48.549 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T23:28:48.549 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T23:28:48.549 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T23:28:48.549 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T23:28:48.549 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T23:28:48.549 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T23:28:48.549 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T23:28:48.549 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T23:28:48.549 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T23:28:48.549 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T23:28:48.549 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T23:28:48.550 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T23:28:48.550 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T23:28:48.550 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T23:28:48.550 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T23:28:48.550 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T23:28:48.550 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T23:28:48.550 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T23:28:48.550 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T23:28:48.550 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T23:28:48.550 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T23:28:48.550 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T23:28:48.550 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T23:28:48.550 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T23:28:48.551 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T23:28:48.551 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T23:28:48.551 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T23:28:48.551 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T23:28:48.551 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T23:28:48.551 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T23:28:48.551 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T23:28:48.551 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T23:28:48.551 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T23:28:48.551 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T23:28:48.551 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T23:28:48.551 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T23:28:48.551 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T23:28:48.551 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 151/198 2024-03-28T23:28:48.552 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T23:28:48.552 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 153/198 2024-03-28T23:28:48.552 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T23:28:48.552 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 155/198 2024-03-28T23:28:48.552 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-03-28T23:28:48.552 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T23:28:48.552 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T23:28:48.552 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T23:28:48.552 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T23:28:48.552 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T23:28:48.552 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T23:28:48.552 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-03-28T23:28:48.552 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-03-28T23:28:48.552 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T23:28:48.553 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T23:28:48.553 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T23:28:48.553 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T23:28:48.553 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T23:28:48.553 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T23:28:48.553 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T23:28:48.553 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T23:28:48.553 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T23:28:48.553 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T23:28:48.553 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T23:28:48.553 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T23:28:48.553 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T23:28:48.553 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T23:28:48.553 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T23:28:48.554 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T23:28:48.554 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T23:28:48.554 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T23:28:48.554 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T23:28:48.554 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T23:28:48.554 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T23:28:48.554 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T23:28:48.554 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T23:28:48.554 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T23:28:48.554 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T23:28:48.554 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T23:28:48.554 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T23:28:48.554 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T23:28:48.555 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T23:28:48.555 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T23:28:48.555 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T23:28:48.555 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T23:28:48.555 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T23:28:50.190 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T23:28:50.191 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-2:19.0.0-1899.g323d23dc.el9.x86_64 1/198 2024-03-28T23:28:50.191 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 2/198 2024-03-28T23:28:50.191 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 3/198 2024-03-28T23:28:50.191 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 4/198 2024-03-28T23:28:50.191 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1899.g323d2 5/198 2024-03-28T23:28:50.191 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 6/198 2024-03-28T23:28:50.191 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 7/198 2024-03-28T23:28:50.191 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 8/198 2024-03-28T23:28:50.191 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 9/198 2024-03-28T23:28:50.191 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 10/198 2024-03-28T23:28:50.191 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 11/198 2024-03-28T23:28:50.192 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-test-2:19.0.0-1899.g323d23dc.el9.x86_64 12/198 2024-03-28T23:28:50.192 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 13/198 2024-03-28T23:28:50.192 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephfs-shell-2:19.0.0-1899.g323d23dc.el9.x86_64 14/198 2024-03-28T23:28:50.192 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs-devel-2:19.0.0-1899.g323d23dc.el9.x86_ 15/198 2024-03-28T23:28:50.192 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 16/198 2024-03-28T23:28:50.192 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 17/198 2024-03-28T23:28:50.192 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados-devel-2:19.0.0-1899.g323d23dc.el9.x86_6 18/198 2024-03-28T23:28:50.192 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86 19/198 2024-03-28T23:28:50.192 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 20/198 2024-03-28T23:28:50.192 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1899.g323d23dc.el 21/198 2024-03-28T23:28:50.192 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ceph-common-2:19.0.0-1899.g323d23dc.el9. 22/198 2024-03-28T23:28:50.192 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cephfs-2:19.0.0-1899.g323d23dc.el9.x86_6 23/198 2024-03-28T23:28:50.192 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rados-2:19.0.0-1899.g323d23dc.el9.x86_64 24/198 2024-03-28T23:28:50.192 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rbd-2:19.0.0-1899.g323d23dc.el9.x86_64 25/198 2024-03-28T23:28:50.211 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rgw-2:19.0.0-1899.g323d23dc.el9.x86_64 26/198 2024-03-28T23:28:50.211 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 27/198 2024-03-28T23:28:50.211 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 28/198 2024-03-28T23:28:50.211 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-nbd-2:19.0.0-1899.g323d23dc.el9.x86_64 29/198 2024-03-28T23:28:50.211 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1899.g323d23dc. 30/198 2024-03-28T23:28:50.211 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noa 31/198 2024-03-28T23:28:50.211 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.n 32/198 2024-03-28T23:28:50.212 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1899.g323 33/198 2024-03-28T23:28:50.212 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1899.g323d23dc.el 34/198 2024-03-28T23:28:50.212 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 35/198 2024-03-28T23:28:50.212 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1899.g323d23dc.e 36/198 2024-03-28T23:28:50.212 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 37/198 2024-03-28T23:28:50.212 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 38/198 2024-03-28T23:28:50.212 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephfs-top-2:19.0.0-1899.g323d23dc.el9.noarch 39/198 2024-03-28T23:28:50.212 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T23:28:50.212 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T23:28:50.212 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T23:28:50.212 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T23:28:50.212 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T23:28:50.212 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T23:28:50.212 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T23:28:50.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T23:28:50.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T23:28:50.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T23:28:50.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T23:28:50.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T23:28:50.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T23:28:50.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T23:28:50.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T23:28:50.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T23:28:50.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T23:28:50.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T23:28:50.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-03-28T23:28:50.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-03-28T23:28:50.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-03-28T23:28:50.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-03-28T23:28:50.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-28T23:28:50.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-28T23:28:50.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-28T23:28:50.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-28T23:28:50.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-28T23:28:50.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-28T23:28:50.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-28T23:28:50.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-28T23:28:50.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-28T23:28:50.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-28T23:28:50.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-28T23:28:50.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-28T23:28:50.215 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-28T23:28:50.215 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-28T23:28:50.215 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-28T23:28:50.215 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-28T23:28:50.215 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-28T23:28:50.215 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-28T23:28:50.215 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-28T23:28:50.215 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-28T23:28:50.215 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-03-28T23:28:50.215 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-03-28T23:28:50.215 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-03-28T23:28:50.215 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-03-28T23:28:50.215 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-03-28T23:28:50.215 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-03-28T23:28:50.216 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-03-28T23:28:50.216 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T23:28:50.216 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-03-28T23:28:50.216 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-03-28T23:28:50.216 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-03-28T23:28:50.216 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-03-28T23:28:50.216 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-03-28T23:28:50.216 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-03-28T23:28:50.216 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-03-28T23:28:50.216 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-03-28T23:28:50.216 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-03-28T23:28:50.216 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-03-28T23:28:50.216 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-03-28T23:28:50.216 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-03-28T23:28:50.217 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-03-28T23:28:50.217 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-03-28T23:28:50.217 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T23:28:50.217 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T23:28:50.217 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T23:28:50.217 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T23:28:50.217 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T23:28:50.217 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T23:28:50.217 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T23:28:50.217 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T23:28:50.217 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T23:28:50.218 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T23:28:50.218 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T23:28:50.218 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T23:28:50.218 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T23:28:50.218 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T23:28:50.218 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T23:28:50.218 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T23:28:50.218 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T23:28:50.218 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T23:28:50.218 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T23:28:50.218 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T23:28:50.218 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T23:28:50.218 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T23:28:50.219 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T23:28:50.219 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T23:28:50.219 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T23:28:50.219 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T23:28:50.219 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T23:28:50.219 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T23:28:50.219 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T23:28:50.219 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T23:28:50.219 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T23:28:50.219 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T23:28:50.219 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T23:28:50.219 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T23:28:50.219 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T23:28:50.220 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T23:28:50.220 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T23:28:50.220 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T23:28:50.220 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T23:28:50.220 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T23:28:50.220 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T23:28:50.220 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T23:28:50.220 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T23:28:50.220 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T23:28:50.220 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T23:28:50.220 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T23:28:50.220 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T23:28:50.220 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 151/198 2024-03-28T23:28:50.220 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T23:28:50.221 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 153/198 2024-03-28T23:28:50.221 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T23:28:50.221 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 155/198 2024-03-28T23:28:50.221 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-03-28T23:28:50.221 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T23:28:50.221 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T23:28:50.221 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T23:28:50.221 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T23:28:50.221 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T23:28:50.221 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T23:28:50.221 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-03-28T23:28:50.221 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-03-28T23:28:50.221 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T23:28:50.222 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T23:28:50.222 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T23:28:50.222 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T23:28:50.222 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T23:28:50.222 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T23:28:50.222 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T23:28:50.222 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T23:28:50.222 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T23:28:50.222 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T23:28:50.222 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T23:28:50.222 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T23:28:50.222 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T23:28:50.222 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T23:28:50.223 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T23:28:50.223 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T23:28:50.223 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T23:28:50.223 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T23:28:50.223 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T23:28:50.223 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T23:28:50.223 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T23:28:50.223 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T23:28:50.223 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T23:28:50.223 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T23:28:50.223 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T23:28:50.223 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T23:28:50.223 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T23:28:50.224 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T23:28:50.224 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T23:28:50.224 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T23:28:50.224 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T23:28:50.224 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T23:28:50.224 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T23:28:51.623 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T23:28:51.624 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:51.624 INFO:teuthology.orchestra.run.smithi170.stdout:Upgraded: 2024-03-28T23:28:51.624 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T23:28:51.624 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T23:28:51.624 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T23:28:51.624 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T23:28:51.624 INFO:teuthology.orchestra.run.smithi170.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T23:28:51.624 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T23:28:51.624 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T23:28:51.624 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T23:28:51.624 INFO:teuthology.orchestra.run.smithi170.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T23:28:51.624 INFO:teuthology.orchestra.run.smithi170.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T23:28:51.625 INFO:teuthology.orchestra.run.smithi170.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T23:28:51.625 INFO:teuthology.orchestra.run.smithi170.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T23:28:51.625 INFO:teuthology.orchestra.run.smithi170.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T23:28:51.625 INFO:teuthology.orchestra.run.smithi170.stdout: librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.625 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.625 INFO:teuthology.orchestra.run.smithi170.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T23:28:51.625 INFO:teuthology.orchestra.run.smithi170.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T23:28:51.625 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T23:28:51.625 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T23:28:51.625 INFO:teuthology.orchestra.run.smithi170.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T23:28:51.625 INFO:teuthology.orchestra.run.smithi170.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T23:28:51.625 INFO:teuthology.orchestra.run.smithi170.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T23:28:51.625 INFO:teuthology.orchestra.run.smithi170.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T23:28:51.625 INFO:teuthology.orchestra.run.smithi170.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T23:28:51.625 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-28T23:28:51.626 INFO:teuthology.orchestra.run.smithi170.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T23:28:51.626 INFO:teuthology.orchestra.run.smithi170.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T23:28:51.626 INFO:teuthology.orchestra.run.smithi170.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T23:28:51.626 INFO:teuthology.orchestra.run.smithi170.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T23:28:51.626 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.626 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.626 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.626 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.626 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T23:28:51.626 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.626 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.626 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.626 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T23:28:51.627 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T23:28:51.627 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T23:28:51.627 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T23:28:51.627 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T23:28:51.627 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.627 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.627 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T23:28:51.627 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.627 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.627 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.627 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T23:28:51.627 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T23:28:51.627 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.628 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-shell-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.628 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-top-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T23:28:51.628 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T23:28:51.628 INFO:teuthology.orchestra.run.smithi170.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T23:28:51.628 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T23:28:51.628 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T23:28:51.628 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T23:28:51.628 INFO:teuthology.orchestra.run.smithi170.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T23:28:51.628 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T23:28:51.628 INFO:teuthology.orchestra.run.smithi170.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T23:28:51.628 INFO:teuthology.orchestra.run.smithi170.stdout: jq-1.6-16.el9.x86_64 2024-03-28T23:28:51.628 INFO:teuthology.orchestra.run.smithi170.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T23:28:51.628 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T23:28:51.628 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T23:28:51.629 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.629 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.629 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.629 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T23:28:51.629 INFO:teuthology.orchestra.run.smithi170.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T23:28:51.629 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T23:28:51.629 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T23:28:51.629 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T23:28:51.629 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.629 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.629 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T23:28:51.629 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.629 INFO:teuthology.orchestra.run.smithi170.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T23:28:51.629 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T23:28:51.629 INFO:teuthology.orchestra.run.smithi170.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T23:28:51.630 INFO:teuthology.orchestra.run.smithi170.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T23:28:51.630 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T23:28:51.630 INFO:teuthology.orchestra.run.smithi170.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T23:28:51.630 INFO:teuthology.orchestra.run.smithi170.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T23:28:51.630 INFO:teuthology.orchestra.run.smithi170.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T23:28:51.630 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T23:28:51.630 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T23:28:51.630 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T23:28:51.630 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T23:28:51.630 INFO:teuthology.orchestra.run.smithi170.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T23:28:51.630 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T23:28:51.630 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T23:28:51.630 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T23:28:51.631 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T23:28:51.631 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T23:28:51.631 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T23:28:51.631 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T23:28:51.631 INFO:teuthology.orchestra.run.smithi170.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T23:28:51.631 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T23:28:51.631 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T23:28:51.631 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T23:28:51.631 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.631 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.631 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.631 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T23:28:51.631 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T23:28:51.631 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T23:28:51.631 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T23:28:51.632 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T23:28:51.632 INFO:teuthology.orchestra.run.smithi170.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T23:28:51.632 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T23:28:51.632 INFO:teuthology.orchestra.run.smithi170.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T23:28:51.632 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T23:28:51.632 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T23:28:51.632 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T23:28:51.632 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T23:28:51.632 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T23:28:51.632 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T23:28:51.632 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T23:28:51.632 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T23:28:51.632 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T23:28:51.632 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T23:28:51.633 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T23:28:51.633 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T23:28:51.633 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T23:28:51.633 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T23:28:51.633 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T23:28:51.633 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T23:28:51.633 INFO:teuthology.orchestra.run.smithi170.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T23:28:51.633 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T23:28:51.633 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T23:28:51.633 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T23:28:51.633 INFO:teuthology.orchestra.run.smithi170.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T23:28:51.633 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T23:28:51.633 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T23:28:51.633 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T23:28:51.634 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T23:28:51.634 INFO:teuthology.orchestra.run.smithi170.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T23:28:51.634 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T23:28:51.634 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T23:28:51.634 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T23:28:51.634 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T23:28:51.634 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T23:28:51.634 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T23:28:51.634 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.634 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.634 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T23:28:51.634 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T23:28:51.634 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T23:28:51.634 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.635 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T23:28:51.635 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T23:28:51.635 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T23:28:51.635 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T23:28:51.635 INFO:teuthology.orchestra.run.smithi170.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T23:28:51.635 INFO:teuthology.orchestra.run.smithi170.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T23:28:51.635 INFO:teuthology.orchestra.run.smithi170.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T23:28:51.635 INFO:teuthology.orchestra.run.smithi170.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T23:28:51.635 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T23:28:51.635 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T23:28:51.635 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T23:28:51.635 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T23:28:51.635 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T23:28:51.635 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T23:28:51.636 INFO:teuthology.orchestra.run.smithi170.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T23:28:51.636 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.636 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.636 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:51.636 INFO:teuthology.orchestra.run.smithi170.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T23:28:51.636 INFO:teuthology.orchestra.run.smithi170.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T23:28:51.636 INFO:teuthology.orchestra.run.smithi170.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T23:28:51.636 INFO:teuthology.orchestra.run.smithi170.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T23:28:51.636 INFO:teuthology.orchestra.run.smithi170.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T23:28:51.636 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T23:28:51.637 INFO:teuthology.orchestra.run.smithi170.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T23:28:51.637 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:51.637 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-28T23:28:52.110 DEBUG:teuthology.parallel:result is None 2024-03-28T23:28:52.958 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T23:28:52.958 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:52.958 INFO:teuthology.orchestra.run.smithi161.stdout:Upgraded: 2024-03-28T23:28:52.959 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T23:28:52.959 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T23:28:52.959 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T23:28:52.959 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T23:28:52.959 INFO:teuthology.orchestra.run.smithi161.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T23:28:52.959 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T23:28:52.959 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T23:28:52.959 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T23:28:52.959 INFO:teuthology.orchestra.run.smithi161.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T23:28:52.959 INFO:teuthology.orchestra.run.smithi161.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T23:28:52.959 INFO:teuthology.orchestra.run.smithi161.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T23:28:52.959 INFO:teuthology.orchestra.run.smithi161.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T23:28:52.959 INFO:teuthology.orchestra.run.smithi161.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T23:28:52.960 INFO:teuthology.orchestra.run.smithi161.stdout: librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.960 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.960 INFO:teuthology.orchestra.run.smithi161.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T23:28:52.960 INFO:teuthology.orchestra.run.smithi161.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T23:28:52.960 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T23:28:52.960 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T23:28:52.960 INFO:teuthology.orchestra.run.smithi161.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T23:28:52.960 INFO:teuthology.orchestra.run.smithi161.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T23:28:52.960 INFO:teuthology.orchestra.run.smithi161.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T23:28:52.960 INFO:teuthology.orchestra.run.smithi161.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T23:28:52.960 INFO:teuthology.orchestra.run.smithi161.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T23:28:52.961 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-28T23:28:52.961 INFO:teuthology.orchestra.run.smithi161.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T23:28:52.961 INFO:teuthology.orchestra.run.smithi161.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T23:28:52.961 INFO:teuthology.orchestra.run.smithi161.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T23:28:52.961 INFO:teuthology.orchestra.run.smithi161.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T23:28:52.961 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.961 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.961 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.961 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.962 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-grafana-dashboards-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T23:28:52.962 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.962 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.963 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.963 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T23:28:52.963 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T23:28:52.963 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T23:28:52.963 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T23:28:52.963 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T23:28:52.963 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.963 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.963 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-prometheus-alerts-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T23:28:52.963 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.963 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.963 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.963 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T23:28:52.964 INFO:teuthology.orchestra.run.smithi161.stdout: cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T23:28:52.964 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.964 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-shell-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.964 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-top-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T23:28:52.964 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T23:28:52.964 INFO:teuthology.orchestra.run.smithi161.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T23:28:52.964 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T23:28:52.964 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T23:28:52.964 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T23:28:52.964 INFO:teuthology.orchestra.run.smithi161.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T23:28:52.964 INFO:teuthology.orchestra.run.smithi161.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T23:28:52.964 INFO:teuthology.orchestra.run.smithi161.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T23:28:52.964 INFO:teuthology.orchestra.run.smithi161.stdout: jq-1.6-16.el9.x86_64 2024-03-28T23:28:52.964 INFO:teuthology.orchestra.run.smithi161.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T23:28:52.965 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T23:28:52.965 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T23:28:52.965 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-devel-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.965 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.965 INFO:teuthology.orchestra.run.smithi161.stdout: libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.965 INFO:teuthology.orchestra.run.smithi161.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T23:28:52.965 INFO:teuthology.orchestra.run.smithi161.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T23:28:52.965 INFO:teuthology.orchestra.run.smithi161.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T23:28:52.965 INFO:teuthology.orchestra.run.smithi161.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T23:28:52.965 INFO:teuthology.orchestra.run.smithi161.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T23:28:52.965 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.965 INFO:teuthology.orchestra.run.smithi161.stdout: libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.965 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T23:28:52.966 INFO:teuthology.orchestra.run.smithi161.stdout: librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.966 INFO:teuthology.orchestra.run.smithi161.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T23:28:52.966 INFO:teuthology.orchestra.run.smithi161.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T23:28:52.966 INFO:teuthology.orchestra.run.smithi161.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T23:28:52.966 INFO:teuthology.orchestra.run.smithi161.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T23:28:52.966 INFO:teuthology.orchestra.run.smithi161.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T23:28:52.966 INFO:teuthology.orchestra.run.smithi161.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T23:28:52.966 INFO:teuthology.orchestra.run.smithi161.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T23:28:52.966 INFO:teuthology.orchestra.run.smithi161.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T23:28:52.966 INFO:teuthology.orchestra.run.smithi161.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T23:28:52.966 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T23:28:52.966 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T23:28:52.966 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T23:28:52.966 INFO:teuthology.orchestra.run.smithi161.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T23:28:52.967 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T23:28:52.967 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T23:28:52.967 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T23:28:52.967 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T23:28:52.967 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T23:28:52.967 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T23:28:52.967 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T23:28:52.967 INFO:teuthology.orchestra.run.smithi161.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T23:28:52.967 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T23:28:52.967 INFO:teuthology.orchestra.run.smithi161.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T23:28:52.967 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T23:28:52.967 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.967 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.968 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.968 INFO:teuthology.orchestra.run.smithi161.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T23:28:52.968 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T23:28:52.968 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T23:28:52.968 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T23:28:52.968 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T23:28:52.968 INFO:teuthology.orchestra.run.smithi161.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T23:28:52.968 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T23:28:52.968 INFO:teuthology.orchestra.run.smithi161.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T23:28:52.968 INFO:teuthology.orchestra.run.smithi161.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T23:28:52.968 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T23:28:52.968 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T23:28:52.968 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T23:28:52.968 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T23:28:52.969 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T23:28:52.969 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T23:28:52.969 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T23:28:52.969 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T23:28:52.969 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T23:28:52.969 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T23:28:52.969 INFO:teuthology.orchestra.run.smithi161.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T23:28:52.969 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T23:28:52.969 INFO:teuthology.orchestra.run.smithi161.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T23:28:52.969 INFO:teuthology.orchestra.run.smithi161.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T23:28:52.969 INFO:teuthology.orchestra.run.smithi161.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T23:28:52.969 INFO:teuthology.orchestra.run.smithi161.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T23:28:52.969 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T23:28:52.969 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T23:28:52.970 INFO:teuthology.orchestra.run.smithi161.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T23:28:52.970 INFO:teuthology.orchestra.run.smithi161.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T23:28:52.970 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T23:28:52.970 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T23:28:52.970 INFO:teuthology.orchestra.run.smithi161.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T23:28:52.970 INFO:teuthology.orchestra.run.smithi161.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T23:28:52.970 INFO:teuthology.orchestra.run.smithi161.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T23:28:52.970 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T23:28:52.970 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T23:28:52.970 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T23:28:52.970 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T23:28:52.970 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T23:28:52.970 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T23:28:52.970 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.971 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.971 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T23:28:52.971 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T23:28:52.971 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T23:28:52.971 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.971 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T23:28:52.971 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T23:28:52.971 INFO:teuthology.orchestra.run.smithi161.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T23:28:52.971 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T23:28:52.971 INFO:teuthology.orchestra.run.smithi161.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T23:28:52.971 INFO:teuthology.orchestra.run.smithi161.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T23:28:52.971 INFO:teuthology.orchestra.run.smithi161.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T23:28:52.971 INFO:teuthology.orchestra.run.smithi161.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T23:28:52.971 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T23:28:52.972 INFO:teuthology.orchestra.run.smithi161.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T23:28:52.972 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T23:28:52.972 INFO:teuthology.orchestra.run.smithi161.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T23:28:52.972 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T23:28:52.972 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T23:28:52.972 INFO:teuthology.orchestra.run.smithi161.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T23:28:52.972 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.972 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.972 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T23:28:52.972 INFO:teuthology.orchestra.run.smithi161.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T23:28:52.972 INFO:teuthology.orchestra.run.smithi161.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T23:28:52.972 INFO:teuthology.orchestra.run.smithi161.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T23:28:52.972 INFO:teuthology.orchestra.run.smithi161.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T23:28:52.972 INFO:teuthology.orchestra.run.smithi161.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T23:28:52.972 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T23:28:52.973 INFO:teuthology.orchestra.run.smithi161.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T23:28:52.973 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:52.973 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-28T23:28:53.473 DEBUG:teuthology.parallel:result is None 2024-03-28T23:28:53.474 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T23:28:53.611 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T23:28:53.633 INFO:teuthology.orchestra.run.smithi161.stdout:19.0.0-1899.g323d23dc.el9 2024-03-28T23:28:53.634 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1899.g323d23dc.el9 2024-03-28T23:28:53.634 INFO:teuthology.task.install:The correct ceph version 19.0.0-1899.g323d23dc is installed. 2024-03-28T23:28:53.636 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T23:28:53.772 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T23:28:53.798 INFO:teuthology.orchestra.run.smithi170.stdout:19.0.0-1899.g323d23dc.el9 2024-03-28T23:28:53.798 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1899.g323d23dc.el9 2024-03-28T23:28:53.798 INFO:teuthology.task.install:The correct ceph version 19.0.0-1899.g323d23dc is installed. 2024-03-28T23:28:53.800 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-28T23:28:53.801 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:28:53.801 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T23:28:53.830 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:28:53.831 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T23:28:53.870 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-28T23:28:53.871 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:28:53.871 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T23:28:53.897 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T23:28:53.964 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:28:53.965 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T23:28:53.994 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T23:28:54.062 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-28T23:28:54.062 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:28:54.062 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T23:28:54.088 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T23:28:54.156 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:28:54.156 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T23:28:54.185 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T23:28:54.253 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-28T23:28:54.253 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:28:54.253 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T23:28:54.280 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T23:28:54.347 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:28:54.347 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T23:28:54.375 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T23:28:54.443 INFO:teuthology.run_tasks:Running task ceph... 2024-03-28T23:28:54.527 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-28T23:28:54.527 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 777 /var/log/ceph 2024-03-28T23:28:54.529 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 777 /var/log/ceph 2024-03-28T23:28:54.556 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-28T23:28:54.556 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-28T23:28:54.596 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-28T23:28:54.623 INFO:tasks.ceph:Creating extra log directories... 2024-03-28T23:28:54.623 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-28T23:28:54.663 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-28T23:28:54.695 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-28T23:28:54.695 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-28T23:28:54.695 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-28T23:28:54.696 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:28:54.696 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-28T23:28:54.742 DEBUG:teuthology.orchestra.run.smithi161:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-28T23:28:54.859 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-28T23:28:54.886 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:28:54.886 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-28T23:28:54.901 DEBUG:teuthology.orchestra.run.smithi170:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-28T23:28:55.018 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-28T23:28:55.049 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-28T23:28:55.050 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds', '\\(MDS_INSUFFICIENT_STANDBY\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MDS_DAMAGE\\)', '\\(FS_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-28T23:28:55.050 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627196', 'branch': 'squid', 'description': 'fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7627196', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds', '\\(MDS_INSUFFICIENT_STANDBY\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MDS_DAMAGE\\)', '\\(FS_DEGRADED\\)'], 'sha1': '323d23dc228bbcb7f16f188fdc9c0dcb801844be'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'ceph-fuse': {'disabled': True}, 'install': {'ceph': {'flavor': 'default', 'sha1': '323d23dc228bbcb7f16f188fdc9c0dcb801844be'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': '55f5138942aa4866e49329c9103468889cec3f73'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']], 'seed': 7214, 'sha1': '323d23dc228bbcb7f16f188fdc9c0dcb801844be', 'sleep_before_teardown': 0, 'subset': '39/512', 'suite': 'fs', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '55f5138942aa4866e49329c9103468889cec3f73', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi161.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBByTCof/EqEz1jIVz78GqGKb+p3QSZbaeon8+Zw2GmOCmoalhHdC1zWHiRPrTT1QZKvq2jFe7DSq61l/GXWj/YU=', 'smithi170.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOcXNoCIPqw31oOW4OMimd9Wdo5l/Jf3E136yIF9WJYmGzcPjP9ayv8TR+bXAGJHUV+4rOjmqVjUj7KgskU0mUk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '323d23dc228bbcb7f16f188fdc9c0dcb801844be'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds', '\\(MDS_INSUFFICIENT_STANDBY\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MDS_DAMAGE\\)', '\\(FS_DEGRADED\\)'], 'sha1': '323d23dc228bbcb7f16f188fdc9c0dcb801844be', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'exec': {'mon.a': ['ceph config set mgr mgr/crash/warn_recent_interval 0']}}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_failover']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-27_21:24:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-28T23:28:55.050 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-28T23:28:55.053 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-28T23:28:55.099 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -d -m0777 -- /var/run/ceph 2024-03-28T23:28:55.108 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m0777 -- /var/run/ceph 2024-03-28T23:28:55.165 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:28:55.165 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T23:28:55.188 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T23:28:55.188 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_1 2024-03-28T23:28:55.242 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T23:28:55.243 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T23:28:55.243 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-28T23:28:55.243 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T23:28:55.243 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T23:28:55.243 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-03-28 23:28:45.240729715 +0000 2024-03-28T23:28:55.243 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-03-28 23:26:29.726156309 +0000 2024-03-28T23:28:55.243 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-03-28 23:26:29.726156309 +0000 2024-03-28T23:28:55.243 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-03-28 23:26:29.726156309 +0000 2024-03-28T23:28:55.243 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T23:28:55.308 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-03-28T23:28:55.308 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-03-28T23:28:55.308 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000183933 s, 2.8 MB/s 2024-03-28T23:28:55.309 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T23:28:55.365 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_2 2024-03-28T23:28:55.420 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T23:28:55.421 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T23:28:55.421 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-28T23:28:55.421 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T23:28:55.421 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T23:28:55.421 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-03-28 23:28:45.240729715 +0000 2024-03-28T23:28:55.421 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-03-28 23:26:30.061149237 +0000 2024-03-28T23:28:55.421 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-03-28 23:26:30.061149237 +0000 2024-03-28T23:28:55.421 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-03-28 23:26:30.061149237 +0000 2024-03-28T23:28:55.422 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T23:28:55.486 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-03-28T23:28:55.486 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-03-28T23:28:55.487 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000234862 s, 2.2 MB/s 2024-03-28T23:28:55.488 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T23:28:55.543 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_3 2024-03-28T23:28:55.598 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T23:28:55.599 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T23:28:55.599 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-28T23:28:55.599 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T23:28:55.599 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T23:28:55.599 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-03-28 23:28:45.240729715 +0000 2024-03-28T23:28:55.599 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-03-28 23:26:30.575138385 +0000 2024-03-28T23:28:55.599 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-03-28 23:26:30.575138385 +0000 2024-03-28T23:28:55.599 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-03-28 23:26:30.575138385 +0000 2024-03-28T23:28:55.600 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T23:28:55.664 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-03-28T23:28:55.664 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-03-28T23:28:55.664 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000227056 s, 2.3 MB/s 2024-03-28T23:28:55.666 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T23:28:55.721 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_4 2024-03-28T23:28:55.776 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T23:28:55.776 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T23:28:55.777 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-28T23:28:55.777 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T23:28:55.777 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T23:28:55.777 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-03-28 23:28:45.241729694 +0000 2024-03-28T23:28:55.777 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-03-28 23:26:30.928130932 +0000 2024-03-28T23:28:55.777 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-03-28 23:26:30.928130932 +0000 2024-03-28T23:28:55.777 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-03-28 23:26:30.928130932 +0000 2024-03-28T23:28:55.777 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T23:28:55.842 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-03-28T23:28:55.842 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-03-28T23:28:55.842 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000208613 s, 2.5 MB/s 2024-03-28T23:28:55.843 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T23:28:55.899 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T23:28:55.900 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:28:55.900 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T23:28:55.918 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T23:28:55.919 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_1 2024-03-28T23:28:55.976 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T23:28:55.976 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T23:28:55.977 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-28T23:28:55.977 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T23:28:55.977 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T23:28:55.977 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-28 23:28:43.388803796 +0000 2024-03-28T23:28:55.977 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-28 23:26:38.893738783 +0000 2024-03-28T23:28:55.977 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-28 23:26:38.893738783 +0000 2024-03-28T23:28:55.977 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-28 23:26:38.893738783 +0000 2024-03-28T23:28:55.977 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T23:28:56.046 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-28T23:28:56.046 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-28T23:28:56.046 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 9.586e-05 s, 5.3 MB/s 2024-03-28T23:28:56.047 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T23:28:56.106 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_2 2024-03-28T23:28:56.163 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T23:28:56.164 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T23:28:56.164 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-28T23:28:56.164 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T23:28:56.164 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T23:28:56.164 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-28 23:28:43.389803774 +0000 2024-03-28T23:28:56.164 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-28 23:26:39.262730321 +0000 2024-03-28T23:28:56.164 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-28 23:26:39.262730321 +0000 2024-03-28T23:28:56.164 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-28 23:26:39.262730321 +0000 2024-03-28T23:28:56.164 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T23:28:56.231 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-28T23:28:56.231 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-28T23:28:56.231 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000110731 s, 4.6 MB/s 2024-03-28T23:28:56.232 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T23:28:56.289 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_3 2024-03-28T23:28:56.345 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T23:28:56.346 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T23:28:56.346 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-28T23:28:56.346 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T23:28:56.346 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T23:28:56.346 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-28 23:28:43.389803774 +0000 2024-03-28T23:28:56.346 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-28 23:26:39.606722432 +0000 2024-03-28T23:28:56.346 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-28 23:26:39.606722432 +0000 2024-03-28T23:28:56.346 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-28 23:26:39.606722432 +0000 2024-03-28T23:28:56.347 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T23:28:56.413 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-28T23:28:56.413 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-28T23:28:56.413 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 8.5425e-05 s, 6.0 MB/s 2024-03-28T23:28:56.414 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T23:28:56.470 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_4 2024-03-28T23:28:56.526 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T23:28:56.526 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T23:28:56.526 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-28T23:28:56.526 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T23:28:56.526 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T23:28:56.526 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-28 23:28:43.389803774 +0000 2024-03-28T23:28:56.526 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-28 23:26:39.938714819 +0000 2024-03-28T23:28:56.526 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-28 23:26:39.938714819 +0000 2024-03-28T23:28:56.526 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-28 23:26:39.938714819 +0000 2024-03-28T23:28:56.527 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T23:28:56.592 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-28T23:28:56.592 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-28T23:28:56.592 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000104447 s, 4.9 MB/s 2024-03-28T23:28:56.593 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T23:28:56.650 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T23:28:56.650 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi161.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi170.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-28T23:28:56.650 INFO:tasks.ceph:Generating config... 2024-03-28T23:28:56.651 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-28T23:28:56.651 INFO:tasks.ceph:[client] debug client = 20 2024-03-28T23:28:56.651 INFO:tasks.ceph:[client] debug ms = 1 2024-03-28T23:28:56.651 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-28T23:28:56.652 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-28T23:28:56.652 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-28T23:28:56.652 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-28T23:28:56.652 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-28T23:28:56.652 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-28T23:28:56.652 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-28T23:28:56.652 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-28T23:28:56.652 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-28T23:28:56.652 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-28T23:28:56.652 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-28T23:28:56.652 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-28T23:28:56.652 INFO:tasks.ceph:[mgr] debug client = 20 2024-03-28T23:28:56.652 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-28T23:28:56.652 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-28T23:28:56.652 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-28T23:28:56.653 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-28T23:28:56.653 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-28T23:28:56.653 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-03-28T23:28:56.653 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-28T23:28:56.653 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-28T23:28:56.653 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-28T23:28:56.653 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-28T23:28:56.653 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-28T23:28:56.653 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-28T23:28:56.653 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-28T23:28:56.653 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-28T23:28:56.653 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-28T23:28:56.653 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-28T23:28:56.653 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-28T23:28:56.654 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-28T23:28:56.654 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-28T23:28:56.654 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-28T23:28:56.654 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-28T23:28:56.654 INFO:tasks.ceph:Setting up mon.a... 2024-03-28T23:28:56.654 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-28T23:28:56.717 INFO:teuthology.orchestra.run.smithi161.stdout:creating /etc/ceph/ceph.keyring 2024-03-28T23:28:56.720 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-28T23:28:56.765 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-28T23:28:56.793 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.161'), ('mon.b', '172.21.15.170'), ('mon.c', '[v2:172.21.15.170:3301,v1:172.21.15.170:6790]')] 2024-03-28T23:28:56.793 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.161,172.21.15.170,[v2:172.21.15.170:3301,v1:172.21.15.170:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-28T23:28:56.794 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:28:56.794 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-28T23:28:56.848 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.161 --add b 172.21.15.170 --addv c '[v2:172.21.15.170:3301,v1:172.21.15.170:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-28T23:28:56.934 INFO:teuthology.orchestra.run.smithi161.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-28T23:28:56.934 INFO:teuthology.orchestra.run.smithi161.stdout:monmaptool: generated fsid cc975a39-1de3-40a7-9dc0-0e645d96eff8 2024-03-28T23:28:56.934 INFO:teuthology.orchestra.run.smithi161.stdout:setting min_mon_release = quincy 2024-03-28T23:28:56.934 INFO:teuthology.orchestra.run.smithi161.stdout:epoch 0 2024-03-28T23:28:56.934 INFO:teuthology.orchestra.run.smithi161.stdout:fsid cc975a39-1de3-40a7-9dc0-0e645d96eff8 2024-03-28T23:28:56.934 INFO:teuthology.orchestra.run.smithi161.stdout:last_changed 2024-03-28T23:28:56.934202+0000 2024-03-28T23:28:56.934 INFO:teuthology.orchestra.run.smithi161.stdout:created 2024-03-28T23:28:56.934202+0000 2024-03-28T23:28:56.934 INFO:teuthology.orchestra.run.smithi161.stdout:min_mon_release 17 (quincy) 2024-03-28T23:28:56.934 INFO:teuthology.orchestra.run.smithi161.stdout:election_strategy: 1 2024-03-28T23:28:56.934 INFO:teuthology.orchestra.run.smithi161.stdout:0: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.a 2024-03-28T23:28:56.935 INFO:teuthology.orchestra.run.smithi161.stdout:1: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.b 2024-03-28T23:28:56.935 INFO:teuthology.orchestra.run.smithi161.stdout:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2024-03-28T23:28:56.935 INFO:teuthology.orchestra.run.smithi161.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-28T23:28:56.936 DEBUG:teuthology.orchestra.run.smithi161:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-28T23:28:56.950 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID cc975a39-1de3-40a7-9dc0-0e645d96eff8... 2024-03-28T23:28:56.951 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-28T23:28:56.993 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-28T23:28:57.040 INFO:teuthology.orchestra.run.smithi161.stdout:[global] 2024-03-28T23:28:57.040 INFO:teuthology.orchestra.run.smithi161.stdout: chdir = "" 2024-03-28T23:28:57.040 INFO:teuthology.orchestra.run.smithi161.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-28T23:28:57.041 INFO:teuthology.orchestra.run.smithi161.stdout: auth supported = cephx 2024-03-28T23:28:57.041 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.041 INFO:teuthology.orchestra.run.smithi161.stdout: filestore xattr use omap = true 2024-03-28T23:28:57.041 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.041 INFO:teuthology.orchestra.run.smithi161.stdout: mon clock drift allowed = 1.000 2024-03-28T23:28:57.041 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.041 INFO:teuthology.orchestra.run.smithi161.stdout: osd crush chooseleaf type = 0 2024-03-28T23:28:57.041 INFO:teuthology.orchestra.run.smithi161.stdout: auth debug = true 2024-03-28T23:28:57.041 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.041 INFO:teuthology.orchestra.run.smithi161.stdout: ms die on old message = true 2024-03-28T23:28:57.041 INFO:teuthology.orchestra.run.smithi161.stdout: ms die on bug = true 2024-03-28T23:28:57.041 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.041 INFO:teuthology.orchestra.run.smithi161.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-28T23:28:57.042 INFO:teuthology.orchestra.run.smithi161.stdout: mon pg warn max object skew = 0 2024-03-28T23:28:57.042 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.042 INFO:teuthology.orchestra.run.smithi161.stdout: # disable pg_autoscaler by default for new pools 2024-03-28T23:28:57.042 INFO:teuthology.orchestra.run.smithi161.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-28T23:28:57.042 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.042 INFO:teuthology.orchestra.run.smithi161.stdout: osd pool default size = 2 2024-03-28T23:28:57.042 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.042 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd allow primary affinity = true 2024-03-28T23:28:57.042 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd allow pg remap = true 2024-03-28T23:28:57.042 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on legacy crush tunables = false 2024-03-28T23:28:57.042 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on crush straw calc version zero = false 2024-03-28T23:28:57.042 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on no sortbitwise = false 2024-03-28T23:28:57.042 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on osd down out interval zero = false 2024-03-28T23:28:57.042 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on too few osds = false 2024-03-28T23:28:57.043 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-28T23:28:57.043 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-28T23:28:57.043 INFO:teuthology.orchestra.run.smithi161.stdout: mon_allow_pool_size_one = true 2024-03-28T23:28:57.043 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.043 INFO:teuthology.orchestra.run.smithi161.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-28T23:28:57.043 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.043 INFO:teuthology.orchestra.run.smithi161.stdout: osd default data pool replay window = 5 2024-03-28T23:28:57.043 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.043 INFO:teuthology.orchestra.run.smithi161.stdout: mon allow pool delete = true 2024-03-28T23:28:57.043 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.043 INFO:teuthology.orchestra.run.smithi161.stdout: mon cluster log file level = debug 2024-03-28T23:28:57.043 INFO:teuthology.orchestra.run.smithi161.stdout: debug asserts on shutdown = true 2024-03-28T23:28:57.043 INFO:teuthology.orchestra.run.smithi161.stdout: mon health detail to clog = false 2024-03-28T23:28:57.043 INFO:teuthology.orchestra.run.smithi161.stdout: mon host = "172.21.15.161,172.21.15.170,[v2:172.21.15.170:3301,v1:172.21.15.170:6790]" 2024-03-28T23:28:57.043 INFO:teuthology.orchestra.run.smithi161.stdout: fsid = cc975a39-1de3-40a7-9dc0-0e645d96eff8 2024-03-28T23:28:57.044 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.044 INFO:teuthology.orchestra.run.smithi161.stdout:[osd] 2024-03-28T23:28:57.044 INFO:teuthology.orchestra.run.smithi161.stdout: osd journal size = 100 2024-03-28T23:28:57.044 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.044 INFO:teuthology.orchestra.run.smithi161.stdout: osd scrub load threshold = 5.0 2024-03-28T23:28:57.044 INFO:teuthology.orchestra.run.smithi161.stdout: osd scrub max interval = 600 2024-03-28T23:28:57.044 INFO:teuthology.orchestra.run.smithi161.stdout: osd mclock profile = high_recovery_ops 2024-03-28T23:28:57.044 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.044 INFO:teuthology.orchestra.run.smithi161.stdout: osd recover clone overlap = true 2024-03-28T23:28:57.044 INFO:teuthology.orchestra.run.smithi161.stdout: osd recovery max chunk = 1048576 2024-03-28T23:28:57.044 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.044 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug shutdown = true 2024-03-28T23:28:57.044 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug op order = true 2024-03-28T23:28:57.044 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug verify stray on activate = true 2024-03-28T23:28:57.044 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug trim objects = true 2024-03-28T23:28:57.045 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.045 INFO:teuthology.orchestra.run.smithi161.stdout: osd open classes on start = true 2024-03-28T23:28:57.045 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug pg log writeout = true 2024-03-28T23:28:57.045 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.045 INFO:teuthology.orchestra.run.smithi161.stdout: osd deep scrub update digest min age = 30 2024-03-28T23:28:57.045 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.045 INFO:teuthology.orchestra.run.smithi161.stdout: osd map max advance = 10 2024-03-28T23:28:57.045 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.045 INFO:teuthology.orchestra.run.smithi161.stdout: journal zero on create = true 2024-03-28T23:28:57.045 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.045 INFO:teuthology.orchestra.run.smithi161.stdout: filestore ondisk finisher threads = 3 2024-03-28T23:28:57.045 INFO:teuthology.orchestra.run.smithi161.stdout: filestore apply finisher threads = 3 2024-03-28T23:28:57.045 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.045 INFO:teuthology.orchestra.run.smithi161.stdout: bdev debug aio = true 2024-03-28T23:28:57.046 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug misdirected ops = true 2024-03-28T23:28:57.046 INFO:teuthology.orchestra.run.smithi161.stdout: bluestore block size = 96636764160 2024-03-28T23:28:57.046 INFO:teuthology.orchestra.run.smithi161.stdout: bluestore compression mode = aggressive 2024-03-28T23:28:57.046 INFO:teuthology.orchestra.run.smithi161.stdout: bluestore fsck on mount = True 2024-03-28T23:28:57.046 INFO:teuthology.orchestra.run.smithi161.stdout: debug bluefs = 20 2024-03-28T23:28:57.046 INFO:teuthology.orchestra.run.smithi161.stdout: debug bluestore = 20 2024-03-28T23:28:57.046 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-03-28T23:28:57.046 INFO:teuthology.orchestra.run.smithi161.stdout: debug osd = 20 2024-03-28T23:28:57.046 INFO:teuthology.orchestra.run.smithi161.stdout: debug rocksdb = 10 2024-03-28T23:28:57.046 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd backfillfull_ratio = 0.85 2024-03-28T23:28:57.046 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd full ratio = 0.9 2024-03-28T23:28:57.046 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd nearfull ratio = 0.8 2024-03-28T23:28:57.046 INFO:teuthology.orchestra.run.smithi161.stdout: osd failsafe full ratio = 0.95 2024-03-28T23:28:57.046 INFO:teuthology.orchestra.run.smithi161.stdout: osd objectstore = bluestore 2024-03-28T23:28:57.046 INFO:teuthology.orchestra.run.smithi161.stdout: osd op complaint time = 180 2024-03-28T23:28:57.047 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.047 INFO:teuthology.orchestra.run.smithi161.stdout:[mgr] 2024-03-28T23:28:57.047 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-03-28T23:28:57.047 INFO:teuthology.orchestra.run.smithi161.stdout: debug mgr = 20 2024-03-28T23:28:57.047 INFO:teuthology.orchestra.run.smithi161.stdout: debug mon = 20 2024-03-28T23:28:57.047 INFO:teuthology.orchestra.run.smithi161.stdout: debug auth = 20 2024-03-28T23:28:57.047 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min pgs per osd = 4 2024-03-28T23:28:57.047 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min bytes per osd = 10 2024-03-28T23:28:57.047 INFO:teuthology.orchestra.run.smithi161.stdout: mgr/telemetry/nag = false 2024-03-28T23:28:57.047 INFO:teuthology.orchestra.run.smithi161.stdout: debug client = 20 2024-03-28T23:28:57.047 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.047 INFO:teuthology.orchestra.run.smithi161.stdout:[mon] 2024-03-28T23:28:57.047 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-03-28T23:28:57.047 INFO:teuthology.orchestra.run.smithi161.stdout: debug mon = 20 2024-03-28T23:28:57.047 INFO:teuthology.orchestra.run.smithi161.stdout: debug paxos = 20 2024-03-28T23:28:57.048 INFO:teuthology.orchestra.run.smithi161.stdout: debug auth = 20 2024-03-28T23:28:57.048 INFO:teuthology.orchestra.run.smithi161.stdout: mon data avail warn = 5 2024-03-28T23:28:57.048 INFO:teuthology.orchestra.run.smithi161.stdout: mon mgr mkfs grace = 240 2024-03-28T23:28:57.048 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min pgs per osd = 4 2024-03-28T23:28:57.048 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd reporter subtree level = osd 2024-03-28T23:28:57.048 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd prime pg temp = true 2024-03-28T23:28:57.048 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min bytes per osd = 10 2024-03-28T23:28:57.048 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.048 INFO:teuthology.orchestra.run.smithi161.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-28T23:28:57.048 INFO:teuthology.orchestra.run.smithi161.stdout: auth mon ticket ttl = 660 # 11m 2024-03-28T23:28:57.048 INFO:teuthology.orchestra.run.smithi161.stdout: auth service ticket ttl = 240 # 4m 2024-03-28T23:28:57.048 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.048 INFO:teuthology.orchestra.run.smithi161.stdout: # don't complain about insecure global_id in the test suite 2024-03-28T23:28:57.048 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-28T23:28:57.049 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-28T23:28:57.049 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.049 INFO:teuthology.orchestra.run.smithi161.stdout: # 1m isn't quite enough 2024-03-28T23:28:57.049 INFO:teuthology.orchestra.run.smithi161.stdout: mon_down_mkfs_grace = 2m 2024-03-28T23:28:57.049 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.049 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_filestore_osds = false 2024-03-28T23:28:57.049 INFO:teuthology.orchestra.run.smithi161.stdout: mon down mkfs grace = 300 2024-03-28T23:28:57.049 INFO:teuthology.orchestra.run.smithi161.stdout: mon op complaint time = 120 2024-03-28T23:28:57.049 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:28:57.049 INFO:teuthology.orchestra.run.smithi161.stdout:[client] 2024-03-28T23:28:57.049 INFO:teuthology.orchestra.run.smithi161.stdout: rgw cache enabled = true 2024-03-28T23:28:57.049 INFO:teuthology.orchestra.run.smithi161.stdout: rgw enable ops log = true 2024-03-28T23:28:57.049 INFO:teuthology.orchestra.run.smithi161.stdout: rgw enable usage log = true 2024-03-28T23:28:57.049 INFO:teuthology.orchestra.run.smithi161.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-28T23:28:57.049 INFO:teuthology.orchestra.run.smithi161.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-28T23:28:57.050 INFO:teuthology.orchestra.run.smithi161.stdout: client mount timeout = 600 2024-03-28T23:28:57.050 INFO:teuthology.orchestra.run.smithi161.stdout: debug client = 20 2024-03-28T23:28:57.050 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-03-28T23:28:57.050 INFO:teuthology.orchestra.run.smithi161.stdout: rados mon op timeout = 900 2024-03-28T23:28:57.050 INFO:teuthology.orchestra.run.smithi161.stdout: rados osd op timeout = 900 2024-03-28T23:28:57.050 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.a] 2024-03-28T23:28:57.050 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.b] 2024-03-28T23:28:57.050 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.c] 2024-03-28T23:28:57.050 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.a] 2024-03-28T23:28:57.050 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.c] 2024-03-28T23:28:57.050 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.b] 2024-03-28T23:28:57.050 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.d] 2024-03-28T23:28:57.050 INFO:teuthology.orchestra.run.smithi161.stdout:[mds] 2024-03-28T23:28:57.050 INFO:teuthology.orchestra.run.smithi161.stdout: debug mds = 20 2024-03-28T23:28:57.050 INFO:teuthology.orchestra.run.smithi161.stdout: debug mds balancer = 20 2024-03-28T23:28:57.051 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-03-28T23:28:57.051 INFO:teuthology.orchestra.run.smithi161.stdout: mds debug frag = True 2024-03-28T23:28:57.051 INFO:teuthology.orchestra.run.smithi161.stdout: mds debug scatterstat = True 2024-03-28T23:28:57.051 INFO:teuthology.orchestra.run.smithi161.stdout: mds op complaint time = 180 2024-03-28T23:28:57.051 INFO:teuthology.orchestra.run.smithi161.stdout: mds verify scatter = True 2024-03-28T23:28:57.051 INFO:teuthology.orchestra.run.smithi161.stdout: osd op complaint time = 180 2024-03-28T23:28:57.051 INFO:teuthology.orchestra.run.smithi161.stdout: rados mon op timeout = 900 2024-03-28T23:28:57.051 INFO:teuthology.orchestra.run.smithi161.stdout: rados osd op timeout = 900 2024-03-28T23:28:57.051 INFO:teuthology.orchestra.run.smithi170.stdout:[global] 2024-03-28T23:28:57.052 INFO:teuthology.orchestra.run.smithi170.stdout: chdir = "" 2024-03-28T23:28:57.052 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-28T23:28:57.052 INFO:teuthology.orchestra.run.smithi170.stdout: auth supported = cephx 2024-03-28T23:28:57.052 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.052 INFO:teuthology.orchestra.run.smithi170.stdout: filestore xattr use omap = true 2024-03-28T23:28:57.052 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.052 INFO:teuthology.orchestra.run.smithi170.stdout: mon clock drift allowed = 1.000 2024-03-28T23:28:57.052 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.052 INFO:teuthology.orchestra.run.smithi170.stdout: osd crush chooseleaf type = 0 2024-03-28T23:28:57.052 INFO:teuthology.orchestra.run.smithi170.stdout: auth debug = true 2024-03-28T23:28:57.052 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.052 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on old message = true 2024-03-28T23:28:57.052 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on bug = true 2024-03-28T23:28:57.052 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.053 INFO:teuthology.orchestra.run.smithi170.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-28T23:28:57.053 INFO:teuthology.orchestra.run.smithi170.stdout: mon pg warn max object skew = 0 2024-03-28T23:28:57.053 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.053 INFO:teuthology.orchestra.run.smithi170.stdout: # disable pg_autoscaler by default for new pools 2024-03-28T23:28:57.053 INFO:teuthology.orchestra.run.smithi170.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-28T23:28:57.053 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.053 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default size = 2 2024-03-28T23:28:57.053 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.053 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow primary affinity = true 2024-03-28T23:28:57.053 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow pg remap = true 2024-03-28T23:28:57.053 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on legacy crush tunables = false 2024-03-28T23:28:57.053 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on crush straw calc version zero = false 2024-03-28T23:28:57.053 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on no sortbitwise = false 2024-03-28T23:28:57.053 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on osd down out interval zero = false 2024-03-28T23:28:57.054 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on too few osds = false 2024-03-28T23:28:57.054 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-28T23:28:57.054 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-28T23:28:57.054 INFO:teuthology.orchestra.run.smithi170.stdout: mon_allow_pool_size_one = true 2024-03-28T23:28:57.054 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.054 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-28T23:28:57.054 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.054 INFO:teuthology.orchestra.run.smithi170.stdout: osd default data pool replay window = 5 2024-03-28T23:28:57.054 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.054 INFO:teuthology.orchestra.run.smithi170.stdout: mon allow pool delete = true 2024-03-28T23:28:57.054 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.054 INFO:teuthology.orchestra.run.smithi170.stdout: mon cluster log file level = debug 2024-03-28T23:28:57.054 INFO:teuthology.orchestra.run.smithi170.stdout: debug asserts on shutdown = true 2024-03-28T23:28:57.054 INFO:teuthology.orchestra.run.smithi170.stdout: mon health detail to clog = false 2024-03-28T23:28:57.055 INFO:teuthology.orchestra.run.smithi170.stdout: mon host = "172.21.15.161,172.21.15.170,[v2:172.21.15.170:3301,v1:172.21.15.170:6790]" 2024-03-28T23:28:57.055 INFO:teuthology.orchestra.run.smithi170.stdout: fsid = cc975a39-1de3-40a7-9dc0-0e645d96eff8 2024-03-28T23:28:57.055 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.055 INFO:teuthology.orchestra.run.smithi170.stdout:[osd] 2024-03-28T23:28:57.055 INFO:teuthology.orchestra.run.smithi170.stdout: osd journal size = 100 2024-03-28T23:28:57.055 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.055 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub load threshold = 5.0 2024-03-28T23:28:57.055 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub max interval = 600 2024-03-28T23:28:57.055 INFO:teuthology.orchestra.run.smithi170.stdout: osd mclock profile = high_recovery_ops 2024-03-28T23:28:57.055 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.055 INFO:teuthology.orchestra.run.smithi170.stdout: osd recover clone overlap = true 2024-03-28T23:28:57.055 INFO:teuthology.orchestra.run.smithi170.stdout: osd recovery max chunk = 1048576 2024-03-28T23:28:57.055 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.055 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug shutdown = true 2024-03-28T23:28:57.056 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug op order = true 2024-03-28T23:28:57.056 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug verify stray on activate = true 2024-03-28T23:28:57.056 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug trim objects = true 2024-03-28T23:28:57.056 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.056 INFO:teuthology.orchestra.run.smithi170.stdout: osd open classes on start = true 2024-03-28T23:28:57.056 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug pg log writeout = true 2024-03-28T23:28:57.056 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.056 INFO:teuthology.orchestra.run.smithi170.stdout: osd deep scrub update digest min age = 30 2024-03-28T23:28:57.056 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.056 INFO:teuthology.orchestra.run.smithi170.stdout: osd map max advance = 10 2024-03-28T23:28:57.056 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.056 INFO:teuthology.orchestra.run.smithi170.stdout: journal zero on create = true 2024-03-28T23:28:57.056 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.056 INFO:teuthology.orchestra.run.smithi170.stdout: filestore ondisk finisher threads = 3 2024-03-28T23:28:57.056 INFO:teuthology.orchestra.run.smithi170.stdout: filestore apply finisher threads = 3 2024-03-28T23:28:57.057 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.057 INFO:teuthology.orchestra.run.smithi170.stdout: bdev debug aio = true 2024-03-28T23:28:57.057 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug misdirected ops = true 2024-03-28T23:28:57.057 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore block size = 96636764160 2024-03-28T23:28:57.057 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore compression mode = aggressive 2024-03-28T23:28:57.057 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore fsck on mount = True 2024-03-28T23:28:57.057 INFO:teuthology.orchestra.run.smithi170.stdout: debug bluefs = 20 2024-03-28T23:28:57.057 INFO:teuthology.orchestra.run.smithi170.stdout: debug bluestore = 20 2024-03-28T23:28:57.057 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2024-03-28T23:28:57.057 INFO:teuthology.orchestra.run.smithi170.stdout: debug osd = 20 2024-03-28T23:28:57.057 INFO:teuthology.orchestra.run.smithi170.stdout: debug rocksdb = 10 2024-03-28T23:28:57.057 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd backfillfull_ratio = 0.85 2024-03-28T23:28:57.057 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd full ratio = 0.9 2024-03-28T23:28:57.057 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd nearfull ratio = 0.8 2024-03-28T23:28:57.057 INFO:teuthology.orchestra.run.smithi170.stdout: osd failsafe full ratio = 0.95 2024-03-28T23:28:57.058 INFO:teuthology.orchestra.run.smithi170.stdout: osd objectstore = bluestore 2024-03-28T23:28:57.058 INFO:teuthology.orchestra.run.smithi170.stdout: osd op complaint time = 180 2024-03-28T23:28:57.058 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.058 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr] 2024-03-28T23:28:57.058 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2024-03-28T23:28:57.058 INFO:teuthology.orchestra.run.smithi170.stdout: debug mgr = 20 2024-03-28T23:28:57.058 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2024-03-28T23:28:57.058 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2024-03-28T23:28:57.058 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2024-03-28T23:28:57.058 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2024-03-28T23:28:57.058 INFO:teuthology.orchestra.run.smithi170.stdout: mgr/telemetry/nag = false 2024-03-28T23:28:57.058 INFO:teuthology.orchestra.run.smithi170.stdout: debug client = 20 2024-03-28T23:28:57.058 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.058 INFO:teuthology.orchestra.run.smithi170.stdout:[mon] 2024-03-28T23:28:57.059 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2024-03-28T23:28:57.059 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2024-03-28T23:28:57.059 INFO:teuthology.orchestra.run.smithi170.stdout: debug paxos = 20 2024-03-28T23:28:57.059 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2024-03-28T23:28:57.059 INFO:teuthology.orchestra.run.smithi170.stdout: mon data avail warn = 5 2024-03-28T23:28:57.059 INFO:teuthology.orchestra.run.smithi170.stdout: mon mgr mkfs grace = 240 2024-03-28T23:28:57.059 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2024-03-28T23:28:57.059 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd reporter subtree level = osd 2024-03-28T23:28:57.059 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd prime pg temp = true 2024-03-28T23:28:57.059 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2024-03-28T23:28:57.059 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.059 INFO:teuthology.orchestra.run.smithi170.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-28T23:28:57.059 INFO:teuthology.orchestra.run.smithi170.stdout: auth mon ticket ttl = 660 # 11m 2024-03-28T23:28:57.059 INFO:teuthology.orchestra.run.smithi170.stdout: auth service ticket ttl = 240 # 4m 2024-03-28T23:28:57.060 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.060 INFO:teuthology.orchestra.run.smithi170.stdout: # don't complain about insecure global_id in the test suite 2024-03-28T23:28:57.060 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-28T23:28:57.060 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-28T23:28:57.060 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.060 INFO:teuthology.orchestra.run.smithi170.stdout: # 1m isn't quite enough 2024-03-28T23:28:57.060 INFO:teuthology.orchestra.run.smithi170.stdout: mon_down_mkfs_grace = 2m 2024-03-28T23:28:57.060 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.061 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_filestore_osds = false 2024-03-28T23:28:57.061 INFO:teuthology.orchestra.run.smithi170.stdout: mon down mkfs grace = 300 2024-03-28T23:28:57.061 INFO:teuthology.orchestra.run.smithi170.stdout: mon op complaint time = 120 2024-03-28T23:28:57.061 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-28T23:28:57.061 INFO:teuthology.orchestra.run.smithi170.stdout:[client] 2024-03-28T23:28:57.061 INFO:teuthology.orchestra.run.smithi170.stdout: rgw cache enabled = true 2024-03-28T23:28:57.061 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable ops log = true 2024-03-28T23:28:57.061 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable usage log = true 2024-03-28T23:28:57.061 INFO:teuthology.orchestra.run.smithi170.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-28T23:28:57.062 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-28T23:28:57.062 INFO:teuthology.orchestra.run.smithi170.stdout: client mount timeout = 600 2024-03-28T23:28:57.062 INFO:teuthology.orchestra.run.smithi170.stdout: debug client = 20 2024-03-28T23:28:57.062 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2024-03-28T23:28:57.062 INFO:teuthology.orchestra.run.smithi170.stdout: rados mon op timeout = 900 2024-03-28T23:28:57.062 INFO:teuthology.orchestra.run.smithi170.stdout: rados osd op timeout = 900 2024-03-28T23:28:57.062 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.a] 2024-03-28T23:28:57.062 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.b] 2024-03-28T23:28:57.062 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.c] 2024-03-28T23:28:57.062 INFO:teuthology.orchestra.run.smithi170.stdout:[mds.a] 2024-03-28T23:28:57.062 INFO:teuthology.orchestra.run.smithi170.stdout:[mds.c] 2024-03-28T23:28:57.062 INFO:teuthology.orchestra.run.smithi170.stdout:[mds.b] 2024-03-28T23:28:57.062 INFO:teuthology.orchestra.run.smithi170.stdout:[mds.d] 2024-03-28T23:28:57.062 INFO:teuthology.orchestra.run.smithi170.stdout:[mds] 2024-03-28T23:28:57.063 INFO:teuthology.orchestra.run.smithi170.stdout: debug mds = 20 2024-03-28T23:28:57.063 INFO:teuthology.orchestra.run.smithi170.stdout: debug mds balancer = 20 2024-03-28T23:28:57.063 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2024-03-28T23:28:57.063 INFO:teuthology.orchestra.run.smithi170.stdout: mds debug frag = True 2024-03-28T23:28:57.063 INFO:teuthology.orchestra.run.smithi170.stdout: mds debug scatterstat = True 2024-03-28T23:28:57.063 INFO:teuthology.orchestra.run.smithi170.stdout: mds op complaint time = 180 2024-03-28T23:28:57.063 INFO:teuthology.orchestra.run.smithi170.stdout: mds verify scatter = True 2024-03-28T23:28:57.063 INFO:teuthology.orchestra.run.smithi170.stdout: osd op complaint time = 180 2024-03-28T23:28:57.063 INFO:teuthology.orchestra.run.smithi170.stdout: rados mon op timeout = 900 2024-03-28T23:28:57.063 INFO:teuthology.orchestra.run.smithi170.stdout: rados osd op timeout = 900 2024-03-28T23:28:57.065 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-28T23:28:57.065 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-28T23:28:57.162 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-28T23:28:57.162 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:28:57.162 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-28T23:28:57.177 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:28:57.178 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-28T23:28:57.232 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi161.front.sepia.ceph.com 2024-03-28T23:28:57.233 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:28:57.233 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-28T23:28:57.233 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-28T23:28:57.312 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:28:57.312 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-28T23:28:57.366 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi170.front.sepia.ceph.com 2024-03-28T23:28:57.366 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:28:57.367 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-28T23:28:57.367 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-28T23:28:57.406 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:28:57.407 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-28T23:28:57.461 INFO:tasks.ceph:Setting up mon nodes... 2024-03-28T23:28:57.462 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-28T23:28:57.462 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-28T23:28:57.518 INFO:teuthology.orchestra.run.smithi161.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-28T23:28:57.520 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-28T23:28:57.586 INFO:teuthology.orchestra.run.smithi170.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-28T23:28:57.588 INFO:tasks.ceph:Setting up mds nodes... 2024-03-28T23:28:57.589 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-28T23:28:57.644 INFO:teuthology.orchestra.run.smithi161.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-28T23:28:57.646 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-28T23:28:57.673 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-28T23:28:57.770 INFO:teuthology.orchestra.run.smithi161.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-28T23:28:57.772 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-28T23:28:57.799 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-28T23:28:57.856 INFO:teuthology.orchestra.run.smithi170.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-28T23:28:57.858 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-28T23:28:57.885 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-28T23:28:57.980 INFO:teuthology.orchestra.run.smithi170.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-28T23:28:57.983 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-28T23:28:58.009 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-28T23:28:58.009 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-28T23:28:58.052 INFO:teuthology.orchestra.run.smithi161.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-28T23:28:58.068 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-28T23:28:58.110 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-28T23:28:58.125 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-28T23:28:58.125 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi161.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi170.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-28T23:28:58.126 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-28T23:28:58.152 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T23:28:58.152 INFO:tasks.ceph:role: osd.0 2024-03-28T23:28:58.152 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi161.front.sepia.ceph.com 2024-03-28T23:28:58.153 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-28T23:28:58.226 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T23:28:58.227 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T23:28:58.227 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T23:28:58.227 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T23:28:58.227 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T23:28:58.227 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2024-03-28T23:28:58.227 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T23:28:58.227 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T23:28:58.227 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T23:28:58.227 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T23:28:58.738 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2024-03-28T23:28:58.740 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2024-03-28T23:28:58.740 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-28T23:28:58.895 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-28T23:28:58.930 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-28T23:28:58.998 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T23:28:58.998 INFO:tasks.ceph:role: osd.1 2024-03-28T23:28:58.998 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi161.front.sepia.ceph.com 2024-03-28T23:28:58.998 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-28T23:28:59.074 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T23:28:59.074 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T23:28:59.074 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T23:28:59.074 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T23:28:59.074 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T23:28:59.074 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2024-03-28T23:28:59.074 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T23:28:59.074 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T23:28:59.074 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T23:28:59.074 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T23:28:59.524 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2024-03-28T23:28:59.527 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2024-03-28T23:28:59.527 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-28T23:28:59.626 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-28T23:28:59.662 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-28T23:28:59.731 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T23:28:59.731 INFO:tasks.ceph:role: osd.2 2024-03-28T23:28:59.731 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi161.front.sepia.ceph.com 2024-03-28T23:28:59.731 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-28T23:28:59.808 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T23:28:59.808 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T23:28:59.808 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T23:28:59.808 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T23:28:59.809 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T23:28:59.809 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2024-03-28T23:28:59.809 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T23:28:59.809 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T23:28:59.809 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T23:28:59.809 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T23:29:00.297 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2024-03-28T23:29:00.300 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2024-03-28T23:29:00.300 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-28T23:29:00.450 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-28T23:29:00.485 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-28T23:29:00.553 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T23:29:00.553 INFO:tasks.ceph:role: osd.3 2024-03-28T23:29:00.553 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi161.front.sepia.ceph.com 2024-03-28T23:29:00.554 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-28T23:29:00.629 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T23:29:00.629 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T23:29:00.629 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T23:29:00.629 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T23:29:00.629 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T23:29:00.629 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2024-03-28T23:29:00.629 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T23:29:00.629 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T23:29:00.629 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T23:29:00.629 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T23:29:01.130 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2024-03-28T23:29:01.132 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2024-03-28T23:29:01.132 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-28T23:29:01.282 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-28T23:29:01.314 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T23:29:01.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:01.416+0000 7fe9c82c4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-28T23:29:01.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:01.417+0000 7fe9c82c4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-28T23:29:01.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:01.417+0000 7fe9c82c4640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-28T23:29:01.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:01.417+0000 7fe9c82c4640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-28T23:29:01.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:01.417+0000 7fe9c82c4640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-28T23:29:01.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:01.417+0000 7fe9c82c4640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-28T23:29:04.272 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-28T23:29:04.305 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T23:29:04.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:04.398+0000 7fcec481f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-28T23:29:04.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:04.399+0000 7fcec481f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-28T23:29:04.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:04.399+0000 7fcec481f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-28T23:29:04.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:04.399+0000 7fcec481f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-28T23:29:04.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:04.399+0000 7fcec481f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-28T23:29:04.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:04.399+0000 7fcec481f640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-28T23:29:07.250 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-28T23:29:07.280 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T23:29:07.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:07.369+0000 7f1877414640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-28T23:29:07.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:07.370+0000 7f1877414640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-28T23:29:07.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:07.370+0000 7f1877414640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-28T23:29:07.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:07.370+0000 7f1877414640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-28T23:29:07.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:07.370+0000 7f1877414640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-28T23:29:07.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:07.370+0000 7f1877414640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-28T23:29:10.222 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-28T23:29:10.254 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T23:29:10.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:10.347+0000 7f1a3d40c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-28T23:29:10.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:10.347+0000 7f1a3d40c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-28T23:29:10.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:10.347+0000 7f1a3d40c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-28T23:29:10.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:10.347+0000 7f1a3d40c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-28T23:29:10.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:10.347+0000 7f1a3d40c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-28T23:29:10.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:10.348+0000 7f1a3d40c640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-28T23:29:13.194 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-28T23:29:13.225 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-28T23:29:13.257 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T23:29:13.257 INFO:tasks.ceph:role: osd.4 2024-03-28T23:29:13.257 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi170.front.sepia.ceph.com 2024-03-28T23:29:13.258 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-28T23:29:13.328 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T23:29:13.328 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T23:29:13.328 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T23:29:13.328 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T23:29:13.328 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T23:29:13.328 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2024-03-28T23:29:13.328 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T23:29:13.328 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T23:29:13.328 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T23:29:13.328 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T23:29:13.361 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2024-03-28T23:29:13.362 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2024-03-28T23:29:13.363 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-28T23:29:13.436 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-28T23:29:13.507 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-28T23:29:13.575 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T23:29:13.575 INFO:tasks.ceph:role: osd.5 2024-03-28T23:29:13.575 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi170.front.sepia.ceph.com 2024-03-28T23:29:13.576 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-28T23:29:13.642 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T23:29:13.642 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T23:29:13.642 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T23:29:13.642 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T23:29:13.642 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T23:29:13.642 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2024-03-28T23:29:13.642 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T23:29:13.642 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T23:29:13.643 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T23:29:13.643 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T23:29:13.675 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2024-03-28T23:29:13.676 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2024-03-28T23:29:13.677 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-28T23:29:13.747 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-28T23:29:13.818 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-28T23:29:13.886 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T23:29:13.886 INFO:tasks.ceph:role: osd.6 2024-03-28T23:29:13.886 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi170.front.sepia.ceph.com 2024-03-28T23:29:13.886 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-28T23:29:13.961 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T23:29:13.961 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T23:29:13.961 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T23:29:13.961 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T23:29:13.961 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T23:29:13.962 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2024-03-28T23:29:13.962 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T23:29:13.962 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T23:29:13.962 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T23:29:13.962 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T23:29:13.992 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2024-03-28T23:29:13.994 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2024-03-28T23:29:13.994 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-28T23:29:14.065 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-28T23:29:14.136 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-28T23:29:14.205 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T23:29:14.205 INFO:tasks.ceph:role: osd.7 2024-03-28T23:29:14.205 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi170.front.sepia.ceph.com 2024-03-28T23:29:14.205 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-28T23:29:14.271 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T23:29:14.271 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T23:29:14.271 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T23:29:14.271 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T23:29:14.271 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T23:29:14.271 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2024-03-28T23:29:14.271 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T23:29:14.271 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T23:29:14.271 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T23:29:14.271 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T23:29:14.303 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2024-03-28T23:29:14.305 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2024-03-28T23:29:14.305 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-28T23:29:14.375 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-28T23:29:14.445 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T23:29:14.543 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:14.541+0000 7efc8180e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-28T23:29:14.543 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:14.541+0000 7efc8180e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-28T23:29:14.543 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:14.541+0000 7efc8180e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-28T23:29:14.543 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:14.541+0000 7efc8180e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-28T23:29:14.543 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:14.541+0000 7efc8180e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-28T23:29:14.543 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:14.542+0000 7efc8180e640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-28T23:29:17.384 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-28T23:29:17.415 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T23:29:17.508 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:17.506+0000 7f844d00f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-28T23:29:17.508 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:17.506+0000 7f844d00f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-28T23:29:17.508 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:17.506+0000 7f844d00f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-28T23:29:17.508 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:17.506+0000 7f844d00f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-28T23:29:17.508 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:17.506+0000 7f844d00f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-28T23:29:17.508 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:17.507+0000 7f844d00f640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-28T23:29:20.351 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-28T23:29:20.383 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T23:29:20.475 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:20.474+0000 7f2427048640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-28T23:29:20.475 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:20.474+0000 7f2427048640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-28T23:29:20.475 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:20.474+0000 7f2427048640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-28T23:29:20.475 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:20.474+0000 7f2427048640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-28T23:29:20.476 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:20.474+0000 7f2427048640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-28T23:29:20.476 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:20.474+0000 7f2427048640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-28T23:29:23.328 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-28T23:29:23.359 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T23:29:23.451 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:23.449+0000 7fa26bec4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-28T23:29:23.451 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:23.449+0000 7fa26bec4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-28T23:29:23.451 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:23.449+0000 7fa26bec4640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-28T23:29:23.451 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:23.450+0000 7fa26bec4640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-28T23:29:23.451 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:23.450+0000 7fa26bec4640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-28T23:29:23.451 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:23.450+0000 7fa26bec4640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-28T23:29:25.052 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:29:25.055 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:29:26.305 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-28T23:29:26.336 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-28T23:29:26.336 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:29:26.336 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-28T23:29:26.369 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:29:26.369 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-28T23:29:26.439 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:29:26.440 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-28T23:29:26.509 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:29:26.509 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-28T23:29:26.579 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:29:26.579 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-28T23:29:26.647 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:29:26.647 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-28T23:29:26.715 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:29:26.715 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-28T23:29:26.786 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:29:26.786 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-28T23:29:26.818 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:29:26.818 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-28T23:29:26.889 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:29:26.889 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-28T23:29:26.957 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:29:26.957 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-28T23:29:27.027 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:29:27.027 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-28T23:29:27.095 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:29:27.095 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-28T23:29:27.162 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:29:27.162 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-28T23:29:27.230 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:29:27.231 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-28T23:29:27.247 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:29:27.247 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-28T23:29:27.285 INFO:tasks.ceph:Adding keys to all mons... 2024-03-28T23:29:27.286 DEBUG:teuthology.orchestra.run.smithi161:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-28T23:29:27.289 DEBUG:teuthology.orchestra.run.smithi170:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-28T23:29:27.330 INFO:teuthology.orchestra.run.smithi161.stdout:[mgr.y] 2024-03-28T23:29:27.331 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQA5/QVm0hLlHhAAh8cewPtZ/TbKrU488kxrBg== 2024-03-28T23:29:27.331 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.a] 2024-03-28T23:29:27.331 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQA5/QVmggtoJhAASLXbUICLBhjRr19aUAuZHw== 2024-03-28T23:29:27.331 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.c] 2024-03-28T23:29:27.331 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQA5/QVmjf/qLRAAosLbpT8aCWvrKAQnj8Uh3g== 2024-03-28T23:29:27.331 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.0] 2024-03-28T23:29:27.331 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQA9/QVmAFPmGBAAS0SkRRwsupm6jm1sqFXJ6g== 2024-03-28T23:29:27.331 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.1] 2024-03-28T23:29:27.332 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBA/QVmNBTVFxAA1htU9ilznfLUDqKmJaQwzg== 2024-03-28T23:29:27.332 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.2] 2024-03-28T23:29:27.332 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBD/QVmg+4YFhAA9TvfFVKT3nQT7/QUOf3Spw== 2024-03-28T23:29:27.332 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.3] 2024-03-28T23:29:27.332 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBG/QVmgiHDFBAAQPtbZT6a1ZGmkisBX0NMgA== 2024-03-28T23:29:27.332 INFO:teuthology.orchestra.run.smithi161.stdout:[mgr.x] 2024-03-28T23:29:27.332 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQA5/QVm7TD3IhAACByGRGpvJWDpF22NuovHWw== 2024-03-28T23:29:27.332 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.b] 2024-03-28T23:29:27.332 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQA5/QVmQbEOMxAA74al6+dMzcN/ngwNZK+aHA== 2024-03-28T23:29:27.332 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.d] 2024-03-28T23:29:27.332 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQA5/QVmpiZ1OhAAjYrFAaDosDS2Y+BvU4c6JQ== 2024-03-28T23:29:27.332 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.4] 2024-03-28T23:29:27.332 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBK/QVmahhZIBAAJfotg/dusbB3usreZlRuPA== 2024-03-28T23:29:27.333 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.5] 2024-03-28T23:29:27.333 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBN/QVmzjlDHhAAli7pLyIo+MVdwSBNJYVg5g== 2024-03-28T23:29:27.333 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.6] 2024-03-28T23:29:27.333 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBQ/QVmtCFSHBAASVfV/5mY/3JNiI0YMT7Vww== 2024-03-28T23:29:27.333 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.7] 2024-03-28T23:29:27.333 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBT/QVm2J7eGhAAaIeZxmLwQBBE1glCxjKbhA== 2024-03-28T23:29:27.333 INFO:teuthology.orchestra.run.smithi161.stdout:[client.0] 2024-03-28T23:29:27.333 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQA6/QVmc1krAxAAEKsn1Tuuv9OlFZpB4PFhDg== 2024-03-28T23:29:27.333 INFO:teuthology.orchestra.run.smithi161.stdout:[client.1] 2024-03-28T23:29:27.333 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQA6/QVm1B2cBhAAmHDeeFwpFlV/y3NIGWUXHQ== 2024-03-28T23:29:27.352 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr.y] 2024-03-28T23:29:27.352 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQA5/QVm0hLlHhAAh8cewPtZ/TbKrU488kxrBg== 2024-03-28T23:29:27.352 INFO:teuthology.orchestra.run.smithi170.stdout:[mds.a] 2024-03-28T23:29:27.352 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQA5/QVmggtoJhAASLXbUICLBhjRr19aUAuZHw== 2024-03-28T23:29:27.352 INFO:teuthology.orchestra.run.smithi170.stdout:[mds.c] 2024-03-28T23:29:27.352 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQA5/QVmjf/qLRAAosLbpT8aCWvrKAQnj8Uh3g== 2024-03-28T23:29:27.352 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.0] 2024-03-28T23:29:27.352 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQA9/QVmAFPmGBAAS0SkRRwsupm6jm1sqFXJ6g== 2024-03-28T23:29:27.353 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.1] 2024-03-28T23:29:27.353 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBA/QVmNBTVFxAA1htU9ilznfLUDqKmJaQwzg== 2024-03-28T23:29:27.353 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.2] 2024-03-28T23:29:27.353 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBD/QVmg+4YFhAA9TvfFVKT3nQT7/QUOf3Spw== 2024-03-28T23:29:27.353 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.3] 2024-03-28T23:29:27.353 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBG/QVmgiHDFBAAQPtbZT6a1ZGmkisBX0NMgA== 2024-03-28T23:29:27.353 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr.x] 2024-03-28T23:29:27.353 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQA5/QVm7TD3IhAACByGRGpvJWDpF22NuovHWw== 2024-03-28T23:29:27.354 INFO:teuthology.orchestra.run.smithi170.stdout:[mds.b] 2024-03-28T23:29:27.354 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQA5/QVmQbEOMxAA74al6+dMzcN/ngwNZK+aHA== 2024-03-28T23:29:27.354 INFO:teuthology.orchestra.run.smithi170.stdout:[mds.d] 2024-03-28T23:29:27.354 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQA5/QVmpiZ1OhAAjYrFAaDosDS2Y+BvU4c6JQ== 2024-03-28T23:29:27.354 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.4] 2024-03-28T23:29:27.354 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBK/QVmahhZIBAAJfotg/dusbB3usreZlRuPA== 2024-03-28T23:29:27.354 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.5] 2024-03-28T23:29:27.354 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBN/QVmzjlDHhAAli7pLyIo+MVdwSBNJYVg5g== 2024-03-28T23:29:27.354 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.6] 2024-03-28T23:29:27.355 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBQ/QVmtCFSHBAASVfV/5mY/3JNiI0YMT7Vww== 2024-03-28T23:29:27.355 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.7] 2024-03-28T23:29:27.355 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBT/QVm2J7eGhAAaIeZxmLwQBBE1glCxjKbhA== 2024-03-28T23:29:27.355 INFO:teuthology.orchestra.run.smithi170.stdout:[client.0] 2024-03-28T23:29:27.355 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQA6/QVmc1krAxAAEKsn1Tuuv9OlFZpB4PFhDg== 2024-03-28T23:29:27.355 INFO:teuthology.orchestra.run.smithi170.stdout:[client.1] 2024-03-28T23:29:27.355 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQA6/QVm1B2cBhAAmHDeeFwpFlV/y3NIGWUXHQ== 2024-03-28T23:29:27.356 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-28T23:29:27.378 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-28T23:29:27.448 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T23:29:27.450 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T23:29:27.505 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T23:29:27.508 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T23:29:27.562 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T23:29:27.565 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T23:29:27.617 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T23:29:27.619 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T23:29:27.671 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T23:29:27.674 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T23:29:27.725 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T23:29:27.727 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T23:29:27.778 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-28T23:29:27.781 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-28T23:29:27.832 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T23:29:27.835 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T23:29:27.889 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T23:29:27.892 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T23:29:27.946 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T23:29:27.949 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T23:29:28.000 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T23:29:28.002 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T23:29:28.055 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T23:29:28.057 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T23:29:28.109 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T23:29:28.111 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T23:29:28.163 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-28T23:29:28.165 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-28T23:29:28.221 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-28T23:29:28.223 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-28T23:29:28.278 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-28T23:29:28.278 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-28T23:29:28.305 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-28T23:29:28.402 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-28T23:29:28.429 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-28T23:29:28.457 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-28T23:29:28.553 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-28T23:29:28.580 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-28T23:29:28.647 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-28T23:29:28.744 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-28T23:29:28.770 DEBUG:teuthology.orchestra.run.smithi161:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-28T23:29:28.773 DEBUG:teuthology.orchestra.run.smithi170:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-28T23:29:28.826 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-28T23:29:28.830 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-28T23:29:28.830 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-28T23:29:28.833 INFO:tasks.ceph.mon.a:Started 2024-03-28T23:29:28.833 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-28T23:29:28.833 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-28T23:29:28.868 INFO:tasks.ceph.mon.b:Started 2024-03-28T23:29:28.868 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-28T23:29:28.868 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-28T23:29:28.870 INFO:tasks.ceph.mon.c:Started 2024-03-28T23:29:28.870 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-28T23:29:28.870 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-28T23:29:28.870 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-28T23:29:28.872 INFO:tasks.ceph.mgr.y:Started 2024-03-28T23:29:28.873 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-28T23:29:28.873 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-28T23:29:28.874 INFO:tasks.ceph.mgr.x:Started 2024-03-28T23:29:28.875 INFO:tasks.ceph:Setting crush tunables to default 2024-03-28T23:29:28.875 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd crush tunables default 2024-03-28T23:29:28.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:28.976+0000 7fe49bd0f640 1 Processor -- start 2024-03-28T23:29:28.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:28.976+0000 7fe49bd0f640 1 -- start start 2024-03-28T23:29:28.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:28.977+0000 7fe49bd0f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe4940c77e0 0x7fe4940c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:28.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:28.977+0000 7fe49bd0f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe494159120 0x7fe49415fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:28.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:28.977+0000 7fe49bd0f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4940c2480 con 0x7fe494072a50 2024-03-28T23:29:28.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:28.977+0000 7fe49bd0f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fe4940c2230 con 0x7fe4940c77e0 2024-03-28T23:29:28.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:28.977+0000 7fe49bd0f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe4940725c0 con 0x7fe494159120 2024-03-28T23:29:28.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:28.977+0000 7fe4992aa640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe494072a50 0x7fe494158a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48802/0 (socket says 172.21.15.161:48802) 2024-03-28T23:29:28.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:28.977+0000 7fe4992aa640 1 -- 172.21.15.161:0/3250263202 learned_addr learned my addr 172.21.15.161:0/3250263202 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:28.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:28.977+0000 7fe49a2ac640 1 --2- 172.21.15.161:0/3250263202 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe494159120 0x7fe49415fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:28.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:28.977+0000 7fe499aab640 1 --2- 172.21.15.161:0/3250263202 >> v2:172.21.15.170:3300/0 conn(0x7fe4940c77e0 0x7fe4940c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:28.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:28.978+0000 7fe49a2ac640 1 -- 172.21.15.161:0/3250263202 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe494159120 msgr2=0x7fe49415fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-28T23:29:28.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:28.978+0000 7fe49a2ac640 1 -- 172.21.15.161:0/3250263202 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe494159120 msgr2=0x7fe49415fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T23:29:28.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:28.978+0000 7fe49a2ac640 1 --2- 172.21.15.161:0/3250263202 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe494159120 0x7fe49415fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T23:29:28.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:28.978+0000 7fe49a2ac640 1 --2- 172.21.15.161:0/3250263202 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe494159120 0x7fe49415fd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T23:29:28.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:28.978+0000 7fe499aab640 1 -- 172.21.15.161:0/3250263202 >> v2:172.21.15.170:3300/0 conn(0x7fe4940c77e0 msgr2=0x7fe4940c7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-28T23:29:28.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:28.978+0000 7fe499aab640 1 -- 172.21.15.161:0/3250263202 >> v2:172.21.15.170:3300/0 conn(0x7fe4940c77e0 msgr2=0x7fe4940c7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T23:29:28.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:28.978+0000 7fe499aab640 1 --2- 172.21.15.161:0/3250263202 >> v2:172.21.15.170:3300/0 conn(0x7fe4940c77e0 0x7fe4940c7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T23:29:28.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:28.978+0000 7fe499aab640 1 --2- 172.21.15.161:0/3250263202 >> v2:172.21.15.170:3300/0 conn(0x7fe4940c77e0 0x7fe4940c7bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T23:29:29.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.137+0000 7fe498aa9640 1 -- 172.21.15.161:0/3250263202 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1148674021 0 0) 0x7fe4940c2480 con 0x7fe494072a50 2024-03-28T23:29:29.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.137+0000 7fe498aa9640 1 -- 172.21.15.161:0/3250263202 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe480003520 con 0x7fe494072a50 2024-03-28T23:29:29.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.137+0000 7fe498aa9640 1 -- 172.21.15.161:0/3250263202 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3396491617 0 0) 0x7fe480003520 con 0x7fe494072a50 2024-03-28T23:29:29.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.137+0000 7fe498aa9640 1 -- 172.21.15.161:0/3250263202 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe494159120 msgr2=0x7fe49415fd40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T23:29:29.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.137+0000 7fe498aa9640 1 --2- 172.21.15.161:0/3250263202 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe494159120 0x7fe49415fd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:29.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.137+0000 7fe498aa9640 1 -- 172.21.15.161:0/3250263202 >> v2:172.21.15.170:3300/0 conn(0x7fe4940c77e0 msgr2=0x7fe4940c7bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T23:29:29.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.137+0000 7fe498aa9640 1 --2- 172.21.15.161:0/3250263202 >> v2:172.21.15.170:3300/0 conn(0x7fe4940c77e0 0x7fe4940c7bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:29.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.137+0000 7fe498aa9640 1 -- 172.21.15.161:0/3250263202 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4940c2760 con 0x7fe494072a50 2024-03-28T23:29:29.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.137+0000 7fe498aa9640 1 -- 172.21.15.161:0/3250263202 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fe484001670 con 0x7fe494072a50 2024-03-28T23:29:29.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.137+0000 7fe498aa9640 1 -- 172.21.15.161:0/3250263202 >> v1:172.21.15.161:6789/0 conn(0x7fe494072a50 legacy=0x7fe494158a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:29.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.137+0000 7fe498aa9640 1 --2- 172.21.15.161:0/3250263202 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe480003a90 0x7fe480003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:29.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.137+0000 7fe498aa9640 1 --2- 172.21.15.161:0/3250263202 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe480004800 0x7fe480004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:29.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.137+0000 7fe498aa9640 1 --2- 172.21.15.161:0/3250263202 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe480005180 0x7fe48000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:29.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.137+0000 7fe498aa9640 1 -- 172.21.15.161:0/3250263202 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe4940c2230 con 0x7fe480003a90 2024-03-28T23:29:29.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.137+0000 7fe498aa9640 1 -- 172.21.15.161:0/3250263202 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe4940725c0 con 0x7fe480005180 2024-03-28T23:29:29.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.137+0000 7fe498aa9640 1 -- 172.21.15.161:0/3250263202 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe48000bf30 con 0x7fe480004800 2024-03-28T23:29:29.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.138+0000 7fe499aab640 1 --2- 172.21.15.161:0/3250263202 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe480003a90 0x7fe480003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:29.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.138+0000 7fe4992aa640 1 --2- 172.21.15.161:0/3250263202 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe480004800 0x7fe480004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:29.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.138+0000 7fe49a2ac640 1 --2- 172.21.15.161:0/3250263202 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe480005180 0x7fe48000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:29.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.138+0000 7fe499aab640 1 -- 172.21.15.161:0/3250263202 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe480004800 msgr2=0x7fe480004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:29.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.138+0000 7fe499aab640 1 --2- 172.21.15.161:0/3250263202 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe480004800 0x7fe480004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:29.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.138+0000 7fe499aab640 1 -- 172.21.15.161:0/3250263202 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe480005180 msgr2=0x7fe48000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:29.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.138+0000 7fe499aab640 1 --2- 172.21.15.161:0/3250263202 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe480005180 0x7fe48000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:29.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.138+0000 7fe499aab640 1 -- 172.21.15.161:0/3250263202 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe480003520 con 0x7fe480003a90 2024-03-28T23:29:29.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.138+0000 7fe49a2ac640 1 --2- 172.21.15.161:0/3250263202 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe480005180 0x7fe48000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:29:29.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.139+0000 7fe499aab640 1 --2- 172.21.15.161:0/3250263202 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe480003a90 0x7fe480003eb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe490009fd0 tx=0x7fe480005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3e78abdbcf0bec6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:29.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.139+0000 7fe498aa9640 1 -- 172.21.15.161:0/3250263202 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe490039070 con 0x7fe480003a90 2024-03-28T23:29:29.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.139+0000 7fe498aa9640 1 -- 172.21.15.161:0/3250263202 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe49002cca0 con 0x7fe480003a90 2024-03-28T23:29:29.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.139+0000 7fe498aa9640 1 -- 172.21.15.161:0/3250263202 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe490009bd0 con 0x7fe480003a90 2024-03-28T23:29:29.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.140+0000 7fe49bd0f640 1 -- 172.21.15.161:0/3250263202 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe480003a90 msgr2=0x7fe480003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:29.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.140+0000 7fe49bd0f640 1 --2- 172.21.15.161:0/3250263202 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe480003a90 0x7fe480003eb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe490009fd0 tx=0x7fe480005830 comp rx=0 tx=0).stop 2024-03-28T23:29:29.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.140+0000 7fe49a2ac640 1 -- 172.21.15.161:0/3250263202 reap_dead start 2024-03-28T23:29:29.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.140+0000 7fe49bd0f640 1 -- 172.21.15.161:0/3250263202 shutdown_connections 2024-03-28T23:29:29.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.140+0000 7fe49bd0f640 1 -- 172.21.15.161:0/3250263202 >> 172.21.15.161:0/3250263202 conn(0x7fe4940a97c0 msgr2=0x7fe4940a9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:29.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.140+0000 7fe49bd0f640 1 -- 172.21.15.161:0/3250263202 shutdown_connections 2024-03-28T23:29:29.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.140+0000 7fe49bd0f640 1 -- 172.21.15.161:0/3250263202 wait complete. 2024-03-28T23:29:29.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.141+0000 7fe49bd0f640 1 Processor -- start 2024-03-28T23:29:29.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.141+0000 7fe49bd0f640 1 -- start start 2024-03-28T23:29:29.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.142+0000 7fe49bd0f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe480003a90 0x7fe4940a9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:29.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.142+0000 7fe49bd0f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4940aa6a0 0x7fe4941eaab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:29.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.142+0000 7fe49bd0f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe49414e160 0x7fe4941ee790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:29.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.142+0000 7fe49bd0f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe4940c36e0 con 0x7fe480003a90 2024-03-28T23:29:29.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.142+0000 7fe49bd0f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe494071f20 con 0x7fe4940aa6a0 2024-03-28T23:29:29.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.142+0000 7fe49bd0f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe494156390 con 0x7fe49414e160 2024-03-28T23:29:29.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.142+0000 7fe499aab640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe480003a90 0x7fe4940a9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:29.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.142+0000 7fe4992aa640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4940aa6a0 0x7fe4941eaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:29.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.142+0000 7fe499aab640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe480003a90 0x7fe4940a9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:44376/0 (socket says 172.21.15.161:44376) 2024-03-28T23:29:29.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.142+0000 7fe499aab640 1 -- 172.21.15.161:0/845399069 learned_addr learned my addr 172.21.15.161:0/845399069 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:29.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.142+0000 7fe49a2ac640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe49414e160 0x7fe4941ee790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:29.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.142+0000 7fe499aab640 1 -- 172.21.15.161:0/845399069 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe49414e160 msgr2=0x7fe4941ee790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:29.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.142+0000 7fe499aab640 1 --2- 172.21.15.161:0/845399069 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe49414e160 0x7fe4941ee790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:29.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.142+0000 7fe499aab640 1 -- 172.21.15.161:0/845399069 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4940aa6a0 msgr2=0x7fe4941eaab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:29.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.142+0000 7fe499aab640 1 --2- 172.21.15.161:0/845399069 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4940aa6a0 0x7fe4941eaab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:29.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.142+0000 7fe499aab640 1 -- 172.21.15.161:0/845399069 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4940ab070 con 0x7fe480003a90 2024-03-28T23:29:29.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.142+0000 7fe499aab640 1 --2- 172.21.15.161:0/845399069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe480003a90 0x7fe4940a9050 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe490009fd0 tx=0x7fe490004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:29.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.143+0000 7fe48a7fc640 1 -- 172.21.15.161:0/845399069 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe490044020 con 0x7fe480003a90 2024-03-28T23:29:29.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.143+0000 7fe48a7fc640 1 -- 172.21.15.161:0/845399069 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe49002ce50 con 0x7fe480003a90 2024-03-28T23:29:29.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.143+0000 7fe49bd0f640 1 -- 172.21.15.161:0/845399069 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4940ab350 con 0x7fe480003a90 2024-03-28T23:29:29.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.143+0000 7fe48a7fc640 1 -- 172.21.15.161:0/845399069 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe490039040 con 0x7fe480003a90 2024-03-28T23:29:29.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.143+0000 7fe49bd0f640 1 -- 172.21.15.161:0/845399069 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4941ee270 con 0x7fe480003a90 2024-03-28T23:29:29.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.143+0000 7fe48a7fc640 1 -- 172.21.15.161:0/845399069 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe490047020 con 0x7fe480003a90 2024-03-28T23:29:29.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.143+0000 7fe48a7fc640 1 -- 172.21.15.161:0/845399069 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe490040830 con 0x7fe480003a90 2024-03-28T23:29:29.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.144+0000 7fe49bd0f640 1 -- 172.21.15.161:0/845399069 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe49404f4b0 con 0x7fe480003a90 2024-03-28T23:29:29.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.146+0000 7fe48a7fc640 1 -- 172.21.15.161:0/845399069 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe49004e030 con 0x7fe480003a90 2024-03-28T23:29:29.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.211+0000 7fe49bd0f640 1 -- 172.21.15.161:0/845399069 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fe49414f500 con 0x7fe480003a90 2024-03-28T23:29:29.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.213+0000 7fe48a7fc640 1 -- 172.21.15.161:0/845399069 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fe490009bd0 con 0x7fe480003a90 2024-03-28T23:29:29.213 INFO:teuthology.orchestra.run.smithi161.stderr:adjusted tunables profile to default 2024-03-28T23:29:29.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.213+0000 7fe49bd0f640 1 -- 172.21.15.161:0/845399069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe480003a90 msgr2=0x7fe4940a9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:29.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.213+0000 7fe49bd0f640 1 --2- 172.21.15.161:0/845399069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe480003a90 0x7fe4940a9050 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe490009fd0 tx=0x7fe490004ab0 comp rx=0 tx=0).stop 2024-03-28T23:29:29.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.213+0000 7fe49bd0f640 1 -- 172.21.15.161:0/845399069 shutdown_connections 2024-03-28T23:29:29.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.213+0000 7fe49bd0f640 1 --2- 172.21.15.161:0/845399069 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe49414e160 0x7fe4941ee790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:29.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.213+0000 7fe49bd0f640 1 --2- 172.21.15.161:0/845399069 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4940aa6a0 0x7fe4941eaab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:29.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.213+0000 7fe49bd0f640 1 --2- 172.21.15.161:0/845399069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe480003a90 0x7fe4940a9050 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:29.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.213+0000 7fe49bd0f640 1 -- 172.21.15.161:0/845399069 >> 172.21.15.161:0/845399069 conn(0x7fe4940a97c0 msgr2=0x7fe4940c5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:29.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.214+0000 7fe49bd0f640 1 -- 172.21.15.161:0/845399069 shutdown_connections 2024-03-28T23:29:29.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:29.214+0000 7fe49bd0f640 1 -- 172.21.15.161:0/845399069 wait complete. 2024-03-28T23:29:29.225 INFO:tasks.ceph:check_enable_crimson: False 2024-03-28T23:29:29.225 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-28T23:29:29.225 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:29:29.225 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-28T23:29:29.252 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:29:29.252 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-28T23:29:29.320 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:29:29.320 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-28T23:29:29.392 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:29:29.530 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-28T23:29:29.557 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:29:29.557 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-28T23:29:29.585 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:29:29.585 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-28T23:29:29.590 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:29.589+0000 7f772ca2c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T23:29:29.616 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:29:29.617 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-28T23:29:29.622 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:29.622+0000 7f637f216040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T23:29:29.647 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:29.646+0000 7f772ca2c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T23:29:29.674 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:29:29.674 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-28T23:29:29.676 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:29.676+0000 7f637f216040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T23:29:29.741 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 7b59decd-6f16-4eea-a2d3-1e0618870d6e 0 2024-03-28T23:29:29.883 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.882+0000 7f268a240640 1 Processor -- start 2024-03-28T23:29:29.884 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.883+0000 7f268a240640 1 -- start start 2024-03-28T23:29:29.885 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.883+0000 7f268a240640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2684152bc0 0x7f26840c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:29.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.883+0000 7f268a240640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f26841589b0 0x7f2684158e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:29.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.883+0000 7f268a240640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26840c6f60 con 0x7f26840c8a80 2024-03-28T23:29:29.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.883+0000 7f268a240640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f26840c6120 con 0x7f26841589b0 2024-03-28T23:29:29.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.883+0000 7f268a240640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f26840c6400 con 0x7f2684152bc0 2024-03-28T23:29:29.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.883+0000 7f26837fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2684152bc0 0x7f26840c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:29.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.883+0000 7f26837fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2684152bc0 0x7f26840c0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:50422/0 (socket says 172.21.15.170:50422) 2024-03-28T23:29:29.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.883+0000 7f26837fe640 1 -- 172.21.15.170:0/553608194 learned_addr learned my addr 172.21.15.170:0/553608194 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T23:29:29.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.884+0000 7f2683fff640 1 --2- 172.21.15.170:0/553608194 >> v2:172.21.15.161:3300/0 conn(0x7f26841589b0 0x7f2684158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:29.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.884+0000 7f26837fe640 1 -- 172.21.15.170:0/553608194 >> v2:172.21.15.161:3300/0 conn(0x7f26841589b0 msgr2=0x7f2684158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:29.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.884+0000 7f26837fe640 1 --2- 172.21.15.170:0/553608194 >> v2:172.21.15.161:3300/0 conn(0x7f26841589b0 0x7f2684158e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:29.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.884+0000 7f26837fe640 1 -- 172.21.15.170:0/553608194 >> v1:172.21.15.161:6789/0 conn(0x7f26840c8a80 legacy=0x7f26840c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:29.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.884+0000 7f26837fe640 1 -- 172.21.15.170:0/553608194 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26840c7240 con 0x7f2684152bc0 2024-03-28T23:29:29.887 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.884+0000 7f2683fff640 1 --2- 172.21.15.170:0/553608194 >> v2:172.21.15.161:3300/0 conn(0x7f26841589b0 0x7f2684158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:29:29.887 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.884+0000 7f26837fe640 1 --2- 172.21.15.170:0/553608194 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2684152bc0 0x7f26840c0330 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2678009870 tx=0x7f267802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29832b519f15e7f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:29.887 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.885+0000 7f26827fc640 1 -- 172.21.15.170:0/553608194 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2678034070 con 0x7f2684152bc0 2024-03-28T23:29:29.887 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.885+0000 7f26827fc640 1 -- 172.21.15.170:0/553608194 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2678030840 con 0x7f2684152bc0 2024-03-28T23:29:29.887 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.885+0000 7f26827fc640 1 -- 172.21.15.170:0/553608194 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2678030c10 con 0x7f2684152bc0 2024-03-28T23:29:29.891 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.885+0000 7f268a240640 1 -- 172.21.15.170:0/553608194 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2684152bc0 msgr2=0x7f26840c0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:29.891 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.885+0000 7f268a240640 1 --2- 172.21.15.170:0/553608194 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2684152bc0 0x7f26840c0330 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2678009870 tx=0x7f267802b980 comp rx=0 tx=0).stop 2024-03-28T23:29:29.891 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.885+0000 7f268a240640 1 -- 172.21.15.170:0/553608194 shutdown_connections 2024-03-28T23:29:29.891 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.885+0000 7f268a240640 1 --2- 172.21.15.170:0/553608194 >> v2:172.21.15.161:3300/0 conn(0x7f26841589b0 0x7f2684158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:29.891 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.885+0000 7f268a240640 1 --2- 172.21.15.170:0/553608194 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2684152bc0 0x7f26840c0330 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:29.891 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.885+0000 7f268a240640 1 -- 172.21.15.170:0/553608194 >> 172.21.15.170:0/553608194 conn(0x7f26840caa80 msgr2=0x7f26840cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:29.891 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.885+0000 7f268a240640 1 -- 172.21.15.170:0/553608194 shutdown_connections 2024-03-28T23:29:29.891 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.885+0000 7f268a240640 1 -- 172.21.15.170:0/553608194 wait complete. 2024-03-28T23:29:29.892 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.886+0000 7f268a240640 1 Processor -- start 2024-03-28T23:29:29.892 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.887+0000 7f268a240640 1 -- start start 2024-03-28T23:29:29.892 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.887+0000 7f268a240640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26840c8a80 0x7f268406c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:29.892 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.887+0000 7f268a240640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2684152bc0 0x7f2684153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:29.892 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.887+0000 7f268a240640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26841589b0 0x7f2684153880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:29.892 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.887+0000 7f268a240640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2684152fa0 con 0x7f26840c8a80 2024-03-28T23:29:29.892 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.887+0000 7f268a240640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f26840c95f0 con 0x7f2684152bc0 2024-03-28T23:29:29.892 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.887+0000 7f268a240640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f268406b840 con 0x7f26841589b0 2024-03-28T23:29:29.892 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.887+0000 7f2683fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26841589b0 0x7f2684153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:29.893 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.887+0000 7f2682ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2684152bc0 0x7f2684153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:29.893 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.887+0000 7f2683fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26841589b0 0x7f2684153880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:50428/0 (socket says 172.21.15.170:50428) 2024-03-28T23:29:29.893 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.887+0000 7f2683fff640 1 -- 172.21.15.170:0/2140466878 learned_addr learned my addr 172.21.15.170:0/2140466878 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T23:29:29.893 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.887+0000 7f26837fe640 1 --2- 172.21.15.170:0/2140466878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26840c8a80 0x7f268406c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:29.893 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.888+0000 7f2683fff640 1 -- 172.21.15.170:0/2140466878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2684152bc0 msgr2=0x7f2684153340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:29.893 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.888+0000 7f2683fff640 1 --2- 172.21.15.170:0/2140466878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2684152bc0 0x7f2684153340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:29.893 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.888+0000 7f2683fff640 1 -- 172.21.15.170:0/2140466878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26840c8a80 msgr2=0x7f268406c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:29.893 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.888+0000 7f2683fff640 1 --2- 172.21.15.170:0/2140466878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26840c8a80 0x7f268406c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:29.893 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.888+0000 7f2683fff640 1 -- 172.21.15.170:0/2140466878 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f268406ced0 con 0x7f26841589b0 2024-03-28T23:29:29.893 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.888+0000 7f2682ffd640 1 --2- 172.21.15.170:0/2140466878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2684152bc0 0x7f2684153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:29:29.893 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.888+0000 7f2683fff640 1 --2- 172.21.15.170:0/2140466878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26841589b0 0x7f2684153880 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f267c007e20 tx=0x7f267c00d4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:29.893 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.888+0000 7f266bfff640 1 -- 172.21.15.170:0/2140466878 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c01a070 con 0x7f26841589b0 2024-03-28T23:29:29.893 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.888+0000 7f266bfff640 1 -- 172.21.15.170:0/2140466878 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f267c003b90 con 0x7f26841589b0 2024-03-28T23:29:29.893 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.888+0000 7f268a240640 1 -- 172.21.15.170:0/2140466878 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f268406d1b0 con 0x7f26841589b0 2024-03-28T23:29:29.894 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.888+0000 7f266bfff640 1 -- 172.21.15.170:0/2140466878 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c00de80 con 0x7f26841589b0 2024-03-28T23:29:29.894 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.888+0000 7f268a240640 1 -- 172.21.15.170:0/2140466878 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f268406c9b0 con 0x7f26841589b0 2024-03-28T23:29:29.894 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.888+0000 7f266bfff640 1 -- 172.21.15.170:0/2140466878 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f267c01d070 con 0x7f26841589b0 2024-03-28T23:29:29.894 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.888+0000 7f266bfff640 1 -- 172.21.15.170:0/2140466878 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f267c01fbc0 con 0x7f26841589b0 2024-03-28T23:29:29.894 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.889+0000 7f268a240640 1 -- 172.21.15.170:0/2140466878 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f268404f4b0 con 0x7f26841589b0 2024-03-28T23:29:29.894 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.891+0000 7f266bfff640 1 -- 172.21.15.170:0/2140466878 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f267c024020 con 0x7f26841589b0 2024-03-28T23:29:29.965 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.964+0000 7f268a240640 1 -- 172.21.15.170:0/2140466878 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd new", "uuid": "7b59decd-6f16-4eea-a2d3-1e0618870d6e", "id": 0} v 0) -- 0x7f26840c9260 con 0x7f26841589b0 2024-03-28T23:29:29.967 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.967+0000 7f266bfff640 1 -- 172.21.15.170:0/2140466878 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7b59decd-6f16-4eea-a2d3-1e0618870d6e", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f267c010070 con 0x7f26841589b0 2024-03-28T23:29:29.968 INFO:teuthology.orchestra.run.smithi170.stdout:0 2024-03-28T23:29:29.969 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.968+0000 7f268a240640 1 -- 172.21.15.170:0/2140466878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26841589b0 msgr2=0x7f2684153880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:29.969 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.968+0000 7f268a240640 1 --2- 172.21.15.170:0/2140466878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26841589b0 0x7f2684153880 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f267c007e20 tx=0x7f267c00d4a0 comp rx=0 tx=0).stop 2024-03-28T23:29:29.969 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.968+0000 7f268a240640 1 -- 172.21.15.170:0/2140466878 shutdown_connections 2024-03-28T23:29:29.969 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.968+0000 7f268a240640 1 --2- 172.21.15.170:0/2140466878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26841589b0 0x7f2684153880 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:29.969 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.968+0000 7f268a240640 1 --2- 172.21.15.170:0/2140466878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2684152bc0 0x7f2684153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:29.969 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.968+0000 7f268a240640 1 --2- 172.21.15.170:0/2140466878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26840c8a80 0x7f268406c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:29.969 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.968+0000 7f268a240640 1 -- 172.21.15.170:0/2140466878 >> 172.21.15.170:0/2140466878 conn(0x7f26840caa80 msgr2=0x7f26840aa620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:29.969 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.968+0000 7f268a240640 1 -- 172.21.15.170:0/2140466878 shutdown_connections 2024-03-28T23:29:29.970 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:29.968+0000 7f268a240640 1 -- 172.21.15.170:0/2140466878 wait complete. 2024-03-28T23:29:29.979 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new ed6fe366-2832-4857-98f7-66f63a7b0f27 1 2024-03-28T23:29:30.079 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.078+0000 7fb87ec5c640 1 Processor -- start 2024-03-28T23:29:30.080 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.078+0000 7fb87ec5c640 1 -- start start 2024-03-28T23:29:30.081 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.079+0000 7fb87ec5c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb8780c5620 0x7fb8780c92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.082 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.079+0000 7fb87ec5c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb8780c5af0 0x7fb878158810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.082 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.079+0000 7fb87ec5c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8780c0450 con 0x7fb878152b60 2024-03-28T23:29:30.082 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.079+0000 7fb87ec5c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb8781597c0 con 0x7fb8780c5af0 2024-03-28T23:29:30.082 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.079+0000 7fb87ec5c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fb8780c0040 con 0x7fb8780c5620 2024-03-28T23:29:30.082 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.079+0000 7fb877fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb8780c5620 0x7fb8780c92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.082 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.079+0000 7fb877fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb8780c5620 0x7fb8780c92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:55768/0 (socket says 172.21.15.170:55768) 2024-03-28T23:29:30.082 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.079+0000 7fb877fff640 1 -- 172.21.15.170:0/1675925014 learned_addr learned my addr 172.21.15.170:0/1675925014 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T23:29:30.082 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.079+0000 7fb87d1f9640 1 --2- 172.21.15.170:0/1675925014 >> v2:172.21.15.161:3300/0 conn(0x7fb8780c5af0 0x7fb878158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.082 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.080+0000 7fb877fff640 1 -- 172.21.15.170:0/1675925014 >> v2:172.21.15.161:3300/0 conn(0x7fb8780c5af0 msgr2=0x7fb878158810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:30.082 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.080+0000 7fb877fff640 1 --2- 172.21.15.170:0/1675925014 >> v2:172.21.15.161:3300/0 conn(0x7fb8780c5af0 0x7fb878158810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.082 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.080+0000 7fb877fff640 1 -- 172.21.15.170:0/1675925014 >> v1:172.21.15.161:6789/0 conn(0x7fb878152b60 legacy=0x7fb8780c6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:30.082 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.080+0000 7fb877fff640 1 -- 172.21.15.170:0/1675925014 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8781594e0 con 0x7fb8780c5620 2024-03-28T23:29:30.083 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.080+0000 7fb877fff640 1 --2- 172.21.15.170:0/1675925014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8780c5620 0x7fb8780c92e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb878073800 tx=0x7fb86402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8456acf61b923466 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:30.083 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.080+0000 7fb8777fe640 1 -- 172.21.15.170:0/1675925014 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb864035020 con 0x7fb8780c5620 2024-03-28T23:29:30.083 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.081+0000 7fb8777fe640 1 -- 172.21.15.170:0/1675925014 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb864031870 con 0x7fb8780c5620 2024-03-28T23:29:30.083 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.081+0000 7fb8777fe640 1 -- 172.21.15.170:0/1675925014 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb864031c40 con 0x7fb8780c5620 2024-03-28T23:29:30.087 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.081+0000 7fb87ec5c640 1 -- 172.21.15.170:0/1675925014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8780c5620 msgr2=0x7fb8780c92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:30.088 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.081+0000 7fb87ec5c640 1 --2- 172.21.15.170:0/1675925014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8780c5620 0x7fb8780c92e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb878073800 tx=0x7fb86402b910 comp rx=0 tx=0).stop 2024-03-28T23:29:30.088 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.081+0000 7fb87ec5c640 1 -- 172.21.15.170:0/1675925014 shutdown_connections 2024-03-28T23:29:30.088 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.081+0000 7fb87ec5c640 1 --2- 172.21.15.170:0/1675925014 >> v2:172.21.15.161:3300/0 conn(0x7fb8780c5af0 0x7fb878158810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.088 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.081+0000 7fb87ec5c640 1 --2- 172.21.15.170:0/1675925014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8780c5620 0x7fb8780c92e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.088 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.081+0000 7fb87ec5c640 1 -- 172.21.15.170:0/1675925014 >> 172.21.15.170:0/1675925014 conn(0x7fb8780caa00 msgr2=0x7fb8780cae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:30.088 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.081+0000 7fb87ec5c640 1 -- 172.21.15.170:0/1675925014 shutdown_connections 2024-03-28T23:29:30.088 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.081+0000 7fb87ec5c640 1 -- 172.21.15.170:0/1675925014 wait complete. 2024-03-28T23:29:30.088 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.082+0000 7fb87ec5c640 1 Processor -- start 2024-03-28T23:29:30.088 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.082+0000 7fb87ec5c640 1 -- start start 2024-03-28T23:29:30.088 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.083+0000 7fb87ec5c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8780c5620 0x7fb878157e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.088 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.083+0000 7fb87ec5c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8780c5af0 0x7fb8781eea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.089 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.083+0000 7fb87ec5c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb878152b60 0x7fb8780aa560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.089 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.083+0000 7fb87ec5c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb878158d50 con 0x7fb878152b60 2024-03-28T23:29:30.089 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.083+0000 7fb87ec5c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb8780c0200 con 0x7fb8780c5620 2024-03-28T23:29:30.089 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.083+0000 7fb87ec5c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb878158ed0 con 0x7fb8780c5af0 2024-03-28T23:29:30.089 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.083+0000 7fb87c9f8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8780c5620 0x7fb878157e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.089 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.083+0000 7fb877fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8780c5af0 0x7fb8781eea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.089 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.083+0000 7fb87c9f8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8780c5620 0x7fb878157e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:55784/0 (socket says 172.21.15.170:55784) 2024-03-28T23:29:30.089 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.083+0000 7fb87c9f8640 1 -- 172.21.15.170:0/3531652060 learned_addr learned my addr 172.21.15.170:0/3531652060 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T23:29:30.089 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.083+0000 7fb87d1f9640 1 --2- 172.21.15.170:0/3531652060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb878152b60 0x7fb8780aa560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.090 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.084+0000 7fb87c9f8640 1 -- 172.21.15.170:0/3531652060 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8780c5af0 msgr2=0x7fb8781eea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:30.090 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.084+0000 7fb87c9f8640 1 --2- 172.21.15.170:0/3531652060 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8780c5af0 0x7fb8781eea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.090 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.084+0000 7fb87c9f8640 1 -- 172.21.15.170:0/3531652060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb878152b60 msgr2=0x7fb8780aa560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:30.090 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.084+0000 7fb87c9f8640 1 --2- 172.21.15.170:0/3531652060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb878152b60 0x7fb8780aa560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.090 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.084+0000 7fb87c9f8640 1 -- 172.21.15.170:0/3531652060 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb87814a830 con 0x7fb8780c5620 2024-03-28T23:29:30.090 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.084+0000 7fb877fff640 1 --2- 172.21.15.170:0/3531652060 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8780c5af0 0x7fb8781eea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:29:30.090 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.084+0000 7fb87d1f9640 1 --2- 172.21.15.170:0/3531652060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb878152b60 0x7fb8780aa560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:29:30.090 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.084+0000 7fb87c9f8640 1 --2- 172.21.15.170:0/3531652060 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8780c5620 0x7fb878157e80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb86c00c3c0 tx=0x7fb86c00c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:30.090 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.084+0000 7fb8757fa640 1 -- 172.21.15.170:0/3531652060 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86c0052d0 con 0x7fb8780c5620 2024-03-28T23:29:30.090 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.084+0000 7fb8757fa640 1 -- 172.21.15.170:0/3531652060 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb86c005490 con 0x7fb8780c5620 2024-03-28T23:29:30.090 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.084+0000 7fb87ec5c640 1 -- 172.21.15.170:0/3531652060 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8780abc40 con 0x7fb8780c5620 2024-03-28T23:29:30.090 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.084+0000 7fb8757fa640 1 -- 172.21.15.170:0/3531652060 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86c007260 con 0x7fb8780c5620 2024-03-28T23:29:30.090 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.084+0000 7fb87ec5c640 1 -- 172.21.15.170:0/3531652060 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8780ac200 con 0x7fb8780c5620 2024-03-28T23:29:30.091 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.084+0000 7fb8757fa640 1 -- 172.21.15.170:0/3531652060 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb86c007400 con 0x7fb8780c5620 2024-03-28T23:29:30.091 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.084+0000 7fb8757fa640 1 -- 172.21.15.170:0/3531652060 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fb86c003a90 con 0x7fb8780c5620 2024-03-28T23:29:30.091 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.085+0000 7fb87ec5c640 1 -- 172.21.15.170:0/3531652060 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb878068f90 con 0x7fb8780c5620 2024-03-28T23:29:30.091 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.087+0000 7fb8757fa640 1 -- 172.21.15.170:0/3531652060 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb878068f90 con 0x7fb8780c5620 2024-03-28T23:29:30.162 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.161+0000 7fb87ec5c640 1 -- 172.21.15.170:0/3531652060 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ed6fe366-2832-4857-98f7-66f63a7b0f27", "id": 1} v 0) -- 0x7fb8781ee360 con 0x7fb8780c5620 2024-03-28T23:29:30.165 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.164+0000 7fb8757fa640 1 -- 172.21.15.170:0/3531652060 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ed6fe366-2832-4857-98f7-66f63a7b0f27", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fb86c0035f0 con 0x7fb8780c5620 2024-03-28T23:29:30.165 INFO:teuthology.orchestra.run.smithi170.stdout:1 2024-03-28T23:29:30.167 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.165+0000 7fb87ec5c640 1 -- 172.21.15.170:0/3531652060 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8780c5620 msgr2=0x7fb878157e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:30.167 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.165+0000 7fb87ec5c640 1 --2- 172.21.15.170:0/3531652060 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8780c5620 0x7fb878157e80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb86c00c3c0 tx=0x7fb86c00c880 comp rx=0 tx=0).stop 2024-03-28T23:29:30.167 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.165+0000 7fb87ec5c640 1 -- 172.21.15.170:0/3531652060 shutdown_connections 2024-03-28T23:29:30.167 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.165+0000 7fb87ec5c640 1 --2- 172.21.15.170:0/3531652060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb878152b60 0x7fb8780aa560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.167 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.165+0000 7fb87ec5c640 1 --2- 172.21.15.170:0/3531652060 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8780c5af0 0x7fb8781eea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.167 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.165+0000 7fb87ec5c640 1 --2- 172.21.15.170:0/3531652060 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8780c5620 0x7fb878157e80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.167 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.165+0000 7fb87ec5c640 1 -- 172.21.15.170:0/3531652060 >> 172.21.15.170:0/3531652060 conn(0x7fb8780caa00 msgr2=0x7fb87814b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:30.167 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.166+0000 7fb87ec5c640 1 -- 172.21.15.170:0/3531652060 shutdown_connections 2024-03-28T23:29:30.167 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.166+0000 7fb87ec5c640 1 -- 172.21.15.170:0/3531652060 wait complete. 2024-03-28T23:29:30.177 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 91d1af65-9bf9-4271-98ac-b66ba5254e8d 2 2024-03-28T23:29:30.199 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:30.198+0000 7f772ca2c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T23:29:30.200 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:30.200+0000 7f637f216040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T23:29:30.253 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:30.253+0000 7f637f216040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T23:29:30.255 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:30.254+0000 7f772ca2c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T23:29:30.277 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.275+0000 7f59ba60e640 1 Processor -- start 2024-03-28T23:29:30.277 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.276+0000 7f59ba60e640 1 -- start start 2024-03-28T23:29:30.279 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.276+0000 7f59ba60e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f59b4152bc0 0x7f59b40c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.279 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.276+0000 7f59ba60e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59b40c8a80 0x7f59b40c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.279 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.276+0000 7f59ba60e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f59b40c9420 0x7f59b41589b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.280 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.276+0000 7f59ba60e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f59b40c6f60 con 0x7f59b40c9420 2024-03-28T23:29:30.280 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.276+0000 7f59ba60e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f59b40c6120 con 0x7f59b4152bc0 2024-03-28T23:29:30.280 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.276+0000 7f59ba60e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f59b40c6400 con 0x7f59b40c8a80 2024-03-28T23:29:30.280 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.276+0000 7f59b3fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f59b4152bc0 0x7f59b40c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.280 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.276+0000 7f59b37fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59b40c8a80 0x7f59b40c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.280 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.276+0000 7f59b37fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59b40c8a80 0x7f59b40c8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:50450/0 (socket says 172.21.15.170:50450) 2024-03-28T23:29:30.280 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.276+0000 7f59b3fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f59b4152bc0 0x7f59b40c0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:55792/0 (socket says 172.21.15.170:55792) 2024-03-28T23:29:30.280 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.276+0000 7f59b37fe640 1 -- 172.21.15.170:0/813472118 learned_addr learned my addr 172.21.15.170:0/813472118 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T23:29:30.280 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.277+0000 7f59b8bab640 1 --2- 172.21.15.170:0/813472118 >> v2:172.21.15.161:3300/0 conn(0x7f59b40c9420 0x7f59b41589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.280 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.277+0000 7f59b37fe640 1 -- 172.21.15.170:0/813472118 >> v2:172.21.15.170:3300/0 conn(0x7f59b4152bc0 msgr2=0x7f59b40c0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:30.280 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.277+0000 7f59b37fe640 1 --2- 172.21.15.170:0/813472118 >> v2:172.21.15.170:3300/0 conn(0x7f59b4152bc0 0x7f59b40c0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.280 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.277+0000 7f59b37fe640 1 -- 172.21.15.170:0/813472118 >> v2:172.21.15.161:3300/0 conn(0x7f59b40c9420 msgr2=0x7f59b41589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:30.281 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.277+0000 7f59b37fe640 1 --2- 172.21.15.170:0/813472118 >> v2:172.21.15.161:3300/0 conn(0x7f59b40c9420 0x7f59b41589b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.281 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.277+0000 7f59b37fe640 1 -- 172.21.15.170:0/813472118 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59b40c7240 con 0x7f59b40c8a80 2024-03-28T23:29:30.281 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.277+0000 7f59b3fff640 1 --2- 172.21.15.170:0/813472118 >> v2:172.21.15.170:3300/0 conn(0x7f59b4152bc0 0x7f59b40c0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:29:30.281 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.277+0000 7f59b37fe640 1 --2- 172.21.15.170:0/813472118 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59b40c8a80 0x7f59b40c8ee0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f59b40aa410 tx=0x7f59a802b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4252d01d8ca72ad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:30.281 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.278+0000 7f59b2ffd640 1 -- 172.21.15.170:0/813472118 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a8035020 con 0x7f59b40c8a80 2024-03-28T23:29:30.281 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.278+0000 7f59b2ffd640 1 -- 172.21.15.170:0/813472118 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59a802d040 con 0x7f59b40c8a80 2024-03-28T23:29:30.281 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.278+0000 7f59b2ffd640 1 -- 172.21.15.170:0/813472118 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a8031a80 con 0x7f59b40c8a80 2024-03-28T23:29:30.281 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.278+0000 7f59ba60e640 1 -- 172.21.15.170:0/813472118 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59b40c8a80 msgr2=0x7f59b40c8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:30.281 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.278+0000 7f59ba60e640 1 --2- 172.21.15.170:0/813472118 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59b40c8a80 0x7f59b40c8ee0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f59b40aa410 tx=0x7f59a802b900 comp rx=0 tx=0).stop 2024-03-28T23:29:30.281 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.278+0000 7f59ba60e640 1 -- 172.21.15.170:0/813472118 shutdown_connections 2024-03-28T23:29:30.281 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.278+0000 7f59ba60e640 1 --2- 172.21.15.170:0/813472118 >> v2:172.21.15.161:3300/0 conn(0x7f59b40c9420 0x7f59b41589b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.281 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.278+0000 7f59ba60e640 1 --2- 172.21.15.170:0/813472118 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59b40c8a80 0x7f59b40c8ee0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.281 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.278+0000 7f59ba60e640 1 --2- 172.21.15.170:0/813472118 >> v2:172.21.15.170:3300/0 conn(0x7f59b4152bc0 0x7f59b40c0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.282 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.278+0000 7f59ba60e640 1 -- 172.21.15.170:0/813472118 >> 172.21.15.170:0/813472118 conn(0x7f59b40caa80 msgr2=0x7f59b40cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:30.282 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.278+0000 7f59ba60e640 1 -- 172.21.15.170:0/813472118 shutdown_connections 2024-03-28T23:29:30.282 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.278+0000 7f59ba60e640 1 -- 172.21.15.170:0/813472118 wait complete. 2024-03-28T23:29:30.284 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.279+0000 7f59ba60e640 1 Processor -- start 2024-03-28T23:29:30.284 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.280+0000 7f59ba60e640 1 -- start start 2024-03-28T23:29:30.284 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.280+0000 7f59ba60e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59b40c8a80 0x7f59b406c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.284 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.280+0000 7f59ba60e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59b40c9420 0x7f59b40ae030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.284 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.280+0000 7f59ba60e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59b4152bc0 0x7f59b40ae570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.285 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.280+0000 7f59ba60e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f59b4152fa0 con 0x7f59b40c8a80 2024-03-28T23:29:30.285 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.280+0000 7f59ba60e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f59b4158ef0 con 0x7f59b4152bc0 2024-03-28T23:29:30.285 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.280+0000 7f59ba60e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f59b4150d80 con 0x7f59b40c9420 2024-03-28T23:29:30.285 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.280+0000 7f59b37fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59b40c9420 0x7f59b40ae030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.285 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.280+0000 7f59b8bab640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59b4152bc0 0x7f59b40ae570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.285 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.280+0000 7f59b37fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59b40c9420 0x7f59b40ae030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:50452/0 (socket says 172.21.15.170:50452) 2024-03-28T23:29:30.285 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.280+0000 7f59b8bab640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59b4152bc0 0x7f59b40ae570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:55798/0 (socket says 172.21.15.170:55798) 2024-03-28T23:29:30.285 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.280+0000 7f59b37fe640 1 -- 172.21.15.170:0/2185318423 learned_addr learned my addr 172.21.15.170:0/2185318423 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T23:29:30.285 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.280+0000 7f59b3fff640 1 --2- 172.21.15.170:0/2185318423 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59b40c8a80 0x7f59b406c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.285 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.281+0000 7f59b37fe640 1 -- 172.21.15.170:0/2185318423 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59b4152bc0 msgr2=0x7f59b40ae570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:30.285 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.281+0000 7f59b37fe640 1 --2- 172.21.15.170:0/2185318423 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59b4152bc0 0x7f59b40ae570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.285 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.281+0000 7f59b37fe640 1 -- 172.21.15.170:0/2185318423 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59b40c8a80 msgr2=0x7f59b406c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:30.285 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.281+0000 7f59b37fe640 1 --2- 172.21.15.170:0/2185318423 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59b40c8a80 0x7f59b406c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.286 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.281+0000 7f59b37fe640 1 -- 172.21.15.170:0/2185318423 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59b4151210 con 0x7f59b40c9420 2024-03-28T23:29:30.286 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.281+0000 7f59b8bab640 1 --2- 172.21.15.170:0/2185318423 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59b4152bc0 0x7f59b40ae570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:29:30.286 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.281+0000 7f59b3fff640 1 --2- 172.21.15.170:0/2185318423 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59b40c8a80 0x7f59b406c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:29:30.286 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.281+0000 7f59b37fe640 1 --2- 172.21.15.170:0/2185318423 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59b40c9420 0x7f59b40ae030 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f59a8000c00 tx=0x7f59a80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:30.286 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.281+0000 7f59b0ff9640 1 -- 172.21.15.170:0/2185318423 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a8040070 con 0x7f59b40c9420 2024-03-28T23:29:30.286 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.281+0000 7f59b0ff9640 1 -- 172.21.15.170:0/2185318423 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59a80398e0 con 0x7f59b40c9420 2024-03-28T23:29:30.286 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.281+0000 7f59ba60e640 1 -- 172.21.15.170:0/2185318423 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59b40aeab0 con 0x7f59b40c9420 2024-03-28T23:29:30.286 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.281+0000 7f59b0ff9640 1 -- 172.21.15.170:0/2185318423 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a8035040 con 0x7f59b40c9420 2024-03-28T23:29:30.286 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.281+0000 7f59ba60e640 1 -- 172.21.15.170:0/2185318423 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f59b4150730 con 0x7f59b40c9420 2024-03-28T23:29:30.286 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.281+0000 7f59b0ff9640 1 -- 172.21.15.170:0/2185318423 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f59a8038020 con 0x7f59b40c9420 2024-03-28T23:29:30.286 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.281+0000 7f59b0ff9640 1 -- 172.21.15.170:0/2185318423 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f59a803abf0 con 0x7f59b40c9420 2024-03-28T23:29:30.286 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.282+0000 7f59ba60e640 1 -- 172.21.15.170:0/2185318423 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59b404f4b0 con 0x7f59b40c9420 2024-03-28T23:29:30.287 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.284+0000 7f59b0ff9640 1 -- 172.21.15.170:0/2185318423 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f59a8039cd0 con 0x7f59b40c9420 2024-03-28T23:29:30.299 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:30.298+0000 7f637f216040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T23:29:30.302 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:30.302+0000 7f772ca2c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T23:29:30.353 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:30.352+0000 7f637f216040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T23:29:30.357 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.357+0000 7f59ba60e640 1 -- 172.21.15.170:0/2185318423 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd new", "uuid": "91d1af65-9bf9-4271-98ac-b66ba5254e8d", "id": 2} v 0) -- 0x7f59b40cb120 con 0x7f59b40c9420 2024-03-28T23:29:30.358 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:30.357+0000 7f772ca2c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T23:29:30.362 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.362+0000 7f59b0ff9640 1 -- 172.21.15.170:0/2185318423 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "91d1af65-9bf9-4271-98ac-b66ba5254e8d", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f59a803a420 con 0x7f59b40c9420 2024-03-28T23:29:30.363 INFO:teuthology.orchestra.run.smithi170.stdout:2 2024-03-28T23:29:30.364 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.363+0000 7f59ba60e640 1 -- 172.21.15.170:0/2185318423 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59b40c9420 msgr2=0x7f59b40ae030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:30.364 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.363+0000 7f59ba60e640 1 --2- 172.21.15.170:0/2185318423 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59b40c9420 0x7f59b40ae030 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f59a8000c00 tx=0x7f59a80046c0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.364 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.363+0000 7f59ba60e640 1 -- 172.21.15.170:0/2185318423 shutdown_connections 2024-03-28T23:29:30.364 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.363+0000 7f59ba60e640 1 --2- 172.21.15.170:0/2185318423 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59b4152bc0 0x7f59b40ae570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.364 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.363+0000 7f59ba60e640 1 --2- 172.21.15.170:0/2185318423 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59b40c9420 0x7f59b40ae030 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.364 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.363+0000 7f59ba60e640 1 --2- 172.21.15.170:0/2185318423 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59b40c8a80 0x7f59b406c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.364 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.363+0000 7f59ba60e640 1 -- 172.21.15.170:0/2185318423 >> 172.21.15.170:0/2185318423 conn(0x7f59b40caa80 msgr2=0x7f59b414d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:30.365 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.363+0000 7f59ba60e640 1 -- 172.21.15.170:0/2185318423 shutdown_connections 2024-03-28T23:29:30.365 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.363+0000 7f59ba60e640 1 -- 172.21.15.170:0/2185318423 wait complete. 2024-03-28T23:29:30.375 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 0a6cc81f-c226-4929-9405-3efb9c0e0723 3 2024-03-28T23:29:30.395 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:30.395+0000 7f637f216040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T23:29:30.403 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:30.402+0000 7f772ca2c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T23:29:30.504 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.503+0000 7efd91d02640 1 Processor -- start 2024-03-28T23:29:30.504 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.503+0000 7efd91d02640 1 -- start start 2024-03-28T23:29:30.507 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.504+0000 7efd91d02640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7efd8c0c0330 0x7efd8c0c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.507 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.504+0000 7efd91d02640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd8c1589b0 0x7efd8c158e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.507 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.504+0000 7efd91d02640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd8c0c6f60 con 0x7efd8c152bc0 2024-03-28T23:29:30.507 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.504+0000 7efd91d02640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7efd8c0c6120 con 0x7efd8c0c0330 2024-03-28T23:29:30.507 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.504+0000 7efd91d02640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7efd8c0c6400 con 0x7efd8c1589b0 2024-03-28T23:29:30.508 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.504+0000 7efd8affd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7efd8c0c0330 0x7efd8c0c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.508 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.504+0000 7efd8bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd8c1589b0 0x7efd8c158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.508 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.504+0000 7efd8affd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7efd8c0c0330 0x7efd8c0c9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:55800/0 (socket says 172.21.15.170:55800) 2024-03-28T23:29:30.508 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.504+0000 7efd8affd640 1 -- 172.21.15.170:0/2629336737 learned_addr learned my addr 172.21.15.170:0/2629336737 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T23:29:30.508 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.504+0000 7efd8bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd8c1589b0 0x7efd8c158e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:50460/0 (socket says 172.21.15.170:50460) 2024-03-28T23:29:30.508 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.504+0000 7efd8affd640 1 -- 172.21.15.170:0/2629336737 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd8c1589b0 msgr2=0x7efd8c158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:30.508 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.504+0000 7efd8affd640 1 --2- 172.21.15.170:0/2629336737 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd8c1589b0 0x7efd8c158e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.508 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.504+0000 7efd8affd640 1 -- 172.21.15.170:0/2629336737 >> v1:172.21.15.161:6789/0 conn(0x7efd8c152bc0 legacy=0x7efd8c0c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:30.508 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.504+0000 7efd8affd640 1 -- 172.21.15.170:0/2629336737 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd8c0c96d0 con 0x7efd8c0c0330 2024-03-28T23:29:30.509 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.504+0000 7efd8bfff640 1 --2- 172.21.15.170:0/2629336737 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd8c1589b0 0x7efd8c158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:29:30.509 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.505+0000 7efd8affd640 1 --2- 172.21.15.170:0/2629336737 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd8c0c0330 0x7efd8c0c9190 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efd7c009870 tx=0x7efd7c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8a17f46c2d38e39 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:30.509 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.505+0000 7efd8a7fc640 1 -- 172.21.15.170:0/2629336737 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7c034070 con 0x7efd8c0c0330 2024-03-28T23:29:30.509 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.505+0000 7efd8a7fc640 1 -- 172.21.15.170:0/2629336737 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd7c030840 con 0x7efd8c0c0330 2024-03-28T23:29:30.509 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.505+0000 7efd8a7fc640 1 -- 172.21.15.170:0/2629336737 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7c030c10 con 0x7efd8c0c0330 2024-03-28T23:29:30.509 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.506+0000 7efd91d02640 1 -- 172.21.15.170:0/2629336737 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd8c0c0330 msgr2=0x7efd8c0c9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:30.509 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.506+0000 7efd91d02640 1 --2- 172.21.15.170:0/2629336737 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd8c0c0330 0x7efd8c0c9190 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efd7c009870 tx=0x7efd7c02b980 comp rx=0 tx=0).stop 2024-03-28T23:29:30.509 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.506+0000 7efd91d02640 1 -- 172.21.15.170:0/2629336737 shutdown_connections 2024-03-28T23:29:30.509 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.506+0000 7efd91d02640 1 --2- 172.21.15.170:0/2629336737 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd8c1589b0 0x7efd8c158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.509 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.506+0000 7efd91d02640 1 --2- 172.21.15.170:0/2629336737 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd8c0c0330 0x7efd8c0c9190 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.509 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.506+0000 7efd91d02640 1 -- 172.21.15.170:0/2629336737 >> 172.21.15.170:0/2629336737 conn(0x7efd8c0caa80 msgr2=0x7efd8c0cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:30.510 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.506+0000 7efd91d02640 1 -- 172.21.15.170:0/2629336737 shutdown_connections 2024-03-28T23:29:30.510 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.506+0000 7efd91d02640 1 -- 172.21.15.170:0/2629336737 wait complete. 2024-03-28T23:29:30.513 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.507+0000 7efd91d02640 1 Processor -- start 2024-03-28T23:29:30.514 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.507+0000 7efd91d02640 1 -- start start 2024-03-28T23:29:30.514 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.508+0000 7efd91d02640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd8c0c0330 0x7efd8c1575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.514 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.508+0000 7efd91d02640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd8c152bc0 0x7efd8c157af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.514 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.508+0000 7efd91d02640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd8c1589b0 0x7efd8c1eeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.514 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.508+0000 7efd91d02640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efd8c152fa0 con 0x7efd8c0c0330 2024-03-28T23:29:30.514 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.508+0000 7efd91d02640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7efd8c0c6bf0 con 0x7efd8c1589b0 2024-03-28T23:29:30.514 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.508+0000 7efd91d02640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7efd8c1519a0 con 0x7efd8c152bc0 2024-03-28T23:29:30.514 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.508+0000 7efd8bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd8c1589b0 0x7efd8c1eeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.514 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.508+0000 7efd8bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd8c1589b0 0x7efd8c1eeb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:55802/0 (socket says 172.21.15.170:55802) 2024-03-28T23:29:30.514 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.508+0000 7efd8affd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd8c152bc0 0x7efd8c157af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.514 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.508+0000 7efd8bfff640 1 -- 172.21.15.170:0/1592676595 learned_addr learned my addr 172.21.15.170:0/1592676595 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T23:29:30.514 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.508+0000 7efd8b7fe640 1 --2- 172.21.15.170:0/1592676595 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd8c0c0330 0x7efd8c1575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.515 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.508+0000 7efd8bfff640 1 -- 172.21.15.170:0/1592676595 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd8c152bc0 msgr2=0x7efd8c157af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:30.515 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.508+0000 7efd8bfff640 1 --2- 172.21.15.170:0/1592676595 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd8c152bc0 0x7efd8c157af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.515 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.508+0000 7efd8bfff640 1 -- 172.21.15.170:0/1592676595 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd8c0c0330 msgr2=0x7efd8c1575b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:30.515 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.508+0000 7efd8bfff640 1 --2- 172.21.15.170:0/1592676595 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd8c0c0330 0x7efd8c1575b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.515 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.508+0000 7efd8bfff640 1 -- 172.21.15.170:0/1592676595 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd8c151bf0 con 0x7efd8c1589b0 2024-03-28T23:29:30.515 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.508+0000 7efd8bfff640 1 --2- 172.21.15.170:0/1592676595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd8c1589b0 0x7efd8c1eeb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efd8400efc0 tx=0x7efd8400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:30.515 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.508+0000 7efd73fff640 1 -- 172.21.15.170:0/1592676595 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd84018030 con 0x7efd8c1589b0 2024-03-28T23:29:30.515 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.508+0000 7efd73fff640 1 -- 172.21.15.170:0/1592676595 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd8400cdf0 con 0x7efd8c1589b0 2024-03-28T23:29:30.515 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.508+0000 7efd91d02640 1 -- 172.21.15.170:0/1592676595 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd8c151ed0 con 0x7efd8c1589b0 2024-03-28T23:29:30.515 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.508+0000 7efd73fff640 1 -- 172.21.15.170:0/1592676595 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd84003bb0 con 0x7efd8c1589b0 2024-03-28T23:29:30.515 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.508+0000 7efd91d02640 1 -- 172.21.15.170:0/1592676595 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd8c158030 con 0x7efd8c1589b0 2024-03-28T23:29:30.515 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.509+0000 7efd73fff640 1 -- 172.21.15.170:0/1592676595 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efd8401b070 con 0x7efd8c1589b0 2024-03-28T23:29:30.515 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.509+0000 7efd73fff640 1 -- 172.21.15.170:0/1592676595 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7efd8401d8f0 con 0x7efd8c1589b0 2024-03-28T23:29:30.515 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.509+0000 7efd91d02640 1 -- 172.21.15.170:0/1592676595 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd8c04f4b0 con 0x7efd8c1589b0 2024-03-28T23:29:30.516 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.511+0000 7efd73fff640 1 -- 172.21.15.170:0/1592676595 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7efd84010040 con 0x7efd8c1589b0 2024-03-28T23:29:30.516 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:30.510+0000 7f637f216040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T23:29:30.525 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:30.525+0000 7f772ca2c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T23:29:30.580 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:30.579+0000 7f637f216040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T23:29:30.585 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.584+0000 7efd91d02640 1 -- 172.21.15.170:0/1592676595 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0a6cc81f-c226-4929-9405-3efb9c0e0723", "id": 3} v 0) -- 0x7efd8c06d400 con 0x7efd8c1589b0 2024-03-28T23:29:30.588 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.587+0000 7efd73fff640 1 -- 172.21.15.170:0/1592676595 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0a6cc81f-c226-4929-9405-3efb9c0e0723", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7efd8401dd60 con 0x7efd8c1589b0 2024-03-28T23:29:30.588 INFO:teuthology.orchestra.run.smithi170.stdout:3 2024-03-28T23:29:30.589 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.588+0000 7efd91d02640 1 -- 172.21.15.170:0/1592676595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd8c1589b0 msgr2=0x7efd8c1eeb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:30.590 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.588+0000 7efd91d02640 1 --2- 172.21.15.170:0/1592676595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd8c1589b0 0x7efd8c1eeb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efd8400efc0 tx=0x7efd8400c490 comp rx=0 tx=0).stop 2024-03-28T23:29:30.590 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.588+0000 7efd91d02640 1 -- 172.21.15.170:0/1592676595 shutdown_connections 2024-03-28T23:29:30.590 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.588+0000 7efd91d02640 1 --2- 172.21.15.170:0/1592676595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd8c1589b0 0x7efd8c1eeb20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.590 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.588+0000 7efd91d02640 1 --2- 172.21.15.170:0/1592676595 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd8c152bc0 0x7efd8c157af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.590 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.588+0000 7efd91d02640 1 --2- 172.21.15.170:0/1592676595 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd8c0c0330 0x7efd8c1575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.590 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.588+0000 7efd91d02640 1 -- 172.21.15.170:0/1592676595 >> 172.21.15.170:0/1592676595 conn(0x7efd8c0caa80 msgr2=0x7efd8c0c5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:30.590 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.589+0000 7efd91d02640 1 -- 172.21.15.170:0/1592676595 shutdown_connections 2024-03-28T23:29:30.590 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.589+0000 7efd91d02640 1 -- 172.21.15.170:0/1592676595 wait complete. 2024-03-28T23:29:30.598 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:30.598+0000 7f772ca2c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T23:29:30.600 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 268e2360-1999-475b-b35f-4e7c443997d2 4 2024-03-28T23:29:30.626 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:30.626+0000 7f637f216040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T23:29:30.647 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:30.647+0000 7f772ca2c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T23:29:30.679 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:30.678+0000 7f637f216040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T23:29:30.701 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.700+0000 7fe820580640 1 Processor -- start 2024-03-28T23:29:30.701 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.700+0000 7fe820580640 1 -- start start 2024-03-28T23:29:30.703 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.701+0000 7fe820580640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe818148980 0x7fe8180c6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.703 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.701+0000 7fe820580640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe8180ac1a0 0x7fe81815c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.703 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.701+0000 7fe820580640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe818152b30 con 0x7fe8180c2dd0 2024-03-28T23:29:30.703 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.701+0000 7fe820580640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe818152e10 con 0x7fe8180ac1a0 2024-03-28T23:29:30.703 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.701+0000 7fe820580640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fe818148d60 con 0x7fe818148980 2024-03-28T23:29:30.703 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.701+0000 7fe81e31c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe818148980 0x7fe8180c6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.703 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.701+0000 7fe81e31c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe818148980 0x7fe8180c6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:55806/0 (socket says 172.21.15.170:55806) 2024-03-28T23:29:30.703 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.701+0000 7fe81e31c640 1 -- 172.21.15.170:0/3068221535 learned_addr learned my addr 172.21.15.170:0/3068221535 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T23:29:30.703 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.701+0000 7fe81eb1d640 1 --2- 172.21.15.170:0/3068221535 >> v2:172.21.15.161:3300/0 conn(0x7fe8180ac1a0 0x7fe81815c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.703 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.701+0000 7fe81e31c640 1 -- 172.21.15.170:0/3068221535 >> v2:172.21.15.161:3300/0 conn(0x7fe8180ac1a0 msgr2=0x7fe81815c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:30.703 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.701+0000 7fe81e31c640 1 --2- 172.21.15.170:0/3068221535 >> v2:172.21.15.161:3300/0 conn(0x7fe8180ac1a0 0x7fe81815c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.703 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.701+0000 7fe81e31c640 1 -- 172.21.15.170:0/3068221535 >> v1:172.21.15.170:6789/0 conn(0x7fe8180c2dd0 legacy=0x7fe8180aba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:30.704 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.701+0000 7fe81e31c640 1 -- 172.21.15.170:0/3068221535 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8180c22f0 con 0x7fe818148980 2024-03-28T23:29:30.704 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.702+0000 7fe81e31c640 1 --2- 172.21.15.170:0/3068221535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe818148980 0x7fe8180c6660 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe8180710e0 tx=0x7fe80802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be9c78cdab0f2642 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:30.704 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:30.702+0000 7f772ca2c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T23:29:30.707 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.702+0000 7fe81d31a640 1 -- 172.21.15.170:0/3068221535 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe808034070 con 0x7fe818148980 2024-03-28T23:29:30.707 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.702+0000 7fe81d31a640 1 -- 172.21.15.170:0/3068221535 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe808030860 con 0x7fe818148980 2024-03-28T23:29:30.707 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.702+0000 7fe81d31a640 1 -- 172.21.15.170:0/3068221535 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe808030c30 con 0x7fe818148980 2024-03-28T23:29:30.707 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.703+0000 7fe820580640 1 -- 172.21.15.170:0/3068221535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe818148980 msgr2=0x7fe8180c6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:30.707 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.703+0000 7fe820580640 1 --2- 172.21.15.170:0/3068221535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe818148980 0x7fe8180c6660 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe8180710e0 tx=0x7fe80802b9a0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.707 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.703+0000 7fe820580640 1 -- 172.21.15.170:0/3068221535 shutdown_connections 2024-03-28T23:29:30.708 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.703+0000 7fe820580640 1 --2- 172.21.15.170:0/3068221535 >> v2:172.21.15.161:3300/0 conn(0x7fe8180ac1a0 0x7fe81815c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.708 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.703+0000 7fe820580640 1 --2- 172.21.15.170:0/3068221535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe818148980 0x7fe8180c6660 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.708 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.703+0000 7fe820580640 1 -- 172.21.15.170:0/3068221535 >> 172.21.15.170:0/3068221535 conn(0x7fe8180c4250 msgr2=0x7fe8180c4660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:30.708 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.703+0000 7fe820580640 1 -- 172.21.15.170:0/3068221535 shutdown_connections 2024-03-28T23:29:30.708 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.703+0000 7fe820580640 1 -- 172.21.15.170:0/3068221535 wait complete. 2024-03-28T23:29:30.708 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.704+0000 7fe820580640 1 Processor -- start 2024-03-28T23:29:30.708 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.704+0000 7fe820580640 1 -- start start 2024-03-28T23:29:30.708 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.705+0000 7fe820580640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8180c2dd0 0x7fe8181dd520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.708 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.705+0000 7fe820580640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe818148980 0x7fe8181536e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.708 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.705+0000 7fe820580640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe81814f820 0x7fe8181ea330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.708 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.705+0000 7fe820580640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe8180c2940 con 0x7fe81814f820 2024-03-28T23:29:30.708 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.705+0000 7fe820580640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe8181528e0 con 0x7fe818148980 2024-03-28T23:29:30.708 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.705+0000 7fe820580640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe8180c3280 con 0x7fe8180c2dd0 2024-03-28T23:29:30.708 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.705+0000 7fe81db1b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe818148980 0x7fe8181536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.709 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.705+0000 7fe81e31c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8180c2dd0 0x7fe8181dd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.709 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.705+0000 7fe81db1b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe818148980 0x7fe8181536e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:55822/0 (socket says 172.21.15.170:55822) 2024-03-28T23:29:30.709 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.705+0000 7fe81e31c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8180c2dd0 0x7fe8181dd520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:50476/0 (socket says 172.21.15.170:50476) 2024-03-28T23:29:30.709 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.705+0000 7fe81db1b640 1 -- 172.21.15.170:0/2641785346 learned_addr learned my addr 172.21.15.170:0/2641785346 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T23:29:30.709 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.705+0000 7fe81eb1d640 1 --2- 172.21.15.170:0/2641785346 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe81814f820 0x7fe8181ea330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.709 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.705+0000 7fe81db1b640 1 -- 172.21.15.170:0/2641785346 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8180c2dd0 msgr2=0x7fe8181dd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:30.709 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.705+0000 7fe81db1b640 1 --2- 172.21.15.170:0/2641785346 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8180c2dd0 0x7fe8181dd520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.709 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.705+0000 7fe81db1b640 1 -- 172.21.15.170:0/2641785346 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe81814f820 msgr2=0x7fe8181ea330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:30.709 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.705+0000 7fe81db1b640 1 --2- 172.21.15.170:0/2641785346 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe81814f820 0x7fe8181ea330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.709 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.705+0000 7fe81db1b640 1 -- 172.21.15.170:0/2641785346 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe818155df0 con 0x7fe818148980 2024-03-28T23:29:30.709 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.705+0000 7fe81eb1d640 1 --2- 172.21.15.170:0/2641785346 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe81814f820 0x7fe8181ea330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:29:30.709 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.705+0000 7fe81db1b640 1 --2- 172.21.15.170:0/2641785346 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe818148980 0x7fe8181536e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe81400c4a0 tx=0x7fe814001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:30.709 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.705+0000 7fe80effd640 1 -- 172.21.15.170:0/2641785346 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8140052d0 con 0x7fe818148980 2024-03-28T23:29:30.710 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.706+0000 7fe80effd640 1 -- 172.21.15.170:0/2641785346 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe814005470 con 0x7fe818148980 2024-03-28T23:29:30.710 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.706+0000 7fe820580640 1 -- 172.21.15.170:0/2641785346 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8181560d0 con 0x7fe818148980 2024-03-28T23:29:30.710 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.706+0000 7fe80effd640 1 -- 172.21.15.170:0/2641785346 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe814007260 con 0x7fe818148980 2024-03-28T23:29:30.710 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.706+0000 7fe820580640 1 -- 172.21.15.170:0/2641785346 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe818154bb0 con 0x7fe818148980 2024-03-28T23:29:30.710 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.706+0000 7fe80effd640 1 -- 172.21.15.170:0/2641785346 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe814007400 con 0x7fe818148980 2024-03-28T23:29:30.710 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.706+0000 7fe80effd640 1 -- 172.21.15.170:0/2641785346 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fe814011590 con 0x7fe818148980 2024-03-28T23:29:30.710 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.706+0000 7fe820580640 1 -- 172.21.15.170:0/2641785346 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe81804f4b0 con 0x7fe818148980 2024-03-28T23:29:30.710 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.708+0000 7fe80effd640 1 -- 172.21.15.170:0/2641785346 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe8140035f0 con 0x7fe818148980 2024-03-28T23:29:30.727 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:30.726+0000 7f637f216040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T23:29:30.752 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:30.752+0000 7f772ca2c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T23:29:30.782 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.781+0000 7fe820580640 1 -- 172.21.15.170:0/2641785346 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd new", "uuid": "268e2360-1999-475b-b35f-4e7c443997d2", "id": 4} v 0) -- 0x7fe818071690 con 0x7fe818148980 2024-03-28T23:29:30.787 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.786+0000 7fe80effd640 1 -- 172.21.15.170:0/2641785346 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "268e2360-1999-475b-b35f-4e7c443997d2", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fe814003a00 con 0x7fe818148980 2024-03-28T23:29:30.787 INFO:teuthology.orchestra.run.smithi170.stdout:4 2024-03-28T23:29:30.788 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.787+0000 7fe820580640 1 -- 172.21.15.170:0/2641785346 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe818148980 msgr2=0x7fe8181536e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:30.788 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.787+0000 7fe820580640 1 --2- 172.21.15.170:0/2641785346 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe818148980 0x7fe8181536e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe81400c4a0 tx=0x7fe814001e70 comp rx=0 tx=0).stop 2024-03-28T23:29:30.788 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.787+0000 7fe820580640 1 -- 172.21.15.170:0/2641785346 shutdown_connections 2024-03-28T23:29:30.788 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.787+0000 7fe820580640 1 --2- 172.21.15.170:0/2641785346 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe81814f820 0x7fe8181ea330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.788 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.787+0000 7fe820580640 1 --2- 172.21.15.170:0/2641785346 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe818148980 0x7fe8181536e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.788 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.787+0000 7fe820580640 1 --2- 172.21.15.170:0/2641785346 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8180c2dd0 0x7fe8181dd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.788 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.787+0000 7fe820580640 1 -- 172.21.15.170:0/2641785346 >> 172.21.15.170:0/2641785346 conn(0x7fe8180c4250 msgr2=0x7fe8180c5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:30.789 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.787+0000 7fe820580640 1 -- 172.21.15.170:0/2641785346 shutdown_connections 2024-03-28T23:29:30.789 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.787+0000 7fe820580640 1 -- 172.21.15.170:0/2641785346 wait complete. 2024-03-28T23:29:30.798 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 08c1ede6-ff85-4dc2-96b3-9a04170152ee 5 2024-03-28T23:29:30.898 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.897+0000 7fc8b87a1640 1 Processor -- start 2024-03-28T23:29:30.899 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.897+0000 7fc8b87a1640 1 -- start start 2024-03-28T23:29:30.900 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.898+0000 7fc8b87a1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc8b00c0330 0x7fc8b00c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.901 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.898+0000 7fc8b87a1640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8b00c6f60 con 0x7fc8b01589b0 2024-03-28T23:29:30.901 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.898+0000 7fc8b87a1640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8b00c6120 con 0x7fc8b0152bc0 2024-03-28T23:29:30.901 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.898+0000 7fc8b87a1640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc8b00c6400 con 0x7fc8b00c0330 2024-03-28T23:29:30.901 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.898+0000 7fc8b653d640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fc8b0152bc0 0x7fc8b00c8a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:53976/0 (socket says 172.21.15.170:53976) 2024-03-28T23:29:30.901 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.898+0000 7fc8b653d640 1 -- 172.21.15.170:0/1631485293 learned_addr learned my addr 172.21.15.170:0/1631485293 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-28T23:29:30.901 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.899+0000 7fc8b5d3c640 1 --2- 172.21.15.170:0/1631485293 >> v2:172.21.15.161:3300/0 conn(0x7fc8b00c0330 0x7fc8b00c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.901 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.899+0000 7fc8b553b640 1 -- 172.21.15.170:0/1631485293 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 175417760 0 0) 0x7fc8b00c6120 con 0x7fc8b0152bc0 2024-03-28T23:29:30.901 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.899+0000 7fc8b553b640 1 -- 172.21.15.170:0/1631485293 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc89c003520 con 0x7fc8b0152bc0 2024-03-28T23:29:30.901 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.899+0000 7fc8b553b640 1 -- 172.21.15.170:0/1631485293 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1366455484 0 0) 0x7fc8b00c6f60 con 0x7fc8b01589b0 2024-03-28T23:29:30.901 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.899+0000 7fc8b553b640 1 -- 172.21.15.170:0/1631485293 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8b00c6120 con 0x7fc8b01589b0 2024-03-28T23:29:30.901 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.899+0000 7fc8b5d3c640 1 -- 172.21.15.170:0/1631485293 >> v1:172.21.15.170:6789/0 conn(0x7fc8b0152bc0 legacy=0x7fc8b00c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:30.901 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.899+0000 7fc8b553b640 1 -- 172.21.15.170:0/1631485293 <== mon.1 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2983245850 0 0) 0x7fc89c003520 con 0x7fc8b0152bc0 2024-03-28T23:29:30.901 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.899+0000 7fc8b5d3c640 1 -- 172.21.15.170:0/1631485293 >> v1:172.21.15.161:6789/0 conn(0x7fc8b01589b0 legacy=0x7fc8b0158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:30.902 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.899+0000 7fc8b5d3c640 1 -- 172.21.15.170:0/1631485293 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8b00c96d0 con 0x7fc8b00c0330 2024-03-28T23:29:30.902 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.900+0000 7fc8b5d3c640 1 --2- 172.21.15.170:0/1631485293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8b00c0330 0x7fc8b00c9190 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0009720 tx=0x7fc8a002b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a39b541bca4ad5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:30.902 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.900+0000 7fc8b553b640 1 -- 172.21.15.170:0/1631485293 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a0038070 con 0x7fc8b00c0330 2024-03-28T23:29:30.902 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.900+0000 7fc8b553b640 1 -- 172.21.15.170:0/1631485293 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8a0031520 con 0x7fc8b00c0330 2024-03-28T23:29:30.902 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.900+0000 7fc8b553b640 1 -- 172.21.15.170:0/1631485293 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac001670 con 0x7fc8b00c0330 2024-03-28T23:29:30.905 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.900+0000 7fc8b87a1640 1 -- 172.21.15.170:0/1631485293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8b00c0330 msgr2=0x7fc8b00c9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:30.905 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.900+0000 7fc8b87a1640 1 --2- 172.21.15.170:0/1631485293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8b00c0330 0x7fc8b00c9190 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0009720 tx=0x7fc8a002b7c0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.905 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.900+0000 7fc8b87a1640 1 -- 172.21.15.170:0/1631485293 shutdown_connections 2024-03-28T23:29:30.905 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.900+0000 7fc8b87a1640 1 --2- 172.21.15.170:0/1631485293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8b00c0330 0x7fc8b00c9190 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.905 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.900+0000 7fc8b87a1640 1 -- 172.21.15.170:0/1631485293 >> 172.21.15.170:0/1631485293 conn(0x7fc8b00caa80 msgr2=0x7fc8b00cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:30.905 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.901+0000 7fc8b87a1640 1 -- 172.21.15.170:0/1631485293 shutdown_connections 2024-03-28T23:29:30.905 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.901+0000 7fc8b87a1640 1 -- 172.21.15.170:0/1631485293 wait complete. 2024-03-28T23:29:30.905 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.902+0000 7fc8b87a1640 1 Processor -- start 2024-03-28T23:29:30.905 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.902+0000 7fc8b87a1640 1 -- start start 2024-03-28T23:29:30.905 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.902+0000 7fc8b87a1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8b00c0330 0x7fc8b00ab040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.905 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.902+0000 7fc8b87a1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8b0152bc0 0x7fc8b0155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.905 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.902+0000 7fc8b87a1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8b01589b0 0x7fc8b0149830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:30.906 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.902+0000 7fc8b87a1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b0152fa0 con 0x7fc8b01589b0 2024-03-28T23:29:30.906 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.902+0000 7fc8b87a1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b00c6bf0 con 0x7fc8b0152bc0 2024-03-28T23:29:30.906 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.902+0000 7fc8b87a1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc8b01585c0 con 0x7fc8b00c0330 2024-03-28T23:29:30.906 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.902+0000 7fc8b5d3c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8b0152bc0 0x7fc8b0155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.906 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.902+0000 7fc8b653d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8b00c0330 0x7fc8b00ab040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.906 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.902+0000 7fc8b5d3c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8b0152bc0 0x7fc8b0155830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:55832/0 (socket says 172.21.15.170:55832) 2024-03-28T23:29:30.906 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.902+0000 7fc8b5d3c640 1 -- 172.21.15.170:0/348031179 learned_addr learned my addr 172.21.15.170:0/348031179 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T23:29:30.906 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.902+0000 7fc8b6d3e640 1 --2- 172.21.15.170:0/348031179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8b01589b0 0x7fc8b0149830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:30.906 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.903+0000 7fc8b5d3c640 1 -- 172.21.15.170:0/348031179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8b00c0330 msgr2=0x7fc8b00ab040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:30.906 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.903+0000 7fc8b5d3c640 1 --2- 172.21.15.170:0/348031179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8b00c0330 0x7fc8b00ab040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.906 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.903+0000 7fc8b5d3c640 1 -- 172.21.15.170:0/348031179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8b01589b0 msgr2=0x7fc8b0149830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:30.906 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.903+0000 7fc8b5d3c640 1 --2- 172.21.15.170:0/348031179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8b01589b0 0x7fc8b0149830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:30.906 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.903+0000 7fc8b5d3c640 1 -- 172.21.15.170:0/348031179 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8b01eec90 con 0x7fc8b0152bc0 2024-03-28T23:29:30.907 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.903+0000 7fc8b653d640 1 --2- 172.21.15.170:0/348031179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8b00c0330 0x7fc8b00ab040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:29:30.907 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.903+0000 7fc8b5d3c640 1 --2- 172.21.15.170:0/348031179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8b0152bc0 0x7fc8b0155830 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0036040 tx=0x7fc8a00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:30.907 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.903+0000 7fc8a6ffd640 1 -- 172.21.15.170:0/348031179 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a0042070 con 0x7fc8b0152bc0 2024-03-28T23:29:30.907 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.903+0000 7fc8a6ffd640 1 -- 172.21.15.170:0/348031179 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8a003d870 con 0x7fc8b0152bc0 2024-03-28T23:29:30.907 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.903+0000 7fc8a6ffd640 1 -- 172.21.15.170:0/348031179 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a0038040 con 0x7fc8b0152bc0 2024-03-28T23:29:30.907 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.903+0000 7fc8b87a1640 1 -- 172.21.15.170:0/348031179 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8b01eded0 con 0x7fc8b0152bc0 2024-03-28T23:29:30.907 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.903+0000 7fc8b87a1640 1 -- 172.21.15.170:0/348031179 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8b01ee490 con 0x7fc8b0152bc0 2024-03-28T23:29:30.907 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.903+0000 7fc8a6ffd640 1 -- 172.21.15.170:0/348031179 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc8a0045070 con 0x7fc8b0152bc0 2024-03-28T23:29:30.907 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.903+0000 7fc8a6ffd640 1 -- 172.21.15.170:0/348031179 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fc8a0033410 con 0x7fc8b0152bc0 2024-03-28T23:29:30.908 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.904+0000 7fc8b87a1640 1 -- 172.21.15.170:0/348031179 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8b004f4b0 con 0x7fc8b0152bc0 2024-03-28T23:29:30.908 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:30.906+0000 7fc8a6ffd640 1 -- 172.21.15.170:0/348031179 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc8a0047450 con 0x7fc8b0152bc0 2024-03-28T23:29:31.058 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.057+0000 7fc8b87a1640 1 -- 172.21.15.170:0/348031179 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd new", "uuid": "08c1ede6-ff85-4dc2-96b3-9a04170152ee", "id": 5} v 0) -- 0x7fc8b00ab580 con 0x7fc8b0152bc0 2024-03-28T23:29:31.064 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.063+0000 7fc8a6ffd640 1 -- 172.21.15.170:0/348031179 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "08c1ede6-ff85-4dc2-96b3-9a04170152ee", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fc8a003dc40 con 0x7fc8b0152bc0 2024-03-28T23:29:31.064 INFO:teuthology.orchestra.run.smithi170.stdout:5 2024-03-28T23:29:31.065 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.064+0000 7fc8b87a1640 1 -- 172.21.15.170:0/348031179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8b0152bc0 msgr2=0x7fc8b0155830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:31.065 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.064+0000 7fc8b87a1640 1 --2- 172.21.15.170:0/348031179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8b0152bc0 0x7fc8b0155830 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0036040 tx=0x7fc8a00046c0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.065 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.064+0000 7fc8b87a1640 1 -- 172.21.15.170:0/348031179 shutdown_connections 2024-03-28T23:29:31.065 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.064+0000 7fc8b87a1640 1 --2- 172.21.15.170:0/348031179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8b01589b0 0x7fc8b0149830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.065 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.064+0000 7fc8b87a1640 1 --2- 172.21.15.170:0/348031179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8b0152bc0 0x7fc8b0155830 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.065 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.064+0000 7fc8b87a1640 1 --2- 172.21.15.170:0/348031179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8b00c0330 0x7fc8b00ab040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.065 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.064+0000 7fc8b87a1640 1 -- 172.21.15.170:0/348031179 >> 172.21.15.170:0/348031179 conn(0x7fc8b00caa80 msgr2=0x7fc8b014e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:31.065 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.064+0000 7fc8b87a1640 1 -- 172.21.15.170:0/348031179 shutdown_connections 2024-03-28T23:29:31.065 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.064+0000 7fc8b87a1640 1 -- 172.21.15.170:0/348031179 wait complete. 2024-03-28T23:29:31.075 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 9e6e8b34-9805-4653-944f-e8b07dbcfadb 6 2024-03-28T23:29:31.103 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:31.103+0000 7f637f216040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T23:29:31.177 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.176+0000 7f2fd50ae640 1 Processor -- start 2024-03-28T23:29:31.177 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.176+0000 7f2fd50ae640 1 -- start start 2024-03-28T23:29:31.179 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.177+0000 7f2fd50ae640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2fd006c160 0x7f2fd006e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:31.179 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.177+0000 7f2fd50ae640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2fd006c540 0x7f2fd00c1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:31.179 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.177+0000 7f2fd50ae640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fd006c9a0 con 0x7f2fd0153bb0 2024-03-28T23:29:31.179 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.177+0000 7f2fd50ae640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2fd006e290 con 0x7f2fd006c540 2024-03-28T23:29:31.179 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.177+0000 7f2fd50ae640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f2fd006f650 con 0x7f2fd006c160 2024-03-28T23:29:31.179 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.177+0000 7f2fced9d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2fd006c160 0x7f2fd006e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:31.179 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.177+0000 7f2fced9d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2fd006c160 0x7f2fd006e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:55836/0 (socket says 172.21.15.170:55836) 2024-03-28T23:29:31.179 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.177+0000 7f2fced9d640 1 -- 172.21.15.170:0/253205768 learned_addr learned my addr 172.21.15.170:0/253205768 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T23:29:31.179 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.177+0000 7f2fce59c640 1 --2- 172.21.15.170:0/253205768 >> v2:172.21.15.161:3300/0 conn(0x7f2fd006c540 0x7f2fd00c1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:31.180 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.177+0000 7f2fced9d640 1 -- 172.21.15.170:0/253205768 >> v2:172.21.15.161:3300/0 conn(0x7f2fd006c540 msgr2=0x7f2fd00c1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:31.180 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.177+0000 7f2fced9d640 1 --2- 172.21.15.170:0/253205768 >> v2:172.21.15.161:3300/0 conn(0x7f2fd006c540 0x7f2fd00c1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.180 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.177+0000 7f2fced9d640 1 -- 172.21.15.170:0/253205768 >> v1:172.21.15.161:6789/0 conn(0x7f2fd0153bb0 legacy=0x7f2fd0150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:31.180 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.177+0000 7f2fced9d640 1 -- 172.21.15.170:0/253205768 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fd006f370 con 0x7f2fd006c160 2024-03-28T23:29:31.180 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.178+0000 7f2fced9d640 1 --2- 172.21.15.170:0/253205768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fd006c160 0x7f2fd006e720 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00708b0 tx=0x7f2fc802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e12f6aa65b4ed49 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:31.180 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.178+0000 7f2fcdd9b640 1 -- 172.21.15.170:0/253205768 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc8035020 con 0x7f2fd006c160 2024-03-28T23:29:31.180 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.178+0000 7f2fcdd9b640 1 -- 172.21.15.170:0/253205768 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fc8031870 con 0x7f2fd006c160 2024-03-28T23:29:31.180 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.178+0000 7f2fcdd9b640 1 -- 172.21.15.170:0/253205768 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc8031c40 con 0x7f2fd006c160 2024-03-28T23:29:31.181 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.179+0000 7f2fd50ae640 1 -- 172.21.15.170:0/253205768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fd006c160 msgr2=0x7f2fd006e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:31.181 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.179+0000 7f2fd50ae640 1 --2- 172.21.15.170:0/253205768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fd006c160 0x7f2fd006e720 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00708b0 tx=0x7f2fc802b910 comp rx=0 tx=0).stop 2024-03-28T23:29:31.181 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.179+0000 7f2fd50ae640 1 -- 172.21.15.170:0/253205768 shutdown_connections 2024-03-28T23:29:31.181 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.179+0000 7f2fd50ae640 1 --2- 172.21.15.170:0/253205768 >> v2:172.21.15.161:3300/0 conn(0x7f2fd006c540 0x7f2fd00c1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.181 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.179+0000 7f2fd50ae640 1 --2- 172.21.15.170:0/253205768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fd006c160 0x7f2fd006e720 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.181 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.179+0000 7f2fd50ae640 1 -- 172.21.15.170:0/253205768 >> 172.21.15.170:0/253205768 conn(0x7f2fd00c3be0 msgr2=0x7f2fd00c3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:31.182 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.179+0000 7f2fd50ae640 1 -- 172.21.15.170:0/253205768 shutdown_connections 2024-03-28T23:29:31.182 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.179+0000 7f2fd50ae640 1 -- 172.21.15.170:0/253205768 wait complete. 2024-03-28T23:29:31.182 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.180+0000 7f2fd50ae640 1 Processor -- start 2024-03-28T23:29:31.182 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.180+0000 7f2fd50ae640 1 -- start start 2024-03-28T23:29:31.184 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.181+0000 7f2fd50ae640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fd006c160 0x7f2fd01ec880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:31.184 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.181+0000 7f2fd50ae640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fd006c540 0x7f2fd01def90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:31.184 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.181+0000 7f2fd50ae640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fd0153bb0 0x7f2fd01e87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:31.184 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.181+0000 7f2fd50ae640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd0154040 con 0x7f2fd006c160 2024-03-28T23:29:31.184 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.181+0000 7f2fd50ae640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd006df20 con 0x7f2fd006c540 2024-03-28T23:29:31.184 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.181+0000 7f2fd50ae640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2fd00c2390 con 0x7f2fd0153bb0 2024-03-28T23:29:31.184 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.181+0000 7f2fce59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fd006c540 0x7f2fd01def90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:31.184 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.181+0000 7f2fce59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fd006c540 0x7f2fd01def90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:55842/0 (socket says 172.21.15.170:55842) 2024-03-28T23:29:31.184 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.181+0000 7f2fcf59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fd0153bb0 0x7f2fd01e87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:31.185 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.181+0000 7f2fce59c640 1 -- 172.21.15.170:0/2433644919 learned_addr learned my addr 172.21.15.170:0/2433644919 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T23:29:31.185 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.181+0000 7f2fced9d640 1 --2- 172.21.15.170:0/2433644919 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fd006c160 0x7f2fd01ec880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:31.185 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.181+0000 7f2fce59c640 1 -- 172.21.15.170:0/2433644919 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fd0153bb0 msgr2=0x7f2fd01e87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:31.185 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.181+0000 7f2fce59c640 1 --2- 172.21.15.170:0/2433644919 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fd0153bb0 0x7f2fd01e87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.185 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.181+0000 7f2fce59c640 1 -- 172.21.15.170:0/2433644919 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fd006c160 msgr2=0x7f2fd01ec880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:31.185 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.181+0000 7f2fce59c640 1 --2- 172.21.15.170:0/2433644919 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fd006c160 0x7f2fd01ec880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.185 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.181+0000 7f2fce59c640 1 -- 172.21.15.170:0/2433644919 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fd00bd540 con 0x7f2fd006c540 2024-03-28T23:29:31.185 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.181+0000 7f2fce59c640 1 --2- 172.21.15.170:0/2433644919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fd006c540 0x7f2fd01def90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2fc400d800 tx=0x7f2fc400dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:31.185 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.182+0000 7f2fb77fe640 1 -- 172.21.15.170:0/2433644919 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc400b6c0 con 0x7f2fd006c540 2024-03-28T23:29:31.185 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.182+0000 7f2fb77fe640 1 -- 172.21.15.170:0/2433644919 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fc4019070 con 0x7f2fd006c540 2024-03-28T23:29:31.185 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.182+0000 7f2fb77fe640 1 -- 172.21.15.170:0/2433644919 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc400bac0 con 0x7f2fd006c540 2024-03-28T23:29:31.185 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.182+0000 7f2fd50ae640 1 -- 172.21.15.170:0/2433644919 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fd01dfaf0 con 0x7f2fd006c540 2024-03-28T23:29:31.185 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.182+0000 7f2fd50ae640 1 -- 172.21.15.170:0/2433644919 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fd01e00b0 con 0x7f2fd006c540 2024-03-28T23:29:31.185 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.182+0000 7f2fb77fe640 1 -- 172.21.15.170:0/2433644919 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2fc400bc60 con 0x7f2fd006c540 2024-03-28T23:29:31.186 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.182+0000 7f2fb77fe640 1 -- 172.21.15.170:0/2433644919 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f2fc4014b60 con 0x7f2fd006c540 2024-03-28T23:29:31.186 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.183+0000 7f2fd50ae640 1 -- 172.21.15.170:0/2433644919 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fd004f4b0 con 0x7f2fd006c540 2024-03-28T23:29:31.186 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.185+0000 7f2fb77fe640 1 -- 172.21.15.170:0/2433644919 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2fc4014e00 con 0x7f2fd006c540 2024-03-28T23:29:31.210 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:31.209+0000 7f772ca2c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T23:29:31.240 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:31.239+0000 7f637f216040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T23:29:31.258 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.258+0000 7f2fd50ae640 1 -- 172.21.15.170:0/2433644919 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9e6e8b34-9805-4653-944f-e8b07dbcfadb", "id": 6} v 0) -- 0x7f2fd00c1150 con 0x7f2fd006c540 2024-03-28T23:29:31.263 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.263+0000 7f2fb77fe640 1 -- 172.21.15.170:0/2433644919 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9e6e8b34-9805-4653-944f-e8b07dbcfadb", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f2fc40146a0 con 0x7f2fd006c540 2024-03-28T23:29:31.264 INFO:teuthology.orchestra.run.smithi170.stdout:6 2024-03-28T23:29:31.265 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.264+0000 7f2fd50ae640 1 -- 172.21.15.170:0/2433644919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fd006c540 msgr2=0x7f2fd01def90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:31.265 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.264+0000 7f2fd50ae640 1 --2- 172.21.15.170:0/2433644919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fd006c540 0x7f2fd01def90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2fc400d800 tx=0x7f2fc400dcc0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.265 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.264+0000 7f2fd50ae640 1 -- 172.21.15.170:0/2433644919 shutdown_connections 2024-03-28T23:29:31.265 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.264+0000 7f2fd50ae640 1 --2- 172.21.15.170:0/2433644919 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fd0153bb0 0x7f2fd01e87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.265 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.264+0000 7f2fd50ae640 1 --2- 172.21.15.170:0/2433644919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fd006c540 0x7f2fd01def90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.265 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.264+0000 7f2fd50ae640 1 --2- 172.21.15.170:0/2433644919 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fd006c160 0x7f2fd01ec880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.265 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.264+0000 7f2fd50ae640 1 -- 172.21.15.170:0/2433644919 >> 172.21.15.170:0/2433644919 conn(0x7f2fd00c3be0 msgr2=0x7f2fd00c4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:31.266 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.264+0000 7f2fd50ae640 1 -- 172.21.15.170:0/2433644919 shutdown_connections 2024-03-28T23:29:31.266 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.264+0000 7f2fd50ae640 1 -- 172.21.15.170:0/2433644919 wait complete. 2024-03-28T23:29:31.275 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new fd44cf2b-243a-4e71-9d85-bf2fdb1adc39 7 2024-03-28T23:29:31.352 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:31.351+0000 7f772ca2c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T23:29:31.364 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:31.363+0000 7f637f216040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T23:29:31.376 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.376+0000 7f5cdd948640 1 Processor -- start 2024-03-28T23:29:31.377 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.376+0000 7f5cdd948640 1 -- start start 2024-03-28T23:29:31.379 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.376+0000 7f5cdd948640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5cd80c4e40 0x7f5cd80c8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:31.379 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.376+0000 7f5cdd948640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cd8156060 con 0x7f5cd8158860 2024-03-28T23:29:31.379 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.376+0000 7f5cdd948640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cd8072870 con 0x7f5cd8072fe0 2024-03-28T23:29:31.379 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.376+0000 7f5cdd948640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5cd8072b50 con 0x7f5cd80c4e40 2024-03-28T23:29:31.379 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.376+0000 7f5cd67fc640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f5cd8072fe0 0x7f5cd80c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:53978/0 (socket says 172.21.15.170:53978) 2024-03-28T23:29:31.379 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.376+0000 7f5cd67fc640 1 -- 172.21.15.170:0/594985426 learned_addr learned my addr 172.21.15.170:0/594985426 (peer_addr_for_me v1:172.21.15.170:0/0) 2024-03-28T23:29:31.379 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.376+0000 7f5cd6ffd640 1 --2- 172.21.15.170:0/594985426 >> v2:172.21.15.161:3300/0 conn(0x7f5cd80c4e40 0x7f5cd80c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:31.379 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.377+0000 7f5cd5ffb640 1 -- 172.21.15.170:0/594985426 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 296370541 0 0) 0x7f5cd8072870 con 0x7f5cd8072fe0 2024-03-28T23:29:31.379 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.377+0000 7f5cd5ffb640 1 -- 172.21.15.170:0/594985426 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cc4003520 con 0x7f5cd8072fe0 2024-03-28T23:29:31.379 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.377+0000 7f5cd5ffb640 1 -- 172.21.15.170:0/594985426 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1587699488 0 0) 0x7f5cd8156060 con 0x7f5cd8158860 2024-03-28T23:29:31.379 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.377+0000 7f5cd5ffb640 1 -- 172.21.15.170:0/594985426 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cd8072870 con 0x7f5cd8158860 2024-03-28T23:29:31.379 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.377+0000 7f5cd6ffd640 1 -- 172.21.15.170:0/594985426 >> v1:172.21.15.170:6789/0 conn(0x7f5cd8072fe0 legacy=0x7f5cd80c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:31.379 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.377+0000 7f5cd5ffb640 1 -- 172.21.15.170:0/594985426 <== mon.1 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4001712500 0 0) 0x7f5cc4003520 con 0x7f5cd8072fe0 2024-03-28T23:29:31.380 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.377+0000 7f5cd6ffd640 1 -- 172.21.15.170:0/594985426 >> v1:172.21.15.161:6789/0 conn(0x7f5cd8158860 legacy=0x7f5cd8158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:31.380 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.377+0000 7f5cd6ffd640 1 -- 172.21.15.170:0/594985426 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cd8155cf0 con 0x7f5cd80c4e40 2024-03-28T23:29:31.380 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.378+0000 7f5cd6ffd640 1 --2- 172.21.15.170:0/594985426 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cd80c4e40 0x7f5cd80c8340 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0009720 tx=0x7f5cc002b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dd62b66b820ae38 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:31.380 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.378+0000 7f5cd5ffb640 1 -- 172.21.15.170:0/594985426 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc0038070 con 0x7f5cd80c4e40 2024-03-28T23:29:31.380 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.378+0000 7f5cd5ffb640 1 -- 172.21.15.170:0/594985426 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cc0031520 con 0x7f5cd80c4e40 2024-03-28T23:29:31.380 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.378+0000 7f5cd5ffb640 1 -- 172.21.15.170:0/594985426 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc8001670 con 0x7f5cd80c4e40 2024-03-28T23:29:31.383 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.378+0000 7f5cdd948640 1 -- 172.21.15.170:0/594985426 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cd80c4e40 msgr2=0x7f5cd80c8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:31.383 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.378+0000 7f5cdd948640 1 --2- 172.21.15.170:0/594985426 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cd80c4e40 0x7f5cd80c8340 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0009720 tx=0x7f5cc002b7c0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.383 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.378+0000 7f5cdd948640 1 -- 172.21.15.170:0/594985426 shutdown_connections 2024-03-28T23:29:31.383 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.378+0000 7f5cdd948640 1 --2- 172.21.15.170:0/594985426 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cd80c4e40 0x7f5cd80c8340 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.383 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.378+0000 7f5cdd948640 1 -- 172.21.15.170:0/594985426 >> 172.21.15.170:0/594985426 conn(0x7f5cd80c88b0 msgr2=0x7f5cd80c8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:31.383 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.378+0000 7f5cdd948640 1 -- 172.21.15.170:0/594985426 shutdown_connections 2024-03-28T23:29:31.383 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.379+0000 7f5cdd948640 1 -- 172.21.15.170:0/594985426 wait complete. 2024-03-28T23:29:31.384 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.380+0000 7f5cdd948640 1 Processor -- start 2024-03-28T23:29:31.384 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.380+0000 7f5cdd948640 1 -- start start 2024-03-28T23:29:31.384 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.380+0000 7f5cdd948640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cd8072fe0 0x7f5cd80a9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:31.384 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.380+0000 7f5cdd948640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cd80c4e40 0x7f5cd81467d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:31.385 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.380+0000 7f5cdd948640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cd8158860 0x7f5cd8146d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:31.385 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.380+0000 7f5cdd948640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd80c6eb0 con 0x7f5cd80c4e40 2024-03-28T23:29:31.385 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.380+0000 7f5cdd948640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd80c6d30 con 0x7f5cd8158860 2024-03-28T23:29:31.385 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.380+0000 7f5cdd948640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5cd8160c00 con 0x7f5cd8072fe0 2024-03-28T23:29:31.385 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.380+0000 7f5cd6ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cd8072fe0 0x7f5cd80a9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:31.385 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.380+0000 7f5cd77fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cd8158860 0x7f5cd8146d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:31.385 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.380+0000 7f5cd77fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cd8158860 0x7f5cd8146d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:55846/0 (socket says 172.21.15.170:55846) 2024-03-28T23:29:31.385 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.381+0000 7f5cd77fe640 1 -- 172.21.15.170:0/2630327845 learned_addr learned my addr 172.21.15.170:0/2630327845 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-28T23:29:31.385 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.381+0000 7f5cd67fc640 1 --2- 172.21.15.170:0/2630327845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cd80c4e40 0x7f5cd81467d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:31.385 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.381+0000 7f5cd77fe640 1 -- 172.21.15.170:0/2630327845 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cd8072fe0 msgr2=0x7f5cd80a9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:31.385 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.381+0000 7f5cd77fe640 1 --2- 172.21.15.170:0/2630327845 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cd8072fe0 0x7f5cd80a9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.385 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.381+0000 7f5cd77fe640 1 -- 172.21.15.170:0/2630327845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cd80c4e40 msgr2=0x7f5cd81467d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:31.385 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.381+0000 7f5cd77fe640 1 --2- 172.21.15.170:0/2630327845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cd80c4e40 0x7f5cd81467d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.385 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.381+0000 7f5cd77fe640 1 -- 172.21.15.170:0/2630327845 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cd81eebd0 con 0x7f5cd8158860 2024-03-28T23:29:31.386 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.381+0000 7f5cd6ffd640 1 --2- 172.21.15.170:0/2630327845 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cd8072fe0 0x7f5cd80a9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:29:31.386 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.381+0000 7f5cd67fc640 1 --2- 172.21.15.170:0/2630327845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cd80c4e40 0x7f5cd81467d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:29:31.386 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.381+0000 7f5cd77fe640 1 --2- 172.21.15.170:0/2630327845 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cd8158860 0x7f5cd8146d10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5cd000c790 tx=0x7f5cd000cc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:31.386 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.381+0000 7f5ccffff640 1 -- 172.21.15.170:0/2630327845 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd0003b70 con 0x7f5cd8158860 2024-03-28T23:29:31.386 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.381+0000 7f5ccffff640 1 -- 172.21.15.170:0/2630327845 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cd0003d10 con 0x7f5cd8158860 2024-03-28T23:29:31.386 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.381+0000 7f5cdd948640 1 -- 172.21.15.170:0/2630327845 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cd8145fd0 con 0x7f5cd8158860 2024-03-28T23:29:31.386 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.381+0000 7f5ccffff640 1 -- 172.21.15.170:0/2630327845 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd000aa50 con 0x7f5cd8158860 2024-03-28T23:29:31.386 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.381+0000 7f5cdd948640 1 -- 172.21.15.170:0/2630327845 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cd806d4d0 con 0x7f5cd8158860 2024-03-28T23:29:31.386 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.381+0000 7f5ccffff640 1 -- 172.21.15.170:0/2630327845 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5cd000abf0 con 0x7f5cd8158860 2024-03-28T23:29:31.386 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.381+0000 7f5ccffff640 1 -- 172.21.15.170:0/2630327845 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f5cd0012cb0 con 0x7f5cd8158860 2024-03-28T23:29:31.386 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.382+0000 7f5cdd948640 1 -- 172.21.15.170:0/2630327845 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cd804f4b0 con 0x7f5cd8158860 2024-03-28T23:29:31.387 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.386+0000 7f5ccffff640 1 -- 172.21.15.170:0/2630327845 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f5cd0012650 con 0x7f5cd8158860 2024-03-28T23:29:31.411 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:31.410+0000 7f637f216040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T23:29:31.459 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.459+0000 7f5cdd948640 1 -- 172.21.15.170:0/2630327845 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fd44cf2b-243a-4e71-9d85-bf2fdb1adc39", "id": 7} v 0) -- 0x7f5cd80a7f30 con 0x7f5cd8158860 2024-03-28T23:29:31.465 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.464+0000 7f5ccffff640 1 -- 172.21.15.170:0/2630327845 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fd44cf2b-243a-4e71-9d85-bf2fdb1adc39", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f5cd0012830 con 0x7f5cd8158860 2024-03-28T23:29:31.465 INFO:teuthology.orchestra.run.smithi170.stdout:7 2024-03-28T23:29:31.466 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.465+0000 7f5cdd948640 1 -- 172.21.15.170:0/2630327845 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cd8158860 msgr2=0x7f5cd8146d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:31.466 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.465+0000 7f5cdd948640 1 --2- 172.21.15.170:0/2630327845 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cd8158860 0x7f5cd8146d10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5cd000c790 tx=0x7f5cd000cc50 comp rx=0 tx=0).stop 2024-03-28T23:29:31.466 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.465+0000 7f5cdd948640 1 -- 172.21.15.170:0/2630327845 shutdown_connections 2024-03-28T23:29:31.466 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.465+0000 7f5cdd948640 1 --2- 172.21.15.170:0/2630327845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cd80c4e40 0x7f5cd81467d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.466 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.465+0000 7f5cdd948640 1 --2- 172.21.15.170:0/2630327845 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cd8158860 0x7f5cd8146d10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.466 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.465+0000 7f5cdd948640 1 --2- 172.21.15.170:0/2630327845 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cd8072fe0 0x7f5cd80a9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.467 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.465+0000 7f5cdd948640 1 -- 172.21.15.170:0/2630327845 >> 172.21.15.170:0/2630327845 conn(0x7f5cd80c88b0 msgr2=0x7f5cd80c5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:31.467 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.465+0000 7f5cdd948640 1 -- 172.21.15.170:0/2630327845 shutdown_connections 2024-03-28T23:29:31.467 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:29:31.465+0000 7f5cdd948640 1 -- 172.21.15.170:0/2630327845 wait complete. 2024-03-28T23:29:31.469 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:31.469+0000 7f772ca2c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T23:29:31.476 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-28T23:29:31.476 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-28T23:29:31.478 INFO:tasks.ceph.osd.0:Started 2024-03-28T23:29:31.479 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-28T23:29:31.479 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-28T23:29:31.481 INFO:tasks.ceph.osd.1:Started 2024-03-28T23:29:31.481 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-28T23:29:31.481 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-28T23:29:31.483 INFO:tasks.ceph.osd.2:Started 2024-03-28T23:29:31.483 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-28T23:29:31.483 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-28T23:29:31.485 INFO:tasks.ceph.osd.3:Started 2024-03-28T23:29:31.485 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-28T23:29:31.485 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-28T23:29:31.487 INFO:tasks.ceph.osd.4:Started 2024-03-28T23:29:31.488 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-28T23:29:31.488 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-28T23:29:31.490 INFO:tasks.ceph.osd.5:Started 2024-03-28T23:29:31.490 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-28T23:29:31.490 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-28T23:29:31.492 INFO:tasks.ceph.osd.6:Started 2024-03-28T23:29:31.492 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-28T23:29:31.492 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-28T23:29:31.494 INFO:tasks.ceph.osd.7:Started 2024-03-28T23:29:31.495 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:29:31.524 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:31.523+0000 7f772ca2c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T23:29:31.555 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:31.555+0000 7f637f216040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T23:29:31.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.615+0000 7f0f75b69640 1 Processor -- start 2024-03-28T23:29:31.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.615+0000 7f0f75b69640 1 -- start start 2024-03-28T23:29:31.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.616+0000 7f0f75b69640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f700c6940 0x7f0f700c6d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:31.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.616+0000 7f0f75b69640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0f701543f0 0x7f0f70153980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:31.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.616+0000 7f0f75b69640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0f70156e90 0x7f0f70150100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:31.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.616+0000 7f0f75b69640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0f700c7260 con 0x7f0f701543f0 2024-03-28T23:29:31.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.616+0000 7f0f75b69640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f0f70153ec0 con 0x7f0f70156e90 2024-03-28T23:29:31.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.616+0000 7f0f75b69640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0f700c3f50 con 0x7f0f700c6940 2024-03-28T23:29:31.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.616+0000 7f0f67fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0f701543f0 0x7f0f70153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:31.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.616+0000 7f0f67fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0f701543f0 0x7f0f70153980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:56294/0 (socket says 172.21.15.161:56294) 2024-03-28T23:29:31.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.616+0000 7f0f67fff640 1 -- 172.21.15.161:0/1062516225 learned_addr learned my addr 172.21.15.161:0/1062516225 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:31.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.616+0000 7f0f6ffff640 1 --2- 172.21.15.161:0/1062516225 >> v2:172.21.15.170:3300/0 conn(0x7f0f70156e90 0x7f0f70150100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:31.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.616+0000 7f0f6f7fe640 1 --2- 172.21.15.161:0/1062516225 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f700c6940 0x7f0f700c6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:31.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.616+0000 7f0f67fff640 1 -- 172.21.15.161:0/1062516225 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f700c6940 msgr2=0x7f0f700c6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:31.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.616+0000 7f0f67fff640 1 --2- 172.21.15.161:0/1062516225 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f700c6940 0x7f0f700c6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.616+0000 7f0f67fff640 1 -- 172.21.15.161:0/1062516225 >> v2:172.21.15.170:3300/0 conn(0x7f0f70156e90 msgr2=0x7f0f70150100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:31.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.616+0000 7f0f67fff640 1 --2- 172.21.15.161:0/1062516225 >> v2:172.21.15.170:3300/0 conn(0x7f0f70156e90 0x7f0f70150100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.616+0000 7f0f67fff640 1 -- 172.21.15.161:0/1062516225 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f700c3540 con 0x7f0f701543f0 2024-03-28T23:29:31.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.617+0000 7f0f67fff640 1 --2- 172.21.15.161:0/1062516225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f701543f0 0x7f0f70153980 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0f7006f5e0 tx=0x7f0f5802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a007d179c2766af6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:31.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.617+0000 7f0f6effd640 1 -- 172.21.15.161:0/1062516225 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f5802e070 con 0x7f0f701543f0 2024-03-28T23:29:31.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.617+0000 7f0f6effd640 1 -- 172.21.15.161:0/1062516225 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f58037070 con 0x7f0f701543f0 2024-03-28T23:29:31.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.617+0000 7f0f6effd640 1 -- 172.21.15.161:0/1062516225 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f58031a00 con 0x7f0f701543f0 2024-03-28T23:29:31.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.618+0000 7f0f75b69640 1 -- 172.21.15.161:0/1062516225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f701543f0 msgr2=0x7f0f70153980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:31.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.618+0000 7f0f75b69640 1 --2- 172.21.15.161:0/1062516225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f701543f0 0x7f0f70153980 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0f7006f5e0 tx=0x7f0f5802b880 comp rx=0 tx=0).stop 2024-03-28T23:29:31.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.618+0000 7f0f75b69640 1 -- 172.21.15.161:0/1062516225 shutdown_connections 2024-03-28T23:29:31.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.618+0000 7f0f75b69640 1 --2- 172.21.15.161:0/1062516225 >> v2:172.21.15.170:3300/0 conn(0x7f0f70156e90 0x7f0f70150100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.618+0000 7f0f75b69640 1 --2- 172.21.15.161:0/1062516225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f701543f0 0x7f0f70153980 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.618+0000 7f0f75b69640 1 --2- 172.21.15.161:0/1062516225 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f700c6940 0x7f0f700c6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.618+0000 7f0f75b69640 1 -- 172.21.15.161:0/1062516225 >> 172.21.15.161:0/1062516225 conn(0x7f0f700c42c0 msgr2=0x7f0f700c46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:31.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.618+0000 7f0f75b69640 1 -- 172.21.15.161:0/1062516225 shutdown_connections 2024-03-28T23:29:31.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.618+0000 7f0f75b69640 1 -- 172.21.15.161:0/1062516225 wait complete. 2024-03-28T23:29:31.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.619+0000 7f0f75b69640 1 Processor -- start 2024-03-28T23:29:31.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.619+0000 7f0f75b69640 1 -- start start 2024-03-28T23:29:31.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.620+0000 7f0f75b69640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f700c6940 0x7f0f700ab940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:31.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.620+0000 7f0f75b69640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f701543f0 0x7f0f700c1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:31.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.620+0000 7f0f75b69640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f70156e90 0x7f0f700c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:31.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.620+0000 7f0f75b69640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0f70156970 con 0x7f0f70156e90 2024-03-28T23:29:31.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.620+0000 7f0f75b69640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0f70156690 con 0x7f0f701543f0 2024-03-28T23:29:31.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.620+0000 7f0f75b69640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0f70154850 con 0x7f0f700c6940 2024-03-28T23:29:31.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.620+0000 7f0f6ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f70156e90 0x7f0f700c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:31.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.620+0000 7f0f6f7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f700c6940 0x7f0f700ab940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:31.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.620+0000 7f0f6ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f70156e90 0x7f0f700c18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:56310/0 (socket says 172.21.15.161:56310) 2024-03-28T23:29:31.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.620+0000 7f0f6ffff640 1 -- 172.21.15.161:0/417676098 learned_addr learned my addr 172.21.15.161:0/417676098 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:31.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.620+0000 7f0f67fff640 1 --2- 172.21.15.161:0/417676098 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f701543f0 0x7f0f700c1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:31.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.620+0000 7f0f6f7fe640 1 -- 172.21.15.161:0/417676098 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f701543f0 msgr2=0x7f0f700c1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:31.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.620+0000 7f0f6f7fe640 1 --2- 172.21.15.161:0/417676098 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f701543f0 0x7f0f700c1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.620+0000 7f0f6f7fe640 1 -- 172.21.15.161:0/417676098 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f70156e90 msgr2=0x7f0f700c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:31.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.620+0000 7f0f6f7fe640 1 --2- 172.21.15.161:0/417676098 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f70156e90 0x7f0f700c18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.620+0000 7f0f6f7fe640 1 -- 172.21.15.161:0/417676098 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f700c1de0 con 0x7f0f700c6940 2024-03-28T23:29:31.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.620+0000 7f0f6ffff640 1 --2- 172.21.15.161:0/417676098 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f70156e90 0x7f0f700c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:29:31.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.620+0000 7f0f6f7fe640 1 --2- 172.21.15.161:0/417676098 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f700c6940 0x7f0f700ab940 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0f6000e870 tx=0x7f0f6000ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:31.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.621+0000 7f0f6cff9640 1 -- 172.21.15.161:0/417676098 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f60017070 con 0x7f0f700c6940 2024-03-28T23:29:31.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.621+0000 7f0f6cff9640 1 -- 172.21.15.161:0/417676098 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f6000c940 con 0x7f0f700c6940 2024-03-28T23:29:31.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.621+0000 7f0f75b69640 1 -- 172.21.15.161:0/417676098 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f701ec200 con 0x7f0f700c6940 2024-03-28T23:29:31.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.621+0000 7f0f6cff9640 1 -- 172.21.15.161:0/417676098 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f6000cd50 con 0x7f0f700c6940 2024-03-28T23:29:31.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.621+0000 7f0f75b69640 1 -- 172.21.15.161:0/417676098 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f701ec7c0 con 0x7f0f700c6940 2024-03-28T23:29:31.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.621+0000 7f0f6cff9640 1 -- 172.21.15.161:0/417676098 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0f6001a070 con 0x7f0f700c6940 2024-03-28T23:29:31.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.621+0000 7f0f6cff9640 1 -- 172.21.15.161:0/417676098 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f0f6001cbf0 con 0x7f0f700c6940 2024-03-28T23:29:31.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.621+0000 7f0f75b69640 1 -- 172.21.15.161:0/417676098 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f7004f4b0 con 0x7f0f700c6940 2024-03-28T23:29:31.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.624+0000 7f0f6cff9640 1 -- 172.21.15.161:0/417676098 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f0f6001c4e0 con 0x7f0f700c6940 2024-03-28T23:29:31.654 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:31.653+0000 7f772ca2c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T23:29:31.662 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:31.661+0000 7f637f216040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T23:29:31.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.684+0000 7f0f75b69640 1 -- 172.21.15.161:0/417676098 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0f7006f600 con 0x7f0f700c6940 2024-03-28T23:29:31.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.685+0000 7f0f6cff9640 1 -- 172.21.15.161:0/417676098 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f0f6001c6c0 con 0x7f0f700c6940 2024-03-28T23:29:31.686 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:29:31.686 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":10,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:29:31.460946+0000","last_up_change":"0.000000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:29:31.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.686+0000 7f0f75b69640 1 -- 172.21.15.161:0/417676098 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f700c6940 msgr2=0x7f0f700ab940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:31.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.686+0000 7f0f75b69640 1 --2- 172.21.15.161:0/417676098 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f700c6940 0x7f0f700ab940 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0f6000e870 tx=0x7f0f6000ec70 comp rx=0 tx=0).stop 2024-03-28T23:29:31.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.686+0000 7f0f75b69640 1 -- 172.21.15.161:0/417676098 shutdown_connections 2024-03-28T23:29:31.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.686+0000 7f0f75b69640 1 --2- 172.21.15.161:0/417676098 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f70156e90 0x7f0f700c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.686+0000 7f0f75b69640 1 --2- 172.21.15.161:0/417676098 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f701543f0 0x7f0f700c1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.686+0000 7f0f75b69640 1 --2- 172.21.15.161:0/417676098 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f700c6940 0x7f0f700ab940 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:31.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.686+0000 7f0f75b69640 1 -- 172.21.15.161:0/417676098 >> 172.21.15.161:0/417676098 conn(0x7f0f700c42c0 msgr2=0x7f0f7014ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:31.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.686+0000 7f0f75b69640 1 -- 172.21.15.161:0/417676098 shutdown_connections 2024-03-28T23:29:31.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:31.686+0000 7f0f75b69640 1 -- 172.21.15.161:0/417676098 wait complete. 2024-03-28T23:29:31.698 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-28T23:29:31.698 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-28T23:29:31.751 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:31.750+0000 7f772ca2c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T23:29:31.821 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:31.820+0000 7f637f216040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T23:29:31.864 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:31.863+0000 7f637f216040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T23:29:31.906 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:31.905+0000 7f772ca2c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T23:29:31.948 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:31.947+0000 7f772ca2c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T23:29:32.020 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:32.019+0000 7f637f216040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T23:29:32.081 INFO:tasks.ceph.osd.4.smithi170.stderr:2024-03-28T23:29:32.080+0000 7f237560b640 -1 Falling back to public interface 2024-03-28T23:29:32.095 INFO:tasks.ceph.osd.1.smithi161.stderr:2024-03-28T23:29:32.094+0000 7f2cdd2b5640 -1 Falling back to public interface 2024-03-28T23:29:32.098 INFO:tasks.ceph.osd.6.smithi170.stderr:2024-03-28T23:29:32.097+0000 7f475f52a640 -1 Falling back to public interface 2024-03-28T23:29:32.102 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:32.101+0000 7f772ca2c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T23:29:32.104 INFO:tasks.ceph.osd.2.smithi161.stderr:2024-03-28T23:29:32.103+0000 7f29fe711640 -1 Falling back to public interface 2024-03-28T23:29:32.108 INFO:tasks.ceph.osd.0.smithi161.stderr:2024-03-28T23:29:32.107+0000 7f950f145640 -1 Falling back to public interface 2024-03-28T23:29:32.117 INFO:tasks.ceph.osd.3.smithi161.stderr:2024-03-28T23:29:32.116+0000 7fe739848640 -1 Falling back to public interface 2024-03-28T23:29:32.119 INFO:tasks.ceph.osd.7.smithi170.stderr:2024-03-28T23:29:32.118+0000 7ff91660f640 -1 Falling back to public interface 2024-03-28T23:29:32.122 INFO:tasks.ceph.osd.5.smithi170.stderr:2024-03-28T23:29:32.121+0000 7f7fd7848640 -1 Falling back to public interface 2024-03-28T23:29:32.504 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-28T23:29:32.596 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.595+0000 7f1576bc9640 1 Processor -- start 2024-03-28T23:29:32.596 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.595+0000 7f1576bc9640 1 -- start start 2024-03-28T23:29:32.598 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.596+0000 7f1576bc9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15700734b0 0x7f1570072890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:32.598 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.596+0000 7f1576bc9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f157006f850 con 0x7f15701529d0 2024-03-28T23:29:32.598 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.596+0000 7f1576bc9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1570155cf0 con 0x7f15700702f0 2024-03-28T23:29:32.598 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.596+0000 7f1576bc9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f15701562b0 con 0x7f15700734b0 2024-03-28T23:29:32.599 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.596+0000 7f1574965640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f15701529d0 0x7f15700c8710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:58908/0 (socket says 172.21.15.161:58908) 2024-03-28T23:29:32.599 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.596+0000 7f1574965640 1 -- 172.21.15.161:0/92799382 learned_addr learned my addr 172.21.15.161:0/92799382 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:32.599 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.596+0000 7f156ffff640 1 --2- 172.21.15.161:0/92799382 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15700734b0 0x7f1570072890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:32.599 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.596+0000 7f156f7fe640 1 -- 172.21.15.161:0/92799382 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2037414580 0 0) 0x7f157006f850 con 0x7f15701529d0 2024-03-28T23:29:32.599 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.596+0000 7f156f7fe640 1 -- 172.21.15.161:0/92799382 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1560003520 con 0x7f15701529d0 2024-03-28T23:29:32.599 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.596+0000 7f156f7fe640 1 -- 172.21.15.161:0/92799382 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4286565590 0 0) 0x7f1570155cf0 con 0x7f15700702f0 2024-03-28T23:29:32.599 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.597+0000 7f156f7fe640 1 -- 172.21.15.161:0/92799382 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f157006f850 con 0x7f15700702f0 2024-03-28T23:29:32.599 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.597+0000 7f156ffff640 1 -- 172.21.15.161:0/92799382 >> v1:172.21.15.170:6789/0 conn(0x7f15700702f0 legacy=0x7f1570070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:32.599 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.597+0000 7f156ffff640 1 -- 172.21.15.161:0/92799382 >> v1:172.21.15.161:6789/0 conn(0x7f15701529d0 legacy=0x7f15700c8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:32.599 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.597+0000 7f156ffff640 1 -- 172.21.15.161:0/92799382 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1570155fd0 con 0x7f15700734b0 2024-03-28T23:29:32.599 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.597+0000 7f156ffff640 1 --2- 172.21.15.161:0/92799382 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15700734b0 0x7f1570072890 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f155c0097d0 tx=0x7f155c02b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70f1edb8f2961e93 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:32.599 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.597+0000 7f156f7fe640 1 -- 172.21.15.161:0/92799382 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c031770 con 0x7f15700734b0 2024-03-28T23:29:32.599 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.597+0000 7f156f7fe640 1 -- 172.21.15.161:0/92799382 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f155c037070 con 0x7f15700734b0 2024-03-28T23:29:32.599 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.597+0000 7f156f7fe640 1 -- 172.21.15.161:0/92799382 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c031b30 con 0x7f15700734b0 2024-03-28T23:29:32.603 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.598+0000 7f1576bc9640 1 -- 172.21.15.161:0/92799382 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15700734b0 msgr2=0x7f1570072890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:32.603 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.598+0000 7f1576bc9640 1 --2- 172.21.15.161:0/92799382 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15700734b0 0x7f1570072890 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f155c0097d0 tx=0x7f155c02b810 comp rx=0 tx=0).stop 2024-03-28T23:29:32.603 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.598+0000 7f1576bc9640 1 -- 172.21.15.161:0/92799382 shutdown_connections 2024-03-28T23:29:32.603 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.598+0000 7f1576bc9640 1 --2- 172.21.15.161:0/92799382 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15700734b0 0x7f1570072890 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:32.603 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.598+0000 7f1576bc9640 1 -- 172.21.15.161:0/92799382 >> 172.21.15.161:0/92799382 conn(0x7f15700a9860 msgr2=0x7f157006b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:32.603 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.598+0000 7f1576bc9640 1 -- 172.21.15.161:0/92799382 shutdown_connections 2024-03-28T23:29:32.603 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.598+0000 7f1576bc9640 1 -- 172.21.15.161:0/92799382 wait complete. 2024-03-28T23:29:32.603 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.599+0000 7f1576bc9640 1 Processor -- start 2024-03-28T23:29:32.604 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.599+0000 7f1576bc9640 1 -- start start 2024-03-28T23:29:32.604 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.600+0000 7f1576bc9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15700702f0 0x7f15701543d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:32.604 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.600+0000 7f1576bc9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15700734b0 0x7f157014c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:32.604 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.600+0000 7f1576bc9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15701529d0 0x7f157014cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:32.604 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.600+0000 7f1576bc9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f15700c8e20 con 0x7f15700734b0 2024-03-28T23:29:32.604 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.600+0000 7f1576bc9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f157006fe60 con 0x7f15701529d0 2024-03-28T23:29:32.604 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.600+0000 7f1576bc9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f15700c8fa0 con 0x7f15700702f0 2024-03-28T23:29:32.604 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.600+0000 7f156ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15700734b0 0x7f157014c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:32.604 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.600+0000 7f156ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15700734b0 0x7f157014c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:56318/0 (socket says 172.21.15.161:56318) 2024-03-28T23:29:32.604 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.600+0000 7f156ffff640 1 -- 172.21.15.161:0/815091912 learned_addr learned my addr 172.21.15.161:0/815091912 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:32.604 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.600+0000 7f1575166640 1 --2- 172.21.15.161:0/815091912 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15701529d0 0x7f157014cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:32.604 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.600+0000 7f1574965640 1 --2- 172.21.15.161:0/815091912 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15700702f0 0x7f15701543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:32.604 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.601+0000 7f1575166640 1 -- 172.21.15.161:0/815091912 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15700702f0 msgr2=0x7f15701543d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:32.605 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.601+0000 7f1575166640 1 --2- 172.21.15.161:0/815091912 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15700702f0 0x7f15701543d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:32.605 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.601+0000 7f1575166640 1 -- 172.21.15.161:0/815091912 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15700734b0 msgr2=0x7f157014c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:32.605 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.601+0000 7f1575166640 1 --2- 172.21.15.161:0/815091912 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15700734b0 0x7f157014c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:32.605 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.601+0000 7f1575166640 1 -- 172.21.15.161:0/815091912 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f157014d0e0 con 0x7f15701529d0 2024-03-28T23:29:32.605 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.601+0000 7f156ffff640 1 --2- 172.21.15.161:0/815091912 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15700734b0 0x7f157014c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:29:32.605 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.601+0000 7f1575166640 1 --2- 172.21.15.161:0/815091912 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15701529d0 0x7f157014cba0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f156800c6c0 tx=0x7f156800cb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:32.605 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.601+0000 7f156d7fa640 1 -- 172.21.15.161:0/815091912 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1568003b70 con 0x7f15701529d0 2024-03-28T23:29:32.605 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.601+0000 7f156d7fa640 1 -- 172.21.15.161:0/815091912 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1568003d10 con 0x7f15701529d0 2024-03-28T23:29:32.605 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.601+0000 7f1576bc9640 1 -- 172.21.15.161:0/815091912 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15701ebba0 con 0x7f15701529d0 2024-03-28T23:29:32.605 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.601+0000 7f156d7fa640 1 -- 172.21.15.161:0/815091912 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1568005310 con 0x7f15701529d0 2024-03-28T23:29:32.605 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.601+0000 7f1576bc9640 1 -- 172.21.15.161:0/815091912 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15700c8960 con 0x7f15701529d0 2024-03-28T23:29:32.605 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.601+0000 7f156d7fa640 1 -- 172.21.15.161:0/815091912 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f15680054b0 con 0x7f15701529d0 2024-03-28T23:29:32.605 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.601+0000 7f156d7fa640 1 -- 172.21.15.161:0/815091912 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f15680057a0 con 0x7f15701529d0 2024-03-28T23:29:32.605 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.602+0000 7f1576bc9640 1 -- 172.21.15.161:0/815091912 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f157004f4b0 con 0x7f15701529d0 2024-03-28T23:29:32.606 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.604+0000 7f156d7fa640 1 -- 172.21.15.161:0/815091912 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f156800a640 con 0x7f15701529d0 2024-03-28T23:29:32.667 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.666+0000 7f1576bc9640 1 -- 172.21.15.161:0/815091912 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f15700c2af0 con 0x7f15701529d0 2024-03-28T23:29:32.667 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.667+0000 7f156d7fa640 1 -- 172.21.15.161:0/815091912 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f156800a820 con 0x7f15701529d0 2024-03-28T23:29:32.668 INFO:teuthology.misc.health.smithi161.stdout: 2024-03-28T23:29:32.668 INFO:teuthology.misc.health.smithi161.stdout:{"epoch":10,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:29:31.460946+0000","last_up_change":"0.000000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:29:32.669 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.668+0000 7f1576bc9640 1 -- 172.21.15.161:0/815091912 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15701529d0 msgr2=0x7f157014cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:32.669 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.668+0000 7f1576bc9640 1 --2- 172.21.15.161:0/815091912 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15701529d0 0x7f157014cba0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f156800c6c0 tx=0x7f156800cb80 comp rx=0 tx=0).stop 2024-03-28T23:29:32.669 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.668+0000 7f1576bc9640 1 -- 172.21.15.161:0/815091912 shutdown_connections 2024-03-28T23:29:32.669 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.668+0000 7f1576bc9640 1 --2- 172.21.15.161:0/815091912 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15700734b0 0x7f157014c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:32.669 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.668+0000 7f1576bc9640 1 --2- 172.21.15.161:0/815091912 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15701529d0 0x7f157014cba0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:32.669 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.668+0000 7f1576bc9640 1 --2- 172.21.15.161:0/815091912 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15700702f0 0x7f15701543d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:32.669 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.668+0000 7f1576bc9640 1 -- 172.21.15.161:0/815091912 >> 172.21.15.161:0/815091912 conn(0x7f15700a9860 msgr2=0x7f15701556c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:32.669 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.668+0000 7f1576bc9640 1 -- 172.21.15.161:0/815091912 shutdown_connections 2024-03-28T23:29:32.669 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:32.668+0000 7f1576bc9640 1 -- 172.21.15.161:0/815091912 wait complete. 2024-03-28T23:29:32.679 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-28T23:29:32.856 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:32.855+0000 7f637f216040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T23:29:32.890 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:32.890+0000 7f772ca2c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T23:29:33.134 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:33.132+0000 7f637f216040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T23:29:33.166 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:33.166+0000 7f772ca2c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T23:29:33.185 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:33.185+0000 7f637f216040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T23:29:33.219 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:33.219+0000 7f772ca2c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T23:29:33.301 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:33.301+0000 7f637f216040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T23:29:33.337 INFO:tasks.ceph.mgr.x.smithi170.stderr:2024-03-28T23:29:33.336+0000 7f772ca2c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T23:29:34.316 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:34.316+0000 7f631fb91640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-28T23:29:34.317 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-28T23:29:34.316+0000 7f631fb91640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-28T23:29:35.531 INFO:tasks.ceph.osd.4.smithi170.stderr:2024-03-28T23:29:35.530+0000 7f237560b640 -1 osd.4 0 log_to_monitors true 2024-03-28T23:29:35.579 INFO:tasks.ceph.osd.6.smithi170.stderr:2024-03-28T23:29:35.578+0000 7f475f52a640 -1 osd.6 0 log_to_monitors true 2024-03-28T23:29:35.596 INFO:tasks.ceph.osd.1.smithi161.stderr:2024-03-28T23:29:35.595+0000 7f2cdd2b5640 -1 osd.1 0 log_to_monitors true 2024-03-28T23:29:35.636 INFO:tasks.ceph.osd.2.smithi161.stderr:2024-03-28T23:29:35.635+0000 7f29fe711640 -1 osd.2 0 log_to_monitors true 2024-03-28T23:29:35.639 INFO:tasks.ceph.osd.3.smithi161.stderr:2024-03-28T23:29:35.638+0000 7fe739848640 -1 osd.3 0 log_to_monitors true 2024-03-28T23:29:35.641 INFO:tasks.ceph.osd.5.smithi170.stderr:2024-03-28T23:29:35.640+0000 7f7fd7848640 -1 osd.5 0 log_to_monitors true 2024-03-28T23:29:35.647 INFO:tasks.ceph.osd.7.smithi170.stderr:2024-03-28T23:29:35.646+0000 7ff91660f640 -1 osd.7 0 log_to_monitors true 2024-03-28T23:29:35.649 INFO:tasks.ceph.osd.0.smithi161.stderr:2024-03-28T23:29:35.648+0000 7f950f145640 -1 osd.0 0 log_to_monitors true 2024-03-28T23:29:38.227 INFO:tasks.ceph.osd.5.smithi170.stderr:2024-03-28T23:29:38.225+0000 7f7fd37de640 -1 osd.5 0 waiting for initial osdmap 2024-03-28T23:29:38.240 INFO:tasks.ceph.osd.5.smithi170.stderr:2024-03-28T23:29:38.239+0000 7f7fcb5fe640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T23:29:38.250 INFO:tasks.ceph.osd.4.smithi170.stderr:2024-03-28T23:29:38.249+0000 7f23715a1640 -1 osd.4 0 waiting for initial osdmap 2024-03-28T23:29:38.251 INFO:tasks.ceph.osd.7.smithi170.stderr:2024-03-28T23:29:38.251+0000 7ff911f3a640 -1 osd.7 0 waiting for initial osdmap 2024-03-28T23:29:38.259 INFO:tasks.ceph.osd.4.smithi170.stderr:2024-03-28T23:29:38.259+0000 7f23693c1640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T23:29:38.264 INFO:tasks.ceph.osd.7.smithi170.stderr:2024-03-28T23:29:38.264+0000 7ff909d5a640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T23:29:38.275 INFO:tasks.ceph.osd.6.smithi170.stderr:2024-03-28T23:29:38.275+0000 7f475b4c0640 -1 osd.6 0 waiting for initial osdmap 2024-03-28T23:29:38.289 INFO:tasks.ceph.osd.6.smithi170.stderr:2024-03-28T23:29:38.288+0000 7f47532e0640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T23:29:38.301 INFO:tasks.ceph.osd.1.smithi161.stderr:2024-03-28T23:29:38.300+0000 7f2cd9a5f640 -1 osd.1 0 waiting for initial osdmap 2024-03-28T23:29:38.316 INFO:tasks.ceph.osd.1.smithi161.stderr:2024-03-28T23:29:38.315+0000 7f2cd106b640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T23:29:38.326 INFO:tasks.ceph.osd.2.smithi161.stderr:2024-03-28T23:29:38.325+0000 7f29fa6a7640 -1 osd.2 0 waiting for initial osdmap 2024-03-28T23:29:38.334 INFO:tasks.ceph.osd.3.smithi161.stderr:2024-03-28T23:29:38.334+0000 7fe7357de640 -1 osd.3 0 waiting for initial osdmap 2024-03-28T23:29:38.338 INFO:tasks.ceph.osd.0.smithi161.stderr:2024-03-28T23:29:38.338+0000 7f950b0db640 -1 osd.0 0 waiting for initial osdmap 2024-03-28T23:29:38.341 INFO:tasks.ceph.osd.2.smithi161.stderr:2024-03-28T23:29:38.341+0000 7f29f24c7640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T23:29:38.342 INFO:tasks.ceph.osd.3.smithi161.stderr:2024-03-28T23:29:38.342+0000 7fe72d5fe640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T23:29:38.350 INFO:tasks.ceph.osd.0.smithi161.stderr:2024-03-28T23:29:38.350+0000 7f9502efb640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T23:29:39.486 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-28T23:29:39.586 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.586+0000 7f563a910640 1 Processor -- start 2024-03-28T23:29:39.587 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.586+0000 7f563a910640 1 -- start start 2024-03-28T23:29:39.588 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.586+0000 7f563a910640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f56341526c0 0x7f56340c4f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:39.588 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.586+0000 7f563a910640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5634155c50 con 0x7f56340c3580 2024-03-28T23:29:39.588 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.586+0000 7f563a910640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5634073660 con 0x7f56340c2d00 2024-03-28T23:29:39.588 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.586+0000 7f563a910640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5634155f50 con 0x7f56341526c0 2024-03-28T23:29:39.589 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.587+0000 7f5633fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f56341526c0 0x7f56340c4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:39.589 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.587+0000 7f5633fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f56341526c0 0x7f56340c4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:56424/0 (socket says 172.21.15.161:56424) 2024-03-28T23:29:39.589 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.587+0000 7f5633fff640 1 -- 172.21.15.161:0/863105277 learned_addr learned my addr 172.21.15.161:0/863105277 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:39.589 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.587+0000 7f5638ead640 1 --1- 172.21.15.161:0/863105277 >> v1:172.21.15.161:6789/0 conn(0x7f56340c3580 0x7f5634072870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:58910/0 (socket says 172.21.15.161:58910) 2024-03-28T23:29:39.589 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.587+0000 7f5633fff640 1 -- 172.21.15.161:0/863105277 >> v1:172.21.15.170:6789/0 conn(0x7f56340c2d00 legacy=0x7f563406ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:39.589 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.587+0000 7f5633fff640 1 -- 172.21.15.161:0/863105277 >> v1:172.21.15.161:6789/0 conn(0x7f56340c3580 legacy=0x7f5634072870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:39.589 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.587+0000 7f5632ffd640 1 -- 172.21.15.161:0/863105277 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2452177153 0 0) 0x7f5634155c50 con 0x7f56340c3580 2024-03-28T23:29:39.589 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.587+0000 7f5633fff640 1 -- 172.21.15.161:0/863105277 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56340a8210 con 0x7f56341526c0 2024-03-28T23:29:39.589 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.588+0000 7f5633fff640 1 --2- 172.21.15.161:0/863105277 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56341526c0 0x7f56340c4f50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5624009870 tx=0x7f562402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4248bac6609a99d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:39.589 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.588+0000 7f5632ffd640 1 -- 172.21.15.161:0/863105277 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5624034070 con 0x7f56341526c0 2024-03-28T23:29:39.589 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.588+0000 7f5632ffd640 1 -- 172.21.15.161:0/863105277 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5624030860 con 0x7f56341526c0 2024-03-28T23:29:39.589 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.588+0000 7f5632ffd640 1 -- 172.21.15.161:0/863105277 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5624030c30 con 0x7f56341526c0 2024-03-28T23:29:39.591 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.588+0000 7f563a910640 1 -- 172.21.15.161:0/863105277 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56341526c0 msgr2=0x7f56340c4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:39.591 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.588+0000 7f563a910640 1 --2- 172.21.15.161:0/863105277 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56341526c0 0x7f56340c4f50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5624009870 tx=0x7f562402b9a0 comp rx=0 tx=0).stop 2024-03-28T23:29:39.591 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.588+0000 7f563a910640 1 -- 172.21.15.161:0/863105277 shutdown_connections 2024-03-28T23:29:39.591 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.588+0000 7f563a910640 1 --2- 172.21.15.161:0/863105277 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56341526c0 0x7f56340c4f50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:39.591 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.588+0000 7f563a910640 1 -- 172.21.15.161:0/863105277 >> 172.21.15.161:0/863105277 conn(0x7f56340aaa10 msgr2=0x7f56340aadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:39.591 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.588+0000 7f563a910640 1 -- 172.21.15.161:0/863105277 shutdown_connections 2024-03-28T23:29:39.591 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.588+0000 7f563a910640 1 -- 172.21.15.161:0/863105277 wait complete. 2024-03-28T23:29:39.591 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.589+0000 7f563a910640 1 Processor -- start 2024-03-28T23:29:39.591 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.590+0000 7f563a910640 1 -- start start 2024-03-28T23:29:39.594 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.590+0000 7f563a910640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56340c2d00 0x7f56341ee680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:39.594 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.590+0000 7f563a910640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56340c3580 0x7f56341eaf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:39.594 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.590+0000 7f563a910640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56341526c0 0x7f56341512c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:39.594 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.590+0000 7f563a910640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5634152aa0 con 0x7f56340c3580 2024-03-28T23:29:39.594 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.590+0000 7f563a910640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f56340a84f0 con 0x7f56341526c0 2024-03-28T23:29:39.594 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.590+0000 7f563a910640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5634072f80 con 0x7f56340c2d00 2024-03-28T23:29:39.595 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.591+0000 7f56337fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56340c3580 0x7f56341eaf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:39.595 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.591+0000 7f56337fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56340c3580 0x7f56341eaf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:56426/0 (socket says 172.21.15.161:56426) 2024-03-28T23:29:39.595 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.591+0000 7f56337fe640 1 -- 172.21.15.161:0/3200583527 learned_addr learned my addr 172.21.15.161:0/3200583527 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:39.595 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.591+0000 7f5638ead640 1 --2- 172.21.15.161:0/3200583527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56341526c0 0x7f56341512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:39.595 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.591+0000 7f5633fff640 1 --2- 172.21.15.161:0/3200583527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56340c2d00 0x7f56341ee680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:39.595 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.591+0000 7f56337fe640 1 -- 172.21.15.161:0/3200583527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56340c2d00 msgr2=0x7f56341ee680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:39.595 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.591+0000 7f56337fe640 1 --2- 172.21.15.161:0/3200583527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56340c2d00 0x7f56341ee680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:39.595 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.591+0000 7f56337fe640 1 -- 172.21.15.161:0/3200583527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56341526c0 msgr2=0x7f56341512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:39.595 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.591+0000 7f56337fe640 1 --2- 172.21.15.161:0/3200583527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56341526c0 0x7f56341512c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:39.595 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.591+0000 7f56337fe640 1 -- 172.21.15.161:0/3200583527 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56341eb4d0 con 0x7f56340c3580 2024-03-28T23:29:39.595 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.591+0000 7f56337fe640 1 --2- 172.21.15.161:0/3200583527 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56340c3580 0x7f56341eaf90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f562800c3f0 tx=0x7f562800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:39.595 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.591+0000 7f5630ff9640 1 -- 172.21.15.161:0/3200583527 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5628007020 con 0x7f56340c3580 2024-03-28T23:29:39.595 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.591+0000 7f5630ff9640 1 -- 172.21.15.161:0/3200583527 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56280071c0 con 0x7f56340c3580 2024-03-28T23:29:39.595 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.591+0000 7f563a910640 1 -- 172.21.15.161:0/3200583527 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5634156cc0 con 0x7f56340c3580 2024-03-28T23:29:39.596 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.591+0000 7f5630ff9640 1 -- 172.21.15.161:0/3200583527 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5628007590 con 0x7f56340c3580 2024-03-28T23:29:39.596 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.591+0000 7f563a910640 1 -- 172.21.15.161:0/3200583527 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56341edba0 con 0x7f56340c3580 2024-03-28T23:29:39.596 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.592+0000 7f563a910640 1 -- 172.21.15.161:0/3200583527 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56340c3300 con 0x7f56340c3580 2024-03-28T23:29:39.596 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.593+0000 7f5630ff9640 1 -- 172.21.15.161:0/3200583527 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 4) ==== 89724+0+0 (secure 0 0 0) 0x7f5628019020 con 0x7f56340c3580 2024-03-28T23:29:39.596 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.593+0000 7f5630ff9640 1 --2- 172.21.15.161:0/3200583527 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f562006b470 0x7f562006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:39.596 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.593+0000 7f5630ff9640 1 -- 172.21.15.161:0/3200583527 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 4471+0+0 (secure 0 0 0) 0x7f5628089320 con 0x7f56340c3580 2024-03-28T23:29:39.596 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.593+0000 7f5633fff640 1 --2- 172.21.15.161:0/3200583527 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f562006b470 0x7f562006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:39.596 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.594+0000 7f5633fff640 1 --2- 172.21.15.161:0/3200583527 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f562006b470 0x7f562006d930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5624036000 tx=0x7f5624008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:39.598 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.598+0000 7f5630ff9640 1 -- 172.21.15.161:0/3200583527 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f56341eb4d0 con 0x7f56340c3580 2024-03-28T23:29:39.774 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.773+0000 7f563a910640 1 -- 172.21.15.161:0/3200583527 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f563404f4b0 con 0x7f56340c3580 2024-03-28T23:29:39.775 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.774+0000 7f5630ff9640 1 -- 172.21.15.161:0/3200583527 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+10945 (secure 0 0 0) 0x7f56280587e0 con 0x7f56340c3580 2024-03-28T23:29:39.775 INFO:teuthology.misc.health.smithi161.stdout: 2024-03-28T23:29:39.775 INFO:teuthology.misc.health.smithi161.stdout:{"epoch":13,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:29:39.231019+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:29:39.778 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.776+0000 7f563a910640 1 -- 172.21.15.161:0/3200583527 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f562006b470 msgr2=0x7f562006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:39.778 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.776+0000 7f563a910640 1 --2- 172.21.15.161:0/3200583527 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f562006b470 0x7f562006d930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5624036000 tx=0x7f5624008030 comp rx=0 tx=0).stop 2024-03-28T23:29:39.778 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.776+0000 7f563a910640 1 -- 172.21.15.161:0/3200583527 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56340c3580 msgr2=0x7f56341eaf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:39.778 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.776+0000 7f563a910640 1 --2- 172.21.15.161:0/3200583527 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56340c3580 0x7f56341eaf90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f562800c3f0 tx=0x7f562800c8b0 comp rx=0 tx=0).stop 2024-03-28T23:29:39.778 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.776+0000 7f563a910640 1 -- 172.21.15.161:0/3200583527 shutdown_connections 2024-03-28T23:29:39.778 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.776+0000 7f563a910640 1 --2- 172.21.15.161:0/3200583527 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f562006b470 0x7f562006d930 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:39.778 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.776+0000 7f563a910640 1 --2- 172.21.15.161:0/3200583527 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56340c3580 0x7f56341eaf90 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:39.778 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.776+0000 7f563a910640 1 --2- 172.21.15.161:0/3200583527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56341526c0 0x7f56341512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:39.778 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.776+0000 7f563a910640 1 --2- 172.21.15.161:0/3200583527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56340c2d00 0x7f56341ee680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:39.778 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.776+0000 7f563a910640 1 -- 172.21.15.161:0/3200583527 >> 172.21.15.161:0/3200583527 conn(0x7f56340aaa10 msgr2=0x7f56340a9080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:39.779 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.776+0000 7f563a910640 1 -- 172.21.15.161:0/3200583527 shutdown_connections 2024-03-28T23:29:39.779 INFO:teuthology.misc.health.smithi161.stderr:2024-03-28T23:29:39.776+0000 7f563a910640 1 -- 172.21.15.161:0/3200583527 wait complete. 2024-03-28T23:29:39.786 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-28T23:29:39.787 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-28T23:29:39.787 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T23:29:39.787 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T23:29:39.791 INFO:tasks.ceph.mds.a:Started 2024-03-28T23:29:39.791 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T23:29:39.791 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T23:29:39.793 INFO:tasks.ceph.mds.c:Started 2024-03-28T23:29:39.793 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-28T23:29:39.793 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-28T23:29:39.796 INFO:tasks.ceph.mds.b:Started 2024-03-28T23:29:39.797 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-28T23:29:39.797 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-28T23:29:39.799 INFO:tasks.ceph.mds.d:Started 2024-03-28T23:29:39.799 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-28T23:29:39.801 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:29:39.866 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-28T23:29:39.867 INFO:tasks.ceph.mds.c.smithi161.stdout:starting mds.c at 2024-03-28T23:29:39.883 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2024-03-28T23:29:39.883 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2024-03-28T23:29:39.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.907+0000 7fe88c1c9640 1 Processor -- start 2024-03-28T23:29:39.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.907+0000 7fe88c1c9640 1 -- start start 2024-03-28T23:29:39.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.908+0000 7fe88c1c9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe884148980 0x7fe8840c6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:39.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.908+0000 7fe88c1c9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8840c2dd0 0x7fe8840aba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:39.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.908+0000 7fe88c1c9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe884152b30 con 0x7fe8840c35c0 2024-03-28T23:29:39.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.908+0000 7fe88c1c9640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe884152e10 con 0x7fe884148980 2024-03-28T23:29:39.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.908+0000 7fe88c1c9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe884148d60 con 0x7fe8840c2dd0 2024-03-28T23:29:39.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.908+0000 7fe889f65640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe884148980 0x7fe8840c6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:39.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.908+0000 7fe889f65640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe884148980 0x7fe8840c6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:56470/0 (socket says 172.21.15.161:56470) 2024-03-28T23:29:39.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.908+0000 7fe889f65640 1 -- 172.21.15.161:0/120376399 learned_addr learned my addr 172.21.15.161:0/120376399 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:39.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.908+0000 7fe889764640 1 --2- 172.21.15.161:0/120376399 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8840c2dd0 0x7fe8840aba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:39.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.908+0000 7fe889f65640 1 -- 172.21.15.161:0/120376399 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8840c2dd0 msgr2=0x7fe8840aba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:39.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.908+0000 7fe889f65640 1 --2- 172.21.15.161:0/120376399 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8840c2dd0 0x7fe8840aba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:39.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.908+0000 7fe889f65640 1 -- 172.21.15.161:0/120376399 >> v1:172.21.15.170:6789/0 conn(0x7fe8840c35c0 legacy=0x7fe884155450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:39.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.908+0000 7fe889f65640 1 -- 172.21.15.161:0/120376399 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8840c22f0 con 0x7fe884148980 2024-03-28T23:29:39.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.908+0000 7fe889764640 1 --2- 172.21.15.161:0/120376399 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8840c2dd0 0x7fe8840aba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:29:39.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.909+0000 7fe889f65640 1 --2- 172.21.15.161:0/120376399 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe884148980 0x7fe8840c6660 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe88000bb10 tx=0x7fe88002e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d4bd3b8c5433e2e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:39.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.909+0000 7fe888f63640 1 -- 172.21.15.161:0/120376399 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88003b070 con 0x7fe884148980 2024-03-28T23:29:39.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.909+0000 7fe888f63640 1 -- 172.21.15.161:0/120376399 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe880036440 con 0x7fe884148980 2024-03-28T23:29:39.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.909+0000 7fe888f63640 1 -- 172.21.15.161:0/120376399 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe880036810 con 0x7fe884148980 2024-03-28T23:29:39.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.910+0000 7fe88c1c9640 1 -- 172.21.15.161:0/120376399 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe884148980 msgr2=0x7fe8840c6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:39.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.910+0000 7fe88c1c9640 1 --2- 172.21.15.161:0/120376399 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe884148980 0x7fe8840c6660 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe88000bb10 tx=0x7fe88002e150 comp rx=0 tx=0).stop 2024-03-28T23:29:39.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.910+0000 7fe88c1c9640 1 -- 172.21.15.161:0/120376399 shutdown_connections 2024-03-28T23:29:39.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.910+0000 7fe88c1c9640 1 --2- 172.21.15.161:0/120376399 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8840c2dd0 0x7fe8840aba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:39.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.910+0000 7fe88c1c9640 1 --2- 172.21.15.161:0/120376399 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe884148980 0x7fe8840c6660 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:39.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.910+0000 7fe88c1c9640 1 -- 172.21.15.161:0/120376399 >> 172.21.15.161:0/120376399 conn(0x7fe8840c4250 msgr2=0x7fe8840c4660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:39.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.910+0000 7fe88c1c9640 1 -- 172.21.15.161:0/120376399 shutdown_connections 2024-03-28T23:29:39.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.910+0000 7fe88c1c9640 1 -- 172.21.15.161:0/120376399 wait complete. 2024-03-28T23:29:39.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.911+0000 7fe88c1c9640 1 Processor -- start 2024-03-28T23:29:39.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.911+0000 7fe88c1c9640 1 -- start start 2024-03-28T23:29:39.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.912+0000 7fe88c1c9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8840c2dd0 0x7fe884144be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:39.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.912+0000 7fe88c1c9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8840c35c0 0x7fe88406cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:39.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.912+0000 7fe88c1c9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe884148980 0x7fe884142740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:39.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.912+0000 7fe88c1c9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe8840c2940 con 0x7fe8840c35c0 2024-03-28T23:29:39.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.912+0000 7fe88c1c9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe8841528e0 con 0x7fe884148980 2024-03-28T23:29:39.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.912+0000 7fe88c1c9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe8840c3280 con 0x7fe8840c2dd0 2024-03-28T23:29:39.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.912+0000 7fe889764640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8840c35c0 0x7fe88406cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:39.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.912+0000 7fe889764640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8840c35c0 0x7fe88406cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:56486/0 (socket says 172.21.15.161:56486) 2024-03-28T23:29:39.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.912+0000 7fe889764640 1 -- 172.21.15.161:0/2089600952 learned_addr learned my addr 172.21.15.161:0/2089600952 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:39.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.912+0000 7fe88a766640 1 --2- 172.21.15.161:0/2089600952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe884148980 0x7fe884142740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:39.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.912+0000 7fe889f65640 1 --2- 172.21.15.161:0/2089600952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8840c2dd0 0x7fe884144be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:39.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.912+0000 7fe889764640 1 -- 172.21.15.161:0/2089600952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8840c2dd0 msgr2=0x7fe884144be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:39.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.912+0000 7fe889764640 1 --2- 172.21.15.161:0/2089600952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8840c2dd0 0x7fe884144be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:39.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.912+0000 7fe889764640 1 -- 172.21.15.161:0/2089600952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe884148980 msgr2=0x7fe884142740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:39.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.912+0000 7fe889764640 1 --2- 172.21.15.161:0/2089600952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe884148980 0x7fe884142740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:39.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.912+0000 7fe889764640 1 -- 172.21.15.161:0/2089600952 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe884145120 con 0x7fe8840c35c0 2024-03-28T23:29:39.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.912+0000 7fe889764640 1 --2- 172.21.15.161:0/2089600952 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8840c35c0 0x7fe88406cdc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe87400e810 tx=0x7fe87400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:39.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.912+0000 7fe87a7fc640 1 -- 172.21.15.161:0/2089600952 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87400f040 con 0x7fe8840c35c0 2024-03-28T23:29:39.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.912+0000 7fe87a7fc640 1 -- 172.21.15.161:0/2089600952 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe874004030 con 0x7fe8840c35c0 2024-03-28T23:29:39.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.912+0000 7fe87a7fc640 1 -- 172.21.15.161:0/2089600952 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe874004400 con 0x7fe8840c35c0 2024-03-28T23:29:39.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.913+0000 7fe88c1c9640 1 -- 172.21.15.161:0/2089600952 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe884144420 con 0x7fe8840c35c0 2024-03-28T23:29:39.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.913+0000 7fe88c1c9640 1 -- 172.21.15.161:0/2089600952 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe88406b560 con 0x7fe8840c35c0 2024-03-28T23:29:39.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.913+0000 7fe88c1c9640 1 -- 172.21.15.161:0/2089600952 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe88404f4b0 con 0x7fe8840c35c0 2024-03-28T23:29:39.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.914+0000 7fe87a7fc640 1 -- 172.21.15.161:0/2089600952 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 4) ==== 89724+0+0 (secure 0 0 0) 0x7fe87400c6d0 con 0x7fe8840c35c0 2024-03-28T23:29:39.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.914+0000 7fe87a7fc640 1 --2- 172.21.15.161:0/2089600952 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe86806b440 0x7fe86806d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:39.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.914+0000 7fe87a7fc640 1 -- 172.21.15.161:0/2089600952 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 4471+0+0 (secure 0 0 0) 0x7fe8740898e0 con 0x7fe8840c35c0 2024-03-28T23:29:39.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.915+0000 7fe889f65640 1 --2- 172.21.15.161:0/2089600952 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe86806b440 0x7fe86806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:39.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.915+0000 7fe889f65640 1 --2- 172.21.15.161:0/2089600952 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe86806b440 0x7fe86806d900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe880038000 tx=0x7fe880032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:39.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:39.919+0000 7fe87a7fc640 1 -- 172.21.15.161:0/2089600952 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe884145120 con 0x7fe8840c35c0 2024-03-28T23:29:40.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.094+0000 7fe88c1c9640 1 -- 172.21.15.161:0/2089600952 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe8840c4a40 con 0x7fe8840c35c0 2024-03-28T23:29:40.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.095+0000 7fe87a7fc640 1 -- 172.21.15.161:0/2089600952 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+10945 (secure 0 0 0) 0x7fe874017020 con 0x7fe8840c35c0 2024-03-28T23:29:40.096 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:29:40.096 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":13,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:29:39.231019+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:29:40.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.097+0000 7fe88c1c9640 1 -- 172.21.15.161:0/2089600952 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe86806b440 msgr2=0x7fe86806d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:40.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.097+0000 7fe88c1c9640 1 --2- 172.21.15.161:0/2089600952 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe86806b440 0x7fe86806d900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe880038000 tx=0x7fe880032000 comp rx=0 tx=0).stop 2024-03-28T23:29:40.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.097+0000 7fe88c1c9640 1 -- 172.21.15.161:0/2089600952 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8840c35c0 msgr2=0x7fe88406cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:40.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.097+0000 7fe88c1c9640 1 --2- 172.21.15.161:0/2089600952 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8840c35c0 0x7fe88406cdc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe87400e810 tx=0x7fe87400ecd0 comp rx=0 tx=0).stop 2024-03-28T23:29:40.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.097+0000 7fe88c1c9640 1 -- 172.21.15.161:0/2089600952 shutdown_connections 2024-03-28T23:29:40.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.097+0000 7fe88c1c9640 1 --2- 172.21.15.161:0/2089600952 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe86806b440 0x7fe86806d900 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:40.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.097+0000 7fe88c1c9640 1 --2- 172.21.15.161:0/2089600952 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8840c35c0 0x7fe88406cdc0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:40.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.097+0000 7fe88c1c9640 1 --2- 172.21.15.161:0/2089600952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe884148980 0x7fe884142740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:40.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.097+0000 7fe88c1c9640 1 --2- 172.21.15.161:0/2089600952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8840c2dd0 0x7fe884144be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:40.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.097+0000 7fe88c1c9640 1 -- 172.21.15.161:0/2089600952 >> 172.21.15.161:0/2089600952 conn(0x7fe8840c4250 msgr2=0x7fe8840a72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:40.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.097+0000 7fe88c1c9640 1 -- 172.21.15.161:0/2089600952 shutdown_connections 2024-03-28T23:29:40.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.097+0000 7fe88c1c9640 1 -- 172.21.15.161:0/2089600952 wait complete. 2024-03-28T23:29:40.110 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-03-28T23:29:40.110 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:29:40.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.213+0000 7f9593aeb640 1 Processor -- start 2024-03-28T23:29:40.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.214+0000 7f9593aeb640 1 -- start start 2024-03-28T23:29:40.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.214+0000 7f9593aeb640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f958c155df0 0x7f958c159240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:40.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.214+0000 7f9593aeb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f958c0c23f0 con 0x7f958c152920 2024-03-28T23:29:40.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.214+0000 7f9593aeb640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f958c155350 con 0x7f958c0c67b0 2024-03-28T23:29:40.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.214+0000 7f9593aeb640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f958c152d00 con 0x7f958c155df0 2024-03-28T23:29:40.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.214+0000 7f9591887640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f958c152920 0x7f958c0c1ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:58926/0 (socket says 172.21.15.161:58926) 2024-03-28T23:29:40.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.214+0000 7f9591887640 1 -- 172.21.15.161:0/515937677 learned_addr learned my addr 172.21.15.161:0/515937677 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:40.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.215+0000 7f9591086640 1 --2- 172.21.15.161:0/515937677 >> v2:172.21.15.170:3300/0 conn(0x7f958c155df0 0x7f958c159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:40.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.215+0000 7f9590885640 1 -- 172.21.15.161:0/515937677 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3735212129 0 0) 0x7f958c0c23f0 con 0x7f958c152920 2024-03-28T23:29:40.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.215+0000 7f9590885640 1 -- 172.21.15.161:0/515937677 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9578003520 con 0x7f958c152920 2024-03-28T23:29:40.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.215+0000 7f9590885640 1 -- 172.21.15.161:0/515937677 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2516751751 0 0) 0x7f9578003520 con 0x7f958c152920 2024-03-28T23:29:40.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.215+0000 7f9590885640 1 -- 172.21.15.161:0/515937677 >> v2:172.21.15.170:3300/0 conn(0x7f958c155df0 msgr2=0x7f958c159240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:40.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.215+0000 7f9590885640 1 --2- 172.21.15.161:0/515937677 >> v2:172.21.15.170:3300/0 conn(0x7f958c155df0 0x7f958c159240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:40.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.215+0000 7f9590885640 1 -- 172.21.15.161:0/515937677 >> v1:172.21.15.170:6789/0 conn(0x7f958c0c67b0 legacy=0x7f958c15fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:40.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.215+0000 7f9590885640 1 -- 172.21.15.161:0/515937677 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f958c0a8070 con 0x7f958c152920 2024-03-28T23:29:40.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.215+0000 7f9590885640 1 -- 172.21.15.161:0/515937677 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f9588001670 con 0x7f958c152920 2024-03-28T23:29:40.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.215+0000 7f9590885640 1 -- 172.21.15.161:0/515937677 >> v1:172.21.15.161:6789/0 conn(0x7f958c152920 legacy=0x7f958c0c1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:40.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.215+0000 7f9590885640 1 --2- 172.21.15.161:0/515937677 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9578003a90 0x7f9578003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:40.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.215+0000 7f9590885640 1 --2- 172.21.15.161:0/515937677 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9578004800 0x7f9578004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:40.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.215+0000 7f9590885640 1 --2- 172.21.15.161:0/515937677 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9578005290 0x7f957800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:40.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.215+0000 7f9590885640 1 -- 172.21.15.161:0/515937677 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f958c152d00 con 0x7f9578005290 2024-03-28T23:29:40.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.215+0000 7f9590885640 1 -- 172.21.15.161:0/515937677 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f957800c1e0 con 0x7f9578004800 2024-03-28T23:29:40.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.215+0000 7f9590885640 1 -- 172.21.15.161:0/515937677 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f957800c3e0 con 0x7f9578003a90 2024-03-28T23:29:40.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.215+0000 7f9592088640 1 --2- 172.21.15.161:0/515937677 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9578005290 0x7f957800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:40.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.215+0000 7f9591887640 1 --2- 172.21.15.161:0/515937677 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9578003a90 0x7f9578003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:40.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.215+0000 7f9591086640 1 --2- 172.21.15.161:0/515937677 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9578004800 0x7f9578004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:40.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.215+0000 7f9592088640 1 -- 172.21.15.161:0/515937677 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9578003a90 msgr2=0x7f9578003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:40.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.215+0000 7f9592088640 1 --2- 172.21.15.161:0/515937677 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9578003a90 0x7f9578003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:40.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.215+0000 7f9592088640 1 -- 172.21.15.161:0/515937677 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9578004800 msgr2=0x7f9578004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:40.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.215+0000 7f9592088640 1 --2- 172.21.15.161:0/515937677 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9578004800 0x7f9578004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:40.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.215+0000 7f9592088640 1 -- 172.21.15.161:0/515937677 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9578003520 con 0x7f9578005290 2024-03-28T23:29:40.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.216+0000 7f9592088640 1 -- 172.21.15.161:0/515937677 reap_dead start 2024-03-28T23:29:40.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.216+0000 7f9592088640 1 --2- 172.21.15.161:0/515937677 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9578005290 0x7f957800b9c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9584001e50 tx=0x7f9578005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e04551ae4ced1d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:40.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.216+0000 7f9590885640 1 -- 172.21.15.161:0/515937677 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9584001a70 con 0x7f9578005290 2024-03-28T23:29:40.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.216+0000 7f9590885640 1 -- 172.21.15.161:0/515937677 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9584003bb0 con 0x7f9578005290 2024-03-28T23:29:40.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.216+0000 7f9590885640 1 -- 172.21.15.161:0/515937677 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95840050d0 con 0x7f9578005290 2024-03-28T23:29:40.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.217+0000 7f9593aeb640 1 -- 172.21.15.161:0/515937677 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9578005290 msgr2=0x7f957800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:40.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.217+0000 7f9593aeb640 1 --2- 172.21.15.161:0/515937677 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9578005290 0x7f957800b9c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9584001e50 tx=0x7f9578005830 comp rx=0 tx=0).stop 2024-03-28T23:29:40.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.217+0000 7f9593aeb640 1 -- 172.21.15.161:0/515937677 shutdown_connections 2024-03-28T23:29:40.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.217+0000 7f9593aeb640 1 --2- 172.21.15.161:0/515937677 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9578005290 0x7f957800b9c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:40.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.217+0000 7f9593aeb640 1 -- 172.21.15.161:0/515937677 >> 172.21.15.161:0/515937677 conn(0x7f958c0c88b0 msgr2=0x7f958c0c8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:40.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.217+0000 7f9593aeb640 1 -- 172.21.15.161:0/515937677 shutdown_connections 2024-03-28T23:29:40.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.217+0000 7f9593aeb640 1 -- 172.21.15.161:0/515937677 wait complete. 2024-03-28T23:29:40.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.218+0000 7f9593aeb640 1 Processor -- start 2024-03-28T23:29:40.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.218+0000 7f9593aeb640 1 -- start start 2024-03-28T23:29:40.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.219+0000 7f9593aeb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9578005290 0x7f958c149900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:40.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.219+0000 7f9593aeb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f958c1564d0 0x7f958c148bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:40.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.219+0000 7f9593aeb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f958c1592e0 0x7f958c149110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:40.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.219+0000 7f9593aeb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f958c0a8350 con 0x7f958c1592e0 2024-03-28T23:29:40.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.219+0000 7f9593aeb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f958c155960 con 0x7f9578005290 2024-03-28T23:29:40.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.219+0000 7f9593aeb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f958c0a7d00 con 0x7f958c1564d0 2024-03-28T23:29:40.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.219+0000 7f9592088640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f958c1592e0 0x7f958c149110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:40.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.219+0000 7f9592088640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f958c1592e0 0x7f958c149110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:56498/0 (socket says 172.21.15.161:56498) 2024-03-28T23:29:40.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.219+0000 7f9592088640 1 -- 172.21.15.161:0/2375307544 learned_addr learned my addr 172.21.15.161:0/2375307544 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:40.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.219+0000 7f9591086640 1 --2- 172.21.15.161:0/2375307544 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f958c1564d0 0x7f958c148bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:40.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.219+0000 7f9591887640 1 --2- 172.21.15.161:0/2375307544 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9578005290 0x7f958c149900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:40.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.219+0000 7f9592088640 1 -- 172.21.15.161:0/2375307544 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f958c1564d0 msgr2=0x7f958c148bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:40.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.219+0000 7f9592088640 1 --2- 172.21.15.161:0/2375307544 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f958c1564d0 0x7f958c148bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:40.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.219+0000 7f9592088640 1 -- 172.21.15.161:0/2375307544 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9578005290 msgr2=0x7f958c149900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:40.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.219+0000 7f9592088640 1 --2- 172.21.15.161:0/2375307544 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9578005290 0x7f958c149900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:40.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.219+0000 7f9592088640 1 -- 172.21.15.161:0/2375307544 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f958c148130 con 0x7f958c1592e0 2024-03-28T23:29:40.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.219+0000 7f9592088640 1 --2- 172.21.15.161:0/2375307544 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f958c1592e0 0x7f958c149110 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f958402c060 tx=0x7f95840014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:40.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.219+0000 7f95827fc640 1 -- 172.21.15.161:0/2375307544 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9584006d60 con 0x7f958c1592e0 2024-03-28T23:29:40.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.219+0000 7f95827fc640 1 -- 172.21.15.161:0/2375307544 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9584039480 con 0x7f958c1592e0 2024-03-28T23:29:40.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.219+0000 7f95827fc640 1 -- 172.21.15.161:0/2375307544 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9584039830 con 0x7f958c1592e0 2024-03-28T23:29:40.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.220+0000 7f9593aeb640 1 -- 172.21.15.161:0/2375307544 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f958c1eba10 con 0x7f958c1592e0 2024-03-28T23:29:40.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.220+0000 7f9593aeb640 1 -- 172.21.15.161:0/2375307544 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f958c1486b0 con 0x7f958c1592e0 2024-03-28T23:29:40.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.221+0000 7f9593aeb640 1 -- 172.21.15.161:0/2375307544 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f958c1488b0 con 0x7f958c1592e0 2024-03-28T23:29:40.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.221+0000 7f95827fc640 1 -- 172.21.15.161:0/2375307544 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 4) ==== 89724+0+0 (secure 0 0 0) 0x7f95840399d0 con 0x7f958c1592e0 2024-03-28T23:29:40.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.221+0000 7f95827fc640 1 --2- 172.21.15.161:0/2375307544 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f957006b470 0x7f957006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:40.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.221+0000 7f95827fc640 1 -- 172.21.15.161:0/2375307544 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 4471+0+0 (secure 0 0 0) 0x7f95840ab940 con 0x7f958c1592e0 2024-03-28T23:29:40.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.221+0000 7f9591887640 1 --2- 172.21.15.161:0/2375307544 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f957006b470 0x7f957006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:40.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.222+0000 7f9591887640 1 --2- 172.21.15.161:0/2375307544 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f957006b470 0x7f957006d930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9588001eb0 tx=0x7f9588009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:40.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.226+0000 7f95827fc640 1 -- 172.21.15.161:0/2375307544 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f958c1488b0 con 0x7f958c1592e0 2024-03-28T23:29:40.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.447+0000 7f9593aeb640 1 -- 172.21.15.161:0/2375307544 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f958c0c6c20 con 0x7f958c1592e0 2024-03-28T23:29:40.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.448+0000 7f95827fc640 1 -- 172.21.15.161:0/2375307544 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1395 (secure 0 0 0) 0x7f958403e070 con 0x7f958c1592e0 2024-03-28T23:29:40.449 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:29:40.449 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4107992135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4107992135},{"type":"v1","addr":"172.21.15.161:6836","nonce":4107992135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-28T23:29:40.449 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 2 2024-03-28T23:29:40.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.450+0000 7f9593aeb640 1 -- 172.21.15.161:0/2375307544 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f957006b470 msgr2=0x7f957006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:40.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.450+0000 7f9593aeb640 1 --2- 172.21.15.161:0/2375307544 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f957006b470 0x7f957006d930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9588001eb0 tx=0x7f9588009040 comp rx=0 tx=0).stop 2024-03-28T23:29:40.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.450+0000 7f9593aeb640 1 -- 172.21.15.161:0/2375307544 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f958c1592e0 msgr2=0x7f958c149110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:40.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.450+0000 7f9593aeb640 1 --2- 172.21.15.161:0/2375307544 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f958c1592e0 0x7f958c149110 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f958402c060 tx=0x7f95840014a0 comp rx=0 tx=0).stop 2024-03-28T23:29:40.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.450+0000 7f9593aeb640 1 -- 172.21.15.161:0/2375307544 shutdown_connections 2024-03-28T23:29:40.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.450+0000 7f9593aeb640 1 --2- 172.21.15.161:0/2375307544 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f957006b470 0x7f957006d930 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:40.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.450+0000 7f9593aeb640 1 --2- 172.21.15.161:0/2375307544 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f958c1592e0 0x7f958c149110 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:40.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.450+0000 7f9593aeb640 1 --2- 172.21.15.161:0/2375307544 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f958c1564d0 0x7f958c148bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:40.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.450+0000 7f9593aeb640 1 --2- 172.21.15.161:0/2375307544 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9578005290 0x7f958c149900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:40.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.450+0000 7f9593aeb640 1 -- 172.21.15.161:0/2375307544 >> 172.21.15.161:0/2375307544 conn(0x7f958c0c88b0 msgr2=0x7f958c0c31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:40.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.450+0000 7f9593aeb640 1 -- 172.21.15.161:0/2375307544 shutdown_connections 2024-03-28T23:29:40.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:40.450+0000 7f9593aeb640 1 -- 172.21.15.161:0/2375307544 wait complete. 2024-03-28T23:29:42.462 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:29:42.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.571+0000 7f299955e640 1 Processor -- start 2024-03-28T23:29:42.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.571+0000 7f299955e640 1 -- start start 2024-03-28T23:29:42.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.572+0000 7f299955e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2994152920 0x7f2994155df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:42.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.572+0000 7f299955e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29940c23f0 con 0x7f29940c67b0 2024-03-28T23:29:42.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.572+0000 7f299955e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2994155350 con 0x7f2994159240 2024-03-28T23:29:42.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.572+0000 7f299955e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2994152d00 con 0x7f2994152920 2024-03-28T23:29:42.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.572+0000 7f29937fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f29940c67b0 0x7f299415fbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48210/0 (socket says 172.21.15.161:48210) 2024-03-28T23:29:42.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.572+0000 7f29937fe640 1 -- 172.21.15.161:0/2646561713 learned_addr learned my addr 172.21.15.161:0/2646561713 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:42.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.572+0000 7f2992ffd640 1 --2- 172.21.15.161:0/2646561713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2994152920 0x7f2994155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:42.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.572+0000 7f2991ffb640 1 -- 172.21.15.161:0/2646561713 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3047010884 0 0) 0x7f29940c23f0 con 0x7f29940c67b0 2024-03-28T23:29:42.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.572+0000 7f2991ffb640 1 -- 172.21.15.161:0/2646561713 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f297c003520 con 0x7f29940c67b0 2024-03-28T23:29:42.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.573+0000 7f2991ffb640 1 -- 172.21.15.161:0/2646561713 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3980465741 0 0) 0x7f2994155350 con 0x7f2994159240 2024-03-28T23:29:42.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.573+0000 7f2991ffb640 1 -- 172.21.15.161:0/2646561713 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29940c23f0 con 0x7f2994159240 2024-03-28T23:29:42.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.573+0000 7f2991ffb640 1 -- 172.21.15.161:0/2646561713 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 402814518 0 0) 0x7f297c003520 con 0x7f29940c67b0 2024-03-28T23:29:42.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.573+0000 7f2991ffb640 1 -- 172.21.15.161:0/2646561713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2994152920 msgr2=0x7f2994155df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:42.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.573+0000 7f2991ffb640 1 --2- 172.21.15.161:0/2646561713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2994152920 0x7f2994155df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:42.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.573+0000 7f2991ffb640 1 -- 172.21.15.161:0/2646561713 >> v1:172.21.15.170:6789/0 conn(0x7f2994159240 legacy=0x7f29940c1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:42.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.573+0000 7f2991ffb640 1 -- 172.21.15.161:0/2646561713 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29940a8070 con 0x7f29940c67b0 2024-03-28T23:29:42.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.573+0000 7f2991ffb640 1 -- 172.21.15.161:0/2646561713 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f298c0019f0 con 0x7f29940c67b0 2024-03-28T23:29:42.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.573+0000 7f2991ffb640 1 -- 172.21.15.161:0/2646561713 >> v1:172.21.15.161:6789/0 conn(0x7f29940c67b0 legacy=0x7f299415fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:42.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.573+0000 7f2991ffb640 1 --2- 172.21.15.161:0/2646561713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f297c003c50 0x7f297c004050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:42.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.573+0000 7f2991ffb640 1 --2- 172.21.15.161:0/2646561713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f297c004800 0x7f297c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:42.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.573+0000 7f2991ffb640 1 --2- 172.21.15.161:0/2646561713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2994152920 0x7f297c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:42.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.573+0000 7f2991ffb640 1 -- 172.21.15.161:0/2646561713 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2994152d00 con 0x7f297c004800 2024-03-28T23:29:42.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.573+0000 7f2991ffb640 1 -- 172.21.15.161:0/2646561713 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f297c005510 con 0x7f297c003c50 2024-03-28T23:29:42.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.573+0000 7f2991ffb640 1 -- 172.21.15.161:0/2646561713 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f297c00bf00 con 0x7f2994152920 2024-03-28T23:29:42.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.573+0000 7f29927fc640 1 --2- 172.21.15.161:0/2646561713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f297c004800 0x7f297c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:42.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.573+0000 7f2992ffd640 1 --2- 172.21.15.161:0/2646561713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f297c003c50 0x7f297c004050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:42.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.573+0000 7f29937fe640 1 --2- 172.21.15.161:0/2646561713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2994152920 0x7f297c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:42.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.573+0000 7f29927fc640 1 -- 172.21.15.161:0/2646561713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2994152920 msgr2=0x7f297c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:42.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.573+0000 7f29927fc640 1 --2- 172.21.15.161:0/2646561713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2994152920 0x7f297c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:42.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.573+0000 7f29927fc640 1 -- 172.21.15.161:0/2646561713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f297c003c50 msgr2=0x7f297c004050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:42.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.573+0000 7f29927fc640 1 --2- 172.21.15.161:0/2646561713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f297c003c50 0x7f297c004050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:42.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.573+0000 7f29927fc640 1 -- 172.21.15.161:0/2646561713 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f297c003520 con 0x7f297c004800 2024-03-28T23:29:42.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.574+0000 7f29927fc640 1 --2- 172.21.15.161:0/2646561713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f297c004800 0x7f297c004c80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f29880018d0 tx=0x7f297c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edd9cbb8b43812f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:42.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.574+0000 7f2991ffb640 1 -- 172.21.15.161:0/2646561713 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2988037070 con 0x7f297c004800 2024-03-28T23:29:42.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.574+0000 7f2991ffb640 1 -- 172.21.15.161:0/2646561713 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29880052d0 con 0x7f297c004800 2024-03-28T23:29:42.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.574+0000 7f2991ffb640 1 -- 172.21.15.161:0/2646561713 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29880014f0 con 0x7f297c004800 2024-03-28T23:29:42.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.574+0000 7f299955e640 1 -- 172.21.15.161:0/2646561713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f297c004800 msgr2=0x7f297c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:42.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.574+0000 7f299955e640 1 --2- 172.21.15.161:0/2646561713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f297c004800 0x7f297c004c80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f29880018d0 tx=0x7f297c005830 comp rx=0 tx=0).stop 2024-03-28T23:29:42.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.574+0000 7f29937fe640 1 -- 172.21.15.161:0/2646561713 reap_dead start 2024-03-28T23:29:42.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.574+0000 7f299955e640 1 -- 172.21.15.161:0/2646561713 shutdown_connections 2024-03-28T23:29:42.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.575+0000 7f299955e640 1 -- 172.21.15.161:0/2646561713 >> 172.21.15.161:0/2646561713 conn(0x7f29940c88b0 msgr2=0x7f29940c8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:42.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.575+0000 7f299955e640 1 -- 172.21.15.161:0/2646561713 shutdown_connections 2024-03-28T23:29:42.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.575+0000 7f299955e640 1 -- 172.21.15.161:0/2646561713 wait complete. 2024-03-28T23:29:42.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.576+0000 7f299955e640 1 Processor -- start 2024-03-28T23:29:42.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.576+0000 7f299955e640 1 -- start start 2024-03-28T23:29:42.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.576+0000 7f299955e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f297c004800 0x7f29941ee200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:42.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.576+0000 7f299955e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29940ac730 0x7f29940acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:42.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.576+0000 7f299955e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29940ad0f0 0x7f299406c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:42.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.576+0000 7f299955e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f29940a8350 con 0x7f29940ad0f0 2024-03-28T23:29:42.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.576+0000 7f299955e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2994155960 con 0x7f297c004800 2024-03-28T23:29:42.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.576+0000 7f299955e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f29941596a0 con 0x7f29940ac730 2024-03-28T23:29:42.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.577+0000 7f29937fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29940ad0f0 0x7f299406c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:42.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.577+0000 7f29937fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29940ad0f0 0x7f299406c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50994/0 (socket says 172.21.15.161:50994) 2024-03-28T23:29:42.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.577+0000 7f29937fe640 1 -- 172.21.15.161:0/701737803 learned_addr learned my addr 172.21.15.161:0/701737803 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:42.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.577+0000 7f29927fc640 1 --2- 172.21.15.161:0/701737803 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29940ac730 0x7f29940acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:42.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.577+0000 7f2992ffd640 1 --2- 172.21.15.161:0/701737803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f297c004800 0x7f29941ee200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:42.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.577+0000 7f29937fe640 1 -- 172.21.15.161:0/701737803 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29940ac730 msgr2=0x7f29940acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:42.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.577+0000 7f29937fe640 1 --2- 172.21.15.161:0/701737803 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29940ac730 0x7f29940acbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:42.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.577+0000 7f29937fe640 1 -- 172.21.15.161:0/701737803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f297c004800 msgr2=0x7f29941ee200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:42.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.577+0000 7f29937fe640 1 --2- 172.21.15.161:0/701737803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f297c004800 0x7f29941ee200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:42.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.577+0000 7f29937fe640 1 -- 172.21.15.161:0/701737803 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29940c40b0 con 0x7f29940ad0f0 2024-03-28T23:29:42.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.577+0000 7f29937fe640 1 --2- 172.21.15.161:0/701737803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29940ad0f0 0x7f299406c220 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f298c002230 tx=0x7f298c007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:42.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.577+0000 7f297b7fe640 1 -- 172.21.15.161:0/701737803 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298c00a9d0 con 0x7f29940ad0f0 2024-03-28T23:29:42.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.577+0000 7f297b7fe640 1 -- 172.21.15.161:0/701737803 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f298c018070 con 0x7f29940ad0f0 2024-03-28T23:29:42.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.577+0000 7f299955e640 1 -- 172.21.15.161:0/701737803 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f299406e3b0 con 0x7f29940ad0f0 2024-03-28T23:29:42.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.577+0000 7f297b7fe640 1 -- 172.21.15.161:0/701737803 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298c00add0 con 0x7f29940ad0f0 2024-03-28T23:29:42.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.577+0000 7f299955e640 1 -- 172.21.15.161:0/701737803 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29941eea40 con 0x7f29940ad0f0 2024-03-28T23:29:42.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.578+0000 7f299955e640 1 -- 172.21.15.161:0/701737803 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f299404f4b0 con 0x7f29940ad0f0 2024-03-28T23:29:42.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.579+0000 7f297b7fe640 1 -- 172.21.15.161:0/701737803 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 4) ==== 89724+0+0 (secure 0 0 0) 0x7f298c01d460 con 0x7f29940ad0f0 2024-03-28T23:29:42.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.579+0000 7f297b7fe640 1 --2- 172.21.15.161:0/701737803 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f298006b470 0x7f298006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:42.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.579+0000 7f297b7fe640 1 -- 172.21.15.161:0/701737803 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f298c08a0a0 con 0x7f29940ad0f0 2024-03-28T23:29:42.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.579+0000 7f2992ffd640 1 --2- 172.21.15.161:0/701737803 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f298006b470 0x7f298006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:42.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.580+0000 7f2992ffd640 1 --2- 172.21.15.161:0/701737803 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f298006b470 0x7f298006d930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f29941551b0 tx=0x7f2984009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:42.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.584+0000 7f297b7fe640 1 -- 172.21.15.161:0/701737803 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f298c0593e0 con 0x7f29940ad0f0 2024-03-28T23:29:42.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.793+0000 7f299955e640 1 -- 172.21.15.161:0/701737803 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2994072b20 con 0x7f29940ad0f0 2024-03-28T23:29:42.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.794+0000 7f297b7fe640 1 -- 172.21.15.161:0/701737803 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3876 (secure 0 0 0) 0x7f298c058ce0 con 0x7f29940ad0f0 2024-03-28T23:29:42.795 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:29:42.795 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/1577284938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":1577284938},{"type":"v1","addr":"172.21.15.170:6835","nonce":1577284938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4258,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/1107796931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1107796931},{"type":"v1","addr":"172.21.15.170:6834","nonce":1107796931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4107992135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4107992135},{"type":"v1","addr":"172.21.15.161:6836","nonce":4107992135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/3832540545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3832540545},{"type":"v1","addr":"172.21.15.161:6837","nonce":3832540545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-28T23:29:42.795 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 3 2024-03-28T23:29:42.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.796+0000 7f299955e640 1 -- 172.21.15.161:0/701737803 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f298006b470 msgr2=0x7f298006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:42.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.796+0000 7f299955e640 1 --2- 172.21.15.161:0/701737803 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f298006b470 0x7f298006d930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f29941551b0 tx=0x7f2984009190 comp rx=0 tx=0).stop 2024-03-28T23:29:42.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.796+0000 7f299955e640 1 -- 172.21.15.161:0/701737803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29940ad0f0 msgr2=0x7f299406c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:42.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.796+0000 7f299955e640 1 --2- 172.21.15.161:0/701737803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29940ad0f0 0x7f299406c220 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f298c002230 tx=0x7f298c007b10 comp rx=0 tx=0).stop 2024-03-28T23:29:42.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.796+0000 7f299955e640 1 -- 172.21.15.161:0/701737803 shutdown_connections 2024-03-28T23:29:42.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.796+0000 7f299955e640 1 --2- 172.21.15.161:0/701737803 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f298006b470 0x7f298006d930 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:42.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.796+0000 7f299955e640 1 --2- 172.21.15.161:0/701737803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29940ad0f0 0x7f299406c220 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:42.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.796+0000 7f299955e640 1 --2- 172.21.15.161:0/701737803 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29940ac730 0x7f29940acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:42.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.796+0000 7f299955e640 1 --2- 172.21.15.161:0/701737803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f297c004800 0x7f29941ee200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:42.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.796+0000 7f299955e640 1 -- 172.21.15.161:0/701737803 >> 172.21.15.161:0/701737803 conn(0x7f29940c88b0 msgr2=0x7f29940c48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:42.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.796+0000 7f299955e640 1 -- 172.21.15.161:0/701737803 shutdown_connections 2024-03-28T23:29:42.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.796+0000 7f299955e640 1 -- 172.21.15.161:0/701737803 wait complete. 2024-03-28T23:29:42.808 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:29:42.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.907+0000 7f153997a640 1 Processor -- start 2024-03-28T23:29:42.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.907+0000 7f153997a640 1 -- start start 2024-03-28T23:29:42.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.907+0000 7f153997a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f15340c57e0 0x7f153406c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:42.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.907+0000 7f153997a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15340c7ed0 con 0x7f15340c53a0 2024-03-28T23:29:42.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.907+0000 7f153997a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15340c5c90 con 0x7f15341528b0 2024-03-28T23:29:42.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.907+0000 7f153997a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1534155f90 con 0x7f15340c57e0 2024-03-28T23:29:42.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.908+0000 7f15337fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f15340c57e0 0x7f153406c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:42.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.908+0000 7f15327fc640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f15340c53a0 0x7f153406c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48224/0 (socket says 172.21.15.161:48224) 2024-03-28T23:29:42.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.908+0000 7f15337fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f15340c57e0 0x7f153406c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51006/0 (socket says 172.21.15.161:51006) 2024-03-28T23:29:42.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.908+0000 7f15327fc640 1 -- 172.21.15.161:0/3710781294 learned_addr learned my addr 172.21.15.161:0/3710781294 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:42.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.908+0000 7f15337fe640 1 -- 172.21.15.161:0/3710781294 >> v1:172.21.15.170:6789/0 conn(0x7f15341528b0 legacy=0x7f15340c8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:42.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.908+0000 7f15337fe640 1 -- 172.21.15.161:0/3710781294 >> v1:172.21.15.161:6789/0 conn(0x7f15340c53a0 legacy=0x7f153406c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:42.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.908+0000 7f15337fe640 1 -- 172.21.15.161:0/3710781294 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1534152c90 con 0x7f15340c57e0 2024-03-28T23:29:42.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.909+0000 7f15337fe640 1 --2- 172.21.15.161:0/3710781294 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15340c57e0 0x7f153406c850 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f152c009d90 tx=0x7f152c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80ed775748b853f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:42.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.909+0000 7f1531ffb640 1 -- 172.21.15.161:0/3710781294 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f152c039070 con 0x7f15340c57e0 2024-03-28T23:29:42.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.909+0000 7f1531ffb640 1 -- 172.21.15.161:0/3710781294 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f152c02cc60 con 0x7f15340c57e0 2024-03-28T23:29:42.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.909+0000 7f1531ffb640 1 -- 172.21.15.161:0/3710781294 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f152c034660 con 0x7f15340c57e0 2024-03-28T23:29:42.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.909+0000 7f153997a640 1 -- 172.21.15.161:0/3710781294 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15340c57e0 msgr2=0x7f153406c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:42.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.909+0000 7f153997a640 1 --2- 172.21.15.161:0/3710781294 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15340c57e0 0x7f153406c850 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f152c009d90 tx=0x7f152c02beb0 comp rx=0 tx=0).stop 2024-03-28T23:29:42.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.909+0000 7f153997a640 1 -- 172.21.15.161:0/3710781294 shutdown_connections 2024-03-28T23:29:42.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.909+0000 7f153997a640 1 --2- 172.21.15.161:0/3710781294 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15340c57e0 0x7f153406c850 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:42.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.909+0000 7f153997a640 1 -- 172.21.15.161:0/3710781294 >> 172.21.15.161:0/3710781294 conn(0x7f15340ac5b0 msgr2=0x7f15340ac9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:42.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.909+0000 7f153997a640 1 -- 172.21.15.161:0/3710781294 shutdown_connections 2024-03-28T23:29:42.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.909+0000 7f153997a640 1 -- 172.21.15.161:0/3710781294 wait complete. 2024-03-28T23:29:42.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.910+0000 7f153997a640 1 Processor -- start 2024-03-28T23:29:42.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.911+0000 7f153997a640 1 -- start start 2024-03-28T23:29:42.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.911+0000 7f153997a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15340c53a0 0x7f153414f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:42.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.911+0000 7f153997a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15340c57e0 0x7f15341ec400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:42.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.911+0000 7f153997a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15341528b0 0x7f15341ec940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:42.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.911+0000 7f153997a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1534155940 con 0x7f15341528b0 2024-03-28T23:29:42.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.911+0000 7f153997a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1534155660 con 0x7f15340c53a0 2024-03-28T23:29:42.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.911+0000 7f153997a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1534155c20 con 0x7f15340c57e0 2024-03-28T23:29:42.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.911+0000 7f15337fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15341528b0 0x7f15341ec940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:42.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.911+0000 7f15337fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15341528b0 0x7f15341ec940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51010/0 (socket says 172.21.15.161:51010) 2024-03-28T23:29:42.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.911+0000 7f15337fe640 1 -- 172.21.15.161:0/2059297374 learned_addr learned my addr 172.21.15.161:0/2059297374 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:42.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.911+0000 7f15327fc640 1 --2- 172.21.15.161:0/2059297374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15340c57e0 0x7f15341ec400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:42.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.911+0000 7f1532ffd640 1 --2- 172.21.15.161:0/2059297374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15340c53a0 0x7f153414f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:42.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.912+0000 7f15337fe640 1 -- 172.21.15.161:0/2059297374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15340c57e0 msgr2=0x7f15341ec400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:42.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.912+0000 7f15337fe640 1 --2- 172.21.15.161:0/2059297374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15340c57e0 0x7f15341ec400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:42.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.912+0000 7f15337fe640 1 -- 172.21.15.161:0/2059297374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15340c53a0 msgr2=0x7f153414f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:42.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.912+0000 7f15337fe640 1 --2- 172.21.15.161:0/2059297374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15340c53a0 0x7f153414f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:42.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.912+0000 7f15337fe640 1 -- 172.21.15.161:0/2059297374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f153414d7b0 con 0x7f15341528b0 2024-03-28T23:29:42.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.912+0000 7f15337fe640 1 --2- 172.21.15.161:0/2059297374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15341528b0 0x7f15341ec940 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f152c009d90 tx=0x7f152c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:42.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.912+0000 7f151bfff640 1 -- 172.21.15.161:0/2059297374 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f152c039050 con 0x7f15341528b0 2024-03-28T23:29:42.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.912+0000 7f151bfff640 1 -- 172.21.15.161:0/2059297374 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f152c045070 con 0x7f15341528b0 2024-03-28T23:29:42.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.912+0000 7f153997a640 1 -- 172.21.15.161:0/2059297374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f153414da90 con 0x7f15341528b0 2024-03-28T23:29:42.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.912+0000 7f151bfff640 1 -- 172.21.15.161:0/2059297374 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f152c03e830 con 0x7f15341528b0 2024-03-28T23:29:42.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.912+0000 7f153997a640 1 -- 172.21.15.161:0/2059297374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f153414e050 con 0x7f15341528b0 2024-03-28T23:29:42.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.913+0000 7f153997a640 1 -- 172.21.15.161:0/2059297374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f153404f4b0 con 0x7f15341528b0 2024-03-28T23:29:42.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.913+0000 7f151bfff640 1 -- 172.21.15.161:0/2059297374 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 4) ==== 89724+0+0 (secure 0 0 0) 0x7f152c03e9d0 con 0x7f15341528b0 2024-03-28T23:29:42.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.914+0000 7f151bfff640 1 --2- 172.21.15.161:0/2059297374 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f152006b470 0x7f152006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:42.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.914+0000 7f151bfff640 1 -- 172.21.15.161:0/2059297374 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f152c0acc10 con 0x7f15341528b0 2024-03-28T23:29:42.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.914+0000 7f1532ffd640 1 --2- 172.21.15.161:0/2059297374 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f152006b470 0x7f152006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:42.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.914+0000 7f1532ffd640 1 --2- 172.21.15.161:0/2059297374 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f152006b470 0x7f152006d930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1524001780 tx=0x7f1524009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:42.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:42.918+0000 7f151bfff640 1 -- 172.21.15.161:0/2059297374 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f152c07b990 con 0x7f15341528b0 2024-03-28T23:29:43.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.092+0000 7f153997a640 1 -- 172.21.15.161:0/2059297374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f15340ac310 con 0x7f15341528b0 2024-03-28T23:29:43.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.093+0000 7f151bfff640 1 -- 172.21.15.161:0/2059297374 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12920 (secure 0 0 0) 0x7f152c07fa30 con 0x7f15341528b0 2024-03-28T23:29:43.094 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:29:43.094 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":16,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:29:42.326203+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:29:43.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.095+0000 7f153997a640 1 -- 172.21.15.161:0/2059297374 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f152006b470 msgr2=0x7f152006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:43.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.095+0000 7f153997a640 1 --2- 172.21.15.161:0/2059297374 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f152006b470 0x7f152006d930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1524001780 tx=0x7f1524009040 comp rx=0 tx=0).stop 2024-03-28T23:29:43.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.095+0000 7f153997a640 1 -- 172.21.15.161:0/2059297374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15341528b0 msgr2=0x7f15341ec940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:43.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.095+0000 7f153997a640 1 --2- 172.21.15.161:0/2059297374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15341528b0 0x7f15341ec940 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f152c009d90 tx=0x7f152c004a40 comp rx=0 tx=0).stop 2024-03-28T23:29:43.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.095+0000 7f153997a640 1 -- 172.21.15.161:0/2059297374 shutdown_connections 2024-03-28T23:29:43.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.095+0000 7f153997a640 1 --2- 172.21.15.161:0/2059297374 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f152006b470 0x7f152006d930 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.095+0000 7f153997a640 1 --2- 172.21.15.161:0/2059297374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15341528b0 0x7f15341ec940 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.095+0000 7f153997a640 1 --2- 172.21.15.161:0/2059297374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15340c57e0 0x7f15341ec400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.095+0000 7f153997a640 1 --2- 172.21.15.161:0/2059297374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15340c53a0 0x7f153414f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.095+0000 7f153997a640 1 -- 172.21.15.161:0/2059297374 >> 172.21.15.161:0/2059297374 conn(0x7f15340ac5b0 msgr2=0x7f153406c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:43.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.095+0000 7f153997a640 1 -- 172.21.15.161:0/2059297374 shutdown_connections 2024-03-28T23:29:43.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.095+0000 7f153997a640 1 -- 172.21.15.161:0/2059297374 wait complete. 2024-03-28T23:29:43.107 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:29:43.107 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:29:43.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.210+0000 7fbfae763640 1 Processor -- start 2024-03-28T23:29:43.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.210+0000 7fbfae763640 1 -- start start 2024-03-28T23:29:43.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.211+0000 7fbfae763640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fbfa80c6350 0x7fbfa806c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:43.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.211+0000 7fbfae763640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfa8152960 con 0x7fbfa8151e70 2024-03-28T23:29:43.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.211+0000 7fbfae763640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfa80c1e40 con 0x7fbfa80a5fb0 2024-03-28T23:29:43.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.211+0000 7fbfae763640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fbfa80c2120 con 0x7fbfa80c6350 2024-03-28T23:29:43.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.211+0000 7fbfa7fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fbfa8151e70 0x7fbfa8152250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48236/0 (socket says 172.21.15.161:48236) 2024-03-28T23:29:43.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.211+0000 7fbfa7fff640 1 -- 172.21.15.161:0/3561066463 learned_addr learned my addr 172.21.15.161:0/3561066463 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:43.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.211+0000 7fbfa77fe640 1 --2- 172.21.15.161:0/3561066463 >> v2:172.21.15.170:3300/0 conn(0x7fbfa80c6350 0x7fbfa806c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:43.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.211+0000 7fbfa6ffd640 1 -- 172.21.15.161:0/3561066463 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2595761205 0 0) 0x7fbfa8152960 con 0x7fbfa8151e70 2024-03-28T23:29:43.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.211+0000 7fbfa6ffd640 1 -- 172.21.15.161:0/3561066463 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf90003520 con 0x7fbfa8151e70 2024-03-28T23:29:43.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.211+0000 7fbfa77fe640 1 -- 172.21.15.161:0/3561066463 >> v1:172.21.15.170:6789/0 conn(0x7fbfa80a5fb0 legacy=0x7fbfa815c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:43.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.211+0000 7fbfa6ffd640 1 -- 172.21.15.161:0/3561066463 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3274690087 0 0) 0x7fbf90003520 con 0x7fbfa8151e70 2024-03-28T23:29:43.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.211+0000 7fbfa77fe640 1 -- 172.21.15.161:0/3561066463 >> v1:172.21.15.161:6789/0 conn(0x7fbfa8151e70 legacy=0x7fbfa8152250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:43.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.211+0000 7fbfa77fe640 1 -- 172.21.15.161:0/3561066463 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa80c2400 con 0x7fbfa80c6350 2024-03-28T23:29:43.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.212+0000 7fbfa77fe640 1 --2- 172.21.15.161:0/3561066463 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfa80c6350 0x7fbfa806c6c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c0097a0 tx=0x7fbf9c02b600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb29fc22b62bea5b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:43.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.212+0000 7fbfa6ffd640 1 -- 172.21.15.161:0/3561066463 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c037070 con 0x7fbfa80c6350 2024-03-28T23:29:43.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.212+0000 7fbfa6ffd640 1 -- 172.21.15.161:0/3561066463 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf9c030510 con 0x7fbfa80c6350 2024-03-28T23:29:43.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.212+0000 7fbfa6ffd640 1 -- 172.21.15.161:0/3561066463 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf98001670 con 0x7fbfa80c6350 2024-03-28T23:29:43.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.213+0000 7fbfae763640 1 -- 172.21.15.161:0/3561066463 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfa80c6350 msgr2=0x7fbfa806c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:43.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.213+0000 7fbfae763640 1 --2- 172.21.15.161:0/3561066463 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfa80c6350 0x7fbfa806c6c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c0097a0 tx=0x7fbf9c02b600 comp rx=0 tx=0).stop 2024-03-28T23:29:43.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.213+0000 7fbfae763640 1 -- 172.21.15.161:0/3561066463 shutdown_connections 2024-03-28T23:29:43.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.213+0000 7fbfae763640 1 --2- 172.21.15.161:0/3561066463 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfa80c6350 0x7fbfa806c6c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.213+0000 7fbfae763640 1 -- 172.21.15.161:0/3561066463 >> 172.21.15.161:0/3561066463 conn(0x7fbfa80c3390 msgr2=0x7fbfa80c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:43.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.213+0000 7fbfae763640 1 -- 172.21.15.161:0/3561066463 shutdown_connections 2024-03-28T23:29:43.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.213+0000 7fbfae763640 1 -- 172.21.15.161:0/3561066463 wait complete. 2024-03-28T23:29:43.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.214+0000 7fbfae763640 1 Processor -- start 2024-03-28T23:29:43.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.214+0000 7fbfae763640 1 -- start start 2024-03-28T23:29:43.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.214+0000 7fbfae763640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfa80a5fb0 0x7fbfa80c5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:43.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.215+0000 7fbfae763640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbfa80c6350 0x7fbfa8144b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:43.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.215+0000 7fbfae763640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfa8151e70 0x7fbfa81450d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:43.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.215+0000 7fbfae763640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa80c2c00 con 0x7fbfa80c6350 2024-03-28T23:29:43.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.215+0000 7fbfae763640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa814f810 con 0x7fbfa80a5fb0 2024-03-28T23:29:43.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.215+0000 7fbfae763640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbfa80c2d80 con 0x7fbfa8151e70 2024-03-28T23:29:43.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.215+0000 7fbfa77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbfa80c6350 0x7fbfa8144b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:43.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.215+0000 7fbfa77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbfa80c6350 0x7fbfa8144b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51020/0 (socket says 172.21.15.161:51020) 2024-03-28T23:29:43.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.215+0000 7fbfa77fe640 1 -- 172.21.15.161:0/1860207043 learned_addr learned my addr 172.21.15.161:0/1860207043 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:43.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.215+0000 7fbfacd00640 1 --2- 172.21.15.161:0/1860207043 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfa8151e70 0x7fbfa81450d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:43.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.215+0000 7fbfa7fff640 1 --2- 172.21.15.161:0/1860207043 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfa80a5fb0 0x7fbfa80c5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:43.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.215+0000 7fbfa77fe640 1 -- 172.21.15.161:0/1860207043 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfa8151e70 msgr2=0x7fbfa81450d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:43.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.215+0000 7fbfa77fe640 1 --2- 172.21.15.161:0/1860207043 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfa8151e70 0x7fbfa81450d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.215+0000 7fbfa77fe640 1 -- 172.21.15.161:0/1860207043 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfa80a5fb0 msgr2=0x7fbfa80c5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:43.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.215+0000 7fbfa77fe640 1 --2- 172.21.15.161:0/1860207043 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfa80a5fb0 0x7fbfa80c5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.215+0000 7fbfa77fe640 1 -- 172.21.15.161:0/1860207043 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa81eb570 con 0x7fbfa80c6350 2024-03-28T23:29:43.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.215+0000 7fbfa77fe640 1 --2- 172.21.15.161:0/1860207043 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbfa80c6350 0x7fbfa8144b90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c035040 tx=0x7fbf9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:43.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.215+0000 7fbfa4ff9640 1 -- 172.21.15.161:0/1860207043 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c042070 con 0x7fbfa80c6350 2024-03-28T23:29:43.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.215+0000 7fbfa4ff9640 1 -- 172.21.15.161:0/1860207043 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf9c004240 con 0x7fbfa80c6350 2024-03-28T23:29:43.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.215+0000 7fbfae763640 1 -- 172.21.15.161:0/1860207043 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfa81eb850 con 0x7fbfa80c6350 2024-03-28T23:29:43.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.215+0000 7fbfa4ff9640 1 -- 172.21.15.161:0/1860207043 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c037040 con 0x7fbfa80c6350 2024-03-28T23:29:43.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.215+0000 7fbfae763640 1 -- 172.21.15.161:0/1860207043 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfa81ebe10 con 0x7fbfa80c6350 2024-03-28T23:29:43.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.216+0000 7fbfae763640 1 -- 172.21.15.161:0/1860207043 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfa804f4b0 con 0x7fbfa80c6350 2024-03-28T23:29:43.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.217+0000 7fbfa4ff9640 1 -- 172.21.15.161:0/1860207043 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 4) ==== 89724+0+0 (secure 0 0 0) 0x7fbf9c03a040 con 0x7fbfa80c6350 2024-03-28T23:29:43.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.217+0000 7fbfa4ff9640 1 --2- 172.21.15.161:0/1860207043 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf9406b440 0x7fbf9406d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:43.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.217+0000 7fbfa7fff640 1 --2- 172.21.15.161:0/1860207043 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf9406b440 0x7fbf9406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:43.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.217+0000 7fbfa4ff9640 1 -- 172.21.15.161:0/1860207043 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fbf9c0ab7b0 con 0x7fbfa80c6350 2024-03-28T23:29:43.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.217+0000 7fbfa7fff640 1 --2- 172.21.15.161:0/1860207043 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf9406b440 0x7fbf9406d900 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbf98001eb0 tx=0x7fbf98001e20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:43.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.222+0000 7fbfa4ff9640 1 -- 172.21.15.161:0/1860207043 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbf9c07a4a0 con 0x7fbfa80c6350 2024-03-28T23:29:43.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.397+0000 7fbfae763640 1 -- 172.21.15.161:0/1860207043 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbfa80c2f80 con 0x7fbfa80c6350 2024-03-28T23:29:43.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.397+0000 7fbfa4ff9640 1 -- 172.21.15.161:0/1860207043 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7fbf9c07e540 con 0x7fbfa80c6350 2024-03-28T23:29:43.398 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:29:43.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.399+0000 7fbfae763640 1 -- 172.21.15.161:0/1860207043 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf9406b440 msgr2=0x7fbf9406d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:43.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.399+0000 7fbfae763640 1 --2- 172.21.15.161:0/1860207043 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf9406b440 0x7fbf9406d900 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbf98001eb0 tx=0x7fbf98001e20 comp rx=0 tx=0).stop 2024-03-28T23:29:43.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.399+0000 7fbfae763640 1 -- 172.21.15.161:0/1860207043 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbfa80c6350 msgr2=0x7fbfa8144b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:43.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.399+0000 7fbfae763640 1 --2- 172.21.15.161:0/1860207043 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbfa80c6350 0x7fbfa8144b90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c035040 tx=0x7fbf9c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.399+0000 7fbfae763640 1 -- 172.21.15.161:0/1860207043 shutdown_connections 2024-03-28T23:29:43.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.399+0000 7fbfae763640 1 --2- 172.21.15.161:0/1860207043 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf9406b440 0x7fbf9406d900 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.399+0000 7fbfae763640 1 --2- 172.21.15.161:0/1860207043 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbfa80c6350 0x7fbfa8144b90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.399+0000 7fbfae763640 1 --2- 172.21.15.161:0/1860207043 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfa8151e70 0x7fbfa81450d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.399+0000 7fbfae763640 1 --2- 172.21.15.161:0/1860207043 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfa80a5fb0 0x7fbfa80c5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.399+0000 7fbfae763640 1 -- 172.21.15.161:0/1860207043 >> 172.21.15.161:0/1860207043 conn(0x7fbfa80c3390 msgr2=0x7fbfa814e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:43.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.399+0000 7fbfae763640 1 -- 172.21.15.161:0/1860207043 shutdown_connections 2024-03-28T23:29:43.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.399+0000 7fbfae763640 1 -- 172.21.15.161:0/1860207043 wait complete. 2024-03-28T23:29:43.411 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T23:29:43.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.511+0000 7f475d91e640 1 Processor -- start 2024-03-28T23:29:43.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.511+0000 7f475d91e640 1 -- start start 2024-03-28T23:29:43.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.512+0000 7f475d91e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f47580caf60 0x7f47581592e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:43.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.512+0000 7f475d91e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f47580c8970 0x7f47580c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:43.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.512+0000 7f475d91e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47580cb4e0 con 0x7f475814f260 2024-03-28T23:29:43.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.512+0000 7f475d91e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f4758158e50 con 0x7f47580caf60 2024-03-28T23:29:43.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.512+0000 7f475d91e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f47580cb7c0 con 0x7f47580c8970 2024-03-28T23:29:43.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.512+0000 7f474ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f47580caf60 0x7f47581592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:43.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.512+0000 7f4756ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f475814f260 0x7f4758155c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48246/0 (socket says 172.21.15.161:48246) 2024-03-28T23:29:43.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.512+0000 7f4756ffd640 1 -- 172.21.15.161:0/2994939381 learned_addr learned my addr 172.21.15.161:0/2994939381 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:43.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.512+0000 7f47577fe640 1 --2- 172.21.15.161:0/2994939381 >> v2:172.21.15.170:3300/0 conn(0x7f47580c8970 0x7f47580c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:43.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.512+0000 7f474ffff640 1 -- 172.21.15.161:0/2994939381 >> v2:172.21.15.170:3300/0 conn(0x7f47580c8970 msgr2=0x7f47580c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:43.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.512+0000 7f474ffff640 1 --2- 172.21.15.161:0/2994939381 >> v2:172.21.15.170:3300/0 conn(0x7f47580c8970 0x7f47580c53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.512+0000 7f474ffff640 1 -- 172.21.15.161:0/2994939381 >> v1:172.21.15.161:6789/0 conn(0x7f475814f260 legacy=0x7f4758155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:43.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.512+0000 7f47567fc640 1 -- 172.21.15.161:0/2994939381 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3299666138 0 0) 0x7f47580cb4e0 con 0x7f475814f260 2024-03-28T23:29:43.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.512+0000 7f474ffff640 1 -- 172.21.15.161:0/2994939381 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4758158800 con 0x7f47580caf60 2024-03-28T23:29:43.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.512+0000 7f47577fe640 1 --2- 172.21.15.161:0/2994939381 >> v2:172.21.15.170:3300/0 conn(0x7f47580c8970 0x7f47580c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:29:43.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.513+0000 7f474ffff640 1 --2- 172.21.15.161:0/2994939381 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47580caf60 0x7f47581592e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4758158fd0 tx=0x7f474002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61346493e89e3916 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:43.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.513+0000 7f47567fc640 1 -- 172.21.15.161:0/2994939381 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47400315a0 con 0x7f47580caf60 2024-03-28T23:29:43.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.513+0000 7f47567fc640 1 -- 172.21.15.161:0/2994939381 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4740037070 con 0x7f47580caf60 2024-03-28T23:29:43.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.513+0000 7f47567fc640 1 -- 172.21.15.161:0/2994939381 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4740031960 con 0x7f47580caf60 2024-03-28T23:29:43.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.513+0000 7f475d91e640 1 -- 172.21.15.161:0/2994939381 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47580caf60 msgr2=0x7f47581592e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:43.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.513+0000 7f475d91e640 1 --2- 172.21.15.161:0/2994939381 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47580caf60 0x7f47581592e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4758158fd0 tx=0x7f474002b840 comp rx=0 tx=0).stop 2024-03-28T23:29:43.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.514+0000 7f475d91e640 1 -- 172.21.15.161:0/2994939381 shutdown_connections 2024-03-28T23:29:43.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.514+0000 7f475d91e640 1 --2- 172.21.15.161:0/2994939381 >> v2:172.21.15.170:3300/0 conn(0x7f47580c8970 0x7f47580c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.514+0000 7f475d91e640 1 --2- 172.21.15.161:0/2994939381 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47580caf60 0x7f47581592e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.514+0000 7f475d91e640 1 -- 172.21.15.161:0/2994939381 >> 172.21.15.161:0/2994939381 conn(0x7f4758072ca0 msgr2=0x7f47580730b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:43.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.514+0000 7f475d91e640 1 -- 172.21.15.161:0/2994939381 shutdown_connections 2024-03-28T23:29:43.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.514+0000 7f475d91e640 1 -- 172.21.15.161:0/2994939381 wait complete. 2024-03-28T23:29:43.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.515+0000 7f475d91e640 1 Processor -- start 2024-03-28T23:29:43.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.515+0000 7f475d91e640 1 -- start start 2024-03-28T23:29:43.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.515+0000 7f475d91e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47580c8970 0x7f47580abc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:43.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.515+0000 7f475d91e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47580caf60 0x7f47581ead90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:43.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.515+0000 7f475d91e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f475814f260 0x7f47581eb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:43.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.515+0000 7f475d91e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f47580c8dc0 con 0x7f47580caf60 2024-03-28T23:29:43.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.515+0000 7f475d91e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f475814f640 con 0x7f475814f260 2024-03-28T23:29:43.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.515+0000 7f475d91e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f47580c8f40 con 0x7f47580c8970 2024-03-28T23:29:43.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.515+0000 7f474ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47580caf60 0x7f47581ead90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:43.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.515+0000 7f474ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47580caf60 0x7f47581ead90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51038/0 (socket says 172.21.15.161:51038) 2024-03-28T23:29:43.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.515+0000 7f474ffff640 1 -- 172.21.15.161:0/3534459843 learned_addr learned my addr 172.21.15.161:0/3534459843 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:43.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.516+0000 7f4756ffd640 1 --2- 172.21.15.161:0/3534459843 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47580c8970 0x7f47580abc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:43.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.516+0000 7f47577fe640 1 --2- 172.21.15.161:0/3534459843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f475814f260 0x7f47581eb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:43.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.516+0000 7f474ffff640 1 -- 172.21.15.161:0/3534459843 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47580c8970 msgr2=0x7f47580abc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:43.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.516+0000 7f474ffff640 1 --2- 172.21.15.161:0/3534459843 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47580c8970 0x7f47580abc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.516+0000 7f474ffff640 1 -- 172.21.15.161:0/3534459843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f475814f260 msgr2=0x7f47581eb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:43.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.516+0000 7f474ffff640 1 --2- 172.21.15.161:0/3534459843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f475814f260 0x7f47581eb2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.516+0000 7f474ffff640 1 -- 172.21.15.161:0/3534459843 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47581552b0 con 0x7f47580caf60 2024-03-28T23:29:43.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.516+0000 7f474ffff640 1 --2- 172.21.15.161:0/3534459843 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47580caf60 0x7f47581ead90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4740009860 tx=0x7f47400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:43.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.516+0000 7f4754ff9640 1 -- 172.21.15.161:0/3534459843 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4740032840 con 0x7f47580caf60 2024-03-28T23:29:43.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.516+0000 7f4754ff9640 1 -- 172.21.15.161:0/3534459843 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4740032a00 con 0x7f47580caf60 2024-03-28T23:29:43.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.516+0000 7f475d91e640 1 -- 172.21.15.161:0/3534459843 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4758155520 con 0x7f47580caf60 2024-03-28T23:29:43.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.516+0000 7f4754ff9640 1 -- 172.21.15.161:0/3534459843 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4740032df0 con 0x7f47580caf60 2024-03-28T23:29:43.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.516+0000 7f475d91e640 1 -- 172.21.15.161:0/3534459843 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47580ac1d0 con 0x7f47580caf60 2024-03-28T23:29:43.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.517+0000 7f475d91e640 1 -- 172.21.15.161:0/3534459843 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f475804f550 con 0x7f47580caf60 2024-03-28T23:29:43.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.517+0000 7f4754ff9640 1 -- 172.21.15.161:0/3534459843 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 4) ==== 89724+0+0 (secure 0 0 0) 0x7f474004c020 con 0x7f47580caf60 2024-03-28T23:29:43.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.518+0000 7f4754ff9640 1 --2- 172.21.15.161:0/3534459843 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f473806b470 0x7f473806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:43.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.518+0000 7f4756ffd640 1 --2- 172.21.15.161:0/3534459843 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f473806b470 0x7f473806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:43.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.518+0000 7f4754ff9640 1 -- 172.21.15.161:0/3534459843 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f474002f020 con 0x7f47580caf60 2024-03-28T23:29:43.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.518+0000 7f4756ffd640 1 --2- 172.21.15.161:0/3534459843 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f473806b470 0x7f473806d930 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4748000f60 tx=0x7f4748009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:43.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.522+0000 7f4754ff9640 1 -- 172.21.15.161:0/3534459843 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4740040070 con 0x7f47580caf60 2024-03-28T23:29:43.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.534+0000 7f4754ff9640 1 -- 172.21.15.161:0/3534459843 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4740076d90 con 0x7f47580caf60 2024-03-28T23:29:43.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.699+0000 7f475d91e640 1 -- 172.21.15.161:0/3534459843 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f47580695f0 con 0x7f47580caf60 2024-03-28T23:29:43.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.700+0000 7f4754ff9640 1 -- 172.21.15.161:0/3534459843 <== mon.0 v2:172.21.15.161:3300/0 8 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) ==== 84+0+65 (secure 0 0 0) 0x7f474007ae20 con 0x7f47580caf60 2024-03-28T23:29:43.700 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:29:43.701 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2024-03-28T23:29:43.701 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:29:43.701 INFO:teuthology.orchestra.run.smithi161.stdout: "poolnum": 1, 2024-03-28T23:29:43.701 INFO:teuthology.orchestra.run.smithi161.stdout: "poolname": ".mgr" 2024-03-28T23:29:43.701 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:29:43.701 INFO:teuthology.orchestra.run.smithi161.stdout:] 2024-03-28T23:29:43.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.702+0000 7f475d91e640 1 -- 172.21.15.161:0/3534459843 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f473806b470 msgr2=0x7f473806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:43.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.702+0000 7f475d91e640 1 --2- 172.21.15.161:0/3534459843 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f473806b470 0x7f473806d930 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4748000f60 tx=0x7f4748009040 comp rx=0 tx=0).stop 2024-03-28T23:29:43.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.702+0000 7f475d91e640 1 -- 172.21.15.161:0/3534459843 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47580caf60 msgr2=0x7f47581ead90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:43.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.702+0000 7f475d91e640 1 --2- 172.21.15.161:0/3534459843 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47580caf60 0x7f47581ead90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4740009860 tx=0x7f47400046c0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.702+0000 7f475d91e640 1 -- 172.21.15.161:0/3534459843 shutdown_connections 2024-03-28T23:29:43.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.702+0000 7f475d91e640 1 --2- 172.21.15.161:0/3534459843 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f473806b470 0x7f473806d930 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.702+0000 7f475d91e640 1 --2- 172.21.15.161:0/3534459843 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47580caf60 0x7f47581ead90 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.702+0000 7f475d91e640 1 --2- 172.21.15.161:0/3534459843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f475814f260 0x7f47581eb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.702+0000 7f475d91e640 1 --2- 172.21.15.161:0/3534459843 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47580c8970 0x7f47580abc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.702+0000 7f475d91e640 1 -- 172.21.15.161:0/3534459843 >> 172.21.15.161:0/3534459843 conn(0x7f4758072ca0 msgr2=0x7f475814d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:43.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.702+0000 7f475d91e640 1 -- 172.21.15.161:0/3534459843 shutdown_connections 2024-03-28T23:29:43.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.702+0000 7f475d91e640 1 -- 172.21.15.161:0/3534459843 wait complete. 2024-03-28T23:29:43.713 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-28T23:29:43.713 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-28T23:29:43.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.817+0000 7fa93e16c640 1 Processor -- start 2024-03-28T23:29:43.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.817+0000 7fa93e16c640 1 -- start start 2024-03-28T23:29:43.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.818+0000 7fa93e16c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa9380c30c0 0x7fa938152880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:43.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.818+0000 7fa93e16c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa9380c19b0 0x7fa9380c1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:43.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.818+0000 7fa93e16c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9380c3540 con 0x7fa9380ac430 2024-03-28T23:29:43.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.818+0000 7fa93e16c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa938152dc0 con 0x7fa9380c30c0 2024-03-28T23:29:43.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.818+0000 7fa93e16c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fa9380c2300 con 0x7fa9380c19b0 2024-03-28T23:29:43.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.818+0000 7fa936ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa9380c30c0 0x7fa938152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:43.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.818+0000 7fa936ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa9380c30c0 0x7fa938152880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51044/0 (socket says 172.21.15.161:51044) 2024-03-28T23:29:43.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.818+0000 7fa936ffd640 1 -- 172.21.15.161:0/4143578319 learned_addr learned my addr 172.21.15.161:0/4143578319 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:43.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.818+0000 7fa937fff640 1 --2- 172.21.15.161:0/4143578319 >> v2:172.21.15.170:3300/0 conn(0x7fa9380c19b0 0x7fa9380c1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:43.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.818+0000 7fa936ffd640 1 -- 172.21.15.161:0/4143578319 >> v2:172.21.15.170:3300/0 conn(0x7fa9380c19b0 msgr2=0x7fa9380c1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:43.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.818+0000 7fa936ffd640 1 --2- 172.21.15.161:0/4143578319 >> v2:172.21.15.170:3300/0 conn(0x7fa9380c19b0 0x7fa9380c1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.818+0000 7fa936ffd640 1 -- 172.21.15.161:0/4143578319 >> v1:172.21.15.170:6789/0 conn(0x7fa9380ac430 legacy=0x7fa9380c24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:43.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.818+0000 7fa936ffd640 1 -- 172.21.15.161:0/4143578319 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9380c3820 con 0x7fa9380c30c0 2024-03-28T23:29:43.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.819+0000 7fa936ffd640 1 --2- 172.21.15.161:0/4143578319 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9380c30c0 0x7fa938152880 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa9380c61b0 tx=0x7fa92c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e02def6dcecff63d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:43.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.819+0000 7fa9367fc640 1 -- 172.21.15.161:0/4143578319 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c035020 con 0x7fa9380c30c0 2024-03-28T23:29:43.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.819+0000 7fa9367fc640 1 -- 172.21.15.161:0/4143578319 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa92c031870 con 0x7fa9380c30c0 2024-03-28T23:29:43.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.819+0000 7fa9367fc640 1 -- 172.21.15.161:0/4143578319 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c031c40 con 0x7fa9380c30c0 2024-03-28T23:29:43.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.820+0000 7fa93e16c640 1 -- 172.21.15.161:0/4143578319 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9380c30c0 msgr2=0x7fa938152880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:43.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.820+0000 7fa93e16c640 1 --2- 172.21.15.161:0/4143578319 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9380c30c0 0x7fa938152880 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa9380c61b0 tx=0x7fa92c02b910 comp rx=0 tx=0).stop 2024-03-28T23:29:43.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.820+0000 7fa93e16c640 1 -- 172.21.15.161:0/4143578319 shutdown_connections 2024-03-28T23:29:43.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.820+0000 7fa93e16c640 1 --2- 172.21.15.161:0/4143578319 >> v2:172.21.15.170:3300/0 conn(0x7fa9380c19b0 0x7fa9380c1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.820+0000 7fa93e16c640 1 --2- 172.21.15.161:0/4143578319 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9380c30c0 0x7fa938152880 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.820+0000 7fa93e16c640 1 -- 172.21.15.161:0/4143578319 >> 172.21.15.161:0/4143578319 conn(0x7fa93806b150 msgr2=0x7fa93806b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:43.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.820+0000 7fa93e16c640 1 -- 172.21.15.161:0/4143578319 shutdown_connections 2024-03-28T23:29:43.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.820+0000 7fa93e16c640 1 -- 172.21.15.161:0/4143578319 wait complete. 2024-03-28T23:29:43.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.821+0000 7fa93e16c640 1 Processor -- start 2024-03-28T23:29:43.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.821+0000 7fa93e16c640 1 -- start start 2024-03-28T23:29:43.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.822+0000 7fa93e16c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9380ac430 0x7fa9381463e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:43.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.822+0000 7fa93e16c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9380c19b0 0x7fa9381dd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:43.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.822+0000 7fa93e16c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9380c30c0 0x7fa938144be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:43.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.822+0000 7fa93e16c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa9380c1490 con 0x7fa9380ac430 2024-03-28T23:29:43.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.822+0000 7fa93e16c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa9380c11b0 con 0x7fa9380c19b0 2024-03-28T23:29:43.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.822+0000 7fa93e16c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa9380c6440 con 0x7fa9380c30c0 2024-03-28T23:29:43.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.822+0000 7fa9377fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9380ac430 0x7fa9381463e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:43.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.822+0000 7fa9377fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9380ac430 0x7fa9381463e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51046/0 (socket says 172.21.15.161:51046) 2024-03-28T23:29:43.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.822+0000 7fa9377fe640 1 -- 172.21.15.161:0/980735653 learned_addr learned my addr 172.21.15.161:0/980735653 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:43.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.822+0000 7fa937fff640 1 --2- 172.21.15.161:0/980735653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9380c30c0 0x7fa938144be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:43.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.822+0000 7fa936ffd640 1 --2- 172.21.15.161:0/980735653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9380c19b0 0x7fa9381dd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:43.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.822+0000 7fa9377fe640 1 -- 172.21.15.161:0/980735653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9380c30c0 msgr2=0x7fa938144be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:43.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.822+0000 7fa9377fe640 1 --2- 172.21.15.161:0/980735653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9380c30c0 0x7fa938144be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.822+0000 7fa9377fe640 1 -- 172.21.15.161:0/980735653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9380c19b0 msgr2=0x7fa9381dd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:43.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.822+0000 7fa9377fe640 1 --2- 172.21.15.161:0/980735653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9380c19b0 0x7fa9381dd8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:43.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.822+0000 7fa9377fe640 1 -- 172.21.15.161:0/980735653 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9381e9b70 con 0x7fa9380ac430 2024-03-28T23:29:43.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.822+0000 7fa936ffd640 1 --2- 172.21.15.161:0/980735653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9380c19b0 0x7fa9381dd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:29:43.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.822+0000 7fa9377fe640 1 --2- 172.21.15.161:0/980735653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9380ac430 0x7fa9381463e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa92800c3c0 tx=0x7fa92800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:43.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.822+0000 7fa91ffff640 1 -- 172.21.15.161:0/980735653 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9280052d0 con 0x7fa9380ac430 2024-03-28T23:29:43.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.822+0000 7fa91ffff640 1 -- 172.21.15.161:0/980735653 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa928005490 con 0x7fa9380ac430 2024-03-28T23:29:43.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.822+0000 7fa93e16c640 1 -- 172.21.15.161:0/980735653 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9381e8db0 con 0x7fa9380ac430 2024-03-28T23:29:43.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.822+0000 7fa91ffff640 1 -- 172.21.15.161:0/980735653 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa928007260 con 0x7fa9380ac430 2024-03-28T23:29:43.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.822+0000 7fa93e16c640 1 -- 172.21.15.161:0/980735653 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9381e9370 con 0x7fa9380ac430 2024-03-28T23:29:43.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.823+0000 7fa93e16c640 1 -- 172.21.15.161:0/980735653 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa938068f90 con 0x7fa9380ac430 2024-03-28T23:29:43.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.824+0000 7fa91ffff640 1 -- 172.21.15.161:0/980735653 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa928007400 con 0x7fa9380ac430 2024-03-28T23:29:43.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.824+0000 7fa91ffff640 1 --2- 172.21.15.161:0/980735653 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa92406b4d0 0x7fa92406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:43.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.824+0000 7fa91ffff640 1 -- 172.21.15.161:0/980735653 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fa9280892c0 con 0x7fa9380ac430 2024-03-28T23:29:43.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.824+0000 7fa936ffd640 1 --2- 172.21.15.161:0/980735653 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa92406b4d0 0x7fa92406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:43.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.824+0000 7fa936ffd640 1 --2- 172.21.15.161:0/980735653 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa92406b4d0 0x7fa92406d990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa92c036000 tx=0x7fa92c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:43.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:43.828+0000 7fa91ffff640 1 -- 172.21.15.161:0/980735653 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa928058520 con 0x7fa9380ac430 2024-03-28T23:29:44.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.002+0000 7fa93e16c640 1 -- 172.21.15.161:0/980735653 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fa938152a20 con 0x7fa9380ac430 2024-03-28T23:29:44.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.535+0000 7fa91ffff640 1 -- 172.21.15.161:0/980735653 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) ==== 138+0+0 (secure 0 0 0) 0x7fa928057e20 con 0x7fa9380ac430 2024-03-28T23:29:44.536 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_metadata' created 2024-03-28T23:29:44.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.538+0000 7fa93e16c640 1 -- 172.21.15.161:0/980735653 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa92406b4d0 msgr2=0x7fa92406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:44.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.538+0000 7fa93e16c640 1 --2- 172.21.15.161:0/980735653 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa92406b4d0 0x7fa92406d990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa92c036000 tx=0x7fa92c008030 comp rx=0 tx=0).stop 2024-03-28T23:29:44.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.538+0000 7fa93e16c640 1 -- 172.21.15.161:0/980735653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9380ac430 msgr2=0x7fa9381463e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:44.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.538+0000 7fa93e16c640 1 --2- 172.21.15.161:0/980735653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9380ac430 0x7fa9381463e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa92800c3c0 tx=0x7fa92800c880 comp rx=0 tx=0).stop 2024-03-28T23:29:44.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.538+0000 7fa93e16c640 1 -- 172.21.15.161:0/980735653 shutdown_connections 2024-03-28T23:29:44.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.538+0000 7fa93e16c640 1 --2- 172.21.15.161:0/980735653 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa92406b4d0 0x7fa92406d990 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:44.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.538+0000 7fa93e16c640 1 --2- 172.21.15.161:0/980735653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9380c30c0 0x7fa938144be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:44.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.538+0000 7fa93e16c640 1 --2- 172.21.15.161:0/980735653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9380c19b0 0x7fa9381dd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:44.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.538+0000 7fa93e16c640 1 --2- 172.21.15.161:0/980735653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9380ac430 0x7fa9381463e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:44.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.538+0000 7fa93e16c640 1 -- 172.21.15.161:0/980735653 >> 172.21.15.161:0/980735653 conn(0x7fa93806b150 msgr2=0x7fa938150780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:44.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.538+0000 7fa93e16c640 1 -- 172.21.15.161:0/980735653 shutdown_connections 2024-03-28T23:29:44.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.539+0000 7fa93e16c640 1 -- 172.21.15.161:0/980735653 wait complete. 2024-03-28T23:29:44.555 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-28T23:29:44.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.666+0000 7fd095169640 1 Processor -- start 2024-03-28T23:29:44.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.666+0000 7fd095169640 1 -- start start 2024-03-28T23:29:44.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.667+0000 7fd095169640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd09006cdd0 0x7fd0901524c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:44.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.667+0000 7fd095169640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd09006ba50 0x7fd090152a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:44.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.667+0000 7fd095169640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0900c2f10 con 0x7fd09014f5c0 2024-03-28T23:29:44.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.667+0000 7fd095169640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd0900c74d0 con 0x7fd09006cdd0 2024-03-28T23:29:44.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.667+0000 7fd095169640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd0900c6fa0 con 0x7fd09006ba50 2024-03-28T23:29:44.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.667+0000 7fd08ed9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd09014f5c0 0x7fd0900c2150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48252/0 (socket says 172.21.15.161:48252) 2024-03-28T23:29:44.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.667+0000 7fd08ed9d640 1 -- 172.21.15.161:0/2129097707 learned_addr learned my addr 172.21.15.161:0/2129097707 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:44.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.667+0000 7fd08dd9b640 1 -- 172.21.15.161:0/2129097707 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 905696938 0 0) 0x7fd080001110 con 0x7fd09014f5c0 2024-03-28T23:29:44.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.667+0000 7fd08f59e640 1 --2- 172.21.15.161:0/2129097707 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd09006ba50 0x7fd090152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:44.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.667+0000 7fd08e59c640 1 --2- 172.21.15.161:0/2129097707 >> v2:172.21.15.170:3300/0 conn(0x7fd09006cdd0 0x7fd0901524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:44.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.667+0000 7fd08dd9b640 1 -- 172.21.15.161:0/2129097707 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd078003520 con 0x7fd09014f5c0 2024-03-28T23:29:44.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.667+0000 7fd08dd9b640 1 -- 172.21.15.161:0/2129097707 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3846296675 0 0) 0x7fd078003520 con 0x7fd09014f5c0 2024-03-28T23:29:44.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.667+0000 7fd08dd9b640 1 -- 172.21.15.161:0/2129097707 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd09006ba50 msgr2=0x7fd090152a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:44.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.667+0000 7fd08dd9b640 1 --2- 172.21.15.161:0/2129097707 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd09006ba50 0x7fd090152a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:44.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.667+0000 7fd08dd9b640 1 -- 172.21.15.161:0/2129097707 >> v2:172.21.15.170:3300/0 conn(0x7fd09006cdd0 msgr2=0x7fd0901524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:44.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.667+0000 7fd08dd9b640 1 --2- 172.21.15.161:0/2129097707 >> v2:172.21.15.170:3300/0 conn(0x7fd09006cdd0 0x7fd0901524c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:44.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.667+0000 7fd08dd9b640 1 -- 172.21.15.161:0/2129097707 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09014f9a0 con 0x7fd09014f5c0 2024-03-28T23:29:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.667+0000 7fd08dd9b640 1 -- 172.21.15.161:0/2129097707 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fd080001840 con 0x7fd09014f5c0 2024-03-28T23:29:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.668+0000 7fd08dd9b640 1 -- 172.21.15.161:0/2129097707 >> v1:172.21.15.161:6789/0 conn(0x7fd09014f5c0 legacy=0x7fd0900c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.668+0000 7fd08dd9b640 1 --2- 172.21.15.161:0/2129097707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd078003a90 0x7fd078003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.668+0000 7fd08dd9b640 1 --2- 172.21.15.161:0/2129097707 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd078004800 0x7fd078004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.668+0000 7fd08dd9b640 1 --2- 172.21.15.161:0/2129097707 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd09006ba50 0x7fd0780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.668+0000 7fd08dd9b640 1 -- 172.21.15.161:0/2129097707 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd0900c74d0 con 0x7fd078003a90 2024-03-28T23:29:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.668+0000 7fd08dd9b640 1 -- 172.21.15.161:0/2129097707 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd0900c6fa0 con 0x7fd078004800 2024-03-28T23:29:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.668+0000 7fd08dd9b640 1 -- 172.21.15.161:0/2129097707 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd07800ba40 con 0x7fd09006ba50 2024-03-28T23:29:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.668+0000 7fd08ed9d640 1 --2- 172.21.15.161:0/2129097707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd078003a90 0x7fd078003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.668+0000 7fd08e59c640 1 --2- 172.21.15.161:0/2129097707 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd078004800 0x7fd078004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.668+0000 7fd08f59e640 1 --2- 172.21.15.161:0/2129097707 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd09006ba50 0x7fd0780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.668+0000 7fd08ed9d640 1 -- 172.21.15.161:0/2129097707 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd09006ba50 msgr2=0x7fd0780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.668+0000 7fd08ed9d640 1 --2- 172.21.15.161:0/2129097707 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd09006ba50 0x7fd0780051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.668+0000 7fd08ed9d640 1 -- 172.21.15.161:0/2129097707 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd078004800 msgr2=0x7fd078004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.668+0000 7fd08ed9d640 1 --2- 172.21.15.161:0/2129097707 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd078004800 0x7fd078004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.668+0000 7fd08ed9d640 1 -- 172.21.15.161:0/2129097707 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd078003520 con 0x7fd078003a90 2024-03-28T23:29:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.669+0000 7fd08ed9d640 1 --2- 172.21.15.161:0/2129097707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd078003a90 0x7fd078003eb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd08000c490 tx=0x7fd078005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d9253850d0aec15 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.669+0000 7fd08dd9b640 1 -- 172.21.15.161:0/2129097707 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08000c0e0 con 0x7fd078003a90 2024-03-28T23:29:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.669+0000 7fd08dd9b640 1 -- 172.21.15.161:0/2129097707 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd080030070 con 0x7fd078003a90 2024-03-28T23:29:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.669+0000 7fd08dd9b640 1 -- 172.21.15.161:0/2129097707 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0800058d0 con 0x7fd078003a90 2024-03-28T23:29:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.669+0000 7fd095169640 1 -- 172.21.15.161:0/2129097707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd078003a90 msgr2=0x7fd078003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.669+0000 7fd095169640 1 --2- 172.21.15.161:0/2129097707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd078003a90 0x7fd078003eb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd08000c490 tx=0x7fd078005830 comp rx=0 tx=0).stop 2024-03-28T23:29:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.669+0000 7fd08f59e640 1 -- 172.21.15.161:0/2129097707 reap_dead start 2024-03-28T23:29:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.669+0000 7fd095169640 1 -- 172.21.15.161:0/2129097707 shutdown_connections 2024-03-28T23:29:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.669+0000 7fd095169640 1 -- 172.21.15.161:0/2129097707 >> 172.21.15.161:0/2129097707 conn(0x7fd0900a83f0 msgr2=0x7fd0900a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.669+0000 7fd095169640 1 -- 172.21.15.161:0/2129097707 shutdown_connections 2024-03-28T23:29:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.669+0000 7fd095169640 1 -- 172.21.15.161:0/2129097707 wait complete. 2024-03-28T23:29:44.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.671+0000 7fd095169640 1 Processor -- start 2024-03-28T23:29:44.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.671+0000 7fd095169640 1 -- start start 2024-03-28T23:29:44.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.671+0000 7fd095169640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd078003a90 0x7fd090148a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:44.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.671+0000 7fd095169640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0901ea020 0x7fd090147ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:44.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.671+0000 7fd095169640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0901ea4a0 0x7fd090148020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:44.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.672+0000 7fd095169640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd09006c140 con 0x7fd0901ea4a0 2024-03-28T23:29:44.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.672+0000 7fd095169640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd090152030 con 0x7fd0901ea020 2024-03-28T23:29:44.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.672+0000 7fd095169640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd09006c3e0 con 0x7fd078003a90 2024-03-28T23:29:44.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.672+0000 7fd08f59e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0901ea4a0 0x7fd090148020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:44.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.672+0000 7fd08f59e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0901ea4a0 0x7fd090148020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51064/0 (socket says 172.21.15.161:51064) 2024-03-28T23:29:44.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.672+0000 7fd08f59e640 1 -- 172.21.15.161:0/1016703870 learned_addr learned my addr 172.21.15.161:0/1016703870 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:44.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.672+0000 7fd08ed9d640 1 --2- 172.21.15.161:0/1016703870 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd078003a90 0x7fd090148a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:44.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.672+0000 7fd08e59c640 1 --2- 172.21.15.161:0/1016703870 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0901ea020 0x7fd090147ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:44.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.672+0000 7fd08f59e640 1 -- 172.21.15.161:0/1016703870 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd078003a90 msgr2=0x7fd090148a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:44.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.672+0000 7fd08f59e640 1 --2- 172.21.15.161:0/1016703870 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd078003a90 0x7fd090148a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:44.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.672+0000 7fd08f59e640 1 -- 172.21.15.161:0/1016703870 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0901ea020 msgr2=0x7fd090147ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:44.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.672+0000 7fd08f59e640 1 --2- 172.21.15.161:0/1016703870 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0901ea020 0x7fd090147ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:44.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.672+0000 7fd08f59e640 1 -- 172.21.15.161:0/1016703870 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd090148fd0 con 0x7fd0901ea4a0 2024-03-28T23:29:44.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.672+0000 7fd08f59e640 1 --2- 172.21.15.161:0/1016703870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0901ea4a0 0x7fd090148020 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd0900a7cf0 tx=0x7fd08800f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:44.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.672+0000 7fd0777fe640 1 -- 172.21.15.161:0/1016703870 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08800fce0 con 0x7fd0901ea4a0 2024-03-28T23:29:44.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.672+0000 7fd0777fe640 1 -- 172.21.15.161:0/1016703870 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd08800fe80 con 0x7fd0901ea4a0 2024-03-28T23:29:44.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.672+0000 7fd095169640 1 -- 172.21.15.161:0/1016703870 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0901533d0 con 0x7fd0901ea4a0 2024-03-28T23:29:44.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.672+0000 7fd0777fe640 1 -- 172.21.15.161:0/1016703870 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088015670 con 0x7fd0901ea4a0 2024-03-28T23:29:44.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.672+0000 7fd095169640 1 -- 172.21.15.161:0/1016703870 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0901eac40 con 0x7fd0901ea4a0 2024-03-28T23:29:44.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.673+0000 7fd095169640 1 -- 172.21.15.161:0/1016703870 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd09004f4b0 con 0x7fd0901ea4a0 2024-03-28T23:29:44.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.674+0000 7fd0777fe640 1 -- 172.21.15.161:0/1016703870 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd088015830 con 0x7fd0901ea4a0 2024-03-28T23:29:44.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.674+0000 7fd0777fe640 1 --2- 172.21.15.161:0/1016703870 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd07c06b470 0x7fd07c06d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:44.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.674+0000 7fd08ed9d640 1 --2- 172.21.15.161:0/1016703870 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd07c06b470 0x7fd07c06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:44.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.674+0000 7fd0777fe640 1 -- 172.21.15.161:0/1016703870 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 5227+0+0 (secure 0 0 0) 0x7fd08808acd0 con 0x7fd0901ea4a0 2024-03-28T23:29:44.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.675+0000 7fd08ed9d640 1 --2- 172.21.15.161:0/1016703870 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd07c06b470 0x7fd07c06d930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd080001050 tx=0x7fd080007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:44.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.679+0000 7fd0777fe640 1 -- 172.21.15.161:0/1016703870 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd088019030 con 0x7fd0901ea4a0 2024-03-28T23:29:44.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:44.853+0000 7fd095169640 1 -- 172.21.15.161:0/1016703870 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fd090071350 con 0x7fd0901ea4a0 2024-03-28T23:29:45.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.541+0000 7fd0777fe640 1 -- 172.21.15.161:0/1016703870 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) ==== 171+0+0 (secure 0 0 0) 0x7fd088010040 con 0x7fd0901ea4a0 2024-03-28T23:29:45.542 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data' created 2024-03-28T23:29:45.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.547+0000 7fd095169640 1 -- 172.21.15.161:0/1016703870 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd07c06b470 msgr2=0x7fd07c06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:45.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.547+0000 7fd095169640 1 --2- 172.21.15.161:0/1016703870 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd07c06b470 0x7fd07c06d930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd080001050 tx=0x7fd080007000 comp rx=0 tx=0).stop 2024-03-28T23:29:45.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.547+0000 7fd095169640 1 -- 172.21.15.161:0/1016703870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0901ea4a0 msgr2=0x7fd090148020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:45.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.547+0000 7fd095169640 1 --2- 172.21.15.161:0/1016703870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0901ea4a0 0x7fd090148020 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd0900a7cf0 tx=0x7fd08800f1b0 comp rx=0 tx=0).stop 2024-03-28T23:29:45.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.547+0000 7fd095169640 1 -- 172.21.15.161:0/1016703870 shutdown_connections 2024-03-28T23:29:45.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.547+0000 7fd095169640 1 --2- 172.21.15.161:0/1016703870 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd07c06b470 0x7fd07c06d930 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:45.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.547+0000 7fd095169640 1 --2- 172.21.15.161:0/1016703870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0901ea4a0 0x7fd090148020 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:45.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.547+0000 7fd095169640 1 --2- 172.21.15.161:0/1016703870 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0901ea020 0x7fd090147ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:45.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.547+0000 7fd095169640 1 --2- 172.21.15.161:0/1016703870 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd078003a90 0x7fd090148a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:45.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.547+0000 7fd095169640 1 -- 172.21.15.161:0/1016703870 >> 172.21.15.161:0/1016703870 conn(0x7fd0900a83f0 msgr2=0x7fd0900c1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:45.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.549+0000 7fd095169640 1 -- 172.21.15.161:0/1016703870 shutdown_connections 2024-03-28T23:29:45.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.552+0000 7fd095169640 1 -- 172.21.15.161:0/1016703870 wait complete. 2024-03-28T23:29:45.575 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-28T23:29:45.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.682+0000 7f30f20f4640 1 Processor -- start 2024-03-28T23:29:45.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.682+0000 7f30f20f4640 1 -- start start 2024-03-28T23:29:45.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.682+0000 7f30f20f4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f30ec072fe0 0x7f30ec0c8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:45.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.682+0000 7f30f20f4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30ec156060 con 0x7f30ec158860 2024-03-28T23:29:45.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.682+0000 7f30f20f4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30ec072870 con 0x7f30ec0c4e40 2024-03-28T23:29:45.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.682+0000 7f30f20f4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f30ec072b50 con 0x7f30ec072fe0 2024-03-28T23:29:45.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.682+0000 7f30eaffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f30ec072fe0 0x7f30ec0c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:45.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.682+0000 7f30ebfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f30ec158860 0x7f30ec158cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48262/0 (socket says 172.21.15.161:48262) 2024-03-28T23:29:45.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.683+0000 7f30ebfff640 1 -- 172.21.15.161:0/68886064 learned_addr learned my addr 172.21.15.161:0/68886064 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:45.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.682+0000 7f30eaffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f30ec072fe0 0x7f30ec0c8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51066/0 (socket says 172.21.15.161:51066) 2024-03-28T23:29:45.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.683+0000 7f30eaffd640 1 -- 172.21.15.161:0/68886064 >> v1:172.21.15.170:6789/0 conn(0x7f30ec0c4e40 legacy=0x7f30ec0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:45.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.683+0000 7f30eaffd640 1 -- 172.21.15.161:0/68886064 >> v1:172.21.15.161:6789/0 conn(0x7f30ec158860 legacy=0x7f30ec158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:45.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.683+0000 7f30eaffd640 1 -- 172.21.15.161:0/68886064 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30ec155cf0 con 0x7f30ec072fe0 2024-03-28T23:29:45.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.683+0000 7f30ea7fc640 1 -- 172.21.15.161:0/68886064 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1049432491 0 0) 0x7f30ec156060 con 0x7f30ec158860 2024-03-28T23:29:45.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.683+0000 7f30eaffd640 1 --2- 172.21.15.161:0/68886064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30ec072fe0 0x7f30ec0c8340 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f30e00097a0 tx=0x7f30e002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c49aac95fb1e002 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:45.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.684+0000 7f30ea7fc640 1 -- 172.21.15.161:0/68886064 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e0035020 con 0x7f30ec072fe0 2024-03-28T23:29:45.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.684+0000 7f30ea7fc640 1 -- 172.21.15.161:0/68886064 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30e0031830 con 0x7f30ec072fe0 2024-03-28T23:29:45.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.684+0000 7f30ea7fc640 1 -- 172.21.15.161:0/68886064 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e0031c00 con 0x7f30ec072fe0 2024-03-28T23:29:45.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.684+0000 7f30f20f4640 1 -- 172.21.15.161:0/68886064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30ec072fe0 msgr2=0x7f30ec0c8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:45.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.684+0000 7f30f20f4640 1 --2- 172.21.15.161:0/68886064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30ec072fe0 0x7f30ec0c8340 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f30e00097a0 tx=0x7f30e002b8d0 comp rx=0 tx=0).stop 2024-03-28T23:29:45.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.684+0000 7f30f20f4640 1 -- 172.21.15.161:0/68886064 shutdown_connections 2024-03-28T23:29:45.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.684+0000 7f30f20f4640 1 --2- 172.21.15.161:0/68886064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30ec072fe0 0x7f30ec0c8340 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:45.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.684+0000 7f30f20f4640 1 -- 172.21.15.161:0/68886064 >> 172.21.15.161:0/68886064 conn(0x7f30ec0c88b0 msgr2=0x7f30ec0c8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:45.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.684+0000 7f30f20f4640 1 -- 172.21.15.161:0/68886064 shutdown_connections 2024-03-28T23:29:45.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.684+0000 7f30f20f4640 1 -- 172.21.15.161:0/68886064 wait complete. 2024-03-28T23:29:45.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.685+0000 7f30f20f4640 1 Processor -- start 2024-03-28T23:29:45.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.685+0000 7f30f20f4640 1 -- start start 2024-03-28T23:29:45.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.686+0000 7f30f20f4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30ec072fe0 0x7f30ec1ee9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:45.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.686+0000 7f30f20f4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30ec0c4e40 0x7f30ec06d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:45.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.686+0000 7f30f20f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30ec158860 0x7f30ec0a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:45.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.686+0000 7f30f20f4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec0c6eb0 con 0x7f30ec072fe0 2024-03-28T23:29:45.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.686+0000 7f30f20f4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec0c6d30 con 0x7f30ec0c4e40 2024-03-28T23:29:45.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.686+0000 7f30f20f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f30ec160c00 con 0x7f30ec158860 2024-03-28T23:29:45.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.686+0000 7f30eb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30ec072fe0 0x7f30ec1ee9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:45.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.686+0000 7f30eb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30ec072fe0 0x7f30ec1ee9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51078/0 (socket says 172.21.15.161:51078) 2024-03-28T23:29:45.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.686+0000 7f30eb7fe640 1 -- 172.21.15.161:0/155170806 learned_addr learned my addr 172.21.15.161:0/155170806 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:45.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.686+0000 7f30eb7fe640 1 -- 172.21.15.161:0/155170806 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30ec158860 msgr2=0x7f30ec0a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:45.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.686+0000 7f30eb7fe640 1 --2- 172.21.15.161:0/155170806 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30ec158860 0x7f30ec0a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:45.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.686+0000 7f30eb7fe640 1 -- 172.21.15.161:0/155170806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30ec0c4e40 msgr2=0x7f30ec06d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:45.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.686+0000 7f30eb7fe640 1 --2- 172.21.15.161:0/155170806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30ec0c4e40 0x7f30ec06d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:45.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.686+0000 7f30eb7fe640 1 -- 172.21.15.161:0/155170806 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30ec14fb20 con 0x7f30ec072fe0 2024-03-28T23:29:45.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.687+0000 7f30eb7fe640 1 --2- 172.21.15.161:0/155170806 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30ec072fe0 0x7f30ec1ee9e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f30dc00b3f0 tx=0x7f30dc00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:45.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.687+0000 7f30d3fff640 1 -- 172.21.15.161:0/155170806 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30dc015020 con 0x7f30ec072fe0 2024-03-28T23:29:45.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.687+0000 7f30d3fff640 1 -- 172.21.15.161:0/155170806 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30dc011830 con 0x7f30ec072fe0 2024-03-28T23:29:45.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.687+0000 7f30f20f4640 1 -- 172.21.15.161:0/155170806 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30ec0a8130 con 0x7f30ec072fe0 2024-03-28T23:29:45.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.687+0000 7f30d3fff640 1 -- 172.21.15.161:0/155170806 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30dc011c20 con 0x7f30ec072fe0 2024-03-28T23:29:45.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.687+0000 7f30f20f4640 1 -- 172.21.15.161:0/155170806 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30ec148140 con 0x7f30ec072fe0 2024-03-28T23:29:45.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.688+0000 7f30f20f4640 1 -- 172.21.15.161:0/155170806 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30ec04f4b0 con 0x7f30ec072fe0 2024-03-28T23:29:45.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.688+0000 7f30d3fff640 1 -- 172.21.15.161:0/155170806 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f30dc018020 con 0x7f30ec072fe0 2024-03-28T23:29:45.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.688+0000 7f30d3fff640 1 --2- 172.21.15.161:0/155170806 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f30d806b500 0x7f30d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:45.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.689+0000 7f30d3fff640 1 -- 172.21.15.161:0/155170806 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5609+0+0 (secure 0 0 0) 0x7f30dc089d10 con 0x7f30ec072fe0 2024-03-28T23:29:45.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.689+0000 7f30eaffd640 1 --2- 172.21.15.161:0/155170806 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f30d806b500 0x7f30d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:45.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.689+0000 7f30eaffd640 1 --2- 172.21.15.161:0/155170806 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f30d806b500 0x7f30d806d9c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f30e0036000 tx=0x7f30e0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:45.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.693+0000 7f30d3fff640 1 -- 172.21.15.161:0/155170806 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f30dc01f030 con 0x7f30ec072fe0 2024-03-28T23:29:45.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:45.902+0000 7f30f20f4640 1 -- 172.21.15.161:0/155170806 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f30ec0c5680 con 0x7f30ec072fe0 2024-03-28T23:29:46.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.542+0000 7f30d3fff640 1 -- 172.21.15.161:0/155170806 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f30dc0172c0 con 0x7f30ec072fe0 2024-03-28T23:29:46.543 INFO:teuthology.orchestra.run.smithi161.stderr:new fs with metadata pool 2 and data pool 3 2024-03-28T23:29:46.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.546+0000 7f30f20f4640 1 -- 172.21.15.161:0/155170806 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f30d806b500 msgr2=0x7f30d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:46.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.546+0000 7f30f20f4640 1 --2- 172.21.15.161:0/155170806 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f30d806b500 0x7f30d806d9c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f30e0036000 tx=0x7f30e0030000 comp rx=0 tx=0).stop 2024-03-28T23:29:46.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.546+0000 7f30f20f4640 1 -- 172.21.15.161:0/155170806 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30ec072fe0 msgr2=0x7f30ec1ee9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:46.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.546+0000 7f30f20f4640 1 --2- 172.21.15.161:0/155170806 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30ec072fe0 0x7f30ec1ee9e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f30dc00b3f0 tx=0x7f30dc00b8b0 comp rx=0 tx=0).stop 2024-03-28T23:29:46.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.549+0000 7f30f20f4640 1 -- 172.21.15.161:0/155170806 shutdown_connections 2024-03-28T23:29:46.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.549+0000 7f30f20f4640 1 --2- 172.21.15.161:0/155170806 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f30d806b500 0x7f30d806d9c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:46.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.549+0000 7f30f20f4640 1 --2- 172.21.15.161:0/155170806 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30ec158860 0x7f30ec0a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:46.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.549+0000 7f30f20f4640 1 --2- 172.21.15.161:0/155170806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30ec0c4e40 0x7f30ec06d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:46.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.549+0000 7f30f20f4640 1 --2- 172.21.15.161:0/155170806 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30ec072fe0 0x7f30ec1ee9e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:46.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.549+0000 7f30f20f4640 1 -- 172.21.15.161:0/155170806 >> 172.21.15.161:0/155170806 conn(0x7f30ec0c88b0 msgr2=0x7f30ec0c8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:46.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.550+0000 7f30f20f4640 1 -- 172.21.15.161:0/155170806 shutdown_connections 2024-03-28T23:29:46.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.551+0000 7f30f20f4640 1 -- 172.21.15.161:0/155170806 wait complete. 2024-03-28T23:29:46.583 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-28T23:29:46.583 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-28T23:29:46.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.690+0000 7f21311a0640 1 Processor -- start 2024-03-28T23:29:46.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.690+0000 7f21311a0640 1 -- start start 2024-03-28T23:29:46.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.690+0000 7f21311a0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f212c1592c0 0x7f212c0c0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:46.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.690+0000 7f21311a0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f212c0a6230 0x7f212c0bfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:46.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.690+0000 7f21311a0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f212c0c03e0 0x7f212c0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:46.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.690+0000 7f21311a0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f212c0c6120 con 0x7f212c1592c0 2024-03-28T23:29:46.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.690+0000 7f21311a0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f212c0c6400 con 0x7f212c0c03e0 2024-03-28T23:29:46.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.690+0000 7f21311a0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f212c0c66e0 con 0x7f212c0a6230 2024-03-28T23:29:46.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.690+0000 7f212bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f212c1592c0 0x7f212c0c0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:46.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.691+0000 7f212bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f212c1592c0 0x7f212c0c0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51084/0 (socket says 172.21.15.161:51084) 2024-03-28T23:29:46.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.691+0000 7f212bfff640 1 -- 172.21.15.161:0/291565435 learned_addr learned my addr 172.21.15.161:0/291565435 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:46.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.691+0000 7f212b7fe640 1 --2- 172.21.15.161:0/291565435 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f212c0a6230 0x7f212c0bfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:46.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.691+0000 7f213099f640 1 --2- 172.21.15.161:0/291565435 >> v2:172.21.15.170:3300/0 conn(0x7f212c0c03e0 0x7f212c0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:46.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.691+0000 7f212bfff640 1 -- 172.21.15.161:0/291565435 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f212c0a6230 msgr2=0x7f212c0bfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:46.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.691+0000 7f212bfff640 1 --2- 172.21.15.161:0/291565435 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f212c0a6230 0x7f212c0bfea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:46.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.691+0000 7f212bfff640 1 -- 172.21.15.161:0/291565435 >> v2:172.21.15.170:3300/0 conn(0x7f212c0c03e0 msgr2=0x7f212c0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:46.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.691+0000 7f212bfff640 1 --2- 172.21.15.161:0/291565435 >> v2:172.21.15.170:3300/0 conn(0x7f212c0c03e0 0x7f212c0c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:46.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.691+0000 7f212bfff640 1 -- 172.21.15.161:0/291565435 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f212c0ab820 con 0x7f212c1592c0 2024-03-28T23:29:46.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.691+0000 7f212b7fe640 1 --2- 172.21.15.161:0/291565435 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f212c0a6230 0x7f212c0bfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:29:46.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.691+0000 7f213099f640 1 --2- 172.21.15.161:0/291565435 >> v2:172.21.15.170:3300/0 conn(0x7f212c0c03e0 0x7f212c0c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:29:46.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.692+0000 7f212bfff640 1 --2- 172.21.15.161:0/291565435 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f212c1592c0 0x7f212c0c0cb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f212c0c45b0 tx=0x7f211c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16ea23b0eff23d52 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:46.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.692+0000 7f212affd640 1 -- 172.21.15.161:0/291565435 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211c035020 con 0x7f212c1592c0 2024-03-28T23:29:46.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.692+0000 7f212affd640 1 -- 172.21.15.161:0/291565435 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f211c031870 con 0x7f212c1592c0 2024-03-28T23:29:46.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.692+0000 7f212affd640 1 -- 172.21.15.161:0/291565435 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211c031c40 con 0x7f212c1592c0 2024-03-28T23:29:46.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.692+0000 7f21311a0640 1 -- 172.21.15.161:0/291565435 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f212c1592c0 msgr2=0x7f212c0c0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:46.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.692+0000 7f21311a0640 1 --2- 172.21.15.161:0/291565435 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f212c1592c0 0x7f212c0c0cb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f212c0c45b0 tx=0x7f211c02b910 comp rx=0 tx=0).stop 2024-03-28T23:29:46.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.692+0000 7f21311a0640 1 -- 172.21.15.161:0/291565435 shutdown_connections 2024-03-28T23:29:46.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.692+0000 7f21311a0640 1 --2- 172.21.15.161:0/291565435 >> v2:172.21.15.170:3300/0 conn(0x7f212c0c03e0 0x7f212c0c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:46.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.692+0000 7f21311a0640 1 --2- 172.21.15.161:0/291565435 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f212c0a6230 0x7f212c0bfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:46.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.692+0000 7f21311a0640 1 --2- 172.21.15.161:0/291565435 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f212c1592c0 0x7f212c0c0cb0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:46.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.692+0000 7f21311a0640 1 -- 172.21.15.161:0/291565435 >> 172.21.15.161:0/291565435 conn(0x7f212c0a8820 msgr2=0x7f212c0a8c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:46.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.692+0000 7f21311a0640 1 -- 172.21.15.161:0/291565435 shutdown_connections 2024-03-28T23:29:46.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.692+0000 7f21311a0640 1 -- 172.21.15.161:0/291565435 wait complete. 2024-03-28T23:29:46.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.693+0000 7f21311a0640 1 Processor -- start 2024-03-28T23:29:46.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.694+0000 7f21311a0640 1 -- start start 2024-03-28T23:29:46.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.694+0000 7f21311a0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f212c0a6230 0x7f212c14c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:46.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.694+0000 7f21311a0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f212c0c03e0 0x7f212c157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:46.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.694+0000 7f21311a0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f212c1592c0 0x7f212c072960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:46.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.694+0000 7f21311a0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f212c0c1470 con 0x7f212c0c03e0 2024-03-28T23:29:46.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.694+0000 7f21311a0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f212c1596a0 con 0x7f212c1592c0 2024-03-28T23:29:46.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.694+0000 7f21311a0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f212c14bc90 con 0x7f212c0a6230 2024-03-28T23:29:46.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.694+0000 7f212b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f212c0c03e0 0x7f212c157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:46.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.694+0000 7f212b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f212c0c03e0 0x7f212c157f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51094/0 (socket says 172.21.15.161:51094) 2024-03-28T23:29:46.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.694+0000 7f212b7fe640 1 -- 172.21.15.161:0/1928562285 learned_addr learned my addr 172.21.15.161:0/1928562285 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:46.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.694+0000 7f212bfff640 1 --2- 172.21.15.161:0/1928562285 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f212c0a6230 0x7f212c14c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:46.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.695+0000 7f213099f640 1 --2- 172.21.15.161:0/1928562285 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f212c1592c0 0x7f212c072960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:46.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.695+0000 7f212b7fe640 1 -- 172.21.15.161:0/1928562285 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f212c0a6230 msgr2=0x7f212c14c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:46.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.695+0000 7f212b7fe640 1 --2- 172.21.15.161:0/1928562285 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f212c0a6230 0x7f212c14c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:46.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.695+0000 7f212b7fe640 1 -- 172.21.15.161:0/1928562285 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f212c1592c0 msgr2=0x7f212c072960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:46.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.695+0000 7f212b7fe640 1 --2- 172.21.15.161:0/1928562285 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f212c1592c0 0x7f212c072960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:46.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.695+0000 7f212b7fe640 1 -- 172.21.15.161:0/1928562285 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f212c1eefc0 con 0x7f212c0c03e0 2024-03-28T23:29:46.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.695+0000 7f213099f640 1 --2- 172.21.15.161:0/1928562285 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f212c1592c0 0x7f212c072960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:29:46.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.695+0000 7f212b7fe640 1 --2- 172.21.15.161:0/1928562285 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f212c0c03e0 0x7f212c157f60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f212000d7d0 tx=0x7f212000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:46.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.695+0000 7f2128ff9640 1 -- 172.21.15.161:0/1928562285 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2120004030 con 0x7f212c0c03e0 2024-03-28T23:29:46.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.695+0000 7f2128ff9640 1 -- 172.21.15.161:0/1928562285 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21200041d0 con 0x7f212c0c03e0 2024-03-28T23:29:46.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.695+0000 7f2128ff9640 1 -- 172.21.15.161:0/1928562285 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21200096a0 con 0x7f212c0c03e0 2024-03-28T23:29:46.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.695+0000 7f21311a0640 1 -- 172.21.15.161:0/1928562285 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f212c14cac0 con 0x7f212c0c03e0 2024-03-28T23:29:46.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.695+0000 7f21311a0640 1 -- 172.21.15.161:0/1928562285 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f212c072ea0 con 0x7f212c0c03e0 2024-03-28T23:29:46.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.696+0000 7f21311a0640 1 -- 172.21.15.161:0/1928562285 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f212c0682b0 con 0x7f212c0c03e0 2024-03-28T23:29:46.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.696+0000 7f2128ff9640 1 -- 172.21.15.161:0/1928562285 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2120009840 con 0x7f212c0c03e0 2024-03-28T23:29:46.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.697+0000 7f2128ff9640 1 --2- 172.21.15.161:0/1928562285 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f211806b4d0 0x7f211806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:46.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.697+0000 7f2128ff9640 1 -- 172.21.15.161:0/1928562285 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7f212008a990 con 0x7f212c0c03e0 2024-03-28T23:29:46.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.697+0000 7f212bfff640 1 --2- 172.21.15.161:0/1928562285 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f211806b4d0 0x7f211806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:46.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.697+0000 7f212bfff640 1 --2- 172.21.15.161:0/1928562285 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f211806b4d0 0x7f211806d990 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f211c036000 tx=0x7f211c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:46.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.701+0000 7f2128ff9640 1 -- 172.21.15.161:0/1928562285 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f212c0682b0 con 0x7f212c0c03e0 2024-03-28T23:29:46.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:46.881+0000 7f21311a0640 1 -- 172.21.15.161:0/1928562285 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f212c073130 con 0x7f212c0c03e0 2024-03-28T23:29:47.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.544+0000 7f2128ff9640 1 -- 172.21.15.161:0/1928562285 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) ==== 157+0+0 (secure 0 0 0) 0x7f2120011340 con 0x7f212c0c03e0 2024-03-28T23:29:47.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.546+0000 7f21311a0640 1 -- 172.21.15.161:0/1928562285 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f211806b4d0 msgr2=0x7f211806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:47.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.546+0000 7f21311a0640 1 --2- 172.21.15.161:0/1928562285 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f211806b4d0 0x7f211806d990 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f211c036000 tx=0x7f211c03c040 comp rx=0 tx=0).stop 2024-03-28T23:29:47.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.546+0000 7f21311a0640 1 -- 172.21.15.161:0/1928562285 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f212c0c03e0 msgr2=0x7f212c157f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:47.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.546+0000 7f21311a0640 1 --2- 172.21.15.161:0/1928562285 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f212c0c03e0 0x7f212c157f60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f212000d7d0 tx=0x7f212000dc90 comp rx=0 tx=0).stop 2024-03-28T23:29:47.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.546+0000 7f21311a0640 1 -- 172.21.15.161:0/1928562285 shutdown_connections 2024-03-28T23:29:47.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.546+0000 7f21311a0640 1 --2- 172.21.15.161:0/1928562285 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f211806b4d0 0x7f211806d990 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:47.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.546+0000 7f21311a0640 1 --2- 172.21.15.161:0/1928562285 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f212c0c03e0 0x7f212c157f60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:47.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.546+0000 7f21311a0640 1 --2- 172.21.15.161:0/1928562285 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f212c1592c0 0x7f212c072960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:47.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.546+0000 7f21311a0640 1 --2- 172.21.15.161:0/1928562285 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f212c0a6230 0x7f212c14c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:47.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.546+0000 7f21311a0640 1 -- 172.21.15.161:0/1928562285 >> 172.21.15.161:0/1928562285 conn(0x7f212c0a8820 msgr2=0x7f212c0a7990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:47.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.546+0000 7f21311a0640 1 -- 172.21.15.161:0/1928562285 shutdown_connections 2024-03-28T23:29:47.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.546+0000 7f21311a0640 1 -- 172.21.15.161:0/1928562285 wait complete. 2024-03-28T23:29:47.560 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T23:29:47.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.670+0000 7f26762b1640 1 Processor -- start 2024-03-28T23:29:47.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.670+0000 7f26762b1640 1 -- start start 2024-03-28T23:29:47.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.670+0000 7f26762b1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f26700ca580 0x7f26700c9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:47.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.670+0000 7f26762b1640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26700c6df0 con 0x7f2670152940 2024-03-28T23:29:47.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.670+0000 7f26762b1640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26700c70d0 con 0x7f2670155220 2024-03-28T23:29:47.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.670+0000 7f26762b1640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f26700c5830 con 0x7f26700ca580 2024-03-28T23:29:47.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.670+0000 7f266f7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f26700ca580 0x7f26700c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:47.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.670+0000 7f266ffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f2670152940 0x7f26700c8a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48264/0 (socket says 172.21.15.161:48264) 2024-03-28T23:29:47.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.670+0000 7f266f7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f26700ca580 0x7f26700c9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51096/0 (socket says 172.21.15.161:51096) 2024-03-28T23:29:47.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.670+0000 7f266ffff640 1 -- 172.21.15.161:0/3993637185 learned_addr learned my addr 172.21.15.161:0/3993637185 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:47.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.671+0000 7f266f7fe640 1 -- 172.21.15.161:0/3993637185 >> v1:172.21.15.170:6789/0 conn(0x7f2670155220 legacy=0x7f2670155640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:47.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.671+0000 7f266f7fe640 1 -- 172.21.15.161:0/3993637185 >> v1:172.21.15.161:6789/0 conn(0x7f2670152940 legacy=0x7f26700c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:47.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.671+0000 7f266f7fe640 1 -- 172.21.15.161:0/3993637185 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26700c5b10 con 0x7f26700ca580 2024-03-28T23:29:47.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.671+0000 7f266effd640 1 -- 172.21.15.161:0/3993637185 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 575987203 0 0) 0x7f26700c6df0 con 0x7f2670152940 2024-03-28T23:29:47.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.671+0000 7f266f7fe640 1 --2- 172.21.15.161:0/3993637185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26700ca580 0x7f26700c9110 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f26600097a0 tx=0x7f266002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11a5b619661603a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:47.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.672+0000 7f266effd640 1 -- 172.21.15.161:0/3993637185 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2660038070 con 0x7f26700ca580 2024-03-28T23:29:47.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.672+0000 7f266effd640 1 -- 172.21.15.161:0/3993637185 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2660031700 con 0x7f26700ca580 2024-03-28T23:29:47.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.672+0000 7f266effd640 1 -- 172.21.15.161:0/3993637185 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2660031ad0 con 0x7f26700ca580 2024-03-28T23:29:47.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.672+0000 7f26762b1640 1 -- 172.21.15.161:0/3993637185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26700ca580 msgr2=0x7f26700c9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:47.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.672+0000 7f26762b1640 1 --2- 172.21.15.161:0/3993637185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26700ca580 0x7f26700c9110 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f26600097a0 tx=0x7f266002b7a0 comp rx=0 tx=0).stop 2024-03-28T23:29:47.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.672+0000 7f26762b1640 1 -- 172.21.15.161:0/3993637185 shutdown_connections 2024-03-28T23:29:47.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.672+0000 7f26762b1640 1 --2- 172.21.15.161:0/3993637185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26700ca580 0x7f26700c9110 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:47.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.672+0000 7f26762b1640 1 -- 172.21.15.161:0/3993637185 >> 172.21.15.161:0/3993637185 conn(0x7f26700caa00 msgr2=0x7f26700cae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:47.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.672+0000 7f26762b1640 1 -- 172.21.15.161:0/3993637185 shutdown_connections 2024-03-28T23:29:47.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.672+0000 7f26762b1640 1 -- 172.21.15.161:0/3993637185 wait complete. 2024-03-28T23:29:47.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.673+0000 7f26762b1640 1 Processor -- start 2024-03-28T23:29:47.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.674+0000 7f26762b1640 1 -- start start 2024-03-28T23:29:47.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.674+0000 7f26762b1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26700ca580 0x7f26700aacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:47.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.674+0000 7f26762b1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2670152940 0x7f26701516f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:47.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.674+0000 7f26762b1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2670155220 0x7f26701ee800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:47.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.674+0000 7f26762b1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2670152ea0 con 0x7f26700ca580 2024-03-28T23:29:47.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.674+0000 7f26762b1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2670152d20 con 0x7f2670155220 2024-03-28T23:29:47.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.674+0000 7f26762b1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f26700c6040 con 0x7f2670152940 2024-03-28T23:29:47.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.674+0000 7f266ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26700ca580 0x7f26700aacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:47.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.674+0000 7f266ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26700ca580 0x7f26700aacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51110/0 (socket says 172.21.15.161:51110) 2024-03-28T23:29:47.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.674+0000 7f266ffff640 1 -- 172.21.15.161:0/626783474 learned_addr learned my addr 172.21.15.161:0/626783474 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:47.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.675+0000 7f266f7fe640 1 --2- 172.21.15.161:0/626783474 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2670152940 0x7f26701516f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:47.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.675+0000 7f267484e640 1 --2- 172.21.15.161:0/626783474 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2670155220 0x7f26701ee800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:47.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.675+0000 7f266ffff640 1 -- 172.21.15.161:0/626783474 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2670152940 msgr2=0x7f26701516f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:47.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.675+0000 7f266ffff640 1 --2- 172.21.15.161:0/626783474 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2670152940 0x7f26701516f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:47.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.675+0000 7f266ffff640 1 -- 172.21.15.161:0/626783474 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2670155220 msgr2=0x7f26701ee800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:47.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.675+0000 7f266ffff640 1 --2- 172.21.15.161:0/626783474 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2670155220 0x7f26701ee800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:47.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.675+0000 7f266ffff640 1 -- 172.21.15.161:0/626783474 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f267006d3f0 con 0x7f26700ca580 2024-03-28T23:29:47.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.675+0000 7f266ffff640 1 --2- 172.21.15.161:0/626783474 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26700ca580 0x7f26700aacd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f265800c500 tx=0x7f265800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:47.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.675+0000 7f266cff9640 1 -- 172.21.15.161:0/626783474 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2658018070 con 0x7f26700ca580 2024-03-28T23:29:47.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.675+0000 7f266cff9640 1 -- 172.21.15.161:0/626783474 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2658007580 con 0x7f26700ca580 2024-03-28T23:29:47.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.675+0000 7f266cff9640 1 -- 172.21.15.161:0/626783474 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2658004a20 con 0x7f26700ca580 2024-03-28T23:29:47.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.675+0000 7f26762b1640 1 -- 172.21.15.161:0/626783474 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f267006ce30 con 0x7f26700ca580 2024-03-28T23:29:47.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.675+0000 7f26762b1640 1 -- 172.21.15.161:0/626783474 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26701ee000 con 0x7f26700ca580 2024-03-28T23:29:47.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.676+0000 7f26762b1640 1 -- 172.21.15.161:0/626783474 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f267004f4b0 con 0x7f26700ca580 2024-03-28T23:29:47.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.677+0000 7f266cff9640 1 -- 172.21.15.161:0/626783474 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2658004bc0 con 0x7f26700ca580 2024-03-28T23:29:47.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.677+0000 7f266cff9640 1 --2- 172.21.15.161:0/626783474 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f265006b4d0 0x7f265006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:47.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.677+0000 7f266f7fe640 1 --2- 172.21.15.161:0/626783474 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f265006b4d0 0x7f265006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:47.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.677+0000 7f266cff9640 1 -- 172.21.15.161:0/626783474 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 6007+0+0 (secure 0 0 0) 0x7f26580895d0 con 0x7f26700ca580 2024-03-28T23:29:47.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.678+0000 7f266f7fe640 1 --2- 172.21.15.161:0/626783474 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f265006b4d0 0x7f265006d990 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2660035000 tx=0x7f2660008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:47.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.682+0000 7f266cff9640 1 -- 172.21.15.161:0/626783474 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2658058330 con 0x7f26700ca580 2024-03-28T23:29:47.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:47.861+0000 7f26762b1640 1 -- 172.21.15.161:0/626783474 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f26700c6aa0 con 0x7f26700ca580 2024-03-28T23:29:49.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.574+0000 7f266cff9640 1 -- 172.21.15.161:0/626783474 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) ==== 228+0+0 (secure 0 0 0) 0x7f2658057c30 con 0x7f26700ca580 2024-03-28T23:29:49.575 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data_ec' created 2024-03-28T23:29:49.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.579+0000 7f26762b1640 1 -- 172.21.15.161:0/626783474 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f265006b4d0 msgr2=0x7f265006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:49.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.579+0000 7f26762b1640 1 --2- 172.21.15.161:0/626783474 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f265006b4d0 0x7f265006d990 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2660035000 tx=0x7f2660008030 comp rx=0 tx=0).stop 2024-03-28T23:29:49.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.579+0000 7f26762b1640 1 -- 172.21.15.161:0/626783474 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26700ca580 msgr2=0x7f26700aacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:49.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.579+0000 7f26762b1640 1 --2- 172.21.15.161:0/626783474 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26700ca580 0x7f26700aacd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f265800c500 tx=0x7f265800c9c0 comp rx=0 tx=0).stop 2024-03-28T23:29:49.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.580+0000 7f26762b1640 1 -- 172.21.15.161:0/626783474 shutdown_connections 2024-03-28T23:29:49.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.580+0000 7f26762b1640 1 --2- 172.21.15.161:0/626783474 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f265006b4d0 0x7f265006d990 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:49.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.580+0000 7f26762b1640 1 --2- 172.21.15.161:0/626783474 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2670155220 0x7f26701ee800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:49.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.580+0000 7f26762b1640 1 --2- 172.21.15.161:0/626783474 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2670152940 0x7f26701516f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:49.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.580+0000 7f26762b1640 1 --2- 172.21.15.161:0/626783474 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26700ca580 0x7f26700aacd0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:49.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.580+0000 7f26762b1640 1 -- 172.21.15.161:0/626783474 >> 172.21.15.161:0/626783474 conn(0x7f26700caa00 msgr2=0x7f26700cade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:49.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.580+0000 7f26762b1640 1 -- 172.21.15.161:0/626783474 shutdown_connections 2024-03-28T23:29:49.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.580+0000 7f26762b1640 1 -- 172.21.15.161:0/626783474 wait complete. 2024-03-28T23:29:49.596 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-28T23:29:49.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.737+0000 7fc160be9640 1 Processor -- start 2024-03-28T23:29:49.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.737+0000 7fc160be9640 1 -- start start 2024-03-28T23:29:49.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.738+0000 7fc160be9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc15c0a81b0 0x7fc15c0c89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:49.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.738+0000 7fc160be9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc15c155fc0 0x7fc15c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:49.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.738+0000 7fc160be9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc15c152d10 con 0x7fc15c0c5020 2024-03-28T23:29:49.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.738+0000 7fc160be9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fc15c159380 con 0x7fc15c0a81b0 2024-03-28T23:29:49.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.738+0000 7fc160be9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc15c0c35d0 con 0x7fc15c155fc0 2024-03-28T23:29:49.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.738+0000 7fc15ad9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc15c155fc0 0x7fc15c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:49.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.738+0000 7fc15a59c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc15c0a81b0 0x7fc15c0c89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:49.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.738+0000 7fc15a59c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc15c0a81b0 0x7fc15c0c89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:43454/0 (socket says 172.21.15.161:43454) 2024-03-28T23:29:49.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.738+0000 7fc15ad9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc15c155fc0 0x7fc15c06d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:33304/0 (socket says 172.21.15.161:33304) 2024-03-28T23:29:49.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.738+0000 7fc15a59c640 1 -- 172.21.15.161:0/3424570880 learned_addr learned my addr 172.21.15.161:0/3424570880 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:49.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.738+0000 7fc153fff640 1 --1- 172.21.15.161:0/3424570880 >> v1:172.21.15.170:6789/0 conn(0x7fc15c0c5020 0x7fc15c1558b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:42788/0 (socket says 172.21.15.161:42788) 2024-03-28T23:29:49.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.739+0000 7fc15a59c640 1 -- 172.21.15.161:0/3424570880 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc15c155fc0 msgr2=0x7fc15c06d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:49.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.739+0000 7fc15a59c640 1 --2- 172.21.15.161:0/3424570880 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc15c155fc0 0x7fc15c06d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:49.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.739+0000 7fc15a59c640 1 -- 172.21.15.161:0/3424570880 >> v1:172.21.15.170:6789/0 conn(0x7fc15c0c5020 legacy=0x7fc15c1558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:49.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.739+0000 7fc15a59c640 1 -- 172.21.15.161:0/3424570880 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc15c155390 con 0x7fc15c0a81b0 2024-03-28T23:29:49.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.739+0000 7fc15ad9d640 1 --2- 172.21.15.161:0/3424570880 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc15c155fc0 0x7fc15c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:29:49.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.739+0000 7fc15a59c640 1 --2- 172.21.15.161:0/3424570880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc15c0a81b0 0x7fc15c0c89c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc14c009870 tx=0x7fc14c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5063b11b40739d82 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:49.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.740+0000 7fc159d9b640 1 -- 172.21.15.161:0/3424570880 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14c034070 con 0x7fc15c0a81b0 2024-03-28T23:29:49.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.740+0000 7fc159d9b640 1 -- 172.21.15.161:0/3424570880 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc14c030840 con 0x7fc15c0a81b0 2024-03-28T23:29:49.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.740+0000 7fc159d9b640 1 -- 172.21.15.161:0/3424570880 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14c030c10 con 0x7fc15c0a81b0 2024-03-28T23:29:49.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.740+0000 7fc160be9640 1 -- 172.21.15.161:0/3424570880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc15c0a81b0 msgr2=0x7fc15c0c89c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:49.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.740+0000 7fc160be9640 1 --2- 172.21.15.161:0/3424570880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc15c0a81b0 0x7fc15c0c89c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc14c009870 tx=0x7fc14c02b980 comp rx=0 tx=0).stop 2024-03-28T23:29:49.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.740+0000 7fc160be9640 1 -- 172.21.15.161:0/3424570880 shutdown_connections 2024-03-28T23:29:49.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.740+0000 7fc160be9640 1 --2- 172.21.15.161:0/3424570880 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc15c155fc0 0x7fc15c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:49.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.740+0000 7fc160be9640 1 --2- 172.21.15.161:0/3424570880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc15c0a81b0 0x7fc15c0c89c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:49.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.740+0000 7fc160be9640 1 -- 172.21.15.161:0/3424570880 >> 172.21.15.161:0/3424570880 conn(0x7fc15c072a60 msgr2=0x7fc15c072e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:49.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.740+0000 7fc160be9640 1 -- 172.21.15.161:0/3424570880 shutdown_connections 2024-03-28T23:29:49.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.740+0000 7fc160be9640 1 -- 172.21.15.161:0/3424570880 wait complete. 2024-03-28T23:29:49.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.741+0000 7fc160be9640 1 Processor -- start 2024-03-28T23:29:49.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.741+0000 7fc160be9640 1 -- start start 2024-03-28T23:29:49.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.742+0000 7fc160be9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc15c0a81b0 0x7fc15c06eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:49.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.742+0000 7fc160be9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc15c0c5020 0x7fc15c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:49.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.742+0000 7fc160be9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc15c155fc0 0x7fc15c1eec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:49.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.742+0000 7fc160be9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc15c071090 con 0x7fc15c0a81b0 2024-03-28T23:29:49.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.742+0000 7fc160be9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc15c159640 con 0x7fc15c155fc0 2024-03-28T23:29:49.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.742+0000 7fc160be9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc15c071210 con 0x7fc15c0c5020 2024-03-28T23:29:49.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.742+0000 7fc15a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc15c0a81b0 0x7fc15c06eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:49.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.742+0000 7fc15a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc15c0a81b0 0x7fc15c06eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51120/0 (socket says 172.21.15.161:51120) 2024-03-28T23:29:49.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.742+0000 7fc15a59c640 1 -- 172.21.15.161:0/300018893 learned_addr learned my addr 172.21.15.161:0/300018893 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:49.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.742+0000 7fc153fff640 1 --2- 172.21.15.161:0/300018893 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc15c0c5020 0x7fc15c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:49.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.742+0000 7fc15ad9d640 1 --2- 172.21.15.161:0/300018893 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc15c155fc0 0x7fc15c1eec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:49.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.742+0000 7fc15a59c640 1 -- 172.21.15.161:0/300018893 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc15c0c5020 msgr2=0x7fc15c06d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:49.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.742+0000 7fc15a59c640 1 --2- 172.21.15.161:0/300018893 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc15c0c5020 0x7fc15c06d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:49.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.742+0000 7fc15a59c640 1 -- 172.21.15.161:0/300018893 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc15c155fc0 msgr2=0x7fc15c1eec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:49.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.742+0000 7fc15a59c640 1 --2- 172.21.15.161:0/300018893 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc15c155fc0 0x7fc15c1eec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:49.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.742+0000 7fc15a59c640 1 -- 172.21.15.161:0/300018893 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc15c0706e0 con 0x7fc15c0a81b0 2024-03-28T23:29:49.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.743+0000 7fc15a59c640 1 --2- 172.21.15.161:0/300018893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc15c0a81b0 0x7fc15c06eb80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc14c000c00 tx=0x7fc14c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:49.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.743+0000 7fc152ffd640 1 -- 172.21.15.161:0/300018893 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14c048070 con 0x7fc15c0a81b0 2024-03-28T23:29:49.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.743+0000 7fc152ffd640 1 -- 172.21.15.161:0/300018893 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc14c030640 con 0x7fc15c0a81b0 2024-03-28T23:29:49.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.743+0000 7fc152ffd640 1 -- 172.21.15.161:0/300018893 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14c034040 con 0x7fc15c0a81b0 2024-03-28T23:29:49.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.743+0000 7fc160be9640 1 -- 172.21.15.161:0/300018893 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc15c1ee430 con 0x7fc15c0a81b0 2024-03-28T23:29:49.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.743+0000 7fc160be9640 1 -- 172.21.15.161:0/300018893 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc15c1ef170 con 0x7fc15c0a81b0 2024-03-28T23:29:49.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.744+0000 7fc160be9640 1 -- 172.21.15.161:0/300018893 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc15c068f90 con 0x7fc15c0a81b0 2024-03-28T23:29:49.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.744+0000 7fc152ffd640 1 -- 172.21.15.161:0/300018893 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc14c04b070 con 0x7fc15c0a81b0 2024-03-28T23:29:49.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.745+0000 7fc152ffd640 1 --2- 172.21.15.161:0/300018893 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc13c06b500 0x7fc13c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:49.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.745+0000 7fc152ffd640 1 -- 172.21.15.161:0/300018893 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 6500+0+0 (secure 0 0 0) 0x7fc14c0ac340 con 0x7fc15c0a81b0 2024-03-28T23:29:49.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.745+0000 7fc153fff640 1 --2- 172.21.15.161:0/300018893 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc13c06b500 0x7fc13c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:49.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.745+0000 7fc153fff640 1 --2- 172.21.15.161:0/300018893 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc13c06b500 0x7fc13c06d9c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0c9520 tx=0x7fc144009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:49.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.749+0000 7fc152ffd640 1 -- 172.21.15.161:0/300018893 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc14c07b050 con 0x7fc15c0a81b0 2024-03-28T23:29:49.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:49.928+0000 7fc160be9640 1 -- 172.21.15.161:0/300018893 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fc15c0c2b20 con 0x7fc15c0a81b0 2024-03-28T23:29:50.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.575+0000 7fc152ffd640 1 -- 172.21.15.161:0/300018893 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) ==== 169+0+0 (secure 0 0 0) 0x7fc14c030810 con 0x7fc15c0a81b0 2024-03-28T23:29:50.576 INFO:teuthology.orchestra.run.smithi161.stderr:set pool 4 allow_ec_overwrites to true 2024-03-28T23:29:50.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.578+0000 7fc160be9640 1 -- 172.21.15.161:0/300018893 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc13c06b500 msgr2=0x7fc13c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:50.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.578+0000 7fc160be9640 1 --2- 172.21.15.161:0/300018893 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc13c06b500 0x7fc13c06d9c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0c9520 tx=0x7fc144009040 comp rx=0 tx=0).stop 2024-03-28T23:29:50.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.578+0000 7fc160be9640 1 -- 172.21.15.161:0/300018893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc15c0a81b0 msgr2=0x7fc15c06eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:50.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.578+0000 7fc160be9640 1 --2- 172.21.15.161:0/300018893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc15c0a81b0 0x7fc15c06eb80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc14c000c00 tx=0x7fc14c004730 comp rx=0 tx=0).stop 2024-03-28T23:29:50.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.579+0000 7fc160be9640 1 -- 172.21.15.161:0/300018893 shutdown_connections 2024-03-28T23:29:50.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.579+0000 7fc160be9640 1 --2- 172.21.15.161:0/300018893 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc13c06b500 0x7fc13c06d9c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:50.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.579+0000 7fc160be9640 1 --2- 172.21.15.161:0/300018893 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc15c155fc0 0x7fc15c1eec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:50.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.579+0000 7fc160be9640 1 --2- 172.21.15.161:0/300018893 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc15c0c5020 0x7fc15c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:50.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.579+0000 7fc160be9640 1 --2- 172.21.15.161:0/300018893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc15c0a81b0 0x7fc15c06eb80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:50.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.579+0000 7fc160be9640 1 -- 172.21.15.161:0/300018893 >> 172.21.15.161:0/300018893 conn(0x7fc15c072a60 msgr2=0x7fc15c1530f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:50.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.579+0000 7fc160be9640 1 -- 172.21.15.161:0/300018893 shutdown_connections 2024-03-28T23:29:50.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.579+0000 7fc160be9640 1 -- 172.21.15.161:0/300018893 wait complete. 2024-03-28T23:29:50.608 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-28T23:29:50.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.715+0000 7f737897a640 1 Processor -- start 2024-03-28T23:29:50.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.715+0000 7f737897a640 1 -- start start 2024-03-28T23:29:50.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.716+0000 7f737897a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f73740c77e0 0x7f73740c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:50.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.716+0000 7f737897a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73740c2480 con 0x7f7374072a50 2024-03-28T23:29:50.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.716+0000 7f737897a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73740c2760 con 0x7f7374159120 2024-03-28T23:29:50.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.716+0000 7f737897a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f73740c2230 con 0x7f73740c77e0 2024-03-28T23:29:50.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.716+0000 7f7371d9b640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f7374072a50 0x7f7374158a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52192/0 (socket says 172.21.15.161:52192) 2024-03-28T23:29:50.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.716+0000 7f7371d9b640 1 -- 172.21.15.161:0/1293740960 learned_addr learned my addr 172.21.15.161:0/1293740960 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:50.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.716+0000 7f737259c640 1 --2- 172.21.15.161:0/1293740960 >> v2:172.21.15.170:3300/0 conn(0x7f73740c77e0 0x7f73740c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:50.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.716+0000 7f737159a640 1 -- 172.21.15.161:0/1293740960 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3793303889 0 0) 0x7f73740c2480 con 0x7f7374072a50 2024-03-28T23:29:50.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.716+0000 7f737159a640 1 -- 172.21.15.161:0/1293740960 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f735c003520 con 0x7f7374072a50 2024-03-28T23:29:50.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.717+0000 7f737259c640 1 -- 172.21.15.161:0/1293740960 >> v1:172.21.15.170:6789/0 conn(0x7f7374159120 legacy=0x7f737415fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:50.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.717+0000 7f737259c640 1 -- 172.21.15.161:0/1293740960 >> v1:172.21.15.161:6789/0 conn(0x7f7374072a50 legacy=0x7f7374158a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:50.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.717+0000 7f737259c640 1 -- 172.21.15.161:0/1293740960 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7374072250 con 0x7f73740c77e0 2024-03-28T23:29:50.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.717+0000 7f737259c640 1 --2- 172.21.15.161:0/1293740960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73740c77e0 0x7f73740c7bc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f73680097a0 tx=0x7f736802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60c6b0102588c023 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:50.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.718+0000 7f737159a640 1 -- 172.21.15.161:0/1293740960 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7368034070 con 0x7f73740c77e0 2024-03-28T23:29:50.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.718+0000 7f737159a640 1 -- 172.21.15.161:0/1293740960 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7368030610 con 0x7f73740c77e0 2024-03-28T23:29:50.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.718+0000 7f737159a640 1 -- 172.21.15.161:0/1293740960 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73680309e0 con 0x7f73740c77e0 2024-03-28T23:29:50.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.718+0000 7f737897a640 1 -- 172.21.15.161:0/1293740960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73740c77e0 msgr2=0x7f73740c7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:50.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.718+0000 7f737897a640 1 --2- 172.21.15.161:0/1293740960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73740c77e0 0x7f73740c7bc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f73680097a0 tx=0x7f736802b960 comp rx=0 tx=0).stop 2024-03-28T23:29:50.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.718+0000 7f737897a640 1 -- 172.21.15.161:0/1293740960 shutdown_connections 2024-03-28T23:29:50.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.718+0000 7f737897a640 1 --2- 172.21.15.161:0/1293740960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73740c77e0 0x7f73740c7bc0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:50.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.718+0000 7f737897a640 1 -- 172.21.15.161:0/1293740960 >> 172.21.15.161:0/1293740960 conn(0x7f73740a97c0 msgr2=0x7f73740a9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:50.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.718+0000 7f737897a640 1 -- 172.21.15.161:0/1293740960 shutdown_connections 2024-03-28T23:29:50.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.718+0000 7f737897a640 1 -- 172.21.15.161:0/1293740960 wait complete. 2024-03-28T23:29:50.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.720+0000 7f737897a640 1 Processor -- start 2024-03-28T23:29:50.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.720+0000 7f737897a640 1 -- start start 2024-03-28T23:29:50.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.720+0000 7f737897a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7374072a50 0x7f7374157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:50.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.720+0000 7f737897a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73740c77e0 0x7f737414cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:50.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.720+0000 7f737897a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7374157ff0 0x7f737414d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:50.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.720+0000 7f737897a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f73740c36e0 con 0x7f73740c77e0 2024-03-28T23:29:50.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.720+0000 7f737897a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f73740725c0 con 0x7f7374157ff0 2024-03-28T23:29:50.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.720+0000 7f737897a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7374156390 con 0x7f7374072a50 2024-03-28T23:29:50.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.721+0000 7f7371d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73740c77e0 0x7f737414cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:50.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.721+0000 7f7371d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73740c77e0 0x7f737414cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55022/0 (socket says 172.21.15.161:55022) 2024-03-28T23:29:50.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.721+0000 7f7371d9b640 1 -- 172.21.15.161:0/1253339671 learned_addr learned my addr 172.21.15.161:0/1253339671 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:50.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.721+0000 7f737259c640 1 --2- 172.21.15.161:0/1253339671 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7374072a50 0x7f7374157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:50.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.721+0000 7f7372d9d640 1 --2- 172.21.15.161:0/1253339671 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7374157ff0 0x7f737414d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:50.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.721+0000 7f7371d9b640 1 -- 172.21.15.161:0/1253339671 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7374072a50 msgr2=0x7f7374157ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:50.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.721+0000 7f7371d9b640 1 --2- 172.21.15.161:0/1253339671 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7374072a50 0x7f7374157ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:50.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.721+0000 7f7371d9b640 1 -- 172.21.15.161:0/1253339671 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7374157ff0 msgr2=0x7f737414d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:50.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.721+0000 7f7371d9b640 1 --2- 172.21.15.161:0/1253339671 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7374157ff0 0x7f737414d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:50.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.721+0000 7f7371d9b640 1 -- 172.21.15.161:0/1253339671 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73740aa740 con 0x7f73740c77e0 2024-03-28T23:29:50.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.721+0000 7f7371d9b640 1 --2- 172.21.15.161:0/1253339671 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73740c77e0 0x7f737414cce0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f736400c500 tx=0x7f736400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:50.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.721+0000 7f7362ffd640 1 -- 172.21.15.161:0/1253339671 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7364017070 con 0x7f73740c77e0 2024-03-28T23:29:50.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.721+0000 7f737897a640 1 -- 172.21.15.161:0/1253339671 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73740aaa20 con 0x7f73740c77e0 2024-03-28T23:29:50.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.721+0000 7f7362ffd640 1 -- 172.21.15.161:0/1253339671 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7364005ce0 con 0x7f73740c77e0 2024-03-28T23:29:50.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.721+0000 7f7362ffd640 1 -- 172.21.15.161:0/1253339671 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73640072f0 con 0x7f73740c77e0 2024-03-28T23:29:50.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.722+0000 7f737897a640 1 -- 172.21.15.161:0/1253339671 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73741e90c0 con 0x7f73740c77e0 2024-03-28T23:29:50.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.723+0000 7f737897a640 1 -- 172.21.15.161:0/1253339671 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73740aabf0 con 0x7f73740c77e0 2024-03-28T23:29:50.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.723+0000 7f7362ffd640 1 -- 172.21.15.161:0/1253339671 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f736401a070 con 0x7f73740c77e0 2024-03-28T23:29:50.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.724+0000 7f7362ffd640 1 --2- 172.21.15.161:0/1253339671 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f735406b500 0x7f735406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:50.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.724+0000 7f7362ffd640 1 -- 172.21.15.161:0/1253339671 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6500+0+0 (secure 0 0 0) 0x7f73640897d0 con 0x7f73740c77e0 2024-03-28T23:29:50.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.724+0000 7f737259c640 1 --2- 172.21.15.161:0/1253339671 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f735406b500 0x7f735406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:50.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.724+0000 7f737259c640 1 --2- 172.21.15.161:0/1253339671 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f735406b500 0x7f735406d9c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7368036000 tx=0x7f736803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:50.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.728+0000 7f7362ffd640 1 -- 172.21.15.161:0/1253339671 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f73740aabf0 con 0x7f73740c77e0 2024-03-28T23:29:50.878 INFO:tasks.ceph.mds.c.smithi161.stderr:2024-03-28T23:29:50.877+0000 7ff59132f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:29:50.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:50.945+0000 7f737897a640 1 -- 172.21.15.161:0/1253339671 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f73741e9380 con 0x7f73740c77e0 2024-03-28T23:29:51.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.592+0000 7f7362ffd640 1 -- 172.21.15.161:0/1253339671 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) ==== 137+0+0 (secure 0 0 0) 0x7f7364057cf0 con 0x7f73740c77e0 2024-03-28T23:29:51.593 INFO:teuthology.orchestra.run.smithi161.stderr:added data pool 4 to fsmap 2024-03-28T23:29:51.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.595+0000 7f737897a640 1 -- 172.21.15.161:0/1253339671 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f735406b500 msgr2=0x7f735406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:51.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.595+0000 7f737897a640 1 --2- 172.21.15.161:0/1253339671 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f735406b500 0x7f735406d9c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7368036000 tx=0x7f736803b040 comp rx=0 tx=0).stop 2024-03-28T23:29:51.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.595+0000 7f737897a640 1 -- 172.21.15.161:0/1253339671 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73740c77e0 msgr2=0x7f737414cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:51.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.595+0000 7f737897a640 1 --2- 172.21.15.161:0/1253339671 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73740c77e0 0x7f737414cce0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f736400c500 tx=0x7f736400c9c0 comp rx=0 tx=0).stop 2024-03-28T23:29:51.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.595+0000 7f737897a640 1 -- 172.21.15.161:0/1253339671 shutdown_connections 2024-03-28T23:29:51.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.595+0000 7f737897a640 1 --2- 172.21.15.161:0/1253339671 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f735406b500 0x7f735406d9c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:51.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.595+0000 7f737897a640 1 --2- 172.21.15.161:0/1253339671 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73740c77e0 0x7f737414cce0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:51.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.595+0000 7f737897a640 1 --2- 172.21.15.161:0/1253339671 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7374157ff0 0x7f737414d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:51.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.595+0000 7f737897a640 1 --2- 172.21.15.161:0/1253339671 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7374072a50 0x7f7374157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:51.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.595+0000 7f737897a640 1 -- 172.21.15.161:0/1253339671 >> 172.21.15.161:0/1253339671 conn(0x7f73740a97c0 msgr2=0x7f73740c6d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:51.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.595+0000 7f737897a640 1 -- 172.21.15.161:0/1253339671 shutdown_connections 2024-03-28T23:29:51.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.595+0000 7f737897a640 1 -- 172.21.15.161:0/1253339671 wait complete. 2024-03-28T23:29:51.606 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:29:51.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.708+0000 7ff6c5bbc640 1 Processor -- start 2024-03-28T23:29:51.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.708+0000 7ff6c5bbc640 1 -- start start 2024-03-28T23:29:51.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.709+0000 7ff6c5bbc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff6c00c5410 0x7ff6c00c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:51.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.709+0000 7ff6c5bbc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff6c01553a0 0x7ff6c0155850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:51.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.709+0000 7ff6c5bbc640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6c0158a30 con 0x7ff6c0152890 2024-03-28T23:29:51.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.709+0000 7ff6c5bbc640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff6c00c8480 con 0x7ff6c01553a0 2024-03-28T23:29:51.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.709+0000 7ff6c5bbc640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7ff6c00c5d90 con 0x7ff6c00c5410 2024-03-28T23:29:51.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.709+0000 7ff6bf7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ff6c0152890 0x7ff6c00c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52196/0 (socket says 172.21.15.161:52196) 2024-03-28T23:29:51.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.709+0000 7ff6bf7fe640 1 -- 172.21.15.161:0/2877032497 learned_addr learned my addr 172.21.15.161:0/2877032497 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:51.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.709+0000 7ff6bffff640 1 --2- 172.21.15.161:0/2877032497 >> v2:172.21.15.161:3300/0 conn(0x7ff6c01553a0 0x7ff6c0155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:51.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.709+0000 7ff6bffff640 1 -- 172.21.15.161:0/2877032497 >> v2:172.21.15.170:3300/0 conn(0x7ff6c00c5410 msgr2=0x7ff6c00c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:51.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.709+0000 7ff6b6ffd640 1 --2- 172.21.15.161:0/2877032497 >> v2:172.21.15.170:3300/0 conn(0x7ff6c00c5410 0x7ff6c00c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:51.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.709+0000 7ff6beffd640 1 -- 172.21.15.161:0/2877032497 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 152374763 0 0) 0x7ff6c0158a30 con 0x7ff6c0152890 2024-03-28T23:29:51.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.709+0000 7ff6bffff640 1 --2- 172.21.15.161:0/2877032497 >> v2:172.21.15.170:3300/0 conn(0x7ff6c00c5410 0x7ff6c00c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:51.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.709+0000 7ff6bffff640 1 -- 172.21.15.161:0/2877032497 >> v1:172.21.15.161:6789/0 conn(0x7ff6c0152890 legacy=0x7ff6c00c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:51.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.709+0000 7ff6bffff640 1 -- 172.21.15.161:0/2877032497 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6c0158d10 con 0x7ff6c01553a0 2024-03-28T23:29:51.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.709+0000 7ff6b6ffd640 1 --2- 172.21.15.161:0/2877032497 >> v2:172.21.15.170:3300/0 conn(0x7ff6c00c5410 0x7ff6c00c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:29:51.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.710+0000 7ff6bffff640 1 --2- 172.21.15.161:0/2877032497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6c01553a0 0x7ff6c0155850 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00ad6e0 tx=0x7ff6b802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21055a7048d4198b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:51.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.710+0000 7ff6beffd640 1 -- 172.21.15.161:0/2877032497 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8039070 con 0x7ff6c01553a0 2024-03-28T23:29:51.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.710+0000 7ff6beffd640 1 -- 172.21.15.161:0/2877032497 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6b802cbf0 con 0x7ff6c01553a0 2024-03-28T23:29:51.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.710+0000 7ff6beffd640 1 -- 172.21.15.161:0/2877032497 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8034440 con 0x7ff6c01553a0 2024-03-28T23:29:51.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.711+0000 7ff6c5bbc640 1 -- 172.21.15.161:0/2877032497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6c01553a0 msgr2=0x7ff6c0155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:51.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.711+0000 7ff6c5bbc640 1 --2- 172.21.15.161:0/2877032497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6c01553a0 0x7ff6c0155850 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00ad6e0 tx=0x7ff6b802be40 comp rx=0 tx=0).stop 2024-03-28T23:29:51.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.711+0000 7ff6c5bbc640 1 -- 172.21.15.161:0/2877032497 shutdown_connections 2024-03-28T23:29:51.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.711+0000 7ff6c5bbc640 1 --2- 172.21.15.161:0/2877032497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6c01553a0 0x7ff6c0155850 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:51.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.711+0000 7ff6c5bbc640 1 --2- 172.21.15.161:0/2877032497 >> v2:172.21.15.170:3300/0 conn(0x7ff6c00c5410 0x7ff6c00c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:51.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.711+0000 7ff6c5bbc640 1 -- 172.21.15.161:0/2877032497 >> 172.21.15.161:0/2877032497 conn(0x7ff6c00ac620 msgr2=0x7ff6c00aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:51.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.711+0000 7ff6c5bbc640 1 -- 172.21.15.161:0/2877032497 shutdown_connections 2024-03-28T23:29:51.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.711+0000 7ff6c5bbc640 1 -- 172.21.15.161:0/2877032497 wait complete. 2024-03-28T23:29:51.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.712+0000 7ff6c5bbc640 1 Processor -- start 2024-03-28T23:29:51.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.712+0000 7ff6c5bbc640 1 -- start start 2024-03-28T23:29:51.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.713+0000 7ff6c5bbc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6c00c5410 0x7ff6c014aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:51.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.713+0000 7ff6c5bbc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6c0152890 0x7ff6c014e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:51.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.713+0000 7ff6c5bbc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6c01553a0 0x7ff6c014e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:51.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.713+0000 7ff6c5bbc640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c01592d0 con 0x7ff6c0152890 2024-03-28T23:29:51.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.713+0000 7ff6c5bbc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c0159640 con 0x7ff6c01553a0 2024-03-28T23:29:51.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.713+0000 7ff6c5bbc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff6c00c90a0 con 0x7ff6c00c5410 2024-03-28T23:29:51.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.713+0000 7ff6b6ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6c0152890 0x7ff6c014e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:51.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.713+0000 7ff6b6ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6c0152890 0x7ff6c014e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55028/0 (socket says 172.21.15.161:55028) 2024-03-28T23:29:51.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.713+0000 7ff6b6ffd640 1 -- 172.21.15.161:0/1132123615 learned_addr learned my addr 172.21.15.161:0/1132123615 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:51.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.713+0000 7ff6bf7fe640 1 --2- 172.21.15.161:0/1132123615 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6c00c5410 0x7ff6c014aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:51.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.713+0000 7ff6bffff640 1 --2- 172.21.15.161:0/1132123615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6c01553a0 0x7ff6c014e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:51.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.713+0000 7ff6b6ffd640 1 -- 172.21.15.161:0/1132123615 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6c00c5410 msgr2=0x7ff6c014aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:51.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.713+0000 7ff6b6ffd640 1 --2- 172.21.15.161:0/1132123615 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6c00c5410 0x7ff6c014aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:51.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.713+0000 7ff6b6ffd640 1 -- 172.21.15.161:0/1132123615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6c01553a0 msgr2=0x7ff6c014e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:51.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.713+0000 7ff6b6ffd640 1 --2- 172.21.15.161:0/1132123615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6c01553a0 0x7ff6c014e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:51.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.713+0000 7ff6b6ffd640 1 -- 172.21.15.161:0/1132123615 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6c01ef1d0 con 0x7ff6c0152890 2024-03-28T23:29:51.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.714+0000 7ff6b6ffd640 1 --2- 172.21.15.161:0/1132123615 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6c0152890 0x7ff6c014e2a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff6a800c7f0 tx=0x7ff6a800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:51.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.714+0000 7ff6bcff9640 1 -- 172.21.15.161:0/1132123615 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a8005b90 con 0x7ff6c0152890 2024-03-28T23:29:51.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.714+0000 7ff6bcff9640 1 -- 172.21.15.161:0/1132123615 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6a8005d30 con 0x7ff6c0152890 2024-03-28T23:29:51.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.714+0000 7ff6c5bbc640 1 -- 172.21.15.161:0/1132123615 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6c0157be0 con 0x7ff6c0152890 2024-03-28T23:29:51.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.714+0000 7ff6bcff9640 1 -- 172.21.15.161:0/1132123615 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a8006270 con 0x7ff6c0152890 2024-03-28T23:29:51.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.714+0000 7ff6c5bbc640 1 -- 172.21.15.161:0/1132123615 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6c014ed20 con 0x7ff6c0152890 2024-03-28T23:29:51.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.715+0000 7ff6c5bbc640 1 -- 172.21.15.161:0/1132123615 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6c004f4b0 con 0x7ff6c0152890 2024-03-28T23:29:51.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.715+0000 7ff6bcff9640 1 -- 172.21.15.161:0/1132123615 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff6a8006620 con 0x7ff6c0152890 2024-03-28T23:29:51.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.716+0000 7ff6bcff9640 1 --2- 172.21.15.161:0/1132123615 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff6a406b500 0x7ff6a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:51.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.716+0000 7ff6bcff9640 1 -- 172.21.15.161:0/1132123615 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff6a808a660 con 0x7ff6c0152890 2024-03-28T23:29:51.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.716+0000 7ff6bf7fe640 1 --2- 172.21.15.161:0/1132123615 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff6a406b500 0x7ff6a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:51.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.716+0000 7ff6bf7fe640 1 --2- 172.21.15.161:0/1132123615 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff6a406b500 0x7ff6a406d9c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0000f60 tx=0x7ff6b0007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:51.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.720+0000 7ff6bcff9640 1 -- 172.21.15.161:0/1132123615 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff6a8015020 con 0x7ff6c0152890 2024-03-28T23:29:51.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.974+0000 7ff6c5bbc640 1 -- 172.21.15.161:0/1132123615 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6c00c7ea0 con 0x7ff6c0152890 2024-03-28T23:29:51.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.974+0000 7ff6bcff9640 1 -- 172.21.15.161:0/1132123615 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5189 (secure 0 0 0) 0x7ff6a8058be0 con 0x7ff6c0152890 2024-03-28T23:29:51.975 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:29:51.975 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/1577284938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":1577284938},{"type":"v1","addr":"172.21.15.170:6835","nonce":1577284938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4258,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/1107796931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1107796931},{"type":"v1","addr":"172.21.15.170:6834","nonce":1107796931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4107992135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4107992135},{"type":"v1","addr":"172.21.15.161:6836","nonce":4107992135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:29:45.903844+0000","modified":"2024-03-28T23:29:50.947049+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6837/3832540545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3832540545},{"type":"v1","addr":"172.21.15.161:6837","nonce":3832540545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4275,"qdb_cluster":[4275]},"id":1}]} 2024-03-28T23:29:51.975 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 7 2024-03-28T23:29:51.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.976+0000 7ff6c5bbc640 1 -- 172.21.15.161:0/1132123615 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff6a406b500 msgr2=0x7ff6a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:51.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.976+0000 7ff6c5bbc640 1 --2- 172.21.15.161:0/1132123615 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff6a406b500 0x7ff6a406d9c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0000f60 tx=0x7ff6b0007d50 comp rx=0 tx=0).stop 2024-03-28T23:29:51.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.976+0000 7ff6c5bbc640 1 -- 172.21.15.161:0/1132123615 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6c0152890 msgr2=0x7ff6c014e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:51.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.976+0000 7ff6c5bbc640 1 --2- 172.21.15.161:0/1132123615 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6c0152890 0x7ff6c014e2a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff6a800c7f0 tx=0x7ff6a800ccb0 comp rx=0 tx=0).stop 2024-03-28T23:29:51.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.976+0000 7ff6c5bbc640 1 -- 172.21.15.161:0/1132123615 shutdown_connections 2024-03-28T23:29:51.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.976+0000 7ff6c5bbc640 1 --2- 172.21.15.161:0/1132123615 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff6a406b500 0x7ff6a406d9c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:51.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.976+0000 7ff6c5bbc640 1 --2- 172.21.15.161:0/1132123615 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6c0152890 0x7ff6c014e2a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:51.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.976+0000 7ff6c5bbc640 1 --2- 172.21.15.161:0/1132123615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6c01553a0 0x7ff6c014e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:51.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.976+0000 7ff6c5bbc640 1 --2- 172.21.15.161:0/1132123615 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6c00c5410 0x7ff6c014aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:51.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.976+0000 7ff6c5bbc640 1 -- 172.21.15.161:0/1132123615 >> 172.21.15.161:0/1132123615 conn(0x7ff6c00ac620 msgr2=0x7ff6c00ca020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:51.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.976+0000 7ff6c5bbc640 1 -- 172.21.15.161:0/1132123615 shutdown_connections 2024-03-28T23:29:51.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:51.976+0000 7ff6c5bbc640 1 -- 172.21.15.161:0/1132123615 wait complete. 2024-03-28T23:29:51.988 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:29:52.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.089+0000 7f2c23fe9640 1 Processor -- start 2024-03-28T23:29:52.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.089+0000 7f2c23fe9640 1 -- start start 2024-03-28T23:29:52.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.090+0000 7f2c23fe9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2c1c06fc60 0x7f2c1c070110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:52.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.090+0000 7f2c23fe9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c1c153050 con 0x7f2c1c072810 2024-03-28T23:29:52.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.090+0000 7f2c23fe9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c1c153310 con 0x7f2c1c0c3390 2024-03-28T23:29:52.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.090+0000 7f2c23fe9640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2c1c0a8520 con 0x7f2c1c06fc60 2024-03-28T23:29:52.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.090+0000 7f2c21584640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2c1c06fc60 0x7f2c1c070110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:52.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.090+0000 7f2c21d85640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f2c1c072810 0x7f2c1c0c4840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52208/0 (socket says 172.21.15.161:52208) 2024-03-28T23:29:52.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.090+0000 7f2c21d85640 1 -- 172.21.15.161:0/1796439461 learned_addr learned my addr 172.21.15.161:0/1796439461 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:52.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.091+0000 7f2c21584640 1 -- 172.21.15.161:0/1796439461 >> v1:172.21.15.170:6789/0 conn(0x7f2c1c0c3390 legacy=0x7f2c1c0c37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:52.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.091+0000 7f2c21584640 1 -- 172.21.15.161:0/1796439461 >> v1:172.21.15.161:6789/0 conn(0x7f2c1c072810 legacy=0x7f2c1c0c4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:52.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.091+0000 7f2c21584640 1 -- 172.21.15.161:0/1796439461 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c1c1535f0 con 0x7f2c1c06fc60 2024-03-28T23:29:52.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.091+0000 7f2c20d83640 1 -- 172.21.15.161:0/1796439461 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 537027625 0 0) 0x7f2c1c153050 con 0x7f2c1c072810 2024-03-28T23:29:52.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.091+0000 7f2c21584640 1 --2- 172.21.15.161:0/1796439461 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c1c06fc60 0x7f2c1c070110 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0097a0 tx=0x7f2c0c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7676fc9721a80a68 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:52.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.092+0000 7f2c20d83640 1 -- 172.21.15.161:0/1796439461 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c0315a0 con 0x7f2c1c06fc60 2024-03-28T23:29:52.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.092+0000 7f2c20d83640 1 -- 172.21.15.161:0/1796439461 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c0c037070 con 0x7f2c1c06fc60 2024-03-28T23:29:52.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.092+0000 7f2c20d83640 1 -- 172.21.15.161:0/1796439461 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c031960 con 0x7f2c1c06fc60 2024-03-28T23:29:52.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.092+0000 7f2c23fe9640 1 -- 172.21.15.161:0/1796439461 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c1c06fc60 msgr2=0x7f2c1c070110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:52.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.092+0000 7f2c23fe9640 1 --2- 172.21.15.161:0/1796439461 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c1c06fc60 0x7f2c1c070110 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0097a0 tx=0x7f2c0c02b840 comp rx=0 tx=0).stop 2024-03-28T23:29:52.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.092+0000 7f2c23fe9640 1 -- 172.21.15.161:0/1796439461 shutdown_connections 2024-03-28T23:29:52.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.092+0000 7f2c23fe9640 1 --2- 172.21.15.161:0/1796439461 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c1c06fc60 0x7f2c1c070110 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.092+0000 7f2c23fe9640 1 -- 172.21.15.161:0/1796439461 >> 172.21.15.161:0/1796439461 conn(0x7f2c1c0aa740 msgr2=0x7f2c1c0aab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:52.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.092+0000 7f2c23fe9640 1 -- 172.21.15.161:0/1796439461 shutdown_connections 2024-03-28T23:29:52.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.092+0000 7f2c23fe9640 1 -- 172.21.15.161:0/1796439461 wait complete. 2024-03-28T23:29:52.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.093+0000 7f2c23fe9640 1 Processor -- start 2024-03-28T23:29:52.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.093+0000 7f2c23fe9640 1 -- start start 2024-03-28T23:29:52.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.094+0000 7f2c23fe9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2c1c06fc60 0x7f2c1c154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:52.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.094+0000 7f2c23fe9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c1c072810 0x7f2c1c156aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:52.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.094+0000 7f2c23fe9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c1c0c3390 0x7f2c1c0ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:52.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.094+0000 7f2c23fe9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2c1c0c3ed0 con 0x7f2c1c0c3390 2024-03-28T23:29:52.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.094+0000 7f2c23fe9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2c1c153bb0 con 0x7f2c1c06fc60 2024-03-28T23:29:52.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.094+0000 7f2c23fe9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2c1c0c4f50 con 0x7f2c1c072810 2024-03-28T23:29:52.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.094+0000 7f2c22586640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c1c0c3390 0x7f2c1c0ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:52.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.094+0000 7f2c22586640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c1c0c3390 0x7f2c1c0ad1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55034/0 (socket says 172.21.15.161:55034) 2024-03-28T23:29:52.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.094+0000 7f2c22586640 1 -- 172.21.15.161:0/952493984 learned_addr learned my addr 172.21.15.161:0/952493984 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:52.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.094+0000 7f2c21d85640 1 --2- 172.21.15.161:0/952493984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2c1c06fc60 0x7f2c1c154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:52.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.094+0000 7f2c21584640 1 --2- 172.21.15.161:0/952493984 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c1c072810 0x7f2c1c156aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:52.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.094+0000 7f2c22586640 1 -- 172.21.15.161:0/952493984 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c1c072810 msgr2=0x7f2c1c156aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:52.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.094+0000 7f2c22586640 1 --2- 172.21.15.161:0/952493984 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c1c072810 0x7f2c1c156aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.094+0000 7f2c22586640 1 -- 172.21.15.161:0/952493984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2c1c06fc60 msgr2=0x7f2c1c154440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:52.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.094+0000 7f2c22586640 1 --2- 172.21.15.161:0/952493984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2c1c06fc60 0x7f2c1c154440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.094+0000 7f2c22586640 1 -- 172.21.15.161:0/952493984 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c1c156fe0 con 0x7f2c1c0c3390 2024-03-28T23:29:52.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.095+0000 7f2c22586640 1 --2- 172.21.15.161:0/952493984 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c1c0c3390 0x7f2c1c0ad1c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2c1400c780 tx=0x7f2c1400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:52.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.095+0000 7f2c127fc640 1 -- 172.21.15.161:0/952493984 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c14004b60 con 0x7f2c1c0c3390 2024-03-28T23:29:52.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.095+0000 7f2c127fc640 1 -- 172.21.15.161:0/952493984 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c14004d00 con 0x7f2c1c0c3390 2024-03-28T23:29:52.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.095+0000 7f2c23fe9640 1 -- 172.21.15.161:0/952493984 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c1c1ecb90 con 0x7f2c1c0c3390 2024-03-28T23:29:52.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.095+0000 7f2c127fc640 1 -- 172.21.15.161:0/952493984 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1400aa30 con 0x7f2c1c0c3390 2024-03-28T23:29:52.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.095+0000 7f2c23fe9640 1 -- 172.21.15.161:0/952493984 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c1c0ac290 con 0x7f2c1c0c3390 2024-03-28T23:29:52.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.096+0000 7f2c23fe9640 1 -- 172.21.15.161:0/952493984 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c1c0abda0 con 0x7f2c1c0c3390 2024-03-28T23:29:52.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.096+0000 7f2c127fc640 1 -- 172.21.15.161:0/952493984 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2c1400abd0 con 0x7f2c1c0c3390 2024-03-28T23:29:52.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.096+0000 7f2c127fc640 1 --2- 172.21.15.161:0/952493984 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2c0006b4d0 0x7f2c0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:52.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.097+0000 7f2c127fc640 1 -- 172.21.15.161:0/952493984 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2c1408a570 con 0x7f2c1c0c3390 2024-03-28T23:29:52.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.097+0000 7f2c21d85640 1 --2- 172.21.15.161:0/952493984 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2c0006b4d0 0x7f2c0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:52.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.097+0000 7f2c21d85640 1 --2- 172.21.15.161:0/952493984 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2c0006b4d0 0x7f2c0006d990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2c18000f60 tx=0x7f2c18007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:52.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.101+0000 7f2c127fc640 1 -- 172.21.15.161:0/952493984 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2c14020070 con 0x7f2c1c0c3390 2024-03-28T23:29:52.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.274+0000 7f2c23fe9640 1 -- 172.21.15.161:0/952493984 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2c1c04f4b0 con 0x7f2c1c0c3390 2024-03-28T23:29:52.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.275+0000 7f2c127fc640 1 -- 172.21.15.161:0/952493984 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7f2c14058af0 con 0x7f2c1c0c3390 2024-03-28T23:29:52.276 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:29:52.276 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":24,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:29:50.946843+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:29:52.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.278+0000 7f2c23fe9640 1 -- 172.21.15.161:0/952493984 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2c0006b4d0 msgr2=0x7f2c0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:52.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.278+0000 7f2c23fe9640 1 --2- 172.21.15.161:0/952493984 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2c0006b4d0 0x7f2c0006d990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2c18000f60 tx=0x7f2c18007d50 comp rx=0 tx=0).stop 2024-03-28T23:29:52.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.278+0000 7f2c23fe9640 1 -- 172.21.15.161:0/952493984 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c1c0c3390 msgr2=0x7f2c1c0ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:52.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.278+0000 7f2c23fe9640 1 --2- 172.21.15.161:0/952493984 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c1c0c3390 0x7f2c1c0ad1c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2c1400c780 tx=0x7f2c1400cc40 comp rx=0 tx=0).stop 2024-03-28T23:29:52.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.278+0000 7f2c23fe9640 1 -- 172.21.15.161:0/952493984 shutdown_connections 2024-03-28T23:29:52.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.278+0000 7f2c23fe9640 1 --2- 172.21.15.161:0/952493984 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2c0006b4d0 0x7f2c0006d990 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.278+0000 7f2c23fe9640 1 --2- 172.21.15.161:0/952493984 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c1c0c3390 0x7f2c1c0ad1c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.278+0000 7f2c23fe9640 1 --2- 172.21.15.161:0/952493984 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c1c072810 0x7f2c1c156aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.278+0000 7f2c23fe9640 1 --2- 172.21.15.161:0/952493984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2c1c06fc60 0x7f2c1c154440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.278+0000 7f2c23fe9640 1 -- 172.21.15.161:0/952493984 >> 172.21.15.161:0/952493984 conn(0x7f2c1c0aa740 msgr2=0x7f2c1c1504a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:52.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.278+0000 7f2c23fe9640 1 -- 172.21.15.161:0/952493984 shutdown_connections 2024-03-28T23:29:52.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.278+0000 7f2c23fe9640 1 -- 172.21.15.161:0/952493984 wait complete. 2024-03-28T23:29:52.290 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:29:52.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.392+0000 7fa9a47b5640 1 Processor -- start 2024-03-28T23:29:52.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.392+0000 7fa9a47b5640 1 -- start start 2024-03-28T23:29:52.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.393+0000 7fa9a47b5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa99c0a6cf0 0x7fa99c155120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:52.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.393+0000 7fa9a47b5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa99c0c2260 con 0x7fa99c14f2a0 2024-03-28T23:29:52.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.393+0000 7fa9a47b5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa99c155770 con 0x7fa99c0c2710 2024-03-28T23:29:52.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.393+0000 7fa9a47b5640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa99c155a50 con 0x7fa99c0a6cf0 2024-03-28T23:29:52.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.393+0000 7fa9a1d50640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa99c0a6cf0 0x7fa99c155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:52.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.393+0000 7fa9a2551640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa99c14f2a0 0x7fa99c0c66d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52224/0 (socket says 172.21.15.161:52224) 2024-03-28T23:29:52.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.393+0000 7fa9a2551640 1 -- 172.21.15.161:0/3304914475 learned_addr learned my addr 172.21.15.161:0/3304914475 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:52.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.393+0000 7fa9a1d50640 1 -- 172.21.15.161:0/3304914475 >> v1:172.21.15.170:6789/0 conn(0x7fa99c0c2710 legacy=0x7fa99c0c2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:52.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.393+0000 7fa9a1d50640 1 -- 172.21.15.161:0/3304914475 >> v1:172.21.15.161:6789/0 conn(0x7fa99c14f2a0 legacy=0x7fa99c0c66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:52.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.393+0000 7fa9a1d50640 1 -- 172.21.15.161:0/3304914475 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa99c155d30 con 0x7fa99c0a6cf0 2024-03-28T23:29:52.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.393+0000 7fa9a154f640 1 -- 172.21.15.161:0/3304914475 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2664352866 0 0) 0x7fa99c0c2260 con 0x7fa99c14f2a0 2024-03-28T23:29:52.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.394+0000 7fa9a1d50640 1 --2- 172.21.15.161:0/3304914475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa99c0a6cf0 0x7fa99c155120 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa9980097a0 tx=0x7fa99802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af72bc03ed632650 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:52.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.394+0000 7fa9a154f640 1 -- 172.21.15.161:0/3304914475 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa998035020 con 0x7fa99c0a6cf0 2024-03-28T23:29:52.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.394+0000 7fa9a154f640 1 -- 172.21.15.161:0/3304914475 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa998031830 con 0x7fa99c0a6cf0 2024-03-28T23:29:52.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.394+0000 7fa9a154f640 1 -- 172.21.15.161:0/3304914475 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa998031c00 con 0x7fa99c0a6cf0 2024-03-28T23:29:52.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.395+0000 7fa9a47b5640 1 -- 172.21.15.161:0/3304914475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa99c0a6cf0 msgr2=0x7fa99c155120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:52.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.395+0000 7fa9a47b5640 1 --2- 172.21.15.161:0/3304914475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa99c0a6cf0 0x7fa99c155120 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa9980097a0 tx=0x7fa99802b8d0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.395+0000 7fa9a47b5640 1 -- 172.21.15.161:0/3304914475 shutdown_connections 2024-03-28T23:29:52.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.395+0000 7fa9a47b5640 1 --2- 172.21.15.161:0/3304914475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa99c0a6cf0 0x7fa99c155120 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.395+0000 7fa9a47b5640 1 -- 172.21.15.161:0/3304914475 >> 172.21.15.161:0/3304914475 conn(0x7fa99c0c42c0 msgr2=0x7fa99c0c46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:52.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.395+0000 7fa9a47b5640 1 -- 172.21.15.161:0/3304914475 shutdown_connections 2024-03-28T23:29:52.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.395+0000 7fa9a47b5640 1 -- 172.21.15.161:0/3304914475 wait complete. 2024-03-28T23:29:52.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.396+0000 7fa9a47b5640 1 Processor -- start 2024-03-28T23:29:52.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.396+0000 7fa9a47b5640 1 -- start start 2024-03-28T23:29:52.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.397+0000 7fa9a47b5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa99c0a6cf0 0x7fa99c146ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:52.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.397+0000 7fa9a47b5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa99c0c2710 0x7fa99c14a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:52.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.397+0000 7fa9a47b5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa99c14f2a0 0x7fa99c14cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:52.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.397+0000 7fa9a47b5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa99c0c6de0 con 0x7fa99c0a6cf0 2024-03-28T23:29:52.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.397+0000 7fa9a47b5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa99c156010 con 0x7fa99c14f2a0 2024-03-28T23:29:52.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.397+0000 7fa9a47b5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa99c0c6f60 con 0x7fa99c0c2710 2024-03-28T23:29:52.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.397+0000 7fa9a2551640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa99c0a6cf0 0x7fa99c146ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:52.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.397+0000 7fa9a2551640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa99c0a6cf0 0x7fa99c146ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55054/0 (socket says 172.21.15.161:55054) 2024-03-28T23:29:52.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.397+0000 7fa9a2551640 1 -- 172.21.15.161:0/902301404 learned_addr learned my addr 172.21.15.161:0/902301404 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:52.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.397+0000 7fa9a1d50640 1 --2- 172.21.15.161:0/902301404 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa99c0c2710 0x7fa99c14a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:52.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.397+0000 7fa9a2d52640 1 --2- 172.21.15.161:0/902301404 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa99c14f2a0 0x7fa99c14cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:52.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.397+0000 7fa9a2551640 1 -- 172.21.15.161:0/902301404 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa99c0c2710 msgr2=0x7fa99c14a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:52.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.397+0000 7fa9a2551640 1 --2- 172.21.15.161:0/902301404 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa99c0c2710 0x7fa99c14a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.397+0000 7fa9a2551640 1 -- 172.21.15.161:0/902301404 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa99c14f2a0 msgr2=0x7fa99c14cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:52.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.397+0000 7fa9a2551640 1 --2- 172.21.15.161:0/902301404 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa99c14f2a0 0x7fa99c14cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.397+0000 7fa9a2551640 1 -- 172.21.15.161:0/902301404 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa99c14a920 con 0x7fa99c0a6cf0 2024-03-28T23:29:52.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.397+0000 7fa9a2551640 1 --2- 172.21.15.161:0/902301404 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa99c0a6cf0 0x7fa99c146ab0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa99000c4d0 tx=0x7fa99000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:52.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.397+0000 7fa98effd640 1 -- 172.21.15.161:0/902301404 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa990018070 con 0x7fa99c0a6cf0 2024-03-28T23:29:52.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.397+0000 7fa98effd640 1 -- 172.21.15.161:0/902301404 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa990007500 con 0x7fa99c0a6cf0 2024-03-28T23:29:52.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.397+0000 7fa9a47b5640 1 -- 172.21.15.161:0/902301404 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa99c0c6b90 con 0x7fa99c0a6cf0 2024-03-28T23:29:52.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.397+0000 7fa98effd640 1 -- 172.21.15.161:0/902301404 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa990004a20 con 0x7fa99c0a6cf0 2024-03-28T23:29:52.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.398+0000 7fa9a47b5640 1 -- 172.21.15.161:0/902301404 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa99c14b0a0 con 0x7fa99c0a6cf0 2024-03-28T23:29:52.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.398+0000 7fa9a47b5640 1 -- 172.21.15.161:0/902301404 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa99c04f4b0 con 0x7fa99c0a6cf0 2024-03-28T23:29:52.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.399+0000 7fa98effd640 1 -- 172.21.15.161:0/902301404 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa99001b070 con 0x7fa99c0a6cf0 2024-03-28T23:29:52.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.399+0000 7fa98effd640 1 --2- 172.21.15.161:0/902301404 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa98006b4d0 0x7fa98006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:52.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.399+0000 7fa98effd640 1 -- 172.21.15.161:0/902301404 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa990089750 con 0x7fa99c0a6cf0 2024-03-28T23:29:52.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.399+0000 7fa9a1d50640 1 --2- 172.21.15.161:0/902301404 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa98006b4d0 0x7fa98006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:52.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.400+0000 7fa9a1d50640 1 --2- 172.21.15.161:0/902301404 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa98006b4d0 0x7fa98006d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa998036000 tx=0x7fa99803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:52.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.407+0000 7fa98effd640 1 -- 172.21.15.161:0/902301404 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa990057cd0 con 0x7fa99c0a6cf0 2024-03-28T23:29:52.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.586+0000 7fa9a47b5640 1 -- 172.21.15.161:0/902301404 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa99c14d000 con 0x7fa99c0a6cf0 2024-03-28T23:29:52.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.588+0000 7fa98effd640 1 -- 172.21.15.161:0/902301404 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7fa99005bd70 con 0x7fa99c0a6cf0 2024-03-28T23:29:52.589 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:29:52.589 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":24,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:29:50.946843+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:29:52.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.590+0000 7fa9a47b5640 1 -- 172.21.15.161:0/902301404 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa98006b4d0 msgr2=0x7fa98006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:52.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.590+0000 7fa9a47b5640 1 --2- 172.21.15.161:0/902301404 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa98006b4d0 0x7fa98006d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa998036000 tx=0x7fa99803d040 comp rx=0 tx=0).stop 2024-03-28T23:29:52.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.590+0000 7fa9a47b5640 1 -- 172.21.15.161:0/902301404 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa99c0a6cf0 msgr2=0x7fa99c146ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:52.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.590+0000 7fa9a47b5640 1 --2- 172.21.15.161:0/902301404 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa99c0a6cf0 0x7fa99c146ab0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa99000c4d0 tx=0x7fa99000c990 comp rx=0 tx=0).stop 2024-03-28T23:29:52.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.590+0000 7fa9a47b5640 1 -- 172.21.15.161:0/902301404 shutdown_connections 2024-03-28T23:29:52.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.590+0000 7fa9a47b5640 1 --2- 172.21.15.161:0/902301404 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa98006b4d0 0x7fa98006d990 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.590+0000 7fa9a47b5640 1 --2- 172.21.15.161:0/902301404 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa99c14f2a0 0x7fa99c14cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.590+0000 7fa9a47b5640 1 --2- 172.21.15.161:0/902301404 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa99c0c2710 0x7fa99c14a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.590+0000 7fa9a47b5640 1 --2- 172.21.15.161:0/902301404 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa99c0a6cf0 0x7fa99c146ab0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.590+0000 7fa9a47b5640 1 -- 172.21.15.161:0/902301404 >> 172.21.15.161:0/902301404 conn(0x7fa99c0c42c0 msgr2=0x7fa99c0ac860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:52.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.591+0000 7fa9a47b5640 1 -- 172.21.15.161:0/902301404 shutdown_connections 2024-03-28T23:29:52.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.591+0000 7fa9a47b5640 1 -- 172.21.15.161:0/902301404 wait complete. 2024-03-28T23:29:52.619 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:29:52.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.721+0000 7f3e2153e640 1 Processor -- start 2024-03-28T23:29:52.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.721+0000 7f3e2153e640 1 -- start start 2024-03-28T23:29:52.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.722+0000 7f3e2153e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3e1c06ca20 0x7f3e1c06ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:52.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.722+0000 7f3e2153e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3e1c0a98c0 0x7f3e1c0a9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:52.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.722+0000 7f3e2153e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e1c151850 con 0x7f3e1c14eca0 2024-03-28T23:29:52.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.722+0000 7f3e2153e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f3e1c152460 con 0x7f3e1c06ca20 2024-03-28T23:29:52.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.722+0000 7f3e2153e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f3e1c151b30 con 0x7f3e1c0a98c0 2024-03-28T23:29:52.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.722+0000 7f3e13fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3e1c06ca20 0x7f3e1c06ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.722+0000 7f3e1affd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f3e1c14eca0 0x7f3e1c0c1880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52228/0 (socket says 172.21.15.161:52228) 2024-03-28T23:29:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.722+0000 7f3e1affd640 1 -- 172.21.15.161:0/3200337507 learned_addr learned my addr 172.21.15.161:0/3200337507 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.722+0000 7f3e1b7fe640 1 --2- 172.21.15.161:0/3200337507 >> v2:172.21.15.170:3300/0 conn(0x7f3e1c0a98c0 0x7f3e1c0a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.722+0000 7f3e13fff640 1 -- 172.21.15.161:0/3200337507 >> v2:172.21.15.170:3300/0 conn(0x7f3e1c0a98c0 msgr2=0x7f3e1c0a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.722+0000 7f3e13fff640 1 --2- 172.21.15.161:0/3200337507 >> v2:172.21.15.170:3300/0 conn(0x7f3e1c0a98c0 0x7f3e1c0a9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.722+0000 7f3e13fff640 1 -- 172.21.15.161:0/3200337507 >> v1:172.21.15.161:6789/0 conn(0x7f3e1c14eca0 legacy=0x7f3e1c0c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.722+0000 7f3e1a7fc640 1 -- 172.21.15.161:0/3200337507 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2426348297 0 0) 0x7f3e1c151850 con 0x7f3e1c14eca0 2024-03-28T23:29:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.722+0000 7f3e13fff640 1 -- 172.21.15.161:0/3200337507 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1c151e10 con 0x7f3e1c06ca20 2024-03-28T23:29:52.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.723+0000 7f3e13fff640 1 --2- 172.21.15.161:0/3200337507 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e1c06ca20 0x7f3e1c06ce80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c0c2f90 tx=0x7f3e0402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f67e8abc81965ef7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:52.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.723+0000 7f3e1a7fc640 1 -- 172.21.15.161:0/3200337507 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e040315a0 con 0x7f3e1c06ca20 2024-03-28T23:29:52.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.723+0000 7f3e1a7fc640 1 -- 172.21.15.161:0/3200337507 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e04037070 con 0x7f3e1c06ca20 2024-03-28T23:29:52.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.723+0000 7f3e1a7fc640 1 -- 172.21.15.161:0/3200337507 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e04031960 con 0x7f3e1c06ca20 2024-03-28T23:29:52.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.724+0000 7f3e2153e640 1 -- 172.21.15.161:0/3200337507 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e1c06ca20 msgr2=0x7f3e1c06ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:52.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.724+0000 7f3e2153e640 1 --2- 172.21.15.161:0/3200337507 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e1c06ca20 0x7f3e1c06ce80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c0c2f90 tx=0x7f3e0402b840 comp rx=0 tx=0).stop 2024-03-28T23:29:52.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.724+0000 7f3e2153e640 1 -- 172.21.15.161:0/3200337507 shutdown_connections 2024-03-28T23:29:52.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.724+0000 7f3e2153e640 1 --2- 172.21.15.161:0/3200337507 >> v2:172.21.15.170:3300/0 conn(0x7f3e1c0a98c0 0x7f3e1c0a9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.724+0000 7f3e2153e640 1 --2- 172.21.15.161:0/3200337507 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e1c06ca20 0x7f3e1c06ce80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.724+0000 7f3e2153e640 1 -- 172.21.15.161:0/3200337507 >> 172.21.15.161:0/3200337507 conn(0x7f3e1c0c6670 msgr2=0x7f3e1c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:52.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.724+0000 7f3e2153e640 1 -- 172.21.15.161:0/3200337507 shutdown_connections 2024-03-28T23:29:52.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.724+0000 7f3e2153e640 1 -- 172.21.15.161:0/3200337507 wait complete. 2024-03-28T23:29:52.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.725+0000 7f3e2153e640 1 Processor -- start 2024-03-28T23:29:52.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.725+0000 7f3e2153e640 1 -- start start 2024-03-28T23:29:52.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.726+0000 7f3e2153e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e1c06ca20 0x7f3e1c0718a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:52.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.726+0000 7f3e2153e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e1c0a98c0 0x7f3e1c072330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:52.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.726+0000 7f3e2153e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e1c14eca0 0x7f3e1c14b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:52.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.726+0000 7f3e2153e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c1514b0 con 0x7f3e1c0a98c0 2024-03-28T23:29:52.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.726+0000 7f3e2153e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c14f080 con 0x7f3e1c14eca0 2024-03-28T23:29:52.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.726+0000 7f3e2153e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3e1c151630 con 0x7f3e1c06ca20 2024-03-28T23:29:52.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.726+0000 7f3e13fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e1c0a98c0 0x7f3e1c072330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:52.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.726+0000 7f3e13fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e1c0a98c0 0x7f3e1c072330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55064/0 (socket says 172.21.15.161:55064) 2024-03-28T23:29:52.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.726+0000 7f3e13fff640 1 -- 172.21.15.161:0/3384332174 learned_addr learned my addr 172.21.15.161:0/3384332174 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:52.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.726+0000 7f3e1affd640 1 --2- 172.21.15.161:0/3384332174 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e1c06ca20 0x7f3e1c0718a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:52.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.726+0000 7f3e1b7fe640 1 --2- 172.21.15.161:0/3384332174 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e1c14eca0 0x7f3e1c14b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:52.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.726+0000 7f3e13fff640 1 -- 172.21.15.161:0/3384332174 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e1c06ca20 msgr2=0x7f3e1c0718a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:52.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.726+0000 7f3e13fff640 1 --2- 172.21.15.161:0/3384332174 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e1c06ca20 0x7f3e1c0718a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.726+0000 7f3e13fff640 1 -- 172.21.15.161:0/3384332174 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e1c14eca0 msgr2=0x7f3e1c14b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:52.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.726+0000 7f3e13fff640 1 --2- 172.21.15.161:0/3384332174 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e1c14eca0 0x7f3e1c14b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.726+0000 7f3e13fff640 1 -- 172.21.15.161:0/3384332174 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1c1557b0 con 0x7f3e1c0a98c0 2024-03-28T23:29:52.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.726+0000 7f3e13fff640 1 --2- 172.21.15.161:0/3384332174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e1c0a98c0 0x7f3e1c072330 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3e04000c00 tx=0x7f3e040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:52.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.727+0000 7f3e137fe640 1 -- 172.21.15.161:0/3384332174 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e04047070 con 0x7f3e1c0a98c0 2024-03-28T23:29:52.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.727+0000 7f3e137fe640 1 -- 172.21.15.161:0/3384332174 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e04042890 con 0x7f3e1c0a98c0 2024-03-28T23:29:52.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.727+0000 7f3e2153e640 1 -- 172.21.15.161:0/3384332174 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e1c1531b0 con 0x7f3e1c0a98c0 2024-03-28T23:29:52.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.727+0000 7f3e137fe640 1 -- 172.21.15.161:0/3384332174 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e04042c80 con 0x7f3e1c0a98c0 2024-03-28T23:29:52.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.727+0000 7f3e2153e640 1 -- 172.21.15.161:0/3384332174 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e1c1e9ac0 con 0x7f3e1c0a98c0 2024-03-28T23:29:52.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.728+0000 7f3e2153e640 1 -- 172.21.15.161:0/3384332174 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e1c04f4b0 con 0x7f3e1c0a98c0 2024-03-28T23:29:52.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.728+0000 7f3e137fe640 1 -- 172.21.15.161:0/3384332174 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3e0404a070 con 0x7f3e1c0a98c0 2024-03-28T23:29:52.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.728+0000 7f3e137fe640 1 --2- 172.21.15.161:0/3384332174 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3dfc06b4d0 0x7f3dfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:52.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.729+0000 7f3e137fe640 1 -- 172.21.15.161:0/3384332174 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f3e0402f020 con 0x7f3e1c0a98c0 2024-03-28T23:29:52.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.729+0000 7f3e1affd640 1 --2- 172.21.15.161:0/3384332174 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3dfc06b4d0 0x7f3dfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:52.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.729+0000 7f3e1affd640 1 --2- 172.21.15.161:0/3384332174 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3dfc06b4d0 0x7f3dfc06d990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c007fd0 tx=0x7f3e0c007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:52.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.733+0000 7f3e137fe640 1 -- 172.21.15.161:0/3384332174 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3e04051030 con 0x7f3e1c0a98c0 2024-03-28T23:29:52.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.947+0000 7f3e2153e640 1 -- 172.21.15.161:0/3384332174 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e1c0c6e60 con 0x7f3e1c0a98c0 2024-03-28T23:29:52.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.948+0000 7f3e137fe640 1 -- 172.21.15.161:0/3384332174 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5189 (secure 0 0 0) 0x7f3e0407d850 con 0x7f3e1c0a98c0 2024-03-28T23:29:52.949 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:29:52.949 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/1577284938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":1577284938},{"type":"v1","addr":"172.21.15.170:6835","nonce":1577284938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4258,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/1107796931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1107796931},{"type":"v1","addr":"172.21.15.170:6834","nonce":1107796931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4107992135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4107992135},{"type":"v1","addr":"172.21.15.161:6836","nonce":4107992135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:29:45.903844+0000","modified":"2024-03-28T23:29:50.947049+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6837/3832540545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3832540545},{"type":"v1","addr":"172.21.15.161:6837","nonce":3832540545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4275,"qdb_cluster":[4275]},"id":1}]} 2024-03-28T23:29:52.949 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 7 2024-03-28T23:29:52.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.950+0000 7f3e2153e640 1 -- 172.21.15.161:0/3384332174 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3dfc06b4d0 msgr2=0x7f3dfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:52.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.950+0000 7f3e2153e640 1 --2- 172.21.15.161:0/3384332174 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3dfc06b4d0 0x7f3dfc06d990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c007fd0 tx=0x7f3e0c007de0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.950+0000 7f3e2153e640 1 -- 172.21.15.161:0/3384332174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e1c0a98c0 msgr2=0x7f3e1c072330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:52.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.950+0000 7f3e2153e640 1 --2- 172.21.15.161:0/3384332174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e1c0a98c0 0x7f3e1c072330 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3e04000c00 tx=0x7f3e040046c0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.950+0000 7f3e2153e640 1 -- 172.21.15.161:0/3384332174 shutdown_connections 2024-03-28T23:29:52.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.950+0000 7f3e2153e640 1 --2- 172.21.15.161:0/3384332174 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3dfc06b4d0 0x7f3dfc06d990 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.950+0000 7f3e2153e640 1 --2- 172.21.15.161:0/3384332174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e1c0a98c0 0x7f3e1c072330 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.950+0000 7f3e2153e640 1 --2- 172.21.15.161:0/3384332174 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e1c14eca0 0x7f3e1c14b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.950+0000 7f3e2153e640 1 --2- 172.21.15.161:0/3384332174 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e1c06ca20 0x7f3e1c0718a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:52.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.950+0000 7f3e2153e640 1 -- 172.21.15.161:0/3384332174 >> 172.21.15.161:0/3384332174 conn(0x7f3e1c0c6670 msgr2=0x7f3e1c0c5d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:52.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.950+0000 7f3e2153e640 1 -- 172.21.15.161:0/3384332174 shutdown_connections 2024-03-28T23:29:52.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:52.950+0000 7f3e2153e640 1 -- 172.21.15.161:0/3384332174 wait complete. 2024-03-28T23:29:52.962 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:29:53.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.062+0000 7f837451e640 1 Processor -- start 2024-03-28T23:29:53.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.062+0000 7f837451e640 1 -- start start 2024-03-28T23:29:53.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.063+0000 7f837451e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f836c152890 0x7f836c0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.063+0000 7f837451e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f836c0c5b20 0x7f836c1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.063+0000 7f837451e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f836c158a30 con 0x7f836c0c8ed0 2024-03-28T23:29:53.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.063+0000 7f837451e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f836c0c8480 con 0x7f836c152890 2024-03-28T23:29:53.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.063+0000 7f837451e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f836c159640 con 0x7f836c0c5b20 2024-03-28T23:29:53.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.063+0000 7f83722ba640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f836c152890 0x7f836c0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.063+0000 7f83722ba640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f836c152890 0x7f836c0c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55066/0 (socket says 172.21.15.161:55066) 2024-03-28T23:29:53.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.063+0000 7f83722ba640 1 -- 172.21.15.161:0/1669250419 learned_addr learned my addr 172.21.15.161:0/1669250419 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:53.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.063+0000 7f8372abb640 1 --2- 172.21.15.161:0/1669250419 >> v2:172.21.15.170:3300/0 conn(0x7f836c0c5b20 0x7f836c1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.064+0000 7f83722ba640 1 -- 172.21.15.161:0/1669250419 >> v2:172.21.15.170:3300/0 conn(0x7f836c0c5b20 msgr2=0x7f836c1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:53.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.064+0000 7f83722ba640 1 --2- 172.21.15.161:0/1669250419 >> v2:172.21.15.170:3300/0 conn(0x7f836c0c5b20 0x7f836c1553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.064+0000 7f83722ba640 1 -- 172.21.15.161:0/1669250419 >> v1:172.21.15.170:6789/0 conn(0x7f836c0c8ed0 legacy=0x7f836c0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:53.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.064+0000 7f83722ba640 1 -- 172.21.15.161:0/1669250419 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f836c158d10 con 0x7f836c152890 2024-03-28T23:29:53.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.065+0000 7f83722ba640 1 --2- 172.21.15.161:0/1669250419 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f836c152890 0x7f836c0c8990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f836c0ad480 tx=0x7f836802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=629661547347cba6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:53.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.065+0000 7f83712b8640 1 -- 172.21.15.161:0/1669250419 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836802e070 con 0x7f836c152890 2024-03-28T23:29:53.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.065+0000 7f83712b8640 1 -- 172.21.15.161:0/1669250419 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8368037070 con 0x7f836c152890 2024-03-28T23:29:53.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.065+0000 7f83712b8640 1 -- 172.21.15.161:0/1669250419 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8368031a00 con 0x7f836c152890 2024-03-28T23:29:53.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.065+0000 7f837451e640 1 -- 172.21.15.161:0/1669250419 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f836c152890 msgr2=0x7f836c0c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:53.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.065+0000 7f837451e640 1 --2- 172.21.15.161:0/1669250419 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f836c152890 0x7f836c0c8990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f836c0ad480 tx=0x7f836802b880 comp rx=0 tx=0).stop 2024-03-28T23:29:53.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.065+0000 7f837451e640 1 -- 172.21.15.161:0/1669250419 shutdown_connections 2024-03-28T23:29:53.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.065+0000 7f837451e640 1 --2- 172.21.15.161:0/1669250419 >> v2:172.21.15.170:3300/0 conn(0x7f836c0c5b20 0x7f836c1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.065+0000 7f837451e640 1 --2- 172.21.15.161:0/1669250419 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f836c152890 0x7f836c0c8990 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.065+0000 7f837451e640 1 -- 172.21.15.161:0/1669250419 >> 172.21.15.161:0/1669250419 conn(0x7f836c0ac620 msgr2=0x7f836c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:53.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.065+0000 7f837451e640 1 -- 172.21.15.161:0/1669250419 shutdown_connections 2024-03-28T23:29:53.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.065+0000 7f837451e640 1 -- 172.21.15.161:0/1669250419 wait complete. 2024-03-28T23:29:53.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.066+0000 7f837451e640 1 Processor -- start 2024-03-28T23:29:53.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.067+0000 7f837451e640 1 -- start start 2024-03-28T23:29:53.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.067+0000 7f837451e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f836c0c5b20 0x7f836c1ee090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.067+0000 7f837451e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f836c0c8ed0 0x7f836c1e7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.067+0000 7f837451e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f836c152890 0x7f836c1e7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.067+0000 7f837451e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f836c152c70 con 0x7f836c0c5b20 2024-03-28T23:29:53.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.067+0000 7f837451e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f836c1592d0 con 0x7f836c152890 2024-03-28T23:29:53.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.067+0000 7f837451e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f836c152df0 con 0x7f836c0c8ed0 2024-03-28T23:29:53.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.067+0000 7f83722ba640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f836c0c5b20 0x7f836c1ee090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.067+0000 7f83722ba640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f836c0c5b20 0x7f836c1ee090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55078/0 (socket says 172.21.15.161:55078) 2024-03-28T23:29:53.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.067+0000 7f83722ba640 1 -- 172.21.15.161:0/2349808192 learned_addr learned my addr 172.21.15.161:0/2349808192 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:53.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.067+0000 7f8372abb640 1 --2- 172.21.15.161:0/2349808192 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f836c152890 0x7f836c1e7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.067+0000 7f8371ab9640 1 --2- 172.21.15.161:0/2349808192 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f836c0c8ed0 0x7f836c1e7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.068+0000 7f83722ba640 1 -- 172.21.15.161:0/2349808192 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f836c0c8ed0 msgr2=0x7f836c1e7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:53.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.068+0000 7f83722ba640 1 --2- 172.21.15.161:0/2349808192 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f836c0c8ed0 0x7f836c1e7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.068+0000 7f83722ba640 1 -- 172.21.15.161:0/2349808192 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f836c152890 msgr2=0x7f836c1e7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:53.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.068+0000 7f83722ba640 1 --2- 172.21.15.161:0/2349808192 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f836c152890 0x7f836c1e7fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.068+0000 7f83722ba640 1 -- 172.21.15.161:0/2349808192 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f836c1e8510 con 0x7f836c0c5b20 2024-03-28T23:29:53.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.068+0000 7f8372abb640 1 --2- 172.21.15.161:0/2349808192 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f836c152890 0x7f836c1e7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:29:53.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.068+0000 7f83722ba640 1 --2- 172.21.15.161:0/2349808192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f836c0c5b20 0x7f836c1ee090 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8368000c00 tx=0x7f83680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:53.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.068+0000 7f8362ffd640 1 -- 172.21.15.161:0/2349808192 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8368032880 con 0x7f836c0c5b20 2024-03-28T23:29:53.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.068+0000 7f8362ffd640 1 -- 172.21.15.161:0/2349808192 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8368032a20 con 0x7f836c0c5b20 2024-03-28T23:29:53.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.068+0000 7f837451e640 1 -- 172.21.15.161:0/2349808192 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f836c156070 con 0x7f836c0c5b20 2024-03-28T23:29:53.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.068+0000 7f8362ffd640 1 -- 172.21.15.161:0/2349808192 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8368032e10 con 0x7f836c0c5b20 2024-03-28T23:29:53.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.068+0000 7f837451e640 1 -- 172.21.15.161:0/2349808192 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f836c157ae0 con 0x7f836c0c5b20 2024-03-28T23:29:53.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.069+0000 7f837451e640 1 -- 172.21.15.161:0/2349808192 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f836c157900 con 0x7f836c0c5b20 2024-03-28T23:29:53.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.069+0000 7f8362ffd640 1 -- 172.21.15.161:0/2349808192 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f836804c060 con 0x7f836c0c5b20 2024-03-28T23:29:53.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.070+0000 7f8362ffd640 1 --2- 172.21.15.161:0/2349808192 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f835006b500 0x7f835006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.070+0000 7f8362ffd640 1 -- 172.21.15.161:0/2349808192 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f83680ac0f0 con 0x7f836c0c5b20 2024-03-28T23:29:53.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.070+0000 7f8371ab9640 1 --2- 172.21.15.161:0/2349808192 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f835006b500 0x7f835006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.070+0000 7f8371ab9640 1 --2- 172.21.15.161:0/2349808192 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f835006b500 0x7f835006d9c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f836c14ab20 tx=0x7f835c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:53.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.074+0000 7f8362ffd640 1 -- 172.21.15.161:0/2349808192 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f836c157900 con 0x7f836c0c5b20 2024-03-28T23:29:53.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.253+0000 7f837451e640 1 -- 172.21.15.161:0/2349808192 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f836c04f4b0 con 0x7f836c0c5b20 2024-03-28T23:29:53.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.254+0000 7f8362ffd640 1 -- 172.21.15.161:0/2349808192 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7f8368040070 con 0x7f836c0c5b20 2024-03-28T23:29:53.255 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:29:53.255 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":24,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:29:50.946843+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:29:53.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.256+0000 7f837451e640 1 -- 172.21.15.161:0/2349808192 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f835006b500 msgr2=0x7f835006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:53.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.256+0000 7f837451e640 1 --2- 172.21.15.161:0/2349808192 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f835006b500 0x7f835006d9c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f836c14ab20 tx=0x7f835c009040 comp rx=0 tx=0).stop 2024-03-28T23:29:53.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.256+0000 7f837451e640 1 -- 172.21.15.161:0/2349808192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f836c0c5b20 msgr2=0x7f836c1ee090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:53.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.256+0000 7f837451e640 1 --2- 172.21.15.161:0/2349808192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f836c0c5b20 0x7f836c1ee090 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8368000c00 tx=0x7f83680046c0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.256+0000 7f837451e640 1 -- 172.21.15.161:0/2349808192 shutdown_connections 2024-03-28T23:29:53.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.256+0000 7f837451e640 1 --2- 172.21.15.161:0/2349808192 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f835006b500 0x7f835006d9c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.256+0000 7f837451e640 1 --2- 172.21.15.161:0/2349808192 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f836c152890 0x7f836c1e7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.256+0000 7f837451e640 1 --2- 172.21.15.161:0/2349808192 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f836c0c8ed0 0x7f836c1e7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.256+0000 7f837451e640 1 --2- 172.21.15.161:0/2349808192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f836c0c5b20 0x7f836c1ee090 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.256+0000 7f837451e640 1 -- 172.21.15.161:0/2349808192 >> 172.21.15.161:0/2349808192 conn(0x7f836c0ac620 msgr2=0x7f836c0c5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:53.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.256+0000 7f837451e640 1 -- 172.21.15.161:0/2349808192 shutdown_connections 2024-03-28T23:29:53.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.256+0000 7f837451e640 1 -- 172.21.15.161:0/2349808192 wait complete. 2024-03-28T23:29:53.671 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:29:45.903844+0000', 'modified': '2024-03-28T23:29:50.947049+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4275}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4275': {'gid': 4275, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.161:6837/3832540545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 3832540545}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 3832540545}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4275, 'qdb_cluster': [4275]} 2024-03-28T23:29:53.671 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:29:53.671 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-28T23:29:53.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.772+0000 7fad4e53e640 1 Processor -- start 2024-03-28T23:29:53.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.772+0000 7fad4e53e640 1 -- start start 2024-03-28T23:29:53.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.773+0000 7fad4e53e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fad48152890 0x7fad480c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.773+0000 7fad4e53e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad48158a30 con 0x7fad480c5b20 2024-03-28T23:29:53.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.773+0000 7fad4e53e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad48158d10 con 0x7fad480c8ed0 2024-03-28T23:29:53.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.773+0000 7fad4e53e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fad480c8480 con 0x7fad48152890 2024-03-28T23:29:53.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.773+0000 7fad47fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fad48152890 0x7fad480c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.773+0000 7fad47fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fad48152890 0x7fad480c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55092/0 (socket says 172.21.15.161:55092) 2024-03-28T23:29:53.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.773+0000 7fad47fff640 1 -- 172.21.15.161:0/399704810 learned_addr learned my addr 172.21.15.161:0/399704810 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:53.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.773+0000 7fad4cadb640 1 --1- 172.21.15.161:0/399704810 >> v1:172.21.15.161:6789/0 conn(0x7fad480c5b20 0x7fad481553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52244/0 (socket says 172.21.15.161:52244) 2024-03-28T23:29:53.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.774+0000 7fad47fff640 1 -- 172.21.15.161:0/399704810 >> v1:172.21.15.170:6789/0 conn(0x7fad480c8ed0 legacy=0x7fad480c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:53.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.774+0000 7fad47fff640 1 -- 172.21.15.161:0/399704810 >> v1:172.21.15.161:6789/0 conn(0x7fad480c5b20 legacy=0x7fad481553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:53.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.774+0000 7fad47fff640 1 -- 172.21.15.161:0/399704810 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad48158ff0 con 0x7fad48152890 2024-03-28T23:29:53.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.774+0000 7fad46ffd640 1 -- 172.21.15.161:0/399704810 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 792612197 0 0) 0x7fad48158a30 con 0x7fad480c5b20 2024-03-28T23:29:53.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.775+0000 7fad47fff640 1 --2- 172.21.15.161:0/399704810 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad48152890 0x7fad480c8990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fad38009870 tx=0x7fad3802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=210c7d5371be5830 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:53.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.775+0000 7fad46ffd640 1 -- 172.21.15.161:0/399704810 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3802e070 con 0x7fad48152890 2024-03-28T23:29:53.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.775+0000 7fad46ffd640 1 -- 172.21.15.161:0/399704810 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad38037070 con 0x7fad48152890 2024-03-28T23:29:53.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.775+0000 7fad46ffd640 1 -- 172.21.15.161:0/399704810 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad38031a00 con 0x7fad48152890 2024-03-28T23:29:53.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.775+0000 7fad4e53e640 1 -- 172.21.15.161:0/399704810 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad48152890 msgr2=0x7fad480c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:53.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.775+0000 7fad4e53e640 1 --2- 172.21.15.161:0/399704810 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad48152890 0x7fad480c8990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fad38009870 tx=0x7fad3802b880 comp rx=0 tx=0).stop 2024-03-28T23:29:53.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.775+0000 7fad4e53e640 1 -- 172.21.15.161:0/399704810 shutdown_connections 2024-03-28T23:29:53.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.775+0000 7fad4e53e640 1 --2- 172.21.15.161:0/399704810 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad48152890 0x7fad480c8990 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.775+0000 7fad4e53e640 1 -- 172.21.15.161:0/399704810 >> 172.21.15.161:0/399704810 conn(0x7fad480ac620 msgr2=0x7fad480aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:53.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.776+0000 7fad4e53e640 1 -- 172.21.15.161:0/399704810 shutdown_connections 2024-03-28T23:29:53.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.776+0000 7fad4e53e640 1 -- 172.21.15.161:0/399704810 wait complete. 2024-03-28T23:29:53.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.777+0000 7fad4e53e640 1 Processor -- start 2024-03-28T23:29:53.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.777+0000 7fad4e53e640 1 -- start start 2024-03-28T23:29:53.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.778+0000 7fad4e53e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad480c5b20 0x7fad48153c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.778+0000 7fad4e53e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad480c8ed0 0x7fad48155c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.778+0000 7fad4e53e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad48152890 0x7fad480aa3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.778+0000 7fad4e53e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fad48152c70 con 0x7fad480c8ed0 2024-03-28T23:29:53.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.778+0000 7fad4e53e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fad48159640 con 0x7fad480c5b20 2024-03-28T23:29:53.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.778+0000 7fad4e53e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fad48152df0 con 0x7fad48152890 2024-03-28T23:29:53.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.779+0000 7fad477fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad480c8ed0 0x7fad48155c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.779+0000 7fad477fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad480c8ed0 0x7fad48155c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55098/0 (socket says 172.21.15.161:55098) 2024-03-28T23:29:53.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.779+0000 7fad477fe640 1 -- 172.21.15.161:0/4123351906 learned_addr learned my addr 172.21.15.161:0/4123351906 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:53.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.779+0000 7fad47fff640 1 --2- 172.21.15.161:0/4123351906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad480c5b20 0x7fad48153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.779+0000 7fad4cadb640 1 --2- 172.21.15.161:0/4123351906 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad48152890 0x7fad480aa3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.779+0000 7fad477fe640 1 -- 172.21.15.161:0/4123351906 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad48152890 msgr2=0x7fad480aa3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:53.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.779+0000 7fad477fe640 1 --2- 172.21.15.161:0/4123351906 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad48152890 0x7fad480aa3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.779+0000 7fad477fe640 1 -- 172.21.15.161:0/4123351906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad480c5b20 msgr2=0x7fad48153c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:53.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.779+0000 7fad477fe640 1 --2- 172.21.15.161:0/4123351906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad480c5b20 0x7fad48153c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.779+0000 7fad477fe640 1 -- 172.21.15.161:0/4123351906 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad48153390 con 0x7fad480c8ed0 2024-03-28T23:29:53.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.779+0000 7fad47fff640 1 --2- 172.21.15.161:0/4123351906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad480c5b20 0x7fad48153c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:29:53.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.779+0000 7fad477fe640 1 --2- 172.21.15.161:0/4123351906 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad480c8ed0 0x7fad48155c00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fad3c00c3f0 tx=0x7fad3c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:53.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.779+0000 7fad44ff9640 1 -- 172.21.15.161:0/4123351906 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3c011790 con 0x7fad480c8ed0 2024-03-28T23:29:53.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.779+0000 7fad44ff9640 1 -- 172.21.15.161:0/4123351906 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad3c017070 con 0x7fad480c8ed0 2024-03-28T23:29:53.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.779+0000 7fad4e53e640 1 -- 172.21.15.161:0/4123351906 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad48154450 con 0x7fad480c8ed0 2024-03-28T23:29:53.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.779+0000 7fad44ff9640 1 -- 172.21.15.161:0/4123351906 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3c011b90 con 0x7fad480c8ed0 2024-03-28T23:29:53.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.779+0000 7fad4e53e640 1 -- 172.21.15.161:0/4123351906 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad48154160 con 0x7fad480c8ed0 2024-03-28T23:29:53.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.781+0000 7fad44ff9640 1 -- 172.21.15.161:0/4123351906 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fad3c012420 con 0x7fad480c8ed0 2024-03-28T23:29:53.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.781+0000 7fad44ff9640 1 --2- 172.21.15.161:0/4123351906 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fad3406b4d0 0x7fad3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.782+0000 7fad44ff9640 1 -- 172.21.15.161:0/4123351906 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fad3c08a100 con 0x7fad480c8ed0 2024-03-28T23:29:53.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.782+0000 7fad47fff640 1 --2- 172.21.15.161:0/4123351906 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fad3406b4d0 0x7fad3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.782+0000 7fad47fff640 1 --2- 172.21.15.161:0/4123351906 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fad3406b4d0 0x7fad3406d990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fad38036000 tx=0x7fad38008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:53.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.782+0000 7fad511fe740 1 Processor -- start 2024-03-28T23:29:53.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.782+0000 7fad511fe740 1 -- start start 2024-03-28T23:29:53.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.783+0000 7fad511fe740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a149c15db0 0x55a149c13890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.783+0000 7fad511fe740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a149c16190 0x55a149c13dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.783+0000 7fad511fe740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c14310 0x55a149c104e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.783+0000 7fad511fe740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55a149b79af0 con 0x55a149c14310 2024-03-28T23:29:53.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.783+0000 7fad511fe740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55a149b229d0 con 0x55a149c16190 2024-03-28T23:29:53.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.783+0000 7fad511fe740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55a149b75b90 con 0x55a149c15db0 2024-03-28T23:29:53.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.783+0000 7fad4cadb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c14310 0x55a149c104e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.783+0000 7fad4cadb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c14310 0x55a149c104e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55110/0 (socket says 172.21.15.161:55110) 2024-03-28T23:29:53.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.783+0000 7fad4cadb640 1 -- 172.21.15.161:0/2878015436 learned_addr learned my addr 172.21.15.161:0/2878015436 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:53.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.783+0000 7fad477fe640 1 --2- 172.21.15.161:0/2878015436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a149c16190 0x55a149c13dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.783+0000 7fad47fff640 1 --2- 172.21.15.161:0/2878015436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a149c15db0 0x55a149c13890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.783+0000 7fad4cadb640 1 -- 172.21.15.161:0/2878015436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a149c15db0 msgr2=0x55a149c13890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:53.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.783+0000 7fad4cadb640 1 --2- 172.21.15.161:0/2878015436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a149c15db0 0x55a149c13890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.783+0000 7fad4cadb640 1 -- 172.21.15.161:0/2878015436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a149c16190 msgr2=0x55a149c13dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:53.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.783+0000 7fad4cadb640 1 --2- 172.21.15.161:0/2878015436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a149c16190 0x55a149c13dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.783+0000 7fad4cadb640 1 -- 172.21.15.161:0/2878015436 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a14988e2d0 con 0x55a149c14310 2024-03-28T23:29:53.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.784+0000 7fad4cadb640 1 --2- 172.21.15.161:0/2878015436 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c14310 0x55a149c104e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fad40001d00 tx=0x7fad400021c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90aa58877c8d4d9a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:53.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.784+0000 7fad2dffb640 1 -- 172.21.15.161:0/2878015436 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad4000d8e0 con 0x55a149c14310 2024-03-28T23:29:53.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.784+0000 7fad2dffb640 1 -- 172.21.15.161:0/2878015436 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad4000da80 con 0x55a149c14310 2024-03-28T23:29:53.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.784+0000 7fad2dffb640 1 -- 172.21.15.161:0/2878015436 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad4000de30 con 0x55a149c14310 2024-03-28T23:29:53.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.784+0000 7fad511fe740 1 -- 172.21.15.161:0/2878015436 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c14310 msgr2=0x55a149c104e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:53.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.784+0000 7fad511fe740 1 --2- 172.21.15.161:0/2878015436 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c14310 0x55a149c104e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fad40001d00 tx=0x7fad400021c0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.784+0000 7fad511fe740 1 -- 172.21.15.161:0/2878015436 shutdown_connections 2024-03-28T23:29:53.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.784+0000 7fad511fe740 1 --2- 172.21.15.161:0/2878015436 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c14310 0x55a149c104e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.784+0000 7fad511fe740 1 --2- 172.21.15.161:0/2878015436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a149c16190 0x55a149c13dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.784+0000 7fad511fe740 1 --2- 172.21.15.161:0/2878015436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a149c15db0 0x55a149c13890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.784+0000 7fad511fe740 1 -- 172.21.15.161:0/2878015436 >> 172.21.15.161:0/2878015436 conn(0x55a149c12630 msgr2=0x55a149c37220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:53.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.784+0000 7fad511fe740 1 -- 172.21.15.161:0/2878015436 shutdown_connections 2024-03-28T23:29:53.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.784+0000 7fad511fe740 1 -- 172.21.15.161:0/2878015436 wait complete. 2024-03-28T23:29:53.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.785+0000 7fad511fe740 1 -- start start 2024-03-28T23:29:53.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.785+0000 7fad511fe740 1 Processor -- start 2024-03-28T23:29:53.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.786+0000 7fad511fe740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a149c15db0 0x55a149c3afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.786+0000 7fad511fe740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a149c16190 0x55a149c36610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.786+0000 7fad511fe740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c62910 0x55a149c62d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.786+0000 7fad511fe740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55a149c38090 con 0x55a149c62910 2024-03-28T23:29:53.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.786+0000 7fad511fe740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55a149b35f70 con 0x55a149c15db0 2024-03-28T23:29:53.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.786+0000 7fad511fe740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55a149c37ad0 con 0x55a149c16190 2024-03-28T23:29:53.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.786+0000 7fad511fe740 10 client.0 _ll_register_callbacks cb 0x55a14988dac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:29:53.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.786+0000 7fad4cadb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c62910 0x55a149c62d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.786+0000 7fad4cadb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c62910 0x55a149c62d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55118/0 (socket says 172.21.15.161:55118) 2024-03-28T23:29:53.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.786+0000 7fad4cadb640 1 -- 172.21.15.161:0/2673629617 learned_addr learned my addr 172.21.15.161:0/2673629617 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:53.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.786+0000 7fad47fff640 1 --2- 172.21.15.161:0/2673629617 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a149c15db0 0x55a149c3afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.786+0000 7fad477fe640 1 --2- 172.21.15.161:0/2673629617 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a149c16190 0x55a149c36610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.786+0000 7fad4cadb640 1 -- 172.21.15.161:0/2673629617 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a149c16190 msgr2=0x55a149c36610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:53.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.786+0000 7fad4cadb640 1 --2- 172.21.15.161:0/2673629617 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a149c16190 0x55a149c36610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.786+0000 7fad4cadb640 1 -- 172.21.15.161:0/2673629617 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a149c15db0 msgr2=0x55a149c3afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:53.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.786+0000 7fad4cadb640 1 --2- 172.21.15.161:0/2673629617 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a149c15db0 0x55a149c3afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.786+0000 7fad4cadb640 1 -- 172.21.15.161:0/2673629617 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1499586b0 con 0x55a149c62910 2024-03-28T23:29:53.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.786+0000 7fad4cadb640 1 --2- 172.21.15.161:0/2673629617 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c62910 0x55a149c62d90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fad40000f80 tx=0x7fad40001480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb6a8d2f330ff242 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:53.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.786+0000 7fad26ffd640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:29:53.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.786+0000 7fad26ffd640 1 -- 172.21.15.161:0/2673629617 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad40023020 con 0x55a149c62910 2024-03-28T23:29:53.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.787+0000 7fad511fe740 1 -- 172.21.15.161:0/2673629617 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad4000da80 con 0x55a149c62910 2024-03-28T23:29:53.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.787+0000 7fad26ffd640 1 -- 172.21.15.161:0/2673629617 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fad4001f680 con 0x55a149c62910 2024-03-28T23:29:53.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.787+0000 7fad26ffd640 1 -- 172.21.15.161:0/2673629617 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7fad40001900 con 0x55a149c62910 2024-03-28T23:29:53.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.787+0000 7fad511fe740 10 client.4443 fetch_fsmap learned FSMap version 7 2024-03-28T23:29:53.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.787+0000 7fad511fe740 1 -- 172.21.15.161:0/2673629617 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55a149c623f0 con 0x55a149c62910 2024-03-28T23:29:53.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.787+0000 7fad26ffd640 1 -- 172.21.15.161:0/2673629617 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad4000da80 con 0x55a149c62910 2024-03-28T23:29:53.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.787+0000 7fad26ffd640 1 -- 172.21.15.161:0/2673629617 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 7) ==== 3502+0+0 (secure 0 0 0) 0x7fad4001f880 con 0x55a149c62910 2024-03-28T23:29:53.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.787+0000 7fad511fe740 10 client.4443 fetch_fsmap finished waiting for FSMap version 7 2024-03-28T23:29:53.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.787+0000 7fad511fe740 10 client.4443 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T23:29:53.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.787+0000 7fad511fe740 20 client.4443 populate_metadata read hostname 'smithi161' 2024-03-28T23:29:53.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.787+0000 7fad511fe740 1 --2- 172.21.15.161:0/2673629617 >> [v2:172.21.15.161:6835/3832540545,v1:172.21.15.161:6837/3832540545] conn(0x55a149c65e70 0x55a149c70d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.787+0000 7fad511fe740 4 client.4443 mds_command: new command op to 4275 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:29:53.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.787+0000 7fad511fe740 1 -- 172.21.15.161:0/2673629617 --> [v2:172.21.15.161:6835/3832540545,v1:172.21.15.161:6837/3832540545] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a149b87030 con 0x55a149c65e70 2024-03-28T23:29:53.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.787+0000 7fad47fff640 1 --2- 172.21.15.161:0/2673629617 >> [v2:172.21.15.161:6835/3832540545,v1:172.21.15.161:6837/3832540545] conn(0x55a149c65e70 0x55a149c70d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.787+0000 7fad26ffd640 10 client.4443 ms_handle_connect on v2:172.21.15.161:6835/3832540545 2024-03-28T23:29:53.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.787+0000 7fad47fff640 1 --2- 172.21.15.161:0/2673629617 >> [v2:172.21.15.161:6835/3832540545,v1:172.21.15.161:6837/3832540545] conn(0x55a149c65e70 0x55a149c70d90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ed5817dcafca25b server_cookie=d62fb09e512d9582 in_seq=0 out_seq=0 2024-03-28T23:29:53.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.788+0000 7fad26ffd640 1 -- 172.21.15.161:0/2673629617 <== mds.0 v2:172.21.15.161:6835/3832540545 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a149b87030 con 0x55a149c65e70 2024-03-28T23:29:53.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.788+0000 7fad26ffd640 10 client.4443 handle_command_reply: tid=0 2024-03-28T23:29:53.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.788+0000 7fad511fe740 1 client.4443 shutdown 2024-03-28T23:29:53.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.788+0000 7fad511fe740 1 -- 172.21.15.161:0/2673629617 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c62910 msgr2=0x55a149c62d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:53.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.788+0000 7fad511fe740 1 --2- 172.21.15.161:0/2673629617 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c62910 0x55a149c62d90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fad40000f80 tx=0x7fad40001480 comp rx=0 tx=0).stop 2024-03-28T23:29:53.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.788+0000 7fad511fe740 1 -- 172.21.15.161:0/2673629617 shutdown_connections 2024-03-28T23:29:53.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.788+0000 7fad511fe740 1 --2- 172.21.15.161:0/2673629617 >> [v2:172.21.15.161:6835/3832540545,v1:172.21.15.161:6837/3832540545] conn(0x55a149c65e70 0x55a149c70d90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.788+0000 7fad511fe740 1 --2- 172.21.15.161:0/2673629617 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c62910 0x55a149c62d90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.788+0000 7fad511fe740 1 --2- 172.21.15.161:0/2673629617 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a149c16190 0x55a149c36610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.788+0000 7fad511fe740 1 --2- 172.21.15.161:0/2673629617 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a149c15db0 0x55a149c3afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.788+0000 7fad511fe740 1 -- 172.21.15.161:0/2673629617 >> 172.21.15.161:0/2673629617 conn(0x55a149c14310 msgr2=0x55a149c0f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:53.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.788+0000 7fad26ffd640 0 client.4443 ms_handle_reset on v2:172.21.15.161:6835/3832540545 2024-03-28T23:29:53.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.788+0000 7fad511fe740 1 -- 172.21.15.161:0/2673629617 shutdown_connections 2024-03-28T23:29:53.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.788+0000 7fad511fe740 1 -- 172.21.15.161:0/2673629617 wait complete. 2024-03-28T23:29:53.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.789+0000 7fad511fe740 20 client.4443 trim_cache size 0 max 16384 2024-03-28T23:29:53.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.806+0000 7fad511fe740 1 Processor -- start 2024-03-28T23:29:53.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.806+0000 7fad511fe740 1 -- start start 2024-03-28T23:29:53.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.807+0000 7fad511fe740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a149c15db0 0x55a149c5fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.807+0000 7fad511fe740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a149c16190 0x55a149c16b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.807+0000 7fad511fe740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c62910 0x55a149c6bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.807+0000 7fad511fe740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fad481f7e50 con 0x55a149c62910 2024-03-28T23:29:53.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.807+0000 7fad511fe740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fad481eea30 con 0x55a149c16190 2024-03-28T23:29:53.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.807+0000 7fad511fe740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fad481f7800 con 0x55a149c15db0 2024-03-28T23:29:53.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.807+0000 7fad4cadb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c62910 0x55a149c6bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.807+0000 7fad4cadb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c62910 0x55a149c6bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55124/0 (socket says 172.21.15.161:55124) 2024-03-28T23:29:53.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.807+0000 7fad4cadb640 1 -- 172.21.15.161:0/4115979529 learned_addr learned my addr 172.21.15.161:0/4115979529 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:53.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.807+0000 7fad47fff640 1 --2- 172.21.15.161:0/4115979529 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a149c15db0 0x55a149c5fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.807+0000 7fad4cadb640 1 -- 172.21.15.161:0/4115979529 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a149c15db0 msgr2=0x55a149c5fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:53.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.807+0000 7fad4cadb640 1 --2- 172.21.15.161:0/4115979529 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a149c15db0 0x55a149c5fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.807+0000 7fad4cadb640 1 -- 172.21.15.161:0/4115979529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a149c16190 msgr2=0x55a149c16b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:53.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.807+0000 7fad477fe640 1 --2- 172.21.15.161:0/4115979529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a149c16190 0x55a149c16b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.807+0000 7fad4cadb640 1 --2- 172.21.15.161:0/4115979529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a149c16190 0x55a149c16b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.807+0000 7fad4cadb640 1 -- 172.21.15.161:0/4115979529 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a149c54540 con 0x55a149c62910 2024-03-28T23:29:53.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.807+0000 7fad4cadb640 1 --2- 172.21.15.161:0/4115979529 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c62910 0x55a149c6bca0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fad4000df60 tx=0x7fad4001e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bc56eb64eba24dd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:53.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.808+0000 7fad2d7fa640 1 -- 172.21.15.161:0/4115979529 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad40023070 con 0x55a149c62910 2024-03-28T23:29:53.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.808+0000 7fad2d7fa640 1 -- 172.21.15.161:0/4115979529 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad4001f680 con 0x55a149c62910 2024-03-28T23:29:53.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.808+0000 7fad2d7fa640 1 -- 172.21.15.161:0/4115979529 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad40028400 con 0x55a149c62910 2024-03-28T23:29:53.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.808+0000 7fad511fe740 1 -- 172.21.15.161:0/4115979529 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c62910 msgr2=0x55a149c6bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:53.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.808+0000 7fad511fe740 1 --2- 172.21.15.161:0/4115979529 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c62910 0x55a149c6bca0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fad4000df60 tx=0x7fad4001e000 comp rx=0 tx=0).stop 2024-03-28T23:29:53.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.808+0000 7fad511fe740 1 -- 172.21.15.161:0/4115979529 shutdown_connections 2024-03-28T23:29:53.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.808+0000 7fad511fe740 1 --2- 172.21.15.161:0/4115979529 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c62910 0x55a149c6bca0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.808+0000 7fad511fe740 1 --2- 172.21.15.161:0/4115979529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a149c16190 0x55a149c16b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.808+0000 7fad511fe740 1 --2- 172.21.15.161:0/4115979529 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a149c15db0 0x55a149c5fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.808+0000 7fad511fe740 1 -- 172.21.15.161:0/4115979529 >> 172.21.15.161:0/4115979529 conn(0x55a149c14310 msgr2=0x55a149c36f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:53.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.808+0000 7fad511fe740 1 -- 172.21.15.161:0/4115979529 shutdown_connections 2024-03-28T23:29:53.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.808+0000 7fad511fe740 1 -- 172.21.15.161:0/4115979529 wait complete. 2024-03-28T23:29:53.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.809+0000 7fad511fe740 1 -- start start 2024-03-28T23:29:53.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.809+0000 7fad511fe740 1 Processor -- start 2024-03-28T23:29:53.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad511fe740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a149c15db0 0x55a149c76480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad511fe740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c16190 0x55a149c5f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad511fe740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a149c62910 0x55a149cc5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad511fe740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55a149c3b890 con 0x55a149c16190 2024-03-28T23:29:53.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad511fe740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55a149c38090 con 0x55a149c62910 2024-03-28T23:29:53.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad511fe740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55a149c33400 con 0x55a149c15db0 2024-03-28T23:29:53.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad511fe740 10 client.0 _ll_register_callbacks cb 0x7fad3c058310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:29:53.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad477fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c16190 0x55a149c5f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad477fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c16190 0x55a149c5f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55130/0 (socket says 172.21.15.161:55130) 2024-03-28T23:29:53.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad477fe640 1 -- 172.21.15.161:0/3917281363 learned_addr learned my addr 172.21.15.161:0/3917281363 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:53.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad4cadb640 1 --2- 172.21.15.161:0/3917281363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a149c62910 0x55a149cc5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad47fff640 1 --2- 172.21.15.161:0/3917281363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a149c15db0 0x55a149c76480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad477fe640 1 -- 172.21.15.161:0/3917281363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a149c15db0 msgr2=0x55a149c76480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:53.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad477fe640 1 --2- 172.21.15.161:0/3917281363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a149c15db0 0x55a149c76480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad477fe640 1 -- 172.21.15.161:0/3917281363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a149c62910 msgr2=0x55a149cc5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:53.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad477fe640 1 --2- 172.21.15.161:0/3917281363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a149c62910 0x55a149cc5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad477fe640 1 -- 172.21.15.161:0/3917281363 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a149c168a0 con 0x55a149c16190 2024-03-28T23:29:53.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad47fff640 1 --2- 172.21.15.161:0/3917281363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a149c15db0 0x55a149c76480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:29:53.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad477fe640 1 --2- 172.21.15.161:0/3917281363 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c16190 0x55a149c5f330 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fad3c0882f0 tx=0x7fad3c08a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ed1e66d5199129b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:53.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad26ffd640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:29:53.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad26ffd640 1 -- 172.21.15.161:0/3917281363 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3c054bb0 con 0x55a149c16190 2024-03-28T23:29:53.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad26ffd640 1 -- 172.21.15.161:0/3917281363 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fad3c0607c0 con 0x55a149c16190 2024-03-28T23:29:53.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad511fe740 1 -- 172.21.15.161:0/3917281363 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad4001f680 con 0x55a149c16190 2024-03-28T23:29:53.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad26ffd640 1 -- 172.21.15.161:0/3917281363 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7fad3c054d50 con 0x55a149c16190 2024-03-28T23:29:53.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad511fe740 10 client.4449 fetch_fsmap learned FSMap version 7 2024-03-28T23:29:53.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.810+0000 7fad511fe740 1 -- 172.21.15.161:0/3917281363 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55a149c55ab0 con 0x55a149c16190 2024-03-28T23:29:53.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.811+0000 7fad26ffd640 1 -- 172.21.15.161:0/3917281363 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad4001f680 con 0x55a149c16190 2024-03-28T23:29:53.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.811+0000 7fad26ffd640 1 -- 172.21.15.161:0/3917281363 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 7) ==== 3502+0+0 (secure 0 0 0) 0x7fad3c057990 con 0x55a149c16190 2024-03-28T23:29:53.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.811+0000 7fad511fe740 10 client.4449 fetch_fsmap finished waiting for FSMap version 7 2024-03-28T23:29:53.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.811+0000 7fad511fe740 10 client.4449 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T23:29:53.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.811+0000 7fad511fe740 20 client.4449 populate_metadata read hostname 'smithi161' 2024-03-28T23:29:53.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.811+0000 7fad511fe740 1 --2- 172.21.15.161:0/3917281363 >> [v2:172.21.15.161:6835/3832540545,v1:172.21.15.161:6837/3832540545] conn(0x55a149c317f0 0x55a149c31bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.811+0000 7fad511fe740 4 client.4449 mds_command: new command op to 4275 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:29:53.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.811+0000 7fad511fe740 1 -- 172.21.15.161:0/3917281363 --> [v2:172.21.15.161:6835/3832540545,v1:172.21.15.161:6837/3832540545] -- command(tid 0: {"prefix": "status"}) -- 0x55a149b81380 con 0x55a149c317f0 2024-03-28T23:29:53.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.811+0000 7fad47fff640 1 --2- 172.21.15.161:0/3917281363 >> [v2:172.21.15.161:6835/3832540545,v1:172.21.15.161:6837/3832540545] conn(0x55a149c317f0 0x55a149c31bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.811+0000 7fad26ffd640 10 client.4449 ms_handle_connect on v2:172.21.15.161:6835/3832540545 2024-03-28T23:29:53.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.811+0000 7fad47fff640 1 --2- 172.21.15.161:0/3917281363 >> [v2:172.21.15.161:6835/3832540545,v1:172.21.15.161:6837/3832540545] conn(0x55a149c317f0 0x55a149c31bd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c97c7f3af337094 server_cookie=cfd23276a2989f6a in_seq=0 out_seq=0 2024-03-28T23:29:53.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.811+0000 7fad26ffd640 1 -- 172.21.15.161:0/3917281363 <== mds.0 v2:172.21.15.161:6835/3832540545 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x55a149b81380 con 0x55a149c317f0 2024-03-28T23:29:53.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.811+0000 7fad26ffd640 10 client.4449 handle_command_reply: tid=0 2024-03-28T23:29:53.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.811+0000 7fad511fe740 1 client.4449 shutdown 2024-03-28T23:29:53.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.811+0000 7fad511fe740 1 -- 172.21.15.161:0/3917281363 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c16190 msgr2=0x55a149c5f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:53.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.811+0000 7fad511fe740 1 --2- 172.21.15.161:0/3917281363 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c16190 0x55a149c5f330 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fad3c0882f0 tx=0x7fad3c08a050 comp rx=0 tx=0).stop 2024-03-28T23:29:53.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.811+0000 7fad511fe740 1 -- 172.21.15.161:0/3917281363 shutdown_connections 2024-03-28T23:29:53.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.811+0000 7fad511fe740 1 --2- 172.21.15.161:0/3917281363 >> [v2:172.21.15.161:6835/3832540545,v1:172.21.15.161:6837/3832540545] conn(0x55a149c317f0 0x55a149c31bd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.811+0000 7fad511fe740 1 --2- 172.21.15.161:0/3917281363 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a149c16190 0x55a149c5f330 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.811+0000 7fad511fe740 1 --2- 172.21.15.161:0/3917281363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a149c62910 0x55a149cc5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.811+0000 7fad511fe740 1 --2- 172.21.15.161:0/3917281363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a149c15db0 0x55a149c76480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.811+0000 7fad511fe740 1 -- 172.21.15.161:0/3917281363 >> 172.21.15.161:0/3917281363 conn(0x55a149c14310 msgr2=0x55a149c3bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:53.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.811+0000 7fad26ffd640 0 client.4449 ms_handle_reset on v2:172.21.15.161:6835/3832540545 2024-03-28T23:29:53.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.812+0000 7fad511fe740 1 -- 172.21.15.161:0/3917281363 shutdown_connections 2024-03-28T23:29:53.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.812+0000 7fad511fe740 1 -- 172.21.15.161:0/3917281363 wait complete. 2024-03-28T23:29:53.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.812+0000 7fad511fe740 20 client.4449 trim_cache size 0 max 16384 2024-03-28T23:29:53.822 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:29:53.822 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:29:53.823 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:29:53.823 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 4275, 2024-03-28T23:29:53.823 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:29:53.823 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:29:53.823 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:29:53.823 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 7.2684856230000001, 2024-03-28T23:29:53.823 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 7, 2024-03-28T23:29:53.823 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 24, 2024-03-28T23:29:53.823 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 24, 2024-03-28T23:29:53.823 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 13.943834580000001 2024-03-28T23:29:53.823 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:29:53.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.812+0000 7fad4e53e640 1 -- 172.21.15.161:0/4123351906 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fad3406b4d0 msgr2=0x7fad3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:53.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.812+0000 7fad4e53e640 1 --2- 172.21.15.161:0/4123351906 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fad3406b4d0 0x7fad3406d990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fad38036000 tx=0x7fad38008030 comp rx=0 tx=0).stop 2024-03-28T23:29:53.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.812+0000 7fad4e53e640 1 -- 172.21.15.161:0/4123351906 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad480c8ed0 msgr2=0x7fad48155c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:53.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.812+0000 7fad4e53e640 1 --2- 172.21.15.161:0/4123351906 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad480c8ed0 0x7fad48155c00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fad3c00c3f0 tx=0x7fad3c00c8b0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.812+0000 7fad4e53e640 1 -- 172.21.15.161:0/4123351906 shutdown_connections 2024-03-28T23:29:53.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.812+0000 7fad4e53e640 1 --2- 172.21.15.161:0/4123351906 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fad3406b4d0 0x7fad3406d990 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.812+0000 7fad4e53e640 1 --2- 172.21.15.161:0/4123351906 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad480c8ed0 0x7fad48155c00 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.812+0000 7fad4e53e640 1 --2- 172.21.15.161:0/4123351906 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad48152890 0x7fad480aa3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.812+0000 7fad4e53e640 1 --2- 172.21.15.161:0/4123351906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad480c5b20 0x7fad48153c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.812+0000 7fad4e53e640 1 -- 172.21.15.161:0/4123351906 >> 172.21.15.161:0/4123351906 conn(0x7fad480ac620 msgr2=0x7fad48151070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:53.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.812+0000 7fad4e53e640 1 -- 172.21.15.161:0/4123351906 shutdown_connections 2024-03-28T23:29:53.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.812+0000 7fad4e53e640 1 -- 172.21.15.161:0/4123351906 wait complete. 2024-03-28T23:29:53.826 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-28T23:29:53.826 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-28T23:29:53.827 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-28T23:29:53.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.928+0000 7f36daec9640 1 Processor -- start 2024-03-28T23:29:53.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.929+0000 7f36daec9640 1 -- start start 2024-03-28T23:29:53.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.929+0000 7f36daec9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f36d414f240 0x7f36d41591e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.929+0000 7f36daec9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36d4155d30 con 0x7f36d40c3ae0 2024-03-28T23:29:53.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.929+0000 7f36daec9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36d40c1e10 con 0x7f36d40c3410 2024-03-28T23:29:53.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.929+0000 7f36daec9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f36d4156d10 con 0x7f36d414f240 2024-03-28T23:29:53.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.929+0000 7f36d9466640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f36d40c3ae0 0x7f36d40c66a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52252/0 (socket says 172.21.15.161:52252) 2024-03-28T23:29:53.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.929+0000 7f36d9466640 1 -- 172.21.15.161:0/371475874 learned_addr learned my addr 172.21.15.161:0/371475874 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:53.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d8c65640 1 --2- 172.21.15.161:0/371475874 >> v2:172.21.15.170:3300/0 conn(0x7f36d414f240 0x7f36d41591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d37fe640 1 -- 172.21.15.161:0/371475874 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1457792361 0 0) 0x7f36d4155d30 con 0x7f36d40c3ae0 2024-03-28T23:29:53.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d37fe640 1 -- 172.21.15.161:0/371475874 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36c0003520 con 0x7f36d40c3ae0 2024-03-28T23:29:53.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d37fe640 1 -- 172.21.15.161:0/371475874 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3243498199 0 0) 0x7f36c0003520 con 0x7f36d40c3ae0 2024-03-28T23:29:53.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d37fe640 1 -- 172.21.15.161:0/371475874 >> v2:172.21.15.170:3300/0 conn(0x7f36d414f240 msgr2=0x7f36d41591e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:53.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d37fe640 1 --2- 172.21.15.161:0/371475874 >> v2:172.21.15.170:3300/0 conn(0x7f36d414f240 0x7f36d41591e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d37fe640 1 -- 172.21.15.161:0/371475874 >> v1:172.21.15.170:6789/0 conn(0x7f36d40c3410 legacy=0x7f36d4156600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:53.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d37fe640 1 -- 172.21.15.161:0/371475874 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36d4155fd0 con 0x7f36d40c3ae0 2024-03-28T23:29:53.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d37fe640 1 -- 172.21.15.161:0/371475874 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f36cc0019f0 con 0x7f36d40c3ae0 2024-03-28T23:29:53.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d8c65640 1 --2- 172.21.15.161:0/371475874 >> v2:172.21.15.170:3300/0 conn(0x7f36d414f240 0x7f36d41591e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:29:53.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d37fe640 1 -- 172.21.15.161:0/371475874 >> v1:172.21.15.161:6789/0 conn(0x7f36d40c3ae0 legacy=0x7f36d40c66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:53.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d37fe640 1 --2- 172.21.15.161:0/371475874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36c0003a90 0x7f36c0003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d37fe640 1 --2- 172.21.15.161:0/371475874 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36c0004800 0x7f36c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d37fe640 1 --2- 172.21.15.161:0/371475874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36c0005290 0x7f36c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d37fe640 1 -- 172.21.15.161:0/371475874 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f36d4156d10 con 0x7f36c0003a90 2024-03-28T23:29:53.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d37fe640 1 -- 172.21.15.161:0/371475874 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f36c000c1e0 con 0x7f36c0005290 2024-03-28T23:29:53.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d37fe640 1 -- 172.21.15.161:0/371475874 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f36c000c3e0 con 0x7f36c0004800 2024-03-28T23:29:53.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d8c65640 1 --2- 172.21.15.161:0/371475874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36c0003a90 0x7f36c0003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d3fff640 1 --2- 172.21.15.161:0/371475874 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36c0004800 0x7f36c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d9466640 1 --2- 172.21.15.161:0/371475874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36c0005290 0x7f36c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d8c65640 1 -- 172.21.15.161:0/371475874 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36c0004800 msgr2=0x7f36c0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:53.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d8c65640 1 --2- 172.21.15.161:0/371475874 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36c0004800 0x7f36c0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d8c65640 1 -- 172.21.15.161:0/371475874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36c0005290 msgr2=0x7f36c000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:53.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d8c65640 1 --2- 172.21.15.161:0/371475874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36c0005290 0x7f36c000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d9466640 1 -- 172.21.15.161:0/371475874 reap_dead start 2024-03-28T23:29:53.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.930+0000 7f36d8c65640 1 -- 172.21.15.161:0/371475874 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36c0003520 con 0x7f36c0003a90 2024-03-28T23:29:53.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.932+0000 7f36d8c65640 1 --2- 172.21.15.161:0/371475874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36c0003a90 0x7f36c0003eb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f36bc009c90 tx=0x7f36c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d81c5d78fc22f5d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:53.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.932+0000 7f36d37fe640 1 -- 172.21.15.161:0/371475874 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36bc039070 con 0x7f36c0003a90 2024-03-28T23:29:53.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.932+0000 7f36d37fe640 1 -- 172.21.15.161:0/371475874 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36bc02c8f0 con 0x7f36c0003a90 2024-03-28T23:29:53.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.932+0000 7f36d37fe640 1 -- 172.21.15.161:0/371475874 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36bc009860 con 0x7f36c0003a90 2024-03-28T23:29:53.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.932+0000 7f36daec9640 1 -- 172.21.15.161:0/371475874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36c0003a90 msgr2=0x7f36c0003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:53.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.932+0000 7f36daec9640 1 --2- 172.21.15.161:0/371475874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36c0003a90 0x7f36c0003eb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f36bc009c90 tx=0x7f36c0005830 comp rx=0 tx=0).stop 2024-03-28T23:29:53.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.932+0000 7f36daec9640 1 -- 172.21.15.161:0/371475874 shutdown_connections 2024-03-28T23:29:53.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.932+0000 7f36daec9640 1 --2- 172.21.15.161:0/371475874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36c0003a90 0x7f36c0003eb0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.932+0000 7f36daec9640 1 -- 172.21.15.161:0/371475874 >> 172.21.15.161:0/371475874 conn(0x7f36d40c7790 msgr2=0x7f36d40c7ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:53.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.932+0000 7f36daec9640 1 -- 172.21.15.161:0/371475874 shutdown_connections 2024-03-28T23:29:53.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.932+0000 7f36daec9640 1 -- 172.21.15.161:0/371475874 wait complete. 2024-03-28T23:29:53.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.933+0000 7f36daec9640 1 Processor -- start 2024-03-28T23:29:53.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.933+0000 7f36daec9640 1 -- start start 2024-03-28T23:29:53.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.934+0000 7f36daec9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36c0003a90 0x7f36d41eed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.934+0000 7f36daec9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36d4149ca0 0x7f36d4145b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.934+0000 7f36daec9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36d4146060 0x7f36d41eae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.934+0000 7f36daec9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f36d41589e0 con 0x7f36c0003a90 2024-03-28T23:29:53.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.934+0000 7f36daec9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f36d414f620 con 0x7f36d4149ca0 2024-03-28T23:29:53.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.934+0000 7f36daec9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f36d4158cc0 con 0x7f36d4146060 2024-03-28T23:29:53.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.934+0000 7f36d8c65640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36c0003a90 0x7f36d41eed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.934+0000 7f36d8c65640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36c0003a90 0x7f36d41eed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55134/0 (socket says 172.21.15.161:55134) 2024-03-28T23:29:53.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.934+0000 7f36d8c65640 1 -- 172.21.15.161:0/4188110492 learned_addr learned my addr 172.21.15.161:0/4188110492 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:53.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.934+0000 7f36d9466640 1 --2- 172.21.15.161:0/4188110492 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36d4146060 0x7f36d41eae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.934+0000 7f36d3fff640 1 --2- 172.21.15.161:0/4188110492 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36d4149ca0 0x7f36d4145b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.935+0000 7f36d8c65640 1 -- 172.21.15.161:0/4188110492 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36d4146060 msgr2=0x7f36d41eae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:53.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.935+0000 7f36d8c65640 1 --2- 172.21.15.161:0/4188110492 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36d4146060 0x7f36d41eae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.935+0000 7f36d8c65640 1 -- 172.21.15.161:0/4188110492 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36d4149ca0 msgr2=0x7f36d4145b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:53.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.935+0000 7f36d8c65640 1 --2- 172.21.15.161:0/4188110492 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36d4149ca0 0x7f36d4145b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:53.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.935+0000 7f36d8c65640 1 -- 172.21.15.161:0/4188110492 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36d4148570 con 0x7f36c0003a90 2024-03-28T23:29:53.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.935+0000 7f36d3fff640 1 --2- 172.21.15.161:0/4188110492 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36d4149ca0 0x7f36d4145b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:29:53.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.935+0000 7f36d8c65640 1 --2- 172.21.15.161:0/4188110492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36c0003a90 0x7f36d41eed80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f36bc009db0 tx=0x7f36bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:53.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.935+0000 7f36d17fa640 1 -- 172.21.15.161:0/4188110492 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36bc044020 con 0x7f36c0003a90 2024-03-28T23:29:53.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.935+0000 7f36d17fa640 1 -- 172.21.15.161:0/4188110492 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36bc03a040 con 0x7f36c0003a90 2024-03-28T23:29:53.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.935+0000 7f36daec9640 1 -- 172.21.15.161:0/4188110492 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36d4148850 con 0x7f36c0003a90 2024-03-28T23:29:53.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.935+0000 7f36d17fa640 1 -- 172.21.15.161:0/4188110492 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36bc039050 con 0x7f36c0003a90 2024-03-28T23:29:53.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.935+0000 7f36daec9640 1 -- 172.21.15.161:0/4188110492 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36d41464e0 con 0x7f36c0003a90 2024-03-28T23:29:53.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.936+0000 7f36daec9640 1 -- 172.21.15.161:0/4188110492 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36d404f4b0 con 0x7f36c0003a90 2024-03-28T23:29:53.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.936+0000 7f36d17fa640 1 -- 172.21.15.161:0/4188110492 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f36bc005990 con 0x7f36c0003a90 2024-03-28T23:29:53.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.937+0000 7f36d17fa640 1 --2- 172.21.15.161:0/4188110492 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f36b406b500 0x7f36b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:53.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.937+0000 7f36d17fa640 1 -- 172.21.15.161:0/4188110492 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f36bc0ac140 con 0x7f36c0003a90 2024-03-28T23:29:53.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.937+0000 7f36d3fff640 1 --2- 172.21.15.161:0/4188110492 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f36b406b500 0x7f36b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:53.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.937+0000 7f36d3fff640 1 --2- 172.21.15.161:0/4188110492 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f36b406b500 0x7f36b406d9c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f36d41e8c40 tx=0x7f36c4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:53.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:53.941+0000 7f36d17fa640 1 -- 172.21.15.161:0/4188110492 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f36bc04e060 con 0x7f36c0003a90 2024-03-28T23:29:54.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:54.150+0000 7f36daec9640 1 -- 172.21.15.161:0/4188110492 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f36d40c31f0 con 0x7f36c0003a90 2024-03-28T23:29:54.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:54.150+0000 7f36d17fa640 1 -- 172.21.15.161:0/4188110492 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+88 (secure 0 0 0) 0x7f36bc07ae30 con 0x7f36c0003a90 2024-03-28T23:29:54.151 INFO:teuthology.orchestra.run.smithi161.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-28T23:29:54.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:54.152+0000 7f36daec9640 1 -- 172.21.15.161:0/4188110492 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f36b406b500 msgr2=0x7f36b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:54.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:54.152+0000 7f36daec9640 1 --2- 172.21.15.161:0/4188110492 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f36b406b500 0x7f36b406d9c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f36d41e8c40 tx=0x7f36c4009040 comp rx=0 tx=0).stop 2024-03-28T23:29:54.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:54.152+0000 7f36daec9640 1 -- 172.21.15.161:0/4188110492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36c0003a90 msgr2=0x7f36d41eed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:54.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:54.152+0000 7f36daec9640 1 --2- 172.21.15.161:0/4188110492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36c0003a90 0x7f36d41eed80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f36bc009db0 tx=0x7f36bc004730 comp rx=0 tx=0).stop 2024-03-28T23:29:54.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:54.152+0000 7f36daec9640 1 -- 172.21.15.161:0/4188110492 shutdown_connections 2024-03-28T23:29:54.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:54.152+0000 7f36daec9640 1 --2- 172.21.15.161:0/4188110492 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f36b406b500 0x7f36b406d9c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:54.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:54.152+0000 7f36daec9640 1 --2- 172.21.15.161:0/4188110492 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36d4146060 0x7f36d41eae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:54.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:54.152+0000 7f36daec9640 1 --2- 172.21.15.161:0/4188110492 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36d4149ca0 0x7f36d4145b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:54.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:54.152+0000 7f36daec9640 1 --2- 172.21.15.161:0/4188110492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36c0003a90 0x7f36d41eed80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:54.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:54.152+0000 7f36daec9640 1 -- 172.21.15.161:0/4188110492 >> 172.21.15.161:0/4188110492 conn(0x7f36d40c7790 msgr2=0x7f36d4156210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:54.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:54.152+0000 7f36daec9640 1 -- 172.21.15.161:0/4188110492 shutdown_connections 2024-03-28T23:29:54.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:54.152+0000 7f36daec9640 1 -- 172.21.15.161:0/4188110492 wait complete. 2024-03-28T23:29:54.164 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-28T23:29:54.164 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T23:29:54.164 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi161.front.sepia.ceph.com 2024-03-28T23:29:54.164 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-28T23:29:54.164 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-28T23:29:54.164 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-28T23:29:54.164 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T23:29:54.164 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T23:29:54.164 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-28T23:29:54.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:29:54.165 DEBUG:teuthology.orchestra.run.smithi161:> ip addr 2024-03-28T23:29:54.180 INFO:teuthology.orchestra.run.smithi161.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T23:29:54.180 INFO:teuthology.orchestra.run.smithi161.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T23:29:54.180 INFO:teuthology.orchestra.run.smithi161.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T23:29:54.180 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-28T23:29:54.180 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 ::1/128 scope host 2024-03-28T23:29:54.180 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-28T23:29:54.180 INFO:teuthology.orchestra.run.smithi161.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T23:29:54.180 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:d9:36:7c brd ff:ff:ff:ff:ff:ff 2024-03-28T23:29:54.180 INFO:teuthology.orchestra.run.smithi161.stdout: altname enp7s0f0 2024-03-28T23:29:54.180 INFO:teuthology.orchestra.run.smithi161.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T23:29:54.181 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:d9:36:7d brd ff:ff:ff:ff:ff:ff 2024-03-28T23:29:54.181 INFO:teuthology.orchestra.run.smithi161.stdout: altname enp7s0f1 2024-03-28T23:29:54.181 INFO:teuthology.orchestra.run.smithi161.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T23:29:54.181 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:88:72:50 brd ff:ff:ff:ff:ff:ff 2024-03-28T23:29:54.181 INFO:teuthology.orchestra.run.smithi161.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T23:29:54.181 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:88:72:51 brd ff:ff:ff:ff:ff:ff 2024-03-28T23:29:54.181 INFO:teuthology.orchestra.run.smithi161.stdout: inet 172.21.15.161/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-28T23:29:54.181 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft 42763sec preferred_lft 42763sec 2024-03-28T23:29:54.181 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::ec4:7aff:fe88:7251/64 scope link 2024-03-28T23:29:54.181 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-28T23:29:54.182 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-28T23:29:54.182 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T23:29:54.182 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-28T23:29:54.182 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link add name ceph-brx type bridge 2024-03-28T23:29:54.182 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip addr flush dev ceph-brx 2024-03-28T23:29:54.182 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link set ceph-brx up 2024-03-28T23:29:54.182 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-28T23:29:54.182 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-28T23:29:54.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T23:29:54.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T23:29:54.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:29:54.387 DEBUG:teuthology.orchestra.run.smithi161:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T23:29:54.415 INFO:teuthology.orchestra.run.smithi161.stdout:1 2024-03-28T23:29:54.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:29:54.417 DEBUG:teuthology.orchestra.run.smithi161:> route 2024-03-28T23:29:54.484 INFO:teuthology.orchestra.run.smithi161.stdout:Kernel IP routing table 2024-03-28T23:29:54.484 INFO:teuthology.orchestra.run.smithi161.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T23:29:54.484 INFO:teuthology.orchestra.run.smithi161.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-28T23:29:54.485 INFO:teuthology.orchestra.run.smithi161.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-28T23:29:54.485 INFO:teuthology.orchestra.run.smithi161.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T23:29:54.485 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T23:29:54.485 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-28T23:29:54.485 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-28T23:29:54.485 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-28T23:29:54.485 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-28T23:29:54.485 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-28T23:29:54.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T23:29:54.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T23:29:54.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:29:54.780 DEBUG:teuthology.orchestra.run.smithi161:> ip netns list 2024-03-28T23:29:54.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:29:54.795 DEBUG:teuthology.orchestra.run.smithi161:> ip netns list-id 2024-03-28T23:29:54.850 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T23:29:54.850 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-28T23:29:54.850 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T23:29:54.850 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-28T23:29:54.850 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-28T23:29:54.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T23:29:54.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T23:29:54.974 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-28T23:29:54.974 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T23:29:54.974 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-28T23:29:54.974 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-28T23:29:54.974 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-28T23:29:54.975 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-28T23:29:54.975 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-28T23:29:54.975 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-28T23:29:54.975 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-28T23:29:55.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T23:29:55.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T23:29:55.111 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T23:29:55.111 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-28T23:29:55.112 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link set brx.0 up 2024-03-28T23:29:55.112 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link set dev brx.0 master ceph-brx 2024-03-28T23:29:55.112 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-28T23:29:55.134 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:29:55.157 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:29:55.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T23:29:55.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T23:29:55.197 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-28T23:29:55.198 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T23:29:55.198 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-28T23:29:55.269 INFO:teuthology.orchestra.run.smithi161.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-28T23:29:55.270 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T23:29:55.270 DEBUG:teuthology.orchestra.run.smithi161:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-28T23:29:55.324 DEBUG:teuthology.orchestra.run.smithi161:> sudo modprobe fuse 2024-03-28T23:29:55.395 DEBUG:teuthology.orchestra.run.smithi161:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T23:29:55.455 INFO:teuthology.orchestra.run.smithi161.stdout:/proc 2024-03-28T23:29:55.455 INFO:teuthology.orchestra.run.smithi161.stdout:/sys 2024-03-28T23:29:55.455 INFO:teuthology.orchestra.run.smithi161.stdout:/dev 2024-03-28T23:29:55.455 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/security 2024-03-28T23:29:55.455 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/shm 2024-03-28T23:29:55.455 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/pts 2024-03-28T23:29:55.455 INFO:teuthology.orchestra.run.smithi161.stdout:/run 2024-03-28T23:29:55.455 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/cgroup 2024-03-28T23:29:55.455 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/pstore 2024-03-28T23:29:55.455 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/bpf 2024-03-28T23:29:55.456 INFO:teuthology.orchestra.run.smithi161.stdout:/ 2024-03-28T23:29:55.456 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/selinux 2024-03-28T23:29:55.456 INFO:teuthology.orchestra.run.smithi161.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T23:29:55.456 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/mqueue 2024-03-28T23:29:55.456 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/hugepages 2024-03-28T23:29:55.456 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/debug 2024-03-28T23:29:55.456 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/tracing 2024-03-28T23:29:55.456 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/fuse/connections 2024-03-28T23:29:55.456 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/config 2024-03-28T23:29:55.456 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T23:29:55.456 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T23:29:55.456 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T23:29:55.456 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T23:29:55.456 INFO:teuthology.orchestra.run.smithi161.stdout:/run/user/1000 2024-03-28T23:29:55.457 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph 2024-03-28T23:29:55.457 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-0 2024-03-28T23:29:55.457 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-1 2024-03-28T23:29:55.457 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-2 2024-03-28T23:29:55.457 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-3 2024-03-28T23:29:55.457 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns 2024-03-28T23:29:55.457 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T23:29:55.457 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T23:29:55.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:29:55.457 DEBUG:teuthology.orchestra.run.smithi161:> ls /sys/fs/fuse/connections 2024-03-28T23:29:55.513 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-28T23:29:55.514 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-28T23:29:55.561 DEBUG:teuthology.orchestra.run.smithi161:> sudo modprobe fuse 2024-03-28T23:29:55.588 DEBUG:teuthology.orchestra.run.smithi161:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T23:29:55.647 INFO:teuthology.orchestra.run.smithi161.stdout:/proc 2024-03-28T23:29:55.647 INFO:teuthology.orchestra.run.smithi161.stdout:/sys 2024-03-28T23:29:55.647 INFO:teuthology.orchestra.run.smithi161.stdout:/dev 2024-03-28T23:29:55.647 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/security 2024-03-28T23:29:55.647 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/shm 2024-03-28T23:29:55.647 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/pts 2024-03-28T23:29:55.647 INFO:teuthology.orchestra.run.smithi161.stdout:/run 2024-03-28T23:29:55.648 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/cgroup 2024-03-28T23:29:55.648 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/pstore 2024-03-28T23:29:55.648 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/bpf 2024-03-28T23:29:55.648 INFO:teuthology.orchestra.run.smithi161.stdout:/ 2024-03-28T23:29:55.648 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/selinux 2024-03-28T23:29:55.648 INFO:teuthology.orchestra.run.smithi161.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T23:29:55.648 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/mqueue 2024-03-28T23:29:55.648 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/hugepages 2024-03-28T23:29:55.648 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/debug 2024-03-28T23:29:55.648 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/tracing 2024-03-28T23:29:55.648 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/fuse/connections 2024-03-28T23:29:55.648 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/config 2024-03-28T23:29:55.648 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T23:29:55.649 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T23:29:55.649 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T23:29:55.649 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T23:29:55.649 INFO:teuthology.orchestra.run.smithi161.stdout:/run/user/1000 2024-03-28T23:29:55.649 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph 2024-03-28T23:29:55.649 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-0 2024-03-28T23:29:55.649 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-1 2024-03-28T23:29:55.649 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-2 2024-03-28T23:29:55.649 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-3 2024-03-28T23:29:55.649 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns 2024-03-28T23:29:55.649 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T23:29:55.649 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T23:29:55.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:29:55.650 DEBUG:teuthology.orchestra.run.smithi161:> ls /sys/fs/fuse/connections 2024-03-28T23:29:55.651 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi161.stderr:2024-03-28T23:29:55.649+0000 7faefc3f4480 -1 init, newargv = 0x55feefbd9a20 newargc=15 2024-03-28T23:29:55.651 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi161.stderr:ceph-fuse[42482]: starting ceph client 2024-03-28T23:29:55.655 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi161.stderr:ceph-fuse[42482]: starting fuse 2024-03-28T23:29:55.672 INFO:teuthology.orchestra.run.smithi161.stdout:41 2024-03-28T23:29:55.672 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-28T23:29:55.672 DEBUG:teuthology.orchestra.run.smithi161:> sudo stdin-killer -- python3 -c ' 2024-03-28T23:29:55.672 DEBUG:teuthology.orchestra.run.smithi161:> import glob 2024-03-28T23:29:55.672 DEBUG:teuthology.orchestra.run.smithi161:> import re 2024-03-28T23:29:55.672 DEBUG:teuthology.orchestra.run.smithi161:> import os 2024-03-28T23:29:55.673 DEBUG:teuthology.orchestra.run.smithi161:> import subprocess 2024-03-28T23:29:55.673 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-28T23:29:55.673 DEBUG:teuthology.orchestra.run.smithi161:> def _find_admin_socket(client_name): 2024-03-28T23:29:55.673 DEBUG:teuthology.orchestra.run.smithi161:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-28T23:29:55.673 DEBUG:teuthology.orchestra.run.smithi161:> files = glob.glob(asok_path) 2024-03-28T23:29:55.673 DEBUG:teuthology.orchestra.run.smithi161:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-28T23:29:55.673 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-28T23:29:55.673 DEBUG:teuthology.orchestra.run.smithi161:> # Given a non-glob path, it better be there 2024-03-28T23:29:55.673 DEBUG:teuthology.orchestra.run.smithi161:> if "*" not in asok_path: 2024-03-28T23:29:55.673 DEBUG:teuthology.orchestra.run.smithi161:> assert(len(files) == 1) 2024-03-28T23:29:55.673 DEBUG:teuthology.orchestra.run.smithi161:> return files[0] 2024-03-28T23:29:55.673 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-28T23:29:55.673 DEBUG:teuthology.orchestra.run.smithi161:> for f in files: 2024-03-28T23:29:55.673 DEBUG:teuthology.orchestra.run.smithi161:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-28T23:29:55.673 DEBUG:teuthology.orchestra.run.smithi161:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-28T23:29:55.673 DEBUG:teuthology.orchestra.run.smithi161:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-28T23:29:55.674 DEBUG:teuthology.orchestra.run.smithi161:> contents = proc_f.read() 2024-03-28T23:29:55.674 DEBUG:teuthology.orchestra.run.smithi161:> if mountpoint in contents: 2024-03-28T23:29:55.674 DEBUG:teuthology.orchestra.run.smithi161:> return f 2024-03-28T23:29:55.674 DEBUG:teuthology.orchestra.run.smithi161:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-28T23:29:55.674 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-28T23:29:55.674 DEBUG:teuthology.orchestra.run.smithi161:> print(_find_admin_socket("client.admin")) 2024-03-28T23:29:55.674 DEBUG:teuthology.orchestra.run.smithi161:> ' 2024-03-28T23:29:55.796 INFO:teuthology.orchestra.run.smithi161.stdout:/var/run/ceph/ceph-client.admin.42482.asok 2024-03-28T23:29:55.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T23:29:55.808 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.42482.asok 2024-03-28T23:29:55.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:29:55.808 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.42482.asok status 2024-03-28T23:29:55.949 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:29:55.949 INFO:teuthology.orchestra.run.smithi161.stdout: "metadata": { 2024-03-28T23:29:55.949 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_sha1": "323d23dc228bbcb7f16f188fdc9c0dcb801844be", 2024-03-28T23:29:55.949 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_version": "ceph version 19.0.0-1899-g323d23dc (323d23dc228bbcb7f16f188fdc9c0dcb801844be) squid (dev)", 2024-03-28T23:29:55.949 INFO:teuthology.orchestra.run.smithi161.stdout: "entity_id": "admin", 2024-03-28T23:29:55.949 INFO:teuthology.orchestra.run.smithi161.stdout: "hostname": "smithi161", 2024-03-28T23:29:55.949 INFO:teuthology.orchestra.run.smithi161.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-28T23:29:55.950 INFO:teuthology.orchestra.run.smithi161.stdout: "pid": "42482", 2024-03-28T23:29:55.950 INFO:teuthology.orchestra.run.smithi161.stdout: "root": "/" 2024-03-28T23:29:55.950 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:29:55.950 INFO:teuthology.orchestra.run.smithi161.stdout: "dentry_count": 0, 2024-03-28T23:29:55.950 INFO:teuthology.orchestra.run.smithi161.stdout: "dentry_pinned_count": 0, 2024-03-28T23:29:55.950 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 4461, 2024-03-28T23:29:55.950 INFO:teuthology.orchestra.run.smithi161.stdout: "inst": { 2024-03-28T23:29:55.950 INFO:teuthology.orchestra.run.smithi161.stdout: "name": { 2024-03-28T23:29:55.950 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "client", 2024-03-28T23:29:55.950 INFO:teuthology.orchestra.run.smithi161.stdout: "num": 4461 2024-03-28T23:29:55.950 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:29:55.951 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": { 2024-03-28T23:29:55.951 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "v1", 2024-03-28T23:29:55.951 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": "192.168.0.1:0", 2024-03-28T23:29:55.951 INFO:teuthology.orchestra.run.smithi161.stdout: "nonce": 4268201179 2024-03-28T23:29:55.951 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:29:55.951 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:29:55.951 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": { 2024-03-28T23:29:55.951 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "v1", 2024-03-28T23:29:55.951 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": "192.168.0.1:0", 2024-03-28T23:29:55.951 INFO:teuthology.orchestra.run.smithi161.stdout: "nonce": 4268201179 2024-03-28T23:29:55.951 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:29:55.952 INFO:teuthology.orchestra.run.smithi161.stdout: "inst_str": "client.4461 192.168.0.1:0/4268201179", 2024-03-28T23:29:55.952 INFO:teuthology.orchestra.run.smithi161.stdout: "addr_str": "192.168.0.1:0/4268201179", 2024-03-28T23:29:55.952 INFO:teuthology.orchestra.run.smithi161.stdout: "inode_count": 1, 2024-03-28T23:29:55.952 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_epoch": 7, 2024-03-28T23:29:55.952 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_epoch": 24, 2024-03-28T23:29:55.952 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_epoch_barrier": 0, 2024-03-28T23:29:55.952 INFO:teuthology.orchestra.run.smithi161.stdout: "blocklisted": false, 2024-03-28T23:29:55.952 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs" 2024-03-28T23:29:55.952 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:29:55.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:29:55.957 DEBUG:teuthology.orchestra.run.smithi161:> stat --file-system '--printf=%T 2024-03-28T23:29:55.957 DEBUG:teuthology.orchestra.run.smithi161:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-28T23:29:55.972 INFO:teuthology.orchestra.run.smithi161.stdout:fuseblk 2024-03-28T23:29:55.972 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-28T23:29:55.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:29:55.973 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-28T23:29:56.042 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-28T23:29:56.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-28T23:29:56.152 INFO:teuthology.orchestra.run.smithi161.stdout:# file: . 2024-03-28T23:29:56.153 INFO:teuthology.orchestra.run.smithi161.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-28T23:29:56.153 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:29:56.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T23:29:56.159 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:29:56.159 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:29:56.175 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:29:56.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.316+0000 7f6c011e2640 1 Processor -- start 2024-03-28T23:29:56.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.316+0000 7f6c011e2640 1 -- start start 2024-03-28T23:29:56.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.317+0000 7f6c011e2640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6bfc152890 0x7f6bfc0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.317+0000 7f6c011e2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6bfc0c8ed0 0x7f6bfc0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.317+0000 7f6c011e2640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bfc158a30 con 0x7f6bfc0c5950 2024-03-28T23:29:56.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.317+0000 7f6c011e2640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f6bfc0c8480 con 0x7f6bfc152890 2024-03-28T23:29:56.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.317+0000 7f6c011e2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6bfc159640 con 0x7f6bfc0c8ed0 2024-03-28T23:29:56.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.317+0000 7f6bfb59e640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6bfc0c5950 0x7f6bfc1553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52254/0 (socket says 172.21.15.161:52254) 2024-03-28T23:29:56.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.317+0000 7f6bfb59e640 1 -- 172.21.15.161:0/2320160970 learned_addr learned my addr 172.21.15.161:0/2320160970 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:56.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bfa59c640 1 -- 172.21.15.161:0/2320160970 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 167864599 0 0) 0x7f6bfc158a30 con 0x7f6bfc0c5950 2024-03-28T23:29:56.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bf3fff640 1 --2- 172.21.15.161:0/2320160970 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6bfc0c8ed0 0x7f6bfc0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bfa59c640 1 -- 172.21.15.161:0/2320160970 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6be8003520 con 0x7f6bfc0c5950 2024-03-28T23:29:56.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bfad9d640 1 --2- 172.21.15.161:0/2320160970 >> v2:172.21.15.170:3300/0 conn(0x7f6bfc152890 0x7f6bfc0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bfa59c640 1 -- 172.21.15.161:0/2320160970 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3319516671 0 0) 0x7f6be8003520 con 0x7f6bfc0c5950 2024-03-28T23:29:56.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bfa59c640 1 -- 172.21.15.161:0/2320160970 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6bfc0c8ed0 msgr2=0x7f6bfc0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:56.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bfa59c640 1 --2- 172.21.15.161:0/2320160970 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6bfc0c8ed0 0x7f6bfc0c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bfa59c640 1 -- 172.21.15.161:0/2320160970 >> v2:172.21.15.170:3300/0 conn(0x7f6bfc152890 msgr2=0x7f6bfc0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:56.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bfa59c640 1 --2- 172.21.15.161:0/2320160970 >> v2:172.21.15.170:3300/0 conn(0x7f6bfc152890 0x7f6bfc0c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bfa59c640 1 -- 172.21.15.161:0/2320160970 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bfc158d10 con 0x7f6bfc0c5950 2024-03-28T23:29:56.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bfa59c640 1 -- 172.21.15.161:0/2320160970 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f6bf40019f0 con 0x7f6bfc0c5950 2024-03-28T23:29:56.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bfa59c640 1 -- 172.21.15.161:0/2320160970 >> v1:172.21.15.161:6789/0 conn(0x7f6bfc0c5950 legacy=0x7f6bfc1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:56.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bfa59c640 1 --2- 172.21.15.161:0/2320160970 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6bfc0c8ed0 0x7f6be8003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bfa59c640 1 --2- 172.21.15.161:0/2320160970 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6be8004010 0x7f6be8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bfa59c640 1 --2- 172.21.15.161:0/2320160970 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6be8004d40 0x7f6be80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bfa59c640 1 -- 172.21.15.161:0/2320160970 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc159640 con 0x7f6be8004010 2024-03-28T23:29:56.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bfa59c640 1 -- 172.21.15.161:0/2320160970 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6be800bb00 con 0x7f6be8004d40 2024-03-28T23:29:56.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bfa59c640 1 -- 172.21.15.161:0/2320160970 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6be800bd00 con 0x7f6bfc0c8ed0 2024-03-28T23:29:56.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bf3fff640 1 --2- 172.21.15.161:0/2320160970 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6be8004010 0x7f6be8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bfad9d640 1 --2- 172.21.15.161:0/2320160970 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6bfc0c8ed0 0x7f6be8003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bfb59e640 1 --2- 172.21.15.161:0/2320160970 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6be8004d40 0x7f6be80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bf3fff640 1 -- 172.21.15.161:0/2320160970 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6bfc0c8ed0 msgr2=0x7f6be8003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:56.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bf3fff640 1 --2- 172.21.15.161:0/2320160970 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6bfc0c8ed0 0x7f6be8003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bf3fff640 1 -- 172.21.15.161:0/2320160970 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6be8004d40 msgr2=0x7f6be80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:56.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bf3fff640 1 --2- 172.21.15.161:0/2320160970 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6be8004d40 0x7f6be80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bf3fff640 1 -- 172.21.15.161:0/2320160970 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6be8003520 con 0x7f6be8004010 2024-03-28T23:29:56.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.318+0000 7f6bfad9d640 1 --2- 172.21.15.161:0/2320160970 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6bfc0c8ed0 0x7f6be8003a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:29:56.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.319+0000 7f6bf3fff640 1 --2- 172.21.15.161:0/2320160970 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6be8004010 0x7f6be8004800 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6be4009a80 tx=0x7f6be8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92de519e380a1583 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:56.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.319+0000 7f6bfa59c640 1 -- 172.21.15.161:0/2320160970 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be4009680 con 0x7f6be8004010 2024-03-28T23:29:56.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.319+0000 7f6bfa59c640 1 -- 172.21.15.161:0/2320160970 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6be4005990 con 0x7f6be8004010 2024-03-28T23:29:56.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.319+0000 7f6bfa59c640 1 -- 172.21.15.161:0/2320160970 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be4005d80 con 0x7f6be8004010 2024-03-28T23:29:56.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.320+0000 7f6c011e2640 1 -- 172.21.15.161:0/2320160970 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6be8004010 msgr2=0x7f6be8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:56.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.320+0000 7f6c011e2640 1 --2- 172.21.15.161:0/2320160970 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6be8004010 0x7f6be8004800 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6be4009a80 tx=0x7f6be8005830 comp rx=0 tx=0).stop 2024-03-28T23:29:56.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.320+0000 7f6bfb59e640 1 -- 172.21.15.161:0/2320160970 reap_dead start 2024-03-28T23:29:56.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.320+0000 7f6c011e2640 1 -- 172.21.15.161:0/2320160970 shutdown_connections 2024-03-28T23:29:56.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.320+0000 7f6c011e2640 1 -- 172.21.15.161:0/2320160970 >> 172.21.15.161:0/2320160970 conn(0x7f6bfc0ac620 msgr2=0x7f6bfc0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:56.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.320+0000 7f6c011e2640 1 -- 172.21.15.161:0/2320160970 shutdown_connections 2024-03-28T23:29:56.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.320+0000 7f6c011e2640 1 -- 172.21.15.161:0/2320160970 wait complete. 2024-03-28T23:29:56.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.321+0000 7f6c011e2640 1 Processor -- start 2024-03-28T23:29:56.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.321+0000 7f6c011e2640 1 -- start start 2024-03-28T23:29:56.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.322+0000 7f6c011e2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6be8004010 0x7f6bfc1444e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.322+0000 7f6c011e2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6bfc1ee810 0x7f6bfc14ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.322+0000 7f6c011e2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6bfc14e2f0 0x7f6bfc143390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.322+0000 7f6c011e2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc152c70 con 0x7f6bfc1ee810 2024-03-28T23:29:56.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.322+0000 7f6c011e2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc1592d0 con 0x7f6bfc14e2f0 2024-03-28T23:29:56.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.322+0000 7f6c011e2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6bfc152df0 con 0x7f6be8004010 2024-03-28T23:29:56.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.322+0000 7f6bf3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6bfc1ee810 0x7f6bfc14ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.322+0000 7f6bf3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6bfc1ee810 0x7f6bfc14ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55160/0 (socket says 172.21.15.161:55160) 2024-03-28T23:29:56.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.322+0000 7f6bf3fff640 1 -- 172.21.15.161:0/3704704798 learned_addr learned my addr 172.21.15.161:0/3704704798 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:56.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.322+0000 7f6bfad9d640 1 --2- 172.21.15.161:0/3704704798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6be8004010 0x7f6bfc1444e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.322+0000 7f6bfb59e640 1 --2- 172.21.15.161:0/3704704798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6bfc14e2f0 0x7f6bfc143390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.322+0000 7f6bf3fff640 1 -- 172.21.15.161:0/3704704798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6be8004010 msgr2=0x7f6bfc1444e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:56.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.322+0000 7f6bf3fff640 1 --2- 172.21.15.161:0/3704704798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6be8004010 0x7f6bfc1444e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.322+0000 7f6bf3fff640 1 -- 172.21.15.161:0/3704704798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6bfc14e2f0 msgr2=0x7f6bfc143390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:56.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.322+0000 7f6bf3fff640 1 --2- 172.21.15.161:0/3704704798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6bfc14e2f0 0x7f6bfc143390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.322+0000 7f6bf3fff640 1 -- 172.21.15.161:0/3704704798 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bfc1eb340 con 0x7f6bfc1ee810 2024-03-28T23:29:56.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.322+0000 7f6bfad9d640 1 --2- 172.21.15.161:0/3704704798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6be8004010 0x7f6bfc1444e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:29:56.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.322+0000 7f6bfb59e640 1 --2- 172.21.15.161:0/3704704798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6bfc14e2f0 0x7f6bfc143390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:29:56.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.322+0000 7f6bf3fff640 1 --2- 172.21.15.161:0/3704704798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6bfc1ee810 0x7f6bfc14ddb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6be4009ba0 tx=0x7f6be4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:56.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.323+0000 7f6bf37fe640 1 -- 172.21.15.161:0/3704704798 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be4043020 con 0x7f6bfc1ee810 2024-03-28T23:29:56.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.323+0000 7f6bf37fe640 1 -- 172.21.15.161:0/3704704798 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6be402cca0 con 0x7f6bfc1ee810 2024-03-28T23:29:56.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.323+0000 7f6bf37fe640 1 -- 172.21.15.161:0/3704704798 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be4006270 con 0x7f6bfc1ee810 2024-03-28T23:29:56.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.323+0000 7f6c011e2640 1 -- 172.21.15.161:0/3704704798 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bfc1edd30 con 0x7f6bfc1ee810 2024-03-28T23:29:56.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.323+0000 7f6c011e2640 1 -- 172.21.15.161:0/3704704798 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bfc1eafd0 con 0x7f6bfc1ee810 2024-03-28T23:29:56.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.324+0000 7f6c011e2640 1 -- 172.21.15.161:0/3704704798 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bfc04f4b0 con 0x7f6bfc1ee810 2024-03-28T23:29:56.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.324+0000 7f6bf37fe640 1 -- 172.21.15.161:0/3704704798 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6be4046020 con 0x7f6bfc1ee810 2024-03-28T23:29:56.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.324+0000 7f6bf37fe640 1 --2- 172.21.15.161:0/3704704798 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6bdc06b500 0x7f6bdc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.325+0000 7f6bf37fe640 1 -- 172.21.15.161:0/3704704798 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6be40ac240 con 0x7f6bfc1ee810 2024-03-28T23:29:56.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.325+0000 7f6bfad9d640 1 --2- 172.21.15.161:0/3704704798 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6bdc06b500 0x7f6bdc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.325+0000 7f6bfad9d640 1 --2- 172.21.15.161:0/3704704798 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6bdc06b500 0x7f6bdc06d9c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6bec009de0 tx=0x7f6bec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:56.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.329+0000 7f6bf37fe640 1 -- 172.21.15.161:0/3704704798 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6be404d030 con 0x7f6bfc1ee810 2024-03-28T23:29:56.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.508+0000 7f6c011e2640 1 -- 172.21.15.161:0/3704704798 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6bfc0c8140 con 0x7f6bfc1ee810 2024-03-28T23:29:56.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.509+0000 7f6bf37fe640 1 -- 172.21.15.161:0/3704704798 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7f6be40a3e20 con 0x7f6bfc1ee810 2024-03-28T23:29:56.510 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:29:56.510 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":24,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:29:50.946843+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:29:56.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.512+0000 7f6c011e2640 1 -- 172.21.15.161:0/3704704798 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6bdc06b500 msgr2=0x7f6bdc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:56.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.512+0000 7f6c011e2640 1 --2- 172.21.15.161:0/3704704798 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6bdc06b500 0x7f6bdc06d9c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6bec009de0 tx=0x7f6bec008040 comp rx=0 tx=0).stop 2024-03-28T23:29:56.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.512+0000 7f6c011e2640 1 -- 172.21.15.161:0/3704704798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6bfc1ee810 msgr2=0x7f6bfc14ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:56.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.512+0000 7f6c011e2640 1 --2- 172.21.15.161:0/3704704798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6bfc1ee810 0x7f6bfc14ddb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6be4009ba0 tx=0x7f6be4004730 comp rx=0 tx=0).stop 2024-03-28T23:29:56.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.512+0000 7f6c011e2640 1 -- 172.21.15.161:0/3704704798 shutdown_connections 2024-03-28T23:29:56.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.512+0000 7f6c011e2640 1 --2- 172.21.15.161:0/3704704798 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6bdc06b500 0x7f6bdc06d9c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.512+0000 7f6c011e2640 1 --2- 172.21.15.161:0/3704704798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6bfc1ee810 0x7f6bfc14ddb0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.512+0000 7f6c011e2640 1 --2- 172.21.15.161:0/3704704798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6bfc14e2f0 0x7f6bfc143390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.512+0000 7f6c011e2640 1 --2- 172.21.15.161:0/3704704798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6be8004010 0x7f6bfc1444e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.512+0000 7f6c011e2640 1 -- 172.21.15.161:0/3704704798 >> 172.21.15.161:0/3704704798 conn(0x7f6bfc0ac620 msgr2=0x7f6bfc14a910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:56.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.512+0000 7f6c011e2640 1 -- 172.21.15.161:0/3704704798 shutdown_connections 2024-03-28T23:29:56.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.512+0000 7f6c011e2640 1 -- 172.21.15.161:0/3704704798 wait complete. 2024-03-28T23:29:56.524 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:29:44.021375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:29:44.872102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T23:29:48.591063+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T23:29:56.524 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:29:56.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.630+0000 7f84a8055640 1 Processor -- start 2024-03-28T23:29:56.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.630+0000 7f84a8055640 1 -- start start 2024-03-28T23:29:56.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.630+0000 7f84a8055640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f84a01529d0 0x7f84a006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.631+0000 7f84a8055640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84a0152db0 con 0x7f84a00c6750 2024-03-28T23:29:56.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.631+0000 7f84a8055640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84a00c8d40 con 0x7f84a00c88b0 2024-03-28T23:29:56.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.631+0000 7f84a8055640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f84a00c7210 con 0x7f84a01529d0 2024-03-28T23:29:56.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.631+0000 7f84a55f0640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f84a00c6750 0x7f84a0155c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52268/0 (socket says 172.21.15.161:52268) 2024-03-28T23:29:56.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.631+0000 7f84a55f0640 1 -- 172.21.15.161:0/3648252130 learned_addr learned my addr 172.21.15.161:0/3648252130 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:56.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.631+0000 7f84a5df1640 1 --2- 172.21.15.161:0/3648252130 >> v2:172.21.15.170:3300/0 conn(0x7f84a01529d0 0x7f84a006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.631+0000 7f84a4def640 1 -- 172.21.15.161:0/3648252130 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1145007008 0 0) 0x7f84a0152db0 con 0x7f84a00c6750 2024-03-28T23:29:56.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.631+0000 7f84a4def640 1 -- 172.21.15.161:0/3648252130 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f848c003520 con 0x7f84a00c6750 2024-03-28T23:29:56.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.631+0000 7f84a4def640 1 -- 172.21.15.161:0/3648252130 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2061262269 0 0) 0x7f848c003520 con 0x7f84a00c6750 2024-03-28T23:29:56.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.631+0000 7f84a4def640 1 -- 172.21.15.161:0/3648252130 >> v2:172.21.15.170:3300/0 conn(0x7f84a01529d0 msgr2=0x7f84a006bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:56.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.631+0000 7f84a4def640 1 --2- 172.21.15.161:0/3648252130 >> v2:172.21.15.170:3300/0 conn(0x7f84a01529d0 0x7f84a006bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.631+0000 7f84a4def640 1 -- 172.21.15.161:0/3648252130 >> v1:172.21.15.170:6789/0 conn(0x7f84a00c88b0 legacy=0x7f84a00c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:56.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.631+0000 7f84a4def640 1 -- 172.21.15.161:0/3648252130 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84a00c6c50 con 0x7f84a00c6750 2024-03-28T23:29:56.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.631+0000 7f84a4def640 1 -- 172.21.15.161:0/3648252130 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f8490001670 con 0x7f84a00c6750 2024-03-28T23:29:56.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.631+0000 7f84a4def640 1 -- 172.21.15.161:0/3648252130 >> v1:172.21.15.161:6789/0 conn(0x7f84a00c6750 legacy=0x7f84a0155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:56.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.631+0000 7f84a4def640 1 --2- 172.21.15.161:0/3648252130 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f848c0039b0 0x7f848c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.631+0000 7f84a4def640 1 --2- 172.21.15.161:0/3648252130 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f848c004800 0x7f848c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.631+0000 7f84a4def640 1 --2- 172.21.15.161:0/3648252130 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f848c0051c0 0x7f848c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.631+0000 7f84a4def640 1 -- 172.21.15.161:0/3648252130 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f84a00c7210 con 0x7f848c004800 2024-03-28T23:29:56.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.631+0000 7f84a4def640 1 -- 172.21.15.161:0/3648252130 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f848c00c060 con 0x7f848c0051c0 2024-03-28T23:29:56.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.631+0000 7f84a4def640 1 -- 172.21.15.161:0/3648252130 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f848c00c260 con 0x7f848c0039b0 2024-03-28T23:29:56.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.631+0000 7f84a55f0640 1 --2- 172.21.15.161:0/3648252130 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f848c004800 0x7f848c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.631+0000 7f84a5df1640 1 --2- 172.21.15.161:0/3648252130 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f848c0039b0 0x7f848c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.631+0000 7f84a65f2640 1 --2- 172.21.15.161:0/3648252130 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f848c0051c0 0x7f848c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.632+0000 7f84a55f0640 1 -- 172.21.15.161:0/3648252130 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f848c0039b0 msgr2=0x7f848c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:56.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.632+0000 7f84a55f0640 1 --2- 172.21.15.161:0/3648252130 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f848c0039b0 0x7f848c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.632+0000 7f84a55f0640 1 -- 172.21.15.161:0/3648252130 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f848c0051c0 msgr2=0x7f848c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:56.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.632+0000 7f84a55f0640 1 --2- 172.21.15.161:0/3648252130 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f848c0051c0 0x7f848c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.632+0000 7f84a55f0640 1 -- 172.21.15.161:0/3648252130 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f848c003520 con 0x7f848c004800 2024-03-28T23:29:56.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.632+0000 7f84a65f2640 1 -- 172.21.15.161:0/3648252130 reap_dead start 2024-03-28T23:29:56.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.632+0000 7f84a55f0640 1 --2- 172.21.15.161:0/3648252130 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f848c004800 0x7f848c004c80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f849000c210 tx=0x7f848c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e6dca5f7a618c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:56.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.633+0000 7f84a4def640 1 -- 172.21.15.161:0/3648252130 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8490007d30 con 0x7f848c004800 2024-03-28T23:29:56.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.633+0000 7f84a4def640 1 -- 172.21.15.161:0/3648252130 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8490030070 con 0x7f848c004800 2024-03-28T23:29:56.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.633+0000 7f84a4def640 1 -- 172.21.15.161:0/3648252130 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f849002cda0 con 0x7f848c004800 2024-03-28T23:29:56.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.633+0000 7f84a8055640 1 -- 172.21.15.161:0/3648252130 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f848c004800 msgr2=0x7f848c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:56.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.633+0000 7f84a8055640 1 --2- 172.21.15.161:0/3648252130 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f848c004800 0x7f848c004c80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f849000c210 tx=0x7f848c005830 comp rx=0 tx=0).stop 2024-03-28T23:29:56.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.633+0000 7f84a8055640 1 -- 172.21.15.161:0/3648252130 shutdown_connections 2024-03-28T23:29:56.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.633+0000 7f84a8055640 1 --2- 172.21.15.161:0/3648252130 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f848c004800 0x7f848c004c80 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.633+0000 7f84a8055640 1 -- 172.21.15.161:0/3648252130 >> 172.21.15.161:0/3648252130 conn(0x7f84a00c7430 msgr2=0x7f84a00c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:56.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.633+0000 7f84a8055640 1 -- 172.21.15.161:0/3648252130 shutdown_connections 2024-03-28T23:29:56.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.633+0000 7f84a8055640 1 -- 172.21.15.161:0/3648252130 wait complete. 2024-03-28T23:29:56.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.634+0000 7f84a8055640 1 Processor -- start 2024-03-28T23:29:56.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.634+0000 7f84a8055640 1 -- start start 2024-03-28T23:29:56.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.635+0000 7f84a8055640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f848c004800 0x7f84a00b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.635+0000 7f84a8055640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f84a00c48c0 0x7f84a0154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.635+0000 7f84a8055640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84a01546c0 0x7f84a014fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.635+0000 7f84a8055640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f84a0155710 con 0x7f84a01546c0 2024-03-28T23:29:56.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.635+0000 7f84a8055640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f84a0155430 con 0x7f848c004800 2024-03-28T23:29:56.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.635+0000 7f84a8055640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f84a00aa160 con 0x7f84a00c48c0 2024-03-28T23:29:56.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.635+0000 7f84a65f2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84a01546c0 0x7f84a014fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.635+0000 7f84a65f2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84a01546c0 0x7f84a014fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55178/0 (socket says 172.21.15.161:55178) 2024-03-28T23:29:56.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.635+0000 7f84a65f2640 1 -- 172.21.15.161:0/1773193795 learned_addr learned my addr 172.21.15.161:0/1773193795 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:56.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.635+0000 7f84a5df1640 1 --2- 172.21.15.161:0/1773193795 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f848c004800 0x7f84a00b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.635+0000 7f84a55f0640 1 --2- 172.21.15.161:0/1773193795 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f84a00c48c0 0x7f84a0154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.635+0000 7f84a65f2640 1 -- 172.21.15.161:0/1773193795 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f84a00c48c0 msgr2=0x7f84a0154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:56.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.635+0000 7f84a65f2640 1 --2- 172.21.15.161:0/1773193795 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f84a00c48c0 0x7f84a0154180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.635+0000 7f84a65f2640 1 -- 172.21.15.161:0/1773193795 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f848c004800 msgr2=0x7f84a00b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:56.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.635+0000 7f84a65f2640 1 --2- 172.21.15.161:0/1773193795 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f848c004800 0x7f84a00b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.635+0000 7f84a65f2640 1 -- 172.21.15.161:0/1773193795 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84a0159780 con 0x7f84a01546c0 2024-03-28T23:29:56.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.635+0000 7f84a55f0640 1 --2- 172.21.15.161:0/1773193795 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f84a00c48c0 0x7f84a0154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:29:56.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.635+0000 7f84a65f2640 1 --2- 172.21.15.161:0/1773193795 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84a01546c0 0x7f84a014fed0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f84a006cad0 tx=0x7f849800f890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:56.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.636+0000 7f84967fc640 1 -- 172.21.15.161:0/1773193795 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f849801a070 con 0x7f84a01546c0 2024-03-28T23:29:56.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.636+0000 7f84967fc640 1 -- 172.21.15.161:0/1773193795 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8498005320 con 0x7f84a01546c0 2024-03-28T23:29:56.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.636+0000 7f84967fc640 1 -- 172.21.15.161:0/1773193795 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8498007250 con 0x7f84a01546c0 2024-03-28T23:29:56.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.636+0000 7f84a8055640 1 -- 172.21.15.161:0/1773193795 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84a00b0810 con 0x7f84a01546c0 2024-03-28T23:29:56.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.636+0000 7f84a8055640 1 -- 172.21.15.161:0/1773193795 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84a00af2b0 con 0x7f84a01546c0 2024-03-28T23:29:56.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.637+0000 7f84a8055640 1 -- 172.21.15.161:0/1773193795 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84a004f4b0 con 0x7f84a01546c0 2024-03-28T23:29:56.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.638+0000 7f84967fc640 1 -- 172.21.15.161:0/1773193795 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f849801d070 con 0x7f84a01546c0 2024-03-28T23:29:56.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.638+0000 7f84967fc640 1 --2- 172.21.15.161:0/1773193795 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f848406b4d0 0x7f848406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.638+0000 7f84967fc640 1 -- 172.21.15.161:0/1773193795 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f84980895d0 con 0x7f84a01546c0 2024-03-28T23:29:56.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.638+0000 7f84a5df1640 1 --2- 172.21.15.161:0/1773193795 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f848406b4d0 0x7f848406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.638+0000 7f84a5df1640 1 --2- 172.21.15.161:0/1773193795 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f848406b4d0 0x7f848406d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f849c014510 tx=0x7f849c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:56.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.642+0000 7f84967fc640 1 -- 172.21.15.161:0/1773193795 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f84a00b0810 con 0x7f84a01546c0 2024-03-28T23:29:56.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.814+0000 7f84a8055640 1 -- 172.21.15.161:0/1773193795 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f84a00c8180 con 0x7f84a01546c0 2024-03-28T23:29:56.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.815+0000 7f84967fc640 1 -- 172.21.15.161:0/1773193795 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f8498057b50 con 0x7f84a01546c0 2024-03-28T23:29:56.816 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:29:56.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.817+0000 7f84a8055640 1 -- 172.21.15.161:0/1773193795 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f848406b4d0 msgr2=0x7f848406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:56.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.817+0000 7f84a8055640 1 --2- 172.21.15.161:0/1773193795 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f848406b4d0 0x7f848406d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f849c014510 tx=0x7f849c00b040 comp rx=0 tx=0).stop 2024-03-28T23:29:56.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.817+0000 7f84a8055640 1 -- 172.21.15.161:0/1773193795 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84a01546c0 msgr2=0x7f84a014fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:56.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.817+0000 7f84a8055640 1 --2- 172.21.15.161:0/1773193795 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84a01546c0 0x7f84a014fed0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f84a006cad0 tx=0x7f849800f890 comp rx=0 tx=0).stop 2024-03-28T23:29:56.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.817+0000 7f84a8055640 1 -- 172.21.15.161:0/1773193795 shutdown_connections 2024-03-28T23:29:56.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.817+0000 7f84a8055640 1 --2- 172.21.15.161:0/1773193795 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f848406b4d0 0x7f848406d990 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.817+0000 7f84a8055640 1 --2- 172.21.15.161:0/1773193795 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84a01546c0 0x7f84a014fed0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.817+0000 7f84a8055640 1 --2- 172.21.15.161:0/1773193795 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f84a00c48c0 0x7f84a0154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.817+0000 7f84a8055640 1 --2- 172.21.15.161:0/1773193795 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f848c004800 0x7f84a00b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.817+0000 7f84a8055640 1 -- 172.21.15.161:0/1773193795 >> 172.21.15.161:0/1773193795 conn(0x7f84a00c7430 msgr2=0x7f84a0150a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:56.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.817+0000 7f84a8055640 1 -- 172.21.15.161:0/1773193795 shutdown_connections 2024-03-28T23:29:56.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.817+0000 7f84a8055640 1 -- 172.21.15.161:0/1773193795 wait complete. 2024-03-28T23:29:56.828 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:29:56.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.933+0000 7f667761c640 1 Processor -- start 2024-03-28T23:29:56.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.933+0000 7f667761c640 1 -- start start 2024-03-28T23:29:56.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.934+0000 7f667761c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f66701589b0 0x7f6670158e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.934+0000 7f667761c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66700c6f60 con 0x7f6670152bc0 2024-03-28T23:29:56.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.934+0000 7f667761c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66700c6120 con 0x7f66700c0330 2024-03-28T23:29:56.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.934+0000 7f667761c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f66700c6400 con 0x7f66701589b0 2024-03-28T23:29:56.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.934+0000 7f66753b8640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6670152bc0 0x7f66700c8a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52270/0 (socket says 172.21.15.161:52270) 2024-03-28T23:29:56.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.934+0000 7f66753b8640 1 -- 172.21.15.161:0/3307132068 learned_addr learned my addr 172.21.15.161:0/3307132068 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:56.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.934+0000 7f6667fff640 1 -- 172.21.15.161:0/3307132068 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3373124429 0 0) 0x7f66700c6f60 con 0x7f6670152bc0 2024-03-28T23:29:56.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.934+0000 7f6675bb9640 1 --2- 172.21.15.161:0/3307132068 >> v2:172.21.15.170:3300/0 conn(0x7f66701589b0 0x7f6670158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.934+0000 7f6667fff640 1 -- 172.21.15.161:0/3307132068 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f665c003520 con 0x7f6670152bc0 2024-03-28T23:29:56.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.934+0000 7f6667fff640 1 -- 172.21.15.161:0/3307132068 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3812229059 0 0) 0x7f665c003520 con 0x7f6670152bc0 2024-03-28T23:29:56.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.934+0000 7f6667fff640 1 -- 172.21.15.161:0/3307132068 >> v2:172.21.15.170:3300/0 conn(0x7f66701589b0 msgr2=0x7f6670158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:56.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.934+0000 7f6667fff640 1 --2- 172.21.15.161:0/3307132068 >> v2:172.21.15.170:3300/0 conn(0x7f66701589b0 0x7f6670158e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.934+0000 7f6667fff640 1 -- 172.21.15.161:0/3307132068 >> v1:172.21.15.170:6789/0 conn(0x7f66700c0330 legacy=0x7f66700c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:56.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.934+0000 7f6667fff640 1 -- 172.21.15.161:0/3307132068 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66700c7240 con 0x7f6670152bc0 2024-03-28T23:29:56.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.934+0000 7f6667fff640 1 -- 172.21.15.161:0/3307132068 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f666c001670 con 0x7f6670152bc0 2024-03-28T23:29:56.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.934+0000 7f6667fff640 1 -- 172.21.15.161:0/3307132068 >> v1:172.21.15.161:6789/0 conn(0x7f6670152bc0 legacy=0x7f66700c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:56.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.935+0000 7f6667fff640 1 --2- 172.21.15.161:0/3307132068 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f665c003a90 0x7f665c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.935+0000 7f6667fff640 1 --2- 172.21.15.161:0/3307132068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f665c004800 0x7f665c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.935+0000 7f6667fff640 1 --2- 172.21.15.161:0/3307132068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f665c005210 0x7f665c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.935+0000 7f6667fff640 1 -- 172.21.15.161:0/3307132068 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f66700c6400 con 0x7f665c005210 2024-03-28T23:29:56.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.935+0000 7f6667fff640 1 -- 172.21.15.161:0/3307132068 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f665c00c160 con 0x7f665c004800 2024-03-28T23:29:56.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.935+0000 7f6667fff640 1 -- 172.21.15.161:0/3307132068 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f665c00c360 con 0x7f665c003a90 2024-03-28T23:29:56.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.935+0000 7f6675bb9640 1 --2- 172.21.15.161:0/3307132068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f665c005210 0x7f665c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.935+0000 7f6674bb7640 1 --2- 172.21.15.161:0/3307132068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f665c004800 0x7f665c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.935+0000 7f66753b8640 1 --2- 172.21.15.161:0/3307132068 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f665c003a90 0x7f665c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.935+0000 7f6675bb9640 1 -- 172.21.15.161:0/3307132068 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f665c003a90 msgr2=0x7f665c003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:56.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.935+0000 7f6675bb9640 1 --2- 172.21.15.161:0/3307132068 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f665c003a90 0x7f665c003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.935+0000 7f6675bb9640 1 -- 172.21.15.161:0/3307132068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f665c004800 msgr2=0x7f665c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:56.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.935+0000 7f6675bb9640 1 --2- 172.21.15.161:0/3307132068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f665c004800 0x7f665c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.935+0000 7f6675bb9640 1 -- 172.21.15.161:0/3307132068 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f665c003520 con 0x7f665c005210 2024-03-28T23:29:56.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.936+0000 7f6675bb9640 1 -- 172.21.15.161:0/3307132068 reap_dead start 2024-03-28T23:29:56.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.936+0000 7f6675bb9640 1 --2- 172.21.15.161:0/3307132068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f665c005210 0x7f665c00b9c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6668009f90 tx=0x7f665c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=139c7239b9ad0889 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:56.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.936+0000 7f6667fff640 1 -- 172.21.15.161:0/3307132068 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6668009c10 con 0x7f665c005210 2024-03-28T23:29:56.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.936+0000 7f6667fff640 1 -- 172.21.15.161:0/3307132068 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f666802cc20 con 0x7f665c005210 2024-03-28T23:29:56.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.936+0000 7f6667fff640 1 -- 172.21.15.161:0/3307132068 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6668003990 con 0x7f665c005210 2024-03-28T23:29:56.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.936+0000 7f667761c640 1 -- 172.21.15.161:0/3307132068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f665c005210 msgr2=0x7f665c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:56.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.936+0000 7f667761c640 1 --2- 172.21.15.161:0/3307132068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f665c005210 0x7f665c00b9c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6668009f90 tx=0x7f665c005830 comp rx=0 tx=0).stop 2024-03-28T23:29:56.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.937+0000 7f667761c640 1 -- 172.21.15.161:0/3307132068 shutdown_connections 2024-03-28T23:29:56.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.937+0000 7f667761c640 1 --2- 172.21.15.161:0/3307132068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f665c005210 0x7f665c00b9c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.937+0000 7f667761c640 1 -- 172.21.15.161:0/3307132068 >> 172.21.15.161:0/3307132068 conn(0x7f66700caa80 msgr2=0x7f66700cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:56.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.937+0000 7f667761c640 1 -- 172.21.15.161:0/3307132068 shutdown_connections 2024-03-28T23:29:56.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.937+0000 7f667761c640 1 -- 172.21.15.161:0/3307132068 wait complete. 2024-03-28T23:29:56.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.938+0000 7f667761c640 1 Processor -- start 2024-03-28T23:29:56.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.938+0000 7f667761c640 1 -- start start 2024-03-28T23:29:56.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.938+0000 7f667761c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f665c005210 0x7f66701520a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.938+0000 7f667761c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66701512a0 0x7f667006ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.938+0000 7f667761c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66701ed9e0 0x7f66701ede60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.938+0000 7f667761c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6670152fa0 con 0x7f66701512a0 2024-03-28T23:29:56.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.938+0000 7f667761c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6670158e00 con 0x7f66701ed9e0 2024-03-28T23:29:56.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.938+0000 7f667761c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6670151a80 con 0x7f665c005210 2024-03-28T23:29:56.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.939+0000 7f6674bb7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66701512a0 0x7f667006ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.939+0000 7f6674bb7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66701512a0 0x7f667006ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55198/0 (socket says 172.21.15.161:55198) 2024-03-28T23:29:56.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.939+0000 7f6674bb7640 1 -- 172.21.15.161:0/3207114961 learned_addr learned my addr 172.21.15.161:0/3207114961 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:56.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.939+0000 7f6675bb9640 1 --2- 172.21.15.161:0/3207114961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66701ed9e0 0x7f66701ede60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.939+0000 7f66753b8640 1 --2- 172.21.15.161:0/3207114961 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f665c005210 0x7f66701520a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.939+0000 7f6674bb7640 1 -- 172.21.15.161:0/3207114961 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f665c005210 msgr2=0x7f66701520a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:56.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.939+0000 7f6674bb7640 1 --2- 172.21.15.161:0/3207114961 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f665c005210 0x7f66701520a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.939+0000 7f6674bb7640 1 -- 172.21.15.161:0/3207114961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66701ed9e0 msgr2=0x7f66701ede60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:56.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.939+0000 7f6674bb7640 1 --2- 172.21.15.161:0/3207114961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66701ed9e0 0x7f66701ede60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:56.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.939+0000 7f6674bb7640 1 -- 172.21.15.161:0/3207114961 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66701ea620 con 0x7f66701512a0 2024-03-28T23:29:56.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.939+0000 7f66753b8640 1 --2- 172.21.15.161:0/3207114961 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f665c005210 0x7f66701520a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:29:56.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.939+0000 7f6675bb9640 1 --2- 172.21.15.161:0/3207114961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66701ed9e0 0x7f66701ede60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:29:56.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.939+0000 7f6674bb7640 1 --2- 172.21.15.161:0/3207114961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66701512a0 0x7f667006ca30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f666000f870 tx=0x7f666000fd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:56.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.939+0000 7f6665ffb640 1 -- 172.21.15.161:0/3207114961 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666001a070 con 0x7f66701512a0 2024-03-28T23:29:56.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.939+0000 7f6665ffb640 1 -- 172.21.15.161:0/3207114961 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6660005320 con 0x7f66701512a0 2024-03-28T23:29:56.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.939+0000 7f667761c640 1 -- 172.21.15.161:0/3207114961 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66701ea900 con 0x7f66701512a0 2024-03-28T23:29:56.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.939+0000 7f6665ffb640 1 -- 172.21.15.161:0/3207114961 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6660007290 con 0x7f66701512a0 2024-03-28T23:29:56.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.939+0000 7f667761c640 1 -- 172.21.15.161:0/3207114961 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f667006cf70 con 0x7f66701512a0 2024-03-28T23:29:56.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.940+0000 7f667761c640 1 -- 172.21.15.161:0/3207114961 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66701ea410 con 0x7f66701512a0 2024-03-28T23:29:56.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.941+0000 7f6665ffb640 1 -- 172.21.15.161:0/3207114961 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f666001d070 con 0x7f66701512a0 2024-03-28T23:29:56.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.941+0000 7f6665ffb640 1 --2- 172.21.15.161:0/3207114961 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f665406b500 0x7f665406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:56.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.941+0000 7f6665ffb640 1 -- 172.21.15.161:0/3207114961 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6660089ca0 con 0x7f66701512a0 2024-03-28T23:29:56.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.941+0000 7f66753b8640 1 --2- 172.21.15.161:0/3207114961 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f665406b500 0x7f665406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:56.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.942+0000 7f66753b8640 1 --2- 172.21.15.161:0/3207114961 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f665406b500 0x7f665406d9c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f666c007b80 tx=0x7f666c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:56.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:56.945+0000 7f6665ffb640 1 -- 172.21.15.161:0/3207114961 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f66600587e0 con 0x7f66701512a0 2024-03-28T23:29:57.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.120+0000 7f667761c640 1 -- 172.21.15.161:0/3207114961 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f667004f4b0 con 0x7f66701512a0 2024-03-28T23:29:57.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.120+0000 7f6665ffb640 1 -- 172.21.15.161:0/3207114961 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f66600580e0 con 0x7f66701512a0 2024-03-28T23:29:57.121 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:29:57.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.122+0000 7f667761c640 1 -- 172.21.15.161:0/3207114961 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f665406b500 msgr2=0x7f665406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:57.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.122+0000 7f667761c640 1 --2- 172.21.15.161:0/3207114961 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f665406b500 0x7f665406d9c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f666c007b80 tx=0x7f666c009040 comp rx=0 tx=0).stop 2024-03-28T23:29:57.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.122+0000 7f667761c640 1 -- 172.21.15.161:0/3207114961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66701512a0 msgr2=0x7f667006ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:57.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.122+0000 7f667761c640 1 --2- 172.21.15.161:0/3207114961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66701512a0 0x7f667006ca30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f666000f870 tx=0x7f666000fd30 comp rx=0 tx=0).stop 2024-03-28T23:29:57.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.122+0000 7f667761c640 1 -- 172.21.15.161:0/3207114961 shutdown_connections 2024-03-28T23:29:57.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.122+0000 7f667761c640 1 --2- 172.21.15.161:0/3207114961 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f665406b500 0x7f665406d9c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.122+0000 7f667761c640 1 --2- 172.21.15.161:0/3207114961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66701512a0 0x7f667006ca30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.122+0000 7f667761c640 1 --2- 172.21.15.161:0/3207114961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66701ed9e0 0x7f66701ede60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.122+0000 7f667761c640 1 --2- 172.21.15.161:0/3207114961 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f665c005210 0x7f66701520a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.122+0000 7f667761c640 1 -- 172.21.15.161:0/3207114961 >> 172.21.15.161:0/3207114961 conn(0x7f66700caa80 msgr2=0x7f66700cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:57.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.122+0000 7f667761c640 1 -- 172.21.15.161:0/3207114961 shutdown_connections 2024-03-28T23:29:57.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.122+0000 7f667761c640 1 -- 172.21.15.161:0/3207114961 wait complete. 2024-03-28T23:29:57.134 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:29:57.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.238+0000 7f9eb9d8a640 1 Processor -- start 2024-03-28T23:29:57.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.238+0000 7f9eb9d8a640 1 -- start start 2024-03-28T23:29:57.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.239+0000 7f9eb9d8a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9eb406bb60 0x7f9eb40c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:57.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.239+0000 7f9eb9d8a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9eb4152db0 con 0x7f9eb40c6750 2024-03-28T23:29:57.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.239+0000 7f9eb9d8a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9eb40c7210 con 0x7f9eb41529d0 2024-03-28T23:29:57.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.239+0000 7f9eb9d8a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f9eb40c6c50 con 0x7f9eb406bb60 2024-03-28T23:29:57.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.239+0000 7f9eabfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f9eb40c6750 0x7f9eb40c54f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52278/0 (socket says 172.21.15.161:52278) 2024-03-28T23:29:57.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.239+0000 7f9eabfff640 1 -- 172.21.15.161:0/2317250841 learned_addr learned my addr 172.21.15.161:0/2317250841 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:57.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.239+0000 7f9eb2ffd640 1 -- 172.21.15.161:0/2317250841 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 245353297 0 0) 0x7f9eb4152db0 con 0x7f9eb40c6750 2024-03-28T23:29:57.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.239+0000 7f9eb3fff640 1 --2- 172.21.15.161:0/2317250841 >> v2:172.21.15.170:3300/0 conn(0x7f9eb406bb60 0x7f9eb40c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:57.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.240+0000 7f9eb2ffd640 1 -- 172.21.15.161:0/2317250841 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ea0003520 con 0x7f9eb40c6750 2024-03-28T23:29:57.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.240+0000 7f9eb2ffd640 1 -- 172.21.15.161:0/2317250841 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 812250285 0 0) 0x7f9ea0003520 con 0x7f9eb40c6750 2024-03-28T23:29:57.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.240+0000 7f9eb2ffd640 1 -- 172.21.15.161:0/2317250841 >> v2:172.21.15.170:3300/0 conn(0x7f9eb406bb60 msgr2=0x7f9eb40c88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:57.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.240+0000 7f9eb2ffd640 1 --2- 172.21.15.161:0/2317250841 >> v2:172.21.15.170:3300/0 conn(0x7f9eb406bb60 0x7f9eb40c88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.240+0000 7f9eb2ffd640 1 -- 172.21.15.161:0/2317250841 >> v1:172.21.15.170:6789/0 conn(0x7f9eb41529d0 legacy=0x7f9eb4155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:57.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.240+0000 7f9eb2ffd640 1 -- 172.21.15.161:0/2317250841 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eb40c6f30 con 0x7f9eb40c6750 2024-03-28T23:29:57.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.240+0000 7f9eb2ffd640 1 -- 172.21.15.161:0/2317250841 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f9e9c001670 con 0x7f9eb40c6750 2024-03-28T23:29:57.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.240+0000 7f9eb2ffd640 1 -- 172.21.15.161:0/2317250841 >> v1:172.21.15.161:6789/0 conn(0x7f9eb40c6750 legacy=0x7f9eb40c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:57.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.240+0000 7f9eb2ffd640 1 --2- 172.21.15.161:0/2317250841 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ea00039b0 0x7f9ea0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:57.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.240+0000 7f9eb2ffd640 1 --2- 172.21.15.161:0/2317250841 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ea0004800 0x7f9ea0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:57.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.240+0000 7f9eb2ffd640 1 --2- 172.21.15.161:0/2317250841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ea00051c0 0x7f9ea000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:57.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.240+0000 7f9eb2ffd640 1 -- 172.21.15.161:0/2317250841 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb40c6c50 con 0x7f9ea00051c0 2024-03-28T23:29:57.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.240+0000 7f9eb2ffd640 1 -- 172.21.15.161:0/2317250841 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea000c060 con 0x7f9ea0004800 2024-03-28T23:29:57.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.240+0000 7f9eb2ffd640 1 -- 172.21.15.161:0/2317250841 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9ea000c260 con 0x7f9ea00039b0 2024-03-28T23:29:57.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.240+0000 7f9eb3fff640 1 --2- 172.21.15.161:0/2317250841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ea00051c0 0x7f9ea000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:57.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.240+0000 7f9eb37fe640 1 --2- 172.21.15.161:0/2317250841 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ea00039b0 0x7f9ea0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:57.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.240+0000 7f9eabfff640 1 --2- 172.21.15.161:0/2317250841 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ea0004800 0x7f9ea0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:57.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.240+0000 7f9eb3fff640 1 -- 172.21.15.161:0/2317250841 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ea00039b0 msgr2=0x7f9ea0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:57.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.240+0000 7f9eb3fff640 1 --2- 172.21.15.161:0/2317250841 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ea00039b0 0x7f9ea0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.240+0000 7f9eb3fff640 1 -- 172.21.15.161:0/2317250841 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ea0004800 msgr2=0x7f9ea0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:57.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.240+0000 7f9eb3fff640 1 --2- 172.21.15.161:0/2317250841 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ea0004800 0x7f9ea0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.240+0000 7f9eb3fff640 1 -- 172.21.15.161:0/2317250841 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ea0003520 con 0x7f9ea00051c0 2024-03-28T23:29:57.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.241+0000 7f9eb3fff640 1 -- 172.21.15.161:0/2317250841 reap_dead start 2024-03-28T23:29:57.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.241+0000 7f9eb3fff640 1 --2- 172.21.15.161:0/2317250841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ea00051c0 0x7f9ea000b9c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9eac009fc0 tx=0x7f9ea0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f27a5604b21564 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:57.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.241+0000 7f9eb2ffd640 1 -- 172.21.15.161:0/2317250841 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac009c10 con 0x7f9ea00051c0 2024-03-28T23:29:57.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.241+0000 7f9eb2ffd640 1 -- 172.21.15.161:0/2317250841 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9eac02cca0 con 0x7f9ea00051c0 2024-03-28T23:29:57.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.241+0000 7f9eb2ffd640 1 -- 172.21.15.161:0/2317250841 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac003bd0 con 0x7f9ea00051c0 2024-03-28T23:29:57.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.242+0000 7f9eb9d8a640 1 -- 172.21.15.161:0/2317250841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ea00051c0 msgr2=0x7f9ea000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:57.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.242+0000 7f9eb9d8a640 1 --2- 172.21.15.161:0/2317250841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ea00051c0 0x7f9ea000b9c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9eac009fc0 tx=0x7f9ea0005830 comp rx=0 tx=0).stop 2024-03-28T23:29:57.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.242+0000 7f9eb9d8a640 1 -- 172.21.15.161:0/2317250841 shutdown_connections 2024-03-28T23:29:57.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.242+0000 7f9eb9d8a640 1 --2- 172.21.15.161:0/2317250841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ea00051c0 0x7f9ea000b9c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.242+0000 7f9eb9d8a640 1 -- 172.21.15.161:0/2317250841 >> 172.21.15.161:0/2317250841 conn(0x7f9eb40c7430 msgr2=0x7f9eb40c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:57.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.242+0000 7f9eb9d8a640 1 -- 172.21.15.161:0/2317250841 shutdown_connections 2024-03-28T23:29:57.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.242+0000 7f9eb9d8a640 1 -- 172.21.15.161:0/2317250841 wait complete. 2024-03-28T23:29:57.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.243+0000 7f9eb9d8a640 1 Processor -- start 2024-03-28T23:29:57.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.243+0000 7f9eb9d8a640 1 -- start start 2024-03-28T23:29:57.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.244+0000 7f9eb9d8a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ea00051c0 0x7f9eb41eb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:57.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.244+0000 7f9eb9d8a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9eb40af4d0 0x7f9eb414aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:57.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.244+0000 7f9eb9d8a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9eb414b7a0 0x7f9eb40adff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:57.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.244+0000 7f9eb9d8a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb40aa160 con 0x7f9ea00051c0 2024-03-28T23:29:57.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.244+0000 7f9eb9d8a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb4155710 con 0x7f9eb414b7a0 2024-03-28T23:29:57.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.244+0000 7f9eb9d8a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9eb40c88b0 con 0x7f9eb40af4d0 2024-03-28T23:29:57.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.244+0000 7f9eb37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ea00051c0 0x7f9eb41eb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:57.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.244+0000 7f9eb37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ea00051c0 0x7f9eb41eb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55226/0 (socket says 172.21.15.161:55226) 2024-03-28T23:29:57.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.244+0000 7f9eb37fe640 1 -- 172.21.15.161:0/3789156502 learned_addr learned my addr 172.21.15.161:0/3789156502 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:57.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.244+0000 7f9eabfff640 1 --2- 172.21.15.161:0/3789156502 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9eb40af4d0 0x7f9eb414aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:57.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.244+0000 7f9eb3fff640 1 --2- 172.21.15.161:0/3789156502 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9eb414b7a0 0x7f9eb40adff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:57.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.244+0000 7f9eb37fe640 1 -- 172.21.15.161:0/3789156502 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9eb40af4d0 msgr2=0x7f9eb414aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:57.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.244+0000 7f9eb37fe640 1 --2- 172.21.15.161:0/3789156502 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9eb40af4d0 0x7f9eb414aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.244+0000 7f9eb37fe640 1 -- 172.21.15.161:0/3789156502 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9eb414b7a0 msgr2=0x7f9eb40adff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:57.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.244+0000 7f9eb37fe640 1 --2- 172.21.15.161:0/3789156502 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9eb414b7a0 0x7f9eb40adff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.244+0000 7f9eb37fe640 1 -- 172.21.15.161:0/3789156502 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eb41ebac0 con 0x7f9ea00051c0 2024-03-28T23:29:57.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.244+0000 7f9eabfff640 1 --2- 172.21.15.161:0/3789156502 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9eb40af4d0 0x7f9eb414aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:29:57.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.244+0000 7f9eb3fff640 1 --2- 172.21.15.161:0/3789156502 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9eb414b7a0 0x7f9eb40adff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:29:57.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.244+0000 7f9eb37fe640 1 --2- 172.21.15.161:0/3789156502 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ea00051c0 0x7f9eb41eb580 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9ea400f650 tx=0x7f9ea400fb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:57.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.244+0000 7f9eb0ff9640 1 -- 172.21.15.161:0/3789156502 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea40050d0 con 0x7f9ea00051c0 2024-03-28T23:29:57.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.244+0000 7f9eb0ff9640 1 -- 172.21.15.161:0/3789156502 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ea4018070 con 0x7f9ea00051c0 2024-03-28T23:29:57.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.244+0000 7f9eb9d8a640 1 -- 172.21.15.161:0/3789156502 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eb414afc0 con 0x7f9ea00051c0 2024-03-28T23:29:57.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.244+0000 7f9eb0ff9640 1 -- 172.21.15.161:0/3789156502 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4005490 con 0x7f9ea00051c0 2024-03-28T23:29:57.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.244+0000 7f9eb9d8a640 1 -- 172.21.15.161:0/3789156502 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eb4154810 con 0x7f9ea00051c0 2024-03-28T23:29:57.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.245+0000 7f9eb9d8a640 1 -- 172.21.15.161:0/3789156502 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9eb404f4b0 con 0x7f9ea00051c0 2024-03-28T23:29:57.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.246+0000 7f9eb0ff9640 1 -- 172.21.15.161:0/3789156502 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9ea400d730 con 0x7f9ea00051c0 2024-03-28T23:29:57.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.246+0000 7f9eb0ff9640 1 --2- 172.21.15.161:0/3789156502 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9e9406b500 0x7f9e9406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:57.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.246+0000 7f9eabfff640 1 --2- 172.21.15.161:0/3789156502 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9e9406b500 0x7f9e9406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:57.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.246+0000 7f9eb0ff9640 1 -- 172.21.15.161:0/3789156502 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9ea408a2a0 con 0x7f9ea00051c0 2024-03-28T23:29:57.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.247+0000 7f9eabfff640 1 --2- 172.21.15.161:0/3789156502 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9e9406b500 0x7f9e9406d9c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9eb40c4b00 tx=0x7f9e9c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:57.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.251+0000 7f9eb0ff9640 1 -- 172.21.15.161:0/3789156502 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9ea4058de0 con 0x7f9ea00051c0 2024-03-28T23:29:57.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.428+0000 7f9eb9d8a640 1 -- 172.21.15.161:0/3789156502 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9eb40c8180 con 0x7f9ea00051c0 2024-03-28T23:29:57.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.429+0000 7f9eb0ff9640 1 -- 172.21.15.161:0/3789156502 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f9ea40586e0 con 0x7f9ea00051c0 2024-03-28T23:29:57.429 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:29:57.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.431+0000 7f9eb9d8a640 1 -- 172.21.15.161:0/3789156502 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9e9406b500 msgr2=0x7f9e9406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:57.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.431+0000 7f9eb9d8a640 1 --2- 172.21.15.161:0/3789156502 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9e9406b500 0x7f9e9406d9c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9eb40c4b00 tx=0x7f9e9c009040 comp rx=0 tx=0).stop 2024-03-28T23:29:57.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.431+0000 7f9eb9d8a640 1 -- 172.21.15.161:0/3789156502 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ea00051c0 msgr2=0x7f9eb41eb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:57.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.431+0000 7f9eb9d8a640 1 --2- 172.21.15.161:0/3789156502 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ea00051c0 0x7f9eb41eb580 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9ea400f650 tx=0x7f9ea400fb10 comp rx=0 tx=0).stop 2024-03-28T23:29:57.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.431+0000 7f9eb9d8a640 1 -- 172.21.15.161:0/3789156502 shutdown_connections 2024-03-28T23:29:57.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.431+0000 7f9eb9d8a640 1 --2- 172.21.15.161:0/3789156502 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9e9406b500 0x7f9e9406d9c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.431+0000 7f9eb9d8a640 1 --2- 172.21.15.161:0/3789156502 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9eb414b7a0 0x7f9eb40adff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.431+0000 7f9eb9d8a640 1 --2- 172.21.15.161:0/3789156502 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9eb40af4d0 0x7f9eb414aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.431+0000 7f9eb9d8a640 1 --2- 172.21.15.161:0/3789156502 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ea00051c0 0x7f9eb41eb580 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.431+0000 7f9eb9d8a640 1 -- 172.21.15.161:0/3789156502 >> 172.21.15.161:0/3789156502 conn(0x7f9eb40c7430 msgr2=0x7f9eb4074020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:57.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.431+0000 7f9eb9d8a640 1 -- 172.21.15.161:0/3789156502 shutdown_connections 2024-03-28T23:29:57.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.431+0000 7f9eb9d8a640 1 -- 172.21.15.161:0/3789156502 wait complete. 2024-03-28T23:29:57.442 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T23:29:57.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.546+0000 7f8436ed4640 1 Processor -- start 2024-03-28T23:29:57.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.546+0000 7f8436ed4640 1 -- start start 2024-03-28T23:29:57.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.547+0000 7f8436ed4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f84300c0330 0x7f84300c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:57.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.547+0000 7f8436ed4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84300c6f60 con 0x7f8430152bc0 2024-03-28T23:29:57.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.547+0000 7f8436ed4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84300c6120 con 0x7f84301589b0 2024-03-28T23:29:57.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.547+0000 7f8436ed4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f84300c6400 con 0x7f84300c0330 2024-03-28T23:29:57.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.547+0000 7f842ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f84300c0330 0x7f84300c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:57.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.547+0000 7f8434c70640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f8430152bc0 0x7f84300c8a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52294/0 (socket says 172.21.15.161:52294) 2024-03-28T23:29:57.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.547+0000 7f8434c70640 1 -- 172.21.15.161:0/3059778862 learned_addr learned my addr 172.21.15.161:0/3059778862 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:57.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.548+0000 7f842ffff640 1 -- 172.21.15.161:0/3059778862 >> v1:172.21.15.170:6789/0 conn(0x7f84301589b0 legacy=0x7f8430158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:57.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.548+0000 7f842ffff640 1 -- 172.21.15.161:0/3059778862 >> v1:172.21.15.161:6789/0 conn(0x7f8430152bc0 legacy=0x7f84300c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:57.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.548+0000 7f842ffff640 1 -- 172.21.15.161:0/3059778862 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84300c96d0 con 0x7f84300c0330 2024-03-28T23:29:57.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.548+0000 7f842ffff640 1 --2- 172.21.15.161:0/3059778862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84300c0330 0x7f84300c9190 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f84240097a0 tx=0x7f842402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7607a37e06e07dbc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:57.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.548+0000 7f842f7fe640 1 -- 172.21.15.161:0/3059778862 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8424035020 con 0x7f84300c0330 2024-03-28T23:29:57.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.548+0000 7f842f7fe640 1 -- 172.21.15.161:0/3059778862 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8424031830 con 0x7f84300c0330 2024-03-28T23:29:57.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.549+0000 7f842f7fe640 1 -- 172.21.15.161:0/3059778862 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8424031c00 con 0x7f84300c0330 2024-03-28T23:29:57.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.549+0000 7f8436ed4640 1 -- 172.21.15.161:0/3059778862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84300c0330 msgr2=0x7f84300c9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:57.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.549+0000 7f8436ed4640 1 --2- 172.21.15.161:0/3059778862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84300c0330 0x7f84300c9190 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f84240097a0 tx=0x7f842402b8d0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.549+0000 7f8436ed4640 1 -- 172.21.15.161:0/3059778862 shutdown_connections 2024-03-28T23:29:57.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.549+0000 7f8436ed4640 1 --2- 172.21.15.161:0/3059778862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84300c0330 0x7f84300c9190 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.549+0000 7f8436ed4640 1 -- 172.21.15.161:0/3059778862 >> 172.21.15.161:0/3059778862 conn(0x7f84300caa80 msgr2=0x7f84300cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:57.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.549+0000 7f8436ed4640 1 -- 172.21.15.161:0/3059778862 shutdown_connections 2024-03-28T23:29:57.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.549+0000 7f8436ed4640 1 -- 172.21.15.161:0/3059778862 wait complete. 2024-03-28T23:29:57.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.550+0000 7f8436ed4640 1 Processor -- start 2024-03-28T23:29:57.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.550+0000 7f8436ed4640 1 -- start start 2024-03-28T23:29:57.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.551+0000 7f8436ed4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84300c0330 0x7f843006b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:57.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.551+0000 7f8436ed4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8430152bc0 0x7f8430157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:57.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.551+0000 7f8436ed4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f84301589b0 0x7f84301ee590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:57.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.551+0000 7f8436ed4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8430152fa0 con 0x7f84300c0330 2024-03-28T23:29:57.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.551+0000 7f8436ed4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f84300c6bf0 con 0x7f84301589b0 2024-03-28T23:29:57.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.551+0000 7f8436ed4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f843014e200 con 0x7f8430152bc0 2024-03-28T23:29:57.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.551+0000 7f8434c70640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84300c0330 0x7f843006b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:57.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.551+0000 7f8434c70640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84300c0330 0x7f843006b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55234/0 (socket says 172.21.15.161:55234) 2024-03-28T23:29:57.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.551+0000 7f8434c70640 1 -- 172.21.15.161:0/619737153 learned_addr learned my addr 172.21.15.161:0/619737153 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:57.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.551+0000 7f842ffff640 1 --2- 172.21.15.161:0/619737153 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8430152bc0 0x7f8430157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:57.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.551+0000 7f8435471640 1 --2- 172.21.15.161:0/619737153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f84301589b0 0x7f84301ee590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:57.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.551+0000 7f8434c70640 1 -- 172.21.15.161:0/619737153 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8430152bc0 msgr2=0x7f8430157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:57.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.551+0000 7f8434c70640 1 --2- 172.21.15.161:0/619737153 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8430152bc0 0x7f8430157ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.551+0000 7f8434c70640 1 -- 172.21.15.161:0/619737153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f84301589b0 msgr2=0x7f84301ee590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:57.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.551+0000 7f8434c70640 1 --2- 172.21.15.161:0/619737153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f84301589b0 0x7f84301ee590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.551+0000 7f8434c70640 1 -- 172.21.15.161:0/619737153 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84301eddd0 con 0x7f84300c0330 2024-03-28T23:29:57.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.552+0000 7f8434c70640 1 --2- 172.21.15.161:0/619737153 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84300c0330 0x7f843006b720 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f842000c400 tx=0x7f842000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:57.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.552+0000 7f842d7fa640 1 -- 172.21.15.161:0/619737153 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8420017070 con 0x7f84300c0330 2024-03-28T23:29:57.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.552+0000 7f842d7fa640 1 -- 172.21.15.161:0/619737153 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8420005ce0 con 0x7f84300c0330 2024-03-28T23:29:57.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.552+0000 7f8436ed4640 1 -- 172.21.15.161:0/619737153 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8430150f90 con 0x7f84300c0330 2024-03-28T23:29:57.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.552+0000 7f842d7fa640 1 -- 172.21.15.161:0/619737153 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84200072f0 con 0x7f84300c0330 2024-03-28T23:29:57.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.552+0000 7f8436ed4640 1 -- 172.21.15.161:0/619737153 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8430151270 con 0x7f84300c0330 2024-03-28T23:29:57.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.553+0000 7f8436ed4640 1 -- 172.21.15.161:0/619737153 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f843004f4b0 con 0x7f84300c0330 2024-03-28T23:29:57.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.553+0000 7f842d7fa640 1 -- 172.21.15.161:0/619737153 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f842001a070 con 0x7f84300c0330 2024-03-28T23:29:57.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.554+0000 7f842d7fa640 1 --2- 172.21.15.161:0/619737153 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f841c06b4d0 0x7f841c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:57.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.554+0000 7f842d7fa640 1 -- 172.21.15.161:0/619737153 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f8420089930 con 0x7f84300c0330 2024-03-28T23:29:57.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.554+0000 7f842ffff640 1 --2- 172.21.15.161:0/619737153 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f841c06b4d0 0x7f841c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:57.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.554+0000 7f842ffff640 1 --2- 172.21.15.161:0/619737153 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f841c06b4d0 0x7f841c06d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8424036000 tx=0x7f8424008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:57.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.558+0000 7f842d7fa640 1 -- 172.21.15.161:0/619737153 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8420058470 con 0x7f84300c0330 2024-03-28T23:29:57.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.737+0000 7f8436ed4640 1 -- 172.21.15.161:0/619737153 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f84300ab900 con 0x7f84300c0330 2024-03-28T23:29:57.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.737+0000 7f842d7fa640 1 -- 172.21.15.161:0/619737153 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f8420057d70 con 0x7f84300c0330 2024-03-28T23:29:57.738 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:29:57.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.739+0000 7f8436ed4640 1 -- 172.21.15.161:0/619737153 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f841c06b4d0 msgr2=0x7f841c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:57.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.739+0000 7f8436ed4640 1 --2- 172.21.15.161:0/619737153 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f841c06b4d0 0x7f841c06d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8424036000 tx=0x7f8424008030 comp rx=0 tx=0).stop 2024-03-28T23:29:57.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.739+0000 7f8436ed4640 1 -- 172.21.15.161:0/619737153 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84300c0330 msgr2=0x7f843006b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:57.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.739+0000 7f8436ed4640 1 --2- 172.21.15.161:0/619737153 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84300c0330 0x7f843006b720 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f842000c400 tx=0x7f842000c8c0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.739+0000 7f8436ed4640 1 -- 172.21.15.161:0/619737153 shutdown_connections 2024-03-28T23:29:57.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.739+0000 7f8436ed4640 1 --2- 172.21.15.161:0/619737153 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f841c06b4d0 0x7f841c06d990 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.739+0000 7f8436ed4640 1 --2- 172.21.15.161:0/619737153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f84301589b0 0x7f84301ee590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.739+0000 7f8436ed4640 1 --2- 172.21.15.161:0/619737153 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8430152bc0 0x7f8430157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.739+0000 7f8436ed4640 1 --2- 172.21.15.161:0/619737153 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84300c0330 0x7f843006b720 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.739+0000 7f8436ed4640 1 -- 172.21.15.161:0/619737153 >> 172.21.15.161:0/619737153 conn(0x7f84300caa80 msgr2=0x7f84300cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:57.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.739+0000 7f8436ed4640 1 -- 172.21.15.161:0/619737153 shutdown_connections 2024-03-28T23:29:57.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.739+0000 7f8436ed4640 1 -- 172.21.15.161:0/619737153 wait complete. 2024-03-28T23:29:57.755 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T23:29:57.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.857+0000 7f4ac6cdf640 1 Processor -- start 2024-03-28T23:29:57.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.857+0000 7f4ac6cdf640 1 -- start start 2024-03-28T23:29:57.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.858+0000 7f4ac6cdf640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4ac00c3390 0x7f4ac00c37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:57.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.858+0000 7f4ac6cdf640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ac0155480 con 0x7f4ac006dd60 2024-03-28T23:29:57.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.858+0000 7f4ac6cdf640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ac0155760 con 0x7f4ac01589d0 2024-03-28T23:29:57.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.858+0000 7f4ac6cdf640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f4ac0155a40 con 0x7f4ac00c3390 2024-03-28T23:29:57.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.858+0000 7f4ac4a7b640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f4ac006dd60 0x7f4ac00a7d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52308/0 (socket says 172.21.15.161:52308) 2024-03-28T23:29:57.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.858+0000 7f4ac4a7b640 1 -- 172.21.15.161:0/35097993 learned_addr learned my addr 172.21.15.161:0/35097993 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:57.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.858+0000 7f4ab7fff640 1 --2- 172.21.15.161:0/35097993 >> v2:172.21.15.170:3300/0 conn(0x7f4ac00c3390 0x7f4ac00c37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:57.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.858+0000 7f4ab77fe640 1 -- 172.21.15.161:0/35097993 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3824072571 0 0) 0x7f4ac0155480 con 0x7f4ac006dd60 2024-03-28T23:29:57.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.859+0000 7f4ab77fe640 1 -- 172.21.15.161:0/35097993 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4aa8003520 con 0x7f4ac006dd60 2024-03-28T23:29:57.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.859+0000 7f4ab77fe640 1 -- 172.21.15.161:0/35097993 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4228576396 0 0) 0x7f4aa8003520 con 0x7f4ac006dd60 2024-03-28T23:29:57.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.859+0000 7f4ab77fe640 1 -- 172.21.15.161:0/35097993 >> v2:172.21.15.170:3300/0 conn(0x7f4ac00c3390 msgr2=0x7f4ac00c37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:57.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.859+0000 7f4ab77fe640 1 --2- 172.21.15.161:0/35097993 >> v2:172.21.15.170:3300/0 conn(0x7f4ac00c3390 0x7f4ac00c37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.859+0000 7f4ab77fe640 1 -- 172.21.15.161:0/35097993 >> v1:172.21.15.170:6789/0 conn(0x7f4ac01589d0 legacy=0x7f4ac0158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:57.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.859+0000 7f4ab77fe640 1 -- 172.21.15.161:0/35097993 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ac0155d20 con 0x7f4ac006dd60 2024-03-28T23:29:57.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.859+0000 7f4ab77fe640 1 -- 172.21.15.161:0/35097993 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f4abc001670 con 0x7f4ac006dd60 2024-03-28T23:29:57.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.859+0000 7f4ab77fe640 1 -- 172.21.15.161:0/35097993 >> v1:172.21.15.161:6789/0 conn(0x7f4ac006dd60 legacy=0x7f4ac00a7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:57.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.859+0000 7f4ab77fe640 1 --2- 172.21.15.161:0/35097993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4aa8003a90 0x7f4aa8003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:57.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.859+0000 7f4ab77fe640 1 --2- 172.21.15.161:0/35097993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4aa8004800 0x7f4aa8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:57.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.859+0000 7f4ab77fe640 1 --2- 172.21.15.161:0/35097993 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4aa80053c0 0x7f4aa800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:57.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.859+0000 7f4ab77fe640 1 -- 172.21.15.161:0/35097993 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac0155a40 con 0x7f4aa80053c0 2024-03-28T23:29:57.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.859+0000 7f4ab77fe640 1 -- 172.21.15.161:0/35097993 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa800c280 con 0x7f4aa8003a90 2024-03-28T23:29:57.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.859+0000 7f4ab77fe640 1 -- 172.21.15.161:0/35097993 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4aa800c480 con 0x7f4aa8004800 2024-03-28T23:29:57.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.859+0000 7f4ac527c640 1 --2- 172.21.15.161:0/35097993 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4aa80053c0 0x7f4aa800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:57.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.859+0000 7f4ac4a7b640 1 --2- 172.21.15.161:0/35097993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4aa8003a90 0x7f4aa8003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:57.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.859+0000 7f4ab7fff640 1 --2- 172.21.15.161:0/35097993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4aa8004800 0x7f4aa8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:57.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.859+0000 7f4ac527c640 1 -- 172.21.15.161:0/35097993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4aa8004800 msgr2=0x7f4aa8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:57.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.859+0000 7f4ac527c640 1 --2- 172.21.15.161:0/35097993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4aa8004800 0x7f4aa8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.859+0000 7f4ac527c640 1 -- 172.21.15.161:0/35097993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4aa8003a90 msgr2=0x7f4aa8003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:57.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.859+0000 7f4ac527c640 1 --2- 172.21.15.161:0/35097993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4aa8003a90 0x7f4aa8003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.859+0000 7f4ac527c640 1 -- 172.21.15.161:0/35097993 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa8003520 con 0x7f4aa80053c0 2024-03-28T23:29:57.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.860+0000 7f4ac527c640 1 -- 172.21.15.161:0/35097993 reap_dead start 2024-03-28T23:29:57.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.860+0000 7f4ac527c640 1 --2- 172.21.15.161:0/35097993 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4aa80053c0 0x7f4aa800b9c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8001e50 tx=0x7f4aa8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d71615727965a0f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:57.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.860+0000 7f4ab77fe640 1 -- 172.21.15.161:0/35097993 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab8001a70 con 0x7f4aa80053c0 2024-03-28T23:29:57.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.860+0000 7f4ab77fe640 1 -- 172.21.15.161:0/35097993 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ab8003b90 con 0x7f4aa80053c0 2024-03-28T23:29:57.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.860+0000 7f4ab77fe640 1 -- 172.21.15.161:0/35097993 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab80050d0 con 0x7f4aa80053c0 2024-03-28T23:29:57.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.861+0000 7f4ac6cdf640 1 -- 172.21.15.161:0/35097993 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4aa80053c0 msgr2=0x7f4aa800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:57.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.861+0000 7f4ac6cdf640 1 --2- 172.21.15.161:0/35097993 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4aa80053c0 0x7f4aa800b9c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8001e50 tx=0x7f4aa8005830 comp rx=0 tx=0).stop 2024-03-28T23:29:57.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.861+0000 7f4ac6cdf640 1 -- 172.21.15.161:0/35097993 shutdown_connections 2024-03-28T23:29:57.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.861+0000 7f4ac6cdf640 1 --2- 172.21.15.161:0/35097993 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4aa80053c0 0x7f4aa800b9c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.861+0000 7f4ac6cdf640 1 -- 172.21.15.161:0/35097993 >> 172.21.15.161:0/35097993 conn(0x7f4ac00aa740 msgr2=0x7f4ac00aab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:57.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.861+0000 7f4ac6cdf640 1 -- 172.21.15.161:0/35097993 shutdown_connections 2024-03-28T23:29:57.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.861+0000 7f4ac6cdf640 1 -- 172.21.15.161:0/35097993 wait complete. 2024-03-28T23:29:57.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.862+0000 7f4ac6cdf640 1 Processor -- start 2024-03-28T23:29:57.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.862+0000 7f4ac6cdf640 1 -- start start 2024-03-28T23:29:57.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.863+0000 7f4ac6cdf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4aa80053c0 0x7f4ac0154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:57.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.863+0000 7f4ac6cdf640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ac0156370 0x7f4ac006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:57.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.863+0000 7f4ac6cdf640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ac0154850 0x7f4ac006ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:57.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.863+0000 7f4ac6cdf640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac00c37f0 con 0x7f4aa80053c0 2024-03-28T23:29:57.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.863+0000 7f4ac6cdf640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac00c6ab0 con 0x7f4ac0156370 2024-03-28T23:29:57.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.863+0000 7f4ac6cdf640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4ac00c3970 con 0x7f4ac0154850 2024-03-28T23:29:57.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.863+0000 7f4ac4a7b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4aa80053c0 0x7f4ac0154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:57.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.863+0000 7f4ac4a7b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4aa80053c0 0x7f4ac0154310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55250/0 (socket says 172.21.15.161:55250) 2024-03-28T23:29:57.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.863+0000 7f4ac4a7b640 1 -- 172.21.15.161:0/1092459561 learned_addr learned my addr 172.21.15.161:0/1092459561 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:57.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.863+0000 7f4ac527c640 1 --2- 172.21.15.161:0/1092459561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ac0154850 0x7f4ac006ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:57.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.863+0000 7f4ab7fff640 1 --2- 172.21.15.161:0/1092459561 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ac0156370 0x7f4ac006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:57.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.863+0000 7f4ac4a7b640 1 -- 172.21.15.161:0/1092459561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ac0154850 msgr2=0x7f4ac006ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:57.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.863+0000 7f4ac4a7b640 1 --2- 172.21.15.161:0/1092459561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ac0154850 0x7f4ac006ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.863+0000 7f4ac4a7b640 1 -- 172.21.15.161:0/1092459561 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ac0156370 msgr2=0x7f4ac006e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:57.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.863+0000 7f4ac4a7b640 1 --2- 172.21.15.161:0/1092459561 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ac0156370 0x7f4ac006e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:57.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.863+0000 7f4ac4a7b640 1 -- 172.21.15.161:0/1092459561 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ac00acf60 con 0x7f4aa80053c0 2024-03-28T23:29:57.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.863+0000 7f4ac4a7b640 1 --2- 172.21.15.161:0/1092459561 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4aa80053c0 0x7f4ac0154310 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4abc004ea0 tx=0x7f4abc004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:57.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.863+0000 7f4ab57fa640 1 -- 172.21.15.161:0/1092459561 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4abc00f680 con 0x7f4aa80053c0 2024-03-28T23:29:57.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.863+0000 7f4ab57fa640 1 -- 172.21.15.161:0/1092459561 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4abc018070 con 0x7f4aa80053c0 2024-03-28T23:29:57.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.863+0000 7f4ac6cdf640 1 -- 172.21.15.161:0/1092459561 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ac00701d0 con 0x7f4aa80053c0 2024-03-28T23:29:57.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.863+0000 7f4ab57fa640 1 -- 172.21.15.161:0/1092459561 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4abc00fa40 con 0x7f4aa80053c0 2024-03-28T23:29:57.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.863+0000 7f4ac6cdf640 1 -- 172.21.15.161:0/1092459561 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ac01ec050 con 0x7f4aa80053c0 2024-03-28T23:29:57.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.864+0000 7f4ac6cdf640 1 -- 172.21.15.161:0/1092459561 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ac004f4b0 con 0x7f4aa80053c0 2024-03-28T23:29:57.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.865+0000 7f4ab57fa640 1 -- 172.21.15.161:0/1092459561 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4abc00fbe0 con 0x7f4aa80053c0 2024-03-28T23:29:57.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.865+0000 7f4ab57fa640 1 --2- 172.21.15.161:0/1092459561 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4aac06b4d0 0x7f4aac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:57.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.865+0000 7f4ab57fa640 1 -- 172.21.15.161:0/1092459561 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4abc08a370 con 0x7f4aa80053c0 2024-03-28T23:29:57.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.865+0000 7f4ab7fff640 1 --2- 172.21.15.161:0/1092459561 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4aac06b4d0 0x7f4aac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:57.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.866+0000 7f4ab7fff640 1 --2- 172.21.15.161:0/1092459561 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4aac06b4d0 0x7f4aac06d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4ac00a7e00 tx=0x7f4ab0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:57.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:57.870+0000 7f4ab57fa640 1 -- 172.21.15.161:0/1092459561 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4abc058f40 con 0x7f4aa80053c0 2024-03-28T23:29:58.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.060+0000 7f4ac6cdf640 1 -- 172.21.15.161:0/1092459561 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4ac00734c0 con 0x7f4aa80053c0 2024-03-28T23:29:58.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.060+0000 7f4ab57fa640 1 -- 172.21.15.161:0/1092459561 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7f4abc058840 con 0x7f4aa80053c0 2024-03-28T23:29:58.061 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-28T23:29:58.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.062+0000 7f4ac6cdf640 1 -- 172.21.15.161:0/1092459561 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4aac06b4d0 msgr2=0x7f4aac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:58.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.062+0000 7f4ac6cdf640 1 --2- 172.21.15.161:0/1092459561 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4aac06b4d0 0x7f4aac06d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4ac00a7e00 tx=0x7f4ab0008040 comp rx=0 tx=0).stop 2024-03-28T23:29:58.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.062+0000 7f4ac6cdf640 1 -- 172.21.15.161:0/1092459561 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4aa80053c0 msgr2=0x7f4ac0154310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:58.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.062+0000 7f4ac6cdf640 1 --2- 172.21.15.161:0/1092459561 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4aa80053c0 0x7f4ac0154310 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4abc004ea0 tx=0x7f4abc004f90 comp rx=0 tx=0).stop 2024-03-28T23:29:58.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.062+0000 7f4ac6cdf640 1 -- 172.21.15.161:0/1092459561 shutdown_connections 2024-03-28T23:29:58.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.062+0000 7f4ac6cdf640 1 --2- 172.21.15.161:0/1092459561 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4aac06b4d0 0x7f4aac06d990 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.062+0000 7f4ac6cdf640 1 --2- 172.21.15.161:0/1092459561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ac0154850 0x7f4ac006ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.062+0000 7f4ac6cdf640 1 --2- 172.21.15.161:0/1092459561 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ac0156370 0x7f4ac006e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.062+0000 7f4ac6cdf640 1 --2- 172.21.15.161:0/1092459561 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4aa80053c0 0x7f4ac0154310 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.062+0000 7f4ac6cdf640 1 -- 172.21.15.161:0/1092459561 >> 172.21.15.161:0/1092459561 conn(0x7f4ac00aa740 msgr2=0x7f4ac00c5f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:58.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.062+0000 7f4ac6cdf640 1 -- 172.21.15.161:0/1092459561 shutdown_connections 2024-03-28T23:29:58.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.062+0000 7f4ac6cdf640 1 -- 172.21.15.161:0/1092459561 wait complete. 2024-03-28T23:29:58.073 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:29:58.074 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:29:58.089 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:29:58.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.230+0000 7fb87dac0640 1 Processor -- start 2024-03-28T23:29:58.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.230+0000 7fb87dac0640 1 -- start start 2024-03-28T23:29:58.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.230+0000 7fb87dac0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb8780c5410 0x7fb8780c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:58.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.230+0000 7fb87dac0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb878158a30 con 0x7fb878152890 2024-03-28T23:29:58.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.230+0000 7fb87dac0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb878158d10 con 0x7fb8781553a0 2024-03-28T23:29:58.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.230+0000 7fb87dac0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fb8780c8480 con 0x7fb8780c5410 2024-03-28T23:29:58.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.231+0000 7fb8777fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb878152890 0x7fb8780c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52318/0 (socket says 172.21.15.161:52318) 2024-03-28T23:29:58.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.231+0000 7fb8777fe640 1 -- 172.21.15.161:0/381499220 learned_addr learned my addr 172.21.15.161:0/381499220 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:58.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.231+0000 7fb876ffd640 1 --2- 172.21.15.161:0/381499220 >> v2:172.21.15.170:3300/0 conn(0x7fb8780c5410 0x7fb8780c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:58.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.231+0000 7fb8767fc640 1 -- 172.21.15.161:0/381499220 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1016976812 0 0) 0x7fb878158a30 con 0x7fb878152890 2024-03-28T23:29:58.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.231+0000 7fb8767fc640 1 -- 172.21.15.161:0/381499220 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb860003520 con 0x7fb878152890 2024-03-28T23:29:58.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.231+0000 7fb8767fc640 1 -- 172.21.15.161:0/381499220 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 296346123 0 0) 0x7fb860003520 con 0x7fb878152890 2024-03-28T23:29:58.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.231+0000 7fb8767fc640 1 -- 172.21.15.161:0/381499220 >> v2:172.21.15.170:3300/0 conn(0x7fb8780c5410 msgr2=0x7fb8780c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:58.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.231+0000 7fb8767fc640 1 --2- 172.21.15.161:0/381499220 >> v2:172.21.15.170:3300/0 conn(0x7fb8780c5410 0x7fb8780c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.231+0000 7fb8767fc640 1 -- 172.21.15.161:0/381499220 >> v1:172.21.15.170:6789/0 conn(0x7fb8781553a0 legacy=0x7fb878155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:58.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.231+0000 7fb8767fc640 1 -- 172.21.15.161:0/381499220 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb878158ff0 con 0x7fb878152890 2024-03-28T23:29:58.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.231+0000 7fb8767fc640 1 -- 172.21.15.161:0/381499220 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fb868001670 con 0x7fb878152890 2024-03-28T23:29:58.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.231+0000 7fb8767fc640 1 -- 172.21.15.161:0/381499220 >> v1:172.21.15.161:6789/0 conn(0x7fb878152890 legacy=0x7fb8780c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:58.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.231+0000 7fb8767fc640 1 --2- 172.21.15.161:0/381499220 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb860003a90 0x7fb860003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:58.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.231+0000 7fb8767fc640 1 --2- 172.21.15.161:0/381499220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb860004800 0x7fb860004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:58.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.231+0000 7fb8767fc640 1 --2- 172.21.15.161:0/381499220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb860005290 0x7fb86000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:58.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.231+0000 7fb8767fc640 1 -- 172.21.15.161:0/381499220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb86000c1b0 con 0x7fb860005290 2024-03-28T23:29:58.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.231+0000 7fb8767fc640 1 -- 172.21.15.161:0/381499220 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb86000c360 con 0x7fb860004800 2024-03-28T23:29:58.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.231+0000 7fb8767fc640 1 -- 172.21.15.161:0/381499220 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb86000c560 con 0x7fb860003a90 2024-03-28T23:29:58.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.231+0000 7fb877fff640 1 --2- 172.21.15.161:0/381499220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb860005290 0x7fb86000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:58.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.231+0000 7fb8777fe640 1 --2- 172.21.15.161:0/381499220 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb860003a90 0x7fb860003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:58.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.231+0000 7fb876ffd640 1 --2- 172.21.15.161:0/381499220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb860004800 0x7fb860004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:58.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.232+0000 7fb877fff640 1 -- 172.21.15.161:0/381499220 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb860003a90 msgr2=0x7fb860003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:58.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.232+0000 7fb877fff640 1 --2- 172.21.15.161:0/381499220 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb860003a90 0x7fb860003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.232+0000 7fb877fff640 1 -- 172.21.15.161:0/381499220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb860004800 msgr2=0x7fb860004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:58.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.232+0000 7fb877fff640 1 --2- 172.21.15.161:0/381499220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb860004800 0x7fb860004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.232+0000 7fb877fff640 1 -- 172.21.15.161:0/381499220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb860003520 con 0x7fb860005290 2024-03-28T23:29:58.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.232+0000 7fb877fff640 1 -- 172.21.15.161:0/381499220 reap_dead start 2024-03-28T23:29:58.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.232+0000 7fb877fff640 1 --2- 172.21.15.161:0/381499220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb860005290 0x7fb86000b9c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb870001e50 tx=0x7fb860005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9e0b5bf17f9083f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:58.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.233+0000 7fb8767fc640 1 -- 172.21.15.161:0/381499220 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb870001a70 con 0x7fb860005290 2024-03-28T23:29:58.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.233+0000 7fb8767fc640 1 -- 172.21.15.161:0/381499220 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb870003b90 con 0x7fb860005290 2024-03-28T23:29:58.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.233+0000 7fb8767fc640 1 -- 172.21.15.161:0/381499220 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8700050d0 con 0x7fb860005290 2024-03-28T23:29:58.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.233+0000 7fb87dac0640 1 -- 172.21.15.161:0/381499220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb860005290 msgr2=0x7fb86000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:58.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.233+0000 7fb87dac0640 1 --2- 172.21.15.161:0/381499220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb860005290 0x7fb86000b9c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb870001e50 tx=0x7fb860005830 comp rx=0 tx=0).stop 2024-03-28T23:29:58.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.233+0000 7fb87dac0640 1 -- 172.21.15.161:0/381499220 shutdown_connections 2024-03-28T23:29:58.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.233+0000 7fb87dac0640 1 --2- 172.21.15.161:0/381499220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb860005290 0x7fb86000b9c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.233+0000 7fb87dac0640 1 -- 172.21.15.161:0/381499220 >> 172.21.15.161:0/381499220 conn(0x7fb8780ac620 msgr2=0x7fb8780aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:58.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.233+0000 7fb87dac0640 1 -- 172.21.15.161:0/381499220 shutdown_connections 2024-03-28T23:29:58.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.233+0000 7fb87dac0640 1 -- 172.21.15.161:0/381499220 wait complete. 2024-03-28T23:29:58.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.234+0000 7fb87dac0640 1 Processor -- start 2024-03-28T23:29:58.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.234+0000 7fb87dac0640 1 -- start start 2024-03-28T23:29:58.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.235+0000 7fb87dac0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb860005290 0x7fb8780c92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:58.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.235+0000 7fb87dac0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8781ee890 0x7fb878157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:58.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.235+0000 7fb87dac0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb87814cfe0 0x7fb8780c4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:58.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.235+0000 7fb87dac0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb878152c70 con 0x7fb87814cfe0 2024-03-28T23:29:58.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.235+0000 7fb87dac0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb878159640 con 0x7fb8781ee890 2024-03-28T23:29:58.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.235+0000 7fb87dac0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb878152df0 con 0x7fb860005290 2024-03-28T23:29:58.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.235+0000 7fb877fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb87814cfe0 0x7fb8780c4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:58.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.235+0000 7fb877fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb87814cfe0 0x7fb8780c4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55278/0 (socket says 172.21.15.161:55278) 2024-03-28T23:29:58.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.235+0000 7fb877fff640 1 -- 172.21.15.161:0/1826351506 learned_addr learned my addr 172.21.15.161:0/1826351506 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:58.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.235+0000 7fb876ffd640 1 --2- 172.21.15.161:0/1826351506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8781ee890 0x7fb878157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:58.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.235+0000 7fb8777fe640 1 --2- 172.21.15.161:0/1826351506 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb860005290 0x7fb8780c92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:58.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.235+0000 7fb877fff640 1 -- 172.21.15.161:0/1826351506 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb860005290 msgr2=0x7fb8780c92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:58.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.235+0000 7fb877fff640 1 --2- 172.21.15.161:0/1826351506 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb860005290 0x7fb8780c92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.236+0000 7fb877fff640 1 -- 172.21.15.161:0/1826351506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8781ee890 msgr2=0x7fb878157290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:58.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.236+0000 7fb877fff640 1 --2- 172.21.15.161:0/1826351506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8781ee890 0x7fb878157290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.236+0000 7fb877fff640 1 -- 172.21.15.161:0/1826351506 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8780c9e50 con 0x7fb87814cfe0 2024-03-28T23:29:58.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.236+0000 7fb876ffd640 1 --2- 172.21.15.161:0/1826351506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8781ee890 0x7fb878157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:29:58.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.236+0000 7fb8777fe640 1 --2- 172.21.15.161:0/1826351506 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb860005290 0x7fb8780c92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:29:58.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.236+0000 7fb877fff640 1 --2- 172.21.15.161:0/1826351506 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb87814cfe0 0x7fb8780c4320 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb87002c060 tx=0x7fb8700014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:58.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.236+0000 7fb85ffff640 1 -- 172.21.15.161:0/1826351506 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb870007db0 con 0x7fb87814cfe0 2024-03-28T23:29:58.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.236+0000 7fb85ffff640 1 -- 172.21.15.161:0/1826351506 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb870006d50 con 0x7fb87814cfe0 2024-03-28T23:29:58.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.236+0000 7fb87dac0640 1 -- 172.21.15.161:0/1826351506 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8780c8800 con 0x7fb87814cfe0 2024-03-28T23:29:58.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.236+0000 7fb85ffff640 1 -- 172.21.15.161:0/1826351506 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb870039680 con 0x7fb87814cfe0 2024-03-28T23:29:58.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.236+0000 7fb87dac0640 1 -- 172.21.15.161:0/1826351506 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8780c8dc0 con 0x7fb87814cfe0 2024-03-28T23:29:58.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.237+0000 7fb87dac0640 1 -- 172.21.15.161:0/1826351506 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb878067a50 con 0x7fb87814cfe0 2024-03-28T23:29:58.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.237+0000 7fb85ffff640 1 -- 172.21.15.161:0/1826351506 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb870039820 con 0x7fb87814cfe0 2024-03-28T23:29:58.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.238+0000 7fb85ffff640 1 --2- 172.21.15.161:0/1826351506 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb86406b500 0x7fb86406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:58.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.238+0000 7fb8777fe640 1 --2- 172.21.15.161:0/1826351506 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb86406b500 0x7fb86406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:58.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.238+0000 7fb85ffff640 1 -- 172.21.15.161:0/1826351506 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb8700ac550 con 0x7fb87814cfe0 2024-03-28T23:29:58.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.238+0000 7fb8777fe640 1 --2- 172.21.15.161:0/1826351506 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb86406b500 0x7fb86406d9c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb868007bf0 tx=0x7fb868009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:58.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.242+0000 7fb85ffff640 1 -- 172.21.15.161:0/1826351506 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb878067a50 con 0x7fb87814cfe0 2024-03-28T23:29:58.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.443+0000 7fb87dac0640 1 -- 172.21.15.161:0/1826351506 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb87804f4b0 con 0x7fb87814cfe0 2024-03-28T23:29:58.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.444+0000 7fb85ffff640 1 -- 172.21.15.161:0/1826351506 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7fb870033030 con 0x7fb87814cfe0 2024-03-28T23:29:58.445 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:29:58.445 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":24,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:29:50.946843+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:29:58.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.446+0000 7fb87dac0640 1 -- 172.21.15.161:0/1826351506 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb86406b500 msgr2=0x7fb86406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:58.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.446+0000 7fb87dac0640 1 --2- 172.21.15.161:0/1826351506 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb86406b500 0x7fb86406d9c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb868007bf0 tx=0x7fb868009040 comp rx=0 tx=0).stop 2024-03-28T23:29:58.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.446+0000 7fb87dac0640 1 -- 172.21.15.161:0/1826351506 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb87814cfe0 msgr2=0x7fb8780c4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:58.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.446+0000 7fb87dac0640 1 --2- 172.21.15.161:0/1826351506 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb87814cfe0 0x7fb8780c4320 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb87002c060 tx=0x7fb8700014a0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.446+0000 7fb87dac0640 1 -- 172.21.15.161:0/1826351506 shutdown_connections 2024-03-28T23:29:58.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.446+0000 7fb87dac0640 1 --2- 172.21.15.161:0/1826351506 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb86406b500 0x7fb86406d9c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.446+0000 7fb87dac0640 1 --2- 172.21.15.161:0/1826351506 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb87814cfe0 0x7fb8780c4320 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.446+0000 7fb87dac0640 1 --2- 172.21.15.161:0/1826351506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8781ee890 0x7fb878157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.446+0000 7fb87dac0640 1 --2- 172.21.15.161:0/1826351506 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb860005290 0x7fb8780c92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.447+0000 7fb87dac0640 1 -- 172.21.15.161:0/1826351506 >> 172.21.15.161:0/1826351506 conn(0x7fb8780ac620 msgr2=0x7fb87806c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:58.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.447+0000 7fb87dac0640 1 -- 172.21.15.161:0/1826351506 shutdown_connections 2024-03-28T23:29:58.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.447+0000 7fb87dac0640 1 -- 172.21.15.161:0/1826351506 wait complete. 2024-03-28T23:29:58.463 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:29:44.021375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:29:44.872102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T23:29:48.591063+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T23:29:58.463 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:29:58.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.568+0000 7fc10b29d640 1 Processor -- start 2024-03-28T23:29:58.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.568+0000 7fc10b29d640 1 -- start start 2024-03-28T23:29:58.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.569+0000 7fc10b29d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc10406bb60 0x7fc1040c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:58.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.569+0000 7fc10b29d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc104152db0 con 0x7fc1041529d0 2024-03-28T23:29:58.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.569+0000 7fc10b29d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1040c7210 con 0x7fc1040c6750 2024-03-28T23:29:58.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.569+0000 7fc10b29d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc1040c6c50 con 0x7fc10406bb60 2024-03-28T23:29:58.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.569+0000 7fc109039640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc1041529d0 0x7fc104155c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52330/0 (socket says 172.21.15.161:52330) 2024-03-28T23:29:58.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.569+0000 7fc109039640 1 -- 172.21.15.161:0/508420448 learned_addr learned my addr 172.21.15.161:0/508420448 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:58.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.569+0000 7fc10983a640 1 --2- 172.21.15.161:0/508420448 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc10406bb60 0x7fc1040c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:58.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.569+0000 7fc0fbfff640 1 -- 172.21.15.161:0/508420448 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1401759699 0 0) 0x7fc104152db0 con 0x7fc1041529d0 2024-03-28T23:29:58.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.569+0000 7fc0fbfff640 1 -- 172.21.15.161:0/508420448 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0f0003520 con 0x7fc1041529d0 2024-03-28T23:29:58.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.570+0000 7fc0fbfff640 1 -- 172.21.15.161:0/508420448 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2074143839 0 0) 0x7fc1040c7210 con 0x7fc1040c6750 2024-03-28T23:29:58.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.570+0000 7fc0fbfff640 1 -- 172.21.15.161:0/508420448 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc104152db0 con 0x7fc1040c6750 2024-03-28T23:29:58.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.570+0000 7fc0fbfff640 1 -- 172.21.15.161:0/508420448 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3817353399 0 0) 0x7fc0f0003520 con 0x7fc1041529d0 2024-03-28T23:29:58.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.570+0000 7fc10983a640 1 -- 172.21.15.161:0/508420448 >> v1:172.21.15.170:6789/0 conn(0x7fc1040c6750 legacy=0x7fc1040c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:58.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.570+0000 7fc10983a640 1 -- 172.21.15.161:0/508420448 >> v1:172.21.15.161:6789/0 conn(0x7fc1041529d0 legacy=0x7fc104155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:58.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.570+0000 7fc10983a640 1 -- 172.21.15.161:0/508420448 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1040c6f30 con 0x7fc10406bb60 2024-03-28T23:29:58.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.571+0000 7fc10983a640 1 --2- 172.21.15.161:0/508420448 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc10406bb60 0x7fc1040c88b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc009dc0 tx=0x7fc0fc02bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bb925abe1cd9e76 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:58.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.571+0000 7fc0fbfff640 1 -- 172.21.15.161:0/508420448 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc039070 con 0x7fc10406bb60 2024-03-28T23:29:58.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.571+0000 7fc0fbfff640 1 -- 172.21.15.161:0/508420448 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0fc02c880 con 0x7fc10406bb60 2024-03-28T23:29:58.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.571+0000 7fc0fbfff640 1 -- 172.21.15.161:0/508420448 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f4001670 con 0x7fc10406bb60 2024-03-28T23:29:58.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.571+0000 7fc10b29d640 1 -- 172.21.15.161:0/508420448 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc10406bb60 msgr2=0x7fc1040c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:58.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.571+0000 7fc10b29d640 1 --2- 172.21.15.161:0/508420448 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc10406bb60 0x7fc1040c88b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc009dc0 tx=0x7fc0fc02bd50 comp rx=0 tx=0).stop 2024-03-28T23:29:58.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.571+0000 7fc10b29d640 1 -- 172.21.15.161:0/508420448 shutdown_connections 2024-03-28T23:29:58.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.571+0000 7fc10b29d640 1 --2- 172.21.15.161:0/508420448 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc10406bb60 0x7fc1040c88b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.571+0000 7fc10b29d640 1 -- 172.21.15.161:0/508420448 >> 172.21.15.161:0/508420448 conn(0x7fc1040c7430 msgr2=0x7fc1040c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:58.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.572+0000 7fc10b29d640 1 -- 172.21.15.161:0/508420448 shutdown_connections 2024-03-28T23:29:58.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.572+0000 7fc10b29d640 1 -- 172.21.15.161:0/508420448 wait complete. 2024-03-28T23:29:58.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.573+0000 7fc10b29d640 1 Processor -- start 2024-03-28T23:29:58.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.573+0000 7fc10b29d640 1 -- start start 2024-03-28T23:29:58.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.574+0000 7fc10b29d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc10406bb60 0x7fc1041eb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:58.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.574+0000 7fc10b29d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1040c6750 0x7fc1040ab5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:58.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.574+0000 7fc10b29d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1041529d0 0x7fc1040c2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:58.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.574+0000 7fc10b29d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc1040aa160 con 0x7fc1040c6750 2024-03-28T23:29:58.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.574+0000 7fc10b29d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc104155710 con 0x7fc10406bb60 2024-03-28T23:29:58.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.574+0000 7fc10b29d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc1040c5c00 con 0x7fc1041529d0 2024-03-28T23:29:58.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.574+0000 7fc108838640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1040c6750 0x7fc1040ab5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:58.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.574+0000 7fc108838640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1040c6750 0x7fc1040ab5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55286/0 (socket says 172.21.15.161:55286) 2024-03-28T23:29:58.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.574+0000 7fc108838640 1 -- 172.21.15.161:0/3577660128 learned_addr learned my addr 172.21.15.161:0/3577660128 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:58.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.575+0000 7fc109039640 1 --2- 172.21.15.161:0/3577660128 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc10406bb60 0x7fc1041eb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:58.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.575+0000 7fc10983a640 1 --2- 172.21.15.161:0/3577660128 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1041529d0 0x7fc1040c2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:58.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.575+0000 7fc108838640 1 -- 172.21.15.161:0/3577660128 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1041529d0 msgr2=0x7fc1040c2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:58.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.575+0000 7fc108838640 1 --2- 172.21.15.161:0/3577660128 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1041529d0 0x7fc1040c2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.575+0000 7fc108838640 1 -- 172.21.15.161:0/3577660128 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc10406bb60 msgr2=0x7fc1041eb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:58.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.575+0000 7fc108838640 1 --2- 172.21.15.161:0/3577660128 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc10406bb60 0x7fc1041eb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.575+0000 7fc108838640 1 -- 172.21.15.161:0/3577660128 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc104153330 con 0x7fc1040c6750 2024-03-28T23:29:58.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.575+0000 7fc10983a640 1 --2- 172.21.15.161:0/3577660128 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1041529d0 0x7fc1040c2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:29:58.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.575+0000 7fc109039640 1 --2- 172.21.15.161:0/3577660128 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc10406bb60 0x7fc1041eb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:29:58.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.575+0000 7fc108838640 1 --2- 172.21.15.161:0/3577660128 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1040c6750 0x7fc1040ab5a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc10000c500 tx=0x7fc10000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:58.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.575+0000 7fc0f9ffb640 1 -- 172.21.15.161:0/3577660128 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1000052d0 con 0x7fc1040c6750 2024-03-28T23:29:58.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.575+0000 7fc0f9ffb640 1 -- 172.21.15.161:0/3577660128 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc100005490 con 0x7fc1040c6750 2024-03-28T23:29:58.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.575+0000 7fc0f9ffb640 1 -- 172.21.15.161:0/3577660128 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc100012870 con 0x7fc1040c6750 2024-03-28T23:29:58.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.575+0000 7fc10b29d640 1 -- 172.21.15.161:0/3577660128 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc104154ce0 con 0x7fc1040c6750 2024-03-28T23:29:58.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.575+0000 7fc10b29d640 1 -- 172.21.15.161:0/3577660128 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1041538b0 con 0x7fc1040c6750 2024-03-28T23:29:58.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.576+0000 7fc10b29d640 1 -- 172.21.15.161:0/3577660128 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc10414ca90 con 0x7fc1040c6750 2024-03-28T23:29:58.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.577+0000 7fc0f9ffb640 1 -- 172.21.15.161:0/3577660128 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc100012a10 con 0x7fc1040c6750 2024-03-28T23:29:58.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.577+0000 7fc0f9ffb640 1 --2- 172.21.15.161:0/3577660128 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc0e806b4d0 0x7fc0e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:58.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.578+0000 7fc0f9ffb640 1 -- 172.21.15.161:0/3577660128 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc10008a230 con 0x7fc1040c6750 2024-03-28T23:29:58.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.578+0000 7fc109039640 1 --2- 172.21.15.161:0/3577660128 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc0e806b4d0 0x7fc0e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:58.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.578+0000 7fc109039640 1 --2- 172.21.15.161:0/3577660128 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc0e806b4d0 0x7fc0e806d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4001eb0 tx=0x7fc0f4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:58.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.582+0000 7fc0f9ffb640 1 -- 172.21.15.161:0/3577660128 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc10414ca90 con 0x7fc1040c6750 2024-03-28T23:29:58.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.765+0000 7fc10b29d640 1 -- 172.21.15.161:0/3577660128 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc104153160 con 0x7fc1040c6750 2024-03-28T23:29:58.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.766+0000 7fc0f9ffb640 1 -- 172.21.15.161:0/3577660128 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7fc100058d80 con 0x7fc1040c6750 2024-03-28T23:29:58.767 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:29:58.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.768+0000 7fc10b29d640 1 -- 172.21.15.161:0/3577660128 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc0e806b4d0 msgr2=0x7fc0e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:58.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.768+0000 7fc10b29d640 1 --2- 172.21.15.161:0/3577660128 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc0e806b4d0 0x7fc0e806d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4001eb0 tx=0x7fc0f4009040 comp rx=0 tx=0).stop 2024-03-28T23:29:58.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.768+0000 7fc10b29d640 1 -- 172.21.15.161:0/3577660128 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1040c6750 msgr2=0x7fc1040ab5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:58.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.768+0000 7fc10b29d640 1 --2- 172.21.15.161:0/3577660128 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1040c6750 0x7fc1040ab5a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc10000c500 tx=0x7fc10000c9c0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.768+0000 7fc10b29d640 1 -- 172.21.15.161:0/3577660128 shutdown_connections 2024-03-28T23:29:58.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.768+0000 7fc10b29d640 1 --2- 172.21.15.161:0/3577660128 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc0e806b4d0 0x7fc0e806d990 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.768+0000 7fc10b29d640 1 --2- 172.21.15.161:0/3577660128 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1040c6750 0x7fc1040ab5a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.768+0000 7fc10b29d640 1 --2- 172.21.15.161:0/3577660128 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1041529d0 0x7fc1040c2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.768+0000 7fc10b29d640 1 --2- 172.21.15.161:0/3577660128 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc10406bb60 0x7fc1041eb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.768+0000 7fc10b29d640 1 -- 172.21.15.161:0/3577660128 >> 172.21.15.161:0/3577660128 conn(0x7fc1040c7430 msgr2=0x7fc104074020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:58.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.768+0000 7fc10b29d640 1 -- 172.21.15.161:0/3577660128 shutdown_connections 2024-03-28T23:29:58.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.768+0000 7fc10b29d640 1 -- 172.21.15.161:0/3577660128 wait complete. 2024-03-28T23:29:58.783 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:29:58.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.888+0000 7f1415b55640 1 Processor -- start 2024-03-28T23:29:58.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.888+0000 7f1415b55640 1 -- start start 2024-03-28T23:29:58.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.889+0000 7f1415b55640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1410152bc0 0x7f14100c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:58.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.889+0000 7f1415b55640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14100c8a80 0x7f14100c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:58.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.889+0000 7f1415b55640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14100c6f60 con 0x7f14100c9420 2024-03-28T23:29:58.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.889+0000 7f1415b55640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f14100c6120 con 0x7f1410152bc0 2024-03-28T23:29:58.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.889+0000 7f1415b55640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f14100c6400 con 0x7f14100c8a80 2024-03-28T23:29:58.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.889+0000 7f140ffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f14100c9420 0x7f14101589b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52336/0 (socket says 172.21.15.161:52336) 2024-03-28T23:29:58.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.889+0000 7f140ffff640 1 -- 172.21.15.161:0/2314711850 learned_addr learned my addr 172.21.15.161:0/2314711850 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:58.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.889+0000 7f140f7fe640 1 --2- 172.21.15.161:0/2314711850 >> v2:172.21.15.170:3300/0 conn(0x7f1410152bc0 0x7f14100c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:58.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140effd640 1 --2- 172.21.15.161:0/2314711850 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14100c8a80 0x7f14100c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:58.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140e7fc640 1 -- 172.21.15.161:0/2314711850 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2321784471 0 0) 0x7f14100c6f60 con 0x7f14100c9420 2024-03-28T23:29:58.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140e7fc640 1 -- 172.21.15.161:0/2314711850 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13fc003520 con 0x7f14100c9420 2024-03-28T23:29:58.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140e7fc640 1 -- 172.21.15.161:0/2314711850 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4044421277 0 0) 0x7f13fc003520 con 0x7f14100c9420 2024-03-28T23:29:58.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140e7fc640 1 -- 172.21.15.161:0/2314711850 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14100c8a80 msgr2=0x7f14100c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:58.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140e7fc640 1 --2- 172.21.15.161:0/2314711850 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14100c8a80 0x7f14100c8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140e7fc640 1 -- 172.21.15.161:0/2314711850 >> v2:172.21.15.170:3300/0 conn(0x7f1410152bc0 msgr2=0x7f14100c0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:58.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140e7fc640 1 --2- 172.21.15.161:0/2314711850 >> v2:172.21.15.170:3300/0 conn(0x7f1410152bc0 0x7f14100c0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140e7fc640 1 -- 172.21.15.161:0/2314711850 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14100c7240 con 0x7f14100c9420 2024-03-28T23:29:58.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140e7fc640 1 -- 172.21.15.161:0/2314711850 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f14080019f0 con 0x7f14100c9420 2024-03-28T23:29:58.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140f7fe640 1 --2- 172.21.15.161:0/2314711850 >> v2:172.21.15.170:3300/0 conn(0x7f1410152bc0 0x7f14100c0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:29:58.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140e7fc640 1 -- 172.21.15.161:0/2314711850 >> v1:172.21.15.161:6789/0 conn(0x7f14100c9420 legacy=0x7f14101589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:58.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140e7fc640 1 --2- 172.21.15.161:0/2314711850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13fc003a90 0x7f13fc003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:58.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140e7fc640 1 --2- 172.21.15.161:0/2314711850 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13fc004800 0x7f13fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:58.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140e7fc640 1 --2- 172.21.15.161:0/2314711850 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14100c8a80 0x7f13fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:58.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140e7fc640 1 -- 172.21.15.161:0/2314711850 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f14100c6120 con 0x7f13fc003a90 2024-03-28T23:29:58.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140e7fc640 1 -- 172.21.15.161:0/2314711850 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f14100c6400 con 0x7f13fc004800 2024-03-28T23:29:58.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140e7fc640 1 -- 172.21.15.161:0/2314711850 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f13fc00b9f0 con 0x7f14100c8a80 2024-03-28T23:29:58.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140f7fe640 1 --2- 172.21.15.161:0/2314711850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13fc003a90 0x7f13fc003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:58.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140effd640 1 --2- 172.21.15.161:0/2314711850 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13fc004800 0x7f13fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:58.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140ffff640 1 --2- 172.21.15.161:0/2314711850 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14100c8a80 0x7f13fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:58.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140f7fe640 1 -- 172.21.15.161:0/2314711850 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14100c8a80 msgr2=0x7f13fc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:58.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140f7fe640 1 --2- 172.21.15.161:0/2314711850 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14100c8a80 0x7f13fc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140f7fe640 1 -- 172.21.15.161:0/2314711850 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13fc004800 msgr2=0x7f13fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:58.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140f7fe640 1 --2- 172.21.15.161:0/2314711850 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13fc004800 0x7f13fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.890+0000 7f140f7fe640 1 -- 172.21.15.161:0/2314711850 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13fc003520 con 0x7f13fc003a90 2024-03-28T23:29:58.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.891+0000 7f140f7fe640 1 --2- 172.21.15.161:0/2314711850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13fc003a90 0x7f13fc003eb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1400009c90 tx=0x7f13fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2af0bcc19a2e43c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:58.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.891+0000 7f140e7fc640 1 -- 172.21.15.161:0/2314711850 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1400039070 con 0x7f13fc003a90 2024-03-28T23:29:58.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.891+0000 7f140e7fc640 1 -- 172.21.15.161:0/2314711850 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f140002c980 con 0x7f13fc003a90 2024-03-28T23:29:58.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.891+0000 7f140e7fc640 1 -- 172.21.15.161:0/2314711850 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1400009860 con 0x7f13fc003a90 2024-03-28T23:29:58.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.892+0000 7f1415b55640 1 -- 172.21.15.161:0/2314711850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13fc003a90 msgr2=0x7f13fc003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:58.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.892+0000 7f1415b55640 1 --2- 172.21.15.161:0/2314711850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13fc003a90 0x7f13fc003eb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1400009c90 tx=0x7f13fc005830 comp rx=0 tx=0).stop 2024-03-28T23:29:58.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.892+0000 7f140ffff640 1 -- 172.21.15.161:0/2314711850 reap_dead start 2024-03-28T23:29:58.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.892+0000 7f1415b55640 1 -- 172.21.15.161:0/2314711850 shutdown_connections 2024-03-28T23:29:58.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.892+0000 7f1415b55640 1 -- 172.21.15.161:0/2314711850 >> 172.21.15.161:0/2314711850 conn(0x7f14100caa80 msgr2=0x7f14100cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:58.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.892+0000 7f1415b55640 1 -- 172.21.15.161:0/2314711850 shutdown_connections 2024-03-28T23:29:58.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.892+0000 7f1415b55640 1 -- 172.21.15.161:0/2314711850 wait complete. 2024-03-28T23:29:58.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.893+0000 7f1415b55640 1 Processor -- start 2024-03-28T23:29:58.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.893+0000 7f1415b55640 1 -- start start 2024-03-28T23:29:58.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.894+0000 7f1415b55640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13fc003a90 0x7f1410141680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:58.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.894+0000 7f1415b55640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1410141150 0x7f1410157800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:58.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.894+0000 7f1415b55640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14100c3e00 0x7f14101eb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:58.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.894+0000 7f1415b55640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1410152fa0 con 0x7f14100c3e00 2024-03-28T23:29:58.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.894+0000 7f1415b55640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f14100c8ee0 con 0x7f13fc003a90 2024-03-28T23:29:58.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.894+0000 7f1415b55640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f14101eb000 con 0x7f1410141150 2024-03-28T23:29:58.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.894+0000 7f140ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14100c3e00 0x7f14101eb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:58.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.894+0000 7f140ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14100c3e00 0x7f14101eb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55292/0 (socket says 172.21.15.161:55292) 2024-03-28T23:29:58.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.894+0000 7f140ffff640 1 -- 172.21.15.161:0/4120321400 learned_addr learned my addr 172.21.15.161:0/4120321400 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:58.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.895+0000 7f140f7fe640 1 --2- 172.21.15.161:0/4120321400 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13fc003a90 0x7f1410141680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:58.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.895+0000 7f140effd640 1 --2- 172.21.15.161:0/4120321400 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1410141150 0x7f1410157800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:58.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.895+0000 7f140ffff640 1 -- 172.21.15.161:0/4120321400 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1410141150 msgr2=0x7f1410157800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:58.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.895+0000 7f140ffff640 1 --2- 172.21.15.161:0/4120321400 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1410141150 0x7f1410157800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.895+0000 7f140ffff640 1 -- 172.21.15.161:0/4120321400 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13fc003a90 msgr2=0x7f1410141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:58.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.895+0000 7f140ffff640 1 --2- 172.21.15.161:0/4120321400 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13fc003a90 0x7f1410141680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:58.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.895+0000 7f140ffff640 1 -- 172.21.15.161:0/4120321400 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1410159790 con 0x7f14100c3e00 2024-03-28T23:29:58.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.895+0000 7f140ffff640 1 --2- 172.21.15.161:0/4120321400 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14100c3e00 0x7f14101eb490 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1408000f80 tx=0x7f140800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:58.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.895+0000 7f1407fff640 1 -- 172.21.15.161:0/4120321400 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140800ebb0 con 0x7f14100c3e00 2024-03-28T23:29:58.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.895+0000 7f1407fff640 1 -- 172.21.15.161:0/4120321400 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f140800ed50 con 0x7f14100c3e00 2024-03-28T23:29:58.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.895+0000 7f1415b55640 1 -- 172.21.15.161:0/4120321400 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14101591d0 con 0x7f14100c3e00 2024-03-28T23:29:58.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.895+0000 7f1407fff640 1 -- 172.21.15.161:0/4120321400 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1408014670 con 0x7f14100c3e00 2024-03-28T23:29:58.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.895+0000 7f1415b55640 1 -- 172.21.15.161:0/4120321400 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14101eac90 con 0x7f14100c3e00 2024-03-28T23:29:58.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.896+0000 7f1415b55640 1 -- 172.21.15.161:0/4120321400 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f141004f4b0 con 0x7f14100c3e00 2024-03-28T23:29:58.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.897+0000 7f1407fff640 1 -- 172.21.15.161:0/4120321400 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1408014810 con 0x7f14100c3e00 2024-03-28T23:29:58.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.897+0000 7f1407fff640 1 --2- 172.21.15.161:0/4120321400 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f13f006b500 0x7f13f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:58.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.897+0000 7f1407fff640 1 -- 172.21.15.161:0/4120321400 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f140808a260 con 0x7f14100c3e00 2024-03-28T23:29:58.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.897+0000 7f140f7fe640 1 --2- 172.21.15.161:0/4120321400 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f13f006b500 0x7f13f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:58.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.898+0000 7f140f7fe640 1 --2- 172.21.15.161:0/4120321400 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f13f006b500 0x7f13f006d9c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1400036000 tx=0x7f1400030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:58.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:58.902+0000 7f1407fff640 1 -- 172.21.15.161:0/4120321400 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1410159790 con 0x7f14100c3e00 2024-03-28T23:29:59.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.074+0000 7f1415b55640 1 -- 172.21.15.161:0/4120321400 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f14100c5c70 con 0x7f14100c3e00 2024-03-28T23:29:59.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.075+0000 7f1407fff640 1 -- 172.21.15.161:0/4120321400 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f1408058e30 con 0x7f14100c3e00 2024-03-28T23:29:59.076 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:29:59.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.077+0000 7f1415b55640 1 -- 172.21.15.161:0/4120321400 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f13f006b500 msgr2=0x7f13f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:59.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.077+0000 7f1415b55640 1 --2- 172.21.15.161:0/4120321400 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f13f006b500 0x7f13f006d9c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1400036000 tx=0x7f1400030000 comp rx=0 tx=0).stop 2024-03-28T23:29:59.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.077+0000 7f1415b55640 1 -- 172.21.15.161:0/4120321400 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14100c3e00 msgr2=0x7f14101eb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:59.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.077+0000 7f1415b55640 1 --2- 172.21.15.161:0/4120321400 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14100c3e00 0x7f14101eb490 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1408000f80 tx=0x7f140800e530 comp rx=0 tx=0).stop 2024-03-28T23:29:59.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.077+0000 7f1415b55640 1 -- 172.21.15.161:0/4120321400 shutdown_connections 2024-03-28T23:29:59.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.077+0000 7f1415b55640 1 --2- 172.21.15.161:0/4120321400 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f13f006b500 0x7f13f006d9c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.077+0000 7f1415b55640 1 --2- 172.21.15.161:0/4120321400 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14100c3e00 0x7f14101eb490 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.077+0000 7f1415b55640 1 --2- 172.21.15.161:0/4120321400 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1410141150 0x7f1410157800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.077+0000 7f1415b55640 1 --2- 172.21.15.161:0/4120321400 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13fc003a90 0x7f1410141680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.077+0000 7f1415b55640 1 -- 172.21.15.161:0/4120321400 >> 172.21.15.161:0/4120321400 conn(0x7f14100caa80 msgr2=0x7f14100abd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:59.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.077+0000 7f1415b55640 1 -- 172.21.15.161:0/4120321400 shutdown_connections 2024-03-28T23:29:59.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.077+0000 7f1415b55640 1 -- 172.21.15.161:0/4120321400 wait complete. 2024-03-28T23:29:59.092 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:29:59.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.196+0000 7f363b69a640 1 Processor -- start 2024-03-28T23:29:59.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.197+0000 7f363b69a640 1 -- start start 2024-03-28T23:29:59.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.197+0000 7f363b69a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36340c6350 0x7f363406c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:59.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.197+0000 7f363b69a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f36340a5fb0 0x7f363406cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:59.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.197+0000 7f363b69a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3634152960 con 0x7f3634151e70 2024-03-28T23:29:59.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.197+0000 7f363b69a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f36340c1e40 con 0x7f36340a5fb0 2024-03-28T23:29:59.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.197+0000 7f363b69a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f36340c2120 con 0x7f36340c6350 2024-03-28T23:29:59.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.198+0000 7f3639436640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f3634151e70 0x7f3634152250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52348/0 (socket says 172.21.15.161:52348) 2024-03-28T23:29:59.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.198+0000 7f3639436640 1 -- 172.21.15.161:0/2323976613 learned_addr learned my addr 172.21.15.161:0/2323976613 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:59.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.198+0000 7f362bfff640 1 -- 172.21.15.161:0/2323976613 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3698226022 0 0) 0x7f3634152960 con 0x7f3634151e70 2024-03-28T23:29:59.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.198+0000 7f3639c37640 1 --2- 172.21.15.161:0/2323976613 >> v2:172.21.15.170:3300/0 conn(0x7f36340a5fb0 0x7f363406cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:59.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.198+0000 7f3638c35640 1 --2- 172.21.15.161:0/2323976613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36340c6350 0x7f363406c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:59.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.198+0000 7f362bfff640 1 -- 172.21.15.161:0/2323976613 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3620003520 con 0x7f3634151e70 2024-03-28T23:29:59.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.198+0000 7f362bfff640 1 -- 172.21.15.161:0/2323976613 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 689267658 0 0) 0x7f3620003520 con 0x7f3634151e70 2024-03-28T23:29:59.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.198+0000 7f362bfff640 1 -- 172.21.15.161:0/2323976613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36340c6350 msgr2=0x7f363406c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:59.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.198+0000 7f362bfff640 1 --2- 172.21.15.161:0/2323976613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36340c6350 0x7f363406c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.198+0000 7f362bfff640 1 -- 172.21.15.161:0/2323976613 >> v2:172.21.15.170:3300/0 conn(0x7f36340a5fb0 msgr2=0x7f363406cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:59.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.198+0000 7f362bfff640 1 --2- 172.21.15.161:0/2323976613 >> v2:172.21.15.170:3300/0 conn(0x7f36340a5fb0 0x7f363406cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.198+0000 7f362bfff640 1 -- 172.21.15.161:0/2323976613 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36340c2400 con 0x7f3634151e70 2024-03-28T23:29:59.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.198+0000 7f362bfff640 1 -- 172.21.15.161:0/2323976613 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f3624001670 con 0x7f3634151e70 2024-03-28T23:29:59.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.198+0000 7f3638c35640 1 --2- 172.21.15.161:0/2323976613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36340c6350 0x7f363406c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:29:59.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.198+0000 7f362bfff640 1 -- 172.21.15.161:0/2323976613 >> v1:172.21.15.161:6789/0 conn(0x7f3634151e70 legacy=0x7f3634152250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:59.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.198+0000 7f362bfff640 1 --2- 172.21.15.161:0/2323976613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36200039b0 0x7f3620003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:59.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.198+0000 7f362bfff640 1 --2- 172.21.15.161:0/2323976613 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3620004800 0x7f3620004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:59.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.198+0000 7f362bfff640 1 --2- 172.21.15.161:0/2323976613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36340c6350 0x7f36200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:59.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.198+0000 7f362bfff640 1 -- 172.21.15.161:0/2323976613 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f36340c1e40 con 0x7f3620004800 2024-03-28T23:29:59.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.198+0000 7f362bfff640 1 -- 172.21.15.161:0/2323976613 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f36340c2120 con 0x7f36200039b0 2024-03-28T23:29:59.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.198+0000 7f362bfff640 1 -- 172.21.15.161:0/2323976613 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f362000ba70 con 0x7f36340c6350 2024-03-28T23:29:59.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.198+0000 7f3638c35640 1 --2- 172.21.15.161:0/2323976613 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3620004800 0x7f3620004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:59.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.198+0000 7f3639436640 1 --2- 172.21.15.161:0/2323976613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36200039b0 0x7f3620003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:59.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.198+0000 7f3639c37640 1 --2- 172.21.15.161:0/2323976613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36340c6350 0x7f36200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:59.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.199+0000 7f3638c35640 1 -- 172.21.15.161:0/2323976613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36340c6350 msgr2=0x7f36200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:59.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.199+0000 7f3638c35640 1 --2- 172.21.15.161:0/2323976613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36340c6350 0x7f36200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.199+0000 7f3638c35640 1 -- 172.21.15.161:0/2323976613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36200039b0 msgr2=0x7f3620003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:59.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.199+0000 7f3638c35640 1 --2- 172.21.15.161:0/2323976613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36200039b0 0x7f3620003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.199+0000 7f3638c35640 1 -- 172.21.15.161:0/2323976613 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3620003520 con 0x7f3620004800 2024-03-28T23:29:59.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.199+0000 7f3638c35640 1 --2- 172.21.15.161:0/2323976613 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3620004800 0x7f3620004c80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f36340c6770 tx=0x7f3620005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=348cdcf645cabbf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:59.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.199+0000 7f362bfff640 1 -- 172.21.15.161:0/2323976613 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363000f040 con 0x7f3620004800 2024-03-28T23:29:59.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.199+0000 7f362bfff640 1 -- 172.21.15.161:0/2323976613 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3630004030 con 0x7f3620004800 2024-03-28T23:29:59.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.199+0000 7f362bfff640 1 -- 172.21.15.161:0/2323976613 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3630004400 con 0x7f3620004800 2024-03-28T23:29:59.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.200+0000 7f363b69a640 1 -- 172.21.15.161:0/2323976613 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3620004800 msgr2=0x7f3620004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:59.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.200+0000 7f363b69a640 1 --2- 172.21.15.161:0/2323976613 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3620004800 0x7f3620004c80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f36340c6770 tx=0x7f3620005830 comp rx=0 tx=0).stop 2024-03-28T23:29:59.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.200+0000 7f3639c37640 1 -- 172.21.15.161:0/2323976613 reap_dead start 2024-03-28T23:29:59.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.200+0000 7f363b69a640 1 -- 172.21.15.161:0/2323976613 shutdown_connections 2024-03-28T23:29:59.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.200+0000 7f363b69a640 1 -- 172.21.15.161:0/2323976613 >> 172.21.15.161:0/2323976613 conn(0x7f36340c3390 msgr2=0x7f36340c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:59.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.200+0000 7f363b69a640 1 -- 172.21.15.161:0/2323976613 shutdown_connections 2024-03-28T23:29:59.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.200+0000 7f363b69a640 1 -- 172.21.15.161:0/2323976613 wait complete. 2024-03-28T23:29:59.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.201+0000 7f363b69a640 1 Processor -- start 2024-03-28T23:29:59.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.201+0000 7f363b69a640 1 -- start start 2024-03-28T23:29:59.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.202+0000 7f363b69a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3620004800 0x7f36341ee340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:59.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.202+0000 7f363b69a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36340a6d40 0x7f36340bff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:59.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.202+0000 7f363b69a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f363414d550 0x7f3634143bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:59.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.202+0000 7f363b69a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f36340c2c00 con 0x7f363414d550 2024-03-28T23:29:59.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.202+0000 7f363b69a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f363414f810 con 0x7f36340a6d40 2024-03-28T23:29:59.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.202+0000 7f363b69a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f36340c2d80 con 0x7f3620004800 2024-03-28T23:29:59.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.202+0000 7f3639c37640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f363414d550 0x7f3634143bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:59.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.202+0000 7f3639c37640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f363414d550 0x7f3634143bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55314/0 (socket says 172.21.15.161:55314) 2024-03-28T23:29:59.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.202+0000 7f3639c37640 1 -- 172.21.15.161:0/751408595 learned_addr learned my addr 172.21.15.161:0/751408595 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:59.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.202+0000 7f3639436640 1 --2- 172.21.15.161:0/751408595 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3620004800 0x7f36341ee340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:59.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.202+0000 7f3638c35640 1 --2- 172.21.15.161:0/751408595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36340a6d40 0x7f36340bff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:59.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.202+0000 7f3639c37640 1 -- 172.21.15.161:0/751408595 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3620004800 msgr2=0x7f36341ee340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:59.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.202+0000 7f3639c37640 1 --2- 172.21.15.161:0/751408595 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3620004800 0x7f36341ee340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.202+0000 7f3639c37640 1 -- 172.21.15.161:0/751408595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36340a6d40 msgr2=0x7f36340bff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:59.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.202+0000 7f3639c37640 1 --2- 172.21.15.161:0/751408595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36340a6d40 0x7f36340bff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.202+0000 7f3639c37640 1 -- 172.21.15.161:0/751408595 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36340a82f0 con 0x7f363414d550 2024-03-28T23:29:59.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.202+0000 7f3639c37640 1 --2- 172.21.15.161:0/751408595 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f363414d550 0x7f3634143bf0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f363406fdc0 tx=0x7f362c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:59.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.203+0000 7f3629ffb640 1 -- 172.21.15.161:0/751408595 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362c015430 con 0x7f363414d550 2024-03-28T23:29:59.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.203+0000 7f3629ffb640 1 -- 172.21.15.161:0/751408595 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f362c0155d0 con 0x7f363414d550 2024-03-28T23:29:59.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.203+0000 7f363b69a640 1 -- 172.21.15.161:0/751408595 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36340a6700 con 0x7f363414d550 2024-03-28T23:29:59.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.203+0000 7f3629ffb640 1 -- 172.21.15.161:0/751408595 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362c0159c0 con 0x7f363414d550 2024-03-28T23:29:59.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.203+0000 7f363b69a640 1 -- 172.21.15.161:0/751408595 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36340c0470 con 0x7f363414d550 2024-03-28T23:29:59.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.204+0000 7f363b69a640 1 -- 172.21.15.161:0/751408595 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f363404f4b0 con 0x7f363414d550 2024-03-28T23:29:59.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.204+0000 7f3629ffb640 1 -- 172.21.15.161:0/751408595 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f362c015b60 con 0x7f363414d550 2024-03-28T23:29:59.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.205+0000 7f3629ffb640 1 --2- 172.21.15.161:0/751408595 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f361806b4d0 0x7f361806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:59.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.205+0000 7f3639436640 1 --2- 172.21.15.161:0/751408595 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f361806b4d0 0x7f361806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:59.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.205+0000 7f3629ffb640 1 -- 172.21.15.161:0/751408595 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f362c08b390 con 0x7f363414d550 2024-03-28T23:29:59.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.205+0000 7f3639436640 1 --2- 172.21.15.161:0/751408595 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f361806b4d0 0x7f361806d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f36240015d0 tx=0x7f3624009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:59.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.209+0000 7f3629ffb640 1 -- 172.21.15.161:0/751408595 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f362c019030 con 0x7f363414d550 2024-03-28T23:29:59.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.385+0000 7f363b69a640 1 -- 172.21.15.161:0/751408595 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f36340c2f80 con 0x7f363414d550 2024-03-28T23:29:59.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.385+0000 7f3629ffb640 1 -- 172.21.15.161:0/751408595 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f362c010040 con 0x7f363414d550 2024-03-28T23:29:59.386 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:29:59.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.387+0000 7f363b69a640 1 -- 172.21.15.161:0/751408595 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f361806b4d0 msgr2=0x7f361806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:59.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.387+0000 7f363b69a640 1 --2- 172.21.15.161:0/751408595 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f361806b4d0 0x7f361806d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f36240015d0 tx=0x7f3624009040 comp rx=0 tx=0).stop 2024-03-28T23:29:59.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.387+0000 7f363b69a640 1 -- 172.21.15.161:0/751408595 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f363414d550 msgr2=0x7f3634143bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:59.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.387+0000 7f363b69a640 1 --2- 172.21.15.161:0/751408595 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f363414d550 0x7f3634143bf0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f363406fdc0 tx=0x7f362c00f580 comp rx=0 tx=0).stop 2024-03-28T23:29:59.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.387+0000 7f363b69a640 1 -- 172.21.15.161:0/751408595 shutdown_connections 2024-03-28T23:29:59.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.387+0000 7f363b69a640 1 --2- 172.21.15.161:0/751408595 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f361806b4d0 0x7f361806d990 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.387+0000 7f363b69a640 1 --2- 172.21.15.161:0/751408595 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f363414d550 0x7f3634143bf0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.387+0000 7f363b69a640 1 --2- 172.21.15.161:0/751408595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f36340a6d40 0x7f36340bff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.387+0000 7f363b69a640 1 --2- 172.21.15.161:0/751408595 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3620004800 0x7f36341ee340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.387+0000 7f363b69a640 1 -- 172.21.15.161:0/751408595 >> 172.21.15.161:0/751408595 conn(0x7f36340c3390 msgr2=0x7f36341483a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:59.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.387+0000 7f363b69a640 1 -- 172.21.15.161:0/751408595 shutdown_connections 2024-03-28T23:29:59.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.387+0000 7f363b69a640 1 -- 172.21.15.161:0/751408595 wait complete. 2024-03-28T23:29:59.399 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T23:29:59.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.503+0000 7f18046ed640 1 Processor -- start 2024-03-28T23:29:59.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.503+0000 7f18046ed640 1 -- start start 2024-03-28T23:29:59.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.504+0000 7f18046ed640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f17fc152bc0 0x7f17fc0c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:59.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.504+0000 7f18046ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f17fc0c8a80 0x7f17fc0c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:59.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.504+0000 7f18046ed640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f17fc0c9420 0x7f17fc1589b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:59.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.504+0000 7f18046ed640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f17fc0c6f60 con 0x7f17fc0c9420 2024-03-28T23:29:59.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.504+0000 7f18046ed640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f17fc0c6120 con 0x7f17fc152bc0 2024-03-28T23:29:59.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.504+0000 7f18046ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f17fc0c6400 con 0x7f17fc0c8a80 2024-03-28T23:29:59.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.504+0000 7f1802c8a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f17fc0c9420 0x7f17fc1589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:59.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.504+0000 7f1802c8a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f17fc0c9420 0x7f17fc1589b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55328/0 (socket says 172.21.15.161:55328) 2024-03-28T23:29:59.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.504+0000 7f1802c8a640 1 -- 172.21.15.161:0/2443658809 learned_addr learned my addr 172.21.15.161:0/2443658809 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:59.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.505+0000 7f1801c88640 1 --2- 172.21.15.161:0/2443658809 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f17fc0c8a80 0x7f17fc0c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:59.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.505+0000 7f1802489640 1 --2- 172.21.15.161:0/2443658809 >> v2:172.21.15.170:3300/0 conn(0x7f17fc152bc0 0x7f17fc0c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:59.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.505+0000 7f1802c8a640 1 -- 172.21.15.161:0/2443658809 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f17fc0c8a80 msgr2=0x7f17fc0c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:59.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.505+0000 7f1802c8a640 1 --2- 172.21.15.161:0/2443658809 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f17fc0c8a80 0x7f17fc0c8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.505+0000 7f1802c8a640 1 -- 172.21.15.161:0/2443658809 >> v2:172.21.15.170:3300/0 conn(0x7f17fc152bc0 msgr2=0x7f17fc0c0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:59.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.505+0000 7f1802c8a640 1 --2- 172.21.15.161:0/2443658809 >> v2:172.21.15.170:3300/0 conn(0x7f17fc152bc0 0x7f17fc0c0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.505+0000 7f1802c8a640 1 -- 172.21.15.161:0/2443658809 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17fc0c7240 con 0x7f17fc0c9420 2024-03-28T23:29:59.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.505+0000 7f1802c8a640 1 --2- 172.21.15.161:0/2443658809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17fc0c9420 0x7f17fc1589b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f17fc0aa410 tx=0x7f17f402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2fb24f3c0844f50 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:59.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.506+0000 7f1801487640 1 -- 172.21.15.161:0/2443658809 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f402ca70 con 0x7f17fc0c9420 2024-03-28T23:29:59.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.506+0000 7f1801487640 1 -- 172.21.15.161:0/2443658809 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17f402cc10 con 0x7f17fc0c9420 2024-03-28T23:29:59.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.506+0000 7f1801487640 1 -- 172.21.15.161:0/2443658809 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f40343f0 con 0x7f17fc0c9420 2024-03-28T23:29:59.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.506+0000 7f18046ed640 1 -- 172.21.15.161:0/2443658809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17fc0c9420 msgr2=0x7f17fc1589b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:59.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.506+0000 7f18046ed640 1 --2- 172.21.15.161:0/2443658809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17fc0c9420 0x7f17fc1589b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f17fc0aa410 tx=0x7f17f402bed0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.506+0000 7f18046ed640 1 -- 172.21.15.161:0/2443658809 shutdown_connections 2024-03-28T23:29:59.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.506+0000 7f18046ed640 1 --2- 172.21.15.161:0/2443658809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17fc0c9420 0x7f17fc1589b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.506+0000 7f18046ed640 1 --2- 172.21.15.161:0/2443658809 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f17fc0c8a80 0x7f17fc0c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.506+0000 7f18046ed640 1 --2- 172.21.15.161:0/2443658809 >> v2:172.21.15.170:3300/0 conn(0x7f17fc152bc0 0x7f17fc0c0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.506+0000 7f18046ed640 1 -- 172.21.15.161:0/2443658809 >> 172.21.15.161:0/2443658809 conn(0x7f17fc0caa80 msgr2=0x7f17fc0cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:59.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.506+0000 7f18046ed640 1 -- 172.21.15.161:0/2443658809 shutdown_connections 2024-03-28T23:29:59.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.506+0000 7f18046ed640 1 -- 172.21.15.161:0/2443658809 wait complete. 2024-03-28T23:29:59.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.507+0000 7f18046ed640 1 Processor -- start 2024-03-28T23:29:59.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.507+0000 7f18046ed640 1 -- start start 2024-03-28T23:29:59.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.508+0000 7f18046ed640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17fc0c8a80 0x7f17fc1503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:59.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.508+0000 7f18046ed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17fc0c9420 0x7f17fc06c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:59.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.508+0000 7f18046ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f17fc152bc0 0x7f17fc154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:59.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.508+0000 7f18046ed640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f17fc152fa0 con 0x7f17fc0c8a80 2024-03-28T23:29:59.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.508+0000 7f18046ed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f17fc158ef0 con 0x7f17fc0c9420 2024-03-28T23:29:59.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.508+0000 7f18046ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f17fc154a50 con 0x7f17fc152bc0 2024-03-28T23:29:59.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.508+0000 7f1802489640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17fc0c8a80 0x7f17fc1503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:59.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.508+0000 7f1802489640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17fc0c8a80 0x7f17fc1503e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55342/0 (socket says 172.21.15.161:55342) 2024-03-28T23:29:59.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.508+0000 7f1802489640 1 -- 172.21.15.161:0/3622278509 learned_addr learned my addr 172.21.15.161:0/3622278509 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:59.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.508+0000 7f1801c88640 1 --2- 172.21.15.161:0/3622278509 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17fc0c9420 0x7f17fc06c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:59.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.508+0000 7f1802c8a640 1 --2- 172.21.15.161:0/3622278509 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f17fc152bc0 0x7f17fc154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:59.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.508+0000 7f1802489640 1 -- 172.21.15.161:0/3622278509 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f17fc152bc0 msgr2=0x7f17fc154ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:59.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.508+0000 7f1802489640 1 --2- 172.21.15.161:0/3622278509 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f17fc152bc0 0x7f17fc154ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.509+0000 7f1802489640 1 -- 172.21.15.161:0/3622278509 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17fc0c9420 msgr2=0x7f17fc06c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:59.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.509+0000 7f1802489640 1 --2- 172.21.15.161:0/3622278509 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17fc0c9420 0x7f17fc06c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.509+0000 7f1802489640 1 -- 172.21.15.161:0/3622278509 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17fc06ceb0 con 0x7f17fc0c8a80 2024-03-28T23:29:59.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.509+0000 7f1802489640 1 --2- 172.21.15.161:0/3622278509 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17fc0c8a80 0x7f17fc1503e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f17ec00c7f0 tx=0x7f17ec00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:59.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.509+0000 7f17faffd640 1 -- 172.21.15.161:0/3622278509 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17ec015070 con 0x7f17fc0c8a80 2024-03-28T23:29:59.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.509+0000 7f17faffd640 1 -- 172.21.15.161:0/3622278509 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17ec004220 con 0x7f17fc0c8a80 2024-03-28T23:29:59.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.509+0000 7f18046ed640 1 -- 172.21.15.161:0/3622278509 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17fc150a70 con 0x7f17fc0c8a80 2024-03-28T23:29:59.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.509+0000 7f17faffd640 1 -- 172.21.15.161:0/3622278509 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17ec007620 con 0x7f17fc0c8a80 2024-03-28T23:29:59.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.509+0000 7f18046ed640 1 -- 172.21.15.161:0/3622278509 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17fc151030 con 0x7f17fc0c8a80 2024-03-28T23:29:59.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.510+0000 7f18046ed640 1 -- 172.21.15.161:0/3622278509 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17fc156570 con 0x7f17fc0c8a80 2024-03-28T23:29:59.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.510+0000 7f17faffd640 1 -- 172.21.15.161:0/3622278509 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f17ec0079d0 con 0x7f17fc0c8a80 2024-03-28T23:29:59.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.511+0000 7f17faffd640 1 --2- 172.21.15.161:0/3622278509 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f17e806b4d0 0x7f17e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:59.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.511+0000 7f17faffd640 1 -- 172.21.15.161:0/3622278509 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f17ec08a590 con 0x7f17fc0c8a80 2024-03-28T23:29:59.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.511+0000 7f1801c88640 1 --2- 172.21.15.161:0/3622278509 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f17e806b4d0 0x7f17e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:59.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.511+0000 7f1801c88640 1 --2- 172.21.15.161:0/3622278509 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f17e806b4d0 0x7f17e806d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f17fc152560 tx=0x7f17f0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:59.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.515+0000 7f17faffd640 1 -- 172.21.15.161:0/3622278509 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f17fc156570 con 0x7f17fc0c8a80 2024-03-28T23:29:59.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.696+0000 7f18046ed640 1 -- 172.21.15.161:0/3622278509 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f17fc0ab7a0 con 0x7f17fc0c8a80 2024-03-28T23:29:59.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.696+0000 7f17faffd640 1 -- 172.21.15.161:0/3622278509 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f17fc0ab7a0 con 0x7f17fc0c8a80 2024-03-28T23:29:59.697 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:29:59.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.699+0000 7f18046ed640 1 -- 172.21.15.161:0/3622278509 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f17e806b4d0 msgr2=0x7f17e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:59.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.699+0000 7f18046ed640 1 --2- 172.21.15.161:0/3622278509 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f17e806b4d0 0x7f17e806d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f17fc152560 tx=0x7f17f0008040 comp rx=0 tx=0).stop 2024-03-28T23:29:59.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.699+0000 7f18046ed640 1 -- 172.21.15.161:0/3622278509 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17fc0c8a80 msgr2=0x7f17fc1503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:59.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.699+0000 7f18046ed640 1 --2- 172.21.15.161:0/3622278509 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17fc0c8a80 0x7f17fc1503e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f17ec00c7f0 tx=0x7f17ec00ccb0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.699+0000 7f18046ed640 1 -- 172.21.15.161:0/3622278509 shutdown_connections 2024-03-28T23:29:59.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.699+0000 7f18046ed640 1 --2- 172.21.15.161:0/3622278509 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f17e806b4d0 0x7f17e806d990 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.699+0000 7f18046ed640 1 --2- 172.21.15.161:0/3622278509 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f17fc152bc0 0x7f17fc154ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.699+0000 7f18046ed640 1 --2- 172.21.15.161:0/3622278509 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17fc0c9420 0x7f17fc06c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.699+0000 7f18046ed640 1 --2- 172.21.15.161:0/3622278509 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17fc0c8a80 0x7f17fc1503e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.699+0000 7f18046ed640 1 -- 172.21.15.161:0/3622278509 >> 172.21.15.161:0/3622278509 conn(0x7f17fc0caa80 msgr2=0x7f17fc0c9800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:59.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.699+0000 7f18046ed640 1 -- 172.21.15.161:0/3622278509 shutdown_connections 2024-03-28T23:29:59.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.699+0000 7f18046ed640 1 -- 172.21.15.161:0/3622278509 wait complete. 2024-03-28T23:29:59.710 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T23:29:59.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.824+0000 7f5e524a4640 1 Processor -- start 2024-03-28T23:29:59.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.825+0000 7f5e524a4640 1 -- start start 2024-03-28T23:29:59.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.826+0000 7f5e524a4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e4c152890 0x7f5e4c0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:59.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.826+0000 7f5e524a4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5e4c0c8ed0 0x7f5e4c0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:59.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.826+0000 7f5e524a4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e4c158a30 con 0x7f5e4c0c5950 2024-03-28T23:29:59.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.826+0000 7f5e524a4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5e4c0c5db0 con 0x7f5e4c0c8ed0 2024-03-28T23:29:59.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.826+0000 7f5e524a4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5e4c0c8480 con 0x7f5e4c152890 2024-03-28T23:29:59.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.826+0000 7f5e4b7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5e4c0c8ed0 0x7f5e4c0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:59.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.826+0000 7f5e50a41640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f5e4c0c5950 0x7f5e4c1553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52358/0 (socket says 172.21.15.161:52358) 2024-03-28T23:29:59.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.826+0000 7f5e50a41640 1 -- 172.21.15.161:0/1260943468 learned_addr learned my addr 172.21.15.161:0/1260943468 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:29:59.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.826+0000 7f5e4bfff640 1 --2- 172.21.15.161:0/1260943468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e4c152890 0x7f5e4c0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:59.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.826+0000 7f5e4b7fe640 1 -- 172.21.15.161:0/1260943468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e4c152890 msgr2=0x7f5e4c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:29:59.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.826+0000 7f5e4b7fe640 1 --2- 172.21.15.161:0/1260943468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e4c152890 0x7f5e4c0c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.826+0000 7f5e4b7fe640 1 -- 172.21.15.161:0/1260943468 >> v1:172.21.15.161:6789/0 conn(0x7f5e4c0c5950 legacy=0x7f5e4c1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:59.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.826+0000 7f5e4b7fe640 1 -- 172.21.15.161:0/1260943468 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e4c158d10 con 0x7f5e4c0c8ed0 2024-03-28T23:29:59.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.826+0000 7f5e4affd640 1 -- 172.21.15.161:0/1260943468 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 89669296 0 0) 0x7f5e4c158a30 con 0x7f5e4c0c5950 2024-03-28T23:29:59.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.827+0000 7f5e4b7fe640 1 --2- 172.21.15.161:0/1260943468 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e4c0c8ed0 0x7f5e4c0c5410 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c009790 tx=0x7f5e3c02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3feee688bb07837b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:59.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.827+0000 7f5e4affd640 1 -- 172.21.15.161:0/1260943468 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c039070 con 0x7f5e4c0c8ed0 2024-03-28T23:29:59.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.827+0000 7f5e4affd640 1 -- 172.21.15.161:0/1260943468 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e3c0316f0 con 0x7f5e4c0c8ed0 2024-03-28T23:29:59.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.827+0000 7f5e4affd640 1 -- 172.21.15.161:0/1260943468 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c031ac0 con 0x7f5e4c0c8ed0 2024-03-28T23:29:59.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.828+0000 7f5e524a4640 1 -- 172.21.15.161:0/1260943468 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e4c0c8ed0 msgr2=0x7f5e4c0c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:59.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.828+0000 7f5e524a4640 1 --2- 172.21.15.161:0/1260943468 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e4c0c8ed0 0x7f5e4c0c5410 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c009790 tx=0x7f5e3c02b790 comp rx=0 tx=0).stop 2024-03-28T23:29:59.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.828+0000 7f5e524a4640 1 -- 172.21.15.161:0/1260943468 shutdown_connections 2024-03-28T23:29:59.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.828+0000 7f5e524a4640 1 --2- 172.21.15.161:0/1260943468 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e4c0c8ed0 0x7f5e4c0c5410 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.828+0000 7f5e524a4640 1 --2- 172.21.15.161:0/1260943468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e4c152890 0x7f5e4c0c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.828+0000 7f5e524a4640 1 -- 172.21.15.161:0/1260943468 >> 172.21.15.161:0/1260943468 conn(0x7f5e4c0ac620 msgr2=0x7f5e4c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:29:59.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.828+0000 7f5e524a4640 1 -- 172.21.15.161:0/1260943468 shutdown_connections 2024-03-28T23:29:59.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.828+0000 7f5e524a4640 1 -- 172.21.15.161:0/1260943468 wait complete. 2024-03-28T23:29:59.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.829+0000 7f5e524a4640 1 Processor -- start 2024-03-28T23:29:59.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.830+0000 7f5e524a4640 1 -- start start 2024-03-28T23:29:59.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.831+0000 7f5e524a4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e4c0c5950 0x7f5e4c147080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:59.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.831+0000 7f5e524a4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e4c0c8ed0 0x7f5e4c1e7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:59.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.831+0000 7f5e524a4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e4c152890 0x7f5e4c1e84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:59.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.831+0000 7f5e524a4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5e4c1592d0 con 0x7f5e4c0c8ed0 2024-03-28T23:29:59.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.831+0000 7f5e524a4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5e4c159640 con 0x7f5e4c152890 2024-03-28T23:29:59.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.831+0000 7f5e524a4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5e4c152c70 con 0x7f5e4c0c5950 2024-03-28T23:29:59.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.831+0000 7f5e4b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e4c0c8ed0 0x7f5e4c1e7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:59.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.831+0000 7f5e4b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e4c0c8ed0 0x7f5e4c1e7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55356/0 (socket says 172.21.15.161:55356) 2024-03-28T23:29:59.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.831+0000 7f5e4b7fe640 1 -- 172.21.15.161:0/395942019 learned_addr learned my addr 172.21.15.161:0/395942019 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:29:59.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.831+0000 7f5e4bfff640 1 --2- 172.21.15.161:0/395942019 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e4c0c5950 0x7f5e4c147080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:59.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.831+0000 7f5e50a41640 1 --2- 172.21.15.161:0/395942019 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e4c152890 0x7f5e4c1e84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:59.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.831+0000 7f5e4b7fe640 1 -- 172.21.15.161:0/395942019 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e4c0c5950 msgr2=0x7f5e4c147080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:59.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.831+0000 7f5e4b7fe640 1 --2- 172.21.15.161:0/395942019 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e4c0c5950 0x7f5e4c147080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.831+0000 7f5e4b7fe640 1 -- 172.21.15.161:0/395942019 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e4c152890 msgr2=0x7f5e4c1e84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:29:59.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.831+0000 7f5e4b7fe640 1 --2- 172.21.15.161:0/395942019 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e4c152890 0x7f5e4c1e84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:29:59.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.831+0000 7f5e4b7fe640 1 -- 172.21.15.161:0/395942019 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e4c1507a0 con 0x7f5e4c0c8ed0 2024-03-28T23:29:59.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.831+0000 7f5e4b7fe640 1 --2- 172.21.15.161:0/395942019 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e4c0c8ed0 0x7f5e4c1e7f70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c009790 tx=0x7f5e3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:59.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.832+0000 7f5e48ff9640 1 -- 172.21.15.161:0/395942019 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c039040 con 0x7f5e4c0c8ed0 2024-03-28T23:29:59.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.832+0000 7f5e48ff9640 1 -- 172.21.15.161:0/395942019 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e3c004020 con 0x7f5e4c0c8ed0 2024-03-28T23:29:59.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.832+0000 7f5e48ff9640 1 -- 172.21.15.161:0/395942019 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c004410 con 0x7f5e4c0c8ed0 2024-03-28T23:29:59.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.832+0000 7f5e524a4640 1 -- 172.21.15.161:0/395942019 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e4c1ee390 con 0x7f5e4c0c8ed0 2024-03-28T23:29:59.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.832+0000 7f5e524a4640 1 -- 172.21.15.161:0/395942019 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e4c1ee950 con 0x7f5e4c0c8ed0 2024-03-28T23:29:59.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.833+0000 7f5e524a4640 1 -- 172.21.15.161:0/395942019 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e4c04f4b0 con 0x7f5e4c0c8ed0 2024-03-28T23:29:59.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.833+0000 7f5e48ff9640 1 -- 172.21.15.161:0/395942019 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5e3c03c070 con 0x7f5e4c0c8ed0 2024-03-28T23:29:59.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.833+0000 7f5e48ff9640 1 --2- 172.21.15.161:0/395942019 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5e3806b500 0x7f5e3806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:29:59.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.834+0000 7f5e48ff9640 1 -- 172.21.15.161:0/395942019 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5e3c0abd80 con 0x7f5e4c0c8ed0 2024-03-28T23:29:59.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.834+0000 7f5e4bfff640 1 --2- 172.21.15.161:0/395942019 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5e3806b500 0x7f5e3806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:29:59.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.834+0000 7f5e4bfff640 1 --2- 172.21.15.161:0/395942019 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5e3806b500 0x7f5e3806d9c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5e40006fd0 tx=0x7f5e40008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:29:59.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:29:59.838+0000 7f5e48ff9640 1 -- 172.21.15.161:0/395942019 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5e3c04c030 con 0x7f5e4c0c8ed0 2024-03-28T23:30:00.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:00.028+0000 7f5e524a4640 1 -- 172.21.15.161:0/395942019 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5e4c14eac0 con 0x7f5e4c0c8ed0 2024-03-28T23:30:00.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:00.029+0000 7f5e48ff9640 1 -- 172.21.15.161:0/395942019 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7f5e3c043030 con 0x7f5e4c0c8ed0 2024-03-28T23:30:00.030 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-28T23:30:00.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:00.031+0000 7f5e524a4640 1 -- 172.21.15.161:0/395942019 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5e3806b500 msgr2=0x7f5e3806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:00.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:00.031+0000 7f5e524a4640 1 --2- 172.21.15.161:0/395942019 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5e3806b500 0x7f5e3806d9c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5e40006fd0 tx=0x7f5e40008040 comp rx=0 tx=0).stop 2024-03-28T23:30:00.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:00.031+0000 7f5e524a4640 1 -- 172.21.15.161:0/395942019 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e4c0c8ed0 msgr2=0x7f5e4c1e7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:00.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:00.031+0000 7f5e524a4640 1 --2- 172.21.15.161:0/395942019 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e4c0c8ed0 0x7f5e4c1e7f70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c009790 tx=0x7f5e3c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:00.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:00.031+0000 7f5e524a4640 1 -- 172.21.15.161:0/395942019 shutdown_connections 2024-03-28T23:30:00.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:00.031+0000 7f5e524a4640 1 --2- 172.21.15.161:0/395942019 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5e3806b500 0x7f5e3806d9c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:00.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:00.031+0000 7f5e524a4640 1 --2- 172.21.15.161:0/395942019 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e4c0c8ed0 0x7f5e4c1e7f70 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:00.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:00.031+0000 7f5e524a4640 1 --2- 172.21.15.161:0/395942019 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e4c152890 0x7f5e4c1e84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:00.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:00.031+0000 7f5e524a4640 1 --2- 172.21.15.161:0/395942019 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e4c0c5950 0x7f5e4c147080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:00.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:00.031+0000 7f5e524a4640 1 -- 172.21.15.161:0/395942019 >> 172.21.15.161:0/395942019 conn(0x7f5e4c0ac620 msgr2=0x7f5e4c0c8660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:00.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:00.031+0000 7f5e524a4640 1 -- 172.21.15.161:0/395942019 shutdown_connections 2024-03-28T23:30:00.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:00.031+0000 7f5e524a4640 1 -- 172.21.15.161:0/395942019 wait complete. 2024-03-28T23:30:00.042 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi161.front.sepia.ceph.com... 2024-03-28T23:30:00.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:00.043 DEBUG:teuthology.orchestra.run.smithi161:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-28T23:30:00.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi161.stderr:ceph-fuse[42482]: fuse finished with error 0 and tester_r 0 2024-03-28T23:30:00.071 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:30:06.173 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi161.front.sepia.ceph.com 2024-03-28T23:30:06.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:06.174 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-28T23:30:06.197 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:30:06.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.340+0000 7f5929f56640 1 Processor -- start 2024-03-28T23:30:06.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.341+0000 7f5929f56640 1 -- start start 2024-03-28T23:30:06.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.341+0000 7f5929f56640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5924152890 0x7f59240c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:06.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.341+0000 7f5929f56640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f59240c8ed0 0x7f59240c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:06.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.341+0000 7f5929f56640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5924158a30 con 0x7f59240c5950 2024-03-28T23:30:06.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.341+0000 7f5929f56640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f59240c8480 con 0x7f59240c8ed0 2024-03-28T23:30:06.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.341+0000 7f5929f56640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f5924159640 con 0x7f5924152890 2024-03-28T23:30:06.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.341+0000 7f5922ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f59240c8ed0 0x7f59240c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:06.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.341+0000 7f5922ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f59240c8ed0 0x7f59240c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45538/0 (socket says 172.21.15.161:45538) 2024-03-28T23:30:06.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.341+0000 7f5922ffd640 1 -- 172.21.15.161:0/390079489 learned_addr learned my addr 172.21.15.161:0/390079489 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:06.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.342+0000 7f59237fe640 1 --2- 172.21.15.161:0/390079489 >> v2:172.21.15.170:3300/0 conn(0x7f5924152890 0x7f59240c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:06.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.342+0000 7f5922ffd640 1 -- 172.21.15.161:0/390079489 >> v2:172.21.15.170:3300/0 conn(0x7f5924152890 msgr2=0x7f59240c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:06.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.342+0000 7f5922ffd640 1 --2- 172.21.15.161:0/390079489 >> v2:172.21.15.170:3300/0 conn(0x7f5924152890 0x7f59240c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:06.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.342+0000 7f5922ffd640 1 -- 172.21.15.161:0/390079489 >> v1:172.21.15.170:6789/0 conn(0x7f59240c5950 legacy=0x7f59241553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:06.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.342+0000 7f5922ffd640 1 -- 172.21.15.161:0/390079489 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5924158d10 con 0x7f59240c8ed0 2024-03-28T23:30:06.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.342+0000 7f5922ffd640 1 --2- 172.21.15.161:0/390079489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59240c8ed0 0x7f59240c5410 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f592408cc80 tx=0x7f591802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a90b61c1b054495 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:06.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.343+0000 7f59227fc640 1 -- 172.21.15.161:0/390079489 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5918037070 con 0x7f59240c8ed0 2024-03-28T23:30:06.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.343+0000 7f59227fc640 1 -- 172.21.15.161:0/390079489 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5918004230 con 0x7f59240c8ed0 2024-03-28T23:30:06.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.343+0000 7f59227fc640 1 -- 172.21.15.161:0/390079489 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5918003610 con 0x7f59240c8ed0 2024-03-28T23:30:06.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.343+0000 7f5929f56640 1 -- 172.21.15.161:0/390079489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59240c8ed0 msgr2=0x7f59240c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:06.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.343+0000 7f5929f56640 1 --2- 172.21.15.161:0/390079489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59240c8ed0 0x7f59240c5410 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f592408cc80 tx=0x7f591802b860 comp rx=0 tx=0).stop 2024-03-28T23:30:06.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.343+0000 7f5929f56640 1 -- 172.21.15.161:0/390079489 shutdown_connections 2024-03-28T23:30:06.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.343+0000 7f5929f56640 1 --2- 172.21.15.161:0/390079489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59240c8ed0 0x7f59240c5410 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:06.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.343+0000 7f5929f56640 1 --2- 172.21.15.161:0/390079489 >> v2:172.21.15.170:3300/0 conn(0x7f5924152890 0x7f59240c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:06.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.343+0000 7f5929f56640 1 -- 172.21.15.161:0/390079489 >> 172.21.15.161:0/390079489 conn(0x7f59240ac620 msgr2=0x7f59240aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:06.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.343+0000 7f5929f56640 1 -- 172.21.15.161:0/390079489 shutdown_connections 2024-03-28T23:30:06.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.343+0000 7f5929f56640 1 -- 172.21.15.161:0/390079489 wait complete. 2024-03-28T23:30:06.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.344+0000 7f5929f56640 1 Processor -- start 2024-03-28T23:30:06.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.345+0000 7f5929f56640 1 -- start start 2024-03-28T23:30:06.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.345+0000 7f5929f56640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59240c5950 0x7f59241eb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:06.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.345+0000 7f5929f56640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59240c8ed0 0x7f5924157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:06.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.345+0000 7f5929f56640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5924152890 0x7f592414d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:06.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.345+0000 7f5929f56640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5924152c70 con 0x7f59240c8ed0 2024-03-28T23:30:06.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.345+0000 7f5929f56640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f59241592d0 con 0x7f5924152890 2024-03-28T23:30:06.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.345+0000 7f5929f56640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5924152df0 con 0x7f59240c5950 2024-03-28T23:30:06.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.345+0000 7f5922ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59240c8ed0 0x7f5924157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:06.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.345+0000 7f5922ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59240c8ed0 0x7f5924157830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45550/0 (socket says 172.21.15.161:45550) 2024-03-28T23:30:06.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.345+0000 7f5922ffd640 1 -- 172.21.15.161:0/3283668352 learned_addr learned my addr 172.21.15.161:0/3283668352 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:06.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.346+0000 7f5923fff640 1 --2- 172.21.15.161:0/3283668352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5924152890 0x7f592414d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:06.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.346+0000 7f59237fe640 1 --2- 172.21.15.161:0/3283668352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59240c5950 0x7f59241eb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:06.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.346+0000 7f5922ffd640 1 -- 172.21.15.161:0/3283668352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59240c5950 msgr2=0x7f59241eb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:06.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.346+0000 7f5922ffd640 1 --2- 172.21.15.161:0/3283668352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59240c5950 0x7f59241eb4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:06.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.346+0000 7f5922ffd640 1 -- 172.21.15.161:0/3283668352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5924152890 msgr2=0x7f592414d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:06.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.346+0000 7f5922ffd640 1 --2- 172.21.15.161:0/3283668352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5924152890 0x7f592414d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:06.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.346+0000 7f5922ffd640 1 -- 172.21.15.161:0/3283668352 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f592414e250 con 0x7f59240c8ed0 2024-03-28T23:30:06.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.346+0000 7f5923fff640 1 --2- 172.21.15.161:0/3283668352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5924152890 0x7f592414d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:06.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.346+0000 7f5922ffd640 1 --2- 172.21.15.161:0/3283668352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59240c8ed0 0x7f5924157830 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f591802b8a0 tx=0x7f59180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:06.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.346+0000 7f590bfff640 1 -- 172.21.15.161:0/3283668352 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5918037050 con 0x7f59240c8ed0 2024-03-28T23:30:06.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.346+0000 7f590bfff640 1 -- 172.21.15.161:0/3283668352 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5918043070 con 0x7f59240c8ed0 2024-03-28T23:30:06.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.346+0000 7f5929f56640 1 -- 172.21.15.161:0/3283668352 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f592414e530 con 0x7f59240c8ed0 2024-03-28T23:30:06.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.346+0000 7f590bfff640 1 -- 172.21.15.161:0/3283668352 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5918006040 con 0x7f59240c8ed0 2024-03-28T23:30:06.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.346+0000 7f5929f56640 1 -- 172.21.15.161:0/3283668352 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f592414ce10 con 0x7f59240c8ed0 2024-03-28T23:30:06.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.347+0000 7f5929f56640 1 -- 172.21.15.161:0/3283668352 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f592404f4b0 con 0x7f59240c8ed0 2024-03-28T23:30:06.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.347+0000 7f590bfff640 1 -- 172.21.15.161:0/3283668352 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f59180061e0 con 0x7f59240c8ed0 2024-03-28T23:30:06.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.348+0000 7f590bfff640 1 --2- 172.21.15.161:0/3283668352 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f591006b4d0 0x7f591006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:06.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.348+0000 7f59237fe640 1 --2- 172.21.15.161:0/3283668352 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f591006b4d0 0x7f591006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:06.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.348+0000 7f590bfff640 1 -- 172.21.15.161:0/3283668352 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f59180ac7c0 con 0x7f59240c8ed0 2024-03-28T23:30:06.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.348+0000 7f59237fe640 1 --2- 172.21.15.161:0/3283668352 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f591006b4d0 0x7f591006d990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5914006fd0 tx=0x7f5914008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:06.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.352+0000 7f590bfff640 1 -- 172.21.15.161:0/3283668352 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f591807b420 con 0x7f59240c8ed0 2024-03-28T23:30:06.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.632+0000 7f5929f56640 1 -- 172.21.15.161:0/3283668352 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5924150580 con 0x7f59240c8ed0 2024-03-28T23:30:06.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.633+0000 7f590bfff640 1 -- 172.21.15.161:0/3283668352 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7f591807ad20 con 0x7f59240c8ed0 2024-03-28T23:30:06.634 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:06.635 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":24,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:29:50.946843+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:30:06.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.635+0000 7f5929f56640 1 -- 172.21.15.161:0/3283668352 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f591006b4d0 msgr2=0x7f591006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:06.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.635+0000 7f5929f56640 1 --2- 172.21.15.161:0/3283668352 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f591006b4d0 0x7f591006d990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5914006fd0 tx=0x7f5914008040 comp rx=0 tx=0).stop 2024-03-28T23:30:06.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.635+0000 7f5929f56640 1 -- 172.21.15.161:0/3283668352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59240c8ed0 msgr2=0x7f5924157830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:06.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.635+0000 7f5929f56640 1 --2- 172.21.15.161:0/3283668352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59240c8ed0 0x7f5924157830 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f591802b8a0 tx=0x7f59180046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:06.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.635+0000 7f5929f56640 1 -- 172.21.15.161:0/3283668352 shutdown_connections 2024-03-28T23:30:06.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.635+0000 7f5929f56640 1 --2- 172.21.15.161:0/3283668352 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f591006b4d0 0x7f591006d990 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:06.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.635+0000 7f5929f56640 1 --2- 172.21.15.161:0/3283668352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59240c8ed0 0x7f5924157830 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:06.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.635+0000 7f5929f56640 1 --2- 172.21.15.161:0/3283668352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5924152890 0x7f592414d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:06.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.635+0000 7f5929f56640 1 --2- 172.21.15.161:0/3283668352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59240c5950 0x7f59241eb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:06.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.635+0000 7f5929f56640 1 -- 172.21.15.161:0/3283668352 >> 172.21.15.161:0/3283668352 conn(0x7f59240ac620 msgr2=0x7f59240c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:06.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.635+0000 7f5929f56640 1 -- 172.21.15.161:0/3283668352 shutdown_connections 2024-03-28T23:30:06.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.635+0000 7f5929f56640 1 -- 172.21.15.161:0/3283668352 wait complete. 2024-03-28T23:30:06.648 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:30:06.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.770+0000 7f839e7bf640 1 Processor -- start 2024-03-28T23:30:06.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.770+0000 7f839e7bf640 1 -- start start 2024-03-28T23:30:06.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.771+0000 7f839e7bf640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f839806c160 0x7f839806e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:06.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.771+0000 7f839e7bf640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f83980c1cf0 0x7f8398153bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:06.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.771+0000 7f839e7bf640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f839806c9a0 con 0x7f839806c540 2024-03-28T23:30:06.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.771+0000 7f839e7bf640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f83981540f0 con 0x7f83980c1cf0 2024-03-28T23:30:06.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.771+0000 7f839e7bf640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f839806e290 con 0x7f839806c160 2024-03-28T23:30:06.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.771+0000 7f8397fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f839806c160 0x7f839806e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:06.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.771+0000 7f839cd5c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f83980c1cf0 0x7f8398153bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:06.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.771+0000 7f8397fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f839806c160 0x7f839806e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:33210/0 (socket says 172.21.15.161:33210) 2024-03-28T23:30:06.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.771+0000 7f839cd5c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f83980c1cf0 0x7f8398153bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:35172/0 (socket says 172.21.15.161:35172) 2024-03-28T23:30:06.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.771+0000 7f8397fff640 1 -- 172.21.15.161:0/4288246601 learned_addr learned my addr 172.21.15.161:0/4288246601 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:06.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.771+0000 7f83977fe640 1 --1- 172.21.15.161:0/4288246601 >> v1:172.21.15.170:6789/0 conn(0x7f839806c540 0x7f8398150560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:46950/0 (socket says 172.21.15.161:46950) 2024-03-28T23:30:06.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.772+0000 7f8397fff640 1 -- 172.21.15.161:0/4288246601 >> v2:172.21.15.170:3300/0 conn(0x7f83980c1cf0 msgr2=0x7f8398153bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:06.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.772+0000 7f8397fff640 1 --2- 172.21.15.161:0/4288246601 >> v2:172.21.15.170:3300/0 conn(0x7f83980c1cf0 0x7f8398153bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:06.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.772+0000 7f8397fff640 1 -- 172.21.15.161:0/4288246601 >> v1:172.21.15.170:6789/0 conn(0x7f839806c540 legacy=0x7f8398150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:06.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.772+0000 7f8397fff640 1 -- 172.21.15.161:0/4288246601 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f839806f650 con 0x7f839806c160 2024-03-28T23:30:06.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.772+0000 7f839cd5c640 1 --2- 172.21.15.161:0/4288246601 >> v2:172.21.15.170:3300/0 conn(0x7f83980c1cf0 0x7f8398153bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:06.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.773+0000 7f8397fff640 1 --2- 172.21.15.161:0/4288246601 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f839806c160 0x7f839806e720 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f83880098a0 tx=0x7f838802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62484297e2f87359 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:06.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.773+0000 7f8396ffd640 1 -- 172.21.15.161:0/4288246601 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8388035020 con 0x7f839806c160 2024-03-28T23:30:06.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.773+0000 7f8396ffd640 1 -- 172.21.15.161:0/4288246601 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8388031640 con 0x7f839806c160 2024-03-28T23:30:06.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.773+0000 7f8396ffd640 1 -- 172.21.15.161:0/4288246601 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838802d040 con 0x7f839806c160 2024-03-28T23:30:06.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.773+0000 7f839e7bf640 1 -- 172.21.15.161:0/4288246601 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f839806c160 msgr2=0x7f839806e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:06.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.773+0000 7f839e7bf640 1 --2- 172.21.15.161:0/4288246601 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f839806c160 0x7f839806e720 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f83880098a0 tx=0x7f838802b8f0 comp rx=0 tx=0).stop 2024-03-28T23:30:06.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.773+0000 7f839e7bf640 1 -- 172.21.15.161:0/4288246601 shutdown_connections 2024-03-28T23:30:06.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.773+0000 7f839e7bf640 1 --2- 172.21.15.161:0/4288246601 >> v2:172.21.15.170:3300/0 conn(0x7f83980c1cf0 0x7f8398153bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:06.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.773+0000 7f839e7bf640 1 --2- 172.21.15.161:0/4288246601 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f839806c160 0x7f839806e720 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:06.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.773+0000 7f839e7bf640 1 -- 172.21.15.161:0/4288246601 >> 172.21.15.161:0/4288246601 conn(0x7f83980c3be0 msgr2=0x7f83980c3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:06.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.773+0000 7f839e7bf640 1 -- 172.21.15.161:0/4288246601 shutdown_connections 2024-03-28T23:30:06.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.773+0000 7f839e7bf640 1 -- 172.21.15.161:0/4288246601 wait complete. 2024-03-28T23:30:06.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.774+0000 7f839e7bf640 1 Processor -- start 2024-03-28T23:30:06.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.775+0000 7f839e7bf640 1 -- start start 2024-03-28T23:30:06.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.775+0000 7f839e7bf640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f839806c160 0x7f83981dfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:06.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.775+0000 7f839e7bf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f839806c540 0x7f83980c0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:06.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.775+0000 7f839e7bf640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83980c1cf0 0x7f83981ea8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:06.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.775+0000 7f839e7bf640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f839806dc40 con 0x7f839806c540 2024-03-28T23:30:06.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.775+0000 7f839e7bf640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f83980c2180 con 0x7f839806c160 2024-03-28T23:30:06.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.775+0000 7f839e7bf640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f839806df20 con 0x7f83980c1cf0 2024-03-28T23:30:06.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.775+0000 7f83977fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f839806c540 0x7f83980c0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:06.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.775+0000 7f83977fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f839806c540 0x7f83980c0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45562/0 (socket says 172.21.15.161:45562) 2024-03-28T23:30:06.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.775+0000 7f83977fe640 1 -- 172.21.15.161:0/2767467665 learned_addr learned my addr 172.21.15.161:0/2767467665 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:06.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.775+0000 7f839cd5c640 1 --2- 172.21.15.161:0/2767467665 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83980c1cf0 0x7f83981ea8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:06.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.775+0000 7f8397fff640 1 --2- 172.21.15.161:0/2767467665 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f839806c160 0x7f83981dfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:06.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.776+0000 7f83977fe640 1 -- 172.21.15.161:0/2767467665 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83980c1cf0 msgr2=0x7f83981ea8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:06.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.776+0000 7f83977fe640 1 --2- 172.21.15.161:0/2767467665 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83980c1cf0 0x7f83981ea8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:06.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.776+0000 7f83977fe640 1 -- 172.21.15.161:0/2767467665 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f839806c160 msgr2=0x7f83981dfdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:06.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.776+0000 7f83977fe640 1 --2- 172.21.15.161:0/2767467665 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f839806c160 0x7f83981dfdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:06.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.776+0000 7f83977fe640 1 -- 172.21.15.161:0/2767467665 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83980abe30 con 0x7f839806c540 2024-03-28T23:30:06.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.776+0000 7f83977fe640 1 --2- 172.21.15.161:0/2767467665 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f839806c540 0x7f83980c0780 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f838c00c400 tx=0x7f838c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:06.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.776+0000 7f8394ff9640 1 -- 172.21.15.161:0/2767467665 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c0074e0 con 0x7f839806c540 2024-03-28T23:30:06.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.776+0000 7f8394ff9640 1 -- 172.21.15.161:0/2767467665 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f838c007680 con 0x7f839806c540 2024-03-28T23:30:06.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.776+0000 7f839e7bf640 1 -- 172.21.15.161:0/2767467665 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83980aad40 con 0x7f839806c540 2024-03-28T23:30:06.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.776+0000 7f8394ff9640 1 -- 172.21.15.161:0/2767467665 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c004a00 con 0x7f839806c540 2024-03-28T23:30:06.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.776+0000 7f839e7bf640 1 -- 172.21.15.161:0/2767467665 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83980c0cc0 con 0x7f839806c540 2024-03-28T23:30:06.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.777+0000 7f839e7bf640 1 -- 172.21.15.161:0/2767467665 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f839814a550 con 0x7f839806c540 2024-03-28T23:30:06.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.778+0000 7f8394ff9640 1 -- 172.21.15.161:0/2767467665 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f838c01a020 con 0x7f839806c540 2024-03-28T23:30:06.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.778+0000 7f8394ff9640 1 --2- 172.21.15.161:0/2767467665 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f838406b500 0x7f838406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:06.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.778+0000 7f8397fff640 1 --2- 172.21.15.161:0/2767467665 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f838406b500 0x7f838406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:06.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.778+0000 7f8394ff9640 1 -- 172.21.15.161:0/2767467665 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f838c089b10 con 0x7f839806c540 2024-03-28T23:30:06.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.779+0000 7f8397fff640 1 --2- 172.21.15.161:0/2767467665 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f838406b500 0x7f838406d9c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8388036000 tx=0x7f838803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:06.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.783+0000 7f8394ff9640 1 -- 172.21.15.161:0/2767467665 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f838c0586e0 con 0x7f839806c540 2024-03-28T23:30:06.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.962+0000 7f839e7bf640 1 -- 172.21.15.161:0/2767467665 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f839804f4b0 con 0x7f839806c540 2024-03-28T23:30:06.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.963+0000 7f8394ff9640 1 -- 172.21.15.161:0/2767467665 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7f838c057fe0 con 0x7f839806c540 2024-03-28T23:30:06.964 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:06.964 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":24,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:29:50.946843+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:30:06.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.965+0000 7f839e7bf640 1 -- 172.21.15.161:0/2767467665 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f838406b500 msgr2=0x7f838406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:06.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.965+0000 7f839e7bf640 1 --2- 172.21.15.161:0/2767467665 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f838406b500 0x7f838406d9c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8388036000 tx=0x7f838803c040 comp rx=0 tx=0).stop 2024-03-28T23:30:06.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.965+0000 7f839e7bf640 1 -- 172.21.15.161:0/2767467665 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f839806c540 msgr2=0x7f83980c0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:06.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.965+0000 7f839e7bf640 1 --2- 172.21.15.161:0/2767467665 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f839806c540 0x7f83980c0780 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f838c00c400 tx=0x7f838c00c800 comp rx=0 tx=0).stop 2024-03-28T23:30:06.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.965+0000 7f839e7bf640 1 -- 172.21.15.161:0/2767467665 shutdown_connections 2024-03-28T23:30:06.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.965+0000 7f839e7bf640 1 --2- 172.21.15.161:0/2767467665 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f838406b500 0x7f838406d9c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:06.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.965+0000 7f839e7bf640 1 --2- 172.21.15.161:0/2767467665 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f839806c540 0x7f83980c0780 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:06.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.965+0000 7f839e7bf640 1 --2- 172.21.15.161:0/2767467665 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83980c1cf0 0x7f83981ea8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:06.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.965+0000 7f839e7bf640 1 --2- 172.21.15.161:0/2767467665 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f839806c160 0x7f83981dfdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:06.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.965+0000 7f839e7bf640 1 -- 172.21.15.161:0/2767467665 >> 172.21.15.161:0/2767467665 conn(0x7f83980c3be0 msgr2=0x7f8398151480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:06.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.965+0000 7f839e7bf640 1 -- 172.21.15.161:0/2767467665 shutdown_connections 2024-03-28T23:30:06.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:06.965+0000 7f839e7bf640 1 -- 172.21.15.161:0/2767467665 wait complete. 2024-03-28T23:30:06.977 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-28T23:30:07.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.082+0000 7fd37161c640 1 Processor -- start 2024-03-28T23:30:07.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.082+0000 7fd37161c640 1 -- start start 2024-03-28T23:30:07.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.083+0000 7fd37161c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd36c152bc0 0x7fd36c0c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:07.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.083+0000 7fd37161c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd36c1589b0 0x7fd36c158e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:07.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.083+0000 7fd37161c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd36c0c6f60 con 0x7fd36c0c8a80 2024-03-28T23:30:07.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.083+0000 7fd37161c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd36c0c6120 con 0x7fd36c1589b0 2024-03-28T23:30:07.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.083+0000 7fd37161c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd36c0c6400 con 0x7fd36c152bc0 2024-03-28T23:30:07.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.084+0000 7fd36affd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd36c152bc0 0x7fd36c0c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:07.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.084+0000 7fd36b7fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd36c1589b0 0x7fd36c158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:07.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.084+0000 7fd36affd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd36c152bc0 0x7fd36c0c0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:33214/0 (socket says 172.21.15.161:33214) 2024-03-28T23:30:07.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.084+0000 7fd36affd640 1 -- 172.21.15.161:0/1481415152 learned_addr learned my addr 172.21.15.161:0/1481415152 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:07.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.084+0000 7fd36affd640 1 -- 172.21.15.161:0/1481415152 >> v2:172.21.15.170:3300/0 conn(0x7fd36c1589b0 msgr2=0x7fd36c158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:07.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.084+0000 7fd36affd640 1 --2- 172.21.15.161:0/1481415152 >> v2:172.21.15.170:3300/0 conn(0x7fd36c1589b0 0x7fd36c158e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:07.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.084+0000 7fd36affd640 1 -- 172.21.15.161:0/1481415152 >> v1:172.21.15.170:6789/0 conn(0x7fd36c0c8a80 legacy=0x7fd36c0c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:07.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.084+0000 7fd36affd640 1 -- 172.21.15.161:0/1481415152 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd36c0c7240 con 0x7fd36c152bc0 2024-03-28T23:30:07.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.085+0000 7fd36affd640 1 --2- 172.21.15.161:0/1481415152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd36c152bc0 0x7fd36c0c0330 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd35c009e50 tx=0x7fd35c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5145a23a20572a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:07.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.085+0000 7fd369ffb640 1 -- 172.21.15.161:0/1481415152 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd35c02cb00 con 0x7fd36c152bc0 2024-03-28T23:30:07.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.085+0000 7fd369ffb640 1 -- 172.21.15.161:0/1481415152 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd35c02cca0 con 0x7fd36c152bc0 2024-03-28T23:30:07.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.085+0000 7fd369ffb640 1 -- 172.21.15.161:0/1481415152 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd35c034610 con 0x7fd36c152bc0 2024-03-28T23:30:07.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.086+0000 7fd37161c640 1 -- 172.21.15.161:0/1481415152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd36c152bc0 msgr2=0x7fd36c0c0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:07.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.086+0000 7fd37161c640 1 --2- 172.21.15.161:0/1481415152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd36c152bc0 0x7fd36c0c0330 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd35c009e50 tx=0x7fd35c02bf60 comp rx=0 tx=0).stop 2024-03-28T23:30:07.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.086+0000 7fd37161c640 1 -- 172.21.15.161:0/1481415152 shutdown_connections 2024-03-28T23:30:07.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.086+0000 7fd37161c640 1 --2- 172.21.15.161:0/1481415152 >> v2:172.21.15.170:3300/0 conn(0x7fd36c1589b0 0x7fd36c158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:07.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.086+0000 7fd37161c640 1 --2- 172.21.15.161:0/1481415152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd36c152bc0 0x7fd36c0c0330 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:07.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.086+0000 7fd37161c640 1 -- 172.21.15.161:0/1481415152 >> 172.21.15.161:0/1481415152 conn(0x7fd36c0caa80 msgr2=0x7fd36c0cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:07.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.086+0000 7fd37161c640 1 -- 172.21.15.161:0/1481415152 shutdown_connections 2024-03-28T23:30:07.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.086+0000 7fd37161c640 1 -- 172.21.15.161:0/1481415152 wait complete. 2024-03-28T23:30:07.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.088+0000 7fd37161c640 1 Processor -- start 2024-03-28T23:30:07.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.088+0000 7fd37161c640 1 -- start start 2024-03-28T23:30:07.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.089+0000 7fd37161c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd36c0c8a80 0x7fd36c14a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:07.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.089+0000 7fd37161c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd36c152bc0 0x7fd36c1558c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:07.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.089+0000 7fd37161c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd36c1589b0 0x7fd36c153b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:07.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.089+0000 7fd37161c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd36c152fa0 con 0x7fd36c152bc0 2024-03-28T23:30:07.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.089+0000 7fd37161c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd36c0c95f0 con 0x7fd36c1589b0 2024-03-28T23:30:07.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.089+0000 7fd37161c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd36c14acd0 con 0x7fd36c0c8a80 2024-03-28T23:30:07.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.089+0000 7fd36a7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd36c152bc0 0x7fd36c1558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.089+0000 7fd36a7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd36c152bc0 0x7fd36c1558c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45576/0 (socket says 172.21.15.161:45576) 2024-03-28T23:30:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.089+0000 7fd36a7fc640 1 -- 172.21.15.161:0/4219264537 learned_addr learned my addr 172.21.15.161:0/4219264537 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.089+0000 7fd36affd640 1 --2- 172.21.15.161:0/4219264537 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd36c0c8a80 0x7fd36c14a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.089+0000 7fd36b7fe640 1 --2- 172.21.15.161:0/4219264537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd36c1589b0 0x7fd36c153b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.089+0000 7fd36a7fc640 1 -- 172.21.15.161:0/4219264537 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd36c0c8a80 msgr2=0x7fd36c14a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.089+0000 7fd36a7fc640 1 --2- 172.21.15.161:0/4219264537 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd36c0c8a80 0x7fd36c14a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.089+0000 7fd36a7fc640 1 -- 172.21.15.161:0/4219264537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd36c1589b0 msgr2=0x7fd36c153b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.089+0000 7fd36a7fc640 1 --2- 172.21.15.161:0/4219264537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd36c1589b0 0x7fd36c153b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.089+0000 7fd36a7fc640 1 -- 172.21.15.161:0/4219264537 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd36c153350 con 0x7fd36c152bc0 2024-03-28T23:30:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.089+0000 7fd36a7fc640 1 --2- 172.21.15.161:0/4219264537 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd36c152bc0 0x7fd36c1558c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd35400c3c0 tx=0x7fd35400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.090+0000 7fd3637fe640 1 -- 172.21.15.161:0/4219264537 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3540052d0 con 0x7fd36c152bc0 2024-03-28T23:30:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.090+0000 7fd3637fe640 1 -- 172.21.15.161:0/4219264537 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd354005490 con 0x7fd36c152bc0 2024-03-28T23:30:07.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.090+0000 7fd3637fe640 1 -- 172.21.15.161:0/4219264537 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd354007260 con 0x7fd36c152bc0 2024-03-28T23:30:07.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.090+0000 7fd37161c640 1 -- 172.21.15.161:0/4219264537 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd36c1eb6a0 con 0x7fd36c152bc0 2024-03-28T23:30:07.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.090+0000 7fd37161c640 1 -- 172.21.15.161:0/4219264537 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd36c155e00 con 0x7fd36c152bc0 2024-03-28T23:30:07.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.091+0000 7fd37161c640 1 -- 172.21.15.161:0/4219264537 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd36c04f4b0 con 0x7fd36c152bc0 2024-03-28T23:30:07.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.091+0000 7fd3637fe640 1 -- 172.21.15.161:0/4219264537 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd354007400 con 0x7fd36c152bc0 2024-03-28T23:30:07.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.091+0000 7fd3637fe640 1 --2- 172.21.15.161:0/4219264537 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd34c06b4d0 0x7fd34c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:07.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.092+0000 7fd3637fe640 1 -- 172.21.15.161:0/4219264537 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd354089930 con 0x7fd36c152bc0 2024-03-28T23:30:07.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.092+0000 7fd36affd640 1 --2- 172.21.15.161:0/4219264537 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd34c06b4d0 0x7fd34c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:07.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.092+0000 7fd36affd640 1 --2- 172.21.15.161:0/4219264537 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd34c06b4d0 0x7fd34c06d990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd35c035000 tx=0x7fd35c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:07.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.096+0000 7fd3637fe640 1 -- 172.21.15.161:0/4219264537 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd354057e20 con 0x7fd36c152bc0 2024-03-28T23:30:07.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.305+0000 7fd37161c640 1 -- 172.21.15.161:0/4219264537 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fd36c0c7fb0 con 0x7fd36c152bc0 2024-03-28T23:30:07.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.630+0000 7fd3637fe640 1 -- 172.21.15.161:0/4219264537 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) ==== 118+0+0 (secure 0 0 0) 0x7fd35405bec0 con 0x7fd36c152bc0 2024-03-28T23:30:07.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.633+0000 7fd37161c640 1 -- 172.21.15.161:0/4219264537 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd34c06b4d0 msgr2=0x7fd34c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:07.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.633+0000 7fd37161c640 1 --2- 172.21.15.161:0/4219264537 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd34c06b4d0 0x7fd34c06d990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd35c035000 tx=0x7fd35c030000 comp rx=0 tx=0).stop 2024-03-28T23:30:07.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.633+0000 7fd37161c640 1 -- 172.21.15.161:0/4219264537 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd36c152bc0 msgr2=0x7fd36c1558c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:07.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.633+0000 7fd37161c640 1 --2- 172.21.15.161:0/4219264537 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd36c152bc0 0x7fd36c1558c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd35400c3c0 tx=0x7fd35400c880 comp rx=0 tx=0).stop 2024-03-28T23:30:07.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.633+0000 7fd37161c640 1 -- 172.21.15.161:0/4219264537 shutdown_connections 2024-03-28T23:30:07.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.633+0000 7fd37161c640 1 --2- 172.21.15.161:0/4219264537 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd34c06b4d0 0x7fd34c06d990 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:07.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.633+0000 7fd37161c640 1 --2- 172.21.15.161:0/4219264537 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd36c152bc0 0x7fd36c1558c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:07.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.633+0000 7fd37161c640 1 --2- 172.21.15.161:0/4219264537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd36c1589b0 0x7fd36c153b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:07.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.633+0000 7fd37161c640 1 --2- 172.21.15.161:0/4219264537 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd36c0c8a80 0x7fd36c14a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:07.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.633+0000 7fd37161c640 1 -- 172.21.15.161:0/4219264537 >> 172.21.15.161:0/4219264537 conn(0x7fd36c0caa80 msgr2=0x7fd36c14cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:07.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.633+0000 7fd37161c640 1 -- 172.21.15.161:0/4219264537 shutdown_connections 2024-03-28T23:30:07.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.633+0000 7fd37161c640 1 -- 172.21.15.161:0/4219264537 wait complete. 2024-03-28T23:30:07.648 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-03-28T23:30:07.650 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-28T23:30:07.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.754+0000 7f67f3b81640 1 Processor -- start 2024-03-28T23:30:07.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.754+0000 7f67f3b81640 1 -- start start 2024-03-28T23:30:07.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.755+0000 7f67f3b81640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f67ec152bc0 0x7f67ec0c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:07.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.756+0000 7f67f3b81640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67ec0c6f60 con 0x7f67ec0c8a80 2024-03-28T23:30:07.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.756+0000 7f67f3b81640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67ec0c6120 con 0x7f67ec1589b0 2024-03-28T23:30:07.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.756+0000 7f67f3b81640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f67ec0c6400 con 0x7f67ec152bc0 2024-03-28T23:30:07.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.756+0000 7f67f191d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f67ec152bc0 0x7f67ec0c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:07.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.756+0000 7f67f111c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f67ec0c8a80 0x7f67ec0c8ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:57700/0 (socket says 172.21.15.161:57700) 2024-03-28T23:30:07.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.756+0000 7f67f111c640 1 -- 172.21.15.161:0/742199134 learned_addr learned my addr 172.21.15.161:0/742199134 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:07.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.756+0000 7f67f191d640 1 -- 172.21.15.161:0/742199134 >> v1:172.21.15.170:6789/0 conn(0x7f67ec1589b0 legacy=0x7f67ec158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:07.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.756+0000 7f67f191d640 1 -- 172.21.15.161:0/742199134 >> v1:172.21.15.161:6789/0 conn(0x7f67ec0c8a80 legacy=0x7f67ec0c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:07.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.756+0000 7f67f191d640 1 -- 172.21.15.161:0/742199134 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67ec0c7240 con 0x7f67ec152bc0 2024-03-28T23:30:07.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.757+0000 7f67f191d640 1 --2- 172.21.15.161:0/742199134 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67ec152bc0 0x7f67ec0c0330 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f67dc0097a0 tx=0x7f67dc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c81dfab82ff9fef2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:07.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.757+0000 7f67f091b640 1 -- 172.21.15.161:0/742199134 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc035020 con 0x7f67ec152bc0 2024-03-28T23:30:07.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.757+0000 7f67f091b640 1 -- 172.21.15.161:0/742199134 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67dc031830 con 0x7f67ec152bc0 2024-03-28T23:30:07.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.757+0000 7f67f091b640 1 -- 172.21.15.161:0/742199134 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc031c00 con 0x7f67ec152bc0 2024-03-28T23:30:07.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.757+0000 7f67f3b81640 1 -- 172.21.15.161:0/742199134 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67ec152bc0 msgr2=0x7f67ec0c0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:07.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.757+0000 7f67f3b81640 1 --2- 172.21.15.161:0/742199134 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67ec152bc0 0x7f67ec0c0330 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f67dc0097a0 tx=0x7f67dc02b8d0 comp rx=0 tx=0).stop 2024-03-28T23:30:07.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.757+0000 7f67f3b81640 1 -- 172.21.15.161:0/742199134 shutdown_connections 2024-03-28T23:30:07.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.757+0000 7f67f3b81640 1 --2- 172.21.15.161:0/742199134 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67ec152bc0 0x7f67ec0c0330 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:07.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.757+0000 7f67f3b81640 1 -- 172.21.15.161:0/742199134 >> 172.21.15.161:0/742199134 conn(0x7f67ec0caa80 msgr2=0x7f67ec0cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:07.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.758+0000 7f67f3b81640 1 -- 172.21.15.161:0/742199134 shutdown_connections 2024-03-28T23:30:07.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.758+0000 7f67f3b81640 1 -- 172.21.15.161:0/742199134 wait complete. 2024-03-28T23:30:07.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.759+0000 7f67f3b81640 1 Processor -- start 2024-03-28T23:30:07.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.759+0000 7f67f3b81640 1 -- start start 2024-03-28T23:30:07.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.759+0000 7f67f3b81640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67ec0c8a80 0x7f67ec14f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:07.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.759+0000 7f67f3b81640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67ec152bc0 0x7f67ec149610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:07.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.759+0000 7f67f3b81640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67ec1589b0 0x7f67ec149b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:07.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.759+0000 7f67f3b81640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f67ec152fa0 con 0x7f67ec0c8a80 2024-03-28T23:30:07.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.759+0000 7f67f3b81640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f67ec0c95f0 con 0x7f67ec1589b0 2024-03-28T23:30:07.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.759+0000 7f67f3b81640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f67ec157bd0 con 0x7f67ec152bc0 2024-03-28T23:30:07.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.759+0000 7f67f191d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67ec0c8a80 0x7f67ec14f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:07.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.759+0000 7f67f191d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67ec0c8a80 0x7f67ec14f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45586/0 (socket says 172.21.15.161:45586) 2024-03-28T23:30:07.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.759+0000 7f67f191d640 1 -- 172.21.15.161:0/2149455114 learned_addr learned my addr 172.21.15.161:0/2149455114 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:07.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.760+0000 7f67f211e640 1 --2- 172.21.15.161:0/2149455114 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67ec1589b0 0x7f67ec149b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:07.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.760+0000 7f67f111c640 1 --2- 172.21.15.161:0/2149455114 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67ec152bc0 0x7f67ec149610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:07.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.760+0000 7f67f191d640 1 -- 172.21.15.161:0/2149455114 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67ec152bc0 msgr2=0x7f67ec149610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:07.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.760+0000 7f67f191d640 1 --2- 172.21.15.161:0/2149455114 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67ec152bc0 0x7f67ec149610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:07.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.760+0000 7f67f191d640 1 -- 172.21.15.161:0/2149455114 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67ec1589b0 msgr2=0x7f67ec149b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:07.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.760+0000 7f67f191d640 1 --2- 172.21.15.161:0/2149455114 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67ec1589b0 0x7f67ec149b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:07.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.760+0000 7f67f191d640 1 -- 172.21.15.161:0/2149455114 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67ec157eb0 con 0x7f67ec0c8a80 2024-03-28T23:30:07.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.760+0000 7f67f111c640 1 --2- 172.21.15.161:0/2149455114 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67ec152bc0 0x7f67ec149610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:07.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.760+0000 7f67f191d640 1 --2- 172.21.15.161:0/2149455114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67ec0c8a80 0x7f67ec14f5d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f67dc0098c0 tx=0x7f67dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:07.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.760+0000 7f67e27fc640 1 -- 172.21.15.161:0/2149455114 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc041070 con 0x7f67ec0c8a80 2024-03-28T23:30:07.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.760+0000 7f67e27fc640 1 -- 172.21.15.161:0/2149455114 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67dc03aa30 con 0x7f67ec0c8a80 2024-03-28T23:30:07.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.760+0000 7f67e27fc640 1 -- 172.21.15.161:0/2149455114 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc035040 con 0x7f67ec0c8a80 2024-03-28T23:30:07.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.760+0000 7f67f3b81640 1 -- 172.21.15.161:0/2149455114 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67ec158190 con 0x7f67ec0c8a80 2024-03-28T23:30:07.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.760+0000 7f67f3b81640 1 -- 172.21.15.161:0/2149455114 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67ec158750 con 0x7f67ec0c8a80 2024-03-28T23:30:07.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.761+0000 7f67f3b81640 1 -- 172.21.15.161:0/2149455114 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67ec04f4b0 con 0x7f67ec0c8a80 2024-03-28T23:30:07.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.762+0000 7f67e27fc640 1 -- 172.21.15.161:0/2149455114 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f67dc04c060 con 0x7f67ec0c8a80 2024-03-28T23:30:07.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.762+0000 7f67e27fc640 1 --2- 172.21.15.161:0/2149455114 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f67d006b4d0 0x7f67d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:07.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.762+0000 7f67e27fc640 1 -- 172.21.15.161:0/2149455114 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f67dc0abfd0 con 0x7f67ec0c8a80 2024-03-28T23:30:07.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.762+0000 7f67f111c640 1 --2- 172.21.15.161:0/2149455114 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f67d006b4d0 0x7f67d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:07.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.762+0000 7f67f111c640 1 --2- 172.21.15.161:0/2149455114 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f67d006b4d0 0x7f67d006d990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f67ec141e80 tx=0x7f67e8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:07.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.766+0000 7f67e27fc640 1 -- 172.21.15.161:0/2149455114 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f67dc07ab10 con 0x7f67ec0c8a80 2024-03-28T23:30:07.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:07.983+0000 7f67f3b81640 1 -- 172.21.15.161:0/2149455114 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f67ec149370 con 0x7f67ec0c8a80 2024-03-28T23:30:08.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.636+0000 7f67e27fc640 1 -- 172.21.15.161:0/2149455114 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) ==== 122+0+0 (secure 0 0 0) 0x7f67dc07a410 con 0x7f67ec0c8a80 2024-03-28T23:30:08.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.638+0000 7f67f3b81640 1 -- 172.21.15.161:0/2149455114 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f67d006b4d0 msgr2=0x7f67d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:08.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.638+0000 7f67f3b81640 1 --2- 172.21.15.161:0/2149455114 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f67d006b4d0 0x7f67d006d990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f67ec141e80 tx=0x7f67e8009040 comp rx=0 tx=0).stop 2024-03-28T23:30:08.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.638+0000 7f67f3b81640 1 -- 172.21.15.161:0/2149455114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67ec0c8a80 msgr2=0x7f67ec14f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:08.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.638+0000 7f67f3b81640 1 --2- 172.21.15.161:0/2149455114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67ec0c8a80 0x7f67ec14f5d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f67dc0098c0 tx=0x7f67dc0046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:08.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.638+0000 7f67f3b81640 1 -- 172.21.15.161:0/2149455114 shutdown_connections 2024-03-28T23:30:08.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.638+0000 7f67f3b81640 1 --2- 172.21.15.161:0/2149455114 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f67d006b4d0 0x7f67d006d990 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:08.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.638+0000 7f67f3b81640 1 --2- 172.21.15.161:0/2149455114 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67ec1589b0 0x7f67ec149b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:08.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.638+0000 7f67f3b81640 1 --2- 172.21.15.161:0/2149455114 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67ec152bc0 0x7f67ec149610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:08.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.638+0000 7f67f3b81640 1 --2- 172.21.15.161:0/2149455114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67ec0c8a80 0x7f67ec14f5d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:08.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.638+0000 7f67f3b81640 1 -- 172.21.15.161:0/2149455114 >> 172.21.15.161:0/2149455114 conn(0x7f67ec0caa80 msgr2=0x7f67ec14b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:08.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.638+0000 7f67f3b81640 1 -- 172.21.15.161:0/2149455114 shutdown_connections 2024-03-28T23:30:08.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.638+0000 7f67f3b81640 1 -- 172.21.15.161:0/2149455114 wait complete. 2024-03-28T23:30:08.651 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-28T23:30:08.651 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:30:08.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.753+0000 7fcfde62e640 1 Processor -- start 2024-03-28T23:30:08.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.753+0000 7fcfde62e640 1 -- start start 2024-03-28T23:30:08.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.754+0000 7fcfde62e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfd8152890 0x7fcfd80c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:08.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.754+0000 7fcfde62e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfd8158a30 con 0x7fcfd80c8ed0 2024-03-28T23:30:08.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.754+0000 7fcfde62e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfd8158d10 con 0x7fcfd80c5b20 2024-03-28T23:30:08.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.754+0000 7fcfde62e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fcfd80c8480 con 0x7fcfd8152890 2024-03-28T23:30:08.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.754+0000 7fcfd77fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fcfd80c8ed0 0x7fcfd80c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:57702/0 (socket says 172.21.15.161:57702) 2024-03-28T23:30:08.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.754+0000 7fcfd77fe640 1 -- 172.21.15.161:0/1898345506 learned_addr learned my addr 172.21.15.161:0/1898345506 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:08.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.754+0000 7fcfd6ffd640 1 -- 172.21.15.161:0/1898345506 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 936966205 0 0) 0x7fcfd8158a30 con 0x7fcfd80c8ed0 2024-03-28T23:30:08.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.754+0000 7fcfd7fff640 1 --2- 172.21.15.161:0/1898345506 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfd8152890 0x7fcfd80c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:08.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.755+0000 7fcfd6ffd640 1 -- 172.21.15.161:0/1898345506 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfc4003520 con 0x7fcfd80c8ed0 2024-03-28T23:30:08.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.755+0000 7fcfd6ffd640 1 -- 172.21.15.161:0/1898345506 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1245367545 0 0) 0x7fcfc4003520 con 0x7fcfd80c8ed0 2024-03-28T23:30:08.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.755+0000 7fcfd6ffd640 1 -- 172.21.15.161:0/1898345506 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfd8152890 msgr2=0x7fcfd80c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:08.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.755+0000 7fcfd6ffd640 1 --2- 172.21.15.161:0/1898345506 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfd8152890 0x7fcfd80c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:08.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.755+0000 7fcfd6ffd640 1 -- 172.21.15.161:0/1898345506 >> v1:172.21.15.170:6789/0 conn(0x7fcfd80c5b20 legacy=0x7fcfd81553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:08.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.755+0000 7fcfd6ffd640 1 -- 172.21.15.161:0/1898345506 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd8158ff0 con 0x7fcfd80c8ed0 2024-03-28T23:30:08.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.755+0000 7fcfd6ffd640 1 -- 172.21.15.161:0/1898345506 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fcfc80017f0 con 0x7fcfd80c8ed0 2024-03-28T23:30:08.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.755+0000 7fcfd6ffd640 1 -- 172.21.15.161:0/1898345506 >> v1:172.21.15.161:6789/0 conn(0x7fcfd80c8ed0 legacy=0x7fcfd80c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:08.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.755+0000 7fcfd6ffd640 1 --2- 172.21.15.161:0/1898345506 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfd8152890 0x7fcfc4003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:08.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.755+0000 7fcfd6ffd640 1 --2- 172.21.15.161:0/1898345506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfc4004040 0x7fcfc4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:08.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.755+0000 7fcfd6ffd640 1 --2- 172.21.15.161:0/1898345506 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfc4004db0 0x7fcfc4005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:08.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.755+0000 7fcfd6ffd640 1 -- 172.21.15.161:0/1898345506 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc400bc90 con 0x7fcfc4004db0 2024-03-28T23:30:08.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.755+0000 7fcfd6ffd640 1 -- 172.21.15.161:0/1898345506 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc400be40 con 0x7fcfc4004040 2024-03-28T23:30:08.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.755+0000 7fcfd6ffd640 1 -- 172.21.15.161:0/1898345506 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fcfc400c040 con 0x7fcfd8152890 2024-03-28T23:30:08.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.755+0000 7fcfdcbcb640 1 --2- 172.21.15.161:0/1898345506 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfc4004db0 0x7fcfc4005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:08.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.755+0000 7fcfd7fff640 1 --2- 172.21.15.161:0/1898345506 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfd8152890 0x7fcfc4003ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:08.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.755+0000 7fcfd77fe640 1 --2- 172.21.15.161:0/1898345506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfc4004040 0x7fcfc4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:08.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.755+0000 7fcfdcbcb640 1 -- 172.21.15.161:0/1898345506 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfd8152890 msgr2=0x7fcfc4003ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:08.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.755+0000 7fcfdcbcb640 1 --2- 172.21.15.161:0/1898345506 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfd8152890 0x7fcfc4003ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:08.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.755+0000 7fcfdcbcb640 1 -- 172.21.15.161:0/1898345506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfc4004040 msgr2=0x7fcfc4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:08.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.755+0000 7fcfdcbcb640 1 --2- 172.21.15.161:0/1898345506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfc4004040 0x7fcfc4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:08.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.755+0000 7fcfdcbcb640 1 -- 172.21.15.161:0/1898345506 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfc4003520 con 0x7fcfc4004db0 2024-03-28T23:30:08.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.755+0000 7fcfd7fff640 1 --2- 172.21.15.161:0/1898345506 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfd8152890 0x7fcfc4003ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:08.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.756+0000 7fcfdcbcb640 1 --2- 172.21.15.161:0/1898345506 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfc4004db0 0x7fcfc4005230 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fcfd80caa40 tx=0x7fcfc4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4209f36a2ecb2a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:08.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.756+0000 7fcfd6ffd640 1 -- 172.21.15.161:0/1898345506 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0001a70 con 0x7fcfc4004db0 2024-03-28T23:30:08.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.756+0000 7fcfd6ffd640 1 -- 172.21.15.161:0/1898345506 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfd0003b70 con 0x7fcfc4004db0 2024-03-28T23:30:08.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.756+0000 7fcfd6ffd640 1 -- 172.21.15.161:0/1898345506 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd00050d0 con 0x7fcfc4004db0 2024-03-28T23:30:08.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.757+0000 7fcfde62e640 1 -- 172.21.15.161:0/1898345506 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfc4004db0 msgr2=0x7fcfc4005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:08.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.757+0000 7fcfde62e640 1 --2- 172.21.15.161:0/1898345506 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfc4004db0 0x7fcfc4005230 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fcfd80caa40 tx=0x7fcfc4005830 comp rx=0 tx=0).stop 2024-03-28T23:30:08.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.757+0000 7fcfdcbcb640 1 -- 172.21.15.161:0/1898345506 reap_dead start 2024-03-28T23:30:08.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.757+0000 7fcfde62e640 1 -- 172.21.15.161:0/1898345506 shutdown_connections 2024-03-28T23:30:08.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.757+0000 7fcfde62e640 1 -- 172.21.15.161:0/1898345506 >> 172.21.15.161:0/1898345506 conn(0x7fcfd80ac620 msgr2=0x7fcfd80aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:08.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.757+0000 7fcfde62e640 1 -- 172.21.15.161:0/1898345506 shutdown_connections 2024-03-28T23:30:08.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.757+0000 7fcfde62e640 1 -- 172.21.15.161:0/1898345506 wait complete. 2024-03-28T23:30:08.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.758+0000 7fcfde62e640 1 Processor -- start 2024-03-28T23:30:08.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.758+0000 7fcfde62e640 1 -- start start 2024-03-28T23:30:08.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.759+0000 7fcfde62e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfc4004db0 0x7fcfd80c14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:08.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.759+0000 7fcfde62e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfd8155760 0x7fcfd8157620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:08.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.759+0000 7fcfde62e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfd8157b60 0x7fcfd81eada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:08.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.759+0000 7fcfde62e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd8152c70 con 0x7fcfd8157b60 2024-03-28T23:30:08.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.759+0000 7fcfde62e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd8159640 con 0x7fcfd8155760 2024-03-28T23:30:08.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.759+0000 7fcfde62e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fcfd8152df0 con 0x7fcfc4004db0 2024-03-28T23:30:08.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.759+0000 7fcfdcbcb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfd8157b60 0x7fcfd81eada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:08.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.759+0000 7fcfdcbcb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfd8157b60 0x7fcfd81eada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45608/0 (socket says 172.21.15.161:45608) 2024-03-28T23:30:08.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.759+0000 7fcfdcbcb640 1 -- 172.21.15.161:0/2201142605 learned_addr learned my addr 172.21.15.161:0/2201142605 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:08.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.759+0000 7fcfd7fff640 1 --2- 172.21.15.161:0/2201142605 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfc4004db0 0x7fcfd80c14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:08.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.759+0000 7fcfd77fe640 1 --2- 172.21.15.161:0/2201142605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfd8155760 0x7fcfd8157620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:08.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.759+0000 7fcfdcbcb640 1 -- 172.21.15.161:0/2201142605 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfc4004db0 msgr2=0x7fcfd80c14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:08.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.760+0000 7fcfdcbcb640 1 --2- 172.21.15.161:0/2201142605 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfc4004db0 0x7fcfd80c14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:08.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.760+0000 7fcfdcbcb640 1 -- 172.21.15.161:0/2201142605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfd8155760 msgr2=0x7fcfd8157620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:08.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.760+0000 7fcfdcbcb640 1 --2- 172.21.15.161:0/2201142605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfd8155760 0x7fcfd8157620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:08.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.760+0000 7fcfdcbcb640 1 -- 172.21.15.161:0/2201142605 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd8145430 con 0x7fcfd8157b60 2024-03-28T23:30:08.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.760+0000 7fcfdcbcb640 1 --2- 172.21.15.161:0/2201142605 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfd8157b60 0x7fcfd81eada0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0001a90 tx=0x7fcfd00014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:08.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.760+0000 7fcfd4ff9640 1 -- 172.21.15.161:0/2201142605 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc0009690 con 0x7fcfd8157b60 2024-03-28T23:30:08.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.760+0000 7fcfd4ff9640 1 -- 172.21.15.161:0/2201142605 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfd0003970 con 0x7fcfd8157b60 2024-03-28T23:30:08.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.760+0000 7fcfde62e640 1 -- 172.21.15.161:0/2201142605 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd8145bf0 con 0x7fcfd8157b60 2024-03-28T23:30:08.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.760+0000 7fcfd4ff9640 1 -- 172.21.15.161:0/2201142605 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0007db0 con 0x7fcfd8157b60 2024-03-28T23:30:08.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.760+0000 7fcfde62e640 1 -- 172.21.15.161:0/2201142605 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd814f9e0 con 0x7fcfd8157b60 2024-03-28T23:30:08.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.761+0000 7fcfde62e640 1 -- 172.21.15.161:0/2201142605 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfd80ac440 con 0x7fcfd8157b60 2024-03-28T23:30:08.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.761+0000 7fcfd4ff9640 1 -- 172.21.15.161:0/2201142605 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcfd0041030 con 0x7fcfd8157b60 2024-03-28T23:30:08.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.762+0000 7fcfd4ff9640 1 --2- 172.21.15.161:0/2201142605 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcfb806b500 0x7fcfb806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:08.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.762+0000 7fcfd4ff9640 1 -- 172.21.15.161:0/2201142605 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fcfd00ab200 con 0x7fcfd8157b60 2024-03-28T23:30:08.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.762+0000 7fcfd7fff640 1 --2- 172.21.15.161:0/2201142605 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcfb806b500 0x7fcfb806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:08.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.762+0000 7fcfd7fff640 1 --2- 172.21.15.161:0/2201142605 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcfb806b500 0x7fcfb806d9c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8158540 tx=0x7fcfc0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:08.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.766+0000 7fcfd4ff9640 1 -- 172.21.15.161:0/2201142605 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcfd80ac440 con 0x7fcfd8157b60 2024-03-28T23:30:08.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.976+0000 7fcfde62e640 1 -- 172.21.15.161:0/2201142605 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcfd804f4b0 con 0x7fcfd8157b60 2024-03-28T23:30:08.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.977+0000 7fcfd4ff9640 1 -- 172.21.15.161:0/2201142605 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5189 (secure 0 0 0) 0x7fcfd0033020 con 0x7fcfd8157b60 2024-03-28T23:30:08.979 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:08.979 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/1577284938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":1577284938},{"type":"v1","addr":"172.21.15.170:6835","nonce":1577284938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4258,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/1107796931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1107796931},{"type":"v1","addr":"172.21.15.170:6834","nonce":1107796931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4107992135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4107992135},{"type":"v1","addr":"172.21.15.161:6836","nonce":4107992135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:29:45.903844+0000","modified":"2024-03-28T23:30:07.984387+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6837/3832540545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3832540545},{"type":"v1","addr":"172.21.15.161:6837","nonce":3832540545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4275,"qdb_cluster":[4275]},"id":1}]} 2024-03-28T23:30:08.979 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 9 2024-03-28T23:30:08.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.979+0000 7fcfde62e640 1 -- 172.21.15.161:0/2201142605 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcfb806b500 msgr2=0x7fcfb806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:08.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.979+0000 7fcfde62e640 1 --2- 172.21.15.161:0/2201142605 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcfb806b500 0x7fcfb806d9c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8158540 tx=0x7fcfc0009190 comp rx=0 tx=0).stop 2024-03-28T23:30:08.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.979+0000 7fcfde62e640 1 -- 172.21.15.161:0/2201142605 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfd8157b60 msgr2=0x7fcfd81eada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:08.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.979+0000 7fcfde62e640 1 --2- 172.21.15.161:0/2201142605 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfd8157b60 0x7fcfd81eada0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0001a90 tx=0x7fcfd00014a0 comp rx=0 tx=0).stop 2024-03-28T23:30:08.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.979+0000 7fcfde62e640 1 -- 172.21.15.161:0/2201142605 shutdown_connections 2024-03-28T23:30:08.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.979+0000 7fcfde62e640 1 --2- 172.21.15.161:0/2201142605 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcfb806b500 0x7fcfb806d9c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:08.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.979+0000 7fcfde62e640 1 --2- 172.21.15.161:0/2201142605 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfd8157b60 0x7fcfd81eada0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:08.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.979+0000 7fcfde62e640 1 --2- 172.21.15.161:0/2201142605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfd8155760 0x7fcfd8157620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:08.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.979+0000 7fcfde62e640 1 --2- 172.21.15.161:0/2201142605 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfc4004db0 0x7fcfd80c14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:08.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.979+0000 7fcfde62e640 1 -- 172.21.15.161:0/2201142605 >> 172.21.15.161:0/2201142605 conn(0x7fcfd80ac620 msgr2=0x7fcfd8150910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:08.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.979+0000 7fcfde62e640 1 -- 172.21.15.161:0/2201142605 shutdown_connections 2024-03-28T23:30:08.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:08.979+0000 7fcfde62e640 1 -- 172.21.15.161:0/2201142605 wait complete. 2024-03-28T23:30:08.992 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:30:09.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.119+0000 7f85baa9d640 1 Processor -- start 2024-03-28T23:30:09.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.119+0000 7f85baa9d640 1 -- start start 2024-03-28T23:30:09.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.120+0000 7f85baa9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f85b4072810 0x7f85b40c4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:09.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.120+0000 7f85baa9d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f85b4070370 0x7f85b40c3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:09.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.120+0000 7f85baa9d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85b40707e0 con 0x7f85b40c4d80 2024-03-28T23:30:09.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.120+0000 7f85baa9d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f85b40a8520 con 0x7f85b4072810 2024-03-28T23:30:09.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.120+0000 7f85baa9d640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f85b4153050 con 0x7f85b4070370 2024-03-28T23:30:09.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.120+0000 7f85b8839640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f85b4072810 0x7f85b40c4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:09.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.120+0000 7f85b8839640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f85b4072810 0x7f85b40c4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45620/0 (socket says 172.21.15.161:45620) 2024-03-28T23:30:09.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.120+0000 7f85b8839640 1 -- 172.21.15.161:0/623950226 learned_addr learned my addr 172.21.15.161:0/623950226 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:09.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.120+0000 7f85b903a640 1 --2- 172.21.15.161:0/623950226 >> v2:172.21.15.170:3300/0 conn(0x7f85b4070370 0x7f85b40c3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:09.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.120+0000 7f85b8839640 1 -- 172.21.15.161:0/623950226 >> v2:172.21.15.170:3300/0 conn(0x7f85b4070370 msgr2=0x7f85b40c3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:09.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.120+0000 7f85b8839640 1 --2- 172.21.15.161:0/623950226 >> v2:172.21.15.170:3300/0 conn(0x7f85b4070370 0x7f85b40c3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.120+0000 7f85b8839640 1 -- 172.21.15.161:0/623950226 >> v1:172.21.15.170:6789/0 conn(0x7f85b40c4d80 legacy=0x7f85b406fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:09.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.120+0000 7f85b8839640 1 -- 172.21.15.161:0/623950226 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85b4153310 con 0x7f85b4072810 2024-03-28T23:30:09.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.121+0000 7f85b8839640 1 --2- 172.21.15.161:0/623950226 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85b4072810 0x7f85b40c4840 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f85b4150c20 tx=0x7f85a802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9083c4ebb3ddce5a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:09.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.121+0000 7f85b37fe640 1 -- 172.21.15.161:0/623950226 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a8035020 con 0x7f85b4072810 2024-03-28T23:30:09.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.121+0000 7f85b37fe640 1 -- 172.21.15.161:0/623950226 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85a8031870 con 0x7f85b4072810 2024-03-28T23:30:09.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.121+0000 7f85b37fe640 1 -- 172.21.15.161:0/623950226 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a8031c40 con 0x7f85b4072810 2024-03-28T23:30:09.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.122+0000 7f85baa9d640 1 -- 172.21.15.161:0/623950226 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85b4072810 msgr2=0x7f85b40c4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:09.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.122+0000 7f85baa9d640 1 --2- 172.21.15.161:0/623950226 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85b4072810 0x7f85b40c4840 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f85b4150c20 tx=0x7f85a802b910 comp rx=0 tx=0).stop 2024-03-28T23:30:09.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.122+0000 7f85baa9d640 1 -- 172.21.15.161:0/623950226 shutdown_connections 2024-03-28T23:30:09.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.122+0000 7f85baa9d640 1 --2- 172.21.15.161:0/623950226 >> v2:172.21.15.170:3300/0 conn(0x7f85b4070370 0x7f85b40c3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.122+0000 7f85baa9d640 1 --2- 172.21.15.161:0/623950226 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85b4072810 0x7f85b40c4840 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.122+0000 7f85baa9d640 1 -- 172.21.15.161:0/623950226 >> 172.21.15.161:0/623950226 conn(0x7f85b40aa740 msgr2=0x7f85b40aab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:09.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.122+0000 7f85baa9d640 1 -- 172.21.15.161:0/623950226 shutdown_connections 2024-03-28T23:30:09.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.122+0000 7f85baa9d640 1 -- 172.21.15.161:0/623950226 wait complete. 2024-03-28T23:30:09.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.123+0000 7f85baa9d640 1 Processor -- start 2024-03-28T23:30:09.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.123+0000 7f85baa9d640 1 -- start start 2024-03-28T23:30:09.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.125+0000 7f85baa9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85b4070370 0x7f85b414b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:09.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.125+0000 7f85baa9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85b4072810 0x7f85b40aba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:09.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.125+0000 7f85baa9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85b40c4d80 0x7f85b4146140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:09.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.125+0000 7f85baa9d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f85b4153bb0 con 0x7f85b4072810 2024-03-28T23:30:09.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.125+0000 7f85baa9d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f85b41538d0 con 0x7f85b4070370 2024-03-28T23:30:09.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.125+0000 7f85baa9d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f85b4072bf0 con 0x7f85b40c4d80 2024-03-28T23:30:09.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.125+0000 7f85b3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85b4072810 0x7f85b40aba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:09.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.125+0000 7f85b3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85b4072810 0x7f85b40aba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45626/0 (socket says 172.21.15.161:45626) 2024-03-28T23:30:09.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.125+0000 7f85b3fff640 1 -- 172.21.15.161:0/2708313063 learned_addr learned my addr 172.21.15.161:0/2708313063 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:09.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.125+0000 7f85b8839640 1 --2- 172.21.15.161:0/2708313063 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85b4070370 0x7f85b414b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:09.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.125+0000 7f85b903a640 1 --2- 172.21.15.161:0/2708313063 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85b40c4d80 0x7f85b4146140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:09.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.125+0000 7f85b3fff640 1 -- 172.21.15.161:0/2708313063 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85b40c4d80 msgr2=0x7f85b4146140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:09.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.125+0000 7f85b3fff640 1 --2- 172.21.15.161:0/2708313063 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85b40c4d80 0x7f85b4146140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.125+0000 7f85b3fff640 1 -- 172.21.15.161:0/2708313063 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85b4070370 msgr2=0x7f85b414b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:09.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.125+0000 7f85b3fff640 1 --2- 172.21.15.161:0/2708313063 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85b4070370 0x7f85b414b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.125+0000 7f85b3fff640 1 -- 172.21.15.161:0/2708313063 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85b40adc30 con 0x7f85b4072810 2024-03-28T23:30:09.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.125+0000 7f85b3fff640 1 --2- 172.21.15.161:0/2708313063 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85b4072810 0x7f85b40aba40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f85a000c470 tx=0x7f85a000c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:09.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.125+0000 7f85b17fa640 1 -- 172.21.15.161:0/2708313063 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a00052d0 con 0x7f85b4072810 2024-03-28T23:30:09.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.125+0000 7f85b17fa640 1 -- 172.21.15.161:0/2708313063 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85a0005490 con 0x7f85b4072810 2024-03-28T23:30:09.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.125+0000 7f85b17fa640 1 -- 172.21.15.161:0/2708313063 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a0007260 con 0x7f85b4072810 2024-03-28T23:30:09.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.125+0000 7f85baa9d640 1 -- 172.21.15.161:0/2708313063 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85b40adf10 con 0x7f85b4072810 2024-03-28T23:30:09.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.126+0000 7f85baa9d640 1 -- 172.21.15.161:0/2708313063 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85b40ae4d0 con 0x7f85b4072810 2024-03-28T23:30:09.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.127+0000 7f85baa9d640 1 -- 172.21.15.161:0/2708313063 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85b404f4b0 con 0x7f85b4072810 2024-03-28T23:30:09.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.127+0000 7f85b17fa640 1 -- 172.21.15.161:0/2708313063 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f85a0007400 con 0x7f85b4072810 2024-03-28T23:30:09.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.127+0000 7f85b17fa640 1 --2- 172.21.15.161:0/2708313063 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f859806b500 0x7f859806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:09.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.128+0000 7f85b17fa640 1 -- 172.21.15.161:0/2708313063 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f85a00899a0 con 0x7f85b4072810 2024-03-28T23:30:09.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.128+0000 7f85b8839640 1 --2- 172.21.15.161:0/2708313063 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f859806b500 0x7f859806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:09.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.128+0000 7f85b8839640 1 --2- 172.21.15.161:0/2708313063 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f859806b500 0x7f859806d9c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f85a8036000 tx=0x7f85a803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:09.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.132+0000 7f85b17fa640 1 -- 172.21.15.161:0/2708313063 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f85b40707e0 con 0x7f85b4072810 2024-03-28T23:30:09.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.381+0000 7f85baa9d640 1 -- 172.21.15.161:0/2708313063 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f85b40c42a0 con 0x7f85b4072810 2024-03-28T23:30:09.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.383+0000 7f85b17fa640 1 -- 172.21.15.161:0/2708313063 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7f85a0057e00 con 0x7f85b4072810 2024-03-28T23:30:09.385 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:09.385 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":24,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:29:50.946843+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:30:09.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.385+0000 7f85baa9d640 1 -- 172.21.15.161:0/2708313063 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f859806b500 msgr2=0x7f859806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:09.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.385+0000 7f85baa9d640 1 --2- 172.21.15.161:0/2708313063 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f859806b500 0x7f859806d9c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f85a8036000 tx=0x7f85a803c040 comp rx=0 tx=0).stop 2024-03-28T23:30:09.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.385+0000 7f85baa9d640 1 -- 172.21.15.161:0/2708313063 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85b4072810 msgr2=0x7f85b40aba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:09.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.385+0000 7f85baa9d640 1 --2- 172.21.15.161:0/2708313063 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85b4072810 0x7f85b40aba40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f85a000c470 tx=0x7f85a000c930 comp rx=0 tx=0).stop 2024-03-28T23:30:09.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.386+0000 7f85baa9d640 1 -- 172.21.15.161:0/2708313063 shutdown_connections 2024-03-28T23:30:09.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.386+0000 7f85baa9d640 1 --2- 172.21.15.161:0/2708313063 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f859806b500 0x7f859806d9c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.386+0000 7f85baa9d640 1 --2- 172.21.15.161:0/2708313063 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85b4072810 0x7f85b40aba40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.386+0000 7f85baa9d640 1 --2- 172.21.15.161:0/2708313063 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85b40c4d80 0x7f85b4146140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.386+0000 7f85baa9d640 1 --2- 172.21.15.161:0/2708313063 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85b4070370 0x7f85b414b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.386+0000 7f85baa9d640 1 -- 172.21.15.161:0/2708313063 >> 172.21.15.161:0/2708313063 conn(0x7f85b40aa740 msgr2=0x7f85b40a90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:09.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.386+0000 7f85baa9d640 1 -- 172.21.15.161:0/2708313063 shutdown_connections 2024-03-28T23:30:09.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.386+0000 7f85baa9d640 1 -- 172.21.15.161:0/2708313063 wait complete. 2024-03-28T23:30:09.400 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T23:30:09.400 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:30:09.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.500+0000 7f0fd4fbf640 1 Processor -- start 2024-03-28T23:30:09.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.500+0000 7f0fd4fbf640 1 -- start start 2024-03-28T23:30:09.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.501+0000 7f0fd4fbf640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0fd0152890 0x7f0fd00c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:09.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.501+0000 7f0fd4fbf640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0fd00c8ed0 0x7f0fd00c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:09.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.501+0000 7f0fd4fbf640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fd0158a30 con 0x7f0fd00c5950 2024-03-28T23:30:09.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.501+0000 7f0fd4fbf640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0fd00c8480 con 0x7f0fd00c8ed0 2024-03-28T23:30:09.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.501+0000 7f0fd4fbf640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f0fd0159640 con 0x7f0fd0152890 2024-03-28T23:30:09.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.501+0000 7f0fcdd9b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0fd00c8ed0 0x7f0fd00c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:09.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.501+0000 7f0fced9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f0fd00c5950 0x7f0fd01553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:57718/0 (socket says 172.21.15.161:57718) 2024-03-28T23:30:09.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.501+0000 7f0fced9d640 1 -- 172.21.15.161:0/564560579 learned_addr learned my addr 172.21.15.161:0/564560579 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:09.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.501+0000 7f0fce59c640 1 --2- 172.21.15.161:0/564560579 >> v2:172.21.15.170:3300/0 conn(0x7f0fd0152890 0x7f0fd00c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:09.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.502+0000 7f0fcdd9b640 1 -- 172.21.15.161:0/564560579 >> v2:172.21.15.170:3300/0 conn(0x7f0fd0152890 msgr2=0x7f0fd00c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:09.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.502+0000 7f0fcdd9b640 1 --2- 172.21.15.161:0/564560579 >> v2:172.21.15.170:3300/0 conn(0x7f0fd0152890 0x7f0fd00c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.502+0000 7f0fcdd9b640 1 -- 172.21.15.161:0/564560579 >> v1:172.21.15.161:6789/0 conn(0x7f0fd00c5950 legacy=0x7f0fd01553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:09.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.502+0000 7f0fcdd9b640 1 -- 172.21.15.161:0/564560579 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fd0158d10 con 0x7f0fd00c8ed0 2024-03-28T23:30:09.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.502+0000 7f0fcd59a640 1 -- 172.21.15.161:0/564560579 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2938626996 0 0) 0x7f0fd0158a30 con 0x7f0fd00c5950 2024-03-28T23:30:09.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.502+0000 7f0fcdd9b640 1 --2- 172.21.15.161:0/564560579 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0fd00c8ed0 0x7f0fd00c5410 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0fd008cc80 tx=0x7f0fb802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad55545ec4edc01 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:09.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.502+0000 7f0fcd59a640 1 -- 172.21.15.161:0/564560579 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8038070 con 0x7f0fd00c8ed0 2024-03-28T23:30:09.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.502+0000 7f0fcd59a640 1 -- 172.21.15.161:0/564560579 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fb8031700 con 0x7f0fd00c8ed0 2024-03-28T23:30:09.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.502+0000 7f0fcd59a640 1 -- 172.21.15.161:0/564560579 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8031ad0 con 0x7f0fd00c8ed0 2024-03-28T23:30:09.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.503+0000 7f0fd4fbf640 1 -- 172.21.15.161:0/564560579 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0fd00c8ed0 msgr2=0x7f0fd00c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:09.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.503+0000 7f0fd4fbf640 1 --2- 172.21.15.161:0/564560579 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0fd00c8ed0 0x7f0fd00c5410 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0fd008cc80 tx=0x7f0fb802b7a0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.503+0000 7f0fd4fbf640 1 -- 172.21.15.161:0/564560579 shutdown_connections 2024-03-28T23:30:09.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.503+0000 7f0fd4fbf640 1 --2- 172.21.15.161:0/564560579 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0fd00c8ed0 0x7f0fd00c5410 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.503+0000 7f0fd4fbf640 1 --2- 172.21.15.161:0/564560579 >> v2:172.21.15.170:3300/0 conn(0x7f0fd0152890 0x7f0fd00c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.503+0000 7f0fd4fbf640 1 -- 172.21.15.161:0/564560579 >> 172.21.15.161:0/564560579 conn(0x7f0fd00ac620 msgr2=0x7f0fd00aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:09.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.503+0000 7f0fd4fbf640 1 -- 172.21.15.161:0/564560579 shutdown_connections 2024-03-28T23:30:09.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.503+0000 7f0fd4fbf640 1 -- 172.21.15.161:0/564560579 wait complete. 2024-03-28T23:30:09.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.505+0000 7f0fd4fbf640 1 Processor -- start 2024-03-28T23:30:09.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.505+0000 7f0fd4fbf640 1 -- start start 2024-03-28T23:30:09.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.506+0000 7f0fd4fbf640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fd00c5950 0x7f0fd01ee630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:09.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.506+0000 7f0fd4fbf640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fd00c8ed0 0x7f0fd01eb040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:09.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.506+0000 7f0fd4fbf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0fd0152890 0x7f0fd00c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:09.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.506+0000 7f0fd4fbf640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0fd0152c70 con 0x7f0fd0152890 2024-03-28T23:30:09.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.506+0000 7f0fd4fbf640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0fd01592d0 con 0x7f0fd00c5950 2024-03-28T23:30:09.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.506+0000 7f0fd4fbf640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0fd0152df0 con 0x7f0fd00c8ed0 2024-03-28T23:30:09.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.506+0000 7f0fced9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0fd0152890 0x7f0fd00c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:09.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.506+0000 7f0fced9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0fd0152890 0x7f0fd00c9de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45642/0 (socket says 172.21.15.161:45642) 2024-03-28T23:30:09.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.506+0000 7f0fced9d640 1 -- 172.21.15.161:0/893101738 learned_addr learned my addr 172.21.15.161:0/893101738 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:09.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.506+0000 7f0fce59c640 1 --2- 172.21.15.161:0/893101738 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fd00c5950 0x7f0fd01ee630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:09.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.506+0000 7f0fcdd9b640 1 --2- 172.21.15.161:0/893101738 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fd00c8ed0 0x7f0fd01eb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:09.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.506+0000 7f0fced9d640 1 -- 172.21.15.161:0/893101738 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fd00c8ed0 msgr2=0x7f0fd01eb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:09.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.506+0000 7f0fced9d640 1 --2- 172.21.15.161:0/893101738 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fd00c8ed0 0x7f0fd01eb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.506+0000 7f0fced9d640 1 -- 172.21.15.161:0/893101738 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fd00c5950 msgr2=0x7f0fd01ee630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:09.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.506+0000 7f0fced9d640 1 --2- 172.21.15.161:0/893101738 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fd00c5950 0x7f0fd01ee630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.507+0000 7f0fced9d640 1 -- 172.21.15.161:0/893101738 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fd00c1f20 con 0x7f0fd0152890 2024-03-28T23:30:09.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.507+0000 7f0fce59c640 1 --2- 172.21.15.161:0/893101738 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fd00c5950 0x7f0fd01ee630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:09.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.507+0000 7f0fced9d640 1 --2- 172.21.15.161:0/893101738 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0fd0152890 0x7f0fd00c9de0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000c700 tx=0x7f0fc000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:09.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.507+0000 7f0fc6ffd640 1 -- 172.21.15.161:0/893101738 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0018070 con 0x7f0fd0152890 2024-03-28T23:30:09.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.507+0000 7f0fc6ffd640 1 -- 172.21.15.161:0/893101738 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fc000ce10 con 0x7f0fd0152890 2024-03-28T23:30:09.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.507+0000 7f0fd4fbf640 1 -- 172.21.15.161:0/893101738 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fd00c2200 con 0x7f0fd0152890 2024-03-28T23:30:09.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.507+0000 7f0fc6ffd640 1 -- 172.21.15.161:0/893101738 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc000a860 con 0x7f0fd0152890 2024-03-28T23:30:09.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.507+0000 7f0fd4fbf640 1 -- 172.21.15.161:0/893101738 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fd01ede30 con 0x7f0fd0152890 2024-03-28T23:30:09.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.508+0000 7f0fd4fbf640 1 -- 172.21.15.161:0/893101738 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fd004f4b0 con 0x7f0fd0152890 2024-03-28T23:30:09.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.508+0000 7f0fc6ffd640 1 -- 172.21.15.161:0/893101738 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0fc001b070 con 0x7f0fd0152890 2024-03-28T23:30:09.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.509+0000 7f0fc6ffd640 1 --2- 172.21.15.161:0/893101738 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0fb006b4d0 0x7f0fb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:09.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.509+0000 7f0fce59c640 1 --2- 172.21.15.161:0/893101738 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0fb006b4d0 0x7f0fb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:09.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.509+0000 7f0fc6ffd640 1 -- 172.21.15.161:0/893101738 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0fc0089a00 con 0x7f0fd0152890 2024-03-28T23:30:09.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.509+0000 7f0fce59c640 1 --2- 172.21.15.161:0/893101738 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0fb006b4d0 0x7f0fb006d990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8009f30 tx=0x7f0fc8009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:09.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.513+0000 7f0fc6ffd640 1 -- 172.21.15.161:0/893101738 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0fc0057f80 con 0x7f0fd0152890 2024-03-28T23:30:09.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.750+0000 7f0fd4fbf640 1 -- 172.21.15.161:0/893101738 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0fd00c8140 con 0x7f0fb006b4d0 2024-03-28T23:30:09.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.759+0000 7f0fc6ffd640 1 -- 172.21.15.161:0/893101738 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332611 (secure 0 0 0) 0x7f0fd00c8140 con 0x7f0fb006b4d0 2024-03-28T23:30:09.765 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:09.766 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:30:09.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.762+0000 7f0fd4fbf640 1 -- 172.21.15.161:0/893101738 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0fb006b4d0 msgr2=0x7f0fb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:09.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.762+0000 7f0fd4fbf640 1 --2- 172.21.15.161:0/893101738 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0fb006b4d0 0x7f0fb006d990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8009f30 tx=0x7f0fc8009190 comp rx=0 tx=0).stop 2024-03-28T23:30:09.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.762+0000 7f0fd4fbf640 1 -- 172.21.15.161:0/893101738 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0fd0152890 msgr2=0x7f0fd00c9de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:09.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.762+0000 7f0fd4fbf640 1 --2- 172.21.15.161:0/893101738 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0fd0152890 0x7f0fd00c9de0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000c700 tx=0x7f0fc000cbc0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.762+0000 7f0fd4fbf640 1 -- 172.21.15.161:0/893101738 shutdown_connections 2024-03-28T23:30:09.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.762+0000 7f0fd4fbf640 1 --2- 172.21.15.161:0/893101738 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0fb006b4d0 0x7f0fb006d990 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.762+0000 7f0fd4fbf640 1 --2- 172.21.15.161:0/893101738 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0fd0152890 0x7f0fd00c9de0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.762+0000 7f0fd4fbf640 1 --2- 172.21.15.161:0/893101738 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fd00c8ed0 0x7f0fd01eb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.762+0000 7f0fd4fbf640 1 --2- 172.21.15.161:0/893101738 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fd00c5950 0x7f0fd01ee630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.762+0000 7f0fd4fbf640 1 -- 172.21.15.161:0/893101738 >> 172.21.15.161:0/893101738 conn(0x7f0fd00ac620 msgr2=0x7f0fd0150630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:09.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.762+0000 7f0fd4fbf640 1 -- 172.21.15.161:0/893101738 shutdown_connections 2024-03-28T23:30:09.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.762+0000 7f0fd4fbf640 1 -- 172.21.15.161:0/893101738 wait complete. 2024-03-28T23:30:09.775 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-03-28T23:30:09.316678+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222516,"kb_used_data":2176,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754752204,"statfs":{"total":773094113280,"available":772866256896,"internally_reserved":0,"allocated":2228224,"data_stored":1992064,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002338"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687840+0000","last_change":"2024-03-28T23:29:45.558575+0000","last_active":"2024-03-28T23:29:51.687840+0000","last_peered":"2024-03-28T23:29:51.687840+0000","last_clean":"2024-03-28T23:29:51.687840+0000","last_became_active":"2024-03-28T23:29:45.557910+0000","last_became_peered":"2024-03-28T23:29:45.557910+0000","last_unstale":"2024-03-28T23:29:51.687840+0000","last_undegraded":"2024-03-28T23:29:51.687840+0000","last_fullsized":"2024-03-28T23:29:51.687840+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:59.719252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686777+0000","last_change":"2024-03-28T23:29:46.557562+0000","last_active":"2024-03-28T23:29:51.686777+0000","last_peered":"2024-03-28T23:29:51.686777+0000","last_clean":"2024-03-28T23:29:51.686777+0000","last_became_active":"2024-03-28T23:29:46.557265+0000","last_became_peered":"2024-03-28T23:29:46.557265+0000","last_unstale":"2024-03-28T23:29:51.686777+0000","last_undegraded":"2024-03-28T23:29:51.686777+0000","last_fullsized":"2024-03-28T23:29:51.686777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:06.731262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686496+0000","last_change":"2024-03-28T23:29:50.707752+0000","last_active":"2024-03-28T23:29:51.686496+0000","last_peered":"2024-03-28T23:29:51.686496+0000","last_clean":"2024-03-28T23:29:51.686496+0000","last_became_active":"2024-03-28T23:29:50.707364+0000","last_became_peered":"2024-03-28T23:29:50.707364+0000","last_unstale":"2024-03-28T23:29:51.686496+0000","last_undegraded":"2024-03-28T23:29:51.686496+0000","last_fullsized":"2024-03-28T23:29:51.686496+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:58:38.680360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686927+0000","last_change":"2024-03-28T23:29:45.558356+0000","last_active":"2024-03-28T23:29:51.686927+0000","last_peered":"2024-03-28T23:29:51.686927+0000","last_clean":"2024-03-28T23:29:51.686927+0000","last_became_active":"2024-03-28T23:29:45.558070+0000","last_became_peered":"2024-03-28T23:29:45.558070+0000","last_unstale":"2024-03-28T23:29:51.686927+0000","last_undegraded":"2024-03-28T23:29:51.686927+0000","last_fullsized":"2024-03-28T23:29:51.686927+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:59.719252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686591+0000","last_change":"2024-03-28T23:29:46.579153+0000","last_active":"2024-03-28T23:29:51.686591+0000","last_peered":"2024-03-28T23:29:51.686591+0000","last_clean":"2024-03-28T23:29:51.686591+0000","last_became_active":"2024-03-28T23:29:46.578341+0000","last_became_peered":"2024-03-28T23:29:46.578341+0000","last_unstale":"2024-03-28T23:29:51.686591+0000","last_undegraded":"2024-03-28T23:29:51.686591+0000","last_fullsized":"2024-03-28T23:29:51.686591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:38.074388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591979+0000","last_change":"2024-03-28T23:29:50.597011+0000","last_active":"2024-03-28T23:29:51.591979+0000","last_peered":"2024-03-28T23:29:51.591979+0000","last_clean":"2024-03-28T23:29:51.591979+0000","last_became_active":"2024-03-28T23:29:50.596239+0000","last_became_peered":"2024-03-28T23:29:50.596239+0000","last_unstale":"2024-03-28T23:29:51.591979+0000","last_undegraded":"2024-03-28T23:29:51.591979+0000","last_fullsized":"2024-03-28T23:29:51.591979+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:29:24.703997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686674+0000","last_change":"2024-03-28T23:29:45.562460+0000","last_active":"2024-03-28T23:29:51.686674+0000","last_peered":"2024-03-28T23:29:51.686674+0000","last_clean":"2024-03-28T23:29:51.686674+0000","last_became_active":"2024-03-28T23:29:45.562101+0000","last_became_peered":"2024-03-28T23:29:45.562101+0000","last_unstale":"2024-03-28T23:29:51.686674+0000","last_undegraded":"2024-03-28T23:29:51.686674+0000","last_fullsized":"2024-03-28T23:29:51.686674+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:42:08.707703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685709+0000","last_change":"2024-03-28T23:29:46.567420+0000","last_active":"2024-03-28T23:29:51.685709+0000","last_peered":"2024-03-28T23:29:51.685709+0000","last_clean":"2024-03-28T23:29:51.685709+0000","last_became_active":"2024-03-28T23:29:46.567157+0000","last_became_peered":"2024-03-28T23:29:46.567157+0000","last_unstale":"2024-03-28T23:29:51.685709+0000","last_undegraded":"2024-03-28T23:29:51.685709+0000","last_fullsized":"2024-03-28T23:29:51.685709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:54.470202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.592139+0000","last_change":"2024-03-28T23:29:50.599020+0000","last_active":"2024-03-28T23:29:51.592139+0000","last_peered":"2024-03-28T23:29:51.592139+0000","last_clean":"2024-03-28T23:29:51.592139+0000","last_became_active":"2024-03-28T23:29:50.598588+0000","last_became_peered":"2024-03-28T23:29:50.598588+0000","last_unstale":"2024-03-28T23:29:51.592139+0000","last_undegraded":"2024-03-28T23:29:51.592139+0000","last_fullsized":"2024-03-28T23:29:51.592139+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:06.894069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686561+0000","last_change":"2024-03-28T23:29:45.564688+0000","last_active":"2024-03-28T23:29:51.686561+0000","last_peered":"2024-03-28T23:29:51.686561+0000","last_clean":"2024-03-28T23:29:51.686561+0000","last_became_active":"2024-03-28T23:29:45.564381+0000","last_became_peered":"2024-03-28T23:29:45.564381+0000","last_unstale":"2024-03-28T23:29:51.686561+0000","last_undegraded":"2024-03-28T23:29:51.686561+0000","last_fullsized":"2024-03-28T23:29:51.686561+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:36.113647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686025+0000","last_change":"2024-03-28T23:29:46.566644+0000","last_active":"2024-03-28T23:29:51.686025+0000","last_peered":"2024-03-28T23:29:51.686025+0000","last_clean":"2024-03-28T23:29:51.686025+0000","last_became_active":"2024-03-28T23:29:46.566340+0000","last_became_peered":"2024-03-28T23:29:46.566340+0000","last_unstale":"2024-03-28T23:29:51.686025+0000","last_undegraded":"2024-03-28T23:29:51.686025+0000","last_fullsized":"2024-03-28T23:29:51.686025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:02.859502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591969+0000","last_change":"2024-03-28T23:29:50.597019+0000","last_active":"2024-03-28T23:29:51.591969+0000","last_peered":"2024-03-28T23:29:51.591969+0000","last_clean":"2024-03-28T23:29:51.591969+0000","last_became_active":"2024-03-28T23:29:50.596284+0000","last_became_peered":"2024-03-28T23:29:50.596284+0000","last_unstale":"2024-03-28T23:29:51.591969+0000","last_undegraded":"2024-03-28T23:29:51.591969+0000","last_fullsized":"2024-03-28T23:29:51.591969+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:08:37.836411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.590591+0000","last_change":"2024-03-28T23:29:46.565325+0000","last_active":"2024-03-28T23:29:51.590591+0000","last_peered":"2024-03-28T23:29:51.590591+0000","last_clean":"2024-03-28T23:29:51.590591+0000","last_became_active":"2024-03-28T23:29:46.564195+0000","last_became_peered":"2024-03-28T23:29:46.564195+0000","last_unstale":"2024-03-28T23:29:51.590591+0000","last_undegraded":"2024-03-28T23:29:51.590591+0000","last_fullsized":"2024-03-28T23:29:51.590591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:32.936386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685919+0000","last_change":"2024-03-28T23:29:45.560329+0000","last_active":"2024-03-28T23:29:51.685919+0000","last_peered":"2024-03-28T23:29:51.685919+0000","last_clean":"2024-03-28T23:29:51.685919+0000","last_became_active":"2024-03-28T23:29:45.560073+0000","last_became_peered":"2024-03-28T23:29:45.560073+0000","last_unstale":"2024-03-28T23:29:51.685919+0000","last_undegraded":"2024-03-28T23:29:51.685919+0000","last_fullsized":"2024-03-28T23:29:51.685919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:09.929984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591983+0000","last_change":"2024-03-28T23:29:50.592083+0000","last_active":"2024-03-28T23:29:51.591983+0000","last_peered":"2024-03-28T23:29:51.591983+0000","last_clean":"2024-03-28T23:29:51.591983+0000","last_became_active":"2024-03-28T23:29:50.591626+0000","last_became_peered":"2024-03-28T23:29:50.591626+0000","last_unstale":"2024-03-28T23:29:51.591983+0000","last_undegraded":"2024-03-28T23:29:51.591983+0000","last_fullsized":"2024-03-28T23:29:51.591983+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:17:54.013778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685967+0000","last_change":"2024-03-28T23:29:46.581125+0000","last_active":"2024-03-28T23:29:51.685967+0000","last_peered":"2024-03-28T23:29:51.685967+0000","last_clean":"2024-03-28T23:29:51.685967+0000","last_became_active":"2024-03-28T23:29:46.580827+0000","last_became_peered":"2024-03-28T23:29:46.580827+0000","last_unstale":"2024-03-28T23:29:51.685967+0000","last_undegraded":"2024-03-28T23:29:51.685967+0000","last_fullsized":"2024-03-28T23:29:51.685967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:10.853710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686404+0000","last_change":"2024-03-28T23:29:45.562525+0000","last_active":"2024-03-28T23:29:51.686404+0000","last_peered":"2024-03-28T23:29:51.686404+0000","last_clean":"2024-03-28T23:29:51.686404+0000","last_became_active":"2024-03-28T23:29:45.561757+0000","last_became_peered":"2024-03-28T23:29:45.561757+0000","last_unstale":"2024-03-28T23:29:51.686404+0000","last_undegraded":"2024-03-28T23:29:51.686404+0000","last_fullsized":"2024-03-28T23:29:51.686404+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:24:58.784056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591567+0000","last_change":"2024-03-28T23:29:50.599617+0000","last_active":"2024-03-28T23:29:51.591567+0000","last_peered":"2024-03-28T23:29:51.591567+0000","last_clean":"2024-03-28T23:29:51.591567+0000","last_became_active":"2024-03-28T23:29:50.599031+0000","last_became_peered":"2024-03-28T23:29:50.599031+0000","last_unstale":"2024-03-28T23:29:51.591567+0000","last_undegraded":"2024-03-28T23:29:51.591567+0000","last_fullsized":"2024-03-28T23:29:51.591567+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:15:03.216756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686223+0000","last_change":"2024-03-28T23:29:46.555057+0000","last_active":"2024-03-28T23:29:51.686223+0000","last_peered":"2024-03-28T23:29:51.686223+0000","last_clean":"2024-03-28T23:29:51.686223+0000","last_became_active":"2024-03-28T23:29:46.554796+0000","last_became_peered":"2024-03-28T23:29:46.554796+0000","last_unstale":"2024-03-28T23:29:51.686223+0000","last_undegraded":"2024-03-28T23:29:51.686223+0000","last_fullsized":"2024-03-28T23:29:51.686223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:36:20.485861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685609+0000","last_change":"2024-03-28T23:29:45.561273+0000","last_active":"2024-03-28T23:29:51.685609+0000","last_peered":"2024-03-28T23:29:51.685609+0000","last_clean":"2024-03-28T23:29:51.685609+0000","last_became_active":"2024-03-28T23:29:45.561006+0000","last_became_peered":"2024-03-28T23:29:45.561006+0000","last_unstale":"2024-03-28T23:29:51.685609+0000","last_undegraded":"2024-03-28T23:29:51.685609+0000","last_fullsized":"2024-03-28T23:29:51.685609+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:41.621188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687569+0000","last_change":"2024-03-28T23:29:50.706824+0000","last_active":"2024-03-28T23:29:51.687569+0000","last_peered":"2024-03-28T23:29:51.687569+0000","last_clean":"2024-03-28T23:29:51.687569+0000","last_became_active":"2024-03-28T23:29:50.706411+0000","last_became_peered":"2024-03-28T23:29:50.706411+0000","last_unstale":"2024-03-28T23:29:51.687569+0000","last_undegraded":"2024-03-28T23:29:51.687569+0000","last_fullsized":"2024-03-28T23:29:51.687569+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:43:14.415039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.592115+0000","last_change":"2024-03-28T23:29:46.558900+0000","last_active":"2024-03-28T23:29:51.592115+0000","last_peered":"2024-03-28T23:29:51.592115+0000","last_clean":"2024-03-28T23:29:51.592115+0000","last_became_active":"2024-03-28T23:29:46.558628+0000","last_became_peered":"2024-03-28T23:29:46.558628+0000","last_unstale":"2024-03-28T23:29:51.592115+0000","last_undegraded":"2024-03-28T23:29:51.592115+0000","last_fullsized":"2024-03-28T23:29:51.592115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:37.122394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685397+0000","last_change":"2024-03-28T23:29:45.563365+0000","last_active":"2024-03-28T23:29:51.685397+0000","last_peered":"2024-03-28T23:29:51.685397+0000","last_clean":"2024-03-28T23:29:51.685397+0000","last_became_active":"2024-03-28T23:29:45.563093+0000","last_became_peered":"2024-03-28T23:29:45.563093+0000","last_unstale":"2024-03-28T23:29:51.685397+0000","last_undegraded":"2024-03-28T23:29:51.685397+0000","last_fullsized":"2024-03-28T23:29:51.685397+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:01.983721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591200+0000","last_change":"2024-03-28T23:29:50.596634+0000","last_active":"2024-03-28T23:29:51.591200+0000","last_peered":"2024-03-28T23:29:51.591200+0000","last_clean":"2024-03-28T23:29:51.591200+0000","last_became_active":"2024-03-28T23:29:50.596057+0000","last_became_peered":"2024-03-28T23:29:50.596057+0000","last_unstale":"2024-03-28T23:29:51.591200+0000","last_undegraded":"2024-03-28T23:29:51.591200+0000","last_fullsized":"2024-03-28T23:29:51.591200+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:02.708894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687515+0000","last_change":"2024-03-28T23:29:46.578357+0000","last_active":"2024-03-28T23:29:51.687515+0000","last_peered":"2024-03-28T23:29:51.687515+0000","last_clean":"2024-03-28T23:29:51.687515+0000","last_became_active":"2024-03-28T23:29:46.578075+0000","last_became_peered":"2024-03-28T23:29:46.578075+0000","last_unstale":"2024-03-28T23:29:51.687515+0000","last_undegraded":"2024-03-28T23:29:51.687515+0000","last_fullsized":"2024-03-28T23:29:51.687515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:40:22.556997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591419+0000","last_change":"2024-03-28T23:29:45.560606+0000","last_active":"2024-03-28T23:29:51.591419+0000","last_peered":"2024-03-28T23:29:51.591419+0000","last_clean":"2024-03-28T23:29:51.591419+0000","last_became_active":"2024-03-28T23:29:45.560355+0000","last_became_peered":"2024-03-28T23:29:45.560355+0000","last_unstale":"2024-03-28T23:29:51.591419+0000","last_undegraded":"2024-03-28T23:29:51.591419+0000","last_fullsized":"2024-03-28T23:29:51.591419+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:01.983721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591966+0000","last_change":"2024-03-28T23:29:50.708102+0000","last_active":"2024-03-28T23:29:51.591966+0000","last_peered":"2024-03-28T23:29:51.591966+0000","last_clean":"2024-03-28T23:29:51.591966+0000","last_became_active":"2024-03-28T23:29:50.707657+0000","last_became_peered":"2024-03-28T23:29:50.707657+0000","last_unstale":"2024-03-28T23:29:51.591966+0000","last_undegraded":"2024-03-28T23:29:51.591966+0000","last_fullsized":"2024-03-28T23:29:51.591966+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:58.504770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687431+0000","last_change":"2024-03-28T23:29:46.561641+0000","last_active":"2024-03-28T23:29:51.687431+0000","last_peered":"2024-03-28T23:29:51.687431+0000","last_clean":"2024-03-28T23:29:51.687431+0000","last_became_active":"2024-03-28T23:29:46.561380+0000","last_became_peered":"2024-03-28T23:29:46.561380+0000","last_unstale":"2024-03-28T23:29:51.687431+0000","last_undegraded":"2024-03-28T23:29:51.687431+0000","last_fullsized":"2024-03-28T23:29:51.687431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:54.470202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686791+0000","last_change":"2024-03-28T23:29:45.562177+0000","last_active":"2024-03-28T23:29:51.686791+0000","last_peered":"2024-03-28T23:29:51.686791+0000","last_clean":"2024-03-28T23:29:51.686791+0000","last_became_active":"2024-03-28T23:29:45.561852+0000","last_became_peered":"2024-03-28T23:29:45.561852+0000","last_unstale":"2024-03-28T23:29:51.686791+0000","last_undegraded":"2024-03-28T23:29:51.686791+0000","last_fullsized":"2024-03-28T23:29:51.686791+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:36.113647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685836+0000","last_change":"2024-03-28T23:29:50.706769+0000","last_active":"2024-03-28T23:29:51.685836+0000","last_peered":"2024-03-28T23:29:51.685836+0000","last_clean":"2024-03-28T23:29:51.685836+0000","last_became_active":"2024-03-28T23:29:50.706415+0000","last_became_peered":"2024-03-28T23:29:50.706415+0000","last_unstale":"2024-03-28T23:29:51.685836+0000","last_undegraded":"2024-03-28T23:29:51.685836+0000","last_fullsized":"2024-03-28T23:29:51.685836+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:47.094789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687279+0000","last_change":"2024-03-28T23:29:46.560067+0000","last_active":"2024-03-28T23:29:51.687279+0000","last_peered":"2024-03-28T23:29:51.687279+0000","last_clean":"2024-03-28T23:29:51.687279+0000","last_became_active":"2024-03-28T23:29:46.559790+0000","last_became_peered":"2024-03-28T23:29:46.559790+0000","last_unstale":"2024-03-28T23:29:51.687279+0000","last_undegraded":"2024-03-28T23:29:51.687279+0000","last_fullsized":"2024-03-28T23:29:51.687279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:32.936386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686796+0000","last_change":"2024-03-28T23:29:45.565279+0000","last_active":"2024-03-28T23:29:51.686796+0000","last_peered":"2024-03-28T23:29:51.686796+0000","last_clean":"2024-03-28T23:29:51.686796+0000","last_became_active":"2024-03-28T23:29:45.564739+0000","last_became_peered":"2024-03-28T23:29:45.564739+0000","last_unstale":"2024-03-28T23:29:51.686796+0000","last_undegraded":"2024-03-28T23:29:51.686796+0000","last_fullsized":"2024-03-28T23:29:51.686796+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:17.783972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686745+0000","last_change":"2024-03-28T23:29:50.706805+0000","last_active":"2024-03-28T23:29:51.686745+0000","last_peered":"2024-03-28T23:29:51.686745+0000","last_clean":"2024-03-28T23:29:51.686745+0000","last_became_active":"2024-03-28T23:29:50.706429+0000","last_became_peered":"2024-03-28T23:29:50.706429+0000","last_unstale":"2024-03-28T23:29:51.686745+0000","last_undegraded":"2024-03-28T23:29:51.686745+0000","last_fullsized":"2024-03-28T23:29:51.686745+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:06:40.917529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686157+0000","last_change":"2024-03-28T23:29:46.560912+0000","last_active":"2024-03-28T23:29:51.686157+0000","last_peered":"2024-03-28T23:29:51.686157+0000","last_clean":"2024-03-28T23:29:51.686157+0000","last_became_active":"2024-03-28T23:29:46.560613+0000","last_became_peered":"2024-03-28T23:29:46.560613+0000","last_unstale":"2024-03-28T23:29:51.686157+0000","last_undegraded":"2024-03-28T23:29:51.686157+0000","last_fullsized":"2024-03-28T23:29:51.686157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:17:49.979210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687372+0000","last_change":"2024-03-28T23:29:45.561918+0000","last_active":"2024-03-28T23:29:51.687372+0000","last_peered":"2024-03-28T23:29:51.687372+0000","last_clean":"2024-03-28T23:29:51.687372+0000","last_became_active":"2024-03-28T23:29:45.561577+0000","last_became_peered":"2024-03-28T23:29:45.561577+0000","last_unstale":"2024-03-28T23:29:51.687372+0000","last_undegraded":"2024-03-28T23:29:51.687372+0000","last_fullsized":"2024-03-28T23:29:51.687372+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:09.929984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686073+0000","last_change":"2024-03-28T23:29:50.594835+0000","last_active":"2024-03-28T23:29:51.686073+0000","last_peered":"2024-03-28T23:29:51.686073+0000","last_clean":"2024-03-28T23:29:51.686073+0000","last_became_active":"2024-03-28T23:29:50.594167+0000","last_became_peered":"2024-03-28T23:29:50.594167+0000","last_unstale":"2024-03-28T23:29:51.686073+0000","last_undegraded":"2024-03-28T23:29:51.686073+0000","last_fullsized":"2024-03-28T23:29:51.686073+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:06:40.917529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.592269+0000","last_change":"2024-03-28T23:29:46.579930+0000","last_active":"2024-03-28T23:29:51.592269+0000","last_peered":"2024-03-28T23:29:51.592269+0000","last_clean":"2024-03-28T23:29:51.592269+0000","last_became_active":"2024-03-28T23:29:46.579622+0000","last_became_peered":"2024-03-28T23:29:46.579622+0000","last_unstale":"2024-03-28T23:29:51.592269+0000","last_undegraded":"2024-03-28T23:29:51.592269+0000","last_fullsized":"2024-03-28T23:29:51.592269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:18.792719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686219+0000","last_change":"2024-03-28T23:29:45.557802+0000","last_active":"2024-03-28T23:29:51.686219+0000","last_peered":"2024-03-28T23:29:51.686219+0000","last_clean":"2024-03-28T23:29:51.686219+0000","last_became_active":"2024-03-28T23:29:45.557552+0000","last_became_peered":"2024-03-28T23:29:45.557552+0000","last_unstale":"2024-03-28T23:29:51.686219+0000","last_undegraded":"2024-03-28T23:29:51.686219+0000","last_fullsized":"2024-03-28T23:29:51.686219+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:01.983721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687230+0000","last_change":"2024-03-28T23:29:50.601086+0000","last_active":"2024-03-28T23:29:51.687230+0000","last_peered":"2024-03-28T23:29:51.687230+0000","last_clean":"2024-03-28T23:29:51.687230+0000","last_became_active":"2024-03-28T23:29:50.600720+0000","last_became_peered":"2024-03-28T23:29:50.600720+0000","last_unstale":"2024-03-28T23:29:51.687230+0000","last_undegraded":"2024-03-28T23:29:51.687230+0000","last_fullsized":"2024-03-28T23:29:51.687230+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:30:48.379529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685445+0000","last_change":"2024-03-28T23:29:46.560331+0000","last_active":"2024-03-28T23:29:51.685445+0000","last_peered":"2024-03-28T23:29:51.685445+0000","last_clean":"2024-03-28T23:29:51.685445+0000","last_became_active":"2024-03-28T23:29:46.559135+0000","last_became_peered":"2024-03-28T23:29:46.559135+0000","last_unstale":"2024-03-28T23:29:51.685445+0000","last_undegraded":"2024-03-28T23:29:51.685445+0000","last_fullsized":"2024-03-28T23:29:51.685445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:58.674327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589305+0000","last_change":"2024-03-28T23:29:45.559376+0000","last_active":"2024-03-28T23:29:51.589305+0000","last_peered":"2024-03-28T23:29:51.589305+0000","last_clean":"2024-03-28T23:29:51.589305+0000","last_became_active":"2024-03-28T23:29:45.559076+0000","last_became_peered":"2024-03-28T23:29:45.559076+0000","last_unstale":"2024-03-28T23:29:51.589305+0000","last_undegraded":"2024-03-28T23:29:51.589305+0000","last_fullsized":"2024-03-28T23:29:51.589305+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:21.707459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687612+0000","last_change":"2024-03-28T23:29:50.600059+0000","last_active":"2024-03-28T23:29:51.687612+0000","last_peered":"2024-03-28T23:29:51.687612+0000","last_clean":"2024-03-28T23:29:51.687612+0000","last_became_active":"2024-03-28T23:29:50.599692+0000","last_became_peered":"2024-03-28T23:29:50.599692+0000","last_unstale":"2024-03-28T23:29:51.687612+0000","last_undegraded":"2024-03-28T23:29:51.687612+0000","last_fullsized":"2024-03-28T23:29:51.687612+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:22.085225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686352+0000","last_change":"2024-03-28T23:29:46.578521+0000","last_active":"2024-03-28T23:29:51.686352+0000","last_peered":"2024-03-28T23:29:51.686352+0000","last_clean":"2024-03-28T23:29:51.686352+0000","last_became_active":"2024-03-28T23:29:46.578076+0000","last_became_peered":"2024-03-28T23:29:46.578076+0000","last_unstale":"2024-03-28T23:29:51.686352+0000","last_undegraded":"2024-03-28T23:29:51.686352+0000","last_fullsized":"2024-03-28T23:29:51.686352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:20.039954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685395+0000","last_change":"2024-03-28T23:29:45.563206+0000","last_active":"2024-03-28T23:29:51.685395+0000","last_peered":"2024-03-28T23:29:51.685395+0000","last_clean":"2024-03-28T23:29:51.685395+0000","last_became_active":"2024-03-28T23:29:45.562929+0000","last_became_peered":"2024-03-28T23:29:45.562929+0000","last_unstale":"2024-03-28T23:29:51.685395+0000","last_undegraded":"2024-03-28T23:29:51.685395+0000","last_fullsized":"2024-03-28T23:29:51.685395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:41.621188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687463+0000","last_change":"2024-03-28T23:29:50.709169+0000","last_active":"2024-03-28T23:29:51.687463+0000","last_peered":"2024-03-28T23:29:51.687463+0000","last_clean":"2024-03-28T23:29:51.687463+0000","last_became_active":"2024-03-28T23:29:50.707306+0000","last_became_peered":"2024-03-28T23:29:50.707306+0000","last_unstale":"2024-03-28T23:29:51.687463+0000","last_undegraded":"2024-03-28T23:29:51.687463+0000","last_fullsized":"2024-03-28T23:29:51.687463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:22.085225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685675+0000","last_change":"2024-03-28T23:29:46.564922+0000","last_active":"2024-03-28T23:29:51.685675+0000","last_peered":"2024-03-28T23:29:51.685675+0000","last_clean":"2024-03-28T23:29:51.685675+0000","last_became_active":"2024-03-28T23:29:46.564644+0000","last_became_peered":"2024-03-28T23:29:46.564644+0000","last_unstale":"2024-03-28T23:29:51.685675+0000","last_undegraded":"2024-03-28T23:29:51.685675+0000","last_fullsized":"2024-03-28T23:29:51.685675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:58.674327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686316+0000","last_change":"2024-03-28T23:29:45.556656+0000","last_active":"2024-03-28T23:29:51.686316+0000","last_peered":"2024-03-28T23:29:51.686316+0000","last_clean":"2024-03-28T23:29:51.686316+0000","last_became_active":"2024-03-28T23:29:45.556329+0000","last_became_peered":"2024-03-28T23:29:45.556329+0000","last_unstale":"2024-03-28T23:29:51.686316+0000","last_undegraded":"2024-03-28T23:29:51.686316+0000","last_fullsized":"2024-03-28T23:29:51.686316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:21.707459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686403+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:29:51.686403+0000","last_peered":"2024-03-28T23:29:51.686403+0000","last_clean":"2024-03-28T23:29:51.686403+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:29:51.686403+0000","last_undegraded":"2024-03-28T23:29:51.686403+0000","last_fullsized":"2024-03-28T23:29:51.686403+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591319+0000","last_change":"2024-03-28T23:29:50.711182+0000","last_active":"2024-03-28T23:29:51.591319+0000","last_peered":"2024-03-28T23:29:51.591319+0000","last_clean":"2024-03-28T23:29:51.591319+0000","last_became_active":"2024-03-28T23:29:50.710737+0000","last_became_peered":"2024-03-28T23:29:50.710737+0000","last_unstale":"2024-03-28T23:29:51.591319+0000","last_undegraded":"2024-03-28T23:29:51.591319+0000","last_fullsized":"2024-03-28T23:29:51.591319+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:17:54.013778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686573+0000","last_change":"2024-03-28T23:29:46.572966+0000","last_active":"2024-03-28T23:29:51.686573+0000","last_peered":"2024-03-28T23:29:51.686573+0000","last_clean":"2024-03-28T23:29:51.686573+0000","last_became_active":"2024-03-28T23:29:46.572405+0000","last_became_peered":"2024-03-28T23:29:46.572405+0000","last_unstale":"2024-03-28T23:29:51.686573+0000","last_undegraded":"2024-03-28T23:29:51.686573+0000","last_fullsized":"2024-03-28T23:29:51.686573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:06:36.882961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589560+0000","last_change":"2024-03-28T23:29:45.560681+0000","last_active":"2024-03-28T23:29:51.589560+0000","last_peered":"2024-03-28T23:29:51.589560+0000","last_clean":"2024-03-28T23:29:51.589560+0000","last_became_active":"2024-03-28T23:29:45.560420+0000","last_became_peered":"2024-03-28T23:29:45.560420+0000","last_unstale":"2024-03-28T23:29:51.589560+0000","last_undegraded":"2024-03-28T23:29:51.589560+0000","last_fullsized":"2024-03-28T23:29:51.589560+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:17:17.207826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.590530+0000","last_change":"2024-03-28T23:29:50.711247+0000","last_active":"2024-03-28T23:29:51.590530+0000","last_peered":"2024-03-28T23:29:51.590530+0000","last_clean":"2024-03-28T23:29:51.590530+0000","last_became_active":"2024-03-28T23:29:50.710836+0000","last_became_peered":"2024-03-28T23:29:50.710836+0000","last_unstale":"2024-03-28T23:29:51.590530+0000","last_undegraded":"2024-03-28T23:29:51.590530+0000","last_fullsized":"2024-03-28T23:29:51.590530+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:58.504770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685843+0000","last_change":"2024-03-28T23:29:46.561283+0000","last_active":"2024-03-28T23:29:51.685843+0000","last_peered":"2024-03-28T23:29:51.685843+0000","last_clean":"2024-03-28T23:29:51.685843+0000","last_became_active":"2024-03-28T23:29:46.560584+0000","last_became_peered":"2024-03-28T23:29:46.560584+0000","last_unstale":"2024-03-28T23:29:51.685843+0000","last_undegraded":"2024-03-28T23:29:51.685843+0000","last_fullsized":"2024-03-28T23:29:51.685843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:06:36.882961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686047+0000","last_change":"2024-03-28T23:29:45.560013+0000","last_active":"2024-03-28T23:29:51.686047+0000","last_peered":"2024-03-28T23:29:51.686047+0000","last_clean":"2024-03-28T23:29:51.686047+0000","last_became_active":"2024-03-28T23:29:45.559643+0000","last_became_peered":"2024-03-28T23:29:45.559643+0000","last_unstale":"2024-03-28T23:29:51.686047+0000","last_undegraded":"2024-03-28T23:29:51.686047+0000","last_fullsized":"2024-03-28T23:29:51.686047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:01.983721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687005+0000","last_change":"2024-03-28T23:29:50.595120+0000","last_active":"2024-03-28T23:29:51.687005+0000","last_peered":"2024-03-28T23:29:51.687005+0000","last_clean":"2024-03-28T23:29:51.687005+0000","last_became_active":"2024-03-28T23:29:50.594780+0000","last_became_peered":"2024-03-28T23:29:50.594780+0000","last_unstale":"2024-03-28T23:29:51.687005+0000","last_undegraded":"2024-03-28T23:29:51.687005+0000","last_fullsized":"2024-03-28T23:29:51.687005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:06.894069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686843+0000","last_change":"2024-03-28T23:29:46.567707+0000","last_active":"2024-03-28T23:29:51.686843+0000","last_peered":"2024-03-28T23:29:51.686843+0000","last_clean":"2024-03-28T23:29:51.686843+0000","last_became_active":"2024-03-28T23:29:46.565675+0000","last_became_peered":"2024-03-28T23:29:46.565675+0000","last_unstale":"2024-03-28T23:29:51.686843+0000","last_undegraded":"2024-03-28T23:29:51.686843+0000","last_fullsized":"2024-03-28T23:29:51.686843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:07:21.524344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589416+0000","last_change":"2024-03-28T23:29:45.558538+0000","last_active":"2024-03-28T23:29:51.589416+0000","last_peered":"2024-03-28T23:29:51.589416+0000","last_clean":"2024-03-28T23:29:51.589416+0000","last_became_active":"2024-03-28T23:29:45.558248+0000","last_became_peered":"2024-03-28T23:29:45.558248+0000","last_unstale":"2024-03-28T23:29:51.589416+0000","last_undegraded":"2024-03-28T23:29:51.589416+0000","last_fullsized":"2024-03-28T23:29:51.589416+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:59.719252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685418+0000","last_change":"2024-03-28T23:29:50.707640+0000","last_active":"2024-03-28T23:29:51.685418+0000","last_peered":"2024-03-28T23:29:51.685418+0000","last_clean":"2024-03-28T23:29:51.685418+0000","last_became_active":"2024-03-28T23:29:50.707258+0000","last_became_peered":"2024-03-28T23:29:50.707258+0000","last_unstale":"2024-03-28T23:29:51.685418+0000","last_undegraded":"2024-03-28T23:29:51.685418+0000","last_fullsized":"2024-03-28T23:29:51.685418+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:47.094789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686505+0000","last_change":"2024-03-28T23:29:46.565431+0000","last_active":"2024-03-28T23:29:51.686505+0000","last_peered":"2024-03-28T23:29:51.686505+0000","last_clean":"2024-03-28T23:29:51.686505+0000","last_became_active":"2024-03-28T23:29:46.565163+0000","last_became_peered":"2024-03-28T23:29:46.565163+0000","last_unstale":"2024-03-28T23:29:51.686505+0000","last_undegraded":"2024-03-28T23:29:51.686505+0000","last_fullsized":"2024-03-28T23:29:51.686505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:02.859502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591420+0000","last_change":"2024-03-28T23:29:45.564138+0000","last_active":"2024-03-28T23:29:51.591420+0000","last_peered":"2024-03-28T23:29:51.591420+0000","last_clean":"2024-03-28T23:29:51.591420+0000","last_became_active":"2024-03-28T23:29:45.563858+0000","last_became_peered":"2024-03-28T23:29:45.563858+0000","last_unstale":"2024-03-28T23:29:51.591420+0000","last_undegraded":"2024-03-28T23:29:51.591420+0000","last_fullsized":"2024-03-28T23:29:51.591420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:42:38.893826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.592126+0000","last_change":"2024-03-28T23:29:50.599484+0000","last_active":"2024-03-28T23:29:51.592126+0000","last_peered":"2024-03-28T23:29:51.592126+0000","last_clean":"2024-03-28T23:29:51.592126+0000","last_became_active":"2024-03-28T23:29:50.599035+0000","last_became_peered":"2024-03-28T23:29:50.599035+0000","last_unstale":"2024-03-28T23:29:51.592126+0000","last_undegraded":"2024-03-28T23:29:51.592126+0000","last_fullsized":"2024-03-28T23:29:51.592126+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:10.765830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686919+0000","last_change":"2024-03-28T23:29:46.567472+0000","last_active":"2024-03-28T23:29:51.686919+0000","last_peered":"2024-03-28T23:29:51.686919+0000","last_clean":"2024-03-28T23:29:51.686919+0000","last_became_active":"2024-03-28T23:29:46.565485+0000","last_became_peered":"2024-03-28T23:29:46.565485+0000","last_unstale":"2024-03-28T23:29:51.686919+0000","last_undegraded":"2024-03-28T23:29:51.686919+0000","last_fullsized":"2024-03-28T23:29:51.686919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:29:20.669429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589103+0000","last_change":"2024-03-28T23:29:45.556004+0000","last_active":"2024-03-28T23:29:51.589103+0000","last_peered":"2024-03-28T23:29:51.589103+0000","last_clean":"2024-03-28T23:29:51.589103+0000","last_became_active":"2024-03-28T23:29:45.555701+0000","last_became_peered":"2024-03-28T23:29:45.555701+0000","last_unstale":"2024-03-28T23:29:51.589103+0000","last_undegraded":"2024-03-28T23:29:51.589103+0000","last_fullsized":"2024-03-28T23:29:51.589103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:06:18.025605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687430+0000","last_change":"2024-03-28T23:29:50.596439+0000","last_active":"2024-03-28T23:29:51.687430+0000","last_peered":"2024-03-28T23:29:51.687430+0000","last_clean":"2024-03-28T23:29:51.687430+0000","last_became_active":"2024-03-28T23:29:50.596065+0000","last_became_peered":"2024-03-28T23:29:50.596065+0000","last_unstale":"2024-03-28T23:29:51.687430+0000","last_undegraded":"2024-03-28T23:29:51.687430+0000","last_fullsized":"2024-03-28T23:29:51.687430+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:43:14.415039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.590719+0000","last_change":"2024-03-28T23:29:46.581708+0000","last_active":"2024-03-28T23:29:51.590719+0000","last_peered":"2024-03-28T23:29:51.590719+0000","last_clean":"2024-03-28T23:29:51.590719+0000","last_became_active":"2024-03-28T23:29:46.581391+0000","last_became_peered":"2024-03-28T23:29:46.581391+0000","last_unstale":"2024-03-28T23:29:51.590719+0000","last_undegraded":"2024-03-28T23:29:51.590719+0000","last_fullsized":"2024-03-28T23:29:51.590719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:24:19.353047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591187+0000","last_change":"2024-03-28T23:29:45.560112+0000","last_active":"2024-03-28T23:29:51.591187+0000","last_peered":"2024-03-28T23:29:51.591187+0000","last_clean":"2024-03-28T23:29:51.591187+0000","last_became_active":"2024-03-28T23:29:45.559817+0000","last_became_peered":"2024-03-28T23:29:45.559817+0000","last_unstale":"2024-03-28T23:29:51.591187+0000","last_undegraded":"2024-03-28T23:29:51.591187+0000","last_fullsized":"2024-03-28T23:29:51.591187+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:21.707459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687297+0000","last_change":"2024-03-28T23:29:50.712185+0000","last_active":"2024-03-28T23:29:51.687297+0000","last_peered":"2024-03-28T23:29:51.687297+0000","last_clean":"2024-03-28T23:29:51.687297+0000","last_became_active":"2024-03-28T23:29:50.711823+0000","last_became_peered":"2024-03-28T23:29:50.711823+0000","last_unstale":"2024-03-28T23:29:51.687297+0000","last_undegraded":"2024-03-28T23:29:51.687297+0000","last_fullsized":"2024-03-28T23:29:51.687297+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:40.107649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685755+0000","last_change":"2024-03-28T23:29:46.570826+0000","last_active":"2024-03-28T23:29:51.685755+0000","last_peered":"2024-03-28T23:29:51.685755+0000","last_clean":"2024-03-28T23:29:51.685755+0000","last_became_active":"2024-03-28T23:29:46.570513+0000","last_became_peered":"2024-03-28T23:29:46.570513+0000","last_unstale":"2024-03-28T23:29:51.685755+0000","last_undegraded":"2024-03-28T23:29:51.685755+0000","last_fullsized":"2024-03-28T23:29:51.685755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:08:33.801844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686189+0000","last_change":"2024-03-28T23:29:45.554954+0000","last_active":"2024-03-28T23:29:51.686189+0000","last_peered":"2024-03-28T23:29:51.686189+0000","last_clean":"2024-03-28T23:29:51.686189+0000","last_became_active":"2024-03-28T23:29:45.554687+0000","last_became_peered":"2024-03-28T23:29:45.554687+0000","last_unstale":"2024-03-28T23:29:51.686189+0000","last_undegraded":"2024-03-28T23:29:51.686189+0000","last_fullsized":"2024-03-28T23:29:51.686189+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:24:58.784056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687871+0000","last_change":"2024-03-28T23:29:50.707207+0000","last_active":"2024-03-28T23:29:51.687871+0000","last_peered":"2024-03-28T23:29:51.687871+0000","last_clean":"2024-03-28T23:29:51.687871+0000","last_became_active":"2024-03-28T23:29:50.706585+0000","last_became_peered":"2024-03-28T23:29:50.706585+0000","last_unstale":"2024-03-28T23:29:51.687871+0000","last_undegraded":"2024-03-28T23:29:51.687871+0000","last_fullsized":"2024-03-28T23:29:51.687871+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:10:11.936256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687068+0000","last_change":"2024-03-28T23:29:46.578519+0000","last_active":"2024-03-28T23:29:51.687068+0000","last_peered":"2024-03-28T23:29:51.687068+0000","last_clean":"2024-03-28T23:29:51.687068+0000","last_became_active":"2024-03-28T23:29:46.577803+0000","last_became_peered":"2024-03-28T23:29:46.577803+0000","last_unstale":"2024-03-28T23:29:51.687068+0000","last_undegraded":"2024-03-28T23:29:51.687068+0000","last_fullsized":"2024-03-28T23:29:51.687068+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:32.936386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687237+0000","last_change":"2024-03-28T23:29:45.557114+0000","last_active":"2024-03-28T23:29:51.687237+0000","last_peered":"2024-03-28T23:29:51.687237+0000","last_clean":"2024-03-28T23:29:51.687237+0000","last_became_active":"2024-03-28T23:29:45.556800+0000","last_became_peered":"2024-03-28T23:29:45.556800+0000","last_unstale":"2024-03-28T23:29:51.687237+0000","last_undegraded":"2024-03-28T23:29:51.687237+0000","last_fullsized":"2024-03-28T23:29:51.687237+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:06:18.025605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686083+0000","last_change":"2024-03-28T23:29:50.707356+0000","last_active":"2024-03-28T23:29:51.686083+0000","last_peered":"2024-03-28T23:29:51.686083+0000","last_clean":"2024-03-28T23:29:51.686083+0000","last_became_active":"2024-03-28T23:29:50.706320+0000","last_became_peered":"2024-03-28T23:29:50.706320+0000","last_unstale":"2024-03-28T23:29:51.686083+0000","last_undegraded":"2024-03-28T23:29:51.686083+0000","last_fullsized":"2024-03-28T23:29:51.686083+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:36.341186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686444+0000","last_change":"2024-03-28T23:29:46.573192+0000","last_active":"2024-03-28T23:29:51.686444+0000","last_peered":"2024-03-28T23:29:51.686444+0000","last_clean":"2024-03-28T23:29:51.686444+0000","last_became_active":"2024-03-28T23:29:46.572701+0000","last_became_peered":"2024-03-28T23:29:46.572701+0000","last_unstale":"2024-03-28T23:29:51.686444+0000","last_undegraded":"2024-03-28T23:29:51.686444+0000","last_fullsized":"2024-03-28T23:29:51.686444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:17:49.979210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:56.151164+0000","last_change":"2024-03-28T23:29:45.558929+0000","last_active":"2024-03-28T23:29:56.151164+0000","last_peered":"2024-03-28T23:29:56.151164+0000","last_clean":"2024-03-28T23:29:56.151164+0000","last_became_active":"2024-03-28T23:29:45.558657+0000","last_became_peered":"2024-03-28T23:29:45.558657+0000","last_unstale":"2024-03-28T23:29:56.151164+0000","last_undegraded":"2024-03-28T23:29:56.151164+0000","last_fullsized":"2024-03-28T23:29:56.151164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:24:58.784056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686265+0000","last_change":"2024-03-28T23:29:50.594405+0000","last_active":"2024-03-28T23:29:51.686265+0000","last_peered":"2024-03-28T23:29:51.686265+0000","last_clean":"2024-03-28T23:29:51.686265+0000","last_became_active":"2024-03-28T23:29:50.594017+0000","last_became_peered":"2024-03-28T23:29:50.594017+0000","last_unstale":"2024-03-28T23:29:51.686265+0000","last_undegraded":"2024-03-28T23:29:51.686265+0000","last_fullsized":"2024-03-28T23:29:51.686265+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:06.894069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591773+0000","last_change":"2024-03-28T23:29:46.561971+0000","last_active":"2024-03-28T23:29:51.591773+0000","last_peered":"2024-03-28T23:29:51.591773+0000","last_clean":"2024-03-28T23:29:51.591773+0000","last_became_active":"2024-03-28T23:29:46.561684+0000","last_became_peered":"2024-03-28T23:29:46.561684+0000","last_unstale":"2024-03-28T23:29:51.591773+0000","last_undegraded":"2024-03-28T23:29:51.591773+0000","last_fullsized":"2024-03-28T23:29:51.591773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:42.629935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685953+0000","last_change":"2024-03-28T23:29:45.563130+0000","last_active":"2024-03-28T23:29:51.685953+0000","last_peered":"2024-03-28T23:29:51.685953+0000","last_clean":"2024-03-28T23:29:51.685953+0000","last_became_active":"2024-03-28T23:29:45.562696+0000","last_became_peered":"2024-03-28T23:29:45.562696+0000","last_unstale":"2024-03-28T23:29:51.685953+0000","last_undegraded":"2024-03-28T23:29:51.685953+0000","last_fullsized":"2024-03-28T23:29:51.685953+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:17.783972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.592114+0000","last_change":"2024-03-28T23:29:50.587580+0000","last_active":"2024-03-28T23:29:51.592114+0000","last_peered":"2024-03-28T23:29:51.592114+0000","last_clean":"2024-03-28T23:29:51.592114+0000","last_became_active":"2024-03-28T23:29:50.587175+0000","last_became_peered":"2024-03-28T23:29:50.587175+0000","last_unstale":"2024-03-28T23:29:51.592114+0000","last_undegraded":"2024-03-28T23:29:51.592114+0000","last_fullsized":"2024-03-28T23:29:51.592114+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:06:40.917529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687664+0000","last_change":"2024-03-28T23:29:46.578684+0000","last_active":"2024-03-28T23:29:51.687664+0000","last_peered":"2024-03-28T23:29:51.687664+0000","last_clean":"2024-03-28T23:29:51.687664+0000","last_became_active":"2024-03-28T23:29:46.577924+0000","last_became_peered":"2024-03-28T23:29:46.577924+0000","last_unstale":"2024-03-28T23:29:51.687664+0000","last_undegraded":"2024-03-28T23:29:51.687664+0000","last_fullsized":"2024-03-28T23:29:51.687664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:36:20.485861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685874+0000","last_change":"2024-03-28T23:29:45.559534+0000","last_active":"2024-03-28T23:29:51.685874+0000","last_peered":"2024-03-28T23:29:51.685874+0000","last_clean":"2024-03-28T23:29:51.685874+0000","last_became_active":"2024-03-28T23:29:45.559243+0000","last_became_peered":"2024-03-28T23:29:45.559243+0000","last_unstale":"2024-03-28T23:29:51.685874+0000","last_undegraded":"2024-03-28T23:29:51.685874+0000","last_fullsized":"2024-03-28T23:29:51.685874+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:17:17.207826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686108+0000","last_change":"2024-03-28T23:29:50.601743+0000","last_active":"2024-03-28T23:29:51.686108+0000","last_peered":"2024-03-28T23:29:51.686108+0000","last_clean":"2024-03-28T23:29:51.686108+0000","last_became_active":"2024-03-28T23:29:50.601327+0000","last_became_peered":"2024-03-28T23:29:50.601327+0000","last_unstale":"2024-03-28T23:29:51.686108+0000","last_undegraded":"2024-03-28T23:29:51.686108+0000","last_fullsized":"2024-03-28T23:29:51.686108+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:36.341186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685724+0000","last_change":"2024-03-28T23:29:46.573188+0000","last_active":"2024-03-28T23:29:51.685724+0000","last_peered":"2024-03-28T23:29:51.685724+0000","last_clean":"2024-03-28T23:29:51.685724+0000","last_became_active":"2024-03-28T23:29:46.572885+0000","last_became_peered":"2024-03-28T23:29:46.572885+0000","last_unstale":"2024-03-28T23:29:51.685724+0000","last_undegraded":"2024-03-28T23:29:51.685724+0000","last_fullsized":"2024-03-28T23:29:51.685724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:08:33.801844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591244+0000","last_change":"2024-03-28T23:29:45.564128+0000","last_active":"2024-03-28T23:29:51.591244+0000","last_peered":"2024-03-28T23:29:51.591244+0000","last_clean":"2024-03-28T23:29:51.591244+0000","last_became_active":"2024-03-28T23:29:45.563837+0000","last_became_peered":"2024-03-28T23:29:45.563837+0000","last_unstale":"2024-03-28T23:29:51.591244+0000","last_undegraded":"2024-03-28T23:29:51.591244+0000","last_fullsized":"2024-03-28T23:29:51.591244+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:17:17.207826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686626+0000","last_change":"2024-03-28T23:29:50.708325+0000","last_active":"2024-03-28T23:29:51.686626+0000","last_peered":"2024-03-28T23:29:51.686626+0000","last_clean":"2024-03-28T23:29:51.686626+0000","last_became_active":"2024-03-28T23:29:50.707929+0000","last_became_peered":"2024-03-28T23:29:50.707929+0000","last_unstale":"2024-03-28T23:29:51.686626+0000","last_undegraded":"2024-03-28T23:29:51.686626+0000","last_fullsized":"2024-03-28T23:29:51.686626+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:01.743074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687303+0000","last_change":"2024-03-28T23:29:50.711341+0000","last_active":"2024-03-28T23:29:51.687303+0000","last_peered":"2024-03-28T23:29:51.687303+0000","last_clean":"2024-03-28T23:29:51.687303+0000","last_became_active":"2024-03-28T23:29:50.710983+0000","last_became_peered":"2024-03-28T23:29:50.710983+0000","last_unstale":"2024-03-28T23:29:51.687303+0000","last_undegraded":"2024-03-28T23:29:51.687303+0000","last_fullsized":"2024-03-28T23:29:51.687303+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:30:48.379529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685433+0000","last_change":"2024-03-28T23:29:46.561654+0000","last_active":"2024-03-28T23:29:51.685433+0000","last_peered":"2024-03-28T23:29:51.685433+0000","last_clean":"2024-03-28T23:29:51.685433+0000","last_became_active":"2024-03-28T23:29:46.561250+0000","last_became_peered":"2024-03-28T23:29:46.561250+0000","last_unstale":"2024-03-28T23:29:51.685433+0000","last_undegraded":"2024-03-28T23:29:51.685433+0000","last_fullsized":"2024-03-28T23:29:51.685433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:38.074388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685648+0000","last_change":"2024-03-28T23:29:45.553472+0000","last_active":"2024-03-28T23:29:51.685648+0000","last_peered":"2024-03-28T23:29:51.685648+0000","last_clean":"2024-03-28T23:29:51.685648+0000","last_became_active":"2024-03-28T23:29:45.553154+0000","last_became_peered":"2024-03-28T23:29:45.553154+0000","last_unstale":"2024-03-28T23:29:51.685648+0000","last_undegraded":"2024-03-28T23:29:51.685648+0000","last_fullsized":"2024-03-28T23:29:51.685648+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:36.113647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591126+0000","last_change":"2024-03-28T23:29:50.600387+0000","last_active":"2024-03-28T23:29:51.591126+0000","last_peered":"2024-03-28T23:29:51.591126+0000","last_clean":"2024-03-28T23:29:51.591126+0000","last_became_active":"2024-03-28T23:29:50.599850+0000","last_became_peered":"2024-03-28T23:29:50.599850+0000","last_unstale":"2024-03-28T23:29:51.591126+0000","last_undegraded":"2024-03-28T23:29:51.591126+0000","last_fullsized":"2024-03-28T23:29:51.591126+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:08:37.836411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686149+0000","last_change":"2024-03-28T23:29:46.581533+0000","last_active":"2024-03-28T23:29:51.686149+0000","last_peered":"2024-03-28T23:29:51.686149+0000","last_clean":"2024-03-28T23:29:51.686149+0000","last_became_active":"2024-03-28T23:29:46.581288+0000","last_became_peered":"2024-03-28T23:29:46.581288+0000","last_unstale":"2024-03-28T23:29:51.686149+0000","last_undegraded":"2024-03-28T23:29:51.686149+0000","last_fullsized":"2024-03-28T23:29:51.686149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:24:19.353047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686836+0000","last_change":"2024-03-28T23:29:45.561047+0000","last_active":"2024-03-28T23:29:51.686836+0000","last_peered":"2024-03-28T23:29:51.686836+0000","last_clean":"2024-03-28T23:29:51.686836+0000","last_became_active":"2024-03-28T23:29:45.560778+0000","last_became_peered":"2024-03-28T23:29:45.560778+0000","last_unstale":"2024-03-28T23:29:51.686836+0000","last_undegraded":"2024-03-28T23:29:51.686836+0000","last_fullsized":"2024-03-28T23:29:51.686836+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:21.707459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.590800+0000","last_change":"2024-03-28T23:29:45.559446+0000","last_active":"2024-03-28T23:29:51.590800+0000","last_peered":"2024-03-28T23:29:51.590800+0000","last_clean":"2024-03-28T23:29:51.590800+0000","last_became_active":"2024-03-28T23:29:45.559015+0000","last_became_peered":"2024-03-28T23:29:45.559015+0000","last_unstale":"2024-03-28T23:29:51.590800+0000","last_undegraded":"2024-03-28T23:29:51.590800+0000","last_fullsized":"2024-03-28T23:29:51.590800+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:59.719252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687152+0000","last_change":"2024-03-28T23:29:46.578355+0000","last_active":"2024-03-28T23:29:51.687152+0000","last_peered":"2024-03-28T23:29:51.687152+0000","last_clean":"2024-03-28T23:29:51.687152+0000","last_became_active":"2024-03-28T23:29:46.577898+0000","last_became_peered":"2024-03-28T23:29:46.577898+0000","last_unstale":"2024-03-28T23:29:51.687152+0000","last_undegraded":"2024-03-28T23:29:51.687152+0000","last_fullsized":"2024-03-28T23:29:51.687152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:58.674327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589392+0000","last_change":"2024-03-28T23:29:50.601015+0000","last_active":"2024-03-28T23:29:51.589392+0000","last_peered":"2024-03-28T23:29:51.589392+0000","last_clean":"2024-03-28T23:29:51.589392+0000","last_became_active":"2024-03-28T23:29:50.600573+0000","last_became_peered":"2024-03-28T23:29:50.600573+0000","last_unstale":"2024-03-28T23:29:51.589392+0000","last_undegraded":"2024-03-28T23:29:51.589392+0000","last_fullsized":"2024-03-28T23:29:51.589392+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:38:05.665416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686050+0000","last_change":"2024-03-28T23:29:45.556159+0000","last_active":"2024-03-28T23:29:51.686050+0000","last_peered":"2024-03-28T23:29:51.686050+0000","last_clean":"2024-03-28T23:29:51.686050+0000","last_became_active":"2024-03-28T23:29:45.555866+0000","last_became_peered":"2024-03-28T23:29:45.555866+0000","last_unstale":"2024-03-28T23:29:51.686050+0000","last_undegraded":"2024-03-28T23:29:51.686050+0000","last_fullsized":"2024-03-28T23:29:51.686050+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:17:17.207826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591783+0000","last_change":"2024-03-28T23:29:46.554056+0000","last_active":"2024-03-28T23:29:51.591783+0000","last_peered":"2024-03-28T23:29:51.591783+0000","last_clean":"2024-03-28T23:29:51.591783+0000","last_became_active":"2024-03-28T23:29:46.553772+0000","last_became_peered":"2024-03-28T23:29:46.553772+0000","last_unstale":"2024-03-28T23:29:51.591783+0000","last_undegraded":"2024-03-28T23:29:51.591783+0000","last_fullsized":"2024-03-28T23:29:51.591783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:02.992468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686381+0000","last_change":"2024-03-28T23:29:50.603254+0000","last_active":"2024-03-28T23:29:51.686381+0000","last_peered":"2024-03-28T23:29:51.686381+0000","last_clean":"2024-03-28T23:29:51.686381+0000","last_became_active":"2024-03-28T23:29:50.602868+0000","last_became_peered":"2024-03-28T23:29:50.602868+0000","last_unstale":"2024-03-28T23:29:51.686381+0000","last_undegraded":"2024-03-28T23:29:51.686381+0000","last_fullsized":"2024-03-28T23:29:51.686381+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:42:43.014557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.590502+0000","last_change":"2024-03-28T23:29:46.581634+0000","last_active":"2024-03-28T23:29:51.590502+0000","last_peered":"2024-03-28T23:29:51.590502+0000","last_clean":"2024-03-28T23:29:51.590502+0000","last_became_active":"2024-03-28T23:29:46.581353+0000","last_became_peered":"2024-03-28T23:29:46.581353+0000","last_unstale":"2024-03-28T23:29:51.590502+0000","last_undegraded":"2024-03-28T23:29:51.590502+0000","last_fullsized":"2024-03-28T23:29:51.590502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:20.039954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589110+0000","last_change":"2024-03-28T23:29:46.563584+0000","last_active":"2024-03-28T23:29:51.589110+0000","last_peered":"2024-03-28T23:29:51.589110+0000","last_clean":"2024-03-28T23:29:51.589110+0000","last_became_active":"2024-03-28T23:29:46.563265+0000","last_became_peered":"2024-03-28T23:29:46.563265+0000","last_unstale":"2024-03-28T23:29:51.589110+0000","last_undegraded":"2024-03-28T23:29:51.589110+0000","last_fullsized":"2024-03-28T23:29:51.589110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:42.629935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685798+0000","last_change":"2024-03-28T23:29:46.561377+0000","last_active":"2024-03-28T23:29:51.685798+0000","last_peered":"2024-03-28T23:29:51.685798+0000","last_clean":"2024-03-28T23:29:51.685798+0000","last_became_active":"2024-03-28T23:29:46.561091+0000","last_became_peered":"2024-03-28T23:29:46.561091+0000","last_unstale":"2024-03-28T23:29:51.685798+0000","last_undegraded":"2024-03-28T23:29:51.685798+0000","last_fullsized":"2024-03-28T23:29:51.685798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:30:44.344961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687062+0000","last_change":"2024-03-28T23:29:46.578999+0000","last_active":"2024-03-28T23:29:51.687062+0000","last_peered":"2024-03-28T23:29:51.687062+0000","last_clean":"2024-03-28T23:29:51.687062+0000","last_became_active":"2024-03-28T23:29:46.578263+0000","last_became_peered":"2024-03-28T23:29:46.578263+0000","last_unstale":"2024-03-28T23:29:51.687062+0000","last_undegraded":"2024-03-28T23:29:51.687062+0000","last_fullsized":"2024-03-28T23:29:51.687062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:10.853710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687925+0000","last_change":"2024-03-28T23:29:46.566611+0000","last_active":"2024-03-28T23:29:51.687925+0000","last_peered":"2024-03-28T23:29:51.687925+0000","last_clean":"2024-03-28T23:29:51.687925+0000","last_became_active":"2024-03-28T23:29:46.566361+0000","last_became_peered":"2024-03-28T23:29:46.566361+0000","last_unstale":"2024-03-28T23:29:51.687925+0000","last_undegraded":"2024-03-28T23:29:51.687925+0000","last_fullsized":"2024-03-28T23:29:51.687925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:29:20.669429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685784+0000","last_change":"2024-03-28T23:29:46.561125+0000","last_active":"2024-03-28T23:29:51.685784+0000","last_peered":"2024-03-28T23:29:51.685784+0000","last_clean":"2024-03-28T23:29:51.685784+0000","last_became_active":"2024-03-28T23:29:46.559223+0000","last_became_peered":"2024-03-28T23:29:46.559223+0000","last_unstale":"2024-03-28T23:29:51.685784+0000","last_undegraded":"2024-03-28T23:29:51.685784+0000","last_fullsized":"2024-03-28T23:29:51.685784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:36:20.485861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591456+0000","last_change":"2024-03-28T23:29:46.574608+0000","last_active":"2024-03-28T23:29:51.591456+0000","last_peered":"2024-03-28T23:29:51.591456+0000","last_clean":"2024-03-28T23:29:51.591456+0000","last_became_active":"2024-03-28T23:29:46.574329+0000","last_became_peered":"2024-03-28T23:29:46.574329+0000","last_unstale":"2024-03-28T23:29:51.591456+0000","last_undegraded":"2024-03-28T23:29:51.591456+0000","last_fullsized":"2024-03-28T23:29:51.591456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:15.386423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687344+0000","last_change":"2024-03-28T23:29:46.566187+0000","last_active":"2024-03-28T23:29:51.687344+0000","last_peered":"2024-03-28T23:29:51.687344+0000","last_clean":"2024-03-28T23:29:51.687344+0000","last_became_active":"2024-03-28T23:29:46.565873+0000","last_became_peered":"2024-03-28T23:29:46.565873+0000","last_unstale":"2024-03-28T23:29:51.687344+0000","last_undegraded":"2024-03-28T23:29:51.687344+0000","last_fullsized":"2024-03-28T23:29:51.687344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:14:59.182189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685520+0000","last_change":"2024-03-28T23:29:46.559531+0000","last_active":"2024-03-28T23:29:51.685520+0000","last_peered":"2024-03-28T23:29:51.685520+0000","last_clean":"2024-03-28T23:29:51.685520+0000","last_became_active":"2024-03-28T23:29:46.559236+0000","last_became_peered":"2024-03-28T23:29:46.559236+0000","last_unstale":"2024-03-28T23:29:51.685520+0000","last_undegraded":"2024-03-28T23:29:51.685520+0000","last_fullsized":"2024-03-28T23:29:51.685520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:06.731262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686762+0000","last_change":"2024-03-28T23:29:46.555898+0000","last_active":"2024-03-28T23:29:51.686762+0000","last_peered":"2024-03-28T23:29:51.686762+0000","last_clean":"2024-03-28T23:29:51.686762+0000","last_became_active":"2024-03-28T23:29:46.555610+0000","last_became_peered":"2024-03-28T23:29:46.555610+0000","last_unstale":"2024-03-28T23:29:51.686762+0000","last_undegraded":"2024-03-28T23:29:51.686762+0000","last_fullsized":"2024-03-28T23:29:51.686762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:38.074388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685565+0000","last_change":"2024-03-28T23:29:46.573905+0000","last_active":"2024-03-28T23:29:51.685565+0000","last_peered":"2024-03-28T23:29:51.685565+0000","last_clean":"2024-03-28T23:29:51.685565+0000","last_became_active":"2024-03-28T23:29:46.573631+0000","last_became_peered":"2024-03-28T23:29:46.573631+0000","last_unstale":"2024-03-28T23:29:51.685565+0000","last_undegraded":"2024-03-28T23:29:51.685565+0000","last_fullsized":"2024-03-28T23:29:51.685565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:14:59.182189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686795+0000","last_change":"2024-03-28T23:29:46.579400+0000","last_active":"2024-03-28T23:29:51.686795+0000","last_peered":"2024-03-28T23:29:51.686795+0000","last_clean":"2024-03-28T23:29:51.686795+0000","last_became_active":"2024-03-28T23:29:46.579070+0000","last_became_peered":"2024-03-28T23:29:46.579070+0000","last_unstale":"2024-03-28T23:29:51.686795+0000","last_undegraded":"2024-03-28T23:29:51.686795+0000","last_fullsized":"2024-03-28T23:29:51.686795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:15.386423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591130+0000","last_change":"2024-03-28T23:29:46.569441+0000","last_active":"2024-03-28T23:29:51.591130+0000","last_peered":"2024-03-28T23:29:51.591130+0000","last_clean":"2024-03-28T23:29:51.591130+0000","last_became_active":"2024-03-28T23:29:46.569151+0000","last_became_peered":"2024-03-28T23:29:46.569151+0000","last_unstale":"2024-03-28T23:29:51.591130+0000","last_undegraded":"2024-03-28T23:29:51.591130+0000","last_fullsized":"2024-03-28T23:29:51.591130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:38.074388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686720+0000","last_change":"2024-03-28T23:29:46.573927+0000","last_active":"2024-03-28T23:29:51.686720+0000","last_peered":"2024-03-28T23:29:51.686720+0000","last_clean":"2024-03-28T23:29:51.686720+0000","last_became_active":"2024-03-28T23:29:46.573668+0000","last_became_peered":"2024-03-28T23:29:46.573668+0000","last_unstale":"2024-03-28T23:29:51.686720+0000","last_undegraded":"2024-03-28T23:29:51.686720+0000","last_fullsized":"2024-03-28T23:29:51.686720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:30:44.344961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589322+0000","last_change":"2024-03-28T23:29:46.558629+0000","last_active":"2024-03-28T23:29:51.589322+0000","last_peered":"2024-03-28T23:29:51.589322+0000","last_clean":"2024-03-28T23:29:51.589322+0000","last_became_active":"2024-03-28T23:29:46.558311+0000","last_became_peered":"2024-03-28T23:29:46.558311+0000","last_unstale":"2024-03-28T23:29:51.589322+0000","last_undegraded":"2024-03-28T23:29:51.589322+0000","last_fullsized":"2024-03-28T23:29:51.589322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:10.938731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686262+0000","last_change":"2024-03-28T23:29:46.573394+0000","last_active":"2024-03-28T23:29:51.686262+0000","last_peered":"2024-03-28T23:29:51.686262+0000","last_clean":"2024-03-28T23:29:51.686262+0000","last_became_active":"2024-03-28T23:29:46.573130+0000","last_became_peered":"2024-03-28T23:29:46.573130+0000","last_unstale":"2024-03-28T23:29:51.686262+0000","last_undegraded":"2024-03-28T23:29:51.686262+0000","last_fullsized":"2024-03-28T23:29:51.686262+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:43:10.380471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685367+0000","last_change":"2024-03-28T23:29:46.581612+0000","last_active":"2024-03-28T23:29:51.685367+0000","last_peered":"2024-03-28T23:29:51.685367+0000","last_clean":"2024-03-28T23:29:51.685367+0000","last_became_active":"2024-03-28T23:29:46.581324+0000","last_became_peered":"2024-03-28T23:29:46.581324+0000","last_unstale":"2024-03-28T23:29:51.685367+0000","last_undegraded":"2024-03-28T23:29:51.685367+0000","last_fullsized":"2024-03-28T23:29:51.685367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:10.853710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685527+0000","last_change":"2024-03-28T23:29:46.552565+0000","last_active":"2024-03-28T23:29:51.685527+0000","last_peered":"2024-03-28T23:29:51.685527+0000","last_clean":"2024-03-28T23:29:51.685527+0000","last_became_active":"2024-03-28T23:29:46.552304+0000","last_became_peered":"2024-03-28T23:29:46.552304+0000","last_unstale":"2024-03-28T23:29:51.685527+0000","last_undegraded":"2024-03-28T23:29:51.685527+0000","last_fullsized":"2024-03-28T23:29:51.685527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:15.386423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686970+0000","last_change":"2024-03-28T23:29:46.579218+0000","last_active":"2024-03-28T23:29:51.686970+0000","last_peered":"2024-03-28T23:29:51.686970+0000","last_clean":"2024-03-28T23:29:51.686970+0000","last_became_active":"2024-03-28T23:29:46.578756+0000","last_became_peered":"2024-03-28T23:29:46.578756+0000","last_unstale":"2024-03-28T23:29:51.686970+0000","last_undegraded":"2024-03-28T23:29:51.686970+0000","last_fullsized":"2024-03-28T23:29:51.686970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:24:19.353047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685891+0000","last_change":"2024-03-28T23:29:46.556797+0000","last_active":"2024-03-28T23:29:51.685891+0000","last_peered":"2024-03-28T23:29:51.685891+0000","last_clean":"2024-03-28T23:29:51.685891+0000","last_became_active":"2024-03-28T23:29:46.556520+0000","last_became_peered":"2024-03-28T23:29:46.556520+0000","last_unstale":"2024-03-28T23:29:51.685891+0000","last_undegraded":"2024-03-28T23:29:51.685891+0000","last_fullsized":"2024-03-28T23:29:51.685891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:15.386423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591677+0000","last_change":"2024-03-28T23:29:46.559749+0000","last_active":"2024-03-28T23:29:51.591677+0000","last_peered":"2024-03-28T23:29:51.591677+0000","last_clean":"2024-03-28T23:29:51.591677+0000","last_became_active":"2024-03-28T23:29:46.559386+0000","last_became_peered":"2024-03-28T23:29:46.559386+0000","last_unstale":"2024-03-28T23:29:51.591677+0000","last_undegraded":"2024-03-28T23:29:51.591677+0000","last_fullsized":"2024-03-28T23:29:51.591677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:10.938731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591382+0000","last_change":"2024-03-28T23:29:46.571568+0000","last_active":"2024-03-28T23:29:51.591382+0000","last_peered":"2024-03-28T23:29:51.591382+0000","last_clean":"2024-03-28T23:29:51.591382+0000","last_became_active":"2024-03-28T23:29:46.571268+0000","last_became_peered":"2024-03-28T23:29:46.571268+0000","last_unstale":"2024-03-28T23:29:51.591382+0000","last_undegraded":"2024-03-28T23:29:51.591382+0000","last_fullsized":"2024-03-28T23:29:51.591382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:36:20.485861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591361+0000","last_change":"2024-03-28T23:29:46.571722+0000","last_active":"2024-03-28T23:29:51.591361+0000","last_peered":"2024-03-28T23:29:51.591361+0000","last_clean":"2024-03-28T23:29:51.591361+0000","last_became_active":"2024-03-28T23:29:46.571437+0000","last_became_peered":"2024-03-28T23:29:46.571437+0000","last_unstale":"2024-03-28T23:29:51.591361+0000","last_undegraded":"2024-03-28T23:29:51.591361+0000","last_fullsized":"2024-03-28T23:29:51.591361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:10.853710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686822+0000","last_change":"2024-03-28T23:29:46.565939+0000","last_active":"2024-03-28T23:29:51.686822+0000","last_peered":"2024-03-28T23:29:51.686822+0000","last_clean":"2024-03-28T23:29:51.686822+0000","last_became_active":"2024-03-28T23:29:46.564853+0000","last_became_peered":"2024-03-28T23:29:46.564853+0000","last_unstale":"2024-03-28T23:29:51.686822+0000","last_undegraded":"2024-03-28T23:29:51.686822+0000","last_fullsized":"2024-03-28T23:29:51.686822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:18.050658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685938+0000","last_change":"2024-03-28T23:29:46.567115+0000","last_active":"2024-03-28T23:29:51.685938+0000","last_peered":"2024-03-28T23:29:51.685938+0000","last_clean":"2024-03-28T23:29:51.685938+0000","last_became_active":"2024-03-28T23:29:46.566742+0000","last_became_peered":"2024-03-28T23:29:46.566742+0000","last_unstale":"2024-03-28T23:29:51.685938+0000","last_undegraded":"2024-03-28T23:29:51.685938+0000","last_fullsized":"2024-03-28T23:29:51.685938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:32.936386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686798+0000","last_change":"2024-03-28T23:29:46.579765+0000","last_active":"2024-03-28T23:29:51.686798+0000","last_peered":"2024-03-28T23:29:51.686798+0000","last_clean":"2024-03-28T23:29:51.686798+0000","last_became_active":"2024-03-28T23:29:46.579469+0000","last_became_peered":"2024-03-28T23:29:46.579469+0000","last_unstale":"2024-03-28T23:29:51.686798+0000","last_undegraded":"2024-03-28T23:29:51.686798+0000","last_fullsized":"2024-03-28T23:29:51.686798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:06.731262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685479+0000","last_change":"2024-03-28T23:29:46.580994+0000","last_active":"2024-03-28T23:29:51.685479+0000","last_peered":"2024-03-28T23:29:51.685479+0000","last_clean":"2024-03-28T23:29:51.685479+0000","last_became_active":"2024-03-28T23:29:46.580685+0000","last_became_peered":"2024-03-28T23:29:46.580685+0000","last_unstale":"2024-03-28T23:29:51.685479+0000","last_undegraded":"2024-03-28T23:29:51.685479+0000","last_fullsized":"2024-03-28T23:29:51.685479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:58.674327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685558+0000","last_change":"2024-03-28T23:29:46.571922+0000","last_active":"2024-03-28T23:29:51.685558+0000","last_peered":"2024-03-28T23:29:51.685558+0000","last_clean":"2024-03-28T23:29:51.685558+0000","last_became_active":"2024-03-28T23:29:46.571256+0000","last_became_peered":"2024-03-28T23:29:46.571256+0000","last_unstale":"2024-03-28T23:29:51.685558+0000","last_undegraded":"2024-03-28T23:29:51.685558+0000","last_fullsized":"2024-03-28T23:29:51.685558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:54.470202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685672+0000","last_change":"2024-03-28T23:29:46.561038+0000","last_active":"2024-03-28T23:29:51.685672+0000","last_peered":"2024-03-28T23:29:51.685672+0000","last_clean":"2024-03-28T23:29:51.685672+0000","last_became_active":"2024-03-28T23:29:46.560679+0000","last_became_peered":"2024-03-28T23:29:46.560679+0000","last_unstale":"2024-03-28T23:29:51.685672+0000","last_undegraded":"2024-03-28T23:29:51.685672+0000","last_fullsized":"2024-03-28T23:29:51.685672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:29:20.669429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687099+0000","last_change":"2024-03-28T23:29:46.578506+0000","last_active":"2024-03-28T23:29:51.687099+0000","last_peered":"2024-03-28T23:29:51.687099+0000","last_clean":"2024-03-28T23:29:51.687099+0000","last_became_active":"2024-03-28T23:29:46.577958+0000","last_became_peered":"2024-03-28T23:29:46.577958+0000","last_unstale":"2024-03-28T23:29:51.687099+0000","last_undegraded":"2024-03-28T23:29:51.687099+0000","last_fullsized":"2024-03-28T23:29:51.687099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:38:01.630848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686703+0000","last_change":"2024-03-28T23:29:46.579555+0000","last_active":"2024-03-28T23:29:51.686703+0000","last_peered":"2024-03-28T23:29:51.686703+0000","last_clean":"2024-03-28T23:29:51.686703+0000","last_became_active":"2024-03-28T23:29:46.579273+0000","last_became_peered":"2024-03-28T23:29:46.579273+0000","last_unstale":"2024-03-28T23:29:51.686703+0000","last_undegraded":"2024-03-28T23:29:51.686703+0000","last_fullsized":"2024-03-28T23:29:51.686703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:10.853710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686627+0000","last_change":"2024-03-28T23:29:46.568698+0000","last_active":"2024-03-28T23:29:51.686627+0000","last_peered":"2024-03-28T23:29:51.686627+0000","last_clean":"2024-03-28T23:29:51.686627+0000","last_became_active":"2024-03-28T23:29:46.568318+0000","last_became_peered":"2024-03-28T23:29:46.568318+0000","last_unstale":"2024-03-28T23:29:51.686627+0000","last_undegraded":"2024-03-28T23:29:51.686627+0000","last_fullsized":"2024-03-28T23:29:51.686627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:43:10.380471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31504,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340336,"statfs":{"total":96636764160,"available":96604504064,"internally_reserved":0,"allocated":323584,"data_stored":691206,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":331776,"data_stored":691262,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:30:09.781 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:30:09.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.935+0000 7f929a672640 1 Processor -- start 2024-03-28T23:30:09.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.936+0000 7f929a672640 1 -- start start 2024-03-28T23:30:09.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.936+0000 7f929a672640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92940c5410 0x7f92940c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:09.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.936+0000 7f929a672640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f92941553a0 0x7f9294155820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:09.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.936+0000 7f929a672640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9294158a30 con 0x7f9294152890 2024-03-28T23:30:09.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.936+0000 7f929a672640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f92940c8480 con 0x7f92941553a0 2024-03-28T23:30:09.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.936+0000 7f929a672640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f92940c5d90 con 0x7f92940c5410 2024-03-28T23:30:09.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.936+0000 7f9298c0f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f92941553a0 0x7f9294155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:09.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.936+0000 7f9298c0f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f92941553a0 0x7f9294155820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45648/0 (socket says 172.21.15.161:45648) 2024-03-28T23:30:09.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.936+0000 7f9298c0f640 1 -- 172.21.15.161:0/3686091028 learned_addr learned my addr 172.21.15.161:0/3686091028 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:09.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.937+0000 7f92937fe640 1 --2- 172.21.15.161:0/3686091028 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92940c5410 0x7f92940c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:09.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.937+0000 7f9298c0f640 1 -- 172.21.15.161:0/3686091028 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92940c5410 msgr2=0x7f92940c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:09.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.937+0000 7f9298c0f640 1 --2- 172.21.15.161:0/3686091028 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92940c5410 0x7f92940c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.937+0000 7f9298c0f640 1 -- 172.21.15.161:0/3686091028 >> v1:172.21.15.170:6789/0 conn(0x7f9294152890 legacy=0x7f92940c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:09.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.937+0000 7f9298c0f640 1 -- 172.21.15.161:0/3686091028 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9294158d10 con 0x7f92941553a0 2024-03-28T23:30:09.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.938+0000 7f9298c0f640 1 --2- 172.21.15.161:0/3686091028 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92941553a0 0x7f9294155820 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f928c009e50 tx=0x7f928c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bc1ca4f308c8e33 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:09.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.938+0000 7f9292ffd640 1 -- 172.21.15.161:0/3686091028 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c02ca70 con 0x7f92941553a0 2024-03-28T23:30:09.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.938+0000 7f9292ffd640 1 -- 172.21.15.161:0/3686091028 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f928c02cc10 con 0x7f92941553a0 2024-03-28T23:30:09.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.938+0000 7f9292ffd640 1 -- 172.21.15.161:0/3686091028 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c0343f0 con 0x7f92941553a0 2024-03-28T23:30:09.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.938+0000 7f929a672640 1 -- 172.21.15.161:0/3686091028 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92941553a0 msgr2=0x7f9294155820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:09.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.938+0000 7f929a672640 1 --2- 172.21.15.161:0/3686091028 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92941553a0 0x7f9294155820 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f928c009e50 tx=0x7f928c02bed0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.938+0000 7f929a672640 1 -- 172.21.15.161:0/3686091028 shutdown_connections 2024-03-28T23:30:09.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.938+0000 7f929a672640 1 --2- 172.21.15.161:0/3686091028 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92941553a0 0x7f9294155820 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.938+0000 7f929a672640 1 --2- 172.21.15.161:0/3686091028 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92940c5410 0x7f92940c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.938+0000 7f929a672640 1 -- 172.21.15.161:0/3686091028 >> 172.21.15.161:0/3686091028 conn(0x7f92940ac620 msgr2=0x7f92940aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:09.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.938+0000 7f929a672640 1 -- 172.21.15.161:0/3686091028 shutdown_connections 2024-03-28T23:30:09.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.938+0000 7f929a672640 1 -- 172.21.15.161:0/3686091028 wait complete. 2024-03-28T23:30:09.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.939+0000 7f929a672640 1 Processor -- start 2024-03-28T23:30:09.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.940+0000 7f929a672640 1 -- start start 2024-03-28T23:30:09.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.940+0000 7f929a672640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92940c5410 0x7f9294157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:09.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.940+0000 7f929a672640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9294152890 0x7f9294149040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:09.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.940+0000 7f929a672640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92941553a0 0x7f9294149580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:09.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.940+0000 7f929a672640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f92941592d0 con 0x7f92941553a0 2024-03-28T23:30:09.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.940+0000 7f929a672640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9294159640 con 0x7f92940c5410 2024-03-28T23:30:09.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.940+0000 7f929a672640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f92940c90a0 con 0x7f9294152890 2024-03-28T23:30:09.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.940+0000 7f9298c0f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92941553a0 0x7f9294149580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:09.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.940+0000 7f9298c0f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92941553a0 0x7f9294149580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45652/0 (socket says 172.21.15.161:45652) 2024-03-28T23:30:09.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.940+0000 7f9298c0f640 1 -- 172.21.15.161:0/1257019423 learned_addr learned my addr 172.21.15.161:0/1257019423 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:09.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.941+0000 7f9293fff640 1 --2- 172.21.15.161:0/1257019423 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92940c5410 0x7f9294157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:09.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.941+0000 7f92937fe640 1 --2- 172.21.15.161:0/1257019423 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9294152890 0x7f9294149040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:09.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.941+0000 7f9298c0f640 1 -- 172.21.15.161:0/1257019423 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9294152890 msgr2=0x7f9294149040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:09.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.941+0000 7f9298c0f640 1 --2- 172.21.15.161:0/1257019423 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9294152890 0x7f9294149040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.941+0000 7f9298c0f640 1 -- 172.21.15.161:0/1257019423 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92940c5410 msgr2=0x7f9294157a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:09.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.941+0000 7f9298c0f640 1 --2- 172.21.15.161:0/1257019423 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92940c5410 0x7f9294157a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:09.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.941+0000 7f9298c0f640 1 -- 172.21.15.161:0/1257019423 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92940bf0b0 con 0x7f92941553a0 2024-03-28T23:30:09.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.941+0000 7f9293fff640 1 --2- 172.21.15.161:0/1257019423 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92940c5410 0x7f9294157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:09.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.941+0000 7f9298c0f640 1 --2- 172.21.15.161:0/1257019423 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92941553a0 0x7f9294149580 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f928c036040 tx=0x7f928c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:09.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.941+0000 7f9290ff9640 1 -- 172.21.15.161:0/1257019423 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c043020 con 0x7f92941553a0 2024-03-28T23:30:09.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.941+0000 7f9290ff9640 1 -- 172.21.15.161:0/1257019423 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f928c03e450 con 0x7f92941553a0 2024-03-28T23:30:09.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.941+0000 7f929a672640 1 -- 172.21.15.161:0/1257019423 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f929414fb40 con 0x7f92941553a0 2024-03-28T23:30:09.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.941+0000 7f9290ff9640 1 -- 172.21.15.161:0/1257019423 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c009510 con 0x7f92941553a0 2024-03-28T23:30:09.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.941+0000 7f929a672640 1 -- 172.21.15.161:0/1257019423 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92940bf320 con 0x7f92941553a0 2024-03-28T23:30:09.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.942+0000 7f929a672640 1 -- 172.21.15.161:0/1257019423 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92940c6de0 con 0x7f92941553a0 2024-03-28T23:30:09.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.943+0000 7f9290ff9640 1 -- 172.21.15.161:0/1257019423 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f928c046020 con 0x7f92941553a0 2024-03-28T23:30:09.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.944+0000 7f9290ff9640 1 --2- 172.21.15.161:0/1257019423 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f928006b4d0 0x7f928006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:09.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.944+0000 7f9290ff9640 1 -- 172.21.15.161:0/1257019423 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f928c0acc90 con 0x7f92941553a0 2024-03-28T23:30:09.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.944+0000 7f9293fff640 1 --2- 172.21.15.161:0/1257019423 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f928006b4d0 0x7f928006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:09.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.944+0000 7f9293fff640 1 --2- 172.21.15.161:0/1257019423 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f928006b4d0 0x7f928006d990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9284001680 tx=0x7f9284009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:09.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:09.948+0000 7f9290ff9640 1 -- 172.21.15.161:0/1257019423 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f92940c6de0 con 0x7f92941553a0 2024-03-28T23:30:10.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.118+0000 7f929a672640 1 -- 172.21.15.161:0/1257019423 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f929404f4b0 con 0x7f928006b4d0 2024-03-28T23:30:10.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.125+0000 7f9290ff9640 1 -- 172.21.15.161:0/1257019423 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332611 (secure 0 0 0) 0x7f929404f4b0 con 0x7f928006b4d0 2024-03-28T23:30:10.131 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:10.133 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:30:10.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.129+0000 7f929a672640 1 -- 172.21.15.161:0/1257019423 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f928006b4d0 msgr2=0x7f928006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:10.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.129+0000 7f929a672640 1 --2- 172.21.15.161:0/1257019423 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f928006b4d0 0x7f928006d990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9284001680 tx=0x7f9284009040 comp rx=0 tx=0).stop 2024-03-28T23:30:10.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.129+0000 7f929a672640 1 -- 172.21.15.161:0/1257019423 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92941553a0 msgr2=0x7f9294149580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:10.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.129+0000 7f929a672640 1 --2- 172.21.15.161:0/1257019423 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92941553a0 0x7f9294149580 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f928c036040 tx=0x7f928c004a40 comp rx=0 tx=0).stop 2024-03-28T23:30:10.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.129+0000 7f929a672640 1 -- 172.21.15.161:0/1257019423 shutdown_connections 2024-03-28T23:30:10.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.129+0000 7f929a672640 1 --2- 172.21.15.161:0/1257019423 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f928006b4d0 0x7f928006d990 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:10.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.129+0000 7f929a672640 1 --2- 172.21.15.161:0/1257019423 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92941553a0 0x7f9294149580 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:10.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.129+0000 7f929a672640 1 --2- 172.21.15.161:0/1257019423 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9294152890 0x7f9294149040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:10.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.129+0000 7f929a672640 1 --2- 172.21.15.161:0/1257019423 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92940c5410 0x7f9294157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:10.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.129+0000 7f929a672640 1 -- 172.21.15.161:0/1257019423 >> 172.21.15.161:0/1257019423 conn(0x7f92940ac620 msgr2=0x7f92940ca230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:10.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.129+0000 7f929a672640 1 -- 172.21.15.161:0/1257019423 shutdown_connections 2024-03-28T23:30:10.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.129+0000 7f929a672640 1 -- 172.21.15.161:0/1257019423 wait complete. 2024-03-28T23:30:10.142 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-03-28T23:30:09.316678+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222516,"kb_used_data":2176,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754752204,"statfs":{"total":773094113280,"available":772866256896,"internally_reserved":0,"allocated":2228224,"data_stored":1992064,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002338"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687840+0000","last_change":"2024-03-28T23:29:45.558575+0000","last_active":"2024-03-28T23:29:51.687840+0000","last_peered":"2024-03-28T23:29:51.687840+0000","last_clean":"2024-03-28T23:29:51.687840+0000","last_became_active":"2024-03-28T23:29:45.557910+0000","last_became_peered":"2024-03-28T23:29:45.557910+0000","last_unstale":"2024-03-28T23:29:51.687840+0000","last_undegraded":"2024-03-28T23:29:51.687840+0000","last_fullsized":"2024-03-28T23:29:51.687840+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:59.719252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686777+0000","last_change":"2024-03-28T23:29:46.557562+0000","last_active":"2024-03-28T23:29:51.686777+0000","last_peered":"2024-03-28T23:29:51.686777+0000","last_clean":"2024-03-28T23:29:51.686777+0000","last_became_active":"2024-03-28T23:29:46.557265+0000","last_became_peered":"2024-03-28T23:29:46.557265+0000","last_unstale":"2024-03-28T23:29:51.686777+0000","last_undegraded":"2024-03-28T23:29:51.686777+0000","last_fullsized":"2024-03-28T23:29:51.686777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:06.731262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686496+0000","last_change":"2024-03-28T23:29:50.707752+0000","last_active":"2024-03-28T23:29:51.686496+0000","last_peered":"2024-03-28T23:29:51.686496+0000","last_clean":"2024-03-28T23:29:51.686496+0000","last_became_active":"2024-03-28T23:29:50.707364+0000","last_became_peered":"2024-03-28T23:29:50.707364+0000","last_unstale":"2024-03-28T23:29:51.686496+0000","last_undegraded":"2024-03-28T23:29:51.686496+0000","last_fullsized":"2024-03-28T23:29:51.686496+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:58:38.680360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686927+0000","last_change":"2024-03-28T23:29:45.558356+0000","last_active":"2024-03-28T23:29:51.686927+0000","last_peered":"2024-03-28T23:29:51.686927+0000","last_clean":"2024-03-28T23:29:51.686927+0000","last_became_active":"2024-03-28T23:29:45.558070+0000","last_became_peered":"2024-03-28T23:29:45.558070+0000","last_unstale":"2024-03-28T23:29:51.686927+0000","last_undegraded":"2024-03-28T23:29:51.686927+0000","last_fullsized":"2024-03-28T23:29:51.686927+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:59.719252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686591+0000","last_change":"2024-03-28T23:29:46.579153+0000","last_active":"2024-03-28T23:29:51.686591+0000","last_peered":"2024-03-28T23:29:51.686591+0000","last_clean":"2024-03-28T23:29:51.686591+0000","last_became_active":"2024-03-28T23:29:46.578341+0000","last_became_peered":"2024-03-28T23:29:46.578341+0000","last_unstale":"2024-03-28T23:29:51.686591+0000","last_undegraded":"2024-03-28T23:29:51.686591+0000","last_fullsized":"2024-03-28T23:29:51.686591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:38.074388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591979+0000","last_change":"2024-03-28T23:29:50.597011+0000","last_active":"2024-03-28T23:29:51.591979+0000","last_peered":"2024-03-28T23:29:51.591979+0000","last_clean":"2024-03-28T23:29:51.591979+0000","last_became_active":"2024-03-28T23:29:50.596239+0000","last_became_peered":"2024-03-28T23:29:50.596239+0000","last_unstale":"2024-03-28T23:29:51.591979+0000","last_undegraded":"2024-03-28T23:29:51.591979+0000","last_fullsized":"2024-03-28T23:29:51.591979+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:29:24.703997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686674+0000","last_change":"2024-03-28T23:29:45.562460+0000","last_active":"2024-03-28T23:29:51.686674+0000","last_peered":"2024-03-28T23:29:51.686674+0000","last_clean":"2024-03-28T23:29:51.686674+0000","last_became_active":"2024-03-28T23:29:45.562101+0000","last_became_peered":"2024-03-28T23:29:45.562101+0000","last_unstale":"2024-03-28T23:29:51.686674+0000","last_undegraded":"2024-03-28T23:29:51.686674+0000","last_fullsized":"2024-03-28T23:29:51.686674+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:42:08.707703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685709+0000","last_change":"2024-03-28T23:29:46.567420+0000","last_active":"2024-03-28T23:29:51.685709+0000","last_peered":"2024-03-28T23:29:51.685709+0000","last_clean":"2024-03-28T23:29:51.685709+0000","last_became_active":"2024-03-28T23:29:46.567157+0000","last_became_peered":"2024-03-28T23:29:46.567157+0000","last_unstale":"2024-03-28T23:29:51.685709+0000","last_undegraded":"2024-03-28T23:29:51.685709+0000","last_fullsized":"2024-03-28T23:29:51.685709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:54.470202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.592139+0000","last_change":"2024-03-28T23:29:50.599020+0000","last_active":"2024-03-28T23:29:51.592139+0000","last_peered":"2024-03-28T23:29:51.592139+0000","last_clean":"2024-03-28T23:29:51.592139+0000","last_became_active":"2024-03-28T23:29:50.598588+0000","last_became_peered":"2024-03-28T23:29:50.598588+0000","last_unstale":"2024-03-28T23:29:51.592139+0000","last_undegraded":"2024-03-28T23:29:51.592139+0000","last_fullsized":"2024-03-28T23:29:51.592139+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:06.894069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686561+0000","last_change":"2024-03-28T23:29:45.564688+0000","last_active":"2024-03-28T23:29:51.686561+0000","last_peered":"2024-03-28T23:29:51.686561+0000","last_clean":"2024-03-28T23:29:51.686561+0000","last_became_active":"2024-03-28T23:29:45.564381+0000","last_became_peered":"2024-03-28T23:29:45.564381+0000","last_unstale":"2024-03-28T23:29:51.686561+0000","last_undegraded":"2024-03-28T23:29:51.686561+0000","last_fullsized":"2024-03-28T23:29:51.686561+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:36.113647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686025+0000","last_change":"2024-03-28T23:29:46.566644+0000","last_active":"2024-03-28T23:29:51.686025+0000","last_peered":"2024-03-28T23:29:51.686025+0000","last_clean":"2024-03-28T23:29:51.686025+0000","last_became_active":"2024-03-28T23:29:46.566340+0000","last_became_peered":"2024-03-28T23:29:46.566340+0000","last_unstale":"2024-03-28T23:29:51.686025+0000","last_undegraded":"2024-03-28T23:29:51.686025+0000","last_fullsized":"2024-03-28T23:29:51.686025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:02.859502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591969+0000","last_change":"2024-03-28T23:29:50.597019+0000","last_active":"2024-03-28T23:29:51.591969+0000","last_peered":"2024-03-28T23:29:51.591969+0000","last_clean":"2024-03-28T23:29:51.591969+0000","last_became_active":"2024-03-28T23:29:50.596284+0000","last_became_peered":"2024-03-28T23:29:50.596284+0000","last_unstale":"2024-03-28T23:29:51.591969+0000","last_undegraded":"2024-03-28T23:29:51.591969+0000","last_fullsized":"2024-03-28T23:29:51.591969+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:08:37.836411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.590591+0000","last_change":"2024-03-28T23:29:46.565325+0000","last_active":"2024-03-28T23:29:51.590591+0000","last_peered":"2024-03-28T23:29:51.590591+0000","last_clean":"2024-03-28T23:29:51.590591+0000","last_became_active":"2024-03-28T23:29:46.564195+0000","last_became_peered":"2024-03-28T23:29:46.564195+0000","last_unstale":"2024-03-28T23:29:51.590591+0000","last_undegraded":"2024-03-28T23:29:51.590591+0000","last_fullsized":"2024-03-28T23:29:51.590591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:32.936386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685919+0000","last_change":"2024-03-28T23:29:45.560329+0000","last_active":"2024-03-28T23:29:51.685919+0000","last_peered":"2024-03-28T23:29:51.685919+0000","last_clean":"2024-03-28T23:29:51.685919+0000","last_became_active":"2024-03-28T23:29:45.560073+0000","last_became_peered":"2024-03-28T23:29:45.560073+0000","last_unstale":"2024-03-28T23:29:51.685919+0000","last_undegraded":"2024-03-28T23:29:51.685919+0000","last_fullsized":"2024-03-28T23:29:51.685919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:09.929984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591983+0000","last_change":"2024-03-28T23:29:50.592083+0000","last_active":"2024-03-28T23:29:51.591983+0000","last_peered":"2024-03-28T23:29:51.591983+0000","last_clean":"2024-03-28T23:29:51.591983+0000","last_became_active":"2024-03-28T23:29:50.591626+0000","last_became_peered":"2024-03-28T23:29:50.591626+0000","last_unstale":"2024-03-28T23:29:51.591983+0000","last_undegraded":"2024-03-28T23:29:51.591983+0000","last_fullsized":"2024-03-28T23:29:51.591983+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:17:54.013778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685967+0000","last_change":"2024-03-28T23:29:46.581125+0000","last_active":"2024-03-28T23:29:51.685967+0000","last_peered":"2024-03-28T23:29:51.685967+0000","last_clean":"2024-03-28T23:29:51.685967+0000","last_became_active":"2024-03-28T23:29:46.580827+0000","last_became_peered":"2024-03-28T23:29:46.580827+0000","last_unstale":"2024-03-28T23:29:51.685967+0000","last_undegraded":"2024-03-28T23:29:51.685967+0000","last_fullsized":"2024-03-28T23:29:51.685967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:10.853710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686404+0000","last_change":"2024-03-28T23:29:45.562525+0000","last_active":"2024-03-28T23:29:51.686404+0000","last_peered":"2024-03-28T23:29:51.686404+0000","last_clean":"2024-03-28T23:29:51.686404+0000","last_became_active":"2024-03-28T23:29:45.561757+0000","last_became_peered":"2024-03-28T23:29:45.561757+0000","last_unstale":"2024-03-28T23:29:51.686404+0000","last_undegraded":"2024-03-28T23:29:51.686404+0000","last_fullsized":"2024-03-28T23:29:51.686404+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:24:58.784056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591567+0000","last_change":"2024-03-28T23:29:50.599617+0000","last_active":"2024-03-28T23:29:51.591567+0000","last_peered":"2024-03-28T23:29:51.591567+0000","last_clean":"2024-03-28T23:29:51.591567+0000","last_became_active":"2024-03-28T23:29:50.599031+0000","last_became_peered":"2024-03-28T23:29:50.599031+0000","last_unstale":"2024-03-28T23:29:51.591567+0000","last_undegraded":"2024-03-28T23:29:51.591567+0000","last_fullsized":"2024-03-28T23:29:51.591567+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:15:03.216756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686223+0000","last_change":"2024-03-28T23:29:46.555057+0000","last_active":"2024-03-28T23:29:51.686223+0000","last_peered":"2024-03-28T23:29:51.686223+0000","last_clean":"2024-03-28T23:29:51.686223+0000","last_became_active":"2024-03-28T23:29:46.554796+0000","last_became_peered":"2024-03-28T23:29:46.554796+0000","last_unstale":"2024-03-28T23:29:51.686223+0000","last_undegraded":"2024-03-28T23:29:51.686223+0000","last_fullsized":"2024-03-28T23:29:51.686223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:36:20.485861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685609+0000","last_change":"2024-03-28T23:29:45.561273+0000","last_active":"2024-03-28T23:29:51.685609+0000","last_peered":"2024-03-28T23:29:51.685609+0000","last_clean":"2024-03-28T23:29:51.685609+0000","last_became_active":"2024-03-28T23:29:45.561006+0000","last_became_peered":"2024-03-28T23:29:45.561006+0000","last_unstale":"2024-03-28T23:29:51.685609+0000","last_undegraded":"2024-03-28T23:29:51.685609+0000","last_fullsized":"2024-03-28T23:29:51.685609+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:41.621188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687569+0000","last_change":"2024-03-28T23:29:50.706824+0000","last_active":"2024-03-28T23:29:51.687569+0000","last_peered":"2024-03-28T23:29:51.687569+0000","last_clean":"2024-03-28T23:29:51.687569+0000","last_became_active":"2024-03-28T23:29:50.706411+0000","last_became_peered":"2024-03-28T23:29:50.706411+0000","last_unstale":"2024-03-28T23:29:51.687569+0000","last_undegraded":"2024-03-28T23:29:51.687569+0000","last_fullsized":"2024-03-28T23:29:51.687569+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:43:14.415039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.592115+0000","last_change":"2024-03-28T23:29:46.558900+0000","last_active":"2024-03-28T23:29:51.592115+0000","last_peered":"2024-03-28T23:29:51.592115+0000","last_clean":"2024-03-28T23:29:51.592115+0000","last_became_active":"2024-03-28T23:29:46.558628+0000","last_became_peered":"2024-03-28T23:29:46.558628+0000","last_unstale":"2024-03-28T23:29:51.592115+0000","last_undegraded":"2024-03-28T23:29:51.592115+0000","last_fullsized":"2024-03-28T23:29:51.592115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:37.122394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685397+0000","last_change":"2024-03-28T23:29:45.563365+0000","last_active":"2024-03-28T23:29:51.685397+0000","last_peered":"2024-03-28T23:29:51.685397+0000","last_clean":"2024-03-28T23:29:51.685397+0000","last_became_active":"2024-03-28T23:29:45.563093+0000","last_became_peered":"2024-03-28T23:29:45.563093+0000","last_unstale":"2024-03-28T23:29:51.685397+0000","last_undegraded":"2024-03-28T23:29:51.685397+0000","last_fullsized":"2024-03-28T23:29:51.685397+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:01.983721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591200+0000","last_change":"2024-03-28T23:29:50.596634+0000","last_active":"2024-03-28T23:29:51.591200+0000","last_peered":"2024-03-28T23:29:51.591200+0000","last_clean":"2024-03-28T23:29:51.591200+0000","last_became_active":"2024-03-28T23:29:50.596057+0000","last_became_peered":"2024-03-28T23:29:50.596057+0000","last_unstale":"2024-03-28T23:29:51.591200+0000","last_undegraded":"2024-03-28T23:29:51.591200+0000","last_fullsized":"2024-03-28T23:29:51.591200+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:02.708894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687515+0000","last_change":"2024-03-28T23:29:46.578357+0000","last_active":"2024-03-28T23:29:51.687515+0000","last_peered":"2024-03-28T23:29:51.687515+0000","last_clean":"2024-03-28T23:29:51.687515+0000","last_became_active":"2024-03-28T23:29:46.578075+0000","last_became_peered":"2024-03-28T23:29:46.578075+0000","last_unstale":"2024-03-28T23:29:51.687515+0000","last_undegraded":"2024-03-28T23:29:51.687515+0000","last_fullsized":"2024-03-28T23:29:51.687515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:40:22.556997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591419+0000","last_change":"2024-03-28T23:29:45.560606+0000","last_active":"2024-03-28T23:29:51.591419+0000","last_peered":"2024-03-28T23:29:51.591419+0000","last_clean":"2024-03-28T23:29:51.591419+0000","last_became_active":"2024-03-28T23:29:45.560355+0000","last_became_peered":"2024-03-28T23:29:45.560355+0000","last_unstale":"2024-03-28T23:29:51.591419+0000","last_undegraded":"2024-03-28T23:29:51.591419+0000","last_fullsized":"2024-03-28T23:29:51.591419+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:01.983721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591966+0000","last_change":"2024-03-28T23:29:50.708102+0000","last_active":"2024-03-28T23:29:51.591966+0000","last_peered":"2024-03-28T23:29:51.591966+0000","last_clean":"2024-03-28T23:29:51.591966+0000","last_became_active":"2024-03-28T23:29:50.707657+0000","last_became_peered":"2024-03-28T23:29:50.707657+0000","last_unstale":"2024-03-28T23:29:51.591966+0000","last_undegraded":"2024-03-28T23:29:51.591966+0000","last_fullsized":"2024-03-28T23:29:51.591966+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:58.504770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687431+0000","last_change":"2024-03-28T23:29:46.561641+0000","last_active":"2024-03-28T23:29:51.687431+0000","last_peered":"2024-03-28T23:29:51.687431+0000","last_clean":"2024-03-28T23:29:51.687431+0000","last_became_active":"2024-03-28T23:29:46.561380+0000","last_became_peered":"2024-03-28T23:29:46.561380+0000","last_unstale":"2024-03-28T23:29:51.687431+0000","last_undegraded":"2024-03-28T23:29:51.687431+0000","last_fullsized":"2024-03-28T23:29:51.687431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:54.470202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686791+0000","last_change":"2024-03-28T23:29:45.562177+0000","last_active":"2024-03-28T23:29:51.686791+0000","last_peered":"2024-03-28T23:29:51.686791+0000","last_clean":"2024-03-28T23:29:51.686791+0000","last_became_active":"2024-03-28T23:29:45.561852+0000","last_became_peered":"2024-03-28T23:29:45.561852+0000","last_unstale":"2024-03-28T23:29:51.686791+0000","last_undegraded":"2024-03-28T23:29:51.686791+0000","last_fullsized":"2024-03-28T23:29:51.686791+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:36.113647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685836+0000","last_change":"2024-03-28T23:29:50.706769+0000","last_active":"2024-03-28T23:29:51.685836+0000","last_peered":"2024-03-28T23:29:51.685836+0000","last_clean":"2024-03-28T23:29:51.685836+0000","last_became_active":"2024-03-28T23:29:50.706415+0000","last_became_peered":"2024-03-28T23:29:50.706415+0000","last_unstale":"2024-03-28T23:29:51.685836+0000","last_undegraded":"2024-03-28T23:29:51.685836+0000","last_fullsized":"2024-03-28T23:29:51.685836+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:47.094789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687279+0000","last_change":"2024-03-28T23:29:46.560067+0000","last_active":"2024-03-28T23:29:51.687279+0000","last_peered":"2024-03-28T23:29:51.687279+0000","last_clean":"2024-03-28T23:29:51.687279+0000","last_became_active":"2024-03-28T23:29:46.559790+0000","last_became_peered":"2024-03-28T23:29:46.559790+0000","last_unstale":"2024-03-28T23:29:51.687279+0000","last_undegraded":"2024-03-28T23:29:51.687279+0000","last_fullsized":"2024-03-28T23:29:51.687279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:32.936386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686796+0000","last_change":"2024-03-28T23:29:45.565279+0000","last_active":"2024-03-28T23:29:51.686796+0000","last_peered":"2024-03-28T23:29:51.686796+0000","last_clean":"2024-03-28T23:29:51.686796+0000","last_became_active":"2024-03-28T23:29:45.564739+0000","last_became_peered":"2024-03-28T23:29:45.564739+0000","last_unstale":"2024-03-28T23:29:51.686796+0000","last_undegraded":"2024-03-28T23:29:51.686796+0000","last_fullsized":"2024-03-28T23:29:51.686796+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:17.783972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686745+0000","last_change":"2024-03-28T23:29:50.706805+0000","last_active":"2024-03-28T23:29:51.686745+0000","last_peered":"2024-03-28T23:29:51.686745+0000","last_clean":"2024-03-28T23:29:51.686745+0000","last_became_active":"2024-03-28T23:29:50.706429+0000","last_became_peered":"2024-03-28T23:29:50.706429+0000","last_unstale":"2024-03-28T23:29:51.686745+0000","last_undegraded":"2024-03-28T23:29:51.686745+0000","last_fullsized":"2024-03-28T23:29:51.686745+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:06:40.917529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686157+0000","last_change":"2024-03-28T23:29:46.560912+0000","last_active":"2024-03-28T23:29:51.686157+0000","last_peered":"2024-03-28T23:29:51.686157+0000","last_clean":"2024-03-28T23:29:51.686157+0000","last_became_active":"2024-03-28T23:29:46.560613+0000","last_became_peered":"2024-03-28T23:29:46.560613+0000","last_unstale":"2024-03-28T23:29:51.686157+0000","last_undegraded":"2024-03-28T23:29:51.686157+0000","last_fullsized":"2024-03-28T23:29:51.686157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:17:49.979210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687372+0000","last_change":"2024-03-28T23:29:45.561918+0000","last_active":"2024-03-28T23:29:51.687372+0000","last_peered":"2024-03-28T23:29:51.687372+0000","last_clean":"2024-03-28T23:29:51.687372+0000","last_became_active":"2024-03-28T23:29:45.561577+0000","last_became_peered":"2024-03-28T23:29:45.561577+0000","last_unstale":"2024-03-28T23:29:51.687372+0000","last_undegraded":"2024-03-28T23:29:51.687372+0000","last_fullsized":"2024-03-28T23:29:51.687372+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:09.929984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686073+0000","last_change":"2024-03-28T23:29:50.594835+0000","last_active":"2024-03-28T23:29:51.686073+0000","last_peered":"2024-03-28T23:29:51.686073+0000","last_clean":"2024-03-28T23:29:51.686073+0000","last_became_active":"2024-03-28T23:29:50.594167+0000","last_became_peered":"2024-03-28T23:29:50.594167+0000","last_unstale":"2024-03-28T23:29:51.686073+0000","last_undegraded":"2024-03-28T23:29:51.686073+0000","last_fullsized":"2024-03-28T23:29:51.686073+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:06:40.917529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.592269+0000","last_change":"2024-03-28T23:29:46.579930+0000","last_active":"2024-03-28T23:29:51.592269+0000","last_peered":"2024-03-28T23:29:51.592269+0000","last_clean":"2024-03-28T23:29:51.592269+0000","last_became_active":"2024-03-28T23:29:46.579622+0000","last_became_peered":"2024-03-28T23:29:46.579622+0000","last_unstale":"2024-03-28T23:29:51.592269+0000","last_undegraded":"2024-03-28T23:29:51.592269+0000","last_fullsized":"2024-03-28T23:29:51.592269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:18.792719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686219+0000","last_change":"2024-03-28T23:29:45.557802+0000","last_active":"2024-03-28T23:29:51.686219+0000","last_peered":"2024-03-28T23:29:51.686219+0000","last_clean":"2024-03-28T23:29:51.686219+0000","last_became_active":"2024-03-28T23:29:45.557552+0000","last_became_peered":"2024-03-28T23:29:45.557552+0000","last_unstale":"2024-03-28T23:29:51.686219+0000","last_undegraded":"2024-03-28T23:29:51.686219+0000","last_fullsized":"2024-03-28T23:29:51.686219+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:01.983721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687230+0000","last_change":"2024-03-28T23:29:50.601086+0000","last_active":"2024-03-28T23:29:51.687230+0000","last_peered":"2024-03-28T23:29:51.687230+0000","last_clean":"2024-03-28T23:29:51.687230+0000","last_became_active":"2024-03-28T23:29:50.600720+0000","last_became_peered":"2024-03-28T23:29:50.600720+0000","last_unstale":"2024-03-28T23:29:51.687230+0000","last_undegraded":"2024-03-28T23:29:51.687230+0000","last_fullsized":"2024-03-28T23:29:51.687230+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:30:48.379529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685445+0000","last_change":"2024-03-28T23:29:46.560331+0000","last_active":"2024-03-28T23:29:51.685445+0000","last_peered":"2024-03-28T23:29:51.685445+0000","last_clean":"2024-03-28T23:29:51.685445+0000","last_became_active":"2024-03-28T23:29:46.559135+0000","last_became_peered":"2024-03-28T23:29:46.559135+0000","last_unstale":"2024-03-28T23:29:51.685445+0000","last_undegraded":"2024-03-28T23:29:51.685445+0000","last_fullsized":"2024-03-28T23:29:51.685445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:58.674327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589305+0000","last_change":"2024-03-28T23:29:45.559376+0000","last_active":"2024-03-28T23:29:51.589305+0000","last_peered":"2024-03-28T23:29:51.589305+0000","last_clean":"2024-03-28T23:29:51.589305+0000","last_became_active":"2024-03-28T23:29:45.559076+0000","last_became_peered":"2024-03-28T23:29:45.559076+0000","last_unstale":"2024-03-28T23:29:51.589305+0000","last_undegraded":"2024-03-28T23:29:51.589305+0000","last_fullsized":"2024-03-28T23:29:51.589305+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:21.707459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687612+0000","last_change":"2024-03-28T23:29:50.600059+0000","last_active":"2024-03-28T23:29:51.687612+0000","last_peered":"2024-03-28T23:29:51.687612+0000","last_clean":"2024-03-28T23:29:51.687612+0000","last_became_active":"2024-03-28T23:29:50.599692+0000","last_became_peered":"2024-03-28T23:29:50.599692+0000","last_unstale":"2024-03-28T23:29:51.687612+0000","last_undegraded":"2024-03-28T23:29:51.687612+0000","last_fullsized":"2024-03-28T23:29:51.687612+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:22.085225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686352+0000","last_change":"2024-03-28T23:29:46.578521+0000","last_active":"2024-03-28T23:29:51.686352+0000","last_peered":"2024-03-28T23:29:51.686352+0000","last_clean":"2024-03-28T23:29:51.686352+0000","last_became_active":"2024-03-28T23:29:46.578076+0000","last_became_peered":"2024-03-28T23:29:46.578076+0000","last_unstale":"2024-03-28T23:29:51.686352+0000","last_undegraded":"2024-03-28T23:29:51.686352+0000","last_fullsized":"2024-03-28T23:29:51.686352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:20.039954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685395+0000","last_change":"2024-03-28T23:29:45.563206+0000","last_active":"2024-03-28T23:29:51.685395+0000","last_peered":"2024-03-28T23:29:51.685395+0000","last_clean":"2024-03-28T23:29:51.685395+0000","last_became_active":"2024-03-28T23:29:45.562929+0000","last_became_peered":"2024-03-28T23:29:45.562929+0000","last_unstale":"2024-03-28T23:29:51.685395+0000","last_undegraded":"2024-03-28T23:29:51.685395+0000","last_fullsized":"2024-03-28T23:29:51.685395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:41.621188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687463+0000","last_change":"2024-03-28T23:29:50.709169+0000","last_active":"2024-03-28T23:29:51.687463+0000","last_peered":"2024-03-28T23:29:51.687463+0000","last_clean":"2024-03-28T23:29:51.687463+0000","last_became_active":"2024-03-28T23:29:50.707306+0000","last_became_peered":"2024-03-28T23:29:50.707306+0000","last_unstale":"2024-03-28T23:29:51.687463+0000","last_undegraded":"2024-03-28T23:29:51.687463+0000","last_fullsized":"2024-03-28T23:29:51.687463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:22.085225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685675+0000","last_change":"2024-03-28T23:29:46.564922+0000","last_active":"2024-03-28T23:29:51.685675+0000","last_peered":"2024-03-28T23:29:51.685675+0000","last_clean":"2024-03-28T23:29:51.685675+0000","last_became_active":"2024-03-28T23:29:46.564644+0000","last_became_peered":"2024-03-28T23:29:46.564644+0000","last_unstale":"2024-03-28T23:29:51.685675+0000","last_undegraded":"2024-03-28T23:29:51.685675+0000","last_fullsized":"2024-03-28T23:29:51.685675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:58.674327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686316+0000","last_change":"2024-03-28T23:29:45.556656+0000","last_active":"2024-03-28T23:29:51.686316+0000","last_peered":"2024-03-28T23:29:51.686316+0000","last_clean":"2024-03-28T23:29:51.686316+0000","last_became_active":"2024-03-28T23:29:45.556329+0000","last_became_peered":"2024-03-28T23:29:45.556329+0000","last_unstale":"2024-03-28T23:29:51.686316+0000","last_undegraded":"2024-03-28T23:29:51.686316+0000","last_fullsized":"2024-03-28T23:29:51.686316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:21.707459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686403+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:29:51.686403+0000","last_peered":"2024-03-28T23:29:51.686403+0000","last_clean":"2024-03-28T23:29:51.686403+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:29:51.686403+0000","last_undegraded":"2024-03-28T23:29:51.686403+0000","last_fullsized":"2024-03-28T23:29:51.686403+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591319+0000","last_change":"2024-03-28T23:29:50.711182+0000","last_active":"2024-03-28T23:29:51.591319+0000","last_peered":"2024-03-28T23:29:51.591319+0000","last_clean":"2024-03-28T23:29:51.591319+0000","last_became_active":"2024-03-28T23:29:50.710737+0000","last_became_peered":"2024-03-28T23:29:50.710737+0000","last_unstale":"2024-03-28T23:29:51.591319+0000","last_undegraded":"2024-03-28T23:29:51.591319+0000","last_fullsized":"2024-03-28T23:29:51.591319+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:17:54.013778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686573+0000","last_change":"2024-03-28T23:29:46.572966+0000","last_active":"2024-03-28T23:29:51.686573+0000","last_peered":"2024-03-28T23:29:51.686573+0000","last_clean":"2024-03-28T23:29:51.686573+0000","last_became_active":"2024-03-28T23:29:46.572405+0000","last_became_peered":"2024-03-28T23:29:46.572405+0000","last_unstale":"2024-03-28T23:29:51.686573+0000","last_undegraded":"2024-03-28T23:29:51.686573+0000","last_fullsized":"2024-03-28T23:29:51.686573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:06:36.882961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589560+0000","last_change":"2024-03-28T23:29:45.560681+0000","last_active":"2024-03-28T23:29:51.589560+0000","last_peered":"2024-03-28T23:29:51.589560+0000","last_clean":"2024-03-28T23:29:51.589560+0000","last_became_active":"2024-03-28T23:29:45.560420+0000","last_became_peered":"2024-03-28T23:29:45.560420+0000","last_unstale":"2024-03-28T23:29:51.589560+0000","last_undegraded":"2024-03-28T23:29:51.589560+0000","last_fullsized":"2024-03-28T23:29:51.589560+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:17:17.207826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.590530+0000","last_change":"2024-03-28T23:29:50.711247+0000","last_active":"2024-03-28T23:29:51.590530+0000","last_peered":"2024-03-28T23:29:51.590530+0000","last_clean":"2024-03-28T23:29:51.590530+0000","last_became_active":"2024-03-28T23:29:50.710836+0000","last_became_peered":"2024-03-28T23:29:50.710836+0000","last_unstale":"2024-03-28T23:29:51.590530+0000","last_undegraded":"2024-03-28T23:29:51.590530+0000","last_fullsized":"2024-03-28T23:29:51.590530+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:58.504770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685843+0000","last_change":"2024-03-28T23:29:46.561283+0000","last_active":"2024-03-28T23:29:51.685843+0000","last_peered":"2024-03-28T23:29:51.685843+0000","last_clean":"2024-03-28T23:29:51.685843+0000","last_became_active":"2024-03-28T23:29:46.560584+0000","last_became_peered":"2024-03-28T23:29:46.560584+0000","last_unstale":"2024-03-28T23:29:51.685843+0000","last_undegraded":"2024-03-28T23:29:51.685843+0000","last_fullsized":"2024-03-28T23:29:51.685843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:06:36.882961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686047+0000","last_change":"2024-03-28T23:29:45.560013+0000","last_active":"2024-03-28T23:29:51.686047+0000","last_peered":"2024-03-28T23:29:51.686047+0000","last_clean":"2024-03-28T23:29:51.686047+0000","last_became_active":"2024-03-28T23:29:45.559643+0000","last_became_peered":"2024-03-28T23:29:45.559643+0000","last_unstale":"2024-03-28T23:29:51.686047+0000","last_undegraded":"2024-03-28T23:29:51.686047+0000","last_fullsized":"2024-03-28T23:29:51.686047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:01.983721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687005+0000","last_change":"2024-03-28T23:29:50.595120+0000","last_active":"2024-03-28T23:29:51.687005+0000","last_peered":"2024-03-28T23:29:51.687005+0000","last_clean":"2024-03-28T23:29:51.687005+0000","last_became_active":"2024-03-28T23:29:50.594780+0000","last_became_peered":"2024-03-28T23:29:50.594780+0000","last_unstale":"2024-03-28T23:29:51.687005+0000","last_undegraded":"2024-03-28T23:29:51.687005+0000","last_fullsized":"2024-03-28T23:29:51.687005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:06.894069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686843+0000","last_change":"2024-03-28T23:29:46.567707+0000","last_active":"2024-03-28T23:29:51.686843+0000","last_peered":"2024-03-28T23:29:51.686843+0000","last_clean":"2024-03-28T23:29:51.686843+0000","last_became_active":"2024-03-28T23:29:46.565675+0000","last_became_peered":"2024-03-28T23:29:46.565675+0000","last_unstale":"2024-03-28T23:29:51.686843+0000","last_undegraded":"2024-03-28T23:29:51.686843+0000","last_fullsized":"2024-03-28T23:29:51.686843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:07:21.524344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589416+0000","last_change":"2024-03-28T23:29:45.558538+0000","last_active":"2024-03-28T23:29:51.589416+0000","last_peered":"2024-03-28T23:29:51.589416+0000","last_clean":"2024-03-28T23:29:51.589416+0000","last_became_active":"2024-03-28T23:29:45.558248+0000","last_became_peered":"2024-03-28T23:29:45.558248+0000","last_unstale":"2024-03-28T23:29:51.589416+0000","last_undegraded":"2024-03-28T23:29:51.589416+0000","last_fullsized":"2024-03-28T23:29:51.589416+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:59.719252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685418+0000","last_change":"2024-03-28T23:29:50.707640+0000","last_active":"2024-03-28T23:29:51.685418+0000","last_peered":"2024-03-28T23:29:51.685418+0000","last_clean":"2024-03-28T23:29:51.685418+0000","last_became_active":"2024-03-28T23:29:50.707258+0000","last_became_peered":"2024-03-28T23:29:50.707258+0000","last_unstale":"2024-03-28T23:29:51.685418+0000","last_undegraded":"2024-03-28T23:29:51.685418+0000","last_fullsized":"2024-03-28T23:29:51.685418+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:47.094789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686505+0000","last_change":"2024-03-28T23:29:46.565431+0000","last_active":"2024-03-28T23:29:51.686505+0000","last_peered":"2024-03-28T23:29:51.686505+0000","last_clean":"2024-03-28T23:29:51.686505+0000","last_became_active":"2024-03-28T23:29:46.565163+0000","last_became_peered":"2024-03-28T23:29:46.565163+0000","last_unstale":"2024-03-28T23:29:51.686505+0000","last_undegraded":"2024-03-28T23:29:51.686505+0000","last_fullsized":"2024-03-28T23:29:51.686505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:02.859502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591420+0000","last_change":"2024-03-28T23:29:45.564138+0000","last_active":"2024-03-28T23:29:51.591420+0000","last_peered":"2024-03-28T23:29:51.591420+0000","last_clean":"2024-03-28T23:29:51.591420+0000","last_became_active":"2024-03-28T23:29:45.563858+0000","last_became_peered":"2024-03-28T23:29:45.563858+0000","last_unstale":"2024-03-28T23:29:51.591420+0000","last_undegraded":"2024-03-28T23:29:51.591420+0000","last_fullsized":"2024-03-28T23:29:51.591420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:42:38.893826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.592126+0000","last_change":"2024-03-28T23:29:50.599484+0000","last_active":"2024-03-28T23:29:51.592126+0000","last_peered":"2024-03-28T23:29:51.592126+0000","last_clean":"2024-03-28T23:29:51.592126+0000","last_became_active":"2024-03-28T23:29:50.599035+0000","last_became_peered":"2024-03-28T23:29:50.599035+0000","last_unstale":"2024-03-28T23:29:51.592126+0000","last_undegraded":"2024-03-28T23:29:51.592126+0000","last_fullsized":"2024-03-28T23:29:51.592126+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:10.765830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686919+0000","last_change":"2024-03-28T23:29:46.567472+0000","last_active":"2024-03-28T23:29:51.686919+0000","last_peered":"2024-03-28T23:29:51.686919+0000","last_clean":"2024-03-28T23:29:51.686919+0000","last_became_active":"2024-03-28T23:29:46.565485+0000","last_became_peered":"2024-03-28T23:29:46.565485+0000","last_unstale":"2024-03-28T23:29:51.686919+0000","last_undegraded":"2024-03-28T23:29:51.686919+0000","last_fullsized":"2024-03-28T23:29:51.686919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:29:20.669429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589103+0000","last_change":"2024-03-28T23:29:45.556004+0000","last_active":"2024-03-28T23:29:51.589103+0000","last_peered":"2024-03-28T23:29:51.589103+0000","last_clean":"2024-03-28T23:29:51.589103+0000","last_became_active":"2024-03-28T23:29:45.555701+0000","last_became_peered":"2024-03-28T23:29:45.555701+0000","last_unstale":"2024-03-28T23:29:51.589103+0000","last_undegraded":"2024-03-28T23:29:51.589103+0000","last_fullsized":"2024-03-28T23:29:51.589103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:06:18.025605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687430+0000","last_change":"2024-03-28T23:29:50.596439+0000","last_active":"2024-03-28T23:29:51.687430+0000","last_peered":"2024-03-28T23:29:51.687430+0000","last_clean":"2024-03-28T23:29:51.687430+0000","last_became_active":"2024-03-28T23:29:50.596065+0000","last_became_peered":"2024-03-28T23:29:50.596065+0000","last_unstale":"2024-03-28T23:29:51.687430+0000","last_undegraded":"2024-03-28T23:29:51.687430+0000","last_fullsized":"2024-03-28T23:29:51.687430+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:43:14.415039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.590719+0000","last_change":"2024-03-28T23:29:46.581708+0000","last_active":"2024-03-28T23:29:51.590719+0000","last_peered":"2024-03-28T23:29:51.590719+0000","last_clean":"2024-03-28T23:29:51.590719+0000","last_became_active":"2024-03-28T23:29:46.581391+0000","last_became_peered":"2024-03-28T23:29:46.581391+0000","last_unstale":"2024-03-28T23:29:51.590719+0000","last_undegraded":"2024-03-28T23:29:51.590719+0000","last_fullsized":"2024-03-28T23:29:51.590719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:24:19.353047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591187+0000","last_change":"2024-03-28T23:29:45.560112+0000","last_active":"2024-03-28T23:29:51.591187+0000","last_peered":"2024-03-28T23:29:51.591187+0000","last_clean":"2024-03-28T23:29:51.591187+0000","last_became_active":"2024-03-28T23:29:45.559817+0000","last_became_peered":"2024-03-28T23:29:45.559817+0000","last_unstale":"2024-03-28T23:29:51.591187+0000","last_undegraded":"2024-03-28T23:29:51.591187+0000","last_fullsized":"2024-03-28T23:29:51.591187+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:21.707459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687297+0000","last_change":"2024-03-28T23:29:50.712185+0000","last_active":"2024-03-28T23:29:51.687297+0000","last_peered":"2024-03-28T23:29:51.687297+0000","last_clean":"2024-03-28T23:29:51.687297+0000","last_became_active":"2024-03-28T23:29:50.711823+0000","last_became_peered":"2024-03-28T23:29:50.711823+0000","last_unstale":"2024-03-28T23:29:51.687297+0000","last_undegraded":"2024-03-28T23:29:51.687297+0000","last_fullsized":"2024-03-28T23:29:51.687297+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:40.107649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685755+0000","last_change":"2024-03-28T23:29:46.570826+0000","last_active":"2024-03-28T23:29:51.685755+0000","last_peered":"2024-03-28T23:29:51.685755+0000","last_clean":"2024-03-28T23:29:51.685755+0000","last_became_active":"2024-03-28T23:29:46.570513+0000","last_became_peered":"2024-03-28T23:29:46.570513+0000","last_unstale":"2024-03-28T23:29:51.685755+0000","last_undegraded":"2024-03-28T23:29:51.685755+0000","last_fullsized":"2024-03-28T23:29:51.685755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:08:33.801844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686189+0000","last_change":"2024-03-28T23:29:45.554954+0000","last_active":"2024-03-28T23:29:51.686189+0000","last_peered":"2024-03-28T23:29:51.686189+0000","last_clean":"2024-03-28T23:29:51.686189+0000","last_became_active":"2024-03-28T23:29:45.554687+0000","last_became_peered":"2024-03-28T23:29:45.554687+0000","last_unstale":"2024-03-28T23:29:51.686189+0000","last_undegraded":"2024-03-28T23:29:51.686189+0000","last_fullsized":"2024-03-28T23:29:51.686189+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:24:58.784056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687871+0000","last_change":"2024-03-28T23:29:50.707207+0000","last_active":"2024-03-28T23:29:51.687871+0000","last_peered":"2024-03-28T23:29:51.687871+0000","last_clean":"2024-03-28T23:29:51.687871+0000","last_became_active":"2024-03-28T23:29:50.706585+0000","last_became_peered":"2024-03-28T23:29:50.706585+0000","last_unstale":"2024-03-28T23:29:51.687871+0000","last_undegraded":"2024-03-28T23:29:51.687871+0000","last_fullsized":"2024-03-28T23:29:51.687871+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:10:11.936256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687068+0000","last_change":"2024-03-28T23:29:46.578519+0000","last_active":"2024-03-28T23:29:51.687068+0000","last_peered":"2024-03-28T23:29:51.687068+0000","last_clean":"2024-03-28T23:29:51.687068+0000","last_became_active":"2024-03-28T23:29:46.577803+0000","last_became_peered":"2024-03-28T23:29:46.577803+0000","last_unstale":"2024-03-28T23:29:51.687068+0000","last_undegraded":"2024-03-28T23:29:51.687068+0000","last_fullsized":"2024-03-28T23:29:51.687068+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:32.936386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687237+0000","last_change":"2024-03-28T23:29:45.557114+0000","last_active":"2024-03-28T23:29:51.687237+0000","last_peered":"2024-03-28T23:29:51.687237+0000","last_clean":"2024-03-28T23:29:51.687237+0000","last_became_active":"2024-03-28T23:29:45.556800+0000","last_became_peered":"2024-03-28T23:29:45.556800+0000","last_unstale":"2024-03-28T23:29:51.687237+0000","last_undegraded":"2024-03-28T23:29:51.687237+0000","last_fullsized":"2024-03-28T23:29:51.687237+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:06:18.025605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686083+0000","last_change":"2024-03-28T23:29:50.707356+0000","last_active":"2024-03-28T23:29:51.686083+0000","last_peered":"2024-03-28T23:29:51.686083+0000","last_clean":"2024-03-28T23:29:51.686083+0000","last_became_active":"2024-03-28T23:29:50.706320+0000","last_became_peered":"2024-03-28T23:29:50.706320+0000","last_unstale":"2024-03-28T23:29:51.686083+0000","last_undegraded":"2024-03-28T23:29:51.686083+0000","last_fullsized":"2024-03-28T23:29:51.686083+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:36.341186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686444+0000","last_change":"2024-03-28T23:29:46.573192+0000","last_active":"2024-03-28T23:29:51.686444+0000","last_peered":"2024-03-28T23:29:51.686444+0000","last_clean":"2024-03-28T23:29:51.686444+0000","last_became_active":"2024-03-28T23:29:46.572701+0000","last_became_peered":"2024-03-28T23:29:46.572701+0000","last_unstale":"2024-03-28T23:29:51.686444+0000","last_undegraded":"2024-03-28T23:29:51.686444+0000","last_fullsized":"2024-03-28T23:29:51.686444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:17:49.979210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:56.151164+0000","last_change":"2024-03-28T23:29:45.558929+0000","last_active":"2024-03-28T23:29:56.151164+0000","last_peered":"2024-03-28T23:29:56.151164+0000","last_clean":"2024-03-28T23:29:56.151164+0000","last_became_active":"2024-03-28T23:29:45.558657+0000","last_became_peered":"2024-03-28T23:29:45.558657+0000","last_unstale":"2024-03-28T23:29:56.151164+0000","last_undegraded":"2024-03-28T23:29:56.151164+0000","last_fullsized":"2024-03-28T23:29:56.151164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:24:58.784056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686265+0000","last_change":"2024-03-28T23:29:50.594405+0000","last_active":"2024-03-28T23:29:51.686265+0000","last_peered":"2024-03-28T23:29:51.686265+0000","last_clean":"2024-03-28T23:29:51.686265+0000","last_became_active":"2024-03-28T23:29:50.594017+0000","last_became_peered":"2024-03-28T23:29:50.594017+0000","last_unstale":"2024-03-28T23:29:51.686265+0000","last_undegraded":"2024-03-28T23:29:51.686265+0000","last_fullsized":"2024-03-28T23:29:51.686265+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:06.894069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591773+0000","last_change":"2024-03-28T23:29:46.561971+0000","last_active":"2024-03-28T23:29:51.591773+0000","last_peered":"2024-03-28T23:29:51.591773+0000","last_clean":"2024-03-28T23:29:51.591773+0000","last_became_active":"2024-03-28T23:29:46.561684+0000","last_became_peered":"2024-03-28T23:29:46.561684+0000","last_unstale":"2024-03-28T23:29:51.591773+0000","last_undegraded":"2024-03-28T23:29:51.591773+0000","last_fullsized":"2024-03-28T23:29:51.591773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:42.629935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685953+0000","last_change":"2024-03-28T23:29:45.563130+0000","last_active":"2024-03-28T23:29:51.685953+0000","last_peered":"2024-03-28T23:29:51.685953+0000","last_clean":"2024-03-28T23:29:51.685953+0000","last_became_active":"2024-03-28T23:29:45.562696+0000","last_became_peered":"2024-03-28T23:29:45.562696+0000","last_unstale":"2024-03-28T23:29:51.685953+0000","last_undegraded":"2024-03-28T23:29:51.685953+0000","last_fullsized":"2024-03-28T23:29:51.685953+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:17.783972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.592114+0000","last_change":"2024-03-28T23:29:50.587580+0000","last_active":"2024-03-28T23:29:51.592114+0000","last_peered":"2024-03-28T23:29:51.592114+0000","last_clean":"2024-03-28T23:29:51.592114+0000","last_became_active":"2024-03-28T23:29:50.587175+0000","last_became_peered":"2024-03-28T23:29:50.587175+0000","last_unstale":"2024-03-28T23:29:51.592114+0000","last_undegraded":"2024-03-28T23:29:51.592114+0000","last_fullsized":"2024-03-28T23:29:51.592114+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:06:40.917529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687664+0000","last_change":"2024-03-28T23:29:46.578684+0000","last_active":"2024-03-28T23:29:51.687664+0000","last_peered":"2024-03-28T23:29:51.687664+0000","last_clean":"2024-03-28T23:29:51.687664+0000","last_became_active":"2024-03-28T23:29:46.577924+0000","last_became_peered":"2024-03-28T23:29:46.577924+0000","last_unstale":"2024-03-28T23:29:51.687664+0000","last_undegraded":"2024-03-28T23:29:51.687664+0000","last_fullsized":"2024-03-28T23:29:51.687664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:36:20.485861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685874+0000","last_change":"2024-03-28T23:29:45.559534+0000","last_active":"2024-03-28T23:29:51.685874+0000","last_peered":"2024-03-28T23:29:51.685874+0000","last_clean":"2024-03-28T23:29:51.685874+0000","last_became_active":"2024-03-28T23:29:45.559243+0000","last_became_peered":"2024-03-28T23:29:45.559243+0000","last_unstale":"2024-03-28T23:29:51.685874+0000","last_undegraded":"2024-03-28T23:29:51.685874+0000","last_fullsized":"2024-03-28T23:29:51.685874+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:17:17.207826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686108+0000","last_change":"2024-03-28T23:29:50.601743+0000","last_active":"2024-03-28T23:29:51.686108+0000","last_peered":"2024-03-28T23:29:51.686108+0000","last_clean":"2024-03-28T23:29:51.686108+0000","last_became_active":"2024-03-28T23:29:50.601327+0000","last_became_peered":"2024-03-28T23:29:50.601327+0000","last_unstale":"2024-03-28T23:29:51.686108+0000","last_undegraded":"2024-03-28T23:29:51.686108+0000","last_fullsized":"2024-03-28T23:29:51.686108+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:36.341186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685724+0000","last_change":"2024-03-28T23:29:46.573188+0000","last_active":"2024-03-28T23:29:51.685724+0000","last_peered":"2024-03-28T23:29:51.685724+0000","last_clean":"2024-03-28T23:29:51.685724+0000","last_became_active":"2024-03-28T23:29:46.572885+0000","last_became_peered":"2024-03-28T23:29:46.572885+0000","last_unstale":"2024-03-28T23:29:51.685724+0000","last_undegraded":"2024-03-28T23:29:51.685724+0000","last_fullsized":"2024-03-28T23:29:51.685724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:08:33.801844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591244+0000","last_change":"2024-03-28T23:29:45.564128+0000","last_active":"2024-03-28T23:29:51.591244+0000","last_peered":"2024-03-28T23:29:51.591244+0000","last_clean":"2024-03-28T23:29:51.591244+0000","last_became_active":"2024-03-28T23:29:45.563837+0000","last_became_peered":"2024-03-28T23:29:45.563837+0000","last_unstale":"2024-03-28T23:29:51.591244+0000","last_undegraded":"2024-03-28T23:29:51.591244+0000","last_fullsized":"2024-03-28T23:29:51.591244+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:17:17.207826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686626+0000","last_change":"2024-03-28T23:29:50.708325+0000","last_active":"2024-03-28T23:29:51.686626+0000","last_peered":"2024-03-28T23:29:51.686626+0000","last_clean":"2024-03-28T23:29:51.686626+0000","last_became_active":"2024-03-28T23:29:50.707929+0000","last_became_peered":"2024-03-28T23:29:50.707929+0000","last_unstale":"2024-03-28T23:29:51.686626+0000","last_undegraded":"2024-03-28T23:29:51.686626+0000","last_fullsized":"2024-03-28T23:29:51.686626+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:01.743074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687303+0000","last_change":"2024-03-28T23:29:50.711341+0000","last_active":"2024-03-28T23:29:51.687303+0000","last_peered":"2024-03-28T23:29:51.687303+0000","last_clean":"2024-03-28T23:29:51.687303+0000","last_became_active":"2024-03-28T23:29:50.710983+0000","last_became_peered":"2024-03-28T23:29:50.710983+0000","last_unstale":"2024-03-28T23:29:51.687303+0000","last_undegraded":"2024-03-28T23:29:51.687303+0000","last_fullsized":"2024-03-28T23:29:51.687303+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:30:48.379529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685433+0000","last_change":"2024-03-28T23:29:46.561654+0000","last_active":"2024-03-28T23:29:51.685433+0000","last_peered":"2024-03-28T23:29:51.685433+0000","last_clean":"2024-03-28T23:29:51.685433+0000","last_became_active":"2024-03-28T23:29:46.561250+0000","last_became_peered":"2024-03-28T23:29:46.561250+0000","last_unstale":"2024-03-28T23:29:51.685433+0000","last_undegraded":"2024-03-28T23:29:51.685433+0000","last_fullsized":"2024-03-28T23:29:51.685433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:38.074388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685648+0000","last_change":"2024-03-28T23:29:45.553472+0000","last_active":"2024-03-28T23:29:51.685648+0000","last_peered":"2024-03-28T23:29:51.685648+0000","last_clean":"2024-03-28T23:29:51.685648+0000","last_became_active":"2024-03-28T23:29:45.553154+0000","last_became_peered":"2024-03-28T23:29:45.553154+0000","last_unstale":"2024-03-28T23:29:51.685648+0000","last_undegraded":"2024-03-28T23:29:51.685648+0000","last_fullsized":"2024-03-28T23:29:51.685648+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:36.113647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591126+0000","last_change":"2024-03-28T23:29:50.600387+0000","last_active":"2024-03-28T23:29:51.591126+0000","last_peered":"2024-03-28T23:29:51.591126+0000","last_clean":"2024-03-28T23:29:51.591126+0000","last_became_active":"2024-03-28T23:29:50.599850+0000","last_became_peered":"2024-03-28T23:29:50.599850+0000","last_unstale":"2024-03-28T23:29:51.591126+0000","last_undegraded":"2024-03-28T23:29:51.591126+0000","last_fullsized":"2024-03-28T23:29:51.591126+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:08:37.836411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686149+0000","last_change":"2024-03-28T23:29:46.581533+0000","last_active":"2024-03-28T23:29:51.686149+0000","last_peered":"2024-03-28T23:29:51.686149+0000","last_clean":"2024-03-28T23:29:51.686149+0000","last_became_active":"2024-03-28T23:29:46.581288+0000","last_became_peered":"2024-03-28T23:29:46.581288+0000","last_unstale":"2024-03-28T23:29:51.686149+0000","last_undegraded":"2024-03-28T23:29:51.686149+0000","last_fullsized":"2024-03-28T23:29:51.686149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:24:19.353047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686836+0000","last_change":"2024-03-28T23:29:45.561047+0000","last_active":"2024-03-28T23:29:51.686836+0000","last_peered":"2024-03-28T23:29:51.686836+0000","last_clean":"2024-03-28T23:29:51.686836+0000","last_became_active":"2024-03-28T23:29:45.560778+0000","last_became_peered":"2024-03-28T23:29:45.560778+0000","last_unstale":"2024-03-28T23:29:51.686836+0000","last_undegraded":"2024-03-28T23:29:51.686836+0000","last_fullsized":"2024-03-28T23:29:51.686836+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:21.707459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.590800+0000","last_change":"2024-03-28T23:29:45.559446+0000","last_active":"2024-03-28T23:29:51.590800+0000","last_peered":"2024-03-28T23:29:51.590800+0000","last_clean":"2024-03-28T23:29:51.590800+0000","last_became_active":"2024-03-28T23:29:45.559015+0000","last_became_peered":"2024-03-28T23:29:45.559015+0000","last_unstale":"2024-03-28T23:29:51.590800+0000","last_undegraded":"2024-03-28T23:29:51.590800+0000","last_fullsized":"2024-03-28T23:29:51.590800+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:59.719252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687152+0000","last_change":"2024-03-28T23:29:46.578355+0000","last_active":"2024-03-28T23:29:51.687152+0000","last_peered":"2024-03-28T23:29:51.687152+0000","last_clean":"2024-03-28T23:29:51.687152+0000","last_became_active":"2024-03-28T23:29:46.577898+0000","last_became_peered":"2024-03-28T23:29:46.577898+0000","last_unstale":"2024-03-28T23:29:51.687152+0000","last_undegraded":"2024-03-28T23:29:51.687152+0000","last_fullsized":"2024-03-28T23:29:51.687152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:58.674327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589392+0000","last_change":"2024-03-28T23:29:50.601015+0000","last_active":"2024-03-28T23:29:51.589392+0000","last_peered":"2024-03-28T23:29:51.589392+0000","last_clean":"2024-03-28T23:29:51.589392+0000","last_became_active":"2024-03-28T23:29:50.600573+0000","last_became_peered":"2024-03-28T23:29:50.600573+0000","last_unstale":"2024-03-28T23:29:51.589392+0000","last_undegraded":"2024-03-28T23:29:51.589392+0000","last_fullsized":"2024-03-28T23:29:51.589392+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:38:05.665416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686050+0000","last_change":"2024-03-28T23:29:45.556159+0000","last_active":"2024-03-28T23:29:51.686050+0000","last_peered":"2024-03-28T23:29:51.686050+0000","last_clean":"2024-03-28T23:29:51.686050+0000","last_became_active":"2024-03-28T23:29:45.555866+0000","last_became_peered":"2024-03-28T23:29:45.555866+0000","last_unstale":"2024-03-28T23:29:51.686050+0000","last_undegraded":"2024-03-28T23:29:51.686050+0000","last_fullsized":"2024-03-28T23:29:51.686050+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:17:17.207826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591783+0000","last_change":"2024-03-28T23:29:46.554056+0000","last_active":"2024-03-28T23:29:51.591783+0000","last_peered":"2024-03-28T23:29:51.591783+0000","last_clean":"2024-03-28T23:29:51.591783+0000","last_became_active":"2024-03-28T23:29:46.553772+0000","last_became_peered":"2024-03-28T23:29:46.553772+0000","last_unstale":"2024-03-28T23:29:51.591783+0000","last_undegraded":"2024-03-28T23:29:51.591783+0000","last_fullsized":"2024-03-28T23:29:51.591783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:02.992468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686381+0000","last_change":"2024-03-28T23:29:50.603254+0000","last_active":"2024-03-28T23:29:51.686381+0000","last_peered":"2024-03-28T23:29:51.686381+0000","last_clean":"2024-03-28T23:29:51.686381+0000","last_became_active":"2024-03-28T23:29:50.602868+0000","last_became_peered":"2024-03-28T23:29:50.602868+0000","last_unstale":"2024-03-28T23:29:51.686381+0000","last_undegraded":"2024-03-28T23:29:51.686381+0000","last_fullsized":"2024-03-28T23:29:51.686381+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:42:43.014557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.590502+0000","last_change":"2024-03-28T23:29:46.581634+0000","last_active":"2024-03-28T23:29:51.590502+0000","last_peered":"2024-03-28T23:29:51.590502+0000","last_clean":"2024-03-28T23:29:51.590502+0000","last_became_active":"2024-03-28T23:29:46.581353+0000","last_became_peered":"2024-03-28T23:29:46.581353+0000","last_unstale":"2024-03-28T23:29:51.590502+0000","last_undegraded":"2024-03-28T23:29:51.590502+0000","last_fullsized":"2024-03-28T23:29:51.590502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:20.039954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589110+0000","last_change":"2024-03-28T23:29:46.563584+0000","last_active":"2024-03-28T23:29:51.589110+0000","last_peered":"2024-03-28T23:29:51.589110+0000","last_clean":"2024-03-28T23:29:51.589110+0000","last_became_active":"2024-03-28T23:29:46.563265+0000","last_became_peered":"2024-03-28T23:29:46.563265+0000","last_unstale":"2024-03-28T23:29:51.589110+0000","last_undegraded":"2024-03-28T23:29:51.589110+0000","last_fullsized":"2024-03-28T23:29:51.589110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:42.629935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685798+0000","last_change":"2024-03-28T23:29:46.561377+0000","last_active":"2024-03-28T23:29:51.685798+0000","last_peered":"2024-03-28T23:29:51.685798+0000","last_clean":"2024-03-28T23:29:51.685798+0000","last_became_active":"2024-03-28T23:29:46.561091+0000","last_became_peered":"2024-03-28T23:29:46.561091+0000","last_unstale":"2024-03-28T23:29:51.685798+0000","last_undegraded":"2024-03-28T23:29:51.685798+0000","last_fullsized":"2024-03-28T23:29:51.685798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:30:44.344961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687062+0000","last_change":"2024-03-28T23:29:46.578999+0000","last_active":"2024-03-28T23:29:51.687062+0000","last_peered":"2024-03-28T23:29:51.687062+0000","last_clean":"2024-03-28T23:29:51.687062+0000","last_became_active":"2024-03-28T23:29:46.578263+0000","last_became_peered":"2024-03-28T23:29:46.578263+0000","last_unstale":"2024-03-28T23:29:51.687062+0000","last_undegraded":"2024-03-28T23:29:51.687062+0000","last_fullsized":"2024-03-28T23:29:51.687062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:10.853710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687925+0000","last_change":"2024-03-28T23:29:46.566611+0000","last_active":"2024-03-28T23:29:51.687925+0000","last_peered":"2024-03-28T23:29:51.687925+0000","last_clean":"2024-03-28T23:29:51.687925+0000","last_became_active":"2024-03-28T23:29:46.566361+0000","last_became_peered":"2024-03-28T23:29:46.566361+0000","last_unstale":"2024-03-28T23:29:51.687925+0000","last_undegraded":"2024-03-28T23:29:51.687925+0000","last_fullsized":"2024-03-28T23:29:51.687925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:29:20.669429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685784+0000","last_change":"2024-03-28T23:29:46.561125+0000","last_active":"2024-03-28T23:29:51.685784+0000","last_peered":"2024-03-28T23:29:51.685784+0000","last_clean":"2024-03-28T23:29:51.685784+0000","last_became_active":"2024-03-28T23:29:46.559223+0000","last_became_peered":"2024-03-28T23:29:46.559223+0000","last_unstale":"2024-03-28T23:29:51.685784+0000","last_undegraded":"2024-03-28T23:29:51.685784+0000","last_fullsized":"2024-03-28T23:29:51.685784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:36:20.485861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591456+0000","last_change":"2024-03-28T23:29:46.574608+0000","last_active":"2024-03-28T23:29:51.591456+0000","last_peered":"2024-03-28T23:29:51.591456+0000","last_clean":"2024-03-28T23:29:51.591456+0000","last_became_active":"2024-03-28T23:29:46.574329+0000","last_became_peered":"2024-03-28T23:29:46.574329+0000","last_unstale":"2024-03-28T23:29:51.591456+0000","last_undegraded":"2024-03-28T23:29:51.591456+0000","last_fullsized":"2024-03-28T23:29:51.591456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:15.386423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687344+0000","last_change":"2024-03-28T23:29:46.566187+0000","last_active":"2024-03-28T23:29:51.687344+0000","last_peered":"2024-03-28T23:29:51.687344+0000","last_clean":"2024-03-28T23:29:51.687344+0000","last_became_active":"2024-03-28T23:29:46.565873+0000","last_became_peered":"2024-03-28T23:29:46.565873+0000","last_unstale":"2024-03-28T23:29:51.687344+0000","last_undegraded":"2024-03-28T23:29:51.687344+0000","last_fullsized":"2024-03-28T23:29:51.687344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:14:59.182189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685520+0000","last_change":"2024-03-28T23:29:46.559531+0000","last_active":"2024-03-28T23:29:51.685520+0000","last_peered":"2024-03-28T23:29:51.685520+0000","last_clean":"2024-03-28T23:29:51.685520+0000","last_became_active":"2024-03-28T23:29:46.559236+0000","last_became_peered":"2024-03-28T23:29:46.559236+0000","last_unstale":"2024-03-28T23:29:51.685520+0000","last_undegraded":"2024-03-28T23:29:51.685520+0000","last_fullsized":"2024-03-28T23:29:51.685520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:06.731262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686762+0000","last_change":"2024-03-28T23:29:46.555898+0000","last_active":"2024-03-28T23:29:51.686762+0000","last_peered":"2024-03-28T23:29:51.686762+0000","last_clean":"2024-03-28T23:29:51.686762+0000","last_became_active":"2024-03-28T23:29:46.555610+0000","last_became_peered":"2024-03-28T23:29:46.555610+0000","last_unstale":"2024-03-28T23:29:51.686762+0000","last_undegraded":"2024-03-28T23:29:51.686762+0000","last_fullsized":"2024-03-28T23:29:51.686762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:38.074388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685565+0000","last_change":"2024-03-28T23:29:46.573905+0000","last_active":"2024-03-28T23:29:51.685565+0000","last_peered":"2024-03-28T23:29:51.685565+0000","last_clean":"2024-03-28T23:29:51.685565+0000","last_became_active":"2024-03-28T23:29:46.573631+0000","last_became_peered":"2024-03-28T23:29:46.573631+0000","last_unstale":"2024-03-28T23:29:51.685565+0000","last_undegraded":"2024-03-28T23:29:51.685565+0000","last_fullsized":"2024-03-28T23:29:51.685565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:14:59.182189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686795+0000","last_change":"2024-03-28T23:29:46.579400+0000","last_active":"2024-03-28T23:29:51.686795+0000","last_peered":"2024-03-28T23:29:51.686795+0000","last_clean":"2024-03-28T23:29:51.686795+0000","last_became_active":"2024-03-28T23:29:46.579070+0000","last_became_peered":"2024-03-28T23:29:46.579070+0000","last_unstale":"2024-03-28T23:29:51.686795+0000","last_undegraded":"2024-03-28T23:29:51.686795+0000","last_fullsized":"2024-03-28T23:29:51.686795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:15.386423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591130+0000","last_change":"2024-03-28T23:29:46.569441+0000","last_active":"2024-03-28T23:29:51.591130+0000","last_peered":"2024-03-28T23:29:51.591130+0000","last_clean":"2024-03-28T23:29:51.591130+0000","last_became_active":"2024-03-28T23:29:46.569151+0000","last_became_peered":"2024-03-28T23:29:46.569151+0000","last_unstale":"2024-03-28T23:29:51.591130+0000","last_undegraded":"2024-03-28T23:29:51.591130+0000","last_fullsized":"2024-03-28T23:29:51.591130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:38.074388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686720+0000","last_change":"2024-03-28T23:29:46.573927+0000","last_active":"2024-03-28T23:29:51.686720+0000","last_peered":"2024-03-28T23:29:51.686720+0000","last_clean":"2024-03-28T23:29:51.686720+0000","last_became_active":"2024-03-28T23:29:46.573668+0000","last_became_peered":"2024-03-28T23:29:46.573668+0000","last_unstale":"2024-03-28T23:29:51.686720+0000","last_undegraded":"2024-03-28T23:29:51.686720+0000","last_fullsized":"2024-03-28T23:29:51.686720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:30:44.344961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589322+0000","last_change":"2024-03-28T23:29:46.558629+0000","last_active":"2024-03-28T23:29:51.589322+0000","last_peered":"2024-03-28T23:29:51.589322+0000","last_clean":"2024-03-28T23:29:51.589322+0000","last_became_active":"2024-03-28T23:29:46.558311+0000","last_became_peered":"2024-03-28T23:29:46.558311+0000","last_unstale":"2024-03-28T23:29:51.589322+0000","last_undegraded":"2024-03-28T23:29:51.589322+0000","last_fullsized":"2024-03-28T23:29:51.589322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:10.938731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686262+0000","last_change":"2024-03-28T23:29:46.573394+0000","last_active":"2024-03-28T23:29:51.686262+0000","last_peered":"2024-03-28T23:29:51.686262+0000","last_clean":"2024-03-28T23:29:51.686262+0000","last_became_active":"2024-03-28T23:29:46.573130+0000","last_became_peered":"2024-03-28T23:29:46.573130+0000","last_unstale":"2024-03-28T23:29:51.686262+0000","last_undegraded":"2024-03-28T23:29:51.686262+0000","last_fullsized":"2024-03-28T23:29:51.686262+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:43:10.380471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685367+0000","last_change":"2024-03-28T23:29:46.581612+0000","last_active":"2024-03-28T23:29:51.685367+0000","last_peered":"2024-03-28T23:29:51.685367+0000","last_clean":"2024-03-28T23:29:51.685367+0000","last_became_active":"2024-03-28T23:29:46.581324+0000","last_became_peered":"2024-03-28T23:29:46.581324+0000","last_unstale":"2024-03-28T23:29:51.685367+0000","last_undegraded":"2024-03-28T23:29:51.685367+0000","last_fullsized":"2024-03-28T23:29:51.685367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:10.853710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685527+0000","last_change":"2024-03-28T23:29:46.552565+0000","last_active":"2024-03-28T23:29:51.685527+0000","last_peered":"2024-03-28T23:29:51.685527+0000","last_clean":"2024-03-28T23:29:51.685527+0000","last_became_active":"2024-03-28T23:29:46.552304+0000","last_became_peered":"2024-03-28T23:29:46.552304+0000","last_unstale":"2024-03-28T23:29:51.685527+0000","last_undegraded":"2024-03-28T23:29:51.685527+0000","last_fullsized":"2024-03-28T23:29:51.685527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:15.386423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686970+0000","last_change":"2024-03-28T23:29:46.579218+0000","last_active":"2024-03-28T23:29:51.686970+0000","last_peered":"2024-03-28T23:29:51.686970+0000","last_clean":"2024-03-28T23:29:51.686970+0000","last_became_active":"2024-03-28T23:29:46.578756+0000","last_became_peered":"2024-03-28T23:29:46.578756+0000","last_unstale":"2024-03-28T23:29:51.686970+0000","last_undegraded":"2024-03-28T23:29:51.686970+0000","last_fullsized":"2024-03-28T23:29:51.686970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:24:19.353047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685891+0000","last_change":"2024-03-28T23:29:46.556797+0000","last_active":"2024-03-28T23:29:51.685891+0000","last_peered":"2024-03-28T23:29:51.685891+0000","last_clean":"2024-03-28T23:29:51.685891+0000","last_became_active":"2024-03-28T23:29:46.556520+0000","last_became_peered":"2024-03-28T23:29:46.556520+0000","last_unstale":"2024-03-28T23:29:51.685891+0000","last_undegraded":"2024-03-28T23:29:51.685891+0000","last_fullsized":"2024-03-28T23:29:51.685891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:15.386423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591677+0000","last_change":"2024-03-28T23:29:46.559749+0000","last_active":"2024-03-28T23:29:51.591677+0000","last_peered":"2024-03-28T23:29:51.591677+0000","last_clean":"2024-03-28T23:29:51.591677+0000","last_became_active":"2024-03-28T23:29:46.559386+0000","last_became_peered":"2024-03-28T23:29:46.559386+0000","last_unstale":"2024-03-28T23:29:51.591677+0000","last_undegraded":"2024-03-28T23:29:51.591677+0000","last_fullsized":"2024-03-28T23:29:51.591677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:10.938731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591382+0000","last_change":"2024-03-28T23:29:46.571568+0000","last_active":"2024-03-28T23:29:51.591382+0000","last_peered":"2024-03-28T23:29:51.591382+0000","last_clean":"2024-03-28T23:29:51.591382+0000","last_became_active":"2024-03-28T23:29:46.571268+0000","last_became_peered":"2024-03-28T23:29:46.571268+0000","last_unstale":"2024-03-28T23:29:51.591382+0000","last_undegraded":"2024-03-28T23:29:51.591382+0000","last_fullsized":"2024-03-28T23:29:51.591382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:36:20.485861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591361+0000","last_change":"2024-03-28T23:29:46.571722+0000","last_active":"2024-03-28T23:29:51.591361+0000","last_peered":"2024-03-28T23:29:51.591361+0000","last_clean":"2024-03-28T23:29:51.591361+0000","last_became_active":"2024-03-28T23:29:46.571437+0000","last_became_peered":"2024-03-28T23:29:46.571437+0000","last_unstale":"2024-03-28T23:29:51.591361+0000","last_undegraded":"2024-03-28T23:29:51.591361+0000","last_fullsized":"2024-03-28T23:29:51.591361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:10.853710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686822+0000","last_change":"2024-03-28T23:29:46.565939+0000","last_active":"2024-03-28T23:29:51.686822+0000","last_peered":"2024-03-28T23:29:51.686822+0000","last_clean":"2024-03-28T23:29:51.686822+0000","last_became_active":"2024-03-28T23:29:46.564853+0000","last_became_peered":"2024-03-28T23:29:46.564853+0000","last_unstale":"2024-03-28T23:29:51.686822+0000","last_undegraded":"2024-03-28T23:29:51.686822+0000","last_fullsized":"2024-03-28T23:29:51.686822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:18.050658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685938+0000","last_change":"2024-03-28T23:29:46.567115+0000","last_active":"2024-03-28T23:29:51.685938+0000","last_peered":"2024-03-28T23:29:51.685938+0000","last_clean":"2024-03-28T23:29:51.685938+0000","last_became_active":"2024-03-28T23:29:46.566742+0000","last_became_peered":"2024-03-28T23:29:46.566742+0000","last_unstale":"2024-03-28T23:29:51.685938+0000","last_undegraded":"2024-03-28T23:29:51.685938+0000","last_fullsized":"2024-03-28T23:29:51.685938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:32.936386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686798+0000","last_change":"2024-03-28T23:29:46.579765+0000","last_active":"2024-03-28T23:29:51.686798+0000","last_peered":"2024-03-28T23:29:51.686798+0000","last_clean":"2024-03-28T23:29:51.686798+0000","last_became_active":"2024-03-28T23:29:46.579469+0000","last_became_peered":"2024-03-28T23:29:46.579469+0000","last_unstale":"2024-03-28T23:29:51.686798+0000","last_undegraded":"2024-03-28T23:29:51.686798+0000","last_fullsized":"2024-03-28T23:29:51.686798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:06.731262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685479+0000","last_change":"2024-03-28T23:29:46.580994+0000","last_active":"2024-03-28T23:29:51.685479+0000","last_peered":"2024-03-28T23:29:51.685479+0000","last_clean":"2024-03-28T23:29:51.685479+0000","last_became_active":"2024-03-28T23:29:46.580685+0000","last_became_peered":"2024-03-28T23:29:46.580685+0000","last_unstale":"2024-03-28T23:29:51.685479+0000","last_undegraded":"2024-03-28T23:29:51.685479+0000","last_fullsized":"2024-03-28T23:29:51.685479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:58.674327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685558+0000","last_change":"2024-03-28T23:29:46.571922+0000","last_active":"2024-03-28T23:29:51.685558+0000","last_peered":"2024-03-28T23:29:51.685558+0000","last_clean":"2024-03-28T23:29:51.685558+0000","last_became_active":"2024-03-28T23:29:46.571256+0000","last_became_peered":"2024-03-28T23:29:46.571256+0000","last_unstale":"2024-03-28T23:29:51.685558+0000","last_undegraded":"2024-03-28T23:29:51.685558+0000","last_fullsized":"2024-03-28T23:29:51.685558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:54.470202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685672+0000","last_change":"2024-03-28T23:29:46.561038+0000","last_active":"2024-03-28T23:29:51.685672+0000","last_peered":"2024-03-28T23:29:51.685672+0000","last_clean":"2024-03-28T23:29:51.685672+0000","last_became_active":"2024-03-28T23:29:46.560679+0000","last_became_peered":"2024-03-28T23:29:46.560679+0000","last_unstale":"2024-03-28T23:29:51.685672+0000","last_undegraded":"2024-03-28T23:29:51.685672+0000","last_fullsized":"2024-03-28T23:29:51.685672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:29:20.669429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687099+0000","last_change":"2024-03-28T23:29:46.578506+0000","last_active":"2024-03-28T23:29:51.687099+0000","last_peered":"2024-03-28T23:29:51.687099+0000","last_clean":"2024-03-28T23:29:51.687099+0000","last_became_active":"2024-03-28T23:29:46.577958+0000","last_became_peered":"2024-03-28T23:29:46.577958+0000","last_unstale":"2024-03-28T23:29:51.687099+0000","last_undegraded":"2024-03-28T23:29:51.687099+0000","last_fullsized":"2024-03-28T23:29:51.687099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:38:01.630848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686703+0000","last_change":"2024-03-28T23:29:46.579555+0000","last_active":"2024-03-28T23:29:51.686703+0000","last_peered":"2024-03-28T23:29:51.686703+0000","last_clean":"2024-03-28T23:29:51.686703+0000","last_became_active":"2024-03-28T23:29:46.579273+0000","last_became_peered":"2024-03-28T23:29:46.579273+0000","last_unstale":"2024-03-28T23:29:51.686703+0000","last_undegraded":"2024-03-28T23:29:51.686703+0000","last_fullsized":"2024-03-28T23:29:51.686703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:10.853710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686627+0000","last_change":"2024-03-28T23:29:46.568698+0000","last_active":"2024-03-28T23:29:51.686627+0000","last_peered":"2024-03-28T23:29:51.686627+0000","last_clean":"2024-03-28T23:29:51.686627+0000","last_became_active":"2024-03-28T23:29:46.568318+0000","last_became_peered":"2024-03-28T23:29:46.568318+0000","last_unstale":"2024-03-28T23:29:51.686627+0000","last_undegraded":"2024-03-28T23:29:51.686627+0000","last_fullsized":"2024-03-28T23:29:51.686627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:43:10.380471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31504,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340336,"statfs":{"total":96636764160,"available":96604504064,"internally_reserved":0,"allocated":323584,"data_stored":691206,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":331776,"data_stored":691262,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:30:10.147 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T23:30:10.147 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-28T23:30:10.147 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-28T23:30:10.148 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-28T23:30:10.148 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-28T23:30:10.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.284+0000 7f8b30d0f640 1 Processor -- start 2024-03-28T23:30:10.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.285+0000 7f8b30d0f640 1 -- start start 2024-03-28T23:30:10.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.285+0000 7f8b30d0f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8b2c06ca20 0x7f8b2c06ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:10.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.285+0000 7f8b30d0f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8b2c0a98c0 0x7f8b2c0a9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:10.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.285+0000 7f8b30d0f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b2c151850 con 0x7f8b2c14eca0 2024-03-28T23:30:10.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.285+0000 7f8b30d0f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8b2c152460 con 0x7f8b2c06ca20 2024-03-28T23:30:10.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.285+0000 7f8b30d0f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f8b2c151b30 con 0x7f8b2c0a98c0 2024-03-28T23:30:10.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.285+0000 7f8b23fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8b2c06ca20 0x7f8b2c06ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:10.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.285+0000 7f8b23fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8b2c06ca20 0x7f8b2c06ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45658/0 (socket says 172.21.15.161:45658) 2024-03-28T23:30:10.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.285+0000 7f8b23fff640 1 -- 172.21.15.161:0/4247968017 learned_addr learned my addr 172.21.15.161:0/4247968017 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:10.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.286+0000 7f8b2ad9d640 1 --2- 172.21.15.161:0/4247968017 >> v2:172.21.15.170:3300/0 conn(0x7f8b2c0a98c0 0x7f8b2c0a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:10.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.286+0000 7f8b23fff640 1 -- 172.21.15.161:0/4247968017 >> v2:172.21.15.170:3300/0 conn(0x7f8b2c0a98c0 msgr2=0x7f8b2c0a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:10.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.286+0000 7f8b23fff640 1 --2- 172.21.15.161:0/4247968017 >> v2:172.21.15.170:3300/0 conn(0x7f8b2c0a98c0 0x7f8b2c0a9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:10.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.286+0000 7f8b23fff640 1 -- 172.21.15.161:0/4247968017 >> v1:172.21.15.170:6789/0 conn(0x7f8b2c14eca0 legacy=0x7f8b2c0c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:10.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.286+0000 7f8b23fff640 1 -- 172.21.15.161:0/4247968017 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b2c151e10 con 0x7f8b2c06ca20 2024-03-28T23:30:10.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.286+0000 7f8b2ad9d640 1 --2- 172.21.15.161:0/4247968017 >> v2:172.21.15.170:3300/0 conn(0x7f8b2c0a98c0 0x7f8b2c0a9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:10.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.287+0000 7f8b23fff640 1 --2- 172.21.15.161:0/4247968017 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c06ca20 0x7f8b2c06ce80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c0c2f90 tx=0x7f8b1402cdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e64dd5d66864ac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:10.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.287+0000 7f8b29d9b640 1 -- 172.21.15.161:0/4247968017 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1403a070 con 0x7f8b2c06ca20 2024-03-28T23:30:10.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.287+0000 7f8b29d9b640 1 -- 172.21.15.161:0/4247968017 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b1402dba0 con 0x7f8b2c06ca20 2024-03-28T23:30:10.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.287+0000 7f8b29d9b640 1 -- 172.21.15.161:0/4247968017 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1400b040 con 0x7f8b2c06ca20 2024-03-28T23:30:10.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.288+0000 7f8b30d0f640 1 -- 172.21.15.161:0/4247968017 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c06ca20 msgr2=0x7f8b2c06ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:10.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.288+0000 7f8b30d0f640 1 --2- 172.21.15.161:0/4247968017 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c06ca20 0x7f8b2c06ce80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c0c2f90 tx=0x7f8b1402cdf0 comp rx=0 tx=0).stop 2024-03-28T23:30:10.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.288+0000 7f8b30d0f640 1 -- 172.21.15.161:0/4247968017 shutdown_connections 2024-03-28T23:30:10.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.288+0000 7f8b30d0f640 1 --2- 172.21.15.161:0/4247968017 >> v2:172.21.15.170:3300/0 conn(0x7f8b2c0a98c0 0x7f8b2c0a9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:10.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.288+0000 7f8b30d0f640 1 --2- 172.21.15.161:0/4247968017 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c06ca20 0x7f8b2c06ce80 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:10.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.288+0000 7f8b30d0f640 1 -- 172.21.15.161:0/4247968017 >> 172.21.15.161:0/4247968017 conn(0x7f8b2c0c6670 msgr2=0x7f8b2c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:10.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.288+0000 7f8b30d0f640 1 -- 172.21.15.161:0/4247968017 shutdown_connections 2024-03-28T23:30:10.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.288+0000 7f8b30d0f640 1 -- 172.21.15.161:0/4247968017 wait complete. 2024-03-28T23:30:10.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.289+0000 7f8b30d0f640 1 Processor -- start 2024-03-28T23:30:10.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.289+0000 7f8b30d0f640 1 -- start start 2024-03-28T23:30:10.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.290+0000 7f8b30d0f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c06ca20 0x7f8b2c1ee9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:10.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.290+0000 7f8b30d0f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b2c0a98c0 0x7f8b2c154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:10.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.290+0000 7f8b30d0f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b2c14eca0 0x7f8b2c1db300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:10.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.290+0000 7f8b30d0f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2c1514b0 con 0x7f8b2c06ca20 2024-03-28T23:30:10.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.290+0000 7f8b30d0f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2c14f080 con 0x7f8b2c0a98c0 2024-03-28T23:30:10.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.290+0000 7f8b30d0f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8b2c151630 con 0x7f8b2c14eca0 2024-03-28T23:30:10.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.290+0000 7f8b2a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c06ca20 0x7f8b2c1ee9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:10.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.290+0000 7f8b2a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c06ca20 0x7f8b2c1ee9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45666/0 (socket says 172.21.15.161:45666) 2024-03-28T23:30:10.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.290+0000 7f8b2a59c640 1 -- 172.21.15.161:0/741971561 learned_addr learned my addr 172.21.15.161:0/741971561 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:10.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.290+0000 7f8b23fff640 1 --2- 172.21.15.161:0/741971561 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b2c0a98c0 0x7f8b2c154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:10.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.290+0000 7f8b2ad9d640 1 --2- 172.21.15.161:0/741971561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b2c14eca0 0x7f8b2c1db300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:10.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.290+0000 7f8b2a59c640 1 -- 172.21.15.161:0/741971561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b2c14eca0 msgr2=0x7f8b2c1db300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:10.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.290+0000 7f8b2a59c640 1 --2- 172.21.15.161:0/741971561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b2c14eca0 0x7f8b2c1db300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:10.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.290+0000 7f8b2a59c640 1 -- 172.21.15.161:0/741971561 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b2c0a98c0 msgr2=0x7f8b2c154a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:10.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.290+0000 7f8b2a59c640 1 --2- 172.21.15.161:0/741971561 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b2c0a98c0 0x7f8b2c154a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:10.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.290+0000 7f8b2a59c640 1 -- 172.21.15.161:0/741971561 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b2c1551e0 con 0x7f8b2c06ca20 2024-03-28T23:30:10.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.290+0000 7f8b2ad9d640 1 --2- 172.21.15.161:0/741971561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b2c14eca0 0x7f8b2c1db300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:10.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.290+0000 7f8b2a59c640 1 --2- 172.21.15.161:0/741971561 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c06ca20 0x7f8b2c1ee9d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c00c3c0 tx=0x7f8b1c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:10.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.291+0000 7f8b22ffd640 1 -- 172.21.15.161:0/741971561 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c0052d0 con 0x7f8b2c06ca20 2024-03-28T23:30:10.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.291+0000 7f8b22ffd640 1 -- 172.21.15.161:0/741971561 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b1c005490 con 0x7f8b2c06ca20 2024-03-28T23:30:10.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.291+0000 7f8b30d0f640 1 -- 172.21.15.161:0/741971561 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b2c1554c0 con 0x7f8b2c06ca20 2024-03-28T23:30:10.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.291+0000 7f8b22ffd640 1 -- 172.21.15.161:0/741971561 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c007260 con 0x7f8b2c06ca20 2024-03-28T23:30:10.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.291+0000 7f8b30d0f640 1 -- 172.21.15.161:0/741971561 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b2c1dd5f0 con 0x7f8b2c06ca20 2024-03-28T23:30:10.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.292+0000 7f8b30d0f640 1 -- 172.21.15.161:0/741971561 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b2c04f4b0 con 0x7f8b2c06ca20 2024-03-28T23:30:10.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.292+0000 7f8b22ffd640 1 -- 172.21.15.161:0/741971561 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8b1c007400 con 0x7f8b2c06ca20 2024-03-28T23:30:10.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.293+0000 7f8b22ffd640 1 --2- 172.21.15.161:0/741971561 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8b0c06b500 0x7f8b0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:10.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.293+0000 7f8b23fff640 1 --2- 172.21.15.161:0/741971561 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8b0c06b500 0x7f8b0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:10.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.293+0000 7f8b22ffd640 1 -- 172.21.15.161:0/741971561 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f8b1c0898a0 con 0x7f8b2c06ca20 2024-03-28T23:30:10.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.293+0000 7f8b23fff640 1 --2- 172.21.15.161:0/741971561 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8b0c06b500 0x7f8b0c06d9c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8b14037000 tx=0x7f8b14031000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:10.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.297+0000 7f8b22ffd640 1 -- 172.21.15.161:0/741971561 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8b1c057e20 con 0x7f8b2c06ca20 2024-03-28T23:30:10.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.585+0000 7f8b30d0f640 1 -- 172.21.15.161:0/741971561 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f8b2c14f790 con 0x7f8b2c06ca20 2024-03-28T23:30:10.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.588+0000 7f8b22ffd640 1 -- 172.21.15.161:0/741971561 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+171790 (secure 0 0 0) 0x7f8b1c05bec0 con 0x7f8b2c06ca20 2024-03-28T23:30:10.591 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:10.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.590+0000 7f8b30d0f640 1 -- 172.21.15.161:0/741971561 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8b0c06b500 msgr2=0x7f8b0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:10.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.590+0000 7f8b30d0f640 1 --2- 172.21.15.161:0/741971561 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8b0c06b500 0x7f8b0c06d9c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8b14037000 tx=0x7f8b14031000 comp rx=0 tx=0).stop 2024-03-28T23:30:10.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.590+0000 7f8b30d0f640 1 -- 172.21.15.161:0/741971561 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c06ca20 msgr2=0x7f8b2c1ee9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:10.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.590+0000 7f8b30d0f640 1 --2- 172.21.15.161:0/741971561 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c06ca20 0x7f8b2c1ee9d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c00c3c0 tx=0x7f8b1c00c880 comp rx=0 tx=0).stop 2024-03-28T23:30:10.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.590+0000 7f8b30d0f640 1 -- 172.21.15.161:0/741971561 shutdown_connections 2024-03-28T23:30:10.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.590+0000 7f8b30d0f640 1 --2- 172.21.15.161:0/741971561 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8b0c06b500 0x7f8b0c06d9c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:10.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.590+0000 7f8b30d0f640 1 --2- 172.21.15.161:0/741971561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b2c14eca0 0x7f8b2c1db300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:10.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.590+0000 7f8b30d0f640 1 --2- 172.21.15.161:0/741971561 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b2c0a98c0 0x7f8b2c154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:10.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.590+0000 7f8b30d0f640 1 --2- 172.21.15.161:0/741971561 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c06ca20 0x7f8b2c1ee9d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:10.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.591+0000 7f8b30d0f640 1 -- 172.21.15.161:0/741971561 >> 172.21.15.161:0/741971561 conn(0x7f8b2c0c6670 msgr2=0x7f8b2c14dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:10.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.591+0000 7f8b30d0f640 1 -- 172.21.15.161:0/741971561 shutdown_connections 2024-03-28T23:30:10.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.591+0000 7f8b30d0f640 1 -- 172.21.15.161:0/741971561 wait complete. 2024-03-28T23:30:10.603 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":5,"flags":0,"active_gid":4107,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":1776925385},{"type":"v1","addr":"172.21.15.161:6833","nonce":1776925385}]},"active_addr":"172.21.15.161:6833/1776925385","active_change":"2024-03-28T23:29:33.303256+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4105,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.161:0","nonce":3700433692}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.161:0","nonce":1736135566}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.161:0","nonce":832273606}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.161:0","nonce":554803565}]}]} 2024-03-28T23:30:10.607 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-28T23:30:10.607 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-28T23:30:10.607 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:30:10.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.716+0000 7f04a413e640 1 Processor -- start 2024-03-28T23:30:10.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.716+0000 7f04a413e640 1 -- start start 2024-03-28T23:30:10.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.717+0000 7f04a413e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f049c06ca20 0x7f049c06ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:10.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.717+0000 7f04a413e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f049c0a98c0 0x7f049c0a9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:10.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.717+0000 7f04a413e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f049c151850 con 0x7f049c14eca0 2024-03-28T23:30:10.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.717+0000 7f04a413e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f049c152460 con 0x7f049c0a98c0 2024-03-28T23:30:10.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.717+0000 7f04a413e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f049c151b30 con 0x7f049c06ca20 2024-03-28T23:30:10.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.717+0000 7f04a26db640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f049c0a98c0 0x7f049c0a9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:10.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.718+0000 7f04a1eda640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f049c14eca0 0x7f049c0c1880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:46034/0 (socket says 172.21.15.161:46034) 2024-03-28T23:30:10.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.718+0000 7f04a1eda640 1 -- 172.21.15.161:0/315389627 learned_addr learned my addr 172.21.15.161:0/315389627 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:10.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.718+0000 7f04a16d9640 1 --2- 172.21.15.161:0/315389627 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f049c06ca20 0x7f049c06ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:10.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.718+0000 7f04a26db640 1 -- 172.21.15.161:0/315389627 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f049c06ca20 msgr2=0x7f049c06ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:10.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.718+0000 7f04a26db640 1 --2- 172.21.15.161:0/315389627 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f049c06ca20 0x7f049c06ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:10.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.718+0000 7f04a26db640 1 -- 172.21.15.161:0/315389627 >> v1:172.21.15.170:6789/0 conn(0x7f049c14eca0 legacy=0x7f049c0c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:10.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.718+0000 7f04a26db640 1 -- 172.21.15.161:0/315389627 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f049c151e10 con 0x7f049c0a98c0 2024-03-28T23:30:10.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.718+0000 7f04a0ed8640 1 -- 172.21.15.161:0/315389627 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3306407802 0 0) 0x7f049c151850 con 0x7f049c14eca0 2024-03-28T23:30:10.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.719+0000 7f04a26db640 1 --2- 172.21.15.161:0/315389627 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f049c0a98c0 0x7f049c0a9d20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f049400bfa0 tx=0x7f049402e010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cea792d37ca6380 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:10.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.719+0000 7f04a0ed8640 1 -- 172.21.15.161:0/315389627 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049402e9b0 con 0x7f049c0a98c0 2024-03-28T23:30:10.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.719+0000 7f04a0ed8640 1 -- 172.21.15.161:0/315389627 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f049402eb50 con 0x7f049c0a98c0 2024-03-28T23:30:10.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.719+0000 7f04a0ed8640 1 -- 172.21.15.161:0/315389627 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0494009540 con 0x7f049c0a98c0 2024-03-28T23:30:10.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.720+0000 7f04a413e640 1 -- 172.21.15.161:0/315389627 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f049c0a98c0 msgr2=0x7f049c0a9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:10.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.720+0000 7f04a413e640 1 --2- 172.21.15.161:0/315389627 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f049c0a98c0 0x7f049c0a9d20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f049400bfa0 tx=0x7f049402e010 comp rx=0 tx=0).stop 2024-03-28T23:30:10.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.720+0000 7f04a413e640 1 -- 172.21.15.161:0/315389627 shutdown_connections 2024-03-28T23:30:10.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.720+0000 7f04a413e640 1 --2- 172.21.15.161:0/315389627 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f049c0a98c0 0x7f049c0a9d20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:10.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.720+0000 7f04a413e640 1 --2- 172.21.15.161:0/315389627 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f049c06ca20 0x7f049c06ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:10.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.720+0000 7f04a413e640 1 -- 172.21.15.161:0/315389627 >> 172.21.15.161:0/315389627 conn(0x7f049c0c6670 msgr2=0x7f049c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:10.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.720+0000 7f04a413e640 1 -- 172.21.15.161:0/315389627 shutdown_connections 2024-03-28T23:30:10.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.720+0000 7f04a413e640 1 -- 172.21.15.161:0/315389627 wait complete. 2024-03-28T23:30:10.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.721+0000 7f04a413e640 1 Processor -- start 2024-03-28T23:30:10.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.721+0000 7f04a413e640 1 -- start start 2024-03-28T23:30:10.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.722+0000 7f04a413e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f049c06ca20 0x7f049c072d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:10.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.722+0000 7f04a413e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f049c0a98c0 0x7f049c06b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:10.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.722+0000 7f04a413e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f049c14eca0 0x7f049c1e9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:10.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.722+0000 7f04a413e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f049c1514b0 con 0x7f049c0a98c0 2024-03-28T23:30:10.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.722+0000 7f04a413e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f049c14f080 con 0x7f049c14eca0 2024-03-28T23:30:10.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.722+0000 7f04a413e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f049c151630 con 0x7f049c06ca20 2024-03-28T23:30:10.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.722+0000 7f04a16d9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f049c0a98c0 0x7f049c06b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:10.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.722+0000 7f04a1eda640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f049c06ca20 0x7f049c072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:10.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.722+0000 7f04a16d9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f049c0a98c0 0x7f049c06b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54024/0 (socket says 172.21.15.161:54024) 2024-03-28T23:30:10.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.722+0000 7f04a16d9640 1 -- 172.21.15.161:0/365615820 learned_addr learned my addr 172.21.15.161:0/365615820 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:10.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.722+0000 7f04a26db640 1 --2- 172.21.15.161:0/365615820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f049c14eca0 0x7f049c1e9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:10.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.722+0000 7f04a1eda640 1 -- 172.21.15.161:0/365615820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f049c14eca0 msgr2=0x7f049c1e9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:10.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.722+0000 7f04a1eda640 1 --2- 172.21.15.161:0/365615820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f049c14eca0 0x7f049c1e9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:10.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.722+0000 7f04a1eda640 1 -- 172.21.15.161:0/365615820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f049c0a98c0 msgr2=0x7f049c06b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:10.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.722+0000 7f04a1eda640 1 --2- 172.21.15.161:0/365615820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f049c0a98c0 0x7f049c06b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:10.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.722+0000 7f04a1eda640 1 -- 172.21.15.161:0/365615820 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f049c073430 con 0x7f049c06ca20 2024-03-28T23:30:10.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.722+0000 7f04a16d9640 1 --2- 172.21.15.161:0/365615820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f049c0a98c0 0x7f049c06b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:10.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.723+0000 7f04a1eda640 1 --2- 172.21.15.161:0/365615820 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f049c06ca20 0x7f049c072d70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f049800c410 tx=0x7f049800c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:10.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.723+0000 7f04927fc640 1 -- 172.21.15.161:0/365615820 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04980074e0 con 0x7f049c06ca20 2024-03-28T23:30:10.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.723+0000 7f04927fc640 1 -- 172.21.15.161:0/365615820 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04980047e0 con 0x7f049c06ca20 2024-03-28T23:30:10.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.723+0000 7f04a413e640 1 -- 172.21.15.161:0/365615820 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f049c1e6410 con 0x7f049c06ca20 2024-03-28T23:30:10.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.723+0000 7f04927fc640 1 -- 172.21.15.161:0/365615820 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0498004bd0 con 0x7f049c06ca20 2024-03-28T23:30:10.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.723+0000 7f04a413e640 1 -- 172.21.15.161:0/365615820 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f049c1e69d0 con 0x7f049c06ca20 2024-03-28T23:30:10.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.724+0000 7f04927fc640 1 -- 172.21.15.161:0/365615820 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0498011070 con 0x7f049c06ca20 2024-03-28T23:30:10.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.724+0000 7f04a413e640 1 -- 172.21.15.161:0/365615820 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f049c1e6bd0 con 0x7f049c06ca20 2024-03-28T23:30:10.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.725+0000 7f04927fc640 1 --2- 172.21.15.161:0/365615820 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f048006b500 0x7f048006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:10.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.725+0000 7f04a16d9640 1 --2- 172.21.15.161:0/365615820 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f048006b500 0x7f048006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:10.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.725+0000 7f04927fc640 1 -- 172.21.15.161:0/365615820 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0498089ec0 con 0x7f049c06ca20 2024-03-28T23:30:10.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.725+0000 7f04a16d9640 1 --2- 172.21.15.161:0/365615820 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f048006b500 0x7f048006d9c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f049c072d00 tx=0x7f048c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:10.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:10.730+0000 7f04927fc640 1 -- 172.21.15.161:0/365615820 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f049c1e6bd0 con 0x7f049c06ca20 2024-03-28T23:30:11.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.049+0000 7f04a413e640 1 -- 172.21.15.161:0/365615820 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f049c04f4b0 con 0x7f049c06ca20 2024-03-28T23:30:11.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.051+0000 7f04927fc640 1 -- 172.21.15.161:0/365615820 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7f04980584e0 con 0x7f049c06ca20 2024-03-28T23:30:11.053 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:11.053 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":24,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:29:50.946843+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:30:11.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.053+0000 7f04a413e640 1 -- 172.21.15.161:0/365615820 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f048006b500 msgr2=0x7f048006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.053+0000 7f04a413e640 1 --2- 172.21.15.161:0/365615820 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f048006b500 0x7f048006d9c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f049c072d00 tx=0x7f048c009190 comp rx=0 tx=0).stop 2024-03-28T23:30:11.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.053+0000 7f04a413e640 1 -- 172.21.15.161:0/365615820 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f049c06ca20 msgr2=0x7f049c072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.053+0000 7f04a413e640 1 --2- 172.21.15.161:0/365615820 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f049c06ca20 0x7f049c072d70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f049800c410 tx=0x7f049800c8d0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.053+0000 7f04a413e640 1 -- 172.21.15.161:0/365615820 shutdown_connections 2024-03-28T23:30:11.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.053+0000 7f04a413e640 1 --2- 172.21.15.161:0/365615820 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f048006b500 0x7f048006d9c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.053+0000 7f04a413e640 1 --2- 172.21.15.161:0/365615820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f049c0a98c0 0x7f049c06b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.053+0000 7f04a413e640 1 --2- 172.21.15.161:0/365615820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f049c14eca0 0x7f049c1e9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.053+0000 7f04a413e640 1 --2- 172.21.15.161:0/365615820 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f049c06ca20 0x7f049c072d70 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.053+0000 7f04a413e640 1 -- 172.21.15.161:0/365615820 >> 172.21.15.161:0/365615820 conn(0x7f049c0c6670 msgr2=0x7f049c0aa8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.053+0000 7f04a413e640 1 -- 172.21.15.161:0/365615820 shutdown_connections 2024-03-28T23:30:11.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.053+0000 7f04a413e640 1 -- 172.21.15.161:0/365615820 wait complete. 2024-03-28T23:30:11.066 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-28T23:30:11.067 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:30:11.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.168+0000 7f2fd4eeb640 1 Processor -- start 2024-03-28T23:30:11.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.168+0000 7f2fd4eeb640 1 -- start start 2024-03-28T23:30:11.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.169+0000 7f2fd4eeb640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2fd00c5410 0x7f2fd00c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.169+0000 7f2fd4eeb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fd0158a30 con 0x7f2fd0152890 2024-03-28T23:30:11.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.169+0000 7f2fd4eeb640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fd0158d10 con 0x7f2fd01553a0 2024-03-28T23:30:11.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.169+0000 7f2fd4eeb640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f2fd00c8480 con 0x7f2fd00c5410 2024-03-28T23:30:11.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.169+0000 7f2fce59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f2fd0152890 0x7f2fd00c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59550/0 (socket says 172.21.15.161:59550) 2024-03-28T23:30:11.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.169+0000 7f2fce59c640 1 -- 172.21.15.161:0/428447663 learned_addr learned my addr 172.21.15.161:0/428447663 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:11.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.169+0000 7f2fc5d9b640 1 --2- 172.21.15.161:0/428447663 >> v2:172.21.15.170:3300/0 conn(0x7f2fd00c5410 0x7f2fd00c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.169+0000 7f2fcdd9b640 1 -- 172.21.15.161:0/428447663 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2261178883 0 0) 0x7f2fd0158a30 con 0x7f2fd0152890 2024-03-28T23:30:11.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.169+0000 7f2fcdd9b640 1 -- 172.21.15.161:0/428447663 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2fbc003520 con 0x7f2fd0152890 2024-03-28T23:30:11.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.170+0000 7f2fcdd9b640 1 -- 172.21.15.161:0/428447663 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2585483224 0 0) 0x7f2fbc003520 con 0x7f2fd0152890 2024-03-28T23:30:11.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.170+0000 7f2fcdd9b640 1 -- 172.21.15.161:0/428447663 >> v2:172.21.15.170:3300/0 conn(0x7f2fd00c5410 msgr2=0x7f2fd00c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.170+0000 7f2fcdd9b640 1 --2- 172.21.15.161:0/428447663 >> v2:172.21.15.170:3300/0 conn(0x7f2fd00c5410 0x7f2fd00c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.170+0000 7f2fcdd9b640 1 -- 172.21.15.161:0/428447663 >> v1:172.21.15.170:6789/0 conn(0x7f2fd01553a0 legacy=0x7f2fd0155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.170+0000 7f2fcdd9b640 1 -- 172.21.15.161:0/428447663 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fd0158ff0 con 0x7f2fd0152890 2024-03-28T23:30:11.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.170+0000 7f2fcdd9b640 1 -- 172.21.15.161:0/428447663 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f2fc0001670 con 0x7f2fd0152890 2024-03-28T23:30:11.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.170+0000 7f2fcdd9b640 1 -- 172.21.15.161:0/428447663 >> v1:172.21.15.161:6789/0 conn(0x7f2fd0152890 legacy=0x7f2fd00c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.170+0000 7f2fcdd9b640 1 --2- 172.21.15.161:0/428447663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fbc003a90 0x7f2fbc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.170+0000 7f2fcdd9b640 1 --2- 172.21.15.161:0/428447663 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fbc004800 0x7f2fbc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.170+0000 7f2fcdd9b640 1 --2- 172.21.15.161:0/428447663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fbc005210 0x7f2fbc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.170+0000 7f2fcdd9b640 1 -- 172.21.15.161:0/428447663 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2fbc00c130 con 0x7f2fbc004800 2024-03-28T23:30:11.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.170+0000 7f2fcdd9b640 1 -- 172.21.15.161:0/428447663 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2fbc00c2e0 con 0x7f2fbc005210 2024-03-28T23:30:11.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.170+0000 7f2fcdd9b640 1 -- 172.21.15.161:0/428447663 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2fbc00c4e0 con 0x7f2fbc003a90 2024-03-28T23:30:11.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.170+0000 7f2fc5d9b640 1 --2- 172.21.15.161:0/428447663 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fbc004800 0x7f2fbc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.170+0000 7f2fce59c640 1 --2- 172.21.15.161:0/428447663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fbc003a90 0x7f2fbc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.170+0000 7f2fced9d640 1 --2- 172.21.15.161:0/428447663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fbc005210 0x7f2fbc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.170+0000 7f2fc5d9b640 1 -- 172.21.15.161:0/428447663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fbc003a90 msgr2=0x7f2fbc003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.170+0000 7f2fc5d9b640 1 --2- 172.21.15.161:0/428447663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fbc003a90 0x7f2fbc003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.170+0000 7f2fc5d9b640 1 -- 172.21.15.161:0/428447663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fbc005210 msgr2=0x7f2fbc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.170+0000 7f2fc5d9b640 1 --2- 172.21.15.161:0/428447663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fbc005210 0x7f2fbc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.170+0000 7f2fc5d9b640 1 -- 172.21.15.161:0/428447663 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fbc003520 con 0x7f2fbc004800 2024-03-28T23:30:11.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.170+0000 7f2fced9d640 1 -- 172.21.15.161:0/428447663 reap_dead start 2024-03-28T23:30:11.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.171+0000 7f2fc5d9b640 1 --2- 172.21.15.161:0/428447663 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fbc004800 0x7f2fbc004c80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00ad1f0 tx=0x7f2fbc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c2666bde9ad9d82 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.171+0000 7f2fcdd9b640 1 -- 172.21.15.161:0/428447663 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8009680 con 0x7f2fbc004800 2024-03-28T23:30:11.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.171+0000 7f2fcdd9b640 1 -- 172.21.15.161:0/428447663 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fb8005990 con 0x7f2fbc004800 2024-03-28T23:30:11.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.171+0000 7f2fcdd9b640 1 -- 172.21.15.161:0/428447663 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8005d80 con 0x7f2fbc004800 2024-03-28T23:30:11.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.172+0000 7f2fd4eeb640 1 -- 172.21.15.161:0/428447663 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fbc004800 msgr2=0x7f2fbc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.172+0000 7f2fd4eeb640 1 --2- 172.21.15.161:0/428447663 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fbc004800 0x7f2fbc004c80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00ad1f0 tx=0x7f2fbc005830 comp rx=0 tx=0).stop 2024-03-28T23:30:11.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.172+0000 7f2fd4eeb640 1 -- 172.21.15.161:0/428447663 shutdown_connections 2024-03-28T23:30:11.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.172+0000 7f2fd4eeb640 1 --2- 172.21.15.161:0/428447663 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fbc004800 0x7f2fbc004c80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.172+0000 7f2fd4eeb640 1 -- 172.21.15.161:0/428447663 >> 172.21.15.161:0/428447663 conn(0x7f2fd00ac620 msgr2=0x7f2fd00aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.172+0000 7f2fd4eeb640 1 -- 172.21.15.161:0/428447663 shutdown_connections 2024-03-28T23:30:11.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.172+0000 7f2fd4eeb640 1 -- 172.21.15.161:0/428447663 wait complete. 2024-03-28T23:30:11.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.174+0000 7f2fd4eeb640 1 Processor -- start 2024-03-28T23:30:11.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.174+0000 7f2fd4eeb640 1 -- start start 2024-03-28T23:30:11.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.174+0000 7f2fd4eeb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fbc004800 0x7f2fd00c0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.174+0000 7f2fd4eeb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fd00be5c0 0x7f2fd0150940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.174+0000 7f2fd4eeb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fd01ed820 0x7f2fd01edca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.174+0000 7f2fd4eeb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd0152c70 con 0x7f2fd01ed820 2024-03-28T23:30:11.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.174+0000 7f2fd4eeb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd0159640 con 0x7f2fd00be5c0 2024-03-28T23:30:11.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.174+0000 7f2fd4eeb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2fd0152df0 con 0x7f2fbc004800 2024-03-28T23:30:11.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.174+0000 7f2fced9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fd01ed820 0x7f2fd01edca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.174+0000 7f2fced9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fd01ed820 0x7f2fd01edca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54036/0 (socket says 172.21.15.161:54036) 2024-03-28T23:30:11.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.174+0000 7f2fced9d640 1 -- 172.21.15.161:0/3531270834 learned_addr learned my addr 172.21.15.161:0/3531270834 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:11.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.175+0000 7f2fce59c640 1 --2- 172.21.15.161:0/3531270834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fbc004800 0x7f2fd00c0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.175+0000 7f2fc5d9b640 1 --2- 172.21.15.161:0/3531270834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fd00be5c0 0x7f2fd0150940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.175+0000 7f2fced9d640 1 -- 172.21.15.161:0/3531270834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fbc004800 msgr2=0x7f2fd00c0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.175+0000 7f2fced9d640 1 --2- 172.21.15.161:0/3531270834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fbc004800 0x7f2fd00c0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.175+0000 7f2fced9d640 1 -- 172.21.15.161:0/3531270834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fd00be5c0 msgr2=0x7f2fd0150940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.175+0000 7f2fced9d640 1 --2- 172.21.15.161:0/3531270834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fd00be5c0 0x7f2fd0150940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.175+0000 7f2fced9d640 1 -- 172.21.15.161:0/3531270834 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fd01eb160 con 0x7f2fd01ed820 2024-03-28T23:30:11.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.175+0000 7f2fced9d640 1 --2- 172.21.15.161:0/3531270834 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fd01ed820 0x7f2fd01edca0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00ca970 tx=0x7f2fc8007b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.175+0000 7f2fc77fe640 1 -- 172.21.15.161:0/3531270834 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc8003c40 con 0x7f2fd01ed820 2024-03-28T23:30:11.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.175+0000 7f2fc77fe640 1 -- 172.21.15.161:0/3531270834 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fc8018070 con 0x7f2fd01ed820 2024-03-28T23:30:11.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.175+0000 7f2fd4eeb640 1 -- 172.21.15.161:0/3531270834 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fd01ea3a0 con 0x7f2fd01ed820 2024-03-28T23:30:11.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.175+0000 7f2fc77fe640 1 -- 172.21.15.161:0/3531270834 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc80050d0 con 0x7f2fd01ed820 2024-03-28T23:30:11.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.175+0000 7f2fd4eeb640 1 -- 172.21.15.161:0/3531270834 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fd01ea960 con 0x7f2fd01ed820 2024-03-28T23:30:11.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.176+0000 7f2fd4eeb640 1 -- 172.21.15.161:0/3531270834 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fd004f4b0 con 0x7f2fd01ed820 2024-03-28T23:30:11.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.177+0000 7f2fc77fe640 1 -- 172.21.15.161:0/3531270834 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2fc8005270 con 0x7f2fd01ed820 2024-03-28T23:30:11.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.177+0000 7f2fc77fe640 1 --2- 172.21.15.161:0/3531270834 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2fb006b500 0x7f2fb006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.177+0000 7f2fc77fe640 1 -- 172.21.15.161:0/3531270834 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2fc808a380 con 0x7f2fd01ed820 2024-03-28T23:30:11.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.177+0000 7f2fce59c640 1 --2- 172.21.15.161:0/3531270834 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2fb006b500 0x7f2fb006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.178+0000 7f2fce59c640 1 --2- 172.21.15.161:0/3531270834 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2fb006b500 0x7f2fb006d9c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0151170 tx=0x7f2fc0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.182+0000 7f2fc77fe640 1 -- 172.21.15.161:0/3531270834 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2fc801a310 con 0x7f2fd01ed820 2024-03-28T23:30:11.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.362+0000 7f2fd4eeb640 1 -- 172.21.15.161:0/3531270834 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2fd0153c60 con 0x7f2fd01ed820 2024-03-28T23:30:11.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.363+0000 7f2fc77fe640 1 -- 172.21.15.161:0/3531270834 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7f2fc8058f50 con 0x7f2fd01ed820 2024-03-28T23:30:11.365 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:11.365 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":24,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:29:50.946843+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:30:11.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.365+0000 7f2fd4eeb640 1 -- 172.21.15.161:0/3531270834 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2fb006b500 msgr2=0x7f2fb006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.365+0000 7f2fd4eeb640 1 --2- 172.21.15.161:0/3531270834 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2fb006b500 0x7f2fb006d9c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0151170 tx=0x7f2fc0009040 comp rx=0 tx=0).stop 2024-03-28T23:30:11.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.365+0000 7f2fd4eeb640 1 -- 172.21.15.161:0/3531270834 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fd01ed820 msgr2=0x7f2fd01edca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.365+0000 7f2fd4eeb640 1 --2- 172.21.15.161:0/3531270834 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fd01ed820 0x7f2fd01edca0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00ca970 tx=0x7f2fc8007b80 comp rx=0 tx=0).stop 2024-03-28T23:30:11.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.365+0000 7f2fd4eeb640 1 -- 172.21.15.161:0/3531270834 shutdown_connections 2024-03-28T23:30:11.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.365+0000 7f2fd4eeb640 1 --2- 172.21.15.161:0/3531270834 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2fb006b500 0x7f2fb006d9c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.365+0000 7f2fd4eeb640 1 --2- 172.21.15.161:0/3531270834 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fd01ed820 0x7f2fd01edca0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.365+0000 7f2fd4eeb640 1 --2- 172.21.15.161:0/3531270834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fd00be5c0 0x7f2fd0150940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.365+0000 7f2fd4eeb640 1 --2- 172.21.15.161:0/3531270834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fbc004800 0x7f2fd00c0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.365+0000 7f2fd4eeb640 1 -- 172.21.15.161:0/3531270834 >> 172.21.15.161:0/3531270834 conn(0x7f2fd00ac620 msgr2=0x7f2fd014c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.365+0000 7f2fd4eeb640 1 -- 172.21.15.161:0/3531270834 shutdown_connections 2024-03-28T23:30:11.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.365+0000 7f2fd4eeb640 1 -- 172.21.15.161:0/3531270834 wait complete. 2024-03-28T23:30:11.378 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-28T23:30:11.378 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-28T23:30:11.379 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-28T23:30:11.379 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-28T23:30:11.379 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-28T23:30:11.380 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-28T23:30:11.380 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-28T23:30:11.381 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-28T23:30:11.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.554+0000 7fdb27f01640 1 Processor -- start 2024-03-28T23:30:11.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.555+0000 7fdb27f01640 1 -- start start 2024-03-28T23:30:11.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.556+0000 7fdb27f01640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fdb20072e10 0x7fdb2006b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.556+0000 7fdb27f01640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb200c1eb0 con 0x7fdb20151fd0 2024-03-28T23:30:11.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.556+0000 7fdb27f01640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb200c2190 con 0x7fdb2006b690 2024-03-28T23:30:11.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.556+0000 7fdb27f01640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fdb200c2470 con 0x7fdb20072e10 2024-03-28T23:30:11.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.556+0000 7fdb25c9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fdb20151fd0 0x7fdb200aa4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59566/0 (socket says 172.21.15.161:59566) 2024-03-28T23:30:11.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.556+0000 7fdb25c9d640 1 -- 172.21.15.161:0/925030568 learned_addr learned my addr 172.21.15.161:0/925030568 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:11.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.557+0000 7fdb24c9b640 1 -- 172.21.15.161:0/925030568 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3532618175 0 0) 0x7fdb200c1eb0 con 0x7fdb20151fd0 2024-03-28T23:30:11.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.557+0000 7fdb24c9b640 1 -- 172.21.15.161:0/925030568 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb0c003520 con 0x7fdb20151fd0 2024-03-28T23:30:11.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.557+0000 7fdb24c9b640 1 -- 172.21.15.161:0/925030568 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4284963140 0 0) 0x7fdb0c003520 con 0x7fdb20151fd0 2024-03-28T23:30:11.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.557+0000 7fdb24c9b640 1 -- 172.21.15.161:0/925030568 >> v2:172.21.15.170:3300/0 conn(0x7fdb20072e10 msgr2=0x7fdb2006b150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T23:30:11.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.557+0000 7fdb24c9b640 1 --2- 172.21.15.161:0/925030568 >> v2:172.21.15.170:3300/0 conn(0x7fdb20072e10 0x7fdb2006b150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.557+0000 7fdb24c9b640 1 -- 172.21.15.161:0/925030568 >> v1:172.21.15.170:6789/0 conn(0x7fdb2006b690 legacy=0x7fdb2015bbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T23:30:11.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.557+0000 7fdb24c9b640 1 -- 172.21.15.161:0/925030568 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb200c2750 con 0x7fdb20151fd0 2024-03-28T23:30:11.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.557+0000 7fdb24c9b640 1 -- 172.21.15.161:0/925030568 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fdb10001670 con 0x7fdb20151fd0 2024-03-28T23:30:11.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.558+0000 7fdb24c9b640 1 -- 172.21.15.161:0/925030568 >> v1:172.21.15.161:6789/0 conn(0x7fdb20151fd0 legacy=0x7fdb200aa4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.558+0000 7fdb24c9b640 1 --2- 172.21.15.161:0/925030568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb0c003ae0 0x7fdb0c003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.558+0000 7fdb24c9b640 1 --2- 172.21.15.161:0/925030568 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdb0c004800 0x7fdb0c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.558+0000 7fdb24c9b640 1 --2- 172.21.15.161:0/925030568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb0c0099b0 0x7fdb0c00bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.558+0000 7fdb24c9b640 1 -- 172.21.15.161:0/925030568 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdb200c2470 con 0x7fdb0c0099b0 2024-03-28T23:30:11.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.558+0000 7fdb25c9d640 1 --2- 172.21.15.161:0/925030568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb0c003ae0 0x7fdb0c003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.558+0000 7fdb24c9b640 1 -- 172.21.15.161:0/925030568 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdb0c00c3c0 con 0x7fdb0c003ae0 2024-03-28T23:30:11.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.558+0000 7fdb24c9b640 1 -- 172.21.15.161:0/925030568 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdb0c00c5c0 con 0x7fdb0c004800 2024-03-28T23:30:11.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.559+0000 7fdb2649e640 1 --2- 172.21.15.161:0/925030568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb0c0099b0 0x7fdb0c00bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.559+0000 7fdb25c9d640 1 -- 172.21.15.161:0/925030568 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdb0c004800 msgr2=0x7fdb0c004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T23:30:11.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.559+0000 7fdb25c9d640 1 --2- 172.21.15.161:0/925030568 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdb0c004800 0x7fdb0c004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.559+0000 7fdb25c9d640 1 -- 172.21.15.161:0/925030568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb0c0099b0 msgr2=0x7fdb0c00bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.559+0000 7fdb25c9d640 1 --2- 172.21.15.161:0/925030568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb0c0099b0 0x7fdb0c00bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.559+0000 7fdb25c9d640 1 -- 172.21.15.161:0/925030568 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb0c003520 con 0x7fdb0c003ae0 2024-03-28T23:30:11.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.560+0000 7fdb2649e640 1 --2- 172.21.15.161:0/925030568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb0c0099b0 0x7fdb0c00bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:11.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.560+0000 7fdb2649e640 1 -- 172.21.15.161:0/925030568 reap_dead start 2024-03-28T23:30:11.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.561+0000 7fdb25c9d640 1 --2- 172.21.15.161:0/925030568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb0c003ae0 0x7fdb0c003ee0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdb1000c190 tx=0x7fdb0c005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d96450ab2c4a838e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.562+0000 7fe58c48f640 1 Processor -- start 2024-03-28T23:30:11.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.563+0000 7fe58c48f640 1 -- start start 2024-03-28T23:30:11.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.563+0000 7fdb24c9b640 1 -- 172.21.15.161:0/925030568 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb10007c60 con 0x7fdb0c003ae0 2024-03-28T23:30:11.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.564+0000 7fdb24c9b640 1 -- 172.21.15.161:0/925030568 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb10030070 con 0x7fdb0c003ae0 2024-03-28T23:30:11.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.564+0000 7fdb24c9b640 1 -- 172.21.15.161:0/925030568 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb10005d70 con 0x7fdb0c003ae0 2024-03-28T23:30:11.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.564+0000 7fe58c48f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe584072e10 0x7fe58406b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.564+0000 7fe58c48f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe58406b690 0x7fe58415bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.564+0000 7fe58c48f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5840c1eb0 con 0x7fe584151fd0 2024-03-28T23:30:11.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.564+0000 7fe58c48f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fe5840c2190 con 0x7fe584072e10 2024-03-28T23:30:11.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.565+0000 7fe58c48f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe5840c2470 con 0x7fe58406b690 2024-03-28T23:30:11.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.565+0000 7fe58aa2c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe58406b690 0x7fe58415bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.565+0000 7fe58aa2c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe58406b690 0x7fe58415bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:36720/0 (socket says 172.21.15.161:36720) 2024-03-28T23:30:11.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.565+0000 7fe58aa2c640 1 -- 172.21.15.161:0/3239806650 learned_addr learned my addr 172.21.15.161:0/3239806650 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:11.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.566+0000 7fdb27f01640 1 -- 172.21.15.161:0/925030568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb0c003ae0 msgr2=0x7fdb0c003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.566+0000 7fdb27f01640 1 --2- 172.21.15.161:0/925030568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb0c003ae0 0x7fdb0c003ee0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdb1000c190 tx=0x7fdb0c005830 comp rx=0 tx=0).stop 2024-03-28T23:30:11.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.566+0000 7fe589a2a640 1 --2- 172.21.15.161:0/3239806650 >> v2:172.21.15.170:3300/0 conn(0x7fe584072e10 0x7fe58406b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.567+0000 7fdb27f01640 1 -- 172.21.15.161:0/925030568 shutdown_connections 2024-03-28T23:30:11.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.567+0000 7fdb27f01640 1 --2- 172.21.15.161:0/925030568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb0c003ae0 0x7fdb0c003ee0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.567+0000 7fdb27f01640 1 -- 172.21.15.161:0/925030568 >> 172.21.15.161:0/925030568 conn(0x7fdb2006d8e0 msgr2=0x7fdb2006dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.567+0000 7fe589a2a640 1 -- 172.21.15.161:0/3239806650 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe58406b690 msgr2=0x7fe58415bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.567+0000 7fe589a2a640 1 --2- 172.21.15.161:0/3239806650 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe58406b690 0x7fe58415bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.567+0000 7fe589a2a640 1 -- 172.21.15.161:0/3239806650 >> v1:172.21.15.170:6789/0 conn(0x7fe584151fd0 legacy=0x7fe5840aa4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.567+0000 7fe589a2a640 1 -- 172.21.15.161:0/3239806650 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5840c2750 con 0x7fe584072e10 2024-03-28T23:30:11.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.567+0000 7fdb27f01640 1 -- 172.21.15.161:0/925030568 shutdown_connections 2024-03-28T23:30:11.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.567+0000 7fdb27f01640 1 -- 172.21.15.161:0/925030568 wait complete. 2024-03-28T23:30:11.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.569+0000 7fdb27f01640 1 Processor -- start 2024-03-28T23:30:11.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.569+0000 7fdb27f01640 1 -- start start 2024-03-28T23:30:11.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.570+0000 7fdb27f01640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb0c003ae0 0x7fdb20156310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.570+0000 7fdb27f01640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdb20158a30 0x7fdb201f2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.570+0000 7fdb27f01640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb20070b50 0x7fdb201f3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.570+0000 7fdb27f01640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdb200c1ad0 con 0x7fdb0c003ae0 2024-03-28T23:30:11.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.570+0000 7fe589a2a640 1 --2- 172.21.15.161:0/3239806650 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe584072e10 0x7fe58406b150 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe5800097a0 tx=0x7fe58002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf7be2bbbed88ff3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.570+0000 7fdb27f01640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdb200aad70 con 0x7fdb20070b50 2024-03-28T23:30:11.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.570+0000 7fdb27f01640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdb200c1c50 con 0x7fdb20158a30 2024-03-28T23:30:11.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.570+0000 7fdb2649e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb20070b50 0x7fdb201f3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.570+0000 7fdb2649e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb20070b50 0x7fdb201f3500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:39042/0 (socket says 172.21.15.161:39042) 2024-03-28T23:30:11.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.570+0000 7fdb2649e640 1 -- 172.21.15.161:0/1183318047 learned_addr learned my addr 172.21.15.161:0/1183318047 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:11.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.571+0000 7fdb25c9d640 1 --2- 172.21.15.161:0/1183318047 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb0c003ae0 0x7fdb20156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.571+0000 7fdb2549c640 1 --2- 172.21.15.161:0/1183318047 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdb20158a30 0x7fdb201f2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.572+0000 7fdb25c9d640 1 -- 172.21.15.161:0/1183318047 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdb20158a30 msgr2=0x7fdb201f2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.572+0000 7fdb25c9d640 1 --2- 172.21.15.161:0/1183318047 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdb20158a30 0x7fdb201f2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.572+0000 7fdb25c9d640 1 -- 172.21.15.161:0/1183318047 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb20070b50 msgr2=0x7fdb201f3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.572+0000 7fdb25c9d640 1 --2- 172.21.15.161:0/1183318047 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb20070b50 0x7fdb201f3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.572+0000 7fe589229640 1 -- 172.21.15.161:0/3239806650 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5800315a0 con 0x7fe584072e10 2024-03-28T23:30:11.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.573+0000 7f154010d640 1 Processor -- start 2024-03-28T23:30:11.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.573+0000 7fe589229640 1 -- 172.21.15.161:0/3239806650 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe580037070 con 0x7fe584072e10 2024-03-28T23:30:11.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.573+0000 7fe589229640 1 -- 172.21.15.161:0/3239806650 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe580031960 con 0x7fe584072e10 2024-03-28T23:30:11.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.574+0000 7fe58c48f640 1 -- 172.21.15.161:0/3239806650 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe584072e10 msgr2=0x7fe58406b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.574+0000 7fe58c48f640 1 --2- 172.21.15.161:0/3239806650 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe584072e10 0x7fe58406b150 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe5800097a0 tx=0x7fe58002b840 comp rx=0 tx=0).stop 2024-03-28T23:30:11.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.574+0000 7fe58c48f640 1 -- 172.21.15.161:0/3239806650 shutdown_connections 2024-03-28T23:30:11.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.574+0000 7fe58c48f640 1 --2- 172.21.15.161:0/3239806650 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe58406b690 0x7fe58415bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.574+0000 7fe58c48f640 1 --2- 172.21.15.161:0/3239806650 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe584072e10 0x7fe58406b150 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.574+0000 7fe58c48f640 1 -- 172.21.15.161:0/3239806650 >> 172.21.15.161:0/3239806650 conn(0x7fe58406d8e0 msgr2=0x7fe58406dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.574+0000 7fdb25c9d640 1 -- 172.21.15.161:0/1183318047 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb20152700 con 0x7fdb0c003ae0 2024-03-28T23:30:11.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.573+0000 7f154010d640 1 -- start start 2024-03-28T23:30:11.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.574+0000 7f154010d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1538071150 0x7f1538071530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.574+0000 7f154010d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f15380c2bf0 0x7f153806b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.574+0000 7f154010d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f15380c35d0 0x7f153806b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.574+0000 7f154010d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1538155bd0 con 0x7f15380c2bf0 2024-03-28T23:30:11.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.574+0000 7f154010d640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f1538154d70 con 0x7f15380c35d0 2024-03-28T23:30:11.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.574+0000 7f154010d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f153814f1e0 con 0x7f1538071150 2024-03-28T23:30:11.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.575+0000 7fdb25c9d640 1 --2- 172.21.15.161:0/1183318047 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb0c003ae0 0x7fdb20156310 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c00c590 tx=0x7fdb200c2920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.575+0000 7f153e6aa640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f15380c35d0 0x7f153806b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.575+0000 7f153dea9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1538071150 0x7f1538071530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.575+0000 7f153dea9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1538071150 0x7f1538071530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:36736/0 (socket says 172.21.15.161:36736) 2024-03-28T23:30:11.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.575+0000 7f153dea9640 1 -- 172.21.15.161:0/4146552698 learned_addr learned my addr 172.21.15.161:0/4146552698 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:11.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.575+0000 7fdb167fc640 1 -- 172.21.15.161:0/1183318047 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb10046070 con 0x7fdb0c003ae0 2024-03-28T23:30:11.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.575+0000 7fdb27f01640 1 -- 172.21.15.161:0/1183318047 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb20153010 con 0x7fdb0c003ae0 2024-03-28T23:30:11.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.576+0000 7fdb27f01640 1 -- 172.21.15.161:0/1183318047 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb201529c0 con 0x7fdb0c003ae0 2024-03-28T23:30:11.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.577+0000 7fdb167fc640 1 -- 172.21.15.161:0/1183318047 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb100047e0 con 0x7fdb0c003ae0 2024-03-28T23:30:11.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.578+0000 7fdb167fc640 1 -- 172.21.15.161:0/1183318047 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb10004bb0 con 0x7fdb0c003ae0 2024-03-28T23:30:11.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.578+0000 7fdb27f01640 1 -- 172.21.15.161:0/1183318047 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdb20068f90 con 0x7fdb0c003ae0 2024-03-28T23:30:11.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.576+0000 7fe58c48f640 1 -- 172.21.15.161:0/3239806650 shutdown_connections 2024-03-28T23:30:11.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.576+0000 7f153dea9640 1 -- 172.21.15.161:0/4146552698 >> v2:172.21.15.170:3300/0 conn(0x7f15380c35d0 msgr2=0x7f153806b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.576+0000 7f153dea9640 1 --2- 172.21.15.161:0/4146552698 >> v2:172.21.15.170:3300/0 conn(0x7f15380c35d0 0x7f153806b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.576+0000 7f153dea9640 1 -- 172.21.15.161:0/4146552698 >> v2:172.21.15.161:3300/0 conn(0x7f15380c2bf0 msgr2=0x7f153806b300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T23:30:11.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.576+0000 7f153dea9640 1 --2- 172.21.15.161:0/4146552698 >> v2:172.21.15.161:3300/0 conn(0x7f15380c2bf0 0x7f153806b300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.576+0000 7f153dea9640 1 -- 172.21.15.161:0/4146552698 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1538154fc0 con 0x7f1538071150 2024-03-28T23:30:11.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.576+0000 7fe58c48f640 1 -- 172.21.15.161:0/3239806650 wait complete. 2024-03-28T23:30:11.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.578+0000 7fe58c48f640 1 Processor -- start 2024-03-28T23:30:11.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.578+0000 7fe58c48f640 1 -- start start 2024-03-28T23:30:11.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.579+0000 7f758f71e640 1 Processor -- start 2024-03-28T23:30:11.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.579+0000 7fdb167fc640 1 -- 172.21.15.161:0/1183318047 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdb10007070 con 0x7fdb0c003ae0 2024-03-28T23:30:11.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.579+0000 7f758f71e640 1 -- start start 2024-03-28T23:30:11.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.580+0000 7fdb167fc640 1 --2- 172.21.15.161:0/1183318047 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdb0406b5d0 0x7fdb0406da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.580+0000 7fdb167fc640 1 -- 172.21.15.161:0/1183318047 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fdb100abb00 con 0x7fdb0c003ae0 2024-03-28T23:30:11.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.579+0000 7fe58c48f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe58406b690 0x7fe584154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.579+0000 7fe58c48f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe584072e10 0x7fe584154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.580+0000 7fdb167fc640 1 --2- 172.21.15.161:0/1183318047 >> [v2:172.21.15.161:6808/3120803198,v1:172.21.15.161:6809/3120803198] conn(0x7fdb04074d50 0x7fdb04077190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.579+0000 7fe58c48f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe584151fd0 0x7fe58414a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.580+0000 7f758f71e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f758806c160 0x7f75881588c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.580+0000 7f758f71e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7588156250 con 0x7f758806c540 2024-03-28T23:30:11.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.580+0000 7f758f71e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75880723e0 con 0x7f7588070fa0 2024-03-28T23:30:11.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.580+0000 7f758f71e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f75881556b0 con 0x7f758806c160 2024-03-28T23:30:11.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.580+0000 7fdb167fc640 1 -- 172.21.15.161:0/1183318047 --> [v2:172.21.15.161:6808/3120803198,v1:172.21.15.161:6809/3120803198] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdb100047e0 con 0x7fdb04074d50 2024-03-28T23:30:11.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.580+0000 7fdb167fc640 1 -- 172.21.15.161:0/1183318047 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdb100abef0 con 0x7fdb0c003ae0 2024-03-28T23:30:11.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.580+0000 7fdb2649e640 1 --2- 172.21.15.161:0/1183318047 >> [v2:172.21.15.161:6808/3120803198,v1:172.21.15.161:6809/3120803198] conn(0x7fdb04074d50 0x7fdb04077190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.579+0000 7fe58c48f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe5840aad70 con 0x7fe58406b690 2024-03-28T23:30:11.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.580+0000 7fe58c48f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe5840aabf0 con 0x7fe584072e10 2024-03-28T23:30:11.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.580+0000 7fe58c48f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe5840c1ad0 con 0x7fe584151fd0 2024-03-28T23:30:11.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.580+0000 7fe58a22b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe58406b690 0x7fe584154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7fe58a22b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe58406b690 0x7fe584154440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54066/0 (socket says 172.21.15.161:54066) 2024-03-28T23:30:11.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7fe58a22b640 1 -- 172.21.15.161:0/3292470588 learned_addr learned my addr 172.21.15.161:0/3292470588 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:11.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.580+0000 7f153dea9640 1 --2- 172.21.15.161:0/4146552698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1538071150 0x7f1538071530 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f15380ad270 tx=0x7f153402fcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc6a3a9b52b551e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7f758ccb9640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f7588070fa0 0x7f7588155b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:46082/0 (socket says 172.21.15.161:46082) 2024-03-28T23:30:11.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7f758ccb9640 1 -- 172.21.15.161:0/1566662224 learned_addr learned my addr 172.21.15.161:0/1566662224 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:11.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7f758d4ba640 1 --2- 172.21.15.161:0/1566662224 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f758806c160 0x7f75881588c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7fdb2549c640 1 --2- 172.21.15.161:0/1183318047 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdb0406b5d0 0x7fdb0406da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7f153cea7640 1 -- 172.21.15.161:0/4146552698 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153403d070 con 0x7f1538071150 2024-03-28T23:30:11.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7f757ffff640 1 -- 172.21.15.161:0/1566662224 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 430497350 0 0) 0x7f75880723e0 con 0x7f7588070fa0 2024-03-28T23:30:11.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7f757ffff640 1 -- 172.21.15.161:0/1566662224 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7574003520 con 0x7f7588070fa0 2024-03-28T23:30:11.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7fe589a2a640 1 --2- 172.21.15.161:0/3292470588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe584072e10 0x7fe584154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7fe58aa2c640 1 --2- 172.21.15.161:0/3292470588 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe584151fd0 0x7fe58414a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7f153cea7640 1 -- 172.21.15.161:0/4146552698 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1534030a70 con 0x7f1538071150 2024-03-28T23:30:11.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7f153cea7640 1 -- 172.21.15.161:0/4146552698 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1534030e40 con 0x7f1538071150 2024-03-28T23:30:11.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7f757ffff640 1 -- 172.21.15.161:0/1566662224 <== mon.1 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2039719928 0 0) 0x7f7574003520 con 0x7f7588070fa0 2024-03-28T23:30:11.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7f757ffff640 1 -- 172.21.15.161:0/1566662224 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f758806c160 msgr2=0x7f75881588c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7f757ffff640 1 --2- 172.21.15.161:0/1566662224 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f758806c160 0x7f75881588c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7f757ffff640 1 -- 172.21.15.161:0/1566662224 >> v1:172.21.15.161:6789/0 conn(0x7f758806c540 legacy=0x7f758815fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7f757ffff640 1 -- 172.21.15.161:0/1566662224 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75880726c0 con 0x7f7588070fa0 2024-03-28T23:30:11.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7fdb2549c640 1 --2- 172.21.15.161:0/1183318047 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdb0406b5d0 0x7fdb0406da90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdb200c0860 tx=0x7fdb18007270 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7fdb2649e640 1 --2- 172.21.15.161:0/1183318047 >> [v2:172.21.15.161:6808/3120803198,v1:172.21.15.161:6809/3120803198] conn(0x7fdb04074d50 0x7fdb04077190 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7f757ffff640 1 -- 172.21.15.161:0/1566662224 <== mon.1 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f7584001670 con 0x7f7588070fa0 2024-03-28T23:30:11.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.582+0000 7f757ffff640 1 -- 172.21.15.161:0/1566662224 >> v1:172.21.15.170:6789/0 conn(0x7f7588070fa0 legacy=0x7f7588155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.582+0000 7f757ffff640 1 --2- 172.21.15.161:0/1566662224 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7574003a90 0x7f7574003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.582+0000 7f757ffff640 1 --2- 172.21.15.161:0/1566662224 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7574004800 0x7f7574004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.582+0000 7f757ffff640 1 --2- 172.21.15.161:0/1566662224 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75740051c0 0x7f757400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.582+0000 7f757ffff640 1 -- 172.21.15.161:0/1566662224 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f757400c090 con 0x7f75740051c0 2024-03-28T23:30:11.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7fe58aa2c640 1 -- 172.21.15.161:0/3292470588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe584072e10 msgr2=0x7fe584154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7fe58aa2c640 1 --2- 172.21.15.161:0/3292470588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe584072e10 0x7fe584154980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7fe58aa2c640 1 -- 172.21.15.161:0/3292470588 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe58406b690 msgr2=0x7fe584154440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.581+0000 7fe58aa2c640 1 --2- 172.21.15.161:0/3292470588 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe58406b690 0x7fe584154440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.582+0000 7f757ffff640 1 -- 172.21.15.161:0/1566662224 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f757400c210 con 0x7f7574003a90 2024-03-28T23:30:11.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.582+0000 7f757ffff640 1 -- 172.21.15.161:0/1566662224 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f757400c3e0 con 0x7f7574004800 2024-03-28T23:30:11.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.582+0000 7fe58aa2c640 1 -- 172.21.15.161:0/3292470588 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5841de4b0 con 0x7fe584151fd0 2024-03-28T23:30:11.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.582+0000 7fe58aa2c640 1 --2- 172.21.15.161:0/3292470588 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe584151fd0 0x7fe58414a7a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe57c0089d0 tx=0x7fe57c00b410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.582+0000 7f154010d640 1 -- 172.21.15.161:0/4146552698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1538071150 msgr2=0x7f1538071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.582+0000 7f154010d640 1 --2- 172.21.15.161:0/4146552698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1538071150 0x7f1538071530 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f15380ad270 tx=0x7f153402fcc0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.583+0000 7f154010d640 1 -- 172.21.15.161:0/4146552698 shutdown_connections 2024-03-28T23:30:11.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.583+0000 7f154010d640 1 --2- 172.21.15.161:0/4146552698 >> v2:172.21.15.170:3300/0 conn(0x7f15380c35d0 0x7f153806b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.583+0000 7f154010d640 1 --2- 172.21.15.161:0/4146552698 >> v2:172.21.15.161:3300/0 conn(0x7f15380c2bf0 0x7f153806b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.583+0000 7f154010d640 1 --2- 172.21.15.161:0/4146552698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1538071150 0x7f1538071530 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.583+0000 7f154010d640 1 -- 172.21.15.161:0/4146552698 >> 172.21.15.161:0/4146552698 conn(0x7f15380c3b70 msgr2=0x7f15380c3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.582+0000 7f758dcbb640 1 --2- 172.21.15.161:0/1566662224 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75740051c0 0x7f757400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.583+0000 7f758dcbb640 1 -- 172.21.15.161:0/1566662224 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7574004800 msgr2=0x7f7574004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.583+0000 7f758dcbb640 1 --2- 172.21.15.161:0/1566662224 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7574004800 0x7f7574004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.583+0000 7f758dcbb640 1 -- 172.21.15.161:0/1566662224 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7574003a90 msgr2=0x7f7574003e90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T23:30:11.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.583+0000 7f758dcbb640 1 --2- 172.21.15.161:0/1566662224 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7574003a90 0x7f7574003e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.583+0000 7f758dcbb640 1 -- 172.21.15.161:0/1566662224 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7574003520 con 0x7f75740051c0 2024-03-28T23:30:11.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.583+0000 7fe57affd640 1 -- 172.21.15.161:0/3292470588 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c003990 con 0x7fe584151fd0 2024-03-28T23:30:11.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.583+0000 7fdb167fc640 1 -- 172.21.15.161:0/1183318047 <== osd.2 v2:172.21.15.161:6808/3120803198 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdb100047e0 con 0x7fdb04074d50 2024-03-28T23:30:11.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.583+0000 7fe57affd640 1 -- 172.21.15.161:0/3292470588 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe57c003b30 con 0x7fe584151fd0 2024-03-28T23:30:11.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.583+0000 7fe57affd640 1 -- 172.21.15.161:0/3292470588 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c004020 con 0x7fe584151fd0 2024-03-28T23:30:11.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.583+0000 7fe58c48f640 1 -- 172.21.15.161:0/3292470588 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5841528a0 con 0x7fe584151fd0 2024-03-28T23:30:11.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.583+0000 7fe58c48f640 1 -- 172.21.15.161:0/3292470588 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5840c0cc0 con 0x7fe584151fd0 2024-03-28T23:30:11.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.584+0000 7f758dcbb640 1 --2- 172.21.15.161:0/1566662224 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75740051c0 0x7f757400b9c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7580001900 tx=0x7f7574005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f01a69f8ebfd4c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.584+0000 7f154010d640 1 -- 172.21.15.161:0/4146552698 shutdown_connections 2024-03-28T23:30:11.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.584+0000 7f154010d640 1 -- 172.21.15.161:0/4146552698 wait complete. 2024-03-28T23:30:11.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.584+0000 7fe58c48f640 1 -- 172.21.15.161:0/3292470588 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe5840aaa10 con 0x7fe584151fd0 2024-03-28T23:30:11.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.585+0000 7f757ffff640 1 -- 172.21.15.161:0/1566662224 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7580001940 con 0x7f75740051c0 2024-03-28T23:30:11.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.585+0000 7f757ffff640 1 -- 172.21.15.161:0/1566662224 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7580038070 con 0x7f75740051c0 2024-03-28T23:30:11.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.585+0000 7f757ffff640 1 -- 172.21.15.161:0/1566662224 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7580003990 con 0x7f75740051c0 2024-03-28T23:30:11.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.585+0000 7f5e8666b640 1 Processor -- start 2024-03-28T23:30:11.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.585+0000 7f5e8666b640 1 -- start start 2024-03-28T23:30:11.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.586+0000 7f5e8666b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e8015cbe0 0x7f5e8015efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.587+0000 7f5e8666b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e800a63c0 con 0x7f5e80157280 2024-03-28T23:30:11.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.587+0000 7f5e8666b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e800a9590 con 0x7f5e800c66d0 2024-03-28T23:30:11.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.587+0000 7f5e8666b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5e800a9340 con 0x7f5e8015cbe0 2024-03-28T23:30:11.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.587+0000 7f5e84c08640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e8015cbe0 0x7f5e8015efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.587+0000 7f5e84c08640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e8015cbe0 0x7f5e8015efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:36774/0 (socket says 172.21.15.161:36774) 2024-03-28T23:30:11.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.586+0000 7f154010d640 1 Processor -- start 2024-03-28T23:30:11.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.587+0000 7f5e84c08640 1 -- 172.21.15.161:0/4047614496 learned_addr learned my addr 172.21.15.161:0/4047614496 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:11.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.586+0000 7f154010d640 1 -- start start 2024-03-28T23:30:11.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.587+0000 7fe57affd640 1 -- 172.21.15.161:0/3292470588 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe57c0041c0 con 0x7fe584151fd0 2024-03-28T23:30:11.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.587+0000 7f5e7effd640 1 -- 172.21.15.161:0/4047614496 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1591223945 0 0) 0x7f5e800a9590 con 0x7f5e800c66d0 2024-03-28T23:30:11.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.587+0000 7f5e7effd640 1 -- 172.21.15.161:0/4047614496 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e68003520 con 0x7f5e800c66d0 2024-03-28T23:30:11.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.587+0000 7fe57affd640 1 --2- 172.21.15.161:0/3292470588 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe56806b5a0 0x7fe56806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.587+0000 7fe57affd640 1 -- 172.21.15.161:0/3292470588 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe57c08add0 con 0x7fe584151fd0 2024-03-28T23:30:11.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.588+0000 7f5e7effd640 1 -- 172.21.15.161:0/4047614496 <== mon.1 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 143786907 0 0) 0x7f5e68003520 con 0x7f5e800c66d0 2024-03-28T23:30:11.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.588+0000 7f5e7effd640 1 -- 172.21.15.161:0/4047614496 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e8015cbe0 msgr2=0x7f5e8015efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.588+0000 7f5e7effd640 1 --2- 172.21.15.161:0/4047614496 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e8015cbe0 0x7f5e8015efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.588+0000 7f5e7effd640 1 -- 172.21.15.161:0/4047614496 >> v1:172.21.15.161:6789/0 conn(0x7f5e80157280 legacy=0x7f5e800c1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.588+0000 7f5e7effd640 1 -- 172.21.15.161:0/4047614496 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e800c5ed0 con 0x7f5e800c66d0 2024-03-28T23:30:11.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.588+0000 7f5e7effd640 1 -- 172.21.15.161:0/4047614496 <== mon.1 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f5e70001670 con 0x7f5e800c66d0 2024-03-28T23:30:11.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.588+0000 7f5e7effd640 1 -- 172.21.15.161:0/4047614496 >> v1:172.21.15.170:6789/0 conn(0x7f5e800c66d0 legacy=0x7f5e800c3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.588+0000 7f5e7effd640 1 --2- 172.21.15.161:0/4047614496 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e68003a90 0x7f5e68003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.588+0000 7f5e7effd640 1 --2- 172.21.15.161:0/4047614496 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e68004800 0x7f5e68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.588+0000 7f5e7effd640 1 --2- 172.21.15.161:0/4047614496 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e680051c0 0x7f5e680145c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.588+0000 7f5e7effd640 1 -- 172.21.15.161:0/4047614496 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5e800a9340 con 0x7f5e680051c0 2024-03-28T23:30:11.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.588+0000 7f5e7effd640 1 -- 172.21.15.161:0/4047614496 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5e68009c10 con 0x7f5e68003a90 2024-03-28T23:30:11.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.588+0000 7f5e7effd640 1 -- 172.21.15.161:0/4047614496 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5e68014b30 con 0x7f5e68004800 2024-03-28T23:30:11.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.588+0000 7fe57affd640 1 --2- 172.21.15.161:0/3292470588 >> [v2:172.21.15.161:6824/3880429893,v1:172.21.15.161:6825/3880429893] conn(0x7fe568074d10 0x7fe568077150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.588+0000 7fe57affd640 1 -- 172.21.15.161:0/3292470588 --> [v2:172.21.15.161:6824/3880429893,v1:172.21.15.161:6825/3880429893] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe57c00fe50 con 0x7fe568074d10 2024-03-28T23:30:11.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.588+0000 7fe57affd640 1 -- 172.21.15.161:0/3292470588 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe57c08b1c0 con 0x7fe584151fd0 2024-03-28T23:30:11.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.588+0000 7fe58a22b640 1 --2- 172.21.15.161:0/3292470588 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe56806b5a0 0x7fe56806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.588+0000 7fe589a2a640 1 --2- 172.21.15.161:0/3292470588 >> [v2:172.21.15.161:6824/3880429893,v1:172.21.15.161:6825/3880429893] conn(0x7fe568074d10 0x7fe568077150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.589+0000 7f758f71e640 1 -- 172.21.15.161:0/1566662224 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75740051c0 msgr2=0x7f757400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.589+0000 7f758f71e640 1 --2- 172.21.15.161:0/1566662224 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75740051c0 0x7f757400b9c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7580001900 tx=0x7f7574005830 comp rx=0 tx=0).stop 2024-03-28T23:30:11.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.589+0000 7f758dcbb640 1 -- 172.21.15.161:0/1566662224 reap_dead start 2024-03-28T23:30:11.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.589+0000 7f758f71e640 1 -- 172.21.15.161:0/1566662224 shutdown_connections 2024-03-28T23:30:11.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.589+0000 7f758f71e640 1 -- 172.21.15.161:0/1566662224 >> 172.21.15.161:0/1566662224 conn(0x7f75880c3cf0 msgr2=0x7f75880c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.589+0000 7fe58a22b640 1 --2- 172.21.15.161:0/3292470588 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe56806b5a0 0x7fe56806da60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe5740017f0 tx=0x7fe574009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.589+0000 7fe589a2a640 1 --2- 172.21.15.161:0/3292470588 >> [v2:172.21.15.161:6824/3880429893,v1:172.21.15.161:6825/3880429893] conn(0x7fe568074d10 0x7fe568077150 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.588+0000 7f5e7f7fe640 1 --2- 172.21.15.161:0/4047614496 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e68004800 0x7f5e68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.589+0000 7f5e7ffff640 1 --2- 172.21.15.161:0/4047614496 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e68003a90 0x7f5e68003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.589+0000 7f5e7f7fe640 1 -- 172.21.15.161:0/4047614496 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e68003a90 msgr2=0x7f5e68003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.589+0000 7f5e7f7fe640 1 --2- 172.21.15.161:0/4047614496 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e68003a90 0x7f5e68003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.589+0000 7f5e7f7fe640 1 -- 172.21.15.161:0/4047614496 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e680051c0 msgr2=0x7f5e680145c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T23:30:11.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.589+0000 7f5e7f7fe640 1 --2- 172.21.15.161:0/4047614496 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e680051c0 0x7f5e680145c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.589+0000 7f5e7f7fe640 1 -- 172.21.15.161:0/4047614496 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e800c5ed0 con 0x7f5e68004800 2024-03-28T23:30:11.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.590+0000 7f5e7ffff640 1 --2- 172.21.15.161:0/4047614496 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e68003a90 0x7f5e68003eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:11.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.588+0000 7f154010d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1538071150 0x7f15380ab2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.589+0000 7f154010d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15380c2bf0 0x7f1538145620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.589+0000 7f154010d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15380c35d0 0x7f1538143f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.589+0000 7f154010d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1538155860 con 0x7f1538071150 2024-03-28T23:30:11.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.589+0000 7f154010d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1538155580 con 0x7f15380c2bf0 2024-03-28T23:30:11.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.589+0000 7f154010d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f15380c4720 con 0x7f15380c35d0 2024-03-28T23:30:11.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.589+0000 7f153d6a8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15380c2bf0 0x7f1538145620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.589+0000 7f153d6a8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15380c2bf0 0x7f1538145620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:39090/0 (socket says 172.21.15.161:39090) 2024-03-28T23:30:11.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.589+0000 7f153d6a8640 1 -- 172.21.15.161:0/2371161997 learned_addr learned my addr 172.21.15.161:0/2371161997 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:11.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.589+0000 7f153e6aa640 1 --2- 172.21.15.161:0/2371161997 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15380c35d0 0x7f1538143f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.590+0000 7f153d6a8640 1 -- 172.21.15.161:0/2371161997 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15380c35d0 msgr2=0x7f1538143f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.590+0000 7f153d6a8640 1 --2- 172.21.15.161:0/2371161997 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15380c35d0 0x7f1538143f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.590+0000 7f153d6a8640 1 -- 172.21.15.161:0/2371161997 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1538071150 msgr2=0x7f15380ab2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T23:30:11.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.590+0000 7f153d6a8640 1 --2- 172.21.15.161:0/2371161997 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1538071150 0x7f15380ab2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.590+0000 7f153d6a8640 1 -- 172.21.15.161:0/2371161997 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f153806be60 con 0x7f15380c2bf0 2024-03-28T23:30:11.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.590+0000 7f153d6a8640 1 --2- 172.21.15.161:0/2371161997 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15380c2bf0 0x7f1538145620 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f153000ba80 tx=0x7f153000bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.590+0000 7f758f71e640 1 -- 172.21.15.161:0/1566662224 shutdown_connections 2024-03-28T23:30:11.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.590+0000 7fe57affd640 1 -- 172.21.15.161:0/3292470588 <== osd.3 v2:172.21.15.161:6824/3880429893 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe57c00fe50 con 0x7fe568074d10 2024-03-28T23:30:11.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.590+0000 7f152e7fc640 1 -- 172.21.15.161:0/2371161997 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1530019070 con 0x7f15380c2bf0 2024-03-28T23:30:11.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.590+0000 7f154010d640 1 -- 172.21.15.161:0/2371161997 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1538143190 con 0x7f15380c2bf0 2024-03-28T23:30:11.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.591+0000 7f154010d640 1 -- 172.21.15.161:0/2371161997 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1538143750 con 0x7f15380c2bf0 2024-03-28T23:30:11.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.591+0000 7f5e7f7fe640 1 --2- 172.21.15.161:0/4047614496 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e68004800 0x7f5e68004c80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5e7000c160 tx=0x7f5e68005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b12624e9b9fe9cb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.591+0000 7f152e7fc640 1 -- 172.21.15.161:0/2371161997 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f153000cc20 con 0x7f15380c2bf0 2024-03-28T23:30:11.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.591+0000 7f152e7fc640 1 -- 172.21.15.161:0/2371161997 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1530014440 con 0x7f15380c2bf0 2024-03-28T23:30:11.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.592+0000 7f5e7effd640 1 -- 172.21.15.161:0/4047614496 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e70007c80 con 0x7f5e68004800 2024-03-28T23:30:11.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.592+0000 7f5e7effd640 1 -- 172.21.15.161:0/4047614496 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e70030070 con 0x7f5e68004800 2024-03-28T23:30:11.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.592+0000 7f5e7effd640 1 -- 172.21.15.161:0/4047614496 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e7002cd00 con 0x7f5e68004800 2024-03-28T23:30:11.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.590+0000 7f758f71e640 1 -- 172.21.15.161:0/1566662224 wait complete. 2024-03-28T23:30:11.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.592+0000 7f758f71e640 1 Processor -- start 2024-03-28T23:30:11.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.592+0000 7f758f71e640 1 -- start start 2024-03-28T23:30:11.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.592+0000 7f5e8666b640 1 -- 172.21.15.161:0/4047614496 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e68004800 msgr2=0x7f5e68004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.592+0000 7f5e8666b640 1 --2- 172.21.15.161:0/4047614496 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e68004800 0x7f5e68004c80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5e7000c160 tx=0x7f5e68005830 comp rx=0 tx=0).stop 2024-03-28T23:30:11.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.593+0000 7f5e84c08640 1 -- 172.21.15.161:0/4047614496 reap_dead start 2024-03-28T23:30:11.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.593+0000 7f5e8666b640 1 -- 172.21.15.161:0/4047614496 shutdown_connections 2024-03-28T23:30:11.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.593+0000 7f5e8666b640 1 -- 172.21.15.161:0/4047614496 >> 172.21.15.161:0/4047614496 conn(0x7f5e800a8820 msgr2=0x7f5e800a8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.593+0000 7f758f71e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75740051c0 0x7f7588071a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.593+0000 7f758f71e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f758814fcf0 0x7f75881503f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.593+0000 7f154010d640 1 -- 172.21.15.161:0/2371161997 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1520001030 con 0x7f15380c2bf0 2024-03-28T23:30:11.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.593+0000 7f758f71e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75880c0610 0x7f75880c0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.593+0000 7f5e8666b640 1 -- 172.21.15.161:0/4047614496 shutdown_connections 2024-03-28T23:30:11.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.593+0000 7f5e8666b640 1 -- 172.21.15.161:0/4047614496 wait complete. 2024-03-28T23:30:11.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.594+0000 7f758ccb9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f758814fcf0 0x7f75881503f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.594+0000 7f758d4ba640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75740051c0 0x7f7588071a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.594+0000 7f758ccb9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f758814fcf0 0x7f75881503f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:36802/0 (socket says 172.21.15.161:36802) 2024-03-28T23:30:11.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.594+0000 7f758ccb9640 1 -- 172.21.15.161:0/382349642 learned_addr learned my addr 172.21.15.161:0/382349642 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:11.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.594+0000 7f758dcbb640 1 --2- 172.21.15.161:0/382349642 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75880c0610 0x7f75880c0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.594+0000 7f758f71e640 1 -- 172.21.15.161:0/382349642 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f75881552d0 con 0x7f75740051c0 2024-03-28T23:30:11.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.594+0000 7f758f71e640 1 -- 172.21.15.161:0/382349642 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f758806c9d0 con 0x7f75880c0610 2024-03-28T23:30:11.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.594+0000 7f758f71e640 1 -- 172.21.15.161:0/382349642 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7588152b30 con 0x7f758814fcf0 2024-03-28T23:30:11.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.595+0000 7f758ccb9640 1 -- 172.21.15.161:0/382349642 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75880c0610 msgr2=0x7f75880c0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.595+0000 7f758ccb9640 1 --2- 172.21.15.161:0/382349642 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75880c0610 0x7f75880c0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.595+0000 7f758ccb9640 1 -- 172.21.15.161:0/382349642 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75740051c0 msgr2=0x7f7588071a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.595+0000 7f758ccb9640 1 --2- 172.21.15.161:0/382349642 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75740051c0 0x7f7588071a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.595+0000 7f758ccb9640 1 -- 172.21.15.161:0/382349642 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75881f7770 con 0x7f758814fcf0 2024-03-28T23:30:11.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.595+0000 7f758d4ba640 1 --2- 172.21.15.161:0/382349642 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75740051c0 0x7f7588071a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:11.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.595+0000 7f758ccb9640 1 --2- 172.21.15.161:0/382349642 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f758814fcf0 0x7f75881503f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f758400c800 tx=0x7f758400ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.595+0000 7fa55f72c640 1 Processor -- start 2024-03-28T23:30:11.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.595+0000 7f757dffb640 1 -- 172.21.15.161:0/382349642 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7584018070 con 0x7f758814fcf0 2024-03-28T23:30:11.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.595+0000 7f5e8666b640 1 Processor -- start 2024-03-28T23:30:11.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.595+0000 7fa55f72c640 1 -- start start 2024-03-28T23:30:11.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.595+0000 7f758f71e640 1 -- 172.21.15.161:0/382349642 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75881f7a50 con 0x7f758814fcf0 2024-03-28T23:30:11.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.596+0000 7f758f71e640 1 -- 172.21.15.161:0/382349642 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f75881f8010 con 0x7f758814fcf0 2024-03-28T23:30:11.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.595+0000 7f5e8666b640 1 -- start start 2024-03-28T23:30:11.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.596+0000 7f5e8666b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e800c5690 0x7f5e801e8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.597+0000 7fa55f72c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa558072ee0 0x7fa5580c4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.597+0000 7fa55f72c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa5580733c0 0x7fa5580c4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.597+0000 7fa55f72c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5580701b0 con 0x7fa558150690 2024-03-28T23:30:11.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.597+0000 7fa55f72c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa558070490 con 0x7fa558072ee0 2024-03-28T23:30:11.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.597+0000 7fa55f72c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa558153ad0 con 0x7fa5580733c0 2024-03-28T23:30:11.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.596+0000 7f152e7fc640 1 -- 172.21.15.161:0/2371161997 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f153001c070 con 0x7f15380c2bf0 2024-03-28T23:30:11.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.597+0000 7f152e7fc640 1 --2- 172.21.15.161:0/2371161997 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f151c06b5d0 0x7f151c06da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.597+0000 7f152e7fc640 1 -- 172.21.15.161:0/2371161997 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f153008a1d0 con 0x7f15380c2bf0 2024-03-28T23:30:11.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.597+0000 7f152e7fc640 1 --2- 172.21.15.161:0/2371161997 >> [v2:172.21.15.170:6808/844763145,v1:172.21.15.170:6809/844763145] conn(0x7f151c074d40 0x7f151c077180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.597+0000 7f757dffb640 1 -- 172.21.15.161:0/382349642 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7584007520 con 0x7f758814fcf0 2024-03-28T23:30:11.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.597+0000 7f152e7fc640 1 -- 172.21.15.161:0/2371161997 --> [v2:172.21.15.170:6808/844763145,v1:172.21.15.170:6809/844763145] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f151c077850 con 0x7f151c074d40 2024-03-28T23:30:11.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.597+0000 7f152e7fc640 1 -- 172.21.15.161:0/2371161997 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1530091050 con 0x7f15380c2bf0 2024-03-28T23:30:11.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.597+0000 7f75737fe640 1 -- 172.21.15.161:0/382349642 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f75880c1000 con 0x7f758814fcf0 2024-03-28T23:30:11.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.597+0000 7f757dffb640 1 -- 172.21.15.161:0/382349642 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7584004a20 con 0x7f758814fcf0 2024-03-28T23:30:11.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.597+0000 7f5e7ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e800c5690 0x7f5e801e8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.597+0000 7f5e7ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e800c5690 0x7f5e801e8040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54096/0 (socket says 172.21.15.161:54096) 2024-03-28T23:30:11.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.597+0000 7f5e7ffff640 1 -- 172.21.15.161:0/2848689575 learned_addr learned my addr 172.21.15.161:0/2848689575 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:11.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.597+0000 7f5e8666b640 1 --2- 172.21.15.161:0/2848689575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e800bf4e0 0x7f5e80151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.597+0000 7f5e8666b640 1 --2- 172.21.15.161:0/2848689575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e801ea9a0 0x7f5e801502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.597+0000 7fa55d4c8640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fa558150690 0x7fa55806f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:46088/0 (socket says 172.21.15.161:46088) 2024-03-28T23:30:11.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.597+0000 7fa55d4c8640 1 -- 172.21.15.161:0/3838265860 learned_addr learned my addr 172.21.15.161:0/3838265860 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:11.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.597+0000 7fa55ccc7640 1 --2- 172.21.15.161:0/3838265860 >> v2:172.21.15.161:3300/0 conn(0x7fa558072ee0 0x7fa5580c4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.597+0000 7f5e7f7fe640 1 --2- 172.21.15.161:0/2848689575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e800bf4e0 0x7f5e80151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.598+0000 7f5e8666b640 1 -- 172.21.15.161:0/2848689575 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5e800c1b80 con 0x7f5e800c5690 2024-03-28T23:30:11.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.598+0000 7f5e8666b640 1 -- 172.21.15.161:0/2848689575 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5e800c6240 con 0x7f5e800bf4e0 2024-03-28T23:30:11.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.598+0000 7f5e8666b640 1 -- 172.21.15.161:0/2848689575 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5e80151550 con 0x7f5e801ea9a0 2024-03-28T23:30:11.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.598+0000 7f5e84c08640 1 --2- 172.21.15.161:0/2848689575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e801ea9a0 0x7f5e801502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.599+0000 7f5e7ffff640 1 -- 172.21.15.161:0/2848689575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e801ea9a0 msgr2=0x7f5e801502c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.599+0000 7f757dffb640 1 -- 172.21.15.161:0/382349642 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f758401b070 con 0x7f758814fcf0 2024-03-28T23:30:11.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.598+0000 7fa55dcc9640 1 --2- 172.21.15.161:0/3838265860 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa5580733c0 0x7fa5580c4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.598+0000 7fa55ccc7640 1 -- 172.21.15.161:0/3838265860 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa5580733c0 msgr2=0x7fa5580c4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.598+0000 7fa55ccc7640 1 --2- 172.21.15.161:0/3838265860 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa5580733c0 0x7fa5580c4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.598+0000 7fa55ccc7640 1 -- 172.21.15.161:0/3838265860 >> v1:172.21.15.170:6789/0 conn(0x7fa558150690 legacy=0x7fa55806f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.598+0000 7fa55ccc7640 1 -- 172.21.15.161:0/3838265860 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa558153db0 con 0x7fa558072ee0 2024-03-28T23:30:11.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.599+0000 7f5e7ffff640 1 --2- 172.21.15.161:0/2848689575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e801ea9a0 0x7f5e801502c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.599+0000 7f5e7ffff640 1 -- 172.21.15.161:0/2848689575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e800bf4e0 msgr2=0x7f5e80151010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.599+0000 7fa55ccc7640 1 --2- 172.21.15.161:0/3838265860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa558072ee0 0x7fa5580c4710 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa550009d90 tx=0x7fa5500346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=789e99b686829639 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.599+0000 7f5e7ffff640 1 --2- 172.21.15.161:0/2848689575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e800bf4e0 0x7f5e80151010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.599+0000 7f5e7ffff640 1 -- 172.21.15.161:0/2848689575 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e801ea1e0 con 0x7f5e800c5690 2024-03-28T23:30:11.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.599+0000 7f153dea9640 1 --2- 172.21.15.161:0/2371161997 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f151c06b5d0 0x7f151c06da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.599+0000 7f153e6aa640 1 --2- 172.21.15.161:0/2371161997 >> [v2:172.21.15.170:6808/844763145,v1:172.21.15.170:6809/844763145] conn(0x7f151c074d40 0x7f151c077180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.599+0000 7f5e7ffff640 1 --2- 172.21.15.161:0/2848689575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e800c5690 0x7f5e801e8040 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5e7800e8d0 tx=0x7f5e7800ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.599+0000 7fa54ffff640 1 -- 172.21.15.161:0/3838265860 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa550007740 con 0x7fa558072ee0 2024-03-28T23:30:11.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.599+0000 7f757dffb640 1 --2- 172.21.15.161:0/382349642 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f756c06b5d0 0x7f756c06da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.599+0000 7fa54ffff640 1 -- 172.21.15.161:0/3838265860 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5500078e0 con 0x7fa558072ee0 2024-03-28T23:30:11.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.599+0000 7f757dffb640 1 -- 172.21.15.161:0/382349642 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f758403b020 con 0x7f758814fcf0 2024-03-28T23:30:11.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.600+0000 7f757dffb640 1 --2- 172.21.15.161:0/382349642 >> [v2:172.21.15.161:6816/4264782123,v1:172.21.15.161:6817/4264782123] conn(0x7f756c074d50 0x7f756c077190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.600+0000 7f757dffb640 1 -- 172.21.15.161:0/382349642 --> [v2:172.21.15.161:6816/4264782123,v1:172.21.15.161:6817/4264782123] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7584015ec0 con 0x7f756c074d50 2024-03-28T23:30:11.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.600+0000 7f757dffb640 1 -- 172.21.15.161:0/382349642 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f758408a3a0 con 0x7f758814fcf0 2024-03-28T23:30:11.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.600+0000 7f5e7f7fe640 1 --2- 172.21.15.161:0/2848689575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e800bf4e0 0x7f5e80151010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:11.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.600+0000 7f0570823640 1 Processor -- start 2024-03-28T23:30:11.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.600+0000 7f758dcbb640 1 --2- 172.21.15.161:0/382349642 >> [v2:172.21.15.161:6816/4264782123,v1:172.21.15.161:6817/4264782123] conn(0x7f756c074d50 0x7f756c077190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.600+0000 7f5e84c08640 1 --2- 172.21.15.161:0/2848689575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e801ea9a0 0x7f5e801502c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:11.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.600+0000 7f5e7cff9640 1 -- 172.21.15.161:0/2848689575 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e78019020 con 0x7f5e800c5690 2024-03-28T23:30:11.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.600+0000 7f5e7cff9640 1 -- 172.21.15.161:0/2848689575 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e78006b50 con 0x7f5e800c5690 2024-03-28T23:30:11.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.600+0000 7f0570823640 1 -- start start 2024-03-28T23:30:11.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.600+0000 7f153dea9640 1 --2- 172.21.15.161:0/2371161997 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f151c06b5d0 0x7f151c06da90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f153403a000 tx=0x7f1534034000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.600+0000 7f758d4ba640 1 --2- 172.21.15.161:0/382349642 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f756c06b5d0 0x7f756c06da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.601+0000 7f5e8666b640 1 -- 172.21.15.161:0/2848689575 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e800bba90 con 0x7f5e800c5690 2024-03-28T23:30:11.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.600+0000 7fa55f72c640 1 -- 172.21.15.161:0/3838265860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa558072ee0 msgr2=0x7fa5580c4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.601+0000 7f153e6aa640 1 --2- 172.21.15.161:0/2371161997 >> [v2:172.21.15.170:6808/844763145,v1:172.21.15.170:6809/844763145] conn(0x7f151c074d40 0x7f151c077180 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.600+0000 7fa55f72c640 1 --2- 172.21.15.161:0/3838265860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa558072ee0 0x7fa5580c4710 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa550009d90 tx=0x7fa5500346d0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.601+0000 7fa55f72c640 1 -- 172.21.15.161:0/3838265860 shutdown_connections 2024-03-28T23:30:11.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.601+0000 7fa55f72c640 1 --2- 172.21.15.161:0/3838265860 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa5580733c0 0x7fa5580c4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.601+0000 7f5e8666b640 1 -- 172.21.15.161:0/2848689575 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e801ea480 con 0x7f5e800c5690 2024-03-28T23:30:11.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.601+0000 7fa55f72c640 1 --2- 172.21.15.161:0/3838265860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa558072ee0 0x7fa5580c4710 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.601+0000 7fa55f72c640 1 -- 172.21.15.161:0/3838265860 >> 172.21.15.161:0/3838265860 conn(0x7fa5580c88b0 msgr2=0x7fa5580c8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.601+0000 7f0570823640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f056c0c1c60 0x7f056c15cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.601+0000 7f0570823640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f056c0c3410 0x7f056c15d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.601+0000 7f0570823640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f056c153ac0 con 0x7f056c1536e0 2024-03-28T23:30:11.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.601+0000 7f0570823640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f056c15a590 con 0x7f056c0c1c60 2024-03-28T23:30:11.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.601+0000 7f0570823640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f056c159a10 con 0x7f056c0c3410 2024-03-28T23:30:11.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.601+0000 7f056a59c640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f056c1536e0 0x7f056c0c6660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:46092/0 (socket says 172.21.15.161:46092) 2024-03-28T23:30:11.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.601+0000 7f056a59c640 1 -- 172.21.15.161:0/997854638 learned_addr learned my addr 172.21.15.161:0/997854638 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:11.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.602+0000 7f056ad9d640 1 --2- 172.21.15.161:0/997854638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f056c0c3410 0x7f056c15d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.601+0000 7f758dcbb640 1 --2- 172.21.15.161:0/382349642 >> [v2:172.21.15.161:6816/4264782123,v1:172.21.15.161:6817/4264782123] conn(0x7f756c074d50 0x7f756c077190 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.602+0000 7f5e667fc640 1 -- 172.21.15.161:0/2848689575 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5e60001030 con 0x7f5e800c5690 2024-03-28T23:30:11.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.603+0000 7f0563fff640 1 --2- 172.21.15.161:0/997854638 >> v2:172.21.15.161:3300/0 conn(0x7f056c0c1c60 0x7f056c15cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.602+0000 7f152e7fc640 1 -- 172.21.15.161:0/2371161997 <== osd.6 v2:172.21.15.170:6808/844763145 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f151c077850 con 0x7f151c074d40 2024-03-28T23:30:11.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.603+0000 7f056ad9d640 1 -- 172.21.15.161:0/997854638 >> v2:172.21.15.161:3300/0 conn(0x7f056c0c1c60 msgr2=0x7f056c15cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.603+0000 7f056ad9d640 1 --2- 172.21.15.161:0/997854638 >> v2:172.21.15.161:3300/0 conn(0x7f056c0c1c60 0x7f056c15cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.603+0000 7f056ad9d640 1 -- 172.21.15.161:0/997854638 >> v1:172.21.15.170:6789/0 conn(0x7f056c1536e0 legacy=0x7f056c0c6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.603+0000 7f758d4ba640 1 --2- 172.21.15.161:0/382349642 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f756c06b5d0 0x7f756c06da90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f75880c2530 tx=0x7f7578009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.603+0000 7f056ad9d640 1 -- 172.21.15.161:0/997854638 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f056c159cf0 con 0x7f056c0c3410 2024-03-28T23:30:11.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.603+0000 7f757dffb640 1 -- 172.21.15.161:0/382349642 <== osd.0 v2:172.21.15.161:6816/4264782123 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7584015ec0 con 0x7f756c074d50 2024-03-28T23:30:11.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.603+0000 7f5e7cff9640 1 -- 172.21.15.161:0/2848689575 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e78004960 con 0x7f5e800c5690 2024-03-28T23:30:11.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.604+0000 7fa55f72c640 1 -- 172.21.15.161:0/3838265860 shutdown_connections 2024-03-28T23:30:11.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.604+0000 7f5e7cff9640 1 -- 172.21.15.161:0/2848689575 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5e78004b00 con 0x7f5e800c5690 2024-03-28T23:30:11.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.604+0000 7fa55f72c640 1 -- 172.21.15.161:0/3838265860 wait complete. 2024-03-28T23:30:11.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.604+0000 7f5e7cff9640 1 --2- 172.21.15.161:0/2848689575 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5e6c06b5a0 0x7f5e6c06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.605+0000 7f5e7f7fe640 1 --2- 172.21.15.161:0/2848689575 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5e6c06b5a0 0x7f5e6c06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.606+0000 7f5e7f7fe640 1 --2- 172.21.15.161:0/2848689575 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5e6c06b5a0 0x7f5e6c06da60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5e70035000 tx=0x7f5e70009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.606+0000 7fa55f72c640 1 Processor -- start 2024-03-28T23:30:11.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.606+0000 7f5e7cff9640 1 -- 172.21.15.161:0/2848689575 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5e7808a700 con 0x7f5e800c5690 2024-03-28T23:30:11.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.606+0000 7fa55f72c640 1 -- start start 2024-03-28T23:30:11.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.606+0000 7f5e7cff9640 1 --2- 172.21.15.161:0/2848689575 >> [v2:172.21.15.170:6824/211299856,v1:172.21.15.170:6825/211299856] conn(0x7f5e6c074d10 0x7f5e6c077150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.606+0000 7f5e7cff9640 1 -- 172.21.15.161:0/2848689575 --> [v2:172.21.15.170:6824/211299856,v1:172.21.15.170:6825/211299856] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5e780020b0 con 0x7f5e6c074d10 2024-03-28T23:30:11.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.606+0000 7f5e7cff9640 1 -- 172.21.15.161:0/2848689575 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5e78091050 con 0x7f5e800c5690 2024-03-28T23:30:11.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.606+0000 7fa55f72c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa558072ee0 0x7fa5581db440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.606+0000 7f056ad9d640 1 --2- 172.21.15.161:0/997854638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f056c0c3410 0x7f056c15d430 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0564009d90 tx=0x7f056402bda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8d0f103b7bd2fdd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.606+0000 7fa55f72c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5580733c0 0x7fa5581e78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.607+0000 7fa55f72c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa558150690 0x7fa5581e7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.607+0000 7fa55f72c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa55806ff90 con 0x7fa5580733c0 2024-03-28T23:30:11.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.607+0000 7fa55f72c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa55806fe10 con 0x7fa558072ee0 2024-03-28T23:30:11.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.607+0000 7fa55f72c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa558150a70 con 0x7fa558150690 2024-03-28T23:30:11.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.607+0000 7fa55d4c8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa558072ee0 0x7fa5581db440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.607+0000 7f0569d9b640 1 -- 172.21.15.161:0/997854638 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0564039070 con 0x7f056c0c3410 2024-03-28T23:30:11.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.607+0000 7fa55d4c8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa558072ee0 0x7fa5581db440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:39132/0 (socket says 172.21.15.161:39132) 2024-03-28T23:30:11.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.607+0000 7fa55d4c8640 1 -- 172.21.15.161:0/2053581074 learned_addr learned my addr 172.21.15.161:0/2053581074 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:11.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.607+0000 7fa55dcc9640 1 --2- 172.21.15.161:0/2053581074 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa558150690 0x7fa5581e7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.607+0000 7f0569d9b640 1 -- 172.21.15.161:0/997854638 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f056402cb50 con 0x7f056c0c3410 2024-03-28T23:30:11.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.607+0000 7f0569d9b640 1 -- 172.21.15.161:0/997854638 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0564034440 con 0x7f056c0c3410 2024-03-28T23:30:11.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.607+0000 7fa55d4c8640 1 -- 172.21.15.161:0/2053581074 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa558150690 msgr2=0x7fa5581e7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.607+0000 7fa55d4c8640 1 --2- 172.21.15.161:0/2053581074 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa558150690 0x7fa5581e7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.607+0000 7fa55d4c8640 1 -- 172.21.15.161:0/2053581074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5580733c0 msgr2=0x7fa5581e78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.607+0000 7fa55d4c8640 1 --2- 172.21.15.161:0/2053581074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5580733c0 0x7fa5581e78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.607+0000 7fa55d4c8640 1 -- 172.21.15.161:0/2053581074 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5580be9f0 con 0x7fa558072ee0 2024-03-28T23:30:11.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.607+0000 7fa55d4c8640 1 --2- 172.21.15.161:0/2053581074 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa558072ee0 0x7fa5581db440 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa55400c3f0 tx=0x7fa55400c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.607+0000 7fa54dffb640 1 -- 172.21.15.161:0/2053581074 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa554017070 con 0x7fa558072ee0 2024-03-28T23:30:11.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.608+0000 7fa54dffb640 1 -- 172.21.15.161:0/2053581074 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa554005ce0 con 0x7fa558072ee0 2024-03-28T23:30:11.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.608+0000 7fa54dffb640 1 -- 172.21.15.161:0/2053581074 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5540072f0 con 0x7fa558072ee0 2024-03-28T23:30:11.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.608+0000 7fa55f72c640 1 -- 172.21.15.161:0/2053581074 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5581d9fa0 con 0x7fa558072ee0 2024-03-28T23:30:11.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.608+0000 7f0570823640 1 -- 172.21.15.161:0/997854638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f056c0c3410 msgr2=0x7f056c15d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.608+0000 7f0570823640 1 --2- 172.21.15.161:0/997854638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f056c0c3410 0x7f056c15d430 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0564009d90 tx=0x7f056402bda0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.608+0000 7fa55f72c640 1 -- 172.21.15.161:0/2053581074 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5581da560 con 0x7fa558072ee0 2024-03-28T23:30:11.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.608+0000 7f0570823640 1 -- 172.21.15.161:0/997854638 shutdown_connections 2024-03-28T23:30:11.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.608+0000 7f0570823640 1 --2- 172.21.15.161:0/997854638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f056c0c3410 0x7f056c15d430 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.608+0000 7f0570823640 1 --2- 172.21.15.161:0/997854638 >> v2:172.21.15.161:3300/0 conn(0x7f056c0c1c60 0x7f056c15cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.608+0000 7f0570823640 1 -- 172.21.15.161:0/997854638 >> 172.21.15.161:0/997854638 conn(0x7f056c0c4250 msgr2=0x7f056c0c4660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.609+0000 7fa54dffb640 1 -- 172.21.15.161:0/2053581074 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa55401a070 con 0x7fa558072ee0 2024-03-28T23:30:11.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.610+0000 7fa55f72c640 1 -- 172.21.15.161:0/2053581074 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa5581dac20 con 0x7fa558072ee0 2024-03-28T23:30:11.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.610+0000 7f5e84c08640 1 --2- 172.21.15.161:0/2848689575 >> [v2:172.21.15.170:6824/211299856,v1:172.21.15.170:6825/211299856] conn(0x7f5e6c074d10 0x7f5e6c077150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.610+0000 7f0570823640 1 -- 172.21.15.161:0/997854638 shutdown_connections 2024-03-28T23:30:11.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.610+0000 7f0570823640 1 -- 172.21.15.161:0/997854638 wait complete. 2024-03-28T23:30:11.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.610+0000 7fa54dffb640 1 --2- 172.21.15.161:0/2053581074 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa53c06b4d0 0x7fa53c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.610+0000 7fa54dffb640 1 -- 172.21.15.161:0/2053581074 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa5540898b0 con 0x7fa558072ee0 2024-03-28T23:30:11.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.610+0000 7fa54dffb640 1 --2- 172.21.15.161:0/2053581074 >> [v2:172.21.15.161:6800/3551918247,v1:172.21.15.161:6801/3551918247] conn(0x7fa53c074ac0 0x7fa53c076f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.610+0000 7fa54dffb640 1 -- 172.21.15.161:0/2053581074 --> [v2:172.21.15.161:6800/3551918247,v1:172.21.15.161:6801/3551918247] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa554015e70 con 0x7fa53c074ac0 2024-03-28T23:30:11.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.610+0000 7fa54dffb640 1 -- 172.21.15.161:0/2053581074 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa554053fb0 con 0x7fa558072ee0 2024-03-28T23:30:11.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.610+0000 7fa55ccc7640 1 --2- 172.21.15.161:0/2053581074 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa53c06b4d0 0x7fa53c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.610+0000 7fa55dcc9640 1 --2- 172.21.15.161:0/2053581074 >> [v2:172.21.15.161:6800/3551918247,v1:172.21.15.161:6801/3551918247] conn(0x7fa53c074ac0 0x7fa53c076f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.611+0000 7f5e84c08640 1 --2- 172.21.15.161:0/2848689575 >> [v2:172.21.15.170:6824/211299856,v1:172.21.15.170:6825/211299856] conn(0x7f5e6c074d10 0x7f5e6c077150 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.611+0000 7fa55dcc9640 1 --2- 172.21.15.161:0/2053581074 >> [v2:172.21.15.161:6800/3551918247,v1:172.21.15.161:6801/3551918247] conn(0x7fa53c074ac0 0x7fa53c076f00 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.611+0000 7fa55ccc7640 1 --2- 172.21.15.161:0/2053581074 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa53c06b4d0 0x7fa53c06d990 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa55003e000 tx=0x7fa550038000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.612+0000 7f0570823640 1 Processor -- start 2024-03-28T23:30:11.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.612+0000 7f0570823640 1 -- start start 2024-03-28T23:30:11.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.612+0000 7f5e7cff9640 1 -- 172.21.15.161:0/2848689575 <== osd.5 v2:172.21.15.170:6824/211299856 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5e780020b0 con 0x7f5e6c074d10 2024-03-28T23:30:11.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.612+0000 7fa54dffb640 1 -- 172.21.15.161:0/2053581074 <== osd.1 v2:172.21.15.161:6800/3551918247 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa554015e70 con 0x7fa53c074ac0 2024-03-28T23:30:11.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.615+0000 7fdb27f01640 1 -- 172.21.15.161:0/1183318047 --> [v2:172.21.15.161:6808/3120803198,v1:172.21.15.161:6809/3120803198] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdb2004f4b0 con 0x7fdb04074d50 2024-03-28T23:30:11.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.615+0000 7fdb167fc640 1 -- 172.21.15.161:0/1183318047 <== osd.2 v2:172.21.15.161:6808/3120803198 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdb2004f4b0 con 0x7fdb04074d50 2024-03-28T23:30:11.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.616+0000 7fdb27f01640 1 -- 172.21.15.161:0/1183318047 >> [v2:172.21.15.161:6808/3120803198,v1:172.21.15.161:6809/3120803198] conn(0x7fdb04074d50 msgr2=0x7fdb04077190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.616+0000 7fdb27f01640 1 --2- 172.21.15.161:0/1183318047 >> [v2:172.21.15.161:6808/3120803198,v1:172.21.15.161:6809/3120803198] conn(0x7fdb04074d50 0x7fdb04077190 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.613+0000 7f0570823640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f056c0c3410 0x7f056c0acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.613+0000 7f0570823640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f056c1536e0 0x7f056c14e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.613+0000 7f0570823640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f056c14feb0 0x7f056c150310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.613+0000 7f056ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f056c14feb0 0x7f056c150310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.613+0000 7f056ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f056c14feb0 0x7f056c150310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54146/0 (socket says 172.21.15.161:54146) 2024-03-28T23:30:11.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.613+0000 7f056ad9d640 1 -- 172.21.15.161:0/1352005790 learned_addr learned my addr 172.21.15.161:0/1352005790 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:11.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.613+0000 7f0570823640 1 -- 172.21.15.161:0/1352005790 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f056c0c6d70 con 0x7f056c14feb0 2024-03-28T23:30:11.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.613+0000 7f0570823640 1 -- 172.21.15.161:0/1352005790 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f056c15a2b0 con 0x7f056c0c3410 2024-03-28T23:30:11.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.613+0000 7f0570823640 1 -- 172.21.15.161:0/1352005790 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f056c0c6ef0 con 0x7f056c1536e0 2024-03-28T23:30:11.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.614+0000 7f056ad9d640 1 -- 172.21.15.161:0/1352005790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f056c1536e0 msgr2=0x7f056c14e680 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T23:30:11.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.614+0000 7f056ad9d640 1 --2- 172.21.15.161:0/1352005790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f056c1536e0 0x7f056c14e680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.614+0000 7f056ad9d640 1 -- 172.21.15.161:0/1352005790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f056c0c3410 msgr2=0x7f056c0acdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.614+0000 7f056a59c640 1 --2- 172.21.15.161:0/1352005790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f056c0c3410 0x7f056c0acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.614+0000 7f056ad9d640 1 --2- 172.21.15.161:0/1352005790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f056c0c3410 0x7f056c0acdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.614+0000 7f056ad9d640 1 -- 172.21.15.161:0/1352005790 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f056c1efcc0 con 0x7f056c14feb0 2024-03-28T23:30:11.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.614+0000 7f056ad9d640 1 --2- 172.21.15.161:0/1352005790 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f056c14feb0 0x7f056c150310 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0564009d90 tx=0x7f0564004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.615+0000 7f0562ffd640 1 -- 172.21.15.161:0/1352005790 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0564044020 con 0x7f056c14feb0 2024-03-28T23:30:11.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.615+0000 7f0562ffd640 1 -- 172.21.15.161:0/1352005790 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0564032670 con 0x7f056c14feb0 2024-03-28T23:30:11.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.615+0000 7f0562ffd640 1 -- 172.21.15.161:0/1352005790 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0564039040 con 0x7f056c14feb0 2024-03-28T23:30:11.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.615+0000 7f0570823640 1 -- 172.21.15.161:0/1352005790 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f056c14beb0 con 0x7f056c14feb0 2024-03-28T23:30:11.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.616+0000 7fdb27f01640 1 -- 172.21.15.161:0/1183318047 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdb0406b5d0 msgr2=0x7fdb0406da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.616+0000 7fdb27f01640 1 --2- 172.21.15.161:0/1183318047 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdb0406b5d0 0x7fdb0406da90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdb200c0860 tx=0x7fdb18007270 comp rx=0 tx=0).stop 2024-03-28T23:30:11.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.616+0000 7fdb27f01640 1 -- 172.21.15.161:0/1183318047 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb0c003ae0 msgr2=0x7fdb20156310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.616+0000 7fdb27f01640 1 --2- 172.21.15.161:0/1183318047 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb0c003ae0 0x7fdb20156310 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c00c590 tx=0x7fdb200c2920 comp rx=0 tx=0).stop 2024-03-28T23:30:11.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.616+0000 7fdb2649e640 1 -- 172.21.15.161:0/1183318047 reap_dead start 2024-03-28T23:30:11.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.617+0000 7f0570823640 1 -- 172.21.15.161:0/1352005790 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f056c1ef390 con 0x7f056c14feb0 2024-03-28T23:30:11.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.618+0000 7fdb27f01640 1 -- 172.21.15.161:0/1183318047 shutdown_connections 2024-03-28T23:30:11.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.618+0000 7fdb27f01640 1 -- 172.21.15.161:0/1183318047 >> 172.21.15.161:0/1183318047 conn(0x7fdb2006d8e0 msgr2=0x7fdb2006dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.618+0000 7f0562ffd640 1 -- 172.21.15.161:0/1352005790 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0564047020 con 0x7f056c14feb0 2024-03-28T23:30:11.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.618+0000 7f0570823640 1 -- 172.21.15.161:0/1352005790 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f056c04f4b0 con 0x7f056c14feb0 2024-03-28T23:30:11.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.619+0000 7f0562ffd640 1 --2- 172.21.15.161:0/1352005790 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f054c06b5a0 0x7f054c06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.620+0000 7fdb27f01640 1 -- 172.21.15.161:0/1183318047 shutdown_connections 2024-03-28T23:30:11.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.620+0000 7fdb27f01640 1 -- 172.21.15.161:0/1183318047 wait complete. 2024-03-28T23:30:11.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.619+0000 7f0562ffd640 1 -- 172.21.15.161:0/1352005790 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f05640b1060 con 0x7f056c14feb0 2024-03-28T23:30:11.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.619+0000 7f0562ffd640 1 --2- 172.21.15.161:0/1352005790 >> [v2:172.21.15.170:6800/1557068995,v1:172.21.15.170:6801/1557068995] conn(0x7f054c074d10 0x7f054c077150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.619+0000 7f0562ffd640 1 -- 172.21.15.161:0/1352005790 --> [v2:172.21.15.170:6800/1557068995,v1:172.21.15.170:6801/1557068995] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f056403ce50 con 0x7f054c074d10 2024-03-28T23:30:11.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.619+0000 7f0562ffd640 1 -- 172.21.15.161:0/1352005790 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f05640ac8c0 con 0x7f056c14feb0 2024-03-28T23:30:11.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.620+0000 7f056a59c640 1 --2- 172.21.15.161:0/1352005790 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f054c06b5a0 0x7f054c06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.620+0000 7f056a59c640 1 --2- 172.21.15.161:0/1352005790 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f054c06b5a0 0x7f054c06da60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f055c000f60 tx=0x7f055c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.620+0000 7f0563fff640 1 --2- 172.21.15.161:0/1352005790 >> [v2:172.21.15.170:6800/1557068995,v1:172.21.15.170:6801/1557068995] conn(0x7f054c074d10 0x7f054c077150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.621+0000 7fe58c48f640 1 -- 172.21.15.161:0/3292470588 --> [v2:172.21.15.161:6824/3880429893,v1:172.21.15.161:6825/3880429893] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe5840aaa10 con 0x7fe568074d10 2024-03-28T23:30:11.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.621+0000 7fe57affd640 1 -- 172.21.15.161:0/3292470588 <== osd.3 v2:172.21.15.161:6824/3880429893 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe5840aaa10 con 0x7fe568074d10 2024-03-28T23:30:11.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.622+0000 7f0563fff640 1 --2- 172.21.15.161:0/1352005790 >> [v2:172.21.15.170:6800/1557068995,v1:172.21.15.170:6801/1557068995] conn(0x7f054c074d10 0x7f054c077150 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.622+0000 7fe58c48f640 1 -- 172.21.15.161:0/3292470588 >> [v2:172.21.15.161:6824/3880429893,v1:172.21.15.161:6825/3880429893] conn(0x7fe568074d10 msgr2=0x7fe568077150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.622+0000 7fe58c48f640 1 --2- 172.21.15.161:0/3292470588 >> [v2:172.21.15.161:6824/3880429893,v1:172.21.15.161:6825/3880429893] conn(0x7fe568074d10 0x7fe568077150 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.622+0000 7fe58c48f640 1 -- 172.21.15.161:0/3292470588 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe56806b5a0 msgr2=0x7fe56806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.622+0000 7fe58c48f640 1 --2- 172.21.15.161:0/3292470588 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe56806b5a0 0x7fe56806da60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe5740017f0 tx=0x7fe574009040 comp rx=0 tx=0).stop 2024-03-28T23:30:11.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.622+0000 7fe58c48f640 1 -- 172.21.15.161:0/3292470588 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe584151fd0 msgr2=0x7fe58414a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.622+0000 7fe58c48f640 1 --2- 172.21.15.161:0/3292470588 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe584151fd0 0x7fe58414a7a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe57c0089d0 tx=0x7fe57c00b410 comp rx=0 tx=0).stop 2024-03-28T23:30:11.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.622+0000 7fe58aa2c640 1 -- 172.21.15.161:0/3292470588 reap_dead start 2024-03-28T23:30:11.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.622+0000 7fe58c48f640 1 -- 172.21.15.161:0/3292470588 shutdown_connections 2024-03-28T23:30:11.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.622+0000 7fe58c48f640 1 -- 172.21.15.161:0/3292470588 >> 172.21.15.161:0/3292470588 conn(0x7fe58406d8e0 msgr2=0x7fe58414d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.623+0000 7f0562ffd640 1 -- 172.21.15.161:0/1352005790 <== osd.4 v2:172.21.15.170:6800/1557068995 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f056403ce50 con 0x7f054c074d10 2024-03-28T23:30:11.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.623+0000 7fe58c48f640 1 -- 172.21.15.161:0/3292470588 shutdown_connections 2024-03-28T23:30:11.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.623+0000 7fe58c48f640 1 -- 172.21.15.161:0/3292470588 wait complete. 2024-03-28T23:30:11.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.630+0000 7f218d0ed640 1 Processor -- start 2024-03-28T23:30:11.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.630+0000 7f218d0ed640 1 -- start start 2024-03-28T23:30:11.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.631+0000 7f218d0ed640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f21880c3510 0x7f21881596c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.631+0000 7f218d0ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2188071e70 0x7f218815d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.631+0000 7f218d0ed640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21880728f0 con 0x7f218806e7f0 2024-03-28T23:30:11.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.631+0000 7f218d0ed640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f2188159c00 con 0x7f21880c3510 2024-03-28T23:30:11.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.631+0000 7f218d0ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2188072330 con 0x7f2188071e70 2024-03-28T23:30:11.661 INFO:teuthology.orchestra.run.smithi161.stdout:55834574856 2024-03-28T23:30:11.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.635+0000 7f218759e640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f218806e7f0 0x7f218806c160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:46102/0 (socket says 172.21.15.161:46102) 2024-03-28T23:30:11.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.635+0000 7f218759e640 1 -- 172.21.15.161:0/2658554261 learned_addr learned my addr 172.21.15.161:0/2658554261 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:11.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.635+0000 7f218659c640 1 --2- 172.21.15.161:0/2658554261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2188071e70 0x7f218815d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.636+0000 7f2186d9d640 1 --2- 172.21.15.161:0/2658554261 >> v2:172.21.15.170:3300/0 conn(0x7f21880c3510 0x7f21881596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.662 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-28T23:30:11.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.636+0000 7f2185d9b640 1 -- 172.21.15.161:0/2658554261 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3927023614 0 0) 0x7f21880728f0 con 0x7f218806e7f0 2024-03-28T23:30:11.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.636+0000 7f2185d9b640 1 -- 172.21.15.161:0/2658554261 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2170003520 con 0x7f218806e7f0 2024-03-28T23:30:11.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.636+0000 7f2185d9b640 1 -- 172.21.15.161:0/2658554261 <== mon.1 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4266519384 0 0) 0x7f2170003520 con 0x7f218806e7f0 2024-03-28T23:30:11.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.637+0000 7f2185d9b640 1 -- 172.21.15.161:0/2658554261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2188071e70 msgr2=0x7f218815d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.637+0000 7f2185d9b640 1 --2- 172.21.15.161:0/2658554261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2188071e70 0x7f218815d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.637+0000 7f2185d9b640 1 -- 172.21.15.161:0/2658554261 >> v2:172.21.15.170:3300/0 conn(0x7f21880c3510 msgr2=0x7f21881596c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.637+0000 7f2185d9b640 1 --2- 172.21.15.161:0/2658554261 >> v2:172.21.15.170:3300/0 conn(0x7f21880c3510 0x7f21881596c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.637+0000 7f2185d9b640 1 -- 172.21.15.161:0/2658554261 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2188072610 con 0x7f218806e7f0 2024-03-28T23:30:11.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.637+0000 7f2185d9b640 1 -- 172.21.15.161:0/2658554261 <== mon.1 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f2178001670 con 0x7f218806e7f0 2024-03-28T23:30:11.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.638+0000 7f2185d9b640 1 -- 172.21.15.161:0/2658554261 >> v1:172.21.15.170:6789/0 conn(0x7f218806e7f0 legacy=0x7f218806c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.638+0000 7f2185d9b640 1 --2- 172.21.15.161:0/2658554261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2170003a90 0x7f2170003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.638+0000 7f2185d9b640 1 --2- 172.21.15.161:0/2658554261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2188071e70 0x7f2170004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.638+0000 7f2185d9b640 1 --2- 172.21.15.161:0/2658554261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2170004d90 0x7f2170005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.638+0000 7f2186d9d640 1 --2- 172.21.15.161:0/2658554261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2170003a90 0x7f2170003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.639+0000 7f218659c640 1 --2- 172.21.15.161:0/2658554261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2188071e70 0x7f2170004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.639+0000 7f2185d9b640 1 -- 172.21.15.161:0/2658554261 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2188072330 con 0x7f2170004d90 2024-03-28T23:30:11.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.639+0000 7f2185d9b640 1 -- 172.21.15.161:0/2658554261 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f217000bc20 con 0x7f2170003a90 2024-03-28T23:30:11.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.639+0000 7f2185d9b640 1 -- 172.21.15.161:0/2658554261 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f217000be20 con 0x7f2188071e70 2024-03-28T23:30:11.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.639+0000 7f218759e640 1 --2- 172.21.15.161:0/2658554261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2170004d90 0x7f2170005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.665 INFO:teuthology.orchestra.run.smithi161.stdout:55834574856 2024-03-28T23:30:11.666 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-28T23:30:11.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.641+0000 7f5e667fc640 1 -- 172.21.15.161:0/2848689575 --> [v2:172.21.15.170:6824/211299856,v1:172.21.15.170:6825/211299856] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5e60002dd0 con 0x7f5e6c074d10 2024-03-28T23:30:11.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.640+0000 7f218759e640 1 -- 172.21.15.161:0/2658554261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2188071e70 msgr2=0x7f2170004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.640+0000 7f218759e640 1 --2- 172.21.15.161:0/2658554261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2188071e70 0x7f2170004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.640+0000 7f218759e640 1 -- 172.21.15.161:0/2658554261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2170003a90 msgr2=0x7f2170003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.640+0000 7f218759e640 1 --2- 172.21.15.161:0/2658554261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2170003a90 0x7f2170003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.640+0000 7f218759e640 1 -- 172.21.15.161:0/2658554261 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2170003520 con 0x7f2170004d90 2024-03-28T23:30:11.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.641+0000 7f218759e640 1 --2- 172.21.15.161:0/2658554261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2170004d90 0x7f2170005210 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f21780050b0 tx=0x7f2170005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a4ceef818b17db server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.641+0000 7f5e7cff9640 1 -- 172.21.15.161:0/2848689575 <== osd.5 v2:172.21.15.170:6824/211299856 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5e60002dd0 con 0x7f5e6c074d10 2024-03-28T23:30:11.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.641+0000 7f2185d9b640 1 -- 172.21.15.161:0/2658554261 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2178004c80 con 0x7f2170004d90 2024-03-28T23:30:11.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.641+0000 7f2185d9b640 1 -- 172.21.15.161:0/2658554261 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2178005bb0 con 0x7f2170004d90 2024-03-28T23:30:11.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.641+0000 7f2185d9b640 1 -- 172.21.15.161:0/2658554261 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21780080d0 con 0x7f2170004d90 2024-03-28T23:30:11.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.642+0000 7f75737fe640 1 -- 172.21.15.161:0/382349642 --> [v2:172.21.15.161:6816/4264782123,v1:172.21.15.161:6817/4264782123] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f758804f4b0 con 0x7f756c074d50 2024-03-28T23:30:11.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.642+0000 7f154010d640 1 -- 172.21.15.161:0/2371161997 --> [v2:172.21.15.170:6808/844763145,v1:172.21.15.170:6809/844763145] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1520002dd0 con 0x7f151c074d40 2024-03-28T23:30:11.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.642+0000 7f5e667fc640 1 -- 172.21.15.161:0/2848689575 >> [v2:172.21.15.170:6824/211299856,v1:172.21.15.170:6825/211299856] conn(0x7f5e6c074d10 msgr2=0x7f5e6c077150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.642+0000 7f5e667fc640 1 --2- 172.21.15.161:0/2848689575 >> [v2:172.21.15.170:6824/211299856,v1:172.21.15.170:6825/211299856] conn(0x7f5e6c074d10 0x7f5e6c077150 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.642+0000 7f757dffb640 1 -- 172.21.15.161:0/382349642 <== osd.0 v2:172.21.15.161:6816/4264782123 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f758804f4b0 con 0x7f756c074d50 2024-03-28T23:30:11.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.642+0000 7f5e667fc640 1 -- 172.21.15.161:0/2848689575 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5e6c06b5a0 msgr2=0x7f5e6c06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.642+0000 7f5e667fc640 1 --2- 172.21.15.161:0/2848689575 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5e6c06b5a0 0x7f5e6c06da60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5e70035000 tx=0x7f5e70009030 comp rx=0 tx=0).stop 2024-03-28T23:30:11.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.642+0000 7f5e667fc640 1 -- 172.21.15.161:0/2848689575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e800c5690 msgr2=0x7f5e801e8040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.642+0000 7f5e667fc640 1 --2- 172.21.15.161:0/2848689575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e800c5690 0x7f5e801e8040 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5e7800e8d0 tx=0x7f5e7800ed90 comp rx=0 tx=0).stop 2024-03-28T23:30:11.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.642+0000 7f5e84c08640 1 -- 172.21.15.161:0/2848689575 reap_dead start 2024-03-28T23:30:11.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f152e7fc640 1 -- 172.21.15.161:0/2371161997 <== osd.6 v2:172.21.15.170:6808/844763145 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1520002dd0 con 0x7f151c074d40 2024-03-28T23:30:11.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7fa55f72c640 1 -- 172.21.15.161:0/2053581074 --> [v2:172.21.15.161:6800/3551918247,v1:172.21.15.161:6801/3551918247] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa55804f4b0 con 0x7fa53c074ac0 2024-03-28T23:30:11.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f5e667fc640 1 -- 172.21.15.161:0/2848689575 shutdown_connections 2024-03-28T23:30:11.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f5e667fc640 1 -- 172.21.15.161:0/2848689575 >> 172.21.15.161:0/2848689575 conn(0x7f5e800a8820 msgr2=0x7f5e8006d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f218d0ed640 1 -- 172.21.15.161:0/2658554261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2170004d90 msgr2=0x7f2170005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f218d0ed640 1 --2- 172.21.15.161:0/2658554261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2170004d90 0x7f2170005210 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f21780050b0 tx=0x7f2170005830 comp rx=0 tx=0).stop 2024-03-28T23:30:11.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f218759e640 1 -- 172.21.15.161:0/2658554261 reap_dead start 2024-03-28T23:30:11.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f75737fe640 1 -- 172.21.15.161:0/382349642 >> [v2:172.21.15.161:6816/4264782123,v1:172.21.15.161:6817/4264782123] conn(0x7f756c074d50 msgr2=0x7f756c077190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f5e667fc640 1 -- 172.21.15.161:0/2848689575 shutdown_connections 2024-03-28T23:30:11.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f75737fe640 1 --2- 172.21.15.161:0/382349642 >> [v2:172.21.15.161:6816/4264782123,v1:172.21.15.161:6817/4264782123] conn(0x7f756c074d50 0x7f756c077190 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f5e667fc640 1 -- 172.21.15.161:0/2848689575 wait complete. 2024-03-28T23:30:11.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7fa54dffb640 1 -- 172.21.15.161:0/2053581074 <== osd.1 v2:172.21.15.161:6800/3551918247 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa55804f4b0 con 0x7fa53c074ac0 2024-03-28T23:30:11.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f218d0ed640 1 -- 172.21.15.161:0/2658554261 shutdown_connections 2024-03-28T23:30:11.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f218d0ed640 1 -- 172.21.15.161:0/2658554261 >> 172.21.15.161:0/2658554261 conn(0x7f21880c3cf0 msgr2=0x7f21880c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f75737fe640 1 -- 172.21.15.161:0/382349642 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f756c06b5d0 msgr2=0x7f756c06da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f75737fe640 1 --2- 172.21.15.161:0/382349642 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f756c06b5d0 0x7f756c06da90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f75880c2530 tx=0x7f7578009190 comp rx=0 tx=0).stop 2024-03-28T23:30:11.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f75737fe640 1 -- 172.21.15.161:0/382349642 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f758814fcf0 msgr2=0x7f75881503f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f75737fe640 1 --2- 172.21.15.161:0/382349642 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f758814fcf0 0x7f75881503f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f758400c800 tx=0x7f758400ccc0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f758dcbb640 1 -- 172.21.15.161:0/382349642 reap_dead start 2024-03-28T23:30:11.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f154010d640 1 -- 172.21.15.161:0/2371161997 >> [v2:172.21.15.170:6808/844763145,v1:172.21.15.170:6809/844763145] conn(0x7f151c074d40 msgr2=0x7f151c077180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f154010d640 1 --2- 172.21.15.161:0/2371161997 >> [v2:172.21.15.170:6808/844763145,v1:172.21.15.170:6809/844763145] conn(0x7f151c074d40 0x7f151c077180 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f218d0ed640 1 -- 172.21.15.161:0/2658554261 shutdown_connections 2024-03-28T23:30:11.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f218d0ed640 1 -- 172.21.15.161:0/2658554261 wait complete. 2024-03-28T23:30:11.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f154010d640 1 -- 172.21.15.161:0/2371161997 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f151c06b5d0 msgr2=0x7f151c06da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f154010d640 1 --2- 172.21.15.161:0/2371161997 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f151c06b5d0 0x7f151c06da90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f153403a000 tx=0x7f1534034000 comp rx=0 tx=0).stop 2024-03-28T23:30:11.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f154010d640 1 -- 172.21.15.161:0/2371161997 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15380c2bf0 msgr2=0x7f1538145620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f154010d640 1 --2- 172.21.15.161:0/2371161997 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15380c2bf0 0x7f1538145620 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f153000ba80 tx=0x7f153000bf40 comp rx=0 tx=0).stop 2024-03-28T23:30:11.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7f153e6aa640 1 -- 172.21.15.161:0/2371161997 reap_dead start 2024-03-28T23:30:11.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.643+0000 7fa55f72c640 1 -- 172.21.15.161:0/2053581074 >> [v2:172.21.15.161:6800/3551918247,v1:172.21.15.161:6801/3551918247] conn(0x7fa53c074ac0 msgr2=0x7fa53c076f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.644+0000 7fa55f72c640 1 --2- 172.21.15.161:0/2053581074 >> [v2:172.21.15.161:6800/3551918247,v1:172.21.15.161:6801/3551918247] conn(0x7fa53c074ac0 0x7fa53c076f00 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.644+0000 7f75737fe640 1 -- 172.21.15.161:0/382349642 shutdown_connections 2024-03-28T23:30:11.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.644+0000 7f75737fe640 1 -- 172.21.15.161:0/382349642 >> 172.21.15.161:0/382349642 conn(0x7f75880c3cf0 msgr2=0x7f7588153d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.644+0000 7f154010d640 1 -- 172.21.15.161:0/2371161997 shutdown_connections 2024-03-28T23:30:11.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.644+0000 7f154010d640 1 -- 172.21.15.161:0/2371161997 >> 172.21.15.161:0/2371161997 conn(0x7f15380c3b70 msgr2=0x7f1538066eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.644+0000 7fa55f72c640 1 -- 172.21.15.161:0/2053581074 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa53c06b4d0 msgr2=0x7fa53c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.644+0000 7fa55f72c640 1 --2- 172.21.15.161:0/2053581074 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa53c06b4d0 0x7fa53c06d990 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa55003e000 tx=0x7fa550038000 comp rx=0 tx=0).stop 2024-03-28T23:30:11.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.644+0000 7fa55f72c640 1 -- 172.21.15.161:0/2053581074 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa558072ee0 msgr2=0x7fa5581db440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.644+0000 7fa55f72c640 1 --2- 172.21.15.161:0/2053581074 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa558072ee0 0x7fa5581db440 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa55400c3f0 tx=0x7fa55400c8b0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.644+0000 7fa55dcc9640 1 -- 172.21.15.161:0/2053581074 reap_dead start 2024-03-28T23:30:11.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.644+0000 7fa55f72c640 1 -- 172.21.15.161:0/2053581074 shutdown_connections 2024-03-28T23:30:11.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.644+0000 7fa55f72c640 1 -- 172.21.15.161:0/2053581074 >> 172.21.15.161:0/2053581074 conn(0x7fa5580c88b0 msgr2=0x7fa558151fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.644+0000 7f154010d640 1 -- 172.21.15.161:0/2371161997 shutdown_connections 2024-03-28T23:30:11.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.644+0000 7f154010d640 1 -- 172.21.15.161:0/2371161997 wait complete. 2024-03-28T23:30:11.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.644+0000 7f75737fe640 1 -- 172.21.15.161:0/382349642 shutdown_connections 2024-03-28T23:30:11.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.644+0000 7fa55f72c640 1 -- 172.21.15.161:0/2053581074 shutdown_connections 2024-03-28T23:30:11.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.644+0000 7fa55f72c640 1 -- 172.21.15.161:0/2053581074 wait complete. 2024-03-28T23:30:11.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.644+0000 7f75737fe640 1 -- 172.21.15.161:0/382349642 wait complete. 2024-03-28T23:30:11.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.645+0000 7f218d0ed640 1 Processor -- start 2024-03-28T23:30:11.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.645+0000 7f218d0ed640 1 -- start start 2024-03-28T23:30:11.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.646+0000 7f218d0ed640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2170004d90 0x7f2188159f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.646+0000 7f218d0ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21880ad1e0 0x7f2188152450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.646+0000 7f218d0ed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2188150650 0x7f218806b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.647+0000 7f218659c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21880ad1e0 0x7f2188152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.647+0000 7f2186d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2170004d90 0x7f2188159f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.647+0000 7f2186d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2170004d90 0x7f2188159f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54172/0 (socket says 172.21.15.161:54172) 2024-03-28T23:30:11.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.647+0000 7f218659c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21880ad1e0 0x7f2188152450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:36872/0 (socket says 172.21.15.161:36872) 2024-03-28T23:30:11.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.647+0000 7f218d0ed640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f218806c870 con 0x7f2170004d90 2024-03-28T23:30:11.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.647+0000 7f218d0ed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f218815d030 con 0x7f2188150650 2024-03-28T23:30:11.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.647+0000 7f218d0ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f218806c9f0 con 0x7f21880ad1e0 2024-03-28T23:30:11.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.647+0000 7f2186d9d640 1 -- 172.21.15.161:0/954927601 learned_addr learned my addr 172.21.15.161:0/954927601 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:11.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.647+0000 7f218759e640 1 --2- 172.21.15.161:0/954927601 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2188150650 0x7f218806b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.647+0000 7f2186d9d640 1 -- 172.21.15.161:0/954927601 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21880ad1e0 msgr2=0x7f2188152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.647+0000 7f2186d9d640 1 --2- 172.21.15.161:0/954927601 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21880ad1e0 0x7f2188152450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.648+0000 7f2186d9d640 1 -- 172.21.15.161:0/954927601 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2188150650 msgr2=0x7f218806b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.648+0000 7f2186d9d640 1 --2- 172.21.15.161:0/954927601 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2188150650 0x7f218806b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.648+0000 7f2186d9d640 1 -- 172.21.15.161:0/954927601 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f218815a480 con 0x7f2170004d90 2024-03-28T23:30:11.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.648+0000 7f2186d9d640 1 --2- 172.21.15.161:0/954927601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2170004d90 0x7f2188159f40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f218000feb0 tx=0x7f218000d750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.650+0000 7f216f7fe640 1 -- 172.21.15.161:0/954927601 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f218001c070 con 0x7f2170004d90 2024-03-28T23:30:11.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.650+0000 7f218d0ed640 1 -- 172.21.15.161:0/954927601 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21881519a0 con 0x7f2170004d90 2024-03-28T23:30:11.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.650+0000 7f216f7fe640 1 -- 172.21.15.161:0/954927601 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2180017410 con 0x7f2170004d90 2024-03-28T23:30:11.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.650+0000 7f216f7fe640 1 -- 172.21.15.161:0/954927601 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f218001c070 con 0x7f2170004d90 2024-03-28T23:30:11.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.650+0000 7f218d0ed640 1 -- 172.21.15.161:0/954927601 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2188151f60 con 0x7f2170004d90 2024-03-28T23:30:11.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.652+0000 7f0570823640 1 -- 172.21.15.161:0/1352005790 --> [v2:172.21.15.170:6800/1557068995,v1:172.21.15.170:6801/1557068995] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f056c1f2300 con 0x7f054c074d10 2024-03-28T23:30:11.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.653+0000 7f0562ffd640 1 -- 172.21.15.161:0/1352005790 <== osd.4 v2:172.21.15.170:6800/1557068995 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f056c1f2300 con 0x7f054c074d10 2024-03-28T23:30:11.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.653+0000 7f216f7fe640 1 -- 172.21.15.161:0/954927601 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2180005d10 con 0x7f2170004d90 2024-03-28T23:30:11.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.653+0000 7f216f7fe640 1 --2- 172.21.15.161:0/954927601 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f217406b6d0 0x7f217406db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.654+0000 7f216f7fe640 1 -- 172.21.15.161:0/954927601 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f218008abe0 con 0x7f2170004d90 2024-03-28T23:30:11.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.654+0000 7f218659c640 1 --2- 172.21.15.161:0/954927601 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f217406b6d0 0x7f217406db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.654+0000 7f0570823640 1 -- 172.21.15.161:0/1352005790 >> [v2:172.21.15.170:6800/1557068995,v1:172.21.15.170:6801/1557068995] conn(0x7f054c074d10 msgr2=0x7f054c077150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.654+0000 7f0570823640 1 --2- 172.21.15.161:0/1352005790 >> [v2:172.21.15.170:6800/1557068995,v1:172.21.15.170:6801/1557068995] conn(0x7f054c074d10 0x7f054c077150 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.654+0000 7f0570823640 1 -- 172.21.15.161:0/1352005790 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f054c06b5a0 msgr2=0x7f054c06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.654+0000 7f218659c640 1 --2- 172.21.15.161:0/954927601 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f217406b6d0 0x7f217406db90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2188155120 tx=0x7f217c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.654+0000 7f0570823640 1 --2- 172.21.15.161:0/1352005790 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f054c06b5a0 0x7f054c06da60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f055c000f60 tx=0x7f055c009040 comp rx=0 tx=0).stop 2024-03-28T23:30:11.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.654+0000 7f0570823640 1 -- 172.21.15.161:0/1352005790 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f056c14feb0 msgr2=0x7f056c150310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.654+0000 7f0570823640 1 --2- 172.21.15.161:0/1352005790 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f056c14feb0 0x7f056c150310 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0564009d90 tx=0x7f0564004a40 comp rx=0 tx=0).stop 2024-03-28T23:30:11.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.654+0000 7f056ad9d640 1 -- 172.21.15.161:0/1352005790 reap_dead start 2024-03-28T23:30:11.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.655+0000 7f0570823640 1 -- 172.21.15.161:0/1352005790 shutdown_connections 2024-03-28T23:30:11.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.655+0000 7f0570823640 1 -- 172.21.15.161:0/1352005790 >> 172.21.15.161:0/1352005790 conn(0x7f056c0c4250 msgr2=0x7f056c0c6400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.655+0000 7f0570823640 1 -- 172.21.15.161:0/1352005790 shutdown_connections 2024-03-28T23:30:11.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.655+0000 7f0570823640 1 -- 172.21.15.161:0/1352005790 wait complete. 2024-03-28T23:30:11.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.658+0000 7f218d0ed640 1 --2- 172.21.15.161:0/954927601 >> [v2:172.21.15.170:6816/3081832568,v1:172.21.15.170:6817/3081832568] conn(0x7f218815c7d0 0x7f218806b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.658+0000 7f218d0ed640 1 -- 172.21.15.161:0/954927601 --> [v2:172.21.15.170:6816/3081832568,v1:172.21.15.170:6817/3081832568] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f218804f4b0 con 0x7f218815c7d0 2024-03-28T23:30:11.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.659+0000 7f218759e640 1 --2- 172.21.15.161:0/954927601 >> [v2:172.21.15.170:6816/3081832568,v1:172.21.15.170:6817/3081832568] conn(0x7f218815c7d0 0x7f218806b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.659+0000 7f218759e640 1 --2- 172.21.15.161:0/954927601 >> [v2:172.21.15.170:6816/3081832568,v1:172.21.15.170:6817/3081832568] conn(0x7f218815c7d0 0x7f218806b690 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.660+0000 7f216f7fe640 1 -- 172.21.15.161:0/954927601 <== osd.7 v2:172.21.15.170:6816/3081832568 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f218804f4b0 con 0x7f218815c7d0 2024-03-28T23:30:11.695 INFO:teuthology.orchestra.run.smithi161.stdout:55834574857 2024-03-28T23:30:11.695 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-28T23:30:11.696 INFO:teuthology.orchestra.run.smithi161.stdout:55834574856 2024-03-28T23:30:11.696 INFO:teuthology.orchestra.run.smithi161.stdout:55834574857 2024-03-28T23:30:11.697 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-28T23:30:11.697 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-28T23:30:11.699 INFO:teuthology.orchestra.run.smithi161.stdout:55834574856 2024-03-28T23:30:11.699 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-28T23:30:11.700 INFO:teuthology.orchestra.run.smithi161.stdout:55834574856 2024-03-28T23:30:11.700 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-28T23:30:11.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.686+0000 7f218d0ed640 1 -- 172.21.15.161:0/954927601 --> [v2:172.21.15.170:6816/3081832568,v1:172.21.15.170:6817/3081832568] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2188152990 con 0x7f218815c7d0 2024-03-28T23:30:11.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.687+0000 7f216f7fe640 1 -- 172.21.15.161:0/954927601 <== osd.7 v2:172.21.15.170:6816/3081832568 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2188152990 con 0x7f218815c7d0 2024-03-28T23:30:11.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.687+0000 7f218d0ed640 1 -- 172.21.15.161:0/954927601 >> [v2:172.21.15.170:6816/3081832568,v1:172.21.15.170:6817/3081832568] conn(0x7f218815c7d0 msgr2=0x7f218806b690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.687+0000 7f218d0ed640 1 --2- 172.21.15.161:0/954927601 >> [v2:172.21.15.170:6816/3081832568,v1:172.21.15.170:6817/3081832568] conn(0x7f218815c7d0 0x7f218806b690 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.687+0000 7f218d0ed640 1 -- 172.21.15.161:0/954927601 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f217406b6d0 msgr2=0x7f217406db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.687+0000 7f218d0ed640 1 --2- 172.21.15.161:0/954927601 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f217406b6d0 0x7f217406db90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2188155120 tx=0x7f217c008040 comp rx=0 tx=0).stop 2024-03-28T23:30:11.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.687+0000 7f218d0ed640 1 -- 172.21.15.161:0/954927601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2170004d90 msgr2=0x7f2188159f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.687+0000 7f218d0ed640 1 --2- 172.21.15.161:0/954927601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2170004d90 0x7f2188159f40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f218000feb0 tx=0x7f218000d750 comp rx=0 tx=0).stop 2024-03-28T23:30:11.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.687+0000 7f218759e640 1 -- 172.21.15.161:0/954927601 reap_dead start 2024-03-28T23:30:11.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.687+0000 7f218d0ed640 1 -- 172.21.15.161:0/954927601 shutdown_connections 2024-03-28T23:30:11.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.687+0000 7f218d0ed640 1 -- 172.21.15.161:0/954927601 >> 172.21.15.161:0/954927601 conn(0x7f21880c3cf0 msgr2=0x7f218815aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.687+0000 7f218d0ed640 1 -- 172.21.15.161:0/954927601 shutdown_connections 2024-03-28T23:30:11.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.687+0000 7f218d0ed640 1 -- 172.21.15.161:0/954927601 wait complete. 2024-03-28T23:30:11.705 INFO:teuthology.orchestra.run.smithi161.stdout:55834574856 2024-03-28T23:30:11.705 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-28T23:30:11.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.843+0000 7f9cab2bc640 1 Processor -- start 2024-03-28T23:30:11.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.842+0000 7f456c0ae640 1 Processor -- start 2024-03-28T23:30:11.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.842+0000 7f456c0ae640 1 -- start start 2024-03-28T23:30:11.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.843+0000 7f9cab2bc640 1 -- start start 2024-03-28T23:30:11.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.844+0000 7f9cab2bc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9ca4153300 0x7f9ca40c2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.844+0000 7f9cab2bc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ca40c0f60 0x7f9ca41593b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.844+0000 7f9cab2bc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9ca415a8a0 0x7f9ca406d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.844+0000 7f9cab2bc640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f9ca40c20e0 con 0x7f9ca415a8a0 2024-03-28T23:30:11.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.844+0000 7f456c0ae640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f455c006e90 0x7f455c007340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.844+0000 7f456c0ae640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f455c0d9c50 con 0x7f455c04b100 2024-03-28T23:30:11.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.844+0000 7f456c0ae640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f455c0d9f30 con 0x7f455c00bd40 2024-03-28T23:30:11.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.844+0000 7f456c0ae640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f455c0da210 con 0x7f455c006e90 2024-03-28T23:30:11.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.844+0000 7f4569e4a640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f455c00bd40 0x7f455c046b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:46114/0 (socket says 172.21.15.161:46114) 2024-03-28T23:30:11.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.844+0000 7f4569e4a640 1 -- 172.21.15.161:0/3579194664 learned_addr learned my addr 172.21.15.161:0/3579194664 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:11.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.844+0000 7f456a64b640 1 --2- 172.21.15.161:0/3579194664 >> v2:172.21.15.161:3300/0 conn(0x7f455c006e90 0x7f455c007340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.844+0000 7f456a64b640 1 -- 172.21.15.161:0/3579194664 >> v1:172.21.15.170:6789/0 conn(0x7f455c00bd40 legacy=0x7f455c046b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.844+0000 7f456a64b640 1 -- 172.21.15.161:0/3579194664 >> v1:172.21.15.161:6789/0 conn(0x7f455c04b100 legacy=0x7f455c0d3aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T23:30:11.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.845+0000 7f456a64b640 1 -- 172.21.15.161:0/3579194664 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f455c0da4f0 con 0x7f455c006e90 2024-03-28T23:30:11.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.844+0000 7f9cab2bc640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f9ca40c14d0 con 0x7f9ca4153300 2024-03-28T23:30:11.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.844+0000 7f9cab2bc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9ca40c17b0 con 0x7f9ca40c0f60 2024-03-28T23:30:11.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.845+0000 7f9ca9ab9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ca40c0f60 0x7f9ca41593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.846+0000 7f456a64b640 1 --2- 172.21.15.161:0/3579194664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f455c006e90 0x7f455c007340 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4564062d10 tx=0x7f456406aef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc7ee5a32c347e6d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.846+0000 7f4568e48640 1 -- 172.21.15.161:0/3579194664 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45640bc070 con 0x7f455c006e90 2024-03-28T23:30:11.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.846+0000 7f4568e48640 1 -- 172.21.15.161:0/3579194664 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45640b7440 con 0x7f455c006e90 2024-03-28T23:30:11.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.846+0000 7f4568e48640 1 -- 172.21.15.161:0/3579194664 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45640b78b0 con 0x7f455c006e90 2024-03-28T23:30:11.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.845+0000 7f9ca9ab9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ca40c0f60 0x7f9ca41593b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:36874/0 (socket says 172.21.15.161:36874) 2024-03-28T23:30:11.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.845+0000 7f9ca9ab9640 1 -- 172.21.15.161:0/2318107549 learned_addr learned my addr 172.21.15.161:0/2318107549 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:11.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.845+0000 7f9caa2ba640 1 --2- 172.21.15.161:0/2318107549 >> v2:172.21.15.170:3300/0 conn(0x7f9ca4153300 0x7f9ca40c2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.845+0000 7f9ca9ab9640 1 -- 172.21.15.161:0/2318107549 >> v2:172.21.15.170:3300/0 conn(0x7f9ca4153300 msgr2=0x7f9ca40c2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.845+0000 7f9ca9ab9640 1 --2- 172.21.15.161:0/2318107549 >> v2:172.21.15.170:3300/0 conn(0x7f9ca4153300 0x7f9ca40c2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.845+0000 7f9ca9ab9640 1 -- 172.21.15.161:0/2318107549 >> v2:172.21.15.161:3300/0 conn(0x7f9ca415a8a0 msgr2=0x7f9ca406d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.845+0000 7f9caaabb640 1 --2- 172.21.15.161:0/2318107549 >> v2:172.21.15.161:3300/0 conn(0x7f9ca415a8a0 0x7f9ca406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.846+0000 7f9ca9ab9640 1 --2- 172.21.15.161:0/2318107549 >> v2:172.21.15.161:3300/0 conn(0x7f9ca415a8a0 0x7f9ca406d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.846+0000 7f9ca9ab9640 1 -- 172.21.15.161:0/2318107549 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca40c1a90 con 0x7f9ca40c0f60 2024-03-28T23:30:11.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.847+0000 7f9caaabb640 1 --2- 172.21.15.161:0/2318107549 >> v2:172.21.15.161:3300/0 conn(0x7f9ca415a8a0 0x7f9ca406d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:11.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.847+0000 7f9ca9ab9640 1 --2- 172.21.15.161:0/2318107549 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ca40c0f60 0x7f9ca41593b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9ca406e040 tx=0x7f9c9c02bf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6d9f179c194b88d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.847+0000 7f456c0ae640 1 -- 172.21.15.161:0/3579194664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f455c006e90 msgr2=0x7f455c007340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.847+0000 7f456c0ae640 1 --2- 172.21.15.161:0/3579194664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f455c006e90 0x7f455c007340 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4564062d10 tx=0x7f456406aef0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.848+0000 7f456c0ae640 1 -- 172.21.15.161:0/3579194664 shutdown_connections 2024-03-28T23:30:11.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.848+0000 7f456c0ae640 1 --2- 172.21.15.161:0/3579194664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f455c006e90 0x7f455c007340 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.848+0000 7f456c0ae640 1 -- 172.21.15.161:0/3579194664 >> 172.21.15.161:0/3579194664 conn(0x7f455c048240 msgr2=0x7f455c048650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.848+0000 7f456c0ae640 1 -- 172.21.15.161:0/3579194664 shutdown_connections 2024-03-28T23:30:11.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.848+0000 7f456c0ae640 1 -- 172.21.15.161:0/3579194664 wait complete. 2024-03-28T23:30:11.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.850+0000 7f456c0ae640 1 Processor -- start 2024-03-28T23:30:11.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.850+0000 7f456c0ae640 1 -- start start 2024-03-28T23:30:11.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.851+0000 7f9ca92b8640 1 -- 172.21.15.161:0/2318107549 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c02caa0 con 0x7f9ca40c0f60 2024-03-28T23:30:11.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.851+0000 7f456c0ae640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f455c006e90 0x7f455c16f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.851+0000 7f456c0ae640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f455c00bd40 0x7f455c0d8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.851+0000 7f456c0ae640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f455c04b100 0x7f455c0ce900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.851+0000 7f456c0ae640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f455c0d6700 con 0x7f455c00bd40 2024-03-28T23:30:11.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.851+0000 7f456c0ae640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f455c0d9920 con 0x7f455c04b100 2024-03-28T23:30:11.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.851+0000 7f456c0ae640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f455c0d6880 con 0x7f455c006e90 2024-03-28T23:30:11.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.851+0000 7f4569649640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f455c00bd40 0x7f455c0d8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.851+0000 7f4569649640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f455c00bd40 0x7f455c0d8100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54204/0 (socket says 172.21.15.161:54204) 2024-03-28T23:30:11.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.852+0000 7f4569649640 1 -- 172.21.15.161:0/502774829 learned_addr learned my addr 172.21.15.161:0/502774829 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:11.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.852+0000 7f456a64b640 1 --2- 172.21.15.161:0/502774829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f455c04b100 0x7f455c0ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.852+0000 7f4569e4a640 1 --2- 172.21.15.161:0/502774829 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f455c006e90 0x7f455c16f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.852+0000 7f4569649640 1 -- 172.21.15.161:0/502774829 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f455c006e90 msgr2=0x7f455c16f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.852+0000 7f4569649640 1 --2- 172.21.15.161:0/502774829 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f455c006e90 0x7f455c16f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.852+0000 7f4569649640 1 -- 172.21.15.161:0/502774829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f455c04b100 msgr2=0x7f455c0ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.852+0000 7f4569649640 1 --2- 172.21.15.161:0/502774829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f455c04b100 0x7f455c0ce900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.852+0000 7f4569649640 1 -- 172.21.15.161:0/502774829 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f455c0d5330 con 0x7f455c00bd40 2024-03-28T23:30:11.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.851+0000 7f9ca92b8640 1 -- 172.21.15.161:0/2318107549 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c9c02cc40 con 0x7f9ca40c0f60 2024-03-28T23:30:11.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.852+0000 7f4569649640 1 --2- 172.21.15.161:0/502774829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f455c00bd40 0x7f455c0d8100 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f455400b440 tx=0x7f455400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.852+0000 7f9cab2bc640 1 -- 172.21.15.161:0/2318107549 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ca40c0f60 msgr2=0x7f9ca41593b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.852+0000 7f9cab2bc640 1 --2- 172.21.15.161:0/2318107549 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ca40c0f60 0x7f9ca41593b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9ca406e040 tx=0x7f9c9c02bf00 comp rx=0 tx=0).stop 2024-03-28T23:30:11.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.852+0000 7f9cab2bc640 1 -- 172.21.15.161:0/2318107549 shutdown_connections 2024-03-28T23:30:11.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.852+0000 7f9cab2bc640 1 --2- 172.21.15.161:0/2318107549 >> v2:172.21.15.161:3300/0 conn(0x7f9ca415a8a0 0x7f9ca406d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.852+0000 7f9cab2bc640 1 --2- 172.21.15.161:0/2318107549 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ca40c0f60 0x7f9ca41593b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.852+0000 7f9cab2bc640 1 --2- 172.21.15.161:0/2318107549 >> v2:172.21.15.170:3300/0 conn(0x7f9ca4153300 0x7f9ca40c2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.852+0000 7f9cab2bc640 1 -- 172.21.15.161:0/2318107549 >> 172.21.15.161:0/2318107549 conn(0x7f9ca40c71a0 msgr2=0x7f9ca40c3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.853+0000 7f9cab2bc640 1 -- 172.21.15.161:0/2318107549 shutdown_connections 2024-03-28T23:30:11.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.856+0000 7f455a7fc640 1 -- 172.21.15.161:0/502774829 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4554015020 con 0x7f455c00bd40 2024-03-28T23:30:11.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.856+0000 7f455a7fc640 1 -- 172.21.15.161:0/502774829 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4554011830 con 0x7f455c00bd40 2024-03-28T23:30:11.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.856+0000 7f455a7fc640 1 -- 172.21.15.161:0/502774829 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4554011c00 con 0x7f455c00bd40 2024-03-28T23:30:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.853+0000 7f9cab2bc640 1 -- 172.21.15.161:0/2318107549 wait complete. 2024-03-28T23:30:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.854+0000 7f9cab2bc640 1 Processor -- start 2024-03-28T23:30:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.854+0000 7f9cab2bc640 1 -- start start 2024-03-28T23:30:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.855+0000 7f9cab2bc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ca40c0f60 0x7f9ca4071ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.855+0000 7f9cab2bc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ca4153300 0x7f9ca4072570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.857+0000 7f9cab2bc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ca415a8a0 0x7f9ca4072ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.857+0000 7f9cab2bc640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca41536e0 con 0x7f9ca415a8a0 2024-03-28T23:30:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.857+0000 7f9cab2bc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca40c0b70 con 0x7f9ca40c0f60 2024-03-28T23:30:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.857+0000 7f9cab2bc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9ca4153860 con 0x7f9ca4153300 2024-03-28T23:30:11.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.857+0000 7f9caa2ba640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ca40c0f60 0x7f9ca4071ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.857+0000 7f9caa2ba640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ca40c0f60 0x7f9ca4071ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:39188/0 (socket says 172.21.15.161:39188) 2024-03-28T23:30:11.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.857+0000 7f9caa2ba640 1 -- 172.21.15.161:0/2385433305 learned_addr learned my addr 172.21.15.161:0/2385433305 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:11.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.858+0000 7f9caaabb640 1 --2- 172.21.15.161:0/2385433305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ca415a8a0 0x7f9ca4072ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.858+0000 7f9caa2ba640 1 -- 172.21.15.161:0/2385433305 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ca4153300 msgr2=0x7f9ca4072570 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T23:30:11.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.858+0000 7f9caa2ba640 1 --2- 172.21.15.161:0/2385433305 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ca4153300 0x7f9ca4072570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.858+0000 7f9caa2ba640 1 -- 172.21.15.161:0/2385433305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ca415a8a0 msgr2=0x7f9ca4072ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.858+0000 7f9caa2ba640 1 --2- 172.21.15.161:0/2385433305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ca415a8a0 0x7f9ca4072ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.858+0000 7f9caa2ba640 1 -- 172.21.15.161:0/2385433305 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca413d210 con 0x7f9ca40c0f60 2024-03-28T23:30:11.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.858+0000 7f9caaabb640 1 --2- 172.21.15.161:0/2385433305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ca415a8a0 0x7f9ca4072ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:11.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.859+0000 7f9caa2ba640 1 --2- 172.21.15.161:0/2385433305 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ca40c0f60 0x7f9ca4071ec0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9ca000e8c0 tx=0x7f9ca000ed80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.860+0000 7f9c9affd640 1 -- 172.21.15.161:0/2385433305 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca0017070 con 0x7f9ca40c0f60 2024-03-28T23:30:11.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.860+0000 7f9cab2bc640 1 -- 172.21.15.161:0/2385433305 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca413dc90 con 0x7f9ca40c0f60 2024-03-28T23:30:11.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.860+0000 7f9cab2bc640 1 -- 172.21.15.161:0/2385433305 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ca4072ff0 con 0x7f9ca40c0f60 2024-03-28T23:30:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.860+0000 7f9c9affd640 1 -- 172.21.15.161:0/2385433305 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ca000c940 con 0x7f9ca40c0f60 2024-03-28T23:30:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.860+0000 7f9c9affd640 1 -- 172.21.15.161:0/2385433305 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca000cd30 con 0x7f9ca40c0f60 2024-03-28T23:30:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.863+0000 7f9c9affd640 1 -- 172.21.15.161:0/2385433305 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9ca001a070 con 0x7f9ca40c0f60 2024-03-28T23:30:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.862+0000 7f456c0ae640 1 -- 172.21.15.161:0/502774829 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f455c02edb0 con 0x7f455c00bd40 2024-03-28T23:30:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.862+0000 7f456c0ae640 1 -- 172.21.15.161:0/502774829 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f455c02f370 con 0x7f455c00bd40 2024-03-28T23:30:11.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.863+0000 7f9c9affd640 1 --2- 172.21.15.161:0/2385433305 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9c9006b6e0 0x7f9c9006dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.864+0000 7f9c9affd640 1 -- 172.21.15.161:0/2385433305 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9ca008a500 con 0x7f9ca40c0f60 2024-03-28T23:30:11.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.864+0000 7f9ca9ab9640 1 --2- 172.21.15.161:0/2385433305 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9c9006b6e0 0x7f9c9006dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.865+0000 7f9ca9ab9640 1 --2- 172.21.15.161:0/2385433305 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9c9006b6e0 0x7f9c9006dba0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c035000 tx=0x7f9c9c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.865+0000 7f9cab2bc640 1 -- 172.21.15.161:0/2385433305 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c8c005290 con 0x7f9ca40c0f60 2024-03-28T23:30:11.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.865+0000 7f456c0ae640 1 -- 172.21.15.161:0/502774829 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f455c02f570 con 0x7f455c00bd40 2024-03-28T23:30:11.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.868+0000 7f455a7fc640 1 -- 172.21.15.161:0/502774829 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4554018020 con 0x7f455c00bd40 2024-03-28T23:30:11.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.868+0000 7f455a7fc640 1 --2- 172.21.15.161:0/502774829 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f454806b5d0 0x7f454806da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.870+0000 7f4569e4a640 1 --2- 172.21.15.161:0/502774829 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f454806b5d0 0x7f454806da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.872+0000 7f9c9affd640 1 -- 172.21.15.161:0/2385433305 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9ca00590b0 con 0x7f9ca40c0f60 2024-03-28T23:30:11.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.873+0000 7f4569e4a640 1 --2- 172.21.15.161:0/502774829 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f454806b5d0 0x7f454806da90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f456000c840 tx=0x7f4560009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.873+0000 7f455a7fc640 1 -- 172.21.15.161:0/502774829 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f455408aaa0 con 0x7f455c00bd40 2024-03-28T23:30:11.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.875+0000 7f455a7fc640 1 -- 172.21.15.161:0/502774829 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f455c02f570 con 0x7f455c00bd40 2024-03-28T23:30:11.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.919+0000 7fcddfe3b640 1 Processor -- start 2024-03-28T23:30:11.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.922+0000 7f26f1bad640 1 Processor -- start 2024-03-28T23:30:11.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.921+0000 7fcddfe3b640 1 -- start start 2024-03-28T23:30:11.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.922+0000 7fcddfe3b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdd8151fd0 0x7fcdd8072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.922+0000 7fcddfe3b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fcdd80aa960 0x7fcdd806b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.922+0000 7fcddfe3b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdd80c1eb0 con 0x7fcdd80aa4e0 2024-03-28T23:30:11.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.922+0000 7fcddfe3b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fcdd806bda0 con 0x7fcdd80aa960 2024-03-28T23:30:11.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.922+0000 7fcddfe3b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fcdd80c2190 con 0x7fcdd8151fd0 2024-03-28T23:30:11.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.922+0000 7f26f1bad640 1 -- start start 2024-03-28T23:30:11.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.923+0000 7fcdddbd7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdd8151fd0 0x7fcdd8072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.923+0000 7fcdde3d8640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fcdd80aa960 0x7fcdd806b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.923+0000 7fcdde3d8640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fcdd80aa960 0x7fcdd806b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:39196/0 (socket says 172.21.15.161:39196) 2024-03-28T23:30:11.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.923+0000 7fcdde3d8640 1 -- 172.21.15.161:0/2536260956 learned_addr learned my addr 172.21.15.161:0/2536260956 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:11.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.923+0000 7f26f1bad640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26e404b8b0 0x7f26e4048330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.923+0000 7f26f0bab640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f26e40d5940 0x7f26e404ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59608/0 (socket says 172.21.15.161:59608) 2024-03-28T23:30:11.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.924+0000 7f26f0bab640 1 -- 172.21.15.161:0/1201281700 learned_addr learned my addr 172.21.15.161:0/1201281700 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:11.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.924+0000 7f26ebfff640 1 --2- 172.21.15.161:0/1201281700 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26e404b8b0 0x7f26e4048330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.924+0000 7fcddcbd5640 1 -- 172.21.15.161:0/2536260956 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2659452445 0 0) 0x7fcdd80c1eb0 con 0x7fcdd80aa4e0 2024-03-28T23:30:11.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.924+0000 7fcddcbd5640 1 -- 172.21.15.161:0/2536260956 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcdc4003520 con 0x7fcdd80aa4e0 2024-03-28T23:30:11.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.924+0000 7f26f1bad640 1 -- 172.21.15.161:0/1201281700 --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26e40d8400 con 0x7f26e40d5940 2024-03-28T23:30:11.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.924+0000 7f26f1bad640 1 -- 172.21.15.161:0/1201281700 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26e404c6d0 con 0x7f26e400be30 2024-03-28T23:30:11.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.924+0000 7f26f1bad640 1 -- 172.21.15.161:0/1201281700 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f26e404c480 con 0x7f26e404b8b0 2024-03-28T23:30:11.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.925+0000 7f26ebfff640 1 -- 172.21.15.161:0/1201281700 >> v1:172.21.15.170:6789/0 conn(0x7f26e400be30 legacy=0x7f26e40e2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.925+0000 7fcddcbd5640 1 -- 172.21.15.161:0/2536260956 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3731801049 0 0) 0x7fcdc4003520 con 0x7fcdd80aa4e0 2024-03-28T23:30:11.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.925+0000 7f26ebfff640 1 -- 172.21.15.161:0/1201281700 >> v1:172.21.15.161:6789/0 conn(0x7f26e40d5940 legacy=0x7f26e404ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.925+0000 7f26ebfff640 1 -- 172.21.15.161:0/1201281700 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26e404bdc0 con 0x7f26e404b8b0 2024-03-28T23:30:11.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.925+0000 7fcddcbd5640 1 -- 172.21.15.161:0/2536260956 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdd8151fd0 msgr2=0x7fcdd8072e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.925+0000 7fcddcbd5640 1 --2- 172.21.15.161:0/2536260956 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdd8151fd0 0x7fcdd8072e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.925+0000 7fcddcbd5640 1 -- 172.21.15.161:0/2536260956 >> v2:172.21.15.170:3300/0 conn(0x7fcdd80aa960 msgr2=0x7fcdd806b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.925+0000 7fcddcbd5640 1 --2- 172.21.15.161:0/2536260956 >> v2:172.21.15.170:3300/0 conn(0x7fcdd80aa960 0x7fcdd806b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.925+0000 7fcddcbd5640 1 -- 172.21.15.161:0/2536260956 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd80c2470 con 0x7fcdd80aa4e0 2024-03-28T23:30:11.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.925+0000 7fcddcbd5640 1 -- 172.21.15.161:0/2536260956 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fcdc8001670 con 0x7fcdd80aa4e0 2024-03-28T23:30:11.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.925+0000 7fcddcbd5640 1 -- 172.21.15.161:0/2536260956 >> v1:172.21.15.161:6789/0 conn(0x7fcdd80aa4e0 legacy=0x7fcdd806b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.925+0000 7fcddcbd5640 1 --2- 172.21.15.161:0/2536260956 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdc4003a90 0x7fcdc4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.925+0000 7fcddcbd5640 1 --2- 172.21.15.161:0/2536260956 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdc4004800 0x7fcdc4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.925+0000 7fcddcbd5640 1 --2- 172.21.15.161:0/2536260956 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdd8151fd0 0x7fcdc400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.925+0000 7fcddcbd5640 1 -- 172.21.15.161:0/2536260956 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd806bda0 con 0x7fcdc4004800 2024-03-28T23:30:11.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.925+0000 7fcddcbd5640 1 -- 172.21.15.161:0/2536260956 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd80c2190 con 0x7fcdc4003a90 2024-03-28T23:30:11.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.925+0000 7fcddcbd5640 1 -- 172.21.15.161:0/2536260956 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fcdc4005680 con 0x7fcdd8151fd0 2024-03-28T23:30:11.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.926+0000 7fcdddbd7640 1 --2- 172.21.15.161:0/2536260956 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdc4003a90 0x7fcdc4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.926+0000 7fcddd3d6640 1 --2- 172.21.15.161:0/2536260956 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdc4004800 0x7fcdc4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.926+0000 7fcdddbd7640 1 -- 172.21.15.161:0/2536260956 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdd8151fd0 msgr2=0x7fcdc400b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T23:30:11.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.926+0000 7fcdddbd7640 1 --2- 172.21.15.161:0/2536260956 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdd8151fd0 0x7fcdc400b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.926+0000 7fcdddbd7640 1 -- 172.21.15.161:0/2536260956 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdc4004800 msgr2=0x7fcdc4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.926+0000 7fcdddbd7640 1 --2- 172.21.15.161:0/2536260956 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdc4004800 0x7fcdc4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.926+0000 7fcdddbd7640 1 -- 172.21.15.161:0/2536260956 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc4003520 con 0x7fcdc4003a90 2024-03-28T23:30:11.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.926+0000 7fcddd3d6640 1 --2- 172.21.15.161:0/2536260956 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdc4004800 0x7fcdc4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:11.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.928+0000 7f26ebfff640 1 --2- 172.21.15.161:0/1201281700 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26e404b8b0 0x7f26e4048330 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f26ec02d700 tx=0x7f26ec06aef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2570b86200f01c08 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.928+0000 7fcdddbd7640 1 --2- 172.21.15.161:0/2536260956 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdc4003a90 0x7fcdc4003e70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400ecd0 tx=0x7fcdc4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a84704b8fc0cdd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.928+0000 7fcddcbd5640 1 -- 172.21.15.161:0/2536260956 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd400e980 con 0x7fcdc4003a90 2024-03-28T23:30:11.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.928+0000 7fcddcbd5640 1 -- 172.21.15.161:0/2536260956 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdd400e2d0 con 0x7fcdc4003a90 2024-03-28T23:30:11.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.928+0000 7f26eb7fe640 1 -- 172.21.15.161:0/1201281700 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec0ab710 con 0x7f26e404b8b0 2024-03-28T23:30:11.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.928+0000 7f26eb7fe640 1 -- 172.21.15.161:0/1201281700 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26ec0ab8b0 con 0x7f26e404b8b0 2024-03-28T23:30:11.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.928+0000 7f26eb7fe640 1 -- 172.21.15.161:0/1201281700 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec0ab710 con 0x7f26e404b8b0 2024-03-28T23:30:11.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.929+0000 7f26f1bad640 1 -- 172.21.15.161:0/1201281700 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26e404b8b0 msgr2=0x7f26e4048330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.929+0000 7f26f1bad640 1 --2- 172.21.15.161:0/1201281700 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26e404b8b0 0x7f26e4048330 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f26ec02d700 tx=0x7f26ec06aef0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.929+0000 7fcddfe3b640 1 -- 172.21.15.161:0/2536260956 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdc4003a90 msgr2=0x7fcdc4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.929+0000 7fcddfe3b640 1 --2- 172.21.15.161:0/2536260956 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdc4003a90 0x7fcdc4003e70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400ecd0 tx=0x7fcdc4005830 comp rx=0 tx=0).stop 2024-03-28T23:30:11.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.929+0000 7fcddcbd5640 1 -- 172.21.15.161:0/2536260956 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4034ab0 con 0x7fcdc4003a90 2024-03-28T23:30:11.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.929+0000 7fcdde3d8640 1 -- 172.21.15.161:0/2536260956 reap_dead start 2024-03-28T23:30:11.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.929+0000 7fcddfe3b640 1 -- 172.21.15.161:0/2536260956 shutdown_connections 2024-03-28T23:30:11.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.929+0000 7f26f1bad640 1 -- 172.21.15.161:0/1201281700 shutdown_connections 2024-03-28T23:30:11.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.929+0000 7f26f1bad640 1 --2- 172.21.15.161:0/1201281700 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26e404b8b0 0x7f26e4048330 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.929+0000 7f26f1bad640 1 -- 172.21.15.161:0/1201281700 >> 172.21.15.161:0/1201281700 conn(0x7f26e402f4a0 msgr2=0x7f26e402f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.929+0000 7f26f1bad640 1 -- 172.21.15.161:0/1201281700 shutdown_connections 2024-03-28T23:30:11.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.929+0000 7fcddfe3b640 1 -- 172.21.15.161:0/2536260956 >> 172.21.15.161:0/2536260956 conn(0x7fcdd806d8e0 msgr2=0x7fcdd806dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.930+0000 7fcddfe3b640 1 -- 172.21.15.161:0/2536260956 shutdown_connections 2024-03-28T23:30:11.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.929+0000 7f26f1bad640 1 -- 172.21.15.161:0/1201281700 wait complete. 2024-03-28T23:30:11.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.931+0000 7fd7299fd640 1 Processor -- start 2024-03-28T23:30:11.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.931+0000 7fd7299fd640 1 -- start start 2024-03-28T23:30:11.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.930+0000 7fcddfe3b640 1 -- 172.21.15.161:0/2536260956 wait complete. 2024-03-28T23:30:11.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.932+0000 7fd7299fd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd724156ec0 0x7fd7240c5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.932+0000 7fd7299fd640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7241572a0 con 0x7fd7240c3560 2024-03-28T23:30:11.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.932+0000 7fd7299fd640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd72415d540 con 0x7fd72415cbd0 2024-03-28T23:30:11.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.932+0000 7fd7299fd640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd72415d820 con 0x7fd724156ec0 2024-03-28T23:30:11.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.932+0000 7fd7227fc640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd7240c3560 0x7fd72415a000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59630/0 (socket says 172.21.15.161:59630) 2024-03-28T23:30:11.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.932+0000 7fd7227fc640 1 -- 172.21.15.161:0/2349571948 learned_addr learned my addr 172.21.15.161:0/2349571948 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:11.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.932+0000 7fd722ffd640 1 --2- 172.21.15.161:0/2349571948 >> v2:172.21.15.161:3300/0 conn(0x7fd724156ec0 0x7fd7240c5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fb102f25640 1 Processor -- start 2024-03-28T23:30:11.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fb102f25640 1 -- start start 2024-03-28T23:30:11.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fd721ffb640 1 -- 172.21.15.161:0/2349571948 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2972120597 0 0) 0x7fd72415d540 con 0x7fd72415cbd0 2024-03-28T23:30:11.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fd721ffb640 1 -- 172.21.15.161:0/2349571948 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd70c003520 con 0x7fd72415cbd0 2024-03-28T23:30:11.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fd721ffb640 1 -- 172.21.15.161:0/2349571948 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3621968724 0 0) 0x7fd7241572a0 con 0x7fd7240c3560 2024-03-28T23:30:11.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fd721ffb640 1 -- 172.21.15.161:0/2349571948 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd72415d540 con 0x7fd7240c3560 2024-03-28T23:30:11.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fd721ffb640 1 -- 172.21.15.161:0/2349571948 <== mon.1 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3191700689 0 0) 0x7fd70c003520 con 0x7fd72415cbd0 2024-03-28T23:30:11.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fd721ffb640 1 -- 172.21.15.161:0/2349571948 >> v2:172.21.15.161:3300/0 conn(0x7fd724156ec0 msgr2=0x7fd7240c5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fd721ffb640 1 --2- 172.21.15.161:0/2349571948 >> v2:172.21.15.161:3300/0 conn(0x7fd724156ec0 0x7fd7240c5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.932+0000 7fcddfe3b640 1 Processor -- start 2024-03-28T23:30:11.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.932+0000 7fcddfe3b640 1 -- start start 2024-03-28T23:30:11.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fcddfe3b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdc4003a90 0x7fcdd80bf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fcddfe3b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdd80bfef0 0x7fcdd80c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fcddfe3b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdd80bbd60 0x7fcdd80bc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fcddfe3b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd80c1ad0 con 0x7fcdc4003a90 2024-03-28T23:30:11.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fcddfe3b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd80c2a30 con 0x7fcdd80bfef0 2024-03-28T23:30:11.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fcddfe3b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fcdd80c1c50 con 0x7fcdd80bbd60 2024-03-28T23:30:11.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fcddd3d6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdd80bfef0 0x7fcdd80c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fcddd3d6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdd80bfef0 0x7fcdd80c6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:39224/0 (socket says 172.21.15.161:39224) 2024-03-28T23:30:11.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fcddd3d6640 1 -- 172.21.15.161:0/1587801827 learned_addr learned my addr 172.21.15.161:0/1587801827 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:11.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fd721ffb640 1 -- 172.21.15.161:0/2349571948 >> v1:172.21.15.161:6789/0 conn(0x7fd7240c3560 legacy=0x7fd72415a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fd721ffb640 1 -- 172.21.15.161:0/2349571948 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd724159800 con 0x7fd72415cbd0 2024-03-28T23:30:11.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fd721ffb640 1 -- 172.21.15.161:0/2349571948 <== mon.1 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fd718001670 con 0x7fd72415cbd0 2024-03-28T23:30:11.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7fcdde3d8640 1 --2- 172.21.15.161:0/1587801827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdd80bbd60 0x7fcdd80bc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fd721ffb640 1 -- 172.21.15.161:0/2349571948 >> v1:172.21.15.170:6789/0 conn(0x7fd72415cbd0 legacy=0x7fd7240c24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.931+0000 7f26f1bad640 1 Processor -- start 2024-03-28T23:30:11.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.931+0000 7f26f1bad640 1 -- start start 2024-03-28T23:30:11.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7f26f1bad640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26e400be30 0x7f26e416f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7f26f1bad640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26e404b8b0 0x7f26e40d7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7f26f1bad640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26e40d5940 0x7f26e416b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7f26f1bad640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f26e404d5e0 con 0x7f26e400be30 2024-03-28T23:30:11.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7f26f1bad640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f26e404ca40 con 0x7f26e40d5940 2024-03-28T23:30:11.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7f26f1bad640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f26e4049a40 con 0x7f26e404b8b0 2024-03-28T23:30:11.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.931+0000 7f2b5531e640 1 Processor -- start 2024-03-28T23:30:11.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.931+0000 7f2b5531e640 1 -- start start 2024-03-28T23:30:11.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.932+0000 7f2b5531e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2b50072ee0 0x7f2b500c4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.932+0000 7f2b5531e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b500733c0 0x7f2b500c4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.932+0000 7f2b5531e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b500701b0 con 0x7f2b50150690 2024-03-28T23:30:11.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.932+0000 7f2b5531e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f2b50070490 con 0x7f2b50072ee0 2024-03-28T23:30:11.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.932+0000 7f2b5531e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2b50153ad0 con 0x7f2b500733c0 2024-03-28T23:30:11.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fd721ffb640 1 --2- 172.21.15.161:0/2349571948 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd70c003c70 0x7fd70c004070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.932+0000 7f2b4e7fc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2b50072ee0 0x7f2b500c4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.932+0000 7f2b4e7fc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2b50072ee0 0x7f2b500c4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:39222/0 (socket says 172.21.15.161:39222) 2024-03-28T23:30:11.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.932+0000 7f2b4e7fc640 1 -- 172.21.15.161:0/3554872847 learned_addr learned my addr 172.21.15.161:0/3554872847 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:11.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7f2b4f7fe640 1 --2- 172.21.15.161:0/3554872847 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b500733c0 0x7f2b500c4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7f2b4dffb640 1 -- 172.21.15.161:0/3554872847 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4107242711 0 0) 0x7f2b500701b0 con 0x7f2b50150690 2024-03-28T23:30:11.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7f2b4dffb640 1 -- 172.21.15.161:0/3554872847 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b38003520 con 0x7f2b50150690 2024-03-28T23:30:11.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7f2b4f7fe640 1 -- 172.21.15.161:0/3554872847 >> v2:172.21.15.170:3300/0 conn(0x7f2b50072ee0 msgr2=0x7f2b500c4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7f2b4f7fe640 1 --2- 172.21.15.161:0/3554872847 >> v2:172.21.15.170:3300/0 conn(0x7f2b50072ee0 0x7f2b500c4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7f2b4f7fe640 1 -- 172.21.15.161:0/3554872847 >> v1:172.21.15.161:6789/0 conn(0x7f2b50150690 legacy=0x7f2b5006f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7f2b4f7fe640 1 -- 172.21.15.161:0/3554872847 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b50153db0 con 0x7f2b500733c0 2024-03-28T23:30:11.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7fcdddbd7640 1 --2- 172.21.15.161:0/1587801827 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdc4003a90 0x7fcdd80bf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fd721ffb640 1 --2- 172.21.15.161:0/2349571948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd70c004800 0x7fd70c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fd721ffb640 1 --2- 172.21.15.161:0/2349571948 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd70c0052e0 0x7fd70c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.933+0000 7fd7227fc640 1 --2- 172.21.15.161:0/2349571948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd70c004800 0x7fd70c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7fd721ffb640 1 -- 172.21.15.161:0/2349571948 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd72415d820 con 0x7fd70c003c70 2024-03-28T23:30:11.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7fd721ffb640 1 -- 172.21.15.161:0/2349571948 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c00c190 con 0x7fd70c004800 2024-03-28T23:30:11.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7fd721ffb640 1 -- 172.21.15.161:0/2349571948 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd70c00c390 con 0x7fd70c0052e0 2024-03-28T23:30:11.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7fd722ffd640 1 --2- 172.21.15.161:0/2349571948 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd70c003c70 0x7fd70c004070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7fb102f25640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0fc0a8d10 0x7fb0fc0ab460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7fcdddbd7640 1 -- 172.21.15.161:0/1587801827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdd80bbd60 msgr2=0x7fcdd80bc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7fcdddbd7640 1 --2- 172.21.15.161:0/1587801827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdd80bbd60 0x7fcdd80bc3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7fcdddbd7640 1 -- 172.21.15.161:0/1587801827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdd80bfef0 msgr2=0x7fcdd80c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7fcdddbd7640 1 --2- 172.21.15.161:0/1587801827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdd80bfef0 0x7fcdd80c6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7fcdddbd7640 1 -- 172.21.15.161:0/1587801827 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd80bb720 con 0x7fcdc4003a90 2024-03-28T23:30:11.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7fd7237fe640 1 --2- 172.21.15.161:0/2349571948 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd70c0052e0 0x7fd70c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7fb102f25640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0fc1550d0 con 0x7fb0fc152a90 2024-03-28T23:30:11.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7fb102f25640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0fc155390 con 0x7fb0fc0c2390 2024-03-28T23:30:11.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7fb102f25640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb0fc155fa0 con 0x7fb0fc0a8d10 2024-03-28T23:30:11.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7fb100cc1640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb0fc152a90 0x7fb0fc0bfa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59636/0 (socket says 172.21.15.161:59636) 2024-03-28T23:30:11.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7fb100cc1640 1 -- 172.21.15.161:0/1659132636 learned_addr learned my addr 172.21.15.161:0/1659132636 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:11.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.935+0000 7fb1014c2640 1 --2- 172.21.15.161:0/1659132636 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0fc0a8d10 0x7fb0fc0ab460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.935+0000 7fd7237fe640 1 -- 172.21.15.161:0/2349571948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd70c004800 msgr2=0x7fd70c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.935+0000 7fd7237fe640 1 --2- 172.21.15.161:0/2349571948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd70c004800 0x7fd70c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.935+0000 7fd7237fe640 1 -- 172.21.15.161:0/2349571948 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd70c003c70 msgr2=0x7fd70c004070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.935+0000 7fd7237fe640 1 --2- 172.21.15.161:0/2349571948 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd70c003c70 0x7fd70c004070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.935+0000 7fd7237fe640 1 -- 172.21.15.161:0/2349571948 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd70c003520 con 0x7fd70c0052e0 2024-03-28T23:30:11.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7f26f0bab640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26e400be30 0x7f26e416f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7f26f0bab640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26e400be30 0x7f26e416f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54252/0 (socket says 172.21.15.161:54252) 2024-03-28T23:30:11.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7f26f0bab640 1 -- 172.21.15.161:0/1425582772 learned_addr learned my addr 172.21.15.161:0/1425582772 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:11.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.934+0000 7f26f13ac640 1 --2- 172.21.15.161:0/1425582772 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26e40d5940 0x7f26e416b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.935+0000 7f26ebfff640 1 --2- 172.21.15.161:0/1425582772 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26e404b8b0 0x7f26e40d7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.935+0000 7fcdddbd7640 1 --2- 172.21.15.161:0/1587801827 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdc4003a90 0x7fcdd80bf9b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcdd403a000 tx=0x7fcdd4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.935+0000 7fb0fb7fe640 1 -- 172.21.15.161:0/1659132636 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 736902548 0 0) 0x7fb0fc1550d0 con 0x7fb0fc152a90 2024-03-28T23:30:11.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.935+0000 7fb0fb7fe640 1 -- 172.21.15.161:0/1659132636 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0e4003520 con 0x7fb0fc152a90 2024-03-28T23:30:11.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.935+0000 7f26ebfff640 1 -- 172.21.15.161:0/1425582772 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26e40d5940 msgr2=0x7f26e416b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.935+0000 7f26ebfff640 1 --2- 172.21.15.161:0/1425582772 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26e40d5940 0x7f26e416b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.935+0000 7f26ebfff640 1 -- 172.21.15.161:0/1425582772 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26e400be30 msgr2=0x7f26e416f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.935+0000 7f26ebfff640 1 --2- 172.21.15.161:0/1425582772 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26e400be30 0x7f26e416f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.935+0000 7f26ebfff640 1 -- 172.21.15.161:0/1425582772 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26e416fcc0 con 0x7f26e404b8b0 2024-03-28T23:30:11.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.935+0000 7f26ebfff640 1 --2- 172.21.15.161:0/1425582772 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26e404b8b0 0x7f26e40d7760 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f26ec07a9e0 tx=0x7f26ec06a8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.935+0000 7f26e97fa640 1 -- 172.21.15.161:0/1425582772 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec0ab710 con 0x7f26e404b8b0 2024-03-28T23:30:11.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.936+0000 7fcdce7fc640 1 -- 172.21.15.161:0/1587801827 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4015070 con 0x7fcdc4003a90 2024-03-28T23:30:11.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.936+0000 7fcdce7fc640 1 -- 172.21.15.161:0/1587801827 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdd40184b0 con 0x7fcdc4003a90 2024-03-28T23:30:11.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.936+0000 7fcdce7fc640 1 -- 172.21.15.161:0/1587801827 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd400e980 con 0x7fcdc4003a90 2024-03-28T23:30:11.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.936+0000 7f26f1bad640 1 -- 172.21.15.161:0/1425582772 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26e400a030 con 0x7f26e404b8b0 2024-03-28T23:30:11.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.936+0000 7f26f1bad640 1 -- 172.21.15.161:0/1425582772 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f26e400a650 con 0x7f26e404b8b0 2024-03-28T23:30:11.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.936+0000 7fcddfe3b640 1 -- 172.21.15.161:0/1587801827 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdd80bba00 con 0x7fcdc4003a90 2024-03-28T23:30:11.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.936+0000 7fcddfe3b640 1 -- 172.21.15.161:0/1587801827 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdd80c0800 con 0x7fcdc4003a90 2024-03-28T23:30:11.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.936+0000 7fb0fb7fe640 1 -- 172.21.15.161:0/1659132636 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2605185667 0 0) 0x7fb0e4003520 con 0x7fb0fc152a90 2024-03-28T23:30:11.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.936+0000 7fb0fb7fe640 1 -- 172.21.15.161:0/1659132636 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0fc0a8d10 msgr2=0x7fb0fc0ab460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.936+0000 7fb0fb7fe640 1 --2- 172.21.15.161:0/1659132636 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0fc0a8d10 0x7fb0fc0ab460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.936+0000 7fb0fb7fe640 1 -- 172.21.15.161:0/1659132636 >> v1:172.21.15.170:6789/0 conn(0x7fb0fc0c2390 legacy=0x7fb0fc0aad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.936+0000 7fb0fb7fe640 1 -- 172.21.15.161:0/1659132636 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0fc155670 con 0x7fb0fc152a90 2024-03-28T23:30:11.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.936+0000 7fb0fb7fe640 1 -- 172.21.15.161:0/1659132636 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fb0f0001670 con 0x7fb0fc152a90 2024-03-28T23:30:11.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.936+0000 7fb0fb7fe640 1 -- 172.21.15.161:0/1659132636 >> v1:172.21.15.161:6789/0 conn(0x7fb0fc152a90 legacy=0x7fb0fc0bfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.936+0000 7fb0fb7fe640 1 --2- 172.21.15.161:0/1659132636 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0e4003ac0 0x7fb0e4003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.936+0000 7fb0fb7fe640 1 --2- 172.21.15.161:0/1659132636 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0e4004800 0x7fb0e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.936+0000 7fb0fb7fe640 1 --2- 172.21.15.161:0/1659132636 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0e40052e0 0x7fb0e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.936+0000 7fb0fb7fe640 1 -- 172.21.15.161:0/1659132636 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e400c0e0 con 0x7fb0e4004800 2024-03-28T23:30:11.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.936+0000 7fb0fb7fe640 1 -- 172.21.15.161:0/1659132636 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e400c290 con 0x7fb0e4003ac0 2024-03-28T23:30:11.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.936+0000 7fb0fb7fe640 1 -- 172.21.15.161:0/1659132636 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb0e400c490 con 0x7fb0e40052e0 2024-03-28T23:30:11.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.936+0000 7f26e97fa640 1 -- 172.21.15.161:0/1425582772 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26ec0b2bf0 con 0x7f26e404b8b0 2024-03-28T23:30:11.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.936+0000 7f26e97fa640 1 -- 172.21.15.161:0/1425582772 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec0b2de0 con 0x7f26e404b8b0 2024-03-28T23:30:11.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.937+0000 7fb100cc1640 1 --2- 172.21.15.161:0/1659132636 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0e4003ac0 0x7fb0e4003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.936+0000 7fd7237fe640 1 -- 172.21.15.161:0/2349571948 reap_dead start 2024-03-28T23:30:11.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.937+0000 7fb0fbfff640 1 --2- 172.21.15.161:0/1659132636 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0e4004800 0x7fb0e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.938+0000 7fb0fbfff640 1 -- 172.21.15.161:0/1659132636 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0e40052e0 msgr2=0x7fb0e400b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T23:30:11.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.938+0000 7fb0fbfff640 1 --2- 172.21.15.161:0/1659132636 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0e40052e0 0x7fb0e400b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.938+0000 7fb0fbfff640 1 -- 172.21.15.161:0/1659132636 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0e4003ac0 msgr2=0x7fb0e4003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.938+0000 7fb0fbfff640 1 --2- 172.21.15.161:0/1659132636 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0e4003ac0 0x7fb0e4003f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.938+0000 7fb0fbfff640 1 -- 172.21.15.161:0/1659132636 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0fc155670 con 0x7fb0e4004800 2024-03-28T23:30:11.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.938+0000 7f26e97fa640 1 -- 172.21.15.161:0/1425582772 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f26ec0837b0 con 0x7f26e404b8b0 2024-03-28T23:30:11.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.938+0000 7fcdce7fc640 1 -- 172.21.15.161:0/1587801827 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcdd400c870 con 0x7fcdc4003a90 2024-03-28T23:30:11.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.938+0000 7fcddfe3b640 1 -- 172.21.15.161:0/1587801827 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdd804f4b0 con 0x7fcdc4003a90 2024-03-28T23:30:11.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.938+0000 7fb100cc1640 1 --2- 172.21.15.161:0/1659132636 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0e4003ac0 0x7fb0e4003f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:11.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.939+0000 7f26e97fa640 1 --2- 172.21.15.161:0/1425582772 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f26d006b5a0 0x7f26d006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.939+0000 7f26e97fa640 1 -- 172.21.15.161:0/1425582772 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f26ec11e140 con 0x7f26e404b8b0 2024-03-28T23:30:11.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.939+0000 7fb0fbfff640 1 --2- 172.21.15.161:0/1659132636 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0e4004800 0x7fb0e4004c80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec003dd0 tx=0x7fb0e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=424c547cb94d6c17 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.939+0000 7f26f0bab640 1 --2- 172.21.15.161:0/1425582772 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f26d006b5a0 0x7f26d006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.939+0000 7fcdce7fc640 1 --2- 172.21.15.161:0/1587801827 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcdbc06b520 0x7fcdbc06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.939+0000 7fcdce7fc640 1 -- 172.21.15.161:0/1587801827 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fcdd40ab370 con 0x7fcdc4003a90 2024-03-28T23:30:11.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.939+0000 7f2b4f7fe640 1 --2- 172.21.15.161:0/3554872847 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b500733c0 0x7f2b500c4c50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2b440097a0 tx=0x7f2b4402fd00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e9ce9fea51b8ef4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.939+0000 7f26f1bad640 1 -- 172.21.15.161:0/1425582772 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26cc005290 con 0x7f26e404b8b0 2024-03-28T23:30:11.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.939+0000 7fcddd3d6640 1 --2- 172.21.15.161:0/1587801827 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcdbc06b520 0x7fcdbc06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.940+0000 7fcddd3d6640 1 --2- 172.21.15.161:0/1587801827 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcdbc06b520 0x7fcdbc06d9e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80c3270 tx=0x7fcdc8001b00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.940+0000 7f26f0bab640 1 --2- 172.21.15.161:0/1425582772 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f26d006b5a0 0x7f26d006da60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f26d8005880 tx=0x7f26d8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.940+0000 7f2b4dffb640 1 -- 172.21.15.161:0/3554872847 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b4403d070 con 0x7f2b500733c0 2024-03-28T23:30:11.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.940+0000 7f2b4dffb640 1 -- 172.21.15.161:0/3554872847 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b44030a90 con 0x7f2b500733c0 2024-03-28T23:30:11.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.940+0000 7fd7237fe640 1 --2- 172.21.15.161:0/2349571948 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd70c0052e0 0x7fd70c00b9c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd718007ec0 tx=0x7fd70c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5ea6823a90fa043 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.940+0000 7f2b4dffb640 1 -- 172.21.15.161:0/3554872847 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b44038440 con 0x7f2b500733c0 2024-03-28T23:30:11.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.941+0000 7f2b5531e640 1 -- 172.21.15.161:0/3554872847 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b500733c0 msgr2=0x7f2b500c4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.941+0000 7f2b5531e640 1 --2- 172.21.15.161:0/3554872847 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b500733c0 0x7f2b500c4c50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2b440097a0 tx=0x7f2b4402fd00 comp rx=0 tx=0).stop 2024-03-28T23:30:11.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.941+0000 7f2b5531e640 1 -- 172.21.15.161:0/3554872847 shutdown_connections 2024-03-28T23:30:11.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.941+0000 7f2b5531e640 1 --2- 172.21.15.161:0/3554872847 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b500733c0 0x7f2b500c4c50 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.941+0000 7f2b5531e640 1 --2- 172.21.15.161:0/3554872847 >> v2:172.21.15.170:3300/0 conn(0x7f2b50072ee0 0x7f2b500c4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.941+0000 7f2b5531e640 1 -- 172.21.15.161:0/3554872847 >> 172.21.15.161:0/3554872847 conn(0x7f2b500c88b0 msgr2=0x7f2b500c8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.941+0000 7f2b5531e640 1 -- 172.21.15.161:0/3554872847 shutdown_connections 2024-03-28T23:30:11.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.941+0000 7f2b5531e640 1 -- 172.21.15.161:0/3554872847 wait complete. 2024-03-28T23:30:11.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.943+0000 7f2b5531e640 1 Processor -- start 2024-03-28T23:30:11.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.943+0000 7f2b5531e640 1 -- start start 2024-03-28T23:30:11.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.943+0000 7f89bd9d0640 1 Processor -- start 2024-03-28T23:30:11.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.943+0000 7f89bd9d0640 1 -- start start 2024-03-28T23:30:11.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.944+0000 7f2b5531e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b50072ee0 0x7f2b5014b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.944+0000 7f2b5531e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b500733c0 0x7f2b501ea000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.944+0000 7f2b5531e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b50150690 0x7f2b501ea540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.944+0000 7f2b5531e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5006ff90 con 0x7f2b500733c0 2024-03-28T23:30:11.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.944+0000 7f2b5531e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5006fe10 con 0x7f2b50072ee0 2024-03-28T23:30:11.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.944+0000 7f2b5531e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2b50150a70 con 0x7f2b50150690 2024-03-28T23:30:11.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.944+0000 7f2b4e7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b500733c0 0x7f2b501ea000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.944+0000 7f2b4e7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b500733c0 0x7f2b501ea000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54266/0 (socket says 172.21.15.161:54266) 2024-03-28T23:30:11.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.944+0000 7f2b4f7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b50150690 0x7f2b501ea540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.944+0000 7f2b4e7fc640 1 -- 172.21.15.161:0/2976794870 learned_addr learned my addr 172.21.15.161:0/2976794870 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:11.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.945+0000 7f89bd9d0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f89b80c3510 0x7f89b81596c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.945+0000 7f89bd9d0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f89b8071e70 0x7f89b815d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.945+0000 7f89bd9d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89b806e7f0 0x7f89b806c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.945+0000 7f89bd9d0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f89b8159c00 con 0x7f89b80c3510 2024-03-28T23:30:11.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.945+0000 7f89bd9d0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f89b80728f0 con 0x7f89b8071e70 2024-03-28T23:30:11.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.945+0000 7f89bd9d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f89b8072330 con 0x7f89b806e7f0 2024-03-28T23:30:11.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.945+0000 7fd721ffb640 1 -- 172.21.15.161:0/2349571948 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd718039070 con 0x7fd70c0052e0 2024-03-28T23:30:11.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.945+0000 7fd721ffb640 1 -- 172.21.15.161:0/2349571948 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd71802c8b0 con 0x7fd70c0052e0 2024-03-28T23:30:11.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.945+0000 7f2b4effd640 1 --2- 172.21.15.161:0/2976794870 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b50072ee0 0x7f2b5014b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.945+0000 7f2b4f7fe640 1 -- 172.21.15.161:0/2976794870 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b50072ee0 msgr2=0x7f2b5014b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.945+0000 7f2b4f7fe640 1 --2- 172.21.15.161:0/2976794870 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b50072ee0 0x7f2b5014b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.945+0000 7f2b4f7fe640 1 -- 172.21.15.161:0/2976794870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b500733c0 msgr2=0x7f2b501ea000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.945+0000 7f2b4f7fe640 1 --2- 172.21.15.161:0/2976794870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b500733c0 0x7f2b501ea000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.945+0000 7f2b4f7fe640 1 -- 172.21.15.161:0/2976794870 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b50150ec0 con 0x7f2b50150690 2024-03-28T23:30:11.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.945+0000 7f26e97fa640 1 -- 172.21.15.161:0/1425582772 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f26ec0ec6d0 con 0x7f26e404b8b0 2024-03-28T23:30:11.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.945+0000 7fb0fb7fe640 1 -- 172.21.15.161:0/1659132636 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec037070 con 0x7fb0e4004800 2024-03-28T23:30:11.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.945+0000 7f89b6ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f89b80c3510 0x7f89b81596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.945+0000 7f89b6ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f89b80c3510 0x7f89b81596c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54278/0 (socket says 172.21.15.161:54278) 2024-03-28T23:30:11.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.945+0000 7f89b6ffd640 1 -- 172.21.15.161:0/3761104067 learned_addr learned my addr 172.21.15.161:0/3761104067 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:11.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.945+0000 7f89b67fc640 1 --2- 172.21.15.161:0/3761104067 >> v2:172.21.15.170:3300/0 conn(0x7f89b8071e70 0x7f89b815d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.945+0000 7fcdce7fc640 1 -- 172.21.15.161:0/1587801827 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcdd404a070 con 0x7fcdc4003a90 2024-03-28T23:30:11.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.945+0000 7fb0fb7fe640 1 -- 172.21.15.161:0/1659132636 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0ec032600 con 0x7fb0e4004800 2024-03-28T23:30:11.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.945+0000 7fb0fb7fe640 1 -- 172.21.15.161:0/1659132636 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec0039a0 con 0x7fb0e4004800 2024-03-28T23:30:11.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.945+0000 7f2b4f7fe640 1 --2- 172.21.15.161:0/2976794870 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b50150690 0x7f2b501ea540 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2b44037000 tx=0x7f2b440046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.946+0000 7f89b77fe640 1 --2- 172.21.15.161:0/3761104067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89b806e7f0 0x7f89b806c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.946+0000 7f89b6ffd640 1 -- 172.21.15.161:0/3761104067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89b806e7f0 msgr2=0x7f89b806c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.946+0000 7f89b6ffd640 1 --2- 172.21.15.161:0/3761104067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89b806e7f0 0x7f89b806c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.946+0000 7f89b6ffd640 1 -- 172.21.15.161:0/3761104067 >> v2:172.21.15.170:3300/0 conn(0x7f89b8071e70 msgr2=0x7f89b815d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:11.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.946+0000 7f89b6ffd640 1 --2- 172.21.15.161:0/3761104067 >> v2:172.21.15.170:3300/0 conn(0x7f89b8071e70 0x7f89b815d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.946+0000 7f89b6ffd640 1 -- 172.21.15.161:0/3761104067 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b8072610 con 0x7f89b80c3510 2024-03-28T23:30:11.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.946+0000 7f2b377fe640 1 -- 172.21.15.161:0/2976794870 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b4403d050 con 0x7f2b50150690 2024-03-28T23:30:11.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.946+0000 7fd7299fd640 1 -- 172.21.15.161:0/2349571948 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd70c0052e0 msgr2=0x7fd70c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.946+0000 7fd7299fd640 1 --2- 172.21.15.161:0/2349571948 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd70c0052e0 0x7fd70c00b9c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd718007ec0 tx=0x7fd70c005830 comp rx=0 tx=0).stop 2024-03-28T23:30:11.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.946+0000 7f2b377fe640 1 -- 172.21.15.161:0/2976794870 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b44049070 con 0x7f2b50150690 2024-03-28T23:30:11.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.946+0000 7f2b377fe640 1 -- 172.21.15.161:0/2976794870 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b44003c60 con 0x7f2b50150690 2024-03-28T23:30:11.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.946+0000 7f2b5531e640 1 -- 172.21.15.161:0/2976794870 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b501511a0 con 0x7f2b50150690 2024-03-28T23:30:11.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.946+0000 7f2b5531e640 1 -- 172.21.15.161:0/2976794870 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b5014c220 con 0x7f2b50150690 2024-03-28T23:30:11.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.947+0000 7fd7299fd640 1 -- 172.21.15.161:0/2349571948 shutdown_connections 2024-03-28T23:30:11.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.947+0000 7fd7299fd640 1 --2- 172.21.15.161:0/2349571948 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd70c0052e0 0x7fd70c00b9c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.947+0000 7fd7299fd640 1 -- 172.21.15.161:0/2349571948 >> 172.21.15.161:0/2349571948 conn(0x7fd7240a8820 msgr2=0x7fd7240a8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.947+0000 7fb102f25640 1 -- 172.21.15.161:0/1659132636 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0e4004800 msgr2=0x7fb0e4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.947+0000 7fb102f25640 1 --2- 172.21.15.161:0/1659132636 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0e4004800 0x7fb0e4004c80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec003dd0 tx=0x7fb0e4005830 comp rx=0 tx=0).stop 2024-03-28T23:30:11.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.947+0000 7fb1014c2640 1 -- 172.21.15.161:0/1659132636 reap_dead start 2024-03-28T23:30:11.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.947+0000 7f89b6ffd640 1 --2- 172.21.15.161:0/3761104067 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89b80c3510 0x7f89b81596c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f89b806f8f0 tx=0x7f89a802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96e36529fa822275 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.947+0000 7fb102f25640 1 -- 172.21.15.161:0/1659132636 shutdown_connections 2024-03-28T23:30:11.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.947+0000 7fb102f25640 1 -- 172.21.15.161:0/1659132636 >> 172.21.15.161:0/1659132636 conn(0x7fb0fc0c6670 msgr2=0x7fb0fc0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.947+0000 7f89b5ffb640 1 -- 172.21.15.161:0/3761104067 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a802e070 con 0x7f89b80c3510 2024-03-28T23:30:11.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.947+0000 7f89b5ffb640 1 -- 172.21.15.161:0/3761104067 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89a8037070 con 0x7f89b80c3510 2024-03-28T23:30:11.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.947+0000 7f89b5ffb640 1 -- 172.21.15.161:0/3761104067 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a8031a00 con 0x7f89b80c3510 2024-03-28T23:30:11.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.948+0000 7f89bd9d0640 1 -- 172.21.15.161:0/3761104067 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89b80c3510 msgr2=0x7f89b81596c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.948+0000 7f89bd9d0640 1 --2- 172.21.15.161:0/3761104067 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89b80c3510 0x7f89b81596c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f89b806f8f0 tx=0x7f89a802b880 comp rx=0 tx=0).stop 2024-03-28T23:30:11.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.948+0000 7fb102f25640 1 -- 172.21.15.161:0/1659132636 shutdown_connections 2024-03-28T23:30:11.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.949+0000 7fb102f25640 1 -- 172.21.15.161:0/1659132636 wait complete. 2024-03-28T23:30:11.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.949+0000 7f2b377fe640 1 -- 172.21.15.161:0/2976794870 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2b4404e460 con 0x7f2b50150690 2024-03-28T23:30:11.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.949+0000 7f89bd9d0640 1 -- 172.21.15.161:0/3761104067 shutdown_connections 2024-03-28T23:30:11.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.949+0000 7f89bd9d0640 1 --2- 172.21.15.161:0/3761104067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89b806e7f0 0x7f89b806c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.949+0000 7f89bd9d0640 1 --2- 172.21.15.161:0/3761104067 >> v2:172.21.15.170:3300/0 conn(0x7f89b8071e70 0x7f89b815d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.949+0000 7f89bd9d0640 1 --2- 172.21.15.161:0/3761104067 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89b80c3510 0x7f89b81596c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.949+0000 7f89bd9d0640 1 -- 172.21.15.161:0/3761104067 >> 172.21.15.161:0/3761104067 conn(0x7f89b80c3cf0 msgr2=0x7f89b80c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:11.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.949+0000 7f89bd9d0640 1 -- 172.21.15.161:0/3761104067 shutdown_connections 2024-03-28T23:30:11.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.949+0000 7f89bd9d0640 1 -- 172.21.15.161:0/3761104067 wait complete. 2024-03-28T23:30:11.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.949+0000 7fd7299fd640 1 -- 172.21.15.161:0/2349571948 shutdown_connections 2024-03-28T23:30:11.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.949+0000 7fd7299fd640 1 -- 172.21.15.161:0/2349571948 wait complete. 2024-03-28T23:30:11.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.951+0000 7fb102f25640 1 Processor -- start 2024-03-28T23:30:11.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.951+0000 7fd7299fd640 1 Processor -- start 2024-03-28T23:30:11.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.951+0000 7f89bd9d0640 1 Processor -- start 2024-03-28T23:30:11.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.951+0000 7fd7299fd640 1 -- start start 2024-03-28T23:30:11.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.951+0000 7f89bd9d0640 1 -- start start 2024-03-28T23:30:11.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.951+0000 7fb102f25640 1 -- start start 2024-03-28T23:30:11.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.952+0000 7fb102f25640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0fc153340 0x7fb0fc153720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.952+0000 7fb102f25640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0fc14e840 0x7fb0fc147890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.952+0000 7fb102f25640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0fc147dd0 0x7fb0fc158550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.952+0000 7fb102f25640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc155c30 con 0x7fb0fc14e840 2024-03-28T23:30:11.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.952+0000 7fb102f25640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc0a91a0 con 0x7fb0fc153340 2024-03-28T23:30:11.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.952+0000 7fb102f25640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb0fc0c0110 con 0x7fb0fc147dd0 2024-03-28T23:30:11.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.952+0000 7fb0fbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0fc14e840 0x7fb0fc147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.952+0000 7fb0fbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0fc14e840 0x7fb0fc147890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54290/0 (socket says 172.21.15.161:54290) 2024-03-28T23:30:11.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.952+0000 7fb0fbfff640 1 -- 172.21.15.161:0/3602223949 learned_addr learned my addr 172.21.15.161:0/3602223949 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:11.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.952+0000 7f89bd9d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89b806e7f0 0x7f89b81e3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.953+0000 7fd7299fd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd70c0052e0 0x7fd72415bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.953+0000 7fd7299fd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7240a3810 0x7fd7241efc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.953+0000 7f89bd9d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89b8071e70 0x7f89b80acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.953+0000 7f89b6ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89b806e7f0 0x7f89b81e3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.953+0000 7f89b6ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89b806e7f0 0x7f89b81e3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:36990/0 (socket says 172.21.15.161:36990) 2024-03-28T23:30:11.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.953+0000 7f89b6ffd640 1 -- 172.21.15.161:0/92183516 learned_addr learned my addr 172.21.15.161:0/92183516 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:11.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.953+0000 7fb0fbfff640 1 -- 172.21.15.161:0/3602223949 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0fc147dd0 msgr2=0x7fb0fc158550 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T23:30:11.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.953+0000 7fb0fbfff640 1 --2- 172.21.15.161:0/3602223949 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0fc147dd0 0x7fb0fc158550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.953+0000 7f89b67fc640 1 --2- 172.21.15.161:0/92183516 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89b8071e70 0x7f89b80acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.953+0000 7fb0fbfff640 1 -- 172.21.15.161:0/3602223949 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0fc153340 msgr2=0x7fb0fc153720 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T23:30:11.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.953+0000 7fb0fbfff640 1 --2- 172.21.15.161:0/3602223949 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0fc153340 0x7fb0fc153720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:11.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.953+0000 7fb0fbfff640 1 -- 172.21.15.161:0/3602223949 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0fc157ae0 con 0x7fb0fc14e840 2024-03-28T23:30:11.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.953+0000 7f89bd9d0640 1 --2- 172.21.15.161:0/92183516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89b80c3510 0x7f89b8158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.953+0000 7fd7227fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7240a3810 0x7fd7241efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.953+0000 7f89bd9d0640 1 -- 172.21.15.161:0/92183516 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f89b80c38f0 con 0x7f89b80c3510 2024-03-28T23:30:11.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.953+0000 7fb0fbfff640 1 --2- 172.21.15.161:0/3602223949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0fc14e840 0x7fb0fc147890 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec003ef0 tx=0x7fb0ec0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:11.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.953+0000 7f89bd9d0640 1 -- 172.21.15.161:0/92183516 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f89b815d030 con 0x7f89b8071e70 2024-03-28T23:30:11.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.953+0000 7f89bd9d0640 1 -- 172.21.15.161:0/92183516 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f89b80ad2f0 con 0x7f89b806e7f0 2024-03-28T23:30:11.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.953+0000 7fd7227fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7240a3810 0x7fd7241efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:39292/0 (socket says 172.21.15.161:39292) 2024-03-28T23:30:11.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.953+0000 7fd7227fc640 1 -- 172.21.15.161:0/231323191 learned_addr learned my addr 172.21.15.161:0/231323191 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:11.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.953+0000 7f89b77fe640 1 --2- 172.21.15.161:0/92183516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89b80c3510 0x7f89b8158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:11.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.953+0000 7fb0f97fa640 1 -- 172.21.15.161:0/3602223949 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec037040 con 0x7fb0fc14e840 2024-03-28T23:30:11.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.954+0000 7fd7299fd640 1 --2- 172.21.15.161:0/231323191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd724158d30 0x7fd72415ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.954+0000 7fd7299fd640 1 -- 172.21.15.161:0/231323191 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd72415d1b0 con 0x7fd724158d30 2024-03-28T23:30:11.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.954+0000 7f2b5531e640 1 -- 172.21.15.161:0/2976794870 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b30005290 con 0x7f2b50150690 2024-03-28T23:30:11.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.954+0000 7f2b377fe640 1 --2- 172.21.15.161:0/2976794870 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2b3c06b520 0x7f2b3c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:11.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.954+0000 7f89b77fe640 1 -- 172.21.15.161:0/92183516 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89b806e7f0 msgr2=0x7f89b81e3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:11.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.954+0000 7f89b77fe640 1 --2- 172.21.15.161:0/92183516 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89b806e7f0 0x7f89b81e3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.954+0000 7f89b77fe640 1 -- 172.21.15.161:0/92183516 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89b8071e70 msgr2=0x7f89b80acdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:12.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.954+0000 7f89b77fe640 1 --2- 172.21.15.161:0/92183516 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89b8071e70 0x7f89b80acdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.954+0000 7f89b77fe640 1 -- 172.21.15.161:0/92183516 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b81efa80 con 0x7f89b80c3510 2024-03-28T23:30:12.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.954+0000 7fd7299fd640 1 -- 172.21.15.161:0/231323191 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd72415d030 con 0x7fd7240a3810 2024-03-28T23:30:12.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.954+0000 7fd7299fd640 1 -- 172.21.15.161:0/231323191 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd7240c5880 con 0x7fd70c0052e0 2024-03-28T23:30:12.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.954+0000 7fd7237fe640 1 --2- 172.21.15.161:0/231323191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd724158d30 0x7fd72415ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:12.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.955+0000 7fd722ffd640 1 --2- 172.21.15.161:0/231323191 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd70c0052e0 0x7fd72415bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:12.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.954+0000 7fb102f25640 1 -- 172.21.15.161:0/3602223949 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0fc14d8e0 con 0x7fb0fc14e840 2024-03-28T23:30:12.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.954+0000 7fb102f25640 1 -- 172.21.15.161:0/3602223949 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0fc157d50 con 0x7fb0fc14e840 2024-03-28T23:30:12.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.955+0000 7fb0f97fa640 1 -- 172.21.15.161:0/3602223949 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0ec032400 con 0x7fb0fc14e840 2024-03-28T23:30:12.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.955+0000 7fb0f97fa640 1 -- 172.21.15.161:0/3602223949 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec0039a0 con 0x7fb0fc14e840 2024-03-28T23:30:12.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.955+0000 7f2b4effd640 1 --2- 172.21.15.161:0/2976794870 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2b3c06b520 0x7f2b3c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:12.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.956+0000 7f2b4effd640 1 --2- 172.21.15.161:0/2976794870 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2b3c06b520 0x7f2b3c06d9e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2b5014c720 tx=0x7f2b40009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:12.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.956+0000 7f2b377fe640 1 -- 172.21.15.161:0/2976794870 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2b440b0560 con 0x7f2b50150690 2024-03-28T23:30:12.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.956+0000 7fd7237fe640 1 -- 172.21.15.161:0/231323191 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd70c0052e0 msgr2=0x7fd72415bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:12.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.956+0000 7fd7237fe640 1 --2- 172.21.15.161:0/231323191 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd70c0052e0 0x7fd72415bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.956+0000 7fd7237fe640 1 -- 172.21.15.161:0/231323191 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7240a3810 msgr2=0x7fd7241efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:12.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.956+0000 7fd7237fe640 1 --2- 172.21.15.161:0/231323191 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7240a3810 0x7fd7241efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.956+0000 7fd7237fe640 1 -- 172.21.15.161:0/231323191 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72415c4f0 con 0x7fd724158d30 2024-03-28T23:30:12.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.956+0000 7f89b77fe640 1 --2- 172.21.15.161:0/92183516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89b80c3510 0x7f89b8158d50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f89b000ccf0 tx=0x7f89b000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:12.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.957+0000 7fd7237fe640 1 --2- 172.21.15.161:0/231323191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd724158d30 0x7fd72415ac80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd72406c6f0 tx=0x7fd7240c55c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:12.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.957+0000 7fd70bfff640 1 -- 172.21.15.161:0/231323191 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd718045070 con 0x7fd724158d30 2024-03-28T23:30:12.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.957+0000 7fd7299fd640 1 -- 172.21.15.161:0/231323191 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd72415c7d0 con 0x7fd724158d30 2024-03-28T23:30:12.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.958+0000 7fd7299fd640 1 -- 172.21.15.161:0/231323191 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd72415a760 con 0x7fd724158d30 2024-03-28T23:30:12.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.958+0000 7fb0f97fa640 1 -- 172.21.15.161:0/3602223949 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb0ec03a070 con 0x7fb0fc14e840 2024-03-28T23:30:12.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.958+0000 7fb0f97fa640 1 --2- 172.21.15.161:0/3602223949 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb0e806b5a0 0x7fb0e806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:12.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.958+0000 7fb0f97fa640 1 -- 172.21.15.161:0/3602223949 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb0ec0abf20 con 0x7fb0fc14e840 2024-03-28T23:30:12.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.958+0000 7fd70bfff640 1 -- 172.21.15.161:0/231323191 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd718007b10 con 0x7fd724158d30 2024-03-28T23:30:12.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.958+0000 7fd70bfff640 1 -- 172.21.15.161:0/231323191 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71804f060 con 0x7fd724158d30 2024-03-28T23:30:12.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.959+0000 7f2b377fe640 1 -- 172.21.15.161:0/2976794870 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2b44053020 con 0x7f2b50150690 2024-03-28T23:30:12.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.960+0000 7fb102f25640 1 -- 172.21.15.161:0/3602223949 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0dc005290 con 0x7fb0fc14e840 2024-03-28T23:30:12.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.960+0000 7f899ffff640 1 -- 172.21.15.161:0/92183516 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b0003b90 con 0x7f89b80c3510 2024-03-28T23:30:12.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.960+0000 7f899ffff640 1 -- 172.21.15.161:0/92183516 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89b0003d30 con 0x7f89b80c3510 2024-03-28T23:30:12.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.960+0000 7f899ffff640 1 -- 172.21.15.161:0/92183516 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b0004240 con 0x7f89b80c3510 2024-03-28T23:30:12.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.961+0000 7fd7299fd640 1 -- 172.21.15.161:0/231323191 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd704005290 con 0x7fd724158d30 2024-03-28T23:30:12.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.963+0000 7f89bd9d0640 1 -- 172.21.15.161:0/92183516 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89b81e4970 con 0x7f89b80c3510 2024-03-28T23:30:12.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.963+0000 7f89bd9d0640 1 -- 172.21.15.161:0/92183516 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89b8159290 con 0x7f89b80c3510 2024-03-28T23:30:12.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.963+0000 7fb100cc1640 1 --2- 172.21.15.161:0/3602223949 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb0e806b5a0 0x7fb0e806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:12.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.964+0000 7fd70bfff640 1 -- 172.21.15.161:0/231323191 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd718048070 con 0x7fd724158d30 2024-03-28T23:30:12.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.964+0000 7fb100cc1640 1 --2- 172.21.15.161:0/3602223949 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb0e806b5a0 0x7fb0e806da60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0000ff0 tx=0x7fb0f0001110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:12.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.964+0000 7fd70bfff640 1 --2- 172.21.15.161:0/231323191 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd71006b500 0x7fd71006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:12.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.964+0000 7fd70bfff640 1 -- 172.21.15.161:0/231323191 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd7180ab260 con 0x7fd724158d30 2024-03-28T23:30:12.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.966+0000 7f89bd9d0640 1 -- 172.21.15.161:0/92183516 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89b804f4b0 con 0x7f89b80c3510 2024-03-28T23:30:12.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.971+0000 7fd722ffd640 1 --2- 172.21.15.161:0/231323191 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd71006b500 0x7fd71006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:12.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.971+0000 7fd722ffd640 1 --2- 172.21.15.161:0/231323191 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd71006b500 0x7fd71006d9c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd714006fd0 tx=0x7fd714008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:12.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.971+0000 7fb0f97fa640 1 -- 172.21.15.161:0/3602223949 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb0ec07ab90 con 0x7fb0fc14e840 2024-03-28T23:30:12.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.988+0000 7fd70bfff640 1 -- 172.21.15.161:0/231323191 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd718079740 con 0x7fd724158d30 2024-03-28T23:30:12.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.988+0000 7f899ffff640 1 -- 172.21.15.161:0/92183516 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f89b00043e0 con 0x7f89b80c3510 2024-03-28T23:30:12.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.988+0000 7f899ffff640 1 --2- 172.21.15.161:0/92183516 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f89a406b520 0x7f89a406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:12.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.989+0000 7f89b6ffd640 1 --2- 172.21.15.161:0/92183516 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f89a406b520 0x7f89a406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:12.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.989+0000 7f899ffff640 1 -- 172.21.15.161:0/92183516 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f89b008b2a0 con 0x7f89b80c3510 2024-03-28T23:30:12.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.990+0000 7f89b6ffd640 1 --2- 172.21.15.161:0/92183516 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f89a406b520 0x7f89a406d9e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f89a8009630 tx=0x7f89a803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:12.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:11.990+0000 7f899ffff640 1 -- 172.21.15.161:0/92183516 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f89b0059ed0 con 0x7f89b80c3510 2024-03-28T23:30:12.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.284+0000 7f456c0ae640 1 -- 172.21.15.161:0/502774829 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f455c006c50 con 0x7f455c00bd40 2024-03-28T23:30:12.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.292+0000 7f455a7fc640 1 -- 172.21.15.161:0/502774829 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f455c006c50 con 0x7f455c00bd40 2024-03-28T23:30:12.301 INFO:teuthology.orchestra.run.smithi161.stdout:55834574855 2024-03-28T23:30:12.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.305+0000 7f456c0ae640 1 -- 172.21.15.161:0/502774829 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f454806b5d0 msgr2=0x7f454806da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:12.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.305+0000 7f456c0ae640 1 --2- 172.21.15.161:0/502774829 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f454806b5d0 0x7f454806da90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f456000c840 tx=0x7f4560009040 comp rx=0 tx=0).stop 2024-03-28T23:30:12.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.305+0000 7f456c0ae640 1 -- 172.21.15.161:0/502774829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f455c00bd40 msgr2=0x7f455c0d8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:12.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.305+0000 7f456c0ae640 1 --2- 172.21.15.161:0/502774829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f455c00bd40 0x7f455c0d8100 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f455400b440 tx=0x7f455400b900 comp rx=0 tx=0).stop 2024-03-28T23:30:12.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.306+0000 7f456c0ae640 1 -- 172.21.15.161:0/502774829 shutdown_connections 2024-03-28T23:30:12.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.306+0000 7f456c0ae640 1 --2- 172.21.15.161:0/502774829 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f454806b5d0 0x7f454806da90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.306+0000 7f456c0ae640 1 --2- 172.21.15.161:0/502774829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f455c00bd40 0x7f455c0d8100 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.306+0000 7f456c0ae640 1 --2- 172.21.15.161:0/502774829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f455c04b100 0x7f455c0ce900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.306+0000 7f456c0ae640 1 --2- 172.21.15.161:0/502774829 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f455c006e90 0x7f455c16f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.306+0000 7f456c0ae640 1 -- 172.21.15.161:0/502774829 >> 172.21.15.161:0/502774829 conn(0x7f455c048240 msgr2=0x7f455c0d6c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:12.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.309+0000 7f456c0ae640 1 -- 172.21.15.161:0/502774829 shutdown_connections 2024-03-28T23:30:12.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.309+0000 7f456c0ae640 1 -- 172.21.15.161:0/502774829 wait complete. 2024-03-28T23:30:12.334 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.2 2024-03-28T23:30:12.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.343+0000 7fd7299fd640 1 -- 172.21.15.161:0/231323191 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fd704005580 con 0x7fd724158d30 2024-03-28T23:30:12.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.343+0000 7fd70bfff640 1 -- 172.21.15.161:0/231323191 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd71807d7e0 con 0x7fd724158d30 2024-03-28T23:30:12.345 INFO:teuthology.orchestra.run.smithi161.stdout:55834574855 2024-03-28T23:30:12.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.348+0000 7fd7299fd640 1 -- 172.21.15.161:0/231323191 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd71006b500 msgr2=0x7fd71006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:12.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.348+0000 7fd7299fd640 1 --2- 172.21.15.161:0/231323191 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd71006b500 0x7fd71006d9c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd714006fd0 tx=0x7fd714008040 comp rx=0 tx=0).stop 2024-03-28T23:30:12.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.348+0000 7fd7299fd640 1 -- 172.21.15.161:0/231323191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd724158d30 msgr2=0x7fd72415ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:12.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.348+0000 7fd7299fd640 1 --2- 172.21.15.161:0/231323191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd724158d30 0x7fd72415ac80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd72406c6f0 tx=0x7fd7240c55c0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.348+0000 7fd7299fd640 1 -- 172.21.15.161:0/231323191 shutdown_connections 2024-03-28T23:30:12.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.348+0000 7fd7299fd640 1 --2- 172.21.15.161:0/231323191 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd71006b500 0x7fd71006d9c0 secure :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd714006fd0 tx=0x7fd714008040 comp rx=0 tx=0).stop 2024-03-28T23:30:12.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.348+0000 7fd7299fd640 1 --2- 172.21.15.161:0/231323191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd724158d30 0x7fd72415ac80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.348+0000 7fd7299fd640 1 --2- 172.21.15.161:0/231323191 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7240a3810 0x7fd7241efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.348+0000 7fd7299fd640 1 --2- 172.21.15.161:0/231323191 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd70c0052e0 0x7fd72415bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.348+0000 7fd7299fd640 1 -- 172.21.15.161:0/231323191 >> 172.21.15.161:0/231323191 conn(0x7fd7240a8820 msgr2=0x7fd7240c3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:12.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.349+0000 7fd7299fd640 1 -- 172.21.15.161:0/231323191 shutdown_connections 2024-03-28T23:30:12.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.349+0000 7fd7299fd640 1 -- 172.21.15.161:0/231323191 wait complete. 2024-03-28T23:30:12.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.349+0000 7f9cab2bc640 1 -- 172.21.15.161:0/2385433305 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f9c8c005580 con 0x7f9ca40c0f60 2024-03-28T23:30:12.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.354+0000 7f9c9affd640 1 -- 172.21.15.161:0/2385433305 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9ca00589b0 con 0x7f9ca40c0f60 2024-03-28T23:30:12.355 INFO:teuthology.orchestra.run.smithi161.stdout:55834574855 2024-03-28T23:30:12.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.357+0000 7f9cab2bc640 1 -- 172.21.15.161:0/2385433305 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9c9006b6e0 msgr2=0x7f9c9006dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:12.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.357+0000 7f9cab2bc640 1 --2- 172.21.15.161:0/2385433305 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9c9006b6e0 0x7f9c9006dba0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c035000 tx=0x7f9c9c030000 comp rx=0 tx=0).stop 2024-03-28T23:30:12.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.357+0000 7f9cab2bc640 1 -- 172.21.15.161:0/2385433305 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ca40c0f60 msgr2=0x7f9ca4071ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:12.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.357+0000 7f9cab2bc640 1 --2- 172.21.15.161:0/2385433305 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ca40c0f60 0x7f9ca4071ec0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9ca000e8c0 tx=0x7f9ca000ed80 comp rx=0 tx=0).stop 2024-03-28T23:30:12.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.357+0000 7f9cab2bc640 1 -- 172.21.15.161:0/2385433305 shutdown_connections 2024-03-28T23:30:12.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.357+0000 7f9cab2bc640 1 --2- 172.21.15.161:0/2385433305 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9c9006b6e0 0x7f9c9006dba0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.357+0000 7f9cab2bc640 1 --2- 172.21.15.161:0/2385433305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ca415a8a0 0x7f9ca4072ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.357+0000 7f9cab2bc640 1 --2- 172.21.15.161:0/2385433305 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ca4153300 0x7f9ca4072570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.357+0000 7f9cab2bc640 1 --2- 172.21.15.161:0/2385433305 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ca40c0f60 0x7f9ca4071ec0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.357+0000 7f9cab2bc640 1 -- 172.21.15.161:0/2385433305 >> 172.21.15.161:0/2385433305 conn(0x7f9ca40c71a0 msgr2=0x7f9ca415b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:12.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.358+0000 7f9cab2bc640 1 -- 172.21.15.161:0/2385433305 shutdown_connections 2024-03-28T23:30:12.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.358+0000 7f9cab2bc640 1 -- 172.21.15.161:0/2385433305 wait complete. 2024-03-28T23:30:12.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.360+0000 7fcddfe3b640 1 -- 172.21.15.161:0/1587801827 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fcdd80a8fc0 con 0x7fcdc4003a90 2024-03-28T23:30:12.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.361+0000 7fcdce7fc640 1 -- 172.21.15.161:0/1587801827 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcdd40798f0 con 0x7fcdc4003a90 2024-03-28T23:30:12.362 INFO:teuthology.orchestra.run.smithi161.stdout:55834574856 2024-03-28T23:30:12.365 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.1 2024-03-28T23:30:12.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.364+0000 7fcddfe3b640 1 -- 172.21.15.161:0/1587801827 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcdbc06b520 msgr2=0x7fcdbc06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:12.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.364+0000 7fcddfe3b640 1 --2- 172.21.15.161:0/1587801827 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcdbc06b520 0x7fcdbc06d9e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80c3270 tx=0x7fcdc8001b00 comp rx=0 tx=0).stop 2024-03-28T23:30:12.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.364+0000 7fcddfe3b640 1 -- 172.21.15.161:0/1587801827 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdc4003a90 msgr2=0x7fcdd80bf9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:12.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.364+0000 7fcddfe3b640 1 --2- 172.21.15.161:0/1587801827 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdc4003a90 0x7fcdd80bf9b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcdd403a000 tx=0x7fcdd4004730 comp rx=0 tx=0).stop 2024-03-28T23:30:12.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.364+0000 7fcddfe3b640 1 -- 172.21.15.161:0/1587801827 shutdown_connections 2024-03-28T23:30:12.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.364+0000 7fcddfe3b640 1 --2- 172.21.15.161:0/1587801827 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcdbc06b520 0x7fcdbc06d9e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.364+0000 7fcddfe3b640 1 --2- 172.21.15.161:0/1587801827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdd80bbd60 0x7fcdd80bc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.364+0000 7fcddfe3b640 1 --2- 172.21.15.161:0/1587801827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdd80bfef0 0x7fcdd80c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.364+0000 7fcddfe3b640 1 --2- 172.21.15.161:0/1587801827 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdc4003a90 0x7fcdd80bf9b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.364+0000 7fcddfe3b640 1 -- 172.21.15.161:0/1587801827 >> 172.21.15.161:0/1587801827 conn(0x7fcdd806d8e0 msgr2=0x7fcdd806b400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:12.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.364+0000 7fcddfe3b640 1 -- 172.21.15.161:0/1587801827 shutdown_connections 2024-03-28T23:30:12.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.364+0000 7fcddfe3b640 1 -- 172.21.15.161:0/1587801827 wait complete. 2024-03-28T23:30:12.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.371+0000 7f26f1bad640 1 -- 172.21.15.161:0/1425582772 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f26cc005580 con 0x7f26e404b8b0 2024-03-28T23:30:12.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.371+0000 7f26e97fa640 1 -- 172.21.15.161:0/1425582772 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f26ec0f0770 con 0x7f26e404b8b0 2024-03-28T23:30:12.373 INFO:teuthology.orchestra.run.smithi161.stdout:55834574855 2024-03-28T23:30:12.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.375+0000 7f26f1bad640 1 -- 172.21.15.161:0/1425582772 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f26d006b5a0 msgr2=0x7f26d006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:12.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.375+0000 7f26f1bad640 1 --2- 172.21.15.161:0/1425582772 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f26d006b5a0 0x7f26d006da60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f26d8005880 tx=0x7f26d8009040 comp rx=0 tx=0).stop 2024-03-28T23:30:12.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.375+0000 7f26f1bad640 1 -- 172.21.15.161:0/1425582772 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26e404b8b0 msgr2=0x7f26e40d7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:12.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.375+0000 7f26f1bad640 1 --2- 172.21.15.161:0/1425582772 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26e404b8b0 0x7f26e40d7760 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f26ec07a9e0 tx=0x7f26ec06a8f0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.375+0000 7f26f1bad640 1 -- 172.21.15.161:0/1425582772 shutdown_connections 2024-03-28T23:30:12.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.375+0000 7f26f1bad640 1 --2- 172.21.15.161:0/1425582772 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f26d006b5a0 0x7f26d006da60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.375+0000 7f26f1bad640 1 --2- 172.21.15.161:0/1425582772 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26e40d5940 0x7f26e416b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.375+0000 7f26f1bad640 1 --2- 172.21.15.161:0/1425582772 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26e404b8b0 0x7f26e40d7760 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.375+0000 7f26f1bad640 1 --2- 172.21.15.161:0/1425582772 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26e400be30 0x7f26e416f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.375+0000 7f26f1bad640 1 -- 172.21.15.161:0/1425582772 >> 172.21.15.161:0/1425582772 conn(0x7f26e402f4a0 msgr2=0x7f26e4048230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:12.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.375+0000 7f26f1bad640 1 -- 172.21.15.161:0/1425582772 shutdown_connections 2024-03-28T23:30:12.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.375+0000 7f26f1bad640 1 -- 172.21.15.161:0/1425582772 wait complete. 2024-03-28T23:30:12.383 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.5 2024-03-28T23:30:12.384 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.3 2024-03-28T23:30:12.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.383+0000 7fb102f25640 1 -- 172.21.15.161:0/3602223949 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fb0dc005580 con 0x7fb0fc14e840 2024-03-28T23:30:12.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.383+0000 7fb0f97fa640 1 -- 172.21.15.161:0/3602223949 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb0ec07a490 con 0x7fb0fc14e840 2024-03-28T23:30:12.385 INFO:teuthology.orchestra.run.smithi161.stdout:55834574855 2024-03-28T23:30:12.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.386+0000 7fb102f25640 1 -- 172.21.15.161:0/3602223949 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb0e806b5a0 msgr2=0x7fb0e806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:12.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.386+0000 7fb102f25640 1 --2- 172.21.15.161:0/3602223949 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb0e806b5a0 0x7fb0e806da60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0000ff0 tx=0x7fb0f0001110 comp rx=0 tx=0).stop 2024-03-28T23:30:12.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.386+0000 7fb102f25640 1 -- 172.21.15.161:0/3602223949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0fc14e840 msgr2=0x7fb0fc147890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:12.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.386+0000 7fb102f25640 1 --2- 172.21.15.161:0/3602223949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0fc14e840 0x7fb0fc147890 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec003ef0 tx=0x7fb0ec0036f0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.386+0000 7fb102f25640 1 -- 172.21.15.161:0/3602223949 shutdown_connections 2024-03-28T23:30:12.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.386+0000 7fb102f25640 1 --2- 172.21.15.161:0/3602223949 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb0e806b5a0 0x7fb0e806da60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.386+0000 7fb102f25640 1 --2- 172.21.15.161:0/3602223949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0fc14e840 0x7fb0fc147890 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.386+0000 7fb102f25640 1 --2- 172.21.15.161:0/3602223949 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0fc147dd0 0x7fb0fc158550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.386+0000 7fb102f25640 1 --2- 172.21.15.161:0/3602223949 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0fc153340 0x7fb0fc153720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.386+0000 7fb102f25640 1 -- 172.21.15.161:0/3602223949 >> 172.21.15.161:0/3602223949 conn(0x7fb0fc0c6670 msgr2=0x7fb0fc0c0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:12.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.386+0000 7fb102f25640 1 -- 172.21.15.161:0/3602223949 shutdown_connections 2024-03-28T23:30:12.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.386+0000 7fb102f25640 1 -- 172.21.15.161:0/3602223949 wait complete. 2024-03-28T23:30:12.394 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.4 2024-03-28T23:30:12.402 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.0 2024-03-28T23:30:12.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.430+0000 7f89bd9d0640 1 -- 172.21.15.161:0/92183516 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f89b80c3a70 con 0x7f89b80c3510 2024-03-28T23:30:12.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.431+0000 7f899ffff640 1 -- 172.21.15.161:0/92183516 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f89b00597d0 con 0x7f89b80c3510 2024-03-28T23:30:12.432 INFO:teuthology.orchestra.run.smithi161.stdout:55834574856 2024-03-28T23:30:12.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.432+0000 7f89bd9d0640 1 -- 172.21.15.161:0/92183516 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f89a406b520 msgr2=0x7f89a406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:12.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.432+0000 7f89bd9d0640 1 --2- 172.21.15.161:0/92183516 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f89a406b520 0x7f89a406d9e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f89a8009630 tx=0x7f89a803d040 comp rx=0 tx=0).stop 2024-03-28T23:30:12.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.432+0000 7f89bd9d0640 1 -- 172.21.15.161:0/92183516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89b80c3510 msgr2=0x7f89b8158d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:12.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.432+0000 7f89bd9d0640 1 --2- 172.21.15.161:0/92183516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89b80c3510 0x7f89b8158d50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f89b000ccf0 tx=0x7f89b000a660 comp rx=0 tx=0).stop 2024-03-28T23:30:12.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.433+0000 7f89bd9d0640 1 -- 172.21.15.161:0/92183516 shutdown_connections 2024-03-28T23:30:12.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.433+0000 7f89bd9d0640 1 --2- 172.21.15.161:0/92183516 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f89a406b520 0x7f89a406d9e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.433+0000 7f89bd9d0640 1 --2- 172.21.15.161:0/92183516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89b80c3510 0x7f89b8158d50 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.433+0000 7f89bd9d0640 1 --2- 172.21.15.161:0/92183516 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89b8071e70 0x7f89b80acdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.433+0000 7f89bd9d0640 1 --2- 172.21.15.161:0/92183516 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89b806e7f0 0x7f89b81e3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.433+0000 7f89bd9d0640 1 -- 172.21.15.161:0/92183516 >> 172.21.15.161:0/92183516 conn(0x7f89b80c3cf0 msgr2=0x7f89b80713d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:12.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.433+0000 7f89bd9d0640 1 -- 172.21.15.161:0/92183516 shutdown_connections 2024-03-28T23:30:12.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.433+0000 7f89bd9d0640 1 -- 172.21.15.161:0/92183516 wait complete. 2024-03-28T23:30:12.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.434+0000 7f2b5531e640 1 -- 172.21.15.161:0/2976794870 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f2b30005580 con 0x7f2b50150690 2024-03-28T23:30:12.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.435+0000 7f2b377fe640 1 -- 172.21.15.161:0/2976794870 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2b4407f110 con 0x7f2b50150690 2024-03-28T23:30:12.437 INFO:teuthology.orchestra.run.smithi161.stdout:55834574855 2024-03-28T23:30:12.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.437+0000 7f2b5531e640 1 -- 172.21.15.161:0/2976794870 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2b3c06b520 msgr2=0x7f2b3c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:12.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.437+0000 7f2b5531e640 1 --2- 172.21.15.161:0/2976794870 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2b3c06b520 0x7f2b3c06d9e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2b5014c720 tx=0x7f2b40009040 comp rx=0 tx=0).stop 2024-03-28T23:30:12.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.437+0000 7f2b5531e640 1 -- 172.21.15.161:0/2976794870 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b50150690 msgr2=0x7f2b501ea540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:12.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.437+0000 7f2b5531e640 1 --2- 172.21.15.161:0/2976794870 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b50150690 0x7f2b501ea540 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2b44037000 tx=0x7f2b440046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.437+0000 7f2b5531e640 1 -- 172.21.15.161:0/2976794870 shutdown_connections 2024-03-28T23:30:12.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.437+0000 7f2b5531e640 1 --2- 172.21.15.161:0/2976794870 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2b3c06b520 0x7f2b3c06d9e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.437+0000 7f2b5531e640 1 --2- 172.21.15.161:0/2976794870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b500733c0 0x7f2b501ea000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.437+0000 7f2b5531e640 1 --2- 172.21.15.161:0/2976794870 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b50150690 0x7f2b501ea540 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.437+0000 7f2b5531e640 1 --2- 172.21.15.161:0/2976794870 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b50072ee0 0x7f2b5014b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:12.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.437+0000 7f2b5531e640 1 -- 172.21.15.161:0/2976794870 >> 172.21.15.161:0/2976794870 conn(0x7f2b500c88b0 msgr2=0x7f2b5014d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:12.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.437+0000 7f2b5531e640 1 -- 172.21.15.161:0/2976794870 shutdown_connections 2024-03-28T23:30:12.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:12.437+0000 7f2b5531e640 1 -- 172.21.15.161:0/2976794870 wait complete. 2024-03-28T23:30:12.446 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.6 2024-03-28T23:30:12.450 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.7 2024-03-28T23:30:13.334 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-28T23:30:13.366 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-28T23:30:13.384 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-28T23:30:13.385 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-28T23:30:13.394 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-28T23:30:13.403 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-28T23:30:13.447 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-28T23:30:13.451 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-28T23:30:13.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.459+0000 7fc593240640 1 Processor -- start 2024-03-28T23:30:13.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.459+0000 7fc593240640 1 -- start start 2024-03-28T23:30:13.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.460+0000 7fc593240640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc58c06c3d0 0x7fc58c14e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.460+0000 7fc593240640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc58c151e20 0x7fc58c15c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.460+0000 7fc593240640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc58c155c00 con 0x7fc58c14ee30 2024-03-28T23:30:13.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.460+0000 7fc593240640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fc58c06d3f0 con 0x7fc58c06c3d0 2024-03-28T23:30:13.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.460+0000 7fc593240640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc58c155920 con 0x7fc58c151e20 2024-03-28T23:30:13.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.460+0000 7fc58bfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc58c14ee30 0x7fc58c151710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59644/0 (socket says 172.21.15.161:59644) 2024-03-28T23:30:13.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.460+0000 7fc590fdc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc58c06c3d0 0x7fc58c14e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.460+0000 7fc58bfff640 1 -- 172.21.15.161:0/3329468985 learned_addr learned my addr 172.21.15.161:0/3329468985 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:13.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.461+0000 7fc5917dd640 1 --2- 172.21.15.161:0/3329468985 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc58c151e20 0x7fc58c15c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.461+0000 7fc590fdc640 1 -- 172.21.15.161:0/3329468985 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc58c151e20 msgr2=0x7fc58c15c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:13.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.461+0000 7fc590fdc640 1 --2- 172.21.15.161:0/3329468985 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc58c151e20 0x7fc58c15c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.461+0000 7fc590fdc640 1 -- 172.21.15.161:0/3329468985 >> v1:172.21.15.161:6789/0 conn(0x7fc58c14ee30 legacy=0x7fc58c151710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.461+0000 7fc590fdc640 1 -- 172.21.15.161:0/3329468985 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc58c06d080 con 0x7fc58c06c3d0 2024-03-28T23:30:13.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.462+0000 7fc590fdc640 1 --2- 172.21.15.161:0/3329468985 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc58c06c3d0 0x7fc58c14e8f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc5800097a0 tx=0x7fc58002cb60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ff251fb3c00ce37 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:13.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.462+0000 7fc58b7fe640 1 -- 172.21.15.161:0/3329468985 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58002e920 con 0x7fc58c06c3d0 2024-03-28T23:30:13.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.462+0000 7fc58b7fe640 1 -- 172.21.15.161:0/3329468985 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc58002eac0 con 0x7fc58c06c3d0 2024-03-28T23:30:13.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.462+0000 7fc58b7fe640 1 -- 172.21.15.161:0/3329468985 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58002d440 con 0x7fc58c06c3d0 2024-03-28T23:30:13.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.463+0000 7fc593240640 1 -- 172.21.15.161:0/3329468985 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc58c06c3d0 msgr2=0x7fc58c14e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.463+0000 7fc593240640 1 --2- 172.21.15.161:0/3329468985 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc58c06c3d0 0x7fc58c14e8f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc5800097a0 tx=0x7fc58002cb60 comp rx=0 tx=0).stop 2024-03-28T23:30:13.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.463+0000 7fc593240640 1 -- 172.21.15.161:0/3329468985 shutdown_connections 2024-03-28T23:30:13.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.463+0000 7fc593240640 1 --2- 172.21.15.161:0/3329468985 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc58c151e20 0x7fc58c15c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.463+0000 7fc593240640 1 --2- 172.21.15.161:0/3329468985 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc58c06c3d0 0x7fc58c14e8f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.463+0000 7fc593240640 1 -- 172.21.15.161:0/3329468985 >> 172.21.15.161:0/3329468985 conn(0x7fc58c0c3cf0 msgr2=0x7fc58c0c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:13.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.463+0000 7fc593240640 1 -- 172.21.15.161:0/3329468985 shutdown_connections 2024-03-28T23:30:13.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.464+0000 7fc593240640 1 -- 172.21.15.161:0/3329468985 wait complete. 2024-03-28T23:30:13.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.465+0000 7fc593240640 1 Processor -- start 2024-03-28T23:30:13.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.466+0000 7fc593240640 1 -- start start 2024-03-28T23:30:13.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.467+0000 7fc593240640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc58c06c3d0 0x7fc58c1e7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.467+0000 7fc593240640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc58c14ee30 0x7fc58c152ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.467+0000 7fc593240640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc58c151e20 0x7fc58c150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.467+0000 7fc593240640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc58c0c1d70 con 0x7fc58c151e20 2024-03-28T23:30:13.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.467+0000 7fc593240640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc58c0c1bf0 con 0x7fc58c14ee30 2024-03-28T23:30:13.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.467+0000 7fc593240640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc58c072950 con 0x7fc58c06c3d0 2024-03-28T23:30:13.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.467+0000 7fc590fdc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc58c06c3d0 0x7fc58c1e7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.467+0000 7fc590fdc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc58c06c3d0 0x7fc58c1e7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:37018/0 (socket says 172.21.15.161:37018) 2024-03-28T23:30:13.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.467+0000 7fc590fdc640 1 -- 172.21.15.161:0/1273977562 learned_addr learned my addr 172.21.15.161:0/1273977562 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:13.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.467+0000 7fc58bfff640 1 --2- 172.21.15.161:0/1273977562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc58c14ee30 0x7fc58c152ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.468+0000 7fc590fdc640 1 -- 172.21.15.161:0/1273977562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc58c14ee30 msgr2=0x7fc58c152ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.468+0000 7fc590fdc640 1 --2- 172.21.15.161:0/1273977562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc58c14ee30 0x7fc58c152ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.468+0000 7fc590fdc640 1 -- 172.21.15.161:0/1273977562 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc58c151e20 msgr2=0x7fc58c150ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T23:30:13.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.468+0000 7fc590fdc640 1 --2- 172.21.15.161:0/1273977562 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc58c151e20 0x7fc58c150ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.468+0000 7fc590fdc640 1 -- 172.21.15.161:0/1273977562 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc58c1e6b00 con 0x7fc58c06c3d0 2024-03-28T23:30:13.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.468+0000 7fc590fdc640 1 --2- 172.21.15.161:0/1273977562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc58c06c3d0 0x7fc58c1e7570 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc580038040 tx=0x7fc5800046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:13.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.469+0000 7fc5897fa640 1 -- 172.21.15.161:0/1273977562 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc580045030 con 0x7fc58c06c3d0 2024-03-28T23:30:13.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.469+0000 7fc5897fa640 1 -- 172.21.15.161:0/1273977562 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc58002e720 con 0x7fc58c06c3d0 2024-03-28T23:30:13.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.469+0000 7fc5897fa640 1 -- 172.21.15.161:0/1273977562 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5800416a0 con 0x7fc58c06c3d0 2024-03-28T23:30:13.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.469+0000 7fc593240640 1 -- 172.21.15.161:0/1273977562 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc58c0c15e0 con 0x7fc58c06c3d0 2024-03-28T23:30:13.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.470+0000 7fc593240640 1 -- 172.21.15.161:0/1273977562 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc58c1e7050 con 0x7fc58c06c3d0 2024-03-28T23:30:13.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.472+0000 7fc593240640 1 -- 172.21.15.161:0/1273977562 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc58c04f4b0 con 0x7fc58c06c3d0 2024-03-28T23:30:13.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.473+0000 7fc5897fa640 1 -- 172.21.15.161:0/1273977562 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc580048070 con 0x7fc58c06c3d0 2024-03-28T23:30:13.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.476+0000 7fc5897fa640 1 --2- 172.21.15.161:0/1273977562 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc57006b5a0 0x7fc57006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.477+0000 7fc58bfff640 1 --2- 172.21.15.161:0/1273977562 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc57006b5a0 0x7fc57006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.480+0000 7fc58bfff640 1 --2- 172.21.15.161:0/1273977562 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc57006b5a0 0x7fc57006da60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc58c1e4830 tx=0x7fc578009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:13.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.483+0000 7fc5897fa640 1 -- 172.21.15.161:0/1273977562 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc5800acbe0 con 0x7fc58c06c3d0 2024-03-28T23:30:13.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.484+0000 7fc5897fa640 1 -- 172.21.15.161:0/1273977562 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc58007b790 con 0x7fc58c06c3d0 2024-03-28T23:30:13.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.519+0000 7f0131476640 1 Processor -- start 2024-03-28T23:30:13.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.519+0000 7f0131476640 1 -- start start 2024-03-28T23:30:13.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.520+0000 7f0131476640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f012c151fd0 0x7f012c072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.520+0000 7f0131476640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f012c0aa960 0x7f012c06b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.520+0000 7f0131476640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f012c0c1eb0 con 0x7f012c0aa4e0 2024-03-28T23:30:13.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.520+0000 7f0131476640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f012c06bda0 con 0x7f012c0aa960 2024-03-28T23:30:13.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.520+0000 7f0131476640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f012c0c2190 con 0x7f012c151fd0 2024-03-28T23:30:13.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.520+0000 7f012affd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f012c151fd0 0x7f012c072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.520+0000 7f012affd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f012c151fd0 0x7f012c072e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:37030/0 (socket says 172.21.15.161:37030) 2024-03-28T23:30:13.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.520+0000 7f012affd640 1 -- 172.21.15.161:0/1884937294 learned_addr learned my addr 172.21.15.161:0/1884937294 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:13.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.521+0000 7f012affd640 1 -- 172.21.15.161:0/1884937294 >> v2:172.21.15.170:3300/0 conn(0x7f012c0aa960 msgr2=0x7f012c06b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:13.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.521+0000 7f012affd640 1 --2- 172.21.15.161:0/1884937294 >> v2:172.21.15.170:3300/0 conn(0x7f012c0aa960 0x7f012c06b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.521+0000 7f012affd640 1 -- 172.21.15.161:0/1884937294 >> v1:172.21.15.170:6789/0 conn(0x7f012c0aa4e0 legacy=0x7f012c06b150 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T23:30:13.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.521+0000 7f012affd640 1 -- 172.21.15.161:0/1884937294 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f012c0c2470 con 0x7f012c151fd0 2024-03-28T23:30:13.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.523+0000 7f012affd640 1 --2- 172.21.15.161:0/1884937294 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f012c151fd0 0x7f012c072e10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0124009bd0 tx=0x7f012402d340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c37b85dcb737e03e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:13.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.523+0000 7f0129ffb640 1 -- 172.21.15.161:0/1884937294 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01240353f0 con 0x7f012c151fd0 2024-03-28T23:30:13.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.523+0000 7f0129ffb640 1 -- 172.21.15.161:0/1884937294 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0124035590 con 0x7f012c151fd0 2024-03-28T23:30:13.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.524+0000 7f0131476640 1 -- 172.21.15.161:0/1884937294 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f012c151fd0 msgr2=0x7f012c072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.524+0000 7f0131476640 1 --2- 172.21.15.161:0/1884937294 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f012c151fd0 0x7f012c072e10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0124009bd0 tx=0x7f012402d340 comp rx=0 tx=0).stop 2024-03-28T23:30:13.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.524+0000 7f0131476640 1 -- 172.21.15.161:0/1884937294 shutdown_connections 2024-03-28T23:30:13.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.524+0000 7f0131476640 1 --2- 172.21.15.161:0/1884937294 >> v2:172.21.15.170:3300/0 conn(0x7f012c0aa960 0x7f012c06b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.524+0000 7f0131476640 1 --2- 172.21.15.161:0/1884937294 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f012c151fd0 0x7f012c072e10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.524+0000 7f0131476640 1 -- 172.21.15.161:0/1884937294 >> 172.21.15.161:0/1884937294 conn(0x7f012c06d8e0 msgr2=0x7f012c06dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:13.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.525+0000 7f0131476640 1 -- 172.21.15.161:0/1884937294 shutdown_connections 2024-03-28T23:30:13.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.526+0000 7f0131476640 1 -- 172.21.15.161:0/1884937294 wait complete. 2024-03-28T23:30:13.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.528+0000 7f0131476640 1 Processor -- start 2024-03-28T23:30:13.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.528+0000 7f0131476640 1 -- start start 2024-03-28T23:30:13.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.529+0000 7f0131476640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f012c0aa4e0 0x7f012c1dd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.530+0000 7f012affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f012c0aa4e0 0x7f012c1dd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.530+0000 7f012affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f012c0aa4e0 0x7f012c1dd010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54320/0 (socket says 172.21.15.161:54320) 2024-03-28T23:30:13.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.530+0000 7f012affd640 1 -- 172.21.15.161:0/3804975014 learned_addr learned my addr 172.21.15.161:0/3804975014 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:13.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.530+0000 7f0131476640 1 --2- 172.21.15.161:0/3804975014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f012c0aa960 0x7f012c152800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.530+0000 7f0131476640 1 --2- 172.21.15.161:0/3804975014 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f012c1e0b70 0x7f012c152d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.530+0000 7f0131476640 1 -- 172.21.15.161:0/3804975014 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f012c0c1ad0 con 0x7f012c0aa4e0 2024-03-28T23:30:13.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.530+0000 7f0131476640 1 -- 172.21.15.161:0/3804975014 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f012c0c2a30 con 0x7f012c0aa960 2024-03-28T23:30:13.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.530+0000 7f0131476640 1 -- 172.21.15.161:0/3804975014 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f012c0c1c50 con 0x7f012c1e0b70 2024-03-28T23:30:13.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.531+0000 7f012affd640 1 -- 172.21.15.161:0/3804975014 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f012c1e0b70 msgr2=0x7f012c152d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T23:30:13.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.531+0000 7f012affd640 1 --2- 172.21.15.161:0/3804975014 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f012c1e0b70 0x7f012c152d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.531+0000 7f012affd640 1 -- 172.21.15.161:0/3804975014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f012c0aa960 msgr2=0x7f012c152800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T23:30:13.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.531+0000 7f012affd640 1 --2- 172.21.15.161:0/3804975014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f012c0aa960 0x7f012c152800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.531+0000 7f012affd640 1 -- 172.21.15.161:0/3804975014 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f012c1de7e0 con 0x7f012c0aa4e0 2024-03-28T23:30:13.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.531+0000 7f012affd640 1 --2- 172.21.15.161:0/3804975014 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f012c0aa4e0 0x7f012c1dd010 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0124037040 tx=0x7f0124004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:13.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.532+0000 7f01137fe640 1 -- 172.21.15.161:0/3804975014 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0124043020 con 0x7f012c0aa4e0 2024-03-28T23:30:13.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.532+0000 7f0131476640 1 -- 172.21.15.161:0/3804975014 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f012c1deac0 con 0x7f012c0aa4e0 2024-03-28T23:30:13.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.535+0000 7f0131476640 1 -- 172.21.15.161:0/3804975014 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f012c1e01e0 con 0x7f012c0aa4e0 2024-03-28T23:30:13.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.535+0000 7f01137fe640 1 -- 172.21.15.161:0/3804975014 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0124008af0 con 0x7f012c0aa4e0 2024-03-28T23:30:13.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.536+0000 7f01137fe640 1 -- 172.21.15.161:0/3804975014 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012403e440 con 0x7f012c0aa4e0 2024-03-28T23:30:13.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.536+0000 7f01137fe640 1 -- 172.21.15.161:0/3804975014 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0124046020 con 0x7f012c0aa4e0 2024-03-28T23:30:13.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.536+0000 7f01137fe640 1 --2- 172.21.15.161:0/3804975014 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f011806b5d0 0x7f011806da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.537+0000 7f01137fe640 1 -- 172.21.15.161:0/3804975014 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0124035740 con 0x7f012c0aa4e0 2024-03-28T23:30:13.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.537+0000 7f012a7fc640 1 --2- 172.21.15.161:0/3804975014 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f011806b5d0 0x7f011806da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.538+0000 7f012a7fc640 1 --2- 172.21.15.161:0/3804975014 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f011806b5d0 0x7f011806da90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f012c1dcb30 tx=0x7f01200079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:13.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.538+0000 7f0131476640 1 -- 172.21.15.161:0/3804975014 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f010c005290 con 0x7f012c0aa4e0 2024-03-28T23:30:13.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.548+0000 7f01137fe640 1 -- 172.21.15.161:0/3804975014 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f012407bdd0 con 0x7f012c0aa4e0 2024-03-28T23:30:13.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.585+0000 7eff1759e640 1 Processor -- start 2024-03-28T23:30:13.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.585+0000 7eff1759e640 1 -- start start 2024-03-28T23:30:13.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.587+0000 7eff1759e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7eff18072850 0x7eff180abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.587+0000 7eff1759e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff180c1a20 con 0x7eff180ab920 2024-03-28T23:30:13.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.587+0000 7eff1759e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff180c1d00 con 0x7eff180c13a0 2024-03-28T23:30:13.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.587+0000 7eff1759e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7eff180710b0 con 0x7eff18072850 2024-03-28T23:30:13.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.587+0000 7eff1659c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7eff180ab920 0x7eff18153600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59648/0 (socket says 172.21.15.161:59648) 2024-03-28T23:30:13.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.587+0000 7eff1659c640 1 -- 172.21.15.161:0/2385736066 learned_addr learned my addr 172.21.15.161:0/2385736066 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:13.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.588+0000 7eff1559a640 1 -- 172.21.15.161:0/2385736066 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 125945700 0 0) 0x7eff180c1a20 con 0x7eff180ab920 2024-03-28T23:30:13.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.588+0000 7eff1559a640 1 -- 172.21.15.161:0/2385736066 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff00003520 con 0x7eff180ab920 2024-03-28T23:30:13.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.588+0000 7eff1559a640 1 -- 172.21.15.161:0/2385736066 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1054352408 0 0) 0x7eff180c1d00 con 0x7eff180c13a0 2024-03-28T23:30:13.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.588+0000 7eff1559a640 1 -- 172.21.15.161:0/2385736066 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff180c1a20 con 0x7eff180c13a0 2024-03-28T23:30:13.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.588+0000 7fa67433e640 1 Processor -- start 2024-03-28T23:30:13.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.588+0000 7fa67433e640 1 -- start start 2024-03-28T23:30:13.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.588+0000 7eff1559a640 1 -- 172.21.15.161:0/2385736066 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 265052218 0 0) 0x7eff00003520 con 0x7eff180ab920 2024-03-28T23:30:13.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.588+0000 7eff1559a640 1 -- 172.21.15.161:0/2385736066 >> v2:172.21.15.170:3300/0 conn(0x7eff18072850 msgr2=0x7eff180abd00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T23:30:13.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.588+0000 7eff1559a640 1 --2- 172.21.15.161:0/2385736066 >> v2:172.21.15.170:3300/0 conn(0x7eff18072850 0x7eff180abd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.588+0000 7eff1559a640 1 -- 172.21.15.161:0/2385736066 >> v1:172.21.15.170:6789/0 conn(0x7eff180c13a0 legacy=0x7eff1815b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.588+0000 7eff1559a640 1 -- 172.21.15.161:0/2385736066 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff181530e0 con 0x7eff180ab920 2024-03-28T23:30:13.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.589+0000 7eff1559a640 1 -- 172.21.15.161:0/2385736066 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7eff080019f0 con 0x7eff180ab920 2024-03-28T23:30:13.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.589+0000 7eff1559a640 1 -- 172.21.15.161:0/2385736066 >> v1:172.21.15.161:6789/0 conn(0x7eff180ab920 legacy=0x7eff18153600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.589+0000 7eff1559a640 1 --2- 172.21.15.161:0/2385736066 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff00003b90 0x7eff00003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.589+0000 7eff1559a640 1 --2- 172.21.15.161:0/2385736066 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff00004800 0x7eff00004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.589+0000 7eff1559a640 1 --2- 172.21.15.161:0/2385736066 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff00005410 0x7eff0000ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.589+0000 7eff1559a640 1 -- 172.21.15.161:0/2385736066 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7eff180710b0 con 0x7eff00003b90 2024-03-28T23:30:13.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.589+0000 7eff1559a640 1 -- 172.21.15.161:0/2385736066 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7eff0000c350 con 0x7eff00004800 2024-03-28T23:30:13.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.589+0000 7eff1559a640 1 -- 172.21.15.161:0/2385736066 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7eff0000c550 con 0x7eff00005410 2024-03-28T23:30:13.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.589+0000 7eff1659c640 1 --2- 172.21.15.161:0/2385736066 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff00003b90 0x7eff00003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.590+0000 7fa67433e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa66c15cd40 0x7fa66c15d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.590+0000 7fa67433e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa66c071e80 0x7fa66c164070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.590+0000 7fa67433e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa66c0c3200 con 0x7fa66c15d660 2024-03-28T23:30:13.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.590+0000 7fa67433e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa66c15dac0 con 0x7fa66c071e80 2024-03-28T23:30:13.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.590+0000 7fa67433e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa66c0c2fb0 con 0x7fa66c15cd40 2024-03-28T23:30:13.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.590+0000 7fa6720da640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa66c15cd40 0x7fa66c15d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.590+0000 7fa6720da640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa66c15cd40 0x7fa66c15d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:37042/0 (socket says 172.21.15.161:37042) 2024-03-28T23:30:13.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.590+0000 7fa6720da640 1 -- 172.21.15.161:0/1849509481 learned_addr learned my addr 172.21.15.161:0/1849509481 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:13.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.591+0000 7fa6728db640 1 --2- 172.21.15.161:0/1849509481 >> v2:172.21.15.161:3300/0 conn(0x7fa66c071e80 0x7fa66c164070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.590+0000 7eff1659c640 1 -- 172.21.15.161:0/2385736066 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff00005410 msgr2=0x7eff0000ba70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T23:30:13.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.590+0000 7eff1659c640 1 --2- 172.21.15.161:0/2385736066 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff00005410 0x7eff0000ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.590+0000 7eff1659c640 1 -- 172.21.15.161:0/2385736066 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff00004800 msgr2=0x7eff00004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T23:30:13.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.591+0000 7fa6728db640 1 -- 172.21.15.161:0/1849509481 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa66c15cd40 msgr2=0x7fa66c15d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:13.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.591+0000 7fa6728db640 1 --2- 172.21.15.161:0/1849509481 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa66c15cd40 0x7fa66c15d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.591+0000 7fa6728db640 1 -- 172.21.15.161:0/1849509481 >> v1:172.21.15.161:6789/0 conn(0x7fa66c15d660 legacy=0x7fa66c071770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.590+0000 7eff1659c640 1 --2- 172.21.15.161:0/2385736066 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff00004800 0x7eff00004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.590+0000 7eff1659c640 1 -- 172.21.15.161:0/2385736066 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff00003520 con 0x7eff00003b90 2024-03-28T23:30:13.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.591+0000 7fa6728db640 1 -- 172.21.15.161:0/1849509481 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa66c0c34e0 con 0x7fa66c071e80 2024-03-28T23:30:13.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.592+0000 7eff1659c640 1 --2- 172.21.15.161:0/2385736066 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff00003b90 0x7eff00003f90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7eff0800a330 tx=0x7eff00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2370647f41eab15c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:13.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.592+0000 7eff16d9d640 1 -- 172.21.15.161:0/2385736066 reap_dead start 2024-03-28T23:30:13.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.592+0000 7eff1559a640 1 -- 172.21.15.161:0/2385736066 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff0800a370 con 0x7eff00003b90 2024-03-28T23:30:13.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.592+0000 7eff1559a640 1 -- 172.21.15.161:0/2385736066 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff0803a070 con 0x7eff00003b90 2024-03-28T23:30:13.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.593+0000 7fa6728db640 1 --2- 172.21.15.161:0/1849509481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa66c071e80 0x7fa66c164070 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa664009d90 tx=0x7fa66402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=384f88cefa0035e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:13.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.594+0000 7fa6710d8640 1 -- 172.21.15.161:0/1849509481 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa664039070 con 0x7fa66c071e80 2024-03-28T23:30:13.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.594+0000 7fa6710d8640 1 -- 172.21.15.161:0/1849509481 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa66402cb60 con 0x7fa66c071e80 2024-03-28T23:30:13.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.594+0000 7f3ebaeed640 1 Processor -- start 2024-03-28T23:30:13.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.594+0000 7f3ebaeed640 1 -- start start 2024-03-28T23:30:13.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.594+0000 7eff1559a640 1 -- 172.21.15.161:0/2385736066 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff080358b0 con 0x7eff00003b90 2024-03-28T23:30:13.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.594+0000 7eff1759e640 1 -- 172.21.15.161:0/2385736066 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff00003b90 msgr2=0x7eff00003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.594+0000 7eff1759e640 1 --2- 172.21.15.161:0/2385736066 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff00003b90 0x7eff00003f90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7eff0800a330 tx=0x7eff00005830 comp rx=0 tx=0).stop 2024-03-28T23:30:13.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.594+0000 7eff1759e640 1 -- 172.21.15.161:0/2385736066 shutdown_connections 2024-03-28T23:30:13.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.594+0000 7eff1759e640 1 --2- 172.21.15.161:0/2385736066 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff00003b90 0x7eff00003f90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.594+0000 7eff1759e640 1 -- 172.21.15.161:0/2385736066 >> 172.21.15.161:0/2385736066 conn(0x7eff180c71a0 msgr2=0x7eff180c3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:13.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.595+0000 7fa67433e640 1 -- 172.21.15.161:0/1849509481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa66c071e80 msgr2=0x7fa66c164070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.595+0000 7fa67433e640 1 --2- 172.21.15.161:0/1849509481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa66c071e80 0x7fa66c164070 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa664009d90 tx=0x7fa66402bdb0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.595+0000 7fa67433e640 1 -- 172.21.15.161:0/1849509481 shutdown_connections 2024-03-28T23:30:13.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.595+0000 7eff1759e640 1 -- 172.21.15.161:0/2385736066 shutdown_connections 2024-03-28T23:30:13.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.595+0000 7fa67433e640 1 --2- 172.21.15.161:0/1849509481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa66c071e80 0x7fa66c164070 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.595+0000 7fa67433e640 1 --2- 172.21.15.161:0/1849509481 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa66c15cd40 0x7fa66c15d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.595+0000 7f3ebaeed640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3eb4156c60 0x7f3eb40c2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.595+0000 7f3ebaeed640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3eb41571a0 con 0x7f3eb41504d0 2024-03-28T23:30:13.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.595+0000 7f3ebaeed640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3eb4157440 con 0x7f3eb4156880 2024-03-28T23:30:13.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.595+0000 7f3ebaeed640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f3eb406b150 con 0x7f3eb4156c60 2024-03-28T23:30:13.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.595+0000 7fa67433e640 1 -- 172.21.15.161:0/1849509481 >> 172.21.15.161:0/1849509481 conn(0x7fa66c06c090 msgr2=0x7fa66c0c1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:13.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.595+0000 7eff1759e640 1 -- 172.21.15.161:0/2385736066 wait complete. 2024-03-28T23:30:13.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.596+0000 7f3eb8c89640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f3eb4156880 0x7f3eb4070e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:46174/0 (socket says 172.21.15.161:46174) 2024-03-28T23:30:13.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.596+0000 7f3eb8c89640 1 -- 172.21.15.161:0/2121643436 learned_addr learned my addr 172.21.15.161:0/2121643436 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:13.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.596+0000 7f3eb3fff640 1 --2- 172.21.15.161:0/2121643436 >> v2:172.21.15.161:3300/0 conn(0x7f3eb4156c60 0x7f3eb40c2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.596+0000 7f3eb37fe640 1 -- 172.21.15.161:0/2121643436 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1599904223 0 0) 0x7f3eb4157440 con 0x7f3eb4156880 2024-03-28T23:30:13.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.596+0000 7f3eb37fe640 1 -- 172.21.15.161:0/2121643436 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ea4003520 con 0x7f3eb4156880 2024-03-28T23:30:13.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.596+0000 7f3eb37fe640 1 -- 172.21.15.161:0/2121643436 <== mon.1 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2270702437 0 0) 0x7f3ea4003520 con 0x7f3eb4156880 2024-03-28T23:30:13.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.596+0000 7f3eb37fe640 1 -- 172.21.15.161:0/2121643436 >> v2:172.21.15.161:3300/0 conn(0x7f3eb4156c60 msgr2=0x7f3eb40c2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:13.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.596+0000 7f3eb37fe640 1 --2- 172.21.15.161:0/2121643436 >> v2:172.21.15.161:3300/0 conn(0x7f3eb4156c60 0x7f3eb40c2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.596+0000 7f3eb37fe640 1 -- 172.21.15.161:0/2121643436 >> v1:172.21.15.161:6789/0 conn(0x7f3eb41504d0 legacy=0x7f3eb406b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.596+0000 7f3eb37fe640 1 -- 172.21.15.161:0/2121643436 --> v1:172.21.15.170:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eb4153bc0 con 0x7f3eb4156880 2024-03-28T23:30:13.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.596+0000 7f3eb37fe640 1 -- 172.21.15.161:0/2121643436 <== mon.1 v1:172.21.15.170:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f3ea0001670 con 0x7f3eb4156880 2024-03-28T23:30:13.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.597+0000 7f3eb37fe640 1 -- 172.21.15.161:0/2121643436 >> v1:172.21.15.170:6789/0 conn(0x7f3eb4156880 legacy=0x7f3eb4070e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.597+0000 7f3eb37fe640 1 --2- 172.21.15.161:0/2121643436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ea4003a90 0x7f3ea4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.597+0000 7f3eb37fe640 1 --2- 172.21.15.161:0/2121643436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ea4004800 0x7f3ea4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.597+0000 7f3eb37fe640 1 --2- 172.21.15.161:0/2121643436 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ea40051c0 0x7f3ea400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.597+0000 7f3eb8c89640 1 --2- 172.21.15.161:0/2121643436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ea4003a90 0x7f3ea4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.597+0000 7f3eb37fe640 1 -- 172.21.15.161:0/2121643436 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb406b150 con 0x7f3ea40051c0 2024-03-28T23:30:13.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.597+0000 7f3eb37fe640 1 -- 172.21.15.161:0/2121643436 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea400c160 con 0x7f3ea4004800 2024-03-28T23:30:13.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.597+0000 7f3eb37fe640 1 -- 172.21.15.161:0/2121643436 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3ea400c360 con 0x7f3ea4003a90 2024-03-28T23:30:13.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.597+0000 7f3eb3fff640 1 --2- 172.21.15.161:0/2121643436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ea4004800 0x7f3ea4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.597+0000 7f3eb8c89640 1 -- 172.21.15.161:0/2121643436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ea4004800 msgr2=0x7f3ea4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:13.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.597+0000 7f3eb8c89640 1 --2- 172.21.15.161:0/2121643436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ea4004800 0x7f3ea4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.597+0000 7f3eb8c89640 1 -- 172.21.15.161:0/2121643436 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ea40051c0 msgr2=0x7f3ea400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:13.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.597+0000 7f3eb8c89640 1 --2- 172.21.15.161:0/2121643436 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ea40051c0 0x7f3ea400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.597+0000 7f3eb8c89640 1 -- 172.21.15.161:0/2121643436 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ea4003520 con 0x7f3ea4003a90 2024-03-28T23:30:13.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.597+0000 7f3eb948a640 1 -- 172.21.15.161:0/2121643436 reap_dead start 2024-03-28T23:30:13.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.598+0000 7f3eb3fff640 1 --2- 172.21.15.161:0/2121643436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ea4004800 0x7f3ea4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:13.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.599+0000 7f3eb8c89640 1 --2- 172.21.15.161:0/2121643436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ea4003a90 0x7f3ea4003e70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0004fd0 tx=0x7f3ea4005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2570ccc98ba2354 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:13.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.599+0000 7fa67433e640 1 -- 172.21.15.161:0/1849509481 shutdown_connections 2024-03-28T23:30:13.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.599+0000 7f3eb37fe640 1 -- 172.21.15.161:0/2121643436 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0004ba0 con 0x7f3ea4003a90 2024-03-28T23:30:13.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.599+0000 7f3eb37fe640 1 -- 172.21.15.161:0/2121643436 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ea00058d0 con 0x7f3ea4003a90 2024-03-28T23:30:13.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.600+0000 7f3ebaeed640 1 -- 172.21.15.161:0/2121643436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ea4003a90 msgr2=0x7f3ea4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.600+0000 7f3ebaeed640 1 --2- 172.21.15.161:0/2121643436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ea4003a90 0x7f3ea4003e70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0004fd0 tx=0x7f3ea4005830 comp rx=0 tx=0).stop 2024-03-28T23:30:13.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.599+0000 7fa67433e640 1 -- 172.21.15.161:0/1849509481 wait complete. 2024-03-28T23:30:13.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.601+0000 7fa67433e640 1 Processor -- start 2024-03-28T23:30:13.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.601+0000 7fa67433e640 1 -- start start 2024-03-28T23:30:13.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.602+0000 7fa67433e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa66c071e80 0x7fa66c0ab0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.602+0000 7fa67433e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa66c15cd40 0x7fa66c151fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.602+0000 7fa67433e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa66c15d660 0x7fa66c152500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.602+0000 7fa67433e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa66c0acee0 con 0x7fa66c15cd40 2024-03-28T23:30:13.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.602+0000 7fa67433e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa66c0ac920 con 0x7fa66c15d660 2024-03-28T23:30:13.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.602+0000 7fa67433e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa66c1571c0 con 0x7fa66c071e80 2024-03-28T23:30:13.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.602+0000 7f3ebaeed640 1 -- 172.21.15.161:0/2121643436 shutdown_connections 2024-03-28T23:30:13.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.602+0000 7f3ebaeed640 1 --2- 172.21.15.161:0/2121643436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ea4003a90 0x7f3ea4003e70 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.602+0000 7f3ebaeed640 1 -- 172.21.15.161:0/2121643436 >> 172.21.15.161:0/2121643436 conn(0x7f3eb406d8e0 msgr2=0x7f3eb406dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:13.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.602+0000 7f3ebaeed640 1 -- 172.21.15.161:0/2121643436 shutdown_connections 2024-03-28T23:30:13.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.603+0000 7f3ebaeed640 1 -- 172.21.15.161:0/2121643436 wait complete. 2024-03-28T23:30:13.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.602+0000 7fa6728db640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa66c15d660 0x7fa66c152500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.602+0000 7fa6728db640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa66c15d660 0x7fa66c152500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:39336/0 (socket says 172.21.15.161:39336) 2024-03-28T23:30:13.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.602+0000 7fa6728db640 1 -- 172.21.15.161:0/1011259720 learned_addr learned my addr 172.21.15.161:0/1011259720 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:13.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.603+0000 7fa6720da640 1 --2- 172.21.15.161:0/1011259720 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa66c071e80 0x7fa66c0ab0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.603+0000 7fa6720da640 1 -- 172.21.15.161:0/1011259720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa66c15d660 msgr2=0x7fa66c152500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.603+0000 7fa6720da640 1 --2- 172.21.15.161:0/1011259720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa66c15d660 0x7fa66c152500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.603+0000 7fa6720da640 1 -- 172.21.15.161:0/1011259720 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa66c15cd40 msgr2=0x7fa66c151fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.603+0000 7fa6718d9640 1 --2- 172.21.15.161:0/1011259720 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa66c15cd40 0x7fa66c151fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.603+0000 7fa6720da640 1 --2- 172.21.15.161:0/1011259720 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa66c15cd40 0x7fa66c151fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.603+0000 7fa6720da640 1 -- 172.21.15.161:0/1011259720 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa66c1f8600 con 0x7fa66c071e80 2024-03-28T23:30:13.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.604+0000 7fa6720da640 1 --2- 172.21.15.161:0/1011259720 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa66c071e80 0x7fa66c0ab0a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa66800e600 tx=0x7fa66800eac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:13.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.604+0000 7fa6718d9640 1 --2- 172.21.15.161:0/1011259720 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa66c15cd40 0x7fa66c151fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:13.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.604+0000 7f3ebaeed640 1 Processor -- start 2024-03-28T23:30:13.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.604+0000 7f3ebaeed640 1 -- start start 2024-03-28T23:30:13.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.606+0000 7f3ebaeed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ea4003a90 0x7f3eb41521d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.606+0000 7f3ebaeed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3eb41eb900 0x7f3eb41413e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.606+0000 7f3ebaeed640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3eb414fe30 0x7f3eb4141920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.606+0000 7f3eb8c89640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ea4003a90 0x7f3eb41521d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.607+0000 7f3eb8c89640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ea4003a90 0x7f3eb41521d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:39344/0 (socket says 172.21.15.161:39344) 2024-03-28T23:30:13.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.607+0000 7f3eb8c89640 1 -- 172.21.15.161:0/1616024629 learned_addr learned my addr 172.21.15.161:0/1616024629 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:13.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.607+0000 7f3eb948a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3eb414fe30 0x7f3eb4141920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.607+0000 7f3ebaeed640 1 -- 172.21.15.161:0/1616024629 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb40c2f10 con 0x7f3eb414fe30 2024-03-28T23:30:13.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.607+0000 7f3ebaeed640 1 -- 172.21.15.161:0/1616024629 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb40c2d90 con 0x7f3ea4003a90 2024-03-28T23:30:13.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.607+0000 7f3ebaeed640 1 -- 172.21.15.161:0/1616024629 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3eb40c3090 con 0x7f3eb41eb900 2024-03-28T23:30:13.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.608+0000 7f3eb948a640 1 -- 172.21.15.161:0/1616024629 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3eb41eb900 msgr2=0x7f3eb41413e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T23:30:13.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.608+0000 7f3eb948a640 1 --2- 172.21.15.161:0/1616024629 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3eb41eb900 0x7f3eb41413e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.608+0000 7f3eb948a640 1 -- 172.21.15.161:0/1616024629 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ea4003a90 msgr2=0x7f3eb41521d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.608+0000 7f3eb948a640 1 --2- 172.21.15.161:0/1616024629 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ea4003a90 0x7f3eb41521d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.608+0000 7f3eb948a640 1 -- 172.21.15.161:0/1616024629 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eb40bb690 con 0x7f3eb414fe30 2024-03-28T23:30:13.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.604+0000 7fa662ffd640 1 -- 172.21.15.161:0/1011259720 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa668017070 con 0x7fa66c071e80 2024-03-28T23:30:13.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.604+0000 7fa662ffd640 1 -- 172.21.15.161:0/1011259720 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa66800c620 con 0x7fa66c071e80 2024-03-28T23:30:13.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.604+0000 7fa662ffd640 1 -- 172.21.15.161:0/1011259720 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66800f040 con 0x7fa66c071e80 2024-03-28T23:30:13.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.609+0000 7fa67433e640 1 -- 172.21.15.161:0/1011259720 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa66c1f88a0 con 0x7fa66c071e80 2024-03-28T23:30:13.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.609+0000 7fa67433e640 1 -- 172.21.15.161:0/1011259720 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa66c0bc190 con 0x7fa66c071e80 2024-03-28T23:30:13.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.609+0000 7f3eb948a640 1 --2- 172.21.15.161:0/1616024629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3eb414fe30 0x7f3eb4141920 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4153af0 tx=0x7f3eac00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:13.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.609+0000 7f3eb17fa640 1 -- 172.21.15.161:0/1616024629 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eac017070 con 0x7f3eb414fe30 2024-03-28T23:30:13.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.610+0000 7f3eb17fa640 1 -- 172.21.15.161:0/1616024629 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3eac005d40 con 0x7f3eb414fe30 2024-03-28T23:30:13.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.610+0000 7f3ebaeed640 1 -- 172.21.15.161:0/1616024629 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3eb40bb970 con 0x7f3eb414fe30 2024-03-28T23:30:13.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.611+0000 7fa660ff9640 1 -- 172.21.15.161:0/1011259720 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa66c04f4b0 con 0x7fa66c071e80 2024-03-28T23:30:13.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.612+0000 7eff1759e640 1 Processor -- start 2024-03-28T23:30:13.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.612+0000 7eff1759e640 1 -- start start 2024-03-28T23:30:13.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.612+0000 7f3ebaeed640 1 -- 172.21.15.161:0/1616024629 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3eb40bbf30 con 0x7f3eb414fe30 2024-03-28T23:30:13.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.613+0000 7f3eb17fa640 1 -- 172.21.15.161:0/1616024629 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eac00a7d0 con 0x7f3eb414fe30 2024-03-28T23:30:13.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.613+0000 7f3eb17fa640 1 -- 172.21.15.161:0/1616024629 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3eac01a070 con 0x7f3eb414fe30 2024-03-28T23:30:13.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.614+0000 7f3eb17fa640 1 --2- 172.21.15.161:0/1616024629 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3e9806b5d0 0x7f3e9806da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.614+0000 7f3ebaeed640 1 -- 172.21.15.161:0/1616024629 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e94005290 con 0x7f3eb414fe30 2024-03-28T23:30:13.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.615+0000 7f3eb17fa640 1 -- 172.21.15.161:0/1616024629 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f3eac089490 con 0x7f3eb414fe30 2024-03-28T23:30:13.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.617+0000 7fa662ffd640 1 -- 172.21.15.161:0/1011259720 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa66801a070 con 0x7fa66c071e80 2024-03-28T23:30:13.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.618+0000 7fa662ffd640 1 --2- 172.21.15.161:0/1011259720 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa65006b5a0 0x7fa65006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.618+0000 7fa662ffd640 1 -- 172.21.15.161:0/1011259720 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa66808a450 con 0x7fa66c071e80 2024-03-28T23:30:13.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.618+0000 7fa6718d9640 1 --2- 172.21.15.161:0/1011259720 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa65006b5a0 0x7fa65006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.618+0000 7fa662ffd640 1 -- 172.21.15.161:0/1011259720 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa668054f90 con 0x7fa66c071e80 2024-03-28T23:30:13.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.620+0000 7fa6718d9640 1 --2- 172.21.15.161:0/1011259720 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa65006b5a0 0x7fa65006da60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa66c069350 tx=0x7fa65c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:13.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.619+0000 7eff1759e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff00003b90 0x7eff181489c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.619+0000 7eff1759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff1814eed0 0x7eff1814f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.619+0000 7eff1759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff180a96f0 0x7eff180a9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.619+0000 7eff1759e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7eff18070920 con 0x7eff00003b90 2024-03-28T23:30:13.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.619+0000 7eff1759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7eff180c2d90 con 0x7eff180a96f0 2024-03-28T23:30:13.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.619+0000 7eff1759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7eff180abd00 con 0x7eff1814eed0 2024-03-28T23:30:13.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.620+0000 7eff1659c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff00003b90 0x7eff181489c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.620+0000 7eff1659c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff00003b90 0x7eff181489c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54382/0 (socket says 172.21.15.161:54382) 2024-03-28T23:30:13.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.620+0000 7eff1659c640 1 -- 172.21.15.161:0/619783374 learned_addr learned my addr 172.21.15.161:0/619783374 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:13.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.621+0000 7eff1659c640 1 -- 172.21.15.161:0/619783374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff1814eed0 msgr2=0x7eff1814f350 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T23:30:13.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.621+0000 7eff1659c640 1 --2- 172.21.15.161:0/619783374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff1814eed0 0x7eff1814f350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.621+0000 7eff1659c640 1 -- 172.21.15.161:0/619783374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff180a96f0 msgr2=0x7eff180a9b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T23:30:13.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.621+0000 7eff1659c640 1 --2- 172.21.15.161:0/619783374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff180a96f0 0x7eff180a9b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.621+0000 7eff1659c640 1 -- 172.21.15.161:0/619783374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff180aa0b0 con 0x7eff00003b90 2024-03-28T23:30:13.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.621+0000 7eff1659c640 1 --2- 172.21.15.161:0/619783374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff00003b90 0x7eff181489c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7eff08030f90 tx=0x7eff180c1800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:13.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.621+0000 7f3eb8c89640 1 --2- 172.21.15.161:0/1616024629 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3e9806b5d0 0x7f3e9806da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.622+0000 7f3eb8c89640 1 --2- 172.21.15.161:0/1616024629 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3e9806b5d0 0x7f3e9806da90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4147c50 tx=0x7f3ea002f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:13.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.624+0000 7f3eb17fa640 1 -- 172.21.15.161:0/1616024629 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3eac057a50 con 0x7f3eb414fe30 2024-03-28T23:30:13.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.624+0000 7eff06ffd640 1 -- 172.21.15.161:0/619783374 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff0800a370 con 0x7eff00003b90 2024-03-28T23:30:13.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.624+0000 7eff06ffd640 1 -- 172.21.15.161:0/619783374 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff08045070 con 0x7eff00003b90 2024-03-28T23:30:13.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.624+0000 7eff06ffd640 1 -- 172.21.15.161:0/619783374 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff08035a80 con 0x7eff00003b90 2024-03-28T23:30:13.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.624+0000 7eff1759e640 1 -- 172.21.15.161:0/619783374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff181486e0 con 0x7eff00003b90 2024-03-28T23:30:13.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.624+0000 7eff1759e640 1 -- 172.21.15.161:0/619783374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff181e7790 con 0x7eff00003b90 2024-03-28T23:30:13.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.627+0000 7eff06ffd640 1 -- 172.21.15.161:0/619783374 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7eff08035c20 con 0x7eff00003b90 2024-03-28T23:30:13.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.628+0000 7eff06ffd640 1 --2- 172.21.15.161:0/619783374 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efef806b5a0 0x7efef806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.628+0000 7eff06ffd640 1 -- 172.21.15.161:0/619783374 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7eff080b1c00 con 0x7eff00003b90 2024-03-28T23:30:13.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.629+0000 7eff15d9b640 1 --2- 172.21.15.161:0/619783374 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efef806b5a0 0x7efef806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.629+0000 7eff1759e640 1 -- 172.21.15.161:0/619783374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efefc005290 con 0x7eff00003b90 2024-03-28T23:30:13.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.630+0000 7eff15d9b640 1 --2- 172.21.15.161:0/619783374 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efef806b5a0 0x7efef806da60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7eff0c0010e0 tx=0x7eff0c007990 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:13.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.632+0000 7fc66c20e640 1 Processor -- start 2024-03-28T23:30:13.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.632+0000 7fc66c20e640 1 -- start start 2024-03-28T23:30:13.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.634+0000 7fc66c20e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6640c3110 0x7fc664072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.634+0000 7fc66c20e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6640c2190 con 0x7fc664151d60 2024-03-28T23:30:13.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.634+0000 7fc66c20e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6640c2470 con 0x7fc66406b150 2024-03-28T23:30:13.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.634+0000 7fc66c20e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc6640c2750 con 0x7fc6640c3110 2024-03-28T23:30:13.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.634+0000 7fc669faa640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6640c3110 0x7fc664072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.634+0000 7fc6697a9640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc664151d60 0x7fc6640aa4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59672/0 (socket says 172.21.15.161:59672) 2024-03-28T23:30:13.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.634+0000 7fc6697a9640 1 -- 172.21.15.161:0/316990000 learned_addr learned my addr 172.21.15.161:0/316990000 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:13.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.635+0000 7fc669faa640 1 -- 172.21.15.161:0/316990000 >> v1:172.21.15.170:6789/0 conn(0x7fc66406b150 legacy=0x7fc66406b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:13.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.635+0000 7fc669faa640 1 -- 172.21.15.161:0/316990000 >> v1:172.21.15.161:6789/0 conn(0x7fc664151d60 legacy=0x7fc6640aa4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.635+0000 7fc669faa640 1 -- 172.21.15.161:0/316990000 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6640c2a30 con 0x7fc6640c3110 2024-03-28T23:30:13.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.636+0000 7fc669faa640 1 --2- 172.21.15.161:0/316990000 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6640c3110 0x7fc664072e10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc6600097d0 tx=0x7fc66002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9e0a3eb2ed3b06e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:13.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.636+0000 7fc668fa8640 1 -- 172.21.15.161:0/316990000 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc660035020 con 0x7fc6640c3110 2024-03-28T23:30:13.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.637+0000 7fc668fa8640 1 -- 172.21.15.161:0/316990000 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc660031830 con 0x7fc6640c3110 2024-03-28T23:30:13.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.638+0000 7fc66c20e640 1 -- 172.21.15.161:0/316990000 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6640c3110 msgr2=0x7fc664072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.638+0000 7fc66c20e640 1 --2- 172.21.15.161:0/316990000 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6640c3110 0x7fc664072e10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc6600097d0 tx=0x7fc66002b8d0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.638+0000 7fc66c20e640 1 -- 172.21.15.161:0/316990000 shutdown_connections 2024-03-28T23:30:13.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.638+0000 7fc66c20e640 1 --2- 172.21.15.161:0/316990000 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6640c3110 0x7fc664072e10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.638+0000 7fc66c20e640 1 -- 172.21.15.161:0/316990000 >> 172.21.15.161:0/316990000 conn(0x7fc66406d8e0 msgr2=0x7fc66406dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:13.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.638+0000 7fc66c20e640 1 -- 172.21.15.161:0/316990000 shutdown_connections 2024-03-28T23:30:13.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.638+0000 7f2ae5ec2640 1 Processor -- start 2024-03-28T23:30:13.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.638+0000 7fc66c20e640 1 -- 172.21.15.161:0/316990000 wait complete. 2024-03-28T23:30:13.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.639+0000 7f2ae5ec2640 1 -- start start 2024-03-28T23:30:13.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.639+0000 7eff06ffd640 1 -- 172.21.15.161:0/619783374 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7eff0804f060 con 0x7eff00003b90 2024-03-28T23:30:13.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.640+0000 7f2ae5ec2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ae00a8f10 0x7f2ae00c0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.640+0000 7f2ae5ec2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2ae00aad50 0x7f2ae00ab1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.640+0000 7f2ae5ec2640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ae0155280 con 0x7f2ae0152800 2024-03-28T23:30:13.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.640+0000 7f2ae5ec2640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2ae0155540 con 0x7f2ae00aad50 2024-03-28T23:30:13.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.640+0000 7f2ae5ec2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2ae0155820 con 0x7f2ae00a8f10 2024-03-28T23:30:13.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.640+0000 7f2adffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2ae00aad50 0x7f2ae00ab1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.640+0000 7f2adffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2ae00aad50 0x7f2ae00ab1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54394/0 (socket says 172.21.15.161:54394) 2024-03-28T23:30:13.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.640+0000 7f2adffff640 1 -- 172.21.15.161:0/1827898293 learned_addr learned my addr 172.21.15.161:0/1827898293 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:13.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.639+0000 7fc66c20e640 1 Processor -- start 2024-03-28T23:30:13.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.640+0000 7fc66c20e640 1 -- start start 2024-03-28T23:30:13.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.641+0000 7fc66c20e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc66406b150 0x7fc66415fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.641+0000 7fc66c20e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6640c3110 0x7fc6641479e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.641+0000 7fc66c20e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc664151d60 0x7fc664143400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.641+0000 7fc66c20e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc6640aad70 con 0x7fc66406b150 2024-03-28T23:30:13.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.641+0000 7fc66c20e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc6640aabf0 con 0x7fc6640c3110 2024-03-28T23:30:13.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.641+0000 7fc66c20e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc6641521e0 con 0x7fc664151d60 2024-03-28T23:30:13.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.641+0000 7fc6697a9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6640c3110 0x7fc6641479e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.641+0000 7fc6697a9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6640c3110 0x7fc6641479e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:39358/0 (socket says 172.21.15.161:39358) 2024-03-28T23:30:13.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.641+0000 7fc6697a9640 1 -- 172.21.15.161:0/2264556486 learned_addr learned my addr 172.21.15.161:0/2264556486 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:13.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.641+0000 7fc66a7ab640 1 --2- 172.21.15.161:0/2264556486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc664151d60 0x7fc664143400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.641+0000 7fc6697a9640 1 -- 172.21.15.161:0/2264556486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc664151d60 msgr2=0x7fc664143400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.641+0000 7fc6697a9640 1 --2- 172.21.15.161:0/2264556486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc664151d60 0x7fc664143400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.641+0000 7fc6697a9640 1 -- 172.21.15.161:0/2264556486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc66406b150 msgr2=0x7fc66415fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.641+0000 7f2adffff640 1 -- 172.21.15.161:0/1827898293 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ae00a8f10 msgr2=0x7f2ae00c0830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T23:30:13.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.641+0000 7f2adffff640 1 --2- 172.21.15.161:0/1827898293 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ae00a8f10 0x7f2ae00c0830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.641+0000 7f2adffff640 1 -- 172.21.15.161:0/1827898293 >> v1:172.21.15.161:6789/0 conn(0x7f2ae0152800 legacy=0x7f2ae00c2550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T23:30:13.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.641+0000 7f2adffff640 1 -- 172.21.15.161:0/1827898293 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae0155b00 con 0x7f2ae00aad50 2024-03-28T23:30:13.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.641+0000 7fc6697a9640 1 --2- 172.21.15.161:0/2264556486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc66406b150 0x7fc66415fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.641+0000 7fc6697a9640 1 -- 172.21.15.161:0/2264556486 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6641f3860 con 0x7fc6640c3110 2024-03-28T23:30:13.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.642+0000 7fc6697a9640 1 --2- 172.21.15.161:0/2264556486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6640c3110 0x7fc6641479e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc65c00caa0 tx=0x7fc65c00cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:13.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.642+0000 7f2adffff640 1 --2- 172.21.15.161:0/1827898293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae00aad50 0x7f2ae00ab1e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8009e50 tx=0x7f2ad802be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=990085127e415755 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:13.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.642+0000 7fc65a7fc640 1 -- 172.21.15.161:0/2264556486 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65c017070 con 0x7fc6640c3110 2024-03-28T23:30:13.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.642+0000 7fc65a7fc640 1 -- 172.21.15.161:0/2264556486 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc65c00ac00 con 0x7fc6640c3110 2024-03-28T23:30:13.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.642+0000 7fc65a7fc640 1 -- 172.21.15.161:0/2264556486 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65c012400 con 0x7fc6640c3110 2024-03-28T23:30:13.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.642+0000 7fc66c20e640 1 -- 172.21.15.161:0/2264556486 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc664161d70 con 0x7fc6640c3110 2024-03-28T23:30:13.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.642+0000 7fc66c20e640 1 -- 172.21.15.161:0/2264556486 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6641f3580 con 0x7fc6640c3110 2024-03-28T23:30:13.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.643+0000 7f2ade7fc640 1 -- 172.21.15.161:0/1827898293 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad802ca10 con 0x7f2ae00aad50 2024-03-28T23:30:13.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.643+0000 7f2ade7fc640 1 -- 172.21.15.161:0/1827898293 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ad802cbb0 con 0x7f2ae00aad50 2024-03-28T23:30:13.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.643+0000 7f2ade7fc640 1 -- 172.21.15.161:0/1827898293 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad80343f0 con 0x7f2ae00aad50 2024-03-28T23:30:13.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.644+0000 7f2ae5ec2640 1 -- 172.21.15.161:0/1827898293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae00aad50 msgr2=0x7f2ae00ab1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.644+0000 7f2ae5ec2640 1 --2- 172.21.15.161:0/1827898293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae00aad50 0x7f2ae00ab1e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8009e50 tx=0x7f2ad802be70 comp rx=0 tx=0).stop 2024-03-28T23:30:13.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.644+0000 7f2ae5ec2640 1 -- 172.21.15.161:0/1827898293 shutdown_connections 2024-03-28T23:30:13.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.644+0000 7f2ae5ec2640 1 --2- 172.21.15.161:0/1827898293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae00aad50 0x7f2ae00ab1e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.644+0000 7f2ae5ec2640 1 --2- 172.21.15.161:0/1827898293 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ae00a8f10 0x7f2ae00c0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.644+0000 7f2ae5ec2640 1 -- 172.21.15.161:0/1827898293 >> 172.21.15.161:0/1827898293 conn(0x7f2ae006d8e0 msgr2=0x7f2ae006dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:13.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.644+0000 7fc65a7fc640 1 -- 172.21.15.161:0/2264556486 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc65c01a070 con 0x7fc6640c3110 2024-03-28T23:30:13.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.645+0000 7f2ae5ec2640 1 -- 172.21.15.161:0/1827898293 shutdown_connections 2024-03-28T23:30:13.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.645+0000 7fc66c20e640 1 -- 172.21.15.161:0/2264556486 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc64c005290 con 0x7fc6640c3110 2024-03-28T23:30:13.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.645+0000 7f2ae5ec2640 1 -- 172.21.15.161:0/1827898293 wait complete. 2024-03-28T23:30:13.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.648+0000 7fc65a7fc640 1 --2- 172.21.15.161:0/2264556486 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc64806b4d0 0x7fc64806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.646+0000 7f2ae5ec2640 1 Processor -- start 2024-03-28T23:30:13.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.647+0000 7f2ae5ec2640 1 -- start start 2024-03-28T23:30:13.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.648+0000 7f2ae5ec2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ae00a8f10 0x7f2ae01f85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.648+0000 7f2ae5ec2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ae00aad50 0x7f2ae01f50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.648+0000 7f2ae5ec2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae0152800 0x7f2ae0158ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.648+0000 7f2ae5ec2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae00c2c60 con 0x7f2ae0152800 2024-03-28T23:30:13.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.648+0000 7f2ae5ec2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae0152be0 con 0x7f2ae00a8f10 2024-03-28T23:30:13.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.648+0000 7f2ae5ec2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2ae00c2de0 con 0x7f2ae00aad50 2024-03-28T23:30:13.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.648+0000 7f2adf7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ae00a8f10 0x7f2ae01f85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.648+0000 7f2adffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae0152800 0x7f2ae0158ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.648+0000 7f2adffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae0152800 0x7f2ae0158ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54414/0 (socket says 172.21.15.161:54414) 2024-03-28T23:30:13.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.648+0000 7f2adffff640 1 -- 172.21.15.161:0/2337287531 learned_addr learned my addr 172.21.15.161:0/2337287531 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:13.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.649+0000 7f2adffff640 1 -- 172.21.15.161:0/2337287531 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ae00aad50 msgr2=0x7f2ae01f50b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T23:30:13.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.649+0000 7f2adffff640 1 --2- 172.21.15.161:0/2337287531 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ae00aad50 0x7f2ae01f50b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.649+0000 7f2adffff640 1 -- 172.21.15.161:0/2337287531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ae00a8f10 msgr2=0x7f2ae01f85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.649+0000 7f2adffff640 1 --2- 172.21.15.161:0/2337287531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ae00a8f10 0x7f2ae01f85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.649+0000 7f2adffff640 1 -- 172.21.15.161:0/2337287531 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae0158150 con 0x7f2ae0152800 2024-03-28T23:30:13.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.649+0000 7f2adffff640 1 --2- 172.21.15.161:0/2337287531 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae0152800 0x7f2ae0158ce0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8036040 tx=0x7f2ad8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:13.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.648+0000 7fc669faa640 1 --2- 172.21.15.161:0/2264556486 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc64806b4d0 0x7fc64806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.650+0000 7fc65a7fc640 1 -- 172.21.15.161:0/2264556486 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc65c08a190 con 0x7fc6640c3110 2024-03-28T23:30:13.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.650+0000 7f2ac7fff640 1 -- 172.21.15.161:0/2337287531 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8043020 con 0x7f2ae0152800 2024-03-28T23:30:13.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.650+0000 7f2ac7fff640 1 -- 172.21.15.161:0/2337287531 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ad803e450 con 0x7f2ae0152800 2024-03-28T23:30:13.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.650+0000 7f2ac7fff640 1 -- 172.21.15.161:0/2337287531 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad803e800 con 0x7f2ae0152800 2024-03-28T23:30:13.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.650+0000 7f2ae5ec2640 1 -- 172.21.15.161:0/2337287531 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ae01567f0 con 0x7f2ae0152800 2024-03-28T23:30:13.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.650+0000 7f2ae5ec2640 1 -- 172.21.15.161:0/2337287531 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ae0156db0 con 0x7f2ae0152800 2024-03-28T23:30:13.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.651+0000 7fc669faa640 1 --2- 172.21.15.161:0/2264556486 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc64806b4d0 0x7fc64806d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc660036000 tx=0x7fc660030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:13.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.652+0000 7fc65a7fc640 1 -- 172.21.15.161:0/2264556486 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc65c0585e0 con 0x7fc6640c3110 2024-03-28T23:30:13.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.653+0000 7f2ae5ec2640 1 -- 172.21.15.161:0/2337287531 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ac0005290 con 0x7f2ae0152800 2024-03-28T23:30:13.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.654+0000 7f964e13e640 1 Processor -- start 2024-03-28T23:30:13.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.654+0000 7f2ac7fff640 1 -- 172.21.15.161:0/2337287531 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2ad8046020 con 0x7f2ae0152800 2024-03-28T23:30:13.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.655+0000 7f2ac7fff640 1 --2- 172.21.15.161:0/2337287531 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2ac806b6e0 0x7f2ac806dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.655+0000 7f2adf7fe640 1 --2- 172.21.15.161:0/2337287531 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2ac806b6e0 0x7f2ac806dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.656+0000 7f2ac7fff640 1 -- 172.21.15.161:0/2337287531 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2ad80ace40 con 0x7f2ae0152800 2024-03-28T23:30:13.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.657+0000 7f2adf7fe640 1 --2- 172.21.15.161:0/2337287531 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2ac806b6e0 0x7f2ac806dba0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2ad400a780 tx=0x7f2ad4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:13.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.657+0000 7f964e13e640 1 -- start start 2024-03-28T23:30:13.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.665+0000 7f964e13e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9648072e10 0x7f964806b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.666+0000 7f964e13e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f964806b690 0x7f964815bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.666+0000 7f964e13e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96480c1eb0 con 0x7f9648151fd0 2024-03-28T23:30:13.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.666+0000 7f964e13e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f96480c2190 con 0x7f9648072e10 2024-03-28T23:30:13.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.666+0000 7f964e13e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f96480c2470 con 0x7f964806b690 2024-03-28T23:30:13.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.666+0000 7f9646ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9648072e10 0x7f964806b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.666+0000 7f9646ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9648072e10 0x7f964806b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54420/0 (socket says 172.21.15.161:54420) 2024-03-28T23:30:13.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.667+0000 7f9646ffd640 1 -- 172.21.15.161:0/4108876720 learned_addr learned my addr 172.21.15.161:0/4108876720 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:13.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.667+0000 7f9647fff640 1 --2- 172.21.15.161:0/4108876720 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f964806b690 0x7f964815bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.667+0000 7f2ac7fff640 1 -- 172.21.15.161:0/2337287531 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2ad804d030 con 0x7f2ae0152800 2024-03-28T23:30:13.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.668+0000 7f9646ffd640 1 -- 172.21.15.161:0/4108876720 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f964806b690 msgr2=0x7f964815bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:13.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.668+0000 7f9646ffd640 1 --2- 172.21.15.161:0/4108876720 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f964806b690 0x7f964815bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.668+0000 7f9646ffd640 1 -- 172.21.15.161:0/4108876720 >> v1:172.21.15.170:6789/0 conn(0x7f9648151fd0 legacy=0x7f96480aa4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:13.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.668+0000 7f9646ffd640 1 -- 172.21.15.161:0/4108876720 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96480c2750 con 0x7f9648072e10 2024-03-28T23:30:13.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.676+0000 7f9646ffd640 1 --2- 172.21.15.161:0/4108876720 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9648072e10 0x7f964806b150 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f963c0091e0 tx=0x7f963c02cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd26a0347129027b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:13.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.681+0000 7f96467fc640 1 -- 172.21.15.161:0/4108876720 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f963c03a070 con 0x7f9648072e10 2024-03-28T23:30:13.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.681+0000 7f96467fc640 1 -- 172.21.15.161:0/4108876720 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f963c02d940 con 0x7f9648072e10 2024-03-28T23:30:13.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.681+0000 7f96467fc640 1 -- 172.21.15.161:0/4108876720 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f963c02dd10 con 0x7f9648072e10 2024-03-28T23:30:13.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.682+0000 7f964e13e640 1 -- 172.21.15.161:0/4108876720 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9648072e10 msgr2=0x7f964806b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.682+0000 7f964e13e640 1 --2- 172.21.15.161:0/4108876720 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9648072e10 0x7f964806b150 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f963c0091e0 tx=0x7f963c02cb90 comp rx=0 tx=0).stop 2024-03-28T23:30:13.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.682+0000 7f964e13e640 1 -- 172.21.15.161:0/4108876720 shutdown_connections 2024-03-28T23:30:13.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.682+0000 7f964e13e640 1 --2- 172.21.15.161:0/4108876720 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f964806b690 0x7f964815bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.682+0000 7f964e13e640 1 --2- 172.21.15.161:0/4108876720 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9648072e10 0x7f964806b150 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.682+0000 7f964e13e640 1 -- 172.21.15.161:0/4108876720 >> 172.21.15.161:0/4108876720 conn(0x7f964806d8e0 msgr2=0x7f964806dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:13.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.684+0000 7f964e13e640 1 -- 172.21.15.161:0/4108876720 shutdown_connections 2024-03-28T23:30:13.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.686+0000 7f964e13e640 1 -- 172.21.15.161:0/4108876720 wait complete. 2024-03-28T23:30:13.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.688+0000 7f964e13e640 1 Processor -- start 2024-03-28T23:30:13.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.689+0000 7f964e13e640 1 -- start start 2024-03-28T23:30:13.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.690+0000 7f964e13e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f964806b690 0x7f9648146bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.690+0000 7f964e13e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9648072e10 0x7f9648147110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.690+0000 7f964e13e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9648151fd0 0x7f9648148cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.690+0000 7f964e13e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f96480aad70 con 0x7f9648072e10 2024-03-28T23:30:13.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.690+0000 7f964e13e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f96480aabf0 con 0x7f9648151fd0 2024-03-28T23:30:13.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.690+0000 7f964e13e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f96480c1ad0 con 0x7f964806b690 2024-03-28T23:30:13.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.690+0000 7f96477fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f964806b690 0x7f9648146bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.690+0000 7f9647fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9648151fd0 0x7f9648148cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.690+0000 7f96477fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f964806b690 0x7f9648146bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:37090/0 (socket says 172.21.15.161:37090) 2024-03-28T23:30:13.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.690+0000 7f96477fe640 1 -- 172.21.15.161:0/3405982244 learned_addr learned my addr 172.21.15.161:0/3405982244 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:13.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.691+0000 7f9646ffd640 1 --2- 172.21.15.161:0/3405982244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9648072e10 0x7f9648147110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.691+0000 7f96477fe640 1 -- 172.21.15.161:0/3405982244 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9648151fd0 msgr2=0x7f9648148cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.691+0000 7f96477fe640 1 --2- 172.21.15.161:0/3405982244 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9648151fd0 0x7f9648148cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.691+0000 7f96477fe640 1 -- 172.21.15.161:0/3405982244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9648072e10 msgr2=0x7f9648147110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.691+0000 7f96477fe640 1 --2- 172.21.15.161:0/3405982244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9648072e10 0x7f9648147110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.691+0000 7f96477fe640 1 -- 172.21.15.161:0/3405982244 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96480bebe0 con 0x7f964806b690 2024-03-28T23:30:13.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.691+0000 7f96477fe640 1 --2- 172.21.15.161:0/3405982244 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f964806b690 0x7f9648146bd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f963800c430 tx=0x7f963800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:13.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.692+0000 7f962ffff640 1 -- 172.21.15.161:0/3405982244 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9638011790 con 0x7f964806b690 2024-03-28T23:30:13.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.692+0000 7f962ffff640 1 -- 172.21.15.161:0/3405982244 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9638017070 con 0x7f964806b690 2024-03-28T23:30:13.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.693+0000 7f962ffff640 1 -- 172.21.15.161:0/3405982244 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9638011b70 con 0x7f964806b690 2024-03-28T23:30:13.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.695+0000 7f964e13e640 1 -- 172.21.15.161:0/3405982244 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9648147650 con 0x7f964806b690 2024-03-28T23:30:13.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.695+0000 7f964e13e640 1 -- 172.21.15.161:0/3405982244 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f96480bee80 con 0x7f964806b690 2024-03-28T23:30:13.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.696+0000 7f962dffb640 1 -- 172.21.15.161:0/3405982244 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96480ac950 con 0x7f964806b690 2024-03-28T23:30:13.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.705+0000 7f962ffff640 1 -- 172.21.15.161:0/3405982244 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9638011d10 con 0x7f964806b690 2024-03-28T23:30:13.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.706+0000 7f962ffff640 1 --2- 172.21.15.161:0/3405982244 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f963006b6e0 0x7f963006dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:13.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.706+0000 7f962ffff640 1 -- 172.21.15.161:0/3405982244 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f963808a350 con 0x7f964806b690 2024-03-28T23:30:13.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.706+0000 7f962ffff640 1 -- 172.21.15.161:0/3405982244 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f96480ac950 con 0x7f964806b690 2024-03-28T23:30:13.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.707+0000 7f9646ffd640 1 --2- 172.21.15.161:0/3405982244 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f963006b6e0 0x7f963006dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:13.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.708+0000 7f9646ffd640 1 --2- 172.21.15.161:0/3405982244 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f963006b6e0 0x7f963006dba0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f963c037000 tx=0x7f963c031000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:13.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.876+0000 7fc593240640 1 -- 172.21.15.161:0/1273977562 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fc58c072730 con 0x7fc58c06c3d0 2024-03-28T23:30:13.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.878+0000 7fc5897fa640 1 -- 172.21.15.161:0/1273977562 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc58007b090 con 0x7fc58c06c3d0 2024-03-28T23:30:13.882 INFO:teuthology.orchestra.run.smithi161.stdout:55834574856 2024-03-28T23:30:13.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.881+0000 7fc593240640 1 -- 172.21.15.161:0/1273977562 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc57006b5a0 msgr2=0x7fc57006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.881+0000 7fc593240640 1 --2- 172.21.15.161:0/1273977562 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc57006b5a0 0x7fc57006da60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc58c1e4830 tx=0x7fc578009040 comp rx=0 tx=0).stop 2024-03-28T23:30:13.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.881+0000 7fc593240640 1 -- 172.21.15.161:0/1273977562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc58c06c3d0 msgr2=0x7fc58c1e7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.881+0000 7fc593240640 1 --2- 172.21.15.161:0/1273977562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc58c06c3d0 0x7fc58c1e7570 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc580038040 tx=0x7fc5800046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.881+0000 7fc593240640 1 -- 172.21.15.161:0/1273977562 shutdown_connections 2024-03-28T23:30:13.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.881+0000 7fc593240640 1 --2- 172.21.15.161:0/1273977562 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc57006b5a0 0x7fc57006da60 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.881+0000 7fc593240640 1 --2- 172.21.15.161:0/1273977562 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc58c151e20 0x7fc58c150ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.881+0000 7fc593240640 1 --2- 172.21.15.161:0/1273977562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc58c14ee30 0x7fc58c152ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.881+0000 7fc593240640 1 --2- 172.21.15.161:0/1273977562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc58c06c3d0 0x7fc58c1e7570 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.881+0000 7fc593240640 1 -- 172.21.15.161:0/1273977562 >> 172.21.15.161:0/1273977562 conn(0x7fc58c0c3cf0 msgr2=0x7fc58c14c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:13.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.881+0000 7fc593240640 1 -- 172.21.15.161:0/1273977562 shutdown_connections 2024-03-28T23:30:13.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.884+0000 7fc593240640 1 -- 172.21.15.161:0/1273977562 wait complete. 2024-03-28T23:30:13.902 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.2 2024-03-28T23:30:13.902 DEBUG:teuthology.parallel:result is None 2024-03-28T23:30:13.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.935+0000 7f0131476640 1 -- 172.21.15.161:0/3804975014 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f010c005850 con 0x7f012c0aa4e0 2024-03-28T23:30:13.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.936+0000 7f01137fe640 1 -- 172.21.15.161:0/3804975014 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f012407b6d0 con 0x7f012c0aa4e0 2024-03-28T23:30:13.938 INFO:teuthology.orchestra.run.smithi161.stdout:55834574856 2024-03-28T23:30:13.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.939+0000 7f0131476640 1 -- 172.21.15.161:0/3804975014 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f011806b5d0 msgr2=0x7f011806da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.939+0000 7f0131476640 1 --2- 172.21.15.161:0/3804975014 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f011806b5d0 0x7f011806da90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f012c1dcb30 tx=0x7f01200079b0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.939+0000 7f0131476640 1 -- 172.21.15.161:0/3804975014 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f012c0aa4e0 msgr2=0x7f012c1dd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:13.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.939+0000 7f0131476640 1 --2- 172.21.15.161:0/3804975014 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f012c0aa4e0 0x7f012c1dd010 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0124037040 tx=0x7f0124004a40 comp rx=0 tx=0).stop 2024-03-28T23:30:13.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.940+0000 7f0131476640 1 -- 172.21.15.161:0/3804975014 shutdown_connections 2024-03-28T23:30:13.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.940+0000 7f0131476640 1 --2- 172.21.15.161:0/3804975014 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f011806b5d0 0x7f011806da90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.940+0000 7f0131476640 1 --2- 172.21.15.161:0/3804975014 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f012c1e0b70 0x7f012c152d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.940+0000 7f0131476640 1 --2- 172.21.15.161:0/3804975014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f012c0aa960 0x7f012c152800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.940+0000 7f0131476640 1 --2- 172.21.15.161:0/3804975014 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f012c0aa4e0 0x7f012c1dd010 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:13.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.940+0000 7f0131476640 1 -- 172.21.15.161:0/3804975014 >> 172.21.15.161:0/3804975014 conn(0x7f012c06d8e0 msgr2=0x7f012c14f550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:13.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.940+0000 7f0131476640 1 -- 172.21.15.161:0/3804975014 shutdown_connections 2024-03-28T23:30:13.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:13.940+0000 7f0131476640 1 -- 172.21.15.161:0/3804975014 wait complete. 2024-03-28T23:30:13.958 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.1 2024-03-28T23:30:13.958 DEBUG:teuthology.parallel:result is None 2024-03-28T23:30:14.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.009+0000 7f2ae5ec2640 1 -- 172.21.15.161:0/2337287531 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f2ac0005580 con 0x7f2ae0152800 2024-03-28T23:30:14.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.009+0000 7f2ac7fff640 1 -- 172.21.15.161:0/2337287531 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2ad807ba20 con 0x7f2ae0152800 2024-03-28T23:30:14.011 INFO:teuthology.orchestra.run.smithi161.stdout:55834574856 2024-03-28T23:30:14.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.011+0000 7f2ae5ec2640 1 -- 172.21.15.161:0/2337287531 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2ac806b6e0 msgr2=0x7f2ac806dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.011+0000 7f2ae5ec2640 1 --2- 172.21.15.161:0/2337287531 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2ac806b6e0 0x7f2ac806dba0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2ad400a780 tx=0x7f2ad4008040 comp rx=0 tx=0).stop 2024-03-28T23:30:14.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.011+0000 7f2ae5ec2640 1 -- 172.21.15.161:0/2337287531 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae0152800 msgr2=0x7f2ae0158ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.011+0000 7f2ae5ec2640 1 --2- 172.21.15.161:0/2337287531 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae0152800 0x7f2ae0158ce0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8036040 tx=0x7f2ad8004a40 comp rx=0 tx=0).stop 2024-03-28T23:30:14.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.012+0000 7f2ae5ec2640 1 -- 172.21.15.161:0/2337287531 shutdown_connections 2024-03-28T23:30:14.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.012+0000 7f2ae5ec2640 1 --2- 172.21.15.161:0/2337287531 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2ac806b6e0 0x7f2ac806dba0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.012+0000 7f2ae5ec2640 1 --2- 172.21.15.161:0/2337287531 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae0152800 0x7f2ae0158ce0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.012+0000 7f2ae5ec2640 1 --2- 172.21.15.161:0/2337287531 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ae00aad50 0x7f2ae01f50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.012+0000 7f2ae5ec2640 1 --2- 172.21.15.161:0/2337287531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ae00a8f10 0x7f2ae01f85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.012+0000 7f2ae5ec2640 1 -- 172.21.15.161:0/2337287531 >> 172.21.15.161:0/2337287531 conn(0x7f2ae006d8e0 msgr2=0x7f2ae00c2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:14.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.012+0000 7f2ae5ec2640 1 -- 172.21.15.161:0/2337287531 shutdown_connections 2024-03-28T23:30:14.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.012+0000 7f2ae5ec2640 1 -- 172.21.15.161:0/2337287531 wait complete. 2024-03-28T23:30:14.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.019+0000 7f3ebaeed640 1 -- 172.21.15.161:0/1616024629 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f3e94005850 con 0x7f3eb414fe30 2024-03-28T23:30:14.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.020+0000 7f3eb17fa640 1 -- 172.21.15.161:0/1616024629 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3eac05baf0 con 0x7f3eb414fe30 2024-03-28T23:30:14.022 INFO:teuthology.orchestra.run.smithi161.stdout:55834574856 2024-03-28T23:30:14.025 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.7 2024-03-28T23:30:14.025 DEBUG:teuthology.parallel:result is None 2024-03-28T23:30:14.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.025+0000 7f3ebaeed640 1 -- 172.21.15.161:0/1616024629 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3e9806b5d0 msgr2=0x7f3e9806da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.025+0000 7f3ebaeed640 1 --2- 172.21.15.161:0/1616024629 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3e9806b5d0 0x7f3e9806da90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4147c50 tx=0x7f3ea002f000 comp rx=0 tx=0).stop 2024-03-28T23:30:14.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.025+0000 7f3ebaeed640 1 -- 172.21.15.161:0/1616024629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3eb414fe30 msgr2=0x7f3eb4141920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.025+0000 7f3ebaeed640 1 --2- 172.21.15.161:0/1616024629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3eb414fe30 0x7f3eb4141920 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4153af0 tx=0x7f3eac00cc00 comp rx=0 tx=0).stop 2024-03-28T23:30:14.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.025+0000 7f3ebaeed640 1 -- 172.21.15.161:0/1616024629 shutdown_connections 2024-03-28T23:30:14.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.025+0000 7f3ebaeed640 1 --2- 172.21.15.161:0/1616024629 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3e9806b5d0 0x7f3e9806da90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.025+0000 7f3ebaeed640 1 --2- 172.21.15.161:0/1616024629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3eb414fe30 0x7f3eb4141920 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.025+0000 7f3ebaeed640 1 --2- 172.21.15.161:0/1616024629 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3eb41eb900 0x7f3eb41413e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.025+0000 7f3ebaeed640 1 --2- 172.21.15.161:0/1616024629 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ea4003a90 0x7f3eb41521d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.025+0000 7f3ebaeed640 1 -- 172.21.15.161:0/1616024629 >> 172.21.15.161:0/1616024629 conn(0x7f3eb406d8e0 msgr2=0x7f3eb406dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:14.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.025+0000 7f3ebaeed640 1 -- 172.21.15.161:0/1616024629 shutdown_connections 2024-03-28T23:30:14.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.025+0000 7f3ebaeed640 1 -- 172.21.15.161:0/1616024629 wait complete. 2024-03-28T23:30:14.042 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.3 2024-03-28T23:30:14.042 DEBUG:teuthology.parallel:result is None 2024-03-28T23:30:14.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.045+0000 7fa660ff9640 1 -- 172.21.15.161:0/1011259720 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fa66c073510 con 0x7fa66c071e80 2024-03-28T23:30:14.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.046+0000 7fa662ffd640 1 -- 172.21.15.161:0/1011259720 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa668058920 con 0x7fa66c071e80 2024-03-28T23:30:14.047 INFO:teuthology.orchestra.run.smithi161.stdout:55834574857 2024-03-28T23:30:14.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.048+0000 7fa660ff9640 1 -- 172.21.15.161:0/1011259720 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa65006b5a0 msgr2=0x7fa65006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.048+0000 7fa660ff9640 1 --2- 172.21.15.161:0/1011259720 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa65006b5a0 0x7fa65006da60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa66c069350 tx=0x7fa65c009040 comp rx=0 tx=0).stop 2024-03-28T23:30:14.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.048+0000 7fa660ff9640 1 -- 172.21.15.161:0/1011259720 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa66c071e80 msgr2=0x7fa66c0ab0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.048+0000 7fa660ff9640 1 --2- 172.21.15.161:0/1011259720 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa66c071e80 0x7fa66c0ab0a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa66800e600 tx=0x7fa66800eac0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.048+0000 7fa660ff9640 1 -- 172.21.15.161:0/1011259720 shutdown_connections 2024-03-28T23:30:14.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.048+0000 7fa660ff9640 1 --2- 172.21.15.161:0/1011259720 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa65006b5a0 0x7fa65006da60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.048+0000 7fa660ff9640 1 --2- 172.21.15.161:0/1011259720 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa66c15cd40 0x7fa66c151fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.048+0000 7fa660ff9640 1 --2- 172.21.15.161:0/1011259720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa66c15d660 0x7fa66c152500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.048+0000 7fa660ff9640 1 --2- 172.21.15.161:0/1011259720 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa66c071e80 0x7fa66c0ab0a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.048+0000 7fa660ff9640 1 -- 172.21.15.161:0/1011259720 >> 172.21.15.161:0/1011259720 conn(0x7fa66c06c090 msgr2=0x7fa66c154330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:14.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.049+0000 7fa660ff9640 1 -- 172.21.15.161:0/1011259720 shutdown_connections 2024-03-28T23:30:14.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.049+0000 7fa660ff9640 1 -- 172.21.15.161:0/1011259720 wait complete. 2024-03-28T23:30:14.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.053+0000 7fc66c20e640 1 -- 172.21.15.161:0/2264556486 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fc64c005580 con 0x7fc6640c3110 2024-03-28T23:30:14.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.054+0000 7fc65a7fc640 1 -- 172.21.15.161:0/2264556486 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc65c05c680 con 0x7fc6640c3110 2024-03-28T23:30:14.056 INFO:teuthology.orchestra.run.smithi161.stdout:55834574857 2024-03-28T23:30:14.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.056+0000 7fc66c20e640 1 -- 172.21.15.161:0/2264556486 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc64806b4d0 msgr2=0x7fc64806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.056+0000 7fc66c20e640 1 --2- 172.21.15.161:0/2264556486 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc64806b4d0 0x7fc64806d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc660036000 tx=0x7fc660030000 comp rx=0 tx=0).stop 2024-03-28T23:30:14.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.056+0000 7fc66c20e640 1 -- 172.21.15.161:0/2264556486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6640c3110 msgr2=0x7fc6641479e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.056+0000 7fc66c20e640 1 --2- 172.21.15.161:0/2264556486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6640c3110 0x7fc6641479e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc65c00caa0 tx=0x7fc65c00cf60 comp rx=0 tx=0).stop 2024-03-28T23:30:14.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.056+0000 7fc66c20e640 1 -- 172.21.15.161:0/2264556486 shutdown_connections 2024-03-28T23:30:14.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.056+0000 7fc66c20e640 1 --2- 172.21.15.161:0/2264556486 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc64806b4d0 0x7fc64806d990 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.056+0000 7fc66c20e640 1 --2- 172.21.15.161:0/2264556486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc664151d60 0x7fc664143400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.056+0000 7fc66c20e640 1 --2- 172.21.15.161:0/2264556486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6640c3110 0x7fc6641479e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.056+0000 7fc66c20e640 1 --2- 172.21.15.161:0/2264556486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc66406b150 0x7fc66415fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.056+0000 7fc66c20e640 1 -- 172.21.15.161:0/2264556486 >> 172.21.15.161:0/2264556486 conn(0x7fc66406d8e0 msgr2=0x7fc6640c5e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:14.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.056+0000 7fc66c20e640 1 -- 172.21.15.161:0/2264556486 shutdown_connections 2024-03-28T23:30:14.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.057+0000 7fc66c20e640 1 -- 172.21.15.161:0/2264556486 wait complete. 2024-03-28T23:30:14.062 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.5 2024-03-28T23:30:14.062 DEBUG:teuthology.parallel:result is None 2024-03-28T23:30:14.071 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.6 2024-03-28T23:30:14.071 DEBUG:teuthology.parallel:result is None 2024-03-28T23:30:14.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.076+0000 7eff1759e640 1 -- 172.21.15.161:0/619783374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7efefc005580 con 0x7eff00003b90 2024-03-28T23:30:14.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.077+0000 7eff06ffd640 1 -- 172.21.15.161:0/619783374 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7eff180aa0b0 con 0x7eff00003b90 2024-03-28T23:30:14.079 INFO:teuthology.orchestra.run.smithi161.stdout:55834574856 2024-03-28T23:30:14.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.080+0000 7eff1759e640 1 -- 172.21.15.161:0/619783374 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efef806b5a0 msgr2=0x7efef806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.080+0000 7eff1759e640 1 --2- 172.21.15.161:0/619783374 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efef806b5a0 0x7efef806da60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7eff0c0010e0 tx=0x7eff0c007990 comp rx=0 tx=0).stop 2024-03-28T23:30:14.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.080+0000 7eff1759e640 1 -- 172.21.15.161:0/619783374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff00003b90 msgr2=0x7eff181489c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.080+0000 7eff1759e640 1 --2- 172.21.15.161:0/619783374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff00003b90 0x7eff181489c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7eff08030f90 tx=0x7eff180c1800 comp rx=0 tx=0).stop 2024-03-28T23:30:14.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.080+0000 7eff1759e640 1 -- 172.21.15.161:0/619783374 shutdown_connections 2024-03-28T23:30:14.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.080+0000 7eff1759e640 1 --2- 172.21.15.161:0/619783374 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efef806b5a0 0x7efef806da60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.080+0000 7eff1759e640 1 --2- 172.21.15.161:0/619783374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff180a96f0 0x7eff180a9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.080+0000 7eff1759e640 1 --2- 172.21.15.161:0/619783374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff1814eed0 0x7eff1814f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.080+0000 7eff1759e640 1 --2- 172.21.15.161:0/619783374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff00003b90 0x7eff181489c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.080+0000 7eff1759e640 1 -- 172.21.15.161:0/619783374 >> 172.21.15.161:0/619783374 conn(0x7eff180c71a0 msgr2=0x7eff1806d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:14.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.080+0000 7eff1759e640 1 -- 172.21.15.161:0/619783374 shutdown_connections 2024-03-28T23:30:14.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.080+0000 7eff1759e640 1 -- 172.21.15.161:0/619783374 wait complete. 2024-03-28T23:30:14.096 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.0 2024-03-28T23:30:14.096 DEBUG:teuthology.parallel:result is None 2024-03-28T23:30:14.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.144+0000 7f962dffb640 1 -- 172.21.15.161:0/3405982244 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f964804f4b0 con 0x7f964806b690 2024-03-28T23:30:14.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.144+0000 7f962ffff640 1 -- 172.21.15.161:0/3405982244 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9638020020 con 0x7f964806b690 2024-03-28T23:30:14.146 INFO:teuthology.orchestra.run.smithi161.stdout:55834574856 2024-03-28T23:30:14.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.146+0000 7f962dffb640 1 -- 172.21.15.161:0/3405982244 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f963006b6e0 msgr2=0x7f963006dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.146+0000 7f962dffb640 1 --2- 172.21.15.161:0/3405982244 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f963006b6e0 0x7f963006dba0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f963c037000 tx=0x7f963c031000 comp rx=0 tx=0).stop 2024-03-28T23:30:14.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.146+0000 7f962dffb640 1 -- 172.21.15.161:0/3405982244 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f964806b690 msgr2=0x7f9648146bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.146+0000 7f962dffb640 1 --2- 172.21.15.161:0/3405982244 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f964806b690 0x7f9648146bd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f963800c430 tx=0x7f963800c8f0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.146+0000 7f962dffb640 1 -- 172.21.15.161:0/3405982244 shutdown_connections 2024-03-28T23:30:14.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.146+0000 7f962dffb640 1 --2- 172.21.15.161:0/3405982244 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f963006b6e0 0x7f963006dba0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.146+0000 7f962dffb640 1 --2- 172.21.15.161:0/3405982244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9648072e10 0x7f9648147110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.146+0000 7f962dffb640 1 --2- 172.21.15.161:0/3405982244 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9648151fd0 0x7f9648148cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.146+0000 7f962dffb640 1 --2- 172.21.15.161:0/3405982244 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f964806b690 0x7f9648146bd0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.146+0000 7f962dffb640 1 -- 172.21.15.161:0/3405982244 >> 172.21.15.161:0/3405982244 conn(0x7f964806d8e0 msgr2=0x7f9648150180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:14.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.146+0000 7f962dffb640 1 -- 172.21.15.161:0/3405982244 shutdown_connections 2024-03-28T23:30:14.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.146+0000 7f962dffb640 1 -- 172.21.15.161:0/3405982244 wait complete. 2024-03-28T23:30:14.163 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.4 2024-03-28T23:30:14.163 DEBUG:teuthology.parallel:result is None 2024-03-28T23:30:14.163 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-28T23:30:14.163 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:30:14.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.263+0000 7fbb625e2640 1 Processor -- start 2024-03-28T23:30:14.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.263+0000 7fbb625e2640 1 -- start start 2024-03-28T23:30:14.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.264+0000 7fbb625e2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb5c155b90 0x7fbb5c158750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:14.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.264+0000 7fbb625e2640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb5c066fe0 con 0x7fbb5c0ac150 2024-03-28T23:30:14.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.264+0000 7fbb625e2640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb5c0672c0 con 0x7fbb5c06c3d0 2024-03-28T23:30:14.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.264+0000 7fbb625e2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbb5c156070 con 0x7fbb5c155b90 2024-03-28T23:30:14.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.264+0000 7fbb60b7f640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fbb5c0ac150 0x7fbb5c15fae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59684/0 (socket says 172.21.15.161:59684) 2024-03-28T23:30:14.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.264+0000 7fbb60b7f640 1 -- 172.21.15.161:0/3837669305 learned_addr learned my addr 172.21.15.161:0/3837669305 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:14.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.264+0000 7fbb5b7fe640 1 --2- 172.21.15.161:0/3837669305 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb5c155b90 0x7fbb5c158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:14.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.264+0000 7fbb5affd640 1 -- 172.21.15.161:0/3837669305 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3966376524 0 0) 0x7fbb5c066fe0 con 0x7fbb5c0ac150 2024-03-28T23:30:14.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.264+0000 7fbb5affd640 1 -- 172.21.15.161:0/3837669305 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb44003520 con 0x7fbb5c0ac150 2024-03-28T23:30:14.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.264+0000 7fbb5affd640 1 -- 172.21.15.161:0/3837669305 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1582609205 0 0) 0x7fbb5c0672c0 con 0x7fbb5c06c3d0 2024-03-28T23:30:14.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.264+0000 7fbb5affd640 1 -- 172.21.15.161:0/3837669305 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb5c066fe0 con 0x7fbb5c06c3d0 2024-03-28T23:30:14.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.264+0000 7fbb5affd640 1 -- 172.21.15.161:0/3837669305 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3123592742 0 0) 0x7fbb44003520 con 0x7fbb5c0ac150 2024-03-28T23:30:14.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.264+0000 7fbb5affd640 1 -- 172.21.15.161:0/3837669305 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb5c155b90 msgr2=0x7fbb5c158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:14.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.264+0000 7fbb5affd640 1 --2- 172.21.15.161:0/3837669305 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb5c155b90 0x7fbb5c158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.264+0000 7fbb5affd640 1 -- 172.21.15.161:0/3837669305 >> v1:172.21.15.170:6789/0 conn(0x7fbb5c06c3d0 legacy=0x7fbb5c152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.264+0000 7fbb5affd640 1 -- 172.21.15.161:0/3837669305 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb5c0c1c80 con 0x7fbb5c0ac150 2024-03-28T23:30:14.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.264+0000 7fbb5affd640 1 -- 172.21.15.161:0/3837669305 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fbb540019f0 con 0x7fbb5c0ac150 2024-03-28T23:30:14.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.264+0000 7fbb5b7fe640 1 --2- 172.21.15.161:0/3837669305 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb5c155b90 0x7fbb5c158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:14.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.264+0000 7fbb5affd640 1 -- 172.21.15.161:0/3837669305 >> v1:172.21.15.161:6789/0 conn(0x7fbb5c0ac150 legacy=0x7fbb5c15fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.264+0000 7fbb5affd640 1 --2- 172.21.15.161:0/3837669305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb44003c50 0x7fbb44004050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:14.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.265+0000 7fbb5affd640 1 --2- 172.21.15.161:0/3837669305 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb5c155b90 0x7fbb44004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:14.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.265+0000 7fbb5affd640 1 --2- 172.21.15.161:0/3837669305 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb44004d40 0x7fbb440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:14.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.265+0000 7fbb5affd640 1 -- 172.21.15.161:0/3837669305 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5c156070 con 0x7fbb44003c50 2024-03-28T23:30:14.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.265+0000 7fbb5affd640 1 -- 172.21.15.161:0/3837669305 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbb4400bb00 con 0x7fbb44004d40 2024-03-28T23:30:14.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.265+0000 7fbb5affd640 1 -- 172.21.15.161:0/3837669305 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbb4400bd00 con 0x7fbb5c155b90 2024-03-28T23:30:14.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.265+0000 7fbb5bfff640 1 --2- 172.21.15.161:0/3837669305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb44003c50 0x7fbb44004050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:14.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.265+0000 7fbb60b7f640 1 --2- 172.21.15.161:0/3837669305 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb44004d40 0x7fbb440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:14.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.265+0000 7fbb5b7fe640 1 --2- 172.21.15.161:0/3837669305 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb5c155b90 0x7fbb44004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:14.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.265+0000 7fbb5bfff640 1 -- 172.21.15.161:0/3837669305 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb5c155b90 msgr2=0x7fbb44004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:14.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.265+0000 7fbb5bfff640 1 --2- 172.21.15.161:0/3837669305 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb5c155b90 0x7fbb44004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.265+0000 7fbb5bfff640 1 -- 172.21.15.161:0/3837669305 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb44004d40 msgr2=0x7fbb440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:14.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.265+0000 7fbb5bfff640 1 --2- 172.21.15.161:0/3837669305 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb44004d40 0x7fbb440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.265+0000 7fbb5bfff640 1 -- 172.21.15.161:0/3837669305 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb44003520 con 0x7fbb44003c50 2024-03-28T23:30:14.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.266+0000 7fbb5bfff640 1 --2- 172.21.15.161:0/3837669305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb44003c50 0x7fbb44004050 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbb500018d0 tx=0x7fbb44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bd4380b44078b07 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:14.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.266+0000 7fbb5affd640 1 -- 172.21.15.161:0/3837669305 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb500014f0 con 0x7fbb44003c50 2024-03-28T23:30:14.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.266+0000 7fbb5affd640 1 -- 172.21.15.161:0/3837669305 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb500052d0 con 0x7fbb44003c50 2024-03-28T23:30:14.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.266+0000 7fbb5affd640 1 -- 172.21.15.161:0/3837669305 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb50003810 con 0x7fbb44003c50 2024-03-28T23:30:14.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.266+0000 7fbb625e2640 1 -- 172.21.15.161:0/3837669305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb44003c50 msgr2=0x7fbb44004050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.266+0000 7fbb625e2640 1 --2- 172.21.15.161:0/3837669305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb44003c50 0x7fbb44004050 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbb500018d0 tx=0x7fbb44005830 comp rx=0 tx=0).stop 2024-03-28T23:30:14.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.266+0000 7fbb60b7f640 1 -- 172.21.15.161:0/3837669305 reap_dead start 2024-03-28T23:30:14.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.266+0000 7fbb625e2640 1 -- 172.21.15.161:0/3837669305 shutdown_connections 2024-03-28T23:30:14.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.266+0000 7fbb625e2640 1 -- 172.21.15.161:0/3837669305 >> 172.21.15.161:0/3837669305 conn(0x7fbb5c0c3be0 msgr2=0x7fbb5c0c3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:14.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.266+0000 7fbb625e2640 1 -- 172.21.15.161:0/3837669305 shutdown_connections 2024-03-28T23:30:14.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.266+0000 7fbb625e2640 1 -- 172.21.15.161:0/3837669305 wait complete. 2024-03-28T23:30:14.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.267+0000 7fbb625e2640 1 Processor -- start 2024-03-28T23:30:14.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.268+0000 7fbb625e2640 1 -- start start 2024-03-28T23:30:14.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.268+0000 7fbb625e2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb44003c50 0x7fbb5c14cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:14.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.268+0000 7fbb625e2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb5c14d480 0x7fbb5c1494f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:14.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.268+0000 7fbb625e2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb5c14e100 0x7fbb5c149a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:14.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.268+0000 7fbb625e2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5c0c2320 con 0x7fbb5c14e100 2024-03-28T23:30:14.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.268+0000 7fbb625e2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5c0c21a0 con 0x7fbb44003c50 2024-03-28T23:30:14.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.268+0000 7fbb625e2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbb5c06c7b0 con 0x7fbb5c14d480 2024-03-28T23:30:14.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.268+0000 7fbb60b7f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb5c14e100 0x7fbb5c149a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:14.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.268+0000 7fbb60b7f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb5c14e100 0x7fbb5c149a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54460/0 (socket says 172.21.15.161:54460) 2024-03-28T23:30:14.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.268+0000 7fbb60b7f640 1 -- 172.21.15.161:0/844555906 learned_addr learned my addr 172.21.15.161:0/844555906 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:14.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.269+0000 7fbb5bfff640 1 --2- 172.21.15.161:0/844555906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb44003c50 0x7fbb5c14cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:14.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.269+0000 7fbb5b7fe640 1 --2- 172.21.15.161:0/844555906 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb5c14d480 0x7fbb5c1494f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:14.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.269+0000 7fbb60b7f640 1 -- 172.21.15.161:0/844555906 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb5c14d480 msgr2=0x7fbb5c1494f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.269+0000 7fbb60b7f640 1 --2- 172.21.15.161:0/844555906 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb5c14d480 0x7fbb5c1494f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.269+0000 7fbb60b7f640 1 -- 172.21.15.161:0/844555906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb44003c50 msgr2=0x7fbb5c14cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.269+0000 7fbb60b7f640 1 --2- 172.21.15.161:0/844555906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb44003c50 0x7fbb5c14cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.269+0000 7fbb60b7f640 1 -- 172.21.15.161:0/844555906 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb5c1eb390 con 0x7fbb5c14e100 2024-03-28T23:30:14.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.269+0000 7fbb5bfff640 1 --2- 172.21.15.161:0/844555906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb44003c50 0x7fbb5c14cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:14.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.269+0000 7fbb5b7fe640 1 --2- 172.21.15.161:0/844555906 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb5c14d480 0x7fbb5c1494f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:14.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.269+0000 7fbb60b7f640 1 --2- 172.21.15.161:0/844555906 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb5c14e100 0x7fbb5c149a30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c070970 tx=0x7fbb54007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:14.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.269+0000 7fbb58ff9640 1 -- 172.21.15.161:0/844555906 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb54017020 con 0x7fbb5c14e100 2024-03-28T23:30:14.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.269+0000 7fbb58ff9640 1 -- 172.21.15.161:0/844555906 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb5400ab90 con 0x7fbb5c14e100 2024-03-28T23:30:14.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.269+0000 7fbb625e2640 1 -- 172.21.15.161:0/844555906 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb5c1eb670 con 0x7fbb5c14e100 2024-03-28T23:30:14.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.269+0000 7fbb58ff9640 1 -- 172.21.15.161:0/844555906 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb540133f0 con 0x7fbb5c14e100 2024-03-28T23:30:14.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.269+0000 7fbb625e2640 1 -- 172.21.15.161:0/844555906 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb5c1518a0 con 0x7fbb5c14e100 2024-03-28T23:30:14.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.270+0000 7fbb58ff9640 1 -- 172.21.15.161:0/844555906 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbb5401a020 con 0x7fbb5c14e100 2024-03-28T23:30:14.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.270+0000 7fbb625e2640 1 -- 172.21.15.161:0/844555906 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb5c04f4b0 con 0x7fbb5c14e100 2024-03-28T23:30:14.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.271+0000 7fbb58ff9640 1 --2- 172.21.15.161:0/844555906 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbb4806b4d0 0x7fbb4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:14.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.271+0000 7fbb5bfff640 1 --2- 172.21.15.161:0/844555906 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbb4806b4d0 0x7fbb4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:14.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.271+0000 7fbb58ff9640 1 -- 172.21.15.161:0/844555906 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fbb5408abf0 con 0x7fbb5c14e100 2024-03-28T23:30:14.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.272+0000 7fbb5bfff640 1 --2- 172.21.15.161:0/844555906 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbb4806b4d0 0x7fbb4806d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbb50034000 tx=0x7fbb50031000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:14.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.276+0000 7fbb58ff9640 1 -- 172.21.15.161:0/844555906 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbb54021020 con 0x7fbb5c14e100 2024-03-28T23:30:14.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.467+0000 7fbb625e2640 1 -- 172.21.15.161:0/844555906 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbb5c14e640 con 0x7fbb4806b4d0 2024-03-28T23:30:14.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.475+0000 7fbb58ff9640 1 -- 172.21.15.161:0/844555906 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332610 (secure 0 0 0) 0x7fbb50005340 con 0x7fbb4806b4d0 2024-03-28T23:30:14.480 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:14.482 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:30:14.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.478+0000 7fbb625e2640 1 -- 172.21.15.161:0/844555906 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbb4806b4d0 msgr2=0x7fbb4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.478+0000 7fbb625e2640 1 --2- 172.21.15.161:0/844555906 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbb4806b4d0 0x7fbb4806d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbb50034000 tx=0x7fbb50031000 comp rx=0 tx=0).stop 2024-03-28T23:30:14.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.478+0000 7fbb625e2640 1 -- 172.21.15.161:0/844555906 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb5c14e100 msgr2=0x7fbb5c149a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.478+0000 7fbb625e2640 1 --2- 172.21.15.161:0/844555906 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb5c14e100 0x7fbb5c149a30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c070970 tx=0x7fbb54007bd0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.478+0000 7fbb625e2640 1 -- 172.21.15.161:0/844555906 shutdown_connections 2024-03-28T23:30:14.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.478+0000 7fbb625e2640 1 --2- 172.21.15.161:0/844555906 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbb4806b4d0 0x7fbb4806d990 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.478+0000 7fbb625e2640 1 --2- 172.21.15.161:0/844555906 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb5c14e100 0x7fbb5c149a30 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.478+0000 7fbb625e2640 1 --2- 172.21.15.161:0/844555906 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb5c14d480 0x7fbb5c1494f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.478+0000 7fbb625e2640 1 --2- 172.21.15.161:0/844555906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb44003c50 0x7fbb5c14cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.478+0000 7fbb625e2640 1 -- 172.21.15.161:0/844555906 >> 172.21.15.161:0/844555906 conn(0x7fbb5c0c3be0 msgr2=0x7fbb5c0c6830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:14.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.478+0000 7fbb625e2640 1 -- 172.21.15.161:0/844555906 shutdown_connections 2024-03-28T23:30:14.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.478+0000 7fbb625e2640 1 -- 172.21.15.161:0/844555906 wait complete. 2024-03-28T23:30:14.495 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-28T23:30:13.317522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222516,"kb_used_data":2176,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754752204,"statfs":{"total":773094113280,"available":772866256896,"internally_reserved":0,"allocated":2228224,"data_stored":1992064,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002607"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687840+0000","last_change":"2024-03-28T23:29:45.558575+0000","last_active":"2024-03-28T23:29:51.687840+0000","last_peered":"2024-03-28T23:29:51.687840+0000","last_clean":"2024-03-28T23:29:51.687840+0000","last_became_active":"2024-03-28T23:29:45.557910+0000","last_became_peered":"2024-03-28T23:29:45.557910+0000","last_unstale":"2024-03-28T23:29:51.687840+0000","last_undegraded":"2024-03-28T23:29:51.687840+0000","last_fullsized":"2024-03-28T23:29:51.687840+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:59.719252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686777+0000","last_change":"2024-03-28T23:29:46.557562+0000","last_active":"2024-03-28T23:29:51.686777+0000","last_peered":"2024-03-28T23:29:51.686777+0000","last_clean":"2024-03-28T23:29:51.686777+0000","last_became_active":"2024-03-28T23:29:46.557265+0000","last_became_peered":"2024-03-28T23:29:46.557265+0000","last_unstale":"2024-03-28T23:29:51.686777+0000","last_undegraded":"2024-03-28T23:29:51.686777+0000","last_fullsized":"2024-03-28T23:29:51.686777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:06.731262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686496+0000","last_change":"2024-03-28T23:29:50.707752+0000","last_active":"2024-03-28T23:29:51.686496+0000","last_peered":"2024-03-28T23:29:51.686496+0000","last_clean":"2024-03-28T23:29:51.686496+0000","last_became_active":"2024-03-28T23:29:50.707364+0000","last_became_peered":"2024-03-28T23:29:50.707364+0000","last_unstale":"2024-03-28T23:29:51.686496+0000","last_undegraded":"2024-03-28T23:29:51.686496+0000","last_fullsized":"2024-03-28T23:29:51.686496+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:58:38.680360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686927+0000","last_change":"2024-03-28T23:29:45.558356+0000","last_active":"2024-03-28T23:29:51.686927+0000","last_peered":"2024-03-28T23:29:51.686927+0000","last_clean":"2024-03-28T23:29:51.686927+0000","last_became_active":"2024-03-28T23:29:45.558070+0000","last_became_peered":"2024-03-28T23:29:45.558070+0000","last_unstale":"2024-03-28T23:29:51.686927+0000","last_undegraded":"2024-03-28T23:29:51.686927+0000","last_fullsized":"2024-03-28T23:29:51.686927+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:59.719252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686591+0000","last_change":"2024-03-28T23:29:46.579153+0000","last_active":"2024-03-28T23:29:51.686591+0000","last_peered":"2024-03-28T23:29:51.686591+0000","last_clean":"2024-03-28T23:29:51.686591+0000","last_became_active":"2024-03-28T23:29:46.578341+0000","last_became_peered":"2024-03-28T23:29:46.578341+0000","last_unstale":"2024-03-28T23:29:51.686591+0000","last_undegraded":"2024-03-28T23:29:51.686591+0000","last_fullsized":"2024-03-28T23:29:51.686591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:38.074388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591979+0000","last_change":"2024-03-28T23:29:50.597011+0000","last_active":"2024-03-28T23:29:51.591979+0000","last_peered":"2024-03-28T23:29:51.591979+0000","last_clean":"2024-03-28T23:29:51.591979+0000","last_became_active":"2024-03-28T23:29:50.596239+0000","last_became_peered":"2024-03-28T23:29:50.596239+0000","last_unstale":"2024-03-28T23:29:51.591979+0000","last_undegraded":"2024-03-28T23:29:51.591979+0000","last_fullsized":"2024-03-28T23:29:51.591979+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:29:24.703997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686674+0000","last_change":"2024-03-28T23:29:45.562460+0000","last_active":"2024-03-28T23:29:51.686674+0000","last_peered":"2024-03-28T23:29:51.686674+0000","last_clean":"2024-03-28T23:29:51.686674+0000","last_became_active":"2024-03-28T23:29:45.562101+0000","last_became_peered":"2024-03-28T23:29:45.562101+0000","last_unstale":"2024-03-28T23:29:51.686674+0000","last_undegraded":"2024-03-28T23:29:51.686674+0000","last_fullsized":"2024-03-28T23:29:51.686674+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:42:08.707703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685709+0000","last_change":"2024-03-28T23:29:46.567420+0000","last_active":"2024-03-28T23:29:51.685709+0000","last_peered":"2024-03-28T23:29:51.685709+0000","last_clean":"2024-03-28T23:29:51.685709+0000","last_became_active":"2024-03-28T23:29:46.567157+0000","last_became_peered":"2024-03-28T23:29:46.567157+0000","last_unstale":"2024-03-28T23:29:51.685709+0000","last_undegraded":"2024-03-28T23:29:51.685709+0000","last_fullsized":"2024-03-28T23:29:51.685709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:54.470202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.592139+0000","last_change":"2024-03-28T23:29:50.599020+0000","last_active":"2024-03-28T23:29:51.592139+0000","last_peered":"2024-03-28T23:29:51.592139+0000","last_clean":"2024-03-28T23:29:51.592139+0000","last_became_active":"2024-03-28T23:29:50.598588+0000","last_became_peered":"2024-03-28T23:29:50.598588+0000","last_unstale":"2024-03-28T23:29:51.592139+0000","last_undegraded":"2024-03-28T23:29:51.592139+0000","last_fullsized":"2024-03-28T23:29:51.592139+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:06.894069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686561+0000","last_change":"2024-03-28T23:29:45.564688+0000","last_active":"2024-03-28T23:29:51.686561+0000","last_peered":"2024-03-28T23:29:51.686561+0000","last_clean":"2024-03-28T23:29:51.686561+0000","last_became_active":"2024-03-28T23:29:45.564381+0000","last_became_peered":"2024-03-28T23:29:45.564381+0000","last_unstale":"2024-03-28T23:29:51.686561+0000","last_undegraded":"2024-03-28T23:29:51.686561+0000","last_fullsized":"2024-03-28T23:29:51.686561+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:36.113647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686025+0000","last_change":"2024-03-28T23:29:46.566644+0000","last_active":"2024-03-28T23:29:51.686025+0000","last_peered":"2024-03-28T23:29:51.686025+0000","last_clean":"2024-03-28T23:29:51.686025+0000","last_became_active":"2024-03-28T23:29:46.566340+0000","last_became_peered":"2024-03-28T23:29:46.566340+0000","last_unstale":"2024-03-28T23:29:51.686025+0000","last_undegraded":"2024-03-28T23:29:51.686025+0000","last_fullsized":"2024-03-28T23:29:51.686025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:02.859502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591969+0000","last_change":"2024-03-28T23:29:50.597019+0000","last_active":"2024-03-28T23:29:51.591969+0000","last_peered":"2024-03-28T23:29:51.591969+0000","last_clean":"2024-03-28T23:29:51.591969+0000","last_became_active":"2024-03-28T23:29:50.596284+0000","last_became_peered":"2024-03-28T23:29:50.596284+0000","last_unstale":"2024-03-28T23:29:51.591969+0000","last_undegraded":"2024-03-28T23:29:51.591969+0000","last_fullsized":"2024-03-28T23:29:51.591969+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:08:37.836411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.590591+0000","last_change":"2024-03-28T23:29:46.565325+0000","last_active":"2024-03-28T23:29:51.590591+0000","last_peered":"2024-03-28T23:29:51.590591+0000","last_clean":"2024-03-28T23:29:51.590591+0000","last_became_active":"2024-03-28T23:29:46.564195+0000","last_became_peered":"2024-03-28T23:29:46.564195+0000","last_unstale":"2024-03-28T23:29:51.590591+0000","last_undegraded":"2024-03-28T23:29:51.590591+0000","last_fullsized":"2024-03-28T23:29:51.590591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:32.936386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685919+0000","last_change":"2024-03-28T23:29:45.560329+0000","last_active":"2024-03-28T23:29:51.685919+0000","last_peered":"2024-03-28T23:29:51.685919+0000","last_clean":"2024-03-28T23:29:51.685919+0000","last_became_active":"2024-03-28T23:29:45.560073+0000","last_became_peered":"2024-03-28T23:29:45.560073+0000","last_unstale":"2024-03-28T23:29:51.685919+0000","last_undegraded":"2024-03-28T23:29:51.685919+0000","last_fullsized":"2024-03-28T23:29:51.685919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:09.929984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591983+0000","last_change":"2024-03-28T23:29:50.592083+0000","last_active":"2024-03-28T23:29:51.591983+0000","last_peered":"2024-03-28T23:29:51.591983+0000","last_clean":"2024-03-28T23:29:51.591983+0000","last_became_active":"2024-03-28T23:29:50.591626+0000","last_became_peered":"2024-03-28T23:29:50.591626+0000","last_unstale":"2024-03-28T23:29:51.591983+0000","last_undegraded":"2024-03-28T23:29:51.591983+0000","last_fullsized":"2024-03-28T23:29:51.591983+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:17:54.013778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685967+0000","last_change":"2024-03-28T23:29:46.581125+0000","last_active":"2024-03-28T23:29:51.685967+0000","last_peered":"2024-03-28T23:29:51.685967+0000","last_clean":"2024-03-28T23:29:51.685967+0000","last_became_active":"2024-03-28T23:29:46.580827+0000","last_became_peered":"2024-03-28T23:29:46.580827+0000","last_unstale":"2024-03-28T23:29:51.685967+0000","last_undegraded":"2024-03-28T23:29:51.685967+0000","last_fullsized":"2024-03-28T23:29:51.685967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:10.853710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686404+0000","last_change":"2024-03-28T23:29:45.562525+0000","last_active":"2024-03-28T23:29:51.686404+0000","last_peered":"2024-03-28T23:29:51.686404+0000","last_clean":"2024-03-28T23:29:51.686404+0000","last_became_active":"2024-03-28T23:29:45.561757+0000","last_became_peered":"2024-03-28T23:29:45.561757+0000","last_unstale":"2024-03-28T23:29:51.686404+0000","last_undegraded":"2024-03-28T23:29:51.686404+0000","last_fullsized":"2024-03-28T23:29:51.686404+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:24:58.784056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591567+0000","last_change":"2024-03-28T23:29:50.599617+0000","last_active":"2024-03-28T23:29:51.591567+0000","last_peered":"2024-03-28T23:29:51.591567+0000","last_clean":"2024-03-28T23:29:51.591567+0000","last_became_active":"2024-03-28T23:29:50.599031+0000","last_became_peered":"2024-03-28T23:29:50.599031+0000","last_unstale":"2024-03-28T23:29:51.591567+0000","last_undegraded":"2024-03-28T23:29:51.591567+0000","last_fullsized":"2024-03-28T23:29:51.591567+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:15:03.216756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686223+0000","last_change":"2024-03-28T23:29:46.555057+0000","last_active":"2024-03-28T23:29:51.686223+0000","last_peered":"2024-03-28T23:29:51.686223+0000","last_clean":"2024-03-28T23:29:51.686223+0000","last_became_active":"2024-03-28T23:29:46.554796+0000","last_became_peered":"2024-03-28T23:29:46.554796+0000","last_unstale":"2024-03-28T23:29:51.686223+0000","last_undegraded":"2024-03-28T23:29:51.686223+0000","last_fullsized":"2024-03-28T23:29:51.686223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:36:20.485861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685609+0000","last_change":"2024-03-28T23:29:45.561273+0000","last_active":"2024-03-28T23:29:51.685609+0000","last_peered":"2024-03-28T23:29:51.685609+0000","last_clean":"2024-03-28T23:29:51.685609+0000","last_became_active":"2024-03-28T23:29:45.561006+0000","last_became_peered":"2024-03-28T23:29:45.561006+0000","last_unstale":"2024-03-28T23:29:51.685609+0000","last_undegraded":"2024-03-28T23:29:51.685609+0000","last_fullsized":"2024-03-28T23:29:51.685609+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:41.621188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687569+0000","last_change":"2024-03-28T23:29:50.706824+0000","last_active":"2024-03-28T23:29:51.687569+0000","last_peered":"2024-03-28T23:29:51.687569+0000","last_clean":"2024-03-28T23:29:51.687569+0000","last_became_active":"2024-03-28T23:29:50.706411+0000","last_became_peered":"2024-03-28T23:29:50.706411+0000","last_unstale":"2024-03-28T23:29:51.687569+0000","last_undegraded":"2024-03-28T23:29:51.687569+0000","last_fullsized":"2024-03-28T23:29:51.687569+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:43:14.415039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.592115+0000","last_change":"2024-03-28T23:29:46.558900+0000","last_active":"2024-03-28T23:29:51.592115+0000","last_peered":"2024-03-28T23:29:51.592115+0000","last_clean":"2024-03-28T23:29:51.592115+0000","last_became_active":"2024-03-28T23:29:46.558628+0000","last_became_peered":"2024-03-28T23:29:46.558628+0000","last_unstale":"2024-03-28T23:29:51.592115+0000","last_undegraded":"2024-03-28T23:29:51.592115+0000","last_fullsized":"2024-03-28T23:29:51.592115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:37.122394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685397+0000","last_change":"2024-03-28T23:29:45.563365+0000","last_active":"2024-03-28T23:29:51.685397+0000","last_peered":"2024-03-28T23:29:51.685397+0000","last_clean":"2024-03-28T23:29:51.685397+0000","last_became_active":"2024-03-28T23:29:45.563093+0000","last_became_peered":"2024-03-28T23:29:45.563093+0000","last_unstale":"2024-03-28T23:29:51.685397+0000","last_undegraded":"2024-03-28T23:29:51.685397+0000","last_fullsized":"2024-03-28T23:29:51.685397+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:01.983721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591200+0000","last_change":"2024-03-28T23:29:50.596634+0000","last_active":"2024-03-28T23:29:51.591200+0000","last_peered":"2024-03-28T23:29:51.591200+0000","last_clean":"2024-03-28T23:29:51.591200+0000","last_became_active":"2024-03-28T23:29:50.596057+0000","last_became_peered":"2024-03-28T23:29:50.596057+0000","last_unstale":"2024-03-28T23:29:51.591200+0000","last_undegraded":"2024-03-28T23:29:51.591200+0000","last_fullsized":"2024-03-28T23:29:51.591200+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:02.708894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687515+0000","last_change":"2024-03-28T23:29:46.578357+0000","last_active":"2024-03-28T23:29:51.687515+0000","last_peered":"2024-03-28T23:29:51.687515+0000","last_clean":"2024-03-28T23:29:51.687515+0000","last_became_active":"2024-03-28T23:29:46.578075+0000","last_became_peered":"2024-03-28T23:29:46.578075+0000","last_unstale":"2024-03-28T23:29:51.687515+0000","last_undegraded":"2024-03-28T23:29:51.687515+0000","last_fullsized":"2024-03-28T23:29:51.687515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:40:22.556997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591419+0000","last_change":"2024-03-28T23:29:45.560606+0000","last_active":"2024-03-28T23:29:51.591419+0000","last_peered":"2024-03-28T23:29:51.591419+0000","last_clean":"2024-03-28T23:29:51.591419+0000","last_became_active":"2024-03-28T23:29:45.560355+0000","last_became_peered":"2024-03-28T23:29:45.560355+0000","last_unstale":"2024-03-28T23:29:51.591419+0000","last_undegraded":"2024-03-28T23:29:51.591419+0000","last_fullsized":"2024-03-28T23:29:51.591419+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:01.983721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591966+0000","last_change":"2024-03-28T23:29:50.708102+0000","last_active":"2024-03-28T23:29:51.591966+0000","last_peered":"2024-03-28T23:29:51.591966+0000","last_clean":"2024-03-28T23:29:51.591966+0000","last_became_active":"2024-03-28T23:29:50.707657+0000","last_became_peered":"2024-03-28T23:29:50.707657+0000","last_unstale":"2024-03-28T23:29:51.591966+0000","last_undegraded":"2024-03-28T23:29:51.591966+0000","last_fullsized":"2024-03-28T23:29:51.591966+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:58.504770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687431+0000","last_change":"2024-03-28T23:29:46.561641+0000","last_active":"2024-03-28T23:29:51.687431+0000","last_peered":"2024-03-28T23:29:51.687431+0000","last_clean":"2024-03-28T23:29:51.687431+0000","last_became_active":"2024-03-28T23:29:46.561380+0000","last_became_peered":"2024-03-28T23:29:46.561380+0000","last_unstale":"2024-03-28T23:29:51.687431+0000","last_undegraded":"2024-03-28T23:29:51.687431+0000","last_fullsized":"2024-03-28T23:29:51.687431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:54.470202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686791+0000","last_change":"2024-03-28T23:29:45.562177+0000","last_active":"2024-03-28T23:29:51.686791+0000","last_peered":"2024-03-28T23:29:51.686791+0000","last_clean":"2024-03-28T23:29:51.686791+0000","last_became_active":"2024-03-28T23:29:45.561852+0000","last_became_peered":"2024-03-28T23:29:45.561852+0000","last_unstale":"2024-03-28T23:29:51.686791+0000","last_undegraded":"2024-03-28T23:29:51.686791+0000","last_fullsized":"2024-03-28T23:29:51.686791+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:36.113647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685836+0000","last_change":"2024-03-28T23:29:50.706769+0000","last_active":"2024-03-28T23:29:51.685836+0000","last_peered":"2024-03-28T23:29:51.685836+0000","last_clean":"2024-03-28T23:29:51.685836+0000","last_became_active":"2024-03-28T23:29:50.706415+0000","last_became_peered":"2024-03-28T23:29:50.706415+0000","last_unstale":"2024-03-28T23:29:51.685836+0000","last_undegraded":"2024-03-28T23:29:51.685836+0000","last_fullsized":"2024-03-28T23:29:51.685836+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:47.094789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687279+0000","last_change":"2024-03-28T23:29:46.560067+0000","last_active":"2024-03-28T23:29:51.687279+0000","last_peered":"2024-03-28T23:29:51.687279+0000","last_clean":"2024-03-28T23:29:51.687279+0000","last_became_active":"2024-03-28T23:29:46.559790+0000","last_became_peered":"2024-03-28T23:29:46.559790+0000","last_unstale":"2024-03-28T23:29:51.687279+0000","last_undegraded":"2024-03-28T23:29:51.687279+0000","last_fullsized":"2024-03-28T23:29:51.687279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:32.936386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686796+0000","last_change":"2024-03-28T23:29:45.565279+0000","last_active":"2024-03-28T23:29:51.686796+0000","last_peered":"2024-03-28T23:29:51.686796+0000","last_clean":"2024-03-28T23:29:51.686796+0000","last_became_active":"2024-03-28T23:29:45.564739+0000","last_became_peered":"2024-03-28T23:29:45.564739+0000","last_unstale":"2024-03-28T23:29:51.686796+0000","last_undegraded":"2024-03-28T23:29:51.686796+0000","last_fullsized":"2024-03-28T23:29:51.686796+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:17.783972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686745+0000","last_change":"2024-03-28T23:29:50.706805+0000","last_active":"2024-03-28T23:29:51.686745+0000","last_peered":"2024-03-28T23:29:51.686745+0000","last_clean":"2024-03-28T23:29:51.686745+0000","last_became_active":"2024-03-28T23:29:50.706429+0000","last_became_peered":"2024-03-28T23:29:50.706429+0000","last_unstale":"2024-03-28T23:29:51.686745+0000","last_undegraded":"2024-03-28T23:29:51.686745+0000","last_fullsized":"2024-03-28T23:29:51.686745+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:06:40.917529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686157+0000","last_change":"2024-03-28T23:29:46.560912+0000","last_active":"2024-03-28T23:29:51.686157+0000","last_peered":"2024-03-28T23:29:51.686157+0000","last_clean":"2024-03-28T23:29:51.686157+0000","last_became_active":"2024-03-28T23:29:46.560613+0000","last_became_peered":"2024-03-28T23:29:46.560613+0000","last_unstale":"2024-03-28T23:29:51.686157+0000","last_undegraded":"2024-03-28T23:29:51.686157+0000","last_fullsized":"2024-03-28T23:29:51.686157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:17:49.979210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687372+0000","last_change":"2024-03-28T23:29:45.561918+0000","last_active":"2024-03-28T23:29:51.687372+0000","last_peered":"2024-03-28T23:29:51.687372+0000","last_clean":"2024-03-28T23:29:51.687372+0000","last_became_active":"2024-03-28T23:29:45.561577+0000","last_became_peered":"2024-03-28T23:29:45.561577+0000","last_unstale":"2024-03-28T23:29:51.687372+0000","last_undegraded":"2024-03-28T23:29:51.687372+0000","last_fullsized":"2024-03-28T23:29:51.687372+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:09.929984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686073+0000","last_change":"2024-03-28T23:29:50.594835+0000","last_active":"2024-03-28T23:29:51.686073+0000","last_peered":"2024-03-28T23:29:51.686073+0000","last_clean":"2024-03-28T23:29:51.686073+0000","last_became_active":"2024-03-28T23:29:50.594167+0000","last_became_peered":"2024-03-28T23:29:50.594167+0000","last_unstale":"2024-03-28T23:29:51.686073+0000","last_undegraded":"2024-03-28T23:29:51.686073+0000","last_fullsized":"2024-03-28T23:29:51.686073+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:06:40.917529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.592269+0000","last_change":"2024-03-28T23:29:46.579930+0000","last_active":"2024-03-28T23:29:51.592269+0000","last_peered":"2024-03-28T23:29:51.592269+0000","last_clean":"2024-03-28T23:29:51.592269+0000","last_became_active":"2024-03-28T23:29:46.579622+0000","last_became_peered":"2024-03-28T23:29:46.579622+0000","last_unstale":"2024-03-28T23:29:51.592269+0000","last_undegraded":"2024-03-28T23:29:51.592269+0000","last_fullsized":"2024-03-28T23:29:51.592269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:18.792719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686219+0000","last_change":"2024-03-28T23:29:45.557802+0000","last_active":"2024-03-28T23:29:51.686219+0000","last_peered":"2024-03-28T23:29:51.686219+0000","last_clean":"2024-03-28T23:29:51.686219+0000","last_became_active":"2024-03-28T23:29:45.557552+0000","last_became_peered":"2024-03-28T23:29:45.557552+0000","last_unstale":"2024-03-28T23:29:51.686219+0000","last_undegraded":"2024-03-28T23:29:51.686219+0000","last_fullsized":"2024-03-28T23:29:51.686219+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:01.983721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687230+0000","last_change":"2024-03-28T23:29:50.601086+0000","last_active":"2024-03-28T23:29:51.687230+0000","last_peered":"2024-03-28T23:29:51.687230+0000","last_clean":"2024-03-28T23:29:51.687230+0000","last_became_active":"2024-03-28T23:29:50.600720+0000","last_became_peered":"2024-03-28T23:29:50.600720+0000","last_unstale":"2024-03-28T23:29:51.687230+0000","last_undegraded":"2024-03-28T23:29:51.687230+0000","last_fullsized":"2024-03-28T23:29:51.687230+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:30:48.379529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685445+0000","last_change":"2024-03-28T23:29:46.560331+0000","last_active":"2024-03-28T23:29:51.685445+0000","last_peered":"2024-03-28T23:29:51.685445+0000","last_clean":"2024-03-28T23:29:51.685445+0000","last_became_active":"2024-03-28T23:29:46.559135+0000","last_became_peered":"2024-03-28T23:29:46.559135+0000","last_unstale":"2024-03-28T23:29:51.685445+0000","last_undegraded":"2024-03-28T23:29:51.685445+0000","last_fullsized":"2024-03-28T23:29:51.685445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:58.674327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589305+0000","last_change":"2024-03-28T23:29:45.559376+0000","last_active":"2024-03-28T23:29:51.589305+0000","last_peered":"2024-03-28T23:29:51.589305+0000","last_clean":"2024-03-28T23:29:51.589305+0000","last_became_active":"2024-03-28T23:29:45.559076+0000","last_became_peered":"2024-03-28T23:29:45.559076+0000","last_unstale":"2024-03-28T23:29:51.589305+0000","last_undegraded":"2024-03-28T23:29:51.589305+0000","last_fullsized":"2024-03-28T23:29:51.589305+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:21.707459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687612+0000","last_change":"2024-03-28T23:29:50.600059+0000","last_active":"2024-03-28T23:29:51.687612+0000","last_peered":"2024-03-28T23:29:51.687612+0000","last_clean":"2024-03-28T23:29:51.687612+0000","last_became_active":"2024-03-28T23:29:50.599692+0000","last_became_peered":"2024-03-28T23:29:50.599692+0000","last_unstale":"2024-03-28T23:29:51.687612+0000","last_undegraded":"2024-03-28T23:29:51.687612+0000","last_fullsized":"2024-03-28T23:29:51.687612+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:22.085225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686352+0000","last_change":"2024-03-28T23:29:46.578521+0000","last_active":"2024-03-28T23:29:51.686352+0000","last_peered":"2024-03-28T23:29:51.686352+0000","last_clean":"2024-03-28T23:29:51.686352+0000","last_became_active":"2024-03-28T23:29:46.578076+0000","last_became_peered":"2024-03-28T23:29:46.578076+0000","last_unstale":"2024-03-28T23:29:51.686352+0000","last_undegraded":"2024-03-28T23:29:51.686352+0000","last_fullsized":"2024-03-28T23:29:51.686352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:20.039954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685395+0000","last_change":"2024-03-28T23:29:45.563206+0000","last_active":"2024-03-28T23:29:51.685395+0000","last_peered":"2024-03-28T23:29:51.685395+0000","last_clean":"2024-03-28T23:29:51.685395+0000","last_became_active":"2024-03-28T23:29:45.562929+0000","last_became_peered":"2024-03-28T23:29:45.562929+0000","last_unstale":"2024-03-28T23:29:51.685395+0000","last_undegraded":"2024-03-28T23:29:51.685395+0000","last_fullsized":"2024-03-28T23:29:51.685395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:41.621188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687463+0000","last_change":"2024-03-28T23:29:50.709169+0000","last_active":"2024-03-28T23:29:51.687463+0000","last_peered":"2024-03-28T23:29:51.687463+0000","last_clean":"2024-03-28T23:29:51.687463+0000","last_became_active":"2024-03-28T23:29:50.707306+0000","last_became_peered":"2024-03-28T23:29:50.707306+0000","last_unstale":"2024-03-28T23:29:51.687463+0000","last_undegraded":"2024-03-28T23:29:51.687463+0000","last_fullsized":"2024-03-28T23:29:51.687463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:22.085225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685675+0000","last_change":"2024-03-28T23:29:46.564922+0000","last_active":"2024-03-28T23:29:51.685675+0000","last_peered":"2024-03-28T23:29:51.685675+0000","last_clean":"2024-03-28T23:29:51.685675+0000","last_became_active":"2024-03-28T23:29:46.564644+0000","last_became_peered":"2024-03-28T23:29:46.564644+0000","last_unstale":"2024-03-28T23:29:51.685675+0000","last_undegraded":"2024-03-28T23:29:51.685675+0000","last_fullsized":"2024-03-28T23:29:51.685675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:58.674327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686316+0000","last_change":"2024-03-28T23:29:45.556656+0000","last_active":"2024-03-28T23:29:51.686316+0000","last_peered":"2024-03-28T23:29:51.686316+0000","last_clean":"2024-03-28T23:29:51.686316+0000","last_became_active":"2024-03-28T23:29:45.556329+0000","last_became_peered":"2024-03-28T23:29:45.556329+0000","last_unstale":"2024-03-28T23:29:51.686316+0000","last_undegraded":"2024-03-28T23:29:51.686316+0000","last_fullsized":"2024-03-28T23:29:51.686316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:21.707459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686403+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:29:51.686403+0000","last_peered":"2024-03-28T23:29:51.686403+0000","last_clean":"2024-03-28T23:29:51.686403+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:29:51.686403+0000","last_undegraded":"2024-03-28T23:29:51.686403+0000","last_fullsized":"2024-03-28T23:29:51.686403+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591319+0000","last_change":"2024-03-28T23:29:50.711182+0000","last_active":"2024-03-28T23:29:51.591319+0000","last_peered":"2024-03-28T23:29:51.591319+0000","last_clean":"2024-03-28T23:29:51.591319+0000","last_became_active":"2024-03-28T23:29:50.710737+0000","last_became_peered":"2024-03-28T23:29:50.710737+0000","last_unstale":"2024-03-28T23:29:51.591319+0000","last_undegraded":"2024-03-28T23:29:51.591319+0000","last_fullsized":"2024-03-28T23:29:51.591319+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:17:54.013778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686573+0000","last_change":"2024-03-28T23:29:46.572966+0000","last_active":"2024-03-28T23:29:51.686573+0000","last_peered":"2024-03-28T23:29:51.686573+0000","last_clean":"2024-03-28T23:29:51.686573+0000","last_became_active":"2024-03-28T23:29:46.572405+0000","last_became_peered":"2024-03-28T23:29:46.572405+0000","last_unstale":"2024-03-28T23:29:51.686573+0000","last_undegraded":"2024-03-28T23:29:51.686573+0000","last_fullsized":"2024-03-28T23:29:51.686573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:06:36.882961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589560+0000","last_change":"2024-03-28T23:29:45.560681+0000","last_active":"2024-03-28T23:29:51.589560+0000","last_peered":"2024-03-28T23:29:51.589560+0000","last_clean":"2024-03-28T23:29:51.589560+0000","last_became_active":"2024-03-28T23:29:45.560420+0000","last_became_peered":"2024-03-28T23:29:45.560420+0000","last_unstale":"2024-03-28T23:29:51.589560+0000","last_undegraded":"2024-03-28T23:29:51.589560+0000","last_fullsized":"2024-03-28T23:29:51.589560+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:17:17.207826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.590530+0000","last_change":"2024-03-28T23:29:50.711247+0000","last_active":"2024-03-28T23:29:51.590530+0000","last_peered":"2024-03-28T23:29:51.590530+0000","last_clean":"2024-03-28T23:29:51.590530+0000","last_became_active":"2024-03-28T23:29:50.710836+0000","last_became_peered":"2024-03-28T23:29:50.710836+0000","last_unstale":"2024-03-28T23:29:51.590530+0000","last_undegraded":"2024-03-28T23:29:51.590530+0000","last_fullsized":"2024-03-28T23:29:51.590530+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:58.504770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685843+0000","last_change":"2024-03-28T23:29:46.561283+0000","last_active":"2024-03-28T23:29:51.685843+0000","last_peered":"2024-03-28T23:29:51.685843+0000","last_clean":"2024-03-28T23:29:51.685843+0000","last_became_active":"2024-03-28T23:29:46.560584+0000","last_became_peered":"2024-03-28T23:29:46.560584+0000","last_unstale":"2024-03-28T23:29:51.685843+0000","last_undegraded":"2024-03-28T23:29:51.685843+0000","last_fullsized":"2024-03-28T23:29:51.685843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:06:36.882961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686047+0000","last_change":"2024-03-28T23:29:45.560013+0000","last_active":"2024-03-28T23:29:51.686047+0000","last_peered":"2024-03-28T23:29:51.686047+0000","last_clean":"2024-03-28T23:29:51.686047+0000","last_became_active":"2024-03-28T23:29:45.559643+0000","last_became_peered":"2024-03-28T23:29:45.559643+0000","last_unstale":"2024-03-28T23:29:51.686047+0000","last_undegraded":"2024-03-28T23:29:51.686047+0000","last_fullsized":"2024-03-28T23:29:51.686047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:01.983721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687005+0000","last_change":"2024-03-28T23:29:50.595120+0000","last_active":"2024-03-28T23:29:51.687005+0000","last_peered":"2024-03-28T23:29:51.687005+0000","last_clean":"2024-03-28T23:29:51.687005+0000","last_became_active":"2024-03-28T23:29:50.594780+0000","last_became_peered":"2024-03-28T23:29:50.594780+0000","last_unstale":"2024-03-28T23:29:51.687005+0000","last_undegraded":"2024-03-28T23:29:51.687005+0000","last_fullsized":"2024-03-28T23:29:51.687005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:06.894069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686843+0000","last_change":"2024-03-28T23:29:46.567707+0000","last_active":"2024-03-28T23:29:51.686843+0000","last_peered":"2024-03-28T23:29:51.686843+0000","last_clean":"2024-03-28T23:29:51.686843+0000","last_became_active":"2024-03-28T23:29:46.565675+0000","last_became_peered":"2024-03-28T23:29:46.565675+0000","last_unstale":"2024-03-28T23:29:51.686843+0000","last_undegraded":"2024-03-28T23:29:51.686843+0000","last_fullsized":"2024-03-28T23:29:51.686843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:07:21.524344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589416+0000","last_change":"2024-03-28T23:29:45.558538+0000","last_active":"2024-03-28T23:29:51.589416+0000","last_peered":"2024-03-28T23:29:51.589416+0000","last_clean":"2024-03-28T23:29:51.589416+0000","last_became_active":"2024-03-28T23:29:45.558248+0000","last_became_peered":"2024-03-28T23:29:45.558248+0000","last_unstale":"2024-03-28T23:29:51.589416+0000","last_undegraded":"2024-03-28T23:29:51.589416+0000","last_fullsized":"2024-03-28T23:29:51.589416+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:59.719252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685418+0000","last_change":"2024-03-28T23:29:50.707640+0000","last_active":"2024-03-28T23:29:51.685418+0000","last_peered":"2024-03-28T23:29:51.685418+0000","last_clean":"2024-03-28T23:29:51.685418+0000","last_became_active":"2024-03-28T23:29:50.707258+0000","last_became_peered":"2024-03-28T23:29:50.707258+0000","last_unstale":"2024-03-28T23:29:51.685418+0000","last_undegraded":"2024-03-28T23:29:51.685418+0000","last_fullsized":"2024-03-28T23:29:51.685418+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:47.094789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686505+0000","last_change":"2024-03-28T23:29:46.565431+0000","last_active":"2024-03-28T23:29:51.686505+0000","last_peered":"2024-03-28T23:29:51.686505+0000","last_clean":"2024-03-28T23:29:51.686505+0000","last_became_active":"2024-03-28T23:29:46.565163+0000","last_became_peered":"2024-03-28T23:29:46.565163+0000","last_unstale":"2024-03-28T23:29:51.686505+0000","last_undegraded":"2024-03-28T23:29:51.686505+0000","last_fullsized":"2024-03-28T23:29:51.686505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:02.859502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591420+0000","last_change":"2024-03-28T23:29:45.564138+0000","last_active":"2024-03-28T23:29:51.591420+0000","last_peered":"2024-03-28T23:29:51.591420+0000","last_clean":"2024-03-28T23:29:51.591420+0000","last_became_active":"2024-03-28T23:29:45.563858+0000","last_became_peered":"2024-03-28T23:29:45.563858+0000","last_unstale":"2024-03-28T23:29:51.591420+0000","last_undegraded":"2024-03-28T23:29:51.591420+0000","last_fullsized":"2024-03-28T23:29:51.591420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:42:38.893826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.592126+0000","last_change":"2024-03-28T23:29:50.599484+0000","last_active":"2024-03-28T23:29:51.592126+0000","last_peered":"2024-03-28T23:29:51.592126+0000","last_clean":"2024-03-28T23:29:51.592126+0000","last_became_active":"2024-03-28T23:29:50.599035+0000","last_became_peered":"2024-03-28T23:29:50.599035+0000","last_unstale":"2024-03-28T23:29:51.592126+0000","last_undegraded":"2024-03-28T23:29:51.592126+0000","last_fullsized":"2024-03-28T23:29:51.592126+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:10.765830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686919+0000","last_change":"2024-03-28T23:29:46.567472+0000","last_active":"2024-03-28T23:29:51.686919+0000","last_peered":"2024-03-28T23:29:51.686919+0000","last_clean":"2024-03-28T23:29:51.686919+0000","last_became_active":"2024-03-28T23:29:46.565485+0000","last_became_peered":"2024-03-28T23:29:46.565485+0000","last_unstale":"2024-03-28T23:29:51.686919+0000","last_undegraded":"2024-03-28T23:29:51.686919+0000","last_fullsized":"2024-03-28T23:29:51.686919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:29:20.669429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589103+0000","last_change":"2024-03-28T23:29:45.556004+0000","last_active":"2024-03-28T23:29:51.589103+0000","last_peered":"2024-03-28T23:29:51.589103+0000","last_clean":"2024-03-28T23:29:51.589103+0000","last_became_active":"2024-03-28T23:29:45.555701+0000","last_became_peered":"2024-03-28T23:29:45.555701+0000","last_unstale":"2024-03-28T23:29:51.589103+0000","last_undegraded":"2024-03-28T23:29:51.589103+0000","last_fullsized":"2024-03-28T23:29:51.589103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:06:18.025605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687430+0000","last_change":"2024-03-28T23:29:50.596439+0000","last_active":"2024-03-28T23:29:51.687430+0000","last_peered":"2024-03-28T23:29:51.687430+0000","last_clean":"2024-03-28T23:29:51.687430+0000","last_became_active":"2024-03-28T23:29:50.596065+0000","last_became_peered":"2024-03-28T23:29:50.596065+0000","last_unstale":"2024-03-28T23:29:51.687430+0000","last_undegraded":"2024-03-28T23:29:51.687430+0000","last_fullsized":"2024-03-28T23:29:51.687430+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:43:14.415039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.590719+0000","last_change":"2024-03-28T23:29:46.581708+0000","last_active":"2024-03-28T23:29:51.590719+0000","last_peered":"2024-03-28T23:29:51.590719+0000","last_clean":"2024-03-28T23:29:51.590719+0000","last_became_active":"2024-03-28T23:29:46.581391+0000","last_became_peered":"2024-03-28T23:29:46.581391+0000","last_unstale":"2024-03-28T23:29:51.590719+0000","last_undegraded":"2024-03-28T23:29:51.590719+0000","last_fullsized":"2024-03-28T23:29:51.590719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:24:19.353047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591187+0000","last_change":"2024-03-28T23:29:45.560112+0000","last_active":"2024-03-28T23:29:51.591187+0000","last_peered":"2024-03-28T23:29:51.591187+0000","last_clean":"2024-03-28T23:29:51.591187+0000","last_became_active":"2024-03-28T23:29:45.559817+0000","last_became_peered":"2024-03-28T23:29:45.559817+0000","last_unstale":"2024-03-28T23:29:51.591187+0000","last_undegraded":"2024-03-28T23:29:51.591187+0000","last_fullsized":"2024-03-28T23:29:51.591187+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:21.707459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687297+0000","last_change":"2024-03-28T23:29:50.712185+0000","last_active":"2024-03-28T23:29:51.687297+0000","last_peered":"2024-03-28T23:29:51.687297+0000","last_clean":"2024-03-28T23:29:51.687297+0000","last_became_active":"2024-03-28T23:29:50.711823+0000","last_became_peered":"2024-03-28T23:29:50.711823+0000","last_unstale":"2024-03-28T23:29:51.687297+0000","last_undegraded":"2024-03-28T23:29:51.687297+0000","last_fullsized":"2024-03-28T23:29:51.687297+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:40.107649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685755+0000","last_change":"2024-03-28T23:29:46.570826+0000","last_active":"2024-03-28T23:29:51.685755+0000","last_peered":"2024-03-28T23:29:51.685755+0000","last_clean":"2024-03-28T23:29:51.685755+0000","last_became_active":"2024-03-28T23:29:46.570513+0000","last_became_peered":"2024-03-28T23:29:46.570513+0000","last_unstale":"2024-03-28T23:29:51.685755+0000","last_undegraded":"2024-03-28T23:29:51.685755+0000","last_fullsized":"2024-03-28T23:29:51.685755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:08:33.801844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686189+0000","last_change":"2024-03-28T23:29:45.554954+0000","last_active":"2024-03-28T23:29:51.686189+0000","last_peered":"2024-03-28T23:29:51.686189+0000","last_clean":"2024-03-28T23:29:51.686189+0000","last_became_active":"2024-03-28T23:29:45.554687+0000","last_became_peered":"2024-03-28T23:29:45.554687+0000","last_unstale":"2024-03-28T23:29:51.686189+0000","last_undegraded":"2024-03-28T23:29:51.686189+0000","last_fullsized":"2024-03-28T23:29:51.686189+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:24:58.784056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687871+0000","last_change":"2024-03-28T23:29:50.707207+0000","last_active":"2024-03-28T23:29:51.687871+0000","last_peered":"2024-03-28T23:29:51.687871+0000","last_clean":"2024-03-28T23:29:51.687871+0000","last_became_active":"2024-03-28T23:29:50.706585+0000","last_became_peered":"2024-03-28T23:29:50.706585+0000","last_unstale":"2024-03-28T23:29:51.687871+0000","last_undegraded":"2024-03-28T23:29:51.687871+0000","last_fullsized":"2024-03-28T23:29:51.687871+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:10:11.936256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687068+0000","last_change":"2024-03-28T23:29:46.578519+0000","last_active":"2024-03-28T23:29:51.687068+0000","last_peered":"2024-03-28T23:29:51.687068+0000","last_clean":"2024-03-28T23:29:51.687068+0000","last_became_active":"2024-03-28T23:29:46.577803+0000","last_became_peered":"2024-03-28T23:29:46.577803+0000","last_unstale":"2024-03-28T23:29:51.687068+0000","last_undegraded":"2024-03-28T23:29:51.687068+0000","last_fullsized":"2024-03-28T23:29:51.687068+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:32.936386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687237+0000","last_change":"2024-03-28T23:29:45.557114+0000","last_active":"2024-03-28T23:29:51.687237+0000","last_peered":"2024-03-28T23:29:51.687237+0000","last_clean":"2024-03-28T23:29:51.687237+0000","last_became_active":"2024-03-28T23:29:45.556800+0000","last_became_peered":"2024-03-28T23:29:45.556800+0000","last_unstale":"2024-03-28T23:29:51.687237+0000","last_undegraded":"2024-03-28T23:29:51.687237+0000","last_fullsized":"2024-03-28T23:29:51.687237+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:06:18.025605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686083+0000","last_change":"2024-03-28T23:29:50.707356+0000","last_active":"2024-03-28T23:29:51.686083+0000","last_peered":"2024-03-28T23:29:51.686083+0000","last_clean":"2024-03-28T23:29:51.686083+0000","last_became_active":"2024-03-28T23:29:50.706320+0000","last_became_peered":"2024-03-28T23:29:50.706320+0000","last_unstale":"2024-03-28T23:29:51.686083+0000","last_undegraded":"2024-03-28T23:29:51.686083+0000","last_fullsized":"2024-03-28T23:29:51.686083+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:36.341186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686444+0000","last_change":"2024-03-28T23:29:46.573192+0000","last_active":"2024-03-28T23:29:51.686444+0000","last_peered":"2024-03-28T23:29:51.686444+0000","last_clean":"2024-03-28T23:29:51.686444+0000","last_became_active":"2024-03-28T23:29:46.572701+0000","last_became_peered":"2024-03-28T23:29:46.572701+0000","last_unstale":"2024-03-28T23:29:51.686444+0000","last_undegraded":"2024-03-28T23:29:51.686444+0000","last_fullsized":"2024-03-28T23:29:51.686444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:17:49.979210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:56.151164+0000","last_change":"2024-03-28T23:29:45.558929+0000","last_active":"2024-03-28T23:29:56.151164+0000","last_peered":"2024-03-28T23:29:56.151164+0000","last_clean":"2024-03-28T23:29:56.151164+0000","last_became_active":"2024-03-28T23:29:45.558657+0000","last_became_peered":"2024-03-28T23:29:45.558657+0000","last_unstale":"2024-03-28T23:29:56.151164+0000","last_undegraded":"2024-03-28T23:29:56.151164+0000","last_fullsized":"2024-03-28T23:29:56.151164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:24:58.784056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686265+0000","last_change":"2024-03-28T23:29:50.594405+0000","last_active":"2024-03-28T23:29:51.686265+0000","last_peered":"2024-03-28T23:29:51.686265+0000","last_clean":"2024-03-28T23:29:51.686265+0000","last_became_active":"2024-03-28T23:29:50.594017+0000","last_became_peered":"2024-03-28T23:29:50.594017+0000","last_unstale":"2024-03-28T23:29:51.686265+0000","last_undegraded":"2024-03-28T23:29:51.686265+0000","last_fullsized":"2024-03-28T23:29:51.686265+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:06.894069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591773+0000","last_change":"2024-03-28T23:29:46.561971+0000","last_active":"2024-03-28T23:29:51.591773+0000","last_peered":"2024-03-28T23:29:51.591773+0000","last_clean":"2024-03-28T23:29:51.591773+0000","last_became_active":"2024-03-28T23:29:46.561684+0000","last_became_peered":"2024-03-28T23:29:46.561684+0000","last_unstale":"2024-03-28T23:29:51.591773+0000","last_undegraded":"2024-03-28T23:29:51.591773+0000","last_fullsized":"2024-03-28T23:29:51.591773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:42.629935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685953+0000","last_change":"2024-03-28T23:29:45.563130+0000","last_active":"2024-03-28T23:29:51.685953+0000","last_peered":"2024-03-28T23:29:51.685953+0000","last_clean":"2024-03-28T23:29:51.685953+0000","last_became_active":"2024-03-28T23:29:45.562696+0000","last_became_peered":"2024-03-28T23:29:45.562696+0000","last_unstale":"2024-03-28T23:29:51.685953+0000","last_undegraded":"2024-03-28T23:29:51.685953+0000","last_fullsized":"2024-03-28T23:29:51.685953+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:17.783972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.592114+0000","last_change":"2024-03-28T23:29:50.587580+0000","last_active":"2024-03-28T23:29:51.592114+0000","last_peered":"2024-03-28T23:29:51.592114+0000","last_clean":"2024-03-28T23:29:51.592114+0000","last_became_active":"2024-03-28T23:29:50.587175+0000","last_became_peered":"2024-03-28T23:29:50.587175+0000","last_unstale":"2024-03-28T23:29:51.592114+0000","last_undegraded":"2024-03-28T23:29:51.592114+0000","last_fullsized":"2024-03-28T23:29:51.592114+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:06:40.917529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687664+0000","last_change":"2024-03-28T23:29:46.578684+0000","last_active":"2024-03-28T23:29:51.687664+0000","last_peered":"2024-03-28T23:29:51.687664+0000","last_clean":"2024-03-28T23:29:51.687664+0000","last_became_active":"2024-03-28T23:29:46.577924+0000","last_became_peered":"2024-03-28T23:29:46.577924+0000","last_unstale":"2024-03-28T23:29:51.687664+0000","last_undegraded":"2024-03-28T23:29:51.687664+0000","last_fullsized":"2024-03-28T23:29:51.687664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:36:20.485861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685874+0000","last_change":"2024-03-28T23:29:45.559534+0000","last_active":"2024-03-28T23:29:51.685874+0000","last_peered":"2024-03-28T23:29:51.685874+0000","last_clean":"2024-03-28T23:29:51.685874+0000","last_became_active":"2024-03-28T23:29:45.559243+0000","last_became_peered":"2024-03-28T23:29:45.559243+0000","last_unstale":"2024-03-28T23:29:51.685874+0000","last_undegraded":"2024-03-28T23:29:51.685874+0000","last_fullsized":"2024-03-28T23:29:51.685874+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:17:17.207826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686108+0000","last_change":"2024-03-28T23:29:50.601743+0000","last_active":"2024-03-28T23:29:51.686108+0000","last_peered":"2024-03-28T23:29:51.686108+0000","last_clean":"2024-03-28T23:29:51.686108+0000","last_became_active":"2024-03-28T23:29:50.601327+0000","last_became_peered":"2024-03-28T23:29:50.601327+0000","last_unstale":"2024-03-28T23:29:51.686108+0000","last_undegraded":"2024-03-28T23:29:51.686108+0000","last_fullsized":"2024-03-28T23:29:51.686108+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:36.341186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685724+0000","last_change":"2024-03-28T23:29:46.573188+0000","last_active":"2024-03-28T23:29:51.685724+0000","last_peered":"2024-03-28T23:29:51.685724+0000","last_clean":"2024-03-28T23:29:51.685724+0000","last_became_active":"2024-03-28T23:29:46.572885+0000","last_became_peered":"2024-03-28T23:29:46.572885+0000","last_unstale":"2024-03-28T23:29:51.685724+0000","last_undegraded":"2024-03-28T23:29:51.685724+0000","last_fullsized":"2024-03-28T23:29:51.685724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:08:33.801844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591244+0000","last_change":"2024-03-28T23:29:45.564128+0000","last_active":"2024-03-28T23:29:51.591244+0000","last_peered":"2024-03-28T23:29:51.591244+0000","last_clean":"2024-03-28T23:29:51.591244+0000","last_became_active":"2024-03-28T23:29:45.563837+0000","last_became_peered":"2024-03-28T23:29:45.563837+0000","last_unstale":"2024-03-28T23:29:51.591244+0000","last_undegraded":"2024-03-28T23:29:51.591244+0000","last_fullsized":"2024-03-28T23:29:51.591244+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:17:17.207826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686626+0000","last_change":"2024-03-28T23:29:50.708325+0000","last_active":"2024-03-28T23:29:51.686626+0000","last_peered":"2024-03-28T23:29:51.686626+0000","last_clean":"2024-03-28T23:29:51.686626+0000","last_became_active":"2024-03-28T23:29:50.707929+0000","last_became_peered":"2024-03-28T23:29:50.707929+0000","last_unstale":"2024-03-28T23:29:51.686626+0000","last_undegraded":"2024-03-28T23:29:51.686626+0000","last_fullsized":"2024-03-28T23:29:51.686626+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:01.743074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687303+0000","last_change":"2024-03-28T23:29:50.711341+0000","last_active":"2024-03-28T23:29:51.687303+0000","last_peered":"2024-03-28T23:29:51.687303+0000","last_clean":"2024-03-28T23:29:51.687303+0000","last_became_active":"2024-03-28T23:29:50.710983+0000","last_became_peered":"2024-03-28T23:29:50.710983+0000","last_unstale":"2024-03-28T23:29:51.687303+0000","last_undegraded":"2024-03-28T23:29:51.687303+0000","last_fullsized":"2024-03-28T23:29:51.687303+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:30:48.379529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685433+0000","last_change":"2024-03-28T23:29:46.561654+0000","last_active":"2024-03-28T23:29:51.685433+0000","last_peered":"2024-03-28T23:29:51.685433+0000","last_clean":"2024-03-28T23:29:51.685433+0000","last_became_active":"2024-03-28T23:29:46.561250+0000","last_became_peered":"2024-03-28T23:29:46.561250+0000","last_unstale":"2024-03-28T23:29:51.685433+0000","last_undegraded":"2024-03-28T23:29:51.685433+0000","last_fullsized":"2024-03-28T23:29:51.685433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:38.074388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685648+0000","last_change":"2024-03-28T23:29:45.553472+0000","last_active":"2024-03-28T23:29:51.685648+0000","last_peered":"2024-03-28T23:29:51.685648+0000","last_clean":"2024-03-28T23:29:51.685648+0000","last_became_active":"2024-03-28T23:29:45.553154+0000","last_became_peered":"2024-03-28T23:29:45.553154+0000","last_unstale":"2024-03-28T23:29:51.685648+0000","last_undegraded":"2024-03-28T23:29:51.685648+0000","last_fullsized":"2024-03-28T23:29:51.685648+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:36.113647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591126+0000","last_change":"2024-03-28T23:29:50.600387+0000","last_active":"2024-03-28T23:29:51.591126+0000","last_peered":"2024-03-28T23:29:51.591126+0000","last_clean":"2024-03-28T23:29:51.591126+0000","last_became_active":"2024-03-28T23:29:50.599850+0000","last_became_peered":"2024-03-28T23:29:50.599850+0000","last_unstale":"2024-03-28T23:29:51.591126+0000","last_undegraded":"2024-03-28T23:29:51.591126+0000","last_fullsized":"2024-03-28T23:29:51.591126+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:08:37.836411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686149+0000","last_change":"2024-03-28T23:29:46.581533+0000","last_active":"2024-03-28T23:29:51.686149+0000","last_peered":"2024-03-28T23:29:51.686149+0000","last_clean":"2024-03-28T23:29:51.686149+0000","last_became_active":"2024-03-28T23:29:46.581288+0000","last_became_peered":"2024-03-28T23:29:46.581288+0000","last_unstale":"2024-03-28T23:29:51.686149+0000","last_undegraded":"2024-03-28T23:29:51.686149+0000","last_fullsized":"2024-03-28T23:29:51.686149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:24:19.353047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686836+0000","last_change":"2024-03-28T23:29:45.561047+0000","last_active":"2024-03-28T23:29:51.686836+0000","last_peered":"2024-03-28T23:29:51.686836+0000","last_clean":"2024-03-28T23:29:51.686836+0000","last_became_active":"2024-03-28T23:29:45.560778+0000","last_became_peered":"2024-03-28T23:29:45.560778+0000","last_unstale":"2024-03-28T23:29:51.686836+0000","last_undegraded":"2024-03-28T23:29:51.686836+0000","last_fullsized":"2024-03-28T23:29:51.686836+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:21.707459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.590800+0000","last_change":"2024-03-28T23:29:45.559446+0000","last_active":"2024-03-28T23:29:51.590800+0000","last_peered":"2024-03-28T23:29:51.590800+0000","last_clean":"2024-03-28T23:29:51.590800+0000","last_became_active":"2024-03-28T23:29:45.559015+0000","last_became_peered":"2024-03-28T23:29:45.559015+0000","last_unstale":"2024-03-28T23:29:51.590800+0000","last_undegraded":"2024-03-28T23:29:51.590800+0000","last_fullsized":"2024-03-28T23:29:51.590800+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:59.719252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687152+0000","last_change":"2024-03-28T23:29:46.578355+0000","last_active":"2024-03-28T23:29:51.687152+0000","last_peered":"2024-03-28T23:29:51.687152+0000","last_clean":"2024-03-28T23:29:51.687152+0000","last_became_active":"2024-03-28T23:29:46.577898+0000","last_became_peered":"2024-03-28T23:29:46.577898+0000","last_unstale":"2024-03-28T23:29:51.687152+0000","last_undegraded":"2024-03-28T23:29:51.687152+0000","last_fullsized":"2024-03-28T23:29:51.687152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:58.674327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589392+0000","last_change":"2024-03-28T23:29:50.601015+0000","last_active":"2024-03-28T23:29:51.589392+0000","last_peered":"2024-03-28T23:29:51.589392+0000","last_clean":"2024-03-28T23:29:51.589392+0000","last_became_active":"2024-03-28T23:29:50.600573+0000","last_became_peered":"2024-03-28T23:29:50.600573+0000","last_unstale":"2024-03-28T23:29:51.589392+0000","last_undegraded":"2024-03-28T23:29:51.589392+0000","last_fullsized":"2024-03-28T23:29:51.589392+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:38:05.665416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686050+0000","last_change":"2024-03-28T23:29:45.556159+0000","last_active":"2024-03-28T23:29:51.686050+0000","last_peered":"2024-03-28T23:29:51.686050+0000","last_clean":"2024-03-28T23:29:51.686050+0000","last_became_active":"2024-03-28T23:29:45.555866+0000","last_became_peered":"2024-03-28T23:29:45.555866+0000","last_unstale":"2024-03-28T23:29:51.686050+0000","last_undegraded":"2024-03-28T23:29:51.686050+0000","last_fullsized":"2024-03-28T23:29:51.686050+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:17:17.207826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591783+0000","last_change":"2024-03-28T23:29:46.554056+0000","last_active":"2024-03-28T23:29:51.591783+0000","last_peered":"2024-03-28T23:29:51.591783+0000","last_clean":"2024-03-28T23:29:51.591783+0000","last_became_active":"2024-03-28T23:29:46.553772+0000","last_became_peered":"2024-03-28T23:29:46.553772+0000","last_unstale":"2024-03-28T23:29:51.591783+0000","last_undegraded":"2024-03-28T23:29:51.591783+0000","last_fullsized":"2024-03-28T23:29:51.591783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:02.992468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686381+0000","last_change":"2024-03-28T23:29:50.603254+0000","last_active":"2024-03-28T23:29:51.686381+0000","last_peered":"2024-03-28T23:29:51.686381+0000","last_clean":"2024-03-28T23:29:51.686381+0000","last_became_active":"2024-03-28T23:29:50.602868+0000","last_became_peered":"2024-03-28T23:29:50.602868+0000","last_unstale":"2024-03-28T23:29:51.686381+0000","last_undegraded":"2024-03-28T23:29:51.686381+0000","last_fullsized":"2024-03-28T23:29:51.686381+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:42:43.014557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.590502+0000","last_change":"2024-03-28T23:29:46.581634+0000","last_active":"2024-03-28T23:29:51.590502+0000","last_peered":"2024-03-28T23:29:51.590502+0000","last_clean":"2024-03-28T23:29:51.590502+0000","last_became_active":"2024-03-28T23:29:46.581353+0000","last_became_peered":"2024-03-28T23:29:46.581353+0000","last_unstale":"2024-03-28T23:29:51.590502+0000","last_undegraded":"2024-03-28T23:29:51.590502+0000","last_fullsized":"2024-03-28T23:29:51.590502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:20.039954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589110+0000","last_change":"2024-03-28T23:29:46.563584+0000","last_active":"2024-03-28T23:29:51.589110+0000","last_peered":"2024-03-28T23:29:51.589110+0000","last_clean":"2024-03-28T23:29:51.589110+0000","last_became_active":"2024-03-28T23:29:46.563265+0000","last_became_peered":"2024-03-28T23:29:46.563265+0000","last_unstale":"2024-03-28T23:29:51.589110+0000","last_undegraded":"2024-03-28T23:29:51.589110+0000","last_fullsized":"2024-03-28T23:29:51.589110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:42.629935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685798+0000","last_change":"2024-03-28T23:29:46.561377+0000","last_active":"2024-03-28T23:29:51.685798+0000","last_peered":"2024-03-28T23:29:51.685798+0000","last_clean":"2024-03-28T23:29:51.685798+0000","last_became_active":"2024-03-28T23:29:46.561091+0000","last_became_peered":"2024-03-28T23:29:46.561091+0000","last_unstale":"2024-03-28T23:29:51.685798+0000","last_undegraded":"2024-03-28T23:29:51.685798+0000","last_fullsized":"2024-03-28T23:29:51.685798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:30:44.344961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687062+0000","last_change":"2024-03-28T23:29:46.578999+0000","last_active":"2024-03-28T23:29:51.687062+0000","last_peered":"2024-03-28T23:29:51.687062+0000","last_clean":"2024-03-28T23:29:51.687062+0000","last_became_active":"2024-03-28T23:29:46.578263+0000","last_became_peered":"2024-03-28T23:29:46.578263+0000","last_unstale":"2024-03-28T23:29:51.687062+0000","last_undegraded":"2024-03-28T23:29:51.687062+0000","last_fullsized":"2024-03-28T23:29:51.687062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:10.853710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687925+0000","last_change":"2024-03-28T23:29:46.566611+0000","last_active":"2024-03-28T23:29:51.687925+0000","last_peered":"2024-03-28T23:29:51.687925+0000","last_clean":"2024-03-28T23:29:51.687925+0000","last_became_active":"2024-03-28T23:29:46.566361+0000","last_became_peered":"2024-03-28T23:29:46.566361+0000","last_unstale":"2024-03-28T23:29:51.687925+0000","last_undegraded":"2024-03-28T23:29:51.687925+0000","last_fullsized":"2024-03-28T23:29:51.687925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:29:20.669429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685784+0000","last_change":"2024-03-28T23:29:46.561125+0000","last_active":"2024-03-28T23:29:51.685784+0000","last_peered":"2024-03-28T23:29:51.685784+0000","last_clean":"2024-03-28T23:29:51.685784+0000","last_became_active":"2024-03-28T23:29:46.559223+0000","last_became_peered":"2024-03-28T23:29:46.559223+0000","last_unstale":"2024-03-28T23:29:51.685784+0000","last_undegraded":"2024-03-28T23:29:51.685784+0000","last_fullsized":"2024-03-28T23:29:51.685784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:36:20.485861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591456+0000","last_change":"2024-03-28T23:29:46.574608+0000","last_active":"2024-03-28T23:29:51.591456+0000","last_peered":"2024-03-28T23:29:51.591456+0000","last_clean":"2024-03-28T23:29:51.591456+0000","last_became_active":"2024-03-28T23:29:46.574329+0000","last_became_peered":"2024-03-28T23:29:46.574329+0000","last_unstale":"2024-03-28T23:29:51.591456+0000","last_undegraded":"2024-03-28T23:29:51.591456+0000","last_fullsized":"2024-03-28T23:29:51.591456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:15.386423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687344+0000","last_change":"2024-03-28T23:29:46.566187+0000","last_active":"2024-03-28T23:29:51.687344+0000","last_peered":"2024-03-28T23:29:51.687344+0000","last_clean":"2024-03-28T23:29:51.687344+0000","last_became_active":"2024-03-28T23:29:46.565873+0000","last_became_peered":"2024-03-28T23:29:46.565873+0000","last_unstale":"2024-03-28T23:29:51.687344+0000","last_undegraded":"2024-03-28T23:29:51.687344+0000","last_fullsized":"2024-03-28T23:29:51.687344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:14:59.182189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685520+0000","last_change":"2024-03-28T23:29:46.559531+0000","last_active":"2024-03-28T23:29:51.685520+0000","last_peered":"2024-03-28T23:29:51.685520+0000","last_clean":"2024-03-28T23:29:51.685520+0000","last_became_active":"2024-03-28T23:29:46.559236+0000","last_became_peered":"2024-03-28T23:29:46.559236+0000","last_unstale":"2024-03-28T23:29:51.685520+0000","last_undegraded":"2024-03-28T23:29:51.685520+0000","last_fullsized":"2024-03-28T23:29:51.685520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:06.731262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686762+0000","last_change":"2024-03-28T23:29:46.555898+0000","last_active":"2024-03-28T23:29:51.686762+0000","last_peered":"2024-03-28T23:29:51.686762+0000","last_clean":"2024-03-28T23:29:51.686762+0000","last_became_active":"2024-03-28T23:29:46.555610+0000","last_became_peered":"2024-03-28T23:29:46.555610+0000","last_unstale":"2024-03-28T23:29:51.686762+0000","last_undegraded":"2024-03-28T23:29:51.686762+0000","last_fullsized":"2024-03-28T23:29:51.686762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:38.074388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685565+0000","last_change":"2024-03-28T23:29:46.573905+0000","last_active":"2024-03-28T23:29:51.685565+0000","last_peered":"2024-03-28T23:29:51.685565+0000","last_clean":"2024-03-28T23:29:51.685565+0000","last_became_active":"2024-03-28T23:29:46.573631+0000","last_became_peered":"2024-03-28T23:29:46.573631+0000","last_unstale":"2024-03-28T23:29:51.685565+0000","last_undegraded":"2024-03-28T23:29:51.685565+0000","last_fullsized":"2024-03-28T23:29:51.685565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:14:59.182189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686795+0000","last_change":"2024-03-28T23:29:46.579400+0000","last_active":"2024-03-28T23:29:51.686795+0000","last_peered":"2024-03-28T23:29:51.686795+0000","last_clean":"2024-03-28T23:29:51.686795+0000","last_became_active":"2024-03-28T23:29:46.579070+0000","last_became_peered":"2024-03-28T23:29:46.579070+0000","last_unstale":"2024-03-28T23:29:51.686795+0000","last_undegraded":"2024-03-28T23:29:51.686795+0000","last_fullsized":"2024-03-28T23:29:51.686795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:15.386423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591130+0000","last_change":"2024-03-28T23:29:46.569441+0000","last_active":"2024-03-28T23:29:51.591130+0000","last_peered":"2024-03-28T23:29:51.591130+0000","last_clean":"2024-03-28T23:29:51.591130+0000","last_became_active":"2024-03-28T23:29:46.569151+0000","last_became_peered":"2024-03-28T23:29:46.569151+0000","last_unstale":"2024-03-28T23:29:51.591130+0000","last_undegraded":"2024-03-28T23:29:51.591130+0000","last_fullsized":"2024-03-28T23:29:51.591130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:38.074388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686720+0000","last_change":"2024-03-28T23:29:46.573927+0000","last_active":"2024-03-28T23:29:51.686720+0000","last_peered":"2024-03-28T23:29:51.686720+0000","last_clean":"2024-03-28T23:29:51.686720+0000","last_became_active":"2024-03-28T23:29:46.573668+0000","last_became_peered":"2024-03-28T23:29:46.573668+0000","last_unstale":"2024-03-28T23:29:51.686720+0000","last_undegraded":"2024-03-28T23:29:51.686720+0000","last_fullsized":"2024-03-28T23:29:51.686720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:30:44.344961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589322+0000","last_change":"2024-03-28T23:29:46.558629+0000","last_active":"2024-03-28T23:29:51.589322+0000","last_peered":"2024-03-28T23:29:51.589322+0000","last_clean":"2024-03-28T23:29:51.589322+0000","last_became_active":"2024-03-28T23:29:46.558311+0000","last_became_peered":"2024-03-28T23:29:46.558311+0000","last_unstale":"2024-03-28T23:29:51.589322+0000","last_undegraded":"2024-03-28T23:29:51.589322+0000","last_fullsized":"2024-03-28T23:29:51.589322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:10.938731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686262+0000","last_change":"2024-03-28T23:29:46.573394+0000","last_active":"2024-03-28T23:29:51.686262+0000","last_peered":"2024-03-28T23:29:51.686262+0000","last_clean":"2024-03-28T23:29:51.686262+0000","last_became_active":"2024-03-28T23:29:46.573130+0000","last_became_peered":"2024-03-28T23:29:46.573130+0000","last_unstale":"2024-03-28T23:29:51.686262+0000","last_undegraded":"2024-03-28T23:29:51.686262+0000","last_fullsized":"2024-03-28T23:29:51.686262+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:43:10.380471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685367+0000","last_change":"2024-03-28T23:29:46.581612+0000","last_active":"2024-03-28T23:29:51.685367+0000","last_peered":"2024-03-28T23:29:51.685367+0000","last_clean":"2024-03-28T23:29:51.685367+0000","last_became_active":"2024-03-28T23:29:46.581324+0000","last_became_peered":"2024-03-28T23:29:46.581324+0000","last_unstale":"2024-03-28T23:29:51.685367+0000","last_undegraded":"2024-03-28T23:29:51.685367+0000","last_fullsized":"2024-03-28T23:29:51.685367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:10.853710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685527+0000","last_change":"2024-03-28T23:29:46.552565+0000","last_active":"2024-03-28T23:29:51.685527+0000","last_peered":"2024-03-28T23:29:51.685527+0000","last_clean":"2024-03-28T23:29:51.685527+0000","last_became_active":"2024-03-28T23:29:46.552304+0000","last_became_peered":"2024-03-28T23:29:46.552304+0000","last_unstale":"2024-03-28T23:29:51.685527+0000","last_undegraded":"2024-03-28T23:29:51.685527+0000","last_fullsized":"2024-03-28T23:29:51.685527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:15.386423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686970+0000","last_change":"2024-03-28T23:29:46.579218+0000","last_active":"2024-03-28T23:29:51.686970+0000","last_peered":"2024-03-28T23:29:51.686970+0000","last_clean":"2024-03-28T23:29:51.686970+0000","last_became_active":"2024-03-28T23:29:46.578756+0000","last_became_peered":"2024-03-28T23:29:46.578756+0000","last_unstale":"2024-03-28T23:29:51.686970+0000","last_undegraded":"2024-03-28T23:29:51.686970+0000","last_fullsized":"2024-03-28T23:29:51.686970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:24:19.353047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685891+0000","last_change":"2024-03-28T23:29:46.556797+0000","last_active":"2024-03-28T23:29:51.685891+0000","last_peered":"2024-03-28T23:29:51.685891+0000","last_clean":"2024-03-28T23:29:51.685891+0000","last_became_active":"2024-03-28T23:29:46.556520+0000","last_became_peered":"2024-03-28T23:29:46.556520+0000","last_unstale":"2024-03-28T23:29:51.685891+0000","last_undegraded":"2024-03-28T23:29:51.685891+0000","last_fullsized":"2024-03-28T23:29:51.685891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:15.386423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591677+0000","last_change":"2024-03-28T23:29:46.559749+0000","last_active":"2024-03-28T23:29:51.591677+0000","last_peered":"2024-03-28T23:29:51.591677+0000","last_clean":"2024-03-28T23:29:51.591677+0000","last_became_active":"2024-03-28T23:29:46.559386+0000","last_became_peered":"2024-03-28T23:29:46.559386+0000","last_unstale":"2024-03-28T23:29:51.591677+0000","last_undegraded":"2024-03-28T23:29:51.591677+0000","last_fullsized":"2024-03-28T23:29:51.591677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:10.938731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591382+0000","last_change":"2024-03-28T23:29:46.571568+0000","last_active":"2024-03-28T23:29:51.591382+0000","last_peered":"2024-03-28T23:29:51.591382+0000","last_clean":"2024-03-28T23:29:51.591382+0000","last_became_active":"2024-03-28T23:29:46.571268+0000","last_became_peered":"2024-03-28T23:29:46.571268+0000","last_unstale":"2024-03-28T23:29:51.591382+0000","last_undegraded":"2024-03-28T23:29:51.591382+0000","last_fullsized":"2024-03-28T23:29:51.591382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:36:20.485861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591361+0000","last_change":"2024-03-28T23:29:46.571722+0000","last_active":"2024-03-28T23:29:51.591361+0000","last_peered":"2024-03-28T23:29:51.591361+0000","last_clean":"2024-03-28T23:29:51.591361+0000","last_became_active":"2024-03-28T23:29:46.571437+0000","last_became_peered":"2024-03-28T23:29:46.571437+0000","last_unstale":"2024-03-28T23:29:51.591361+0000","last_undegraded":"2024-03-28T23:29:51.591361+0000","last_fullsized":"2024-03-28T23:29:51.591361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:10.853710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686822+0000","last_change":"2024-03-28T23:29:46.565939+0000","last_active":"2024-03-28T23:29:51.686822+0000","last_peered":"2024-03-28T23:29:51.686822+0000","last_clean":"2024-03-28T23:29:51.686822+0000","last_became_active":"2024-03-28T23:29:46.564853+0000","last_became_peered":"2024-03-28T23:29:46.564853+0000","last_unstale":"2024-03-28T23:29:51.686822+0000","last_undegraded":"2024-03-28T23:29:51.686822+0000","last_fullsized":"2024-03-28T23:29:51.686822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:18.050658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685938+0000","last_change":"2024-03-28T23:29:46.567115+0000","last_active":"2024-03-28T23:29:51.685938+0000","last_peered":"2024-03-28T23:29:51.685938+0000","last_clean":"2024-03-28T23:29:51.685938+0000","last_became_active":"2024-03-28T23:29:46.566742+0000","last_became_peered":"2024-03-28T23:29:46.566742+0000","last_unstale":"2024-03-28T23:29:51.685938+0000","last_undegraded":"2024-03-28T23:29:51.685938+0000","last_fullsized":"2024-03-28T23:29:51.685938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:32.936386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686798+0000","last_change":"2024-03-28T23:29:46.579765+0000","last_active":"2024-03-28T23:29:51.686798+0000","last_peered":"2024-03-28T23:29:51.686798+0000","last_clean":"2024-03-28T23:29:51.686798+0000","last_became_active":"2024-03-28T23:29:46.579469+0000","last_became_peered":"2024-03-28T23:29:46.579469+0000","last_unstale":"2024-03-28T23:29:51.686798+0000","last_undegraded":"2024-03-28T23:29:51.686798+0000","last_fullsized":"2024-03-28T23:29:51.686798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:06.731262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685479+0000","last_change":"2024-03-28T23:29:46.580994+0000","last_active":"2024-03-28T23:29:51.685479+0000","last_peered":"2024-03-28T23:29:51.685479+0000","last_clean":"2024-03-28T23:29:51.685479+0000","last_became_active":"2024-03-28T23:29:46.580685+0000","last_became_peered":"2024-03-28T23:29:46.580685+0000","last_unstale":"2024-03-28T23:29:51.685479+0000","last_undegraded":"2024-03-28T23:29:51.685479+0000","last_fullsized":"2024-03-28T23:29:51.685479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:58.674327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685558+0000","last_change":"2024-03-28T23:29:46.571922+0000","last_active":"2024-03-28T23:29:51.685558+0000","last_peered":"2024-03-28T23:29:51.685558+0000","last_clean":"2024-03-28T23:29:51.685558+0000","last_became_active":"2024-03-28T23:29:46.571256+0000","last_became_peered":"2024-03-28T23:29:46.571256+0000","last_unstale":"2024-03-28T23:29:51.685558+0000","last_undegraded":"2024-03-28T23:29:51.685558+0000","last_fullsized":"2024-03-28T23:29:51.685558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:54.470202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685672+0000","last_change":"2024-03-28T23:29:46.561038+0000","last_active":"2024-03-28T23:29:51.685672+0000","last_peered":"2024-03-28T23:29:51.685672+0000","last_clean":"2024-03-28T23:29:51.685672+0000","last_became_active":"2024-03-28T23:29:46.560679+0000","last_became_peered":"2024-03-28T23:29:46.560679+0000","last_unstale":"2024-03-28T23:29:51.685672+0000","last_undegraded":"2024-03-28T23:29:51.685672+0000","last_fullsized":"2024-03-28T23:29:51.685672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:29:20.669429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687099+0000","last_change":"2024-03-28T23:29:46.578506+0000","last_active":"2024-03-28T23:29:51.687099+0000","last_peered":"2024-03-28T23:29:51.687099+0000","last_clean":"2024-03-28T23:29:51.687099+0000","last_became_active":"2024-03-28T23:29:46.577958+0000","last_became_peered":"2024-03-28T23:29:46.577958+0000","last_unstale":"2024-03-28T23:29:51.687099+0000","last_undegraded":"2024-03-28T23:29:51.687099+0000","last_fullsized":"2024-03-28T23:29:51.687099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:38:01.630848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686703+0000","last_change":"2024-03-28T23:29:46.579555+0000","last_active":"2024-03-28T23:29:51.686703+0000","last_peered":"2024-03-28T23:29:51.686703+0000","last_clean":"2024-03-28T23:29:51.686703+0000","last_became_active":"2024-03-28T23:29:46.579273+0000","last_became_peered":"2024-03-28T23:29:46.579273+0000","last_unstale":"2024-03-28T23:29:51.686703+0000","last_undegraded":"2024-03-28T23:29:51.686703+0000","last_fullsized":"2024-03-28T23:29:51.686703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:10.853710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686627+0000","last_change":"2024-03-28T23:29:46.568698+0000","last_active":"2024-03-28T23:29:51.686627+0000","last_peered":"2024-03-28T23:29:51.686627+0000","last_clean":"2024-03-28T23:29:51.686627+0000","last_became_active":"2024-03-28T23:29:46.568318+0000","last_became_peered":"2024-03-28T23:29:46.568318+0000","last_unstale":"2024-03-28T23:29:51.686627+0000","last_undegraded":"2024-03-28T23:29:51.686627+0000","last_fullsized":"2024-03-28T23:29:51.686627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:43:10.380471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31504,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340336,"statfs":{"total":96636764160,"available":96604504064,"internally_reserved":0,"allocated":323584,"data_stored":691206,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":331776,"data_stored":691262,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:30:14.500 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:30:14.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.652+0000 7f2ae6f8c640 1 Processor -- start 2024-03-28T23:30:14.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.652+0000 7f2ae6f8c640 1 -- start start 2024-03-28T23:30:14.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.653+0000 7f2ae6f8c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2ae0152890 0x7f2ae00c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:14.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.653+0000 7f2ae6f8c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2ae00c5b20 0x7f2ae01553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:14.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.653+0000 7f2ae6f8c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ae0158a30 con 0x7f2ae00c8ed0 2024-03-28T23:30:14.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.653+0000 7f2ae6f8c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2ae00c8480 con 0x7f2ae00c5b20 2024-03-28T23:30:14.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.653+0000 7f2ae6f8c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f2ae0159640 con 0x7f2ae0152890 2024-03-28T23:30:14.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.653+0000 7f2ae5529640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2ae00c5b20 0x7f2ae01553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:14.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.653+0000 7f2ad7fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f2ae00c8ed0 0x7f2ae00c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59700/0 (socket says 172.21.15.161:59700) 2024-03-28T23:30:14.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.653+0000 7f2ad7fff640 1 -- 172.21.15.161:0/318873990 learned_addr learned my addr 172.21.15.161:0/318873990 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:14.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.653+0000 7f2ae4d28640 1 --2- 172.21.15.161:0/318873990 >> v2:172.21.15.170:3300/0 conn(0x7f2ae0152890 0x7f2ae00c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:14.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.653+0000 7f2ae5529640 1 -- 172.21.15.161:0/318873990 >> v2:172.21.15.170:3300/0 conn(0x7f2ae0152890 msgr2=0x7f2ae00c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:14.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.653+0000 7f2ae5529640 1 --2- 172.21.15.161:0/318873990 >> v2:172.21.15.170:3300/0 conn(0x7f2ae0152890 0x7f2ae00c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.653+0000 7f2ae5529640 1 -- 172.21.15.161:0/318873990 >> v1:172.21.15.161:6789/0 conn(0x7f2ae00c8ed0 legacy=0x7f2ae00c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.653+0000 7f2ad77fe640 1 -- 172.21.15.161:0/318873990 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1319194746 0 0) 0x7f2ae0158a30 con 0x7f2ae00c8ed0 2024-03-28T23:30:14.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.653+0000 7f2ae5529640 1 -- 172.21.15.161:0/318873990 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae0158d10 con 0x7f2ae00c5b20 2024-03-28T23:30:14.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.653+0000 7f2ae4d28640 1 --2- 172.21.15.161:0/318873990 >> v2:172.21.15.170:3300/0 conn(0x7f2ae0152890 0x7f2ae00c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:14.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.654+0000 7f2ae5529640 1 --2- 172.21.15.161:0/318873990 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae00c5b20 0x7f2ae01553a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2ae008cc80 tx=0x7f2ad802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd6819c9454d0458 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:14.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.654+0000 7f2ad77fe640 1 -- 172.21.15.161:0/318873990 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8038070 con 0x7f2ae00c5b20 2024-03-28T23:30:14.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.654+0000 7f2ad77fe640 1 -- 172.21.15.161:0/318873990 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ad8031710 con 0x7f2ae00c5b20 2024-03-28T23:30:14.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.654+0000 7f2ad77fe640 1 -- 172.21.15.161:0/318873990 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8031ae0 con 0x7f2ae00c5b20 2024-03-28T23:30:14.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.655+0000 7f2ae6f8c640 1 -- 172.21.15.161:0/318873990 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae00c5b20 msgr2=0x7f2ae01553a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.655+0000 7f2ae6f8c640 1 --2- 172.21.15.161:0/318873990 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae00c5b20 0x7f2ae01553a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2ae008cc80 tx=0x7f2ad802b7b0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.655+0000 7f2ae6f8c640 1 -- 172.21.15.161:0/318873990 shutdown_connections 2024-03-28T23:30:14.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.655+0000 7f2ae6f8c640 1 --2- 172.21.15.161:0/318873990 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae00c5b20 0x7f2ae01553a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.655+0000 7f2ae6f8c640 1 --2- 172.21.15.161:0/318873990 >> v2:172.21.15.170:3300/0 conn(0x7f2ae0152890 0x7f2ae00c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.655+0000 7f2ae6f8c640 1 -- 172.21.15.161:0/318873990 >> 172.21.15.161:0/318873990 conn(0x7f2ae00ac620 msgr2=0x7f2ae00aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:14.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.655+0000 7f2ae6f8c640 1 -- 172.21.15.161:0/318873990 shutdown_connections 2024-03-28T23:30:14.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.655+0000 7f2ae6f8c640 1 -- 172.21.15.161:0/318873990 wait complete. 2024-03-28T23:30:14.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.656+0000 7f2ae6f8c640 1 Processor -- start 2024-03-28T23:30:14.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.656+0000 7f2ae6f8c640 1 -- start start 2024-03-28T23:30:14.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.657+0000 7f2ae6f8c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ae00c5b20 0x7f2ae0157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:14.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.657+0000 7f2ae6f8c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ae00c8ed0 0x7f2ae0153580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:14.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.657+0000 7f2ae6f8c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae0152890 0x7f2ae0153ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:14.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.657+0000 7f2ae6f8c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae0152c70 con 0x7f2ae0152890 2024-03-28T23:30:14.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.657+0000 7f2ae6f8c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae01592d0 con 0x7f2ae00c8ed0 2024-03-28T23:30:14.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.657+0000 7f2ae6f8c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2ae0152df0 con 0x7f2ae00c5b20 2024-03-28T23:30:14.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.657+0000 7f2ae5529640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae0152890 0x7f2ae0153ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:14.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.657+0000 7f2ae5529640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae0152890 0x7f2ae0153ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54478/0 (socket says 172.21.15.161:54478) 2024-03-28T23:30:14.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.657+0000 7f2ae5529640 1 -- 172.21.15.161:0/2952986080 learned_addr learned my addr 172.21.15.161:0/2952986080 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:14.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.657+0000 7f2ae4d28640 1 --2- 172.21.15.161:0/2952986080 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ae00c5b20 0x7f2ae0157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:14.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.657+0000 7f2ad7fff640 1 --2- 172.21.15.161:0/2952986080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ae00c8ed0 0x7f2ae0153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:14.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.657+0000 7f2ae5529640 1 -- 172.21.15.161:0/2952986080 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ae00c5b20 msgr2=0x7f2ae0157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.657+0000 7f2ae5529640 1 --2- 172.21.15.161:0/2952986080 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ae00c5b20 0x7f2ae0157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.657+0000 7f2ae5529640 1 -- 172.21.15.161:0/2952986080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ae00c8ed0 msgr2=0x7f2ae0153580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.657+0000 7f2ae5529640 1 --2- 172.21.15.161:0/2952986080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ae00c8ed0 0x7f2ae0153580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.657+0000 7f2ae5529640 1 -- 172.21.15.161:0/2952986080 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae01eb880 con 0x7f2ae0152890 2024-03-28T23:30:14.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.658+0000 7f2ae5529640 1 --2- 172.21.15.161:0/2952986080 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae0152890 0x7f2ae0153ac0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8009890 tx=0x7f2ad80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:14.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.659+0000 7f2ad57fa640 1 -- 172.21.15.161:0/2952986080 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8042020 con 0x7f2ae0152890 2024-03-28T23:30:14.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.659+0000 7f2ad57fa640 1 -- 172.21.15.161:0/2952986080 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ad80339c0 con 0x7f2ae0152890 2024-03-28T23:30:14.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.659+0000 7f2ad57fa640 1 -- 172.21.15.161:0/2952986080 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8038040 con 0x7f2ae0152890 2024-03-28T23:30:14.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.659+0000 7f2ae6f8c640 1 -- 172.21.15.161:0/2952986080 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ae01ead00 con 0x7f2ae0152890 2024-03-28T23:30:14.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.659+0000 7f2ae6f8c640 1 -- 172.21.15.161:0/2952986080 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ae01eb2c0 con 0x7f2ae0152890 2024-03-28T23:30:14.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.660+0000 7f2ae6f8c640 1 -- 172.21.15.161:0/2952986080 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ae00ac440 con 0x7f2ae0152890 2024-03-28T23:30:14.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.661+0000 7f2ad57fa640 1 -- 172.21.15.161:0/2952986080 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2ad8045020 con 0x7f2ae0152890 2024-03-28T23:30:14.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.661+0000 7f2ad57fa640 1 --2- 172.21.15.161:0/2952986080 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2acc06b4d0 0x7f2acc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:14.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.661+0000 7f2ae4d28640 1 --2- 172.21.15.161:0/2952986080 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2acc06b4d0 0x7f2acc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:14.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.661+0000 7f2ad57fa640 1 -- 172.21.15.161:0/2952986080 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2ad80abda0 con 0x7f2ae0152890 2024-03-28T23:30:14.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.662+0000 7f2ae4d28640 1 --2- 172.21.15.161:0/2952986080 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2acc06b4d0 0x7f2acc06d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2ae00be800 tx=0x7f2adc007770 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:14.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.666+0000 7f2ad57fa640 1 -- 172.21.15.161:0/2952986080 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2ae00ac440 con 0x7f2ae0152890 2024-03-28T23:30:14.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.841+0000 7f2ae6f8c640 1 -- 172.21.15.161:0/2952986080 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2ae004f4b0 con 0x7f2acc06b4d0 2024-03-28T23:30:14.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.849+0000 7f2ad57fa640 1 -- 172.21.15.161:0/2952986080 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332610 (secure 0 0 0) 0x7f2ae004f4b0 con 0x7f2acc06b4d0 2024-03-28T23:30:14.854 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:14.855 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:30:14.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.852+0000 7f2ae6f8c640 1 -- 172.21.15.161:0/2952986080 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2acc06b4d0 msgr2=0x7f2acc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.852+0000 7f2ae6f8c640 1 --2- 172.21.15.161:0/2952986080 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2acc06b4d0 0x7f2acc06d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2ae00be800 tx=0x7f2adc007770 comp rx=0 tx=0).stop 2024-03-28T23:30:14.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.852+0000 7f2ae6f8c640 1 -- 172.21.15.161:0/2952986080 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae0152890 msgr2=0x7f2ae0153ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:14.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.852+0000 7f2ae6f8c640 1 --2- 172.21.15.161:0/2952986080 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae0152890 0x7f2ae0153ac0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8009890 tx=0x7f2ad80046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.852+0000 7f2ae6f8c640 1 -- 172.21.15.161:0/2952986080 shutdown_connections 2024-03-28T23:30:14.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.852+0000 7f2ae6f8c640 1 --2- 172.21.15.161:0/2952986080 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2acc06b4d0 0x7f2acc06d990 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.852+0000 7f2ae6f8c640 1 --2- 172.21.15.161:0/2952986080 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae0152890 0x7f2ae0153ac0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.852+0000 7f2ae6f8c640 1 --2- 172.21.15.161:0/2952986080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ae00c8ed0 0x7f2ae0153580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.852+0000 7f2ae6f8c640 1 --2- 172.21.15.161:0/2952986080 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ae00c5b20 0x7f2ae0157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:14.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.853+0000 7f2ae6f8c640 1 -- 172.21.15.161:0/2952986080 >> 172.21.15.161:0/2952986080 conn(0x7f2ae00ac620 msgr2=0x7f2ae00c5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:14.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.853+0000 7f2ae6f8c640 1 -- 172.21.15.161:0/2952986080 shutdown_connections 2024-03-28T23:30:14.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:14.853+0000 7f2ae6f8c640 1 -- 172.21.15.161:0/2952986080 wait complete. 2024-03-28T23:30:14.865 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-28T23:30:13.317522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222516,"kb_used_data":2176,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754752204,"statfs":{"total":773094113280,"available":772866256896,"internally_reserved":0,"allocated":2228224,"data_stored":1992064,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002607"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687840+0000","last_change":"2024-03-28T23:29:45.558575+0000","last_active":"2024-03-28T23:29:51.687840+0000","last_peered":"2024-03-28T23:29:51.687840+0000","last_clean":"2024-03-28T23:29:51.687840+0000","last_became_active":"2024-03-28T23:29:45.557910+0000","last_became_peered":"2024-03-28T23:29:45.557910+0000","last_unstale":"2024-03-28T23:29:51.687840+0000","last_undegraded":"2024-03-28T23:29:51.687840+0000","last_fullsized":"2024-03-28T23:29:51.687840+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:59.719252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686777+0000","last_change":"2024-03-28T23:29:46.557562+0000","last_active":"2024-03-28T23:29:51.686777+0000","last_peered":"2024-03-28T23:29:51.686777+0000","last_clean":"2024-03-28T23:29:51.686777+0000","last_became_active":"2024-03-28T23:29:46.557265+0000","last_became_peered":"2024-03-28T23:29:46.557265+0000","last_unstale":"2024-03-28T23:29:51.686777+0000","last_undegraded":"2024-03-28T23:29:51.686777+0000","last_fullsized":"2024-03-28T23:29:51.686777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:06.731262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686496+0000","last_change":"2024-03-28T23:29:50.707752+0000","last_active":"2024-03-28T23:29:51.686496+0000","last_peered":"2024-03-28T23:29:51.686496+0000","last_clean":"2024-03-28T23:29:51.686496+0000","last_became_active":"2024-03-28T23:29:50.707364+0000","last_became_peered":"2024-03-28T23:29:50.707364+0000","last_unstale":"2024-03-28T23:29:51.686496+0000","last_undegraded":"2024-03-28T23:29:51.686496+0000","last_fullsized":"2024-03-28T23:29:51.686496+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:58:38.680360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686927+0000","last_change":"2024-03-28T23:29:45.558356+0000","last_active":"2024-03-28T23:29:51.686927+0000","last_peered":"2024-03-28T23:29:51.686927+0000","last_clean":"2024-03-28T23:29:51.686927+0000","last_became_active":"2024-03-28T23:29:45.558070+0000","last_became_peered":"2024-03-28T23:29:45.558070+0000","last_unstale":"2024-03-28T23:29:51.686927+0000","last_undegraded":"2024-03-28T23:29:51.686927+0000","last_fullsized":"2024-03-28T23:29:51.686927+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:59.719252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686591+0000","last_change":"2024-03-28T23:29:46.579153+0000","last_active":"2024-03-28T23:29:51.686591+0000","last_peered":"2024-03-28T23:29:51.686591+0000","last_clean":"2024-03-28T23:29:51.686591+0000","last_became_active":"2024-03-28T23:29:46.578341+0000","last_became_peered":"2024-03-28T23:29:46.578341+0000","last_unstale":"2024-03-28T23:29:51.686591+0000","last_undegraded":"2024-03-28T23:29:51.686591+0000","last_fullsized":"2024-03-28T23:29:51.686591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:38.074388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591979+0000","last_change":"2024-03-28T23:29:50.597011+0000","last_active":"2024-03-28T23:29:51.591979+0000","last_peered":"2024-03-28T23:29:51.591979+0000","last_clean":"2024-03-28T23:29:51.591979+0000","last_became_active":"2024-03-28T23:29:50.596239+0000","last_became_peered":"2024-03-28T23:29:50.596239+0000","last_unstale":"2024-03-28T23:29:51.591979+0000","last_undegraded":"2024-03-28T23:29:51.591979+0000","last_fullsized":"2024-03-28T23:29:51.591979+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:29:24.703997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686674+0000","last_change":"2024-03-28T23:29:45.562460+0000","last_active":"2024-03-28T23:29:51.686674+0000","last_peered":"2024-03-28T23:29:51.686674+0000","last_clean":"2024-03-28T23:29:51.686674+0000","last_became_active":"2024-03-28T23:29:45.562101+0000","last_became_peered":"2024-03-28T23:29:45.562101+0000","last_unstale":"2024-03-28T23:29:51.686674+0000","last_undegraded":"2024-03-28T23:29:51.686674+0000","last_fullsized":"2024-03-28T23:29:51.686674+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:42:08.707703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685709+0000","last_change":"2024-03-28T23:29:46.567420+0000","last_active":"2024-03-28T23:29:51.685709+0000","last_peered":"2024-03-28T23:29:51.685709+0000","last_clean":"2024-03-28T23:29:51.685709+0000","last_became_active":"2024-03-28T23:29:46.567157+0000","last_became_peered":"2024-03-28T23:29:46.567157+0000","last_unstale":"2024-03-28T23:29:51.685709+0000","last_undegraded":"2024-03-28T23:29:51.685709+0000","last_fullsized":"2024-03-28T23:29:51.685709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:54.470202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.592139+0000","last_change":"2024-03-28T23:29:50.599020+0000","last_active":"2024-03-28T23:29:51.592139+0000","last_peered":"2024-03-28T23:29:51.592139+0000","last_clean":"2024-03-28T23:29:51.592139+0000","last_became_active":"2024-03-28T23:29:50.598588+0000","last_became_peered":"2024-03-28T23:29:50.598588+0000","last_unstale":"2024-03-28T23:29:51.592139+0000","last_undegraded":"2024-03-28T23:29:51.592139+0000","last_fullsized":"2024-03-28T23:29:51.592139+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:06.894069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686561+0000","last_change":"2024-03-28T23:29:45.564688+0000","last_active":"2024-03-28T23:29:51.686561+0000","last_peered":"2024-03-28T23:29:51.686561+0000","last_clean":"2024-03-28T23:29:51.686561+0000","last_became_active":"2024-03-28T23:29:45.564381+0000","last_became_peered":"2024-03-28T23:29:45.564381+0000","last_unstale":"2024-03-28T23:29:51.686561+0000","last_undegraded":"2024-03-28T23:29:51.686561+0000","last_fullsized":"2024-03-28T23:29:51.686561+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:36.113647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686025+0000","last_change":"2024-03-28T23:29:46.566644+0000","last_active":"2024-03-28T23:29:51.686025+0000","last_peered":"2024-03-28T23:29:51.686025+0000","last_clean":"2024-03-28T23:29:51.686025+0000","last_became_active":"2024-03-28T23:29:46.566340+0000","last_became_peered":"2024-03-28T23:29:46.566340+0000","last_unstale":"2024-03-28T23:29:51.686025+0000","last_undegraded":"2024-03-28T23:29:51.686025+0000","last_fullsized":"2024-03-28T23:29:51.686025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:02.859502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591969+0000","last_change":"2024-03-28T23:29:50.597019+0000","last_active":"2024-03-28T23:29:51.591969+0000","last_peered":"2024-03-28T23:29:51.591969+0000","last_clean":"2024-03-28T23:29:51.591969+0000","last_became_active":"2024-03-28T23:29:50.596284+0000","last_became_peered":"2024-03-28T23:29:50.596284+0000","last_unstale":"2024-03-28T23:29:51.591969+0000","last_undegraded":"2024-03-28T23:29:51.591969+0000","last_fullsized":"2024-03-28T23:29:51.591969+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:08:37.836411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.590591+0000","last_change":"2024-03-28T23:29:46.565325+0000","last_active":"2024-03-28T23:29:51.590591+0000","last_peered":"2024-03-28T23:29:51.590591+0000","last_clean":"2024-03-28T23:29:51.590591+0000","last_became_active":"2024-03-28T23:29:46.564195+0000","last_became_peered":"2024-03-28T23:29:46.564195+0000","last_unstale":"2024-03-28T23:29:51.590591+0000","last_undegraded":"2024-03-28T23:29:51.590591+0000","last_fullsized":"2024-03-28T23:29:51.590591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:32.936386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685919+0000","last_change":"2024-03-28T23:29:45.560329+0000","last_active":"2024-03-28T23:29:51.685919+0000","last_peered":"2024-03-28T23:29:51.685919+0000","last_clean":"2024-03-28T23:29:51.685919+0000","last_became_active":"2024-03-28T23:29:45.560073+0000","last_became_peered":"2024-03-28T23:29:45.560073+0000","last_unstale":"2024-03-28T23:29:51.685919+0000","last_undegraded":"2024-03-28T23:29:51.685919+0000","last_fullsized":"2024-03-28T23:29:51.685919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:09.929984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591983+0000","last_change":"2024-03-28T23:29:50.592083+0000","last_active":"2024-03-28T23:29:51.591983+0000","last_peered":"2024-03-28T23:29:51.591983+0000","last_clean":"2024-03-28T23:29:51.591983+0000","last_became_active":"2024-03-28T23:29:50.591626+0000","last_became_peered":"2024-03-28T23:29:50.591626+0000","last_unstale":"2024-03-28T23:29:51.591983+0000","last_undegraded":"2024-03-28T23:29:51.591983+0000","last_fullsized":"2024-03-28T23:29:51.591983+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:17:54.013778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685967+0000","last_change":"2024-03-28T23:29:46.581125+0000","last_active":"2024-03-28T23:29:51.685967+0000","last_peered":"2024-03-28T23:29:51.685967+0000","last_clean":"2024-03-28T23:29:51.685967+0000","last_became_active":"2024-03-28T23:29:46.580827+0000","last_became_peered":"2024-03-28T23:29:46.580827+0000","last_unstale":"2024-03-28T23:29:51.685967+0000","last_undegraded":"2024-03-28T23:29:51.685967+0000","last_fullsized":"2024-03-28T23:29:51.685967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:10.853710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686404+0000","last_change":"2024-03-28T23:29:45.562525+0000","last_active":"2024-03-28T23:29:51.686404+0000","last_peered":"2024-03-28T23:29:51.686404+0000","last_clean":"2024-03-28T23:29:51.686404+0000","last_became_active":"2024-03-28T23:29:45.561757+0000","last_became_peered":"2024-03-28T23:29:45.561757+0000","last_unstale":"2024-03-28T23:29:51.686404+0000","last_undegraded":"2024-03-28T23:29:51.686404+0000","last_fullsized":"2024-03-28T23:29:51.686404+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:24:58.784056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591567+0000","last_change":"2024-03-28T23:29:50.599617+0000","last_active":"2024-03-28T23:29:51.591567+0000","last_peered":"2024-03-28T23:29:51.591567+0000","last_clean":"2024-03-28T23:29:51.591567+0000","last_became_active":"2024-03-28T23:29:50.599031+0000","last_became_peered":"2024-03-28T23:29:50.599031+0000","last_unstale":"2024-03-28T23:29:51.591567+0000","last_undegraded":"2024-03-28T23:29:51.591567+0000","last_fullsized":"2024-03-28T23:29:51.591567+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:15:03.216756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686223+0000","last_change":"2024-03-28T23:29:46.555057+0000","last_active":"2024-03-28T23:29:51.686223+0000","last_peered":"2024-03-28T23:29:51.686223+0000","last_clean":"2024-03-28T23:29:51.686223+0000","last_became_active":"2024-03-28T23:29:46.554796+0000","last_became_peered":"2024-03-28T23:29:46.554796+0000","last_unstale":"2024-03-28T23:29:51.686223+0000","last_undegraded":"2024-03-28T23:29:51.686223+0000","last_fullsized":"2024-03-28T23:29:51.686223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:36:20.485861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685609+0000","last_change":"2024-03-28T23:29:45.561273+0000","last_active":"2024-03-28T23:29:51.685609+0000","last_peered":"2024-03-28T23:29:51.685609+0000","last_clean":"2024-03-28T23:29:51.685609+0000","last_became_active":"2024-03-28T23:29:45.561006+0000","last_became_peered":"2024-03-28T23:29:45.561006+0000","last_unstale":"2024-03-28T23:29:51.685609+0000","last_undegraded":"2024-03-28T23:29:51.685609+0000","last_fullsized":"2024-03-28T23:29:51.685609+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:41.621188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687569+0000","last_change":"2024-03-28T23:29:50.706824+0000","last_active":"2024-03-28T23:29:51.687569+0000","last_peered":"2024-03-28T23:29:51.687569+0000","last_clean":"2024-03-28T23:29:51.687569+0000","last_became_active":"2024-03-28T23:29:50.706411+0000","last_became_peered":"2024-03-28T23:29:50.706411+0000","last_unstale":"2024-03-28T23:29:51.687569+0000","last_undegraded":"2024-03-28T23:29:51.687569+0000","last_fullsized":"2024-03-28T23:29:51.687569+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:43:14.415039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.592115+0000","last_change":"2024-03-28T23:29:46.558900+0000","last_active":"2024-03-28T23:29:51.592115+0000","last_peered":"2024-03-28T23:29:51.592115+0000","last_clean":"2024-03-28T23:29:51.592115+0000","last_became_active":"2024-03-28T23:29:46.558628+0000","last_became_peered":"2024-03-28T23:29:46.558628+0000","last_unstale":"2024-03-28T23:29:51.592115+0000","last_undegraded":"2024-03-28T23:29:51.592115+0000","last_fullsized":"2024-03-28T23:29:51.592115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:37.122394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685397+0000","last_change":"2024-03-28T23:29:45.563365+0000","last_active":"2024-03-28T23:29:51.685397+0000","last_peered":"2024-03-28T23:29:51.685397+0000","last_clean":"2024-03-28T23:29:51.685397+0000","last_became_active":"2024-03-28T23:29:45.563093+0000","last_became_peered":"2024-03-28T23:29:45.563093+0000","last_unstale":"2024-03-28T23:29:51.685397+0000","last_undegraded":"2024-03-28T23:29:51.685397+0000","last_fullsized":"2024-03-28T23:29:51.685397+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:01.983721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591200+0000","last_change":"2024-03-28T23:29:50.596634+0000","last_active":"2024-03-28T23:29:51.591200+0000","last_peered":"2024-03-28T23:29:51.591200+0000","last_clean":"2024-03-28T23:29:51.591200+0000","last_became_active":"2024-03-28T23:29:50.596057+0000","last_became_peered":"2024-03-28T23:29:50.596057+0000","last_unstale":"2024-03-28T23:29:51.591200+0000","last_undegraded":"2024-03-28T23:29:51.591200+0000","last_fullsized":"2024-03-28T23:29:51.591200+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:02.708894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687515+0000","last_change":"2024-03-28T23:29:46.578357+0000","last_active":"2024-03-28T23:29:51.687515+0000","last_peered":"2024-03-28T23:29:51.687515+0000","last_clean":"2024-03-28T23:29:51.687515+0000","last_became_active":"2024-03-28T23:29:46.578075+0000","last_became_peered":"2024-03-28T23:29:46.578075+0000","last_unstale":"2024-03-28T23:29:51.687515+0000","last_undegraded":"2024-03-28T23:29:51.687515+0000","last_fullsized":"2024-03-28T23:29:51.687515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:40:22.556997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591419+0000","last_change":"2024-03-28T23:29:45.560606+0000","last_active":"2024-03-28T23:29:51.591419+0000","last_peered":"2024-03-28T23:29:51.591419+0000","last_clean":"2024-03-28T23:29:51.591419+0000","last_became_active":"2024-03-28T23:29:45.560355+0000","last_became_peered":"2024-03-28T23:29:45.560355+0000","last_unstale":"2024-03-28T23:29:51.591419+0000","last_undegraded":"2024-03-28T23:29:51.591419+0000","last_fullsized":"2024-03-28T23:29:51.591419+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:01.983721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591966+0000","last_change":"2024-03-28T23:29:50.708102+0000","last_active":"2024-03-28T23:29:51.591966+0000","last_peered":"2024-03-28T23:29:51.591966+0000","last_clean":"2024-03-28T23:29:51.591966+0000","last_became_active":"2024-03-28T23:29:50.707657+0000","last_became_peered":"2024-03-28T23:29:50.707657+0000","last_unstale":"2024-03-28T23:29:51.591966+0000","last_undegraded":"2024-03-28T23:29:51.591966+0000","last_fullsized":"2024-03-28T23:29:51.591966+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:58.504770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687431+0000","last_change":"2024-03-28T23:29:46.561641+0000","last_active":"2024-03-28T23:29:51.687431+0000","last_peered":"2024-03-28T23:29:51.687431+0000","last_clean":"2024-03-28T23:29:51.687431+0000","last_became_active":"2024-03-28T23:29:46.561380+0000","last_became_peered":"2024-03-28T23:29:46.561380+0000","last_unstale":"2024-03-28T23:29:51.687431+0000","last_undegraded":"2024-03-28T23:29:51.687431+0000","last_fullsized":"2024-03-28T23:29:51.687431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:54.470202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686791+0000","last_change":"2024-03-28T23:29:45.562177+0000","last_active":"2024-03-28T23:29:51.686791+0000","last_peered":"2024-03-28T23:29:51.686791+0000","last_clean":"2024-03-28T23:29:51.686791+0000","last_became_active":"2024-03-28T23:29:45.561852+0000","last_became_peered":"2024-03-28T23:29:45.561852+0000","last_unstale":"2024-03-28T23:29:51.686791+0000","last_undegraded":"2024-03-28T23:29:51.686791+0000","last_fullsized":"2024-03-28T23:29:51.686791+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:36.113647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685836+0000","last_change":"2024-03-28T23:29:50.706769+0000","last_active":"2024-03-28T23:29:51.685836+0000","last_peered":"2024-03-28T23:29:51.685836+0000","last_clean":"2024-03-28T23:29:51.685836+0000","last_became_active":"2024-03-28T23:29:50.706415+0000","last_became_peered":"2024-03-28T23:29:50.706415+0000","last_unstale":"2024-03-28T23:29:51.685836+0000","last_undegraded":"2024-03-28T23:29:51.685836+0000","last_fullsized":"2024-03-28T23:29:51.685836+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:47.094789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687279+0000","last_change":"2024-03-28T23:29:46.560067+0000","last_active":"2024-03-28T23:29:51.687279+0000","last_peered":"2024-03-28T23:29:51.687279+0000","last_clean":"2024-03-28T23:29:51.687279+0000","last_became_active":"2024-03-28T23:29:46.559790+0000","last_became_peered":"2024-03-28T23:29:46.559790+0000","last_unstale":"2024-03-28T23:29:51.687279+0000","last_undegraded":"2024-03-28T23:29:51.687279+0000","last_fullsized":"2024-03-28T23:29:51.687279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:32.936386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686796+0000","last_change":"2024-03-28T23:29:45.565279+0000","last_active":"2024-03-28T23:29:51.686796+0000","last_peered":"2024-03-28T23:29:51.686796+0000","last_clean":"2024-03-28T23:29:51.686796+0000","last_became_active":"2024-03-28T23:29:45.564739+0000","last_became_peered":"2024-03-28T23:29:45.564739+0000","last_unstale":"2024-03-28T23:29:51.686796+0000","last_undegraded":"2024-03-28T23:29:51.686796+0000","last_fullsized":"2024-03-28T23:29:51.686796+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:17.783972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686745+0000","last_change":"2024-03-28T23:29:50.706805+0000","last_active":"2024-03-28T23:29:51.686745+0000","last_peered":"2024-03-28T23:29:51.686745+0000","last_clean":"2024-03-28T23:29:51.686745+0000","last_became_active":"2024-03-28T23:29:50.706429+0000","last_became_peered":"2024-03-28T23:29:50.706429+0000","last_unstale":"2024-03-28T23:29:51.686745+0000","last_undegraded":"2024-03-28T23:29:51.686745+0000","last_fullsized":"2024-03-28T23:29:51.686745+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:06:40.917529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686157+0000","last_change":"2024-03-28T23:29:46.560912+0000","last_active":"2024-03-28T23:29:51.686157+0000","last_peered":"2024-03-28T23:29:51.686157+0000","last_clean":"2024-03-28T23:29:51.686157+0000","last_became_active":"2024-03-28T23:29:46.560613+0000","last_became_peered":"2024-03-28T23:29:46.560613+0000","last_unstale":"2024-03-28T23:29:51.686157+0000","last_undegraded":"2024-03-28T23:29:51.686157+0000","last_fullsized":"2024-03-28T23:29:51.686157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:17:49.979210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687372+0000","last_change":"2024-03-28T23:29:45.561918+0000","last_active":"2024-03-28T23:29:51.687372+0000","last_peered":"2024-03-28T23:29:51.687372+0000","last_clean":"2024-03-28T23:29:51.687372+0000","last_became_active":"2024-03-28T23:29:45.561577+0000","last_became_peered":"2024-03-28T23:29:45.561577+0000","last_unstale":"2024-03-28T23:29:51.687372+0000","last_undegraded":"2024-03-28T23:29:51.687372+0000","last_fullsized":"2024-03-28T23:29:51.687372+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:09.929984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686073+0000","last_change":"2024-03-28T23:29:50.594835+0000","last_active":"2024-03-28T23:29:51.686073+0000","last_peered":"2024-03-28T23:29:51.686073+0000","last_clean":"2024-03-28T23:29:51.686073+0000","last_became_active":"2024-03-28T23:29:50.594167+0000","last_became_peered":"2024-03-28T23:29:50.594167+0000","last_unstale":"2024-03-28T23:29:51.686073+0000","last_undegraded":"2024-03-28T23:29:51.686073+0000","last_fullsized":"2024-03-28T23:29:51.686073+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:06:40.917529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.592269+0000","last_change":"2024-03-28T23:29:46.579930+0000","last_active":"2024-03-28T23:29:51.592269+0000","last_peered":"2024-03-28T23:29:51.592269+0000","last_clean":"2024-03-28T23:29:51.592269+0000","last_became_active":"2024-03-28T23:29:46.579622+0000","last_became_peered":"2024-03-28T23:29:46.579622+0000","last_unstale":"2024-03-28T23:29:51.592269+0000","last_undegraded":"2024-03-28T23:29:51.592269+0000","last_fullsized":"2024-03-28T23:29:51.592269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:18.792719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686219+0000","last_change":"2024-03-28T23:29:45.557802+0000","last_active":"2024-03-28T23:29:51.686219+0000","last_peered":"2024-03-28T23:29:51.686219+0000","last_clean":"2024-03-28T23:29:51.686219+0000","last_became_active":"2024-03-28T23:29:45.557552+0000","last_became_peered":"2024-03-28T23:29:45.557552+0000","last_unstale":"2024-03-28T23:29:51.686219+0000","last_undegraded":"2024-03-28T23:29:51.686219+0000","last_fullsized":"2024-03-28T23:29:51.686219+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:01.983721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687230+0000","last_change":"2024-03-28T23:29:50.601086+0000","last_active":"2024-03-28T23:29:51.687230+0000","last_peered":"2024-03-28T23:29:51.687230+0000","last_clean":"2024-03-28T23:29:51.687230+0000","last_became_active":"2024-03-28T23:29:50.600720+0000","last_became_peered":"2024-03-28T23:29:50.600720+0000","last_unstale":"2024-03-28T23:29:51.687230+0000","last_undegraded":"2024-03-28T23:29:51.687230+0000","last_fullsized":"2024-03-28T23:29:51.687230+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:30:48.379529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685445+0000","last_change":"2024-03-28T23:29:46.560331+0000","last_active":"2024-03-28T23:29:51.685445+0000","last_peered":"2024-03-28T23:29:51.685445+0000","last_clean":"2024-03-28T23:29:51.685445+0000","last_became_active":"2024-03-28T23:29:46.559135+0000","last_became_peered":"2024-03-28T23:29:46.559135+0000","last_unstale":"2024-03-28T23:29:51.685445+0000","last_undegraded":"2024-03-28T23:29:51.685445+0000","last_fullsized":"2024-03-28T23:29:51.685445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:58.674327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589305+0000","last_change":"2024-03-28T23:29:45.559376+0000","last_active":"2024-03-28T23:29:51.589305+0000","last_peered":"2024-03-28T23:29:51.589305+0000","last_clean":"2024-03-28T23:29:51.589305+0000","last_became_active":"2024-03-28T23:29:45.559076+0000","last_became_peered":"2024-03-28T23:29:45.559076+0000","last_unstale":"2024-03-28T23:29:51.589305+0000","last_undegraded":"2024-03-28T23:29:51.589305+0000","last_fullsized":"2024-03-28T23:29:51.589305+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:21.707459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687612+0000","last_change":"2024-03-28T23:29:50.600059+0000","last_active":"2024-03-28T23:29:51.687612+0000","last_peered":"2024-03-28T23:29:51.687612+0000","last_clean":"2024-03-28T23:29:51.687612+0000","last_became_active":"2024-03-28T23:29:50.599692+0000","last_became_peered":"2024-03-28T23:29:50.599692+0000","last_unstale":"2024-03-28T23:29:51.687612+0000","last_undegraded":"2024-03-28T23:29:51.687612+0000","last_fullsized":"2024-03-28T23:29:51.687612+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:22.085225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686352+0000","last_change":"2024-03-28T23:29:46.578521+0000","last_active":"2024-03-28T23:29:51.686352+0000","last_peered":"2024-03-28T23:29:51.686352+0000","last_clean":"2024-03-28T23:29:51.686352+0000","last_became_active":"2024-03-28T23:29:46.578076+0000","last_became_peered":"2024-03-28T23:29:46.578076+0000","last_unstale":"2024-03-28T23:29:51.686352+0000","last_undegraded":"2024-03-28T23:29:51.686352+0000","last_fullsized":"2024-03-28T23:29:51.686352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:20.039954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685395+0000","last_change":"2024-03-28T23:29:45.563206+0000","last_active":"2024-03-28T23:29:51.685395+0000","last_peered":"2024-03-28T23:29:51.685395+0000","last_clean":"2024-03-28T23:29:51.685395+0000","last_became_active":"2024-03-28T23:29:45.562929+0000","last_became_peered":"2024-03-28T23:29:45.562929+0000","last_unstale":"2024-03-28T23:29:51.685395+0000","last_undegraded":"2024-03-28T23:29:51.685395+0000","last_fullsized":"2024-03-28T23:29:51.685395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:41.621188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687463+0000","last_change":"2024-03-28T23:29:50.709169+0000","last_active":"2024-03-28T23:29:51.687463+0000","last_peered":"2024-03-28T23:29:51.687463+0000","last_clean":"2024-03-28T23:29:51.687463+0000","last_became_active":"2024-03-28T23:29:50.707306+0000","last_became_peered":"2024-03-28T23:29:50.707306+0000","last_unstale":"2024-03-28T23:29:51.687463+0000","last_undegraded":"2024-03-28T23:29:51.687463+0000","last_fullsized":"2024-03-28T23:29:51.687463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:22.085225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685675+0000","last_change":"2024-03-28T23:29:46.564922+0000","last_active":"2024-03-28T23:29:51.685675+0000","last_peered":"2024-03-28T23:29:51.685675+0000","last_clean":"2024-03-28T23:29:51.685675+0000","last_became_active":"2024-03-28T23:29:46.564644+0000","last_became_peered":"2024-03-28T23:29:46.564644+0000","last_unstale":"2024-03-28T23:29:51.685675+0000","last_undegraded":"2024-03-28T23:29:51.685675+0000","last_fullsized":"2024-03-28T23:29:51.685675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:58.674327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686316+0000","last_change":"2024-03-28T23:29:45.556656+0000","last_active":"2024-03-28T23:29:51.686316+0000","last_peered":"2024-03-28T23:29:51.686316+0000","last_clean":"2024-03-28T23:29:51.686316+0000","last_became_active":"2024-03-28T23:29:45.556329+0000","last_became_peered":"2024-03-28T23:29:45.556329+0000","last_unstale":"2024-03-28T23:29:51.686316+0000","last_undegraded":"2024-03-28T23:29:51.686316+0000","last_fullsized":"2024-03-28T23:29:51.686316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:21.707459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686403+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:29:51.686403+0000","last_peered":"2024-03-28T23:29:51.686403+0000","last_clean":"2024-03-28T23:29:51.686403+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:29:51.686403+0000","last_undegraded":"2024-03-28T23:29:51.686403+0000","last_fullsized":"2024-03-28T23:29:51.686403+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591319+0000","last_change":"2024-03-28T23:29:50.711182+0000","last_active":"2024-03-28T23:29:51.591319+0000","last_peered":"2024-03-28T23:29:51.591319+0000","last_clean":"2024-03-28T23:29:51.591319+0000","last_became_active":"2024-03-28T23:29:50.710737+0000","last_became_peered":"2024-03-28T23:29:50.710737+0000","last_unstale":"2024-03-28T23:29:51.591319+0000","last_undegraded":"2024-03-28T23:29:51.591319+0000","last_fullsized":"2024-03-28T23:29:51.591319+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:17:54.013778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686573+0000","last_change":"2024-03-28T23:29:46.572966+0000","last_active":"2024-03-28T23:29:51.686573+0000","last_peered":"2024-03-28T23:29:51.686573+0000","last_clean":"2024-03-28T23:29:51.686573+0000","last_became_active":"2024-03-28T23:29:46.572405+0000","last_became_peered":"2024-03-28T23:29:46.572405+0000","last_unstale":"2024-03-28T23:29:51.686573+0000","last_undegraded":"2024-03-28T23:29:51.686573+0000","last_fullsized":"2024-03-28T23:29:51.686573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:06:36.882961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589560+0000","last_change":"2024-03-28T23:29:45.560681+0000","last_active":"2024-03-28T23:29:51.589560+0000","last_peered":"2024-03-28T23:29:51.589560+0000","last_clean":"2024-03-28T23:29:51.589560+0000","last_became_active":"2024-03-28T23:29:45.560420+0000","last_became_peered":"2024-03-28T23:29:45.560420+0000","last_unstale":"2024-03-28T23:29:51.589560+0000","last_undegraded":"2024-03-28T23:29:51.589560+0000","last_fullsized":"2024-03-28T23:29:51.589560+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:17:17.207826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.590530+0000","last_change":"2024-03-28T23:29:50.711247+0000","last_active":"2024-03-28T23:29:51.590530+0000","last_peered":"2024-03-28T23:29:51.590530+0000","last_clean":"2024-03-28T23:29:51.590530+0000","last_became_active":"2024-03-28T23:29:50.710836+0000","last_became_peered":"2024-03-28T23:29:50.710836+0000","last_unstale":"2024-03-28T23:29:51.590530+0000","last_undegraded":"2024-03-28T23:29:51.590530+0000","last_fullsized":"2024-03-28T23:29:51.590530+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:58.504770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685843+0000","last_change":"2024-03-28T23:29:46.561283+0000","last_active":"2024-03-28T23:29:51.685843+0000","last_peered":"2024-03-28T23:29:51.685843+0000","last_clean":"2024-03-28T23:29:51.685843+0000","last_became_active":"2024-03-28T23:29:46.560584+0000","last_became_peered":"2024-03-28T23:29:46.560584+0000","last_unstale":"2024-03-28T23:29:51.685843+0000","last_undegraded":"2024-03-28T23:29:51.685843+0000","last_fullsized":"2024-03-28T23:29:51.685843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:06:36.882961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686047+0000","last_change":"2024-03-28T23:29:45.560013+0000","last_active":"2024-03-28T23:29:51.686047+0000","last_peered":"2024-03-28T23:29:51.686047+0000","last_clean":"2024-03-28T23:29:51.686047+0000","last_became_active":"2024-03-28T23:29:45.559643+0000","last_became_peered":"2024-03-28T23:29:45.559643+0000","last_unstale":"2024-03-28T23:29:51.686047+0000","last_undegraded":"2024-03-28T23:29:51.686047+0000","last_fullsized":"2024-03-28T23:29:51.686047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:01.983721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687005+0000","last_change":"2024-03-28T23:29:50.595120+0000","last_active":"2024-03-28T23:29:51.687005+0000","last_peered":"2024-03-28T23:29:51.687005+0000","last_clean":"2024-03-28T23:29:51.687005+0000","last_became_active":"2024-03-28T23:29:50.594780+0000","last_became_peered":"2024-03-28T23:29:50.594780+0000","last_unstale":"2024-03-28T23:29:51.687005+0000","last_undegraded":"2024-03-28T23:29:51.687005+0000","last_fullsized":"2024-03-28T23:29:51.687005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:06.894069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686843+0000","last_change":"2024-03-28T23:29:46.567707+0000","last_active":"2024-03-28T23:29:51.686843+0000","last_peered":"2024-03-28T23:29:51.686843+0000","last_clean":"2024-03-28T23:29:51.686843+0000","last_became_active":"2024-03-28T23:29:46.565675+0000","last_became_peered":"2024-03-28T23:29:46.565675+0000","last_unstale":"2024-03-28T23:29:51.686843+0000","last_undegraded":"2024-03-28T23:29:51.686843+0000","last_fullsized":"2024-03-28T23:29:51.686843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:07:21.524344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589416+0000","last_change":"2024-03-28T23:29:45.558538+0000","last_active":"2024-03-28T23:29:51.589416+0000","last_peered":"2024-03-28T23:29:51.589416+0000","last_clean":"2024-03-28T23:29:51.589416+0000","last_became_active":"2024-03-28T23:29:45.558248+0000","last_became_peered":"2024-03-28T23:29:45.558248+0000","last_unstale":"2024-03-28T23:29:51.589416+0000","last_undegraded":"2024-03-28T23:29:51.589416+0000","last_fullsized":"2024-03-28T23:29:51.589416+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:59.719252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685418+0000","last_change":"2024-03-28T23:29:50.707640+0000","last_active":"2024-03-28T23:29:51.685418+0000","last_peered":"2024-03-28T23:29:51.685418+0000","last_clean":"2024-03-28T23:29:51.685418+0000","last_became_active":"2024-03-28T23:29:50.707258+0000","last_became_peered":"2024-03-28T23:29:50.707258+0000","last_unstale":"2024-03-28T23:29:51.685418+0000","last_undegraded":"2024-03-28T23:29:51.685418+0000","last_fullsized":"2024-03-28T23:29:51.685418+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:47.094789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686505+0000","last_change":"2024-03-28T23:29:46.565431+0000","last_active":"2024-03-28T23:29:51.686505+0000","last_peered":"2024-03-28T23:29:51.686505+0000","last_clean":"2024-03-28T23:29:51.686505+0000","last_became_active":"2024-03-28T23:29:46.565163+0000","last_became_peered":"2024-03-28T23:29:46.565163+0000","last_unstale":"2024-03-28T23:29:51.686505+0000","last_undegraded":"2024-03-28T23:29:51.686505+0000","last_fullsized":"2024-03-28T23:29:51.686505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:02.859502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591420+0000","last_change":"2024-03-28T23:29:45.564138+0000","last_active":"2024-03-28T23:29:51.591420+0000","last_peered":"2024-03-28T23:29:51.591420+0000","last_clean":"2024-03-28T23:29:51.591420+0000","last_became_active":"2024-03-28T23:29:45.563858+0000","last_became_peered":"2024-03-28T23:29:45.563858+0000","last_unstale":"2024-03-28T23:29:51.591420+0000","last_undegraded":"2024-03-28T23:29:51.591420+0000","last_fullsized":"2024-03-28T23:29:51.591420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:42:38.893826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.592126+0000","last_change":"2024-03-28T23:29:50.599484+0000","last_active":"2024-03-28T23:29:51.592126+0000","last_peered":"2024-03-28T23:29:51.592126+0000","last_clean":"2024-03-28T23:29:51.592126+0000","last_became_active":"2024-03-28T23:29:50.599035+0000","last_became_peered":"2024-03-28T23:29:50.599035+0000","last_unstale":"2024-03-28T23:29:51.592126+0000","last_undegraded":"2024-03-28T23:29:51.592126+0000","last_fullsized":"2024-03-28T23:29:51.592126+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:10.765830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686919+0000","last_change":"2024-03-28T23:29:46.567472+0000","last_active":"2024-03-28T23:29:51.686919+0000","last_peered":"2024-03-28T23:29:51.686919+0000","last_clean":"2024-03-28T23:29:51.686919+0000","last_became_active":"2024-03-28T23:29:46.565485+0000","last_became_peered":"2024-03-28T23:29:46.565485+0000","last_unstale":"2024-03-28T23:29:51.686919+0000","last_undegraded":"2024-03-28T23:29:51.686919+0000","last_fullsized":"2024-03-28T23:29:51.686919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:29:20.669429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589103+0000","last_change":"2024-03-28T23:29:45.556004+0000","last_active":"2024-03-28T23:29:51.589103+0000","last_peered":"2024-03-28T23:29:51.589103+0000","last_clean":"2024-03-28T23:29:51.589103+0000","last_became_active":"2024-03-28T23:29:45.555701+0000","last_became_peered":"2024-03-28T23:29:45.555701+0000","last_unstale":"2024-03-28T23:29:51.589103+0000","last_undegraded":"2024-03-28T23:29:51.589103+0000","last_fullsized":"2024-03-28T23:29:51.589103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:06:18.025605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687430+0000","last_change":"2024-03-28T23:29:50.596439+0000","last_active":"2024-03-28T23:29:51.687430+0000","last_peered":"2024-03-28T23:29:51.687430+0000","last_clean":"2024-03-28T23:29:51.687430+0000","last_became_active":"2024-03-28T23:29:50.596065+0000","last_became_peered":"2024-03-28T23:29:50.596065+0000","last_unstale":"2024-03-28T23:29:51.687430+0000","last_undegraded":"2024-03-28T23:29:51.687430+0000","last_fullsized":"2024-03-28T23:29:51.687430+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:43:14.415039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.590719+0000","last_change":"2024-03-28T23:29:46.581708+0000","last_active":"2024-03-28T23:29:51.590719+0000","last_peered":"2024-03-28T23:29:51.590719+0000","last_clean":"2024-03-28T23:29:51.590719+0000","last_became_active":"2024-03-28T23:29:46.581391+0000","last_became_peered":"2024-03-28T23:29:46.581391+0000","last_unstale":"2024-03-28T23:29:51.590719+0000","last_undegraded":"2024-03-28T23:29:51.590719+0000","last_fullsized":"2024-03-28T23:29:51.590719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:24:19.353047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591187+0000","last_change":"2024-03-28T23:29:45.560112+0000","last_active":"2024-03-28T23:29:51.591187+0000","last_peered":"2024-03-28T23:29:51.591187+0000","last_clean":"2024-03-28T23:29:51.591187+0000","last_became_active":"2024-03-28T23:29:45.559817+0000","last_became_peered":"2024-03-28T23:29:45.559817+0000","last_unstale":"2024-03-28T23:29:51.591187+0000","last_undegraded":"2024-03-28T23:29:51.591187+0000","last_fullsized":"2024-03-28T23:29:51.591187+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:21.707459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687297+0000","last_change":"2024-03-28T23:29:50.712185+0000","last_active":"2024-03-28T23:29:51.687297+0000","last_peered":"2024-03-28T23:29:51.687297+0000","last_clean":"2024-03-28T23:29:51.687297+0000","last_became_active":"2024-03-28T23:29:50.711823+0000","last_became_peered":"2024-03-28T23:29:50.711823+0000","last_unstale":"2024-03-28T23:29:51.687297+0000","last_undegraded":"2024-03-28T23:29:51.687297+0000","last_fullsized":"2024-03-28T23:29:51.687297+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:40.107649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685755+0000","last_change":"2024-03-28T23:29:46.570826+0000","last_active":"2024-03-28T23:29:51.685755+0000","last_peered":"2024-03-28T23:29:51.685755+0000","last_clean":"2024-03-28T23:29:51.685755+0000","last_became_active":"2024-03-28T23:29:46.570513+0000","last_became_peered":"2024-03-28T23:29:46.570513+0000","last_unstale":"2024-03-28T23:29:51.685755+0000","last_undegraded":"2024-03-28T23:29:51.685755+0000","last_fullsized":"2024-03-28T23:29:51.685755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:08:33.801844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686189+0000","last_change":"2024-03-28T23:29:45.554954+0000","last_active":"2024-03-28T23:29:51.686189+0000","last_peered":"2024-03-28T23:29:51.686189+0000","last_clean":"2024-03-28T23:29:51.686189+0000","last_became_active":"2024-03-28T23:29:45.554687+0000","last_became_peered":"2024-03-28T23:29:45.554687+0000","last_unstale":"2024-03-28T23:29:51.686189+0000","last_undegraded":"2024-03-28T23:29:51.686189+0000","last_fullsized":"2024-03-28T23:29:51.686189+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:24:58.784056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687871+0000","last_change":"2024-03-28T23:29:50.707207+0000","last_active":"2024-03-28T23:29:51.687871+0000","last_peered":"2024-03-28T23:29:51.687871+0000","last_clean":"2024-03-28T23:29:51.687871+0000","last_became_active":"2024-03-28T23:29:50.706585+0000","last_became_peered":"2024-03-28T23:29:50.706585+0000","last_unstale":"2024-03-28T23:29:51.687871+0000","last_undegraded":"2024-03-28T23:29:51.687871+0000","last_fullsized":"2024-03-28T23:29:51.687871+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:10:11.936256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687068+0000","last_change":"2024-03-28T23:29:46.578519+0000","last_active":"2024-03-28T23:29:51.687068+0000","last_peered":"2024-03-28T23:29:51.687068+0000","last_clean":"2024-03-28T23:29:51.687068+0000","last_became_active":"2024-03-28T23:29:46.577803+0000","last_became_peered":"2024-03-28T23:29:46.577803+0000","last_unstale":"2024-03-28T23:29:51.687068+0000","last_undegraded":"2024-03-28T23:29:51.687068+0000","last_fullsized":"2024-03-28T23:29:51.687068+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:32.936386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687237+0000","last_change":"2024-03-28T23:29:45.557114+0000","last_active":"2024-03-28T23:29:51.687237+0000","last_peered":"2024-03-28T23:29:51.687237+0000","last_clean":"2024-03-28T23:29:51.687237+0000","last_became_active":"2024-03-28T23:29:45.556800+0000","last_became_peered":"2024-03-28T23:29:45.556800+0000","last_unstale":"2024-03-28T23:29:51.687237+0000","last_undegraded":"2024-03-28T23:29:51.687237+0000","last_fullsized":"2024-03-28T23:29:51.687237+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:06:18.025605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686083+0000","last_change":"2024-03-28T23:29:50.707356+0000","last_active":"2024-03-28T23:29:51.686083+0000","last_peered":"2024-03-28T23:29:51.686083+0000","last_clean":"2024-03-28T23:29:51.686083+0000","last_became_active":"2024-03-28T23:29:50.706320+0000","last_became_peered":"2024-03-28T23:29:50.706320+0000","last_unstale":"2024-03-28T23:29:51.686083+0000","last_undegraded":"2024-03-28T23:29:51.686083+0000","last_fullsized":"2024-03-28T23:29:51.686083+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:36.341186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686444+0000","last_change":"2024-03-28T23:29:46.573192+0000","last_active":"2024-03-28T23:29:51.686444+0000","last_peered":"2024-03-28T23:29:51.686444+0000","last_clean":"2024-03-28T23:29:51.686444+0000","last_became_active":"2024-03-28T23:29:46.572701+0000","last_became_peered":"2024-03-28T23:29:46.572701+0000","last_unstale":"2024-03-28T23:29:51.686444+0000","last_undegraded":"2024-03-28T23:29:51.686444+0000","last_fullsized":"2024-03-28T23:29:51.686444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:17:49.979210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:56.151164+0000","last_change":"2024-03-28T23:29:45.558929+0000","last_active":"2024-03-28T23:29:56.151164+0000","last_peered":"2024-03-28T23:29:56.151164+0000","last_clean":"2024-03-28T23:29:56.151164+0000","last_became_active":"2024-03-28T23:29:45.558657+0000","last_became_peered":"2024-03-28T23:29:45.558657+0000","last_unstale":"2024-03-28T23:29:56.151164+0000","last_undegraded":"2024-03-28T23:29:56.151164+0000","last_fullsized":"2024-03-28T23:29:56.151164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:24:58.784056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686265+0000","last_change":"2024-03-28T23:29:50.594405+0000","last_active":"2024-03-28T23:29:51.686265+0000","last_peered":"2024-03-28T23:29:51.686265+0000","last_clean":"2024-03-28T23:29:51.686265+0000","last_became_active":"2024-03-28T23:29:50.594017+0000","last_became_peered":"2024-03-28T23:29:50.594017+0000","last_unstale":"2024-03-28T23:29:51.686265+0000","last_undegraded":"2024-03-28T23:29:51.686265+0000","last_fullsized":"2024-03-28T23:29:51.686265+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:06.894069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591773+0000","last_change":"2024-03-28T23:29:46.561971+0000","last_active":"2024-03-28T23:29:51.591773+0000","last_peered":"2024-03-28T23:29:51.591773+0000","last_clean":"2024-03-28T23:29:51.591773+0000","last_became_active":"2024-03-28T23:29:46.561684+0000","last_became_peered":"2024-03-28T23:29:46.561684+0000","last_unstale":"2024-03-28T23:29:51.591773+0000","last_undegraded":"2024-03-28T23:29:51.591773+0000","last_fullsized":"2024-03-28T23:29:51.591773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:42.629935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685953+0000","last_change":"2024-03-28T23:29:45.563130+0000","last_active":"2024-03-28T23:29:51.685953+0000","last_peered":"2024-03-28T23:29:51.685953+0000","last_clean":"2024-03-28T23:29:51.685953+0000","last_became_active":"2024-03-28T23:29:45.562696+0000","last_became_peered":"2024-03-28T23:29:45.562696+0000","last_unstale":"2024-03-28T23:29:51.685953+0000","last_undegraded":"2024-03-28T23:29:51.685953+0000","last_fullsized":"2024-03-28T23:29:51.685953+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:17.783972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.592114+0000","last_change":"2024-03-28T23:29:50.587580+0000","last_active":"2024-03-28T23:29:51.592114+0000","last_peered":"2024-03-28T23:29:51.592114+0000","last_clean":"2024-03-28T23:29:51.592114+0000","last_became_active":"2024-03-28T23:29:50.587175+0000","last_became_peered":"2024-03-28T23:29:50.587175+0000","last_unstale":"2024-03-28T23:29:51.592114+0000","last_undegraded":"2024-03-28T23:29:51.592114+0000","last_fullsized":"2024-03-28T23:29:51.592114+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:06:40.917529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687664+0000","last_change":"2024-03-28T23:29:46.578684+0000","last_active":"2024-03-28T23:29:51.687664+0000","last_peered":"2024-03-28T23:29:51.687664+0000","last_clean":"2024-03-28T23:29:51.687664+0000","last_became_active":"2024-03-28T23:29:46.577924+0000","last_became_peered":"2024-03-28T23:29:46.577924+0000","last_unstale":"2024-03-28T23:29:51.687664+0000","last_undegraded":"2024-03-28T23:29:51.687664+0000","last_fullsized":"2024-03-28T23:29:51.687664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:36:20.485861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685874+0000","last_change":"2024-03-28T23:29:45.559534+0000","last_active":"2024-03-28T23:29:51.685874+0000","last_peered":"2024-03-28T23:29:51.685874+0000","last_clean":"2024-03-28T23:29:51.685874+0000","last_became_active":"2024-03-28T23:29:45.559243+0000","last_became_peered":"2024-03-28T23:29:45.559243+0000","last_unstale":"2024-03-28T23:29:51.685874+0000","last_undegraded":"2024-03-28T23:29:51.685874+0000","last_fullsized":"2024-03-28T23:29:51.685874+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:17:17.207826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686108+0000","last_change":"2024-03-28T23:29:50.601743+0000","last_active":"2024-03-28T23:29:51.686108+0000","last_peered":"2024-03-28T23:29:51.686108+0000","last_clean":"2024-03-28T23:29:51.686108+0000","last_became_active":"2024-03-28T23:29:50.601327+0000","last_became_peered":"2024-03-28T23:29:50.601327+0000","last_unstale":"2024-03-28T23:29:51.686108+0000","last_undegraded":"2024-03-28T23:29:51.686108+0000","last_fullsized":"2024-03-28T23:29:51.686108+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:36.341186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685724+0000","last_change":"2024-03-28T23:29:46.573188+0000","last_active":"2024-03-28T23:29:51.685724+0000","last_peered":"2024-03-28T23:29:51.685724+0000","last_clean":"2024-03-28T23:29:51.685724+0000","last_became_active":"2024-03-28T23:29:46.572885+0000","last_became_peered":"2024-03-28T23:29:46.572885+0000","last_unstale":"2024-03-28T23:29:51.685724+0000","last_undegraded":"2024-03-28T23:29:51.685724+0000","last_fullsized":"2024-03-28T23:29:51.685724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:08:33.801844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591244+0000","last_change":"2024-03-28T23:29:45.564128+0000","last_active":"2024-03-28T23:29:51.591244+0000","last_peered":"2024-03-28T23:29:51.591244+0000","last_clean":"2024-03-28T23:29:51.591244+0000","last_became_active":"2024-03-28T23:29:45.563837+0000","last_became_peered":"2024-03-28T23:29:45.563837+0000","last_unstale":"2024-03-28T23:29:51.591244+0000","last_undegraded":"2024-03-28T23:29:51.591244+0000","last_fullsized":"2024-03-28T23:29:51.591244+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:17:17.207826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686626+0000","last_change":"2024-03-28T23:29:50.708325+0000","last_active":"2024-03-28T23:29:51.686626+0000","last_peered":"2024-03-28T23:29:51.686626+0000","last_clean":"2024-03-28T23:29:51.686626+0000","last_became_active":"2024-03-28T23:29:50.707929+0000","last_became_peered":"2024-03-28T23:29:50.707929+0000","last_unstale":"2024-03-28T23:29:51.686626+0000","last_undegraded":"2024-03-28T23:29:51.686626+0000","last_fullsized":"2024-03-28T23:29:51.686626+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:01.743074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687303+0000","last_change":"2024-03-28T23:29:50.711341+0000","last_active":"2024-03-28T23:29:51.687303+0000","last_peered":"2024-03-28T23:29:51.687303+0000","last_clean":"2024-03-28T23:29:51.687303+0000","last_became_active":"2024-03-28T23:29:50.710983+0000","last_became_peered":"2024-03-28T23:29:50.710983+0000","last_unstale":"2024-03-28T23:29:51.687303+0000","last_undegraded":"2024-03-28T23:29:51.687303+0000","last_fullsized":"2024-03-28T23:29:51.687303+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:30:48.379529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685433+0000","last_change":"2024-03-28T23:29:46.561654+0000","last_active":"2024-03-28T23:29:51.685433+0000","last_peered":"2024-03-28T23:29:51.685433+0000","last_clean":"2024-03-28T23:29:51.685433+0000","last_became_active":"2024-03-28T23:29:46.561250+0000","last_became_peered":"2024-03-28T23:29:46.561250+0000","last_unstale":"2024-03-28T23:29:51.685433+0000","last_undegraded":"2024-03-28T23:29:51.685433+0000","last_fullsized":"2024-03-28T23:29:51.685433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:38.074388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685648+0000","last_change":"2024-03-28T23:29:45.553472+0000","last_active":"2024-03-28T23:29:51.685648+0000","last_peered":"2024-03-28T23:29:51.685648+0000","last_clean":"2024-03-28T23:29:51.685648+0000","last_became_active":"2024-03-28T23:29:45.553154+0000","last_became_peered":"2024-03-28T23:29:45.553154+0000","last_unstale":"2024-03-28T23:29:51.685648+0000","last_undegraded":"2024-03-28T23:29:51.685648+0000","last_fullsized":"2024-03-28T23:29:51.685648+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:36.113647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591126+0000","last_change":"2024-03-28T23:29:50.600387+0000","last_active":"2024-03-28T23:29:51.591126+0000","last_peered":"2024-03-28T23:29:51.591126+0000","last_clean":"2024-03-28T23:29:51.591126+0000","last_became_active":"2024-03-28T23:29:50.599850+0000","last_became_peered":"2024-03-28T23:29:50.599850+0000","last_unstale":"2024-03-28T23:29:51.591126+0000","last_undegraded":"2024-03-28T23:29:51.591126+0000","last_fullsized":"2024-03-28T23:29:51.591126+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:08:37.836411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686149+0000","last_change":"2024-03-28T23:29:46.581533+0000","last_active":"2024-03-28T23:29:51.686149+0000","last_peered":"2024-03-28T23:29:51.686149+0000","last_clean":"2024-03-28T23:29:51.686149+0000","last_became_active":"2024-03-28T23:29:46.581288+0000","last_became_peered":"2024-03-28T23:29:46.581288+0000","last_unstale":"2024-03-28T23:29:51.686149+0000","last_undegraded":"2024-03-28T23:29:51.686149+0000","last_fullsized":"2024-03-28T23:29:51.686149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:24:19.353047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686836+0000","last_change":"2024-03-28T23:29:45.561047+0000","last_active":"2024-03-28T23:29:51.686836+0000","last_peered":"2024-03-28T23:29:51.686836+0000","last_clean":"2024-03-28T23:29:51.686836+0000","last_became_active":"2024-03-28T23:29:45.560778+0000","last_became_peered":"2024-03-28T23:29:45.560778+0000","last_unstale":"2024-03-28T23:29:51.686836+0000","last_undegraded":"2024-03-28T23:29:51.686836+0000","last_fullsized":"2024-03-28T23:29:51.686836+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:21.707459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.590800+0000","last_change":"2024-03-28T23:29:45.559446+0000","last_active":"2024-03-28T23:29:51.590800+0000","last_peered":"2024-03-28T23:29:51.590800+0000","last_clean":"2024-03-28T23:29:51.590800+0000","last_became_active":"2024-03-28T23:29:45.559015+0000","last_became_peered":"2024-03-28T23:29:45.559015+0000","last_unstale":"2024-03-28T23:29:51.590800+0000","last_undegraded":"2024-03-28T23:29:51.590800+0000","last_fullsized":"2024-03-28T23:29:51.590800+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:59.719252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687152+0000","last_change":"2024-03-28T23:29:46.578355+0000","last_active":"2024-03-28T23:29:51.687152+0000","last_peered":"2024-03-28T23:29:51.687152+0000","last_clean":"2024-03-28T23:29:51.687152+0000","last_became_active":"2024-03-28T23:29:46.577898+0000","last_became_peered":"2024-03-28T23:29:46.577898+0000","last_unstale":"2024-03-28T23:29:51.687152+0000","last_undegraded":"2024-03-28T23:29:51.687152+0000","last_fullsized":"2024-03-28T23:29:51.687152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:58.674327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589392+0000","last_change":"2024-03-28T23:29:50.601015+0000","last_active":"2024-03-28T23:29:51.589392+0000","last_peered":"2024-03-28T23:29:51.589392+0000","last_clean":"2024-03-28T23:29:51.589392+0000","last_became_active":"2024-03-28T23:29:50.600573+0000","last_became_peered":"2024-03-28T23:29:50.600573+0000","last_unstale":"2024-03-28T23:29:51.589392+0000","last_undegraded":"2024-03-28T23:29:51.589392+0000","last_fullsized":"2024-03-28T23:29:51.589392+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:38:05.665416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686050+0000","last_change":"2024-03-28T23:29:45.556159+0000","last_active":"2024-03-28T23:29:51.686050+0000","last_peered":"2024-03-28T23:29:51.686050+0000","last_clean":"2024-03-28T23:29:51.686050+0000","last_became_active":"2024-03-28T23:29:45.555866+0000","last_became_peered":"2024-03-28T23:29:45.555866+0000","last_unstale":"2024-03-28T23:29:51.686050+0000","last_undegraded":"2024-03-28T23:29:51.686050+0000","last_fullsized":"2024-03-28T23:29:51.686050+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:44.528076+0000","last_clean_scrub_stamp":"2024-03-28T23:29:44.528076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:17:17.207826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591783+0000","last_change":"2024-03-28T23:29:46.554056+0000","last_active":"2024-03-28T23:29:51.591783+0000","last_peered":"2024-03-28T23:29:51.591783+0000","last_clean":"2024-03-28T23:29:51.591783+0000","last_became_active":"2024-03-28T23:29:46.553772+0000","last_became_peered":"2024-03-28T23:29:46.553772+0000","last_unstale":"2024-03-28T23:29:51.591783+0000","last_undegraded":"2024-03-28T23:29:51.591783+0000","last_fullsized":"2024-03-28T23:29:51.591783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:02.992468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686381+0000","last_change":"2024-03-28T23:29:50.603254+0000","last_active":"2024-03-28T23:29:51.686381+0000","last_peered":"2024-03-28T23:29:51.686381+0000","last_clean":"2024-03-28T23:29:51.686381+0000","last_became_active":"2024-03-28T23:29:50.602868+0000","last_became_peered":"2024-03-28T23:29:50.602868+0000","last_unstale":"2024-03-28T23:29:51.686381+0000","last_undegraded":"2024-03-28T23:29:51.686381+0000","last_fullsized":"2024-03-28T23:29:51.686381+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:49.571390+0000","last_clean_scrub_stamp":"2024-03-28T23:29:49.571390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:42:43.014557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.590502+0000","last_change":"2024-03-28T23:29:46.581634+0000","last_active":"2024-03-28T23:29:51.590502+0000","last_peered":"2024-03-28T23:29:51.590502+0000","last_clean":"2024-03-28T23:29:51.590502+0000","last_became_active":"2024-03-28T23:29:46.581353+0000","last_became_peered":"2024-03-28T23:29:46.581353+0000","last_unstale":"2024-03-28T23:29:51.590502+0000","last_undegraded":"2024-03-28T23:29:51.590502+0000","last_fullsized":"2024-03-28T23:29:51.590502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:18:20.039954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589110+0000","last_change":"2024-03-28T23:29:46.563584+0000","last_active":"2024-03-28T23:29:51.589110+0000","last_peered":"2024-03-28T23:29:51.589110+0000","last_clean":"2024-03-28T23:29:51.589110+0000","last_became_active":"2024-03-28T23:29:46.563265+0000","last_became_peered":"2024-03-28T23:29:46.563265+0000","last_unstale":"2024-03-28T23:29:51.589110+0000","last_undegraded":"2024-03-28T23:29:51.589110+0000","last_fullsized":"2024-03-28T23:29:51.589110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:42.629935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685798+0000","last_change":"2024-03-28T23:29:46.561377+0000","last_active":"2024-03-28T23:29:51.685798+0000","last_peered":"2024-03-28T23:29:51.685798+0000","last_clean":"2024-03-28T23:29:51.685798+0000","last_became_active":"2024-03-28T23:29:46.561091+0000","last_became_peered":"2024-03-28T23:29:46.561091+0000","last_unstale":"2024-03-28T23:29:51.685798+0000","last_undegraded":"2024-03-28T23:29:51.685798+0000","last_fullsized":"2024-03-28T23:29:51.685798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:30:44.344961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687062+0000","last_change":"2024-03-28T23:29:46.578999+0000","last_active":"2024-03-28T23:29:51.687062+0000","last_peered":"2024-03-28T23:29:51.687062+0000","last_clean":"2024-03-28T23:29:51.687062+0000","last_became_active":"2024-03-28T23:29:46.578263+0000","last_became_peered":"2024-03-28T23:29:46.578263+0000","last_unstale":"2024-03-28T23:29:51.687062+0000","last_undegraded":"2024-03-28T23:29:51.687062+0000","last_fullsized":"2024-03-28T23:29:51.687062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:10.853710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687925+0000","last_change":"2024-03-28T23:29:46.566611+0000","last_active":"2024-03-28T23:29:51.687925+0000","last_peered":"2024-03-28T23:29:51.687925+0000","last_clean":"2024-03-28T23:29:51.687925+0000","last_became_active":"2024-03-28T23:29:46.566361+0000","last_became_peered":"2024-03-28T23:29:46.566361+0000","last_unstale":"2024-03-28T23:29:51.687925+0000","last_undegraded":"2024-03-28T23:29:51.687925+0000","last_fullsized":"2024-03-28T23:29:51.687925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:29:20.669429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685784+0000","last_change":"2024-03-28T23:29:46.561125+0000","last_active":"2024-03-28T23:29:51.685784+0000","last_peered":"2024-03-28T23:29:51.685784+0000","last_clean":"2024-03-28T23:29:51.685784+0000","last_became_active":"2024-03-28T23:29:46.559223+0000","last_became_peered":"2024-03-28T23:29:46.559223+0000","last_unstale":"2024-03-28T23:29:51.685784+0000","last_undegraded":"2024-03-28T23:29:51.685784+0000","last_fullsized":"2024-03-28T23:29:51.685784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:36:20.485861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591456+0000","last_change":"2024-03-28T23:29:46.574608+0000","last_active":"2024-03-28T23:29:51.591456+0000","last_peered":"2024-03-28T23:29:51.591456+0000","last_clean":"2024-03-28T23:29:51.591456+0000","last_became_active":"2024-03-28T23:29:46.574329+0000","last_became_peered":"2024-03-28T23:29:46.574329+0000","last_unstale":"2024-03-28T23:29:51.591456+0000","last_undegraded":"2024-03-28T23:29:51.591456+0000","last_fullsized":"2024-03-28T23:29:51.591456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:15.386423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687344+0000","last_change":"2024-03-28T23:29:46.566187+0000","last_active":"2024-03-28T23:29:51.687344+0000","last_peered":"2024-03-28T23:29:51.687344+0000","last_clean":"2024-03-28T23:29:51.687344+0000","last_became_active":"2024-03-28T23:29:46.565873+0000","last_became_peered":"2024-03-28T23:29:46.565873+0000","last_unstale":"2024-03-28T23:29:51.687344+0000","last_undegraded":"2024-03-28T23:29:51.687344+0000","last_fullsized":"2024-03-28T23:29:51.687344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:14:59.182189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685520+0000","last_change":"2024-03-28T23:29:46.559531+0000","last_active":"2024-03-28T23:29:51.685520+0000","last_peered":"2024-03-28T23:29:51.685520+0000","last_clean":"2024-03-28T23:29:51.685520+0000","last_became_active":"2024-03-28T23:29:46.559236+0000","last_became_peered":"2024-03-28T23:29:46.559236+0000","last_unstale":"2024-03-28T23:29:51.685520+0000","last_undegraded":"2024-03-28T23:29:51.685520+0000","last_fullsized":"2024-03-28T23:29:51.685520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:06.731262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686762+0000","last_change":"2024-03-28T23:29:46.555898+0000","last_active":"2024-03-28T23:29:51.686762+0000","last_peered":"2024-03-28T23:29:51.686762+0000","last_clean":"2024-03-28T23:29:51.686762+0000","last_became_active":"2024-03-28T23:29:46.555610+0000","last_became_peered":"2024-03-28T23:29:46.555610+0000","last_unstale":"2024-03-28T23:29:51.686762+0000","last_undegraded":"2024-03-28T23:29:51.686762+0000","last_fullsized":"2024-03-28T23:29:51.686762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:38.074388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685565+0000","last_change":"2024-03-28T23:29:46.573905+0000","last_active":"2024-03-28T23:29:51.685565+0000","last_peered":"2024-03-28T23:29:51.685565+0000","last_clean":"2024-03-28T23:29:51.685565+0000","last_became_active":"2024-03-28T23:29:46.573631+0000","last_became_peered":"2024-03-28T23:29:46.573631+0000","last_unstale":"2024-03-28T23:29:51.685565+0000","last_undegraded":"2024-03-28T23:29:51.685565+0000","last_fullsized":"2024-03-28T23:29:51.685565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:14:59.182189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686795+0000","last_change":"2024-03-28T23:29:46.579400+0000","last_active":"2024-03-28T23:29:51.686795+0000","last_peered":"2024-03-28T23:29:51.686795+0000","last_clean":"2024-03-28T23:29:51.686795+0000","last_became_active":"2024-03-28T23:29:46.579070+0000","last_became_peered":"2024-03-28T23:29:46.579070+0000","last_unstale":"2024-03-28T23:29:51.686795+0000","last_undegraded":"2024-03-28T23:29:51.686795+0000","last_fullsized":"2024-03-28T23:29:51.686795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:15.386423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591130+0000","last_change":"2024-03-28T23:29:46.569441+0000","last_active":"2024-03-28T23:29:51.591130+0000","last_peered":"2024-03-28T23:29:51.591130+0000","last_clean":"2024-03-28T23:29:51.591130+0000","last_became_active":"2024-03-28T23:29:46.569151+0000","last_became_peered":"2024-03-28T23:29:46.569151+0000","last_unstale":"2024-03-28T23:29:51.591130+0000","last_undegraded":"2024-03-28T23:29:51.591130+0000","last_fullsized":"2024-03-28T23:29:51.591130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:38.074388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686720+0000","last_change":"2024-03-28T23:29:46.573927+0000","last_active":"2024-03-28T23:29:51.686720+0000","last_peered":"2024-03-28T23:29:51.686720+0000","last_clean":"2024-03-28T23:29:51.686720+0000","last_became_active":"2024-03-28T23:29:46.573668+0000","last_became_peered":"2024-03-28T23:29:46.573668+0000","last_unstale":"2024-03-28T23:29:51.686720+0000","last_undegraded":"2024-03-28T23:29:51.686720+0000","last_fullsized":"2024-03-28T23:29:51.686720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:30:44.344961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.589322+0000","last_change":"2024-03-28T23:29:46.558629+0000","last_active":"2024-03-28T23:29:51.589322+0000","last_peered":"2024-03-28T23:29:51.589322+0000","last_clean":"2024-03-28T23:29:51.589322+0000","last_became_active":"2024-03-28T23:29:46.558311+0000","last_became_peered":"2024-03-28T23:29:46.558311+0000","last_unstale":"2024-03-28T23:29:51.589322+0000","last_undegraded":"2024-03-28T23:29:51.589322+0000","last_fullsized":"2024-03-28T23:29:51.589322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:10.938731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686262+0000","last_change":"2024-03-28T23:29:46.573394+0000","last_active":"2024-03-28T23:29:51.686262+0000","last_peered":"2024-03-28T23:29:51.686262+0000","last_clean":"2024-03-28T23:29:51.686262+0000","last_became_active":"2024-03-28T23:29:46.573130+0000","last_became_peered":"2024-03-28T23:29:46.573130+0000","last_unstale":"2024-03-28T23:29:51.686262+0000","last_undegraded":"2024-03-28T23:29:51.686262+0000","last_fullsized":"2024-03-28T23:29:51.686262+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:43:10.380471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685367+0000","last_change":"2024-03-28T23:29:46.581612+0000","last_active":"2024-03-28T23:29:51.685367+0000","last_peered":"2024-03-28T23:29:51.685367+0000","last_clean":"2024-03-28T23:29:51.685367+0000","last_became_active":"2024-03-28T23:29:46.581324+0000","last_became_peered":"2024-03-28T23:29:46.581324+0000","last_unstale":"2024-03-28T23:29:51.685367+0000","last_undegraded":"2024-03-28T23:29:51.685367+0000","last_fullsized":"2024-03-28T23:29:51.685367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:10.853710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685527+0000","last_change":"2024-03-28T23:29:46.552565+0000","last_active":"2024-03-28T23:29:51.685527+0000","last_peered":"2024-03-28T23:29:51.685527+0000","last_clean":"2024-03-28T23:29:51.685527+0000","last_became_active":"2024-03-28T23:29:46.552304+0000","last_became_peered":"2024-03-28T23:29:46.552304+0000","last_unstale":"2024-03-28T23:29:51.685527+0000","last_undegraded":"2024-03-28T23:29:51.685527+0000","last_fullsized":"2024-03-28T23:29:51.685527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:15.386423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686970+0000","last_change":"2024-03-28T23:29:46.579218+0000","last_active":"2024-03-28T23:29:51.686970+0000","last_peered":"2024-03-28T23:29:51.686970+0000","last_clean":"2024-03-28T23:29:51.686970+0000","last_became_active":"2024-03-28T23:29:46.578756+0000","last_became_peered":"2024-03-28T23:29:46.578756+0000","last_unstale":"2024-03-28T23:29:51.686970+0000","last_undegraded":"2024-03-28T23:29:51.686970+0000","last_fullsized":"2024-03-28T23:29:51.686970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:24:19.353047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685891+0000","last_change":"2024-03-28T23:29:46.556797+0000","last_active":"2024-03-28T23:29:51.685891+0000","last_peered":"2024-03-28T23:29:51.685891+0000","last_clean":"2024-03-28T23:29:51.685891+0000","last_became_active":"2024-03-28T23:29:46.556520+0000","last_became_peered":"2024-03-28T23:29:46.556520+0000","last_unstale":"2024-03-28T23:29:51.685891+0000","last_undegraded":"2024-03-28T23:29:51.685891+0000","last_fullsized":"2024-03-28T23:29:51.685891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:15.386423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591677+0000","last_change":"2024-03-28T23:29:46.559749+0000","last_active":"2024-03-28T23:29:51.591677+0000","last_peered":"2024-03-28T23:29:51.591677+0000","last_clean":"2024-03-28T23:29:51.591677+0000","last_became_active":"2024-03-28T23:29:46.559386+0000","last_became_peered":"2024-03-28T23:29:46.559386+0000","last_unstale":"2024-03-28T23:29:51.591677+0000","last_undegraded":"2024-03-28T23:29:51.591677+0000","last_fullsized":"2024-03-28T23:29:51.591677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:10.938731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591382+0000","last_change":"2024-03-28T23:29:46.571568+0000","last_active":"2024-03-28T23:29:51.591382+0000","last_peered":"2024-03-28T23:29:51.591382+0000","last_clean":"2024-03-28T23:29:51.591382+0000","last_became_active":"2024-03-28T23:29:46.571268+0000","last_became_peered":"2024-03-28T23:29:46.571268+0000","last_unstale":"2024-03-28T23:29:51.591382+0000","last_undegraded":"2024-03-28T23:29:51.591382+0000","last_fullsized":"2024-03-28T23:29:51.591382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:36:20.485861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.591361+0000","last_change":"2024-03-28T23:29:46.571722+0000","last_active":"2024-03-28T23:29:51.591361+0000","last_peered":"2024-03-28T23:29:51.591361+0000","last_clean":"2024-03-28T23:29:51.591361+0000","last_became_active":"2024-03-28T23:29:46.571437+0000","last_became_peered":"2024-03-28T23:29:46.571437+0000","last_unstale":"2024-03-28T23:29:51.591361+0000","last_undegraded":"2024-03-28T23:29:51.591361+0000","last_fullsized":"2024-03-28T23:29:51.591361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:10.853710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686822+0000","last_change":"2024-03-28T23:29:46.565939+0000","last_active":"2024-03-28T23:29:51.686822+0000","last_peered":"2024-03-28T23:29:51.686822+0000","last_clean":"2024-03-28T23:29:51.686822+0000","last_became_active":"2024-03-28T23:29:46.564853+0000","last_became_peered":"2024-03-28T23:29:46.564853+0000","last_unstale":"2024-03-28T23:29:51.686822+0000","last_undegraded":"2024-03-28T23:29:51.686822+0000","last_fullsized":"2024-03-28T23:29:51.686822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:18.050658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685938+0000","last_change":"2024-03-28T23:29:46.567115+0000","last_active":"2024-03-28T23:29:51.685938+0000","last_peered":"2024-03-28T23:29:51.685938+0000","last_clean":"2024-03-28T23:29:51.685938+0000","last_became_active":"2024-03-28T23:29:46.566742+0000","last_became_peered":"2024-03-28T23:29:46.566742+0000","last_unstale":"2024-03-28T23:29:51.685938+0000","last_undegraded":"2024-03-28T23:29:51.685938+0000","last_fullsized":"2024-03-28T23:29:51.685938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:32.936386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686798+0000","last_change":"2024-03-28T23:29:46.579765+0000","last_active":"2024-03-28T23:29:51.686798+0000","last_peered":"2024-03-28T23:29:51.686798+0000","last_clean":"2024-03-28T23:29:51.686798+0000","last_became_active":"2024-03-28T23:29:46.579469+0000","last_became_peered":"2024-03-28T23:29:46.579469+0000","last_unstale":"2024-03-28T23:29:51.686798+0000","last_undegraded":"2024-03-28T23:29:51.686798+0000","last_fullsized":"2024-03-28T23:29:51.686798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:06.731262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685479+0000","last_change":"2024-03-28T23:29:46.580994+0000","last_active":"2024-03-28T23:29:51.685479+0000","last_peered":"2024-03-28T23:29:51.685479+0000","last_clean":"2024-03-28T23:29:51.685479+0000","last_became_active":"2024-03-28T23:29:46.580685+0000","last_became_peered":"2024-03-28T23:29:46.580685+0000","last_unstale":"2024-03-28T23:29:51.685479+0000","last_undegraded":"2024-03-28T23:29:51.685479+0000","last_fullsized":"2024-03-28T23:29:51.685479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:58.674327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685558+0000","last_change":"2024-03-28T23:29:46.571922+0000","last_active":"2024-03-28T23:29:51.685558+0000","last_peered":"2024-03-28T23:29:51.685558+0000","last_clean":"2024-03-28T23:29:51.685558+0000","last_became_active":"2024-03-28T23:29:46.571256+0000","last_became_peered":"2024-03-28T23:29:46.571256+0000","last_unstale":"2024-03-28T23:29:51.685558+0000","last_undegraded":"2024-03-28T23:29:51.685558+0000","last_fullsized":"2024-03-28T23:29:51.685558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:54.470202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.685672+0000","last_change":"2024-03-28T23:29:46.561038+0000","last_active":"2024-03-28T23:29:51.685672+0000","last_peered":"2024-03-28T23:29:51.685672+0000","last_clean":"2024-03-28T23:29:51.685672+0000","last_became_active":"2024-03-28T23:29:46.560679+0000","last_became_peered":"2024-03-28T23:29:46.560679+0000","last_unstale":"2024-03-28T23:29:51.685672+0000","last_undegraded":"2024-03-28T23:29:51.685672+0000","last_fullsized":"2024-03-28T23:29:51.685672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:29:20.669429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.687099+0000","last_change":"2024-03-28T23:29:46.578506+0000","last_active":"2024-03-28T23:29:51.687099+0000","last_peered":"2024-03-28T23:29:51.687099+0000","last_clean":"2024-03-28T23:29:51.687099+0000","last_became_active":"2024-03-28T23:29:46.577958+0000","last_became_peered":"2024-03-28T23:29:46.577958+0000","last_unstale":"2024-03-28T23:29:51.687099+0000","last_undegraded":"2024-03-28T23:29:51.687099+0000","last_fullsized":"2024-03-28T23:29:51.687099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:38:01.630848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686703+0000","last_change":"2024-03-28T23:29:46.579555+0000","last_active":"2024-03-28T23:29:51.686703+0000","last_peered":"2024-03-28T23:29:51.686703+0000","last_clean":"2024-03-28T23:29:51.686703+0000","last_became_active":"2024-03-28T23:29:46.579273+0000","last_became_peered":"2024-03-28T23:29:46.579273+0000","last_unstale":"2024-03-28T23:29:51.686703+0000","last_undegraded":"2024-03-28T23:29:51.686703+0000","last_fullsized":"2024-03-28T23:29:51.686703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:10.853710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T23:29:51.686627+0000","last_change":"2024-03-28T23:29:46.568698+0000","last_active":"2024-03-28T23:29:51.686627+0000","last_peered":"2024-03-28T23:29:51.686627+0000","last_clean":"2024-03-28T23:29:51.686627+0000","last_became_active":"2024-03-28T23:29:46.568318+0000","last_became_peered":"2024-03-28T23:29:46.568318+0000","last_unstale":"2024-03-28T23:29:51.686627+0000","last_undegraded":"2024-03-28T23:29:51.686627+0000","last_fullsized":"2024-03-28T23:29:51.686627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:45.536823+0000","last_clean_scrub_stamp":"2024-03-28T23:29:45.536823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:43:10.380471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31504,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340336,"statfs":{"total":96636764160,"available":96604504064,"internally_reserved":0,"allocated":323584,"data_stored":691206,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":331776,"data_stored":691262,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:30:14.870 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-28T23:30:14.870 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-28T23:30:14.871 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-28T23:30:14.871 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-28T23:30:15.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.005+0000 7fdf5a969640 1 Processor -- start 2024-03-28T23:30:15.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.006+0000 7fdf5a969640 1 -- start start 2024-03-28T23:30:15.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.006+0000 7fdf5a969640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fdf540caf60 0x7fdf541592e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:15.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.006+0000 7fdf5a969640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf540c8970 0x7fdf540c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:15.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.006+0000 7fdf5a969640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf540cb4e0 con 0x7fdf5414f260 2024-03-28T23:30:15.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.006+0000 7fdf5a969640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fdf54158e50 con 0x7fdf540caf60 2024-03-28T23:30:15.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.006+0000 7fdf5a969640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdf540cb7c0 con 0x7fdf540c8970 2024-03-28T23:30:15.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.007+0000 7fdf537fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fdf540caf60 0x7fdf541592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:15.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.007+0000 7fdf58f06640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf540c8970 0x7fdf540c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:15.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.007+0000 7fdf537fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fdf540caf60 0x7fdf541592e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:39438/0 (socket says 172.21.15.161:39438) 2024-03-28T23:30:15.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.007+0000 7fdf53fff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fdf5414f260 0x7fdf54155c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:46216/0 (socket says 172.21.15.161:46216) 2024-03-28T23:30:15.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.007+0000 7fdf537fe640 1 -- 172.21.15.161:0/2646055938 learned_addr learned my addr 172.21.15.161:0/2646055938 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:15.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.007+0000 7fdf58f06640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf540c8970 0x7fdf540c53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:37114/0 (socket says 172.21.15.161:37114) 2024-03-28T23:30:15.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.007+0000 7fdf537fe640 1 -- 172.21.15.161:0/2646055938 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf540c8970 msgr2=0x7fdf540c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:15.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.007+0000 7fdf537fe640 1 --2- 172.21.15.161:0/2646055938 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf540c8970 0x7fdf540c53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.007+0000 7fdf537fe640 1 -- 172.21.15.161:0/2646055938 >> v1:172.21.15.170:6789/0 conn(0x7fdf5414f260 legacy=0x7fdf54155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:15.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.007+0000 7fdf537fe640 1 -- 172.21.15.161:0/2646055938 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf54158800 con 0x7fdf540caf60 2024-03-28T23:30:15.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.007+0000 7fdf58f06640 1 --2- 172.21.15.161:0/2646055938 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf540c8970 0x7fdf540c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:15.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.008+0000 7fdf537fe640 1 --2- 172.21.15.161:0/2646055938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf540caf60 0x7fdf541592e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdf48009870 tx=0x7fdf4802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e801d32f7f9c1a59 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:15.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.008+0000 7fdf52ffd640 1 -- 172.21.15.161:0/2646055938 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf48035020 con 0x7fdf540caf60 2024-03-28T23:30:15.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.008+0000 7fdf52ffd640 1 -- 172.21.15.161:0/2646055938 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf48031640 con 0x7fdf540caf60 2024-03-28T23:30:15.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.008+0000 7fdf52ffd640 1 -- 172.21.15.161:0/2646055938 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf4802d040 con 0x7fdf540caf60 2024-03-28T23:30:15.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.009+0000 7fdf5a969640 1 -- 172.21.15.161:0/2646055938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf540caf60 msgr2=0x7fdf541592e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:15.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.009+0000 7fdf5a969640 1 --2- 172.21.15.161:0/2646055938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf540caf60 0x7fdf541592e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdf48009870 tx=0x7fdf4802b8f0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.009+0000 7fdf5a969640 1 -- 172.21.15.161:0/2646055938 shutdown_connections 2024-03-28T23:30:15.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.009+0000 7fdf5a969640 1 --2- 172.21.15.161:0/2646055938 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf540c8970 0x7fdf540c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.009+0000 7fdf5a969640 1 --2- 172.21.15.161:0/2646055938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf540caf60 0x7fdf541592e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.009+0000 7fdf5a969640 1 -- 172.21.15.161:0/2646055938 >> 172.21.15.161:0/2646055938 conn(0x7fdf54072ca0 msgr2=0x7fdf540730b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:15.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.009+0000 7fdf5a969640 1 -- 172.21.15.161:0/2646055938 shutdown_connections 2024-03-28T23:30:15.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.009+0000 7fdf5a969640 1 -- 172.21.15.161:0/2646055938 wait complete. 2024-03-28T23:30:15.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.010+0000 7fdf5a969640 1 Processor -- start 2024-03-28T23:30:15.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.010+0000 7fdf5a969640 1 -- start start 2024-03-28T23:30:15.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.011+0000 7fdf5a969640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf540c8970 0x7fdf541546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:15.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.011+0000 7fdf5a969640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf540caf60 0x7fdf54150cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:15.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.011+0000 7fdf5a969640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf5414f260 0x7fdf54153080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:15.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.011+0000 7fdf5a969640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdf540c8e00 con 0x7fdf540caf60 2024-03-28T23:30:15.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.011+0000 7fdf5a969640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5414f640 con 0x7fdf5414f260 2024-03-28T23:30:15.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.011+0000 7fdf5a969640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdf540c8f80 con 0x7fdf540c8970 2024-03-28T23:30:15.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.011+0000 7fdf537fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf540caf60 0x7fdf54150cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:15.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.011+0000 7fdf58f06640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf5414f260 0x7fdf54153080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:15.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.011+0000 7fdf537fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf540caf60 0x7fdf54150cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54488/0 (socket says 172.21.15.161:54488) 2024-03-28T23:30:15.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.011+0000 7fdf53fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf540c8970 0x7fdf541546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:15.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.011+0000 7fdf537fe640 1 -- 172.21.15.161:0/525040495 learned_addr learned my addr 172.21.15.161:0/525040495 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:15.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.011+0000 7fdf537fe640 1 -- 172.21.15.161:0/525040495 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf540c8970 msgr2=0x7fdf541546a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:15.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.011+0000 7fdf537fe640 1 --2- 172.21.15.161:0/525040495 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf540c8970 0x7fdf541546a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.011+0000 7fdf537fe640 1 -- 172.21.15.161:0/525040495 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf5414f260 msgr2=0x7fdf54153080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:15.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.011+0000 7fdf537fe640 1 --2- 172.21.15.161:0/525040495 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf5414f260 0x7fdf54153080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.011+0000 7fdf537fe640 1 -- 172.21.15.161:0/525040495 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf54154ca0 con 0x7fdf540caf60 2024-03-28T23:30:15.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.011+0000 7fdf537fe640 1 --2- 172.21.15.161:0/525040495 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf540caf60 0x7fdf54150cd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdf48000c00 tx=0x7fdf480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:15.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.011+0000 7fdf50ff9640 1 -- 172.21.15.161:0/525040495 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf48040070 con 0x7fdf540caf60 2024-03-28T23:30:15.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.012+0000 7fdf50ff9640 1 -- 172.21.15.161:0/525040495 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf480399f0 con 0x7fdf540caf60 2024-03-28T23:30:15.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.012+0000 7fdf5a969640 1 -- 172.21.15.161:0/525040495 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf5414f860 con 0x7fdf540caf60 2024-03-28T23:30:15.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.012+0000 7fdf50ff9640 1 -- 172.21.15.161:0/525040495 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf48035040 con 0x7fdf540caf60 2024-03-28T23:30:15.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.012+0000 7fdf5a969640 1 -- 172.21.15.161:0/525040495 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf5414fe20 con 0x7fdf540caf60 2024-03-28T23:30:15.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.012+0000 7fdf5a969640 1 -- 172.21.15.161:0/525040495 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf5404f550 con 0x7fdf540caf60 2024-03-28T23:30:15.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.013+0000 7fdf50ff9640 1 -- 172.21.15.161:0/525040495 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdf4804b060 con 0x7fdf540caf60 2024-03-28T23:30:15.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.013+0000 7fdf50ff9640 1 --2- 172.21.15.161:0/525040495 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdf4006b4d0 0x7fdf4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:15.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.014+0000 7fdf50ff9640 1 -- 172.21.15.161:0/525040495 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fdf480abc40 con 0x7fdf540caf60 2024-03-28T23:30:15.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.014+0000 7fdf53fff640 1 --2- 172.21.15.161:0/525040495 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdf4006b4d0 0x7fdf4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:15.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.014+0000 7fdf53fff640 1 --2- 172.21.15.161:0/525040495 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdf4006b4d0 0x7fdf4006d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdf44000f60 tx=0x7fdf44000ed0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:15.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.018+0000 7fdf50ff9640 1 -- 172.21.15.161:0/525040495 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdf4807a780 con 0x7fdf540caf60 2024-03-28T23:30:15.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.349+0000 7fdf5a969640 1 -- 172.21.15.161:0/525040495 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fdf5414e560 con 0x7fdf540caf60 2024-03-28T23:30:15.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.350+0000 7fdf50ff9640 1 -- 172.21.15.161:0/525040495 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fdf4807a080 con 0x7fdf540caf60 2024-03-28T23:30:15.351 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:15.351 INFO:teuthology.orchestra.run.smithi161.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-28T23:30:15.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.351+0000 7fdf5a969640 1 -- 172.21.15.161:0/525040495 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdf4006b4d0 msgr2=0x7fdf4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:15.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.351+0000 7fdf5a969640 1 --2- 172.21.15.161:0/525040495 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdf4006b4d0 0x7fdf4006d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdf44000f60 tx=0x7fdf44000ed0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.351+0000 7fdf5a969640 1 -- 172.21.15.161:0/525040495 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf540caf60 msgr2=0x7fdf54150cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:15.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.351+0000 7fdf5a969640 1 --2- 172.21.15.161:0/525040495 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf540caf60 0x7fdf54150cd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdf48000c00 tx=0x7fdf480046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.352+0000 7fdf5a969640 1 -- 172.21.15.161:0/525040495 shutdown_connections 2024-03-28T23:30:15.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.352+0000 7fdf5a969640 1 --2- 172.21.15.161:0/525040495 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdf4006b4d0 0x7fdf4006d990 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.352+0000 7fdf5a969640 1 --2- 172.21.15.161:0/525040495 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf540caf60 0x7fdf54150cd0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.352+0000 7fdf5a969640 1 --2- 172.21.15.161:0/525040495 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf5414f260 0x7fdf54153080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.352+0000 7fdf5a969640 1 --2- 172.21.15.161:0/525040495 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf540c8970 0x7fdf541546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.352+0000 7fdf5a969640 1 -- 172.21.15.161:0/525040495 >> 172.21.15.161:0/525040495 conn(0x7fdf54072ca0 msgr2=0x7fdf5414dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:15.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.352+0000 7fdf5a969640 1 -- 172.21.15.161:0/525040495 shutdown_connections 2024-03-28T23:30:15.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.352+0000 7fdf5a969640 1 -- 172.21.15.161:0/525040495 wait complete. 2024-03-28T23:30:15.364 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-28T23:30:15.364 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:30:15.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.465+0000 7f0e73b9a640 1 Processor -- start 2024-03-28T23:30:15.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.465+0000 7f0e73b9a640 1 -- start start 2024-03-28T23:30:15.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.466+0000 7f0e73b9a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0e6c06ca20 0x7f0e6c06ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:15.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.466+0000 7f0e73b9a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0e6c0a98c0 0x7f0e6c0a9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:15.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.466+0000 7f0e73b9a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e6c151850 con 0x7f0e6c14eca0 2024-03-28T23:30:15.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.466+0000 7f0e73b9a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f0e6c152460 con 0x7f0e6c06ca20 2024-03-28T23:30:15.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.466+0000 7f0e73b9a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0e6c151b30 con 0x7f0e6c0a98c0 2024-03-28T23:30:15.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.466+0000 7f0e71936640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f0e6c14eca0 0x7f0e6c0c1880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59714/0 (socket says 172.21.15.161:59714) 2024-03-28T23:30:15.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.466+0000 7f0e71936640 1 -- 172.21.15.161:0/4176218599 learned_addr learned my addr 172.21.15.161:0/4176218599 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:15.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.466+0000 7f0e71135640 1 --2- 172.21.15.161:0/4176218599 >> v2:172.21.15.170:3300/0 conn(0x7f0e6c06ca20 0x7f0e6c06ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:15.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.466+0000 7f0e72137640 1 --2- 172.21.15.161:0/4176218599 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0e6c0a98c0 0x7f0e6c0a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:15.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.466+0000 7f0e70934640 1 -- 172.21.15.161:0/4176218599 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2544632560 0 0) 0x7f0e6c151850 con 0x7f0e6c14eca0 2024-03-28T23:30:15.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.466+0000 7f0e70934640 1 -- 172.21.15.161:0/4176218599 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e58003520 con 0x7f0e6c14eca0 2024-03-28T23:30:15.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.467+0000 7f0e70934640 1 -- 172.21.15.161:0/4176218599 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1746831403 0 0) 0x7f0e58003520 con 0x7f0e6c14eca0 2024-03-28T23:30:15.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.467+0000 7f0e70934640 1 -- 172.21.15.161:0/4176218599 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0e6c0a98c0 msgr2=0x7f0e6c0a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:15.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.467+0000 7f0e70934640 1 --2- 172.21.15.161:0/4176218599 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0e6c0a98c0 0x7f0e6c0a9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.467+0000 7f0e70934640 1 -- 172.21.15.161:0/4176218599 >> v2:172.21.15.170:3300/0 conn(0x7f0e6c06ca20 msgr2=0x7f0e6c06ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:15.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.467+0000 7f0e70934640 1 --2- 172.21.15.161:0/4176218599 >> v2:172.21.15.170:3300/0 conn(0x7f0e6c06ca20 0x7f0e6c06ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.467+0000 7f0e70934640 1 -- 172.21.15.161:0/4176218599 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e6c151e10 con 0x7f0e6c14eca0 2024-03-28T23:30:15.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.467+0000 7f0e70934640 1 -- 172.21.15.161:0/4176218599 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f0e5c001670 con 0x7f0e6c14eca0 2024-03-28T23:30:15.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.467+0000 7f0e70934640 1 -- 172.21.15.161:0/4176218599 >> v1:172.21.15.161:6789/0 conn(0x7f0e6c14eca0 legacy=0x7f0e6c0c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:15.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.467+0000 7f0e70934640 1 --2- 172.21.15.161:0/4176218599 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e58003a90 0x7f0e58003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:15.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.467+0000 7f0e70934640 1 --2- 172.21.15.161:0/4176218599 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0e6c0a98c0 0x7f0e58004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:15.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.467+0000 7f0e70934640 1 --2- 172.21.15.161:0/4176218599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e58004d40 0x7f0e580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:15.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.467+0000 7f0e70934640 1 -- 172.21.15.161:0/4176218599 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6c152460 con 0x7f0e58003a90 2024-03-28T23:30:15.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.467+0000 7f0e70934640 1 -- 172.21.15.161:0/4176218599 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6c151b30 con 0x7f0e58004d40 2024-03-28T23:30:15.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.467+0000 7f0e70934640 1 -- 172.21.15.161:0/4176218599 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0e5800bac0 con 0x7f0e6c0a98c0 2024-03-28T23:30:15.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.467+0000 7f0e71936640 1 --2- 172.21.15.161:0/4176218599 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e58003a90 0x7f0e58003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:15.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.467+0000 7f0e71135640 1 --2- 172.21.15.161:0/4176218599 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0e6c0a98c0 0x7f0e58004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:15.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.467+0000 7f0e72137640 1 --2- 172.21.15.161:0/4176218599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e58004d40 0x7f0e580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:15.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.467+0000 7f0e71936640 1 -- 172.21.15.161:0/4176218599 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0e6c0a98c0 msgr2=0x7f0e58004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:15.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.467+0000 7f0e71936640 1 --2- 172.21.15.161:0/4176218599 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0e6c0a98c0 0x7f0e58004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.467+0000 7f0e71936640 1 -- 172.21.15.161:0/4176218599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e58004d40 msgr2=0x7f0e580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:15.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.467+0000 7f0e71936640 1 --2- 172.21.15.161:0/4176218599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e58004d40 0x7f0e580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.467+0000 7f0e71936640 1 -- 172.21.15.161:0/4176218599 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e58003520 con 0x7f0e58003a90 2024-03-28T23:30:15.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.468+0000 7f0e71936640 1 --2- 172.21.15.161:0/4176218599 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e58003a90 0x7f0e58003eb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c00c210 tx=0x7f0e58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d924ad5b89535e2c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:15.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.468+0000 7f0e70934640 1 -- 172.21.15.161:0/4176218599 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c007d50 con 0x7f0e58003a90 2024-03-28T23:30:15.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.468+0000 7f0e70934640 1 -- 172.21.15.161:0/4176218599 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e5c030070 con 0x7f0e58003a90 2024-03-28T23:30:15.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.468+0000 7f0e70934640 1 -- 172.21.15.161:0/4176218599 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c005d30 con 0x7f0e58003a90 2024-03-28T23:30:15.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.468+0000 7f0e73b9a640 1 -- 172.21.15.161:0/4176218599 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e58003a90 msgr2=0x7f0e58003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:15.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.468+0000 7f0e73b9a640 1 --2- 172.21.15.161:0/4176218599 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e58003a90 0x7f0e58003eb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c00c210 tx=0x7f0e58005830 comp rx=0 tx=0).stop 2024-03-28T23:30:15.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.468+0000 7f0e72137640 1 -- 172.21.15.161:0/4176218599 reap_dead start 2024-03-28T23:30:15.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.469+0000 7f0e73b9a640 1 -- 172.21.15.161:0/4176218599 shutdown_connections 2024-03-28T23:30:15.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.469+0000 7f0e73b9a640 1 -- 172.21.15.161:0/4176218599 >> 172.21.15.161:0/4176218599 conn(0x7f0e6c0c6670 msgr2=0x7f0e6c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:15.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.469+0000 7f0e73b9a640 1 -- 172.21.15.161:0/4176218599 shutdown_connections 2024-03-28T23:30:15.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.469+0000 7f0e73b9a640 1 -- 172.21.15.161:0/4176218599 wait complete. 2024-03-28T23:30:15.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.470+0000 7f0e73b9a640 1 Processor -- start 2024-03-28T23:30:15.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.470+0000 7f0e73b9a640 1 -- start start 2024-03-28T23:30:15.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.471+0000 7f0e73b9a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e58003a90 0x7f0e6c0a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:15.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.471+0000 7f0e73b9a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e6c144a20 0x7f0e6c1528d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:15.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.471+0000 7f0e73b9a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0e6c155660 0x7f0e6c153550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:15.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.471+0000 7f0e73b9a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6c1514b0 con 0x7f0e58003a90 2024-03-28T23:30:15.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.471+0000 7f0e73b9a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6c14f080 con 0x7f0e6c144a20 2024-03-28T23:30:15.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.471+0000 7f0e73b9a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0e6c151630 con 0x7f0e6c155660 2024-03-28T23:30:15.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.471+0000 7f0e71936640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e58003a90 0x7f0e6c0a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:15.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.471+0000 7f0e71936640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e58003a90 0x7f0e6c0a9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54520/0 (socket says 172.21.15.161:54520) 2024-03-28T23:30:15.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.471+0000 7f0e71936640 1 -- 172.21.15.161:0/2167761679 learned_addr learned my addr 172.21.15.161:0/2167761679 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:15.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.472+0000 7f0e72137640 1 --2- 172.21.15.161:0/2167761679 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0e6c155660 0x7f0e6c153550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:15.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.472+0000 7f0e71135640 1 --2- 172.21.15.161:0/2167761679 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e6c144a20 0x7f0e6c1528d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:15.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.472+0000 7f0e71936640 1 -- 172.21.15.161:0/2167761679 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0e6c155660 msgr2=0x7f0e6c153550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:15.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.472+0000 7f0e71936640 1 --2- 172.21.15.161:0/2167761679 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0e6c155660 0x7f0e6c153550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.472+0000 7f0e71936640 1 -- 172.21.15.161:0/2167761679 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e6c144a20 msgr2=0x7f0e6c1528d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:15.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.472+0000 7f0e71936640 1 --2- 172.21.15.161:0/2167761679 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e6c144a20 0x7f0e6c1528d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.472+0000 7f0e71936640 1 -- 172.21.15.161:0/2167761679 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e6c1ecfe0 con 0x7f0e58003a90 2024-03-28T23:30:15.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.472+0000 7f0e72137640 1 --2- 172.21.15.161:0/2167761679 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0e6c155660 0x7f0e6c153550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:15.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.472+0000 7f0e71135640 1 --2- 172.21.15.161:0/2167761679 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e6c144a20 0x7f0e6c1528d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:15.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.472+0000 7f0e71936640 1 --2- 172.21.15.161:0/2167761679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e58003a90 0x7f0e6c0a9f40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c02c4e0 tx=0x7f0e5c001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:15.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.472+0000 7f0e627fc640 1 -- 172.21.15.161:0/2167761679 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c001c90 con 0x7f0e58003a90 2024-03-28T23:30:15.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.472+0000 7f0e627fc640 1 -- 172.21.15.161:0/2167761679 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e5c004bf0 con 0x7f0e58003a90 2024-03-28T23:30:15.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.472+0000 7f0e627fc640 1 -- 172.21.15.161:0/2167761679 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c007d50 con 0x7f0e58003a90 2024-03-28T23:30:15.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.472+0000 7f0e73b9a640 1 -- 172.21.15.161:0/2167761679 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e6c072040 con 0x7f0e58003a90 2024-03-28T23:30:15.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.472+0000 7f0e73b9a640 1 -- 172.21.15.161:0/2167761679 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e6c1ecc70 con 0x7f0e58003a90 2024-03-28T23:30:15.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.473+0000 7f0e73b9a640 1 -- 172.21.15.161:0/2167761679 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e6c04f4b0 con 0x7f0e58003a90 2024-03-28T23:30:15.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.473+0000 7f0e627fc640 1 -- 172.21.15.161:0/2167761679 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0e5c047020 con 0x7f0e58003a90 2024-03-28T23:30:15.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.474+0000 7f0e627fc640 1 --2- 172.21.15.161:0/2167761679 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0e5006b500 0x7f0e5006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:15.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.474+0000 7f0e627fc640 1 -- 172.21.15.161:0/2167761679 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0e5c0ab560 con 0x7f0e58003a90 2024-03-28T23:30:15.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.474+0000 7f0e71135640 1 --2- 172.21.15.161:0/2167761679 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0e5006b500 0x7f0e5006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:15.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.474+0000 7f0e71135640 1 --2- 172.21.15.161:0/2167761679 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0e5006b500 0x7f0e5006d9c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c146330 tx=0x7f0e68008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:15.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.478+0000 7f0e627fc640 1 -- 172.21.15.161:0/2167761679 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0e5c07a130 con 0x7f0e58003a90 2024-03-28T23:30:15.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.655+0000 7f0e73b9a640 1 -- 172.21.15.161:0/2167761679 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0e6c0c63e0 con 0x7f0e58003a90 2024-03-28T23:30:15.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.656+0000 7f0e627fc640 1 -- 172.21.15.161:0/2167761679 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7f0e5c079a30 con 0x7f0e58003a90 2024-03-28T23:30:15.657 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:15.657 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":24,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:29:50.946843+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:30:15.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.658+0000 7f0e73b9a640 1 -- 172.21.15.161:0/2167761679 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0e5006b500 msgr2=0x7f0e5006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:15.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.658+0000 7f0e73b9a640 1 --2- 172.21.15.161:0/2167761679 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0e5006b500 0x7f0e5006d9c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c146330 tx=0x7f0e68008040 comp rx=0 tx=0).stop 2024-03-28T23:30:15.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.658+0000 7f0e73b9a640 1 -- 172.21.15.161:0/2167761679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e58003a90 msgr2=0x7f0e6c0a9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:15.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.658+0000 7f0e73b9a640 1 --2- 172.21.15.161:0/2167761679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e58003a90 0x7f0e6c0a9f40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c02c4e0 tx=0x7f0e5c001a40 comp rx=0 tx=0).stop 2024-03-28T23:30:15.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.658+0000 7f0e73b9a640 1 -- 172.21.15.161:0/2167761679 shutdown_connections 2024-03-28T23:30:15.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.658+0000 7f0e73b9a640 1 --2- 172.21.15.161:0/2167761679 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0e5006b500 0x7f0e5006d9c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.658+0000 7f0e73b9a640 1 --2- 172.21.15.161:0/2167761679 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0e6c155660 0x7f0e6c153550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.658+0000 7f0e73b9a640 1 --2- 172.21.15.161:0/2167761679 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0e6c144a20 0x7f0e6c1528d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.658+0000 7f0e73b9a640 1 --2- 172.21.15.161:0/2167761679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e58003a90 0x7f0e6c0a9f40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.658+0000 7f0e73b9a640 1 -- 172.21.15.161:0/2167761679 >> 172.21.15.161:0/2167761679 conn(0x7f0e6c0c6670 msgr2=0x7f0e6c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:15.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.658+0000 7f0e73b9a640 1 -- 172.21.15.161:0/2167761679 shutdown_connections 2024-03-28T23:30:15.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.658+0000 7f0e73b9a640 1 -- 172.21.15.161:0/2167761679 wait complete. 2024-03-28T23:30:15.671 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:29:44.021375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:29:44.872102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T23:29:48.591063+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T23:30:15.672 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:30:15.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.776+0000 7f69f3e9d640 1 Processor -- start 2024-03-28T23:30:15.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.776+0000 7f69f3e9d640 1 -- start start 2024-03-28T23:30:15.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.777+0000 7f69f3e9d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f69ec06efb0 0x7f69ec14fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:15.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.777+0000 7f69f3e9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f69ec1500b0 0x7f69ec0aac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:15.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.777+0000 7f69f3e9d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69ec0acaf0 con 0x7f69ec0a9480 2024-03-28T23:30:15.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.777+0000 7f69f3e9d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f69ec15a490 con 0x7f69ec1500b0 2024-03-28T23:30:15.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.777+0000 7f69f3e9d640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f69ec159910 con 0x7f69ec06efb0 2024-03-28T23:30:15.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.777+0000 7f69f243a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f69ec1500b0 0x7f69ec0aac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:15.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.777+0000 7f69f1438640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f69ec0a9480 0x7f69ec0aa560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59730/0 (socket says 172.21.15.161:59730) 2024-03-28T23:30:15.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.777+0000 7f69f1438640 1 -- 172.21.15.161:0/2321259765 learned_addr learned my addr 172.21.15.161:0/2321259765 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:15.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.777+0000 7f69f1c39640 1 --2- 172.21.15.161:0/2321259765 >> v2:172.21.15.170:3300/0 conn(0x7f69ec06efb0 0x7f69ec14fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:15.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.778+0000 7f69f243a640 1 -- 172.21.15.161:0/2321259765 >> v2:172.21.15.170:3300/0 conn(0x7f69ec06efb0 msgr2=0x7f69ec14fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:15.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.778+0000 7f69f243a640 1 --2- 172.21.15.161:0/2321259765 >> v2:172.21.15.170:3300/0 conn(0x7f69ec06efb0 0x7f69ec14fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.778+0000 7f69f243a640 1 -- 172.21.15.161:0/2321259765 >> v1:172.21.15.161:6789/0 conn(0x7f69ec0a9480 legacy=0x7f69ec0aa560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:15.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.778+0000 7f69f0c37640 1 -- 172.21.15.161:0/2321259765 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2733895765 0 0) 0x7f69ec0acaf0 con 0x7f69ec0a9480 2024-03-28T23:30:15.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.778+0000 7f69f243a640 1 -- 172.21.15.161:0/2321259765 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69ec159bf0 con 0x7f69ec1500b0 2024-03-28T23:30:15.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.778+0000 7f69f1c39640 1 --2- 172.21.15.161:0/2321259765 >> v2:172.21.15.170:3300/0 conn(0x7f69ec06efb0 0x7f69ec14fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:15.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.778+0000 7f69f243a640 1 --2- 172.21.15.161:0/2321259765 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69ec1500b0 0x7f69ec0aac70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f69ec0c6dd0 tx=0x7f69e402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b18ced3f7026d62b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:15.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.778+0000 7f69f0c37640 1 -- 172.21.15.161:0/2321259765 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e4039070 con 0x7f69ec1500b0 2024-03-28T23:30:15.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.778+0000 7f69f0c37640 1 -- 172.21.15.161:0/2321259765 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69e402cc80 con 0x7f69ec1500b0 2024-03-28T23:30:15.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.778+0000 7f69f0c37640 1 -- 172.21.15.161:0/2321259765 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e4034660 con 0x7f69ec1500b0 2024-03-28T23:30:15.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.779+0000 7f69f3e9d640 1 -- 172.21.15.161:0/2321259765 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69ec1500b0 msgr2=0x7f69ec0aac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:15.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.779+0000 7f69f3e9d640 1 --2- 172.21.15.161:0/2321259765 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69ec1500b0 0x7f69ec0aac70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f69ec0c6dd0 tx=0x7f69e402bed0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.779+0000 7f69f3e9d640 1 -- 172.21.15.161:0/2321259765 shutdown_connections 2024-03-28T23:30:15.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.779+0000 7f69f3e9d640 1 --2- 172.21.15.161:0/2321259765 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69ec1500b0 0x7f69ec0aac70 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.779+0000 7f69f3e9d640 1 --2- 172.21.15.161:0/2321259765 >> v2:172.21.15.170:3300/0 conn(0x7f69ec06efb0 0x7f69ec14fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.779+0000 7f69f3e9d640 1 -- 172.21.15.161:0/2321259765 >> 172.21.15.161:0/2321259765 conn(0x7f69ec0c4e70 msgr2=0x7f69ec0c5280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:15.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.779+0000 7f69f3e9d640 1 -- 172.21.15.161:0/2321259765 shutdown_connections 2024-03-28T23:30:15.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.779+0000 7f69f3e9d640 1 -- 172.21.15.161:0/2321259765 wait complete. 2024-03-28T23:30:15.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.780+0000 7f69f3e9d640 1 Processor -- start 2024-03-28T23:30:15.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.780+0000 7f69f3e9d640 1 -- start start 2024-03-28T23:30:15.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.781+0000 7f69f3e9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69ec06efb0 0x7f69ec1568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:15.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.781+0000 7f69f3e9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69ec0a9480 0x7f69ec1f2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:15.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.781+0000 7f69f3e9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69ec1500b0 0x7f69ec154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:15.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.781+0000 7f69f3e9d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f69ec0ab1b0 con 0x7f69ec0a9480 2024-03-28T23:30:15.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.781+0000 7f69f3e9d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f69ec15a1b0 con 0x7f69ec06efb0 2024-03-28T23:30:15.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.781+0000 7f69f3e9d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f69ec1570b0 con 0x7f69ec1500b0 2024-03-28T23:30:15.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.781+0000 7f69f1438640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69ec0a9480 0x7f69ec1f2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:15.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.781+0000 7f69f1438640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69ec0a9480 0x7f69ec1f2070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54546/0 (socket says 172.21.15.161:54546) 2024-03-28T23:30:15.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.781+0000 7f69f1438640 1 -- 172.21.15.161:0/638043777 learned_addr learned my addr 172.21.15.161:0/638043777 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:15.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.781+0000 7f69f1c39640 1 --2- 172.21.15.161:0/638043777 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69ec06efb0 0x7f69ec1568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:15.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.781+0000 7f69f243a640 1 --2- 172.21.15.161:0/638043777 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69ec1500b0 0x7f69ec154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:15.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.781+0000 7f69f1438640 1 -- 172.21.15.161:0/638043777 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69ec1500b0 msgr2=0x7f69ec154d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:15.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.781+0000 7f69f1438640 1 --2- 172.21.15.161:0/638043777 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69ec1500b0 0x7f69ec154d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.781+0000 7f69f1438640 1 -- 172.21.15.161:0/638043777 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69ec06efb0 msgr2=0x7f69ec1568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:15.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.781+0000 7f69f1438640 1 --2- 172.21.15.161:0/638043777 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69ec06efb0 0x7f69ec1568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.781+0000 7f69f1438640 1 -- 172.21.15.161:0/638043777 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69ec15d800 con 0x7f69ec0a9480 2024-03-28T23:30:15.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.781+0000 7f69f1c39640 1 --2- 172.21.15.161:0/638043777 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69ec06efb0 0x7f69ec1568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:15.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.781+0000 7f69f1438640 1 --2- 172.21.15.161:0/638043777 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69ec0a9480 0x7f69ec1f2070 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f69dc00c4d0 tx=0x7f69dc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:15.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.782+0000 7f69e27fc640 1 -- 172.21.15.161:0/638043777 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69dc018070 con 0x7f69ec0a9480 2024-03-28T23:30:15.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.782+0000 7f69e27fc640 1 -- 172.21.15.161:0/638043777 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69dc007500 con 0x7f69ec0a9480 2024-03-28T23:30:15.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.782+0000 7f69e27fc640 1 -- 172.21.15.161:0/638043777 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69dc004a20 con 0x7f69ec0a9480 2024-03-28T23:30:15.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.782+0000 7f69f3e9d640 1 -- 172.21.15.161:0/638043777 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69ec15c220 con 0x7f69ec0a9480 2024-03-28T23:30:15.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.782+0000 7f69f3e9d640 1 -- 172.21.15.161:0/638043777 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69ec15c4c0 con 0x7f69ec0a9480 2024-03-28T23:30:15.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.783+0000 7f69f3e9d640 1 -- 172.21.15.161:0/638043777 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69ec04f4b0 con 0x7f69ec0a9480 2024-03-28T23:30:15.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.783+0000 7f69e27fc640 1 -- 172.21.15.161:0/638043777 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f69dc01b070 con 0x7f69ec0a9480 2024-03-28T23:30:15.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.784+0000 7f69e27fc640 1 --2- 172.21.15.161:0/638043777 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f69d006b500 0x7f69d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:15.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.784+0000 7f69e27fc640 1 -- 172.21.15.161:0/638043777 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f69dc089780 con 0x7f69ec0a9480 2024-03-28T23:30:15.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.784+0000 7f69f1c39640 1 --2- 172.21.15.161:0/638043777 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f69d006b500 0x7f69d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:15.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.784+0000 7f69f1c39640 1 --2- 172.21.15.161:0/638043777 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f69d006b500 0x7f69d006d9c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f69e8006fd0 tx=0x7f69e8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:15.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.788+0000 7f69e27fc640 1 -- 172.21.15.161:0/638043777 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f69dc058350 con 0x7f69ec0a9480 2024-03-28T23:30:15.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.964+0000 7f69f3e9d640 1 -- 172.21.15.161:0/638043777 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f69ec0c6260 con 0x7f69ec0a9480 2024-03-28T23:30:15.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.964+0000 7f69e27fc640 1 -- 172.21.15.161:0/638043777 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f69dc057c50 con 0x7f69ec0a9480 2024-03-28T23:30:15.965 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:30:15.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.966+0000 7f69f3e9d640 1 -- 172.21.15.161:0/638043777 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f69d006b500 msgr2=0x7f69d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:15.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.966+0000 7f69f3e9d640 1 --2- 172.21.15.161:0/638043777 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f69d006b500 0x7f69d006d9c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f69e8006fd0 tx=0x7f69e8008040 comp rx=0 tx=0).stop 2024-03-28T23:30:15.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.966+0000 7f69f3e9d640 1 -- 172.21.15.161:0/638043777 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69ec0a9480 msgr2=0x7f69ec1f2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:15.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.966+0000 7f69f3e9d640 1 --2- 172.21.15.161:0/638043777 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69ec0a9480 0x7f69ec1f2070 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f69dc00c4d0 tx=0x7f69dc00c990 comp rx=0 tx=0).stop 2024-03-28T23:30:15.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.966+0000 7f69f3e9d640 1 -- 172.21.15.161:0/638043777 shutdown_connections 2024-03-28T23:30:15.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.966+0000 7f69f3e9d640 1 --2- 172.21.15.161:0/638043777 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f69d006b500 0x7f69d006d9c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.966+0000 7f69f3e9d640 1 --2- 172.21.15.161:0/638043777 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69ec0a9480 0x7f69ec1f2070 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.966+0000 7f69f3e9d640 1 --2- 172.21.15.161:0/638043777 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69ec1500b0 0x7f69ec154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.966+0000 7f69f3e9d640 1 --2- 172.21.15.161:0/638043777 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69ec06efb0 0x7f69ec1568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:15.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.966+0000 7f69f3e9d640 1 -- 172.21.15.161:0/638043777 >> 172.21.15.161:0/638043777 conn(0x7f69ec0c4e70 msgr2=0x7f69ec157be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:15.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.966+0000 7f69f3e9d640 1 -- 172.21.15.161:0/638043777 shutdown_connections 2024-03-28T23:30:15.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:15.966+0000 7f69f3e9d640 1 -- 172.21.15.161:0/638043777 wait complete. 2024-03-28T23:30:15.978 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:30:16.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.085+0000 7f0f2c7b5640 1 Processor -- start 2024-03-28T23:30:16.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.085+0000 7f0f2c7b5640 1 -- start start 2024-03-28T23:30:16.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.087+0000 7f0f2c7b5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0f24156080 0x7f0f240c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:16.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.087+0000 7f0f2c7b5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0f2406ff90 0x7f0f24156b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:16.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.087+0000 7f0f2c7b5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f24072790 con 0x7f0f24156460 2024-03-28T23:30:16.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.087+0000 7f0f2c7b5640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0f240c67b0 con 0x7f0f24156080 2024-03-28T23:30:16.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.087+0000 7f0f2c7b5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f0f240a7ea0 con 0x7f0f2406ff90 2024-03-28T23:30:16.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.087+0000 7f0f2a551640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0f24156080 0x7f0f240c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:16.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.087+0000 7f0f2a551640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0f24156080 0x7f0f240c5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54550/0 (socket says 172.21.15.161:54550) 2024-03-28T23:30:16.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.087+0000 7f0f2a551640 1 -- 172.21.15.161:0/3998574568 learned_addr learned my addr 172.21.15.161:0/3998574568 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:16.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.087+0000 7f0f29d50640 1 --2- 172.21.15.161:0/3998574568 >> v2:172.21.15.170:3300/0 conn(0x7f0f2406ff90 0x7f0f24156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:16.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.087+0000 7f0f2a551640 1 -- 172.21.15.161:0/3998574568 >> v2:172.21.15.170:3300/0 conn(0x7f0f2406ff90 msgr2=0x7f0f24156b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:16.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.087+0000 7f0f2a551640 1 --2- 172.21.15.161:0/3998574568 >> v2:172.21.15.170:3300/0 conn(0x7f0f2406ff90 0x7f0f24156b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.087+0000 7f0f2a551640 1 -- 172.21.15.161:0/3998574568 >> v1:172.21.15.170:6789/0 conn(0x7f0f24156460 legacy=0x7f0f241570c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:16.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.087+0000 7f0f2a551640 1 -- 172.21.15.161:0/3998574568 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f240a8180 con 0x7f0f24156080 2024-03-28T23:30:16.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.087+0000 7f0f29d50640 1 --2- 172.21.15.161:0/3998574568 >> v2:172.21.15.170:3300/0 conn(0x7f0f2406ff90 0x7f0f24156b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:16.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.088+0000 7f0f2a551640 1 --2- 172.21.15.161:0/3998574568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f24156080 0x7f0f240c5340 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0f2406f870 tx=0x7f0f1c02bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcc7d369d2f63011 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:16.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.088+0000 7f0f2954f640 1 -- 172.21.15.161:0/3998574568 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c02c960 con 0x7f0f24156080 2024-03-28T23:30:16.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.088+0000 7f0f2954f640 1 -- 172.21.15.161:0/3998574568 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f1c02cb00 con 0x7f0f24156080 2024-03-28T23:30:16.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.088+0000 7f0f2954f640 1 -- 172.21.15.161:0/3998574568 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c0343f0 con 0x7f0f24156080 2024-03-28T23:30:16.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.089+0000 7f0f2c7b5640 1 -- 172.21.15.161:0/3998574568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f24156080 msgr2=0x7f0f240c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:16.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.089+0000 7f0f2c7b5640 1 --2- 172.21.15.161:0/3998574568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f24156080 0x7f0f240c5340 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0f2406f870 tx=0x7f0f1c02bdc0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.089+0000 7f0f2c7b5640 1 -- 172.21.15.161:0/3998574568 shutdown_connections 2024-03-28T23:30:16.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.089+0000 7f0f2c7b5640 1 --2- 172.21.15.161:0/3998574568 >> v2:172.21.15.170:3300/0 conn(0x7f0f2406ff90 0x7f0f24156b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.089+0000 7f0f2c7b5640 1 --2- 172.21.15.161:0/3998574568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f24156080 0x7f0f240c5340 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.089+0000 7f0f2c7b5640 1 -- 172.21.15.161:0/3998574568 >> 172.21.15.161:0/3998574568 conn(0x7f0f2406eba0 msgr2=0x7f0f2406b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:16.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.089+0000 7f0f2c7b5640 1 -- 172.21.15.161:0/3998574568 shutdown_connections 2024-03-28T23:30:16.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.089+0000 7f0f2c7b5640 1 -- 172.21.15.161:0/3998574568 wait complete. 2024-03-28T23:30:16.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.090+0000 7f0f2c7b5640 1 Processor -- start 2024-03-28T23:30:16.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.091+0000 7f0f2c7b5640 1 -- start start 2024-03-28T23:30:16.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.092+0000 7f0f2c7b5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f2406ff90 0x7f0f240be360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:16.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.092+0000 7f0f2c7b5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f24156080 0x7f0f240a8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:16.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.092+0000 7f0f2c7b5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f24156460 0x7f0f2414de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:16.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.092+0000 7f0f2c7b5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0f24070620 con 0x7f0f24156460 2024-03-28T23:30:16.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.092+0000 7f0f2c7b5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0f240704a0 con 0x7f0f24156080 2024-03-28T23:30:16.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.092+0000 7f0f2c7b5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0f240707a0 con 0x7f0f2406ff90 2024-03-28T23:30:16.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.092+0000 7f0f2ad52640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f24156460 0x7f0f2414de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:16.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.092+0000 7f0f2ad52640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f24156460 0x7f0f2414de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54552/0 (socket says 172.21.15.161:54552) 2024-03-28T23:30:16.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.092+0000 7f0f2ad52640 1 -- 172.21.15.161:0/1970917716 learned_addr learned my addr 172.21.15.161:0/1970917716 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:16.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.092+0000 7f0f2a551640 1 --2- 172.21.15.161:0/1970917716 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f2406ff90 0x7f0f240be360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:16.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.092+0000 7f0f29d50640 1 --2- 172.21.15.161:0/1970917716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f24156080 0x7f0f240a8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:16.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.092+0000 7f0f2ad52640 1 -- 172.21.15.161:0/1970917716 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f2406ff90 msgr2=0x7f0f240be360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:16.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.092+0000 7f0f2ad52640 1 --2- 172.21.15.161:0/1970917716 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f2406ff90 0x7f0f240be360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.092+0000 7f0f2ad52640 1 -- 172.21.15.161:0/1970917716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f24156080 msgr2=0x7f0f240a8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:16.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.092+0000 7f0f2ad52640 1 --2- 172.21.15.161:0/1970917716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f24156080 0x7f0f240a8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.092+0000 7f0f2ad52640 1 -- 172.21.15.161:0/1970917716 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f2414d060 con 0x7f0f24156460 2024-03-28T23:30:16.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.092+0000 7f0f2a551640 1 --2- 172.21.15.161:0/1970917716 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f2406ff90 0x7f0f240be360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:16.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.092+0000 7f0f2ad52640 1 --2- 172.21.15.161:0/1970917716 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f24156460 0x7f0f2414de20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0f2000c360 tx=0x7f0f2000c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:16.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.092+0000 7f0f1affd640 1 -- 172.21.15.161:0/1970917716 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f20017070 con 0x7f0f24156460 2024-03-28T23:30:16.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.092+0000 7f0f1affd640 1 -- 172.21.15.161:0/1970917716 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f200035f0 con 0x7f0f24156460 2024-03-28T23:30:16.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.092+0000 7f0f2c7b5640 1 -- 172.21.15.161:0/1970917716 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f2414d340 con 0x7f0f24156460 2024-03-28T23:30:16.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.092+0000 7f0f1affd640 1 -- 172.21.15.161:0/1970917716 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f200039e0 con 0x7f0f24156460 2024-03-28T23:30:16.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.092+0000 7f0f2c7b5640 1 -- 172.21.15.161:0/1970917716 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f2414d900 con 0x7f0f24156460 2024-03-28T23:30:16.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.094+0000 7f0f2c7b5640 1 -- 172.21.15.161:0/1970917716 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f2404f4b0 con 0x7f0f24156460 2024-03-28T23:30:16.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.094+0000 7f0f1affd640 1 -- 172.21.15.161:0/1970917716 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0f2001a070 con 0x7f0f24156460 2024-03-28T23:30:16.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.094+0000 7f0f1affd640 1 --2- 172.21.15.161:0/1970917716 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0f0806b4d0 0x7f0f0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:16.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.094+0000 7f0f1affd640 1 -- 172.21.15.161:0/1970917716 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0f20089870 con 0x7f0f24156460 2024-03-28T23:30:16.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.094+0000 7f0f2a551640 1 --2- 172.21.15.161:0/1970917716 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0f0806b4d0 0x7f0f0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:16.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.095+0000 7f0f2a551640 1 --2- 172.21.15.161:0/1970917716 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0f0806b4d0 0x7f0f0806d990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c035000 tx=0x7f0f1c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:16.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.098+0000 7f0f1affd640 1 -- 172.21.15.161:0/1970917716 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0f20058440 con 0x7f0f24156460 2024-03-28T23:30:16.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.287+0000 7f0f2c7b5640 1 -- 172.21.15.161:0/1970917716 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0f241544e0 con 0x7f0f24156460 2024-03-28T23:30:16.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.287+0000 7f0f1affd640 1 -- 172.21.15.161:0/1970917716 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f0f20057d40 con 0x7f0f24156460 2024-03-28T23:30:16.288 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:30:16.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.290+0000 7f0f2c7b5640 1 -- 172.21.15.161:0/1970917716 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0f0806b4d0 msgr2=0x7f0f0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:16.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.290+0000 7f0f2c7b5640 1 --2- 172.21.15.161:0/1970917716 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0f0806b4d0 0x7f0f0806d990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c035000 tx=0x7f0f1c030000 comp rx=0 tx=0).stop 2024-03-28T23:30:16.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.290+0000 7f0f2c7b5640 1 -- 172.21.15.161:0/1970917716 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f24156460 msgr2=0x7f0f2414de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:16.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.290+0000 7f0f2c7b5640 1 --2- 172.21.15.161:0/1970917716 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f24156460 0x7f0f2414de20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0f2000c360 tx=0x7f0f2000c820 comp rx=0 tx=0).stop 2024-03-28T23:30:16.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.290+0000 7f0f2c7b5640 1 -- 172.21.15.161:0/1970917716 shutdown_connections 2024-03-28T23:30:16.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.290+0000 7f0f2c7b5640 1 --2- 172.21.15.161:0/1970917716 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0f0806b4d0 0x7f0f0806d990 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.290+0000 7f0f2c7b5640 1 --2- 172.21.15.161:0/1970917716 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f24156460 0x7f0f2414de20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.290+0000 7f0f2c7b5640 1 --2- 172.21.15.161:0/1970917716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f24156080 0x7f0f240a8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.290+0000 7f0f2c7b5640 1 --2- 172.21.15.161:0/1970917716 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f2406ff90 0x7f0f240be360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.290+0000 7f0f2c7b5640 1 -- 172.21.15.161:0/1970917716 >> 172.21.15.161:0/1970917716 conn(0x7f0f2406eba0 msgr2=0x7f0f24156b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:16.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.290+0000 7f0f2c7b5640 1 -- 172.21.15.161:0/1970917716 shutdown_connections 2024-03-28T23:30:16.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.290+0000 7f0f2c7b5640 1 -- 172.21.15.161:0/1970917716 wait complete. 2024-03-28T23:30:16.301 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:30:16.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.405+0000 7f72ca0e1640 1 Processor -- start 2024-03-28T23:30:16.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.405+0000 7f72ca0e1640 1 -- start start 2024-03-28T23:30:16.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.406+0000 7f72ca0e1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f72c406bb60 0x7f72c40c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:16.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.406+0000 7f72ca0e1640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72c4152db0 con 0x7f72c40c6750 2024-03-28T23:30:16.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.406+0000 7f72ca0e1640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72c40c7210 con 0x7f72c41529d0 2024-03-28T23:30:16.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.406+0000 7f72ca0e1640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f72c40c6c50 con 0x7f72c406bb60 2024-03-28T23:30:16.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.406+0000 7f72c3fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f72c406bb60 0x7f72c40c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:16.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.406+0000 7f72c2ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f72c40c6750 0x7f72c40c54f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59742/0 (socket says 172.21.15.161:59742) 2024-03-28T23:30:16.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.406+0000 7f72c2ffd640 1 -- 172.21.15.161:0/943080535 learned_addr learned my addr 172.21.15.161:0/943080535 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:16.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.407+0000 7f72c3fff640 1 -- 172.21.15.161:0/943080535 >> v1:172.21.15.170:6789/0 conn(0x7f72c41529d0 legacy=0x7f72c4155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:16.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.407+0000 7f72c27fc640 1 -- 172.21.15.161:0/943080535 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3104116735 0 0) 0x7f72c4152db0 con 0x7f72c40c6750 2024-03-28T23:30:16.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.407+0000 7f72c3fff640 1 -- 172.21.15.161:0/943080535 >> v1:172.21.15.161:6789/0 conn(0x7f72c40c6750 legacy=0x7f72c40c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:16.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.407+0000 7f72c3fff640 1 -- 172.21.15.161:0/943080535 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72c40c6f30 con 0x7f72c406bb60 2024-03-28T23:30:16.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.408+0000 7f72c3fff640 1 --2- 172.21.15.161:0/943080535 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72c406bb60 0x7f72c40c88b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f72bc009d90 tx=0x7f72bc02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ad5ef6dfb5d3f72 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:16.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.408+0000 7f72c27fc640 1 -- 172.21.15.161:0/943080535 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc039070 con 0x7f72c406bb60 2024-03-28T23:30:16.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.408+0000 7f72c27fc640 1 -- 172.21.15.161:0/943080535 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72bc02cd00 con 0x7f72c406bb60 2024-03-28T23:30:16.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.408+0000 7f72c27fc640 1 -- 172.21.15.161:0/943080535 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc034660 con 0x7f72c406bb60 2024-03-28T23:30:16.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.408+0000 7f72ca0e1640 1 -- 172.21.15.161:0/943080535 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72c406bb60 msgr2=0x7f72c40c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:16.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.408+0000 7f72ca0e1640 1 --2- 172.21.15.161:0/943080535 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72c406bb60 0x7f72c40c88b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f72bc009d90 tx=0x7f72bc02bf50 comp rx=0 tx=0).stop 2024-03-28T23:30:16.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.408+0000 7f72ca0e1640 1 -- 172.21.15.161:0/943080535 shutdown_connections 2024-03-28T23:30:16.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.409+0000 7f72ca0e1640 1 --2- 172.21.15.161:0/943080535 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72c406bb60 0x7f72c40c88b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.409+0000 7f72ca0e1640 1 -- 172.21.15.161:0/943080535 >> 172.21.15.161:0/943080535 conn(0x7f72c40c7430 msgr2=0x7f72c40c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:16.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.409+0000 7f72ca0e1640 1 -- 172.21.15.161:0/943080535 shutdown_connections 2024-03-28T23:30:16.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.409+0000 7f72ca0e1640 1 -- 172.21.15.161:0/943080535 wait complete. 2024-03-28T23:30:16.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.410+0000 7f72ca0e1640 1 Processor -- start 2024-03-28T23:30:16.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.410+0000 7f72ca0e1640 1 -- start start 2024-03-28T23:30:16.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.410+0000 7f72ca0e1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72c406bb60 0x7f72c4158af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:16.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.410+0000 7f72ca0e1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72c40c6750 0x7f72c40afc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:16.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.410+0000 7f72ca0e1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72c41529d0 0x7f72c40b0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:16.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.410+0000 7f72ca0e1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f72c40aa160 con 0x7f72c40c6750 2024-03-28T23:30:16.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.410+0000 7f72ca0e1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f72c4155710 con 0x7f72c406bb60 2024-03-28T23:30:16.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.410+0000 7f72ca0e1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f72c40c5c00 con 0x7f72c41529d0 2024-03-28T23:30:16.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.411+0000 7f72c2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72c40c6750 0x7f72c40afc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:16.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.411+0000 7f72c2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72c40c6750 0x7f72c40afc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54566/0 (socket says 172.21.15.161:54566) 2024-03-28T23:30:16.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.411+0000 7f72c2ffd640 1 -- 172.21.15.161:0/414208300 learned_addr learned my addr 172.21.15.161:0/414208300 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:16.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.411+0000 7f72c37fe640 1 --2- 172.21.15.161:0/414208300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72c406bb60 0x7f72c4158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:16.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.411+0000 7f72c3fff640 1 --2- 172.21.15.161:0/414208300 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72c41529d0 0x7f72c40b0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:16.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.411+0000 7f72c2ffd640 1 -- 172.21.15.161:0/414208300 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72c41529d0 msgr2=0x7f72c40b0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:16.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.411+0000 7f72c2ffd640 1 --2- 172.21.15.161:0/414208300 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72c41529d0 0x7f72c40b0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.411+0000 7f72c2ffd640 1 -- 172.21.15.161:0/414208300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72c406bb60 msgr2=0x7f72c4158af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:16.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.411+0000 7f72c2ffd640 1 --2- 172.21.15.161:0/414208300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72c406bb60 0x7f72c4158af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.411+0000 7f72c2ffd640 1 -- 172.21.15.161:0/414208300 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72c414ec90 con 0x7f72c40c6750 2024-03-28T23:30:16.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.411+0000 7f72c37fe640 1 --2- 172.21.15.161:0/414208300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72c406bb60 0x7f72c4158af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:16.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.411+0000 7f72c2ffd640 1 --2- 172.21.15.161:0/414208300 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72c40c6750 0x7f72c40afc10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f72b800c4a0 tx=0x7f72b800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:16.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.411+0000 7f72abfff640 1 -- 172.21.15.161:0/414208300 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b8018070 con 0x7f72c40c6750 2024-03-28T23:30:16.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.411+0000 7f72abfff640 1 -- 172.21.15.161:0/414208300 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72b80074e0 con 0x7f72c40c6750 2024-03-28T23:30:16.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.411+0000 7f72abfff640 1 -- 172.21.15.161:0/414208300 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b8004a20 con 0x7f72c40c6750 2024-03-28T23:30:16.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.411+0000 7f72ca0e1640 1 -- 172.21.15.161:0/414208300 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72c41eba50 con 0x7f72c40c6750 2024-03-28T23:30:16.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.411+0000 7f72ca0e1640 1 -- 172.21.15.161:0/414208300 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72c4159030 con 0x7f72c40c6750 2024-03-28T23:30:16.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.412+0000 7f72ca0e1640 1 -- 172.21.15.161:0/414208300 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72c4068f90 con 0x7f72c40c6750 2024-03-28T23:30:16.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.413+0000 7f72abfff640 1 -- 172.21.15.161:0/414208300 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f72b801b070 con 0x7f72c40c6750 2024-03-28T23:30:16.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.413+0000 7f72abfff640 1 --2- 172.21.15.161:0/414208300 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f72b006b4d0 0x7f72b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:16.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.413+0000 7f72abfff640 1 -- 172.21.15.161:0/414208300 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f72b80896e0 con 0x7f72c40c6750 2024-03-28T23:30:16.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.413+0000 7f72c37fe640 1 --2- 172.21.15.161:0/414208300 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f72b006b4d0 0x7f72b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:16.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.414+0000 7f72c37fe640 1 --2- 172.21.15.161:0/414208300 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f72b006b4d0 0x7f72b006d990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f72c4159910 tx=0x7f72b4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:16.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.418+0000 7f72abfff640 1 -- 172.21.15.161:0/414208300 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f72c4068f90 con 0x7f72c40c6750 2024-03-28T23:30:16.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.591+0000 7f72ca0e1640 1 -- 172.21.15.161:0/414208300 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f72c4151460 con 0x7f72c40c6750 2024-03-28T23:30:16.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.591+0000 7f72abfff640 1 -- 172.21.15.161:0/414208300 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f72c4151460 con 0x7f72c40c6750 2024-03-28T23:30:16.592 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:30:16.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.593+0000 7f72ca0e1640 1 -- 172.21.15.161:0/414208300 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f72b006b4d0 msgr2=0x7f72b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:16.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.593+0000 7f72ca0e1640 1 --2- 172.21.15.161:0/414208300 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f72b006b4d0 0x7f72b006d990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f72c4159910 tx=0x7f72b4009040 comp rx=0 tx=0).stop 2024-03-28T23:30:16.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.593+0000 7f72ca0e1640 1 -- 172.21.15.161:0/414208300 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72c40c6750 msgr2=0x7f72c40afc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:16.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.593+0000 7f72ca0e1640 1 --2- 172.21.15.161:0/414208300 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72c40c6750 0x7f72c40afc10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f72b800c4a0 tx=0x7f72b800c960 comp rx=0 tx=0).stop 2024-03-28T23:30:16.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.593+0000 7f72ca0e1640 1 -- 172.21.15.161:0/414208300 shutdown_connections 2024-03-28T23:30:16.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.593+0000 7f72ca0e1640 1 --2- 172.21.15.161:0/414208300 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f72b006b4d0 0x7f72b006d990 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.593+0000 7f72ca0e1640 1 --2- 172.21.15.161:0/414208300 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72c40c6750 0x7f72c40afc10 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.593+0000 7f72ca0e1640 1 --2- 172.21.15.161:0/414208300 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72c41529d0 0x7f72c40b0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.593+0000 7f72ca0e1640 1 --2- 172.21.15.161:0/414208300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72c406bb60 0x7f72c4158af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.593+0000 7f72ca0e1640 1 -- 172.21.15.161:0/414208300 >> 172.21.15.161:0/414208300 conn(0x7f72c40c7430 msgr2=0x7f72c40c88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:16.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.593+0000 7f72ca0e1640 1 -- 172.21.15.161:0/414208300 shutdown_connections 2024-03-28T23:30:16.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.593+0000 7f72ca0e1640 1 -- 172.21.15.161:0/414208300 wait complete. 2024-03-28T23:30:16.605 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T23:30:16.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.710+0000 7fde6dfc7640 1 Processor -- start 2024-03-28T23:30:16.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.710+0000 7fde6dfc7640 1 -- start start 2024-03-28T23:30:16.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.710+0000 7fde6dfc7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fde68156080 0x7fde680c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:16.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.710+0000 7fde6dfc7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fde6806ff90 0x7fde68156b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:16.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.710+0000 7fde6dfc7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde68072790 con 0x7fde68156460 2024-03-28T23:30:16.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.710+0000 7fde6dfc7640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fde680c67b0 con 0x7fde6806ff90 2024-03-28T23:30:16.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.710+0000 7fde6dfc7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fde680a7ea0 con 0x7fde68156080 2024-03-28T23:30:16.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.711+0000 7fde66ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fde6806ff90 0x7fde68156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:16.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.711+0000 7fde66ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fde6806ff90 0x7fde68156b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54570/0 (socket says 172.21.15.161:54570) 2024-03-28T23:30:16.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.711+0000 7fde66ffd640 1 -- 172.21.15.161:0/1564511505 learned_addr learned my addr 172.21.15.161:0/1564511505 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:16.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.711+0000 7fde677fe640 1 --2- 172.21.15.161:0/1564511505 >> v2:172.21.15.170:3300/0 conn(0x7fde68156080 0x7fde680c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:16.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.711+0000 7fde66ffd640 1 -- 172.21.15.161:0/1564511505 >> v2:172.21.15.170:3300/0 conn(0x7fde68156080 msgr2=0x7fde680c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:16.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.711+0000 7fde66ffd640 1 --2- 172.21.15.161:0/1564511505 >> v2:172.21.15.170:3300/0 conn(0x7fde68156080 0x7fde680c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.711+0000 7fde66ffd640 1 -- 172.21.15.161:0/1564511505 >> v1:172.21.15.170:6789/0 conn(0x7fde68156460 legacy=0x7fde681570c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:16.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.711+0000 7fde66ffd640 1 -- 172.21.15.161:0/1564511505 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde680a8180 con 0x7fde6806ff90 2024-03-28T23:30:16.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.712+0000 7fde66ffd640 1 --2- 172.21.15.161:0/1564511505 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde6806ff90 0x7fde68156b80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fde680ab6a0 tx=0x7fde5c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef9e24f8b0a5a141 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:16.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.712+0000 7fde667fc640 1 -- 172.21.15.161:0/1564511505 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c038070 con 0x7fde6806ff90 2024-03-28T23:30:16.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.712+0000 7fde667fc640 1 -- 172.21.15.161:0/1564511505 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde5c02f050 con 0x7fde6806ff90 2024-03-28T23:30:16.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.712+0000 7fde667fc640 1 -- 172.21.15.161:0/1564511505 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c031970 con 0x7fde6806ff90 2024-03-28T23:30:16.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.712+0000 7fde6dfc7640 1 -- 172.21.15.161:0/1564511505 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde6806ff90 msgr2=0x7fde68156b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:16.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.712+0000 7fde6dfc7640 1 --2- 172.21.15.161:0/1564511505 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde6806ff90 0x7fde68156b80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fde680ab6a0 tx=0x7fde5c02b7f0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.712+0000 7fde6dfc7640 1 -- 172.21.15.161:0/1564511505 shutdown_connections 2024-03-28T23:30:16.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.712+0000 7fde6dfc7640 1 --2- 172.21.15.161:0/1564511505 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde6806ff90 0x7fde68156b80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.712+0000 7fde6dfc7640 1 --2- 172.21.15.161:0/1564511505 >> v2:172.21.15.170:3300/0 conn(0x7fde68156080 0x7fde680c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.712+0000 7fde6dfc7640 1 -- 172.21.15.161:0/1564511505 >> 172.21.15.161:0/1564511505 conn(0x7fde6806eba0 msgr2=0x7fde6806b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:16.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.712+0000 7fde6dfc7640 1 -- 172.21.15.161:0/1564511505 shutdown_connections 2024-03-28T23:30:16.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.712+0000 7fde6dfc7640 1 -- 172.21.15.161:0/1564511505 wait complete. 2024-03-28T23:30:16.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.713+0000 7fde6dfc7640 1 Processor -- start 2024-03-28T23:30:16.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.714+0000 7fde6dfc7640 1 -- start start 2024-03-28T23:30:16.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.714+0000 7fde6dfc7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde6806ff90 0x7fde680a83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:16.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.714+0000 7fde6dfc7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde68156080 0x7fde6814ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:16.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.714+0000 7fde6dfc7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde68156460 0x7fde680c1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:16.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.714+0000 7fde6dfc7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fde68070620 con 0x7fde68156460 2024-03-28T23:30:16.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.714+0000 7fde6dfc7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fde680704a0 con 0x7fde68156080 2024-03-28T23:30:16.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.714+0000 7fde6dfc7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fde680707a0 con 0x7fde6806ff90 2024-03-28T23:30:16.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.714+0000 7fde67fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde68156460 0x7fde680c1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:16.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.715+0000 7fde67fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde68156460 0x7fde680c1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54576/0 (socket says 172.21.15.161:54576) 2024-03-28T23:30:16.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.715+0000 7fde67fff640 1 -- 172.21.15.161:0/504415633 learned_addr learned my addr 172.21.15.161:0/504415633 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:16.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.715+0000 7fde677fe640 1 --2- 172.21.15.161:0/504415633 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde6806ff90 0x7fde680a83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:16.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.715+0000 7fde66ffd640 1 --2- 172.21.15.161:0/504415633 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde68156080 0x7fde6814ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:16.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.715+0000 7fde67fff640 1 -- 172.21.15.161:0/504415633 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde6806ff90 msgr2=0x7fde680a83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:16.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.715+0000 7fde67fff640 1 --2- 172.21.15.161:0/504415633 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde6806ff90 0x7fde680a83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.715+0000 7fde67fff640 1 -- 172.21.15.161:0/504415633 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde68156080 msgr2=0x7fde6814ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:16.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.715+0000 7fde67fff640 1 --2- 172.21.15.161:0/504415633 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde68156080 0x7fde6814ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.715+0000 7fde67fff640 1 -- 172.21.15.161:0/504415633 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde6806db70 con 0x7fde68156460 2024-03-28T23:30:16.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.715+0000 7fde67fff640 1 --2- 172.21.15.161:0/504415633 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde68156460 0x7fde680c1ce0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fde6000b9e0 tx=0x7fde6000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:16.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.715+0000 7fde4ffff640 1 -- 172.21.15.161:0/504415633 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde60003b70 con 0x7fde68156460 2024-03-28T23:30:16.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.715+0000 7fde4ffff640 1 -- 172.21.15.161:0/504415633 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde60003d10 con 0x7fde68156460 2024-03-28T23:30:16.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.715+0000 7fde6dfc7640 1 -- 172.21.15.161:0/504415633 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde681eb8d0 con 0x7fde68156460 2024-03-28T23:30:16.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.715+0000 7fde4ffff640 1 -- 172.21.15.161:0/504415633 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde6000cb60 con 0x7fde68156460 2024-03-28T23:30:16.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.715+0000 7fde6dfc7640 1 -- 172.21.15.161:0/504415633 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde6814f220 con 0x7fde68156460 2024-03-28T23:30:16.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.716+0000 7fde6dfc7640 1 -- 172.21.15.161:0/504415633 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde6804f4b0 con 0x7fde68156460 2024-03-28T23:30:16.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.717+0000 7fde4ffff640 1 -- 172.21.15.161:0/504415633 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fde6000cd00 con 0x7fde68156460 2024-03-28T23:30:16.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.717+0000 7fde4ffff640 1 --2- 172.21.15.161:0/504415633 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fde5406b4d0 0x7fde5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:16.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.718+0000 7fde4ffff640 1 -- 172.21.15.161:0/504415633 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fde6008a290 con 0x7fde68156460 2024-03-28T23:30:16.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.718+0000 7fde677fe640 1 --2- 172.21.15.161:0/504415633 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fde5406b4d0 0x7fde5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:16.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.718+0000 7fde677fe640 1 --2- 172.21.15.161:0/504415633 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fde5406b4d0 0x7fde5406d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fde58006fd0 tx=0x7fde58008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:16.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.722+0000 7fde4ffff640 1 -- 172.21.15.161:0/504415633 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fde60058810 con 0x7fde68156460 2024-03-28T23:30:16.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.893+0000 7fde6dfc7640 1 -- 172.21.15.161:0/504415633 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fde680c3fc0 con 0x7fde68156460 2024-03-28T23:30:16.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.893+0000 7fde4ffff640 1 -- 172.21.15.161:0/504415633 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7fde6005c8b0 con 0x7fde68156460 2024-03-28T23:30:16.894 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:30:16.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.895+0000 7fde6dfc7640 1 -- 172.21.15.161:0/504415633 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fde5406b4d0 msgr2=0x7fde5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:16.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.895+0000 7fde6dfc7640 1 --2- 172.21.15.161:0/504415633 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fde5406b4d0 0x7fde5406d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fde58006fd0 tx=0x7fde58008040 comp rx=0 tx=0).stop 2024-03-28T23:30:16.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.895+0000 7fde6dfc7640 1 -- 172.21.15.161:0/504415633 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde68156460 msgr2=0x7fde680c1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:16.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.895+0000 7fde6dfc7640 1 --2- 172.21.15.161:0/504415633 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde68156460 0x7fde680c1ce0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fde6000b9e0 tx=0x7fde6000bea0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.895+0000 7fde6dfc7640 1 -- 172.21.15.161:0/504415633 shutdown_connections 2024-03-28T23:30:16.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.895+0000 7fde6dfc7640 1 --2- 172.21.15.161:0/504415633 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fde5406b4d0 0x7fde5406d990 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.895+0000 7fde6dfc7640 1 --2- 172.21.15.161:0/504415633 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde68156460 0x7fde680c1ce0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.895+0000 7fde6dfc7640 1 --2- 172.21.15.161:0/504415633 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde68156080 0x7fde6814ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.895+0000 7fde6dfc7640 1 --2- 172.21.15.161:0/504415633 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde6806ff90 0x7fde680a83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:16.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.895+0000 7fde6dfc7640 1 -- 172.21.15.161:0/504415633 >> 172.21.15.161:0/504415633 conn(0x7fde6806eba0 msgr2=0x7fde680c41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:16.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.895+0000 7fde6dfc7640 1 -- 172.21.15.161:0/504415633 shutdown_connections 2024-03-28T23:30:16.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:16.895+0000 7fde6dfc7640 1 -- 172.21.15.161:0/504415633 wait complete. 2024-03-28T23:30:16.907 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:30:17.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.009+0000 7ff3695a7640 1 Processor -- start 2024-03-28T23:30:17.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.009+0000 7ff3695a7640 1 -- start start 2024-03-28T23:30:17.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.010+0000 7ff3695a7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff364152890 0x7ff3640c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:17.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.010+0000 7ff3695a7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3640c5b20 0x7ff3641553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:17.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.010+0000 7ff3695a7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff364158a30 con 0x7ff3640c8ed0 2024-03-28T23:30:17.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.010+0000 7ff3695a7640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff3640c8480 con 0x7ff364152890 2024-03-28T23:30:17.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.010+0000 7ff3695a7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff364159640 con 0x7ff3640c5b20 2024-03-28T23:30:17.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.010+0000 7ff362ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff364152890 0x7ff3640c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:17.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.010+0000 7ff362ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff364152890 0x7ff3640c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54586/0 (socket says 172.21.15.161:54586) 2024-03-28T23:30:17.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.010+0000 7ff362ffd640 1 -- 172.21.15.161:0/505492782 learned_addr learned my addr 172.21.15.161:0/505492782 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:17.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.010+0000 7ff3637fe640 1 --2- 172.21.15.161:0/505492782 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3640c5b20 0x7ff3641553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:17.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.010+0000 7ff362ffd640 1 -- 172.21.15.161:0/505492782 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3640c5b20 msgr2=0x7ff3641553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:17.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.010+0000 7ff362ffd640 1 --2- 172.21.15.161:0/505492782 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3640c5b20 0x7ff3641553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.010+0000 7ff362ffd640 1 -- 172.21.15.161:0/505492782 >> v1:172.21.15.170:6789/0 conn(0x7ff3640c8ed0 legacy=0x7ff3640c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:17.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.010+0000 7ff362ffd640 1 -- 172.21.15.161:0/505492782 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff364158d10 con 0x7ff364152890 2024-03-28T23:30:17.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.011+0000 7ff362ffd640 1 --2- 172.21.15.161:0/505492782 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff364152890 0x7ff3640c8990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff3641597c0 tx=0x7ff35402b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0d812fb902ee160 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:17.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.011+0000 7ff3627fc640 1 -- 172.21.15.161:0/505492782 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff354038070 con 0x7ff364152890 2024-03-28T23:30:17.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.011+0000 7ff3627fc640 1 -- 172.21.15.161:0/505492782 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff354031520 con 0x7ff364152890 2024-03-28T23:30:17.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.011+0000 7ff3627fc640 1 -- 172.21.15.161:0/505492782 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3540318f0 con 0x7ff364152890 2024-03-28T23:30:17.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.012+0000 7ff3695a7640 1 -- 172.21.15.161:0/505492782 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff364152890 msgr2=0x7ff3640c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:17.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.012+0000 7ff3695a7640 1 --2- 172.21.15.161:0/505492782 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff364152890 0x7ff3640c8990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff3641597c0 tx=0x7ff35402b7c0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.012+0000 7ff3695a7640 1 -- 172.21.15.161:0/505492782 shutdown_connections 2024-03-28T23:30:17.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.012+0000 7ff3695a7640 1 --2- 172.21.15.161:0/505492782 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3640c5b20 0x7ff3641553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.012+0000 7ff3695a7640 1 --2- 172.21.15.161:0/505492782 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff364152890 0x7ff3640c8990 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.012+0000 7ff3695a7640 1 -- 172.21.15.161:0/505492782 >> 172.21.15.161:0/505492782 conn(0x7ff3640ac620 msgr2=0x7ff3640aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:17.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.012+0000 7ff3695a7640 1 -- 172.21.15.161:0/505492782 shutdown_connections 2024-03-28T23:30:17.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.012+0000 7ff3695a7640 1 -- 172.21.15.161:0/505492782 wait complete. 2024-03-28T23:30:17.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.013+0000 7ff3695a7640 1 Processor -- start 2024-03-28T23:30:17.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.013+0000 7ff3695a7640 1 -- start start 2024-03-28T23:30:17.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.013+0000 7ff3695a7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3640c5b20 0x7ff3641eb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:17.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.013+0000 7ff3695a7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3640c8ed0 0x7ff3640c06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:17.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.013+0000 7ff3695a7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff364152890 0x7ff36414f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:17.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.013+0000 7ff3695a7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff364152c70 con 0x7ff364152890 2024-03-28T23:30:17.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.013+0000 7ff3695a7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff3641592d0 con 0x7ff3640c8ed0 2024-03-28T23:30:17.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.013+0000 7ff3695a7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff364152df0 con 0x7ff3640c5b20 2024-03-28T23:30:17.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.014+0000 7ff3637fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff364152890 0x7ff36414f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:17.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.014+0000 7ff3637fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff364152890 0x7ff36414f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54588/0 (socket says 172.21.15.161:54588) 2024-03-28T23:30:17.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.014+0000 7ff3637fe640 1 -- 172.21.15.161:0/1786489904 learned_addr learned my addr 172.21.15.161:0/1786489904 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:17.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.014+0000 7ff362ffd640 1 --2- 172.21.15.161:0/1786489904 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3640c5b20 0x7ff3641eb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:17.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.014+0000 7ff35bfff640 1 --2- 172.21.15.161:0/1786489904 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3640c8ed0 0x7ff3640c06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:17.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.014+0000 7ff3637fe640 1 -- 172.21.15.161:0/1786489904 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3640c5b20 msgr2=0x7ff3641eb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:17.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.014+0000 7ff3637fe640 1 --2- 172.21.15.161:0/1786489904 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3640c5b20 0x7ff3641eb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.014+0000 7ff3637fe640 1 -- 172.21.15.161:0/1786489904 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3640c8ed0 msgr2=0x7ff3640c06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:17.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.014+0000 7ff3637fe640 1 --2- 172.21.15.161:0/1786489904 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3640c8ed0 0x7ff3640c06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.014+0000 7ff3637fe640 1 -- 172.21.15.161:0/1786489904 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff36414e960 con 0x7ff364152890 2024-03-28T23:30:17.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.014+0000 7ff362ffd640 1 --2- 172.21.15.161:0/1786489904 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3640c5b20 0x7ff3641eb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:17.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.014+0000 7ff35bfff640 1 --2- 172.21.15.161:0/1786489904 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3640c8ed0 0x7ff3640c06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:17.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.014+0000 7ff3637fe640 1 --2- 172.21.15.161:0/1786489904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff364152890 0x7ff36414f540 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff35c00cc80 tx=0x7ff35c00a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:17.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.014+0000 7ff35b7fe640 1 -- 172.21.15.161:0/1786489904 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35c003990 con 0x7ff364152890 2024-03-28T23:30:17.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.014+0000 7ff35b7fe640 1 -- 172.21.15.161:0/1786489904 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff35c003b50 con 0x7ff364152890 2024-03-28T23:30:17.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.014+0000 7ff3695a7640 1 -- 172.21.15.161:0/1786489904 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3641eae90 con 0x7ff364152890 2024-03-28T23:30:17.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.014+0000 7ff35b7fe640 1 -- 172.21.15.161:0/1786489904 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35c004020 con 0x7ff364152890 2024-03-28T23:30:17.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.014+0000 7ff3695a7640 1 -- 172.21.15.161:0/1786489904 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3640c0bf0 con 0x7ff364152890 2024-03-28T23:30:17.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.015+0000 7ff3695a7640 1 -- 172.21.15.161:0/1786489904 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff36404f4b0 con 0x7ff364152890 2024-03-28T23:30:17.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.016+0000 7ff35b7fe640 1 -- 172.21.15.161:0/1786489904 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff35c0041c0 con 0x7ff364152890 2024-03-28T23:30:17.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.016+0000 7ff35b7fe640 1 --2- 172.21.15.161:0/1786489904 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff34406b4d0 0x7ff34406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:17.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.016+0000 7ff35b7fe640 1 -- 172.21.15.161:0/1786489904 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff35c08a920 con 0x7ff364152890 2024-03-28T23:30:17.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.016+0000 7ff362ffd640 1 --2- 172.21.15.161:0/1786489904 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff34406b4d0 0x7ff34406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:17.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.017+0000 7ff362ffd640 1 --2- 172.21.15.161:0/1786489904 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff34406b4d0 0x7ff34406d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff354035000 tx=0x7ff354030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:17.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.021+0000 7ff35b7fe640 1 -- 172.21.15.161:0/1786489904 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff35c014030 con 0x7ff364152890 2024-03-28T23:30:17.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.236+0000 7ff3695a7640 1 -- 172.21.15.161:0/1786489904 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3640be290 con 0x7ff364152890 2024-03-28T23:30:17.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.237+0000 7ff35b7fe640 1 -- 172.21.15.161:0/1786489904 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5189 (secure 0 0 0) 0x7ff35c0594f0 con 0x7ff364152890 2024-03-28T23:30:17.238 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:17.238 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/1577284938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":1577284938},{"type":"v1","addr":"172.21.15.170:6835","nonce":1577284938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4258,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/1107796931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1107796931},{"type":"v1","addr":"172.21.15.170:6834","nonce":1107796931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4107992135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4107992135},{"type":"v1","addr":"172.21.15.161:6836","nonce":4107992135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:29:45.903844+0000","modified":"2024-03-28T23:30:07.984387+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6837/3832540545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3832540545},{"type":"v1","addr":"172.21.15.161:6837","nonce":3832540545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4275,"qdb_cluster":[4275]},"id":1}]} 2024-03-28T23:30:17.238 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 9 2024-03-28T23:30:17.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.239+0000 7ff3695a7640 1 -- 172.21.15.161:0/1786489904 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff34406b4d0 msgr2=0x7ff34406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:17.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.239+0000 7ff3695a7640 1 --2- 172.21.15.161:0/1786489904 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff34406b4d0 0x7ff34406d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff354035000 tx=0x7ff354030000 comp rx=0 tx=0).stop 2024-03-28T23:30:17.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.239+0000 7ff3695a7640 1 -- 172.21.15.161:0/1786489904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff364152890 msgr2=0x7ff36414f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:17.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.239+0000 7ff3695a7640 1 --2- 172.21.15.161:0/1786489904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff364152890 0x7ff36414f540 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff35c00cc80 tx=0x7ff35c00a5c0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.239+0000 7ff3695a7640 1 -- 172.21.15.161:0/1786489904 shutdown_connections 2024-03-28T23:30:17.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.239+0000 7ff3695a7640 1 --2- 172.21.15.161:0/1786489904 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff34406b4d0 0x7ff34406d990 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.239+0000 7ff3695a7640 1 --2- 172.21.15.161:0/1786489904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff364152890 0x7ff36414f540 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.239+0000 7ff3695a7640 1 --2- 172.21.15.161:0/1786489904 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3640c8ed0 0x7ff3640c06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.239+0000 7ff3695a7640 1 --2- 172.21.15.161:0/1786489904 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3640c5b20 0x7ff3641eb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.239+0000 7ff3695a7640 1 -- 172.21.15.161:0/1786489904 >> 172.21.15.161:0/1786489904 conn(0x7ff3640ac620 msgr2=0x7ff364150d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:17.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.239+0000 7ff3695a7640 1 -- 172.21.15.161:0/1786489904 shutdown_connections 2024-03-28T23:30:17.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.239+0000 7ff3695a7640 1 -- 172.21.15.161:0/1786489904 wait complete. 2024-03-28T23:30:17.251 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:30:17.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.353+0000 7f338f684640 1 Processor -- start 2024-03-28T23:30:17.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.353+0000 7f338f684640 1 -- start start 2024-03-28T23:30:17.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.354+0000 7f338f684640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3388152890 0x7f33880c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:17.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.354+0000 7f338f684640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3388158a30 con 0x7f33880c8ed0 2024-03-28T23:30:17.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.354+0000 7f338f684640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3388158d10 con 0x7f33880c5b20 2024-03-28T23:30:17.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.354+0000 7f338f684640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f33880c8480 con 0x7f3388152890 2024-03-28T23:30:17.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.354+0000 7f338d420640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3388152890 0x7f33880c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:17.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.354+0000 7f338d420640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3388152890 0x7f33880c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54602/0 (socket says 172.21.15.161:54602) 2024-03-28T23:30:17.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.354+0000 7f338d420640 1 -- 172.21.15.161:0/258562036 learned_addr learned my addr 172.21.15.161:0/258562036 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:17.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.355+0000 7f338d420640 1 -- 172.21.15.161:0/258562036 >> v1:172.21.15.170:6789/0 conn(0x7f33880c5b20 legacy=0x7f33881553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:17.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.355+0000 7f338d420640 1 -- 172.21.15.161:0/258562036 >> v1:172.21.15.161:6789/0 conn(0x7f33880c8ed0 legacy=0x7f33880c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:17.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.355+0000 7f337ffff640 1 -- 172.21.15.161:0/258562036 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3487372009 0 0) 0x7f3388158a30 con 0x7f33880c8ed0 2024-03-28T23:30:17.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.355+0000 7f338d420640 1 -- 172.21.15.161:0/258562036 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3388158ff0 con 0x7f3388152890 2024-03-28T23:30:17.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.355+0000 7f338d420640 1 --2- 172.21.15.161:0/258562036 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3388152890 0x7f33880c8990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3384009870 tx=0x7f338402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f49f4fe83bfc0df server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:17.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.355+0000 7f337ffff640 1 -- 172.21.15.161:0/258562036 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3384035020 con 0x7f3388152890 2024-03-28T23:30:17.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.355+0000 7f337ffff640 1 -- 172.21.15.161:0/258562036 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3384031870 con 0x7f3388152890 2024-03-28T23:30:17.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.355+0000 7f337ffff640 1 -- 172.21.15.161:0/258562036 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3384031c40 con 0x7f3388152890 2024-03-28T23:30:17.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.356+0000 7f338f684640 1 -- 172.21.15.161:0/258562036 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3388152890 msgr2=0x7f33880c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:17.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.356+0000 7f338f684640 1 --2- 172.21.15.161:0/258562036 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3388152890 0x7f33880c8990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3384009870 tx=0x7f338402b910 comp rx=0 tx=0).stop 2024-03-28T23:30:17.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.356+0000 7f338f684640 1 -- 172.21.15.161:0/258562036 shutdown_connections 2024-03-28T23:30:17.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.356+0000 7f338f684640 1 --2- 172.21.15.161:0/258562036 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3388152890 0x7f33880c8990 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.356+0000 7f338f684640 1 -- 172.21.15.161:0/258562036 >> 172.21.15.161:0/258562036 conn(0x7f33880ac620 msgr2=0x7f33880aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:17.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.356+0000 7f338f684640 1 -- 172.21.15.161:0/258562036 shutdown_connections 2024-03-28T23:30:17.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.356+0000 7f338f684640 1 -- 172.21.15.161:0/258562036 wait complete. 2024-03-28T23:30:17.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.357+0000 7f338f684640 1 Processor -- start 2024-03-28T23:30:17.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.357+0000 7f338f684640 1 -- start start 2024-03-28T23:30:17.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.359+0000 7f338f684640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33880c5b20 0x7f33881579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:17.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.359+0000 7f338f684640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f33880c8ed0 0x7f33881ee600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:17.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.359+0000 7f338f684640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3388152890 0x7f33881eeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:17.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.359+0000 7f338f684640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3388152c70 con 0x7f3388152890 2024-03-28T23:30:17.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.359+0000 7f338f684640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3388159640 con 0x7f33880c5b20 2024-03-28T23:30:17.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.359+0000 7f338f684640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3388152df0 con 0x7f33880c8ed0 2024-03-28T23:30:17.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.359+0000 7f338dc21640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3388152890 0x7f33881eeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:17.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.359+0000 7f338dc21640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3388152890 0x7f33881eeb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54618/0 (socket says 172.21.15.161:54618) 2024-03-28T23:30:17.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.359+0000 7f338dc21640 1 -- 172.21.15.161:0/1639772259 learned_addr learned my addr 172.21.15.161:0/1639772259 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:17.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.359+0000 7f338d420640 1 --2- 172.21.15.161:0/1639772259 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33880c5b20 0x7f33881579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:17.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.359+0000 7f338cc1f640 1 --2- 172.21.15.161:0/1639772259 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f33880c8ed0 0x7f33881ee600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:17.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.359+0000 7f338dc21640 1 -- 172.21.15.161:0/1639772259 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f33880c8ed0 msgr2=0x7f33881ee600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:17.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.359+0000 7f338dc21640 1 --2- 172.21.15.161:0/1639772259 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f33880c8ed0 0x7f33881ee600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.359+0000 7f338dc21640 1 -- 172.21.15.161:0/1639772259 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33880c5b20 msgr2=0x7f33881579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:17.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.359+0000 7f338dc21640 1 --2- 172.21.15.161:0/1639772259 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33880c5b20 0x7f33881579f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.359+0000 7f338dc21640 1 -- 172.21.15.161:0/1639772259 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33881ee120 con 0x7f3388152890 2024-03-28T23:30:17.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.359+0000 7f338d420640 1 --2- 172.21.15.161:0/1639772259 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33880c5b20 0x7f33881579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:17.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.359+0000 7f338dc21640 1 --2- 172.21.15.161:0/1639772259 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3388152890 0x7f33881eeb40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f338000c730 tx=0x7f338000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:17.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.359+0000 7f337dffb640 1 -- 172.21.15.161:0/1639772259 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3380003b70 con 0x7f3388152890 2024-03-28T23:30:17.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.359+0000 7f337dffb640 1 -- 172.21.15.161:0/1639772259 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3380003d10 con 0x7f3388152890 2024-03-28T23:30:17.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.359+0000 7f338f684640 1 -- 172.21.15.161:0/1639772259 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3388153ba0 con 0x7f3388152890 2024-03-28T23:30:17.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.359+0000 7f337dffb640 1 -- 172.21.15.161:0/1639772259 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3380005310 con 0x7f3388152890 2024-03-28T23:30:17.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.360+0000 7f338f684640 1 -- 172.21.15.161:0/1639772259 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3388157f30 con 0x7f3388152890 2024-03-28T23:30:17.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.361+0000 7f338f684640 1 -- 172.21.15.161:0/1639772259 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f338804f4b0 con 0x7f3388152890 2024-03-28T23:30:17.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.361+0000 7f337dffb640 1 -- 172.21.15.161:0/1639772259 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f33800056c0 con 0x7f3388152890 2024-03-28T23:30:17.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.361+0000 7f337dffb640 1 --2- 172.21.15.161:0/1639772259 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f336c06b4d0 0x7f336c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:17.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.361+0000 7f337dffb640 1 -- 172.21.15.161:0/1639772259 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f338008a500 con 0x7f3388152890 2024-03-28T23:30:17.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.361+0000 7f338d420640 1 --2- 172.21.15.161:0/1639772259 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f336c06b4d0 0x7f336c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:17.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.362+0000 7f338d420640 1 --2- 172.21.15.161:0/1639772259 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f336c06b4d0 0x7f336c06d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3384036000 tx=0x7f3384008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:17.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.370+0000 7f337dffb640 1 -- 172.21.15.161:0/1639772259 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f33800590d0 con 0x7f3388152890 2024-03-28T23:30:17.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.549+0000 7f338f684640 1 -- 172.21.15.161:0/1639772259 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3388074210 con 0x7f3388152890 2024-03-28T23:30:17.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.550+0000 7f337dffb640 1 -- 172.21.15.161:0/1639772259 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7f33800589d0 con 0x7f3388152890 2024-03-28T23:30:17.551 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:17.551 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":24,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:29:50.946843+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:30:17.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.552+0000 7f338f684640 1 -- 172.21.15.161:0/1639772259 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f336c06b4d0 msgr2=0x7f336c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:17.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.552+0000 7f338f684640 1 --2- 172.21.15.161:0/1639772259 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f336c06b4d0 0x7f336c06d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3384036000 tx=0x7f3384008030 comp rx=0 tx=0).stop 2024-03-28T23:30:17.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.552+0000 7f338f684640 1 -- 172.21.15.161:0/1639772259 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3388152890 msgr2=0x7f33881eeb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:17.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.552+0000 7f338f684640 1 --2- 172.21.15.161:0/1639772259 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3388152890 0x7f33881eeb40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f338000c730 tx=0x7f338000cbf0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.552+0000 7f338f684640 1 -- 172.21.15.161:0/1639772259 shutdown_connections 2024-03-28T23:30:17.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.552+0000 7f338f684640 1 --2- 172.21.15.161:0/1639772259 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f336c06b4d0 0x7f336c06d990 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.552+0000 7f338f684640 1 --2- 172.21.15.161:0/1639772259 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3388152890 0x7f33881eeb40 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.552+0000 7f338f684640 1 --2- 172.21.15.161:0/1639772259 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f33880c8ed0 0x7f33881ee600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.552+0000 7f338f684640 1 --2- 172.21.15.161:0/1639772259 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33880c5b20 0x7f33881579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.552+0000 7f338f684640 1 -- 172.21.15.161:0/1639772259 >> 172.21.15.161:0/1639772259 conn(0x7f33880ac620 msgr2=0x7f33880c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:17.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.553+0000 7f338f684640 1 -- 172.21.15.161:0/1639772259 shutdown_connections 2024-03-28T23:30:17.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.553+0000 7f338f684640 1 -- 172.21.15.161:0/1639772259 wait complete. 2024-03-28T23:30:17.566 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:29:44.021375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:29:44.872102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T23:29:48.591063+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T23:30:17.566 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:30:17.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.671+0000 7f1ed4078640 1 Processor -- start 2024-03-28T23:30:17.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.671+0000 7f1ed4078640 1 -- start start 2024-03-28T23:30:17.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.672+0000 7f1ed4078640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1ecc152800 0x7f1ecc0c32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:17.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.672+0000 7f1ed4078640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1ecc0c37f0 0x7f1ecc158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:17.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.672+0000 7f1ed4078640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ecc155330 con 0x7f1ecc0c67b0 2024-03-28T23:30:17.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.672+0000 7f1ed4078640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1ecc155610 con 0x7f1ecc152800 2024-03-28T23:30:17.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.672+0000 7f1ed4078640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f1ecc1558f0 con 0x7f1ecc0c37f0 2024-03-28T23:30:17.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.672+0000 7f1ed1e14640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1ecc152800 0x7f1ecc0c32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:17.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.672+0000 7f1ed1e14640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1ecc152800 0x7f1ecc0c32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54622/0 (socket says 172.21.15.161:54622) 2024-03-28T23:30:17.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.672+0000 7f1ed1e14640 1 -- 172.21.15.161:0/4286931750 learned_addr learned my addr 172.21.15.161:0/4286931750 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:17.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.672+0000 7f1ed2615640 1 --2- 172.21.15.161:0/4286931750 >> v2:172.21.15.170:3300/0 conn(0x7f1ecc0c37f0 0x7f1ecc158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:17.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.672+0000 7f1ed1e14640 1 -- 172.21.15.161:0/4286931750 >> v2:172.21.15.170:3300/0 conn(0x7f1ecc0c37f0 msgr2=0x7f1ecc158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:17.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.672+0000 7f1ed1e14640 1 --2- 172.21.15.161:0/4286931750 >> v2:172.21.15.170:3300/0 conn(0x7f1ecc0c37f0 0x7f1ecc158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.672+0000 7f1ed1e14640 1 -- 172.21.15.161:0/4286931750 >> v1:172.21.15.170:6789/0 conn(0x7f1ecc0c67b0 legacy=0x7f1ecc1588e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:17.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.672+0000 7f1ed1e14640 1 -- 172.21.15.161:0/4286931750 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ecc155bd0 con 0x7f1ecc152800 2024-03-28T23:30:17.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.673+0000 7f1ed1e14640 1 --2- 172.21.15.161:0/4286931750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ecc152800 0x7f1ecc0c32b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc0a8440 tx=0x7f1ebc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4862d7c182e331 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:17.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.673+0000 7f1ed0e12640 1 -- 172.21.15.161:0/4286931750 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc035020 con 0x7f1ecc152800 2024-03-28T23:30:17.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.673+0000 7f1ed0e12640 1 -- 172.21.15.161:0/4286931750 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ebc031870 con 0x7f1ecc152800 2024-03-28T23:30:17.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.673+0000 7f1ed0e12640 1 -- 172.21.15.161:0/4286931750 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc031c40 con 0x7f1ecc152800 2024-03-28T23:30:17.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.674+0000 7f1ed4078640 1 -- 172.21.15.161:0/4286931750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ecc152800 msgr2=0x7f1ecc0c32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:17.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.674+0000 7f1ed4078640 1 --2- 172.21.15.161:0/4286931750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ecc152800 0x7f1ecc0c32b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc0a8440 tx=0x7f1ebc02b910 comp rx=0 tx=0).stop 2024-03-28T23:30:17.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.674+0000 7f1ed4078640 1 -- 172.21.15.161:0/4286931750 shutdown_connections 2024-03-28T23:30:17.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.674+0000 7f1ed4078640 1 --2- 172.21.15.161:0/4286931750 >> v2:172.21.15.170:3300/0 conn(0x7f1ecc0c37f0 0x7f1ecc158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.674+0000 7f1ed4078640 1 --2- 172.21.15.161:0/4286931750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ecc152800 0x7f1ecc0c32b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.674+0000 7f1ed4078640 1 -- 172.21.15.161:0/4286931750 >> 172.21.15.161:0/4286931750 conn(0x7f1ecc0aa660 msgr2=0x7f1ecc0aaa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:17.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.674+0000 7f1ed4078640 1 -- 172.21.15.161:0/4286931750 shutdown_connections 2024-03-28T23:30:17.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.674+0000 7f1ed4078640 1 -- 172.21.15.161:0/4286931750 wait complete. 2024-03-28T23:30:17.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.675+0000 7f1ed4078640 1 Processor -- start 2024-03-28T23:30:17.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.675+0000 7f1ed4078640 1 -- start start 2024-03-28T23:30:17.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.676+0000 7f1ed4078640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ecc0c37f0 0x7f1ecc157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:17.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.676+0000 7f1ed4078640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ecc0c67b0 0x7f1ecc156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:17.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.676+0000 7f1ed4078640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ecc152800 0x7f1ecc06c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:17.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.676+0000 7f1ed4078640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1ecc159530 con 0x7f1ecc0c67b0 2024-03-28T23:30:17.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.676+0000 7f1ed4078640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1ecc073680 con 0x7f1ecc0c37f0 2024-03-28T23:30:17.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.676+0000 7f1ed4078640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1ecc0a7d00 con 0x7f1ecc152800 2024-03-28T23:30:17.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.676+0000 7f1ed1613640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ecc0c67b0 0x7f1ecc156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:17.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.676+0000 7f1ed1613640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ecc0c67b0 0x7f1ecc156ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54624/0 (socket says 172.21.15.161:54624) 2024-03-28T23:30:17.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.676+0000 7f1ed1613640 1 -- 172.21.15.161:0/353145847 learned_addr learned my addr 172.21.15.161:0/353145847 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:17.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.676+0000 7f1ed1e14640 1 --2- 172.21.15.161:0/353145847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ecc0c37f0 0x7f1ecc157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:17.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.676+0000 7f1ed2615640 1 --2- 172.21.15.161:0/353145847 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ecc152800 0x7f1ecc06c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:17.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.676+0000 7f1ed1613640 1 -- 172.21.15.161:0/353145847 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ecc152800 msgr2=0x7f1ecc06c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:17.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.676+0000 7f1ed1613640 1 --2- 172.21.15.161:0/353145847 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ecc152800 0x7f1ecc06c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.676+0000 7f1ed1613640 1 -- 172.21.15.161:0/353145847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ecc0c37f0 msgr2=0x7f1ecc157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:17.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.676+0000 7f1ed1613640 1 --2- 172.21.15.161:0/353145847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ecc0c37f0 0x7f1ecc157970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.676+0000 7f1ed1613640 1 -- 172.21.15.161:0/353145847 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ecc070710 con 0x7f1ecc0c67b0 2024-03-28T23:30:17.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.676+0000 7f1ed2615640 1 --2- 172.21.15.161:0/353145847 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ecc152800 0x7f1ecc06c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:17.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.676+0000 7f1ed1613640 1 --2- 172.21.15.161:0/353145847 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ecc0c67b0 0x7f1ecc156ca0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1ec800c440 tx=0x7f1ec8001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:17.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.676+0000 7f1ec27fc640 1 -- 172.21.15.161:0/353145847 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec80052d0 con 0x7f1ecc0c67b0 2024-03-28T23:30:17.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.676+0000 7f1ec27fc640 1 -- 172.21.15.161:0/353145847 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ec8005470 con 0x7f1ecc0c67b0 2024-03-28T23:30:17.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.676+0000 7f1ec27fc640 1 -- 172.21.15.161:0/353145847 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec8007260 con 0x7f1ecc0c67b0 2024-03-28T23:30:17.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.676+0000 7f1ed4078640 1 -- 172.21.15.161:0/353145847 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ecc157690 con 0x7f1ecc0c67b0 2024-03-28T23:30:17.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.677+0000 7f1ed4078640 1 -- 172.21.15.161:0/353145847 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ecc06f2e0 con 0x7f1ecc0c67b0 2024-03-28T23:30:17.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.678+0000 7f1ed4078640 1 -- 172.21.15.161:0/353145847 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ecc04f4b0 con 0x7f1ecc0c67b0 2024-03-28T23:30:17.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.678+0000 7f1ec27fc640 1 -- 172.21.15.161:0/353145847 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1ec8007400 con 0x7f1ecc0c67b0 2024-03-28T23:30:17.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.678+0000 7f1ec27fc640 1 --2- 172.21.15.161:0/353145847 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1eb006b500 0x7f1eb006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:17.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.678+0000 7f1ec27fc640 1 -- 172.21.15.161:0/353145847 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f1ec8089890 con 0x7f1ecc0c67b0 2024-03-28T23:30:17.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.678+0000 7f1ed1e14640 1 --2- 172.21.15.161:0/353145847 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1eb006b500 0x7f1eb006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:17.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.679+0000 7f1ed1e14640 1 --2- 172.21.15.161:0/353145847 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1eb006b500 0x7f1eb006d9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc036000 tx=0x7f1ebc03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:17.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.683+0000 7f1ec27fc640 1 -- 172.21.15.161:0/353145847 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1ec8057e10 con 0x7f1ecc0c67b0 2024-03-28T23:30:17.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.860+0000 7f1ed4078640 1 -- 172.21.15.161:0/353145847 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1ecc0aaa40 con 0x7f1ecc0c67b0 2024-03-28T23:30:17.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.860+0000 7f1ec27fc640 1 -- 172.21.15.161:0/353145847 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f1ec805beb0 con 0x7f1ecc0c67b0 2024-03-28T23:30:17.861 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:30:17.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.862+0000 7f1ed4078640 1 -- 172.21.15.161:0/353145847 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1eb006b500 msgr2=0x7f1eb006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:17.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.862+0000 7f1ed4078640 1 --2- 172.21.15.161:0/353145847 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1eb006b500 0x7f1eb006d9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc036000 tx=0x7f1ebc03c040 comp rx=0 tx=0).stop 2024-03-28T23:30:17.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.862+0000 7f1ed4078640 1 -- 172.21.15.161:0/353145847 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ecc0c67b0 msgr2=0x7f1ecc156ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:17.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.862+0000 7f1ed4078640 1 --2- 172.21.15.161:0/353145847 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ecc0c67b0 0x7f1ecc156ca0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1ec800c440 tx=0x7f1ec8001e70 comp rx=0 tx=0).stop 2024-03-28T23:30:17.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.862+0000 7f1ed4078640 1 -- 172.21.15.161:0/353145847 shutdown_connections 2024-03-28T23:30:17.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.862+0000 7f1ed4078640 1 --2- 172.21.15.161:0/353145847 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1eb006b500 0x7f1eb006d9c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.862+0000 7f1ed4078640 1 --2- 172.21.15.161:0/353145847 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ecc0c67b0 0x7f1ecc156ca0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.862+0000 7f1ed4078640 1 --2- 172.21.15.161:0/353145847 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ecc152800 0x7f1ecc06c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.862+0000 7f1ed4078640 1 --2- 172.21.15.161:0/353145847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ecc0c37f0 0x7f1ecc157970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.862+0000 7f1ed4078640 1 -- 172.21.15.161:0/353145847 >> 172.21.15.161:0/353145847 conn(0x7f1ecc0aa660 msgr2=0x7f1ecc152df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:17.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.862+0000 7f1ed4078640 1 -- 172.21.15.161:0/353145847 shutdown_connections 2024-03-28T23:30:17.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.862+0000 7f1ed4078640 1 -- 172.21.15.161:0/353145847 wait complete. 2024-03-28T23:30:17.874 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:30:17.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.979+0000 7f0147881640 1 Processor -- start 2024-03-28T23:30:17.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.979+0000 7f0147881640 1 -- start start 2024-03-28T23:30:17.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.980+0000 7f0147881640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0140156080 0x7f01400c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:17.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.980+0000 7f0147881640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0140156460 0x7f0140157290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:17.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.980+0000 7f0147881640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0140072790 con 0x7f014006ff90 2024-03-28T23:30:17.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.980+0000 7f0147881640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f01400c67b0 con 0x7f0140156080 2024-03-28T23:30:17.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.980+0000 7f0147881640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f01400a7ea0 con 0x7f0140156460 2024-03-28T23:30:17.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.980+0000 7f014561d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0140156080 0x7f01400c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:17.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.980+0000 7f014561d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0140156080 0x7f01400c5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54632/0 (socket says 172.21.15.161:54632) 2024-03-28T23:30:17.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.980+0000 7f014561d640 1 -- 172.21.15.161:0/2863125365 learned_addr learned my addr 172.21.15.161:0/2863125365 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:17.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.980+0000 7f0145e1e640 1 --2- 172.21.15.161:0/2863125365 >> v2:172.21.15.170:3300/0 conn(0x7f0140156460 0x7f0140157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:17.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.981+0000 7f014561d640 1 -- 172.21.15.161:0/2863125365 >> v2:172.21.15.170:3300/0 conn(0x7f0140156460 msgr2=0x7f0140157290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:17.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.981+0000 7f014561d640 1 --2- 172.21.15.161:0/2863125365 >> v2:172.21.15.170:3300/0 conn(0x7f0140156460 0x7f0140157290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.981+0000 7f014561d640 1 -- 172.21.15.161:0/2863125365 >> v1:172.21.15.170:6789/0 conn(0x7f014006ff90 legacy=0x7f0140156b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:17.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.981+0000 7f014561d640 1 -- 172.21.15.161:0/2863125365 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01400a8180 con 0x7f0140156080 2024-03-28T23:30:17.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.981+0000 7f014561d640 1 --2- 172.21.15.161:0/2863125365 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0140156080 0x7f01400c5340 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f01400c7ff0 tx=0x7f013c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e91dca2c2114b5f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:17.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.981+0000 7f0137fff640 1 -- 172.21.15.161:0/2863125365 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013c038070 con 0x7f0140156080 2024-03-28T23:30:17.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.981+0000 7f0137fff640 1 -- 172.21.15.161:0/2863125365 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f013c02f050 con 0x7f0140156080 2024-03-28T23:30:17.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.981+0000 7f0137fff640 1 -- 172.21.15.161:0/2863125365 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013c031970 con 0x7f0140156080 2024-03-28T23:30:17.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.982+0000 7f0147881640 1 -- 172.21.15.161:0/2863125365 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0140156080 msgr2=0x7f01400c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:17.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.982+0000 7f0147881640 1 --2- 172.21.15.161:0/2863125365 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0140156080 0x7f01400c5340 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f01400c7ff0 tx=0x7f013c02b7f0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.982+0000 7f0147881640 1 -- 172.21.15.161:0/2863125365 shutdown_connections 2024-03-28T23:30:17.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.982+0000 7f0147881640 1 --2- 172.21.15.161:0/2863125365 >> v2:172.21.15.170:3300/0 conn(0x7f0140156460 0x7f0140157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.982+0000 7f0147881640 1 --2- 172.21.15.161:0/2863125365 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0140156080 0x7f01400c5340 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.982+0000 7f0147881640 1 -- 172.21.15.161:0/2863125365 >> 172.21.15.161:0/2863125365 conn(0x7f014006eba0 msgr2=0x7f014006b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:17.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.982+0000 7f0147881640 1 -- 172.21.15.161:0/2863125365 shutdown_connections 2024-03-28T23:30:17.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.982+0000 7f0147881640 1 -- 172.21.15.161:0/2863125365 wait complete. 2024-03-28T23:30:17.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.983+0000 7f0147881640 1 Processor -- start 2024-03-28T23:30:17.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.983+0000 7f0147881640 1 -- start start 2024-03-28T23:30:17.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.984+0000 7f0147881640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f014006ff90 0x7f01401e01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:17.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.984+0000 7f0147881640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0140156080 0x7f01401de520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:17.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.984+0000 7f0147881640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0140156460 0x7f01400a9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:17.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.984+0000 7f0147881640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0140070620 con 0x7f0140156460 2024-03-28T23:30:17.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.984+0000 7f0147881640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f01400704a0 con 0x7f014006ff90 2024-03-28T23:30:17.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.984+0000 7f0147881640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f01400707a0 con 0x7f0140156080 2024-03-28T23:30:17.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.984+0000 7f0145e1e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0140156460 0x7f01400a9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:17.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.984+0000 7f0145e1e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0140156460 0x7f01400a9e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54636/0 (socket says 172.21.15.161:54636) 2024-03-28T23:30:17.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.984+0000 7f0145e1e640 1 -- 172.21.15.161:0/3422957012 learned_addr learned my addr 172.21.15.161:0/3422957012 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:17.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.984+0000 7f014561d640 1 --2- 172.21.15.161:0/3422957012 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f014006ff90 0x7f01401e01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:17.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.984+0000 7f0144e1c640 1 --2- 172.21.15.161:0/3422957012 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0140156080 0x7f01401de520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:17.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.985+0000 7f0145e1e640 1 -- 172.21.15.161:0/3422957012 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0140156080 msgr2=0x7f01401de520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:17.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.985+0000 7f0145e1e640 1 --2- 172.21.15.161:0/3422957012 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0140156080 0x7f01401de520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.985+0000 7f0145e1e640 1 -- 172.21.15.161:0/3422957012 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f014006ff90 msgr2=0x7f01401e01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:17.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.985+0000 7f0145e1e640 1 --2- 172.21.15.161:0/3422957012 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f014006ff90 0x7f01401e01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:17.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.985+0000 7f0145e1e640 1 -- 172.21.15.161:0/3422957012 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01400c25f0 con 0x7f0140156460 2024-03-28T23:30:17.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.985+0000 7f0145e1e640 1 --2- 172.21.15.161:0/3422957012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0140156460 0x7f01400a9e20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0138007d80 tx=0x7f013800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:17.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.985+0000 7f0135ffb640 1 -- 172.21.15.161:0/3422957012 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0138014420 con 0x7f0140156460 2024-03-28T23:30:17.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.985+0000 7f0135ffb640 1 -- 172.21.15.161:0/3422957012 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0138019070 con 0x7f0140156460 2024-03-28T23:30:17.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.985+0000 7f0147881640 1 -- 172.21.15.161:0/3422957012 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01401ecb10 con 0x7f0140156460 2024-03-28T23:30:17.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.985+0000 7f0135ffb640 1 -- 172.21.15.161:0/3422957012 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0138014800 con 0x7f0140156460 2024-03-28T23:30:17.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.985+0000 7f0147881640 1 -- 172.21.15.161:0/3422957012 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01401dea60 con 0x7f0140156460 2024-03-28T23:30:17.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.986+0000 7f0147881640 1 -- 172.21.15.161:0/3422957012 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f014004f4b0 con 0x7f0140156460 2024-03-28T23:30:17.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.987+0000 7f0135ffb640 1 -- 172.21.15.161:0/3422957012 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f01380149a0 con 0x7f0140156460 2024-03-28T23:30:17.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.987+0000 7f0135ffb640 1 --2- 172.21.15.161:0/3422957012 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f012406b4d0 0x7f012406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:17.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.987+0000 7f0135ffb640 1 -- 172.21.15.161:0/3422957012 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f013808ab90 con 0x7f0140156460 2024-03-28T23:30:17.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.988+0000 7f014561d640 1 --2- 172.21.15.161:0/3422957012 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f012406b4d0 0x7f012406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:17.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.988+0000 7f014561d640 1 --2- 172.21.15.161:0/3422957012 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f012406b4d0 0x7f012406d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f013c035000 tx=0x7f013c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:17.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:17.996+0000 7f0135ffb640 1 -- 172.21.15.161:0/3422957012 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0138059110 con 0x7f0140156460 2024-03-28T23:30:18.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.173+0000 7f0147881640 1 -- 172.21.15.161:0/3422957012 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f014006fd30 con 0x7f0140156460 2024-03-28T23:30:18.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.173+0000 7f0135ffb640 1 -- 172.21.15.161:0/3422957012 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f013805d1b0 con 0x7f0140156460 2024-03-28T23:30:18.174 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:30:18.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.175+0000 7f0147881640 1 -- 172.21.15.161:0/3422957012 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f012406b4d0 msgr2=0x7f012406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:18.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.175+0000 7f0147881640 1 --2- 172.21.15.161:0/3422957012 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f012406b4d0 0x7f012406d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f013c035000 tx=0x7f013c008030 comp rx=0 tx=0).stop 2024-03-28T23:30:18.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.175+0000 7f0147881640 1 -- 172.21.15.161:0/3422957012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0140156460 msgr2=0x7f01400a9e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:18.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.175+0000 7f0147881640 1 --2- 172.21.15.161:0/3422957012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0140156460 0x7f01400a9e20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0138007d80 tx=0x7f013800e4b0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.175+0000 7f0147881640 1 -- 172.21.15.161:0/3422957012 shutdown_connections 2024-03-28T23:30:18.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.175+0000 7f0147881640 1 --2- 172.21.15.161:0/3422957012 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f012406b4d0 0x7f012406d990 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.175+0000 7f0147881640 1 --2- 172.21.15.161:0/3422957012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0140156460 0x7f01400a9e20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.175+0000 7f0147881640 1 --2- 172.21.15.161:0/3422957012 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0140156080 0x7f01401de520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.175+0000 7f0147881640 1 --2- 172.21.15.161:0/3422957012 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f014006ff90 0x7f01401e01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.175+0000 7f0147881640 1 -- 172.21.15.161:0/3422957012 >> 172.21.15.161:0/3422957012 conn(0x7f014006eba0 msgr2=0x7f01400c4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:18.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.175+0000 7f0147881640 1 -- 172.21.15.161:0/3422957012 shutdown_connections 2024-03-28T23:30:18.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.176+0000 7f0147881640 1 -- 172.21.15.161:0/3422957012 wait complete. 2024-03-28T23:30:18.187 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:30:18.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.291+0000 7fc1ca302640 1 Processor -- start 2024-03-28T23:30:18.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.291+0000 7fc1ca302640 1 -- start start 2024-03-28T23:30:18.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.292+0000 7fc1ca302640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1c40c3890 0x7fc1c414b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:18.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.292+0000 7fc1ca302640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc1c40ac520 0x7fc1c414bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:18.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.292+0000 7fc1ca302640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1c406eb30 con 0x7fc1c40ab850 2024-03-28T23:30:18.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.292+0000 7fc1ca302640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc1c40c23e0 con 0x7fc1c40ac520 2024-03-28T23:30:18.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.292+0000 7fc1ca302640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc1c40c1e20 con 0x7fc1c40c3890 2024-03-28T23:30:18.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.292+0000 7fc1c889f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc1c40ac520 0x7fc1c414bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:18.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.292+0000 7fc1c3fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc1c40ab850 0x7fc1c40c06b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59768/0 (socket says 172.21.15.161:59768) 2024-03-28T23:30:18.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.292+0000 7fc1c3fff640 1 -- 172.21.15.161:0/743592960 learned_addr learned my addr 172.21.15.161:0/743592960 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:18.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.293+0000 7fc1c37fe640 1 --2- 172.21.15.161:0/743592960 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1c40c3890 0x7fc1c414b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:18.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.293+0000 7fc1c889f640 1 -- 172.21.15.161:0/743592960 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1c40c3890 msgr2=0x7fc1c414b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:18.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.293+0000 7fc1c889f640 1 --2- 172.21.15.161:0/743592960 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1c40c3890 0x7fc1c414b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.293+0000 7fc1c889f640 1 -- 172.21.15.161:0/743592960 >> v1:172.21.15.161:6789/0 conn(0x7fc1c40ab850 legacy=0x7fc1c40c06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:18.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.293+0000 7fc1c889f640 1 -- 172.21.15.161:0/743592960 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1c40c2100 con 0x7fc1c40ac520 2024-03-28T23:30:18.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.293+0000 7fc1c37fe640 1 --2- 172.21.15.161:0/743592960 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1c40c3890 0x7fc1c414b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:18.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.293+0000 7fc1c889f640 1 --2- 172.21.15.161:0/743592960 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1c40ac520 0x7fc1c414bbf0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc009d90 tx=0x7fc1bc02bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b914e4c0b11badd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:18.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.294+0000 7fc1c2ffd640 1 -- 172.21.15.161:0/743592960 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc039070 con 0x7fc1c40ac520 2024-03-28T23:30:18.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.294+0000 7fc1c2ffd640 1 -- 172.21.15.161:0/743592960 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1bc02cb30 con 0x7fc1c40ac520 2024-03-28T23:30:18.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.294+0000 7fc1c2ffd640 1 -- 172.21.15.161:0/743592960 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc034440 con 0x7fc1c40ac520 2024-03-28T23:30:18.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.294+0000 7fc1ca302640 1 -- 172.21.15.161:0/743592960 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1c40ac520 msgr2=0x7fc1c414bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:18.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.294+0000 7fc1ca302640 1 --2- 172.21.15.161:0/743592960 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1c40ac520 0x7fc1c414bbf0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc009d90 tx=0x7fc1bc02bd80 comp rx=0 tx=0).stop 2024-03-28T23:30:18.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.294+0000 7fc1ca302640 1 -- 172.21.15.161:0/743592960 shutdown_connections 2024-03-28T23:30:18.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.294+0000 7fc1ca302640 1 --2- 172.21.15.161:0/743592960 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1c40ac520 0x7fc1c414bbf0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.294+0000 7fc1ca302640 1 --2- 172.21.15.161:0/743592960 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1c40c3890 0x7fc1c414b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.294+0000 7fc1ca302640 1 -- 172.21.15.161:0/743592960 >> 172.21.15.161:0/743592960 conn(0x7fc1c40c7280 msgr2=0x7fc1c40c7660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:18.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.294+0000 7fc1ca302640 1 -- 172.21.15.161:0/743592960 shutdown_connections 2024-03-28T23:30:18.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.294+0000 7fc1ca302640 1 -- 172.21.15.161:0/743592960 wait complete. 2024-03-28T23:30:18.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.296+0000 7fc1ca302640 1 Processor -- start 2024-03-28T23:30:18.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.296+0000 7fc1ca302640 1 -- start start 2024-03-28T23:30:18.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.297+0000 7fc1ca302640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1c40ab850 0x7fc1c4154410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:18.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.297+0000 7fc1ca302640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1c40ac520 0x7fc1c41eed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:18.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.297+0000 7fc1ca302640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1c40c3890 0x7fc1c40c6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:18.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.297+0000 7fc1ca302640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c406e7c0 con 0x7fc1c40ab850 2024-03-28T23:30:18.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.297+0000 7fc1ca302640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c414c130 con 0x7fc1c40ac520 2024-03-28T23:30:18.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.297+0000 7fc1ca302640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc1c40abc30 con 0x7fc1c40c3890 2024-03-28T23:30:18.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.298+0000 7fc1c3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1c40ab850 0x7fc1c4154410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:18.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.298+0000 7fc1c3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1c40ab850 0x7fc1c4154410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54640/0 (socket says 172.21.15.161:54640) 2024-03-28T23:30:18.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.298+0000 7fc1c3fff640 1 -- 172.21.15.161:0/692156657 learned_addr learned my addr 172.21.15.161:0/692156657 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:18.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.298+0000 7fc1c37fe640 1 --2- 172.21.15.161:0/692156657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1c40ac520 0x7fc1c41eed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:18.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.298+0000 7fc1c889f640 1 --2- 172.21.15.161:0/692156657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1c40c3890 0x7fc1c40c6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:18.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.298+0000 7fc1c3fff640 1 -- 172.21.15.161:0/692156657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1c40c3890 msgr2=0x7fc1c40c6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:18.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.298+0000 7fc1c3fff640 1 --2- 172.21.15.161:0/692156657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1c40c3890 0x7fc1c40c6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.298+0000 7fc1c3fff640 1 -- 172.21.15.161:0/692156657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1c40ac520 msgr2=0x7fc1c41eed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:18.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.298+0000 7fc1c3fff640 1 --2- 172.21.15.161:0/692156657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1c40ac520 0x7fc1c41eed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.298+0000 7fc1c3fff640 1 -- 172.21.15.161:0/692156657 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1c41ef250 con 0x7fc1c40ab850 2024-03-28T23:30:18.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.298+0000 7fc1c3fff640 1 --2- 172.21.15.161:0/692156657 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1c40ab850 0x7fc1c4154410 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc1b800c400 tx=0x7fc1b800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:18.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.298+0000 7fc1c0ff9640 1 -- 172.21.15.161:0/692156657 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8017070 con 0x7fc1c40ab850 2024-03-28T23:30:18.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.298+0000 7fc1c0ff9640 1 -- 172.21.15.161:0/692156657 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1b8005ce0 con 0x7fc1c40ab850 2024-03-28T23:30:18.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.298+0000 7fc1ca302640 1 -- 172.21.15.161:0/692156657 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1c41eadc0 con 0x7fc1c40ab850 2024-03-28T23:30:18.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.298+0000 7fc1c0ff9640 1 -- 172.21.15.161:0/692156657 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b80072f0 con 0x7fc1c40ab850 2024-03-28T23:30:18.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.298+0000 7fc1ca302640 1 -- 172.21.15.161:0/692156657 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1c4153490 con 0x7fc1c40ab850 2024-03-28T23:30:18.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.299+0000 7fc1ca302640 1 -- 172.21.15.161:0/692156657 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1c404f4b0 con 0x7fc1c40ab850 2024-03-28T23:30:18.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.300+0000 7fc1c0ff9640 1 -- 172.21.15.161:0/692156657 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc1b801a070 con 0x7fc1c40ab850 2024-03-28T23:30:18.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.300+0000 7fc1c0ff9640 1 --2- 172.21.15.161:0/692156657 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc1b006b4d0 0x7fc1b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:18.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.301+0000 7fc1c37fe640 1 --2- 172.21.15.161:0/692156657 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc1b006b4d0 0x7fc1b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:18.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.301+0000 7fc1c0ff9640 1 -- 172.21.15.161:0/692156657 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc1b8089840 con 0x7fc1c40ab850 2024-03-28T23:30:18.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.301+0000 7fc1c37fe640 1 --2- 172.21.15.161:0/692156657 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc1b006b4d0 0x7fc1b006d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc1c40c1b80 tx=0x7fc1b4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:18.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.305+0000 7fc1c0ff9640 1 -- 172.21.15.161:0/692156657 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc1b8057dc0 con 0x7fc1c40ab850 2024-03-28T23:30:18.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.482+0000 7fc1ca302640 1 -- 172.21.15.161:0/692156657 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc1c41541f0 con 0x7fc1c40ab850 2024-03-28T23:30:18.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.482+0000 7fc1c0ff9640 1 -- 172.21.15.161:0/692156657 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7fc1b805be60 con 0x7fc1c40ab850 2024-03-28T23:30:18.483 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:30:18.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.485+0000 7fc1ca302640 1 -- 172.21.15.161:0/692156657 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc1b006b4d0 msgr2=0x7fc1b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:18.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.485+0000 7fc1ca302640 1 --2- 172.21.15.161:0/692156657 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc1b006b4d0 0x7fc1b006d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc1c40c1b80 tx=0x7fc1b4008040 comp rx=0 tx=0).stop 2024-03-28T23:30:18.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.485+0000 7fc1ca302640 1 -- 172.21.15.161:0/692156657 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1c40ab850 msgr2=0x7fc1c4154410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:18.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.485+0000 7fc1ca302640 1 --2- 172.21.15.161:0/692156657 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1c40ab850 0x7fc1c4154410 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc1b800c400 tx=0x7fc1b800c8c0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.485+0000 7fc1ca302640 1 -- 172.21.15.161:0/692156657 shutdown_connections 2024-03-28T23:30:18.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.485+0000 7fc1ca302640 1 --2- 172.21.15.161:0/692156657 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc1b006b4d0 0x7fc1b006d990 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.485+0000 7fc1ca302640 1 --2- 172.21.15.161:0/692156657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1c40c3890 0x7fc1c40c6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.485+0000 7fc1ca302640 1 --2- 172.21.15.161:0/692156657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1c40ac520 0x7fc1c41eed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.485+0000 7fc1ca302640 1 --2- 172.21.15.161:0/692156657 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1c40ab850 0x7fc1c4154410 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.485+0000 7fc1ca302640 1 -- 172.21.15.161:0/692156657 >> 172.21.15.161:0/692156657 conn(0x7fc1c40c7280 msgr2=0x7fc1c4072800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:18.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.485+0000 7fc1ca302640 1 -- 172.21.15.161:0/692156657 shutdown_connections 2024-03-28T23:30:18.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.485+0000 7fc1ca302640 1 -- 172.21.15.161:0/692156657 wait complete. 2024-03-28T23:30:18.500 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T23:30:18.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.608+0000 7f735a156640 1 Processor -- start 2024-03-28T23:30:18.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.608+0000 7f735a156640 1 -- start start 2024-03-28T23:30:18.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.608+0000 7f735a156640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f73540ca580 0x7f73540c9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:18.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.608+0000 7f735a156640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73540c6df0 con 0x7f7354155220 2024-03-28T23:30:18.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.608+0000 7f735a156640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73540c70d0 con 0x7f7354152940 2024-03-28T23:30:18.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.608+0000 7f735a156640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f73540c5830 con 0x7f73540ca580 2024-03-28T23:30:18.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.609+0000 7f7353fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f7354155220 0x7f7354155640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59774/0 (socket says 172.21.15.161:59774) 2024-03-28T23:30:18.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.609+0000 7f7353fff640 1 -- 172.21.15.161:0/2910054829 learned_addr learned my addr 172.21.15.161:0/2910054829 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:18.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.609+0000 7f7352ffd640 1 --2- 172.21.15.161:0/2910054829 >> v2:172.21.15.170:3300/0 conn(0x7f73540ca580 0x7f73540c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:18.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.609+0000 7f73527fc640 1 -- 172.21.15.161:0/2910054829 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3369405208 0 0) 0x7f73540c6df0 con 0x7f7354155220 2024-03-28T23:30:18.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.609+0000 7f73527fc640 1 -- 172.21.15.161:0/2910054829 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f733c003520 con 0x7f7354155220 2024-03-28T23:30:18.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.609+0000 7f73527fc640 1 -- 172.21.15.161:0/2910054829 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1089639709 0 0) 0x7f733c003520 con 0x7f7354155220 2024-03-28T23:30:18.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.609+0000 7f73527fc640 1 -- 172.21.15.161:0/2910054829 >> v2:172.21.15.170:3300/0 conn(0x7f73540ca580 msgr2=0x7f73540c9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:18.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.609+0000 7f73527fc640 1 --2- 172.21.15.161:0/2910054829 >> v2:172.21.15.170:3300/0 conn(0x7f73540ca580 0x7f73540c9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.609+0000 7f73527fc640 1 -- 172.21.15.161:0/2910054829 >> v1:172.21.15.170:6789/0 conn(0x7f7354152940 legacy=0x7f73540c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:18.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.609+0000 7f73527fc640 1 -- 172.21.15.161:0/2910054829 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73540c5b10 con 0x7f7354155220 2024-03-28T23:30:18.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.609+0000 7f73527fc640 1 -- 172.21.15.161:0/2910054829 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f734c0019f0 con 0x7f7354155220 2024-03-28T23:30:18.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.609+0000 7f73527fc640 1 -- 172.21.15.161:0/2910054829 >> v1:172.21.15.161:6789/0 conn(0x7f7354155220 legacy=0x7f7354155640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:18.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.609+0000 7f73527fc640 1 --2- 172.21.15.161:0/2910054829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f733c003a90 0x7f733c003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:18.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.609+0000 7f73527fc640 1 --2- 172.21.15.161:0/2910054829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f733c004800 0x7f733c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:18.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.609+0000 7f73527fc640 1 --2- 172.21.15.161:0/2910054829 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f733c0053c0 0x7f733c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:18.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.609+0000 7f73527fc640 1 -- 172.21.15.161:0/2910054829 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f73540c5830 con 0x7f733c003a90 2024-03-28T23:30:18.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.609+0000 7f73527fc640 1 -- 172.21.15.161:0/2910054829 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f733c00c280 con 0x7f733c004800 2024-03-28T23:30:18.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.609+0000 7f73527fc640 1 -- 172.21.15.161:0/2910054829 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f733c00c480 con 0x7f733c0053c0 2024-03-28T23:30:18.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.609+0000 7f73537fe640 1 --2- 172.21.15.161:0/2910054829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f733c003a90 0x7f733c003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:18.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.609+0000 7f7352ffd640 1 --2- 172.21.15.161:0/2910054829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f733c004800 0x7f733c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:18.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.610+0000 7f7353fff640 1 --2- 172.21.15.161:0/2910054829 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f733c0053c0 0x7f733c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:18.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.610+0000 7f73537fe640 1 -- 172.21.15.161:0/2910054829 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f733c0053c0 msgr2=0x7f733c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:18.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.610+0000 7f73537fe640 1 --2- 172.21.15.161:0/2910054829 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f733c0053c0 0x7f733c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.610+0000 7f73537fe640 1 -- 172.21.15.161:0/2910054829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f733c004800 msgr2=0x7f733c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:18.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.610+0000 7f73537fe640 1 --2- 172.21.15.161:0/2910054829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f733c004800 0x7f733c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.610+0000 7f73537fe640 1 -- 172.21.15.161:0/2910054829 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f733c003520 con 0x7f733c003a90 2024-03-28T23:30:18.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.610+0000 7f7353fff640 1 -- 172.21.15.161:0/2910054829 reap_dead start 2024-03-28T23:30:18.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.610+0000 7f7352ffd640 1 --2- 172.21.15.161:0/2910054829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f733c004800 0x7f733c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:18.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.610+0000 7f73537fe640 1 --2- 172.21.15.161:0/2910054829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f733c003a90 0x7f733c003eb0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7344001750 tx=0x7f733c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ace7759eec1b1b82 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:18.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.610+0000 7f73527fc640 1 -- 172.21.15.161:0/2910054829 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7344001370 con 0x7f733c003a90 2024-03-28T23:30:18.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.610+0000 7f73527fc640 1 -- 172.21.15.161:0/2910054829 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73440052d0 con 0x7f733c003a90 2024-03-28T23:30:18.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.611+0000 7f73527fc640 1 -- 172.21.15.161:0/2910054829 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7344003830 con 0x7f733c003a90 2024-03-28T23:30:18.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.611+0000 7f735a156640 1 -- 172.21.15.161:0/2910054829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f733c003a90 msgr2=0x7f733c003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:18.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.611+0000 7f735a156640 1 --2- 172.21.15.161:0/2910054829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f733c003a90 0x7f733c003eb0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7344001750 tx=0x7f733c005830 comp rx=0 tx=0).stop 2024-03-28T23:30:18.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.611+0000 7f735a156640 1 -- 172.21.15.161:0/2910054829 shutdown_connections 2024-03-28T23:30:18.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.611+0000 7f735a156640 1 --2- 172.21.15.161:0/2910054829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f733c003a90 0x7f733c003eb0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.611+0000 7f735a156640 1 -- 172.21.15.161:0/2910054829 >> 172.21.15.161:0/2910054829 conn(0x7f73540caa00 msgr2=0x7f73540cae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:18.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.611+0000 7f735a156640 1 -- 172.21.15.161:0/2910054829 shutdown_connections 2024-03-28T23:30:18.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.611+0000 7f735a156640 1 -- 172.21.15.161:0/2910054829 wait complete. 2024-03-28T23:30:18.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.612+0000 7f735a156640 1 Processor -- start 2024-03-28T23:30:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.612+0000 7f735a156640 1 -- start start 2024-03-28T23:30:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.613+0000 7f735a156640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f733c003a90 0x7f73541ec860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.613+0000 7f735a156640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73540af210 0x7f73541eb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.613+0000 7f735a156640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73541ebbb0 0x7f7354147160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.613+0000 7f735a156640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f73540c61c0 con 0x7f733c003a90 2024-03-28T23:30:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.613+0000 7f735a156640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f73540c6040 con 0x7f73540af210 2024-03-28T23:30:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.613+0000 7f735a156640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7354155640 con 0x7f73541ebbb0 2024-03-28T23:30:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.613+0000 7f73537fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f733c003a90 0x7f73541ec860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.613+0000 7f73537fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f733c003a90 0x7f73541ec860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54660/0 (socket says 172.21.15.161:54660) 2024-03-28T23:30:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.613+0000 7f73537fe640 1 -- 172.21.15.161:0/3514572026 learned_addr learned my addr 172.21.15.161:0/3514572026 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.613+0000 7f7352ffd640 1 --2- 172.21.15.161:0/3514572026 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73540af210 0x7f73541eb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.613+0000 7f7353fff640 1 --2- 172.21.15.161:0/3514572026 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73541ebbb0 0x7f7354147160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.613+0000 7f73537fe640 1 -- 172.21.15.161:0/3514572026 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73541ebbb0 msgr2=0x7f7354147160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.613+0000 7f73537fe640 1 --2- 172.21.15.161:0/3514572026 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73541ebbb0 0x7f7354147160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.613+0000 7f73537fe640 1 -- 172.21.15.161:0/3514572026 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73540af210 msgr2=0x7f73541eb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.613+0000 7f73537fe640 1 --2- 172.21.15.161:0/3514572026 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73540af210 0x7f73541eb670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.613+0000 7f73537fe640 1 -- 172.21.15.161:0/3514572026 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73541ecda0 con 0x7f733c003a90 2024-03-28T23:30:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.614+0000 7f73537fe640 1 --2- 172.21.15.161:0/3514572026 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f733c003a90 0x7f73541ec860 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7344000c00 tx=0x7f7344001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.614+0000 7f733bfff640 1 -- 172.21.15.161:0/3514572026 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7344041020 con 0x7f733c003a90 2024-03-28T23:30:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.614+0000 7f733bfff640 1 -- 172.21.15.161:0/3514572026 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7344005480 con 0x7f733c003a90 2024-03-28T23:30:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.614+0000 7f733bfff640 1 -- 172.21.15.161:0/3514572026 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7344007a30 con 0x7f733c003a90 2024-03-28T23:30:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.614+0000 7f735a156640 1 -- 172.21.15.161:0/3514572026 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73540c1c70 con 0x7f733c003a90 2024-03-28T23:30:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.614+0000 7f735a156640 1 -- 172.21.15.161:0/3514572026 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73540aea10 con 0x7f733c003a90 2024-03-28T23:30:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.615+0000 7f735a156640 1 -- 172.21.15.161:0/3514572026 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f735406da50 con 0x7f733c003a90 2024-03-28T23:30:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.615+0000 7f733bfff640 1 -- 172.21.15.161:0/3514572026 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7344044020 con 0x7f733c003a90 2024-03-28T23:30:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.616+0000 7f733bfff640 1 --2- 172.21.15.161:0/3514572026 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f734006b500 0x7f734006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.616+0000 7f733bfff640 1 -- 172.21.15.161:0/3514572026 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f73440abe90 con 0x7f733c003a90 2024-03-28T23:30:18.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.616+0000 7f7352ffd640 1 --2- 172.21.15.161:0/3514572026 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f734006b500 0x7f734006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:18.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.616+0000 7f7352ffd640 1 --2- 172.21.15.161:0/3514572026 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f734006b500 0x7f734006d9c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f73541e91d0 tx=0x7f7348008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:18.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.620+0000 7f733bfff640 1 -- 172.21.15.161:0/3514572026 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f735406da50 con 0x7f733c003a90 2024-03-28T23:30:18.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.801+0000 7f735a156640 1 -- 172.21.15.161:0/3514572026 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f735404f4b0 con 0x7f733c003a90 2024-03-28T23:30:18.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.801+0000 7f733bfff640 1 -- 172.21.15.161:0/3514572026 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f734404b030 con 0x7f733c003a90 2024-03-28T23:30:18.802 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:30:18.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.803+0000 7f735a156640 1 -- 172.21.15.161:0/3514572026 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f734006b500 msgr2=0x7f734006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:18.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.803+0000 7f735a156640 1 --2- 172.21.15.161:0/3514572026 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f734006b500 0x7f734006d9c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f73541e91d0 tx=0x7f7348008040 comp rx=0 tx=0).stop 2024-03-28T23:30:18.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.803+0000 7f735a156640 1 -- 172.21.15.161:0/3514572026 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f733c003a90 msgr2=0x7f73541ec860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:18.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.803+0000 7f735a156640 1 --2- 172.21.15.161:0/3514572026 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f733c003a90 0x7f73541ec860 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7344000c00 tx=0x7f7344001120 comp rx=0 tx=0).stop 2024-03-28T23:30:18.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.803+0000 7f735a156640 1 -- 172.21.15.161:0/3514572026 shutdown_connections 2024-03-28T23:30:18.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.803+0000 7f735a156640 1 --2- 172.21.15.161:0/3514572026 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f734006b500 0x7f734006d9c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.803+0000 7f735a156640 1 --2- 172.21.15.161:0/3514572026 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73541ebbb0 0x7f7354147160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.803+0000 7f735a156640 1 --2- 172.21.15.161:0/3514572026 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73540af210 0x7f73541eb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.803+0000 7f735a156640 1 --2- 172.21.15.161:0/3514572026 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f733c003a90 0x7f73541ec860 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.803+0000 7f735a156640 1 -- 172.21.15.161:0/3514572026 >> 172.21.15.161:0/3514572026 conn(0x7f73540caa00 msgr2=0x7f7354153a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:18.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.803+0000 7f735a156640 1 -- 172.21.15.161:0/3514572026 shutdown_connections 2024-03-28T23:30:18.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.803+0000 7f735a156640 1 -- 172.21.15.161:0/3514572026 wait complete. 2024-03-28T23:30:18.815 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:30:18.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.917+0000 7f415a7d6640 1 Processor -- start 2024-03-28T23:30:18.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.917+0000 7f415a7d6640 1 -- start start 2024-03-28T23:30:18.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.917+0000 7f415a7d6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4154072810 0x7f41540c4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:18.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.917+0000 7f415a7d6640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4154070370 0x7f41540c3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:18.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.917+0000 7f415a7d6640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41540707e0 con 0x7f41540c4d80 2024-03-28T23:30:18.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.917+0000 7f415a7d6640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f41540a8520 con 0x7f4154070370 2024-03-28T23:30:18.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.917+0000 7f415a7d6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4154153050 con 0x7f4154072810 2024-03-28T23:30:18.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.917+0000 7f4158d73640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4154070370 0x7f41540c3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:18.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.918+0000 7f4158d73640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4154070370 0x7f41540c3390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54676/0 (socket says 172.21.15.161:54676) 2024-03-28T23:30:18.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.918+0000 7f4158d73640 1 -- 172.21.15.161:0/2489490012 learned_addr learned my addr 172.21.15.161:0/2489490012 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:18.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.918+0000 7f4153fff640 1 --2- 172.21.15.161:0/2489490012 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4154072810 0x7f41540c4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:18.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.918+0000 7f4158d73640 1 -- 172.21.15.161:0/2489490012 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4154072810 msgr2=0x7f41540c4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:18.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.918+0000 7f4158d73640 1 --2- 172.21.15.161:0/2489490012 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4154072810 0x7f41540c4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.918+0000 7f4158d73640 1 -- 172.21.15.161:0/2489490012 >> v1:172.21.15.170:6789/0 conn(0x7f41540c4d80 legacy=0x7f415406fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:18.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.918+0000 7f4158d73640 1 -- 172.21.15.161:0/2489490012 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4154153310 con 0x7f4154070370 2024-03-28T23:30:18.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.918+0000 7f4153fff640 1 --2- 172.21.15.161:0/2489490012 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4154072810 0x7f41540c4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:18.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.919+0000 7f4158d73640 1 --2- 172.21.15.161:0/2489490012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4154070370 0x7f41540c3390 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f414c009e50 tx=0x7f414c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=729f4fc5a7e0dd68 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:18.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.919+0000 7f4152ffd640 1 -- 172.21.15.161:0/2489490012 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c02c9e0 con 0x7f4154070370 2024-03-28T23:30:18.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.919+0000 7f4152ffd640 1 -- 172.21.15.161:0/2489490012 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f414c02cb80 con 0x7f4154070370 2024-03-28T23:30:18.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.919+0000 7f4152ffd640 1 -- 172.21.15.161:0/2489490012 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c0343f0 con 0x7f4154070370 2024-03-28T23:30:18.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.919+0000 7f415a7d6640 1 -- 172.21.15.161:0/2489490012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4154070370 msgr2=0x7f41540c3390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:18.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.919+0000 7f415a7d6640 1 --2- 172.21.15.161:0/2489490012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4154070370 0x7f41540c3390 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f414c009e50 tx=0x7f414c02be40 comp rx=0 tx=0).stop 2024-03-28T23:30:18.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.919+0000 7f415a7d6640 1 -- 172.21.15.161:0/2489490012 shutdown_connections 2024-03-28T23:30:18.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.919+0000 7f415a7d6640 1 --2- 172.21.15.161:0/2489490012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4154070370 0x7f41540c3390 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.919+0000 7f415a7d6640 1 --2- 172.21.15.161:0/2489490012 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4154072810 0x7f41540c4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.919+0000 7f415a7d6640 1 -- 172.21.15.161:0/2489490012 >> 172.21.15.161:0/2489490012 conn(0x7f41540aa740 msgr2=0x7f41540aab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:18.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.919+0000 7f415a7d6640 1 -- 172.21.15.161:0/2489490012 shutdown_connections 2024-03-28T23:30:18.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.919+0000 7f415a7d6640 1 -- 172.21.15.161:0/2489490012 wait complete. 2024-03-28T23:30:18.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.921+0000 7f415a7d6640 1 Processor -- start 2024-03-28T23:30:18.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.921+0000 7f415a7d6640 1 -- start start 2024-03-28T23:30:18.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.922+0000 7f415a7d6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4154070370 0x7f4154146860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:18.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.922+0000 7f415a7d6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4154072810 0x7f415414b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:18.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.922+0000 7f415a7d6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41540c4d80 0x7f41541e58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:18.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.922+0000 7f415a7d6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4154153bb0 con 0x7f4154070370 2024-03-28T23:30:18.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.922+0000 7f415a7d6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f41541538d0 con 0x7f4154072810 2024-03-28T23:30:18.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.922+0000 7f415a7d6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4154072bf0 con 0x7f41540c4d80 2024-03-28T23:30:18.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.922+0000 7f4153fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4154070370 0x7f4154146860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:18.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.922+0000 7f4153fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4154070370 0x7f4154146860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54682/0 (socket says 172.21.15.161:54682) 2024-03-28T23:30:18.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.922+0000 7f4153fff640 1 -- 172.21.15.161:0/2720688787 learned_addr learned my addr 172.21.15.161:0/2720688787 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:18.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.922+0000 7f4158d73640 1 --2- 172.21.15.161:0/2720688787 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41540c4d80 0x7f41541e58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:18.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.922+0000 7f41537fe640 1 --2- 172.21.15.161:0/2720688787 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4154072810 0x7f415414b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:18.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.922+0000 7f4153fff640 1 -- 172.21.15.161:0/2720688787 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41540c4d80 msgr2=0x7f41541e58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:18.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.922+0000 7f4153fff640 1 --2- 172.21.15.161:0/2720688787 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41540c4d80 0x7f41541e58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.922+0000 7f4153fff640 1 -- 172.21.15.161:0/2720688787 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4154072810 msgr2=0x7f415414b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:18.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.922+0000 7f4153fff640 1 --2- 172.21.15.161:0/2720688787 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4154072810 0x7f415414b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:18.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.922+0000 7f4153fff640 1 -- 172.21.15.161:0/2720688787 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4154146da0 con 0x7f4154070370 2024-03-28T23:30:18.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.923+0000 7f4153fff640 1 --2- 172.21.15.161:0/2720688787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4154070370 0x7f4154146860 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f414400c7f0 tx=0x7f414400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:18.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.923+0000 7f4150ff9640 1 -- 172.21.15.161:0/2720688787 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4144015070 con 0x7f4154070370 2024-03-28T23:30:18.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.923+0000 7f4150ff9640 1 -- 172.21.15.161:0/2720688787 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41440079d0 con 0x7f4154070370 2024-03-28T23:30:18.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.923+0000 7f4150ff9640 1 -- 172.21.15.161:0/2720688787 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4144007da0 con 0x7f4154070370 2024-03-28T23:30:18.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.923+0000 7f415a7d6640 1 -- 172.21.15.161:0/2720688787 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41541e86c0 con 0x7f4154070370 2024-03-28T23:30:18.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.923+0000 7f415a7d6640 1 -- 172.21.15.161:0/2720688787 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41541e8c80 con 0x7f4154070370 2024-03-28T23:30:18.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.924+0000 7f415a7d6640 1 -- 172.21.15.161:0/2720688787 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f415406e470 con 0x7f4154070370 2024-03-28T23:30:18.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.924+0000 7f4150ff9640 1 -- 172.21.15.161:0/2720688787 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4144018070 con 0x7f4154070370 2024-03-28T23:30:18.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.925+0000 7f4150ff9640 1 --2- 172.21.15.161:0/2720688787 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f414006b4d0 0x7f414006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:18.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.925+0000 7f4150ff9640 1 -- 172.21.15.161:0/2720688787 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f414408a440 con 0x7f4154070370 2024-03-28T23:30:18.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.925+0000 7f41537fe640 1 --2- 172.21.15.161:0/2720688787 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f414006b4d0 0x7f414006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:18.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.925+0000 7f41537fe640 1 --2- 172.21.15.161:0/2720688787 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f414006b4d0 0x7f414006d990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f41540ac040 tx=0x7f4148009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:18.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:18.929+0000 7f4150ff9640 1 -- 172.21.15.161:0/2720688787 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f415406e470 con 0x7f4154070370 2024-03-28T23:30:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.144+0000 7f415a7d6640 1 -- 172.21.15.161:0/2720688787 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f415404f4b0 con 0x7f4154070370 2024-03-28T23:30:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.144+0000 7f4150ff9640 1 -- 172.21.15.161:0/2720688787 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5189 (secure 0 0 0) 0x7f4144059010 con 0x7f4154070370 2024-03-28T23:30:19.146 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:19.146 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/1577284938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":1577284938},{"type":"v1","addr":"172.21.15.170:6835","nonce":1577284938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4258,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/1107796931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1107796931},{"type":"v1","addr":"172.21.15.170:6834","nonce":1107796931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4107992135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4107992135},{"type":"v1","addr":"172.21.15.161:6836","nonce":4107992135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:29:45.903844+0000","modified":"2024-03-28T23:30:07.984387+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6837/3832540545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3832540545},{"type":"v1","addr":"172.21.15.161:6837","nonce":3832540545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4275,"qdb_cluster":[4275]},"id":1}]} 2024-03-28T23:30:19.146 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 9 2024-03-28T23:30:19.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.146+0000 7f415a7d6640 1 -- 172.21.15.161:0/2720688787 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f414006b4d0 msgr2=0x7f414006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:19.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.146+0000 7f415a7d6640 1 --2- 172.21.15.161:0/2720688787 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f414006b4d0 0x7f414006d990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f41540ac040 tx=0x7f4148009040 comp rx=0 tx=0).stop 2024-03-28T23:30:19.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.146+0000 7f415a7d6640 1 -- 172.21.15.161:0/2720688787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4154070370 msgr2=0x7f4154146860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:19.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.146+0000 7f415a7d6640 1 --2- 172.21.15.161:0/2720688787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4154070370 0x7f4154146860 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f414400c7f0 tx=0x7f414400cbf0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.146+0000 7f415a7d6640 1 -- 172.21.15.161:0/2720688787 shutdown_connections 2024-03-28T23:30:19.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.146+0000 7f415a7d6640 1 --2- 172.21.15.161:0/2720688787 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f414006b4d0 0x7f414006d990 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.146+0000 7f415a7d6640 1 --2- 172.21.15.161:0/2720688787 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41540c4d80 0x7f41541e58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.146+0000 7f415a7d6640 1 --2- 172.21.15.161:0/2720688787 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4154072810 0x7f415414b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.146+0000 7f415a7d6640 1 --2- 172.21.15.161:0/2720688787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4154070370 0x7f4154146860 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.146+0000 7f415a7d6640 1 -- 172.21.15.161:0/2720688787 >> 172.21.15.161:0/2720688787 conn(0x7f41540aa740 msgr2=0x7f41540aab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:19.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.147+0000 7f415a7d6640 1 -- 172.21.15.161:0/2720688787 shutdown_connections 2024-03-28T23:30:19.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.147+0000 7f415a7d6640 1 -- 172.21.15.161:0/2720688787 wait complete. 2024-03-28T23:30:19.159 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:30:19.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.261+0000 7f9002040640 1 Processor -- start 2024-03-28T23:30:19.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.261+0000 7f9002040640 1 -- start start 2024-03-28T23:30:19.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.261+0000 7f9002040640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ffc152890 0x7f8ffc0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:19.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.261+0000 7f9002040640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8ffc0c5b20 0x7f8ffc1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:19.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.261+0000 7f9002040640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ffc158a30 con 0x7f8ffc0c8ed0 2024-03-28T23:30:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.261+0000 7f9002040640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8ffc0c8480 con 0x7f8ffc0c5b20 2024-03-28T23:30:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.261+0000 7f9002040640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8ffc159640 con 0x7f8ffc152890 2024-03-28T23:30:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.262+0000 7f8ffbfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8ffc0c5b20 0x7f8ffc1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.262+0000 7f8ffbfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8ffc0c5b20 0x7f8ffc1553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54696/0 (socket says 172.21.15.161:54696) 2024-03-28T23:30:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.262+0000 7f8ffbfff640 1 -- 172.21.15.161:0/1308409801 learned_addr learned my addr 172.21.15.161:0/1308409801 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.262+0000 7f8ffb7fe640 1 --2- 172.21.15.161:0/1308409801 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ffc152890 0x7f8ffc0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.262+0000 7f8ffbfff640 1 -- 172.21.15.161:0/1308409801 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ffc152890 msgr2=0x7f8ffc0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.262+0000 7f8ffbfff640 1 --2- 172.21.15.161:0/1308409801 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ffc152890 0x7f8ffc0c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.262+0000 7f8ffbfff640 1 -- 172.21.15.161:0/1308409801 >> v1:172.21.15.170:6789/0 conn(0x7f8ffc0c8ed0 legacy=0x7f8ffc0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.262+0000 7f8ffbfff640 1 -- 172.21.15.161:0/1308409801 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ffc158d10 con 0x7f8ffc0c5b20 2024-03-28T23:30:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.263+0000 7f8ffbfff640 1 --2- 172.21.15.161:0/1308409801 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ffc0c5b20 0x7f8ffc1553a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc1597c0 tx=0x7f8ff402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88eb11ed5e220f24 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.263+0000 7f8ffa7fc640 1 -- 172.21.15.161:0/1308409801 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff402c9c0 con 0x7f8ffc0c5b20 2024-03-28T23:30:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.263+0000 7f8ffa7fc640 1 -- 172.21.15.161:0/1308409801 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ff402cb60 con 0x7f8ffc0c5b20 2024-03-28T23:30:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.263+0000 7f8ffa7fc640 1 -- 172.21.15.161:0/1308409801 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff40343f0 con 0x7f8ffc0c5b20 2024-03-28T23:30:19.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.264+0000 7f9002040640 1 -- 172.21.15.161:0/1308409801 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ffc0c5b20 msgr2=0x7f8ffc1553a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:19.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.264+0000 7f9002040640 1 --2- 172.21.15.161:0/1308409801 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ffc0c5b20 0x7f8ffc1553a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc1597c0 tx=0x7f8ff402be20 comp rx=0 tx=0).stop 2024-03-28T23:30:19.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.264+0000 7f9002040640 1 -- 172.21.15.161:0/1308409801 shutdown_connections 2024-03-28T23:30:19.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.264+0000 7f9002040640 1 --2- 172.21.15.161:0/1308409801 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ffc0c5b20 0x7f8ffc1553a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.264+0000 7f9002040640 1 --2- 172.21.15.161:0/1308409801 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ffc152890 0x7f8ffc0c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.264+0000 7f9002040640 1 -- 172.21.15.161:0/1308409801 >> 172.21.15.161:0/1308409801 conn(0x7f8ffc0ac620 msgr2=0x7f8ffc0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:19.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.264+0000 7f9002040640 1 -- 172.21.15.161:0/1308409801 shutdown_connections 2024-03-28T23:30:19.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.264+0000 7f9002040640 1 -- 172.21.15.161:0/1308409801 wait complete. 2024-03-28T23:30:19.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.265+0000 7f9002040640 1 Processor -- start 2024-03-28T23:30:19.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.265+0000 7f9002040640 1 -- start start 2024-03-28T23:30:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.266+0000 7f9002040640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ffc0c5b20 0x7f8ffc14ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.266+0000 7f9002040640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ffc0c8ed0 0x7f8ffc146ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.266+0000 7f9002040640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ffc152890 0x7f8ffc147420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.266+0000 7f9002040640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8ffc152c70 con 0x7f8ffc0c5b20 2024-03-28T23:30:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.266+0000 7f9002040640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8ffc1592d0 con 0x7f8ffc152890 2024-03-28T23:30:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.266+0000 7f9002040640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8ffc152df0 con 0x7f8ffc0c8ed0 2024-03-28T23:30:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.266+0000 7f8ffb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ffc0c5b20 0x7f8ffc14ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.266+0000 7f8ffb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ffc0c5b20 0x7f8ffc14ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54700/0 (socket says 172.21.15.161:54700) 2024-03-28T23:30:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.266+0000 7f8ffb7fe640 1 -- 172.21.15.161:0/960561728 learned_addr learned my addr 172.21.15.161:0/960561728 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.266+0000 7f8ffaffd640 1 --2- 172.21.15.161:0/960561728 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ffc0c8ed0 0x7f8ffc146ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.266+0000 7f8ffbfff640 1 --2- 172.21.15.161:0/960561728 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ffc152890 0x7f8ffc147420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.266+0000 7f8ffb7fe640 1 -- 172.21.15.161:0/960561728 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ffc0c8ed0 msgr2=0x7f8ffc146ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.266+0000 7f8ffb7fe640 1 --2- 172.21.15.161:0/960561728 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ffc0c8ed0 0x7f8ffc146ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.266+0000 7f8ffb7fe640 1 -- 172.21.15.161:0/960561728 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ffc152890 msgr2=0x7f8ffc147420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.266+0000 7f8ffb7fe640 1 --2- 172.21.15.161:0/960561728 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ffc152890 0x7f8ffc147420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.266+0000 7f8ffb7fe640 1 -- 172.21.15.161:0/960561728 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ffc158740 con 0x7f8ffc0c5b20 2024-03-28T23:30:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.266+0000 7f8ffaffd640 1 --2- 172.21.15.161:0/960561728 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ffc0c8ed0 0x7f8ffc146ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.266+0000 7f8ffbfff640 1 --2- 172.21.15.161:0/960561728 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ffc152890 0x7f8ffc147420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:19.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.266+0000 7f8ffb7fe640 1 --2- 172.21.15.161:0/960561728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ffc0c5b20 0x7f8ffc14ea40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8fec00d800 tx=0x7f8fec00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:19.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.267+0000 7f8fe3fff640 1 -- 172.21.15.161:0/960561728 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fec00b6c0 con 0x7f8ffc0c5b20 2024-03-28T23:30:19.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.267+0000 7f8fe3fff640 1 -- 172.21.15.161:0/960561728 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fec019070 con 0x7f8ffc0c5b20 2024-03-28T23:30:19.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.267+0000 7f9002040640 1 -- 172.21.15.161:0/960561728 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ffc148230 con 0x7f8ffc0c5b20 2024-03-28T23:30:19.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.267+0000 7f8fe3fff640 1 -- 172.21.15.161:0/960561728 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fec00bac0 con 0x7f8ffc0c5b20 2024-03-28T23:30:19.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.267+0000 7f9002040640 1 -- 172.21.15.161:0/960561728 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ffc145000 con 0x7f8ffc0c5b20 2024-03-28T23:30:19.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.268+0000 7f9002040640 1 -- 172.21.15.161:0/960561728 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ffc04f4b0 con 0x7f8ffc0c5b20 2024-03-28T23:30:19.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.268+0000 7f8fe3fff640 1 -- 172.21.15.161:0/960561728 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8fec00bc60 con 0x7f8ffc0c5b20 2024-03-28T23:30:19.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.269+0000 7f8fe3fff640 1 --2- 172.21.15.161:0/960561728 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8fe806b4d0 0x7f8fe806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:19.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.269+0000 7f8ffaffd640 1 --2- 172.21.15.161:0/960561728 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8fe806b4d0 0x7f8fe806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:19.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.269+0000 7f8fe3fff640 1 -- 172.21.15.161:0/960561728 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f8fec08a990 con 0x7f8ffc0c5b20 2024-03-28T23:30:19.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.269+0000 7f8ffaffd640 1 --2- 172.21.15.161:0/960561728 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8fe806b4d0 0x7f8fe806d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc1522a0 tx=0x7f8ff0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:19.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.273+0000 7f8fe3fff640 1 -- 172.21.15.161:0/960561728 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8fec059560 con 0x7f8ffc0c5b20 2024-03-28T23:30:19.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.453+0000 7f9002040640 1 -- 172.21.15.161:0/960561728 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8ffc14c410 con 0x7f8ffc0c5b20 2024-03-28T23:30:19.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.454+0000 7f8fe3fff640 1 -- 172.21.15.161:0/960561728 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7f8fec058e60 con 0x7f8ffc0c5b20 2024-03-28T23:30:19.455 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:19.455 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":24,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:29:50.946843+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:30:19.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.456+0000 7f9002040640 1 -- 172.21.15.161:0/960561728 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8fe806b4d0 msgr2=0x7f8fe806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:19.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.456+0000 7f9002040640 1 --2- 172.21.15.161:0/960561728 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8fe806b4d0 0x7f8fe806d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc1522a0 tx=0x7f8ff0009040 comp rx=0 tx=0).stop 2024-03-28T23:30:19.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.456+0000 7f9002040640 1 -- 172.21.15.161:0/960561728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ffc0c5b20 msgr2=0x7f8ffc14ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:19.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.456+0000 7f9002040640 1 --2- 172.21.15.161:0/960561728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ffc0c5b20 0x7f8ffc14ea40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8fec00d800 tx=0x7f8fec00dcc0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.456+0000 7f9002040640 1 -- 172.21.15.161:0/960561728 shutdown_connections 2024-03-28T23:30:19.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.456+0000 7f9002040640 1 --2- 172.21.15.161:0/960561728 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8fe806b4d0 0x7f8fe806d990 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.456+0000 7f9002040640 1 --2- 172.21.15.161:0/960561728 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ffc152890 0x7f8ffc147420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.456+0000 7f9002040640 1 --2- 172.21.15.161:0/960561728 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ffc0c8ed0 0x7f8ffc146ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.456+0000 7f9002040640 1 --2- 172.21.15.161:0/960561728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ffc0c5b20 0x7f8ffc14ea40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.456+0000 7f9002040640 1 -- 172.21.15.161:0/960561728 >> 172.21.15.161:0/960561728 conn(0x7f8ffc0ac620 msgr2=0x7f8ffc06cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:19.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.457+0000 7f9002040640 1 -- 172.21.15.161:0/960561728 shutdown_connections 2024-03-28T23:30:19.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.457+0000 7f9002040640 1 -- 172.21.15.161:0/960561728 wait complete. 2024-03-28T23:30:19.469 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:30:19.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.570+0000 7fa2c9264640 1 Processor -- start 2024-03-28T23:30:19.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.570+0000 7fa2c9264640 1 -- start start 2024-03-28T23:30:19.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.570+0000 7fa2c9264640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa2c4152890 0x7fa2c40c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:19.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.570+0000 7fa2c9264640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa2c40c8ed0 0x7fa2c40c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:19.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.570+0000 7fa2c9264640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2c4158a30 con 0x7fa2c40c5950 2024-03-28T23:30:19.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.570+0000 7fa2c9264640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa2c40c8480 con 0x7fa2c40c8ed0 2024-03-28T23:30:19.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.570+0000 7fa2c9264640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fa2c4159640 con 0x7fa2c4152890 2024-03-28T23:30:19.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.570+0000 7fa2bbfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa2c40c8ed0 0x7fa2c40c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:19.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.570+0000 7fa2bbfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa2c40c8ed0 0x7fa2c40c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54702/0 (socket says 172.21.15.161:54702) 2024-03-28T23:30:19.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.570+0000 7fa2bbfff640 1 -- 172.21.15.161:0/2975487755 learned_addr learned my addr 172.21.15.161:0/2975487755 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:19.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.571+0000 7fa2c2ffd640 1 --2- 172.21.15.161:0/2975487755 >> v2:172.21.15.170:3300/0 conn(0x7fa2c4152890 0x7fa2c40c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.571+0000 7fa2bbfff640 1 -- 172.21.15.161:0/2975487755 >> v2:172.21.15.170:3300/0 conn(0x7fa2c4152890 msgr2=0x7fa2c40c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.571+0000 7fa2bbfff640 1 --2- 172.21.15.161:0/2975487755 >> v2:172.21.15.170:3300/0 conn(0x7fa2c4152890 0x7fa2c40c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.571+0000 7fa2bbfff640 1 -- 172.21.15.161:0/2975487755 >> v1:172.21.15.170:6789/0 conn(0x7fa2c40c5950 legacy=0x7fa2c41553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.571+0000 7fa2bbfff640 1 -- 172.21.15.161:0/2975487755 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2c4158d10 con 0x7fa2c40c8ed0 2024-03-28T23:30:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.572+0000 7fa2bbfff640 1 --2- 172.21.15.161:0/2975487755 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2c40c8ed0 0x7fa2c40c5410 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa2c408cc80 tx=0x7fa2ac02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1019acd2cadc4029 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.572+0000 7fa2c27fc640 1 -- 172.21.15.161:0/2975487755 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac038070 con 0x7fa2c40c8ed0 2024-03-28T23:30:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.572+0000 7fa2c27fc640 1 -- 172.21.15.161:0/2975487755 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2ac004230 con 0x7fa2c40c8ed0 2024-03-28T23:30:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.572+0000 7fa2c27fc640 1 -- 172.21.15.161:0/2975487755 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac02f050 con 0x7fa2c40c8ed0 2024-03-28T23:30:19.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.573+0000 7fa2c9264640 1 -- 172.21.15.161:0/2975487755 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2c40c8ed0 msgr2=0x7fa2c40c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:19.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.573+0000 7fa2c9264640 1 --2- 172.21.15.161:0/2975487755 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2c40c8ed0 0x7fa2c40c5410 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa2c408cc80 tx=0x7fa2ac02b7e0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.573+0000 7fa2c9264640 1 -- 172.21.15.161:0/2975487755 shutdown_connections 2024-03-28T23:30:19.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.573+0000 7fa2c9264640 1 --2- 172.21.15.161:0/2975487755 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2c40c8ed0 0x7fa2c40c5410 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.573+0000 7fa2c9264640 1 --2- 172.21.15.161:0/2975487755 >> v2:172.21.15.170:3300/0 conn(0x7fa2c4152890 0x7fa2c40c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.573+0000 7fa2c9264640 1 -- 172.21.15.161:0/2975487755 >> 172.21.15.161:0/2975487755 conn(0x7fa2c40ac620 msgr2=0x7fa2c40aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:19.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.573+0000 7fa2c9264640 1 -- 172.21.15.161:0/2975487755 shutdown_connections 2024-03-28T23:30:19.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.573+0000 7fa2c9264640 1 -- 172.21.15.161:0/2975487755 wait complete. 2024-03-28T23:30:19.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.574+0000 7fa2c9264640 1 Processor -- start 2024-03-28T23:30:19.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.574+0000 7fa2c9264640 1 -- start start 2024-03-28T23:30:19.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.575+0000 7fa2c9264640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2c40c5950 0x7fa2c41578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:19.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.575+0000 7fa2c9264640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2c40c8ed0 0x7fa2c41444b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:19.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.575+0000 7fa2c9264640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2c4152890 0x7fa2c41449f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:19.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.575+0000 7fa2c9264640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa2c4152c70 con 0x7fa2c4152890 2024-03-28T23:30:19.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.575+0000 7fa2c9264640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa2c41592d0 con 0x7fa2c40c5950 2024-03-28T23:30:19.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.575+0000 7fa2c9264640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa2c4152df0 con 0x7fa2c40c8ed0 2024-03-28T23:30:19.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.575+0000 7fa2c37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2c4152890 0x7fa2c41449f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:19.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.575+0000 7fa2c37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2c4152890 0x7fa2c41449f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54718/0 (socket says 172.21.15.161:54718) 2024-03-28T23:30:19.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.575+0000 7fa2c37fe640 1 -- 172.21.15.161:0/1153918392 learned_addr learned my addr 172.21.15.161:0/1153918392 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:19.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.575+0000 7fa2c2ffd640 1 --2- 172.21.15.161:0/1153918392 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2c40c5950 0x7fa2c41578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:19.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.575+0000 7fa2bbfff640 1 --2- 172.21.15.161:0/1153918392 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2c40c8ed0 0x7fa2c41444b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:19.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.575+0000 7fa2c37fe640 1 -- 172.21.15.161:0/1153918392 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2c40c8ed0 msgr2=0x7fa2c41444b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:19.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.575+0000 7fa2c37fe640 1 --2- 172.21.15.161:0/1153918392 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2c40c8ed0 0x7fa2c41444b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.575+0000 7fa2c37fe640 1 -- 172.21.15.161:0/1153918392 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2c40c5950 msgr2=0x7fa2c41578c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:19.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.575+0000 7fa2c37fe640 1 --2- 172.21.15.161:0/1153918392 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2c40c5950 0x7fa2c41578c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.575+0000 7fa2c37fe640 1 -- 172.21.15.161:0/1153918392 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2c40aa5d0 con 0x7fa2c4152890 2024-03-28T23:30:19.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.575+0000 7fa2bbfff640 1 --2- 172.21.15.161:0/1153918392 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2c40c8ed0 0x7fa2c41444b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:19.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.576+0000 7fa2c37fe640 1 --2- 172.21.15.161:0/1153918392 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2c4152890 0x7fa2c41449f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc00b9a0 tx=0x7fa2bc00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:19.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.576+0000 7fa2bb7fe640 1 -- 172.21.15.161:0/1153918392 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc019070 con 0x7fa2c4152890 2024-03-28T23:30:19.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.576+0000 7fa2bb7fe640 1 -- 172.21.15.161:0/1153918392 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2bc00cba0 con 0x7fa2c4152890 2024-03-28T23:30:19.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.576+0000 7fa2c9264640 1 -- 172.21.15.161:0/1153918392 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2c40aa8b0 con 0x7fa2c4152890 2024-03-28T23:30:19.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.576+0000 7fa2bb7fe640 1 -- 172.21.15.161:0/1153918392 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc014440 con 0x7fa2c4152890 2024-03-28T23:30:19.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.576+0000 7fa2c9264640 1 -- 172.21.15.161:0/1153918392 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2c40aae70 con 0x7fa2c4152890 2024-03-28T23:30:19.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.577+0000 7fa2c9264640 1 -- 172.21.15.161:0/1153918392 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2c404f4b0 con 0x7fa2c4152890 2024-03-28T23:30:19.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.577+0000 7fa2bb7fe640 1 -- 172.21.15.161:0/1153918392 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa2bc01c070 con 0x7fa2c4152890 2024-03-28T23:30:19.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.578+0000 7fa2bb7fe640 1 --2- 172.21.15.161:0/1153918392 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa2a406b500 0x7fa2a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:19.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.578+0000 7fa2bb7fe640 1 -- 172.21.15.161:0/1153918392 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa2bc08a110 con 0x7fa2c4152890 2024-03-28T23:30:19.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.578+0000 7fa2c2ffd640 1 --2- 172.21.15.161:0/1153918392 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa2a406b500 0x7fa2a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:19.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.578+0000 7fa2c2ffd640 1 --2- 172.21.15.161:0/1153918392 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa2a406b500 0x7fa2a406d9c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4006fd0 tx=0x7fa2b4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:19.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.582+0000 7fa2bb7fe640 1 -- 172.21.15.161:0/1153918392 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa2bc058600 con 0x7fa2c4152890 2024-03-28T23:30:19.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.798+0000 7fa2c9264640 1 -- 172.21.15.161:0/1153918392 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2c40c8140 con 0x7fa2c4152890 2024-03-28T23:30:19.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.799+0000 7fa2bb7fe640 1 -- 172.21.15.161:0/1153918392 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5189 (secure 0 0 0) 0x7fa2bc05c6a0 con 0x7fa2c4152890 2024-03-28T23:30:19.800 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:19.800 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/1577284938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":1577284938},{"type":"v1","addr":"172.21.15.170:6835","nonce":1577284938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4258,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/1107796931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1107796931},{"type":"v1","addr":"172.21.15.170:6834","nonce":1107796931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4107992135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4107992135},{"type":"v1","addr":"172.21.15.161:6836","nonce":4107992135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:29:45.903844+0000","modified":"2024-03-28T23:30:07.984387+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6837/3832540545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3832540545},{"type":"v1","addr":"172.21.15.161:6837","nonce":3832540545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4275,"qdb_cluster":[4275]},"id":1}]} 2024-03-28T23:30:19.801 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 9 2024-03-28T23:30:19.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.801+0000 7fa2c9264640 1 -- 172.21.15.161:0/1153918392 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa2a406b500 msgr2=0x7fa2a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:19.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.801+0000 7fa2c9264640 1 --2- 172.21.15.161:0/1153918392 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa2a406b500 0x7fa2a406d9c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4006fd0 tx=0x7fa2b4008040 comp rx=0 tx=0).stop 2024-03-28T23:30:19.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.801+0000 7fa2c9264640 1 -- 172.21.15.161:0/1153918392 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2c4152890 msgr2=0x7fa2c41449f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:19.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.801+0000 7fa2c9264640 1 --2- 172.21.15.161:0/1153918392 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2c4152890 0x7fa2c41449f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc00b9a0 tx=0x7fa2bc00be60 comp rx=0 tx=0).stop 2024-03-28T23:30:19.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.801+0000 7fa2c9264640 1 -- 172.21.15.161:0/1153918392 shutdown_connections 2024-03-28T23:30:19.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.801+0000 7fa2c9264640 1 --2- 172.21.15.161:0/1153918392 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa2a406b500 0x7fa2a406d9c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.801+0000 7fa2c9264640 1 --2- 172.21.15.161:0/1153918392 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2c4152890 0x7fa2c41449f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.801+0000 7fa2c9264640 1 --2- 172.21.15.161:0/1153918392 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2c40c8ed0 0x7fa2c41444b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.801+0000 7fa2c9264640 1 --2- 172.21.15.161:0/1153918392 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2c40c5950 0x7fa2c41578c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:19.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.801+0000 7fa2c9264640 1 -- 172.21.15.161:0/1153918392 >> 172.21.15.161:0/1153918392 conn(0x7fa2c40ac620 msgr2=0x7fa2c414b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:19.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.801+0000 7fa2c9264640 1 -- 172.21.15.161:0/1153918392 shutdown_connections 2024-03-28T23:30:19.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:19.801+0000 7fa2c9264640 1 -- 172.21.15.161:0/1153918392 wait complete. 2024-03-28T23:30:20.219 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:29:45.903844+0000', 'modified': '2024-03-28T23:30:07.984387+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4275}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4275': {'gid': 4275, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.161:6837/3832540545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 3832540545}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 3832540545}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4275, 'qdb_cluster': [4275]} 2024-03-28T23:30:20.219 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:30:20.219 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-28T23:30:20.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.320+0000 7f66a216c640 1 Processor -- start 2024-03-28T23:30:20.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.320+0000 7f66a216c640 1 -- start start 2024-03-28T23:30:20.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.321+0000 7f66a216c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f669c152890 0x7f669c0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.321+0000 7f66a216c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f669c0c5b20 0x7f669c1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.321+0000 7f66a216c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f669c158a30 con 0x7f669c0c8ed0 2024-03-28T23:30:20.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.321+0000 7f66a216c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f669c0c8480 con 0x7f669c0c5b20 2024-03-28T23:30:20.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.321+0000 7f66a216c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f669c159640 con 0x7f669c152890 2024-03-28T23:30:20.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.321+0000 7f669b7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f669c152890 0x7f669c0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.321+0000 7f669bfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f669c0c5b20 0x7f669c1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.321+0000 7f669b7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f669c152890 0x7f669c0c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:37310/0 (socket says 172.21.15.161:37310) 2024-03-28T23:30:20.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.321+0000 7f669b7fe640 1 -- 172.21.15.161:0/608833363 learned_addr learned my addr 172.21.15.161:0/608833363 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:20.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.321+0000 7f6692ffd640 1 --1- 172.21.15.161:0/608833363 >> v1:172.21.15.170:6789/0 conn(0x7f669c0c8ed0 0x7f669c0c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:46282/0 (socket says 172.21.15.161:46282) 2024-03-28T23:30:20.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.322+0000 7f669affd640 1 -- 172.21.15.161:0/608833363 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2300609530 0 0) 0x7f669c158a30 con 0x7f669c0c8ed0 2024-03-28T23:30:20.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.322+0000 7f669affd640 1 -- 172.21.15.161:0/608833363 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6688003520 con 0x7f669c0c8ed0 2024-03-28T23:30:20.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.322+0000 7f669b7fe640 1 -- 172.21.15.161:0/608833363 >> v2:172.21.15.170:3300/0 conn(0x7f669c0c5b20 msgr2=0x7f669c1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:20.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.322+0000 7f669b7fe640 1 --2- 172.21.15.161:0/608833363 >> v2:172.21.15.170:3300/0 conn(0x7f669c0c5b20 0x7f669c1553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.322+0000 7f669b7fe640 1 -- 172.21.15.161:0/608833363 >> v1:172.21.15.170:6789/0 conn(0x7f669c0c8ed0 legacy=0x7f669c0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:20.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.322+0000 7f669b7fe640 1 -- 172.21.15.161:0/608833363 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f669c158d10 con 0x7f669c152890 2024-03-28T23:30:20.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.322+0000 7f669bfff640 1 --2- 172.21.15.161:0/608833363 >> v2:172.21.15.170:3300/0 conn(0x7f669c0c5b20 0x7f669c1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:20.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.322+0000 7f669b7fe640 1 --2- 172.21.15.161:0/608833363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f669c152890 0x7f669c0c8990 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f669c1597c0 tx=0x7f668c02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cbc1469ecd2808b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:20.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.323+0000 7f669affd640 1 -- 172.21.15.161:0/608833363 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c0317c0 con 0x7f669c152890 2024-03-28T23:30:20.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.323+0000 7f669affd640 1 -- 172.21.15.161:0/608833363 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f668c037070 con 0x7f669c152890 2024-03-28T23:30:20.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.323+0000 7f669affd640 1 -- 172.21.15.161:0/608833363 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c031b60 con 0x7f669c152890 2024-03-28T23:30:20.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.323+0000 7f66a216c640 1 -- 172.21.15.161:0/608833363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f669c152890 msgr2=0x7f669c0c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:20.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.323+0000 7f66a216c640 1 --2- 172.21.15.161:0/608833363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f669c152890 0x7f669c0c8990 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f669c1597c0 tx=0x7f668c02b860 comp rx=0 tx=0).stop 2024-03-28T23:30:20.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.323+0000 7f66a216c640 1 -- 172.21.15.161:0/608833363 shutdown_connections 2024-03-28T23:30:20.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.323+0000 7f66a216c640 1 --2- 172.21.15.161:0/608833363 >> v2:172.21.15.170:3300/0 conn(0x7f669c0c5b20 0x7f669c1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.323+0000 7f66a216c640 1 --2- 172.21.15.161:0/608833363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f669c152890 0x7f669c0c8990 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.323+0000 7f66a216c640 1 -- 172.21.15.161:0/608833363 >> 172.21.15.161:0/608833363 conn(0x7f669c0ac620 msgr2=0x7f669c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:20.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.323+0000 7f66a216c640 1 -- 172.21.15.161:0/608833363 shutdown_connections 2024-03-28T23:30:20.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.323+0000 7f66a216c640 1 -- 172.21.15.161:0/608833363 wait complete. 2024-03-28T23:30:20.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.324+0000 7f66a216c640 1 Processor -- start 2024-03-28T23:30:20.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.325+0000 7f66a216c640 1 -- start start 2024-03-28T23:30:20.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.325+0000 7f66a216c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f669c0c5b20 0x7f669c1eb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.325+0000 7f66a216c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f669c0c8ed0 0x7f669c0aae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.325+0000 7f66a216c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f669c152890 0x7f669c147da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.325+0000 7f66a216c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f669c152c70 con 0x7f669c0c5b20 2024-03-28T23:30:20.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.325+0000 7f66a216c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f669c1592d0 con 0x7f669c152890 2024-03-28T23:30:20.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.325+0000 7f66a216c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f669c152df0 con 0x7f669c0c8ed0 2024-03-28T23:30:20.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.325+0000 7f669b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f669c0c5b20 0x7f669c1eb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.325+0000 7f669b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f669c0c5b20 0x7f669c1eb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54726/0 (socket says 172.21.15.161:54726) 2024-03-28T23:30:20.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.325+0000 7f669b7fe640 1 -- 172.21.15.161:0/265135571 learned_addr learned my addr 172.21.15.161:0/265135571 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:20.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.326+0000 7f6692ffd640 1 --2- 172.21.15.161:0/265135571 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f669c0c8ed0 0x7f669c0aae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.326+0000 7f669bfff640 1 --2- 172.21.15.161:0/265135571 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f669c152890 0x7f669c147da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.326+0000 7f669b7fe640 1 -- 172.21.15.161:0/265135571 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f669c0c8ed0 msgr2=0x7f669c0aae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:20.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.326+0000 7f669b7fe640 1 --2- 172.21.15.161:0/265135571 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f669c0c8ed0 0x7f669c0aae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.326+0000 7f669b7fe640 1 -- 172.21.15.161:0/265135571 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f669c152890 msgr2=0x7f669c147da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:20.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.326+0000 7f669b7fe640 1 --2- 172.21.15.161:0/265135571 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f669c152890 0x7f669c147da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.326+0000 7f669b7fe640 1 -- 172.21.15.161:0/265135571 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f669c1ebac0 con 0x7f669c0c5b20 2024-03-28T23:30:20.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.326+0000 7f669bfff640 1 --2- 172.21.15.161:0/265135571 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f669c152890 0x7f669c147da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:20.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.326+0000 7f6692ffd640 1 --2- 172.21.15.161:0/265135571 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f669c0c8ed0 0x7f669c0aae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:20.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.326+0000 7f669b7fe640 1 --2- 172.21.15.161:0/265135571 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f669c0c5b20 0x7f669c1eb580 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f668c02b8a0 tx=0x7f668c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:20.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.326+0000 7f6698ff9640 1 -- 172.21.15.161:0/265135571 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c032650 con 0x7f669c0c5b20 2024-03-28T23:30:20.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.326+0000 7f6698ff9640 1 -- 172.21.15.161:0/265135571 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f668c032810 con 0x7f669c0c5b20 2024-03-28T23:30:20.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.326+0000 7f66a216c640 1 -- 172.21.15.161:0/265135571 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f669c0c2fb0 con 0x7f669c0c5b20 2024-03-28T23:30:20.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.326+0000 7f6698ff9640 1 -- 172.21.15.161:0/265135571 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c032c00 con 0x7f669c0c5b20 2024-03-28T23:30:20.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.326+0000 7f66a216c640 1 -- 172.21.15.161:0/265135571 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f669c0c3570 con 0x7f669c0c5b20 2024-03-28T23:30:20.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.328+0000 7f6698ff9640 1 -- 172.21.15.161:0/265135571 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f668c04c060 con 0x7f669c0c5b20 2024-03-28T23:30:20.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.328+0000 7f6698ff9640 1 --2- 172.21.15.161:0/265135571 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f667c06b4d0 0x7f667c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.328+0000 7f6698ff9640 1 -- 172.21.15.161:0/265135571 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f668c0abf60 con 0x7f669c0c5b20 2024-03-28T23:30:20.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.328+0000 7f6692ffd640 1 --2- 172.21.15.161:0/265135571 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f667c06b4d0 0x7f667c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.329+0000 7f6692ffd640 1 --2- 172.21.15.161:0/265135571 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f667c06b4d0 0x7f667c06d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f669c0c50b0 tx=0x7f6684009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:20.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.329+0000 7f66a4e2d740 1 Processor -- start 2024-03-28T23:30:20.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.329+0000 7f66a4e2d740 1 -- start start 2024-03-28T23:30:20.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.330+0000 7f66a4e2d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463213db0 0x556463211890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.330+0000 7f66a4e2d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556463214190 0x556463211dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.330+0000 7f66a4e2d740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556463212310 0x55646320e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.330+0000 7f66a4e2d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x556463177af0 con 0x556463213db0 2024-03-28T23:30:20.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.330+0000 7f66a4e2d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5564631209d0 con 0x556463214190 2024-03-28T23:30:20.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.330+0000 7f66a4e2d740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x556463173b90 con 0x556463212310 2024-03-28T23:30:20.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.330+0000 7f669b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463213db0 0x556463211890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.330+0000 7f669b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463213db0 0x556463211890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54738/0 (socket says 172.21.15.161:54738) 2024-03-28T23:30:20.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.330+0000 7f669b7fe640 1 -- 172.21.15.161:0/1876531923 learned_addr learned my addr 172.21.15.161:0/1876531923 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:20.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.330+0000 7f6692ffd640 1 --2- 172.21.15.161:0/1876531923 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556463214190 0x556463211dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.330+0000 7f669bfff640 1 --2- 172.21.15.161:0/1876531923 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556463212310 0x55646320e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.330+0000 7f669b7fe640 1 -- 172.21.15.161:0/1876531923 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556463212310 msgr2=0x55646320e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:20.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.330+0000 7f669b7fe640 1 --2- 172.21.15.161:0/1876531923 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556463212310 0x55646320e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.330+0000 7f669b7fe640 1 -- 172.21.15.161:0/1876531923 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556463214190 msgr2=0x556463211dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:20.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.330+0000 7f669b7fe640 1 --2- 172.21.15.161:0/1876531923 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556463214190 0x556463211dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.330+0000 7f669b7fe640 1 -- 172.21.15.161:0/1876531923 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556462e8c2d0 con 0x556463213db0 2024-03-28T23:30:20.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.330+0000 7f6692ffd640 1 --2- 172.21.15.161:0/1876531923 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556463214190 0x556463211dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:20.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.331+0000 7f669b7fe640 1 --2- 172.21.15.161:0/1876531923 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463213db0 0x556463211890 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f668c076890 tx=0x7f668c07a330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28aefcef4e04171d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:20.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.331+0000 7f66917fa640 1 -- 172.21.15.161:0/1876531923 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c0a8070 con 0x556463213db0 2024-03-28T23:30:20.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.331+0000 7f66917fa640 1 -- 172.21.15.161:0/1876531923 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f668c037040 con 0x556463213db0 2024-03-28T23:30:20.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.331+0000 7f66917fa640 1 -- 172.21.15.161:0/1876531923 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c07e580 con 0x556463213db0 2024-03-28T23:30:20.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.331+0000 7f66a4e2d740 1 -- 172.21.15.161:0/1876531923 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463213db0 msgr2=0x556463211890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:20.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.331+0000 7f66a4e2d740 1 --2- 172.21.15.161:0/1876531923 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463213db0 0x556463211890 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f668c076890 tx=0x7f668c07a330 comp rx=0 tx=0).stop 2024-03-28T23:30:20.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.331+0000 7f66a4e2d740 1 -- 172.21.15.161:0/1876531923 shutdown_connections 2024-03-28T23:30:20.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.331+0000 7f66a4e2d740 1 --2- 172.21.15.161:0/1876531923 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556463212310 0x55646320e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.331+0000 7f66a4e2d740 1 --2- 172.21.15.161:0/1876531923 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556463214190 0x556463211dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.331+0000 7f66a4e2d740 1 --2- 172.21.15.161:0/1876531923 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463213db0 0x556463211890 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.331+0000 7f66a4e2d740 1 -- 172.21.15.161:0/1876531923 >> 172.21.15.161:0/1876531923 conn(0x556463210630 msgr2=0x556463235220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:20.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.331+0000 7f66a4e2d740 1 -- 172.21.15.161:0/1876531923 shutdown_connections 2024-03-28T23:30:20.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.331+0000 7f66a4e2d740 1 -- 172.21.15.161:0/1876531923 wait complete. 2024-03-28T23:30:20.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.332+0000 7f66a4e2d740 1 -- start start 2024-03-28T23:30:20.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.332+0000 7f66a4e2d740 1 Processor -- start 2024-03-28T23:30:20.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.333+0000 7f66a4e2d740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556463213db0 0x556463238ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.333+0000 7f66a4e2d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556463214190 0x556463234bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.333+0000 7f66a4e2d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463239a50 0x5564632364f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.333+0000 7f66a4e2d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5564631209d0 con 0x556463239a50 2024-03-28T23:30:20.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.333+0000 7f66a4e2d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x556463133f70 con 0x556463214190 2024-03-28T23:30:20.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.333+0000 7f66a4e2d740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x556463177af0 con 0x556463213db0 2024-03-28T23:30:20.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.333+0000 7f66a4e2d740 10 client.0 _ll_register_callbacks cb 0x556462e8bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:30:20.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.333+0000 7f669bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463239a50 0x5564632364f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.333+0000 7f669bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463239a50 0x5564632364f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54752/0 (socket says 172.21.15.161:54752) 2024-03-28T23:30:20.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.333+0000 7f669bfff640 1 -- 172.21.15.161:0/3814939504 learned_addr learned my addr 172.21.15.161:0/3814939504 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:20.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.333+0000 7f6692ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556463214190 0x556463234bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.333+0000 7f669b7fe640 1 --2- 172.21.15.161:0/3814939504 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556463213db0 0x556463238ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.333+0000 7f669bfff640 1 -- 172.21.15.161:0/3814939504 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556463213db0 msgr2=0x556463238ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:20.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.333+0000 7f669bfff640 1 --2- 172.21.15.161:0/3814939504 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556463213db0 0x556463238ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.333+0000 7f669bfff640 1 -- 172.21.15.161:0/3814939504 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556463214190 msgr2=0x556463234bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:20.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.333+0000 7f669bfff640 1 --2- 172.21.15.161:0/3814939504 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556463214190 0x556463234bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.333+0000 7f669bfff640 1 -- 172.21.15.161:0/3814939504 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556462f566b0 con 0x556463239a50 2024-03-28T23:30:20.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.333+0000 7f669bfff640 1 --2- 172.21.15.161:0/3814939504 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463239a50 0x5564632364f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6694007dd0 tx=0x7f66940115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=141389a64d1653e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:20.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.333+0000 7f667a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:30:20.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.333+0000 7f667a7fc640 1 -- 172.21.15.161:0/3814939504 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f669401b020 con 0x556463239a50 2024-03-28T23:30:20.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.333+0000 7f667a7fc640 1 -- 172.21.15.161:0/3814939504 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6694011c30 con 0x556463239a50 2024-03-28T23:30:20.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.333+0000 7f66a4e2d740 1 -- 172.21.15.161:0/3814939504 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f668c037040 con 0x556463239a50 2024-03-28T23:30:20.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.334+0000 7f667a7fc640 1 -- 172.21.15.161:0/3814939504 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7f6694025050 con 0x556463239a50 2024-03-28T23:30:20.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.334+0000 7f66a4e2d740 10 client.4833 fetch_fsmap learned FSMap version 9 2024-03-28T23:30:20.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.334+0000 7f66a4e2d740 1 -- 172.21.15.161:0/3814939504 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) -- 0x556463263a00 con 0x556463239a50 2024-03-28T23:30:20.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.334+0000 7f667a7fc640 1 -- 172.21.15.161:0/3814939504 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f668c037040 con 0x556463239a50 2024-03-28T23:30:20.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.334+0000 7f667a7fc640 1 -- 172.21.15.161:0/3814939504 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 9) ==== 3502+0+0 (secure 0 0 0) 0x7f6694028020 con 0x556463239a50 2024-03-28T23:30:20.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.334+0000 7f66a4e2d740 10 client.4833 fetch_fsmap finished waiting for FSMap version 9 2024-03-28T23:30:20.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.334+0000 7f66a4e2d740 10 client.4833 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T23:30:20.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.334+0000 7f66a4e2d740 20 client.4833 populate_metadata read hostname 'smithi161' 2024-03-28T23:30:20.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.334+0000 7f66a4e2d740 1 --2- 172.21.15.161:0/3814939504 >> [v2:172.21.15.161:6835/3832540545,v1:172.21.15.161:6837/3832540545] conn(0x556463236a30 0x556463226e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.334+0000 7f66a4e2d740 4 client.4833 mds_command: new command op to 4275 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:30:20.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.334+0000 7f66a4e2d740 1 -- 172.21.15.161:0/3814939504 --> [v2:172.21.15.161:6835/3832540545,v1:172.21.15.161:6837/3832540545] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556463185030 con 0x556463236a30 2024-03-28T23:30:20.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.334+0000 7f669b7fe640 1 --2- 172.21.15.161:0/3814939504 >> [v2:172.21.15.161:6835/3832540545,v1:172.21.15.161:6837/3832540545] conn(0x556463236a30 0x556463226e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.334+0000 7f669b7fe640 1 --2- 172.21.15.161:0/3814939504 >> [v2:172.21.15.161:6835/3832540545,v1:172.21.15.161:6837/3832540545] conn(0x556463236a30 0x556463226e20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b85ca4e3b32b2035 server_cookie=f99ab790d2ac1515 in_seq=0 out_seq=0 2024-03-28T23:30:20.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.334+0000 7f667a7fc640 10 client.4833 ms_handle_connect on v2:172.21.15.161:6835/3832540545 2024-03-28T23:30:20.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.335+0000 7f667a7fc640 1 -- 172.21.15.161:0/3814939504 <== mds.0 v2:172.21.15.161:6835/3832540545 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556463185030 con 0x556463236a30 2024-03-28T23:30:20.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.335+0000 7f667a7fc640 10 client.4833 handle_command_reply: tid=0 2024-03-28T23:30:20.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.335+0000 7f66a4e2d740 1 client.4833 shutdown 2024-03-28T23:30:20.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.335+0000 7f66a4e2d740 1 -- 172.21.15.161:0/3814939504 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463239a50 msgr2=0x5564632364f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:20.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.335+0000 7f66a4e2d740 1 --2- 172.21.15.161:0/3814939504 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463239a50 0x5564632364f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6694007dd0 tx=0x7f66940115a0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.335+0000 7f66a4e2d740 1 -- 172.21.15.161:0/3814939504 shutdown_connections 2024-03-28T23:30:20.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.335+0000 7f66a4e2d740 1 --2- 172.21.15.161:0/3814939504 >> [v2:172.21.15.161:6835/3832540545,v1:172.21.15.161:6837/3832540545] conn(0x556463236a30 0x556463226e20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.335+0000 7f66a4e2d740 1 --2- 172.21.15.161:0/3814939504 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463239a50 0x5564632364f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.335+0000 7f66a4e2d740 1 --2- 172.21.15.161:0/3814939504 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556463214190 0x556463234bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.335+0000 7f667a7fc640 0 client.4833 ms_handle_reset on v2:172.21.15.161:6835/3832540545 2024-03-28T23:30:20.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.335+0000 7f66a4e2d740 1 --2- 172.21.15.161:0/3814939504 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556463213db0 0x556463238ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.335+0000 7f66a4e2d740 1 -- 172.21.15.161:0/3814939504 >> 172.21.15.161:0/3814939504 conn(0x556463212310 msgr2=0x5564632509b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:20.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.335+0000 7f66a4e2d740 1 -- 172.21.15.161:0/3814939504 shutdown_connections 2024-03-28T23:30:20.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.335+0000 7f66a4e2d740 1 -- 172.21.15.161:0/3814939504 wait complete. 2024-03-28T23:30:20.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.335+0000 7f66a4e2d740 20 client.4833 trim_cache size 0 max 16384 2024-03-28T23:30:20.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.353+0000 7f66a4e2d740 1 Processor -- start 2024-03-28T23:30:20.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.353+0000 7f66a4e2d740 1 -- start start 2024-03-28T23:30:20.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.354+0000 7f66a4e2d740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556463213db0 0x55646320d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.354+0000 7f66a4e2d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556463214190 0x556463211630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.354+0000 7f66a4e2d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463239a50 0x556463211b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.354+0000 7f66a4e2d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f669c1f7d80 con 0x556463239a50 2024-03-28T23:30:20.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.354+0000 7f66a4e2d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f669c1f70e0 con 0x556463214190 2024-03-28T23:30:20.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.354+0000 7f66a4e2d740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f669c1f7730 con 0x556463213db0 2024-03-28T23:30:20.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.354+0000 7f669bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463239a50 0x556463211b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.354+0000 7f669bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463239a50 0x556463211b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54764/0 (socket says 172.21.15.161:54764) 2024-03-28T23:30:20.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.354+0000 7f669bfff640 1 -- 172.21.15.161:0/3308946581 learned_addr learned my addr 172.21.15.161:0/3308946581 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:20.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.354+0000 7f669b7fe640 1 --2- 172.21.15.161:0/3308946581 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556463213db0 0x55646320d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.354+0000 7f6692ffd640 1 --2- 172.21.15.161:0/3308946581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556463214190 0x556463211630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.354+0000 7f669bfff640 1 -- 172.21.15.161:0/3308946581 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556463213db0 msgr2=0x55646320d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:20.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.354+0000 7f669bfff640 1 --2- 172.21.15.161:0/3308946581 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556463213db0 0x55646320d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.354+0000 7f669bfff640 1 -- 172.21.15.161:0/3308946581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556463214190 msgr2=0x556463211630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:20.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.354+0000 7f669bfff640 1 --2- 172.21.15.161:0/3308946581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556463214190 0x556463211630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.354+0000 7f669bfff640 1 -- 172.21.15.161:0/3308946581 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556463236860 con 0x556463239a50 2024-03-28T23:30:20.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.354+0000 7f669bfff640 1 --2- 172.21.15.161:0/3308946581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463239a50 0x556463211b70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6694011df0 tx=0x7f6694017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4cb19c8f0eab9dd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:20.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.354+0000 7f6690ff9640 1 -- 172.21.15.161:0/3308946581 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f669401b050 con 0x556463239a50 2024-03-28T23:30:20.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.354+0000 7f6690ff9640 1 -- 172.21.15.161:0/3308946581 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6694020710 con 0x556463239a50 2024-03-28T23:30:20.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.354+0000 7f6690ff9640 1 -- 172.21.15.161:0/3308946581 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6694020b00 con 0x556463239a50 2024-03-28T23:30:20.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.355+0000 7f66a4e2d740 1 -- 172.21.15.161:0/3308946581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463239a50 msgr2=0x556463211b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:20.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.355+0000 7f66a4e2d740 1 --2- 172.21.15.161:0/3308946581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463239a50 0x556463211b70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6694011df0 tx=0x7f6694017000 comp rx=0 tx=0).stop 2024-03-28T23:30:20.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.355+0000 7f66a4e2d740 1 -- 172.21.15.161:0/3308946581 shutdown_connections 2024-03-28T23:30:20.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.355+0000 7f66a4e2d740 1 --2- 172.21.15.161:0/3308946581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463239a50 0x556463211b70 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.355+0000 7f66a4e2d740 1 --2- 172.21.15.161:0/3308946581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556463214190 0x556463211630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.355+0000 7f66a4e2d740 1 --2- 172.21.15.161:0/3308946581 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556463213db0 0x55646320d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.355+0000 7f66a4e2d740 1 -- 172.21.15.161:0/3308946581 >> 172.21.15.161:0/3308946581 conn(0x556463212310 msgr2=0x556463273fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:20.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.355+0000 7f66a4e2d740 1 -- 172.21.15.161:0/3308946581 shutdown_connections 2024-03-28T23:30:20.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.355+0000 7f66a4e2d740 1 -- 172.21.15.161:0/3308946581 wait complete. 2024-03-28T23:30:20.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.356+0000 7f66a4e2d740 1 -- start start 2024-03-28T23:30:20.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.356+0000 7f66a4e2d740 1 Processor -- start 2024-03-28T23:30:20.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.356+0000 7f66a4e2d740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556463213db0 0x556463211800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.356+0000 7f66a4e2d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463214190 0x5564632ce320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.356+0000 7f66a4e2d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556463239a50 0x55646323a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.356+0000 7f66a4e2d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x556463238470 con 0x556463214190 2024-03-28T23:30:20.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.356+0000 7f66a4e2d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x556463263da0 con 0x556463239a50 2024-03-28T23:30:20.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.356+0000 7f66a4e2d740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x556463236e60 con 0x556463213db0 2024-03-28T23:30:20.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.356+0000 7f66a4e2d740 10 client.0 _ll_register_callbacks cb 0x7f6694009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:30:20.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.356+0000 7f6692ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463214190 0x5564632ce320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.356+0000 7f6692ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463214190 0x5564632ce320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54772/0 (socket says 172.21.15.161:54772) 2024-03-28T23:30:20.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.356+0000 7f6692ffd640 1 -- 172.21.15.161:0/2533036068 learned_addr learned my addr 172.21.15.161:0/2533036068 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:20.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.356+0000 7f669b7fe640 1 --2- 172.21.15.161:0/2533036068 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556463213db0 0x556463211800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.357+0000 7f669bfff640 1 --2- 172.21.15.161:0/2533036068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556463239a50 0x55646323a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.357+0000 7f6692ffd640 1 -- 172.21.15.161:0/2533036068 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556463213db0 msgr2=0x556463211800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:20.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.357+0000 7f6692ffd640 1 --2- 172.21.15.161:0/2533036068 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556463213db0 0x556463211800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.357+0000 7f6692ffd640 1 -- 172.21.15.161:0/2533036068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556463239a50 msgr2=0x55646323a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:20.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.357+0000 7f6692ffd640 1 --2- 172.21.15.161:0/2533036068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556463239a50 0x55646323a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.357+0000 7f6692ffd640 1 -- 172.21.15.161:0/2533036068 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55646323c500 con 0x556463214190 2024-03-28T23:30:20.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.357+0000 7f6692ffd640 1 --2- 172.21.15.161:0/2533036068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463214190 0x5564632ce320 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6684000fb0 tx=0x7f6684000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8202c162fdc3e085 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:20.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.357+0000 7f667a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:30:20.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.357+0000 7f667a7fc640 1 -- 172.21.15.161:0/2533036068 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6684029060 con 0x556463214190 2024-03-28T23:30:20.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.357+0000 7f667a7fc640 1 -- 172.21.15.161:0/2533036068 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f668400dd80 con 0x556463214190 2024-03-28T23:30:20.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.357+0000 7f66a4e2d740 1 -- 172.21.15.161:0/2533036068 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6694020710 con 0x556463214190 2024-03-28T23:30:20.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.357+0000 7f667a7fc640 1 -- 172.21.15.161:0/2533036068 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7f6684012410 con 0x556463214190 2024-03-28T23:30:20.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.357+0000 7f66a4e2d740 10 client.4839 fetch_fsmap learned FSMap version 9 2024-03-28T23:30:20.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.357+0000 7f66a4e2d740 1 -- 172.21.15.161:0/2533036068 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) -- 0x556463237940 con 0x556463214190 2024-03-28T23:30:20.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.357+0000 7f667a7fc640 1 -- 172.21.15.161:0/2533036068 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6694020710 con 0x556463214190 2024-03-28T23:30:20.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.357+0000 7f667a7fc640 1 -- 172.21.15.161:0/2533036068 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 9) ==== 3502+0+0 (secure 0 0 0) 0x7f66840125b0 con 0x556463214190 2024-03-28T23:30:20.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.357+0000 7f66a4e2d740 10 client.4839 fetch_fsmap finished waiting for FSMap version 9 2024-03-28T23:30:20.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.357+0000 7f66a4e2d740 10 client.4839 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T23:30:20.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.357+0000 7f66a4e2d740 20 client.4839 populate_metadata read hostname 'smithi161' 2024-03-28T23:30:20.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.357+0000 7f66a4e2d740 1 --2- 172.21.15.161:0/2533036068 >> [v2:172.21.15.161:6835/3832540545,v1:172.21.15.161:6837/3832540545] conn(0x5564632c7c10 0x5564632c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.357+0000 7f66a4e2d740 4 client.4839 mds_command: new command op to 4275 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:30:20.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.357+0000 7f66a4e2d740 1 -- 172.21.15.161:0/2533036068 --> [v2:172.21.15.161:6835/3832540545,v1:172.21.15.161:6837/3832540545] -- command(tid 0: {"prefix": "status"}) -- 0x55646317f380 con 0x5564632c7c10 2024-03-28T23:30:20.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.358+0000 7f669b7fe640 1 --2- 172.21.15.161:0/2533036068 >> [v2:172.21.15.161:6835/3832540545,v1:172.21.15.161:6837/3832540545] conn(0x5564632c7c10 0x5564632c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.358+0000 7f667a7fc640 10 client.4839 ms_handle_connect on v2:172.21.15.161:6835/3832540545 2024-03-28T23:30:20.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.358+0000 7f669b7fe640 1 --2- 172.21.15.161:0/2533036068 >> [v2:172.21.15.161:6835/3832540545,v1:172.21.15.161:6837/3832540545] conn(0x5564632c7c10 0x5564632c7ff0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e67e91b7654e27a server_cookie=30ae7284d707deae in_seq=0 out_seq=0 2024-03-28T23:30:20.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.358+0000 7f667a7fc640 1 -- 172.21.15.161:0/2533036068 <== mds.0 v2:172.21.15.161:6835/3832540545 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x55646317f380 con 0x5564632c7c10 2024-03-28T23:30:20.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.358+0000 7f667a7fc640 10 client.4839 handle_command_reply: tid=0 2024-03-28T23:30:20.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.358+0000 7f66a4e2d740 1 client.4839 shutdown 2024-03-28T23:30:20.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.358+0000 7f66a4e2d740 1 -- 172.21.15.161:0/2533036068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463214190 msgr2=0x5564632ce320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:20.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.358+0000 7f66a4e2d740 1 --2- 172.21.15.161:0/2533036068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463214190 0x5564632ce320 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6684000fb0 tx=0x7f6684000bc0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.358+0000 7f66a4e2d740 1 -- 172.21.15.161:0/2533036068 shutdown_connections 2024-03-28T23:30:20.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.358+0000 7f66a4e2d740 1 --2- 172.21.15.161:0/2533036068 >> [v2:172.21.15.161:6835/3832540545,v1:172.21.15.161:6837/3832540545] conn(0x5564632c7c10 0x5564632c7ff0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.358+0000 7f66a4e2d740 1 --2- 172.21.15.161:0/2533036068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556463214190 0x5564632ce320 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.358+0000 7f66a4e2d740 1 --2- 172.21.15.161:0/2533036068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556463239a50 0x55646323a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.358+0000 7f667a7fc640 0 client.4839 ms_handle_reset on v2:172.21.15.161:6835/3832540545 2024-03-28T23:30:20.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.358+0000 7f66a4e2d740 1 --2- 172.21.15.161:0/2533036068 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556463213db0 0x556463211800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.358+0000 7f66a4e2d740 1 -- 172.21.15.161:0/2533036068 >> 172.21.15.161:0/2533036068 conn(0x556463212310 msgr2=0x55646320d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:20.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.358+0000 7f66a4e2d740 1 -- 172.21.15.161:0/2533036068 shutdown_connections 2024-03-28T23:30:20.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.358+0000 7f66a4e2d740 1 -- 172.21.15.161:0/2533036068 wait complete. 2024-03-28T23:30:20.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.358+0000 7f66a4e2d740 20 client.4839 trim_cache size 0 max 16384 2024-03-28T23:30:20.365 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:30:20.366 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:30:20.366 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:30:20.366 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 4275, 2024-03-28T23:30:20.366 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:30:20.366 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:30:20.366 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:30:20.366 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 33.815234042, 2024-03-28T23:30:20.366 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 9, 2024-03-28T23:30:20.366 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 24, 2024-03-28T23:30:20.366 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 24, 2024-03-28T23:30:20.366 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 40.490584273000003 2024-03-28T23:30:20.366 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:30:20.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.359+0000 7f66a216c640 1 -- 172.21.15.161:0/265135571 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f667c06b4d0 msgr2=0x7f667c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:20.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.359+0000 7f66a216c640 1 --2- 172.21.15.161:0/265135571 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f667c06b4d0 0x7f667c06d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f669c0c50b0 tx=0x7f6684009040 comp rx=0 tx=0).stop 2024-03-28T23:30:20.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.359+0000 7f66a216c640 1 -- 172.21.15.161:0/265135571 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f669c0c5b20 msgr2=0x7f669c1eb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:20.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.359+0000 7f66a216c640 1 --2- 172.21.15.161:0/265135571 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f669c0c5b20 0x7f669c1eb580 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f668c02b8a0 tx=0x7f668c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.359+0000 7f66a216c640 1 -- 172.21.15.161:0/265135571 shutdown_connections 2024-03-28T23:30:20.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.359+0000 7f66a216c640 1 --2- 172.21.15.161:0/265135571 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f667c06b4d0 0x7f667c06d990 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.359+0000 7f66a216c640 1 --2- 172.21.15.161:0/265135571 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f669c152890 0x7f669c147da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.359+0000 7f66a216c640 1 --2- 172.21.15.161:0/265135571 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f669c0c8ed0 0x7f669c0aae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.359+0000 7f66a216c640 1 --2- 172.21.15.161:0/265135571 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f669c0c5b20 0x7f669c1eb580 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.359+0000 7f66a216c640 1 -- 172.21.15.161:0/265135571 >> 172.21.15.161:0/265135571 conn(0x7f669c0ac620 msgr2=0x7f669c150700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:20.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.359+0000 7f66a216c640 1 -- 172.21.15.161:0/265135571 shutdown_connections 2024-03-28T23:30:20.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.359+0000 7f66a216c640 1 -- 172.21.15.161:0/265135571 wait complete. 2024-03-28T23:30:20.374 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-28T23:30:20.384 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-28T23:30:20.384 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-03-28T23:30:20.384 INFO:tasks.ceph_fuse:client.0 config is {'disabled': True} 2024-03-28T23:30:20.384 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-28T23:30:20.384 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T23:30:20.384 INFO:tasks.ceph_fuse:client.1 config is {'disabled': True} 2024-03-28T23:30:20.384 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-28T23:30:20.385 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-28T23:30:20.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:20.385 DEBUG:teuthology.orchestra.run.smithi170:> ip netns list 2024-03-28T23:30:20.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:20.404 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip link delete ceph-brx 2024-03-28T23:30:20.480 INFO:teuthology.orchestra.run.smithi170.stderr:Cannot find device "ceph-brx" 2024-03-28T23:30:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:30:20.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:20.482 DEBUG:teuthology.orchestra.run.smithi161:> ip netns list 2024-03-28T23:30:20.497 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-28T23:30:20.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:20.497 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T23:30:20.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:20.566 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link delete ceph-brx 2024-03-28T23:30:20.665 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-28T23:30:20.665 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-28T23:30:20.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.769+0000 7f068e28f640 1 Processor -- start 2024-03-28T23:30:20.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.769+0000 7f068e28f640 1 -- start start 2024-03-28T23:30:20.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.770+0000 7f068e28f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06880a8fc0 0x7f0688153150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.770+0000 7f068e28f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06881566f0 con 0x7f06880aacb0 2024-03-28T23:30:20.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.770+0000 7f068e28f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06881569d0 con 0x7f0688150590 2024-03-28T23:30:20.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.770+0000 7f068e28f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0688156cb0 con 0x7f06880a8fc0 2024-03-28T23:30:20.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.770+0000 7f068c82c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f06880aacb0 0x7f0688153690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38394/0 (socket says 172.21.15.161:38394) 2024-03-28T23:30:20.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.770+0000 7f068c82c640 1 -- 172.21.15.161:0/1856088894 learned_addr learned my addr 172.21.15.161:0/1856088894 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:20.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.770+0000 7f06877fe640 1 --2- 172.21.15.161:0/1856088894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06880a8fc0 0x7f0688153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.770+0000 7f0686ffd640 1 -- 172.21.15.161:0/1856088894 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3848734384 0 0) 0x7f06881566f0 con 0x7f06880aacb0 2024-03-28T23:30:20.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.770+0000 7f0686ffd640 1 -- 172.21.15.161:0/1856088894 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0670003520 con 0x7f06880aacb0 2024-03-28T23:30:20.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.770+0000 7f0686ffd640 1 -- 172.21.15.161:0/1856088894 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2701253570 0 0) 0x7f06881569d0 con 0x7f0688150590 2024-03-28T23:30:20.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.770+0000 7f0686ffd640 1 -- 172.21.15.161:0/1856088894 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06881566f0 con 0x7f0688150590 2024-03-28T23:30:20.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.770+0000 7f06877fe640 1 -- 172.21.15.161:0/1856088894 >> v1:172.21.15.170:6789/0 conn(0x7f0688150590 legacy=0x7f06880a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:20.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.770+0000 7f06877fe640 1 -- 172.21.15.161:0/1856088894 >> v1:172.21.15.161:6789/0 conn(0x7f06880aacb0 legacy=0x7f0688153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:20.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.770+0000 7f06877fe640 1 -- 172.21.15.161:0/1856088894 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0688156f90 con 0x7f06880a8fc0 2024-03-28T23:30:20.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.771+0000 7f06877fe640 1 --2- 172.21.15.161:0/1856088894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06880a8fc0 0x7f0688153150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f06780097d0 tx=0x7f067802b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b874581adae08a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:20.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.771+0000 7f0686ffd640 1 -- 172.21.15.161:0/1856088894 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0678031770 con 0x7f06880a8fc0 2024-03-28T23:30:20.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.771+0000 7f0686ffd640 1 -- 172.21.15.161:0/1856088894 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0678037070 con 0x7f06880a8fc0 2024-03-28T23:30:20.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.771+0000 7f0686ffd640 1 -- 172.21.15.161:0/1856088894 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0678031b30 con 0x7f06880a8fc0 2024-03-28T23:30:20.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.772+0000 7f068e28f640 1 -- 172.21.15.161:0/1856088894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06880a8fc0 msgr2=0x7f0688153150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:20.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.772+0000 7f068e28f640 1 --2- 172.21.15.161:0/1856088894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06880a8fc0 0x7f0688153150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f06780097d0 tx=0x7f067802b810 comp rx=0 tx=0).stop 2024-03-28T23:30:20.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.772+0000 7f068e28f640 1 -- 172.21.15.161:0/1856088894 shutdown_connections 2024-03-28T23:30:20.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.772+0000 7f068e28f640 1 --2- 172.21.15.161:0/1856088894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06880a8fc0 0x7f0688153150 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.772+0000 7f068e28f640 1 -- 172.21.15.161:0/1856088894 >> 172.21.15.161:0/1856088894 conn(0x7f06880c6670 msgr2=0x7f06880c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:20.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.772+0000 7f068e28f640 1 -- 172.21.15.161:0/1856088894 shutdown_connections 2024-03-28T23:30:20.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.772+0000 7f068e28f640 1 -- 172.21.15.161:0/1856088894 wait complete. 2024-03-28T23:30:20.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.773+0000 7f068e28f640 1 Processor -- start 2024-03-28T23:30:20.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.773+0000 7f068e28f640 1 -- start start 2024-03-28T23:30:20.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.774+0000 7f068e28f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06880a8fc0 0x7f06881ecbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.774+0000 7f068e28f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06880aacb0 0x7f0688152640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.774+0000 7f068e28f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0688150590 0x7f06880c4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.774+0000 7f068e28f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0688150970 con 0x7f0688150590 2024-03-28T23:30:20.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.774+0000 7f068e28f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f06880aa9c0 con 0x7f06880a8fc0 2024-03-28T23:30:20.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.774+0000 7f068e28f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f06880aa650 con 0x7f06880aacb0 2024-03-28T23:30:20.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.774+0000 7f068c82c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0688150590 0x7f06880c4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.774+0000 7f068c82c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0688150590 0x7f06880c4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51630/0 (socket says 172.21.15.161:51630) 2024-03-28T23:30:20.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.774+0000 7f068c82c640 1 -- 172.21.15.161:0/466948105 learned_addr learned my addr 172.21.15.161:0/466948105 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:20.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.774+0000 7f0687fff640 1 --2- 172.21.15.161:0/466948105 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06880a8fc0 0x7f06881ecbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.774+0000 7f06877fe640 1 --2- 172.21.15.161:0/466948105 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06880aacb0 0x7f0688152640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.774+0000 7f068c82c640 1 -- 172.21.15.161:0/466948105 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06880aacb0 msgr2=0x7f0688152640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:20.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.774+0000 7f068c82c640 1 --2- 172.21.15.161:0/466948105 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06880aacb0 0x7f0688152640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.774+0000 7f068c82c640 1 -- 172.21.15.161:0/466948105 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06880a8fc0 msgr2=0x7f06881ecbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:20.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.774+0000 7f068c82c640 1 --2- 172.21.15.161:0/466948105 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06880a8fc0 0x7f06881ecbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:20.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.774+0000 7f068c82c640 1 -- 172.21.15.161:0/466948105 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0688156360 con 0x7f0688150590 2024-03-28T23:30:20.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.775+0000 7f068c82c640 1 --2- 172.21.15.161:0/466948105 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0688150590 0x7f06880c4240 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f068000cb50 tx=0x7f068000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:20.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.775+0000 7f0684ff9640 1 -- 172.21.15.161:0/466948105 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0680017070 con 0x7f0688150590 2024-03-28T23:30:20.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.775+0000 7f0684ff9640 1 -- 172.21.15.161:0/466948105 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f068000ad40 con 0x7f0688150590 2024-03-28T23:30:20.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.775+0000 7f068e28f640 1 -- 172.21.15.161:0/466948105 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06881e8f90 con 0x7f0688150590 2024-03-28T23:30:20.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.775+0000 7f0684ff9640 1 -- 172.21.15.161:0/466948105 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0680012620 con 0x7f0688150590 2024-03-28T23:30:20.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.775+0000 7f068e28f640 1 -- 172.21.15.161:0/466948105 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06881e9550 con 0x7f0688150590 2024-03-28T23:30:20.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.776+0000 7f068e28f640 1 -- 172.21.15.161:0/466948105 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f068804f4b0 con 0x7f0688150590 2024-03-28T23:30:20.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.777+0000 7f0684ff9640 1 -- 172.21.15.161:0/466948105 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f068001a070 con 0x7f0688150590 2024-03-28T23:30:20.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.777+0000 7f0684ff9640 1 --2- 172.21.15.161:0/466948105 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f067406b4d0 0x7f067406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:20.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.777+0000 7f0687fff640 1 --2- 172.21.15.161:0/466948105 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f067406b4d0 0x7f067406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:20.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.777+0000 7f0684ff9640 1 -- 172.21.15.161:0/466948105 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f068008a240 con 0x7f0688150590 2024-03-28T23:30:20.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.778+0000 7f0687fff640 1 --2- 172.21.15.161:0/466948105 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f067406b4d0 0x7f067406d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f067c007fd0 tx=0x7f067c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:20.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:20.782+0000 7f0684ff9640 1 -- 172.21.15.161:0/466948105 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0680058730 con 0x7f0688150590 2024-03-28T23:30:21.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:21.017+0000 7f068e28f640 1 -- 172.21.15.161:0/466948105 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f06880a9d20 con 0x7f0688150590 2024-03-28T23:30:21.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:21.017+0000 7f0684ff9640 1 -- 172.21.15.161:0/466948105 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f068005c7d0 con 0x7f0688150590 2024-03-28T23:30:21.018 INFO:teuthology.orchestra.run.smithi161.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-28T23:30:21.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:21.019+0000 7f068e28f640 1 -- 172.21.15.161:0/466948105 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f067406b4d0 msgr2=0x7f067406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:21.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:21.019+0000 7f068e28f640 1 --2- 172.21.15.161:0/466948105 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f067406b4d0 0x7f067406d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f067c007fd0 tx=0x7f067c009040 comp rx=0 tx=0).stop 2024-03-28T23:30:21.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:21.019+0000 7f068e28f640 1 -- 172.21.15.161:0/466948105 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0688150590 msgr2=0x7f06880c4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:21.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:21.019+0000 7f068e28f640 1 --2- 172.21.15.161:0/466948105 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0688150590 0x7f06880c4240 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f068000cb50 tx=0x7f068000a430 comp rx=0 tx=0).stop 2024-03-28T23:30:21.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:21.019+0000 7f068e28f640 1 -- 172.21.15.161:0/466948105 shutdown_connections 2024-03-28T23:30:21.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:21.019+0000 7f068e28f640 1 --2- 172.21.15.161:0/466948105 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f067406b4d0 0x7f067406d990 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:21.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:21.019+0000 7f068e28f640 1 --2- 172.21.15.161:0/466948105 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0688150590 0x7f06880c4240 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:21.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:21.019+0000 7f068e28f640 1 --2- 172.21.15.161:0/466948105 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06880aacb0 0x7f0688152640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:21.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:21.019+0000 7f068e28f640 1 --2- 172.21.15.161:0/466948105 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06880a8fc0 0x7f06881ecbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:21.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:21.019+0000 7f068e28f640 1 -- 172.21.15.161:0/466948105 >> 172.21.15.161:0/466948105 conn(0x7f06880c6670 msgr2=0x7f06881489a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:21.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:21.019+0000 7f068e28f640 1 -- 172.21.15.161:0/466948105 shutdown_connections 2024-03-28T23:30:21.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:21.019+0000 7f068e28f640 1 -- 172.21.15.161:0/466948105 wait complete. 2024-03-28T23:30:21.031 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-28T23:30:21.031 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T23:30:21.031 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi161.front.sepia.ceph.com 2024-03-28T23:30:21.031 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-28T23:30:21.031 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T23:30:21.031 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-28T23:30:21.031 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T23:30:21.031 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T23:30:21.031 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-28T23:30:21.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:21.032 DEBUG:teuthology.orchestra.run.smithi161:> ip addr 2024-03-28T23:30:21.047 INFO:teuthology.orchestra.run.smithi161.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T23:30:21.047 INFO:teuthology.orchestra.run.smithi161.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T23:30:21.047 INFO:teuthology.orchestra.run.smithi161.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T23:30:21.047 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-28T23:30:21.047 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 ::1/128 scope host 2024-03-28T23:30:21.047 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-28T23:30:21.047 INFO:teuthology.orchestra.run.smithi161.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T23:30:21.047 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:d9:36:7c brd ff:ff:ff:ff:ff:ff 2024-03-28T23:30:21.047 INFO:teuthology.orchestra.run.smithi161.stdout: altname enp7s0f0 2024-03-28T23:30:21.047 INFO:teuthology.orchestra.run.smithi161.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T23:30:21.047 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:d9:36:7d brd ff:ff:ff:ff:ff:ff 2024-03-28T23:30:21.048 INFO:teuthology.orchestra.run.smithi161.stdout: altname enp7s0f1 2024-03-28T23:30:21.048 INFO:teuthology.orchestra.run.smithi161.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T23:30:21.048 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:88:72:50 brd ff:ff:ff:ff:ff:ff 2024-03-28T23:30:21.048 INFO:teuthology.orchestra.run.smithi161.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T23:30:21.048 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:88:72:51 brd ff:ff:ff:ff:ff:ff 2024-03-28T23:30:21.048 INFO:teuthology.orchestra.run.smithi161.stdout: inet 172.21.15.161/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-28T23:30:21.048 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft 42736sec preferred_lft 42736sec 2024-03-28T23:30:21.048 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::ec4:7aff:fe88:7251/64 scope link 2024-03-28T23:30:21.048 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-28T23:30:21.048 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-28T23:30:21.048 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T23:30:21.048 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-28T23:30:21.048 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link add name ceph-brx type bridge 2024-03-28T23:30:21.049 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip addr flush dev ceph-brx 2024-03-28T23:30:21.049 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link set ceph-brx up 2024-03-28T23:30:21.049 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-28T23:30:21.049 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-28T23:30:21.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T23:30:21.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T23:30:21.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:21.200 DEBUG:teuthology.orchestra.run.smithi161:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T23:30:21.225 INFO:teuthology.orchestra.run.smithi161.stdout:1 2024-03-28T23:30:21.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:21.227 DEBUG:teuthology.orchestra.run.smithi161:> route 2024-03-28T23:30:21.283 INFO:teuthology.orchestra.run.smithi161.stdout:Kernel IP routing table 2024-03-28T23:30:21.283 INFO:teuthology.orchestra.run.smithi161.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T23:30:21.283 INFO:teuthology.orchestra.run.smithi161.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-28T23:30:21.284 INFO:teuthology.orchestra.run.smithi161.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-28T23:30:21.284 INFO:teuthology.orchestra.run.smithi161.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T23:30:21.284 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T23:30:21.284 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-28T23:30:21.284 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-28T23:30:21.284 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-28T23:30:21.284 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-28T23:30:21.284 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-28T23:30:21.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T23:30:21.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T23:30:21.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:21.427 DEBUG:teuthology.orchestra.run.smithi161:> ip netns list 2024-03-28T23:30:21.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:21.442 DEBUG:teuthology.orchestra.run.smithi161:> ip netns list-id 2024-03-28T23:30:21.496 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T23:30:21.496 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-28T23:30:21.496 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T23:30:21.496 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-28T23:30:21.496 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-28T23:30:21.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T23:30:21.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T23:30:21.622 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-28T23:30:21.622 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T23:30:21.623 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-28T23:30:21.623 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-28T23:30:21.623 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-28T23:30:21.623 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-28T23:30:21.623 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-28T23:30:21.623 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-28T23:30:21.623 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-28T23:30:21.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T23:30:21.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T23:30:21.750 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T23:30:21.751 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-28T23:30:21.751 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link set brx.0 up 2024-03-28T23:30:21.751 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link set dev brx.0 master ceph-brx 2024-03-28T23:30:21.751 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-28T23:30:21.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T23:30:21.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T23:30:21.833 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'disabled': True} 2024-03-28T23:30:21.833 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T23:30:21.834 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-28T23:30:21.849 INFO:teuthology.orchestra.run.smithi161.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-28T23:30:21.849 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T23:30:21.849 DEBUG:teuthology.orchestra.run.smithi161:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-28T23:30:21.904 DEBUG:teuthology.orchestra.run.smithi161:> sudo modprobe fuse 2024-03-28T23:30:21.976 DEBUG:teuthology.orchestra.run.smithi161:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T23:30:22.034 INFO:teuthology.orchestra.run.smithi161.stdout:/proc 2024-03-28T23:30:22.034 INFO:teuthology.orchestra.run.smithi161.stdout:/sys 2024-03-28T23:30:22.034 INFO:teuthology.orchestra.run.smithi161.stdout:/dev 2024-03-28T23:30:22.034 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/security 2024-03-28T23:30:22.034 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/shm 2024-03-28T23:30:22.034 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/pts 2024-03-28T23:30:22.034 INFO:teuthology.orchestra.run.smithi161.stdout:/run 2024-03-28T23:30:22.034 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/cgroup 2024-03-28T23:30:22.034 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/pstore 2024-03-28T23:30:22.035 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/bpf 2024-03-28T23:30:22.035 INFO:teuthology.orchestra.run.smithi161.stdout:/ 2024-03-28T23:30:22.035 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/selinux 2024-03-28T23:30:22.035 INFO:teuthology.orchestra.run.smithi161.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T23:30:22.035 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/mqueue 2024-03-28T23:30:22.035 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/hugepages 2024-03-28T23:30:22.035 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/debug 2024-03-28T23:30:22.035 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/tracing 2024-03-28T23:30:22.035 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/fuse/connections 2024-03-28T23:30:22.035 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/config 2024-03-28T23:30:22.035 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T23:30:22.035 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T23:30:22.035 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T23:30:22.035 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T23:30:22.036 INFO:teuthology.orchestra.run.smithi161.stdout:/run/user/1000 2024-03-28T23:30:22.036 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph 2024-03-28T23:30:22.036 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-0 2024-03-28T23:30:22.036 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-1 2024-03-28T23:30:22.036 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-2 2024-03-28T23:30:22.036 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-3 2024-03-28T23:30:22.036 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns 2024-03-28T23:30:22.036 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T23:30:22.036 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T23:30:22.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:22.036 DEBUG:teuthology.orchestra.run.smithi161:> ls /sys/fs/fuse/connections 2024-03-28T23:30:22.089 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-28T23:30:22.090 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-28T23:30:22.132 DEBUG:teuthology.orchestra.run.smithi161:> sudo modprobe fuse 2024-03-28T23:30:22.160 DEBUG:teuthology.orchestra.run.smithi161:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T23:30:22.218 INFO:teuthology.orchestra.run.smithi161.stdout:/proc 2024-03-28T23:30:22.218 INFO:teuthology.orchestra.run.smithi161.stdout:/sys 2024-03-28T23:30:22.218 INFO:teuthology.orchestra.run.smithi161.stdout:/dev 2024-03-28T23:30:22.218 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/security 2024-03-28T23:30:22.218 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/shm 2024-03-28T23:30:22.218 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/pts 2024-03-28T23:30:22.218 INFO:teuthology.orchestra.run.smithi161.stdout:/run 2024-03-28T23:30:22.218 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/cgroup 2024-03-28T23:30:22.218 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/pstore 2024-03-28T23:30:22.219 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/bpf 2024-03-28T23:30:22.219 INFO:teuthology.orchestra.run.smithi161.stdout:/ 2024-03-28T23:30:22.219 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/selinux 2024-03-28T23:30:22.219 INFO:teuthology.orchestra.run.smithi161.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T23:30:22.219 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/mqueue 2024-03-28T23:30:22.219 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/hugepages 2024-03-28T23:30:22.219 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/debug 2024-03-28T23:30:22.219 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/tracing 2024-03-28T23:30:22.219 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/fuse/connections 2024-03-28T23:30:22.219 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/config 2024-03-28T23:30:22.219 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T23:30:22.219 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T23:30:22.219 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T23:30:22.219 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T23:30:22.219 INFO:teuthology.orchestra.run.smithi161.stdout:/run/user/1000 2024-03-28T23:30:22.220 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph 2024-03-28T23:30:22.220 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-0 2024-03-28T23:30:22.220 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-1 2024-03-28T23:30:22.220 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-2 2024-03-28T23:30:22.220 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-3 2024-03-28T23:30:22.220 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns 2024-03-28T23:30:22.220 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T23:30:22.220 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T23:30:22.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:22.220 DEBUG:teuthology.orchestra.run.smithi161:> ls /sys/fs/fuse/connections 2024-03-28T23:30:22.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:2024-03-28T23:30:22.218+0000 7f5a7c041480 -1 init, newargv = 0x5623e0ada9c0 newargc=15 2024-03-28T23:30:22.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:ceph-fuse[45946]: starting ceph client 2024-03-28T23:30:22.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:ceph-fuse[45946]: starting fuse 2024-03-28T23:30:22.236 INFO:teuthology.orchestra.run.smithi161.stdout:41 2024-03-28T23:30:22.236 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-28T23:30:22.237 DEBUG:teuthology.orchestra.run.smithi161:> sudo stdin-killer -- python3 -c ' 2024-03-28T23:30:22.237 DEBUG:teuthology.orchestra.run.smithi161:> import glob 2024-03-28T23:30:22.237 DEBUG:teuthology.orchestra.run.smithi161:> import re 2024-03-28T23:30:22.237 DEBUG:teuthology.orchestra.run.smithi161:> import os 2024-03-28T23:30:22.237 DEBUG:teuthology.orchestra.run.smithi161:> import subprocess 2024-03-28T23:30:22.237 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-28T23:30:22.237 DEBUG:teuthology.orchestra.run.smithi161:> def _find_admin_socket(client_name): 2024-03-28T23:30:22.237 DEBUG:teuthology.orchestra.run.smithi161:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-28T23:30:22.237 DEBUG:teuthology.orchestra.run.smithi161:> files = glob.glob(asok_path) 2024-03-28T23:30:22.237 DEBUG:teuthology.orchestra.run.smithi161:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-28T23:30:22.237 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-28T23:30:22.237 DEBUG:teuthology.orchestra.run.smithi161:> # Given a non-glob path, it better be there 2024-03-28T23:30:22.237 DEBUG:teuthology.orchestra.run.smithi161:> if "*" not in asok_path: 2024-03-28T23:30:22.237 DEBUG:teuthology.orchestra.run.smithi161:> assert(len(files) == 1) 2024-03-28T23:30:22.238 DEBUG:teuthology.orchestra.run.smithi161:> return files[0] 2024-03-28T23:30:22.238 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-28T23:30:22.238 DEBUG:teuthology.orchestra.run.smithi161:> for f in files: 2024-03-28T23:30:22.238 DEBUG:teuthology.orchestra.run.smithi161:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-28T23:30:22.238 DEBUG:teuthology.orchestra.run.smithi161:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-28T23:30:22.238 DEBUG:teuthology.orchestra.run.smithi161:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-28T23:30:22.238 DEBUG:teuthology.orchestra.run.smithi161:> contents = proc_f.read() 2024-03-28T23:30:22.238 DEBUG:teuthology.orchestra.run.smithi161:> if mountpoint in contents: 2024-03-28T23:30:22.238 DEBUG:teuthology.orchestra.run.smithi161:> return f 2024-03-28T23:30:22.238 DEBUG:teuthology.orchestra.run.smithi161:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-28T23:30:22.238 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-28T23:30:22.238 DEBUG:teuthology.orchestra.run.smithi161:> print(_find_admin_socket("client.0")) 2024-03-28T23:30:22.238 DEBUG:teuthology.orchestra.run.smithi161:> ' 2024-03-28T23:30:22.357 INFO:teuthology.orchestra.run.smithi161.stdout:/var/run/ceph/ceph-client.0.45946.asok 2024-03-28T23:30:22.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T23:30:22.368 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.45946.asok 2024-03-28T23:30:22.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:22.368 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.45946.asok status 2024-03-28T23:30:22.505 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:30:22.505 INFO:teuthology.orchestra.run.smithi161.stdout: "metadata": { 2024-03-28T23:30:22.506 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_sha1": "323d23dc228bbcb7f16f188fdc9c0dcb801844be", 2024-03-28T23:30:22.506 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_version": "ceph version 19.0.0-1899-g323d23dc (323d23dc228bbcb7f16f188fdc9c0dcb801844be) squid (dev)", 2024-03-28T23:30:22.506 INFO:teuthology.orchestra.run.smithi161.stdout: "entity_id": "0", 2024-03-28T23:30:22.506 INFO:teuthology.orchestra.run.smithi161.stdout: "hostname": "smithi161", 2024-03-28T23:30:22.506 INFO:teuthology.orchestra.run.smithi161.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-28T23:30:22.506 INFO:teuthology.orchestra.run.smithi161.stdout: "pid": "45946", 2024-03-28T23:30:22.506 INFO:teuthology.orchestra.run.smithi161.stdout: "root": "/" 2024-03-28T23:30:22.506 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:22.506 INFO:teuthology.orchestra.run.smithi161.stdout: "dentry_count": 0, 2024-03-28T23:30:22.506 INFO:teuthology.orchestra.run.smithi161.stdout: "dentry_pinned_count": 0, 2024-03-28T23:30:22.506 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 4854, 2024-03-28T23:30:22.506 INFO:teuthology.orchestra.run.smithi161.stdout: "inst": { 2024-03-28T23:30:22.506 INFO:teuthology.orchestra.run.smithi161.stdout: "name": { 2024-03-28T23:30:22.506 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "client", 2024-03-28T23:30:22.507 INFO:teuthology.orchestra.run.smithi161.stdout: "num": 4854 2024-03-28T23:30:22.507 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:22.507 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": { 2024-03-28T23:30:22.507 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "v1", 2024-03-28T23:30:22.507 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": "192.168.0.1:0", 2024-03-28T23:30:22.507 INFO:teuthology.orchestra.run.smithi161.stdout: "nonce": 1172484348 2024-03-28T23:30:22.507 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:22.507 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:22.507 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": { 2024-03-28T23:30:22.507 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "v1", 2024-03-28T23:30:22.507 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": "192.168.0.1:0", 2024-03-28T23:30:22.507 INFO:teuthology.orchestra.run.smithi161.stdout: "nonce": 1172484348 2024-03-28T23:30:22.507 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:22.507 INFO:teuthology.orchestra.run.smithi161.stdout: "inst_str": "client.4854 192.168.0.1:0/1172484348", 2024-03-28T23:30:22.507 INFO:teuthology.orchestra.run.smithi161.stdout: "addr_str": "192.168.0.1:0/1172484348", 2024-03-28T23:30:22.508 INFO:teuthology.orchestra.run.smithi161.stdout: "inode_count": 1, 2024-03-28T23:30:22.508 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_epoch": 9, 2024-03-28T23:30:22.508 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_epoch": 24, 2024-03-28T23:30:22.508 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_epoch_barrier": 0, 2024-03-28T23:30:22.508 INFO:teuthology.orchestra.run.smithi161.stdout: "blocklisted": false, 2024-03-28T23:30:22.508 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs" 2024-03-28T23:30:22.508 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:30:22.514 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-28T23:30:22.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.621+0000 7f4f96756640 1 Processor -- start 2024-03-28T23:30:22.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.621+0000 7f4f96756640 1 -- start start 2024-03-28T23:30:22.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.622+0000 7f4f96756640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4f90152890 0x7f4f900c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:22.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.622+0000 7f4f96756640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f900c8ed0 0x7f4f900c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:22.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.622+0000 7f4f96756640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f90158a30 con 0x7f4f900c5950 2024-03-28T23:30:22.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.622+0000 7f4f96756640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f4f900c8480 con 0x7f4f90152890 2024-03-28T23:30:22.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.622+0000 7f4f96756640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4f90159640 con 0x7f4f900c8ed0 2024-03-28T23:30:22.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.622+0000 7f4f94cf3640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f4f900c5950 0x7f4f901553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38400/0 (socket says 172.21.15.161:38400) 2024-03-28T23:30:22.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.622+0000 7f4f94cf3640 1 -- 172.21.15.161:0/1136918703 learned_addr learned my addr 172.21.15.161:0/1136918703 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:22.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.622+0000 7f4f8f7fe640 1 --2- 172.21.15.161:0/1136918703 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f900c8ed0 0x7f4f900c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:22.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.622+0000 7f4f8ffff640 1 --2- 172.21.15.161:0/1136918703 >> v2:172.21.15.170:3300/0 conn(0x7f4f90152890 0x7f4f900c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:22.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.622+0000 7f4f8effd640 1 -- 172.21.15.161:0/1136918703 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 241410745 0 0) 0x7f4f90158a30 con 0x7f4f900c5950 2024-03-28T23:30:22.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.622+0000 7f4f8effd640 1 -- 172.21.15.161:0/1136918703 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f78003520 con 0x7f4f900c5950 2024-03-28T23:30:22.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.623+0000 7f4f8effd640 1 -- 172.21.15.161:0/1136918703 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1017263174 0 0) 0x7f4f78003520 con 0x7f4f900c5950 2024-03-28T23:30:22.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.623+0000 7f4f8effd640 1 -- 172.21.15.161:0/1136918703 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f900c8ed0 msgr2=0x7f4f900c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:22.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.623+0000 7f4f8effd640 1 --2- 172.21.15.161:0/1136918703 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f900c8ed0 0x7f4f900c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:22.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.623+0000 7f4f8effd640 1 -- 172.21.15.161:0/1136918703 >> v2:172.21.15.170:3300/0 conn(0x7f4f90152890 msgr2=0x7f4f900c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:22.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.623+0000 7f4f8effd640 1 --2- 172.21.15.161:0/1136918703 >> v2:172.21.15.170:3300/0 conn(0x7f4f90152890 0x7f4f900c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:22.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.623+0000 7f4f8effd640 1 -- 172.21.15.161:0/1136918703 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f90158d10 con 0x7f4f900c5950 2024-03-28T23:30:22.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.623+0000 7f4f8effd640 1 -- 172.21.15.161:0/1136918703 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f4f880019f0 con 0x7f4f900c5950 2024-03-28T23:30:22.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.623+0000 7f4f8effd640 1 -- 172.21.15.161:0/1136918703 >> v1:172.21.15.161:6789/0 conn(0x7f4f900c5950 legacy=0x7f4f901553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:22.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.623+0000 7f4f8effd640 1 --2- 172.21.15.161:0/1136918703 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f900c8ed0 0x7f4f78003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:22.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.623+0000 7f4f8effd640 1 --2- 172.21.15.161:0/1136918703 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f78004010 0x7f4f78004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:22.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.623+0000 7f4f8effd640 1 --2- 172.21.15.161:0/1136918703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f78004d40 0x7f4f780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:22.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.623+0000 7f4f8effd640 1 -- 172.21.15.161:0/1136918703 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4f90159640 con 0x7f4f78004010 2024-03-28T23:30:22.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.623+0000 7f4f8effd640 1 -- 172.21.15.161:0/1136918703 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7800bb00 con 0x7f4f78004d40 2024-03-28T23:30:22.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.623+0000 7f4f8effd640 1 -- 172.21.15.161:0/1136918703 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4f7800bd00 con 0x7f4f900c8ed0 2024-03-28T23:30:22.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.623+0000 7f4f8f7fe640 1 --2- 172.21.15.161:0/1136918703 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f78004010 0x7f4f78004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:22.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.623+0000 7f4f8ffff640 1 --2- 172.21.15.161:0/1136918703 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f900c8ed0 0x7f4f78003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:22.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.623+0000 7f4f94cf3640 1 --2- 172.21.15.161:0/1136918703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f78004d40 0x7f4f780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:22.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.623+0000 7f4f8f7fe640 1 -- 172.21.15.161:0/1136918703 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f900c8ed0 msgr2=0x7f4f78003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:22.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.623+0000 7f4f8f7fe640 1 --2- 172.21.15.161:0/1136918703 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f900c8ed0 0x7f4f78003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:22.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.623+0000 7f4f8f7fe640 1 -- 172.21.15.161:0/1136918703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f78004d40 msgr2=0x7f4f780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:22.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.623+0000 7f4f8f7fe640 1 --2- 172.21.15.161:0/1136918703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f78004d40 0x7f4f780051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:22.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.623+0000 7f4f8f7fe640 1 -- 172.21.15.161:0/1136918703 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f78003520 con 0x7f4f78004010 2024-03-28T23:30:22.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.624+0000 7f4f8f7fe640 1 --2- 172.21.15.161:0/1136918703 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f78004010 0x7f4f78004800 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4f84009a80 tx=0x7f4f78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f690a9f3cccccb95 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:22.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.624+0000 7f4f8effd640 1 -- 172.21.15.161:0/1136918703 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f84009680 con 0x7f4f78004010 2024-03-28T23:30:22.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.624+0000 7f4f8effd640 1 -- 172.21.15.161:0/1136918703 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f84005990 con 0x7f4f78004010 2024-03-28T23:30:22.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.624+0000 7f4f8effd640 1 -- 172.21.15.161:0/1136918703 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f84005d80 con 0x7f4f78004010 2024-03-28T23:30:22.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.624+0000 7f4f96756640 1 -- 172.21.15.161:0/1136918703 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f78004010 msgr2=0x7f4f78004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:22.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.624+0000 7f4f96756640 1 --2- 172.21.15.161:0/1136918703 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f78004010 0x7f4f78004800 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4f84009a80 tx=0x7f4f78005830 comp rx=0 tx=0).stop 2024-03-28T23:30:22.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.625+0000 7f4f94cf3640 1 -- 172.21.15.161:0/1136918703 reap_dead start 2024-03-28T23:30:22.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.625+0000 7f4f96756640 1 -- 172.21.15.161:0/1136918703 shutdown_connections 2024-03-28T23:30:22.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.625+0000 7f4f96756640 1 -- 172.21.15.161:0/1136918703 >> 172.21.15.161:0/1136918703 conn(0x7f4f900ac620 msgr2=0x7f4f900aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:22.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.625+0000 7f4f96756640 1 -- 172.21.15.161:0/1136918703 shutdown_connections 2024-03-28T23:30:22.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.625+0000 7f4f96756640 1 -- 172.21.15.161:0/1136918703 wait complete. 2024-03-28T23:30:22.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.626+0000 7f4f96756640 1 Processor -- start 2024-03-28T23:30:22.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.626+0000 7f4f96756640 1 -- start start 2024-03-28T23:30:22.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.626+0000 7f4f96756640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f78004010 0x7f4f90143fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:22.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.626+0000 7f4f96756640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f90150320 0x7f4f901419c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:22.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.626+0000 7f4f96756640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f901e7ee0 0x7f4f901e8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:22.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.627+0000 7f4f96756640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4f90152c70 con 0x7f4f901e7ee0 2024-03-28T23:30:22.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.627+0000 7f4f96756640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4f901592d0 con 0x7f4f78004010 2024-03-28T23:30:22.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.627+0000 7f4f96756640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4f90152df0 con 0x7f4f90150320 2024-03-28T23:30:22.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.627+0000 7f4f94cf3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f901e7ee0 0x7f4f901e8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:22.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.627+0000 7f4f94cf3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f901e7ee0 0x7f4f901e8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51652/0 (socket says 172.21.15.161:51652) 2024-03-28T23:30:22.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.627+0000 7f4f94cf3640 1 -- 172.21.15.161:0/716875291 learned_addr learned my addr 172.21.15.161:0/716875291 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:22.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.627+0000 7f4f8f7fe640 1 --2- 172.21.15.161:0/716875291 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f90150320 0x7f4f901419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:22.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.627+0000 7f4f8ffff640 1 --2- 172.21.15.161:0/716875291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f78004010 0x7f4f90143fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:22.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.627+0000 7f4f94cf3640 1 -- 172.21.15.161:0/716875291 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f90150320 msgr2=0x7f4f901419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:22.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.627+0000 7f4f94cf3640 1 --2- 172.21.15.161:0/716875291 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f90150320 0x7f4f901419c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:22.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.627+0000 7f4f94cf3640 1 -- 172.21.15.161:0/716875291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f78004010 msgr2=0x7f4f90143fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:22.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.627+0000 7f4f94cf3640 1 --2- 172.21.15.161:0/716875291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f78004010 0x7f4f90143fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:22.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.627+0000 7f4f94cf3640 1 -- 172.21.15.161:0/716875291 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f90144620 con 0x7f4f901e7ee0 2024-03-28T23:30:22.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.627+0000 7f4f94cf3640 1 --2- 172.21.15.161:0/716875291 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f901e7ee0 0x7f4f901e8360 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4f88002230 tx=0x7f4f88007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:22.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.627+0000 7f4f8cff9640 1 -- 172.21.15.161:0/716875291 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8800aaa0 con 0x7f4f901e7ee0 2024-03-28T23:30:22.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.627+0000 7f4f8cff9640 1 -- 172.21.15.161:0/716875291 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f8800ac60 con 0x7f4f901e7ee0 2024-03-28T23:30:22.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.627+0000 7f4f8cff9640 1 -- 172.21.15.161:0/716875291 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f88013630 con 0x7f4f901e7ee0 2024-03-28T23:30:22.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.627+0000 7f4f96756640 1 -- 172.21.15.161:0/716875291 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f900c2c10 con 0x7f4f901e7ee0 2024-03-28T23:30:22.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.627+0000 7f4f96756640 1 -- 172.21.15.161:0/716875291 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f90144bc0 con 0x7f4f901e7ee0 2024-03-28T23:30:22.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.628+0000 7f4f96756640 1 -- 172.21.15.161:0/716875291 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f9004f4b0 con 0x7f4f901e7ee0 2024-03-28T23:30:22.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.629+0000 7f4f8cff9640 1 -- 172.21.15.161:0/716875291 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4f880137d0 con 0x7f4f901e7ee0 2024-03-28T23:30:22.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.629+0000 7f4f8cff9640 1 --2- 172.21.15.161:0/716875291 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4f7c06b500 0x7f4f7c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:22.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.629+0000 7f4f8ffff640 1 --2- 172.21.15.161:0/716875291 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4f7c06b500 0x7f4f7c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:22.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.629+0000 7f4f8cff9640 1 -- 172.21.15.161:0/716875291 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4f8808a7d0 con 0x7f4f901e7ee0 2024-03-28T23:30:22.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.630+0000 7f4f8ffff640 1 --2- 172.21.15.161:0/716875291 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4f7c06b500 0x7f4f7c06d9c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4f80009b80 tx=0x7f4f80008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:22.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.634+0000 7f4f8cff9640 1 -- 172.21.15.161:0/716875291 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4f880593a0 con 0x7f4f901e7ee0 2024-03-28T23:30:22.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.847+0000 7f4f96756640 1 -- 172.21.15.161:0/716875291 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4f900c8140 con 0x7f4f901e7ee0 2024-03-28T23:30:22.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.848+0000 7f4f8cff9640 1 -- 172.21.15.161:0/716875291 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f4f88058ca0 con 0x7f4f901e7ee0 2024-03-28T23:30:22.848 INFO:teuthology.orchestra.run.smithi161.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-28T23:30:22.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.849+0000 7f4f96756640 1 -- 172.21.15.161:0/716875291 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4f7c06b500 msgr2=0x7f4f7c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:22.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.849+0000 7f4f96756640 1 --2- 172.21.15.161:0/716875291 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4f7c06b500 0x7f4f7c06d9c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4f80009b80 tx=0x7f4f80008040 comp rx=0 tx=0).stop 2024-03-28T23:30:22.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.849+0000 7f4f96756640 1 -- 172.21.15.161:0/716875291 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f901e7ee0 msgr2=0x7f4f901e8360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:22.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.849+0000 7f4f96756640 1 --2- 172.21.15.161:0/716875291 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f901e7ee0 0x7f4f901e8360 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4f88002230 tx=0x7f4f88007a20 comp rx=0 tx=0).stop 2024-03-28T23:30:22.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.849+0000 7f4f96756640 1 -- 172.21.15.161:0/716875291 shutdown_connections 2024-03-28T23:30:22.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.849+0000 7f4f96756640 1 --2- 172.21.15.161:0/716875291 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4f7c06b500 0x7f4f7c06d9c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:22.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.849+0000 7f4f96756640 1 --2- 172.21.15.161:0/716875291 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f901e7ee0 0x7f4f901e8360 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:22.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.849+0000 7f4f96756640 1 --2- 172.21.15.161:0/716875291 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f90150320 0x7f4f901419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:22.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.849+0000 7f4f96756640 1 --2- 172.21.15.161:0/716875291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f78004010 0x7f4f90143fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:22.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.849+0000 7f4f96756640 1 -- 172.21.15.161:0/716875291 >> 172.21.15.161:0/716875291 conn(0x7f4f900ac620 msgr2=0x7f4f900c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:22.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.850+0000 7f4f96756640 1 -- 172.21.15.161:0/716875291 shutdown_connections 2024-03-28T23:30:22.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:22.850+0000 7f4f96756640 1 -- 172.21.15.161:0/716875291 wait complete. 2024-03-28T23:30:22.861 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-28T23:30:22.861 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T23:30:22.861 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi170.front.sepia.ceph.com 2024-03-28T23:30:22.862 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-28T23:30:22.862 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-28T23:30:22.862 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-28T23:30:22.862 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T23:30:22.862 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T23:30:22.862 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-28T23:30:22.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:22.862 DEBUG:teuthology.orchestra.run.smithi170:> ip addr 2024-03-28T23:30:22.881 INFO:teuthology.orchestra.run.smithi170.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T23:30:22.881 INFO:teuthology.orchestra.run.smithi170.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T23:30:22.881 INFO:teuthology.orchestra.run.smithi170.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T23:30:22.881 INFO:teuthology.orchestra.run.smithi170.stdout: valid_lft forever preferred_lft forever 2024-03-28T23:30:22.881 INFO:teuthology.orchestra.run.smithi170.stdout: inet6 ::1/128 scope host 2024-03-28T23:30:22.881 INFO:teuthology.orchestra.run.smithi170.stdout: valid_lft forever preferred_lft forever 2024-03-28T23:30:22.881 INFO:teuthology.orchestra.run.smithi170.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T23:30:22.881 INFO:teuthology.orchestra.run.smithi170.stdout: link/ether 0c:c4:7a:d9:3b:54 brd ff:ff:ff:ff:ff:ff 2024-03-28T23:30:22.881 INFO:teuthology.orchestra.run.smithi170.stdout: altname enp7s0f0 2024-03-28T23:30:22.881 INFO:teuthology.orchestra.run.smithi170.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T23:30:22.882 INFO:teuthology.orchestra.run.smithi170.stdout: link/ether 0c:c4:7a:d9:3b:55 brd ff:ff:ff:ff:ff:ff 2024-03-28T23:30:22.882 INFO:teuthology.orchestra.run.smithi170.stdout: altname enp7s0f1 2024-03-28T23:30:22.882 INFO:teuthology.orchestra.run.smithi170.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T23:30:22.882 INFO:teuthology.orchestra.run.smithi170.stdout: link/ether 0c:c4:7a:88:72:54 brd ff:ff:ff:ff:ff:ff 2024-03-28T23:30:22.882 INFO:teuthology.orchestra.run.smithi170.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T23:30:22.882 INFO:teuthology.orchestra.run.smithi170.stdout: link/ether 0c:c4:7a:88:72:55 brd ff:ff:ff:ff:ff:ff 2024-03-28T23:30:22.882 INFO:teuthology.orchestra.run.smithi170.stdout: inet 172.21.15.170/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-28T23:30:22.882 INFO:teuthology.orchestra.run.smithi170.stdout: valid_lft 42705sec preferred_lft 42705sec 2024-03-28T23:30:22.882 INFO:teuthology.orchestra.run.smithi170.stdout: inet6 fe80::ec4:7aff:fe88:7255/64 scope link 2024-03-28T23:30:22.882 INFO:teuthology.orchestra.run.smithi170.stdout: valid_lft forever preferred_lft forever 2024-03-28T23:30:22.882 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-28T23:30:22.883 DEBUG:teuthology.orchestra.run.smithi170:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T23:30:22.883 DEBUG:teuthology.orchestra.run.smithi170:> set -e 2024-03-28T23:30:22.883 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip link add name ceph-brx type bridge 2024-03-28T23:30:22.883 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip addr flush dev ceph-brx 2024-03-28T23:30:22.883 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip link set ceph-brx up 2024-03-28T23:30:22.883 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-28T23:30:22.883 DEBUG:teuthology.orchestra.run.smithi170:> ') 2024-03-28T23:30:22.975 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:30:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T23:30:23.078 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:30:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T23:30:23.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:23.084 DEBUG:teuthology.orchestra.run.smithi170:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T23:30:23.111 INFO:teuthology.orchestra.run.smithi170.stdout:1 2024-03-28T23:30:23.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:23.113 DEBUG:teuthology.orchestra.run.smithi170:> route 2024-03-28T23:30:23.183 INFO:teuthology.orchestra.run.smithi170.stdout:Kernel IP routing table 2024-03-28T23:30:23.184 INFO:teuthology.orchestra.run.smithi170.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T23:30:23.184 INFO:teuthology.orchestra.run.smithi170.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-28T23:30:23.184 INFO:teuthology.orchestra.run.smithi170.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-28T23:30:23.184 INFO:teuthology.orchestra.run.smithi170.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T23:30:23.184 DEBUG:teuthology.orchestra.run.smithi170:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T23:30:23.184 DEBUG:teuthology.orchestra.run.smithi170:> set -e 2024-03-28T23:30:23.184 DEBUG:teuthology.orchestra.run.smithi170:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-28T23:30:23.184 DEBUG:teuthology.orchestra.run.smithi170:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-28T23:30:23.184 DEBUG:teuthology.orchestra.run.smithi170:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-28T23:30:23.184 DEBUG:teuthology.orchestra.run.smithi170:> ') 2024-03-28T23:30:23.271 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:30:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T23:30:23.484 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:30:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T23:30:23.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:23.491 DEBUG:teuthology.orchestra.run.smithi170:> ip netns list 2024-03-28T23:30:23.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:23.506 DEBUG:teuthology.orchestra.run.smithi170:> ip netns list-id 2024-03-28T23:30:23.561 DEBUG:teuthology.orchestra.run.smithi170:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T23:30:23.561 DEBUG:teuthology.orchestra.run.smithi170:> set -e 2024-03-28T23:30:23.561 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-28T23:30:23.561 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-28T23:30:23.561 DEBUG:teuthology.orchestra.run.smithi170:> ') 2024-03-28T23:30:23.650 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:30:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T23:30:23.679 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:30:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T23:30:23.688 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-28T23:30:23.688 DEBUG:teuthology.orchestra.run.smithi170:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T23:30:23.689 DEBUG:teuthology.orchestra.run.smithi170:> set -e 2024-03-28T23:30:23.689 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-28T23:30:23.689 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-28T23:30:23.689 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-28T23:30:23.689 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-28T23:30:23.689 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-28T23:30:23.689 DEBUG:teuthology.orchestra.run.smithi170:> ') 2024-03-28T23:30:23.739 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:30:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T23:30:23.873 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:30:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T23:30:23.879 DEBUG:teuthology.orchestra.run.smithi170:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T23:30:23.880 DEBUG:teuthology.orchestra.run.smithi170:> set -e 2024-03-28T23:30:23.880 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip link set brx.0 up 2024-03-28T23:30:23.880 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip link set dev brx.0 master ceph-brx 2024-03-28T23:30:23.880 DEBUG:teuthology.orchestra.run.smithi170:> ') 2024-03-28T23:30:23.928 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:30:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T23:30:23.957 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:30:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T23:30:23.963 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {'disabled': True} 2024-03-28T23:30:23.963 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T23:30:23.963 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-28T23:30:23.984 INFO:teuthology.orchestra.run.smithi170.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-28T23:30:23.985 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T23:30:23.985 DEBUG:teuthology.orchestra.run.smithi170:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-28T23:30:24.039 DEBUG:teuthology.orchestra.run.smithi170:> sudo modprobe fuse 2024-03-28T23:30:24.107 DEBUG:teuthology.orchestra.run.smithi170:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T23:30:24.163 INFO:teuthology.orchestra.run.smithi170.stdout:/proc 2024-03-28T23:30:24.163 INFO:teuthology.orchestra.run.smithi170.stdout:/sys 2024-03-28T23:30:24.163 INFO:teuthology.orchestra.run.smithi170.stdout:/dev 2024-03-28T23:30:24.163 INFO:teuthology.orchestra.run.smithi170.stdout:/sys/kernel/security 2024-03-28T23:30:24.163 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/shm 2024-03-28T23:30:24.163 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/pts 2024-03-28T23:30:24.163 INFO:teuthology.orchestra.run.smithi170.stdout:/run 2024-03-28T23:30:24.163 INFO:teuthology.orchestra.run.smithi170.stdout:/sys/fs/cgroup 2024-03-28T23:30:24.164 INFO:teuthology.orchestra.run.smithi170.stdout:/sys/fs/pstore 2024-03-28T23:30:24.164 INFO:teuthology.orchestra.run.smithi170.stdout:/sys/fs/bpf 2024-03-28T23:30:24.164 INFO:teuthology.orchestra.run.smithi170.stdout:/ 2024-03-28T23:30:24.164 INFO:teuthology.orchestra.run.smithi170.stdout:/sys/fs/selinux 2024-03-28T23:30:24.164 INFO:teuthology.orchestra.run.smithi170.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T23:30:24.164 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/mqueue 2024-03-28T23:30:24.164 INFO:teuthology.orchestra.run.smithi170.stdout:/sys/kernel/debug 2024-03-28T23:30:24.164 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/hugepages 2024-03-28T23:30:24.164 INFO:teuthology.orchestra.run.smithi170.stdout:/sys/kernel/tracing 2024-03-28T23:30:24.164 INFO:teuthology.orchestra.run.smithi170.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T23:30:24.164 INFO:teuthology.orchestra.run.smithi170.stdout:/sys/fs/fuse/connections 2024-03-28T23:30:24.164 INFO:teuthology.orchestra.run.smithi170.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T23:30:24.164 INFO:teuthology.orchestra.run.smithi170.stdout:/sys/kernel/config 2024-03-28T23:30:24.165 INFO:teuthology.orchestra.run.smithi170.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T23:30:24.165 INFO:teuthology.orchestra.run.smithi170.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T23:30:24.165 INFO:teuthology.orchestra.run.smithi170.stdout:/run/user/1000 2024-03-28T23:30:24.165 INFO:teuthology.orchestra.run.smithi170.stdout:/var/lib/ceph 2024-03-28T23:30:24.165 INFO:teuthology.orchestra.run.smithi170.stdout:/var/lib/ceph/osd/ceph-4 2024-03-28T23:30:24.165 INFO:teuthology.orchestra.run.smithi170.stdout:/var/lib/ceph/osd/ceph-5 2024-03-28T23:30:24.165 INFO:teuthology.orchestra.run.smithi170.stdout:/var/lib/ceph/osd/ceph-6 2024-03-28T23:30:24.165 INFO:teuthology.orchestra.run.smithi170.stdout:/var/lib/ceph/osd/ceph-7 2024-03-28T23:30:24.165 INFO:teuthology.orchestra.run.smithi170.stdout:/run/netns 2024-03-28T23:30:24.165 INFO:teuthology.orchestra.run.smithi170.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-28T23:30:24.165 INFO:teuthology.orchestra.run.smithi170.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-28T23:30:24.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:24.166 DEBUG:teuthology.orchestra.run.smithi170:> ls /sys/fs/fuse/connections 2024-03-28T23:30:24.217 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-28T23:30:24.218 DEBUG:teuthology.orchestra.run.smithi170:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-28T23:30:24.260 DEBUG:teuthology.orchestra.run.smithi170:> sudo modprobe fuse 2024-03-28T23:30:24.286 DEBUG:teuthology.orchestra.run.smithi170:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T23:30:24.342 INFO:teuthology.orchestra.run.smithi170.stdout:/proc 2024-03-28T23:30:24.342 INFO:teuthology.orchestra.run.smithi170.stdout:/sys 2024-03-28T23:30:24.342 INFO:teuthology.orchestra.run.smithi170.stdout:/dev 2024-03-28T23:30:24.342 INFO:teuthology.orchestra.run.smithi170.stdout:/sys/kernel/security 2024-03-28T23:30:24.342 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/shm 2024-03-28T23:30:24.342 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/pts 2024-03-28T23:30:24.342 INFO:teuthology.orchestra.run.smithi170.stdout:/run 2024-03-28T23:30:24.342 INFO:teuthology.orchestra.run.smithi170.stdout:/sys/fs/cgroup 2024-03-28T23:30:24.342 INFO:teuthology.orchestra.run.smithi170.stdout:/sys/fs/pstore 2024-03-28T23:30:24.343 INFO:teuthology.orchestra.run.smithi170.stdout:/sys/fs/bpf 2024-03-28T23:30:24.343 INFO:teuthology.orchestra.run.smithi170.stdout:/ 2024-03-28T23:30:24.343 INFO:teuthology.orchestra.run.smithi170.stdout:/sys/fs/selinux 2024-03-28T23:30:24.343 INFO:teuthology.orchestra.run.smithi170.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T23:30:24.343 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/mqueue 2024-03-28T23:30:24.343 INFO:teuthology.orchestra.run.smithi170.stdout:/sys/kernel/debug 2024-03-28T23:30:24.343 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/hugepages 2024-03-28T23:30:24.343 INFO:teuthology.orchestra.run.smithi170.stdout:/sys/kernel/tracing 2024-03-28T23:30:24.343 INFO:teuthology.orchestra.run.smithi170.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T23:30:24.343 INFO:teuthology.orchestra.run.smithi170.stdout:/sys/fs/fuse/connections 2024-03-28T23:30:24.343 INFO:teuthology.orchestra.run.smithi170.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T23:30:24.343 INFO:teuthology.orchestra.run.smithi170.stdout:/sys/kernel/config 2024-03-28T23:30:24.343 INFO:teuthology.orchestra.run.smithi170.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T23:30:24.344 INFO:teuthology.orchestra.run.smithi170.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T23:30:24.344 INFO:teuthology.orchestra.run.smithi170.stdout:/run/user/1000 2024-03-28T23:30:24.344 INFO:teuthology.orchestra.run.smithi170.stdout:/var/lib/ceph 2024-03-28T23:30:24.344 INFO:teuthology.orchestra.run.smithi170.stdout:/var/lib/ceph/osd/ceph-4 2024-03-28T23:30:24.344 INFO:teuthology.orchestra.run.smithi170.stdout:/var/lib/ceph/osd/ceph-5 2024-03-28T23:30:24.344 INFO:teuthology.orchestra.run.smithi170.stdout:/var/lib/ceph/osd/ceph-6 2024-03-28T23:30:24.344 INFO:teuthology.orchestra.run.smithi170.stdout:/var/lib/ceph/osd/ceph-7 2024-03-28T23:30:24.344 INFO:teuthology.orchestra.run.smithi170.stdout:/run/netns 2024-03-28T23:30:24.344 INFO:teuthology.orchestra.run.smithi170.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-28T23:30:24.344 INFO:teuthology.orchestra.run.smithi170.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-28T23:30:24.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:24.344 DEBUG:teuthology.orchestra.run.smithi170:> ls /sys/fs/fuse/connections 2024-03-28T23:30:24.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi170.stderr:2024-03-28T23:30:24.345+0000 7f694f852480 -1 init, newargv = 0x562e6905b1b0 newargc=15 2024-03-28T23:30:24.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi170.stderr:ceph-fuse[42037]: starting ceph client 2024-03-28T23:30:24.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi170.stderr:ceph-fuse[42037]: starting fuse 2024-03-28T23:30:24.360 INFO:teuthology.orchestra.run.smithi170.stdout:42 2024-03-28T23:30:24.361 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-28T23:30:24.361 DEBUG:teuthology.orchestra.run.smithi170:> sudo stdin-killer -- python3 -c ' 2024-03-28T23:30:24.361 DEBUG:teuthology.orchestra.run.smithi170:> import glob 2024-03-28T23:30:24.361 DEBUG:teuthology.orchestra.run.smithi170:> import re 2024-03-28T23:30:24.361 DEBUG:teuthology.orchestra.run.smithi170:> import os 2024-03-28T23:30:24.361 DEBUG:teuthology.orchestra.run.smithi170:> import subprocess 2024-03-28T23:30:24.361 DEBUG:teuthology.orchestra.run.smithi170:> 2024-03-28T23:30:24.361 DEBUG:teuthology.orchestra.run.smithi170:> def _find_admin_socket(client_name): 2024-03-28T23:30:24.362 DEBUG:teuthology.orchestra.run.smithi170:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-28T23:30:24.362 DEBUG:teuthology.orchestra.run.smithi170:> files = glob.glob(asok_path) 2024-03-28T23:30:24.362 DEBUG:teuthology.orchestra.run.smithi170:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-28T23:30:24.362 DEBUG:teuthology.orchestra.run.smithi170:> 2024-03-28T23:30:24.362 DEBUG:teuthology.orchestra.run.smithi170:> # Given a non-glob path, it better be there 2024-03-28T23:30:24.362 DEBUG:teuthology.orchestra.run.smithi170:> if "*" not in asok_path: 2024-03-28T23:30:24.362 DEBUG:teuthology.orchestra.run.smithi170:> assert(len(files) == 1) 2024-03-28T23:30:24.362 DEBUG:teuthology.orchestra.run.smithi170:> return files[0] 2024-03-28T23:30:24.362 DEBUG:teuthology.orchestra.run.smithi170:> 2024-03-28T23:30:24.362 DEBUG:teuthology.orchestra.run.smithi170:> for f in files: 2024-03-28T23:30:24.362 DEBUG:teuthology.orchestra.run.smithi170:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-28T23:30:24.362 DEBUG:teuthology.orchestra.run.smithi170:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-28T23:30:24.363 DEBUG:teuthology.orchestra.run.smithi170:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-28T23:30:24.363 DEBUG:teuthology.orchestra.run.smithi170:> contents = proc_f.read() 2024-03-28T23:30:24.363 DEBUG:teuthology.orchestra.run.smithi170:> if mountpoint in contents: 2024-03-28T23:30:24.363 DEBUG:teuthology.orchestra.run.smithi170:> return f 2024-03-28T23:30:24.363 DEBUG:teuthology.orchestra.run.smithi170:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-28T23:30:24.363 DEBUG:teuthology.orchestra.run.smithi170:> 2024-03-28T23:30:24.363 DEBUG:teuthology.orchestra.run.smithi170:> print(_find_admin_socket("client.1")) 2024-03-28T23:30:24.363 DEBUG:teuthology.orchestra.run.smithi170:> ' 2024-03-28T23:30:24.482 INFO:teuthology.orchestra.run.smithi170.stdout:/var/run/ceph/ceph-client.1.42037.asok 2024-03-28T23:30:24.486 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-28T23:30:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T23:30:24.493 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.42037.asok 2024-03-28T23:30:24.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:24.494 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.42037.asok status 2024-03-28T23:30:24.629 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-03-28T23:30:24.629 INFO:teuthology.orchestra.run.smithi170.stdout: "metadata": { 2024-03-28T23:30:24.629 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_sha1": "323d23dc228bbcb7f16f188fdc9c0dcb801844be", 2024-03-28T23:30:24.629 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_version": "ceph version 19.0.0-1899-g323d23dc (323d23dc228bbcb7f16f188fdc9c0dcb801844be) squid (dev)", 2024-03-28T23:30:24.629 INFO:teuthology.orchestra.run.smithi170.stdout: "entity_id": "1", 2024-03-28T23:30:24.629 INFO:teuthology.orchestra.run.smithi170.stdout: "hostname": "smithi170", 2024-03-28T23:30:24.629 INFO:teuthology.orchestra.run.smithi170.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-28T23:30:24.629 INFO:teuthology.orchestra.run.smithi170.stdout: "pid": "42037", 2024-03-28T23:30:24.629 INFO:teuthology.orchestra.run.smithi170.stdout: "root": "/" 2024-03-28T23:30:24.629 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-28T23:30:24.630 INFO:teuthology.orchestra.run.smithi170.stdout: "dentry_count": 0, 2024-03-28T23:30:24.630 INFO:teuthology.orchestra.run.smithi170.stdout: "dentry_pinned_count": 0, 2024-03-28T23:30:24.630 INFO:teuthology.orchestra.run.smithi170.stdout: "id": 4565, 2024-03-28T23:30:24.630 INFO:teuthology.orchestra.run.smithi170.stdout: "inst": { 2024-03-28T23:30:24.630 INFO:teuthology.orchestra.run.smithi170.stdout: "name": { 2024-03-28T23:30:24.630 INFO:teuthology.orchestra.run.smithi170.stdout: "type": "client", 2024-03-28T23:30:24.630 INFO:teuthology.orchestra.run.smithi170.stdout: "num": 4565 2024-03-28T23:30:24.630 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-28T23:30:24.630 INFO:teuthology.orchestra.run.smithi170.stdout: "addr": { 2024-03-28T23:30:24.630 INFO:teuthology.orchestra.run.smithi170.stdout: "type": "v1", 2024-03-28T23:30:24.630 INFO:teuthology.orchestra.run.smithi170.stdout: "addr": "192.168.0.1:0", 2024-03-28T23:30:24.630 INFO:teuthology.orchestra.run.smithi170.stdout: "nonce": 2899780024 2024-03-28T23:30:24.630 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-28T23:30:24.630 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-28T23:30:24.630 INFO:teuthology.orchestra.run.smithi170.stdout: "addr": { 2024-03-28T23:30:24.631 INFO:teuthology.orchestra.run.smithi170.stdout: "type": "v1", 2024-03-28T23:30:24.631 INFO:teuthology.orchestra.run.smithi170.stdout: "addr": "192.168.0.1:0", 2024-03-28T23:30:24.631 INFO:teuthology.orchestra.run.smithi170.stdout: "nonce": 2899780024 2024-03-28T23:30:24.631 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-28T23:30:24.631 INFO:teuthology.orchestra.run.smithi170.stdout: "inst_str": "client.4565 192.168.0.1:0/2899780024", 2024-03-28T23:30:24.631 INFO:teuthology.orchestra.run.smithi170.stdout: "addr_str": "192.168.0.1:0/2899780024", 2024-03-28T23:30:24.631 INFO:teuthology.orchestra.run.smithi170.stdout: "inode_count": 1, 2024-03-28T23:30:24.631 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_epoch": 9, 2024-03-28T23:30:24.631 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_epoch": 24, 2024-03-28T23:30:24.631 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_epoch_barrier": 0, 2024-03-28T23:30:24.631 INFO:teuthology.orchestra.run.smithi170.stdout: "blocklisted": false, 2024-03-28T23:30:24.631 INFO:teuthology.orchestra.run.smithi170.stdout: "fs_name": "cephfs" 2024-03-28T23:30:24.631 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-03-28T23:30:24.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:24.637 DEBUG:teuthology.orchestra.run.smithi161:> stat --file-system '--printf=%T 2024-03-28T23:30:24.637 DEBUG:teuthology.orchestra.run.smithi161:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-28T23:30:24.652 INFO:teuthology.orchestra.run.smithi161.stdout:fuseblk 2024-03-28T23:30:24.653 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-28T23:30:24.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:24.653 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-28T23:30:24.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:24.720 DEBUG:teuthology.orchestra.run.smithi170:> stat --file-system '--printf=%T 2024-03-28T23:30:24.720 DEBUG:teuthology.orchestra.run.smithi170:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-28T23:30:24.735 INFO:teuthology.orchestra.run.smithi170.stdout:fuseblk 2024-03-28T23:30:24.735 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-28T23:30:24.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:24.736 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-28T23:30:24.803 INFO:teuthology.run_tasks:Running task exec... 2024-03-28T23:30:24.813 INFO:teuthology.task.exec:Executing custom commands... 2024-03-28T23:30:24.813 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi161.front.sepia.ceph.com 2024-03-28T23:30:24.813 DEBUG:teuthology.orchestra.run.smithi161:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/crash/warn_recent_interval 0' 2024-03-28T23:30:24.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.916+0000 7fab5888f640 1 Processor -- start 2024-03-28T23:30:24.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.916+0000 7fab5888f640 1 -- start start 2024-03-28T23:30:24.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.916+0000 7fab5888f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fab5406c180 0x7fab5406c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:24.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.916+0000 7fab5888f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fab5406f430 0x7fab540bde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:24.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.916+0000 7fab5888f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fab5406dea0 0x7fab5406e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:24.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.916+0000 7fab5888f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fab54155d70 con 0x7fab5406c180 2024-03-28T23:30:24.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.916+0000 7fab5888f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fab540bc630 con 0x7fab5406dea0 2024-03-28T23:30:24.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.916+0000 7fab5888f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fab5414f7c0 con 0x7fab5406f430 2024-03-28T23:30:24.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.917+0000 7fab5259c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fab5406c180 0x7fab5406c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:24.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.917+0000 7fab5259c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fab5406c180 0x7fab5406c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51654/0 (socket says 172.21.15.161:51654) 2024-03-28T23:30:24.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.917+0000 7fab5259c640 1 -- 172.21.15.161:0/31541682 learned_addr learned my addr 172.21.15.161:0/31541682 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:24.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.917+0000 7fab49d9b640 1 --2- 172.21.15.161:0/31541682 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fab5406f430 0x7fab540bde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:24.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.917+0000 7fab52d9d640 1 --2- 172.21.15.161:0/31541682 >> v2:172.21.15.170:3300/0 conn(0x7fab5406dea0 0x7fab5406e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:24.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.917+0000 7fab5259c640 1 -- 172.21.15.161:0/31541682 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fab5406f430 msgr2=0x7fab540bde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:24.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.917+0000 7fab5259c640 1 --2- 172.21.15.161:0/31541682 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fab5406f430 0x7fab540bde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:24.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.917+0000 7fab5259c640 1 -- 172.21.15.161:0/31541682 >> v2:172.21.15.170:3300/0 conn(0x7fab5406dea0 msgr2=0x7fab5406e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:24.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.917+0000 7fab5259c640 1 --2- 172.21.15.161:0/31541682 >> v2:172.21.15.170:3300/0 conn(0x7fab5406dea0 0x7fab5406e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:24.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.917+0000 7fab5259c640 1 -- 172.21.15.161:0/31541682 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab541562a0 con 0x7fab5406c180 2024-03-28T23:30:24.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.918+0000 7fab5259c640 1 --2- 172.21.15.161:0/31541682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab5406c180 0x7fab5406c560 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fab54070c40 tx=0x7fab4402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=259116d39a9f7d78 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.918+0000 7fab51d9b640 1 -- 172.21.15.161:0/31541682 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4402e070 con 0x7fab5406c180 2024-03-28T23:30:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.918+0000 7fab51d9b640 1 -- 172.21.15.161:0/31541682 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab44037070 con 0x7fab5406c180 2024-03-28T23:30:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.918+0000 7fab51d9b640 1 -- 172.21.15.161:0/31541682 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab44031a00 con 0x7fab5406c180 2024-03-28T23:30:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.918+0000 7fab5888f640 1 -- 172.21.15.161:0/31541682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab5406c180 msgr2=0x7fab5406c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.918+0000 7fab5888f640 1 --2- 172.21.15.161:0/31541682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab5406c180 0x7fab5406c560 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fab54070c40 tx=0x7fab4402b880 comp rx=0 tx=0).stop 2024-03-28T23:30:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.918+0000 7fab5888f640 1 -- 172.21.15.161:0/31541682 shutdown_connections 2024-03-28T23:30:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.918+0000 7fab5888f640 1 --2- 172.21.15.161:0/31541682 >> v2:172.21.15.170:3300/0 conn(0x7fab5406dea0 0x7fab5406e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.918+0000 7fab5888f640 1 --2- 172.21.15.161:0/31541682 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fab5406f430 0x7fab540bde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.918+0000 7fab5888f640 1 --2- 172.21.15.161:0/31541682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab5406c180 0x7fab5406c560 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.918+0000 7fab5888f640 1 -- 172.21.15.161:0/31541682 >> 172.21.15.161:0/31541682 conn(0x7fab540c3cb0 msgr2=0x7fab540c40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.918+0000 7fab5888f640 1 -- 172.21.15.161:0/31541682 shutdown_connections 2024-03-28T23:30:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.918+0000 7fab5888f640 1 -- 172.21.15.161:0/31541682 wait complete. 2024-03-28T23:30:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.920+0000 7fab5888f640 1 Processor -- start 2024-03-28T23:30:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.920+0000 7fab5888f640 1 -- start start 2024-03-28T23:30:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.920+0000 7fab5888f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab5406c180 0x7fab540729a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.920+0000 7fab5888f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fab5406dea0 0x7fab54071b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.920+0000 7fab5888f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fab5406f430 0x7fab5413a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.920+0000 7fab5888f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fab540bc300 con 0x7fab5406c180 2024-03-28T23:30:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.920+0000 7fab5888f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fab540c1e40 con 0x7fab5406f430 2024-03-28T23:30:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.920+0000 7fab5888f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fab5406e850 con 0x7fab5406dea0 2024-03-28T23:30:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.920+0000 7fab5259c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab5406c180 0x7fab540729a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.921+0000 7fab5259c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab5406c180 0x7fab540729a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51670/0 (socket says 172.21.15.161:51670) 2024-03-28T23:30:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.921+0000 7fab52d9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fab5406f430 0x7fab5413a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.921+0000 7fab5259c640 1 -- 172.21.15.161:0/2746572834 learned_addr learned my addr 172.21.15.161:0/2746572834 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.921+0000 7fab49d9b640 1 --2- 172.21.15.161:0/2746572834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fab5406dea0 0x7fab54071b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.921+0000 7fab5259c640 1 -- 172.21.15.161:0/2746572834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fab5406dea0 msgr2=0x7fab54071b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:24.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.921+0000 7fab5259c640 1 --2- 172.21.15.161:0/2746572834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fab5406dea0 0x7fab54071b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:24.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.921+0000 7fab5259c640 1 -- 172.21.15.161:0/2746572834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fab5406f430 msgr2=0x7fab5413a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:24.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.921+0000 7fab5259c640 1 --2- 172.21.15.161:0/2746572834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fab5406f430 0x7fab5413a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:24.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.921+0000 7fab5259c640 1 -- 172.21.15.161:0/2746572834 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab541ef3e0 con 0x7fab5406c180 2024-03-28T23:30:24.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.921+0000 7fab49d9b640 1 --2- 172.21.15.161:0/2746572834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fab5406dea0 0x7fab54071b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:24.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.921+0000 7fab52d9d640 1 --2- 172.21.15.161:0/2746572834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fab5406f430 0x7fab5413a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:24.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.921+0000 7fab5259c640 1 --2- 172.21.15.161:0/2746572834 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab5406c180 0x7fab540729a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fab44000c00 tx=0x7fab440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:24.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.921+0000 7fab4b7fe640 1 -- 172.21.15.161:0/2746572834 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab44032670 con 0x7fab5406c180 2024-03-28T23:30:24.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.921+0000 7fab4b7fe640 1 -- 172.21.15.161:0/2746572834 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab44032810 con 0x7fab5406c180 2024-03-28T23:30:24.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.921+0000 7fab5888f640 1 -- 172.21.15.161:0/2746572834 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab5413b790 con 0x7fab5406c180 2024-03-28T23:30:24.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.921+0000 7fab4b7fe640 1 -- 172.21.15.161:0/2746572834 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab44032c00 con 0x7fab5406c180 2024-03-28T23:30:24.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.921+0000 7fab5888f640 1 -- 172.21.15.161:0/2746572834 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab5413c100 con 0x7fab5406c180 2024-03-28T23:30:24.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.923+0000 7fab4b7fe640 1 -- 172.21.15.161:0/2746572834 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fab4404c060 con 0x7fab5406c180 2024-03-28T23:30:24.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.923+0000 7fab5888f640 1 -- 172.21.15.161:0/2746572834 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab541eeef0 con 0x7fab5406c180 2024-03-28T23:30:24.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.923+0000 7fab4b7fe640 1 --2- 172.21.15.161:0/2746572834 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fab3406b4d0 0x7fab3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:24.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.923+0000 7fab49d9b640 1 --2- 172.21.15.161:0/2746572834 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fab3406b4d0 0x7fab3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:24.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.923+0000 7fab4b7fe640 1 -- 172.21.15.161:0/2746572834 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fab440abcb0 con 0x7fab5406c180 2024-03-28T23:30:24.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.924+0000 7fab49d9b640 1 --2- 172.21.15.161:0/2746572834 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fab3406b4d0 0x7fab3406d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fab54152d30 tx=0x7fab3c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:24.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:24.930+0000 7fab4b7fe640 1 -- 172.21.15.161:0/2746572834 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fab44040070 con 0x7fab5406c180 2024-03-28T23:30:25.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.106+0000 7fab5888f640 1 -- 172.21.15.161:0/2746572834 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mgr/crash/warn_recent_interval}] v 0) -- 0x7fab540bcd30 con 0x7fab5406c180 2024-03-28T23:30:25.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.110+0000 7fab4b7fe640 1 -- 172.21.15.161:0/2746572834 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/crash/warn_recent_interval}]=0 v10)=0 v10) ==== 128+0+0 (secure 0 0 0) 0x7fab4407a270 con 0x7fab5406c180 2024-03-28T23:30:25.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.112+0000 7fab5888f640 1 -- 172.21.15.161:0/2746572834 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fab3406b4d0 msgr2=0x7fab3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:25.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.112+0000 7fab5888f640 1 --2- 172.21.15.161:0/2746572834 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fab3406b4d0 0x7fab3406d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fab54152d30 tx=0x7fab3c009190 comp rx=0 tx=0).stop 2024-03-28T23:30:25.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.112+0000 7fab5888f640 1 -- 172.21.15.161:0/2746572834 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab5406c180 msgr2=0x7fab540729a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:25.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.112+0000 7fab5888f640 1 --2- 172.21.15.161:0/2746572834 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab5406c180 0x7fab540729a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fab44000c00 tx=0x7fab440046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.112+0000 7fab5888f640 1 -- 172.21.15.161:0/2746572834 shutdown_connections 2024-03-28T23:30:25.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.112+0000 7fab5888f640 1 --2- 172.21.15.161:0/2746572834 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fab3406b4d0 0x7fab3406d990 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.112+0000 7fab5888f640 1 --2- 172.21.15.161:0/2746572834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fab5406f430 0x7fab5413a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.112+0000 7fab5888f640 1 --2- 172.21.15.161:0/2746572834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fab5406dea0 0x7fab54071b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.112+0000 7fab5888f640 1 --2- 172.21.15.161:0/2746572834 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab5406c180 0x7fab540729a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.112+0000 7fab5888f640 1 -- 172.21.15.161:0/2746572834 >> 172.21.15.161:0/2746572834 conn(0x7fab540c3cb0 msgr2=0x7fab540c4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:25.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.112+0000 7fab5888f640 1 -- 172.21.15.161:0/2746572834 shutdown_connections 2024-03-28T23:30:25.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.112+0000 7fab5888f640 1 -- 172.21.15.161:0/2746572834 wait complete. 2024-03-28T23:30:25.124 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-28T23:30:25.139 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:30:25.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.241+0000 7ff881884640 1 Processor -- start 2024-03-28T23:30:25.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.241+0000 7ff881884640 1 -- start start 2024-03-28T23:30:25.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.242+0000 7ff881884640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff87c152890 0x7ff87c0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:25.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.242+0000 7ff881884640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff87c0c8ed0 0x7ff87c0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:25.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.242+0000 7ff881884640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff87c158a30 con 0x7ff87c0c5950 2024-03-28T23:30:25.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.242+0000 7ff881884640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff87c0c8480 con 0x7ff87c0c8ed0 2024-03-28T23:30:25.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.242+0000 7ff881884640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7ff87c159640 con 0x7ff87c152890 2024-03-28T23:30:25.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.242+0000 7ff87a7fc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff87c0c8ed0 0x7ff87c0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:25.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.242+0000 7ff87b7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ff87c0c5950 0x7ff87c1553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38408/0 (socket says 172.21.15.161:38408) 2024-03-28T23:30:25.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.242+0000 7ff87b7fe640 1 -- 172.21.15.161:0/3198896434 learned_addr learned my addr 172.21.15.161:0/3198896434 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:25.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.242+0000 7ff87affd640 1 --2- 172.21.15.161:0/3198896434 >> v2:172.21.15.170:3300/0 conn(0x7ff87c152890 0x7ff87c0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:25.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.242+0000 7ff87a7fc640 1 -- 172.21.15.161:0/3198896434 >> v2:172.21.15.170:3300/0 conn(0x7ff87c152890 msgr2=0x7ff87c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:25.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.242+0000 7ff87a7fc640 1 --2- 172.21.15.161:0/3198896434 >> v2:172.21.15.170:3300/0 conn(0x7ff87c152890 0x7ff87c0c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.242+0000 7ff87a7fc640 1 -- 172.21.15.161:0/3198896434 >> v1:172.21.15.161:6789/0 conn(0x7ff87c0c5950 legacy=0x7ff87c1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:25.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.242+0000 7ff87a7fc640 1 -- 172.21.15.161:0/3198896434 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff87c158d10 con 0x7ff87c0c8ed0 2024-03-28T23:30:25.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.242+0000 7ff879ffb640 1 -- 172.21.15.161:0/3198896434 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2939213925 0 0) 0x7ff87c158a30 con 0x7ff87c0c5950 2024-03-28T23:30:25.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.244+0000 7ff87a7fc640 1 --2- 172.21.15.161:0/3198896434 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff87c0c8ed0 0x7ff87c0c5410 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff87c08cc80 tx=0x7ff86c02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1bc57704133a21e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:25.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.244+0000 7ff879ffb640 1 -- 172.21.15.161:0/3198896434 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86c038070 con 0x7ff87c0c8ed0 2024-03-28T23:30:25.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.244+0000 7ff879ffb640 1 -- 172.21.15.161:0/3198896434 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff86c031700 con 0x7ff87c0c8ed0 2024-03-28T23:30:25.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.244+0000 7ff879ffb640 1 -- 172.21.15.161:0/3198896434 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86c031ad0 con 0x7ff87c0c8ed0 2024-03-28T23:30:25.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.244+0000 7ff881884640 1 -- 172.21.15.161:0/3198896434 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff87c0c8ed0 msgr2=0x7ff87c0c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:25.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.244+0000 7ff881884640 1 --2- 172.21.15.161:0/3198896434 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff87c0c8ed0 0x7ff87c0c5410 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff87c08cc80 tx=0x7ff86c02b7a0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.244+0000 7ff881884640 1 -- 172.21.15.161:0/3198896434 shutdown_connections 2024-03-28T23:30:25.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.244+0000 7ff881884640 1 --2- 172.21.15.161:0/3198896434 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff87c0c8ed0 0x7ff87c0c5410 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.244+0000 7ff881884640 1 --2- 172.21.15.161:0/3198896434 >> v2:172.21.15.170:3300/0 conn(0x7ff87c152890 0x7ff87c0c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.244+0000 7ff881884640 1 -- 172.21.15.161:0/3198896434 >> 172.21.15.161:0/3198896434 conn(0x7ff87c0ac620 msgr2=0x7ff87c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:25.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.244+0000 7ff881884640 1 -- 172.21.15.161:0/3198896434 shutdown_connections 2024-03-28T23:30:25.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.244+0000 7ff881884640 1 -- 172.21.15.161:0/3198896434 wait complete. 2024-03-28T23:30:25.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.245+0000 7ff881884640 1 Processor -- start 2024-03-28T23:30:25.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.246+0000 7ff881884640 1 -- start start 2024-03-28T23:30:25.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.246+0000 7ff881884640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff87c0c5950 0x7ff87c157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:25.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.246+0000 7ff881884640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff87c0c8ed0 0x7ff87c1ecf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:25.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.246+0000 7ff881884640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff87c152890 0x7ff87c1ed480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:25.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.246+0000 7ff881884640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff87c152c70 con 0x7ff87c0c5950 2024-03-28T23:30:25.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.246+0000 7ff881884640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff87c1592d0 con 0x7ff87c152890 2024-03-28T23:30:25.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.246+0000 7ff881884640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff87c152df0 con 0x7ff87c0c8ed0 2024-03-28T23:30:25.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.246+0000 7ff87affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff87c0c5950 0x7ff87c157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:25.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.246+0000 7ff87affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff87c0c5950 0x7ff87c157790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51678/0 (socket says 172.21.15.161:51678) 2024-03-28T23:30:25.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.246+0000 7ff87affd640 1 -- 172.21.15.161:0/841546426 learned_addr learned my addr 172.21.15.161:0/841546426 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:25.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.247+0000 7ff87b7fe640 1 --2- 172.21.15.161:0/841546426 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff87c152890 0x7ff87c1ed480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:25.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.247+0000 7ff87a7fc640 1 --2- 172.21.15.161:0/841546426 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff87c0c8ed0 0x7ff87c1ecf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:25.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.247+0000 7ff87affd640 1 -- 172.21.15.161:0/841546426 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff87c0c8ed0 msgr2=0x7ff87c1ecf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:25.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.247+0000 7ff87affd640 1 --2- 172.21.15.161:0/841546426 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff87c0c8ed0 0x7ff87c1ecf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.247+0000 7ff87affd640 1 -- 172.21.15.161:0/841546426 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff87c152890 msgr2=0x7ff87c1ed480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:25.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.247+0000 7ff87affd640 1 --2- 172.21.15.161:0/841546426 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff87c152890 0x7ff87c1ed480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.247+0000 7ff87affd640 1 -- 172.21.15.161:0/841546426 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff87c1580c0 con 0x7ff87c0c5950 2024-03-28T23:30:25.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.247+0000 7ff87b7fe640 1 --2- 172.21.15.161:0/841546426 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff87c152890 0x7ff87c1ed480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:25.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.247+0000 7ff87a7fc640 1 --2- 172.21.15.161:0/841546426 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff87c0c8ed0 0x7ff87c1ecf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:25.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.247+0000 7ff87affd640 1 --2- 172.21.15.161:0/841546426 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff87c0c5950 0x7ff87c157790 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff86400d800 tx=0x7ff86400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:25.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.247+0000 7ff873fff640 1 -- 172.21.15.161:0/841546426 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86400b6c0 con 0x7ff87c0c5950 2024-03-28T23:30:25.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.247+0000 7ff873fff640 1 -- 172.21.15.161:0/841546426 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff864019070 con 0x7ff87c0c5950 2024-03-28T23:30:25.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.247+0000 7ff873fff640 1 -- 172.21.15.161:0/841546426 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86400ba80 con 0x7ff87c0c5950 2024-03-28T23:30:25.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.247+0000 7ff881884640 1 -- 172.21.15.161:0/841546426 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff87c1583a0 con 0x7ff87c0c5950 2024-03-28T23:30:25.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.247+0000 7ff881884640 1 -- 172.21.15.161:0/841546426 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff87c157cd0 con 0x7ff87c0c5950 2024-03-28T23:30:25.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.248+0000 7ff881884640 1 -- 172.21.15.161:0/841546426 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff87c0c1b40 con 0x7ff87c0c5950 2024-03-28T23:30:25.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.248+0000 7ff873fff640 1 -- 172.21.15.161:0/841546426 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff86400bc20 con 0x7ff87c0c5950 2024-03-28T23:30:25.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.249+0000 7ff873fff640 1 --2- 172.21.15.161:0/841546426 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff85c06b500 0x7ff85c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:25.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.249+0000 7ff873fff640 1 -- 172.21.15.161:0/841546426 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff86408a970 con 0x7ff87c0c5950 2024-03-28T23:30:25.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.249+0000 7ff87a7fc640 1 --2- 172.21.15.161:0/841546426 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff85c06b500 0x7ff85c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:25.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.249+0000 7ff87a7fc640 1 --2- 172.21.15.161:0/841546426 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff85c06b500 0x7ff85c06d9c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff86c035000 tx=0x7ff86c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:25.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.253+0000 7ff873fff640 1 -- 172.21.15.161:0/841546426 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff86401c360 con 0x7ff87c0c5950 2024-03-28T23:30:25.303 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:30:25.306 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:30:25.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.436+0000 7ff881884640 1 -- 172.21.15.161:0/841546426 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff87c1ea510 con 0x7ff87c0c5950 2024-03-28T23:30:25.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.438+0000 7ff873fff640 1 -- 172.21.15.161:0/841546426 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7ff864059540 con 0x7ff87c0c5950 2024-03-28T23:30:25.439 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:25.439 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":24,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:29:50.946843+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:30:25.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.440+0000 7ff881884640 1 -- 172.21.15.161:0/841546426 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff85c06b500 msgr2=0x7ff85c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:25.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.440+0000 7ff881884640 1 --2- 172.21.15.161:0/841546426 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff85c06b500 0x7ff85c06d9c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff86c035000 tx=0x7ff86c008030 comp rx=0 tx=0).stop 2024-03-28T23:30:25.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.440+0000 7ff881884640 1 -- 172.21.15.161:0/841546426 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff87c0c5950 msgr2=0x7ff87c157790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:25.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.440+0000 7ff881884640 1 --2- 172.21.15.161:0/841546426 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff87c0c5950 0x7ff87c157790 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff86400d800 tx=0x7ff86400dcc0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.440+0000 7ff881884640 1 -- 172.21.15.161:0/841546426 shutdown_connections 2024-03-28T23:30:25.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.440+0000 7ff881884640 1 --2- 172.21.15.161:0/841546426 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff85c06b500 0x7ff85c06d9c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.440+0000 7ff881884640 1 --2- 172.21.15.161:0/841546426 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff87c152890 0x7ff87c1ed480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.440+0000 7ff881884640 1 --2- 172.21.15.161:0/841546426 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff87c0c8ed0 0x7ff87c1ecf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.440+0000 7ff881884640 1 --2- 172.21.15.161:0/841546426 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff87c0c5950 0x7ff87c157790 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.440+0000 7ff881884640 1 -- 172.21.15.161:0/841546426 >> 172.21.15.161:0/841546426 conn(0x7ff87c0ac620 msgr2=0x7ff87c0c9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:25.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.440+0000 7ff881884640 1 -- 172.21.15.161:0/841546426 shutdown_connections 2024-03-28T23:30:25.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.440+0000 7ff881884640 1 -- 172.21.15.161:0/841546426 wait complete. 2024-03-28T23:30:25.452 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:29:44.021375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:29:44.872102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T23:29:48.591063+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T23:30:25.453 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:30:25.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.557+0000 7fac1979a640 1 Processor -- start 2024-03-28T23:30:25.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.557+0000 7fac1979a640 1 -- start start 2024-03-28T23:30:25.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.558+0000 7fac1979a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac140c1a70 0x7fac14071c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:25.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.558+0000 7fac1979a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fac14072190 0x7fac1415c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:25.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.558+0000 7fac1979a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac140c47b0 con 0x7fac14071870 2024-03-28T23:30:25.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.558+0000 7fac1979a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fac14072620 con 0x7fac14072190 2024-03-28T23:30:25.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.558+0000 7fac1979a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fac14071320 con 0x7fac140c1a70 2024-03-28T23:30:25.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.558+0000 7fac137fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fac14072190 0x7fac1415c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:25.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.558+0000 7fac137fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fac14072190 0x7fac1415c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:50210/0 (socket says 172.21.15.161:50210) 2024-03-28T23:30:25.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.558+0000 7fac137fe640 1 -- 172.21.15.161:0/4162053348 learned_addr learned my addr 172.21.15.161:0/4162053348 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:25.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.558+0000 7fac127fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac140c1a70 0x7fac14071c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:25.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.558+0000 7fac12ffd640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fac14071870 0x7fac1406c260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:34364/0 (socket says 172.21.15.161:34364) 2024-03-28T23:30:25.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.558+0000 7fac137fe640 1 -- 172.21.15.161:0/4162053348 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac140c1a70 msgr2=0x7fac14071c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:25.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.558+0000 7fac137fe640 1 --2- 172.21.15.161:0/4162053348 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac140c1a70 0x7fac14071c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.558+0000 7fac137fe640 1 -- 172.21.15.161:0/4162053348 >> v1:172.21.15.170:6789/0 conn(0x7fac14071870 legacy=0x7fac1406c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:25.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.558+0000 7fac137fe640 1 -- 172.21.15.161:0/4162053348 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac14070de0 con 0x7fac14072190 2024-03-28T23:30:25.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.558+0000 7fac11ffb640 1 -- 172.21.15.161:0/4162053348 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3226089542 0 0) 0x7fac04001110 con 0x7fac14071870 2024-03-28T23:30:25.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.558+0000 7fac127fc640 1 --2- 172.21.15.161:0/4162053348 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac140c1a70 0x7fac14071c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:25.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.560+0000 7fac137fe640 1 --2- 172.21.15.161:0/4162053348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac14072190 0x7fac1415c860 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fac0c00d150 tx=0x7fac0c02e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0b04a9d4926ec75 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:25.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.560+0000 7fac11ffb640 1 -- 172.21.15.161:0/4162053348 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0c02e9e0 con 0x7fac14072190 2024-03-28T23:30:25.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.560+0000 7fac11ffb640 1 -- 172.21.15.161:0/4162053348 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac0c02eb80 con 0x7fac14072190 2024-03-28T23:30:25.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.560+0000 7fac11ffb640 1 -- 172.21.15.161:0/4162053348 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0c009600 con 0x7fac14072190 2024-03-28T23:30:25.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.560+0000 7fac1979a640 1 -- 172.21.15.161:0/4162053348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac14072190 msgr2=0x7fac1415c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:25.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.560+0000 7fac1979a640 1 --2- 172.21.15.161:0/4162053348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac14072190 0x7fac1415c860 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fac0c00d150 tx=0x7fac0c02e040 comp rx=0 tx=0).stop 2024-03-28T23:30:25.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.560+0000 7fac1979a640 1 -- 172.21.15.161:0/4162053348 shutdown_connections 2024-03-28T23:30:25.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.560+0000 7fac1979a640 1 --2- 172.21.15.161:0/4162053348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac14072190 0x7fac1415c860 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.560+0000 7fac1979a640 1 --2- 172.21.15.161:0/4162053348 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac140c1a70 0x7fac14071c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.560+0000 7fac1979a640 1 -- 172.21.15.161:0/4162053348 >> 172.21.15.161:0/4162053348 conn(0x7fac140c3b70 msgr2=0x7fac140c3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:25.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.561+0000 7fac1979a640 1 -- 172.21.15.161:0/4162053348 shutdown_connections 2024-03-28T23:30:25.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.561+0000 7fac1979a640 1 -- 172.21.15.161:0/4162053348 wait complete. 2024-03-28T23:30:25.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.562+0000 7fac1979a640 1 Processor -- start 2024-03-28T23:30:25.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.562+0000 7fac1979a640 1 -- start start 2024-03-28T23:30:25.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.563+0000 7fac1979a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac14071870 0x7fac14144640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:25.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.563+0000 7fac1979a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac14072190 0x7fac141e6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:25.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.563+0000 7fac1979a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac140c1a70 0x7fac141e6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:25.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.563+0000 7fac1979a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fac14152ac0 con 0x7fac140c1a70 2024-03-28T23:30:25.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.563+0000 7fac1979a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fac140c1ef0 con 0x7fac14071870 2024-03-28T23:30:25.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.563+0000 7fac1979a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fac1414c470 con 0x7fac14072190 2024-03-28T23:30:25.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.564+0000 7fac137fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac140c1a70 0x7fac141e6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:25.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.564+0000 7fac12ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac14071870 0x7fac14144640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:25.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.564+0000 7fac127fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac14072190 0x7fac141e6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:25.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.564+0000 7fac137fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac140c1a70 0x7fac141e6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51688/0 (socket says 172.21.15.161:51688) 2024-03-28T23:30:25.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.564+0000 7fac127fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac14072190 0x7fac141e6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:38078/0 (socket says 172.21.15.161:38078) 2024-03-28T23:30:25.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.564+0000 7fac12ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac14071870 0x7fac14144640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:50212/0 (socket says 172.21.15.161:50212) 2024-03-28T23:30:25.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.564+0000 7fac137fe640 1 -- 172.21.15.161:0/1496362764 learned_addr learned my addr 172.21.15.161:0/1496362764 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:25.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.564+0000 7fac137fe640 1 -- 172.21.15.161:0/1496362764 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac14072190 msgr2=0x7fac141e6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:25.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.564+0000 7fac137fe640 1 --2- 172.21.15.161:0/1496362764 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac14072190 0x7fac141e6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.564+0000 7fac137fe640 1 -- 172.21.15.161:0/1496362764 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac14071870 msgr2=0x7fac14144640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:25.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.564+0000 7fac137fe640 1 --2- 172.21.15.161:0/1496362764 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac14071870 0x7fac14144640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.564+0000 7fac137fe640 1 -- 172.21.15.161:0/1496362764 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac140c6770 con 0x7fac140c1a70 2024-03-28T23:30:25.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.564+0000 7fac137fe640 1 --2- 172.21.15.161:0/1496362764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac140c1a70 0x7fac141e6e90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fac0c00d150 tx=0x7fac0c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:25.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.564+0000 7fabfb7fe640 1 -- 172.21.15.161:0/1496362764 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0c02e9e0 con 0x7fac140c1a70 2024-03-28T23:30:25.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.564+0000 7fabfb7fe640 1 -- 172.21.15.161:0/1496362764 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac0c045070 con 0x7fac140c1a70 2024-03-28T23:30:25.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.564+0000 7fac1979a640 1 -- 172.21.15.161:0/1496362764 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac140c6cc0 con 0x7fac140c1a70 2024-03-28T23:30:25.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.564+0000 7fabfb7fe640 1 -- 172.21.15.161:0/1496362764 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0c036a60 con 0x7fac140c1a70 2024-03-28T23:30:25.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.564+0000 7fac1979a640 1 -- 172.21.15.161:0/1496362764 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac140c69e0 con 0x7fac140c1a70 2024-03-28T23:30:25.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.565+0000 7fac1979a640 1 -- 172.21.15.161:0/1496362764 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac1404f4b0 con 0x7fac140c1a70 2024-03-28T23:30:25.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.566+0000 7fabfb7fe640 1 -- 172.21.15.161:0/1496362764 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fac0c036c00 con 0x7fac140c1a70 2024-03-28T23:30:25.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.567+0000 7fabfb7fe640 1 --2- 172.21.15.161:0/1496362764 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fac0006b4d0 0x7fac0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:25.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.567+0000 7fabfb7fe640 1 -- 172.21.15.161:0/1496362764 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fac0c0ad330 con 0x7fac140c1a70 2024-03-28T23:30:25.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.567+0000 7fac12ffd640 1 --2- 172.21.15.161:0/1496362764 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fac0006b4d0 0x7fac0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:25.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.567+0000 7fac12ffd640 1 --2- 172.21.15.161:0/1496362764 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fac0006b4d0 0x7fac0006d990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fac04000f60 tx=0x7fac04007b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:25.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.571+0000 7fabfb7fe640 1 -- 172.21.15.161:0/1496362764 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fac0c07b950 con 0x7fac140c1a70 2024-03-28T23:30:25.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.833+0000 7fac1979a640 1 -- 172.21.15.161:0/1496362764 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fac14066eb0 con 0x7fac140c1a70 2024-03-28T23:30:25.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.834+0000 7fabfb7fe640 1 -- 172.21.15.161:0/1496362764 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7fac0c07f9f0 con 0x7fac140c1a70 2024-03-28T23:30:25.835 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:30:25.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.836+0000 7fac1979a640 1 -- 172.21.15.161:0/1496362764 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fac0006b4d0 msgr2=0x7fac0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:25.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.836+0000 7fac1979a640 1 --2- 172.21.15.161:0/1496362764 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fac0006b4d0 0x7fac0006d990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fac04000f60 tx=0x7fac04007b50 comp rx=0 tx=0).stop 2024-03-28T23:30:25.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.836+0000 7fac1979a640 1 -- 172.21.15.161:0/1496362764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac140c1a70 msgr2=0x7fac141e6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:25.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.836+0000 7fac1979a640 1 --2- 172.21.15.161:0/1496362764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac140c1a70 0x7fac141e6e90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fac0c00d150 tx=0x7fac0c004a40 comp rx=0 tx=0).stop 2024-03-28T23:30:25.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.836+0000 7fac1979a640 1 -- 172.21.15.161:0/1496362764 shutdown_connections 2024-03-28T23:30:25.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.836+0000 7fac1979a640 1 --2- 172.21.15.161:0/1496362764 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fac0006b4d0 0x7fac0006d990 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.836+0000 7fac1979a640 1 --2- 172.21.15.161:0/1496362764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac140c1a70 0x7fac141e6e90 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.836+0000 7fac1979a640 1 --2- 172.21.15.161:0/1496362764 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac14072190 0x7fac141e6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.836+0000 7fac1979a640 1 --2- 172.21.15.161:0/1496362764 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac14071870 0x7fac14144640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.836+0000 7fac1979a640 1 -- 172.21.15.161:0/1496362764 >> 172.21.15.161:0/1496362764 conn(0x7fac140c3b70 msgr2=0x7fac1406dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:25.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.836+0000 7fac1979a640 1 -- 172.21.15.161:0/1496362764 shutdown_connections 2024-03-28T23:30:25.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.836+0000 7fac1979a640 1 -- 172.21.15.161:0/1496362764 wait complete. 2024-03-28T23:30:25.847 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:30:25.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.950+0000 7ff914eaf640 1 Processor -- start 2024-03-28T23:30:25.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.950+0000 7ff914eaf640 1 -- start start 2024-03-28T23:30:25.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.951+0000 7ff914eaf640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff9101589b0 0x7ff910158e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:25.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.951+0000 7ff914eaf640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9100c6f60 con 0x7ff910152bc0 2024-03-28T23:30:25.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.951+0000 7ff914eaf640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9100c6120 con 0x7ff9100c0330 2024-03-28T23:30:25.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.951+0000 7ff914eaf640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7ff9100c6400 con 0x7ff9101589b0 2024-03-28T23:30:25.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.952+0000 7ff90e59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ff910152bc0 0x7ff9100c8a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38418/0 (socket says 172.21.15.161:38418) 2024-03-28T23:30:25.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.952+0000 7ff90e59c640 1 -- 172.21.15.161:0/1146039673 learned_addr learned my addr 172.21.15.161:0/1146039673 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:25.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.952+0000 7ff90ed9d640 1 --2- 172.21.15.161:0/1146039673 >> v2:172.21.15.170:3300/0 conn(0x7ff9101589b0 0x7ff910158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:25.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.952+0000 7ff90d59a640 1 -- 172.21.15.161:0/1146039673 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2990028821 0 0) 0x7ff9100c6f60 con 0x7ff910152bc0 2024-03-28T23:30:25.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.952+0000 7ff90d59a640 1 -- 172.21.15.161:0/1146039673 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8fc003520 con 0x7ff910152bc0 2024-03-28T23:30:25.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.952+0000 7ff90ed9d640 1 -- 172.21.15.161:0/1146039673 >> v1:172.21.15.170:6789/0 conn(0x7ff9100c0330 legacy=0x7ff9100c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:25.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.952+0000 7ff90ed9d640 1 -- 172.21.15.161:0/1146039673 >> v1:172.21.15.161:6789/0 conn(0x7ff910152bc0 legacy=0x7ff9100c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:25.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.952+0000 7ff90ed9d640 1 -- 172.21.15.161:0/1146039673 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9100c7240 con 0x7ff9101589b0 2024-03-28T23:30:25.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.953+0000 7ff90ed9d640 1 --2- 172.21.15.161:0/1146039673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9101589b0 0x7ff910158e00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff908009d90 tx=0x7ff90802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5257c954b9680ca2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:25.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.953+0000 7ff90d59a640 1 -- 172.21.15.161:0/1146039673 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff908039070 con 0x7ff9101589b0 2024-03-28T23:30:25.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.953+0000 7ff90d59a640 1 -- 172.21.15.161:0/1146039673 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff90802cc60 con 0x7ff9101589b0 2024-03-28T23:30:25.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.953+0000 7ff90d59a640 1 -- 172.21.15.161:0/1146039673 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff908034660 con 0x7ff9101589b0 2024-03-28T23:30:25.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.953+0000 7ff914eaf640 1 -- 172.21.15.161:0/1146039673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9101589b0 msgr2=0x7ff910158e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:25.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.953+0000 7ff914eaf640 1 --2- 172.21.15.161:0/1146039673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9101589b0 0x7ff910158e00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff908009d90 tx=0x7ff90802beb0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.953+0000 7ff914eaf640 1 -- 172.21.15.161:0/1146039673 shutdown_connections 2024-03-28T23:30:25.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.953+0000 7ff914eaf640 1 --2- 172.21.15.161:0/1146039673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9101589b0 0x7ff910158e00 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.953+0000 7ff914eaf640 1 -- 172.21.15.161:0/1146039673 >> 172.21.15.161:0/1146039673 conn(0x7ff9100caa80 msgr2=0x7ff9100cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:25.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.954+0000 7ff914eaf640 1 -- 172.21.15.161:0/1146039673 shutdown_connections 2024-03-28T23:30:25.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.954+0000 7ff914eaf640 1 -- 172.21.15.161:0/1146039673 wait complete. 2024-03-28T23:30:25.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.955+0000 7ff914eaf640 1 Processor -- start 2024-03-28T23:30:25.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.955+0000 7ff914eaf640 1 -- start start 2024-03-28T23:30:25.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.955+0000 7ff914eaf640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9100c0330 0x7ff9101ee830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:25.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.955+0000 7ff914eaf640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff910152bc0 0x7ff910156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:25.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.955+0000 7ff914eaf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9101589b0 0x7ff91006b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:25.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.955+0000 7ff914eaf640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff910152fa0 con 0x7ff9101589b0 2024-03-28T23:30:25.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.955+0000 7ff914eaf640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff910159340 con 0x7ff9100c0330 2024-03-28T23:30:25.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.955+0000 7ff914eaf640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff9101edde0 con 0x7ff910152bc0 2024-03-28T23:30:25.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.955+0000 7ff90ed9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9101589b0 0x7ff91006b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:25.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.956+0000 7ff90ed9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9101589b0 0x7ff91006b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51704/0 (socket says 172.21.15.161:51704) 2024-03-28T23:30:25.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.956+0000 7ff90ed9d640 1 -- 172.21.15.161:0/2311161142 learned_addr learned my addr 172.21.15.161:0/2311161142 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:25.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.956+0000 7ff90e59c640 1 --2- 172.21.15.161:0/2311161142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9100c0330 0x7ff9101ee830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:25.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.956+0000 7ff90dd9b640 1 --2- 172.21.15.161:0/2311161142 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff910152bc0 0x7ff910156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:25.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.956+0000 7ff90ed9d640 1 -- 172.21.15.161:0/2311161142 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff910152bc0 msgr2=0x7ff910156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:25.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.956+0000 7ff90ed9d640 1 --2- 172.21.15.161:0/2311161142 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff910152bc0 0x7ff910156870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.956+0000 7ff90ed9d640 1 -- 172.21.15.161:0/2311161142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9100c0330 msgr2=0x7ff9101ee830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:25.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.956+0000 7ff90ed9d640 1 --2- 172.21.15.161:0/2311161142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9100c0330 0x7ff9101ee830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:25.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.956+0000 7ff90ed9d640 1 -- 172.21.15.161:0/2311161142 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9100abac0 con 0x7ff9101589b0 2024-03-28T23:30:25.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.956+0000 7ff90ed9d640 1 --2- 172.21.15.161:0/2311161142 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9101589b0 0x7ff91006b920 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff908009eb0 tx=0x7ff908004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:25.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.956+0000 7ff906ffd640 1 -- 172.21.15.161:0/2311161142 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff908039050 con 0x7ff9101589b0 2024-03-28T23:30:25.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.956+0000 7ff906ffd640 1 -- 172.21.15.161:0/2311161142 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff908045070 con 0x7ff9101589b0 2024-03-28T23:30:25.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.956+0000 7ff914eaf640 1 -- 172.21.15.161:0/2311161142 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9100abda0 con 0x7ff9101589b0 2024-03-28T23:30:25.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.956+0000 7ff906ffd640 1 -- 172.21.15.161:0/2311161142 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff908009450 con 0x7ff9101589b0 2024-03-28T23:30:25.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.956+0000 7ff914eaf640 1 -- 172.21.15.161:0/2311161142 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9101ee310 con 0x7ff9101589b0 2024-03-28T23:30:25.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.958+0000 7ff914eaf640 1 -- 172.21.15.161:0/2311161142 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff91004f4b0 con 0x7ff9101589b0 2024-03-28T23:30:25.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.958+0000 7ff906ffd640 1 -- 172.21.15.161:0/2311161142 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff90803e850 con 0x7ff9101589b0 2024-03-28T23:30:25.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.958+0000 7ff906ffd640 1 --2- 172.21.15.161:0/2311161142 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff8f006b500 0x7ff8f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:25.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.958+0000 7ff90e59c640 1 --2- 172.21.15.161:0/2311161142 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff8f006b500 0x7ff8f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:25.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.958+0000 7ff906ffd640 1 -- 172.21.15.161:0/2311161142 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff9080ad260 con 0x7ff9101589b0 2024-03-28T23:30:25.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.959+0000 7ff90e59c640 1 --2- 172.21.15.161:0/2311161142 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff8f006b500 0x7ff8f006d9c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff8f80011e0 tx=0x7ff8f8007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:25.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:25.963+0000 7ff906ffd640 1 -- 172.21.15.161:0/2311161142 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff90807b7e0 con 0x7ff9101589b0 2024-03-28T23:30:26.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.141+0000 7ff914eaf640 1 -- 172.21.15.161:0/2311161142 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff9100c0810 con 0x7ff9101589b0 2024-03-28T23:30:26.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.142+0000 7ff906ffd640 1 -- 172.21.15.161:0/2311161142 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7ff90807f880 con 0x7ff9101589b0 2024-03-28T23:30:26.143 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:30:26.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.144+0000 7ff914eaf640 1 -- 172.21.15.161:0/2311161142 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff8f006b500 msgr2=0x7ff8f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:26.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.144+0000 7ff914eaf640 1 --2- 172.21.15.161:0/2311161142 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff8f006b500 0x7ff8f006d9c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff8f80011e0 tx=0x7ff8f8007d30 comp rx=0 tx=0).stop 2024-03-28T23:30:26.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.144+0000 7ff914eaf640 1 -- 172.21.15.161:0/2311161142 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9101589b0 msgr2=0x7ff91006b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:26.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.144+0000 7ff914eaf640 1 --2- 172.21.15.161:0/2311161142 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9101589b0 0x7ff91006b920 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff908009eb0 tx=0x7ff908004a40 comp rx=0 tx=0).stop 2024-03-28T23:30:26.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.144+0000 7ff914eaf640 1 -- 172.21.15.161:0/2311161142 shutdown_connections 2024-03-28T23:30:26.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.144+0000 7ff914eaf640 1 --2- 172.21.15.161:0/2311161142 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff8f006b500 0x7ff8f006d9c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.144+0000 7ff914eaf640 1 --2- 172.21.15.161:0/2311161142 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9101589b0 0x7ff91006b920 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.144+0000 7ff914eaf640 1 --2- 172.21.15.161:0/2311161142 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff910152bc0 0x7ff910156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.144+0000 7ff914eaf640 1 --2- 172.21.15.161:0/2311161142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9100c0330 0x7ff9101ee830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.144+0000 7ff914eaf640 1 -- 172.21.15.161:0/2311161142 >> 172.21.15.161:0/2311161142 conn(0x7ff9100caa80 msgr2=0x7ff9100cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:26.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.144+0000 7ff914eaf640 1 -- 172.21.15.161:0/2311161142 shutdown_connections 2024-03-28T23:30:26.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.144+0000 7ff914eaf640 1 -- 172.21.15.161:0/2311161142 wait complete. 2024-03-28T23:30:26.156 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:30:26.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.261+0000 7f63d5c84640 1 Processor -- start 2024-03-28T23:30:26.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.261+0000 7f63d5c84640 1 -- start start 2024-03-28T23:30:26.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.261+0000 7f63d5c84640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63d01511e0 0x7f63d006d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:26.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.261+0000 7f63d5c84640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f63d006d610 0x7f63d01522b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:26.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.261+0000 7f63d5c84640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63d00c2fc0 con 0x7f63d00c6470 2024-03-28T23:30:26.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.261+0000 7f63d5c84640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f63d00c1290 con 0x7f63d006d610 2024-03-28T23:30:26.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.261+0000 7f63d5c84640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f63d006c8a0 con 0x7f63d01511e0 2024-03-28T23:30:26.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.261+0000 7f63cffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f63d006d610 0x7f63d01522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:26.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.262+0000 7f63cffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f63d006d610 0x7f63d01522b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51720/0 (socket says 172.21.15.161:51720) 2024-03-28T23:30:26.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.262+0000 7f63cffff640 1 -- 172.21.15.161:0/2829720197 learned_addr learned my addr 172.21.15.161:0/2829720197 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:26.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.262+0000 7f63cf7fe640 1 --2- 172.21.15.161:0/2829720197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63d01511e0 0x7f63d006d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:26.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.262+0000 7f63cffff640 1 -- 172.21.15.161:0/2829720197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63d01511e0 msgr2=0x7f63d006d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:26.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.262+0000 7f63cffff640 1 --2- 172.21.15.161:0/2829720197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63d01511e0 0x7f63d006d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.262+0000 7f63cffff640 1 -- 172.21.15.161:0/2829720197 >> v1:172.21.15.170:6789/0 conn(0x7f63d00c6470 legacy=0x7f63d0151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:26.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.262+0000 7f63cffff640 1 -- 172.21.15.161:0/2829720197 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63d006cb80 con 0x7f63d006d610 2024-03-28T23:30:26.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.262+0000 7f63ce7fc640 1 -- 172.21.15.161:0/2829720197 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1836847593 0 0) 0x7f63d00c2fc0 con 0x7f63d00c6470 2024-03-28T23:30:26.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.263+0000 7f63cffff640 1 --2- 172.21.15.161:0/2829720197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63d006d610 0x7f63d01522b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f63c800d150 tx=0x7f63c802e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df44a1d028a7ed47 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:26.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.263+0000 7f63ce7fc640 1 -- 172.21.15.161:0/2829720197 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c802e9e0 con 0x7f63d006d610 2024-03-28T23:30:26.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.263+0000 7f63ce7fc640 1 -- 172.21.15.161:0/2829720197 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63c802eb80 con 0x7f63d006d610 2024-03-28T23:30:26.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.263+0000 7f63ce7fc640 1 -- 172.21.15.161:0/2829720197 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c8009600 con 0x7f63d006d610 2024-03-28T23:30:26.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.264+0000 7f63d5c84640 1 -- 172.21.15.161:0/2829720197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63d006d610 msgr2=0x7f63d01522b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:26.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.264+0000 7f63d5c84640 1 --2- 172.21.15.161:0/2829720197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63d006d610 0x7f63d01522b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f63c800d150 tx=0x7f63c802e040 comp rx=0 tx=0).stop 2024-03-28T23:30:26.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.264+0000 7f63d5c84640 1 -- 172.21.15.161:0/2829720197 shutdown_connections 2024-03-28T23:30:26.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.264+0000 7f63d5c84640 1 --2- 172.21.15.161:0/2829720197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63d006d610 0x7f63d01522b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.264+0000 7f63d5c84640 1 --2- 172.21.15.161:0/2829720197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63d01511e0 0x7f63d006d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.264+0000 7f63d5c84640 1 -- 172.21.15.161:0/2829720197 >> 172.21.15.161:0/2829720197 conn(0x7f63d00c3390 msgr2=0x7f63d00c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:26.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.264+0000 7f63d5c84640 1 -- 172.21.15.161:0/2829720197 shutdown_connections 2024-03-28T23:30:26.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.264+0000 7f63d5c84640 1 -- 172.21.15.161:0/2829720197 wait complete. 2024-03-28T23:30:26.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.265+0000 7f63d5c84640 1 Processor -- start 2024-03-28T23:30:26.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.265+0000 7f63d5c84640 1 -- start start 2024-03-28T23:30:26.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.266+0000 7f63d5c84640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63d006d610 0x7f63d01e7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:26.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.266+0000 7f63d5c84640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63d01511e0 0x7f63d01ea550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:26.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.266+0000 7f63d5c84640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63d01e9600 0x7f63d0072de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:26.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.266+0000 7f63d5c84640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f63d006c2e0 con 0x7f63d006d610 2024-03-28T23:30:26.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.266+0000 7f63d5c84640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f63d006c160 con 0x7f63d01511e0 2024-03-28T23:30:26.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.266+0000 7f63d5c84640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f63d00c5f20 con 0x7f63d01e9600 2024-03-28T23:30:26.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.266+0000 7f63cf7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63d006d610 0x7f63d01e7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:26.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.266+0000 7f63cf7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63d006d610 0x7f63d01e7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51730/0 (socket says 172.21.15.161:51730) 2024-03-28T23:30:26.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.266+0000 7f63cf7fe640 1 -- 172.21.15.161:0/3575060460 learned_addr learned my addr 172.21.15.161:0/3575060460 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:26.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.266+0000 7f63cffff640 1 --2- 172.21.15.161:0/3575060460 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63d01e9600 0x7f63d0072de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:26.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.266+0000 7f63ceffd640 1 --2- 172.21.15.161:0/3575060460 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63d01511e0 0x7f63d01ea550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:26.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.266+0000 7f63cf7fe640 1 -- 172.21.15.161:0/3575060460 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63d01e9600 msgr2=0x7f63d0072de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:26.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.266+0000 7f63cf7fe640 1 --2- 172.21.15.161:0/3575060460 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63d01e9600 0x7f63d0072de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.266+0000 7f63cf7fe640 1 -- 172.21.15.161:0/3575060460 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63d01511e0 msgr2=0x7f63d01ea550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:26.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.266+0000 7f63cf7fe640 1 --2- 172.21.15.161:0/3575060460 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63d01511e0 0x7f63d01ea550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.266+0000 7f63cf7fe640 1 -- 172.21.15.161:0/3575060460 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63d0072650 con 0x7f63d006d610 2024-03-28T23:30:26.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.266+0000 7f63cf7fe640 1 --2- 172.21.15.161:0/3575060460 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63d006d610 0x7f63d01e7a40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f63c400e8a0 tx=0x7f63c400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:26.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.266+0000 7f63b7fff640 1 -- 172.21.15.161:0/3575060460 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c4017070 con 0x7f63d006d610 2024-03-28T23:30:26.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.266+0000 7f63b7fff640 1 -- 172.21.15.161:0/3575060460 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63c400c940 con 0x7f63d006d610 2024-03-28T23:30:26.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.266+0000 7f63d5c84640 1 -- 172.21.15.161:0/3575060460 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63d01eaa90 con 0x7f63d006d610 2024-03-28T23:30:26.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.266+0000 7f63b7fff640 1 -- 172.21.15.161:0/3575060460 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c400cd30 con 0x7f63d006d610 2024-03-28T23:30:26.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.266+0000 7f63d5c84640 1 -- 172.21.15.161:0/3575060460 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63d0070c00 con 0x7f63d006d610 2024-03-28T23:30:26.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.267+0000 7f63d5c84640 1 -- 172.21.15.161:0/3575060460 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63d004f4b0 con 0x7f63d006d610 2024-03-28T23:30:26.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.268+0000 7f63b7fff640 1 -- 172.21.15.161:0/3575060460 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f63c401a070 con 0x7f63d006d610 2024-03-28T23:30:26.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.268+0000 7f63b7fff640 1 --2- 172.21.15.161:0/3575060460 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f63bc06b500 0x7f63bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:26.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.268+0000 7f63ceffd640 1 --2- 172.21.15.161:0/3575060460 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f63bc06b500 0x7f63bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:26.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.268+0000 7f63b7fff640 1 -- 172.21.15.161:0/3575060460 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f63c408a250 con 0x7f63d006d610 2024-03-28T23:30:26.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.269+0000 7f63ceffd640 1 --2- 172.21.15.161:0/3575060460 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f63bc06b500 0x7f63bc06d9c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f63d00c0b90 tx=0x7f63c0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:26.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.273+0000 7f63b7fff640 1 -- 172.21.15.161:0/3575060460 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f63c40587d0 con 0x7f63d006d610 2024-03-28T23:30:26.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.451+0000 7f63d5c84640 1 -- 172.21.15.161:0/3575060460 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f63d014c470 con 0x7f63d006d610 2024-03-28T23:30:26.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.451+0000 7f63b7fff640 1 -- 172.21.15.161:0/3575060460 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f63c405c870 con 0x7f63d006d610 2024-03-28T23:30:26.452 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:30:26.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.453+0000 7f63d5c84640 1 -- 172.21.15.161:0/3575060460 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f63bc06b500 msgr2=0x7f63bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:26.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.453+0000 7f63d5c84640 1 --2- 172.21.15.161:0/3575060460 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f63bc06b500 0x7f63bc06d9c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f63d00c0b90 tx=0x7f63c0009040 comp rx=0 tx=0).stop 2024-03-28T23:30:26.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.453+0000 7f63d5c84640 1 -- 172.21.15.161:0/3575060460 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63d006d610 msgr2=0x7f63d01e7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:26.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.453+0000 7f63d5c84640 1 --2- 172.21.15.161:0/3575060460 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63d006d610 0x7f63d01e7a40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f63c400e8a0 tx=0x7f63c400ed60 comp rx=0 tx=0).stop 2024-03-28T23:30:26.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.453+0000 7f63d5c84640 1 -- 172.21.15.161:0/3575060460 shutdown_connections 2024-03-28T23:30:26.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.453+0000 7f63d5c84640 1 --2- 172.21.15.161:0/3575060460 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f63bc06b500 0x7f63bc06d9c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.453+0000 7f63d5c84640 1 --2- 172.21.15.161:0/3575060460 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63d01e9600 0x7f63d0072de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.453+0000 7f63d5c84640 1 --2- 172.21.15.161:0/3575060460 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63d01511e0 0x7f63d01ea550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.453+0000 7f63d5c84640 1 --2- 172.21.15.161:0/3575060460 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63d006d610 0x7f63d01e7a40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.453+0000 7f63d5c84640 1 -- 172.21.15.161:0/3575060460 >> 172.21.15.161:0/3575060460 conn(0x7f63d00c3390 msgr2=0x7f63d0074020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:26.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.453+0000 7f63d5c84640 1 -- 172.21.15.161:0/3575060460 shutdown_connections 2024-03-28T23:30:26.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.453+0000 7f63d5c84640 1 -- 172.21.15.161:0/3575060460 wait complete. 2024-03-28T23:30:26.465 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T23:30:26.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.569+0000 7f3637b6b640 1 Processor -- start 2024-03-28T23:30:26.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.570+0000 7f3637b6b640 1 -- start start 2024-03-28T23:30:26.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.570+0000 7f3637b6b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3630152bc0 0x7f36300c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:26.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.570+0000 7f3637b6b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36300c8a80 0x7f36300c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:26.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.570+0000 7f3637b6b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f36300c9420 0x7f36301589b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:26.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.570+0000 7f3637b6b640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f36300c6f60 con 0x7f36300c9420 2024-03-28T23:30:26.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.570+0000 7f3637b6b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f36300c6120 con 0x7f3630152bc0 2024-03-28T23:30:26.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.570+0000 7f3637b6b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f36300c6400 con 0x7f36300c8a80 2024-03-28T23:30:26.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.571+0000 7f3636108640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f36300c9420 0x7f36301589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:26.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.571+0000 7f3636108640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f36300c9420 0x7f36301589b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51732/0 (socket says 172.21.15.161:51732) 2024-03-28T23:30:26.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.571+0000 7f3636108640 1 -- 172.21.15.161:0/3678284324 learned_addr learned my addr 172.21.15.161:0/3678284324 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:26.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.571+0000 7f3635907640 1 --2- 172.21.15.161:0/3678284324 >> v2:172.21.15.170:3300/0 conn(0x7f3630152bc0 0x7f36300c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:26.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.571+0000 7f3635106640 1 --2- 172.21.15.161:0/3678284324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36300c8a80 0x7f36300c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:26.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.571+0000 7f3636108640 1 -- 172.21.15.161:0/3678284324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36300c8a80 msgr2=0x7f36300c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:26.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.571+0000 7f3636108640 1 --2- 172.21.15.161:0/3678284324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36300c8a80 0x7f36300c8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.571+0000 7f3636108640 1 -- 172.21.15.161:0/3678284324 >> v2:172.21.15.170:3300/0 conn(0x7f3630152bc0 msgr2=0x7f36300c0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:26.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.571+0000 7f3636108640 1 --2- 172.21.15.161:0/3678284324 >> v2:172.21.15.170:3300/0 conn(0x7f3630152bc0 0x7f36300c0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.571+0000 7f3636108640 1 -- 172.21.15.161:0/3678284324 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36300c7240 con 0x7f36300c9420 2024-03-28T23:30:26.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.572+0000 7f3636108640 1 --2- 172.21.15.161:0/3678284324 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36300c9420 0x7f36301589b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f36300aa410 tx=0x7f362802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c25e7aae881a19ce server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:26.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.572+0000 7f3634905640 1 -- 172.21.15.161:0/3678284324 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3628035020 con 0x7f36300c9420 2024-03-28T23:30:26.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.572+0000 7f3634905640 1 -- 172.21.15.161:0/3678284324 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3628031870 con 0x7f36300c9420 2024-03-28T23:30:26.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.572+0000 7f3634905640 1 -- 172.21.15.161:0/3678284324 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3628031c40 con 0x7f36300c9420 2024-03-28T23:30:26.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.572+0000 7f3637b6b640 1 -- 172.21.15.161:0/3678284324 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36300c9420 msgr2=0x7f36301589b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:26.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.572+0000 7f3637b6b640 1 --2- 172.21.15.161:0/3678284324 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36300c9420 0x7f36301589b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f36300aa410 tx=0x7f362802b910 comp rx=0 tx=0).stop 2024-03-28T23:30:26.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.572+0000 7f3637b6b640 1 -- 172.21.15.161:0/3678284324 shutdown_connections 2024-03-28T23:30:26.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.572+0000 7f3637b6b640 1 --2- 172.21.15.161:0/3678284324 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36300c9420 0x7f36301589b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.572+0000 7f3637b6b640 1 --2- 172.21.15.161:0/3678284324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36300c8a80 0x7f36300c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.572+0000 7f3637b6b640 1 --2- 172.21.15.161:0/3678284324 >> v2:172.21.15.170:3300/0 conn(0x7f3630152bc0 0x7f36300c0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.572+0000 7f3637b6b640 1 -- 172.21.15.161:0/3678284324 >> 172.21.15.161:0/3678284324 conn(0x7f36300caa80 msgr2=0x7f36300cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:26.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.573+0000 7f3637b6b640 1 -- 172.21.15.161:0/3678284324 shutdown_connections 2024-03-28T23:30:26.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.573+0000 7f3637b6b640 1 -- 172.21.15.161:0/3678284324 wait complete. 2024-03-28T23:30:26.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.574+0000 7f3637b6b640 1 Processor -- start 2024-03-28T23:30:26.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.574+0000 7f3637b6b640 1 -- start start 2024-03-28T23:30:26.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.574+0000 7f3637b6b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36300c8a80 0x7f36300ae110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:26.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.574+0000 7f3637b6b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36300c9420 0x7f36301ebe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:26.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.574+0000 7f3637b6b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3630152bc0 0x7f36301ee1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:26.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.574+0000 7f3637b6b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3630152fa0 con 0x7f36300c9420 2024-03-28T23:30:26.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.574+0000 7f3637b6b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3630158ef0 con 0x7f3630152bc0 2024-03-28T23:30:26.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.574+0000 7f3637b6b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3630157990 con 0x7f36300c8a80 2024-03-28T23:30:26.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.574+0000 7f3635106640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36300c9420 0x7f36301ebe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:26.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.574+0000 7f3635106640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36300c9420 0x7f36301ebe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51738/0 (socket says 172.21.15.161:51738) 2024-03-28T23:30:26.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.574+0000 7f3635106640 1 -- 172.21.15.161:0/4261891142 learned_addr learned my addr 172.21.15.161:0/4261891142 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:26.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.574+0000 7f3635907640 1 --2- 172.21.15.161:0/4261891142 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36300c8a80 0x7f36300ae110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:26.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.574+0000 7f3636108640 1 --2- 172.21.15.161:0/4261891142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3630152bc0 0x7f36301ee1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:26.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.575+0000 7f3635106640 1 -- 172.21.15.161:0/4261891142 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36300c8a80 msgr2=0x7f36300ae110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:26.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.575+0000 7f3635106640 1 --2- 172.21.15.161:0/4261891142 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36300c8a80 0x7f36300ae110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.575+0000 7f3635106640 1 -- 172.21.15.161:0/4261891142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3630152bc0 msgr2=0x7f36301ee1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:26.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.575+0000 7f3635106640 1 --2- 172.21.15.161:0/4261891142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3630152bc0 0x7f36301ee1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.575+0000 7f3635106640 1 -- 172.21.15.161:0/4261891142 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36301536f0 con 0x7f36300c9420 2024-03-28T23:30:26.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.575+0000 7f3635907640 1 --2- 172.21.15.161:0/4261891142 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36300c8a80 0x7f36300ae110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:26.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.575+0000 7f3636108640 1 --2- 172.21.15.161:0/4261891142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3630152bc0 0x7f36301ee1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:26.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.575+0000 7f3635106640 1 --2- 172.21.15.161:0/4261891142 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36300c9420 0x7f36301ebe60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f362000e890 tx=0x7f362000ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:26.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.575+0000 7f36267fc640 1 -- 172.21.15.161:0/4261891142 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3620017070 con 0x7f36300c9420 2024-03-28T23:30:26.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.575+0000 7f36267fc640 1 -- 172.21.15.161:0/4261891142 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f362000c940 con 0x7f36300c9420 2024-03-28T23:30:26.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.575+0000 7f3637b6b640 1 -- 172.21.15.161:0/4261891142 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3630157340 con 0x7f36300c9420 2024-03-28T23:30:26.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.575+0000 7f36267fc640 1 -- 172.21.15.161:0/4261891142 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362000cd10 con 0x7f36300c9420 2024-03-28T23:30:26.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.575+0000 7f3637b6b640 1 -- 172.21.15.161:0/4261891142 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36301533f0 con 0x7f36300c9420 2024-03-28T23:30:26.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.576+0000 7f3637b6b640 1 -- 172.21.15.161:0/4261891142 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3630157170 con 0x7f36300c9420 2024-03-28T23:30:26.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.577+0000 7f36267fc640 1 -- 172.21.15.161:0/4261891142 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f362001a070 con 0x7f36300c9420 2024-03-28T23:30:26.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.577+0000 7f36267fc640 1 --2- 172.21.15.161:0/4261891142 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f361406b4d0 0x7f361406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:26.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.577+0000 7f36267fc640 1 -- 172.21.15.161:0/4261891142 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f362008a760 con 0x7f36300c9420 2024-03-28T23:30:26.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.577+0000 7f3635907640 1 --2- 172.21.15.161:0/4261891142 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f361406b4d0 0x7f361406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:26.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.578+0000 7f3635907640 1 --2- 172.21.15.161:0/4261891142 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f361406b4d0 0x7f361406d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f362c009fd0 tx=0x7f362c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:26.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.583+0000 7f36267fc640 1 -- 172.21.15.161:0/4261891142 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3620058ce0 con 0x7f36300c9420 2024-03-28T23:30:26.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.761+0000 7f3637b6b640 1 -- 172.21.15.161:0/4261891142 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f363004f4b0 con 0x7f36300c9420 2024-03-28T23:30:26.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.761+0000 7f36267fc640 1 -- 172.21.15.161:0/4261891142 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f362005cd80 con 0x7f36300c9420 2024-03-28T23:30:26.762 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:30:26.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.764+0000 7f3637b6b640 1 -- 172.21.15.161:0/4261891142 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f361406b4d0 msgr2=0x7f361406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:26.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.764+0000 7f3637b6b640 1 --2- 172.21.15.161:0/4261891142 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f361406b4d0 0x7f361406d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f362c009fd0 tx=0x7f362c00b040 comp rx=0 tx=0).stop 2024-03-28T23:30:26.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.764+0000 7f3637b6b640 1 -- 172.21.15.161:0/4261891142 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36300c9420 msgr2=0x7f36301ebe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:26.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.764+0000 7f3637b6b640 1 --2- 172.21.15.161:0/4261891142 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36300c9420 0x7f36301ebe60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f362000e890 tx=0x7f362000ed50 comp rx=0 tx=0).stop 2024-03-28T23:30:26.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.764+0000 7f3637b6b640 1 -- 172.21.15.161:0/4261891142 shutdown_connections 2024-03-28T23:30:26.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.764+0000 7f3637b6b640 1 --2- 172.21.15.161:0/4261891142 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f361406b4d0 0x7f361406d990 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.764+0000 7f3637b6b640 1 --2- 172.21.15.161:0/4261891142 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36300c9420 0x7f36301ebe60 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.764+0000 7f3637b6b640 1 --2- 172.21.15.161:0/4261891142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3630152bc0 0x7f36301ee1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.764+0000 7f3637b6b640 1 --2- 172.21.15.161:0/4261891142 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36300c8a80 0x7f36300ae110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.764+0000 7f3637b6b640 1 -- 172.21.15.161:0/4261891142 >> 172.21.15.161:0/4261891142 conn(0x7f36300caa80 msgr2=0x7f363014e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:26.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.764+0000 7f3637b6b640 1 -- 172.21.15.161:0/4261891142 shutdown_connections 2024-03-28T23:30:26.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.764+0000 7f3637b6b640 1 -- 172.21.15.161:0/4261891142 wait complete. 2024-03-28T23:30:26.775 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:30:26.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.877+0000 7f390b8f4640 1 Processor -- start 2024-03-28T23:30:26.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.877+0000 7f390b8f4640 1 -- start start 2024-03-28T23:30:26.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.878+0000 7f390b8f4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3904152890 0x7f39040c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:26.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.878+0000 7f390b8f4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f39040c8ed0 0x7f39040c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:26.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.878+0000 7f390b8f4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3904158a30 con 0x7f39040c5950 2024-03-28T23:30:26.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.878+0000 7f390b8f4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f39040c8480 con 0x7f3904152890 2024-03-28T23:30:26.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.878+0000 7f390b8f4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f3904159640 con 0x7f39040c8ed0 2024-03-28T23:30:26.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.878+0000 7f3909690640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3904152890 0x7f39040c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:26.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.878+0000 7f3909e91640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f39040c5950 0x7f39041553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38430/0 (socket says 172.21.15.161:38430) 2024-03-28T23:30:26.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.878+0000 7f3909e91640 1 -- 172.21.15.161:0/3675470622 learned_addr learned my addr 172.21.15.161:0/3675470622 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:26.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.879+0000 7f3908e8f640 1 --2- 172.21.15.161:0/3675470622 >> v2:172.21.15.170:3300/0 conn(0x7f39040c8ed0 0x7f39040c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:26.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.879+0000 7f3909690640 1 -- 172.21.15.161:0/3675470622 >> v2:172.21.15.170:3300/0 conn(0x7f39040c8ed0 msgr2=0x7f39040c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:26.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.879+0000 7f3909690640 1 --2- 172.21.15.161:0/3675470622 >> v2:172.21.15.170:3300/0 conn(0x7f39040c8ed0 0x7f39040c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.879+0000 7f3909690640 1 -- 172.21.15.161:0/3675470622 >> v1:172.21.15.161:6789/0 conn(0x7f39040c5950 legacy=0x7f39041553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:26.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.879+0000 7f3909690640 1 -- 172.21.15.161:0/3675470622 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3904158d10 con 0x7f3904152890 2024-03-28T23:30:26.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.879+0000 7f38fbfff640 1 -- 172.21.15.161:0/3675470622 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 823182581 0 0) 0x7f3904158a30 con 0x7f39040c5950 2024-03-28T23:30:26.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.879+0000 7f3908e8f640 1 --2- 172.21.15.161:0/3675470622 >> v2:172.21.15.170:3300/0 conn(0x7f39040c8ed0 0x7f39040c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:26.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.880+0000 7f3909690640 1 --2- 172.21.15.161:0/3675470622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3904152890 0x7f39040c8990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f39040ad6e0 tx=0x7f390002b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=560d75fcc345f100 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:26.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.880+0000 7f38fbfff640 1 -- 172.21.15.161:0/3675470622 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3900038070 con 0x7f3904152890 2024-03-28T23:30:26.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.880+0000 7f38fbfff640 1 -- 172.21.15.161:0/3675470622 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3900031710 con 0x7f3904152890 2024-03-28T23:30:26.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.880+0000 7f38fbfff640 1 -- 172.21.15.161:0/3675470622 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3900031ae0 con 0x7f3904152890 2024-03-28T23:30:26.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.880+0000 7f390b8f4640 1 -- 172.21.15.161:0/3675470622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3904152890 msgr2=0x7f39040c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:26.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.880+0000 7f390b8f4640 1 --2- 172.21.15.161:0/3675470622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3904152890 0x7f39040c8990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f39040ad6e0 tx=0x7f390002b7b0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.880+0000 7f390b8f4640 1 -- 172.21.15.161:0/3675470622 shutdown_connections 2024-03-28T23:30:26.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.880+0000 7f390b8f4640 1 --2- 172.21.15.161:0/3675470622 >> v2:172.21.15.170:3300/0 conn(0x7f39040c8ed0 0x7f39040c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.880+0000 7f390b8f4640 1 --2- 172.21.15.161:0/3675470622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3904152890 0x7f39040c8990 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.880+0000 7f390b8f4640 1 -- 172.21.15.161:0/3675470622 >> 172.21.15.161:0/3675470622 conn(0x7f39040ac620 msgr2=0x7f39040aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:26.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.881+0000 7f390b8f4640 1 -- 172.21.15.161:0/3675470622 shutdown_connections 2024-03-28T23:30:26.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.881+0000 7f390b8f4640 1 -- 172.21.15.161:0/3675470622 wait complete. 2024-03-28T23:30:26.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.882+0000 7f390b8f4640 1 Processor -- start 2024-03-28T23:30:26.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.882+0000 7f390b8f4640 1 -- start start 2024-03-28T23:30:26.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.882+0000 7f390b8f4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39040c5950 0x7f39041480a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:26.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.882+0000 7f390b8f4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39040c8ed0 0x7f3904149f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:26.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.882+0000 7f390b8f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3904152890 0x7f39040c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:26.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.882+0000 7f390b8f4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3904152c70 con 0x7f39040c8ed0 2024-03-28T23:30:26.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.882+0000 7f390b8f4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f39041592d0 con 0x7f39040c5950 2024-03-28T23:30:26.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.882+0000 7f390b8f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3904152df0 con 0x7f3904152890 2024-03-28T23:30:26.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.882+0000 7f3908e8f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39040c8ed0 0x7f3904149f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:26.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.882+0000 7f3908e8f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39040c8ed0 0x7f3904149f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51766/0 (socket says 172.21.15.161:51766) 2024-03-28T23:30:26.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.882+0000 7f3908e8f640 1 -- 172.21.15.161:0/4292038042 learned_addr learned my addr 172.21.15.161:0/4292038042 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:26.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.883+0000 7f3909690640 1 --2- 172.21.15.161:0/4292038042 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39040c5950 0x7f39041480a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:26.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.883+0000 7f3909e91640 1 --2- 172.21.15.161:0/4292038042 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3904152890 0x7f39040c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:26.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.883+0000 7f3908e8f640 1 -- 172.21.15.161:0/4292038042 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3904152890 msgr2=0x7f39040c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:26.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.883+0000 7f3908e8f640 1 --2- 172.21.15.161:0/4292038042 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3904152890 0x7f39040c0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.883+0000 7f3908e8f640 1 -- 172.21.15.161:0/4292038042 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39040c5950 msgr2=0x7f39041480a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:26.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.883+0000 7f3908e8f640 1 --2- 172.21.15.161:0/4292038042 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39040c5950 0x7f39041480a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:26.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.883+0000 7f3908e8f640 1 -- 172.21.15.161:0/4292038042 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3904155c70 con 0x7f39040c8ed0 2024-03-28T23:30:26.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.883+0000 7f3908e8f640 1 --2- 172.21.15.161:0/4292038042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39040c8ed0 0x7f3904149f80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f38f400c820 tx=0x7f38f400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:26.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.883+0000 7f38f9ffb640 1 -- 172.21.15.161:0/4292038042 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f4005b90 con 0x7f39040c8ed0 2024-03-28T23:30:26.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.883+0000 7f38f9ffb640 1 -- 172.21.15.161:0/4292038042 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38f4005d30 con 0x7f39040c8ed0 2024-03-28T23:30:26.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.883+0000 7f390b8f4640 1 -- 172.21.15.161:0/4292038042 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f390414ad70 con 0x7f39040c8ed0 2024-03-28T23:30:26.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.883+0000 7f38f9ffb640 1 -- 172.21.15.161:0/4292038042 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f4007a70 con 0x7f39040c8ed0 2024-03-28T23:30:26.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.883+0000 7f390b8f4640 1 -- 172.21.15.161:0/4292038042 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39041561f0 con 0x7f39040c8ed0 2024-03-28T23:30:26.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.884+0000 7f390b8f4640 1 -- 172.21.15.161:0/4292038042 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f390404f4b0 con 0x7f39040c8ed0 2024-03-28T23:30:26.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.884+0000 7f38f9ffb640 1 -- 172.21.15.161:0/4292038042 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f38f4007c10 con 0x7f39040c8ed0 2024-03-28T23:30:26.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.885+0000 7f38f9ffb640 1 --2- 172.21.15.161:0/4292038042 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f38e806b500 0x7f38e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:26.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.885+0000 7f3909690640 1 --2- 172.21.15.161:0/4292038042 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f38e806b500 0x7f38e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:26.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.885+0000 7f38f9ffb640 1 -- 172.21.15.161:0/4292038042 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f38f408a530 con 0x7f39040c8ed0 2024-03-28T23:30:26.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.885+0000 7f3909690640 1 --2- 172.21.15.161:0/4292038042 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f38e806b500 0x7f38e806d9c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3900035000 tx=0x7f3900008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:26.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:26.889+0000 7f38f9ffb640 1 -- 172.21.15.161:0/4292038042 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f38f4015020 con 0x7f39040c8ed0 2024-03-28T23:30:27.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.067+0000 7f390b8f4640 1 -- 172.21.15.161:0/4292038042 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3904156570 con 0x7f39040c8ed0 2024-03-28T23:30:27.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.068+0000 7f38f9ffb640 1 -- 172.21.15.161:0/4292038042 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7f38f4059100 con 0x7f39040c8ed0 2024-03-28T23:30:27.069 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:27.069 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":24,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:29:50.946843+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:30:27.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.070+0000 7f390b8f4640 1 -- 172.21.15.161:0/4292038042 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f38e806b500 msgr2=0x7f38e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:27.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.070+0000 7f390b8f4640 1 --2- 172.21.15.161:0/4292038042 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f38e806b500 0x7f38e806d9c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3900035000 tx=0x7f3900008030 comp rx=0 tx=0).stop 2024-03-28T23:30:27.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.070+0000 7f390b8f4640 1 -- 172.21.15.161:0/4292038042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39040c8ed0 msgr2=0x7f3904149f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:27.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.070+0000 7f390b8f4640 1 --2- 172.21.15.161:0/4292038042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39040c8ed0 0x7f3904149f80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f38f400c820 tx=0x7f38f400cce0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.070+0000 7f390b8f4640 1 -- 172.21.15.161:0/4292038042 shutdown_connections 2024-03-28T23:30:27.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.070+0000 7f390b8f4640 1 --2- 172.21.15.161:0/4292038042 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f38e806b500 0x7f38e806d9c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.070+0000 7f390b8f4640 1 --2- 172.21.15.161:0/4292038042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39040c8ed0 0x7f3904149f80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.070+0000 7f390b8f4640 1 --2- 172.21.15.161:0/4292038042 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3904152890 0x7f39040c0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.070+0000 7f390b8f4640 1 --2- 172.21.15.161:0/4292038042 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39040c5950 0x7f39041480a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.070+0000 7f390b8f4640 1 -- 172.21.15.161:0/4292038042 >> 172.21.15.161:0/4292038042 conn(0x7f39040ac620 msgr2=0x7f39040c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:27.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.070+0000 7f390b8f4640 1 -- 172.21.15.161:0/4292038042 shutdown_connections 2024-03-28T23:30:27.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.070+0000 7f390b8f4640 1 -- 172.21.15.161:0/4292038042 wait complete. 2024-03-28T23:30:27.082 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:29:44.021375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:29:44.872102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T23:29:48.591063+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T23:30:27.083 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:30:27.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.188+0000 7eff685b6640 1 Processor -- start 2024-03-28T23:30:27.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.188+0000 7eff685b6640 1 -- start start 2024-03-28T23:30:27.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.189+0000 7eff685b6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff601529d0 0x7eff6006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:27.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.189+0000 7eff685b6640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7eff600c6750 0x7eff600c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:27.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.189+0000 7eff685b6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff60152db0 con 0x7eff60155c30 2024-03-28T23:30:27.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.189+0000 7eff685b6640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7eff600c7210 con 0x7eff600c6750 2024-03-28T23:30:27.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.189+0000 7eff685b6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7eff600c6c50 con 0x7eff601529d0 2024-03-28T23:30:27.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.189+0000 7eff66b53640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7eff60155c30 0x7eff600c54f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38444/0 (socket says 172.21.15.161:38444) 2024-03-28T23:30:27.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.189+0000 7eff66b53640 1 -- 172.21.15.161:0/798428818 learned_addr learned my addr 172.21.15.161:0/798428818 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:27.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.189+0000 7eff65350640 1 -- 172.21.15.161:0/798428818 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 582136264 0 0) 0x7eff60152db0 con 0x7eff60155c30 2024-03-28T23:30:27.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.189+0000 7eff66352640 1 --2- 172.21.15.161:0/798428818 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff601529d0 0x7eff6006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:27.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.189+0000 7eff65350640 1 -- 172.21.15.161:0/798428818 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff48003520 con 0x7eff60155c30 2024-03-28T23:30:27.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.189+0000 7eff65b51640 1 --2- 172.21.15.161:0/798428818 >> v2:172.21.15.170:3300/0 conn(0x7eff600c6750 0x7eff600c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:27.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.190+0000 7eff65350640 1 -- 172.21.15.161:0/798428818 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3415974811 0 0) 0x7eff48003520 con 0x7eff60155c30 2024-03-28T23:30:27.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.190+0000 7eff65350640 1 -- 172.21.15.161:0/798428818 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff601529d0 msgr2=0x7eff6006bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:27.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.190+0000 7eff65350640 1 --2- 172.21.15.161:0/798428818 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff601529d0 0x7eff6006bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.190+0000 7eff65350640 1 -- 172.21.15.161:0/798428818 >> v2:172.21.15.170:3300/0 conn(0x7eff600c6750 msgr2=0x7eff600c88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:27.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.190+0000 7eff65350640 1 --2- 172.21.15.161:0/798428818 >> v2:172.21.15.170:3300/0 conn(0x7eff600c6750 0x7eff600c88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.190+0000 7eff65350640 1 -- 172.21.15.161:0/798428818 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff600c6f30 con 0x7eff60155c30 2024-03-28T23:30:27.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.190+0000 7eff65350640 1 -- 172.21.15.161:0/798428818 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7eff50001670 con 0x7eff60155c30 2024-03-28T23:30:27.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.190+0000 7eff65350640 1 -- 172.21.15.161:0/798428818 >> v1:172.21.15.161:6789/0 conn(0x7eff60155c30 legacy=0x7eff600c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:27.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.190+0000 7eff65350640 1 --2- 172.21.15.161:0/798428818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff480039b0 0x7eff48003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:27.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.190+0000 7eff65350640 1 --2- 172.21.15.161:0/798428818 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff601529d0 0x7eff48004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:27.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.190+0000 7eff65350640 1 --2- 172.21.15.161:0/798428818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff48004d40 0x7eff480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:27.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.190+0000 7eff65350640 1 -- 172.21.15.161:0/798428818 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7eff600c7210 con 0x7eff480039b0 2024-03-28T23:30:27.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.190+0000 7eff65350640 1 -- 172.21.15.161:0/798428818 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7eff600c6c50 con 0x7eff48004d40 2024-03-28T23:30:27.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.190+0000 7eff65350640 1 -- 172.21.15.161:0/798428818 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7eff4800b9c0 con 0x7eff601529d0 2024-03-28T23:30:27.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.190+0000 7eff66352640 1 --2- 172.21.15.161:0/798428818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff480039b0 0x7eff48003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:27.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.190+0000 7eff66b53640 1 --2- 172.21.15.161:0/798428818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff48004d40 0x7eff480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:27.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.190+0000 7eff65b51640 1 --2- 172.21.15.161:0/798428818 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff601529d0 0x7eff48004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:27.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.190+0000 7eff66352640 1 -- 172.21.15.161:0/798428818 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff601529d0 msgr2=0x7eff48004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:27.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.190+0000 7eff66352640 1 --2- 172.21.15.161:0/798428818 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff601529d0 0x7eff48004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.190+0000 7eff66352640 1 -- 172.21.15.161:0/798428818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff48004d40 msgr2=0x7eff480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:27.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.190+0000 7eff66352640 1 --2- 172.21.15.161:0/798428818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff48004d40 0x7eff480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.190+0000 7eff66352640 1 -- 172.21.15.161:0/798428818 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff48003520 con 0x7eff480039b0 2024-03-28T23:30:27.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.190+0000 7eff65b51640 1 --2- 172.21.15.161:0/798428818 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff601529d0 0x7eff48004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:27.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.191+0000 7eff66352640 1 --2- 172.21.15.161:0/798428818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff480039b0 0x7eff48003dd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7eff58009fc0 tx=0x7eff48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce1fdf54e482717e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:27.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.191+0000 7eff65350640 1 -- 172.21.15.161:0/798428818 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff58039070 con 0x7eff480039b0 2024-03-28T23:30:27.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.191+0000 7eff65350640 1 -- 172.21.15.161:0/798428818 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff5802cc20 con 0x7eff480039b0 2024-03-28T23:30:27.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.191+0000 7eff65350640 1 -- 172.21.15.161:0/798428818 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff58009c10 con 0x7eff480039b0 2024-03-28T23:30:27.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.191+0000 7eff685b6640 1 -- 172.21.15.161:0/798428818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff480039b0 msgr2=0x7eff48003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:27.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.191+0000 7eff685b6640 1 --2- 172.21.15.161:0/798428818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff480039b0 0x7eff48003dd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7eff58009fc0 tx=0x7eff48005830 comp rx=0 tx=0).stop 2024-03-28T23:30:27.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.191+0000 7eff66b53640 1 -- 172.21.15.161:0/798428818 reap_dead start 2024-03-28T23:30:27.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.192+0000 7eff685b6640 1 -- 172.21.15.161:0/798428818 shutdown_connections 2024-03-28T23:30:27.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.192+0000 7eff685b6640 1 -- 172.21.15.161:0/798428818 >> 172.21.15.161:0/798428818 conn(0x7eff600c7430 msgr2=0x7eff600c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:27.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.192+0000 7eff685b6640 1 -- 172.21.15.161:0/798428818 shutdown_connections 2024-03-28T23:30:27.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.192+0000 7eff685b6640 1 -- 172.21.15.161:0/798428818 wait complete. 2024-03-28T23:30:27.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.193+0000 7eff685b6640 1 Processor -- start 2024-03-28T23:30:27.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.193+0000 7eff685b6640 1 -- start start 2024-03-28T23:30:27.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.193+0000 7eff685b6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff480039b0 0x7eff600b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:27.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.193+0000 7eff685b6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff600ae9a0 0x7eff60154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:27.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.193+0000 7eff685b6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff601eb1d0 0x7eff601eb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.193+0000 7eff685b6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7eff600aa160 con 0x7eff601eb1d0 2024-03-28T23:30:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.193+0000 7eff685b6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7eff60155710 con 0x7eff600ae9a0 2024-03-28T23:30:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.193+0000 7eff685b6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7eff600c5c00 con 0x7eff480039b0 2024-03-28T23:30:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.193+0000 7eff66b53640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff601eb1d0 0x7eff601eb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.193+0000 7eff66b53640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff601eb1d0 0x7eff601eb650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51794/0 (socket says 172.21.15.161:51794) 2024-03-28T23:30:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.193+0000 7eff66b53640 1 -- 172.21.15.161:0/3642594051 learned_addr learned my addr 172.21.15.161:0/3642594051 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.194+0000 7eff66352640 1 --2- 172.21.15.161:0/3642594051 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff480039b0 0x7eff600b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.194+0000 7eff65b51640 1 --2- 172.21.15.161:0/3642594051 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff600ae9a0 0x7eff60154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.194+0000 7eff66b53640 1 -- 172.21.15.161:0/3642594051 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff480039b0 msgr2=0x7eff600b02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.194+0000 7eff66b53640 1 --2- 172.21.15.161:0/3642594051 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff480039b0 0x7eff600b02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.194+0000 7eff66b53640 1 -- 172.21.15.161:0/3642594051 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff600ae9a0 msgr2=0x7eff60154450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.194+0000 7eff66b53640 1 --2- 172.21.15.161:0/3642594051 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff600ae9a0 0x7eff60154450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.194+0000 7eff66b53640 1 -- 172.21.15.161:0/3642594051 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff60159670 con 0x7eff601eb1d0 2024-03-28T23:30:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.194+0000 7eff65b51640 1 --2- 172.21.15.161:0/3642594051 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff600ae9a0 0x7eff60154450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:27.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.194+0000 7eff66352640 1 --2- 172.21.15.161:0/3642594051 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff480039b0 0x7eff600b02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:27.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.194+0000 7eff66b53640 1 --2- 172.21.15.161:0/3642594051 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff601eb1d0 0x7eff601eb650 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7eff60159950 tx=0x7eff50007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:27.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.194+0000 7eff5effd640 1 -- 172.21.15.161:0/3642594051 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5001a070 con 0x7eff601eb1d0 2024-03-28T23:30:27.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.194+0000 7eff5effd640 1 -- 172.21.15.161:0/3642594051 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff50007740 con 0x7eff601eb1d0 2024-03-28T23:30:27.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.194+0000 7eff5effd640 1 -- 172.21.15.161:0/3642594051 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5000fa20 con 0x7eff601eb1d0 2024-03-28T23:30:27.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.194+0000 7eff685b6640 1 -- 172.21.15.161:0/3642594051 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff600ada10 con 0x7eff601eb1d0 2024-03-28T23:30:27.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.194+0000 7eff685b6640 1 -- 172.21.15.161:0/3642594051 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff600c46d0 con 0x7eff601eb1d0 2024-03-28T23:30:27.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.195+0000 7eff5effd640 1 -- 172.21.15.161:0/3642594051 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7eff5001d070 con 0x7eff601eb1d0 2024-03-28T23:30:27.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.195+0000 7eff685b6640 1 -- 172.21.15.161:0/3642594051 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff6004f4b0 con 0x7eff601eb1d0 2024-03-28T23:30:27.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.196+0000 7eff5effd640 1 --2- 172.21.15.161:0/3642594051 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7eff4c06b4d0 0x7eff4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:27.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.196+0000 7eff5effd640 1 -- 172.21.15.161:0/3642594051 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7eff500899f0 con 0x7eff601eb1d0 2024-03-28T23:30:27.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.196+0000 7eff66352640 1 --2- 172.21.15.161:0/3642594051 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7eff4c06b4d0 0x7eff4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:27.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.196+0000 7eff66352640 1 --2- 172.21.15.161:0/3642594051 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7eff4c06b4d0 0x7eff4c06d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7eff58036000 tx=0x7eff5802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:27.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.200+0000 7eff5effd640 1 -- 172.21.15.161:0/3642594051 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7eff50058530 con 0x7eff601eb1d0 2024-03-28T23:30:27.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.489+0000 7eff685b6640 1 -- 172.21.15.161:0/3642594051 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7eff601502f0 con 0x7eff601eb1d0 2024-03-28T23:30:27.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.489+0000 7eff5effd640 1 -- 172.21.15.161:0/3642594051 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7eff50057e30 con 0x7eff601eb1d0 2024-03-28T23:30:27.490 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:30:27.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.491+0000 7eff685b6640 1 -- 172.21.15.161:0/3642594051 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7eff4c06b4d0 msgr2=0x7eff4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:27.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.491+0000 7eff685b6640 1 --2- 172.21.15.161:0/3642594051 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7eff4c06b4d0 0x7eff4c06d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7eff58036000 tx=0x7eff5802e000 comp rx=0 tx=0).stop 2024-03-28T23:30:27.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.491+0000 7eff685b6640 1 -- 172.21.15.161:0/3642594051 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff601eb1d0 msgr2=0x7eff601eb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:27.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.491+0000 7eff685b6640 1 --2- 172.21.15.161:0/3642594051 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff601eb1d0 0x7eff601eb650 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7eff60159950 tx=0x7eff50007430 comp rx=0 tx=0).stop 2024-03-28T23:30:27.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.491+0000 7eff685b6640 1 -- 172.21.15.161:0/3642594051 shutdown_connections 2024-03-28T23:30:27.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.491+0000 7eff685b6640 1 --2- 172.21.15.161:0/3642594051 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7eff4c06b4d0 0x7eff4c06d990 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.491+0000 7eff685b6640 1 --2- 172.21.15.161:0/3642594051 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff601eb1d0 0x7eff601eb650 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.491+0000 7eff685b6640 1 --2- 172.21.15.161:0/3642594051 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff600ae9a0 0x7eff60154450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.491+0000 7eff685b6640 1 --2- 172.21.15.161:0/3642594051 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff480039b0 0x7eff600b02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.491+0000 7eff685b6640 1 -- 172.21.15.161:0/3642594051 >> 172.21.15.161:0/3642594051 conn(0x7eff600c7430 msgr2=0x7eff60150510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:27.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.491+0000 7eff685b6640 1 -- 172.21.15.161:0/3642594051 shutdown_connections 2024-03-28T23:30:27.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.491+0000 7eff685b6640 1 -- 172.21.15.161:0/3642594051 wait complete. 2024-03-28T23:30:27.503 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:30:27.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.606+0000 7fa05d8fb640 1 Processor -- start 2024-03-28T23:30:27.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.607+0000 7fa05d8fb640 1 -- start start 2024-03-28T23:30:27.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.607+0000 7fa05d8fb640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa0580c4e40 0x7fa0580c8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:27.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.607+0000 7fa05d8fb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa058158860 0x7fa058158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:27.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.607+0000 7fa05d8fb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa058156060 con 0x7fa058072fe0 2024-03-28T23:30:27.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.607+0000 7fa05d8fb640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa058072870 con 0x7fa058158860 2024-03-28T23:30:27.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.607+0000 7fa05d8fb640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fa058072b50 con 0x7fa0580c4e40 2024-03-28T23:30:27.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.607+0000 7fa0577fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa058158860 0x7fa058158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:27.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.607+0000 7fa0567fc640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa058072fe0 0x7fa0580c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38446/0 (socket says 172.21.15.161:38446) 2024-03-28T23:30:27.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.607+0000 7fa0567fc640 1 -- 172.21.15.161:0/1882163789 learned_addr learned my addr 172.21.15.161:0/1882163789 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:27.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.608+0000 7fa0577fe640 1 -- 172.21.15.161:0/1882163789 >> v2:172.21.15.170:3300/0 conn(0x7fa0580c4e40 msgr2=0x7fa0580c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:27.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.608+0000 7fa0577fe640 1 --2- 172.21.15.161:0/1882163789 >> v2:172.21.15.170:3300/0 conn(0x7fa0580c4e40 0x7fa0580c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.608+0000 7fa0577fe640 1 -- 172.21.15.161:0/1882163789 >> v1:172.21.15.161:6789/0 conn(0x7fa058072fe0 legacy=0x7fa0580c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:27.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.608+0000 7fa0577fe640 1 -- 172.21.15.161:0/1882163789 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa058155cf0 con 0x7fa058158860 2024-03-28T23:30:27.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.608+0000 7fa055ffb640 1 -- 172.21.15.161:0/1882163789 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3843676913 0 0) 0x7fa058156060 con 0x7fa058072fe0 2024-03-28T23:30:27.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.609+0000 7fa0577fe640 1 --2- 172.21.15.161:0/1882163789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa058158860 0x7fa058158cd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa0580c98f0 tx=0x7fa05002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=134bb5b4d1de7ac4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:27.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.609+0000 7fa055ffb640 1 -- 172.21.15.161:0/1882163789 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa050039070 con 0x7fa058158860 2024-03-28T23:30:27.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.609+0000 7fa055ffb640 1 -- 172.21.15.161:0/1882163789 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa05002cc80 con 0x7fa058158860 2024-03-28T23:30:27.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.609+0000 7fa055ffb640 1 -- 172.21.15.161:0/1882163789 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa050034660 con 0x7fa058158860 2024-03-28T23:30:27.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.609+0000 7fa05d8fb640 1 -- 172.21.15.161:0/1882163789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa058158860 msgr2=0x7fa058158cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:27.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.609+0000 7fa05d8fb640 1 --2- 172.21.15.161:0/1882163789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa058158860 0x7fa058158cd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa0580c98f0 tx=0x7fa05002bed0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.609+0000 7fa05d8fb640 1 -- 172.21.15.161:0/1882163789 shutdown_connections 2024-03-28T23:30:27.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.609+0000 7fa05d8fb640 1 --2- 172.21.15.161:0/1882163789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa058158860 0x7fa058158cd0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.609+0000 7fa05d8fb640 1 --2- 172.21.15.161:0/1882163789 >> v2:172.21.15.170:3300/0 conn(0x7fa0580c4e40 0x7fa0580c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.609+0000 7fa05d8fb640 1 -- 172.21.15.161:0/1882163789 >> 172.21.15.161:0/1882163789 conn(0x7fa0580c88b0 msgr2=0x7fa0580c8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:27.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.610+0000 7fa05d8fb640 1 -- 172.21.15.161:0/1882163789 shutdown_connections 2024-03-28T23:30:27.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.610+0000 7fa05d8fb640 1 -- 172.21.15.161:0/1882163789 wait complete. 2024-03-28T23:30:27.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.611+0000 7fa05d8fb640 1 Processor -- start 2024-03-28T23:30:27.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.611+0000 7fa05d8fb640 1 -- start start 2024-03-28T23:30:27.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.613+0000 7fa05d8fb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa058072fe0 0x7fa05806c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:27.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.613+0000 7fa05d8fb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0580c4e40 0x7fa05814ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:27.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.613+0000 7fa05d8fb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa058158860 0x7fa05814bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:27.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.613+0000 7fa05d8fb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa0580c5bd0 con 0x7fa058158860 2024-03-28T23:30:27.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.613+0000 7fa05d8fb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa0580c5a50 con 0x7fa058072fe0 2024-03-28T23:30:27.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.613+0000 7fa05d8fb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa0580c6d30 con 0x7fa0580c4e40 2024-03-28T23:30:27.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.613+0000 7fa0577fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa058158860 0x7fa05814bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:27.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.613+0000 7fa0577fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa058158860 0x7fa05814bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51800/0 (socket says 172.21.15.161:51800) 2024-03-28T23:30:27.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.613+0000 7fa0577fe640 1 -- 172.21.15.161:0/1979834507 learned_addr learned my addr 172.21.15.161:0/1979834507 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:27.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.613+0000 7fa056ffd640 1 --2- 172.21.15.161:0/1979834507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa058072fe0 0x7fa05806c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:27.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.613+0000 7fa0567fc640 1 --2- 172.21.15.161:0/1979834507 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0580c4e40 0x7fa05814ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:27.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.613+0000 7fa0577fe640 1 -- 172.21.15.161:0/1979834507 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0580c4e40 msgr2=0x7fa05814ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:27.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.613+0000 7fa0577fe640 1 --2- 172.21.15.161:0/1979834507 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0580c4e40 0x7fa05814ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.613+0000 7fa0577fe640 1 -- 172.21.15.161:0/1979834507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa058072fe0 msgr2=0x7fa05806c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:27.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.613+0000 7fa0577fe640 1 --2- 172.21.15.161:0/1979834507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa058072fe0 0x7fa05806c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.613+0000 7fa0577fe640 1 -- 172.21.15.161:0/1979834507 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa05814dfb0 con 0x7fa058158860 2024-03-28T23:30:27.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.613+0000 7fa0577fe640 1 --2- 172.21.15.161:0/1979834507 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa058158860 0x7fa05814bc00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa050009e80 tx=0x7fa050004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:27.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.613+0000 7fa04ffff640 1 -- 172.21.15.161:0/1979834507 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa050044020 con 0x7fa058158860 2024-03-28T23:30:27.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.613+0000 7fa04ffff640 1 -- 172.21.15.161:0/1979834507 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa05003f470 con 0x7fa058158860 2024-03-28T23:30:27.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.613+0000 7fa04ffff640 1 -- 172.21.15.161:0/1979834507 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa050039040 con 0x7fa058158860 2024-03-28T23:30:27.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.613+0000 7fa05d8fb640 1 -- 172.21.15.161:0/1979834507 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa05806dac0 con 0x7fa058158860 2024-03-28T23:30:27.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.613+0000 7fa05d8fb640 1 -- 172.21.15.161:0/1979834507 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa05806e080 con 0x7fa058158860 2024-03-28T23:30:27.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.614+0000 7fa05d8fb640 1 -- 172.21.15.161:0/1979834507 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa05804f4b0 con 0x7fa058158860 2024-03-28T23:30:27.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.615+0000 7fa04ffff640 1 -- 172.21.15.161:0/1979834507 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa050047020 con 0x7fa058158860 2024-03-28T23:30:27.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.615+0000 7fa04ffff640 1 --2- 172.21.15.161:0/1979834507 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa03806b4d0 0x7fa03806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:27.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.615+0000 7fa056ffd640 1 --2- 172.21.15.161:0/1979834507 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa03806b4d0 0x7fa03806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:27.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.615+0000 7fa04ffff640 1 -- 172.21.15.161:0/1979834507 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa0500ad1c0 con 0x7fa058158860 2024-03-28T23:30:27.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.616+0000 7fa056ffd640 1 --2- 172.21.15.161:0/1979834507 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa03806b4d0 0x7fa03806d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa04000b810 tx=0x7fa040008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:27.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.620+0000 7fa04ffff640 1 -- 172.21.15.161:0/1979834507 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa05004e030 con 0x7fa058158860 2024-03-28T23:30:27.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.793+0000 7fa05d8fb640 1 -- 172.21.15.161:0/1979834507 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa058069590 con 0x7fa058158860 2024-03-28T23:30:27.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.794+0000 7fa04ffff640 1 -- 172.21.15.161:0/1979834507 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7fa05007bd00 con 0x7fa058158860 2024-03-28T23:30:27.795 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:30:27.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.796+0000 7fa05d8fb640 1 -- 172.21.15.161:0/1979834507 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa03806b4d0 msgr2=0x7fa03806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:27.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.796+0000 7fa05d8fb640 1 --2- 172.21.15.161:0/1979834507 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa03806b4d0 0x7fa03806d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa04000b810 tx=0x7fa040008040 comp rx=0 tx=0).stop 2024-03-28T23:30:27.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.796+0000 7fa05d8fb640 1 -- 172.21.15.161:0/1979834507 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa058158860 msgr2=0x7fa05814bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:27.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.796+0000 7fa05d8fb640 1 --2- 172.21.15.161:0/1979834507 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa058158860 0x7fa05814bc00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa050009e80 tx=0x7fa050004a40 comp rx=0 tx=0).stop 2024-03-28T23:30:27.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.796+0000 7fa05d8fb640 1 -- 172.21.15.161:0/1979834507 shutdown_connections 2024-03-28T23:30:27.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.796+0000 7fa05d8fb640 1 --2- 172.21.15.161:0/1979834507 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa03806b4d0 0x7fa03806d990 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.796+0000 7fa05d8fb640 1 --2- 172.21.15.161:0/1979834507 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa058158860 0x7fa05814bc00 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.796+0000 7fa05d8fb640 1 --2- 172.21.15.161:0/1979834507 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0580c4e40 0x7fa05814ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.796+0000 7fa05d8fb640 1 --2- 172.21.15.161:0/1979834507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa058072fe0 0x7fa05806c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.796+0000 7fa05d8fb640 1 -- 172.21.15.161:0/1979834507 >> 172.21.15.161:0/1979834507 conn(0x7fa0580c88b0 msgr2=0x7fa0580c5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:27.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.796+0000 7fa05d8fb640 1 -- 172.21.15.161:0/1979834507 shutdown_connections 2024-03-28T23:30:27.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.796+0000 7fa05d8fb640 1 -- 172.21.15.161:0/1979834507 wait complete. 2024-03-28T23:30:27.812 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:30:27.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.916+0000 7fdd372f4640 1 Processor -- start 2024-03-28T23:30:27.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.916+0000 7fdd372f4640 1 -- start start 2024-03-28T23:30:27.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.917+0000 7fdd372f4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdd301529d0 0x7fdd3006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:27.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.917+0000 7fdd372f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd300c6750 0x7fdd300c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:27.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.917+0000 7fdd372f4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd30152db0 con 0x7fdd30155c30 2024-03-28T23:30:27.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.917+0000 7fdd372f4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fdd300c7210 con 0x7fdd301529d0 2024-03-28T23:30:27.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.917+0000 7fdd372f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdd300c6c50 con 0x7fdd300c6750 2024-03-28T23:30:27.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.917+0000 7fdd35891640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fdd30155c30 0x7fdd300c54f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38460/0 (socket says 172.21.15.161:38460) 2024-03-28T23:30:27.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.917+0000 7fdd35090640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdd301529d0 0x7fdd3006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:27.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.917+0000 7fdd35891640 1 -- 172.21.15.161:0/265730918 learned_addr learned my addr 172.21.15.161:0/265730918 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:27.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.917+0000 7fdd3488f640 1 --2- 172.21.15.161:0/265730918 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd300c6750 0x7fdd300c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:27.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.917+0000 7fdd35090640 1 -- 172.21.15.161:0/265730918 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd300c6750 msgr2=0x7fdd300c88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:27.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.917+0000 7fdd35090640 1 --2- 172.21.15.161:0/265730918 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd300c6750 0x7fdd300c88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.917+0000 7fdd27fff640 1 -- 172.21.15.161:0/265730918 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1341166604 0 0) 0x7fdd30152db0 con 0x7fdd30155c30 2024-03-28T23:30:27.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.918+0000 7fdd35090640 1 -- 172.21.15.161:0/265730918 >> v1:172.21.15.161:6789/0 conn(0x7fdd30155c30 legacy=0x7fdd300c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:27.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.918+0000 7fdd35090640 1 -- 172.21.15.161:0/265730918 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd300c6f30 con 0x7fdd301529d0 2024-03-28T23:30:27.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.918+0000 7fdd35090640 1 --2- 172.21.15.161:0/265730918 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd301529d0 0x7fdd3006bb60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fdd200097a0 tx=0x7fdd2002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a56cc27d78f927a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:27.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.918+0000 7fdd27fff640 1 -- 172.21.15.161:0/265730918 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd20035020 con 0x7fdd301529d0 2024-03-28T23:30:27.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.918+0000 7fdd27fff640 1 -- 172.21.15.161:0/265730918 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd20031830 con 0x7fdd301529d0 2024-03-28T23:30:27.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.918+0000 7fdd27fff640 1 -- 172.21.15.161:0/265730918 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd20031c00 con 0x7fdd301529d0 2024-03-28T23:30:27.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.919+0000 7fdd372f4640 1 -- 172.21.15.161:0/265730918 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd301529d0 msgr2=0x7fdd3006bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:27.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.919+0000 7fdd372f4640 1 --2- 172.21.15.161:0/265730918 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd301529d0 0x7fdd3006bb60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fdd200097a0 tx=0x7fdd2002b8d0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.919+0000 7fdd372f4640 1 -- 172.21.15.161:0/265730918 shutdown_connections 2024-03-28T23:30:27.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.919+0000 7fdd372f4640 1 --2- 172.21.15.161:0/265730918 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd300c6750 0x7fdd300c88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.919+0000 7fdd372f4640 1 --2- 172.21.15.161:0/265730918 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd301529d0 0x7fdd3006bb60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.919+0000 7fdd372f4640 1 -- 172.21.15.161:0/265730918 >> 172.21.15.161:0/265730918 conn(0x7fdd300c7430 msgr2=0x7fdd300c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:27.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.919+0000 7fdd372f4640 1 -- 172.21.15.161:0/265730918 shutdown_connections 2024-03-28T23:30:27.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.919+0000 7fdd372f4640 1 -- 172.21.15.161:0/265730918 wait complete. 2024-03-28T23:30:27.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.921+0000 7fdd372f4640 1 Processor -- start 2024-03-28T23:30:27.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.921+0000 7fdd372f4640 1 -- start start 2024-03-28T23:30:27.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.922+0000 7fdd372f4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd300c6750 0x7fdd301500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:27.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.922+0000 7fdd372f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd301529d0 0x7fdd30148ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:27.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.922+0000 7fdd372f4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd30155c30 0x7fdd301491e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:27.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.922+0000 7fdd372f4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdd300aa160 con 0x7fdd30155c30 2024-03-28T23:30:27.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.922+0000 7fdd372f4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdd30155710 con 0x7fdd300c6750 2024-03-28T23:30:27.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.922+0000 7fdd372f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdd300c5c00 con 0x7fdd301529d0 2024-03-28T23:30:27.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.922+0000 7fdd35891640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd30155c30 0x7fdd301491e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:27.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.922+0000 7fdd35891640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd30155c30 0x7fdd301491e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51810/0 (socket says 172.21.15.161:51810) 2024-03-28T23:30:27.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.922+0000 7fdd35891640 1 -- 172.21.15.161:0/1242302386 learned_addr learned my addr 172.21.15.161:0/1242302386 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:27.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.922+0000 7fdd35090640 1 --2- 172.21.15.161:0/1242302386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd300c6750 0x7fdd301500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:27.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.922+0000 7fdd3488f640 1 --2- 172.21.15.161:0/1242302386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd301529d0 0x7fdd30148ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:27.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.922+0000 7fdd35891640 1 -- 172.21.15.161:0/1242302386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd301529d0 msgr2=0x7fdd30148ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:27.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.922+0000 7fdd35891640 1 --2- 172.21.15.161:0/1242302386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd301529d0 0x7fdd30148ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.922+0000 7fdd35891640 1 -- 172.21.15.161:0/1242302386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd300c6750 msgr2=0x7fdd301500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:27.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.922+0000 7fdd35891640 1 --2- 172.21.15.161:0/1242302386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd300c6750 0x7fdd301500a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:27.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.922+0000 7fdd35891640 1 -- 172.21.15.161:0/1242302386 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd30154ac0 con 0x7fdd30155c30 2024-03-28T23:30:27.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.922+0000 7fdd35090640 1 --2- 172.21.15.161:0/1242302386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd300c6750 0x7fdd301500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:27.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.922+0000 7fdd35891640 1 --2- 172.21.15.161:0/1242302386 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd30155c30 0x7fdd301491e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fdd2800c7d0 tx=0x7fdd2800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:27.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.922+0000 7fdd25ffb640 1 -- 172.21.15.161:0/1242302386 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd28018070 con 0x7fdd30155c30 2024-03-28T23:30:27.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.922+0000 7fdd25ffb640 1 -- 172.21.15.161:0/1242302386 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd2800a640 con 0x7fdd30155c30 2024-03-28T23:30:27.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.922+0000 7fdd372f4640 1 -- 172.21.15.161:0/1242302386 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd30153a00 con 0x7fdd30155c30 2024-03-28T23:30:27.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.923+0000 7fdd25ffb640 1 -- 172.21.15.161:0/1242302386 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2800aa70 con 0x7fdd30155c30 2024-03-28T23:30:27.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.923+0000 7fdd372f4640 1 -- 172.21.15.161:0/1242302386 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd30149a90 con 0x7fdd30155c30 2024-03-28T23:30:27.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.924+0000 7fdd372f4640 1 -- 172.21.15.161:0/1242302386 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd30068f90 con 0x7fdd30155c30 2024-03-28T23:30:27.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.924+0000 7fdd25ffb640 1 -- 172.21.15.161:0/1242302386 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdd2801b070 con 0x7fdd30155c30 2024-03-28T23:30:27.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.924+0000 7fdd25ffb640 1 --2- 172.21.15.161:0/1242302386 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdd1406b500 0x7fdd1406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:27.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.925+0000 7fdd25ffb640 1 -- 172.21.15.161:0/1242302386 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fdd28089a40 con 0x7fdd30155c30 2024-03-28T23:30:27.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.925+0000 7fdd35090640 1 --2- 172.21.15.161:0/1242302386 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdd1406b500 0x7fdd1406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:27.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.925+0000 7fdd35090640 1 --2- 172.21.15.161:0/1242302386 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdd1406b500 0x7fdd1406d9c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdd20036000 tx=0x7fdd20008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:27.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:27.929+0000 7fdd25ffb640 1 -- 172.21.15.161:0/1242302386 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdd30068f90 con 0x7fdd30155c30 2024-03-28T23:30:28.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.107+0000 7fdd372f4640 1 -- 172.21.15.161:0/1242302386 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdd3004f4b0 con 0x7fdd30155c30 2024-03-28T23:30:28.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.108+0000 7fdd25ffb640 1 -- 172.21.15.161:0/1242302386 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7fdd28058610 con 0x7fdd30155c30 2024-03-28T23:30:28.109 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:30:28.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.110+0000 7fdd372f4640 1 -- 172.21.15.161:0/1242302386 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdd1406b500 msgr2=0x7fdd1406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:28.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.110+0000 7fdd372f4640 1 --2- 172.21.15.161:0/1242302386 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdd1406b500 0x7fdd1406d9c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdd20036000 tx=0x7fdd20008030 comp rx=0 tx=0).stop 2024-03-28T23:30:28.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.110+0000 7fdd372f4640 1 -- 172.21.15.161:0/1242302386 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd30155c30 msgr2=0x7fdd301491e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:28.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.110+0000 7fdd372f4640 1 --2- 172.21.15.161:0/1242302386 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd30155c30 0x7fdd301491e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fdd2800c7d0 tx=0x7fdd2800cc90 comp rx=0 tx=0).stop 2024-03-28T23:30:28.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.110+0000 7fdd372f4640 1 -- 172.21.15.161:0/1242302386 shutdown_connections 2024-03-28T23:30:28.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.110+0000 7fdd372f4640 1 --2- 172.21.15.161:0/1242302386 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdd1406b500 0x7fdd1406d9c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.110+0000 7fdd372f4640 1 --2- 172.21.15.161:0/1242302386 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd30155c30 0x7fdd301491e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.110+0000 7fdd372f4640 1 --2- 172.21.15.161:0/1242302386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd301529d0 0x7fdd30148ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.110+0000 7fdd372f4640 1 --2- 172.21.15.161:0/1242302386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd300c6750 0x7fdd301500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.110+0000 7fdd372f4640 1 -- 172.21.15.161:0/1242302386 >> 172.21.15.161:0/1242302386 conn(0x7fdd300c7430 msgr2=0x7fdd301de8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:28.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.110+0000 7fdd372f4640 1 -- 172.21.15.161:0/1242302386 shutdown_connections 2024-03-28T23:30:28.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.110+0000 7fdd372f4640 1 -- 172.21.15.161:0/1242302386 wait complete. 2024-03-28T23:30:28.123 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T23:30:28.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.228+0000 7f2603db5640 1 Processor -- start 2024-03-28T23:30:28.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.228+0000 7f2603db5640 1 -- start start 2024-03-28T23:30:28.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.229+0000 7f2603db5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f25fc152bc0 0x7f25fc0c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:28.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.229+0000 7f2603db5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25fc1589b0 0x7f25fc158e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:28.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.229+0000 7f2603db5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25fc0c6f60 con 0x7f25fc0c8a80 2024-03-28T23:30:28.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.229+0000 7f2603db5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f25fc0c6120 con 0x7f25fc152bc0 2024-03-28T23:30:28.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.229+0000 7f2603db5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f25fc0c6400 con 0x7f25fc1589b0 2024-03-28T23:30:28.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.229+0000 7f2601350640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f25fc0c8a80 0x7f25fc0c8ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38468/0 (socket says 172.21.15.161:38468) 2024-03-28T23:30:28.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.229+0000 7f2601350640 1 -- 172.21.15.161:0/3080267377 learned_addr learned my addr 172.21.15.161:0/3080267377 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:28.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.229+0000 7f2601b51640 1 --2- 172.21.15.161:0/3080267377 >> v2:172.21.15.170:3300/0 conn(0x7f25fc152bc0 0x7f25fc0c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:28.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.229+0000 7f2602352640 1 --2- 172.21.15.161:0/3080267377 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25fc1589b0 0x7f25fc158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:28.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.229+0000 7f2600b4f640 1 -- 172.21.15.161:0/3080267377 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2921729550 0 0) 0x7f25fc0c6f60 con 0x7f25fc0c8a80 2024-03-28T23:30:28.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.229+0000 7f2600b4f640 1 -- 172.21.15.161:0/3080267377 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25e8003520 con 0x7f25fc0c8a80 2024-03-28T23:30:28.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.230+0000 7f2601b51640 1 -- 172.21.15.161:0/3080267377 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25fc1589b0 msgr2=0x7f25fc158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:28.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.230+0000 7f2601b51640 1 --2- 172.21.15.161:0/3080267377 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25fc1589b0 0x7f25fc158e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.230+0000 7f2601b51640 1 -- 172.21.15.161:0/3080267377 >> v1:172.21.15.161:6789/0 conn(0x7f25fc0c8a80 legacy=0x7f25fc0c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:28.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.230+0000 7f2601b51640 1 -- 172.21.15.161:0/3080267377 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25fc0c7240 con 0x7f25fc152bc0 2024-03-28T23:30:28.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.230+0000 7f2602352640 1 --2- 172.21.15.161:0/3080267377 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25fc1589b0 0x7f25fc158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:28.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.230+0000 7f2601b51640 1 --2- 172.21.15.161:0/3080267377 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25fc152bc0 0x7f25fc0c0330 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f25f40097a0 tx=0x7f25f402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=291dd15da4c20b00 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:28.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.231+0000 7f2600b4f640 1 -- 172.21.15.161:0/3080267377 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f4035020 con 0x7f25fc152bc0 2024-03-28T23:30:28.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.231+0000 7f2600b4f640 1 -- 172.21.15.161:0/3080267377 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25f4031830 con 0x7f25fc152bc0 2024-03-28T23:30:28.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.231+0000 7f2600b4f640 1 -- 172.21.15.161:0/3080267377 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f4031c00 con 0x7f25fc152bc0 2024-03-28T23:30:28.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.231+0000 7f2603db5640 1 -- 172.21.15.161:0/3080267377 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25fc152bc0 msgr2=0x7f25fc0c0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:28.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.231+0000 7f2603db5640 1 --2- 172.21.15.161:0/3080267377 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25fc152bc0 0x7f25fc0c0330 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f25f40097a0 tx=0x7f25f402b8d0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.231+0000 7f2603db5640 1 -- 172.21.15.161:0/3080267377 shutdown_connections 2024-03-28T23:30:28.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.231+0000 7f2603db5640 1 --2- 172.21.15.161:0/3080267377 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25fc1589b0 0x7f25fc158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.231+0000 7f2603db5640 1 --2- 172.21.15.161:0/3080267377 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25fc152bc0 0x7f25fc0c0330 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.231+0000 7f2603db5640 1 -- 172.21.15.161:0/3080267377 >> 172.21.15.161:0/3080267377 conn(0x7f25fc0caa80 msgr2=0x7f25fc0cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:28.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.231+0000 7f2603db5640 1 -- 172.21.15.161:0/3080267377 shutdown_connections 2024-03-28T23:30:28.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.231+0000 7f2603db5640 1 -- 172.21.15.161:0/3080267377 wait complete. 2024-03-28T23:30:28.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.233+0000 7f2603db5640 1 Processor -- start 2024-03-28T23:30:28.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.233+0000 7f2603db5640 1 -- start start 2024-03-28T23:30:28.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.233+0000 7f2603db5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25fc0c8a80 0x7f25fc149fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:28.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.233+0000 7f2603db5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25fc152bc0 0x7f25fc1ec1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:28.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.233+0000 7f2603db5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25fc1589b0 0x7f25fc1ec6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:28.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.233+0000 7f2603db5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc152fa0 con 0x7f25fc0c8a80 2024-03-28T23:30:28.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.233+0000 7f2603db5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc0c95f0 con 0x7f25fc1589b0 2024-03-28T23:30:28.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.233+0000 7f2603db5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f25fc06c480 con 0x7f25fc152bc0 2024-03-28T23:30:28.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.233+0000 7f2601b51640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25fc0c8a80 0x7f25fc149fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:28.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.233+0000 7f2601b51640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25fc0c8a80 0x7f25fc149fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51820/0 (socket says 172.21.15.161:51820) 2024-03-28T23:30:28.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.233+0000 7f2601b51640 1 -- 172.21.15.161:0/1241500922 learned_addr learned my addr 172.21.15.161:0/1241500922 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:28.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.234+0000 7f2602352640 1 --2- 172.21.15.161:0/1241500922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25fc1589b0 0x7f25fc1ec6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:28.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.234+0000 7f2601b51640 1 -- 172.21.15.161:0/1241500922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25fc152bc0 msgr2=0x7f25fc1ec1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:28.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.234+0000 7f2601350640 1 --2- 172.21.15.161:0/1241500922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25fc152bc0 0x7f25fc1ec1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:28.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.234+0000 7f2601b51640 1 --2- 172.21.15.161:0/1241500922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25fc152bc0 0x7f25fc1ec1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.234+0000 7f2601b51640 1 -- 172.21.15.161:0/1241500922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25fc1589b0 msgr2=0x7f25fc1ec6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:28.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.234+0000 7f2601b51640 1 --2- 172.21.15.161:0/1241500922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25fc1589b0 0x7f25fc1ec6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.234+0000 7f2601b51640 1 -- 172.21.15.161:0/1241500922 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25fc1ead40 con 0x7f25fc0c8a80 2024-03-28T23:30:28.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.234+0000 7f2602352640 1 --2- 172.21.15.161:0/1241500922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25fc1589b0 0x7f25fc1ec6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:28.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.234+0000 7f2601350640 1 --2- 172.21.15.161:0/1241500922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25fc152bc0 0x7f25fc1ec1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:28.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.234+0000 7f2601b51640 1 --2- 172.21.15.161:0/1241500922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25fc0c8a80 0x7f25fc149fe0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f25f402bee0 tx=0x7f25f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:28.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.234+0000 7f25f27fc640 1 -- 172.21.15.161:0/1241500922 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f4035040 con 0x7f25fc0c8a80 2024-03-28T23:30:28.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.234+0000 7f25f27fc640 1 -- 172.21.15.161:0/1241500922 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25f403f070 con 0x7f25fc0c8a80 2024-03-28T23:30:28.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.234+0000 7f25f27fc640 1 -- 172.21.15.161:0/1241500922 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f403bb60 con 0x7f25fc0c8a80 2024-03-28T23:30:28.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.234+0000 7f2603db5640 1 -- 172.21.15.161:0/1241500922 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25fc1ef030 con 0x7f25fc0c8a80 2024-03-28T23:30:28.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.234+0000 7f2603db5640 1 -- 172.21.15.161:0/1241500922 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25fc1eb2e0 con 0x7f25fc0c8a80 2024-03-28T23:30:28.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.235+0000 7f2603db5640 1 -- 172.21.15.161:0/1241500922 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25fc04f4b0 con 0x7f25fc0c8a80 2024-03-28T23:30:28.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.236+0000 7f25f27fc640 1 -- 172.21.15.161:0/1241500922 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f25f4038020 con 0x7f25fc0c8a80 2024-03-28T23:30:28.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.236+0000 7f25f27fc640 1 --2- 172.21.15.161:0/1241500922 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f25e006b4d0 0x7f25e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:28.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.236+0000 7f25f27fc640 1 -- 172.21.15.161:0/1241500922 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f25f40ac060 con 0x7f25fc0c8a80 2024-03-28T23:30:28.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.236+0000 7f2601350640 1 --2- 172.21.15.161:0/1241500922 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f25e006b4d0 0x7f25e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:28.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.237+0000 7f2601350640 1 --2- 172.21.15.161:0/1241500922 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f25e006b4d0 0x7f25e006d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f25fc1ee210 tx=0x7f25ec009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:28.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.241+0000 7f25f27fc640 1 -- 172.21.15.161:0/1241500922 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f25f407ac30 con 0x7f25fc0c8a80 2024-03-28T23:30:28.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.414+0000 7f2603db5640 1 -- 172.21.15.161:0/1241500922 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f25fc0c0860 con 0x7f25fc0c8a80 2024-03-28T23:30:28.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.415+0000 7f25f27fc640 1 -- 172.21.15.161:0/1241500922 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f25f407a530 con 0x7f25fc0c8a80 2024-03-28T23:30:28.415 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:30:28.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.416+0000 7f2603db5640 1 -- 172.21.15.161:0/1241500922 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f25e006b4d0 msgr2=0x7f25e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:28.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.416+0000 7f2603db5640 1 --2- 172.21.15.161:0/1241500922 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f25e006b4d0 0x7f25e006d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f25fc1ee210 tx=0x7f25ec009040 comp rx=0 tx=0).stop 2024-03-28T23:30:28.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.416+0000 7f2603db5640 1 -- 172.21.15.161:0/1241500922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25fc0c8a80 msgr2=0x7f25fc149fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:28.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.416+0000 7f2603db5640 1 --2- 172.21.15.161:0/1241500922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25fc0c8a80 0x7f25fc149fe0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f25f402bee0 tx=0x7f25f40046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.416+0000 7f2603db5640 1 -- 172.21.15.161:0/1241500922 shutdown_connections 2024-03-28T23:30:28.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.416+0000 7f2603db5640 1 --2- 172.21.15.161:0/1241500922 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f25e006b4d0 0x7f25e006d990 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.416+0000 7f2603db5640 1 --2- 172.21.15.161:0/1241500922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25fc1589b0 0x7f25fc1ec6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.416+0000 7f2603db5640 1 --2- 172.21.15.161:0/1241500922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25fc152bc0 0x7f25fc1ec1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.416+0000 7f2603db5640 1 --2- 172.21.15.161:0/1241500922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25fc0c8a80 0x7f25fc149fe0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.417+0000 7f2603db5640 1 -- 172.21.15.161:0/1241500922 >> 172.21.15.161:0/1241500922 conn(0x7f25fc0caa80 msgr2=0x7f25fc14cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:28.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.417+0000 7f2603db5640 1 -- 172.21.15.161:0/1241500922 shutdown_connections 2024-03-28T23:30:28.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.417+0000 7f2603db5640 1 -- 172.21.15.161:0/1241500922 wait complete. 2024-03-28T23:30:28.428 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:30:28.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.534+0000 7f1203ed4640 1 Processor -- start 2024-03-28T23:30:28.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.534+0000 7f1203ed4640 1 -- start start 2024-03-28T23:30:28.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.535+0000 7f1203ed4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11fc150600 0x7f11fc0a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:28.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.535+0000 7f1203ed4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f11fc0aacb0 0x7f11fc153980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:28.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.535+0000 7f1203ed4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11fc1567c0 con 0x7f11fc0a8fc0 2024-03-28T23:30:28.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.535+0000 7f1203ed4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f11fc156570 con 0x7f11fc0aacb0 2024-03-28T23:30:28.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.535+0000 7f1203ed4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f11fc156aa0 con 0x7f11fc150600 2024-03-28T23:30:28.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.536+0000 7f1202471640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f11fc0aacb0 0x7f11fc153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:28.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.536+0000 7f1201c70640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11fc150600 0x7f11fc0a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:28.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.536+0000 7f1202471640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f11fc0aacb0 0x7f11fc153980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:50362/0 (socket says 172.21.15.161:50362) 2024-03-28T23:30:28.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.536+0000 7f1201c70640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11fc150600 0x7f11fc0a9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:38208/0 (socket says 172.21.15.161:38208) 2024-03-28T23:30:28.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.536+0000 7f1202471640 1 -- 172.21.15.161:0/2287000151 learned_addr learned my addr 172.21.15.161:0/2287000151 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:28.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.536+0000 7f120146f640 1 --1- 172.21.15.161:0/2287000151 >> v1:172.21.15.170:6789/0 conn(0x7f11fc0a8fc0 0x7f11fc153270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:34382/0 (socket says 172.21.15.161:34382) 2024-03-28T23:30:28.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.536+0000 7f1202471640 1 -- 172.21.15.161:0/2287000151 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11fc150600 msgr2=0x7f11fc0a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:28.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.536+0000 7f1202471640 1 --2- 172.21.15.161:0/2287000151 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11fc150600 0x7f11fc0a9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.536+0000 7f1202471640 1 -- 172.21.15.161:0/2287000151 >> v1:172.21.15.170:6789/0 conn(0x7f11fc0a8fc0 legacy=0x7f11fc153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:28.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.536+0000 7f1202471640 1 -- 172.21.15.161:0/2287000151 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11fc156d80 con 0x7f11fc0aacb0 2024-03-28T23:30:28.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.536+0000 7f1201c70640 1 --2- 172.21.15.161:0/2287000151 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11fc150600 0x7f11fc0a9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:28.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.537+0000 7f1202471640 1 --2- 172.21.15.161:0/2287000151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11fc0aacb0 0x7f11fc153980 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f11f8009e50 tx=0x7f11f802bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f4571dd1357a442 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:28.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.537+0000 7f1200c6e640 1 -- 172.21.15.161:0/2287000151 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f802ca70 con 0x7f11fc0aacb0 2024-03-28T23:30:28.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.537+0000 7f1200c6e640 1 -- 172.21.15.161:0/2287000151 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11f802cc10 con 0x7f11fc0aacb0 2024-03-28T23:30:28.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.537+0000 7f1200c6e640 1 -- 172.21.15.161:0/2287000151 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f80343f0 con 0x7f11fc0aacb0 2024-03-28T23:30:28.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.538+0000 7f1203ed4640 1 -- 172.21.15.161:0/2287000151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11fc0aacb0 msgr2=0x7f11fc153980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:28.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.538+0000 7f1203ed4640 1 --2- 172.21.15.161:0/2287000151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11fc0aacb0 0x7f11fc153980 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f11f8009e50 tx=0x7f11f802bed0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.538+0000 7f1203ed4640 1 -- 172.21.15.161:0/2287000151 shutdown_connections 2024-03-28T23:30:28.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.538+0000 7f1203ed4640 1 --2- 172.21.15.161:0/2287000151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11fc0aacb0 0x7f11fc153980 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.538+0000 7f1203ed4640 1 --2- 172.21.15.161:0/2287000151 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11fc150600 0x7f11fc0a9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.538+0000 7f1203ed4640 1 -- 172.21.15.161:0/2287000151 >> 172.21.15.161:0/2287000151 conn(0x7f11fc0c6670 msgr2=0x7f11fc0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:28.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.538+0000 7f1203ed4640 1 -- 172.21.15.161:0/2287000151 shutdown_connections 2024-03-28T23:30:28.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.538+0000 7f1203ed4640 1 -- 172.21.15.161:0/2287000151 wait complete. 2024-03-28T23:30:28.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.539+0000 7f1203ed4640 1 Processor -- start 2024-03-28T23:30:28.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.539+0000 7f1203ed4640 1 -- start start 2024-03-28T23:30:28.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.540+0000 7f1203ed4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11fc0a8fc0 0x7f11fc1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:28.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.540+0000 7f1203ed4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11fc0aacb0 0x7f11fc0c1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:28.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.540+0000 7f1203ed4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11fc150600 0x7f11fc0c1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:28.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.540+0000 7f1203ed4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f11fc153ec0 con 0x7f11fc150600 2024-03-28T23:30:28.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.540+0000 7f1203ed4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f11fc157340 con 0x7f11fc0aacb0 2024-03-28T23:30:28.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.540+0000 7f1203ed4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f11fc1509e0 con 0x7f11fc0a8fc0 2024-03-28T23:30:28.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.540+0000 7f1202471640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11fc150600 0x7f11fc0c1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:28.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.540+0000 7f1202471640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11fc150600 0x7f11fc0c1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51824/0 (socket says 172.21.15.161:51824) 2024-03-28T23:30:28.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.540+0000 7f1202471640 1 -- 172.21.15.161:0/3484001903 learned_addr learned my addr 172.21.15.161:0/3484001903 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:28.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.540+0000 7f1201c70640 1 --2- 172.21.15.161:0/3484001903 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11fc0a8fc0 0x7f11fc1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:28.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.540+0000 7f120146f640 1 --2- 172.21.15.161:0/3484001903 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11fc0aacb0 0x7f11fc0c1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:28.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.540+0000 7f1202471640 1 -- 172.21.15.161:0/3484001903 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11fc0a8fc0 msgr2=0x7f11fc1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:28.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.540+0000 7f1202471640 1 --2- 172.21.15.161:0/3484001903 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11fc0a8fc0 0x7f11fc1553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.540+0000 7f1202471640 1 -- 172.21.15.161:0/3484001903 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11fc0aacb0 msgr2=0x7f11fc0c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:28.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.540+0000 7f1202471640 1 --2- 172.21.15.161:0/3484001903 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11fc0aacb0 0x7f11fc0c1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.540+0000 7f1202471640 1 -- 172.21.15.161:0/3484001903 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11fc0c36c0 con 0x7f11fc150600 2024-03-28T23:30:28.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.540+0000 7f1202471640 1 --2- 172.21.15.161:0/3484001903 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11fc150600 0x7f11fc0c1a80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f11f8009e50 tx=0x7f11f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:28.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.541+0000 7f11ee7fc640 1 -- 172.21.15.161:0/3484001903 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f8043020 con 0x7f11fc150600 2024-03-28T23:30:28.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.541+0000 7f11ee7fc640 1 -- 172.21.15.161:0/3484001903 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11f803e450 con 0x7f11fc150600 2024-03-28T23:30:28.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.541+0000 7f11ee7fc640 1 -- 172.21.15.161:0/3484001903 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f8009510 con 0x7f11fc150600 2024-03-28T23:30:28.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.541+0000 7f1203ed4640 1 -- 172.21.15.161:0/3484001903 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11fc0c39a0 con 0x7f11fc150600 2024-03-28T23:30:28.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.541+0000 7f1203ed4640 1 -- 172.21.15.161:0/3484001903 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11fc1558e0 con 0x7f11fc150600 2024-03-28T23:30:28.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.542+0000 7f1203ed4640 1 -- 172.21.15.161:0/3484001903 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11fc04f4b0 con 0x7f11fc150600 2024-03-28T23:30:28.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.542+0000 7f11ee7fc640 1 -- 172.21.15.161:0/3484001903 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f11f8046020 con 0x7f11fc150600 2024-03-28T23:30:28.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.542+0000 7f11ee7fc640 1 --2- 172.21.15.161:0/3484001903 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f11e006b4d0 0x7f11e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:28.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.543+0000 7f11ee7fc640 1 -- 172.21.15.161:0/3484001903 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f11f80acad0 con 0x7f11fc150600 2024-03-28T23:30:28.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.543+0000 7f1201c70640 1 --2- 172.21.15.161:0/3484001903 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f11e006b4d0 0x7f11e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:28.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.543+0000 7f1201c70640 1 --2- 172.21.15.161:0/3484001903 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f11e006b4d0 0x7f11e006d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009560 tx=0x7f11f4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:28.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.547+0000 7f11ee7fc640 1 -- 172.21.15.161:0/3484001903 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f11f804d030 con 0x7f11fc150600 2024-03-28T23:30:28.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.721+0000 7f1203ed4640 1 -- 172.21.15.161:0/3484001903 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f11fc0a9d20 con 0x7f11fc150600 2024-03-28T23:30:28.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.723+0000 7f11ee7fc640 1 -- 172.21.15.161:0/3484001903 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7f11f807b050 con 0x7f11fc150600 2024-03-28T23:30:28.723 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:28.724 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":24,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:29:50.946843+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:30:28.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.724+0000 7f1203ed4640 1 -- 172.21.15.161:0/3484001903 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f11e006b4d0 msgr2=0x7f11e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:28.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.724+0000 7f1203ed4640 1 --2- 172.21.15.161:0/3484001903 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f11e006b4d0 0x7f11e006d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009560 tx=0x7f11f4009190 comp rx=0 tx=0).stop 2024-03-28T23:30:28.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.724+0000 7f1203ed4640 1 -- 172.21.15.161:0/3484001903 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11fc150600 msgr2=0x7f11fc0c1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:28.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.724+0000 7f1203ed4640 1 --2- 172.21.15.161:0/3484001903 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11fc150600 0x7f11fc0c1a80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f11f8009e50 tx=0x7f11f8004a40 comp rx=0 tx=0).stop 2024-03-28T23:30:28.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.725+0000 7f1203ed4640 1 -- 172.21.15.161:0/3484001903 shutdown_connections 2024-03-28T23:30:28.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.725+0000 7f1203ed4640 1 --2- 172.21.15.161:0/3484001903 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f11e006b4d0 0x7f11e006d990 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.725+0000 7f1203ed4640 1 --2- 172.21.15.161:0/3484001903 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11fc150600 0x7f11fc0c1a80 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.725+0000 7f1203ed4640 1 --2- 172.21.15.161:0/3484001903 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11fc0aacb0 0x7f11fc0c1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.725+0000 7f1203ed4640 1 --2- 172.21.15.161:0/3484001903 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11fc0a8fc0 0x7f11fc1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.725+0000 7f1203ed4640 1 -- 172.21.15.161:0/3484001903 >> 172.21.15.161:0/3484001903 conn(0x7f11fc0c6670 msgr2=0x7f11fc0a9f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:28.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.725+0000 7f1203ed4640 1 -- 172.21.15.161:0/3484001903 shutdown_connections 2024-03-28T23:30:28.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.725+0000 7f1203ed4640 1 -- 172.21.15.161:0/3484001903 wait complete. 2024-03-28T23:30:28.737 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:29:44.021375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:29:44.872102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T23:29:48.591063+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T23:30:28.738 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:30:28.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.842+0000 7f13321bc640 1 Processor -- start 2024-03-28T23:30:28.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.842+0000 7f13321bc640 1 -- start start 2024-03-28T23:30:28.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.842+0000 7f13321bc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f132c152850 0x7f132c0c8b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:28.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.842+0000 7f13321bc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f132c1552a0 0x7f132c1587b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:28.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.842+0000 7f13321bc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f132c0c7d00 0x7f132c0c53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:28.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.842+0000 7f13321bc640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f132c152c30 con 0x7f132c1552a0 2024-03-28T23:30:28.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.842+0000 7f13321bc640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f132c1590b0 con 0x7f132c0c7d00 2024-03-28T23:30:28.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.842+0000 7f13321bc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f132c159390 con 0x7f132c152850 2024-03-28T23:30:28.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.842+0000 7f132affd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f132c1552a0 0x7f132c1587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:28.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.842+0000 7f132affd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f132c1552a0 0x7f132c1587b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51830/0 (socket says 172.21.15.161:51830) 2024-03-28T23:30:28.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.842+0000 7f132affd640 1 -- 172.21.15.161:0/3570842115 learned_addr learned my addr 172.21.15.161:0/3570842115 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:28.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.843+0000 7f132affd640 1 -- 172.21.15.161:0/3570842115 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f132c152850 msgr2=0x7f132c0c8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:28.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.843+0000 7f132affd640 1 --2- 172.21.15.161:0/3570842115 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f132c152850 0x7f132c0c8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.843+0000 7f132affd640 1 -- 172.21.15.161:0/3570842115 >> v2:172.21.15.170:3300/0 conn(0x7f132c0c7d00 msgr2=0x7f132c0c53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:28.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.843+0000 7f132bfff640 1 --2- 172.21.15.161:0/3570842115 >> v2:172.21.15.170:3300/0 conn(0x7f132c0c7d00 0x7f132c0c53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:28.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.843+0000 7f132affd640 1 --2- 172.21.15.161:0/3570842115 >> v2:172.21.15.170:3300/0 conn(0x7f132c0c7d00 0x7f132c0c53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.843+0000 7f132affd640 1 -- 172.21.15.161:0/3570842115 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132c159670 con 0x7f132c1552a0 2024-03-28T23:30:28.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.844+0000 7f132affd640 1 --2- 172.21.15.161:0/3570842115 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f132c1552a0 0x7f132c1587b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f132c0ad3f0 tx=0x7f132002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2ee32067c767db2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:28.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.844+0000 7f132a7fc640 1 -- 172.21.15.161:0/3570842115 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132002e070 con 0x7f132c1552a0 2024-03-28T23:30:28.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.844+0000 7f132a7fc640 1 -- 172.21.15.161:0/3570842115 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1320037070 con 0x7f132c1552a0 2024-03-28T23:30:28.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.844+0000 7f132a7fc640 1 -- 172.21.15.161:0/3570842115 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1320031a00 con 0x7f132c1552a0 2024-03-28T23:30:28.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.844+0000 7f13321bc640 1 -- 172.21.15.161:0/3570842115 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f132c1552a0 msgr2=0x7f132c1587b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:28.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.844+0000 7f13321bc640 1 --2- 172.21.15.161:0/3570842115 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f132c1552a0 0x7f132c1587b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f132c0ad3f0 tx=0x7f132002b880 comp rx=0 tx=0).stop 2024-03-28T23:30:28.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.844+0000 7f13321bc640 1 -- 172.21.15.161:0/3570842115 shutdown_connections 2024-03-28T23:30:28.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.844+0000 7f13321bc640 1 --2- 172.21.15.161:0/3570842115 >> v2:172.21.15.170:3300/0 conn(0x7f132c0c7d00 0x7f132c0c53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.844+0000 7f13321bc640 1 --2- 172.21.15.161:0/3570842115 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f132c1552a0 0x7f132c1587b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.844+0000 7f13321bc640 1 --2- 172.21.15.161:0/3570842115 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f132c152850 0x7f132c0c8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.844+0000 7f13321bc640 1 -- 172.21.15.161:0/3570842115 >> 172.21.15.161:0/3570842115 conn(0x7f132c0ac5e0 msgr2=0x7f132c0ac9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:28.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.844+0000 7f13321bc640 1 -- 172.21.15.161:0/3570842115 shutdown_connections 2024-03-28T23:30:28.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.844+0000 7f13321bc640 1 -- 172.21.15.161:0/3570842115 wait complete. 2024-03-28T23:30:28.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.846+0000 7f13321bc640 1 Processor -- start 2024-03-28T23:30:28.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.846+0000 7f13321bc640 1 -- start start 2024-03-28T23:30:28.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.847+0000 7f13321bc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f132c0c7d00 0x7f132c0ca340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:28.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.847+0000 7f13321bc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f132c152850 0x7f132c0ab880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:28.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.847+0000 7f13321bc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f132c1552a0 0x7f132c0abdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:28.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.847+0000 7f13321bc640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f132c158cf0 con 0x7f132c1552a0 2024-03-28T23:30:28.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.847+0000 7f13321bc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f132c0c77e0 con 0x7f132c152850 2024-03-28T23:30:28.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.847+0000 7f13321bc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f132c158e70 con 0x7f132c0c7d00 2024-03-28T23:30:28.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.847+0000 7f132bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f132c1552a0 0x7f132c0abdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:28.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.848+0000 7f132bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f132c1552a0 0x7f132c0abdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51834/0 (socket says 172.21.15.161:51834) 2024-03-28T23:30:28.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.848+0000 7f132bfff640 1 -- 172.21.15.161:0/2649338519 learned_addr learned my addr 172.21.15.161:0/2649338519 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:28.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.848+0000 7f132affd640 1 --2- 172.21.15.161:0/2649338519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f132c152850 0x7f132c0ab880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:28.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.848+0000 7f132b7fe640 1 --2- 172.21.15.161:0/2649338519 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f132c0c7d00 0x7f132c0ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:28.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.848+0000 7f132bfff640 1 -- 172.21.15.161:0/2649338519 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f132c0c7d00 msgr2=0x7f132c0ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:28.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.848+0000 7f132bfff640 1 --2- 172.21.15.161:0/2649338519 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f132c0c7d00 0x7f132c0ca340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.848+0000 7f132bfff640 1 -- 172.21.15.161:0/2649338519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f132c152850 msgr2=0x7f132c0ab880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:28.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.848+0000 7f132bfff640 1 --2- 172.21.15.161:0/2649338519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f132c152850 0x7f132c0ab880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:28.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.848+0000 7f132bfff640 1 -- 172.21.15.161:0/2649338519 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132c0aaac0 con 0x7f132c1552a0 2024-03-28T23:30:28.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.848+0000 7f132bfff640 1 --2- 172.21.15.161:0/2649338519 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f132c1552a0 0x7f132c0abdc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f132400cc60 tx=0x7f132400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:28.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.848+0000 7f1313fff640 1 -- 172.21.15.161:0/2649338519 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1324003990 con 0x7f132c1552a0 2024-03-28T23:30:28.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.848+0000 7f1313fff640 1 -- 172.21.15.161:0/2649338519 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1324003b30 con 0x7f132c1552a0 2024-03-28T23:30:28.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.848+0000 7f1313fff640 1 -- 172.21.15.161:0/2649338519 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1324004020 con 0x7f132c1552a0 2024-03-28T23:30:28.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.848+0000 7f13321bc640 1 -- 172.21.15.161:0/2649338519 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f132c0aada0 con 0x7f132c1552a0 2024-03-28T23:30:28.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.848+0000 7f13321bc640 1 -- 172.21.15.161:0/2649338519 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f132c0ab360 con 0x7f132c1552a0 2024-03-28T23:30:28.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.849+0000 7f13321bc640 1 -- 172.21.15.161:0/2649338519 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f132c0c8190 con 0x7f132c1552a0 2024-03-28T23:30:28.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.850+0000 7f1313fff640 1 -- 172.21.15.161:0/2649338519 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f13240041c0 con 0x7f132c1552a0 2024-03-28T23:30:28.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.850+0000 7f1313fff640 1 --2- 172.21.15.161:0/2649338519 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f131806b500 0x7f131806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:28.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.850+0000 7f1313fff640 1 -- 172.21.15.161:0/2649338519 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f132408a9e0 con 0x7f132c1552a0 2024-03-28T23:30:28.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.850+0000 7f132b7fe640 1 --2- 172.21.15.161:0/2649338519 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f131806b500 0x7f131806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:28.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.851+0000 7f132b7fe640 1 --2- 172.21.15.161:0/2649338519 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f131806b500 0x7f131806d9c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f131c009890 tx=0x7f131c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:28.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:28.855+0000 7f1313fff640 1 -- 172.21.15.161:0/2649338519 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f132c0c8190 con 0x7f132c1552a0 2024-03-28T23:30:29.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.032+0000 7f13321bc640 1 -- 172.21.15.161:0/2649338519 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f132c14d880 con 0x7f132c1552a0 2024-03-28T23:30:29.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.033+0000 7f1313fff640 1 -- 172.21.15.161:0/2649338519 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f132c14d880 con 0x7f132c1552a0 2024-03-28T23:30:29.034 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:30:29.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.035+0000 7f13321bc640 1 -- 172.21.15.161:0/2649338519 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f131806b500 msgr2=0x7f131806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:29.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.035+0000 7f13321bc640 1 --2- 172.21.15.161:0/2649338519 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f131806b500 0x7f131806d9c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f131c009890 tx=0x7f131c008040 comp rx=0 tx=0).stop 2024-03-28T23:30:29.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.035+0000 7f13321bc640 1 -- 172.21.15.161:0/2649338519 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f132c1552a0 msgr2=0x7f132c0abdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:29.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.035+0000 7f13321bc640 1 --2- 172.21.15.161:0/2649338519 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f132c1552a0 0x7f132c0abdc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f132400cc60 tx=0x7f132400a5c0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.035+0000 7f13321bc640 1 -- 172.21.15.161:0/2649338519 shutdown_connections 2024-03-28T23:30:29.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.035+0000 7f13321bc640 1 --2- 172.21.15.161:0/2649338519 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f131806b500 0x7f131806d9c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.035+0000 7f13321bc640 1 --2- 172.21.15.161:0/2649338519 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f132c1552a0 0x7f132c0abdc0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.035+0000 7f13321bc640 1 --2- 172.21.15.161:0/2649338519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f132c152850 0x7f132c0ab880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.035+0000 7f13321bc640 1 --2- 172.21.15.161:0/2649338519 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f132c0c7d00 0x7f132c0ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.035+0000 7f13321bc640 1 -- 172.21.15.161:0/2649338519 >> 172.21.15.161:0/2649338519 conn(0x7f132c0ac5e0 msgr2=0x7f132c14e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:29.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.035+0000 7f13321bc640 1 -- 172.21.15.161:0/2649338519 shutdown_connections 2024-03-28T23:30:29.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.035+0000 7f13321bc640 1 -- 172.21.15.161:0/2649338519 wait complete. 2024-03-28T23:30:29.047 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:30:29.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.152+0000 7f252c27c640 1 Processor -- start 2024-03-28T23:30:29.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.152+0000 7f252c27c640 1 -- start start 2024-03-28T23:30:29.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.153+0000 7f252c27c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2524072fe0 0x7f25240c8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:29.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.153+0000 7f252c27c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2524158860 0x7f2524158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:29.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.153+0000 7f252c27c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2524156060 con 0x7f25240c4e40 2024-03-28T23:30:29.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.153+0000 7f252c27c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2524072870 con 0x7f2524072fe0 2024-03-28T23:30:29.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.153+0000 7f252c27c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f2524072b50 con 0x7f2524158860 2024-03-28T23:30:29.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.153+0000 7f2529817640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2524072fe0 0x7f25240c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:29.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.153+0000 7f252a018640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f25240c4e40 0x7f25240c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38470/0 (socket says 172.21.15.161:38470) 2024-03-28T23:30:29.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.153+0000 7f2529817640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2524072fe0 0x7f25240c8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51844/0 (socket says 172.21.15.161:51844) 2024-03-28T23:30:29.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.153+0000 7f252a018640 1 -- 172.21.15.161:0/1615930423 learned_addr learned my addr 172.21.15.161:0/1615930423 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:29.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.153+0000 7f252a819640 1 --2- 172.21.15.161:0/1615930423 >> v2:172.21.15.170:3300/0 conn(0x7f2524158860 0x7f2524158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:29.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.153+0000 7f2529817640 1 -- 172.21.15.161:0/1615930423 >> v2:172.21.15.170:3300/0 conn(0x7f2524158860 msgr2=0x7f2524158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:29.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.153+0000 7f2529817640 1 --2- 172.21.15.161:0/1615930423 >> v2:172.21.15.170:3300/0 conn(0x7f2524158860 0x7f2524158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.153+0000 7f2529817640 1 -- 172.21.15.161:0/1615930423 >> v1:172.21.15.161:6789/0 conn(0x7f25240c4e40 legacy=0x7f25240c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:29.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.153+0000 7f2529817640 1 -- 172.21.15.161:0/1615930423 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2524155cf0 con 0x7f2524072fe0 2024-03-28T23:30:29.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.153+0000 7f2529016640 1 -- 172.21.15.161:0/1615930423 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 709815658 0 0) 0x7f2524156060 con 0x7f25240c4e40 2024-03-28T23:30:29.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.153+0000 7f252a819640 1 --2- 172.21.15.161:0/1615930423 >> v2:172.21.15.170:3300/0 conn(0x7f2524158860 0x7f2524158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:29.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.154+0000 7f2529817640 1 --2- 172.21.15.161:0/1615930423 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2524072fe0 0x7f25240c8340 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2524073670 tx=0x7f252002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d247c7cb84cf3e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:29.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.154+0000 7f2529016640 1 -- 172.21.15.161:0/1615930423 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2520035020 con 0x7f2524072fe0 2024-03-28T23:30:29.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.154+0000 7f2529016640 1 -- 172.21.15.161:0/1615930423 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2520031830 con 0x7f2524072fe0 2024-03-28T23:30:29.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.154+0000 7f2529016640 1 -- 172.21.15.161:0/1615930423 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2520031c00 con 0x7f2524072fe0 2024-03-28T23:30:29.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.155+0000 7f252c27c640 1 -- 172.21.15.161:0/1615930423 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2524072fe0 msgr2=0x7f25240c8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:29.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.155+0000 7f252c27c640 1 --2- 172.21.15.161:0/1615930423 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2524072fe0 0x7f25240c8340 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2524073670 tx=0x7f252002b8d0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.155+0000 7f252c27c640 1 -- 172.21.15.161:0/1615930423 shutdown_connections 2024-03-28T23:30:29.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.155+0000 7f252c27c640 1 --2- 172.21.15.161:0/1615930423 >> v2:172.21.15.170:3300/0 conn(0x7f2524158860 0x7f2524158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.155+0000 7f252c27c640 1 --2- 172.21.15.161:0/1615930423 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2524072fe0 0x7f25240c8340 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.155+0000 7f252c27c640 1 -- 172.21.15.161:0/1615930423 >> 172.21.15.161:0/1615930423 conn(0x7f25240c88b0 msgr2=0x7f25240c8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:29.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.155+0000 7f252c27c640 1 -- 172.21.15.161:0/1615930423 shutdown_connections 2024-03-28T23:30:29.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.155+0000 7f252c27c640 1 -- 172.21.15.161:0/1615930423 wait complete. 2024-03-28T23:30:29.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.156+0000 7f252c27c640 1 Processor -- start 2024-03-28T23:30:29.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.156+0000 7f252c27c640 1 -- start start 2024-03-28T23:30:29.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.157+0000 7f252c27c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2524072fe0 0x7f252414ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:29.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.157+0000 7f252c27c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25240c4e40 0x7f252406d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:29.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.157+0000 7f252c27c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2524158860 0x7f25241eb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:29.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.157+0000 7f252c27c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f25240c5bd0 con 0x7f25240c4e40 2024-03-28T23:30:29.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.157+0000 7f252c27c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f25240c5a50 con 0x7f2524072fe0 2024-03-28T23:30:29.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.157+0000 7f252c27c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f25240c6d30 con 0x7f2524158860 2024-03-28T23:30:29.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.157+0000 7f2529817640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25240c4e40 0x7f252406d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:29.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.157+0000 7f2529817640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25240c4e40 0x7f252406d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51848/0 (socket says 172.21.15.161:51848) 2024-03-28T23:30:29.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.157+0000 7f2529817640 1 -- 172.21.15.161:0/2919757666 learned_addr learned my addr 172.21.15.161:0/2919757666 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:29.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.157+0000 7f252a018640 1 --2- 172.21.15.161:0/2919757666 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2524072fe0 0x7f252414ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:29.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.157+0000 7f252a819640 1 --2- 172.21.15.161:0/2919757666 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2524158860 0x7f25241eb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:29.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.157+0000 7f2529817640 1 -- 172.21.15.161:0/2919757666 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2524158860 msgr2=0x7f25241eb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:29.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.157+0000 7f2529817640 1 --2- 172.21.15.161:0/2919757666 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2524158860 0x7f25241eb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.157+0000 7f2529817640 1 -- 172.21.15.161:0/2919757666 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2524072fe0 msgr2=0x7f252414ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:29.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.157+0000 7f2529817640 1 --2- 172.21.15.161:0/2919757666 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2524072fe0 0x7f252414ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.157+0000 7f2529817640 1 -- 172.21.15.161:0/2919757666 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25241eb0e0 con 0x7f25240c4e40 2024-03-28T23:30:29.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.157+0000 7f252a018640 1 --2- 172.21.15.161:0/2919757666 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2524072fe0 0x7f252414ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:29.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.157+0000 7f2529817640 1 --2- 172.21.15.161:0/2919757666 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25240c4e40 0x7f252406d840 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2520000c00 tx=0x7f25200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:29.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.157+0000 7f2516ffd640 1 -- 172.21.15.161:0/2919757666 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2520040070 con 0x7f25240c4e40 2024-03-28T23:30:29.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.157+0000 7f2516ffd640 1 -- 172.21.15.161:0/2919757666 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2520039a20 con 0x7f25240c4e40 2024-03-28T23:30:29.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.157+0000 7f252c27c640 1 -- 172.21.15.161:0/2919757666 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f252414b1c0 con 0x7f25240c4e40 2024-03-28T23:30:29.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.157+0000 7f2516ffd640 1 -- 172.21.15.161:0/2919757666 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2520035040 con 0x7f25240c4e40 2024-03-28T23:30:29.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.157+0000 7f252c27c640 1 -- 172.21.15.161:0/2919757666 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25241ef0f0 con 0x7f25240c4e40 2024-03-28T23:30:29.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.158+0000 7f252c27c640 1 -- 172.21.15.161:0/2919757666 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f252404f4b0 con 0x7f25240c4e40 2024-03-28T23:30:29.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.159+0000 7f2516ffd640 1 -- 172.21.15.161:0/2919757666 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f252004b060 con 0x7f25240c4e40 2024-03-28T23:30:29.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.160+0000 7f2516ffd640 1 --2- 172.21.15.161:0/2919757666 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f250806b4d0 0x7f250806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:29.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.160+0000 7f2516ffd640 1 -- 172.21.15.161:0/2919757666 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f25200abf60 con 0x7f25240c4e40 2024-03-28T23:30:29.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.160+0000 7f252a018640 1 --2- 172.21.15.161:0/2919757666 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f250806b4d0 0x7f250806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:29.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.160+0000 7f252a018640 1 --2- 172.21.15.161:0/2919757666 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f250806b4d0 0x7f250806d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2518000f60 tx=0x7f2518007cc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:29.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.164+0000 7f2516ffd640 1 -- 172.21.15.161:0/2919757666 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f252007aaa0 con 0x7f25240c4e40 2024-03-28T23:30:29.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.355+0000 7f252c27c640 1 -- 172.21.15.161:0/2919757666 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f25240aeaa0 con 0x7f25240c4e40 2024-03-28T23:30:29.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.355+0000 7f2516ffd640 1 -- 172.21.15.161:0/2919757666 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f252007a3a0 con 0x7f25240c4e40 2024-03-28T23:30:29.356 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:30:29.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.357+0000 7f252c27c640 1 -- 172.21.15.161:0/2919757666 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f250806b4d0 msgr2=0x7f250806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:29.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.357+0000 7f252c27c640 1 --2- 172.21.15.161:0/2919757666 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f250806b4d0 0x7f250806d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2518000f60 tx=0x7f2518007cc0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.357+0000 7f252c27c640 1 -- 172.21.15.161:0/2919757666 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25240c4e40 msgr2=0x7f252406d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:29.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.357+0000 7f252c27c640 1 --2- 172.21.15.161:0/2919757666 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25240c4e40 0x7f252406d840 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2520000c00 tx=0x7f25200046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.357+0000 7f252c27c640 1 -- 172.21.15.161:0/2919757666 shutdown_connections 2024-03-28T23:30:29.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.357+0000 7f252c27c640 1 --2- 172.21.15.161:0/2919757666 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f250806b4d0 0x7f250806d990 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.357+0000 7f252c27c640 1 --2- 172.21.15.161:0/2919757666 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25240c4e40 0x7f252406d840 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.357+0000 7f252c27c640 1 --2- 172.21.15.161:0/2919757666 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2524158860 0x7f25241eb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.357+0000 7f252c27c640 1 --2- 172.21.15.161:0/2919757666 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2524072fe0 0x7f252414ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.357+0000 7f252c27c640 1 -- 172.21.15.161:0/2919757666 >> 172.21.15.161:0/2919757666 conn(0x7f25240c88b0 msgr2=0x7f252406fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:29.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.357+0000 7f252c27c640 1 -- 172.21.15.161:0/2919757666 shutdown_connections 2024-03-28T23:30:29.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.357+0000 7f252c27c640 1 -- 172.21.15.161:0/2919757666 wait complete. 2024-03-28T23:30:29.369 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:30:29.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.474+0000 7fb2d610f640 1 Processor -- start 2024-03-28T23:30:29.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.474+0000 7fb2d610f640 1 -- start start 2024-03-28T23:30:29.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.474+0000 7fb2d610f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb2d01529d0 0x7fb2d006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:29.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.474+0000 7fb2d610f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2d00c6750 0x7fb2d00c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:29.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.474+0000 7fb2d610f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb2d0155c30 0x7fb2d00c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:29.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.474+0000 7fb2d610f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb2d0152db0 con 0x7fb2d01529d0 2024-03-28T23:30:29.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.474+0000 7fb2d610f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fb2d00c7210 con 0x7fb2d0155c30 2024-03-28T23:30:29.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.474+0000 7fb2d610f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb2d00c6c50 con 0x7fb2d00c6750 2024-03-28T23:30:29.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.474+0000 7fb2cf7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb2d01529d0 0x7fb2d006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:29.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.475+0000 7fb2cf7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb2d01529d0 0x7fb2d006bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51850/0 (socket says 172.21.15.161:51850) 2024-03-28T23:30:29.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.475+0000 7fb2cf7fe640 1 -- 172.21.15.161:0/228853378 learned_addr learned my addr 172.21.15.161:0/228853378 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:29.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.475+0000 7fb2c7fff640 1 --2- 172.21.15.161:0/228853378 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2d00c6750 0x7fb2d00c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:29.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.475+0000 7fb2cffff640 1 --2- 172.21.15.161:0/228853378 >> v2:172.21.15.170:3300/0 conn(0x7fb2d0155c30 0x7fb2d00c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:29.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.475+0000 7fb2cf7fe640 1 -- 172.21.15.161:0/228853378 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2d00c6750 msgr2=0x7fb2d00c88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:29.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.475+0000 7fb2cf7fe640 1 --2- 172.21.15.161:0/228853378 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2d00c6750 0x7fb2d00c88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.475+0000 7fb2cf7fe640 1 -- 172.21.15.161:0/228853378 >> v2:172.21.15.170:3300/0 conn(0x7fb2d0155c30 msgr2=0x7fb2d00c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:29.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.475+0000 7fb2cf7fe640 1 --2- 172.21.15.161:0/228853378 >> v2:172.21.15.170:3300/0 conn(0x7fb2d0155c30 0x7fb2d00c54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.475+0000 7fb2cf7fe640 1 -- 172.21.15.161:0/228853378 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2d00c6f30 con 0x7fb2d01529d0 2024-03-28T23:30:29.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.476+0000 7fb2cf7fe640 1 --2- 172.21.15.161:0/228853378 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2d01529d0 0x7fb2d006bb60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb2d00749f0 tx=0x7fb2c002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b2b804a5d3afb9f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:29.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.476+0000 7fb2ceffd640 1 -- 172.21.15.161:0/228853378 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0035020 con 0x7fb2d01529d0 2024-03-28T23:30:29.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.476+0000 7fb2ceffd640 1 -- 172.21.15.161:0/228853378 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2c0031870 con 0x7fb2d01529d0 2024-03-28T23:30:29.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.476+0000 7fb2ceffd640 1 -- 172.21.15.161:0/228853378 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0031c40 con 0x7fb2d01529d0 2024-03-28T23:30:29.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.476+0000 7fb2d610f640 1 -- 172.21.15.161:0/228853378 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2d01529d0 msgr2=0x7fb2d006bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:29.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.476+0000 7fb2d610f640 1 --2- 172.21.15.161:0/228853378 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2d01529d0 0x7fb2d006bb60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb2d00749f0 tx=0x7fb2c002b910 comp rx=0 tx=0).stop 2024-03-28T23:30:29.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.477+0000 7fb2d610f640 1 -- 172.21.15.161:0/228853378 shutdown_connections 2024-03-28T23:30:29.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.477+0000 7fb2d610f640 1 --2- 172.21.15.161:0/228853378 >> v2:172.21.15.170:3300/0 conn(0x7fb2d0155c30 0x7fb2d00c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.477+0000 7fb2d610f640 1 --2- 172.21.15.161:0/228853378 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2d00c6750 0x7fb2d00c88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.477+0000 7fb2d610f640 1 --2- 172.21.15.161:0/228853378 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2d01529d0 0x7fb2d006bb60 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.477+0000 7fb2d610f640 1 -- 172.21.15.161:0/228853378 >> 172.21.15.161:0/228853378 conn(0x7fb2d00c7430 msgr2=0x7fb2d00c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:29.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.477+0000 7fb2d610f640 1 -- 172.21.15.161:0/228853378 shutdown_connections 2024-03-28T23:30:29.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.477+0000 7fb2d610f640 1 -- 172.21.15.161:0/228853378 wait complete. 2024-03-28T23:30:29.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.478+0000 7fb2d610f640 1 Processor -- start 2024-03-28T23:30:29.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.478+0000 7fb2d610f640 1 -- start start 2024-03-28T23:30:29.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.479+0000 7fb2d610f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2d00c6750 0x7fb2d01ec020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:29.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.479+0000 7fb2d610f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2d01529d0 0x7fb2d0144c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:29.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.479+0000 7fb2d610f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2d0155c30 0x7fb2d01451a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:29.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.479+0000 7fb2d610f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d00aa160 con 0x7fb2d0155c30 2024-03-28T23:30:29.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.479+0000 7fb2d610f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d0155710 con 0x7fb2d01529d0 2024-03-28T23:30:29.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.479+0000 7fb2d610f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb2d01560e0 con 0x7fb2d00c6750 2024-03-28T23:30:29.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.480+0000 7fb2cffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2d0155c30 0x7fb2d01451a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:29.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.480+0000 7fb2cffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2d0155c30 0x7fb2d01451a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51856/0 (socket says 172.21.15.161:51856) 2024-03-28T23:30:29.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.480+0000 7fb2cffff640 1 -- 172.21.15.161:0/1901553383 learned_addr learned my addr 172.21.15.161:0/1901553383 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:29.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.480+0000 7fb2c7fff640 1 --2- 172.21.15.161:0/1901553383 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2d01529d0 0x7fb2d0144c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:29.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.480+0000 7fb2cf7fe640 1 --2- 172.21.15.161:0/1901553383 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2d00c6750 0x7fb2d01ec020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:29.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.480+0000 7fb2cffff640 1 -- 172.21.15.161:0/1901553383 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2d00c6750 msgr2=0x7fb2d01ec020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:29.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.480+0000 7fb2cffff640 1 --2- 172.21.15.161:0/1901553383 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2d00c6750 0x7fb2d01ec020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.480+0000 7fb2cffff640 1 -- 172.21.15.161:0/1901553383 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2d01529d0 msgr2=0x7fb2d0144c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:29.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.480+0000 7fb2cffff640 1 --2- 172.21.15.161:0/1901553383 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2d01529d0 0x7fb2d0144c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.480+0000 7fb2cffff640 1 -- 172.21.15.161:0/1901553383 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2d01570a0 con 0x7fb2d0155c30 2024-03-28T23:30:29.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.480+0000 7fb2cffff640 1 --2- 172.21.15.161:0/1901553383 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2d0155c30 0x7fb2d01451a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8007e40 tx=0x7fb2c800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:29.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.480+0000 7fb2ccff9640 1 -- 172.21.15.161:0/1901553383 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8014620 con 0x7fb2d0155c30 2024-03-28T23:30:29.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.480+0000 7fb2ccff9640 1 -- 172.21.15.161:0/1901553383 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2c8019070 con 0x7fb2d0155c30 2024-03-28T23:30:29.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.480+0000 7fb2d610f640 1 -- 172.21.15.161:0/1901553383 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2d0158d70 con 0x7fb2d0155c30 2024-03-28T23:30:29.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.480+0000 7fb2ccff9640 1 -- 172.21.15.161:0/1901553383 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8014a20 con 0x7fb2d0155c30 2024-03-28T23:30:29.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.480+0000 7fb2d610f640 1 -- 172.21.15.161:0/1901553383 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2d0157640 con 0x7fb2d0155c30 2024-03-28T23:30:29.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.481+0000 7fb2d610f640 1 -- 172.21.15.161:0/1901553383 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2d004f4b0 con 0x7fb2d0155c30 2024-03-28T23:30:29.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.482+0000 7fb2ccff9640 1 -- 172.21.15.161:0/1901553383 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb2c8014bc0 con 0x7fb2d0155c30 2024-03-28T23:30:29.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.482+0000 7fb2ccff9640 1 --2- 172.21.15.161:0/1901553383 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb2b006b4d0 0x7fb2b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:29.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.483+0000 7fb2cf7fe640 1 --2- 172.21.15.161:0/1901553383 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb2b006b4d0 0x7fb2b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:29.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.483+0000 7fb2ccff9640 1 -- 172.21.15.161:0/1901553383 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb2c808ad20 con 0x7fb2d0155c30 2024-03-28T23:30:29.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.483+0000 7fb2cf7fe640 1 --2- 172.21.15.161:0/1901553383 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb2b006b4d0 0x7fb2b006d990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0036000 tx=0x7fb2c003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:29.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.487+0000 7fb2ccff9640 1 -- 172.21.15.161:0/1901553383 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb2c8059210 con 0x7fb2d0155c30 2024-03-28T23:30:29.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.667+0000 7fb2d610f640 1 -- 172.21.15.161:0/1901553383 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb2d014ba70 con 0x7fb2d0155c30 2024-03-28T23:30:29.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.667+0000 7fb2ccff9640 1 -- 172.21.15.161:0/1901553383 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7fb2c805d2b0 con 0x7fb2d0155c30 2024-03-28T23:30:29.668 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:30:29.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.669+0000 7fb2d610f640 1 -- 172.21.15.161:0/1901553383 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb2b006b4d0 msgr2=0x7fb2b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:29.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.669+0000 7fb2d610f640 1 --2- 172.21.15.161:0/1901553383 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb2b006b4d0 0x7fb2b006d990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0036000 tx=0x7fb2c003c040 comp rx=0 tx=0).stop 2024-03-28T23:30:29.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.669+0000 7fb2d610f640 1 -- 172.21.15.161:0/1901553383 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2d0155c30 msgr2=0x7fb2d01451a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:29.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.669+0000 7fb2d610f640 1 --2- 172.21.15.161:0/1901553383 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2d0155c30 0x7fb2d01451a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8007e40 tx=0x7fb2c800e4b0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.669+0000 7fb2d610f640 1 -- 172.21.15.161:0/1901553383 shutdown_connections 2024-03-28T23:30:29.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.669+0000 7fb2d610f640 1 --2- 172.21.15.161:0/1901553383 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb2b006b4d0 0x7fb2b006d990 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.669+0000 7fb2d610f640 1 --2- 172.21.15.161:0/1901553383 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2d0155c30 0x7fb2d01451a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.669+0000 7fb2d610f640 1 --2- 172.21.15.161:0/1901553383 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2d01529d0 0x7fb2d0144c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.669+0000 7fb2d610f640 1 --2- 172.21.15.161:0/1901553383 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2d00c6750 0x7fb2d01ec020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.669+0000 7fb2d610f640 1 -- 172.21.15.161:0/1901553383 >> 172.21.15.161:0/1901553383 conn(0x7fb2d00c7430 msgr2=0x7fb2d01495a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:29.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.669+0000 7fb2d610f640 1 -- 172.21.15.161:0/1901553383 shutdown_connections 2024-03-28T23:30:29.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.669+0000 7fb2d610f640 1 -- 172.21.15.161:0/1901553383 wait complete. 2024-03-28T23:30:29.681 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T23:30:29.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.784+0000 7f905fb20640 1 Processor -- start 2024-03-28T23:30:29.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.785+0000 7f905fb20640 1 -- start start 2024-03-28T23:30:29.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.785+0000 7f905fb20640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9058152bc0 0x7f90580c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:29.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.785+0000 7f905fb20640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90580c6f60 con 0x7f90580c8a80 2024-03-28T23:30:29.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.785+0000 7f905fb20640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90580c6120 con 0x7f90581589b0 2024-03-28T23:30:29.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.785+0000 7f905fb20640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f90580c6400 con 0x7f9058152bc0 2024-03-28T23:30:29.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.785+0000 7f905d0bb640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f90580c8a80 0x7f90580c8ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38472/0 (socket says 172.21.15.161:38472) 2024-03-28T23:30:29.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.785+0000 7f905d8bc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9058152bc0 0x7f90580c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:29.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.785+0000 7f905d0bb640 1 -- 172.21.15.161:0/3761492998 learned_addr learned my addr 172.21.15.161:0/3761492998 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:29.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.786+0000 7f905d8bc640 1 -- 172.21.15.161:0/3761492998 >> v1:172.21.15.170:6789/0 conn(0x7f90581589b0 legacy=0x7f9058158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:29.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.786+0000 7f905d8bc640 1 -- 172.21.15.161:0/3761492998 >> v1:172.21.15.161:6789/0 conn(0x7f90580c8a80 legacy=0x7f90580c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:29.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.786+0000 7f905c8ba640 1 -- 172.21.15.161:0/3761492998 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4215424628 0 0) 0x7f90580c6f60 con 0x7f90580c8a80 2024-03-28T23:30:29.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.786+0000 7f905d8bc640 1 -- 172.21.15.161:0/3761492998 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90580c7240 con 0x7f9058152bc0 2024-03-28T23:30:29.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.787+0000 7f905d8bc640 1 --2- 172.21.15.161:0/3761492998 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9058152bc0 0x7f90580c0330 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f90540097a0 tx=0x7f905402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8db33e75aefe659 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:29.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.787+0000 7f905c8ba640 1 -- 172.21.15.161:0/3761492998 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9054034070 con 0x7f9058152bc0 2024-03-28T23:30:29.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.787+0000 7f905c8ba640 1 -- 172.21.15.161:0/3761492998 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9054030610 con 0x7f9058152bc0 2024-03-28T23:30:29.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.787+0000 7f905c8ba640 1 -- 172.21.15.161:0/3761492998 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90540309e0 con 0x7f9058152bc0 2024-03-28T23:30:29.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.787+0000 7f905fb20640 1 -- 172.21.15.161:0/3761492998 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9058152bc0 msgr2=0x7f90580c0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:29.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.787+0000 7f905fb20640 1 --2- 172.21.15.161:0/3761492998 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9058152bc0 0x7f90580c0330 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f90540097a0 tx=0x7f905402b960 comp rx=0 tx=0).stop 2024-03-28T23:30:29.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.787+0000 7f905fb20640 1 -- 172.21.15.161:0/3761492998 shutdown_connections 2024-03-28T23:30:29.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.787+0000 7f905fb20640 1 --2- 172.21.15.161:0/3761492998 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9058152bc0 0x7f90580c0330 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.787+0000 7f905fb20640 1 -- 172.21.15.161:0/3761492998 >> 172.21.15.161:0/3761492998 conn(0x7f90580caa80 msgr2=0x7f90580cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:29.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.787+0000 7f905fb20640 1 -- 172.21.15.161:0/3761492998 shutdown_connections 2024-03-28T23:30:29.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.787+0000 7f905fb20640 1 -- 172.21.15.161:0/3761492998 wait complete. 2024-03-28T23:30:29.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.788+0000 7f905fb20640 1 Processor -- start 2024-03-28T23:30:29.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.788+0000 7f905fb20640 1 -- start start 2024-03-28T23:30:29.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.789+0000 7f905fb20640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90580c8a80 0x7f9058157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:29.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.789+0000 7f905fb20640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9058152bc0 0x7f90581411f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:29.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.789+0000 7f905fb20640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90581589b0 0x7f9058141730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:29.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.789+0000 7f905fb20640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9058152fa0 con 0x7f9058152bc0 2024-03-28T23:30:29.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.789+0000 7f905fb20640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f90580c95f0 con 0x7f90581589b0 2024-03-28T23:30:29.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.789+0000 7f905fb20640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9058141c70 con 0x7f90580c8a80 2024-03-28T23:30:29.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.789+0000 7f905d0bb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9058152bc0 0x7f90581411f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:29.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.789+0000 7f905d0bb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9058152bc0 0x7f90581411f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51868/0 (socket says 172.21.15.161:51868) 2024-03-28T23:30:29.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.789+0000 7f905d0bb640 1 -- 172.21.15.161:0/3480829635 learned_addr learned my addr 172.21.15.161:0/3480829635 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:29.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.789+0000 7f905d8bc640 1 --2- 172.21.15.161:0/3480829635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90580c8a80 0x7f9058157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:29.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.789+0000 7f905e0bd640 1 --2- 172.21.15.161:0/3480829635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90581589b0 0x7f9058141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:29.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.790+0000 7f905d0bb640 1 -- 172.21.15.161:0/3480829635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90580c8a80 msgr2=0x7f9058157ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:29.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.790+0000 7f905d0bb640 1 --2- 172.21.15.161:0/3480829635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90580c8a80 0x7f9058157ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.790+0000 7f905d0bb640 1 -- 172.21.15.161:0/3480829635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90581589b0 msgr2=0x7f9058141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:29.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.790+0000 7f905d0bb640 1 --2- 172.21.15.161:0/3480829635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90581589b0 0x7f9058141730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.790+0000 7f905d0bb640 1 -- 172.21.15.161:0/3480829635 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90580ab780 con 0x7f9058152bc0 2024-03-28T23:30:29.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.790+0000 7f905d0bb640 1 --2- 172.21.15.161:0/3480829635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9058152bc0 0x7f90581411f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f904800c4d0 tx=0x7f904800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:29.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.790+0000 7f904e7fc640 1 -- 172.21.15.161:0/3480829635 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9048018070 con 0x7f9058152bc0 2024-03-28T23:30:29.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.790+0000 7f904e7fc640 1 -- 172.21.15.161:0/3480829635 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90480074e0 con 0x7f9058152bc0 2024-03-28T23:30:29.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.790+0000 7f905fb20640 1 -- 172.21.15.161:0/3480829635 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90580aba60 con 0x7f9058152bc0 2024-03-28T23:30:29.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.790+0000 7f904e7fc640 1 -- 172.21.15.161:0/3480829635 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9048004a20 con 0x7f9058152bc0 2024-03-28T23:30:29.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.790+0000 7f905fb20640 1 -- 172.21.15.161:0/3480829635 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90580ac020 con 0x7f9058152bc0 2024-03-28T23:30:29.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.791+0000 7f905fb20640 1 -- 172.21.15.161:0/3480829635 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9058159550 con 0x7f9058152bc0 2024-03-28T23:30:29.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.791+0000 7f904e7fc640 1 -- 172.21.15.161:0/3480829635 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f904801b070 con 0x7f9058152bc0 2024-03-28T23:30:29.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.792+0000 7f904e7fc640 1 --2- 172.21.15.161:0/3480829635 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f903c06b500 0x7f903c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:29.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.792+0000 7f904e7fc640 1 -- 172.21.15.161:0/3480829635 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f90480897d0 con 0x7f9058152bc0 2024-03-28T23:30:29.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.792+0000 7f905d8bc640 1 --2- 172.21.15.161:0/3480829635 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f903c06b500 0x7f903c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:29.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.792+0000 7f905d8bc640 1 --2- 172.21.15.161:0/3480829635 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f903c06b500 0x7f903c06d9c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9054036000 tx=0x7f905403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:29.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.796+0000 7f904e7fc640 1 -- 172.21.15.161:0/3480829635 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9058159550 con 0x7f9058152bc0 2024-03-28T23:30:29.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.974+0000 7f905fb20640 1 -- 172.21.15.161:0/3480829635 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9058067af0 con 0x7f9058152bc0 2024-03-28T23:30:29.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.975+0000 7f904e7fc640 1 -- 172.21.15.161:0/3480829635 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f9058067af0 con 0x7f9058152bc0 2024-03-28T23:30:29.976 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:30:29.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.976+0000 7f905fb20640 1 -- 172.21.15.161:0/3480829635 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f903c06b500 msgr2=0x7f903c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:29.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.976+0000 7f905fb20640 1 --2- 172.21.15.161:0/3480829635 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f903c06b500 0x7f903c06d9c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9054036000 tx=0x7f905403b040 comp rx=0 tx=0).stop 2024-03-28T23:30:29.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.976+0000 7f905fb20640 1 -- 172.21.15.161:0/3480829635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9058152bc0 msgr2=0x7f90581411f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:29.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.976+0000 7f905fb20640 1 --2- 172.21.15.161:0/3480829635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9058152bc0 0x7f90581411f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f904800c4d0 tx=0x7f904800c990 comp rx=0 tx=0).stop 2024-03-28T23:30:29.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.977+0000 7f905fb20640 1 -- 172.21.15.161:0/3480829635 shutdown_connections 2024-03-28T23:30:29.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.977+0000 7f905fb20640 1 --2- 172.21.15.161:0/3480829635 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f903c06b500 0x7f903c06d9c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.977+0000 7f905fb20640 1 --2- 172.21.15.161:0/3480829635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9058152bc0 0x7f90581411f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.977+0000 7f905fb20640 1 --2- 172.21.15.161:0/3480829635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90581589b0 0x7f9058141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.977+0000 7f905fb20640 1 --2- 172.21.15.161:0/3480829635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90580c8a80 0x7f9058157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:29.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.977+0000 7f905fb20640 1 -- 172.21.15.161:0/3480829635 >> 172.21.15.161:0/3480829635 conn(0x7f90580caa80 msgr2=0x7f90580c5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:29.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.977+0000 7f905fb20640 1 -- 172.21.15.161:0/3480829635 shutdown_connections 2024-03-28T23:30:29.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:29.977+0000 7f905fb20640 1 -- 172.21.15.161:0/3480829635 wait complete. 2024-03-28T23:30:29.989 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:30:30.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.089+0000 7f6d9a16c640 1 Processor -- start 2024-03-28T23:30:30.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.089+0000 7f6d9a16c640 1 -- start start 2024-03-28T23:30:30.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.089+0000 7f6d9a16c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6d94152890 0x7f6d940c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:30.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.089+0000 7f6d9a16c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d940c8ed0 0x7f6d940c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:30.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.089+0000 7f6d9a16c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d94158a30 con 0x7f6d940c5950 2024-03-28T23:30:30.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.089+0000 7f6d9a16c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6d940c8480 con 0x7f6d94152890 2024-03-28T23:30:30.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.089+0000 7f6d9a16c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6d94159640 con 0x7f6d940c8ed0 2024-03-28T23:30:30.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.090+0000 7f6d937fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6d94152890 0x7f6d940c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:30.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.090+0000 7f6d93fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6d940c5950 0x7f6d941553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38488/0 (socket says 172.21.15.161:38488) 2024-03-28T23:30:30.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.090+0000 7f6d93fff640 1 -- 172.21.15.161:0/1267514546 learned_addr learned my addr 172.21.15.161:0/1267514546 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:30.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.090+0000 7f6d92ffd640 1 --2- 172.21.15.161:0/1267514546 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d940c8ed0 0x7f6d940c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:30.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.090+0000 7f6d937fe640 1 -- 172.21.15.161:0/1267514546 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d940c8ed0 msgr2=0x7f6d940c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:30.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.090+0000 7f6d937fe640 1 --2- 172.21.15.161:0/1267514546 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d940c8ed0 0x7f6d940c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.090+0000 7f6d937fe640 1 -- 172.21.15.161:0/1267514546 >> v1:172.21.15.161:6789/0 conn(0x7f6d940c5950 legacy=0x7f6d941553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:30.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.090+0000 7f6d927fc640 1 -- 172.21.15.161:0/1267514546 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2754463988 0 0) 0x7f6d94158a30 con 0x7f6d940c5950 2024-03-28T23:30:30.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.090+0000 7f6d937fe640 1 -- 172.21.15.161:0/1267514546 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d94158d10 con 0x7f6d94152890 2024-03-28T23:30:30.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.090+0000 7f6d92ffd640 1 --2- 172.21.15.161:0/1267514546 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d940c8ed0 0x7f6d940c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:30.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.091+0000 7f6d937fe640 1 --2- 172.21.15.161:0/1267514546 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d94152890 0x7f6d940c8990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6d840097a0 tx=0x7f6d8402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1836f82ca0e31e80 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:30.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.091+0000 7f6d927fc640 1 -- 172.21.15.161:0/1267514546 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d84038070 con 0x7f6d94152890 2024-03-28T23:30:30.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.091+0000 7f6d927fc640 1 -- 172.21.15.161:0/1267514546 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d84031700 con 0x7f6d94152890 2024-03-28T23:30:30.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.091+0000 7f6d927fc640 1 -- 172.21.15.161:0/1267514546 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d84031ad0 con 0x7f6d94152890 2024-03-28T23:30:30.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.092+0000 7f6d9a16c640 1 -- 172.21.15.161:0/1267514546 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d94152890 msgr2=0x7f6d940c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:30.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.092+0000 7f6d9a16c640 1 --2- 172.21.15.161:0/1267514546 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d94152890 0x7f6d940c8990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6d840097a0 tx=0x7f6d8402b7a0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.092+0000 7f6d9a16c640 1 -- 172.21.15.161:0/1267514546 shutdown_connections 2024-03-28T23:30:30.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.092+0000 7f6d9a16c640 1 --2- 172.21.15.161:0/1267514546 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d940c8ed0 0x7f6d940c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.092+0000 7f6d9a16c640 1 --2- 172.21.15.161:0/1267514546 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d94152890 0x7f6d940c8990 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.092+0000 7f6d9a16c640 1 -- 172.21.15.161:0/1267514546 >> 172.21.15.161:0/1267514546 conn(0x7f6d940ac620 msgr2=0x7f6d940aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:30.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.092+0000 7f6d9a16c640 1 -- 172.21.15.161:0/1267514546 shutdown_connections 2024-03-28T23:30:30.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.092+0000 7f6d9a16c640 1 -- 172.21.15.161:0/1267514546 wait complete. 2024-03-28T23:30:30.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.093+0000 7f6d9a16c640 1 Processor -- start 2024-03-28T23:30:30.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.093+0000 7f6d9a16c640 1 -- start start 2024-03-28T23:30:30.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.094+0000 7f6d9a16c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d940c5950 0x7f6d94153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:30.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.094+0000 7f6d9a16c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d940c8ed0 0x7f6d9406c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:30.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.094+0000 7f6d9a16c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d94152890 0x7f6d941ee560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:30.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.094+0000 7f6d9a16c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6d94152c70 con 0x7f6d94152890 2024-03-28T23:30:30.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.094+0000 7f6d9a16c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6d941592d0 con 0x7f6d940c8ed0 2024-03-28T23:30:30.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.094+0000 7f6d9a16c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6d94152df0 con 0x7f6d940c5950 2024-03-28T23:30:30.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.094+0000 7f6d93fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d94152890 0x7f6d941ee560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:30.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.094+0000 7f6d93fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d94152890 0x7f6d941ee560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51890/0 (socket says 172.21.15.161:51890) 2024-03-28T23:30:30.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.094+0000 7f6d93fff640 1 -- 172.21.15.161:0/3399770349 learned_addr learned my addr 172.21.15.161:0/3399770349 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:30.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.094+0000 7f6d937fe640 1 --2- 172.21.15.161:0/3399770349 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d940c5950 0x7f6d94153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:30.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.094+0000 7f6d92ffd640 1 --2- 172.21.15.161:0/3399770349 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d940c8ed0 0x7f6d9406c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:30.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.094+0000 7f6d93fff640 1 -- 172.21.15.161:0/3399770349 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d940c5950 msgr2=0x7f6d94153210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:30.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.094+0000 7f6d93fff640 1 --2- 172.21.15.161:0/3399770349 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d940c5950 0x7f6d94153210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.094+0000 7f6d93fff640 1 -- 172.21.15.161:0/3399770349 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d940c8ed0 msgr2=0x7f6d9406c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:30.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.094+0000 7f6d93fff640 1 --2- 172.21.15.161:0/3399770349 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d940c8ed0 0x7f6d9406c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.094+0000 7f6d93fff640 1 -- 172.21.15.161:0/3399770349 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d941eb820 con 0x7f6d94152890 2024-03-28T23:30:30.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.094+0000 7f6d92ffd640 1 --2- 172.21.15.161:0/3399770349 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d940c8ed0 0x7f6d9406c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:30.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.094+0000 7f6d93fff640 1 --2- 172.21.15.161:0/3399770349 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d94152890 0x7f6d941ee560 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c00c730 tx=0x7f6d8c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:30.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.094+0000 7f6d7bfff640 1 -- 172.21.15.161:0/3399770349 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c018070 con 0x7f6d94152890 2024-03-28T23:30:30.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.094+0000 7f6d7bfff640 1 -- 172.21.15.161:0/3399770349 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d8c00ce40 con 0x7f6d94152890 2024-03-28T23:30:30.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.094+0000 7f6d9a16c640 1 -- 172.21.15.161:0/3399770349 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d9414d740 con 0x7f6d94152890 2024-03-28T23:30:30.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.094+0000 7f6d7bfff640 1 -- 172.21.15.161:0/3399770349 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c00a860 con 0x7f6d94152890 2024-03-28T23:30:30.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.094+0000 7f6d9a16c640 1 -- 172.21.15.161:0/3399770349 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d941ee040 con 0x7f6d94152890 2024-03-28T23:30:30.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.095+0000 7f6d9a16c640 1 -- 172.21.15.161:0/3399770349 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d9404f4b0 con 0x7f6d94152890 2024-03-28T23:30:30.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.096+0000 7f6d7bfff640 1 -- 172.21.15.161:0/3399770349 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6d8c01b070 con 0x7f6d94152890 2024-03-28T23:30:30.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.096+0000 7f6d7bfff640 1 --2- 172.21.15.161:0/3399770349 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6d8006b4d0 0x7f6d8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:30.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.096+0000 7f6d937fe640 1 --2- 172.21.15.161:0/3399770349 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6d8006b4d0 0x7f6d8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:30.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.097+0000 7f6d7bfff640 1 -- 172.21.15.161:0/3399770349 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6d8c089960 con 0x7f6d94152890 2024-03-28T23:30:30.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.097+0000 7f6d937fe640 1 --2- 172.21.15.161:0/3399770349 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6d8006b4d0 0x7f6d8006d990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6d84035000 tx=0x7f6d84008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:30.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.101+0000 7f6d7bfff640 1 -- 172.21.15.161:0/3399770349 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6d8c057ee0 con 0x7f6d94152890 2024-03-28T23:30:30.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.280+0000 7f6d9a16c640 1 -- 172.21.15.161:0/3399770349 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6d940ac400 con 0x7f6d94152890 2024-03-28T23:30:30.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.281+0000 7f6d7bfff640 1 -- 172.21.15.161:0/3399770349 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7f6d8c05bf80 con 0x7f6d94152890 2024-03-28T23:30:30.283 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:30.283 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":24,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:29:50.946843+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:30:30.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.284+0000 7f6d9a16c640 1 -- 172.21.15.161:0/3399770349 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6d8006b4d0 msgr2=0x7f6d8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:30.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.284+0000 7f6d9a16c640 1 --2- 172.21.15.161:0/3399770349 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6d8006b4d0 0x7f6d8006d990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6d84035000 tx=0x7f6d84008030 comp rx=0 tx=0).stop 2024-03-28T23:30:30.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.284+0000 7f6d9a16c640 1 -- 172.21.15.161:0/3399770349 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d94152890 msgr2=0x7f6d941ee560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:30.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.284+0000 7f6d9a16c640 1 --2- 172.21.15.161:0/3399770349 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d94152890 0x7f6d941ee560 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c00c730 tx=0x7f6d8c00cbf0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.284+0000 7f6d9a16c640 1 -- 172.21.15.161:0/3399770349 shutdown_connections 2024-03-28T23:30:30.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.284+0000 7f6d9a16c640 1 --2- 172.21.15.161:0/3399770349 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6d8006b4d0 0x7f6d8006d990 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.284+0000 7f6d9a16c640 1 --2- 172.21.15.161:0/3399770349 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d94152890 0x7f6d941ee560 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.284+0000 7f6d9a16c640 1 --2- 172.21.15.161:0/3399770349 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d940c8ed0 0x7f6d9406c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.284+0000 7f6d9a16c640 1 --2- 172.21.15.161:0/3399770349 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d940c5950 0x7f6d94153210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.284+0000 7f6d9a16c640 1 -- 172.21.15.161:0/3399770349 >> 172.21.15.161:0/3399770349 conn(0x7f6d940ac620 msgr2=0x7f6d940c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:30.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.284+0000 7f6d9a16c640 1 -- 172.21.15.161:0/3399770349 shutdown_connections 2024-03-28T23:30:30.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.285+0000 7f6d9a16c640 1 -- 172.21.15.161:0/3399770349 wait complete. 2024-03-28T23:30:30.297 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:29:44.021375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:29:44.872102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T23:29:48.591063+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T23:30:30.297 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:30:30.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.404+0000 7f78a8156640 1 Processor -- start 2024-03-28T23:30:30.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.404+0000 7f78a8156640 1 -- start start 2024-03-28T23:30:30.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.405+0000 7f78a8156640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78a01529d0 0x7f78a006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:30.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.405+0000 7f78a8156640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f78a00c6750 0x7f78a00c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:30.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.405+0000 7f78a8156640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78a0152db0 con 0x7f78a0155c30 2024-03-28T23:30:30.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.405+0000 7f78a8156640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f78a00c7210 con 0x7f78a00c6750 2024-03-28T23:30:30.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.405+0000 7f78a8156640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f78a00c6c50 con 0x7f78a01529d0 2024-03-28T23:30:30.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.405+0000 7f78a56f1640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f78a00c6750 0x7f78a00c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:30.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.405+0000 7f78a5ef2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78a01529d0 0x7f78a006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:30.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.405+0000 7f78a56f1640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f78a00c6750 0x7f78a00c88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:50462/0 (socket says 172.21.15.161:50462) 2024-03-28T23:30:30.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.405+0000 7f78a56f1640 1 -- 172.21.15.161:0/1584020925 learned_addr learned my addr 172.21.15.161:0/1584020925 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:30.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.405+0000 7f78a66f3640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f78a0155c30 0x7f78a00c54f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:34400/0 (socket says 172.21.15.161:34400) 2024-03-28T23:30:30.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.405+0000 7f78a56f1640 1 -- 172.21.15.161:0/1584020925 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78a01529d0 msgr2=0x7f78a006bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:30.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.405+0000 7f78a56f1640 1 --2- 172.21.15.161:0/1584020925 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78a01529d0 0x7f78a006bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.405+0000 7f78a56f1640 1 -- 172.21.15.161:0/1584020925 >> v1:172.21.15.170:6789/0 conn(0x7f78a0155c30 legacy=0x7f78a00c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:30.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.405+0000 7f78a56f1640 1 -- 172.21.15.161:0/1584020925 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a00c6f30 con 0x7f78a00c6750 2024-03-28T23:30:30.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.405+0000 7f78a5ef2640 1 --2- 172.21.15.161:0/1584020925 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78a01529d0 0x7f78a006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:30.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.406+0000 7f78a56f1640 1 --2- 172.21.15.161:0/1584020925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78a00c6750 0x7f78a00c88b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7890009870 tx=0x7f789002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca0e7200cc869564 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:30.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.406+0000 7f78a4ef0640 1 -- 172.21.15.161:0/1584020925 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7890034070 con 0x7f78a00c6750 2024-03-28T23:30:30.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.406+0000 7f78a4ef0640 1 -- 172.21.15.161:0/1584020925 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7890030840 con 0x7f78a00c6750 2024-03-28T23:30:30.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.406+0000 7f78a4ef0640 1 -- 172.21.15.161:0/1584020925 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7890030c10 con 0x7f78a00c6750 2024-03-28T23:30:30.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.407+0000 7f78a8156640 1 -- 172.21.15.161:0/1584020925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78a00c6750 msgr2=0x7f78a00c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:30.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.407+0000 7f78a8156640 1 --2- 172.21.15.161:0/1584020925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78a00c6750 0x7f78a00c88b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7890009870 tx=0x7f789002b980 comp rx=0 tx=0).stop 2024-03-28T23:30:30.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.407+0000 7f78a8156640 1 -- 172.21.15.161:0/1584020925 shutdown_connections 2024-03-28T23:30:30.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.407+0000 7f78a8156640 1 --2- 172.21.15.161:0/1584020925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78a00c6750 0x7f78a00c88b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.407+0000 7f78a8156640 1 --2- 172.21.15.161:0/1584020925 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78a01529d0 0x7f78a006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.407+0000 7f78a8156640 1 -- 172.21.15.161:0/1584020925 >> 172.21.15.161:0/1584020925 conn(0x7f78a00c7430 msgr2=0x7f78a00c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:30.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.407+0000 7f78a8156640 1 -- 172.21.15.161:0/1584020925 shutdown_connections 2024-03-28T23:30:30.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.407+0000 7f78a8156640 1 -- 172.21.15.161:0/1584020925 wait complete. 2024-03-28T23:30:30.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.409+0000 7f78a8156640 1 Processor -- start 2024-03-28T23:30:30.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.409+0000 7f78a8156640 1 -- start start 2024-03-28T23:30:30.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.410+0000 7f78a8156640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f78a00c6750 0x7f78a0158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:30.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.410+0000 7f78a8156640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78a01529d0 0x7f78a014e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:30.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.410+0000 7f78a8156640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78a0155c30 0x7f78a00ad890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:30.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.410+0000 7f78a8156640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f78a00aa160 con 0x7f78a00c6750 2024-03-28T23:30:30.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.410+0000 7f78a8156640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f78a0155710 con 0x7f78a01529d0 2024-03-28T23:30:30.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.410+0000 7f78a8156640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f78a00c5c00 con 0x7f78a0155c30 2024-03-28T23:30:30.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.410+0000 7f78a5ef2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f78a00c6750 0x7f78a0158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:30.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.410+0000 7f78a5ef2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f78a00c6750 0x7f78a0158dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51892/0 (socket says 172.21.15.161:51892) 2024-03-28T23:30:30.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.410+0000 7f78a5ef2640 1 -- 172.21.15.161:0/137910821 learned_addr learned my addr 172.21.15.161:0/137910821 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:30.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.411+0000 7f78a56f1640 1 --2- 172.21.15.161:0/137910821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78a01529d0 0x7f78a014e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:30.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.411+0000 7f78a66f3640 1 --2- 172.21.15.161:0/137910821 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78a0155c30 0x7f78a00ad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:30.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.411+0000 7f78a5ef2640 1 -- 172.21.15.161:0/137910821 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78a0155c30 msgr2=0x7f78a00ad890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:30.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.411+0000 7f78a5ef2640 1 --2- 172.21.15.161:0/137910821 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78a0155c30 0x7f78a00ad890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.411+0000 7f78a5ef2640 1 -- 172.21.15.161:0/137910821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78a01529d0 msgr2=0x7f78a014e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:30.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.411+0000 7f78a5ef2640 1 --2- 172.21.15.161:0/137910821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78a01529d0 0x7f78a014e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.411+0000 7f78a5ef2640 1 -- 172.21.15.161:0/137910821 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a0159440 con 0x7f78a00c6750 2024-03-28T23:30:30.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.411+0000 7f78a56f1640 1 --2- 172.21.15.161:0/137910821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78a01529d0 0x7f78a014e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:30.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.411+0000 7f78a66f3640 1 --2- 172.21.15.161:0/137910821 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78a0155c30 0x7f78a00ad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:30.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.411+0000 7f78a5ef2640 1 --2- 172.21.15.161:0/137910821 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f78a00c6750 0x7f78a0158dd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f789c00ec10 tx=0x7f789c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:30.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.411+0000 7f78967fc640 1 -- 172.21.15.161:0/137910821 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c017070 con 0x7f78a00c6750 2024-03-28T23:30:30.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.411+0000 7f78967fc640 1 -- 172.21.15.161:0/137910821 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f789c00cc30 con 0x7f78a00c6750 2024-03-28T23:30:30.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.411+0000 7f78a8156640 1 -- 172.21.15.161:0/137910821 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78a01ee1a0 con 0x7f78a00c6750 2024-03-28T23:30:30.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.411+0000 7f78967fc640 1 -- 172.21.15.161:0/137910821 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c013470 con 0x7f78a00c6750 2024-03-28T23:30:30.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.411+0000 7f78a8156640 1 -- 172.21.15.161:0/137910821 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78a014e570 con 0x7f78a00c6750 2024-03-28T23:30:30.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.412+0000 7f78a8156640 1 -- 172.21.15.161:0/137910821 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78a004f4b0 con 0x7f78a00c6750 2024-03-28T23:30:30.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.413+0000 7f78967fc640 1 -- 172.21.15.161:0/137910821 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f789c01a070 con 0x7f78a00c6750 2024-03-28T23:30:30.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.413+0000 7f78967fc640 1 --2- 172.21.15.161:0/137910821 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f788406b4d0 0x7f788406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:30.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.413+0000 7f78967fc640 1 -- 172.21.15.161:0/137910821 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f789c08a840 con 0x7f78a00c6750 2024-03-28T23:30:30.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.413+0000 7f78a56f1640 1 --2- 172.21.15.161:0/137910821 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f788406b4d0 0x7f788406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:30.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.413+0000 7f78a56f1640 1 --2- 172.21.15.161:0/137910821 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f788406b4d0 0x7f788406d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7890036000 tx=0x7f789003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:30.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.421+0000 7f78967fc640 1 -- 172.21.15.161:0/137910821 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f789c059390 con 0x7f78a00c6750 2024-03-28T23:30:30.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.597+0000 7f78a8156640 1 -- 172.21.15.161:0/137910821 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f78a00ca650 con 0x7f78a00c6750 2024-03-28T23:30:30.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.598+0000 7f78967fc640 1 -- 172.21.15.161:0/137910821 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f789c058c90 con 0x7f78a00c6750 2024-03-28T23:30:30.599 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:30:30.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.600+0000 7f78a8156640 1 -- 172.21.15.161:0/137910821 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f788406b4d0 msgr2=0x7f788406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:30.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.600+0000 7f78a8156640 1 --2- 172.21.15.161:0/137910821 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f788406b4d0 0x7f788406d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7890036000 tx=0x7f789003b040 comp rx=0 tx=0).stop 2024-03-28T23:30:30.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.600+0000 7f78a8156640 1 -- 172.21.15.161:0/137910821 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f78a00c6750 msgr2=0x7f78a0158dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:30.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.600+0000 7f78a8156640 1 --2- 172.21.15.161:0/137910821 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f78a00c6750 0x7f78a0158dd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f789c00ec10 tx=0x7f789c00c520 comp rx=0 tx=0).stop 2024-03-28T23:30:30.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.601+0000 7f78a8156640 1 -- 172.21.15.161:0/137910821 shutdown_connections 2024-03-28T23:30:30.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.601+0000 7f78a8156640 1 --2- 172.21.15.161:0/137910821 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f788406b4d0 0x7f788406d990 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.601+0000 7f78a8156640 1 --2- 172.21.15.161:0/137910821 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78a0155c30 0x7f78a00ad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.601+0000 7f78a8156640 1 --2- 172.21.15.161:0/137910821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78a01529d0 0x7f78a014e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.601+0000 7f78a8156640 1 --2- 172.21.15.161:0/137910821 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f78a00c6750 0x7f78a0158dd0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.601+0000 7f78a8156640 1 -- 172.21.15.161:0/137910821 >> 172.21.15.161:0/137910821 conn(0x7f78a00c7430 msgr2=0x7f78a01501f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:30.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.601+0000 7f78a8156640 1 -- 172.21.15.161:0/137910821 shutdown_connections 2024-03-28T23:30:30.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.601+0000 7f78a8156640 1 -- 172.21.15.161:0/137910821 wait complete. 2024-03-28T23:30:30.613 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:30:30.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.723+0000 7fa465f7a640 1 Processor -- start 2024-03-28T23:30:30.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.723+0000 7fa465f7a640 1 -- start start 2024-03-28T23:30:30.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.723+0000 7fa465f7a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa4600a9960 0x7fa460072970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:30.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.723+0000 7fa465f7a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4600abc00 con 0x7fa4600733e0 2024-03-28T23:30:30.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.723+0000 7fa465f7a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa46006bf70 con 0x7fa4600c7330 2024-03-28T23:30:30.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.723+0000 7fa465f7a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa4600aa3a0 con 0x7fa4600a9960 2024-03-28T23:30:30.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.724+0000 7fa45ffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa4600733e0 0x7fa46014eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60058/0 (socket says 172.21.15.161:60058) 2024-03-28T23:30:30.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.724+0000 7fa45ffff640 1 -- 172.21.15.161:0/2836425994 learned_addr learned my addr 172.21.15.161:0/2836425994 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:30.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.724+0000 7fa45f7fe640 1 --2- 172.21.15.161:0/2836425994 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa4600a9960 0x7fa460072970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:30.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.724+0000 7fa45e7fc640 1 -- 172.21.15.161:0/2836425994 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1107875825 0 0) 0x7fa4600abc00 con 0x7fa4600733e0 2024-03-28T23:30:30.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.724+0000 7fa45e7fc640 1 -- 172.21.15.161:0/2836425994 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa448003520 con 0x7fa4600733e0 2024-03-28T23:30:30.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.724+0000 7fa45e7fc640 1 -- 172.21.15.161:0/2836425994 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1203505495 0 0) 0x7fa46006bf70 con 0x7fa4600c7330 2024-03-28T23:30:30.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.724+0000 7fa45e7fc640 1 -- 172.21.15.161:0/2836425994 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4600abc00 con 0x7fa4600c7330 2024-03-28T23:30:30.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.724+0000 7fa45f7fe640 1 -- 172.21.15.161:0/2836425994 >> v1:172.21.15.170:6789/0 conn(0x7fa4600c7330 legacy=0x7fa46014e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:30.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.724+0000 7fa45f7fe640 1 -- 172.21.15.161:0/2836425994 >> v1:172.21.15.161:6789/0 conn(0x7fa4600733e0 legacy=0x7fa46014eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:30.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.724+0000 7fa45f7fe640 1 -- 172.21.15.161:0/2836425994 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa46006bcd0 con 0x7fa4600a9960 2024-03-28T23:30:30.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.725+0000 7fa45f7fe640 1 --2- 172.21.15.161:0/2836425994 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa4600a9960 0x7fa460072970 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa4500097d0 tx=0x7fa45002b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ee9dcf6dbad083b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:30.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.725+0000 7fa45e7fc640 1 -- 172.21.15.161:0/2836425994 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa450031770 con 0x7fa4600a9960 2024-03-28T23:30:30.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.725+0000 7fa45e7fc640 1 -- 172.21.15.161:0/2836425994 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa450037070 con 0x7fa4600a9960 2024-03-28T23:30:30.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.725+0000 7fa45e7fc640 1 -- 172.21.15.161:0/2836425994 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa450031b10 con 0x7fa4600a9960 2024-03-28T23:30:30.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.726+0000 7fa465f7a640 1 -- 172.21.15.161:0/2836425994 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa4600a9960 msgr2=0x7fa460072970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:30.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.726+0000 7fa465f7a640 1 --2- 172.21.15.161:0/2836425994 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa4600a9960 0x7fa460072970 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa4500097d0 tx=0x7fa45002b810 comp rx=0 tx=0).stop 2024-03-28T23:30:30.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.726+0000 7fa465f7a640 1 -- 172.21.15.161:0/2836425994 shutdown_connections 2024-03-28T23:30:30.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.726+0000 7fa465f7a640 1 --2- 172.21.15.161:0/2836425994 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa4600a9960 0x7fa460072970 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.726+0000 7fa465f7a640 1 -- 172.21.15.161:0/2836425994 >> 172.21.15.161:0/2836425994 conn(0x7fa4600a83f0 msgr2=0x7fa4600a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:30.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.726+0000 7fa465f7a640 1 -- 172.21.15.161:0/2836425994 shutdown_connections 2024-03-28T23:30:30.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.726+0000 7fa465f7a640 1 -- 172.21.15.161:0/2836425994 wait complete. 2024-03-28T23:30:30.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.728+0000 7fa465f7a640 1 Processor -- start 2024-03-28T23:30:30.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.728+0000 7fa465f7a640 1 -- start start 2024-03-28T23:30:30.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.729+0000 7fa465f7a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4600733e0 0x7fa4600bbf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:30.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.729+0000 7fa465f7a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4600a9960 0x7fa4601e71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:30.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.729+0000 7fa465f7a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa4600c7330 0x7fa4601e9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:30.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.729+0000 7fa465f7a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa4600a9d40 con 0x7fa4600a9960 2024-03-28T23:30:30.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.729+0000 7fa465f7a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa460152250 con 0x7fa4600733e0 2024-03-28T23:30:30.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.729+0000 7fa465f7a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa4600a9ec0 con 0x7fa4600c7330 2024-03-28T23:30:30.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.729+0000 7fa45effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4600a9960 0x7fa4601e71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:30.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.729+0000 7fa45effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4600a9960 0x7fa4601e71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48860/0 (socket says 172.21.15.161:48860) 2024-03-28T23:30:30.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.729+0000 7fa45effd640 1 -- 172.21.15.161:0/1330953878 learned_addr learned my addr 172.21.15.161:0/1330953878 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:30.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.729+0000 7fa45ffff640 1 --2- 172.21.15.161:0/1330953878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa4600c7330 0x7fa4601e9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:30.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.729+0000 7fa45f7fe640 1 --2- 172.21.15.161:0/1330953878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4600733e0 0x7fa4600bbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:30.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.729+0000 7fa45effd640 1 -- 172.21.15.161:0/1330953878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa4600c7330 msgr2=0x7fa4601e9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:30.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.729+0000 7fa45effd640 1 --2- 172.21.15.161:0/1330953878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa4600c7330 0x7fa4601e9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.729+0000 7fa45effd640 1 -- 172.21.15.161:0/1330953878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4600733e0 msgr2=0x7fa4600bbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:30.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.729+0000 7fa45effd640 1 --2- 172.21.15.161:0/1330953878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4600733e0 0x7fa4600bbf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.729+0000 7fa45effd640 1 -- 172.21.15.161:0/1330953878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4600bfe00 con 0x7fa4600a9960 2024-03-28T23:30:30.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.730+0000 7fa45effd640 1 --2- 172.21.15.161:0/1330953878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4600a9960 0x7fa4601e71b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa45400c7a0 tx=0x7fa45400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:30.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.730+0000 7fa447fff640 1 -- 172.21.15.161:0/1330953878 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa454003b70 con 0x7fa4600a9960 2024-03-28T23:30:30.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.730+0000 7fa447fff640 1 -- 172.21.15.161:0/1330953878 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa454003d30 con 0x7fa4600a9960 2024-03-28T23:30:30.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.730+0000 7fa465f7a640 1 -- 172.21.15.161:0/1330953878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4601e69d0 con 0x7fa4600a9960 2024-03-28T23:30:30.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.730+0000 7fa447fff640 1 -- 172.21.15.161:0/1330953878 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45400aa30 con 0x7fa4600a9960 2024-03-28T23:30:30.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.730+0000 7fa465f7a640 1 -- 172.21.15.161:0/1330953878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4600c05d0 con 0x7fa4600a9960 2024-03-28T23:30:30.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.731+0000 7fa465f7a640 1 -- 172.21.15.161:0/1330953878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa46004f4b0 con 0x7fa4600a9960 2024-03-28T23:30:30.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.731+0000 7fa447fff640 1 -- 172.21.15.161:0/1330953878 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa45400abd0 con 0x7fa4600a9960 2024-03-28T23:30:30.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.732+0000 7fa447fff640 1 --2- 172.21.15.161:0/1330953878 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa44c06b4d0 0x7fa44c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:30.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.732+0000 7fa45f7fe640 1 --2- 172.21.15.161:0/1330953878 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa44c06b4d0 0x7fa44c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:30.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.732+0000 7fa447fff640 1 -- 172.21.15.161:0/1330953878 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa45408a340 con 0x7fa4600a9960 2024-03-28T23:30:30.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.732+0000 7fa45f7fe640 1 --2- 172.21.15.161:0/1330953878 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa44c06b4d0 0x7fa44c06d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa450036000 tx=0x7fa45003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:30.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.736+0000 7fa447fff640 1 -- 172.21.15.161:0/1330953878 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa454058840 con 0x7fa4600a9960 2024-03-28T23:30:30.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.914+0000 7fa465f7a640 1 -- 172.21.15.161:0/1330953878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa46006b1b0 con 0x7fa4600a9960 2024-03-28T23:30:30.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.914+0000 7fa447fff640 1 -- 172.21.15.161:0/1330953878 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7fa45405c8e0 con 0x7fa4600a9960 2024-03-28T23:30:30.915 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:30:30.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.916+0000 7fa465f7a640 1 -- 172.21.15.161:0/1330953878 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa44c06b4d0 msgr2=0x7fa44c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:30.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.916+0000 7fa465f7a640 1 --2- 172.21.15.161:0/1330953878 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa44c06b4d0 0x7fa44c06d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa450036000 tx=0x7fa45003d040 comp rx=0 tx=0).stop 2024-03-28T23:30:30.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.916+0000 7fa465f7a640 1 -- 172.21.15.161:0/1330953878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4600a9960 msgr2=0x7fa4601e71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:30.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.916+0000 7fa465f7a640 1 --2- 172.21.15.161:0/1330953878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4600a9960 0x7fa4601e71b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa45400c7a0 tx=0x7fa45400cc60 comp rx=0 tx=0).stop 2024-03-28T23:30:30.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.916+0000 7fa465f7a640 1 -- 172.21.15.161:0/1330953878 shutdown_connections 2024-03-28T23:30:30.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.916+0000 7fa465f7a640 1 --2- 172.21.15.161:0/1330953878 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa44c06b4d0 0x7fa44c06d990 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.916+0000 7fa465f7a640 1 --2- 172.21.15.161:0/1330953878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4600a9960 0x7fa4601e71b0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.916+0000 7fa465f7a640 1 --2- 172.21.15.161:0/1330953878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa4600c7330 0x7fa4601e9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.916+0000 7fa465f7a640 1 --2- 172.21.15.161:0/1330953878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4600733e0 0x7fa4600bbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:30.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.916+0000 7fa465f7a640 1 -- 172.21.15.161:0/1330953878 >> 172.21.15.161:0/1330953878 conn(0x7fa4600a83f0 msgr2=0x7fa46014dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:30.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.916+0000 7fa465f7a640 1 -- 172.21.15.161:0/1330953878 shutdown_connections 2024-03-28T23:30:30.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:30.916+0000 7fa465f7a640 1 -- 172.21.15.161:0/1330953878 wait complete. 2024-03-28T23:30:30.928 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:30:31.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.032+0000 7f9ef536c640 1 Processor -- start 2024-03-28T23:30:31.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.032+0000 7f9ef536c640 1 -- start start 2024-03-28T23:30:31.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.032+0000 7f9ef536c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ef01529d0 0x7f9ef006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:31.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.032+0000 7f9ef536c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9ef00c6750 0x7f9ef00c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:31.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.032+0000 7f9ef536c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ef0152db0 con 0x7f9ef0155c30 2024-03-28T23:30:31.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.032+0000 7f9ef536c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f9ef00c7210 con 0x7f9ef00c6750 2024-03-28T23:30:31.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.032+0000 7f9ef536c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9ef00c6c50 con 0x7f9ef01529d0 2024-03-28T23:30:31.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.033+0000 7f9eef7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f9ef0155c30 0x7f9ef00c54f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60060/0 (socket says 172.21.15.161:60060) 2024-03-28T23:30:31.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.033+0000 7f9eef7fe640 1 -- 172.21.15.161:0/1283821442 learned_addr learned my addr 172.21.15.161:0/1283821442 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:31.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.033+0000 7f9ee67fc640 1 --2- 172.21.15.161:0/1283821442 >> v2:172.21.15.170:3300/0 conn(0x7f9ef00c6750 0x7f9ef00c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:31.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.033+0000 7f9eeeffd640 1 --2- 172.21.15.161:0/1283821442 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ef01529d0 0x7f9ef006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:31.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.033+0000 7f9eee7fc640 1 -- 172.21.15.161:0/1283821442 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 473253478 0 0) 0x7f9ef0152db0 con 0x7f9ef0155c30 2024-03-28T23:30:31.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.033+0000 7f9eee7fc640 1 -- 172.21.15.161:0/1283821442 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9edc003520 con 0x7f9ef0155c30 2024-03-28T23:30:31.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.033+0000 7f9ee67fc640 1 -- 172.21.15.161:0/1283821442 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ef01529d0 msgr2=0x7f9ef006bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:31.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.033+0000 7f9ee67fc640 1 --2- 172.21.15.161:0/1283821442 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ef01529d0 0x7f9ef006bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.033+0000 7f9ee67fc640 1 -- 172.21.15.161:0/1283821442 >> v1:172.21.15.161:6789/0 conn(0x7f9ef0155c30 legacy=0x7f9ef00c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:31.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.033+0000 7f9ee67fc640 1 -- 172.21.15.161:0/1283821442 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ef00c6f30 con 0x7f9ef00c6750 2024-03-28T23:30:31.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.033+0000 7f9eeeffd640 1 --2- 172.21.15.161:0/1283821442 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ef01529d0 0x7f9ef006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:31.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.034+0000 7f9ee67fc640 1 --2- 172.21.15.161:0/1283821442 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ef00c6750 0x7f9ef00c88b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9ed80097a0 tx=0x7f9ed802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff0fa7f03d7a74c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:31.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.034+0000 7f9eee7fc640 1 -- 172.21.15.161:0/1283821442 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed8035020 con 0x7f9ef00c6750 2024-03-28T23:30:31.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.034+0000 7f9eee7fc640 1 -- 172.21.15.161:0/1283821442 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ed8031830 con 0x7f9ef00c6750 2024-03-28T23:30:31.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.034+0000 7f9eee7fc640 1 -- 172.21.15.161:0/1283821442 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed8031be0 con 0x7f9ef00c6750 2024-03-28T23:30:31.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.035+0000 7f9ef536c640 1 -- 172.21.15.161:0/1283821442 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ef00c6750 msgr2=0x7f9ef00c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:31.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.035+0000 7f9ef536c640 1 --2- 172.21.15.161:0/1283821442 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ef00c6750 0x7f9ef00c88b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9ed80097a0 tx=0x7f9ed802b8d0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.035+0000 7f9ef536c640 1 -- 172.21.15.161:0/1283821442 shutdown_connections 2024-03-28T23:30:31.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.035+0000 7f9ef536c640 1 --2- 172.21.15.161:0/1283821442 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ef00c6750 0x7f9ef00c88b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.035+0000 7f9ef536c640 1 --2- 172.21.15.161:0/1283821442 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ef01529d0 0x7f9ef006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.035+0000 7f9ef536c640 1 -- 172.21.15.161:0/1283821442 >> 172.21.15.161:0/1283821442 conn(0x7f9ef00c7430 msgr2=0x7f9ef00c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:31.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.035+0000 7f9ef536c640 1 -- 172.21.15.161:0/1283821442 shutdown_connections 2024-03-28T23:30:31.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.035+0000 7f9ef536c640 1 -- 172.21.15.161:0/1283821442 wait complete. 2024-03-28T23:30:31.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.036+0000 7f9ef536c640 1 Processor -- start 2024-03-28T23:30:31.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.036+0000 7f9ef536c640 1 -- start start 2024-03-28T23:30:31.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.037+0000 7f9ef536c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ef00c6750 0x7f9ef00adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:31.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.037+0000 7f9ef536c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ef01529d0 0x7f9ef00af960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:31.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.037+0000 7f9ef536c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ef0155c30 0x7f9ef00afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:31.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.037+0000 7f9ef536c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef00aa160 con 0x7f9ef00c6750 2024-03-28T23:30:31.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.037+0000 7f9ef536c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef0155710 con 0x7f9ef0155c30 2024-03-28T23:30:31.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.037+0000 7f9ef536c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9ef00c5c00 con 0x7f9ef01529d0 2024-03-28T23:30:31.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.037+0000 7f9eeeffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ef00c6750 0x7f9ef00adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:31.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.037+0000 7f9eeeffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ef00c6750 0x7f9ef00adb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48864/0 (socket says 172.21.15.161:48864) 2024-03-28T23:30:31.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.037+0000 7f9eeeffd640 1 -- 172.21.15.161:0/1411272708 learned_addr learned my addr 172.21.15.161:0/1411272708 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:31.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.037+0000 7f9eef7fe640 1 --2- 172.21.15.161:0/1411272708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ef0155c30 0x7f9ef00afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:31.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.037+0000 7f9ee67fc640 1 --2- 172.21.15.161:0/1411272708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ef01529d0 0x7f9ef00af960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:31.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.037+0000 7f9eeeffd640 1 -- 172.21.15.161:0/1411272708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ef01529d0 msgr2=0x7f9ef00af960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:31.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.037+0000 7f9eeeffd640 1 --2- 172.21.15.161:0/1411272708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ef01529d0 0x7f9ef00af960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.037+0000 7f9eeeffd640 1 -- 172.21.15.161:0/1411272708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ef0155c30 msgr2=0x7f9ef00afea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:31.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.037+0000 7f9eeeffd640 1 --2- 172.21.15.161:0/1411272708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ef0155c30 0x7f9ef00afea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.037+0000 7f9eeeffd640 1 -- 172.21.15.161:0/1411272708 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ef00c8ea0 con 0x7f9ef00c6750 2024-03-28T23:30:31.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.037+0000 7f9eef7fe640 1 --2- 172.21.15.161:0/1411272708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ef0155c30 0x7f9ef00afea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:31.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.038+0000 7f9eeeffd640 1 --2- 172.21.15.161:0/1411272708 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ef00c6750 0x7f9ef00adb20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9ee000e990 tx=0x7f9ee000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:31.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.038+0000 7f9ee7fff640 1 -- 172.21.15.161:0/1411272708 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee0017070 con 0x7f9ef00c6750 2024-03-28T23:30:31.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.038+0000 7f9ee7fff640 1 -- 172.21.15.161:0/1411272708 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ee000c9e0 con 0x7f9ef00c6750 2024-03-28T23:30:31.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.038+0000 7f9ef536c640 1 -- 172.21.15.161:0/1411272708 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ef00b03e0 con 0x7f9ef00c6750 2024-03-28T23:30:31.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.038+0000 7f9ee7fff640 1 -- 172.21.15.161:0/1411272708 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee000cdb0 con 0x7f9ef00c6750 2024-03-28T23:30:31.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.038+0000 7f9ef536c640 1 -- 172.21.15.161:0/1411272708 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ef01e0120 con 0x7f9ef00c6750 2024-03-28T23:30:31.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.039+0000 7f9ef536c640 1 -- 172.21.15.161:0/1411272708 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ef004f4b0 con 0x7f9ef00c6750 2024-03-28T23:30:31.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.039+0000 7f9ee7fff640 1 -- 172.21.15.161:0/1411272708 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9ee001a070 con 0x7f9ef00c6750 2024-03-28T23:30:31.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.040+0000 7f9ee7fff640 1 --2- 172.21.15.161:0/1411272708 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9ed006b4d0 0x7f9ed006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:31.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.040+0000 7f9ee7fff640 1 -- 172.21.15.161:0/1411272708 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9ee008a2c0 con 0x7f9ef00c6750 2024-03-28T23:30:31.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.040+0000 7f9ee67fc640 1 --2- 172.21.15.161:0/1411272708 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9ed006b4d0 0x7f9ed006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:31.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.040+0000 7f9ee67fc640 1 --2- 172.21.15.161:0/1411272708 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9ed006b4d0 0x7f9ed006d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8036000 tx=0x7f9ed803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:31.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.048+0000 7f9ee7fff640 1 -- 172.21.15.161:0/1411272708 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9ef00c8ea0 con 0x7f9ef00c6750 2024-03-28T23:30:31.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.248+0000 7f9ef536c640 1 -- 172.21.15.161:0/1411272708 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9ef00743e0 con 0x7f9ef00c6750 2024-03-28T23:30:31.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.248+0000 7f9ee7fff640 1 -- 172.21.15.161:0/1411272708 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f9ee0058840 con 0x7f9ef00c6750 2024-03-28T23:30:31.249 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:30:31.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.250+0000 7f9ef536c640 1 -- 172.21.15.161:0/1411272708 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9ed006b4d0 msgr2=0x7f9ed006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:31.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.250+0000 7f9ef536c640 1 --2- 172.21.15.161:0/1411272708 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9ed006b4d0 0x7f9ed006d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8036000 tx=0x7f9ed803c040 comp rx=0 tx=0).stop 2024-03-28T23:30:31.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.250+0000 7f9ef536c640 1 -- 172.21.15.161:0/1411272708 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ef00c6750 msgr2=0x7f9ef00adb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:31.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.250+0000 7f9ef536c640 1 --2- 172.21.15.161:0/1411272708 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ef00c6750 0x7f9ef00adb20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9ee000e990 tx=0x7f9ee000ee50 comp rx=0 tx=0).stop 2024-03-28T23:30:31.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.250+0000 7f9ef536c640 1 -- 172.21.15.161:0/1411272708 shutdown_connections 2024-03-28T23:30:31.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.250+0000 7f9ef536c640 1 --2- 172.21.15.161:0/1411272708 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9ed006b4d0 0x7f9ed006d990 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.250+0000 7f9ef536c640 1 --2- 172.21.15.161:0/1411272708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ef0155c30 0x7f9ef00afea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.250+0000 7f9ef536c640 1 --2- 172.21.15.161:0/1411272708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ef01529d0 0x7f9ef00af960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.250+0000 7f9ef536c640 1 --2- 172.21.15.161:0/1411272708 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ef00c6750 0x7f9ef00adb20 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.250+0000 7f9ef536c640 1 -- 172.21.15.161:0/1411272708 >> 172.21.15.161:0/1411272708 conn(0x7f9ef00c7430 msgr2=0x7f9ef0151bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:31.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.251+0000 7f9ef536c640 1 -- 172.21.15.161:0/1411272708 shutdown_connections 2024-03-28T23:30:31.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.251+0000 7f9ef536c640 1 -- 172.21.15.161:0/1411272708 wait complete. 2024-03-28T23:30:31.262 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T23:30:31.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.367+0000 7f5134e40640 1 Processor -- start 2024-03-28T23:30:31.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.367+0000 7f5134e40640 1 -- start start 2024-03-28T23:30:31.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.367+0000 7f5134e40640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f513014f5c0 0x7f51300c2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:31.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.367+0000 7f5134e40640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f513006cdd0 0x7f51301524c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:31.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.367+0000 7f5134e40640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51300c2690 0x7f5130152a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:31.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.367+0000 7f5134e40640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f51300c74d0 con 0x7f513014f5c0 2024-03-28T23:30:31.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.368+0000 7f5134e40640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f51300c6fa0 con 0x7f513006cdd0 2024-03-28T23:30:31.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.368+0000 7f5134e40640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5130152030 con 0x7f51300c2690 2024-03-28T23:30:31.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.368+0000 7f512e59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f513014f5c0 0x7f51300c2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:31.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.368+0000 7f512e59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f513014f5c0 0x7f51300c2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48876/0 (socket says 172.21.15.161:48876) 2024-03-28T23:30:31.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.368+0000 7f512e59c640 1 -- 172.21.15.161:0/2773643215 learned_addr learned my addr 172.21.15.161:0/2773643215 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:31.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.368+0000 7f512e59c640 1 -- 172.21.15.161:0/2773643215 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51300c2690 msgr2=0x7f5130152a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:31.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.368+0000 7f512dd9b640 1 --2- 172.21.15.161:0/2773643215 >> v2:172.21.15.170:3300/0 conn(0x7f513006cdd0 0x7f51301524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:31.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.368+0000 7f512ed9d640 1 --2- 172.21.15.161:0/2773643215 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51300c2690 0x7f5130152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:31.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.368+0000 7f512e59c640 1 --2- 172.21.15.161:0/2773643215 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51300c2690 0x7f5130152a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.368+0000 7f512e59c640 1 -- 172.21.15.161:0/2773643215 >> v2:172.21.15.170:3300/0 conn(0x7f513006cdd0 msgr2=0x7f51301524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:31.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.368+0000 7f512e59c640 1 --2- 172.21.15.161:0/2773643215 >> v2:172.21.15.170:3300/0 conn(0x7f513006cdd0 0x7f51301524c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.368+0000 7f512e59c640 1 -- 172.21.15.161:0/2773643215 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51300c2f10 con 0x7f513014f5c0 2024-03-28T23:30:31.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.369+0000 7f512e59c640 1 --2- 172.21.15.161:0/2773643215 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f513014f5c0 0x7f51300c2150 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f513006b800 tx=0x7f512002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c01532245ee1472 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:31.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.369+0000 7f512d59a640 1 -- 172.21.15.161:0/2773643215 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5120035020 con 0x7f513014f5c0 2024-03-28T23:30:31.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.369+0000 7f512d59a640 1 -- 172.21.15.161:0/2773643215 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5120031870 con 0x7f513014f5c0 2024-03-28T23:30:31.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.369+0000 7f512d59a640 1 -- 172.21.15.161:0/2773643215 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5120031c40 con 0x7f513014f5c0 2024-03-28T23:30:31.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.370+0000 7f5134e40640 1 -- 172.21.15.161:0/2773643215 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f513014f5c0 msgr2=0x7f51300c2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:31.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.370+0000 7f5134e40640 1 --2- 172.21.15.161:0/2773643215 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f513014f5c0 0x7f51300c2150 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f513006b800 tx=0x7f512002b910 comp rx=0 tx=0).stop 2024-03-28T23:30:31.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.370+0000 7f5134e40640 1 -- 172.21.15.161:0/2773643215 shutdown_connections 2024-03-28T23:30:31.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.370+0000 7f5134e40640 1 --2- 172.21.15.161:0/2773643215 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51300c2690 0x7f5130152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.370+0000 7f5134e40640 1 --2- 172.21.15.161:0/2773643215 >> v2:172.21.15.170:3300/0 conn(0x7f513006cdd0 0x7f51301524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.370+0000 7f5134e40640 1 --2- 172.21.15.161:0/2773643215 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f513014f5c0 0x7f51300c2150 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.370+0000 7f5134e40640 1 -- 172.21.15.161:0/2773643215 >> 172.21.15.161:0/2773643215 conn(0x7f51300a83f0 msgr2=0x7f51300a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:31.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.370+0000 7f5134e40640 1 -- 172.21.15.161:0/2773643215 shutdown_connections 2024-03-28T23:30:31.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.370+0000 7f5134e40640 1 -- 172.21.15.161:0/2773643215 wait complete. 2024-03-28T23:30:31.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.371+0000 7f5134e40640 1 Processor -- start 2024-03-28T23:30:31.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.371+0000 7f5134e40640 1 -- start start 2024-03-28T23:30:31.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.372+0000 7f5134e40640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f513006cdd0 0x7f51301ec070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:31.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.372+0000 7f5134e40640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51300c2690 0x7f5130072b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:31.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.372+0000 7f5134e40640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f513014f5c0 0x7f51301e92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:31.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.372+0000 7f5134e40640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f513006c140 con 0x7f513006cdd0 2024-03-28T23:30:31.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.372+0000 7f5134e40640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f513014f9a0 con 0x7f513014f5c0 2024-03-28T23:30:31.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.372+0000 7f5134e40640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f513006c3e0 con 0x7f51300c2690 2024-03-28T23:30:31.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.372+0000 7f512e59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f513006cdd0 0x7f51301ec070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:31.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.372+0000 7f512e59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f513006cdd0 0x7f51301ec070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48880/0 (socket says 172.21.15.161:48880) 2024-03-28T23:30:31.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.372+0000 7f512e59c640 1 -- 172.21.15.161:0/2365928981 learned_addr learned my addr 172.21.15.161:0/2365928981 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:31.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.372+0000 7f512dd9b640 1 --2- 172.21.15.161:0/2365928981 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51300c2690 0x7f5130072b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:31.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.372+0000 7f512ed9d640 1 --2- 172.21.15.161:0/2365928981 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f513014f5c0 0x7f51301e92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:31.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.372+0000 7f512e59c640 1 -- 172.21.15.161:0/2365928981 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51300c2690 msgr2=0x7f5130072b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:31.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.372+0000 7f512e59c640 1 --2- 172.21.15.161:0/2365928981 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51300c2690 0x7f5130072b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.372+0000 7f512e59c640 1 -- 172.21.15.161:0/2365928981 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f513014f5c0 msgr2=0x7f51301e92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:31.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.372+0000 7f512e59c640 1 --2- 172.21.15.161:0/2365928981 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f513014f5c0 0x7f51301e92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.372+0000 7f512e59c640 1 -- 172.21.15.161:0/2365928981 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5130073530 con 0x7f513006cdd0 2024-03-28T23:30:31.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.373+0000 7f512e59c640 1 --2- 172.21.15.161:0/2365928981 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f513006cdd0 0x7f51301ec070 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f512002b950 tx=0x7f51200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:31.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.373+0000 7f511effd640 1 -- 172.21.15.161:0/2365928981 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5120040070 con 0x7f513006cdd0 2024-03-28T23:30:31.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.373+0000 7f511effd640 1 -- 172.21.15.161:0/2365928981 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f512003a420 con 0x7f513006cdd0 2024-03-28T23:30:31.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.373+0000 7f5134e40640 1 -- 172.21.15.161:0/2365928981 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51301e8b00 con 0x7f513006cdd0 2024-03-28T23:30:31.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.373+0000 7f511effd640 1 -- 172.21.15.161:0/2365928981 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5120035040 con 0x7f513006cdd0 2024-03-28T23:30:31.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.373+0000 7f5134e40640 1 -- 172.21.15.161:0/2365928981 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5130153540 con 0x7f513006cdd0 2024-03-28T23:30:31.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.374+0000 7f5134e40640 1 -- 172.21.15.161:0/2365928981 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51300abba0 con 0x7f513006cdd0 2024-03-28T23:30:31.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.374+0000 7f511effd640 1 -- 172.21.15.161:0/2365928981 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f512004b060 con 0x7f513006cdd0 2024-03-28T23:30:31.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.375+0000 7f511effd640 1 --2- 172.21.15.161:0/2365928981 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f511006b4d0 0x7f511006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:31.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.375+0000 7f512dd9b640 1 --2- 172.21.15.161:0/2365928981 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f511006b4d0 0x7f511006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:31.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.375+0000 7f511effd640 1 -- 172.21.15.161:0/2365928981 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f51200ac2c0 con 0x7f513006cdd0 2024-03-28T23:30:31.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.375+0000 7f512dd9b640 1 --2- 172.21.15.161:0/2365928981 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f511006b4d0 0x7f511006d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f513014ccf0 tx=0x7f5124008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:31.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.379+0000 7f511effd640 1 -- 172.21.15.161:0/2365928981 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f51300abba0 con 0x7f513006cdd0 2024-03-28T23:30:31.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.618+0000 7f5134e40640 1 -- 172.21.15.161:0/2365928981 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f513006e570 con 0x7f513006cdd0 2024-03-28T23:30:31.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.619+0000 7f511effd640 1 -- 172.21.15.161:0/2365928981 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f513006e570 con 0x7f513006cdd0 2024-03-28T23:30:31.619 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:30:31.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.620+0000 7f5134e40640 1 -- 172.21.15.161:0/2365928981 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f511006b4d0 msgr2=0x7f511006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:31.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.620+0000 7f5134e40640 1 --2- 172.21.15.161:0/2365928981 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f511006b4d0 0x7f511006d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f513014ccf0 tx=0x7f5124008040 comp rx=0 tx=0).stop 2024-03-28T23:30:31.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.620+0000 7f5134e40640 1 -- 172.21.15.161:0/2365928981 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f513006cdd0 msgr2=0x7f51301ec070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:31.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.620+0000 7f5134e40640 1 --2- 172.21.15.161:0/2365928981 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f513006cdd0 0x7f51301ec070 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f512002b950 tx=0x7f51200046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.621+0000 7f5134e40640 1 -- 172.21.15.161:0/2365928981 shutdown_connections 2024-03-28T23:30:31.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.621+0000 7f5134e40640 1 --2- 172.21.15.161:0/2365928981 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f511006b4d0 0x7f511006d990 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.621+0000 7f5134e40640 1 --2- 172.21.15.161:0/2365928981 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f513014f5c0 0x7f51301e92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.621+0000 7f5134e40640 1 --2- 172.21.15.161:0/2365928981 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51300c2690 0x7f5130072b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.621+0000 7f5134e40640 1 --2- 172.21.15.161:0/2365928981 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f513006cdd0 0x7f51301ec070 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.621+0000 7f5134e40640 1 -- 172.21.15.161:0/2365928981 >> 172.21.15.161:0/2365928981 conn(0x7f51300a83f0 msgr2=0x7f51300a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:31.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.621+0000 7f5134e40640 1 -- 172.21.15.161:0/2365928981 shutdown_connections 2024-03-28T23:30:31.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.621+0000 7f5134e40640 1 -- 172.21.15.161:0/2365928981 wait complete. 2024-03-28T23:30:31.658 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_drop (tasks.cephfs.test_failover.TestClusterAffinity) That unsetting mds_join_fs will not cause failover if no high-affinity standby exists. 2024-03-28T23:30:31.659 INFO:tasks.cephfs_test_runner:test_join_fs_drop (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-28T23:30:31.659 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:30:31.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.760+0000 7fa860e74640 1 Processor -- start 2024-03-28T23:30:31.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.760+0000 7fa860e74640 1 -- start start 2024-03-28T23:30:31.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.761+0000 7fa860e74640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa85c0c3ae0 0x7fa85c1591e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:31.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.761+0000 7fa860e74640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa85c155d30 con 0x7fa85c14f240 2024-03-28T23:30:31.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.761+0000 7fa860e74640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa85c0c1e10 con 0x7fa85c0c3410 2024-03-28T23:30:31.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.761+0000 7fa860e74640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fa85c156d10 con 0x7fa85c0c3ae0 2024-03-28T23:30:31.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.761+0000 7fa85a59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa85c14f240 0x7fa85c156600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60076/0 (socket says 172.21.15.161:60076) 2024-03-28T23:30:31.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.761+0000 7fa85a59c640 1 -- 172.21.15.161:0/2906655052 learned_addr learned my addr 172.21.15.161:0/2906655052 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:31.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.761+0000 7fa85ad9d640 1 --2- 172.21.15.161:0/2906655052 >> v2:172.21.15.170:3300/0 conn(0x7fa85c0c3ae0 0x7fa85c1591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:31.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.761+0000 7fa85959a640 1 -- 172.21.15.161:0/2906655052 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2319685372 0 0) 0x7fa85c155d30 con 0x7fa85c14f240 2024-03-28T23:30:31.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.761+0000 7fa85959a640 1 -- 172.21.15.161:0/2906655052 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa844003520 con 0x7fa85c14f240 2024-03-28T23:30:31.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.762+0000 7fa85959a640 1 -- 172.21.15.161:0/2906655052 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3607285447 0 0) 0x7fa844003520 con 0x7fa85c14f240 2024-03-28T23:30:31.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.762+0000 7fa85959a640 1 -- 172.21.15.161:0/2906655052 >> v2:172.21.15.170:3300/0 conn(0x7fa85c0c3ae0 msgr2=0x7fa85c1591e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:31.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.762+0000 7fa85959a640 1 --2- 172.21.15.161:0/2906655052 >> v2:172.21.15.170:3300/0 conn(0x7fa85c0c3ae0 0x7fa85c1591e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.762+0000 7fa85959a640 1 -- 172.21.15.161:0/2906655052 >> v1:172.21.15.170:6789/0 conn(0x7fa85c0c3410 legacy=0x7fa85c0c66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:31.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.762+0000 7fa85959a640 1 -- 172.21.15.161:0/2906655052 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa85c155fd0 con 0x7fa85c14f240 2024-03-28T23:30:31.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.762+0000 7fa85959a640 1 -- 172.21.15.161:0/2906655052 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fa8540019f0 con 0x7fa85c14f240 2024-03-28T23:30:31.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.762+0000 7fa85959a640 1 -- 172.21.15.161:0/2906655052 >> v1:172.21.15.161:6789/0 conn(0x7fa85c14f240 legacy=0x7fa85c156600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:31.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.762+0000 7fa85959a640 1 --2- 172.21.15.161:0/2906655052 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa844003a90 0x7fa844003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:31.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.762+0000 7fa85959a640 1 --2- 172.21.15.161:0/2906655052 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa844004800 0x7fa844004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:31.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.762+0000 7fa85959a640 1 --2- 172.21.15.161:0/2906655052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa844005210 0x7fa84400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:31.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.762+0000 7fa85959a640 1 -- 172.21.15.161:0/2906655052 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa85c156d10 con 0x7fa844005210 2024-03-28T23:30:31.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.762+0000 7fa85959a640 1 -- 172.21.15.161:0/2906655052 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa84400c160 con 0x7fa844003a90 2024-03-28T23:30:31.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.762+0000 7fa85959a640 1 -- 172.21.15.161:0/2906655052 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa84400c360 con 0x7fa844004800 2024-03-28T23:30:31.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.762+0000 7fa85ad9d640 1 --2- 172.21.15.161:0/2906655052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa844005210 0x7fa84400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:31.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.762+0000 7fa85a59c640 1 --2- 172.21.15.161:0/2906655052 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa844003a90 0x7fa844003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:31.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.762+0000 7fa859d9b640 1 --2- 172.21.15.161:0/2906655052 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa844004800 0x7fa844004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:31.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.762+0000 7fa85ad9d640 1 -- 172.21.15.161:0/2906655052 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa844004800 msgr2=0x7fa844004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:31.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.762+0000 7fa85ad9d640 1 --2- 172.21.15.161:0/2906655052 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa844004800 0x7fa844004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.762+0000 7fa85ad9d640 1 -- 172.21.15.161:0/2906655052 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa844003a90 msgr2=0x7fa844003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:31.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.762+0000 7fa85ad9d640 1 --2- 172.21.15.161:0/2906655052 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa844003a90 0x7fa844003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.762+0000 7fa85ad9d640 1 -- 172.21.15.161:0/2906655052 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa844003520 con 0x7fa844005210 2024-03-28T23:30:31.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.763+0000 7fa85ad9d640 1 -- 172.21.15.161:0/2906655052 reap_dead start 2024-03-28T23:30:31.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.763+0000 7fa85ad9d640 1 --2- 172.21.15.161:0/2906655052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa844005210 0x7fa84400b9c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa84c009a90 tx=0x7fa844005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a680506bca620e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:31.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.763+0000 7fa85959a640 1 -- 172.21.15.161:0/2906655052 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84c009690 con 0x7fa844005210 2024-03-28T23:30:31.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.763+0000 7fa85959a640 1 -- 172.21.15.161:0/2906655052 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa84c004020 con 0x7fa844005210 2024-03-28T23:30:31.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.763+0000 7fa85959a640 1 -- 172.21.15.161:0/2906655052 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84c004410 con 0x7fa844005210 2024-03-28T23:30:31.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.764+0000 7fa860e74640 1 -- 172.21.15.161:0/2906655052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa844005210 msgr2=0x7fa84400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:31.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.764+0000 7fa860e74640 1 --2- 172.21.15.161:0/2906655052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa844005210 0x7fa84400b9c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa84c009a90 tx=0x7fa844005830 comp rx=0 tx=0).stop 2024-03-28T23:30:31.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.764+0000 7fa860e74640 1 -- 172.21.15.161:0/2906655052 shutdown_connections 2024-03-28T23:30:31.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.764+0000 7fa860e74640 1 --2- 172.21.15.161:0/2906655052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa844005210 0x7fa84400b9c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.764+0000 7fa860e74640 1 -- 172.21.15.161:0/2906655052 >> 172.21.15.161:0/2906655052 conn(0x7fa85c0c7790 msgr2=0x7fa85c0c7ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:31.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.764+0000 7fa860e74640 1 -- 172.21.15.161:0/2906655052 shutdown_connections 2024-03-28T23:30:31.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.764+0000 7fa860e74640 1 -- 172.21.15.161:0/2906655052 wait complete. 2024-03-28T23:30:31.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.765+0000 7fa860e74640 1 Processor -- start 2024-03-28T23:30:31.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.765+0000 7fa860e74640 1 -- start start 2024-03-28T23:30:31.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.766+0000 7fa860e74640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa844005210 0x7fa85c0a85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:31.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.766+0000 7fa860e74640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa85c0723a0 0x7fa85c1eb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:31.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.766+0000 7fa860e74640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa85c1eb550 0x7fa85c14d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:31.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.766+0000 7fa860e74640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa85c1589e0 con 0x7fa844005210 2024-03-28T23:30:31.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.766+0000 7fa860e74640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa85c14f620 con 0x7fa85c0723a0 2024-03-28T23:30:31.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.766+0000 7fa860e74640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa85c158cc0 con 0x7fa85c1eb550 2024-03-28T23:30:31.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.766+0000 7fa85a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa844005210 0x7fa85c0a85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:31.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.766+0000 7fa85a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa844005210 0x7fa85c0a85d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48908/0 (socket says 172.21.15.161:48908) 2024-03-28T23:30:31.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.766+0000 7fa85a59c640 1 -- 172.21.15.161:0/71073381 learned_addr learned my addr 172.21.15.161:0/71073381 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:31.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.766+0000 7fa85ad9d640 1 --2- 172.21.15.161:0/71073381 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa85c1eb550 0x7fa85c14d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:31.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.766+0000 7fa859d9b640 1 --2- 172.21.15.161:0/71073381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa85c0723a0 0x7fa85c1eb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:31.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.766+0000 7fa85a59c640 1 -- 172.21.15.161:0/71073381 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa85c1eb550 msgr2=0x7fa85c14d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:31.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.766+0000 7fa85a59c640 1 --2- 172.21.15.161:0/71073381 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa85c1eb550 0x7fa85c14d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.766+0000 7fa85a59c640 1 -- 172.21.15.161:0/71073381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa85c0723a0 msgr2=0x7fa85c1eb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:31.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.766+0000 7fa85a59c640 1 --2- 172.21.15.161:0/71073381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa85c0723a0 0x7fa85c1eb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.766+0000 7fa85a59c640 1 -- 172.21.15.161:0/71073381 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa85c072cd0 con 0x7fa844005210 2024-03-28T23:30:31.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.766+0000 7fa85a59c640 1 --2- 172.21.15.161:0/71073381 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa844005210 0x7fa85c0a85d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa8540079d0 tx=0x7fa85400a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:31.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.766+0000 7fa84affd640 1 -- 172.21.15.161:0/71073381 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8540019b0 con 0x7fa844005210 2024-03-28T23:30:31.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.766+0000 7fa84affd640 1 -- 172.21.15.161:0/71073381 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa854018070 con 0x7fa844005210 2024-03-28T23:30:31.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.766+0000 7fa860e74640 1 -- 172.21.15.161:0/71073381 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa85c072fb0 con 0x7fa844005210 2024-03-28T23:30:31.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.766+0000 7fa84affd640 1 -- 172.21.15.161:0/71073381 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa854013630 con 0x7fa844005210 2024-03-28T23:30:31.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.766+0000 7fa860e74640 1 -- 172.21.15.161:0/71073381 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa85c073570 con 0x7fa844005210 2024-03-28T23:30:31.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.767+0000 7fa860e74640 1 -- 172.21.15.161:0/71073381 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa85c04f4b0 con 0x7fa844005210 2024-03-28T23:30:31.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.768+0000 7fa84affd640 1 -- 172.21.15.161:0/71073381 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa8540137d0 con 0x7fa844005210 2024-03-28T23:30:31.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.768+0000 7fa84affd640 1 --2- 172.21.15.161:0/71073381 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa83c06b500 0x7fa83c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:31.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.769+0000 7fa859d9b640 1 --2- 172.21.15.161:0/71073381 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa83c06b500 0x7fa83c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:31.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.769+0000 7fa84affd640 1 -- 172.21.15.161:0/71073381 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa85408ac20 con 0x7fa844005210 2024-03-28T23:30:31.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.769+0000 7fa859d9b640 1 --2- 172.21.15.161:0/71073381 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa83c06b500 0x7fa83c06d9c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa85c149b50 tx=0x7fa850009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:31.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.773+0000 7fa84affd640 1 -- 172.21.15.161:0/71073381 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa8540597f0 con 0x7fa844005210 2024-03-28T23:30:31.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.995+0000 7fa860e74640 1 -- 172.21.15.161:0/71073381 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa85c0c31f0 con 0x7fa844005210 2024-03-28T23:30:31.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.996+0000 7fa84affd640 1 -- 172.21.15.161:0/71073381 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7fa8540590f0 con 0x7fa844005210 2024-03-28T23:30:31.997 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:31.997 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":24,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:29:50.946843+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:30:31.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.997+0000 7fa860e74640 1 -- 172.21.15.161:0/71073381 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa83c06b500 msgr2=0x7fa83c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:31.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.997+0000 7fa860e74640 1 --2- 172.21.15.161:0/71073381 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa83c06b500 0x7fa83c06d9c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa85c149b50 tx=0x7fa850009040 comp rx=0 tx=0).stop 2024-03-28T23:30:31.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.998+0000 7fa860e74640 1 -- 172.21.15.161:0/71073381 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa844005210 msgr2=0x7fa85c0a85d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:31.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.998+0000 7fa860e74640 1 --2- 172.21.15.161:0/71073381 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa844005210 0x7fa85c0a85d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa8540079d0 tx=0x7fa85400a6c0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.998+0000 7fa860e74640 1 -- 172.21.15.161:0/71073381 shutdown_connections 2024-03-28T23:30:31.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.998+0000 7fa860e74640 1 --2- 172.21.15.161:0/71073381 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa83c06b500 0x7fa83c06d9c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.998+0000 7fa860e74640 1 --2- 172.21.15.161:0/71073381 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa85c1eb550 0x7fa85c14d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.998+0000 7fa860e74640 1 --2- 172.21.15.161:0/71073381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa85c0723a0 0x7fa85c1eb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.998+0000 7fa860e74640 1 --2- 172.21.15.161:0/71073381 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa844005210 0x7fa85c0a85d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:31.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.998+0000 7fa860e74640 1 -- 172.21.15.161:0/71073381 >> 172.21.15.161:0/71073381 conn(0x7fa85c0c7790 msgr2=0x7fa85c148c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:31.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.998+0000 7fa860e74640 1 -- 172.21.15.161:0/71073381 shutdown_connections 2024-03-28T23:30:31.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:31.998+0000 7fa860e74640 1 -- 172.21.15.161:0/71073381 wait complete. 2024-03-28T23:30:32.009 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:29:44.021375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:29:44.872102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T23:29:48.591063+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T23:30:32.010 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:30:32.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.118+0000 7f00caf7a640 1 Processor -- start 2024-03-28T23:30:32.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.118+0000 7f00caf7a640 1 -- start start 2024-03-28T23:30:32.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.120+0000 7f00caf7a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00c40ab690 0x7f00c4071690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:32.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.120+0000 7f00caf7a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f00c40aba70 0x7f00c406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:32.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.120+0000 7f00caf7a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00c40abea0 con 0x7f00c406f290 2024-03-28T23:30:32.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.120+0000 7f00caf7a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f00c4154000 con 0x7f00c40aba70 2024-03-28T23:30:32.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.120+0000 7f00caf7a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f00c4072fa0 con 0x7f00c40ab690 2024-03-28T23:30:32.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.120+0000 7f00c3fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f00c406f290 0x7f00c41538f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60084/0 (socket says 172.21.15.161:60084) 2024-03-28T23:30:32.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.120+0000 7f00c3fff640 1 -- 172.21.15.161:0/4032134387 learned_addr learned my addr 172.21.15.161:0/4032134387 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:32.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.120+0000 7f00c9517640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f00c40aba70 0x7f00c406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:32.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.120+0000 7f00c8d16640 1 --2- 172.21.15.161:0/4032134387 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00c40ab690 0x7f00c4071690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:32.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.120+0000 7f00c37fe640 1 -- 172.21.15.161:0/4032134387 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2998051547 0 0) 0x7f00c40abea0 con 0x7f00c406f290 2024-03-28T23:30:32.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.120+0000 7f00c37fe640 1 -- 172.21.15.161:0/4032134387 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00ac003520 con 0x7f00c406f290 2024-03-28T23:30:32.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.120+0000 7f00c9517640 1 -- 172.21.15.161:0/4032134387 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00c40ab690 msgr2=0x7f00c4071690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:32.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.120+0000 7f00c9517640 1 --2- 172.21.15.161:0/4032134387 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00c40ab690 0x7f00c4071690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.120+0000 7f00c9517640 1 -- 172.21.15.161:0/4032134387 >> v1:172.21.15.161:6789/0 conn(0x7f00c406f290 legacy=0x7f00c41538f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:32.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.120+0000 7f00c9517640 1 -- 172.21.15.161:0/4032134387 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c41530f0 con 0x7f00c40aba70 2024-03-28T23:30:32.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.121+0000 7f00c9517640 1 --2- 172.21.15.161:0/4032134387 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c40aba70 0x7f00c406dd20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f00bc009d90 tx=0x7f00bc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da29bcfe6bca4d56 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:32.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.121+0000 7f00c37fe640 1 -- 172.21.15.161:0/4032134387 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00bc039070 con 0x7f00c40aba70 2024-03-28T23:30:32.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.121+0000 7f00c37fe640 1 -- 172.21.15.161:0/4032134387 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00bc02cc50 con 0x7f00c40aba70 2024-03-28T23:30:32.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.121+0000 7f00c37fe640 1 -- 172.21.15.161:0/4032134387 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00bc034440 con 0x7f00c40aba70 2024-03-28T23:30:32.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.122+0000 7f00caf7a640 1 -- 172.21.15.161:0/4032134387 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c40aba70 msgr2=0x7f00c406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:32.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.122+0000 7f00caf7a640 1 --2- 172.21.15.161:0/4032134387 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c40aba70 0x7f00c406dd20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f00bc009d90 tx=0x7f00bc02bea0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.122+0000 7f00caf7a640 1 -- 172.21.15.161:0/4032134387 shutdown_connections 2024-03-28T23:30:32.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.122+0000 7f00caf7a640 1 --2- 172.21.15.161:0/4032134387 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c40aba70 0x7f00c406dd20 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.122+0000 7f00caf7a640 1 --2- 172.21.15.161:0/4032134387 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00c40ab690 0x7f00c4071690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.122+0000 7f00caf7a640 1 -- 172.21.15.161:0/4032134387 >> 172.21.15.161:0/4032134387 conn(0x7f00c40c4e70 msgr2=0x7f00c40c5280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:32.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.122+0000 7f00caf7a640 1 -- 172.21.15.161:0/4032134387 shutdown_connections 2024-03-28T23:30:32.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.122+0000 7f00caf7a640 1 -- 172.21.15.161:0/4032134387 wait complete. 2024-03-28T23:30:32.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.124+0000 7f00caf7a640 1 Processor -- start 2024-03-28T23:30:32.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.124+0000 7f00caf7a640 1 -- start start 2024-03-28T23:30:32.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.125+0000 7f00caf7a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00c406f290 0x7f00c4151850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:32.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.125+0000 7f00caf7a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00c40ab690 0x7f00c40c0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:32.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.125+0000 7f00caf7a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c40aba70 0x7f00c40c10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:32.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.125+0000 7f00caf7a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f00c4073280 con 0x7f00c40aba70 2024-03-28T23:30:32.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.125+0000 7f00caf7a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f00c40aca00 con 0x7f00c406f290 2024-03-28T23:30:32.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.125+0000 7f00caf7a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f00c4150890 con 0x7f00c40ab690 2024-03-28T23:30:32.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.125+0000 7f00c9517640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c40aba70 0x7f00c40c10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:32.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.125+0000 7f00c9517640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c40aba70 0x7f00c40c10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48926/0 (socket says 172.21.15.161:48926) 2024-03-28T23:30:32.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.125+0000 7f00c9517640 1 -- 172.21.15.161:0/1089267596 learned_addr learned my addr 172.21.15.161:0/1089267596 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:32.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.126+0000 7f00c3fff640 1 --2- 172.21.15.161:0/1089267596 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00c40ab690 0x7f00c40c0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:32.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.126+0000 7f00c8d16640 1 --2- 172.21.15.161:0/1089267596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00c406f290 0x7f00c4151850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:32.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.126+0000 7f00c9517640 1 -- 172.21.15.161:0/1089267596 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00c40ab690 msgr2=0x7f00c40c0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:32.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.126+0000 7f00c9517640 1 --2- 172.21.15.161:0/1089267596 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00c40ab690 0x7f00c40c0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.126+0000 7f00c9517640 1 -- 172.21.15.161:0/1089267596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00c406f290 msgr2=0x7f00c4151850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:32.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.126+0000 7f00c9517640 1 --2- 172.21.15.161:0/1089267596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00c406f290 0x7f00c4151850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.126+0000 7f00c9517640 1 -- 172.21.15.161:0/1089267596 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c41ecd80 con 0x7f00c40aba70 2024-03-28T23:30:32.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.126+0000 7f00c3fff640 1 --2- 172.21.15.161:0/1089267596 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00c40ab690 0x7f00c40c0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:32.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.126+0000 7f00c9517640 1 --2- 172.21.15.161:0/1089267596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c40aba70 0x7f00c40c10f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f00bc009eb0 tx=0x7f00bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:32.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.126+0000 7f00c17fa640 1 -- 172.21.15.161:0/1089267596 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00bc039050 con 0x7f00c40aba70 2024-03-28T23:30:32.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.126+0000 7f00c17fa640 1 -- 172.21.15.161:0/1089267596 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00bc045070 con 0x7f00c40aba70 2024-03-28T23:30:32.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.126+0000 7f00caf7a640 1 -- 172.21.15.161:0/1089267596 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00c4154ae0 con 0x7f00c40aba70 2024-03-28T23:30:32.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.126+0000 7f00c17fa640 1 -- 172.21.15.161:0/1089267596 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00bc009450 con 0x7f00c40aba70 2024-03-28T23:30:32.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.126+0000 7f00caf7a640 1 -- 172.21.15.161:0/1089267596 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00c4151d90 con 0x7f00c40aba70 2024-03-28T23:30:32.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.127+0000 7f00caf7a640 1 -- 172.21.15.161:0/1089267596 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00c404f4b0 con 0x7f00c40aba70 2024-03-28T23:30:32.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.128+0000 7f00c17fa640 1 -- 172.21.15.161:0/1089267596 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f00bc03e610 con 0x7f00c40aba70 2024-03-28T23:30:32.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.128+0000 7f00c17fa640 1 --2- 172.21.15.161:0/1089267596 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f00b006b500 0x7f00b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:32.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.128+0000 7f00c17fa640 1 -- 172.21.15.161:0/1089267596 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f00bc0ac800 con 0x7f00c40aba70 2024-03-28T23:30:32.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.128+0000 7f00c8d16640 1 --2- 172.21.15.161:0/1089267596 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f00b006b500 0x7f00b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:32.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.129+0000 7f00c8d16640 1 --2- 172.21.15.161:0/1089267596 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f00b006b500 0x7f00b006d9c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f00b4009a10 tx=0x7f00b4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:32.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.133+0000 7f00c17fa640 1 -- 172.21.15.161:0/1089267596 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f00bc04e020 con 0x7f00c40aba70 2024-03-28T23:30:32.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.309+0000 7f00caf7a640 1 -- 172.21.15.161:0/1089267596 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f00c4072250 con 0x7f00c40aba70 2024-03-28T23:30:32.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.309+0000 7f00c17fa640 1 -- 172.21.15.161:0/1089267596 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f00bc07b3d0 con 0x7f00c40aba70 2024-03-28T23:30:32.310 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:30:32.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.312+0000 7f00caf7a640 1 -- 172.21.15.161:0/1089267596 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f00b006b500 msgr2=0x7f00b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:32.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.312+0000 7f00caf7a640 1 --2- 172.21.15.161:0/1089267596 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f00b006b500 0x7f00b006d9c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f00b4009a10 tx=0x7f00b4008040 comp rx=0 tx=0).stop 2024-03-28T23:30:32.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.312+0000 7f00caf7a640 1 -- 172.21.15.161:0/1089267596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c40aba70 msgr2=0x7f00c40c10f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:32.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.312+0000 7f00caf7a640 1 --2- 172.21.15.161:0/1089267596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c40aba70 0x7f00c40c10f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f00bc009eb0 tx=0x7f00bc004a40 comp rx=0 tx=0).stop 2024-03-28T23:30:32.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.312+0000 7f00caf7a640 1 -- 172.21.15.161:0/1089267596 shutdown_connections 2024-03-28T23:30:32.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.312+0000 7f00caf7a640 1 --2- 172.21.15.161:0/1089267596 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f00b006b500 0x7f00b006d9c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.312+0000 7f00caf7a640 1 --2- 172.21.15.161:0/1089267596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c40aba70 0x7f00c40c10f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.312+0000 7f00caf7a640 1 --2- 172.21.15.161:0/1089267596 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00c40ab690 0x7f00c40c0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.312+0000 7f00caf7a640 1 --2- 172.21.15.161:0/1089267596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00c406f290 0x7f00c4151850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.312+0000 7f00caf7a640 1 -- 172.21.15.161:0/1089267596 >> 172.21.15.161:0/1089267596 conn(0x7f00c40c4e70 msgr2=0x7f00c40c60e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:32.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.312+0000 7f00caf7a640 1 -- 172.21.15.161:0/1089267596 shutdown_connections 2024-03-28T23:30:32.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.312+0000 7f00caf7a640 1 -- 172.21.15.161:0/1089267596 wait complete. 2024-03-28T23:30:32.324 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:30:32.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.427+0000 7f8c77f9a640 1 Processor -- start 2024-03-28T23:30:32.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.428+0000 7f8c77f9a640 1 -- start start 2024-03-28T23:30:32.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.428+0000 7f8c77f9a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8c700c0330 0x7f8c700c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:32.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.428+0000 7f8c77f9a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c701589b0 0x7f8c70158e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:32.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.428+0000 7f8c77f9a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c700c6f60 con 0x7f8c70152bc0 2024-03-28T23:30:32.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.428+0000 7f8c77f9a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8c700c6120 con 0x7f8c700c0330 2024-03-28T23:30:32.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.428+0000 7f8c77f9a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8c700c6400 con 0x7f8c701589b0 2024-03-28T23:30:32.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.428+0000 7f8c75535640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8c700c0330 0x7f8c700c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:32.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.428+0000 7f8c75535640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8c700c0330 0x7f8c700c9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48928/0 (socket says 172.21.15.161:48928) 2024-03-28T23:30:32.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.428+0000 7f8c75535640 1 -- 172.21.15.161:0/3514307559 learned_addr learned my addr 172.21.15.161:0/3514307559 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:32.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.429+0000 7f8c76537640 1 --2- 172.21.15.161:0/3514307559 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c701589b0 0x7f8c70158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:32.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.429+0000 7f8c75535640 1 -- 172.21.15.161:0/3514307559 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c701589b0 msgr2=0x7f8c70158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:32.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.429+0000 7f8c75535640 1 --2- 172.21.15.161:0/3514307559 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c701589b0 0x7f8c70158e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.429+0000 7f8c75535640 1 -- 172.21.15.161:0/3514307559 >> v1:172.21.15.170:6789/0 conn(0x7f8c70152bc0 legacy=0x7f8c700c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:32.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.429+0000 7f8c75535640 1 -- 172.21.15.161:0/3514307559 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c700c96d0 con 0x7f8c700c0330 2024-03-28T23:30:32.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.429+0000 7f8c75535640 1 --2- 172.21.15.161:0/3514307559 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c700c0330 0x7f8c700c9190 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8c60009870 tx=0x7f8c6002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c14a710e77622ecd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:32.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.430+0000 7f8c74d34640 1 -- 172.21.15.161:0/3514307559 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c60035020 con 0x7f8c700c0330 2024-03-28T23:30:32.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.430+0000 7f8c74d34640 1 -- 172.21.15.161:0/3514307559 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c60031640 con 0x7f8c700c0330 2024-03-28T23:30:32.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.430+0000 7f8c74d34640 1 -- 172.21.15.161:0/3514307559 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6002d040 con 0x7f8c700c0330 2024-03-28T23:30:32.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.430+0000 7f8c77f9a640 1 -- 172.21.15.161:0/3514307559 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c700c0330 msgr2=0x7f8c700c9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:32.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.430+0000 7f8c77f9a640 1 --2- 172.21.15.161:0/3514307559 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c700c0330 0x7f8c700c9190 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8c60009870 tx=0x7f8c6002b8f0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.430+0000 7f8c77f9a640 1 -- 172.21.15.161:0/3514307559 shutdown_connections 2024-03-28T23:30:32.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.430+0000 7f8c77f9a640 1 --2- 172.21.15.161:0/3514307559 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c701589b0 0x7f8c70158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.430+0000 7f8c77f9a640 1 --2- 172.21.15.161:0/3514307559 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c700c0330 0x7f8c700c9190 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.430+0000 7f8c77f9a640 1 -- 172.21.15.161:0/3514307559 >> 172.21.15.161:0/3514307559 conn(0x7f8c700caa80 msgr2=0x7f8c700cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:32.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.431+0000 7f8c77f9a640 1 -- 172.21.15.161:0/3514307559 shutdown_connections 2024-03-28T23:30:32.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.431+0000 7f8c77f9a640 1 -- 172.21.15.161:0/3514307559 wait complete. 2024-03-28T23:30:32.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.432+0000 7f8c77f9a640 1 Processor -- start 2024-03-28T23:30:32.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.432+0000 7f8c77f9a640 1 -- start start 2024-03-28T23:30:32.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.432+0000 7f8c77f9a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c700c0330 0x7f8c701420b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:32.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.432+0000 7f8c77f9a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c70152bc0 0x7f8c70147e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:32.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.432+0000 7f8c77f9a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c701589b0 0x7f8c70157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:32.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.432+0000 7f8c77f9a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8c70152fa0 con 0x7f8c70152bc0 2024-03-28T23:30:32.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.432+0000 7f8c77f9a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8c700c6bf0 con 0x7f8c700c0330 2024-03-28T23:30:32.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.432+0000 7f8c77f9a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8c701eb7d0 con 0x7f8c701589b0 2024-03-28T23:30:32.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.432+0000 7f8c75535640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c70152bc0 0x7f8c70147e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:32.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.432+0000 7f8c75535640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c70152bc0 0x7f8c70147e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48930/0 (socket says 172.21.15.161:48930) 2024-03-28T23:30:32.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.432+0000 7f8c75535640 1 -- 172.21.15.161:0/308850618 learned_addr learned my addr 172.21.15.161:0/308850618 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:32.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.433+0000 7f8c76537640 1 --2- 172.21.15.161:0/308850618 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c701589b0 0x7f8c70157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:32.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.433+0000 7f8c75d36640 1 --2- 172.21.15.161:0/308850618 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c700c0330 0x7f8c701420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:32.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.433+0000 7f8c75535640 1 -- 172.21.15.161:0/308850618 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c701589b0 msgr2=0x7f8c70157750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:32.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.433+0000 7f8c75535640 1 --2- 172.21.15.161:0/308850618 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c701589b0 0x7f8c70157750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.433+0000 7f8c75535640 1 -- 172.21.15.161:0/308850618 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c700c0330 msgr2=0x7f8c701420b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:32.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.433+0000 7f8c75535640 1 --2- 172.21.15.161:0/308850618 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c700c0330 0x7f8c701420b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.433+0000 7f8c75535640 1 -- 172.21.15.161:0/308850618 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c701507a0 con 0x7f8c70152bc0 2024-03-28T23:30:32.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.433+0000 7f8c75535640 1 --2- 172.21.15.161:0/308850618 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c70152bc0 0x7f8c70147e10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8c60000c00 tx=0x7f8c600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:32.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.433+0000 7f8c667fc640 1 -- 172.21.15.161:0/308850618 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c60040070 con 0x7f8c70152bc0 2024-03-28T23:30:32.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.433+0000 7f8c667fc640 1 -- 172.21.15.161:0/308850618 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c60039c10 con 0x7f8c70152bc0 2024-03-28T23:30:32.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.433+0000 7f8c77f9a640 1 -- 172.21.15.161:0/308850618 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c701eaec0 con 0x7f8c70152bc0 2024-03-28T23:30:32.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.433+0000 7f8c667fc640 1 -- 172.21.15.161:0/308850618 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c60035040 con 0x7f8c70152bc0 2024-03-28T23:30:32.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.433+0000 7f8c77f9a640 1 -- 172.21.15.161:0/308850618 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c701eb180 con 0x7f8c70152bc0 2024-03-28T23:30:32.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.434+0000 7f8c77f9a640 1 -- 172.21.15.161:0/308850618 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c701eb950 con 0x7f8c70152bc0 2024-03-28T23:30:32.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.435+0000 7f8c667fc640 1 -- 172.21.15.161:0/308850618 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8c60038020 con 0x7f8c70152bc0 2024-03-28T23:30:32.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.435+0000 7f8c667fc640 1 --2- 172.21.15.161:0/308850618 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8c5406b500 0x7f8c5406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:32.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.436+0000 7f8c667fc640 1 -- 172.21.15.161:0/308850618 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f8c600ac0e0 con 0x7f8c70152bc0 2024-03-28T23:30:32.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.436+0000 7f8c75d36640 1 --2- 172.21.15.161:0/308850618 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8c5406b500 0x7f8c5406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:32.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.436+0000 7f8c75d36640 1 --2- 172.21.15.161:0/308850618 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8c5406b500 0x7f8c5406d9c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8c70148f20 tx=0x7f8c6c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:32.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.444+0000 7f8c667fc640 1 -- 172.21.15.161:0/308850618 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8c701507a0 con 0x7f8c70152bc0 2024-03-28T23:30:32.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.638+0000 7f8c77f9a640 1 -- 172.21.15.161:0/308850618 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8c70158e40 con 0x7f8c70152bc0 2024-03-28T23:30:32.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.638+0000 7f8c667fc640 1 -- 172.21.15.161:0/308850618 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f8c70158e40 con 0x7f8c70152bc0 2024-03-28T23:30:32.639 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:30:32.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.640+0000 7f8c77f9a640 1 -- 172.21.15.161:0/308850618 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8c5406b500 msgr2=0x7f8c5406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:32.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.640+0000 7f8c77f9a640 1 --2- 172.21.15.161:0/308850618 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8c5406b500 0x7f8c5406d9c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8c70148f20 tx=0x7f8c6c009040 comp rx=0 tx=0).stop 2024-03-28T23:30:32.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.640+0000 7f8c77f9a640 1 -- 172.21.15.161:0/308850618 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c70152bc0 msgr2=0x7f8c70147e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:32.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.640+0000 7f8c77f9a640 1 --2- 172.21.15.161:0/308850618 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c70152bc0 0x7f8c70147e10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8c60000c00 tx=0x7f8c600046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.640+0000 7f8c77f9a640 1 -- 172.21.15.161:0/308850618 shutdown_connections 2024-03-28T23:30:32.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.640+0000 7f8c77f9a640 1 --2- 172.21.15.161:0/308850618 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8c5406b500 0x7f8c5406d9c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.640+0000 7f8c77f9a640 1 --2- 172.21.15.161:0/308850618 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c70152bc0 0x7f8c70147e10 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.640+0000 7f8c77f9a640 1 --2- 172.21.15.161:0/308850618 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c701589b0 0x7f8c70157750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.640+0000 7f8c77f9a640 1 --2- 172.21.15.161:0/308850618 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c700c0330 0x7f8c701420b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.640+0000 7f8c77f9a640 1 -- 172.21.15.161:0/308850618 >> 172.21.15.161:0/308850618 conn(0x7f8c700caa80 msgr2=0x7f8c7014e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:32.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.640+0000 7f8c77f9a640 1 -- 172.21.15.161:0/308850618 shutdown_connections 2024-03-28T23:30:32.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.640+0000 7f8c77f9a640 1 -- 172.21.15.161:0/308850618 wait complete. 2024-03-28T23:30:32.652 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:30:32.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.758+0000 7f378509c640 1 Processor -- start 2024-03-28T23:30:32.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.758+0000 7f378509c640 1 -- start start 2024-03-28T23:30:32.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.759+0000 7f378509c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f37801529d0 0x7f378006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:32.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.759+0000 7f378509c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f37800c6750 0x7f37800c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:32.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.759+0000 7f378509c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3780155c30 0x7f37800c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:32.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.759+0000 7f378509c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f3780152db0 con 0x7f37800c6750 2024-03-28T23:30:32.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.759+0000 7f378509c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f37800c7210 con 0x7f37801529d0 2024-03-28T23:30:32.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.759+0000 7f378509c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f37800c6c50 con 0x7f3780155c30 2024-03-28T23:30:32.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.759+0000 7f377e59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f37800c6750 0x7f37800c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:32.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.759+0000 7f377e59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f37800c6750 0x7f37800c88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48946/0 (socket says 172.21.15.161:48946) 2024-03-28T23:30:32.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.759+0000 7f377e59c640 1 -- 172.21.15.161:0/909448979 learned_addr learned my addr 172.21.15.161:0/909448979 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:32.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.759+0000 7f377ed9d640 1 --2- 172.21.15.161:0/909448979 >> v2:172.21.15.170:3300/0 conn(0x7f37801529d0 0x7f378006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:32.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.759+0000 7f377f59e640 1 --2- 172.21.15.161:0/909448979 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3780155c30 0x7f37800c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:32.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.759+0000 7f377e59c640 1 -- 172.21.15.161:0/909448979 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3780155c30 msgr2=0x7f37800c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:32.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.759+0000 7f377e59c640 1 --2- 172.21.15.161:0/909448979 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3780155c30 0x7f37800c54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.759+0000 7f377e59c640 1 -- 172.21.15.161:0/909448979 >> v2:172.21.15.170:3300/0 conn(0x7f37801529d0 msgr2=0x7f378006bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:32.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.759+0000 7f377e59c640 1 --2- 172.21.15.161:0/909448979 >> v2:172.21.15.170:3300/0 conn(0x7f37801529d0 0x7f378006bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.759+0000 7f377e59c640 1 -- 172.21.15.161:0/909448979 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37800c6f30 con 0x7f37800c6750 2024-03-28T23:30:32.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.760+0000 7f377e59c640 1 --2- 172.21.15.161:0/909448979 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37800c6750 0x7f37800c88b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f37800689e0 tx=0x7f377402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63857e92145cfa88 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:32.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.760+0000 7f377dd9b640 1 -- 172.21.15.161:0/909448979 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3774035020 con 0x7f37800c6750 2024-03-28T23:30:32.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.760+0000 7f377dd9b640 1 -- 172.21.15.161:0/909448979 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3774031870 con 0x7f37800c6750 2024-03-28T23:30:32.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.760+0000 7f377dd9b640 1 -- 172.21.15.161:0/909448979 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3774031c40 con 0x7f37800c6750 2024-03-28T23:30:32.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.761+0000 7f378509c640 1 -- 172.21.15.161:0/909448979 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37800c6750 msgr2=0x7f37800c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:32.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.761+0000 7f378509c640 1 --2- 172.21.15.161:0/909448979 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37800c6750 0x7f37800c88b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f37800689e0 tx=0x7f377402b910 comp rx=0 tx=0).stop 2024-03-28T23:30:32.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.761+0000 7f378509c640 1 -- 172.21.15.161:0/909448979 shutdown_connections 2024-03-28T23:30:32.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.761+0000 7f378509c640 1 --2- 172.21.15.161:0/909448979 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3780155c30 0x7f37800c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.761+0000 7f378509c640 1 --2- 172.21.15.161:0/909448979 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37800c6750 0x7f37800c88b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.761+0000 7f378509c640 1 --2- 172.21.15.161:0/909448979 >> v2:172.21.15.170:3300/0 conn(0x7f37801529d0 0x7f378006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.761+0000 7f378509c640 1 -- 172.21.15.161:0/909448979 >> 172.21.15.161:0/909448979 conn(0x7f37800c7430 msgr2=0x7f37800c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:32.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.761+0000 7f378509c640 1 -- 172.21.15.161:0/909448979 shutdown_connections 2024-03-28T23:30:32.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.761+0000 7f378509c640 1 -- 172.21.15.161:0/909448979 wait complete. 2024-03-28T23:30:32.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.763+0000 7f378509c640 1 Processor -- start 2024-03-28T23:30:32.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.763+0000 7f378509c640 1 -- start start 2024-03-28T23:30:32.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.764+0000 7f378509c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37800c6750 0x7f3780154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:32.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.764+0000 7f378509c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37801529d0 0x7f37801eb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:32.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.764+0000 7f378509c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3780155c30 0x7f3780145270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:32.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.764+0000 7f378509c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f37800aa160 con 0x7f37800c6750 2024-03-28T23:30:32.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.764+0000 7f378509c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3780155710 con 0x7f3780155c30 2024-03-28T23:30:32.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.764+0000 7f378509c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f37801560e0 con 0x7f37801529d0 2024-03-28T23:30:32.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.764+0000 7f377ed9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37800c6750 0x7f3780154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:32.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.764+0000 7f377ed9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37800c6750 0x7f3780154450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48958/0 (socket says 172.21.15.161:48958) 2024-03-28T23:30:32.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.764+0000 7f377ed9d640 1 -- 172.21.15.161:0/1959674351 learned_addr learned my addr 172.21.15.161:0/1959674351 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:32.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.764+0000 7f377f59e640 1 --2- 172.21.15.161:0/1959674351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3780155c30 0x7f3780145270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:32.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.764+0000 7f377e59c640 1 --2- 172.21.15.161:0/1959674351 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37801529d0 0x7f37801eb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:32.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.764+0000 7f377ed9d640 1 -- 172.21.15.161:0/1959674351 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37801529d0 msgr2=0x7f37801eb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:32.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.764+0000 7f377ed9d640 1 --2- 172.21.15.161:0/1959674351 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37801529d0 0x7f37801eb2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.764+0000 7f377ed9d640 1 -- 172.21.15.161:0/1959674351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3780155c30 msgr2=0x7f3780145270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:32.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.764+0000 7f377ed9d640 1 --2- 172.21.15.161:0/1959674351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3780155c30 0x7f3780145270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:32.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.764+0000 7f377ed9d640 1 -- 172.21.15.161:0/1959674351 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37800adb10 con 0x7f37800c6750 2024-03-28T23:30:32.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.765+0000 7f377ed9d640 1 --2- 172.21.15.161:0/1959674351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37800c6750 0x7f3780154450 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f377800d800 tx=0x7f377800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:32.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.765+0000 7f37677fe640 1 -- 172.21.15.161:0/1959674351 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3778018020 con 0x7f37800c6750 2024-03-28T23:30:32.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.765+0000 7f37677fe640 1 -- 172.21.15.161:0/1959674351 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37780096a0 con 0x7f37800c6750 2024-03-28T23:30:32.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.765+0000 7f378509c640 1 -- 172.21.15.161:0/1959674351 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37800addf0 con 0x7f37800c6750 2024-03-28T23:30:32.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.765+0000 7f37677fe640 1 -- 172.21.15.161:0/1959674351 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3778009a70 con 0x7f37800c6750 2024-03-28T23:30:32.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.765+0000 7f378509c640 1 -- 172.21.15.161:0/1959674351 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3780154990 con 0x7f37800c6750 2024-03-28T23:30:32.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.766+0000 7f378509c640 1 -- 172.21.15.161:0/1959674351 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37801eabe0 con 0x7f37800c6750 2024-03-28T23:30:32.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.766+0000 7f37677fe640 1 -- 172.21.15.161:0/1959674351 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f377801b020 con 0x7f37800c6750 2024-03-28T23:30:32.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.767+0000 7f37677fe640 1 --2- 172.21.15.161:0/1959674351 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f376c06b4d0 0x7f376c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:32.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.767+0000 7f37677fe640 1 -- 172.21.15.161:0/1959674351 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f377808a5d0 con 0x7f37800c6750 2024-03-28T23:30:32.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.767+0000 7f377e59c640 1 --2- 172.21.15.161:0/1959674351 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f376c06b4d0 0x7f376c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:32.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.767+0000 7f377e59c640 1 --2- 172.21.15.161:0/1959674351 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f376c06b4d0 0x7f376c06d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3774036000 tx=0x7f377403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:32.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:32.775+0000 7f37677fe640 1 -- 172.21.15.161:0/1959674351 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3778022030 con 0x7f37800c6750 2024-03-28T23:30:33.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.024+0000 7f378509c640 1 -- 172.21.15.161:0/1959674351 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f378004f4b0 con 0x7f37800c6750 2024-03-28T23:30:33.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.024+0000 7f37677fe640 1 -- 172.21.15.161:0/1959674351 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f3778058b50 con 0x7f37800c6750 2024-03-28T23:30:33.025 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:30:33.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.027+0000 7f378509c640 1 -- 172.21.15.161:0/1959674351 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f376c06b4d0 msgr2=0x7f376c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:33.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.027+0000 7f378509c640 1 --2- 172.21.15.161:0/1959674351 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f376c06b4d0 0x7f376c06d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3774036000 tx=0x7f377403c040 comp rx=0 tx=0).stop 2024-03-28T23:30:33.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.027+0000 7f378509c640 1 -- 172.21.15.161:0/1959674351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37800c6750 msgr2=0x7f3780154450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:33.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.027+0000 7f378509c640 1 --2- 172.21.15.161:0/1959674351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37800c6750 0x7f3780154450 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f377800d800 tx=0x7f377800dcc0 comp rx=0 tx=0).stop 2024-03-28T23:30:33.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.027+0000 7f378509c640 1 -- 172.21.15.161:0/1959674351 shutdown_connections 2024-03-28T23:30:33.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.027+0000 7f378509c640 1 --2- 172.21.15.161:0/1959674351 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f376c06b4d0 0x7f376c06d990 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:33.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.027+0000 7f378509c640 1 --2- 172.21.15.161:0/1959674351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3780155c30 0x7f3780145270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:33.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.027+0000 7f378509c640 1 --2- 172.21.15.161:0/1959674351 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37801529d0 0x7f37801eb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:33.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.027+0000 7f378509c640 1 --2- 172.21.15.161:0/1959674351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37800c6750 0x7f3780154450 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:33.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.027+0000 7f378509c640 1 -- 172.21.15.161:0/1959674351 >> 172.21.15.161:0/1959674351 conn(0x7f37800c7430 msgr2=0x7f3780151bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:33.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.027+0000 7f378509c640 1 -- 172.21.15.161:0/1959674351 shutdown_connections 2024-03-28T23:30:33.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.027+0000 7f378509c640 1 -- 172.21.15.161:0/1959674351 wait complete. 2024-03-28T23:30:33.039 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T23:30:33.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.142+0000 7f8f2a57a640 1 Processor -- start 2024-03-28T23:30:33.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.142+0000 7f8f2a57a640 1 -- start start 2024-03-28T23:30:33.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.143+0000 7f8f2a57a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8f240725f0 0x7f8f24153a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:33.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.143+0000 7f8f2a57a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8f240c14b0 0x7f8f24156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:33.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.143+0000 7f8f2a57a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f241566e0 con 0x7f8f24073310 2024-03-28T23:30:33.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.143+0000 7f8f2a57a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8f240736f0 con 0x7f8f240725f0 2024-03-28T23:30:33.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.143+0000 7f8f2a57a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f8f24070ab0 con 0x7f8f240c14b0 2024-03-28T23:30:33.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.143+0000 7f8f237fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8f240725f0 0x7f8f24153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:33.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.143+0000 7f8f23fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f8f24073310 0x7f8f2406e8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60086/0 (socket says 172.21.15.161:60086) 2024-03-28T23:30:33.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.143+0000 7f8f23fff640 1 -- 172.21.15.161:0/152126084 learned_addr learned my addr 172.21.15.161:0/152126084 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:33.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.143+0000 7f8f28b17640 1 --2- 172.21.15.161:0/152126084 >> v2:172.21.15.170:3300/0 conn(0x7f8f240c14b0 0x7f8f24156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:33.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.143+0000 7f8f237fe640 1 -- 172.21.15.161:0/152126084 >> v2:172.21.15.170:3300/0 conn(0x7f8f240c14b0 msgr2=0x7f8f24156bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:33.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.143+0000 7f8f237fe640 1 --2- 172.21.15.161:0/152126084 >> v2:172.21.15.170:3300/0 conn(0x7f8f240c14b0 0x7f8f24156bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:33.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.143+0000 7f8f237fe640 1 -- 172.21.15.161:0/152126084 >> v1:172.21.15.161:6789/0 conn(0x7f8f24073310 legacy=0x7f8f2406e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:33.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.143+0000 7f8f22ffd640 1 -- 172.21.15.161:0/152126084 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3708431300 0 0) 0x7f8f241566e0 con 0x7f8f24073310 2024-03-28T23:30:33.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.143+0000 7f8f237fe640 1 -- 172.21.15.161:0/152126084 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f24070d90 con 0x7f8f240725f0 2024-03-28T23:30:33.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.144+0000 7f8f237fe640 1 --2- 172.21.15.161:0/152126084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f240725f0 0x7f8f24153a40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8f240c1940 tx=0x7f8f1802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3e9d25b483eb8c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:33.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.144+0000 7f8f22ffd640 1 -- 172.21.15.161:0/152126084 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f180315a0 con 0x7f8f240725f0 2024-03-28T23:30:33.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.144+0000 7f8f22ffd640 1 -- 172.21.15.161:0/152126084 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f18037070 con 0x7f8f240725f0 2024-03-28T23:30:33.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.144+0000 7f8f22ffd640 1 -- 172.21.15.161:0/152126084 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f18031960 con 0x7f8f240725f0 2024-03-28T23:30:33.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.145+0000 7f8f2a57a640 1 -- 172.21.15.161:0/152126084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f240725f0 msgr2=0x7f8f24153a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:33.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.145+0000 7f8f2a57a640 1 --2- 172.21.15.161:0/152126084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f240725f0 0x7f8f24153a40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8f240c1940 tx=0x7f8f1802b840 comp rx=0 tx=0).stop 2024-03-28T23:30:33.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.145+0000 7f8f2a57a640 1 -- 172.21.15.161:0/152126084 shutdown_connections 2024-03-28T23:30:33.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.145+0000 7f8f2a57a640 1 --2- 172.21.15.161:0/152126084 >> v2:172.21.15.170:3300/0 conn(0x7f8f240c14b0 0x7f8f24156bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:33.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.145+0000 7f8f2a57a640 1 --2- 172.21.15.161:0/152126084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f240725f0 0x7f8f24153a40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:33.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.145+0000 7f8f2a57a640 1 -- 172.21.15.161:0/152126084 >> 172.21.15.161:0/152126084 conn(0x7f8f240c6b30 msgr2=0x7f8f240c6f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:33.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.145+0000 7f8f2a57a640 1 -- 172.21.15.161:0/152126084 shutdown_connections 2024-03-28T23:30:33.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.145+0000 7f8f2a57a640 1 -- 172.21.15.161:0/152126084 wait complete. 2024-03-28T23:30:33.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.146+0000 7f8f2a57a640 1 Processor -- start 2024-03-28T23:30:33.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.147+0000 7f8f2a57a640 1 -- start start 2024-03-28T23:30:33.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.147+0000 7f8f2a57a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f240725f0 0x7f8f240a9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:33.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.147+0000 7f8f2a57a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f24073310 0x7f8f241eca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:33.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.147+0000 7f8f2a57a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f240c14b0 0x7f8f240c2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:33.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.147+0000 7f8f2a57a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2406efb0 con 0x7f8f240c14b0 2024-03-28T23:30:33.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.147+0000 7f8f2a57a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8f240c6840 con 0x7f8f24073310 2024-03-28T23:30:33.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.147+0000 7f8f2a57a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8f2406f130 con 0x7f8f240725f0 2024-03-28T23:30:33.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.147+0000 7f8f28b17640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f240c14b0 0x7f8f240c2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:33.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.147+0000 7f8f28b17640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f240c14b0 0x7f8f240c2780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48986/0 (socket says 172.21.15.161:48986) 2024-03-28T23:30:33.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.147+0000 7f8f28b17640 1 -- 172.21.15.161:0/1916026789 learned_addr learned my addr 172.21.15.161:0/1916026789 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:33.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.148+0000 7f8f237fe640 1 --2- 172.21.15.161:0/1916026789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f24073310 0x7f8f241eca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:33.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.148+0000 7f8f23fff640 1 --2- 172.21.15.161:0/1916026789 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f240725f0 0x7f8f240a9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:33.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.148+0000 7f8f28b17640 1 -- 172.21.15.161:0/1916026789 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f240725f0 msgr2=0x7f8f240a9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:33.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.148+0000 7f8f28b17640 1 --2- 172.21.15.161:0/1916026789 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f240725f0 0x7f8f240a9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:33.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.148+0000 7f8f28b17640 1 -- 172.21.15.161:0/1916026789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f24073310 msgr2=0x7f8f241eca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:33.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.148+0000 7f8f28b17640 1 --2- 172.21.15.161:0/1916026789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f24073310 0x7f8f241eca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:33.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.148+0000 7f8f28b17640 1 -- 172.21.15.161:0/1916026789 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f241ecfd0 con 0x7f8f240c14b0 2024-03-28T23:30:33.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.148+0000 7f8f237fe640 1 --2- 172.21.15.161:0/1916026789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f24073310 0x7f8f241eca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:33.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.148+0000 7f8f28b17640 1 --2- 172.21.15.161:0/1916026789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f240c14b0 0x7f8f240c2780 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c00efc0 tx=0x7f8f1c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:33.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.148+0000 7f8f20ff9640 1 -- 172.21.15.161:0/1916026789 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c018020 con 0x7f8f240c14b0 2024-03-28T23:30:33.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.148+0000 7f8f20ff9640 1 -- 172.21.15.161:0/1916026789 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f1c014480 con 0x7f8f240c14b0 2024-03-28T23:30:33.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.148+0000 7f8f2a57a640 1 -- 172.21.15.161:0/1916026789 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f241e8b40 con 0x7f8f240c14b0 2024-03-28T23:30:33.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.148+0000 7f8f20ff9640 1 -- 172.21.15.161:0/1916026789 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c014870 con 0x7f8f240c14b0 2024-03-28T23:30:33.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.148+0000 7f8f2a57a640 1 -- 172.21.15.161:0/1916026789 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f2414eac0 con 0x7f8f240c14b0 2024-03-28T23:30:33.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.149+0000 7f8f2a57a640 1 -- 172.21.15.161:0/1916026789 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f2404f4b0 con 0x7f8f240c14b0 2024-03-28T23:30:33.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.150+0000 7f8f20ff9640 1 -- 172.21.15.161:0/1916026789 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8f1c005d10 con 0x7f8f240c14b0 2024-03-28T23:30:33.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.150+0000 7f8f20ff9640 1 --2- 172.21.15.161:0/1916026789 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8f1006b4d0 0x7f8f1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:33.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.150+0000 7f8f20ff9640 1 -- 172.21.15.161:0/1916026789 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f8f1c08af00 con 0x7f8f240c14b0 2024-03-28T23:30:33.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.150+0000 7f8f23fff640 1 --2- 172.21.15.161:0/1916026789 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8f1006b4d0 0x7f8f1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:33.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.151+0000 7f8f23fff640 1 --2- 172.21.15.161:0/1916026789 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8f1006b4d0 0x7f8f1006d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8f14000f60 tx=0x7f8f14007b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:33.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.154+0000 7f8f20ff9640 1 -- 172.21.15.161:0/1916026789 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8f1c022070 con 0x7f8f240c14b0 2024-03-28T23:30:33.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.337+0000 7f8f2a57a640 1 -- 172.21.15.161:0/1916026789 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8f2414f740 con 0x7f8f240c14b0 2024-03-28T23:30:33.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.338+0000 7f8f20ff9640 1 -- 172.21.15.161:0/1916026789 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f8f241e8b40 con 0x7f8f240c14b0 2024-03-28T23:30:33.339 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:30:33.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.340+0000 7f8f2a57a640 1 -- 172.21.15.161:0/1916026789 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8f1006b4d0 msgr2=0x7f8f1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:33.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.340+0000 7f8f2a57a640 1 --2- 172.21.15.161:0/1916026789 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8f1006b4d0 0x7f8f1006d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8f14000f60 tx=0x7f8f14007b50 comp rx=0 tx=0).stop 2024-03-28T23:30:33.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.340+0000 7f8f2a57a640 1 -- 172.21.15.161:0/1916026789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f240c14b0 msgr2=0x7f8f240c2780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:33.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.340+0000 7f8f2a57a640 1 --2- 172.21.15.161:0/1916026789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f240c14b0 0x7f8f240c2780 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c00efc0 tx=0x7f8f1c00c490 comp rx=0 tx=0).stop 2024-03-28T23:30:33.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.340+0000 7f8f2a57a640 1 -- 172.21.15.161:0/1916026789 shutdown_connections 2024-03-28T23:30:33.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.340+0000 7f8f2a57a640 1 --2- 172.21.15.161:0/1916026789 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8f1006b4d0 0x7f8f1006d990 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:33.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.340+0000 7f8f2a57a640 1 --2- 172.21.15.161:0/1916026789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f240c14b0 0x7f8f240c2780 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:33.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.340+0000 7f8f2a57a640 1 --2- 172.21.15.161:0/1916026789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f24073310 0x7f8f241eca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:33.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.340+0000 7f8f2a57a640 1 --2- 172.21.15.161:0/1916026789 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f240725f0 0x7f8f240a9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:33.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.340+0000 7f8f2a57a640 1 -- 172.21.15.161:0/1916026789 >> 172.21.15.161:0/1916026789 conn(0x7f8f240c6b30 msgr2=0x7f8f24156bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:33.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.340+0000 7f8f2a57a640 1 -- 172.21.15.161:0/1916026789 shutdown_connections 2024-03-28T23:30:33.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.340+0000 7f8f2a57a640 1 -- 172.21.15.161:0/1916026789 wait complete. 2024-03-28T23:30:33.352 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop' 2024-03-28T23:30:33.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.452+0000 7fee3fec9640 1 Processor -- start 2024-03-28T23:30:33.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.452+0000 7fee3fec9640 1 -- start start 2024-03-28T23:30:33.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.453+0000 7fee3fec9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee380c5410 0x7fee380c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:33.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.453+0000 7fee3fec9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fee381553a0 0x7fee38155820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:33.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.453+0000 7fee3fec9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee38158a30 con 0x7fee38152890 2024-03-28T23:30:33.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.453+0000 7fee3fec9640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fee380c8480 con 0x7fee381553a0 2024-03-28T23:30:33.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.453+0000 7fee3fec9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fee380c5d90 con 0x7fee380c5410 2024-03-28T23:30:33.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.453+0000 7fee3e466640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fee381553a0 0x7fee38155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:33.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.453+0000 7fee3e466640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fee381553a0 0x7fee38155820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48988/0 (socket says 172.21.15.161:48988) 2024-03-28T23:30:33.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.453+0000 7fee3e466640 1 -- 172.21.15.161:0/910710097 learned_addr learned my addr 172.21.15.161:0/910710097 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:33.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.453+0000 7fee3d464640 1 --2- 172.21.15.161:0/910710097 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee380c5410 0x7fee380c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:33.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.454+0000 7fee3e466640 1 -- 172.21.15.161:0/910710097 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee380c5410 msgr2=0x7fee380c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:33.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.454+0000 7fee3e466640 1 --2- 172.21.15.161:0/910710097 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee380c5410 0x7fee380c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:33.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.454+0000 7fee3e466640 1 -- 172.21.15.161:0/910710097 >> v1:172.21.15.170:6789/0 conn(0x7fee38152890 legacy=0x7fee380c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:33.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.454+0000 7fee3e466640 1 -- 172.21.15.161:0/910710097 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee38158d10 con 0x7fee381553a0 2024-03-28T23:30:33.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.454+0000 7fee3e466640 1 --2- 172.21.15.161:0/910710097 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee381553a0 0x7fee38155820 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fee30009e50 tx=0x7fee3002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea05d6c5a29ae7c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:33.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.455+0000 7fee3cc63640 1 -- 172.21.15.161:0/910710097 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3002ca70 con 0x7fee381553a0 2024-03-28T23:30:33.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.455+0000 7fee3cc63640 1 -- 172.21.15.161:0/910710097 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee3002cc10 con 0x7fee381553a0 2024-03-28T23:30:33.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.455+0000 7fee3cc63640 1 -- 172.21.15.161:0/910710097 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee300343f0 con 0x7fee381553a0 2024-03-28T23:30:33.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.455+0000 7fee3fec9640 1 -- 172.21.15.161:0/910710097 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee381553a0 msgr2=0x7fee38155820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:33.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.455+0000 7fee3fec9640 1 --2- 172.21.15.161:0/910710097 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee381553a0 0x7fee38155820 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fee30009e50 tx=0x7fee3002bed0 comp rx=0 tx=0).stop 2024-03-28T23:30:33.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.455+0000 7fee3fec9640 1 -- 172.21.15.161:0/910710097 shutdown_connections 2024-03-28T23:30:33.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.455+0000 7fee3fec9640 1 --2- 172.21.15.161:0/910710097 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee381553a0 0x7fee38155820 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:33.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.455+0000 7fee3fec9640 1 --2- 172.21.15.161:0/910710097 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee380c5410 0x7fee380c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:33.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.455+0000 7fee3fec9640 1 -- 172.21.15.161:0/910710097 >> 172.21.15.161:0/910710097 conn(0x7fee380ac620 msgr2=0x7fee380aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:33.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.455+0000 7fee3fec9640 1 -- 172.21.15.161:0/910710097 shutdown_connections 2024-03-28T23:30:33.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.455+0000 7fee3fec9640 1 -- 172.21.15.161:0/910710097 wait complete. 2024-03-28T23:30:33.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.456+0000 7fee3fec9640 1 Processor -- start 2024-03-28T23:30:33.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.456+0000 7fee3fec9640 1 -- start start 2024-03-28T23:30:33.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.457+0000 7fee3fec9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee380c5410 0x7fee38157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:33.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.457+0000 7fee3fec9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee38152890 0x7fee38149040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:33.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.457+0000 7fee3fec9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee381553a0 0x7fee38149580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:33.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.457+0000 7fee3fec9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fee381592d0 con 0x7fee38152890 2024-03-28T23:30:33.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.457+0000 7fee3fec9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fee38159640 con 0x7fee380c5410 2024-03-28T23:30:33.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.457+0000 7fee3fec9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fee380c90a0 con 0x7fee381553a0 2024-03-28T23:30:33.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.457+0000 7fee3d464640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee38152890 0x7fee38149040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:33.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.457+0000 7fee3d464640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee38152890 0x7fee38149040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48992/0 (socket says 172.21.15.161:48992) 2024-03-28T23:30:33.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.457+0000 7fee3d464640 1 -- 172.21.15.161:0/3252041096 learned_addr learned my addr 172.21.15.161:0/3252041096 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:33.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.457+0000 7fee3dc65640 1 --2- 172.21.15.161:0/3252041096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee380c5410 0x7fee38157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:33.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.457+0000 7fee3e466640 1 --2- 172.21.15.161:0/3252041096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee381553a0 0x7fee38149580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:33.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.457+0000 7fee3d464640 1 -- 172.21.15.161:0/3252041096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee381553a0 msgr2=0x7fee38149580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:33.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.457+0000 7fee3d464640 1 --2- 172.21.15.161:0/3252041096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee381553a0 0x7fee38149580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:33.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.457+0000 7fee3d464640 1 -- 172.21.15.161:0/3252041096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee380c5410 msgr2=0x7fee38157a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:33.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.457+0000 7fee3d464640 1 --2- 172.21.15.161:0/3252041096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee380c5410 0x7fee38157a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:33.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.457+0000 7fee3d464640 1 -- 172.21.15.161:0/3252041096 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee380bf0b0 con 0x7fee38152890 2024-03-28T23:30:33.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.458+0000 7fee3d464640 1 --2- 172.21.15.161:0/3252041096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee38152890 0x7fee38149040 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fee2c00e870 tx=0x7fee2c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:33.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.458+0000 7fee367fc640 1 -- 172.21.15.161:0/3252041096 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c017070 con 0x7fee38152890 2024-03-28T23:30:33.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.458+0000 7fee367fc640 1 -- 172.21.15.161:0/3252041096 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee2c00c940 con 0x7fee38152890 2024-03-28T23:30:33.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.458+0000 7fee367fc640 1 -- 172.21.15.161:0/3252041096 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c00cd10 con 0x7fee38152890 2024-03-28T23:30:33.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.458+0000 7fee3fec9640 1 -- 172.21.15.161:0/3252041096 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee3814fb40 con 0x7fee38152890 2024-03-28T23:30:33.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.458+0000 7fee3fec9640 1 -- 172.21.15.161:0/3252041096 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee380bf320 con 0x7fee38152890 2024-03-28T23:30:33.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.459+0000 7fee3fec9640 1 -- 172.21.15.161:0/3252041096 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee380c6de0 con 0x7fee38152890 2024-03-28T23:30:33.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.459+0000 7fee367fc640 1 -- 172.21.15.161:0/3252041096 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fee2c01a070 con 0x7fee38152890 2024-03-28T23:30:33.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.460+0000 7fee367fc640 1 --2- 172.21.15.161:0/3252041096 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fee2406b4d0 0x7fee2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:33.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.460+0000 7fee367fc640 1 -- 172.21.15.161:0/3252041096 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fee2c08a700 con 0x7fee38152890 2024-03-28T23:30:33.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.460+0000 7fee3dc65640 1 --2- 172.21.15.161:0/3252041096 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fee2406b4d0 0x7fee2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:33.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.460+0000 7fee3dc65640 1 --2- 172.21.15.161:0/3252041096 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fee2406b4d0 0x7fee2406d990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fee28001780 tx=0x7fee28009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:33.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.464+0000 7fee367fc640 1 -- 172.21.15.161:0/3252041096 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fee380c6de0 con 0x7fee38152890 2024-03-28T23:30:33.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:33.728+0000 7fee3fec9640 1 -- 172.21.15.161:0/3252041096 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop"]} v 0) -- 0x7fee3804f4b0 con 0x7fee38152890 2024-03-28T23:30:34.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.139+0000 7fee367fc640 1 -- 172.21.15.161:0/3252041096 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop"]}]=0 v61) ==== 146+0+0 (secure 0 0 0) 0x7fee2c058d40 con 0x7fee38152890 2024-03-28T23:30:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.141+0000 7fee3fec9640 1 -- 172.21.15.161:0/3252041096 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fee2406b4d0 msgr2=0x7fee2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.141+0000 7fee3fec9640 1 --2- 172.21.15.161:0/3252041096 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fee2406b4d0 0x7fee2406d990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fee28001780 tx=0x7fee28009040 comp rx=0 tx=0).stop 2024-03-28T23:30:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.141+0000 7fee3fec9640 1 -- 172.21.15.161:0/3252041096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee38152890 msgr2=0x7fee38149040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.141+0000 7fee3fec9640 1 --2- 172.21.15.161:0/3252041096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee38152890 0x7fee38149040 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fee2c00e870 tx=0x7fee2c00ed30 comp rx=0 tx=0).stop 2024-03-28T23:30:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.141+0000 7fee3fec9640 1 -- 172.21.15.161:0/3252041096 shutdown_connections 2024-03-28T23:30:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.141+0000 7fee3fec9640 1 --2- 172.21.15.161:0/3252041096 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fee2406b4d0 0x7fee2406d990 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.141+0000 7fee3fec9640 1 --2- 172.21.15.161:0/3252041096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee38152890 0x7fee38149040 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.141+0000 7fee3fec9640 1 --2- 172.21.15.161:0/3252041096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee381553a0 0x7fee38149580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.141+0000 7fee3fec9640 1 --2- 172.21.15.161:0/3252041096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee380c5410 0x7fee38157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:34.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.141+0000 7fee3fec9640 1 -- 172.21.15.161:0/3252041096 >> 172.21.15.161:0/3252041096 conn(0x7fee380ac620 msgr2=0x7fee380ca230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:34.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.141+0000 7fee3fec9640 1 -- 172.21.15.161:0/3252041096 shutdown_connections 2024-03-28T23:30:34.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.141+0000 7fee3fec9640 1 -- 172.21.15.161:0/3252041096 wait complete. 2024-03-28T23:30:34.157 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-28T23:30:34.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.263+0000 7faadb31e640 1 Processor -- start 2024-03-28T23:30:34.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.263+0000 7faadb31e640 1 -- start start 2024-03-28T23:30:34.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.264+0000 7faadb31e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7faad40c0330 0x7faad40c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:34.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.264+0000 7faadb31e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faad41589b0 0x7faad4158e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:34.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.264+0000 7faadb31e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faad40c6f60 con 0x7faad4152bc0 2024-03-28T23:30:34.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.264+0000 7faadb31e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7faad40c6120 con 0x7faad40c0330 2024-03-28T23:30:34.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.264+0000 7faadb31e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7faad40c6400 con 0x7faad41589b0 2024-03-28T23:30:34.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.264+0000 7faad90ba640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7faad4152bc0 0x7faad40c8a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60098/0 (socket says 172.21.15.161:60098) 2024-03-28T23:30:34.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.264+0000 7faad90ba640 1 -- 172.21.15.161:0/925211217 learned_addr learned my addr 172.21.15.161:0/925211217 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:34.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.264+0000 7faad98bb640 1 --2- 172.21.15.161:0/925211217 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faad41589b0 0x7faad4158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:34.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.264+0000 7faad88b9640 1 --2- 172.21.15.161:0/925211217 >> v2:172.21.15.170:3300/0 conn(0x7faad40c0330 0x7faad40c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:34.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.264+0000 7faacbfff640 1 -- 172.21.15.161:0/925211217 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 845504112 0 0) 0x7faad40c6f60 con 0x7faad4152bc0 2024-03-28T23:30:34.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.264+0000 7faacbfff640 1 -- 172.21.15.161:0/925211217 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faac0003520 con 0x7faad4152bc0 2024-03-28T23:30:34.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.265+0000 7faacbfff640 1 -- 172.21.15.161:0/925211217 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3451172660 0 0) 0x7faac0003520 con 0x7faad4152bc0 2024-03-28T23:30:34.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.265+0000 7faacbfff640 1 -- 172.21.15.161:0/925211217 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faad41589b0 msgr2=0x7faad4158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:34.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.265+0000 7faacbfff640 1 --2- 172.21.15.161:0/925211217 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faad41589b0 0x7faad4158e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:34.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.265+0000 7faacbfff640 1 -- 172.21.15.161:0/925211217 >> v2:172.21.15.170:3300/0 conn(0x7faad40c0330 msgr2=0x7faad40c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:34.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.265+0000 7faacbfff640 1 --2- 172.21.15.161:0/925211217 >> v2:172.21.15.170:3300/0 conn(0x7faad40c0330 0x7faad40c9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:34.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.265+0000 7faacbfff640 1 -- 172.21.15.161:0/925211217 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faad40c96d0 con 0x7faad4152bc0 2024-03-28T23:30:34.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.265+0000 7faacbfff640 1 -- 172.21.15.161:0/925211217 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7faad0001670 con 0x7faad4152bc0 2024-03-28T23:30:34.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.265+0000 7faacbfff640 1 -- 172.21.15.161:0/925211217 >> v1:172.21.15.161:6789/0 conn(0x7faad4152bc0 legacy=0x7faad40c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:34.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.265+0000 7faacbfff640 1 --2- 172.21.15.161:0/925211217 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faad41589b0 0x7faac0003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:34.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.265+0000 7faacbfff640 1 --2- 172.21.15.161:0/925211217 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faac0004010 0x7faac0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:34.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.265+0000 7faacbfff640 1 --2- 172.21.15.161:0/925211217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faac0004d40 0x7faac00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:34.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.265+0000 7faacbfff640 1 -- 172.21.15.161:0/925211217 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7faad40c6120 con 0x7faac0004010 2024-03-28T23:30:34.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.265+0000 7faacbfff640 1 -- 172.21.15.161:0/925211217 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7faad40c6400 con 0x7faac0004d40 2024-03-28T23:30:34.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.265+0000 7faacbfff640 1 -- 172.21.15.161:0/925211217 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7faac000ba40 con 0x7faad41589b0 2024-03-28T23:30:34.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.265+0000 7faad88b9640 1 --2- 172.21.15.161:0/925211217 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faac0004010 0x7faac0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:34.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.265+0000 7faad90ba640 1 --2- 172.21.15.161:0/925211217 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faad41589b0 0x7faac0003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:34.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.265+0000 7faad98bb640 1 --2- 172.21.15.161:0/925211217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faac0004d40 0x7faac00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:34.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.265+0000 7faad88b9640 1 -- 172.21.15.161:0/925211217 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faad41589b0 msgr2=0x7faac0003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:34.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.265+0000 7faad88b9640 1 --2- 172.21.15.161:0/925211217 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faad41589b0 0x7faac0003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:34.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.265+0000 7faad88b9640 1 -- 172.21.15.161:0/925211217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faac0004d40 msgr2=0x7faac00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:34.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.265+0000 7faad88b9640 1 --2- 172.21.15.161:0/925211217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faac0004d40 0x7faac00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:34.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.265+0000 7faad88b9640 1 -- 172.21.15.161:0/925211217 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faac0003520 con 0x7faac0004010 2024-03-28T23:30:34.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.266+0000 7faad88b9640 1 --2- 172.21.15.161:0/925211217 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faac0004010 0x7faac0004800 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7faac4009ac0 tx=0x7faac0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d9c12604c86079a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:34.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.266+0000 7faacbfff640 1 -- 172.21.15.161:0/925211217 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac4039070 con 0x7faac0004010 2024-03-28T23:30:34.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.266+0000 7faacbfff640 1 -- 172.21.15.161:0/925211217 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faac402c710 con 0x7faac0004010 2024-03-28T23:30:34.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.266+0000 7faacbfff640 1 -- 172.21.15.161:0/925211217 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac4009690 con 0x7faac0004010 2024-03-28T23:30:34.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.267+0000 7faadb31e640 1 -- 172.21.15.161:0/925211217 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faac0004010 msgr2=0x7faac0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:34.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.267+0000 7faadb31e640 1 --2- 172.21.15.161:0/925211217 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faac0004010 0x7faac0004800 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7faac4009ac0 tx=0x7faac0005830 comp rx=0 tx=0).stop 2024-03-28T23:30:34.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.267+0000 7faad98bb640 1 -- 172.21.15.161:0/925211217 reap_dead start 2024-03-28T23:30:34.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.267+0000 7faadb31e640 1 -- 172.21.15.161:0/925211217 shutdown_connections 2024-03-28T23:30:34.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.267+0000 7faadb31e640 1 -- 172.21.15.161:0/925211217 >> 172.21.15.161:0/925211217 conn(0x7faad40caa80 msgr2=0x7faad40cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:34.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.267+0000 7faadb31e640 1 -- 172.21.15.161:0/925211217 shutdown_connections 2024-03-28T23:30:34.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.267+0000 7faadb31e640 1 -- 172.21.15.161:0/925211217 wait complete. 2024-03-28T23:30:34.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.268+0000 7faadb31e640 1 Processor -- start 2024-03-28T23:30:34.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.268+0000 7faadb31e640 1 -- start start 2024-03-28T23:30:34.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.269+0000 7faadb31e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faac0004010 0x7faad41ea560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:34.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.269+0000 7faadb31e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faad406bd30 0x7faad4151790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:34.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.269+0000 7faadb31e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faad406b500 0x7faad40aa4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:34.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.269+0000 7faadb31e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7faad4152fa0 con 0x7faad406b500 2024-03-28T23:30:34.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.269+0000 7faadb31e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7faad40c6bf0 con 0x7faac0004010 2024-03-28T23:30:34.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.269+0000 7faadb31e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7faad41ee4e0 con 0x7faad406bd30 2024-03-28T23:30:34.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.269+0000 7faad98bb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faad406b500 0x7faad40aa4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:34.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.269+0000 7faad98bb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faad406b500 0x7faad40aa4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49020/0 (socket says 172.21.15.161:49020) 2024-03-28T23:30:34.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.269+0000 7faad98bb640 1 -- 172.21.15.161:0/828670052 learned_addr learned my addr 172.21.15.161:0/828670052 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:34.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.269+0000 7faad90ba640 1 --2- 172.21.15.161:0/828670052 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faac0004010 0x7faad41ea560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:34.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.269+0000 7faad88b9640 1 --2- 172.21.15.161:0/828670052 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faad406bd30 0x7faad4151790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:34.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.269+0000 7faad98bb640 1 -- 172.21.15.161:0/828670052 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faad406bd30 msgr2=0x7faad4151790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:34.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.269+0000 7faad98bb640 1 --2- 172.21.15.161:0/828670052 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faad406bd30 0x7faad4151790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:34.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.269+0000 7faad98bb640 1 -- 172.21.15.161:0/828670052 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faac0004010 msgr2=0x7faad41ea560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:34.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.269+0000 7faad98bb640 1 --2- 172.21.15.161:0/828670052 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faac0004010 0x7faad41ea560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:34.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.269+0000 7faad98bb640 1 -- 172.21.15.161:0/828670052 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faad41eaaa0 con 0x7faad406b500 2024-03-28T23:30:34.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.269+0000 7faad98bb640 1 --2- 172.21.15.161:0/828670052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faad406b500 0x7faad40aa4c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7faad40c0fc0 tx=0x7faacc00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:34.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.270+0000 7faac9ffb640 1 -- 172.21.15.161:0/828670052 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faacc00fde0 con 0x7faad406b500 2024-03-28T23:30:34.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.270+0000 7faac9ffb640 1 -- 172.21.15.161:0/828670052 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faacc015430 con 0x7faad406b500 2024-03-28T23:30:34.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.270+0000 7faac9ffb640 1 -- 172.21.15.161:0/828670052 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faacc015820 con 0x7faad406b500 2024-03-28T23:30:34.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.270+0000 7faadb31e640 1 -- 172.21.15.161:0/828670052 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faad40ab670 con 0x7faad406b500 2024-03-28T23:30:34.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.270+0000 7faadb31e640 1 -- 172.21.15.161:0/828670052 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faad4158370 con 0x7faad406b500 2024-03-28T23:30:34.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.271+0000 7faadb31e640 1 -- 172.21.15.161:0/828670052 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faad404f4b0 con 0x7faad406b500 2024-03-28T23:30:34.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.271+0000 7faac9ffb640 1 -- 172.21.15.161:0/828670052 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7faacc0159c0 con 0x7faad406b500 2024-03-28T23:30:34.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.272+0000 7faac9ffb640 1 --2- 172.21.15.161:0/828670052 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faab806b500 0x7faab806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:34.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.272+0000 7faac9ffb640 1 -- 172.21.15.161:0/828670052 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6735+0+0 (secure 0 0 0) 0x7faacc08b1b0 con 0x7faad406b500 2024-03-28T23:30:34.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.272+0000 7faad90ba640 1 --2- 172.21.15.161:0/828670052 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faab806b500 0x7faab806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:34.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.272+0000 7faad90ba640 1 --2- 172.21.15.161:0/828670052 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faab806b500 0x7faab806d9c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7faad4152070 tx=0x7faad00078f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:34.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.276+0000 7faac9ffb640 1 -- 172.21.15.161:0/828670052 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7faacc019030 con 0x7faad406b500 2024-03-28T23:30:34.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.482+0000 7faadb31e640 1 -- 172.21.15.161:0/828670052 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7faad41ee970 con 0x7faad406b500 2024-03-28T23:30:34.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.485+0000 7faac9ffb640 1 -- 172.21.15.161:0/828670052 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v11)=0 v11) ==== 122+0+0 (secure 0 0 0) 0x7faacc010040 con 0x7faad406b500 2024-03-28T23:30:34.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.487+0000 7faadb31e640 1 -- 172.21.15.161:0/828670052 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faab806b500 msgr2=0x7faab806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:34.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.487+0000 7faadb31e640 1 --2- 172.21.15.161:0/828670052 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faab806b500 0x7faab806d9c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7faad4152070 tx=0x7faad00078f0 comp rx=0 tx=0).stop 2024-03-28T23:30:34.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.487+0000 7faadb31e640 1 -- 172.21.15.161:0/828670052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faad406b500 msgr2=0x7faad40aa4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:34.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.487+0000 7faadb31e640 1 --2- 172.21.15.161:0/828670052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faad406b500 0x7faad40aa4c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7faad40c0fc0 tx=0x7faacc00f580 comp rx=0 tx=0).stop 2024-03-28T23:30:34.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.487+0000 7faadb31e640 1 -- 172.21.15.161:0/828670052 shutdown_connections 2024-03-28T23:30:34.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.487+0000 7faadb31e640 1 --2- 172.21.15.161:0/828670052 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faab806b500 0x7faab806d9c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:34.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.487+0000 7faadb31e640 1 --2- 172.21.15.161:0/828670052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faad406b500 0x7faad40aa4c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:34.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.487+0000 7faadb31e640 1 --2- 172.21.15.161:0/828670052 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faad406bd30 0x7faad4151790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:34.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.487+0000 7faadb31e640 1 --2- 172.21.15.161:0/828670052 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faac0004010 0x7faad41ea560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:34.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.487+0000 7faadb31e640 1 -- 172.21.15.161:0/828670052 >> 172.21.15.161:0/828670052 conn(0x7faad40caa80 msgr2=0x7faad40c8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:34.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.487+0000 7faadb31e640 1 -- 172.21.15.161:0/828670052 shutdown_connections 2024-03-28T23:30:34.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.487+0000 7faadb31e640 1 -- 172.21.15.161:0/828670052 wait complete. 2024-03-28T23:30:34.499 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-28T23:30:34.499 DEBUG:teuthology.orchestra.run.smithi161:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:30:34.552 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:30:34.601 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-28T23:30:34.602 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:30:34.602 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:30:34.617 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:30:34.618 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:30:34.673 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:30:34.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.816+0000 7f1e0530d640 1 Processor -- start 2024-03-28T23:30:34.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.816+0000 7f1e0530d640 1 -- start start 2024-03-28T23:30:34.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.817+0000 7f1e0530d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e000c5410 0x7f1e000c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:34.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.817+0000 7f1e0530d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e00158a30 con 0x7f1e001553a0 2024-03-28T23:30:34.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.817+0000 7f1e0530d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e00158d10 con 0x7f1e00152890 2024-03-28T23:30:34.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.817+0000 7f1e0530d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1e000c8480 con 0x7f1e000c5410 2024-03-28T23:30:34.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.817+0000 7f1dff7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f1e001553a0 0x7f1e00155820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60108/0 (socket says 172.21.15.161:60108) 2024-03-28T23:30:34.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.817+0000 7f1dff7fe640 1 -- 172.21.15.161:0/1410836877 learned_addr learned my addr 172.21.15.161:0/1410836877 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:34.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.817+0000 7f1dfe7fc640 1 --2- 172.21.15.161:0/1410836877 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e000c5410 0x7f1e000c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:34.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.818+0000 7f1dfdffb640 1 -- 172.21.15.161:0/1410836877 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2626546733 0 0) 0x7f1e00158a30 con 0x7f1e001553a0 2024-03-28T23:30:34.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.818+0000 7f1dfdffb640 1 -- 172.21.15.161:0/1410836877 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1dec003520 con 0x7f1e001553a0 2024-03-28T23:30:34.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.818+0000 7f1dfdffb640 1 -- 172.21.15.161:0/1410836877 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3675844971 0 0) 0x7f1dec003520 con 0x7f1e001553a0 2024-03-28T23:30:34.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.818+0000 7f1dfdffb640 1 -- 172.21.15.161:0/1410836877 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e000c5410 msgr2=0x7f1e000c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:34.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.818+0000 7f1dfdffb640 1 --2- 172.21.15.161:0/1410836877 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e000c5410 0x7f1e000c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:34.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.818+0000 7f1dfdffb640 1 -- 172.21.15.161:0/1410836877 >> v1:172.21.15.170:6789/0 conn(0x7f1e00152890 legacy=0x7f1e000c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:34.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.818+0000 7f1dfdffb640 1 -- 172.21.15.161:0/1410836877 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e00158ff0 con 0x7f1e001553a0 2024-03-28T23:30:34.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.818+0000 7f1dfdffb640 1 -- 172.21.15.161:0/1410836877 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f1df80019f0 con 0x7f1e001553a0 2024-03-28T23:30:34.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.818+0000 7f1dfdffb640 1 -- 172.21.15.161:0/1410836877 >> v1:172.21.15.161:6789/0 conn(0x7f1e001553a0 legacy=0x7f1e00155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:34.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.818+0000 7f1dfdffb640 1 --2- 172.21.15.161:0/1410836877 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1dec003ac0 0x7f1dec003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:34.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.818+0000 7f1dfdffb640 1 --2- 172.21.15.161:0/1410836877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1dec004800 0x7f1dec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:34.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.818+0000 7f1dfdffb640 1 --2- 172.21.15.161:0/1410836877 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1dec0051c0 0x7f1dec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:34.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.818+0000 7f1dfdffb640 1 -- 172.21.15.161:0/1410836877 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1dec00c090 con 0x7f1dec0051c0 2024-03-28T23:30:34.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.818+0000 7f1dfdffb640 1 -- 172.21.15.161:0/1410836877 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1dec00c240 con 0x7f1dec004800 2024-03-28T23:30:34.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.818+0000 7f1dfdffb640 1 -- 172.21.15.161:0/1410836877 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1dec00c440 con 0x7f1dec003ac0 2024-03-28T23:30:34.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.818+0000 7f1dfe7fc640 1 --2- 172.21.15.161:0/1410836877 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e000c5410 0x7f1e000c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:34.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.818+0000 7f1dff7fe640 1 --2- 172.21.15.161:0/1410836877 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1dec0051c0 0x7f1dec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:34.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.818+0000 7f1dfeffd640 1 --2- 172.21.15.161:0/1410836877 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1dec003ac0 0x7f1dec003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:34.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.818+0000 7f1dfe7fc640 1 --2- 172.21.15.161:0/1410836877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1dec004800 0x7f1dec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:34.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.818+0000 7f1dff7fe640 1 -- 172.21.15.161:0/1410836877 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1dec003ac0 msgr2=0x7f1dec003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:34.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.818+0000 7f1dff7fe640 1 --2- 172.21.15.161:0/1410836877 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1dec003ac0 0x7f1dec003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:34.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.818+0000 7f1dff7fe640 1 -- 172.21.15.161:0/1410836877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1dec004800 msgr2=0x7f1dec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:34.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.818+0000 7f1dff7fe640 1 --2- 172.21.15.161:0/1410836877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1dec004800 0x7f1dec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:34.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.818+0000 7f1dff7fe640 1 -- 172.21.15.161:0/1410836877 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dec003520 con 0x7f1dec0051c0 2024-03-28T23:30:34.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.819+0000 7f1dff7fe640 1 --2- 172.21.15.161:0/1410836877 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1dec0051c0 0x7f1dec00b9c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1df800a810 tx=0x7f1dec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31bf039ca010d710 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:34.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.819+0000 7f1dfdffb640 1 -- 172.21.15.161:0/1410836877 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df800a430 con 0x7f1dec0051c0 2024-03-28T23:30:34.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.819+0000 7f1dfdffb640 1 -- 172.21.15.161:0/1410836877 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1df8034630 con 0x7f1dec0051c0 2024-03-28T23:30:34.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.819+0000 7f1dfdffb640 1 -- 172.21.15.161:0/1410836877 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df8034a60 con 0x7f1dec0051c0 2024-03-28T23:30:34.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.820+0000 7f1e0530d640 1 -- 172.21.15.161:0/1410836877 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1dec0051c0 msgr2=0x7f1dec00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:34.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.820+0000 7f1e0530d640 1 --2- 172.21.15.161:0/1410836877 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1dec0051c0 0x7f1dec00b9c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1df800a810 tx=0x7f1dec005830 comp rx=0 tx=0).stop 2024-03-28T23:30:34.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.820+0000 7f1dff7fe640 1 -- 172.21.15.161:0/1410836877 reap_dead start 2024-03-28T23:30:34.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.820+0000 7f1e0530d640 1 -- 172.21.15.161:0/1410836877 shutdown_connections 2024-03-28T23:30:34.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.820+0000 7f1e0530d640 1 -- 172.21.15.161:0/1410836877 >> 172.21.15.161:0/1410836877 conn(0x7f1e000ac620 msgr2=0x7f1e000aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:34.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.820+0000 7f1e0530d640 1 -- 172.21.15.161:0/1410836877 shutdown_connections 2024-03-28T23:30:34.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.820+0000 7f1e0530d640 1 -- 172.21.15.161:0/1410836877 wait complete. 2024-03-28T23:30:34.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.821+0000 7f1e0530d640 1 Processor -- start 2024-03-28T23:30:34.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.821+0000 7f1e0530d640 1 -- start start 2024-03-28T23:30:34.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.822+0000 7f1e0530d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1dec0051c0 0x7f1e000ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:34.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.822+0000 7f1e0530d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e001500b0 0x7f1e001517e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:34.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.822+0000 7f1e0530d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e000c0e30 0x7f1e000c12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:34.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.822+0000 7f1e0530d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1e000c90a0 con 0x7f1e000c0e30 2024-03-28T23:30:34.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.822+0000 7f1e0530d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1e00159640 con 0x7f1e001500b0 2024-03-28T23:30:34.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.822+0000 7f1e0530d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1e000c9220 con 0x7f1dec0051c0 2024-03-28T23:30:34.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.822+0000 7f1dff7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e000c0e30 0x7f1e000c12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:34.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.822+0000 7f1dff7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e000c0e30 0x7f1e000c12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49038/0 (socket says 172.21.15.161:49038) 2024-03-28T23:30:34.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.822+0000 7f1dff7fe640 1 -- 172.21.15.161:0/975314582 learned_addr learned my addr 172.21.15.161:0/975314582 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:34.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.822+0000 7f1dfeffd640 1 --2- 172.21.15.161:0/975314582 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1dec0051c0 0x7f1e000ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:34.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.822+0000 7f1dfe7fc640 1 --2- 172.21.15.161:0/975314582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e001500b0 0x7f1e001517e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:34.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.822+0000 7f1dff7fe640 1 -- 172.21.15.161:0/975314582 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1dec0051c0 msgr2=0x7f1e000ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:34.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.822+0000 7f1dff7fe640 1 --2- 172.21.15.161:0/975314582 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1dec0051c0 0x7f1e000ab6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:34.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.822+0000 7f1dff7fe640 1 -- 172.21.15.161:0/975314582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e001500b0 msgr2=0x7f1e001517e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:34.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.822+0000 7f1dff7fe640 1 --2- 172.21.15.161:0/975314582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e001500b0 0x7f1e001517e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:34.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.822+0000 7f1dff7fe640 1 -- 172.21.15.161:0/975314582 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e001eec00 con 0x7f1e000c0e30 2024-03-28T23:30:34.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.822+0000 7f1dff7fe640 1 --2- 172.21.15.161:0/975314582 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e000c0e30 0x7f1e000c12e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1df8001c70 tx=0x7f1df8001d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:34.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.823+0000 7f1df77fe640 1 -- 172.21.15.161:0/975314582 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df800a430 con 0x7f1e000c0e30 2024-03-28T23:30:34.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.823+0000 7f1df77fe640 1 -- 172.21.15.161:0/975314582 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1df8044070 con 0x7f1e000c0e30 2024-03-28T23:30:34.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.823+0000 7f1df77fe640 1 -- 172.21.15.161:0/975314582 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df80349f0 con 0x7f1e000c0e30 2024-03-28T23:30:34.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.823+0000 7f1e0530d640 1 -- 172.21.15.161:0/975314582 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e001520a0 con 0x7f1e000c0e30 2024-03-28T23:30:34.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.823+0000 7f1e0530d640 1 -- 172.21.15.161:0/975314582 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e001ee120 con 0x7f1e000c0e30 2024-03-28T23:30:34.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.824+0000 7f1e0530d640 1 -- 172.21.15.161:0/975314582 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e0004f4b0 con 0x7f1e000c0e30 2024-03-28T23:30:34.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.824+0000 7f1df77fe640 1 -- 172.21.15.161:0/975314582 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1df8034b90 con 0x7f1e000c0e30 2024-03-28T23:30:34.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.824+0000 7f1df77fe640 1 --2- 172.21.15.161:0/975314582 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1de006b4d0 0x7f1de006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:34.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.825+0000 7f1df77fe640 1 -- 172.21.15.161:0/975314582 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6735+0+0 (secure 0 0 0) 0x7f1df80ac980 con 0x7f1e000c0e30 2024-03-28T23:30:34.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.825+0000 7f1dfeffd640 1 --2- 172.21.15.161:0/975314582 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1de006b4d0 0x7f1de006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:34.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.825+0000 7f1dfeffd640 1 --2- 172.21.15.161:0/975314582 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1de006b4d0 0x7f1de006d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1de8000c00 tx=0x7f1de8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:34.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:34.829+0000 7f1df77fe640 1 -- 172.21.15.161:0/975314582 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1df807b480 con 0x7f1e000c0e30 2024-03-28T23:30:35.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.007+0000 7f1e0530d640 1 -- 172.21.15.161:0/975314582 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1e000c1820 con 0x7f1e000c0e30 2024-03-28T23:30:35.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.008+0000 7f1df77fe640 1 -- 172.21.15.161:0/975314582 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18937 (secure 0 0 0) 0x7f1df807ad80 con 0x7f1e000c0e30 2024-03-28T23:30:35.009 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:35.009 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":25,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:30:34.139273+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:30:35.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.010+0000 7f1e0530d640 1 -- 172.21.15.161:0/975314582 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1de006b4d0 msgr2=0x7f1de006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:35.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.010+0000 7f1e0530d640 1 --2- 172.21.15.161:0/975314582 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1de006b4d0 0x7f1de006d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1de8000c00 tx=0x7f1de8009040 comp rx=0 tx=0).stop 2024-03-28T23:30:35.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.010+0000 7f1e0530d640 1 -- 172.21.15.161:0/975314582 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e000c0e30 msgr2=0x7f1e000c12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:35.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.010+0000 7f1e0530d640 1 --2- 172.21.15.161:0/975314582 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e000c0e30 0x7f1e000c12e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1df8001c70 tx=0x7f1df8001d40 comp rx=0 tx=0).stop 2024-03-28T23:30:35.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.010+0000 7f1e0530d640 1 -- 172.21.15.161:0/975314582 shutdown_connections 2024-03-28T23:30:35.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.010+0000 7f1e0530d640 1 --2- 172.21.15.161:0/975314582 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1de006b4d0 0x7f1de006d990 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.010+0000 7f1e0530d640 1 --2- 172.21.15.161:0/975314582 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e000c0e30 0x7f1e000c12e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.010+0000 7f1e0530d640 1 --2- 172.21.15.161:0/975314582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e001500b0 0x7f1e001517e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.010+0000 7f1e0530d640 1 --2- 172.21.15.161:0/975314582 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1dec0051c0 0x7f1e000ab6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.010+0000 7f1e0530d640 1 -- 172.21.15.161:0/975314582 >> 172.21.15.161:0/975314582 conn(0x7f1e000ac620 msgr2=0x7f1e0014b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:35.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.010+0000 7f1e0530d640 1 -- 172.21.15.161:0/975314582 shutdown_connections 2024-03-28T23:30:35.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.010+0000 7f1e0530d640 1 -- 172.21.15.161:0/975314582 wait complete. 2024-03-28T23:30:35.026 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:29:44.021375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:29:44.872102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T23:29:48.591063+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T23:30:35.026 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:30:35.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.130+0000 7f457f029640 1 Processor -- start 2024-03-28T23:30:35.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.130+0000 7f457f029640 1 -- start start 2024-03-28T23:30:35.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.131+0000 7f457f029640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f45780a9bb0 0x7f4578153860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:35.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.131+0000 7f457f029640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f45780c57a0 0x7f45780c5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:35.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.131+0000 7f457f029640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45780abb60 con 0x7f45780c1290 2024-03-28T23:30:35.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.131+0000 7f457f029640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f45780ac770 con 0x7f45780c57a0 2024-03-28T23:30:35.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.131+0000 7f457f029640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f45780abe40 con 0x7f45780a9bb0 2024-03-28T23:30:35.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.131+0000 7f457d5c6640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f45780c57a0 0x7f45780c5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:35.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.131+0000 7f456ffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f45780c1290 0x7f4578149660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60120/0 (socket says 172.21.15.161:60120) 2024-03-28T23:30:35.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.131+0000 7f456ffff640 1 -- 172.21.15.161:0/579967276 learned_addr learned my addr 172.21.15.161:0/579967276 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:35.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.131+0000 7f457d5c6640 1 -- 172.21.15.161:0/579967276 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f45780a9bb0 msgr2=0x7f4578153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:35.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.131+0000 7f457d5c6640 1 --2- 172.21.15.161:0/579967276 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f45780a9bb0 0x7f4578153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.131+0000 7f457d5c6640 1 -- 172.21.15.161:0/579967276 >> v1:172.21.15.161:6789/0 conn(0x7f45780c1290 legacy=0x7f4578149660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:35.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.131+0000 7f456f7fe640 1 -- 172.21.15.161:0/579967276 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3609620113 0 0) 0x7f45780abb60 con 0x7f45780c1290 2024-03-28T23:30:35.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.131+0000 7f457d5c6640 1 -- 172.21.15.161:0/579967276 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45780ac120 con 0x7f45780c57a0 2024-03-28T23:30:35.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.132+0000 7f457d5c6640 1 --2- 172.21.15.161:0/579967276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45780c57a0 0x7f45780c5c30 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4570009d90 tx=0x7f457002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=150a2b8dc93a9848 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:35.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.132+0000 7f456f7fe640 1 -- 172.21.15.161:0/579967276 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4570039070 con 0x7f45780c57a0 2024-03-28T23:30:35.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.132+0000 7f456f7fe640 1 -- 172.21.15.161:0/579967276 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f457002cbc0 con 0x7f45780c57a0 2024-03-28T23:30:35.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.132+0000 7f456f7fe640 1 -- 172.21.15.161:0/579967276 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4570034440 con 0x7f45780c57a0 2024-03-28T23:30:35.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.132+0000 7f457f029640 1 -- 172.21.15.161:0/579967276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45780c57a0 msgr2=0x7f45780c5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:35.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.132+0000 7f457f029640 1 --2- 172.21.15.161:0/579967276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45780c57a0 0x7f45780c5c30 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4570009d90 tx=0x7f457002be10 comp rx=0 tx=0).stop 2024-03-28T23:30:35.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.133+0000 7f457f029640 1 -- 172.21.15.161:0/579967276 shutdown_connections 2024-03-28T23:30:35.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.133+0000 7f457f029640 1 --2- 172.21.15.161:0/579967276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45780c57a0 0x7f45780c5c30 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.133+0000 7f457f029640 1 --2- 172.21.15.161:0/579967276 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f45780a9bb0 0x7f4578153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.133+0000 7f457f029640 1 -- 172.21.15.161:0/579967276 >> 172.21.15.161:0/579967276 conn(0x7f45780c3390 msgr2=0x7f45780c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:35.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.133+0000 7f457f029640 1 -- 172.21.15.161:0/579967276 shutdown_connections 2024-03-28T23:30:35.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.133+0000 7f457f029640 1 -- 172.21.15.161:0/579967276 wait complete. 2024-03-28T23:30:35.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.134+0000 7f457f029640 1 Processor -- start 2024-03-28T23:30:35.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.134+0000 7f457f029640 1 -- start start 2024-03-28T23:30:35.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.134+0000 7f457f029640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45780a9bb0 0x7f4578148ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:35.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.134+0000 7f457f029640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45780c1290 0x7f457814a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:35.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.135+0000 7f457f029640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f45780c57a0 0x7f457814ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:35.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.135+0000 7f457f029640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4578153da0 con 0x7f45780a9bb0 2024-03-28T23:30:35.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.135+0000 7f457f029640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f45780c6170 con 0x7f45780c1290 2024-03-28T23:30:35.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.135+0000 7f457f029640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4578149d70 con 0x7f45780c57a0 2024-03-28T23:30:35.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.135+0000 7f457cdc5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45780a9bb0 0x7f4578148ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:35.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.135+0000 7f457cdc5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45780a9bb0 0x7f4578148ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49066/0 (socket says 172.21.15.161:49066) 2024-03-28T23:30:35.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.135+0000 7f457cdc5640 1 -- 172.21.15.161:0/1383211115 learned_addr learned my addr 172.21.15.161:0/1383211115 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:35.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.135+0000 7f457d5c6640 1 --2- 172.21.15.161:0/1383211115 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f45780c57a0 0x7f457814ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:35.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.135+0000 7f456ffff640 1 --2- 172.21.15.161:0/1383211115 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45780c1290 0x7f457814a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:35.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.135+0000 7f457cdc5640 1 -- 172.21.15.161:0/1383211115 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f45780c57a0 msgr2=0x7f457814ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:35.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.135+0000 7f457cdc5640 1 --2- 172.21.15.161:0/1383211115 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f45780c57a0 0x7f457814ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.135+0000 7f457cdc5640 1 -- 172.21.15.161:0/1383211115 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45780c1290 msgr2=0x7f457814a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:35.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.135+0000 7f457cdc5640 1 --2- 172.21.15.161:0/1383211115 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45780c1290 0x7f457814a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.135+0000 7f457cdc5640 1 -- 172.21.15.161:0/1383211115 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f457814e590 con 0x7f45780a9bb0 2024-03-28T23:30:35.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.135+0000 7f457d5c6640 1 --2- 172.21.15.161:0/1383211115 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f45780c57a0 0x7f457814ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:35.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.135+0000 7f457cdc5640 1 --2- 172.21.15.161:0/1383211115 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45780a9bb0 0x7f4578148ba0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f457400b4d0 tx=0x7f457400b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:35.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.135+0000 7f456d7fa640 1 -- 172.21.15.161:0/1383211115 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4574014070 con 0x7f45780a9bb0 2024-03-28T23:30:35.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.135+0000 7f456d7fa640 1 -- 172.21.15.161:0/1383211115 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4574010870 con 0x7f45780a9bb0 2024-03-28T23:30:35.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.135+0000 7f457f029640 1 -- 172.21.15.161:0/1383211115 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45780721d0 con 0x7f45780a9bb0 2024-03-28T23:30:35.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.135+0000 7f456d7fa640 1 -- 172.21.15.161:0/1383211115 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4574010c60 con 0x7f45780a9bb0 2024-03-28T23:30:35.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.135+0000 7f457f029640 1 -- 172.21.15.161:0/1383211115 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4578072790 con 0x7f45780a9bb0 2024-03-28T23:30:35.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.136+0000 7f457f029640 1 -- 172.21.15.161:0/1383211115 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4578068870 con 0x7f45780a9bb0 2024-03-28T23:30:35.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.137+0000 7f456d7fa640 1 -- 172.21.15.161:0/1383211115 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4574017070 con 0x7f45780a9bb0 2024-03-28T23:30:35.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.137+0000 7f456d7fa640 1 --2- 172.21.15.161:0/1383211115 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f456406b500 0x7f456406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:35.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.137+0000 7f456d7fa640 1 -- 172.21.15.161:0/1383211115 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6735+0+0 (secure 0 0 0) 0x7f457408a290 con 0x7f45780a9bb0 2024-03-28T23:30:35.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.137+0000 7f456ffff640 1 --2- 172.21.15.161:0/1383211115 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f456406b500 0x7f456406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:35.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.138+0000 7f456ffff640 1 --2- 172.21.15.161:0/1383211115 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f456406b500 0x7f456406d9c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4578150b20 tx=0x7f4568007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:35.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.142+0000 7f456d7fa640 1 -- 172.21.15.161:0/1383211115 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4578068870 con 0x7f45780a9bb0 2024-03-28T23:30:35.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.327+0000 7f457f029640 1 -- 172.21.15.161:0/1383211115 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f45780c63d0 con 0x7f45780a9bb0 2024-03-28T23:30:35.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.328+0000 7f456d7fa640 1 -- 172.21.15.161:0/1383211115 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f45740586c0 con 0x7f45780a9bb0 2024-03-28T23:30:35.329 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:30:35.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.330+0000 7f457f029640 1 -- 172.21.15.161:0/1383211115 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f456406b500 msgr2=0x7f456406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:35.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.330+0000 7f457f029640 1 --2- 172.21.15.161:0/1383211115 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f456406b500 0x7f456406d9c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4578150b20 tx=0x7f4568007d50 comp rx=0 tx=0).stop 2024-03-28T23:30:35.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.330+0000 7f457f029640 1 -- 172.21.15.161:0/1383211115 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45780a9bb0 msgr2=0x7f4578148ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:35.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.330+0000 7f457f029640 1 --2- 172.21.15.161:0/1383211115 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45780a9bb0 0x7f4578148ba0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f457400b4d0 tx=0x7f457400b990 comp rx=0 tx=0).stop 2024-03-28T23:30:35.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.330+0000 7f457f029640 1 -- 172.21.15.161:0/1383211115 shutdown_connections 2024-03-28T23:30:35.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.330+0000 7f457f029640 1 --2- 172.21.15.161:0/1383211115 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f456406b500 0x7f456406d9c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.330+0000 7f457f029640 1 --2- 172.21.15.161:0/1383211115 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f45780c57a0 0x7f457814ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.330+0000 7f457f029640 1 --2- 172.21.15.161:0/1383211115 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45780c1290 0x7f457814a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.330+0000 7f457f029640 1 --2- 172.21.15.161:0/1383211115 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45780a9bb0 0x7f4578148ba0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.330+0000 7f457f029640 1 -- 172.21.15.161:0/1383211115 >> 172.21.15.161:0/1383211115 conn(0x7f45780c3390 msgr2=0x7f45780c3770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:35.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.330+0000 7f457f029640 1 -- 172.21.15.161:0/1383211115 shutdown_connections 2024-03-28T23:30:35.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.330+0000 7f457f029640 1 -- 172.21.15.161:0/1383211115 wait complete. 2024-03-28T23:30:35.342 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:30:35.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.456+0000 7f396f73e640 1 Processor -- start 2024-03-28T23:30:35.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.456+0000 7f396f73e640 1 -- start start 2024-03-28T23:30:35.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.457+0000 7f396f73e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3968152bc0 0x7f39680c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:35.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.457+0000 7f396f73e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39681589b0 0x7f3968158e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:35.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.457+0000 7f396f73e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39680c6f60 con 0x7f39680c8a80 2024-03-28T23:30:35.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.457+0000 7f396f73e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f39680c6120 con 0x7f3968152bc0 2024-03-28T23:30:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.457+0000 7f396f73e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f39680c6400 con 0x7f39681589b0 2024-03-28T23:30:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.457+0000 7f396ccd9640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f39680c8a80 0x7f39680c8ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60134/0 (socket says 172.21.15.161:60134) 2024-03-28T23:30:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.457+0000 7f396ccd9640 1 -- 172.21.15.161:0/2891671999 learned_addr learned my addr 172.21.15.161:0/2891671999 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.457+0000 7f396d4da640 1 --2- 172.21.15.161:0/2891671999 >> v2:172.21.15.170:3300/0 conn(0x7f3968152bc0 0x7f39680c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.457+0000 7f396dcdb640 1 --2- 172.21.15.161:0/2891671999 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39681589b0 0x7f3968158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.457+0000 7f395ffff640 1 -- 172.21.15.161:0/2891671999 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2776175638 0 0) 0x7f39680c6f60 con 0x7f39680c8a80 2024-03-28T23:30:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.457+0000 7f395ffff640 1 -- 172.21.15.161:0/2891671999 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3954003520 con 0x7f39680c8a80 2024-03-28T23:30:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.457+0000 7f396d4da640 1 -- 172.21.15.161:0/2891671999 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39681589b0 msgr2=0x7f3968158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.457+0000 7f396d4da640 1 --2- 172.21.15.161:0/2891671999 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39681589b0 0x7f3968158e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.457+0000 7f396d4da640 1 -- 172.21.15.161:0/2891671999 >> v1:172.21.15.161:6789/0 conn(0x7f39680c8a80 legacy=0x7f39680c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.457+0000 7f396d4da640 1 -- 172.21.15.161:0/2891671999 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39680c7240 con 0x7f3968152bc0 2024-03-28T23:30:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.457+0000 7f396dcdb640 1 --2- 172.21.15.161:0/2891671999 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39681589b0 0x7f3968158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:35.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.458+0000 7f396d4da640 1 --2- 172.21.15.161:0/2891671999 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3968152bc0 0x7f39680c0330 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f39640097a0 tx=0x7f396402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8461b7964568bf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:35.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.458+0000 7f395ffff640 1 -- 172.21.15.161:0/2891671999 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3964035020 con 0x7f3968152bc0 2024-03-28T23:30:35.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.458+0000 7f395ffff640 1 -- 172.21.15.161:0/2891671999 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3964031830 con 0x7f3968152bc0 2024-03-28T23:30:35.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.458+0000 7f395ffff640 1 -- 172.21.15.161:0/2891671999 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3964031be0 con 0x7f3968152bc0 2024-03-28T23:30:35.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.459+0000 7f396f73e640 1 -- 172.21.15.161:0/2891671999 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3968152bc0 msgr2=0x7f39680c0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:35.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.459+0000 7f396f73e640 1 --2- 172.21.15.161:0/2891671999 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3968152bc0 0x7f39680c0330 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f39640097a0 tx=0x7f396402b8d0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.459+0000 7f396f73e640 1 -- 172.21.15.161:0/2891671999 shutdown_connections 2024-03-28T23:30:35.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.459+0000 7f396f73e640 1 --2- 172.21.15.161:0/2891671999 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39681589b0 0x7f3968158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.459+0000 7f396f73e640 1 --2- 172.21.15.161:0/2891671999 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3968152bc0 0x7f39680c0330 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.459+0000 7f396f73e640 1 -- 172.21.15.161:0/2891671999 >> 172.21.15.161:0/2891671999 conn(0x7f39680caa80 msgr2=0x7f39680cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:35.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.459+0000 7f396f73e640 1 -- 172.21.15.161:0/2891671999 shutdown_connections 2024-03-28T23:30:35.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.459+0000 7f396f73e640 1 -- 172.21.15.161:0/2891671999 wait complete. 2024-03-28T23:30:35.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.460+0000 7f396f73e640 1 Processor -- start 2024-03-28T23:30:35.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.460+0000 7f396f73e640 1 -- start start 2024-03-28T23:30:35.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.461+0000 7f396f73e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39680c8a80 0x7f39681eec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:35.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.461+0000 7f396f73e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3968152bc0 0x7f396806bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:35.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.461+0000 7f396f73e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39681589b0 0x7f39681412d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:35.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.461+0000 7f396f73e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3968152fa0 con 0x7f39681589b0 2024-03-28T23:30:35.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.461+0000 7f396f73e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f39680c95f0 con 0x7f3968152bc0 2024-03-28T23:30:35.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.461+0000 7f396f73e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f39681ee790 con 0x7f39680c8a80 2024-03-28T23:30:35.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.461+0000 7f396dcdb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39681589b0 0x7f39681412d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:35.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.461+0000 7f396dcdb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39681589b0 0x7f39681412d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49070/0 (socket says 172.21.15.161:49070) 2024-03-28T23:30:35.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.461+0000 7f396dcdb640 1 -- 172.21.15.161:0/2523491413 learned_addr learned my addr 172.21.15.161:0/2523491413 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:35.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.461+0000 7f396ccd9640 1 --2- 172.21.15.161:0/2523491413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3968152bc0 0x7f396806bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:35.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.461+0000 7f396d4da640 1 --2- 172.21.15.161:0/2523491413 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39680c8a80 0x7f39681eec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:35.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.461+0000 7f396dcdb640 1 -- 172.21.15.161:0/2523491413 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39680c8a80 msgr2=0x7f39681eec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:35.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.461+0000 7f396dcdb640 1 --2- 172.21.15.161:0/2523491413 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39680c8a80 0x7f39681eec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.461+0000 7f396dcdb640 1 -- 172.21.15.161:0/2523491413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3968152bc0 msgr2=0x7f396806bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:35.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.461+0000 7f396dcdb640 1 --2- 172.21.15.161:0/2523491413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3968152bc0 0x7f396806bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.461+0000 7f396dcdb640 1 -- 172.21.15.161:0/2523491413 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3968141810 con 0x7f39681589b0 2024-03-28T23:30:35.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.461+0000 7f396dcdb640 1 --2- 172.21.15.161:0/2523491413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39681589b0 0x7f39681412d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f396000efc0 tx=0x7f396000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:35.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.462+0000 7f395dffb640 1 -- 172.21.15.161:0/2523491413 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3960018030 con 0x7f39681589b0 2024-03-28T23:30:35.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.462+0000 7f395dffb640 1 -- 172.21.15.161:0/2523491413 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f396000cdf0 con 0x7f39681589b0 2024-03-28T23:30:35.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.462+0000 7f396f73e640 1 -- 172.21.15.161:0/2523491413 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3968151650 con 0x7f39681589b0 2024-03-28T23:30:35.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.462+0000 7f395dffb640 1 -- 172.21.15.161:0/2523491413 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3960003bb0 con 0x7f39681589b0 2024-03-28T23:30:35.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.462+0000 7f396f73e640 1 -- 172.21.15.161:0/2523491413 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39681ee140 con 0x7f39681589b0 2024-03-28T23:30:35.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.463+0000 7f396f73e640 1 -- 172.21.15.161:0/2523491413 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39680c66e0 con 0x7f39681589b0 2024-03-28T23:30:35.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.464+0000 7f395dffb640 1 -- 172.21.15.161:0/2523491413 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3960005d10 con 0x7f39681589b0 2024-03-28T23:30:35.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.464+0000 7f395dffb640 1 --2- 172.21.15.161:0/2523491413 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f394c06b4d0 0x7f394c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:35.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.464+0000 7f395dffb640 1 -- 172.21.15.161:0/2523491413 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f396008adf0 con 0x7f39681589b0 2024-03-28T23:30:35.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.464+0000 7f396d4da640 1 --2- 172.21.15.161:0/2523491413 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f394c06b4d0 0x7f394c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:35.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.465+0000 7f396d4da640 1 --2- 172.21.15.161:0/2523491413 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f394c06b4d0 0x7f394c06d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3964036000 tx=0x7f3964008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:35.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.469+0000 7f395dffb640 1 -- 172.21.15.161:0/2523491413 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f39680c66e0 con 0x7f39681589b0 2024-03-28T23:30:35.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.639+0000 7f396f73e640 1 -- 172.21.15.161:0/2523491413 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f396804f4b0 con 0x7f39681589b0 2024-03-28T23:30:35.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.640+0000 7f395dffb640 1 -- 172.21.15.161:0/2523491413 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f3960022070 con 0x7f39681589b0 2024-03-28T23:30:35.641 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:30:35.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.642+0000 7f396f73e640 1 -- 172.21.15.161:0/2523491413 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f394c06b4d0 msgr2=0x7f394c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:35.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.642+0000 7f396f73e640 1 --2- 172.21.15.161:0/2523491413 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f394c06b4d0 0x7f394c06d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3964036000 tx=0x7f3964008030 comp rx=0 tx=0).stop 2024-03-28T23:30:35.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.642+0000 7f396f73e640 1 -- 172.21.15.161:0/2523491413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39681589b0 msgr2=0x7f39681412d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:35.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.642+0000 7f396f73e640 1 --2- 172.21.15.161:0/2523491413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39681589b0 0x7f39681412d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f396000efc0 tx=0x7f396000c490 comp rx=0 tx=0).stop 2024-03-28T23:30:35.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.642+0000 7f396f73e640 1 -- 172.21.15.161:0/2523491413 shutdown_connections 2024-03-28T23:30:35.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.642+0000 7f396f73e640 1 --2- 172.21.15.161:0/2523491413 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f394c06b4d0 0x7f394c06d990 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.642+0000 7f396f73e640 1 --2- 172.21.15.161:0/2523491413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39681589b0 0x7f39681412d0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.642+0000 7f396f73e640 1 --2- 172.21.15.161:0/2523491413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3968152bc0 0x7f396806bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.642+0000 7f396f73e640 1 --2- 172.21.15.161:0/2523491413 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39680c8a80 0x7f39681eec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.642+0000 7f396f73e640 1 -- 172.21.15.161:0/2523491413 >> 172.21.15.161:0/2523491413 conn(0x7f39680caa80 msgr2=0x7f396814cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:35.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.642+0000 7f396f73e640 1 -- 172.21.15.161:0/2523491413 shutdown_connections 2024-03-28T23:30:35.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.642+0000 7f396f73e640 1 -- 172.21.15.161:0/2523491413 wait complete. 2024-03-28T23:30:35.654 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:30:35.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.761+0000 7f6c7f0f4640 1 Processor -- start 2024-03-28T23:30:35.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.761+0000 7f6c7f0f4640 1 -- start start 2024-03-28T23:30:35.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.762+0000 7f6c7f0f4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6c780c6750 0x7f6c7806bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:35.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.762+0000 7f6c7f0f4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c78152db0 con 0x7f6c781529d0 2024-03-28T23:30:35.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.762+0000 7f6c7f0f4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c780c8d40 con 0x7f6c780c88b0 2024-03-28T23:30:35.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.762+0000 7f6c7f0f4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6c780c7210 con 0x7f6c780c6750 2024-03-28T23:30:35.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.762+0000 7f6c77fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6c780c6750 0x7f6c7806bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:35.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.762+0000 7f6c7ce90640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6c781529d0 0x7f6c78155c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60148/0 (socket says 172.21.15.161:60148) 2024-03-28T23:30:35.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.762+0000 7f6c7ce90640 1 -- 172.21.15.161:0/2569315722 learned_addr learned my addr 172.21.15.161:0/2569315722 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:35.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.762+0000 7f6c77fff640 1 --2- 172.21.15.161:0/2569315722 >> v2:172.21.15.161:3300/0 conn(0x7f6c780c6750 0x7f6c7806bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49084/0 (socket says 172.21.15.161:49084) 2024-03-28T23:30:35.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.763+0000 7f6c77fff640 1 -- 172.21.15.161:0/2569315722 >> v1:172.21.15.170:6789/0 conn(0x7f6c780c88b0 legacy=0x7f6c780c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:35.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.763+0000 7f6c77fff640 1 -- 172.21.15.161:0/2569315722 >> v1:172.21.15.161:6789/0 conn(0x7f6c781529d0 legacy=0x7f6c78155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:35.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.763+0000 7f6c77fff640 1 -- 172.21.15.161:0/2569315722 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c780c6c50 con 0x7f6c780c6750 2024-03-28T23:30:35.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.763+0000 7f6c77fff640 1 --2- 172.21.15.161:0/2569315722 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c780c6750 0x7f6c7806bb60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c0097a0 tx=0x7f6c6c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=984cc0664ca678e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:35.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.763+0000 7f6c777fe640 1 -- 172.21.15.161:0/2569315722 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c035020 con 0x7f6c780c6750 2024-03-28T23:30:35.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.764+0000 7f6c777fe640 1 -- 172.21.15.161:0/2569315722 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c6c031830 con 0x7f6c780c6750 2024-03-28T23:30:35.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.764+0000 7f6c777fe640 1 -- 172.21.15.161:0/2569315722 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c031c00 con 0x7f6c780c6750 2024-03-28T23:30:35.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.764+0000 7f6c7f0f4640 1 -- 172.21.15.161:0/2569315722 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c780c6750 msgr2=0x7f6c7806bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:35.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.764+0000 7f6c7f0f4640 1 --2- 172.21.15.161:0/2569315722 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c780c6750 0x7f6c7806bb60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c0097a0 tx=0x7f6c6c02b8d0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.764+0000 7f6c7f0f4640 1 -- 172.21.15.161:0/2569315722 shutdown_connections 2024-03-28T23:30:35.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.764+0000 7f6c7f0f4640 1 --2- 172.21.15.161:0/2569315722 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c780c6750 0x7f6c7806bb60 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.764+0000 7f6c7f0f4640 1 -- 172.21.15.161:0/2569315722 >> 172.21.15.161:0/2569315722 conn(0x7f6c780c7430 msgr2=0x7f6c780c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:35.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.764+0000 7f6c7f0f4640 1 -- 172.21.15.161:0/2569315722 shutdown_connections 2024-03-28T23:30:35.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.764+0000 7f6c7f0f4640 1 -- 172.21.15.161:0/2569315722 wait complete. 2024-03-28T23:30:35.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.765+0000 7f6c7f0f4640 1 Processor -- start 2024-03-28T23:30:35.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.765+0000 7f6c7f0f4640 1 -- start start 2024-03-28T23:30:35.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.766+0000 7f6c7f0f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c780c6750 0x7f6c781df1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:35.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.766+0000 7f6c7f0f4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c780c88b0 0x7f6c781df730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:35.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.766+0000 7f6c7f0f4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c781529d0 0x7f6c78144af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:35.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.766+0000 7f6c7f0f4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6c78155710 con 0x7f6c780c88b0 2024-03-28T23:30:35.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.766+0000 7f6c7f0f4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6c78155430 con 0x7f6c781529d0 2024-03-28T23:30:35.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.766+0000 7f6c7f0f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6c780aa160 con 0x7f6c780c6750 2024-03-28T23:30:35.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.766+0000 7f6c77fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c780c88b0 0x7f6c781df730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:35.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.766+0000 7f6c77fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c780c88b0 0x7f6c781df730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49092/0 (socket says 172.21.15.161:49092) 2024-03-28T23:30:35.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.766+0000 7f6c77fff640 1 -- 172.21.15.161:0/1354881539 learned_addr learned my addr 172.21.15.161:0/1354881539 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:35.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.766+0000 7f6c77fff640 1 -- 172.21.15.161:0/1354881539 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c780c6750 msgr2=0x7f6c781df1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:35.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.766+0000 7f6c77fff640 1 --2- 172.21.15.161:0/1354881539 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c780c6750 0x7f6c781df1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.766+0000 7f6c77fff640 1 -- 172.21.15.161:0/1354881539 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c781529d0 msgr2=0x7f6c78144af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:35.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.766+0000 7f6c77fff640 1 --2- 172.21.15.161:0/1354881539 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c781529d0 0x7f6c78144af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.766+0000 7f6c77fff640 1 -- 172.21.15.161:0/1354881539 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c781dfc70 con 0x7f6c780c88b0 2024-03-28T23:30:35.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.767+0000 7f6c77fff640 1 --2- 172.21.15.161:0/1354881539 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c780c88b0 0x7f6c781df730 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c0098c0 tx=0x7f6c6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:35.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.767+0000 7f6c757fa640 1 -- 172.21.15.161:0/1354881539 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c03ba80 con 0x7f6c780c88b0 2024-03-28T23:30:35.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.767+0000 7f6c757fa640 1 -- 172.21.15.161:0/1354881539 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c6c03bc40 con 0x7f6c780c88b0 2024-03-28T23:30:35.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.767+0000 7f6c7f0f4640 1 -- 172.21.15.161:0/1354881539 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c781e15e0 con 0x7f6c780c88b0 2024-03-28T23:30:35.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.767+0000 7f6c757fa640 1 -- 172.21.15.161:0/1354881539 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c0453f0 con 0x7f6c780c88b0 2024-03-28T23:30:35.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.767+0000 7f6c7f0f4640 1 -- 172.21.15.161:0/1354881539 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c781ebc50 con 0x7f6c780c88b0 2024-03-28T23:30:35.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.768+0000 7f6c7f0f4640 1 -- 172.21.15.161:0/1354881539 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c7804f4b0 con 0x7f6c780c88b0 2024-03-28T23:30:35.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.768+0000 7f6c757fa640 1 -- 172.21.15.161:0/1354881539 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6c6c04c060 con 0x7f6c780c88b0 2024-03-28T23:30:35.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.769+0000 7f6c757fa640 1 --2- 172.21.15.161:0/1354881539 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6c6406b500 0x7f6c6406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:35.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.769+0000 7f6c757fa640 1 -- 172.21.15.161:0/1354881539 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f6c6c0ac390 con 0x7f6c780c88b0 2024-03-28T23:30:35.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.769+0000 7f6c7ce90640 1 --2- 172.21.15.161:0/1354881539 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6c6406b500 0x7f6c6406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:35.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.769+0000 7f6c7ce90640 1 --2- 172.21.15.161:0/1354881539 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6c6406b500 0x7f6c6406d9c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6c700012e0 tx=0x7f6c70001590 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:35.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.773+0000 7f6c757fa640 1 -- 172.21.15.161:0/1354881539 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6c6c040070 con 0x7f6c780c88b0 2024-03-28T23:30:35.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.948+0000 7f6c7f0f4640 1 -- 172.21.15.161:0/1354881539 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6c780c8180 con 0x7f6c780c88b0 2024-03-28T23:30:35.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.949+0000 7f6c757fa640 1 -- 172.21.15.161:0/1354881539 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f6c6c07ae10 con 0x7f6c780c88b0 2024-03-28T23:30:35.950 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:30:35.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.951+0000 7f6c7f0f4640 1 -- 172.21.15.161:0/1354881539 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6c6406b500 msgr2=0x7f6c6406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:35.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.951+0000 7f6c7f0f4640 1 --2- 172.21.15.161:0/1354881539 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6c6406b500 0x7f6c6406d9c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6c700012e0 tx=0x7f6c70001590 comp rx=0 tx=0).stop 2024-03-28T23:30:35.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.951+0000 7f6c7f0f4640 1 -- 172.21.15.161:0/1354881539 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c780c88b0 msgr2=0x7f6c781df730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:35.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.951+0000 7f6c7f0f4640 1 --2- 172.21.15.161:0/1354881539 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c780c88b0 0x7f6c781df730 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c0098c0 tx=0x7f6c6c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.951+0000 7f6c7f0f4640 1 -- 172.21.15.161:0/1354881539 shutdown_connections 2024-03-28T23:30:35.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.951+0000 7f6c7f0f4640 1 --2- 172.21.15.161:0/1354881539 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6c6406b500 0x7f6c6406d9c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.951+0000 7f6c7f0f4640 1 --2- 172.21.15.161:0/1354881539 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c780c88b0 0x7f6c781df730 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.951+0000 7f6c7f0f4640 1 --2- 172.21.15.161:0/1354881539 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c781529d0 0x7f6c78144af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.951+0000 7f6c7f0f4640 1 --2- 172.21.15.161:0/1354881539 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c780c6750 0x7f6c781df1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:35.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.951+0000 7f6c7f0f4640 1 -- 172.21.15.161:0/1354881539 >> 172.21.15.161:0/1354881539 conn(0x7f6c780c7430 msgr2=0x7f6c78150fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:35.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.951+0000 7f6c7f0f4640 1 -- 172.21.15.161:0/1354881539 shutdown_connections 2024-03-28T23:30:35.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:35.951+0000 7f6c7f0f4640 1 -- 172.21.15.161:0/1354881539 wait complete. 2024-03-28T23:30:35.963 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T23:30:36.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.066+0000 7f752d0fb640 1 Processor -- start 2024-03-28T23:30:36.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.067+0000 7f752d0fb640 1 -- start start 2024-03-28T23:30:36.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.067+0000 7f752d0fb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7528152bc0 0x7f75280c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:36.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.067+0000 7f752d0fb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75281589b0 0x7f7528158e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:36.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.067+0000 7f752d0fb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75280c6f60 con 0x7f75280c8a80 2024-03-28T23:30:36.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.067+0000 7f752d0fb640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f75280c6120 con 0x7f7528152bc0 2024-03-28T23:30:36.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.067+0000 7f752d0fb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f75280c6400 con 0x7f75281589b0 2024-03-28T23:30:36.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.067+0000 7f7526d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7528152bc0 0x7f75280c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:36.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.067+0000 7f751ffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f75280c8a80 0x7f75280c8ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60164/0 (socket says 172.21.15.161:60164) 2024-03-28T23:30:36.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.067+0000 7f751ffff640 1 -- 172.21.15.161:0/430057555 learned_addr learned my addr 172.21.15.161:0/430057555 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:36.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.068+0000 7f7526d9d640 1 -- 172.21.15.161:0/430057555 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75281589b0 msgr2=0x7f7528158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:36.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.068+0000 7f7526d9d640 1 --2- 172.21.15.161:0/430057555 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75281589b0 0x7f7528158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.068+0000 7f7526d9d640 1 -- 172.21.15.161:0/430057555 >> v1:172.21.15.161:6789/0 conn(0x7f75280c8a80 legacy=0x7f75280c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:36.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.068+0000 7f752659c640 1 -- 172.21.15.161:0/430057555 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2966297261 0 0) 0x7f75280c6f60 con 0x7f75280c8a80 2024-03-28T23:30:36.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.068+0000 7f7526d9d640 1 -- 172.21.15.161:0/430057555 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75280c7240 con 0x7f7528152bc0 2024-03-28T23:30:36.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.068+0000 7f7526d9d640 1 --2- 172.21.15.161:0/430057555 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7528152bc0 0x7f75280c0330 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f75180097a0 tx=0x7f751802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54e2db761fe22fc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:36.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.069+0000 7f752659c640 1 -- 172.21.15.161:0/430057555 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7518035020 con 0x7f7528152bc0 2024-03-28T23:30:36.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.069+0000 7f752659c640 1 -- 172.21.15.161:0/430057555 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7518031830 con 0x7f7528152bc0 2024-03-28T23:30:36.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.069+0000 7f752659c640 1 -- 172.21.15.161:0/430057555 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7518031c00 con 0x7f7528152bc0 2024-03-28T23:30:36.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.069+0000 7f752d0fb640 1 -- 172.21.15.161:0/430057555 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7528152bc0 msgr2=0x7f75280c0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:36.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.069+0000 7f752d0fb640 1 --2- 172.21.15.161:0/430057555 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7528152bc0 0x7f75280c0330 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f75180097a0 tx=0x7f751802b8d0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.069+0000 7f752d0fb640 1 -- 172.21.15.161:0/430057555 shutdown_connections 2024-03-28T23:30:36.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.069+0000 7f752d0fb640 1 --2- 172.21.15.161:0/430057555 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75281589b0 0x7f7528158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.069+0000 7f752d0fb640 1 --2- 172.21.15.161:0/430057555 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7528152bc0 0x7f75280c0330 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.069+0000 7f752d0fb640 1 -- 172.21.15.161:0/430057555 >> 172.21.15.161:0/430057555 conn(0x7f75280caa80 msgr2=0x7f75280cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:36.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.069+0000 7f752d0fb640 1 -- 172.21.15.161:0/430057555 shutdown_connections 2024-03-28T23:30:36.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.069+0000 7f752d0fb640 1 -- 172.21.15.161:0/430057555 wait complete. 2024-03-28T23:30:36.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.070+0000 7f752d0fb640 1 Processor -- start 2024-03-28T23:30:36.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.071+0000 7f752d0fb640 1 -- start start 2024-03-28T23:30:36.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.071+0000 7f752d0fb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75280c8a80 0x7f75280adc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:36.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.071+0000 7f752d0fb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7528152bc0 0x7f75281ee5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:36.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.071+0000 7f752d0fb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75281589b0 0x7f75281eeb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:36.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.071+0000 7f752d0fb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7528152fa0 con 0x7f7528152bc0 2024-03-28T23:30:36.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.071+0000 7f752d0fb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f75280c95f0 con 0x7f75280c8a80 2024-03-28T23:30:36.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.071+0000 7f752d0fb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f752806d2f0 con 0x7f75281589b0 2024-03-28T23:30:36.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.071+0000 7f751ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7528152bc0 0x7f75281ee5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:36.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.071+0000 7f751ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7528152bc0 0x7f75281ee5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49122/0 (socket says 172.21.15.161:49122) 2024-03-28T23:30:36.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.071+0000 7f751ffff640 1 -- 172.21.15.161:0/1926233959 learned_addr learned my addr 172.21.15.161:0/1926233959 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:36.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.072+0000 7f752759e640 1 --2- 172.21.15.161:0/1926233959 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75281589b0 0x7f75281eeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:36.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.072+0000 7f751ffff640 1 -- 172.21.15.161:0/1926233959 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75281589b0 msgr2=0x7f75281eeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:36.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.072+0000 7f7526d9d640 1 --2- 172.21.15.161:0/1926233959 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75280c8a80 0x7f75280adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:36.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.072+0000 7f751ffff640 1 --2- 172.21.15.161:0/1926233959 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75281589b0 0x7f75281eeb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.072+0000 7f751ffff640 1 -- 172.21.15.161:0/1926233959 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75280c8a80 msgr2=0x7f75280adc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:36.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.072+0000 7f751ffff640 1 --2- 172.21.15.161:0/1926233959 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75280c8a80 0x7f75280adc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.072+0000 7f751ffff640 1 -- 172.21.15.161:0/1926233959 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7528149030 con 0x7f7528152bc0 2024-03-28T23:30:36.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.072+0000 7f752759e640 1 --2- 172.21.15.161:0/1926233959 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75281589b0 0x7f75281eeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:36.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.072+0000 7f7526d9d640 1 --2- 172.21.15.161:0/1926233959 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75280c8a80 0x7f75280adc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:36.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.072+0000 7f751ffff640 1 --2- 172.21.15.161:0/1926233959 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7528152bc0 0x7f75281ee5c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f751000c4d0 tx=0x7f751000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:36.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.072+0000 7f751f7fe640 1 -- 172.21.15.161:0/1926233959 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7510018070 con 0x7f7528152bc0 2024-03-28T23:30:36.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.072+0000 7f751f7fe640 1 -- 172.21.15.161:0/1926233959 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7510007500 con 0x7f7528152bc0 2024-03-28T23:30:36.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.072+0000 7f752d0fb640 1 -- 172.21.15.161:0/1926233959 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75281ef040 con 0x7f7528152bc0 2024-03-28T23:30:36.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.072+0000 7f751f7fe640 1 -- 172.21.15.161:0/1926233959 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7510004a20 con 0x7f7528152bc0 2024-03-28T23:30:36.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.072+0000 7f752d0fb640 1 -- 172.21.15.161:0/1926233959 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75281492d0 con 0x7f7528152bc0 2024-03-28T23:30:36.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.073+0000 7f752d0fb640 1 -- 172.21.15.161:0/1926233959 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f752804f4b0 con 0x7f7528152bc0 2024-03-28T23:30:36.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.074+0000 7f751f7fe640 1 -- 172.21.15.161:0/1926233959 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f751001b070 con 0x7f7528152bc0 2024-03-28T23:30:36.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.074+0000 7f751f7fe640 1 --2- 172.21.15.161:0/1926233959 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f750806b500 0x7f750806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:36.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.074+0000 7f751f7fe640 1 -- 172.21.15.161:0/1926233959 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f7510089830 con 0x7f7528152bc0 2024-03-28T23:30:36.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.074+0000 7f7526d9d640 1 --2- 172.21.15.161:0/1926233959 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f750806b500 0x7f750806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:36.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.075+0000 7f7526d9d640 1 --2- 172.21.15.161:0/1926233959 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f750806b500 0x7f750806d9c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7518036000 tx=0x7f7518008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:36.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.079+0000 7f751f7fe640 1 -- 172.21.15.161:0/1926233959 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7510058340 con 0x7f7528152bc0 2024-03-28T23:30:36.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.258+0000 7f752d0fb640 1 -- 172.21.15.161:0/1926233959 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f75280c4eb0 con 0x7f7528152bc0 2024-03-28T23:30:36.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.258+0000 7f751f7fe640 1 -- 172.21.15.161:0/1926233959 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f7510057c40 con 0x7f7528152bc0 2024-03-28T23:30:36.259 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:30:36.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.260+0000 7f752d0fb640 1 -- 172.21.15.161:0/1926233959 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f750806b500 msgr2=0x7f750806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:36.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.260+0000 7f752d0fb640 1 --2- 172.21.15.161:0/1926233959 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f750806b500 0x7f750806d9c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7518036000 tx=0x7f7518008030 comp rx=0 tx=0).stop 2024-03-28T23:30:36.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.260+0000 7f752d0fb640 1 -- 172.21.15.161:0/1926233959 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7528152bc0 msgr2=0x7f75281ee5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:36.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.260+0000 7f752d0fb640 1 --2- 172.21.15.161:0/1926233959 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7528152bc0 0x7f75281ee5c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f751000c4d0 tx=0x7f751000c990 comp rx=0 tx=0).stop 2024-03-28T23:30:36.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.260+0000 7f752d0fb640 1 -- 172.21.15.161:0/1926233959 shutdown_connections 2024-03-28T23:30:36.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.260+0000 7f752d0fb640 1 --2- 172.21.15.161:0/1926233959 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f750806b500 0x7f750806d9c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.260+0000 7f752d0fb640 1 --2- 172.21.15.161:0/1926233959 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7528152bc0 0x7f75281ee5c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.260+0000 7f752d0fb640 1 --2- 172.21.15.161:0/1926233959 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75281589b0 0x7f75281eeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.260+0000 7f752d0fb640 1 --2- 172.21.15.161:0/1926233959 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75280c8a80 0x7f75280adc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.260+0000 7f752d0fb640 1 -- 172.21.15.161:0/1926233959 >> 172.21.15.161:0/1926233959 conn(0x7f75280caa80 msgr2=0x7f752814b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:36.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.260+0000 7f752d0fb640 1 -- 172.21.15.161:0/1926233959 shutdown_connections 2024-03-28T23:30:36.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.260+0000 7f752d0fb640 1 -- 172.21.15.161:0/1926233959 wait complete. 2024-03-28T23:30:36.272 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T23:30:36.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.390+0000 7f3287b8a640 1 Processor -- start 2024-03-28T23:30:36.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.390+0000 7f3287b8a640 1 -- start start 2024-03-28T23:30:36.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.391+0000 7f3287b8a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3280152890 0x7f32800c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:36.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.391+0000 7f3287b8a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f32800c8ed0 0x7f32800c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:36.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.391+0000 7f3287b8a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3280158a30 con 0x7f32800c5950 2024-03-28T23:30:36.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.391+0000 7f3287b8a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f32800c5db0 con 0x7f32800c8ed0 2024-03-28T23:30:36.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.391+0000 7f3287b8a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f32800c8480 con 0x7f3280152890 2024-03-28T23:30:36.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.391+0000 7f3285125640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f32800c8ed0 0x7f32800c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:36.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.391+0000 7f3285125640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f32800c8ed0 0x7f32800c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49134/0 (socket says 172.21.15.161:49134) 2024-03-28T23:30:36.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.391+0000 7f3285125640 1 -- 172.21.15.161:0/830712857 learned_addr learned my addr 172.21.15.161:0/830712857 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:36.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.391+0000 7f3285125640 1 -- 172.21.15.161:0/830712857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3280152890 msgr2=0x7f32800c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:36.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.391+0000 7f3285926640 1 --2- 172.21.15.161:0/830712857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3280152890 0x7f32800c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:36.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.391+0000 7f3285125640 1 --2- 172.21.15.161:0/830712857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3280152890 0x7f32800c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.391+0000 7f3285125640 1 -- 172.21.15.161:0/830712857 >> v1:172.21.15.170:6789/0 conn(0x7f32800c5950 legacy=0x7f32801553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:36.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.391+0000 7f3285125640 1 -- 172.21.15.161:0/830712857 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3280158d10 con 0x7f32800c8ed0 2024-03-28T23:30:36.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.392+0000 7f3285125640 1 --2- 172.21.15.161:0/830712857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32800c8ed0 0x7f32800c5410 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f327c009850 tx=0x7f327c02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ba174e97b81a2c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:36.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.392+0000 7f3284924640 1 -- 172.21.15.161:0/830712857 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327c038070 con 0x7f32800c8ed0 2024-03-28T23:30:36.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.392+0000 7f3284924640 1 -- 172.21.15.161:0/830712857 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f327c004230 con 0x7f32800c8ed0 2024-03-28T23:30:36.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.392+0000 7f3284924640 1 -- 172.21.15.161:0/830712857 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327c003610 con 0x7f32800c8ed0 2024-03-28T23:30:36.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.393+0000 7f3287b8a640 1 -- 172.21.15.161:0/830712857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32800c8ed0 msgr2=0x7f32800c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:36.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.393+0000 7f3287b8a640 1 --2- 172.21.15.161:0/830712857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32800c8ed0 0x7f32800c5410 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f327c009850 tx=0x7f327c02b7b0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.393+0000 7f3287b8a640 1 -- 172.21.15.161:0/830712857 shutdown_connections 2024-03-28T23:30:36.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.393+0000 7f3287b8a640 1 --2- 172.21.15.161:0/830712857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32800c8ed0 0x7f32800c5410 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.393+0000 7f3287b8a640 1 --2- 172.21.15.161:0/830712857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3280152890 0x7f32800c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.393+0000 7f3287b8a640 1 -- 172.21.15.161:0/830712857 >> 172.21.15.161:0/830712857 conn(0x7f32800ac620 msgr2=0x7f32800aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:36.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.393+0000 7f3287b8a640 1 -- 172.21.15.161:0/830712857 shutdown_connections 2024-03-28T23:30:36.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.393+0000 7f3287b8a640 1 -- 172.21.15.161:0/830712857 wait complete. 2024-03-28T23:30:36.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.394+0000 7f3287b8a640 1 Processor -- start 2024-03-28T23:30:36.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.394+0000 7f3287b8a640 1 -- start start 2024-03-28T23:30:36.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.395+0000 7f3287b8a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32800c5950 0x7f32800ab6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:36.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.395+0000 7f3287b8a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32800c8ed0 0x7f32800bdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:36.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.395+0000 7f3287b8a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3280152890 0x7f32800be110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:36.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.395+0000 7f3287b8a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f32801592d0 con 0x7f32800c8ed0 2024-03-28T23:30:36.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.395+0000 7f3287b8a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3280159640 con 0x7f32800c5950 2024-03-28T23:30:36.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.395+0000 7f3287b8a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3280152c70 con 0x7f3280152890 2024-03-28T23:30:36.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.395+0000 7f3285125640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32800c8ed0 0x7f32800bdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:36.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.395+0000 7f3285125640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32800c8ed0 0x7f32800bdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49148/0 (socket says 172.21.15.161:49148) 2024-03-28T23:30:36.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.395+0000 7f3285125640 1 -- 172.21.15.161:0/3025779428 learned_addr learned my addr 172.21.15.161:0/3025779428 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:36.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.395+0000 7f3285926640 1 --2- 172.21.15.161:0/3025779428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32800c5950 0x7f32800ab6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:36.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.395+0000 7f3286127640 1 --2- 172.21.15.161:0/3025779428 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3280152890 0x7f32800be110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:36.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.395+0000 7f3285125640 1 -- 172.21.15.161:0/3025779428 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3280152890 msgr2=0x7f32800be110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:36.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.395+0000 7f3285125640 1 --2- 172.21.15.161:0/3025779428 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3280152890 0x7f32800be110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.395+0000 7f3285125640 1 -- 172.21.15.161:0/3025779428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32800c5950 msgr2=0x7f32800ab6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:36.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.395+0000 7f3285125640 1 --2- 172.21.15.161:0/3025779428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32800c5950 0x7f32800ab6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.395+0000 7f3285125640 1 -- 172.21.15.161:0/3025779428 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f328014ffe0 con 0x7f32800c8ed0 2024-03-28T23:30:36.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.395+0000 7f3285125640 1 --2- 172.21.15.161:0/3025779428 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32800c8ed0 0x7f32800bdbd0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f327c000c00 tx=0x7f327c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:36.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.395+0000 7f32767fc640 1 -- 172.21.15.161:0/3025779428 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327c042020 con 0x7f32800c8ed0 2024-03-28T23:30:36.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.395+0000 7f32767fc640 1 -- 172.21.15.161:0/3025779428 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f327c003c40 con 0x7f32800c8ed0 2024-03-28T23:30:36.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.395+0000 7f3287b8a640 1 -- 172.21.15.161:0/3025779428 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32800c1ce0 con 0x7f32800c8ed0 2024-03-28T23:30:36.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.396+0000 7f32767fc640 1 -- 172.21.15.161:0/3025779428 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327c038040 con 0x7f32800c8ed0 2024-03-28T23:30:36.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.396+0000 7f3287b8a640 1 -- 172.21.15.161:0/3025779428 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32800c22a0 con 0x7f32800c8ed0 2024-03-28T23:30:36.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.396+0000 7f3287b8a640 1 -- 172.21.15.161:0/3025779428 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f328004f4b0 con 0x7f32800c8ed0 2024-03-28T23:30:36.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.397+0000 7f32767fc640 1 -- 172.21.15.161:0/3025779428 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f327c045020 con 0x7f32800c8ed0 2024-03-28T23:30:36.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.397+0000 7f32767fc640 1 --2- 172.21.15.161:0/3025779428 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f326406b4d0 0x7f326406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:36.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.398+0000 7f3285926640 1 --2- 172.21.15.161:0/3025779428 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f326406b4d0 0x7f326406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:36.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.398+0000 7f32767fc640 1 -- 172.21.15.161:0/3025779428 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f327c0ac000 con 0x7f32800c8ed0 2024-03-28T23:30:36.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.398+0000 7f3285926640 1 --2- 172.21.15.161:0/3025779428 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f326406b4d0 0x7f326406d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f327000aac0 tx=0x7f3270008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:36.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.404+0000 7f32767fc640 1 -- 172.21.15.161:0/3025779428 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f328014ffe0 con 0x7f32800c8ed0 2024-03-28T23:30:36.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.592+0000 7f3287b8a640 1 -- 172.21.15.161:0/3025779428 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f32800be650 con 0x7f32800c8ed0 2024-03-28T23:30:36.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.592+0000 7f32767fc640 1 -- 172.21.15.161:0/3025779428 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f327c07a4c0 con 0x7f32800c8ed0 2024-03-28T23:30:36.593 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-28T23:30:36.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.594+0000 7f3287b8a640 1 -- 172.21.15.161:0/3025779428 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f326406b4d0 msgr2=0x7f326406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:36.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.594+0000 7f3287b8a640 1 --2- 172.21.15.161:0/3025779428 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f326406b4d0 0x7f326406d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f327000aac0 tx=0x7f3270008040 comp rx=0 tx=0).stop 2024-03-28T23:30:36.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.594+0000 7f3287b8a640 1 -- 172.21.15.161:0/3025779428 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32800c8ed0 msgr2=0x7f32800bdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:36.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.594+0000 7f3287b8a640 1 --2- 172.21.15.161:0/3025779428 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32800c8ed0 0x7f32800bdbd0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f327c000c00 tx=0x7f327c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.594+0000 7f3287b8a640 1 -- 172.21.15.161:0/3025779428 shutdown_connections 2024-03-28T23:30:36.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.594+0000 7f3287b8a640 1 --2- 172.21.15.161:0/3025779428 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f326406b4d0 0x7f326406d990 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.594+0000 7f3287b8a640 1 --2- 172.21.15.161:0/3025779428 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32800c8ed0 0x7f32800bdbd0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.594+0000 7f3287b8a640 1 --2- 172.21.15.161:0/3025779428 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3280152890 0x7f32800be110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.594+0000 7f3287b8a640 1 --2- 172.21.15.161:0/3025779428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32800c5950 0x7f32800ab6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.594+0000 7f3287b8a640 1 -- 172.21.15.161:0/3025779428 >> 172.21.15.161:0/3025779428 conn(0x7f32800ac620 msgr2=0x7f328014aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:36.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.595+0000 7f3287b8a640 1 -- 172.21.15.161:0/3025779428 shutdown_connections 2024-03-28T23:30:36.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.595+0000 7f3287b8a640 1 -- 172.21.15.161:0/3025779428 wait complete. 2024-03-28T23:30:36.606 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:30:36.606 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:30:36.622 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:30:36.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.763+0000 7fbd204f4640 1 Processor -- start 2024-03-28T23:30:36.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.763+0000 7fbd204f4640 1 -- start start 2024-03-28T23:30:36.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.764+0000 7fbd204f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd180c5410 0x7fbd180c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:36.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.764+0000 7fbd204f4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbd181553a0 0x7fbd18155820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:36.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.764+0000 7fbd204f4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd18158a30 con 0x7fbd18152890 2024-03-28T23:30:36.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.764+0000 7fbd204f4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fbd180c8480 con 0x7fbd181553a0 2024-03-28T23:30:36.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.764+0000 7fbd204f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbd180c5d90 con 0x7fbd180c5410 2024-03-28T23:30:36.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.764+0000 7fbd1ea91640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbd181553a0 0x7fbd18155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:36.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.764+0000 7fbd1e290640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fbd18152890 0x7fbd180c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60170/0 (socket says 172.21.15.161:60170) 2024-03-28T23:30:36.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.764+0000 7fbd1e290640 1 -- 172.21.15.161:0/2255932192 learned_addr learned my addr 172.21.15.161:0/2255932192 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:36.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.765+0000 7fbd1da8f640 1 --2- 172.21.15.161:0/2255932192 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd180c5410 0x7fbd180c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:36.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.765+0000 7fbd1ea91640 1 -- 172.21.15.161:0/2255932192 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd180c5410 msgr2=0x7fbd180c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:36.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.765+0000 7fbd1ea91640 1 --2- 172.21.15.161:0/2255932192 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd180c5410 0x7fbd180c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.765+0000 7fbd1ea91640 1 -- 172.21.15.161:0/2255932192 >> v1:172.21.15.161:6789/0 conn(0x7fbd18152890 legacy=0x7fbd180c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:36.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.765+0000 7fbd1d28e640 1 -- 172.21.15.161:0/2255932192 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1739641371 0 0) 0x7fbd18158a30 con 0x7fbd18152890 2024-03-28T23:30:36.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.765+0000 7fbd1ea91640 1 -- 172.21.15.161:0/2255932192 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd18158d10 con 0x7fbd181553a0 2024-03-28T23:30:36.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.766+0000 7fbd1ea91640 1 --2- 172.21.15.161:0/2255932192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd181553a0 0x7fbd18155820 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fbd10009d90 tx=0x7fbd1002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4b4a3270bb6d469 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:36.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.766+0000 7fbd1d28e640 1 -- 172.21.15.161:0/2255932192 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd10039070 con 0x7fbd181553a0 2024-03-28T23:30:36.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.766+0000 7fbd1d28e640 1 -- 172.21.15.161:0/2255932192 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd1002cbc0 con 0x7fbd181553a0 2024-03-28T23:30:36.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.766+0000 7fbd1d28e640 1 -- 172.21.15.161:0/2255932192 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd10034440 con 0x7fbd181553a0 2024-03-28T23:30:36.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.766+0000 7fbd204f4640 1 -- 172.21.15.161:0/2255932192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd181553a0 msgr2=0x7fbd18155820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:36.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.766+0000 7fbd204f4640 1 --2- 172.21.15.161:0/2255932192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd181553a0 0x7fbd18155820 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fbd10009d90 tx=0x7fbd1002be10 comp rx=0 tx=0).stop 2024-03-28T23:30:36.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.766+0000 7fbd204f4640 1 -- 172.21.15.161:0/2255932192 shutdown_connections 2024-03-28T23:30:36.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.766+0000 7fbd204f4640 1 --2- 172.21.15.161:0/2255932192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd181553a0 0x7fbd18155820 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.766+0000 7fbd204f4640 1 --2- 172.21.15.161:0/2255932192 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd180c5410 0x7fbd180c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.766+0000 7fbd204f4640 1 -- 172.21.15.161:0/2255932192 >> 172.21.15.161:0/2255932192 conn(0x7fbd180ac620 msgr2=0x7fbd180aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:36.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.766+0000 7fbd204f4640 1 -- 172.21.15.161:0/2255932192 shutdown_connections 2024-03-28T23:30:36.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.766+0000 7fbd204f4640 1 -- 172.21.15.161:0/2255932192 wait complete. 2024-03-28T23:30:36.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.767+0000 7fbd204f4640 1 Processor -- start 2024-03-28T23:30:36.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.767+0000 7fbd204f4640 1 -- start start 2024-03-28T23:30:36.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.768+0000 7fbd204f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd180c5410 0x7fbd1814f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:36.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.768+0000 7fbd204f4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd18152890 0x7fbd18150c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:36.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.768+0000 7fbd204f4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd181553a0 0x7fbd1814cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:36.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.768+0000 7fbd204f4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbd181592d0 con 0x7fbd18152890 2024-03-28T23:30:36.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.768+0000 7fbd204f4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbd18159640 con 0x7fbd181553a0 2024-03-28T23:30:36.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.768+0000 7fbd204f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbd180c90a0 con 0x7fbd180c5410 2024-03-28T23:30:36.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.768+0000 7fbd1da8f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd18152890 0x7fbd18150c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:36.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.768+0000 7fbd1da8f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd18152890 0x7fbd18150c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49176/0 (socket says 172.21.15.161:49176) 2024-03-28T23:30:36.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.768+0000 7fbd1da8f640 1 -- 172.21.15.161:0/3263083351 learned_addr learned my addr 172.21.15.161:0/3263083351 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:36.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.768+0000 7fbd1ea91640 1 --2- 172.21.15.161:0/3263083351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd181553a0 0x7fbd1814cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:36.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.768+0000 7fbd1e290640 1 --2- 172.21.15.161:0/3263083351 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd180c5410 0x7fbd1814f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:36.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.768+0000 7fbd1da8f640 1 -- 172.21.15.161:0/3263083351 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd180c5410 msgr2=0x7fbd1814f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:36.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.768+0000 7fbd1da8f640 1 --2- 172.21.15.161:0/3263083351 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd180c5410 0x7fbd1814f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.768+0000 7fbd1da8f640 1 -- 172.21.15.161:0/3263083351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd181553a0 msgr2=0x7fbd1814cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:36.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.768+0000 7fbd1da8f640 1 --2- 172.21.15.161:0/3263083351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd181553a0 0x7fbd1814cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.768+0000 7fbd1da8f640 1 -- 172.21.15.161:0/3263083351 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd18151980 con 0x7fbd18152890 2024-03-28T23:30:36.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.769+0000 7fbd1ea91640 1 --2- 172.21.15.161:0/3263083351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd181553a0 0x7fbd1814cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:36.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.769+0000 7fbd1e290640 1 --2- 172.21.15.161:0/3263083351 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd180c5410 0x7fbd1814f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:36.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.769+0000 7fbd1da8f640 1 --2- 172.21.15.161:0/3263083351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd18152890 0x7fbd18150c20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbd0800d830 tx=0x7fbd0800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:36.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.769+0000 7fbd0effd640 1 -- 172.21.15.161:0/3263083351 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd08005990 con 0x7fbd18152890 2024-03-28T23:30:36.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.769+0000 7fbd0effd640 1 -- 172.21.15.161:0/3263083351 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd08005b50 con 0x7fbd18152890 2024-03-28T23:30:36.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.769+0000 7fbd204f4640 1 -- 172.21.15.161:0/3263083351 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd18150290 con 0x7fbd18152890 2024-03-28T23:30:36.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.769+0000 7fbd0effd640 1 -- 172.21.15.161:0/3263083351 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd080096a0 con 0x7fbd18152890 2024-03-28T23:30:36.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.769+0000 7fbd204f4640 1 -- 172.21.15.161:0/3263083351 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd181eba60 con 0x7fbd18152890 2024-03-28T23:30:36.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.770+0000 7fbd204f4640 1 -- 172.21.15.161:0/3263083351 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd18067a50 con 0x7fbd18152890 2024-03-28T23:30:36.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.770+0000 7fbd0effd640 1 -- 172.21.15.161:0/3263083351 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbd08009a50 con 0x7fbd18152890 2024-03-28T23:30:36.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.771+0000 7fbd0effd640 1 --2- 172.21.15.161:0/3263083351 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbcfc06b500 0x7fbcfc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:36.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.771+0000 7fbd1e290640 1 --2- 172.21.15.161:0/3263083351 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbcfc06b500 0x7fbcfc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:36.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.771+0000 7fbd0effd640 1 -- 172.21.15.161:0/3263083351 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fbd0808ab50 con 0x7fbd18152890 2024-03-28T23:30:36.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.771+0000 7fbd1e290640 1 --2- 172.21.15.161:0/3263083351 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbcfc06b500 0x7fbcfc06d9c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbd14000f60 tx=0x7fbd14007c10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:36.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.775+0000 7fbd0effd640 1 -- 172.21.15.161:0/3263083351 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbd18067a50 con 0x7fbd18152890 2024-03-28T23:30:36.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.953+0000 7fbd204f4640 1 -- 172.21.15.161:0/3263083351 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbd1804f4b0 con 0x7fbd18152890 2024-03-28T23:30:36.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.954+0000 7fbd0effd640 1 -- 172.21.15.161:0/3263083351 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18937 (secure 0 0 0) 0x7fbd08059540 con 0x7fbd18152890 2024-03-28T23:30:36.955 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:36.955 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":26,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:30:35.148429+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:30:36.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.956+0000 7fbd204f4640 1 -- 172.21.15.161:0/3263083351 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbcfc06b500 msgr2=0x7fbcfc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:36.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.956+0000 7fbd204f4640 1 --2- 172.21.15.161:0/3263083351 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbcfc06b500 0x7fbcfc06d9c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbd14000f60 tx=0x7fbd14007c10 comp rx=0 tx=0).stop 2024-03-28T23:30:36.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.957+0000 7fbd204f4640 1 -- 172.21.15.161:0/3263083351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd18152890 msgr2=0x7fbd18150c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:36.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.957+0000 7fbd204f4640 1 --2- 172.21.15.161:0/3263083351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd18152890 0x7fbd18150c20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbd0800d830 tx=0x7fbd0800dcf0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.957+0000 7fbd204f4640 1 -- 172.21.15.161:0/3263083351 shutdown_connections 2024-03-28T23:30:36.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.957+0000 7fbd204f4640 1 --2- 172.21.15.161:0/3263083351 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbcfc06b500 0x7fbcfc06d9c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.957+0000 7fbd204f4640 1 --2- 172.21.15.161:0/3263083351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd18152890 0x7fbd18150c20 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.957+0000 7fbd204f4640 1 --2- 172.21.15.161:0/3263083351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd181553a0 0x7fbd1814cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.957+0000 7fbd204f4640 1 --2- 172.21.15.161:0/3263083351 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd180c5410 0x7fbd1814f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:36.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.957+0000 7fbd204f4640 1 -- 172.21.15.161:0/3263083351 >> 172.21.15.161:0/3263083351 conn(0x7fbd180ac620 msgr2=0x7fbd18152f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:36.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.957+0000 7fbd204f4640 1 -- 172.21.15.161:0/3263083351 shutdown_connections 2024-03-28T23:30:36.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:36.957+0000 7fbd204f4640 1 -- 172.21.15.161:0/3263083351 wait complete. 2024-03-28T23:30:36.969 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:29:44.021375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:29:44.872102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T23:29:48.591063+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T23:30:36.969 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:30:37.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.073+0000 7ffad930d640 1 Processor -- start 2024-03-28T23:30:37.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.073+0000 7ffad930d640 1 -- start start 2024-03-28T23:30:37.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.074+0000 7ffad930d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffad40c6750 0x7ffad406bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:37.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.074+0000 7ffad930d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ffad40c88b0 0x7ffad40c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:37.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.074+0000 7ffad930d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffad4152db0 con 0x7ffad41529d0 2024-03-28T23:30:37.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.074+0000 7ffad930d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ffad40c8d60 con 0x7ffad40c88b0 2024-03-28T23:30:37.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.074+0000 7ffad930d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ffad40c7210 con 0x7ffad40c6750 2024-03-28T23:30:37.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.074+0000 7ffad37fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ffad40c88b0 0x7ffad40c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:37.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.074+0000 7ffad37fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ffad40c88b0 0x7ffad40c54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49186/0 (socket says 172.21.15.161:49186) 2024-03-28T23:30:37.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.074+0000 7ffad37fe640 1 -- 172.21.15.161:0/1157896106 learned_addr learned my addr 172.21.15.161:0/1157896106 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:37.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.075+0000 7ffacbfff640 1 --2- 172.21.15.161:0/1157896106 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffad40c6750 0x7ffad406bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:37.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.075+0000 7ffad37fe640 1 -- 172.21.15.161:0/1157896106 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffad40c6750 msgr2=0x7ffad406bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:37.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.075+0000 7ffad37fe640 1 --2- 172.21.15.161:0/1157896106 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffad40c6750 0x7ffad406bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.075+0000 7ffad37fe640 1 -- 172.21.15.161:0/1157896106 >> v1:172.21.15.170:6789/0 conn(0x7ffad41529d0 legacy=0x7ffad4155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:37.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.075+0000 7ffad37fe640 1 -- 172.21.15.161:0/1157896106 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffad40c6c50 con 0x7ffad40c88b0 2024-03-28T23:30:37.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.075+0000 7ffad37fe640 1 --2- 172.21.15.161:0/1157896106 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffad40c88b0 0x7ffad40c54f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ffacc009e50 tx=0x7ffacc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=105b81310d8fbd98 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:37.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.076+0000 7ffad27fc640 1 -- 172.21.15.161:0/1157896106 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffacc02cb00 con 0x7ffad40c88b0 2024-03-28T23:30:37.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.076+0000 7ffad27fc640 1 -- 172.21.15.161:0/1157896106 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffacc02cca0 con 0x7ffad40c88b0 2024-03-28T23:30:37.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.076+0000 7ffad27fc640 1 -- 172.21.15.161:0/1157896106 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffacc034610 con 0x7ffad40c88b0 2024-03-28T23:30:37.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.076+0000 7ffad930d640 1 -- 172.21.15.161:0/1157896106 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffad40c88b0 msgr2=0x7ffad40c54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:37.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.076+0000 7ffad930d640 1 --2- 172.21.15.161:0/1157896106 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffad40c88b0 0x7ffad40c54f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ffacc009e50 tx=0x7ffacc02bf60 comp rx=0 tx=0).stop 2024-03-28T23:30:37.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.076+0000 7ffad930d640 1 -- 172.21.15.161:0/1157896106 shutdown_connections 2024-03-28T23:30:37.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.076+0000 7ffad930d640 1 --2- 172.21.15.161:0/1157896106 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffad40c88b0 0x7ffad40c54f0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.076+0000 7ffad930d640 1 --2- 172.21.15.161:0/1157896106 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffad40c6750 0x7ffad406bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.076+0000 7ffad930d640 1 -- 172.21.15.161:0/1157896106 >> 172.21.15.161:0/1157896106 conn(0x7ffad40c7430 msgr2=0x7ffad40c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:37.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.076+0000 7ffad930d640 1 -- 172.21.15.161:0/1157896106 shutdown_connections 2024-03-28T23:30:37.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.076+0000 7ffad930d640 1 -- 172.21.15.161:0/1157896106 wait complete. 2024-03-28T23:30:37.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.077+0000 7ffad930d640 1 Processor -- start 2024-03-28T23:30:37.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.077+0000 7ffad930d640 1 -- start start 2024-03-28T23:30:37.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.078+0000 7ffad930d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffad40c6750 0x7ffad41eb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:37.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.078+0000 7ffad930d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffad40c88b0 0x7ffad4147d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:37.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.078+0000 7ffad930d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffad41529d0 0x7ffad40b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:37.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.078+0000 7ffad930d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ffad4155710 con 0x7ffad40c6750 2024-03-28T23:30:37.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.078+0000 7ffad930d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ffad4155430 con 0x7ffad41529d0 2024-03-28T23:30:37.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.078+0000 7ffad930d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ffad40aa160 con 0x7ffad40c88b0 2024-03-28T23:30:37.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.078+0000 7ffad2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffad40c6750 0x7ffad41eb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:37.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.078+0000 7ffad2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffad40c6750 0x7ffad41eb9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49196/0 (socket says 172.21.15.161:49196) 2024-03-28T23:30:37.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.078+0000 7ffad2ffd640 1 -- 172.21.15.161:0/1878098150 learned_addr learned my addr 172.21.15.161:0/1878098150 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:37.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.078+0000 7ffacbfff640 1 --2- 172.21.15.161:0/1878098150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffad40c88b0 0x7ffad4147d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:37.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.078+0000 7ffad37fe640 1 --2- 172.21.15.161:0/1878098150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffad41529d0 0x7ffad40b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:37.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.078+0000 7ffad2ffd640 1 -- 172.21.15.161:0/1878098150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffad40c88b0 msgr2=0x7ffad4147d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:37.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.078+0000 7ffad2ffd640 1 --2- 172.21.15.161:0/1878098150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffad40c88b0 0x7ffad4147d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.078+0000 7ffad2ffd640 1 -- 172.21.15.161:0/1878098150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffad41529d0 msgr2=0x7ffad40b00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:37.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.078+0000 7ffad2ffd640 1 --2- 172.21.15.161:0/1878098150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffad41529d0 0x7ffad40b00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.078+0000 7ffad2ffd640 1 -- 172.21.15.161:0/1878098150 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffad4154d60 con 0x7ffad40c6750 2024-03-28T23:30:37.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.079+0000 7ffad2ffd640 1 --2- 172.21.15.161:0/1878098150 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffad40c6750 0x7ffad41eb9b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ffac400c3c0 tx=0x7ffac400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:37.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.079+0000 7ffacb7fe640 1 -- 172.21.15.161:0/1878098150 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac40052d0 con 0x7ffad40c6750 2024-03-28T23:30:37.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.079+0000 7ffacb7fe640 1 -- 172.21.15.161:0/1878098150 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffac4005490 con 0x7ffad40c6750 2024-03-28T23:30:37.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.079+0000 7ffad930d640 1 -- 172.21.15.161:0/1878098150 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffad41547a0 con 0x7ffad40c6750 2024-03-28T23:30:37.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.079+0000 7ffacb7fe640 1 -- 172.21.15.161:0/1878098150 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac4007260 con 0x7ffad40c6750 2024-03-28T23:30:37.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.079+0000 7ffad930d640 1 -- 172.21.15.161:0/1878098150 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffad414a280 con 0x7ffad40c6750 2024-03-28T23:30:37.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.080+0000 7ffad930d640 1 -- 172.21.15.161:0/1878098150 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffad404f4b0 con 0x7ffad40c6750 2024-03-28T23:30:37.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.080+0000 7ffacb7fe640 1 -- 172.21.15.161:0/1878098150 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ffac4007400 con 0x7ffad40c6750 2024-03-28T23:30:37.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.081+0000 7ffacb7fe640 1 --2- 172.21.15.161:0/1878098150 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffab406b500 0x7ffab406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:37.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.081+0000 7ffacb7fe640 1 -- 172.21.15.161:0/1878098150 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7ffac4089a00 con 0x7ffad40c6750 2024-03-28T23:30:37.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.081+0000 7ffacbfff640 1 --2- 172.21.15.161:0/1878098150 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffab406b500 0x7ffab406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:37.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.081+0000 7ffacbfff640 1 --2- 172.21.15.161:0/1878098150 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffab406b500 0x7ffab406d9c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ffad40ab9b0 tx=0x7ffabc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:37.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.085+0000 7ffacb7fe640 1 -- 172.21.15.161:0/1878098150 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ffac4058510 con 0x7ffad40c6750 2024-03-28T23:30:37.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.264+0000 7ffad930d640 1 -- 172.21.15.161:0/1878098150 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ffad406d720 con 0x7ffad40c6750 2024-03-28T23:30:37.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.265+0000 7ffacb7fe640 1 -- 172.21.15.161:0/1878098150 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7ffac4057e10 con 0x7ffad40c6750 2024-03-28T23:30:37.266 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:30:37.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.267+0000 7ffad930d640 1 -- 172.21.15.161:0/1878098150 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffab406b500 msgr2=0x7ffab406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:37.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.267+0000 7ffad930d640 1 --2- 172.21.15.161:0/1878098150 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffab406b500 0x7ffab406d9c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ffad40ab9b0 tx=0x7ffabc008040 comp rx=0 tx=0).stop 2024-03-28T23:30:37.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.267+0000 7ffad930d640 1 -- 172.21.15.161:0/1878098150 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffad40c6750 msgr2=0x7ffad41eb9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:37.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.267+0000 7ffad930d640 1 --2- 172.21.15.161:0/1878098150 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffad40c6750 0x7ffad41eb9b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ffac400c3c0 tx=0x7ffac400c880 comp rx=0 tx=0).stop 2024-03-28T23:30:37.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.267+0000 7ffad930d640 1 -- 172.21.15.161:0/1878098150 shutdown_connections 2024-03-28T23:30:37.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.267+0000 7ffad930d640 1 --2- 172.21.15.161:0/1878098150 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffab406b500 0x7ffab406d9c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.267+0000 7ffad930d640 1 --2- 172.21.15.161:0/1878098150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffad41529d0 0x7ffad40b00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.267+0000 7ffad930d640 1 --2- 172.21.15.161:0/1878098150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffad40c88b0 0x7ffad4147d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.267+0000 7ffad930d640 1 --2- 172.21.15.161:0/1878098150 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffad40c6750 0x7ffad41eb9b0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.267+0000 7ffad930d640 1 -- 172.21.15.161:0/1878098150 >> 172.21.15.161:0/1878098150 conn(0x7ffad40c7430 msgr2=0x7ffad406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:37.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.267+0000 7ffad930d640 1 -- 172.21.15.161:0/1878098150 shutdown_connections 2024-03-28T23:30:37.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.267+0000 7ffad930d640 1 -- 172.21.15.161:0/1878098150 wait complete. 2024-03-28T23:30:37.279 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:30:37.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.403+0000 7f07184c0640 1 Processor -- start 2024-03-28T23:30:37.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.403+0000 7f07184c0640 1 -- start start 2024-03-28T23:30:37.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.404+0000 7f07184c0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f07100a9960 0x7f0710072970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:37.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.404+0000 7f07184c0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f07100c7330 0x7f071014e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:37.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.404+0000 7f07184c0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07100abc00 con 0x7f07100733e0 2024-03-28T23:30:37.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.404+0000 7f07184c0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f07100aa3a0 con 0x7f07100a9960 2024-03-28T23:30:37.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.404+0000 7f07184c0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f071006bf70 con 0x7f07100c7330 2024-03-28T23:30:37.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.404+0000 7f0716a5d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f07100733e0 0x7f071014e9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60182/0 (socket says 172.21.15.161:60182) 2024-03-28T23:30:37.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.404+0000 7f0716a5d640 1 -- 172.21.15.161:0/1998795716 learned_addr learned my addr 172.21.15.161:0/1998795716 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:37.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.404+0000 7f0715a5b640 1 --2- 172.21.15.161:0/1998795716 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f07100c7330 0x7f071014e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:37.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.404+0000 7f071625c640 1 --2- 172.21.15.161:0/1998795716 >> v2:172.21.15.170:3300/0 conn(0x7f07100a9960 0x7f0710072970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:37.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.404+0000 7f071525a640 1 -- 172.21.15.161:0/1998795716 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3225698508 0 0) 0x7f07100abc00 con 0x7f07100733e0 2024-03-28T23:30:37.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.405+0000 7f071525a640 1 -- 172.21.15.161:0/1998795716 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06fc005730 con 0x7f07100733e0 2024-03-28T23:30:37.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.405+0000 7f071525a640 1 -- 172.21.15.161:0/1998795716 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4096290553 0 0) 0x7f06fc005730 con 0x7f07100733e0 2024-03-28T23:30:37.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.405+0000 7f071525a640 1 -- 172.21.15.161:0/1998795716 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f07100c7330 msgr2=0x7f071014e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:37.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.405+0000 7f071525a640 1 --2- 172.21.15.161:0/1998795716 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f07100c7330 0x7f071014e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.405+0000 7f071525a640 1 -- 172.21.15.161:0/1998795716 >> v2:172.21.15.170:3300/0 conn(0x7f07100a9960 msgr2=0x7f0710072970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:37.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.405+0000 7f071525a640 1 --2- 172.21.15.161:0/1998795716 >> v2:172.21.15.170:3300/0 conn(0x7f07100a9960 0x7f0710072970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.405+0000 7f071525a640 1 -- 172.21.15.161:0/1998795716 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f071006bcd0 con 0x7f07100733e0 2024-03-28T23:30:37.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.405+0000 7f071525a640 1 -- 172.21.15.161:0/1998795716 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f07080019f0 con 0x7f07100733e0 2024-03-28T23:30:37.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.405+0000 7f071525a640 1 -- 172.21.15.161:0/1998795716 >> v1:172.21.15.161:6789/0 conn(0x7f07100733e0 legacy=0x7f071014e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:37.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.405+0000 7f071525a640 1 --2- 172.21.15.161:0/1998795716 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06fc005ca0 0x7f06fc0060c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:37.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.405+0000 7f071525a640 1 --2- 172.21.15.161:0/1998795716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06fc006a10 0x7f06fc006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:37.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.405+0000 7f071525a640 1 --2- 172.21.15.161:0/1998795716 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f07100c7330 0x7f06fc00dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:37.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.405+0000 7f071525a640 1 -- 172.21.15.161:0/1998795716 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f071006bf70 con 0x7f06fc005ca0 2024-03-28T23:30:37.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.405+0000 7f071525a640 1 -- 172.21.15.161:0/1998795716 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f06fc007730 con 0x7f06fc006a10 2024-03-28T23:30:37.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.405+0000 7f071525a640 1 -- 172.21.15.161:0/1998795716 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f06fc00e110 con 0x7f07100c7330 2024-03-28T23:30:37.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.405+0000 7f071625c640 1 --2- 172.21.15.161:0/1998795716 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06fc005ca0 0x7f06fc0060c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:37.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.405+0000 7f0716a5d640 1 --2- 172.21.15.161:0/1998795716 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f07100c7330 0x7f06fc00dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:37.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.405+0000 7f0715a5b640 1 --2- 172.21.15.161:0/1998795716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06fc006a10 0x7f06fc006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:37.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.405+0000 7f071625c640 1 -- 172.21.15.161:0/1998795716 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f07100c7330 msgr2=0x7f06fc00dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:37.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.405+0000 7f071625c640 1 --2- 172.21.15.161:0/1998795716 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f07100c7330 0x7f06fc00dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.405+0000 7f071625c640 1 -- 172.21.15.161:0/1998795716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06fc006a10 msgr2=0x7f06fc006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:37.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.405+0000 7f071625c640 1 --2- 172.21.15.161:0/1998795716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06fc006a10 0x7f06fc006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.405+0000 7f071625c640 1 -- 172.21.15.161:0/1998795716 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06fc005730 con 0x7f06fc005ca0 2024-03-28T23:30:37.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.405+0000 7f0715a5b640 1 --2- 172.21.15.161:0/1998795716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06fc006a10 0x7f06fc006e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:37.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.406+0000 7f071625c640 1 --2- 172.21.15.161:0/1998795716 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06fc005ca0 0x7f06fc0060c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f070c00ba30 tx=0x7f06fc007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4026b577da0de371 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:37.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.406+0000 7f071525a640 1 -- 172.21.15.161:0/1998795716 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f070c036020 con 0x7f06fc005ca0 2024-03-28T23:30:37.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.406+0000 7f071525a640 1 -- 172.21.15.161:0/1998795716 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f070c032680 con 0x7f06fc005ca0 2024-03-28T23:30:37.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.406+0000 7f071525a640 1 -- 172.21.15.161:0/1998795716 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f070c00b600 con 0x7f06fc005ca0 2024-03-28T23:30:37.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.407+0000 7f07184c0640 1 -- 172.21.15.161:0/1998795716 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06fc005ca0 msgr2=0x7f06fc0060c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:37.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.407+0000 7f07184c0640 1 --2- 172.21.15.161:0/1998795716 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06fc005ca0 0x7f06fc0060c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f070c00ba30 tx=0x7f06fc007a40 comp rx=0 tx=0).stop 2024-03-28T23:30:37.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.407+0000 7f0716a5d640 1 -- 172.21.15.161:0/1998795716 reap_dead start 2024-03-28T23:30:37.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.407+0000 7f07184c0640 1 -- 172.21.15.161:0/1998795716 shutdown_connections 2024-03-28T23:30:37.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.407+0000 7f07184c0640 1 -- 172.21.15.161:0/1998795716 >> 172.21.15.161:0/1998795716 conn(0x7f07100a83f0 msgr2=0x7f07100a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:37.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.407+0000 7f07184c0640 1 -- 172.21.15.161:0/1998795716 shutdown_connections 2024-03-28T23:30:37.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.407+0000 7f07184c0640 1 -- 172.21.15.161:0/1998795716 wait complete. 2024-03-28T23:30:37.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.408+0000 7f07184c0640 1 Processor -- start 2024-03-28T23:30:37.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.408+0000 7f07184c0640 1 -- start start 2024-03-28T23:30:37.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.408+0000 7f07184c0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06fc005ca0 0x7f07101ea4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:37.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.408+0000 7f07184c0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f071014bcf0 0x7f07101e65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:37.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.408+0000 7f07184c0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f071014fde0 0x7f0710150260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:37.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.408+0000 7f07184c0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f07100ab820 con 0x7f071014fde0 2024-03-28T23:30:37.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.408+0000 7f07184c0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0710152250 con 0x7f071014bcf0 2024-03-28T23:30:37.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.408+0000 7f07184c0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f07100ab9a0 con 0x7f06fc005ca0 2024-03-28T23:30:37.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.409+0000 7f0716a5d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f071014fde0 0x7f0710150260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:37.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.409+0000 7f0716a5d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f071014fde0 0x7f0710150260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49224/0 (socket says 172.21.15.161:49224) 2024-03-28T23:30:37.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.409+0000 7f0716a5d640 1 -- 172.21.15.161:0/3128025112 learned_addr learned my addr 172.21.15.161:0/3128025112 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:37.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.409+0000 7f0715a5b640 1 --2- 172.21.15.161:0/3128025112 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f071014bcf0 0x7f07101e65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:37.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.409+0000 7f071625c640 1 --2- 172.21.15.161:0/3128025112 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06fc005ca0 0x7f07101ea4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:37.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.409+0000 7f0716a5d640 1 -- 172.21.15.161:0/3128025112 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06fc005ca0 msgr2=0x7f07101ea4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:37.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.409+0000 7f0716a5d640 1 --2- 172.21.15.161:0/3128025112 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06fc005ca0 0x7f07101ea4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.409+0000 7f0716a5d640 1 -- 172.21.15.161:0/3128025112 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f071014bcf0 msgr2=0x7f07101e65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:37.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.409+0000 7f0716a5d640 1 --2- 172.21.15.161:0/3128025112 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f071014bcf0 0x7f07101e65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.409+0000 7f0716a5d640 1 -- 172.21.15.161:0/3128025112 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07101eaa10 con 0x7f071014fde0 2024-03-28T23:30:37.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.409+0000 7f0716a5d640 1 --2- 172.21.15.161:0/3128025112 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f071014fde0 0x7f0710150260 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f071006ec30 tx=0x7f0708007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:37.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.409+0000 7f0706ffd640 1 -- 172.21.15.161:0/3128025112 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0708017020 con 0x7f071014fde0 2024-03-28T23:30:37.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.409+0000 7f0706ffd640 1 -- 172.21.15.161:0/3128025112 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f070800aa40 con 0x7f071014fde0 2024-03-28T23:30:37.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.409+0000 7f07184c0640 1 -- 172.21.15.161:0/3128025112 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0710155600 con 0x7f071014fde0 2024-03-28T23:30:37.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.409+0000 7f0706ffd640 1 -- 172.21.15.161:0/3128025112 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f070800ae30 con 0x7f071014fde0 2024-03-28T23:30:37.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.409+0000 7f07184c0640 1 -- 172.21.15.161:0/3128025112 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07101e9cd0 con 0x7f071014fde0 2024-03-28T23:30:37.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.411+0000 7f07184c0640 1 -- 172.21.15.161:0/3128025112 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0710155b20 con 0x7f071014fde0 2024-03-28T23:30:37.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.411+0000 7f0706ffd640 1 -- 172.21.15.161:0/3128025112 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f070801a020 con 0x7f071014fde0 2024-03-28T23:30:37.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.411+0000 7f0706ffd640 1 --2- 172.21.15.161:0/3128025112 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f06f406b500 0x7f06f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:37.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.411+0000 7f0706ffd640 1 -- 172.21.15.161:0/3128025112 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f070808a980 con 0x7f071014fde0 2024-03-28T23:30:37.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.411+0000 7f071625c640 1 --2- 172.21.15.161:0/3128025112 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f06f406b500 0x7f06f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:37.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.412+0000 7f071625c640 1 --2- 172.21.15.161:0/3128025112 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f06f406b500 0x7f06f406d9c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f070c00ba00 tx=0x7f070c03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:37.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.415+0000 7f0706ffd640 1 -- 172.21.15.161:0/3128025112 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0710155b20 con 0x7f071014fde0 2024-03-28T23:30:37.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.608+0000 7f07184c0640 1 -- 172.21.15.161:0/3128025112 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f071014e590 con 0x7f071014fde0 2024-03-28T23:30:37.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.608+0000 7f0706ffd640 1 -- 172.21.15.161:0/3128025112 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f071014e590 con 0x7f071014fde0 2024-03-28T23:30:37.609 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:30:37.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.611+0000 7f07184c0640 1 -- 172.21.15.161:0/3128025112 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f06f406b500 msgr2=0x7f06f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:37.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.611+0000 7f07184c0640 1 --2- 172.21.15.161:0/3128025112 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f06f406b500 0x7f06f406d9c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f070c00ba00 tx=0x7f070c03d040 comp rx=0 tx=0).stop 2024-03-28T23:30:37.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.611+0000 7f07184c0640 1 -- 172.21.15.161:0/3128025112 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f071014fde0 msgr2=0x7f0710150260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:37.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.611+0000 7f07184c0640 1 --2- 172.21.15.161:0/3128025112 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f071014fde0 0x7f0710150260 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f071006ec30 tx=0x7f0708007a90 comp rx=0 tx=0).stop 2024-03-28T23:30:37.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.611+0000 7f07184c0640 1 -- 172.21.15.161:0/3128025112 shutdown_connections 2024-03-28T23:30:37.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.611+0000 7f07184c0640 1 --2- 172.21.15.161:0/3128025112 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f06f406b500 0x7f06f406d9c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.611+0000 7f07184c0640 1 --2- 172.21.15.161:0/3128025112 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f071014fde0 0x7f0710150260 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.611+0000 7f07184c0640 1 --2- 172.21.15.161:0/3128025112 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f071014bcf0 0x7f07101e65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.611+0000 7f07184c0640 1 --2- 172.21.15.161:0/3128025112 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06fc005ca0 0x7f07101ea4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.611+0000 7f07184c0640 1 -- 172.21.15.161:0/3128025112 >> 172.21.15.161:0/3128025112 conn(0x7f07100a83f0 msgr2=0x7f071014de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:37.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.611+0000 7f07184c0640 1 -- 172.21.15.161:0/3128025112 shutdown_connections 2024-03-28T23:30:37.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.611+0000 7f07184c0640 1 -- 172.21.15.161:0/3128025112 wait complete. 2024-03-28T23:30:37.622 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:30:37.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.726+0000 7f9cf1155640 1 Processor -- start 2024-03-28T23:30:37.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.726+0000 7f9cf1155640 1 -- start start 2024-03-28T23:30:37.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9cf1155640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cec0704c0 0x7f9cec155130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:37.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9cf1155640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9cec0708e0 0x7f9cec155670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:37.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9cf1155640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cec152ba0 con 0x7f9cec1527c0 2024-03-28T23:30:37.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9cf1155640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f9cec0c3aa0 con 0x7f9cec0708e0 2024-03-28T23:30:37.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9cf1155640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9cec070dc0 con 0x7f9cec0704c0 2024-03-28T23:30:37.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9cead9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f9cec1527c0 0x7f9cec0c1b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60192/0 (socket says 172.21.15.161:60192) 2024-03-28T23:30:37.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9cead9d640 1 -- 172.21.15.161:0/1812587135 learned_addr learned my addr 172.21.15.161:0/1812587135 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:37.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9ceb59e640 1 --2- 172.21.15.161:0/1812587135 >> v2:172.21.15.170:3300/0 conn(0x7f9cec0708e0 0x7f9cec155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:37.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9cea59c640 1 -- 172.21.15.161:0/1812587135 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3997272538 0 0) 0x7f9cec152ba0 con 0x7f9cec1527c0 2024-03-28T23:30:37.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9ce3fff640 1 --2- 172.21.15.161:0/1812587135 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cec0704c0 0x7f9cec155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:37.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9cea59c640 1 -- 172.21.15.161:0/1812587135 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9cd8003520 con 0x7f9cec1527c0 2024-03-28T23:30:37.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9cea59c640 1 -- 172.21.15.161:0/1812587135 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1992685438 0 0) 0x7f9cd8003520 con 0x7f9cec1527c0 2024-03-28T23:30:37.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9cea59c640 1 -- 172.21.15.161:0/1812587135 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cec0704c0 msgr2=0x7f9cec155130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:37.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9cea59c640 1 --2- 172.21.15.161:0/1812587135 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cec0704c0 0x7f9cec155130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9cea59c640 1 -- 172.21.15.161:0/1812587135 >> v2:172.21.15.170:3300/0 conn(0x7f9cec0708e0 msgr2=0x7f9cec155670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:37.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9cea59c640 1 --2- 172.21.15.161:0/1812587135 >> v2:172.21.15.170:3300/0 conn(0x7f9cec0708e0 0x7f9cec155670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9cea59c640 1 -- 172.21.15.161:0/1812587135 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cec0c3cf0 con 0x7f9cec1527c0 2024-03-28T23:30:37.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9cea59c640 1 -- 172.21.15.161:0/1812587135 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f9cdc001670 con 0x7f9cec1527c0 2024-03-28T23:30:37.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9cea59c640 1 -- 172.21.15.161:0/1812587135 >> v1:172.21.15.161:6789/0 conn(0x7f9cec1527c0 legacy=0x7f9cec0c1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:37.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9cea59c640 1 --2- 172.21.15.161:0/1812587135 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cd80039b0 0x7f9cd8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:37.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9cea59c640 1 --2- 172.21.15.161:0/1812587135 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cec0704c0 0x7f9cd8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:37.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9cea59c640 1 --2- 172.21.15.161:0/1812587135 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cd8004d40 0x7f9cd80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:37.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9cea59c640 1 -- 172.21.15.161:0/1812587135 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec0c3aa0 con 0x7f9cd80039b0 2024-03-28T23:30:37.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9cea59c640 1 -- 172.21.15.161:0/1812587135 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec070dc0 con 0x7f9cd8004d40 2024-03-28T23:30:37.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.727+0000 7f9cea59c640 1 -- 172.21.15.161:0/1812587135 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9cd800ba40 con 0x7f9cec0704c0 2024-03-28T23:30:37.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.728+0000 7f9cead9d640 1 --2- 172.21.15.161:0/1812587135 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cd80039b0 0x7f9cd8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:37.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.728+0000 7f9ce3fff640 1 --2- 172.21.15.161:0/1812587135 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cec0704c0 0x7f9cd8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:37.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.728+0000 7f9ceb59e640 1 --2- 172.21.15.161:0/1812587135 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cd8004d40 0x7f9cd80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:37.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.728+0000 7f9cead9d640 1 -- 172.21.15.161:0/1812587135 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cec0704c0 msgr2=0x7f9cd8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:37.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.728+0000 7f9cead9d640 1 --2- 172.21.15.161:0/1812587135 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cec0704c0 0x7f9cd8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.728+0000 7f9cead9d640 1 -- 172.21.15.161:0/1812587135 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cd8004d40 msgr2=0x7f9cd80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:37.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.728+0000 7f9cead9d640 1 --2- 172.21.15.161:0/1812587135 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cd8004d40 0x7f9cd80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.728+0000 7f9cead9d640 1 -- 172.21.15.161:0/1812587135 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cd8003520 con 0x7f9cd80039b0 2024-03-28T23:30:37.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.728+0000 7f9ce3fff640 1 --2- 172.21.15.161:0/1812587135 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cec0704c0 0x7f9cd8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:37.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.728+0000 7f9cead9d640 1 --2- 172.21.15.161:0/1812587135 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cd80039b0 0x7f9cd8003dd0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc007fd0 tx=0x7f9cd8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa1c9edaf1bfa8e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:37.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.729+0000 7f9cea59c640 1 -- 172.21.15.161:0/1812587135 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc007d50 con 0x7f9cd80039b0 2024-03-28T23:30:37.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.729+0000 7f9cea59c640 1 -- 172.21.15.161:0/1812587135 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cdc030070 con 0x7f9cd80039b0 2024-03-28T23:30:37.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.729+0000 7f9cea59c640 1 -- 172.21.15.161:0/1812587135 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc02cd40 con 0x7f9cd80039b0 2024-03-28T23:30:37.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.729+0000 7f9cf1155640 1 -- 172.21.15.161:0/1812587135 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cd80039b0 msgr2=0x7f9cd8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:37.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.729+0000 7f9cf1155640 1 --2- 172.21.15.161:0/1812587135 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cd80039b0 0x7f9cd8003dd0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc007fd0 tx=0x7f9cd8005830 comp rx=0 tx=0).stop 2024-03-28T23:30:37.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.729+0000 7f9ceb59e640 1 -- 172.21.15.161:0/1812587135 reap_dead start 2024-03-28T23:30:37.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.729+0000 7f9cf1155640 1 -- 172.21.15.161:0/1812587135 shutdown_connections 2024-03-28T23:30:37.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.729+0000 7f9cf1155640 1 -- 172.21.15.161:0/1812587135 >> 172.21.15.161:0/1812587135 conn(0x7f9cec0c8810 msgr2=0x7f9cec0c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:37.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.729+0000 7f9cf1155640 1 -- 172.21.15.161:0/1812587135 shutdown_connections 2024-03-28T23:30:37.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.729+0000 7f9cf1155640 1 -- 172.21.15.161:0/1812587135 wait complete. 2024-03-28T23:30:37.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.731+0000 7f9cf1155640 1 Processor -- start 2024-03-28T23:30:37.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.731+0000 7f9cf1155640 1 -- start start 2024-03-28T23:30:37.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.731+0000 7f9cf1155640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cd80039b0 0x7f9cec0c76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:37.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.731+0000 7f9cf1155640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cec14ec80 0x7f9cec14f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:37.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.731+0000 7f9cf1155640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cec14f640 0x7f9cec148220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:37.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.731+0000 7f9cf1155640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec0c47d0 con 0x7f9cd80039b0 2024-03-28T23:30:37.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.731+0000 7f9cf1155640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec0c42b0 con 0x7f9cec14ec80 2024-03-28T23:30:37.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.731+0000 7f9cf1155640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9cec155bb0 con 0x7f9cec14f640 2024-03-28T23:30:37.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.731+0000 7f9cead9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cd80039b0 0x7f9cec0c76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:37.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.731+0000 7f9cead9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cd80039b0 0x7f9cec0c76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49250/0 (socket says 172.21.15.161:49250) 2024-03-28T23:30:37.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.731+0000 7f9cead9d640 1 -- 172.21.15.161:0/58627863 learned_addr learned my addr 172.21.15.161:0/58627863 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:37.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.732+0000 7f9ceb59e640 1 --2- 172.21.15.161:0/58627863 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cec14f640 0x7f9cec148220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:37.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.732+0000 7f9ce3fff640 1 --2- 172.21.15.161:0/58627863 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cec14ec80 0x7f9cec14f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:37.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.732+0000 7f9cead9d640 1 -- 172.21.15.161:0/58627863 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cec14f640 msgr2=0x7f9cec148220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:37.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.732+0000 7f9cead9d640 1 --2- 172.21.15.161:0/58627863 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cec14f640 0x7f9cec148220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.732+0000 7f9cead9d640 1 -- 172.21.15.161:0/58627863 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cec14ec80 msgr2=0x7f9cec14f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:37.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.732+0000 7f9cead9d640 1 --2- 172.21.15.161:0/58627863 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cec14ec80 0x7f9cec14f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.732+0000 7f9cead9d640 1 -- 172.21.15.161:0/58627863 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cec156d50 con 0x7f9cd80039b0 2024-03-28T23:30:37.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.732+0000 7f9cead9d640 1 --2- 172.21.15.161:0/58627863 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cd80039b0 0x7f9cec0c76a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc000f60 tx=0x7f9cdc000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:37.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.732+0000 7f9ce37fe640 1 -- 172.21.15.161:0/58627863 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc001cb0 con 0x7f9cd80039b0 2024-03-28T23:30:37.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.732+0000 7f9ce37fe640 1 -- 172.21.15.161:0/58627863 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cdc004bf0 con 0x7f9cd80039b0 2024-03-28T23:30:37.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.732+0000 7f9ce37fe640 1 -- 172.21.15.161:0/58627863 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc007070 con 0x7f9cd80039b0 2024-03-28T23:30:37.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.732+0000 7f9cf1155640 1 -- 172.21.15.161:0/58627863 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cec157030 con 0x7f9cd80039b0 2024-03-28T23:30:37.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.732+0000 7f9cf1155640 1 -- 172.21.15.161:0/58627863 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cec0c7be0 con 0x7f9cd80039b0 2024-03-28T23:30:37.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.733+0000 7f9cf1155640 1 -- 172.21.15.161:0/58627863 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cec04f4b0 con 0x7f9cd80039b0 2024-03-28T23:30:37.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.734+0000 7f9ce37fe640 1 -- 172.21.15.161:0/58627863 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9cdc048020 con 0x7f9cd80039b0 2024-03-28T23:30:37.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.734+0000 7f9ce37fe640 1 --2- 172.21.15.161:0/58627863 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9ccc06b500 0x7f9ccc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:37.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.734+0000 7f9ce37fe640 1 -- 172.21.15.161:0/58627863 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f9cdc0ab700 con 0x7f9cd80039b0 2024-03-28T23:30:37.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.734+0000 7f9ce3fff640 1 --2- 172.21.15.161:0/58627863 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9ccc06b500 0x7f9ccc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:37.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.734+0000 7f9ce3fff640 1 --2- 172.21.15.161:0/58627863 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9ccc06b500 0x7f9ccc06d9c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9cec1538c0 tx=0x7f9cd4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:37.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.739+0000 7f9ce37fe640 1 -- 172.21.15.161:0/58627863 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9cdc079bc0 con 0x7f9cd80039b0 2024-03-28T23:30:37.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.910+0000 7f9cf1155640 1 -- 172.21.15.161:0/58627863 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9cec0c8d20 con 0x7f9cd80039b0 2024-03-28T23:30:37.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.910+0000 7f9ce37fe640 1 -- 172.21.15.161:0/58627863 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f9cdc07dc60 con 0x7f9cd80039b0 2024-03-28T23:30:37.911 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:30:37.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.912+0000 7f9cf1155640 1 -- 172.21.15.161:0/58627863 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9ccc06b500 msgr2=0x7f9ccc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:37.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.912+0000 7f9cf1155640 1 --2- 172.21.15.161:0/58627863 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9ccc06b500 0x7f9ccc06d9c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9cec1538c0 tx=0x7f9cd4008040 comp rx=0 tx=0).stop 2024-03-28T23:30:37.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.912+0000 7f9cf1155640 1 -- 172.21.15.161:0/58627863 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cd80039b0 msgr2=0x7f9cec0c76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:37.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.912+0000 7f9cf1155640 1 --2- 172.21.15.161:0/58627863 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cd80039b0 0x7f9cec0c76a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc000f60 tx=0x7f9cdc000fe0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.912+0000 7f9cf1155640 1 -- 172.21.15.161:0/58627863 shutdown_connections 2024-03-28T23:30:37.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.912+0000 7f9cf1155640 1 --2- 172.21.15.161:0/58627863 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9ccc06b500 0x7f9ccc06d9c0 secure :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9cec1538c0 tx=0x7f9cd4008040 comp rx=0 tx=0).stop 2024-03-28T23:30:37.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.912+0000 7f9cf1155640 1 --2- 172.21.15.161:0/58627863 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cec14f640 0x7f9cec148220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.912+0000 7f9cf1155640 1 --2- 172.21.15.161:0/58627863 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cec14ec80 0x7f9cec14f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.912+0000 7f9cf1155640 1 --2- 172.21.15.161:0/58627863 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cd80039b0 0x7f9cec0c76a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:37.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.912+0000 7f9cf1155640 1 -- 172.21.15.161:0/58627863 >> 172.21.15.161:0/58627863 conn(0x7f9cec0c8810 msgr2=0x7f9cec06cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:37.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.912+0000 7f9cf1155640 1 -- 172.21.15.161:0/58627863 shutdown_connections 2024-03-28T23:30:37.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:37.912+0000 7f9cf1155640 1 -- 172.21.15.161:0/58627863 wait complete. 2024-03-28T23:30:37.924 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T23:30:38.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.027+0000 7eff9428a640 1 Processor -- start 2024-03-28T23:30:38.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.027+0000 7eff9428a640 1 -- start start 2024-03-28T23:30:38.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.028+0000 7eff9428a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff8c0c0330 0x7eff8c0c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:38.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.028+0000 7eff9428a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7eff8c1589b0 0x7eff8c158e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:38.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.028+0000 7eff9428a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff8c0c6f60 con 0x7eff8c152bc0 2024-03-28T23:30:38.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.028+0000 7eff9428a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7eff8c0c6120 con 0x7eff8c1589b0 2024-03-28T23:30:38.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.028+0000 7eff9428a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7eff8c0c6400 con 0x7eff8c0c0330 2024-03-28T23:30:38.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.029+0000 7eff91825640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff8c0c0330 0x7eff8c0c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:38.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.029+0000 7eff92827640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7eff8c1589b0 0x7eff8c158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:38.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.029+0000 7eff91825640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff8c0c0330 0x7eff8c0c9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:54240/0 (socket says 172.21.15.161:54240) 2024-03-28T23:30:38.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.029+0000 7eff92827640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7eff8c1589b0 0x7eff8c158e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:54626/0 (socket says 172.21.15.161:54626) 2024-03-28T23:30:38.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.029+0000 7eff91825640 1 -- 172.21.15.161:0/3317285312 learned_addr learned my addr 172.21.15.161:0/3317285312 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:38.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.029+0000 7eff92827640 1 -- 172.21.15.161:0/3317285312 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff8c0c0330 msgr2=0x7eff8c0c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:38.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.029+0000 7eff92827640 1 --2- 172.21.15.161:0/3317285312 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff8c0c0330 0x7eff8c0c9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.029+0000 7eff92827640 1 -- 172.21.15.161:0/3317285312 >> v1:172.21.15.170:6789/0 conn(0x7eff8c152bc0 legacy=0x7eff8c0c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:38.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.029+0000 7eff92827640 1 -- 172.21.15.161:0/3317285312 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff8c0c96d0 con 0x7eff8c1589b0 2024-03-28T23:30:38.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.029+0000 7eff91825640 1 --2- 172.21.15.161:0/3317285312 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff8c0c0330 0x7eff8c0c9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:38.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.030+0000 7eff92827640 1 --2- 172.21.15.161:0/3317285312 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff8c1589b0 0x7eff8c158e10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7eff7c0097a0 tx=0x7eff7c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f90d8c2deca9e0e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:38.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.030+0000 7eff91024640 1 -- 172.21.15.161:0/3317285312 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7c035020 con 0x7eff8c1589b0 2024-03-28T23:30:38.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.030+0000 7eff91024640 1 -- 172.21.15.161:0/3317285312 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff7c031830 con 0x7eff8c1589b0 2024-03-28T23:30:38.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.030+0000 7eff91024640 1 -- 172.21.15.161:0/3317285312 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7c031c00 con 0x7eff8c1589b0 2024-03-28T23:30:38.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.031+0000 7eff9428a640 1 -- 172.21.15.161:0/3317285312 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff8c1589b0 msgr2=0x7eff8c158e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:38.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.031+0000 7eff9428a640 1 --2- 172.21.15.161:0/3317285312 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff8c1589b0 0x7eff8c158e10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7eff7c0097a0 tx=0x7eff7c02b8d0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.031+0000 7eff9428a640 1 -- 172.21.15.161:0/3317285312 shutdown_connections 2024-03-28T23:30:38.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.031+0000 7eff9428a640 1 --2- 172.21.15.161:0/3317285312 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff8c1589b0 0x7eff8c158e10 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.031+0000 7eff9428a640 1 --2- 172.21.15.161:0/3317285312 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff8c0c0330 0x7eff8c0c9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.031+0000 7eff9428a640 1 -- 172.21.15.161:0/3317285312 >> 172.21.15.161:0/3317285312 conn(0x7eff8c0caa80 msgr2=0x7eff8c0cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:38.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.031+0000 7eff9428a640 1 -- 172.21.15.161:0/3317285312 shutdown_connections 2024-03-28T23:30:38.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.031+0000 7eff9428a640 1 -- 172.21.15.161:0/3317285312 wait complete. 2024-03-28T23:30:38.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.032+0000 7eff9428a640 1 Processor -- start 2024-03-28T23:30:38.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.032+0000 7eff9428a640 1 -- start start 2024-03-28T23:30:38.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.033+0000 7eff9428a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff8c0c0330 0x7eff8c157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.033+0000 7eff9428a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff8c152bc0 0x7eff8c1515c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.033+0000 7eff9428a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff8c1589b0 0x7eff8c151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.033+0000 7eff9428a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7eff8c152fa0 con 0x7eff8c1589b0 2024-03-28T23:30:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.033+0000 7eff9428a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7eff8c0c6bf0 con 0x7eff8c0c0330 2024-03-28T23:30:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.033+0000 7eff9428a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7eff8c14ada0 con 0x7eff8c152bc0 2024-03-28T23:30:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.033+0000 7eff92827640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff8c1589b0 0x7eff8c151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.033+0000 7eff92827640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff8c1589b0 0x7eff8c151b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49254/0 (socket says 172.21.15.161:49254) 2024-03-28T23:30:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.033+0000 7eff92827640 1 -- 172.21.15.161:0/3072100137 learned_addr learned my addr 172.21.15.161:0/3072100137 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.033+0000 7eff92026640 1 --2- 172.21.15.161:0/3072100137 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff8c0c0330 0x7eff8c157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.033+0000 7eff91825640 1 --2- 172.21.15.161:0/3072100137 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff8c152bc0 0x7eff8c1515c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.033+0000 7eff92827640 1 -- 172.21.15.161:0/3072100137 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff8c152bc0 msgr2=0x7eff8c1515c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.033+0000 7eff92827640 1 --2- 172.21.15.161:0/3072100137 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff8c152bc0 0x7eff8c1515c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.034+0000 7eff92827640 1 -- 172.21.15.161:0/3072100137 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff8c0c0330 msgr2=0x7eff8c157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.034+0000 7eff92827640 1 --2- 172.21.15.161:0/3072100137 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff8c0c0330 0x7eff8c157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.034+0000 7eff92827640 1 -- 172.21.15.161:0/3072100137 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff8c145fb0 con 0x7eff8c1589b0 2024-03-28T23:30:38.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.034+0000 7eff92827640 1 --2- 172.21.15.161:0/3072100137 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff8c1589b0 0x7eff8c151b00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7eff7c034040 tx=0x7eff7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:38.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.034+0000 7eff8affd640 1 -- 172.21.15.161:0/3072100137 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7c040070 con 0x7eff8c1589b0 2024-03-28T23:30:38.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.034+0000 7eff8affd640 1 -- 172.21.15.161:0/3072100137 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff7c03a850 con 0x7eff8c1589b0 2024-03-28T23:30:38.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.034+0000 7eff8affd640 1 -- 172.21.15.161:0/3072100137 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7c035040 con 0x7eff8c1589b0 2024-03-28T23:30:38.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.034+0000 7eff9428a640 1 -- 172.21.15.161:0/3072100137 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff8c150760 con 0x7eff8c1589b0 2024-03-28T23:30:38.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.034+0000 7eff9428a640 1 -- 172.21.15.161:0/3072100137 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff8c150e00 con 0x7eff8c1589b0 2024-03-28T23:30:38.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.035+0000 7eff9428a640 1 -- 172.21.15.161:0/3072100137 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff8c068f90 con 0x7eff8c1589b0 2024-03-28T23:30:38.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.035+0000 7eff8affd640 1 -- 172.21.15.161:0/3072100137 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7eff7c04b060 con 0x7eff8c1589b0 2024-03-28T23:30:38.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.036+0000 7eff8affd640 1 --2- 172.21.15.161:0/3072100137 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7eff7806b500 0x7eff7806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:38.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.036+0000 7eff92026640 1 --2- 172.21.15.161:0/3072100137 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7eff7806b500 0x7eff7806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:38.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.036+0000 7eff8affd640 1 -- 172.21.15.161:0/3072100137 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7eff7c0abe80 con 0x7eff8c1589b0 2024-03-28T23:30:38.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.036+0000 7eff92026640 1 --2- 172.21.15.161:0/3072100137 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7eff7806b500 0x7eff7806d9c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7eff84001d40 tx=0x7eff84009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:38.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.040+0000 7eff8affd640 1 -- 172.21.15.161:0/3072100137 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7eff8c068f90 con 0x7eff8c1589b0 2024-03-28T23:30:38.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.219+0000 7eff9428a640 1 -- 172.21.15.161:0/3072100137 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7eff8c04f4b0 con 0x7eff8c1589b0 2024-03-28T23:30:38.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.219+0000 7eff8affd640 1 -- 172.21.15.161:0/3072100137 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7eff7c031800 con 0x7eff8c1589b0 2024-03-28T23:30:38.220 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:30:38.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.221+0000 7eff9428a640 1 -- 172.21.15.161:0/3072100137 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7eff7806b500 msgr2=0x7eff7806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:38.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.221+0000 7eff9428a640 1 --2- 172.21.15.161:0/3072100137 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7eff7806b500 0x7eff7806d9c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7eff84001d40 tx=0x7eff84009040 comp rx=0 tx=0).stop 2024-03-28T23:30:38.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.221+0000 7eff9428a640 1 -- 172.21.15.161:0/3072100137 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff8c1589b0 msgr2=0x7eff8c151b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:38.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.221+0000 7eff9428a640 1 --2- 172.21.15.161:0/3072100137 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff8c1589b0 0x7eff8c151b00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7eff7c034040 tx=0x7eff7c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.221+0000 7eff9428a640 1 -- 172.21.15.161:0/3072100137 shutdown_connections 2024-03-28T23:30:38.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.221+0000 7eff9428a640 1 --2- 172.21.15.161:0/3072100137 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7eff7806b500 0x7eff7806d9c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.221+0000 7eff9428a640 1 --2- 172.21.15.161:0/3072100137 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff8c1589b0 0x7eff8c151b00 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.221+0000 7eff9428a640 1 --2- 172.21.15.161:0/3072100137 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff8c152bc0 0x7eff8c1515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.221+0000 7eff9428a640 1 --2- 172.21.15.161:0/3072100137 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff8c0c0330 0x7eff8c157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.221+0000 7eff9428a640 1 -- 172.21.15.161:0/3072100137 >> 172.21.15.161:0/3072100137 conn(0x7eff8c0caa80 msgr2=0x7eff8c0c5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:38.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.221+0000 7eff9428a640 1 -- 172.21.15.161:0/3072100137 shutdown_connections 2024-03-28T23:30:38.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.221+0000 7eff9428a640 1 -- 172.21.15.161:0/3072100137 wait complete. 2024-03-28T23:30:38.234 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T23:30:38.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.336+0000 7f3795de9640 1 Processor -- start 2024-03-28T23:30:38.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.336+0000 7f3795de9640 1 -- start start 2024-03-28T23:30:38.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.337+0000 7f3795de9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3790152890 0x7f37900c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:38.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.337+0000 7f3795de9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37900c8ed0 0x7f37900c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:38.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.337+0000 7f3795de9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3790158a30 con 0x7f37900c5950 2024-03-28T23:30:38.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.337+0000 7f3795de9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f37900c8480 con 0x7f3790152890 2024-03-28T23:30:38.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.337+0000 7f3795de9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3790159640 con 0x7f37900c8ed0 2024-03-28T23:30:38.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.337+0000 7f378effd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37900c8ed0 0x7f37900c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:38.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.337+0000 7f378f7fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3790152890 0x7f37900c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:38.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.337+0000 7f378f7fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3790152890 0x7f37900c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:54638/0 (socket says 172.21.15.161:54638) 2024-03-28T23:30:38.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.337+0000 7f378effd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37900c8ed0 0x7f37900c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:54256/0 (socket says 172.21.15.161:54256) 2024-03-28T23:30:38.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.337+0000 7f378effd640 1 -- 172.21.15.161:0/991866953 learned_addr learned my addr 172.21.15.161:0/991866953 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:38.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.337+0000 7f378ffff640 1 --1- 172.21.15.161:0/991866953 >> v1:172.21.15.170:6789/0 conn(0x7f37900c5950 0x7f37901553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:34098/0 (socket says 172.21.15.161:34098) 2024-03-28T23:30:38.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.338+0000 7f378f7fe640 1 -- 172.21.15.161:0/991866953 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37900c8ed0 msgr2=0x7f37900c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:38.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.338+0000 7f378f7fe640 1 --2- 172.21.15.161:0/991866953 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37900c8ed0 0x7f37900c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.338+0000 7f378f7fe640 1 -- 172.21.15.161:0/991866953 >> v1:172.21.15.170:6789/0 conn(0x7f37900c5950 legacy=0x7f37901553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:38.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.338+0000 7f378f7fe640 1 -- 172.21.15.161:0/991866953 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3790158d10 con 0x7f3790152890 2024-03-28T23:30:38.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.338+0000 7f378effd640 1 --2- 172.21.15.161:0/991866953 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37900c8ed0 0x7f37900c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:38.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.338+0000 7f378f7fe640 1 --2- 172.21.15.161:0/991866953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3790152890 0x7f37900c8990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f37800097a0 tx=0x7f378002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89215bb0b5922ab9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:38.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.339+0000 7f378e7fc640 1 -- 172.21.15.161:0/991866953 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3780038070 con 0x7f3790152890 2024-03-28T23:30:38.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.339+0000 7f378e7fc640 1 -- 172.21.15.161:0/991866953 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3780031700 con 0x7f3790152890 2024-03-28T23:30:38.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.339+0000 7f378e7fc640 1 -- 172.21.15.161:0/991866953 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3780031ad0 con 0x7f3790152890 2024-03-28T23:30:38.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.339+0000 7f3795de9640 1 -- 172.21.15.161:0/991866953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3790152890 msgr2=0x7f37900c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:38.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.339+0000 7f3795de9640 1 --2- 172.21.15.161:0/991866953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3790152890 0x7f37900c8990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f37800097a0 tx=0x7f378002b7a0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.339+0000 7f3795de9640 1 -- 172.21.15.161:0/991866953 shutdown_connections 2024-03-28T23:30:38.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.339+0000 7f3795de9640 1 --2- 172.21.15.161:0/991866953 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37900c8ed0 0x7f37900c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.339+0000 7f3795de9640 1 --2- 172.21.15.161:0/991866953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3790152890 0x7f37900c8990 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.339+0000 7f3795de9640 1 -- 172.21.15.161:0/991866953 >> 172.21.15.161:0/991866953 conn(0x7f37900ac620 msgr2=0x7f37900aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:38.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.339+0000 7f3795de9640 1 -- 172.21.15.161:0/991866953 shutdown_connections 2024-03-28T23:30:38.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.339+0000 7f3795de9640 1 -- 172.21.15.161:0/991866953 wait complete. 2024-03-28T23:30:38.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.340+0000 7f3795de9640 1 Processor -- start 2024-03-28T23:30:38.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.340+0000 7f3795de9640 1 -- start start 2024-03-28T23:30:38.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.341+0000 7f3795de9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37900c5950 0x7f3790156600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:38.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.341+0000 7f3795de9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37900c8ed0 0x7f37901eb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:38.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.341+0000 7f3795de9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3790152890 0x7f3790148830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:38.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.341+0000 7f3795de9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3790152c70 con 0x7f3790152890 2024-03-28T23:30:38.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.341+0000 7f3795de9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f37901592d0 con 0x7f37900c8ed0 2024-03-28T23:30:38.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.341+0000 7f3795de9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3790152df0 con 0x7f37900c5950 2024-03-28T23:30:38.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.341+0000 7f378ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3790152890 0x7f3790148830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:38.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.341+0000 7f378ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3790152890 0x7f3790148830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49258/0 (socket says 172.21.15.161:49258) 2024-03-28T23:30:38.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.341+0000 7f378ffff640 1 -- 172.21.15.161:0/38002663 learned_addr learned my addr 172.21.15.161:0/38002663 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:38.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.341+0000 7f378f7fe640 1 --2- 172.21.15.161:0/38002663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37900c5950 0x7f3790156600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:38.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.341+0000 7f378effd640 1 --2- 172.21.15.161:0/38002663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37900c8ed0 0x7f37901eb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:38.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.342+0000 7f378ffff640 1 -- 172.21.15.161:0/38002663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37900c5950 msgr2=0x7f3790156600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:38.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.342+0000 7f378ffff640 1 --2- 172.21.15.161:0/38002663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37900c5950 0x7f3790156600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.342+0000 7f378ffff640 1 -- 172.21.15.161:0/38002663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37900c8ed0 msgr2=0x7f37901eb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:38.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.342+0000 7f378ffff640 1 --2- 172.21.15.161:0/38002663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37900c8ed0 0x7f37901eb8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.342+0000 7f378ffff640 1 -- 172.21.15.161:0/38002663 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3790156b40 con 0x7f3790152890 2024-03-28T23:30:38.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.342+0000 7f378ffff640 1 --2- 172.21.15.161:0/38002663 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3790152890 0x7f3790148830 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f378800c760 tx=0x7f378800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:38.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.342+0000 7f3777fff640 1 -- 172.21.15.161:0/38002663 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3788017070 con 0x7f3790152890 2024-03-28T23:30:38.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.342+0000 7f3777fff640 1 -- 172.21.15.161:0/38002663 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f378800a640 con 0x7f3790152890 2024-03-28T23:30:38.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.342+0000 7f3795de9640 1 -- 172.21.15.161:0/38002663 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3790158730 con 0x7f3790152890 2024-03-28T23:30:38.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.342+0000 7f3777fff640 1 -- 172.21.15.161:0/38002663 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f378800a9f0 con 0x7f3790152890 2024-03-28T23:30:38.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.342+0000 7f3795de9640 1 -- 172.21.15.161:0/38002663 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3790158450 con 0x7f3790152890 2024-03-28T23:30:38.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.343+0000 7f3795de9640 1 -- 172.21.15.161:0/38002663 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f379004f4b0 con 0x7f3790152890 2024-03-28T23:30:38.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.344+0000 7f3777fff640 1 -- 172.21.15.161:0/38002663 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f378801a070 con 0x7f3790152890 2024-03-28T23:30:38.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.344+0000 7f3777fff640 1 --2- 172.21.15.161:0/38002663 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f377c06b4d0 0x7f377c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:38.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.345+0000 7f3777fff640 1 -- 172.21.15.161:0/38002663 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f3788089b20 con 0x7f3790152890 2024-03-28T23:30:38.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.345+0000 7f378f7fe640 1 --2- 172.21.15.161:0/38002663 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f377c06b4d0 0x7f377c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:38.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.345+0000 7f378f7fe640 1 --2- 172.21.15.161:0/38002663 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f377c06b4d0 0x7f377c06d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3780035000 tx=0x7f3780008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:38.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.349+0000 7f3777fff640 1 -- 172.21.15.161:0/38002663 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3788057ec0 con 0x7f3790152890 2024-03-28T23:30:38.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.538+0000 7f3795de9640 1 -- 172.21.15.161:0/38002663 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f37900c8140 con 0x7f3790152890 2024-03-28T23:30:38.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.539+0000 7f3777fff640 1 -- 172.21.15.161:0/38002663 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f378805bf60 con 0x7f3790152890 2024-03-28T23:30:38.540 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-28T23:30:38.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.541+0000 7f3795de9640 1 -- 172.21.15.161:0/38002663 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f377c06b4d0 msgr2=0x7f377c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:38.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.541+0000 7f3795de9640 1 --2- 172.21.15.161:0/38002663 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f377c06b4d0 0x7f377c06d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3780035000 tx=0x7f3780008030 comp rx=0 tx=0).stop 2024-03-28T23:30:38.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.541+0000 7f3795de9640 1 -- 172.21.15.161:0/38002663 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3790152890 msgr2=0x7f3790148830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:38.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.541+0000 7f3795de9640 1 --2- 172.21.15.161:0/38002663 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3790152890 0x7f3790148830 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f378800c760 tx=0x7f378800cc20 comp rx=0 tx=0).stop 2024-03-28T23:30:38.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.541+0000 7f3795de9640 1 -- 172.21.15.161:0/38002663 shutdown_connections 2024-03-28T23:30:38.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.541+0000 7f3795de9640 1 --2- 172.21.15.161:0/38002663 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f377c06b4d0 0x7f377c06d990 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.541+0000 7f3795de9640 1 --2- 172.21.15.161:0/38002663 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3790152890 0x7f3790148830 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.541+0000 7f3795de9640 1 --2- 172.21.15.161:0/38002663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37900c8ed0 0x7f37901eb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.541+0000 7f3795de9640 1 --2- 172.21.15.161:0/38002663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37900c5950 0x7f3790156600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.541+0000 7f3795de9640 1 -- 172.21.15.161:0/38002663 >> 172.21.15.161:0/38002663 conn(0x7f37900ac620 msgr2=0x7f37901514a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:38.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.541+0000 7f3795de9640 1 -- 172.21.15.161:0/38002663 shutdown_connections 2024-03-28T23:30:38.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.541+0000 7f3795de9640 1 -- 172.21.15.161:0/38002663 wait complete. 2024-03-28T23:30:38.553 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi161.front.sepia.ceph.com... 2024-03-28T23:30:38.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:38.553 DEBUG:teuthology.orchestra.run.smithi161:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-28T23:30:38.579 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:30:38.580 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi161.front.sepia.ceph.com 2024-03-28T23:30:38.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:38.580 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-28T23:30:38.635 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:30:38.635 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:30:38.655 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:30:38.655 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:30:38.715 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:30:38.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.817+0000 7fd7fe09d640 1 Processor -- start 2024-03-28T23:30:38.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.818+0000 7fd7fe09d640 1 -- start start 2024-03-28T23:30:38.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.819+0000 7fd7fe09d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd7f8152890 0x7fd7f80c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:38.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.819+0000 7fd7fe09d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7f8158a30 con 0x7fd7f80c5b20 2024-03-28T23:30:38.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.819+0000 7fd7fe09d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7f8158d10 con 0x7fd7f80c8ed0 2024-03-28T23:30:38.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.819+0000 7fd7fe09d640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd7f80c8480 con 0x7fd7f8152890 2024-03-28T23:30:38.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.819+0000 7fd7f7fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd7f80c5b20 0x7fd7f81553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60194/0 (socket says 172.21.15.161:60194) 2024-03-28T23:30:38.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.819+0000 7fd7f7fff640 1 -- 172.21.15.161:0/803074210 learned_addr learned my addr 172.21.15.161:0/803074210 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:38.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.819+0000 7fd7f67fc640 1 -- 172.21.15.161:0/803074210 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 546976377 0 0) 0x7fd7f8158a30 con 0x7fd7f80c5b20 2024-03-28T23:30:38.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.819+0000 7fd7f67fc640 1 -- 172.21.15.161:0/803074210 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7e0003520 con 0x7fd7f80c5b20 2024-03-28T23:30:38.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.819+0000 7fd7f77fe640 1 --2- 172.21.15.161:0/803074210 >> v2:172.21.15.170:3300/0 conn(0x7fd7f8152890 0x7fd7f80c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:38.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.819+0000 7fd7f67fc640 1 -- 172.21.15.161:0/803074210 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2323332337 0 0) 0x7fd7e0003520 con 0x7fd7f80c5b20 2024-03-28T23:30:38.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.819+0000 7fd7f67fc640 1 -- 172.21.15.161:0/803074210 >> v2:172.21.15.170:3300/0 conn(0x7fd7f8152890 msgr2=0x7fd7f80c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:38.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.819+0000 7fd7f67fc640 1 --2- 172.21.15.161:0/803074210 >> v2:172.21.15.170:3300/0 conn(0x7fd7f8152890 0x7fd7f80c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.819+0000 7fd7f67fc640 1 -- 172.21.15.161:0/803074210 >> v1:172.21.15.170:6789/0 conn(0x7fd7f80c8ed0 legacy=0x7fd7f80c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:38.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.819+0000 7fd7f67fc640 1 -- 172.21.15.161:0/803074210 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f8158ff0 con 0x7fd7f80c5b20 2024-03-28T23:30:38.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.819+0000 7fd7f67fc640 1 -- 172.21.15.161:0/803074210 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fd7f00019f0 con 0x7fd7f80c5b20 2024-03-28T23:30:38.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.819+0000 7fd7f67fc640 1 -- 172.21.15.161:0/803074210 >> v1:172.21.15.161:6789/0 conn(0x7fd7f80c5b20 legacy=0x7fd7f81553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:38.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.819+0000 7fd7f67fc640 1 --2- 172.21.15.161:0/803074210 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7e0003ae0 0x7fd7e0003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:38.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.819+0000 7fd7f67fc640 1 --2- 172.21.15.161:0/803074210 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7e0004800 0x7fd7e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:38.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.819+0000 7fd7f67fc640 1 --2- 172.21.15.161:0/803074210 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7e0005410 0x7fd7e000ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:38.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.820+0000 7fd7f67fc640 1 -- 172.21.15.161:0/803074210 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e000c320 con 0x7fd7e0004800 2024-03-28T23:30:38.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.820+0000 7fd7f67fc640 1 -- 172.21.15.161:0/803074210 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e000c4d0 con 0x7fd7e0003ae0 2024-03-28T23:30:38.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.820+0000 7fd7f67fc640 1 -- 172.21.15.161:0/803074210 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd7e000c6d0 con 0x7fd7e0005410 2024-03-28T23:30:38.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.820+0000 7fd7f6ffd640 1 --2- 172.21.15.161:0/803074210 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7e0004800 0x7fd7e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:38.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.820+0000 7fd7f77fe640 1 --2- 172.21.15.161:0/803074210 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7e0003ae0 0x7fd7e0003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:38.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.820+0000 7fd7f7fff640 1 --2- 172.21.15.161:0/803074210 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7e0005410 0x7fd7e000ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:38.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.820+0000 7fd7f6ffd640 1 -- 172.21.15.161:0/803074210 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7e0005410 msgr2=0x7fd7e000ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:38.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.820+0000 7fd7f6ffd640 1 --2- 172.21.15.161:0/803074210 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7e0005410 0x7fd7e000ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.820+0000 7fd7f6ffd640 1 -- 172.21.15.161:0/803074210 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7e0003ae0 msgr2=0x7fd7e0003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:38.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.820+0000 7fd7f6ffd640 1 --2- 172.21.15.161:0/803074210 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7e0003ae0 0x7fd7e0003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.820+0000 7fd7f6ffd640 1 -- 172.21.15.161:0/803074210 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e0003520 con 0x7fd7e0004800 2024-03-28T23:30:38.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.820+0000 7fd7f7fff640 1 --2- 172.21.15.161:0/803074210 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7e0005410 0x7fd7e000ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:38.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.820+0000 7fd7f7fff640 1 -- 172.21.15.161:0/803074210 reap_dead start 2024-03-28T23:30:38.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.821+0000 7fd7f6ffd640 1 --2- 172.21.15.161:0/803074210 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7e0004800 0x7fd7e0004c80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd7f80ad1f0 tx=0x7fd7e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d86efa61da37f4d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:38.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.821+0000 7fd7f67fc640 1 -- 172.21.15.161:0/803074210 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e8037070 con 0x7fd7e0004800 2024-03-28T23:30:38.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.821+0000 7fd7f67fc640 1 -- 172.21.15.161:0/803074210 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7e8032600 con 0x7fd7e0004800 2024-03-28T23:30:38.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.821+0000 7fd7f67fc640 1 -- 172.21.15.161:0/803074210 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e80039d0 con 0x7fd7e0004800 2024-03-28T23:30:38.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.821+0000 7fd7fe09d640 1 -- 172.21.15.161:0/803074210 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7e0004800 msgr2=0x7fd7e0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:38.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.821+0000 7fd7fe09d640 1 --2- 172.21.15.161:0/803074210 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7e0004800 0x7fd7e0004c80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd7f80ad1f0 tx=0x7fd7e0005830 comp rx=0 tx=0).stop 2024-03-28T23:30:38.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.821+0000 7fd7fe09d640 1 -- 172.21.15.161:0/803074210 shutdown_connections 2024-03-28T23:30:38.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.821+0000 7fd7fe09d640 1 --2- 172.21.15.161:0/803074210 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7e0004800 0x7fd7e0004c80 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.821+0000 7fd7fe09d640 1 -- 172.21.15.161:0/803074210 >> 172.21.15.161:0/803074210 conn(0x7fd7f80ac620 msgr2=0x7fd7f80aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:38.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.821+0000 7fd7fe09d640 1 -- 172.21.15.161:0/803074210 shutdown_connections 2024-03-28T23:30:38.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.821+0000 7fd7fe09d640 1 -- 172.21.15.161:0/803074210 wait complete. 2024-03-28T23:30:38.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.822+0000 7fd7fe09d640 1 Processor -- start 2024-03-28T23:30:38.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.823+0000 7fd7fe09d640 1 -- start start 2024-03-28T23:30:38.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.823+0000 7fd7fe09d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7e0004800 0x7fd7f8146910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:38.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.823+0000 7fd7fe09d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7f81eb760 0x7fd7f8155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:38.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.823+0000 7fd7fe09d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7f8149bd0 0x7fd7f8155e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:38.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.823+0000 7fd7fe09d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f8152c70 con 0x7fd7e0004800 2024-03-28T23:30:38.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.823+0000 7fd7fe09d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f8159640 con 0x7fd7f81eb760 2024-03-28T23:30:38.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.823+0000 7fd7fe09d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd7f8152df0 con 0x7fd7f8149bd0 2024-03-28T23:30:38.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.823+0000 7fd7f77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7e0004800 0x7fd7f8146910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:38.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.823+0000 7fd7f77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7e0004800 0x7fd7f8146910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49282/0 (socket says 172.21.15.161:49282) 2024-03-28T23:30:38.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.823+0000 7fd7f77fe640 1 -- 172.21.15.161:0/844227104 learned_addr learned my addr 172.21.15.161:0/844227104 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:38.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.824+0000 7fd7f6ffd640 1 --2- 172.21.15.161:0/844227104 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7f81eb760 0x7fd7f8155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:38.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.824+0000 7fd7f7fff640 1 --2- 172.21.15.161:0/844227104 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7f8149bd0 0x7fd7f8155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:38.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.824+0000 7fd7f77fe640 1 -- 172.21.15.161:0/844227104 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7f8149bd0 msgr2=0x7fd7f8155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.824+0000 7fd7f77fe640 1 --2- 172.21.15.161:0/844227104 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7f8149bd0 0x7fd7f8155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.824+0000 7fd7f77fe640 1 -- 172.21.15.161:0/844227104 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7f81eb760 msgr2=0x7fd7f8155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.824+0000 7fd7f77fe640 1 --2- 172.21.15.161:0/844227104 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7f81eb760 0x7fd7f8155950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.824+0000 7fd7f77fe640 1 -- 172.21.15.161:0/844227104 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f814e110 con 0x7fd7e0004800 2024-03-28T23:30:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.824+0000 7fd7f6ffd640 1 --2- 172.21.15.161:0/844227104 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7f81eb760 0x7fd7f8155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.824+0000 7fd7f77fe640 1 --2- 172.21.15.161:0/844227104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7e0004800 0x7fd7f8146910 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec00b820 tx=0x7fd7ec00bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.824+0000 7fd7dffff640 1 -- 172.21.15.161:0/844227104 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec018070 con 0x7fd7e0004800 2024-03-28T23:30:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.824+0000 7fd7dffff640 1 -- 172.21.15.161:0/844227104 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7ec010830 con 0x7fd7e0004800 2024-03-28T23:30:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.824+0000 7fd7fe09d640 1 -- 172.21.15.161:0/844227104 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7f814e690 con 0x7fd7e0004800 2024-03-28T23:30:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.824+0000 7fd7dffff640 1 -- 172.21.15.161:0/844227104 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec010c00 con 0x7fd7e0004800 2024-03-28T23:30:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.824+0000 7fd7fe09d640 1 -- 172.21.15.161:0/844227104 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7f8146e50 con 0x7fd7e0004800 2024-03-28T23:30:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.825+0000 7fd7fe09d640 1 -- 172.21.15.161:0/844227104 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7f8151150 con 0x7fd7e0004800 2024-03-28T23:30:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.826+0000 7fd7dffff640 1 -- 172.21.15.161:0/844227104 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd7ec01b070 con 0x7fd7e0004800 2024-03-28T23:30:38.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.826+0000 7fd7dffff640 1 --2- 172.21.15.161:0/844227104 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd7e406b500 0x7fd7e406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:38.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.827+0000 7fd7dffff640 1 -- 172.21.15.161:0/844227104 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fd7ec08a910 con 0x7fd7e0004800 2024-03-28T23:30:38.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.827+0000 7fd7f6ffd640 1 --2- 172.21.15.161:0/844227104 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd7e406b500 0x7fd7e406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:38.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.827+0000 7fd7f6ffd640 1 --2- 172.21.15.161:0/844227104 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd7e406b500 0x7fd7e406d9c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8034000 tx=0x7fd7e8009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:38.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:38.835+0000 7fd7dffff640 1 -- 172.21.15.161:0/844227104 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd7f8151150 con 0x7fd7e0004800 2024-03-28T23:30:39.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.009+0000 7fd7fe09d640 1 -- 172.21.15.161:0/844227104 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd7f804f4b0 con 0x7fd7e0004800 2024-03-28T23:30:39.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.011+0000 7fd7dffff640 1 -- 172.21.15.161:0/844227104 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18937 (secure 0 0 0) 0x7fd7ec059420 con 0x7fd7e0004800 2024-03-28T23:30:39.012 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:39.012 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":26,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:30:35.148429+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:30:39.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.013+0000 7fd7fe09d640 1 -- 172.21.15.161:0/844227104 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd7e406b500 msgr2=0x7fd7e406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:39.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.013+0000 7fd7fe09d640 1 --2- 172.21.15.161:0/844227104 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd7e406b500 0x7fd7e406d9c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8034000 tx=0x7fd7e8009030 comp rx=0 tx=0).stop 2024-03-28T23:30:39.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.013+0000 7fd7fe09d640 1 -- 172.21.15.161:0/844227104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7e0004800 msgr2=0x7fd7f8146910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:39.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.013+0000 7fd7fe09d640 1 --2- 172.21.15.161:0/844227104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7e0004800 0x7fd7f8146910 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec00b820 tx=0x7fd7ec00bc20 comp rx=0 tx=0).stop 2024-03-28T23:30:39.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.013+0000 7fd7fe09d640 1 -- 172.21.15.161:0/844227104 shutdown_connections 2024-03-28T23:30:39.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.013+0000 7fd7fe09d640 1 --2- 172.21.15.161:0/844227104 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd7e406b500 0x7fd7e406d9c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.013+0000 7fd7fe09d640 1 --2- 172.21.15.161:0/844227104 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7f8149bd0 0x7fd7f8155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.013+0000 7fd7fe09d640 1 --2- 172.21.15.161:0/844227104 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7f81eb760 0x7fd7f8155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.013+0000 7fd7fe09d640 1 --2- 172.21.15.161:0/844227104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7e0004800 0x7fd7f8146910 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.013+0000 7fd7fe09d640 1 -- 172.21.15.161:0/844227104 >> 172.21.15.161:0/844227104 conn(0x7fd7f80ac620 msgr2=0x7fd7f80c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:39.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.014+0000 7fd7fe09d640 1 -- 172.21.15.161:0/844227104 shutdown_connections 2024-03-28T23:30:39.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.014+0000 7fd7fe09d640 1 -- 172.21.15.161:0/844227104 wait complete. 2024-03-28T23:30:39.029 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:29:44.021375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:29:44.872102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T23:29:48.591063+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T23:30:39.029 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:30:39.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.134+0000 7f0a94725640 1 Processor -- start 2024-03-28T23:30:39.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.134+0000 7f0a94725640 1 -- start start 2024-03-28T23:30:39.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.134+0000 7f0a94725640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0a8c1529d0 0x7f0a8c06bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:39.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.135+0000 7f0a94725640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0a8c0c88b0 0x7f0a8c0c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:39.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.135+0000 7f0a94725640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a8c152db0 con 0x7f0a8c0c6750 2024-03-28T23:30:39.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.135+0000 7f0a94725640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0a8c0c8d60 con 0x7f0a8c0c88b0 2024-03-28T23:30:39.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.135+0000 7f0a94725640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0a8c0c7210 con 0x7f0a8c1529d0 2024-03-28T23:30:39.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.135+0000 7f0a92cc2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0a8c0c88b0 0x7f0a8c0c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:39.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.135+0000 7f0a91cc0640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f0a8c0c6750 0x7f0a8c155c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60198/0 (socket says 172.21.15.161:60198) 2024-03-28T23:30:39.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.135+0000 7f0a91cc0640 1 -- 172.21.15.161:0/1851982594 learned_addr learned my addr 172.21.15.161:0/1851982594 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:39.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.135+0000 7f0a924c1640 1 --2- 172.21.15.161:0/1851982594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0a8c1529d0 0x7f0a8c06bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:39.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.135+0000 7f0a92cc2640 1 -- 172.21.15.161:0/1851982594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0a8c1529d0 msgr2=0x7f0a8c06bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:39.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.135+0000 7f0a92cc2640 1 --2- 172.21.15.161:0/1851982594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0a8c1529d0 0x7f0a8c06bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.135+0000 7f0a92cc2640 1 -- 172.21.15.161:0/1851982594 >> v1:172.21.15.161:6789/0 conn(0x7f0a8c0c6750 legacy=0x7f0a8c155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:39.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.135+0000 7f0a92cc2640 1 -- 172.21.15.161:0/1851982594 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a8c0c6c50 con 0x7f0a8c0c88b0 2024-03-28T23:30:39.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.135+0000 7f0a914bf640 1 -- 172.21.15.161:0/1851982594 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1721635457 0 0) 0x7f0a8c152db0 con 0x7f0a8c0c6750 2024-03-28T23:30:39.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.135+0000 7f0a924c1640 1 --2- 172.21.15.161:0/1851982594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0a8c1529d0 0x7f0a8c06bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:39.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.136+0000 7f0a92cc2640 1 --2- 172.21.15.161:0/1851982594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a8c0c88b0 0x7f0a8c0c54f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0a88009d90 tx=0x7f0a8802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b94589002337a6e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:39.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.136+0000 7f0a914bf640 1 -- 172.21.15.161:0/1851982594 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a88039070 con 0x7f0a8c0c88b0 2024-03-28T23:30:39.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.136+0000 7f0a914bf640 1 -- 172.21.15.161:0/1851982594 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a8802cc50 con 0x7f0a8c0c88b0 2024-03-28T23:30:39.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.136+0000 7f0a914bf640 1 -- 172.21.15.161:0/1851982594 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a88034440 con 0x7f0a8c0c88b0 2024-03-28T23:30:39.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.137+0000 7f0a94725640 1 -- 172.21.15.161:0/1851982594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a8c0c88b0 msgr2=0x7f0a8c0c54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:39.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.137+0000 7f0a94725640 1 --2- 172.21.15.161:0/1851982594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a8c0c88b0 0x7f0a8c0c54f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0a88009d90 tx=0x7f0a8802bea0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.137+0000 7f0a94725640 1 -- 172.21.15.161:0/1851982594 shutdown_connections 2024-03-28T23:30:39.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.137+0000 7f0a94725640 1 --2- 172.21.15.161:0/1851982594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a8c0c88b0 0x7f0a8c0c54f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.137+0000 7f0a94725640 1 --2- 172.21.15.161:0/1851982594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0a8c1529d0 0x7f0a8c06bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.137+0000 7f0a94725640 1 -- 172.21.15.161:0/1851982594 >> 172.21.15.161:0/1851982594 conn(0x7f0a8c0c7430 msgr2=0x7f0a8c0c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:39.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.137+0000 7f0a94725640 1 -- 172.21.15.161:0/1851982594 shutdown_connections 2024-03-28T23:30:39.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.137+0000 7f0a94725640 1 -- 172.21.15.161:0/1851982594 wait complete. 2024-03-28T23:30:39.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.138+0000 7f0a94725640 1 Processor -- start 2024-03-28T23:30:39.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.138+0000 7f0a94725640 1 -- start start 2024-03-28T23:30:39.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.139+0000 7f0a94725640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a8c0c6750 0x7f0a8c1eab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:39.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.139+0000 7f0a94725640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a8c0c88b0 0x7f0a8c0aed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:39.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.139+0000 7f0a94725640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0a8c1529d0 0x7f0a8c1492f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:39.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.139+0000 7f0a94725640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c155710 con 0x7f0a8c0c88b0 2024-03-28T23:30:39.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.139+0000 7f0a94725640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c155430 con 0x7f0a8c0c6750 2024-03-28T23:30:39.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.139+0000 7f0a94725640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0a8c0aa160 con 0x7f0a8c1529d0 2024-03-28T23:30:39.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.139+0000 7f0a91cc0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a8c0c88b0 0x7f0a8c0aed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:39.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.139+0000 7f0a91cc0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a8c0c88b0 0x7f0a8c0aed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49304/0 (socket says 172.21.15.161:49304) 2024-03-28T23:30:39.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.139+0000 7f0a91cc0640 1 -- 172.21.15.161:0/1594509384 learned_addr learned my addr 172.21.15.161:0/1594509384 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:39.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.139+0000 7f0a924c1640 1 --2- 172.21.15.161:0/1594509384 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a8c0c6750 0x7f0a8c1eab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:39.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.139+0000 7f0a92cc2640 1 --2- 172.21.15.161:0/1594509384 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0a8c1529d0 0x7f0a8c1492f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:39.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.139+0000 7f0a91cc0640 1 -- 172.21.15.161:0/1594509384 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0a8c1529d0 msgr2=0x7f0a8c1492f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:39.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.139+0000 7f0a91cc0640 1 --2- 172.21.15.161:0/1594509384 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0a8c1529d0 0x7f0a8c1492f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.139+0000 7f0a91cc0640 1 -- 172.21.15.161:0/1594509384 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a8c0c6750 msgr2=0x7f0a8c1eab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:39.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.139+0000 7f0a91cc0640 1 --2- 172.21.15.161:0/1594509384 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a8c0c6750 0x7f0a8c1eab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.139+0000 7f0a91cc0640 1 -- 172.21.15.161:0/1594509384 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a8c0adc60 con 0x7f0a8c0c88b0 2024-03-28T23:30:39.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.139+0000 7f0a924c1640 1 --2- 172.21.15.161:0/1594509384 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a8c0c6750 0x7f0a8c1eab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:39.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.139+0000 7f0a91cc0640 1 --2- 172.21.15.161:0/1594509384 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a8c0c88b0 0x7f0a8c0aed00 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c00c460 tx=0x7f0a7c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:39.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.139+0000 7f0a82ffd640 1 -- 172.21.15.161:0/1594509384 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c018070 con 0x7f0a8c0c88b0 2024-03-28T23:30:39.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.140+0000 7f0a82ffd640 1 -- 172.21.15.161:0/1594509384 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a7c007500 con 0x7f0a8c0c88b0 2024-03-28T23:30:39.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.140+0000 7f0a94725640 1 -- 172.21.15.161:0/1594509384 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a8c0adf40 con 0x7f0a8c0c88b0 2024-03-28T23:30:39.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.140+0000 7f0a82ffd640 1 -- 172.21.15.161:0/1594509384 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c004a20 con 0x7f0a8c0c88b0 2024-03-28T23:30:39.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.140+0000 7f0a94725640 1 -- 172.21.15.161:0/1594509384 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a8c0ae500 con 0x7f0a8c0c88b0 2024-03-28T23:30:39.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.141+0000 7f0a94725640 1 -- 172.21.15.161:0/1594509384 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a8c04f4b0 con 0x7f0a8c0c88b0 2024-03-28T23:30:39.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.141+0000 7f0a82ffd640 1 -- 172.21.15.161:0/1594509384 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0a7c01b070 con 0x7f0a8c0c88b0 2024-03-28T23:30:39.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.141+0000 7f0a82ffd640 1 --2- 172.21.15.161:0/1594509384 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0a7006b4d0 0x7f0a7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:39.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.141+0000 7f0a82ffd640 1 -- 172.21.15.161:0/1594509384 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f0a7c0897d0 con 0x7f0a8c0c88b0 2024-03-28T23:30:39.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.141+0000 7f0a924c1640 1 --2- 172.21.15.161:0/1594509384 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0a7006b4d0 0x7f0a7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:39.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.142+0000 7f0a924c1640 1 --2- 172.21.15.161:0/1594509384 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0a7006b4d0 0x7f0a7006d990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0a8400aa30 tx=0x7f0a84008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:39.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.146+0000 7f0a82ffd640 1 -- 172.21.15.161:0/1594509384 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0a7c0582e0 con 0x7f0a8c0c88b0 2024-03-28T23:30:39.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.326+0000 7f0a94725640 1 -- 172.21.15.161:0/1594509384 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0a8c0c0cc0 con 0x7f0a8c0c88b0 2024-03-28T23:30:39.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.326+0000 7f0a82ffd640 1 -- 172.21.15.161:0/1594509384 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f0a7c057be0 con 0x7f0a8c0c88b0 2024-03-28T23:30:39.327 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:30:39.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.328+0000 7f0a94725640 1 -- 172.21.15.161:0/1594509384 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0a7006b4d0 msgr2=0x7f0a7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:39.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.328+0000 7f0a94725640 1 --2- 172.21.15.161:0/1594509384 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0a7006b4d0 0x7f0a7006d990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0a8400aa30 tx=0x7f0a84008040 comp rx=0 tx=0).stop 2024-03-28T23:30:39.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.328+0000 7f0a94725640 1 -- 172.21.15.161:0/1594509384 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a8c0c88b0 msgr2=0x7f0a8c0aed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:39.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.328+0000 7f0a94725640 1 --2- 172.21.15.161:0/1594509384 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a8c0c88b0 0x7f0a8c0aed00 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c00c460 tx=0x7f0a7c00c920 comp rx=0 tx=0).stop 2024-03-28T23:30:39.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.328+0000 7f0a94725640 1 -- 172.21.15.161:0/1594509384 shutdown_connections 2024-03-28T23:30:39.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.328+0000 7f0a94725640 1 --2- 172.21.15.161:0/1594509384 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0a7006b4d0 0x7f0a7006d990 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.328+0000 7f0a94725640 1 --2- 172.21.15.161:0/1594509384 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a8c0c88b0 0x7f0a8c0aed00 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.328+0000 7f0a94725640 1 --2- 172.21.15.161:0/1594509384 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0a8c1529d0 0x7f0a8c1492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.328+0000 7f0a94725640 1 --2- 172.21.15.161:0/1594509384 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a8c0c6750 0x7f0a8c1eab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.328+0000 7f0a94725640 1 -- 172.21.15.161:0/1594509384 >> 172.21.15.161:0/1594509384 conn(0x7f0a8c0c7430 msgr2=0x7f0a8c074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:39.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.328+0000 7f0a94725640 1 -- 172.21.15.161:0/1594509384 shutdown_connections 2024-03-28T23:30:39.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.328+0000 7f0a94725640 1 -- 172.21.15.161:0/1594509384 wait complete. 2024-03-28T23:30:39.340 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:30:39.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.444+0000 7fd1da809640 1 Processor -- start 2024-03-28T23:30:39.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.444+0000 7fd1da809640 1 -- start start 2024-03-28T23:30:39.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.445+0000 7fd1da809640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd1d40c0330 0x7fd1d40c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:39.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.445+0000 7fd1da809640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1d40c6f60 con 0x7fd1d4152bc0 2024-03-28T23:30:39.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.445+0000 7fd1da809640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1d40c6120 con 0x7fd1d41589b0 2024-03-28T23:30:39.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.445+0000 7fd1da809640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd1d40c6400 con 0x7fd1d40c0330 2024-03-28T23:30:39.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.445+0000 7fd1d37fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd1d40c0330 0x7fd1d40c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:39.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.445+0000 7fd1d3fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd1d4152bc0 0x7fd1d40c8a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60214/0 (socket says 172.21.15.161:60214) 2024-03-28T23:30:39.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.445+0000 7fd1d3fff640 1 -- 172.21.15.161:0/3624514574 learned_addr learned my addr 172.21.15.161:0/3624514574 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:39.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.445+0000 7fd1d37fe640 1 --2- 172.21.15.161:0/3624514574 >> v2:172.21.15.161:3300/0 conn(0x7fd1d40c0330 0x7fd1d40c9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49310/0 (socket says 172.21.15.161:49310) 2024-03-28T23:30:39.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.445+0000 7fd1d37fe640 1 -- 172.21.15.161:0/3624514574 >> v1:172.21.15.170:6789/0 conn(0x7fd1d41589b0 legacy=0x7fd1d4158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:39.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.445+0000 7fd1d37fe640 1 -- 172.21.15.161:0/3624514574 >> v1:172.21.15.161:6789/0 conn(0x7fd1d4152bc0 legacy=0x7fd1d40c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:39.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.445+0000 7fd1d37fe640 1 -- 172.21.15.161:0/3624514574 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d40c96d0 con 0x7fd1d40c0330 2024-03-28T23:30:39.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.445+0000 7fd1d2ffd640 1 -- 172.21.15.161:0/3624514574 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4045852428 0 0) 0x7fd1d40c6f60 con 0x7fd1d4152bc0 2024-03-28T23:30:39.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.446+0000 7fd1d37fe640 1 --2- 172.21.15.161:0/3624514574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1d40c0330 0x7fd1d40c9190 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd1c80097a0 tx=0x7fd1c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a977b59b3ea198a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:39.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.446+0000 7fd1d2ffd640 1 -- 172.21.15.161:0/3624514574 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8035020 con 0x7fd1d40c0330 2024-03-28T23:30:39.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.446+0000 7fd1d2ffd640 1 -- 172.21.15.161:0/3624514574 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1c8031830 con 0x7fd1d40c0330 2024-03-28T23:30:39.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.446+0000 7fd1d2ffd640 1 -- 172.21.15.161:0/3624514574 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8031c00 con 0x7fd1d40c0330 2024-03-28T23:30:39.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.446+0000 7fd1da809640 1 -- 172.21.15.161:0/3624514574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1d40c0330 msgr2=0x7fd1d40c9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:39.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.446+0000 7fd1da809640 1 --2- 172.21.15.161:0/3624514574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1d40c0330 0x7fd1d40c9190 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd1c80097a0 tx=0x7fd1c802b8d0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.446+0000 7fd1da809640 1 -- 172.21.15.161:0/3624514574 shutdown_connections 2024-03-28T23:30:39.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.446+0000 7fd1da809640 1 --2- 172.21.15.161:0/3624514574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1d40c0330 0x7fd1d40c9190 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.446+0000 7fd1da809640 1 -- 172.21.15.161:0/3624514574 >> 172.21.15.161:0/3624514574 conn(0x7fd1d40caa80 msgr2=0x7fd1d40cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:39.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.446+0000 7fd1da809640 1 -- 172.21.15.161:0/3624514574 shutdown_connections 2024-03-28T23:30:39.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.446+0000 7fd1da809640 1 -- 172.21.15.161:0/3624514574 wait complete. 2024-03-28T23:30:39.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.448+0000 7fd1da809640 1 Processor -- start 2024-03-28T23:30:39.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.448+0000 7fd1da809640 1 -- start start 2024-03-28T23:30:39.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.449+0000 7fd1da809640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1d40c0330 0x7fd1d4149430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:39.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.449+0000 7fd1da809640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1d4152bc0 0x7fd1d4151340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:39.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.449+0000 7fd1da809640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1d41589b0 0x7fd1d4151880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:39.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.449+0000 7fd1da809640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d4152fa0 con 0x7fd1d40c0330 2024-03-28T23:30:39.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.449+0000 7fd1da809640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d40c6bf0 con 0x7fd1d41589b0 2024-03-28T23:30:39.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.449+0000 7fd1da809640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd1d4150eb0 con 0x7fd1d4152bc0 2024-03-28T23:30:39.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.449+0000 7fd1d3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1d40c0330 0x7fd1d4149430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:39.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.449+0000 7fd1d3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1d40c0330 0x7fd1d4149430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49318/0 (socket says 172.21.15.161:49318) 2024-03-28T23:30:39.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.449+0000 7fd1d3fff640 1 -- 172.21.15.161:0/474696184 learned_addr learned my addr 172.21.15.161:0/474696184 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:39.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.449+0000 7fd1d8da6640 1 --2- 172.21.15.161:0/474696184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1d41589b0 0x7fd1d4151880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:39.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.449+0000 7fd1d37fe640 1 --2- 172.21.15.161:0/474696184 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1d4152bc0 0x7fd1d4151340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:39.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.449+0000 7fd1d3fff640 1 -- 172.21.15.161:0/474696184 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1d4152bc0 msgr2=0x7fd1d4151340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:39.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.449+0000 7fd1d3fff640 1 --2- 172.21.15.161:0/474696184 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1d4152bc0 0x7fd1d4151340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.449+0000 7fd1d3fff640 1 -- 172.21.15.161:0/474696184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1d41589b0 msgr2=0x7fd1d4151880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:39.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.449+0000 7fd1d3fff640 1 --2- 172.21.15.161:0/474696184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1d41589b0 0x7fd1d4151880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.449+0000 7fd1d3fff640 1 -- 172.21.15.161:0/474696184 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d4150580 con 0x7fd1d40c0330 2024-03-28T23:30:39.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.449+0000 7fd1d3fff640 1 --2- 172.21.15.161:0/474696184 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1d40c0330 0x7fd1d4149430 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400c4d0 tx=0x7fd1c400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:39.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.450+0000 7fd1d0ff9640 1 -- 172.21.15.161:0/474696184 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c4018070 con 0x7fd1d40c0330 2024-03-28T23:30:39.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.450+0000 7fd1d0ff9640 1 -- 172.21.15.161:0/474696184 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1c4007500 con 0x7fd1d40c0330 2024-03-28T23:30:39.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.450+0000 7fd1da809640 1 -- 172.21.15.161:0/474696184 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1d4150860 con 0x7fd1d40c0330 2024-03-28T23:30:39.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.450+0000 7fd1d0ff9640 1 -- 172.21.15.161:0/474696184 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c4004a20 con 0x7fd1d40c0330 2024-03-28T23:30:39.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.450+0000 7fd1da809640 1 -- 172.21.15.161:0/474696184 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1d4149970 con 0x7fd1d40c0330 2024-03-28T23:30:39.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.451+0000 7fd1da809640 1 -- 172.21.15.161:0/474696184 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1d404f4b0 con 0x7fd1d40c0330 2024-03-28T23:30:39.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.451+0000 7fd1d0ff9640 1 -- 172.21.15.161:0/474696184 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd1c401b070 con 0x7fd1d40c0330 2024-03-28T23:30:39.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.451+0000 7fd1d0ff9640 1 --2- 172.21.15.161:0/474696184 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd1c006b4d0 0x7fd1c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:39.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.452+0000 7fd1d0ff9640 1 -- 172.21.15.161:0/474696184 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fd1c40897e0 con 0x7fd1d40c0330 2024-03-28T23:30:39.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.452+0000 7fd1d37fe640 1 --2- 172.21.15.161:0/474696184 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd1c006b4d0 0x7fd1c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:39.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.452+0000 7fd1d37fe640 1 --2- 172.21.15.161:0/474696184 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd1c006b4d0 0x7fd1c006d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8036000 tx=0x7fd1c803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:39.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.456+0000 7fd1d0ff9640 1 -- 172.21.15.161:0/474696184 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd1c4057ca0 con 0x7fd1d40c0330 2024-03-28T23:30:39.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.638+0000 7fd1da809640 1 -- 172.21.15.161:0/474696184 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd1d4148d00 con 0x7fd1d40c0330 2024-03-28T23:30:39.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.638+0000 7fd1d0ff9640 1 -- 172.21.15.161:0/474696184 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fd1c405bd40 con 0x7fd1d40c0330 2024-03-28T23:30:39.639 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:30:39.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.640+0000 7fd1da809640 1 -- 172.21.15.161:0/474696184 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd1c006b4d0 msgr2=0x7fd1c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:39.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.640+0000 7fd1da809640 1 --2- 172.21.15.161:0/474696184 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd1c006b4d0 0x7fd1c006d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8036000 tx=0x7fd1c803c040 comp rx=0 tx=0).stop 2024-03-28T23:30:39.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.640+0000 7fd1da809640 1 -- 172.21.15.161:0/474696184 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1d40c0330 msgr2=0x7fd1d4149430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:39.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.640+0000 7fd1da809640 1 --2- 172.21.15.161:0/474696184 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1d40c0330 0x7fd1d4149430 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400c4d0 tx=0x7fd1c400c990 comp rx=0 tx=0).stop 2024-03-28T23:30:39.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.640+0000 7fd1da809640 1 -- 172.21.15.161:0/474696184 shutdown_connections 2024-03-28T23:30:39.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.640+0000 7fd1da809640 1 --2- 172.21.15.161:0/474696184 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd1c006b4d0 0x7fd1c006d990 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.640+0000 7fd1da809640 1 --2- 172.21.15.161:0/474696184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1d41589b0 0x7fd1d4151880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.640+0000 7fd1da809640 1 --2- 172.21.15.161:0/474696184 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1d4152bc0 0x7fd1d4151340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.640+0000 7fd1da809640 1 --2- 172.21.15.161:0/474696184 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1d40c0330 0x7fd1d4149430 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.640+0000 7fd1da809640 1 -- 172.21.15.161:0/474696184 >> 172.21.15.161:0/474696184 conn(0x7fd1d40caa80 msgr2=0x7fd1d414cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:39.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.640+0000 7fd1da809640 1 -- 172.21.15.161:0/474696184 shutdown_connections 2024-03-28T23:30:39.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.640+0000 7fd1da809640 1 -- 172.21.15.161:0/474696184 wait complete. 2024-03-28T23:30:39.652 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:30:39.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.756+0000 7f914cd1e640 1 Processor -- start 2024-03-28T23:30:39.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.756+0000 7f914cd1e640 1 -- start start 2024-03-28T23:30:39.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.757+0000 7f914cd1e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f91480c1b40 0x7f91480c2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:39.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.757+0000 7f914cd1e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f914815a3a0 con 0x7f9148156b20 2024-03-28T23:30:39.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.757+0000 7f914cd1e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91480c2250 con 0x7f91480c1290 2024-03-28T23:30:39.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.757+0000 7f914cd1e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f91480c2000 con 0x7f91480c1b40 2024-03-28T23:30:39.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.757+0000 7f9145d9b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f91480c1b40 0x7f91480c2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:39.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.757+0000 7f9145d9b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f91480c1b40 0x7f91480c2d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49326/0 (socket says 172.21.15.161:49326) 2024-03-28T23:30:39.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.757+0000 7f914659c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f9148156b20 0x7f91480c4790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60224/0 (socket says 172.21.15.161:60224) 2024-03-28T23:30:39.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.757+0000 7f9145d9b640 1 -- 172.21.15.161:0/939298745 learned_addr learned my addr 172.21.15.161:0/939298745 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:39.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.758+0000 7f9145d9b640 1 -- 172.21.15.161:0/939298745 >> v1:172.21.15.170:6789/0 conn(0x7f91480c1290 legacy=0x7f914806c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:39.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.758+0000 7f9145d9b640 1 -- 172.21.15.161:0/939298745 >> v1:172.21.15.161:6789/0 conn(0x7f9148156b20 legacy=0x7f91480c4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:39.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.758+0000 7f914559a640 1 -- 172.21.15.161:0/939298745 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1691416293 0 0) 0x7f914815a3a0 con 0x7f9148156b20 2024-03-28T23:30:39.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.758+0000 7f9145d9b640 1 -- 172.21.15.161:0/939298745 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91480c2530 con 0x7f91480c1b40 2024-03-28T23:30:39.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.758+0000 7f9145d9b640 1 --2- 172.21.15.161:0/939298745 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f91480c1b40 0x7f91480c2d30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9138009870 tx=0x7f913802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77353f6d34db20ad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:39.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.758+0000 7f914559a640 1 -- 172.21.15.161:0/939298745 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f913802f840 con 0x7f91480c1b40 2024-03-28T23:30:39.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.759+0000 7f914559a640 1 -- 172.21.15.161:0/939298745 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f913802f9e0 con 0x7f91480c1b40 2024-03-28T23:30:39.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.759+0000 7f914559a640 1 -- 172.21.15.161:0/939298745 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f913802fdb0 con 0x7f91480c1b40 2024-03-28T23:30:39.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.759+0000 7f914cd1e640 1 -- 172.21.15.161:0/939298745 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f91480c1b40 msgr2=0x7f91480c2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:39.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.759+0000 7f914cd1e640 1 --2- 172.21.15.161:0/939298745 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f91480c1b40 0x7f91480c2d30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9138009870 tx=0x7f913802ba20 comp rx=0 tx=0).stop 2024-03-28T23:30:39.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.759+0000 7f914cd1e640 1 -- 172.21.15.161:0/939298745 shutdown_connections 2024-03-28T23:30:39.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.759+0000 7f914cd1e640 1 --2- 172.21.15.161:0/939298745 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f91480c1b40 0x7f91480c2d30 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.759+0000 7f914cd1e640 1 -- 172.21.15.161:0/939298745 >> 172.21.15.161:0/939298745 conn(0x7f91480c3390 msgr2=0x7f91480c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:39.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.759+0000 7f914cd1e640 1 -- 172.21.15.161:0/939298745 shutdown_connections 2024-03-28T23:30:39.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.759+0000 7f914cd1e640 1 -- 172.21.15.161:0/939298745 wait complete. 2024-03-28T23:30:39.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.760+0000 7f914cd1e640 1 Processor -- start 2024-03-28T23:30:39.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.760+0000 7f914cd1e640 1 -- start start 2024-03-28T23:30:39.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.761+0000 7f914cd1e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91480c1290 0x7f91480ab870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:39.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.761+0000 7f914cd1e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91480c1b40 0x7f91481ef580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:39.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.761+0000 7f914cd1e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9148156b20 0x7f91481efac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:39.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.761+0000 7f914cd1e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f914806cdd0 con 0x7f9148156b20 2024-03-28T23:30:39.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.761+0000 7f914cd1e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9148156f00 con 0x7f91480c1b40 2024-03-28T23:30:39.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.761+0000 7f914cd1e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f914806cf50 con 0x7f91480c1290 2024-03-28T23:30:39.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.761+0000 7f9146d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9148156b20 0x7f91481efac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:39.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.761+0000 7f9146d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9148156b20 0x7f91481efac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49332/0 (socket says 172.21.15.161:49332) 2024-03-28T23:30:39.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.761+0000 7f9146d9d640 1 -- 172.21.15.161:0/161251856 learned_addr learned my addr 172.21.15.161:0/161251856 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:39.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.761+0000 7f914659c640 1 --2- 172.21.15.161:0/161251856 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91480c1290 0x7f91480ab870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:39.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.761+0000 7f9145d9b640 1 --2- 172.21.15.161:0/161251856 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91480c1b40 0x7f91481ef580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:39.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.762+0000 7f9146d9d640 1 -- 172.21.15.161:0/161251856 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91480c1290 msgr2=0x7f91480ab870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:39.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.762+0000 7f9146d9d640 1 --2- 172.21.15.161:0/161251856 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91480c1290 0x7f91480ab870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.762+0000 7f9146d9d640 1 -- 172.21.15.161:0/161251856 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91480c1b40 msgr2=0x7f91481ef580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:39.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.762+0000 7f9146d9d640 1 --2- 172.21.15.161:0/161251856 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91480c1b40 0x7f91481ef580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.762+0000 7f9146d9d640 1 -- 172.21.15.161:0/161251856 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9148154cf0 con 0x7f9148156b20 2024-03-28T23:30:39.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.762+0000 7f914659c640 1 --2- 172.21.15.161:0/161251856 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91480c1290 0x7f91480ab870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:39.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.762+0000 7f9145d9b640 1 --2- 172.21.15.161:0/161251856 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91480c1b40 0x7f91481ef580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:39.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.762+0000 7f9146d9d640 1 --2- 172.21.15.161:0/161251856 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9148156b20 0x7f91481efac0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f914000c7d0 tx=0x7f914000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:39.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.762+0000 7f913effd640 1 -- 172.21.15.161:0/161251856 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9140003b70 con 0x7f9148156b20 2024-03-28T23:30:39.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.762+0000 7f913effd640 1 -- 172.21.15.161:0/161251856 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9140003d30 con 0x7f9148156b20 2024-03-28T23:30:39.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.762+0000 7f914cd1e640 1 -- 172.21.15.161:0/161251856 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f914815cb50 con 0x7f9148156b20 2024-03-28T23:30:39.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.762+0000 7f913effd640 1 -- 172.21.15.161:0/161251856 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914000aa90 con 0x7f9148156b20 2024-03-28T23:30:39.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.762+0000 7f914cd1e640 1 -- 172.21.15.161:0/161251856 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f914815b0e0 con 0x7f9148156b20 2024-03-28T23:30:39.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.763+0000 7f914cd1e640 1 -- 172.21.15.161:0/161251856 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9148159f90 con 0x7f9148156b20 2024-03-28T23:30:39.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.763+0000 7f913effd640 1 -- 172.21.15.161:0/161251856 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f914000ac30 con 0x7f9148156b20 2024-03-28T23:30:39.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.764+0000 7f913effd640 1 --2- 172.21.15.161:0/161251856 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f912806b500 0x7f912806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:39.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.764+0000 7f914659c640 1 --2- 172.21.15.161:0/161251856 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f912806b500 0x7f912806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:39.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.764+0000 7f913effd640 1 -- 172.21.15.161:0/161251856 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f914008a570 con 0x7f9148156b20 2024-03-28T23:30:39.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.764+0000 7f914659c640 1 --2- 172.21.15.161:0/161251856 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f912806b500 0x7f912806d9c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9130007fd0 tx=0x7f9130007dc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:39.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.768+0000 7f913effd640 1 -- 172.21.15.161:0/161251856 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9148159f90 con 0x7f9148156b20 2024-03-28T23:30:39.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.992+0000 7f914cd1e640 1 -- 172.21.15.161:0/161251856 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f914804f4b0 con 0x7f9148156b20 2024-03-28T23:30:39.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.993+0000 7f913effd640 1 -- 172.21.15.161:0/161251856 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f9140058ff0 con 0x7f9148156b20 2024-03-28T23:30:39.994 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:30:39.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.995+0000 7f914cd1e640 1 -- 172.21.15.161:0/161251856 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f912806b500 msgr2=0x7f912806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:39.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.995+0000 7f914cd1e640 1 --2- 172.21.15.161:0/161251856 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f912806b500 0x7f912806d9c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9130007fd0 tx=0x7f9130007dc0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.995+0000 7f914cd1e640 1 -- 172.21.15.161:0/161251856 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9148156b20 msgr2=0x7f91481efac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:39.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.995+0000 7f914cd1e640 1 --2- 172.21.15.161:0/161251856 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9148156b20 0x7f91481efac0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f914000c7d0 tx=0x7f914000cc90 comp rx=0 tx=0).stop 2024-03-28T23:30:39.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.995+0000 7f914cd1e640 1 -- 172.21.15.161:0/161251856 shutdown_connections 2024-03-28T23:30:39.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.995+0000 7f914cd1e640 1 --2- 172.21.15.161:0/161251856 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f912806b500 0x7f912806d9c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.995+0000 7f914cd1e640 1 --2- 172.21.15.161:0/161251856 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9148156b20 0x7f91481efac0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.995+0000 7f914cd1e640 1 --2- 172.21.15.161:0/161251856 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91480c1b40 0x7f91481ef580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.995+0000 7f914cd1e640 1 --2- 172.21.15.161:0/161251856 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91480c1290 0x7f91480ab870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:39.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.995+0000 7f914cd1e640 1 -- 172.21.15.161:0/161251856 >> 172.21.15.161:0/161251856 conn(0x7f91480c3390 msgr2=0x7f914806c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:39.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.995+0000 7f914cd1e640 1 -- 172.21.15.161:0/161251856 shutdown_connections 2024-03-28T23:30:39.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:39.995+0000 7f914cd1e640 1 -- 172.21.15.161:0/161251856 wait complete. 2024-03-28T23:30:40.010 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T23:30:40.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.114+0000 7fa50861c640 1 Processor -- start 2024-03-28T23:30:40.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.114+0000 7fa50861c640 1 -- start start 2024-03-28T23:30:40.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.115+0000 7fa50861c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa500152bc0 0x7fa5000c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:40.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.115+0000 7fa50861c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa5000c8a80 0x7fa5000c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:40.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.115+0000 7fa50861c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5000c6f60 con 0x7fa5000c9420 2024-03-28T23:30:40.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.115+0000 7fa50861c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa5000c6120 con 0x7fa5000c8a80 2024-03-28T23:30:40.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.115+0000 7fa50861c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fa5000c6400 con 0x7fa500152bc0 2024-03-28T23:30:40.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.115+0000 7fa505bb7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa5000c8a80 0x7fa5000c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:40.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.116+0000 7fa505bb7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa5000c8a80 0x7fa5000c8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49342/0 (socket says 172.21.15.161:49342) 2024-03-28T23:30:40.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.116+0000 7fa505bb7640 1 -- 172.21.15.161:0/3484753125 learned_addr learned my addr 172.21.15.161:0/3484753125 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:40.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.116+0000 7fa506bb9640 1 --1- 172.21.15.161:0/3484753125 >> v1:172.21.15.161:6789/0 conn(0x7fa5000c9420 0x7fa5001589b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60232/0 (socket says 172.21.15.161:60232) 2024-03-28T23:30:40.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.116+0000 7fa5063b8640 1 --2- 172.21.15.161:0/3484753125 >> v2:172.21.15.170:3300/0 conn(0x7fa500152bc0 0x7fa5000c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:40.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.116+0000 7fa505bb7640 1 -- 172.21.15.161:0/3484753125 >> v2:172.21.15.170:3300/0 conn(0x7fa500152bc0 msgr2=0x7fa5000c0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:40.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.116+0000 7fa505bb7640 1 --2- 172.21.15.161:0/3484753125 >> v2:172.21.15.170:3300/0 conn(0x7fa500152bc0 0x7fa5000c0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:40.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.116+0000 7fa505bb7640 1 -- 172.21.15.161:0/3484753125 >> v1:172.21.15.161:6789/0 conn(0x7fa5000c9420 legacy=0x7fa5001589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:40.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.116+0000 7fa5053b6640 1 -- 172.21.15.161:0/3484753125 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 642621601 0 0) 0x7fa5000c6f60 con 0x7fa5000c9420 2024-03-28T23:30:40.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.116+0000 7fa505bb7640 1 -- 172.21.15.161:0/3484753125 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5000c7240 con 0x7fa5000c8a80 2024-03-28T23:30:40.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.117+0000 7fa505bb7640 1 --2- 172.21.15.161:0/3484753125 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5000c8a80 0x7fa5000c8ee0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa5000aa410 tx=0x7fa4f002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3f23c2078cdcf42 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:40.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.117+0000 7fa5053b6640 1 -- 172.21.15.161:0/3484753125 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f0034070 con 0x7fa5000c8a80 2024-03-28T23:30:40.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.117+0000 7fa5053b6640 1 -- 172.21.15.161:0/3484753125 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4f0030860 con 0x7fa5000c8a80 2024-03-28T23:30:40.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.117+0000 7fa5053b6640 1 -- 172.21.15.161:0/3484753125 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f0030c30 con 0x7fa5000c8a80 2024-03-28T23:30:40.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.117+0000 7fa50861c640 1 -- 172.21.15.161:0/3484753125 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5000c8a80 msgr2=0x7fa5000c8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:40.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.117+0000 7fa50861c640 1 --2- 172.21.15.161:0/3484753125 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5000c8a80 0x7fa5000c8ee0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa5000aa410 tx=0x7fa4f002b9a0 comp rx=0 tx=0).stop 2024-03-28T23:30:40.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.117+0000 7fa50861c640 1 -- 172.21.15.161:0/3484753125 shutdown_connections 2024-03-28T23:30:40.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.117+0000 7fa50861c640 1 --2- 172.21.15.161:0/3484753125 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5000c8a80 0x7fa5000c8ee0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:40.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.117+0000 7fa50861c640 1 --2- 172.21.15.161:0/3484753125 >> v2:172.21.15.170:3300/0 conn(0x7fa500152bc0 0x7fa5000c0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:40.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.117+0000 7fa50861c640 1 -- 172.21.15.161:0/3484753125 >> 172.21.15.161:0/3484753125 conn(0x7fa5000caa80 msgr2=0x7fa5000cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:40.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.117+0000 7fa50861c640 1 -- 172.21.15.161:0/3484753125 shutdown_connections 2024-03-28T23:30:40.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.117+0000 7fa50861c640 1 -- 172.21.15.161:0/3484753125 wait complete. 2024-03-28T23:30:40.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.119+0000 7fa50861c640 1 Processor -- start 2024-03-28T23:30:40.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.119+0000 7fa50861c640 1 -- start start 2024-03-28T23:30:40.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.120+0000 7fa50861c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa5000c8a80 0x7fa50014bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:40.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.120+0000 7fa50861c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5000c9420 0x7fa5001e0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:40.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.120+0000 7fa50861c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa500152bc0 0x7fa5001e0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:40.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.120+0000 7fa50861c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa500152fa0 con 0x7fa5000c9420 2024-03-28T23:30:40.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.120+0000 7fa50861c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa5001590c0 con 0x7fa5000c8a80 2024-03-28T23:30:40.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.120+0000 7fa50861c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa500146660 con 0x7fa500152bc0 2024-03-28T23:30:40.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.120+0000 7fa505bb7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5000c9420 0x7fa5001e0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:40.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.120+0000 7fa505bb7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5000c9420 0x7fa5001e0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49350/0 (socket says 172.21.15.161:49350) 2024-03-28T23:30:40.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.120+0000 7fa505bb7640 1 -- 172.21.15.161:0/3642322855 learned_addr learned my addr 172.21.15.161:0/3642322855 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:40.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.120+0000 7fa506bb9640 1 --2- 172.21.15.161:0/3642322855 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa500152bc0 0x7fa5001e0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:40.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.120+0000 7fa5063b8640 1 --2- 172.21.15.161:0/3642322855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa5000c8a80 0x7fa50014bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:40.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.121+0000 7fa505bb7640 1 -- 172.21.15.161:0/3642322855 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa500152bc0 msgr2=0x7fa5001e0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:40.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.121+0000 7fa505bb7640 1 --2- 172.21.15.161:0/3642322855 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa500152bc0 0x7fa5001e0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:40.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.121+0000 7fa505bb7640 1 -- 172.21.15.161:0/3642322855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa5000c8a80 msgr2=0x7fa50014bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:40.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.121+0000 7fa505bb7640 1 --2- 172.21.15.161:0/3642322855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa5000c8a80 0x7fa50014bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:40.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.121+0000 7fa505bb7640 1 -- 172.21.15.161:0/3642322855 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5001461b0 con 0x7fa5000c9420 2024-03-28T23:30:40.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.121+0000 7fa505bb7640 1 --2- 172.21.15.161:0/3642322855 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5000c9420 0x7fa5001e0020 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa4f0000c00 tx=0x7fa4f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:40.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.121+0000 7fa4f6ffd640 1 -- 172.21.15.161:0/3642322855 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f0048070 con 0x7fa5000c9420 2024-03-28T23:30:40.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.121+0000 7fa4f6ffd640 1 -- 172.21.15.161:0/3642322855 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4f0030660 con 0x7fa5000c9420 2024-03-28T23:30:40.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.121+0000 7fa50861c640 1 -- 172.21.15.161:0/3642322855 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa50014c4a0 con 0x7fa5000c9420 2024-03-28T23:30:40.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.121+0000 7fa4f6ffd640 1 -- 172.21.15.161:0/3642322855 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f0034040 con 0x7fa5000c9420 2024-03-28T23:30:40.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.121+0000 7fa50861c640 1 -- 172.21.15.161:0/3642322855 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5001515f0 con 0x7fa5000c9420 2024-03-28T23:30:40.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.122+0000 7fa50861c640 1 -- 172.21.15.161:0/3642322855 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa50004f4b0 con 0x7fa5000c9420 2024-03-28T23:30:40.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.122+0000 7fa4f6ffd640 1 -- 172.21.15.161:0/3642322855 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa4f004b070 con 0x7fa5000c9420 2024-03-28T23:30:40.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.123+0000 7fa4f6ffd640 1 --2- 172.21.15.161:0/3642322855 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa4e406b4d0 0x7fa4e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:40.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.123+0000 7fa5063b8640 1 --2- 172.21.15.161:0/3642322855 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa4e406b4d0 0x7fa4e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:40.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.123+0000 7fa4f6ffd640 1 -- 172.21.15.161:0/3642322855 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fa4f00ac720 con 0x7fa5000c9420 2024-03-28T23:30:40.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.123+0000 7fa5063b8640 1 --2- 172.21.15.161:0/3642322855 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa4e406b4d0 0x7fa4e406d990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc0079a0 tx=0x7fa4fc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:40.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.127+0000 7fa4f6ffd640 1 -- 172.21.15.161:0/3642322855 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa4f007ac70 con 0x7fa5000c9420 2024-03-28T23:30:40.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.307+0000 7fa50861c640 1 -- 172.21.15.161:0/3642322855 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa50014f8f0 con 0x7fa5000c9420 2024-03-28T23:30:40.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.307+0000 7fa4f6ffd640 1 -- 172.21.15.161:0/3642322855 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fa4f007ed10 con 0x7fa5000c9420 2024-03-28T23:30:40.309 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:30:40.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.309+0000 7fa50861c640 1 -- 172.21.15.161:0/3642322855 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa4e406b4d0 msgr2=0x7fa4e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:40.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.309+0000 7fa50861c640 1 --2- 172.21.15.161:0/3642322855 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa4e406b4d0 0x7fa4e406d990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc0079a0 tx=0x7fa4fc008040 comp rx=0 tx=0).stop 2024-03-28T23:30:40.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.309+0000 7fa50861c640 1 -- 172.21.15.161:0/3642322855 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5000c9420 msgr2=0x7fa5001e0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:40.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.309+0000 7fa50861c640 1 --2- 172.21.15.161:0/3642322855 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5000c9420 0x7fa5001e0020 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa4f0000c00 tx=0x7fa4f0004730 comp rx=0 tx=0).stop 2024-03-28T23:30:40.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.309+0000 7fa50861c640 1 -- 172.21.15.161:0/3642322855 shutdown_connections 2024-03-28T23:30:40.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.309+0000 7fa50861c640 1 --2- 172.21.15.161:0/3642322855 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa4e406b4d0 0x7fa4e406d990 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:40.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.309+0000 7fa50861c640 1 --2- 172.21.15.161:0/3642322855 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5000c9420 0x7fa5001e0020 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:40.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.309+0000 7fa50861c640 1 --2- 172.21.15.161:0/3642322855 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa500152bc0 0x7fa5001e0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:40.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.309+0000 7fa50861c640 1 --2- 172.21.15.161:0/3642322855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa5000c8a80 0x7fa50014bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:40.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.309+0000 7fa50861c640 1 -- 172.21.15.161:0/3642322855 >> 172.21.15.161:0/3642322855 conn(0x7fa5000caa80 msgr2=0x7fa50014d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:40.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.309+0000 7fa50861c640 1 -- 172.21.15.161:0/3642322855 shutdown_connections 2024-03-28T23:30:40.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.309+0000 7fa50861c640 1 -- 172.21.15.161:0/3642322855 wait complete. 2024-03-28T23:30:40.322 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T23:30:40.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.425+0000 7f1b6c22e640 1 Processor -- start 2024-03-28T23:30:40.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.425+0000 7f1b6c22e640 1 -- start start 2024-03-28T23:30:40.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.426+0000 7f1b6c22e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b641528b0 0x7f1b640c8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:40.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.426+0000 7f1b6c22e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b640c7ed0 con 0x7f1b640c5ab0 2024-03-28T23:30:40.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.426+0000 7f1b6c22e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b64152c90 con 0x7f1b640c8e60 2024-03-28T23:30:40.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.426+0000 7f1b6c22e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1b64155f90 con 0x7f1b641528b0 2024-03-28T23:30:40.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.426+0000 7f1b6a7cb640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f1b640c5ab0 0x7f1b6406c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60238/0 (socket says 172.21.15.161:60238) 2024-03-28T23:30:40.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.426+0000 7f1b6a7cb640 1 -- 172.21.15.161:0/2168641572 learned_addr learned my addr 172.21.15.161:0/2168641572 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:40.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.427+0000 7f1b69fca640 1 --2- 172.21.15.161:0/2168641572 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b641528b0 0x7f1b640c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:40.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.427+0000 7f1b68fc8640 1 -- 172.21.15.161:0/2168641572 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 278124381 0 0) 0x7f1b640c7ed0 con 0x7f1b640c5ab0 2024-03-28T23:30:40.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.427+0000 7f1b68fc8640 1 -- 172.21.15.161:0/2168641572 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b50003520 con 0x7f1b640c5ab0 2024-03-28T23:30:40.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.427+0000 7f1b68fc8640 1 -- 172.21.15.161:0/2168641572 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3319806571 0 0) 0x7f1b64152c90 con 0x7f1b640c8e60 2024-03-28T23:30:40.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.427+0000 7f1b69fca640 1 -- 172.21.15.161:0/2168641572 >> v1:172.21.15.170:6789/0 conn(0x7f1b640c8e60 legacy=0x7f1b640c53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:40.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.427+0000 7f1b69fca640 1 -- 172.21.15.161:0/2168641572 >> v1:172.21.15.161:6789/0 conn(0x7f1b640c5ab0 legacy=0x7f1b6406c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:40.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.427+0000 7f1b69fca640 1 -- 172.21.15.161:0/2168641572 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b64155380 con 0x7f1b641528b0 2024-03-28T23:30:40.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.428+0000 7f1b69fca640 1 --2- 172.21.15.161:0/2168641572 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b641528b0 0x7f1b640c8920 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1b600097d0 tx=0x7f1b6002b600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37400ce2bf1c9e97 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:40.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.428+0000 7f1b68fc8640 1 -- 172.21.15.161:0/2168641572 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b60037070 con 0x7f1b641528b0 2024-03-28T23:30:40.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.428+0000 7f1b68fc8640 1 -- 172.21.15.161:0/2168641572 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b60030510 con 0x7f1b641528b0 2024-03-28T23:30:40.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.428+0000 7f1b68fc8640 1 -- 172.21.15.161:0/2168641572 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c0019f0 con 0x7f1b641528b0 2024-03-28T23:30:40.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.429+0000 7f1b6c22e640 1 -- 172.21.15.161:0/2168641572 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b641528b0 msgr2=0x7f1b640c8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:40.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.429+0000 7f1b6c22e640 1 --2- 172.21.15.161:0/2168641572 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b641528b0 0x7f1b640c8920 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1b600097d0 tx=0x7f1b6002b600 comp rx=0 tx=0).stop 2024-03-28T23:30:40.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.429+0000 7f1b6c22e640 1 -- 172.21.15.161:0/2168641572 shutdown_connections 2024-03-28T23:30:40.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.429+0000 7f1b6c22e640 1 --2- 172.21.15.161:0/2168641572 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b641528b0 0x7f1b640c8920 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:40.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.429+0000 7f1b6c22e640 1 -- 172.21.15.161:0/2168641572 >> 172.21.15.161:0/2168641572 conn(0x7f1b640ac5b0 msgr2=0x7f1b640ac9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:40.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.429+0000 7f1b6c22e640 1 -- 172.21.15.161:0/2168641572 shutdown_connections 2024-03-28T23:30:40.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.429+0000 7f1b6c22e640 1 -- 172.21.15.161:0/2168641572 wait complete. 2024-03-28T23:30:40.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.431+0000 7f1b6c22e640 1 Processor -- start 2024-03-28T23:30:40.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.431+0000 7f1b6c22e640 1 -- start start 2024-03-28T23:30:40.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.431+0000 7f1b6c22e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b640c8e60 0x7f1b641ee110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:40.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.431+0000 7f1b6c22e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b641528b0 0x7f1b64151110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:40.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.431+0000 7f1b6c22e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b6414d350 0x7f1b640c3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:40.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.431+0000 7f1b6c22e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1b64155c20 con 0x7f1b641528b0 2024-03-28T23:30:40.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.431+0000 7f1b6c22e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1b64155940 con 0x7f1b640c8e60 2024-03-28T23:30:40.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.431+0000 7f1b6c22e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1b6406c850 con 0x7f1b6414d350 2024-03-28T23:30:40.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.431+0000 7f1b697c9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b641528b0 0x7f1b64151110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:40.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.432+0000 7f1b697c9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b641528b0 0x7f1b64151110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49356/0 (socket says 172.21.15.161:49356) 2024-03-28T23:30:40.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.432+0000 7f1b697c9640 1 -- 172.21.15.161:0/339579096 learned_addr learned my addr 172.21.15.161:0/339579096 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:40.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.432+0000 7f1b69fca640 1 --2- 172.21.15.161:0/339579096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b640c8e60 0x7f1b641ee110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:40.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.432+0000 7f1b6a7cb640 1 --2- 172.21.15.161:0/339579096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b6414d350 0x7f1b640c3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:40.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.432+0000 7f1b697c9640 1 -- 172.21.15.161:0/339579096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b6414d350 msgr2=0x7f1b640c3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:40.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.432+0000 7f1b697c9640 1 --2- 172.21.15.161:0/339579096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b6414d350 0x7f1b640c3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:40.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.432+0000 7f1b697c9640 1 -- 172.21.15.161:0/339579096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b640c8e60 msgr2=0x7f1b641ee110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:40.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.432+0000 7f1b697c9640 1 --2- 172.21.15.161:0/339579096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b640c8e60 0x7f1b641ee110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:40.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.432+0000 7f1b697c9640 1 -- 172.21.15.161:0/339579096 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b6414cc00 con 0x7f1b641528b0 2024-03-28T23:30:40.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.432+0000 7f1b697c9640 1 --2- 172.21.15.161:0/339579096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b641528b0 0x7f1b64151110 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1b5400c4e0 tx=0x7f1b5400c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:40.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.432+0000 7f1b5a7fc640 1 -- 172.21.15.161:0/339579096 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b54017070 con 0x7f1b641528b0 2024-03-28T23:30:40.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.432+0000 7f1b5a7fc640 1 -- 172.21.15.161:0/339579096 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b540035f0 con 0x7f1b641528b0 2024-03-28T23:30:40.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.432+0000 7f1b6c22e640 1 -- 172.21.15.161:0/339579096 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b6414f770 con 0x7f1b641528b0 2024-03-28T23:30:40.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.432+0000 7f1b5a7fc640 1 -- 172.21.15.161:0/339579096 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b540039e0 con 0x7f1b641528b0 2024-03-28T23:30:40.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.432+0000 7f1b6c22e640 1 -- 172.21.15.161:0/339579096 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b640c3210 con 0x7f1b641528b0 2024-03-28T23:30:40.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.433+0000 7f1b6c22e640 1 -- 172.21.15.161:0/339579096 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b6404f4b0 con 0x7f1b641528b0 2024-03-28T23:30:40.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.434+0000 7f1b5a7fc640 1 -- 172.21.15.161:0/339579096 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1b5401a070 con 0x7f1b641528b0 2024-03-28T23:30:40.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.435+0000 7f1b5a7fc640 1 --2- 172.21.15.161:0/339579096 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1b4806b4d0 0x7f1b4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:40.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.435+0000 7f1b5a7fc640 1 -- 172.21.15.161:0/339579096 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f1b54089750 con 0x7f1b641528b0 2024-03-28T23:30:40.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.435+0000 7f1b69fca640 1 --2- 172.21.15.161:0/339579096 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1b4806b4d0 0x7f1b4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:40.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.435+0000 7f1b69fca640 1 --2- 172.21.15.161:0/339579096 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1b4806b4d0 0x7f1b4806d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1b60034000 tx=0x7f1b60008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:40.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.443+0000 7f1b5a7fc640 1 -- 172.21.15.161:0/339579096 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1b6414cc00 con 0x7f1b641528b0 2024-03-28T23:30:40.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.694+0000 7f1b6c22e640 1 -- 172.21.15.161:0/339579096 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1b640c80d0 con 0x7f1b641528b0 2024-03-28T23:30:40.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.694+0000 7f1b5a7fc640 1 -- 172.21.15.161:0/339579096 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f1b54057c10 con 0x7f1b641528b0 2024-03-28T23:30:40.695 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-28T23:30:40.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.697+0000 7f1b6c22e640 1 -- 172.21.15.161:0/339579096 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1b4806b4d0 msgr2=0x7f1b4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:40.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.697+0000 7f1b6c22e640 1 --2- 172.21.15.161:0/339579096 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1b4806b4d0 0x7f1b4806d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1b60034000 tx=0x7f1b60008030 comp rx=0 tx=0).stop 2024-03-28T23:30:40.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.697+0000 7f1b6c22e640 1 -- 172.21.15.161:0/339579096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b641528b0 msgr2=0x7f1b64151110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:40.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.697+0000 7f1b6c22e640 1 --2- 172.21.15.161:0/339579096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b641528b0 0x7f1b64151110 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1b5400c4e0 tx=0x7f1b5400c9a0 comp rx=0 tx=0).stop 2024-03-28T23:30:40.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.697+0000 7f1b6c22e640 1 -- 172.21.15.161:0/339579096 shutdown_connections 2024-03-28T23:30:40.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.697+0000 7f1b6c22e640 1 --2- 172.21.15.161:0/339579096 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1b4806b4d0 0x7f1b4806d990 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:40.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.697+0000 7f1b6c22e640 1 --2- 172.21.15.161:0/339579096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b641528b0 0x7f1b64151110 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:40.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.697+0000 7f1b6c22e640 1 --2- 172.21.15.161:0/339579096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b6414d350 0x7f1b640c3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:40.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.697+0000 7f1b6c22e640 1 --2- 172.21.15.161:0/339579096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b640c8e60 0x7f1b641ee110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:40.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.697+0000 7f1b6c22e640 1 -- 172.21.15.161:0/339579096 >> 172.21.15.161:0/339579096 conn(0x7f1b640ac5b0 msgr2=0x7f1b640abbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:40.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.698+0000 7f1b6c22e640 1 -- 172.21.15.161:0/339579096 shutdown_connections 2024-03-28T23:30:40.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.698+0000 7f1b6c22e640 1 -- 172.21.15.161:0/339579096 wait complete. 2024-03-28T23:30:40.710 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:30:40.710 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:30:40.729 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:30:40.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.834+0000 7fd5ab713640 1 Processor -- start 2024-03-28T23:30:40.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.834+0000 7fd5ab713640 1 -- start start 2024-03-28T23:30:40.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.835+0000 7fd5ab713640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd5a406ce80 0x7fd5a40aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:40.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.835+0000 7fd5ab713640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5a40a9680 0x7fd5a40aab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:40.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.835+0000 7fd5ab713640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5a414c250 con 0x7fd5a414be70 2024-03-28T23:30:40.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.835+0000 7fd5ab713640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd5a41529c0 con 0x7fd5a406ce80 2024-03-28T23:30:40.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.835+0000 7fd5ab713640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd5a40a9dc0 con 0x7fd5a40a9680 2024-03-28T23:30:40.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.835+0000 7fd5a9cb0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5a40a9680 0x7fd5a40aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:40.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.835+0000 7fd5a8cae640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd5a406ce80 0x7fd5a40aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:40.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.835+0000 7fd5a9cb0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5a40a9680 0x7fd5a40aab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:33098/0 (socket says 172.21.15.161:33098) 2024-03-28T23:30:40.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.835+0000 7fd5a9cb0640 1 -- 172.21.15.161:0/1382871942 learned_addr learned my addr 172.21.15.161:0/1382871942 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:40.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.835+0000 7fd5a94af640 1 --1- 172.21.15.161:0/1382871942 >> v1:172.21.15.170:6789/0 conn(0x7fd5a414be70 0x7fd5a40ab1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:60366/0 (socket says 172.21.15.161:60366) 2024-03-28T23:30:40.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.836+0000 7fd5a8cae640 1 -- 172.21.15.161:0/1382871942 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5a40a9680 msgr2=0x7fd5a40aab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:40.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.836+0000 7fd5a8cae640 1 --2- 172.21.15.161:0/1382871942 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5a40a9680 0x7fd5a40aab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:40.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.836+0000 7fd5a8cae640 1 -- 172.21.15.161:0/1382871942 >> v1:172.21.15.170:6789/0 conn(0x7fd5a414be70 legacy=0x7fd5a40ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:40.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.836+0000 7fd5a8cae640 1 -- 172.21.15.161:0/1382871942 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a40aa0a0 con 0x7fd5a406ce80 2024-03-28T23:30:40.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.836+0000 7fd5a8cae640 1 --2- 172.21.15.161:0/1382871942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5a406ce80 0x7fd5a40aa5c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd5940097a0 tx=0x7fd59402b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a06a3400b759ee8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:40.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.837+0000 7fd59bfff640 1 -- 172.21.15.161:0/1382871942 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd594038070 con 0x7fd5a406ce80 2024-03-28T23:30:40.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.837+0000 7fd59bfff640 1 -- 172.21.15.161:0/1382871942 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd594031700 con 0x7fd5a406ce80 2024-03-28T23:30:40.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.837+0000 7fd59bfff640 1 -- 172.21.15.161:0/1382871942 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd594031ad0 con 0x7fd5a406ce80 2024-03-28T23:30:40.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.837+0000 7fd5ab713640 1 -- 172.21.15.161:0/1382871942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5a406ce80 msgr2=0x7fd5a40aa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:40.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.837+0000 7fd5ab713640 1 --2- 172.21.15.161:0/1382871942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5a406ce80 0x7fd5a40aa5c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd5940097a0 tx=0x7fd59402b7a0 comp rx=0 tx=0).stop 2024-03-28T23:30:40.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.837+0000 7fd5ab713640 1 -- 172.21.15.161:0/1382871942 shutdown_connections 2024-03-28T23:30:40.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.837+0000 7fd5ab713640 1 --2- 172.21.15.161:0/1382871942 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5a40a9680 0x7fd5a40aab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:40.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.837+0000 7fd5ab713640 1 --2- 172.21.15.161:0/1382871942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5a406ce80 0x7fd5a40aa5c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:40.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.837+0000 7fd5ab713640 1 -- 172.21.15.161:0/1382871942 >> 172.21.15.161:0/1382871942 conn(0x7fd5a40c6670 msgr2=0x7fd5a40c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:40.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.837+0000 7fd5ab713640 1 -- 172.21.15.161:0/1382871942 shutdown_connections 2024-03-28T23:30:40.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.837+0000 7fd5ab713640 1 -- 172.21.15.161:0/1382871942 wait complete. 2024-03-28T23:30:40.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.838+0000 7fd5ab713640 1 Processor -- start 2024-03-28T23:30:40.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.838+0000 7fd5ab713640 1 -- start start 2024-03-28T23:30:40.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.839+0000 7fd5ab713640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5a406ce80 0x7fd5a4151650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:40.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.839+0000 7fd5ab713640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5a40a9680 0x7fd5a41e6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:40.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.839+0000 7fd5ab713640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5a414be70 0x7fd5a41e7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:40.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.839+0000 7fd5ab713640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a40aba30 con 0x7fd5a414be70 2024-03-28T23:30:40.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.839+0000 7fd5ab713640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a40ab8b0 con 0x7fd5a406ce80 2024-03-28T23:30:40.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.839+0000 7fd5ab713640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd5a406b150 con 0x7fd5a40a9680 2024-03-28T23:30:40.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.839+0000 7fd5a9cb0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5a414be70 0x7fd5a41e7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:40.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.839+0000 7fd5a9cb0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5a414be70 0x7fd5a41e7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51826/0 (socket says 172.21.15.161:51826) 2024-03-28T23:30:40.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.839+0000 7fd5a9cb0640 1 -- 172.21.15.161:0/3674144366 learned_addr learned my addr 172.21.15.161:0/3674144366 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:40.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.839+0000 7fd5a8cae640 1 --2- 172.21.15.161:0/3674144366 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5a40a9680 0x7fd5a41e6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:40.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.839+0000 7fd5a94af640 1 --2- 172.21.15.161:0/3674144366 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5a406ce80 0x7fd5a4151650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:40.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.840+0000 7fd5a9cb0640 1 -- 172.21.15.161:0/3674144366 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5a40a9680 msgr2=0x7fd5a41e6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:40.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.840+0000 7fd5a9cb0640 1 --2- 172.21.15.161:0/3674144366 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5a40a9680 0x7fd5a41e6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:40.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.840+0000 7fd5a9cb0640 1 -- 172.21.15.161:0/3674144366 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5a406ce80 msgr2=0x7fd5a4151650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:40.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.840+0000 7fd5a9cb0640 1 --2- 172.21.15.161:0/3674144366 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5a406ce80 0x7fd5a4151650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:40.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.840+0000 7fd5a9cb0640 1 -- 172.21.15.161:0/3674144366 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a41ea740 con 0x7fd5a414be70 2024-03-28T23:30:40.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.840+0000 7fd5a9cb0640 1 --2- 172.21.15.161:0/3674144366 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5a414be70 0x7fd5a41e7030 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd59c00efc0 tx=0x7fd59c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:40.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.840+0000 7fd599ffb640 1 -- 172.21.15.161:0/3674144366 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c00cd20 con 0x7fd5a414be70 2024-03-28T23:30:40.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.840+0000 7fd599ffb640 1 -- 172.21.15.161:0/3674144366 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd59c003990 con 0x7fd5a414be70 2024-03-28T23:30:40.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.840+0000 7fd599ffb640 1 -- 172.21.15.161:0/3674144366 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c003d80 con 0x7fd5a414be70 2024-03-28T23:30:40.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.840+0000 7fd5ab713640 1 -- 172.21.15.161:0/3674144366 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5a41eaa20 con 0x7fd5a414be70 2024-03-28T23:30:40.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.840+0000 7fd5ab713640 1 -- 172.21.15.161:0/3674144366 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5a41dd870 con 0x7fd5a414be70 2024-03-28T23:30:40.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.841+0000 7fd5ab713640 1 -- 172.21.15.161:0/3674144366 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5a404f4b0 con 0x7fd5a414be70 2024-03-28T23:30:40.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.842+0000 7fd599ffb640 1 -- 172.21.15.161:0/3674144366 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd59c012050 con 0x7fd5a414be70 2024-03-28T23:30:40.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.842+0000 7fd599ffb640 1 --2- 172.21.15.161:0/3674144366 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd58806b4d0 0x7fd58806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:40.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.842+0000 7fd5a94af640 1 --2- 172.21.15.161:0/3674144366 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd58806b4d0 0x7fd58806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:40.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.842+0000 7fd599ffb640 1 -- 172.21.15.161:0/3674144366 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fd59c08aa00 con 0x7fd5a414be70 2024-03-28T23:30:40.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.843+0000 7fd5a94af640 1 --2- 172.21.15.161:0/3674144366 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd58806b4d0 0x7fd58806d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0007fd0 tx=0x7fd5a0007c40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:40.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:40.846+0000 7fd599ffb640 1 -- 172.21.15.161:0/3674144366 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd59c010040 con 0x7fd5a414be70 2024-03-28T23:30:41.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.028+0000 7fd5ab713640 1 -- 172.21.15.161:0/3674144366 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd5a40c0ff0 con 0x7fd5a414be70 2024-03-28T23:30:41.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.030+0000 7fd599ffb640 1 -- 172.21.15.161:0/3674144366 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18937 (secure 0 0 0) 0x7fd59c059510 con 0x7fd5a414be70 2024-03-28T23:30:41.031 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:41.031 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":26,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:30:35.148429+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:30:41.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.032+0000 7fd5ab713640 1 -- 172.21.15.161:0/3674144366 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd58806b4d0 msgr2=0x7fd58806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:41.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.032+0000 7fd5ab713640 1 --2- 172.21.15.161:0/3674144366 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd58806b4d0 0x7fd58806d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0007fd0 tx=0x7fd5a0007c40 comp rx=0 tx=0).stop 2024-03-28T23:30:41.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.032+0000 7fd5ab713640 1 -- 172.21.15.161:0/3674144366 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5a414be70 msgr2=0x7fd5a41e7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:41.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.032+0000 7fd5ab713640 1 --2- 172.21.15.161:0/3674144366 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5a414be70 0x7fd5a41e7030 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd59c00efc0 tx=0x7fd59c00c490 comp rx=0 tx=0).stop 2024-03-28T23:30:41.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.032+0000 7fd5ab713640 1 -- 172.21.15.161:0/3674144366 shutdown_connections 2024-03-28T23:30:41.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.032+0000 7fd5ab713640 1 --2- 172.21.15.161:0/3674144366 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd58806b4d0 0x7fd58806d990 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.032+0000 7fd5ab713640 1 --2- 172.21.15.161:0/3674144366 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5a414be70 0x7fd5a41e7030 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.032+0000 7fd5ab713640 1 --2- 172.21.15.161:0/3674144366 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5a40a9680 0x7fd5a41e6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.032+0000 7fd5ab713640 1 --2- 172.21.15.161:0/3674144366 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5a406ce80 0x7fd5a4151650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.032+0000 7fd5ab713640 1 -- 172.21.15.161:0/3674144366 >> 172.21.15.161:0/3674144366 conn(0x7fd5a40c6670 msgr2=0x7fd5a414a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:41.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.032+0000 7fd5ab713640 1 -- 172.21.15.161:0/3674144366 shutdown_connections 2024-03-28T23:30:41.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.032+0000 7fd5ab713640 1 -- 172.21.15.161:0/3674144366 wait complete. 2024-03-28T23:30:41.044 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:29:44.021375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:29:44.872102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T23:29:48.591063+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T23:30:41.044 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:30:41.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.148+0000 7f7af8bc9640 1 Processor -- start 2024-03-28T23:30:41.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.148+0000 7f7af8bc9640 1 -- start start 2024-03-28T23:30:41.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.148+0000 7f7af8bc9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7af41529d0 0x7f7af406bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:41.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.148+0000 7f7af8bc9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7af40c6750 0x7f7af40c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:41.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.149+0000 7f7af8bc9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7af4155c30 0x7f7af40c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:41.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.149+0000 7f7af8bc9640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f7af4152db0 con 0x7f7af4155c30 2024-03-28T23:30:41.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.149+0000 7f7af8bc9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f7af40c7210 con 0x7f7af40c6750 2024-03-28T23:30:41.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.149+0000 7f7af8bc9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7af40c6c50 con 0x7f7af41529d0 2024-03-28T23:30:41.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.149+0000 7f7af2d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7af4155c30 0x7f7af40c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:41.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.149+0000 7f7af2d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7af4155c30 0x7f7af40c54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51832/0 (socket says 172.21.15.161:51832) 2024-03-28T23:30:41.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.149+0000 7f7af2d9d640 1 -- 172.21.15.161:0/1593614894 learned_addr learned my addr 172.21.15.161:0/1593614894 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:41.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.149+0000 7f7af1d9b640 1 --2- 172.21.15.161:0/1593614894 >> v2:172.21.15.170:3300/0 conn(0x7f7af40c6750 0x7f7af40c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:41.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.149+0000 7f7af259c640 1 --2- 172.21.15.161:0/1593614894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7af41529d0 0x7f7af406bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:41.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.149+0000 7f7af2d9d640 1 -- 172.21.15.161:0/1593614894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7af41529d0 msgr2=0x7f7af406bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:41.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.149+0000 7f7af2d9d640 1 --2- 172.21.15.161:0/1593614894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7af41529d0 0x7f7af406bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.149+0000 7f7af2d9d640 1 -- 172.21.15.161:0/1593614894 >> v2:172.21.15.170:3300/0 conn(0x7f7af40c6750 msgr2=0x7f7af40c88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:41.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.149+0000 7f7af2d9d640 1 --2- 172.21.15.161:0/1593614894 >> v2:172.21.15.170:3300/0 conn(0x7f7af40c6750 0x7f7af40c88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.149+0000 7f7af2d9d640 1 -- 172.21.15.161:0/1593614894 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af40c6f30 con 0x7f7af4155c30 2024-03-28T23:30:41.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.150+0000 7f7af2d9d640 1 --2- 172.21.15.161:0/1593614894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7af4155c30 0x7f7af40c54f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7af4068a40 tx=0x7f7aec02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13127774f677bc09 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:41.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.150+0000 7f7af159a640 1 -- 172.21.15.161:0/1593614894 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aec02ca70 con 0x7f7af4155c30 2024-03-28T23:30:41.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.150+0000 7f7af159a640 1 -- 172.21.15.161:0/1593614894 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7aec02cc10 con 0x7f7af4155c30 2024-03-28T23:30:41.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.150+0000 7f7af159a640 1 -- 172.21.15.161:0/1593614894 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aec0343f0 con 0x7f7af4155c30 2024-03-28T23:30:41.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.151+0000 7f7af8bc9640 1 -- 172.21.15.161:0/1593614894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7af4155c30 msgr2=0x7f7af40c54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:41.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.151+0000 7f7af8bc9640 1 --2- 172.21.15.161:0/1593614894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7af4155c30 0x7f7af40c54f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7af4068a40 tx=0x7f7aec02bed0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.151+0000 7f7af8bc9640 1 -- 172.21.15.161:0/1593614894 shutdown_connections 2024-03-28T23:30:41.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.151+0000 7f7af8bc9640 1 --2- 172.21.15.161:0/1593614894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7af4155c30 0x7f7af40c54f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.151+0000 7f7af8bc9640 1 --2- 172.21.15.161:0/1593614894 >> v2:172.21.15.170:3300/0 conn(0x7f7af40c6750 0x7f7af40c88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.151+0000 7f7af8bc9640 1 --2- 172.21.15.161:0/1593614894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7af41529d0 0x7f7af406bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.151+0000 7f7af8bc9640 1 -- 172.21.15.161:0/1593614894 >> 172.21.15.161:0/1593614894 conn(0x7f7af40c7430 msgr2=0x7f7af40c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:41.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.151+0000 7f7af8bc9640 1 -- 172.21.15.161:0/1593614894 shutdown_connections 2024-03-28T23:30:41.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.151+0000 7f7af8bc9640 1 -- 172.21.15.161:0/1593614894 wait complete. 2024-03-28T23:30:41.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.152+0000 7f7af8bc9640 1 Processor -- start 2024-03-28T23:30:41.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.152+0000 7f7af8bc9640 1 -- start start 2024-03-28T23:30:41.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.153+0000 7f7af8bc9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7af40c6750 0x7f7af40b0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:41.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.153+0000 7f7af8bc9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7af41529d0 0x7f7af41eb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:41.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.153+0000 7f7af8bc9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7af4155c30 0x7f7af414a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:41.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.153+0000 7f7af8bc9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7af40aa160 con 0x7f7af41529d0 2024-03-28T23:30:41.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.153+0000 7f7af8bc9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7af4155710 con 0x7f7af4155c30 2024-03-28T23:30:41.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.153+0000 7f7af8bc9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7af41560e0 con 0x7f7af40c6750 2024-03-28T23:30:41.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.153+0000 7f7af1d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7af41529d0 0x7f7af41eb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:41.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.153+0000 7f7af1d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7af41529d0 0x7f7af41eb840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51838/0 (socket says 172.21.15.161:51838) 2024-03-28T23:30:41.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.153+0000 7f7af1d9b640 1 -- 172.21.15.161:0/1985269351 learned_addr learned my addr 172.21.15.161:0/1985269351 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:41.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.153+0000 7f7af259c640 1 --2- 172.21.15.161:0/1985269351 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7af40c6750 0x7f7af40b0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:41.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.153+0000 7f7af2d9d640 1 --2- 172.21.15.161:0/1985269351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7af4155c30 0x7f7af414a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:41.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.153+0000 7f7af1d9b640 1 -- 172.21.15.161:0/1985269351 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7af40c6750 msgr2=0x7f7af40b0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:41.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.153+0000 7f7af1d9b640 1 --2- 172.21.15.161:0/1985269351 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7af40c6750 0x7f7af40b0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.153+0000 7f7af1d9b640 1 -- 172.21.15.161:0/1985269351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7af4155c30 msgr2=0x7f7af414a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:41.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.153+0000 7f7af1d9b640 1 --2- 172.21.15.161:0/1985269351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7af4155c30 0x7f7af414a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.153+0000 7f7af1d9b640 1 -- 172.21.15.161:0/1985269351 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af414a490 con 0x7f7af41529d0 2024-03-28T23:30:41.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.153+0000 7f7af1d9b640 1 --2- 172.21.15.161:0/1985269351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7af41529d0 0x7f7af41eb840 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7ae800e8c0 tx=0x7f7ae800ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:41.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.153+0000 7f7ae2ffd640 1 -- 172.21.15.161:0/1985269351 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae8017070 con 0x7f7af41529d0 2024-03-28T23:30:41.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.153+0000 7f7ae2ffd640 1 -- 172.21.15.161:0/1985269351 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ae800c940 con 0x7f7af41529d0 2024-03-28T23:30:41.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.153+0000 7f7af8bc9640 1 -- 172.21.15.161:0/1985269351 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7af41592d0 con 0x7f7af41529d0 2024-03-28T23:30:41.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.154+0000 7f7ae2ffd640 1 -- 172.21.15.161:0/1985269351 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae800cd10 con 0x7f7af41529d0 2024-03-28T23:30:41.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.154+0000 7f7af8bc9640 1 -- 172.21.15.161:0/1985269351 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7af4148f70 con 0x7f7af41529d0 2024-03-28T23:30:41.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.154+0000 7f7af8bc9640 1 -- 172.21.15.161:0/1985269351 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7af404f4b0 con 0x7f7af41529d0 2024-03-28T23:30:41.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.155+0000 7f7ae2ffd640 1 -- 172.21.15.161:0/1985269351 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7ae801a070 con 0x7f7af41529d0 2024-03-28T23:30:41.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.155+0000 7f7ae2ffd640 1 --2- 172.21.15.161:0/1985269351 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7ad406b4d0 0x7f7ad406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:41.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.156+0000 7f7af259c640 1 --2- 172.21.15.161:0/1985269351 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7ad406b4d0 0x7f7ad406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:41.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.156+0000 7f7ae2ffd640 1 -- 172.21.15.161:0/1985269351 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f7ae808a810 con 0x7f7af41529d0 2024-03-28T23:30:41.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.156+0000 7f7af259c640 1 --2- 172.21.15.161:0/1985269351 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7ad406b4d0 0x7f7ad406d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4009b40 tx=0x7f7ae4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:41.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.160+0000 7f7ae2ffd640 1 -- 172.21.15.161:0/1985269351 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7ae8058cd0 con 0x7f7af41529d0 2024-03-28T23:30:41.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.339+0000 7f7af8bc9640 1 -- 172.21.15.161:0/1985269351 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7af414be00 con 0x7f7af41529d0 2024-03-28T23:30:41.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.340+0000 7f7ae2ffd640 1 -- 172.21.15.161:0/1985269351 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f7ae805cd70 con 0x7f7af41529d0 2024-03-28T23:30:41.341 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:30:41.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.342+0000 7f7af8bc9640 1 -- 172.21.15.161:0/1985269351 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7ad406b4d0 msgr2=0x7f7ad406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:41.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.342+0000 7f7af8bc9640 1 --2- 172.21.15.161:0/1985269351 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7ad406b4d0 0x7f7ad406d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4009b40 tx=0x7f7ae4009190 comp rx=0 tx=0).stop 2024-03-28T23:30:41.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.342+0000 7f7af8bc9640 1 -- 172.21.15.161:0/1985269351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7af41529d0 msgr2=0x7f7af41eb840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:41.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.342+0000 7f7af8bc9640 1 --2- 172.21.15.161:0/1985269351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7af41529d0 0x7f7af41eb840 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7ae800e8c0 tx=0x7f7ae800ed80 comp rx=0 tx=0).stop 2024-03-28T23:30:41.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.342+0000 7f7af8bc9640 1 -- 172.21.15.161:0/1985269351 shutdown_connections 2024-03-28T23:30:41.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.342+0000 7f7af8bc9640 1 --2- 172.21.15.161:0/1985269351 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7ad406b4d0 0x7f7ad406d990 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.342+0000 7f7af8bc9640 1 --2- 172.21.15.161:0/1985269351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7af41529d0 0x7f7af41eb840 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.342+0000 7f7af8bc9640 1 --2- 172.21.15.161:0/1985269351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7af4155c30 0x7f7af414a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.342+0000 7f7af8bc9640 1 --2- 172.21.15.161:0/1985269351 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7af40c6750 0x7f7af40b0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.342+0000 7f7af8bc9640 1 -- 172.21.15.161:0/1985269351 >> 172.21.15.161:0/1985269351 conn(0x7f7af40c7430 msgr2=0x7f7af40aa2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:41.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.342+0000 7f7af8bc9640 1 -- 172.21.15.161:0/1985269351 shutdown_connections 2024-03-28T23:30:41.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.342+0000 7f7af8bc9640 1 -- 172.21.15.161:0/1985269351 wait complete. 2024-03-28T23:30:41.354 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:30:41.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.458+0000 7fd527288640 1 Processor -- start 2024-03-28T23:30:41.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.458+0000 7fd527288640 1 -- start start 2024-03-28T23:30:41.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.459+0000 7fd527288640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd520072a50 0x7fd5200c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:41.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.459+0000 7fd527288640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5200c2480 con 0x7fd520159120 2024-03-28T23:30:41.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.459+0000 7fd527288640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5200c2760 con 0x7fd5200c77e0 2024-03-28T23:30:41.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.459+0000 7fd527288640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd5200c2230 con 0x7fd520072a50 2024-03-28T23:30:41.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.459+0000 7fd524823640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd520159120 0x7fd52015dd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:34946/0 (socket says 172.21.15.161:34946) 2024-03-28T23:30:41.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.459+0000 7fd524823640 1 -- 172.21.15.161:0/1838241663 learned_addr learned my addr 172.21.15.161:0/1838241663 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:41.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.459+0000 7fd525825640 1 --2- 172.21.15.161:0/1838241663 >> v2:172.21.15.170:3300/0 conn(0x7fd520072a50 0x7fd5200c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:41.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.459+0000 7fd517fff640 1 -- 172.21.15.161:0/1838241663 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3158605929 0 0) 0x7fd5200c2480 con 0x7fd520159120 2024-03-28T23:30:41.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.459+0000 7fd517fff640 1 -- 172.21.15.161:0/1838241663 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd50c003520 con 0x7fd520159120 2024-03-28T23:30:41.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.459+0000 7fd517fff640 1 -- 172.21.15.161:0/1838241663 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 504549592 0 0) 0x7fd50c003520 con 0x7fd520159120 2024-03-28T23:30:41.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.459+0000 7fd517fff640 1 -- 172.21.15.161:0/1838241663 >> v2:172.21.15.170:3300/0 conn(0x7fd520072a50 msgr2=0x7fd5200c7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:41.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.459+0000 7fd517fff640 1 --2- 172.21.15.161:0/1838241663 >> v2:172.21.15.170:3300/0 conn(0x7fd520072a50 0x7fd5200c7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.459+0000 7fd517fff640 1 -- 172.21.15.161:0/1838241663 >> v1:172.21.15.170:6789/0 conn(0x7fd5200c77e0 legacy=0x7fd520158a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:41.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.460+0000 7fd517fff640 1 -- 172.21.15.161:0/1838241663 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd520072250 con 0x7fd520159120 2024-03-28T23:30:41.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.460+0000 7fd517fff640 1 -- 172.21.15.161:0/1838241663 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fd51c001670 con 0x7fd520159120 2024-03-28T23:30:41.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.460+0000 7fd517fff640 1 -- 172.21.15.161:0/1838241663 >> v1:172.21.15.161:6789/0 conn(0x7fd520159120 legacy=0x7fd52015dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:41.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.460+0000 7fd517fff640 1 --2- 172.21.15.161:0/1838241663 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd50c003a90 0x7fd50c003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:41.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.460+0000 7fd517fff640 1 --2- 172.21.15.161:0/1838241663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd50c004800 0x7fd50c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:41.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.460+0000 7fd517fff640 1 --2- 172.21.15.161:0/1838241663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd50c005210 0x7fd50c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:41.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.460+0000 7fd517fff640 1 -- 172.21.15.161:0/1838241663 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd5200c2230 con 0x7fd50c003a90 2024-03-28T23:30:41.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.460+0000 7fd517fff640 1 -- 172.21.15.161:0/1838241663 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd50c00c140 con 0x7fd50c004800 2024-03-28T23:30:41.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.460+0000 7fd517fff640 1 -- 172.21.15.161:0/1838241663 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd50c00c340 con 0x7fd50c005210 2024-03-28T23:30:41.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.460+0000 7fd525024640 1 --2- 172.21.15.161:0/1838241663 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd50c003a90 0x7fd50c003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:41.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.460+0000 7fd525825640 1 --2- 172.21.15.161:0/1838241663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd50c005210 0x7fd50c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:41.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.460+0000 7fd524823640 1 --2- 172.21.15.161:0/1838241663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd50c004800 0x7fd50c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:41.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.460+0000 7fd525024640 1 -- 172.21.15.161:0/1838241663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd50c005210 msgr2=0x7fd50c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:41.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.460+0000 7fd525024640 1 --2- 172.21.15.161:0/1838241663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd50c005210 0x7fd50c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.460+0000 7fd525024640 1 -- 172.21.15.161:0/1838241663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd50c004800 msgr2=0x7fd50c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:41.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.460+0000 7fd525024640 1 --2- 172.21.15.161:0/1838241663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd50c004800 0x7fd50c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.460+0000 7fd525024640 1 -- 172.21.15.161:0/1838241663 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd50c003520 con 0x7fd50c003a90 2024-03-28T23:30:41.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.460+0000 7fd525825640 1 -- 172.21.15.161:0/1838241663 reap_dead start 2024-03-28T23:30:41.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.460+0000 7fd524823640 1 --2- 172.21.15.161:0/1838241663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd50c004800 0x7fd50c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:41.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.461+0000 7fd525024640 1 --2- 172.21.15.161:0/1838241663 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd50c003a90 0x7fd50c003eb0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd510001720 tx=0x7fd50c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af7641843bd9106 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:41.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.461+0000 7fd517fff640 1 -- 172.21.15.161:0/1838241663 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd510001370 con 0x7fd50c003a90 2024-03-28T23:30:41.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.461+0000 7fd517fff640 1 -- 172.21.15.161:0/1838241663 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5100052d0 con 0x7fd50c003a90 2024-03-28T23:30:41.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.461+0000 7fd517fff640 1 -- 172.21.15.161:0/1838241663 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd510003830 con 0x7fd50c003a90 2024-03-28T23:30:41.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.462+0000 7fd527288640 1 -- 172.21.15.161:0/1838241663 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd50c003a90 msgr2=0x7fd50c003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:41.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.462+0000 7fd527288640 1 --2- 172.21.15.161:0/1838241663 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd50c003a90 0x7fd50c003eb0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd510001720 tx=0x7fd50c005830 comp rx=0 tx=0).stop 2024-03-28T23:30:41.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.462+0000 7fd527288640 1 -- 172.21.15.161:0/1838241663 shutdown_connections 2024-03-28T23:30:41.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.462+0000 7fd527288640 1 --2- 172.21.15.161:0/1838241663 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd50c003a90 0x7fd50c003eb0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.462+0000 7fd527288640 1 -- 172.21.15.161:0/1838241663 >> 172.21.15.161:0/1838241663 conn(0x7fd5200a97c0 msgr2=0x7fd5200a9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:41.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.462+0000 7fd527288640 1 -- 172.21.15.161:0/1838241663 shutdown_connections 2024-03-28T23:30:41.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.462+0000 7fd527288640 1 -- 172.21.15.161:0/1838241663 wait complete. 2024-03-28T23:30:41.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.464+0000 7fd527288640 1 Processor -- start 2024-03-28T23:30:41.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.464+0000 7fd527288640 1 -- start start 2024-03-28T23:30:41.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.465+0000 7fd527288640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd50c003a90 0x7fd5201ef070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:41.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.465+0000 7fd527288640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5200a9100 0x7fd520151ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:41.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.465+0000 7fd527288640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd520157840 0x7fd52014e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:41.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.465+0000 7fd527288640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd5200c36e0 con 0x7fd520157840 2024-03-28T23:30:41.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.465+0000 7fd527288640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd5200725c0 con 0x7fd5200a9100 2024-03-28T23:30:41.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.465+0000 7fd527288640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd520156390 con 0x7fd50c003a90 2024-03-28T23:30:41.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.465+0000 7fd525825640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd520157840 0x7fd52014e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:41.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.465+0000 7fd525825640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd520157840 0x7fd52014e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51858/0 (socket says 172.21.15.161:51858) 2024-03-28T23:30:41.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.465+0000 7fd525825640 1 -- 172.21.15.161:0/1046673803 learned_addr learned my addr 172.21.15.161:0/1046673803 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:41.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.465+0000 7fd524823640 1 --2- 172.21.15.161:0/1046673803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5200a9100 0x7fd520151ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:41.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.465+0000 7fd525024640 1 --2- 172.21.15.161:0/1046673803 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd50c003a90 0x7fd5201ef070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:41.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.465+0000 7fd525825640 1 -- 172.21.15.161:0/1046673803 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd50c003a90 msgr2=0x7fd5201ef070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:41.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.465+0000 7fd525825640 1 --2- 172.21.15.161:0/1046673803 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd50c003a90 0x7fd5201ef070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.465+0000 7fd525825640 1 -- 172.21.15.161:0/1046673803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5200a9100 msgr2=0x7fd520151ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:41.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.465+0000 7fd525825640 1 --2- 172.21.15.161:0/1046673803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5200a9100 0x7fd520151ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.465+0000 7fd525825640 1 -- 172.21.15.161:0/1046673803 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd520157d40 con 0x7fd520157840 2024-03-28T23:30:41.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.465+0000 7fd525024640 1 --2- 172.21.15.161:0/1046673803 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd50c003a90 0x7fd5201ef070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:41.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.465+0000 7fd525825640 1 --2- 172.21.15.161:0/1046673803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd520157840 0x7fd52014e790 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd5200716a0 tx=0x7fd51800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:41.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.466+0000 7fd515ffb640 1 -- 172.21.15.161:0/1046673803 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd518019070 con 0x7fd520157840 2024-03-28T23:30:41.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.466+0000 7fd515ffb640 1 -- 172.21.15.161:0/1046673803 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd518011e90 con 0x7fd520157840 2024-03-28T23:30:41.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.466+0000 7fd527288640 1 -- 172.21.15.161:0/1046673803 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd520158020 con 0x7fd520157840 2024-03-28T23:30:41.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.466+0000 7fd515ffb640 1 -- 172.21.15.161:0/1046673803 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd518003bd0 con 0x7fd520157840 2024-03-28T23:30:41.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.466+0000 7fd527288640 1 -- 172.21.15.161:0/1046673803 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5201eeb50 con 0x7fd520157840 2024-03-28T23:30:41.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.467+0000 7fd527288640 1 -- 172.21.15.161:0/1046673803 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5200c31d0 con 0x7fd520157840 2024-03-28T23:30:41.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.467+0000 7fd515ffb640 1 -- 172.21.15.161:0/1046673803 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd51801c070 con 0x7fd520157840 2024-03-28T23:30:41.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.468+0000 7fd515ffb640 1 --2- 172.21.15.161:0/1046673803 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd50406b500 0x7fd50406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:41.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.468+0000 7fd515ffb640 1 -- 172.21.15.161:0/1046673803 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fd51808adf0 con 0x7fd520157840 2024-03-28T23:30:41.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.468+0000 7fd525024640 1 --2- 172.21.15.161:0/1046673803 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd50406b500 0x7fd50406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:41.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.469+0000 7fd525024640 1 --2- 172.21.15.161:0/1046673803 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd50406b500 0x7fd50406d9c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd510033000 tx=0x7fd510009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:41.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.472+0000 7fd515ffb640 1 -- 172.21.15.161:0/1046673803 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd5200c31d0 con 0x7fd520157840 2024-03-28T23:30:41.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.654+0000 7fd527288640 1 -- 172.21.15.161:0/1046673803 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd52004f4b0 con 0x7fd520157840 2024-03-28T23:30:41.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.655+0000 7fd515ffb640 1 -- 172.21.15.161:0/1046673803 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fd518010040 con 0x7fd520157840 2024-03-28T23:30:41.656 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:30:41.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.657+0000 7fd527288640 1 -- 172.21.15.161:0/1046673803 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd50406b500 msgr2=0x7fd50406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:41.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.657+0000 7fd527288640 1 --2- 172.21.15.161:0/1046673803 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd50406b500 0x7fd50406d9c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd510033000 tx=0x7fd510009030 comp rx=0 tx=0).stop 2024-03-28T23:30:41.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.657+0000 7fd527288640 1 -- 172.21.15.161:0/1046673803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd520157840 msgr2=0x7fd52014e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:41.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.657+0000 7fd527288640 1 --2- 172.21.15.161:0/1046673803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd520157840 0x7fd52014e790 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd5200716a0 tx=0x7fd51800f580 comp rx=0 tx=0).stop 2024-03-28T23:30:41.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.657+0000 7fd527288640 1 -- 172.21.15.161:0/1046673803 shutdown_connections 2024-03-28T23:30:41.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.657+0000 7fd527288640 1 --2- 172.21.15.161:0/1046673803 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd50406b500 0x7fd50406d9c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.657+0000 7fd527288640 1 --2- 172.21.15.161:0/1046673803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd520157840 0x7fd52014e790 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.657+0000 7fd527288640 1 --2- 172.21.15.161:0/1046673803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5200a9100 0x7fd520151ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.657+0000 7fd527288640 1 --2- 172.21.15.161:0/1046673803 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd50c003a90 0x7fd5201ef070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.657+0000 7fd527288640 1 -- 172.21.15.161:0/1046673803 >> 172.21.15.161:0/1046673803 conn(0x7fd5200a97c0 msgr2=0x7fd520151090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:41.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.657+0000 7fd527288640 1 -- 172.21.15.161:0/1046673803 shutdown_connections 2024-03-28T23:30:41.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.657+0000 7fd527288640 1 -- 172.21.15.161:0/1046673803 wait complete. 2024-03-28T23:30:41.669 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:30:41.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.773+0000 7fa8758d4640 1 Processor -- start 2024-03-28T23:30:41.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.773+0000 7fa8758d4640 1 -- start start 2024-03-28T23:30:41.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.774+0000 7fa8758d4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa870152fb0 0x7fa8700c2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:41.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.774+0000 7fa8758d4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa870072b50 0x7fa870072f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:41.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.774+0000 7fa8758d4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa870153500 con 0x7fa87013c430 2024-03-28T23:30:41.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.774+0000 7fa8758d4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa87013c810 con 0x7fa870152fb0 2024-03-28T23:30:41.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.774+0000 7fa8758d4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fa8701537e0 con 0x7fa870072b50 2024-03-28T23:30:41.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.774+0000 7fa86e7fc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa870152fb0 0x7fa8700c2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:41.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.774+0000 7fa86effd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa87013c430 0x7fa87006e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:34960/0 (socket says 172.21.15.161:34960) 2024-03-28T23:30:41.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.774+0000 7fa86effd640 1 -- 172.21.15.161:0/1397228845 learned_addr learned my addr 172.21.15.161:0/1397228845 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:41.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.774+0000 7fa86f7fe640 1 --2- 172.21.15.161:0/1397228845 >> v2:172.21.15.170:3300/0 conn(0x7fa870072b50 0x7fa870072f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:41.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.775+0000 7fa86e7fc640 1 -- 172.21.15.161:0/1397228845 >> v2:172.21.15.170:3300/0 conn(0x7fa870072b50 msgr2=0x7fa870072f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:41.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.775+0000 7fa86e7fc640 1 --2- 172.21.15.161:0/1397228845 >> v2:172.21.15.170:3300/0 conn(0x7fa870072b50 0x7fa870072f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.775+0000 7fa86e7fc640 1 -- 172.21.15.161:0/1397228845 >> v1:172.21.15.161:6789/0 conn(0x7fa87013c430 legacy=0x7fa87006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:41.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.775+0000 7fa86e7fc640 1 -- 172.21.15.161:0/1397228845 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa870153ac0 con 0x7fa870152fb0 2024-03-28T23:30:41.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.775+0000 7fa86e7fc640 1 --2- 172.21.15.161:0/1397228845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa870152fb0 0x7fa8700c2c10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa8700714a0 tx=0x7fa86002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed7308b800ba72e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:41.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.775+0000 7fa86dffb640 1 -- 172.21.15.161:0/1397228845 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8600315a0 con 0x7fa870152fb0 2024-03-28T23:30:41.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.775+0000 7fa86dffb640 1 -- 172.21.15.161:0/1397228845 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa860037070 con 0x7fa870152fb0 2024-03-28T23:30:41.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.775+0000 7fa86dffb640 1 -- 172.21.15.161:0/1397228845 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa860031960 con 0x7fa870152fb0 2024-03-28T23:30:41.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.776+0000 7fa8758d4640 1 -- 172.21.15.161:0/1397228845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa870152fb0 msgr2=0x7fa8700c2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:41.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.776+0000 7fa8758d4640 1 --2- 172.21.15.161:0/1397228845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa870152fb0 0x7fa8700c2c10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa8700714a0 tx=0x7fa86002b840 comp rx=0 tx=0).stop 2024-03-28T23:30:41.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.776+0000 7fa8758d4640 1 -- 172.21.15.161:0/1397228845 shutdown_connections 2024-03-28T23:30:41.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.776+0000 7fa8758d4640 1 --2- 172.21.15.161:0/1397228845 >> v2:172.21.15.170:3300/0 conn(0x7fa870072b50 0x7fa870072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.776+0000 7fa8758d4640 1 --2- 172.21.15.161:0/1397228845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa870152fb0 0x7fa8700c2c10 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.776+0000 7fa8758d4640 1 -- 172.21.15.161:0/1397228845 >> 172.21.15.161:0/1397228845 conn(0x7fa8700c7760 msgr2=0x7fa8700c7b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:41.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.776+0000 7fa8758d4640 1 -- 172.21.15.161:0/1397228845 shutdown_connections 2024-03-28T23:30:41.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.776+0000 7fa8758d4640 1 -- 172.21.15.161:0/1397228845 wait complete. 2024-03-28T23:30:41.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.778+0000 7fa8758d4640 1 Processor -- start 2024-03-28T23:30:41.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.778+0000 7fa8758d4640 1 -- start start 2024-03-28T23:30:41.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.779+0000 7fa8758d4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa870072b50 0x7fa8700c1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:41.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.779+0000 7fa8758d4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa87013c430 0x7fa8700bd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:41.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.779+0000 7fa8758d4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa870152fb0 0x7fa8700bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:41.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.779+0000 7fa8758d4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa87006eb30 con 0x7fa870072b50 2024-03-28T23:30:41.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.779+0000 7fa8758d4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa8700c3650 con 0x7fa87013c430 2024-03-28T23:30:41.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.779+0000 7fa8758d4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa87006ecb0 con 0x7fa870152fb0 2024-03-28T23:30:41.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.779+0000 7fa86effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa870072b50 0x7fa8700c1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:41.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.779+0000 7fa86effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa870072b50 0x7fa8700c1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51882/0 (socket says 172.21.15.161:51882) 2024-03-28T23:30:41.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.779+0000 7fa86effd640 1 -- 172.21.15.161:0/2207494297 learned_addr learned my addr 172.21.15.161:0/2207494297 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:41.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.779+0000 7fa86e7fc640 1 --2- 172.21.15.161:0/2207494297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa87013c430 0x7fa8700bd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:41.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.779+0000 7fa86f7fe640 1 --2- 172.21.15.161:0/2207494297 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa870152fb0 0x7fa8700bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:41.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.779+0000 7fa86effd640 1 -- 172.21.15.161:0/2207494297 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa870152fb0 msgr2=0x7fa8700bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:41.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.779+0000 7fa86effd640 1 --2- 172.21.15.161:0/2207494297 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa870152fb0 0x7fa8700bfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.779+0000 7fa86effd640 1 -- 172.21.15.161:0/2207494297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa87013c430 msgr2=0x7fa8700bd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:41.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.779+0000 7fa86effd640 1 --2- 172.21.15.161:0/2207494297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa87013c430 0x7fa8700bd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:41.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.779+0000 7fa86effd640 1 -- 172.21.15.161:0/2207494297 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8700c7340 con 0x7fa870072b50 2024-03-28T23:30:41.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.779+0000 7fa86f7fe640 1 --2- 172.21.15.161:0/2207494297 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa870152fb0 0x7fa8700bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:41.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.779+0000 7fa86effd640 1 --2- 172.21.15.161:0/2207494297 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa870072b50 0x7fa8700c1210 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa86400c3f0 tx=0x7fa86400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:41.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.780+0000 7fa857fff640 1 -- 172.21.15.161:0/2207494297 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa864017070 con 0x7fa870072b50 2024-03-28T23:30:41.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.780+0000 7fa857fff640 1 -- 172.21.15.161:0/2207494297 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa864005ce0 con 0x7fa870072b50 2024-03-28T23:30:41.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.780+0000 7fa857fff640 1 -- 172.21.15.161:0/2207494297 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8640072f0 con 0x7fa870072b50 2024-03-28T23:30:41.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.780+0000 7fa8758d4640 1 -- 172.21.15.161:0/2207494297 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa87012b170 con 0x7fa870072b50 2024-03-28T23:30:41.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.780+0000 7fa8758d4640 1 -- 172.21.15.161:0/2207494297 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8700bd560 con 0x7fa870072b50 2024-03-28T23:30:41.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.781+0000 7fa8758d4640 1 -- 172.21.15.161:0/2207494297 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa87004f4b0 con 0x7fa870072b50 2024-03-28T23:30:41.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.781+0000 7fa857fff640 1 -- 172.21.15.161:0/2207494297 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa86401a070 con 0x7fa870072b50 2024-03-28T23:30:41.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.781+0000 7fa857fff640 1 --2- 172.21.15.161:0/2207494297 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa85c06b4d0 0x7fa85c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:41.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.782+0000 7fa857fff640 1 -- 172.21.15.161:0/2207494297 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fa864089860 con 0x7fa870072b50 2024-03-28T23:30:41.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.782+0000 7fa86e7fc640 1 --2- 172.21.15.161:0/2207494297 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa85c06b4d0 0x7fa85c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:41.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.782+0000 7fa86e7fc640 1 --2- 172.21.15.161:0/2207494297 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa85c06b4d0 0x7fa85c06d990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa860036000 tx=0x7fa86003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:41.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:41.790+0000 7fa857fff640 1 -- 172.21.15.161:0/2207494297 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa864057d20 con 0x7fa870072b50 2024-03-28T23:30:42.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.048+0000 7fa8758d4640 1 -- 172.21.15.161:0/2207494297 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa8700be390 con 0x7fa870072b50 2024-03-28T23:30:42.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.048+0000 7fa857fff640 1 -- 172.21.15.161:0/2207494297 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fa86405bdc0 con 0x7fa870072b50 2024-03-28T23:30:42.049 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:30:42.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.050+0000 7fa8758d4640 1 -- 172.21.15.161:0/2207494297 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa85c06b4d0 msgr2=0x7fa85c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:42.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.050+0000 7fa8758d4640 1 --2- 172.21.15.161:0/2207494297 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa85c06b4d0 0x7fa85c06d990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa860036000 tx=0x7fa86003d040 comp rx=0 tx=0).stop 2024-03-28T23:30:42.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.050+0000 7fa8758d4640 1 -- 172.21.15.161:0/2207494297 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa870072b50 msgr2=0x7fa8700c1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:42.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.050+0000 7fa8758d4640 1 --2- 172.21.15.161:0/2207494297 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa870072b50 0x7fa8700c1210 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa86400c3f0 tx=0x7fa86400c8b0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.050+0000 7fa8758d4640 1 -- 172.21.15.161:0/2207494297 shutdown_connections 2024-03-28T23:30:42.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.050+0000 7fa8758d4640 1 --2- 172.21.15.161:0/2207494297 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa85c06b4d0 0x7fa85c06d990 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.050+0000 7fa8758d4640 1 --2- 172.21.15.161:0/2207494297 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa870152fb0 0x7fa8700bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.050+0000 7fa8758d4640 1 --2- 172.21.15.161:0/2207494297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa87013c430 0x7fa8700bd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.050+0000 7fa8758d4640 1 --2- 172.21.15.161:0/2207494297 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa870072b50 0x7fa8700c1210 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.050+0000 7fa8758d4640 1 -- 172.21.15.161:0/2207494297 >> 172.21.15.161:0/2207494297 conn(0x7fa8700c7760 msgr2=0x7fa870151be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:42.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.051+0000 7fa8758d4640 1 -- 172.21.15.161:0/2207494297 shutdown_connections 2024-03-28T23:30:42.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.051+0000 7fa8758d4640 1 -- 172.21.15.161:0/2207494297 wait complete. 2024-03-28T23:30:42.063 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T23:30:42.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.169+0000 7fc62de8f640 1 Processor -- start 2024-03-28T23:30:42.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.169+0000 7fc62de8f640 1 -- start start 2024-03-28T23:30:42.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.170+0000 7fc62de8f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc6280c0330 0x7fc6280c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:42.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.170+0000 7fc62de8f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6281589b0 0x7fc628158e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:42.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.170+0000 7fc62de8f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6280c6f60 con 0x7fc628152bc0 2024-03-28T23:30:42.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.170+0000 7fc62de8f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc6280c6120 con 0x7fc6280c0330 2024-03-28T23:30:42.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.170+0000 7fc62de8f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc6280c6400 con 0x7fc6281589b0 2024-03-28T23:30:42.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.170+0000 7fc626ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc6280c0330 0x7fc6280c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:42.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.170+0000 7fc6277fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc628152bc0 0x7fc6280c8a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:34968/0 (socket says 172.21.15.161:34968) 2024-03-28T23:30:42.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.170+0000 7fc626ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc6280c0330 0x7fc6280c9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51888/0 (socket says 172.21.15.161:51888) 2024-03-28T23:30:42.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.170+0000 7fc6277fe640 1 -- 172.21.15.161:0/2209466145 learned_addr learned my addr 172.21.15.161:0/2209466145 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:42.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.170+0000 7fc627fff640 1 --2- 172.21.15.161:0/2209466145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6281589b0 0x7fc628158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:42.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.170+0000 7fc626ffd640 1 -- 172.21.15.161:0/2209466145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6281589b0 msgr2=0x7fc628158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:42.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.170+0000 7fc626ffd640 1 --2- 172.21.15.161:0/2209466145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6281589b0 0x7fc628158e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.170+0000 7fc626ffd640 1 -- 172.21.15.161:0/2209466145 >> v1:172.21.15.161:6789/0 conn(0x7fc628152bc0 legacy=0x7fc6280c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:42.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.170+0000 7fc6267fc640 1 -- 172.21.15.161:0/2209466145 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 542920327 0 0) 0x7fc6280c6f60 con 0x7fc628152bc0 2024-03-28T23:30:42.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.170+0000 7fc626ffd640 1 -- 172.21.15.161:0/2209466145 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6280c96d0 con 0x7fc6280c0330 2024-03-28T23:30:42.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.171+0000 7fc626ffd640 1 --2- 172.21.15.161:0/2209466145 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6280c0330 0x7fc6280c9190 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc61c0097a0 tx=0x7fc61c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96f30c48018ccb31 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:42.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.171+0000 7fc6267fc640 1 -- 172.21.15.161:0/2209466145 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61c035020 con 0x7fc6280c0330 2024-03-28T23:30:42.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.171+0000 7fc6267fc640 1 -- 172.21.15.161:0/2209466145 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc61c031830 con 0x7fc6280c0330 2024-03-28T23:30:42.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.171+0000 7fc6267fc640 1 -- 172.21.15.161:0/2209466145 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61c031c00 con 0x7fc6280c0330 2024-03-28T23:30:42.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.172+0000 7fc62de8f640 1 -- 172.21.15.161:0/2209466145 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6280c0330 msgr2=0x7fc6280c9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:42.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.172+0000 7fc62de8f640 1 --2- 172.21.15.161:0/2209466145 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6280c0330 0x7fc6280c9190 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc61c0097a0 tx=0x7fc61c02b8d0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.172+0000 7fc62de8f640 1 -- 172.21.15.161:0/2209466145 shutdown_connections 2024-03-28T23:30:42.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.172+0000 7fc62de8f640 1 --2- 172.21.15.161:0/2209466145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6281589b0 0x7fc628158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.172+0000 7fc62de8f640 1 --2- 172.21.15.161:0/2209466145 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6280c0330 0x7fc6280c9190 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.172+0000 7fc62de8f640 1 -- 172.21.15.161:0/2209466145 >> 172.21.15.161:0/2209466145 conn(0x7fc6280caa80 msgr2=0x7fc6280cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:42.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.172+0000 7fc62de8f640 1 -- 172.21.15.161:0/2209466145 shutdown_connections 2024-03-28T23:30:42.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.172+0000 7fc62de8f640 1 -- 172.21.15.161:0/2209466145 wait complete. 2024-03-28T23:30:42.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.173+0000 7fc62de8f640 1 Processor -- start 2024-03-28T23:30:42.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.173+0000 7fc62de8f640 1 -- start start 2024-03-28T23:30:42.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.174+0000 7fc62de8f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6280c0330 0x7fc6281558c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:42.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.174+0000 7fc62de8f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc628152bc0 0x7fc62806c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:42.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.174+0000 7fc62de8f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6281589b0 0x7fc6281ea850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:42.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.174+0000 7fc62de8f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc628152fa0 con 0x7fc628152bc0 2024-03-28T23:30:42.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.174+0000 7fc62de8f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc6280c6bf0 con 0x7fc6281589b0 2024-03-28T23:30:42.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.174+0000 7fc62de8f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc628155ec0 con 0x7fc6280c0330 2024-03-28T23:30:42.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.174+0000 7fc626ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc628152bc0 0x7fc62806c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:42.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.174+0000 7fc626ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc628152bc0 0x7fc62806c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51898/0 (socket says 172.21.15.161:51898) 2024-03-28T23:30:42.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.174+0000 7fc626ffd640 1 -- 172.21.15.161:0/1751680475 learned_addr learned my addr 172.21.15.161:0/1751680475 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:42.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.174+0000 7fc6277fe640 1 --2- 172.21.15.161:0/1751680475 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6280c0330 0x7fc6281558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:42.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.174+0000 7fc627fff640 1 --2- 172.21.15.161:0/1751680475 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6281589b0 0x7fc6281ea850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:42.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.174+0000 7fc626ffd640 1 -- 172.21.15.161:0/1751680475 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6280c0330 msgr2=0x7fc6281558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:42.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.174+0000 7fc626ffd640 1 --2- 172.21.15.161:0/1751680475 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6280c0330 0x7fc6281558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.174+0000 7fc626ffd640 1 -- 172.21.15.161:0/1751680475 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6281589b0 msgr2=0x7fc6281ea850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:42.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.174+0000 7fc626ffd640 1 --2- 172.21.15.161:0/1751680475 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6281589b0 0x7fc6281ea850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.175+0000 7fc626ffd640 1 -- 172.21.15.161:0/1751680475 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc628150580 con 0x7fc628152bc0 2024-03-28T23:30:42.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.175+0000 7fc626ffd640 1 --2- 172.21.15.161:0/1751680475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc628152bc0 0x7fc62806c610 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc61c000c00 tx=0x7fc61c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:42.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.176+0000 7fc60ffff640 1 -- 172.21.15.161:0/1751680475 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61c035050 con 0x7fc628152bc0 2024-03-28T23:30:42.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.176+0000 7fc60ffff640 1 -- 172.21.15.161:0/1751680475 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc61c004230 con 0x7fc628152bc0 2024-03-28T23:30:42.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.176+0000 7fc62de8f640 1 -- 172.21.15.161:0/1751680475 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc628150860 con 0x7fc628152bc0 2024-03-28T23:30:42.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.176+0000 7fc60ffff640 1 -- 172.21.15.161:0/1751680475 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61c03b800 con 0x7fc628152bc0 2024-03-28T23:30:42.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.176+0000 7fc62de8f640 1 -- 172.21.15.161:0/1751680475 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc62806cb50 con 0x7fc628152bc0 2024-03-28T23:30:42.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.177+0000 7fc62de8f640 1 -- 172.21.15.161:0/1751680475 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc62814ff20 con 0x7fc628152bc0 2024-03-28T23:30:42.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.177+0000 7fc60ffff640 1 -- 172.21.15.161:0/1751680475 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc61c038020 con 0x7fc628152bc0 2024-03-28T23:30:42.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.178+0000 7fc60ffff640 1 --2- 172.21.15.161:0/1751680475 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc61406b4d0 0x7fc61406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:42.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.178+0000 7fc60ffff640 1 -- 172.21.15.161:0/1751680475 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fc61c0ac2b0 con 0x7fc628152bc0 2024-03-28T23:30:42.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.178+0000 7fc6277fe640 1 --2- 172.21.15.161:0/1751680475 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc61406b4d0 0x7fc61406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:42.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.178+0000 7fc6277fe640 1 --2- 172.21.15.161:0/1751680475 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc61406b4d0 0x7fc61406d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc618000f60 tx=0x7fc618007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:42.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.187+0000 7fc60ffff640 1 -- 172.21.15.161:0/1751680475 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc62814ff20 con 0x7fc628152bc0 2024-03-28T23:30:42.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.365+0000 7fc62de8f640 1 -- 172.21.15.161:0/1751680475 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc62804f4b0 con 0x7fc628152bc0 2024-03-28T23:30:42.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.365+0000 7fc60ffff640 1 -- 172.21.15.161:0/1751680475 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fc61c040070 con 0x7fc628152bc0 2024-03-28T23:30:42.366 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:30:42.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.367+0000 7fc62de8f640 1 -- 172.21.15.161:0/1751680475 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc61406b4d0 msgr2=0x7fc61406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:42.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.367+0000 7fc62de8f640 1 --2- 172.21.15.161:0/1751680475 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc61406b4d0 0x7fc61406d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc618000f60 tx=0x7fc618007d50 comp rx=0 tx=0).stop 2024-03-28T23:30:42.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.367+0000 7fc62de8f640 1 -- 172.21.15.161:0/1751680475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc628152bc0 msgr2=0x7fc62806c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:42.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.367+0000 7fc62de8f640 1 --2- 172.21.15.161:0/1751680475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc628152bc0 0x7fc62806c610 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc61c000c00 tx=0x7fc61c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.367+0000 7fc62de8f640 1 -- 172.21.15.161:0/1751680475 shutdown_connections 2024-03-28T23:30:42.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.367+0000 7fc62de8f640 1 --2- 172.21.15.161:0/1751680475 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc61406b4d0 0x7fc61406d990 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.367+0000 7fc62de8f640 1 --2- 172.21.15.161:0/1751680475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc628152bc0 0x7fc62806c610 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.367+0000 7fc62de8f640 1 --2- 172.21.15.161:0/1751680475 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6281589b0 0x7fc6281ea850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.367+0000 7fc62de8f640 1 --2- 172.21.15.161:0/1751680475 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6280c0330 0x7fc6281558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.367+0000 7fc62de8f640 1 -- 172.21.15.161:0/1751680475 >> 172.21.15.161:0/1751680475 conn(0x7fc6280caa80 msgr2=0x7fc6280cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:42.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.367+0000 7fc62de8f640 1 -- 172.21.15.161:0/1751680475 shutdown_connections 2024-03-28T23:30:42.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.367+0000 7fc62de8f640 1 -- 172.21.15.161:0/1751680475 wait complete. 2024-03-28T23:30:42.379 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T23:30:42.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.482+0000 7f604b9b5640 1 Processor -- start 2024-03-28T23:30:42.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.482+0000 7f604b9b5640 1 -- start start 2024-03-28T23:30:42.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.483+0000 7f604b9b5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6044152890 0x7f60440c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:42.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.483+0000 7f604b9b5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f60440c8ed0 0x7f60440c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:42.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.483+0000 7f604b9b5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6044158a30 con 0x7f60440c5950 2024-03-28T23:30:42.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.483+0000 7f604b9b5640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f60440c8480 con 0x7f6044152890 2024-03-28T23:30:42.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.483+0000 7f604b9b5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f6044159640 con 0x7f60440c8ed0 2024-03-28T23:30:42.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.483+0000 7f6049751640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6044152890 0x7f60440c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:42.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.483+0000 7f6049751640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6044152890 0x7f60440c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51908/0 (socket says 172.21.15.161:51908) 2024-03-28T23:30:42.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.483+0000 7f6049751640 1 -- 172.21.15.161:0/2102043658 learned_addr learned my addr 172.21.15.161:0/2102043658 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:42.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.484+0000 7f6049751640 1 -- 172.21.15.161:0/2102043658 >> v2:172.21.15.170:3300/0 conn(0x7f60440c8ed0 msgr2=0x7f60440c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:42.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.484+0000 7f6049751640 1 --2- 172.21.15.161:0/2102043658 >> v2:172.21.15.170:3300/0 conn(0x7f60440c8ed0 0x7f60440c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.484+0000 7f6049751640 1 -- 172.21.15.161:0/2102043658 >> v1:172.21.15.170:6789/0 conn(0x7f60440c5950 legacy=0x7f60441553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:42.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.484+0000 7f6049751640 1 -- 172.21.15.161:0/2102043658 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6044158d10 con 0x7f6044152890 2024-03-28T23:30:42.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.484+0000 7f6049751640 1 --2- 172.21.15.161:0/2102043658 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6044152890 0x7f60440c8990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f60440ad6e0 tx=0x7f604002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6955c677575bf7b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:42.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.484+0000 7f603bfff640 1 -- 172.21.15.161:0/2102043658 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6040038070 con 0x7f6044152890 2024-03-28T23:30:42.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.484+0000 7f603bfff640 1 -- 172.21.15.161:0/2102043658 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f604002f050 con 0x7f6044152890 2024-03-28T23:30:42.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.485+0000 7f603bfff640 1 -- 172.21.15.161:0/2102043658 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6040031970 con 0x7f6044152890 2024-03-28T23:30:42.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.485+0000 7f604b9b5640 1 -- 172.21.15.161:0/2102043658 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6044152890 msgr2=0x7f60440c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:42.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.485+0000 7f604b9b5640 1 --2- 172.21.15.161:0/2102043658 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6044152890 0x7f60440c8990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f60440ad6e0 tx=0x7f604002b7f0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.485+0000 7f604b9b5640 1 -- 172.21.15.161:0/2102043658 shutdown_connections 2024-03-28T23:30:42.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.485+0000 7f604b9b5640 1 --2- 172.21.15.161:0/2102043658 >> v2:172.21.15.170:3300/0 conn(0x7f60440c8ed0 0x7f60440c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.485+0000 7f604b9b5640 1 --2- 172.21.15.161:0/2102043658 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6044152890 0x7f60440c8990 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.485+0000 7f604b9b5640 1 -- 172.21.15.161:0/2102043658 >> 172.21.15.161:0/2102043658 conn(0x7f60440ac620 msgr2=0x7f60440aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:42.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.485+0000 7f604b9b5640 1 -- 172.21.15.161:0/2102043658 shutdown_connections 2024-03-28T23:30:42.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.485+0000 7f604b9b5640 1 -- 172.21.15.161:0/2102043658 wait complete. 2024-03-28T23:30:42.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.486+0000 7f604b9b5640 1 Processor -- start 2024-03-28T23:30:42.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.486+0000 7f604b9b5640 1 -- start start 2024-03-28T23:30:42.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.487+0000 7f604b9b5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60440c5950 0x7f60440ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:42.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.487+0000 7f604b9b5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60440c8ed0 0x7f6044148e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:42.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.487+0000 7f604b9b5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6044152890 0x7f6044149360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:42.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.487+0000 7f604b9b5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6044152c70 con 0x7f60440c5950 2024-03-28T23:30:42.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.487+0000 7f604b9b5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f60441592d0 con 0x7f6044152890 2024-03-28T23:30:42.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.487+0000 7f604b9b5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6044152df0 con 0x7f60440c8ed0 2024-03-28T23:30:42.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.487+0000 7f6049751640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60440c5950 0x7f60440ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:42.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.487+0000 7f6049751640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60440c5950 0x7f60440ab6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51916/0 (socket says 172.21.15.161:51916) 2024-03-28T23:30:42.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.487+0000 7f6049751640 1 -- 172.21.15.161:0/3437954004 learned_addr learned my addr 172.21.15.161:0/3437954004 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:42.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.487+0000 7f6049f52640 1 --2- 172.21.15.161:0/3437954004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6044152890 0x7f6044149360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:42.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.487+0000 7f6048f50640 1 --2- 172.21.15.161:0/3437954004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60440c8ed0 0x7f6044148e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:42.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.487+0000 7f6049751640 1 -- 172.21.15.161:0/3437954004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60440c8ed0 msgr2=0x7f6044148e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:42.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.487+0000 7f6049751640 1 --2- 172.21.15.161:0/3437954004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60440c8ed0 0x7f6044148e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.487+0000 7f6049751640 1 -- 172.21.15.161:0/3437954004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6044152890 msgr2=0x7f6044149360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:42.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.487+0000 7f6049751640 1 --2- 172.21.15.161:0/3437954004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6044152890 0x7f6044149360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.487+0000 7f6049751640 1 -- 172.21.15.161:0/3437954004 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60441498a0 con 0x7f60440c5950 2024-03-28T23:30:42.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.487+0000 7f6049751640 1 --2- 172.21.15.161:0/3437954004 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60440c5950 0x7f60440ab6f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f604002b830 tx=0x7f60400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:42.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.488+0000 7f6039ffb640 1 -- 172.21.15.161:0/3437954004 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6040043070 con 0x7f60440c5950 2024-03-28T23:30:42.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.488+0000 7f6039ffb640 1 -- 172.21.15.161:0/3437954004 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f604003f870 con 0x7f60440c5950 2024-03-28T23:30:42.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.488+0000 7f604b9b5640 1 -- 172.21.15.161:0/3437954004 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60440c4580 con 0x7f60440c5950 2024-03-28T23:30:42.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.488+0000 7f6039ffb640 1 -- 172.21.15.161:0/3437954004 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6040038040 con 0x7f60440c5950 2024-03-28T23:30:42.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.488+0000 7f604b9b5640 1 -- 172.21.15.161:0/3437954004 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60440c4b40 con 0x7f60440c5950 2024-03-28T23:30:42.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.489+0000 7f604b9b5640 1 -- 172.21.15.161:0/3437954004 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f604404f4b0 con 0x7f60440c5950 2024-03-28T23:30:42.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.490+0000 7f6039ffb640 1 -- 172.21.15.161:0/3437954004 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6040046070 con 0x7f60440c5950 2024-03-28T23:30:42.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.490+0000 7f6039ffb640 1 --2- 172.21.15.161:0/3437954004 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f602806b500 0x7f602806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:42.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.490+0000 7f6039ffb640 1 -- 172.21.15.161:0/3437954004 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f60400ac4c0 con 0x7f60440c5950 2024-03-28T23:30:42.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.490+0000 7f6048f50640 1 --2- 172.21.15.161:0/3437954004 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f602806b500 0x7f602806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:42.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.491+0000 7f6048f50640 1 --2- 172.21.15.161:0/3437954004 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f602806b500 0x7f602806d9c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6034006fd0 tx=0x7f6034008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:42.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.495+0000 7f6039ffb640 1 -- 172.21.15.161:0/3437954004 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f604007b060 con 0x7f60440c5950 2024-03-28T23:30:42.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.688+0000 7f604b9b5640 1 -- 172.21.15.161:0/3437954004 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f604414f780 con 0x7f60440c5950 2024-03-28T23:30:42.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.689+0000 7f6039ffb640 1 -- 172.21.15.161:0/3437954004 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f604007a960 con 0x7f60440c5950 2024-03-28T23:30:42.689 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-28T23:30:42.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.690+0000 7f604b9b5640 1 -- 172.21.15.161:0/3437954004 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f602806b500 msgr2=0x7f602806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:42.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.690+0000 7f604b9b5640 1 --2- 172.21.15.161:0/3437954004 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f602806b500 0x7f602806d9c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6034006fd0 tx=0x7f6034008040 comp rx=0 tx=0).stop 2024-03-28T23:30:42.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.691+0000 7f604b9b5640 1 -- 172.21.15.161:0/3437954004 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60440c5950 msgr2=0x7f60440ab6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:42.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.691+0000 7f604b9b5640 1 --2- 172.21.15.161:0/3437954004 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60440c5950 0x7f60440ab6f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f604002b830 tx=0x7f60400046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.691+0000 7f604b9b5640 1 -- 172.21.15.161:0/3437954004 shutdown_connections 2024-03-28T23:30:42.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.691+0000 7f604b9b5640 1 --2- 172.21.15.161:0/3437954004 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f602806b500 0x7f602806d9c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.691+0000 7f604b9b5640 1 --2- 172.21.15.161:0/3437954004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6044152890 0x7f6044149360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.691+0000 7f604b9b5640 1 --2- 172.21.15.161:0/3437954004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60440c8ed0 0x7f6044148e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.691+0000 7f604b9b5640 1 --2- 172.21.15.161:0/3437954004 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60440c5950 0x7f60440ab6f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.691+0000 7f604b9b5640 1 -- 172.21.15.161:0/3437954004 >> 172.21.15.161:0/3437954004 conn(0x7f60440ac620 msgr2=0x7f604406be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:42.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.691+0000 7f604b9b5640 1 -- 172.21.15.161:0/3437954004 shutdown_connections 2024-03-28T23:30:42.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.691+0000 7f604b9b5640 1 -- 172.21.15.161:0/3437954004 wait complete. 2024-03-28T23:30:42.702 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi170.front.sepia.ceph.com... 2024-03-28T23:30:42.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:42.702 DEBUG:teuthology.orchestra.run.smithi170:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-03-28T23:30:42.735 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:30:42.735 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi170.front.sepia.ceph.com 2024-03-28T23:30:42.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:30:42.735 DEBUG:teuthology.orchestra.run.smithi170:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-28T23:30:42.803 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-28T23:30:42.803 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:30:42.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.906+0000 7f5cec2c0640 1 Processor -- start 2024-03-28T23:30:42.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.906+0000 7f5cec2c0640 1 -- start start 2024-03-28T23:30:42.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.907+0000 7f5cec2c0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5ce4152890 0x7f5ce40c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:42.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.907+0000 7f5cec2c0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ce40c8ed0 0x7f5ce40c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:42.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.907+0000 7f5cec2c0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ce4158a30 con 0x7f5ce40c5950 2024-03-28T23:30:42.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.907+0000 7f5cec2c0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f5ce40c8480 con 0x7f5ce4152890 2024-03-28T23:30:42.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.907+0000 7f5cec2c0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5ce4159640 con 0x7f5ce40c8ed0 2024-03-28T23:30:42.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.907+0000 7f5cea85d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f5ce40c5950 0x7f5ce41553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:34976/0 (socket says 172.21.15.161:34976) 2024-03-28T23:30:42.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.907+0000 7f5cea85d640 1 -- 172.21.15.161:0/2228841649 learned_addr learned my addr 172.21.15.161:0/2228841649 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:42.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.908+0000 7f5ce905a640 1 -- 172.21.15.161:0/2228841649 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3617379550 0 0) 0x7f5ce4158a30 con 0x7f5ce40c5950 2024-03-28T23:30:42.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.908+0000 7f5ce905a640 1 -- 172.21.15.161:0/2228841649 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cd0003520 con 0x7f5ce40c5950 2024-03-28T23:30:42.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.908+0000 7f5ce985b640 1 --2- 172.21.15.161:0/2228841649 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ce40c8ed0 0x7f5ce40c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:42.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.908+0000 7f5cea05c640 1 --2- 172.21.15.161:0/2228841649 >> v2:172.21.15.170:3300/0 conn(0x7f5ce4152890 0x7f5ce40c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:42.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.908+0000 7f5ce905a640 1 -- 172.21.15.161:0/2228841649 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 421259945 0 0) 0x7f5cd0003520 con 0x7f5ce40c5950 2024-03-28T23:30:42.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.908+0000 7f5ce905a640 1 -- 172.21.15.161:0/2228841649 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ce40c8ed0 msgr2=0x7f5ce40c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:42.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.908+0000 7f5ce905a640 1 --2- 172.21.15.161:0/2228841649 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ce40c8ed0 0x7f5ce40c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.908+0000 7f5ce905a640 1 -- 172.21.15.161:0/2228841649 >> v2:172.21.15.170:3300/0 conn(0x7f5ce4152890 msgr2=0x7f5ce40c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:42.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.908+0000 7f5ce905a640 1 --2- 172.21.15.161:0/2228841649 >> v2:172.21.15.170:3300/0 conn(0x7f5ce4152890 0x7f5ce40c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.908+0000 7f5ce905a640 1 -- 172.21.15.161:0/2228841649 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce4158d10 con 0x7f5ce40c5950 2024-03-28T23:30:42.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.908+0000 7f5ce905a640 1 -- 172.21.15.161:0/2228841649 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f5cdc0019f0 con 0x7f5ce40c5950 2024-03-28T23:30:42.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.908+0000 7f5ce905a640 1 -- 172.21.15.161:0/2228841649 >> v1:172.21.15.161:6789/0 conn(0x7f5ce40c5950 legacy=0x7f5ce41553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:42.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.908+0000 7f5ce905a640 1 --2- 172.21.15.161:0/2228841649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cd0003a90 0x7f5cd0003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:42.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.908+0000 7f5ce905a640 1 --2- 172.21.15.161:0/2228841649 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ce40c8ed0 0x7f5cd0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:42.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.908+0000 7f5ce905a640 1 --2- 172.21.15.161:0/2228841649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cd0004d40 0x7f5cd00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:42.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.908+0000 7f5ce905a640 1 -- 172.21.15.161:0/2228841649 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce4159640 con 0x7f5cd0003a90 2024-03-28T23:30:42.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.908+0000 7f5ce905a640 1 -- 172.21.15.161:0/2228841649 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd000baa0 con 0x7f5cd0004d40 2024-03-28T23:30:42.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.908+0000 7f5ce905a640 1 -- 172.21.15.161:0/2228841649 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5cd000bca0 con 0x7f5ce40c8ed0 2024-03-28T23:30:42.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.908+0000 7f5cea05c640 1 --2- 172.21.15.161:0/2228841649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cd0003a90 0x7f5cd0003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:42.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.908+0000 7f5ce985b640 1 --2- 172.21.15.161:0/2228841649 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ce40c8ed0 0x7f5cd0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:42.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.908+0000 7f5cea85d640 1 --2- 172.21.15.161:0/2228841649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cd0004d40 0x7f5cd00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:42.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.909+0000 7f5cea05c640 1 -- 172.21.15.161:0/2228841649 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ce40c8ed0 msgr2=0x7f5cd0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:42.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.909+0000 7f5cea05c640 1 --2- 172.21.15.161:0/2228841649 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ce40c8ed0 0x7f5cd0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.909+0000 7f5cea05c640 1 -- 172.21.15.161:0/2228841649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cd0004d40 msgr2=0x7f5cd00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:42.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.909+0000 7f5cea05c640 1 --2- 172.21.15.161:0/2228841649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cd0004d40 0x7f5cd00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.909+0000 7f5cea05c640 1 -- 172.21.15.161:0/2228841649 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cd0003520 con 0x7f5cd0003a90 2024-03-28T23:30:42.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.909+0000 7f5cea05c640 1 --2- 172.21.15.161:0/2228841649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cd0003a90 0x7f5cd0003eb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5ce0009ac0 tx=0x7f5cd0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=303c2009b5c6d405 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:42.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.909+0000 7f5ce905a640 1 -- 172.21.15.161:0/2228841649 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce0039070 con 0x7f5cd0003a90 2024-03-28T23:30:42.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.909+0000 7f5ce905a640 1 -- 172.21.15.161:0/2228841649 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ce002c710 con 0x7f5cd0003a90 2024-03-28T23:30:42.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.909+0000 7f5ce905a640 1 -- 172.21.15.161:0/2228841649 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce0009690 con 0x7f5cd0003a90 2024-03-28T23:30:42.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.910+0000 7f5cec2c0640 1 -- 172.21.15.161:0/2228841649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cd0003a90 msgr2=0x7f5cd0003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:42.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.910+0000 7f5cec2c0640 1 --2- 172.21.15.161:0/2228841649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cd0003a90 0x7f5cd0003eb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5ce0009ac0 tx=0x7f5cd0005830 comp rx=0 tx=0).stop 2024-03-28T23:30:42.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.910+0000 7f5cea85d640 1 -- 172.21.15.161:0/2228841649 reap_dead start 2024-03-28T23:30:42.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.910+0000 7f5cec2c0640 1 -- 172.21.15.161:0/2228841649 shutdown_connections 2024-03-28T23:30:42.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.910+0000 7f5cec2c0640 1 -- 172.21.15.161:0/2228841649 >> 172.21.15.161:0/2228841649 conn(0x7f5ce40ac620 msgr2=0x7f5ce40aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:42.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.910+0000 7f5cec2c0640 1 -- 172.21.15.161:0/2228841649 shutdown_connections 2024-03-28T23:30:42.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.910+0000 7f5cec2c0640 1 -- 172.21.15.161:0/2228841649 wait complete. 2024-03-28T23:30:42.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.911+0000 7f5cec2c0640 1 Processor -- start 2024-03-28T23:30:42.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.911+0000 7f5cec2c0640 1 -- start start 2024-03-28T23:30:42.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.912+0000 7f5cec2c0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cd0003a90 0x7f5ce414f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:42.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.912+0000 7f5cec2c0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ce414d040 0x7f5ce40ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:42.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.912+0000 7f5cec2c0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ce40c0d60 0x7f5ce40c11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:42.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.912+0000 7f5cec2c0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce4152c70 con 0x7f5ce40c0d60 2024-03-28T23:30:42.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.912+0000 7f5cec2c0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce41592d0 con 0x7f5cd0003a90 2024-03-28T23:30:42.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.912+0000 7f5cec2c0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5ce4152df0 con 0x7f5ce414d040 2024-03-28T23:30:42.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.912+0000 7f5cea85d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ce40c0d60 0x7f5ce40c11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:42.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.912+0000 7f5cea85d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ce40c0d60 0x7f5ce40c11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51940/0 (socket says 172.21.15.161:51940) 2024-03-28T23:30:42.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.912+0000 7f5cea85d640 1 -- 172.21.15.161:0/2105123096 learned_addr learned my addr 172.21.15.161:0/2105123096 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:42.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.912+0000 7f5ce985b640 1 --2- 172.21.15.161:0/2105123096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ce414d040 0x7f5ce40ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:42.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.912+0000 7f5cea05c640 1 --2- 172.21.15.161:0/2105123096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cd0003a90 0x7f5ce414f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:42.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.912+0000 7f5cea85d640 1 -- 172.21.15.161:0/2105123096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ce414d040 msgr2=0x7f5ce40ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:42.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.912+0000 7f5cea85d640 1 --2- 172.21.15.161:0/2105123096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ce414d040 0x7f5ce40ab6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.912+0000 7f5cea85d640 1 -- 172.21.15.161:0/2105123096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cd0003a90 msgr2=0x7f5ce414f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:42.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.912+0000 7f5cea85d640 1 --2- 172.21.15.161:0/2105123096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cd0003a90 0x7f5ce414f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:42.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.912+0000 7f5cea85d640 1 -- 172.21.15.161:0/2105123096 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce414cd60 con 0x7f5ce40c0d60 2024-03-28T23:30:42.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.912+0000 7f5cea05c640 1 --2- 172.21.15.161:0/2105123096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cd0003a90 0x7f5ce414f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:42.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.912+0000 7f5cea85d640 1 --2- 172.21.15.161:0/2105123096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ce40c0d60 0x7f5ce40c11e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc0013d0 tx=0x7f5cdc0079d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:42.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.913+0000 7f5cdaffd640 1 -- 172.21.15.161:0/2105123096 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc019070 con 0x7f5ce40c0d60 2024-03-28T23:30:42.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.913+0000 7f5cdaffd640 1 -- 172.21.15.161:0/2105123096 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cdc00aaa0 con 0x7f5ce40c0d60 2024-03-28T23:30:42.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.913+0000 7f5cdaffd640 1 -- 172.21.15.161:0/2105123096 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc0133f0 con 0x7f5ce40c0d60 2024-03-28T23:30:42.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.913+0000 7f5cec2c0640 1 -- 172.21.15.161:0/2105123096 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ce40c47a0 con 0x7f5ce40c0d60 2024-03-28T23:30:42.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.913+0000 7f5cec2c0640 1 -- 172.21.15.161:0/2105123096 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ce40abc30 con 0x7f5ce40c0d60 2024-03-28T23:30:42.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.914+0000 7f5cec2c0640 1 -- 172.21.15.161:0/2105123096 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ce4067a50 con 0x7f5ce40c0d60 2024-03-28T23:30:42.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.914+0000 7f5cdaffd640 1 -- 172.21.15.161:0/2105123096 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5cdc01c070 con 0x7f5ce40c0d60 2024-03-28T23:30:42.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.915+0000 7f5cdaffd640 1 --2- 172.21.15.161:0/2105123096 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5cc806b4d0 0x7f5cc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:42.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.915+0000 7f5cdaffd640 1 -- 172.21.15.161:0/2105123096 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f5cdc08a040 con 0x7f5ce40c0d60 2024-03-28T23:30:42.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.915+0000 7f5cea05c640 1 --2- 172.21.15.161:0/2105123096 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5cc806b4d0 0x7f5cc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:42.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.915+0000 7f5cea05c640 1 --2- 172.21.15.161:0/2105123096 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5cc806b4d0 0x7f5cc806d990 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5ce0036000 tx=0x7f5ce0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:42.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:42.923+0000 7f5cdaffd640 1 -- 172.21.15.161:0/2105123096 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5ce4067a50 con 0x7f5ce40c0d60 2024-03-28T23:30:43.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.130+0000 7f5cec2c0640 1 -- 172.21.15.161:0/2105123096 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ce404f4b0 con 0x7f5ce40c0d60 2024-03-28T23:30:43.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.131+0000 7f5cdaffd640 1 -- 172.21.15.161:0/2105123096 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5189 (secure 0 0 0) 0x7f5cdc0583e0 con 0x7f5ce40c0d60 2024-03-28T23:30:43.131 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:43.132 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/1577284938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":1577284938},{"type":"v1","addr":"172.21.15.170:6835","nonce":1577284938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4258,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/1107796931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1107796931},{"type":"v1","addr":"172.21.15.170:6834","nonce":1107796931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4107992135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4107992135},{"type":"v1","addr":"172.21.15.161:6836","nonce":4107992135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:29:45.903844+0000","modified":"2024-03-28T23:30:07.984387+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6837/3832540545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3832540545},{"type":"v1","addr":"172.21.15.161:6837","nonce":3832540545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4275,"qdb_cluster":[4275]},"id":1}]} 2024-03-28T23:30:43.132 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 9 2024-03-28T23:30:43.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.133+0000 7f5cec2c0640 1 -- 172.21.15.161:0/2105123096 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5cc806b4d0 msgr2=0x7f5cc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:43.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.133+0000 7f5cec2c0640 1 --2- 172.21.15.161:0/2105123096 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5cc806b4d0 0x7f5cc806d990 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5ce0036000 tx=0x7f5ce0030000 comp rx=0 tx=0).stop 2024-03-28T23:30:43.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.133+0000 7f5cec2c0640 1 -- 172.21.15.161:0/2105123096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ce40c0d60 msgr2=0x7f5ce40c11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:43.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.133+0000 7f5cec2c0640 1 --2- 172.21.15.161:0/2105123096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ce40c0d60 0x7f5ce40c11e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc0013d0 tx=0x7f5cdc0079d0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.133+0000 7f5cec2c0640 1 -- 172.21.15.161:0/2105123096 shutdown_connections 2024-03-28T23:30:43.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.133+0000 7f5cec2c0640 1 --2- 172.21.15.161:0/2105123096 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5cc806b4d0 0x7f5cc806d990 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.133+0000 7f5cec2c0640 1 --2- 172.21.15.161:0/2105123096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ce40c0d60 0x7f5ce40c11e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.133+0000 7f5cec2c0640 1 --2- 172.21.15.161:0/2105123096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ce414d040 0x7f5ce40ab6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.133+0000 7f5cec2c0640 1 --2- 172.21.15.161:0/2105123096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cd0003a90 0x7f5ce414f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.133+0000 7f5cec2c0640 1 -- 172.21.15.161:0/2105123096 >> 172.21.15.161:0/2105123096 conn(0x7f5ce40ac620 msgr2=0x7f5ce406be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:43.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.133+0000 7f5cec2c0640 1 -- 172.21.15.161:0/2105123096 shutdown_connections 2024-03-28T23:30:43.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.133+0000 7f5cec2c0640 1 -- 172.21.15.161:0/2105123096 wait complete. 2024-03-28T23:30:43.145 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:30:43.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.247+0000 7f641fd0f640 1 Processor -- start 2024-03-28T23:30:43.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.247+0000 7f641fd0f640 1 -- start start 2024-03-28T23:30:43.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.247+0000 7f641fd0f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6418150600 0x7f64180a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:43.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.247+0000 7f641fd0f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f64180a8fc0 0x7f6418153270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:43.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.247+0000 7f641fd0f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64181567c0 con 0x7f64180aacb0 2024-03-28T23:30:43.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.247+0000 7f641fd0f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6418156570 con 0x7f6418150600 2024-03-28T23:30:43.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.247+0000 7f641fd0f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f6418156aa0 con 0x7f64180a8fc0 2024-03-28T23:30:43.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.248+0000 7f641daab640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6418150600 0x7f64180a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:43.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.248+0000 7f641daab640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6418150600 0x7f64180a9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51948/0 (socket says 172.21.15.161:51948) 2024-03-28T23:30:43.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.248+0000 7f641daab640 1 -- 172.21.15.161:0/3184180453 learned_addr learned my addr 172.21.15.161:0/3184180453 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:43.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.248+0000 7f641d2aa640 1 --2- 172.21.15.161:0/3184180453 >> v2:172.21.15.170:3300/0 conn(0x7f64180a8fc0 0x7f6418153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:43.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.248+0000 7f641daab640 1 -- 172.21.15.161:0/3184180453 >> v2:172.21.15.170:3300/0 conn(0x7f64180a8fc0 msgr2=0x7f6418153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:43.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.248+0000 7f641daab640 1 --2- 172.21.15.161:0/3184180453 >> v2:172.21.15.170:3300/0 conn(0x7f64180a8fc0 0x7f6418153270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.248+0000 7f641daab640 1 -- 172.21.15.161:0/3184180453 >> v1:172.21.15.170:6789/0 conn(0x7f64180aacb0 legacy=0x7f64181537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:43.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.248+0000 7f641daab640 1 -- 172.21.15.161:0/3184180453 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6418156d80 con 0x7f6418150600 2024-03-28T23:30:43.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.249+0000 7f641daab640 1 --2- 172.21.15.161:0/3184180453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6418150600 0x7f64180a9f40 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f641806f570 tx=0x7f641402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62d929b1331d4dcd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:43.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.249+0000 7f641caa9640 1 -- 172.21.15.161:0/3184180453 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6414035020 con 0x7f6418150600 2024-03-28T23:30:43.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.249+0000 7f641caa9640 1 -- 172.21.15.161:0/3184180453 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6414031870 con 0x7f6418150600 2024-03-28T23:30:43.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.249+0000 7f641caa9640 1 -- 172.21.15.161:0/3184180453 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6414031c40 con 0x7f6418150600 2024-03-28T23:30:43.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.249+0000 7f641fd0f640 1 -- 172.21.15.161:0/3184180453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6418150600 msgr2=0x7f64180a9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:43.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.249+0000 7f641fd0f640 1 --2- 172.21.15.161:0/3184180453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6418150600 0x7f64180a9f40 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f641806f570 tx=0x7f641402b910 comp rx=0 tx=0).stop 2024-03-28T23:30:43.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.249+0000 7f641fd0f640 1 -- 172.21.15.161:0/3184180453 shutdown_connections 2024-03-28T23:30:43.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.249+0000 7f641fd0f640 1 --2- 172.21.15.161:0/3184180453 >> v2:172.21.15.170:3300/0 conn(0x7f64180a8fc0 0x7f6418153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.249+0000 7f641fd0f640 1 --2- 172.21.15.161:0/3184180453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6418150600 0x7f64180a9f40 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.249+0000 7f641fd0f640 1 -- 172.21.15.161:0/3184180453 >> 172.21.15.161:0/3184180453 conn(0x7f64180c6670 msgr2=0x7f64180c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:43.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.249+0000 7f641fd0f640 1 -- 172.21.15.161:0/3184180453 shutdown_connections 2024-03-28T23:30:43.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.249+0000 7f641fd0f640 1 -- 172.21.15.161:0/3184180453 wait complete. 2024-03-28T23:30:43.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.251+0000 7f641fd0f640 1 Processor -- start 2024-03-28T23:30:43.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.251+0000 7f641fd0f640 1 -- start start 2024-03-28T23:30:43.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.251+0000 7f641fd0f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64180a8fc0 0x7f64181eb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:43.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.251+0000 7f641fd0f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64180aacb0 0x7f64180c5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:43.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.251+0000 7f641fd0f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6418150600 0x7f641806b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:43.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.251+0000 7f641fd0f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6418153ec0 con 0x7f6418150600 2024-03-28T23:30:43.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.251+0000 7f641fd0f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6418157340 con 0x7f64180a8fc0 2024-03-28T23:30:43.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.251+0000 7f641fd0f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f64181509e0 con 0x7f64180aacb0 2024-03-28T23:30:43.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.251+0000 7f641e2ac640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6418150600 0x7f641806b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:43.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.251+0000 7f641e2ac640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6418150600 0x7f641806b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51964/0 (socket says 172.21.15.161:51964) 2024-03-28T23:30:43.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.251+0000 7f641e2ac640 1 -- 172.21.15.161:0/1679572785 learned_addr learned my addr 172.21.15.161:0/1679572785 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:43.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.252+0000 7f641daab640 1 --2- 172.21.15.161:0/1679572785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64180a8fc0 0x7f64181eb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:43.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.252+0000 7f641d2aa640 1 --2- 172.21.15.161:0/1679572785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64180aacb0 0x7f64180c5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:43.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.252+0000 7f641e2ac640 1 -- 172.21.15.161:0/1679572785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64180aacb0 msgr2=0x7f64180c5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:43.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.252+0000 7f641e2ac640 1 --2- 172.21.15.161:0/1679572785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64180aacb0 0x7f64180c5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.252+0000 7f641e2ac640 1 -- 172.21.15.161:0/1679572785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64180a8fc0 msgr2=0x7f64181eb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:43.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.252+0000 7f641e2ac640 1 --2- 172.21.15.161:0/1679572785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64180a8fc0 0x7f64181eb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.252+0000 7f641e2ac640 1 -- 172.21.15.161:0/1679572785 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f641806c420 con 0x7f6418150600 2024-03-28T23:30:43.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.252+0000 7f641e2ac640 1 --2- 172.21.15.161:0/1679572785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6418150600 0x7f641806b7a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f641000c6b0 tx=0x7f641000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:43.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.252+0000 7f640e7fc640 1 -- 172.21.15.161:0/1679572785 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6410003b70 con 0x7f6418150600 2024-03-28T23:30:43.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.252+0000 7f640e7fc640 1 -- 172.21.15.161:0/1679572785 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6410003d30 con 0x7f6418150600 2024-03-28T23:30:43.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.252+0000 7f641fd0f640 1 -- 172.21.15.161:0/1679572785 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64181eb700 con 0x7f6418150600 2024-03-28T23:30:43.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.252+0000 7f640e7fc640 1 -- 172.21.15.161:0/1679572785 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6410005310 con 0x7f6418150600 2024-03-28T23:30:43.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.252+0000 7f641fd0f640 1 -- 172.21.15.161:0/1679572785 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f641806c9a0 con 0x7f6418150600 2024-03-28T23:30:43.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.253+0000 7f641fd0f640 1 -- 172.21.15.161:0/1679572785 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f641806cb70 con 0x7f6418150600 2024-03-28T23:30:43.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.253+0000 7f640e7fc640 1 -- 172.21.15.161:0/1679572785 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f64100054b0 con 0x7f6418150600 2024-03-28T23:30:43.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.254+0000 7f640e7fc640 1 --2- 172.21.15.161:0/1679572785 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f63fc06b4d0 0x7f63fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:43.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.254+0000 7f640e7fc640 1 -- 172.21.15.161:0/1679572785 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f641008a290 con 0x7f6418150600 2024-03-28T23:30:43.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.254+0000 7f641daab640 1 --2- 172.21.15.161:0/1679572785 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f63fc06b4d0 0x7f63fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:43.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.254+0000 7f641daab640 1 --2- 172.21.15.161:0/1679572785 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f63fc06b4d0 0x7f63fc06d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6414036000 tx=0x7f641403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:43.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.258+0000 7f640e7fc640 1 -- 172.21.15.161:0/1679572785 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6410058750 con 0x7f6418150600 2024-03-28T23:30:43.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.437+0000 7f641fd0f640 1 -- 172.21.15.161:0/1679572785 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f641804f4b0 con 0x7f6418150600 2024-03-28T23:30:43.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.438+0000 7f640e7fc640 1 -- 172.21.15.161:0/1679572785 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18937 (secure 0 0 0) 0x7f641005c7f0 con 0x7f6418150600 2024-03-28T23:30:43.439 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:43.439 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":26,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:30:35.148429+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:30:43.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.440+0000 7f641fd0f640 1 -- 172.21.15.161:0/1679572785 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f63fc06b4d0 msgr2=0x7f63fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:43.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.440+0000 7f641fd0f640 1 --2- 172.21.15.161:0/1679572785 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f63fc06b4d0 0x7f63fc06d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6414036000 tx=0x7f641403c040 comp rx=0 tx=0).stop 2024-03-28T23:30:43.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.440+0000 7f641fd0f640 1 -- 172.21.15.161:0/1679572785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6418150600 msgr2=0x7f641806b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:43.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.440+0000 7f641fd0f640 1 --2- 172.21.15.161:0/1679572785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6418150600 0x7f641806b7a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f641000c6b0 tx=0x7f641000cb70 comp rx=0 tx=0).stop 2024-03-28T23:30:43.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.440+0000 7f641fd0f640 1 -- 172.21.15.161:0/1679572785 shutdown_connections 2024-03-28T23:30:43.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.440+0000 7f641fd0f640 1 --2- 172.21.15.161:0/1679572785 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f63fc06b4d0 0x7f63fc06d990 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.440+0000 7f641fd0f640 1 --2- 172.21.15.161:0/1679572785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6418150600 0x7f641806b7a0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.440+0000 7f641fd0f640 1 --2- 172.21.15.161:0/1679572785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64180aacb0 0x7f64180c5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.440+0000 7f641fd0f640 1 --2- 172.21.15.161:0/1679572785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64180a8fc0 0x7f64181eb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.440+0000 7f641fd0f640 1 -- 172.21.15.161:0/1679572785 >> 172.21.15.161:0/1679572785 conn(0x7f64180c6670 msgr2=0x7f64180c3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:43.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.440+0000 7f641fd0f640 1 -- 172.21.15.161:0/1679572785 shutdown_connections 2024-03-28T23:30:43.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.440+0000 7f641fd0f640 1 -- 172.21.15.161:0/1679572785 wait complete. 2024-03-28T23:30:43.452 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:29:44.021375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:29:44.872102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T23:29:48.591063+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T23:30:43.452 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:30:43.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.557+0000 7f37feaeb640 1 Processor -- start 2024-03-28T23:30:43.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.557+0000 7f37feaeb640 1 -- start start 2024-03-28T23:30:43.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.558+0000 7f37feaeb640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f37f81504f0 0x7f37f8156e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:43.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.558+0000 7f37feaeb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f37f8070cd0 0x7f37f80725d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:43.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.558+0000 7f37feaeb640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37f80c3400 con 0x7f37f8072b10 2024-03-28T23:30:43.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.558+0000 7f37feaeb640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f37f81508d0 con 0x7f37f8070cd0 2024-03-28T23:30:43.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.558+0000 7f37feaeb640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f37f80a8120 con 0x7f37f81504f0 2024-03-28T23:30:43.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.558+0000 7f37f7fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f37f8070cd0 0x7f37f80725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:43.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.558+0000 7f37f7fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f37f8070cd0 0x7f37f80725d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51968/0 (socket says 172.21.15.161:51968) 2024-03-28T23:30:43.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.558+0000 7f37f7fff640 1 -- 172.21.15.161:0/132181446 learned_addr learned my addr 172.21.15.161:0/132181446 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:43.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.558+0000 7f37fc887640 1 --2- 172.21.15.161:0/132181446 >> v2:172.21.15.170:3300/0 conn(0x7f37f81504f0 0x7f37f8156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:43.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.558+0000 7f37f7fff640 1 -- 172.21.15.161:0/132181446 >> v2:172.21.15.170:3300/0 conn(0x7f37f81504f0 msgr2=0x7f37f8156e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:43.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.558+0000 7f37f7fff640 1 --2- 172.21.15.161:0/132181446 >> v2:172.21.15.170:3300/0 conn(0x7f37f81504f0 0x7f37f8156e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.558+0000 7f37f7fff640 1 -- 172.21.15.161:0/132181446 >> v1:172.21.15.170:6789/0 conn(0x7f37f8072b10 legacy=0x7f37f8153010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:43.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.558+0000 7f37f7fff640 1 -- 172.21.15.161:0/132181446 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37f80a8400 con 0x7f37f8070cd0 2024-03-28T23:30:43.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.559+0000 7f37f7fff640 1 --2- 172.21.15.161:0/132181446 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37f8070cd0 0x7f37f80725d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f37f80aea60 tx=0x7f37e402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca25463201b5404b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:43.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.559+0000 7f37f77fe640 1 -- 172.21.15.161:0/132181446 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e402e070 con 0x7f37f8070cd0 2024-03-28T23:30:43.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.559+0000 7f37f77fe640 1 -- 172.21.15.161:0/132181446 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37e4037070 con 0x7f37f8070cd0 2024-03-28T23:30:43.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.559+0000 7f37f77fe640 1 -- 172.21.15.161:0/132181446 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e4031a00 con 0x7f37f8070cd0 2024-03-28T23:30:43.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.560+0000 7f37feaeb640 1 -- 172.21.15.161:0/132181446 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37f8070cd0 msgr2=0x7f37f80725d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:43.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.560+0000 7f37feaeb640 1 --2- 172.21.15.161:0/132181446 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37f8070cd0 0x7f37f80725d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f37f80aea60 tx=0x7f37e402b880 comp rx=0 tx=0).stop 2024-03-28T23:30:43.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.560+0000 7f37feaeb640 1 -- 172.21.15.161:0/132181446 shutdown_connections 2024-03-28T23:30:43.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.560+0000 7f37feaeb640 1 --2- 172.21.15.161:0/132181446 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37f8070cd0 0x7f37f80725d0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.560+0000 7f37feaeb640 1 --2- 172.21.15.161:0/132181446 >> v2:172.21.15.170:3300/0 conn(0x7f37f81504f0 0x7f37f8156e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.560+0000 7f37feaeb640 1 -- 172.21.15.161:0/132181446 >> 172.21.15.161:0/132181446 conn(0x7f37f80c8810 msgr2=0x7f37f80c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:43.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.560+0000 7f37feaeb640 1 -- 172.21.15.161:0/132181446 shutdown_connections 2024-03-28T23:30:43.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.560+0000 7f37feaeb640 1 -- 172.21.15.161:0/132181446 wait complete. 2024-03-28T23:30:43.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.561+0000 7f37feaeb640 1 Processor -- start 2024-03-28T23:30:43.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.561+0000 7f37feaeb640 1 -- start start 2024-03-28T23:30:43.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.562+0000 7f37feaeb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37f8070cd0 0x7f37f814ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:43.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.562+0000 7f37feaeb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37f8072b10 0x7f37f8149a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:43.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.562+0000 7f37feaeb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37f81504f0 0x7f37f8149f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:43.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.562+0000 7f37feaeb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f37f8153720 con 0x7f37f8072b10 2024-03-28T23:30:43.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.562+0000 7f37feaeb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f37f80a89c0 con 0x7f37f81504f0 2024-03-28T23:30:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.562+0000 7f37feaeb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f37f81538a0 con 0x7f37f8070cd0 2024-03-28T23:30:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.562+0000 7f37f7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37f8072b10 0x7f37f8149a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.562+0000 7f37f7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37f8072b10 0x7f37f8149a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51976/0 (socket says 172.21.15.161:51976) 2024-03-28T23:30:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.562+0000 7f37f7fff640 1 -- 172.21.15.161:0/2727661149 learned_addr learned my addr 172.21.15.161:0/2727661149 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.562+0000 7f37fd088640 1 --2- 172.21.15.161:0/2727661149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37f81504f0 0x7f37f8149f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.562+0000 7f37fc887640 1 --2- 172.21.15.161:0/2727661149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37f8070cd0 0x7f37f814ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.562+0000 7f37f7fff640 1 -- 172.21.15.161:0/2727661149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37f8070cd0 msgr2=0x7f37f814ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.562+0000 7f37f7fff640 1 --2- 172.21.15.161:0/2727661149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37f8070cd0 0x7f37f814ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.562+0000 7f37f7fff640 1 -- 172.21.15.161:0/2727661149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37f81504f0 msgr2=0x7f37f8149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.562+0000 7f37f7fff640 1 --2- 172.21.15.161:0/2727661149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37f81504f0 0x7f37f8149f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.562+0000 7f37f7fff640 1 -- 172.21.15.161:0/2727661149 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37f806d3d0 con 0x7f37f8072b10 2024-03-28T23:30:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.563+0000 7f37f7fff640 1 --2- 172.21.15.161:0/2727661149 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37f8072b10 0x7f37f8149a30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f37e402b8c0 tx=0x7f37e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:43.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.563+0000 7f37f57fa640 1 -- 172.21.15.161:0/2727661149 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e4032880 con 0x7f37f8072b10 2024-03-28T23:30:43.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.563+0000 7f37f57fa640 1 -- 172.21.15.161:0/2727661149 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37e4032a40 con 0x7f37f8072b10 2024-03-28T23:30:43.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.563+0000 7f37feaeb640 1 -- 172.21.15.161:0/2727661149 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37f806d6b0 con 0x7f37f8072b10 2024-03-28T23:30:43.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.563+0000 7f37f57fa640 1 -- 172.21.15.161:0/2727661149 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e4032e30 con 0x7f37f8072b10 2024-03-28T23:30:43.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.563+0000 7f37feaeb640 1 -- 172.21.15.161:0/2727661149 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37f8151080 con 0x7f37f8072b10 2024-03-28T23:30:43.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.564+0000 7f37feaeb640 1 -- 172.21.15.161:0/2727661149 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37f804f4b0 con 0x7f37f8072b10 2024-03-28T23:30:43.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.564+0000 7f37f57fa640 1 -- 172.21.15.161:0/2727661149 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f37e404c060 con 0x7f37f8072b10 2024-03-28T23:30:43.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.565+0000 7f37f57fa640 1 --2- 172.21.15.161:0/2727661149 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f37dc06b500 0x7f37dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:43.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.565+0000 7f37f57fa640 1 -- 172.21.15.161:0/2727661149 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f37e40abf50 con 0x7f37f8072b10 2024-03-28T23:30:43.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.565+0000 7f37fc887640 1 --2- 172.21.15.161:0/2727661149 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f37dc06b500 0x7f37dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:43.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.565+0000 7f37fc887640 1 --2- 172.21.15.161:0/2727661149 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f37dc06b500 0x7f37dc06d9c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f37ec006fd0 tx=0x7f37ec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:43.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.569+0000 7f37f57fa640 1 -- 172.21.15.161:0/2727661149 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f37e4040070 con 0x7f37f8072b10 2024-03-28T23:30:43.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.810+0000 7f37feaeb640 1 -- 172.21.15.161:0/2727661149 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f37f8070ab0 con 0x7f37f8072b10 2024-03-28T23:30:43.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.810+0000 7f37f57fa640 1 -- 172.21.15.161:0/2727661149 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f37e407a380 con 0x7f37f8072b10 2024-03-28T23:30:43.811 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:30:43.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.812+0000 7f37feaeb640 1 -- 172.21.15.161:0/2727661149 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f37dc06b500 msgr2=0x7f37dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:43.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.812+0000 7f37feaeb640 1 --2- 172.21.15.161:0/2727661149 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f37dc06b500 0x7f37dc06d9c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f37ec006fd0 tx=0x7f37ec008040 comp rx=0 tx=0).stop 2024-03-28T23:30:43.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.812+0000 7f37feaeb640 1 -- 172.21.15.161:0/2727661149 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37f8072b10 msgr2=0x7f37f8149a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:43.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.812+0000 7f37feaeb640 1 --2- 172.21.15.161:0/2727661149 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37f8072b10 0x7f37f8149a30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f37e402b8c0 tx=0x7f37e40046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.812+0000 7f37feaeb640 1 -- 172.21.15.161:0/2727661149 shutdown_connections 2024-03-28T23:30:43.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.812+0000 7f37feaeb640 1 --2- 172.21.15.161:0/2727661149 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f37dc06b500 0x7f37dc06d9c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.812+0000 7f37feaeb640 1 --2- 172.21.15.161:0/2727661149 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37f8072b10 0x7f37f8149a30 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.812+0000 7f37feaeb640 1 --2- 172.21.15.161:0/2727661149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37f81504f0 0x7f37f8149f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.812+0000 7f37feaeb640 1 --2- 172.21.15.161:0/2727661149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37f8070cd0 0x7f37f814ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.812+0000 7f37feaeb640 1 -- 172.21.15.161:0/2727661149 >> 172.21.15.161:0/2727661149 conn(0x7f37f80c8810 msgr2=0x7f37f814c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:43.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.813+0000 7f37feaeb640 1 -- 172.21.15.161:0/2727661149 shutdown_connections 2024-03-28T23:30:43.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.813+0000 7f37feaeb640 1 -- 172.21.15.161:0/2727661149 wait complete. 2024-03-28T23:30:43.824 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:30:43.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.932+0000 7f830069c640 1 Processor -- start 2024-03-28T23:30:43.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.932+0000 7f830069c640 1 -- start start 2024-03-28T23:30:43.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.933+0000 7f830069c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f82f8150520 0x7f82f80ab280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:43.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.933+0000 7f830069c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f82f80c1040 0x7f82f81535b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:43.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.933+0000 7f830069c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82f80c2f50 con 0x7f82f80a96f0 2024-03-28T23:30:43.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.933+0000 7f830069c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f82f80a9c20 con 0x7f82f80c1040 2024-03-28T23:30:43.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.933+0000 7f830069c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f82f8150900 con 0x7f82f8150520 2024-03-28T23:30:43.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.933+0000 7f82fdc37640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f82f80c1040 0x7f82f81535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:43.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.933+0000 7f82fec39640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f82f80a96f0 0x7f82f8156520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:34990/0 (socket says 172.21.15.161:34990) 2024-03-28T23:30:43.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.933+0000 7f82fec39640 1 -- 172.21.15.161:0/335195165 learned_addr learned my addr 172.21.15.161:0/335195165 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:43.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.933+0000 7f82fe438640 1 --2- 172.21.15.161:0/335195165 >> v2:172.21.15.170:3300/0 conn(0x7f82f8150520 0x7f82f80ab280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:43.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.933+0000 7f82fdc37640 1 -- 172.21.15.161:0/335195165 >> v2:172.21.15.170:3300/0 conn(0x7f82f8150520 msgr2=0x7f82f80ab280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:43.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.933+0000 7f82fdc37640 1 --2- 172.21.15.161:0/335195165 >> v2:172.21.15.170:3300/0 conn(0x7f82f8150520 0x7f82f80ab280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.933+0000 7f82fdc37640 1 -- 172.21.15.161:0/335195165 >> v1:172.21.15.161:6789/0 conn(0x7f82f80a96f0 legacy=0x7f82f8156520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:43.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.933+0000 7f82fd436640 1 -- 172.21.15.161:0/335195165 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 342232902 0 0) 0x7f82f80c2f50 con 0x7f82f80a96f0 2024-03-28T23:30:43.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.933+0000 7f82fdc37640 1 -- 172.21.15.161:0/335195165 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82f81530d0 con 0x7f82f80c1040 2024-03-28T23:30:43.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.934+0000 7f82fdc37640 1 --2- 172.21.15.161:0/335195165 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82f80c1040 0x7f82f81535b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f82f806e9c0 tx=0x7f82ec02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a44afda8dd7d41eb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:43.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.935+0000 7f82fd436640 1 -- 172.21.15.161:0/335195165 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ec0315a0 con 0x7f82f80c1040 2024-03-28T23:30:43.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.935+0000 7f82fd436640 1 -- 172.21.15.161:0/335195165 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82ec037070 con 0x7f82f80c1040 2024-03-28T23:30:43.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.935+0000 7f82fd436640 1 -- 172.21.15.161:0/335195165 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ec031960 con 0x7f82f80c1040 2024-03-28T23:30:43.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.935+0000 7f830069c640 1 -- 172.21.15.161:0/335195165 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82f80c1040 msgr2=0x7f82f81535b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:43.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.935+0000 7f830069c640 1 --2- 172.21.15.161:0/335195165 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82f80c1040 0x7f82f81535b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f82f806e9c0 tx=0x7f82ec02b840 comp rx=0 tx=0).stop 2024-03-28T23:30:43.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.935+0000 7f830069c640 1 -- 172.21.15.161:0/335195165 shutdown_connections 2024-03-28T23:30:43.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.935+0000 7f830069c640 1 --2- 172.21.15.161:0/335195165 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82f80c1040 0x7f82f81535b0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.935+0000 7f830069c640 1 --2- 172.21.15.161:0/335195165 >> v2:172.21.15.170:3300/0 conn(0x7f82f8150520 0x7f82f80ab280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.935+0000 7f830069c640 1 -- 172.21.15.161:0/335195165 >> 172.21.15.161:0/335195165 conn(0x7f82f80a83f0 msgr2=0x7f82f80a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:43.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.935+0000 7f830069c640 1 -- 172.21.15.161:0/335195165 shutdown_connections 2024-03-28T23:30:43.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.935+0000 7f830069c640 1 -- 172.21.15.161:0/335195165 wait complete. 2024-03-28T23:30:43.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.936+0000 7f830069c640 1 Processor -- start 2024-03-28T23:30:43.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.936+0000 7f830069c640 1 -- start start 2024-03-28T23:30:43.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.937+0000 7f830069c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82f80a96f0 0x7f82f80bfce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:43.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.937+0000 7f830069c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82f80c1040 0x7f82f8156670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:43.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.937+0000 7f830069c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82f8150520 0x7f82f8156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:43.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.937+0000 7f830069c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f82f80c0c00 con 0x7f82f80c1040 2024-03-28T23:30:43.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.937+0000 7f830069c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f82f80c2c70 con 0x7f82f80a96f0 2024-03-28T23:30:43.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.937+0000 7f830069c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f82f80c0d80 con 0x7f82f8150520 2024-03-28T23:30:43.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.937+0000 7f82fdc37640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82f80c1040 0x7f82f8156670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:43.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.937+0000 7f82fdc37640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82f80c1040 0x7f82f8156670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:52000/0 (socket says 172.21.15.161:52000) 2024-03-28T23:30:43.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.937+0000 7f82fdc37640 1 -- 172.21.15.161:0/3967678817 learned_addr learned my addr 172.21.15.161:0/3967678817 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:43.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.937+0000 7f82fe438640 1 --2- 172.21.15.161:0/3967678817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82f80a96f0 0x7f82f80bfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:43.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.937+0000 7f82fec39640 1 --2- 172.21.15.161:0/3967678817 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82f8150520 0x7f82f8156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:43.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.937+0000 7f82fdc37640 1 -- 172.21.15.161:0/3967678817 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82f8150520 msgr2=0x7f82f8156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:43.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.937+0000 7f82fdc37640 1 --2- 172.21.15.161:0/3967678817 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82f8150520 0x7f82f8156bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.937+0000 7f82fdc37640 1 -- 172.21.15.161:0/3967678817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82f80a96f0 msgr2=0x7f82f80bfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:43.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.937+0000 7f82fdc37640 1 --2- 172.21.15.161:0/3967678817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82f80a96f0 0x7f82f80bfce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:43.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.937+0000 7f82fdc37640 1 -- 172.21.15.161:0/3967678817 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82f806c990 con 0x7f82f80c1040 2024-03-28T23:30:43.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.938+0000 7f82fdc37640 1 --2- 172.21.15.161:0/3967678817 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82f80c1040 0x7f82f8156670 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f82ec000c00 tx=0x7f82ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:43.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.938+0000 7f82eaffd640 1 -- 172.21.15.161:0/3967678817 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ec032630 con 0x7f82f80c1040 2024-03-28T23:30:43.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.938+0000 7f82eaffd640 1 -- 172.21.15.161:0/3967678817 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82ec0327f0 con 0x7f82f80c1040 2024-03-28T23:30:43.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.938+0000 7f830069c640 1 -- 172.21.15.161:0/3967678817 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82f81570f0 con 0x7f82f80c1040 2024-03-28T23:30:43.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.938+0000 7f82eaffd640 1 -- 172.21.15.161:0/3967678817 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ec032be0 con 0x7f82f80c1040 2024-03-28T23:30:43.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.938+0000 7f830069c640 1 -- 172.21.15.161:0/3967678817 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82f8150f70 con 0x7f82f80c1040 2024-03-28T23:30:43.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.939+0000 7f830069c640 1 -- 172.21.15.161:0/3967678817 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82f8151a30 con 0x7f82f80c1040 2024-03-28T23:30:43.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.939+0000 7f82eaffd640 1 -- 172.21.15.161:0/3967678817 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f82ec04c020 con 0x7f82f80c1040 2024-03-28T23:30:43.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.940+0000 7f82eaffd640 1 --2- 172.21.15.161:0/3967678817 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f82dc06b4d0 0x7f82dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:43.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.940+0000 7f82fe438640 1 --2- 172.21.15.161:0/3967678817 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f82dc06b4d0 0x7f82dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:43.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.940+0000 7f82eaffd640 1 -- 172.21.15.161:0/3967678817 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f82ec02f020 con 0x7f82f80c1040 2024-03-28T23:30:43.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.940+0000 7f82fe438640 1 --2- 172.21.15.161:0/3967678817 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f82dc06b4d0 0x7f82dc06d990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f82f80c07a0 tx=0x7f82f4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:43.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:43.945+0000 7f82eaffd640 1 -- 172.21.15.161:0/3967678817 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f82ec040070 con 0x7f82f80c1040 2024-03-28T23:30:44.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.118+0000 7f830069c640 1 -- 172.21.15.161:0/3967678817 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f82f814dd40 con 0x7f82f80c1040 2024-03-28T23:30:44.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.118+0000 7f82eaffd640 1 -- 172.21.15.161:0/3967678817 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f82f814dd40 con 0x7f82f80c1040 2024-03-28T23:30:44.119 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:30:44.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.120+0000 7f830069c640 1 -- 172.21.15.161:0/3967678817 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f82dc06b4d0 msgr2=0x7f82dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:44.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.120+0000 7f830069c640 1 --2- 172.21.15.161:0/3967678817 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f82dc06b4d0 0x7f82dc06d990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f82f80c07a0 tx=0x7f82f4008040 comp rx=0 tx=0).stop 2024-03-28T23:30:44.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.120+0000 7f830069c640 1 -- 172.21.15.161:0/3967678817 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82f80c1040 msgr2=0x7f82f8156670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:44.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.120+0000 7f830069c640 1 --2- 172.21.15.161:0/3967678817 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82f80c1040 0x7f82f8156670 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f82ec000c00 tx=0x7f82ec0046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.120+0000 7f830069c640 1 -- 172.21.15.161:0/3967678817 shutdown_connections 2024-03-28T23:30:44.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.120+0000 7f830069c640 1 --2- 172.21.15.161:0/3967678817 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f82dc06b4d0 0x7f82dc06d990 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.120+0000 7f830069c640 1 --2- 172.21.15.161:0/3967678817 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82f80c1040 0x7f82f8156670 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.120+0000 7f830069c640 1 --2- 172.21.15.161:0/3967678817 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82f8150520 0x7f82f8156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.120+0000 7f830069c640 1 --2- 172.21.15.161:0/3967678817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82f80a96f0 0x7f82f80bfce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.120+0000 7f830069c640 1 -- 172.21.15.161:0/3967678817 >> 172.21.15.161:0/3967678817 conn(0x7f82f80a83f0 msgr2=0x7f82f8141ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:44.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.120+0000 7f830069c640 1 -- 172.21.15.161:0/3967678817 shutdown_connections 2024-03-28T23:30:44.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.120+0000 7f830069c640 1 -- 172.21.15.161:0/3967678817 wait complete. 2024-03-28T23:30:44.132 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:30:44.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.242+0000 7fede73d0640 1 Processor -- start 2024-03-28T23:30:44.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.242+0000 7fede73d0640 1 -- start start 2024-03-28T23:30:44.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.243+0000 7fede73d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fede01529d0 0x7fede006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:44.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.243+0000 7fede73d0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fede00c88b0 0x7fede00c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:44.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.243+0000 7fede73d0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fede0152db0 con 0x7fede00c6750 2024-03-28T23:30:44.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.243+0000 7fede73d0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fede00c8d60 con 0x7fede00c88b0 2024-03-28T23:30:44.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.243+0000 7fede73d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fede00c7210 con 0x7fede01529d0 2024-03-28T23:30:44.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.243+0000 7fede596d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fede00c88b0 0x7fede00c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:44.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.243+0000 7fede596d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fede00c88b0 0x7fede00c54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:52014/0 (socket says 172.21.15.161:52014) 2024-03-28T23:30:44.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.243+0000 7fede596d640 1 -- 172.21.15.161:0/2269093939 learned_addr learned my addr 172.21.15.161:0/2269093939 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:44.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.243+0000 7fede596d640 1 -- 172.21.15.161:0/2269093939 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fede01529d0 msgr2=0x7fede006bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:44.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.243+0000 7fede596d640 1 --2- 172.21.15.161:0/2269093939 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fede01529d0 0x7fede006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.243+0000 7fede596d640 1 -- 172.21.15.161:0/2269093939 >> v1:172.21.15.170:6789/0 conn(0x7fede00c6750 legacy=0x7fede0155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:44.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.243+0000 7fede596d640 1 -- 172.21.15.161:0/2269093939 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fede00c6c50 con 0x7fede00c88b0 2024-03-28T23:30:44.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.244+0000 7fede596d640 1 --2- 172.21.15.161:0/2269093939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fede00c88b0 0x7fede00c54f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fedd8009e50 tx=0x7fedd802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=358a51e641e55a59 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:44.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.244+0000 7fedd7fff640 1 -- 172.21.15.161:0/2269093939 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd802ca70 con 0x7fede00c88b0 2024-03-28T23:30:44.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.244+0000 7fedd7fff640 1 -- 172.21.15.161:0/2269093939 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedd802cc10 con 0x7fede00c88b0 2024-03-28T23:30:44.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.244+0000 7fedd7fff640 1 -- 172.21.15.161:0/2269093939 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd80343f0 con 0x7fede00c88b0 2024-03-28T23:30:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.244+0000 7fede73d0640 1 -- 172.21.15.161:0/2269093939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fede00c88b0 msgr2=0x7fede00c54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.244+0000 7fede73d0640 1 --2- 172.21.15.161:0/2269093939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fede00c88b0 0x7fede00c54f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fedd8009e50 tx=0x7fedd802bed0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.244+0000 7fede73d0640 1 -- 172.21.15.161:0/2269093939 shutdown_connections 2024-03-28T23:30:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.244+0000 7fede73d0640 1 --2- 172.21.15.161:0/2269093939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fede00c88b0 0x7fede00c54f0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.244+0000 7fede73d0640 1 --2- 172.21.15.161:0/2269093939 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fede01529d0 0x7fede006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.244+0000 7fede73d0640 1 -- 172.21.15.161:0/2269093939 >> 172.21.15.161:0/2269093939 conn(0x7fede00c7430 msgr2=0x7fede00c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.245+0000 7fede73d0640 1 -- 172.21.15.161:0/2269093939 shutdown_connections 2024-03-28T23:30:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.245+0000 7fede73d0640 1 -- 172.21.15.161:0/2269093939 wait complete. 2024-03-28T23:30:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.246+0000 7fede73d0640 1 Processor -- start 2024-03-28T23:30:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.246+0000 7fede73d0640 1 -- start start 2024-03-28T23:30:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.246+0000 7fede73d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fede00c6750 0x7fede00c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.246+0000 7fede73d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fede00c88b0 0x7fede01505b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.246+0000 7fede73d0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fede01529d0 0x7fede0150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.246+0000 7fede73d0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fede0155710 con 0x7fede01529d0 2024-03-28T23:30:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.246+0000 7fede73d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fede0155430 con 0x7fede00c6750 2024-03-28T23:30:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.246+0000 7fede73d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fede00aa160 con 0x7fede00c88b0 2024-03-28T23:30:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.247+0000 7fede596d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fede01529d0 0x7fede0150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.247+0000 7fede596d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fede01529d0 0x7fede0150af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:52020/0 (socket says 172.21.15.161:52020) 2024-03-28T23:30:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.247+0000 7fede596d640 1 -- 172.21.15.161:0/964837806 learned_addr learned my addr 172.21.15.161:0/964837806 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:44.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.247+0000 7fede596d640 1 -- 172.21.15.161:0/964837806 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fede00c88b0 msgr2=0x7fede01505b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:44.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.247+0000 7fede596d640 1 --2- 172.21.15.161:0/964837806 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fede00c88b0 0x7fede01505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.247+0000 7fede596d640 1 -- 172.21.15.161:0/964837806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fede00c6750 msgr2=0x7fede00c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:44.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.247+0000 7fede596d640 1 --2- 172.21.15.161:0/964837806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fede00c6750 0x7fede00c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.247+0000 7fede596d640 1 -- 172.21.15.161:0/964837806 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fede0154a90 con 0x7fede01529d0 2024-03-28T23:30:44.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.247+0000 7fede596d640 1 --2- 172.21.15.161:0/964837806 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fede01529d0 0x7fede0150af0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fedd8009e50 tx=0x7fedd8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:44.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.247+0000 7fedd5ffb640 1 -- 172.21.15.161:0/964837806 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd8043020 con 0x7fede01529d0 2024-03-28T23:30:44.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.247+0000 7fedd5ffb640 1 -- 172.21.15.161:0/964837806 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedd803e450 con 0x7fede01529d0 2024-03-28T23:30:44.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.247+0000 7fede73d0640 1 -- 172.21.15.161:0/964837806 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fede0151cb0 con 0x7fede01529d0 2024-03-28T23:30:44.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.247+0000 7fedd5ffb640 1 -- 172.21.15.161:0/964837806 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd803e820 con 0x7fede01529d0 2024-03-28T23:30:44.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.247+0000 7fede73d0640 1 -- 172.21.15.161:0/964837806 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fede0154d50 con 0x7fede01529d0 2024-03-28T23:30:44.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.248+0000 7fede73d0640 1 -- 172.21.15.161:0/964837806 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fede01501f0 con 0x7fede01529d0 2024-03-28T23:30:44.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.249+0000 7fedd5ffb640 1 -- 172.21.15.161:0/964837806 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fedd8046020 con 0x7fede01529d0 2024-03-28T23:30:44.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.249+0000 7fedd5ffb640 1 --2- 172.21.15.161:0/964837806 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fedc406b500 0x7fedc406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:44.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.249+0000 7fedd5ffb640 1 -- 172.21.15.161:0/964837806 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fedd80acba0 con 0x7fede01529d0 2024-03-28T23:30:44.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.249+0000 7fede516c640 1 --2- 172.21.15.161:0/964837806 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fedc406b500 0x7fedc406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:44.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.250+0000 7fede516c640 1 --2- 172.21.15.161:0/964837806 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fedc406b500 0x7fedc406d9c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7feddc009670 tx=0x7feddc006bf0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:44.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.254+0000 7fedd5ffb640 1 -- 172.21.15.161:0/964837806 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fedd804d070 con 0x7fede01529d0 2024-03-28T23:30:44.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.509+0000 7fede73d0640 1 -- 172.21.15.161:0/964837806 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fede004f4b0 con 0x7fede01529d0 2024-03-28T23:30:44.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.510+0000 7fedd5ffb640 1 -- 172.21.15.161:0/964837806 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fedd807b6b0 con 0x7fede01529d0 2024-03-28T23:30:44.510 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:30:44.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.511+0000 7fede73d0640 1 -- 172.21.15.161:0/964837806 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fedc406b500 msgr2=0x7fedc406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:44.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.511+0000 7fede73d0640 1 --2- 172.21.15.161:0/964837806 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fedc406b500 0x7fedc406d9c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7feddc009670 tx=0x7feddc006bf0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.511+0000 7fede73d0640 1 -- 172.21.15.161:0/964837806 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fede01529d0 msgr2=0x7fede0150af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:44.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.511+0000 7fede73d0640 1 --2- 172.21.15.161:0/964837806 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fede01529d0 0x7fede0150af0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fedd8009e50 tx=0x7fedd8004a40 comp rx=0 tx=0).stop 2024-03-28T23:30:44.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.512+0000 7fede73d0640 1 -- 172.21.15.161:0/964837806 shutdown_connections 2024-03-28T23:30:44.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.512+0000 7fede73d0640 1 --2- 172.21.15.161:0/964837806 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fedc406b500 0x7fedc406d9c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.512+0000 7fede73d0640 1 --2- 172.21.15.161:0/964837806 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fede01529d0 0x7fede0150af0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.512+0000 7fede73d0640 1 --2- 172.21.15.161:0/964837806 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fede00c88b0 0x7fede01505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.512+0000 7fede73d0640 1 --2- 172.21.15.161:0/964837806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fede00c6750 0x7fede00c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.512+0000 7fede73d0640 1 -- 172.21.15.161:0/964837806 >> 172.21.15.161:0/964837806 conn(0x7fede00c7430 msgr2=0x7fede00c1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:44.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.512+0000 7fede73d0640 1 -- 172.21.15.161:0/964837806 shutdown_connections 2024-03-28T23:30:44.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.512+0000 7fede73d0640 1 -- 172.21.15.161:0/964837806 wait complete. 2024-03-28T23:30:44.523 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T23:30:44.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.643+0000 7f7ad095e640 1 Processor -- start 2024-03-28T23:30:44.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.643+0000 7f7ad095e640 1 -- start start 2024-03-28T23:30:44.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.644+0000 7f7ad095e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7acc0c7330 0x7f7acc14e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:44.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.644+0000 7f7ad095e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7acc0abc00 con 0x7f7acc0733e0 2024-03-28T23:30:44.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.644+0000 7f7ad095e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7acc06bf70 con 0x7f7acc0a9960 2024-03-28T23:30:44.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.644+0000 7f7ad095e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f7acc0aa3a0 con 0x7f7acc0c7330 2024-03-28T23:30:44.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.644+0000 7f7acad9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f7acc0733e0 0x7f7acc14e9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:34994/0 (socket says 172.21.15.161:34994) 2024-03-28T23:30:44.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.644+0000 7f7acad9d640 1 -- 172.21.15.161:0/2004160724 learned_addr learned my addr 172.21.15.161:0/2004160724 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:44.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.644+0000 7f7ac3fff640 1 --2- 172.21.15.161:0/2004160724 >> v2:172.21.15.170:3300/0 conn(0x7f7acc0c7330 0x7f7acc14e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:44.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.644+0000 7f7ac9d9b640 1 -- 172.21.15.161:0/2004160724 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 967311797 0 0) 0x7f7acc0abc00 con 0x7f7acc0733e0 2024-03-28T23:30:44.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.644+0000 7f7ac9d9b640 1 -- 172.21.15.161:0/2004160724 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ab8003520 con 0x7f7acc0733e0 2024-03-28T23:30:44.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.644+0000 7f7ac3fff640 1 -- 172.21.15.161:0/2004160724 >> v1:172.21.15.170:6789/0 conn(0x7f7acc0a9960 legacy=0x7f7acc072970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:44.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.644+0000 7f7ac3fff640 1 -- 172.21.15.161:0/2004160724 >> v1:172.21.15.161:6789/0 conn(0x7f7acc0733e0 legacy=0x7f7acc14e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:44.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.644+0000 7f7ac3fff640 1 -- 172.21.15.161:0/2004160724 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7acc06bcd0 con 0x7f7acc0c7330 2024-03-28T23:30:44.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.646+0000 7f7ac3fff640 1 --2- 172.21.15.161:0/2004160724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7acc0c7330 0x7f7acc14e460 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7ab400a940 tx=0x7f7ab402cd00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30a6adbadb159e42 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:44.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.646+0000 7f7ac9d9b640 1 -- 172.21.15.161:0/2004160724 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab403a070 con 0x7f7acc0c7330 2024-03-28T23:30:44.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.646+0000 7f7ac9d9b640 1 -- 172.21.15.161:0/2004160724 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ab402dab0 con 0x7f7acc0c7330 2024-03-28T23:30:44.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.646+0000 7f7ac9d9b640 1 -- 172.21.15.161:0/2004160724 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab400b040 con 0x7f7acc0c7330 2024-03-28T23:30:44.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.646+0000 7f7ad095e640 1 -- 172.21.15.161:0/2004160724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7acc0c7330 msgr2=0x7f7acc14e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:44.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.646+0000 7f7ad095e640 1 --2- 172.21.15.161:0/2004160724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7acc0c7330 0x7f7acc14e460 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7ab400a940 tx=0x7f7ab402cd00 comp rx=0 tx=0).stop 2024-03-28T23:30:44.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.646+0000 7f7ad095e640 1 -- 172.21.15.161:0/2004160724 shutdown_connections 2024-03-28T23:30:44.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.646+0000 7f7ad095e640 1 --2- 172.21.15.161:0/2004160724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7acc0c7330 0x7f7acc14e460 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.646+0000 7f7ad095e640 1 -- 172.21.15.161:0/2004160724 >> 172.21.15.161:0/2004160724 conn(0x7f7acc0a83f0 msgr2=0x7f7acc0a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:44.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.646+0000 7f7ad095e640 1 -- 172.21.15.161:0/2004160724 shutdown_connections 2024-03-28T23:30:44.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.646+0000 7f7ad095e640 1 -- 172.21.15.161:0/2004160724 wait complete. 2024-03-28T23:30:44.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.648+0000 7f7ad095e640 1 Processor -- start 2024-03-28T23:30:44.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.648+0000 7f7ad095e640 1 -- start start 2024-03-28T23:30:44.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.648+0000 7f7ad095e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7acc0733e0 0x7f7acc0a77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:44.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.648+0000 7f7ad095e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7acc0a9960 0x7f7acc0a6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:44.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.648+0000 7f7ad095e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7acc0c7330 0x7f7acc0a6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:44.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.648+0000 7f7ad095e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7acc0ab820 con 0x7f7acc0a9960 2024-03-28T23:30:44.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.648+0000 7f7ad095e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7acc152250 con 0x7f7acc0c7330 2024-03-28T23:30:44.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.648+0000 7f7ad095e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7acc0ab9a0 con 0x7f7acc0733e0 2024-03-28T23:30:44.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.648+0000 7f7ac3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7acc0a9960 0x7f7acc0a6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:44.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.648+0000 7f7ac3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7acc0a9960 0x7f7acc0a6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:52026/0 (socket says 172.21.15.161:52026) 2024-03-28T23:30:44.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.648+0000 7f7ac3fff640 1 -- 172.21.15.161:0/2169337638 learned_addr learned my addr 172.21.15.161:0/2169337638 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:44.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.649+0000 7f7acad9d640 1 --2- 172.21.15.161:0/2169337638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7acc0c7330 0x7f7acc0a6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:44.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.649+0000 7f7aca59c640 1 --2- 172.21.15.161:0/2169337638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7acc0733e0 0x7f7acc0a77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:44.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.649+0000 7f7ac3fff640 1 -- 172.21.15.161:0/2169337638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7acc0733e0 msgr2=0x7f7acc0a77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:44.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.649+0000 7f7ac3fff640 1 --2- 172.21.15.161:0/2169337638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7acc0733e0 0x7f7acc0a77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.649+0000 7f7ac3fff640 1 -- 172.21.15.161:0/2169337638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7acc0c7330 msgr2=0x7f7acc0a6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:44.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.649+0000 7f7ac3fff640 1 --2- 172.21.15.161:0/2169337638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7acc0c7330 0x7f7acc0a6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.649+0000 7f7ac3fff640 1 -- 172.21.15.161:0/2169337638 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7acc1e9570 con 0x7f7acc0a9960 2024-03-28T23:30:44.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.649+0000 7f7acad9d640 1 --2- 172.21.15.161:0/2169337638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7acc0c7330 0x7f7acc0a6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:44.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.649+0000 7f7aca59c640 1 --2- 172.21.15.161:0/2169337638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7acc0733e0 0x7f7acc0a77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:44.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.649+0000 7f7ac3fff640 1 --2- 172.21.15.161:0/2169337638 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7acc0a9960 0x7f7acc0a6760 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4000c00 tx=0x7f7ab40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:44.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.649+0000 7f7ac2ffd640 1 -- 172.21.15.161:0/2169337638 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab4045070 con 0x7f7acc0a9960 2024-03-28T23:30:44.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.649+0000 7f7ac2ffd640 1 -- 172.21.15.161:0/2169337638 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ab4033550 con 0x7f7acc0a9960 2024-03-28T23:30:44.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.649+0000 7f7ad095e640 1 -- 172.21.15.161:0/2169337638 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7acc0be070 con 0x7f7acc0a9960 2024-03-28T23:30:44.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.649+0000 7f7ac2ffd640 1 -- 172.21.15.161:0/2169337638 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab403a070 con 0x7f7acc0a9960 2024-03-28T23:30:44.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.649+0000 7f7ad095e640 1 -- 172.21.15.161:0/2169337638 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7acc0be630 con 0x7f7acc0a9960 2024-03-28T23:30:44.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.650+0000 7f7ad095e640 1 -- 172.21.15.161:0/2169337638 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7acc04f4b0 con 0x7f7acc0a9960 2024-03-28T23:30:44.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.651+0000 7f7ac2ffd640 1 -- 172.21.15.161:0/2169337638 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7ab4048070 con 0x7f7acc0a9960 2024-03-28T23:30:44.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.651+0000 7f7ac2ffd640 1 --2- 172.21.15.161:0/2169337638 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7aac06b500 0x7f7aac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:44.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.651+0000 7f7ac2ffd640 1 -- 172.21.15.161:0/2169337638 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f7ab40ac050 con 0x7f7acc0a9960 2024-03-28T23:30:44.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.651+0000 7f7aca59c640 1 --2- 172.21.15.161:0/2169337638 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7aac06b500 0x7f7aac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:44.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.652+0000 7f7aca59c640 1 --2- 172.21.15.161:0/2169337638 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7aac06b500 0x7f7aac06d9c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7acc1e6430 tx=0x7f7abc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:44.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.656+0000 7f7ac2ffd640 1 -- 172.21.15.161:0/2169337638 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7ab407ab60 con 0x7f7acc0a9960 2024-03-28T23:30:44.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.832+0000 7f7ad095e640 1 -- 172.21.15.161:0/2169337638 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f7acc06b1b0 con 0x7f7acc0a9960 2024-03-28T23:30:44.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.832+0000 7f7ac2ffd640 1 -- 172.21.15.161:0/2169337638 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f7ab407a460 con 0x7f7acc0a9960 2024-03-28T23:30:44.833 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:30:44.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.834+0000 7f7ad095e640 1 -- 172.21.15.161:0/2169337638 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7aac06b500 msgr2=0x7f7aac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:44.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.834+0000 7f7ad095e640 1 --2- 172.21.15.161:0/2169337638 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7aac06b500 0x7f7aac06d9c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7acc1e6430 tx=0x7f7abc009040 comp rx=0 tx=0).stop 2024-03-28T23:30:44.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.834+0000 7f7ad095e640 1 -- 172.21.15.161:0/2169337638 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7acc0a9960 msgr2=0x7f7acc0a6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:44.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.834+0000 7f7ad095e640 1 --2- 172.21.15.161:0/2169337638 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7acc0a9960 0x7f7acc0a6760 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4000c00 tx=0x7f7ab40046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.835+0000 7f7ad095e640 1 -- 172.21.15.161:0/2169337638 shutdown_connections 2024-03-28T23:30:44.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.835+0000 7f7ad095e640 1 --2- 172.21.15.161:0/2169337638 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7aac06b500 0x7f7aac06d9c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.835+0000 7f7ad095e640 1 --2- 172.21.15.161:0/2169337638 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7acc0a9960 0x7f7acc0a6760 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.835+0000 7f7ad095e640 1 --2- 172.21.15.161:0/2169337638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7acc0c7330 0x7f7acc0a6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.835+0000 7f7ad095e640 1 --2- 172.21.15.161:0/2169337638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7acc0733e0 0x7f7acc0a77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.835+0000 7f7ad095e640 1 -- 172.21.15.161:0/2169337638 >> 172.21.15.161:0/2169337638 conn(0x7f7acc0a83f0 msgr2=0x7f7acc070470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:44.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.835+0000 7f7ad095e640 1 -- 172.21.15.161:0/2169337638 shutdown_connections 2024-03-28T23:30:44.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.835+0000 7f7ad095e640 1 -- 172.21.15.161:0/2169337638 wait complete. 2024-03-28T23:30:44.850 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:30:44.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.951+0000 7fd3bd69a640 1 Processor -- start 2024-03-28T23:30:44.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.951+0000 7fd3bd69a640 1 -- start start 2024-03-28T23:30:44.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.951+0000 7fd3bd69a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3b8152890 0x7fd3b80c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:44.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.951+0000 7fd3bd69a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3b8158a30 con 0x7fd3b80c8ed0 2024-03-28T23:30:44.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.951+0000 7fd3bd69a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3b8158d10 con 0x7fd3b80c5b20 2024-03-28T23:30:44.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.951+0000 7fd3bd69a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd3b80c8480 con 0x7fd3b8152890 2024-03-28T23:30:44.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.951+0000 7fd3b67fc640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd3b80c8ed0 0x7fd3b80c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:35006/0 (socket says 172.21.15.161:35006) 2024-03-28T23:30:44.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.951+0000 7fd3b67fc640 1 -- 172.21.15.161:0/4140499619 learned_addr learned my addr 172.21.15.161:0/4140499619 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:44.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.952+0000 7fd3b6ffd640 1 --2- 172.21.15.161:0/4140499619 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3b8152890 0x7fd3b80c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:44.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.952+0000 7fd3b5ffb640 1 -- 172.21.15.161:0/4140499619 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 870948266 0 0) 0x7fd3b8158a30 con 0x7fd3b80c8ed0 2024-03-28T23:30:44.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.952+0000 7fd3b5ffb640 1 -- 172.21.15.161:0/4140499619 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3a0003520 con 0x7fd3b80c8ed0 2024-03-28T23:30:44.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.952+0000 7fd3b5ffb640 1 -- 172.21.15.161:0/4140499619 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1486155712 0 0) 0x7fd3a0003520 con 0x7fd3b80c8ed0 2024-03-28T23:30:44.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.952+0000 7fd3b5ffb640 1 -- 172.21.15.161:0/4140499619 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3b8152890 msgr2=0x7fd3b80c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:44.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.952+0000 7fd3b5ffb640 1 --2- 172.21.15.161:0/4140499619 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3b8152890 0x7fd3b80c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.952+0000 7fd3b5ffb640 1 -- 172.21.15.161:0/4140499619 >> v1:172.21.15.170:6789/0 conn(0x7fd3b80c5b20 legacy=0x7fd3b81553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:44.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.952+0000 7fd3b5ffb640 1 -- 172.21.15.161:0/4140499619 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b8158ff0 con 0x7fd3b80c8ed0 2024-03-28T23:30:44.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.952+0000 7fd3b5ffb640 1 -- 172.21.15.161:0/4140499619 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fd3ac0017f0 con 0x7fd3b80c8ed0 2024-03-28T23:30:44.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.952+0000 7fd3b5ffb640 1 -- 172.21.15.161:0/4140499619 >> v1:172.21.15.161:6789/0 conn(0x7fd3b80c8ed0 legacy=0x7fd3b80c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:44.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.952+0000 7fd3b5ffb640 1 --2- 172.21.15.161:0/4140499619 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3a0003ac0 0x7fd3a0003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:44.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.952+0000 7fd3b5ffb640 1 --2- 172.21.15.161:0/4140499619 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3b8152890 0x7fd3a0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:44.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.952+0000 7fd3b5ffb640 1 --2- 172.21.15.161:0/4140499619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3a0004d40 0x7fd3a00051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:44.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.952+0000 7fd3b5ffb640 1 -- 172.21.15.161:0/4140499619 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a000bba0 con 0x7fd3a0003ac0 2024-03-28T23:30:44.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.952+0000 7fd3b5ffb640 1 -- 172.21.15.161:0/4140499619 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a000bd50 con 0x7fd3a0004d40 2024-03-28T23:30:44.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.952+0000 7fd3b5ffb640 1 -- 172.21.15.161:0/4140499619 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd3a000bf50 con 0x7fd3b8152890 2024-03-28T23:30:44.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.952+0000 7fd3b6ffd640 1 --2- 172.21.15.161:0/4140499619 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3a0003ac0 0x7fd3a0003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:44.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.952+0000 7fd3b67fc640 1 --2- 172.21.15.161:0/4140499619 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3b8152890 0x7fd3a0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:44.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.952+0000 7fd3b77fe640 1 --2- 172.21.15.161:0/4140499619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3a0004d40 0x7fd3a00051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:44.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.953+0000 7fd3b6ffd640 1 -- 172.21.15.161:0/4140499619 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3b8152890 msgr2=0x7fd3a0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:44.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.953+0000 7fd3b6ffd640 1 --2- 172.21.15.161:0/4140499619 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3b8152890 0x7fd3a0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.953+0000 7fd3b6ffd640 1 -- 172.21.15.161:0/4140499619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3a0004d40 msgr2=0x7fd3a00051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:44.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.953+0000 7fd3b6ffd640 1 --2- 172.21.15.161:0/4140499619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3a0004d40 0x7fd3a00051a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.953+0000 7fd3b6ffd640 1 -- 172.21.15.161:0/4140499619 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3a0003520 con 0x7fd3a0003ac0 2024-03-28T23:30:44.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.953+0000 7fd3b6ffd640 1 --2- 172.21.15.161:0/4140499619 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3a0003ac0 0x7fd3a0003ea0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8009ac0 tx=0x7fd3a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76d93a6e83b33198 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:44.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.953+0000 7fd3b5ffb640 1 -- 172.21.15.161:0/4140499619 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8039070 con 0x7fd3a0003ac0 2024-03-28T23:30:44.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.953+0000 7fd3b5ffb640 1 -- 172.21.15.161:0/4140499619 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3a802c6b0 con 0x7fd3a0003ac0 2024-03-28T23:30:44.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.953+0000 7fd3b5ffb640 1 -- 172.21.15.161:0/4140499619 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8009690 con 0x7fd3a0003ac0 2024-03-28T23:30:44.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.954+0000 7fd3bd69a640 1 -- 172.21.15.161:0/4140499619 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3a0003ac0 msgr2=0x7fd3a0003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:44.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.954+0000 7fd3bd69a640 1 --2- 172.21.15.161:0/4140499619 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3a0003ac0 0x7fd3a0003ea0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8009ac0 tx=0x7fd3a0005830 comp rx=0 tx=0).stop 2024-03-28T23:30:44.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.954+0000 7fd3b77fe640 1 -- 172.21.15.161:0/4140499619 reap_dead start 2024-03-28T23:30:44.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.954+0000 7fd3bd69a640 1 -- 172.21.15.161:0/4140499619 shutdown_connections 2024-03-28T23:30:44.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.954+0000 7fd3bd69a640 1 -- 172.21.15.161:0/4140499619 >> 172.21.15.161:0/4140499619 conn(0x7fd3b80ac620 msgr2=0x7fd3b80aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:44.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.954+0000 7fd3bd69a640 1 -- 172.21.15.161:0/4140499619 shutdown_connections 2024-03-28T23:30:44.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.954+0000 7fd3bd69a640 1 -- 172.21.15.161:0/4140499619 wait complete. 2024-03-28T23:30:44.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.956+0000 7fd3bd69a640 1 Processor -- start 2024-03-28T23:30:44.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.956+0000 7fd3bd69a640 1 -- start start 2024-03-28T23:30:44.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.957+0000 7fd3bd69a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3b81578b0 0x7fd3b80c20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:44.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.957+0000 7fd3bd69a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3b814f8f0 0x7fd3b80c0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:44.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.957+0000 7fd3bd69a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3b80c1530 0x7fd3b81eabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:44.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.957+0000 7fd3bd69a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b8152c70 con 0x7fd3b81578b0 2024-03-28T23:30:44.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.957+0000 7fd3bd69a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b8159640 con 0x7fd3b80c1530 2024-03-28T23:30:44.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.957+0000 7fd3bd69a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd3b8152df0 con 0x7fd3b814f8f0 2024-03-28T23:30:44.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.957+0000 7fd3b6ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3b81578b0 0x7fd3b80c20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:44.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.957+0000 7fd3b6ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3b81578b0 0x7fd3b80c20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:52040/0 (socket says 172.21.15.161:52040) 2024-03-28T23:30:44.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.957+0000 7fd3b6ffd640 1 -- 172.21.15.161:0/802450370 learned_addr learned my addr 172.21.15.161:0/802450370 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:44.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.957+0000 7fd3b77fe640 1 --2- 172.21.15.161:0/802450370 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3b80c1530 0x7fd3b81eabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:44.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.957+0000 7fd3b67fc640 1 --2- 172.21.15.161:0/802450370 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3b814f8f0 0x7fd3b80c0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:44.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.957+0000 7fd3b6ffd640 1 -- 172.21.15.161:0/802450370 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3b814f8f0 msgr2=0x7fd3b80c0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:44.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.957+0000 7fd3b6ffd640 1 --2- 172.21.15.161:0/802450370 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3b814f8f0 0x7fd3b80c0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.957+0000 7fd3b6ffd640 1 -- 172.21.15.161:0/802450370 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3b80c1530 msgr2=0x7fd3b81eabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:44.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.957+0000 7fd3b6ffd640 1 --2- 172.21.15.161:0/802450370 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3b80c1530 0x7fd3b81eabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:44.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.957+0000 7fd3b6ffd640 1 -- 172.21.15.161:0/802450370 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b8156380 con 0x7fd3b81578b0 2024-03-28T23:30:44.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.957+0000 7fd3b77fe640 1 --2- 172.21.15.161:0/802450370 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3b80c1530 0x7fd3b81eabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:44.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.957+0000 7fd3b67fc640 1 --2- 172.21.15.161:0/802450370 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3b814f8f0 0x7fd3b80c0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:44.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.958+0000 7fd3b6ffd640 1 --2- 172.21.15.161:0/802450370 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3b81578b0 0x7fd3b80c20e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd3a802c2a0 tx=0x7fd3a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:44.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.958+0000 7fd39f7fe640 1 -- 172.21.15.161:0/802450370 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8039070 con 0x7fd3b81578b0 2024-03-28T23:30:44.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.958+0000 7fd39f7fe640 1 -- 172.21.15.161:0/802450370 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3a803a040 con 0x7fd3b81578b0 2024-03-28T23:30:44.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.958+0000 7fd3bd69a640 1 -- 172.21.15.161:0/802450370 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3b80c4810 con 0x7fd3b81578b0 2024-03-28T23:30:44.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.958+0000 7fd39f7fe640 1 -- 172.21.15.161:0/802450370 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a80044e0 con 0x7fd3b81578b0 2024-03-28T23:30:44.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.958+0000 7fd3bd69a640 1 -- 172.21.15.161:0/802450370 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3b80c4ab0 con 0x7fd3b81578b0 2024-03-28T23:30:44.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.959+0000 7fd3bd69a640 1 -- 172.21.15.161:0/802450370 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3b80ac440 con 0x7fd3b81578b0 2024-03-28T23:30:44.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.959+0000 7fd39f7fe640 1 -- 172.21.15.161:0/802450370 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd3a8045030 con 0x7fd3b81578b0 2024-03-28T23:30:44.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.960+0000 7fd39f7fe640 1 --2- 172.21.15.161:0/802450370 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd3a406b500 0x7fd3a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:44.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.960+0000 7fd3b67fc640 1 --2- 172.21.15.161:0/802450370 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd3a406b500 0x7fd3a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:44.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.960+0000 7fd39f7fe640 1 -- 172.21.15.161:0/802450370 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fd3a80ab1c0 con 0x7fd3b81578b0 2024-03-28T23:30:44.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.960+0000 7fd3b67fc640 1 --2- 172.21.15.161:0/802450370 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd3a406b500 0x7fd3a406d9c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac001380 tx=0x7fd3ac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:44.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:44.964+0000 7fd39f7fe640 1 -- 172.21.15.161:0/802450370 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd3b80ac440 con 0x7fd3b81578b0 2024-03-28T23:30:45.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.300+0000 7fd3bd69a640 1 -- 172.21.15.161:0/802450370 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3b804f4b0 con 0x7fd3b81578b0 2024-03-28T23:30:45.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.301+0000 7fd39f7fe640 1 -- 172.21.15.161:0/802450370 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5189 (secure 0 0 0) 0x7fd3a804c060 con 0x7fd3b81578b0 2024-03-28T23:30:45.302 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:45.302 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/1577284938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":1577284938},{"type":"v1","addr":"172.21.15.170:6835","nonce":1577284938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4258,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/1107796931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1107796931},{"type":"v1","addr":"172.21.15.170:6834","nonce":1107796931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4107992135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4107992135},{"type":"v1","addr":"172.21.15.161:6836","nonce":4107992135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:29:45.903844+0000","modified":"2024-03-28T23:30:07.984387+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6837/3832540545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3832540545},{"type":"v1","addr":"172.21.15.161:6837","nonce":3832540545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4275,"qdb_cluster":[4275]},"id":1}]} 2024-03-28T23:30:45.302 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 9 2024-03-28T23:30:45.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.303+0000 7fd3bd69a640 1 -- 172.21.15.161:0/802450370 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd3a406b500 msgr2=0x7fd3a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:45.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.303+0000 7fd3bd69a640 1 --2- 172.21.15.161:0/802450370 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd3a406b500 0x7fd3a406d9c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac001380 tx=0x7fd3ac009040 comp rx=0 tx=0).stop 2024-03-28T23:30:45.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.303+0000 7fd3bd69a640 1 -- 172.21.15.161:0/802450370 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3b81578b0 msgr2=0x7fd3b80c20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:45.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.303+0000 7fd3bd69a640 1 --2- 172.21.15.161:0/802450370 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3b81578b0 0x7fd3b80c20e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd3a802c2a0 tx=0x7fd3a8004730 comp rx=0 tx=0).stop 2024-03-28T23:30:45.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.303+0000 7fd3bd69a640 1 -- 172.21.15.161:0/802450370 shutdown_connections 2024-03-28T23:30:45.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.303+0000 7fd3bd69a640 1 --2- 172.21.15.161:0/802450370 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd3a406b500 0x7fd3a406d9c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:45.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.303+0000 7fd3bd69a640 1 --2- 172.21.15.161:0/802450370 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3b80c1530 0x7fd3b81eabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:45.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.303+0000 7fd3bd69a640 1 --2- 172.21.15.161:0/802450370 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3b814f8f0 0x7fd3b80c0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:45.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.303+0000 7fd3bd69a640 1 --2- 172.21.15.161:0/802450370 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3b81578b0 0x7fd3b80c20e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:45.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.303+0000 7fd3bd69a640 1 -- 172.21.15.161:0/802450370 >> 172.21.15.161:0/802450370 conn(0x7fd3b80ac620 msgr2=0x7fd3b81500b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:45.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.303+0000 7fd3bd69a640 1 -- 172.21.15.161:0/802450370 shutdown_connections 2024-03-28T23:30:45.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.303+0000 7fd3bd69a640 1 -- 172.21.15.161:0/802450370 wait complete. 2024-03-28T23:30:45.315 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:30:45.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.415+0000 7f769bc39640 1 Processor -- start 2024-03-28T23:30:45.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.416+0000 7f769bc39640 1 -- start start 2024-03-28T23:30:45.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.416+0000 7f769bc39640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76940c5410 0x7f76940c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:45.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.416+0000 7f769bc39640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7694158a30 con 0x7f76941553a0 2024-03-28T23:30:45.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.416+0000 7f769bc39640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7694158d10 con 0x7f7694152890 2024-03-28T23:30:45.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.416+0000 7f769bc39640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f76940c8480 con 0x7f76940c5410 2024-03-28T23:30:45.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.416+0000 7f769a1d6640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f76941553a0 0x7f7694155820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:35012/0 (socket says 172.21.15.161:35012) 2024-03-28T23:30:45.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.416+0000 7f769a1d6640 1 -- 172.21.15.161:0/2806100159 learned_addr learned my addr 172.21.15.161:0/2806100159 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:45.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.417+0000 7f76991d4640 1 --2- 172.21.15.161:0/2806100159 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76940c5410 0x7f76940c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:45.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.417+0000 7f76989d3640 1 -- 172.21.15.161:0/2806100159 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1717265106 0 0) 0x7f7694158a30 con 0x7f76941553a0 2024-03-28T23:30:45.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.417+0000 7f76989d3640 1 -- 172.21.15.161:0/2806100159 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7680003520 con 0x7f76941553a0 2024-03-28T23:30:45.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.417+0000 7f76989d3640 1 -- 172.21.15.161:0/2806100159 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1150055163 0 0) 0x7f7694158d10 con 0x7f7694152890 2024-03-28T23:30:45.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.417+0000 7f76989d3640 1 -- 172.21.15.161:0/2806100159 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7694158a30 con 0x7f7694152890 2024-03-28T23:30:45.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.417+0000 7f76989d3640 1 -- 172.21.15.161:0/2806100159 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1756980094 0 0) 0x7f7680003520 con 0x7f76941553a0 2024-03-28T23:30:45.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.417+0000 7f76991d4640 1 -- 172.21.15.161:0/2806100159 >> v1:172.21.15.170:6789/0 conn(0x7f7694152890 legacy=0x7f76940c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:45.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.417+0000 7f76991d4640 1 -- 172.21.15.161:0/2806100159 >> v1:172.21.15.161:6789/0 conn(0x7f76941553a0 legacy=0x7f7694155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:45.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.417+0000 7f76991d4640 1 -- 172.21.15.161:0/2806100159 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7694158ff0 con 0x7f76940c5410 2024-03-28T23:30:45.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.418+0000 7f76991d4640 1 --2- 172.21.15.161:0/2806100159 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76940c5410 0x7f76940c5850 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f76840097c0 tx=0x7f768402b760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8f91c5ef1899b63 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:45.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.418+0000 7f76989d3640 1 -- 172.21.15.161:0/2806100159 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7684037070 con 0x7f76940c5410 2024-03-28T23:30:45.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.418+0000 7f76989d3640 1 -- 172.21.15.161:0/2806100159 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7684030510 con 0x7f76940c5410 2024-03-28T23:30:45.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.418+0000 7f76989d3640 1 -- 172.21.15.161:0/2806100159 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76900019f0 con 0x7f76940c5410 2024-03-28T23:30:45.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.418+0000 7f769bc39640 1 -- 172.21.15.161:0/2806100159 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76940c5410 msgr2=0x7f76940c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:45.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.418+0000 7f769bc39640 1 --2- 172.21.15.161:0/2806100159 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76940c5410 0x7f76940c5850 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f76840097c0 tx=0x7f768402b760 comp rx=0 tx=0).stop 2024-03-28T23:30:45.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.419+0000 7f769bc39640 1 -- 172.21.15.161:0/2806100159 shutdown_connections 2024-03-28T23:30:45.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.419+0000 7f769bc39640 1 --2- 172.21.15.161:0/2806100159 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76940c5410 0x7f76940c5850 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:45.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.419+0000 7f769bc39640 1 -- 172.21.15.161:0/2806100159 >> 172.21.15.161:0/2806100159 conn(0x7f76940ac620 msgr2=0x7f76940aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:45.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.419+0000 7f769bc39640 1 -- 172.21.15.161:0/2806100159 shutdown_connections 2024-03-28T23:30:45.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.419+0000 7f769bc39640 1 -- 172.21.15.161:0/2806100159 wait complete. 2024-03-28T23:30:45.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.420+0000 7f769bc39640 1 Processor -- start 2024-03-28T23:30:45.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.420+0000 7f769bc39640 1 -- start start 2024-03-28T23:30:45.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.420+0000 7f769bc39640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76940c5410 0x7f7694150ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:45.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.420+0000 7f769bc39640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7694152890 0x7f76940c0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:45.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.420+0000 7f769bc39640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76941553a0 0x7f76940b0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:45.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.420+0000 7f769bc39640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7694159640 con 0x7f7694152890 2024-03-28T23:30:45.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.420+0000 7f769bc39640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f76940c5d90 con 0x7f76941553a0 2024-03-28T23:30:45.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.420+0000 7f769bc39640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f76940c90a0 con 0x7f76940c5410 2024-03-28T23:30:45.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.420+0000 7f76991d4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7694152890 0x7f76940c0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:45.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.420+0000 7f76991d4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7694152890 0x7f76940c0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:52042/0 (socket says 172.21.15.161:52042) 2024-03-28T23:30:45.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.420+0000 7f76991d4640 1 -- 172.21.15.161:0/3046525829 learned_addr learned my addr 172.21.15.161:0/3046525829 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:45.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.421+0000 7f769a1d6640 1 --2- 172.21.15.161:0/3046525829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76941553a0 0x7f76940b0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:45.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.421+0000 7f76999d5640 1 --2- 172.21.15.161:0/3046525829 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76940c5410 0x7f7694150ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:45.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.421+0000 7f76991d4640 1 -- 172.21.15.161:0/3046525829 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76940c5410 msgr2=0x7f7694150ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:45.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.421+0000 7f76991d4640 1 --2- 172.21.15.161:0/3046525829 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76940c5410 0x7f7694150ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:45.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.421+0000 7f76991d4640 1 -- 172.21.15.161:0/3046525829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76941553a0 msgr2=0x7f76940b0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:45.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.421+0000 7f76991d4640 1 --2- 172.21.15.161:0/3046525829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76941553a0 0x7f76940b0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:45.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.421+0000 7f76991d4640 1 -- 172.21.15.161:0/3046525829 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76941edfb0 con 0x7f7694152890 2024-03-28T23:30:45.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.421+0000 7f769a1d6640 1 --2- 172.21.15.161:0/3046525829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76941553a0 0x7f76940b0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:45.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.421+0000 7f76999d5640 1 --2- 172.21.15.161:0/3046525829 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76940c5410 0x7f7694150ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:45.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.421+0000 7f76991d4640 1 --2- 172.21.15.161:0/3046525829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7694152890 0x7f76940c0300 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7684000c00 tx=0x7f76840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:45.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.421+0000 7f768a7fc640 1 -- 172.21.15.161:0/3046525829 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7684041070 con 0x7f7694152890 2024-03-28T23:30:45.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.421+0000 7f768a7fc640 1 -- 172.21.15.161:0/3046525829 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f768403c870 con 0x7f7694152890 2024-03-28T23:30:45.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.421+0000 7f769bc39640 1 -- 172.21.15.161:0/3046525829 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76941ee290 con 0x7f7694152890 2024-03-28T23:30:45.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.421+0000 7f768a7fc640 1 -- 172.21.15.161:0/3046525829 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7684037040 con 0x7f7694152890 2024-03-28T23:30:45.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.421+0000 7f769bc39640 1 -- 172.21.15.161:0/3046525829 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76941ee850 con 0x7f7694152890 2024-03-28T23:30:45.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.422+0000 7f769bc39640 1 -- 172.21.15.161:0/3046525829 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f769404f4b0 con 0x7f7694152890 2024-03-28T23:30:45.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.423+0000 7f768a7fc640 1 -- 172.21.15.161:0/3046525829 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7684044070 con 0x7f7694152890 2024-03-28T23:30:45.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.423+0000 7f768a7fc640 1 --2- 172.21.15.161:0/3046525829 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f767806b4d0 0x7f767806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:45.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.423+0000 7f768a7fc640 1 -- 172.21.15.161:0/3046525829 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f76840ac4c0 con 0x7f7694152890 2024-03-28T23:30:45.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.423+0000 7f76999d5640 1 --2- 172.21.15.161:0/3046525829 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f767806b4d0 0x7f767806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:45.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.424+0000 7f76999d5640 1 --2- 172.21.15.161:0/3046525829 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f767806b4d0 0x7f767806d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f768c0011e0 tx=0x7f768c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:45.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.427+0000 7f768a7fc640 1 -- 172.21.15.161:0/3046525829 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f768407b070 con 0x7f7694152890 2024-03-28T23:30:45.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.605+0000 7f769bc39640 1 -- 172.21.15.161:0/3046525829 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f76940c8120 con 0x7f7694152890 2024-03-28T23:30:45.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.606+0000 7f768a7fc640 1 -- 172.21.15.161:0/3046525829 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18937 (secure 0 0 0) 0x7f768407a970 con 0x7f7694152890 2024-03-28T23:30:45.607 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:45.607 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":26,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:30:35.148429+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:30:45.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.609+0000 7f769bc39640 1 -- 172.21.15.161:0/3046525829 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f767806b4d0 msgr2=0x7f767806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:45.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.609+0000 7f769bc39640 1 --2- 172.21.15.161:0/3046525829 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f767806b4d0 0x7f767806d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f768c0011e0 tx=0x7f768c009040 comp rx=0 tx=0).stop 2024-03-28T23:30:45.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.609+0000 7f769bc39640 1 -- 172.21.15.161:0/3046525829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7694152890 msgr2=0x7f76940c0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:45.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.609+0000 7f769bc39640 1 --2- 172.21.15.161:0/3046525829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7694152890 0x7f76940c0300 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7684000c00 tx=0x7f76840046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:45.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.609+0000 7f769bc39640 1 -- 172.21.15.161:0/3046525829 shutdown_connections 2024-03-28T23:30:45.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.609+0000 7f769bc39640 1 --2- 172.21.15.161:0/3046525829 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f767806b4d0 0x7f767806d990 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:45.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.609+0000 7f769bc39640 1 --2- 172.21.15.161:0/3046525829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7694152890 0x7f76940c0300 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:45.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.609+0000 7f769bc39640 1 --2- 172.21.15.161:0/3046525829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76941553a0 0x7f76940b0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:45.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.609+0000 7f769bc39640 1 --2- 172.21.15.161:0/3046525829 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76940c5410 0x7f7694150ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:45.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.609+0000 7f769bc39640 1 -- 172.21.15.161:0/3046525829 >> 172.21.15.161:0/3046525829 conn(0x7f76940ac620 msgr2=0x7f76940caa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:45.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.609+0000 7f769bc39640 1 -- 172.21.15.161:0/3046525829 shutdown_connections 2024-03-28T23:30:45.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.609+0000 7f769bc39640 1 -- 172.21.15.161:0/3046525829 wait complete. 2024-03-28T23:30:45.622 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-28T23:30:45.622 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:30:45.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.723+0000 7fd3ab69d640 1 Processor -- start 2024-03-28T23:30:45.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.723+0000 7fd3ab69d640 1 -- start start 2024-03-28T23:30:45.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.724+0000 7fd3ab69d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd3a4072810 0x7fd3a40c4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:45.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.724+0000 7fd3ab69d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd3a40c4d80 0x7fd3a406fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:45.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.724+0000 7fd3ab69d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3a4153050 con 0x7fd3a40701a0 2024-03-28T23:30:45.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.724+0000 7fd3ab69d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd3a40a8520 con 0x7fd3a4072810 2024-03-28T23:30:45.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.724+0000 7fd3ab69d640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd3a4153310 con 0x7fd3a40c4d80 2024-03-28T23:30:45.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.724+0000 7fd3a9439640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd3a4072810 0x7fd3a40c4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:45.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.724+0000 7fd3a9c3a640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd3a40701a0 0x7fd3a40c3390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:35018/0 (socket says 172.21.15.161:35018) 2024-03-28T23:30:45.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.724+0000 7fd3a9c3a640 1 -- 172.21.15.161:0/2853508890 learned_addr learned my addr 172.21.15.161:0/2853508890 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:45.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.724+0000 7fd3a8c38640 1 --2- 172.21.15.161:0/2853508890 >> v2:172.21.15.170:3300/0 conn(0x7fd3a40c4d80 0x7fd3a406fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:45.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.724+0000 7fd3a9439640 1 -- 172.21.15.161:0/2853508890 >> v2:172.21.15.170:3300/0 conn(0x7fd3a40c4d80 msgr2=0x7fd3a406fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:45.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.724+0000 7fd3a9439640 1 --2- 172.21.15.161:0/2853508890 >> v2:172.21.15.170:3300/0 conn(0x7fd3a40c4d80 0x7fd3a406fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:45.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.724+0000 7fd3a9439640 1 -- 172.21.15.161:0/2853508890 >> v1:172.21.15.161:6789/0 conn(0x7fd3a40701a0 legacy=0x7fd3a40c3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:45.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.724+0000 7fd3a9439640 1 -- 172.21.15.161:0/2853508890 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3a41535f0 con 0x7fd3a4072810 2024-03-28T23:30:45.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.724+0000 7fd39bfff640 1 -- 172.21.15.161:0/2853508890 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4097151535 0 0) 0x7fd3a4153050 con 0x7fd3a40701a0 2024-03-28T23:30:45.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.725+0000 7fd3a9439640 1 --2- 172.21.15.161:0/2853508890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3a4072810 0x7fd3a40c4840 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4068990 tx=0x7fd3a002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91be12b8648833df server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:45.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.725+0000 7fd39bfff640 1 -- 172.21.15.161:0/2853508890 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a00315a0 con 0x7fd3a4072810 2024-03-28T23:30:45.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.725+0000 7fd39bfff640 1 -- 172.21.15.161:0/2853508890 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3a0037070 con 0x7fd3a4072810 2024-03-28T23:30:45.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.725+0000 7fd39bfff640 1 -- 172.21.15.161:0/2853508890 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0031960 con 0x7fd3a4072810 2024-03-28T23:30:45.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.726+0000 7fd3ab69d640 1 -- 172.21.15.161:0/2853508890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3a4072810 msgr2=0x7fd3a40c4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:45.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.726+0000 7fd3ab69d640 1 --2- 172.21.15.161:0/2853508890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3a4072810 0x7fd3a40c4840 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4068990 tx=0x7fd3a002b840 comp rx=0 tx=0).stop 2024-03-28T23:30:45.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.726+0000 7fd3ab69d640 1 -- 172.21.15.161:0/2853508890 shutdown_connections 2024-03-28T23:30:45.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.726+0000 7fd3ab69d640 1 --2- 172.21.15.161:0/2853508890 >> v2:172.21.15.170:3300/0 conn(0x7fd3a40c4d80 0x7fd3a406fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:45.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.726+0000 7fd3ab69d640 1 --2- 172.21.15.161:0/2853508890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3a4072810 0x7fd3a40c4840 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:45.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.726+0000 7fd3ab69d640 1 -- 172.21.15.161:0/2853508890 >> 172.21.15.161:0/2853508890 conn(0x7fd3a40aa740 msgr2=0x7fd3a40aab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:45.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.726+0000 7fd3ab69d640 1 -- 172.21.15.161:0/2853508890 shutdown_connections 2024-03-28T23:30:45.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.726+0000 7fd3ab69d640 1 -- 172.21.15.161:0/2853508890 wait complete. 2024-03-28T23:30:45.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.727+0000 7fd3ab69d640 1 Processor -- start 2024-03-28T23:30:45.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.727+0000 7fd3ab69d640 1 -- start start 2024-03-28T23:30:45.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.728+0000 7fd3ab69d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3a40701a0 0x7fd3a41526a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:45.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.728+0000 7fd3ab69d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3a4072810 0x7fd3a4145cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:45.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.728+0000 7fd3ab69d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3a40c4d80 0x7fd3a4146210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:45.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.728+0000 7fd3ab69d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a4070610 con 0x7fd3a40c4d80 2024-03-28T23:30:45.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.728+0000 7fd3ab69d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a4153bb0 con 0x7fd3a4072810 2024-03-28T23:30:45.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.728+0000 7fd3ab69d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd3a4070790 con 0x7fd3a40701a0 2024-03-28T23:30:45.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.728+0000 7fd3a9c3a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3a40c4d80 0x7fd3a4146210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:45.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.728+0000 7fd3a9c3a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3a40c4d80 0x7fd3a4146210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:52060/0 (socket says 172.21.15.161:52060) 2024-03-28T23:30:45.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.728+0000 7fd3a9c3a640 1 -- 172.21.15.161:0/2464404491 learned_addr learned my addr 172.21.15.161:0/2464404491 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:45.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.728+0000 7fd3a9439640 1 --2- 172.21.15.161:0/2464404491 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3a40701a0 0x7fd3a41526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:45.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.728+0000 7fd3a8c38640 1 --2- 172.21.15.161:0/2464404491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3a4072810 0x7fd3a4145cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:45.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.728+0000 7fd3a9c3a640 1 -- 172.21.15.161:0/2464404491 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3a40701a0 msgr2=0x7fd3a41526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:45.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.728+0000 7fd3a9c3a640 1 --2- 172.21.15.161:0/2464404491 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3a40701a0 0x7fd3a41526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:45.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.728+0000 7fd3a9c3a640 1 -- 172.21.15.161:0/2464404491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3a4072810 msgr2=0x7fd3a4145cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:45.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.728+0000 7fd3a9c3a640 1 --2- 172.21.15.161:0/2464404491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3a4072810 0x7fd3a4145cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:45.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.728+0000 7fd3a9c3a640 1 -- 172.21.15.161:0/2464404491 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3a4154ca0 con 0x7fd3a40c4d80 2024-03-28T23:30:45.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.728+0000 7fd3a9c3a640 1 --2- 172.21.15.161:0/2464404491 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3a40c4d80 0x7fd3a4146210 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd39c00c700 tx=0x7fd39c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:45.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.728+0000 7fd399ffb640 1 -- 172.21.15.161:0/2464404491 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd39c018070 con 0x7fd3a40c4d80 2024-03-28T23:30:45.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.729+0000 7fd399ffb640 1 -- 172.21.15.161:0/2464404491 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd39c00ce10 con 0x7fd3a40c4d80 2024-03-28T23:30:45.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.729+0000 7fd3ab69d640 1 -- 172.21.15.161:0/2464404491 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3a40c2000 con 0x7fd3a40c4d80 2024-03-28T23:30:45.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.729+0000 7fd399ffb640 1 -- 172.21.15.161:0/2464404491 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd39c00a880 con 0x7fd3a40c4d80 2024-03-28T23:30:45.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.729+0000 7fd3ab69d640 1 -- 172.21.15.161:0/2464404491 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3a41545e0 con 0x7fd3a40c4d80 2024-03-28T23:30:45.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.730+0000 7fd3ab69d640 1 -- 172.21.15.161:0/2464404491 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3a404f4b0 con 0x7fd3a40c4d80 2024-03-28T23:30:45.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.730+0000 7fd399ffb640 1 -- 172.21.15.161:0/2464404491 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd39c01b070 con 0x7fd3a40c4d80 2024-03-28T23:30:45.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.730+0000 7fd399ffb640 1 --2- 172.21.15.161:0/2464404491 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd38806b4d0 0x7fd38806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:45.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.731+0000 7fd399ffb640 1 -- 172.21.15.161:0/2464404491 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fd39c0899d0 con 0x7fd3a40c4d80 2024-03-28T23:30:45.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.731+0000 7fd3a9439640 1 --2- 172.21.15.161:0/2464404491 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd38806b4d0 0x7fd38806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:45.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.731+0000 7fd3a9439640 1 --2- 172.21.15.161:0/2464404491 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd38806b4d0 0x7fd38806d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0036000 tx=0x7fd3a003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:45.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:45.739+0000 7fd399ffb640 1 -- 172.21.15.161:0/2464404491 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd39c057e90 con 0x7fd3a40c4d80 2024-03-28T23:30:46.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.023+0000 7fd3ab69d640 1 -- 172.21.15.161:0/2464404491 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3a4072dd0 con 0x7fd3a40c4d80 2024-03-28T23:30:46.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.023+0000 7fd399ffb640 1 -- 172.21.15.161:0/2464404491 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5189 (secure 0 0 0) 0x7fd39c05bf30 con 0x7fd3a40c4d80 2024-03-28T23:30:46.024 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:46.024 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/1577284938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":1577284938},{"type":"v1","addr":"172.21.15.170:6835","nonce":1577284938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4258,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/1107796931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1107796931},{"type":"v1","addr":"172.21.15.170:6834","nonce":1107796931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4107992135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4107992135},{"type":"v1","addr":"172.21.15.161:6836","nonce":4107992135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:29:45.903844+0000","modified":"2024-03-28T23:30:07.984387+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6837/3832540545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3832540545},{"type":"v1","addr":"172.21.15.161:6837","nonce":3832540545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4275,"qdb_cluster":[4275]},"id":1}]} 2024-03-28T23:30:46.024 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 9 2024-03-28T23:30:46.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.026+0000 7fd3ab69d640 1 -- 172.21.15.161:0/2464404491 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd38806b4d0 msgr2=0x7fd38806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:46.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.026+0000 7fd3ab69d640 1 --2- 172.21.15.161:0/2464404491 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd38806b4d0 0x7fd38806d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0036000 tx=0x7fd3a003d040 comp rx=0 tx=0).stop 2024-03-28T23:30:46.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.026+0000 7fd3ab69d640 1 -- 172.21.15.161:0/2464404491 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3a40c4d80 msgr2=0x7fd3a4146210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:46.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.026+0000 7fd3ab69d640 1 --2- 172.21.15.161:0/2464404491 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3a40c4d80 0x7fd3a4146210 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd39c00c700 tx=0x7fd39c00cbc0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.026+0000 7fd3ab69d640 1 -- 172.21.15.161:0/2464404491 shutdown_connections 2024-03-28T23:30:46.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.026+0000 7fd3ab69d640 1 --2- 172.21.15.161:0/2464404491 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd38806b4d0 0x7fd38806d990 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.026+0000 7fd3ab69d640 1 --2- 172.21.15.161:0/2464404491 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3a40c4d80 0x7fd3a4146210 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.026+0000 7fd3ab69d640 1 --2- 172.21.15.161:0/2464404491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3a4072810 0x7fd3a4145cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.026+0000 7fd3ab69d640 1 --2- 172.21.15.161:0/2464404491 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3a40701a0 0x7fd3a41526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.026+0000 7fd3ab69d640 1 -- 172.21.15.161:0/2464404491 >> 172.21.15.161:0/2464404491 conn(0x7fd3a40aa740 msgr2=0x7fd3a40c7b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:46.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.026+0000 7fd3ab69d640 1 -- 172.21.15.161:0/2464404491 shutdown_connections 2024-03-28T23:30:46.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.026+0000 7fd3ab69d640 1 -- 172.21.15.161:0/2464404491 wait complete. 2024-03-28T23:30:46.041 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:30:46.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.141+0000 7fe144f6c640 1 Processor -- start 2024-03-28T23:30:46.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.141+0000 7fe144f6c640 1 -- start start 2024-03-28T23:30:46.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.141+0000 7fe144f6c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe1400c5850 0x7fe140155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:46.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.141+0000 7fe144f6c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1400c5d00 con 0x7fe140152890 2024-03-28T23:30:46.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.141+0000 7fe144f6c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe140158a30 con 0x7fe1400c5410 2024-03-28T23:30:46.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.141+0000 7fe144f6c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe1400c8480 con 0x7fe1400c5850 2024-03-28T23:30:46.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.141+0000 7fe13e59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe140152890 0x7fe1400c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:35028/0 (socket says 172.21.15.161:35028) 2024-03-28T23:30:46.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.141+0000 7fe13e59c640 1 -- 172.21.15.161:0/3091420825 learned_addr learned my addr 172.21.15.161:0/3091420825 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:46.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.142+0000 7fe13ed9d640 1 --2- 172.21.15.161:0/3091420825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe1400c5850 0x7fe140155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:46.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.142+0000 7fe13dd9b640 1 -- 172.21.15.161:0/3091420825 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3769631172 0 0) 0x7fe1400c5d00 con 0x7fe140152890 2024-03-28T23:30:46.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.142+0000 7fe13dd9b640 1 -- 172.21.15.161:0/3091420825 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe12c003520 con 0x7fe140152890 2024-03-28T23:30:46.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.142+0000 7fe13dd9b640 1 -- 172.21.15.161:0/3091420825 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2979894154 0 0) 0x7fe140158a30 con 0x7fe1400c5410 2024-03-28T23:30:46.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.142+0000 7fe13dd9b640 1 -- 172.21.15.161:0/3091420825 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1400c5d00 con 0x7fe1400c5410 2024-03-28T23:30:46.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.142+0000 7fe13ed9d640 1 -- 172.21.15.161:0/3091420825 >> v1:172.21.15.170:6789/0 conn(0x7fe1400c5410 legacy=0x7fe1401553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:46.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.142+0000 7fe13ed9d640 1 -- 172.21.15.161:0/3091420825 >> v1:172.21.15.161:6789/0 conn(0x7fe140152890 legacy=0x7fe1400c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:46.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.142+0000 7fe13dd9b640 1 -- 172.21.15.161:0/3091420825 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 339199934 0 0) 0x7fe12c003520 con 0x7fe140152890 2024-03-28T23:30:46.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.142+0000 7fe13ed9d640 1 -- 172.21.15.161:0/3091420825 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe140158d10 con 0x7fe1400c5850 2024-03-28T23:30:46.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.143+0000 7fe13ed9d640 1 --2- 172.21.15.161:0/3091420825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe1400c5850 0x7fe140155ab0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe138009dc0 tx=0x7fe13802bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdd7c5586a5eb8b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:46.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.143+0000 7fe13dd9b640 1 -- 172.21.15.161:0/3091420825 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe138039070 con 0x7fe1400c5850 2024-03-28T23:30:46.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.143+0000 7fe13dd9b640 1 -- 172.21.15.161:0/3091420825 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe13802c880 con 0x7fe1400c5850 2024-03-28T23:30:46.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.143+0000 7fe13dd9b640 1 -- 172.21.15.161:0/3091420825 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe130001670 con 0x7fe1400c5850 2024-03-28T23:30:46.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.144+0000 7fe144f6c640 1 -- 172.21.15.161:0/3091420825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe1400c5850 msgr2=0x7fe140155ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:46.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.144+0000 7fe144f6c640 1 --2- 172.21.15.161:0/3091420825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe1400c5850 0x7fe140155ab0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe138009dc0 tx=0x7fe13802bd50 comp rx=0 tx=0).stop 2024-03-28T23:30:46.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.144+0000 7fe144f6c640 1 -- 172.21.15.161:0/3091420825 shutdown_connections 2024-03-28T23:30:46.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.144+0000 7fe144f6c640 1 --2- 172.21.15.161:0/3091420825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe1400c5850 0x7fe140155ab0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.144+0000 7fe144f6c640 1 -- 172.21.15.161:0/3091420825 >> 172.21.15.161:0/3091420825 conn(0x7fe1400ac620 msgr2=0x7fe1400aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:46.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.144+0000 7fe144f6c640 1 -- 172.21.15.161:0/3091420825 shutdown_connections 2024-03-28T23:30:46.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.144+0000 7fe144f6c640 1 -- 172.21.15.161:0/3091420825 wait complete. 2024-03-28T23:30:46.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.145+0000 7fe144f6c640 1 Processor -- start 2024-03-28T23:30:46.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.145+0000 7fe144f6c640 1 -- start start 2024-03-28T23:30:46.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.145+0000 7fe144f6c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1400c5410 0x7fe1400be5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:46.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.145+0000 7fe144f6c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1400c5850 0x7fe1401eb560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:46.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.146+0000 7fe144f6c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe140152890 0x7fe14014c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:46.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.146+0000 7fe144f6c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe1401592d0 con 0x7fe1400c5410 2024-03-28T23:30:46.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.146+0000 7fe144f6c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe140159640 con 0x7fe1400c5850 2024-03-28T23:30:46.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.146+0000 7fe144f6c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe140155ff0 con 0x7fe140152890 2024-03-28T23:30:46.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.146+0000 7fe13e59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1400c5410 0x7fe1400be5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:46.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.146+0000 7fe13e59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1400c5410 0x7fe1400be5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:52070/0 (socket says 172.21.15.161:52070) 2024-03-28T23:30:46.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.146+0000 7fe13e59c640 1 -- 172.21.15.161:0/4157712855 learned_addr learned my addr 172.21.15.161:0/4157712855 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:46.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.146+0000 7fe13ed9d640 1 --2- 172.21.15.161:0/4157712855 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe140152890 0x7fe14014c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:46.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.146+0000 7fe137fff640 1 --2- 172.21.15.161:0/4157712855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1400c5850 0x7fe1401eb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:46.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.146+0000 7fe13e59c640 1 -- 172.21.15.161:0/4157712855 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe140152890 msgr2=0x7fe14014c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:46.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.146+0000 7fe13e59c640 1 --2- 172.21.15.161:0/4157712855 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe140152890 0x7fe14014c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.146+0000 7fe13e59c640 1 -- 172.21.15.161:0/4157712855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1400c5850 msgr2=0x7fe1401eb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:46.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.146+0000 7fe13e59c640 1 --2- 172.21.15.161:0/4157712855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1400c5850 0x7fe1401eb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.146+0000 7fe13e59c640 1 -- 172.21.15.161:0/4157712855 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1401ebaa0 con 0x7fe1400c5410 2024-03-28T23:30:46.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.146+0000 7fe13e59c640 1 --2- 172.21.15.161:0/4157712855 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1400c5410 0x7fe1400be5c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe13000c550 tx=0x7fe13000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:46.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.146+0000 7fe136ffd640 1 -- 172.21.15.161:0/4157712855 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe130017070 con 0x7fe1400c5410 2024-03-28T23:30:46.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.146+0000 7fe136ffd640 1 -- 172.21.15.161:0/4157712855 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe130005d00 con 0x7fe1400c5410 2024-03-28T23:30:46.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.146+0000 7fe144f6c640 1 -- 172.21.15.161:0/4157712855 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1401576f0 con 0x7fe1400c5410 2024-03-28T23:30:46.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.146+0000 7fe136ffd640 1 -- 172.21.15.161:0/4157712855 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1300072b0 con 0x7fe1400c5410 2024-03-28T23:30:46.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.147+0000 7fe144f6c640 1 -- 172.21.15.161:0/4157712855 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe140157cb0 con 0x7fe1400c5410 2024-03-28T23:30:46.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.148+0000 7fe144f6c640 1 -- 172.21.15.161:0/4157712855 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1400c74a0 con 0x7fe1400c5410 2024-03-28T23:30:46.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.148+0000 7fe136ffd640 1 -- 172.21.15.161:0/4157712855 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe13001a070 con 0x7fe1400c5410 2024-03-28T23:30:46.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.148+0000 7fe136ffd640 1 --2- 172.21.15.161:0/4157712855 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe12006b500 0x7fe12006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:46.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.148+0000 7fe136ffd640 1 -- 172.21.15.161:0/4157712855 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fe130089950 con 0x7fe1400c5410 2024-03-28T23:30:46.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.148+0000 7fe137fff640 1 --2- 172.21.15.161:0/4157712855 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe12006b500 0x7fe12006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:46.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.149+0000 7fe137fff640 1 --2- 172.21.15.161:0/4157712855 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe12006b500 0x7fe12006d9c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe1400c7b60 tx=0x7fe128009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:46.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.153+0000 7fe136ffd640 1 -- 172.21.15.161:0/4157712855 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe1400c74a0 con 0x7fe1400c5410 2024-03-28T23:30:46.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.414+0000 7fe144f6c640 1 -- 172.21.15.161:0/4157712855 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe14004f4b0 con 0x7fe1400c5410 2024-03-28T23:30:46.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.414+0000 7fe136ffd640 1 -- 172.21.15.161:0/4157712855 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5189 (secure 0 0 0) 0x7fe1401ebaa0 con 0x7fe1400c5410 2024-03-28T23:30:46.416 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:46.416 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/1577284938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":1577284938},{"type":"v1","addr":"172.21.15.170:6835","nonce":1577284938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4258,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/1107796931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1107796931},{"type":"v1","addr":"172.21.15.170:6834","nonce":1107796931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4107992135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4107992135},{"type":"v1","addr":"172.21.15.161:6836","nonce":4107992135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:29:45.903844+0000","modified":"2024-03-28T23:30:07.984387+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6837/3832540545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3832540545},{"type":"v1","addr":"172.21.15.161:6837","nonce":3832540545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4275,"qdb_cluster":[4275]},"id":1}]} 2024-03-28T23:30:46.416 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 9 2024-03-28T23:30:46.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.417+0000 7fe144f6c640 1 -- 172.21.15.161:0/4157712855 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe12006b500 msgr2=0x7fe12006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:46.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.417+0000 7fe144f6c640 1 --2- 172.21.15.161:0/4157712855 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe12006b500 0x7fe12006d9c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe1400c7b60 tx=0x7fe128009040 comp rx=0 tx=0).stop 2024-03-28T23:30:46.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.417+0000 7fe144f6c640 1 -- 172.21.15.161:0/4157712855 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1400c5410 msgr2=0x7fe1400be5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:46.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.417+0000 7fe144f6c640 1 --2- 172.21.15.161:0/4157712855 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1400c5410 0x7fe1400be5c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe13000c550 tx=0x7fe13000ca10 comp rx=0 tx=0).stop 2024-03-28T23:30:46.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.417+0000 7fe144f6c640 1 -- 172.21.15.161:0/4157712855 shutdown_connections 2024-03-28T23:30:46.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.417+0000 7fe144f6c640 1 --2- 172.21.15.161:0/4157712855 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe12006b500 0x7fe12006d9c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.417+0000 7fe144f6c640 1 --2- 172.21.15.161:0/4157712855 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe140152890 0x7fe14014c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.417+0000 7fe144f6c640 1 --2- 172.21.15.161:0/4157712855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1400c5850 0x7fe1401eb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.417+0000 7fe144f6c640 1 --2- 172.21.15.161:0/4157712855 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1400c5410 0x7fe1400be5c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.417+0000 7fe144f6c640 1 -- 172.21.15.161:0/4157712855 >> 172.21.15.161:0/4157712855 conn(0x7fe1400ac620 msgr2=0x7fe14014f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:46.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.417+0000 7fe144f6c640 1 -- 172.21.15.161:0/4157712855 shutdown_connections 2024-03-28T23:30:46.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.417+0000 7fe144f6c640 1 -- 172.21.15.161:0/4157712855 wait complete. 2024-03-28T23:30:46.430 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:30:46.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.531+0000 7f32ab75e640 1 Processor -- start 2024-03-28T23:30:46.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.531+0000 7f32ab75e640 1 -- start start 2024-03-28T23:30:46.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.532+0000 7f32ab75e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32a4152890 0x7f32a40c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:46.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.532+0000 7f32ab75e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f32a40c8ed0 0x7f32a40c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:46.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.532+0000 7f32ab75e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32a4158a30 con 0x7f32a40c5950 2024-03-28T23:30:46.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.532+0000 7f32ab75e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f32a40c5db0 con 0x7f32a40c8ed0 2024-03-28T23:30:46.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.532+0000 7f32ab75e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f32a40c8480 con 0x7f32a4152890 2024-03-28T23:30:46.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.532+0000 7f32a8cf9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f32a40c8ed0 0x7f32a40c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:46.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.532+0000 7f32a8cf9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f32a40c8ed0 0x7f32a40c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:52074/0 (socket says 172.21.15.161:52074) 2024-03-28T23:30:46.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.532+0000 7f32a8cf9640 1 -- 172.21.15.161:0/1605843291 learned_addr learned my addr 172.21.15.161:0/1605843291 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:46.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.532+0000 7f32a94fa640 1 --2- 172.21.15.161:0/1605843291 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32a4152890 0x7f32a40c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:46.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.532+0000 7f32a8cf9640 1 -- 172.21.15.161:0/1605843291 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32a4152890 msgr2=0x7f32a40c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:46.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.532+0000 7f32a8cf9640 1 --2- 172.21.15.161:0/1605843291 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32a4152890 0x7f32a40c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.532+0000 7f32a8cf9640 1 -- 172.21.15.161:0/1605843291 >> v1:172.21.15.170:6789/0 conn(0x7f32a40c5950 legacy=0x7f32a41553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:46.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.532+0000 7f32a8cf9640 1 -- 172.21.15.161:0/1605843291 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32a4158d10 con 0x7f32a40c8ed0 2024-03-28T23:30:46.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.533+0000 7f32a8cf9640 1 --2- 172.21.15.161:0/1605843291 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32a40c8ed0 0x7f32a40c5410 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3298009850 tx=0x7f329802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80cdd20b8dfce97d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:46.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.533+0000 7f32a3fff640 1 -- 172.21.15.161:0/1605843291 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3298038070 con 0x7f32a40c8ed0 2024-03-28T23:30:46.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.533+0000 7f32a3fff640 1 -- 172.21.15.161:0/1605843291 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3298004230 con 0x7f32a40c8ed0 2024-03-28T23:30:46.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.533+0000 7f32a3fff640 1 -- 172.21.15.161:0/1605843291 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3298003610 con 0x7f32a40c8ed0 2024-03-28T23:30:46.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.534+0000 7f32ab75e640 1 -- 172.21.15.161:0/1605843291 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32a40c8ed0 msgr2=0x7f32a40c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:46.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.534+0000 7f32ab75e640 1 --2- 172.21.15.161:0/1605843291 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32a40c8ed0 0x7f32a40c5410 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3298009850 tx=0x7f329802b7b0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.534+0000 7f32ab75e640 1 -- 172.21.15.161:0/1605843291 shutdown_connections 2024-03-28T23:30:46.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.534+0000 7f32ab75e640 1 --2- 172.21.15.161:0/1605843291 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32a40c8ed0 0x7f32a40c5410 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.534+0000 7f32ab75e640 1 --2- 172.21.15.161:0/1605843291 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32a4152890 0x7f32a40c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.534+0000 7f32ab75e640 1 -- 172.21.15.161:0/1605843291 >> 172.21.15.161:0/1605843291 conn(0x7f32a40ac620 msgr2=0x7f32a40aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:46.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.534+0000 7f32ab75e640 1 -- 172.21.15.161:0/1605843291 shutdown_connections 2024-03-28T23:30:46.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.534+0000 7f32ab75e640 1 -- 172.21.15.161:0/1605843291 wait complete. 2024-03-28T23:30:46.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.535+0000 7f32ab75e640 1 Processor -- start 2024-03-28T23:30:46.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.535+0000 7f32ab75e640 1 -- start start 2024-03-28T23:30:46.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.536+0000 7f32ab75e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32a40c5950 0x7f32a40ab6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:46.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.536+0000 7f32ab75e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32a40c8ed0 0x7f32a40bdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:46.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.536+0000 7f32ab75e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32a4152890 0x7f32a40be110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:46.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.536+0000 7f32ab75e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f32a41592d0 con 0x7f32a40c8ed0 2024-03-28T23:30:46.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.536+0000 7f32ab75e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f32a4159640 con 0x7f32a4152890 2024-03-28T23:30:46.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.536+0000 7f32ab75e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f32a4152c70 con 0x7f32a40c5950 2024-03-28T23:30:46.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.536+0000 7f32a8cf9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32a40c8ed0 0x7f32a40bdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:46.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.536+0000 7f32a8cf9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32a40c8ed0 0x7f32a40bdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:52090/0 (socket says 172.21.15.161:52090) 2024-03-28T23:30:46.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.536+0000 7f32a8cf9640 1 -- 172.21.15.161:0/1830995778 learned_addr learned my addr 172.21.15.161:0/1830995778 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:46.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.536+0000 7f32a94fa640 1 --2- 172.21.15.161:0/1830995778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32a40c5950 0x7f32a40ab6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:46.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.536+0000 7f32a9cfb640 1 --2- 172.21.15.161:0/1830995778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32a4152890 0x7f32a40be110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:46.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.536+0000 7f32a8cf9640 1 -- 172.21.15.161:0/1830995778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32a40c5950 msgr2=0x7f32a40ab6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:46.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.536+0000 7f32a8cf9640 1 --2- 172.21.15.161:0/1830995778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32a40c5950 0x7f32a40ab6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.536+0000 7f32a8cf9640 1 -- 172.21.15.161:0/1830995778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32a4152890 msgr2=0x7f32a40be110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:46.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.536+0000 7f32a8cf9640 1 --2- 172.21.15.161:0/1830995778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32a4152890 0x7f32a40be110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.536+0000 7f32a8cf9640 1 -- 172.21.15.161:0/1830995778 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32a414ffe0 con 0x7f32a40c8ed0 2024-03-28T23:30:46.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.536+0000 7f32a94fa640 1 --2- 172.21.15.161:0/1830995778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32a40c5950 0x7f32a40ab6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:46.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.536+0000 7f32a9cfb640 1 --2- 172.21.15.161:0/1830995778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32a4152890 0x7f32a40be110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:46.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.536+0000 7f32a8cf9640 1 --2- 172.21.15.161:0/1830995778 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32a40c8ed0 0x7f32a40bdbd0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3298000c00 tx=0x7f32980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:46.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.536+0000 7f32a1ffb640 1 -- 172.21.15.161:0/1830995778 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3298042020 con 0x7f32a40c8ed0 2024-03-28T23:30:46.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.536+0000 7f32a1ffb640 1 -- 172.21.15.161:0/1830995778 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3298003a40 con 0x7f32a40c8ed0 2024-03-28T23:30:46.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.536+0000 7f32a1ffb640 1 -- 172.21.15.161:0/1830995778 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3298038040 con 0x7f32a40c8ed0 2024-03-28T23:30:46.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.537+0000 7f32ab75e640 1 -- 172.21.15.161:0/1830995778 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32a40c1ce0 con 0x7f32a40c8ed0 2024-03-28T23:30:46.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.537+0000 7f32ab75e640 1 -- 172.21.15.161:0/1830995778 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32a40c22a0 con 0x7f32a40c8ed0 2024-03-28T23:30:46.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.538+0000 7f32ab75e640 1 -- 172.21.15.161:0/1830995778 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32a404f4b0 con 0x7f32a40c8ed0 2024-03-28T23:30:46.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.538+0000 7f32a1ffb640 1 -- 172.21.15.161:0/1830995778 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3298045020 con 0x7f32a40c8ed0 2024-03-28T23:30:46.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.538+0000 7f32a1ffb640 1 --2- 172.21.15.161:0/1830995778 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f329006b4d0 0x7f329006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:46.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.539+0000 7f32a1ffb640 1 -- 172.21.15.161:0/1830995778 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f32980abde0 con 0x7f32a40c8ed0 2024-03-28T23:30:46.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.539+0000 7f32a94fa640 1 --2- 172.21.15.161:0/1830995778 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f329006b4d0 0x7f329006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:46.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.539+0000 7f32a94fa640 1 --2- 172.21.15.161:0/1830995778 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f329006b4d0 0x7f329006d990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3294006fd0 tx=0x7f3294008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:46.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.543+0000 7f32a1ffb640 1 -- 172.21.15.161:0/1830995778 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f329804c070 con 0x7f32a40c8ed0 2024-03-28T23:30:46.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.770+0000 7f32ab75e640 1 -- 172.21.15.161:0/1830995778 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f32a40be650 con 0x7f32a40c8ed0 2024-03-28T23:30:46.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.771+0000 7f32a1ffb640 1 -- 172.21.15.161:0/1830995778 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18937 (secure 0 0 0) 0x7f32a414ffe0 con 0x7f32a40c8ed0 2024-03-28T23:30:46.772 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:46.772 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":26,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:30:35.148429+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:29:44.021375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T23:29:44.872102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T23:29:48.591063+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:30:46.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.773+0000 7f32ab75e640 1 -- 172.21.15.161:0/1830995778 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f329006b4d0 msgr2=0x7f329006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:46.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.773+0000 7f32ab75e640 1 --2- 172.21.15.161:0/1830995778 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f329006b4d0 0x7f329006d990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3294006fd0 tx=0x7f3294008040 comp rx=0 tx=0).stop 2024-03-28T23:30:46.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.773+0000 7f32ab75e640 1 -- 172.21.15.161:0/1830995778 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32a40c8ed0 msgr2=0x7f32a40bdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:46.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.773+0000 7f32ab75e640 1 --2- 172.21.15.161:0/1830995778 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32a40c8ed0 0x7f32a40bdbd0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3298000c00 tx=0x7f32980046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.773+0000 7f32ab75e640 1 -- 172.21.15.161:0/1830995778 shutdown_connections 2024-03-28T23:30:46.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.773+0000 7f32ab75e640 1 --2- 172.21.15.161:0/1830995778 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f329006b4d0 0x7f329006d990 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.773+0000 7f32ab75e640 1 --2- 172.21.15.161:0/1830995778 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32a40c8ed0 0x7f32a40bdbd0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.773+0000 7f32ab75e640 1 --2- 172.21.15.161:0/1830995778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32a4152890 0x7f32a40be110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.773+0000 7f32ab75e640 1 --2- 172.21.15.161:0/1830995778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32a40c5950 0x7f32a40ab6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.773+0000 7f32ab75e640 1 -- 172.21.15.161:0/1830995778 >> 172.21.15.161:0/1830995778 conn(0x7f32a40ac620 msgr2=0x7f32a414aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:46.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.773+0000 7f32ab75e640 1 -- 172.21.15.161:0/1830995778 shutdown_connections 2024-03-28T23:30:46.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.773+0000 7f32ab75e640 1 -- 172.21.15.161:0/1830995778 wait complete. 2024-03-28T23:30:46.785 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-28T23:30:46.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.886+0000 7f6201647640 1 Processor -- start 2024-03-28T23:30:46.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.886+0000 7f6201647640 1 -- start start 2024-03-28T23:30:46.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.887+0000 7f6201647640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f61fc152890 0x7f61fc0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:46.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.887+0000 7f6201647640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f61fc0c8ed0 0x7f61fc0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:46.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.887+0000 7f6201647640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61fc158a30 con 0x7f61fc0c5950 2024-03-28T23:30:46.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.887+0000 7f6201647640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f61fc0c8480 con 0x7f61fc152890 2024-03-28T23:30:46.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.887+0000 7f6201647640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f61fc159640 con 0x7f61fc0c8ed0 2024-03-28T23:30:46.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.887+0000 7f61faffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f61fc152890 0x7f61fc0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:46.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.887+0000 7f61fb7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f61fc0c5950 0x7f61fc1553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:35040/0 (socket says 172.21.15.161:35040) 2024-03-28T23:30:46.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.887+0000 7f61fb7fe640 1 -- 172.21.15.161:0/1163242581 learned_addr learned my addr 172.21.15.161:0/1163242581 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:46.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.887+0000 7f61fa7fc640 1 --2- 172.21.15.161:0/1163242581 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f61fc0c8ed0 0x7f61fc0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:46.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.887+0000 7f61faffd640 1 -- 172.21.15.161:0/1163242581 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f61fc0c8ed0 msgr2=0x7f61fc0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:46.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.887+0000 7f61faffd640 1 --2- 172.21.15.161:0/1163242581 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f61fc0c8ed0 0x7f61fc0c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.887+0000 7f61f9ffb640 1 -- 172.21.15.161:0/1163242581 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2603495568 0 0) 0x7f61fc158a30 con 0x7f61fc0c5950 2024-03-28T23:30:46.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.887+0000 7f61faffd640 1 -- 172.21.15.161:0/1163242581 >> v1:172.21.15.161:6789/0 conn(0x7f61fc0c5950 legacy=0x7f61fc1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:46.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.887+0000 7f61faffd640 1 -- 172.21.15.161:0/1163242581 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61fc158d10 con 0x7f61fc152890 2024-03-28T23:30:46.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.888+0000 7f61faffd640 1 --2- 172.21.15.161:0/1163242581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61fc152890 0x7f61fc0c8990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f61ec0097a0 tx=0x7f61ec02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dd71ef555084c61 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:46.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.888+0000 7f61f9ffb640 1 -- 172.21.15.161:0/1163242581 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec038070 con 0x7f61fc152890 2024-03-28T23:30:46.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.888+0000 7f61f9ffb640 1 -- 172.21.15.161:0/1163242581 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61ec031700 con 0x7f61fc152890 2024-03-28T23:30:46.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.888+0000 7f61f9ffb640 1 -- 172.21.15.161:0/1163242581 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec031ad0 con 0x7f61fc152890 2024-03-28T23:30:46.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.889+0000 7f6201647640 1 -- 172.21.15.161:0/1163242581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61fc152890 msgr2=0x7f61fc0c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:46.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.889+0000 7f6201647640 1 --2- 172.21.15.161:0/1163242581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61fc152890 0x7f61fc0c8990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f61ec0097a0 tx=0x7f61ec02b7a0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.889+0000 7f6201647640 1 -- 172.21.15.161:0/1163242581 shutdown_connections 2024-03-28T23:30:46.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.889+0000 7f6201647640 1 --2- 172.21.15.161:0/1163242581 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f61fc0c8ed0 0x7f61fc0c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.889+0000 7f6201647640 1 --2- 172.21.15.161:0/1163242581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61fc152890 0x7f61fc0c8990 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.889+0000 7f6201647640 1 -- 172.21.15.161:0/1163242581 >> 172.21.15.161:0/1163242581 conn(0x7f61fc0ac620 msgr2=0x7f61fc0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:46.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.889+0000 7f6201647640 1 -- 172.21.15.161:0/1163242581 shutdown_connections 2024-03-28T23:30:46.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.889+0000 7f6201647640 1 -- 172.21.15.161:0/1163242581 wait complete. 2024-03-28T23:30:46.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.890+0000 7f6201647640 1 Processor -- start 2024-03-28T23:30:46.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.890+0000 7f6201647640 1 -- start start 2024-03-28T23:30:46.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.891+0000 7f6201647640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61fc0c5950 0x7f61fc1eec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:46.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.891+0000 7f6201647640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f61fc0c8ed0 0x7f61fc1eb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:46.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.891+0000 7f6201647640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f61fc152890 0x7f61fc0c9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:46.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.891+0000 7f6201647640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc152c70 con 0x7f61fc0c5950 2024-03-28T23:30:46.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.891+0000 7f6201647640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc1592d0 con 0x7f61fc0c8ed0 2024-03-28T23:30:46.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.891+0000 7f6201647640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f61fc152df0 con 0x7f61fc152890 2024-03-28T23:30:46.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.891+0000 7f61faffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61fc0c5950 0x7f61fc1eec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:46.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.891+0000 7f61faffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61fc0c5950 0x7f61fc1eec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:52100/0 (socket says 172.21.15.161:52100) 2024-03-28T23:30:46.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.891+0000 7f61faffd640 1 -- 172.21.15.161:0/2442018894 learned_addr learned my addr 172.21.15.161:0/2442018894 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.891+0000 7f61fb7fe640 1 --2- 172.21.15.161:0/2442018894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f61fc152890 0x7f61fc0c9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.891+0000 7f61fa7fc640 1 --2- 172.21.15.161:0/2442018894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f61fc0c8ed0 0x7f61fc1eb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.891+0000 7f61faffd640 1 -- 172.21.15.161:0/2442018894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f61fc152890 msgr2=0x7f61fc0c9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.891+0000 7f61faffd640 1 --2- 172.21.15.161:0/2442018894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f61fc152890 0x7f61fc0c9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.891+0000 7f61faffd640 1 -- 172.21.15.161:0/2442018894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f61fc0c8ed0 msgr2=0x7f61fc1eb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.891+0000 7f61faffd640 1 --2- 172.21.15.161:0/2442018894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f61fc0c8ed0 0x7f61fc1eb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.891+0000 7f61faffd640 1 -- 172.21.15.161:0/2442018894 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61fc1ebb30 con 0x7f61fc0c5950 2024-03-28T23:30:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.891+0000 7f61fb7fe640 1 --2- 172.21.15.161:0/2442018894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f61fc152890 0x7f61fc0c9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.891+0000 7f61faffd640 1 --2- 172.21.15.161:0/2442018894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61fc0c5950 0x7f61fc1eec00 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f61ec036040 tx=0x7f61ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.892+0000 7f61e37fe640 1 -- 172.21.15.161:0/2442018894 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec042070 con 0x7f61fc0c5950 2024-03-28T23:30:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.892+0000 7f61e37fe640 1 -- 172.21.15.161:0/2442018894 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61ec0339b0 con 0x7f61fc0c5950 2024-03-28T23:30:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.892+0000 7f6201647640 1 -- 172.21.15.161:0/2442018894 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61fc1ee160 con 0x7f61fc0c5950 2024-03-28T23:30:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.892+0000 7f61e37fe640 1 -- 172.21.15.161:0/2442018894 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec038040 con 0x7f61fc0c5950 2024-03-28T23:30:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.892+0000 7f6201647640 1 -- 172.21.15.161:0/2442018894 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61fc153a40 con 0x7f61fc0c5950 2024-03-28T23:30:46.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.893+0000 7f6201647640 1 -- 172.21.15.161:0/2442018894 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61fc1ee730 con 0x7f61fc0c5950 2024-03-28T23:30:46.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.893+0000 7f61e37fe640 1 -- 172.21.15.161:0/2442018894 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f61ec03b070 con 0x7f61fc0c5950 2024-03-28T23:30:46.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.894+0000 7f61e37fe640 1 --2- 172.21.15.161:0/2442018894 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f61e806b4d0 0x7f61e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:46.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.894+0000 7f61fa7fc640 1 --2- 172.21.15.161:0/2442018894 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f61e806b4d0 0x7f61e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:46.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.894+0000 7f61e37fe640 1 -- 172.21.15.161:0/2442018894 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f61ec0abe80 con 0x7f61fc0c5950 2024-03-28T23:30:46.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.894+0000 7f61fa7fc640 1 --2- 172.21.15.161:0/2442018894 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f61e806b4d0 0x7f61e806d990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f61fc1518d0 tx=0x7f61f0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:46.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:46.898+0000 7f61e37fe640 1 -- 172.21.15.161:0/2442018894 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f61fc1ebb30 con 0x7f61fc0c5950 2024-03-28T23:30:47.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.112+0000 7f6201647640 1 -- 172.21.15.161:0/2442018894 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f61fc04f4b0 con 0x7f61fc0c5950 2024-03-28T23:30:47.113 INFO:tasks.ceph.mon.a.smithi161.stderr:2024-03-28T23:30:47.113+0000 7fdc7b85b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T23:30:47.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.114+0000 7f61e37fe640 1 -- 172.21.15.161:0/2442018894 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) ==== 161+0+0 (secure 0 0 0) 0x7f61ec030070 con 0x7f61fc0c5950 2024-03-28T23:30:47.115 INFO:teuthology.orchestra.run.smithi161.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-28T23:30:47.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.118+0000 7f6201647640 1 -- 172.21.15.161:0/2442018894 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f61e806b4d0 msgr2=0x7f61e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:47.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.118+0000 7f6201647640 1 --2- 172.21.15.161:0/2442018894 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f61e806b4d0 0x7f61e806d990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f61fc1518d0 tx=0x7f61f0008040 comp rx=0 tx=0).stop 2024-03-28T23:30:47.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.118+0000 7f6201647640 1 -- 172.21.15.161:0/2442018894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61fc0c5950 msgr2=0x7f61fc1eec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:47.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.118+0000 7f6201647640 1 --2- 172.21.15.161:0/2442018894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61fc0c5950 0x7f61fc1eec00 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f61ec036040 tx=0x7f61ec0046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:47.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.118+0000 7f6201647640 1 -- 172.21.15.161:0/2442018894 shutdown_connections 2024-03-28T23:30:47.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.118+0000 7f6201647640 1 --2- 172.21.15.161:0/2442018894 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f61e806b4d0 0x7f61e806d990 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:47.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.118+0000 7f6201647640 1 --2- 172.21.15.161:0/2442018894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f61fc152890 0x7f61fc0c9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:47.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.118+0000 7f6201647640 1 --2- 172.21.15.161:0/2442018894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f61fc0c8ed0 0x7f61fc1eb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:47.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.118+0000 7f6201647640 1 --2- 172.21.15.161:0/2442018894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61fc0c5950 0x7f61fc1eec00 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:47.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.118+0000 7f6201647640 1 -- 172.21.15.161:0/2442018894 >> 172.21.15.161:0/2442018894 conn(0x7f61fc0ac620 msgr2=0x7f61fc150fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:47.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.118+0000 7f6201647640 1 -- 172.21.15.161:0/2442018894 shutdown_connections 2024-03-28T23:30:47.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.118+0000 7f6201647640 1 -- 172.21.15.161:0/2442018894 wait complete. 2024-03-28T23:30:47.121 INFO:tasks.ceph.mds.c.smithi161.stderr: -3468> 2024-03-28T23:29:50.877+0000 7ff59132f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:30:47.132 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-28T23:30:47.152 INFO:tasks.ceph.mds.c.smithi161.stdout:starting mds.c at 2024-03-28T23:30:47.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.236+0000 7fcc4a38a640 1 Processor -- start 2024-03-28T23:30:47.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.236+0000 7fcc4a38a640 1 -- start start 2024-03-28T23:30:47.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.237+0000 7fcc4a38a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fcc44152890 0x7fcc440c04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:47.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.237+0000 7fcc4a38a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcc440c13c0 0x7fcc4406b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:47.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.237+0000 7fcc4a38a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc44155670 con 0x7fcc440670d0 2024-03-28T23:30:47.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.237+0000 7fcc4a38a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fcc4406bda0 con 0x7fcc440c13c0 2024-03-28T23:30:47.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.237+0000 7fcc4a38a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fcc44155950 con 0x7fcc44152890 2024-03-28T23:30:47.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.237+0000 7fcc48927640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcc440c13c0 0x7fcc4406b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:47.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.237+0000 7fcc437fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fcc440670d0 0x7fcc4406b150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:35050/0 (socket says 172.21.15.161:35050) 2024-03-28T23:30:47.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.237+0000 7fcc437fe640 1 -- 172.21.15.161:0/4195236022 learned_addr learned my addr 172.21.15.161:0/4195236022 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:47.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.237+0000 7fcc43fff640 1 --2- 172.21.15.161:0/4195236022 >> v2:172.21.15.170:3300/0 conn(0x7fcc44152890 0x7fcc440c04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:47.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.237+0000 7fcc48927640 1 -- 172.21.15.161:0/4195236022 >> v2:172.21.15.170:3300/0 conn(0x7fcc44152890 msgr2=0x7fcc440c04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:47.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.237+0000 7fcc48927640 1 --2- 172.21.15.161:0/4195236022 >> v2:172.21.15.170:3300/0 conn(0x7fcc44152890 0x7fcc440c04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:47.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.237+0000 7fcc48927640 1 -- 172.21.15.161:0/4195236022 >> v1:172.21.15.161:6789/0 conn(0x7fcc440670d0 legacy=0x7fcc4406b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:47.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.237+0000 7fcc48927640 1 -- 172.21.15.161:0/4195236022 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc44155c30 con 0x7fcc440c13c0 2024-03-28T23:30:47.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.237+0000 7fcc42ffd640 1 -- 172.21.15.161:0/4195236022 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2204921621 0 0) 0x7fcc44155670 con 0x7fcc440670d0 2024-03-28T23:30:47.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.238+0000 7fcc48927640 1 --2- 172.21.15.161:0/4195236022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc440c13c0 0x7fcc4406b860 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fcc440706d0 tx=0x7fcc3c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5326717cb562e0e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:47.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.238+0000 7fcc42ffd640 1 -- 172.21.15.161:0/4195236022 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3c039070 con 0x7fcc440c13c0 2024-03-28T23:30:47.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.238+0000 7fcc42ffd640 1 -- 172.21.15.161:0/4195236022 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc3c02cbf0 con 0x7fcc440c13c0 2024-03-28T23:30:47.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.238+0000 7fcc42ffd640 1 -- 172.21.15.161:0/4195236022 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3c034440 con 0x7fcc440c13c0 2024-03-28T23:30:47.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.238+0000 7fcc4a38a640 1 -- 172.21.15.161:0/4195236022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc440c13c0 msgr2=0x7fcc4406b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:47.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.239+0000 7fcc4a38a640 1 --2- 172.21.15.161:0/4195236022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc440c13c0 0x7fcc4406b860 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fcc440706d0 tx=0x7fcc3c02be40 comp rx=0 tx=0).stop 2024-03-28T23:30:47.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.239+0000 7fcc4a38a640 1 -- 172.21.15.161:0/4195236022 shutdown_connections 2024-03-28T23:30:47.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.239+0000 7fcc4a38a640 1 --2- 172.21.15.161:0/4195236022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc440c13c0 0x7fcc4406b860 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:47.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.239+0000 7fcc4a38a640 1 --2- 172.21.15.161:0/4195236022 >> v2:172.21.15.170:3300/0 conn(0x7fcc44152890 0x7fcc440c04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:47.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.239+0000 7fcc4a38a640 1 -- 172.21.15.161:0/4195236022 >> 172.21.15.161:0/4195236022 conn(0x7fcc440c3b70 msgr2=0x7fcc440c3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:47.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.239+0000 7fcc4a38a640 1 -- 172.21.15.161:0/4195236022 shutdown_connections 2024-03-28T23:30:47.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.239+0000 7fcc4a38a640 1 -- 172.21.15.161:0/4195236022 wait complete. 2024-03-28T23:30:47.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.240+0000 7fcc4a38a640 1 Processor -- start 2024-03-28T23:30:47.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.240+0000 7fcc4a38a640 1 -- start start 2024-03-28T23:30:47.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.241+0000 7fcc4a38a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcc440670d0 0x7fcc441eb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:47.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.241+0000 7fcc4a38a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc440c13c0 0x7fcc440c6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:47.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.241+0000 7fcc4a38a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcc44152890 0x7fcc440c5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:47.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.241+0000 7fcc4a38a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcc44152c70 con 0x7fcc440c13c0 2024-03-28T23:30:47.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.241+0000 7fcc4a38a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fcc44155320 con 0x7fcc44152890 2024-03-28T23:30:47.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.241+0000 7fcc4a38a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fcc440c1b50 con 0x7fcc440670d0 2024-03-28T23:30:47.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.241+0000 7fcc437fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc440c13c0 0x7fcc440c6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:47.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.241+0000 7fcc437fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc440c13c0 0x7fcc440c6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:52120/0 (socket says 172.21.15.161:52120) 2024-03-28T23:30:47.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.241+0000 7fcc437fe640 1 -- 172.21.15.161:0/1292547706 learned_addr learned my addr 172.21.15.161:0/1292547706 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:47.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.241+0000 7fcc48927640 1 --2- 172.21.15.161:0/1292547706 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcc44152890 0x7fcc440c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:47.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.241+0000 7fcc43fff640 1 --2- 172.21.15.161:0/1292547706 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcc440670d0 0x7fcc441eb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:47.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.241+0000 7fcc437fe640 1 -- 172.21.15.161:0/1292547706 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcc440670d0 msgr2=0x7fcc441eb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:47.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.241+0000 7fcc437fe640 1 --2- 172.21.15.161:0/1292547706 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcc440670d0 0x7fcc441eb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:47.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.241+0000 7fcc437fe640 1 -- 172.21.15.161:0/1292547706 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcc44152890 msgr2=0x7fcc440c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:47.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.241+0000 7fcc437fe640 1 --2- 172.21.15.161:0/1292547706 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcc44152890 0x7fcc440c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:47.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.241+0000 7fcc437fe640 1 -- 172.21.15.161:0/1292547706 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc441511a0 con 0x7fcc440c13c0 2024-03-28T23:30:47.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.241+0000 7fcc43fff640 1 --2- 172.21.15.161:0/1292547706 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcc440670d0 0x7fcc441eb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:47.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.241+0000 7fcc437fe640 1 --2- 172.21.15.161:0/1292547706 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc440c13c0 0x7fcc440c6240 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fcc3400c4b0 tx=0x7fcc3400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:47.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.241+0000 7fcc40ff9640 1 -- 172.21.15.161:0/1292547706 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc34018070 con 0x7fcc440c13c0 2024-03-28T23:30:47.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.241+0000 7fcc40ff9640 1 -- 172.21.15.161:0/1292547706 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc34007500 con 0x7fcc440c13c0 2024-03-28T23:30:47.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.241+0000 7fcc40ff9640 1 -- 172.21.15.161:0/1292547706 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc34004a20 con 0x7fcc440c13c0 2024-03-28T23:30:47.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.241+0000 7fcc4a38a640 1 -- 172.21.15.161:0/1292547706 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc44151440 con 0x7fcc440c13c0 2024-03-28T23:30:47.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.241+0000 7fcc4a38a640 1 -- 172.21.15.161:0/1292547706 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc440c4d00 con 0x7fcc440c13c0 2024-03-28T23:30:47.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.242+0000 7fcc4a38a640 1 -- 172.21.15.161:0/1292547706 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc4404f4b0 con 0x7fcc440c13c0 2024-03-28T23:30:47.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.243+0000 7fcc40ff9640 1 -- 172.21.15.161:0/1292547706 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcc3401b070 con 0x7fcc440c13c0 2024-03-28T23:30:47.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.243+0000 7fcc40ff9640 1 --2- 172.21.15.161:0/1292547706 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcc3006b4d0 0x7fcc3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:47.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.243+0000 7fcc43fff640 1 --2- 172.21.15.161:0/1292547706 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcc3006b4d0 0x7fcc3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:47.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.243+0000 7fcc40ff9640 1 -- 172.21.15.161:0/1292547706 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7fcc34089900 con 0x7fcc440c13c0 2024-03-28T23:30:47.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.244+0000 7fcc43fff640 1 --2- 172.21.15.161:0/1292547706 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcc3006b4d0 0x7fcc3006d990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcc38006fd0 tx=0x7fcc38008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:47.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.248+0000 7fcc40ff9640 1 -- 172.21.15.161:0/1292547706 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcc34058470 con 0x7fcc440c13c0 2024-03-28T23:30:47.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:47.458+0000 7fcc4a38a640 1 -- 172.21.15.161:0/1292547706 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fcc4406eae0 con 0x7fcc440c13c0 2024-03-28T23:30:48.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.116+0000 7fcc40ff9640 1 -- 172.21.15.161:0/1292547706 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7fcc34057d70 con 0x7fcc440c13c0 2024-03-28T23:30:48.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.119+0000 7fcc4a38a640 1 -- 172.21.15.161:0/1292547706 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcc3006b4d0 msgr2=0x7fcc3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:48.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.119+0000 7fcc4a38a640 1 --2- 172.21.15.161:0/1292547706 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcc3006b4d0 0x7fcc3006d990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcc38006fd0 tx=0x7fcc38008040 comp rx=0 tx=0).stop 2024-03-28T23:30:48.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.119+0000 7fcc4a38a640 1 -- 172.21.15.161:0/1292547706 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc440c13c0 msgr2=0x7fcc440c6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:48.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.119+0000 7fcc4a38a640 1 --2- 172.21.15.161:0/1292547706 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc440c13c0 0x7fcc440c6240 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fcc3400c4b0 tx=0x7fcc3400c970 comp rx=0 tx=0).stop 2024-03-28T23:30:48.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.119+0000 7fcc4a38a640 1 -- 172.21.15.161:0/1292547706 shutdown_connections 2024-03-28T23:30:48.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.119+0000 7fcc4a38a640 1 --2- 172.21.15.161:0/1292547706 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcc3006b4d0 0x7fcc3006d990 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:48.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.119+0000 7fcc4a38a640 1 --2- 172.21.15.161:0/1292547706 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc440c13c0 0x7fcc440c6240 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:48.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.119+0000 7fcc4a38a640 1 --2- 172.21.15.161:0/1292547706 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcc44152890 0x7fcc440c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:48.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.119+0000 7fcc4a38a640 1 --2- 172.21.15.161:0/1292547706 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcc440670d0 0x7fcc441eb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:48.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.119+0000 7fcc4a38a640 1 -- 172.21.15.161:0/1292547706 >> 172.21.15.161:0/1292547706 conn(0x7fcc440c3b70 msgr2=0x7fcc440714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:48.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.119+0000 7fcc4a38a640 1 -- 172.21.15.161:0/1292547706 shutdown_connections 2024-03-28T23:30:48.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.119+0000 7fcc4a38a640 1 -- 172.21.15.161:0/1292547706 wait complete. 2024-03-28T23:30:48.134 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-28T23:30:48.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.285+0000 7f1aee0c0640 1 Processor -- start 2024-03-28T23:30:48.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.285+0000 7f1aee0c0640 1 -- start start 2024-03-28T23:30:48.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.285+0000 7f1aee0c0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1ae80c6e40 0x7f1ae8160be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:48.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.285+0000 7f1aee0c0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ae80c5da0 con 0x7f1ae8155e20 2024-03-28T23:30:48.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.285+0000 7f1aee0c0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ae80c6080 con 0x7f1ae8152950 2024-03-28T23:30:48.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.285+0000 7f1aee0c0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1ae80c9350 con 0x7f1ae80c6e40 2024-03-28T23:30:48.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.285+0000 7f1ae7fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1ae80c6e40 0x7f1ae8160be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:48.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.286+0000 7f1ae7fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1ae80c6e40 0x7f1ae8160be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:52130/0 (socket says 172.21.15.161:52130) 2024-03-28T23:30:48.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.286+0000 7f1ae7fff640 1 -- 172.21.15.161:0/3715901554 learned_addr learned my addr 172.21.15.161:0/3715901554 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:48.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.286+0000 7f1ae6ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f1ae8155e20 0x7f1ae806bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:35058/0 (socket says 172.21.15.161:35058) 2024-03-28T23:30:48.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.286+0000 7f1ae7fff640 1 -- 172.21.15.161:0/3715901554 >> v1:172.21.15.170:6789/0 conn(0x7f1ae8152950 legacy=0x7f1ae80c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:48.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.286+0000 7f1ae7fff640 1 -- 172.21.15.161:0/3715901554 >> v1:172.21.15.161:6789/0 conn(0x7f1ae8155e20 legacy=0x7f1ae806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:48.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.286+0000 7f1ae7fff640 1 -- 172.21.15.161:0/3715901554 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae80c6360 con 0x7f1ae80c6e40 2024-03-28T23:30:48.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.286+0000 7f1ae67fc640 1 -- 172.21.15.161:0/3715901554 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1176554587 0 0) 0x7f1ae80c5da0 con 0x7f1ae8155e20 2024-03-28T23:30:48.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.287+0000 7f1ae7fff640 1 --2- 172.21.15.161:0/3715901554 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ae80c6e40 0x7f1ae8160be0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009e50 tx=0x7f1ae002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c379c839d2aca61e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:48.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.287+0000 7f1ae67fc640 1 -- 172.21.15.161:0/3715901554 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae002cb20 con 0x7f1ae80c6e40 2024-03-28T23:30:48.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.287+0000 7f1ae67fc640 1 -- 172.21.15.161:0/3715901554 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ae002ccc0 con 0x7f1ae80c6e40 2024-03-28T23:30:48.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.287+0000 7f1ae67fc640 1 -- 172.21.15.161:0/3715901554 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae0034610 con 0x7f1ae80c6e40 2024-03-28T23:30:48.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.287+0000 7f1aee0c0640 1 -- 172.21.15.161:0/3715901554 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ae80c6e40 msgr2=0x7f1ae8160be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:48.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.287+0000 7f1aee0c0640 1 --2- 172.21.15.161:0/3715901554 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ae80c6e40 0x7f1ae8160be0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009e50 tx=0x7f1ae002bf80 comp rx=0 tx=0).stop 2024-03-28T23:30:48.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.287+0000 7f1aee0c0640 1 -- 172.21.15.161:0/3715901554 shutdown_connections 2024-03-28T23:30:48.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.287+0000 7f1aee0c0640 1 --2- 172.21.15.161:0/3715901554 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ae80c6e40 0x7f1ae8160be0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:48.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.287+0000 7f1aee0c0640 1 -- 172.21.15.161:0/3715901554 >> 172.21.15.161:0/3715901554 conn(0x7f1ae80c7690 msgr2=0x7f1ae80c7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:48.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.287+0000 7f1aee0c0640 1 -- 172.21.15.161:0/3715901554 shutdown_connections 2024-03-28T23:30:48.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.287+0000 7f1aee0c0640 1 -- 172.21.15.161:0/3715901554 wait complete. 2024-03-28T23:30:48.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.288+0000 7f1aee0c0640 1 Processor -- start 2024-03-28T23:30:48.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.289+0000 7f1aee0c0640 1 -- start start 2024-03-28T23:30:48.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.289+0000 7f1aee0c0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ae80c6e40 0x7f1ae81ee440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:48.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.289+0000 7f1aee0c0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ae8152950 0x7f1ae8153920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:48.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.289+0000 7f1aee0c0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ae8155e20 0x7f1ae8153e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:48.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.289+0000 7f1aee0c0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae80c69b0 con 0x7f1ae80c6e40 2024-03-28T23:30:48.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.289+0000 7f1aee0c0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae80c72b0 con 0x7f1ae8152950 2024-03-28T23:30:48.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.289+0000 7f1aee0c0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1ae8152d30 con 0x7f1ae8155e20 2024-03-28T23:30:48.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.289+0000 7f1ae77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ae80c6e40 0x7f1ae81ee440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:48.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.289+0000 7f1ae77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ae80c6e40 0x7f1ae81ee440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:52140/0 (socket says 172.21.15.161:52140) 2024-03-28T23:30:48.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.289+0000 7f1ae77fe640 1 -- 172.21.15.161:0/4150599211 learned_addr learned my addr 172.21.15.161:0/4150599211 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:48.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.290+0000 7f1ae7fff640 1 --2- 172.21.15.161:0/4150599211 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ae8155e20 0x7f1ae8153e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:48.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.290+0000 7f1ae6ffd640 1 --2- 172.21.15.161:0/4150599211 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ae8152950 0x7f1ae8153920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:48.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.290+0000 7f1ae77fe640 1 -- 172.21.15.161:0/4150599211 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ae8155e20 msgr2=0x7f1ae8153e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:48.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.290+0000 7f1ae77fe640 1 --2- 172.21.15.161:0/4150599211 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ae8155e20 0x7f1ae8153e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:48.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.290+0000 7f1ae77fe640 1 -- 172.21.15.161:0/4150599211 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ae8152950 msgr2=0x7f1ae8153920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:48.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.290+0000 7f1ae77fe640 1 --2- 172.21.15.161:0/4150599211 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ae8152950 0x7f1ae8153920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:48.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.290+0000 7f1ae77fe640 1 -- 172.21.15.161:0/4150599211 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae8154cf0 con 0x7f1ae80c6e40 2024-03-28T23:30:48.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.290+0000 7f1ae7fff640 1 --2- 172.21.15.161:0/4150599211 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ae8155e20 0x7f1ae8153e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:48.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.290+0000 7f1ae6ffd640 1 --2- 172.21.15.161:0/4150599211 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ae8152950 0x7f1ae8153920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:48.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.290+0000 7f1ae77fe640 1 --2- 172.21.15.161:0/4150599211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ae80c6e40 0x7f1ae81ee440 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1ad800c480 tx=0x7f1ad800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:48.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.290+0000 7f1acffff640 1 -- 172.21.15.161:0/4150599211 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad80052d0 con 0x7f1ae80c6e40 2024-03-28T23:30:48.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.290+0000 7f1acffff640 1 -- 172.21.15.161:0/4150599211 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ad8005470 con 0x7f1ae80c6e40 2024-03-28T23:30:48.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.290+0000 7f1acffff640 1 -- 172.21.15.161:0/4150599211 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad8012870 con 0x7f1ae80c6e40 2024-03-28T23:30:48.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.290+0000 7f1aee0c0640 1 -- 172.21.15.161:0/4150599211 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ae81543a0 con 0x7f1ae80c6e40 2024-03-28T23:30:48.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.290+0000 7f1aee0c0640 1 -- 172.21.15.161:0/4150599211 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ae80aa490 con 0x7f1ae80c6e40 2024-03-28T23:30:48.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.291+0000 7f1aee0c0640 1 -- 172.21.15.161:0/4150599211 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ae8068f90 con 0x7f1ae80c6e40 2024-03-28T23:30:48.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.292+0000 7f1acffff640 1 -- 172.21.15.161:0/4150599211 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1ad8012a10 con 0x7f1ae80c6e40 2024-03-28T23:30:48.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.292+0000 7f1acffff640 1 --2- 172.21.15.161:0/4150599211 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1ad406b4d0 0x7f1ad406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:48.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.292+0000 7f1acffff640 1 -- 172.21.15.161:0/4150599211 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f1ad808a2d0 con 0x7f1ae80c6e40 2024-03-28T23:30:48.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.292+0000 7f1ae6ffd640 1 --2- 172.21.15.161:0/4150599211 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1ad406b4d0 0x7f1ad406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:48.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.292+0000 7f1ae6ffd640 1 --2- 172.21.15.161:0/4150599211 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1ad406b4d0 0x7f1ad406d990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8159960 tx=0x7f1adc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:48.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.296+0000 7f1acffff640 1 -- 172.21.15.161:0/4150599211 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1ad8058d00 con 0x7f1ae80c6e40 2024-03-28T23:30:48.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:48.470+0000 7f1aee0c0640 1 -- 172.21.15.161:0/4150599211 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1ae806b740 con 0x7f1ae80c6e40 2024-03-28T23:30:49.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.211+0000 7f1acffff640 1 -- 172.21.15.161:0/4150599211 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v28) ==== 181+0+0 (secure 0 0 0) 0x7f1ad8058600 con 0x7f1ae80c6e40 2024-03-28T23:30:49.212 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_metadata' removed 2024-03-28T23:30:49.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.213+0000 7f1aee0c0640 1 -- 172.21.15.161:0/4150599211 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1ad406b4d0 msgr2=0x7f1ad406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:49.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.213+0000 7f1aee0c0640 1 --2- 172.21.15.161:0/4150599211 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1ad406b4d0 0x7f1ad406d990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8159960 tx=0x7f1adc009040 comp rx=0 tx=0).stop 2024-03-28T23:30:49.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.213+0000 7f1aee0c0640 1 -- 172.21.15.161:0/4150599211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ae80c6e40 msgr2=0x7f1ae81ee440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:49.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.213+0000 7f1aee0c0640 1 --2- 172.21.15.161:0/4150599211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ae80c6e40 0x7f1ae81ee440 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1ad800c480 tx=0x7f1ad800c940 comp rx=0 tx=0).stop 2024-03-28T23:30:49.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.215+0000 7f1aee0c0640 1 -- 172.21.15.161:0/4150599211 shutdown_connections 2024-03-28T23:30:49.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.215+0000 7f1aee0c0640 1 --2- 172.21.15.161:0/4150599211 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1ad406b4d0 0x7f1ad406d990 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:49.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.215+0000 7f1aee0c0640 1 --2- 172.21.15.161:0/4150599211 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ae8155e20 0x7f1ae8153e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:49.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.215+0000 7f1aee0c0640 1 --2- 172.21.15.161:0/4150599211 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ae8152950 0x7f1ae8153920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:49.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.215+0000 7f1aee0c0640 1 --2- 172.21.15.161:0/4150599211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ae80c6e40 0x7f1ae81ee440 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:49.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.215+0000 7f1aee0c0640 1 -- 172.21.15.161:0/4150599211 >> 172.21.15.161:0/4150599211 conn(0x7f1ae80c7690 msgr2=0x7f1ae8151bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:49.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.217+0000 7f1aee0c0640 1 -- 172.21.15.161:0/4150599211 shutdown_connections 2024-03-28T23:30:49.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.217+0000 7f1aee0c0640 1 -- 172.21.15.161:0/4150599211 wait complete. 2024-03-28T23:30:49.233 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-28T23:30:49.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.341+0000 7f1c3595c640 1 Processor -- start 2024-03-28T23:30:49.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.341+0000 7f1c3595c640 1 -- start start 2024-03-28T23:30:49.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.342+0000 7f1c3595c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1c3014ef80 0x7f1c3006cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:49.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.342+0000 7f1c3595c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c30151730 con 0x7f1c3006d4d0 2024-03-28T23:30:49.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.342+0000 7f1c3595c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c3014f360 con 0x7f1c300c1460 2024-03-28T23:30:49.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.342+0000 7f1c3595c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f1c300c0eb0 con 0x7f1c3014ef80 2024-03-28T23:30:49.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.342+0000 7f1c2e7fc640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f1c3006d4d0 0x7f1c30151c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:35062/0 (socket says 172.21.15.161:35062) 2024-03-28T23:30:49.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.342+0000 7f1c2e7fc640 1 -- 172.21.15.161:0/1752368611 learned_addr learned my addr 172.21.15.161:0/1752368611 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:49.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.342+0000 7f1c2effd640 1 --2- 172.21.15.161:0/1752368611 >> v2:172.21.15.170:3300/0 conn(0x7f1c3014ef80 0x7f1c3006cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:49.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.342+0000 7f1c2dffb640 1 -- 172.21.15.161:0/1752368611 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3237848677 0 0) 0x7f1c30151730 con 0x7f1c3006d4d0 2024-03-28T23:30:49.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.342+0000 7f1c2dffb640 1 -- 172.21.15.161:0/1752368611 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c18003520 con 0x7f1c3006d4d0 2024-03-28T23:30:49.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.343+0000 7f1c2dffb640 1 -- 172.21.15.161:0/1752368611 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1214441686 0 0) 0x7f1c18003520 con 0x7f1c3006d4d0 2024-03-28T23:30:49.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.343+0000 7f1c2dffb640 1 -- 172.21.15.161:0/1752368611 >> v2:172.21.15.170:3300/0 conn(0x7f1c3014ef80 msgr2=0x7f1c3006cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:49.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.343+0000 7f1c2dffb640 1 --2- 172.21.15.161:0/1752368611 >> v2:172.21.15.170:3300/0 conn(0x7f1c3014ef80 0x7f1c3006cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:49.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.343+0000 7f1c2dffb640 1 -- 172.21.15.161:0/1752368611 >> v1:172.21.15.170:6789/0 conn(0x7f1c300c1460 legacy=0x7f1c3015bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:49.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.343+0000 7f1c2dffb640 1 -- 172.21.15.161:0/1752368611 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c300c0bd0 con 0x7f1c3006d4d0 2024-03-28T23:30:49.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.343+0000 7f1c2dffb640 1 -- 172.21.15.161:0/1752368611 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f1c24001670 con 0x7f1c3006d4d0 2024-03-28T23:30:49.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.343+0000 7f1c2dffb640 1 -- 172.21.15.161:0/1752368611 >> v1:172.21.15.161:6789/0 conn(0x7f1c3006d4d0 legacy=0x7f1c30151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:49.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.343+0000 7f1c2dffb640 1 --2- 172.21.15.161:0/1752368611 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c180039b0 0x7f1c18003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:49.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.343+0000 7f1c2dffb640 1 --2- 172.21.15.161:0/1752368611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c18004800 0x7f1c18004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:49.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.343+0000 7f1c2dffb640 1 --2- 172.21.15.161:0/1752368611 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c180052e0 0x7f1c1800dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:49.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.343+0000 7f1c2dffb640 1 -- 172.21.15.161:0/1752368611 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1c300c0eb0 con 0x7f1c180052e0 2024-03-28T23:30:49.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.343+0000 7f1c2dffb640 1 -- 172.21.15.161:0/1752368611 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1800e3c0 con 0x7f1c180039b0 2024-03-28T23:30:49.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.343+0000 7f1c2dffb640 1 -- 172.21.15.161:0/1752368611 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1c1800e5c0 con 0x7f1c18004800 2024-03-28T23:30:49.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.343+0000 7f1c2f7fe640 1 --2- 172.21.15.161:0/1752368611 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c180052e0 0x7f1c1800dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:49.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.343+0000 7f1c2effd640 1 --2- 172.21.15.161:0/1752368611 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c180039b0 0x7f1c18003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:49.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.343+0000 7f1c2e7fc640 1 --2- 172.21.15.161:0/1752368611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c18004800 0x7f1c18004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:49.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.343+0000 7f1c2f7fe640 1 -- 172.21.15.161:0/1752368611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c18004800 msgr2=0x7f1c18004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:49.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.343+0000 7f1c2f7fe640 1 --2- 172.21.15.161:0/1752368611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c18004800 0x7f1c18004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:49.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.343+0000 7f1c2f7fe640 1 -- 172.21.15.161:0/1752368611 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c180039b0 msgr2=0x7f1c18003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:49.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.343+0000 7f1c2f7fe640 1 --2- 172.21.15.161:0/1752368611 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c180039b0 0x7f1c18003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:49.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.343+0000 7f1c2f7fe640 1 -- 172.21.15.161:0/1752368611 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c18003520 con 0x7f1c180052e0 2024-03-28T23:30:49.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.343+0000 7f1c2e7fc640 1 --2- 172.21.15.161:0/1752368611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c18004800 0x7f1c18004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:49.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.344+0000 7f1c2f7fe640 1 -- 172.21.15.161:0/1752368611 reap_dead start 2024-03-28T23:30:49.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.344+0000 7f1c2f7fe640 1 --2- 172.21.15.161:0/1752368611 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c180052e0 0x7f1c1800dbd0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1c28001e20 tx=0x7f1c18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fb870124caba81a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:49.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.344+0000 7f1c2dffb640 1 -- 172.21.15.161:0/1752368611 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c28001a70 con 0x7f1c180052e0 2024-03-28T23:30:49.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.344+0000 7f1c2dffb640 1 -- 172.21.15.161:0/1752368611 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c28003b70 con 0x7f1c180052e0 2024-03-28T23:30:49.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.344+0000 7f1c2dffb640 1 -- 172.21.15.161:0/1752368611 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c280050d0 con 0x7f1c180052e0 2024-03-28T23:30:49.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.345+0000 7f1c3595c640 1 -- 172.21.15.161:0/1752368611 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c180052e0 msgr2=0x7f1c1800dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:49.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.345+0000 7f1c3595c640 1 --2- 172.21.15.161:0/1752368611 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c180052e0 0x7f1c1800dbd0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1c28001e20 tx=0x7f1c18005830 comp rx=0 tx=0).stop 2024-03-28T23:30:49.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.345+0000 7f1c3595c640 1 -- 172.21.15.161:0/1752368611 shutdown_connections 2024-03-28T23:30:49.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.345+0000 7f1c3595c640 1 --2- 172.21.15.161:0/1752368611 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c180052e0 0x7f1c1800dbd0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:49.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.345+0000 7f1c3595c640 1 -- 172.21.15.161:0/1752368611 >> 172.21.15.161:0/1752368611 conn(0x7f1c3006b150 msgr2=0x7f1c3006b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:49.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.345+0000 7f1c3595c640 1 -- 172.21.15.161:0/1752368611 shutdown_connections 2024-03-28T23:30:49.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.345+0000 7f1c3595c640 1 -- 172.21.15.161:0/1752368611 wait complete. 2024-03-28T23:30:49.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.346+0000 7f1c3595c640 1 Processor -- start 2024-03-28T23:30:49.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.346+0000 7f1c3595c640 1 -- start start 2024-03-28T23:30:49.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.347+0000 7f1c3595c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c180052e0 0x7f1c300c3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:49.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.347+0000 7f1c3595c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c300a7d40 0x7f1c300a8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:49.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.347+0000 7f1c3595c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c301ea9c0 0x7f1c301eae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:49.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.347+0000 7f1c3595c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1c300c46b0 con 0x7f1c180052e0 2024-03-28T23:30:49.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.347+0000 7f1c3595c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3006e4d0 con 0x7f1c300a7d40 2024-03-28T23:30:49.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.347+0000 7f1c3595c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1c300c11b0 con 0x7f1c301ea9c0 2024-03-28T23:30:49.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.347+0000 7f1c2effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c180052e0 0x7f1c300c3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:49.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.347+0000 7f1c2effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c180052e0 0x7f1c300c3050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:52154/0 (socket says 172.21.15.161:52154) 2024-03-28T23:30:49.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.347+0000 7f1c2effd640 1 -- 172.21.15.161:0/3548291276 learned_addr learned my addr 172.21.15.161:0/3548291276 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:49.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.347+0000 7f1c2effd640 1 -- 172.21.15.161:0/3548291276 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c301ea9c0 msgr2=0x7f1c301eae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:49.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.347+0000 7f1c2effd640 1 --2- 172.21.15.161:0/3548291276 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c301ea9c0 0x7f1c301eae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:49.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.347+0000 7f1c2effd640 1 -- 172.21.15.161:0/3548291276 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c300a7d40 msgr2=0x7f1c300a8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:49.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.347+0000 7f1c2effd640 1 --2- 172.21.15.161:0/3548291276 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c300a7d40 0x7f1c300a8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:49.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.347+0000 7f1c2effd640 1 -- 172.21.15.161:0/3548291276 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c301eee00 con 0x7f1c180052e0 2024-03-28T23:30:49.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.347+0000 7f1c2effd640 1 --2- 172.21.15.161:0/3548291276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c180052e0 0x7f1c300c3050 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f1c20011910 tx=0x7f1c20011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:49.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.347+0000 7f1c17fff640 1 -- 172.21.15.161:0/3548291276 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2000faf0 con 0x7f1c180052e0 2024-03-28T23:30:49.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.347+0000 7f1c17fff640 1 -- 172.21.15.161:0/3548291276 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c2000fc90 con 0x7f1c180052e0 2024-03-28T23:30:49.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.347+0000 7f1c3595c640 1 -- 172.21.15.161:0/3548291276 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c301eeb20 con 0x7f1c180052e0 2024-03-28T23:30:49.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.347+0000 7f1c17fff640 1 -- 172.21.15.161:0/3548291276 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c20014440 con 0x7f1c180052e0 2024-03-28T23:30:49.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.347+0000 7f1c3595c640 1 -- 172.21.15.161:0/3548291276 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c300a8f50 con 0x7f1c180052e0 2024-03-28T23:30:49.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.348+0000 7f1c3595c640 1 -- 172.21.15.161:0/3548291276 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c30068f90 con 0x7f1c180052e0 2024-03-28T23:30:49.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.349+0000 7f1c17fff640 1 -- 172.21.15.161:0/3548291276 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1c20012040 con 0x7f1c180052e0 2024-03-28T23:30:49.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.349+0000 7f1c17fff640 1 --2- 172.21.15.161:0/3548291276 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1c1c06b500 0x7f1c1c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:49.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.349+0000 7f1c17fff640 1 -- 172.21.15.161:0/3548291276 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6383+0+0 (secure 0 0 0) 0x7f1c2008a980 con 0x7f1c180052e0 2024-03-28T23:30:49.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.349+0000 7f1c2e7fc640 1 --2- 172.21.15.161:0/3548291276 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1c1c06b500 0x7f1c1c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:49.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.350+0000 7f1c2e7fc640 1 --2- 172.21.15.161:0/3548291276 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1c1c06b500 0x7f1c1c06d9c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1c24001b80 tx=0x7f1c24001060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:49.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.354+0000 7f1c17fff640 1 -- 172.21.15.161:0/3548291276 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1c20019030 con 0x7f1c180052e0 2024-03-28T23:30:49.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:49.525+0000 7f1c3595c640 1 -- 172.21.15.161:0/3548291276 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1c30151730 con 0x7f1c180052e0 2024-03-28T23:30:50.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.215+0000 7f1c17fff640 1 -- 172.21.15.161:0/3548291276 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v29) ==== 169+0+0 (secure 0 0 0) 0x7f1c20058fa0 con 0x7f1c180052e0 2024-03-28T23:30:50.216 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data' removed 2024-03-28T23:30:50.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.219+0000 7f1c3595c640 1 -- 172.21.15.161:0/3548291276 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1c1c06b500 msgr2=0x7f1c1c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:50.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.219+0000 7f1c3595c640 1 --2- 172.21.15.161:0/3548291276 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1c1c06b500 0x7f1c1c06d9c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1c24001b80 tx=0x7f1c24001060 comp rx=0 tx=0).stop 2024-03-28T23:30:50.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.219+0000 7f1c3595c640 1 -- 172.21.15.161:0/3548291276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c180052e0 msgr2=0x7f1c300c3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:50.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.219+0000 7f1c3595c640 1 --2- 172.21.15.161:0/3548291276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c180052e0 0x7f1c300c3050 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f1c20011910 tx=0x7f1c20011dd0 comp rx=0 tx=0).stop 2024-03-28T23:30:50.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.219+0000 7f1c3595c640 1 -- 172.21.15.161:0/3548291276 shutdown_connections 2024-03-28T23:30:50.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.219+0000 7f1c3595c640 1 --2- 172.21.15.161:0/3548291276 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1c1c06b500 0x7f1c1c06d9c0 secure :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1c24001b80 tx=0x7f1c24001060 comp rx=0 tx=0).stop 2024-03-28T23:30:50.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.219+0000 7f1c3595c640 1 --2- 172.21.15.161:0/3548291276 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c301ea9c0 0x7f1c301eae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:50.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.219+0000 7f1c3595c640 1 --2- 172.21.15.161:0/3548291276 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c300a7d40 0x7f1c300a8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:50.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.219+0000 7f1c3595c640 1 --2- 172.21.15.161:0/3548291276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c180052e0 0x7f1c300c3050 secure :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f1c20011910 tx=0x7f1c20011dd0 comp rx=0 tx=0).stop 2024-03-28T23:30:50.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.219+0000 7f1c3595c640 1 -- 172.21.15.161:0/3548291276 >> 172.21.15.161:0/3548291276 conn(0x7f1c3006b150 msgr2=0x7f1c300c56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:50.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.221+0000 7f1c3595c640 1 -- 172.21.15.161:0/3548291276 shutdown_connections 2024-03-28T23:30:50.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.222+0000 7f1c3595c640 1 -- 172.21.15.161:0/3548291276 wait complete. 2024-03-28T23:30:50.236 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-28T23:30:50.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.342+0000 7f9e07cc2640 1 Processor -- start 2024-03-28T23:30:50.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.342+0000 7f9e07cc2640 1 -- start start 2024-03-28T23:30:50.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.343+0000 7f9e07cc2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9e0014f1d0 0x7f9e00151db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:50.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.343+0000 7f9e07cc2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e000c3080 0x7f9e000c3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:50.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.343+0000 7f9e07cc2640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e000c15a0 con 0x7f9e000c3fd0 2024-03-28T23:30:50.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.343+0000 7f9e07cc2640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f9e000c6330 con 0x7f9e0014f1d0 2024-03-28T23:30:50.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.343+0000 7f9e07cc2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9e000c1840 con 0x7f9e000c3080 2024-03-28T23:30:50.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.343+0000 7f9e05a5e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9e0014f1d0 0x7f9e00151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:50.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.343+0000 7f9e0625f640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f9e000c3fd0 0x7f9e0006cab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:35072/0 (socket says 172.21.15.161:35072) 2024-03-28T23:30:50.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.343+0000 7f9e0625f640 1 -- 172.21.15.161:0/3436934355 learned_addr learned my addr 172.21.15.161:0/3436934355 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:50.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.343+0000 7f9e0525d640 1 --2- 172.21.15.161:0/3436934355 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e000c3080 0x7f9e000c3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:50.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.343+0000 7f9e04a5c640 1 -- 172.21.15.161:0/3436934355 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4260539239 0 0) 0x7f9e000c15a0 con 0x7f9e000c3fd0 2024-03-28T23:30:50.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.343+0000 7f9e04a5c640 1 -- 172.21.15.161:0/3436934355 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9dec003420 con 0x7f9e000c3fd0 2024-03-28T23:30:50.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.343+0000 7f9e05a5e640 1 -- 172.21.15.161:0/3436934355 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e000c3080 msgr2=0x7f9e000c3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:50.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.343+0000 7f9e05a5e640 1 --2- 172.21.15.161:0/3436934355 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e000c3080 0x7f9e000c3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:50.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.343+0000 7f9e05a5e640 1 -- 172.21.15.161:0/3436934355 >> v1:172.21.15.161:6789/0 conn(0x7f9e000c3fd0 legacy=0x7f9e0006cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:50.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.343+0000 7f9e05a5e640 1 -- 172.21.15.161:0/3436934355 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e000c1b20 con 0x7f9e0014f1d0 2024-03-28T23:30:50.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.344+0000 7f9e05a5e640 1 --2- 172.21.15.161:0/3436934355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e0014f1d0 0x7f9e00151db0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f9df00097d0 tx=0x7f9df002b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb57cf633684fcaf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:50.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.344+0000 7f9e04a5c640 1 -- 172.21.15.161:0/3436934355 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df0031770 con 0x7f9e0014f1d0 2024-03-28T23:30:50.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.344+0000 7f9e04a5c640 1 -- 172.21.15.161:0/3436934355 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9df0037070 con 0x7f9e0014f1d0 2024-03-28T23:30:50.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.344+0000 7f9e04a5c640 1 -- 172.21.15.161:0/3436934355 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df0031b30 con 0x7f9e0014f1d0 2024-03-28T23:30:50.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.345+0000 7f9e07cc2640 1 -- 172.21.15.161:0/3436934355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e0014f1d0 msgr2=0x7f9e00151db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:50.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.345+0000 7f9e07cc2640 1 --2- 172.21.15.161:0/3436934355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e0014f1d0 0x7f9e00151db0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f9df00097d0 tx=0x7f9df002b810 comp rx=0 tx=0).stop 2024-03-28T23:30:50.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.345+0000 7f9e07cc2640 1 -- 172.21.15.161:0/3436934355 shutdown_connections 2024-03-28T23:30:50.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.345+0000 7f9e07cc2640 1 --2- 172.21.15.161:0/3436934355 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e000c3080 0x7f9e000c3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:50.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.345+0000 7f9e07cc2640 1 --2- 172.21.15.161:0/3436934355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e0014f1d0 0x7f9e00151db0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:50.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.345+0000 7f9e07cc2640 1 -- 172.21.15.161:0/3436934355 >> 172.21.15.161:0/3436934355 conn(0x7f9e000c6e20 msgr2=0x7f9e000c7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:50.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.345+0000 7f9e07cc2640 1 -- 172.21.15.161:0/3436934355 shutdown_connections 2024-03-28T23:30:50.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.345+0000 7f9e07cc2640 1 -- 172.21.15.161:0/3436934355 wait complete. 2024-03-28T23:30:50.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.346+0000 7f9e07cc2640 1 Processor -- start 2024-03-28T23:30:50.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.346+0000 7f9e07cc2640 1 -- start start 2024-03-28T23:30:50.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.346+0000 7f9e07cc2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e000c3080 0x7f9e000a9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:50.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.346+0000 7f9e07cc2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9e000c3fd0 0x7f9e001ea7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:50.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.346+0000 7f9e07cc2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e0014f1d0 0x7f9e00150d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:50.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.346+0000 7f9e07cc2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0014f5b0 con 0x7f9e0014f1d0 2024-03-28T23:30:50.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.346+0000 7f9e07cc2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9e000c2600 con 0x7f9e000c3fd0 2024-03-28T23:30:50.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.346+0000 7f9e07cc2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9e0014f730 con 0x7f9e000c3080 2024-03-28T23:30:50.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.347+0000 7f9e0625f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e0014f1d0 0x7f9e00150d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:50.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.347+0000 7f9e0625f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e0014f1d0 0x7f9e00150d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:52176/0 (socket says 172.21.15.161:52176) 2024-03-28T23:30:50.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.347+0000 7f9e0625f640 1 -- 172.21.15.161:0/1331195220 learned_addr learned my addr 172.21.15.161:0/1331195220 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:50.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.347+0000 7f9e05a5e640 1 --2- 172.21.15.161:0/1331195220 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e000c3080 0x7f9e000a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:50.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.347+0000 7f9e0625f640 1 -- 172.21.15.161:0/1331195220 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e000c3080 msgr2=0x7f9e000a9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:50.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.347+0000 7f9e0625f640 1 --2- 172.21.15.161:0/1331195220 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e000c3080 0x7f9e000a9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:50.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.347+0000 7f9e0525d640 1 --2- 172.21.15.161:0/1331195220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9e000c3fd0 0x7f9e001ea7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:50.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.347+0000 7f9e0625f640 1 -- 172.21.15.161:0/1331195220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9e000c3fd0 msgr2=0x7f9e001ea7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:50.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.347+0000 7f9e0625f640 1 --2- 172.21.15.161:0/1331195220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9e000c3fd0 0x7f9e001ea7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:50.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.347+0000 7f9e0625f640 1 -- 172.21.15.161:0/1331195220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e000a91d0 con 0x7f9e0014f1d0 2024-03-28T23:30:50.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.347+0000 7f9e05a5e640 1 --2- 172.21.15.161:0/1331195220 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e000c3080 0x7f9e000a9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:50.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.347+0000 7f9e0625f640 1 --2- 172.21.15.161:0/1331195220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e0014f1d0 0x7f9e00150d60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9df800cb10 tx=0x7f9df800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:50.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.347+0000 7f9df67fc640 1 -- 172.21.15.161:0/1331195220 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df8017070 con 0x7f9e0014f1d0 2024-03-28T23:30:50.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.347+0000 7f9df67fc640 1 -- 172.21.15.161:0/1331195220 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9df800ac70 con 0x7f9e0014f1d0 2024-03-28T23:30:50.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.347+0000 7f9df67fc640 1 -- 172.21.15.161:0/1331195220 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df800ae60 con 0x7f9e0014f1d0 2024-03-28T23:30:50.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.347+0000 7f9e07cc2640 1 -- 172.21.15.161:0/1331195220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e000a94b0 con 0x7f9e0014f1d0 2024-03-28T23:30:50.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.347+0000 7f9e07cc2640 1 -- 172.21.15.161:0/1331195220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e001dcba0 con 0x7f9e0014f1d0 2024-03-28T23:30:50.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.348+0000 7f9e07cc2640 1 -- 172.21.15.161:0/1331195220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e00068f90 con 0x7f9e0014f1d0 2024-03-28T23:30:50.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.349+0000 7f9df67fc640 1 -- 172.21.15.161:0/1331195220 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9df801a070 con 0x7f9e0014f1d0 2024-03-28T23:30:50.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.349+0000 7f9df67fc640 1 --2- 172.21.15.161:0/1331195220 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9de406b4d0 0x7f9de406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:50.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.349+0000 7f9df67fc640 1 -- 172.21.15.161:0/1331195220 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 5853+0+0 (secure 0 0 0) 0x7f9df8089f30 con 0x7f9e0014f1d0 2024-03-28T23:30:50.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.349+0000 7f9e05a5e640 1 --2- 172.21.15.161:0/1331195220 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9de406b4d0 0x7f9de406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:50.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.349+0000 7f9e05a5e640 1 --2- 172.21.15.161:0/1331195220 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9de406b4d0 0x7f9de406d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9df0036000 tx=0x7f9df0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:50.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.353+0000 7f9df67fc640 1 -- 172.21.15.161:0/1331195220 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9df8058630 con 0x7f9e0014f1d0 2024-03-28T23:30:50.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:50.525+0000 7f9e07cc2640 1 -- 172.21.15.161:0/1331195220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9e000c2c50 con 0x7f9e0014f1d0 2024-03-28T23:30:51.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.216+0000 7f9df67fc640 1 -- 172.21.15.161:0/1331195220 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v30) ==== 178+0+0 (secure 0 0 0) 0x7f9df805c6d0 con 0x7f9e0014f1d0 2024-03-28T23:30:51.217 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data_ec' removed 2024-03-28T23:30:51.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.219+0000 7f9e07cc2640 1 -- 172.21.15.161:0/1331195220 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9de406b4d0 msgr2=0x7f9de406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:51.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.219+0000 7f9e07cc2640 1 --2- 172.21.15.161:0/1331195220 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9de406b4d0 0x7f9de406d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9df0036000 tx=0x7f9df0008030 comp rx=0 tx=0).stop 2024-03-28T23:30:51.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.219+0000 7f9e07cc2640 1 -- 172.21.15.161:0/1331195220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e0014f1d0 msgr2=0x7f9e00150d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:51.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.219+0000 7f9e07cc2640 1 --2- 172.21.15.161:0/1331195220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e0014f1d0 0x7f9e00150d60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9df800cb10 tx=0x7f9df800a060 comp rx=0 tx=0).stop 2024-03-28T23:30:51.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.220+0000 7f9e07cc2640 1 -- 172.21.15.161:0/1331195220 shutdown_connections 2024-03-28T23:30:51.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.220+0000 7f9e07cc2640 1 --2- 172.21.15.161:0/1331195220 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9de406b4d0 0x7f9de406d990 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:51.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.220+0000 7f9e07cc2640 1 --2- 172.21.15.161:0/1331195220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e0014f1d0 0x7f9e00150d60 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:51.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.220+0000 7f9e07cc2640 1 --2- 172.21.15.161:0/1331195220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9e000c3fd0 0x7f9e001ea7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:51.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.220+0000 7f9e07cc2640 1 --2- 172.21.15.161:0/1331195220 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e000c3080 0x7f9e000a9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:51.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.220+0000 7f9e07cc2640 1 -- 172.21.15.161:0/1331195220 >> 172.21.15.161:0/1331195220 conn(0x7f9e000c6e20 msgr2=0x7f9e0014d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:51.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.220+0000 7f9e07cc2640 1 -- 172.21.15.161:0/1331195220 shutdown_connections 2024-03-28T23:30:51.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.220+0000 7f9e07cc2640 1 -- 172.21.15.161:0/1331195220 wait complete. 2024-03-28T23:30:51.234 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T23:30:51.234 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-28T23:30:51.234 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T23:30:51.234 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:30:51.234 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T23:30:51.234 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-28T23:30:51.235 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T23:30:51.235 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:30:51.235 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-28T23:30:51.235 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-28T23:30:51.235 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-28T23:30:51.235 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:30:51.235 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-28T23:30:51.236 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-28T23:30:51.236 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-28T23:30:51.236 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:30:51.243 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T23:30:51.243 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T23:30:51.244 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T23:30:51.244 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T23:30:51.247 INFO:tasks.ceph.mds.b:Stopped 2024-03-28T23:30:51.247 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-28T23:30:51.247 INFO:tasks.ceph.mds.d:Stopped 2024-03-28T23:30:51.248 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-28T23:30:51.286 INFO:tasks.ceph.mds.a:Started 2024-03-28T23:30:51.286 DEBUG:teuthology.parallel:result is None 2024-03-28T23:30:51.287 INFO:tasks.ceph.mds.c:Started 2024-03-28T23:30:51.287 DEBUG:teuthology.parallel:result is None 2024-03-28T23:30:51.289 INFO:tasks.ceph.mds.b:Started 2024-03-28T23:30:51.289 DEBUG:teuthology.parallel:result is None 2024-03-28T23:30:51.289 INFO:tasks.ceph.mds.d:Started 2024-03-28T23:30:51.290 DEBUG:teuthology.parallel:result is None 2024-03-28T23:30:51.290 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-28T23:30:51.364 INFO:tasks.ceph.mds.c.smithi161.stdout:starting mds.c at 2024-03-28T23:30:51.365 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-28T23:30:51.369 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2024-03-28T23:30:51.370 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2024-03-28T23:30:51.370 INFO:tasks.ceph.mds.b.smithi170.stderr:2024-03-28T23:30:51.370+0000 7f0120096640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-03-28T23:30:51.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.395+0000 7f4360c23640 1 Processor -- start 2024-03-28T23:30:51.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.395+0000 7f4360c23640 1 -- start start 2024-03-28T23:30:51.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.396+0000 7f4360c23640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f435c0c5850 0x7f435c155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:51.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.396+0000 7f4360c23640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f435c0c5d00 con 0x7f435c0c5410 2024-03-28T23:30:51.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.396+0000 7f4360c23640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f435c158a30 con 0x7f435c152890 2024-03-28T23:30:51.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.396+0000 7f4360c23640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f435c0c8480 con 0x7f435c0c5850 2024-03-28T23:30:51.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.396+0000 7f435ad9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f435c0c5850 0x7f435c155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:51.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.396+0000 7f435ad9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f435c0c5850 0x7f435c155ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46678/0 (socket says 172.21.15.161:46678) 2024-03-28T23:30:51.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.396+0000 7f435ad9d640 1 -- 172.21.15.161:0/3964191407 learned_addr learned my addr 172.21.15.161:0/3964191407 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:51.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.396+0000 7f435ad9d640 1 -- 172.21.15.161:0/3964191407 >> v1:172.21.15.170:6789/0 conn(0x7f435c152890 legacy=0x7f435c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:51.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.397+0000 7f435959a640 1 -- 172.21.15.161:0/3964191407 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2948936743 0 0) 0x7f435c0c5d00 con 0x7f435c0c5410 2024-03-28T23:30:51.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.397+0000 7f435ad9d640 1 -- 172.21.15.161:0/3964191407 >> v1:172.21.15.161:6789/0 conn(0x7f435c0c5410 legacy=0x7f435c1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:51.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.397+0000 7f435ad9d640 1 -- 172.21.15.161:0/3964191407 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f435c158d10 con 0x7f435c0c5850 2024-03-28T23:30:51.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.397+0000 7f435ad9d640 1 --2- 172.21.15.161:0/3964191407 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f435c0c5850 0x7f435c155ab0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4354009e50 tx=0x7f435402bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=528ac977e137fff4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:51.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.397+0000 7f435959a640 1 -- 172.21.15.161:0/3964191407 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435402c900 con 0x7f435c0c5850 2024-03-28T23:30:51.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.397+0000 7f435959a640 1 -- 172.21.15.161:0/3964191407 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f435402caa0 con 0x7f435c0c5850 2024-03-28T23:30:51.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.397+0000 7f435959a640 1 -- 172.21.15.161:0/3964191407 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43540343f0 con 0x7f435c0c5850 2024-03-28T23:30:51.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.398+0000 7f4360c23640 1 -- 172.21.15.161:0/3964191407 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f435c0c5850 msgr2=0x7f435c155ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:51.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.398+0000 7f4360c23640 1 --2- 172.21.15.161:0/3964191407 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f435c0c5850 0x7f435c155ab0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4354009e50 tx=0x7f435402bde0 comp rx=0 tx=0).stop 2024-03-28T23:30:51.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.398+0000 7f4360c23640 1 -- 172.21.15.161:0/3964191407 shutdown_connections 2024-03-28T23:30:51.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.398+0000 7f4360c23640 1 --2- 172.21.15.161:0/3964191407 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f435c0c5850 0x7f435c155ab0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:51.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.398+0000 7f4360c23640 1 -- 172.21.15.161:0/3964191407 >> 172.21.15.161:0/3964191407 conn(0x7f435c0ac620 msgr2=0x7f435c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:51.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.398+0000 7f4360c23640 1 -- 172.21.15.161:0/3964191407 shutdown_connections 2024-03-28T23:30:51.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.398+0000 7f4360c23640 1 -- 172.21.15.161:0/3964191407 wait complete. 2024-03-28T23:30:51.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.399+0000 7f4360c23640 1 Processor -- start 2024-03-28T23:30:51.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.399+0000 7f4360c23640 1 -- start start 2024-03-28T23:30:51.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.400+0000 7f4360c23640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f435c0c5410 0x7f435c1ef120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:51.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.400+0000 7f4360c23640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f435c0c5850 0x7f435c1ebbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:51.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.400+0000 7f4360c23640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f435c152890 0x7f435c0be9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:51.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.400+0000 7f4360c23640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f435c1592d0 con 0x7f435c0c5850 2024-03-28T23:30:51.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.400+0000 7f4360c23640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f435c159640 con 0x7f435c152890 2024-03-28T23:30:51.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.400+0000 7f4360c23640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f435c155ff0 con 0x7f435c0c5410 2024-03-28T23:30:51.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.400+0000 7f4359d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f435c0c5850 0x7f435c1ebbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:51.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.400+0000 7f4359d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f435c0c5850 0x7f435c1ebbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46684/0 (socket says 172.21.15.161:46684) 2024-03-28T23:30:51.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.400+0000 7f4359d9b640 1 -- 172.21.15.161:0/2184760858 learned_addr learned my addr 172.21.15.161:0/2184760858 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:51.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.400+0000 7f435a59c640 1 --2- 172.21.15.161:0/2184760858 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f435c0c5410 0x7f435c1ef120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:51.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.400+0000 7f435ad9d640 1 --2- 172.21.15.161:0/2184760858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f435c152890 0x7f435c0be9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:51.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.400+0000 7f4359d9b640 1 -- 172.21.15.161:0/2184760858 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f435c0c5410 msgr2=0x7f435c1ef120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:51.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.400+0000 7f4359d9b640 1 --2- 172.21.15.161:0/2184760858 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f435c0c5410 0x7f435c1ef120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:51.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.400+0000 7f4359d9b640 1 -- 172.21.15.161:0/2184760858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f435c152890 msgr2=0x7f435c0be9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:51.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.400+0000 7f4359d9b640 1 --2- 172.21.15.161:0/2184760858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f435c152890 0x7f435c0be9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:51.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.400+0000 7f4359d9b640 1 -- 172.21.15.161:0/2184760858 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f435c154e50 con 0x7f435c0c5850 2024-03-28T23:30:51.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.400+0000 7f4359d9b640 1 --2- 172.21.15.161:0/2184760858 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f435c0c5850 0x7f435c1ebbb0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f435000c3e0 tx=0x7f435000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:51.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.401+0000 7f434affd640 1 -- 172.21.15.161:0/2184760858 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4350017070 con 0x7f435c0c5850 2024-03-28T23:30:51.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.401+0000 7f434affd640 1 -- 172.21.15.161:0/2184760858 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43500035f0 con 0x7f435c0c5850 2024-03-28T23:30:51.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.401+0000 7f4360c23640 1 -- 172.21.15.161:0/2184760858 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f435c1ec0f0 con 0x7f435c0c5850 2024-03-28T23:30:51.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.401+0000 7f434affd640 1 -- 172.21.15.161:0/2184760858 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4350003a20 con 0x7f435c0c5850 2024-03-28T23:30:51.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.401+0000 7f4360c23640 1 -- 172.21.15.161:0/2184760858 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f435c1550f0 con 0x7f435c0c5850 2024-03-28T23:30:51.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.402+0000 7f434affd640 1 -- 172.21.15.161:0/2184760858 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f435001a070 con 0x7f435c0c5850 2024-03-28T23:30:51.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.403+0000 7f4360c23640 1 -- 172.21.15.161:0/2184760858 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f435c0c7530 con 0x7f435c0c5850 2024-03-28T23:30:51.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.403+0000 7f434affd640 1 --2- 172.21.15.161:0/2184760858 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f433c06b4d0 0x7f433c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:51.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.403+0000 7f434affd640 1 -- 172.21.15.161:0/2184760858 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5335+0+0 (secure 0 0 0) 0x7f4350089340 con 0x7f435c0c5850 2024-03-28T23:30:51.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.403+0000 7f435a59c640 1 --2- 172.21.15.161:0/2184760858 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f433c06b4d0 0x7f433c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:51.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.403+0000 7f435a59c640 1 --2- 172.21.15.161:0/2184760858 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f433c06b4d0 0x7f433c06d990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f434c000f60 tx=0x7f434c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:51.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.407+0000 7f434affd640 1 -- 172.21.15.161:0/2184760858 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f435c0c7530 con 0x7f435c0c5850 2024-03-28T23:30:51.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:51.590+0000 7f4360c23640 1 -- 172.21.15.161:0/2184760858 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f435c067a50 con 0x7f435c0c5850 2024-03-28T23:30:52.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.230+0000 7f434affd640 1 -- 172.21.15.161:0/2184760858 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v32) ==== 97+0+0 (secure 0 0 0) 0x7f435c067a50 con 0x7f435c0c5850 2024-03-28T23:30:52.231 INFO:teuthology.orchestra.run.smithi161.stderr: removed all blocklist entries 2024-03-28T23:30:52.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.233+0000 7f4360c23640 1 -- 172.21.15.161:0/2184760858 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f433c06b4d0 msgr2=0x7f433c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:52.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.233+0000 7f4360c23640 1 --2- 172.21.15.161:0/2184760858 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f433c06b4d0 0x7f433c06d990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f434c000f60 tx=0x7f434c009040 comp rx=0 tx=0).stop 2024-03-28T23:30:52.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.233+0000 7f4360c23640 1 -- 172.21.15.161:0/2184760858 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f435c0c5850 msgr2=0x7f435c1ebbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:52.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.233+0000 7f4360c23640 1 --2- 172.21.15.161:0/2184760858 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f435c0c5850 0x7f435c1ebbb0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f435000c3e0 tx=0x7f435000c8a0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.233+0000 7f4360c23640 1 -- 172.21.15.161:0/2184760858 shutdown_connections 2024-03-28T23:30:52.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.233+0000 7f4360c23640 1 --2- 172.21.15.161:0/2184760858 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f433c06b4d0 0x7f433c06d990 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.233+0000 7f4360c23640 1 --2- 172.21.15.161:0/2184760858 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f435c0c5850 0x7f435c1ebbb0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.233+0000 7f4360c23640 1 --2- 172.21.15.161:0/2184760858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f435c152890 0x7f435c0be9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.233+0000 7f4360c23640 1 --2- 172.21.15.161:0/2184760858 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f435c0c5410 0x7f435c1ef120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.233+0000 7f4360c23640 1 -- 172.21.15.161:0/2184760858 >> 172.21.15.161:0/2184760858 conn(0x7f435c0ac620 msgr2=0x7f435c14e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:52.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.233+0000 7f4360c23640 1 -- 172.21.15.161:0/2184760858 shutdown_connections 2024-03-28T23:30:52.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.234+0000 7f4360c23640 1 -- 172.21.15.161:0/2184760858 wait complete. 2024-03-28T23:30:52.249 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-28T23:30:52.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.350+0000 7fe291be2640 1 Processor -- start 2024-03-28T23:30:52.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.350+0000 7fe291be2640 1 -- start start 2024-03-28T23:30:52.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.351+0000 7fe291be2640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe28c150590 0x7fe28c0a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:52.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.351+0000 7fe291be2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe28c0a8fc0 0x7fe28c153150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:52.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.351+0000 7fe291be2640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe28c1566f0 con 0x7fe28c0aacb0 2024-03-28T23:30:52.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.351+0000 7fe291be2640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe28c1569d0 con 0x7fe28c0a8fc0 2024-03-28T23:30:52.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.351+0000 7fe291be2640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fe28c156cb0 con 0x7fe28c150590 2024-03-28T23:30:52.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.351+0000 7fe28affd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe28c0a8fc0 0x7fe28c153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:52.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.351+0000 7fe28affd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe28c0a8fc0 0x7fe28c153150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46694/0 (socket says 172.21.15.161:46694) 2024-03-28T23:30:52.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.351+0000 7fe28affd640 1 -- 172.21.15.161:0/1859029953 learned_addr learned my addr 172.21.15.161:0/1859029953 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:52.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.351+0000 7fe28affd640 1 -- 172.21.15.161:0/1859029953 >> v2:172.21.15.170:3300/0 conn(0x7fe28c150590 msgr2=0x7fe28c0a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:52.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.351+0000 7fe28affd640 1 --2- 172.21.15.161:0/1859029953 >> v2:172.21.15.170:3300/0 conn(0x7fe28c150590 0x7fe28c0a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.351+0000 7fe28affd640 1 -- 172.21.15.161:0/1859029953 >> v1:172.21.15.170:6789/0 conn(0x7fe28c0aacb0 legacy=0x7fe28c153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:52.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.351+0000 7fe28affd640 1 -- 172.21.15.161:0/1859029953 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe28c156f90 con 0x7fe28c0a8fc0 2024-03-28T23:30:52.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.352+0000 7fe28affd640 1 --2- 172.21.15.161:0/1859029953 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe28c0a8fc0 0x7fe28c153150 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe28c0c4210 tx=0x7fe28002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d35648e03fa8af server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:52.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.352+0000 7fe28a7fc640 1 -- 172.21.15.161:0/1859029953 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe280038070 con 0x7fe28c0a8fc0 2024-03-28T23:30:52.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.352+0000 7fe28a7fc640 1 -- 172.21.15.161:0/1859029953 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe28002f050 con 0x7fe28c0a8fc0 2024-03-28T23:30:52.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.352+0000 7fe28a7fc640 1 -- 172.21.15.161:0/1859029953 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe280031970 con 0x7fe28c0a8fc0 2024-03-28T23:30:52.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.353+0000 7fe291be2640 1 -- 172.21.15.161:0/1859029953 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe28c0a8fc0 msgr2=0x7fe28c153150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:52.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.353+0000 7fe291be2640 1 --2- 172.21.15.161:0/1859029953 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe28c0a8fc0 0x7fe28c153150 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe28c0c4210 tx=0x7fe28002b7f0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.353+0000 7fe291be2640 1 -- 172.21.15.161:0/1859029953 shutdown_connections 2024-03-28T23:30:52.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.353+0000 7fe291be2640 1 --2- 172.21.15.161:0/1859029953 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe28c0a8fc0 0x7fe28c153150 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.353+0000 7fe291be2640 1 --2- 172.21.15.161:0/1859029953 >> v2:172.21.15.170:3300/0 conn(0x7fe28c150590 0x7fe28c0a9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.353+0000 7fe291be2640 1 -- 172.21.15.161:0/1859029953 >> 172.21.15.161:0/1859029953 conn(0x7fe28c0c6670 msgr2=0x7fe28c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:52.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.353+0000 7fe291be2640 1 -- 172.21.15.161:0/1859029953 shutdown_connections 2024-03-28T23:30:52.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.353+0000 7fe291be2640 1 -- 172.21.15.161:0/1859029953 wait complete. 2024-03-28T23:30:52.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.354+0000 7fe291be2640 1 Processor -- start 2024-03-28T23:30:52.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.354+0000 7fe291be2640 1 -- start start 2024-03-28T23:30:52.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.355+0000 7fe291be2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe28c0a8fc0 0x7fe28c152720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:52.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.355+0000 7fe291be2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe28c0aacb0 0x7fe28c155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:52.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.355+0000 7fe291be2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe28c150590 0x7fe28c1560e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:52.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.355+0000 7fe291be2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c150970 con 0x7fe28c0aacb0 2024-03-28T23:30:52.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.355+0000 7fe291be2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c0aa9c0 con 0x7fe28c150590 2024-03-28T23:30:52.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.355+0000 7fe291be2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe28c0aa480 con 0x7fe28c0a8fc0 2024-03-28T23:30:52.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.355+0000 7fe28affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe28c0aacb0 0x7fe28c155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:52.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.355+0000 7fe28affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe28c0aacb0 0x7fe28c155ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46698/0 (socket says 172.21.15.161:46698) 2024-03-28T23:30:52.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.355+0000 7fe28affd640 1 -- 172.21.15.161:0/718926474 learned_addr learned my addr 172.21.15.161:0/718926474 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:52.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.355+0000 7fe28b7fe640 1 --2- 172.21.15.161:0/718926474 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe28c0a8fc0 0x7fe28c152720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:52.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.355+0000 7fe28bfff640 1 --2- 172.21.15.161:0/718926474 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe28c150590 0x7fe28c1560e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:52.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.355+0000 7fe28affd640 1 -- 172.21.15.161:0/718926474 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe28c0a8fc0 msgr2=0x7fe28c152720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:52.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.355+0000 7fe28affd640 1 --2- 172.21.15.161:0/718926474 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe28c0a8fc0 0x7fe28c152720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.355+0000 7fe28affd640 1 -- 172.21.15.161:0/718926474 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe28c150590 msgr2=0x7fe28c1560e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:52.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.355+0000 7fe28affd640 1 --2- 172.21.15.161:0/718926474 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe28c150590 0x7fe28c1560e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.355+0000 7fe28affd640 1 -- 172.21.15.161:0/718926474 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe28c1544a0 con 0x7fe28c0aacb0 2024-03-28T23:30:52.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.355+0000 7fe28affd640 1 --2- 172.21.15.161:0/718926474 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe28c0aacb0 0x7fe28c155ba0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe280000c00 tx=0x7fe2800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:52.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.355+0000 7fe273fff640 1 -- 172.21.15.161:0/718926474 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe280043070 con 0x7fe28c0aacb0 2024-03-28T23:30:52.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.355+0000 7fe273fff640 1 -- 172.21.15.161:0/718926474 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe28003f870 con 0x7fe28c0aacb0 2024-03-28T23:30:52.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.355+0000 7fe291be2640 1 -- 172.21.15.161:0/718926474 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe28c0c5610 con 0x7fe28c0aacb0 2024-03-28T23:30:52.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.355+0000 7fe273fff640 1 -- 172.21.15.161:0/718926474 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe280038040 con 0x7fe28c0aacb0 2024-03-28T23:30:52.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.355+0000 7fe291be2640 1 -- 172.21.15.161:0/718926474 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe28c0c0f90 con 0x7fe28c0aacb0 2024-03-28T23:30:52.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.356+0000 7fe291be2640 1 -- 172.21.15.161:0/718926474 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe28c06e570 con 0x7fe28c0aacb0 2024-03-28T23:30:52.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.357+0000 7fe273fff640 1 -- 172.21.15.161:0/718926474 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe280046070 con 0x7fe28c0aacb0 2024-03-28T23:30:52.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.357+0000 7fe273fff640 1 --2- 172.21.15.161:0/718926474 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe27806b4d0 0x7fe27806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:52.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.357+0000 7fe28b7fe640 1 --2- 172.21.15.161:0/718926474 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe27806b4d0 0x7fe27806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:52.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.357+0000 7fe273fff640 1 -- 172.21.15.161:0/718926474 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7fe2800ac0b0 con 0x7fe28c0aacb0 2024-03-28T23:30:52.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.358+0000 7fe28b7fe640 1 --2- 172.21.15.161:0/718926474 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe27806b4d0 0x7fe27806d990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe27c00b810 tx=0x7fe27c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:52.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.362+0000 7fe273fff640 1 -- 172.21.15.161:0/718926474 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe28c06e570 con 0x7fe28c0aacb0 2024-03-28T23:30:52.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.596+0000 7fe291be2640 1 -- 172.21.15.161:0/718926474 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fe28c0c1690 con 0x7fe28c0aacb0 2024-03-28T23:30:52.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.596+0000 7fe273fff640 1 -- 172.21.15.161:0/718926474 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v2) ==== 80+0+6065 (secure 0 0 0) 0x7fe28c0c1690 con 0x7fe28c0aacb0 2024-03-28T23:30:52.597 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:52.597 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:30:52.597 INFO:teuthology.orchestra.run.smithi161.stdout: "auth_dump": [ 2024-03-28T23:30:52.597 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:30:52.597 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.a", 2024-03-28T23:30:52.597 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmggtoJhAASLXbUICLBhjRr19aUAuZHw==", 2024-03-28T23:30:52.597 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:30:52.598 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:30:52.598 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:30:52.598 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:30:52.598 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:30:52.598 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:52.598 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:52.598 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:30:52.598 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.b", 2024-03-28T23:30:52.598 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmQbEOMxAA74al6+dMzcN/ngwNZK+aHA==", 2024-03-28T23:30:52.598 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:30:52.598 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:30:52.598 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:30:52.599 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:30:52.599 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:30:52.599 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:52.599 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:52.599 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:30:52.599 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.c", 2024-03-28T23:30:52.599 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmjf/qLRAAosLbpT8aCWvrKAQnj8Uh3g==", 2024-03-28T23:30:52.599 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:30:52.599 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:30:52.599 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:30:52.599 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:30:52.599 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:30:52.600 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:52.600 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:52.600 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:30:52.600 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.d", 2024-03-28T23:30:52.600 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmpiZ1OhAAjYrFAaDosDS2Y+BvU4c6JQ==", 2024-03-28T23:30:52.600 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:30:52.600 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:30:52.600 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:30:52.600 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:30:52.600 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:30:52.600 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:52.600 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:52.600 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:30:52.600 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.0", 2024-03-28T23:30:52.600 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA9/QVmAFPmGBAAS0SkRRwsupm6jm1sqFXJ6g==", 2024-03-28T23:30:52.601 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:30:52.601 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:30:52.601 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:30:52.601 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:30:52.601 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:52.601 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:52.601 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:30:52.601 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.1", 2024-03-28T23:30:52.601 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBA/QVmNBTVFxAA1htU9ilznfLUDqKmJaQwzg==", 2024-03-28T23:30:52.601 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:30:52.601 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:30:52.601 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:30:52.601 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:30:52.601 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:52.601 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:30:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.2", 2024-03-28T23:30:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBD/QVmg+4YFhAA9TvfFVKT3nQT7/QUOf3Spw==", 2024-03-28T23:30:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:30:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:30:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:30:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:30:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:30:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.3", 2024-03-28T23:30:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBG/QVmgiHDFBAAQPtbZT6a1ZGmkisBX0NMgA==", 2024-03-28T23:30:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:30:52.602 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:30:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:30:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:30:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:30:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.4", 2024-03-28T23:30:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBK/QVmahhZIBAAJfotg/dusbB3usreZlRuPA==", 2024-03-28T23:30:52.603 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:30:52.604 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:30:52.604 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:30:52.604 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:30:52.604 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:52.604 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:52.604 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:30:52.604 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.5", 2024-03-28T23:30:52.604 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBN/QVmzjlDHhAAli7pLyIo+MVdwSBNJYVg5g==", 2024-03-28T23:30:52.604 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:30:52.604 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:30:52.605 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:30:52.605 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:30:52.605 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:52.605 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:52.605 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:30:52.605 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.6", 2024-03-28T23:30:52.605 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBQ/QVmtCFSHBAASVfV/5mY/3JNiI0YMT7Vww==", 2024-03-28T23:30:52.605 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:30:52.605 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:30:52.605 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:30:52.605 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:30:52.605 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:52.606 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:52.606 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:30:52.606 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.7", 2024-03-28T23:30:52.606 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBT/QVm2J7eGhAAaIeZxmLwQBBE1glCxjKbhA==", 2024-03-28T23:30:52.606 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:30:52.606 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:30:52.606 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:30:52.606 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:30:52.606 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:52.606 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:52.606 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:30:52.606 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.0", 2024-03-28T23:30:52.606 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA6/QVmc1krAxAAEKsn1Tuuv9OlFZpB4PFhDg==", 2024-03-28T23:30:52.606 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:30:52.607 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:30:52.607 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow r", 2024-03-28T23:30:52.607 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow rw", 2024-03-28T23:30:52.607 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow rwx" 2024-03-28T23:30:52.607 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:52.607 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:52.607 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:30:52.607 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.1", 2024-03-28T23:30:52.607 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA6/QVm1B2cBhAAmHDeeFwpFlV/y3NIGWUXHQ==", 2024-03-28T23:30:52.607 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:30:52.607 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:30:52.607 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow r", 2024-03-28T23:30:52.608 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow rw", 2024-03-28T23:30:52.608 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow rwx" 2024-03-28T23:30:52.608 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:52.608 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:52.608 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:30:52.608 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.admin", 2024-03-28T23:30:52.608 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmzzJ7CRAAYqfhnIkfNJJ3O+VWC2LP+Q==", 2024-03-28T23:30:52.608 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:30:52.608 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-28T23:30:52.608 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:30:52.609 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:30:52.609 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:30:52.609 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:52.609 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:52.609 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:30:52.609 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.bootstrap-mds", 2024-03-28T23:30:52.609 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBZ/QVmTG29BxAADQCGlWihNJ541dfQuyLHuA==", 2024-03-28T23:30:52.609 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:30:52.609 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile bootstrap-mds" 2024-03-28T23:30:52.609 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:52.610 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:52.610 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:30:52.610 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.bootstrap-mgr", 2024-03-28T23:30:52.610 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBZ/QVmg4y9BxAA3sKeUN5USVvLvqs3fMgDTg==", 2024-03-28T23:30:52.610 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:30:52.610 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-28T23:30:52.610 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:52.610 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:52.611 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:30:52.611 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.bootstrap-osd", 2024-03-28T23:30:52.611 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBZ/QVmkJS9BxAAIV8zeigEz7FNJOLSm4YZLg==", 2024-03-28T23:30:52.611 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:30:52.611 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile bootstrap-osd" 2024-03-28T23:30:52.611 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:52.611 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:52.611 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:30:52.611 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.bootstrap-rbd", 2024-03-28T23:30:52.611 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBZ/QVmQpu9BxAAj5CYnm/FhphqQwXuL/m8cQ==", 2024-03-28T23:30:52.611 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:30:52.611 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-28T23:30:52.611 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:52.611 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:52.611 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:30:52.612 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-28T23:30:52.612 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBZ/QVm66G9BxAASV3Lh5wyH1T+GfXn+6MRWA==", 2024-03-28T23:30:52.612 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:30:52.612 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-28T23:30:52.612 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:52.612 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:52.612 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:30:52.612 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.bootstrap-rgw", 2024-03-28T23:30:52.612 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBZ/QVm56i9BxAAdRy2SXhc/DAsuXLHdek9Cw==", 2024-03-28T23:30:52.612 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:30:52.612 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-28T23:30:52.612 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:52.612 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:52.612 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:30:52.612 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mgr.x", 2024-03-28T23:30:52.613 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVm7TD3IhAACByGRGpvJWDpF22NuovHWw==", 2024-03-28T23:30:52.613 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:30:52.613 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-28T23:30:52.613 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile mgr", 2024-03-28T23:30:52.613 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:30:52.613 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:52.613 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:30:52.613 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:30:52.613 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mgr.y", 2024-03-28T23:30:52.613 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVm0hLlHhAAh8cewPtZ/TbKrU488kxrBg==", 2024-03-28T23:30:52.613 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:30:52.613 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-28T23:30:52.613 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile mgr", 2024-03-28T23:30:52.613 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:30:52.613 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:52.614 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:52.614 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-03-28T23:30:52.614 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:30:52.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.598+0000 7fe291be2640 1 -- 172.21.15.161:0/718926474 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe27806b4d0 msgr2=0x7fe27806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:52.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.598+0000 7fe291be2640 1 --2- 172.21.15.161:0/718926474 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe27806b4d0 0x7fe27806d990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe27c00b810 tx=0x7fe27c008040 comp rx=0 tx=0).stop 2024-03-28T23:30:52.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.598+0000 7fe291be2640 1 -- 172.21.15.161:0/718926474 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe28c0aacb0 msgr2=0x7fe28c155ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:52.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.598+0000 7fe291be2640 1 --2- 172.21.15.161:0/718926474 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe28c0aacb0 0x7fe28c155ba0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe280000c00 tx=0x7fe2800046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.598+0000 7fe291be2640 1 -- 172.21.15.161:0/718926474 shutdown_connections 2024-03-28T23:30:52.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.598+0000 7fe291be2640 1 --2- 172.21.15.161:0/718926474 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe27806b4d0 0x7fe27806d990 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.598+0000 7fe291be2640 1 --2- 172.21.15.161:0/718926474 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe28c0aacb0 0x7fe28c155ba0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.598+0000 7fe291be2640 1 --2- 172.21.15.161:0/718926474 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe28c150590 0x7fe28c1560e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.598+0000 7fe291be2640 1 --2- 172.21.15.161:0/718926474 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe28c0a8fc0 0x7fe28c152720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.598+0000 7fe291be2640 1 -- 172.21.15.161:0/718926474 >> 172.21.15.161:0/718926474 conn(0x7fe28c0c6670 msgr2=0x7fe28c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:52.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.598+0000 7fe291be2640 1 -- 172.21.15.161:0/718926474 shutdown_connections 2024-03-28T23:30:52.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.598+0000 7fe291be2640 1 -- 172.21.15.161:0/718926474 wait complete. 2024-03-28T23:30:52.617 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-28T23:30:52.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.717+0000 7f51f3a47640 1 Processor -- start 2024-03-28T23:30:52.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.718+0000 7f51f3a47640 1 -- start start 2024-03-28T23:30:52.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.718+0000 7f51f3a47640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f51ec150590 0x7f51ec0a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:52.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.718+0000 7f51f3a47640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51ec1566f0 con 0x7f51ec0aacb0 2024-03-28T23:30:52.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.718+0000 7f51f3a47640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51ec1569d0 con 0x7f51ec0a8fc0 2024-03-28T23:30:52.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.718+0000 7f51f3a47640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f51ec156cb0 con 0x7f51ec150590 2024-03-28T23:30:52.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.718+0000 7f51f1fe4640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f51ec0aacb0 0x7f51ec153860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:46434/0 (socket says 172.21.15.161:46434) 2024-03-28T23:30:52.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.719+0000 7f51f1fe4640 1 -- 172.21.15.161:0/755899001 learned_addr learned my addr 172.21.15.161:0/755899001 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:52.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.719+0000 7f51dffff640 1 -- 172.21.15.161:0/755899001 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1992948066 0 0) 0x7f51ec1566f0 con 0x7f51ec0aacb0 2024-03-28T23:30:52.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.719+0000 7f51dffff640 1 -- 172.21.15.161:0/755899001 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51d8003520 con 0x7f51ec0aacb0 2024-03-28T23:30:52.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.719+0000 7f51f17e3640 1 --2- 172.21.15.161:0/755899001 >> v2:172.21.15.170:3300/0 conn(0x7f51ec150590 0x7f51ec0a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:52.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.719+0000 7f51dffff640 1 -- 172.21.15.161:0/755899001 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 764118562 0 0) 0x7f51d8003520 con 0x7f51ec0aacb0 2024-03-28T23:30:52.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.719+0000 7f51dffff640 1 -- 172.21.15.161:0/755899001 >> v2:172.21.15.170:3300/0 conn(0x7f51ec150590 msgr2=0x7f51ec0a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:52.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.719+0000 7f51dffff640 1 --2- 172.21.15.161:0/755899001 >> v2:172.21.15.170:3300/0 conn(0x7f51ec150590 0x7f51ec0a9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.719+0000 7f51dffff640 1 -- 172.21.15.161:0/755899001 >> v1:172.21.15.170:6789/0 conn(0x7f51ec0a8fc0 legacy=0x7f51ec153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:52.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.719+0000 7f51dffff640 1 -- 172.21.15.161:0/755899001 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51ec156f90 con 0x7f51ec0aacb0 2024-03-28T23:30:52.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.719+0000 7f51dffff640 1 -- 172.21.15.161:0/755899001 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f51e40019f0 con 0x7f51ec0aacb0 2024-03-28T23:30:52.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.719+0000 7f51dffff640 1 -- 172.21.15.161:0/755899001 >> v1:172.21.15.161:6789/0 conn(0x7f51ec0aacb0 legacy=0x7f51ec153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:52.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.719+0000 7f51dffff640 1 --2- 172.21.15.161:0/755899001 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51d8003ae0 0x7f51d8003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:52.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.719+0000 7f51dffff640 1 --2- 172.21.15.161:0/755899001 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51d8004800 0x7f51d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:52.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.719+0000 7f51dffff640 1 --2- 172.21.15.161:0/755899001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51d8005410 0x7f51d800ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:52.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.719+0000 7f51dffff640 1 -- 172.21.15.161:0/755899001 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec156cb0 con 0x7f51d8003ae0 2024-03-28T23:30:52.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.719+0000 7f51dffff640 1 -- 172.21.15.161:0/755899001 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f51d800c350 con 0x7f51d8005410 2024-03-28T23:30:52.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.719+0000 7f51dffff640 1 -- 172.21.15.161:0/755899001 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f51d800c550 con 0x7f51d8004800 2024-03-28T23:30:52.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.719+0000 7f51f17e3640 1 --2- 172.21.15.161:0/755899001 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51d8003ae0 0x7f51d8003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.719+0000 7f51f0fe2640 1 --2- 172.21.15.161:0/755899001 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51d8004800 0x7f51d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.719+0000 7f51f1fe4640 1 --2- 172.21.15.161:0/755899001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51d8005410 0x7f51d800ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.720+0000 7f51f17e3640 1 -- 172.21.15.161:0/755899001 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51d8004800 msgr2=0x7f51d8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.720+0000 7f51f17e3640 1 --2- 172.21.15.161:0/755899001 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51d8004800 0x7f51d8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.720+0000 7f51f17e3640 1 -- 172.21.15.161:0/755899001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51d8005410 msgr2=0x7f51d800ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.720+0000 7f51f17e3640 1 --2- 172.21.15.161:0/755899001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51d8005410 0x7f51d800ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.720+0000 7f51f17e3640 1 -- 172.21.15.161:0/755899001 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51d8003520 con 0x7f51d8003ae0 2024-03-28T23:30:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.720+0000 7f51f1fe4640 1 --2- 172.21.15.161:0/755899001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51d8005410 0x7f51d800ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.720+0000 7f51f1fe4640 1 -- 172.21.15.161:0/755899001 reap_dead start 2024-03-28T23:30:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.720+0000 7f51f17e3640 1 --2- 172.21.15.161:0/755899001 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51d8003ae0 0x7f51d8003ee0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f51e800ba30 tx=0x7f51d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e069f6a417daefbc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.720+0000 7f51dffff640 1 -- 172.21.15.161:0/755899001 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e8036020 con 0x7f51d8003ae0 2024-03-28T23:30:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.720+0000 7f51dffff640 1 -- 172.21.15.161:0/755899001 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51e8032640 con 0x7f51d8003ae0 2024-03-28T23:30:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.720+0000 7f51dffff640 1 -- 172.21.15.161:0/755899001 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e800b600 con 0x7f51d8003ae0 2024-03-28T23:30:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.721+0000 7f51f3a47640 1 -- 172.21.15.161:0/755899001 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51d8003ae0 msgr2=0x7f51d8003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:52.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.721+0000 7f51f3a47640 1 --2- 172.21.15.161:0/755899001 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51d8003ae0 0x7f51d8003ee0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f51e800ba30 tx=0x7f51d8005830 comp rx=0 tx=0).stop 2024-03-28T23:30:52.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.721+0000 7f51f3a47640 1 -- 172.21.15.161:0/755899001 shutdown_connections 2024-03-28T23:30:52.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.721+0000 7f51f3a47640 1 --2- 172.21.15.161:0/755899001 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51d8003ae0 0x7f51d8003ee0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.721+0000 7f51f3a47640 1 -- 172.21.15.161:0/755899001 >> 172.21.15.161:0/755899001 conn(0x7f51ec0c6670 msgr2=0x7f51ec0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:52.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.721+0000 7f51f3a47640 1 -- 172.21.15.161:0/755899001 shutdown_connections 2024-03-28T23:30:52.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.721+0000 7f51f3a47640 1 -- 172.21.15.161:0/755899001 wait complete. 2024-03-28T23:30:52.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.722+0000 7f51f3a47640 1 Processor -- start 2024-03-28T23:30:52.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.722+0000 7f51f3a47640 1 -- start start 2024-03-28T23:30:52.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.723+0000 7f51f3a47640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51d8003ae0 0x7f51ec147ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:52.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.723+0000 7f51f3a47640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51ec14eae0 0x7f51ec14f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:52.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.723+0000 7f51f3a47640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51ec14f930 0x7f51ec1ec4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:52.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.723+0000 7f51f3a47640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec0a9f40 con 0x7f51ec14eae0 2024-03-28T23:30:52.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.723+0000 7f51f3a47640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec150970 con 0x7f51ec14f930 2024-03-28T23:30:52.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.723+0000 7f51f3a47640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f51ec0aa0c0 con 0x7f51d8003ae0 2024-03-28T23:30:52.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.723+0000 7f51f0fe2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51ec14eae0 0x7f51ec14f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:52.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.723+0000 7f51f0fe2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51ec14eae0 0x7f51ec14f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46722/0 (socket says 172.21.15.161:46722) 2024-03-28T23:30:52.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.723+0000 7f51f0fe2640 1 -- 172.21.15.161:0/1377515248 learned_addr learned my addr 172.21.15.161:0/1377515248 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:52.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.723+0000 7f51f1fe4640 1 --2- 172.21.15.161:0/1377515248 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51ec14f930 0x7f51ec1ec4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:52.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.723+0000 7f51f17e3640 1 --2- 172.21.15.161:0/1377515248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51d8003ae0 0x7f51ec147ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:52.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.723+0000 7f51f0fe2640 1 -- 172.21.15.161:0/1377515248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51d8003ae0 msgr2=0x7f51ec147ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:52.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.723+0000 7f51f0fe2640 1 --2- 172.21.15.161:0/1377515248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51d8003ae0 0x7f51ec147ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.723+0000 7f51f0fe2640 1 -- 172.21.15.161:0/1377515248 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51ec14f930 msgr2=0x7f51ec1ec4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:52.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.723+0000 7f51f0fe2640 1 --2- 172.21.15.161:0/1377515248 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51ec14f930 0x7f51ec1ec4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.723+0000 7f51f0fe2640 1 -- 172.21.15.161:0/1377515248 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51ec1ec080 con 0x7f51ec14eae0 2024-03-28T23:30:52.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.723+0000 7f51f0fe2640 1 --2- 172.21.15.161:0/1377515248 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51ec14eae0 0x7f51ec14f3f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f51e000f660 tx=0x7f51e000fa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:52.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.723+0000 7f51ddffb640 1 -- 172.21.15.161:0/1377515248 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0014680 con 0x7f51ec14eae0 2024-03-28T23:30:52.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.723+0000 7f51ddffb640 1 -- 172.21.15.161:0/1377515248 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51e0019070 con 0x7f51ec14eae0 2024-03-28T23:30:52.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.724+0000 7f51ddffb640 1 -- 172.21.15.161:0/1377515248 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0014a60 con 0x7f51ec14eae0 2024-03-28T23:30:52.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.724+0000 7f51f3a47640 1 -- 172.21.15.161:0/1377515248 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51ec0c4fd0 con 0x7f51ec14eae0 2024-03-28T23:30:52.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.724+0000 7f51f3a47640 1 -- 172.21.15.161:0/1377515248 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51ec0c4520 con 0x7f51ec14eae0 2024-03-28T23:30:52.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.725+0000 7f51f3a47640 1 -- 172.21.15.161:0/1377515248 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51ec04f4b0 con 0x7f51ec14eae0 2024-03-28T23:30:52.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.725+0000 7f51ddffb640 1 -- 172.21.15.161:0/1377515248 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f51e0014c00 con 0x7f51ec14eae0 2024-03-28T23:30:52.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.725+0000 7f51ddffb640 1 --2- 172.21.15.161:0/1377515248 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f51d006b500 0x7f51d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:52.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.725+0000 7f51ddffb640 1 -- 172.21.15.161:0/1377515248 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f51e0089d30 con 0x7f51ec14eae0 2024-03-28T23:30:52.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.725+0000 7f51f17e3640 1 --2- 172.21.15.161:0/1377515248 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f51d006b500 0x7f51d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:52.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.726+0000 7f51f17e3640 1 --2- 172.21.15.161:0/1377515248 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f51d006b500 0x7f51d006d9c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f51e800ba00 tx=0x7f51e803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:52.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.730+0000 7f51ddffb640 1 -- 172.21.15.161:0/1377515248 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f51e0023020 con 0x7f51ec14eae0 2024-03-28T23:30:52.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.968+0000 7f51f3a47640 1 -- 172.21.15.161:0/1377515248 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f51ec0a9d20 con 0x7f51ec14eae0 2024-03-28T23:30:52.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.970+0000 7f51ddffb640 1 -- 172.21.15.161:0/1377515248 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v3) ==== 90+0+0 (secure 0 0 0) 0x7f51e0058e00 con 0x7f51ec14eae0 2024-03-28T23:30:52.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.972+0000 7f51f3a47640 1 -- 172.21.15.161:0/1377515248 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f51d006b500 msgr2=0x7f51d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:52.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.972+0000 7f51f3a47640 1 --2- 172.21.15.161:0/1377515248 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f51d006b500 0x7f51d006d9c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f51e800ba00 tx=0x7f51e803d040 comp rx=0 tx=0).stop 2024-03-28T23:30:52.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.972+0000 7f51f3a47640 1 -- 172.21.15.161:0/1377515248 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51ec14eae0 msgr2=0x7f51ec14f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:52.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.972+0000 7f51f3a47640 1 --2- 172.21.15.161:0/1377515248 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51ec14eae0 0x7f51ec14f3f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f51e000f660 tx=0x7f51e000fa60 comp rx=0 tx=0).stop 2024-03-28T23:30:52.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.972+0000 7f51f3a47640 1 -- 172.21.15.161:0/1377515248 shutdown_connections 2024-03-28T23:30:52.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.972+0000 7f51f3a47640 1 --2- 172.21.15.161:0/1377515248 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f51d006b500 0x7f51d006d9c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.972+0000 7f51f3a47640 1 --2- 172.21.15.161:0/1377515248 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51ec14eae0 0x7f51ec14f3f0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.972+0000 7f51f3a47640 1 --2- 172.21.15.161:0/1377515248 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51ec14f930 0x7f51ec1ec4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.972+0000 7f51f3a47640 1 --2- 172.21.15.161:0/1377515248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51d8003ae0 0x7f51ec147ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:52.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.972+0000 7f51f3a47640 1 -- 172.21.15.161:0/1377515248 >> 172.21.15.161:0/1377515248 conn(0x7f51ec0c6670 msgr2=0x7f51ec0732b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:52.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.972+0000 7f51f3a47640 1 -- 172.21.15.161:0/1377515248 shutdown_connections 2024-03-28T23:30:52.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:52.972+0000 7f51f3a47640 1 -- 172.21.15.161:0/1377515248 wait complete. 2024-03-28T23:30:52.983 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-28T23:30:53.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.084+0000 7f59f8be2640 1 Processor -- start 2024-03-28T23:30:53.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.084+0000 7f59f8be2640 1 -- start start 2024-03-28T23:30:53.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.085+0000 7f59f8be2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f59f4150590 0x7f59f40a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:53.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.085+0000 7f59f8be2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59f40a8fc0 0x7f59f4153150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:53.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.085+0000 7f59f8be2640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59f41566f0 con 0x7f59f40aacb0 2024-03-28T23:30:53.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.085+0000 7f59f8be2640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f59f41569d0 con 0x7f59f4150590 2024-03-28T23:30:53.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.085+0000 7f59f8be2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f59f4156cb0 con 0x7f59f40a8fc0 2024-03-28T23:30:53.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.085+0000 7f59f259c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f59f4150590 0x7f59f40a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:53.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.085+0000 7f59f2d9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f59f40aacb0 0x7f59f4153690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:46448/0 (socket says 172.21.15.161:46448) 2024-03-28T23:30:53.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.085+0000 7f59f259c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f59f4150590 0x7f59f40a9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46738/0 (socket says 172.21.15.161:46738) 2024-03-28T23:30:53.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.085+0000 7f59f2d9d640 1 -- 172.21.15.161:0/2088439162 learned_addr learned my addr 172.21.15.161:0/2088439162 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:53.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.086+0000 7f59f1d9b640 1 --2- 172.21.15.161:0/2088439162 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59f40a8fc0 0x7f59f4153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:53.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.086+0000 7f59f259c640 1 -- 172.21.15.161:0/2088439162 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59f40a8fc0 msgr2=0x7f59f4153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:53.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.086+0000 7f59f259c640 1 --2- 172.21.15.161:0/2088439162 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59f40a8fc0 0x7f59f4153150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.086+0000 7f59f259c640 1 -- 172.21.15.161:0/2088439162 >> v1:172.21.15.161:6789/0 conn(0x7f59f40aacb0 legacy=0x7f59f4153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:53.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.086+0000 7f59f259c640 1 -- 172.21.15.161:0/2088439162 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f4156f90 con 0x7f59f4150590 2024-03-28T23:30:53.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.086+0000 7f59f159a640 1 -- 172.21.15.161:0/2088439162 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1202495530 0 0) 0x7f59f41566f0 con 0x7f59f40aacb0 2024-03-28T23:30:53.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.086+0000 7f59f1d9b640 1 --2- 172.21.15.161:0/2088439162 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59f40a8fc0 0x7f59f4153150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:53.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.086+0000 7f59f259c640 1 --2- 172.21.15.161:0/2088439162 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59f4150590 0x7f59f40a9f40 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f59e4009d90 tx=0x7f59e402bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb4e048490e0efb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:53.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.087+0000 7f59f159a640 1 -- 172.21.15.161:0/2088439162 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e4039070 con 0x7f59f4150590 2024-03-28T23:30:53.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.087+0000 7f59f159a640 1 -- 172.21.15.161:0/2088439162 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59e402cb30 con 0x7f59f4150590 2024-03-28T23:30:53.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.087+0000 7f59f159a640 1 -- 172.21.15.161:0/2088439162 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e4034440 con 0x7f59f4150590 2024-03-28T23:30:53.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.087+0000 7f59f8be2640 1 -- 172.21.15.161:0/2088439162 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59f4150590 msgr2=0x7f59f40a9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:53.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.087+0000 7f59f8be2640 1 --2- 172.21.15.161:0/2088439162 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59f4150590 0x7f59f40a9f40 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f59e4009d90 tx=0x7f59e402bd80 comp rx=0 tx=0).stop 2024-03-28T23:30:53.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.087+0000 7f59f8be2640 1 -- 172.21.15.161:0/2088439162 shutdown_connections 2024-03-28T23:30:53.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.087+0000 7f59f8be2640 1 --2- 172.21.15.161:0/2088439162 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59f40a8fc0 0x7f59f4153150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.087+0000 7f59f8be2640 1 --2- 172.21.15.161:0/2088439162 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59f4150590 0x7f59f40a9f40 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.087+0000 7f59f8be2640 1 -- 172.21.15.161:0/2088439162 >> 172.21.15.161:0/2088439162 conn(0x7f59f40c6670 msgr2=0x7f59f40c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:53.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.087+0000 7f59f8be2640 1 -- 172.21.15.161:0/2088439162 shutdown_connections 2024-03-28T23:30:53.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.087+0000 7f59f8be2640 1 -- 172.21.15.161:0/2088439162 wait complete. 2024-03-28T23:30:53.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.089+0000 7f59f8be2640 1 Processor -- start 2024-03-28T23:30:53.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.089+0000 7f59f8be2640 1 -- start start 2024-03-28T23:30:53.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.089+0000 7f59f8be2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59f40a8fc0 0x7f59f4147f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:53.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.089+0000 7f59f8be2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59f40aacb0 0x7f59f41e90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:53.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.089+0000 7f59f8be2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59f4150590 0x7f59f40c59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:53.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.089+0000 7f59f8be2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f59f4150970 con 0x7f59f40aacb0 2024-03-28T23:30:53.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.089+0000 7f59f8be2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f59f40aa9c0 con 0x7f59f40a8fc0 2024-03-28T23:30:53.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.089+0000 7f59f8be2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f59f40aa480 con 0x7f59f4150590 2024-03-28T23:30:53.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.089+0000 7f59f1d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59f40aacb0 0x7f59f41e90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:53.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.089+0000 7f59f1d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59f40aacb0 0x7f59f41e90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46740/0 (socket says 172.21.15.161:46740) 2024-03-28T23:30:53.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.089+0000 7f59f1d9b640 1 -- 172.21.15.161:0/808580325 learned_addr learned my addr 172.21.15.161:0/808580325 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:53.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.090+0000 7f59f1d9b640 1 -- 172.21.15.161:0/808580325 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59f4150590 msgr2=0x7f59f40c59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:53.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.090+0000 7f59f1d9b640 1 --2- 172.21.15.161:0/808580325 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59f4150590 0x7f59f40c59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.090+0000 7f59f1d9b640 1 -- 172.21.15.161:0/808580325 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59f40a8fc0 msgr2=0x7f59f4147f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:53.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.090+0000 7f59f1d9b640 1 --2- 172.21.15.161:0/808580325 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59f40a8fc0 0x7f59f4147f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.090+0000 7f59f1d9b640 1 -- 172.21.15.161:0/808580325 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f41e9600 con 0x7f59f40aacb0 2024-03-28T23:30:53.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.090+0000 7f59f1d9b640 1 --2- 172.21.15.161:0/808580325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59f40aacb0 0x7f59f41e90c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f59e800c7f0 tx=0x7f59e800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:53.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.090+0000 7f59e2ffd640 1 -- 172.21.15.161:0/808580325 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e8015070 con 0x7f59f40aacb0 2024-03-28T23:30:53.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.090+0000 7f59e2ffd640 1 -- 172.21.15.161:0/808580325 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59e8007a30 con 0x7f59f40aacb0 2024-03-28T23:30:53.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.090+0000 7f59f8be2640 1 -- 172.21.15.161:0/808580325 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59f4151870 con 0x7f59f40aacb0 2024-03-28T23:30:53.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.090+0000 7f59e2ffd640 1 -- 172.21.15.161:0/808580325 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e8007e00 con 0x7f59f40aacb0 2024-03-28T23:30:53.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.090+0000 7f59f8be2640 1 -- 172.21.15.161:0/808580325 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59f4141be0 con 0x7f59f40aacb0 2024-03-28T23:30:53.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.091+0000 7f59f8be2640 1 -- 172.21.15.161:0/808580325 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59f404f4b0 con 0x7f59f40aacb0 2024-03-28T23:30:53.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.092+0000 7f59e2ffd640 1 -- 172.21.15.161:0/808580325 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f59e8018070 con 0x7f59f40aacb0 2024-03-28T23:30:53.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.092+0000 7f59e2ffd640 1 --2- 172.21.15.161:0/808580325 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f59d406b4d0 0x7f59d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:53.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.092+0000 7f59e2ffd640 1 -- 172.21.15.161:0/808580325 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f59e8089ed0 con 0x7f59f40aacb0 2024-03-28T23:30:53.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.092+0000 7f59f259c640 1 --2- 172.21.15.161:0/808580325 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f59d406b4d0 0x7f59d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:53.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.092+0000 7f59f259c640 1 --2- 172.21.15.161:0/808580325 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f59d406b4d0 0x7f59d406d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f59e4036000 tx=0x7f59e402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:53.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.097+0000 7f59e2ffd640 1 -- 172.21.15.161:0/808580325 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f59f41e9600 con 0x7f59f40aacb0 2024-03-28T23:30:53.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.338+0000 7f59f8be2640 1 -- 172.21.15.161:0/808580325 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7f59f4149920 con 0x7f59f40aacb0 2024-03-28T23:30:53.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.340+0000 7f59e2ffd640 1 -- 172.21.15.161:0/808580325 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v4) ==== 90+0+0 (secure 0 0 0) 0x7f59e8058fa0 con 0x7f59f40aacb0 2024-03-28T23:30:53.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.342+0000 7f59f8be2640 1 -- 172.21.15.161:0/808580325 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f59d406b4d0 msgr2=0x7f59d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:53.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.342+0000 7f59f8be2640 1 --2- 172.21.15.161:0/808580325 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f59d406b4d0 0x7f59d406d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f59e4036000 tx=0x7f59e402e000 comp rx=0 tx=0).stop 2024-03-28T23:30:53.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.342+0000 7f59f8be2640 1 -- 172.21.15.161:0/808580325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59f40aacb0 msgr2=0x7f59f41e90c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:53.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.342+0000 7f59f8be2640 1 --2- 172.21.15.161:0/808580325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59f40aacb0 0x7f59f41e90c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f59e800c7f0 tx=0x7f59e800ccb0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.342+0000 7f59f8be2640 1 -- 172.21.15.161:0/808580325 shutdown_connections 2024-03-28T23:30:53.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.342+0000 7f59f8be2640 1 --2- 172.21.15.161:0/808580325 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f59d406b4d0 0x7f59d406d990 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.342+0000 7f59f8be2640 1 --2- 172.21.15.161:0/808580325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59f40aacb0 0x7f59f41e90c0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.342+0000 7f59f8be2640 1 --2- 172.21.15.161:0/808580325 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59f4150590 0x7f59f40c59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.342+0000 7f59f8be2640 1 --2- 172.21.15.161:0/808580325 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59f40a8fc0 0x7f59f4147f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.342+0000 7f59f8be2640 1 -- 172.21.15.161:0/808580325 >> 172.21.15.161:0/808580325 conn(0x7f59f40c6670 msgr2=0x7f59f414d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:53.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.342+0000 7f59f8be2640 1 -- 172.21.15.161:0/808580325 shutdown_connections 2024-03-28T23:30:53.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.342+0000 7f59f8be2640 1 -- 172.21.15.161:0/808580325 wait complete. 2024-03-28T23:30:53.353 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-28T23:30:53.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.456+0000 7f079a607640 1 Processor -- start 2024-03-28T23:30:53.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.456+0000 7f079a607640 1 -- start start 2024-03-28T23:30:53.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.456+0000 7f079a607640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f07940a8fc0 0x7f0794153270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:53.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.456+0000 7f079a607640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f07940aacb0 0x7f07941537b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:53.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.456+0000 7f079a607640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07941567c0 con 0x7f0794150600 2024-03-28T23:30:53.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.456+0000 7f079a607640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0794156570 con 0x7f07940aacb0 2024-03-28T23:30:53.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.456+0000 7f079a607640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f0794156aa0 con 0x7f07940a8fc0 2024-03-28T23:30:53.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.457+0000 7f0798ba4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f07940aacb0 0x7f07941537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:53.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.457+0000 7f0798ba4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f07940aacb0 0x7f07941537b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46750/0 (socket says 172.21.15.161:46750) 2024-03-28T23:30:53.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.457+0000 7f0798ba4640 1 -- 172.21.15.161:0/1865809290 learned_addr learned my addr 172.21.15.161:0/1865809290 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:53.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.457+0000 7f07937fe640 1 --2- 172.21.15.161:0/1865809290 >> v2:172.21.15.170:3300/0 conn(0x7f07940a8fc0 0x7f0794153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:53.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.457+0000 7f0798ba4640 1 -- 172.21.15.161:0/1865809290 >> v2:172.21.15.170:3300/0 conn(0x7f07940a8fc0 msgr2=0x7f0794153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:53.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.457+0000 7f0798ba4640 1 --2- 172.21.15.161:0/1865809290 >> v2:172.21.15.170:3300/0 conn(0x7f07940a8fc0 0x7f0794153270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.457+0000 7f0798ba4640 1 -- 172.21.15.161:0/1865809290 >> v1:172.21.15.170:6789/0 conn(0x7f0794150600 legacy=0x7f07940a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:53.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.457+0000 7f0798ba4640 1 -- 172.21.15.161:0/1865809290 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0794156d80 con 0x7f07940aacb0 2024-03-28T23:30:53.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.457+0000 7f07937fe640 1 --2- 172.21.15.161:0/1865809290 >> v2:172.21.15.170:3300/0 conn(0x7f07940a8fc0 0x7f0794153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:53.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.458+0000 7f0798ba4640 1 --2- 172.21.15.161:0/1865809290 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07940aacb0 0x7f07941537b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f079406f570 tx=0x7f078c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55af533efc48abb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:53.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.458+0000 7f0792ffd640 1 -- 172.21.15.161:0/1865809290 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c02ca80 con 0x7f07940aacb0 2024-03-28T23:30:53.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.458+0000 7f0792ffd640 1 -- 172.21.15.161:0/1865809290 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f078c02cc20 con 0x7f07940aacb0 2024-03-28T23:30:53.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.458+0000 7f0792ffd640 1 -- 172.21.15.161:0/1865809290 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c0343f0 con 0x7f07940aacb0 2024-03-28T23:30:53.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.458+0000 7f079a607640 1 -- 172.21.15.161:0/1865809290 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07940aacb0 msgr2=0x7f07941537b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:53.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.458+0000 7f079a607640 1 --2- 172.21.15.161:0/1865809290 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07940aacb0 0x7f07941537b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f079406f570 tx=0x7f078c02bee0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.458+0000 7f079a607640 1 -- 172.21.15.161:0/1865809290 shutdown_connections 2024-03-28T23:30:53.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.458+0000 7f079a607640 1 --2- 172.21.15.161:0/1865809290 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07940aacb0 0x7f07941537b0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.458+0000 7f079a607640 1 --2- 172.21.15.161:0/1865809290 >> v2:172.21.15.170:3300/0 conn(0x7f07940a8fc0 0x7f0794153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.458+0000 7f079a607640 1 -- 172.21.15.161:0/1865809290 >> 172.21.15.161:0/1865809290 conn(0x7f07940c6670 msgr2=0x7f07940c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:53.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.458+0000 7f079a607640 1 -- 172.21.15.161:0/1865809290 shutdown_connections 2024-03-28T23:30:53.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.458+0000 7f079a607640 1 -- 172.21.15.161:0/1865809290 wait complete. 2024-03-28T23:30:53.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.460+0000 7f079a607640 1 Processor -- start 2024-03-28T23:30:53.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.460+0000 7f079a607640 1 -- start start 2024-03-28T23:30:53.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.460+0000 7f079a607640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07940a8fc0 0x7f07941e92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:53.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.460+0000 7f079a607640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07940aacb0 0x7f07940c5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:53.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.460+0000 7f079a607640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0794150600 0x7f079413e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:53.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.460+0000 7f079a607640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f07941509e0 con 0x7f07940a8fc0 2024-03-28T23:30:53.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.460+0000 7f079a607640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0794157340 con 0x7f07940aacb0 2024-03-28T23:30:53.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.460+0000 7f079a607640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f07940aa9e0 con 0x7f0794150600 2024-03-28T23:30:53.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.460+0000 7f0793fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07940a8fc0 0x7f07941e92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:53.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.461+0000 7f0793fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07940a8fc0 0x7f07941e92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46758/0 (socket says 172.21.15.161:46758) 2024-03-28T23:30:53.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.461+0000 7f0793fff640 1 -- 172.21.15.161:0/4084657786 learned_addr learned my addr 172.21.15.161:0/4084657786 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:53.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.461+0000 7f07937fe640 1 --2- 172.21.15.161:0/4084657786 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07940aacb0 0x7f07940c5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:53.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.461+0000 7f0798ba4640 1 --2- 172.21.15.161:0/4084657786 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0794150600 0x7f079413e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:53.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.461+0000 7f0793fff640 1 -- 172.21.15.161:0/4084657786 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0794150600 msgr2=0x7f079413e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:53.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.461+0000 7f0793fff640 1 --2- 172.21.15.161:0/4084657786 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0794150600 0x7f079413e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.461+0000 7f0793fff640 1 -- 172.21.15.161:0/4084657786 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07940aacb0 msgr2=0x7f07940c5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:53.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.461+0000 7f0793fff640 1 --2- 172.21.15.161:0/4084657786 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07940aacb0 0x7f07940c5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.461+0000 7f0793fff640 1 -- 172.21.15.161:0/4084657786 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f079414aa50 con 0x7f07940a8fc0 2024-03-28T23:30:53.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.461+0000 7f0793fff640 1 --2- 172.21.15.161:0/4084657786 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07940a8fc0 0x7f07941e92a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f078400c430 tx=0x7f078400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:53.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.461+0000 7f0790ff9640 1 -- 172.21.15.161:0/4084657786 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07840052d0 con 0x7f07940a8fc0 2024-03-28T23:30:53.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.461+0000 7f0790ff9640 1 -- 172.21.15.161:0/4084657786 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0784005490 con 0x7f07940a8fc0 2024-03-28T23:30:53.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.461+0000 7f079a607640 1 -- 172.21.15.161:0/4084657786 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0794148fb0 con 0x7f07940a8fc0 2024-03-28T23:30:53.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.461+0000 7f0790ff9640 1 -- 172.21.15.161:0/4084657786 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0784007260 con 0x7f07940a8fc0 2024-03-28T23:30:53.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.461+0000 7f079a607640 1 -- 172.21.15.161:0/4084657786 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0794149570 con 0x7f07940a8fc0 2024-03-28T23:30:53.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.462+0000 7f079a607640 1 -- 172.21.15.161:0/4084657786 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f079404f4b0 con 0x7f07940a8fc0 2024-03-28T23:30:53.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.463+0000 7f0790ff9640 1 -- 172.21.15.161:0/4084657786 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0784007400 con 0x7f07940a8fc0 2024-03-28T23:30:53.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.463+0000 7f0790ff9640 1 --2- 172.21.15.161:0/4084657786 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f078006b500 0x7f078006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:53.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.463+0000 7f0790ff9640 1 -- 172.21.15.161:0/4084657786 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f07840894a0 con 0x7f07940a8fc0 2024-03-28T23:30:53.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.463+0000 7f07937fe640 1 --2- 172.21.15.161:0/4084657786 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f078006b500 0x7f078006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:53.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.463+0000 7f07937fe640 1 --2- 172.21.15.161:0/4084657786 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f078006b500 0x7f078006d9c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f07940733d0 tx=0x7f0788008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:53.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.467+0000 7f0790ff9640 1 -- 172.21.15.161:0/4084657786 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0784058570 con 0x7f07940a8fc0 2024-03-28T23:30:53.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.708+0000 7f079a607640 1 -- 172.21.15.161:0/4084657786 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f07940c3240 con 0x7f07940a8fc0 2024-03-28T23:30:53.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.710+0000 7f0790ff9640 1 -- 172.21.15.161:0/4084657786 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7f0784057e70 con 0x7f07940a8fc0 2024-03-28T23:30:53.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.713+0000 7f079a607640 1 -- 172.21.15.161:0/4084657786 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f078006b500 msgr2=0x7f078006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:53.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.713+0000 7f079a607640 1 --2- 172.21.15.161:0/4084657786 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f078006b500 0x7f078006d9c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f07940733d0 tx=0x7f0788008040 comp rx=0 tx=0).stop 2024-03-28T23:30:53.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.713+0000 7f079a607640 1 -- 172.21.15.161:0/4084657786 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07940a8fc0 msgr2=0x7f07941e92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:53.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.713+0000 7f079a607640 1 --2- 172.21.15.161:0/4084657786 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07940a8fc0 0x7f07941e92a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f078400c430 tx=0x7f078400c8f0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.713+0000 7f079a607640 1 -- 172.21.15.161:0/4084657786 shutdown_connections 2024-03-28T23:30:53.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.713+0000 7f079a607640 1 --2- 172.21.15.161:0/4084657786 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f078006b500 0x7f078006d9c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.713+0000 7f079a607640 1 --2- 172.21.15.161:0/4084657786 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0794150600 0x7f079413e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.713+0000 7f079a607640 1 --2- 172.21.15.161:0/4084657786 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07940aacb0 0x7f07940c5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.713+0000 7f079a607640 1 --2- 172.21.15.161:0/4084657786 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07940a8fc0 0x7f07941e92a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.713+0000 7f079a607640 1 -- 172.21.15.161:0/4084657786 >> 172.21.15.161:0/4084657786 conn(0x7f07940c6670 msgr2=0x7f07940728a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:53.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.713+0000 7f079a607640 1 -- 172.21.15.161:0/4084657786 shutdown_connections 2024-03-28T23:30:53.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.713+0000 7f079a607640 1 -- 172.21.15.161:0/4084657786 wait complete. 2024-03-28T23:30:53.724 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-28T23:30:53.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.825+0000 7fa631f65640 1 Processor -- start 2024-03-28T23:30:53.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.825+0000 7fa631f65640 1 -- start start 2024-03-28T23:30:53.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.826+0000 7fa631f65640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa62c14eca0 0x7fa62c0c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:53.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.826+0000 7fa631f65640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa62c0a98c0 0x7fa62c0a9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:53.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.826+0000 7fa631f65640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa62c151850 con 0x7fa62c0c1dc0 2024-03-28T23:30:53.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.826+0000 7fa631f65640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa62c152460 con 0x7fa62c14eca0 2024-03-28T23:30:53.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.826+0000 7fa631f65640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa62c151b30 con 0x7fa62c0a98c0 2024-03-28T23:30:53.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.826+0000 7fa62b7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa62c14eca0 0x7fa62c0c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:53.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.826+0000 7fa62affd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa62c0c1dc0 0x7fa62c06ca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:46454/0 (socket says 172.21.15.161:46454) 2024-03-28T23:30:53.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.826+0000 7fa62affd640 1 -- 172.21.15.161:0/1499607472 learned_addr learned my addr 172.21.15.161:0/1499607472 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:53.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.826+0000 7fa62bfff640 1 --2- 172.21.15.161:0/1499607472 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa62c0a98c0 0x7fa62c0a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:53.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.826+0000 7fa62b7fe640 1 -- 172.21.15.161:0/1499607472 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa62c0a98c0 msgr2=0x7fa62c0a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:53.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.826+0000 7fa62b7fe640 1 --2- 172.21.15.161:0/1499607472 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa62c0a98c0 0x7fa62c0a9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.826+0000 7fa62b7fe640 1 -- 172.21.15.161:0/1499607472 >> v1:172.21.15.161:6789/0 conn(0x7fa62c0c1dc0 legacy=0x7fa62c06ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:53.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.826+0000 7fa62a7fc640 1 -- 172.21.15.161:0/1499607472 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2252667573 0 0) 0x7fa62c151850 con 0x7fa62c0c1dc0 2024-03-28T23:30:53.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.826+0000 7fa62b7fe640 1 -- 172.21.15.161:0/1499607472 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa62c151e10 con 0x7fa62c14eca0 2024-03-28T23:30:53.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.827+0000 7fa62b7fe640 1 --2- 172.21.15.161:0/1499607472 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa62c14eca0 0x7fa62c0c1880 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa61c0097a0 tx=0x7fa61c02c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28a5b0410916e2c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:53.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.827+0000 7fa62a7fc640 1 -- 172.21.15.161:0/1499607472 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61c035070 con 0x7fa62c14eca0 2024-03-28T23:30:53.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.827+0000 7fa62a7fc640 1 -- 172.21.15.161:0/1499607472 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa61c031610 con 0x7fa62c14eca0 2024-03-28T23:30:53.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.827+0000 7fa62a7fc640 1 -- 172.21.15.161:0/1499607472 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61c0319e0 con 0x7fa62c14eca0 2024-03-28T23:30:53.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.828+0000 7fa631f65640 1 -- 172.21.15.161:0/1499607472 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa62c14eca0 msgr2=0x7fa62c0c1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:53.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.828+0000 7fa631f65640 1 --2- 172.21.15.161:0/1499607472 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa62c14eca0 0x7fa62c0c1880 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa61c0097a0 tx=0x7fa61c02c950 comp rx=0 tx=0).stop 2024-03-28T23:30:53.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.828+0000 7fa631f65640 1 -- 172.21.15.161:0/1499607472 shutdown_connections 2024-03-28T23:30:53.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.828+0000 7fa631f65640 1 --2- 172.21.15.161:0/1499607472 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa62c0a98c0 0x7fa62c0a9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.828+0000 7fa631f65640 1 --2- 172.21.15.161:0/1499607472 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa62c14eca0 0x7fa62c0c1880 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.828+0000 7fa631f65640 1 -- 172.21.15.161:0/1499607472 >> 172.21.15.161:0/1499607472 conn(0x7fa62c0c6670 msgr2=0x7fa62c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:53.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.828+0000 7fa631f65640 1 -- 172.21.15.161:0/1499607472 shutdown_connections 2024-03-28T23:30:53.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.828+0000 7fa631f65640 1 -- 172.21.15.161:0/1499607472 wait complete. 2024-03-28T23:30:53.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.829+0000 7fa631f65640 1 Processor -- start 2024-03-28T23:30:53.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.829+0000 7fa631f65640 1 -- start start 2024-03-28T23:30:53.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.830+0000 7fa631f65640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa62c0a98c0 0x7fa62c0bfc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:53.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.830+0000 7fa631f65640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa62c0c1dc0 0x7fa62c153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:53.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.830+0000 7fa631f65640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa62c14eca0 0x7fa62c154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:53.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.830+0000 7fa631f65640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1514b0 con 0x7fa62c14eca0 2024-03-28T23:30:53.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.830+0000 7fa631f65640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c14f080 con 0x7fa62c0a98c0 2024-03-28T23:30:53.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.830+0000 7fa631f65640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa62c151630 con 0x7fa62c0c1dc0 2024-03-28T23:30:53.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.830+0000 7fa62bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa62c14eca0 0x7fa62c154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:53.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.830+0000 7fa62bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa62c14eca0 0x7fa62c154230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46766/0 (socket says 172.21.15.161:46766) 2024-03-28T23:30:53.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.830+0000 7fa62bfff640 1 -- 172.21.15.161:0/1578359100 learned_addr learned my addr 172.21.15.161:0/1578359100 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:53.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.830+0000 7fa62affd640 1 --2- 172.21.15.161:0/1578359100 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa62c0c1dc0 0x7fa62c153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:53.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.830+0000 7fa62bfff640 1 -- 172.21.15.161:0/1578359100 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa62c0c1dc0 msgr2=0x7fa62c153cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:53.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.830+0000 7fa62b7fe640 1 --2- 172.21.15.161:0/1578359100 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa62c0a98c0 0x7fa62c0bfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:53.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.830+0000 7fa62bfff640 1 --2- 172.21.15.161:0/1578359100 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa62c0c1dc0 0x7fa62c153cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.830+0000 7fa62bfff640 1 -- 172.21.15.161:0/1578359100 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa62c0a98c0 msgr2=0x7fa62c0bfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:53.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.830+0000 7fa62bfff640 1 --2- 172.21.15.161:0/1578359100 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa62c0a98c0 0x7fa62c0bfc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:53.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.830+0000 7fa62bfff640 1 -- 172.21.15.161:0/1578359100 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa62c1ea8f0 con 0x7fa62c14eca0 2024-03-28T23:30:53.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.830+0000 7fa62bfff640 1 --2- 172.21.15.161:0/1578359100 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa62c14eca0 0x7fa62c154230 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fa62400cd70 tx=0x7fa62400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:53.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.830+0000 7fa613fff640 1 -- 172.21.15.161:0/1578359100 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa624003b90 con 0x7fa62c14eca0 2024-03-28T23:30:53.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.830+0000 7fa613fff640 1 -- 172.21.15.161:0/1578359100 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa624003d30 con 0x7fa62c14eca0 2024-03-28T23:30:53.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.830+0000 7fa613fff640 1 -- 172.21.15.161:0/1578359100 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa624004240 con 0x7fa62c14eca0 2024-03-28T23:30:53.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.830+0000 7fa631f65640 1 -- 172.21.15.161:0/1578359100 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa62c0c0510 con 0x7fa62c14eca0 2024-03-28T23:30:53.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.830+0000 7fa631f65640 1 -- 172.21.15.161:0/1578359100 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa62c1eae70 con 0x7fa62c14eca0 2024-03-28T23:30:53.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.831+0000 7fa631f65640 1 -- 172.21.15.161:0/1578359100 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa62c04f4b0 con 0x7fa62c14eca0 2024-03-28T23:30:53.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.832+0000 7fa613fff640 1 -- 172.21.15.161:0/1578359100 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa6240043e0 con 0x7fa62c14eca0 2024-03-28T23:30:53.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.832+0000 7fa613fff640 1 --2- 172.21.15.161:0/1578359100 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa61806b4d0 0x7fa61806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:53.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.832+0000 7fa613fff640 1 -- 172.21.15.161:0/1578359100 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7fa62408aad0 con 0x7fa62c14eca0 2024-03-28T23:30:53.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.832+0000 7fa62b7fe640 1 --2- 172.21.15.161:0/1578359100 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa61806b4d0 0x7fa61806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:53.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.833+0000 7fa62b7fe640 1 --2- 172.21.15.161:0/1578359100 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa61806b4d0 0x7fa61806d990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa61c037000 tx=0x7fa61c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:53.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:53.837+0000 7fa613fff640 1 -- 172.21.15.161:0/1578359100 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa624059ba0 con 0x7fa62c14eca0 2024-03-28T23:30:54.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.077+0000 7fa631f65640 1 -- 172.21.15.161:0/1578359100 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7fa62c14e2d0 con 0x7fa62c14eca0 2024-03-28T23:30:54.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.079+0000 7fa613fff640 1 -- 172.21.15.161:0/1578359100 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7fa6240594a0 con 0x7fa62c14eca0 2024-03-28T23:30:54.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.081+0000 7fa631f65640 1 -- 172.21.15.161:0/1578359100 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa61806b4d0 msgr2=0x7fa61806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:54.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.081+0000 7fa631f65640 1 --2- 172.21.15.161:0/1578359100 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa61806b4d0 0x7fa61806d990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa61c037000 tx=0x7fa61c008030 comp rx=0 tx=0).stop 2024-03-28T23:30:54.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.081+0000 7fa631f65640 1 -- 172.21.15.161:0/1578359100 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa62c14eca0 msgr2=0x7fa62c154230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:54.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.081+0000 7fa631f65640 1 --2- 172.21.15.161:0/1578359100 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa62c14eca0 0x7fa62c154230 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fa62400cd70 tx=0x7fa62400a660 comp rx=0 tx=0).stop 2024-03-28T23:30:54.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.081+0000 7fa631f65640 1 -- 172.21.15.161:0/1578359100 shutdown_connections 2024-03-28T23:30:54.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.081+0000 7fa631f65640 1 --2- 172.21.15.161:0/1578359100 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa61806b4d0 0x7fa61806d990 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.081+0000 7fa631f65640 1 --2- 172.21.15.161:0/1578359100 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa62c14eca0 0x7fa62c154230 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.081+0000 7fa631f65640 1 --2- 172.21.15.161:0/1578359100 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa62c0c1dc0 0x7fa62c153cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.081+0000 7fa631f65640 1 --2- 172.21.15.161:0/1578359100 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa62c0a98c0 0x7fa62c0bfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.081+0000 7fa631f65640 1 -- 172.21.15.161:0/1578359100 >> 172.21.15.161:0/1578359100 conn(0x7fa62c0c6670 msgr2=0x7fa62c0aa8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:54.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.082+0000 7fa631f65640 1 -- 172.21.15.161:0/1578359100 shutdown_connections 2024-03-28T23:30:54.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.082+0000 7fa631f65640 1 -- 172.21.15.161:0/1578359100 wait complete. 2024-03-28T23:30:54.093 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-28T23:30:54.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.194+0000 7fb394bdd640 1 Processor -- start 2024-03-28T23:30:54.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.194+0000 7fb394bdd640 1 -- start start 2024-03-28T23:30:54.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb394bdd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb39006ce80 0x7fb3900aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:54.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb394bdd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb3900a9680 0x7fb3900aab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:54.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb394bdd640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb39014c250 con 0x7fb39014be70 2024-03-28T23:30:54.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb394bdd640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fb3901529c0 con 0x7fb3900a9680 2024-03-28T23:30:54.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb394bdd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb3900a9dc0 con 0x7fb39006ce80 2024-03-28T23:30:54.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb38e59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb39014be70 0x7fb3900ab1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:46462/0 (socket says 172.21.15.161:46462) 2024-03-28T23:30:54.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb38e59c640 1 -- 172.21.15.161:0/1533375254 learned_addr learned my addr 172.21.15.161:0/1533375254 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:54.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb38ed9d640 1 --2- 172.21.15.161:0/1533375254 >> v2:172.21.15.170:3300/0 conn(0x7fb3900a9680 0x7fb3900aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:54.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb38dd9b640 1 -- 172.21.15.161:0/1533375254 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3641053191 0 0) 0x7fb39014c250 con 0x7fb39014be70 2024-03-28T23:30:54.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb385d9b640 1 --2- 172.21.15.161:0/1533375254 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb39006ce80 0x7fb3900aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:54.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb38dd9b640 1 -- 172.21.15.161:0/1533375254 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb37c005730 con 0x7fb39014be70 2024-03-28T23:30:54.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb38dd9b640 1 -- 172.21.15.161:0/1533375254 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2141740885 0 0) 0x7fb37c005730 con 0x7fb39014be70 2024-03-28T23:30:54.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb38dd9b640 1 -- 172.21.15.161:0/1533375254 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb39006ce80 msgr2=0x7fb3900aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:54.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb38dd9b640 1 --2- 172.21.15.161:0/1533375254 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb39006ce80 0x7fb3900aa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb38dd9b640 1 -- 172.21.15.161:0/1533375254 >> v2:172.21.15.170:3300/0 conn(0x7fb3900a9680 msgr2=0x7fb3900aab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:54.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb38dd9b640 1 --2- 172.21.15.161:0/1533375254 >> v2:172.21.15.170:3300/0 conn(0x7fb3900a9680 0x7fb3900aab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb38dd9b640 1 -- 172.21.15.161:0/1533375254 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3900aa0a0 con 0x7fb39014be70 2024-03-28T23:30:54.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb38dd9b640 1 -- 172.21.15.161:0/1533375254 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fb380001670 con 0x7fb39014be70 2024-03-28T23:30:54.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb38dd9b640 1 -- 172.21.15.161:0/1533375254 >> v1:172.21.15.161:6789/0 conn(0x7fb39014be70 legacy=0x7fb3900ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:54.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb38dd9b640 1 --2- 172.21.15.161:0/1533375254 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb39006ce80 0x7fb37c005ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:54.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb38dd9b640 1 --2- 172.21.15.161:0/1533375254 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb37c006220 0x7fb37c006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:54.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb38dd9b640 1 --2- 172.21.15.161:0/1533375254 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb37c006f50 0x7fb37c0073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:54.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb38dd9b640 1 -- 172.21.15.161:0/1533375254 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb3901529c0 con 0x7fb37c006220 2024-03-28T23:30:54.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb38dd9b640 1 -- 172.21.15.161:0/1533375254 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb3900a9dc0 con 0x7fb37c006f50 2024-03-28T23:30:54.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb38dd9b640 1 -- 172.21.15.161:0/1533375254 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb37c00dd90 con 0x7fb39006ce80 2024-03-28T23:30:54.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.195+0000 7fb385d9b640 1 --2- 172.21.15.161:0/1533375254 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb37c006220 0x7fb37c006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:54.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.196+0000 7fb38e59c640 1 --2- 172.21.15.161:0/1533375254 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb39006ce80 0x7fb37c005ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:54.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.196+0000 7fb38ed9d640 1 --2- 172.21.15.161:0/1533375254 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb37c006f50 0x7fb37c0073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:54.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.196+0000 7fb385d9b640 1 -- 172.21.15.161:0/1533375254 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb39006ce80 msgr2=0x7fb37c005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:54.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.196+0000 7fb385d9b640 1 --2- 172.21.15.161:0/1533375254 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb39006ce80 0x7fb37c005ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.196+0000 7fb385d9b640 1 -- 172.21.15.161:0/1533375254 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb37c006f50 msgr2=0x7fb37c0073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:54.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.196+0000 7fb385d9b640 1 --2- 172.21.15.161:0/1533375254 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb37c006f50 0x7fb37c0073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.196+0000 7fb385d9b640 1 -- 172.21.15.161:0/1533375254 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb37c005730 con 0x7fb37c006220 2024-03-28T23:30:54.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.196+0000 7fb385d9b640 1 --2- 172.21.15.161:0/1533375254 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb37c006220 0x7fb37c006a10 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb378009ac0 tx=0x7fb37c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19d073ee2194ea93 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:54.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.197+0000 7fb38dd9b640 1 -- 172.21.15.161:0/1533375254 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb378039070 con 0x7fb37c006220 2024-03-28T23:30:54.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.197+0000 7fb38dd9b640 1 -- 172.21.15.161:0/1533375254 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb37802c6b0 con 0x7fb37c006220 2024-03-28T23:30:54.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.197+0000 7fb38dd9b640 1 -- 172.21.15.161:0/1533375254 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb378009690 con 0x7fb37c006220 2024-03-28T23:30:54.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.197+0000 7fb394bdd640 1 -- 172.21.15.161:0/1533375254 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb37c006220 msgr2=0x7fb37c006a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:54.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.197+0000 7fb394bdd640 1 --2- 172.21.15.161:0/1533375254 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb37c006220 0x7fb37c006a10 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb378009ac0 tx=0x7fb37c007a40 comp rx=0 tx=0).stop 2024-03-28T23:30:54.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.197+0000 7fb38ed9d640 1 -- 172.21.15.161:0/1533375254 reap_dead start 2024-03-28T23:30:54.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.197+0000 7fb394bdd640 1 -- 172.21.15.161:0/1533375254 shutdown_connections 2024-03-28T23:30:54.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.197+0000 7fb394bdd640 1 -- 172.21.15.161:0/1533375254 >> 172.21.15.161:0/1533375254 conn(0x7fb3900c6670 msgr2=0x7fb3900c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:54.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.197+0000 7fb394bdd640 1 -- 172.21.15.161:0/1533375254 shutdown_connections 2024-03-28T23:30:54.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.197+0000 7fb394bdd640 1 -- 172.21.15.161:0/1533375254 wait complete. 2024-03-28T23:30:54.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.199+0000 7fb394bdd640 1 Processor -- start 2024-03-28T23:30:54.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.199+0000 7fb394bdd640 1 -- start start 2024-03-28T23:30:54.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.199+0000 7fb394bdd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb37c006220 0x7fb390148f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:54.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.199+0000 7fb394bdd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb390155110 0x7fb39006c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:54.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.199+0000 7fb394bdd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb39006c9a0 0x7fb39014edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:54.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.199+0000 7fb394bdd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb3900aba30 con 0x7fb390155110 2024-03-28T23:30:54.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.199+0000 7fb394bdd640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb3900ab8b0 con 0x7fb39006c9a0 2024-03-28T23:30:54.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.199+0000 7fb394bdd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb39006b150 con 0x7fb37c006220 2024-03-28T23:30:54.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.199+0000 7fb385d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb390155110 0x7fb39006c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:54.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.199+0000 7fb385d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb390155110 0x7fb39006c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46780/0 (socket says 172.21.15.161:46780) 2024-03-28T23:30:54.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.199+0000 7fb385d9b640 1 -- 172.21.15.161:0/619565553 learned_addr learned my addr 172.21.15.161:0/619565553 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:54.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.200+0000 7fb38ed9d640 1 --2- 172.21.15.161:0/619565553 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb39006c9a0 0x7fb39014edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:54.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.200+0000 7fb38e59c640 1 --2- 172.21.15.161:0/619565553 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb37c006220 0x7fb390148f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:54.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.200+0000 7fb385d9b640 1 -- 172.21.15.161:0/619565553 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb37c006220 msgr2=0x7fb390148f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:54.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.200+0000 7fb385d9b640 1 --2- 172.21.15.161:0/619565553 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb37c006220 0x7fb390148f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.200+0000 7fb385d9b640 1 -- 172.21.15.161:0/619565553 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb39006c9a0 msgr2=0x7fb39014edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:54.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.200+0000 7fb385d9b640 1 --2- 172.21.15.161:0/619565553 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb39006c9a0 0x7fb39014edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.200+0000 7fb385d9b640 1 -- 172.21.15.161:0/619565553 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb39014c480 con 0x7fb390155110 2024-03-28T23:30:54.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.200+0000 7fb38e59c640 1 --2- 172.21.15.161:0/619565553 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb37c006220 0x7fb390148f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:54.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.200+0000 7fb385d9b640 1 --2- 172.21.15.161:0/619565553 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb390155110 0x7fb39006c460 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb378037040 tx=0x7fb378004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:54.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.200+0000 7fb3877fe640 1 -- 172.21.15.161:0/619565553 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb378042020 con 0x7fb390155110 2024-03-28T23:30:54.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.200+0000 7fb3877fe640 1 -- 172.21.15.161:0/619565553 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb37803a040 con 0x7fb390155110 2024-03-28T23:30:54.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.200+0000 7fb394bdd640 1 -- 172.21.15.161:0/619565553 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb39014c760 con 0x7fb390155110 2024-03-28T23:30:54.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.200+0000 7fb3877fe640 1 -- 172.21.15.161:0/619565553 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb378039070 con 0x7fb390155110 2024-03-28T23:30:54.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.200+0000 7fb394bdd640 1 -- 172.21.15.161:0/619565553 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb39014cec0 con 0x7fb390155110 2024-03-28T23:30:54.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.201+0000 7fb394bdd640 1 -- 172.21.15.161:0/619565553 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb39004f4b0 con 0x7fb390155110 2024-03-28T23:30:54.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.201+0000 7fb3877fe640 1 -- 172.21.15.161:0/619565553 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb378045020 con 0x7fb390155110 2024-03-28T23:30:54.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.202+0000 7fb3877fe640 1 --2- 172.21.15.161:0/619565553 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb37006b4d0 0x7fb37006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:54.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.202+0000 7fb3877fe640 1 -- 172.21.15.161:0/619565553 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7fb3780aad40 con 0x7fb390155110 2024-03-28T23:30:54.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.202+0000 7fb38e59c640 1 --2- 172.21.15.161:0/619565553 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb37006b4d0 0x7fb37006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:54.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.202+0000 7fb38e59c640 1 --2- 172.21.15.161:0/619565553 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb37006b4d0 0x7fb37006d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb380001120 tx=0x7fb380009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:54.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.206+0000 7fb3877fe640 1 -- 172.21.15.161:0/619565553 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb378079e10 con 0x7fb390155110 2024-03-28T23:30:54.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.449+0000 7fb394bdd640 1 -- 172.21.15.161:0/619565553 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7fb390155f70 con 0x7fb390155110 2024-03-28T23:30:54.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.451+0000 7fb3877fe640 1 -- 172.21.15.161:0/619565553 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v7) ==== 97+0+0 (secure 0 0 0) 0x7fb378079710 con 0x7fb390155110 2024-03-28T23:30:54.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.453+0000 7fb394bdd640 1 -- 172.21.15.161:0/619565553 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb37006b4d0 msgr2=0x7fb37006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:54.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.453+0000 7fb394bdd640 1 --2- 172.21.15.161:0/619565553 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb37006b4d0 0x7fb37006d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb380001120 tx=0x7fb380009040 comp rx=0 tx=0).stop 2024-03-28T23:30:54.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.453+0000 7fb394bdd640 1 -- 172.21.15.161:0/619565553 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb390155110 msgr2=0x7fb39006c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:54.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.453+0000 7fb394bdd640 1 --2- 172.21.15.161:0/619565553 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb390155110 0x7fb39006c460 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb378037040 tx=0x7fb378004730 comp rx=0 tx=0).stop 2024-03-28T23:30:54.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.453+0000 7fb394bdd640 1 -- 172.21.15.161:0/619565553 shutdown_connections 2024-03-28T23:30:54.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.453+0000 7fb394bdd640 1 --2- 172.21.15.161:0/619565553 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb37006b4d0 0x7fb37006d990 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.453+0000 7fb394bdd640 1 --2- 172.21.15.161:0/619565553 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb390155110 0x7fb39006c460 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.453+0000 7fb394bdd640 1 --2- 172.21.15.161:0/619565553 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb39006c9a0 0x7fb39014edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.453+0000 7fb394bdd640 1 --2- 172.21.15.161:0/619565553 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb37c006220 0x7fb390148f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.453+0000 7fb394bdd640 1 -- 172.21.15.161:0/619565553 >> 172.21.15.161:0/619565553 conn(0x7fb3900c6670 msgr2=0x7fb3900c1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:54.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.453+0000 7fb394bdd640 1 -- 172.21.15.161:0/619565553 shutdown_connections 2024-03-28T23:30:54.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.453+0000 7fb394bdd640 1 -- 172.21.15.161:0/619565553 wait complete. 2024-03-28T23:30:54.465 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-28T23:30:54.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.566+0000 7f1e3106f640 1 Processor -- start 2024-03-28T23:30:54.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.566+0000 7f1e3106f640 1 -- start start 2024-03-28T23:30:54.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.566+0000 7f1e3106f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e2c06c540 0x7f1e2c0c1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:54.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.566+0000 7f1e3106f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e2c06f650 con 0x7f1e2c06c160 2024-03-28T23:30:54.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.566+0000 7f1e3106f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e2c06f370 con 0x7f1e2c06e720 2024-03-28T23:30:54.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.566+0000 7f1e3106f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1e2c06c9d0 con 0x7f1e2c06c540 2024-03-28T23:30:54.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.566+0000 7f1e2ad9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f1e2c06c160 0x7f1e2c150560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:46474/0 (socket says 172.21.15.161:46474) 2024-03-28T23:30:54.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.566+0000 7f1e2ad9d640 1 -- 172.21.15.161:0/3659754355 learned_addr learned my addr 172.21.15.161:0/3659754355 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:54.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.567+0000 7f1e29d9b640 1 -- 172.21.15.161:0/3659754355 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2543474 0 0) 0x7f1e2c06f650 con 0x7f1e2c06c160 2024-03-28T23:30:54.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.567+0000 7f1e29d9b640 1 -- 172.21.15.161:0/3659754355 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e18003520 con 0x7f1e2c06c160 2024-03-28T23:30:54.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.567+0000 7f1e2b59e640 1 --2- 172.21.15.161:0/3659754355 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e2c06c540 0x7f1e2c0c1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:54.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.567+0000 7f1e29d9b640 1 -- 172.21.15.161:0/3659754355 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2668250158 0 0) 0x7f1e18003520 con 0x7f1e2c06c160 2024-03-28T23:30:54.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.567+0000 7f1e29d9b640 1 -- 172.21.15.161:0/3659754355 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e2c06c540 msgr2=0x7f1e2c0c1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:54.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.567+0000 7f1e29d9b640 1 --2- 172.21.15.161:0/3659754355 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e2c06c540 0x7f1e2c0c1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.567+0000 7f1e29d9b640 1 -- 172.21.15.161:0/3659754355 >> v1:172.21.15.170:6789/0 conn(0x7f1e2c06e720 legacy=0x7f1e2c15b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:54.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.567+0000 7f1e29d9b640 1 -- 172.21.15.161:0/3659754355 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e2c06dc40 con 0x7f1e2c06c160 2024-03-28T23:30:54.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.567+0000 7f1e29d9b640 1 -- 172.21.15.161:0/3659754355 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f1e240019f0 con 0x7f1e2c06c160 2024-03-28T23:30:54.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.567+0000 7f1e29d9b640 1 -- 172.21.15.161:0/3659754355 >> v1:172.21.15.161:6789/0 conn(0x7f1e2c06c160 legacy=0x7f1e2c150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:54.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.567+0000 7f1e29d9b640 1 --2- 172.21.15.161:0/3659754355 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e2c06c540 0x7f1e18003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:54.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.567+0000 7f1e29d9b640 1 --2- 172.21.15.161:0/3659754355 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e180040f0 0x7f1e18004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:54.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.567+0000 7f1e29d9b640 1 --2- 172.21.15.161:0/3659754355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e18004e60 0x7f1e1800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:54.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.567+0000 7f1e29d9b640 1 -- 172.21.15.161:0/3659754355 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2c06c9d0 con 0x7f1e18004e60 2024-03-28T23:30:54.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.567+0000 7f1e29d9b640 1 -- 172.21.15.161:0/3659754355 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1e180055e0 con 0x7f1e180040f0 2024-03-28T23:30:54.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.567+0000 7f1e29d9b640 1 -- 172.21.15.161:0/3659754355 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1e1800bf00 con 0x7f1e2c06c540 2024-03-28T23:30:54.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.567+0000 7f1e2b59e640 1 --2- 172.21.15.161:0/3659754355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e18004e60 0x7f1e1800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:54.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.567+0000 7f1e2ad9d640 1 --2- 172.21.15.161:0/3659754355 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e2c06c540 0x7f1e18003ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:54.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.567+0000 7f1e2a59c640 1 --2- 172.21.15.161:0/3659754355 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e180040f0 0x7f1e18004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:54.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.567+0000 7f1e2b59e640 1 -- 172.21.15.161:0/3659754355 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e2c06c540 msgr2=0x7f1e18003ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:54.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.568+0000 7f1e2b59e640 1 --2- 172.21.15.161:0/3659754355 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e2c06c540 0x7f1e18003ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.568+0000 7f1e2b59e640 1 -- 172.21.15.161:0/3659754355 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e180040f0 msgr2=0x7f1e18004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:54.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.568+0000 7f1e2b59e640 1 --2- 172.21.15.161:0/3659754355 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e180040f0 0x7f1e18004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.568+0000 7f1e2b59e640 1 -- 172.21.15.161:0/3659754355 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e18003520 con 0x7f1e18004e60 2024-03-28T23:30:54.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.568+0000 7f1e2b59e640 1 --2- 172.21.15.161:0/3659754355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e18004e60 0x7f1e1800b9c0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c00ba30 tx=0x7f1e18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9568f7f5673cb244 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:54.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.568+0000 7f1e29d9b640 1 -- 172.21.15.161:0/3659754355 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c036020 con 0x7f1e18004e60 2024-03-28T23:30:54.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.568+0000 7f1e29d9b640 1 -- 172.21.15.161:0/3659754355 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e1c032640 con 0x7f1e18004e60 2024-03-28T23:30:54.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.568+0000 7f1e29d9b640 1 -- 172.21.15.161:0/3659754355 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c00b600 con 0x7f1e18004e60 2024-03-28T23:30:54.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.569+0000 7f1e3106f640 1 -- 172.21.15.161:0/3659754355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e18004e60 msgr2=0x7f1e1800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:54.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.569+0000 7f1e3106f640 1 --2- 172.21.15.161:0/3659754355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e18004e60 0x7f1e1800b9c0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c00ba30 tx=0x7f1e18005830 comp rx=0 tx=0).stop 2024-03-28T23:30:54.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.569+0000 7f1e2b59e640 1 -- 172.21.15.161:0/3659754355 reap_dead start 2024-03-28T23:30:54.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.569+0000 7f1e3106f640 1 -- 172.21.15.161:0/3659754355 shutdown_connections 2024-03-28T23:30:54.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.569+0000 7f1e3106f640 1 -- 172.21.15.161:0/3659754355 >> 172.21.15.161:0/3659754355 conn(0x7f1e2c0c3be0 msgr2=0x7f1e2c0c3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:54.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.569+0000 7f1e3106f640 1 -- 172.21.15.161:0/3659754355 shutdown_connections 2024-03-28T23:30:54.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.569+0000 7f1e3106f640 1 -- 172.21.15.161:0/3659754355 wait complete. 2024-03-28T23:30:54.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.570+0000 7f1e3106f640 1 Processor -- start 2024-03-28T23:30:54.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.570+0000 7f1e3106f640 1 -- start start 2024-03-28T23:30:54.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.571+0000 7f1e3106f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e2c153bc0 0x7f1e2c148b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:54.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.571+0000 7f1e3106f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e2c147990 0x7f1e2c147e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:54.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.571+0000 7f1e3106f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e2c1e89f0 0x7f1e2c1e8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:54.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.571+0000 7f1e3106f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2c0c2390 con 0x7f1e2c1e89f0 2024-03-28T23:30:54.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.571+0000 7f1e3106f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2c06e290 con 0x7f1e2c147990 2024-03-28T23:30:54.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.571+0000 7f1e3106f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1e2c06f000 con 0x7f1e2c153bc0 2024-03-28T23:30:54.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.571+0000 7f1e2b59e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e2c1e89f0 0x7f1e2c1e8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:54.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.571+0000 7f1e2b59e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e2c1e89f0 0x7f1e2c1e8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46800/0 (socket says 172.21.15.161:46800) 2024-03-28T23:30:54.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.571+0000 7f1e2b59e640 1 -- 172.21.15.161:0/1019551928 learned_addr learned my addr 172.21.15.161:0/1019551928 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:54.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.571+0000 7f1e2a59c640 1 --2- 172.21.15.161:0/1019551928 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e2c147990 0x7f1e2c147e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:54.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.571+0000 7f1e2ad9d640 1 --2- 172.21.15.161:0/1019551928 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e2c153bc0 0x7f1e2c148b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:54.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.571+0000 7f1e2b59e640 1 -- 172.21.15.161:0/1019551928 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e2c153bc0 msgr2=0x7f1e2c148b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:54.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.571+0000 7f1e2b59e640 1 --2- 172.21.15.161:0/1019551928 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e2c153bc0 0x7f1e2c148b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.571+0000 7f1e2b59e640 1 -- 172.21.15.161:0/1019551928 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e2c147990 msgr2=0x7f1e2c147e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:54.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.571+0000 7f1e2b59e640 1 --2- 172.21.15.161:0/1019551928 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e2c147990 0x7f1e2c147e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.571+0000 7f1e2b59e640 1 -- 172.21.15.161:0/1019551928 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e2c14cb90 con 0x7f1e2c1e89f0 2024-03-28T23:30:54.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.571+0000 7f1e2ad9d640 1 --2- 172.21.15.161:0/1019551928 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e2c153bc0 0x7f1e2c148b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:54.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.571+0000 7f1e2a59c640 1 --2- 172.21.15.161:0/1019551928 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e2c147990 0x7f1e2c147e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:54.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.572+0000 7f1e2b59e640 1 --2- 172.21.15.161:0/1019551928 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e2c1e89f0 0x7f1e2c1e8e50 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c00bb20 tx=0x7f1e1c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:54.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.572+0000 7f1e237fe640 1 -- 172.21.15.161:0/1019551928 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c036040 con 0x7f1e2c1e89f0 2024-03-28T23:30:54.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.572+0000 7f1e237fe640 1 -- 172.21.15.161:0/1019551928 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e1c040070 con 0x7f1e2c1e89f0 2024-03-28T23:30:54.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.572+0000 7f1e3106f640 1 -- 172.21.15.161:0/1019551928 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e2c1ece80 con 0x7f1e2c1e89f0 2024-03-28T23:30:54.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.572+0000 7f1e237fe640 1 -- 172.21.15.161:0/1019551928 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c00b600 con 0x7f1e2c1e89f0 2024-03-28T23:30:54.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.572+0000 7f1e3106f640 1 -- 172.21.15.161:0/1019551928 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e2c1ecba0 con 0x7f1e2c1e89f0 2024-03-28T23:30:54.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.573+0000 7f1e3106f640 1 -- 172.21.15.161:0/1019551928 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e2c067360 con 0x7f1e2c1e89f0 2024-03-28T23:30:54.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.573+0000 7f1e237fe640 1 -- 172.21.15.161:0/1019551928 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1e1c039020 con 0x7f1e2c1e89f0 2024-03-28T23:30:54.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.574+0000 7f1e237fe640 1 --2- 172.21.15.161:0/1019551928 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1e0c06b4d0 0x7f1e0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:54.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.574+0000 7f1e237fe640 1 -- 172.21.15.161:0/1019551928 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f1e1c0ac0d0 con 0x7f1e2c1e89f0 2024-03-28T23:30:54.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.574+0000 7f1e2ad9d640 1 --2- 172.21.15.161:0/1019551928 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1e0c06b4d0 0x7f1e0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:54.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.574+0000 7f1e2ad9d640 1 --2- 172.21.15.161:0/1019551928 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1e0c06b4d0 0x7f1e0c06d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1e24004c20 tx=0x7f1e24006040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:54.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.578+0000 7f1e237fe640 1 -- 172.21.15.161:0/1019551928 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1e2c067360 con 0x7f1e2c1e89f0 2024-03-28T23:30:54.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.815+0000 7f1e3106f640 1 -- 172.21.15.161:0/1019551928 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f1e2c1ec990 con 0x7f1e2c1e89f0 2024-03-28T23:30:54.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.817+0000 7f1e237fe640 1 -- 172.21.15.161:0/1019551928 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7f1e1c04c020 con 0x7f1e2c1e89f0 2024-03-28T23:30:54.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.819+0000 7f1e3106f640 1 -- 172.21.15.161:0/1019551928 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1e0c06b4d0 msgr2=0x7f1e0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:54.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.819+0000 7f1e3106f640 1 --2- 172.21.15.161:0/1019551928 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1e0c06b4d0 0x7f1e0c06d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1e24004c20 tx=0x7f1e24006040 comp rx=0 tx=0).stop 2024-03-28T23:30:54.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.820+0000 7f1e3106f640 1 -- 172.21.15.161:0/1019551928 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e2c1e89f0 msgr2=0x7f1e2c1e8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:54.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.820+0000 7f1e3106f640 1 --2- 172.21.15.161:0/1019551928 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e2c1e89f0 0x7f1e2c1e8e50 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c00bb20 tx=0x7f1e1c004730 comp rx=0 tx=0).stop 2024-03-28T23:30:54.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.820+0000 7f1e3106f640 1 -- 172.21.15.161:0/1019551928 shutdown_connections 2024-03-28T23:30:54.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.820+0000 7f1e3106f640 1 --2- 172.21.15.161:0/1019551928 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1e0c06b4d0 0x7f1e0c06d990 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.820+0000 7f1e3106f640 1 --2- 172.21.15.161:0/1019551928 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e2c1e89f0 0x7f1e2c1e8e50 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.820+0000 7f1e3106f640 1 --2- 172.21.15.161:0/1019551928 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e2c147990 0x7f1e2c147e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.820+0000 7f1e3106f640 1 --2- 172.21.15.161:0/1019551928 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e2c153bc0 0x7f1e2c148b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.820+0000 7f1e3106f640 1 -- 172.21.15.161:0/1019551928 >> 172.21.15.161:0/1019551928 conn(0x7f1e2c0c3be0 msgr2=0x7f1e2c14f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:54.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.820+0000 7f1e3106f640 1 -- 172.21.15.161:0/1019551928 shutdown_connections 2024-03-28T23:30:54.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.820+0000 7f1e3106f640 1 -- 172.21.15.161:0/1019551928 wait complete. 2024-03-28T23:30:54.832 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:30:54.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.932+0000 7f6e8fb5e640 1 Processor -- start 2024-03-28T23:30:54.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.932+0000 7f6e8fb5e640 1 -- start start 2024-03-28T23:30:54.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.932+0000 7f6e8fb5e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6e88150600 0x7f6e880a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:54.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.932+0000 7f6e8fb5e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6e880a8fc0 0x7f6e88153270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:54.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.933+0000 7f6e8fb5e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e881567c0 con 0x7f6e880aacb0 2024-03-28T23:30:54.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.933+0000 7f6e8fb5e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6e88156570 con 0x7f6e88150600 2024-03-28T23:30:54.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.933+0000 7f6e8fb5e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f6e88156aa0 con 0x7f6e880a8fc0 2024-03-28T23:30:54.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.933+0000 7f6e8d8fa640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6e88150600 0x7f6e880a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:54.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.933+0000 7f6e8d8fa640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6e88150600 0x7f6e880a9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46804/0 (socket says 172.21.15.161:46804) 2024-03-28T23:30:54.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.933+0000 7f6e8d8fa640 1 -- 172.21.15.161:0/267819690 learned_addr learned my addr 172.21.15.161:0/267819690 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:54.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.933+0000 7f6e8d0f9640 1 --2- 172.21.15.161:0/267819690 >> v2:172.21.15.170:3300/0 conn(0x7f6e880a8fc0 0x7f6e88153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:54.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.933+0000 7f6e8d8fa640 1 -- 172.21.15.161:0/267819690 >> v2:172.21.15.170:3300/0 conn(0x7f6e880a8fc0 msgr2=0x7f6e88153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:54.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.933+0000 7f6e8d8fa640 1 --2- 172.21.15.161:0/267819690 >> v2:172.21.15.170:3300/0 conn(0x7f6e880a8fc0 0x7f6e88153270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.933+0000 7f6e8d8fa640 1 -- 172.21.15.161:0/267819690 >> v1:172.21.15.170:6789/0 conn(0x7f6e880aacb0 legacy=0x7f6e881537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:54.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.933+0000 7f6e8d8fa640 1 -- 172.21.15.161:0/267819690 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e88156d80 con 0x7f6e88150600 2024-03-28T23:30:54.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.934+0000 7f6e8d8fa640 1 --2- 172.21.15.161:0/267819690 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e88150600 0x7f6e880a9f40 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6e8806f570 tx=0x7f6e8402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa1b8aca63ff90b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:54.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.934+0000 7f6e8c8f8640 1 -- 172.21.15.161:0/267819690 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e84035020 con 0x7f6e88150600 2024-03-28T23:30:54.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.934+0000 7f6e8c8f8640 1 -- 172.21.15.161:0/267819690 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e84031870 con 0x7f6e88150600 2024-03-28T23:30:54.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.934+0000 7f6e8c8f8640 1 -- 172.21.15.161:0/267819690 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e84031c40 con 0x7f6e88150600 2024-03-28T23:30:54.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.934+0000 7f6e8fb5e640 1 -- 172.21.15.161:0/267819690 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e88150600 msgr2=0x7f6e880a9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:54.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.934+0000 7f6e8fb5e640 1 --2- 172.21.15.161:0/267819690 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e88150600 0x7f6e880a9f40 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6e8806f570 tx=0x7f6e8402b910 comp rx=0 tx=0).stop 2024-03-28T23:30:54.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.934+0000 7f6e8fb5e640 1 -- 172.21.15.161:0/267819690 shutdown_connections 2024-03-28T23:30:54.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.934+0000 7f6e8fb5e640 1 --2- 172.21.15.161:0/267819690 >> v2:172.21.15.170:3300/0 conn(0x7f6e880a8fc0 0x7f6e88153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.934+0000 7f6e8fb5e640 1 --2- 172.21.15.161:0/267819690 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e88150600 0x7f6e880a9f40 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.934+0000 7f6e8fb5e640 1 -- 172.21.15.161:0/267819690 >> 172.21.15.161:0/267819690 conn(0x7f6e880c6670 msgr2=0x7f6e880c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:54.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.935+0000 7f6e8fb5e640 1 -- 172.21.15.161:0/267819690 shutdown_connections 2024-03-28T23:30:54.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.935+0000 7f6e8fb5e640 1 -- 172.21.15.161:0/267819690 wait complete. 2024-03-28T23:30:54.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.936+0000 7f6e8fb5e640 1 Processor -- start 2024-03-28T23:30:54.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.936+0000 7f6e8fb5e640 1 -- start start 2024-03-28T23:30:54.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.936+0000 7f6e8fb5e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e880a8fc0 0x7f6e881eb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:54.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.936+0000 7f6e8fb5e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e880aacb0 0x7f6e880c5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:54.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.936+0000 7f6e8fb5e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e88150600 0x7f6e8806b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:54.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.936+0000 7f6e8fb5e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6e88153ec0 con 0x7f6e880a8fc0 2024-03-28T23:30:54.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.936+0000 7f6e8fb5e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6e88157340 con 0x7f6e88150600 2024-03-28T23:30:54.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.936+0000 7f6e8fb5e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6e881509e0 con 0x7f6e880aacb0 2024-03-28T23:30:54.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.936+0000 7f6e8d8fa640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e880a8fc0 0x7f6e881eb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:54.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.936+0000 7f6e8d8fa640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e880a8fc0 0x7f6e881eb1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46820/0 (socket says 172.21.15.161:46820) 2024-03-28T23:30:54.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.936+0000 7f6e8d8fa640 1 -- 172.21.15.161:0/4273281003 learned_addr learned my addr 172.21.15.161:0/4273281003 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:54.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.936+0000 7f6e8d0f9640 1 --2- 172.21.15.161:0/4273281003 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e880aacb0 0x7f6e880c5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:54.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.936+0000 7f6e8e0fb640 1 --2- 172.21.15.161:0/4273281003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e88150600 0x7f6e8806b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:54.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.937+0000 7f6e8d8fa640 1 -- 172.21.15.161:0/4273281003 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e880aacb0 msgr2=0x7f6e880c5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:54.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.937+0000 7f6e8d8fa640 1 --2- 172.21.15.161:0/4273281003 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e880aacb0 0x7f6e880c5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.937+0000 7f6e8d8fa640 1 -- 172.21.15.161:0/4273281003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e88150600 msgr2=0x7f6e8806b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:54.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.937+0000 7f6e8d8fa640 1 --2- 172.21.15.161:0/4273281003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e88150600 0x7f6e8806b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:54.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.937+0000 7f6e8d8fa640 1 -- 172.21.15.161:0/4273281003 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e8806c420 con 0x7f6e880a8fc0 2024-03-28T23:30:54.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.937+0000 7f6e8d8fa640 1 --2- 172.21.15.161:0/4273281003 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e880a8fc0 0x7f6e881eb1c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6e8402b950 tx=0x7f6e840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:54.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.937+0000 7f6e7e7fc640 1 -- 172.21.15.161:0/4273281003 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e84040070 con 0x7f6e880a8fc0 2024-03-28T23:30:54.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.937+0000 7f6e7e7fc640 1 -- 172.21.15.161:0/4273281003 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e84030d00 con 0x7f6e880a8fc0 2024-03-28T23:30:54.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.937+0000 7f6e8fb5e640 1 -- 172.21.15.161:0/4273281003 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e881eb700 con 0x7f6e880a8fc0 2024-03-28T23:30:54.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.937+0000 7f6e7e7fc640 1 -- 172.21.15.161:0/4273281003 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e84035040 con 0x7f6e880a8fc0 2024-03-28T23:30:54.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.937+0000 7f6e8fb5e640 1 -- 172.21.15.161:0/4273281003 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e8806c9a0 con 0x7f6e880a8fc0 2024-03-28T23:30:54.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.938+0000 7f6e8fb5e640 1 -- 172.21.15.161:0/4273281003 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e8804f4b0 con 0x7f6e880a8fc0 2024-03-28T23:30:54.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.938+0000 7f6e7e7fc640 1 -- 172.21.15.161:0/4273281003 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6e8404b060 con 0x7f6e880a8fc0 2024-03-28T23:30:54.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.939+0000 7f6e7e7fc640 1 --2- 172.21.15.161:0/4273281003 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6e6c06b4d0 0x7f6e6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:54.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.939+0000 7f6e7e7fc640 1 -- 172.21.15.161:0/4273281003 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f6e840abca0 con 0x7f6e880a8fc0 2024-03-28T23:30:54.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.939+0000 7f6e8d0f9640 1 --2- 172.21.15.161:0/4273281003 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6e6c06b4d0 0x7f6e6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:54.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.939+0000 7f6e8d0f9640 1 --2- 172.21.15.161:0/4273281003 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6e6c06b4d0 0x7f6e6c06d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6e880c3e40 tx=0x7f6e78008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:54.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:54.943+0000 7f6e7e7fc640 1 -- 172.21.15.161:0/4273281003 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6e8407ad70 con 0x7f6e880a8fc0 2024-03-28T23:30:55.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.115+0000 7f6e8fb5e640 1 -- 172.21.15.161:0/4273281003 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6e880a9d20 con 0x7f6e880a8fc0 2024-03-28T23:30:55.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.115+0000 7f6e7e7fc640 1 -- 172.21.15.161:0/4273281003 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+13200 (secure 0 0 0) 0x7f6e8407a670 con 0x7f6e880a8fc0 2024-03-28T23:30:55.116 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:55.116 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":34,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:30:52.237735+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:30:55.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.117+0000 7f6e8fb5e640 1 -- 172.21.15.161:0/4273281003 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6e6c06b4d0 msgr2=0x7f6e6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:55.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.117+0000 7f6e8fb5e640 1 --2- 172.21.15.161:0/4273281003 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6e6c06b4d0 0x7f6e6c06d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6e880c3e40 tx=0x7f6e78008040 comp rx=0 tx=0).stop 2024-03-28T23:30:55.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.117+0000 7f6e8fb5e640 1 -- 172.21.15.161:0/4273281003 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e880a8fc0 msgr2=0x7f6e881eb1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:55.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.117+0000 7f6e8fb5e640 1 --2- 172.21.15.161:0/4273281003 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e880a8fc0 0x7f6e881eb1c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6e8402b950 tx=0x7f6e840046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.117+0000 7f6e8fb5e640 1 -- 172.21.15.161:0/4273281003 shutdown_connections 2024-03-28T23:30:55.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.117+0000 7f6e8fb5e640 1 --2- 172.21.15.161:0/4273281003 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6e6c06b4d0 0x7f6e6c06d990 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.117+0000 7f6e8fb5e640 1 --2- 172.21.15.161:0/4273281003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e88150600 0x7f6e8806b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.117+0000 7f6e8fb5e640 1 --2- 172.21.15.161:0/4273281003 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e880aacb0 0x7f6e880c5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.117+0000 7f6e8fb5e640 1 --2- 172.21.15.161:0/4273281003 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e880a8fc0 0x7f6e881eb1c0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.117+0000 7f6e8fb5e640 1 -- 172.21.15.161:0/4273281003 >> 172.21.15.161:0/4273281003 conn(0x7f6e880c6670 msgr2=0x7f6e880c3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:55.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.118+0000 7f6e8fb5e640 1 -- 172.21.15.161:0/4273281003 shutdown_connections 2024-03-28T23:30:55.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.118+0000 7f6e8fb5e640 1 -- 172.21.15.161:0/4273281003 wait complete. 2024-03-28T23:30:55.129 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:30:55.130 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:30:55.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.232+0000 7ff1662fb640 1 Processor -- start 2024-03-28T23:30:55.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.232+0000 7ff1662fb640 1 -- start start 2024-03-28T23:30:55.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.233+0000 7ff1662fb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1601529d0 0x7ff16006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:55.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.233+0000 7ff1662fb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff1600c6750 0x7ff1600c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:55.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.233+0000 7ff1662fb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff160152db0 con 0x7ff160155c30 2024-03-28T23:30:55.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.233+0000 7ff1662fb640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff1600c7210 con 0x7ff1600c6750 2024-03-28T23:30:55.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.233+0000 7ff1662fb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff1600c6c50 con 0x7ff1601529d0 2024-03-28T23:30:55.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.233+0000 7ff15f7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff1600c6750 0x7ff1600c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:55.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.233+0000 7ff164898640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ff160155c30 0x7ff1600c54f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:46476/0 (socket says 172.21.15.161:46476) 2024-03-28T23:30:55.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.233+0000 7ff164898640 1 -- 172.21.15.161:0/2731271876 learned_addr learned my addr 172.21.15.161:0/2731271876 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:55.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.234+0000 7ff15ffff640 1 --2- 172.21.15.161:0/2731271876 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1601529d0 0x7ff16006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:55.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.234+0000 7ff15f7fe640 1 -- 172.21.15.161:0/2731271876 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1601529d0 msgr2=0x7ff16006bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:55.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.234+0000 7ff15f7fe640 1 --2- 172.21.15.161:0/2731271876 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1601529d0 0x7ff16006bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.234+0000 7ff15f7fe640 1 -- 172.21.15.161:0/2731271876 >> v1:172.21.15.161:6789/0 conn(0x7ff160155c30 legacy=0x7ff1600c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:55.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.234+0000 7ff15effd640 1 -- 172.21.15.161:0/2731271876 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3654235931 0 0) 0x7ff160152db0 con 0x7ff160155c30 2024-03-28T23:30:55.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.234+0000 7ff15f7fe640 1 -- 172.21.15.161:0/2731271876 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1600c6f30 con 0x7ff1600c6750 2024-03-28T23:30:55.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.234+0000 7ff15f7fe640 1 --2- 172.21.15.161:0/2731271876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1600c6750 0x7ff1600c88b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7ff1540097a0 tx=0x7ff15402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c6d5cc48e17b9f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:55.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.235+0000 7ff15effd640 1 -- 172.21.15.161:0/2731271876 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff154035020 con 0x7ff1600c6750 2024-03-28T23:30:55.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.235+0000 7ff15effd640 1 -- 172.21.15.161:0/2731271876 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff154031830 con 0x7ff1600c6750 2024-03-28T23:30:55.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.235+0000 7ff15effd640 1 -- 172.21.15.161:0/2731271876 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff154031c00 con 0x7ff1600c6750 2024-03-28T23:30:55.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.235+0000 7ff1662fb640 1 -- 172.21.15.161:0/2731271876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1600c6750 msgr2=0x7ff1600c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:55.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.235+0000 7ff1662fb640 1 --2- 172.21.15.161:0/2731271876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1600c6750 0x7ff1600c88b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7ff1540097a0 tx=0x7ff15402b8d0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.235+0000 7ff1662fb640 1 -- 172.21.15.161:0/2731271876 shutdown_connections 2024-03-28T23:30:55.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.235+0000 7ff1662fb640 1 --2- 172.21.15.161:0/2731271876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1600c6750 0x7ff1600c88b0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.235+0000 7ff1662fb640 1 --2- 172.21.15.161:0/2731271876 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1601529d0 0x7ff16006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.235+0000 7ff1662fb640 1 -- 172.21.15.161:0/2731271876 >> 172.21.15.161:0/2731271876 conn(0x7ff1600c7430 msgr2=0x7ff1600c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:55.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.235+0000 7ff1662fb640 1 -- 172.21.15.161:0/2731271876 shutdown_connections 2024-03-28T23:30:55.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.235+0000 7ff1662fb640 1 -- 172.21.15.161:0/2731271876 wait complete. 2024-03-28T23:30:55.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.236+0000 7ff1662fb640 1 Processor -- start 2024-03-28T23:30:55.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.236+0000 7ff1662fb640 1 -- start start 2024-03-28T23:30:55.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.237+0000 7ff1662fb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1600c6750 0x7ff1600abb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:55.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.237+0000 7ff1662fb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1601529d0 0x7ff160145a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:55.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.237+0000 7ff1662fb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff160155c30 0x7ff160145fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:55.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.237+0000 7ff1662fb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff1600aa160 con 0x7ff1601529d0 2024-03-28T23:30:55.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.237+0000 7ff1662fb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff160155710 con 0x7ff160155c30 2024-03-28T23:30:55.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.237+0000 7ff1662fb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff1600c5c00 con 0x7ff1600c6750 2024-03-28T23:30:55.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.237+0000 7ff15f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1601529d0 0x7ff160145a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:55.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.237+0000 7ff15f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1601529d0 0x7ff160145a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46830/0 (socket says 172.21.15.161:46830) 2024-03-28T23:30:55.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.237+0000 7ff15f7fe640 1 -- 172.21.15.161:0/1766026693 learned_addr learned my addr 172.21.15.161:0/1766026693 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:55.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.237+0000 7ff15ffff640 1 --2- 172.21.15.161:0/1766026693 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1600c6750 0x7ff1600abb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:55.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.237+0000 7ff15f7fe640 1 -- 172.21.15.161:0/1766026693 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1600c6750 msgr2=0x7ff1600abb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:55.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.237+0000 7ff164898640 1 --2- 172.21.15.161:0/1766026693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff160155c30 0x7ff160145fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:55.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.237+0000 7ff15f7fe640 1 --2- 172.21.15.161:0/1766026693 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1600c6750 0x7ff1600abb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.237+0000 7ff15f7fe640 1 -- 172.21.15.161:0/1766026693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff160155c30 msgr2=0x7ff160145fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:55.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.237+0000 7ff15f7fe640 1 --2- 172.21.15.161:0/1766026693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff160155c30 0x7ff160145fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.237+0000 7ff15f7fe640 1 -- 172.21.15.161:0/1766026693 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff16014d630 con 0x7ff1601529d0 2024-03-28T23:30:55.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.237+0000 7ff164898640 1 --2- 172.21.15.161:0/1766026693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff160155c30 0x7ff160145fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:30:55.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.238+0000 7ff15f7fe640 1 --2- 172.21.15.161:0/1766026693 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1601529d0 0x7ff160145a70 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff154034040 tx=0x7ff1540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:55.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.238+0000 7ff15cff9640 1 -- 172.21.15.161:0/1766026693 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff154040070 con 0x7ff1601529d0 2024-03-28T23:30:55.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.238+0000 7ff15cff9640 1 -- 172.21.15.161:0/1766026693 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1540399a0 con 0x7ff1601529d0 2024-03-28T23:30:55.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.238+0000 7ff1662fb640 1 -- 172.21.15.161:0/1766026693 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff16014d8d0 con 0x7ff1601529d0 2024-03-28T23:30:55.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.238+0000 7ff15cff9640 1 -- 172.21.15.161:0/1766026693 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff154035040 con 0x7ff1601529d0 2024-03-28T23:30:55.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.238+0000 7ff1662fb640 1 -- 172.21.15.161:0/1766026693 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1600ac0a0 con 0x7ff1601529d0 2024-03-28T23:30:55.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.239+0000 7ff1662fb640 1 -- 172.21.15.161:0/1766026693 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff16004f4b0 con 0x7ff1601529d0 2024-03-28T23:30:55.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.239+0000 7ff15cff9640 1 -- 172.21.15.161:0/1766026693 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff154038020 con 0x7ff1601529d0 2024-03-28T23:30:55.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.240+0000 7ff15cff9640 1 --2- 172.21.15.161:0/1766026693 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff14c06b500 0x7ff14c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:55.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.240+0000 7ff15ffff640 1 --2- 172.21.15.161:0/1766026693 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff14c06b500 0x7ff14c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:55.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.240+0000 7ff15cff9640 1 -- 172.21.15.161:0/1766026693 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7ff1540ab980 con 0x7ff1601529d0 2024-03-28T23:30:55.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.240+0000 7ff15ffff640 1 --2- 172.21.15.161:0/1766026693 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff14c06b500 0x7ff14c06d9c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff150006fd0 tx=0x7ff150008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:55.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.244+0000 7ff15cff9640 1 -- 172.21.15.161:0/1766026693 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff15407a9d0 con 0x7ff1601529d0 2024-03-28T23:30:55.362 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:30:55.364 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:30:55.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.420+0000 7ff1662fb640 1 -- 172.21.15.161:0/1766026693 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff1600ca650 con 0x7ff1601529d0 2024-03-28T23:30:55.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.421+0000 7ff15cff9640 1 -- 172.21.15.161:0/1766026693 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) ==== 93+0+10 (secure 0 0 0) 0x7ff15407a2d0 con 0x7ff1601529d0 2024-03-28T23:30:55.422 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:30:55.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.423+0000 7ff1662fb640 1 -- 172.21.15.161:0/1766026693 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff14c06b500 msgr2=0x7ff14c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:55.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.423+0000 7ff1662fb640 1 --2- 172.21.15.161:0/1766026693 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff14c06b500 0x7ff14c06d9c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff150006fd0 tx=0x7ff150008040 comp rx=0 tx=0).stop 2024-03-28T23:30:55.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.423+0000 7ff1662fb640 1 -- 172.21.15.161:0/1766026693 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1601529d0 msgr2=0x7ff160145a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:55.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.423+0000 7ff1662fb640 1 --2- 172.21.15.161:0/1766026693 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1601529d0 0x7ff160145a70 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff154034040 tx=0x7ff1540046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.423+0000 7ff1662fb640 1 -- 172.21.15.161:0/1766026693 shutdown_connections 2024-03-28T23:30:55.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.423+0000 7ff1662fb640 1 --2- 172.21.15.161:0/1766026693 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff14c06b500 0x7ff14c06d9c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.423+0000 7ff1662fb640 1 --2- 172.21.15.161:0/1766026693 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1601529d0 0x7ff160145a70 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.423+0000 7ff1662fb640 1 --2- 172.21.15.161:0/1766026693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff160155c30 0x7ff160145fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.423+0000 7ff1662fb640 1 --2- 172.21.15.161:0/1766026693 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1600c6750 0x7ff1600abb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.423+0000 7ff1662fb640 1 -- 172.21.15.161:0/1766026693 >> 172.21.15.161:0/1766026693 conn(0x7ff1600c7430 msgr2=0x7ff160151b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:55.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.423+0000 7ff1662fb640 1 -- 172.21.15.161:0/1766026693 shutdown_connections 2024-03-28T23:30:55.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.423+0000 7ff1662fb640 1 -- 172.21.15.161:0/1766026693 wait complete. 2024-03-28T23:30:55.434 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T23:30:55.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.575+0000 7fca45615640 1 Processor -- start 2024-03-28T23:30:55.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.576+0000 7fca45615640 1 -- start start 2024-03-28T23:30:55.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.576+0000 7fca45615640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca4014f260 0x7fca401592e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:55.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.576+0000 7fca45615640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca400cb4e0 con 0x7fca400c8970 2024-03-28T23:30:55.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.576+0000 7fca45615640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca400cb7c0 con 0x7fca400caf60 2024-03-28T23:30:55.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.576+0000 7fca45615640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fca40158e50 con 0x7fca4014f260 2024-03-28T23:30:55.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.576+0000 7fca3f7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fca400c8970 0x7fca400c53f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:46490/0 (socket says 172.21.15.161:46490) 2024-03-28T23:30:55.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.576+0000 7fca3f7fe640 1 -- 172.21.15.161:0/2604264946 learned_addr learned my addr 172.21.15.161:0/2604264946 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:55.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca3e7fc640 1 -- 172.21.15.161:0/2604264946 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1978772530 0 0) 0x7fca400cb4e0 con 0x7fca400c8970 2024-03-28T23:30:55.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca3e7fc640 1 -- 172.21.15.161:0/2604264946 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca2c003520 con 0x7fca400c8970 2024-03-28T23:30:55.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca3effd640 1 --2- 172.21.15.161:0/2604264946 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca4014f260 0x7fca401592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:55.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca3e7fc640 1 -- 172.21.15.161:0/2604264946 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3144941369 0 0) 0x7fca2c003520 con 0x7fca400c8970 2024-03-28T23:30:55.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca3e7fc640 1 -- 172.21.15.161:0/2604264946 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca4014f260 msgr2=0x7fca401592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:55.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca3e7fc640 1 --2- 172.21.15.161:0/2604264946 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca4014f260 0x7fca401592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca3e7fc640 1 -- 172.21.15.161:0/2604264946 >> v1:172.21.15.170:6789/0 conn(0x7fca400caf60 legacy=0x7fca40155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:55.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca3e7fc640 1 -- 172.21.15.161:0/2604264946 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca40158800 con 0x7fca400c8970 2024-03-28T23:30:55.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca3e7fc640 1 -- 172.21.15.161:0/2604264946 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fca380019f0 con 0x7fca400c8970 2024-03-28T23:30:55.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca3e7fc640 1 -- 172.21.15.161:0/2604264946 >> v1:172.21.15.161:6789/0 conn(0x7fca400c8970 legacy=0x7fca400c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:55.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca3e7fc640 1 --2- 172.21.15.161:0/2604264946 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca4014f260 0x7fca2c003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:55.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca3e7fc640 1 --2- 172.21.15.161:0/2604264946 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca2c003fd0 0x7fca2c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:55.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca3e7fc640 1 --2- 172.21.15.161:0/2604264946 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca2c004d40 0x7fca2c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:55.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca3e7fc640 1 -- 172.21.15.161:0/2604264946 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fca40158e50 con 0x7fca2c003fd0 2024-03-28T23:30:55.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca3e7fc640 1 -- 172.21.15.161:0/2604264946 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fca2c00bbb0 con 0x7fca2c004d40 2024-03-28T23:30:55.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca3e7fc640 1 -- 172.21.15.161:0/2604264946 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fca2c00bdb0 con 0x7fca4014f260 2024-03-28T23:30:55.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca37fff640 1 --2- 172.21.15.161:0/2604264946 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca2c003fd0 0x7fca2c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:55.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca3effd640 1 --2- 172.21.15.161:0/2604264946 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca4014f260 0x7fca2c003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:55.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca3f7fe640 1 --2- 172.21.15.161:0/2604264946 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca2c004d40 0x7fca2c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:55.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca37fff640 1 -- 172.21.15.161:0/2604264946 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca4014f260 msgr2=0x7fca2c003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:55.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca37fff640 1 --2- 172.21.15.161:0/2604264946 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca4014f260 0x7fca2c003a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca37fff640 1 -- 172.21.15.161:0/2604264946 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca2c004d40 msgr2=0x7fca2c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:55.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca37fff640 1 --2- 172.21.15.161:0/2604264946 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca2c004d40 0x7fca2c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca37fff640 1 -- 172.21.15.161:0/2604264946 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca2c003520 con 0x7fca2c003fd0 2024-03-28T23:30:55.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.577+0000 7fca3f7fe640 1 --2- 172.21.15.161:0/2604264946 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca2c004d40 0x7fca2c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:30:55.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.578+0000 7fca37fff640 1 --2- 172.21.15.161:0/2604264946 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca2c003fd0 0x7fca2c004800 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fca28001750 tx=0x7fca2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f49253c793527eb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:55.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.578+0000 7fca3e7fc640 1 -- 172.21.15.161:0/2604264946 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28001370 con 0x7fca2c003fd0 2024-03-28T23:30:55.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.578+0000 7fca3e7fc640 1 -- 172.21.15.161:0/2604264946 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca280052d0 con 0x7fca2c003fd0 2024-03-28T23:30:55.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.578+0000 7fca3e7fc640 1 -- 172.21.15.161:0/2604264946 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28003830 con 0x7fca2c003fd0 2024-03-28T23:30:55.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.579+0000 7fca45615640 1 -- 172.21.15.161:0/2604264946 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca2c003fd0 msgr2=0x7fca2c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:55.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.579+0000 7fca45615640 1 --2- 172.21.15.161:0/2604264946 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca2c003fd0 0x7fca2c004800 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fca28001750 tx=0x7fca2c005830 comp rx=0 tx=0).stop 2024-03-28T23:30:55.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.579+0000 7fca3f7fe640 1 -- 172.21.15.161:0/2604264946 reap_dead start 2024-03-28T23:30:55.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.579+0000 7fca45615640 1 -- 172.21.15.161:0/2604264946 shutdown_connections 2024-03-28T23:30:55.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.579+0000 7fca45615640 1 -- 172.21.15.161:0/2604264946 >> 172.21.15.161:0/2604264946 conn(0x7fca40072ca0 msgr2=0x7fca400730b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:55.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.579+0000 7fca45615640 1 -- 172.21.15.161:0/2604264946 shutdown_connections 2024-03-28T23:30:55.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.579+0000 7fca45615640 1 -- 172.21.15.161:0/2604264946 wait complete. 2024-03-28T23:30:55.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.580+0000 7fca45615640 1 Processor -- start 2024-03-28T23:30:55.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.580+0000 7fca45615640 1 -- start start 2024-03-28T23:30:55.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.581+0000 7fca45615640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca2c003fd0 0x7fca400c3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:55.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.581+0000 7fca45615640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca4014e220 0x7fca400ab5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:55.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.581+0000 7fca45615640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca400abb10 0x7fca4014cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:55.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.581+0000 7fca45615640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fca400c5b00 con 0x7fca4014e220 2024-03-28T23:30:55.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.581+0000 7fca45615640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fca4014f640 con 0x7fca400abb10 2024-03-28T23:30:55.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.581+0000 7fca45615640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fca400c5c80 con 0x7fca2c003fd0 2024-03-28T23:30:55.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.581+0000 7fca37fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca4014e220 0x7fca400ab5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:55.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.581+0000 7fca37fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca4014e220 0x7fca400ab5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46838/0 (socket says 172.21.15.161:46838) 2024-03-28T23:30:55.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.581+0000 7fca37fff640 1 -- 172.21.15.161:0/53872420 learned_addr learned my addr 172.21.15.161:0/53872420 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:55.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.581+0000 7fca3effd640 1 --2- 172.21.15.161:0/53872420 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca2c003fd0 0x7fca400c3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:55.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.581+0000 7fca3f7fe640 1 --2- 172.21.15.161:0/53872420 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca400abb10 0x7fca4014cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:55.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.581+0000 7fca37fff640 1 -- 172.21.15.161:0/53872420 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca2c003fd0 msgr2=0x7fca400c3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:55.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.581+0000 7fca37fff640 1 --2- 172.21.15.161:0/53872420 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca2c003fd0 0x7fca400c3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.581+0000 7fca37fff640 1 -- 172.21.15.161:0/53872420 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca400abb10 msgr2=0x7fca4014cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:55.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.581+0000 7fca37fff640 1 --2- 172.21.15.161:0/53872420 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca400abb10 0x7fca4014cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.581+0000 7fca37fff640 1 -- 172.21.15.161:0/53872420 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca400c42d0 con 0x7fca4014e220 2024-03-28T23:30:55.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.581+0000 7fca37fff640 1 --2- 172.21.15.161:0/53872420 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca4014e220 0x7fca400ab5d0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fca28008fd0 tx=0x7fca28001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:55.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.581+0000 7fca377fe640 1 -- 172.21.15.161:0/53872420 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28041070 con 0x7fca4014e220 2024-03-28T23:30:55.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.581+0000 7fca377fe640 1 -- 172.21.15.161:0/53872420 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca28005480 con 0x7fca4014e220 2024-03-28T23:30:55.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.581+0000 7fca45615640 1 -- 172.21.15.161:0/53872420 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca401ebe00 con 0x7fca4014e220 2024-03-28T23:30:55.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.581+0000 7fca377fe640 1 -- 172.21.15.161:0/53872420 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28001370 con 0x7fca4014e220 2024-03-28T23:30:55.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.581+0000 7fca45615640 1 -- 172.21.15.161:0/53872420 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca40151a50 con 0x7fca4014e220 2024-03-28T23:30:55.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.583+0000 7fca45615640 1 -- 172.21.15.161:0/53872420 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca40067a50 con 0x7fca4014e220 2024-03-28T23:30:55.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.583+0000 7fca377fe640 1 -- 172.21.15.161:0/53872420 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fca28044070 con 0x7fca4014e220 2024-03-28T23:30:55.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.583+0000 7fca377fe640 1 --2- 172.21.15.161:0/53872420 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fca2006b500 0x7fca2006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:55.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.583+0000 7fca377fe640 1 -- 172.21.15.161:0/53872420 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7fca280ab9a0 con 0x7fca4014e220 2024-03-28T23:30:55.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.583+0000 7fca3effd640 1 --2- 172.21.15.161:0/53872420 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fca2006b500 0x7fca2006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:55.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.584+0000 7fca3effd640 1 --2- 172.21.15.161:0/53872420 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fca2006b500 0x7fca2006d9c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fca30009bc0 tx=0x7fca30009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:55.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.588+0000 7fca377fe640 1 -- 172.21.15.161:0/53872420 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fca40067a50 con 0x7fca4014e220 2024-03-28T23:30:55.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.763+0000 7fca45615640 1 -- 172.21.15.161:0/53872420 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fca40068f90 con 0x7fca4014e220 2024-03-28T23:30:55.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.764+0000 7fca377fe640 1 -- 172.21.15.161:0/53872420 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v34) ==== 84+0+65 (secure 0 0 0) 0x7fca40068f90 con 0x7fca4014e220 2024-03-28T23:30:55.764 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:55.765 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2024-03-28T23:30:55.765 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:30:55.765 INFO:teuthology.orchestra.run.smithi161.stdout: "poolnum": 1, 2024-03-28T23:30:55.765 INFO:teuthology.orchestra.run.smithi161.stdout: "poolname": ".mgr" 2024-03-28T23:30:55.765 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:30:55.765 INFO:teuthology.orchestra.run.smithi161.stdout:] 2024-03-28T23:30:55.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.765+0000 7fca45615640 1 -- 172.21.15.161:0/53872420 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fca2006b500 msgr2=0x7fca2006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:55.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.765+0000 7fca45615640 1 --2- 172.21.15.161:0/53872420 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fca2006b500 0x7fca2006d9c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fca30009bc0 tx=0x7fca30009190 comp rx=0 tx=0).stop 2024-03-28T23:30:55.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.766+0000 7fca45615640 1 -- 172.21.15.161:0/53872420 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca4014e220 msgr2=0x7fca400ab5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:55.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.766+0000 7fca45615640 1 --2- 172.21.15.161:0/53872420 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca4014e220 0x7fca400ab5d0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fca28008fd0 tx=0x7fca28001120 comp rx=0 tx=0).stop 2024-03-28T23:30:55.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.766+0000 7fca45615640 1 -- 172.21.15.161:0/53872420 shutdown_connections 2024-03-28T23:30:55.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.766+0000 7fca45615640 1 --2- 172.21.15.161:0/53872420 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fca2006b500 0x7fca2006d9c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.766+0000 7fca45615640 1 --2- 172.21.15.161:0/53872420 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca4014e220 0x7fca400ab5d0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.766+0000 7fca45615640 1 --2- 172.21.15.161:0/53872420 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca400abb10 0x7fca4014cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.766+0000 7fca45615640 1 --2- 172.21.15.161:0/53872420 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca2c003fd0 0x7fca400c3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.766+0000 7fca45615640 1 -- 172.21.15.161:0/53872420 >> 172.21.15.161:0/53872420 conn(0x7fca40072ca0 msgr2=0x7fca400ca140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:55.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.766+0000 7fca45615640 1 -- 172.21.15.161:0/53872420 shutdown_connections 2024-03-28T23:30:55.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.766+0000 7fca45615640 1 -- 172.21.15.161:0/53872420 wait complete. 2024-03-28T23:30:55.778 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-28T23:30:55.779 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-28T23:30:55.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.883+0000 7fb7f12b5640 1 Processor -- start 2024-03-28T23:30:55.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.883+0000 7fb7f12b5640 1 -- start start 2024-03-28T23:30:55.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.884+0000 7fb7f12b5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb7ec152720 0x7fb7ec0c30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:55.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.884+0000 7fb7f12b5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7ec0c3630 0x7fb7ec0bfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:55.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.884+0000 7fb7f12b5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb7ec0c4c90 0x7fb7ec0c0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:55.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.884+0000 7fb7f12b5640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb7ec155e10 con 0x7fb7ec152720 2024-03-28T23:30:55.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.884+0000 7fb7f12b5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fb7ec152b00 con 0x7fb7ec0c4c90 2024-03-28T23:30:55.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.884+0000 7fb7f12b5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb7ec155200 con 0x7fb7ec0c3630 2024-03-28T23:30:55.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.884+0000 7fb7eaffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb7ec152720 0x7fb7ec0c30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:55.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.884+0000 7fb7eaffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb7ec152720 0x7fb7ec0c30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46844/0 (socket says 172.21.15.161:46844) 2024-03-28T23:30:55.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.884+0000 7fb7eaffd640 1 -- 172.21.15.161:0/75496468 learned_addr learned my addr 172.21.15.161:0/75496468 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:55.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.884+0000 7fb7eb7fe640 1 --2- 172.21.15.161:0/75496468 >> v2:172.21.15.170:3300/0 conn(0x7fb7ec0c4c90 0x7fb7ec0c0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:55.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.884+0000 7fb7ea7fc640 1 --2- 172.21.15.161:0/75496468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7ec0c3630 0x7fb7ec0bfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:55.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.884+0000 7fb7eaffd640 1 -- 172.21.15.161:0/75496468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7ec0c3630 msgr2=0x7fb7ec0bfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:55.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.884+0000 7fb7eaffd640 1 --2- 172.21.15.161:0/75496468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7ec0c3630 0x7fb7ec0bfbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.884+0000 7fb7eaffd640 1 -- 172.21.15.161:0/75496468 >> v2:172.21.15.170:3300/0 conn(0x7fb7ec0c4c90 msgr2=0x7fb7ec0c0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:55.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.884+0000 7fb7eaffd640 1 --2- 172.21.15.161:0/75496468 >> v2:172.21.15.170:3300/0 conn(0x7fb7ec0c4c90 0x7fb7ec0c0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.884+0000 7fb7eaffd640 1 -- 172.21.15.161:0/75496468 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7ec1554e0 con 0x7fb7ec152720 2024-03-28T23:30:55.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.885+0000 7fb7eaffd640 1 --2- 172.21.15.161:0/75496468 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7ec152720 0x7fb7ec0c30f0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec0c7790 tx=0x7fb7dc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89056128dc4868f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:55.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.885+0000 7fb7e9ffb640 1 -- 172.21.15.161:0/75496468 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc02e070 con 0x7fb7ec152720 2024-03-28T23:30:55.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.885+0000 7fb7e9ffb640 1 -- 172.21.15.161:0/75496468 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7dc037070 con 0x7fb7ec152720 2024-03-28T23:30:55.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.885+0000 7fb7e9ffb640 1 -- 172.21.15.161:0/75496468 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc031a00 con 0x7fb7ec152720 2024-03-28T23:30:55.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.886+0000 7fb7f12b5640 1 -- 172.21.15.161:0/75496468 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7ec152720 msgr2=0x7fb7ec0c30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:55.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.886+0000 7fb7f12b5640 1 --2- 172.21.15.161:0/75496468 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7ec152720 0x7fb7ec0c30f0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec0c7790 tx=0x7fb7dc02b880 comp rx=0 tx=0).stop 2024-03-28T23:30:55.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.886+0000 7fb7f12b5640 1 -- 172.21.15.161:0/75496468 shutdown_connections 2024-03-28T23:30:55.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.886+0000 7fb7f12b5640 1 --2- 172.21.15.161:0/75496468 >> v2:172.21.15.170:3300/0 conn(0x7fb7ec0c4c90 0x7fb7ec0c0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.886+0000 7fb7f12b5640 1 --2- 172.21.15.161:0/75496468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7ec0c3630 0x7fb7ec0bfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.886+0000 7fb7f12b5640 1 --2- 172.21.15.161:0/75496468 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7ec152720 0x7fb7ec0c30f0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.886+0000 7fb7f12b5640 1 -- 172.21.15.161:0/75496468 >> 172.21.15.161:0/75496468 conn(0x7fb7ec06b150 msgr2=0x7fb7ec06b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:55.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.886+0000 7fb7f12b5640 1 -- 172.21.15.161:0/75496468 shutdown_connections 2024-03-28T23:30:55.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.886+0000 7fb7f12b5640 1 -- 172.21.15.161:0/75496468 wait complete. 2024-03-28T23:30:55.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.887+0000 7fb7f12b5640 1 Processor -- start 2024-03-28T23:30:55.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.887+0000 7fb7f12b5640 1 -- start start 2024-03-28T23:30:55.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.888+0000 7fb7f12b5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7ec0c3630 0x7fb7ec071a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:55.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.888+0000 7fb7f12b5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7ec0c4c90 0x7fb7ec1ee5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:55.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.888+0000 7fb7f12b5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7ec152720 0x7fb7ec147000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:55.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.888+0000 7fb7f12b5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ec0c5d80 con 0x7fb7ec0c3630 2024-03-28T23:30:55.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.888+0000 7fb7f12b5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ec155aa0 con 0x7fb7ec0c4c90 2024-03-28T23:30:55.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.888+0000 7fb7f12b5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb7ec0c5f00 con 0x7fb7ec152720 2024-03-28T23:30:55.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.888+0000 7fb7eaffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7ec0c3630 0x7fb7ec071a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:55.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.888+0000 7fb7eaffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7ec0c3630 0x7fb7ec071a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46858/0 (socket says 172.21.15.161:46858) 2024-03-28T23:30:55.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.888+0000 7fb7eaffd640 1 -- 172.21.15.161:0/439928837 learned_addr learned my addr 172.21.15.161:0/439928837 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:55.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.888+0000 7fb7ea7fc640 1 --2- 172.21.15.161:0/439928837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7ec0c4c90 0x7fb7ec1ee5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:55.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.888+0000 7fb7eb7fe640 1 --2- 172.21.15.161:0/439928837 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7ec152720 0x7fb7ec147000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:55.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.888+0000 7fb7eaffd640 1 -- 172.21.15.161:0/439928837 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7ec152720 msgr2=0x7fb7ec147000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:55.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.888+0000 7fb7eaffd640 1 --2- 172.21.15.161:0/439928837 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7ec152720 0x7fb7ec147000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.888+0000 7fb7eaffd640 1 -- 172.21.15.161:0/439928837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7ec0c4c90 msgr2=0x7fb7ec1ee5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:55.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.888+0000 7fb7eaffd640 1 --2- 172.21.15.161:0/439928837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7ec0c4c90 0x7fb7ec1ee5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:55.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.888+0000 7fb7eaffd640 1 -- 172.21.15.161:0/439928837 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7ec070790 con 0x7fb7ec0c3630 2024-03-28T23:30:55.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.888+0000 7fb7eaffd640 1 --2- 172.21.15.161:0/439928837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7ec0c3630 0x7fb7ec071a40 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc000c00 tx=0x7fb7dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:55.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.888+0000 7fb7d37fe640 1 -- 172.21.15.161:0/439928837 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc032880 con 0x7fb7ec0c3630 2024-03-28T23:30:55.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.888+0000 7fb7d37fe640 1 -- 172.21.15.161:0/439928837 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7dc032a20 con 0x7fb7ec0c3630 2024-03-28T23:30:55.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.888+0000 7fb7d37fe640 1 -- 172.21.15.161:0/439928837 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc032e10 con 0x7fb7ec0c3630 2024-03-28T23:30:55.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.888+0000 7fb7f12b5640 1 -- 172.21.15.161:0/439928837 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7ec1ea780 con 0x7fb7ec0c3630 2024-03-28T23:30:55.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.888+0000 7fb7f12b5640 1 -- 172.21.15.161:0/439928837 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7ec070d30 con 0x7fb7ec0c3630 2024-03-28T23:30:55.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.889+0000 7fb7f12b5640 1 -- 172.21.15.161:0/439928837 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7ec068f90 con 0x7fb7ec0c3630 2024-03-28T23:30:55.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.890+0000 7fb7d37fe640 1 -- 172.21.15.161:0/439928837 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb7dc0453f0 con 0x7fb7ec0c3630 2024-03-28T23:30:55.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.890+0000 7fb7d37fe640 1 --2- 172.21.15.161:0/439928837 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb7d806b4d0 0x7fb7d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:55.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.890+0000 7fb7ea7fc640 1 --2- 172.21.15.161:0/439928837 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb7d806b4d0 0x7fb7d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:55.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.890+0000 7fb7d37fe640 1 -- 172.21.15.161:0/439928837 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7fb7dc0abfe0 con 0x7fb7ec0c3630 2024-03-28T23:30:55.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.891+0000 7fb7ea7fc640 1 --2- 172.21.15.161:0/439928837 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb7d806b4d0 0x7fb7d806d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec151ad0 tx=0x7fb7e0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:55.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:55.895+0000 7fb7d37fe640 1 -- 172.21.15.161:0/439928837 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb7dc040070 con 0x7fb7ec0c3630 2024-03-28T23:30:56.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.067+0000 7fb7f12b5640 1 -- 172.21.15.161:0/439928837 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fb7ec06b920 con 0x7fb7ec0c3630 2024-03-28T23:30:56.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.249+0000 7fb7d37fe640 1 -- 172.21.15.161:0/439928837 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v35) ==== 138+0+0 (secure 0 0 0) 0x7fb7dc07ab80 con 0x7fb7ec0c3630 2024-03-28T23:30:56.250 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_metadata' created 2024-03-28T23:30:56.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.252+0000 7fb7f12b5640 1 -- 172.21.15.161:0/439928837 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb7d806b4d0 msgr2=0x7fb7d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:56.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.252+0000 7fb7f12b5640 1 --2- 172.21.15.161:0/439928837 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb7d806b4d0 0x7fb7d806d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec151ad0 tx=0x7fb7e0008040 comp rx=0 tx=0).stop 2024-03-28T23:30:56.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.252+0000 7fb7f12b5640 1 -- 172.21.15.161:0/439928837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7ec0c3630 msgr2=0x7fb7ec071a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:56.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.252+0000 7fb7f12b5640 1 --2- 172.21.15.161:0/439928837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7ec0c3630 0x7fb7ec071a40 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc000c00 tx=0x7fb7dc0046c0 comp rx=0 tx=0).stop 2024-03-28T23:30:56.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.253+0000 7fb7f12b5640 1 -- 172.21.15.161:0/439928837 shutdown_connections 2024-03-28T23:30:56.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.253+0000 7fb7f12b5640 1 --2- 172.21.15.161:0/439928837 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb7d806b4d0 0x7fb7d806d990 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:56.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.253+0000 7fb7f12b5640 1 --2- 172.21.15.161:0/439928837 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7ec152720 0x7fb7ec147000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:56.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.253+0000 7fb7f12b5640 1 --2- 172.21.15.161:0/439928837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7ec0c4c90 0x7fb7ec1ee5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:56.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.253+0000 7fb7f12b5640 1 --2- 172.21.15.161:0/439928837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7ec0c3630 0x7fb7ec071a40 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:56.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.253+0000 7fb7f12b5640 1 -- 172.21.15.161:0/439928837 >> 172.21.15.161:0/439928837 conn(0x7fb7ec06b150 msgr2=0x7fb7ec0bfbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:56.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.253+0000 7fb7f12b5640 1 -- 172.21.15.161:0/439928837 shutdown_connections 2024-03-28T23:30:56.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.253+0000 7fb7f12b5640 1 -- 172.21.15.161:0/439928837 wait complete. 2024-03-28T23:30:56.267 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T23:30:56.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.374+0000 7f5af33c7640 1 Processor -- start 2024-03-28T23:30:56.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.374+0000 7f5af33c7640 1 -- start start 2024-03-28T23:30:56.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.375+0000 7f5af33c7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5aec14f5c0 0x7f5aec0c2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:56.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.375+0000 7f5af33c7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5aec152bd0 0x7f5aec15c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:56.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.375+0000 7f5af33c7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5aec0c2f10 con 0x7f5aec06cdd0 2024-03-28T23:30:56.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.375+0000 7f5af33c7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f5aec0c74d0 con 0x7f5aec14f5c0 2024-03-28T23:30:56.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.375+0000 7f5af33c7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5aec0c6fa0 con 0x7f5aec152bd0 2024-03-28T23:30:56.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.375+0000 7f5af1964640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5aec152bd0 0x7f5aec15c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:56.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.375+0000 7f5af1163640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5aec14f5c0 0x7f5aec0c2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:56.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.375+0000 7f5af1964640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5aec152bd0 0x7f5aec15c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:45590/0 (socket says 172.21.15.161:45590) 2024-03-28T23:30:56.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.375+0000 7f5af0962640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f5aec06cdd0 0x7f5aec1524c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:39074/0 (socket says 172.21.15.161:39074) 2024-03-28T23:30:56.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.375+0000 7f5af1163640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5aec14f5c0 0x7f5aec0c2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:48582/0 (socket says 172.21.15.161:48582) 2024-03-28T23:30:56.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.375+0000 7f5af1964640 1 -- 172.21.15.161:0/1458411003 learned_addr learned my addr 172.21.15.161:0/1458411003 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:56.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.375+0000 7f5af1163640 1 -- 172.21.15.161:0/1458411003 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5aec152bd0 msgr2=0x7f5aec15c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:56.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.375+0000 7f5af1163640 1 --2- 172.21.15.161:0/1458411003 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5aec152bd0 0x7f5aec15c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:56.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.375+0000 7f5af1163640 1 -- 172.21.15.161:0/1458411003 >> v1:172.21.15.170:6789/0 conn(0x7f5aec06cdd0 legacy=0x7f5aec1524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:56.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.375+0000 7f5af1163640 1 -- 172.21.15.161:0/1458411003 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aec14f9a0 con 0x7f5aec14f5c0 2024-03-28T23:30:56.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.375+0000 7f5af1964640 1 --2- 172.21.15.161:0/1458411003 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5aec152bd0 0x7f5aec15c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:56.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.376+0000 7f5af1163640 1 --2- 172.21.15.161:0/1458411003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5aec14f5c0 0x7f5aec0c2150 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5ae80097a0 tx=0x7f5ae802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ed37d00621a2763 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:56.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.376+0000 7f5ae3fff640 1 -- 172.21.15.161:0/1458411003 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8035020 con 0x7f5aec14f5c0 2024-03-28T23:30:56.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.376+0000 7f5ae3fff640 1 -- 172.21.15.161:0/1458411003 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae8031830 con 0x7f5aec14f5c0 2024-03-28T23:30:56.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.376+0000 7f5ae3fff640 1 -- 172.21.15.161:0/1458411003 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8031c00 con 0x7f5aec14f5c0 2024-03-28T23:30:56.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.377+0000 7f5af33c7640 1 -- 172.21.15.161:0/1458411003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5aec14f5c0 msgr2=0x7f5aec0c2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:56.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.377+0000 7f5af33c7640 1 --2- 172.21.15.161:0/1458411003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5aec14f5c0 0x7f5aec0c2150 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5ae80097a0 tx=0x7f5ae802b8d0 comp rx=0 tx=0).stop 2024-03-28T23:30:56.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.377+0000 7f5af33c7640 1 -- 172.21.15.161:0/1458411003 shutdown_connections 2024-03-28T23:30:56.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.377+0000 7f5af33c7640 1 --2- 172.21.15.161:0/1458411003 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5aec152bd0 0x7f5aec15c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:56.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.377+0000 7f5af33c7640 1 --2- 172.21.15.161:0/1458411003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5aec14f5c0 0x7f5aec0c2150 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:56.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.377+0000 7f5af33c7640 1 -- 172.21.15.161:0/1458411003 >> 172.21.15.161:0/1458411003 conn(0x7f5aec0a83f0 msgr2=0x7f5aec0a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:56.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.377+0000 7f5af33c7640 1 -- 172.21.15.161:0/1458411003 shutdown_connections 2024-03-28T23:30:56.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.377+0000 7f5af33c7640 1 -- 172.21.15.161:0/1458411003 wait complete. 2024-03-28T23:30:56.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.378+0000 7f5af33c7640 1 Processor -- start 2024-03-28T23:30:56.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.378+0000 7f5af33c7640 1 -- start start 2024-03-28T23:30:56.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.379+0000 7f5af33c7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5aec06cdd0 0x7f5aec150e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:56.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.379+0000 7f5af33c7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aec14f5c0 0x7f5aec141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:56.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.379+0000 7f5af33c7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5aec152bd0 0x7f5aec141d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:56.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.379+0000 7f5af33c7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5aec06c140 con 0x7f5aec14f5c0 2024-03-28T23:30:56.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.379+0000 7f5af33c7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5aec152030 con 0x7f5aec152bd0 2024-03-28T23:30:56.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.379+0000 7f5af33c7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5aec06c3e0 con 0x7f5aec06cdd0 2024-03-28T23:30:56.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.379+0000 7f5af0962640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aec14f5c0 0x7f5aec141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:56.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.379+0000 7f5af1964640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5aec152bd0 0x7f5aec141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:56.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.379+0000 7f5af1163640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5aec06cdd0 0x7f5aec150e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:56.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.379+0000 7f5af1964640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5aec152bd0 0x7f5aec141d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:48592/0 (socket says 172.21.15.161:48592) 2024-03-28T23:30:56.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.379+0000 7f5af1163640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5aec06cdd0 0x7f5aec150e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:45606/0 (socket says 172.21.15.161:45606) 2024-03-28T23:30:56.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.379+0000 7f5af1964640 1 -- 172.21.15.161:0/1031901842 learned_addr learned my addr 172.21.15.161:0/1031901842 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:56.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.379+0000 7f5af1964640 1 -- 172.21.15.161:0/1031901842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5aec06cdd0 msgr2=0x7f5aec150e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:56.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.379+0000 7f5af1964640 1 --2- 172.21.15.161:0/1031901842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5aec06cdd0 0x7f5aec150e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:56.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.379+0000 7f5af1964640 1 -- 172.21.15.161:0/1031901842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aec14f5c0 msgr2=0x7f5aec141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:56.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.379+0000 7f5af1964640 1 --2- 172.21.15.161:0/1031901842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aec14f5c0 0x7f5aec141850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:56.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.379+0000 7f5af1964640 1 -- 172.21.15.161:0/1031901842 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aec0c4230 con 0x7f5aec152bd0 2024-03-28T23:30:56.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.379+0000 7f5af1163640 1 --2- 172.21.15.161:0/1031901842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5aec06cdd0 0x7f5aec150e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:56.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.379+0000 7f5af1964640 1 --2- 172.21.15.161:0/1031901842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5aec152bd0 0x7f5aec141d90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5ae400efc0 tx=0x7f5ae400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:56.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.379+0000 7f5ae1ffb640 1 -- 172.21.15.161:0/1031901842 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae400cb20 con 0x7f5aec152bd0 2024-03-28T23:30:56.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.379+0000 7f5ae1ffb640 1 -- 172.21.15.161:0/1031901842 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae400ccc0 con 0x7f5aec152bd0 2024-03-28T23:30:56.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.379+0000 7f5af33c7640 1 -- 172.21.15.161:0/1031901842 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aec0c4510 con 0x7f5aec152bd0 2024-03-28T23:30:56.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.379+0000 7f5ae1ffb640 1 -- 172.21.15.161:0/1031901842 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae4003bb0 con 0x7f5aec152bd0 2024-03-28T23:30:56.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.380+0000 7f5af33c7640 1 -- 172.21.15.161:0/1031901842 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aec0c4ad0 con 0x7f5aec152bd0 2024-03-28T23:30:56.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.380+0000 7f5af33c7640 1 -- 172.21.15.161:0/1031901842 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5aec04f4b0 con 0x7f5aec152bd0 2024-03-28T23:30:56.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.381+0000 7f5ae1ffb640 1 -- 172.21.15.161:0/1031901842 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5ae401b030 con 0x7f5aec152bd0 2024-03-28T23:30:56.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.381+0000 7f5ae1ffb640 1 --2- 172.21.15.161:0/1031901842 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5ad006b500 0x7f5ad006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:56.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.381+0000 7f5ae1ffb640 1 -- 172.21.15.161:0/1031901842 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5619+0+0 (secure 0 0 0) 0x7f5ae408a2d0 con 0x7f5aec152bd0 2024-03-28T23:30:56.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.381+0000 7f5af1163640 1 --2- 172.21.15.161:0/1031901842 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5ad006b500 0x7f5ad006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:56.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.382+0000 7f5af1163640 1 --2- 172.21.15.161:0/1031901842 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5ad006b500 0x7f5ad006d9c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8036000 tx=0x7f5ae8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:56.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.386+0000 7f5ae1ffb640 1 -- 172.21.15.161:0/1031901842 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5ae4022020 con 0x7f5aec152bd0 2024-03-28T23:30:56.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:56.555+0000 7f5af33c7640 1 -- 172.21.15.161:0/1031901842 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f5aec0c0240 con 0x7f5aec152bd0 2024-03-28T23:30:57.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.257+0000 7f5ae1ffb640 1 -- 172.21.15.161:0/1031901842 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v36) ==== 170+0+0 (secure 0 0 0) 0x7f5ae4010040 con 0x7f5aec152bd0 2024-03-28T23:30:57.260 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data' created 2024-03-28T23:30:57.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.268+0000 7f5af33c7640 1 -- 172.21.15.161:0/1031901842 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5ad006b500 msgr2=0x7f5ad006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:57.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.268+0000 7f5af33c7640 1 --2- 172.21.15.161:0/1031901842 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5ad006b500 0x7f5ad006d9c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8036000 tx=0x7f5ae8008030 comp rx=0 tx=0).stop 2024-03-28T23:30:57.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.268+0000 7f5af33c7640 1 -- 172.21.15.161:0/1031901842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5aec152bd0 msgr2=0x7f5aec141d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:57.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.268+0000 7f5af33c7640 1 --2- 172.21.15.161:0/1031901842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5aec152bd0 0x7f5aec141d90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5ae400efc0 tx=0x7f5ae400c490 comp rx=0 tx=0).stop 2024-03-28T23:30:57.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.269+0000 7f5af33c7640 1 -- 172.21.15.161:0/1031901842 shutdown_connections 2024-03-28T23:30:57.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.269+0000 7f5af33c7640 1 --2- 172.21.15.161:0/1031901842 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5ad006b500 0x7f5ad006d9c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:57.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.269+0000 7f5af33c7640 1 --2- 172.21.15.161:0/1031901842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aec14f5c0 0x7f5aec141850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:57.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.269+0000 7f5af33c7640 1 --2- 172.21.15.161:0/1031901842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5aec152bd0 0x7f5aec141d90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:57.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.269+0000 7f5af33c7640 1 --2- 172.21.15.161:0/1031901842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5aec06cdd0 0x7f5aec150e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:57.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.269+0000 7f5af33c7640 1 -- 172.21.15.161:0/1031901842 >> 172.21.15.161:0/1031901842 conn(0x7f5aec0a83f0 msgr2=0x7f5aec0a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:57.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.270+0000 7f5af33c7640 1 -- 172.21.15.161:0/1031901842 shutdown_connections 2024-03-28T23:30:57.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.270+0000 7f5af33c7640 1 -- 172.21.15.161:0/1031901842 wait complete. 2024-03-28T23:30:57.289 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-28T23:30:57.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.395+0000 7fd77da40640 1 Processor -- start 2024-03-28T23:30:57.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.395+0000 7fd77da40640 1 -- start start 2024-03-28T23:30:57.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.396+0000 7fd77da40640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd778152bc0 0x7fd7780c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:57.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.396+0000 7fd77da40640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd7781589b0 0x7fd778158e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:57.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.396+0000 7fd77da40640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7780c6f60 con 0x7fd7780c8a80 2024-03-28T23:30:57.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.396+0000 7fd77da40640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd7780c6120 con 0x7fd7781589b0 2024-03-28T23:30:57.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.396+0000 7fd77da40640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd7780c6400 con 0x7fd778152bc0 2024-03-28T23:30:57.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.396+0000 7fd776ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd778152bc0 0x7fd7780c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:57.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.396+0000 7fd7777fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd7781589b0 0x7fd778158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:57.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.396+0000 7fd776ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd778152bc0 0x7fd7780c0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:45620/0 (socket says 172.21.15.161:45620) 2024-03-28T23:30:57.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.396+0000 7fd776ffd640 1 -- 172.21.15.161:0/458335171 learned_addr learned my addr 172.21.15.161:0/458335171 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:57.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.396+0000 7fd776ffd640 1 -- 172.21.15.161:0/458335171 >> v2:172.21.15.170:3300/0 conn(0x7fd7781589b0 msgr2=0x7fd778158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:57.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.396+0000 7fd776ffd640 1 --2- 172.21.15.161:0/458335171 >> v2:172.21.15.170:3300/0 conn(0x7fd7781589b0 0x7fd778158e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:57.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.396+0000 7fd776ffd640 1 -- 172.21.15.161:0/458335171 >> v1:172.21.15.170:6789/0 conn(0x7fd7780c8a80 legacy=0x7fd7780c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:57.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.396+0000 7fd775ffb640 1 -- 172.21.15.161:0/458335171 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 894550131 0 0) 0x7fd7780c6f60 con 0x7fd7780c8a80 2024-03-28T23:30:57.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.396+0000 7fd776ffd640 1 -- 172.21.15.161:0/458335171 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7780c7240 con 0x7fd778152bc0 2024-03-28T23:30:57.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.396+0000 7fd7777fe640 1 --2- 172.21.15.161:0/458335171 >> v2:172.21.15.170:3300/0 conn(0x7fd7781589b0 0x7fd778158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:57.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.397+0000 7fd776ffd640 1 --2- 172.21.15.161:0/458335171 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd778152bc0 0x7fd7780c0330 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd768009e50 tx=0x7fd76802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1b806dfcaba045c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:57.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.397+0000 7fd775ffb640 1 -- 172.21.15.161:0/458335171 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76802cb00 con 0x7fd778152bc0 2024-03-28T23:30:57.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.397+0000 7fd775ffb640 1 -- 172.21.15.161:0/458335171 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd76802cca0 con 0x7fd778152bc0 2024-03-28T23:30:57.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.397+0000 7fd775ffb640 1 -- 172.21.15.161:0/458335171 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd768034610 con 0x7fd778152bc0 2024-03-28T23:30:57.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.398+0000 7fd77da40640 1 -- 172.21.15.161:0/458335171 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd778152bc0 msgr2=0x7fd7780c0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:57.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.398+0000 7fd77da40640 1 --2- 172.21.15.161:0/458335171 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd778152bc0 0x7fd7780c0330 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd768009e50 tx=0x7fd76802bf60 comp rx=0 tx=0).stop 2024-03-28T23:30:57.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.398+0000 7fd77da40640 1 -- 172.21.15.161:0/458335171 shutdown_connections 2024-03-28T23:30:57.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.398+0000 7fd77da40640 1 --2- 172.21.15.161:0/458335171 >> v2:172.21.15.170:3300/0 conn(0x7fd7781589b0 0x7fd778158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:57.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.398+0000 7fd77da40640 1 --2- 172.21.15.161:0/458335171 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd778152bc0 0x7fd7780c0330 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:57.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.398+0000 7fd77da40640 1 -- 172.21.15.161:0/458335171 >> 172.21.15.161:0/458335171 conn(0x7fd7780caa80 msgr2=0x7fd7780cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:57.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.398+0000 7fd77da40640 1 -- 172.21.15.161:0/458335171 shutdown_connections 2024-03-28T23:30:57.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.398+0000 7fd77da40640 1 -- 172.21.15.161:0/458335171 wait complete. 2024-03-28T23:30:57.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.399+0000 7fd77da40640 1 Processor -- start 2024-03-28T23:30:57.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.399+0000 7fd77da40640 1 -- start start 2024-03-28T23:30:57.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.400+0000 7fd77da40640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7780c8a80 0x7fd7781414a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:57.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.400+0000 7fd77da40640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd778152bc0 0x7fd778150260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:57.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.400+0000 7fd77da40640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7781589b0 0x7fd7780aba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:57.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.400+0000 7fd77da40640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd778152fa0 con 0x7fd7781589b0 2024-03-28T23:30:57.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.400+0000 7fd77da40640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd7780c95f0 con 0x7fd7780c8a80 2024-03-28T23:30:57.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.400+0000 7fd77da40640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd778153c80 con 0x7fd778152bc0 2024-03-28T23:30:57.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.400+0000 7fd776ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7780c8a80 0x7fd7781414a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:57.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.400+0000 7fd7767fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd778152bc0 0x7fd778150260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:57.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.400+0000 7fd7777fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7781589b0 0x7fd7780aba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:57.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.400+0000 7fd776ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7780c8a80 0x7fd7781414a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:48606/0 (socket says 172.21.15.161:48606) 2024-03-28T23:30:57.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.400+0000 7fd7767fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd778152bc0 0x7fd778150260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:45626/0 (socket says 172.21.15.161:45626) 2024-03-28T23:30:57.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.400+0000 7fd7777fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7781589b0 0x7fd7780aba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46864/0 (socket says 172.21.15.161:46864) 2024-03-28T23:30:57.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.400+0000 7fd776ffd640 1 -- 172.21.15.161:0/4248358615 learned_addr learned my addr 172.21.15.161:0/4248358615 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:57.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.400+0000 7fd7767fc640 1 -- 172.21.15.161:0/4248358615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7780c8a80 msgr2=0x7fd7781414a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:57.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.400+0000 7fd7767fc640 1 --2- 172.21.15.161:0/4248358615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7780c8a80 0x7fd7781414a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:57.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.400+0000 7fd7767fc640 1 -- 172.21.15.161:0/4248358615 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7781589b0 msgr2=0x7fd7780aba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:57.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.400+0000 7fd7767fc640 1 --2- 172.21.15.161:0/4248358615 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7781589b0 0x7fd7780aba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:57.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.400+0000 7fd7767fc640 1 -- 172.21.15.161:0/4248358615 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd77806c120 con 0x7fd778152bc0 2024-03-28T23:30:57.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.400+0000 7fd7777fe640 1 --2- 172.21.15.161:0/4248358615 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7781589b0 0x7fd7780aba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:57.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.400+0000 7fd776ffd640 1 --2- 172.21.15.161:0/4248358615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7780c8a80 0x7fd7781414a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:30:57.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.400+0000 7fd7767fc640 1 --2- 172.21.15.161:0/4248358615 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd778152bc0 0x7fd778150260 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd76c00c440 tx=0x7fd76c00c840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:57.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.401+0000 7fd75ffff640 1 -- 172.21.15.161:0/4248358615 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c017070 con 0x7fd778152bc0 2024-03-28T23:30:57.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.401+0000 7fd75ffff640 1 -- 172.21.15.161:0/4248358615 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd76c0035f0 con 0x7fd778152bc0 2024-03-28T23:30:57.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.401+0000 7fd77da40640 1 -- 172.21.15.161:0/4248358615 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7781ee930 con 0x7fd778152bc0 2024-03-28T23:30:57.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.401+0000 7fd75ffff640 1 -- 172.21.15.161:0/4248358615 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c0039e0 con 0x7fd778152bc0 2024-03-28T23:30:57.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.401+0000 7fd77da40640 1 -- 172.21.15.161:0/4248358615 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd77806c6a0 con 0x7fd778152bc0 2024-03-28T23:30:57.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.402+0000 7fd77da40640 1 -- 172.21.15.161:0/4248358615 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd77804f4b0 con 0x7fd778152bc0 2024-03-28T23:30:57.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.402+0000 7fd75ffff640 1 -- 172.21.15.161:0/4248358615 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd76c01a070 con 0x7fd778152bc0 2024-03-28T23:30:57.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.402+0000 7fd75ffff640 1 --2- 172.21.15.161:0/4248358615 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd76406b4d0 0x7fd76406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:57.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.403+0000 7fd75ffff640 1 -- 172.21.15.161:0/4248358615 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(36..36 src has 1..36) ==== 6001+0+0 (secure 0 0 0) 0x7fd76c0894b0 con 0x7fd778152bc0 2024-03-28T23:30:57.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.403+0000 7fd776ffd640 1 --2- 172.21.15.161:0/4248358615 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd76406b4d0 0x7fd76406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:57.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.403+0000 7fd776ffd640 1 --2- 172.21.15.161:0/4248358615 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd76406b4d0 0x7fd76406d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd768009e50 tx=0x7fd768030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:57.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.407+0000 7fd75ffff640 1 -- 172.21.15.161:0/4248358615 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd76c057d00 con 0x7fd778152bc0 2024-03-28T23:30:57.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:57.613+0000 7fd77da40640 1 -- 172.21.15.161:0/4248358615 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fd77814b310 con 0x7fd778152bc0 2024-03-28T23:30:58.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.260+0000 7fd75ffff640 1 -- 172.21.15.161:0/4248358615 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v14) ==== 172+0+0 (secure 0 0 0) 0x7fd76c05bda0 con 0x7fd778152bc0 2024-03-28T23:30:58.262 INFO:teuthology.orchestra.run.smithi161.stderr:new fs with metadata pool 5 and data pool 6 2024-03-28T23:30:58.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.270+0000 7fd77da40640 1 -- 172.21.15.161:0/4248358615 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd76406b4d0 msgr2=0x7fd76406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:58.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.270+0000 7fd77da40640 1 --2- 172.21.15.161:0/4248358615 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd76406b4d0 0x7fd76406d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd768009e50 tx=0x7fd768030000 comp rx=0 tx=0).stop 2024-03-28T23:30:58.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.270+0000 7fd77da40640 1 -- 172.21.15.161:0/4248358615 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd778152bc0 msgr2=0x7fd778150260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:58.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.270+0000 7fd77da40640 1 --2- 172.21.15.161:0/4248358615 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd778152bc0 0x7fd778150260 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd76c00c440 tx=0x7fd76c00c840 comp rx=0 tx=0).stop 2024-03-28T23:30:58.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.273+0000 7fd77da40640 1 -- 172.21.15.161:0/4248358615 shutdown_connections 2024-03-28T23:30:58.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.273+0000 7fd77da40640 1 --2- 172.21.15.161:0/4248358615 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd76406b4d0 0x7fd76406d990 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:58.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.273+0000 7fd77da40640 1 --2- 172.21.15.161:0/4248358615 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7781589b0 0x7fd7780aba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:58.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.273+0000 7fd77da40640 1 --2- 172.21.15.161:0/4248358615 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd778152bc0 0x7fd778150260 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd76c00c440 tx=0x7fd76c00c840 comp rx=0 tx=0).stop 2024-03-28T23:30:58.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.273+0000 7fd77da40640 1 --2- 172.21.15.161:0/4248358615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7780c8a80 0x7fd7781414a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:58.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.273+0000 7fd77da40640 1 -- 172.21.15.161:0/4248358615 >> 172.21.15.161:0/4248358615 conn(0x7fd7780caa80 msgr2=0x7fd778158e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:58.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.275+0000 7fd77da40640 1 -- 172.21.15.161:0/4248358615 shutdown_connections 2024-03-28T23:30:58.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.276+0000 7fd77da40640 1 -- 172.21.15.161:0/4248358615 wait complete. 2024-03-28T23:30:58.296 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:30:58.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.396+0000 7f7047a40640 1 Processor -- start 2024-03-28T23:30:58.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.396+0000 7f7047a40640 1 -- start start 2024-03-28T23:30:58.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.396+0000 7f7047a40640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f70400c5850 0x7f7040155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:58.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.396+0000 7f7047a40640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70400c5d00 con 0x7f70400c5410 2024-03-28T23:30:58.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.396+0000 7f7047a40640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7040158a30 con 0x7f7040152890 2024-03-28T23:30:58.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.396+0000 7f7047a40640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f70400c8480 con 0x7f70400c5850 2024-03-28T23:30:58.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.397+0000 7f7044fdb640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f70400c5410 0x7f70401553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:46500/0 (socket says 172.21.15.161:46500) 2024-03-28T23:30:58.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.397+0000 7f7044fdb640 1 -- 172.21.15.161:0/4066955748 learned_addr learned my addr 172.21.15.161:0/4066955748 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:58.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.397+0000 7f7045fdd640 1 --2- 172.21.15.161:0/4066955748 >> v2:172.21.15.170:3300/0 conn(0x7f70400c5850 0x7f7040155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:58.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.397+0000 7f703ffff640 1 -- 172.21.15.161:0/4066955748 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4088613944 0 0) 0x7f70400c5d00 con 0x7f70400c5410 2024-03-28T23:30:58.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.397+0000 7f703ffff640 1 -- 172.21.15.161:0/4066955748 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7028003520 con 0x7f70400c5410 2024-03-28T23:30:58.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.397+0000 7f703ffff640 1 -- 172.21.15.161:0/4066955748 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1822570133 0 0) 0x7f7040158a30 con 0x7f7040152890 2024-03-28T23:30:58.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.397+0000 7f703ffff640 1 -- 172.21.15.161:0/4066955748 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70400c5d00 con 0x7f7040152890 2024-03-28T23:30:58.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.397+0000 7f7045fdd640 1 -- 172.21.15.161:0/4066955748 >> v1:172.21.15.170:6789/0 conn(0x7f7040152890 legacy=0x7f70400c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:58.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.397+0000 7f703ffff640 1 -- 172.21.15.161:0/4066955748 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4154241462 0 0) 0x7f7028003520 con 0x7f70400c5410 2024-03-28T23:30:58.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.397+0000 7f7045fdd640 1 -- 172.21.15.161:0/4066955748 >> v1:172.21.15.161:6789/0 conn(0x7f70400c5410 legacy=0x7f70401553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:58.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.397+0000 7f7045fdd640 1 -- 172.21.15.161:0/4066955748 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7040158d10 con 0x7f70400c5850 2024-03-28T23:30:58.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.398+0000 7f7045fdd640 1 --2- 172.21.15.161:0/4066955748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70400c5850 0x7f7040155ab0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f70300097a0 tx=0x7f703002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83333f892657965 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:58.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.398+0000 7f703ffff640 1 -- 172.21.15.161:0/4066955748 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7030037070 con 0x7f70400c5850 2024-03-28T23:30:58.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.398+0000 7f703ffff640 1 -- 172.21.15.161:0/4066955748 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7030030510 con 0x7f70400c5850 2024-03-28T23:30:58.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.398+0000 7f703ffff640 1 -- 172.21.15.161:0/4066955748 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7034001b70 con 0x7f70400c5850 2024-03-28T23:30:58.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.399+0000 7f7047a40640 1 -- 172.21.15.161:0/4066955748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70400c5850 msgr2=0x7f7040155ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:58.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.399+0000 7f7047a40640 1 --2- 172.21.15.161:0/4066955748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70400c5850 0x7f7040155ab0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f70300097a0 tx=0x7f703002b790 comp rx=0 tx=0).stop 2024-03-28T23:30:58.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.399+0000 7f7047a40640 1 -- 172.21.15.161:0/4066955748 shutdown_connections 2024-03-28T23:30:58.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.399+0000 7f7047a40640 1 --2- 172.21.15.161:0/4066955748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70400c5850 0x7f7040155ab0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:58.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.399+0000 7f7047a40640 1 -- 172.21.15.161:0/4066955748 >> 172.21.15.161:0/4066955748 conn(0x7f70400ac620 msgr2=0x7f70400aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:58.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.399+0000 7f7047a40640 1 -- 172.21.15.161:0/4066955748 shutdown_connections 2024-03-28T23:30:58.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.399+0000 7f7047a40640 1 -- 172.21.15.161:0/4066955748 wait complete. 2024-03-28T23:30:58.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.400+0000 7f7047a40640 1 Processor -- start 2024-03-28T23:30:58.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.400+0000 7f7047a40640 1 -- start start 2024-03-28T23:30:58.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.400+0000 7f7047a40640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70400c5410 0x7f7040157670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:58.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.400+0000 7f7047a40640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70400c5850 0x7f70401eb370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:58.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.400+0000 7f7047a40640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7040152890 0x7f70401eb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:58.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.400+0000 7f7047a40640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f70401592d0 con 0x7f70400c5850 2024-03-28T23:30:58.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.400+0000 7f7047a40640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7040159640 con 0x7f70400c5410 2024-03-28T23:30:58.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.400+0000 7f7047a40640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7040155ff0 con 0x7f7040152890 2024-03-28T23:30:58.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.401+0000 7f7044fdb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70400c5850 0x7f70401eb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:58.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.401+0000 7f7044fdb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70400c5850 0x7f70401eb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46872/0 (socket says 172.21.15.161:46872) 2024-03-28T23:30:58.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.401+0000 7f7044fdb640 1 -- 172.21.15.161:0/2743119083 learned_addr learned my addr 172.21.15.161:0/2743119083 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:58.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.401+0000 7f70457dc640 1 --2- 172.21.15.161:0/2743119083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70400c5410 0x7f7040157670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:58.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.401+0000 7f7045fdd640 1 --2- 172.21.15.161:0/2743119083 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7040152890 0x7f70401eb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:58.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.401+0000 7f7044fdb640 1 -- 172.21.15.161:0/2743119083 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7040152890 msgr2=0x7f70401eb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:58.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.401+0000 7f7044fdb640 1 --2- 172.21.15.161:0/2743119083 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7040152890 0x7f70401eb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:58.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.401+0000 7f7044fdb640 1 -- 172.21.15.161:0/2743119083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70400c5410 msgr2=0x7f7040157670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:58.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.401+0000 7f7044fdb640 1 --2- 172.21.15.161:0/2743119083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70400c5410 0x7f7040157670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:58.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.401+0000 7f7044fdb640 1 -- 172.21.15.161:0/2743119083 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70400c13e0 con 0x7f70400c5850 2024-03-28T23:30:58.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.401+0000 7f7044fdb640 1 --2- 172.21.15.161:0/2743119083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70400c5850 0x7f70401eb370 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f70340014d0 tx=0x7f703400ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:58.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.401+0000 7f703dffb640 1 -- 172.21.15.161:0/2743119083 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7034017070 con 0x7f70400c5850 2024-03-28T23:30:58.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.401+0000 7f703dffb640 1 -- 172.21.15.161:0/2743119083 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f703400a640 con 0x7f70400c5850 2024-03-28T23:30:58.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.401+0000 7f703dffb640 1 -- 172.21.15.161:0/2743119083 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703400aa70 con 0x7f70400c5850 2024-03-28T23:30:58.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.401+0000 7f7047a40640 1 -- 172.21.15.161:0/2743119083 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7040154fa0 con 0x7f70400c5850 2024-03-28T23:30:58.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.401+0000 7f7047a40640 1 -- 172.21.15.161:0/2743119083 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7040154cc0 con 0x7f70400c5850 2024-03-28T23:30:58.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.402+0000 7f7047a40640 1 -- 172.21.15.161:0/2743119083 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f704004f4b0 con 0x7f70400c5850 2024-03-28T23:30:58.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.403+0000 7f703dffb640 1 -- 172.21.15.161:0/2743119083 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f703401a070 con 0x7f70400c5850 2024-03-28T23:30:58.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.403+0000 7f703dffb640 1 --2- 172.21.15.161:0/2743119083 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f702c06b4d0 0x7f702c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:58.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.403+0000 7f703dffb640 1 -- 172.21.15.161:0/2743119083 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6101+0+0 (secure 0 0 0) 0x7f70340899e0 con 0x7f70400c5850 2024-03-28T23:30:58.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.403+0000 7f70457dc640 1 --2- 172.21.15.161:0/2743119083 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f702c06b4d0 0x7f702c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:58.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.404+0000 7f70457dc640 1 --2- 172.21.15.161:0/2743119083 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f702c06b4d0 0x7f702c06d990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f70380011e0 tx=0x7f7038009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:58.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.408+0000 7f703dffb640 1 -- 172.21.15.161:0/2743119083 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7034058110 con 0x7f70400c5850 2024-03-28T23:30:58.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.581+0000 7f7047a40640 1 -- 172.21.15.161:0/2743119083 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f70400c18b0 con 0x7f70400c5850 2024-03-28T23:30:58.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.582+0000 7f703dffb640 1 -- 172.21.15.161:0/2743119083 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+16991 (secure 0 0 0) 0x7f703405c1b0 con 0x7f70400c5850 2024-03-28T23:30:58.584 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:58.584 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":37,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:30:57.616099+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:30:56.085611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-28T23:30:56.575631+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:30:58.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.584+0000 7f7047a40640 1 -- 172.21.15.161:0/2743119083 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f702c06b4d0 msgr2=0x7f702c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:58.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.584+0000 7f7047a40640 1 --2- 172.21.15.161:0/2743119083 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f702c06b4d0 0x7f702c06d990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f70380011e0 tx=0x7f7038009040 comp rx=0 tx=0).stop 2024-03-28T23:30:58.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.584+0000 7f7047a40640 1 -- 172.21.15.161:0/2743119083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70400c5850 msgr2=0x7f70401eb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:58.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.584+0000 7f7047a40640 1 --2- 172.21.15.161:0/2743119083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70400c5850 0x7f70401eb370 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f70340014d0 tx=0x7f703400ced0 comp rx=0 tx=0).stop 2024-03-28T23:30:58.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.584+0000 7f7047a40640 1 -- 172.21.15.161:0/2743119083 shutdown_connections 2024-03-28T23:30:58.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.584+0000 7f7047a40640 1 --2- 172.21.15.161:0/2743119083 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f702c06b4d0 0x7f702c06d990 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:58.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.584+0000 7f7047a40640 1 --2- 172.21.15.161:0/2743119083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70400c5850 0x7f70401eb370 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:58.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.584+0000 7f7047a40640 1 --2- 172.21.15.161:0/2743119083 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7040152890 0x7f70401eb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:58.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.584+0000 7f7047a40640 1 --2- 172.21.15.161:0/2743119083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70400c5410 0x7f7040157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:58.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.584+0000 7f7047a40640 1 -- 172.21.15.161:0/2743119083 >> 172.21.15.161:0/2743119083 conn(0x7f70400ac620 msgr2=0x7f704006cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:58.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.584+0000 7f7047a40640 1 -- 172.21.15.161:0/2743119083 shutdown_connections 2024-03-28T23:30:58.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.584+0000 7f7047a40640 1 -- 172.21.15.161:0/2743119083 wait complete. 2024-03-28T23:30:58.597 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:30:58.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.696+0000 7f733af48640 1 Processor -- start 2024-03-28T23:30:58.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.696+0000 7f733af48640 1 -- start start 2024-03-28T23:30:58.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.697+0000 7f733af48640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7334159380 0x7f73340bfed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:58.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.697+0000 7f733af48640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f73340ab250 0x7f73340c14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:58.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.697+0000 7f733af48640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73340c6840 con 0x7f73340c0410 2024-03-28T23:30:58.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.697+0000 7f733af48640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f73340c1170 con 0x7f73340ab250 2024-03-28T23:30:58.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.697+0000 7f733af48640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f73340c0a00 con 0x7f7334159380 2024-03-28T23:30:58.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.697+0000 7f73394e5640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f73340c0410 0x7f73340c1a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:46516/0 (socket says 172.21.15.161:46516) 2024-03-28T23:30:58.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.697+0000 7f73394e5640 1 -- 172.21.15.161:0/3222283400 learned_addr learned my addr 172.21.15.161:0/3222283400 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:58.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.697+0000 7f7338ce4640 1 --2- 172.21.15.161:0/3222283400 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7334159380 0x7f73340bfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:58.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.697+0000 7f732bfff640 1 --2- 172.21.15.161:0/3222283400 >> v2:172.21.15.170:3300/0 conn(0x7f73340ab250 0x7f73340c14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:58.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.697+0000 7f732b7fe640 1 -- 172.21.15.161:0/3222283400 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2616324077 0 0) 0x7f73340c6840 con 0x7f73340c0410 2024-03-28T23:30:58.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.697+0000 7f732b7fe640 1 -- 172.21.15.161:0/3222283400 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f731c003520 con 0x7f73340c0410 2024-03-28T23:30:58.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.697+0000 7f732b7fe640 1 -- 172.21.15.161:0/3222283400 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1815391692 0 0) 0x7f731c003520 con 0x7f73340c0410 2024-03-28T23:30:58.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.697+0000 7f732bfff640 1 -- 172.21.15.161:0/3222283400 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7334159380 msgr2=0x7f73340bfed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:58.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.697+0000 7f732bfff640 1 --2- 172.21.15.161:0/3222283400 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7334159380 0x7f73340bfed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:58.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.697+0000 7f732bfff640 1 -- 172.21.15.161:0/3222283400 >> v1:172.21.15.161:6789/0 conn(0x7f73340c0410 legacy=0x7f73340c1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:58.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.697+0000 7f732bfff640 1 -- 172.21.15.161:0/3222283400 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73340c5da0 con 0x7f73340ab250 2024-03-28T23:30:58.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.698+0000 7f732bfff640 1 --2- 172.21.15.161:0/3222283400 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73340ab250 0x7f73340c14c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f73240097a0 tx=0x7f732402b660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1fc2285988fa28f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:58.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.698+0000 7f732b7fe640 1 -- 172.21.15.161:0/3222283400 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7324037070 con 0x7f73340ab250 2024-03-28T23:30:58.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.698+0000 7f732b7fe640 1 -- 172.21.15.161:0/3222283400 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7324030510 con 0x7f73340ab250 2024-03-28T23:30:58.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.698+0000 7f732b7fe640 1 -- 172.21.15.161:0/3222283400 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f732402be60 con 0x7f73340ab250 2024-03-28T23:30:58.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.699+0000 7f733af48640 1 -- 172.21.15.161:0/3222283400 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73340ab250 msgr2=0x7f73340c14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:58.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.699+0000 7f733af48640 1 --2- 172.21.15.161:0/3222283400 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73340ab250 0x7f73340c14c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f73240097a0 tx=0x7f732402b660 comp rx=0 tx=0).stop 2024-03-28T23:30:58.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.699+0000 7f733af48640 1 -- 172.21.15.161:0/3222283400 shutdown_connections 2024-03-28T23:30:58.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.699+0000 7f733af48640 1 --2- 172.21.15.161:0/3222283400 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73340ab250 0x7f73340c14c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:58.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.699+0000 7f733af48640 1 --2- 172.21.15.161:0/3222283400 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7334159380 0x7f73340bfed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:58.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.699+0000 7f733af48640 1 -- 172.21.15.161:0/3222283400 >> 172.21.15.161:0/3222283400 conn(0x7f73340c6b40 msgr2=0x7f73340c6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:58.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.699+0000 7f733af48640 1 -- 172.21.15.161:0/3222283400 shutdown_connections 2024-03-28T23:30:58.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.699+0000 7f733af48640 1 -- 172.21.15.161:0/3222283400 wait complete. 2024-03-28T23:30:58.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.700+0000 7f733af48640 1 Processor -- start 2024-03-28T23:30:58.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.700+0000 7f733af48640 1 -- start start 2024-03-28T23:30:58.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.701+0000 7f733af48640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73340ab250 0x7f733406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:58.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.701+0000 7f733af48640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73340c0410 0x7f733414f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:58.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.701+0000 7f733af48640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7334159380 0x7f733414f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:58.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.701+0000 7f733af48640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f73340c64d0 con 0x7f73340ab250 2024-03-28T23:30:58.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.701+0000 7f733af48640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f73340c0c50 con 0x7f73340c0410 2024-03-28T23:30:58.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.701+0000 7f733af48640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7334159760 con 0x7f7334159380 2024-03-28T23:30:58.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.701+0000 7f7338ce4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73340ab250 0x7f733406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:58.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.701+0000 7f7338ce4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73340ab250 0x7f733406eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46874/0 (socket says 172.21.15.161:46874) 2024-03-28T23:30:58.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.701+0000 7f7338ce4640 1 -- 172.21.15.161:0/797725492 learned_addr learned my addr 172.21.15.161:0/797725492 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:58.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.701+0000 7f732bfff640 1 --2- 172.21.15.161:0/797725492 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73340c0410 0x7f733414f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:58.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.701+0000 7f73394e5640 1 --2- 172.21.15.161:0/797725492 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7334159380 0x7f733414f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:58.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.701+0000 7f7338ce4640 1 -- 172.21.15.161:0/797725492 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7334159380 msgr2=0x7f733414f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:58.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.701+0000 7f7338ce4640 1 --2- 172.21.15.161:0/797725492 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7334159380 0x7f733414f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:58.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.701+0000 7f7338ce4640 1 -- 172.21.15.161:0/797725492 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73340c0410 msgr2=0x7f733414f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:58.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.701+0000 7f7338ce4640 1 --2- 172.21.15.161:0/797725492 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73340c0410 0x7f733414f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:58.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.701+0000 7f7338ce4640 1 -- 172.21.15.161:0/797725492 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f733406d190 con 0x7f73340ab250 2024-03-28T23:30:58.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.702+0000 7f7338ce4640 1 --2- 172.21.15.161:0/797725492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73340ab250 0x7f733406eb30 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f733000e9e0 tx=0x7f733000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:58.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.702+0000 7f73297fa640 1 -- 172.21.15.161:0/797725492 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7330017070 con 0x7f73340ab250 2024-03-28T23:30:58.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.702+0000 7f73297fa640 1 -- 172.21.15.161:0/797725492 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f733000c9d0 con 0x7f73340ab250 2024-03-28T23:30:58.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.702+0000 7f733af48640 1 -- 172.21.15.161:0/797725492 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73341eb700 con 0x7f73340ab250 2024-03-28T23:30:58.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.702+0000 7f73297fa640 1 -- 172.21.15.161:0/797725492 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733000cdc0 con 0x7f73340ab250 2024-03-28T23:30:58.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.702+0000 7f733af48640 1 -- 172.21.15.161:0/797725492 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f733406cdf0 con 0x7f73340ab250 2024-03-28T23:30:58.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.703+0000 7f733af48640 1 -- 172.21.15.161:0/797725492 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f733404f4b0 con 0x7f73340ab250 2024-03-28T23:30:58.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.703+0000 7f73297fa640 1 -- 172.21.15.161:0/797725492 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f733001a070 con 0x7f73340ab250 2024-03-28T23:30:58.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.703+0000 7f73297fa640 1 --2- 172.21.15.161:0/797725492 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f732006b4d0 0x7f732006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:58.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.703+0000 7f73297fa640 1 -- 172.21.15.161:0/797725492 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6101+0+0 (secure 0 0 0) 0x7f733008a740 con 0x7f73340ab250 2024-03-28T23:30:58.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.703+0000 7f732bfff640 1 --2- 172.21.15.161:0/797725492 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f732006b4d0 0x7f732006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:58.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.704+0000 7f732bfff640 1 --2- 172.21.15.161:0/797725492 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f732006b4d0 0x7f732006d990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7324034000 tx=0x7f7324008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:58.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.708+0000 7f73297fa640 1 -- 172.21.15.161:0/797725492 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f73300594c0 con 0x7f73340ab250 2024-03-28T23:30:58.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.883+0000 7f733af48640 1 -- 172.21.15.161:0/797725492 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f733406c780 con 0x7f73340ab250 2024-03-28T23:30:58.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.884+0000 7f73297fa640 1 -- 172.21.15.161:0/797725492 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+16991 (secure 0 0 0) 0x7f7330058dc0 con 0x7f73340ab250 2024-03-28T23:30:58.886 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:30:58.886 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":37,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:30:57.616099+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:30:56.085611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-28T23:30:56.575631+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:30:58.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.886+0000 7f733af48640 1 -- 172.21.15.161:0/797725492 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f732006b4d0 msgr2=0x7f732006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:58.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.886+0000 7f733af48640 1 --2- 172.21.15.161:0/797725492 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f732006b4d0 0x7f732006d990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7324034000 tx=0x7f7324008030 comp rx=0 tx=0).stop 2024-03-28T23:30:58.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.886+0000 7f733af48640 1 -- 172.21.15.161:0/797725492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73340ab250 msgr2=0x7f733406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:58.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.886+0000 7f733af48640 1 --2- 172.21.15.161:0/797725492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73340ab250 0x7f733406eb30 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f733000e9e0 tx=0x7f733000eea0 comp rx=0 tx=0).stop 2024-03-28T23:30:58.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.887+0000 7f733af48640 1 -- 172.21.15.161:0/797725492 shutdown_connections 2024-03-28T23:30:58.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.887+0000 7f733af48640 1 --2- 172.21.15.161:0/797725492 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f732006b4d0 0x7f732006d990 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:58.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.887+0000 7f733af48640 1 --2- 172.21.15.161:0/797725492 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7334159380 0x7f733414f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:58.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.887+0000 7f733af48640 1 --2- 172.21.15.161:0/797725492 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73340c0410 0x7f733414f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:58.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.887+0000 7f733af48640 1 --2- 172.21.15.161:0/797725492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73340ab250 0x7f733406eb30 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:58.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.887+0000 7f733af48640 1 -- 172.21.15.161:0/797725492 >> 172.21.15.161:0/797725492 conn(0x7f73340c6b40 msgr2=0x7f7334156730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:58.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.887+0000 7f733af48640 1 -- 172.21.15.161:0/797725492 shutdown_connections 2024-03-28T23:30:58.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:58.887+0000 7f733af48640 1 -- 172.21.15.161:0/797725492 wait complete. 2024-03-28T23:30:58.900 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-28T23:30:59.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.003+0000 7f07063d0640 1 Processor -- start 2024-03-28T23:30:59.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.003+0000 7f07063d0640 1 -- start start 2024-03-28T23:30:59.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.004+0000 7f07063d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f070014f5c0 0x7f07000c2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:59.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.004+0000 7f07063d0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07000c2f10 con 0x7f0700152bd0 2024-03-28T23:30:59.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.004+0000 7f07063d0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f070014f9a0 con 0x7f070006cdd0 2024-03-28T23:30:59.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.004+0000 7f07063d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f07000c74d0 con 0x7f070014f5c0 2024-03-28T23:30:59.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.004+0000 7f070496d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f0700152bd0 0x7f070015c830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:46532/0 (socket says 172.21.15.161:46532) 2024-03-28T23:30:59.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.004+0000 7f070496d640 1 -- 172.21.15.161:0/145063095 learned_addr learned my addr 172.21.15.161:0/145063095 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:30:59.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.004+0000 7f06fffff640 1 --2- 172.21.15.161:0/145063095 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f070014f5c0 0x7f07000c2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:59.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f06feffd640 1 -- 172.21.15.161:0/145063095 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2091724286 0 0) 0x7f06f8001490 con 0x7f0700152bd0 2024-03-28T23:30:59.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f06feffd640 1 -- 172.21.15.161:0/145063095 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06e8003520 con 0x7f0700152bd0 2024-03-28T23:30:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f06feffd640 1 -- 172.21.15.161:0/145063095 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1517478059 0 0) 0x7f070014f9a0 con 0x7f070006cdd0 2024-03-28T23:30:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f06feffd640 1 -- 172.21.15.161:0/145063095 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06f8001490 con 0x7f070006cdd0 2024-03-28T23:30:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f06feffd640 1 -- 172.21.15.161:0/145063095 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2773807541 0 0) 0x7f06e8003520 con 0x7f0700152bd0 2024-03-28T23:30:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f06feffd640 1 -- 172.21.15.161:0/145063095 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f070014f5c0 msgr2=0x7f07000c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f06feffd640 1 --2- 172.21.15.161:0/145063095 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f070014f5c0 0x7f07000c2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f06feffd640 1 -- 172.21.15.161:0/145063095 >> v1:172.21.15.170:6789/0 conn(0x7f070006cdd0 legacy=0x7f07001524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f06feffd640 1 -- 172.21.15.161:0/145063095 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f070006c140 con 0x7f0700152bd0 2024-03-28T23:30:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f06feffd640 1 -- 172.21.15.161:0/145063095 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f06f8001bc0 con 0x7f0700152bd0 2024-03-28T23:30:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f06feffd640 1 -- 172.21.15.161:0/145063095 >> v1:172.21.15.161:6789/0 conn(0x7f0700152bd0 legacy=0x7f070015c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f06feffd640 1 --2- 172.21.15.161:0/145063095 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06e8003c50 0x7f06e8004050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f06feffd640 1 --2- 172.21.15.161:0/145063095 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f070014f5c0 0x7f06e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f06feffd640 1 --2- 172.21.15.161:0/145063095 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06e8004d40 0x7f06e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f06feffd640 1 -- 172.21.15.161:0/145063095 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f07000c74d0 con 0x7f06e8003c50 2024-03-28T23:30:59.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f06feffd640 1 -- 172.21.15.161:0/145063095 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f06e800bb50 con 0x7f06e8004d40 2024-03-28T23:30:59.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f06feffd640 1 -- 172.21.15.161:0/145063095 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f06e800bd20 con 0x7f070014f5c0 2024-03-28T23:30:59.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f06fffff640 1 --2- 172.21.15.161:0/145063095 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06e8003c50 0x7f06e8004050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:59.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f06ff7fe640 1 --2- 172.21.15.161:0/145063095 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f070014f5c0 0x7f06e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:59.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f070496d640 1 --2- 172.21.15.161:0/145063095 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06e8004d40 0x7f06e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:59.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f06fffff640 1 -- 172.21.15.161:0/145063095 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f070014f5c0 msgr2=0x7f06e8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:59.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f06fffff640 1 --2- 172.21.15.161:0/145063095 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f070014f5c0 0x7f06e8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:59.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f06fffff640 1 -- 172.21.15.161:0/145063095 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06e8004d40 msgr2=0x7f06e80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:30:59.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f06fffff640 1 --2- 172.21.15.161:0/145063095 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06e8004d40 0x7f06e80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:59.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.005+0000 7f06fffff640 1 -- 172.21.15.161:0/145063095 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06e8003520 con 0x7f06e8003c50 2024-03-28T23:30:59.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.006+0000 7f06fffff640 1 --2- 172.21.15.161:0/145063095 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06e8003c50 0x7f06e8004050 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f06f4009ac0 tx=0x7f06e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bde5239d259de31 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:59.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.006+0000 7f06feffd640 1 -- 172.21.15.161:0/145063095 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f403b070 con 0x7f06e8003c50 2024-03-28T23:30:59.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.006+0000 7f06feffd640 1 -- 172.21.15.161:0/145063095 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06f402eca0 con 0x7f06e8003c50 2024-03-28T23:30:59.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.006+0000 7f06feffd640 1 -- 172.21.15.161:0/145063095 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f4009690 con 0x7f06e8003c50 2024-03-28T23:30:59.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.007+0000 7f07063d0640 1 -- 172.21.15.161:0/145063095 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06e8003c50 msgr2=0x7f06e8004050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:59.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.007+0000 7f07063d0640 1 --2- 172.21.15.161:0/145063095 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06e8003c50 0x7f06e8004050 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f06f4009ac0 tx=0x7f06e8005830 comp rx=0 tx=0).stop 2024-03-28T23:30:59.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.007+0000 7f070496d640 1 -- 172.21.15.161:0/145063095 reap_dead start 2024-03-28T23:30:59.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.007+0000 7f07063d0640 1 -- 172.21.15.161:0/145063095 shutdown_connections 2024-03-28T23:30:59.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.007+0000 7f07063d0640 1 -- 172.21.15.161:0/145063095 >> 172.21.15.161:0/145063095 conn(0x7f07000a83f0 msgr2=0x7f07000a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:59.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.007+0000 7f07063d0640 1 -- 172.21.15.161:0/145063095 shutdown_connections 2024-03-28T23:30:59.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.007+0000 7f07063d0640 1 -- 172.21.15.161:0/145063095 wait complete. 2024-03-28T23:30:59.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.008+0000 7f07063d0640 1 Processor -- start 2024-03-28T23:30:59.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.008+0000 7f07063d0640 1 -- start start 2024-03-28T23:30:59.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.009+0000 7f07063d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06e8003c50 0x7f07000c5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:59.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.009+0000 7f07063d0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0700156030 0x7f07001488c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:59.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.009+0000 7f07063d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0700155b00 0x7f0700148e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:59.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.009+0000 7f07063d0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0700152030 con 0x7f0700156030 2024-03-28T23:30:59.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.009+0000 7f07063d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f07000c6fa0 con 0x7f06e8003c50 2024-03-28T23:30:59.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.009+0000 7f07063d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f070006c3e0 con 0x7f0700155b00 2024-03-28T23:30:59.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.009+0000 7f06ff7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0700156030 0x7f07001488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:59.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.009+0000 7f06ff7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0700156030 0x7f07001488c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46900/0 (socket says 172.21.15.161:46900) 2024-03-28T23:30:59.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.009+0000 7f06ff7fe640 1 -- 172.21.15.161:0/2415646527 learned_addr learned my addr 172.21.15.161:0/2415646527 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:30:59.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.009+0000 7f070496d640 1 --2- 172.21.15.161:0/2415646527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0700155b00 0x7f0700148e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:59.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.009+0000 7f06fffff640 1 --2- 172.21.15.161:0/2415646527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06e8003c50 0x7f07000c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:59.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.009+0000 7f06ff7fe640 1 -- 172.21.15.161:0/2415646527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0700155b00 msgr2=0x7f0700148e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:59.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.009+0000 7f06ff7fe640 1 --2- 172.21.15.161:0/2415646527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0700155b00 0x7f0700148e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:59.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.009+0000 7f06ff7fe640 1 -- 172.21.15.161:0/2415646527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06e8003c50 msgr2=0x7f07000c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:59.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.009+0000 7f06ff7fe640 1 --2- 172.21.15.161:0/2415646527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06e8003c50 0x7f07000c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:59.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.009+0000 7f06ff7fe640 1 -- 172.21.15.161:0/2415646527 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07001e7920 con 0x7f0700156030 2024-03-28T23:30:59.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.009+0000 7f06ff7fe640 1 --2- 172.21.15.161:0/2415646527 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0700156030 0x7f07001488c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f06f000f770 tx=0x7f06f000fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:59.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.010+0000 7f06fcff9640 1 -- 172.21.15.161:0/2415646527 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f00050d0 con 0x7f0700156030 2024-03-28T23:30:59.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.010+0000 7f06fcff9640 1 -- 172.21.15.161:0/2415646527 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06f0018070 con 0x7f0700156030 2024-03-28T23:30:59.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.010+0000 7f06fcff9640 1 -- 172.21.15.161:0/2415646527 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f000d730 con 0x7f0700156030 2024-03-28T23:30:59.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.010+0000 7f07063d0640 1 -- 172.21.15.161:0/2415646527 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07000731e0 con 0x7f0700156030 2024-03-28T23:30:59.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.010+0000 7f07063d0640 1 -- 172.21.15.161:0/2415646527 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07000c59a0 con 0x7f0700156030 2024-03-28T23:30:59.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.011+0000 7f07063d0640 1 -- 172.21.15.161:0/2415646527 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07000ab060 con 0x7f0700156030 2024-03-28T23:30:59.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.011+0000 7f06fcff9640 1 -- 172.21.15.161:0/2415646527 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f06f000dae0 con 0x7f0700156030 2024-03-28T23:30:59.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.011+0000 7f06fcff9640 1 --2- 172.21.15.161:0/2415646527 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f06ec06b500 0x7f06ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:30:59.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.012+0000 7f06fcff9640 1 -- 172.21.15.161:0/2415646527 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6101+0+0 (secure 0 0 0) 0x7f06f008a310 con 0x7f0700156030 2024-03-28T23:30:59.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.012+0000 7f06fffff640 1 --2- 172.21.15.161:0/2415646527 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f06ec06b500 0x7f06ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:30:59.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.012+0000 7f06fffff640 1 --2- 172.21.15.161:0/2415646527 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f06ec06b500 0x7f06ec06d9c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f06f4038000 tx=0x7f06f4032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:30:59.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.016+0000 7f06fcff9640 1 -- 172.21.15.161:0/2415646527 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f07000ab060 con 0x7f0700156030 2024-03-28T23:30:59.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.231+0000 7f07063d0640 1 -- 172.21.15.161:0/2415646527 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f0700072fd0 con 0x7f0700156030 2024-03-28T23:30:59.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.926+0000 7f06fcff9640 1 -- 172.21.15.161:0/2415646527 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) ==== 118+0+0 (secure 0 0 0) 0x7f06f0058a40 con 0x7f0700156030 2024-03-28T23:30:59.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.929+0000 7f07063d0640 1 -- 172.21.15.161:0/2415646527 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f06ec06b500 msgr2=0x7f06ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:59.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.929+0000 7f07063d0640 1 --2- 172.21.15.161:0/2415646527 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f06ec06b500 0x7f06ec06d9c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f06f4038000 tx=0x7f06f4032000 comp rx=0 tx=0).stop 2024-03-28T23:30:59.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.929+0000 7f07063d0640 1 -- 172.21.15.161:0/2415646527 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0700156030 msgr2=0x7f07001488c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:30:59.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.929+0000 7f07063d0640 1 --2- 172.21.15.161:0/2415646527 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0700156030 0x7f07001488c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f06f000f770 tx=0x7f06f000fc30 comp rx=0 tx=0).stop 2024-03-28T23:30:59.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.929+0000 7f07063d0640 1 -- 172.21.15.161:0/2415646527 shutdown_connections 2024-03-28T23:30:59.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.929+0000 7f07063d0640 1 --2- 172.21.15.161:0/2415646527 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f06ec06b500 0x7f06ec06d9c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:59.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.929+0000 7f07063d0640 1 --2- 172.21.15.161:0/2415646527 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0700156030 0x7f07001488c0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:59.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.929+0000 7f07063d0640 1 --2- 172.21.15.161:0/2415646527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0700155b00 0x7f0700148e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:59.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.929+0000 7f07063d0640 1 --2- 172.21.15.161:0/2415646527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06e8003c50 0x7f07000c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:30:59.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.929+0000 7f07063d0640 1 -- 172.21.15.161:0/2415646527 >> 172.21.15.161:0/2415646527 conn(0x7f07000a83f0 msgr2=0x7f07000a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:30:59.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.929+0000 7f07063d0640 1 -- 172.21.15.161:0/2415646527 shutdown_connections 2024-03-28T23:30:59.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:30:59.929+0000 7f07063d0640 1 -- 172.21.15.161:0/2415646527 wait complete. 2024-03-28T23:30:59.942 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-28T23:30:59.942 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-28T23:31:00.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.073+0000 7f6ab0ef4640 1 Processor -- start 2024-03-28T23:31:00.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.073+0000 7f6ab0ef4640 1 -- start start 2024-03-28T23:31:00.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.074+0000 7f6ab0ef4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6aac0670d0 0x7f6aac06b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:00.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.074+0000 7f6ab0ef4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6aac0c13c0 0x7f6aac06b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:00.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.074+0000 7f6ab0ef4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6aac155670 con 0x7f6aac152890 2024-03-28T23:31:00.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.074+0000 7f6ab0ef4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6aac155950 con 0x7f6aac0c13c0 2024-03-28T23:31:00.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.074+0000 7f6ab0ef4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f6aac155c30 con 0x7f6aac0670d0 2024-03-28T23:31:00.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.074+0000 7f6aaad9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6aac0c13c0 0x7f6aac06b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:00.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.074+0000 7f6aaad9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6aac0c13c0 0x7f6aac06b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46910/0 (socket says 172.21.15.161:46910) 2024-03-28T23:31:00.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.074+0000 7f6aaad9d640 1 -- 172.21.15.161:0/1891000021 learned_addr learned my addr 172.21.15.161:0/1891000021 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:00.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.074+0000 7f6aaad9d640 1 -- 172.21.15.161:0/1891000021 >> v2:172.21.15.170:3300/0 conn(0x7f6aac0670d0 msgr2=0x7f6aac06b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:00.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.074+0000 7f6aa3fff640 1 --2- 172.21.15.161:0/1891000021 >> v2:172.21.15.170:3300/0 conn(0x7f6aac0670d0 0x7f6aac06b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:00.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.074+0000 7f6aaad9d640 1 --2- 172.21.15.161:0/1891000021 >> v2:172.21.15.170:3300/0 conn(0x7f6aac0670d0 0x7f6aac06b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:00.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.074+0000 7f6aaad9d640 1 -- 172.21.15.161:0/1891000021 >> v1:172.21.15.170:6789/0 conn(0x7f6aac152890 legacy=0x7f6aac0c04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:00.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.074+0000 7f6aaad9d640 1 -- 172.21.15.161:0/1891000021 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aac155f10 con 0x7f6aac0c13c0 2024-03-28T23:31:00.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.075+0000 7f6aaad9d640 1 --2- 172.21.15.161:0/1891000021 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6aac0c13c0 0x7f6aac06b690 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f6aac073690 tx=0x7f6aa402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d8871e87bbc9f4e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:00.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.075+0000 7f6aa9d9b640 1 -- 172.21.15.161:0/1891000021 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa402c9f0 con 0x7f6aac0c13c0 2024-03-28T23:31:00.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.075+0000 7f6aa9d9b640 1 -- 172.21.15.161:0/1891000021 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6aa402cb90 con 0x7f6aac0c13c0 2024-03-28T23:31:00.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.075+0000 7f6aa9d9b640 1 -- 172.21.15.161:0/1891000021 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa40343f0 con 0x7f6aac0c13c0 2024-03-28T23:31:00.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.076+0000 7f6ab0ef4640 1 -- 172.21.15.161:0/1891000021 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6aac0c13c0 msgr2=0x7f6aac06b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:00.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.076+0000 7f6ab0ef4640 1 --2- 172.21.15.161:0/1891000021 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6aac0c13c0 0x7f6aac06b690 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f6aac073690 tx=0x7f6aa402be50 comp rx=0 tx=0).stop 2024-03-28T23:31:00.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.076+0000 7f6ab0ef4640 1 -- 172.21.15.161:0/1891000021 shutdown_connections 2024-03-28T23:31:00.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.076+0000 7f6ab0ef4640 1 --2- 172.21.15.161:0/1891000021 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6aac0c13c0 0x7f6aac06b690 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:00.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.076+0000 7f6ab0ef4640 1 --2- 172.21.15.161:0/1891000021 >> v2:172.21.15.170:3300/0 conn(0x7f6aac0670d0 0x7f6aac06b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:00.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.076+0000 7f6ab0ef4640 1 -- 172.21.15.161:0/1891000021 >> 172.21.15.161:0/1891000021 conn(0x7f6aac0c3b70 msgr2=0x7f6aac0c3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:00.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.076+0000 7f6ab0ef4640 1 -- 172.21.15.161:0/1891000021 shutdown_connections 2024-03-28T23:31:00.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.076+0000 7f6ab0ef4640 1 -- 172.21.15.161:0/1891000021 wait complete. 2024-03-28T23:31:00.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.077+0000 7f6ab0ef4640 1 Processor -- start 2024-03-28T23:31:00.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.077+0000 7f6ab0ef4640 1 -- start start 2024-03-28T23:31:00.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.078+0000 7f6ab0ef4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6aac0670d0 0x7f6aac151100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:00.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.078+0000 7f6ab0ef4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6aac0c13c0 0x7f6aac1eb500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:00.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.078+0000 7f6ab0ef4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6aac152890 0x7f6aac1eba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:00.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.078+0000 7f6ab0ef4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6aac0c0bb0 con 0x7f6aac152890 2024-03-28T23:31:00.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.078+0000 7f6ab0ef4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6aac152c70 con 0x7f6aac0c13c0 2024-03-28T23:31:00.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.078+0000 7f6ab0ef4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6aac0c0d30 con 0x7f6aac0670d0 2024-03-28T23:31:00.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.078+0000 7f6aaad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6aac152890 0x7f6aac1eba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:00.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.078+0000 7f6aaad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6aac152890 0x7f6aac1eba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46912/0 (socket says 172.21.15.161:46912) 2024-03-28T23:31:00.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.078+0000 7f6aaad9d640 1 -- 172.21.15.161:0/1298986372 learned_addr learned my addr 172.21.15.161:0/1298986372 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:00.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.078+0000 7f6aaa59c640 1 --2- 172.21.15.161:0/1298986372 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6aac0670d0 0x7f6aac151100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:00.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.078+0000 7f6aa3fff640 1 --2- 172.21.15.161:0/1298986372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6aac0c13c0 0x7f6aac1eb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:00.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.078+0000 7f6aaad9d640 1 -- 172.21.15.161:0/1298986372 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6aac0670d0 msgr2=0x7f6aac151100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:00.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.078+0000 7f6aaad9d640 1 --2- 172.21.15.161:0/1298986372 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6aac0670d0 0x7f6aac151100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:00.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.078+0000 7f6aaad9d640 1 -- 172.21.15.161:0/1298986372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6aac0c13c0 msgr2=0x7f6aac1eb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:00.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.078+0000 7f6aaad9d640 1 --2- 172.21.15.161:0/1298986372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6aac0c13c0 0x7f6aac1eb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:00.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.078+0000 7f6aaad9d640 1 -- 172.21.15.161:0/1298986372 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aac0c6cc0 con 0x7f6aac152890 2024-03-28T23:31:00.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.078+0000 7f6aaad9d640 1 --2- 172.21.15.161:0/1298986372 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6aac152890 0x7f6aac1eba40 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4009f40 tx=0x7f6aa4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:00.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.078+0000 7f6aa2ffd640 1 -- 172.21.15.161:0/1298986372 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4043020 con 0x7f6aac152890 2024-03-28T23:31:00.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.078+0000 7f6aa2ffd640 1 -- 172.21.15.161:0/1298986372 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6aa403e450 con 0x7f6aac152890 2024-03-28T23:31:00.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.078+0000 7f6aa2ffd640 1 -- 172.21.15.161:0/1298986372 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa403e820 con 0x7f6aac152890 2024-03-28T23:31:00.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.078+0000 7f6ab0ef4640 1 -- 172.21.15.161:0/1298986372 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aac0c6700 con 0x7f6aac152890 2024-03-28T23:31:00.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.079+0000 7f6ab0ef4640 1 -- 172.21.15.161:0/1298986372 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aac1483a0 con 0x7f6aac152890 2024-03-28T23:31:00.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.079+0000 7f6ab0ef4640 1 -- 172.21.15.161:0/1298986372 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6aac0736c0 con 0x7f6aac152890 2024-03-28T23:31:00.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.080+0000 7f6aa2ffd640 1 -- 172.21.15.161:0/1298986372 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6aa4046020 con 0x7f6aac152890 2024-03-28T23:31:00.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.080+0000 7f6aa2ffd640 1 --2- 172.21.15.161:0/1298986372 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6a8c06b4d0 0x7f6a8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:00.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.080+0000 7f6aa2ffd640 1 -- 172.21.15.161:0/1298986372 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f6aa40acdf0 con 0x7f6aac152890 2024-03-28T23:31:00.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.080+0000 7f6aaa59c640 1 --2- 172.21.15.161:0/1298986372 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6a8c06b4d0 0x7f6a8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:00.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.081+0000 7f6aaa59c640 1 --2- 172.21.15.161:0/1298986372 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6a8c06b4d0 0x7f6a8c06d990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6aac072490 tx=0x7f6a9c007e30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:00.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.085+0000 7f6aa2ffd640 1 -- 172.21.15.161:0/1298986372 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6aac0736c0 con 0x7f6aac152890 2024-03-28T23:31:00.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:00.322+0000 7f6ab0ef4640 1 -- 172.21.15.161:0/1298986372 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f6aac04f4b0 con 0x7f6aac152890 2024-03-28T23:31:01.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.282+0000 7f6aa2ffd640 1 -- 172.21.15.161:0/1298986372 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) ==== 122+0+0 (secure 0 0 0) 0x7f6aa404d020 con 0x7f6aac152890 2024-03-28T23:31:01.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.284+0000 7f6ab0ef4640 1 -- 172.21.15.161:0/1298986372 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6a8c06b4d0 msgr2=0x7f6a8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:01.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.284+0000 7f6ab0ef4640 1 --2- 172.21.15.161:0/1298986372 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6a8c06b4d0 0x7f6a8c06d990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6aac072490 tx=0x7f6a9c007e30 comp rx=0 tx=0).stop 2024-03-28T23:31:01.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.284+0000 7f6ab0ef4640 1 -- 172.21.15.161:0/1298986372 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6aac152890 msgr2=0x7f6aac1eba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:01.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.284+0000 7f6ab0ef4640 1 --2- 172.21.15.161:0/1298986372 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6aac152890 0x7f6aac1eba40 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4009f40 tx=0x7f6aa4004a40 comp rx=0 tx=0).stop 2024-03-28T23:31:01.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.284+0000 7f6ab0ef4640 1 -- 172.21.15.161:0/1298986372 shutdown_connections 2024-03-28T23:31:01.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.284+0000 7f6ab0ef4640 1 --2- 172.21.15.161:0/1298986372 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6a8c06b4d0 0x7f6a8c06d990 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:01.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.284+0000 7f6ab0ef4640 1 --2- 172.21.15.161:0/1298986372 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6aac152890 0x7f6aac1eba40 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:01.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.284+0000 7f6ab0ef4640 1 --2- 172.21.15.161:0/1298986372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6aac0c13c0 0x7f6aac1eb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:01.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.285+0000 7f6ab0ef4640 1 --2- 172.21.15.161:0/1298986372 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6aac0670d0 0x7f6aac151100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:01.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.285+0000 7f6ab0ef4640 1 -- 172.21.15.161:0/1298986372 >> 172.21.15.161:0/1298986372 conn(0x7f6aac0c3b70 msgr2=0x7f6aac06e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:01.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.285+0000 7f6ab0ef4640 1 -- 172.21.15.161:0/1298986372 shutdown_connections 2024-03-28T23:31:01.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.285+0000 7f6ab0ef4640 1 -- 172.21.15.161:0/1298986372 wait complete. 2024-03-28T23:31:01.297 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-28T23:31:01.298 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:31:01.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.398+0000 7f03fbfc9640 1 Processor -- start 2024-03-28T23:31:01.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.398+0000 7f03fbfc9640 1 -- start start 2024-03-28T23:31:01.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.399+0000 7f03fbfc9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f03f4152890 0x7f03f40c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:01.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.399+0000 7f03fbfc9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03f40c5b20 0x7f03f41553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:01.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.399+0000 7f03fbfc9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03f4158a30 con 0x7f03f40c8ed0 2024-03-28T23:31:01.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.399+0000 7f03fbfc9640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f03f40c8480 con 0x7f03f4152890 2024-03-28T23:31:01.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.399+0000 7f03fbfc9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f03f4159640 con 0x7f03f40c5b20 2024-03-28T23:31:01.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.399+0000 7f03f9564640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f03f40c8ed0 0x7f03f40c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41886/0 (socket says 172.21.15.161:41886) 2024-03-28T23:31:01.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.399+0000 7f03f9564640 1 -- 172.21.15.161:0/630941733 learned_addr learned my addr 172.21.15.161:0/630941733 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:01.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.399+0000 7f03f9d65640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f03f4152890 0x7f03f40c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:01.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.399+0000 7f03fa566640 1 --2- 172.21.15.161:0/630941733 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03f40c5b20 0x7f03f41553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:01.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.400+0000 7f03f9d65640 1 -- 172.21.15.161:0/630941733 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03f40c5b20 msgr2=0x7f03f41553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:01.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.400+0000 7f03f9d65640 1 --2- 172.21.15.161:0/630941733 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03f40c5b20 0x7f03f41553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:01.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.400+0000 7f03f9d65640 1 -- 172.21.15.161:0/630941733 >> v1:172.21.15.161:6789/0 conn(0x7f03f40c8ed0 legacy=0x7f03f40c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:01.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.400+0000 7f03f8d63640 1 -- 172.21.15.161:0/630941733 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 630051331 0 0) 0x7f03f4158a30 con 0x7f03f40c8ed0 2024-03-28T23:31:01.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.400+0000 7f03f9d65640 1 -- 172.21.15.161:0/630941733 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f4158d10 con 0x7f03f4152890 2024-03-28T23:31:01.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.400+0000 7f03f9d65640 1 --2- 172.21.15.161:0/630941733 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03f4152890 0x7f03f40c8990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f03f41597c0 tx=0x7f03f002b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c5c775b43cf142a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:01.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.400+0000 7f03f8d63640 1 -- 172.21.15.161:0/630941733 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f0039070 con 0x7f03f4152890 2024-03-28T23:31:01.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.400+0000 7f03f8d63640 1 -- 172.21.15.161:0/630941733 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03f00316f0 con 0x7f03f4152890 2024-03-28T23:31:01.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.400+0000 7f03f8d63640 1 -- 172.21.15.161:0/630941733 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f0031ac0 con 0x7f03f4152890 2024-03-28T23:31:01.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.401+0000 7f03fbfc9640 1 -- 172.21.15.161:0/630941733 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03f4152890 msgr2=0x7f03f40c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:01.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.401+0000 7f03fbfc9640 1 --2- 172.21.15.161:0/630941733 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03f4152890 0x7f03f40c8990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f03f41597c0 tx=0x7f03f002b790 comp rx=0 tx=0).stop 2024-03-28T23:31:01.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.401+0000 7f03fbfc9640 1 -- 172.21.15.161:0/630941733 shutdown_connections 2024-03-28T23:31:01.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.401+0000 7f03fbfc9640 1 --2- 172.21.15.161:0/630941733 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03f40c5b20 0x7f03f41553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:01.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.401+0000 7f03fbfc9640 1 --2- 172.21.15.161:0/630941733 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03f4152890 0x7f03f40c8990 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:01.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.401+0000 7f03fbfc9640 1 -- 172.21.15.161:0/630941733 >> 172.21.15.161:0/630941733 conn(0x7f03f40ac620 msgr2=0x7f03f40aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:01.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.401+0000 7f03fbfc9640 1 -- 172.21.15.161:0/630941733 shutdown_connections 2024-03-28T23:31:01.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.401+0000 7f03fbfc9640 1 -- 172.21.15.161:0/630941733 wait complete. 2024-03-28T23:31:01.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.402+0000 7f03fbfc9640 1 Processor -- start 2024-03-28T23:31:01.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.402+0000 7f03fbfc9640 1 -- start start 2024-03-28T23:31:01.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.403+0000 7f03fbfc9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03f40c5b20 0x7f03f40c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:01.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.403+0000 7f03fbfc9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03f40c8ed0 0x7f03f4149d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:01.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.403+0000 7f03fbfc9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03f4152890 0x7f03f41eb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:01.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.403+0000 7f03fbfc9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f03f4152c70 con 0x7f03f40c8ed0 2024-03-28T23:31:01.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.403+0000 7f03fbfc9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f03f41592d0 con 0x7f03f4152890 2024-03-28T23:31:01.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.403+0000 7f03fbfc9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f03f4152df0 con 0x7f03f40c5b20 2024-03-28T23:31:01.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.403+0000 7f03f9564640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03f40c8ed0 0x7f03f4149d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:01.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.403+0000 7f03f9564640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03f40c8ed0 0x7f03f4149d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45730/0 (socket says 172.21.15.161:45730) 2024-03-28T23:31:01.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.403+0000 7f03f9564640 1 -- 172.21.15.161:0/4162997954 learned_addr learned my addr 172.21.15.161:0/4162997954 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:01.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.403+0000 7f03f9d65640 1 --2- 172.21.15.161:0/4162997954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03f40c5b20 0x7f03f40c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:01.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.403+0000 7f03fa566640 1 --2- 172.21.15.161:0/4162997954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03f4152890 0x7f03f41eb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:01.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.403+0000 7f03f9564640 1 -- 172.21.15.161:0/4162997954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03f40c5b20 msgr2=0x7f03f40c1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:01.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.403+0000 7f03f9564640 1 --2- 172.21.15.161:0/4162997954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03f40c5b20 0x7f03f40c1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:01.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.403+0000 7f03f9564640 1 -- 172.21.15.161:0/4162997954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03f4152890 msgr2=0x7f03f41eb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:01.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.403+0000 7f03f9564640 1 --2- 172.21.15.161:0/4162997954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03f4152890 0x7f03f41eb450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:01.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.403+0000 7f03f9564640 1 -- 172.21.15.161:0/4162997954 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f40bff70 con 0x7f03f40c8ed0 2024-03-28T23:31:01.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.403+0000 7f03fa566640 1 --2- 172.21.15.161:0/4162997954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03f4152890 0x7f03f41eb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:01.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.403+0000 7f03f9564640 1 --2- 172.21.15.161:0/4162997954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03f40c8ed0 0x7f03f4149d20 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f03e400c760 tx=0x7f03e400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:01.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.404+0000 7f03ea7fc640 1 -- 172.21.15.161:0/4162997954 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e4007270 con 0x7f03f40c8ed0 2024-03-28T23:31:01.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.404+0000 7f03ea7fc640 1 -- 172.21.15.161:0/4162997954 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03e4007410 con 0x7f03f40c8ed0 2024-03-28T23:31:01.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.404+0000 7f03ea7fc640 1 -- 172.21.15.161:0/4162997954 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e4001d00 con 0x7f03f40c8ed0 2024-03-28T23:31:01.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.404+0000 7f03fbfc9640 1 -- 172.21.15.161:0/4162997954 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03f40beea0 con 0x7f03f40c8ed0 2024-03-28T23:31:01.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.404+0000 7f03fbfc9640 1 -- 172.21.15.161:0/4162997954 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03f40bf460 con 0x7f03f40c8ed0 2024-03-28T23:31:01.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.405+0000 7f03fbfc9640 1 -- 172.21.15.161:0/4162997954 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03f40c58c0 con 0x7f03f40c8ed0 2024-03-28T23:31:01.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.405+0000 7f03ea7fc640 1 -- 172.21.15.161:0/4162997954 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f03e40057a0 con 0x7f03f40c8ed0 2024-03-28T23:31:01.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.406+0000 7f03ea7fc640 1 --2- 172.21.15.161:0/4162997954 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f03d806b500 0x7f03d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:01.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.406+0000 7f03f9d65640 1 --2- 172.21.15.161:0/4162997954 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f03d806b500 0x7f03d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:01.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.406+0000 7f03ea7fc640 1 -- 172.21.15.161:0/4162997954 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f03e4089f30 con 0x7f03f40c8ed0 2024-03-28T23:31:01.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.406+0000 7f03f9d65640 1 --2- 172.21.15.161:0/4162997954 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f03d806b500 0x7f03d806d9c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f03f0036000 tx=0x7f03f0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:01.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.410+0000 7f03ea7fc640 1 -- 172.21.15.161:0/4162997954 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f03f40c58c0 con 0x7f03f40c8ed0 2024-03-28T23:31:01.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.625+0000 7f03fbfc9640 1 -- 172.21.15.161:0/4162997954 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03f404f4b0 con 0x7f03f40c8ed0 2024-03-28T23:31:01.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.625+0000 7f03ea7fc640 1 -- 172.21.15.161:0/4162997954 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5181 (secure 0 0 0) 0x7f03e4058cb0 con 0x7f03f40c8ed0 2024-03-28T23:31:01.627 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:01.627 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4853,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/11130685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":11130685},{"type":"v1","addr":"172.21.15.170:6834","nonce":11130685}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5110,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/465771987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":465771987},{"type":"v1","addr":"172.21.15.170:6835","nonce":465771987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5355,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/306644174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":306644174},{"type":"v1","addr":"172.21.15.161:6836","nonce":306644174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:30:57.616423+0000","modified":"2024-03-28T23:31:00.324742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5361},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5361":{"gid":5361,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6837/2636327316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":2636327316},{"type":"v1","addr":"172.21.15.161:6837","nonce":2636327316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5361,"qdb_cluster":[5361]},"id":2}]} 2024-03-28T23:31:01.628 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 17 2024-03-28T23:31:01.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.627+0000 7f03fbfc9640 1 -- 172.21.15.161:0/4162997954 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f03d806b500 msgr2=0x7f03d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:01.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.627+0000 7f03fbfc9640 1 --2- 172.21.15.161:0/4162997954 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f03d806b500 0x7f03d806d9c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f03f0036000 tx=0x7f03f0008030 comp rx=0 tx=0).stop 2024-03-28T23:31:01.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.627+0000 7f03fbfc9640 1 -- 172.21.15.161:0/4162997954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03f40c8ed0 msgr2=0x7f03f4149d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:01.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.627+0000 7f03fbfc9640 1 --2- 172.21.15.161:0/4162997954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03f40c8ed0 0x7f03f4149d20 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f03e400c760 tx=0x7f03e400cc20 comp rx=0 tx=0).stop 2024-03-28T23:31:01.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.628+0000 7f03fbfc9640 1 -- 172.21.15.161:0/4162997954 shutdown_connections 2024-03-28T23:31:01.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.628+0000 7f03fbfc9640 1 --2- 172.21.15.161:0/4162997954 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f03d806b500 0x7f03d806d9c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:01.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.628+0000 7f03fbfc9640 1 --2- 172.21.15.161:0/4162997954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03f40c8ed0 0x7f03f4149d20 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:01.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.628+0000 7f03fbfc9640 1 --2- 172.21.15.161:0/4162997954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03f4152890 0x7f03f41eb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:01.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.628+0000 7f03fbfc9640 1 --2- 172.21.15.161:0/4162997954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03f40c5b20 0x7f03f40c1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:01.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.628+0000 7f03fbfc9640 1 -- 172.21.15.161:0/4162997954 >> 172.21.15.161:0/4162997954 conn(0x7f03f40ac620 msgr2=0x7f03f41512e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:01.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.628+0000 7f03fbfc9640 1 -- 172.21.15.161:0/4162997954 shutdown_connections 2024-03-28T23:31:01.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.628+0000 7f03fbfc9640 1 -- 172.21.15.161:0/4162997954 wait complete. 2024-03-28T23:31:01.640 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:31:01.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.740+0000 7f92e4316640 1 Processor -- start 2024-03-28T23:31:01.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.740+0000 7f92e4316640 1 -- start start 2024-03-28T23:31:01.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.741+0000 7f92e4316640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f92dc06c3d0 0x7f92dc152580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:01.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.741+0000 7f92e4316640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f92dc155b90 0x7f92dc158750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:01.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.741+0000 7f92e4316640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92dc066fe0 con 0x7f92dc0ac150 2024-03-28T23:31:01.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.741+0000 7f92e4316640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f92dc156070 con 0x7f92dc06c3d0 2024-03-28T23:31:01.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.741+0000 7f92e4316640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f92dc0672c0 con 0x7f92dc155b90 2024-03-28T23:31:01.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.741+0000 7f92e20b2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f92dc06c3d0 0x7f92dc152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:01.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.741+0000 7f92e20b2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f92dc06c3d0 0x7f92dc152580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45746/0 (socket says 172.21.15.161:45746) 2024-03-28T23:31:01.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.741+0000 7f92e20b2640 1 -- 172.21.15.161:0/2753632081 learned_addr learned my addr 172.21.15.161:0/2753632081 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:01.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.741+0000 7f92e18b1640 1 --2- 172.21.15.161:0/2753632081 >> v2:172.21.15.170:3300/0 conn(0x7f92dc155b90 0x7f92dc158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:01.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.741+0000 7f92e20b2640 1 -- 172.21.15.161:0/2753632081 >> v2:172.21.15.170:3300/0 conn(0x7f92dc155b90 msgr2=0x7f92dc158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:01.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.741+0000 7f92e20b2640 1 --2- 172.21.15.161:0/2753632081 >> v2:172.21.15.170:3300/0 conn(0x7f92dc155b90 0x7f92dc158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:01.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.741+0000 7f92e20b2640 1 -- 172.21.15.161:0/2753632081 >> v1:172.21.15.170:6789/0 conn(0x7f92dc0ac150 legacy=0x7f92dc0c0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:01.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.741+0000 7f92e20b2640 1 -- 172.21.15.161:0/2753632081 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92dc0c1c80 con 0x7f92dc06c3d0 2024-03-28T23:31:01.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.741+0000 7f92e18b1640 1 --2- 172.21.15.161:0/2753632081 >> v2:172.21.15.170:3300/0 conn(0x7f92dc155b90 0x7f92dc158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:01.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.742+0000 7f92e20b2640 1 --2- 172.21.15.161:0/2753632081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92dc06c3d0 0x7f92dc152580 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f92dc0c3230 tx=0x7f92cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b836cd3d4ae0966d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:01.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.742+0000 7f92e10b0640 1 -- 172.21.15.161:0/2753632081 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92cc035020 con 0x7f92dc06c3d0 2024-03-28T23:31:01.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.742+0000 7f92e10b0640 1 -- 172.21.15.161:0/2753632081 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92cc031870 con 0x7f92dc06c3d0 2024-03-28T23:31:01.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.742+0000 7f92e10b0640 1 -- 172.21.15.161:0/2753632081 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92cc031c40 con 0x7f92dc06c3d0 2024-03-28T23:31:01.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.743+0000 7f92e4316640 1 -- 172.21.15.161:0/2753632081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92dc06c3d0 msgr2=0x7f92dc152580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:01.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.743+0000 7f92e4316640 1 --2- 172.21.15.161:0/2753632081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92dc06c3d0 0x7f92dc152580 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f92dc0c3230 tx=0x7f92cc02b910 comp rx=0 tx=0).stop 2024-03-28T23:31:01.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.743+0000 7f92e4316640 1 -- 172.21.15.161:0/2753632081 shutdown_connections 2024-03-28T23:31:01.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.743+0000 7f92e4316640 1 --2- 172.21.15.161:0/2753632081 >> v2:172.21.15.170:3300/0 conn(0x7f92dc155b90 0x7f92dc158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:01.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.743+0000 7f92e4316640 1 --2- 172.21.15.161:0/2753632081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92dc06c3d0 0x7f92dc152580 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:01.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.743+0000 7f92e4316640 1 -- 172.21.15.161:0/2753632081 >> 172.21.15.161:0/2753632081 conn(0x7f92dc0c3be0 msgr2=0x7f92dc0c3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:01.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.743+0000 7f92e4316640 1 -- 172.21.15.161:0/2753632081 shutdown_connections 2024-03-28T23:31:01.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.743+0000 7f92e4316640 1 -- 172.21.15.161:0/2753632081 wait complete. 2024-03-28T23:31:01.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.744+0000 7f92e4316640 1 Processor -- start 2024-03-28T23:31:01.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.744+0000 7f92e4316640 1 -- start start 2024-03-28T23:31:01.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.745+0000 7f92e4316640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92dc06c3d0 0x7f92dc1ee120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:01.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.745+0000 7f92e4316640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92dc0ac150 0x7f92dc146bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:01.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.745+0000 7f92e4316640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92dc155b90 0x7f92dc1470f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:01.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.745+0000 7f92e4316640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f92dc06c7b0 con 0x7f92dc06c3d0 2024-03-28T23:31:01.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.745+0000 7f92e4316640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f92dc06c930 con 0x7f92dc0ac150 2024-03-28T23:31:01.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.745+0000 7f92e4316640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f92dc0c4900 con 0x7f92dc155b90 2024-03-28T23:31:01.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.745+0000 7f92e20b2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92dc06c3d0 0x7f92dc1ee120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:01.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.745+0000 7f92e20b2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92dc06c3d0 0x7f92dc1ee120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45754/0 (socket says 172.21.15.161:45754) 2024-03-28T23:31:01.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.745+0000 7f92e20b2640 1 -- 172.21.15.161:0/508852492 learned_addr learned my addr 172.21.15.161:0/508852492 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:01.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.745+0000 7f92e18b1640 1 --2- 172.21.15.161:0/508852492 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92dc0ac150 0x7f92dc146bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:01.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.745+0000 7f92e28b3640 1 --2- 172.21.15.161:0/508852492 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92dc155b90 0x7f92dc1470f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:01.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.745+0000 7f92e20b2640 1 -- 172.21.15.161:0/508852492 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92dc155b90 msgr2=0x7f92dc1470f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:01.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.745+0000 7f92e20b2640 1 --2- 172.21.15.161:0/508852492 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92dc155b90 0x7f92dc1470f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:01.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.745+0000 7f92e20b2640 1 -- 172.21.15.161:0/508852492 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92dc0ac150 msgr2=0x7f92dc146bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:01.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.745+0000 7f92e20b2640 1 --2- 172.21.15.161:0/508852492 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92dc0ac150 0x7f92dc146bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:01.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.745+0000 7f92e20b2640 1 -- 172.21.15.161:0/508852492 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92dc1477f0 con 0x7f92dc06c3d0 2024-03-28T23:31:01.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.745+0000 7f92e28b3640 1 --2- 172.21.15.161:0/508852492 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92dc155b90 0x7f92dc1470f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:01.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.745+0000 7f92e20b2640 1 --2- 172.21.15.161:0/508852492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92dc06c3d0 0x7f92dc1ee120 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f92cc02b950 tx=0x7f92cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:01.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.746+0000 7f92d2ffd640 1 -- 172.21.15.161:0/508852492 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92cc040070 con 0x7f92dc06c3d0 2024-03-28T23:31:01.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.746+0000 7f92d2ffd640 1 -- 172.21.15.161:0/508852492 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92cc030d00 con 0x7f92dc06c3d0 2024-03-28T23:31:01.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.746+0000 7f92e4316640 1 -- 172.21.15.161:0/508852492 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92dc147f50 con 0x7f92dc06c3d0 2024-03-28T23:31:01.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.746+0000 7f92d2ffd640 1 -- 172.21.15.161:0/508852492 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92cc035040 con 0x7f92dc06c3d0 2024-03-28T23:31:01.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.746+0000 7f92e4316640 1 -- 172.21.15.161:0/508852492 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92dc1ee960 con 0x7f92dc06c3d0 2024-03-28T23:31:01.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.747+0000 7f92e4316640 1 -- 172.21.15.161:0/508852492 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92dc068470 con 0x7f92dc06c3d0 2024-03-28T23:31:01.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.747+0000 7f92d2ffd640 1 -- 172.21.15.161:0/508852492 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f92cc038020 con 0x7f92dc06c3d0 2024-03-28T23:31:01.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.747+0000 7f92d2ffd640 1 --2- 172.21.15.161:0/508852492 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f92c006b4d0 0x7f92c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:01.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.748+0000 7f92d2ffd640 1 -- 172.21.15.161:0/508852492 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f92cc0ac0c0 con 0x7f92dc06c3d0 2024-03-28T23:31:01.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.748+0000 7f92e18b1640 1 --2- 172.21.15.161:0/508852492 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f92c006b4d0 0x7f92c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:01.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.748+0000 7f92e18b1640 1 --2- 172.21.15.161:0/508852492 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f92c006b4d0 0x7f92c006d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f92dc14b830 tx=0x7f92d8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:01.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.752+0000 7f92d2ffd640 1 -- 172.21.15.161:0/508852492 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f92dc068470 con 0x7f92dc06c3d0 2024-03-28T23:31:01.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.930+0000 7f92e4316640 1 -- 172.21.15.161:0/508852492 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f92dc04f4b0 con 0x7f92dc06c3d0 2024-03-28T23:31:01.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.931+0000 7f92d2ffd640 1 -- 172.21.15.161:0/508852492 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16978 (secure 0 0 0) 0x7f92cc07a770 con 0x7f92dc06c3d0 2024-03-28T23:31:01.933 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:01.933 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":38,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:30:59.266263+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:30:56.085611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-28T23:30:56.575631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:31:01.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.933+0000 7f92e4316640 1 -- 172.21.15.161:0/508852492 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f92c006b4d0 msgr2=0x7f92c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:01.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.933+0000 7f92e4316640 1 --2- 172.21.15.161:0/508852492 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f92c006b4d0 0x7f92c006d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f92dc14b830 tx=0x7f92d8008040 comp rx=0 tx=0).stop 2024-03-28T23:31:01.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.933+0000 7f92e4316640 1 -- 172.21.15.161:0/508852492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92dc06c3d0 msgr2=0x7f92dc1ee120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:01.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.933+0000 7f92e4316640 1 --2- 172.21.15.161:0/508852492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92dc06c3d0 0x7f92dc1ee120 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f92cc02b950 tx=0x7f92cc0046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:01.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.933+0000 7f92e4316640 1 -- 172.21.15.161:0/508852492 shutdown_connections 2024-03-28T23:31:01.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.933+0000 7f92e4316640 1 --2- 172.21.15.161:0/508852492 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f92c006b4d0 0x7f92c006d990 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:01.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.933+0000 7f92e4316640 1 --2- 172.21.15.161:0/508852492 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92dc155b90 0x7f92dc1470f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:01.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.933+0000 7f92e4316640 1 --2- 172.21.15.161:0/508852492 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92dc0ac150 0x7f92dc146bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:01.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.933+0000 7f92e4316640 1 --2- 172.21.15.161:0/508852492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92dc06c3d0 0x7f92dc1ee120 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:01.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.933+0000 7f92e4316640 1 -- 172.21.15.161:0/508852492 >> 172.21.15.161:0/508852492 conn(0x7f92dc0c3be0 msgr2=0x7f92dc0c07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:01.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.933+0000 7f92e4316640 1 -- 172.21.15.161:0/508852492 shutdown_connections 2024-03-28T23:31:01.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:01.933+0000 7f92e4316640 1 -- 172.21.15.161:0/508852492 wait complete. 2024-03-28T23:31:01.946 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T23:31:01.946 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:31:02.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.048+0000 7fd652839640 1 Processor -- start 2024-03-28T23:31:02.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.048+0000 7fd652839640 1 -- start start 2024-03-28T23:31:02.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.049+0000 7fd652839640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd64c152890 0x7fd64c0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:02.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.049+0000 7fd652839640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd64c0c8ed0 0x7fd64c0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:02.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.049+0000 7fd652839640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd64c158a30 con 0x7fd64c0c5950 2024-03-28T23:31:02.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.049+0000 7fd652839640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd64c0c8480 con 0x7fd64c0c8ed0 2024-03-28T23:31:02.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.049+0000 7fd652839640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd64c159640 con 0x7fd64c152890 2024-03-28T23:31:02.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.049+0000 7fd64b7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd64c0c8ed0 0x7fd64c0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:02.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.049+0000 7fd650dd6640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd64c0c5950 0x7fd64c1553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41894/0 (socket says 172.21.15.161:41894) 2024-03-28T23:31:02.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.049+0000 7fd650dd6640 1 -- 172.21.15.161:0/3047040508 learned_addr learned my addr 172.21.15.161:0/3047040508 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:02.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.049+0000 7fd64bfff640 1 --2- 172.21.15.161:0/3047040508 >> v2:172.21.15.170:3300/0 conn(0x7fd64c152890 0x7fd64c0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:02.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.050+0000 7fd64b7fe640 1 -- 172.21.15.161:0/3047040508 >> v2:172.21.15.170:3300/0 conn(0x7fd64c152890 msgr2=0x7fd64c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:02.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.050+0000 7fd64b7fe640 1 --2- 172.21.15.161:0/3047040508 >> v2:172.21.15.170:3300/0 conn(0x7fd64c152890 0x7fd64c0c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.050+0000 7fd64b7fe640 1 -- 172.21.15.161:0/3047040508 >> v1:172.21.15.161:6789/0 conn(0x7fd64c0c5950 legacy=0x7fd64c1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:02.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.050+0000 7fd64b7fe640 1 -- 172.21.15.161:0/3047040508 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd64c158d10 con 0x7fd64c0c8ed0 2024-03-28T23:31:02.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.050+0000 7fd64affd640 1 -- 172.21.15.161:0/3047040508 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2014348374 0 0) 0x7fd64c158a30 con 0x7fd64c0c5950 2024-03-28T23:31:02.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.050+0000 7fd64b7fe640 1 --2- 172.21.15.161:0/3047040508 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd64c0c8ed0 0x7fd64c0c5410 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd64c08cc80 tx=0x7fd64002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be86f1f620d8b79 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:02.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.050+0000 7fd64affd640 1 -- 172.21.15.161:0/3047040508 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd640038070 con 0x7fd64c0c8ed0 2024-03-28T23:31:02.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.051+0000 7fd64affd640 1 -- 172.21.15.161:0/3047040508 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd640031700 con 0x7fd64c0c8ed0 2024-03-28T23:31:02.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.051+0000 7fd64affd640 1 -- 172.21.15.161:0/3047040508 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd640031ad0 con 0x7fd64c0c8ed0 2024-03-28T23:31:02.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.051+0000 7fd652839640 1 -- 172.21.15.161:0/3047040508 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd64c0c8ed0 msgr2=0x7fd64c0c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:02.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.051+0000 7fd652839640 1 --2- 172.21.15.161:0/3047040508 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd64c0c8ed0 0x7fd64c0c5410 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd64c08cc80 tx=0x7fd64002b7a0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.051+0000 7fd652839640 1 -- 172.21.15.161:0/3047040508 shutdown_connections 2024-03-28T23:31:02.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.051+0000 7fd652839640 1 --2- 172.21.15.161:0/3047040508 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd64c0c8ed0 0x7fd64c0c5410 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.051+0000 7fd652839640 1 --2- 172.21.15.161:0/3047040508 >> v2:172.21.15.170:3300/0 conn(0x7fd64c152890 0x7fd64c0c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.051+0000 7fd652839640 1 -- 172.21.15.161:0/3047040508 >> 172.21.15.161:0/3047040508 conn(0x7fd64c0ac620 msgr2=0x7fd64c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:02.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.051+0000 7fd652839640 1 -- 172.21.15.161:0/3047040508 shutdown_connections 2024-03-28T23:31:02.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.051+0000 7fd652839640 1 -- 172.21.15.161:0/3047040508 wait complete. 2024-03-28T23:31:02.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.052+0000 7fd652839640 1 Processor -- start 2024-03-28T23:31:02.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.052+0000 7fd652839640 1 -- start start 2024-03-28T23:31:02.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.053+0000 7fd652839640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd64c0c5950 0x7fd64c1ee630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:02.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.053+0000 7fd652839640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd64c0c8ed0 0x7fd64c1eb040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:02.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.053+0000 7fd652839640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd64c152890 0x7fd64c0c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:02.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.053+0000 7fd652839640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c152c70 con 0x7fd64c152890 2024-03-28T23:31:02.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.053+0000 7fd652839640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c1592d0 con 0x7fd64c0c5950 2024-03-28T23:31:02.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.053+0000 7fd652839640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd64c152df0 con 0x7fd64c0c8ed0 2024-03-28T23:31:02.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.053+0000 7fd650dd6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd64c152890 0x7fd64c0c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:02.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.053+0000 7fd650dd6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd64c152890 0x7fd64c0c9de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45776/0 (socket says 172.21.15.161:45776) 2024-03-28T23:31:02.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.053+0000 7fd650dd6640 1 -- 172.21.15.161:0/2589557998 learned_addr learned my addr 172.21.15.161:0/2589557998 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:02.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.053+0000 7fd64bfff640 1 --2- 172.21.15.161:0/2589557998 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd64c0c5950 0x7fd64c1ee630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:02.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.053+0000 7fd64b7fe640 1 --2- 172.21.15.161:0/2589557998 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd64c0c8ed0 0x7fd64c1eb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:02.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.054+0000 7fd650dd6640 1 -- 172.21.15.161:0/2589557998 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd64c0c8ed0 msgr2=0x7fd64c1eb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:02.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.054+0000 7fd650dd6640 1 --2- 172.21.15.161:0/2589557998 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd64c0c8ed0 0x7fd64c1eb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.054+0000 7fd650dd6640 1 -- 172.21.15.161:0/2589557998 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd64c0c5950 msgr2=0x7fd64c1ee630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:02.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.054+0000 7fd650dd6640 1 --2- 172.21.15.161:0/2589557998 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd64c0c5950 0x7fd64c1ee630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.054+0000 7fd650dd6640 1 -- 172.21.15.161:0/2589557998 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd64c0c1f20 con 0x7fd64c152890 2024-03-28T23:31:02.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.054+0000 7fd650dd6640 1 --2- 172.21.15.161:0/2589557998 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd64c152890 0x7fd64c0c9de0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fd64400c700 tx=0x7fd64400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:02.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.054+0000 7fd648ff9640 1 -- 172.21.15.161:0/2589557998 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd644018070 con 0x7fd64c152890 2024-03-28T23:31:02.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.054+0000 7fd648ff9640 1 -- 172.21.15.161:0/2589557998 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd64400ce10 con 0x7fd64c152890 2024-03-28T23:31:02.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.054+0000 7fd652839640 1 -- 172.21.15.161:0/2589557998 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd64c0c2200 con 0x7fd64c152890 2024-03-28T23:31:02.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.054+0000 7fd648ff9640 1 -- 172.21.15.161:0/2589557998 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64400a860 con 0x7fd64c152890 2024-03-28T23:31:02.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.054+0000 7fd652839640 1 -- 172.21.15.161:0/2589557998 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd64c1ede30 con 0x7fd64c152890 2024-03-28T23:31:02.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.055+0000 7fd652839640 1 -- 172.21.15.161:0/2589557998 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd64c04f4b0 con 0x7fd64c152890 2024-03-28T23:31:02.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.055+0000 7fd648ff9640 1 -- 172.21.15.161:0/2589557998 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd64401b070 con 0x7fd64c152890 2024-03-28T23:31:02.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.056+0000 7fd648ff9640 1 --2- 172.21.15.161:0/2589557998 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd63806b4d0 0x7fd63806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:02.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.056+0000 7fd648ff9640 1 -- 172.21.15.161:0/2589557998 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fd644089850 con 0x7fd64c152890 2024-03-28T23:31:02.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.056+0000 7fd64bfff640 1 --2- 172.21.15.161:0/2589557998 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd63806b4d0 0x7fd63806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:02.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.056+0000 7fd64bfff640 1 --2- 172.21.15.161:0/2589557998 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd63806b4d0 0x7fd63806d990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd63c009b40 tx=0x7fd63c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:02.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.060+0000 7fd648ff9640 1 -- 172.21.15.161:0/2589557998 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd644057f80 con 0x7fd64c152890 2024-03-28T23:31:02.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.231+0000 7fd652839640 1 -- 172.21.15.161:0/2589557998 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd64c0c8140 con 0x7fd63806b4d0 2024-03-28T23:31:02.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.238+0000 7fd648ff9640 1 -- 172.21.15.161:0/2589557998 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251985 (secure 0 0 0) 0x7fd64c0c8140 con 0x7fd63806b4d0 2024-03-28T23:31:02.242 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:02.243 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:31:02.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.240+0000 7fd652839640 1 -- 172.21.15.161:0/2589557998 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd63806b4d0 msgr2=0x7fd63806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:02.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.240+0000 7fd652839640 1 --2- 172.21.15.161:0/2589557998 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd63806b4d0 0x7fd63806d990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd63c009b40 tx=0x7fd63c009190 comp rx=0 tx=0).stop 2024-03-28T23:31:02.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.240+0000 7fd652839640 1 -- 172.21.15.161:0/2589557998 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd64c152890 msgr2=0x7fd64c0c9de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:02.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.240+0000 7fd652839640 1 --2- 172.21.15.161:0/2589557998 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd64c152890 0x7fd64c0c9de0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fd64400c700 tx=0x7fd64400cbc0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.240+0000 7fd652839640 1 -- 172.21.15.161:0/2589557998 shutdown_connections 2024-03-28T23:31:02.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.240+0000 7fd652839640 1 --2- 172.21.15.161:0/2589557998 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd63806b4d0 0x7fd63806d990 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.240+0000 7fd652839640 1 --2- 172.21.15.161:0/2589557998 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd64c152890 0x7fd64c0c9de0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.240+0000 7fd652839640 1 --2- 172.21.15.161:0/2589557998 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd64c0c8ed0 0x7fd64c1eb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.240+0000 7fd652839640 1 --2- 172.21.15.161:0/2589557998 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd64c0c5950 0x7fd64c1ee630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.240+0000 7fd652839640 1 -- 172.21.15.161:0/2589557998 >> 172.21.15.161:0/2589557998 conn(0x7fd64c0ac620 msgr2=0x7fd64c150630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:02.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.240+0000 7fd652839640 1 -- 172.21.15.161:0/2589557998 shutdown_connections 2024-03-28T23:31:02.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.241+0000 7fd652839640 1 -- 172.21.15.161:0/2589557998 wait complete. 2024-03-28T23:31:02.254 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2024-03-28T23:31:01.326242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":84,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":232208,"kb_used_data":3520,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754742512,"statfs":{"total":773094113280,"available":772856332288,"internally_reserved":0,"allocated":3604480,"data_stored":2652308,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.075394"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.283812+0000","last_change":"2024-03-28T23:30:58.276937+0000","last_active":"2024-03-28T23:30:58.283812+0000","last_peered":"2024-03-28T23:30:58.283812+0000","last_clean":"2024-03-28T23:30:58.283812+0000","last_became_active":"2024-03-28T23:30:58.276514+0000","last_became_peered":"2024-03-28T23:30:58.276514+0000","last_unstale":"2024-03-28T23:30:58.283812+0000","last_undegraded":"2024-03-28T23:30:58.283812+0000","last_fullsized":"2024-03-28T23:30:58.283812+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:34:06.371040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.285743+0000","last_change":"2024-03-28T23:30:58.282554+0000","last_active":"2024-03-28T23:30:58.285743+0000","last_peered":"2024-03-28T23:30:58.285743+0000","last_clean":"2024-03-28T23:30:58.285743+0000","last_became_active":"2024-03-28T23:30:58.282301+0000","last_became_peered":"2024-03-28T23:30:58.282301+0000","last_unstale":"2024-03-28T23:30:58.285743+0000","last_undegraded":"2024-03-28T23:30:58.285743+0000","last_fullsized":"2024-03-28T23:30:58.285743+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:36:48.381459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.285802+0000","last_change":"2024-03-28T23:30:58.285256+0000","last_active":"2024-03-28T23:30:58.285802+0000","last_peered":"2024-03-28T23:30:58.285802+0000","last_clean":"2024-03-28T23:30:58.285802+0000","last_became_active":"2024-03-28T23:30:58.284994+0000","last_became_peered":"2024-03-28T23:30:58.284994+0000","last_unstale":"2024-03-28T23:30:58.285802+0000","last_undegraded":"2024-03-28T23:30:58.285802+0000","last_fullsized":"2024-03-28T23:30:58.285802+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:03:27.267088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280479+0000","last_change":"2024-03-28T23:30:58.270738+0000","last_active":"2024-03-28T23:30:59.280479+0000","last_peered":"2024-03-28T23:30:59.280479+0000","last_clean":"2024-03-28T23:30:59.280479+0000","last_became_active":"2024-03-28T23:30:58.270465+0000","last_became_peered":"2024-03-28T23:30:58.270465+0000","last_unstale":"2024-03-28T23:30:59.280479+0000","last_undegraded":"2024-03-28T23:30:59.280479+0000","last_fullsized":"2024-03-28T23:30:59.280479+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:37:38.489135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280501+0000","last_change":"2024-03-28T23:30:58.266623+0000","last_active":"2024-03-28T23:30:59.280501+0000","last_peered":"2024-03-28T23:30:59.280501+0000","last_clean":"2024-03-28T23:30:59.280501+0000","last_became_active":"2024-03-28T23:30:58.266334+0000","last_became_peered":"2024-03-28T23:30:58.266334+0000","last_unstale":"2024-03-28T23:30:59.280501+0000","last_undegraded":"2024-03-28T23:30:59.280501+0000","last_fullsized":"2024-03-28T23:30:59.280501+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:31:37.187370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.285981+0000","last_change":"2024-03-28T23:30:58.280203+0000","last_active":"2024-03-28T23:30:58.285981+0000","last_peered":"2024-03-28T23:30:58.285981+0000","last_clean":"2024-03-28T23:30:58.285981+0000","last_became_active":"2024-03-28T23:30:58.279952+0000","last_became_peered":"2024-03-28T23:30:58.279952+0000","last_unstale":"2024-03-28T23:30:58.285981+0000","last_undegraded":"2024-03-28T23:30:58.285981+0000","last_fullsized":"2024-03-28T23:30:58.285981+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:04.163317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.290009+0000","last_change":"2024-03-28T23:30:58.287596+0000","last_active":"2024-03-28T23:30:58.290009+0000","last_peered":"2024-03-28T23:30:58.290009+0000","last_clean":"2024-03-28T23:30:58.290009+0000","last_became_active":"2024-03-28T23:30:58.287122+0000","last_became_peered":"2024-03-28T23:30:58.287122+0000","last_unstale":"2024-03-28T23:30:58.290009+0000","last_undegraded":"2024-03-28T23:30:58.290009+0000","last_fullsized":"2024-03-28T23:30:58.290009+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:04:49.308170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.293161+0000","last_change":"2024-03-28T23:30:58.290407+0000","last_active":"2024-03-28T23:30:58.293161+0000","last_peered":"2024-03-28T23:30:58.293161+0000","last_clean":"2024-03-28T23:30:58.293161+0000","last_became_active":"2024-03-28T23:30:58.290065+0000","last_became_peered":"2024-03-28T23:30:58.290065+0000","last_unstale":"2024-03-28T23:30:58.293161+0000","last_undegraded":"2024-03-28T23:30:58.293161+0000","last_fullsized":"2024-03-28T23:30:58.293161+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:41:27.549056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.285478+0000","last_change":"2024-03-28T23:30:58.281883+0000","last_active":"2024-03-28T23:30:58.285478+0000","last_peered":"2024-03-28T23:30:58.285478+0000","last_clean":"2024-03-28T23:30:58.285478+0000","last_became_active":"2024-03-28T23:30:58.281618+0000","last_became_peered":"2024-03-28T23:30:58.281618+0000","last_unstale":"2024-03-28T23:30:58.285478+0000","last_undegraded":"2024-03-28T23:30:58.285478+0000","last_fullsized":"2024-03-28T23:30:58.285478+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:05.426156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.288901+0000","last_change":"2024-03-28T23:30:58.279905+0000","last_active":"2024-03-28T23:30:58.288901+0000","last_peered":"2024-03-28T23:30:58.288901+0000","last_clean":"2024-03-28T23:30:58.288901+0000","last_became_active":"2024-03-28T23:30:58.279627+0000","last_became_peered":"2024-03-28T23:30:58.279627+0000","last_unstale":"2024-03-28T23:30:58.288901+0000","last_undegraded":"2024-03-28T23:30:58.288901+0000","last_fullsized":"2024-03-28T23:30:58.288901+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:02:15.637133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.290372+0000","last_change":"2024-03-28T23:30:58.289133+0000","last_active":"2024-03-28T23:30:58.290372+0000","last_peered":"2024-03-28T23:30:58.290372+0000","last_clean":"2024-03-28T23:30:58.290372+0000","last_became_active":"2024-03-28T23:30:58.288634+0000","last_became_peered":"2024-03-28T23:30:58.288634+0000","last_unstale":"2024-03-28T23:30:58.290372+0000","last_undegraded":"2024-03-28T23:30:58.290372+0000","last_fullsized":"2024-03-28T23:30:58.290372+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:38:17.760851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.284973+0000","last_change":"2024-03-28T23:30:58.281639+0000","last_active":"2024-03-28T23:30:58.284973+0000","last_peered":"2024-03-28T23:30:58.284973+0000","last_clean":"2024-03-28T23:30:58.284973+0000","last_became_active":"2024-03-28T23:30:58.276244+0000","last_became_peered":"2024-03-28T23:30:58.276244+0000","last_unstale":"2024-03-28T23:30:58.284973+0000","last_undegraded":"2024-03-28T23:30:58.284973+0000","last_fullsized":"2024-03-28T23:30:58.284973+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:04.163317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.288887+0000","last_change":"2024-03-28T23:30:58.276918+0000","last_active":"2024-03-28T23:30:58.288887+0000","last_peered":"2024-03-28T23:30:58.288887+0000","last_clean":"2024-03-28T23:30:58.288887+0000","last_became_active":"2024-03-28T23:30:58.276211+0000","last_became_peered":"2024-03-28T23:30:58.276211+0000","last_unstale":"2024-03-28T23:30:58.288887+0000","last_undegraded":"2024-03-28T23:30:58.288887+0000","last_fullsized":"2024-03-28T23:30:58.288887+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:02:55.962434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.266633+0000","last_change":"2024-03-28T23:30:58.265193+0000","last_active":"2024-03-28T23:30:58.266633+0000","last_peered":"2024-03-28T23:30:58.266633+0000","last_clean":"2024-03-28T23:30:58.266633+0000","last_became_active":"2024-03-28T23:30:58.264909+0000","last_became_peered":"2024-03-28T23:30:58.264909+0000","last_unstale":"2024-03-28T23:30:58.266633+0000","last_undegraded":"2024-03-28T23:30:58.266633+0000","last_fullsized":"2024-03-28T23:30:58.266633+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:32:35.995508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.289891+0000","last_change":"2024-03-28T23:30:58.286577+0000","last_active":"2024-03-28T23:30:58.289891+0000","last_peered":"2024-03-28T23:30:58.289891+0000","last_clean":"2024-03-28T23:30:58.289891+0000","last_became_active":"2024-03-28T23:30:58.285532+0000","last_became_peered":"2024-03-28T23:30:58.285532+0000","last_unstale":"2024-03-28T23:30:58.289891+0000","last_undegraded":"2024-03-28T23:30:58.289891+0000","last_fullsized":"2024-03-28T23:30:58.289891+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:50:54.095985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279463+0000","last_change":"2024-03-28T23:30:58.284055+0000","last_active":"2024-03-28T23:30:59.279463+0000","last_peered":"2024-03-28T23:30:59.279463+0000","last_clean":"2024-03-28T23:30:59.279463+0000","last_became_active":"2024-03-28T23:30:58.283763+0000","last_became_peered":"2024-03-28T23:30:58.283763+0000","last_unstale":"2024-03-28T23:30:59.279463+0000","last_undegraded":"2024-03-28T23:30:59.279463+0000","last_fullsized":"2024-03-28T23:30:59.279463+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:16.989843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280134+0000","last_change":"2024-03-28T23:30:58.281452+0000","last_active":"2024-03-28T23:30:59.280134+0000","last_peered":"2024-03-28T23:30:59.280134+0000","last_clean":"2024-03-28T23:30:59.280134+0000","last_became_active":"2024-03-28T23:30:58.281176+0000","last_became_peered":"2024-03-28T23:30:58.281176+0000","last_unstale":"2024-03-28T23:30:59.280134+0000","last_undegraded":"2024-03-28T23:30:59.280134+0000","last_fullsized":"2024-03-28T23:30:59.280134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:36:48.381459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279450+0000","last_change":"2024-03-28T23:30:58.266599+0000","last_active":"2024-03-28T23:30:59.279450+0000","last_peered":"2024-03-28T23:30:59.279450+0000","last_clean":"2024-03-28T23:30:59.279450+0000","last_became_active":"2024-03-28T23:30:58.266208+0000","last_became_peered":"2024-03-28T23:30:58.266208+0000","last_unstale":"2024-03-28T23:30:59.279450+0000","last_undegraded":"2024-03-28T23:30:59.279450+0000","last_fullsized":"2024-03-28T23:30:59.279450+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:04.163317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280430+0000","last_change":"2024-03-28T23:30:58.277756+0000","last_active":"2024-03-28T23:30:59.280430+0000","last_peered":"2024-03-28T23:30:59.280430+0000","last_clean":"2024-03-28T23:30:59.280430+0000","last_became_active":"2024-03-28T23:30:58.277498+0000","last_became_peered":"2024-03-28T23:30:58.277498+0000","last_unstale":"2024-03-28T23:30:59.280430+0000","last_undegraded":"2024-03-28T23:30:59.280430+0000","last_fullsized":"2024-03-28T23:30:59.280430+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:50.450151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279356+0000","last_change":"2024-03-28T23:30:58.265431+0000","last_active":"2024-03-28T23:30:59.279356+0000","last_peered":"2024-03-28T23:30:59.279356+0000","last_clean":"2024-03-28T23:30:59.279356+0000","last_became_active":"2024-03-28T23:30:58.265128+0000","last_became_peered":"2024-03-28T23:30:58.265128+0000","last_unstale":"2024-03-28T23:30:59.279356+0000","last_undegraded":"2024-03-28T23:30:59.279356+0000","last_fullsized":"2024-03-28T23:30:59.279356+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:42:28.325629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.288856+0000","last_change":"2024-03-28T23:30:58.276770+0000","last_active":"2024-03-28T23:30:58.288856+0000","last_peered":"2024-03-28T23:30:58.288856+0000","last_clean":"2024-03-28T23:30:58.288856+0000","last_became_active":"2024-03-28T23:30:58.276504+0000","last_became_peered":"2024-03-28T23:30:58.276504+0000","last_unstale":"2024-03-28T23:30:58.288856+0000","last_undegraded":"2024-03-28T23:30:58.288856+0000","last_fullsized":"2024-03-28T23:30:58.288856+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:03:27.267088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.283507+0000","last_change":"2024-03-28T23:30:58.279908+0000","last_active":"2024-03-28T23:30:58.283507+0000","last_peered":"2024-03-28T23:30:58.283507+0000","last_clean":"2024-03-28T23:30:58.283507+0000","last_became_active":"2024-03-28T23:30:58.279640+0000","last_became_peered":"2024-03-28T23:30:58.279640+0000","last_unstale":"2024-03-28T23:30:58.283507+0000","last_undegraded":"2024-03-28T23:30:58.283507+0000","last_fullsized":"2024-03-28T23:30:58.283507+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:04.163317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279011+0000","last_change":"2024-03-28T23:30:58.283847+0000","last_active":"2024-03-28T23:30:59.279011+0000","last_peered":"2024-03-28T23:30:59.279011+0000","last_clean":"2024-03-28T23:30:59.279011+0000","last_became_active":"2024-03-28T23:30:58.283565+0000","last_became_peered":"2024-03-28T23:30:58.283565+0000","last_unstale":"2024-03-28T23:30:59.279011+0000","last_undegraded":"2024-03-28T23:30:59.279011+0000","last_fullsized":"2024-03-28T23:30:59.279011+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:50:54.095985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280195+0000","last_change":"2024-03-28T23:30:58.279115+0000","last_active":"2024-03-28T23:30:59.280195+0000","last_peered":"2024-03-28T23:30:59.280195+0000","last_clean":"2024-03-28T23:30:59.280195+0000","last_became_active":"2024-03-28T23:30:58.278827+0000","last_became_peered":"2024-03-28T23:30:58.278827+0000","last_unstale":"2024-03-28T23:30:59.280195+0000","last_undegraded":"2024-03-28T23:30:59.280195+0000","last_fullsized":"2024-03-28T23:30:59.280195+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:43:32.744982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279522+0000","last_change":"2024-03-28T23:30:58.273441+0000","last_active":"2024-03-28T23:30:59.279522+0000","last_peered":"2024-03-28T23:30:59.279522+0000","last_clean":"2024-03-28T23:30:59.279522+0000","last_became_active":"2024-03-28T23:30:58.273184+0000","last_became_peered":"2024-03-28T23:30:58.273184+0000","last_unstale":"2024-03-28T23:30:59.279522+0000","last_undegraded":"2024-03-28T23:30:59.279522+0000","last_fullsized":"2024-03-28T23:30:59.279522+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:03:15.095404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.293030+0000","last_change":"2024-03-28T23:30:58.288931+0000","last_active":"2024-03-28T23:30:58.293030+0000","last_peered":"2024-03-28T23:30:58.293030+0000","last_clean":"2024-03-28T23:30:58.293030+0000","last_became_active":"2024-03-28T23:30:58.288639+0000","last_became_peered":"2024-03-28T23:30:58.288639+0000","last_unstale":"2024-03-28T23:30:58.293030+0000","last_undegraded":"2024-03-28T23:30:58.293030+0000","last_fullsized":"2024-03-28T23:30:58.293030+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:21:53.272287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279201+0000","last_change":"2024-03-28T23:30:58.275238+0000","last_active":"2024-03-28T23:30:59.279201+0000","last_peered":"2024-03-28T23:30:59.279201+0000","last_clean":"2024-03-28T23:30:59.279201+0000","last_became_active":"2024-03-28T23:30:58.274928+0000","last_became_peered":"2024-03-28T23:30:58.274928+0000","last_unstale":"2024-03-28T23:30:59.279201+0000","last_undegraded":"2024-03-28T23:30:59.279201+0000","last_fullsized":"2024-03-28T23:30:59.279201+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:52:06.639918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.289937+0000","last_change":"2024-03-28T23:30:58.287450+0000","last_active":"2024-03-28T23:30:58.289937+0000","last_peered":"2024-03-28T23:30:58.289937+0000","last_clean":"2024-03-28T23:30:58.289937+0000","last_became_active":"2024-03-28T23:30:58.287158+0000","last_became_peered":"2024-03-28T23:30:58.287158+0000","last_unstale":"2024-03-28T23:30:58.289937+0000","last_undegraded":"2024-03-28T23:30:58.289937+0000","last_fullsized":"2024-03-28T23:30:58.289937+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:16.989843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279539+0000","last_change":"2024-03-28T23:30:58.274865+0000","last_active":"2024-03-28T23:30:59.279539+0000","last_peered":"2024-03-28T23:30:59.279539+0000","last_clean":"2024-03-28T23:30:59.279539+0000","last_became_active":"2024-03-28T23:30:58.274500+0000","last_became_peered":"2024-03-28T23:30:58.274500+0000","last_unstale":"2024-03-28T23:30:59.279539+0000","last_undegraded":"2024-03-28T23:30:59.279539+0000","last_fullsized":"2024-03-28T23:30:59.279539+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:05.426156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279373+0000","last_change":"2024-03-28T23:30:58.276147+0000","last_active":"2024-03-28T23:30:59.279373+0000","last_peered":"2024-03-28T23:30:59.279373+0000","last_clean":"2024-03-28T23:30:59.279373+0000","last_became_active":"2024-03-28T23:30:58.275859+0000","last_became_peered":"2024-03-28T23:30:58.275859+0000","last_unstale":"2024-03-28T23:30:59.279373+0000","last_undegraded":"2024-03-28T23:30:59.279373+0000","last_fullsized":"2024-03-28T23:30:59.279373+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:11.983009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280157+0000","last_change":"2024-03-28T23:30:58.284545+0000","last_active":"2024-03-28T23:30:59.280157+0000","last_peered":"2024-03-28T23:30:59.280157+0000","last_clean":"2024-03-28T23:30:59.280157+0000","last_became_active":"2024-03-28T23:30:58.284258+0000","last_became_peered":"2024-03-28T23:30:58.284258+0000","last_unstale":"2024-03-28T23:30:59.280157+0000","last_undegraded":"2024-03-28T23:30:59.280157+0000","last_fullsized":"2024-03-28T23:30:59.280157+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:41:27.549056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.285528+0000","last_change":"2024-03-28T23:30:58.280540+0000","last_active":"2024-03-28T23:30:58.285528+0000","last_peered":"2024-03-28T23:30:58.285528+0000","last_clean":"2024-03-28T23:30:58.285528+0000","last_became_active":"2024-03-28T23:30:58.280103+0000","last_became_peered":"2024-03-28T23:30:58.280103+0000","last_unstale":"2024-03-28T23:30:58.285528+0000","last_undegraded":"2024-03-28T23:30:58.285528+0000","last_fullsized":"2024-03-28T23:30:58.285528+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:03:15.095404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280012+0000","last_change":"2024-03-28T23:30:58.280692+0000","last_active":"2024-03-28T23:30:59.280012+0000","last_peered":"2024-03-28T23:30:59.280012+0000","last_clean":"2024-03-28T23:30:59.280012+0000","last_became_active":"2024-03-28T23:30:58.280416+0000","last_became_peered":"2024-03-28T23:30:58.280416+0000","last_unstale":"2024-03-28T23:30:59.280012+0000","last_undegraded":"2024-03-28T23:30:59.280012+0000","last_fullsized":"2024-03-28T23:30:59.280012+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:42:55.770179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"37'4","reported_seq":18,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279506+0000","last_change":"2024-03-28T23:30:57.272785+0000","last_active":"2024-03-28T23:30:59.279506+0000","last_peered":"2024-03-28T23:30:59.279506+0000","last_clean":"2024-03-28T23:30:59.279506+0000","last_became_active":"2024-03-28T23:30:57.272546+0000","last_became_peered":"2024-03-28T23:30:57.272546+0000","last_unstale":"2024-03-28T23:30:59.279506+0000","last_undegraded":"2024-03-28T23:30:59.279506+0000","last_fullsized":"2024-03-28T23:30:59.279506+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:43:31.740970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.293043+0000","last_change":"2024-03-28T23:30:58.290118+0000","last_active":"2024-03-28T23:30:58.293043+0000","last_peered":"2024-03-28T23:30:58.293043+0000","last_clean":"2024-03-28T23:30:58.293043+0000","last_became_active":"2024-03-28T23:30:58.289839+0000","last_became_peered":"2024-03-28T23:30:58.289839+0000","last_unstale":"2024-03-28T23:30:58.293043+0000","last_undegraded":"2024-03-28T23:30:58.293043+0000","last_fullsized":"2024-03-28T23:30:58.293043+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:03:27.267088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.288455+0000","last_change":"2024-03-28T23:30:57.273444+0000","last_active":"2024-03-28T23:30:58.288455+0000","last_peered":"2024-03-28T23:30:58.288455+0000","last_clean":"2024-03-28T23:30:58.288455+0000","last_became_active":"2024-03-28T23:30:57.273170+0000","last_became_peered":"2024-03-28T23:30:57.273170+0000","last_unstale":"2024-03-28T23:30:58.288455+0000","last_undegraded":"2024-03-28T23:30:58.288455+0000","last_fullsized":"2024-03-28T23:30:58.288455+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:16:54.807782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.265869+0000","last_change":"2024-03-28T23:30:58.264617+0000","last_active":"2024-03-28T23:30:58.265869+0000","last_peered":"2024-03-28T23:30:58.265869+0000","last_clean":"2024-03-28T23:30:58.265869+0000","last_became_active":"2024-03-28T23:30:58.264329+0000","last_became_peered":"2024-03-28T23:30:58.264329+0000","last_unstale":"2024-03-28T23:30:58.265869+0000","last_undegraded":"2024-03-28T23:30:58.265869+0000","last_fullsized":"2024-03-28T23:30:58.265869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:16:55.811794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.281904+0000","last_change":"2024-03-28T23:30:57.272275+0000","last_active":"2024-03-28T23:30:58.281904+0000","last_peered":"2024-03-28T23:30:58.281904+0000","last_clean":"2024-03-28T23:30:58.281904+0000","last_became_active":"2024-03-28T23:30:57.271973+0000","last_became_peered":"2024-03-28T23:30:57.271973+0000","last_unstale":"2024-03-28T23:30:58.281904+0000","last_undegraded":"2024-03-28T23:30:58.281904+0000","last_fullsized":"2024-03-28T23:30:58.281904+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:42:27.321617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.278579+0000","last_change":"2024-03-28T23:30:58.271502+0000","last_active":"2024-03-28T23:30:58.278579+0000","last_peered":"2024-03-28T23:30:58.278579+0000","last_clean":"2024-03-28T23:30:58.278579+0000","last_became_active":"2024-03-28T23:30:58.271132+0000","last_became_peered":"2024-03-28T23:30:58.271132+0000","last_unstale":"2024-03-28T23:30:58.278579+0000","last_undegraded":"2024-03-28T23:30:58.278579+0000","last_fullsized":"2024-03-28T23:30:58.278579+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:50:54.095985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.284655+0000","last_change":"2024-03-28T23:30:57.274899+0000","last_active":"2024-03-28T23:30:58.284655+0000","last_peered":"2024-03-28T23:30:58.284655+0000","last_clean":"2024-03-28T23:30:58.284655+0000","last_became_active":"2024-03-28T23:30:57.274615+0000","last_became_peered":"2024-03-28T23:30:57.274615+0000","last_unstale":"2024-03-28T23:30:58.284655+0000","last_undegraded":"2024-03-28T23:30:58.284655+0000","last_fullsized":"2024-03-28T23:30:58.284655+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:31:36.183358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.278515+0000","last_change":"2024-03-28T23:30:58.274341+0000","last_active":"2024-03-28T23:30:58.278515+0000","last_peered":"2024-03-28T23:30:58.278515+0000","last_clean":"2024-03-28T23:30:58.278515+0000","last_became_active":"2024-03-28T23:30:58.274076+0000","last_became_peered":"2024-03-28T23:30:58.274076+0000","last_unstale":"2024-03-28T23:30:58.278515+0000","last_undegraded":"2024-03-28T23:30:58.278515+0000","last_fullsized":"2024-03-28T23:30:58.278515+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:52:06.639918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.289008+0000","last_change":"2024-03-28T23:30:57.277760+0000","last_active":"2024-03-28T23:30:58.289008+0000","last_peered":"2024-03-28T23:30:58.289008+0000","last_clean":"2024-03-28T23:30:58.289008+0000","last_became_active":"2024-03-28T23:30:57.277521+0000","last_became_peered":"2024-03-28T23:30:57.277521+0000","last_unstale":"2024-03-28T23:30:58.289008+0000","last_undegraded":"2024-03-28T23:30:58.289008+0000","last_fullsized":"2024-03-28T23:30:58.289008+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:42:27.321617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.287305+0000","last_change":"2024-03-28T23:30:58.276623+0000","last_active":"2024-03-28T23:30:58.287305+0000","last_peered":"2024-03-28T23:30:58.287305+0000","last_clean":"2024-03-28T23:30:58.287305+0000","last_became_active":"2024-03-28T23:30:58.276271+0000","last_became_peered":"2024-03-28T23:30:58.276271+0000","last_unstale":"2024-03-28T23:30:58.287305+0000","last_undegraded":"2024-03-28T23:30:58.287305+0000","last_fullsized":"2024-03-28T23:30:58.287305+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:21:53.272287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279916+0000","last_change":"2024-03-28T23:30:57.269206+0000","last_active":"2024-03-28T23:30:59.279916+0000","last_peered":"2024-03-28T23:30:59.279916+0000","last_clean":"2024-03-28T23:30:59.279916+0000","last_became_active":"2024-03-28T23:30:57.268931+0000","last_became_peered":"2024-03-28T23:30:57.268931+0000","last_unstale":"2024-03-28T23:30:59.279916+0000","last_undegraded":"2024-03-28T23:30:59.279916+0000","last_fullsized":"2024-03-28T23:30:59.279916+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:32:34.991496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.288066+0000","last_change":"2024-03-28T23:30:58.284335+0000","last_active":"2024-03-28T23:30:58.288066+0000","last_peered":"2024-03-28T23:30:58.288066+0000","last_clean":"2024-03-28T23:30:58.288066+0000","last_became_active":"2024-03-28T23:30:58.284074+0000","last_became_peered":"2024-03-28T23:30:58.284074+0000","last_unstale":"2024-03-28T23:30:58.288066+0000","last_undegraded":"2024-03-28T23:30:58.288066+0000","last_fullsized":"2024-03-28T23:30:58.288066+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:03:15.095404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.268717+0000","last_change":"2024-03-28T23:30:57.270537+0000","last_active":"2024-03-28T23:30:58.268717+0000","last_peered":"2024-03-28T23:30:58.268717+0000","last_clean":"2024-03-28T23:30:58.268717+0000","last_became_active":"2024-03-28T23:30:57.270248+0000","last_became_peered":"2024-03-28T23:30:57.270248+0000","last_unstale":"2024-03-28T23:30:58.268717+0000","last_undegraded":"2024-03-28T23:30:58.268717+0000","last_fullsized":"2024-03-28T23:30:58.268717+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:16:54.807782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.285601+0000","last_change":"2024-03-28T23:30:58.279530+0000","last_active":"2024-03-28T23:30:58.285601+0000","last_peered":"2024-03-28T23:30:58.285601+0000","last_clean":"2024-03-28T23:30:58.285601+0000","last_became_active":"2024-03-28T23:30:58.279273+0000","last_became_peered":"2024-03-28T23:30:58.279273+0000","last_unstale":"2024-03-28T23:30:58.285601+0000","last_undegraded":"2024-03-28T23:30:58.285601+0000","last_fullsized":"2024-03-28T23:30:58.285601+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:23:53.595134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.261415+0000","last_change":"2024-03-28T23:30:57.274429+0000","last_active":"2024-03-28T23:30:58.261415+0000","last_peered":"2024-03-28T23:30:58.261415+0000","last_clean":"2024-03-28T23:30:58.261415+0000","last_became_active":"2024-03-28T23:30:57.274171+0000","last_became_peered":"2024-03-28T23:30:57.274171+0000","last_unstale":"2024-03-28T23:30:58.261415+0000","last_undegraded":"2024-03-28T23:30:58.261415+0000","last_fullsized":"2024-03-28T23:30:58.261415+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:49.446139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280503+0000","last_change":"2024-03-28T23:30:58.274997+0000","last_active":"2024-03-28T23:30:59.280503+0000","last_peered":"2024-03-28T23:30:59.280503+0000","last_clean":"2024-03-28T23:30:59.280503+0000","last_became_active":"2024-03-28T23:30:58.273657+0000","last_became_peered":"2024-03-28T23:30:58.273657+0000","last_unstale":"2024-03-28T23:30:59.280503+0000","last_undegraded":"2024-03-28T23:30:59.280503+0000","last_fullsized":"2024-03-28T23:30:59.280503+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:38:41.796270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.268229+0000","last_change":"2024-03-28T23:30:57.271839+0000","last_active":"2024-03-28T23:30:58.268229+0000","last_peered":"2024-03-28T23:30:58.268229+0000","last_clean":"2024-03-28T23:30:58.268229+0000","last_became_active":"2024-03-28T23:30:57.271504+0000","last_became_peered":"2024-03-28T23:30:57.271504+0000","last_unstale":"2024-03-28T23:30:58.268229+0000","last_undegraded":"2024-03-28T23:30:58.268229+0000","last_fullsized":"2024-03-28T23:30:58.268229+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:02:54.958422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280033+0000","last_change":"2024-03-28T23:30:58.278604+0000","last_active":"2024-03-28T23:30:59.280033+0000","last_peered":"2024-03-28T23:30:59.280033+0000","last_clean":"2024-03-28T23:30:59.280033+0000","last_became_active":"2024-03-28T23:30:58.278160+0000","last_became_peered":"2024-03-28T23:30:58.278160+0000","last_unstale":"2024-03-28T23:30:59.280033+0000","last_undegraded":"2024-03-28T23:30:59.280033+0000","last_fullsized":"2024-03-28T23:30:59.280033+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:23:53.595134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.266735+0000","last_change":"2024-03-28T23:30:57.268764+0000","last_active":"2024-03-28T23:30:58.266735+0000","last_peered":"2024-03-28T23:30:58.266735+0000","last_clean":"2024-03-28T23:30:58.266735+0000","last_became_active":"2024-03-28T23:30:57.268525+0000","last_became_peered":"2024-03-28T23:30:57.268525+0000","last_unstale":"2024-03-28T23:30:58.266735+0000","last_undegraded":"2024-03-28T23:30:58.266735+0000","last_fullsized":"2024-03-28T23:30:58.266735+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:10.978997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.285130+0000","last_change":"2024-03-28T23:30:58.281875+0000","last_active":"2024-03-28T23:30:58.285130+0000","last_peered":"2024-03-28T23:30:58.285130+0000","last_clean":"2024-03-28T23:30:58.285130+0000","last_became_active":"2024-03-28T23:30:58.281594+0000","last_became_peered":"2024-03-28T23:30:58.281594+0000","last_unstale":"2024-03-28T23:30:58.285130+0000","last_undegraded":"2024-03-28T23:30:58.285130+0000","last_fullsized":"2024-03-28T23:30:58.285130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:03:15.095404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"37'3","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.289964+0000","last_change":"2024-03-28T23:30:57.272844+0000","last_active":"2024-03-28T23:30:58.289964+0000","last_peered":"2024-03-28T23:30:58.289964+0000","last_clean":"2024-03-28T23:30:58.289964+0000","last_became_active":"2024-03-28T23:30:57.272178+0000","last_became_peered":"2024-03-28T23:30:57.272178+0000","last_unstale":"2024-03-28T23:30:58.289964+0000","last_undegraded":"2024-03-28T23:30:58.289964+0000","last_fullsized":"2024-03-28T23:30:58.289964+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:03.159305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.282181+0000","last_change":"2024-03-28T23:30:58.279118+0000","last_active":"2024-03-28T23:30:58.282181+0000","last_peered":"2024-03-28T23:30:58.282181+0000","last_clean":"2024-03-28T23:30:58.282181+0000","last_became_active":"2024-03-28T23:30:58.278845+0000","last_became_peered":"2024-03-28T23:30:58.278845+0000","last_unstale":"2024-03-28T23:30:58.282181+0000","last_undegraded":"2024-03-28T23:30:58.282181+0000","last_fullsized":"2024-03-28T23:30:58.282181+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:11.983009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"37'6","reported_seq":18,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.293205+0000","last_change":"2024-03-28T23:30:57.267617+0000","last_active":"2024-03-28T23:30:58.293205+0000","last_peered":"2024-03-28T23:30:58.293205+0000","last_clean":"2024-03-28T23:30:58.293205+0000","last_became_active":"2024-03-28T23:30:57.266847+0000","last_became_peered":"2024-03-28T23:30:57.266847+0000","last_unstale":"2024-03-28T23:30:58.293205+0000","last_undegraded":"2024-03-28T23:30:58.293205+0000","last_fullsized":"2024-03-28T23:30:58.293205+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:02:14.633121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.284376+0000","last_change":"2024-03-28T23:30:58.280388+0000","last_active":"2024-03-28T23:30:58.284376+0000","last_peered":"2024-03-28T23:30:58.284376+0000","last_clean":"2024-03-28T23:30:58.284376+0000","last_became_active":"2024-03-28T23:30:58.280132+0000","last_became_peered":"2024-03-28T23:30:58.280132+0000","last_unstale":"2024-03-28T23:30:58.284376+0000","last_undegraded":"2024-03-28T23:30:58.284376+0000","last_fullsized":"2024-03-28T23:30:58.284376+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:11.983009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.283939+0000","last_change":"2024-03-28T23:30:57.275555+0000","last_active":"2024-03-28T23:30:58.283939+0000","last_peered":"2024-03-28T23:30:58.283939+0000","last_clean":"2024-03-28T23:30:58.283939+0000","last_became_active":"2024-03-28T23:30:57.275232+0000","last_became_peered":"2024-03-28T23:30:57.275232+0000","last_unstale":"2024-03-28T23:30:58.283939+0000","last_undegraded":"2024-03-28T23:30:58.283939+0000","last_fullsized":"2024-03-28T23:30:58.283939+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:03:26.263076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.283839+0000","last_change":"2024-03-28T23:30:58.276149+0000","last_active":"2024-03-28T23:30:58.283839+0000","last_peered":"2024-03-28T23:30:58.283839+0000","last_clean":"2024-03-28T23:30:58.283839+0000","last_became_active":"2024-03-28T23:30:58.275887+0000","last_became_peered":"2024-03-28T23:30:58.275887+0000","last_unstale":"2024-03-28T23:30:58.283839+0000","last_undegraded":"2024-03-28T23:30:58.283839+0000","last_fullsized":"2024-03-28T23:30:58.283839+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:16.989843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.270386+0000","last_change":"2024-03-28T23:30:57.271154+0000","last_active":"2024-03-28T23:30:58.270386+0000","last_peered":"2024-03-28T23:30:58.270386+0000","last_clean":"2024-03-28T23:30:58.270386+0000","last_became_active":"2024-03-28T23:30:57.270870+0000","last_became_peered":"2024-03-28T23:30:57.270870+0000","last_unstale":"2024-03-28T23:30:58.270386+0000","last_undegraded":"2024-03-28T23:30:58.270386+0000","last_fullsized":"2024-03-28T23:30:58.270386+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:52:05.635906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279793+0000","last_change":"2024-03-28T23:30:58.286161+0000","last_active":"2024-03-28T23:30:59.279793+0000","last_peered":"2024-03-28T23:30:59.279793+0000","last_clean":"2024-03-28T23:30:59.279793+0000","last_became_active":"2024-03-28T23:30:58.285876+0000","last_became_peered":"2024-03-28T23:30:58.285876+0000","last_unstale":"2024-03-28T23:30:59.279793+0000","last_undegraded":"2024-03-28T23:30:59.279793+0000","last_fullsized":"2024-03-28T23:30:59.279793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:38:17.760851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.278925+0000","last_change":"2024-03-28T23:30:57.271795+0000","last_active":"2024-03-28T23:30:59.278925+0000","last_peered":"2024-03-28T23:30:59.278925+0000","last_clean":"2024-03-28T23:30:59.278925+0000","last_became_active":"2024-03-28T23:30:57.271536+0000","last_became_peered":"2024-03-28T23:30:57.271536+0000","last_unstale":"2024-03-28T23:30:59.278925+0000","last_undegraded":"2024-03-28T23:30:59.278925+0000","last_fullsized":"2024-03-28T23:30:59.278925+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:14:40.512760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.283782+0000","last_change":"2024-03-28T23:30:58.276523+0000","last_active":"2024-03-28T23:30:58.283782+0000","last_peered":"2024-03-28T23:30:58.283782+0000","last_clean":"2024-03-28T23:30:58.283782+0000","last_became_active":"2024-03-28T23:30:58.276253+0000","last_became_peered":"2024-03-28T23:30:58.276253+0000","last_unstale":"2024-03-28T23:30:58.283782+0000","last_undegraded":"2024-03-28T23:30:58.283782+0000","last_fullsized":"2024-03-28T23:30:58.283782+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:04:49.308170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"37'8","reported_seq":22,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279562+0000","last_change":"2024-03-28T23:30:57.269607+0000","last_active":"2024-03-28T23:30:59.279562+0000","last_peered":"2024-03-28T23:30:59.279562+0000","last_clean":"2024-03-28T23:30:59.279562+0000","last_became_active":"2024-03-28T23:30:57.269342+0000","last_became_peered":"2024-03-28T23:30:57.269342+0000","last_unstale":"2024-03-28T23:30:59.279562+0000","last_undegraded":"2024-03-28T23:30:59.279562+0000","last_fullsized":"2024-03-28T23:30:59.279562+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:52:05.635906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.283966+0000","last_change":"2024-03-28T23:30:58.278202+0000","last_active":"2024-03-28T23:30:58.283966+0000","last_peered":"2024-03-28T23:30:58.283966+0000","last_clean":"2024-03-28T23:30:58.283966+0000","last_became_active":"2024-03-28T23:30:58.277935+0000","last_became_peered":"2024-03-28T23:30:58.277935+0000","last_unstale":"2024-03-28T23:30:58.283966+0000","last_undegraded":"2024-03-28T23:30:58.283966+0000","last_fullsized":"2024-03-28T23:30:58.283966+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:38:17.760851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.289548+0000","last_change":"2024-03-28T23:30:57.271432+0000","last_active":"2024-03-28T23:30:58.289548+0000","last_peered":"2024-03-28T23:30:58.289548+0000","last_clean":"2024-03-28T23:30:58.289548+0000","last_became_active":"2024-03-28T23:30:57.271181+0000","last_became_peered":"2024-03-28T23:30:57.271181+0000","last_unstale":"2024-03-28T23:30:58.289548+0000","last_undegraded":"2024-03-28T23:30:58.289548+0000","last_fullsized":"2024-03-28T23:30:58.289548+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:04.422144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.289542+0000","last_change":"2024-03-28T23:30:58.282817+0000","last_active":"2024-03-28T23:30:58.289542+0000","last_peered":"2024-03-28T23:30:58.289542+0000","last_clean":"2024-03-28T23:30:58.289542+0000","last_became_active":"2024-03-28T23:30:58.282571+0000","last_became_peered":"2024-03-28T23:30:58.282571+0000","last_unstale":"2024-03-28T23:30:58.289542+0000","last_undegraded":"2024-03-28T23:30:58.289542+0000","last_fullsized":"2024-03-28T23:30:58.289542+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:16:03.618378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.289869+0000","last_change":"2024-03-28T23:30:57.273183+0000","last_active":"2024-03-28T23:30:58.289869+0000","last_peered":"2024-03-28T23:30:58.289869+0000","last_clean":"2024-03-28T23:30:58.289869+0000","last_became_active":"2024-03-28T23:30:57.272920+0000","last_became_peered":"2024-03-28T23:30:57.272920+0000","last_unstale":"2024-03-28T23:30:58.289869+0000","last_undegraded":"2024-03-28T23:30:58.289869+0000","last_fullsized":"2024-03-28T23:30:58.289869+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:37:37.485123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.279272+0000","last_change":"2024-03-28T23:30:58.277460+0000","last_active":"2024-03-28T23:30:58.279272+0000","last_peered":"2024-03-28T23:30:58.279272+0000","last_clean":"2024-03-28T23:30:58.279272+0000","last_became_active":"2024-03-28T23:30:58.277204+0000","last_became_peered":"2024-03-28T23:30:58.277204+0000","last_unstale":"2024-03-28T23:30:58.279272+0000","last_undegraded":"2024-03-28T23:30:58.279272+0000","last_fullsized":"2024-03-28T23:30:58.279272+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:42:28.325629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280529+0000","last_change":"2024-03-28T23:30:57.264576+0000","last_active":"2024-03-28T23:30:59.280529+0000","last_peered":"2024-03-28T23:30:59.280529+0000","last_clean":"2024-03-28T23:30:59.280529+0000","last_became_active":"2024-03-28T23:30:57.264271+0000","last_became_peered":"2024-03-28T23:30:57.264271+0000","last_unstale":"2024-03-28T23:30:59.280529+0000","last_undegraded":"2024-03-28T23:30:59.280529+0000","last_fullsized":"2024-03-28T23:30:59.280529+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:02.708795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280667+0000","last_change":"2024-03-28T23:30:58.278480+0000","last_active":"2024-03-28T23:30:59.280667+0000","last_peered":"2024-03-28T23:30:59.280667+0000","last_clean":"2024-03-28T23:30:59.280667+0000","last_became_active":"2024-03-28T23:30:58.276636+0000","last_became_peered":"2024-03-28T23:30:58.276636+0000","last_unstale":"2024-03-28T23:30:59.280667+0000","last_undegraded":"2024-03-28T23:30:59.280667+0000","last_fullsized":"2024-03-28T23:30:59.280667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:16:55.811794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.269732+0000","last_change":"2024-03-28T23:30:57.275811+0000","last_active":"2024-03-28T23:30:58.269732+0000","last_peered":"2024-03-28T23:30:58.269732+0000","last_clean":"2024-03-28T23:30:58.269732+0000","last_became_active":"2024-03-28T23:30:57.275515+0000","last_became_peered":"2024-03-28T23:30:57.275515+0000","last_unstale":"2024-03-28T23:30:58.269732+0000","last_undegraded":"2024-03-28T23:30:58.269732+0000","last_fullsized":"2024-03-28T23:30:58.269732+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:32:34.991496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279747+0000","last_change":"2024-03-28T23:30:58.285570+0000","last_active":"2024-03-28T23:30:59.279747+0000","last_peered":"2024-03-28T23:30:59.279747+0000","last_clean":"2024-03-28T23:30:59.279747+0000","last_became_active":"2024-03-28T23:30:58.285243+0000","last_became_peered":"2024-03-28T23:30:58.285243+0000","last_unstale":"2024-03-28T23:30:59.279747+0000","last_undegraded":"2024-03-28T23:30:59.279747+0000","last_fullsized":"2024-03-28T23:30:59.279747+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:04:49.308170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280313+0000","last_change":"2024-03-28T23:30:57.277719+0000","last_active":"2024-03-28T23:30:59.280313+0000","last_peered":"2024-03-28T23:30:59.280313+0000","last_clean":"2024-03-28T23:30:59.280313+0000","last_became_active":"2024-03-28T23:30:57.277396+0000","last_became_peered":"2024-03-28T23:30:57.277396+0000","last_unstale":"2024-03-28T23:30:59.280313+0000","last_undegraded":"2024-03-28T23:30:59.280313+0000","last_fullsized":"2024-03-28T23:30:59.280313+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:14:08.521862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.280815+0000","last_change":"2024-03-28T23:30:58.279179+0000","last_active":"2024-03-28T23:30:58.280815+0000","last_peered":"2024-03-28T23:30:58.280815+0000","last_clean":"2024-03-28T23:30:58.280815+0000","last_became_active":"2024-03-28T23:30:58.278881+0000","last_became_peered":"2024-03-28T23:30:58.278881+0000","last_unstale":"2024-03-28T23:30:58.280815+0000","last_undegraded":"2024-03-28T23:30:58.280815+0000","last_fullsized":"2024-03-28T23:30:58.280815+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:42:28.325629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280289+0000","last_change":"2024-03-28T23:30:57.269324+0000","last_active":"2024-03-28T23:30:59.280289+0000","last_peered":"2024-03-28T23:30:59.280289+0000","last_clean":"2024-03-28T23:30:59.280289+0000","last_became_active":"2024-03-28T23:30:57.269043+0000","last_became_peered":"2024-03-28T23:30:57.269043+0000","last_unstale":"2024-03-28T23:30:59.280289+0000","last_undegraded":"2024-03-28T23:30:59.280289+0000","last_fullsized":"2024-03-28T23:30:59.280289+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:56:10.171229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.289223+0000","last_change":"2024-03-28T23:30:58.284238+0000","last_active":"2024-03-28T23:30:58.289223+0000","last_peered":"2024-03-28T23:30:58.289223+0000","last_clean":"2024-03-28T23:30:58.289223+0000","last_became_active":"2024-03-28T23:30:58.283967+0000","last_became_peered":"2024-03-28T23:30:58.283967+0000","last_unstale":"2024-03-28T23:30:58.289223+0000","last_undegraded":"2024-03-28T23:30:58.289223+0000","last_fullsized":"2024-03-28T23:30:58.289223+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:39:35.383747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279839+0000","last_change":"2024-03-28T23:30:57.266767+0000","last_active":"2024-03-28T23:30:59.279839+0000","last_peered":"2024-03-28T23:30:59.279839+0000","last_clean":"2024-03-28T23:30:59.279839+0000","last_became_active":"2024-03-28T23:30:57.266511+0000","last_became_peered":"2024-03-28T23:30:57.266511+0000","last_unstale":"2024-03-28T23:30:59.279839+0000","last_undegraded":"2024-03-28T23:30:59.279839+0000","last_fullsized":"2024-03-28T23:30:59.279839+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:38:40.792258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.285582+0000","last_change":"2024-03-28T23:30:58.278844+0000","last_active":"2024-03-28T23:30:58.285582+0000","last_peered":"2024-03-28T23:30:58.285582+0000","last_clean":"2024-03-28T23:30:58.285582+0000","last_became_active":"2024-03-28T23:30:58.278585+0000","last_became_peered":"2024-03-28T23:30:58.278585+0000","last_unstale":"2024-03-28T23:30:58.285582+0000","last_undegraded":"2024-03-28T23:30:58.285582+0000","last_fullsized":"2024-03-28T23:30:58.285582+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:42:55.770179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280211+0000","last_change":"2024-03-28T23:30:57.275078+0000","last_active":"2024-03-28T23:30:59.280211+0000","last_peered":"2024-03-28T23:30:59.280211+0000","last_clean":"2024-03-28T23:30:59.280211+0000","last_became_active":"2024-03-28T23:30:57.274771+0000","last_became_peered":"2024-03-28T23:30:57.274771+0000","last_unstale":"2024-03-28T23:30:59.280211+0000","last_undegraded":"2024-03-28T23:30:59.280211+0000","last_fullsized":"2024-03-28T23:30:59.280211+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:54:09.618251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.293008+0000","last_change":"2024-03-28T23:30:58.288797+0000","last_active":"2024-03-28T23:30:58.293008+0000","last_peered":"2024-03-28T23:30:58.293008+0000","last_clean":"2024-03-28T23:30:58.293008+0000","last_became_active":"2024-03-28T23:30:58.288096+0000","last_became_peered":"2024-03-28T23:30:58.288096+0000","last_unstale":"2024-03-28T23:30:58.293008+0000","last_undegraded":"2024-03-28T23:30:58.293008+0000","last_fullsized":"2024-03-28T23:30:58.293008+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:05.426156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.289459+0000","last_change":"2024-03-28T23:30:57.272696+0000","last_active":"2024-03-28T23:30:58.289459+0000","last_peered":"2024-03-28T23:30:58.289459+0000","last_clean":"2024-03-28T23:30:58.289459+0000","last_became_active":"2024-03-28T23:30:57.272137+0000","last_became_peered":"2024-03-28T23:30:57.272137+0000","last_unstale":"2024-03-28T23:30:58.289459+0000","last_undegraded":"2024-03-28T23:30:58.289459+0000","last_fullsized":"2024-03-28T23:30:58.289459+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:03:14.091392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280189+0000","last_change":"2024-03-28T23:30:58.278290+0000","last_active":"2024-03-28T23:30:59.280189+0000","last_peered":"2024-03-28T23:30:59.280189+0000","last_clean":"2024-03-28T23:30:59.280189+0000","last_became_active":"2024-03-28T23:30:58.277768+0000","last_became_peered":"2024-03-28T23:30:58.277768+0000","last_unstale":"2024-03-28T23:30:59.280189+0000","last_undegraded":"2024-03-28T23:30:59.280189+0000","last_fullsized":"2024-03-28T23:30:59.280189+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:36:53.956712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280378+0000","last_change":"2024-03-28T23:30:57.262449+0000","last_active":"2024-03-28T23:30:59.280378+0000","last_peered":"2024-03-28T23:30:59.280378+0000","last_clean":"2024-03-28T23:30:59.280378+0000","last_became_active":"2024-03-28T23:30:57.262160+0000","last_became_peered":"2024-03-28T23:30:57.262160+0000","last_unstale":"2024-03-28T23:30:59.280378+0000","last_undegraded":"2024-03-28T23:30:59.280378+0000","last_fullsized":"2024-03-28T23:30:59.280378+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:29:57.017794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.278610+0000","last_change":"2024-03-28T23:30:58.271365+0000","last_active":"2024-03-28T23:30:58.278610+0000","last_peered":"2024-03-28T23:30:58.278610+0000","last_clean":"2024-03-28T23:30:58.278610+0000","last_became_active":"2024-03-28T23:30:58.271109+0000","last_became_peered":"2024-03-28T23:30:58.271109+0000","last_unstale":"2024-03-28T23:30:58.278610+0000","last_undegraded":"2024-03-28T23:30:58.278610+0000","last_fullsized":"2024-03-28T23:30:58.278610+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:39:35.383747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.266176+0000","last_change":"2024-03-28T23:30:57.272287+0000","last_active":"2024-03-28T23:30:58.266176+0000","last_peered":"2024-03-28T23:30:58.266176+0000","last_clean":"2024-03-28T23:30:58.266176+0000","last_became_active":"2024-03-28T23:30:57.271795+0000","last_became_peered":"2024-03-28T23:30:57.271795+0000","last_unstale":"2024-03-28T23:30:58.266176+0000","last_undegraded":"2024-03-28T23:30:58.266176+0000","last_fullsized":"2024-03-28T23:30:58.266176+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:38:40.792258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.285136+0000","last_change":"2024-03-28T23:30:58.282976+0000","last_active":"2024-03-28T23:30:58.285136+0000","last_peered":"2024-03-28T23:30:58.285136+0000","last_clean":"2024-03-28T23:30:58.285136+0000","last_became_active":"2024-03-28T23:30:58.282698+0000","last_became_peered":"2024-03-28T23:30:58.282698+0000","last_unstale":"2024-03-28T23:30:58.285136+0000","last_undegraded":"2024-03-28T23:30:58.285136+0000","last_fullsized":"2024-03-28T23:30:58.285136+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:21:53.272287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.267042+0000","last_change":"2024-03-28T23:30:57.274772+0000","last_active":"2024-03-28T23:30:58.267042+0000","last_peered":"2024-03-28T23:30:58.267042+0000","last_clean":"2024-03-28T23:30:58.267042+0000","last_became_active":"2024-03-28T23:30:57.274432+0000","last_became_peered":"2024-03-28T23:30:57.274432+0000","last_unstale":"2024-03-28T23:30:58.267042+0000","last_undegraded":"2024-03-28T23:30:58.267042+0000","last_fullsized":"2024-03-28T23:30:58.267042+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:32:34.991496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.276357+0000","last_change":"2024-03-28T23:30:58.273278+0000","last_active":"2024-03-28T23:30:58.276357+0000","last_peered":"2024-03-28T23:30:58.276357+0000","last_clean":"2024-03-28T23:30:58.276357+0000","last_became_active":"2024-03-28T23:30:58.272960+0000","last_became_peered":"2024-03-28T23:30:58.272960+0000","last_unstale":"2024-03-28T23:30:58.276357+0000","last_undegraded":"2024-03-28T23:30:58.276357+0000","last_fullsized":"2024-03-28T23:30:58.276357+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:11.983009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279200+0000","last_change":"2024-03-28T23:30:57.274378+0000","last_active":"2024-03-28T23:30:59.279200+0000","last_peered":"2024-03-28T23:30:59.279200+0000","last_clean":"2024-03-28T23:30:59.279200+0000","last_became_active":"2024-03-28T23:30:57.274122+0000","last_became_peered":"2024-03-28T23:30:57.274122+0000","last_unstale":"2024-03-28T23:30:59.279200+0000","last_undegraded":"2024-03-28T23:30:59.279200+0000","last_fullsized":"2024-03-28T23:30:59.279200+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:58:49.220122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.277841+0000","last_change":"2024-03-28T23:30:58.275485+0000","last_active":"2024-03-28T23:30:58.277841+0000","last_peered":"2024-03-28T23:30:58.277841+0000","last_clean":"2024-03-28T23:30:58.277841+0000","last_became_active":"2024-03-28T23:30:58.275109+0000","last_became_peered":"2024-03-28T23:30:58.275109+0000","last_unstale":"2024-03-28T23:30:58.277841+0000","last_undegraded":"2024-03-28T23:30:58.277841+0000","last_fullsized":"2024-03-28T23:30:58.277841+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:52:06.639918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.261786+0000","last_change":"2024-03-28T23:30:57.272248+0000","last_active":"2024-03-28T23:30:58.261786+0000","last_peered":"2024-03-28T23:30:58.261786+0000","last_clean":"2024-03-28T23:30:58.261786+0000","last_became_active":"2024-03-28T23:30:57.271994+0000","last_became_peered":"2024-03-28T23:30:57.271994+0000","last_unstale":"2024-03-28T23:30:58.261786+0000","last_undegraded":"2024-03-28T23:30:58.261786+0000","last_fullsized":"2024-03-28T23:30:58.261786+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:38:40.792258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279759+0000","last_change":"2024-03-28T23:30:58.278854+0000","last_active":"2024-03-28T23:30:59.279759+0000","last_peered":"2024-03-28T23:30:59.279759+0000","last_clean":"2024-03-28T23:30:59.279759+0000","last_became_active":"2024-03-28T23:30:58.278547+0000","last_became_peered":"2024-03-28T23:30:58.278547+0000","last_unstale":"2024-03-28T23:30:59.279759+0000","last_undegraded":"2024-03-28T23:30:59.279759+0000","last_fullsized":"2024-03-28T23:30:59.279759+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:34:06.371040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.269906+0000","last_change":"2024-03-28T23:30:57.271746+0000","last_active":"2024-03-28T23:30:58.269906+0000","last_peered":"2024-03-28T23:30:58.269906+0000","last_clean":"2024-03-28T23:30:58.269906+0000","last_became_active":"2024-03-28T23:30:57.271435+0000","last_became_peered":"2024-03-28T23:30:57.271435+0000","last_unstale":"2024-03-28T23:30:58.269906+0000","last_undegraded":"2024-03-28T23:30:58.269906+0000","last_fullsized":"2024-03-28T23:30:58.269906+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:16:02.614366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.290069+0000","last_change":"2024-03-28T23:30:58.288122+0000","last_active":"2024-03-28T23:30:58.290069+0000","last_peered":"2024-03-28T23:30:58.290069+0000","last_clean":"2024-03-28T23:30:58.290069+0000","last_became_active":"2024-03-28T23:30:58.287768+0000","last_became_peered":"2024-03-28T23:30:58.287768+0000","last_unstale":"2024-03-28T23:30:58.290069+0000","last_undegraded":"2024-03-28T23:30:58.290069+0000","last_fullsized":"2024-03-28T23:30:58.290069+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:34:06.371040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279984+0000","last_change":"2024-03-28T23:30:57.270123+0000","last_active":"2024-03-28T23:30:59.279984+0000","last_peered":"2024-03-28T23:30:59.279984+0000","last_clean":"2024-03-28T23:30:59.279984+0000","last_became_active":"2024-03-28T23:30:57.269887+0000","last_became_peered":"2024-03-28T23:30:57.269887+0000","last_unstale":"2024-03-28T23:30:59.279984+0000","last_undegraded":"2024-03-28T23:30:59.279984+0000","last_fullsized":"2024-03-28T23:30:59.279984+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:16:54.807782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":203,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280001+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:30:59.280001+0000","last_peered":"2024-03-28T23:30:59.280001+0000","last_clean":"2024-03-28T23:30:59.280001+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:30:59.280001+0000","last_undegraded":"2024-03-28T23:30:59.280001+0000","last_fullsized":"2024-03-28T23:30:59.280001+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":507904,"data_stored":775394,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31608,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340232,"statfs":{"total":96636764160,"available":96604397568,"internally_reserved":0,"allocated":430080,"data_stored":183926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":434176,"data_stored":183456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":434176,"data_stored":183948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":442368,"data_stored":184990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27448,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":430080,"data_stored":183414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27448,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":430080,"data_stored":183434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27504,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344336,"statfs":{"total":96636764160,"available":96608600064,"internally_reserved":0,"allocated":495616,"data_stored":773746,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:31:02.258 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:31:02.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.397+0000 7f501ed8a640 1 Processor -- start 2024-03-28T23:31:02.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.397+0000 7f501ed8a640 1 -- start start 2024-03-28T23:31:02.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.398+0000 7f501ed8a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5018152890 0x7f50180c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:02.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.398+0000 7f501ed8a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f50180c8ed0 0x7f50180c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:02.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.398+0000 7f501ed8a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5018158a30 con 0x7f50180c5950 2024-03-28T23:31:02.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.398+0000 7f501ed8a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f50180c8480 con 0x7f5018152890 2024-03-28T23:31:02.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.398+0000 7f501ed8a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5018159640 con 0x7f50180c8ed0 2024-03-28T23:31:02.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.398+0000 7f5017fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f50180c8ed0 0x7f50180c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:02.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.398+0000 7f501cb26640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5018152890 0x7f50180c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:02.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.398+0000 7f5017fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f50180c8ed0 0x7f50180c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:56124/0 (socket says 172.21.15.161:56124) 2024-03-28T23:31:02.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.398+0000 7f5017fff640 1 -- 172.21.15.161:0/111915539 learned_addr learned my addr 172.21.15.161:0/111915539 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:02.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.399+0000 7f501cb26640 1 -- 172.21.15.161:0/111915539 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f50180c8ed0 msgr2=0x7f50180c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:02.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.399+0000 7f501cb26640 1 --2- 172.21.15.161:0/111915539 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f50180c8ed0 0x7f50180c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.399+0000 7f501cb26640 1 -- 172.21.15.161:0/111915539 >> v1:172.21.15.170:6789/0 conn(0x7f50180c5950 legacy=0x7f50181553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:02.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.399+0000 7f501cb26640 1 -- 172.21.15.161:0/111915539 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5018158d10 con 0x7f5018152890 2024-03-28T23:31:02.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.399+0000 7f5017fff640 1 --2- 172.21.15.161:0/111915539 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f50180c8ed0 0x7f50180c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:02.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.399+0000 7f501cb26640 1 --2- 172.21.15.161:0/111915539 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5018152890 0x7f50180c8990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f500c0097a0 tx=0x7f500c02b5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8161da807945494 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:02.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.400+0000 7f50177fe640 1 -- 172.21.15.161:0/111915539 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c037070 con 0x7f5018152890 2024-03-28T23:31:02.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.400+0000 7f50177fe640 1 -- 172.21.15.161:0/111915539 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f500c030710 con 0x7f5018152890 2024-03-28T23:31:02.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.400+0000 7f50177fe640 1 -- 172.21.15.161:0/111915539 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c030ae0 con 0x7f5018152890 2024-03-28T23:31:02.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.400+0000 7f501ed8a640 1 -- 172.21.15.161:0/111915539 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5018152890 msgr2=0x7f50180c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:02.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.400+0000 7f501ed8a640 1 --2- 172.21.15.161:0/111915539 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5018152890 0x7f50180c8990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f500c0097a0 tx=0x7f500c02b5d0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.400+0000 7f501ed8a640 1 -- 172.21.15.161:0/111915539 shutdown_connections 2024-03-28T23:31:02.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.400+0000 7f501ed8a640 1 --2- 172.21.15.161:0/111915539 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f50180c8ed0 0x7f50180c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.400+0000 7f501ed8a640 1 --2- 172.21.15.161:0/111915539 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5018152890 0x7f50180c8990 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.400+0000 7f501ed8a640 1 -- 172.21.15.161:0/111915539 >> 172.21.15.161:0/111915539 conn(0x7f50180ac620 msgr2=0x7f50180aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:02.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.400+0000 7f501ed8a640 1 -- 172.21.15.161:0/111915539 shutdown_connections 2024-03-28T23:31:02.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.400+0000 7f501ed8a640 1 -- 172.21.15.161:0/111915539 wait complete. 2024-03-28T23:31:02.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.402+0000 7f501ed8a640 1 Processor -- start 2024-03-28T23:31:02.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.402+0000 7f501ed8a640 1 -- start start 2024-03-28T23:31:02.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.403+0000 7f501ed8a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50180c5950 0x7f50181eb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:02.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.403+0000 7f501ed8a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f50180c8ed0 0x7f50181e8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:02.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.403+0000 7f501ed8a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5018152890 0x7f501814ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:02.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.403+0000 7f501ed8a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5018152c70 con 0x7f50180c5950 2024-03-28T23:31:02.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.403+0000 7f501ed8a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f50181592d0 con 0x7f50180c8ed0 2024-03-28T23:31:02.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.403+0000 7f501ed8a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5018152df0 con 0x7f5018152890 2024-03-28T23:31:02.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.403+0000 7f501cb26640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50180c5950 0x7f50181eb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:02.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.403+0000 7f501cb26640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50180c5950 0x7f50181eb4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45788/0 (socket says 172.21.15.161:45788) 2024-03-28T23:31:02.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.403+0000 7f501cb26640 1 -- 172.21.15.161:0/2896252336 learned_addr learned my addr 172.21.15.161:0/2896252336 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:02.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.403+0000 7f5017fff640 1 --2- 172.21.15.161:0/2896252336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f50180c8ed0 0x7f50181e8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:02.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.403+0000 7f501d327640 1 --2- 172.21.15.161:0/2896252336 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5018152890 0x7f501814ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:02.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.403+0000 7f501cb26640 1 -- 172.21.15.161:0/2896252336 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5018152890 msgr2=0x7f501814ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:02.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.403+0000 7f501cb26640 1 --2- 172.21.15.161:0/2896252336 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5018152890 0x7f501814ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.403+0000 7f501cb26640 1 -- 172.21.15.161:0/2896252336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f50180c8ed0 msgr2=0x7f50181e8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:02.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.403+0000 7f501cb26640 1 --2- 172.21.15.161:0/2896252336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f50180c8ed0 0x7f50181e8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.403+0000 7f501cb26640 1 -- 172.21.15.161:0/2896252336 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50180ad7f0 con 0x7f50180c5950 2024-03-28T23:31:02.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.403+0000 7f501d327640 1 --2- 172.21.15.161:0/2896252336 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5018152890 0x7f501814ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:02.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.403+0000 7f5017fff640 1 --2- 172.21.15.161:0/2896252336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f50180c8ed0 0x7f50181e8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:02.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.403+0000 7f501cb26640 1 --2- 172.21.15.161:0/2896252336 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50180c5950 0x7f50181eb4c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f500c035040 tx=0x7f500c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:02.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.404+0000 7f50157fa640 1 -- 172.21.15.161:0/2896252336 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c041030 con 0x7f50180c5950 2024-03-28T23:31:02.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.404+0000 7f50157fa640 1 -- 172.21.15.161:0/2896252336 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f500c03c730 con 0x7f50180c5950 2024-03-28T23:31:02.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.404+0000 7f501ed8a640 1 -- 172.21.15.161:0/2896252336 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50181eba00 con 0x7f50180c5950 2024-03-28T23:31:02.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.404+0000 7f50157fa640 1 -- 172.21.15.161:0/2896252336 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c037040 con 0x7f50180c5950 2024-03-28T23:31:02.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.404+0000 7f501ed8a640 1 -- 172.21.15.161:0/2896252336 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50181e8690 con 0x7f50180c5950 2024-03-28T23:31:02.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.405+0000 7f501ed8a640 1 -- 172.21.15.161:0/2896252336 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f501804f4b0 con 0x7f50180c5950 2024-03-28T23:31:02.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.405+0000 7f50157fa640 1 -- 172.21.15.161:0/2896252336 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f500c044070 con 0x7f50180c5950 2024-03-28T23:31:02.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.405+0000 7f50157fa640 1 --2- 172.21.15.161:0/2896252336 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4ffc06b4d0 0x7f4ffc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:02.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.405+0000 7f50157fa640 1 -- 172.21.15.161:0/2896252336 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f500c0abda0 con 0x7f50180c5950 2024-03-28T23:31:02.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.406+0000 7f5017fff640 1 --2- 172.21.15.161:0/2896252336 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4ffc06b4d0 0x7f4ffc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:02.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.406+0000 7f5017fff640 1 --2- 172.21.15.161:0/2896252336 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4ffc06b4d0 0x7f4ffc06d990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f50180ad1f0 tx=0x7f5004008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:02.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.411+0000 7f50157fa640 1 -- 172.21.15.161:0/2896252336 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f50181eba00 con 0x7f50180c5950 2024-03-28T23:31:02.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.582+0000 7f501ed8a640 1 -- 172.21.15.161:0/2896252336 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5018150a40 con 0x7f4ffc06b4d0 2024-03-28T23:31:02.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.594+0000 7f50157fa640 1 -- 172.21.15.161:0/2896252336 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251985 (secure 0 0 0) 0x7f5018150a40 con 0x7f4ffc06b4d0 2024-03-28T23:31:02.598 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:02.599 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:31:02.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.597+0000 7f501ed8a640 1 -- 172.21.15.161:0/2896252336 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4ffc06b4d0 msgr2=0x7f4ffc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:02.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.597+0000 7f501ed8a640 1 --2- 172.21.15.161:0/2896252336 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4ffc06b4d0 0x7f4ffc06d990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f50180ad1f0 tx=0x7f5004008040 comp rx=0 tx=0).stop 2024-03-28T23:31:02.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.597+0000 7f501ed8a640 1 -- 172.21.15.161:0/2896252336 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50180c5950 msgr2=0x7f50181eb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:02.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.597+0000 7f501ed8a640 1 --2- 172.21.15.161:0/2896252336 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50180c5950 0x7f50181eb4c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f500c035040 tx=0x7f500c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.597+0000 7f501ed8a640 1 -- 172.21.15.161:0/2896252336 shutdown_connections 2024-03-28T23:31:02.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.597+0000 7f501ed8a640 1 --2- 172.21.15.161:0/2896252336 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4ffc06b4d0 0x7f4ffc06d990 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.597+0000 7f501ed8a640 1 --2- 172.21.15.161:0/2896252336 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5018152890 0x7f501814ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.597+0000 7f501ed8a640 1 --2- 172.21.15.161:0/2896252336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f50180c8ed0 0x7f50181e8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.597+0000 7f501ed8a640 1 --2- 172.21.15.161:0/2896252336 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50180c5950 0x7f50181eb4c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.597+0000 7f501ed8a640 1 -- 172.21.15.161:0/2896252336 >> 172.21.15.161:0/2896252336 conn(0x7f50180ac620 msgr2=0x7f501806be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:02.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.597+0000 7f501ed8a640 1 -- 172.21.15.161:0/2896252336 shutdown_connections 2024-03-28T23:31:02.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.597+0000 7f501ed8a640 1 -- 172.21.15.161:0/2896252336 wait complete. 2024-03-28T23:31:02.610 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2024-03-28T23:31:01.326242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":84,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":232208,"kb_used_data":3520,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754742512,"statfs":{"total":773094113280,"available":772856332288,"internally_reserved":0,"allocated":3604480,"data_stored":2652308,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.075394"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.283812+0000","last_change":"2024-03-28T23:30:58.276937+0000","last_active":"2024-03-28T23:30:58.283812+0000","last_peered":"2024-03-28T23:30:58.283812+0000","last_clean":"2024-03-28T23:30:58.283812+0000","last_became_active":"2024-03-28T23:30:58.276514+0000","last_became_peered":"2024-03-28T23:30:58.276514+0000","last_unstale":"2024-03-28T23:30:58.283812+0000","last_undegraded":"2024-03-28T23:30:58.283812+0000","last_fullsized":"2024-03-28T23:30:58.283812+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:34:06.371040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.285743+0000","last_change":"2024-03-28T23:30:58.282554+0000","last_active":"2024-03-28T23:30:58.285743+0000","last_peered":"2024-03-28T23:30:58.285743+0000","last_clean":"2024-03-28T23:30:58.285743+0000","last_became_active":"2024-03-28T23:30:58.282301+0000","last_became_peered":"2024-03-28T23:30:58.282301+0000","last_unstale":"2024-03-28T23:30:58.285743+0000","last_undegraded":"2024-03-28T23:30:58.285743+0000","last_fullsized":"2024-03-28T23:30:58.285743+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:36:48.381459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.285802+0000","last_change":"2024-03-28T23:30:58.285256+0000","last_active":"2024-03-28T23:30:58.285802+0000","last_peered":"2024-03-28T23:30:58.285802+0000","last_clean":"2024-03-28T23:30:58.285802+0000","last_became_active":"2024-03-28T23:30:58.284994+0000","last_became_peered":"2024-03-28T23:30:58.284994+0000","last_unstale":"2024-03-28T23:30:58.285802+0000","last_undegraded":"2024-03-28T23:30:58.285802+0000","last_fullsized":"2024-03-28T23:30:58.285802+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:03:27.267088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280479+0000","last_change":"2024-03-28T23:30:58.270738+0000","last_active":"2024-03-28T23:30:59.280479+0000","last_peered":"2024-03-28T23:30:59.280479+0000","last_clean":"2024-03-28T23:30:59.280479+0000","last_became_active":"2024-03-28T23:30:58.270465+0000","last_became_peered":"2024-03-28T23:30:58.270465+0000","last_unstale":"2024-03-28T23:30:59.280479+0000","last_undegraded":"2024-03-28T23:30:59.280479+0000","last_fullsized":"2024-03-28T23:30:59.280479+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:37:38.489135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280501+0000","last_change":"2024-03-28T23:30:58.266623+0000","last_active":"2024-03-28T23:30:59.280501+0000","last_peered":"2024-03-28T23:30:59.280501+0000","last_clean":"2024-03-28T23:30:59.280501+0000","last_became_active":"2024-03-28T23:30:58.266334+0000","last_became_peered":"2024-03-28T23:30:58.266334+0000","last_unstale":"2024-03-28T23:30:59.280501+0000","last_undegraded":"2024-03-28T23:30:59.280501+0000","last_fullsized":"2024-03-28T23:30:59.280501+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:31:37.187370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.285981+0000","last_change":"2024-03-28T23:30:58.280203+0000","last_active":"2024-03-28T23:30:58.285981+0000","last_peered":"2024-03-28T23:30:58.285981+0000","last_clean":"2024-03-28T23:30:58.285981+0000","last_became_active":"2024-03-28T23:30:58.279952+0000","last_became_peered":"2024-03-28T23:30:58.279952+0000","last_unstale":"2024-03-28T23:30:58.285981+0000","last_undegraded":"2024-03-28T23:30:58.285981+0000","last_fullsized":"2024-03-28T23:30:58.285981+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:04.163317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.290009+0000","last_change":"2024-03-28T23:30:58.287596+0000","last_active":"2024-03-28T23:30:58.290009+0000","last_peered":"2024-03-28T23:30:58.290009+0000","last_clean":"2024-03-28T23:30:58.290009+0000","last_became_active":"2024-03-28T23:30:58.287122+0000","last_became_peered":"2024-03-28T23:30:58.287122+0000","last_unstale":"2024-03-28T23:30:58.290009+0000","last_undegraded":"2024-03-28T23:30:58.290009+0000","last_fullsized":"2024-03-28T23:30:58.290009+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:04:49.308170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.293161+0000","last_change":"2024-03-28T23:30:58.290407+0000","last_active":"2024-03-28T23:30:58.293161+0000","last_peered":"2024-03-28T23:30:58.293161+0000","last_clean":"2024-03-28T23:30:58.293161+0000","last_became_active":"2024-03-28T23:30:58.290065+0000","last_became_peered":"2024-03-28T23:30:58.290065+0000","last_unstale":"2024-03-28T23:30:58.293161+0000","last_undegraded":"2024-03-28T23:30:58.293161+0000","last_fullsized":"2024-03-28T23:30:58.293161+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:41:27.549056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.285478+0000","last_change":"2024-03-28T23:30:58.281883+0000","last_active":"2024-03-28T23:30:58.285478+0000","last_peered":"2024-03-28T23:30:58.285478+0000","last_clean":"2024-03-28T23:30:58.285478+0000","last_became_active":"2024-03-28T23:30:58.281618+0000","last_became_peered":"2024-03-28T23:30:58.281618+0000","last_unstale":"2024-03-28T23:30:58.285478+0000","last_undegraded":"2024-03-28T23:30:58.285478+0000","last_fullsized":"2024-03-28T23:30:58.285478+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:05.426156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.288901+0000","last_change":"2024-03-28T23:30:58.279905+0000","last_active":"2024-03-28T23:30:58.288901+0000","last_peered":"2024-03-28T23:30:58.288901+0000","last_clean":"2024-03-28T23:30:58.288901+0000","last_became_active":"2024-03-28T23:30:58.279627+0000","last_became_peered":"2024-03-28T23:30:58.279627+0000","last_unstale":"2024-03-28T23:30:58.288901+0000","last_undegraded":"2024-03-28T23:30:58.288901+0000","last_fullsized":"2024-03-28T23:30:58.288901+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:02:15.637133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.290372+0000","last_change":"2024-03-28T23:30:58.289133+0000","last_active":"2024-03-28T23:30:58.290372+0000","last_peered":"2024-03-28T23:30:58.290372+0000","last_clean":"2024-03-28T23:30:58.290372+0000","last_became_active":"2024-03-28T23:30:58.288634+0000","last_became_peered":"2024-03-28T23:30:58.288634+0000","last_unstale":"2024-03-28T23:30:58.290372+0000","last_undegraded":"2024-03-28T23:30:58.290372+0000","last_fullsized":"2024-03-28T23:30:58.290372+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:38:17.760851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.284973+0000","last_change":"2024-03-28T23:30:58.281639+0000","last_active":"2024-03-28T23:30:58.284973+0000","last_peered":"2024-03-28T23:30:58.284973+0000","last_clean":"2024-03-28T23:30:58.284973+0000","last_became_active":"2024-03-28T23:30:58.276244+0000","last_became_peered":"2024-03-28T23:30:58.276244+0000","last_unstale":"2024-03-28T23:30:58.284973+0000","last_undegraded":"2024-03-28T23:30:58.284973+0000","last_fullsized":"2024-03-28T23:30:58.284973+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:04.163317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.288887+0000","last_change":"2024-03-28T23:30:58.276918+0000","last_active":"2024-03-28T23:30:58.288887+0000","last_peered":"2024-03-28T23:30:58.288887+0000","last_clean":"2024-03-28T23:30:58.288887+0000","last_became_active":"2024-03-28T23:30:58.276211+0000","last_became_peered":"2024-03-28T23:30:58.276211+0000","last_unstale":"2024-03-28T23:30:58.288887+0000","last_undegraded":"2024-03-28T23:30:58.288887+0000","last_fullsized":"2024-03-28T23:30:58.288887+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:02:55.962434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.266633+0000","last_change":"2024-03-28T23:30:58.265193+0000","last_active":"2024-03-28T23:30:58.266633+0000","last_peered":"2024-03-28T23:30:58.266633+0000","last_clean":"2024-03-28T23:30:58.266633+0000","last_became_active":"2024-03-28T23:30:58.264909+0000","last_became_peered":"2024-03-28T23:30:58.264909+0000","last_unstale":"2024-03-28T23:30:58.266633+0000","last_undegraded":"2024-03-28T23:30:58.266633+0000","last_fullsized":"2024-03-28T23:30:58.266633+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:32:35.995508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.289891+0000","last_change":"2024-03-28T23:30:58.286577+0000","last_active":"2024-03-28T23:30:58.289891+0000","last_peered":"2024-03-28T23:30:58.289891+0000","last_clean":"2024-03-28T23:30:58.289891+0000","last_became_active":"2024-03-28T23:30:58.285532+0000","last_became_peered":"2024-03-28T23:30:58.285532+0000","last_unstale":"2024-03-28T23:30:58.289891+0000","last_undegraded":"2024-03-28T23:30:58.289891+0000","last_fullsized":"2024-03-28T23:30:58.289891+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:50:54.095985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279463+0000","last_change":"2024-03-28T23:30:58.284055+0000","last_active":"2024-03-28T23:30:59.279463+0000","last_peered":"2024-03-28T23:30:59.279463+0000","last_clean":"2024-03-28T23:30:59.279463+0000","last_became_active":"2024-03-28T23:30:58.283763+0000","last_became_peered":"2024-03-28T23:30:58.283763+0000","last_unstale":"2024-03-28T23:30:59.279463+0000","last_undegraded":"2024-03-28T23:30:59.279463+0000","last_fullsized":"2024-03-28T23:30:59.279463+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:16.989843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280134+0000","last_change":"2024-03-28T23:30:58.281452+0000","last_active":"2024-03-28T23:30:59.280134+0000","last_peered":"2024-03-28T23:30:59.280134+0000","last_clean":"2024-03-28T23:30:59.280134+0000","last_became_active":"2024-03-28T23:30:58.281176+0000","last_became_peered":"2024-03-28T23:30:58.281176+0000","last_unstale":"2024-03-28T23:30:59.280134+0000","last_undegraded":"2024-03-28T23:30:59.280134+0000","last_fullsized":"2024-03-28T23:30:59.280134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:36:48.381459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279450+0000","last_change":"2024-03-28T23:30:58.266599+0000","last_active":"2024-03-28T23:30:59.279450+0000","last_peered":"2024-03-28T23:30:59.279450+0000","last_clean":"2024-03-28T23:30:59.279450+0000","last_became_active":"2024-03-28T23:30:58.266208+0000","last_became_peered":"2024-03-28T23:30:58.266208+0000","last_unstale":"2024-03-28T23:30:59.279450+0000","last_undegraded":"2024-03-28T23:30:59.279450+0000","last_fullsized":"2024-03-28T23:30:59.279450+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:04.163317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280430+0000","last_change":"2024-03-28T23:30:58.277756+0000","last_active":"2024-03-28T23:30:59.280430+0000","last_peered":"2024-03-28T23:30:59.280430+0000","last_clean":"2024-03-28T23:30:59.280430+0000","last_became_active":"2024-03-28T23:30:58.277498+0000","last_became_peered":"2024-03-28T23:30:58.277498+0000","last_unstale":"2024-03-28T23:30:59.280430+0000","last_undegraded":"2024-03-28T23:30:59.280430+0000","last_fullsized":"2024-03-28T23:30:59.280430+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:50.450151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279356+0000","last_change":"2024-03-28T23:30:58.265431+0000","last_active":"2024-03-28T23:30:59.279356+0000","last_peered":"2024-03-28T23:30:59.279356+0000","last_clean":"2024-03-28T23:30:59.279356+0000","last_became_active":"2024-03-28T23:30:58.265128+0000","last_became_peered":"2024-03-28T23:30:58.265128+0000","last_unstale":"2024-03-28T23:30:59.279356+0000","last_undegraded":"2024-03-28T23:30:59.279356+0000","last_fullsized":"2024-03-28T23:30:59.279356+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:42:28.325629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.288856+0000","last_change":"2024-03-28T23:30:58.276770+0000","last_active":"2024-03-28T23:30:58.288856+0000","last_peered":"2024-03-28T23:30:58.288856+0000","last_clean":"2024-03-28T23:30:58.288856+0000","last_became_active":"2024-03-28T23:30:58.276504+0000","last_became_peered":"2024-03-28T23:30:58.276504+0000","last_unstale":"2024-03-28T23:30:58.288856+0000","last_undegraded":"2024-03-28T23:30:58.288856+0000","last_fullsized":"2024-03-28T23:30:58.288856+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:03:27.267088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.283507+0000","last_change":"2024-03-28T23:30:58.279908+0000","last_active":"2024-03-28T23:30:58.283507+0000","last_peered":"2024-03-28T23:30:58.283507+0000","last_clean":"2024-03-28T23:30:58.283507+0000","last_became_active":"2024-03-28T23:30:58.279640+0000","last_became_peered":"2024-03-28T23:30:58.279640+0000","last_unstale":"2024-03-28T23:30:58.283507+0000","last_undegraded":"2024-03-28T23:30:58.283507+0000","last_fullsized":"2024-03-28T23:30:58.283507+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:04.163317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279011+0000","last_change":"2024-03-28T23:30:58.283847+0000","last_active":"2024-03-28T23:30:59.279011+0000","last_peered":"2024-03-28T23:30:59.279011+0000","last_clean":"2024-03-28T23:30:59.279011+0000","last_became_active":"2024-03-28T23:30:58.283565+0000","last_became_peered":"2024-03-28T23:30:58.283565+0000","last_unstale":"2024-03-28T23:30:59.279011+0000","last_undegraded":"2024-03-28T23:30:59.279011+0000","last_fullsized":"2024-03-28T23:30:59.279011+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:50:54.095985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280195+0000","last_change":"2024-03-28T23:30:58.279115+0000","last_active":"2024-03-28T23:30:59.280195+0000","last_peered":"2024-03-28T23:30:59.280195+0000","last_clean":"2024-03-28T23:30:59.280195+0000","last_became_active":"2024-03-28T23:30:58.278827+0000","last_became_peered":"2024-03-28T23:30:58.278827+0000","last_unstale":"2024-03-28T23:30:59.280195+0000","last_undegraded":"2024-03-28T23:30:59.280195+0000","last_fullsized":"2024-03-28T23:30:59.280195+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:43:32.744982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279522+0000","last_change":"2024-03-28T23:30:58.273441+0000","last_active":"2024-03-28T23:30:59.279522+0000","last_peered":"2024-03-28T23:30:59.279522+0000","last_clean":"2024-03-28T23:30:59.279522+0000","last_became_active":"2024-03-28T23:30:58.273184+0000","last_became_peered":"2024-03-28T23:30:58.273184+0000","last_unstale":"2024-03-28T23:30:59.279522+0000","last_undegraded":"2024-03-28T23:30:59.279522+0000","last_fullsized":"2024-03-28T23:30:59.279522+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:03:15.095404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.293030+0000","last_change":"2024-03-28T23:30:58.288931+0000","last_active":"2024-03-28T23:30:58.293030+0000","last_peered":"2024-03-28T23:30:58.293030+0000","last_clean":"2024-03-28T23:30:58.293030+0000","last_became_active":"2024-03-28T23:30:58.288639+0000","last_became_peered":"2024-03-28T23:30:58.288639+0000","last_unstale":"2024-03-28T23:30:58.293030+0000","last_undegraded":"2024-03-28T23:30:58.293030+0000","last_fullsized":"2024-03-28T23:30:58.293030+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:21:53.272287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279201+0000","last_change":"2024-03-28T23:30:58.275238+0000","last_active":"2024-03-28T23:30:59.279201+0000","last_peered":"2024-03-28T23:30:59.279201+0000","last_clean":"2024-03-28T23:30:59.279201+0000","last_became_active":"2024-03-28T23:30:58.274928+0000","last_became_peered":"2024-03-28T23:30:58.274928+0000","last_unstale":"2024-03-28T23:30:59.279201+0000","last_undegraded":"2024-03-28T23:30:59.279201+0000","last_fullsized":"2024-03-28T23:30:59.279201+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:52:06.639918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.289937+0000","last_change":"2024-03-28T23:30:58.287450+0000","last_active":"2024-03-28T23:30:58.289937+0000","last_peered":"2024-03-28T23:30:58.289937+0000","last_clean":"2024-03-28T23:30:58.289937+0000","last_became_active":"2024-03-28T23:30:58.287158+0000","last_became_peered":"2024-03-28T23:30:58.287158+0000","last_unstale":"2024-03-28T23:30:58.289937+0000","last_undegraded":"2024-03-28T23:30:58.289937+0000","last_fullsized":"2024-03-28T23:30:58.289937+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:16.989843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279539+0000","last_change":"2024-03-28T23:30:58.274865+0000","last_active":"2024-03-28T23:30:59.279539+0000","last_peered":"2024-03-28T23:30:59.279539+0000","last_clean":"2024-03-28T23:30:59.279539+0000","last_became_active":"2024-03-28T23:30:58.274500+0000","last_became_peered":"2024-03-28T23:30:58.274500+0000","last_unstale":"2024-03-28T23:30:59.279539+0000","last_undegraded":"2024-03-28T23:30:59.279539+0000","last_fullsized":"2024-03-28T23:30:59.279539+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:05.426156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279373+0000","last_change":"2024-03-28T23:30:58.276147+0000","last_active":"2024-03-28T23:30:59.279373+0000","last_peered":"2024-03-28T23:30:59.279373+0000","last_clean":"2024-03-28T23:30:59.279373+0000","last_became_active":"2024-03-28T23:30:58.275859+0000","last_became_peered":"2024-03-28T23:30:58.275859+0000","last_unstale":"2024-03-28T23:30:59.279373+0000","last_undegraded":"2024-03-28T23:30:59.279373+0000","last_fullsized":"2024-03-28T23:30:59.279373+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:11.983009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280157+0000","last_change":"2024-03-28T23:30:58.284545+0000","last_active":"2024-03-28T23:30:59.280157+0000","last_peered":"2024-03-28T23:30:59.280157+0000","last_clean":"2024-03-28T23:30:59.280157+0000","last_became_active":"2024-03-28T23:30:58.284258+0000","last_became_peered":"2024-03-28T23:30:58.284258+0000","last_unstale":"2024-03-28T23:30:59.280157+0000","last_undegraded":"2024-03-28T23:30:59.280157+0000","last_fullsized":"2024-03-28T23:30:59.280157+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:41:27.549056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.285528+0000","last_change":"2024-03-28T23:30:58.280540+0000","last_active":"2024-03-28T23:30:58.285528+0000","last_peered":"2024-03-28T23:30:58.285528+0000","last_clean":"2024-03-28T23:30:58.285528+0000","last_became_active":"2024-03-28T23:30:58.280103+0000","last_became_peered":"2024-03-28T23:30:58.280103+0000","last_unstale":"2024-03-28T23:30:58.285528+0000","last_undegraded":"2024-03-28T23:30:58.285528+0000","last_fullsized":"2024-03-28T23:30:58.285528+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:03:15.095404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280012+0000","last_change":"2024-03-28T23:30:58.280692+0000","last_active":"2024-03-28T23:30:59.280012+0000","last_peered":"2024-03-28T23:30:59.280012+0000","last_clean":"2024-03-28T23:30:59.280012+0000","last_became_active":"2024-03-28T23:30:58.280416+0000","last_became_peered":"2024-03-28T23:30:58.280416+0000","last_unstale":"2024-03-28T23:30:59.280012+0000","last_undegraded":"2024-03-28T23:30:59.280012+0000","last_fullsized":"2024-03-28T23:30:59.280012+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:42:55.770179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"37'4","reported_seq":18,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279506+0000","last_change":"2024-03-28T23:30:57.272785+0000","last_active":"2024-03-28T23:30:59.279506+0000","last_peered":"2024-03-28T23:30:59.279506+0000","last_clean":"2024-03-28T23:30:59.279506+0000","last_became_active":"2024-03-28T23:30:57.272546+0000","last_became_peered":"2024-03-28T23:30:57.272546+0000","last_unstale":"2024-03-28T23:30:59.279506+0000","last_undegraded":"2024-03-28T23:30:59.279506+0000","last_fullsized":"2024-03-28T23:30:59.279506+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:43:31.740970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.293043+0000","last_change":"2024-03-28T23:30:58.290118+0000","last_active":"2024-03-28T23:30:58.293043+0000","last_peered":"2024-03-28T23:30:58.293043+0000","last_clean":"2024-03-28T23:30:58.293043+0000","last_became_active":"2024-03-28T23:30:58.289839+0000","last_became_peered":"2024-03-28T23:30:58.289839+0000","last_unstale":"2024-03-28T23:30:58.293043+0000","last_undegraded":"2024-03-28T23:30:58.293043+0000","last_fullsized":"2024-03-28T23:30:58.293043+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:03:27.267088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.288455+0000","last_change":"2024-03-28T23:30:57.273444+0000","last_active":"2024-03-28T23:30:58.288455+0000","last_peered":"2024-03-28T23:30:58.288455+0000","last_clean":"2024-03-28T23:30:58.288455+0000","last_became_active":"2024-03-28T23:30:57.273170+0000","last_became_peered":"2024-03-28T23:30:57.273170+0000","last_unstale":"2024-03-28T23:30:58.288455+0000","last_undegraded":"2024-03-28T23:30:58.288455+0000","last_fullsized":"2024-03-28T23:30:58.288455+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:16:54.807782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.265869+0000","last_change":"2024-03-28T23:30:58.264617+0000","last_active":"2024-03-28T23:30:58.265869+0000","last_peered":"2024-03-28T23:30:58.265869+0000","last_clean":"2024-03-28T23:30:58.265869+0000","last_became_active":"2024-03-28T23:30:58.264329+0000","last_became_peered":"2024-03-28T23:30:58.264329+0000","last_unstale":"2024-03-28T23:30:58.265869+0000","last_undegraded":"2024-03-28T23:30:58.265869+0000","last_fullsized":"2024-03-28T23:30:58.265869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:16:55.811794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.281904+0000","last_change":"2024-03-28T23:30:57.272275+0000","last_active":"2024-03-28T23:30:58.281904+0000","last_peered":"2024-03-28T23:30:58.281904+0000","last_clean":"2024-03-28T23:30:58.281904+0000","last_became_active":"2024-03-28T23:30:57.271973+0000","last_became_peered":"2024-03-28T23:30:57.271973+0000","last_unstale":"2024-03-28T23:30:58.281904+0000","last_undegraded":"2024-03-28T23:30:58.281904+0000","last_fullsized":"2024-03-28T23:30:58.281904+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:42:27.321617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.278579+0000","last_change":"2024-03-28T23:30:58.271502+0000","last_active":"2024-03-28T23:30:58.278579+0000","last_peered":"2024-03-28T23:30:58.278579+0000","last_clean":"2024-03-28T23:30:58.278579+0000","last_became_active":"2024-03-28T23:30:58.271132+0000","last_became_peered":"2024-03-28T23:30:58.271132+0000","last_unstale":"2024-03-28T23:30:58.278579+0000","last_undegraded":"2024-03-28T23:30:58.278579+0000","last_fullsized":"2024-03-28T23:30:58.278579+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:50:54.095985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.284655+0000","last_change":"2024-03-28T23:30:57.274899+0000","last_active":"2024-03-28T23:30:58.284655+0000","last_peered":"2024-03-28T23:30:58.284655+0000","last_clean":"2024-03-28T23:30:58.284655+0000","last_became_active":"2024-03-28T23:30:57.274615+0000","last_became_peered":"2024-03-28T23:30:57.274615+0000","last_unstale":"2024-03-28T23:30:58.284655+0000","last_undegraded":"2024-03-28T23:30:58.284655+0000","last_fullsized":"2024-03-28T23:30:58.284655+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:31:36.183358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.278515+0000","last_change":"2024-03-28T23:30:58.274341+0000","last_active":"2024-03-28T23:30:58.278515+0000","last_peered":"2024-03-28T23:30:58.278515+0000","last_clean":"2024-03-28T23:30:58.278515+0000","last_became_active":"2024-03-28T23:30:58.274076+0000","last_became_peered":"2024-03-28T23:30:58.274076+0000","last_unstale":"2024-03-28T23:30:58.278515+0000","last_undegraded":"2024-03-28T23:30:58.278515+0000","last_fullsized":"2024-03-28T23:30:58.278515+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:52:06.639918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.289008+0000","last_change":"2024-03-28T23:30:57.277760+0000","last_active":"2024-03-28T23:30:58.289008+0000","last_peered":"2024-03-28T23:30:58.289008+0000","last_clean":"2024-03-28T23:30:58.289008+0000","last_became_active":"2024-03-28T23:30:57.277521+0000","last_became_peered":"2024-03-28T23:30:57.277521+0000","last_unstale":"2024-03-28T23:30:58.289008+0000","last_undegraded":"2024-03-28T23:30:58.289008+0000","last_fullsized":"2024-03-28T23:30:58.289008+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:42:27.321617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.287305+0000","last_change":"2024-03-28T23:30:58.276623+0000","last_active":"2024-03-28T23:30:58.287305+0000","last_peered":"2024-03-28T23:30:58.287305+0000","last_clean":"2024-03-28T23:30:58.287305+0000","last_became_active":"2024-03-28T23:30:58.276271+0000","last_became_peered":"2024-03-28T23:30:58.276271+0000","last_unstale":"2024-03-28T23:30:58.287305+0000","last_undegraded":"2024-03-28T23:30:58.287305+0000","last_fullsized":"2024-03-28T23:30:58.287305+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:21:53.272287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279916+0000","last_change":"2024-03-28T23:30:57.269206+0000","last_active":"2024-03-28T23:30:59.279916+0000","last_peered":"2024-03-28T23:30:59.279916+0000","last_clean":"2024-03-28T23:30:59.279916+0000","last_became_active":"2024-03-28T23:30:57.268931+0000","last_became_peered":"2024-03-28T23:30:57.268931+0000","last_unstale":"2024-03-28T23:30:59.279916+0000","last_undegraded":"2024-03-28T23:30:59.279916+0000","last_fullsized":"2024-03-28T23:30:59.279916+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:32:34.991496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.288066+0000","last_change":"2024-03-28T23:30:58.284335+0000","last_active":"2024-03-28T23:30:58.288066+0000","last_peered":"2024-03-28T23:30:58.288066+0000","last_clean":"2024-03-28T23:30:58.288066+0000","last_became_active":"2024-03-28T23:30:58.284074+0000","last_became_peered":"2024-03-28T23:30:58.284074+0000","last_unstale":"2024-03-28T23:30:58.288066+0000","last_undegraded":"2024-03-28T23:30:58.288066+0000","last_fullsized":"2024-03-28T23:30:58.288066+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:03:15.095404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.268717+0000","last_change":"2024-03-28T23:30:57.270537+0000","last_active":"2024-03-28T23:30:58.268717+0000","last_peered":"2024-03-28T23:30:58.268717+0000","last_clean":"2024-03-28T23:30:58.268717+0000","last_became_active":"2024-03-28T23:30:57.270248+0000","last_became_peered":"2024-03-28T23:30:57.270248+0000","last_unstale":"2024-03-28T23:30:58.268717+0000","last_undegraded":"2024-03-28T23:30:58.268717+0000","last_fullsized":"2024-03-28T23:30:58.268717+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:16:54.807782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.285601+0000","last_change":"2024-03-28T23:30:58.279530+0000","last_active":"2024-03-28T23:30:58.285601+0000","last_peered":"2024-03-28T23:30:58.285601+0000","last_clean":"2024-03-28T23:30:58.285601+0000","last_became_active":"2024-03-28T23:30:58.279273+0000","last_became_peered":"2024-03-28T23:30:58.279273+0000","last_unstale":"2024-03-28T23:30:58.285601+0000","last_undegraded":"2024-03-28T23:30:58.285601+0000","last_fullsized":"2024-03-28T23:30:58.285601+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:23:53.595134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.261415+0000","last_change":"2024-03-28T23:30:57.274429+0000","last_active":"2024-03-28T23:30:58.261415+0000","last_peered":"2024-03-28T23:30:58.261415+0000","last_clean":"2024-03-28T23:30:58.261415+0000","last_became_active":"2024-03-28T23:30:57.274171+0000","last_became_peered":"2024-03-28T23:30:57.274171+0000","last_unstale":"2024-03-28T23:30:58.261415+0000","last_undegraded":"2024-03-28T23:30:58.261415+0000","last_fullsized":"2024-03-28T23:30:58.261415+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:49.446139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280503+0000","last_change":"2024-03-28T23:30:58.274997+0000","last_active":"2024-03-28T23:30:59.280503+0000","last_peered":"2024-03-28T23:30:59.280503+0000","last_clean":"2024-03-28T23:30:59.280503+0000","last_became_active":"2024-03-28T23:30:58.273657+0000","last_became_peered":"2024-03-28T23:30:58.273657+0000","last_unstale":"2024-03-28T23:30:59.280503+0000","last_undegraded":"2024-03-28T23:30:59.280503+0000","last_fullsized":"2024-03-28T23:30:59.280503+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:38:41.796270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.268229+0000","last_change":"2024-03-28T23:30:57.271839+0000","last_active":"2024-03-28T23:30:58.268229+0000","last_peered":"2024-03-28T23:30:58.268229+0000","last_clean":"2024-03-28T23:30:58.268229+0000","last_became_active":"2024-03-28T23:30:57.271504+0000","last_became_peered":"2024-03-28T23:30:57.271504+0000","last_unstale":"2024-03-28T23:30:58.268229+0000","last_undegraded":"2024-03-28T23:30:58.268229+0000","last_fullsized":"2024-03-28T23:30:58.268229+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:02:54.958422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280033+0000","last_change":"2024-03-28T23:30:58.278604+0000","last_active":"2024-03-28T23:30:59.280033+0000","last_peered":"2024-03-28T23:30:59.280033+0000","last_clean":"2024-03-28T23:30:59.280033+0000","last_became_active":"2024-03-28T23:30:58.278160+0000","last_became_peered":"2024-03-28T23:30:58.278160+0000","last_unstale":"2024-03-28T23:30:59.280033+0000","last_undegraded":"2024-03-28T23:30:59.280033+0000","last_fullsized":"2024-03-28T23:30:59.280033+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:23:53.595134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.266735+0000","last_change":"2024-03-28T23:30:57.268764+0000","last_active":"2024-03-28T23:30:58.266735+0000","last_peered":"2024-03-28T23:30:58.266735+0000","last_clean":"2024-03-28T23:30:58.266735+0000","last_became_active":"2024-03-28T23:30:57.268525+0000","last_became_peered":"2024-03-28T23:30:57.268525+0000","last_unstale":"2024-03-28T23:30:58.266735+0000","last_undegraded":"2024-03-28T23:30:58.266735+0000","last_fullsized":"2024-03-28T23:30:58.266735+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:10.978997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.285130+0000","last_change":"2024-03-28T23:30:58.281875+0000","last_active":"2024-03-28T23:30:58.285130+0000","last_peered":"2024-03-28T23:30:58.285130+0000","last_clean":"2024-03-28T23:30:58.285130+0000","last_became_active":"2024-03-28T23:30:58.281594+0000","last_became_peered":"2024-03-28T23:30:58.281594+0000","last_unstale":"2024-03-28T23:30:58.285130+0000","last_undegraded":"2024-03-28T23:30:58.285130+0000","last_fullsized":"2024-03-28T23:30:58.285130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:03:15.095404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"37'3","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.289964+0000","last_change":"2024-03-28T23:30:57.272844+0000","last_active":"2024-03-28T23:30:58.289964+0000","last_peered":"2024-03-28T23:30:58.289964+0000","last_clean":"2024-03-28T23:30:58.289964+0000","last_became_active":"2024-03-28T23:30:57.272178+0000","last_became_peered":"2024-03-28T23:30:57.272178+0000","last_unstale":"2024-03-28T23:30:58.289964+0000","last_undegraded":"2024-03-28T23:30:58.289964+0000","last_fullsized":"2024-03-28T23:30:58.289964+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:03.159305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.282181+0000","last_change":"2024-03-28T23:30:58.279118+0000","last_active":"2024-03-28T23:30:58.282181+0000","last_peered":"2024-03-28T23:30:58.282181+0000","last_clean":"2024-03-28T23:30:58.282181+0000","last_became_active":"2024-03-28T23:30:58.278845+0000","last_became_peered":"2024-03-28T23:30:58.278845+0000","last_unstale":"2024-03-28T23:30:58.282181+0000","last_undegraded":"2024-03-28T23:30:58.282181+0000","last_fullsized":"2024-03-28T23:30:58.282181+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:11.983009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"37'6","reported_seq":18,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.293205+0000","last_change":"2024-03-28T23:30:57.267617+0000","last_active":"2024-03-28T23:30:58.293205+0000","last_peered":"2024-03-28T23:30:58.293205+0000","last_clean":"2024-03-28T23:30:58.293205+0000","last_became_active":"2024-03-28T23:30:57.266847+0000","last_became_peered":"2024-03-28T23:30:57.266847+0000","last_unstale":"2024-03-28T23:30:58.293205+0000","last_undegraded":"2024-03-28T23:30:58.293205+0000","last_fullsized":"2024-03-28T23:30:58.293205+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:02:14.633121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.284376+0000","last_change":"2024-03-28T23:30:58.280388+0000","last_active":"2024-03-28T23:30:58.284376+0000","last_peered":"2024-03-28T23:30:58.284376+0000","last_clean":"2024-03-28T23:30:58.284376+0000","last_became_active":"2024-03-28T23:30:58.280132+0000","last_became_peered":"2024-03-28T23:30:58.280132+0000","last_unstale":"2024-03-28T23:30:58.284376+0000","last_undegraded":"2024-03-28T23:30:58.284376+0000","last_fullsized":"2024-03-28T23:30:58.284376+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:11.983009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.283939+0000","last_change":"2024-03-28T23:30:57.275555+0000","last_active":"2024-03-28T23:30:58.283939+0000","last_peered":"2024-03-28T23:30:58.283939+0000","last_clean":"2024-03-28T23:30:58.283939+0000","last_became_active":"2024-03-28T23:30:57.275232+0000","last_became_peered":"2024-03-28T23:30:57.275232+0000","last_unstale":"2024-03-28T23:30:58.283939+0000","last_undegraded":"2024-03-28T23:30:58.283939+0000","last_fullsized":"2024-03-28T23:30:58.283939+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:03:26.263076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.283839+0000","last_change":"2024-03-28T23:30:58.276149+0000","last_active":"2024-03-28T23:30:58.283839+0000","last_peered":"2024-03-28T23:30:58.283839+0000","last_clean":"2024-03-28T23:30:58.283839+0000","last_became_active":"2024-03-28T23:30:58.275887+0000","last_became_peered":"2024-03-28T23:30:58.275887+0000","last_unstale":"2024-03-28T23:30:58.283839+0000","last_undegraded":"2024-03-28T23:30:58.283839+0000","last_fullsized":"2024-03-28T23:30:58.283839+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:16.989843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.270386+0000","last_change":"2024-03-28T23:30:57.271154+0000","last_active":"2024-03-28T23:30:58.270386+0000","last_peered":"2024-03-28T23:30:58.270386+0000","last_clean":"2024-03-28T23:30:58.270386+0000","last_became_active":"2024-03-28T23:30:57.270870+0000","last_became_peered":"2024-03-28T23:30:57.270870+0000","last_unstale":"2024-03-28T23:30:58.270386+0000","last_undegraded":"2024-03-28T23:30:58.270386+0000","last_fullsized":"2024-03-28T23:30:58.270386+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:52:05.635906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279793+0000","last_change":"2024-03-28T23:30:58.286161+0000","last_active":"2024-03-28T23:30:59.279793+0000","last_peered":"2024-03-28T23:30:59.279793+0000","last_clean":"2024-03-28T23:30:59.279793+0000","last_became_active":"2024-03-28T23:30:58.285876+0000","last_became_peered":"2024-03-28T23:30:58.285876+0000","last_unstale":"2024-03-28T23:30:59.279793+0000","last_undegraded":"2024-03-28T23:30:59.279793+0000","last_fullsized":"2024-03-28T23:30:59.279793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:38:17.760851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.278925+0000","last_change":"2024-03-28T23:30:57.271795+0000","last_active":"2024-03-28T23:30:59.278925+0000","last_peered":"2024-03-28T23:30:59.278925+0000","last_clean":"2024-03-28T23:30:59.278925+0000","last_became_active":"2024-03-28T23:30:57.271536+0000","last_became_peered":"2024-03-28T23:30:57.271536+0000","last_unstale":"2024-03-28T23:30:59.278925+0000","last_undegraded":"2024-03-28T23:30:59.278925+0000","last_fullsized":"2024-03-28T23:30:59.278925+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:14:40.512760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.283782+0000","last_change":"2024-03-28T23:30:58.276523+0000","last_active":"2024-03-28T23:30:58.283782+0000","last_peered":"2024-03-28T23:30:58.283782+0000","last_clean":"2024-03-28T23:30:58.283782+0000","last_became_active":"2024-03-28T23:30:58.276253+0000","last_became_peered":"2024-03-28T23:30:58.276253+0000","last_unstale":"2024-03-28T23:30:58.283782+0000","last_undegraded":"2024-03-28T23:30:58.283782+0000","last_fullsized":"2024-03-28T23:30:58.283782+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:04:49.308170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"37'8","reported_seq":22,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279562+0000","last_change":"2024-03-28T23:30:57.269607+0000","last_active":"2024-03-28T23:30:59.279562+0000","last_peered":"2024-03-28T23:30:59.279562+0000","last_clean":"2024-03-28T23:30:59.279562+0000","last_became_active":"2024-03-28T23:30:57.269342+0000","last_became_peered":"2024-03-28T23:30:57.269342+0000","last_unstale":"2024-03-28T23:30:59.279562+0000","last_undegraded":"2024-03-28T23:30:59.279562+0000","last_fullsized":"2024-03-28T23:30:59.279562+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:52:05.635906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.283966+0000","last_change":"2024-03-28T23:30:58.278202+0000","last_active":"2024-03-28T23:30:58.283966+0000","last_peered":"2024-03-28T23:30:58.283966+0000","last_clean":"2024-03-28T23:30:58.283966+0000","last_became_active":"2024-03-28T23:30:58.277935+0000","last_became_peered":"2024-03-28T23:30:58.277935+0000","last_unstale":"2024-03-28T23:30:58.283966+0000","last_undegraded":"2024-03-28T23:30:58.283966+0000","last_fullsized":"2024-03-28T23:30:58.283966+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:38:17.760851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.289548+0000","last_change":"2024-03-28T23:30:57.271432+0000","last_active":"2024-03-28T23:30:58.289548+0000","last_peered":"2024-03-28T23:30:58.289548+0000","last_clean":"2024-03-28T23:30:58.289548+0000","last_became_active":"2024-03-28T23:30:57.271181+0000","last_became_peered":"2024-03-28T23:30:57.271181+0000","last_unstale":"2024-03-28T23:30:58.289548+0000","last_undegraded":"2024-03-28T23:30:58.289548+0000","last_fullsized":"2024-03-28T23:30:58.289548+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:04.422144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.289542+0000","last_change":"2024-03-28T23:30:58.282817+0000","last_active":"2024-03-28T23:30:58.289542+0000","last_peered":"2024-03-28T23:30:58.289542+0000","last_clean":"2024-03-28T23:30:58.289542+0000","last_became_active":"2024-03-28T23:30:58.282571+0000","last_became_peered":"2024-03-28T23:30:58.282571+0000","last_unstale":"2024-03-28T23:30:58.289542+0000","last_undegraded":"2024-03-28T23:30:58.289542+0000","last_fullsized":"2024-03-28T23:30:58.289542+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:16:03.618378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.289869+0000","last_change":"2024-03-28T23:30:57.273183+0000","last_active":"2024-03-28T23:30:58.289869+0000","last_peered":"2024-03-28T23:30:58.289869+0000","last_clean":"2024-03-28T23:30:58.289869+0000","last_became_active":"2024-03-28T23:30:57.272920+0000","last_became_peered":"2024-03-28T23:30:57.272920+0000","last_unstale":"2024-03-28T23:30:58.289869+0000","last_undegraded":"2024-03-28T23:30:58.289869+0000","last_fullsized":"2024-03-28T23:30:58.289869+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:37:37.485123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.279272+0000","last_change":"2024-03-28T23:30:58.277460+0000","last_active":"2024-03-28T23:30:58.279272+0000","last_peered":"2024-03-28T23:30:58.279272+0000","last_clean":"2024-03-28T23:30:58.279272+0000","last_became_active":"2024-03-28T23:30:58.277204+0000","last_became_peered":"2024-03-28T23:30:58.277204+0000","last_unstale":"2024-03-28T23:30:58.279272+0000","last_undegraded":"2024-03-28T23:30:58.279272+0000","last_fullsized":"2024-03-28T23:30:58.279272+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:42:28.325629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280529+0000","last_change":"2024-03-28T23:30:57.264576+0000","last_active":"2024-03-28T23:30:59.280529+0000","last_peered":"2024-03-28T23:30:59.280529+0000","last_clean":"2024-03-28T23:30:59.280529+0000","last_became_active":"2024-03-28T23:30:57.264271+0000","last_became_peered":"2024-03-28T23:30:57.264271+0000","last_unstale":"2024-03-28T23:30:59.280529+0000","last_undegraded":"2024-03-28T23:30:59.280529+0000","last_fullsized":"2024-03-28T23:30:59.280529+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:09:02.708795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280667+0000","last_change":"2024-03-28T23:30:58.278480+0000","last_active":"2024-03-28T23:30:59.280667+0000","last_peered":"2024-03-28T23:30:59.280667+0000","last_clean":"2024-03-28T23:30:59.280667+0000","last_became_active":"2024-03-28T23:30:58.276636+0000","last_became_peered":"2024-03-28T23:30:58.276636+0000","last_unstale":"2024-03-28T23:30:59.280667+0000","last_undegraded":"2024-03-28T23:30:59.280667+0000","last_fullsized":"2024-03-28T23:30:59.280667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:16:55.811794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.269732+0000","last_change":"2024-03-28T23:30:57.275811+0000","last_active":"2024-03-28T23:30:58.269732+0000","last_peered":"2024-03-28T23:30:58.269732+0000","last_clean":"2024-03-28T23:30:58.269732+0000","last_became_active":"2024-03-28T23:30:57.275515+0000","last_became_peered":"2024-03-28T23:30:57.275515+0000","last_unstale":"2024-03-28T23:30:58.269732+0000","last_undegraded":"2024-03-28T23:30:58.269732+0000","last_fullsized":"2024-03-28T23:30:58.269732+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:32:34.991496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279747+0000","last_change":"2024-03-28T23:30:58.285570+0000","last_active":"2024-03-28T23:30:59.279747+0000","last_peered":"2024-03-28T23:30:59.279747+0000","last_clean":"2024-03-28T23:30:59.279747+0000","last_became_active":"2024-03-28T23:30:58.285243+0000","last_became_peered":"2024-03-28T23:30:58.285243+0000","last_unstale":"2024-03-28T23:30:59.279747+0000","last_undegraded":"2024-03-28T23:30:59.279747+0000","last_fullsized":"2024-03-28T23:30:59.279747+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:04:49.308170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280313+0000","last_change":"2024-03-28T23:30:57.277719+0000","last_active":"2024-03-28T23:30:59.280313+0000","last_peered":"2024-03-28T23:30:59.280313+0000","last_clean":"2024-03-28T23:30:59.280313+0000","last_became_active":"2024-03-28T23:30:57.277396+0000","last_became_peered":"2024-03-28T23:30:57.277396+0000","last_unstale":"2024-03-28T23:30:59.280313+0000","last_undegraded":"2024-03-28T23:30:59.280313+0000","last_fullsized":"2024-03-28T23:30:59.280313+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:14:08.521862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.280815+0000","last_change":"2024-03-28T23:30:58.279179+0000","last_active":"2024-03-28T23:30:58.280815+0000","last_peered":"2024-03-28T23:30:58.280815+0000","last_clean":"2024-03-28T23:30:58.280815+0000","last_became_active":"2024-03-28T23:30:58.278881+0000","last_became_peered":"2024-03-28T23:30:58.278881+0000","last_unstale":"2024-03-28T23:30:58.280815+0000","last_undegraded":"2024-03-28T23:30:58.280815+0000","last_fullsized":"2024-03-28T23:30:58.280815+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:42:28.325629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280289+0000","last_change":"2024-03-28T23:30:57.269324+0000","last_active":"2024-03-28T23:30:59.280289+0000","last_peered":"2024-03-28T23:30:59.280289+0000","last_clean":"2024-03-28T23:30:59.280289+0000","last_became_active":"2024-03-28T23:30:57.269043+0000","last_became_peered":"2024-03-28T23:30:57.269043+0000","last_unstale":"2024-03-28T23:30:59.280289+0000","last_undegraded":"2024-03-28T23:30:59.280289+0000","last_fullsized":"2024-03-28T23:30:59.280289+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:56:10.171229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.289223+0000","last_change":"2024-03-28T23:30:58.284238+0000","last_active":"2024-03-28T23:30:58.289223+0000","last_peered":"2024-03-28T23:30:58.289223+0000","last_clean":"2024-03-28T23:30:58.289223+0000","last_became_active":"2024-03-28T23:30:58.283967+0000","last_became_peered":"2024-03-28T23:30:58.283967+0000","last_unstale":"2024-03-28T23:30:58.289223+0000","last_undegraded":"2024-03-28T23:30:58.289223+0000","last_fullsized":"2024-03-28T23:30:58.289223+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:39:35.383747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279839+0000","last_change":"2024-03-28T23:30:57.266767+0000","last_active":"2024-03-28T23:30:59.279839+0000","last_peered":"2024-03-28T23:30:59.279839+0000","last_clean":"2024-03-28T23:30:59.279839+0000","last_became_active":"2024-03-28T23:30:57.266511+0000","last_became_peered":"2024-03-28T23:30:57.266511+0000","last_unstale":"2024-03-28T23:30:59.279839+0000","last_undegraded":"2024-03-28T23:30:59.279839+0000","last_fullsized":"2024-03-28T23:30:59.279839+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:38:40.792258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.285582+0000","last_change":"2024-03-28T23:30:58.278844+0000","last_active":"2024-03-28T23:30:58.285582+0000","last_peered":"2024-03-28T23:30:58.285582+0000","last_clean":"2024-03-28T23:30:58.285582+0000","last_became_active":"2024-03-28T23:30:58.278585+0000","last_became_peered":"2024-03-28T23:30:58.278585+0000","last_unstale":"2024-03-28T23:30:58.285582+0000","last_undegraded":"2024-03-28T23:30:58.285582+0000","last_fullsized":"2024-03-28T23:30:58.285582+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:42:55.770179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280211+0000","last_change":"2024-03-28T23:30:57.275078+0000","last_active":"2024-03-28T23:30:59.280211+0000","last_peered":"2024-03-28T23:30:59.280211+0000","last_clean":"2024-03-28T23:30:59.280211+0000","last_became_active":"2024-03-28T23:30:57.274771+0000","last_became_peered":"2024-03-28T23:30:57.274771+0000","last_unstale":"2024-03-28T23:30:59.280211+0000","last_undegraded":"2024-03-28T23:30:59.280211+0000","last_fullsized":"2024-03-28T23:30:59.280211+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:54:09.618251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.293008+0000","last_change":"2024-03-28T23:30:58.288797+0000","last_active":"2024-03-28T23:30:58.293008+0000","last_peered":"2024-03-28T23:30:58.293008+0000","last_clean":"2024-03-28T23:30:58.293008+0000","last_became_active":"2024-03-28T23:30:58.288096+0000","last_became_peered":"2024-03-28T23:30:58.288096+0000","last_unstale":"2024-03-28T23:30:58.293008+0000","last_undegraded":"2024-03-28T23:30:58.293008+0000","last_fullsized":"2024-03-28T23:30:58.293008+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:05.426156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.289459+0000","last_change":"2024-03-28T23:30:57.272696+0000","last_active":"2024-03-28T23:30:58.289459+0000","last_peered":"2024-03-28T23:30:58.289459+0000","last_clean":"2024-03-28T23:30:58.289459+0000","last_became_active":"2024-03-28T23:30:57.272137+0000","last_became_peered":"2024-03-28T23:30:57.272137+0000","last_unstale":"2024-03-28T23:30:58.289459+0000","last_undegraded":"2024-03-28T23:30:58.289459+0000","last_fullsized":"2024-03-28T23:30:58.289459+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:03:14.091392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280189+0000","last_change":"2024-03-28T23:30:58.278290+0000","last_active":"2024-03-28T23:30:59.280189+0000","last_peered":"2024-03-28T23:30:59.280189+0000","last_clean":"2024-03-28T23:30:59.280189+0000","last_became_active":"2024-03-28T23:30:58.277768+0000","last_became_peered":"2024-03-28T23:30:58.277768+0000","last_unstale":"2024-03-28T23:30:59.280189+0000","last_undegraded":"2024-03-28T23:30:59.280189+0000","last_fullsized":"2024-03-28T23:30:59.280189+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:36:53.956712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280378+0000","last_change":"2024-03-28T23:30:57.262449+0000","last_active":"2024-03-28T23:30:59.280378+0000","last_peered":"2024-03-28T23:30:59.280378+0000","last_clean":"2024-03-28T23:30:59.280378+0000","last_became_active":"2024-03-28T23:30:57.262160+0000","last_became_peered":"2024-03-28T23:30:57.262160+0000","last_unstale":"2024-03-28T23:30:59.280378+0000","last_undegraded":"2024-03-28T23:30:59.280378+0000","last_fullsized":"2024-03-28T23:30:59.280378+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:29:57.017794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.278610+0000","last_change":"2024-03-28T23:30:58.271365+0000","last_active":"2024-03-28T23:30:58.278610+0000","last_peered":"2024-03-28T23:30:58.278610+0000","last_clean":"2024-03-28T23:30:58.278610+0000","last_became_active":"2024-03-28T23:30:58.271109+0000","last_became_peered":"2024-03-28T23:30:58.271109+0000","last_unstale":"2024-03-28T23:30:58.278610+0000","last_undegraded":"2024-03-28T23:30:58.278610+0000","last_fullsized":"2024-03-28T23:30:58.278610+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:39:35.383747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.266176+0000","last_change":"2024-03-28T23:30:57.272287+0000","last_active":"2024-03-28T23:30:58.266176+0000","last_peered":"2024-03-28T23:30:58.266176+0000","last_clean":"2024-03-28T23:30:58.266176+0000","last_became_active":"2024-03-28T23:30:57.271795+0000","last_became_peered":"2024-03-28T23:30:57.271795+0000","last_unstale":"2024-03-28T23:30:58.266176+0000","last_undegraded":"2024-03-28T23:30:58.266176+0000","last_fullsized":"2024-03-28T23:30:58.266176+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:38:40.792258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.285136+0000","last_change":"2024-03-28T23:30:58.282976+0000","last_active":"2024-03-28T23:30:58.285136+0000","last_peered":"2024-03-28T23:30:58.285136+0000","last_clean":"2024-03-28T23:30:58.285136+0000","last_became_active":"2024-03-28T23:30:58.282698+0000","last_became_peered":"2024-03-28T23:30:58.282698+0000","last_unstale":"2024-03-28T23:30:58.285136+0000","last_undegraded":"2024-03-28T23:30:58.285136+0000","last_fullsized":"2024-03-28T23:30:58.285136+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:21:53.272287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.267042+0000","last_change":"2024-03-28T23:30:57.274772+0000","last_active":"2024-03-28T23:30:58.267042+0000","last_peered":"2024-03-28T23:30:58.267042+0000","last_clean":"2024-03-28T23:30:58.267042+0000","last_became_active":"2024-03-28T23:30:57.274432+0000","last_became_peered":"2024-03-28T23:30:57.274432+0000","last_unstale":"2024-03-28T23:30:58.267042+0000","last_undegraded":"2024-03-28T23:30:58.267042+0000","last_fullsized":"2024-03-28T23:30:58.267042+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:32:34.991496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.276357+0000","last_change":"2024-03-28T23:30:58.273278+0000","last_active":"2024-03-28T23:30:58.276357+0000","last_peered":"2024-03-28T23:30:58.276357+0000","last_clean":"2024-03-28T23:30:58.276357+0000","last_became_active":"2024-03-28T23:30:58.272960+0000","last_became_peered":"2024-03-28T23:30:58.272960+0000","last_unstale":"2024-03-28T23:30:58.276357+0000","last_undegraded":"2024-03-28T23:30:58.276357+0000","last_fullsized":"2024-03-28T23:30:58.276357+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:11.983009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279200+0000","last_change":"2024-03-28T23:30:57.274378+0000","last_active":"2024-03-28T23:30:59.279200+0000","last_peered":"2024-03-28T23:30:59.279200+0000","last_clean":"2024-03-28T23:30:59.279200+0000","last_became_active":"2024-03-28T23:30:57.274122+0000","last_became_peered":"2024-03-28T23:30:57.274122+0000","last_unstale":"2024-03-28T23:30:59.279200+0000","last_undegraded":"2024-03-28T23:30:59.279200+0000","last_fullsized":"2024-03-28T23:30:59.279200+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:58:49.220122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.277841+0000","last_change":"2024-03-28T23:30:58.275485+0000","last_active":"2024-03-28T23:30:58.277841+0000","last_peered":"2024-03-28T23:30:58.277841+0000","last_clean":"2024-03-28T23:30:58.277841+0000","last_became_active":"2024-03-28T23:30:58.275109+0000","last_became_peered":"2024-03-28T23:30:58.275109+0000","last_unstale":"2024-03-28T23:30:58.277841+0000","last_undegraded":"2024-03-28T23:30:58.277841+0000","last_fullsized":"2024-03-28T23:30:58.277841+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:52:06.639918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.261786+0000","last_change":"2024-03-28T23:30:57.272248+0000","last_active":"2024-03-28T23:30:58.261786+0000","last_peered":"2024-03-28T23:30:58.261786+0000","last_clean":"2024-03-28T23:30:58.261786+0000","last_became_active":"2024-03-28T23:30:57.271994+0000","last_became_peered":"2024-03-28T23:30:57.271994+0000","last_unstale":"2024-03-28T23:30:58.261786+0000","last_undegraded":"2024-03-28T23:30:58.261786+0000","last_fullsized":"2024-03-28T23:30:58.261786+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:38:40.792258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279759+0000","last_change":"2024-03-28T23:30:58.278854+0000","last_active":"2024-03-28T23:30:59.279759+0000","last_peered":"2024-03-28T23:30:59.279759+0000","last_clean":"2024-03-28T23:30:59.279759+0000","last_became_active":"2024-03-28T23:30:58.278547+0000","last_became_peered":"2024-03-28T23:30:58.278547+0000","last_unstale":"2024-03-28T23:30:59.279759+0000","last_undegraded":"2024-03-28T23:30:59.279759+0000","last_fullsized":"2024-03-28T23:30:59.279759+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:34:06.371040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.269906+0000","last_change":"2024-03-28T23:30:57.271746+0000","last_active":"2024-03-28T23:30:58.269906+0000","last_peered":"2024-03-28T23:30:58.269906+0000","last_clean":"2024-03-28T23:30:58.269906+0000","last_became_active":"2024-03-28T23:30:57.271435+0000","last_became_peered":"2024-03-28T23:30:57.271435+0000","last_unstale":"2024-03-28T23:30:58.269906+0000","last_undegraded":"2024-03-28T23:30:58.269906+0000","last_fullsized":"2024-03-28T23:30:58.269906+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:16:02.614366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-28T23:30:58.290069+0000","last_change":"2024-03-28T23:30:58.288122+0000","last_active":"2024-03-28T23:30:58.290069+0000","last_peered":"2024-03-28T23:30:58.290069+0000","last_clean":"2024-03-28T23:30:58.290069+0000","last_became_active":"2024-03-28T23:30:58.287768+0000","last_became_peered":"2024-03-28T23:30:58.287768+0000","last_unstale":"2024-03-28T23:30:58.290069+0000","last_undegraded":"2024-03-28T23:30:58.290069+0000","last_fullsized":"2024-03-28T23:30:58.290069+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:57.250798+0000","last_clean_scrub_stamp":"2024-03-28T23:30:57.250798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:34:06.371040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.279984+0000","last_change":"2024-03-28T23:30:57.270123+0000","last_active":"2024-03-28T23:30:59.279984+0000","last_peered":"2024-03-28T23:30:59.279984+0000","last_clean":"2024-03-28T23:30:59.279984+0000","last_became_active":"2024-03-28T23:30:57.269887+0000","last_became_peered":"2024-03-28T23:30:57.269887+0000","last_unstale":"2024-03-28T23:30:59.279984+0000","last_undegraded":"2024-03-28T23:30:59.279984+0000","last_fullsized":"2024-03-28T23:30:59.279984+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:30:56.246786+0000","last_clean_scrub_stamp":"2024-03-28T23:30:56.246786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:16:54.807782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":203,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-28T23:30:59.280001+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:30:59.280001+0000","last_peered":"2024-03-28T23:30:59.280001+0000","last_clean":"2024-03-28T23:30:59.280001+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:30:59.280001+0000","last_undegraded":"2024-03-28T23:30:59.280001+0000","last_fullsized":"2024-03-28T23:30:59.280001+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":507904,"data_stored":775394,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31608,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340232,"statfs":{"total":96636764160,"available":96604397568,"internally_reserved":0,"allocated":430080,"data_stored":183926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":434176,"data_stored":183456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":434176,"data_stored":183948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":442368,"data_stored":184990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27448,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":430080,"data_stored":183414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27448,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":430080,"data_stored":183434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27504,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344336,"statfs":{"total":96636764160,"available":96608600064,"internally_reserved":0,"allocated":495616,"data_stored":773746,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:31:02.614 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T23:31:02.614 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-28T23:31:02.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.757+0000 7f199e92c640 1 Processor -- start 2024-03-28T23:31:02.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.758+0000 7f199e92c640 1 -- start start 2024-03-28T23:31:02.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.759+0000 7f199e92c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f19980c6750 0x7f199806bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:02.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.759+0000 7f199e92c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1998152db0 con 0x7f19980c88b0 2024-03-28T23:31:02.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.759+0000 7f199e92c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19980c8d40 con 0x7f19981529d0 2024-03-28T23:31:02.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.759+0000 7f199e92c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f19980c7210 con 0x7f19980c6750 2024-03-28T23:31:02.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.759+0000 7f19977fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f19980c6750 0x7f199806bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:02.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.759+0000 7f199cec9640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f19980c88b0 0x7f19980c54f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41908/0 (socket says 172.21.15.161:41908) 2024-03-28T23:31:02.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.759+0000 7f199cec9640 1 -- 172.21.15.161:0/168889526 learned_addr learned my addr 172.21.15.161:0/168889526 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:02.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.759+0000 7f19977fe640 1 -- 172.21.15.161:0/168889526 >> v1:172.21.15.170:6789/0 conn(0x7f19981529d0 legacy=0x7f1998155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:02.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.759+0000 7f19977fe640 1 -- 172.21.15.161:0/168889526 >> v1:172.21.15.161:6789/0 conn(0x7f19980c88b0 legacy=0x7f19980c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:02.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.759+0000 7f1996ffd640 1 -- 172.21.15.161:0/168889526 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2561124594 0 0) 0x7f1998152db0 con 0x7f19980c88b0 2024-03-28T23:31:02.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.759+0000 7f19977fe640 1 -- 172.21.15.161:0/168889526 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19980c6c50 con 0x7f19980c6750 2024-03-28T23:31:02.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.760+0000 7f19977fe640 1 --2- 172.21.15.161:0/168889526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19980c6750 0x7f199806bb60 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f19880097a0 tx=0x7f198802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d256fceb50ab564 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:02.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.760+0000 7f1996ffd640 1 -- 172.21.15.161:0/168889526 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1988034070 con 0x7f19980c6750 2024-03-28T23:31:02.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.760+0000 7f1996ffd640 1 -- 172.21.15.161:0/168889526 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1988030610 con 0x7f19980c6750 2024-03-28T23:31:02.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.760+0000 7f1996ffd640 1 -- 172.21.15.161:0/168889526 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19880309e0 con 0x7f19980c6750 2024-03-28T23:31:02.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.761+0000 7f199e92c640 1 -- 172.21.15.161:0/168889526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19980c6750 msgr2=0x7f199806bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:02.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.761+0000 7f199e92c640 1 --2- 172.21.15.161:0/168889526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19980c6750 0x7f199806bb60 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f19880097a0 tx=0x7f198802b960 comp rx=0 tx=0).stop 2024-03-28T23:31:02.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.761+0000 7f199e92c640 1 -- 172.21.15.161:0/168889526 shutdown_connections 2024-03-28T23:31:02.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.761+0000 7f199e92c640 1 --2- 172.21.15.161:0/168889526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19980c6750 0x7f199806bb60 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.761+0000 7f199e92c640 1 -- 172.21.15.161:0/168889526 >> 172.21.15.161:0/168889526 conn(0x7f19980c7430 msgr2=0x7f19980c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:02.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.761+0000 7f199e92c640 1 -- 172.21.15.161:0/168889526 shutdown_connections 2024-03-28T23:31:02.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.761+0000 7f199e92c640 1 -- 172.21.15.161:0/168889526 wait complete. 2024-03-28T23:31:02.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.762+0000 7f199e92c640 1 Processor -- start 2024-03-28T23:31:02.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.762+0000 7f199e92c640 1 -- start start 2024-03-28T23:31:02.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.763+0000 7f199e92c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19980c6750 0x7f1998158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:02.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.763+0000 7f199e92c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19980c88b0 0x7f199814eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:02.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.763+0000 7f199e92c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f19981529d0 0x7f199814a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:02.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.763+0000 7f199e92c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1998155710 con 0x7f19980c6750 2024-03-28T23:31:02.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.763+0000 7f199e92c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1998155430 con 0x7f19981529d0 2024-03-28T23:31:02.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.763+0000 7f199e92c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f19980aa160 con 0x7f19980c88b0 2024-03-28T23:31:02.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.763+0000 7f1997fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19980c6750 0x7f1998158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:02.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.763+0000 7f1997fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19980c6750 0x7f1998158ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45820/0 (socket says 172.21.15.161:45820) 2024-03-28T23:31:02.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.763+0000 7f1997fff640 1 -- 172.21.15.161:0/4145867143 learned_addr learned my addr 172.21.15.161:0/4145867143 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:02.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.763+0000 7f19977fe640 1 --2- 172.21.15.161:0/4145867143 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19980c88b0 0x7f199814eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:02.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.763+0000 7f199cec9640 1 --2- 172.21.15.161:0/4145867143 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f19981529d0 0x7f199814a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:02.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.763+0000 7f1997fff640 1 -- 172.21.15.161:0/4145867143 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19980c88b0 msgr2=0x7f199814eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:02.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.763+0000 7f1997fff640 1 --2- 172.21.15.161:0/4145867143 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19980c88b0 0x7f199814eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.763+0000 7f1997fff640 1 -- 172.21.15.161:0/4145867143 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f19981529d0 msgr2=0x7f199814a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:02.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.763+0000 7f1997fff640 1 --2- 172.21.15.161:0/4145867143 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f19981529d0 0x7f199814a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:02.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.763+0000 7f1997fff640 1 -- 172.21.15.161:0/4145867143 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f199814b9b0 con 0x7f19980c6750 2024-03-28T23:31:02.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.763+0000 7f1997fff640 1 --2- 172.21.15.161:0/4145867143 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19980c6750 0x7f1998158ad0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f198c00c400 tx=0x7f198c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:02.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.763+0000 7f1994ff9640 1 -- 172.21.15.161:0/4145867143 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198c017070 con 0x7f19980c6750 2024-03-28T23:31:02.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.763+0000 7f1994ff9640 1 -- 172.21.15.161:0/4145867143 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f198c0052d0 con 0x7f19980c6750 2024-03-28T23:31:02.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.763+0000 7f1994ff9640 1 -- 172.21.15.161:0/4145867143 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198c003830 con 0x7f19980c6750 2024-03-28T23:31:02.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.763+0000 7f199e92c640 1 -- 172.21.15.161:0/4145867143 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f199814b6d0 con 0x7f19980c6750 2024-03-28T23:31:02.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.763+0000 7f199e92c640 1 -- 172.21.15.161:0/4145867143 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f199814da00 con 0x7f19980c6750 2024-03-28T23:31:02.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.765+0000 7f199e92c640 1 -- 172.21.15.161:0/4145867143 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f199804f4b0 con 0x7f19980c6750 2024-03-28T23:31:02.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.765+0000 7f1994ff9640 1 -- 172.21.15.161:0/4145867143 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f198c01a070 con 0x7f19980c6750 2024-03-28T23:31:02.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.765+0000 7f1994ff9640 1 --2- 172.21.15.161:0/4145867143 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f198406b500 0x7f198406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:02.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.765+0000 7f19977fe640 1 --2- 172.21.15.161:0/4145867143 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f198406b500 0x7f198406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:02.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.765+0000 7f1994ff9640 1 -- 172.21.15.161:0/4145867143 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f198c0896e0 con 0x7f19980c6750 2024-03-28T23:31:02.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.766+0000 7f19977fe640 1 --2- 172.21.15.161:0/4145867143 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f198406b500 0x7f198406d9c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1988036000 tx=0x7f1988008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:02.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:02.769+0000 7f1994ff9640 1 -- 172.21.15.161:0/4145867143 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f198c0583d0 con 0x7f19980c6750 2024-03-28T23:31:03.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.019+0000 7f199e92c640 1 -- 172.21.15.161:0/4145867143 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f19980ca650 con 0x7f19980c6750 2024-03-28T23:31:03.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.021+0000 7f1994ff9640 1 -- 172.21.15.161:0/4145867143 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v9) ==== 190+0+147 (secure 0 0 0) 0x7f198c057cd0 con 0x7f19980c6750 2024-03-28T23:31:03.023 INFO:teuthology.orchestra.run.smithi161.stdout:[client.0] 2024-03-28T23:31:03.023 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQA6/QVmc1krAxAAEKsn1Tuuv9OlFZpB4PFhDg== 2024-03-28T23:31:03.023 INFO:teuthology.orchestra.run.smithi161.stdout: caps mds = "allow" 2024-03-28T23:31:03.023 INFO:teuthology.orchestra.run.smithi161.stdout: caps mon = "allow r" 2024-03-28T23:31:03.023 INFO:teuthology.orchestra.run.smithi161.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-28T23:31:03.023 INFO:teuthology.orchestra.run.smithi161.stderr:updated caps for client.0 2024-03-28T23:31:03.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.023+0000 7f199e92c640 1 -- 172.21.15.161:0/4145867143 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f198406b500 msgr2=0x7f198406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:03.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.023+0000 7f199e92c640 1 --2- 172.21.15.161:0/4145867143 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f198406b500 0x7f198406d9c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1988036000 tx=0x7f1988008030 comp rx=0 tx=0).stop 2024-03-28T23:31:03.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.023+0000 7f199e92c640 1 -- 172.21.15.161:0/4145867143 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19980c6750 msgr2=0x7f1998158ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:03.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.023+0000 7f199e92c640 1 --2- 172.21.15.161:0/4145867143 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19980c6750 0x7f1998158ad0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f198c00c400 tx=0x7f198c00c8c0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.023+0000 7f199e92c640 1 -- 172.21.15.161:0/4145867143 shutdown_connections 2024-03-28T23:31:03.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.023+0000 7f199e92c640 1 --2- 172.21.15.161:0/4145867143 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f198406b500 0x7f198406d9c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.023+0000 7f199e92c640 1 --2- 172.21.15.161:0/4145867143 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f19981529d0 0x7f199814a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.023+0000 7f199e92c640 1 --2- 172.21.15.161:0/4145867143 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19980c88b0 0x7f199814eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.023+0000 7f199e92c640 1 --2- 172.21.15.161:0/4145867143 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19980c6750 0x7f1998158ad0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.023+0000 7f199e92c640 1 -- 172.21.15.161:0/4145867143 >> 172.21.15.161:0/4145867143 conn(0x7f19980c7430 msgr2=0x7f199806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:03.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.023+0000 7f199e92c640 1 -- 172.21.15.161:0/4145867143 shutdown_connections 2024-03-28T23:31:03.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.023+0000 7f199e92c640 1 -- 172.21.15.161:0/4145867143 wait complete. 2024-03-28T23:31:03.036 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:31:03.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.135+0000 7f39f64c0640 1 Processor -- start 2024-03-28T23:31:03.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.135+0000 7f39f64c0640 1 -- start start 2024-03-28T23:31:03.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.136+0000 7f39f64c0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f39f0152890 0x7f39f00c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.136+0000 7f39f64c0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f39f00c8ed0 0x7f39f00c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.136+0000 7f39f64c0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39f0158a30 con 0x7f39f00c5950 2024-03-28T23:31:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.136+0000 7f39f64c0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f39f00c8480 con 0x7f39f00c8ed0 2024-03-28T23:31:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.136+0000 7f39f64c0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f39f0159640 con 0x7f39f0152890 2024-03-28T23:31:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.136+0000 7f39ef7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f39f00c8ed0 0x7f39f00c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.136+0000 7f39ef7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f39f00c8ed0 0x7f39f00c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45824/0 (socket says 172.21.15.161:45824) 2024-03-28T23:31:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.136+0000 7f39ef7fe640 1 -- 172.21.15.161:0/3932279099 learned_addr learned my addr 172.21.15.161:0/3932279099 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.136+0000 7f39ef7fe640 1 -- 172.21.15.161:0/3932279099 >> v2:172.21.15.170:3300/0 conn(0x7f39f0152890 msgr2=0x7f39f00c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.136+0000 7f39ef7fe640 1 --2- 172.21.15.161:0/3932279099 >> v2:172.21.15.170:3300/0 conn(0x7f39f0152890 0x7f39f00c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.136+0000 7f39ef7fe640 1 -- 172.21.15.161:0/3932279099 >> v1:172.21.15.170:6789/0 conn(0x7f39f00c5950 legacy=0x7f39f01553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.136+0000 7f39ef7fe640 1 -- 172.21.15.161:0/3932279099 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39f0158d10 con 0x7f39f00c8ed0 2024-03-28T23:31:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.137+0000 7f39ef7fe640 1 --2- 172.21.15.161:0/3932279099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39f00c8ed0 0x7f39f00c5410 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f39f008cc80 tx=0x7f39e402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcd5c9dffc1ed6e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.137+0000 7f39eeffd640 1 -- 172.21.15.161:0/3932279099 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e4038070 con 0x7f39f00c8ed0 2024-03-28T23:31:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.137+0000 7f39eeffd640 1 -- 172.21.15.161:0/3932279099 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39e4004230 con 0x7f39f00c8ed0 2024-03-28T23:31:03.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.137+0000 7f39eeffd640 1 -- 172.21.15.161:0/3932279099 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e402f050 con 0x7f39f00c8ed0 2024-03-28T23:31:03.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.138+0000 7f39f64c0640 1 -- 172.21.15.161:0/3932279099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39f00c8ed0 msgr2=0x7f39f00c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:03.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.138+0000 7f39f64c0640 1 --2- 172.21.15.161:0/3932279099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39f00c8ed0 0x7f39f00c5410 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f39f008cc80 tx=0x7f39e402b7e0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.138+0000 7f39f64c0640 1 -- 172.21.15.161:0/3932279099 shutdown_connections 2024-03-28T23:31:03.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.138+0000 7f39f64c0640 1 --2- 172.21.15.161:0/3932279099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39f00c8ed0 0x7f39f00c5410 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.138+0000 7f39f64c0640 1 --2- 172.21.15.161:0/3932279099 >> v2:172.21.15.170:3300/0 conn(0x7f39f0152890 0x7f39f00c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.138+0000 7f39f64c0640 1 -- 172.21.15.161:0/3932279099 >> 172.21.15.161:0/3932279099 conn(0x7f39f00ac620 msgr2=0x7f39f00aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:03.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.138+0000 7f39f64c0640 1 -- 172.21.15.161:0/3932279099 shutdown_connections 2024-03-28T23:31:03.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.138+0000 7f39f64c0640 1 -- 172.21.15.161:0/3932279099 wait complete. 2024-03-28T23:31:03.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.139+0000 7f39f64c0640 1 Processor -- start 2024-03-28T23:31:03.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.139+0000 7f39f64c0640 1 -- start start 2024-03-28T23:31:03.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.140+0000 7f39f64c0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39f00c5950 0x7f39f01ebc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.140+0000 7f39f64c0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39f00c8ed0 0x7f39f014f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.140+0000 7f39f64c0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39f0152890 0x7f39f00c0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.140+0000 7f39f64c0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f39f0152c70 con 0x7f39f00c5950 2024-03-28T23:31:03.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.140+0000 7f39f64c0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f39f01592d0 con 0x7f39f00c8ed0 2024-03-28T23:31:03.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.140+0000 7f39f64c0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f39f0152df0 con 0x7f39f0152890 2024-03-28T23:31:03.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.140+0000 7f39effff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39f00c5950 0x7f39f01ebc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.140+0000 7f39effff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39f00c5950 0x7f39f01ebc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45832/0 (socket says 172.21.15.161:45832) 2024-03-28T23:31:03.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.140+0000 7f39effff640 1 -- 172.21.15.161:0/486641558 learned_addr learned my addr 172.21.15.161:0/486641558 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:03.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.140+0000 7f39f4a5d640 1 --2- 172.21.15.161:0/486641558 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39f0152890 0x7f39f00c0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.140+0000 7f39ef7fe640 1 --2- 172.21.15.161:0/486641558 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39f00c8ed0 0x7f39f014f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.140+0000 7f39effff640 1 -- 172.21.15.161:0/486641558 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39f0152890 msgr2=0x7f39f00c0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:03.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.140+0000 7f39effff640 1 --2- 172.21.15.161:0/486641558 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39f0152890 0x7f39f00c0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.140+0000 7f39effff640 1 -- 172.21.15.161:0/486641558 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39f00c8ed0 msgr2=0x7f39f014f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:03.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.140+0000 7f39effff640 1 --2- 172.21.15.161:0/486641558 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39f00c8ed0 0x7f39f014f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.140+0000 7f39effff640 1 -- 172.21.15.161:0/486641558 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39f014d9a0 con 0x7f39f00c5950 2024-03-28T23:31:03.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.140+0000 7f39ef7fe640 1 --2- 172.21.15.161:0/486641558 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39f00c8ed0 0x7f39f014f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:03.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.140+0000 7f39effff640 1 --2- 172.21.15.161:0/486641558 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39f00c5950 0x7f39f01ebc90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f39e000b4a0 tx=0x7f39e000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:03.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.140+0000 7f39ecff9640 1 -- 172.21.15.161:0/486641558 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e0014070 con 0x7f39f00c5950 2024-03-28T23:31:03.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.140+0000 7f39ecff9640 1 -- 172.21.15.161:0/486641558 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39e0010610 con 0x7f39f00c5950 2024-03-28T23:31:03.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.140+0000 7f39ecff9640 1 -- 172.21.15.161:0/486641558 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e0010a00 con 0x7f39f00c5950 2024-03-28T23:31:03.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.140+0000 7f39f64c0640 1 -- 172.21.15.161:0/486641558 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39f00bf1d0 con 0x7f39f00c5950 2024-03-28T23:31:03.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.141+0000 7f39f64c0640 1 -- 172.21.15.161:0/486641558 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39f00bec10 con 0x7f39f00c5950 2024-03-28T23:31:03.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.142+0000 7f39f64c0640 1 -- 172.21.15.161:0/486641558 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39f004f4b0 con 0x7f39f00c5950 2024-03-28T23:31:03.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.142+0000 7f39ecff9640 1 -- 172.21.15.161:0/486641558 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f39e0017070 con 0x7f39f00c5950 2024-03-28T23:31:03.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.142+0000 7f39ecff9640 1 --2- 172.21.15.161:0/486641558 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f39dc06b500 0x7f39dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.142+0000 7f39ecff9640 1 -- 172.21.15.161:0/486641558 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f39e0089c70 con 0x7f39f00c5950 2024-03-28T23:31:03.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.142+0000 7f39ef7fe640 1 --2- 172.21.15.161:0/486641558 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f39dc06b500 0x7f39dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.143+0000 7f39ef7fe640 1 --2- 172.21.15.161:0/486641558 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f39dc06b500 0x7f39dc06d9c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f39e4035000 tx=0x7f39e4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:03.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.147+0000 7f39ecff9640 1 -- 172.21.15.161:0/486641558 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f39f014d9a0 con 0x7f39f00c5950 2024-03-28T23:31:03.237 INFO:tasks.ceph.mds.a.smithi161.stderr:2024-03-28T23:31:03.235+0000 7f4a6e3fd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:31:03.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.360+0000 7f39f64c0640 1 -- 172.21.15.161:0/486641558 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39f0151930 con 0x7f39f00c5950 2024-03-28T23:31:03.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.361+0000 7f39ecff9640 1 -- 172.21.15.161:0/486641558 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5181 (secure 0 0 0) 0x7f39e00583a0 con 0x7f39f00c5950 2024-03-28T23:31:03.363 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:03.363 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4853,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/11130685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":11130685},{"type":"v1","addr":"172.21.15.170:6834","nonce":11130685}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5110,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/465771987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":465771987},{"type":"v1","addr":"172.21.15.170:6835","nonce":465771987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5355,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/306644174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":306644174},{"type":"v1","addr":"172.21.15.161:6836","nonce":306644174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:30:57.616423+0000","modified":"2024-03-28T23:31:00.324742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5361},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5361":{"gid":5361,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6837/2636327316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":2636327316},{"type":"v1","addr":"172.21.15.161:6837","nonce":2636327316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5361,"qdb_cluster":[5361]},"id":2}]} 2024-03-28T23:31:03.363 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 17 2024-03-28T23:31:03.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.362+0000 7f39f64c0640 1 -- 172.21.15.161:0/486641558 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f39dc06b500 msgr2=0x7f39dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:03.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.362+0000 7f39f64c0640 1 --2- 172.21.15.161:0/486641558 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f39dc06b500 0x7f39dc06d9c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f39e4035000 tx=0x7f39e4008030 comp rx=0 tx=0).stop 2024-03-28T23:31:03.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.364+0000 7f39f64c0640 1 -- 172.21.15.161:0/486641558 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39f00c5950 msgr2=0x7f39f01ebc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:03.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.364+0000 7f39f64c0640 1 --2- 172.21.15.161:0/486641558 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39f00c5950 0x7f39f01ebc90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f39e000b4a0 tx=0x7f39e000b960 comp rx=0 tx=0).stop 2024-03-28T23:31:03.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.364+0000 7f39f64c0640 1 -- 172.21.15.161:0/486641558 shutdown_connections 2024-03-28T23:31:03.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.364+0000 7f39f64c0640 1 --2- 172.21.15.161:0/486641558 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f39dc06b500 0x7f39dc06d9c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.364+0000 7f39f64c0640 1 --2- 172.21.15.161:0/486641558 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39f0152890 0x7f39f00c0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.364+0000 7f39f64c0640 1 --2- 172.21.15.161:0/486641558 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39f00c8ed0 0x7f39f014f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.364+0000 7f39f64c0640 1 --2- 172.21.15.161:0/486641558 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39f00c5950 0x7f39f01ebc90 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.364+0000 7f39f64c0640 1 -- 172.21.15.161:0/486641558 >> 172.21.15.161:0/486641558 conn(0x7f39f00ac620 msgr2=0x7f39f014b480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:03.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.364+0000 7f39f64c0640 1 -- 172.21.15.161:0/486641558 shutdown_connections 2024-03-28T23:31:03.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.364+0000 7f39f64c0640 1 -- 172.21.15.161:0/486641558 wait complete. 2024-03-28T23:31:03.779 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:30:57.616423+0000', 'modified': '2024-03-28T23:31:00.324742+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5361}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5361': {'gid': 5361, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.161:6837/2636327316', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 2636327316}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 2636327316}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5361, 'qdb_cluster': [5361]} 2024-03-28T23:31:03.779 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:31:03.779 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-28T23:31:03.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.881+0000 7f504873e640 1 Processor -- start 2024-03-28T23:31:03.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.881+0000 7f504873e640 1 -- start start 2024-03-28T23:31:03.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.882+0000 7f504873e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f504006ce80 0x7f50400aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.882+0000 7f504873e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f50400a9680 0x7f50400aab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.882+0000 7f504873e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f504014c250 con 0x7f504014be70 2024-03-28T23:31:03.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.882+0000 7f504873e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f50401529c0 con 0x7f504006ce80 2024-03-28T23:31:03.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.882+0000 7f504873e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f50400a9dc0 con 0x7f50400a9680 2024-03-28T23:31:03.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.882+0000 7f50464da640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f504014be70 0x7f50400ab1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41918/0 (socket says 172.21.15.161:41918) 2024-03-28T23:31:03.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.882+0000 7f50464da640 1 -- 172.21.15.161:0/917602533 learned_addr learned my addr 172.21.15.161:0/917602533 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:03.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.883+0000 7f5046cdb640 1 --2- 172.21.15.161:0/917602533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f50400a9680 0x7f50400aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.883+0000 7f5045cd9640 1 --2- 172.21.15.161:0/917602533 >> v2:172.21.15.170:3300/0 conn(0x7f504006ce80 0x7f50400aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.883+0000 7f50454d8640 1 -- 172.21.15.161:0/917602533 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 233787781 0 0) 0x7f504014c250 con 0x7f504014be70 2024-03-28T23:31:03.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.883+0000 7f50454d8640 1 -- 172.21.15.161:0/917602533 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f502c003520 con 0x7f504014be70 2024-03-28T23:31:03.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.883+0000 7f50454d8640 1 -- 172.21.15.161:0/917602533 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1767918281 0 0) 0x7f502c003520 con 0x7f504014be70 2024-03-28T23:31:03.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.883+0000 7f50454d8640 1 -- 172.21.15.161:0/917602533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f50400a9680 msgr2=0x7f50400aab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:03.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.883+0000 7f50454d8640 1 --2- 172.21.15.161:0/917602533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f50400a9680 0x7f50400aab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.883+0000 7f50454d8640 1 -- 172.21.15.161:0/917602533 >> v2:172.21.15.170:3300/0 conn(0x7f504006ce80 msgr2=0x7f50400aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:03.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.883+0000 7f50454d8640 1 --2- 172.21.15.161:0/917602533 >> v2:172.21.15.170:3300/0 conn(0x7f504006ce80 0x7f50400aa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.883+0000 7f50454d8640 1 -- 172.21.15.161:0/917602533 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50400aa0a0 con 0x7f504014be70 2024-03-28T23:31:03.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.883+0000 7f50454d8640 1 -- 172.21.15.161:0/917602533 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f5030001690 con 0x7f504014be70 2024-03-28T23:31:03.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.883+0000 7f50454d8640 1 -- 172.21.15.161:0/917602533 >> v1:172.21.15.161:6789/0 conn(0x7f504014be70 legacy=0x7f50400ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:03.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.883+0000 7f50454d8640 1 --2- 172.21.15.161:0/917602533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f50400a9680 0x7f502c003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.883+0000 7f50454d8640 1 --2- 172.21.15.161:0/917602533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f502c004010 0x7f502c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.883+0000 7f50454d8640 1 --2- 172.21.15.161:0/917602533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f502c004d40 0x7f502c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.883+0000 7f50454d8640 1 -- 172.21.15.161:0/917602533 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f50401529c0 con 0x7f502c004d40 2024-03-28T23:31:03.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.883+0000 7f50454d8640 1 -- 172.21.15.161:0/917602533 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f50400a9dc0 con 0x7f502c004010 2024-03-28T23:31:03.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.883+0000 7f50454d8640 1 -- 172.21.15.161:0/917602533 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f502c00bb80 con 0x7f50400a9680 2024-03-28T23:31:03.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.883+0000 7f5046cdb640 1 --2- 172.21.15.161:0/917602533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f502c004d40 0x7f502c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.883+0000 7f50464da640 1 --2- 172.21.15.161:0/917602533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f50400a9680 0x7f502c003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.883+0000 7f5045cd9640 1 --2- 172.21.15.161:0/917602533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f502c004010 0x7f502c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.884+0000 7f5046cdb640 1 -- 172.21.15.161:0/917602533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f50400a9680 msgr2=0x7f502c003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:03.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.884+0000 7f5046cdb640 1 --2- 172.21.15.161:0/917602533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f50400a9680 0x7f502c003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.884+0000 7f5046cdb640 1 -- 172.21.15.161:0/917602533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f502c004010 msgr2=0x7f502c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:03.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.884+0000 7f5046cdb640 1 --2- 172.21.15.161:0/917602533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f502c004010 0x7f502c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.884+0000 7f5046cdb640 1 -- 172.21.15.161:0/917602533 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f502c003520 con 0x7f502c004d40 2024-03-28T23:31:03.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.884+0000 7f5046cdb640 1 --2- 172.21.15.161:0/917602533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f502c004d40 0x7f502c0051c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f5038009fc0 tx=0x7f502c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=163245dea449663c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:03.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.884+0000 7f50454d8640 1 -- 172.21.15.161:0/917602533 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5038039070 con 0x7f502c004d40 2024-03-28T23:31:03.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.884+0000 7f50454d8640 1 -- 172.21.15.161:0/917602533 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f503802cc20 con 0x7f502c004d40 2024-03-28T23:31:03.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.884+0000 7f50454d8640 1 -- 172.21.15.161:0/917602533 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5038009c10 con 0x7f502c004d40 2024-03-28T23:31:03.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.885+0000 7f504873e640 1 -- 172.21.15.161:0/917602533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f502c004d40 msgr2=0x7f502c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.885+0000 7f504873e640 1 --2- 172.21.15.161:0/917602533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f502c004d40 0x7f502c0051c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f5038009fc0 tx=0x7f502c005830 comp rx=0 tx=0).stop 2024-03-28T23:31:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.885+0000 7f5046cdb640 1 -- 172.21.15.161:0/917602533 reap_dead start 2024-03-28T23:31:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.885+0000 7f504873e640 1 -- 172.21.15.161:0/917602533 shutdown_connections 2024-03-28T23:31:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.885+0000 7f504873e640 1 -- 172.21.15.161:0/917602533 >> 172.21.15.161:0/917602533 conn(0x7f50400c6670 msgr2=0x7f50400c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.885+0000 7f504873e640 1 -- 172.21.15.161:0/917602533 shutdown_connections 2024-03-28T23:31:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.885+0000 7f504873e640 1 -- 172.21.15.161:0/917602533 wait complete. 2024-03-28T23:31:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.886+0000 7f504873e640 1 Processor -- start 2024-03-28T23:31:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.886+0000 7f504873e640 1 -- start start 2024-03-28T23:31:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.887+0000 7f504873e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f502c004d40 0x7f50400c2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.887+0000 7f504873e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50400bf750 0x7f50400bfbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.887+0000 7f504873e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f504014c480 0x7f504014c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.887+0000 7f504873e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f50400aba30 con 0x7f50400bf750 2024-03-28T23:31:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.887+0000 7f504873e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f50400ab8b0 con 0x7f504014c480 2024-03-28T23:31:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.887+0000 7f504873e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f504006b150 con 0x7f502c004d40 2024-03-28T23:31:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.887+0000 7f5045cd9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50400bf750 0x7f50400bfbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.887+0000 7f5045cd9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50400bf750 0x7f50400bfbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45850/0 (socket says 172.21.15.161:45850) 2024-03-28T23:31:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.887+0000 7f5045cd9640 1 -- 172.21.15.161:0/1705092804 learned_addr learned my addr 172.21.15.161:0/1705092804 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.887+0000 7f5046cdb640 1 --2- 172.21.15.161:0/1705092804 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f504014c480 0x7f504014c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.887+0000 7f50464da640 1 --2- 172.21.15.161:0/1705092804 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f502c004d40 0x7f50400c2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.887+0000 7f5045cd9640 1 -- 172.21.15.161:0/1705092804 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f502c004d40 msgr2=0x7f50400c2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.887+0000 7f5045cd9640 1 --2- 172.21.15.161:0/1705092804 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f502c004d40 0x7f50400c2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.887+0000 7f5045cd9640 1 -- 172.21.15.161:0/1705092804 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f504014c480 msgr2=0x7f504014c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.887+0000 7f5045cd9640 1 --2- 172.21.15.161:0/1705092804 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f504014c480 0x7f504014c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.887+0000 7f5045cd9640 1 -- 172.21.15.161:0/1705092804 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50401554b0 con 0x7f50400bf750 2024-03-28T23:31:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.888+0000 7f5045cd9640 1 --2- 172.21.15.161:0/1705092804 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50400bf750 0x7f50400bfbb0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f503c011990 tx=0x7f503c011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.888+0000 7f5036ffd640 1 -- 172.21.15.161:0/1705092804 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503c00f9d0 con 0x7f50400bf750 2024-03-28T23:31:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.888+0000 7f5036ffd640 1 -- 172.21.15.161:0/1705092804 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f503c00fb70 con 0x7f50400bf750 2024-03-28T23:31:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.888+0000 7f504873e640 1 -- 172.21.15.161:0/1705092804 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50401ec280 con 0x7f50400bf750 2024-03-28T23:31:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.888+0000 7f5036ffd640 1 -- 172.21.15.161:0/1705092804 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503c014440 con 0x7f50400bf750 2024-03-28T23:31:03.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.888+0000 7f504873e640 1 -- 172.21.15.161:0/1705092804 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50401ec840 con 0x7f50400bf750 2024-03-28T23:31:03.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.889+0000 7f5036ffd640 1 -- 172.21.15.161:0/1705092804 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f503c012040 con 0x7f50400bf750 2024-03-28T23:31:03.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.890+0000 7f5036ffd640 1 --2- 172.21.15.161:0/1705092804 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f502406b4d0 0x7f502406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.890+0000 7f5036ffd640 1 -- 172.21.15.161:0/1705092804 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f503c08a820 con 0x7f50400bf750 2024-03-28T23:31:03.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.890+0000 7f50464da640 1 --2- 172.21.15.161:0/1705092804 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f502406b4d0 0x7f502406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.890+0000 7f50464da640 1 --2- 172.21.15.161:0/1705092804 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f502406b4d0 0x7f502406d990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f50300010e0 tx=0x7f5030007e10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:03.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.891+0000 7f504b3f9740 1 Processor -- start 2024-03-28T23:31:03.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.891+0000 7f504b3f9740 1 -- start start 2024-03-28T23:31:03.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.891+0000 7f504b3f9740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be494d7db0 0x55be494d5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.891+0000 7f504b3f9740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d8190 0x55be494d5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.891+0000 7f504b3f9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be494d6310 0x55be494d24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.891+0000 7f504b3f9740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55be4943baf0 con 0x55be494d8190 2024-03-28T23:31:03.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.891+0000 7f504b3f9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55be493e49d0 con 0x55be494d6310 2024-03-28T23:31:03.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.891+0000 7f504b3f9740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55be49437b90 con 0x55be494d7db0 2024-03-28T23:31:03.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.891+0000 7f5045cd9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d8190 0x55be494d5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.891+0000 7f5045cd9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d8190 0x55be494d5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45858/0 (socket says 172.21.15.161:45858) 2024-03-28T23:31:03.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.891+0000 7f5045cd9640 1 -- 172.21.15.161:0/698236135 learned_addr learned my addr 172.21.15.161:0/698236135 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:03.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.892+0000 7f5045cd9640 1 -- 172.21.15.161:0/698236135 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be494d7db0 msgr2=0x55be494d5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:03.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.892+0000 7f5045cd9640 1 --2- 172.21.15.161:0/698236135 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be494d7db0 0x55be494d5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.892+0000 7f5045cd9640 1 -- 172.21.15.161:0/698236135 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be494d6310 msgr2=0x55be494d24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:03.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.892+0000 7f5045cd9640 1 --2- 172.21.15.161:0/698236135 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be494d6310 0x55be494d24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.892+0000 7f5045cd9640 1 -- 172.21.15.161:0/698236135 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be491502d0 con 0x55be494d8190 2024-03-28T23:31:03.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.892+0000 7f5045cd9640 1 --2- 172.21.15.161:0/698236135 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d8190 0x55be494d5dd0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f503c088b30 tx=0x7f503c059650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36099ffd8ba7b281 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:03.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.892+0000 7f5023fff640 1 -- 172.21.15.161:0/698236135 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503c070020 con 0x55be494d8190 2024-03-28T23:31:03.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.892+0000 7f5023fff640 1 -- 172.21.15.161:0/698236135 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f503c061520 con 0x55be494d8190 2024-03-28T23:31:03.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.892+0000 7f5023fff640 1 -- 172.21.15.161:0/698236135 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503c057110 con 0x55be494d8190 2024-03-28T23:31:03.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.893+0000 7f504b3f9740 1 -- 172.21.15.161:0/698236135 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d8190 msgr2=0x55be494d5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:03.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.893+0000 7f504b3f9740 1 --2- 172.21.15.161:0/698236135 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d8190 0x55be494d5dd0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f503c088b30 tx=0x7f503c059650 comp rx=0 tx=0).stop 2024-03-28T23:31:03.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.893+0000 7f504b3f9740 1 -- 172.21.15.161:0/698236135 shutdown_connections 2024-03-28T23:31:03.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.893+0000 7f504b3f9740 1 --2- 172.21.15.161:0/698236135 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d8190 0x55be494d5dd0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.893+0000 7f504b3f9740 1 --2- 172.21.15.161:0/698236135 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be494d6310 0x55be494d24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.893+0000 7f504b3f9740 1 --2- 172.21.15.161:0/698236135 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be494d7db0 0x55be494d5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.893+0000 7f504b3f9740 1 -- 172.21.15.161:0/698236135 >> 172.21.15.161:0/698236135 conn(0x55be494d4630 msgr2=0x55be494f9220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:03.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.893+0000 7f504b3f9740 1 -- 172.21.15.161:0/698236135 shutdown_connections 2024-03-28T23:31:03.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.893+0000 7f504b3f9740 1 -- 172.21.15.161:0/698236135 wait complete. 2024-03-28T23:31:03.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.893+0000 7f504b3f9740 1 -- start start 2024-03-28T23:31:03.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.893+0000 7f504b3f9740 1 Processor -- start 2024-03-28T23:31:03.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.894+0000 7f504b3f9740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be494d7db0 0x55be4950c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.894+0000 7f504b3f9740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d8190 0x55be494f8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.894+0000 7f504b3f9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be49527d20 0x55be494f5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.894+0000 7f504b3f9740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55be4943baf0 con 0x55be494d8190 2024-03-28T23:31:03.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.894+0000 7f504b3f9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55be493f7f70 con 0x55be49527d20 2024-03-28T23:31:03.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.894+0000 7f504b3f9740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55be494fa090 con 0x55be494d7db0 2024-03-28T23:31:03.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.894+0000 7f504b3f9740 10 client.0 _ll_register_callbacks cb 0x55be4914fac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:31:03.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.894+0000 7f5045cd9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d8190 0x55be494f8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.894+0000 7f5045cd9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d8190 0x55be494f8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45864/0 (socket says 172.21.15.161:45864) 2024-03-28T23:31:03.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.894+0000 7f5045cd9640 1 -- 172.21.15.161:0/4148982506 learned_addr learned my addr 172.21.15.161:0/4148982506 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:03.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.895+0000 7f50464da640 1 --2- 172.21.15.161:0/4148982506 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be494d7db0 0x55be4950c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.895+0000 7f5045cd9640 1 -- 172.21.15.161:0/4148982506 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be494d7db0 msgr2=0x55be4950c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:03.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.895+0000 7f5045cd9640 1 --2- 172.21.15.161:0/4148982506 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be494d7db0 0x55be4950c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.895+0000 7f5045cd9640 1 -- 172.21.15.161:0/4148982506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be49527d20 msgr2=0x55be494f5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:03.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.895+0000 7f5045cd9640 1 --2- 172.21.15.161:0/4148982506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be49527d20 0x55be494f5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.895+0000 7f5045cd9640 1 -- 172.21.15.161:0/4148982506 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be4921a6b0 con 0x55be494d8190 2024-03-28T23:31:03.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.895+0000 7f50464da640 1 --2- 172.21.15.161:0/4148982506 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be494d7db0 0x55be4950c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:03.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.895+0000 7f5045cd9640 1 --2- 172.21.15.161:0/4148982506 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d8190 0x55be494f8700 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f503c060ff0 tx=0x7f503c08a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368459f0fc490de2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:03.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.895+0000 7f50217fa640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:31:03.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.895+0000 7f50217fa640 1 -- 172.21.15.161:0/4148982506 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503c070040 con 0x55be494d8190 2024-03-28T23:31:03.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.895+0000 7f50217fa640 1 -- 172.21.15.161:0/4148982506 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f503c061090 con 0x55be494d8190 2024-03-28T23:31:03.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.895+0000 7f504b3f9740 1 -- 172.21.15.161:0/4148982506 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f503c061520 con 0x55be494d8190 2024-03-28T23:31:03.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.895+0000 7f50217fa640 1 -- 172.21.15.161:0/4148982506 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f503c07e050 con 0x55be494d8190 2024-03-28T23:31:03.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.895+0000 7f504b3f9740 10 client.5538 fetch_fsmap learned FSMap version 17 2024-03-28T23:31:03.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.895+0000 7f504b3f9740 1 -- 172.21.15.161:0/4148982506 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55be4950b400 con 0x55be494d8190 2024-03-28T23:31:03.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.895+0000 7f50217fa640 1 -- 172.21.15.161:0/4148982506 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f503c061520 con 0x55be494d8190 2024-03-28T23:31:03.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.895+0000 7f50217fa640 1 -- 172.21.15.161:0/4148982506 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7f503c077ce0 con 0x55be494d8190 2024-03-28T23:31:03.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.895+0000 7f504b3f9740 10 client.5538 fetch_fsmap finished waiting for FSMap version 17 2024-03-28T23:31:03.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.895+0000 7f504b3f9740 10 client.5538 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T23:31:03.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.895+0000 7f504b3f9740 20 client.5538 populate_metadata read hostname 'smithi161' 2024-03-28T23:31:03.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.895+0000 7f504b3f9740 1 --2- 172.21.15.161:0/4148982506 >> [v2:172.21.15.161:6835/2636327316,v1:172.21.15.161:6837/2636327316] conn(0x55be494fa9f0 0x55be494fadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.895+0000 7f504b3f9740 4 client.5538 mds_command: new command op to 5361 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:31:03.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.895+0000 7f504b3f9740 1 -- 172.21.15.161:0/4148982506 --> [v2:172.21.15.161:6835/2636327316,v1:172.21.15.161:6837/2636327316] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be49449030 con 0x55be494fa9f0 2024-03-28T23:31:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.896+0000 7f50464da640 1 --2- 172.21.15.161:0/4148982506 >> [v2:172.21.15.161:6835/2636327316,v1:172.21.15.161:6837/2636327316] conn(0x55be494fa9f0 0x55be494fadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.896+0000 7f50217fa640 10 client.5538 ms_handle_connect on v2:172.21.15.161:6835/2636327316 2024-03-28T23:31:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.896+0000 7f50464da640 1 --2- 172.21.15.161:0/4148982506 >> [v2:172.21.15.161:6835/2636327316,v1:172.21.15.161:6837/2636327316] conn(0x55be494fa9f0 0x55be494fadd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f56bd7352052f46 server_cookie=63c63b4647a3a2d2 in_seq=0 out_seq=0 2024-03-28T23:31:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.897+0000 7f50217fa640 1 -- 172.21.15.161:0/4148982506 <== mds.0 v2:172.21.15.161:6835/2636327316 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55be49449030 con 0x55be494fa9f0 2024-03-28T23:31:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.897+0000 7f50217fa640 10 client.5538 handle_command_reply: tid=0 2024-03-28T23:31:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.897+0000 7f504b3f9740 1 client.5538 shutdown 2024-03-28T23:31:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.897+0000 7f504b3f9740 1 -- 172.21.15.161:0/4148982506 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d8190 msgr2=0x55be494f8700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.897+0000 7f504b3f9740 1 --2- 172.21.15.161:0/4148982506 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d8190 0x55be494f8700 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f503c060ff0 tx=0x7f503c08a040 comp rx=0 tx=0).stop 2024-03-28T23:31:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.897+0000 7f504b3f9740 1 -- 172.21.15.161:0/4148982506 shutdown_connections 2024-03-28T23:31:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.897+0000 7f504b3f9740 1 --2- 172.21.15.161:0/4148982506 >> [v2:172.21.15.161:6835/2636327316,v1:172.21.15.161:6837/2636327316] conn(0x55be494fa9f0 0x55be494fadd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.897+0000 7f504b3f9740 1 --2- 172.21.15.161:0/4148982506 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d8190 0x55be494f8700 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.897+0000 7f50217fa640 0 client.5538 ms_handle_reset on v2:172.21.15.161:6835/2636327316 2024-03-28T23:31:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.897+0000 7f504b3f9740 1 --2- 172.21.15.161:0/4148982506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be49527d20 0x55be494f5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.897+0000 7f504b3f9740 1 --2- 172.21.15.161:0/4148982506 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be494d7db0 0x55be4950c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.897+0000 7f504b3f9740 1 -- 172.21.15.161:0/4148982506 >> 172.21.15.161:0/4148982506 conn(0x55be494d6310 msgr2=0x55be494d7150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:03.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.897+0000 7f504b3f9740 1 -- 172.21.15.161:0/4148982506 shutdown_connections 2024-03-28T23:31:03.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.897+0000 7f504b3f9740 1 -- 172.21.15.161:0/4148982506 wait complete. 2024-03-28T23:31:03.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.897+0000 7f504b3f9740 20 client.5538 trim_cache size 0 max 16384 2024-03-28T23:31:03.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.915+0000 7f504b3f9740 1 Processor -- start 2024-03-28T23:31:03.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.915+0000 7f504b3f9740 1 -- start start 2024-03-28T23:31:03.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.916+0000 7f504b3f9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be494d7db0 0x55be494d7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.916+0000 7f504b3f9740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d8190 0x55be49510ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.916+0000 7f504b3f9740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be49527d20 0x55be4950a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.916+0000 7f504b3f9740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f50401f5400 con 0x55be494d8190 2024-03-28T23:31:03.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.916+0000 7f504b3f9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f50401f4760 con 0x55be494d7db0 2024-03-28T23:31:03.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.916+0000 7f504b3f9740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f50401f4db0 con 0x55be49527d20 2024-03-28T23:31:03.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.916+0000 7f5045cd9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d8190 0x55be49510ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.916+0000 7f5045cd9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d8190 0x55be49510ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45874/0 (socket says 172.21.15.161:45874) 2024-03-28T23:31:03.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.916+0000 7f5045cd9640 1 -- 172.21.15.161:0/296355386 learned_addr learned my addr 172.21.15.161:0/296355386 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:03.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.916+0000 7f50464da640 1 --2- 172.21.15.161:0/296355386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be494d7db0 0x55be494d7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.916+0000 7f5046cdb640 1 --2- 172.21.15.161:0/296355386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be49527d20 0x55be4950a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.916+0000 7f5045cd9640 1 -- 172.21.15.161:0/296355386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be49527d20 msgr2=0x55be4950a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:03.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.916+0000 7f5045cd9640 1 --2- 172.21.15.161:0/296355386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be49527d20 0x55be4950a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.916+0000 7f5045cd9640 1 -- 172.21.15.161:0/296355386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be494d7db0 msgr2=0x55be494d7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:03.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.916+0000 7f5045cd9640 1 --2- 172.21.15.161:0/296355386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be494d7db0 0x55be494d7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.916+0000 7f5045cd9640 1 -- 172.21.15.161:0/296355386 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be494d33e0 con 0x55be494d8190 2024-03-28T23:31:03.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.916+0000 7f5046cdb640 1 --2- 172.21.15.161:0/296355386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be49527d20 0x55be4950a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:03.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.916+0000 7f5045cd9640 1 --2- 172.21.15.161:0/296355386 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d8190 0x55be49510ca0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f503c055270 tx=0x7f503c06e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb9a3818854c8463 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:03.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.916+0000 7f50237fe640 1 -- 172.21.15.161:0/296355386 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503c070040 con 0x55be494d8190 2024-03-28T23:31:03.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.916+0000 7f50237fe640 1 -- 172.21.15.161:0/296355386 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f503c061520 con 0x55be494d8190 2024-03-28T23:31:03.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.916+0000 7f50237fe640 1 -- 172.21.15.161:0/296355386 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503c070ba0 con 0x55be494d8190 2024-03-28T23:31:03.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.917+0000 7f504b3f9740 1 -- 172.21.15.161:0/296355386 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d8190 msgr2=0x55be49510ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:03.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.917+0000 7f504b3f9740 1 --2- 172.21.15.161:0/296355386 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d8190 0x55be49510ca0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f503c055270 tx=0x7f503c06e070 comp rx=0 tx=0).stop 2024-03-28T23:31:03.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.917+0000 7f504b3f9740 1 -- 172.21.15.161:0/296355386 shutdown_connections 2024-03-28T23:31:03.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.917+0000 7f504b3f9740 1 --2- 172.21.15.161:0/296355386 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d8190 0x55be49510ca0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.917+0000 7f504b3f9740 1 --2- 172.21.15.161:0/296355386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be49527d20 0x55be4950a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.917+0000 7f504b3f9740 1 --2- 172.21.15.161:0/296355386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be494d7db0 0x55be494d7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.917+0000 7f504b3f9740 1 -- 172.21.15.161:0/296355386 >> 172.21.15.161:0/296355386 conn(0x55be494d6310 msgr2=0x55be494f5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:03.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.917+0000 7f504b3f9740 1 -- 172.21.15.161:0/296355386 shutdown_connections 2024-03-28T23:31:03.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.917+0000 7f504b3f9740 1 -- 172.21.15.161:0/296355386 wait complete. 2024-03-28T23:31:03.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.918+0000 7f504b3f9740 1 -- start start 2024-03-28T23:31:03.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.918+0000 7f504b3f9740 1 Processor -- start 2024-03-28T23:31:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.918+0000 7f504b3f9740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d7db0 0x55be494d73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.918+0000 7f504b3f9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be494d8190 0x55be4958f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.918+0000 7f504b3f9740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be49527d20 0x55be494d1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.918+0000 7f504b3f9740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55be4943baf0 con 0x55be494d7db0 2024-03-28T23:31:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.918+0000 7f504b3f9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55be493f7f70 con 0x55be494d8190 2024-03-28T23:31:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.918+0000 7f504b3f9740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55be4950d6e0 con 0x55be49527d20 2024-03-28T23:31:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.919+0000 7f504b3f9740 10 client.0 _ll_register_callbacks cb 0x7f503c058800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:31:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.919+0000 7f50464da640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d7db0 0x55be494d73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.919+0000 7f50464da640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d7db0 0x55be494d73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45880/0 (socket says 172.21.15.161:45880) 2024-03-28T23:31:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.919+0000 7f50464da640 1 -- 172.21.15.161:0/214521271 learned_addr learned my addr 172.21.15.161:0/214521271 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.919+0000 7f5045cd9640 1 --2- 172.21.15.161:0/214521271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be494d8190 0x55be4958f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.919+0000 7f5046cdb640 1 --2- 172.21.15.161:0/214521271 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be49527d20 0x55be494d1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.919+0000 7f50464da640 1 -- 172.21.15.161:0/214521271 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be49527d20 msgr2=0x55be494d1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:03.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.919+0000 7f50464da640 1 --2- 172.21.15.161:0/214521271 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be49527d20 0x55be494d1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.919+0000 7f50464da640 1 -- 172.21.15.161:0/214521271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be494d8190 msgr2=0x55be4958f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:03.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.919+0000 7f50464da640 1 --2- 172.21.15.161:0/214521271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be494d8190 0x55be4958f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.919+0000 7f50464da640 1 -- 172.21.15.161:0/214521271 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be494f37f0 con 0x55be494d7db0 2024-03-28T23:31:03.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.919+0000 7f50464da640 1 --2- 172.21.15.161:0/214521271 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d7db0 0x55be494d73d0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x55be4950d860 tx=0x7f5030007970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10b317abf8e22e80 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:03.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.919+0000 7f50217fa640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:31:03.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.919+0000 7f50217fa640 1 -- 172.21.15.161:0/214521271 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503001b040 con 0x55be494d7db0 2024-03-28T23:31:03.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.919+0000 7f50217fa640 1 -- 172.21.15.161:0/214521271 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f5030015ca0 con 0x55be494d7db0 2024-03-28T23:31:03.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.919+0000 7f504b3f9740 1 -- 172.21.15.161:0/214521271 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f503c061520 con 0x55be494d7db0 2024-03-28T23:31:03.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.919+0000 7f50217fa640 1 -- 172.21.15.161:0/214521271 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f50300260b0 con 0x55be494d7db0 2024-03-28T23:31:03.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.919+0000 7f504b3f9740 10 client.5544 fetch_fsmap learned FSMap version 17 2024-03-28T23:31:03.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.919+0000 7f504b3f9740 1 -- 172.21.15.161:0/214521271 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55be494d5830 con 0x55be494d7db0 2024-03-28T23:31:03.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.920+0000 7f50217fa640 1 -- 172.21.15.161:0/214521271 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f503002a030 con 0x55be494d7db0 2024-03-28T23:31:03.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.920+0000 7f50217fa640 1 -- 172.21.15.161:0/214521271 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7f5030021050 con 0x55be494d7db0 2024-03-28T23:31:03.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.920+0000 7f504b3f9740 10 client.5544 fetch_fsmap finished waiting for FSMap version 17 2024-03-28T23:31:03.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.920+0000 7f504b3f9740 10 client.5544 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T23:31:03.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.920+0000 7f504b3f9740 20 client.5544 populate_metadata read hostname 'smithi161' 2024-03-28T23:31:03.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.920+0000 7f504b3f9740 1 --2- 172.21.15.161:0/214521271 >> [v2:172.21.15.161:6835/2636327316,v1:172.21.15.161:6837/2636327316] conn(0x55be49592120 0x55be49595640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:03.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.920+0000 7f504b3f9740 4 client.5544 mds_command: new command op to 5361 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:31:03.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.920+0000 7f504b3f9740 1 -- 172.21.15.161:0/214521271 --> [v2:172.21.15.161:6835/2636327316,v1:172.21.15.161:6837/2636327316] -- command(tid 0: {"prefix": "status"}) -- 0x55be49443380 con 0x55be49592120 2024-03-28T23:31:03.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.920+0000 7f5045cd9640 1 --2- 172.21.15.161:0/214521271 >> [v2:172.21.15.161:6835/2636327316,v1:172.21.15.161:6837/2636327316] conn(0x55be49592120 0x55be49595640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:03.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.920+0000 7f50217fa640 10 client.5544 ms_handle_connect on v2:172.21.15.161:6835/2636327316 2024-03-28T23:31:03.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.920+0000 7f5045cd9640 1 --2- 172.21.15.161:0/214521271 >> [v2:172.21.15.161:6835/2636327316,v1:172.21.15.161:6837/2636327316] conn(0x55be49592120 0x55be49595640 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=708fbda11797ef3e server_cookie=ef137eb6157d89fa in_seq=0 out_seq=0 2024-03-28T23:31:03.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.920+0000 7f50217fa640 1 -- 172.21.15.161:0/214521271 <== mds.0 v2:172.21.15.161:6835/2636327316 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55be49443380 con 0x55be49592120 2024-03-28T23:31:03.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.920+0000 7f50217fa640 10 client.5544 handle_command_reply: tid=0 2024-03-28T23:31:03.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.920+0000 7f504b3f9740 1 client.5544 shutdown 2024-03-28T23:31:03.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.920+0000 7f504b3f9740 1 -- 172.21.15.161:0/214521271 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d7db0 msgr2=0x55be494d73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:03.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.920+0000 7f504b3f9740 1 --2- 172.21.15.161:0/214521271 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d7db0 0x55be494d73d0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x55be4950d860 tx=0x7f5030007970 comp rx=0 tx=0).stop 2024-03-28T23:31:03.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.920+0000 7f504b3f9740 1 -- 172.21.15.161:0/214521271 shutdown_connections 2024-03-28T23:31:03.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.921+0000 7f504b3f9740 1 --2- 172.21.15.161:0/214521271 >> [v2:172.21.15.161:6835/2636327316,v1:172.21.15.161:6837/2636327316] conn(0x55be49592120 0x55be49595640 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.921+0000 7f504b3f9740 1 --2- 172.21.15.161:0/214521271 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be49527d20 0x55be494d1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.921+0000 7f504b3f9740 1 --2- 172.21.15.161:0/214521271 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be494d8190 0x55be4958f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.921+0000 7f504b3f9740 1 --2- 172.21.15.161:0/214521271 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55be494d7db0 0x55be494d73d0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.921+0000 7f50217fa640 0 client.5544 ms_handle_reset on v2:172.21.15.161:6835/2636327316 2024-03-28T23:31:03.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.921+0000 7f504b3f9740 1 -- 172.21.15.161:0/214521271 >> 172.21.15.161:0/214521271 conn(0x55be494d6310 msgr2=0x55be494cd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:03.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.921+0000 7f504b3f9740 1 -- 172.21.15.161:0/214521271 shutdown_connections 2024-03-28T23:31:03.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.921+0000 7f504b3f9740 1 -- 172.21.15.161:0/214521271 wait complete. 2024-03-28T23:31:03.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.921+0000 7f504b3f9740 20 client.5544 trim_cache size 0 max 16384 2024-03-28T23:31:03.928 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:31:03.928 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:31:03.928 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:31:03.928 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 5361, 2024-03-28T23:31:03.928 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:31:03.928 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:31:03.928 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:31:03.928 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 5.6635812809999999, 2024-03-28T23:31:03.928 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 17, 2024-03-28T23:31:03.928 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 38, 2024-03-28T23:31:03.928 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 38, 2024-03-28T23:31:03.928 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 12.555213011999999 2024-03-28T23:31:03.928 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:31:03.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.921+0000 7f504873e640 1 -- 172.21.15.161:0/1705092804 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f502406b4d0 msgr2=0x7f502406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:03.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.921+0000 7f504873e640 1 --2- 172.21.15.161:0/1705092804 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f502406b4d0 0x7f502406d990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f50300010e0 tx=0x7f5030007e10 comp rx=0 tx=0).stop 2024-03-28T23:31:03.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.921+0000 7f504873e640 1 -- 172.21.15.161:0/1705092804 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50400bf750 msgr2=0x7f50400bfbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:03.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.921+0000 7f504873e640 1 --2- 172.21.15.161:0/1705092804 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50400bf750 0x7f50400bfbb0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f503c011990 tx=0x7f503c011d90 comp rx=0 tx=0).stop 2024-03-28T23:31:03.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.921+0000 7f504873e640 1 -- 172.21.15.161:0/1705092804 shutdown_connections 2024-03-28T23:31:03.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.921+0000 7f504873e640 1 --2- 172.21.15.161:0/1705092804 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f502406b4d0 0x7f502406d990 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.921+0000 7f504873e640 1 --2- 172.21.15.161:0/1705092804 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50400bf750 0x7f50400bfbb0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.921+0000 7f504873e640 1 --2- 172.21.15.161:0/1705092804 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f504014c480 0x7f504014c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.921+0000 7f504873e640 1 --2- 172.21.15.161:0/1705092804 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f502c004d40 0x7f50400c2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:03.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.921+0000 7f504873e640 1 -- 172.21.15.161:0/1705092804 >> 172.21.15.161:0/1705092804 conn(0x7f50400c6670 msgr2=0x7f50400c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:03.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.921+0000 7f504873e640 1 -- 172.21.15.161:0/1705092804 shutdown_connections 2024-03-28T23:31:03.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:03.921+0000 7f504873e640 1 -- 172.21.15.161:0/1705092804 wait complete. 2024-03-28T23:31:03.934 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:31:04.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.039+0000 7fa374113640 1 Processor -- start 2024-03-28T23:31:04.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.039+0000 7fa374113640 1 -- start start 2024-03-28T23:31:04.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.039+0000 7fa374113640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa36c152890 0x7fa36c0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:04.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.039+0000 7fa374113640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa36c0c5b20 0x7fa36c1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:04.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.039+0000 7fa374113640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa36c158a30 con 0x7fa36c0c8ed0 2024-03-28T23:31:04.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.039+0000 7fa374113640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa36c0c8480 con 0x7fa36c152890 2024-03-28T23:31:04.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.039+0000 7fa374113640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fa36c159640 con 0x7fa36c0c5b20 2024-03-28T23:31:04.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.040+0000 7fa371eaf640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa36c152890 0x7fa36c0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:04.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.040+0000 7fa3716ae640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa36c0c8ed0 0x7fa36c0c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41930/0 (socket says 172.21.15.161:41930) 2024-03-28T23:31:04.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.040+0000 7fa3716ae640 1 -- 172.21.15.161:0/1313655852 learned_addr learned my addr 172.21.15.161:0/1313655852 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:04.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.040+0000 7fa3726b0640 1 --2- 172.21.15.161:0/1313655852 >> v2:172.21.15.170:3300/0 conn(0x7fa36c0c5b20 0x7fa36c1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:04.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.040+0000 7fa371eaf640 1 -- 172.21.15.161:0/1313655852 >> v2:172.21.15.170:3300/0 conn(0x7fa36c0c5b20 msgr2=0x7fa36c1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:04.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.040+0000 7fa371eaf640 1 --2- 172.21.15.161:0/1313655852 >> v2:172.21.15.170:3300/0 conn(0x7fa36c0c5b20 0x7fa36c1553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.040+0000 7fa371eaf640 1 -- 172.21.15.161:0/1313655852 >> v1:172.21.15.161:6789/0 conn(0x7fa36c0c8ed0 legacy=0x7fa36c0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:04.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.040+0000 7fa371eaf640 1 -- 172.21.15.161:0/1313655852 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa36c158d10 con 0x7fa36c152890 2024-03-28T23:31:04.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.040+0000 7fa370ead640 1 -- 172.21.15.161:0/1313655852 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1219415008 0 0) 0x7fa36c158a30 con 0x7fa36c0c8ed0 2024-03-28T23:31:04.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.040+0000 7fa3726b0640 1 --2- 172.21.15.161:0/1313655852 >> v2:172.21.15.170:3300/0 conn(0x7fa36c0c5b20 0x7fa36c1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:04.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.041+0000 7fa371eaf640 1 --2- 172.21.15.161:0/1313655852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa36c152890 0x7fa36c0c8990 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa36c0ad480 tx=0x7fa36802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a0f8cb83d81d5b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:04.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.041+0000 7fa370ead640 1 -- 172.21.15.161:0/1313655852 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa368038070 con 0x7fa36c152890 2024-03-28T23:31:04.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.041+0000 7fa370ead640 1 -- 172.21.15.161:0/1313655852 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa368031710 con 0x7fa36c152890 2024-03-28T23:31:04.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.041+0000 7fa370ead640 1 -- 172.21.15.161:0/1313655852 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa368031ae0 con 0x7fa36c152890 2024-03-28T23:31:04.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.042+0000 7fa374113640 1 -- 172.21.15.161:0/1313655852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa36c152890 msgr2=0x7fa36c0c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:04.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.042+0000 7fa374113640 1 --2- 172.21.15.161:0/1313655852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa36c152890 0x7fa36c0c8990 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa36c0ad480 tx=0x7fa36802b7b0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.042+0000 7fa374113640 1 -- 172.21.15.161:0/1313655852 shutdown_connections 2024-03-28T23:31:04.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.042+0000 7fa374113640 1 --2- 172.21.15.161:0/1313655852 >> v2:172.21.15.170:3300/0 conn(0x7fa36c0c5b20 0x7fa36c1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.042+0000 7fa374113640 1 --2- 172.21.15.161:0/1313655852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa36c152890 0x7fa36c0c8990 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.042+0000 7fa374113640 1 -- 172.21.15.161:0/1313655852 >> 172.21.15.161:0/1313655852 conn(0x7fa36c0ac620 msgr2=0x7fa36c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:04.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.042+0000 7fa374113640 1 -- 172.21.15.161:0/1313655852 shutdown_connections 2024-03-28T23:31:04.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.042+0000 7fa374113640 1 -- 172.21.15.161:0/1313655852 wait complete. 2024-03-28T23:31:04.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.044+0000 7fa374113640 1 Processor -- start 2024-03-28T23:31:04.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.044+0000 7fa374113640 1 -- start start 2024-03-28T23:31:04.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.045+0000 7fa374113640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa36c0c5b20 0x7fa36c1eb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:04.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.045+0000 7fa374113640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa36c0c8ed0 0x7fa36c0abb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:04.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.045+0000 7fa374113640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa36c152890 0x7fa36c0bf800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:04.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.045+0000 7fa374113640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa36c152c70 con 0x7fa36c0c8ed0 2024-03-28T23:31:04.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.045+0000 7fa374113640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa36c1592d0 con 0x7fa36c0c5b20 2024-03-28T23:31:04.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.045+0000 7fa374113640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa36c152df0 con 0x7fa36c152890 2024-03-28T23:31:04.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.045+0000 7fa3716ae640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa36c0c8ed0 0x7fa36c0abb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:04.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.045+0000 7fa3716ae640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa36c0c8ed0 0x7fa36c0abb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45896/0 (socket says 172.21.15.161:45896) 2024-03-28T23:31:04.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.045+0000 7fa3716ae640 1 -- 172.21.15.161:0/1702181954 learned_addr learned my addr 172.21.15.161:0/1702181954 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:04.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.045+0000 7fa3726b0640 1 --2- 172.21.15.161:0/1702181954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa36c152890 0x7fa36c0bf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:04.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.045+0000 7fa371eaf640 1 --2- 172.21.15.161:0/1702181954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa36c0c5b20 0x7fa36c1eb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:04.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.045+0000 7fa3716ae640 1 -- 172.21.15.161:0/1702181954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa36c152890 msgr2=0x7fa36c0bf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:04.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.045+0000 7fa3716ae640 1 --2- 172.21.15.161:0/1702181954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa36c152890 0x7fa36c0bf800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.045+0000 7fa3716ae640 1 -- 172.21.15.161:0/1702181954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa36c0c5b20 msgr2=0x7fa36c1eb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:04.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.045+0000 7fa3716ae640 1 --2- 172.21.15.161:0/1702181954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa36c0c5b20 0x7fa36c1eb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.045+0000 7fa3716ae640 1 -- 172.21.15.161:0/1702181954 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa36c1eed40 con 0x7fa36c0c8ed0 2024-03-28T23:31:04.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.045+0000 7fa3716ae640 1 --2- 172.21.15.161:0/1702181954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa36c0c8ed0 0x7fa36c0abb30 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa35c00c760 tx=0x7fa35c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:04.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.045+0000 7fa3627fc640 1 -- 172.21.15.161:0/1702181954 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35c007270 con 0x7fa36c0c8ed0 2024-03-28T23:31:04.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.045+0000 7fa3627fc640 1 -- 172.21.15.161:0/1702181954 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa35c007410 con 0x7fa36c0c8ed0 2024-03-28T23:31:04.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.045+0000 7fa3627fc640 1 -- 172.21.15.161:0/1702181954 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35c001d00 con 0x7fa36c0c8ed0 2024-03-28T23:31:04.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.045+0000 7fa374113640 1 -- 172.21.15.161:0/1702181954 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa36c1ef020 con 0x7fa36c0c8ed0 2024-03-28T23:31:04.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.046+0000 7fa374113640 1 -- 172.21.15.161:0/1702181954 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa36c1ef5e0 con 0x7fa36c0c8ed0 2024-03-28T23:31:04.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.047+0000 7fa374113640 1 -- 172.21.15.161:0/1702181954 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa36c067a50 con 0x7fa36c0c8ed0 2024-03-28T23:31:04.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.047+0000 7fa3627fc640 1 -- 172.21.15.161:0/1702181954 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa35c0057a0 con 0x7fa36c0c8ed0 2024-03-28T23:31:04.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.047+0000 7fa3627fc640 1 --2- 172.21.15.161:0/1702181954 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa35006b4d0 0x7fa35006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:04.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.047+0000 7fa3627fc640 1 -- 172.21.15.161:0/1702181954 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fa35c089f80 con 0x7fa36c0c8ed0 2024-03-28T23:31:04.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.047+0000 7fa371eaf640 1 --2- 172.21.15.161:0/1702181954 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa35006b4d0 0x7fa35006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:04.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.048+0000 7fa371eaf640 1 --2- 172.21.15.161:0/1702181954 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa35006b4d0 0x7fa35006d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa368035000 tx=0x7fa368008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:04.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.052+0000 7fa3627fc640 1 -- 172.21.15.161:0/1702181954 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa36c067a50 con 0x7fa36c0c8ed0 2024-03-28T23:31:04.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.267+0000 7fa374113640 1 -- 172.21.15.161:0/1702181954 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa36c068f90 con 0x7fa36c0c8ed0 2024-03-28T23:31:04.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.267+0000 7fa3627fc640 1 -- 172.21.15.161:0/1702181954 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5181 (secure 0 0 0) 0x7fa36c068f90 con 0x7fa36c0c8ed0 2024-03-28T23:31:04.268 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:04.268 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4853,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/11130685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":11130685},{"type":"v1","addr":"172.21.15.170:6834","nonce":11130685}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5110,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/465771987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":465771987},{"type":"v1","addr":"172.21.15.170:6835","nonce":465771987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5355,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/306644174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":306644174},{"type":"v1","addr":"172.21.15.161:6836","nonce":306644174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:30:57.616423+0000","modified":"2024-03-28T23:31:00.324742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5361},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5361":{"gid":5361,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6837/2636327316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":2636327316},{"type":"v1","addr":"172.21.15.161:6837","nonce":2636327316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5361,"qdb_cluster":[5361]},"id":2}]} 2024-03-28T23:31:04.268 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 17 2024-03-28T23:31:04.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.269+0000 7fa374113640 1 -- 172.21.15.161:0/1702181954 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa35006b4d0 msgr2=0x7fa35006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:04.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.269+0000 7fa374113640 1 --2- 172.21.15.161:0/1702181954 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa35006b4d0 0x7fa35006d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa368035000 tx=0x7fa368008030 comp rx=0 tx=0).stop 2024-03-28T23:31:04.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.269+0000 7fa374113640 1 -- 172.21.15.161:0/1702181954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa36c0c8ed0 msgr2=0x7fa36c0abb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:04.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.269+0000 7fa374113640 1 --2- 172.21.15.161:0/1702181954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa36c0c8ed0 0x7fa36c0abb30 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa35c00c760 tx=0x7fa35c00cc20 comp rx=0 tx=0).stop 2024-03-28T23:31:04.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.269+0000 7fa374113640 1 -- 172.21.15.161:0/1702181954 shutdown_connections 2024-03-28T23:31:04.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.269+0000 7fa374113640 1 --2- 172.21.15.161:0/1702181954 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa35006b4d0 0x7fa35006d990 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.269+0000 7fa374113640 1 --2- 172.21.15.161:0/1702181954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa36c0c8ed0 0x7fa36c0abb30 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.269+0000 7fa374113640 1 --2- 172.21.15.161:0/1702181954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa36c152890 0x7fa36c0bf800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.269+0000 7fa374113640 1 --2- 172.21.15.161:0/1702181954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa36c0c5b20 0x7fa36c1eb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.269+0000 7fa374113640 1 -- 172.21.15.161:0/1702181954 >> 172.21.15.161:0/1702181954 conn(0x7fa36c0ac620 msgr2=0x7fa36c14b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:04.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.269+0000 7fa374113640 1 -- 172.21.15.161:0/1702181954 shutdown_connections 2024-03-28T23:31:04.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.269+0000 7fa374113640 1 -- 172.21.15.161:0/1702181954 wait complete. 2024-03-28T23:31:04.285 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 17, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:30:57.616423+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 17, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5361": { "addr": "172.21.15.161:6837/2636327316", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6835", "nonce": 2636327316, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 2636327316, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5361, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-03-28T23:31:00.324742+0000", "qdb_cluster": [ 5361 ], "qdb_leader": 5361, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5361 } } } ], "standbys": [ { "addr": "172.21.15.170:6834/11130685", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 11130685, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 11130685, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4853, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.170:6835/465771987", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 465771987, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 465771987, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5110, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6836/306644174", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 306644174, "type": "v2" }, { "addr": "172.21.15.161:6836", "nonce": 306644174, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5355, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:31:04.285 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-28T23:31:04.285 INFO:tasks.cephfs.test_failover:current = [{'gid': 5361, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.161:6837/2636327316', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 2636327316}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 2636327316}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 5110, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/465771987', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 465771987}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 465771987}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 13}, {'gid': 5355, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6836/306644174', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 306644174}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 306644174}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 12}, {'gid': 4853, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6834/11130685', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 11130685}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 11130685}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 13}] 2024-03-28T23:31:04.285 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs 2024-03-28T23:31:04.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.409+0000 7fc35702e640 1 Processor -- start 2024-03-28T23:31:04.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.409+0000 7fc35702e640 1 -- start start 2024-03-28T23:31:04.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.410+0000 7fc35702e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc3500c1040 0x7fc3501535b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:04.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.410+0000 7fc35702e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3500a96f0 0x7fc35015d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:04.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.410+0000 7fc35702e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3500c2f50 con 0x7fc350150520 2024-03-28T23:31:04.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.410+0000 7fc35702e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc3500a9c20 con 0x7fc3500c1040 2024-03-28T23:31:04.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.410+0000 7fc35702e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc350150900 con 0x7fc3500a96f0 2024-03-28T23:31:04.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.410+0000 7fc34ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc3500c1040 0x7fc3501535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:04.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.410+0000 7fc354dca640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc350150520 0x7fc3500ab280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41940/0 (socket says 172.21.15.161:41940) 2024-03-28T23:31:04.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.410+0000 7fc354dca640 1 -- 172.21.15.161:0/3039215527 learned_addr learned my addr 172.21.15.161:0/3039215527 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:04.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.410+0000 7fc3555cb640 1 --2- 172.21.15.161:0/3039215527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3500a96f0 0x7fc35015d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:04.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.410+0000 7fc34ffff640 1 -- 172.21.15.161:0/3039215527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3500a96f0 msgr2=0x7fc35015d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:04.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.410+0000 7fc34f7fe640 1 -- 172.21.15.161:0/3039215527 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3963113280 0 0) 0x7fc3500c2f50 con 0x7fc350150520 2024-03-28T23:31:04.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.410+0000 7fc34ffff640 1 --2- 172.21.15.161:0/3039215527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3500a96f0 0x7fc35015d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.410+0000 7fc34ffff640 1 -- 172.21.15.161:0/3039215527 >> v1:172.21.15.161:6789/0 conn(0x7fc350150520 legacy=0x7fc3500ab280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:04.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.410+0000 7fc34ffff640 1 -- 172.21.15.161:0/3039215527 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3501530d0 con 0x7fc3500c1040 2024-03-28T23:31:04.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.411+0000 7fc34ffff640 1 --2- 172.21.15.161:0/3039215527 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3500c1040 0x7fc3501535b0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fc3380097a0 tx=0x7fc33802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7388445820615a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:04.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.411+0000 7fc34f7fe640 1 -- 172.21.15.161:0/3039215527 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3380315a0 con 0x7fc3500c1040 2024-03-28T23:31:04.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.411+0000 7fc34f7fe640 1 -- 172.21.15.161:0/3039215527 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc338037070 con 0x7fc3500c1040 2024-03-28T23:31:04.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.411+0000 7fc34f7fe640 1 -- 172.21.15.161:0/3039215527 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc338031960 con 0x7fc3500c1040 2024-03-28T23:31:04.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.412+0000 7fc35702e640 1 -- 172.21.15.161:0/3039215527 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3500c1040 msgr2=0x7fc3501535b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:04.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.412+0000 7fc35702e640 1 --2- 172.21.15.161:0/3039215527 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3500c1040 0x7fc3501535b0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fc3380097a0 tx=0x7fc33802b840 comp rx=0 tx=0).stop 2024-03-28T23:31:04.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.412+0000 7fc35702e640 1 -- 172.21.15.161:0/3039215527 shutdown_connections 2024-03-28T23:31:04.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.412+0000 7fc35702e640 1 --2- 172.21.15.161:0/3039215527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3500a96f0 0x7fc35015d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.412+0000 7fc35702e640 1 --2- 172.21.15.161:0/3039215527 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3500c1040 0x7fc3501535b0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.412+0000 7fc35702e640 1 -- 172.21.15.161:0/3039215527 >> 172.21.15.161:0/3039215527 conn(0x7fc3500a83f0 msgr2=0x7fc3500a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:04.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.412+0000 7fc35702e640 1 -- 172.21.15.161:0/3039215527 shutdown_connections 2024-03-28T23:31:04.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.412+0000 7fc35702e640 1 -- 172.21.15.161:0/3039215527 wait complete. 2024-03-28T23:31:04.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.413+0000 7fc35702e640 1 Processor -- start 2024-03-28T23:31:04.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.413+0000 7fc35702e640 1 -- start start 2024-03-28T23:31:04.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.414+0000 7fc35702e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3500a96f0 0x7fc350156c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:04.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.414+0000 7fc35702e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3500c1040 0x7fc350142180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:04.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.414+0000 7fc35702e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc350150520 0x7fc3501426c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:04.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.414+0000 7fc35702e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc3500ab990 con 0x7fc3500a96f0 2024-03-28T23:31:04.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.414+0000 7fc35702e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc3500c2c70 con 0x7fc350150520 2024-03-28T23:31:04.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.414+0000 7fc35702e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc3500abb10 con 0x7fc3500c1040 2024-03-28T23:31:04.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.414+0000 7fc354dca640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3500a96f0 0x7fc350156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:04.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.414+0000 7fc354dca640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3500a96f0 0x7fc350156c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45916/0 (socket says 172.21.15.161:45916) 2024-03-28T23:31:04.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.414+0000 7fc354dca640 1 -- 172.21.15.161:0/2597908580 learned_addr learned my addr 172.21.15.161:0/2597908580 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:04.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.414+0000 7fc34ffff640 1 --2- 172.21.15.161:0/2597908580 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3500c1040 0x7fc350142180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:04.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.414+0000 7fc3555cb640 1 --2- 172.21.15.161:0/2597908580 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc350150520 0x7fc3501426c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:04.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.414+0000 7fc354dca640 1 -- 172.21.15.161:0/2597908580 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3500c1040 msgr2=0x7fc350142180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:04.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.414+0000 7fc354dca640 1 --2- 172.21.15.161:0/2597908580 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3500c1040 0x7fc350142180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.414+0000 7fc354dca640 1 -- 172.21.15.161:0/2597908580 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc350150520 msgr2=0x7fc3501426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:04.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.414+0000 7fc354dca640 1 --2- 172.21.15.161:0/2597908580 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc350150520 0x7fc3501426c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.414+0000 7fc354dca640 1 -- 172.21.15.161:0/2597908580 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc350146bb0 con 0x7fc3500a96f0 2024-03-28T23:31:04.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.414+0000 7fc354dca640 1 --2- 172.21.15.161:0/2597908580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3500a96f0 0x7fc350156c20 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc34800c8c0 tx=0x7fc34800cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:04.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.414+0000 7fc34d7fa640 1 -- 172.21.15.161:0/2597908580 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc348018070 con 0x7fc3500a96f0 2024-03-28T23:31:04.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.414+0000 7fc34d7fa640 1 -- 172.21.15.161:0/2597908580 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc34800a640 con 0x7fc3500a96f0 2024-03-28T23:31:04.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.414+0000 7fc35702e640 1 -- 172.21.15.161:0/2597908580 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3501552e0 con 0x7fc3500a96f0 2024-03-28T23:31:04.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.414+0000 7fc34d7fa640 1 -- 172.21.15.161:0/2597908580 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34800aa30 con 0x7fc3500a96f0 2024-03-28T23:31:04.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.414+0000 7fc35702e640 1 -- 172.21.15.161:0/2597908580 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc350155860 con 0x7fc3500a96f0 2024-03-28T23:31:04.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.415+0000 7fc35702e640 1 -- 172.21.15.161:0/2597908580 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc35004f4b0 con 0x7fc3500a96f0 2024-03-28T23:31:04.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.416+0000 7fc34d7fa640 1 -- 172.21.15.161:0/2597908580 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc34801b070 con 0x7fc3500a96f0 2024-03-28T23:31:04.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.416+0000 7fc34d7fa640 1 --2- 172.21.15.161:0/2597908580 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc33006b500 0x7fc33006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:04.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.416+0000 7fc34d7fa640 1 -- 172.21.15.161:0/2597908580 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fc3480899e0 con 0x7fc3500a96f0 2024-03-28T23:31:04.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.416+0000 7fc34ffff640 1 --2- 172.21.15.161:0/2597908580 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc33006b500 0x7fc33006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:04.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.417+0000 7fc34ffff640 1 --2- 172.21.15.161:0/2597908580 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc33006b500 0x7fc33006d9c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc338036000 tx=0x7fc338008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:04.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.421+0000 7fc34d7fa640 1 -- 172.21.15.161:0/2597908580 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc3480586d0 con 0x7fc3500a96f0 2024-03-28T23:31:04.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.618+0000 7fc35702e640 1 -- 172.21.15.161:0/2597908580 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) -- 0x7fc3500c2640 con 0x7fc3500a96f0 2024-03-28T23:31:04.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.621+0000 7fc34d7fa640 1 -- 172.21.15.161:0/2597908580 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v12)=0 v12) ==== 116+0+0 (secure 0 0 0) 0x7fc348057fd0 con 0x7fc3500a96f0 2024-03-28T23:31:04.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.623+0000 7fc35702e640 1 -- 172.21.15.161:0/2597908580 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc33006b500 msgr2=0x7fc33006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:04.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.623+0000 7fc35702e640 1 --2- 172.21.15.161:0/2597908580 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc33006b500 0x7fc33006d9c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc338036000 tx=0x7fc338008030 comp rx=0 tx=0).stop 2024-03-28T23:31:04.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.623+0000 7fc35702e640 1 -- 172.21.15.161:0/2597908580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3500a96f0 msgr2=0x7fc350156c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:04.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.623+0000 7fc35702e640 1 --2- 172.21.15.161:0/2597908580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3500a96f0 0x7fc350156c20 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc34800c8c0 tx=0x7fc34800cd80 comp rx=0 tx=0).stop 2024-03-28T23:31:04.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.623+0000 7fc35702e640 1 -- 172.21.15.161:0/2597908580 shutdown_connections 2024-03-28T23:31:04.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.623+0000 7fc35702e640 1 --2- 172.21.15.161:0/2597908580 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc33006b500 0x7fc33006d9c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.623+0000 7fc35702e640 1 --2- 172.21.15.161:0/2597908580 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc350150520 0x7fc3501426c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.623+0000 7fc35702e640 1 --2- 172.21.15.161:0/2597908580 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3500c1040 0x7fc350142180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.623+0000 7fc35702e640 1 --2- 172.21.15.161:0/2597908580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3500a96f0 0x7fc350156c20 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.623+0000 7fc35702e640 1 -- 172.21.15.161:0/2597908580 >> 172.21.15.161:0/2597908580 conn(0x7fc3500a83f0 msgr2=0x7fc3500ab280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:04.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.623+0000 7fc35702e640 1 -- 172.21.15.161:0/2597908580 shutdown_connections 2024-03-28T23:31:04.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.623+0000 7fc35702e640 1 -- 172.21.15.161:0/2597908580 wait complete. 2024-03-28T23:31:04.635 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:31:04.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.736+0000 7f5a1592c640 1 Processor -- start 2024-03-28T23:31:04.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.736+0000 7f5a1592c640 1 -- start start 2024-03-28T23:31:04.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.737+0000 7f5a1592c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5a10152890 0x7f5a100c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:04.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.737+0000 7f5a1592c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a100c5b20 0x7f5a101553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:04.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.737+0000 7f5a1592c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a10158a30 con 0x7f5a100c8ed0 2024-03-28T23:31:04.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.737+0000 7f5a1592c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f5a100c8480 con 0x7f5a10152890 2024-03-28T23:31:04.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.737+0000 7f5a1592c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5a10159640 con 0x7f5a100c5b20 2024-03-28T23:31:04.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.737+0000 7f5a0f7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a100c5b20 0x7f5a101553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:04.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.737+0000 7f5a0effd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5a10152890 0x7f5a100c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:04.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.738+0000 7f5a0e7fc640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f5a100c8ed0 0x7f5a100c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:55906/0 (socket says 172.21.15.161:55906) 2024-03-28T23:31:04.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.738+0000 7f5a0effd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5a10152890 0x7f5a100c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:57534/0 (socket says 172.21.15.161:57534) 2024-03-28T23:31:04.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.738+0000 7f5a0e7fc640 1 -- 172.21.15.161:0/4045596210 learned_addr learned my addr 172.21.15.161:0/4045596210 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:04.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.738+0000 7f5a0f7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a100c5b20 0x7f5a101553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:56206/0 (socket says 172.21.15.161:56206) 2024-03-28T23:31:04.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.738+0000 7f5a0dffb640 1 -- 172.21.15.161:0/4045596210 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1962375649 0 0) 0x7f5a10158a30 con 0x7f5a100c8ed0 2024-03-28T23:31:04.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.738+0000 7f5a0dffb640 1 -- 172.21.15.161:0/4045596210 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59fc003520 con 0x7f5a100c8ed0 2024-03-28T23:31:04.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.738+0000 7f5a0effd640 1 -- 172.21.15.161:0/4045596210 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a100c5b20 msgr2=0x7f5a101553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:04.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.738+0000 7f5a0effd640 1 --2- 172.21.15.161:0/4045596210 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a100c5b20 0x7f5a101553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.738+0000 7f5a0effd640 1 -- 172.21.15.161:0/4045596210 >> v1:172.21.15.170:6789/0 conn(0x7f5a100c8ed0 legacy=0x7f5a100c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:04.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.738+0000 7f5a0effd640 1 -- 172.21.15.161:0/4045596210 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a10158d10 con 0x7f5a10152890 2024-03-28T23:31:04.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.738+0000 7f5a0f7fe640 1 --2- 172.21.15.161:0/4045596210 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a100c5b20 0x7f5a101553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:04.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.739+0000 7f5a0effd640 1 --2- 172.21.15.161:0/4045596210 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a10152890 0x7f5a100c8990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5a101597c0 tx=0x7f5a0802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb64ac6e2f5a3772 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:04.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.739+0000 7f5a0dffb640 1 -- 172.21.15.161:0/4045596210 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a08039070 con 0x7f5a10152890 2024-03-28T23:31:04.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.739+0000 7f5a0dffb640 1 -- 172.21.15.161:0/4045596210 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a080316f0 con 0x7f5a10152890 2024-03-28T23:31:04.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.739+0000 7f5a0dffb640 1 -- 172.21.15.161:0/4045596210 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a08031aa0 con 0x7f5a10152890 2024-03-28T23:31:04.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.739+0000 7f5a1592c640 1 -- 172.21.15.161:0/4045596210 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a10152890 msgr2=0x7f5a100c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:04.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.739+0000 7f5a1592c640 1 --2- 172.21.15.161:0/4045596210 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a10152890 0x7f5a100c8990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5a101597c0 tx=0x7f5a0802b790 comp rx=0 tx=0).stop 2024-03-28T23:31:04.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.739+0000 7f5a1592c640 1 -- 172.21.15.161:0/4045596210 shutdown_connections 2024-03-28T23:31:04.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.739+0000 7f5a1592c640 1 --2- 172.21.15.161:0/4045596210 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a100c5b20 0x7f5a101553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.739+0000 7f5a1592c640 1 --2- 172.21.15.161:0/4045596210 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a10152890 0x7f5a100c8990 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.739+0000 7f5a1592c640 1 -- 172.21.15.161:0/4045596210 >> 172.21.15.161:0/4045596210 conn(0x7f5a100ac620 msgr2=0x7f5a100aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:04.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.739+0000 7f5a1592c640 1 -- 172.21.15.161:0/4045596210 shutdown_connections 2024-03-28T23:31:04.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.740+0000 7f5a1592c640 1 -- 172.21.15.161:0/4045596210 wait complete. 2024-03-28T23:31:04.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.741+0000 7f5a1592c640 1 Processor -- start 2024-03-28T23:31:04.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.741+0000 7f5a1592c640 1 -- start start 2024-03-28T23:31:04.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.741+0000 7f5a1592c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a100c5b20 0x7f5a101e7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:04.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.741+0000 7f5a1592c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a100c8ed0 0x7f5a101e8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:04.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.741+0000 7f5a1592c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a10152890 0x7f5a100be830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:04.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.741+0000 7f5a1592c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5a10152c70 con 0x7f5a10152890 2024-03-28T23:31:04.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.741+0000 7f5a1592c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5a101592d0 con 0x7f5a100c5b20 2024-03-28T23:31:04.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.741+0000 7f5a1592c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5a10152df0 con 0x7f5a100c8ed0 2024-03-28T23:31:04.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.742+0000 7f5a0f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a10152890 0x7f5a100be830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:04.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.742+0000 7f5a0f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a10152890 0x7f5a100be830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45930/0 (socket says 172.21.15.161:45930) 2024-03-28T23:31:04.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.742+0000 7f5a0f7fe640 1 -- 172.21.15.161:0/1419038814 learned_addr learned my addr 172.21.15.161:0/1419038814 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:04.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.742+0000 7f5a0effd640 1 --2- 172.21.15.161:0/1419038814 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a100c5b20 0x7f5a101e7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:04.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.742+0000 7f5a0e7fc640 1 --2- 172.21.15.161:0/1419038814 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a100c8ed0 0x7f5a101e8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:04.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.742+0000 7f5a0f7fe640 1 -- 172.21.15.161:0/1419038814 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a100c8ed0 msgr2=0x7f5a101e8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:04.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.742+0000 7f5a0f7fe640 1 --2- 172.21.15.161:0/1419038814 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a100c8ed0 0x7f5a101e8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.742+0000 7f5a0f7fe640 1 -- 172.21.15.161:0/1419038814 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a100c5b20 msgr2=0x7f5a101e7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:04.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.742+0000 7f5a0f7fe640 1 --2- 172.21.15.161:0/1419038814 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a100c5b20 0x7f5a101e7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.742+0000 7f5a0f7fe640 1 -- 172.21.15.161:0/1419038814 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a101eef40 con 0x7f5a10152890 2024-03-28T23:31:04.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.742+0000 7f5a0e7fc640 1 --2- 172.21.15.161:0/1419038814 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a100c8ed0 0x7f5a101e8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:04.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.742+0000 7f5a0f7fe640 1 --2- 172.21.15.161:0/1419038814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a10152890 0x7f5a100be830 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f5a0000efc0 tx=0x7f5a0000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:04.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.742+0000 7f5a07fff640 1 -- 172.21.15.161:0/1419038814 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a00018020 con 0x7f5a10152890 2024-03-28T23:31:04.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.742+0000 7f5a07fff640 1 -- 172.21.15.161:0/1419038814 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a00014480 con 0x7f5a10152890 2024-03-28T23:31:04.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.742+0000 7f5a07fff640 1 -- 172.21.15.161:0/1419038814 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a00014870 con 0x7f5a10152890 2024-03-28T23:31:04.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.742+0000 7f5a1592c640 1 -- 172.21.15.161:0/1419038814 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a101eec60 con 0x7f5a10152890 2024-03-28T23:31:04.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.742+0000 7f5a1592c640 1 -- 172.21.15.161:0/1419038814 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a1014ceb0 con 0x7f5a10152890 2024-03-28T23:31:04.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.743+0000 7f5a1592c640 1 -- 172.21.15.161:0/1419038814 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a10152fa0 con 0x7f5a10152890 2024-03-28T23:31:04.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.744+0000 7f5a07fff640 1 -- 172.21.15.161:0/1419038814 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5a00005d10 con 0x7f5a10152890 2024-03-28T23:31:04.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.744+0000 7f5a07fff640 1 --2- 172.21.15.161:0/1419038814 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f59f006b4d0 0x7f59f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:04.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.744+0000 7f5a07fff640 1 -- 172.21.15.161:0/1419038814 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f5a0008af90 con 0x7f5a10152890 2024-03-28T23:31:04.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.744+0000 7f5a0effd640 1 --2- 172.21.15.161:0/1419038814 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f59f006b4d0 0x7f59f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:04.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.745+0000 7f5a0effd640 1 --2- 172.21.15.161:0/1419038814 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f59f006b4d0 0x7f59f006d990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5a08036000 tx=0x7f5a08008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:04.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.749+0000 7f5a07fff640 1 -- 172.21.15.161:0/1419038814 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5a10152fa0 con 0x7f5a10152890 2024-03-28T23:31:04.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.963+0000 7f5a1592c640 1 -- 172.21.15.161:0/1419038814 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a1014d0b0 con 0x7f5a10152890 2024-03-28T23:31:04.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.963+0000 7f5a07fff640 1 -- 172.21.15.161:0/1419038814 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5181 (secure 0 0 0) 0x7f5a1014d0b0 con 0x7f5a10152890 2024-03-28T23:31:04.964 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:04.965 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4853,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/11130685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":11130685},{"type":"v1","addr":"172.21.15.170:6834","nonce":11130685}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5110,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/465771987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":465771987},{"type":"v1","addr":"172.21.15.170:6835","nonce":465771987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5355,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/306644174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":306644174},{"type":"v1","addr":"172.21.15.161:6836","nonce":306644174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:30:57.616423+0000","modified":"2024-03-28T23:31:00.324742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5361},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5361":{"gid":5361,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6837/2636327316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":2636327316},{"type":"v1","addr":"172.21.15.161:6837","nonce":2636327316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5361,"qdb_cluster":[5361]},"id":2}]} 2024-03-28T23:31:04.965 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 17 2024-03-28T23:31:04.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.965+0000 7f5a1592c640 1 -- 172.21.15.161:0/1419038814 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f59f006b4d0 msgr2=0x7f59f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:04.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.965+0000 7f5a1592c640 1 --2- 172.21.15.161:0/1419038814 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f59f006b4d0 0x7f59f006d990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5a08036000 tx=0x7f5a08008030 comp rx=0 tx=0).stop 2024-03-28T23:31:04.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.965+0000 7f5a1592c640 1 -- 172.21.15.161:0/1419038814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a10152890 msgr2=0x7f5a100be830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:04.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.965+0000 7f5a1592c640 1 --2- 172.21.15.161:0/1419038814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a10152890 0x7f5a100be830 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f5a0000efc0 tx=0x7f5a0000c490 comp rx=0 tx=0).stop 2024-03-28T23:31:04.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.965+0000 7f5a1592c640 1 -- 172.21.15.161:0/1419038814 shutdown_connections 2024-03-28T23:31:04.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.965+0000 7f5a1592c640 1 --2- 172.21.15.161:0/1419038814 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f59f006b4d0 0x7f59f006d990 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.965+0000 7f5a1592c640 1 --2- 172.21.15.161:0/1419038814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a10152890 0x7f5a100be830 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.965+0000 7f5a1592c640 1 --2- 172.21.15.161:0/1419038814 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a100c8ed0 0x7f5a101e8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.965+0000 7f5a1592c640 1 --2- 172.21.15.161:0/1419038814 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a100c5b20 0x7f5a101e7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:04.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.965+0000 7f5a1592c640 1 -- 172.21.15.161:0/1419038814 >> 172.21.15.161:0/1419038814 conn(0x7f5a100ac620 msgr2=0x7f5a101512a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:04.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.965+0000 7f5a1592c640 1 -- 172.21.15.161:0/1419038814 shutdown_connections 2024-03-28T23:31:04.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:04.965+0000 7f5a1592c640 1 -- 172.21.15.161:0/1419038814 wait complete. 2024-03-28T23:31:05.380 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:30:57.616423+0000', 'modified': '2024-03-28T23:31:00.324742+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5361}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5361': {'gid': 5361, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.161:6837/2636327316', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 2636327316}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 2636327316}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5361, 'qdb_cluster': [5361]} 2024-03-28T23:31:05.380 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:31:05.380 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-28T23:31:05.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.515+0000 7f9e19da1640 1 Processor -- start 2024-03-28T23:31:05.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.516+0000 7f9e19da1640 1 -- start start 2024-03-28T23:31:05.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.516+0000 7f9e19da1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e140c5850 0x7f9e14155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:05.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.516+0000 7f9e19da1640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e140c5d00 con 0x7f9e14152890 2024-03-28T23:31:05.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.516+0000 7f9e19da1640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e14158a30 con 0x7f9e140c5410 2024-03-28T23:31:05.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.516+0000 7f9e19da1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9e140c8480 con 0x7f9e140c5850 2024-03-28T23:31:05.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.516+0000 7f9e137fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f9e14152890 0x7f9e140c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41950/0 (socket says 172.21.15.161:41950) 2024-03-28T23:31:05.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.516+0000 7f9e137fe640 1 -- 172.21.15.161:0/949061493 learned_addr learned my addr 172.21.15.161:0/949061493 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:05.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.517+0000 7f9e13fff640 1 --2- 172.21.15.161:0/949061493 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e140c5850 0x7f9e14155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:05.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.517+0000 7f9e127fc640 1 -- 172.21.15.161:0/949061493 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2105858534 0 0) 0x7f9e140c5d00 con 0x7f9e14152890 2024-03-28T23:31:05.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.517+0000 7f9e127fc640 1 -- 172.21.15.161:0/949061493 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e00003520 con 0x7f9e14152890 2024-03-28T23:31:05.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.517+0000 7f9e127fc640 1 -- 172.21.15.161:0/949061493 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1949821684 0 0) 0x7f9e00003520 con 0x7f9e14152890 2024-03-28T23:31:05.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.517+0000 7f9e127fc640 1 -- 172.21.15.161:0/949061493 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e140c5850 msgr2=0x7f9e14155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:05.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.517+0000 7f9e127fc640 1 --2- 172.21.15.161:0/949061493 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e140c5850 0x7f9e14155ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.517+0000 7f9e127fc640 1 -- 172.21.15.161:0/949061493 >> v1:172.21.15.170:6789/0 conn(0x7f9e140c5410 legacy=0x7f9e141553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:05.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.517+0000 7f9e127fc640 1 -- 172.21.15.161:0/949061493 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e14158d10 con 0x7f9e14152890 2024-03-28T23:31:05.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.517+0000 7f9e127fc640 1 -- 172.21.15.161:0/949061493 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f9e040019f0 con 0x7f9e14152890 2024-03-28T23:31:05.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.517+0000 7f9e127fc640 1 -- 172.21.15.161:0/949061493 >> v1:172.21.15.161:6789/0 conn(0x7f9e14152890 legacy=0x7f9e140c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:05.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.517+0000 7f9e127fc640 1 --2- 172.21.15.161:0/949061493 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9e00003ac0 0x7f9e00003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:05.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.517+0000 7f9e127fc640 1 --2- 172.21.15.161:0/949061493 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e00004800 0x7f9e00004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:05.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.517+0000 7f9e127fc640 1 --2- 172.21.15.161:0/949061493 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e140c5850 0x7f9e0000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:05.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.517+0000 7f9e127fc640 1 -- 172.21.15.161:0/949061493 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9e00005520 con 0x7f9e00004800 2024-03-28T23:31:05.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.517+0000 7f9e127fc640 1 -- 172.21.15.161:0/949061493 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0000bf00 con 0x7f9e00003ac0 2024-03-28T23:31:05.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.517+0000 7f9e127fc640 1 -- 172.21.15.161:0/949061493 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9e0000c080 con 0x7f9e140c5850 2024-03-28T23:31:05.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.517+0000 7f9e12ffd640 1 --2- 172.21.15.161:0/949061493 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e00004800 0x7f9e00004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:05.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.517+0000 7f9e137fe640 1 --2- 172.21.15.161:0/949061493 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9e00003ac0 0x7f9e00003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:05.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.517+0000 7f9e13fff640 1 --2- 172.21.15.161:0/949061493 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e140c5850 0x7f9e0000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:05.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.517+0000 7f9e12ffd640 1 -- 172.21.15.161:0/949061493 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e140c5850 msgr2=0x7f9e0000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:05.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.517+0000 7f9e12ffd640 1 --2- 172.21.15.161:0/949061493 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e140c5850 0x7f9e0000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.517+0000 7f9e12ffd640 1 -- 172.21.15.161:0/949061493 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9e00003ac0 msgr2=0x7f9e00003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:05.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.517+0000 7f9e12ffd640 1 --2- 172.21.15.161:0/949061493 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9e00003ac0 0x7f9e00003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.517+0000 7f9e12ffd640 1 -- 172.21.15.161:0/949061493 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e00003520 con 0x7f9e00004800 2024-03-28T23:31:05.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.518+0000 7f9e12ffd640 1 --2- 172.21.15.161:0/949061493 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e00004800 0x7f9e00004c80 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f9e140ad1f0 tx=0x7f9e00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=512d33d598989614 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:05.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.518+0000 7f9e127fc640 1 -- 172.21.15.161:0/949061493 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc0013c0 con 0x7f9e00004800 2024-03-28T23:31:05.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.518+0000 7f9e127fc640 1 -- 172.21.15.161:0/949061493 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dfc0052d0 con 0x7f9e00004800 2024-03-28T23:31:05.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.518+0000 7f9e127fc640 1 -- 172.21.15.161:0/949061493 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc003640 con 0x7f9e00004800 2024-03-28T23:31:05.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.519+0000 7f9e19da1640 1 -- 172.21.15.161:0/949061493 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e00004800 msgr2=0x7f9e00004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:05.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.519+0000 7f9e19da1640 1 --2- 172.21.15.161:0/949061493 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e00004800 0x7f9e00004c80 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f9e140ad1f0 tx=0x7f9e00005830 comp rx=0 tx=0).stop 2024-03-28T23:31:05.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.519+0000 7f9e13fff640 1 -- 172.21.15.161:0/949061493 reap_dead start 2024-03-28T23:31:05.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.519+0000 7f9e19da1640 1 -- 172.21.15.161:0/949061493 shutdown_connections 2024-03-28T23:31:05.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.519+0000 7f9e19da1640 1 -- 172.21.15.161:0/949061493 >> 172.21.15.161:0/949061493 conn(0x7f9e140ac620 msgr2=0x7f9e140aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:05.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.519+0000 7f9e19da1640 1 -- 172.21.15.161:0/949061493 shutdown_connections 2024-03-28T23:31:05.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.519+0000 7f9e19da1640 1 -- 172.21.15.161:0/949061493 wait complete. 2024-03-28T23:31:05.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.520+0000 7f9e19da1640 1 Processor -- start 2024-03-28T23:31:05.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.520+0000 7f9e19da1640 1 -- start start 2024-03-28T23:31:05.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.521+0000 7f9e19da1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e00004800 0x7f9e1414ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:05.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.521+0000 7f9e19da1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9e140c0c60 0x7f9e14147ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:05.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.521+0000 7f9e19da1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e14148220 0x7f9e141eb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:05.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.521+0000 7f9e19da1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9e141592d0 con 0x7f9e00004800 2024-03-28T23:31:05.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.521+0000 7f9e19da1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9e14159640 con 0x7f9e140c0c60 2024-03-28T23:31:05.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.521+0000 7f9e19da1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9e140c90a0 con 0x7f9e14148220 2024-03-28T23:31:05.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.521+0000 7f9e137fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e00004800 0x7f9e1414ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:05.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.521+0000 7f9e137fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e00004800 0x7f9e1414ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45942/0 (socket says 172.21.15.161:45942) 2024-03-28T23:31:05.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.521+0000 7f9e137fe640 1 -- 172.21.15.161:0/860531861 learned_addr learned my addr 172.21.15.161:0/860531861 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:05.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.521+0000 7f9e13fff640 1 --2- 172.21.15.161:0/860531861 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e14148220 0x7f9e141eb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:05.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.521+0000 7f9e12ffd640 1 --2- 172.21.15.161:0/860531861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9e140c0c60 0x7f9e14147ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:05.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.521+0000 7f9e137fe640 1 -- 172.21.15.161:0/860531861 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e14148220 msgr2=0x7f9e141eb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:05.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.521+0000 7f9e137fe640 1 --2- 172.21.15.161:0/860531861 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e14148220 0x7f9e141eb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.521+0000 7f9e137fe640 1 -- 172.21.15.161:0/860531861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9e140c0c60 msgr2=0x7f9e14147ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:05.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.521+0000 7f9e137fe640 1 --2- 172.21.15.161:0/860531861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9e140c0c60 0x7f9e14147ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.521+0000 7f9e137fe640 1 -- 172.21.15.161:0/860531861 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e140c18d0 con 0x7f9e00004800 2024-03-28T23:31:05.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.521+0000 7f9e137fe640 1 --2- 172.21.15.161:0/860531861 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e00004800 0x7f9e1414ef20 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f9e040077d0 tx=0x7f9e0400a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:05.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.522+0000 7f9e0bfff640 1 -- 172.21.15.161:0/860531861 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e04007ad0 con 0x7f9e00004800 2024-03-28T23:31:05.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.522+0000 7f9e0bfff640 1 -- 172.21.15.161:0/860531861 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e04018070 con 0x7f9e00004800 2024-03-28T23:31:05.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.522+0000 7f9e19da1640 1 -- 172.21.15.161:0/860531861 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e140becb0 con 0x7f9e00004800 2024-03-28T23:31:05.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.522+0000 7f9e0bfff640 1 -- 172.21.15.161:0/860531861 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e04013610 con 0x7f9e00004800 2024-03-28T23:31:05.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.522+0000 7f9e19da1640 1 -- 172.21.15.161:0/860531861 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e140c15f0 con 0x7f9e00004800 2024-03-28T23:31:05.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.524+0000 7f9e0bfff640 1 -- 172.21.15.161:0/860531861 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9e040137b0 con 0x7f9e00004800 2024-03-28T23:31:05.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.524+0000 7f9e0bfff640 1 --2- 172.21.15.161:0/860531861 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9df406b500 0x7f9df406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:05.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.524+0000 7f9e0bfff640 1 -- 172.21.15.161:0/860531861 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f9e0408ab40 con 0x7f9e00004800 2024-03-28T23:31:05.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.524+0000 7f9e12ffd640 1 --2- 172.21.15.161:0/860531861 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9df406b500 0x7f9df406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:05.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.524+0000 7f9e12ffd640 1 --2- 172.21.15.161:0/860531861 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9df406b500 0x7f9df406d9c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc033000 tx=0x7f9dfc009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:05.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.525+0000 7f9e1c985740 1 Processor -- start 2024-03-28T23:31:05.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.525+0000 7f9e1c985740 1 -- start start 2024-03-28T23:31:05.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.525+0000 7f9e1c985740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558804992db0 0x558804990890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:05.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.525+0000 7f9e1c985740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558804993190 0x558804990dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:05.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.525+0000 7f9e1c985740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558804991310 0x55880498d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:05.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.525+0000 7f9e1c985740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5588048f6af0 con 0x558804993190 2024-03-28T23:31:05.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.526+0000 7f9e1c985740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55880489f9d0 con 0x558804991310 2024-03-28T23:31:05.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.526+0000 7f9e1c985740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x5588048f2b90 con 0x558804992db0 2024-03-28T23:31:05.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.526+0000 7f9e12ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558804993190 0x558804990dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:05.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.526+0000 7f9e12ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558804993190 0x558804990dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45952/0 (socket says 172.21.15.161:45952) 2024-03-28T23:31:05.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.526+0000 7f9e12ffd640 1 -- 172.21.15.161:0/457538485 learned_addr learned my addr 172.21.15.161:0/457538485 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:05.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.526+0000 7f9e137fe640 1 --2- 172.21.15.161:0/457538485 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558804992db0 0x558804990890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:05.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.526+0000 7f9e13fff640 1 --2- 172.21.15.161:0/457538485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558804991310 0x55880498d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:05.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.526+0000 7f9e12ffd640 1 -- 172.21.15.161:0/457538485 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558804992db0 msgr2=0x558804990890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:05.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.526+0000 7f9e12ffd640 1 --2- 172.21.15.161:0/457538485 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558804992db0 0x558804990890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.526+0000 7f9e12ffd640 1 -- 172.21.15.161:0/457538485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558804991310 msgr2=0x55880498d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:05.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.526+0000 7f9e12ffd640 1 --2- 172.21.15.161:0/457538485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558804991310 0x55880498d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.526+0000 7f9e12ffd640 1 -- 172.21.15.161:0/457538485 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55880460b2d0 con 0x558804993190 2024-03-28T23:31:05.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.526+0000 7f9e13fff640 1 --2- 172.21.15.161:0/457538485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558804991310 0x55880498d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:05.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.526+0000 7f9e12ffd640 1 --2- 172.21.15.161:0/457538485 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558804993190 0x558804990dd0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc005040 tx=0x7f9dfc02f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84493f8d38b3d8fe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:05.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.526+0000 7f9e097fa640 1 -- 172.21.15.161:0/457538485 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc043040 con 0x558804993190 2024-03-28T23:31:05.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.526+0000 7f9e097fa640 1 -- 172.21.15.161:0/457538485 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dfc007240 con 0x558804993190 2024-03-28T23:31:05.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.526+0000 7f9e097fa640 1 -- 172.21.15.161:0/457538485 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc0075f0 con 0x558804993190 2024-03-28T23:31:05.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.527+0000 7f9e1c985740 1 -- 172.21.15.161:0/457538485 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558804993190 msgr2=0x558804990dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:05.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.527+0000 7f9e1c985740 1 --2- 172.21.15.161:0/457538485 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558804993190 0x558804990dd0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc005040 tx=0x7f9dfc02f000 comp rx=0 tx=0).stop 2024-03-28T23:31:05.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.527+0000 7f9e1c985740 1 -- 172.21.15.161:0/457538485 shutdown_connections 2024-03-28T23:31:05.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.527+0000 7f9e1c985740 1 --2- 172.21.15.161:0/457538485 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558804993190 0x558804990dd0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.527+0000 7f9e1c985740 1 --2- 172.21.15.161:0/457538485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558804991310 0x55880498d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.527+0000 7f9e1c985740 1 --2- 172.21.15.161:0/457538485 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558804992db0 0x558804990890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.527+0000 7f9e1c985740 1 -- 172.21.15.161:0/457538485 >> 172.21.15.161:0/457538485 conn(0x55880498f630 msgr2=0x5588049b4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:05.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.527+0000 7f9e1c985740 1 -- 172.21.15.161:0/457538485 shutdown_connections 2024-03-28T23:31:05.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.527+0000 7f9e1c985740 1 -- 172.21.15.161:0/457538485 wait complete. 2024-03-28T23:31:05.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.528+0000 7f9e1c985740 1 -- start start 2024-03-28T23:31:05.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.528+0000 7f9e1c985740 1 Processor -- start 2024-03-28T23:31:05.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.528+0000 7f9e1c985740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558804992db0 0x5588049b32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:05.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.528+0000 7f9e1c985740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558804993190 0x5588049b5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:05.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.528+0000 7f9e1c985740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5588049c7430 0x5588049cae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:05.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.528+0000 7f9e1c985740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5588049b5090 con 0x558804992db0 2024-03-28T23:31:05.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.528+0000 7f9e1c985740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5588048b2f70 con 0x558804993190 2024-03-28T23:31:05.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.528+0000 7f9e1c985740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x5588049b4ad0 con 0x5588049c7430 2024-03-28T23:31:05.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.528+0000 7f9e137fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558804992db0 0x5588049b32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:05.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.528+0000 7f9e1c985740 10 client.0 _ll_register_callbacks cb 0x55880460aac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:31:05.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.528+0000 7f9e137fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558804992db0 0x5588049b32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45960/0 (socket says 172.21.15.161:45960) 2024-03-28T23:31:05.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.529+0000 7f9e137fe640 1 -- 172.21.15.161:0/1233953733 learned_addr learned my addr 172.21.15.161:0/1233953733 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:05.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.529+0000 7f9e13fff640 1 --2- 172.21.15.161:0/1233953733 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5588049c7430 0x5588049cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:05.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.529+0000 7f9e12ffd640 1 --2- 172.21.15.161:0/1233953733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558804993190 0x5588049b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:05.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.529+0000 7f9e137fe640 1 -- 172.21.15.161:0/1233953733 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5588049c7430 msgr2=0x5588049cae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:05.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.529+0000 7f9e137fe640 1 --2- 172.21.15.161:0/1233953733 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5588049c7430 0x5588049cae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.529+0000 7f9e137fe640 1 -- 172.21.15.161:0/1233953733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558804993190 msgr2=0x5588049b5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:05.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.529+0000 7f9e137fe640 1 --2- 172.21.15.161:0/1233953733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558804993190 0x5588049b5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.529+0000 7f9e137fe640 1 -- 172.21.15.161:0/1233953733 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588046d56b0 con 0x558804992db0 2024-03-28T23:31:05.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.529+0000 7f9e137fe640 1 --2- 172.21.15.161:0/1233953733 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558804992db0 0x5588049b32b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f9e04059150 tx=0x7f9e040591c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eda1f0ce0f0ccb6f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:05.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.529+0000 7f9df27fc640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:31:05.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.529+0000 7f9df27fc640 1 -- 172.21.15.161:0/1233953733 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e04087060 con 0x558804992db0 2024-03-28T23:31:05.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.529+0000 7f9df27fc640 1 -- 172.21.15.161:0/1233953733 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f9e04088a40 con 0x558804992db0 2024-03-28T23:31:05.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.529+0000 7f9e1c985740 1 -- 172.21.15.161:0/1233953733 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9dfc007240 con 0x558804992db0 2024-03-28T23:31:05.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.529+0000 7f9df27fc640 1 -- 172.21.15.161:0/1233953733 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f9e04075050 con 0x558804992db0 2024-03-28T23:31:05.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.529+0000 7f9e1c985740 10 client.5577 fetch_fsmap learned FSMap version 17 2024-03-28T23:31:05.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.529+0000 7f9e1c985740 1 -- 172.21.15.161:0/1233953733 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x5588049b5f20 con 0x558804992db0 2024-03-28T23:31:05.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.529+0000 7f9df27fc640 1 -- 172.21.15.161:0/1233953733 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dfc007240 con 0x558804992db0 2024-03-28T23:31:05.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.529+0000 7f9df27fc640 1 -- 172.21.15.161:0/1233953733 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7f9e0405f2d0 con 0x558804992db0 2024-03-28T23:31:05.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.529+0000 7f9e1c985740 10 client.5577 fetch_fsmap finished waiting for FSMap version 17 2024-03-28T23:31:05.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.529+0000 7f9e1c985740 10 client.5577 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T23:31:05.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.530+0000 7f9e1c985740 20 client.5577 populate_metadata read hostname 'smithi161' 2024-03-28T23:31:05.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.530+0000 7f9e1c985740 1 --2- 172.21.15.161:0/1233953733 >> [v2:172.21.15.161:6835/2636327316,v1:172.21.15.161:6837/2636327316] conn(0x5588049cb380 0x5588049e2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:05.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.530+0000 7f9e1c985740 4 client.5577 mds_command: new command op to 5361 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:31:05.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.530+0000 7f9e1c985740 1 -- 172.21.15.161:0/1233953733 --> [v2:172.21.15.161:6835/2636327316,v1:172.21.15.161:6837/2636327316] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558804904030 con 0x5588049cb380 2024-03-28T23:31:05.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.530+0000 7f9e12ffd640 1 --2- 172.21.15.161:0/1233953733 >> [v2:172.21.15.161:6835/2636327316,v1:172.21.15.161:6837/2636327316] conn(0x5588049cb380 0x5588049e2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:05.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.530+0000 7f9df27fc640 10 client.5577 ms_handle_connect on v2:172.21.15.161:6835/2636327316 2024-03-28T23:31:05.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.530+0000 7f9e12ffd640 1 --2- 172.21.15.161:0/1233953733 >> [v2:172.21.15.161:6835/2636327316,v1:172.21.15.161:6837/2636327316] conn(0x5588049cb380 0x5588049e2ff0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1b4d1514563a779 server_cookie=1d5a96b3574dca26 in_seq=0 out_seq=0 2024-03-28T23:31:05.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.531+0000 7f9df27fc640 1 -- 172.21.15.161:0/1233953733 <== mds.0 v2:172.21.15.161:6835/2636327316 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558804904030 con 0x5588049cb380 2024-03-28T23:31:05.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.531+0000 7f9df27fc640 10 client.5577 handle_command_reply: tid=0 2024-03-28T23:31:05.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.531+0000 7f9e1c985740 1 client.5577 shutdown 2024-03-28T23:31:05.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.531+0000 7f9e1c985740 1 -- 172.21.15.161:0/1233953733 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558804992db0 msgr2=0x5588049b32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:05.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.531+0000 7f9e1c985740 1 --2- 172.21.15.161:0/1233953733 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558804992db0 0x5588049b32b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f9e04059150 tx=0x7f9e040591c0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.531+0000 7f9e1c985740 1 -- 172.21.15.161:0/1233953733 shutdown_connections 2024-03-28T23:31:05.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.531+0000 7f9e1c985740 1 --2- 172.21.15.161:0/1233953733 >> [v2:172.21.15.161:6835/2636327316,v1:172.21.15.161:6837/2636327316] conn(0x5588049cb380 0x5588049e2ff0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.531+0000 7f9e1c985740 1 --2- 172.21.15.161:0/1233953733 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5588049c7430 0x5588049cae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.531+0000 7f9df27fc640 0 client.5577 ms_handle_reset on v2:172.21.15.161:6835/2636327316 2024-03-28T23:31:05.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.531+0000 7f9e1c985740 1 --2- 172.21.15.161:0/1233953733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558804993190 0x5588049b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.531+0000 7f9e1c985740 1 --2- 172.21.15.161:0/1233953733 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558804992db0 0x5588049b32b0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.531+0000 7f9e1c985740 1 -- 172.21.15.161:0/1233953733 >> 172.21.15.161:0/1233953733 conn(0x558804991310 msgr2=0x558804991740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:05.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.531+0000 7f9e1c985740 1 -- 172.21.15.161:0/1233953733 shutdown_connections 2024-03-28T23:31:05.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.531+0000 7f9e1c985740 1 -- 172.21.15.161:0/1233953733 wait complete. 2024-03-28T23:31:05.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.531+0000 7f9e1c985740 20 client.5577 trim_cache size 0 max 16384 2024-03-28T23:31:05.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.549+0000 7f9e1c985740 1 Processor -- start 2024-03-28T23:31:05.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.549+0000 7f9e1c985740 1 -- start start 2024-03-28T23:31:05.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.550+0000 7f9e1c985740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558804992db0 0x5588049f2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:05.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.550+0000 7f9e1c985740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558804993190 0x55880498c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:05.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.550+0000 7f9e1c985740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588049c7430 0x5588049b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:05.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.550+0000 7f9e1c985740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9e141f74d0 con 0x5588049c7430 2024-03-28T23:31:05.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.550+0000 7f9e1c985740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9e141f6830 con 0x558804992db0 2024-03-28T23:31:05.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.550+0000 7f9e1c985740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9e141f6e80 con 0x558804993190 2024-03-28T23:31:05.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.550+0000 7f9e13fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588049c7430 0x5588049b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:05.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.550+0000 7f9e13fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588049c7430 0x5588049b77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45962/0 (socket says 172.21.15.161:45962) 2024-03-28T23:31:05.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.550+0000 7f9e13fff640 1 -- 172.21.15.161:0/3046752090 learned_addr learned my addr 172.21.15.161:0/3046752090 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:05.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.550+0000 7f9e137fe640 1 --2- 172.21.15.161:0/3046752090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558804992db0 0x5588049f2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:05.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.550+0000 7f9e12ffd640 1 --2- 172.21.15.161:0/3046752090 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558804993190 0x55880498c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:05.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.550+0000 7f9e13fff640 1 -- 172.21.15.161:0/3046752090 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558804993190 msgr2=0x55880498c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:05.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.550+0000 7f9e13fff640 1 --2- 172.21.15.161:0/3046752090 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558804993190 0x55880498c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.550+0000 7f9e13fff640 1 -- 172.21.15.161:0/3046752090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558804992db0 msgr2=0x5588049f2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:05.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.550+0000 7f9e13fff640 1 --2- 172.21.15.161:0/3046752090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558804992db0 0x5588049f2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.550+0000 7f9e13fff640 1 -- 172.21.15.161:0/3046752090 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588049dfd30 con 0x5588049c7430 2024-03-28T23:31:05.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.550+0000 7f9e137fe640 1 --2- 172.21.15.161:0/3046752090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558804992db0 0x5588049f2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:05.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.550+0000 7f9e13fff640 1 --2- 172.21.15.161:0/3046752090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588049c7430 0x5588049b77c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f9e140be590 tx=0x7f9e0c009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57401bb406b67e20 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:05.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.550+0000 7f9e08ff9640 1 -- 172.21.15.161:0/3046752090 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c021420 con 0x5588049c7430 2024-03-28T23:31:05.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.551+0000 7f9e08ff9640 1 -- 172.21.15.161:0/3046752090 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e0c0215c0 con 0x5588049c7430 2024-03-28T23:31:05.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.551+0000 7f9e08ff9640 1 -- 172.21.15.161:0/3046752090 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c021770 con 0x5588049c7430 2024-03-28T23:31:05.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.551+0000 7f9e1c985740 1 -- 172.21.15.161:0/3046752090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588049c7430 msgr2=0x5588049b77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:05.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.551+0000 7f9e1c985740 1 --2- 172.21.15.161:0/3046752090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588049c7430 0x5588049b77c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f9e140be590 tx=0x7f9e0c009190 comp rx=0 tx=0).stop 2024-03-28T23:31:05.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.551+0000 7f9e1c985740 1 -- 172.21.15.161:0/3046752090 shutdown_connections 2024-03-28T23:31:05.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.551+0000 7f9e1c985740 1 --2- 172.21.15.161:0/3046752090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588049c7430 0x5588049b77c0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.551+0000 7f9e1c985740 1 --2- 172.21.15.161:0/3046752090 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558804993190 0x55880498c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.551+0000 7f9e1c985740 1 --2- 172.21.15.161:0/3046752090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558804992db0 0x5588049f2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.551+0000 7f9e1c985740 1 -- 172.21.15.161:0/3046752090 >> 172.21.15.161:0/3046752090 conn(0x558804991310 msgr2=0x5588049cc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:05.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.551+0000 7f9e1c985740 1 -- 172.21.15.161:0/3046752090 shutdown_connections 2024-03-28T23:31:05.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.551+0000 7f9e1c985740 1 -- 172.21.15.161:0/3046752090 wait complete. 2024-03-28T23:31:05.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.552+0000 7f9e1c985740 1 -- start start 2024-03-28T23:31:05.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.552+0000 7f9e1c985740 1 Processor -- start 2024-03-28T23:31:05.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.552+0000 7f9e1c985740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558804992db0 0x558804a4a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:05.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.552+0000 7f9e1c985740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558804993190 0x5588049e8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:05.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.552+0000 7f9e1c985740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588049c7430 0x558804990400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:05.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.552+0000 7f9e1c985740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5588049b5cc0 con 0x5588049c7430 2024-03-28T23:31:05.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.552+0000 7f9e1c985740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5588049df1e0 con 0x558804993190 2024-03-28T23:31:05.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.552+0000 7f9e1c985740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x5588049cb800 con 0x558804992db0 2024-03-28T23:31:05.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.552+0000 7f9e1c985740 10 client.0 _ll_register_callbacks cb 0x7f9e04058d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:31:05.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.553+0000 7f9e13fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588049c7430 0x558804990400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:05.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.553+0000 7f9e13fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588049c7430 0x558804990400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45966/0 (socket says 172.21.15.161:45966) 2024-03-28T23:31:05.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.553+0000 7f9e13fff640 1 -- 172.21.15.161:0/598634780 learned_addr learned my addr 172.21.15.161:0/598634780 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:05.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.553+0000 7f9e12ffd640 1 --2- 172.21.15.161:0/598634780 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558804993190 0x5588049e8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:05.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.553+0000 7f9e137fe640 1 --2- 172.21.15.161:0/598634780 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558804992db0 0x558804a4a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:05.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.553+0000 7f9e13fff640 1 -- 172.21.15.161:0/598634780 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558804992db0 msgr2=0x558804a4a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:05.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.553+0000 7f9e13fff640 1 --2- 172.21.15.161:0/598634780 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558804992db0 0x558804a4a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.553+0000 7f9e13fff640 1 -- 172.21.15.161:0/598634780 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558804993190 msgr2=0x5588049e8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:05.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.553+0000 7f9e13fff640 1 --2- 172.21.15.161:0/598634780 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558804993190 0x5588049e8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.553+0000 7f9e13fff640 1 -- 172.21.15.161:0/598634780 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558804a439b0 con 0x5588049c7430 2024-03-28T23:31:05.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.553+0000 7f9e13fff640 1 --2- 172.21.15.161:0/598634780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588049c7430 0x558804990400 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x5588049c88e0 tx=0x7f9e0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b9e16b22150a78d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:05.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.553+0000 7f9df27fc640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:31:05.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.553+0000 7f9df27fc640 1 -- 172.21.15.161:0/598634780 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c021420 con 0x5588049c7430 2024-03-28T23:31:05.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.553+0000 7f9df27fc640 1 -- 172.21.15.161:0/598634780 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f9e0c026040 con 0x5588049c7430 2024-03-28T23:31:05.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.553+0000 7f9e1c985740 1 -- 172.21.15.161:0/598634780 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e0c0215c0 con 0x5588049c7430 2024-03-28T23:31:05.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.553+0000 7f9df27fc640 1 -- 172.21.15.161:0/598634780 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f9e0c035050 con 0x5588049c7430 2024-03-28T23:31:05.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.553+0000 7f9e1c985740 10 client.5583 fetch_fsmap learned FSMap version 17 2024-03-28T23:31:05.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.553+0000 7f9e1c985740 1 -- 172.21.15.161:0/598634780 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55880498cb40 con 0x5588049c7430 2024-03-28T23:31:05.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.553+0000 7f9df27fc640 1 -- 172.21.15.161:0/598634780 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e0c0215c0 con 0x5588049c7430 2024-03-28T23:31:05.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.554+0000 7f9df27fc640 1 -- 172.21.15.161:0/598634780 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7f9e0c038020 con 0x5588049c7430 2024-03-28T23:31:05.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.554+0000 7f9e1c985740 10 client.5583 fetch_fsmap finished waiting for FSMap version 17 2024-03-28T23:31:05.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.554+0000 7f9e1c985740 10 client.5583 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T23:31:05.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.554+0000 7f9e1c985740 20 client.5583 populate_metadata read hostname 'smithi161' 2024-03-28T23:31:05.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.554+0000 7f9e1c985740 1 --2- 172.21.15.161:0/598634780 >> [v2:172.21.15.161:6835/2636327316,v1:172.21.15.161:6837/2636327316] conn(0x5588049d2720 0x5588049d2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:05.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.554+0000 7f9e1c985740 4 client.5583 mds_command: new command op to 5361 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:31:05.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.554+0000 7f9e1c985740 1 -- 172.21.15.161:0/598634780 --> [v2:172.21.15.161:6835/2636327316,v1:172.21.15.161:6837/2636327316] -- command(tid 0: {"prefix": "status"}) -- 0x5588048fe380 con 0x5588049d2720 2024-03-28T23:31:05.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.554+0000 7f9e137fe640 1 --2- 172.21.15.161:0/598634780 >> [v2:172.21.15.161:6835/2636327316,v1:172.21.15.161:6837/2636327316] conn(0x5588049d2720 0x5588049d2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:05.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.554+0000 7f9df27fc640 10 client.5583 ms_handle_connect on v2:172.21.15.161:6835/2636327316 2024-03-28T23:31:05.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.554+0000 7f9e137fe640 1 --2- 172.21.15.161:0/598634780 >> [v2:172.21.15.161:6835/2636327316,v1:172.21.15.161:6837/2636327316] conn(0x5588049d2720 0x5588049d2b00 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13a7e56d98efaff7 server_cookie=cfe4bebba5bc7a6f in_seq=0 out_seq=0 2024-03-28T23:31:05.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.554+0000 7f9df27fc640 1 -- 172.21.15.161:0/598634780 <== mds.0 v2:172.21.15.161:6835/2636327316 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5588048fe380 con 0x5588049d2720 2024-03-28T23:31:05.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.554+0000 7f9df27fc640 10 client.5583 handle_command_reply: tid=0 2024-03-28T23:31:05.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.554+0000 7f9e1c985740 1 client.5583 shutdown 2024-03-28T23:31:05.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.554+0000 7f9e1c985740 1 -- 172.21.15.161:0/598634780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588049c7430 msgr2=0x558804990400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:05.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.554+0000 7f9e1c985740 1 --2- 172.21.15.161:0/598634780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588049c7430 0x558804990400 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x5588049c88e0 tx=0x7f9e0c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.554+0000 7f9e1c985740 1 -- 172.21.15.161:0/598634780 shutdown_connections 2024-03-28T23:31:05.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.554+0000 7f9e1c985740 1 --2- 172.21.15.161:0/598634780 >> [v2:172.21.15.161:6835/2636327316,v1:172.21.15.161:6837/2636327316] conn(0x5588049d2720 0x5588049d2b00 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.554+0000 7f9e1c985740 1 --2- 172.21.15.161:0/598634780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588049c7430 0x558804990400 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.554+0000 7f9e1c985740 1 --2- 172.21.15.161:0/598634780 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558804993190 0x5588049e8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.554+0000 7f9df27fc640 0 client.5583 ms_handle_reset on v2:172.21.15.161:6835/2636327316 2024-03-28T23:31:05.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.554+0000 7f9e1c985740 1 --2- 172.21.15.161:0/598634780 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558804992db0 0x558804a4a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.554+0000 7f9e1c985740 1 -- 172.21.15.161:0/598634780 >> 172.21.15.161:0/598634780 conn(0x558804991310 msgr2=0x5588049f2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:05.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.555+0000 7f9e1c985740 1 -- 172.21.15.161:0/598634780 shutdown_connections 2024-03-28T23:31:05.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.555+0000 7f9e1c985740 1 -- 172.21.15.161:0/598634780 wait complete. 2024-03-28T23:31:05.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.555+0000 7f9e1c985740 20 client.5583 trim_cache size 0 max 16384 2024-03-28T23:31:05.562 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:31:05.562 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:31:05.562 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:31:05.562 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 5361, 2024-03-28T23:31:05.562 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:31:05.562 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:31:05.562 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:31:05.563 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 7.2975803969999999, 2024-03-28T23:31:05.563 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 17, 2024-03-28T23:31:05.563 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 38, 2024-03-28T23:31:05.563 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 38, 2024-03-28T23:31:05.563 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 14.189211461999999 2024-03-28T23:31:05.563 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:31:05.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.555+0000 7f9e19da1640 1 -- 172.21.15.161:0/860531861 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9df406b500 msgr2=0x7f9df406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:05.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.555+0000 7f9e19da1640 1 --2- 172.21.15.161:0/860531861 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9df406b500 0x7f9df406d9c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc033000 tx=0x7f9dfc009030 comp rx=0 tx=0).stop 2024-03-28T23:31:05.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.555+0000 7f9e19da1640 1 -- 172.21.15.161:0/860531861 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e00004800 msgr2=0x7f9e1414ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:05.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.555+0000 7f9e19da1640 1 --2- 172.21.15.161:0/860531861 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e00004800 0x7f9e1414ef20 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f9e040077d0 tx=0x7f9e0400a730 comp rx=0 tx=0).stop 2024-03-28T23:31:05.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.555+0000 7f9e19da1640 1 -- 172.21.15.161:0/860531861 shutdown_connections 2024-03-28T23:31:05.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.555+0000 7f9e19da1640 1 --2- 172.21.15.161:0/860531861 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9df406b500 0x7f9df406d9c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.555+0000 7f9e19da1640 1 --2- 172.21.15.161:0/860531861 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e14148220 0x7f9e141eb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.555+0000 7f9e19da1640 1 --2- 172.21.15.161:0/860531861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9e140c0c60 0x7f9e14147ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.555+0000 7f9e19da1640 1 --2- 172.21.15.161:0/860531861 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e00004800 0x7f9e1414ef20 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:05.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.555+0000 7f9e19da1640 1 -- 172.21.15.161:0/860531861 >> 172.21.15.161:0/860531861 conn(0x7f9e140ac620 msgr2=0x7f9e14151350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:05.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.555+0000 7f9e19da1640 1 -- 172.21.15.161:0/860531861 shutdown_connections 2024-03-28T23:31:05.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:05.555+0000 7f9e19da1640 1 -- 172.21.15.161:0/860531861 wait complete. 2024-03-28T23:31:05.570 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 17, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:30:57.616423+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 17, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5361": { "addr": "172.21.15.161:6837/2636327316", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6835", "nonce": 2636327316, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 2636327316, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5361, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-03-28T23:31:00.324742+0000", "qdb_cluster": [ 5361 ], "qdb_leader": 5361, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5361 } } } ], "standbys": [ { "addr": "172.21.15.170:6834/11130685", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 11130685, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 11130685, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4853, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.170:6835/465771987", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 465771987, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 465771987, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5110, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6836/306644174", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 306644174, "type": "v2" }, { "addr": "172.21.15.161:6836", "nonce": 306644174, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5355, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:31:05.571 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 2, 'name': 'd', 'state': 'up:active'}] 2024-03-28T23:31:05.571 INFO:tasks.cephfs.test_failover:current = [{'gid': 5361, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.161:6837/2636327316', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 2636327316}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 2636327316}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 5110, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/465771987', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 465771987}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 465771987}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 13}, {'gid': 5355, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6836/306644174', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 306644174}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 306644174}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 12}, {'gid': 4853, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6834/11130685', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 11130685}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 11130685}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 13}] 2024-03-28T23:31:05.571 DEBUG:tasks.cephfs.test_failover:2 != -1 2024-03-28T23:31:05.572 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-28T23:31:07.639 INFO:tasks.ceph.mds.a.smithi161.stderr: -226> 2024-03-28T23:31:03.235+0000 7f4a6e3fd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:31:07.664 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-28T23:31:10.572 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:31:10.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.685+0000 7f4a34702640 1 Processor -- start 2024-03-28T23:31:10.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.685+0000 7f4a34702640 1 -- start start 2024-03-28T23:31:10.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.686+0000 7f4a34702640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a2c152890 0x7f4a2c0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:10.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.686+0000 7f4a34702640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4a2c0c5b20 0x7f4a2c1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:10.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.686+0000 7f4a34702640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a2c158a30 con 0x7f4a2c0c8ed0 2024-03-28T23:31:10.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.686+0000 7f4a34702640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f4a2c0c8480 con 0x7f4a2c0c5b20 2024-03-28T23:31:10.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.686+0000 7f4a34702640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4a2c159640 con 0x7f4a2c152890 2024-03-28T23:31:10.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.686+0000 7f4a31c9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f4a2c0c8ed0 0x7f4a2c0c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:47986/0 (socket says 172.21.15.161:47986) 2024-03-28T23:31:10.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.686+0000 7f4a31c9d640 1 -- 172.21.15.161:0/2373848365 learned_addr learned my addr 172.21.15.161:0/2373848365 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:10.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.687+0000 7f4a3249e640 1 --2- 172.21.15.161:0/2373848365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a2c152890 0x7f4a2c0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:10.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.687+0000 7f4a32c9f640 1 --2- 172.21.15.161:0/2373848365 >> v2:172.21.15.170:3300/0 conn(0x7f4a2c0c5b20 0x7f4a2c1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:10.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.687+0000 7f4a3149c640 1 -- 172.21.15.161:0/2373848365 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3215861724 0 0) 0x7f4a2c158a30 con 0x7f4a2c0c8ed0 2024-03-28T23:31:10.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.687+0000 7f4a3149c640 1 -- 172.21.15.161:0/2373848365 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a18003520 con 0x7f4a2c0c8ed0 2024-03-28T23:31:10.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.687+0000 7f4a3249e640 1 -- 172.21.15.161:0/2373848365 >> v2:172.21.15.170:3300/0 conn(0x7f4a2c0c5b20 msgr2=0x7f4a2c1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:10.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.687+0000 7f4a3249e640 1 --2- 172.21.15.161:0/2373848365 >> v2:172.21.15.170:3300/0 conn(0x7f4a2c0c5b20 0x7f4a2c1553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:10.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.687+0000 7f4a3249e640 1 -- 172.21.15.161:0/2373848365 >> v1:172.21.15.161:6789/0 conn(0x7f4a2c0c8ed0 legacy=0x7f4a2c0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:10.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.687+0000 7f4a3249e640 1 -- 172.21.15.161:0/2373848365 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a2c158d10 con 0x7f4a2c152890 2024-03-28T23:31:10.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.687+0000 7f4a32c9f640 1 --2- 172.21.15.161:0/2373848365 >> v2:172.21.15.170:3300/0 conn(0x7f4a2c0c5b20 0x7f4a2c1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:10.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.688+0000 7f4a3249e640 1 --2- 172.21.15.161:0/2373848365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a2c152890 0x7f4a2c0c8990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c1597c0 tx=0x7f4a2802bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64a2260c105997f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:10.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.688+0000 7f4a3149c640 1 -- 172.21.15.161:0/2373848365 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a28039070 con 0x7f4a2c152890 2024-03-28T23:31:10.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.688+0000 7f4a3149c640 1 -- 172.21.15.161:0/2373848365 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a2802cb20 con 0x7f4a2c152890 2024-03-28T23:31:10.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.688+0000 7f4a3149c640 1 -- 172.21.15.161:0/2373848365 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a28034440 con 0x7f4a2c152890 2024-03-28T23:31:10.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.689+0000 7f4a34702640 1 -- 172.21.15.161:0/2373848365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a2c152890 msgr2=0x7f4a2c0c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:10.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.689+0000 7f4a34702640 1 --2- 172.21.15.161:0/2373848365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a2c152890 0x7f4a2c0c8990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c1597c0 tx=0x7f4a2802bd70 comp rx=0 tx=0).stop 2024-03-28T23:31:10.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.689+0000 7f4a34702640 1 -- 172.21.15.161:0/2373848365 shutdown_connections 2024-03-28T23:31:10.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.689+0000 7f4a34702640 1 --2- 172.21.15.161:0/2373848365 >> v2:172.21.15.170:3300/0 conn(0x7f4a2c0c5b20 0x7f4a2c1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:10.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.689+0000 7f4a34702640 1 --2- 172.21.15.161:0/2373848365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a2c152890 0x7f4a2c0c8990 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:10.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.689+0000 7f4a34702640 1 -- 172.21.15.161:0/2373848365 >> 172.21.15.161:0/2373848365 conn(0x7f4a2c0ac620 msgr2=0x7f4a2c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:10.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.689+0000 7f4a34702640 1 -- 172.21.15.161:0/2373848365 shutdown_connections 2024-03-28T23:31:10.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.689+0000 7f4a34702640 1 -- 172.21.15.161:0/2373848365 wait complete. 2024-03-28T23:31:10.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.690+0000 7f4a34702640 1 Processor -- start 2024-03-28T23:31:10.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.690+0000 7f4a34702640 1 -- start start 2024-03-28T23:31:10.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.691+0000 7f4a34702640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a2c0c5b20 0x7f4a2c1eaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:10.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.691+0000 7f4a34702640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a2c0c8ed0 0x7f4a2c158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:10.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.691+0000 7f4a34702640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a2c152890 0x7f4a2c14d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:10.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.691+0000 7f4a34702640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2c152c70 con 0x7f4a2c0c5b20 2024-03-28T23:31:10.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.691+0000 7f4a34702640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2c1592d0 con 0x7f4a2c152890 2024-03-28T23:31:10.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.691+0000 7f4a34702640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4a2c152df0 con 0x7f4a2c0c8ed0 2024-03-28T23:31:10.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.691+0000 7f4a3249e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a2c0c5b20 0x7f4a2c1eaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:10.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.691+0000 7f4a3249e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a2c0c5b20 0x7f4a2c1eaeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50870/0 (socket says 172.21.15.161:50870) 2024-03-28T23:31:10.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.691+0000 7f4a3249e640 1 -- 172.21.15.161:0/834814058 learned_addr learned my addr 172.21.15.161:0/834814058 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:10.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.691+0000 7f4a31c9d640 1 --2- 172.21.15.161:0/834814058 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a2c0c8ed0 0x7f4a2c158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:10.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.691+0000 7f4a32c9f640 1 --2- 172.21.15.161:0/834814058 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a2c152890 0x7f4a2c14d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:10.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.691+0000 7f4a3249e640 1 -- 172.21.15.161:0/834814058 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a2c0c8ed0 msgr2=0x7f4a2c158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:10.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.691+0000 7f4a3249e640 1 --2- 172.21.15.161:0/834814058 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a2c0c8ed0 0x7f4a2c158070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:10.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.691+0000 7f4a3249e640 1 -- 172.21.15.161:0/834814058 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a2c152890 msgr2=0x7f4a2c14d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:10.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.691+0000 7f4a3249e640 1 --2- 172.21.15.161:0/834814058 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a2c152890 0x7f4a2c14d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:10.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.691+0000 7f4a3249e640 1 -- 172.21.15.161:0/834814058 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a2c1eb3f0 con 0x7f4a2c0c5b20 2024-03-28T23:31:10.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.691+0000 7f4a32c9f640 1 --2- 172.21.15.161:0/834814058 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a2c152890 0x7f4a2c14d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:10.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.691+0000 7f4a3249e640 1 --2- 172.21.15.161:0/834814058 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a2c0c5b20 0x7f4a2c1eaeb0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f4a28037040 tx=0x7f4a28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:10.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.691+0000 7f4a22ffd640 1 -- 172.21.15.161:0/834814058 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a28039050 con 0x7f4a2c0c5b20 2024-03-28T23:31:10.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.691+0000 7f4a22ffd640 1 -- 172.21.15.161:0/834814058 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a28045070 con 0x7f4a2c0c5b20 2024-03-28T23:31:10.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.691+0000 7f4a22ffd640 1 -- 172.21.15.161:0/834814058 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a2803e850 con 0x7f4a2c0c5b20 2024-03-28T23:31:10.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.691+0000 7f4a34702640 1 -- 172.21.15.161:0/834814058 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a2c1eb670 con 0x7f4a2c0c5b20 2024-03-28T23:31:10.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.691+0000 7f4a34702640 1 -- 172.21.15.161:0/834814058 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a2c1ebc30 con 0x7f4a2c0c5b20 2024-03-28T23:31:10.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.692+0000 7f4a34702640 1 -- 172.21.15.161:0/834814058 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a2c1ebe30 con 0x7f4a2c0c5b20 2024-03-28T23:31:10.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.693+0000 7f4a22ffd640 1 -- 172.21.15.161:0/834814058 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4a2803e9f0 con 0x7f4a2c0c5b20 2024-03-28T23:31:10.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.693+0000 7f4a22ffd640 1 --2- 172.21.15.161:0/834814058 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4a1006b4d0 0x7f4a1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:10.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.693+0000 7f4a22ffd640 1 -- 172.21.15.161:0/834814058 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f4a280ad210 con 0x7f4a2c0c5b20 2024-03-28T23:31:10.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.693+0000 7f4a31c9d640 1 --2- 172.21.15.161:0/834814058 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4a1006b4d0 0x7f4a1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:10.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.694+0000 7f4a31c9d640 1 --2- 172.21.15.161:0/834814058 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4a1006b4d0 0x7f4a1006d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c06c8b0 tx=0x7f4a1c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:10.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.698+0000 7f4a22ffd640 1 -- 172.21.15.161:0/834814058 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4a2c1ebe30 con 0x7f4a2c0c5b20 2024-03-28T23:31:10.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.905+0000 7f4a34702640 1 -- 172.21.15.161:0/834814058 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a2c067a50 con 0x7f4a2c0c5b20 2024-03-28T23:31:10.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.906+0000 7f4a22ffd640 1 -- 172.21.15.161:0/834814058 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+5181 (secure 0 0 0) 0x7f4a2c067a50 con 0x7f4a2c0c5b20 2024-03-28T23:31:10.907 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:10.907 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":22,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5110,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/465771987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":465771987},{"type":"v1","addr":"172.21.15.170:6835","nonce":465771987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5355,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/306644174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":306644174},{"type":"v1","addr":"172.21.15.161:6836","nonce":306644174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1649814516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":1649814516},{"type":"v1","addr":"172.21.15.161:6837","nonce":1649814516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:30:57.616423+0000","modified":"2024-03-28T23:31:10.642963+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4853},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4853":{"gid":4853,"name":"d","rank":0,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.170:6834/11130685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":11130685},{"type":"v1","addr":"172.21.15.170:6834","nonce":11130685}]},"join_fscid":2,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4853,"qdb_cluster":[4853]},"id":2}]} 2024-03-28T23:31:10.907 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 22 2024-03-28T23:31:10.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.908+0000 7f4a34702640 1 -- 172.21.15.161:0/834814058 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4a1006b4d0 msgr2=0x7f4a1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:10.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.908+0000 7f4a34702640 1 --2- 172.21.15.161:0/834814058 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4a1006b4d0 0x7f4a1006d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c06c8b0 tx=0x7f4a1c009040 comp rx=0 tx=0).stop 2024-03-28T23:31:10.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.908+0000 7f4a34702640 1 -- 172.21.15.161:0/834814058 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a2c0c5b20 msgr2=0x7f4a2c1eaeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:10.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.908+0000 7f4a34702640 1 --2- 172.21.15.161:0/834814058 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a2c0c5b20 0x7f4a2c1eaeb0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f4a28037040 tx=0x7f4a28004a40 comp rx=0 tx=0).stop 2024-03-28T23:31:10.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.908+0000 7f4a34702640 1 -- 172.21.15.161:0/834814058 shutdown_connections 2024-03-28T23:31:10.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.908+0000 7f4a34702640 1 --2- 172.21.15.161:0/834814058 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4a1006b4d0 0x7f4a1006d990 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:10.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.908+0000 7f4a34702640 1 --2- 172.21.15.161:0/834814058 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a2c152890 0x7f4a2c14d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:10.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.908+0000 7f4a34702640 1 --2- 172.21.15.161:0/834814058 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a2c0c8ed0 0x7f4a2c158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:10.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.908+0000 7f4a34702640 1 --2- 172.21.15.161:0/834814058 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a2c0c5b20 0x7f4a2c1eaeb0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:10.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.908+0000 7f4a34702640 1 -- 172.21.15.161:0/834814058 >> 172.21.15.161:0/834814058 conn(0x7f4a2c0ac620 msgr2=0x7f4a2c0c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:10.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.908+0000 7f4a34702640 1 -- 172.21.15.161:0/834814058 shutdown_connections 2024-03-28T23:31:10.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:10.908+0000 7f4a34702640 1 -- 172.21.15.161:0/834814058 wait complete. 2024-03-28T23:31:11.322 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:30:57.616423+0000', 'modified': '2024-03-28T23:31:10.642963+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 39, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4853}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4853': {'gid': 4853, 'name': 'd', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.170:6834/11130685', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 11130685}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 11130685}]}, 'join_fscid': 2, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4853, 'qdb_cluster': [4853]} 2024-03-28T23:31:11.323 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:31:11.323 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-28T23:31:11.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.426+0000 7fd8a84cd640 1 Processor -- start 2024-03-28T23:31:11.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.426+0000 7fd8a84cd640 1 -- start start 2024-03-28T23:31:11.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.427+0000 7fd8a84cd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd8a006ce80 0x7fd8a00aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.427+0000 7fd8a84cd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8a00a9680 0x7fd8a00aab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.427+0000 7fd8a84cd640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8a014c250 con 0x7fd8a014be70 2024-03-28T23:31:11.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.427+0000 7fd8a84cd640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd8a01529c0 con 0x7fd8a006ce80 2024-03-28T23:31:11.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.427+0000 7fd8a84cd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd8a00a9dc0 con 0x7fd8a00a9680 2024-03-28T23:31:11.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.428+0000 7fd8a5a68640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd8a006ce80 0x7fd8a00aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.428+0000 7fd8a6a6a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8a00a9680 0x7fd8a00aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.428+0000 7fd8a6269640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fd8a014be70 0x7fd8a00ab1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:51332/0 (socket says 172.21.15.161:51332) 2024-03-28T23:31:11.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.428+0000 7fd8a6269640 1 -- 172.21.15.161:0/4222273313 learned_addr learned my addr 172.21.15.161:0/4222273313 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:11.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.428+0000 7fd8a5a68640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd8a006ce80 0x7fd8a00aa5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:52852/0 (socket says 172.21.15.161:52852) 2024-03-28T23:31:11.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.428+0000 7fd8a6a6a640 1 -- 172.21.15.161:0/4222273313 >> v2:172.21.15.170:3300/0 conn(0x7fd8a006ce80 msgr2=0x7fd8a00aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:11.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.428+0000 7fd8a6a6a640 1 --2- 172.21.15.161:0/4222273313 >> v2:172.21.15.170:3300/0 conn(0x7fd8a006ce80 0x7fd8a00aa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.428+0000 7fd8a6a6a640 1 -- 172.21.15.161:0/4222273313 >> v1:172.21.15.170:6789/0 conn(0x7fd8a014be70 legacy=0x7fd8a00ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:11.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.428+0000 7fd8a6a6a640 1 -- 172.21.15.161:0/4222273313 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a00aa0a0 con 0x7fd8a00a9680 2024-03-28T23:31:11.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.428+0000 7fd8a5a68640 1 --2- 172.21.15.161:0/4222273313 >> v2:172.21.15.170:3300/0 conn(0x7fd8a006ce80 0x7fd8a00aa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:11.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.429+0000 7fd8a6a6a640 1 --2- 172.21.15.161:0/4222273313 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8a00a9680 0x7fd8a00aab00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd89c0097a0 tx=0x7fd89c02d9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd5add2c3b74423 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:11.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.429+0000 7fd8a5267640 1 -- 172.21.15.161:0/4222273313 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd89c036070 con 0x7fd8a00a9680 2024-03-28T23:31:11.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.429+0000 7fd8a5267640 1 -- 172.21.15.161:0/4222273313 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd89c032870 con 0x7fd8a00a9680 2024-03-28T23:31:11.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.429+0000 7fd8a5267640 1 -- 172.21.15.161:0/4222273313 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd89c032c40 con 0x7fd8a00a9680 2024-03-28T23:31:11.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.429+0000 7fd8a84cd640 1 -- 172.21.15.161:0/4222273313 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8a00a9680 msgr2=0x7fd8a00aab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:11.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.429+0000 7fd8a84cd640 1 --2- 172.21.15.161:0/4222273313 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8a00a9680 0x7fd8a00aab00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd89c0097a0 tx=0x7fd89c02d9b0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.429+0000 7fd8a84cd640 1 -- 172.21.15.161:0/4222273313 shutdown_connections 2024-03-28T23:31:11.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.429+0000 7fd8a84cd640 1 --2- 172.21.15.161:0/4222273313 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8a00a9680 0x7fd8a00aab00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.429+0000 7fd8a84cd640 1 --2- 172.21.15.161:0/4222273313 >> v2:172.21.15.170:3300/0 conn(0x7fd8a006ce80 0x7fd8a00aa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.429+0000 7fd8a84cd640 1 -- 172.21.15.161:0/4222273313 >> 172.21.15.161:0/4222273313 conn(0x7fd8a00c6670 msgr2=0x7fd8a00c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:11.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.429+0000 7fd8a84cd640 1 -- 172.21.15.161:0/4222273313 shutdown_connections 2024-03-28T23:31:11.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.429+0000 7fd8a84cd640 1 -- 172.21.15.161:0/4222273313 wait complete. 2024-03-28T23:31:11.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.430+0000 7fd8a84cd640 1 Processor -- start 2024-03-28T23:31:11.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.431+0000 7fd8a84cd640 1 -- start start 2024-03-28T23:31:11.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.431+0000 7fd8a84cd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8a006ce80 0x7fd8a01ddae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.431+0000 7fd8a84cd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8a00a9680 0x7fd8a0144840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.431+0000 7fd8a84cd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8a014be70 0x7fd8a0144d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.431+0000 7fd8a84cd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a00aba30 con 0x7fd8a00a9680 2024-03-28T23:31:11.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.431+0000 7fd8a84cd640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a00ab8b0 con 0x7fd8a006ce80 2024-03-28T23:31:11.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.431+0000 7fd8a84cd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd8a006b150 con 0x7fd8a014be70 2024-03-28T23:31:11.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.431+0000 7fd8a6269640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8a006ce80 0x7fd8a01ddae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.431+0000 7fd8a5a68640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8a00a9680 0x7fd8a0144840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.431+0000 7fd8a6a6a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8a014be70 0x7fd8a0144d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.431+0000 7fd8a6269640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8a006ce80 0x7fd8a01ddae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:52862/0 (socket says 172.21.15.161:52862) 2024-03-28T23:31:11.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.431+0000 7fd8a5a68640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8a00a9680 0x7fd8a0144840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50876/0 (socket says 172.21.15.161:50876) 2024-03-28T23:31:11.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.431+0000 7fd8a6269640 1 -- 172.21.15.161:0/1569328534 learned_addr learned my addr 172.21.15.161:0/1569328534 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:11.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.432+0000 7fd8a6269640 1 -- 172.21.15.161:0/1569328534 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8a014be70 msgr2=0x7fd8a0144d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:11.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.432+0000 7fd8a6269640 1 --2- 172.21.15.161:0/1569328534 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8a014be70 0x7fd8a0144d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.432+0000 7fd8a6269640 1 -- 172.21.15.161:0/1569328534 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8a00a9680 msgr2=0x7fd8a0144840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:11.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.432+0000 7fd8a6269640 1 --2- 172.21.15.161:0/1569328534 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8a00a9680 0x7fd8a0144840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.432+0000 7fd8a6269640 1 -- 172.21.15.161:0/1569328534 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a01452c0 con 0x7fd8a006ce80 2024-03-28T23:31:11.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.432+0000 7fd8a6269640 1 --2- 172.21.15.161:0/1569328534 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8a006ce80 0x7fd8a01ddae0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd89800caa0 tx=0x7fd89800cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:11.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.432+0000 7fd896ffd640 1 -- 172.21.15.161:0/1569328534 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd898017070 con 0x7fd8a006ce80 2024-03-28T23:31:11.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.432+0000 7fd896ffd640 1 -- 172.21.15.161:0/1569328534 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd89800ac20 con 0x7fd8a006ce80 2024-03-28T23:31:11.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.432+0000 7fd8a84cd640 1 -- 172.21.15.161:0/1569328534 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8a01dbf40 con 0x7fd8a006ce80 2024-03-28T23:31:11.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.432+0000 7fd896ffd640 1 -- 172.21.15.161:0/1569328534 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd898012400 con 0x7fd8a006ce80 2024-03-28T23:31:11.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.432+0000 7fd8a84cd640 1 -- 172.21.15.161:0/1569328534 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8a01505d0 con 0x7fd8a006ce80 2024-03-28T23:31:11.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.433+0000 7fd896ffd640 1 -- 172.21.15.161:0/1569328534 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd89801a070 con 0x7fd8a006ce80 2024-03-28T23:31:11.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.434+0000 7fd896ffd640 1 --2- 172.21.15.161:0/1569328534 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd88406b500 0x7fd88406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.434+0000 7fd896ffd640 1 -- 172.21.15.161:0/1569328534 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7fd898089ef0 con 0x7fd8a006ce80 2024-03-28T23:31:11.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.434+0000 7fd8a5a68640 1 --2- 172.21.15.161:0/1569328534 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd88406b500 0x7fd88406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.434+0000 7fd8a5a68640 1 --2- 172.21.15.161:0/1569328534 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd88406b500 0x7fd88406d9c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd890009560 tx=0x7fd890009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:11.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.435+0000 7fd8ab192740 1 Processor -- start 2024-03-28T23:31:11.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.435+0000 7fd8ab192740 1 -- start start 2024-03-28T23:31:11.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.436+0000 7fd8ab192740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651d9fcf660 0x5651d9f85e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.436+0000 7fd8ab192740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651d9f863b0 0x5651d9fcfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.436+0000 7fd8ab192740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fc0280 0x5651d9f804e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.436+0000 7fd8ab192740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5651d9ee9af0 con 0x5651d9fc0280 2024-03-28T23:31:11.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.436+0000 7fd8ab192740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5651d9e929d0 con 0x5651d9fcf660 2024-03-28T23:31:11.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.436+0000 7fd8ab192740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x5651d9ee5b90 con 0x5651d9f863b0 2024-03-28T23:31:11.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.436+0000 7fd8a6269640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651d9fcf660 0x5651d9f85e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.436+0000 7fd8a6a6a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fc0280 0x5651d9f804e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.436+0000 7fd8a6269640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651d9fcf660 0x5651d9f85e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:52864/0 (socket says 172.21.15.161:52864) 2024-03-28T23:31:11.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.436+0000 7fd8a6269640 1 -- 172.21.15.161:0/3036209263 learned_addr learned my addr 172.21.15.161:0/3036209263 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:11.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.436+0000 7fd8a5a68640 1 --2- 172.21.15.161:0/3036209263 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651d9f863b0 0x5651d9fcfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.436+0000 7fd8a6269640 1 -- 172.21.15.161:0/3036209263 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651d9f863b0 msgr2=0x5651d9fcfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:11.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.436+0000 7fd8a6269640 1 --2- 172.21.15.161:0/3036209263 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651d9f863b0 0x5651d9fcfa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.436+0000 7fd8a6269640 1 -- 172.21.15.161:0/3036209263 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fc0280 msgr2=0x5651d9f804e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:11.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.436+0000 7fd8a6269640 1 --2- 172.21.15.161:0/3036209263 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fc0280 0x5651d9f804e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.436+0000 7fd8a6269640 1 -- 172.21.15.161:0/3036209263 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651d9bfe2d0 con 0x5651d9fcf660 2024-03-28T23:31:11.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.436+0000 7fd8a6a6a640 1 --2- 172.21.15.161:0/3036209263 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fc0280 0x5651d9f804e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:11.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.436+0000 7fd8a5a68640 1 --2- 172.21.15.161:0/3036209263 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651d9f863b0 0x5651d9fcfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:11.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.436+0000 7fd8a6269640 1 --2- 172.21.15.161:0/3036209263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651d9fcf660 0x5651d9f85e70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd898054a10 tx=0x7fd8980586f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5019ebc1eb23c4d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:11.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.436+0000 7fd883fff640 1 -- 172.21.15.161:0/3036209263 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd898054c20 con 0x5651d9fcf660 2024-03-28T23:31:11.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.436+0000 7fd883fff640 1 -- 172.21.15.161:0/3036209263 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd898060780 con 0x5651d9fcf660 2024-03-28T23:31:11.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.436+0000 7fd883fff640 1 -- 172.21.15.161:0/3036209263 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd898054dc0 con 0x5651d9fcf660 2024-03-28T23:31:11.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.437+0000 7fd8ab192740 1 -- 172.21.15.161:0/3036209263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651d9fcf660 msgr2=0x5651d9f85e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:11.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.437+0000 7fd8ab192740 1 --2- 172.21.15.161:0/3036209263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651d9fcf660 0x5651d9f85e70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd898054a10 tx=0x7fd8980586f0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.437+0000 7fd8ab192740 1 -- 172.21.15.161:0/3036209263 shutdown_connections 2024-03-28T23:31:11.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.437+0000 7fd8ab192740 1 --2- 172.21.15.161:0/3036209263 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fc0280 0x5651d9f804e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.437+0000 7fd8ab192740 1 --2- 172.21.15.161:0/3036209263 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651d9f863b0 0x5651d9fcfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.437+0000 7fd8ab192740 1 --2- 172.21.15.161:0/3036209263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651d9fcf660 0x5651d9f85e70 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.437+0000 7fd8ab192740 1 -- 172.21.15.161:0/3036209263 >> 172.21.15.161:0/3036209263 conn(0x5651d9f82630 msgr2=0x5651d9fa7220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:11.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.437+0000 7fd8ab192740 1 -- 172.21.15.161:0/3036209263 shutdown_connections 2024-03-28T23:31:11.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.437+0000 7fd8ab192740 1 -- 172.21.15.161:0/3036209263 wait complete. 2024-03-28T23:31:11.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.438+0000 7fd8ab192740 1 -- start start 2024-03-28T23:31:11.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.438+0000 7fd8ab192740 1 Processor -- start 2024-03-28T23:31:11.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.439+0000 7fd8ab192740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fc0280 0x5651d9f81120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.439+0000 7fd8ab192740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651d9fcf660 0x5651d9fda660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.439+0000 7fd8ab192740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651d9fd8840 0x5651d9fd8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.439+0000 7fd8ab192740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5651d9fc43f0 con 0x5651d9fc0280 2024-03-28T23:31:11.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.439+0000 7fd8ab192740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5651d9ea5f70 con 0x5651d9fd8840 2024-03-28T23:31:11.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.439+0000 7fd8ab192740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x5651d9fa7c10 con 0x5651d9fcf660 2024-03-28T23:31:11.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.439+0000 7fd8ab192740 10 client.0 _ll_register_callbacks cb 0x5651d9bfdac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:31:11.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.439+0000 7fd8a6269640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fc0280 0x5651d9f81120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.439+0000 7fd8a6269640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fc0280 0x5651d9f81120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50896/0 (socket says 172.21.15.161:50896) 2024-03-28T23:31:11.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.439+0000 7fd8a6269640 1 -- 172.21.15.161:0/1953026291 learned_addr learned my addr 172.21.15.161:0/1953026291 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:11.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.439+0000 7fd8a5a68640 1 --2- 172.21.15.161:0/1953026291 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651d9fcf660 0x5651d9fda660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.439+0000 7fd8a6a6a640 1 --2- 172.21.15.161:0/1953026291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651d9fd8840 0x5651d9fd8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.439+0000 7fd8a6269640 1 -- 172.21.15.161:0/1953026291 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651d9fcf660 msgr2=0x5651d9fda660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:11.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.439+0000 7fd8a6269640 1 --2- 172.21.15.161:0/1953026291 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651d9fcf660 0x5651d9fda660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.439+0000 7fd8a6269640 1 -- 172.21.15.161:0/1953026291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651d9fd8840 msgr2=0x5651d9fd8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:11.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.439+0000 7fd8a6269640 1 --2- 172.21.15.161:0/1953026291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651d9fd8840 0x5651d9fd8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.439+0000 7fd8a6269640 1 -- 172.21.15.161:0/1953026291 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651d9cc86b0 con 0x5651d9fc0280 2024-03-28T23:31:11.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.439+0000 7fd8a5a68640 1 --2- 172.21.15.161:0/1953026291 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651d9fcf660 0x5651d9fda660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:11.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.439+0000 7fd8a6269640 1 --2- 172.21.15.161:0/1953026291 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fc0280 0x5651d9f81120 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd89800cfd0 tx=0x7fd898000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f270ed01de07262 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:11.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.439+0000 7fd8817fa640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:31:11.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.439+0000 7fd8817fa640 1 -- 172.21.15.161:0/1953026291 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd898054850 con 0x5651d9fc0280 2024-03-28T23:31:11.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.439+0000 7fd8817fa640 1 -- 172.21.15.161:0/1953026291 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7fd89805c370 con 0x5651d9fc0280 2024-03-28T23:31:11.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.439+0000 7fd8ab192740 1 -- 172.21.15.161:0/1953026291 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd898060780 con 0x5651d9fc0280 2024-03-28T23:31:11.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.440+0000 7fd8817fa640 1 -- 172.21.15.161:0/1953026291 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fd898077080 con 0x5651d9fc0280 2024-03-28T23:31:11.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.440+0000 7fd8ab192740 10 client.5604 fetch_fsmap learned FSMap version 22 2024-03-28T23:31:11.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.440+0000 7fd8ab192740 1 -- 172.21.15.161:0/1953026291 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=2+}) -- 0x5651d9fd80a0 con 0x5651d9fc0280 2024-03-28T23:31:11.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.440+0000 7fd8817fa640 1 -- 172.21.15.161:0/1953026291 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd898060780 con 0x5651d9fc0280 2024-03-28T23:31:11.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.440+0000 7fd8817fa640 1 -- 172.21.15.161:0/1953026291 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 22) ==== 3494+0+0 (secure 0 0 0) 0x7fd898072070 con 0x5651d9fc0280 2024-03-28T23:31:11.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.440+0000 7fd8ab192740 10 client.5604 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T23:31:11.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.440+0000 7fd8ab192740 10 client.5604 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-28T23:31:11.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.440+0000 7fd8ab192740 20 client.5604 populate_metadata read hostname 'smithi161' 2024-03-28T23:31:11.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.440+0000 7fd8ab192740 1 --2- 172.21.15.161:0/1953026291 >> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] conn(0x5651d9f848e0 0x5651d9f84cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.440+0000 7fd8ab192740 4 client.5604 mds_command: new command op to 4853 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:31:11.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.440+0000 7fd8ab192740 1 -- 172.21.15.161:0/1953026291 --> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651d9ef7030 con 0x5651d9f848e0 2024-03-28T23:31:11.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.440+0000 7fd8a5a68640 1 --2- 172.21.15.161:0/1953026291 >> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] conn(0x5651d9f848e0 0x5651d9f84cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.440+0000 7fd8817fa640 10 client.5604 ms_handle_connect on v2:172.21.15.170:6832/11130685 2024-03-28T23:31:11.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.440+0000 7fd8a5a68640 1 --2- 172.21.15.161:0/1953026291 >> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] conn(0x5651d9f848e0 0x5651d9f84cc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcecf7475f0ce6be server_cookie=c493c6927157b7fa in_seq=0 out_seq=0 2024-03-28T23:31:11.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.441+0000 7fd8817fa640 1 -- 172.21.15.161:0/1953026291 <== mds.0 v2:172.21.15.170:6832/11130685 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5651d9ef7030 con 0x5651d9f848e0 2024-03-28T23:31:11.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.441+0000 7fd8817fa640 10 client.5604 handle_command_reply: tid=0 2024-03-28T23:31:11.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.441+0000 7fd8ab192740 1 client.5604 shutdown 2024-03-28T23:31:11.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.442+0000 7fd8ab192740 1 -- 172.21.15.161:0/1953026291 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fc0280 msgr2=0x5651d9f81120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:11.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.442+0000 7fd8ab192740 1 --2- 172.21.15.161:0/1953026291 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fc0280 0x5651d9f81120 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd89800cfd0 tx=0x7fd898000f40 comp rx=0 tx=0).stop 2024-03-28T23:31:11.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.442+0000 7fd8ab192740 1 -- 172.21.15.161:0/1953026291 shutdown_connections 2024-03-28T23:31:11.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.442+0000 7fd8ab192740 1 --2- 172.21.15.161:0/1953026291 >> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] conn(0x5651d9f848e0 0x5651d9f84cc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.442+0000 7fd8ab192740 1 --2- 172.21.15.161:0/1953026291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651d9fd8840 0x5651d9fd8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.442+0000 7fd8ab192740 1 --2- 172.21.15.161:0/1953026291 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651d9fcf660 0x5651d9fda660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.442+0000 7fd8817fa640 0 client.5604 ms_handle_reset on v2:172.21.15.170:6832/11130685 2024-03-28T23:31:11.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.442+0000 7fd8ab192740 1 --2- 172.21.15.161:0/1953026291 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fc0280 0x5651d9f81120 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.442+0000 7fd8ab192740 1 -- 172.21.15.161:0/1953026291 >> 172.21.15.161:0/1953026291 conn(0x5651d9f863b0 msgr2=0x5651d9fccac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:11.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.442+0000 7fd8ab192740 1 -- 172.21.15.161:0/1953026291 shutdown_connections 2024-03-28T23:31:11.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.442+0000 7fd8ab192740 1 -- 172.21.15.161:0/1953026291 wait complete. 2024-03-28T23:31:11.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.442+0000 7fd8ab192740 20 client.5604 trim_cache size 0 max 16384 2024-03-28T23:31:11.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.460+0000 7fd8ab192740 1 Processor -- start 2024-03-28T23:31:11.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.460+0000 7fd8ab192740 1 -- start start 2024-03-28T23:31:11.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.460+0000 7fd8ab192740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651d9fc0280 0x5651d9fc1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.460+0000 7fd8ab192740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651d9fcf660 0x5651d9fcb580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.460+0000 7fd8ab192740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fd8840 0x5651d9fcbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.460+0000 7fd8ab192740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a01f2d90 con 0x5651d9fd8840 2024-03-28T23:31:11.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.460+0000 7fd8ab192740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a01f20f0 con 0x5651d9fc0280 2024-03-28T23:31:11.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.460+0000 7fd8ab192740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd8a01f2740 con 0x5651d9fcf660 2024-03-28T23:31:11.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.460+0000 7fd8a6a6a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fd8840 0x5651d9fcbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.460+0000 7fd8a6a6a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fd8840 0x5651d9fcbac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50900/0 (socket says 172.21.15.161:50900) 2024-03-28T23:31:11.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.460+0000 7fd8a6a6a640 1 -- 172.21.15.161:0/666519878 learned_addr learned my addr 172.21.15.161:0/666519878 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:11.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.461+0000 7fd8a6269640 1 --2- 172.21.15.161:0/666519878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651d9fc0280 0x5651d9fc1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.461+0000 7fd8a5a68640 1 --2- 172.21.15.161:0/666519878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651d9fcf660 0x5651d9fcb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.461+0000 7fd8a6a6a640 1 -- 172.21.15.161:0/666519878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651d9fcf660 msgr2=0x5651d9fcb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:11.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.461+0000 7fd8a6a6a640 1 --2- 172.21.15.161:0/666519878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651d9fcf660 0x5651d9fcb580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.461+0000 7fd8a6a6a640 1 -- 172.21.15.161:0/666519878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651d9fc0280 msgr2=0x5651d9fc1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:11.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.461+0000 7fd8a6a6a640 1 --2- 172.21.15.161:0/666519878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651d9fc0280 0x5651d9fc1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.461+0000 7fd8a6a6a640 1 -- 172.21.15.161:0/666519878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651d9f7f3d0 con 0x5651d9fd8840 2024-03-28T23:31:11.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.461+0000 7fd8a6269640 1 --2- 172.21.15.161:0/666519878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651d9fc0280 0x5651d9fc1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:11.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.461+0000 7fd8a6a6a640 1 --2- 172.21.15.161:0/666519878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fd8840 0x5651d9fcbac0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fd89c009890 tx=0x7fd89c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=820ff6f143fd059e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:11.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.461+0000 7fd8837fe640 1 -- 172.21.15.161:0/666519878 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd89c036040 con 0x5651d9fd8840 2024-03-28T23:31:11.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.461+0000 7fd8837fe640 1 -- 172.21.15.161:0/666519878 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd89c032c70 con 0x5651d9fd8840 2024-03-28T23:31:11.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.461+0000 7fd8837fe640 1 -- 172.21.15.161:0/666519878 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd89c046660 con 0x5651d9fd8840 2024-03-28T23:31:11.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.461+0000 7fd8ab192740 1 -- 172.21.15.161:0/666519878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fd8840 msgr2=0x5651d9fcbac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:11.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.461+0000 7fd8ab192740 1 --2- 172.21.15.161:0/666519878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fd8840 0x5651d9fcbac0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fd89c009890 tx=0x7fd89c032000 comp rx=0 tx=0).stop 2024-03-28T23:31:11.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.462+0000 7fd8ab192740 1 -- 172.21.15.161:0/666519878 shutdown_connections 2024-03-28T23:31:11.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.462+0000 7fd8ab192740 1 --2- 172.21.15.161:0/666519878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fd8840 0x5651d9fcbac0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.462+0000 7fd8ab192740 1 --2- 172.21.15.161:0/666519878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651d9fcf660 0x5651d9fcb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.462+0000 7fd8ab192740 1 --2- 172.21.15.161:0/666519878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651d9fc0280 0x5651d9fc1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.462+0000 7fd8ab192740 1 -- 172.21.15.161:0/666519878 >> 172.21.15.161:0/666519878 conn(0x5651d9f863b0 msgr2=0x5651d9f7b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:11.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.462+0000 7fd8ab192740 1 -- 172.21.15.161:0/666519878 shutdown_connections 2024-03-28T23:31:11.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.462+0000 7fd8ab192740 1 -- 172.21.15.161:0/666519878 wait complete. 2024-03-28T23:31:11.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.462+0000 7fd8ab192740 1 -- start start 2024-03-28T23:31:11.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.462+0000 7fd8ab192740 1 Processor -- start 2024-03-28T23:31:11.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.463+0000 7fd8ab192740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fc0280 0x5651d9fb8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.463+0000 7fd8ab192740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651d9fcf660 0x5651d9fbb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.463+0000 7fd8ab192740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651d9fd8840 0x5651d9fea580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.463+0000 7fd8ab192740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5651d9ea5f70 con 0x5651d9fc0280 2024-03-28T23:31:11.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.463+0000 7fd8ab192740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5651d9fa48e0 con 0x5651d9fd8840 2024-03-28T23:31:11.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.463+0000 7fd8ab192740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x5651d9fc43f0 con 0x5651d9fcf660 2024-03-28T23:31:11.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.463+0000 7fd8ab192740 10 client.0 _ll_register_callbacks cb 0x7fd89c009620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:31:11.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.463+0000 7fd8a6269640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fc0280 0x5651d9fb8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.463+0000 7fd8a6269640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fc0280 0x5651d9fb8ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50908/0 (socket says 172.21.15.161:50908) 2024-03-28T23:31:11.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.463+0000 7fd8a6269640 1 -- 172.21.15.161:0/784124319 learned_addr learned my addr 172.21.15.161:0/784124319 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:11.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.463+0000 7fd8a6a6a640 1 --2- 172.21.15.161:0/784124319 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651d9fd8840 0x5651d9fea580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.463+0000 7fd8a5a68640 1 --2- 172.21.15.161:0/784124319 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651d9fcf660 0x5651d9fbb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.463+0000 7fd8a6269640 1 -- 172.21.15.161:0/784124319 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651d9fcf660 msgr2=0x5651d9fbb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:11.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.463+0000 7fd8a6269640 1 --2- 172.21.15.161:0/784124319 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651d9fcf660 0x5651d9fbb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.463+0000 7fd8a6269640 1 -- 172.21.15.161:0/784124319 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651d9fd8840 msgr2=0x5651d9fea580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:11.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.463+0000 7fd8a6269640 1 --2- 172.21.15.161:0/784124319 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651d9fd8840 0x5651d9fea580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.463+0000 7fd8a6269640 1 -- 172.21.15.161:0/784124319 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651d9fb7f90 con 0x5651d9fc0280 2024-03-28T23:31:11.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.463+0000 7fd8a6269640 1 --2- 172.21.15.161:0/784124319 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fc0280 0x5651d9fb8ad0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fd89800afd0 tx=0x7fd898001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50685480a16787b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:11.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.464+0000 7fd8817fa640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:31:11.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.464+0000 7fd8817fa640 1 -- 172.21.15.161:0/784124319 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd898054850 con 0x5651d9fc0280 2024-03-28T23:31:11.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.464+0000 7fd8817fa640 1 -- 172.21.15.161:0/784124319 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7fd898079d00 con 0x5651d9fc0280 2024-03-28T23:31:11.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.464+0000 7fd8ab192740 1 -- 172.21.15.161:0/784124319 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd89c032c70 con 0x5651d9fc0280 2024-03-28T23:31:11.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.464+0000 7fd8817fa640 1 -- 172.21.15.161:0/784124319 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fd898060e60 con 0x5651d9fc0280 2024-03-28T23:31:11.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.464+0000 7fd8ab192740 10 client.5610 fetch_fsmap learned FSMap version 22 2024-03-28T23:31:11.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.464+0000 7fd8ab192740 1 -- 172.21.15.161:0/784124319 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=2+}) -- 0x5651d9fb7f90 con 0x5651d9fc0280 2024-03-28T23:31:11.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.464+0000 7fd8817fa640 1 -- 172.21.15.161:0/784124319 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd89c032c70 con 0x5651d9fc0280 2024-03-28T23:31:11.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.464+0000 7fd8817fa640 1 -- 172.21.15.161:0/784124319 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 22) ==== 3494+0+0 (secure 0 0 0) 0x7fd898080060 con 0x5651d9fc0280 2024-03-28T23:31:11.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.464+0000 7fd8ab192740 10 client.5610 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T23:31:11.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.464+0000 7fd8ab192740 10 client.5610 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-28T23:31:11.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.464+0000 7fd8ab192740 20 client.5610 populate_metadata read hostname 'smithi161' 2024-03-28T23:31:11.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.464+0000 7fd8ab192740 1 --2- 172.21.15.161:0/784124319 >> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] conn(0x5651da03dde0 0x5651da043890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.464+0000 7fd8ab192740 4 client.5610 mds_command: new command op to 4853 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:31:11.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.464+0000 7fd8ab192740 1 -- 172.21.15.161:0/784124319 --> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] -- command(tid 0: {"prefix": "status"}) -- 0x5651d9ef1380 con 0x5651da03dde0 2024-03-28T23:31:11.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.464+0000 7fd8a5a68640 1 --2- 172.21.15.161:0/784124319 >> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] conn(0x5651da03dde0 0x5651da043890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.465+0000 7fd8817fa640 10 client.5610 ms_handle_connect on v2:172.21.15.170:6832/11130685 2024-03-28T23:31:11.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.465+0000 7fd8a5a68640 1 --2- 172.21.15.161:0/784124319 >> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] conn(0x5651da03dde0 0x5651da043890 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe25ea9cde07ff36 server_cookie=85868a13718e38a5 in_seq=0 out_seq=0 2024-03-28T23:31:11.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.465+0000 7fd8817fa640 1 -- 172.21.15.161:0/784124319 <== mds.0 v2:172.21.15.170:6832/11130685 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x5651d9ef1380 con 0x5651da03dde0 2024-03-28T23:31:11.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.465+0000 7fd8817fa640 10 client.5610 handle_command_reply: tid=0 2024-03-28T23:31:11.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.465+0000 7fd8ab192740 1 client.5610 shutdown 2024-03-28T23:31:11.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.465+0000 7fd8ab192740 1 -- 172.21.15.161:0/784124319 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fc0280 msgr2=0x5651d9fb8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:11.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.465+0000 7fd8ab192740 1 --2- 172.21.15.161:0/784124319 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fc0280 0x5651d9fb8ad0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fd89800afd0 tx=0x7fd898001250 comp rx=0 tx=0).stop 2024-03-28T23:31:11.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.465+0000 7fd8ab192740 1 -- 172.21.15.161:0/784124319 shutdown_connections 2024-03-28T23:31:11.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.465+0000 7fd8ab192740 1 --2- 172.21.15.161:0/784124319 >> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] conn(0x5651da03dde0 0x5651da043890 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.465+0000 7fd8ab192740 1 --2- 172.21.15.161:0/784124319 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651d9fd8840 0x5651d9fea580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.465+0000 7fd8ab192740 1 --2- 172.21.15.161:0/784124319 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651d9fcf660 0x5651d9fbb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.465+0000 7fd8817fa640 0 client.5610 ms_handle_reset on v2:172.21.15.170:6832/11130685 2024-03-28T23:31:11.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.465+0000 7fd8ab192740 1 --2- 172.21.15.161:0/784124319 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651d9fc0280 0x5651d9fb8ad0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.465+0000 7fd8ab192740 1 -- 172.21.15.161:0/784124319 >> 172.21.15.161:0/784124319 conn(0x5651d9f863b0 msgr2=0x5651d9fd9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:11.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.465+0000 7fd8ab192740 1 -- 172.21.15.161:0/784124319 shutdown_connections 2024-03-28T23:31:11.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.465+0000 7fd8ab192740 1 -- 172.21.15.161:0/784124319 wait complete. 2024-03-28T23:31:11.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.465+0000 7fd8ab192740 20 client.5610 trim_cache size 0 max 16384 2024-03-28T23:31:11.474 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:31:11.474 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:31:11.474 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:31:11.475 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 4853, 2024-03-28T23:31:11.475 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:31:11.475 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:31:11.475 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:31:11.475 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 3.828495947, 2024-03-28T23:31:11.475 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 22, 2024-03-28T23:31:11.475 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 39, 2024-03-28T23:31:11.475 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 39, 2024-03-28T23:31:11.475 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 20.094100682000001 2024-03-28T23:31:11.475 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:31:11.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.466+0000 7fd8a84cd640 1 -- 172.21.15.161:0/1569328534 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd88406b500 msgr2=0x7fd88406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:11.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.466+0000 7fd8a84cd640 1 --2- 172.21.15.161:0/1569328534 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd88406b500 0x7fd88406d9c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd890009560 tx=0x7fd890009190 comp rx=0 tx=0).stop 2024-03-28T23:31:11.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.466+0000 7fd8a84cd640 1 -- 172.21.15.161:0/1569328534 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8a006ce80 msgr2=0x7fd8a01ddae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:11.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.466+0000 7fd8a84cd640 1 --2- 172.21.15.161:0/1569328534 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8a006ce80 0x7fd8a01ddae0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd89800caa0 tx=0x7fd89800cf60 comp rx=0 tx=0).stop 2024-03-28T23:31:11.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.466+0000 7fd8a84cd640 1 -- 172.21.15.161:0/1569328534 shutdown_connections 2024-03-28T23:31:11.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.466+0000 7fd8a84cd640 1 --2- 172.21.15.161:0/1569328534 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd88406b500 0x7fd88406d9c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.466+0000 7fd8a84cd640 1 --2- 172.21.15.161:0/1569328534 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8a00a9680 0x7fd8a0144840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.466+0000 7fd8a84cd640 1 --2- 172.21.15.161:0/1569328534 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8a014be70 0x7fd8a0144d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.466+0000 7fd8a84cd640 1 --2- 172.21.15.161:0/1569328534 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8a006ce80 0x7fd8a01ddae0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.466+0000 7fd8a84cd640 1 -- 172.21.15.161:0/1569328534 >> 172.21.15.161:0/1569328534 conn(0x7fd8a00c6670 msgr2=0x7fd8a014a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:11.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.466+0000 7fd8a84cd640 1 -- 172.21.15.161:0/1569328534 shutdown_connections 2024-03-28T23:31:11.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.466+0000 7fd8a84cd640 1 -- 172.21.15.161:0/1569328534 wait complete. 2024-03-28T23:31:11.478 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 22, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:30:57.616423+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 22, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4853": { "addr": "172.21.15.170:6834/11130685", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 11130685, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 11130685, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4853, "incarnation": 19, "join_fscid": 2, "name": "d", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-03-28T23:31:10.642963+0000", "qdb_cluster": [ 4853 ], "qdb_leader": 4853, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4853 } } } ], "standbys": [ { "addr": "172.21.15.170:6835/465771987", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 465771987, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 465771987, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5110, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6836/306644174", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 306644174, "type": "v2" }, { "addr": "172.21.15.161:6836", "nonce": 306644174, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5355, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6837/1649814516", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6835", "nonce": 1649814516, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 1649814516, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 20, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5589, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:31:11.479 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 2, 'name': 'd', 'state': 'up:active'}] 2024-03-28T23:31:11.479 INFO:tasks.cephfs.test_failover:current = [{'gid': 5589, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6837/1649814516', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 1649814516}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 1649814516}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 20}, {'gid': 5110, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/465771987', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 465771987}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 465771987}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 13}, {'gid': 5355, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6836/306644174', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 306644174}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 306644174}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 12}, {'gid': 4853, 'name': 'd', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.170:6834/11130685', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 11130685}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 11130685}]}, 'join_fscid': 2, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}] 2024-03-28T23:31:11.479 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-28T23:31:11.480 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-03-28T23:31:11.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.582+0000 7f1a90e74640 1 Processor -- start 2024-03-28T23:31:11.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.582+0000 7f1a90e74640 1 -- start start 2024-03-28T23:31:11.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.583+0000 7f1a90e74640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1a8c14f5c0 0x7f1a8c0c2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.583+0000 7f1a90e74640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a8c0c2f10 con 0x7f1a8c06cdd0 2024-03-28T23:31:11.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.583+0000 7f1a90e74640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a8c14f9a0 con 0x7f1a8c152bd0 2024-03-28T23:31:11.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.583+0000 7f1a90e74640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1a8c0c74d0 con 0x7f1a8c14f5c0 2024-03-28T23:31:11.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.583+0000 7f1a8a59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1a8c14f5c0 0x7f1a8c0c2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.583+0000 7f1a8a59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1a8c14f5c0 0x7f1a8c0c2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50916/0 (socket says 172.21.15.161:50916) 2024-03-28T23:31:11.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.583+0000 7f1a8a59c640 1 -- 172.21.15.161:0/2302759579 learned_addr learned my addr 172.21.15.161:0/2302759579 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:11.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.583+0000 7f1a89d9b640 1 --1- 172.21.15.161:0/2302759579 >> v1:172.21.15.161:6789/0 conn(0x7f1a8c06cdd0 0x7f1a8c1524c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:47996/0 (socket says 172.21.15.161:47996) 2024-03-28T23:31:11.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.584+0000 7f1a8959a640 1 -- 172.21.15.161:0/2302759579 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2706734766 0 0) 0x7f1a8c14f9a0 con 0x7f1a8c152bd0 2024-03-28T23:31:11.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.584+0000 7f1a8a59c640 1 -- 172.21.15.161:0/2302759579 >> v1:172.21.15.170:6789/0 conn(0x7f1a8c152bd0 legacy=0x7f1a8c15c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:11.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.584+0000 7f1a8a59c640 1 -- 172.21.15.161:0/2302759579 >> v1:172.21.15.161:6789/0 conn(0x7f1a8c06cdd0 legacy=0x7f1a8c1524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:11.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.584+0000 7f1a8a59c640 1 -- 172.21.15.161:0/2302759579 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a8c06c140 con 0x7f1a8c14f5c0 2024-03-28T23:31:11.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.584+0000 7f1a8a59c640 1 --2- 172.21.15.161:0/2302759579 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a8c14f5c0 0x7f1a8c0c2150 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f1a80009870 tx=0x7f1a8002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fdf7a5492344483 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:11.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.585+0000 7f1a8959a640 1 -- 172.21.15.161:0/2302759579 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a80038070 con 0x7f1a8c14f5c0 2024-03-28T23:31:11.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.585+0000 7f1a8959a640 1 -- 172.21.15.161:0/2302759579 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a80031740 con 0x7f1a8c14f5c0 2024-03-28T23:31:11.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.585+0000 7f1a8959a640 1 -- 172.21.15.161:0/2302759579 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8002f050 con 0x7f1a8c14f5c0 2024-03-28T23:31:11.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.585+0000 7f1a90e74640 1 -- 172.21.15.161:0/2302759579 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a8c14f5c0 msgr2=0x7f1a8c0c2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:11.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.585+0000 7f1a90e74640 1 --2- 172.21.15.161:0/2302759579 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a8c14f5c0 0x7f1a8c0c2150 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f1a80009870 tx=0x7f1a8002b860 comp rx=0 tx=0).stop 2024-03-28T23:31:11.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.585+0000 7f1a90e74640 1 -- 172.21.15.161:0/2302759579 shutdown_connections 2024-03-28T23:31:11.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.585+0000 7f1a90e74640 1 --2- 172.21.15.161:0/2302759579 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a8c14f5c0 0x7f1a8c0c2150 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.585+0000 7f1a90e74640 1 -- 172.21.15.161:0/2302759579 >> 172.21.15.161:0/2302759579 conn(0x7f1a8c0a83f0 msgr2=0x7f1a8c0a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:11.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.585+0000 7f1a90e74640 1 -- 172.21.15.161:0/2302759579 shutdown_connections 2024-03-28T23:31:11.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.585+0000 7f1a90e74640 1 -- 172.21.15.161:0/2302759579 wait complete. 2024-03-28T23:31:11.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.587+0000 7f1a90e74640 1 Processor -- start 2024-03-28T23:31:11.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.587+0000 7f1a90e74640 1 -- start start 2024-03-28T23:31:11.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.587+0000 7f1a90e74640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a8c06cdd0 0x7f1a8c1e6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.587+0000 7f1a90e74640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a8c14f5c0 0x7f1a8c1dcd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.587+0000 7f1a90e74640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a8c152bd0 0x7f1a8c0c3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.587+0000 7f1a90e74640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c152030 con 0x7f1a8c06cdd0 2024-03-28T23:31:11.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.587+0000 7f1a90e74640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c0c6fa0 con 0x7f1a8c152bd0 2024-03-28T23:31:11.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.587+0000 7f1a90e74640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1a8c06c3e0 con 0x7f1a8c14f5c0 2024-03-28T23:31:11.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.587+0000 7f1a8a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a8c06cdd0 0x7f1a8c1e6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.587+0000 7f1a8a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a8c06cdd0 0x7f1a8c1e6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50922/0 (socket says 172.21.15.161:50922) 2024-03-28T23:31:11.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.587+0000 7f1a8a59c640 1 -- 172.21.15.161:0/282687064 learned_addr learned my addr 172.21.15.161:0/282687064 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:11.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.587+0000 7f1a8ad9d640 1 --2- 172.21.15.161:0/282687064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a8c152bd0 0x7f1a8c0c3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.587+0000 7f1a89d9b640 1 --2- 172.21.15.161:0/282687064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a8c14f5c0 0x7f1a8c1dcd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.588+0000 7f1a8a59c640 1 -- 172.21.15.161:0/282687064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a8c14f5c0 msgr2=0x7f1a8c1dcd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:11.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.588+0000 7f1a8a59c640 1 --2- 172.21.15.161:0/282687064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a8c14f5c0 0x7f1a8c1dcd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.588+0000 7f1a8a59c640 1 -- 172.21.15.161:0/282687064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a8c152bd0 msgr2=0x7f1a8c0c3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:11.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.588+0000 7f1a8a59c640 1 --2- 172.21.15.161:0/282687064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a8c152bd0 0x7f1a8c0c3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.588+0000 7f1a8a59c640 1 -- 172.21.15.161:0/282687064 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a8c0c66b0 con 0x7f1a8c06cdd0 2024-03-28T23:31:11.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.588+0000 7f1a8a59c640 1 --2- 172.21.15.161:0/282687064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a8c06cdd0 0x7f1a8c1e6840 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f1a80009870 tx=0x7f1a800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:11.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.588+0000 7f1a7affd640 1 -- 172.21.15.161:0/282687064 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a80043070 con 0x7f1a8c06cdd0 2024-03-28T23:31:11.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.588+0000 7f1a7affd640 1 -- 172.21.15.161:0/282687064 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a80031e80 con 0x7f1a8c06cdd0 2024-03-28T23:31:11.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.588+0000 7f1a7affd640 1 -- 172.21.15.161:0/282687064 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a80038040 con 0x7f1a8c06cdd0 2024-03-28T23:31:11.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.588+0000 7f1a90e74640 1 -- 172.21.15.161:0/282687064 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a8c0c6990 con 0x7f1a8c06cdd0 2024-03-28T23:31:11.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.588+0000 7f1a90e74640 1 -- 172.21.15.161:0/282687064 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a8c1e9db0 con 0x7f1a8c06cdd0 2024-03-28T23:31:11.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.589+0000 7f1a90e74640 1 -- 172.21.15.161:0/282687064 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a8c06e570 con 0x7f1a8c06cdd0 2024-03-28T23:31:11.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.590+0000 7f1a7affd640 1 -- 172.21.15.161:0/282687064 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1a80046070 con 0x7f1a8c06cdd0 2024-03-28T23:31:11.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.590+0000 7f1a7affd640 1 --2- 172.21.15.161:0/282687064 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1a6c06b4d0 0x7f1a6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.590+0000 7f1a7affd640 1 -- 172.21.15.161:0/282687064 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f1a800abf50 con 0x7f1a8c06cdd0 2024-03-28T23:31:11.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.590+0000 7f1a89d9b640 1 --2- 172.21.15.161:0/282687064 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1a6c06b4d0 0x7f1a6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.590+0000 7f1a89d9b640 1 --2- 172.21.15.161:0/282687064 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1a6c06b4d0 0x7f1a6c06d990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c0c1940 tx=0x7f1a7c009f90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:11.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.594+0000 7f1a7affd640 1 -- 172.21.15.161:0/282687064 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1a8c06e570 con 0x7f1a8c06cdd0 2024-03-28T23:31:11.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.769+0000 7f1a90e74640 1 -- 172.21.15.161:0/282687064 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) -- 0x7f1a8c04f4b0 con 0x7f1a8c06cdd0 2024-03-28T23:31:11.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.772+0000 7f1a7affd640 1 -- 172.21.15.161:0/282687064 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v13) ==== 96+0+0 (secure 0 0 0) 0x7f1a8004d030 con 0x7f1a8c06cdd0 2024-03-28T23:31:11.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.774+0000 7f1a90e74640 1 -- 172.21.15.161:0/282687064 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1a6c06b4d0 msgr2=0x7f1a6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:11.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.774+0000 7f1a90e74640 1 --2- 172.21.15.161:0/282687064 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1a6c06b4d0 0x7f1a6c06d990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c0c1940 tx=0x7f1a7c009f90 comp rx=0 tx=0).stop 2024-03-28T23:31:11.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.774+0000 7f1a90e74640 1 -- 172.21.15.161:0/282687064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a8c06cdd0 msgr2=0x7f1a8c1e6840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:11.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.774+0000 7f1a90e74640 1 --2- 172.21.15.161:0/282687064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a8c06cdd0 0x7f1a8c1e6840 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f1a80009870 tx=0x7f1a800046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.774+0000 7f1a90e74640 1 -- 172.21.15.161:0/282687064 shutdown_connections 2024-03-28T23:31:11.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.774+0000 7f1a90e74640 1 --2- 172.21.15.161:0/282687064 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1a6c06b4d0 0x7f1a6c06d990 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.774+0000 7f1a90e74640 1 --2- 172.21.15.161:0/282687064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a8c152bd0 0x7f1a8c0c3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.774+0000 7f1a90e74640 1 --2- 172.21.15.161:0/282687064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a8c14f5c0 0x7f1a8c1dcd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.774+0000 7f1a90e74640 1 --2- 172.21.15.161:0/282687064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a8c06cdd0 0x7f1a8c1e6840 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.774+0000 7f1a90e74640 1 -- 172.21.15.161:0/282687064 >> 172.21.15.161:0/282687064 conn(0x7f1a8c0a83f0 msgr2=0x7f1a8c0a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:11.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.774+0000 7f1a90e74640 1 -- 172.21.15.161:0/282687064 shutdown_connections 2024-03-28T23:31:11.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.774+0000 7f1a90e74640 1 -- 172.21.15.161:0/282687064 wait complete. 2024-03-28T23:31:11.786 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:31:11.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.885+0000 7f72b57d0640 1 Processor -- start 2024-03-28T23:31:11.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.885+0000 7f72b57d0640 1 -- start start 2024-03-28T23:31:11.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.886+0000 7f72b57d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72b006ce80 0x7f72b00a9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.886+0000 7f72b57d0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72b006d310 con 0x7f72b014eca0 2024-03-28T23:31:11.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.886+0000 7f72b57d0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72b0151850 con 0x7f72b006ca20 2024-03-28T23:31:11.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.886+0000 7f72b57d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f72b0152460 con 0x7f72b006ce80 2024-03-28T23:31:11.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.886+0000 7f72aeffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f72b014eca0 0x7f72b00c1880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48008/0 (socket says 172.21.15.161:48008) 2024-03-28T23:31:11.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.886+0000 7f72aeffd640 1 -- 172.21.15.161:0/2270348946 learned_addr learned my addr 172.21.15.161:0/2270348946 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:11.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.886+0000 7f72af7fe640 1 --2- 172.21.15.161:0/2270348946 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72b006ce80 0x7f72b00a9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.887+0000 7f72adffb640 1 -- 172.21.15.161:0/2270348946 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 121717734 0 0) 0x7f72a0001490 con 0x7f72b014eca0 2024-03-28T23:31:11.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.887+0000 7f72adffb640 1 -- 172.21.15.161:0/2270348946 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7298003520 con 0x7f72b014eca0 2024-03-28T23:31:11.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.887+0000 7f72adffb640 1 -- 172.21.15.161:0/2270348946 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2159869723 0 0) 0x7f72b0151850 con 0x7f72b006ca20 2024-03-28T23:31:11.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.887+0000 7f72adffb640 1 -- 172.21.15.161:0/2270348946 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72a0001490 con 0x7f72b006ca20 2024-03-28T23:31:11.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.887+0000 7f72af7fe640 1 -- 172.21.15.161:0/2270348946 >> v1:172.21.15.170:6789/0 conn(0x7f72b006ca20 legacy=0x7f72b00a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:11.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.887+0000 7f72af7fe640 1 -- 172.21.15.161:0/2270348946 >> v1:172.21.15.161:6789/0 conn(0x7f72b014eca0 legacy=0x7f72b00c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:11.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.887+0000 7f72af7fe640 1 -- 172.21.15.161:0/2270348946 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72b0151b30 con 0x7f72b006ce80 2024-03-28T23:31:11.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.887+0000 7f72af7fe640 1 --2- 172.21.15.161:0/2270348946 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72b006ce80 0x7f72b00a9fd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f72a80097d0 tx=0x7f72a802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=525f3fbb0d267825 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:11.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.888+0000 7f72adffb640 1 -- 172.21.15.161:0/2270348946 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a8035020 con 0x7f72b006ce80 2024-03-28T23:31:11.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.888+0000 7f72adffb640 1 -- 172.21.15.161:0/2270348946 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72a8031800 con 0x7f72b006ce80 2024-03-28T23:31:11.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.888+0000 7f72adffb640 1 -- 172.21.15.161:0/2270348946 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a8031bb0 con 0x7f72b006ce80 2024-03-28T23:31:11.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.888+0000 7f72b57d0640 1 -- 172.21.15.161:0/2270348946 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72b006ce80 msgr2=0x7f72b00a9fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:11.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.888+0000 7f72b57d0640 1 --2- 172.21.15.161:0/2270348946 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72b006ce80 0x7f72b00a9fd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f72a80097d0 tx=0x7f72a802b8a0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.888+0000 7f72b57d0640 1 -- 172.21.15.161:0/2270348946 shutdown_connections 2024-03-28T23:31:11.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.888+0000 7f72b57d0640 1 --2- 172.21.15.161:0/2270348946 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72b006ce80 0x7f72b00a9fd0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.888+0000 7f72b57d0640 1 -- 172.21.15.161:0/2270348946 >> 172.21.15.161:0/2270348946 conn(0x7f72b00c6670 msgr2=0x7f72b00c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:11.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.888+0000 7f72b57d0640 1 -- 172.21.15.161:0/2270348946 shutdown_connections 2024-03-28T23:31:11.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.888+0000 7f72b57d0640 1 -- 172.21.15.161:0/2270348946 wait complete. 2024-03-28T23:31:11.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.889+0000 7f72b57d0640 1 Processor -- start 2024-03-28T23:31:11.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.890+0000 7f72b57d0640 1 -- start start 2024-03-28T23:31:11.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.890+0000 7f72b57d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72b006ca20 0x7f72b01e9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.890+0000 7f72b57d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72b006ce80 0x7f72b0155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.890+0000 7f72b57d0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72b014eca0 0x7f72b0154420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.890+0000 7f72b57d0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f72b00aa510 con 0x7f72b014eca0 2024-03-28T23:31:11.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.890+0000 7f72b57d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f72b01520f0 con 0x7f72b006ce80 2024-03-28T23:31:11.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.890+0000 7f72b57d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f72b014f080 con 0x7f72b006ca20 2024-03-28T23:31:11.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.890+0000 7f72af7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72b014eca0 0x7f72b0154420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.890+0000 7f72af7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72b014eca0 0x7f72b0154420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50926/0 (socket says 172.21.15.161:50926) 2024-03-28T23:31:11.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.890+0000 7f72af7fe640 1 -- 172.21.15.161:0/3986837073 learned_addr learned my addr 172.21.15.161:0/3986837073 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:11.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.890+0000 7f72aeffd640 1 --2- 172.21.15.161:0/3986837073 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72b006ca20 0x7f72b01e9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.890+0000 7f72ae7fc640 1 --2- 172.21.15.161:0/3986837073 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72b006ce80 0x7f72b0155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.891+0000 7f72af7fe640 1 -- 172.21.15.161:0/3986837073 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72b006ca20 msgr2=0x7f72b01e9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:11.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.891+0000 7f72af7fe640 1 --2- 172.21.15.161:0/3986837073 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72b006ca20 0x7f72b01e9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.891+0000 7f72af7fe640 1 -- 172.21.15.161:0/3986837073 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72b006ce80 msgr2=0x7f72b0155320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:11.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.891+0000 7f72af7fe640 1 --2- 172.21.15.161:0/3986837073 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72b006ce80 0x7f72b0155320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:11.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.891+0000 7f72af7fe640 1 -- 172.21.15.161:0/3986837073 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72b00a5880 con 0x7f72b014eca0 2024-03-28T23:31:11.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.891+0000 7f72af7fe640 1 --2- 172.21.15.161:0/3986837073 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72b014eca0 0x7f72b0154420 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f72a802b8e0 tx=0x7f72a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:11.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.891+0000 7f72977fe640 1 -- 172.21.15.161:0/3986837073 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a8041070 con 0x7f72b014eca0 2024-03-28T23:31:11.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.891+0000 7f72977fe640 1 -- 172.21.15.161:0/3986837073 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72a8031e30 con 0x7f72b014eca0 2024-03-28T23:31:11.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.891+0000 7f72b57d0640 1 -- 172.21.15.161:0/3986837073 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72b0152dc0 con 0x7f72b014eca0 2024-03-28T23:31:11.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.891+0000 7f72977fe640 1 -- 172.21.15.161:0/3986837073 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a8035040 con 0x7f72b014eca0 2024-03-28T23:31:11.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.891+0000 7f72b57d0640 1 -- 172.21.15.161:0/3986837073 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72b0153c20 con 0x7f72b014eca0 2024-03-28T23:31:11.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.892+0000 7f72b57d0640 1 -- 172.21.15.161:0/3986837073 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72b00737d0 con 0x7f72b014eca0 2024-03-28T23:31:11.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.892+0000 7f72977fe640 1 -- 172.21.15.161:0/3986837073 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f72a8038020 con 0x7f72b014eca0 2024-03-28T23:31:11.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.893+0000 7f72977fe640 1 --2- 172.21.15.161:0/3986837073 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f729c06b4d0 0x7f729c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:11.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.893+0000 7f72977fe640 1 -- 172.21.15.161:0/3986837073 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f72a80abdc0 con 0x7f72b014eca0 2024-03-28T23:31:11.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.893+0000 7f72aeffd640 1 --2- 172.21.15.161:0/3986837073 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f729c06b4d0 0x7f729c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:11.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.893+0000 7f72aeffd640 1 --2- 172.21.15.161:0/3986837073 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f729c06b4d0 0x7f729c06d990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f72a0002230 tx=0x7f72a0001c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:11.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:11.897+0000 7f72977fe640 1 -- 172.21.15.161:0/3986837073 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f72b00737d0 con 0x7f72b014eca0 2024-03-28T23:31:12.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.102+0000 7f72b57d0640 1 -- 172.21.15.161:0/3986837073 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f72b00c6ab0 con 0x7f72b014eca0 2024-03-28T23:31:12.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.103+0000 7f72977fe640 1 -- 172.21.15.161:0/3986837073 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+5181 (secure 0 0 0) 0x7f72b00c6ab0 con 0x7f72b014eca0 2024-03-28T23:31:12.104 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:12.104 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":22,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5110,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/465771987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":465771987},{"type":"v1","addr":"172.21.15.170:6835","nonce":465771987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5355,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/306644174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":306644174},{"type":"v1","addr":"172.21.15.161:6836","nonce":306644174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1649814516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":1649814516},{"type":"v1","addr":"172.21.15.161:6837","nonce":1649814516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:30:57.616423+0000","modified":"2024-03-28T23:31:10.642963+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4853},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4853":{"gid":4853,"name":"d","rank":0,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.170:6834/11130685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":11130685},{"type":"v1","addr":"172.21.15.170:6834","nonce":11130685}]},"join_fscid":2,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4853,"qdb_cluster":[4853]},"id":2}]} 2024-03-28T23:31:12.104 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 22 2024-03-28T23:31:12.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.105+0000 7f72b57d0640 1 -- 172.21.15.161:0/3986837073 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f729c06b4d0 msgr2=0x7f729c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:12.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.105+0000 7f72b57d0640 1 --2- 172.21.15.161:0/3986837073 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f729c06b4d0 0x7f729c06d990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f72a0002230 tx=0x7f72a0001c00 comp rx=0 tx=0).stop 2024-03-28T23:31:12.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.105+0000 7f72b57d0640 1 -- 172.21.15.161:0/3986837073 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72b014eca0 msgr2=0x7f72b0154420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:12.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.105+0000 7f72b57d0640 1 --2- 172.21.15.161:0/3986837073 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72b014eca0 0x7f72b0154420 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f72a802b8e0 tx=0x7f72a80046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.105+0000 7f72b57d0640 1 -- 172.21.15.161:0/3986837073 shutdown_connections 2024-03-28T23:31:12.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.105+0000 7f72b57d0640 1 --2- 172.21.15.161:0/3986837073 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f729c06b4d0 0x7f729c06d990 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.105+0000 7f72b57d0640 1 --2- 172.21.15.161:0/3986837073 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72b014eca0 0x7f72b0154420 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.105+0000 7f72b57d0640 1 --2- 172.21.15.161:0/3986837073 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72b006ce80 0x7f72b0155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.105+0000 7f72b57d0640 1 --2- 172.21.15.161:0/3986837073 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72b006ca20 0x7f72b01e9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.105+0000 7f72b57d0640 1 -- 172.21.15.161:0/3986837073 >> 172.21.15.161:0/3986837073 conn(0x7f72b00c6670 msgr2=0x7f72b01503b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:12.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.105+0000 7f72b57d0640 1 -- 172.21.15.161:0/3986837073 shutdown_connections 2024-03-28T23:31:12.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.105+0000 7f72b57d0640 1 -- 172.21.15.161:0/3986837073 wait complete. 2024-03-28T23:31:12.519 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:30:57.616423+0000', 'modified': '2024-03-28T23:31:10.642963+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 39, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4853}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4853': {'gid': 4853, 'name': 'd', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.170:6834/11130685', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 11130685}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 11130685}]}, 'join_fscid': 2, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4853, 'qdb_cluster': [4853]} 2024-03-28T23:31:12.519 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:31:12.519 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-28T23:31:12.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.622+0000 7f9c6e11e640 1 Processor -- start 2024-03-28T23:31:12.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.622+0000 7f9c6e11e640 1 -- start start 2024-03-28T23:31:12.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.623+0000 7f9c6e11e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9c680aacd0 0x7f9c6815d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:12.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.623+0000 7f9c6e11e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c6814c250 con 0x7f9c6814be70 2024-03-28T23:31:12.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.623+0000 7f9c6e11e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c681529c0 con 0x7f9c6806ce80 2024-03-28T23:31:12.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.623+0000 7f9c6e11e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f9c680a9dc0 con 0x7f9c680aacd0 2024-03-28T23:31:12.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.623+0000 7f9c67fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9c680aacd0 0x7f9c6815d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:12.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.623+0000 7f9c677fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f9c6814be70 0x7f9c680ab1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48024/0 (socket says 172.21.15.161:48024) 2024-03-28T23:31:12.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.623+0000 7f9c677fe640 1 -- 172.21.15.161:0/439857668 learned_addr learned my addr 172.21.15.161:0/439857668 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:12.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.623+0000 7f9c67fff640 1 -- 172.21.15.161:0/439857668 >> v1:172.21.15.170:6789/0 conn(0x7f9c6806ce80 legacy=0x7f9c680aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:12.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.623+0000 7f9c67fff640 1 -- 172.21.15.161:0/439857668 >> v1:172.21.15.161:6789/0 conn(0x7f9c6814be70 legacy=0x7f9c680ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:12.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.623+0000 7f9c67fff640 1 -- 172.21.15.161:0/439857668 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c680aa0a0 con 0x7f9c680aacd0 2024-03-28T23:31:12.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.623+0000 7f9c667fc640 1 -- 172.21.15.161:0/439857668 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4108599424 0 0) 0x7f9c6814c250 con 0x7f9c6814be70 2024-03-28T23:31:12.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.624+0000 7f9c67fff640 1 --2- 172.21.15.161:0/439857668 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c680aacd0 0x7f9c6815d750 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9c60009d90 tx=0x7f9c6002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f4ccf4e9a4b24a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:12.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.624+0000 7f9c667fc640 1 -- 172.21.15.161:0/439857668 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c60039070 con 0x7f9c680aacd0 2024-03-28T23:31:12.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.624+0000 7f9c667fc640 1 -- 172.21.15.161:0/439857668 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c6002cc60 con 0x7f9c680aacd0 2024-03-28T23:31:12.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.624+0000 7f9c667fc640 1 -- 172.21.15.161:0/439857668 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c60034660 con 0x7f9c680aacd0 2024-03-28T23:31:12.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.625+0000 7f9c6e11e640 1 -- 172.21.15.161:0/439857668 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c680aacd0 msgr2=0x7f9c6815d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:12.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.625+0000 7f9c6e11e640 1 --2- 172.21.15.161:0/439857668 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c680aacd0 0x7f9c6815d750 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9c60009d90 tx=0x7f9c6002beb0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.625+0000 7f9c6e11e640 1 -- 172.21.15.161:0/439857668 shutdown_connections 2024-03-28T23:31:12.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.625+0000 7f9c6e11e640 1 --2- 172.21.15.161:0/439857668 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c680aacd0 0x7f9c6815d750 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.625+0000 7f9c6e11e640 1 -- 172.21.15.161:0/439857668 >> 172.21.15.161:0/439857668 conn(0x7f9c680c6670 msgr2=0x7f9c680c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:12.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.625+0000 7f9c6e11e640 1 -- 172.21.15.161:0/439857668 shutdown_connections 2024-03-28T23:31:12.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.625+0000 7f9c6e11e640 1 -- 172.21.15.161:0/439857668 wait complete. 2024-03-28T23:31:12.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.626+0000 7f9c6e11e640 1 Processor -- start 2024-03-28T23:31:12.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.626+0000 7f9c6e11e640 1 -- start start 2024-03-28T23:31:12.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.627+0000 7f9c6e11e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c6806ce80 0x7f9c68145870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:12.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.627+0000 7f9c6e11e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c680aacd0 0x7f9c68145db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:12.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.627+0000 7f9c6e11e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c6814be70 0x7f9c681ec440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:12.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.627+0000 7f9c6e11e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9c680aba30 con 0x7f9c680aacd0 2024-03-28T23:31:12.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.627+0000 7f9c6e11e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9c680ab8b0 con 0x7f9c6806ce80 2024-03-28T23:31:12.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.627+0000 7f9c6e11e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9c6806b150 con 0x7f9c6814be70 2024-03-28T23:31:12.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.627+0000 7f9c66ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c680aacd0 0x7f9c68145db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:12.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.627+0000 7f9c66ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c680aacd0 0x7f9c68145db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50952/0 (socket says 172.21.15.161:50952) 2024-03-28T23:31:12.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.627+0000 7f9c66ffd640 1 -- 172.21.15.161:0/378338925 learned_addr learned my addr 172.21.15.161:0/378338925 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:12.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.627+0000 7f9c677fe640 1 --2- 172.21.15.161:0/378338925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c6806ce80 0x7f9c68145870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:12.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.627+0000 7f9c67fff640 1 --2- 172.21.15.161:0/378338925 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c6814be70 0x7f9c681ec440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:12.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.627+0000 7f9c66ffd640 1 -- 172.21.15.161:0/378338925 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c6814be70 msgr2=0x7f9c681ec440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:12.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.627+0000 7f9c66ffd640 1 --2- 172.21.15.161:0/378338925 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c6814be70 0x7f9c681ec440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.627+0000 7f9c66ffd640 1 -- 172.21.15.161:0/378338925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c6806ce80 msgr2=0x7f9c68145870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:12.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.627+0000 7f9c66ffd640 1 --2- 172.21.15.161:0/378338925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c6806ce80 0x7f9c68145870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.627+0000 7f9c66ffd640 1 -- 172.21.15.161:0/378338925 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c680bbb00 con 0x7f9c680aacd0 2024-03-28T23:31:12.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.627+0000 7f9c677fe640 1 --2- 172.21.15.161:0/378338925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c6806ce80 0x7f9c68145870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:12.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.627+0000 7f9c66ffd640 1 --2- 172.21.15.161:0/378338925 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c680aacd0 0x7f9c68145db0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c00c400 tx=0x7f9c5c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:12.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.627+0000 7f9c4ffff640 1 -- 172.21.15.161:0/378338925 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c017070 con 0x7f9c680aacd0 2024-03-28T23:31:12.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.627+0000 7f9c4ffff640 1 -- 172.21.15.161:0/378338925 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c5c0052d0 con 0x7f9c680aacd0 2024-03-28T23:31:12.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.627+0000 7f9c4ffff640 1 -- 172.21.15.161:0/378338925 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c003830 con 0x7f9c680aacd0 2024-03-28T23:31:12.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.627+0000 7f9c6e11e640 1 -- 172.21.15.161:0/378338925 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c681ea200 con 0x7f9c680aacd0 2024-03-28T23:31:12.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.628+0000 7f9c6e11e640 1 -- 172.21.15.161:0/378338925 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c681ea7c0 con 0x7f9c680aacd0 2024-03-28T23:31:12.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.629+0000 7f9c4ffff640 1 -- 172.21.15.161:0/378338925 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9c5c01a070 con 0x7f9c680aacd0 2024-03-28T23:31:12.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.629+0000 7f9c4ffff640 1 --2- 172.21.15.161:0/378338925 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9c5406b500 0x7f9c5406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:12.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.629+0000 7f9c4ffff640 1 -- 172.21.15.161:0/378338925 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f9c5c089680 con 0x7f9c680aacd0 2024-03-28T23:31:12.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.629+0000 7f9c677fe640 1 --2- 172.21.15.161:0/378338925 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9c5406b500 0x7f9c5406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:12.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.630+0000 7f9c677fe640 1 --2- 172.21.15.161:0/378338925 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9c5406b500 0x7f9c5406d9c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9c58000f60 tx=0x7f9c58007e40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:12.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.630+0000 7f9c70dd9740 1 Processor -- start 2024-03-28T23:31:12.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.630+0000 7f9c70dd9740 1 -- start start 2024-03-28T23:31:12.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.631+0000 7f9c70dd9740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bd98b58660 0x55bd98b0ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:12.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.631+0000 7f9c70dd9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bd98b0f3b0 0x55bd98b58a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:12.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.631+0000 7f9c70dd9740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b49280 0x55bd98b094e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:12.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.631+0000 7f9c70dd9740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55bd98a72af0 con 0x55bd98b49280 2024-03-28T23:31:12.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.631+0000 7f9c70dd9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55bd98a1b9d0 con 0x55bd98b0f3b0 2024-03-28T23:31:12.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.631+0000 7f9c70dd9740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55bd98a6eb90 con 0x55bd98b58660 2024-03-28T23:31:12.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.631+0000 7f9c67fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b49280 0x55bd98b094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:12.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.631+0000 7f9c67fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b49280 0x55bd98b094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50966/0 (socket says 172.21.15.161:50966) 2024-03-28T23:31:12.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.631+0000 7f9c67fff640 1 -- 172.21.15.161:0/1121558381 learned_addr learned my addr 172.21.15.161:0/1121558381 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:12.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.631+0000 7f9c677fe640 1 --2- 172.21.15.161:0/1121558381 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bd98b58660 0x55bd98b0ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:12.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.631+0000 7f9c66ffd640 1 --2- 172.21.15.161:0/1121558381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bd98b0f3b0 0x55bd98b58a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:12.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.632+0000 7f9c67fff640 1 -- 172.21.15.161:0/1121558381 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bd98b58660 msgr2=0x55bd98b0ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:12.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.632+0000 7f9c67fff640 1 --2- 172.21.15.161:0/1121558381 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bd98b58660 0x55bd98b0ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.632+0000 7f9c67fff640 1 -- 172.21.15.161:0/1121558381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bd98b0f3b0 msgr2=0x55bd98b58a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:12.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.632+0000 7f9c67fff640 1 --2- 172.21.15.161:0/1121558381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bd98b0f3b0 0x55bd98b58a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.632+0000 7f9c67fff640 1 -- 172.21.15.161:0/1121558381 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd987872d0 con 0x55bd98b49280 2024-03-28T23:31:12.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.632+0000 7f9c677fe640 1 --2- 172.21.15.161:0/1121558381 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bd98b58660 0x55bd98b0ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:12.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.632+0000 7f9c67fff640 1 --2- 172.21.15.161:0/1121558381 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b49280 0x55bd98b094e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f9c6002bef0 tx=0x7f9c6002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba5343ea77ecc3a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:12.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.632+0000 7f9c4d7fa640 1 -- 172.21.15.161:0/1121558381 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c60039040 con 0x55bd98b49280 2024-03-28T23:31:12.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.632+0000 7f9c4d7fa640 1 -- 172.21.15.161:0/1121558381 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c6003a040 con 0x55bd98b49280 2024-03-28T23:31:12.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.632+0000 7f9c4d7fa640 1 -- 172.21.15.161:0/1121558381 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c60034860 con 0x55bd98b49280 2024-03-28T23:31:12.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.633+0000 7f9c70dd9740 1 -- 172.21.15.161:0/1121558381 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b49280 msgr2=0x55bd98b094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:12.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.633+0000 7f9c70dd9740 1 --2- 172.21.15.161:0/1121558381 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b49280 0x55bd98b094e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f9c6002bef0 tx=0x7f9c6002e000 comp rx=0 tx=0).stop 2024-03-28T23:31:12.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.633+0000 7f9c70dd9740 1 -- 172.21.15.161:0/1121558381 shutdown_connections 2024-03-28T23:31:12.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.633+0000 7f9c70dd9740 1 --2- 172.21.15.161:0/1121558381 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b49280 0x55bd98b094e0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.633+0000 7f9c70dd9740 1 --2- 172.21.15.161:0/1121558381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bd98b0f3b0 0x55bd98b58a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.633+0000 7f9c70dd9740 1 --2- 172.21.15.161:0/1121558381 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bd98b58660 0x55bd98b0ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.633+0000 7f9c70dd9740 1 -- 172.21.15.161:0/1121558381 >> 172.21.15.161:0/1121558381 conn(0x55bd98b0b630 msgr2=0x55bd98b30220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:12.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.633+0000 7f9c70dd9740 1 -- 172.21.15.161:0/1121558381 shutdown_connections 2024-03-28T23:31:12.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.633+0000 7f9c70dd9740 1 -- 172.21.15.161:0/1121558381 wait complete. 2024-03-28T23:31:12.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.633+0000 7f9c70dd9740 1 -- start start 2024-03-28T23:31:12.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.633+0000 7f9c70dd9740 1 Processor -- start 2024-03-28T23:31:12.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.634+0000 7f9c70dd9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bd98b49280 0x55bd98b54260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:12.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.634+0000 7f9c70dd9740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b58660 0x55bd98b4a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:12.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.634+0000 7f9c70dd9740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bd98b4a830 0x55bd98b2cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:12.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.634+0000 7f9c70dd9740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55bd98b4d3f0 con 0x55bd98b58660 2024-03-28T23:31:12.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.634+0000 7f9c70dd9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55bd98a2ef70 con 0x55bd98b49280 2024-03-28T23:31:12.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.634+0000 7f9c70dd9740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55bd98b30c10 con 0x55bd98b4a830 2024-03-28T23:31:12.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.634+0000 7f9c70dd9740 10 client.0 _ll_register_callbacks cb 0x55bd98786ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:31:12.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.634+0000 7f9c66ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b58660 0x55bd98b4a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:12.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.634+0000 7f9c677fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bd98b49280 0x55bd98b54260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:12.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.634+0000 7f9c66ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b58660 0x55bd98b4a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50974/0 (socket says 172.21.15.161:50974) 2024-03-28T23:31:12.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.634+0000 7f9c66ffd640 1 -- 172.21.15.161:0/2825790102 learned_addr learned my addr 172.21.15.161:0/2825790102 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:12.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.634+0000 7f9c67fff640 1 --2- 172.21.15.161:0/2825790102 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bd98b4a830 0x55bd98b2cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:12.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.634+0000 7f9c66ffd640 1 -- 172.21.15.161:0/2825790102 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bd98b4a830 msgr2=0x55bd98b2cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:12.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.634+0000 7f9c66ffd640 1 --2- 172.21.15.161:0/2825790102 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bd98b4a830 0x55bd98b2cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.634+0000 7f9c66ffd640 1 -- 172.21.15.161:0/2825790102 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bd98b49280 msgr2=0x55bd98b54260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:12.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.634+0000 7f9c66ffd640 1 --2- 172.21.15.161:0/2825790102 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bd98b49280 0x55bd98b54260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.634+0000 7f9c66ffd640 1 -- 172.21.15.161:0/2825790102 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd988516b0 con 0x55bd98b58660 2024-03-28T23:31:12.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.635+0000 7f9c66ffd640 1 --2- 172.21.15.161:0/2825790102 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b58660 0x55bd98b4a2f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c057c10 tx=0x7f9c5c005540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c527cc785b3df86 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:12.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.635+0000 7f9c467fc640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:31:12.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.635+0000 7f9c467fc640 1 -- 172.21.15.161:0/2825790102 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c06f020 con 0x55bd98b58660 2024-03-28T23:31:12.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.635+0000 7f9c467fc640 1 -- 172.21.15.161:0/2825790102 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f9c5c054340 con 0x55bd98b58660 2024-03-28T23:31:12.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.635+0000 7f9c70dd9740 1 -- 172.21.15.161:0/2825790102 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c6003a040 con 0x55bd98b58660 2024-03-28T23:31:12.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.635+0000 7f9c467fc640 1 -- 172.21.15.161:0/2825790102 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f9c5c060330 con 0x55bd98b58660 2024-03-28T23:31:12.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.635+0000 7f9c70dd9740 10 client.5640 fetch_fsmap learned FSMap version 22 2024-03-28T23:31:12.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.635+0000 7f9c70dd9740 1 -- 172.21.15.161:0/2825790102 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=2+}) -- 0x55bd98b53d40 con 0x55bd98b58660 2024-03-28T23:31:12.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.635+0000 7f9c467fc640 1 -- 172.21.15.161:0/2825790102 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c6003a040 con 0x55bd98b58660 2024-03-28T23:31:12.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.635+0000 7f9c467fc640 1 -- 172.21.15.161:0/2825790102 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 22) ==== 3494+0+0 (secure 0 0 0) 0x7f9c5c06bbe0 con 0x55bd98b58660 2024-03-28T23:31:12.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.635+0000 7f9c70dd9740 10 client.5640 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T23:31:12.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.635+0000 7f9c70dd9740 10 client.5640 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-28T23:31:12.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.635+0000 7f9c70dd9740 20 client.5640 populate_metadata read hostname 'smithi161' 2024-03-28T23:31:12.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.635+0000 7f9c70dd9740 1 --2- 172.21.15.161:0/2825790102 >> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] conn(0x55bd98b63120 0x55bd98b63500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:12.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.635+0000 7f9c70dd9740 4 client.5640 mds_command: new command op to 4853 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:31:12.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.635+0000 7f9c70dd9740 1 -- 172.21.15.161:0/2825790102 --> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd98a80030 con 0x55bd98b63120 2024-03-28T23:31:12.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.635+0000 7f9c677fe640 1 --2- 172.21.15.161:0/2825790102 >> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] conn(0x55bd98b63120 0x55bd98b63500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:12.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.636+0000 7f9c677fe640 1 --2- 172.21.15.161:0/2825790102 >> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] conn(0x55bd98b63120 0x55bd98b63500 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9effb0d8cd94651 server_cookie=8f2803b4db790fb5 in_seq=0 out_seq=0 2024-03-28T23:31:12.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.636+0000 7f9c467fc640 10 client.5640 ms_handle_connect on v2:172.21.15.170:6832/11130685 2024-03-28T23:31:12.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.636+0000 7f9c467fc640 1 -- 172.21.15.161:0/2825790102 <== mds.0 v2:172.21.15.170:6832/11130685 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bd98a80030 con 0x55bd98b63120 2024-03-28T23:31:12.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.636+0000 7f9c467fc640 10 client.5640 handle_command_reply: tid=0 2024-03-28T23:31:12.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.636+0000 7f9c70dd9740 1 client.5640 shutdown 2024-03-28T23:31:12.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.636+0000 7f9c70dd9740 1 -- 172.21.15.161:0/2825790102 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b58660 msgr2=0x55bd98b4a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:12.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.636+0000 7f9c70dd9740 1 --2- 172.21.15.161:0/2825790102 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b58660 0x55bd98b4a2f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c057c10 tx=0x7f9c5c005540 comp rx=0 tx=0).stop 2024-03-28T23:31:12.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.636+0000 7f9c70dd9740 1 -- 172.21.15.161:0/2825790102 shutdown_connections 2024-03-28T23:31:12.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.636+0000 7f9c70dd9740 1 --2- 172.21.15.161:0/2825790102 >> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] conn(0x55bd98b63120 0x55bd98b63500 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.636+0000 7f9c70dd9740 1 --2- 172.21.15.161:0/2825790102 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b58660 0x55bd98b4a2f0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.636+0000 7f9c70dd9740 1 --2- 172.21.15.161:0/2825790102 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bd98b4a830 0x55bd98b2cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.636+0000 7f9c467fc640 0 client.5640 ms_handle_reset on v2:172.21.15.170:6832/11130685 2024-03-28T23:31:12.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.636+0000 7f9c70dd9740 1 --2- 172.21.15.161:0/2825790102 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bd98b49280 0x55bd98b54260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.636+0000 7f9c70dd9740 1 -- 172.21.15.161:0/2825790102 >> 172.21.15.161:0/2825790102 conn(0x55bd98b0f3b0 msgr2=0x55bd98b55ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:12.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.637+0000 7f9c70dd9740 1 -- 172.21.15.161:0/2825790102 shutdown_connections 2024-03-28T23:31:12.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.637+0000 7f9c70dd9740 1 -- 172.21.15.161:0/2825790102 wait complete. 2024-03-28T23:31:12.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.637+0000 7f9c70dd9740 20 client.5640 trim_cache size 0 max 16384 2024-03-28T23:31:12.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.655+0000 7f9c70dd9740 1 Processor -- start 2024-03-28T23:31:12.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.655+0000 7f9c70dd9740 1 -- start start 2024-03-28T23:31:12.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.655+0000 7f9c70dd9740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b49280 0x55bd98b4c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:12.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.655+0000 7f9c70dd9740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bd98b4a830 0x55bd98b55ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:12.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.655+0000 7f9c70dd9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bd98b58660 0x55bd98b0d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:12.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.655+0000 7f9c70dd9740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9c68161510 con 0x55bd98b49280 2024-03-28T23:31:12.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.655+0000 7f9c70dd9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9c68154810 con 0x55bd98b58660 2024-03-28T23:31:12.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.655+0000 7f9c70dd9740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9c68160ec0 con 0x55bd98b4a830 2024-03-28T23:31:12.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.656+0000 7f9c677fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b49280 0x55bd98b4c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:12.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.656+0000 7f9c677fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b49280 0x55bd98b4c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50984/0 (socket says 172.21.15.161:50984) 2024-03-28T23:31:12.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.656+0000 7f9c677fe640 1 -- 172.21.15.161:0/2026395374 learned_addr learned my addr 172.21.15.161:0/2026395374 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:12.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.656+0000 7f9c66ffd640 1 --2- 172.21.15.161:0/2026395374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bd98b4a830 0x55bd98b55ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:12.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.656+0000 7f9c67fff640 1 --2- 172.21.15.161:0/2026395374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bd98b58660 0x55bd98b0d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:12.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.656+0000 7f9c677fe640 1 -- 172.21.15.161:0/2026395374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bd98b4a830 msgr2=0x55bd98b55ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:12.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.656+0000 7f9c677fe640 1 --2- 172.21.15.161:0/2026395374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bd98b4a830 0x55bd98b55ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.656+0000 7f9c677fe640 1 -- 172.21.15.161:0/2026395374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bd98b58660 msgr2=0x55bd98b0d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:12.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.656+0000 7f9c677fe640 1 --2- 172.21.15.161:0/2026395374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bd98b58660 0x55bd98b0d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.656+0000 7f9c677fe640 1 -- 172.21.15.161:0/2026395374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd98b48f30 con 0x55bd98b49280 2024-03-28T23:31:12.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.656+0000 7f9c677fe640 1 --2- 172.21.15.161:0/2026395374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b49280 0x55bd98b4c1f0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f9c58007510 tx=0x7f9c58013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61d02b65cc63286a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:12.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.656+0000 7f9c4cff9640 1 -- 172.21.15.161:0/2026395374 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c58024190 con 0x55bd98b49280 2024-03-28T23:31:12.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.656+0000 7f9c4cff9640 1 -- 172.21.15.161:0/2026395374 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c580291d0 con 0x55bd98b49280 2024-03-28T23:31:12.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.656+0000 7f9c4cff9640 1 -- 172.21.15.161:0/2026395374 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c58029580 con 0x55bd98b49280 2024-03-28T23:31:12.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.657+0000 7f9c70dd9740 1 -- 172.21.15.161:0/2026395374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b49280 msgr2=0x55bd98b4c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:12.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.657+0000 7f9c70dd9740 1 --2- 172.21.15.161:0/2026395374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b49280 0x55bd98b4c1f0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f9c58007510 tx=0x7f9c58013000 comp rx=0 tx=0).stop 2024-03-28T23:31:12.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.657+0000 7f9c70dd9740 1 -- 172.21.15.161:0/2026395374 shutdown_connections 2024-03-28T23:31:12.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.657+0000 7f9c70dd9740 1 --2- 172.21.15.161:0/2026395374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bd98b58660 0x55bd98b0d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.657+0000 7f9c70dd9740 1 --2- 172.21.15.161:0/2026395374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bd98b4a830 0x55bd98b55ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.657+0000 7f9c70dd9740 1 --2- 172.21.15.161:0/2026395374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b49280 0x55bd98b4c1f0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.657+0000 7f9c70dd9740 1 -- 172.21.15.161:0/2026395374 >> 172.21.15.161:0/2026395374 conn(0x55bd98b0f3b0 msgr2=0x55bd98b4cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:12.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.657+0000 7f9c70dd9740 1 -- 172.21.15.161:0/2026395374 shutdown_connections 2024-03-28T23:31:12.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.657+0000 7f9c70dd9740 1 -- 172.21.15.161:0/2026395374 wait complete. 2024-03-28T23:31:12.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.657+0000 7f9c70dd9740 1 -- start start 2024-03-28T23:31:12.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.658+0000 7f9c70dd9740 1 Processor -- start 2024-03-28T23:31:12.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.658+0000 7f9c70dd9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bd98b49280 0x55bd98b4e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:12.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.658+0000 7f9c70dd9740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b4a830 0x55bd98b2fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:12.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.658+0000 7f9c70dd9740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bd98b58660 0x55bd98bcbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:12.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.658+0000 7f9c70dd9740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55bd98b568d0 con 0x55bd98b4a830 2024-03-28T23:31:12.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.658+0000 7f9c70dd9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55bd98b62fa0 con 0x55bd98b49280 2024-03-28T23:31:12.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.658+0000 7f9c70dd9740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55bd98b4b960 con 0x55bd98b58660 2024-03-28T23:31:12.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.658+0000 7f9c70dd9740 10 client.0 _ll_register_callbacks cb 0x7f9c5c057810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:31:12.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.658+0000 7f9c66ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b4a830 0x55bd98b2fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:12.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.658+0000 7f9c66ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b4a830 0x55bd98b2fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50996/0 (socket says 172.21.15.161:50996) 2024-03-28T23:31:12.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.658+0000 7f9c66ffd640 1 -- 172.21.15.161:0/331005213 learned_addr learned my addr 172.21.15.161:0/331005213 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:12.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.658+0000 7f9c677fe640 1 --2- 172.21.15.161:0/331005213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bd98b49280 0x55bd98b4e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:12.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.658+0000 7f9c67fff640 1 --2- 172.21.15.161:0/331005213 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bd98b58660 0x55bd98bcbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:12.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.658+0000 7f9c66ffd640 1 -- 172.21.15.161:0/331005213 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bd98b58660 msgr2=0x55bd98bcbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:12.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.658+0000 7f9c66ffd640 1 --2- 172.21.15.161:0/331005213 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bd98b58660 0x55bd98bcbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.659+0000 7f9c66ffd640 1 -- 172.21.15.161:0/331005213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bd98b49280 msgr2=0x55bd98b4e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:12.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.659+0000 7f9c66ffd640 1 --2- 172.21.15.161:0/331005213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bd98b49280 0x55bd98b4e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.659+0000 7f9c66ffd640 1 -- 172.21.15.161:0/331005213 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd98b733d0 con 0x55bd98b4a830 2024-03-28T23:31:12.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.659+0000 7f9c66ffd640 1 --2- 172.21.15.161:0/331005213 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b4a830 0x55bd98b2fe40 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c0899a0 tx=0x7f9c5c014070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f8bb00b599f8b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:12.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.659+0000 7f9c467fc640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:31:12.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.659+0000 7f9c467fc640 1 -- 172.21.15.161:0/331005213 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c06f040 con 0x55bd98b4a830 2024-03-28T23:31:12.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.659+0000 7f9c467fc640 1 -- 172.21.15.161:0/331005213 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f9c5c0571c0 con 0x55bd98b4a830 2024-03-28T23:31:12.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.659+0000 7f9c70dd9740 1 -- 172.21.15.161:0/331005213 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c580291d0 con 0x55bd98b4a830 2024-03-28T23:31:12.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.659+0000 7f9c467fc640 1 -- 172.21.15.161:0/331005213 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f9c5c07e050 con 0x55bd98b4a830 2024-03-28T23:31:12.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.659+0000 7f9c70dd9740 10 client.5646 fetch_fsmap learned FSMap version 22 2024-03-28T23:31:12.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.659+0000 7f9c70dd9740 1 -- 172.21.15.161:0/331005213 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=2+}) -- 0x55bd98b0de50 con 0x55bd98b4a830 2024-03-28T23:31:12.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.659+0000 7f9c467fc640 1 -- 172.21.15.161:0/331005213 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c580291d0 con 0x55bd98b4a830 2024-03-28T23:31:12.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.659+0000 7f9c467fc640 1 -- 172.21.15.161:0/331005213 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 22) ==== 3494+0+0 (secure 0 0 0) 0x7f9c5c076af0 con 0x55bd98b4a830 2024-03-28T23:31:12.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.659+0000 7f9c70dd9740 10 client.5646 fetch_fsmap finished waiting for FSMap version 22 2024-03-28T23:31:12.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.659+0000 7f9c70dd9740 10 client.5646 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-28T23:31:12.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.659+0000 7f9c70dd9740 20 client.5646 populate_metadata read hostname 'smithi161' 2024-03-28T23:31:12.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.659+0000 7f9c70dd9740 1 --2- 172.21.15.161:0/331005213 >> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] conn(0x55bd98b4fa70 0x55bd98b4fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:12.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.659+0000 7f9c70dd9740 4 client.5646 mds_command: new command op to 4853 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:31:12.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.659+0000 7f9c70dd9740 1 -- 172.21.15.161:0/331005213 --> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] -- command(tid 0: {"prefix": "status"}) -- 0x55bd98a7a380 con 0x55bd98b4fa70 2024-03-28T23:31:12.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.659+0000 7f9c677fe640 1 --2- 172.21.15.161:0/331005213 >> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] conn(0x55bd98b4fa70 0x55bd98b4fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:12.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.660+0000 7f9c467fc640 10 client.5646 ms_handle_connect on v2:172.21.15.170:6832/11130685 2024-03-28T23:31:12.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.660+0000 7f9c677fe640 1 --2- 172.21.15.161:0/331005213 >> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] conn(0x55bd98b4fa70 0x55bd98b4fe50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9733faaf4e7a36f server_cookie=7a0cb517c88d76 in_seq=0 out_seq=0 2024-03-28T23:31:12.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.660+0000 7f9c467fc640 1 -- 172.21.15.161:0/331005213 <== mds.0 v2:172.21.15.170:6832/11130685 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55bd98a7a380 con 0x55bd98b4fa70 2024-03-28T23:31:12.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.660+0000 7f9c467fc640 10 client.5646 handle_command_reply: tid=0 2024-03-28T23:31:12.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.660+0000 7f9c70dd9740 1 client.5646 shutdown 2024-03-28T23:31:12.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.660+0000 7f9c70dd9740 1 -- 172.21.15.161:0/331005213 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b4a830 msgr2=0x55bd98b2fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:12.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.660+0000 7f9c70dd9740 1 --2- 172.21.15.161:0/331005213 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b4a830 0x55bd98b2fe40 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c0899a0 tx=0x7f9c5c014070 comp rx=0 tx=0).stop 2024-03-28T23:31:12.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.660+0000 7f9c70dd9740 1 -- 172.21.15.161:0/331005213 shutdown_connections 2024-03-28T23:31:12.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.660+0000 7f9c70dd9740 1 --2- 172.21.15.161:0/331005213 >> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] conn(0x55bd98b4fa70 0x55bd98b4fe50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.660+0000 7f9c70dd9740 1 --2- 172.21.15.161:0/331005213 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bd98b4a830 0x55bd98b2fe40 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.660+0000 7f9c70dd9740 1 --2- 172.21.15.161:0/331005213 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bd98b58660 0x55bd98bcbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.660+0000 7f9c70dd9740 1 --2- 172.21.15.161:0/331005213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bd98b49280 0x55bd98b4e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.660+0000 7f9c70dd9740 1 -- 172.21.15.161:0/331005213 >> 172.21.15.161:0/331005213 conn(0x55bd98b0f3b0 msgr2=0x55bd98b55ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:12.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.660+0000 7f9c467fc640 0 client.5646 ms_handle_reset on v2:172.21.15.170:6832/11130685 2024-03-28T23:31:12.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.660+0000 7f9c70dd9740 1 -- 172.21.15.161:0/331005213 shutdown_connections 2024-03-28T23:31:12.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.660+0000 7f9c70dd9740 1 -- 172.21.15.161:0/331005213 wait complete. 2024-03-28T23:31:12.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.660+0000 7f9c70dd9740 20 client.5646 trim_cache size 0 max 16384 2024-03-28T23:31:12.667 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:31:12.668 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:31:12.668 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:31:12.668 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 4853, 2024-03-28T23:31:12.668 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:31:12.668 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:31:12.668 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:31:12.668 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 5.0237041610000004, 2024-03-28T23:31:12.668 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 22, 2024-03-28T23:31:12.668 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 39, 2024-03-28T23:31:12.668 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 39, 2024-03-28T23:31:12.668 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 21.289306522 2024-03-28T23:31:12.668 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:31:12.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.661+0000 7f9c6e11e640 1 -- 172.21.15.161:0/378338925 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9c5406b500 msgr2=0x7f9c5406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:12.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.661+0000 7f9c6e11e640 1 --2- 172.21.15.161:0/378338925 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9c5406b500 0x7f9c5406d9c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9c58000f60 tx=0x7f9c58007e40 comp rx=0 tx=0).stop 2024-03-28T23:31:12.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.661+0000 7f9c6e11e640 1 -- 172.21.15.161:0/378338925 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c680aacd0 msgr2=0x7f9c68145db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:12.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.661+0000 7f9c6e11e640 1 --2- 172.21.15.161:0/378338925 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c680aacd0 0x7f9c68145db0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c00c400 tx=0x7f9c5c00c8c0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.661+0000 7f9c6e11e640 1 -- 172.21.15.161:0/378338925 shutdown_connections 2024-03-28T23:31:12.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.661+0000 7f9c6e11e640 1 --2- 172.21.15.161:0/378338925 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9c5406b500 0x7f9c5406d9c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.661+0000 7f9c6e11e640 1 --2- 172.21.15.161:0/378338925 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c680aacd0 0x7f9c68145db0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.661+0000 7f9c6e11e640 1 --2- 172.21.15.161:0/378338925 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c6814be70 0x7f9c681ec440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.661+0000 7f9c6e11e640 1 --2- 172.21.15.161:0/378338925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c6806ce80 0x7f9c68145870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:12.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.661+0000 7f9c6e11e640 1 -- 172.21.15.161:0/378338925 >> 172.21.15.161:0/378338925 conn(0x7f9c680c6670 msgr2=0x7f9c680c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:12.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.661+0000 7f9c6e11e640 1 -- 172.21.15.161:0/378338925 shutdown_connections 2024-03-28T23:31:12.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:12.661+0000 7f9c6e11e640 1 -- 172.21.15.161:0/378338925 wait complete. 2024-03-28T23:31:12.672 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 22, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:30:57.616423+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 22, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4853": { "addr": "172.21.15.170:6834/11130685", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 11130685, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 11130685, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4853, "incarnation": 19, "join_fscid": 2, "name": "d", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-03-28T23:31:10.642963+0000", "qdb_cluster": [ 4853 ], "qdb_leader": 4853, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4853 } } } ], "standbys": [ { "addr": "172.21.15.170:6835/465771987", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 465771987, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 465771987, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5110, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6836/306644174", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 306644174, "type": "v2" }, { "addr": "172.21.15.161:6836", "nonce": 306644174, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5355, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6837/1649814516", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6835", "nonce": 1649814516, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 1649814516, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 20, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5589, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:31:12.673 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd', 'state': 'up:active'}] 2024-03-28T23:31:12.673 INFO:tasks.cephfs.test_failover:current = [{'gid': 5589, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6837/1649814516', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 1649814516}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 1649814516}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 20}, {'gid': 5110, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/465771987', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 465771987}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 465771987}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 13}, {'gid': 5355, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6836/306644174', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 306644174}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 306644174}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 12}, {'gid': 4853, 'name': 'd', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.170:6834/11130685', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 11130685}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 11130685}]}, 'join_fscid': 2, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}] 2024-03-28T23:31:12.673 DEBUG:tasks.cephfs.test_failover:-1 != 2 2024-03-28T23:31:12.673 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-28T23:31:13.239 INFO:tasks.ceph.mds.d.smithi170.stderr:2024-03-28T23:31:13.238+0000 7fa26681b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:31:17.675 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:31:17.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.787+0000 7f44facc2640 1 Processor -- start 2024-03-28T23:31:17.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.787+0000 7f44facc2640 1 -- start start 2024-03-28T23:31:17.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.787+0000 7f44facc2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44f414eca0 0x7f44f40c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:17.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.787+0000 7f44facc2640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44f4151850 con 0x7f44f40c1dc0 2024-03-28T23:31:17.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.787+0000 7f44facc2640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44f4151b30 con 0x7f44f40a98c0 2024-03-28T23:31:17.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.787+0000 7f44facc2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f44f4152460 con 0x7f44f414eca0 2024-03-28T23:31:17.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.788+0000 7f44f3fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f44f40c1dc0 0x7f44f406ca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48040/0 (socket says 172.21.15.161:48040) 2024-03-28T23:31:17.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.788+0000 7f44f3fff640 1 -- 172.21.15.161:0/1928234917 learned_addr learned my addr 172.21.15.161:0/1928234917 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:17.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.788+0000 7f44f8a5e640 1 --2- 172.21.15.161:0/1928234917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44f414eca0 0x7f44f40c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:17.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.788+0000 7f44f37fe640 1 -- 172.21.15.161:0/1928234917 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2810631389 0 0) 0x7f44f4151850 con 0x7f44f40c1dc0 2024-03-28T23:31:17.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.788+0000 7f44f37fe640 1 -- 172.21.15.161:0/1928234917 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44e4003520 con 0x7f44f40c1dc0 2024-03-28T23:31:17.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.788+0000 7f44f37fe640 1 -- 172.21.15.161:0/1928234917 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 113834080 0 0) 0x7f44f4151b30 con 0x7f44f40a98c0 2024-03-28T23:31:17.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.788+0000 7f44f8a5e640 1 -- 172.21.15.161:0/1928234917 >> v1:172.21.15.170:6789/0 conn(0x7f44f40a98c0 legacy=0x7f44f40a9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:17.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.788+0000 7f44f8a5e640 1 -- 172.21.15.161:0/1928234917 >> v1:172.21.15.161:6789/0 conn(0x7f44f40c1dc0 legacy=0x7f44f406ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:17.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.788+0000 7f44f8a5e640 1 -- 172.21.15.161:0/1928234917 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f4151e10 con 0x7f44f414eca0 2024-03-28T23:31:17.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.789+0000 7f44f8a5e640 1 --2- 172.21.15.161:0/1928234917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44f414eca0 0x7f44f40c1880 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f44e80097d0 tx=0x7f44e802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6491e87dbc271c2e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:17.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.789+0000 7f44f37fe640 1 -- 172.21.15.161:0/1928234917 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e8035020 con 0x7f44f414eca0 2024-03-28T23:31:17.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.789+0000 7f44f37fe640 1 -- 172.21.15.161:0/1928234917 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44e8031800 con 0x7f44f414eca0 2024-03-28T23:31:17.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.789+0000 7f44f37fe640 1 -- 172.21.15.161:0/1928234917 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e8031bd0 con 0x7f44f414eca0 2024-03-28T23:31:17.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.790+0000 7f44facc2640 1 -- 172.21.15.161:0/1928234917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44f414eca0 msgr2=0x7f44f40c1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:17.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.790+0000 7f44facc2640 1 --2- 172.21.15.161:0/1928234917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44f414eca0 0x7f44f40c1880 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f44e80097d0 tx=0x7f44e802b8a0 comp rx=0 tx=0).stop 2024-03-28T23:31:17.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.790+0000 7f44facc2640 1 -- 172.21.15.161:0/1928234917 shutdown_connections 2024-03-28T23:31:17.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.790+0000 7f44facc2640 1 --2- 172.21.15.161:0/1928234917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44f414eca0 0x7f44f40c1880 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:17.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.790+0000 7f44facc2640 1 -- 172.21.15.161:0/1928234917 >> 172.21.15.161:0/1928234917 conn(0x7f44f40c6670 msgr2=0x7f44f40c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:17.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.790+0000 7f44facc2640 1 -- 172.21.15.161:0/1928234917 shutdown_connections 2024-03-28T23:31:17.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.790+0000 7f44facc2640 1 -- 172.21.15.161:0/1928234917 wait complete. 2024-03-28T23:31:17.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.791+0000 7f44facc2640 1 Processor -- start 2024-03-28T23:31:17.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.791+0000 7f44facc2640 1 -- start start 2024-03-28T23:31:17.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.791+0000 7f44facc2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44f40a98c0 0x7f44f413ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:17.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.792+0000 7f44facc2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44f40c1dc0 0x7f44f4154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:17.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.792+0000 7f44facc2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44f414eca0 0x7f44f4155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:17.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.792+0000 7f44facc2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f44f41514b0 con 0x7f44f414eca0 2024-03-28T23:31:17.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.792+0000 7f44facc2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f44f414f080 con 0x7f44f40a98c0 2024-03-28T23:31:17.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.792+0000 7f44facc2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f44f4151630 con 0x7f44f40c1dc0 2024-03-28T23:31:17.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.792+0000 7f44f925f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44f414eca0 0x7f44f4155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:17.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.792+0000 7f44f8a5e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44f40a98c0 0x7f44f413ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:17.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.792+0000 7f44f8a5e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44f40a98c0 0x7f44f413ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:52936/0 (socket says 172.21.15.161:52936) 2024-03-28T23:31:17.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.792+0000 7f44f8a5e640 1 -- 172.21.15.161:0/416324538 learned_addr learned my addr 172.21.15.161:0/416324538 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:17.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.792+0000 7f44f3fff640 1 --2- 172.21.15.161:0/416324538 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44f40c1dc0 0x7f44f4154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:17.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.792+0000 7f44f8a5e640 1 -- 172.21.15.161:0/416324538 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44f40c1dc0 msgr2=0x7f44f4154ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:17.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.792+0000 7f44f8a5e640 1 --2- 172.21.15.161:0/416324538 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44f40c1dc0 0x7f44f4154ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:17.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.792+0000 7f44f8a5e640 1 -- 172.21.15.161:0/416324538 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44f414eca0 msgr2=0x7f44f4155000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:17.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.792+0000 7f44f8a5e640 1 --2- 172.21.15.161:0/416324538 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44f414eca0 0x7f44f4155000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:17.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.792+0000 7f44f8a5e640 1 -- 172.21.15.161:0/416324538 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f40bf140 con 0x7f44f40a98c0 2024-03-28T23:31:17.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.792+0000 7f44f8a5e640 1 --2- 172.21.15.161:0/416324538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44f40a98c0 0x7f44f413ea50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f44e8009720 tx=0x7f44e80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:17.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.792+0000 7f44f17fa640 1 -- 172.21.15.161:0/416324538 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e8006e10 con 0x7f44f40a98c0 2024-03-28T23:31:17.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.792+0000 7f44f17fa640 1 -- 172.21.15.161:0/416324538 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44e803ab50 con 0x7f44f40a98c0 2024-03-28T23:31:17.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.792+0000 7f44facc2640 1 -- 172.21.15.161:0/416324538 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44f40bb790 con 0x7f44f40a98c0 2024-03-28T23:31:17.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.792+0000 7f44f17fa640 1 -- 172.21.15.161:0/416324538 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e8004020 con 0x7f44f40a98c0 2024-03-28T23:31:17.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.792+0000 7f44facc2640 1 -- 172.21.15.161:0/416324538 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44f40c0ef0 con 0x7f44f40a98c0 2024-03-28T23:31:17.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.793+0000 7f44facc2640 1 -- 172.21.15.161:0/416324538 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44f404f4b0 con 0x7f44f40a98c0 2024-03-28T23:31:17.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.794+0000 7f44f17fa640 1 -- 172.21.15.161:0/416324538 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f44e804b060 con 0x7f44f40a98c0 2024-03-28T23:31:17.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.794+0000 7f44f17fa640 1 --2- 172.21.15.161:0/416324538 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f44d806b4d0 0x7f44d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:17.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.794+0000 7f44f3fff640 1 --2- 172.21.15.161:0/416324538 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f44d806b4d0 0x7f44d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:17.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.794+0000 7f44f17fa640 1 -- 172.21.15.161:0/416324538 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f44e80abe90 con 0x7f44f40a98c0 2024-03-28T23:31:17.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.795+0000 7f44f3fff640 1 --2- 172.21.15.161:0/416324538 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f44d806b4d0 0x7f44d806d990 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f44f4155b60 tx=0x7f44e0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:17.804 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:17.803+0000 7f44f17fa640 1 -- 172.21.15.161:0/416324538 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f44e803f070 con 0x7f44f40a98c0 2024-03-28T23:31:18.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.012+0000 7f44facc2640 1 -- 172.21.15.161:0/416324538 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44f40748b0 con 0x7f44f40a98c0 2024-03-28T23:31:18.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.012+0000 7f44f17fa640 1 -- 172.21.15.161:0/416324538 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5182 (secure 0 0 0) 0x7f44e80ad070 con 0x7f44f40a98c0 2024-03-28T23:31:18.013 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:18.013 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5110,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/465771987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":465771987},{"type":"v1","addr":"172.21.15.170:6835","nonce":465771987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5355,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/306644174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":306644174},{"type":"v1","addr":"172.21.15.161:6836","nonce":306644174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1649814516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":1649814516},{"type":"v1","addr":"172.21.15.161:6837","nonce":1649814516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:30:57.616423+0000","modified":"2024-03-28T23:31:13.644123+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4853},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4853":{"gid":4853,"name":"d","rank":0,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.170:6834/11130685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":11130685},{"type":"v1","addr":"172.21.15.170:6834","nonce":11130685}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4853,"qdb_cluster":[4853]},"id":2}]} 2024-03-28T23:31:18.013 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 23 2024-03-28T23:31:18.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.014+0000 7f44facc2640 1 -- 172.21.15.161:0/416324538 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f44d806b4d0 msgr2=0x7f44d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:18.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.014+0000 7f44facc2640 1 --2- 172.21.15.161:0/416324538 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f44d806b4d0 0x7f44d806d990 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f44f4155b60 tx=0x7f44e0009040 comp rx=0 tx=0).stop 2024-03-28T23:31:18.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.014+0000 7f44facc2640 1 -- 172.21.15.161:0/416324538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44f40a98c0 msgr2=0x7f44f413ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:18.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.014+0000 7f44facc2640 1 --2- 172.21.15.161:0/416324538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44f40a98c0 0x7f44f413ea50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f44e8009720 tx=0x7f44e80046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.014+0000 7f44facc2640 1 -- 172.21.15.161:0/416324538 shutdown_connections 2024-03-28T23:31:18.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.014+0000 7f44facc2640 1 --2- 172.21.15.161:0/416324538 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f44d806b4d0 0x7f44d806d990 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.014+0000 7f44facc2640 1 --2- 172.21.15.161:0/416324538 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44f414eca0 0x7f44f4155000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.014+0000 7f44facc2640 1 --2- 172.21.15.161:0/416324538 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44f40c1dc0 0x7f44f4154ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.014+0000 7f44facc2640 1 --2- 172.21.15.161:0/416324538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44f40a98c0 0x7f44f413ea50 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.014+0000 7f44facc2640 1 -- 172.21.15.161:0/416324538 >> 172.21.15.161:0/416324538 conn(0x7f44f40c6670 msgr2=0x7f44f4071f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:18.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.014+0000 7f44facc2640 1 -- 172.21.15.161:0/416324538 shutdown_connections 2024-03-28T23:31:18.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.015+0000 7f44facc2640 1 -- 172.21.15.161:0/416324538 wait complete. 2024-03-28T23:31:18.429 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:30:57.616423+0000', 'modified': '2024-03-28T23:31:13.644123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 39, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4853}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4853': {'gid': 4853, 'name': 'd', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.170:6834/11130685', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 11130685}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 11130685}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4853, 'qdb_cluster': [4853]} 2024-03-28T23:31:18.429 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:31:18.430 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-28T23:31:18.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.533+0000 7f4306840640 1 Processor -- start 2024-03-28T23:31:18.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.533+0000 7f4306840640 1 -- start start 2024-03-28T23:31:18.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.534+0000 7f4306840640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f430014eca0 0x7f43000c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.534+0000 7f4306840640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f43000c1dc0 0x7f430006ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.534+0000 7f4306840640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4300151850 con 0x7f430006cf60 2024-03-28T23:31:18.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.534+0000 7f4306840640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f4300152460 con 0x7f43000c1dc0 2024-03-28T23:31:18.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.534+0000 7f4306840640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f4300151b30 con 0x7f430014eca0 2024-03-28T23:31:18.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.534+0000 7f42ff7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f43000c1dc0 0x7f430006ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.534+0000 7f42ff7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f43000c1dc0 0x7f430006ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51010/0 (socket says 172.21.15.161:51010) 2024-03-28T23:31:18.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.534+0000 7f42ff7fe640 1 -- 172.21.15.161:0/1284364725 learned_addr learned my addr 172.21.15.161:0/1284364725 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:18.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.534+0000 7f42fffff640 1 --2- 172.21.15.161:0/1284364725 >> v2:172.21.15.170:3300/0 conn(0x7f430014eca0 0x7f43000c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.534+0000 7f42ff7fe640 1 -- 172.21.15.161:0/1284364725 >> v2:172.21.15.170:3300/0 conn(0x7f430014eca0 msgr2=0x7f43000c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:18.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.534+0000 7f42ff7fe640 1 --2- 172.21.15.161:0/1284364725 >> v2:172.21.15.170:3300/0 conn(0x7f430014eca0 0x7f43000c1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.534+0000 7f42ff7fe640 1 -- 172.21.15.161:0/1284364725 >> v1:172.21.15.170:6789/0 conn(0x7f430006cf60 legacy=0x7f43000a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:18.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.534+0000 7f42ff7fe640 1 -- 172.21.15.161:0/1284364725 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4300151e10 con 0x7f43000c1dc0 2024-03-28T23:31:18.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.535+0000 7f42ff7fe640 1 --2- 172.21.15.161:0/1284364725 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43000c1dc0 0x7f430006ca20 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f43000c75a0 tx=0x7f42f402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e0d0a4ddc0d5236 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:18.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.535+0000 7f42feffd640 1 -- 172.21.15.161:0/1284364725 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f403b070 con 0x7f43000c1dc0 2024-03-28T23:31:18.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.535+0000 7f42feffd640 1 -- 172.21.15.161:0/1284364725 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42f4036440 con 0x7f43000c1dc0 2024-03-28T23:31:18.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.535+0000 7f42feffd640 1 -- 172.21.15.161:0/1284364725 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f4036810 con 0x7f43000c1dc0 2024-03-28T23:31:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.536+0000 7f4306840640 1 -- 172.21.15.161:0/1284364725 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43000c1dc0 msgr2=0x7f430006ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.536+0000 7f4306840640 1 --2- 172.21.15.161:0/1284364725 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43000c1dc0 0x7f430006ca20 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f43000c75a0 tx=0x7f42f402e0e0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.536+0000 7f4306840640 1 -- 172.21.15.161:0/1284364725 shutdown_connections 2024-03-28T23:31:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.536+0000 7f4306840640 1 --2- 172.21.15.161:0/1284364725 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43000c1dc0 0x7f430006ca20 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.536+0000 7f4306840640 1 --2- 172.21.15.161:0/1284364725 >> v2:172.21.15.170:3300/0 conn(0x7f430014eca0 0x7f43000c1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.536+0000 7f4306840640 1 -- 172.21.15.161:0/1284364725 >> 172.21.15.161:0/1284364725 conn(0x7f43000c6670 msgr2=0x7f43000c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.536+0000 7f4306840640 1 -- 172.21.15.161:0/1284364725 shutdown_connections 2024-03-28T23:31:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.536+0000 7f4306840640 1 -- 172.21.15.161:0/1284364725 wait complete. 2024-03-28T23:31:18.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.537+0000 7f4306840640 1 Processor -- start 2024-03-28T23:31:18.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.537+0000 7f4306840640 1 -- start start 2024-03-28T23:31:18.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.538+0000 7f4306840640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f430006cf60 0x7f43000c1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.538+0000 7f4306840640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43000c1dc0 0x7f43001528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.538+0000 7f4306840640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f430014eca0 0x7f4300145870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.538+0000 7f4306840640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f43001514b0 con 0x7f43000c1dc0 2024-03-28T23:31:18.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.538+0000 7f4306840640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f430014f080 con 0x7f430006cf60 2024-03-28T23:31:18.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.538+0000 7f4306840640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4300151630 con 0x7f430014eca0 2024-03-28T23:31:18.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.538+0000 7f42ff7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43000c1dc0 0x7f43001528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.538+0000 7f42ff7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43000c1dc0 0x7f43001528f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51022/0 (socket says 172.21.15.161:51022) 2024-03-28T23:31:18.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.538+0000 7f42ff7fe640 1 -- 172.21.15.161:0/1155135124 learned_addr learned my addr 172.21.15.161:0/1155135124 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:18.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.538+0000 7f42fffff640 1 --2- 172.21.15.161:0/1155135124 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f430006cf60 0x7f43000c1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.538+0000 7f4304ddd640 1 --2- 172.21.15.161:0/1155135124 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f430014eca0 0x7f4300145870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.538+0000 7f42ff7fe640 1 -- 172.21.15.161:0/1155135124 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f430014eca0 msgr2=0x7f4300145870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:18.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.538+0000 7f42ff7fe640 1 --2- 172.21.15.161:0/1155135124 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f430014eca0 0x7f4300145870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.538+0000 7f42ff7fe640 1 -- 172.21.15.161:0/1155135124 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f430006cf60 msgr2=0x7f43000c1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:18.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.538+0000 7f42ff7fe640 1 --2- 172.21.15.161:0/1155135124 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f430006cf60 0x7f43000c1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.538+0000 7f42ff7fe640 1 -- 172.21.15.161:0/1155135124 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43001e64b0 con 0x7f43000c1dc0 2024-03-28T23:31:18.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.538+0000 7f42ff7fe640 1 --2- 172.21.15.161:0/1155135124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43000c1dc0 0x7f43001528f0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f42f4000c00 tx=0x7f42f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:18.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.538+0000 7f42fcff9640 1 -- 172.21.15.161:0/1155135124 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f403b070 con 0x7f43000c1dc0 2024-03-28T23:31:18.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.538+0000 7f42fcff9640 1 -- 172.21.15.161:0/1155135124 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42f403d050 con 0x7f43000c1dc0 2024-03-28T23:31:18.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.538+0000 7f4306840640 1 -- 172.21.15.161:0/1155135124 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43001e6770 con 0x7f43000c1dc0 2024-03-28T23:31:18.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.538+0000 7f42fcff9640 1 -- 172.21.15.161:0/1155135124 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f4034a90 con 0x7f43000c1dc0 2024-03-28T23:31:18.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.539+0000 7f4306840640 1 -- 172.21.15.161:0/1155135124 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43001e6d30 con 0x7f43000c1dc0 2024-03-28T23:31:18.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.540+0000 7f42fcff9640 1 -- 172.21.15.161:0/1155135124 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f42f4034c30 con 0x7f43000c1dc0 2024-03-28T23:31:18.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.540+0000 7f42fcff9640 1 --2- 172.21.15.161:0/1155135124 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f42ec06b4d0 0x7f42ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.540+0000 7f42fcff9640 1 -- 172.21.15.161:0/1155135124 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f42f40ac150 con 0x7f43000c1dc0 2024-03-28T23:31:18.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.540+0000 7f42fffff640 1 --2- 172.21.15.161:0/1155135124 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f42ec06b4d0 0x7f42ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.541+0000 7f42fffff640 1 --2- 172.21.15.161:0/1155135124 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f42ec06b4d0 0x7f42ec06d990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f42f0009aa0 tx=0x7f42f0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:18.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.541+0000 7f4309424740 1 Processor -- start 2024-03-28T23:31:18.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.541+0000 7f4309424740 1 -- start start 2024-03-28T23:31:18.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.542+0000 7f4309424740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a9e3be5db0 0x55a9e3be3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.542+0000 7f4309424740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3be6190 0x55a9e3be3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.542+0000 7f4309424740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a9e3be4310 0x55a9e3be04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.542+0000 7f4309424740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55a9e3b49af0 con 0x55a9e3be6190 2024-03-28T23:31:18.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.542+0000 7f4309424740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55a9e3af29d0 con 0x55a9e3be5db0 2024-03-28T23:31:18.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.542+0000 7f4309424740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55a9e3b45b90 con 0x55a9e3be4310 2024-03-28T23:31:18.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.542+0000 7f42ff7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3be6190 0x55a9e3be3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.542+0000 7f42ff7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3be6190 0x55a9e3be3dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51034/0 (socket says 172.21.15.161:51034) 2024-03-28T23:31:18.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.542+0000 7f42ff7fe640 1 -- 172.21.15.161:0/2131510482 learned_addr learned my addr 172.21.15.161:0/2131510482 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:18.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.542+0000 7f42fffff640 1 --2- 172.21.15.161:0/2131510482 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a9e3be5db0 0x55a9e3be3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.542+0000 7f4304ddd640 1 --2- 172.21.15.161:0/2131510482 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a9e3be4310 0x55a9e3be04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.542+0000 7f42ff7fe640 1 -- 172.21.15.161:0/2131510482 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a9e3be4310 msgr2=0x55a9e3be04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:18.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.542+0000 7f42ff7fe640 1 --2- 172.21.15.161:0/2131510482 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a9e3be4310 0x55a9e3be04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.542+0000 7f42ff7fe640 1 -- 172.21.15.161:0/2131510482 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a9e3be5db0 msgr2=0x55a9e3be3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:18.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.542+0000 7f42ff7fe640 1 --2- 172.21.15.161:0/2131510482 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a9e3be5db0 0x55a9e3be3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.542+0000 7f42ff7fe640 1 -- 172.21.15.161:0/2131510482 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9e385e2d0 con 0x55a9e3be6190 2024-03-28T23:31:18.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.542+0000 7f4304ddd640 1 --2- 172.21.15.161:0/2131510482 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a9e3be4310 0x55a9e3be04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:18.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.542+0000 7f42ff7fe640 1 --2- 172.21.15.161:0/2131510482 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3be6190 0x55a9e3be3dd0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f42f40aa4a0 tx=0x7f42f4036f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a0833de38eb3cc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:18.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.543+0000 7f42e5ffb640 1 -- 172.21.15.161:0/2131510482 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f407e8d0 con 0x55a9e3be6190 2024-03-28T23:31:18.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.543+0000 7f42e5ffb640 1 -- 172.21.15.161:0/2131510482 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42f4082e90 con 0x55a9e3be6190 2024-03-28T23:31:18.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.543+0000 7f42e5ffb640 1 -- 172.21.15.161:0/2131510482 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f4076ea0 con 0x55a9e3be6190 2024-03-28T23:31:18.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.543+0000 7f4309424740 1 -- 172.21.15.161:0/2131510482 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3be6190 msgr2=0x55a9e3be3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:18.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.543+0000 7f4309424740 1 --2- 172.21.15.161:0/2131510482 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3be6190 0x55a9e3be3dd0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f42f40aa4a0 tx=0x7f42f4036f90 comp rx=0 tx=0).stop 2024-03-28T23:31:18.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.543+0000 7f4309424740 1 -- 172.21.15.161:0/2131510482 shutdown_connections 2024-03-28T23:31:18.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.543+0000 7f4309424740 1 --2- 172.21.15.161:0/2131510482 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3be6190 0x55a9e3be3dd0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.543+0000 7f4309424740 1 --2- 172.21.15.161:0/2131510482 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a9e3be4310 0x55a9e3be04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.543+0000 7f4309424740 1 --2- 172.21.15.161:0/2131510482 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a9e3be5db0 0x55a9e3be3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.543+0000 7f4309424740 1 -- 172.21.15.161:0/2131510482 >> 172.21.15.161:0/2131510482 conn(0x55a9e3be2630 msgr2=0x55a9e3c07220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:18.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.543+0000 7f4309424740 1 -- 172.21.15.161:0/2131510482 shutdown_connections 2024-03-28T23:31:18.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.543+0000 7f4309424740 1 -- 172.21.15.161:0/2131510482 wait complete. 2024-03-28T23:31:18.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.544+0000 7f4309424740 1 -- start start 2024-03-28T23:31:18.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.544+0000 7f4309424740 1 Processor -- start 2024-03-28T23:31:18.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.545+0000 7f4309424740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3be5db0 0x55a9e3c196c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.545+0000 7f4309424740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a9e3be6190 0x55a9e3c062b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.545+0000 7f4309424740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a9e3c19c00 0x55a9e3c03d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.545+0000 7f4309424740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55a9e3b49af0 con 0x55a9e3be5db0 2024-03-28T23:31:18.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.545+0000 7f4309424740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55a9e3b05f70 con 0x55a9e3c19c00 2024-03-28T23:31:18.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.545+0000 7f4309424740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55a9e3c08090 con 0x55a9e3be6190 2024-03-28T23:31:18.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.545+0000 7f42fffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3be5db0 0x55a9e3c196c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.545+0000 7f42fffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3be5db0 0x55a9e3c196c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51042/0 (socket says 172.21.15.161:51042) 2024-03-28T23:31:18.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.545+0000 7f42fffff640 1 -- 172.21.15.161:0/482031308 learned_addr learned my addr 172.21.15.161:0/482031308 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:18.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.545+0000 7f4309424740 10 client.0 _ll_register_callbacks cb 0x55a9e385dac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:31:18.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.545+0000 7f4304ddd640 1 --2- 172.21.15.161:0/482031308 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a9e3c19c00 0x55a9e3c03d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.545+0000 7f42ff7fe640 1 --2- 172.21.15.161:0/482031308 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a9e3be6190 0x55a9e3c062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.545+0000 7f42fffff640 1 -- 172.21.15.161:0/482031308 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a9e3be6190 msgr2=0x55a9e3c062b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:18.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.545+0000 7f42fffff640 1 --2- 172.21.15.161:0/482031308 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a9e3be6190 0x55a9e3c062b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.545+0000 7f42fffff640 1 -- 172.21.15.161:0/482031308 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a9e3c19c00 msgr2=0x55a9e3c03d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:18.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.545+0000 7f42fffff640 1 --2- 172.21.15.161:0/482031308 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a9e3c19c00 0x55a9e3c03d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.545+0000 7f42fffff640 1 -- 172.21.15.161:0/482031308 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9e39286b0 con 0x55a9e3be5db0 2024-03-28T23:31:18.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.545+0000 7f42fffff640 1 --2- 172.21.15.161:0/482031308 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3be5db0 0x55a9e3c196c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f42f0003640 tx=0x7f42f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0566d3cb2e551a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:18.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.545+0000 7f42deffd640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:31:18.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.545+0000 7f42deffd640 1 -- 172.21.15.161:0/482031308 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f0027070 con 0x55a9e3be5db0 2024-03-28T23:31:18.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.545+0000 7f42deffd640 1 -- 172.21.15.161:0/482031308 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f42f0014440 con 0x55a9e3be5db0 2024-03-28T23:31:18.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.545+0000 7f4309424740 1 -- 172.21.15.161:0/482031308 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42f4082e90 con 0x55a9e3be5db0 2024-03-28T23:31:18.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.545+0000 7f42deffd640 1 -- 172.21.15.161:0/482031308 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f42f0014640 con 0x55a9e3be5db0 2024-03-28T23:31:18.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.546+0000 7f4309424740 10 client.5664 fetch_fsmap learned FSMap version 23 2024-03-28T23:31:18.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.546+0000 7f4309424740 1 -- 172.21.15.161:0/482031308 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x55a9e3c32e90 con 0x55a9e3be5db0 2024-03-28T23:31:18.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.546+0000 7f42deffd640 1 -- 172.21.15.161:0/482031308 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42f4082e90 con 0x55a9e3be5db0 2024-03-28T23:31:18.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.546+0000 7f42deffd640 1 -- 172.21.15.161:0/482031308 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 23) ==== 3494+0+0 (secure 0 0 0) 0x7f42f0035080 con 0x55a9e3be5db0 2024-03-28T23:31:18.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.546+0000 7f4309424740 10 client.5664 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T23:31:18.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.546+0000 7f4309424740 10 client.5664 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-28T23:31:18.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.546+0000 7f4309424740 20 client.5664 populate_metadata read hostname 'smithi161' 2024-03-28T23:31:18.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.546+0000 7f4309424740 1 --2- 172.21.15.161:0/482031308 >> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] conn(0x55a9e3c35b20 0x55a9e3c35f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.546+0000 7f4309424740 4 client.5664 mds_command: new command op to 4853 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:31:18.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.546+0000 7f4309424740 1 -- 172.21.15.161:0/482031308 --> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9e3b57030 con 0x55a9e3c35b20 2024-03-28T23:31:18.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.546+0000 7f42ff7fe640 1 --2- 172.21.15.161:0/482031308 >> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] conn(0x55a9e3c35b20 0x55a9e3c35f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.546+0000 7f42deffd640 10 client.5664 ms_handle_connect on v2:172.21.15.170:6832/11130685 2024-03-28T23:31:18.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.546+0000 7f42ff7fe640 1 --2- 172.21.15.161:0/482031308 >> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] conn(0x55a9e3c35b20 0x55a9e3c35f00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b8f51feee64e749 server_cookie=4e19708217ee6249 in_seq=0 out_seq=0 2024-03-28T23:31:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.547+0000 7f42deffd640 1 -- 172.21.15.161:0/482031308 <== mds.0 v2:172.21.15.170:6832/11130685 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a9e3b57030 con 0x55a9e3c35b20 2024-03-28T23:31:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.547+0000 7f42deffd640 10 client.5664 handle_command_reply: tid=0 2024-03-28T23:31:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.547+0000 7f4309424740 1 client.5664 shutdown 2024-03-28T23:31:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.547+0000 7f4309424740 1 -- 172.21.15.161:0/482031308 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3be5db0 msgr2=0x55a9e3c196c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.547+0000 7f4309424740 1 --2- 172.21.15.161:0/482031308 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3be5db0 0x55a9e3c196c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f42f0003640 tx=0x7f42f00046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.547+0000 7f4309424740 1 -- 172.21.15.161:0/482031308 shutdown_connections 2024-03-28T23:31:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.547+0000 7f4309424740 1 --2- 172.21.15.161:0/482031308 >> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] conn(0x55a9e3c35b20 0x55a9e3c35f00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.547+0000 7f4309424740 1 --2- 172.21.15.161:0/482031308 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a9e3c19c00 0x55a9e3c03d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.547+0000 7f4309424740 1 --2- 172.21.15.161:0/482031308 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a9e3be6190 0x55a9e3c062b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.547+0000 7f42deffd640 0 client.5664 ms_handle_reset on v2:172.21.15.170:6832/11130685 2024-03-28T23:31:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.547+0000 7f4309424740 1 --2- 172.21.15.161:0/482031308 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3be5db0 0x55a9e3c196c0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.547+0000 7f4309424740 1 -- 172.21.15.161:0/482031308 >> 172.21.15.161:0/482031308 conn(0x55a9e3be4310 msgr2=0x55a9e3bdf2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.547+0000 7f4309424740 1 -- 172.21.15.161:0/482031308 shutdown_connections 2024-03-28T23:31:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.547+0000 7f4309424740 1 -- 172.21.15.161:0/482031308 wait complete. 2024-03-28T23:31:18.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.547+0000 7f4309424740 20 client.5664 trim_cache size 0 max 16384 2024-03-28T23:31:18.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.565+0000 7f4309424740 1 Processor -- start 2024-03-28T23:31:18.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.566+0000 7f4309424740 1 -- start start 2024-03-28T23:31:18.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.566+0000 7f4309424740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a9e3be5db0 0x55a9e3bdf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.566+0000 7f4309424740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3be6190 0x55a9e3c191d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.566+0000 7f4309424740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a9e3c19c00 0x55a9e3c03800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.566+0000 7f4309424740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f43001f2d40 con 0x55a9e3be6190 2024-03-28T23:31:18.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.566+0000 7f4309424740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f43001f20a0 con 0x55a9e3be5db0 2024-03-28T23:31:18.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.566+0000 7f4309424740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f43001f26f0 con 0x55a9e3c19c00 2024-03-28T23:31:18.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.566+0000 7f42ff7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3be6190 0x55a9e3c191d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.566+0000 7f42ff7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3be6190 0x55a9e3c191d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51050/0 (socket says 172.21.15.161:51050) 2024-03-28T23:31:18.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.566+0000 7f42ff7fe640 1 -- 172.21.15.161:0/242088070 learned_addr learned my addr 172.21.15.161:0/242088070 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:18.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.566+0000 7f42fffff640 1 --2- 172.21.15.161:0/242088070 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a9e3be5db0 0x55a9e3bdf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.566+0000 7f4304ddd640 1 --2- 172.21.15.161:0/242088070 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a9e3c19c00 0x55a9e3c03800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.567+0000 7f42ff7fe640 1 -- 172.21.15.161:0/242088070 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a9e3c19c00 msgr2=0x55a9e3c03800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:18.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.567+0000 7f42ff7fe640 1 --2- 172.21.15.161:0/242088070 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a9e3c19c00 0x55a9e3c03800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.567+0000 7f42ff7fe640 1 -- 172.21.15.161:0/242088070 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a9e3be5db0 msgr2=0x55a9e3bdf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:18.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.567+0000 7f42ff7fe640 1 --2- 172.21.15.161:0/242088070 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a9e3be5db0 0x55a9e3bdf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.567+0000 7f42ff7fe640 1 -- 172.21.15.161:0/242088070 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9e3c1ecd0 con 0x55a9e3be6190 2024-03-28T23:31:18.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.567+0000 7f42ff7fe640 1 --2- 172.21.15.161:0/242088070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3be6190 0x55a9e3c191d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f42f407a7c0 tx=0x7f42f407c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=816374a84b7400d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:18.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.567+0000 7f42e57fa640 1 -- 172.21.15.161:0/242088070 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f4078b30 con 0x55a9e3be6190 2024-03-28T23:31:18.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.567+0000 7f42e57fa640 1 -- 172.21.15.161:0/242088070 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42f4092170 con 0x55a9e3be6190 2024-03-28T23:31:18.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.567+0000 7f42e57fa640 1 -- 172.21.15.161:0/242088070 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f407aa90 con 0x55a9e3be6190 2024-03-28T23:31:18.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.567+0000 7f4309424740 1 -- 172.21.15.161:0/242088070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3be6190 msgr2=0x55a9e3c191d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:18.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.567+0000 7f4309424740 1 --2- 172.21.15.161:0/242088070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3be6190 0x55a9e3c191d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f42f407a7c0 tx=0x7f42f407c000 comp rx=0 tx=0).stop 2024-03-28T23:31:18.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.567+0000 7f4309424740 1 -- 172.21.15.161:0/242088070 shutdown_connections 2024-03-28T23:31:18.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.567+0000 7f4309424740 1 --2- 172.21.15.161:0/242088070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3be6190 0x55a9e3c191d0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.567+0000 7f4309424740 1 --2- 172.21.15.161:0/242088070 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a9e3c19c00 0x55a9e3c03800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.567+0000 7f4309424740 1 --2- 172.21.15.161:0/242088070 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a9e3be5db0 0x55a9e3bdf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.567+0000 7f4309424740 1 -- 172.21.15.161:0/242088070 >> 172.21.15.161:0/242088070 conn(0x55a9e3be4310 msgr2=0x55a9e3c24290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.567+0000 7f4309424740 1 -- 172.21.15.161:0/242088070 shutdown_connections 2024-03-28T23:31:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.567+0000 7f4309424740 1 -- 172.21.15.161:0/242088070 wait complete. 2024-03-28T23:31:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.568+0000 7f4309424740 1 -- start start 2024-03-28T23:31:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.568+0000 7f4309424740 1 Processor -- start 2024-03-28T23:31:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.569+0000 7f4309424740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a9e3be5db0 0x55a9e3be0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.569+0000 7f4309424740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a9e3be6190 0x55a9e3c03c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.569+0000 7f4309424740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3c19c00 0x55a9e3c178f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.569+0000 7f4309424740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55a9e3c0b3a0 con 0x55a9e3c19c00 2024-03-28T23:31:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.569+0000 7f4309424740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55a9e3c359a0 con 0x55a9e3be5db0 2024-03-28T23:31:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.569+0000 7f4309424740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55a9e3c069e0 con 0x55a9e3be6190 2024-03-28T23:31:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.569+0000 7f4309424740 10 client.0 _ll_register_callbacks cb 0x7f42f0004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:31:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.569+0000 7f4304ddd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3c19c00 0x55a9e3c178f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.569+0000 7f4304ddd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3c19c00 0x55a9e3c178f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51054/0 (socket says 172.21.15.161:51054) 2024-03-28T23:31:18.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.569+0000 7f4304ddd640 1 -- 172.21.15.161:0/3437064021 learned_addr learned my addr 172.21.15.161:0/3437064021 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:18.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.569+0000 7f42ff7fe640 1 --2- 172.21.15.161:0/3437064021 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a9e3be6190 0x55a9e3c03c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.569+0000 7f42fffff640 1 --2- 172.21.15.161:0/3437064021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a9e3be5db0 0x55a9e3be0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.569+0000 7f4304ddd640 1 -- 172.21.15.161:0/3437064021 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a9e3be6190 msgr2=0x55a9e3c03c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:18.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.569+0000 7f4304ddd640 1 --2- 172.21.15.161:0/3437064021 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a9e3be6190 0x55a9e3c03c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.569+0000 7f4304ddd640 1 -- 172.21.15.161:0/3437064021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a9e3be5db0 msgr2=0x55a9e3be0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:18.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.569+0000 7f4304ddd640 1 --2- 172.21.15.161:0/3437064021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a9e3be5db0 0x55a9e3be0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.569+0000 7f4304ddd640 1 -- 172.21.15.161:0/3437064021 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9e3ca03e0 con 0x55a9e3c19c00 2024-03-28T23:31:18.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.569+0000 7f4304ddd640 1 --2- 172.21.15.161:0/3437064021 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3c19c00 0x55a9e3c178f0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f42f80012e0 tx=0x7f42f8016a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9f59c1ccfe577db server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:18.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.569+0000 7f42deffd640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:31:18.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.570+0000 7f42deffd640 1 -- 172.21.15.161:0/3437064021 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f8003df0 con 0x55a9e3c19c00 2024-03-28T23:31:18.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.570+0000 7f42deffd640 1 -- 172.21.15.161:0/3437064021 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f42f8005070 con 0x55a9e3c19c00 2024-03-28T23:31:18.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.570+0000 7f4309424740 1 -- 172.21.15.161:0/3437064021 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42f4092170 con 0x55a9e3c19c00 2024-03-28T23:31:18.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.570+0000 7f42deffd640 1 -- 172.21.15.161:0/3437064021 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f42f8027050 con 0x55a9e3c19c00 2024-03-28T23:31:18.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.570+0000 7f4309424740 10 client.5670 fetch_fsmap learned FSMap version 23 2024-03-28T23:31:18.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.570+0000 7f4309424740 1 -- 172.21.15.161:0/3437064021 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x55a9e3ca08b0 con 0x55a9e3c19c00 2024-03-28T23:31:18.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.570+0000 7f42deffd640 1 -- 172.21.15.161:0/3437064021 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42f4092170 con 0x55a9e3c19c00 2024-03-28T23:31:18.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.570+0000 7f42deffd640 1 -- 172.21.15.161:0/3437064021 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 23) ==== 3494+0+0 (secure 0 0 0) 0x7f42f802a020 con 0x55a9e3c19c00 2024-03-28T23:31:18.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.570+0000 7f4309424740 10 client.5670 fetch_fsmap finished waiting for FSMap version 23 2024-03-28T23:31:18.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.570+0000 7f4309424740 10 client.5670 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-28T23:31:18.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.570+0000 7f4309424740 20 client.5670 populate_metadata read hostname 'smithi161' 2024-03-28T23:31:18.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.570+0000 7f4309424740 1 --2- 172.21.15.161:0/3437064021 >> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] conn(0x55a9e3c9b490 0x55a9e3c9b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.570+0000 7f4309424740 4 client.5670 mds_command: new command op to 4853 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:31:18.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.570+0000 7f4309424740 1 -- 172.21.15.161:0/3437064021 --> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] -- command(tid 0: {"prefix": "status"}) -- 0x55a9e3b51380 con 0x55a9e3c9b490 2024-03-28T23:31:18.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.570+0000 7f42fffff640 1 --2- 172.21.15.161:0/3437064021 >> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] conn(0x55a9e3c9b490 0x55a9e3c9b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.570+0000 7f42fffff640 1 --2- 172.21.15.161:0/3437064021 >> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] conn(0x55a9e3c9b490 0x55a9e3c9b870 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99664803ae6d87c server_cookie=b488361763106faa in_seq=0 out_seq=0 2024-03-28T23:31:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.570+0000 7f42deffd640 10 client.5670 ms_handle_connect on v2:172.21.15.170:6832/11130685 2024-03-28T23:31:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.571+0000 7f42deffd640 1 -- 172.21.15.161:0/3437064021 <== mds.0 v2:172.21.15.170:6832/11130685 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55a9e3b51380 con 0x55a9e3c9b490 2024-03-28T23:31:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.571+0000 7f42deffd640 10 client.5670 handle_command_reply: tid=0 2024-03-28T23:31:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.571+0000 7f4309424740 1 client.5670 shutdown 2024-03-28T23:31:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.571+0000 7f4309424740 1 -- 172.21.15.161:0/3437064021 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3c19c00 msgr2=0x55a9e3c178f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.571+0000 7f4309424740 1 --2- 172.21.15.161:0/3437064021 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3c19c00 0x55a9e3c178f0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f42f80012e0 tx=0x7f42f8016a80 comp rx=0 tx=0).stop 2024-03-28T23:31:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.571+0000 7f4309424740 1 -- 172.21.15.161:0/3437064021 shutdown_connections 2024-03-28T23:31:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.571+0000 7f4309424740 1 --2- 172.21.15.161:0/3437064021 >> [v2:172.21.15.170:6832/11130685,v1:172.21.15.170:6834/11130685] conn(0x55a9e3c9b490 0x55a9e3c9b870 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.571+0000 7f4309424740 1 --2- 172.21.15.161:0/3437064021 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a9e3c19c00 0x55a9e3c178f0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.571+0000 7f4309424740 1 --2- 172.21.15.161:0/3437064021 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a9e3be6190 0x55a9e3c03c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.571+0000 7f42deffd640 0 client.5670 ms_handle_reset on v2:172.21.15.170:6832/11130685 2024-03-28T23:31:18.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.571+0000 7f4309424740 1 --2- 172.21.15.161:0/3437064021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a9e3be5db0 0x55a9e3be0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.571+0000 7f4309424740 1 -- 172.21.15.161:0/3437064021 >> 172.21.15.161:0/3437064021 conn(0x55a9e3be4310 msgr2=0x55a9e3c32050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:18.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.571+0000 7f4309424740 1 -- 172.21.15.161:0/3437064021 shutdown_connections 2024-03-28T23:31:18.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.571+0000 7f4309424740 1 -- 172.21.15.161:0/3437064021 wait complete. 2024-03-28T23:31:18.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.571+0000 7f4309424740 20 client.5670 trim_cache size 0 max 16384 2024-03-28T23:31:18.583 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:31:18.583 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:31:18.583 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:31:18.583 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 4853, 2024-03-28T23:31:18.583 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:31:18.584 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:31:18.584 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:31:18.584 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 10.934418395, 2024-03-28T23:31:18.584 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 23, 2024-03-28T23:31:18.584 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 39, 2024-03-28T23:31:18.584 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 39, 2024-03-28T23:31:18.584 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 27.200020440999999 2024-03-28T23:31:18.584 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:31:18.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.571+0000 7f4306840640 1 -- 172.21.15.161:0/1155135124 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f42ec06b4d0 msgr2=0x7f42ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:18.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.571+0000 7f4306840640 1 --2- 172.21.15.161:0/1155135124 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f42ec06b4d0 0x7f42ec06d990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f42f0009aa0 tx=0x7f42f0009190 comp rx=0 tx=0).stop 2024-03-28T23:31:18.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.571+0000 7f4306840640 1 -- 172.21.15.161:0/1155135124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43000c1dc0 msgr2=0x7f43001528f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:18.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.571+0000 7f4306840640 1 --2- 172.21.15.161:0/1155135124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43000c1dc0 0x7f43001528f0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f42f4000c00 tx=0x7f42f40046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.571+0000 7f4306840640 1 -- 172.21.15.161:0/1155135124 shutdown_connections 2024-03-28T23:31:18.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.571+0000 7f4306840640 1 --2- 172.21.15.161:0/1155135124 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f42ec06b4d0 0x7f42ec06d990 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.571+0000 7f4306840640 1 --2- 172.21.15.161:0/1155135124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43000c1dc0 0x7f43001528f0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.572+0000 7f4306840640 1 --2- 172.21.15.161:0/1155135124 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f430014eca0 0x7f4300145870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.572+0000 7f4306840640 1 --2- 172.21.15.161:0/1155135124 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f430006cf60 0x7f43000c1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.572+0000 7f4306840640 1 -- 172.21.15.161:0/1155135124 >> 172.21.15.161:0/1155135124 conn(0x7f43000c6670 msgr2=0x7f43000c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:18.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.572+0000 7f4306840640 1 -- 172.21.15.161:0/1155135124 shutdown_connections 2024-03-28T23:31:18.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.572+0000 7f4306840640 1 -- 172.21.15.161:0/1155135124 wait complete. 2024-03-28T23:31:18.587 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 23, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:30:57.616423+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 23, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4853": { "addr": "172.21.15.170:6834/11130685", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 11130685, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 11130685, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4853, "incarnation": 19, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-03-28T23:31:13.644123+0000", "qdb_cluster": [ 4853 ], "qdb_leader": 4853, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4853 } } } ], "standbys": [ { "addr": "172.21.15.170:6835/465771987", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 465771987, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 465771987, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5110, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6836/306644174", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 306644174, "type": "v2" }, { "addr": "172.21.15.161:6836", "nonce": 306644174, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5355, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6837/1649814516", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6835", "nonce": 1649814516, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 1649814516, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 20, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5589, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:31:18.588 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd', 'state': 'up:active'}] 2024-03-28T23:31:18.588 INFO:tasks.cephfs.test_failover:current = [{'gid': 5589, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6837/1649814516', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 1649814516}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 1649814516}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 20}, {'gid': 5110, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/465771987', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 465771987}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 465771987}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 13}, {'gid': 5355, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6836/306644174', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 306644174}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 306644174}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 12}, {'gid': 4853, 'name': 'd', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.170:6834/11130685', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 11130685}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 11130685}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}] 2024-03-28T23:31:18.588 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-28T23:31:18.588 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-28T23:31:18.588 DEBUG:teuthology.orchestra.run.smithi161:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:31:18.635 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:31:18.679 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-28T23:31:18.679 INFO:tasks.cephfs.mount:Terminating background process 2024-03-28T23:31:18.679 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:31:18.680 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:31:18.695 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20793 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-28T23:31:18.695 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi161.front.sepia.ceph.com 2024-03-28T23:31:18.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:31:18.696 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-28T23:31:18.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:31:18.750 INFO:teuthology.orchestra.run.smithi161.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-28T23:31:18.751 INFO:tasks.cephfs.mount:Terminating background process 2024-03-28T23:31:18.751 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:31:18.751 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:31:18.765 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056374,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19690 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-28T23:31:18.766 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi170.front.sepia.ceph.com 2024-03-28T23:31:18.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:31:18.766 DEBUG:teuthology.orchestra.run.smithi170:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-28T23:31:18.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:31:18.820 INFO:teuthology.orchestra.run.smithi170.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-28T23:31:18.820 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:31:18.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.920+0000 7f120700e640 1 Processor -- start 2024-03-28T23:31:18.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.920+0000 7f120700e640 1 -- start start 2024-03-28T23:31:18.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.921+0000 7f120700e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f120014be70 0x7f12000ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.921+0000 7f120700e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12000ab6e0 0x7f12000aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.921+0000 7f120700e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f120014c250 con 0x7f120006ce80 2024-03-28T23:31:18.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.921+0000 7f120700e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f12001529c0 con 0x7f120014be70 2024-03-28T23:31:18.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.921+0000 7f120700e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f12000a9dc0 con 0x7f12000ab6e0 2024-03-28T23:31:18.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.921+0000 7f1204daa640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f120014be70 0x7f12000ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.921+0000 7f11fffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12000ab6e0 0x7f12000aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.921+0000 7f12055ab640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f120006ce80 0x7f12001553b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:51384/0 (socket says 172.21.15.161:51384) 2024-03-28T23:31:18.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.921+0000 7f12055ab640 1 -- 172.21.15.161:0/2984340488 learned_addr learned my addr 172.21.15.161:0/2984340488 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:18.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.921+0000 7f1204daa640 1 -- 172.21.15.161:0/2984340488 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12000ab6e0 msgr2=0x7f12000aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:18.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.921+0000 7f1204daa640 1 --2- 172.21.15.161:0/2984340488 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12000ab6e0 0x7f12000aa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.921+0000 7f1204daa640 1 -- 172.21.15.161:0/2984340488 >> v1:172.21.15.170:6789/0 conn(0x7f120006ce80 legacy=0x7f12001553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:18.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.921+0000 7f11ff7fe640 1 -- 172.21.15.161:0/2984340488 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 110634527 0 0) 0x7f120014c250 con 0x7f120006ce80 2024-03-28T23:31:18.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.921+0000 7f1204daa640 1 -- 172.21.15.161:0/2984340488 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12000aa0a0 con 0x7f120014be70 2024-03-28T23:31:18.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.921+0000 7f11fffff640 1 --2- 172.21.15.161:0/2984340488 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12000ab6e0 0x7f12000aa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:18.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.922+0000 7f1204daa640 1 --2- 172.21.15.161:0/2984340488 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f120014be70 0x7f12000ab1a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f11f40097a0 tx=0x7f11f402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67280415cba9e0c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:18.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.922+0000 7f11ff7fe640 1 -- 172.21.15.161:0/2984340488 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f40315a0 con 0x7f120014be70 2024-03-28T23:31:18.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.922+0000 7f11ff7fe640 1 -- 172.21.15.161:0/2984340488 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11f4037070 con 0x7f120014be70 2024-03-28T23:31:18.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.922+0000 7f11ff7fe640 1 -- 172.21.15.161:0/2984340488 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f4031960 con 0x7f120014be70 2024-03-28T23:31:18.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.923+0000 7f120700e640 1 -- 172.21.15.161:0/2984340488 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f120014be70 msgr2=0x7f12000ab1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:18.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.923+0000 7f120700e640 1 --2- 172.21.15.161:0/2984340488 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f120014be70 0x7f12000ab1a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f11f40097a0 tx=0x7f11f402b840 comp rx=0 tx=0).stop 2024-03-28T23:31:18.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.923+0000 7f120700e640 1 -- 172.21.15.161:0/2984340488 shutdown_connections 2024-03-28T23:31:18.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.923+0000 7f120700e640 1 --2- 172.21.15.161:0/2984340488 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12000ab6e0 0x7f12000aa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.923+0000 7f120700e640 1 --2- 172.21.15.161:0/2984340488 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f120014be70 0x7f12000ab1a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.923+0000 7f120700e640 1 -- 172.21.15.161:0/2984340488 >> 172.21.15.161:0/2984340488 conn(0x7f12000c6670 msgr2=0x7f12000c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:18.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.923+0000 7f120700e640 1 -- 172.21.15.161:0/2984340488 shutdown_connections 2024-03-28T23:31:18.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.923+0000 7f120700e640 1 -- 172.21.15.161:0/2984340488 wait complete. 2024-03-28T23:31:18.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.924+0000 7f120700e640 1 Processor -- start 2024-03-28T23:31:18.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.924+0000 7f120700e640 1 -- start start 2024-03-28T23:31:18.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.925+0000 7f120700e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f120006ce80 0x7f1200154990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.925+0000 7f120700e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12000ab6e0 0x7f120014f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.925+0000 7f120700e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f120014be70 0x7f120014d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.925+0000 7f120700e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f120006b2d0 con 0x7f12000ab6e0 2024-03-28T23:31:18.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.925+0000 7f120700e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f120006b150 con 0x7f120006ce80 2024-03-28T23:31:18.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.925+0000 7f120700e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f12000c1520 con 0x7f120014be70 2024-03-28T23:31:18.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.925+0000 7f11fffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12000ab6e0 0x7f120014f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.925+0000 7f1204daa640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f120006ce80 0x7f1200154990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.925+0000 7f11fffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12000ab6e0 0x7f120014f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51060/0 (socket says 172.21.15.161:51060) 2024-03-28T23:31:18.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.925+0000 7f11fffff640 1 -- 172.21.15.161:0/4048516789 learned_addr learned my addr 172.21.15.161:0/4048516789 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:18.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.925+0000 7f12055ab640 1 --2- 172.21.15.161:0/4048516789 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f120014be70 0x7f120014d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.925+0000 7f1204daa640 1 -- 172.21.15.161:0/4048516789 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f120014be70 msgr2=0x7f120014d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:18.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.925+0000 7f1204daa640 1 --2- 172.21.15.161:0/4048516789 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f120014be70 0x7f120014d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.925+0000 7f1204daa640 1 -- 172.21.15.161:0/4048516789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12000ab6e0 msgr2=0x7f120014f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:18.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.925+0000 7f1204daa640 1 --2- 172.21.15.161:0/4048516789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12000ab6e0 0x7f120014f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:18.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.925+0000 7f1204daa640 1 -- 172.21.15.161:0/4048516789 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1200155840 con 0x7f120006ce80 2024-03-28T23:31:18.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.925+0000 7f11fffff640 1 --2- 172.21.15.161:0/4048516789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12000ab6e0 0x7f120014f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:18.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.925+0000 7f1204daa640 1 --2- 172.21.15.161:0/4048516789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f120006ce80 0x7f1200154990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f11f4033000 tx=0x7f11f40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:18.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.925+0000 7f11fd7fa640 1 -- 172.21.15.161:0/4048516789 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f4004220 con 0x7f120006ce80 2024-03-28T23:31:18.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.926+0000 7f11fd7fa640 1 -- 172.21.15.161:0/4048516789 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11f40043c0 con 0x7f120006ce80 2024-03-28T23:31:18.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.926+0000 7f120700e640 1 -- 172.21.15.161:0/4048516789 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12001ebbc0 con 0x7f120006ce80 2024-03-28T23:31:18.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.926+0000 7f11fd7fa640 1 -- 172.21.15.161:0/4048516789 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f4003850 con 0x7f120006ce80 2024-03-28T23:31:18.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.926+0000 7f120700e640 1 -- 172.21.15.161:0/4048516789 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12001ec140 con 0x7f120006ce80 2024-03-28T23:31:18.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.927+0000 7f120700e640 1 -- 172.21.15.161:0/4048516789 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f120004f4b0 con 0x7f120006ce80 2024-03-28T23:31:18.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.927+0000 7f11fd7fa640 1 -- 172.21.15.161:0/4048516789 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f11f40039f0 con 0x7f120006ce80 2024-03-28T23:31:18.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.927+0000 7f11fd7fa640 1 --2- 172.21.15.161:0/4048516789 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f11e406b4d0 0x7f11e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:18.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.928+0000 7f11fd7fa640 1 -- 172.21.15.161:0/4048516789 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f11f402f030 con 0x7f120006ce80 2024-03-28T23:31:18.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.928+0000 7f11fffff640 1 --2- 172.21.15.161:0/4048516789 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f11e406b4d0 0x7f11e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:18.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.928+0000 7f11fffff640 1 --2- 172.21.15.161:0/4048516789 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f11e406b4d0 0x7f11e406d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f12000a9320 tx=0x7f11ec009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:18.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:18.936+0000 7f11fd7fa640 1 -- 172.21.15.161:0/4048516789 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f11f404b060 con 0x7f120006ce80 2024-03-28T23:31:19.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.141+0000 7f120700e640 1 -- 172.21.15.161:0/4048516789 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f12000c0ff0 con 0x7f120006ce80 2024-03-28T23:31:19.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.142+0000 7f11fd7fa640 1 -- 172.21.15.161:0/4048516789 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5182 (secure 0 0 0) 0x7f11f4041070 con 0x7f120006ce80 2024-03-28T23:31:19.143 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:19.143 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5110,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/465771987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":465771987},{"type":"v1","addr":"172.21.15.170:6835","nonce":465771987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5355,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/306644174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":306644174},{"type":"v1","addr":"172.21.15.161:6836","nonce":306644174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1649814516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":1649814516},{"type":"v1","addr":"172.21.15.161:6837","nonce":1649814516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:30:57.616423+0000","modified":"2024-03-28T23:31:13.644123+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4853},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4853":{"gid":4853,"name":"d","rank":0,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.170:6834/11130685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":11130685},{"type":"v1","addr":"172.21.15.170:6834","nonce":11130685}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4853,"qdb_cluster":[4853]},"id":2}]} 2024-03-28T23:31:19.144 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 23 2024-03-28T23:31:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.144+0000 7f120700e640 1 -- 172.21.15.161:0/4048516789 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f11e406b4d0 msgr2=0x7f11e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.144+0000 7f120700e640 1 --2- 172.21.15.161:0/4048516789 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f11e406b4d0 0x7f11e406d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f12000a9320 tx=0x7f11ec009190 comp rx=0 tx=0).stop 2024-03-28T23:31:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.144+0000 7f120700e640 1 -- 172.21.15.161:0/4048516789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f120006ce80 msgr2=0x7f1200154990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.144+0000 7f120700e640 1 --2- 172.21.15.161:0/4048516789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f120006ce80 0x7f1200154990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f11f4033000 tx=0x7f11f40046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.144+0000 7f120700e640 1 -- 172.21.15.161:0/4048516789 shutdown_connections 2024-03-28T23:31:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.144+0000 7f120700e640 1 --2- 172.21.15.161:0/4048516789 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f11e406b4d0 0x7f11e406d990 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.144+0000 7f120700e640 1 --2- 172.21.15.161:0/4048516789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12000ab6e0 0x7f120014f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.144+0000 7f120700e640 1 --2- 172.21.15.161:0/4048516789 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f120014be70 0x7f120014d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.144+0000 7f120700e640 1 --2- 172.21.15.161:0/4048516789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f120006ce80 0x7f1200154990 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.144+0000 7f120700e640 1 -- 172.21.15.161:0/4048516789 >> 172.21.15.161:0/4048516789 conn(0x7f12000c6670 msgr2=0x7f12000c20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:19.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.144+0000 7f120700e640 1 -- 172.21.15.161:0/4048516789 shutdown_connections 2024-03-28T23:31:19.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.144+0000 7f120700e640 1 -- 172.21.15.161:0/4048516789 wait complete. 2024-03-28T23:31:19.156 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:31:19.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.256+0000 7f73aa0ed640 1 Processor -- start 2024-03-28T23:31:19.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.256+0000 7f73aa0ed640 1 -- start start 2024-03-28T23:31:19.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.257+0000 7f73aa0ed640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f73a406ce80 0x7f73a40a9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:19.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.257+0000 7f73aa0ed640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73a406d310 con 0x7f73a414eca0 2024-03-28T23:31:19.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.257+0000 7f73aa0ed640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73a4151850 con 0x7f73a406ca20 2024-03-28T23:31:19.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.257+0000 7f73aa0ed640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f73a4152460 con 0x7f73a406ce80 2024-03-28T23:31:19.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.257+0000 7f73a37fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f73a414eca0 0x7f73a40c1880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48042/0 (socket says 172.21.15.161:48042) 2024-03-28T23:31:19.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.257+0000 7f73a37fe640 1 -- 172.21.15.161:0/1895186037 learned_addr learned my addr 172.21.15.161:0/1895186037 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:19.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.257+0000 7f73a3fff640 1 --2- 172.21.15.161:0/1895186037 >> v2:172.21.15.170:3300/0 conn(0x7f73a406ce80 0x7f73a40a9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:19.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.257+0000 7f73a27fc640 1 -- 172.21.15.161:0/1895186037 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1361497360 0 0) 0x7f7394001110 con 0x7f73a414eca0 2024-03-28T23:31:19.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.257+0000 7f73a27fc640 1 -- 172.21.15.161:0/1895186037 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f738c003520 con 0x7f73a414eca0 2024-03-28T23:31:19.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.257+0000 7f73a27fc640 1 -- 172.21.15.161:0/1895186037 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3724722131 0 0) 0x7f73a4151850 con 0x7f73a406ca20 2024-03-28T23:31:19.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.257+0000 7f73a27fc640 1 -- 172.21.15.161:0/1895186037 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7394001110 con 0x7f73a406ca20 2024-03-28T23:31:19.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.257+0000 7f73a3fff640 1 -- 172.21.15.161:0/1895186037 >> v1:172.21.15.170:6789/0 conn(0x7f73a406ca20 legacy=0x7f73a40a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:19.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.257+0000 7f73a3fff640 1 -- 172.21.15.161:0/1895186037 >> v1:172.21.15.161:6789/0 conn(0x7f73a414eca0 legacy=0x7f73a40c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:19.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.257+0000 7f73a3fff640 1 -- 172.21.15.161:0/1895186037 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73a4151b30 con 0x7f73a406ce80 2024-03-28T23:31:19.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.258+0000 7f73a3fff640 1 --2- 172.21.15.161:0/1895186037 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73a406ce80 0x7f73a40a9fd0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f739c009d90 tx=0x7f739c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3190d77bf22373fc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:19.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.258+0000 7f73a27fc640 1 -- 172.21.15.161:0/1895186037 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739c039070 con 0x7f73a406ce80 2024-03-28T23:31:19.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.258+0000 7f73a27fc640 1 -- 172.21.15.161:0/1895186037 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f739c02cc60 con 0x7f73a406ce80 2024-03-28T23:31:19.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.258+0000 7f73a27fc640 1 -- 172.21.15.161:0/1895186037 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739c034640 con 0x7f73a406ce80 2024-03-28T23:31:19.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.259+0000 7f73aa0ed640 1 -- 172.21.15.161:0/1895186037 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73a406ce80 msgr2=0x7f73a40a9fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:19.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.259+0000 7f73aa0ed640 1 --2- 172.21.15.161:0/1895186037 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73a406ce80 0x7f73a40a9fd0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f739c009d90 tx=0x7f739c02beb0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.259+0000 7f73aa0ed640 1 -- 172.21.15.161:0/1895186037 shutdown_connections 2024-03-28T23:31:19.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.259+0000 7f73aa0ed640 1 --2- 172.21.15.161:0/1895186037 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73a406ce80 0x7f73a40a9fd0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.259+0000 7f73aa0ed640 1 -- 172.21.15.161:0/1895186037 >> 172.21.15.161:0/1895186037 conn(0x7f73a40c6670 msgr2=0x7f73a40c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:19.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.259+0000 7f73aa0ed640 1 -- 172.21.15.161:0/1895186037 shutdown_connections 2024-03-28T23:31:19.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.259+0000 7f73aa0ed640 1 -- 172.21.15.161:0/1895186037 wait complete. 2024-03-28T23:31:19.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.260+0000 7f73aa0ed640 1 Processor -- start 2024-03-28T23:31:19.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.260+0000 7f73aa0ed640 1 -- start start 2024-03-28T23:31:19.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.261+0000 7f73aa0ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73a406ca20 0x7f73a414dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:19.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.261+0000 7f73aa0ed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73a406ce80 0x7f73a406b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:19.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.261+0000 7f73aa0ed640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73a414eca0 0x7f73a41ec840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:19.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.261+0000 7f73aa0ed640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f73a40aa510 con 0x7f73a414eca0 2024-03-28T23:31:19.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.261+0000 7f73aa0ed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f73a41520f0 con 0x7f73a406ce80 2024-03-28T23:31:19.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.261+0000 7f73aa0ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f73a414f080 con 0x7f73a406ca20 2024-03-28T23:31:19.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.261+0000 7f73a3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73a414eca0 0x7f73a41ec840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:19.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.261+0000 7f73a3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73a414eca0 0x7f73a41ec840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51070/0 (socket says 172.21.15.161:51070) 2024-03-28T23:31:19.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.261+0000 7f73a37fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73a406ca20 0x7f73a414dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:19.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.261+0000 7f73a3fff640 1 -- 172.21.15.161:0/1755945655 learned_addr learned my addr 172.21.15.161:0/1755945655 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:19.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.261+0000 7f73a2ffd640 1 --2- 172.21.15.161:0/1755945655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73a406ce80 0x7f73a406b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:19.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.261+0000 7f73a3fff640 1 -- 172.21.15.161:0/1755945655 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73a406ca20 msgr2=0x7f73a414dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:19.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.261+0000 7f73a3fff640 1 --2- 172.21.15.161:0/1755945655 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73a406ca20 0x7f73a414dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.261+0000 7f73a3fff640 1 -- 172.21.15.161:0/1755945655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73a406ce80 msgr2=0x7f73a406b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:19.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.261+0000 7f73a3fff640 1 --2- 172.21.15.161:0/1755945655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73a406ce80 0x7f73a406b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.261+0000 7f73a3fff640 1 -- 172.21.15.161:0/1755945655 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73a413ef30 con 0x7f73a414eca0 2024-03-28T23:31:19.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.261+0000 7f73a3fff640 1 --2- 172.21.15.161:0/1755945655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73a414eca0 0x7f73a41ec840 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f739c009eb0 tx=0x7f739c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:19.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.261+0000 7f738bfff640 1 -- 172.21.15.161:0/1755945655 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739c039050 con 0x7f73a414eca0 2024-03-28T23:31:19.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.262+0000 7f738bfff640 1 -- 172.21.15.161:0/1755945655 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f739c045070 con 0x7f73a414eca0 2024-03-28T23:31:19.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.262+0000 7f738bfff640 1 -- 172.21.15.161:0/1755945655 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739c009450 con 0x7f73a414eca0 2024-03-28T23:31:19.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.262+0000 7f73aa0ed640 1 -- 172.21.15.161:0/1755945655 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73a4147ac0 con 0x7f73a414eca0 2024-03-28T23:31:19.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.262+0000 7f73aa0ed640 1 -- 172.21.15.161:0/1755945655 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73a41477e0 con 0x7f73a414eca0 2024-03-28T23:31:19.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.262+0000 7f73aa0ed640 1 -- 172.21.15.161:0/1755945655 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73a404f4b0 con 0x7f73a414eca0 2024-03-28T23:31:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.263+0000 7f738bfff640 1 -- 172.21.15.161:0/1755945655 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f739c03e830 con 0x7f73a414eca0 2024-03-28T23:31:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.263+0000 7f738bfff640 1 --2- 172.21.15.161:0/1755945655 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f739006b500 0x7f739006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.263+0000 7f73a37fe640 1 --2- 172.21.15.161:0/1755945655 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f739006b500 0x7f739006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.263+0000 7f738bfff640 1 -- 172.21.15.161:0/1755945655 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f739c0ac7f0 con 0x7f73a414eca0 2024-03-28T23:31:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.264+0000 7f73a37fe640 1 --2- 172.21.15.161:0/1755945655 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f739006b500 0x7f739006d9c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7394000f60 tx=0x7f7394007d70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.268+0000 7f738bfff640 1 -- 172.21.15.161:0/1755945655 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f739c04e020 con 0x7f73a414eca0 2024-03-28T23:31:19.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.443+0000 7f73aa0ed640 1 -- 172.21.15.161:0/1755945655 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f73a41454e0 con 0x7f73a414eca0 2024-03-28T23:31:19.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.445+0000 7f738bfff640 1 -- 172.21.15.161:0/1755945655 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+17109 (secure 0 0 0) 0x7f739c07af90 con 0x7f73a414eca0 2024-03-28T23:31:19.446 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:19.446 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":39,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:31:07.634589+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:30:56.085611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-28T23:30:56.575631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6837/2636327316":"2024-03-29T23:31:07.634564+0000","172.21.15.161:6835/2636327316":"2024-03-29T23:31:07.634564+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:31:19.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.447+0000 7f73aa0ed640 1 -- 172.21.15.161:0/1755945655 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f739006b500 msgr2=0x7f739006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:19.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.447+0000 7f73aa0ed640 1 --2- 172.21.15.161:0/1755945655 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f739006b500 0x7f739006d9c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7394000f60 tx=0x7f7394007d70 comp rx=0 tx=0).stop 2024-03-28T23:31:19.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.447+0000 7f73aa0ed640 1 -- 172.21.15.161:0/1755945655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73a414eca0 msgr2=0x7f73a41ec840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:19.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.447+0000 7f73aa0ed640 1 --2- 172.21.15.161:0/1755945655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73a414eca0 0x7f73a41ec840 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f739c009eb0 tx=0x7f739c004a40 comp rx=0 tx=0).stop 2024-03-28T23:31:19.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.447+0000 7f73aa0ed640 1 -- 172.21.15.161:0/1755945655 shutdown_connections 2024-03-28T23:31:19.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.447+0000 7f73aa0ed640 1 --2- 172.21.15.161:0/1755945655 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f739006b500 0x7f739006d9c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.447+0000 7f73aa0ed640 1 --2- 172.21.15.161:0/1755945655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73a414eca0 0x7f73a41ec840 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.447+0000 7f73aa0ed640 1 --2- 172.21.15.161:0/1755945655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73a406ce80 0x7f73a406b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.447+0000 7f73aa0ed640 1 --2- 172.21.15.161:0/1755945655 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73a406ca20 0x7f73a414dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.447+0000 7f73aa0ed640 1 -- 172.21.15.161:0/1755945655 >> 172.21.15.161:0/1755945655 conn(0x7f73a40c6670 msgr2=0x7f73a4148e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:19.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.447+0000 7f73aa0ed640 1 -- 172.21.15.161:0/1755945655 shutdown_connections 2024-03-28T23:31:19.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.447+0000 7f73aa0ed640 1 -- 172.21.15.161:0/1755945655 wait complete. 2024-03-28T23:31:19.459 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:30:56.085611+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:30:56.575631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:31:19.460 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:31:19.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.562+0000 7f82574b5640 1 Processor -- start 2024-03-28T23:31:19.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.563+0000 7f82574b5640 1 -- start start 2024-03-28T23:31:19.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.563+0000 7f82574b5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82501511e0 0x7f825006d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:19.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.563+0000 7f82574b5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f82500c6470 0x7f8250151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:19.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.563+0000 7f82574b5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f825006d610 0x7f82501520e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:19.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.563+0000 7f82574b5640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f82500c1290 con 0x7f82500c6470 2024-03-28T23:31:19.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.563+0000 7f82574b5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f82500c2fc0 con 0x7f825006d610 2024-03-28T23:31:19.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.563+0000 7f82574b5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8250152620 con 0x7f82501511e0 2024-03-28T23:31:19.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.564+0000 7f8254a50640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f82500c6470 0x7f8250151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:19.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.564+0000 7f8254a50640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f82500c6470 0x7f8250151ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51074/0 (socket says 172.21.15.161:51074) 2024-03-28T23:31:19.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.564+0000 7f8254a50640 1 -- 172.21.15.161:0/1268283771 learned_addr learned my addr 172.21.15.161:0/1268283771 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:19.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.564+0000 7f8255a52640 1 --2- 172.21.15.161:0/1268283771 >> v2:172.21.15.170:3300/0 conn(0x7f825006d610 0x7f82501520e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:19.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.564+0000 7f8255251640 1 --2- 172.21.15.161:0/1268283771 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82501511e0 0x7f825006d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:19.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.564+0000 7f8254a50640 1 -- 172.21.15.161:0/1268283771 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82501511e0 msgr2=0x7f825006d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:19.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.564+0000 7f8254a50640 1 --2- 172.21.15.161:0/1268283771 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82501511e0 0x7f825006d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.564+0000 7f8254a50640 1 -- 172.21.15.161:0/1268283771 >> v2:172.21.15.170:3300/0 conn(0x7f825006d610 msgr2=0x7f82501520e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:19.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.564+0000 7f8254a50640 1 --2- 172.21.15.161:0/1268283771 >> v2:172.21.15.170:3300/0 conn(0x7f825006d610 0x7f82501520e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.564+0000 7f8254a50640 1 -- 172.21.15.161:0/1268283771 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f825006c8a0 con 0x7f82500c6470 2024-03-28T23:31:19.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.564+0000 7f8255a52640 1 --2- 172.21.15.161:0/1268283771 >> v2:172.21.15.170:3300/0 conn(0x7f825006d610 0x7f82501520e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:19.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.565+0000 7f8254a50640 1 --2- 172.21.15.161:0/1268283771 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82500c6470 0x7f8250151ba0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f82500c4280 tx=0x7f824c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12a9402900c1dba5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:19.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.565+0000 7f8243fff640 1 -- 172.21.15.161:0/1268283771 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c038070 con 0x7f82500c6470 2024-03-28T23:31:19.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.565+0000 7f8243fff640 1 -- 172.21.15.161:0/1268283771 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f824c02f050 con 0x7f82500c6470 2024-03-28T23:31:19.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.565+0000 7f8243fff640 1 -- 172.21.15.161:0/1268283771 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c031970 con 0x7f82500c6470 2024-03-28T23:31:19.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.565+0000 7f82574b5640 1 -- 172.21.15.161:0/1268283771 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82500c6470 msgr2=0x7f8250151ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:19.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.565+0000 7f82574b5640 1 --2- 172.21.15.161:0/1268283771 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82500c6470 0x7f8250151ba0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f82500c4280 tx=0x7f824c02b7f0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.565+0000 7f82574b5640 1 -- 172.21.15.161:0/1268283771 shutdown_connections 2024-03-28T23:31:19.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.565+0000 7f82574b5640 1 --2- 172.21.15.161:0/1268283771 >> v2:172.21.15.170:3300/0 conn(0x7f825006d610 0x7f82501520e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.565+0000 7f82574b5640 1 --2- 172.21.15.161:0/1268283771 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82500c6470 0x7f8250151ba0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.565+0000 7f82574b5640 1 --2- 172.21.15.161:0/1268283771 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82501511e0 0x7f825006d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.565+0000 7f82574b5640 1 -- 172.21.15.161:0/1268283771 >> 172.21.15.161:0/1268283771 conn(0x7f82500c3390 msgr2=0x7f82500c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:19.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.566+0000 7f82574b5640 1 -- 172.21.15.161:0/1268283771 shutdown_connections 2024-03-28T23:31:19.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.566+0000 7f82574b5640 1 -- 172.21.15.161:0/1268283771 wait complete. 2024-03-28T23:31:19.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.567+0000 7f82574b5640 1 Processor -- start 2024-03-28T23:31:19.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.567+0000 7f82574b5640 1 -- start start 2024-03-28T23:31:19.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.567+0000 7f82574b5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f825006d610 0x7f82501e4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:19.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.567+0000 7f82574b5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82501511e0 0x7f8250154c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:19.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.567+0000 7f82574b5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82501e8100 0x7f82501ea1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:19.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.567+0000 7f82574b5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8250151740 con 0x7f82501511e0 2024-03-28T23:31:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.567+0000 7f82574b5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f82501515c0 con 0x7f82501e8100 2024-03-28T23:31:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.567+0000 7f82574b5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f825006c160 con 0x7f825006d610 2024-03-28T23:31:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.567+0000 7f8254a50640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82501511e0 0x7f8250154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.567+0000 7f8254a50640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82501511e0 0x7f8250154c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51086/0 (socket says 172.21.15.161:51086) 2024-03-28T23:31:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.567+0000 7f8254a50640 1 -- 172.21.15.161:0/1857194943 learned_addr learned my addr 172.21.15.161:0/1857194943 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.567+0000 7f8255251640 1 --2- 172.21.15.161:0/1857194943 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f825006d610 0x7f82501e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.567+0000 7f8255a52640 1 --2- 172.21.15.161:0/1857194943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82501e8100 0x7f82501ea1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.568+0000 7f8254a50640 1 -- 172.21.15.161:0/1857194943 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f825006d610 msgr2=0x7f82501e4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.568+0000 7f8254a50640 1 --2- 172.21.15.161:0/1857194943 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f825006d610 0x7f82501e4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.568+0000 7f8254a50640 1 -- 172.21.15.161:0/1857194943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82501e8100 msgr2=0x7f82501ea1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.568+0000 7f8254a50640 1 --2- 172.21.15.161:0/1857194943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82501e8100 0x7f82501ea1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.568+0000 7f8254a50640 1 -- 172.21.15.161:0/1857194943 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8250070870 con 0x7f82501511e0 2024-03-28T23:31:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.568+0000 7f8254a50640 1 --2- 172.21.15.161:0/1857194943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82501511e0 0x7f8250154c10 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f824c000c00 tx=0x7f824c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.568+0000 7f8241ffb640 1 -- 172.21.15.161:0/1857194943 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c042070 con 0x7f82501511e0 2024-03-28T23:31:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.568+0000 7f8241ffb640 1 -- 172.21.15.161:0/1857194943 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f824c03d680 con 0x7f82501511e0 2024-03-28T23:31:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.568+0000 7f8241ffb640 1 -- 172.21.15.161:0/1857194943 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c038040 con 0x7f82501511e0 2024-03-28T23:31:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.568+0000 7f82574b5640 1 -- 172.21.15.161:0/1857194943 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8250070b50 con 0x7f82501511e0 2024-03-28T23:31:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.568+0000 7f82574b5640 1 -- 172.21.15.161:0/1857194943 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82501546f0 con 0x7f82501511e0 2024-03-28T23:31:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.569+0000 7f82574b5640 1 -- 172.21.15.161:0/1857194943 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8250068f90 con 0x7f82501511e0 2024-03-28T23:31:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.570+0000 7f8241ffb640 1 -- 172.21.15.161:0/1857194943 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f824c045070 con 0x7f82501511e0 2024-03-28T23:31:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.570+0000 7f8241ffb640 1 --2- 172.21.15.161:0/1857194943 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f823406b4d0 0x7f823406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.570+0000 7f8241ffb640 1 -- 172.21.15.161:0/1857194943 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f824c0abc70 con 0x7f82501511e0 2024-03-28T23:31:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.570+0000 7f8255251640 1 --2- 172.21.15.161:0/1857194943 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f823406b4d0 0x7f823406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.570+0000 7f8255251640 1 --2- 172.21.15.161:0/1857194943 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f823406b4d0 0x7f823406d990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8244009a20 tx=0x7f8244009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:19.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.574+0000 7f8241ffb640 1 -- 172.21.15.161:0/1857194943 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8250068f90 con 0x7f82501511e0 2024-03-28T23:31:19.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.747+0000 7f82574b5640 1 -- 172.21.15.161:0/1857194943 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f825004f4b0 con 0x7f82501511e0 2024-03-28T23:31:19.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.748+0000 7f8241ffb640 1 -- 172.21.15.161:0/1857194943 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) ==== 93+0+10 (secure 0 0 0) 0x7f824c07a470 con 0x7f82501511e0 2024-03-28T23:31:19.749 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:31:19.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.750+0000 7f82574b5640 1 -- 172.21.15.161:0/1857194943 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f823406b4d0 msgr2=0x7f823406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:19.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.750+0000 7f82574b5640 1 --2- 172.21.15.161:0/1857194943 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f823406b4d0 0x7f823406d990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8244009a20 tx=0x7f8244009190 comp rx=0 tx=0).stop 2024-03-28T23:31:19.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.750+0000 7f82574b5640 1 -- 172.21.15.161:0/1857194943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82501511e0 msgr2=0x7f8250154c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:19.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.750+0000 7f82574b5640 1 --2- 172.21.15.161:0/1857194943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82501511e0 0x7f8250154c10 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f824c000c00 tx=0x7f824c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.750+0000 7f82574b5640 1 -- 172.21.15.161:0/1857194943 shutdown_connections 2024-03-28T23:31:19.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.750+0000 7f82574b5640 1 --2- 172.21.15.161:0/1857194943 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f823406b4d0 0x7f823406d990 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.750+0000 7f82574b5640 1 --2- 172.21.15.161:0/1857194943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82501511e0 0x7f8250154c10 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.750+0000 7f82574b5640 1 --2- 172.21.15.161:0/1857194943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82501e8100 0x7f82501ea1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.750+0000 7f82574b5640 1 --2- 172.21.15.161:0/1857194943 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f825006d610 0x7f82501e4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.750+0000 7f82574b5640 1 -- 172.21.15.161:0/1857194943 >> 172.21.15.161:0/1857194943 conn(0x7f82500c3390 msgr2=0x7f8250151aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:19.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.750+0000 7f82574b5640 1 -- 172.21.15.161:0/1857194943 shutdown_connections 2024-03-28T23:31:19.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.750+0000 7f82574b5640 1 -- 172.21.15.161:0/1857194943 wait complete. 2024-03-28T23:31:19.761 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:31:19.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.864+0000 7fea6264e640 1 Processor -- start 2024-03-28T23:31:19.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.864+0000 7fea6264e640 1 -- start start 2024-03-28T23:31:19.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.865+0000 7fea6264e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fea5c0c7330 0x7fea5c14e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:19.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.865+0000 7fea6264e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea5c0abc00 con 0x7fea5c0733e0 2024-03-28T23:31:19.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.865+0000 7fea6264e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea5c06bf70 con 0x7fea5c0a9960 2024-03-28T23:31:19.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.865+0000 7fea6264e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fea5c0aa3a0 con 0x7fea5c0c7330 2024-03-28T23:31:19.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.865+0000 7fea5b7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fea5c0c7330 0x7fea5c14e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:19.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.865+0000 7fea60beb640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fea5c0733e0 0x7fea5c14e9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48044/0 (socket says 172.21.15.161:48044) 2024-03-28T23:31:19.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.865+0000 7fea60beb640 1 -- 172.21.15.161:0/2483721056 learned_addr learned my addr 172.21.15.161:0/2483721056 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:19.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.866+0000 7fea5b7fe640 1 -- 172.21.15.161:0/2483721056 >> v1:172.21.15.170:6789/0 conn(0x7fea5c0a9960 legacy=0x7fea5c072970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:19.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.866+0000 7fea5affd640 1 -- 172.21.15.161:0/2483721056 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2930067628 0 0) 0x7fea5c06bf70 con 0x7fea5c0a9960 2024-03-28T23:31:19.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.866+0000 7fea5b7fe640 1 -- 172.21.15.161:0/2483721056 >> v1:172.21.15.161:6789/0 conn(0x7fea5c0733e0 legacy=0x7fea5c14e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:19.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.866+0000 7fea5b7fe640 1 -- 172.21.15.161:0/2483721056 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea5c06bcd0 con 0x7fea5c0c7330 2024-03-28T23:31:19.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.866+0000 7fea5b7fe640 1 --2- 172.21.15.161:0/2483721056 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea5c0c7330 0x7fea5c14e460 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fea500097a0 tx=0x7fea5002c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e99f53448fade113 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:19.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.866+0000 7fea5affd640 1 -- 172.21.15.161:0/2483721056 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea50038070 con 0x7fea5c0c7330 2024-03-28T23:31:19.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.866+0000 7fea5affd640 1 -- 172.21.15.161:0/2483721056 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea50031710 con 0x7fea5c0c7330 2024-03-28T23:31:19.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.866+0000 7fea5affd640 1 -- 172.21.15.161:0/2483721056 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea50031ae0 con 0x7fea5c0c7330 2024-03-28T23:31:19.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.867+0000 7fea6264e640 1 -- 172.21.15.161:0/2483721056 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea5c0c7330 msgr2=0x7fea5c14e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:19.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.867+0000 7fea6264e640 1 --2- 172.21.15.161:0/2483721056 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea5c0c7330 0x7fea5c14e460 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fea500097a0 tx=0x7fea5002c790 comp rx=0 tx=0).stop 2024-03-28T23:31:19.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.867+0000 7fea6264e640 1 -- 172.21.15.161:0/2483721056 shutdown_connections 2024-03-28T23:31:19.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.867+0000 7fea6264e640 1 --2- 172.21.15.161:0/2483721056 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea5c0c7330 0x7fea5c14e460 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.867+0000 7fea6264e640 1 -- 172.21.15.161:0/2483721056 >> 172.21.15.161:0/2483721056 conn(0x7fea5c0a83f0 msgr2=0x7fea5c0a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:19.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.867+0000 7fea6264e640 1 -- 172.21.15.161:0/2483721056 shutdown_connections 2024-03-28T23:31:19.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.867+0000 7fea6264e640 1 -- 172.21.15.161:0/2483721056 wait complete. 2024-03-28T23:31:19.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.868+0000 7fea6264e640 1 Processor -- start 2024-03-28T23:31:19.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.868+0000 7fea6264e640 1 -- start start 2024-03-28T23:31:19.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.869+0000 7fea6264e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea5c0733e0 0x7fea5c09df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:19.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.869+0000 7fea6264e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea5c0a9960 0x7fea5c1ec2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:19.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.869+0000 7fea6264e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea5c0c7330 0x7fea5c0c00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:19.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.869+0000 7fea6264e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fea5c0ab820 con 0x7fea5c0a9960 2024-03-28T23:31:19.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.869+0000 7fea6264e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fea5c152250 con 0x7fea5c0c7330 2024-03-28T23:31:19.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.869+0000 7fea6264e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fea5c0ab9a0 con 0x7fea5c0733e0 2024-03-28T23:31:19.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.869+0000 7fea5b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea5c0a9960 0x7fea5c1ec2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:19.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.869+0000 7fea5b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea5c0a9960 0x7fea5c1ec2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51098/0 (socket says 172.21.15.161:51098) 2024-03-28T23:31:19.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.869+0000 7fea5b7fe640 1 -- 172.21.15.161:0/1014108158 learned_addr learned my addr 172.21.15.161:0/1014108158 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:19.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.869+0000 7fea60beb640 1 --2- 172.21.15.161:0/1014108158 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea5c0c7330 0x7fea5c0c00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:19.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.869+0000 7fea5bfff640 1 --2- 172.21.15.161:0/1014108158 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea5c0733e0 0x7fea5c09df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:19.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.869+0000 7fea5b7fe640 1 -- 172.21.15.161:0/1014108158 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea5c0733e0 msgr2=0x7fea5c09df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:19.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.869+0000 7fea5b7fe640 1 --2- 172.21.15.161:0/1014108158 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea5c0733e0 0x7fea5c09df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.869+0000 7fea5b7fe640 1 -- 172.21.15.161:0/1014108158 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea5c0c7330 msgr2=0x7fea5c0c00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:19.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.869+0000 7fea5b7fe640 1 --2- 172.21.15.161:0/1014108158 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea5c0c7330 0x7fea5c0c00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:19.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.869+0000 7fea5b7fe640 1 -- 172.21.15.161:0/1014108158 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea5c1e8a90 con 0x7fea5c0a9960 2024-03-28T23:31:19.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.870+0000 7fea5b7fe640 1 --2- 172.21.15.161:0/1014108158 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea5c0a9960 0x7fea5c1ec2e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fea50000c00 tx=0x7fea500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:19.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.870+0000 7fea58ff9640 1 -- 172.21.15.161:0/1014108158 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea50042070 con 0x7fea5c0a9960 2024-03-28T23:31:19.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.870+0000 7fea58ff9640 1 -- 172.21.15.161:0/1014108158 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea50033a40 con 0x7fea5c0a9960 2024-03-28T23:31:19.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.870+0000 7fea6264e640 1 -- 172.21.15.161:0/1014108158 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea5c1ebac0 con 0x7fea5c0a9960 2024-03-28T23:31:19.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.870+0000 7fea58ff9640 1 -- 172.21.15.161:0/1014108158 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea50038040 con 0x7fea5c0a9960 2024-03-28T23:31:19.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.870+0000 7fea6264e640 1 -- 172.21.15.161:0/1014108158 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea5c0c0630 con 0x7fea5c0a9960 2024-03-28T23:31:19.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.871+0000 7fea6264e640 1 -- 172.21.15.161:0/1014108158 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea5c04f4b0 con 0x7fea5c0a9960 2024-03-28T23:31:19.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.871+0000 7fea58ff9640 1 -- 172.21.15.161:0/1014108158 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fea50045070 con 0x7fea5c0a9960 2024-03-28T23:31:19.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.872+0000 7fea58ff9640 1 --2- 172.21.15.161:0/1014108158 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fea4806b4d0 0x7fea4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:19.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.872+0000 7fea58ff9640 1 -- 172.21.15.161:0/1014108158 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7fea500ac160 con 0x7fea5c0a9960 2024-03-28T23:31:19.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.872+0000 7fea5bfff640 1 --2- 172.21.15.161:0/1014108158 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fea4806b4d0 0x7fea4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:19.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.872+0000 7fea5bfff640 1 --2- 172.21.15.161:0/1014108158 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fea4806b4d0 0x7fea4806d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fea4c000f60 tx=0x7fea4c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:19.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:19.876+0000 7fea58ff9640 1 -- 172.21.15.161:0/1014108158 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fea5007a960 con 0x7fea5c0a9960 2024-03-28T23:31:20.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.051+0000 7fea6264e640 1 -- 172.21.15.161:0/1014108158 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fea5c06b1b0 con 0x7fea5c0a9960 2024-03-28T23:31:20.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.051+0000 7fea58ff9640 1 -- 172.21.15.161:0/1014108158 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v39) ==== 104+0+11 (secure 0 0 0) 0x7fea5007ea00 con 0x7fea5c0a9960 2024-03-28T23:31:20.052 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:31:20.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.053+0000 7fea6264e640 1 -- 172.21.15.161:0/1014108158 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fea4806b4d0 msgr2=0x7fea4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:20.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.053+0000 7fea6264e640 1 --2- 172.21.15.161:0/1014108158 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fea4806b4d0 0x7fea4806d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fea4c000f60 tx=0x7fea4c009040 comp rx=0 tx=0).stop 2024-03-28T23:31:20.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.053+0000 7fea6264e640 1 -- 172.21.15.161:0/1014108158 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea5c0a9960 msgr2=0x7fea5c1ec2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:20.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.053+0000 7fea6264e640 1 --2- 172.21.15.161:0/1014108158 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea5c0a9960 0x7fea5c1ec2e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fea50000c00 tx=0x7fea500046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.053+0000 7fea6264e640 1 -- 172.21.15.161:0/1014108158 shutdown_connections 2024-03-28T23:31:20.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.053+0000 7fea6264e640 1 --2- 172.21.15.161:0/1014108158 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fea4806b4d0 0x7fea4806d990 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.053+0000 7fea6264e640 1 --2- 172.21.15.161:0/1014108158 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea5c0a9960 0x7fea5c1ec2e0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.053+0000 7fea6264e640 1 --2- 172.21.15.161:0/1014108158 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea5c0c7330 0x7fea5c0c00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.053+0000 7fea6264e640 1 --2- 172.21.15.161:0/1014108158 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea5c0733e0 0x7fea5c09df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.053+0000 7fea6264e640 1 -- 172.21.15.161:0/1014108158 >> 172.21.15.161:0/1014108158 conn(0x7fea5c0a83f0 msgr2=0x7fea5c0c66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:20.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.053+0000 7fea6264e640 1 -- 172.21.15.161:0/1014108158 shutdown_connections 2024-03-28T23:31:20.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.053+0000 7fea6264e640 1 -- 172.21.15.161:0/1014108158 wait complete. 2024-03-28T23:31:20.065 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:31:20.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.169+0000 7f378ee4e640 1 Processor -- start 2024-03-28T23:31:20.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.169+0000 7f378ee4e640 1 -- start start 2024-03-28T23:31:20.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.170+0000 7f378ee4e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37880c5dd0 0x7f37880a9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:20.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.170+0000 7f378ee4e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f378806ea30 0x7f37880c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:20.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.170+0000 7f378ee4e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f37880c1290 0x7f37880c23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:20.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.170+0000 7f378ee4e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f37880c61b0 con 0x7f37880c1290 2024-03-28T23:31:20.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.170+0000 7f378ee4e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f37880a6290 con 0x7f378806ea30 2024-03-28T23:31:20.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.170+0000 7f378ee4e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f37881532a0 con 0x7f37880c5dd0 2024-03-28T23:31:20.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.170+0000 7f378d3eb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f37880c1290 0x7f37880c23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:20.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.170+0000 7f378d3eb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f37880c1290 0x7f37880c23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51114/0 (socket says 172.21.15.161:51114) 2024-03-28T23:31:20.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.170+0000 7f378d3eb640 1 -- 172.21.15.161:0/1059120545 learned_addr learned my addr 172.21.15.161:0/1059120545 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:20.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.170+0000 7f3787fff640 1 --2- 172.21.15.161:0/1059120545 >> v2:172.21.15.170:3300/0 conn(0x7f378806ea30 0x7f37880c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:20.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.170+0000 7f378cbea640 1 --2- 172.21.15.161:0/1059120545 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37880c5dd0 0x7f37880a9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:20.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.170+0000 7f3787fff640 1 -- 172.21.15.161:0/1059120545 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37880c5dd0 msgr2=0x7f37880a9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:20.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.170+0000 7f3787fff640 1 --2- 172.21.15.161:0/1059120545 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37880c5dd0 0x7f37880a9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.170+0000 7f3787fff640 1 -- 172.21.15.161:0/1059120545 >> v2:172.21.15.161:3300/0 conn(0x7f37880c1290 msgr2=0x7f37880c23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:20.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.170+0000 7f3787fff640 1 --2- 172.21.15.161:0/1059120545 >> v2:172.21.15.161:3300/0 conn(0x7f37880c1290 0x7f37880c23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.170+0000 7f3787fff640 1 -- 172.21.15.161:0/1059120545 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37880abaa0 con 0x7f378806ea30 2024-03-28T23:31:20.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.170+0000 7f378d3eb640 1 --2- 172.21.15.161:0/1059120545 >> v2:172.21.15.161:3300/0 conn(0x7f37880c1290 0x7f37880c23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:20.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.171+0000 7f3787fff640 1 --2- 172.21.15.161:0/1059120545 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f378806ea30 0x7f37880c1eb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f37880c4450 tx=0x7f377c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f18624aa758da2aa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:20.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.171+0000 7f37877fe640 1 -- 172.21.15.161:0/1059120545 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c0315a0 con 0x7f378806ea30 2024-03-28T23:31:20.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.171+0000 7f37877fe640 1 -- 172.21.15.161:0/1059120545 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f377c037070 con 0x7f378806ea30 2024-03-28T23:31:20.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.171+0000 7f37877fe640 1 -- 172.21.15.161:0/1059120545 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c031960 con 0x7f378806ea30 2024-03-28T23:31:20.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.172+0000 7f378ee4e640 1 -- 172.21.15.161:0/1059120545 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f378806ea30 msgr2=0x7f37880c1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:20.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.172+0000 7f378ee4e640 1 --2- 172.21.15.161:0/1059120545 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f378806ea30 0x7f37880c1eb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f37880c4450 tx=0x7f377c02b840 comp rx=0 tx=0).stop 2024-03-28T23:31:20.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.172+0000 7f378ee4e640 1 -- 172.21.15.161:0/1059120545 shutdown_connections 2024-03-28T23:31:20.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.172+0000 7f378ee4e640 1 --2- 172.21.15.161:0/1059120545 >> v2:172.21.15.161:3300/0 conn(0x7f37880c1290 0x7f37880c23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.172+0000 7f378ee4e640 1 --2- 172.21.15.161:0/1059120545 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f378806ea30 0x7f37880c1eb0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.172+0000 7f378ee4e640 1 --2- 172.21.15.161:0/1059120545 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37880c5dd0 0x7f37880a9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.172+0000 7f378ee4e640 1 -- 172.21.15.161:0/1059120545 >> 172.21.15.161:0/1059120545 conn(0x7f37880c3390 msgr2=0x7f37880c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:20.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.172+0000 7f378ee4e640 1 -- 172.21.15.161:0/1059120545 shutdown_connections 2024-03-28T23:31:20.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.172+0000 7f378ee4e640 1 -- 172.21.15.161:0/1059120545 wait complete. 2024-03-28T23:31:20.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.173+0000 7f378ee4e640 1 Processor -- start 2024-03-28T23:31:20.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.173+0000 7f378ee4e640 1 -- start start 2024-03-28T23:31:20.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.174+0000 7f378ee4e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f378806ea30 0x7f3788147ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:20.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.174+0000 7f378ee4e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37880c1290 0x7f37880c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:20.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.174+0000 7f378ee4e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37880c5dd0 0x7f3788070970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:20.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.174+0000 7f378ee4e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f37880ac340 con 0x7f378806ea30 2024-03-28T23:31:20.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.174+0000 7f378ee4e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f37880ac060 con 0x7f37880c1290 2024-03-28T23:31:20.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.174+0000 7f378ee4e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3788152f50 con 0x7f37880c5dd0 2024-03-28T23:31:20.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.174+0000 7f378cbea640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f378806ea30 0x7f3788147ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:20.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.174+0000 7f378cbea640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f378806ea30 0x7f3788147ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51122/0 (socket says 172.21.15.161:51122) 2024-03-28T23:31:20.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.174+0000 7f378cbea640 1 -- 172.21.15.161:0/2297012247 learned_addr learned my addr 172.21.15.161:0/2297012247 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:20.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.174+0000 7f378d3eb640 1 --2- 172.21.15.161:0/2297012247 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37880c5dd0 0x7f3788070970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:20.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.174+0000 7f3787fff640 1 --2- 172.21.15.161:0/2297012247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37880c1290 0x7f37880c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:20.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.174+0000 7f378cbea640 1 -- 172.21.15.161:0/2297012247 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37880c5dd0 msgr2=0x7f3788070970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:20.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.174+0000 7f378cbea640 1 --2- 172.21.15.161:0/2297012247 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37880c5dd0 0x7f3788070970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.174+0000 7f378cbea640 1 -- 172.21.15.161:0/2297012247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37880c1290 msgr2=0x7f37880c24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:20.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.174+0000 7f378cbea640 1 --2- 172.21.15.161:0/2297012247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37880c1290 0x7f37880c24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.174+0000 7f378cbea640 1 -- 172.21.15.161:0/2297012247 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37881ec6d0 con 0x7f378806ea30 2024-03-28T23:31:20.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.174+0000 7f378cbea640 1 --2- 172.21.15.161:0/2297012247 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f378806ea30 0x7f3788147ae0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f377800e870 tx=0x7f377800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:20.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.175+0000 7f37857fa640 1 -- 172.21.15.161:0/2297012247 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3778017070 con 0x7f378806ea30 2024-03-28T23:31:20.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.175+0000 7f37857fa640 1 -- 172.21.15.161:0/2297012247 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f377800c940 con 0x7f378806ea30 2024-03-28T23:31:20.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.175+0000 7f378ee4e640 1 -- 172.21.15.161:0/2297012247 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37880a5f30 con 0x7f378806ea30 2024-03-28T23:31:20.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.175+0000 7f37857fa640 1 -- 172.21.15.161:0/2297012247 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377800cd30 con 0x7f378806ea30 2024-03-28T23:31:20.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.175+0000 7f378ee4e640 1 -- 172.21.15.161:0/2297012247 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37881564e0 con 0x7f378806ea30 2024-03-28T23:31:20.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.176+0000 7f378ee4e640 1 -- 172.21.15.161:0/2297012247 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f378804f4b0 con 0x7f378806ea30 2024-03-28T23:31:20.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.176+0000 7f37857fa640 1 -- 172.21.15.161:0/2297012247 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f377801a070 con 0x7f378806ea30 2024-03-28T23:31:20.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.177+0000 7f37857fa640 1 --2- 172.21.15.161:0/2297012247 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f377406b4d0 0x7f377406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:20.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.177+0000 7f37857fa640 1 -- 172.21.15.161:0/2297012247 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f377808a160 con 0x7f378806ea30 2024-03-28T23:31:20.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.177+0000 7f3787fff640 1 --2- 172.21.15.161:0/2297012247 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f377406b4d0 0x7f377406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:20.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.177+0000 7f3787fff640 1 --2- 172.21.15.161:0/2297012247 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f377406b4d0 0x7f377406d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f377c036000 tx=0x7f377c03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:20.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.181+0000 7f37857fa640 1 -- 172.21.15.161:0/2297012247 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3778058840 con 0x7f378806ea30 2024-03-28T23:31:20.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.352+0000 7f378ee4e640 1 -- 172.21.15.161:0/2297012247 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f37880c6360 con 0x7f378806ea30 2024-03-28T23:31:20.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.353+0000 7f37857fa640 1 -- 172.21.15.161:0/2297012247 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v39) ==== 100+0+11 (secure 0 0 0) 0x7f377805c8e0 con 0x7f378806ea30 2024-03-28T23:31:20.354 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:31:20.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.354+0000 7f378ee4e640 1 -- 172.21.15.161:0/2297012247 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f377406b4d0 msgr2=0x7f377406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:20.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.354+0000 7f378ee4e640 1 --2- 172.21.15.161:0/2297012247 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f377406b4d0 0x7f377406d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f377c036000 tx=0x7f377c03d040 comp rx=0 tx=0).stop 2024-03-28T23:31:20.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.354+0000 7f378ee4e640 1 -- 172.21.15.161:0/2297012247 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f378806ea30 msgr2=0x7f3788147ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:20.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.354+0000 7f378ee4e640 1 --2- 172.21.15.161:0/2297012247 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f378806ea30 0x7f3788147ae0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f377800e870 tx=0x7f377800ed30 comp rx=0 tx=0).stop 2024-03-28T23:31:20.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.355+0000 7f378ee4e640 1 -- 172.21.15.161:0/2297012247 shutdown_connections 2024-03-28T23:31:20.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.355+0000 7f378ee4e640 1 --2- 172.21.15.161:0/2297012247 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f377406b4d0 0x7f377406d990 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.355+0000 7f378ee4e640 1 --2- 172.21.15.161:0/2297012247 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37880c5dd0 0x7f3788070970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.355+0000 7f378ee4e640 1 --2- 172.21.15.161:0/2297012247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37880c1290 0x7f37880c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.355+0000 7f378ee4e640 1 --2- 172.21.15.161:0/2297012247 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f378806ea30 0x7f3788147ae0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.355+0000 7f378ee4e640 1 -- 172.21.15.161:0/2297012247 >> 172.21.15.161:0/2297012247 conn(0x7f37880c3390 msgr2=0x7f3788152380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:20.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.355+0000 7f378ee4e640 1 -- 172.21.15.161:0/2297012247 shutdown_connections 2024-03-28T23:31:20.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.355+0000 7f378ee4e640 1 -- 172.21.15.161:0/2297012247 wait complete. 2024-03-28T23:31:20.366 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:31:20.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.467+0000 7faa5829d640 1 Processor -- start 2024-03-28T23:31:20.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.467+0000 7faa5829d640 1 -- start start 2024-03-28T23:31:20.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.468+0000 7faa5829d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7faa50150600 0x7faa500a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:20.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.468+0000 7faa5829d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7faa500a8fc0 0x7faa50153270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:20.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.468+0000 7faa5829d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa500aacb0 0x7faa501537b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:20.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.468+0000 7faa5829d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7faa50156570 con 0x7faa500a8fc0 2024-03-28T23:31:20.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.468+0000 7faa5829d640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7faa501567c0 con 0x7faa50150600 2024-03-28T23:31:20.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.468+0000 7faa5829d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7faa50156aa0 con 0x7faa500aacb0 2024-03-28T23:31:20.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.468+0000 7faa55838640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7faa500a8fc0 0x7faa50153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:20.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.468+0000 7faa56039640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7faa50150600 0x7faa500a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:20.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.468+0000 7faa55838640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7faa500a8fc0 0x7faa50153270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51126/0 (socket says 172.21.15.161:51126) 2024-03-28T23:31:20.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.468+0000 7faa5683a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa500aacb0 0x7faa501537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:20.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.468+0000 7faa55838640 1 -- 172.21.15.161:0/1659801858 learned_addr learned my addr 172.21.15.161:0/1659801858 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:20.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.468+0000 7faa56039640 1 -- 172.21.15.161:0/1659801858 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa500aacb0 msgr2=0x7faa501537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:20.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.468+0000 7faa56039640 1 --2- 172.21.15.161:0/1659801858 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa500aacb0 0x7faa501537b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.468+0000 7faa56039640 1 -- 172.21.15.161:0/1659801858 >> v2:172.21.15.161:3300/0 conn(0x7faa500a8fc0 msgr2=0x7faa50153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:20.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.468+0000 7faa56039640 1 --2- 172.21.15.161:0/1659801858 >> v2:172.21.15.161:3300/0 conn(0x7faa500a8fc0 0x7faa50153270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.468+0000 7faa56039640 1 -- 172.21.15.161:0/1659801858 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa50156d80 con 0x7faa50150600 2024-03-28T23:31:20.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.468+0000 7faa55838640 1 --2- 172.21.15.161:0/1659801858 >> v2:172.21.15.161:3300/0 conn(0x7faa500a8fc0 0x7faa50153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:20.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.469+0000 7faa56039640 1 --2- 172.21.15.161:0/1659801858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa50150600 0x7faa500a9f40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faa500c4210 tx=0x7faa4c02b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a79d1dc0d9a6d443 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:20.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.469+0000 7faa55037640 1 -- 172.21.15.161:0/1659801858 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4c038070 con 0x7faa50150600 2024-03-28T23:31:20.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.469+0000 7faa55037640 1 -- 172.21.15.161:0/1659801858 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa4c031700 con 0x7faa50150600 2024-03-28T23:31:20.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.469+0000 7faa55037640 1 -- 172.21.15.161:0/1659801858 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4c031ad0 con 0x7faa50150600 2024-03-28T23:31:20.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.470+0000 7faa5829d640 1 -- 172.21.15.161:0/1659801858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa50150600 msgr2=0x7faa500a9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:20.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.470+0000 7faa5829d640 1 --2- 172.21.15.161:0/1659801858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa50150600 0x7faa500a9f40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faa500c4210 tx=0x7faa4c02b7a0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.470+0000 7faa5829d640 1 -- 172.21.15.161:0/1659801858 shutdown_connections 2024-03-28T23:31:20.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.470+0000 7faa5829d640 1 --2- 172.21.15.161:0/1659801858 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa500aacb0 0x7faa501537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.470+0000 7faa5829d640 1 --2- 172.21.15.161:0/1659801858 >> v2:172.21.15.161:3300/0 conn(0x7faa500a8fc0 0x7faa50153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.470+0000 7faa5829d640 1 --2- 172.21.15.161:0/1659801858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa50150600 0x7faa500a9f40 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.470+0000 7faa5829d640 1 -- 172.21.15.161:0/1659801858 >> 172.21.15.161:0/1659801858 conn(0x7faa500c6670 msgr2=0x7faa500c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:20.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.470+0000 7faa5829d640 1 -- 172.21.15.161:0/1659801858 shutdown_connections 2024-03-28T23:31:20.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.470+0000 7faa5829d640 1 -- 172.21.15.161:0/1659801858 wait complete. 2024-03-28T23:31:20.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.471+0000 7faa5829d640 1 Processor -- start 2024-03-28T23:31:20.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.471+0000 7faa5829d640 1 -- start start 2024-03-28T23:31:20.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.471+0000 7faa5829d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa500a8fc0 0x7faa501ecc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:20.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.471+0000 7faa5829d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa500aacb0 0x7faa501e9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:20.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.471+0000 7faa5829d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa50150600 0x7faa5014cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:20.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.471+0000 7faa5829d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7faa501509e0 con 0x7faa500aacb0 2024-03-28T23:31:20.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.471+0000 7faa5829d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7faa50157340 con 0x7faa50150600 2024-03-28T23:31:20.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.471+0000 7faa5829d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7faa500aa9e0 con 0x7faa500a8fc0 2024-03-28T23:31:20.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.472+0000 7faa55838640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa500aacb0 0x7faa501e9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:20.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.472+0000 7faa55838640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa500aacb0 0x7faa501e9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51136/0 (socket says 172.21.15.161:51136) 2024-03-28T23:31:20.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.472+0000 7faa5683a640 1 --2- 172.21.15.161:0/2491881740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa50150600 0x7faa5014cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:20.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.472+0000 7faa55838640 1 -- 172.21.15.161:0/2491881740 learned_addr learned my addr 172.21.15.161:0/2491881740 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:20.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.472+0000 7faa56039640 1 --2- 172.21.15.161:0/2491881740 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa500a8fc0 0x7faa501ecc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:20.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.472+0000 7faa55838640 1 -- 172.21.15.161:0/2491881740 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa500a8fc0 msgr2=0x7faa501ecc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:20.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.472+0000 7faa55838640 1 --2- 172.21.15.161:0/2491881740 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa500a8fc0 0x7faa501ecc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.472+0000 7faa55838640 1 -- 172.21.15.161:0/2491881740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa50150600 msgr2=0x7faa5014cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:20.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.472+0000 7faa55838640 1 --2- 172.21.15.161:0/2491881740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa50150600 0x7faa5014cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.472+0000 7faa55838640 1 -- 172.21.15.161:0/2491881740 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa5014fd90 con 0x7faa500aacb0 2024-03-28T23:31:20.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.472+0000 7faa5683a640 1 --2- 172.21.15.161:0/2491881740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa50150600 0x7faa5014cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:20.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.472+0000 7faa55838640 1 --2- 172.21.15.161:0/2491881740 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa500aacb0 0x7faa501e9490 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7faa4000e9d0 tx=0x7faa4000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:20.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.472+0000 7faa46ffd640 1 -- 172.21.15.161:0/2491881740 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa40017070 con 0x7faa500aacb0 2024-03-28T23:31:20.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.472+0000 7faa46ffd640 1 -- 172.21.15.161:0/2491881740 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa4000ca60 con 0x7faa500aacb0 2024-03-28T23:31:20.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.472+0000 7faa5829d640 1 -- 172.21.15.161:0/2491881740 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa50150ec0 con 0x7faa500aacb0 2024-03-28T23:31:20.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.472+0000 7faa46ffd640 1 -- 172.21.15.161:0/2491881740 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4000ce10 con 0x7faa500aacb0 2024-03-28T23:31:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.472+0000 7faa5829d640 1 -- 172.21.15.161:0/2491881740 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa5014e950 con 0x7faa500aacb0 2024-03-28T23:31:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.473+0000 7faa5829d640 1 -- 172.21.15.161:0/2491881740 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa5004f4b0 con 0x7faa500aacb0 2024-03-28T23:31:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.474+0000 7faa46ffd640 1 -- 172.21.15.161:0/2491881740 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7faa4001a070 con 0x7faa500aacb0 2024-03-28T23:31:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.474+0000 7faa46ffd640 1 --2- 172.21.15.161:0/2491881740 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faa3406b4d0 0x7faa3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.474+0000 7faa46ffd640 1 -- 172.21.15.161:0/2491881740 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7faa4008a2b0 con 0x7faa500aacb0 2024-03-28T23:31:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.474+0000 7faa56039640 1 --2- 172.21.15.161:0/2491881740 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faa3406b4d0 0x7faa3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.475+0000 7faa56039640 1 --2- 172.21.15.161:0/2491881740 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faa3406b4d0 0x7faa3406d990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7faa4c035000 tx=0x7faa4c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.479+0000 7faa46ffd640 1 -- 172.21.15.161:0/2491881740 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7faa40058910 con 0x7faa500aacb0 2024-03-28T23:31:20.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.686+0000 7faa5829d640 1 -- 172.21.15.161:0/2491881740 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa50156350 con 0x7faa500aacb0 2024-03-28T23:31:20.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.687+0000 7faa46ffd640 1 -- 172.21.15.161:0/2491881740 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5182 (secure 0 0 0) 0x7faa4005c9b0 con 0x7faa500aacb0 2024-03-28T23:31:20.688 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:20.688 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5110,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/465771987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":465771987},{"type":"v1","addr":"172.21.15.170:6835","nonce":465771987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5355,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/306644174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":306644174},{"type":"v1","addr":"172.21.15.161:6836","nonce":306644174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1649814516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":1649814516},{"type":"v1","addr":"172.21.15.161:6837","nonce":1649814516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:30:57.616423+0000","modified":"2024-03-28T23:31:13.644123+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4853},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4853":{"gid":4853,"name":"d","rank":0,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.170:6834/11130685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":11130685},{"type":"v1","addr":"172.21.15.170:6834","nonce":11130685}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4853,"qdb_cluster":[4853]},"id":2}]} 2024-03-28T23:31:20.688 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 23 2024-03-28T23:31:20.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.689+0000 7faa5829d640 1 -- 172.21.15.161:0/2491881740 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faa3406b4d0 msgr2=0x7faa3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:20.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.689+0000 7faa5829d640 1 --2- 172.21.15.161:0/2491881740 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faa3406b4d0 0x7faa3406d990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7faa4c035000 tx=0x7faa4c008030 comp rx=0 tx=0).stop 2024-03-28T23:31:20.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.689+0000 7faa5829d640 1 -- 172.21.15.161:0/2491881740 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa500aacb0 msgr2=0x7faa501e9490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:20.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.689+0000 7faa5829d640 1 --2- 172.21.15.161:0/2491881740 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa500aacb0 0x7faa501e9490 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7faa4000e9d0 tx=0x7faa4000ee90 comp rx=0 tx=0).stop 2024-03-28T23:31:20.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.689+0000 7faa5829d640 1 -- 172.21.15.161:0/2491881740 shutdown_connections 2024-03-28T23:31:20.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.689+0000 7faa5829d640 1 --2- 172.21.15.161:0/2491881740 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faa3406b4d0 0x7faa3406d990 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.689+0000 7faa5829d640 1 --2- 172.21.15.161:0/2491881740 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa500aacb0 0x7faa501e9490 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.689+0000 7faa5829d640 1 --2- 172.21.15.161:0/2491881740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa50150600 0x7faa5014cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.689+0000 7faa5829d640 1 --2- 172.21.15.161:0/2491881740 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa500a8fc0 0x7faa501ecc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.689+0000 7faa5829d640 1 -- 172.21.15.161:0/2491881740 >> 172.21.15.161:0/2491881740 conn(0x7faa500c6670 msgr2=0x7faa50074020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:20.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.689+0000 7faa5829d640 1 -- 172.21.15.161:0/2491881740 shutdown_connections 2024-03-28T23:31:20.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.689+0000 7faa5829d640 1 -- 172.21.15.161:0/2491881740 wait complete. 2024-03-28T23:31:20.701 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:31:20.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.802+0000 7f79c94eb640 1 Processor -- start 2024-03-28T23:31:20.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.802+0000 7f79c94eb640 1 -- start start 2024-03-28T23:31:20.805 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.803+0000 7f79c94eb640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f79c4152a80 0x7f79c4155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:20.805 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.803+0000 7f79c94eb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f79c406cb80 0x7f79c40c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:20.805 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.803+0000 7f79c94eb640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79c406c3c0 con 0x7f79c406d9f0 2024-03-28T23:31:20.805 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.803+0000 7f79c94eb640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f79c4156250 con 0x7f79c406cb80 2024-03-28T23:31:20.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.803+0000 7f79c94eb640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f79c406c6f0 con 0x7f79c4152a80 2024-03-28T23:31:20.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.803+0000 7f79c37fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f79c406cb80 0x7f79c40c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:20.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.803+0000 7f79c37fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f79c406cb80 0x7f79c40c74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35264/0 (socket says 172.21.15.161:35264) 2024-03-28T23:31:20.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.803+0000 7f79c37fe640 1 -- 172.21.15.161:0/956776405 learned_addr learned my addr 172.21.15.161:0/956776405 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:20.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.803+0000 7f79c2ffd640 1 --2- 172.21.15.161:0/956776405 >> v2:172.21.15.170:3300/0 conn(0x7f79c4152a80 0x7f79c4155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:20.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.803+0000 7f79c37fe640 1 -- 172.21.15.161:0/956776405 >> v2:172.21.15.170:3300/0 conn(0x7f79c4152a80 msgr2=0x7f79c4155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:20.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.803+0000 7f79c37fe640 1 --2- 172.21.15.161:0/956776405 >> v2:172.21.15.170:3300/0 conn(0x7f79c4152a80 0x7f79c4155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.803+0000 7f79c37fe640 1 -- 172.21.15.161:0/956776405 >> v1:172.21.15.170:6789/0 conn(0x7f79c406d9f0 legacy=0x7f79c40c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:20.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.803+0000 7f79c37fe640 1 -- 172.21.15.161:0/956776405 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79c40c5b90 con 0x7f79c406cb80 2024-03-28T23:31:20.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.804+0000 7f79c37fe640 1 --2- 172.21.15.161:0/956776405 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79c406cb80 0x7f79c40c74c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f79c40c7ff0 tx=0x7f79bc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da84328f50197c9d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:20.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.804+0000 7f79c1ffb640 1 -- 172.21.15.161:0/956776405 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79bc02e070 con 0x7f79c406cb80 2024-03-28T23:31:20.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.804+0000 7f79c1ffb640 1 -- 172.21.15.161:0/956776405 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79bc037070 con 0x7f79c406cb80 2024-03-28T23:31:20.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.804+0000 7f79c1ffb640 1 -- 172.21.15.161:0/956776405 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79bc031a00 con 0x7f79c406cb80 2024-03-28T23:31:20.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.805+0000 7f79c94eb640 1 -- 172.21.15.161:0/956776405 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79c406cb80 msgr2=0x7f79c40c74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:20.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.805+0000 7f79c94eb640 1 --2- 172.21.15.161:0/956776405 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79c406cb80 0x7f79c40c74c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f79c40c7ff0 tx=0x7f79bc02b880 comp rx=0 tx=0).stop 2024-03-28T23:31:20.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.805+0000 7f79c94eb640 1 -- 172.21.15.161:0/956776405 shutdown_connections 2024-03-28T23:31:20.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.805+0000 7f79c94eb640 1 --2- 172.21.15.161:0/956776405 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79c406cb80 0x7f79c40c74c0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.805+0000 7f79c94eb640 1 --2- 172.21.15.161:0/956776405 >> v2:172.21.15.170:3300/0 conn(0x7f79c4152a80 0x7f79c4155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.805+0000 7f79c94eb640 1 -- 172.21.15.161:0/956776405 >> 172.21.15.161:0/956776405 conn(0x7f79c40ac720 msgr2=0x7f79c40acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:20.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.805+0000 7f79c94eb640 1 -- 172.21.15.161:0/956776405 shutdown_connections 2024-03-28T23:31:20.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.805+0000 7f79c94eb640 1 -- 172.21.15.161:0/956776405 wait complete. 2024-03-28T23:31:20.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.806+0000 7f79c94eb640 1 Processor -- start 2024-03-28T23:31:20.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.806+0000 7f79c94eb640 1 -- start start 2024-03-28T23:31:20.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.806+0000 7f79c94eb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79c406cb80 0x7f79c41590d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:20.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.807+0000 7f79c94eb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79c406d9f0 0x7f79c4154620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:20.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.807+0000 7f79c94eb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79c4152a80 0x7f79c4154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:20.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.807+0000 7f79c94eb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f79c406d010 con 0x7f79c4152a80 2024-03-28T23:31:20.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.807+0000 7f79c94eb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f79c40c58b0 con 0x7f79c406cb80 2024-03-28T23:31:20.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.807+0000 7f79c94eb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f79c406d190 con 0x7f79c406d9f0 2024-03-28T23:31:20.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.807+0000 7f79c37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79c4152a80 0x7f79c4154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:20.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.807+0000 7f79c37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79c4152a80 0x7f79c4154b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35280/0 (socket says 172.21.15.161:35280) 2024-03-28T23:31:20.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.807+0000 7f79c37fe640 1 -- 172.21.15.161:0/2603990093 learned_addr learned my addr 172.21.15.161:0/2603990093 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:20.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.807+0000 7f79c2ffd640 1 --2- 172.21.15.161:0/2603990093 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79c406cb80 0x7f79c41590d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:20.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.807+0000 7f79c27fc640 1 --2- 172.21.15.161:0/2603990093 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79c406d9f0 0x7f79c4154620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:20.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.807+0000 7f79c37fe640 1 -- 172.21.15.161:0/2603990093 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79c406d9f0 msgr2=0x7f79c4154620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:20.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.807+0000 7f79c37fe640 1 --2- 172.21.15.161:0/2603990093 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79c406d9f0 0x7f79c4154620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.807+0000 7f79c37fe640 1 -- 172.21.15.161:0/2603990093 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79c406cb80 msgr2=0x7f79c41590d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:20.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.807+0000 7f79c37fe640 1 --2- 172.21.15.161:0/2603990093 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79c406cb80 0x7f79c41590d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.807+0000 7f79c37fe640 1 -- 172.21.15.161:0/2603990093 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79c41eaca0 con 0x7f79c4152a80 2024-03-28T23:31:20.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.807+0000 7f79c37fe640 1 --2- 172.21.15.161:0/2603990093 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79c4152a80 0x7f79c4154b60 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f79bc0090b0 tx=0x7f79bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:20.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.807+0000 7f79ab7fe640 1 -- 172.21.15.161:0/2603990093 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79bc032670 con 0x7f79c4152a80 2024-03-28T23:31:20.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.807+0000 7f79ab7fe640 1 -- 172.21.15.161:0/2603990093 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79bc032830 con 0x7f79c4152a80 2024-03-28T23:31:20.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.807+0000 7f79c94eb640 1 -- 172.21.15.161:0/2603990093 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79c41eaf80 con 0x7f79c4152a80 2024-03-28T23:31:20.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.807+0000 7f79ab7fe640 1 -- 172.21.15.161:0/2603990093 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79bc032c20 con 0x7f79c4152a80 2024-03-28T23:31:20.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.807+0000 7f79c94eb640 1 -- 172.21.15.161:0/2603990093 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79c41eb540 con 0x7f79c4152a80 2024-03-28T23:31:20.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.808+0000 7f79c94eb640 1 -- 172.21.15.161:0/2603990093 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79c404f4b0 con 0x7f79c4152a80 2024-03-28T23:31:20.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.809+0000 7f79ab7fe640 1 -- 172.21.15.161:0/2603990093 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f79bc04c060 con 0x7f79c4152a80 2024-03-28T23:31:20.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.809+0000 7f79ab7fe640 1 --2- 172.21.15.161:0/2603990093 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f79b006b4d0 0x7f79b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:20.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.809+0000 7f79ab7fe640 1 -- 172.21.15.161:0/2603990093 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f79bc0abca0 con 0x7f79c4152a80 2024-03-28T23:31:20.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.809+0000 7f79c2ffd640 1 --2- 172.21.15.161:0/2603990093 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f79b006b4d0 0x7f79b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:20.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.810+0000 7f79c2ffd640 1 --2- 172.21.15.161:0/2603990093 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f79b006b4d0 0x7f79b006d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f79b4011340 tx=0x7f79b400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:20.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.814+0000 7f79ab7fe640 1 -- 172.21.15.161:0/2603990093 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f79bc040070 con 0x7f79c4152a80 2024-03-28T23:31:20.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.986+0000 7f79c94eb640 1 -- 172.21.15.161:0/2603990093 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f79c40ca3f0 con 0x7f79c4152a80 2024-03-28T23:31:20.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.987+0000 7f79ab7fe640 1 -- 172.21.15.161:0/2603990093 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+17109 (secure 0 0 0) 0x7f79bc07a380 con 0x7f79c4152a80 2024-03-28T23:31:20.988 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:20.988 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":39,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:31:07.634589+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:30:56.085611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-28T23:30:56.575631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6837/2636327316":"2024-03-29T23:31:07.634564+0000","172.21.15.161:6835/2636327316":"2024-03-29T23:31:07.634564+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:31:20.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.989+0000 7f79c94eb640 1 -- 172.21.15.161:0/2603990093 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f79b006b4d0 msgr2=0x7f79b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:20.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.989+0000 7f79c94eb640 1 --2- 172.21.15.161:0/2603990093 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f79b006b4d0 0x7f79b006d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f79b4011340 tx=0x7f79b400b040 comp rx=0 tx=0).stop 2024-03-28T23:31:20.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.989+0000 7f79c94eb640 1 -- 172.21.15.161:0/2603990093 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79c4152a80 msgr2=0x7f79c4154b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:20.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.989+0000 7f79c94eb640 1 --2- 172.21.15.161:0/2603990093 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79c4152a80 0x7f79c4154b60 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f79bc0090b0 tx=0x7f79bc0046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.989+0000 7f79c94eb640 1 -- 172.21.15.161:0/2603990093 shutdown_connections 2024-03-28T23:31:20.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.989+0000 7f79c94eb640 1 --2- 172.21.15.161:0/2603990093 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f79b006b4d0 0x7f79b006d990 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.989+0000 7f79c94eb640 1 --2- 172.21.15.161:0/2603990093 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79c4152a80 0x7f79c4154b60 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.989+0000 7f79c94eb640 1 --2- 172.21.15.161:0/2603990093 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79c406d9f0 0x7f79c4154620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.989+0000 7f79c94eb640 1 --2- 172.21.15.161:0/2603990093 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79c406cb80 0x7f79c41590d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:20.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.989+0000 7f79c94eb640 1 -- 172.21.15.161:0/2603990093 >> 172.21.15.161:0/2603990093 conn(0x7f79c40ac720 msgr2=0x7f79c4151d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:20.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.989+0000 7f79c94eb640 1 -- 172.21.15.161:0/2603990093 shutdown_connections 2024-03-28T23:31:20.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:20.989+0000 7f79c94eb640 1 -- 172.21.15.161:0/2603990093 wait complete. 2024-03-28T23:31:21.001 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-28T23:31:21.001 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:31:21.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.100+0000 7f6a8089a640 1 Processor -- start 2024-03-28T23:31:21.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.100+0000 7f6a8089a640 1 -- start start 2024-03-28T23:31:21.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.101+0000 7f6a8089a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6a7c06ce80 0x7f6a7c0a9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:21.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.101+0000 7f6a8089a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a7c06d310 con 0x7f6a7c14eca0 2024-03-28T23:31:21.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.101+0000 7f6a8089a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a7c151850 con 0x7f6a7c06ca20 2024-03-28T23:31:21.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.101+0000 7f6a8089a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6a7c152460 con 0x7f6a7c06ce80 2024-03-28T23:31:21.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.102+0000 7f6a7ad9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6a7c06ce80 0x7f6a7c0a9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:21.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.102+0000 7f6a7a59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6a7c14eca0 0x7f6a7c0c1880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:46064/0 (socket says 172.21.15.161:46064) 2024-03-28T23:31:21.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.102+0000 7f6a7a59c640 1 -- 172.21.15.161:0/2283037879 learned_addr learned my addr 172.21.15.161:0/2283037879 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:21.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.102+0000 7f6a7ad9d640 1 -- 172.21.15.161:0/2283037879 >> v1:172.21.15.170:6789/0 conn(0x7f6a7c06ca20 legacy=0x7f6a7c0a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:21.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.102+0000 7f6a7ad9d640 1 -- 172.21.15.161:0/2283037879 >> v1:172.21.15.161:6789/0 conn(0x7f6a7c14eca0 legacy=0x7f6a7c0c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:21.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.102+0000 7f6a7959a640 1 -- 172.21.15.161:0/2283037879 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1494073893 0 0) 0x7f6a70001110 con 0x7f6a7c14eca0 2024-03-28T23:31:21.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.102+0000 7f6a7ad9d640 1 -- 172.21.15.161:0/2283037879 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a7c151b30 con 0x7f6a7c06ce80 2024-03-28T23:31:21.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.103+0000 7f6a7ad9d640 1 --2- 172.21.15.161:0/2283037879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a7c06ce80 0x7f6a7c0a9fd0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f6a7400aea0 tx=0x7f6a7402cfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9f3e0ef6c9d55d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:21.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.103+0000 7f6a7959a640 1 -- 172.21.15.161:0/2283037879 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a7402db60 con 0x7f6a7c06ce80 2024-03-28T23:31:21.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.103+0000 7f6a7959a640 1 -- 172.21.15.161:0/2283037879 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a7402dd00 con 0x7f6a7c06ce80 2024-03-28T23:31:21.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.103+0000 7f6a7959a640 1 -- 172.21.15.161:0/2283037879 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a74008660 con 0x7f6a7c06ce80 2024-03-28T23:31:21.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.103+0000 7f6a8089a640 1 -- 172.21.15.161:0/2283037879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a7c06ce80 msgr2=0x7f6a7c0a9fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:21.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.103+0000 7f6a8089a640 1 --2- 172.21.15.161:0/2283037879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a7c06ce80 0x7f6a7c0a9fd0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f6a7400aea0 tx=0x7f6a7402cfc0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.103+0000 7f6a8089a640 1 -- 172.21.15.161:0/2283037879 shutdown_connections 2024-03-28T23:31:21.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.103+0000 7f6a8089a640 1 --2- 172.21.15.161:0/2283037879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a7c06ce80 0x7f6a7c0a9fd0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.103+0000 7f6a8089a640 1 -- 172.21.15.161:0/2283037879 >> 172.21.15.161:0/2283037879 conn(0x7f6a7c0c6670 msgr2=0x7f6a7c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:21.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.103+0000 7f6a8089a640 1 -- 172.21.15.161:0/2283037879 shutdown_connections 2024-03-28T23:31:21.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.103+0000 7f6a8089a640 1 -- 172.21.15.161:0/2283037879 wait complete. 2024-03-28T23:31:21.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.105+0000 7f6a8089a640 1 Processor -- start 2024-03-28T23:31:21.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.105+0000 7f6a8089a640 1 -- start start 2024-03-28T23:31:21.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.105+0000 7f6a8089a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a7c06ca20 0x7f6a7c0c1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:21.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.105+0000 7f6a8089a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a7c06ce80 0x7f6a7c0bc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:21.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.105+0000 7f6a8089a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a7c14eca0 0x7f6a7c14c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:21.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.105+0000 7f6a8089a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6a7c0aa510 con 0x7f6a7c06ca20 2024-03-28T23:31:21.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.105+0000 7f6a8089a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6a7c1520f0 con 0x7f6a7c06ce80 2024-03-28T23:31:21.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.105+0000 7f6a8089a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6a7c14f080 con 0x7f6a7c14eca0 2024-03-28T23:31:21.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.105+0000 7f6a7a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a7c06ca20 0x7f6a7c0c1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:21.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.105+0000 7f6a7a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a7c06ca20 0x7f6a7c0c1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35310/0 (socket says 172.21.15.161:35310) 2024-03-28T23:31:21.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.105+0000 7f6a7a59c640 1 -- 172.21.15.161:0/3412796190 learned_addr learned my addr 172.21.15.161:0/3412796190 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:21.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.105+0000 7f6a79d9b640 1 --2- 172.21.15.161:0/3412796190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a7c06ce80 0x7f6a7c0bc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:21.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.105+0000 7f6a7ad9d640 1 --2- 172.21.15.161:0/3412796190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a7c14eca0 0x7f6a7c14c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:21.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.106+0000 7f6a7a59c640 1 -- 172.21.15.161:0/3412796190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a7c14eca0 msgr2=0x7f6a7c14c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:21.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.106+0000 7f6a7a59c640 1 --2- 172.21.15.161:0/3412796190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a7c14eca0 0x7f6a7c14c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.106+0000 7f6a7a59c640 1 -- 172.21.15.161:0/3412796190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a7c06ce80 msgr2=0x7f6a7c0bc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:21.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.106+0000 7f6a7a59c640 1 --2- 172.21.15.161:0/3412796190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a7c06ce80 0x7f6a7c0bc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.106+0000 7f6a7a59c640 1 -- 172.21.15.161:0/3412796190 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a7c0bc050 con 0x7f6a7c06ca20 2024-03-28T23:31:21.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.106+0000 7f6a7a59c640 1 --2- 172.21.15.161:0/3412796190 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a7c06ca20 0x7f6a7c0c1ad0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6a7000c780 tx=0x7f6a7000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:21.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.106+0000 7f6a6affd640 1 -- 172.21.15.161:0/3412796190 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a70018070 con 0x7f6a7c06ca20 2024-03-28T23:31:21.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.106+0000 7f6a6affd640 1 -- 172.21.15.161:0/3412796190 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a700072d0 con 0x7f6a7c06ca20 2024-03-28T23:31:21.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.106+0000 7f6a8089a640 1 -- 172.21.15.161:0/3412796190 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a7c0bbd70 con 0x7f6a7c06ca20 2024-03-28T23:31:21.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.106+0000 7f6a6affd640 1 -- 172.21.15.161:0/3412796190 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a700047e0 con 0x7f6a7c06ca20 2024-03-28T23:31:21.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.106+0000 7f6a8089a640 1 -- 172.21.15.161:0/3412796190 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a7c1553e0 con 0x7f6a7c06ca20 2024-03-28T23:31:21.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.107+0000 7f6a8089a640 1 -- 172.21.15.161:0/3412796190 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a7c04f4b0 con 0x7f6a7c06ca20 2024-03-28T23:31:21.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.107+0000 7f6a6affd640 1 -- 172.21.15.161:0/3412796190 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6a7001b070 con 0x7f6a7c06ca20 2024-03-28T23:31:21.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.108+0000 7f6a6affd640 1 --2- 172.21.15.161:0/3412796190 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6a5c06b4d0 0x7f6a5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:21.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.108+0000 7f6a6affd640 1 -- 172.21.15.161:0/3412796190 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f6a70089570 con 0x7f6a7c06ca20 2024-03-28T23:31:21.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.108+0000 7f6a79d9b640 1 --2- 172.21.15.161:0/3412796190 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6a5c06b4d0 0x7f6a5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:21.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.108+0000 7f6a79d9b640 1 --2- 172.21.15.161:0/3412796190 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6a5c06b4d0 0x7f6a5c06d990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c0c2380 tx=0x7f6a6c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:21.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.112+0000 7f6a6affd640 1 -- 172.21.15.161:0/3412796190 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6a7c06d310 con 0x7f6a7c06ca20 2024-03-28T23:31:21.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.324+0000 7f6a8089a640 1 -- 172.21.15.161:0/3412796190 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a7c151260 con 0x7f6a7c06ca20 2024-03-28T23:31:21.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.324+0000 7f6a6affd640 1 -- 172.21.15.161:0/3412796190 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5182 (secure 0 0 0) 0x7f6a70007d00 con 0x7f6a7c06ca20 2024-03-28T23:31:21.325 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:21.325 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5110,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/465771987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":465771987},{"type":"v1","addr":"172.21.15.170:6835","nonce":465771987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5355,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/306644174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":306644174},{"type":"v1","addr":"172.21.15.161:6836","nonce":306644174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1649814516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":1649814516},{"type":"v1","addr":"172.21.15.161:6837","nonce":1649814516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:30:57.616423+0000","modified":"2024-03-28T23:31:13.644123+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4853},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4853":{"gid":4853,"name":"d","rank":0,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.170:6834/11130685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":11130685},{"type":"v1","addr":"172.21.15.170:6834","nonce":11130685}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4853,"qdb_cluster":[4853]},"id":2}]} 2024-03-28T23:31:21.326 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 23 2024-03-28T23:31:21.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.326+0000 7f6a8089a640 1 -- 172.21.15.161:0/3412796190 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6a5c06b4d0 msgr2=0x7f6a5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:21.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.326+0000 7f6a8089a640 1 --2- 172.21.15.161:0/3412796190 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6a5c06b4d0 0x7f6a5c06d990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c0c2380 tx=0x7f6a6c009040 comp rx=0 tx=0).stop 2024-03-28T23:31:21.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.326+0000 7f6a8089a640 1 -- 172.21.15.161:0/3412796190 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a7c06ca20 msgr2=0x7f6a7c0c1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:21.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.326+0000 7f6a8089a640 1 --2- 172.21.15.161:0/3412796190 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a7c06ca20 0x7f6a7c0c1ad0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6a7000c780 tx=0x7f6a7000cc40 comp rx=0 tx=0).stop 2024-03-28T23:31:21.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.326+0000 7f6a8089a640 1 -- 172.21.15.161:0/3412796190 shutdown_connections 2024-03-28T23:31:21.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.326+0000 7f6a8089a640 1 --2- 172.21.15.161:0/3412796190 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6a5c06b4d0 0x7f6a5c06d990 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.326+0000 7f6a8089a640 1 --2- 172.21.15.161:0/3412796190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a7c14eca0 0x7f6a7c14c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.326+0000 7f6a8089a640 1 --2- 172.21.15.161:0/3412796190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a7c06ce80 0x7f6a7c0bc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.326+0000 7f6a8089a640 1 --2- 172.21.15.161:0/3412796190 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a7c06ca20 0x7f6a7c0c1ad0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.326+0000 7f6a8089a640 1 -- 172.21.15.161:0/3412796190 >> 172.21.15.161:0/3412796190 conn(0x7f6a7c0c6670 msgr2=0x7f6a7c0aa8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:21.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.327+0000 7f6a8089a640 1 -- 172.21.15.161:0/3412796190 shutdown_connections 2024-03-28T23:31:21.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.327+0000 7f6a8089a640 1 -- 172.21.15.161:0/3412796190 wait complete. 2024-03-28T23:31:21.339 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:31:21.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.438+0000 7f4483dc9640 1 Processor -- start 2024-03-28T23:31:21.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.439+0000 7f4483dc9640 1 -- start start 2024-03-28T23:31:21.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.439+0000 7f4483dc9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f447c150600 0x7f447c0a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:21.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.439+0000 7f4483dc9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f447c0aacb0 0x7f447c153980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:21.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.439+0000 7f4483dc9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f447c1567c0 con 0x7f447c0a8fc0 2024-03-28T23:31:21.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.439+0000 7f4483dc9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f447c156570 con 0x7f447c150600 2024-03-28T23:31:21.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.439+0000 7f4483dc9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f447c156aa0 con 0x7f447c0aacb0 2024-03-28T23:31:21.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.440+0000 7f4481b65640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f447c150600 0x7f447c0a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:21.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.440+0000 7f4481364640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f447c0a8fc0 0x7f447c153270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:46076/0 (socket says 172.21.15.161:46076) 2024-03-28T23:31:21.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.440+0000 7f4481364640 1 -- 172.21.15.161:0/2075734567 learned_addr learned my addr 172.21.15.161:0/2075734567 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:21.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.440+0000 7f4482366640 1 --2- 172.21.15.161:0/2075734567 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f447c0aacb0 0x7f447c153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:21.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.440+0000 7f4480b63640 1 -- 172.21.15.161:0/2075734567 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2370296496 0 0) 0x7f447c1567c0 con 0x7f447c0a8fc0 2024-03-28T23:31:21.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.440+0000 7f4480b63640 1 -- 172.21.15.161:0/2075734567 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4468003520 con 0x7f447c0a8fc0 2024-03-28T23:31:21.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.440+0000 7f4481b65640 1 -- 172.21.15.161:0/2075734567 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f447c0aacb0 msgr2=0x7f447c153980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:21.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.440+0000 7f4481b65640 1 --2- 172.21.15.161:0/2075734567 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f447c0aacb0 0x7f447c153980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.440+0000 7f4481b65640 1 -- 172.21.15.161:0/2075734567 >> v1:172.21.15.161:6789/0 conn(0x7f447c0a8fc0 legacy=0x7f447c153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:21.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.440+0000 7f4481b65640 1 -- 172.21.15.161:0/2075734567 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f447c156d80 con 0x7f447c150600 2024-03-28T23:31:21.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.440+0000 7f4482366640 1 --2- 172.21.15.161:0/2075734567 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f447c0aacb0 0x7f447c153980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:21.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.441+0000 7f4481b65640 1 --2- 172.21.15.161:0/2075734567 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f447c150600 0x7f447c0a9f40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f446c0097a0 tx=0x7f446c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ca1b21a64e366f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:21.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.441+0000 7f4480b63640 1 -- 172.21.15.161:0/2075734567 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446c0315a0 con 0x7f447c150600 2024-03-28T23:31:21.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.441+0000 7f4480b63640 1 -- 172.21.15.161:0/2075734567 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f446c037070 con 0x7f447c150600 2024-03-28T23:31:21.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.441+0000 7f4480b63640 1 -- 172.21.15.161:0/2075734567 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446c031960 con 0x7f447c150600 2024-03-28T23:31:21.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.441+0000 7f4483dc9640 1 -- 172.21.15.161:0/2075734567 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f447c150600 msgr2=0x7f447c0a9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:21.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.441+0000 7f4483dc9640 1 --2- 172.21.15.161:0/2075734567 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f447c150600 0x7f447c0a9f40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f446c0097a0 tx=0x7f446c02b840 comp rx=0 tx=0).stop 2024-03-28T23:31:21.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.441+0000 7f4483dc9640 1 -- 172.21.15.161:0/2075734567 shutdown_connections 2024-03-28T23:31:21.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.441+0000 7f4483dc9640 1 --2- 172.21.15.161:0/2075734567 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f447c0aacb0 0x7f447c153980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.441+0000 7f4483dc9640 1 --2- 172.21.15.161:0/2075734567 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f447c150600 0x7f447c0a9f40 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.441+0000 7f4483dc9640 1 -- 172.21.15.161:0/2075734567 >> 172.21.15.161:0/2075734567 conn(0x7f447c0c6670 msgr2=0x7f447c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:21.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.441+0000 7f4483dc9640 1 -- 172.21.15.161:0/2075734567 shutdown_connections 2024-03-28T23:31:21.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.441+0000 7f4483dc9640 1 -- 172.21.15.161:0/2075734567 wait complete. 2024-03-28T23:31:21.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.442+0000 7f4483dc9640 1 Processor -- start 2024-03-28T23:31:21.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.443+0000 7f4483dc9640 1 -- start start 2024-03-28T23:31:21.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.443+0000 7f4483dc9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f447c0a8fc0 0x7f447c14afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:21.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.443+0000 7f4483dc9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f447c0aacb0 0x7f447c14b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:21.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.443+0000 7f4483dc9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f447c150600 0x7f447c155120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:21.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.443+0000 7f4483dc9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f447c153ec0 con 0x7f447c0a8fc0 2024-03-28T23:31:21.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.443+0000 7f4483dc9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f447c157340 con 0x7f447c0aacb0 2024-03-28T23:31:21.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.443+0000 7f4483dc9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f447c1509e0 con 0x7f447c150600 2024-03-28T23:31:21.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.443+0000 7f4481b65640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f447c0a8fc0 0x7f447c14afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:21.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.443+0000 7f4481b65640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f447c0a8fc0 0x7f447c14afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35312/0 (socket says 172.21.15.161:35312) 2024-03-28T23:31:21.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.443+0000 7f4481b65640 1 -- 172.21.15.161:0/1671154908 learned_addr learned my addr 172.21.15.161:0/1671154908 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:21.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.443+0000 7f4481364640 1 --2- 172.21.15.161:0/1671154908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f447c0aacb0 0x7f447c14b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:21.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.443+0000 7f4482366640 1 --2- 172.21.15.161:0/1671154908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f447c150600 0x7f447c155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:21.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.444+0000 7f4481b65640 1 -- 172.21.15.161:0/1671154908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f447c150600 msgr2=0x7f447c155120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:21.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.444+0000 7f4481b65640 1 --2- 172.21.15.161:0/1671154908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f447c150600 0x7f447c155120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.444+0000 7f4481b65640 1 -- 172.21.15.161:0/1671154908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f447c0aacb0 msgr2=0x7f447c14b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:21.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.444+0000 7f4481b65640 1 --2- 172.21.15.161:0/1671154908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f447c0aacb0 0x7f447c14b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.444+0000 7f4481b65640 1 -- 172.21.15.161:0/1671154908 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f447c154960 con 0x7f447c0a8fc0 2024-03-28T23:31:21.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.444+0000 7f4481b65640 1 --2- 172.21.15.161:0/1671154908 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f447c0a8fc0 0x7f447c14afd0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f446c0098c0 tx=0x7f446c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:21.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.444+0000 7f44727fc640 1 -- 172.21.15.161:0/1671154908 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446c032630 con 0x7f447c0a8fc0 2024-03-28T23:31:21.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.444+0000 7f44727fc640 1 -- 172.21.15.161:0/1671154908 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f446c0327f0 con 0x7f447c0a8fc0 2024-03-28T23:31:21.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.444+0000 7f4483dc9640 1 -- 172.21.15.161:0/1671154908 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f447c14a910 con 0x7f447c0a8fc0 2024-03-28T23:31:21.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.444+0000 7f44727fc640 1 -- 172.21.15.161:0/1671154908 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446c032be0 con 0x7f447c0a8fc0 2024-03-28T23:31:21.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.444+0000 7f4483dc9640 1 -- 172.21.15.161:0/1671154908 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f447c1eee90 con 0x7f447c0a8fc0 2024-03-28T23:31:21.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.445+0000 7f4483dc9640 1 -- 172.21.15.161:0/1671154908 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f447c04f4b0 con 0x7f447c0a8fc0 2024-03-28T23:31:21.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.445+0000 7f44727fc640 1 -- 172.21.15.161:0/1671154908 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f446c03b3f0 con 0x7f447c0a8fc0 2024-03-28T23:31:21.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.446+0000 7f44727fc640 1 --2- 172.21.15.161:0/1671154908 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f446006b4d0 0x7f446006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:21.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.446+0000 7f4481364640 1 --2- 172.21.15.161:0/1671154908 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f446006b4d0 0x7f446006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:21.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.446+0000 7f44727fc640 1 -- 172.21.15.161:0/1671154908 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f446c02f030 con 0x7f447c0a8fc0 2024-03-28T23:31:21.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.446+0000 7f4481364640 1 --2- 172.21.15.161:0/1671154908 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f446006b4d0 0x7f446006d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f447c0c5560 tx=0x7f4478009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:21.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.450+0000 7f44727fc640 1 -- 172.21.15.161:0/1671154908 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f446c040070 con 0x7f447c0a8fc0 2024-03-28T23:31:21.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.660+0000 7f4483dc9640 1 -- 172.21.15.161:0/1671154908 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f447c153100 con 0x7f447c0a8fc0 2024-03-28T23:31:21.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.660+0000 7f44727fc640 1 -- 172.21.15.161:0/1671154908 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5182 (secure 0 0 0) 0x7f446c07aed0 con 0x7f447c0a8fc0 2024-03-28T23:31:21.661 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:21.661 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5110,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/465771987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":465771987},{"type":"v1","addr":"172.21.15.170:6835","nonce":465771987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5355,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/306644174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":306644174},{"type":"v1","addr":"172.21.15.161:6836","nonce":306644174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1649814516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":1649814516},{"type":"v1","addr":"172.21.15.161:6837","nonce":1649814516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:30:57.616423+0000","modified":"2024-03-28T23:31:13.644123+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4853},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4853":{"gid":4853,"name":"d","rank":0,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.170:6834/11130685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":11130685},{"type":"v1","addr":"172.21.15.170:6834","nonce":11130685}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4853,"qdb_cluster":[4853]},"id":2}]} 2024-03-28T23:31:21.662 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 23 2024-03-28T23:31:21.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.662+0000 7f4483dc9640 1 -- 172.21.15.161:0/1671154908 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f446006b4d0 msgr2=0x7f446006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:21.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.662+0000 7f4483dc9640 1 --2- 172.21.15.161:0/1671154908 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f446006b4d0 0x7f446006d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f447c0c5560 tx=0x7f4478009040 comp rx=0 tx=0).stop 2024-03-28T23:31:21.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.662+0000 7f4483dc9640 1 -- 172.21.15.161:0/1671154908 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f447c0a8fc0 msgr2=0x7f447c14afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:21.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.662+0000 7f4483dc9640 1 --2- 172.21.15.161:0/1671154908 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f447c0a8fc0 0x7f447c14afd0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f446c0098c0 tx=0x7f446c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.663+0000 7f4483dc9640 1 -- 172.21.15.161:0/1671154908 shutdown_connections 2024-03-28T23:31:21.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.663+0000 7f4483dc9640 1 --2- 172.21.15.161:0/1671154908 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f446006b4d0 0x7f446006d990 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.663+0000 7f4483dc9640 1 --2- 172.21.15.161:0/1671154908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f447c150600 0x7f447c155120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.663+0000 7f4483dc9640 1 --2- 172.21.15.161:0/1671154908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f447c0aacb0 0x7f447c14b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.663+0000 7f4483dc9640 1 --2- 172.21.15.161:0/1671154908 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f447c0a8fc0 0x7f447c14afd0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.663+0000 7f4483dc9640 1 -- 172.21.15.161:0/1671154908 >> 172.21.15.161:0/1671154908 conn(0x7f447c0c6670 msgr2=0x7f447c0bfa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:21.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.663+0000 7f4483dc9640 1 -- 172.21.15.161:0/1671154908 shutdown_connections 2024-03-28T23:31:21.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.663+0000 7f4483dc9640 1 -- 172.21.15.161:0/1671154908 wait complete. 2024-03-28T23:31:21.676 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:31:21.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.778+0000 7ff45448f640 1 Processor -- start 2024-03-28T23:31:21.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.778+0000 7ff45448f640 1 -- start start 2024-03-28T23:31:21.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.779+0000 7ff45448f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff44c14be70 0x7ff44c0ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:21.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.779+0000 7ff45448f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff44c0ab6e0 0x7ff44c0aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:21.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.779+0000 7ff45448f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff44c14c250 con 0x7ff44c06ce80 2024-03-28T23:31:21.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.779+0000 7ff45448f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff44c1529c0 con 0x7ff44c0ab6e0 2024-03-28T23:31:21.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.779+0000 7ff45448f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7ff44c0a9dc0 con 0x7ff44c14be70 2024-03-28T23:31:21.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.779+0000 7ff451a2a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff44c0ab6e0 0x7ff44c0aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:21.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.779+0000 7ff451a2a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff44c0ab6e0 0x7ff44c0aa5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35320/0 (socket says 172.21.15.161:35320) 2024-03-28T23:31:21.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.779+0000 7ff451a2a640 1 -- 172.21.15.161:0/2001461032 learned_addr learned my addr 172.21.15.161:0/2001461032 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:21.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.779+0000 7ff45222b640 1 --2- 172.21.15.161:0/2001461032 >> v2:172.21.15.170:3300/0 conn(0x7ff44c14be70 0x7ff44c0ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:21.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.780+0000 7ff451a2a640 1 -- 172.21.15.161:0/2001461032 >> v2:172.21.15.170:3300/0 conn(0x7ff44c14be70 msgr2=0x7ff44c0ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:21.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.780+0000 7ff451a2a640 1 --2- 172.21.15.161:0/2001461032 >> v2:172.21.15.170:3300/0 conn(0x7ff44c14be70 0x7ff44c0ab1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.780+0000 7ff451a2a640 1 -- 172.21.15.161:0/2001461032 >> v1:172.21.15.170:6789/0 conn(0x7ff44c06ce80 legacy=0x7ff44c1553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:21.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.780+0000 7ff451a2a640 1 -- 172.21.15.161:0/2001461032 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff44c0aa0a0 con 0x7ff44c0ab6e0 2024-03-28T23:31:21.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.780+0000 7ff451a2a640 1 --2- 172.21.15.161:0/2001461032 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff44c0ab6e0 0x7ff44c0aa5c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff44c0c7520 tx=0x7ff43c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4dd3d5ffe98576 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:21.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.781+0000 7ff451229640 1 -- 172.21.15.161:0/2001461032 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43c035020 con 0x7ff44c0ab6e0 2024-03-28T23:31:21.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.781+0000 7ff451229640 1 -- 172.21.15.161:0/2001461032 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff43c031870 con 0x7ff44c0ab6e0 2024-03-28T23:31:21.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.781+0000 7ff451229640 1 -- 172.21.15.161:0/2001461032 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43c031c40 con 0x7ff44c0ab6e0 2024-03-28T23:31:21.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.781+0000 7ff45448f640 1 -- 172.21.15.161:0/2001461032 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff44c0ab6e0 msgr2=0x7ff44c0aa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:21.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.781+0000 7ff45448f640 1 --2- 172.21.15.161:0/2001461032 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff44c0ab6e0 0x7ff44c0aa5c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff44c0c7520 tx=0x7ff43c02b910 comp rx=0 tx=0).stop 2024-03-28T23:31:21.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.781+0000 7ff45448f640 1 -- 172.21.15.161:0/2001461032 shutdown_connections 2024-03-28T23:31:21.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.781+0000 7ff45448f640 1 --2- 172.21.15.161:0/2001461032 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff44c0ab6e0 0x7ff44c0aa5c0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.781+0000 7ff45448f640 1 --2- 172.21.15.161:0/2001461032 >> v2:172.21.15.170:3300/0 conn(0x7ff44c14be70 0x7ff44c0ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.781+0000 7ff45448f640 1 -- 172.21.15.161:0/2001461032 >> 172.21.15.161:0/2001461032 conn(0x7ff44c0c6670 msgr2=0x7ff44c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:21.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.781+0000 7ff45448f640 1 -- 172.21.15.161:0/2001461032 shutdown_connections 2024-03-28T23:31:21.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.781+0000 7ff45448f640 1 -- 172.21.15.161:0/2001461032 wait complete. 2024-03-28T23:31:21.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.782+0000 7ff45448f640 1 Processor -- start 2024-03-28T23:31:21.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.782+0000 7ff45448f640 1 -- start start 2024-03-28T23:31:21.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.783+0000 7ff45448f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff44c06ce80 0x7ff44c072c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:21.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.783+0000 7ff45448f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff44c0ab6e0 0x7ff44c1537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:21.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.783+0000 7ff45448f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff44c14be70 0x7ff44c153ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:21.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.783+0000 7ff45448f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff44c06b2d0 con 0x7ff44c06ce80 2024-03-28T23:31:21.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.783+0000 7ff45448f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff44c06b150 con 0x7ff44c0ab6e0 2024-03-28T23:31:21.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.783+0000 7ff45448f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff44c0c1520 con 0x7ff44c14be70 2024-03-28T23:31:21.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.783+0000 7ff45222b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff44c06ce80 0x7ff44c072c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:21.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.783+0000 7ff45222b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff44c06ce80 0x7ff44c072c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35332/0 (socket says 172.21.15.161:35332) 2024-03-28T23:31:21.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.783+0000 7ff45222b640 1 -- 172.21.15.161:0/2983092065 learned_addr learned my addr 172.21.15.161:0/2983092065 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:21.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.783+0000 7ff451a2a640 1 --2- 172.21.15.161:0/2983092065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff44c0ab6e0 0x7ff44c1537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:21.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.783+0000 7ff452a2c640 1 --2- 172.21.15.161:0/2983092065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff44c14be70 0x7ff44c153ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:21.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.783+0000 7ff45222b640 1 -- 172.21.15.161:0/2983092065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff44c14be70 msgr2=0x7ff44c153ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:21.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.783+0000 7ff45222b640 1 --2- 172.21.15.161:0/2983092065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff44c14be70 0x7ff44c153ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.783+0000 7ff45222b640 1 -- 172.21.15.161:0/2983092065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff44c0ab6e0 msgr2=0x7ff44c1537a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:21.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.783+0000 7ff45222b640 1 --2- 172.21.15.161:0/2983092065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff44c0ab6e0 0x7ff44c1537a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.783+0000 7ff45222b640 1 -- 172.21.15.161:0/2983092065 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff44c151570 con 0x7ff44c06ce80 2024-03-28T23:31:21.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.783+0000 7ff45222b640 1 --2- 172.21.15.161:0/2983092065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff44c06ce80 0x7ff44c072c10 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff44400e9b0 tx=0x7ff44400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:21.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.784+0000 7ff442ffd640 1 -- 172.21.15.161:0/2983092065 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff444017070 con 0x7ff44c06ce80 2024-03-28T23:31:21.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.784+0000 7ff442ffd640 1 -- 172.21.15.161:0/2983092065 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff44400c9a0 con 0x7ff44c06ce80 2024-03-28T23:31:21.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.784+0000 7ff442ffd640 1 -- 172.21.15.161:0/2983092065 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44400cd90 con 0x7ff44c06ce80 2024-03-28T23:31:21.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.784+0000 7ff45448f640 1 -- 172.21.15.161:0/2983092065 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff44c1509d0 con 0x7ff44c06ce80 2024-03-28T23:31:21.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.784+0000 7ff45448f640 1 -- 172.21.15.161:0/2983092065 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff44c155030 con 0x7ff44c06ce80 2024-03-28T23:31:21.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.785+0000 7ff45448f640 1 -- 172.21.15.161:0/2983092065 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff44c14b9b0 con 0x7ff44c06ce80 2024-03-28T23:31:21.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.785+0000 7ff442ffd640 1 -- 172.21.15.161:0/2983092065 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff44401a070 con 0x7ff44c06ce80 2024-03-28T23:31:21.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.785+0000 7ff442ffd640 1 --2- 172.21.15.161:0/2983092065 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff43006b4d0 0x7ff43006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:21.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.785+0000 7ff442ffd640 1 -- 172.21.15.161:0/2983092065 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7ff44408a270 con 0x7ff44c06ce80 2024-03-28T23:31:21.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.785+0000 7ff451a2a640 1 --2- 172.21.15.161:0/2983092065 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff43006b4d0 0x7ff43006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:21.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.786+0000 7ff451a2a640 1 --2- 172.21.15.161:0/2983092065 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff43006b4d0 0x7ff43006d990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff43c036000 tx=0x7ff43c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:21.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.790+0000 7ff442ffd640 1 -- 172.21.15.161:0/2983092065 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff444058950 con 0x7ff44c06ce80 2024-03-28T23:31:21.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.962+0000 7ff45448f640 1 -- 172.21.15.161:0/2983092065 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff44c04f4b0 con 0x7ff44c06ce80 2024-03-28T23:31:21.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.963+0000 7ff442ffd640 1 -- 172.21.15.161:0/2983092065 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+17109 (secure 0 0 0) 0x7ff44405c9f0 con 0x7ff44c06ce80 2024-03-28T23:31:21.964 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:21.964 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":39,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:31:07.634589+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:30:56.085611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-28T23:30:56.575631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6837/2636327316":"2024-03-29T23:31:07.634564+0000","172.21.15.161:6835/2636327316":"2024-03-29T23:31:07.634564+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:31:21.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.965+0000 7ff45448f640 1 -- 172.21.15.161:0/2983092065 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff43006b4d0 msgr2=0x7ff43006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:21.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.965+0000 7ff45448f640 1 --2- 172.21.15.161:0/2983092065 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff43006b4d0 0x7ff43006d990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff43c036000 tx=0x7ff43c03c040 comp rx=0 tx=0).stop 2024-03-28T23:31:21.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.965+0000 7ff45448f640 1 -- 172.21.15.161:0/2983092065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff44c06ce80 msgr2=0x7ff44c072c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:21.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.965+0000 7ff45448f640 1 --2- 172.21.15.161:0/2983092065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff44c06ce80 0x7ff44c072c10 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff44400e9b0 tx=0x7ff44400ee70 comp rx=0 tx=0).stop 2024-03-28T23:31:21.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.965+0000 7ff45448f640 1 -- 172.21.15.161:0/2983092065 shutdown_connections 2024-03-28T23:31:21.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.965+0000 7ff45448f640 1 --2- 172.21.15.161:0/2983092065 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff43006b4d0 0x7ff43006d990 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.965+0000 7ff45448f640 1 --2- 172.21.15.161:0/2983092065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff44c14be70 0x7ff44c153ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.965+0000 7ff45448f640 1 --2- 172.21.15.161:0/2983092065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff44c0ab6e0 0x7ff44c1537a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.965+0000 7ff45448f640 1 --2- 172.21.15.161:0/2983092065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff44c06ce80 0x7ff44c072c10 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:21.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.965+0000 7ff45448f640 1 -- 172.21.15.161:0/2983092065 >> 172.21.15.161:0/2983092065 conn(0x7ff44c0c6670 msgr2=0x7ff44c0bfa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:21.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.965+0000 7ff45448f640 1 -- 172.21.15.161:0/2983092065 shutdown_connections 2024-03-28T23:31:21.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:21.965+0000 7ff45448f640 1 -- 172.21.15.161:0/2983092065 wait complete. 2024-03-28T23:31:21.977 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-28T23:31:22.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.077+0000 7f7ddb710640 1 Processor -- start 2024-03-28T23:31:22.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.077+0000 7f7ddb710640 1 -- start start 2024-03-28T23:31:22.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.078+0000 7f7ddb710640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7dd406ce80 0x7f7dd40aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:22.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.078+0000 7f7ddb710640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7dd40a9680 0x7f7dd40aab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:22.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.078+0000 7f7ddb710640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7dd414c250 con 0x7f7dd414be70 2024-03-28T23:31:22.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.078+0000 7f7ddb710640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f7dd41529c0 con 0x7f7dd406ce80 2024-03-28T23:31:22.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.078+0000 7f7ddb710640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7dd40a9dc0 con 0x7f7dd40a9680 2024-03-28T23:31:22.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.078+0000 7f7dd9cad640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7dd40a9680 0x7f7dd40aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:22.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.078+0000 7f7dd8cab640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7dd406ce80 0x7f7dd40aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:22.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.078+0000 7f7dd9cad640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7dd40a9680 0x7f7dd40aab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:44872/0 (socket says 172.21.15.161:44872) 2024-03-28T23:31:22.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.078+0000 7f7dd8cab640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7dd406ce80 0x7f7dd40aa5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:56566/0 (socket says 172.21.15.161:56566) 2024-03-28T23:31:22.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.078+0000 7f7dd94ac640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f7dd414be70 0x7f7dd40ab1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:54480/0 (socket says 172.21.15.161:54480) 2024-03-28T23:31:22.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.078+0000 7f7dd9cad640 1 -- 172.21.15.161:0/2738865476 learned_addr learned my addr 172.21.15.161:0/2738865476 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:22.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.078+0000 7f7dd9cad640 1 -- 172.21.15.161:0/2738865476 >> v2:172.21.15.170:3300/0 conn(0x7f7dd406ce80 msgr2=0x7f7dd40aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:22.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.078+0000 7f7dd9cad640 1 --2- 172.21.15.161:0/2738865476 >> v2:172.21.15.170:3300/0 conn(0x7f7dd406ce80 0x7f7dd40aa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:22.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.078+0000 7f7dd9cad640 1 -- 172.21.15.161:0/2738865476 >> v1:172.21.15.170:6789/0 conn(0x7f7dd414be70 legacy=0x7f7dd40ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:22.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.079+0000 7f7dd9cad640 1 -- 172.21.15.161:0/2738865476 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dd40aa0a0 con 0x7f7dd40a9680 2024-03-28T23:31:22.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.079+0000 7f7dd9cad640 1 --2- 172.21.15.161:0/2738865476 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7dd40a9680 0x7f7dd40aab00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc00d150 tx=0x7f7dcc02e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da697d739db4100f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:22.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.079+0000 7f7dcbfff640 1 -- 172.21.15.161:0/2738865476 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc02e9e0 con 0x7f7dd40a9680 2024-03-28T23:31:22.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.079+0000 7f7dcbfff640 1 -- 172.21.15.161:0/2738865476 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dcc02eb80 con 0x7f7dd40a9680 2024-03-28T23:31:22.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.079+0000 7f7dcbfff640 1 -- 172.21.15.161:0/2738865476 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc009600 con 0x7f7dd40a9680 2024-03-28T23:31:22.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.080+0000 7f7ddb710640 1 -- 172.21.15.161:0/2738865476 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7dd40a9680 msgr2=0x7f7dd40aab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:22.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.080+0000 7f7ddb710640 1 --2- 172.21.15.161:0/2738865476 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7dd40a9680 0x7f7dd40aab00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc00d150 tx=0x7f7dcc02e040 comp rx=0 tx=0).stop 2024-03-28T23:31:22.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.080+0000 7f7ddb710640 1 -- 172.21.15.161:0/2738865476 shutdown_connections 2024-03-28T23:31:22.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.080+0000 7f7ddb710640 1 --2- 172.21.15.161:0/2738865476 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7dd40a9680 0x7f7dd40aab00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:22.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.080+0000 7f7ddb710640 1 --2- 172.21.15.161:0/2738865476 >> v2:172.21.15.170:3300/0 conn(0x7f7dd406ce80 0x7f7dd40aa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:22.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.080+0000 7f7ddb710640 1 -- 172.21.15.161:0/2738865476 >> 172.21.15.161:0/2738865476 conn(0x7f7dd40c6670 msgr2=0x7f7dd40c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:22.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.080+0000 7f7ddb710640 1 -- 172.21.15.161:0/2738865476 shutdown_connections 2024-03-28T23:31:22.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.080+0000 7f7ddb710640 1 -- 172.21.15.161:0/2738865476 wait complete. 2024-03-28T23:31:22.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.081+0000 7f7ddb710640 1 Processor -- start 2024-03-28T23:31:22.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.081+0000 7f7ddb710640 1 -- start start 2024-03-28T23:31:22.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.082+0000 7f7ddb710640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7dd406ce80 0x7f7dd41e9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:22.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.082+0000 7f7ddb710640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7dd40a9680 0x7f7dd4148430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:22.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.082+0000 7f7ddb710640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7dd414be70 0x7f7dd4148970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:22.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.082+0000 7f7ddb710640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd40aba30 con 0x7f7dd40a9680 2024-03-28T23:31:22.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.082+0000 7f7ddb710640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd40ab8b0 con 0x7f7dd414be70 2024-03-28T23:31:22.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.082+0000 7f7ddb710640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7dd406b150 con 0x7f7dd406ce80 2024-03-28T23:31:22.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.082+0000 7f7dd8cab640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7dd40a9680 0x7f7dd4148430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:22.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.082+0000 7f7dd8cab640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7dd40a9680 0x7f7dd4148430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35344/0 (socket says 172.21.15.161:35344) 2024-03-28T23:31:22.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.082+0000 7f7dd8cab640 1 -- 172.21.15.161:0/2427031273 learned_addr learned my addr 172.21.15.161:0/2427031273 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:22.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.082+0000 7f7dd94ac640 1 --2- 172.21.15.161:0/2427031273 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7dd406ce80 0x7f7dd41e9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:22.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.082+0000 7f7dd9cad640 1 --2- 172.21.15.161:0/2427031273 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7dd414be70 0x7f7dd4148970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:22.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.082+0000 7f7dd8cab640 1 -- 172.21.15.161:0/2427031273 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7dd406ce80 msgr2=0x7f7dd41e9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:22.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.082+0000 7f7dd8cab640 1 --2- 172.21.15.161:0/2427031273 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7dd406ce80 0x7f7dd41e9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:22.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.082+0000 7f7dd8cab640 1 -- 172.21.15.161:0/2427031273 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7dd414be70 msgr2=0x7f7dd4148970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:22.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.082+0000 7f7dd8cab640 1 --2- 172.21.15.161:0/2427031273 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7dd414be70 0x7f7dd4148970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:22.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.082+0000 7f7dd8cab640 1 -- 172.21.15.161:0/2427031273 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dd4153890 con 0x7f7dd40a9680 2024-03-28T23:31:22.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.083+0000 7f7dd8cab640 1 --2- 172.21.15.161:0/2427031273 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7dd40a9680 0x7f7dd4148430 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400e870 tx=0x7f7dc400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:22.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.083+0000 7f7dc9ffb640 1 -- 172.21.15.161:0/2427031273 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4017070 con 0x7f7dd40a9680 2024-03-28T23:31:22.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.083+0000 7f7dc9ffb640 1 -- 172.21.15.161:0/2427031273 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dc400c940 con 0x7f7dd40a9680 2024-03-28T23:31:22.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.083+0000 7f7ddb710640 1 -- 172.21.15.161:0/2427031273 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dd414d010 con 0x7f7dd40a9680 2024-03-28T23:31:22.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.083+0000 7f7dc9ffb640 1 -- 172.21.15.161:0/2427031273 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc400cd30 con 0x7f7dd40a9680 2024-03-28T23:31:22.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.083+0000 7f7ddb710640 1 -- 172.21.15.161:0/2427031273 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dd4155dd0 con 0x7f7dd40a9680 2024-03-28T23:31:22.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.084+0000 7f7ddb710640 1 -- 172.21.15.161:0/2427031273 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dd404f4b0 con 0x7f7dd40a9680 2024-03-28T23:31:22.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.084+0000 7f7dc9ffb640 1 -- 172.21.15.161:0/2427031273 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7dc401a070 con 0x7f7dd40a9680 2024-03-28T23:31:22.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.085+0000 7f7dc9ffb640 1 --2- 172.21.15.161:0/2427031273 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7db806b4d0 0x7f7db806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:22.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.085+0000 7f7dd94ac640 1 --2- 172.21.15.161:0/2427031273 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7db806b4d0 0x7f7db806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:22.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.085+0000 7f7dc9ffb640 1 -- 172.21.15.161:0/2427031273 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6187+0+0 (secure 0 0 0) 0x7f7dc408a160 con 0x7f7dd40a9680 2024-03-28T23:31:22.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.085+0000 7f7dd94ac640 1 --2- 172.21.15.161:0/2427031273 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7db806b4d0 0x7f7db806d990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0000f60 tx=0x7f7dd0007e20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:22.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.089+0000 7f7dc9ffb640 1 -- 172.21.15.161:0/2427031273 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7dd4153890 con 0x7f7dd40a9680 2024-03-28T23:31:22.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.301+0000 7f7ddb710640 1 -- 172.21.15.161:0/2427031273 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f7dd40c0ff0 con 0x7f7dd40a9680 2024-03-28T23:31:22.302 INFO:tasks.ceph.mon.a.smithi161.stderr:2024-03-28T23:31:22.302+0000 7fdc7b85b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T23:31:22.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.303+0000 7f7dc9ffb640 1 -- 172.21.15.161:0/2427031273 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v24) ==== 161+0+0 (secure 0 0 0) 0x7f7dc4058840 con 0x7f7dd40a9680 2024-03-28T23:31:22.303 INFO:teuthology.orchestra.run.smithi161.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-28T23:31:22.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.306+0000 7f7ddb710640 1 -- 172.21.15.161:0/2427031273 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7db806b4d0 msgr2=0x7f7db806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:22.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.306+0000 7f7ddb710640 1 --2- 172.21.15.161:0/2427031273 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7db806b4d0 0x7f7db806d990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0000f60 tx=0x7f7dd0007e20 comp rx=0 tx=0).stop 2024-03-28T23:31:22.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.306+0000 7f7ddb710640 1 -- 172.21.15.161:0/2427031273 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7dd40a9680 msgr2=0x7f7dd4148430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:22.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.306+0000 7f7ddb710640 1 --2- 172.21.15.161:0/2427031273 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7dd40a9680 0x7f7dd4148430 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400e870 tx=0x7f7dc400ed30 comp rx=0 tx=0).stop 2024-03-28T23:31:22.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.306+0000 7f7ddb710640 1 -- 172.21.15.161:0/2427031273 shutdown_connections 2024-03-28T23:31:22.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.306+0000 7f7ddb710640 1 --2- 172.21.15.161:0/2427031273 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7db806b4d0 0x7f7db806d990 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:22.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.306+0000 7f7ddb710640 1 --2- 172.21.15.161:0/2427031273 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7dd40a9680 0x7f7dd4148430 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:22.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.306+0000 7f7ddb710640 1 --2- 172.21.15.161:0/2427031273 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7dd414be70 0x7f7dd4148970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:22.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.306+0000 7f7ddb710640 1 --2- 172.21.15.161:0/2427031273 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7dd406ce80 0x7f7dd41e9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:22.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.306+0000 7f7ddb710640 1 -- 172.21.15.161:0/2427031273 >> 172.21.15.161:0/2427031273 conn(0x7f7dd40c6670 msgr2=0x7f7dd4150810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:22.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.306+0000 7f7ddb710640 1 -- 172.21.15.161:0/2427031273 shutdown_connections 2024-03-28T23:31:22.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.306+0000 7f7ddb710640 1 -- 172.21.15.161:0/2427031273 wait complete. 2024-03-28T23:31:22.308 INFO:tasks.ceph.mds.d.smithi170.stderr: -342> 2024-03-28T23:31:13.238+0000 7fa26681b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:31:22.318 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-28T23:31:22.332 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2024-03-28T23:31:22.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.420+0000 7f30f1d1e640 1 Processor -- start 2024-03-28T23:31:22.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.421+0000 7f30f1d1e640 1 -- start start 2024-03-28T23:31:22.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.421+0000 7f30f1d1e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30ec0c0330 0x7f30ec0c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:22.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.421+0000 7f30f1d1e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f30ec1589b0 0x7f30ec158e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:22.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.421+0000 7f30f1d1e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30ec0c6f60 con 0x7f30ec152bc0 2024-03-28T23:31:22.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.421+0000 7f30f1d1e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f30ec0c6120 con 0x7f30ec1589b0 2024-03-28T23:31:22.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.421+0000 7f30f1d1e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f30ec0c6400 con 0x7f30ec0c0330 2024-03-28T23:31:22.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.421+0000 7f30eb7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f30ec152bc0 0x7f30ec0c8a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:46080/0 (socket says 172.21.15.161:46080) 2024-03-28T23:31:22.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.421+0000 7f30eb7fe640 1 -- 172.21.15.161:0/2542983023 learned_addr learned my addr 172.21.15.161:0/2542983023 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:22.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.422+0000 7f30e2ffd640 1 --2- 172.21.15.161:0/2542983023 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30ec0c0330 0x7f30ec0c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:22.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.422+0000 7f30ebfff640 1 --2- 172.21.15.161:0/2542983023 >> v2:172.21.15.170:3300/0 conn(0x7f30ec1589b0 0x7f30ec158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:22.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.422+0000 7f30eaffd640 1 -- 172.21.15.161:0/2542983023 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1475070025 0 0) 0x7f30ec0c6f60 con 0x7f30ec152bc0 2024-03-28T23:31:22.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.422+0000 7f30eaffd640 1 -- 172.21.15.161:0/2542983023 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30d8003520 con 0x7f30ec152bc0 2024-03-28T23:31:22.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.422+0000 7f30e2ffd640 1 -- 172.21.15.161:0/2542983023 >> v2:172.21.15.170:3300/0 conn(0x7f30ec1589b0 msgr2=0x7f30ec158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:22.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.422+0000 7f30e2ffd640 1 --2- 172.21.15.161:0/2542983023 >> v2:172.21.15.170:3300/0 conn(0x7f30ec1589b0 0x7f30ec158e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:22.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.422+0000 7f30e2ffd640 1 -- 172.21.15.161:0/2542983023 >> v1:172.21.15.161:6789/0 conn(0x7f30ec152bc0 legacy=0x7f30ec0c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:22.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.422+0000 7f30e2ffd640 1 -- 172.21.15.161:0/2542983023 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30ec0c96d0 con 0x7f30ec0c0330 2024-03-28T23:31:22.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.422+0000 7f30ebfff640 1 --2- 172.21.15.161:0/2542983023 >> v2:172.21.15.170:3300/0 conn(0x7f30ec1589b0 0x7f30ec158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:22.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.423+0000 7f30e2ffd640 1 --2- 172.21.15.161:0/2542983023 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30ec0c0330 0x7f30ec0c9190 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f30d40097a0 tx=0x7f30d402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebce71cb4deb89b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:22.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.423+0000 7f30eaffd640 1 -- 172.21.15.161:0/2542983023 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d4035020 con 0x7f30ec0c0330 2024-03-28T23:31:22.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.423+0000 7f30eaffd640 1 -- 172.21.15.161:0/2542983023 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30d4031830 con 0x7f30ec0c0330 2024-03-28T23:31:22.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.423+0000 7f30eaffd640 1 -- 172.21.15.161:0/2542983023 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d4031c00 con 0x7f30ec0c0330 2024-03-28T23:31:22.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.423+0000 7f30f1d1e640 1 -- 172.21.15.161:0/2542983023 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30ec0c0330 msgr2=0x7f30ec0c9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:22.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.423+0000 7f30f1d1e640 1 --2- 172.21.15.161:0/2542983023 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30ec0c0330 0x7f30ec0c9190 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f30d40097a0 tx=0x7f30d402b8d0 comp rx=0 tx=0).stop 2024-03-28T23:31:22.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.423+0000 7f30f1d1e640 1 -- 172.21.15.161:0/2542983023 shutdown_connections 2024-03-28T23:31:22.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.423+0000 7f30f1d1e640 1 --2- 172.21.15.161:0/2542983023 >> v2:172.21.15.170:3300/0 conn(0x7f30ec1589b0 0x7f30ec158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:22.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.423+0000 7f30f1d1e640 1 --2- 172.21.15.161:0/2542983023 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30ec0c0330 0x7f30ec0c9190 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:22.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.423+0000 7f30f1d1e640 1 -- 172.21.15.161:0/2542983023 >> 172.21.15.161:0/2542983023 conn(0x7f30ec0caa80 msgr2=0x7f30ec0cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:22.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.423+0000 7f30f1d1e640 1 -- 172.21.15.161:0/2542983023 shutdown_connections 2024-03-28T23:31:22.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.423+0000 7f30f1d1e640 1 -- 172.21.15.161:0/2542983023 wait complete. 2024-03-28T23:31:22.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.424+0000 7f30f1d1e640 1 Processor -- start 2024-03-28T23:31:22.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.425+0000 7f30f1d1e640 1 -- start start 2024-03-28T23:31:22.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.425+0000 7f30f1d1e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30ec0c0330 0x7f30ec14a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:22.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.425+0000 7f30f1d1e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30ec152bc0 0x7f30ec157a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:22.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.425+0000 7f30f1d1e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30ec1589b0 0x7f30ec150390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:22.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.425+0000 7f30f1d1e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec152fa0 con 0x7f30ec1589b0 2024-03-28T23:31:22.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.425+0000 7f30f1d1e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec0c6bf0 con 0x7f30ec0c0330 2024-03-28T23:31:22.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.425+0000 7f30f1d1e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f30ec151ef0 con 0x7f30ec152bc0 2024-03-28T23:31:22.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.425+0000 7f30ebfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30ec1589b0 0x7f30ec150390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:22.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.425+0000 7f30ebfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30ec1589b0 0x7f30ec150390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35346/0 (socket says 172.21.15.161:35346) 2024-03-28T23:31:22.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.425+0000 7f30ebfff640 1 -- 172.21.15.161:0/2902227667 learned_addr learned my addr 172.21.15.161:0/2902227667 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:22.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.426+0000 7f30eb7fe640 1 --2- 172.21.15.161:0/2902227667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30ec0c0330 0x7f30ec14a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:22.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.426+0000 7f30e2ffd640 1 --2- 172.21.15.161:0/2902227667 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30ec152bc0 0x7f30ec157a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:22.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.426+0000 7f30ebfff640 1 -- 172.21.15.161:0/2902227667 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30ec152bc0 msgr2=0x7f30ec157a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:22.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.426+0000 7f30ebfff640 1 --2- 172.21.15.161:0/2902227667 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30ec152bc0 0x7f30ec157a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:22.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.426+0000 7f30ebfff640 1 -- 172.21.15.161:0/2902227667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30ec0c0330 msgr2=0x7f30ec14a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:22.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.426+0000 7f30ebfff640 1 --2- 172.21.15.161:0/2902227667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30ec0c0330 0x7f30ec14a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:22.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.426+0000 7f30ebfff640 1 -- 172.21.15.161:0/2902227667 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30ec152380 con 0x7f30ec1589b0 2024-03-28T23:31:22.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.426+0000 7f30ebfff640 1 --2- 172.21.15.161:0/2902227667 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30ec1589b0 0x7f30ec150390 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f30e4009e60 tx=0x7f30e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:22.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.426+0000 7f30e8ff9640 1 -- 172.21.15.161:0/2902227667 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e4018030 con 0x7f30ec1589b0 2024-03-28T23:31:22.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.426+0000 7f30e8ff9640 1 -- 172.21.15.161:0/2902227667 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30e400cdf0 con 0x7f30ec1589b0 2024-03-28T23:31:22.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.426+0000 7f30f1d1e640 1 -- 172.21.15.161:0/2902227667 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30ec14a030 con 0x7f30ec1589b0 2024-03-28T23:31:22.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.426+0000 7f30e8ff9640 1 -- 172.21.15.161:0/2902227667 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e4003bb0 con 0x7f30ec1589b0 2024-03-28T23:31:22.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.426+0000 7f30f1d1e640 1 -- 172.21.15.161:0/2902227667 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30ec1eb780 con 0x7f30ec1589b0 2024-03-28T23:31:22.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.427+0000 7f30f1d1e640 1 -- 172.21.15.161:0/2902227667 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30ec04f4b0 con 0x7f30ec1589b0 2024-03-28T23:31:22.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.427+0000 7f30e8ff9640 1 -- 172.21.15.161:0/2902227667 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f30e4005d10 con 0x7f30ec1589b0 2024-03-28T23:31:22.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.428+0000 7f30e8ff9640 1 --2- 172.21.15.161:0/2902227667 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f30cc06b500 0x7f30cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:22.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.428+0000 7f30e8ff9640 1 -- 172.21.15.161:0/2902227667 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6273+0+0 (secure 0 0 0) 0x7f30e408ac30 con 0x7f30ec1589b0 2024-03-28T23:31:22.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.428+0000 7f30eb7fe640 1 --2- 172.21.15.161:0/2902227667 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f30cc06b500 0x7f30cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:22.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.428+0000 7f30eb7fe640 1 --2- 172.21.15.161:0/2902227667 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f30cc06b500 0x7f30cc06d9c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f30dc005880 tx=0x7f30dc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:22.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.432+0000 7f30e8ff9640 1 -- 172.21.15.161:0/2902227667 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f30e4022070 con 0x7f30ec1589b0 2024-03-28T23:31:22.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:22.642+0000 7f30f1d1e640 1 -- 172.21.15.161:0/2902227667 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f30ec0cb250 con 0x7f30ec1589b0 2024-03-28T23:31:23.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.304+0000 7f30e8ff9640 1 -- 172.21.15.161:0/2902227667 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v25) ==== 104+0+0 (secure 0 0 0) 0x7f30e4010040 con 0x7f30ec1589b0 2024-03-28T23:31:23.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.306+0000 7f30f1d1e640 1 -- 172.21.15.161:0/2902227667 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f30cc06b500 msgr2=0x7f30cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:23.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.306+0000 7f30f1d1e640 1 --2- 172.21.15.161:0/2902227667 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f30cc06b500 0x7f30cc06d9c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f30dc005880 tx=0x7f30dc009040 comp rx=0 tx=0).stop 2024-03-28T23:31:23.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.306+0000 7f30f1d1e640 1 -- 172.21.15.161:0/2902227667 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30ec1589b0 msgr2=0x7f30ec150390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:23.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.306+0000 7f30f1d1e640 1 --2- 172.21.15.161:0/2902227667 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30ec1589b0 0x7f30ec150390 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f30e4009e60 tx=0x7f30e400c490 comp rx=0 tx=0).stop 2024-03-28T23:31:23.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.306+0000 7f30f1d1e640 1 -- 172.21.15.161:0/2902227667 shutdown_connections 2024-03-28T23:31:23.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.306+0000 7f30f1d1e640 1 --2- 172.21.15.161:0/2902227667 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f30cc06b500 0x7f30cc06d9c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:23.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.306+0000 7f30f1d1e640 1 --2- 172.21.15.161:0/2902227667 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30ec1589b0 0x7f30ec150390 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:23.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.306+0000 7f30f1d1e640 1 --2- 172.21.15.161:0/2902227667 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30ec152bc0 0x7f30ec157a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:23.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.306+0000 7f30f1d1e640 1 --2- 172.21.15.161:0/2902227667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30ec0c0330 0x7f30ec14a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:23.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.306+0000 7f30f1d1e640 1 -- 172.21.15.161:0/2902227667 >> 172.21.15.161:0/2902227667 conn(0x7f30ec0caa80 msgr2=0x7f30ec0c9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:23.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.307+0000 7f30f1d1e640 1 -- 172.21.15.161:0/2902227667 shutdown_connections 2024-03-28T23:31:23.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.307+0000 7f30f1d1e640 1 -- 172.21.15.161:0/2902227667 wait complete. 2024-03-28T23:31:23.319 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-28T23:31:23.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.425+0000 7fee3f33e640 1 Processor -- start 2024-03-28T23:31:23.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.425+0000 7fee3f33e640 1 -- start start 2024-03-28T23:31:23.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee3f33e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fee380c3080 0x7fee38151db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:23.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee3f33e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee3806cab0 0x7fee3806cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:23.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee3f33e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee380c15a0 con 0x7fee3814f1d0 2024-03-28T23:31:23.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee3f33e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fee380c6330 con 0x7fee380c3080 2024-03-28T23:31:23.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee3f33e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fee380c1840 con 0x7fee3806cab0 2024-03-28T23:31:23.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee3d0da640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fee3814f1d0 0x7fee380c3a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:46086/0 (socket says 172.21.15.161:46086) 2024-03-28T23:31:23.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee3d0da640 1 -- 172.21.15.161:0/160625092 learned_addr learned my addr 172.21.15.161:0/160625092 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:23.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee3d8db640 1 --2- 172.21.15.161:0/160625092 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee3806cab0 0x7fee3806cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:23.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee2ffff640 1 -- 172.21.15.161:0/160625092 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1898213468 0 0) 0x7fee380c15a0 con 0x7fee3814f1d0 2024-03-28T23:31:23.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee3c8d9640 1 --2- 172.21.15.161:0/160625092 >> v2:172.21.15.170:3300/0 conn(0x7fee380c3080 0x7fee38151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:23.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee2ffff640 1 -- 172.21.15.161:0/160625092 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee24005730 con 0x7fee3814f1d0 2024-03-28T23:31:23.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee2ffff640 1 -- 172.21.15.161:0/160625092 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1557771192 0 0) 0x7fee24005730 con 0x7fee3814f1d0 2024-03-28T23:31:23.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee2ffff640 1 -- 172.21.15.161:0/160625092 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee3806cab0 msgr2=0x7fee3806cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:23.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee2ffff640 1 --2- 172.21.15.161:0/160625092 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee3806cab0 0x7fee3806cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:23.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee2ffff640 1 -- 172.21.15.161:0/160625092 >> v2:172.21.15.170:3300/0 conn(0x7fee380c3080 msgr2=0x7fee38151db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:23.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee2ffff640 1 --2- 172.21.15.161:0/160625092 >> v2:172.21.15.170:3300/0 conn(0x7fee380c3080 0x7fee38151db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:23.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee2ffff640 1 -- 172.21.15.161:0/160625092 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee380c1b20 con 0x7fee3814f1d0 2024-03-28T23:31:23.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee2ffff640 1 -- 172.21.15.161:0/160625092 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fee34001670 con 0x7fee3814f1d0 2024-03-28T23:31:23.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee2ffff640 1 -- 172.21.15.161:0/160625092 >> v1:172.21.15.161:6789/0 conn(0x7fee3814f1d0 legacy=0x7fee380c3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:23.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee2ffff640 1 --2- 172.21.15.161:0/160625092 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee24005bc0 0x7fee24005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:23.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee2ffff640 1 --2- 172.21.15.161:0/160625092 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee3806cab0 0x7fee24006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:23.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee2ffff640 1 --2- 172.21.15.161:0/160625092 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee24006f50 0x7fee240073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:23.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee2ffff640 1 -- 172.21.15.161:0/160625092 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fee380c6330 con 0x7fee24006f50 2024-03-28T23:31:23.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee2ffff640 1 -- 172.21.15.161:0/160625092 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fee380c1840 con 0x7fee24005bc0 2024-03-28T23:31:23.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.426+0000 7fee2ffff640 1 -- 172.21.15.161:0/160625092 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fee2400dbd0 con 0x7fee3806cab0 2024-03-28T23:31:23.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.427+0000 7fee3d8db640 1 --2- 172.21.15.161:0/160625092 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee24006f50 0x7fee240073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:23.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.427+0000 7fee3d0da640 1 --2- 172.21.15.161:0/160625092 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee24005bc0 0x7fee24005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:23.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.427+0000 7fee3c8d9640 1 --2- 172.21.15.161:0/160625092 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee3806cab0 0x7fee24006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:23.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.427+0000 7fee3d8db640 1 -- 172.21.15.161:0/160625092 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee3806cab0 msgr2=0x7fee24006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:23.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.427+0000 7fee3d8db640 1 --2- 172.21.15.161:0/160625092 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee3806cab0 0x7fee24006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:23.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.427+0000 7fee3d8db640 1 -- 172.21.15.161:0/160625092 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee24005bc0 msgr2=0x7fee24005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:23.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.427+0000 7fee3d8db640 1 --2- 172.21.15.161:0/160625092 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee24005bc0 0x7fee24005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:23.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.427+0000 7fee3d8db640 1 -- 172.21.15.161:0/160625092 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee24005730 con 0x7fee24006f50 2024-03-28T23:31:23.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.428+0000 7fee3d8db640 1 --2- 172.21.15.161:0/160625092 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee24006f50 0x7fee240073d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fee30009fc0 tx=0x7fee24007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a4840aa11fa0bc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:23.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.428+0000 7fee2ffff640 1 -- 172.21.15.161:0/160625092 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee30039070 con 0x7fee24006f50 2024-03-28T23:31:23.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.428+0000 7fee2ffff640 1 -- 172.21.15.161:0/160625092 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee3002cc20 con 0x7fee24006f50 2024-03-28T23:31:23.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.428+0000 7fee2ffff640 1 -- 172.21.15.161:0/160625092 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee30009c10 con 0x7fee24006f50 2024-03-28T23:31:23.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.428+0000 7fee3f33e640 1 -- 172.21.15.161:0/160625092 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee24006f50 msgr2=0x7fee240073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:23.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.428+0000 7fee3f33e640 1 --2- 172.21.15.161:0/160625092 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee24006f50 0x7fee240073d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fee30009fc0 tx=0x7fee24007a40 comp rx=0 tx=0).stop 2024-03-28T23:31:23.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.428+0000 7fee3d8db640 1 -- 172.21.15.161:0/160625092 reap_dead start 2024-03-28T23:31:23.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.428+0000 7fee3f33e640 1 -- 172.21.15.161:0/160625092 shutdown_connections 2024-03-28T23:31:23.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.428+0000 7fee3f33e640 1 -- 172.21.15.161:0/160625092 >> 172.21.15.161:0/160625092 conn(0x7fee380c6e20 msgr2=0x7fee380c7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:23.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.428+0000 7fee3f33e640 1 -- 172.21.15.161:0/160625092 shutdown_connections 2024-03-28T23:31:23.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.428+0000 7fee3f33e640 1 -- 172.21.15.161:0/160625092 wait complete. 2024-03-28T23:31:23.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.429+0000 7fee3f33e640 1 Processor -- start 2024-03-28T23:31:23.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.430+0000 7fee3f33e640 1 -- start start 2024-03-28T23:31:23.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.430+0000 7fee3f33e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee381ea1e0 0x7fee381ea5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:23.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.430+0000 7fee3f33e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee38150f40 0x7fee38148560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:23.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.430+0000 7fee3f33e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee38148aa0 0x7fee3814a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:23.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.430+0000 7fee3f33e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fee380c41a0 con 0x7fee381ea1e0 2024-03-28T23:31:23.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.430+0000 7fee3f33e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fee380c2600 con 0x7fee38148aa0 2024-03-28T23:31:23.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.430+0000 7fee3f33e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fee380c4320 con 0x7fee38150f40 2024-03-28T23:31:23.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.430+0000 7fee3d0da640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee381ea1e0 0x7fee381ea5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:23.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.430+0000 7fee3d0da640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee381ea1e0 0x7fee381ea5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35366/0 (socket says 172.21.15.161:35366) 2024-03-28T23:31:23.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.430+0000 7fee3d0da640 1 -- 172.21.15.161:0/1894201297 learned_addr learned my addr 172.21.15.161:0/1894201297 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:23.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.430+0000 7fee3c8d9640 1 --2- 172.21.15.161:0/1894201297 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee38150f40 0x7fee38148560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:23.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.430+0000 7fee3d8db640 1 --2- 172.21.15.161:0/1894201297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee38148aa0 0x7fee3814a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:23.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.430+0000 7fee3d0da640 1 -- 172.21.15.161:0/1894201297 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee38150f40 msgr2=0x7fee38148560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:23.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.430+0000 7fee3d0da640 1 --2- 172.21.15.161:0/1894201297 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee38150f40 0x7fee38148560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:23.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.430+0000 7fee3d0da640 1 -- 172.21.15.161:0/1894201297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee38148aa0 msgr2=0x7fee3814a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:23.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.431+0000 7fee3d0da640 1 --2- 172.21.15.161:0/1894201297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee38148aa0 0x7fee3814a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:23.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.431+0000 7fee3d0da640 1 -- 172.21.15.161:0/1894201297 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee3814ffa0 con 0x7fee381ea1e0 2024-03-28T23:31:23.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.431+0000 7fee3d8db640 1 --2- 172.21.15.161:0/1894201297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee38148aa0 0x7fee3814a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:23.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.431+0000 7fee3d0da640 1 --2- 172.21.15.161:0/1894201297 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee381ea1e0 0x7fee381ea5c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fee340072c0 tx=0x7fee34007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:23.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.431+0000 7fee2dffb640 1 -- 172.21.15.161:0/1894201297 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3400f9c0 con 0x7fee381ea1e0 2024-03-28T23:31:23.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.431+0000 7fee2dffb640 1 -- 172.21.15.161:0/1894201297 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee34018070 con 0x7fee381ea1e0 2024-03-28T23:31:23.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.431+0000 7fee2dffb640 1 -- 172.21.15.161:0/1894201297 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3400fde0 con 0x7fee381ea1e0 2024-03-28T23:31:23.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.431+0000 7fee3f33e640 1 -- 172.21.15.161:0/1894201297 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee381eac60 con 0x7fee381ea1e0 2024-03-28T23:31:23.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.431+0000 7fee3f33e640 1 -- 172.21.15.161:0/1894201297 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee38151c10 con 0x7fee381ea1e0 2024-03-28T23:31:23.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.432+0000 7fee3f33e640 1 -- 172.21.15.161:0/1894201297 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee38151e10 con 0x7fee381ea1e0 2024-03-28T23:31:23.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.432+0000 7fee2dffb640 1 -- 172.21.15.161:0/1894201297 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fee3401d460 con 0x7fee381ea1e0 2024-03-28T23:31:23.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.433+0000 7fee2dffb640 1 --2- 172.21.15.161:0/1894201297 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fee1c06b500 0x7fee1c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:23.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.433+0000 7fee2dffb640 1 -- 172.21.15.161:0/1894201297 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6273+0+0 (secure 0 0 0) 0x7fee3408a5f0 con 0x7fee381ea1e0 2024-03-28T23:31:23.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.433+0000 7fee3c8d9640 1 --2- 172.21.15.161:0/1894201297 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fee1c06b500 0x7fee1c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:23.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.433+0000 7fee3c8d9640 1 --2- 172.21.15.161:0/1894201297 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fee1c06b500 0x7fee1c06d9c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fee3814b2b0 tx=0x7fee28008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:23.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.437+0000 7fee2dffb640 1 -- 172.21.15.161:0/1894201297 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fee34059230 con 0x7fee381ea1e0 2024-03-28T23:31:23.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:23.608+0000 7fee3f33e640 1 -- 172.21.15.161:0/1894201297 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fee38068f90 con 0x7fee381ea1e0 2024-03-28T23:31:24.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.308+0000 7fee2dffb640 1 -- 172.21.15.161:0/1894201297 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v41) ==== 181+0+0 (secure 0 0 0) 0x7fee34058b30 con 0x7fee381ea1e0 2024-03-28T23:31:24.309 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_metadata' removed 2024-03-28T23:31:24.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.312+0000 7fee3f33e640 1 -- 172.21.15.161:0/1894201297 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fee1c06b500 msgr2=0x7fee1c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:24.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.312+0000 7fee3f33e640 1 --2- 172.21.15.161:0/1894201297 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fee1c06b500 0x7fee1c06d9c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fee3814b2b0 tx=0x7fee28008040 comp rx=0 tx=0).stop 2024-03-28T23:31:24.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.312+0000 7fee3f33e640 1 -- 172.21.15.161:0/1894201297 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee381ea1e0 msgr2=0x7fee381ea5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:24.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.312+0000 7fee3f33e640 1 --2- 172.21.15.161:0/1894201297 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee381ea1e0 0x7fee381ea5c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fee340072c0 tx=0x7fee34007780 comp rx=0 tx=0).stop 2024-03-28T23:31:24.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.312+0000 7fee3f33e640 1 -- 172.21.15.161:0/1894201297 shutdown_connections 2024-03-28T23:31:24.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.312+0000 7fee3f33e640 1 --2- 172.21.15.161:0/1894201297 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fee1c06b500 0x7fee1c06d9c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:24.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.312+0000 7fee3f33e640 1 --2- 172.21.15.161:0/1894201297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee38148aa0 0x7fee3814a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:24.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.312+0000 7fee3f33e640 1 --2- 172.21.15.161:0/1894201297 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee38150f40 0x7fee38148560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:24.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.312+0000 7fee3f33e640 1 --2- 172.21.15.161:0/1894201297 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee381ea1e0 0x7fee381ea5c0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:24.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.312+0000 7fee3f33e640 1 -- 172.21.15.161:0/1894201297 >> 172.21.15.161:0/1894201297 conn(0x7fee380c6e20 msgr2=0x7fee38071a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:24.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.312+0000 7fee3f33e640 1 -- 172.21.15.161:0/1894201297 shutdown_connections 2024-03-28T23:31:24.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.312+0000 7fee3f33e640 1 -- 172.21.15.161:0/1894201297 wait complete. 2024-03-28T23:31:24.327 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-28T23:31:24.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.438+0000 7fe52aaed640 1 Processor -- start 2024-03-28T23:31:24.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.438+0000 7fe52aaed640 1 -- start start 2024-03-28T23:31:24.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.439+0000 7fe52aaed640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe5240c20c0 0x7fe524152880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:24.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.439+0000 7fe52aaed640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5240c3540 con 0x7fe5240ac430 2024-03-28T23:31:24.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.439+0000 7fe52aaed640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5240c3820 con 0x7fe5240c30c0 2024-03-28T23:31:24.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.439+0000 7fe52aaed640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe524152dc0 con 0x7fe5240c20c0 2024-03-28T23:31:24.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.439+0000 7fe52908a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe5240c20c0 0x7fe524152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:24.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.439+0000 7fe528889640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe5240ac430 0x7fe5240c24d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:46090/0 (socket says 172.21.15.161:46090) 2024-03-28T23:31:24.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.439+0000 7fe528889640 1 -- 172.21.15.161:0/3852191332 learned_addr learned my addr 172.21.15.161:0/3852191332 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:24.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.439+0000 7fe52908a640 1 -- 172.21.15.161:0/3852191332 >> v1:172.21.15.170:6789/0 conn(0x7fe5240c30c0 legacy=0x7fe5240c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:24.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.439+0000 7fe52908a640 1 -- 172.21.15.161:0/3852191332 >> v1:172.21.15.161:6789/0 conn(0x7fe5240ac430 legacy=0x7fe5240c24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:24.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.439+0000 7fe52908a640 1 -- 172.21.15.161:0/3852191332 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5240c3b00 con 0x7fe5240c20c0 2024-03-28T23:31:24.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.439+0000 7fe5237fe640 1 -- 172.21.15.161:0/3852191332 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 987295894 0 0) 0x7fe5240c3540 con 0x7fe5240ac430 2024-03-28T23:31:24.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.440+0000 7fe52908a640 1 --2- 172.21.15.161:0/3852191332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5240c20c0 0x7fe524152880 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe51c009d90 tx=0x7fe51c02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=952876c2d27a81e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:24.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.440+0000 7fe5237fe640 1 -- 172.21.15.161:0/3852191332 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51c039070 con 0x7fe5240c20c0 2024-03-28T23:31:24.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.440+0000 7fe5237fe640 1 -- 172.21.15.161:0/3852191332 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe51c02cbd0 con 0x7fe5240c20c0 2024-03-28T23:31:24.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.440+0000 7fe5237fe640 1 -- 172.21.15.161:0/3852191332 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51c034440 con 0x7fe5240c20c0 2024-03-28T23:31:24.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.441+0000 7fe52aaed640 1 -- 172.21.15.161:0/3852191332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5240c20c0 msgr2=0x7fe524152880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:24.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.441+0000 7fe52aaed640 1 --2- 172.21.15.161:0/3852191332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5240c20c0 0x7fe524152880 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe51c009d90 tx=0x7fe51c02be20 comp rx=0 tx=0).stop 2024-03-28T23:31:24.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.441+0000 7fe52aaed640 1 -- 172.21.15.161:0/3852191332 shutdown_connections 2024-03-28T23:31:24.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.441+0000 7fe52aaed640 1 --2- 172.21.15.161:0/3852191332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5240c20c0 0x7fe524152880 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:24.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.441+0000 7fe52aaed640 1 -- 172.21.15.161:0/3852191332 >> 172.21.15.161:0/3852191332 conn(0x7fe52406b150 msgr2=0x7fe52406b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:24.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.441+0000 7fe52aaed640 1 -- 172.21.15.161:0/3852191332 shutdown_connections 2024-03-28T23:31:24.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.441+0000 7fe52aaed640 1 -- 172.21.15.161:0/3852191332 wait complete. 2024-03-28T23:31:24.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.442+0000 7fe52aaed640 1 Processor -- start 2024-03-28T23:31:24.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.442+0000 7fe52aaed640 1 -- start start 2024-03-28T23:31:24.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.443+0000 7fe52aaed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5240ac430 0x7fe5240a8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:24.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.443+0000 7fe52aaed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5240c20c0 0x7fe5240a73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:24.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.443+0000 7fe52aaed640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5240c30c0 0x7fe5240a7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:24.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.443+0000 7fe52aaed640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe5240c6440 con 0x7fe5240c30c0 2024-03-28T23:31:24.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.443+0000 7fe52aaed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe5240c1490 con 0x7fe5240ac430 2024-03-28T23:31:24.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.443+0000 7fe52aaed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe5240c2be0 con 0x7fe5240c20c0 2024-03-28T23:31:24.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.443+0000 7fe52908a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5240c30c0 0x7fe5240a7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:24.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.443+0000 7fe52908a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5240c30c0 0x7fe5240a7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35380/0 (socket says 172.21.15.161:35380) 2024-03-28T23:31:24.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.443+0000 7fe52908a640 1 -- 172.21.15.161:0/2660075936 learned_addr learned my addr 172.21.15.161:0/2660075936 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:24.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.443+0000 7fe528889640 1 --2- 172.21.15.161:0/2660075936 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5240ac430 0x7fe5240a8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:24.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.443+0000 7fe523fff640 1 --2- 172.21.15.161:0/2660075936 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5240c20c0 0x7fe5240a73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:24.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.443+0000 7fe52908a640 1 -- 172.21.15.161:0/2660075936 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5240c20c0 msgr2=0x7fe5240a73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:24.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.443+0000 7fe52908a640 1 --2- 172.21.15.161:0/2660075936 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5240c20c0 0x7fe5240a73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:24.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.443+0000 7fe52908a640 1 -- 172.21.15.161:0/2660075936 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5240ac430 msgr2=0x7fe5240a8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:24.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.443+0000 7fe52908a640 1 --2- 172.21.15.161:0/2660075936 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5240ac430 0x7fe5240a8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:24.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.443+0000 7fe52908a640 1 -- 172.21.15.161:0/2660075936 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe524073700 con 0x7fe5240c30c0 2024-03-28T23:31:24.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.443+0000 7fe523fff640 1 --2- 172.21.15.161:0/2660075936 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5240c20c0 0x7fe5240a73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:24.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.443+0000 7fe52908a640 1 --2- 172.21.15.161:0/2660075936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5240c30c0 0x7fe5240a7900 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe51c009eb0 tx=0x7fe51c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:24.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.444+0000 7fe5217fa640 1 -- 172.21.15.161:0/2660075936 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51c044030 con 0x7fe5240c30c0 2024-03-28T23:31:24.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.444+0000 7fe5217fa640 1 -- 172.21.15.161:0/2660075936 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe51c02c9d0 con 0x7fe5240c30c0 2024-03-28T23:31:24.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.444+0000 7fe52aaed640 1 -- 172.21.15.161:0/2660075936 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5240a6600 con 0x7fe5240c30c0 2024-03-28T23:31:24.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.444+0000 7fe5217fa640 1 -- 172.21.15.161:0/2660075936 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51c039040 con 0x7fe5240c30c0 2024-03-28T23:31:24.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.444+0000 7fe52aaed640 1 -- 172.21.15.161:0/2660075936 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5240a6bc0 con 0x7fe5240c30c0 2024-03-28T23:31:24.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.445+0000 7fe52aaed640 1 -- 172.21.15.161:0/2660075936 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5240c3540 con 0x7fe5240c30c0 2024-03-28T23:31:24.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.445+0000 7fe5217fa640 1 -- 172.21.15.161:0/2660075936 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe51c047070 con 0x7fe5240c30c0 2024-03-28T23:31:24.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.446+0000 7fe5217fa640 1 --2- 172.21.15.161:0/2660075936 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe51006b4d0 0x7fe51006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:24.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.446+0000 7fe528889640 1 --2- 172.21.15.161:0/2660075936 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe51006b4d0 0x7fe51006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:24.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.446+0000 7fe5217fa640 1 -- 172.21.15.161:0/2660075936 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 5835+0+0 (secure 0 0 0) 0x7fe51c0aca30 con 0x7fe5240c30c0 2024-03-28T23:31:24.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.446+0000 7fe528889640 1 --2- 172.21.15.161:0/2660075936 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe51006b4d0 0x7fe51006d990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe514000f60 tx=0x7fe514007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:24.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.450+0000 7fe5217fa640 1 -- 172.21.15.161:0/2660075936 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe5240c3540 con 0x7fe5240c30c0 2024-03-28T23:31:24.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:24.629+0000 7fe52aaed640 1 -- 172.21.15.161:0/2660075936 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe524068f90 con 0x7fe5240c30c0 2024-03-28T23:31:25.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.310+0000 7fe5217fa640 1 -- 172.21.15.161:0/2660075936 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v42) ==== 169+0+0 (secure 0 0 0) 0x7fe51c04e070 con 0x7fe5240c30c0 2024-03-28T23:31:25.311 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data' removed 2024-03-28T23:31:25.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.312+0000 7fe52aaed640 1 -- 172.21.15.161:0/2660075936 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe51006b4d0 msgr2=0x7fe51006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:25.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.312+0000 7fe52aaed640 1 --2- 172.21.15.161:0/2660075936 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe51006b4d0 0x7fe51006d990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe514000f60 tx=0x7fe514007d50 comp rx=0 tx=0).stop 2024-03-28T23:31:25.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.312+0000 7fe52aaed640 1 -- 172.21.15.161:0/2660075936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5240c30c0 msgr2=0x7fe5240a7900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:25.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.312+0000 7fe52aaed640 1 --2- 172.21.15.161:0/2660075936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5240c30c0 0x7fe5240a7900 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe51c009eb0 tx=0x7fe51c004a40 comp rx=0 tx=0).stop 2024-03-28T23:31:25.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.312+0000 7fe52aaed640 1 -- 172.21.15.161:0/2660075936 shutdown_connections 2024-03-28T23:31:25.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.312+0000 7fe52aaed640 1 --2- 172.21.15.161:0/2660075936 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe51006b4d0 0x7fe51006d990 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:25.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.312+0000 7fe52aaed640 1 --2- 172.21.15.161:0/2660075936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5240c30c0 0x7fe5240a7900 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:25.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.312+0000 7fe52aaed640 1 --2- 172.21.15.161:0/2660075936 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5240c20c0 0x7fe5240a73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:25.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.312+0000 7fe52aaed640 1 --2- 172.21.15.161:0/2660075936 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5240ac430 0x7fe5240a8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:25.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.312+0000 7fe52aaed640 1 -- 172.21.15.161:0/2660075936 >> 172.21.15.161:0/2660075936 conn(0x7fe52406b150 msgr2=0x7fe5240c4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:25.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.313+0000 7fe52aaed640 1 -- 172.21.15.161:0/2660075936 shutdown_connections 2024-03-28T23:31:25.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.313+0000 7fe52aaed640 1 -- 172.21.15.161:0/2660075936 wait complete. 2024-03-28T23:31:25.327 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-28T23:31:25.422 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:31:25.424 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:31:25.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.436+0000 7f0b72f47640 1 Processor -- start 2024-03-28T23:31:25.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.436+0000 7f0b72f47640 1 -- start start 2024-03-28T23:31:25.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.437+0000 7f0b72f47640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0b6c155f70 0x7f0b6c1595f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:25.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.437+0000 7f0b72f47640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b6c0c3540 con 0x7f0b6c06d510 2024-03-28T23:31:25.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.437+0000 7f0b72f47640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b6c0c3820 con 0x7f0b6c072150 2024-03-28T23:31:25.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.437+0000 7f0b72f47640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f0b6c158b10 con 0x7f0b6c155f70 2024-03-28T23:31:25.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.437+0000 7f0b6bfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f0b6c06d510 0x7f0b6c0c2d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:46104/0 (socket says 172.21.15.161:46104) 2024-03-28T23:31:25.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.437+0000 7f0b6bfff640 1 -- 172.21.15.161:0/3978896914 learned_addr learned my addr 172.21.15.161:0/3978896914 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:25.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.437+0000 7f0b70ce3640 1 --2- 172.21.15.161:0/3978896914 >> v2:172.21.15.170:3300/0 conn(0x7f0b6c155f70 0x7f0b6c1595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:25.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.437+0000 7f0b6b7fe640 1 -- 172.21.15.161:0/3978896914 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 400104485 0 0) 0x7f0b6c0c3540 con 0x7f0b6c06d510 2024-03-28T23:31:25.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.437+0000 7f0b6b7fe640 1 -- 172.21.15.161:0/3978896914 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b5c003520 con 0x7f0b6c06d510 2024-03-28T23:31:25.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.437+0000 7f0b6b7fe640 1 -- 172.21.15.161:0/3978896914 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3842666619 0 0) 0x7f0b6c0c3820 con 0x7f0b6c072150 2024-03-28T23:31:25.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.437+0000 7f0b6b7fe640 1 -- 172.21.15.161:0/3978896914 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b6c0c3540 con 0x7f0b6c072150 2024-03-28T23:31:25.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.438+0000 7f0b6b7fe640 1 -- 172.21.15.161:0/3978896914 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2537696193 0 0) 0x7f0b5c003520 con 0x7f0b6c06d510 2024-03-28T23:31:25.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.438+0000 7f0b70ce3640 1 -- 172.21.15.161:0/3978896914 >> v1:172.21.15.170:6789/0 conn(0x7f0b6c072150 legacy=0x7f0b6c0c77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:25.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.438+0000 7f0b70ce3640 1 -- 172.21.15.161:0/3978896914 >> v1:172.21.15.161:6789/0 conn(0x7f0b6c06d510 legacy=0x7f0b6c0c2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:25.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.438+0000 7f0b70ce3640 1 -- 172.21.15.161:0/3978896914 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b6c158df0 con 0x7f0b6c155f70 2024-03-28T23:31:25.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.438+0000 7f0b70ce3640 1 --2- 172.21.15.161:0/3978896914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b6c155f70 0x7f0b6c1595f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0b600097d0 tx=0x7f0b6002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2fcd759bf4c64de server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:25.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.439+0000 7f0b6b7fe640 1 -- 172.21.15.161:0/3978896914 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b60037070 con 0x7f0b6c155f70 2024-03-28T23:31:25.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.439+0000 7f0b6b7fe640 1 -- 172.21.15.161:0/3978896914 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b60030510 con 0x7f0b6c155f70 2024-03-28T23:31:25.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.439+0000 7f0b6b7fe640 1 -- 172.21.15.161:0/3978896914 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b58001670 con 0x7f0b6c155f70 2024-03-28T23:31:25.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.439+0000 7f0b72f47640 1 -- 172.21.15.161:0/3978896914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b6c155f70 msgr2=0x7f0b6c1595f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:25.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.439+0000 7f0b72f47640 1 --2- 172.21.15.161:0/3978896914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b6c155f70 0x7f0b6c1595f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0b600097d0 tx=0x7f0b6002b790 comp rx=0 tx=0).stop 2024-03-28T23:31:25.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.439+0000 7f0b72f47640 1 -- 172.21.15.161:0/3978896914 shutdown_connections 2024-03-28T23:31:25.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.439+0000 7f0b72f47640 1 --2- 172.21.15.161:0/3978896914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b6c155f70 0x7f0b6c1595f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:25.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.439+0000 7f0b72f47640 1 -- 172.21.15.161:0/3978896914 >> 172.21.15.161:0/3978896914 conn(0x7f0b6c0a97c0 msgr2=0x7f0b6c0a9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:25.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.439+0000 7f0b72f47640 1 -- 172.21.15.161:0/3978896914 shutdown_connections 2024-03-28T23:31:25.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.439+0000 7f0b72f47640 1 -- 172.21.15.161:0/3978896914 wait complete. 2024-03-28T23:31:25.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.440+0000 7f0b72f47640 1 Processor -- start 2024-03-28T23:31:25.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.440+0000 7f0b72f47640 1 -- start start 2024-03-28T23:31:25.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.441+0000 7f0b72f47640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b6c06d510 0x7f0b6c1522c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:25.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.441+0000 7f0b72f47640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0b6c072150 0x7f0b6c0ab8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:25.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.441+0000 7f0b72f47640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b6c155f70 0x7f0b6c1ee170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:25.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.441+0000 7f0b72f47640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0b6c1564d0 con 0x7f0b6c155f70 2024-03-28T23:31:25.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.441+0000 7f0b72f47640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0b6c156350 con 0x7f0b6c06d510 2024-03-28T23:31:25.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.441+0000 7f0b72f47640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0b6c0725c0 con 0x7f0b6c072150 2024-03-28T23:31:25.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.441+0000 7f0b714e4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b6c155f70 0x7f0b6c1ee170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:25.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.441+0000 7f0b714e4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b6c155f70 0x7f0b6c1ee170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35390/0 (socket says 172.21.15.161:35390) 2024-03-28T23:31:25.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.441+0000 7f0b714e4640 1 -- 172.21.15.161:0/2898487851 learned_addr learned my addr 172.21.15.161:0/2898487851 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:25.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.441+0000 7f0b70ce3640 1 --2- 172.21.15.161:0/2898487851 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b6c06d510 0x7f0b6c1522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:25.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.441+0000 7f0b6bfff640 1 --2- 172.21.15.161:0/2898487851 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0b6c072150 0x7f0b6c0ab8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:25.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.442+0000 7f0b714e4640 1 -- 172.21.15.161:0/2898487851 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0b6c072150 msgr2=0x7f0b6c0ab8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:25.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.442+0000 7f0b714e4640 1 --2- 172.21.15.161:0/2898487851 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0b6c072150 0x7f0b6c0ab8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:25.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.442+0000 7f0b714e4640 1 -- 172.21.15.161:0/2898487851 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b6c06d510 msgr2=0x7f0b6c1522c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:25.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.442+0000 7f0b714e4640 1 --2- 172.21.15.161:0/2898487851 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b6c06d510 0x7f0b6c1522c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:25.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.442+0000 7f0b714e4640 1 -- 172.21.15.161:0/2898487851 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b6c09b440 con 0x7f0b6c155f70 2024-03-28T23:31:25.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.442+0000 7f0b714e4640 1 --2- 172.21.15.161:0/2898487851 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b6c155f70 0x7f0b6c1ee170 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0b6400c790 tx=0x7f0b6400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:25.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.442+0000 7f0b697fa640 1 -- 172.21.15.161:0/2898487851 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b64003b70 con 0x7f0b6c155f70 2024-03-28T23:31:25.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.442+0000 7f0b697fa640 1 -- 172.21.15.161:0/2898487851 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b64003d10 con 0x7f0b6c155f70 2024-03-28T23:31:25.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.442+0000 7f0b72f47640 1 -- 172.21.15.161:0/2898487851 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b6c09b720 con 0x7f0b6c155f70 2024-03-28T23:31:25.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.442+0000 7f0b697fa640 1 -- 172.21.15.161:0/2898487851 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6400aa50 con 0x7f0b6c155f70 2024-03-28T23:31:25.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.442+0000 7f0b72f47640 1 -- 172.21.15.161:0/2898487851 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b6c0ad000 con 0x7f0b6c155f70 2024-03-28T23:31:25.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.443+0000 7f0b72f47640 1 -- 172.21.15.161:0/2898487851 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b6c04f4b0 con 0x7f0b6c155f70 2024-03-28T23:31:25.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.444+0000 7f0b697fa640 1 -- 172.21.15.161:0/2898487851 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0b6400abf0 con 0x7f0b6c155f70 2024-03-28T23:31:25.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.444+0000 7f0b697fa640 1 --2- 172.21.15.161:0/2898487851 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0b5006b4d0 0x7f0b5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:25.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.444+0000 7f0b70ce3640 1 --2- 172.21.15.161:0/2898487851 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0b5006b4d0 0x7f0b5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:25.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.444+0000 7f0b697fa640 1 -- 172.21.15.161:0/2898487851 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5421+0+0 (secure 0 0 0) 0x7f0b64089ed0 con 0x7f0b6c155f70 2024-03-28T23:31:25.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.445+0000 7f0b70ce3640 1 --2- 172.21.15.161:0/2898487851 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0b5006b4d0 0x7f0b5006d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0b60034000 tx=0x7f0b6002f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:25.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.449+0000 7f0b697fa640 1 -- 172.21.15.161:0/2898487851 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0b640588b0 con 0x7f0b6c155f70 2024-03-28T23:31:25.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.623+0000 7f0b72f47640 1 -- 172.21.15.161:0/2898487851 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f0b6c0c5f80 con 0x7f0b6c155f70 2024-03-28T23:31:25.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.626+0000 7f0b697fa640 1 -- 172.21.15.161:0/2898487851 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) ==== 104+0+0 (secure 0 0 0) 0x7f0b6405c950 con 0x7f0b6c155f70 2024-03-28T23:31:25.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.628+0000 7f0b72f47640 1 -- 172.21.15.161:0/2898487851 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0b5006b4d0 msgr2=0x7f0b5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:25.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.628+0000 7f0b72f47640 1 --2- 172.21.15.161:0/2898487851 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0b5006b4d0 0x7f0b5006d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0b60034000 tx=0x7f0b6002f000 comp rx=0 tx=0).stop 2024-03-28T23:31:25.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.628+0000 7f0b72f47640 1 -- 172.21.15.161:0/2898487851 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b6c155f70 msgr2=0x7f0b6c1ee170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:25.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.628+0000 7f0b72f47640 1 --2- 172.21.15.161:0/2898487851 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b6c155f70 0x7f0b6c1ee170 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0b6400c790 tx=0x7f0b6400cc50 comp rx=0 tx=0).stop 2024-03-28T23:31:25.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.628+0000 7f0b72f47640 1 -- 172.21.15.161:0/2898487851 shutdown_connections 2024-03-28T23:31:25.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.628+0000 7f0b72f47640 1 --2- 172.21.15.161:0/2898487851 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0b5006b4d0 0x7f0b5006d990 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:25.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.628+0000 7f0b72f47640 1 --2- 172.21.15.161:0/2898487851 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b6c155f70 0x7f0b6c1ee170 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:25.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.628+0000 7f0b72f47640 1 --2- 172.21.15.161:0/2898487851 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0b6c072150 0x7f0b6c0ab8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:25.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.628+0000 7f0b72f47640 1 --2- 172.21.15.161:0/2898487851 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b6c06d510 0x7f0b6c1522c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:25.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.628+0000 7f0b72f47640 1 -- 172.21.15.161:0/2898487851 >> 172.21.15.161:0/2898487851 conn(0x7f0b6c0a97c0 msgr2=0x7f0b6c0c77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:25.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.629+0000 7f0b72f47640 1 -- 172.21.15.161:0/2898487851 shutdown_connections 2024-03-28T23:31:25.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.629+0000 7f0b72f47640 1 -- 172.21.15.161:0/2898487851 wait complete. 2024-03-28T23:31:25.640 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-03-28T23:31:25.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.788+0000 7fa9c5aa4640 1 Processor -- start 2024-03-28T23:31:25.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.788+0000 7fa9c5aa4640 1 -- start start 2024-03-28T23:31:25.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.789+0000 7fa9c5aa4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9c00a9960 0x7fa9c0072970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:25.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.789+0000 7fa9c5aa4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa9c00733e0 0x7fa9c014eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:25.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.789+0000 7fa9c5aa4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9c00abc00 con 0x7fa9c00c7330 2024-03-28T23:31:25.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.789+0000 7fa9c5aa4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fa9c00aa3a0 con 0x7fa9c00733e0 2024-03-28T23:31:25.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.789+0000 7fa9c5aa4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa9c006bf70 con 0x7fa9c00a9960 2024-03-28T23:31:25.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.789+0000 7fa9bffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa9c00733e0 0x7fa9c014eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:25.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.789+0000 7fa9bf7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9c00a9960 0x7fa9c0072970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:25.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.789+0000 7fa9bf7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9c00a9960 0x7fa9c0072970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:44950/0 (socket says 172.21.15.161:44950) 2024-03-28T23:31:25.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.789+0000 7fa9bf7fe640 1 -- 172.21.15.161:0/2020022294 learned_addr learned my addr 172.21.15.161:0/2020022294 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:25.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.789+0000 7fa9bffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa9c00733e0 0x7fa9c014eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:56662/0 (socket says 172.21.15.161:56662) 2024-03-28T23:31:25.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.789+0000 7fa9bffff640 1 -- 172.21.15.161:0/2020022294 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9c00a9960 msgr2=0x7fa9c0072970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:25.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.789+0000 7fa9bffff640 1 --2- 172.21.15.161:0/2020022294 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9c00a9960 0x7fa9c0072970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:25.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.789+0000 7fa9bffff640 1 -- 172.21.15.161:0/2020022294 >> v1:172.21.15.170:6789/0 conn(0x7fa9c00c7330 legacy=0x7fa9c014e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:25.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.790+0000 7fa9bffff640 1 -- 172.21.15.161:0/2020022294 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9c006bcd0 con 0x7fa9c00733e0 2024-03-28T23:31:25.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.790+0000 7fa9bf7fe640 1 --2- 172.21.15.161:0/2020022294 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9c00a9960 0x7fa9c0072970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:25.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.790+0000 7fa9bffff640 1 --2- 172.21.15.161:0/2020022294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9c00733e0 0x7fa9c014eb70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8009d90 tx=0x7fa9b802bbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=895c06d24c19c724 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:25.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.790+0000 7fa9be7fc640 1 -- 172.21.15.161:0/2020022294 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b802d910 con 0x7fa9c00733e0 2024-03-28T23:31:25.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.790+0000 7fa9be7fc640 1 -- 172.21.15.161:0/2020022294 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9b802dab0 con 0x7fa9c00733e0 2024-03-28T23:31:25.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.790+0000 7fa9be7fc640 1 -- 172.21.15.161:0/2020022294 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b802c430 con 0x7fa9c00733e0 2024-03-28T23:31:25.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.791+0000 7fa9c5aa4640 1 -- 172.21.15.161:0/2020022294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9c00733e0 msgr2=0x7fa9c014eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:25.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.791+0000 7fa9c5aa4640 1 --2- 172.21.15.161:0/2020022294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9c00733e0 0x7fa9c014eb70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8009d90 tx=0x7fa9b802bbd0 comp rx=0 tx=0).stop 2024-03-28T23:31:25.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.791+0000 7fa9c5aa4640 1 -- 172.21.15.161:0/2020022294 shutdown_connections 2024-03-28T23:31:25.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.791+0000 7fa9c5aa4640 1 --2- 172.21.15.161:0/2020022294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9c00733e0 0x7fa9c014eb70 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:25.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.791+0000 7fa9c5aa4640 1 --2- 172.21.15.161:0/2020022294 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9c00a9960 0x7fa9c0072970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:25.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.791+0000 7fa9c5aa4640 1 -- 172.21.15.161:0/2020022294 >> 172.21.15.161:0/2020022294 conn(0x7fa9c00a83f0 msgr2=0x7fa9c00a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:25.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.791+0000 7fa9c5aa4640 1 -- 172.21.15.161:0/2020022294 shutdown_connections 2024-03-28T23:31:25.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.791+0000 7fa9c5aa4640 1 -- 172.21.15.161:0/2020022294 wait complete. 2024-03-28T23:31:25.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.792+0000 7fa9c5aa4640 1 Processor -- start 2024-03-28T23:31:25.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.792+0000 7fa9c5aa4640 1 -- start start 2024-03-28T23:31:25.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.793+0000 7fa9c5aa4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9c00733e0 0x7fa9c014e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:25.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.793+0000 7fa9c5aa4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9c00a9960 0x7fa9c0152d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:25.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.793+0000 7fa9c5aa4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9c00c7330 0x7fa9c014ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:25.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.793+0000 7fa9c5aa4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c00ab820 con 0x7fa9c00c7330 2024-03-28T23:31:25.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.793+0000 7fa9c5aa4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c0152250 con 0x7fa9c00733e0 2024-03-28T23:31:25.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.793+0000 7fa9c5aa4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa9c00ab9a0 con 0x7fa9c00a9960 2024-03-28T23:31:25.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.793+0000 7fa9bffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9c00c7330 0x7fa9c014ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:25.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.793+0000 7fa9bffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9c00c7330 0x7fa9c014ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35406/0 (socket says 172.21.15.161:35406) 2024-03-28T23:31:25.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.793+0000 7fa9bffff640 1 -- 172.21.15.161:0/2168492232 learned_addr learned my addr 172.21.15.161:0/2168492232 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:25.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.793+0000 7fa9bf7fe640 1 --2- 172.21.15.161:0/2168492232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9c00733e0 0x7fa9c014e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:25.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.793+0000 7fa9beffd640 1 --2- 172.21.15.161:0/2168492232 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9c00a9960 0x7fa9c0152d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:25.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.794+0000 7fa9bffff640 1 -- 172.21.15.161:0/2168492232 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9c00a9960 msgr2=0x7fa9c0152d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:25.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.794+0000 7fa9bffff640 1 --2- 172.21.15.161:0/2168492232 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9c00a9960 0x7fa9c0152d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:25.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.794+0000 7fa9bffff640 1 -- 172.21.15.161:0/2168492232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9c00733e0 msgr2=0x7fa9c014e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:25.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.794+0000 7fa9bffff640 1 --2- 172.21.15.161:0/2168492232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9c00733e0 0x7fa9c014e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:25.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.794+0000 7fa9bffff640 1 -- 172.21.15.161:0/2168492232 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9c0152860 con 0x7fa9c00c7330 2024-03-28T23:31:25.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.794+0000 7fa9bf7fe640 1 --2- 172.21.15.161:0/2168492232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9c00733e0 0x7fa9c014e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:25.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.794+0000 7fa9bffff640 1 --2- 172.21.15.161:0/2168492232 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9c00c7330 0x7fa9c014ba10 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8009eb0 tx=0x7fa9b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:25.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.794+0000 7fa9a7fff640 1 -- 172.21.15.161:0/2168492232 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b8045070 con 0x7fa9c00c7330 2024-03-28T23:31:25.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.794+0000 7fa9a7fff640 1 -- 172.21.15.161:0/2168492232 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9b803f430 con 0x7fa9c00c7330 2024-03-28T23:31:25.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.794+0000 7fa9c5aa4640 1 -- 172.21.15.161:0/2168492232 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9c01559d0 con 0x7fa9c00c7330 2024-03-28T23:31:25.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.794+0000 7fa9a7fff640 1 -- 172.21.15.161:0/2168492232 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b8009450 con 0x7fa9c00c7330 2024-03-28T23:31:25.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.794+0000 7fa9c5aa4640 1 -- 172.21.15.161:0/2168492232 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9c014c910 con 0x7fa9c00c7330 2024-03-28T23:31:25.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.795+0000 7fa9c5aa4640 1 -- 172.21.15.161:0/2168492232 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9c006b990 con 0x7fa9c00c7330 2024-03-28T23:31:25.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.795+0000 7fa9a7fff640 1 -- 172.21.15.161:0/2168492232 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa9b8048070 con 0x7fa9c00c7330 2024-03-28T23:31:25.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.796+0000 7fa9a7fff640 1 --2- 172.21.15.161:0/2168492232 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa9ac06b4d0 0x7fa9ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:25.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.796+0000 7fa9bf7fe640 1 --2- 172.21.15.161:0/2168492232 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa9ac06b4d0 0x7fa9ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:25.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.796+0000 7fa9a7fff640 1 -- 172.21.15.161:0/2168492232 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5421+0+0 (secure 0 0 0) 0x7fa9b80acb00 con 0x7fa9c00c7330 2024-03-28T23:31:25.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.796+0000 7fa9bf7fe640 1 --2- 172.21.15.161:0/2168492232 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa9ac06b4d0 0x7fa9ac06d990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0009630 tx=0x7fa9b0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:25.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.800+0000 7fa9a7fff640 1 -- 172.21.15.161:0/2168492232 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa9c006b990 con 0x7fa9c00c7330 2024-03-28T23:31:25.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.972+0000 7fa9c5aa4640 1 -- 172.21.15.161:0/2168492232 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) -- 0x7fa9c004f4b0 con 0x7fa9c00c7330 2024-03-28T23:31:25.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.973+0000 7fa9a7fff640 1 -- 172.21.15.161:0/2168492232 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v14) ==== 96+0+0 (secure 0 0 0) 0x7fa9b807bab0 con 0x7fa9c00c7330 2024-03-28T23:31:25.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.974+0000 7fa9c5aa4640 1 -- 172.21.15.161:0/2168492232 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa9ac06b4d0 msgr2=0x7fa9ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:25.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.974+0000 7fa9c5aa4640 1 --2- 172.21.15.161:0/2168492232 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa9ac06b4d0 0x7fa9ac06d990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0009630 tx=0x7fa9b0008040 comp rx=0 tx=0).stop 2024-03-28T23:31:25.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.974+0000 7fa9c5aa4640 1 -- 172.21.15.161:0/2168492232 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9c00c7330 msgr2=0x7fa9c014ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:25.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.974+0000 7fa9c5aa4640 1 --2- 172.21.15.161:0/2168492232 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9c00c7330 0x7fa9c014ba10 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8009eb0 tx=0x7fa9b8004a40 comp rx=0 tx=0).stop 2024-03-28T23:31:25.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.974+0000 7fa9c5aa4640 1 -- 172.21.15.161:0/2168492232 shutdown_connections 2024-03-28T23:31:25.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.974+0000 7fa9c5aa4640 1 --2- 172.21.15.161:0/2168492232 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa9ac06b4d0 0x7fa9ac06d990 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:25.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.974+0000 7fa9c5aa4640 1 --2- 172.21.15.161:0/2168492232 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9c00c7330 0x7fa9c014ba10 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:25.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.974+0000 7fa9c5aa4640 1 --2- 172.21.15.161:0/2168492232 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9c00a9960 0x7fa9c0152d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:25.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.974+0000 7fa9c5aa4640 1 --2- 172.21.15.161:0/2168492232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9c00733e0 0x7fa9c014e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:25.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.975+0000 7fa9c5aa4640 1 -- 172.21.15.161:0/2168492232 >> 172.21.15.161:0/2168492232 conn(0x7fa9c00a83f0 msgr2=0x7fa9c00a7b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:25.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.975+0000 7fa9c5aa4640 1 -- 172.21.15.161:0/2168492232 shutdown_connections 2024-03-28T23:31:25.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:25.975+0000 7fa9c5aa4640 1 -- 172.21.15.161:0/2168492232 wait complete. 2024-03-28T23:31:25.986 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop' 2024-03-28T23:31:26.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.086+0000 7fe9ee9a0640 1 Processor -- start 2024-03-28T23:31:26.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.086+0000 7fe9ee9a0640 1 -- start start 2024-03-28T23:31:26.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.086+0000 7fe9ee9a0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9e8152500 0x7fe9e80c2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:26.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.086+0000 7fe9ee9a0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe9e80725d0 0x7fe9e8072a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:26.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.086+0000 7fe9ee9a0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9e80c3550 con 0x7fe9e8155ab0 2024-03-28T23:31:26.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.086+0000 7fe9ee9a0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fe9e80c7060 con 0x7fe9e80725d0 2024-03-28T23:31:26.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.086+0000 7fe9ee9a0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe9e80c3830 con 0x7fe9e8152500 2024-03-28T23:31:26.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.087+0000 7fe9ed19d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe9e8155ab0 0x7fe9e80c6950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:46108/0 (socket says 172.21.15.161:46108) 2024-03-28T23:31:26.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.087+0000 7fe9ed19d640 1 -- 172.21.15.161:0/506622034 learned_addr learned my addr 172.21.15.161:0/506622034 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:26.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.087+0000 7fe9ee19f640 1 --2- 172.21.15.161:0/506622034 >> v2:172.21.15.170:3300/0 conn(0x7fe9e80725d0 0x7fe9e8072a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:26.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.087+0000 7fe9ed99e640 1 --2- 172.21.15.161:0/506622034 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9e8152500 0x7fe9e80c2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:26.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.087+0000 7fe9ec99c640 1 -- 172.21.15.161:0/506622034 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4087084991 0 0) 0x7fe9e80c3550 con 0x7fe9e8155ab0 2024-03-28T23:31:26.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.087+0000 7fe9ec99c640 1 -- 172.21.15.161:0/506622034 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9d0003520 con 0x7fe9e8155ab0 2024-03-28T23:31:26.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.087+0000 7fe9ec99c640 1 -- 172.21.15.161:0/506622034 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 450962960 0 0) 0x7fe9d0003520 con 0x7fe9e8155ab0 2024-03-28T23:31:26.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.087+0000 7fe9ec99c640 1 -- 172.21.15.161:0/506622034 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9e8152500 msgr2=0x7fe9e80c2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:26.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.087+0000 7fe9ec99c640 1 --2- 172.21.15.161:0/506622034 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9e8152500 0x7fe9e80c2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:26.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.087+0000 7fe9ec99c640 1 -- 172.21.15.161:0/506622034 >> v2:172.21.15.170:3300/0 conn(0x7fe9e80725d0 msgr2=0x7fe9e8072a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:26.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.087+0000 7fe9ec99c640 1 --2- 172.21.15.161:0/506622034 >> v2:172.21.15.170:3300/0 conn(0x7fe9e80725d0 0x7fe9e8072a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:26.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.087+0000 7fe9ec99c640 1 -- 172.21.15.161:0/506622034 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9e80c1c70 con 0x7fe9e8155ab0 2024-03-28T23:31:26.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.087+0000 7fe9ec99c640 1 -- 172.21.15.161:0/506622034 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fe9dc001670 con 0x7fe9e8155ab0 2024-03-28T23:31:26.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.087+0000 7fe9ec99c640 1 -- 172.21.15.161:0/506622034 >> v1:172.21.15.161:6789/0 conn(0x7fe9e8155ab0 legacy=0x7fe9e80c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:26.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.087+0000 7fe9ec99c640 1 --2- 172.21.15.161:0/506622034 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9d00039b0 0x7fe9d0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:26.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.087+0000 7fe9ec99c640 1 --2- 172.21.15.161:0/506622034 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9e8152500 0x7fe9d0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:26.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.087+0000 7fe9ec99c640 1 --2- 172.21.15.161:0/506622034 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9d0004d40 0x7fe9d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:26.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.087+0000 7fe9ec99c640 1 -- 172.21.15.161:0/506622034 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e80c7060 con 0x7fe9d0004d40 2024-03-28T23:31:26.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.087+0000 7fe9ec99c640 1 -- 172.21.15.161:0/506622034 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e80c3830 con 0x7fe9d00039b0 2024-03-28T23:31:26.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.087+0000 7fe9ec99c640 1 -- 172.21.15.161:0/506622034 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe9d000b9c0 con 0x7fe9e8152500 2024-03-28T23:31:26.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.087+0000 7fe9ee19f640 1 --2- 172.21.15.161:0/506622034 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9d0004d40 0x7fe9d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:26.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.087+0000 7fe9ed99e640 1 --2- 172.21.15.161:0/506622034 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9d00039b0 0x7fe9d0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:26.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.087+0000 7fe9ed19d640 1 --2- 172.21.15.161:0/506622034 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9e8152500 0x7fe9d0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:26.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.088+0000 7fe9ee19f640 1 -- 172.21.15.161:0/506622034 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9e8152500 msgr2=0x7fe9d0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:26.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.088+0000 7fe9ee19f640 1 --2- 172.21.15.161:0/506622034 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9e8152500 0x7fe9d0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:26.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.088+0000 7fe9ee19f640 1 -- 172.21.15.161:0/506622034 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9d00039b0 msgr2=0x7fe9d0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:26.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.088+0000 7fe9ee19f640 1 --2- 172.21.15.161:0/506622034 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9d00039b0 0x7fe9d0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:26.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.088+0000 7fe9ee19f640 1 -- 172.21.15.161:0/506622034 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9d0003520 con 0x7fe9d0004d40 2024-03-28T23:31:26.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.088+0000 7fe9ed99e640 1 --2- 172.21.15.161:0/506622034 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9d00039b0 0x7fe9d0003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:26.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.088+0000 7fe9ee19f640 1 --2- 172.21.15.161:0/506622034 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9d0004d40 0x7fe9d00051c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8009ac0 tx=0x7fe9d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c7c3dd86f905b03 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:26.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.088+0000 7fe9ec99c640 1 -- 172.21.15.161:0/506622034 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8039070 con 0x7fe9d0004d40 2024-03-28T23:31:26.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.089+0000 7fe9ec99c640 1 -- 172.21.15.161:0/506622034 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9d802c710 con 0x7fe9d0004d40 2024-03-28T23:31:26.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.089+0000 7fe9ec99c640 1 -- 172.21.15.161:0/506622034 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8009690 con 0x7fe9d0004d40 2024-03-28T23:31:26.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.089+0000 7fe9ee9a0640 1 -- 172.21.15.161:0/506622034 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9d0004d40 msgr2=0x7fe9d00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:26.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.089+0000 7fe9ee9a0640 1 --2- 172.21.15.161:0/506622034 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9d0004d40 0x7fe9d00051c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8009ac0 tx=0x7fe9d0005830 comp rx=0 tx=0).stop 2024-03-28T23:31:26.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.089+0000 7fe9ee19f640 1 -- 172.21.15.161:0/506622034 reap_dead start 2024-03-28T23:31:26.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.089+0000 7fe9ee9a0640 1 -- 172.21.15.161:0/506622034 shutdown_connections 2024-03-28T23:31:26.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.089+0000 7fe9ee9a0640 1 -- 172.21.15.161:0/506622034 >> 172.21.15.161:0/506622034 conn(0x7fe9e80aa5f0 msgr2=0x7fe9e80aa9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:26.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.089+0000 7fe9ee9a0640 1 -- 172.21.15.161:0/506622034 shutdown_connections 2024-03-28T23:31:26.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.089+0000 7fe9ee9a0640 1 -- 172.21.15.161:0/506622034 wait complete. 2024-03-28T23:31:26.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.090+0000 7fe9ee9a0640 1 Processor -- start 2024-03-28T23:31:26.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.090+0000 7fe9ee9a0640 1 -- start start 2024-03-28T23:31:26.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.091+0000 7fe9ee9a0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9e81510e0 0x7fe9e806f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:26.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.091+0000 7fe9ee9a0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9e81591a0 0x7fe9e806fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:26.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.091+0000 7fe9ee9a0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9e81eaa00 0x7fe9e81eae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:26.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.091+0000 7fe9ee9a0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e80c3240 con 0x7fe9e81591a0 2024-03-28T23:31:26.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.091+0000 7fe9ee9a0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e80c2230 con 0x7fe9e81510e0 2024-03-28T23:31:26.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.091+0000 7fe9ee9a0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe9e8155f30 con 0x7fe9e81eaa00 2024-03-28T23:31:26.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.091+0000 7fe9ed19d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9e81591a0 0x7fe9e806fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:26.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.091+0000 7fe9ed19d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9e81591a0 0x7fe9e806fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35430/0 (socket says 172.21.15.161:35430) 2024-03-28T23:31:26.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.091+0000 7fe9ed19d640 1 -- 172.21.15.161:0/3646515598 learned_addr learned my addr 172.21.15.161:0/3646515598 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:26.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.091+0000 7fe9ed19d640 1 -- 172.21.15.161:0/3646515598 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9e81eaa00 msgr2=0x7fe9e81eae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:26.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.091+0000 7fe9ed19d640 1 --2- 172.21.15.161:0/3646515598 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9e81eaa00 0x7fe9e81eae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:26.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.091+0000 7fe9ed19d640 1 -- 172.21.15.161:0/3646515598 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9e81510e0 msgr2=0x7fe9e806f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:26.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.091+0000 7fe9ed19d640 1 --2- 172.21.15.161:0/3646515598 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9e81510e0 0x7fe9e806f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:26.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.091+0000 7fe9ed19d640 1 -- 172.21.15.161:0/3646515598 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9e806ff40 con 0x7fe9e81591a0 2024-03-28T23:31:26.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.091+0000 7fe9ed19d640 1 --2- 172.21.15.161:0/3646515598 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9e81591a0 0x7fe9e806fa00 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc0072f0 tx=0x7fe9dc0077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:26.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.091+0000 7fe9e67fc640 1 -- 172.21.15.161:0/3646515598 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc00f940 con 0x7fe9e81591a0 2024-03-28T23:31:26.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.092+0000 7fe9e67fc640 1 -- 172.21.15.161:0/3646515598 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9dc018070 con 0x7fe9e81591a0 2024-03-28T23:31:26.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.092+0000 7fe9e67fc640 1 -- 172.21.15.161:0/3646515598 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc00fd40 con 0x7fe9e81591a0 2024-03-28T23:31:26.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.092+0000 7fe9ee9a0640 1 -- 172.21.15.161:0/3646515598 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9e81ee530 con 0x7fe9e81591a0 2024-03-28T23:31:26.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.092+0000 7fe9ee9a0640 1 -- 172.21.15.161:0/3646515598 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9e8158860 con 0x7fe9e81591a0 2024-03-28T23:31:26.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.093+0000 7fe9ee9a0640 1 -- 172.21.15.161:0/3646515598 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9e804f4b0 con 0x7fe9e81591a0 2024-03-28T23:31:26.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.093+0000 7fe9e67fc640 1 -- 172.21.15.161:0/3646515598 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe9dc01d460 con 0x7fe9e81591a0 2024-03-28T23:31:26.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.093+0000 7fe9e67fc640 1 --2- 172.21.15.161:0/3646515598 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe9d406b500 0x7fe9d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:26.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.093+0000 7fe9ed99e640 1 --2- 172.21.15.161:0/3646515598 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe9d406b500 0x7fe9d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:26.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.093+0000 7fe9e67fc640 1 -- 172.21.15.161:0/3646515598 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5421+0+0 (secure 0 0 0) 0x7fe9dc08a120 con 0x7fe9e81591a0 2024-03-28T23:31:26.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.094+0000 7fe9ed99e640 1 --2- 172.21.15.161:0/3646515598 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe9d406b500 0x7fe9d406d9c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0011420 tx=0x7fe9e0009750 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:26.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.098+0000 7fe9e67fc640 1 -- 172.21.15.161:0/3646515598 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe9dc059150 con 0x7fe9e81591a0 2024-03-28T23:31:26.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:26.335+0000 7fe9ee9a0640 1 -- 172.21.15.161:0/3646515598 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop"]} v 0) -- 0x7fe9e8073260 con 0x7fe9e81591a0 2024-03-28T23:31:27.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.316+0000 7fe9e67fc640 1 -- 172.21.15.161:0/3646515598 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop"]}]=0 v109) ==== 143+0+0 (secure 0 0 0) 0x7fe9dc058a50 con 0x7fe9e81591a0 2024-03-28T23:31:27.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.319+0000 7fe9ee9a0640 1 -- 172.21.15.161:0/3646515598 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe9d406b500 msgr2=0x7fe9d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:27.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.319+0000 7fe9ee9a0640 1 --2- 172.21.15.161:0/3646515598 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe9d406b500 0x7fe9d406d9c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0011420 tx=0x7fe9e0009750 comp rx=0 tx=0).stop 2024-03-28T23:31:27.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.319+0000 7fe9ee9a0640 1 -- 172.21.15.161:0/3646515598 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9e81591a0 msgr2=0x7fe9e806fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:27.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.319+0000 7fe9ee9a0640 1 --2- 172.21.15.161:0/3646515598 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9e81591a0 0x7fe9e806fa00 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc0072f0 tx=0x7fe9dc0077b0 comp rx=0 tx=0).stop 2024-03-28T23:31:27.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.319+0000 7fe9ee9a0640 1 -- 172.21.15.161:0/3646515598 shutdown_connections 2024-03-28T23:31:27.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.319+0000 7fe9ee9a0640 1 --2- 172.21.15.161:0/3646515598 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe9d406b500 0x7fe9d406d9c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:27.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.319+0000 7fe9ee9a0640 1 --2- 172.21.15.161:0/3646515598 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9e81591a0 0x7fe9e806fa00 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:27.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.319+0000 7fe9ee9a0640 1 --2- 172.21.15.161:0/3646515598 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9e81eaa00 0x7fe9e81eae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:27.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.319+0000 7fe9ee9a0640 1 --2- 172.21.15.161:0/3646515598 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9e81510e0 0x7fe9e806f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:27.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.319+0000 7fe9ee9a0640 1 -- 172.21.15.161:0/3646515598 >> 172.21.15.161:0/3646515598 conn(0x7fe9e80aa5f0 msgr2=0x7fe9e8155110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:27.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.319+0000 7fe9ee9a0640 1 -- 172.21.15.161:0/3646515598 shutdown_connections 2024-03-28T23:31:27.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.319+0000 7fe9ee9a0640 1 -- 172.21.15.161:0/3646515598 wait complete. 2024-03-28T23:31:27.333 INFO:tasks.cephfs_test_runner:That unsetting mds_join_fs will not cause failover if no high-affinity standby exists. ... ok 2024-03-28T23:31:27.333 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_last_resort (tasks.cephfs.test_failover.TestClusterAffinity) That a standby with mds_join_fs set to another fs is still used if necessary. 2024-03-28T23:31:27.334 INFO:tasks.cephfs_test_runner:test_join_fs_last_resort (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-28T23:31:27.334 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:31:27.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.441+0000 7ff315525640 1 Processor -- start 2024-03-28T23:31:27.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.441+0000 7ff315525640 1 -- start start 2024-03-28T23:31:27.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.441+0000 7ff315525640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff310152a80 0x7ff310155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:27.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.441+0000 7ff315525640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff31006d9f0 0x7ff3100c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:27.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.441+0000 7ff315525640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3100c98e0 con 0x7ff31006cb80 2024-03-28T23:31:27.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.441+0000 7ff315525640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff310156250 con 0x7ff310152a80 2024-03-28T23:31:27.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.441+0000 7ff315525640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff31006c6f0 con 0x7ff31006d9f0 2024-03-28T23:31:27.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.441+0000 7ff30effd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff310152a80 0x7ff310155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:27.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.442+0000 7ff30effd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff310152a80 0x7ff310155d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35432/0 (socket says 172.21.15.161:35432) 2024-03-28T23:31:27.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.442+0000 7ff30effd640 1 -- 172.21.15.161:0/1232144447 learned_addr learned my addr 172.21.15.161:0/1232144447 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:27.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.442+0000 7ff30e7fc640 1 --2- 172.21.15.161:0/1232144447 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff31006d9f0 0x7ff3100c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:27.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.442+0000 7ff30effd640 1 -- 172.21.15.161:0/1232144447 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff31006d9f0 msgr2=0x7ff3100c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:27.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.442+0000 7ff30effd640 1 --2- 172.21.15.161:0/1232144447 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff31006d9f0 0x7ff3100c93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:27.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.442+0000 7ff30effd640 1 -- 172.21.15.161:0/1232144447 >> v1:172.21.15.170:6789/0 conn(0x7ff31006cb80 legacy=0x7ff3100c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:27.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.442+0000 7ff30effd640 1 -- 172.21.15.161:0/1232144447 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff31006c3c0 con 0x7ff310152a80 2024-03-28T23:31:27.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.443+0000 7ff30effd640 1 --2- 172.21.15.161:0/1232144447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff310152a80 0x7ff310155d10 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff304009870 tx=0x7ff30402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b56d69636eb23d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:27.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.443+0000 7ff30dffb640 1 -- 172.21.15.161:0/1232144447 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3040317c0 con 0x7ff310152a80 2024-03-28T23:31:27.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.443+0000 7ff30dffb640 1 -- 172.21.15.161:0/1232144447 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff304037070 con 0x7ff310152a80 2024-03-28T23:31:27.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.443+0000 7ff30dffb640 1 -- 172.21.15.161:0/1232144447 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff304031b80 con 0x7ff310152a80 2024-03-28T23:31:27.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.443+0000 7ff315525640 1 -- 172.21.15.161:0/1232144447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff310152a80 msgr2=0x7ff310155d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:27.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.443+0000 7ff315525640 1 --2- 172.21.15.161:0/1232144447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff310152a80 0x7ff310155d10 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff304009870 tx=0x7ff30402b860 comp rx=0 tx=0).stop 2024-03-28T23:31:27.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.443+0000 7ff315525640 1 -- 172.21.15.161:0/1232144447 shutdown_connections 2024-03-28T23:31:27.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.443+0000 7ff315525640 1 --2- 172.21.15.161:0/1232144447 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff31006d9f0 0x7ff3100c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:27.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.443+0000 7ff315525640 1 --2- 172.21.15.161:0/1232144447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff310152a80 0x7ff310155d10 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:27.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.443+0000 7ff315525640 1 -- 172.21.15.161:0/1232144447 >> 172.21.15.161:0/1232144447 conn(0x7ff3100ac720 msgr2=0x7ff3100acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:27.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.443+0000 7ff315525640 1 -- 172.21.15.161:0/1232144447 shutdown_connections 2024-03-28T23:31:27.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.443+0000 7ff315525640 1 -- 172.21.15.161:0/1232144447 wait complete. 2024-03-28T23:31:27.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.445+0000 7ff315525640 1 Processor -- start 2024-03-28T23:31:27.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.445+0000 7ff315525640 1 -- start start 2024-03-28T23:31:27.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.445+0000 7ff315525640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff31006cb80 0x7ff3100c0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:27.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.445+0000 7ff315525640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff31006d9f0 0x7ff310147b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:27.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.445+0000 7ff315525640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff310152a80 0x7ff3101480d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:27.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.445+0000 7ff315525640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff3100c58b0 con 0x7ff31006cb80 2024-03-28T23:31:27.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.445+0000 7ff315525640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff3100c55d0 con 0x7ff310152a80 2024-03-28T23:31:27.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.445+0000 7ff315525640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff3100c7bd0 con 0x7ff31006d9f0 2024-03-28T23:31:27.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.445+0000 7ff30effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff31006cb80 0x7ff3100c0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:27.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.445+0000 7ff30effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff31006cb80 0x7ff3100c0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35442/0 (socket says 172.21.15.161:35442) 2024-03-28T23:31:27.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.445+0000 7ff30effd640 1 -- 172.21.15.161:0/141321918 learned_addr learned my addr 172.21.15.161:0/141321918 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:27.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.446+0000 7ff30e7fc640 1 --2- 172.21.15.161:0/141321918 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff31006d9f0 0x7ff310147b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:27.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.446+0000 7ff30f7fe640 1 --2- 172.21.15.161:0/141321918 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff310152a80 0x7ff3101480d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:27.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.446+0000 7ff30effd640 1 -- 172.21.15.161:0/141321918 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff31006d9f0 msgr2=0x7ff310147b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:27.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.446+0000 7ff30effd640 1 --2- 172.21.15.161:0/141321918 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff31006d9f0 0x7ff310147b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:27.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.446+0000 7ff30effd640 1 -- 172.21.15.161:0/141321918 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff310152a80 msgr2=0x7ff3101480d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:27.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.446+0000 7ff30effd640 1 --2- 172.21.15.161:0/141321918 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff310152a80 0x7ff3101480d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:27.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.446+0000 7ff30effd640 1 -- 172.21.15.161:0/141321918 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff310148990 con 0x7ff31006cb80 2024-03-28T23:31:27.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.446+0000 7ff30e7fc640 1 --2- 172.21.15.161:0/141321918 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff31006d9f0 0x7ff310147b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:27.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.446+0000 7ff30f7fe640 1 --2- 172.21.15.161:0/141321918 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff310152a80 0x7ff3101480d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:27.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.446+0000 7ff30effd640 1 --2- 172.21.15.161:0/141321918 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff31006cb80 0x7ff3100c0d80 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff30402b8a0 tx=0x7ff3040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:27.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.446+0000 7ff2f77fe640 1 -- 172.21.15.161:0/141321918 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff304032860 con 0x7ff31006cb80 2024-03-28T23:31:27.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.446+0000 7ff2f77fe640 1 -- 172.21.15.161:0/141321918 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff304032a20 con 0x7ff31006cb80 2024-03-28T23:31:27.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.446+0000 7ff315525640 1 -- 172.21.15.161:0/141321918 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff310158b50 con 0x7ff31006cb80 2024-03-28T23:31:27.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.446+0000 7ff2f77fe640 1 -- 172.21.15.161:0/141321918 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff304032e10 con 0x7ff31006cb80 2024-03-28T23:31:27.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.446+0000 7ff315525640 1 -- 172.21.15.161:0/141321918 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff310159110 con 0x7ff31006cb80 2024-03-28T23:31:27.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.447+0000 7ff315525640 1 -- 172.21.15.161:0/141321918 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff31004f4b0 con 0x7ff31006cb80 2024-03-28T23:31:27.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.447+0000 7ff2f77fe640 1 -- 172.21.15.161:0/141321918 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff30404c060 con 0x7ff31006cb80 2024-03-28T23:31:27.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.448+0000 7ff2f77fe640 1 --2- 172.21.15.161:0/141321918 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff2fc06b500 0x7ff2fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:27.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.448+0000 7ff30e7fc640 1 --2- 172.21.15.161:0/141321918 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff2fc06b500 0x7ff2fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:27.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.448+0000 7ff2f77fe640 1 -- 172.21.15.161:0/141321918 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5421+0+0 (secure 0 0 0) 0x7ff3040ab9e0 con 0x7ff31006cb80 2024-03-28T23:31:27.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.449+0000 7ff30e7fc640 1 --2- 172.21.15.161:0/141321918 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff2fc06b500 0x7ff2fc06d9c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff300006fd0 tx=0x7ff300008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:27.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.452+0000 7ff2f77fe640 1 -- 172.21.15.161:0/141321918 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff304040070 con 0x7ff31006cb80 2024-03-28T23:31:27.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.628+0000 7ff315525640 1 -- 172.21.15.161:0/141321918 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff3100aa8c0 con 0x7ff31006cb80 2024-03-28T23:31:27.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.629+0000 7ff2f77fe640 1 -- 172.21.15.161:0/141321918 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+13459 (secure 0 0 0) 0x7ff304039310 con 0x7ff31006cb80 2024-03-28T23:31:27.630 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:27.630 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":42,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:31:25.309426+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6834/11130685":"2024-03-29T23:31:22.302664+0000","172.21.15.170:6832/11130685":"2024-03-29T23:31:22.302664+0000","172.21.15.161:6837/2636327316":"2024-03-29T23:31:07.634564+0000","172.21.15.161:6835/2636327316":"2024-03-29T23:31:07.634564+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:31:27.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.631+0000 7ff315525640 1 -- 172.21.15.161:0/141321918 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff2fc06b500 msgr2=0x7ff2fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:27.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.631+0000 7ff315525640 1 --2- 172.21.15.161:0/141321918 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff2fc06b500 0x7ff2fc06d9c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff300006fd0 tx=0x7ff300008040 comp rx=0 tx=0).stop 2024-03-28T23:31:27.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.631+0000 7ff315525640 1 -- 172.21.15.161:0/141321918 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff31006cb80 msgr2=0x7ff3100c0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:27.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.631+0000 7ff315525640 1 --2- 172.21.15.161:0/141321918 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff31006cb80 0x7ff3100c0d80 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff30402b8a0 tx=0x7ff3040046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:27.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.631+0000 7ff315525640 1 -- 172.21.15.161:0/141321918 shutdown_connections 2024-03-28T23:31:27.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.631+0000 7ff315525640 1 --2- 172.21.15.161:0/141321918 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff2fc06b500 0x7ff2fc06d9c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:27.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.631+0000 7ff315525640 1 --2- 172.21.15.161:0/141321918 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff310152a80 0x7ff3101480d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:27.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.631+0000 7ff315525640 1 --2- 172.21.15.161:0/141321918 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff31006d9f0 0x7ff310147b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:27.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.631+0000 7ff315525640 1 --2- 172.21.15.161:0/141321918 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff31006cb80 0x7ff3100c0d80 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:27.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.631+0000 7ff315525640 1 -- 172.21.15.161:0/141321918 >> 172.21.15.161:0/141321918 conn(0x7ff3100ac720 msgr2=0x7ff3100c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:27.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.631+0000 7ff315525640 1 -- 172.21.15.161:0/141321918 shutdown_connections 2024-03-28T23:31:27.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.631+0000 7ff315525640 1 -- 172.21.15.161:0/141321918 wait complete. 2024-03-28T23:31:27.643 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:31:27.643 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:31:27.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.746+0000 7fc812bd6640 1 Processor -- start 2024-03-28T23:31:27.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.746+0000 7fc812bd6640 1 -- start start 2024-03-28T23:31:27.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.747+0000 7fc812bd6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc80c1511e0 0x7fc80c06d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:27.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.747+0000 7fc812bd6640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc80c06d610 0x7fc80c1522b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:27.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.747+0000 7fc812bd6640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc80c0c2fc0 con 0x7fc80c0c6470 2024-03-28T23:31:27.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.747+0000 7fc812bd6640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fc80c0c1290 con 0x7fc80c06d610 2024-03-28T23:31:27.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.747+0000 7fc812bd6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc80c06c8a0 con 0x7fc80c1511e0 2024-03-28T23:31:27.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.747+0000 7fc810972640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc80c1511e0 0x7fc80c06d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:27.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.747+0000 7fc810972640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc80c1511e0 0x7fc80c06d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:45032/0 (socket says 172.21.15.161:45032) 2024-03-28T23:31:27.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.747+0000 7fc810972640 1 -- 172.21.15.161:0/1445464322 learned_addr learned my addr 172.21.15.161:0/1445464322 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:27.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.747+0000 7fc811173640 1 --2- 172.21.15.161:0/1445464322 >> v2:172.21.15.170:3300/0 conn(0x7fc80c06d610 0x7fc80c1522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:27.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.747+0000 7fc810972640 1 -- 172.21.15.161:0/1445464322 >> v2:172.21.15.170:3300/0 conn(0x7fc80c06d610 msgr2=0x7fc80c1522b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:27.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.748+0000 7fc810972640 1 --2- 172.21.15.161:0/1445464322 >> v2:172.21.15.170:3300/0 conn(0x7fc80c06d610 0x7fc80c1522b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:27.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.748+0000 7fc810972640 1 -- 172.21.15.161:0/1445464322 >> v1:172.21.15.170:6789/0 conn(0x7fc80c0c6470 legacy=0x7fc80c151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:27.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.748+0000 7fc810972640 1 -- 172.21.15.161:0/1445464322 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc80c06cb80 con 0x7fc80c1511e0 2024-03-28T23:31:27.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.748+0000 7fc810972640 1 --2- 172.21.15.161:0/1445464322 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc80c1511e0 0x7fc80c06d0d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc800009870 tx=0x7fc80002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b4400f80e28837d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:27.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.748+0000 7fc80b7fe640 1 -- 172.21.15.161:0/1445464322 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8000317c0 con 0x7fc80c1511e0 2024-03-28T23:31:27.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.749+0000 7fc80b7fe640 1 -- 172.21.15.161:0/1445464322 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc800037070 con 0x7fc80c1511e0 2024-03-28T23:31:27.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.749+0000 7fc80b7fe640 1 -- 172.21.15.161:0/1445464322 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc800031b80 con 0x7fc80c1511e0 2024-03-28T23:31:27.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.749+0000 7fc812bd6640 1 -- 172.21.15.161:0/1445464322 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc80c1511e0 msgr2=0x7fc80c06d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:27.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.749+0000 7fc812bd6640 1 --2- 172.21.15.161:0/1445464322 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc80c1511e0 0x7fc80c06d0d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc800009870 tx=0x7fc80002b860 comp rx=0 tx=0).stop 2024-03-28T23:31:27.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.749+0000 7fc812bd6640 1 -- 172.21.15.161:0/1445464322 shutdown_connections 2024-03-28T23:31:27.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.749+0000 7fc812bd6640 1 --2- 172.21.15.161:0/1445464322 >> v2:172.21.15.170:3300/0 conn(0x7fc80c06d610 0x7fc80c1522b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:27.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.749+0000 7fc812bd6640 1 --2- 172.21.15.161:0/1445464322 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc80c1511e0 0x7fc80c06d0d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:27.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.749+0000 7fc812bd6640 1 -- 172.21.15.161:0/1445464322 >> 172.21.15.161:0/1445464322 conn(0x7fc80c0c3390 msgr2=0x7fc80c0c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:27.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.749+0000 7fc812bd6640 1 -- 172.21.15.161:0/1445464322 shutdown_connections 2024-03-28T23:31:27.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.749+0000 7fc812bd6640 1 -- 172.21.15.161:0/1445464322 wait complete. 2024-03-28T23:31:27.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.750+0000 7fc812bd6640 1 Processor -- start 2024-03-28T23:31:27.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.750+0000 7fc812bd6640 1 -- start start 2024-03-28T23:31:27.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.751+0000 7fc812bd6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc80c06d610 0x7fc80c155580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:27.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.751+0000 7fc812bd6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc80c1511e0 0x7fc80c0a6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:27.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.751+0000 7fc812bd6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc80c073220 0x7fc80c14acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:27.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.751+0000 7fc812bd6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc80c06c2e0 con 0x7fc80c06d610 2024-03-28T23:31:27.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.751+0000 7fc812bd6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc80c06c160 con 0x7fc80c1511e0 2024-03-28T23:31:27.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.751+0000 7fc812bd6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc80c0c5f20 con 0x7fc80c073220 2024-03-28T23:31:27.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.751+0000 7fc810972640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc80c06d610 0x7fc80c155580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:27.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.751+0000 7fc810972640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc80c06d610 0x7fc80c155580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35448/0 (socket says 172.21.15.161:35448) 2024-03-28T23:31:27.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.751+0000 7fc810972640 1 -- 172.21.15.161:0/360263466 learned_addr learned my addr 172.21.15.161:0/360263466 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:27.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.751+0000 7fc811173640 1 --2- 172.21.15.161:0/360263466 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc80c073220 0x7fc80c14acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:27.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.751+0000 7fc80bfff640 1 --2- 172.21.15.161:0/360263466 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc80c1511e0 0x7fc80c0a6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:27.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.752+0000 7fc810972640 1 -- 172.21.15.161:0/360263466 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc80c073220 msgr2=0x7fc80c14acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:27.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.752+0000 7fc810972640 1 --2- 172.21.15.161:0/360263466 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc80c073220 0x7fc80c14acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:27.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.752+0000 7fc810972640 1 -- 172.21.15.161:0/360263466 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc80c1511e0 msgr2=0x7fc80c0a6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:27.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.752+0000 7fc810972640 1 --2- 172.21.15.161:0/360263466 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc80c1511e0 0x7fc80c0a6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:27.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.752+0000 7fc810972640 1 -- 172.21.15.161:0/360263466 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc80c072550 con 0x7fc80c06d610 2024-03-28T23:31:27.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.752+0000 7fc810972640 1 --2- 172.21.15.161:0/360263466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc80c06d610 0x7fc80c155580 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc800000c00 tx=0x7fc800004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:27.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.752+0000 7fc8097fa640 1 -- 172.21.15.161:0/360263466 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80004b070 con 0x7fc80c06d610 2024-03-28T23:31:27.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.752+0000 7fc8097fa640 1 -- 172.21.15.161:0/360263466 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8000318d0 con 0x7fc80c06d610 2024-03-28T23:31:27.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.752+0000 7fc812bd6640 1 -- 172.21.15.161:0/360263466 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc80c072bc0 con 0x7fc80c06d610 2024-03-28T23:31:27.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.752+0000 7fc8097fa640 1 -- 172.21.15.161:0/360263466 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc800031c80 con 0x7fc80c06d610 2024-03-28T23:31:27.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.752+0000 7fc812bd6640 1 -- 172.21.15.161:0/360263466 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc80c0a6d60 con 0x7fc80c06d610 2024-03-28T23:31:27.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.753+0000 7fc812bd6640 1 -- 172.21.15.161:0/360263466 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc80c04f4b0 con 0x7fc80c06d610 2024-03-28T23:31:27.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.753+0000 7fc8097fa640 1 -- 172.21.15.161:0/360263466 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc80004e070 con 0x7fc80c06d610 2024-03-28T23:31:27.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.754+0000 7fc8097fa640 1 --2- 172.21.15.161:0/360263466 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc7f006b4d0 0x7fc7f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:27.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.754+0000 7fc8097fa640 1 -- 172.21.15.161:0/360263466 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5421+0+0 (secure 0 0 0) 0x7fc8000adc20 con 0x7fc80c06d610 2024-03-28T23:31:27.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.754+0000 7fc80bfff640 1 --2- 172.21.15.161:0/360263466 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc7f006b4d0 0x7fc7f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:27.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.754+0000 7fc80bfff640 1 --2- 172.21.15.161:0/360263466 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc7f006b4d0 0x7fc7f006d990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc80c1eaf60 tx=0x7fc7f8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:27.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.758+0000 7fc8097fa640 1 -- 172.21.15.161:0/360263466 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc80007c600 con 0x7fc80c06d610 2024-03-28T23:31:27.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.929+0000 7fc812bd6640 1 -- 172.21.15.161:0/360263466 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc80c0c0900 con 0x7fc80c06d610 2024-03-28T23:31:27.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.929+0000 7fc8097fa640 1 -- 172.21.15.161:0/360263466 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7fc8000806a0 con 0x7fc80c06d610 2024-03-28T23:31:27.930 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:31:27.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.931+0000 7fc812bd6640 1 -- 172.21.15.161:0/360263466 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc7f006b4d0 msgr2=0x7fc7f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:27.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.931+0000 7fc812bd6640 1 --2- 172.21.15.161:0/360263466 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc7f006b4d0 0x7fc7f006d990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc80c1eaf60 tx=0x7fc7f8009040 comp rx=0 tx=0).stop 2024-03-28T23:31:27.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.931+0000 7fc812bd6640 1 -- 172.21.15.161:0/360263466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc80c06d610 msgr2=0x7fc80c155580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:27.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.931+0000 7fc812bd6640 1 --2- 172.21.15.161:0/360263466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc80c06d610 0x7fc80c155580 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc800000c00 tx=0x7fc800004730 comp rx=0 tx=0).stop 2024-03-28T23:31:27.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.931+0000 7fc812bd6640 1 -- 172.21.15.161:0/360263466 shutdown_connections 2024-03-28T23:31:27.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.931+0000 7fc812bd6640 1 --2- 172.21.15.161:0/360263466 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc7f006b4d0 0x7fc7f006d990 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:27.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.931+0000 7fc812bd6640 1 --2- 172.21.15.161:0/360263466 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc80c073220 0x7fc80c14acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:27.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.931+0000 7fc812bd6640 1 --2- 172.21.15.161:0/360263466 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc80c1511e0 0x7fc80c0a6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:27.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.931+0000 7fc812bd6640 1 --2- 172.21.15.161:0/360263466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc80c06d610 0x7fc80c155580 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:27.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.931+0000 7fc812bd6640 1 -- 172.21.15.161:0/360263466 >> 172.21.15.161:0/360263466 conn(0x7fc80c0c3390 msgr2=0x7fc80c0c4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:27.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.932+0000 7fc812bd6640 1 -- 172.21.15.161:0/360263466 shutdown_connections 2024-03-28T23:31:27.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:27.932+0000 7fc812bd6640 1 -- 172.21.15.161:0/360263466 wait complete. 2024-03-28T23:31:27.943 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort' 2024-03-28T23:31:28.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.044+0000 7f141cb02640 1 Processor -- start 2024-03-28T23:31:28.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.044+0000 7f141cb02640 1 -- start start 2024-03-28T23:31:28.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.045+0000 7f141cb02640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14180a8fc0 0x7f1418153270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:28.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.045+0000 7f141cb02640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f14180aacb0 0x7f14181537b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:28.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.045+0000 7f141cb02640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14181567c0 con 0x7f1418150600 2024-03-28T23:31:28.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.045+0000 7f141cb02640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1418156570 con 0x7f14180aacb0 2024-03-28T23:31:28.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.045+0000 7f141cb02640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1418156aa0 con 0x7f14180a8fc0 2024-03-28T23:31:28.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.045+0000 7f1416d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f14180aacb0 0x7f14181537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:28.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.045+0000 7f1416d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f14180aacb0 0x7f14181537b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35462/0 (socket says 172.21.15.161:35462) 2024-03-28T23:31:28.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.045+0000 7f1416d9d640 1 -- 172.21.15.161:0/2799922618 learned_addr learned my addr 172.21.15.161:0/2799922618 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:28.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.045+0000 7f1415d9b640 1 --2- 172.21.15.161:0/2799922618 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14180a8fc0 0x7f1418153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:28.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.045+0000 7f1416d9d640 1 -- 172.21.15.161:0/2799922618 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14180a8fc0 msgr2=0x7f1418153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:28.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.045+0000 7f1416d9d640 1 --2- 172.21.15.161:0/2799922618 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14180a8fc0 0x7f1418153270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.045+0000 7f1416d9d640 1 -- 172.21.15.161:0/2799922618 >> v1:172.21.15.170:6789/0 conn(0x7f1418150600 legacy=0x7f14180a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:28.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.045+0000 7f1416d9d640 1 -- 172.21.15.161:0/2799922618 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1418156d80 con 0x7f14180aacb0 2024-03-28T23:31:28.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.046+0000 7f1416d9d640 1 --2- 172.21.15.161:0/2799922618 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14180aacb0 0x7f14181537b0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f1410009e50 tx=0x7f141002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c07eac1c76c977d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:28.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.046+0000 7f141559a640 1 -- 172.21.15.161:0/2799922618 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f141002ca70 con 0x7f14180aacb0 2024-03-28T23:31:28.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.046+0000 7f141559a640 1 -- 172.21.15.161:0/2799922618 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f141002cc10 con 0x7f14180aacb0 2024-03-28T23:31:28.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.046+0000 7f141559a640 1 -- 172.21.15.161:0/2799922618 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14100343f0 con 0x7f14180aacb0 2024-03-28T23:31:28.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.047+0000 7f141cb02640 1 -- 172.21.15.161:0/2799922618 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14180aacb0 msgr2=0x7f14181537b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:28.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.047+0000 7f141cb02640 1 --2- 172.21.15.161:0/2799922618 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14180aacb0 0x7f14181537b0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f1410009e50 tx=0x7f141002bed0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.047+0000 7f141cb02640 1 -- 172.21.15.161:0/2799922618 shutdown_connections 2024-03-28T23:31:28.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.047+0000 7f141cb02640 1 --2- 172.21.15.161:0/2799922618 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14180aacb0 0x7f14181537b0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.047+0000 7f141cb02640 1 --2- 172.21.15.161:0/2799922618 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14180a8fc0 0x7f1418153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.047+0000 7f141cb02640 1 -- 172.21.15.161:0/2799922618 >> 172.21.15.161:0/2799922618 conn(0x7f14180c6670 msgr2=0x7f14180c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:28.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.047+0000 7f141cb02640 1 -- 172.21.15.161:0/2799922618 shutdown_connections 2024-03-28T23:31:28.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.047+0000 7f141cb02640 1 -- 172.21.15.161:0/2799922618 wait complete. 2024-03-28T23:31:28.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.048+0000 7f141cb02640 1 Processor -- start 2024-03-28T23:31:28.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.048+0000 7f141cb02640 1 -- start start 2024-03-28T23:31:28.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.049+0000 7f141cb02640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14180a8fc0 0x7f14180c15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:28.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.049+0000 7f141cb02640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f14180aacb0 0x7f1418155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:28.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.049+0000 7f141cb02640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1418150600 0x7f14180c0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:28.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.049+0000 7f141cb02640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f14181509e0 con 0x7f14180a8fc0 2024-03-28T23:31:28.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.049+0000 7f141cb02640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1418157340 con 0x7f14180aacb0 2024-03-28T23:31:28.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.049+0000 7f141cb02640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f14180aa9e0 con 0x7f1418150600 2024-03-28T23:31:28.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.049+0000 7f141659c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14180a8fc0 0x7f14180c15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:28.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.049+0000 7f141659c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14180a8fc0 0x7f14180c15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35470/0 (socket says 172.21.15.161:35470) 2024-03-28T23:31:28.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.049+0000 7f141659c640 1 -- 172.21.15.161:0/679842707 learned_addr learned my addr 172.21.15.161:0/679842707 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:28.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.049+0000 7f1416d9d640 1 --2- 172.21.15.161:0/679842707 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1418150600 0x7f14180c0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:28.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.049+0000 7f1415d9b640 1 --2- 172.21.15.161:0/679842707 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f14180aacb0 0x7f1418155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:28.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.049+0000 7f141659c640 1 -- 172.21.15.161:0/679842707 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1418150600 msgr2=0x7f14180c0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:28.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.049+0000 7f141659c640 1 --2- 172.21.15.161:0/679842707 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1418150600 0x7f14180c0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.049+0000 7f141659c640 1 -- 172.21.15.161:0/679842707 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f14180aacb0 msgr2=0x7f1418155dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:28.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.049+0000 7f141659c640 1 --2- 172.21.15.161:0/679842707 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f14180aacb0 0x7f1418155dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.049+0000 7f141659c640 1 -- 172.21.15.161:0/679842707 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1418156310 con 0x7f14180a8fc0 2024-03-28T23:31:28.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.049+0000 7f1415d9b640 1 --2- 172.21.15.161:0/679842707 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f14180aacb0 0x7f1418155dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:28.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.049+0000 7f141659c640 1 --2- 172.21.15.161:0/679842707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14180a8fc0 0x7f14180c15b0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f140800c3c0 tx=0x7f140800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:28.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.050+0000 7f1406ffd640 1 -- 172.21.15.161:0/679842707 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14080052d0 con 0x7f14180a8fc0 2024-03-28T23:31:28.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.050+0000 7f1406ffd640 1 -- 172.21.15.161:0/679842707 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1408005490 con 0x7f14180a8fc0 2024-03-28T23:31:28.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.050+0000 7f1406ffd640 1 -- 172.21.15.161:0/679842707 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1408007260 con 0x7f14180a8fc0 2024-03-28T23:31:28.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.050+0000 7f141cb02640 1 -- 172.21.15.161:0/679842707 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1418154e00 con 0x7f14180a8fc0 2024-03-28T23:31:28.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.050+0000 7f141cb02640 1 -- 172.21.15.161:0/679842707 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14181553c0 con 0x7f14180a8fc0 2024-03-28T23:31:28.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.051+0000 7f141cb02640 1 -- 172.21.15.161:0/679842707 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1418071800 con 0x7f14180a8fc0 2024-03-28T23:31:28.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.051+0000 7f1406ffd640 1 -- 172.21.15.161:0/679842707 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1408007400 con 0x7f14180a8fc0 2024-03-28T23:31:28.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.051+0000 7f1406ffd640 1 --2- 172.21.15.161:0/679842707 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f13f806b500 0x7f13f806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:28.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.051+0000 7f1406ffd640 1 -- 172.21.15.161:0/679842707 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5421+0+0 (secure 0 0 0) 0x7f14080893f0 con 0x7f14180a8fc0 2024-03-28T23:31:28.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.051+0000 7f1415d9b640 1 --2- 172.21.15.161:0/679842707 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f13f806b500 0x7f13f806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:28.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.052+0000 7f1415d9b640 1 --2- 172.21.15.161:0/679842707 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f13f806b500 0x7f13f806d9c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1418155a30 tx=0x7f140c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:28.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.056+0000 7f1406ffd640 1 -- 172.21.15.161:0/679842707 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1418071800 con 0x7f14180a8fc0 2024-03-28T23:31:28.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.292+0000 7f141cb02640 1 -- 172.21.15.161:0/679842707 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort"]} v 0) -- 0x7f14181501b0 con 0x7f14180a8fc0 2024-03-28T23:31:28.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.322+0000 7f1406ffd640 1 -- 172.21.15.161:0/679842707 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort"]}]=0 v110) ==== 153+0+0 (secure 0 0 0) 0x7f14181501b0 con 0x7f14180a8fc0 2024-03-28T23:31:28.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.325+0000 7f141cb02640 1 -- 172.21.15.161:0/679842707 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f13f806b500 msgr2=0x7f13f806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:28.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.325+0000 7f141cb02640 1 --2- 172.21.15.161:0/679842707 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f13f806b500 0x7f13f806d9c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1418155a30 tx=0x7f140c008040 comp rx=0 tx=0).stop 2024-03-28T23:31:28.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.325+0000 7f141cb02640 1 -- 172.21.15.161:0/679842707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14180a8fc0 msgr2=0x7f14180c15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:28.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.325+0000 7f141cb02640 1 --2- 172.21.15.161:0/679842707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14180a8fc0 0x7f14180c15b0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f140800c3c0 tx=0x7f140800c880 comp rx=0 tx=0).stop 2024-03-28T23:31:28.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.325+0000 7f141cb02640 1 -- 172.21.15.161:0/679842707 shutdown_connections 2024-03-28T23:31:28.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.325+0000 7f141cb02640 1 --2- 172.21.15.161:0/679842707 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f13f806b500 0x7f13f806d9c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.325+0000 7f141cb02640 1 --2- 172.21.15.161:0/679842707 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1418150600 0x7f14180c0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.325+0000 7f141cb02640 1 --2- 172.21.15.161:0/679842707 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f14180aacb0 0x7f1418155dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.325+0000 7f141cb02640 1 --2- 172.21.15.161:0/679842707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14180a8fc0 0x7f14180c15b0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.325+0000 7f141cb02640 1 -- 172.21.15.161:0/679842707 >> 172.21.15.161:0/679842707 conn(0x7f14180c6670 msgr2=0x7f1418153100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:28.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.325+0000 7f141cb02640 1 -- 172.21.15.161:0/679842707 shutdown_connections 2024-03-28T23:31:28.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.325+0000 7f141cb02640 1 -- 172.21.15.161:0/679842707 wait complete. 2024-03-28T23:31:28.337 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-28T23:31:28.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.440+0000 7f295a9a0640 1 Processor -- start 2024-03-28T23:31:28.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.441+0000 7f295a9a0640 1 -- start start 2024-03-28T23:31:28.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.441+0000 7f295a9a0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2954156f70 0x7f29540c8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:28.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.441+0000 7f295a9a0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29540c75c0 con 0x7f29540c90f0 2024-03-28T23:31:28.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.441+0000 7f295a9a0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29540c72e0 con 0x7f29540c3ba0 2024-03-28T23:31:28.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.441+0000 7f295a9a0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f29540c6740 con 0x7f2954156f70 2024-03-28T23:31:28.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.442+0000 7f295919d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f29540c90f0 0x7f29540c3490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:46118/0 (socket says 172.21.15.161:46118) 2024-03-28T23:31:28.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.442+0000 7f295919d640 1 -- 172.21.15.161:0/169599366 learned_addr learned my addr 172.21.15.161:0/169599366 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:28.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.442+0000 7f295899c640 1 -- 172.21.15.161:0/169599366 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1506201921 0 0) 0x7f29540c75c0 con 0x7f29540c90f0 2024-03-28T23:31:28.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.442+0000 7f295899c640 1 -- 172.21.15.161:0/169599366 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2940003520 con 0x7f29540c90f0 2024-03-28T23:31:28.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.442+0000 7f295999e640 1 --2- 172.21.15.161:0/169599366 >> v2:172.21.15.170:3300/0 conn(0x7f2954156f70 0x7f29540c8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:28.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.442+0000 7f295899c640 1 -- 172.21.15.161:0/169599366 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2955235894 0 0) 0x7f2940003520 con 0x7f29540c90f0 2024-03-28T23:31:28.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.442+0000 7f295899c640 1 -- 172.21.15.161:0/169599366 >> v2:172.21.15.170:3300/0 conn(0x7f2954156f70 msgr2=0x7f29540c8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:28.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.442+0000 7f295899c640 1 --2- 172.21.15.161:0/169599366 >> v2:172.21.15.170:3300/0 conn(0x7f2954156f70 0x7f29540c8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.442+0000 7f295899c640 1 -- 172.21.15.161:0/169599366 >> v1:172.21.15.170:6789/0 conn(0x7f29540c3ba0 legacy=0x7f295415d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:28.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.442+0000 7f295899c640 1 -- 172.21.15.161:0/169599366 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2954073360 con 0x7f29540c90f0 2024-03-28T23:31:28.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.442+0000 7f295899c640 1 -- 172.21.15.161:0/169599366 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f2944001670 con 0x7f29540c90f0 2024-03-28T23:31:28.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.442+0000 7f295899c640 1 -- 172.21.15.161:0/169599366 >> v1:172.21.15.161:6789/0 conn(0x7f29540c90f0 legacy=0x7f29540c3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:28.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.442+0000 7f295899c640 1 --2- 172.21.15.161:0/169599366 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29400039b0 0x7f2940003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:28.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.442+0000 7f295899c640 1 --2- 172.21.15.161:0/169599366 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2940004800 0x7f2940004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:28.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.442+0000 7f295899c640 1 --2- 172.21.15.161:0/169599366 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29400051c0 0x7f294000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:28.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.442+0000 7f295899c640 1 -- 172.21.15.161:0/169599366 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f29540c6740 con 0x7f29400051c0 2024-03-28T23:31:28.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.442+0000 7f295899c640 1 -- 172.21.15.161:0/169599366 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f294000c160 con 0x7f2940004800 2024-03-28T23:31:28.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.442+0000 7f295899c640 1 -- 172.21.15.161:0/169599366 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f294000c360 con 0x7f29400039b0 2024-03-28T23:31:28.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.442+0000 7f295a19f640 1 --2- 172.21.15.161:0/169599366 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29400051c0 0x7f294000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:28.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.442+0000 7f295919d640 1 --2- 172.21.15.161:0/169599366 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2940004800 0x7f2940004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:28.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.442+0000 7f295999e640 1 --2- 172.21.15.161:0/169599366 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29400039b0 0x7f2940003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:28.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.443+0000 7f295a19f640 1 -- 172.21.15.161:0/169599366 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29400039b0 msgr2=0x7f2940003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:28.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.443+0000 7f295a19f640 1 --2- 172.21.15.161:0/169599366 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29400039b0 0x7f2940003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.443+0000 7f295a19f640 1 -- 172.21.15.161:0/169599366 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2940004800 msgr2=0x7f2940004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:28.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.443+0000 7f295a19f640 1 --2- 172.21.15.161:0/169599366 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2940004800 0x7f2940004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.443+0000 7f295a19f640 1 -- 172.21.15.161:0/169599366 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2940003520 con 0x7f29400051c0 2024-03-28T23:31:28.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.443+0000 7f295a19f640 1 -- 172.21.15.161:0/169599366 reap_dead start 2024-03-28T23:31:28.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.443+0000 7f295a19f640 1 --2- 172.21.15.161:0/169599366 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29400051c0 0x7f294000b9c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f294c001750 tx=0x7f2940005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de5d384bfac49e8c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:28.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.443+0000 7f295899c640 1 -- 172.21.15.161:0/169599366 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294c001370 con 0x7f29400051c0 2024-03-28T23:31:28.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.443+0000 7f295899c640 1 -- 172.21.15.161:0/169599366 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f294c005310 con 0x7f29400051c0 2024-03-28T23:31:28.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.443+0000 7f295899c640 1 -- 172.21.15.161:0/169599366 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294c003830 con 0x7f29400051c0 2024-03-28T23:31:28.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.444+0000 7f295a9a0640 1 -- 172.21.15.161:0/169599366 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29400051c0 msgr2=0x7f294000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:28.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.444+0000 7f295a9a0640 1 --2- 172.21.15.161:0/169599366 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29400051c0 0x7f294000b9c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f294c001750 tx=0x7f2940005830 comp rx=0 tx=0).stop 2024-03-28T23:31:28.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.444+0000 7f295a9a0640 1 -- 172.21.15.161:0/169599366 shutdown_connections 2024-03-28T23:31:28.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.444+0000 7f295a9a0640 1 --2- 172.21.15.161:0/169599366 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29400051c0 0x7f294000b9c0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.444+0000 7f295a9a0640 1 -- 172.21.15.161:0/169599366 >> 172.21.15.161:0/169599366 conn(0x7f295406b330 msgr2=0x7f295406b740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:28.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.444+0000 7f295a9a0640 1 -- 172.21.15.161:0/169599366 shutdown_connections 2024-03-28T23:31:28.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.444+0000 7f295a9a0640 1 -- 172.21.15.161:0/169599366 wait complete. 2024-03-28T23:31:28.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.445+0000 7f295a9a0640 1 Processor -- start 2024-03-28T23:31:28.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.445+0000 7f295a9a0640 1 -- start start 2024-03-28T23:31:28.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.446+0000 7f295a9a0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29400051c0 0x7f295414eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:28.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.446+0000 7f295a9a0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f295414f710 0x7f295414f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:28.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.446+0000 7f295a9a0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29541ec2e0 0x7f29541ec760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:28.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.446+0000 7f295a9a0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f29540c6bd0 con 0x7f29400051c0 2024-03-28T23:31:28.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.446+0000 7f295a9a0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f29540c6f90 con 0x7f295414f710 2024-03-28T23:31:28.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.446+0000 7f295a9a0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f29541eacb0 con 0x7f29541ec2e0 2024-03-28T23:31:28.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.446+0000 7f295999e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29400051c0 0x7f295414eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:28.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.446+0000 7f295999e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29400051c0 0x7f295414eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35490/0 (socket says 172.21.15.161:35490) 2024-03-28T23:31:28.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.446+0000 7f295999e640 1 -- 172.21.15.161:0/73981122 learned_addr learned my addr 172.21.15.161:0/73981122 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:28.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.446+0000 7f295919d640 1 --2- 172.21.15.161:0/73981122 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f295414f710 0x7f295414f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:28.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.446+0000 7f295a19f640 1 --2- 172.21.15.161:0/73981122 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29541ec2e0 0x7f29541ec760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:28.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.446+0000 7f295999e640 1 -- 172.21.15.161:0/73981122 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29541ec2e0 msgr2=0x7f29541ec760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:28.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.446+0000 7f295999e640 1 --2- 172.21.15.161:0/73981122 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29541ec2e0 0x7f29541ec760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.446+0000 7f295999e640 1 -- 172.21.15.161:0/73981122 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f295414f710 msgr2=0x7f295414f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:28.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.446+0000 7f295999e640 1 --2- 172.21.15.161:0/73981122 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f295414f710 0x7f295414f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.446+0000 7f295999e640 1 -- 172.21.15.161:0/73981122 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29541ea0a0 con 0x7f29400051c0 2024-03-28T23:31:28.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.446+0000 7f295999e640 1 --2- 172.21.15.161:0/73981122 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29400051c0 0x7f295414eac0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2950009f90 tx=0x7f295000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:28.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.446+0000 7f294a7fc640 1 -- 172.21.15.161:0/73981122 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2950015430 con 0x7f29400051c0 2024-03-28T23:31:28.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.446+0000 7f294a7fc640 1 -- 172.21.15.161:0/73981122 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29500155d0 con 0x7f29400051c0 2024-03-28T23:31:28.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.446+0000 7f295a9a0640 1 -- 172.21.15.161:0/73981122 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29541ea380 con 0x7f29400051c0 2024-03-28T23:31:28.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.447+0000 7f294a7fc640 1 -- 172.21.15.161:0/73981122 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29500159c0 con 0x7f29400051c0 2024-03-28T23:31:28.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.447+0000 7f295a9a0640 1 -- 172.21.15.161:0/73981122 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29541ea940 con 0x7f29400051c0 2024-03-28T23:31:28.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.448+0000 7f295a9a0640 1 -- 172.21.15.161:0/73981122 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f295404f4b0 con 0x7f29400051c0 2024-03-28T23:31:28.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.448+0000 7f294a7fc640 1 -- 172.21.15.161:0/73981122 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2950015b60 con 0x7f29400051c0 2024-03-28T23:31:28.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.448+0000 7f294a7fc640 1 --2- 172.21.15.161:0/73981122 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f293806b4d0 0x7f293806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:28.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.448+0000 7f294a7fc640 1 -- 172.21.15.161:0/73981122 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5421+0+0 (secure 0 0 0) 0x7f295008b070 con 0x7f29400051c0 2024-03-28T23:31:28.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.448+0000 7f295919d640 1 --2- 172.21.15.161:0/73981122 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f293806b4d0 0x7f293806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:28.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.449+0000 7f295919d640 1 --2- 172.21.15.161:0/73981122 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f293806b4d0 0x7f293806d990 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f29541e9c30 tx=0x7f29440078f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:28.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.453+0000 7f294a7fc640 1 -- 172.21.15.161:0/73981122 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2950019030 con 0x7f29400051c0 2024-03-28T23:31:28.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.624+0000 7f295a9a0640 1 -- 172.21.15.161:0/73981122 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f29540685f0 con 0x7f29400051c0 2024-03-28T23:31:28.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.628+0000 7f294a7fc640 1 -- 172.21.15.161:0/73981122 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v15)=0 v15) ==== 122+0+0 (secure 0 0 0) 0x7f2950010040 con 0x7f29400051c0 2024-03-28T23:31:28.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.630+0000 7f295a9a0640 1 -- 172.21.15.161:0/73981122 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f293806b4d0 msgr2=0x7f293806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:28.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.630+0000 7f295a9a0640 1 --2- 172.21.15.161:0/73981122 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f293806b4d0 0x7f293806d990 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f29541e9c30 tx=0x7f29440078f0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.630+0000 7f295a9a0640 1 -- 172.21.15.161:0/73981122 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29400051c0 msgr2=0x7f295414eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:28.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.630+0000 7f295a9a0640 1 --2- 172.21.15.161:0/73981122 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29400051c0 0x7f295414eac0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2950009f90 tx=0x7f295000f580 comp rx=0 tx=0).stop 2024-03-28T23:31:28.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.630+0000 7f295a9a0640 1 -- 172.21.15.161:0/73981122 shutdown_connections 2024-03-28T23:31:28.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.630+0000 7f295a9a0640 1 --2- 172.21.15.161:0/73981122 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f293806b4d0 0x7f293806d990 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.630+0000 7f295a9a0640 1 --2- 172.21.15.161:0/73981122 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29541ec2e0 0x7f29541ec760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.630+0000 7f295a9a0640 1 --2- 172.21.15.161:0/73981122 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f295414f710 0x7f295414f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.630+0000 7f295a9a0640 1 --2- 172.21.15.161:0/73981122 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29400051c0 0x7f295414eac0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.630+0000 7f295a9a0640 1 -- 172.21.15.161:0/73981122 >> 172.21.15.161:0/73981122 conn(0x7f295406b330 msgr2=0x7f295406b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:28.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.630+0000 7f295a9a0640 1 -- 172.21.15.161:0/73981122 shutdown_connections 2024-03-28T23:31:28.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.630+0000 7f295a9a0640 1 -- 172.21.15.161:0/73981122 wait complete. 2024-03-28T23:31:28.641 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-28T23:31:28.641 DEBUG:teuthology.orchestra.run.smithi161:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:31:28.682 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:31:28.733 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-28T23:31:28.733 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:31:28.733 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:31:28.749 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20793 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-28T23:31:28.749 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:31:28.749 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:31:28.766 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056374,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19690 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-28T23:31:28.766 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-28T23:31:28.766 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:31:28.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.890+0000 7fe016674640 1 Processor -- start 2024-03-28T23:31:28.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.890+0000 7fe016674640 1 -- start start 2024-03-28T23:31:28.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.891+0000 7fe016674640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe010152a90 0x7fe0100a8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:28.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.891+0000 7fe016674640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe0100c2390 0x7fe0100bfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:28.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.891+0000 7fe016674640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0101550d0 con 0x7fe0100bff40 2024-03-28T23:31:28.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.891+0000 7fe016674640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe010155fa0 con 0x7fe0100c2390 2024-03-28T23:31:28.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.891+0000 7fe016674640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe010155390 con 0x7fe010152a90 2024-03-28T23:31:28.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.891+0000 7fe00f7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe0100c2390 0x7fe0100bfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:28.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.891+0000 7fe00f7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe0100c2390 0x7fe0100bfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35500/0 (socket says 172.21.15.161:35500) 2024-03-28T23:31:28.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.891+0000 7fe014c11640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe0100bff40 0x7fe0100aad50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:46134/0 (socket says 172.21.15.161:46134) 2024-03-28T23:31:28.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.891+0000 7fe00f7fe640 1 -- 172.21.15.161:0/999987947 learned_addr learned my addr 172.21.15.161:0/999987947 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:28.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.891+0000 7fe00ffff640 1 --2- 172.21.15.161:0/999987947 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe010152a90 0x7fe0100a8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:28.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.891+0000 7fe00f7fe640 1 -- 172.21.15.161:0/999987947 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe010152a90 msgr2=0x7fe0100a8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:28.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.891+0000 7fe00f7fe640 1 --2- 172.21.15.161:0/999987947 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe010152a90 0x7fe0100a8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.891+0000 7fe00f7fe640 1 -- 172.21.15.161:0/999987947 >> v1:172.21.15.161:6789/0 conn(0x7fe0100bff40 legacy=0x7fe0100aad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:28.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.892+0000 7fe00f7fe640 1 -- 172.21.15.161:0/999987947 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe010155670 con 0x7fe0100c2390 2024-03-28T23:31:28.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.891+0000 7fe00effd640 1 -- 172.21.15.161:0/999987947 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2018605131 0 0) 0x7fe0101550d0 con 0x7fe0100bff40 2024-03-28T23:31:28.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.892+0000 7fe00f7fe640 1 --2- 172.21.15.161:0/999987947 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0100c2390 0x7fe0100bfa00 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe004009870 tx=0x7fe00402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4287274f643014ac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:28.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.892+0000 7fe00effd640 1 -- 172.21.15.161:0/999987947 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe004035020 con 0x7fe0100c2390 2024-03-28T23:31:28.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.892+0000 7fe00effd640 1 -- 172.21.15.161:0/999987947 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe004031640 con 0x7fe0100c2390 2024-03-28T23:31:28.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.892+0000 7fe00effd640 1 -- 172.21.15.161:0/999987947 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00402d040 con 0x7fe0100c2390 2024-03-28T23:31:28.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.893+0000 7fe016674640 1 -- 172.21.15.161:0/999987947 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0100c2390 msgr2=0x7fe0100bfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:28.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.893+0000 7fe016674640 1 --2- 172.21.15.161:0/999987947 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0100c2390 0x7fe0100bfa00 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe004009870 tx=0x7fe00402b8f0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.893+0000 7fe016674640 1 -- 172.21.15.161:0/999987947 shutdown_connections 2024-03-28T23:31:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.893+0000 7fe016674640 1 --2- 172.21.15.161:0/999987947 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0100c2390 0x7fe0100bfa00 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.893+0000 7fe016674640 1 --2- 172.21.15.161:0/999987947 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe010152a90 0x7fe0100a8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.893+0000 7fe016674640 1 -- 172.21.15.161:0/999987947 >> 172.21.15.161:0/999987947 conn(0x7fe0100c6670 msgr2=0x7fe0100c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.893+0000 7fe016674640 1 -- 172.21.15.161:0/999987947 shutdown_connections 2024-03-28T23:31:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.893+0000 7fe016674640 1 -- 172.21.15.161:0/999987947 wait complete. 2024-03-28T23:31:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.894+0000 7fe016674640 1 Processor -- start 2024-03-28T23:31:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.894+0000 7fe016674640 1 -- start start 2024-03-28T23:31:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.895+0000 7fe016674640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0100bff40 0x7fe0101577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.895+0000 7fe016674640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0100c2390 0x7fe010153f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.895+0000 7fe016674640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe010152a90 0x7fe01014cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.895+0000 7fe016674640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe0100ab460 con 0x7fe0100bff40 2024-03-28T23:31:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.895+0000 7fe016674640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe010155c30 con 0x7fe010152a90 2024-03-28T23:31:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.895+0000 7fe016674640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe0100ab5e0 con 0x7fe0100c2390 2024-03-28T23:31:28.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.895+0000 7fe00ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0100bff40 0x7fe0101577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:28.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.895+0000 7fe00ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0100bff40 0x7fe0101577c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35516/0 (socket says 172.21.15.161:35516) 2024-03-28T23:31:28.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.895+0000 7fe00ffff640 1 -- 172.21.15.161:0/3473058406 learned_addr learned my addr 172.21.15.161:0/3473058406 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:28.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.895+0000 7fe00f7fe640 1 --2- 172.21.15.161:0/3473058406 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0100c2390 0x7fe010153f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:28.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.895+0000 7fe014c11640 1 --2- 172.21.15.161:0/3473058406 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe010152a90 0x7fe01014cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:28.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.895+0000 7fe00ffff640 1 -- 172.21.15.161:0/3473058406 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0100c2390 msgr2=0x7fe010153f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:28.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.895+0000 7fe00ffff640 1 --2- 172.21.15.161:0/3473058406 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0100c2390 0x7fe010153f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.895+0000 7fe00ffff640 1 -- 172.21.15.161:0/3473058406 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe010152a90 msgr2=0x7fe01014cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:28.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.895+0000 7fe00ffff640 1 --2- 172.21.15.161:0/3473058406 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe010152a90 0x7fe01014cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:28.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.895+0000 7fe00ffff640 1 -- 172.21.15.161:0/3473058406 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe010156560 con 0x7fe0100bff40 2024-03-28T23:31:28.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.895+0000 7fe00ffff640 1 --2- 172.21.15.161:0/3473058406 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0100bff40 0x7fe0101577c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fe00000d5f0 tx=0x7fe00000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:28.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.896+0000 7fe00cff9640 1 -- 172.21.15.161:0/3473058406 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0000125f0 con 0x7fe0100bff40 2024-03-28T23:31:28.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.896+0000 7fe00cff9640 1 -- 172.21.15.161:0/3473058406 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe000019070 con 0x7fe0100bff40 2024-03-28T23:31:28.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.896+0000 7fe016674640 1 -- 172.21.15.161:0/3473058406 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe010072fe0 con 0x7fe0100bff40 2024-03-28T23:31:28.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.896+0000 7fe00cff9640 1 -- 172.21.15.161:0/3473058406 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0000129d0 con 0x7fe0100bff40 2024-03-28T23:31:28.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.896+0000 7fe016674640 1 -- 172.21.15.161:0/3473058406 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe010157d00 con 0x7fe0100bff40 2024-03-28T23:31:28.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.897+0000 7fe016674640 1 -- 172.21.15.161:0/3473058406 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe01004f4b0 con 0x7fe0100bff40 2024-03-28T23:31:28.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.897+0000 7fe00cff9640 1 -- 172.21.15.161:0/3473058406 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe000012b70 con 0x7fe0100bff40 2024-03-28T23:31:28.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.897+0000 7fe00cff9640 1 --2- 172.21.15.161:0/3473058406 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdffc06b4d0 0x7fdffc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:28.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.897+0000 7fe00cff9640 1 -- 172.21.15.161:0/3473058406 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5421+0+0 (secure 0 0 0) 0x7fe00008a370 con 0x7fe0100bff40 2024-03-28T23:31:28.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.897+0000 7fe00f7fe640 1 --2- 172.21.15.161:0/3473058406 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdffc06b4d0 0x7fdffc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:28.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.898+0000 7fe00f7fe640 1 --2- 172.21.15.161:0/3473058406 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdffc06b4d0 0x7fdffc06d990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe004036000 tx=0x7fe00403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:28.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:28.902+0000 7fe00cff9640 1 -- 172.21.15.161:0/3473058406 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe0000593a0 con 0x7fe0100bff40 2024-03-28T23:31:29.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:29.109+0000 7fe016674640 1 -- 172.21.15.161:0/3473058406 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe0100c6bf0 con 0x7fe0100bff40 2024-03-28T23:31:29.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:29.109+0000 7fe00cff9640 1 -- 172.21.15.161:0/3473058406 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+3872 (secure 0 0 0) 0x7fe000058ca0 con 0x7fe0100bff40 2024-03-28T23:31:29.110 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:29.110 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":25,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5110,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/465771987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":465771987},{"type":"v1","addr":"172.21.15.170:6835","nonce":465771987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5114,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/639612864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":639612864},{"type":"v1","addr":"172.21.15.170:6834","nonce":639612864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5355,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/306644174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":306644174},{"type":"v1","addr":"172.21.15.161:6836","nonce":306644174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1649814516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":1649814516},{"type":"v1","addr":"172.21.15.161:6837","nonce":1649814516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[]} 2024-03-28T23:31:29.110 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 25 2024-03-28T23:31:29.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:29.111+0000 7fe016674640 1 -- 172.21.15.161:0/3473058406 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdffc06b4d0 msgr2=0x7fdffc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:29.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:29.111+0000 7fe016674640 1 --2- 172.21.15.161:0/3473058406 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdffc06b4d0 0x7fdffc06d990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe004036000 tx=0x7fe00403c040 comp rx=0 tx=0).stop 2024-03-28T23:31:29.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:29.111+0000 7fe016674640 1 -- 172.21.15.161:0/3473058406 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0100bff40 msgr2=0x7fe0101577c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:29.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:29.111+0000 7fe016674640 1 --2- 172.21.15.161:0/3473058406 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0100bff40 0x7fe0101577c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fe00000d5f0 tx=0x7fe00000dab0 comp rx=0 tx=0).stop 2024-03-28T23:31:29.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:29.111+0000 7fe016674640 1 -- 172.21.15.161:0/3473058406 shutdown_connections 2024-03-28T23:31:29.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:29.111+0000 7fe016674640 1 --2- 172.21.15.161:0/3473058406 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdffc06b4d0 0x7fdffc06d990 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:29.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:29.111+0000 7fe016674640 1 --2- 172.21.15.161:0/3473058406 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe010152a90 0x7fe01014cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:29.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:29.111+0000 7fe016674640 1 --2- 172.21.15.161:0/3473058406 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0100c2390 0x7fe010153f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:29.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:29.111+0000 7fe016674640 1 --2- 172.21.15.161:0/3473058406 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0100bff40 0x7fe0101577c0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:29.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:29.111+0000 7fe016674640 1 -- 172.21.15.161:0/3473058406 >> 172.21.15.161:0/3473058406 conn(0x7fe0100c6670 msgr2=0x7fe01006c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:29.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:29.111+0000 7fe016674640 1 -- 172.21.15.161:0/3473058406 shutdown_connections 2024-03-28T23:31:29.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:29.111+0000 7fe016674640 1 -- 172.21.15.161:0/3473058406 wait complete. 2024-03-28T23:31:29.123 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T23:31:29.123 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-28T23:31:29.123 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T23:31:29.124 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:31:29.124 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T23:31:29.124 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-28T23:31:29.124 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T23:31:29.124 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:31:29.124 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-28T23:31:29.125 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-28T23:31:29.125 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-28T23:31:29.125 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:31:29.125 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-28T23:31:29.125 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-28T23:31:29.125 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-28T23:31:29.126 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:31:29.133 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T23:31:29.133 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T23:31:29.134 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T23:31:29.134 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T23:31:29.137 INFO:tasks.ceph.mds.d:Stopped 2024-03-28T23:31:29.137 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-28T23:31:29.176 INFO:tasks.ceph.mds.a:Started 2024-03-28T23:31:29.176 DEBUG:teuthology.parallel:result is None 2024-03-28T23:31:29.177 INFO:tasks.ceph.mds.c:Started 2024-03-28T23:31:29.177 DEBUG:teuthology.parallel:result is None 2024-03-28T23:31:29.180 INFO:tasks.ceph.mds.d:Started 2024-03-28T23:31:29.180 DEBUG:teuthology.parallel:result is None 2024-03-28T23:31:29.252 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-28T23:31:29.252 INFO:tasks.ceph.mds.c.smithi161.stdout:starting mds.c at 2024-03-28T23:31:29.262 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2024-03-28T23:31:32.320 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-28T23:31:35.137 INFO:tasks.ceph.mds.b:Stopped 2024-03-28T23:31:35.138 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-28T23:31:35.141 INFO:tasks.ceph.mds.b:Started 2024-03-28T23:31:35.141 DEBUG:teuthology.parallel:result is None 2024-03-28T23:31:35.141 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-28T23:31:35.226 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2024-03-28T23:31:35.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.255+0000 7efed553e640 1 Processor -- start 2024-03-28T23:31:35.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.255+0000 7efed553e640 1 -- start start 2024-03-28T23:31:35.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.256+0000 7efed553e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efed006c3d0 0x7efed0152580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:35.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.256+0000 7efed553e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7efed0155b90 0x7efed0158750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:35.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.256+0000 7efed553e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efed00ac150 0x7efed0158c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:35.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.256+0000 7efed553e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7efed01591d0 con 0x7efed00ac150 2024-03-28T23:31:35.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.256+0000 7efed553e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7efed0156070 con 0x7efed0155b90 2024-03-28T23:31:35.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.256+0000 7efed553e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7efed0066fe0 con 0x7efed006c3d0 2024-03-28T23:31:35.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.256+0000 7efecf7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efed00ac150 0x7efed0158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:35.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.256+0000 7efecf7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efed00ac150 0x7efed0158c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59280/0 (socket says 172.21.15.161:59280) 2024-03-28T23:31:35.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.256+0000 7efecf7fe640 1 -- 172.21.15.161:0/1667636361 learned_addr learned my addr 172.21.15.161:0/1667636361 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:35.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.256+0000 7efeceffd640 1 --2- 172.21.15.161:0/1667636361 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efed006c3d0 0x7efed0152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:35.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.256+0000 7efec7fff640 1 --2- 172.21.15.161:0/1667636361 >> v2:172.21.15.170:3300/0 conn(0x7efed0155b90 0x7efed0158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:35.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.256+0000 7efecf7fe640 1 -- 172.21.15.161:0/1667636361 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efed006c3d0 msgr2=0x7efed0152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:35.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.256+0000 7efecf7fe640 1 --2- 172.21.15.161:0/1667636361 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efed006c3d0 0x7efed0152580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:35.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.256+0000 7efecf7fe640 1 -- 172.21.15.161:0/1667636361 >> v2:172.21.15.170:3300/0 conn(0x7efed0155b90 msgr2=0x7efed0158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:35.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.256+0000 7efecf7fe640 1 --2- 172.21.15.161:0/1667636361 >> v2:172.21.15.170:3300/0 conn(0x7efed0155b90 0x7efed0158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:35.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.256+0000 7efecf7fe640 1 -- 172.21.15.161:0/1667636361 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed00672c0 con 0x7efed00ac150 2024-03-28T23:31:35.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.257+0000 7efecf7fe640 1 --2- 172.21.15.161:0/1667636361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efed00ac150 0x7efed0158c90 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7efed0068a40 tx=0x7efec802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f544a2a40514bd20 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:35.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.257+0000 7efece7fc640 1 -- 172.21.15.161:0/1667636361 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec802ca70 con 0x7efed00ac150 2024-03-28T23:31:35.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.257+0000 7efece7fc640 1 -- 172.21.15.161:0/1667636361 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efec802cc10 con 0x7efed00ac150 2024-03-28T23:31:35.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.257+0000 7efece7fc640 1 -- 172.21.15.161:0/1667636361 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec80343f0 con 0x7efed00ac150 2024-03-28T23:31:35.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.258+0000 7efed553e640 1 -- 172.21.15.161:0/1667636361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efed00ac150 msgr2=0x7efed0158c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:35.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.258+0000 7efed553e640 1 --2- 172.21.15.161:0/1667636361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efed00ac150 0x7efed0158c90 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7efed0068a40 tx=0x7efec802bed0 comp rx=0 tx=0).stop 2024-03-28T23:31:35.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.258+0000 7efed553e640 1 -- 172.21.15.161:0/1667636361 shutdown_connections 2024-03-28T23:31:35.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.258+0000 7efed553e640 1 --2- 172.21.15.161:0/1667636361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efed00ac150 0x7efed0158c90 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:35.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.258+0000 7efed553e640 1 --2- 172.21.15.161:0/1667636361 >> v2:172.21.15.170:3300/0 conn(0x7efed0155b90 0x7efed0158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:35.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.258+0000 7efed553e640 1 --2- 172.21.15.161:0/1667636361 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efed006c3d0 0x7efed0152580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:35.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.258+0000 7efed553e640 1 -- 172.21.15.161:0/1667636361 >> 172.21.15.161:0/1667636361 conn(0x7efed00c3be0 msgr2=0x7efed00c3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:35.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.258+0000 7efed553e640 1 -- 172.21.15.161:0/1667636361 shutdown_connections 2024-03-28T23:31:35.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.258+0000 7efed553e640 1 -- 172.21.15.161:0/1667636361 wait complete. 2024-03-28T23:31:35.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.259+0000 7efed553e640 1 Processor -- start 2024-03-28T23:31:35.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.259+0000 7efed553e640 1 -- start start 2024-03-28T23:31:35.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.260+0000 7efed553e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efed006c3d0 0x7efed014efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:35.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.260+0000 7efed553e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efed00ac150 0x7efed00aa3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:35.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.260+0000 7efed553e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efed0155b90 0x7efed01de570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:35.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.260+0000 7efed553e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efed00c2320 con 0x7efed00ac150 2024-03-28T23:31:35.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.260+0000 7efed553e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7efed00c21a0 con 0x7efed006c3d0 2024-03-28T23:31:35.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.260+0000 7efed553e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7efed006c7b0 con 0x7efed0155b90 2024-03-28T23:31:35.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.260+0000 7efec7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efed00ac150 0x7efed00aa3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:35.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.260+0000 7efec7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efed00ac150 0x7efed00aa3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59286/0 (socket says 172.21.15.161:59286) 2024-03-28T23:31:35.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.260+0000 7efec7fff640 1 -- 172.21.15.161:0/218941163 learned_addr learned my addr 172.21.15.161:0/218941163 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:35.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.260+0000 7efeceffd640 1 --2- 172.21.15.161:0/218941163 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efed006c3d0 0x7efed014efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:35.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.260+0000 7efecf7fe640 1 --2- 172.21.15.161:0/218941163 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efed0155b90 0x7efed01de570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:35.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.260+0000 7efec7fff640 1 -- 172.21.15.161:0/218941163 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efed0155b90 msgr2=0x7efed01de570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:35.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.260+0000 7efec7fff640 1 --2- 172.21.15.161:0/218941163 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efed0155b90 0x7efed01de570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:35.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.260+0000 7efec7fff640 1 -- 172.21.15.161:0/218941163 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efed006c3d0 msgr2=0x7efed014efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:35.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.260+0000 7efec7fff640 1 --2- 172.21.15.161:0/218941163 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efed006c3d0 0x7efed014efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:35.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.260+0000 7efec7fff640 1 -- 172.21.15.161:0/218941163 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed01dfb50 con 0x7efed00ac150 2024-03-28T23:31:35.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.260+0000 7efec7fff640 1 --2- 172.21.15.161:0/218941163 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efed00ac150 0x7efed00aa3e0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7efeb800e9e0 tx=0x7efeb800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:35.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.260+0000 7efec77fe640 1 -- 172.21.15.161:0/218941163 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb8017070 con 0x7efed00ac150 2024-03-28T23:31:35.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.260+0000 7efec77fe640 1 -- 172.21.15.161:0/218941163 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeb800c9d0 con 0x7efed00ac150 2024-03-28T23:31:35.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.260+0000 7efed553e640 1 -- 172.21.15.161:0/218941163 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efed014f500 con 0x7efed00ac150 2024-03-28T23:31:35.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.260+0000 7efec77fe640 1 -- 172.21.15.161:0/218941163 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb800cdc0 con 0x7efed00ac150 2024-03-28T23:31:35.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.260+0000 7efed553e640 1 -- 172.21.15.161:0/218941163 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efed0157fc0 con 0x7efed00ac150 2024-03-28T23:31:35.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.261+0000 7efed553e640 1 -- 172.21.15.161:0/218941163 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efed0068470 con 0x7efed00ac150 2024-03-28T23:31:35.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.262+0000 7efec77fe640 1 -- 172.21.15.161:0/218941163 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efeb801a070 con 0x7efed00ac150 2024-03-28T23:31:35.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.262+0000 7efec77fe640 1 --2- 172.21.15.161:0/218941163 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efeb006b500 0x7efeb006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:35.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.262+0000 7efec77fe640 1 -- 172.21.15.161:0/218941163 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7efeb8089f90 con 0x7efed00ac150 2024-03-28T23:31:35.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.262+0000 7efeceffd640 1 --2- 172.21.15.161:0/218941163 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efeb006b500 0x7efeb006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:35.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.263+0000 7efeceffd640 1 --2- 172.21.15.161:0/218941163 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efeb006b500 0x7efeb006d9c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7efed00aacf0 tx=0x7efec0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:35.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.267+0000 7efec77fe640 1 -- 172.21.15.161:0/218941163 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efed0068470 con 0x7efed00ac150 2024-03-28T23:31:35.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:35.456+0000 7efed553e640 1 -- 172.21.15.161:0/218941163 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7efed004f4b0 con 0x7efed00ac150 2024-03-28T23:31:36.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.354+0000 7efec77fe640 1 -- 172.21.15.161:0/218941163 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v47) ==== 97+0+0 (secure 0 0 0) 0x7efed014f500 con 0x7efed00ac150 2024-03-28T23:31:36.357 INFO:teuthology.orchestra.run.smithi161.stderr: removed all blocklist entries 2024-03-28T23:31:36.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.357+0000 7efed553e640 1 -- 172.21.15.161:0/218941163 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efeb006b500 msgr2=0x7efeb006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:36.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.357+0000 7efed553e640 1 --2- 172.21.15.161:0/218941163 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efeb006b500 0x7efeb006d9c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7efed00aacf0 tx=0x7efec0009190 comp rx=0 tx=0).stop 2024-03-28T23:31:36.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.357+0000 7efed553e640 1 -- 172.21.15.161:0/218941163 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efed00ac150 msgr2=0x7efed00aa3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:36.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.357+0000 7efed553e640 1 --2- 172.21.15.161:0/218941163 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efed00ac150 0x7efed00aa3e0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7efeb800e9e0 tx=0x7efeb800eea0 comp rx=0 tx=0).stop 2024-03-28T23:31:36.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.357+0000 7efed553e640 1 -- 172.21.15.161:0/218941163 shutdown_connections 2024-03-28T23:31:36.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.357+0000 7efed553e640 1 --2- 172.21.15.161:0/218941163 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efeb006b500 0x7efeb006d9c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:36.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.357+0000 7efed553e640 1 --2- 172.21.15.161:0/218941163 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efed00ac150 0x7efed00aa3e0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:36.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.357+0000 7efed553e640 1 --2- 172.21.15.161:0/218941163 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efed0155b90 0x7efed01de570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:36.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.357+0000 7efed553e640 1 --2- 172.21.15.161:0/218941163 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efed006c3d0 0x7efed014efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:36.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.357+0000 7efed553e640 1 -- 172.21.15.161:0/218941163 >> 172.21.15.161:0/218941163 conn(0x7efed00c3be0 msgr2=0x7efed006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:36.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.357+0000 7efed553e640 1 -- 172.21.15.161:0/218941163 shutdown_connections 2024-03-28T23:31:36.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.357+0000 7efed553e640 1 -- 172.21.15.161:0/218941163 wait complete. 2024-03-28T23:31:36.370 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-28T23:31:36.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.471+0000 7ff6dbf5e640 1 Processor -- start 2024-03-28T23:31:36.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.471+0000 7ff6dbf5e640 1 -- start start 2024-03-28T23:31:36.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.472+0000 7ff6dbf5e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d414be70 0x7ff6d40ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:36.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.472+0000 7ff6dbf5e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff6d40aacd0 0x7ff6d415c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:36.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.472+0000 7ff6dbf5e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6d414c250 con 0x7ff6d40ab6e0 2024-03-28T23:31:36.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.472+0000 7ff6dbf5e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff6d41529c0 con 0x7ff6d40aacd0 2024-03-28T23:31:36.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.472+0000 7ff6dbf5e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff6d40a9dc0 con 0x7ff6d414be70 2024-03-28T23:31:36.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.472+0000 7ff6da4fb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff6d40aacd0 0x7ff6d415c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:36.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.472+0000 7ff6da4fb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff6d40aacd0 0x7ff6d415c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59296/0 (socket says 172.21.15.161:59296) 2024-03-28T23:31:36.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.472+0000 7ff6da4fb640 1 -- 172.21.15.161:0/331266700 learned_addr learned my addr 172.21.15.161:0/331266700 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:36.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.472+0000 7ff6d9cfa640 1 --2- 172.21.15.161:0/331266700 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d414be70 0x7ff6d40ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:36.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.472+0000 7ff6da4fb640 1 -- 172.21.15.161:0/331266700 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d414be70 msgr2=0x7ff6d40ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:36.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.472+0000 7ff6da4fb640 1 --2- 172.21.15.161:0/331266700 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d414be70 0x7ff6d40ab1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:36.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.473+0000 7ff6da4fb640 1 -- 172.21.15.161:0/331266700 >> v1:172.21.15.170:6789/0 conn(0x7ff6d40ab6e0 legacy=0x7ff6d40aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:36.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.473+0000 7ff6da4fb640 1 -- 172.21.15.161:0/331266700 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d40aa0a0 con 0x7ff6d40aacd0 2024-03-28T23:31:36.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.473+0000 7ff6d9cfa640 1 --2- 172.21.15.161:0/331266700 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d414be70 0x7ff6d40ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:36.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.473+0000 7ff6da4fb640 1 --2- 172.21.15.161:0/331266700 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6d40aacd0 0x7ff6d415c740 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc00d150 tx=0x7ff6cc02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4513e2afa4c0a2af server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:36.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.473+0000 7ff6d8cf8640 1 -- 172.21.15.161:0/331266700 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc02ea80 con 0x7ff6d40aacd0 2024-03-28T23:31:36.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.473+0000 7ff6d8cf8640 1 -- 172.21.15.161:0/331266700 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6cc02ec20 con 0x7ff6d40aacd0 2024-03-28T23:31:36.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.473+0000 7ff6d8cf8640 1 -- 172.21.15.161:0/331266700 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc009600 con 0x7ff6d40aacd0 2024-03-28T23:31:36.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.474+0000 7ff6dbf5e640 1 -- 172.21.15.161:0/331266700 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6d40aacd0 msgr2=0x7ff6d415c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:36.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.474+0000 7ff6dbf5e640 1 --2- 172.21.15.161:0/331266700 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6d40aacd0 0x7ff6d415c740 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc00d150 tx=0x7ff6cc02e0e0 comp rx=0 tx=0).stop 2024-03-28T23:31:36.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.474+0000 7ff6dbf5e640 1 -- 172.21.15.161:0/331266700 shutdown_connections 2024-03-28T23:31:36.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.474+0000 7ff6dbf5e640 1 --2- 172.21.15.161:0/331266700 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6d40aacd0 0x7ff6d415c740 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:36.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.474+0000 7ff6dbf5e640 1 --2- 172.21.15.161:0/331266700 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d414be70 0x7ff6d40ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:36.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.474+0000 7ff6dbf5e640 1 -- 172.21.15.161:0/331266700 >> 172.21.15.161:0/331266700 conn(0x7ff6d40c6670 msgr2=0x7ff6d40c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:36.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.474+0000 7ff6dbf5e640 1 -- 172.21.15.161:0/331266700 shutdown_connections 2024-03-28T23:31:36.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.474+0000 7ff6dbf5e640 1 -- 172.21.15.161:0/331266700 wait complete. 2024-03-28T23:31:36.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.475+0000 7ff6dbf5e640 1 Processor -- start 2024-03-28T23:31:36.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.475+0000 7ff6dbf5e640 1 -- start start 2024-03-28T23:31:36.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.476+0000 7ff6dbf5e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d40aacd0 0x7ff6d41dbe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:36.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.476+0000 7ff6dbf5e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6d40ab6e0 0x7ff6d4151440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:36.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.476+0000 7ff6dbf5e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6d414be70 0x7ff6d4153b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:36.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.476+0000 7ff6dbf5e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d406b2d0 con 0x7ff6d40ab6e0 2024-03-28T23:31:36.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.476+0000 7ff6dbf5e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d406b150 con 0x7ff6d414be70 2024-03-28T23:31:36.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.476+0000 7ff6dbf5e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff6d40c1520 con 0x7ff6d40aacd0 2024-03-28T23:31:36.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.476+0000 7ff6d94f9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6d40ab6e0 0x7ff6d4151440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:36.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.476+0000 7ff6d94f9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6d40ab6e0 0x7ff6d4151440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59306/0 (socket says 172.21.15.161:59306) 2024-03-28T23:31:36.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.476+0000 7ff6d94f9640 1 -- 172.21.15.161:0/2163613195 learned_addr learned my addr 172.21.15.161:0/2163613195 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:36.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.476+0000 7ff6da4fb640 1 --2- 172.21.15.161:0/2163613195 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6d414be70 0x7ff6d4153b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:36.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.476+0000 7ff6d9cfa640 1 --2- 172.21.15.161:0/2163613195 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d40aacd0 0x7ff6d41dbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:36.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.476+0000 7ff6d94f9640 1 -- 172.21.15.161:0/2163613195 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d40aacd0 msgr2=0x7ff6d41dbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:36.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.476+0000 7ff6d94f9640 1 --2- 172.21.15.161:0/2163613195 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d40aacd0 0x7ff6d41dbe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:36.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.476+0000 7ff6d94f9640 1 -- 172.21.15.161:0/2163613195 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6d414be70 msgr2=0x7ff6d4153b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:36.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.476+0000 7ff6d94f9640 1 --2- 172.21.15.161:0/2163613195 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6d414be70 0x7ff6d4153b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:36.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.476+0000 7ff6d94f9640 1 -- 172.21.15.161:0/2163613195 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d41dd2c0 con 0x7ff6d40ab6e0 2024-03-28T23:31:36.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.476+0000 7ff6d9cfa640 1 --2- 172.21.15.161:0/2163613195 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d40aacd0 0x7ff6d41dbe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:36.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.476+0000 7ff6d94f9640 1 --2- 172.21.15.161:0/2163613195 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6d40ab6e0 0x7ff6d4151440 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7ff6c400c370 tx=0x7ff6c400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:36.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.476+0000 7ff6ca7fc640 1 -- 172.21.15.161:0/2163613195 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4017070 con 0x7ff6d40ab6e0 2024-03-28T23:31:36.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.476+0000 7ff6ca7fc640 1 -- 172.21.15.161:0/2163613195 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6c40052d0 con 0x7ff6d40ab6e0 2024-03-28T23:31:36.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.476+0000 7ff6dbf5e640 1 -- 172.21.15.161:0/2163613195 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6d41dd5a0 con 0x7ff6d40ab6e0 2024-03-28T23:31:36.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.477+0000 7ff6ca7fc640 1 -- 172.21.15.161:0/2163613195 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4003850 con 0x7ff6d40ab6e0 2024-03-28T23:31:36.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.477+0000 7ff6dbf5e640 1 -- 172.21.15.161:0/2163613195 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6d4151980 con 0x7ff6d40ab6e0 2024-03-28T23:31:36.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.477+0000 7ff6dbf5e640 1 -- 172.21.15.161:0/2163613195 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6d404f4b0 con 0x7ff6d40ab6e0 2024-03-28T23:31:36.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.478+0000 7ff6ca7fc640 1 -- 172.21.15.161:0/2163613195 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff6c401a070 con 0x7ff6d40ab6e0 2024-03-28T23:31:36.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.478+0000 7ff6ca7fc640 1 --2- 172.21.15.161:0/2163613195 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff6b806b4d0 0x7ff6b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:36.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.478+0000 7ff6ca7fc640 1 -- 172.21.15.161:0/2163613195 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5249+0+0 (secure 0 0 0) 0x7ff6c40892e0 con 0x7ff6d40ab6e0 2024-03-28T23:31:36.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.478+0000 7ff6d9cfa640 1 --2- 172.21.15.161:0/2163613195 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff6b806b4d0 0x7ff6b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:36.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.479+0000 7ff6d9cfa640 1 --2- 172.21.15.161:0/2163613195 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff6b806b4d0 0x7ff6b806d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0006fd0 tx=0x7ff6d0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:36.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.483+0000 7ff6ca7fc640 1 -- 172.21.15.161:0/2163613195 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff6c4057cd0 con 0x7ff6d40ab6e0 2024-03-28T23:31:36.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.719+0000 7ff6dbf5e640 1 -- 172.21.15.161:0/2163613195 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7ff6d40c0ff0 con 0x7ff6d40ab6e0 2024-03-28T23:31:36.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.719+0000 7ff6ca7fc640 1 -- 172.21.15.161:0/2163613195 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v9) ==== 80+0+4724 (secure 0 0 0) 0x7ff6c405bd70 con 0x7ff6d40ab6e0 2024-03-28T23:31:36.720 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:36.720 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:31:36.720 INFO:teuthology.orchestra.run.smithi161.stdout: "auth_dump": [ 2024-03-28T23:31:36.720 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:31:36.720 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.a", 2024-03-28T23:31:36.721 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmggtoJhAASLXbUICLBhjRr19aUAuZHw==", 2024-03-28T23:31:36.721 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:31:36.721 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:31:36.721 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:31:36.721 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:31:36.721 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:31:36.721 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:31:36.721 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:31:36.721 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:31:36.721 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.b", 2024-03-28T23:31:36.721 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmQbEOMxAA74al6+dMzcN/ngwNZK+aHA==", 2024-03-28T23:31:36.721 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:31:36.721 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:31:36.721 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:31:36.722 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:31:36.722 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:31:36.722 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:31:36.722 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:31:36.722 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:31:36.722 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.c", 2024-03-28T23:31:36.722 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmjf/qLRAAosLbpT8aCWvrKAQnj8Uh3g==", 2024-03-28T23:31:36.722 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:31:36.722 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:31:36.722 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:31:36.722 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:31:36.722 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:31:36.722 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:31:36.723 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:31:36.723 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:31:36.723 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.d", 2024-03-28T23:31:36.723 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmpiZ1OhAAjYrFAaDosDS2Y+BvU4c6JQ==", 2024-03-28T23:31:36.723 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:31:36.723 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:31:36.723 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:31:36.723 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:31:36.723 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:31:36.723 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:31:36.723 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:31:36.723 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:31:36.723 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.0", 2024-03-28T23:31:36.723 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA9/QVmAFPmGBAAS0SkRRwsupm6jm1sqFXJ6g==", 2024-03-28T23:31:36.723 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:31:36.724 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:31:36.724 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:31:36.724 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:31:36.724 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:31:36.724 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:31:36.724 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:31:36.724 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.1", 2024-03-28T23:31:36.724 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBA/QVmNBTVFxAA1htU9ilznfLUDqKmJaQwzg==", 2024-03-28T23:31:36.724 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:31:36.724 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:31:36.724 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:31:36.724 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:31:36.724 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:31:36.725 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:31:36.725 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:31:36.725 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.2", 2024-03-28T23:31:36.725 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBD/QVmg+4YFhAA9TvfFVKT3nQT7/QUOf3Spw==", 2024-03-28T23:31:36.725 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:31:36.725 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:31:36.725 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:31:36.725 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:31:36.725 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:31:36.725 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:31:36.725 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:31:36.725 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.3", 2024-03-28T23:31:36.725 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBG/QVmgiHDFBAAQPtbZT6a1ZGmkisBX0NMgA==", 2024-03-28T23:31:36.725 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:31:36.725 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:31:36.726 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:31:36.726 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:31:36.726 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:31:36.726 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:31:36.726 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:31:36.726 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.4", 2024-03-28T23:31:36.726 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBK/QVmahhZIBAAJfotg/dusbB3usreZlRuPA==", 2024-03-28T23:31:36.726 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:31:36.726 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:31:36.726 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:31:36.726 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:31:36.726 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:31:36.726 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:31:36.726 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:31:36.727 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.5", 2024-03-28T23:31:36.727 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBN/QVmzjlDHhAAli7pLyIo+MVdwSBNJYVg5g==", 2024-03-28T23:31:36.727 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:31:36.727 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:31:36.727 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:31:36.727 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:31:36.727 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:31:36.727 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:31:36.727 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:31:36.727 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.6", 2024-03-28T23:31:36.727 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBQ/QVmtCFSHBAASVfV/5mY/3JNiI0YMT7Vww==", 2024-03-28T23:31:36.727 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:31:36.727 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:31:36.727 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:31:36.728 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:31:36.728 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:31:36.728 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:31:36.728 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:31:36.728 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.7", 2024-03-28T23:31:36.728 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBT/QVm2J7eGhAAaIeZxmLwQBBE1glCxjKbhA==", 2024-03-28T23:31:36.728 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:31:36.728 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:31:36.728 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:31:36.728 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:31:36.728 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:31:36.728 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:31:36.728 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:31:36.728 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.0", 2024-03-28T23:31:36.728 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA6/QVmc1krAxAAEKsn1Tuuv9OlFZpB4PFhDg==", 2024-03-28T23:31:36.729 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:31:36.729 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:31:36.729 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow r", 2024-03-28T23:31:36.729 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-28T23:31:36.729 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:31:36.729 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:31:36.729 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:31:36.729 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.1", 2024-03-28T23:31:36.729 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA6/QVm1B2cBhAAmHDeeFwpFlV/y3NIGWUXHQ==", 2024-03-28T23:31:36.729 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:31:36.729 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:31:36.729 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow r", 2024-03-28T23:31:36.729 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow rw", 2024-03-28T23:31:36.729 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow rwx" 2024-03-28T23:31:36.730 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:31:36.730 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:31:36.730 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:31:36.730 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.admin", 2024-03-28T23:31:36.730 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmzzJ7CRAAYqfhnIkfNJJ3O+VWC2LP+Q==", 2024-03-28T23:31:36.730 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:31:36.730 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-28T23:31:36.730 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:31:36.730 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:31:36.730 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:31:36.730 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:31:36.730 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:31:36.731 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:31:36.731 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mgr.x", 2024-03-28T23:31:36.731 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVm7TD3IhAACByGRGpvJWDpF22NuovHWw==", 2024-03-28T23:31:36.731 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:31:36.731 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-28T23:31:36.731 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile mgr", 2024-03-28T23:31:36.731 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:31:36.731 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:31:36.731 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:31:36.731 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:31:36.731 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mgr.y", 2024-03-28T23:31:36.731 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVm0hLlHhAAh8cewPtZ/TbKrU488kxrBg==", 2024-03-28T23:31:36.731 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:31:36.731 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-28T23:31:36.732 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile mgr", 2024-03-28T23:31:36.732 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:31:36.732 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:31:36.732 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:31:36.732 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-03-28T23:31:36.732 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:31:36.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.721+0000 7ff6dbf5e640 1 -- 172.21.15.161:0/2163613195 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff6b806b4d0 msgr2=0x7ff6b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:36.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.721+0000 7ff6dbf5e640 1 --2- 172.21.15.161:0/2163613195 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff6b806b4d0 0x7ff6b806d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0006fd0 tx=0x7ff6d0008040 comp rx=0 tx=0).stop 2024-03-28T23:31:36.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.721+0000 7ff6dbf5e640 1 -- 172.21.15.161:0/2163613195 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6d40ab6e0 msgr2=0x7ff6d4151440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:36.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.721+0000 7ff6dbf5e640 1 --2- 172.21.15.161:0/2163613195 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6d40ab6e0 0x7ff6d4151440 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7ff6c400c370 tx=0x7ff6c400c830 comp rx=0 tx=0).stop 2024-03-28T23:31:36.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.721+0000 7ff6dbf5e640 1 -- 172.21.15.161:0/2163613195 shutdown_connections 2024-03-28T23:31:36.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.721+0000 7ff6dbf5e640 1 --2- 172.21.15.161:0/2163613195 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff6b806b4d0 0x7ff6b806d990 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:36.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.721+0000 7ff6dbf5e640 1 --2- 172.21.15.161:0/2163613195 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6d40ab6e0 0x7ff6d4151440 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:36.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.721+0000 7ff6dbf5e640 1 --2- 172.21.15.161:0/2163613195 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6d414be70 0x7ff6d4153b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:36.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.721+0000 7ff6dbf5e640 1 --2- 172.21.15.161:0/2163613195 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d40aacd0 0x7ff6d41dbe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:36.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.721+0000 7ff6dbf5e640 1 -- 172.21.15.161:0/2163613195 >> 172.21.15.161:0/2163613195 conn(0x7ff6d40c6670 msgr2=0x7ff6d4149cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:36.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.722+0000 7ff6dbf5e640 1 -- 172.21.15.161:0/2163613195 shutdown_connections 2024-03-28T23:31:36.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.722+0000 7ff6dbf5e640 1 -- 172.21.15.161:0/2163613195 wait complete. 2024-03-28T23:31:36.735 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:31:36.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.836+0000 7f69a98cd640 1 Processor -- start 2024-03-28T23:31:36.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.836+0000 7f69a98cd640 1 -- start start 2024-03-28T23:31:36.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.836+0000 7f69a98cd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f69a406d9f0 0x7f69a4155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:36.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.836+0000 7f69a98cd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f69a406cb80 0x7f69a40c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:36.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.836+0000 7f69a98cd640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69a406c3c0 con 0x7f69a4152a80 2024-03-28T23:31:36.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.836+0000 7f69a98cd640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f69a4156250 con 0x7f69a406cb80 2024-03-28T23:31:36.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.836+0000 7f69a98cd640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f69a406c6f0 con 0x7f69a406d9f0 2024-03-28T23:31:36.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.836+0000 7f69a2ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f69a4152a80 0x7f69a40c93a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50572/0 (socket says 172.21.15.161:50572) 2024-03-28T23:31:36.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.836+0000 7f69a37fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f69a406cb80 0x7f69a40c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:36.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.837+0000 7f69a2ffd640 1 -- 172.21.15.161:0/3218552231 learned_addr learned my addr 172.21.15.161:0/3218552231 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:36.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.837+0000 7f69a27fc640 1 --2- 172.21.15.161:0/3218552231 >> v2:172.21.15.170:3300/0 conn(0x7f69a406d9f0 0x7f69a4155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:36.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.837+0000 7f69a37fe640 1 -- 172.21.15.161:0/3218552231 >> v2:172.21.15.170:3300/0 conn(0x7f69a406d9f0 msgr2=0x7f69a4155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:36.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.837+0000 7f69a37fe640 1 --2- 172.21.15.161:0/3218552231 >> v2:172.21.15.170:3300/0 conn(0x7f69a406d9f0 0x7f69a4155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:36.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.837+0000 7f69a37fe640 1 -- 172.21.15.161:0/3218552231 >> v1:172.21.15.161:6789/0 conn(0x7f69a4152a80 legacy=0x7f69a40c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:36.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.837+0000 7f69a1ffb640 1 -- 172.21.15.161:0/3218552231 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 693685450 0 0) 0x7f69a406c3c0 con 0x7f69a4152a80 2024-03-28T23:31:36.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.837+0000 7f69a37fe640 1 -- 172.21.15.161:0/3218552231 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69a40c5b90 con 0x7f69a406cb80 2024-03-28T23:31:36.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.838+0000 7f69a37fe640 1 --2- 172.21.15.161:0/3218552231 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69a406cb80 0x7f69a40c74c0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f69a40cbbb0 tx=0x7f699c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3f519b3b10370b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:36.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.838+0000 7f69a1ffb640 1 -- 172.21.15.161:0/3218552231 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699c039070 con 0x7f69a406cb80 2024-03-28T23:31:36.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.838+0000 7f69a1ffb640 1 -- 172.21.15.161:0/3218552231 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f699c02cb60 con 0x7f69a406cb80 2024-03-28T23:31:36.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.838+0000 7f69a1ffb640 1 -- 172.21.15.161:0/3218552231 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699c034440 con 0x7f69a406cb80 2024-03-28T23:31:36.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.838+0000 7f69a98cd640 1 -- 172.21.15.161:0/3218552231 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69a406cb80 msgr2=0x7f69a40c74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:36.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.838+0000 7f69a98cd640 1 --2- 172.21.15.161:0/3218552231 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69a406cb80 0x7f69a40c74c0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f69a40cbbb0 tx=0x7f699c02bdb0 comp rx=0 tx=0).stop 2024-03-28T23:31:36.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.838+0000 7f69a98cd640 1 -- 172.21.15.161:0/3218552231 shutdown_connections 2024-03-28T23:31:36.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.838+0000 7f69a98cd640 1 --2- 172.21.15.161:0/3218552231 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69a406cb80 0x7f69a40c74c0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:36.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.838+0000 7f69a98cd640 1 --2- 172.21.15.161:0/3218552231 >> v2:172.21.15.170:3300/0 conn(0x7f69a406d9f0 0x7f69a4155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:36.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.838+0000 7f69a98cd640 1 -- 172.21.15.161:0/3218552231 >> 172.21.15.161:0/3218552231 conn(0x7f69a40ac720 msgr2=0x7f69a40acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:36.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.838+0000 7f69a98cd640 1 -- 172.21.15.161:0/3218552231 shutdown_connections 2024-03-28T23:31:36.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.838+0000 7f69a98cd640 1 -- 172.21.15.161:0/3218552231 wait complete. 2024-03-28T23:31:36.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.839+0000 7f69a98cd640 1 Processor -- start 2024-03-28T23:31:36.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.840+0000 7f69a98cd640 1 -- start start 2024-03-28T23:31:36.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.840+0000 7f69a98cd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69a406cb80 0x7f69a40ae7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:36.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.840+0000 7f69a98cd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69a406d9f0 0x7f69a41eb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:36.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.840+0000 7f69a98cd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69a4152a80 0x7f69a414f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:36.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.840+0000 7f69a98cd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f69a406d010 con 0x7f69a4152a80 2024-03-28T23:31:36.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.840+0000 7f69a98cd640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f69a40c58b0 con 0x7f69a406cb80 2024-03-28T23:31:36.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.840+0000 7f69a98cd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f69a406d190 con 0x7f69a406d9f0 2024-03-28T23:31:36.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.840+0000 7f69a37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69a4152a80 0x7f69a414f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:36.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.840+0000 7f69a37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69a4152a80 0x7f69a414f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59318/0 (socket says 172.21.15.161:59318) 2024-03-28T23:31:36.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.840+0000 7f69a37fe640 1 -- 172.21.15.161:0/4139969454 learned_addr learned my addr 172.21.15.161:0/4139969454 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:36.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.841+0000 7f69a27fc640 1 --2- 172.21.15.161:0/4139969454 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69a406d9f0 0x7f69a41eb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:36.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.841+0000 7f69a37fe640 1 -- 172.21.15.161:0/4139969454 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69a406d9f0 msgr2=0x7f69a41eb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:36.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.841+0000 7f69a37fe640 1 --2- 172.21.15.161:0/4139969454 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69a406d9f0 0x7f69a41eb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:36.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.841+0000 7f69a37fe640 1 -- 172.21.15.161:0/4139969454 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69a406cb80 msgr2=0x7f69a40ae7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:36.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.841+0000 7f69a37fe640 1 --2- 172.21.15.161:0/4139969454 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69a406cb80 0x7f69a40ae7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:36.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.841+0000 7f69a37fe640 1 -- 172.21.15.161:0/4139969454 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69a40aed30 con 0x7f69a4152a80 2024-03-28T23:31:36.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.841+0000 7f69a37fe640 1 --2- 172.21.15.161:0/4139969454 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69a4152a80 0x7f69a414f830 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f699c037040 tx=0x7f699c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:36.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.841+0000 7f698bfff640 1 -- 172.21.15.161:0/4139969454 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699c044020 con 0x7f69a4152a80 2024-03-28T23:31:36.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.841+0000 7f698bfff640 1 -- 172.21.15.161:0/4139969454 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f699c03f470 con 0x7f69a4152a80 2024-03-28T23:31:36.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.841+0000 7f69a98cd640 1 -- 172.21.15.161:0/4139969454 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69a41eb600 con 0x7f69a4152a80 2024-03-28T23:31:36.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.841+0000 7f698bfff640 1 -- 172.21.15.161:0/4139969454 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699c039040 con 0x7f69a4152a80 2024-03-28T23:31:36.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.841+0000 7f69a98cd640 1 -- 172.21.15.161:0/4139969454 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69a40c38b0 con 0x7f69a4152a80 2024-03-28T23:31:36.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.842+0000 7f69a98cd640 1 -- 172.21.15.161:0/4139969454 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69a404f4b0 con 0x7f69a4152a80 2024-03-28T23:31:36.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.843+0000 7f698bfff640 1 -- 172.21.15.161:0/4139969454 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f699c047020 con 0x7f69a4152a80 2024-03-28T23:31:36.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.843+0000 7f698bfff640 1 --2- 172.21.15.161:0/4139969454 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f699006b4d0 0x7f699006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:36.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.843+0000 7f698bfff640 1 -- 172.21.15.161:0/4139969454 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5249+0+0 (secure 0 0 0) 0x7f699c0acc30 con 0x7f69a4152a80 2024-03-28T23:31:36.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.843+0000 7f69a2ffd640 1 --2- 172.21.15.161:0/4139969454 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f699006b4d0 0x7f699006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:36.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.843+0000 7f69a2ffd640 1 --2- 172.21.15.161:0/4139969454 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f699006b4d0 0x7f699006d990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6998007fd0 tx=0x7f6998007960 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:36.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:36.847+0000 7f698bfff640 1 -- 172.21.15.161:0/4139969454 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f699c04e030 con 0x7f69a4152a80 2024-03-28T23:31:37.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.020+0000 7f69a98cd640 1 -- 172.21.15.161:0/4139969454 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f69a40ca2d0 con 0x7f69a4152a80 2024-03-28T23:31:37.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.021+0000 7f698bfff640 1 -- 172.21.15.161:0/4139969454 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) ==== 74+0+13200 (secure 0 0 0) 0x7f699c07bc80 con 0x7f69a4152a80 2024-03-28T23:31:37.022 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:37.022 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":47,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:31:36.350577+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:31:37.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.023+0000 7f69a98cd640 1 -- 172.21.15.161:0/4139969454 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f699006b4d0 msgr2=0x7f699006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:37.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.023+0000 7f69a98cd640 1 --2- 172.21.15.161:0/4139969454 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f699006b4d0 0x7f699006d990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6998007fd0 tx=0x7f6998007960 comp rx=0 tx=0).stop 2024-03-28T23:31:37.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.023+0000 7f69a98cd640 1 -- 172.21.15.161:0/4139969454 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69a4152a80 msgr2=0x7f69a414f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:37.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.023+0000 7f69a98cd640 1 --2- 172.21.15.161:0/4139969454 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69a4152a80 0x7f69a414f830 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f699c037040 tx=0x7f699c004a40 comp rx=0 tx=0).stop 2024-03-28T23:31:37.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.023+0000 7f69a98cd640 1 -- 172.21.15.161:0/4139969454 shutdown_connections 2024-03-28T23:31:37.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.023+0000 7f69a98cd640 1 --2- 172.21.15.161:0/4139969454 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f699006b4d0 0x7f699006d990 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.023+0000 7f69a98cd640 1 --2- 172.21.15.161:0/4139969454 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69a4152a80 0x7f69a414f830 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.023+0000 7f69a98cd640 1 --2- 172.21.15.161:0/4139969454 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69a406d9f0 0x7f69a41eb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.023+0000 7f69a98cd640 1 --2- 172.21.15.161:0/4139969454 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69a406cb80 0x7f69a40ae7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.023+0000 7f69a98cd640 1 -- 172.21.15.161:0/4139969454 >> 172.21.15.161:0/4139969454 conn(0x7f69a40ac720 msgr2=0x7f69a40c8bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:37.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.023+0000 7f69a98cd640 1 -- 172.21.15.161:0/4139969454 shutdown_connections 2024-03-28T23:31:37.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.023+0000 7f69a98cd640 1 -- 172.21.15.161:0/4139969454 wait complete. 2024-03-28T23:31:37.034 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:31:37.035 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:31:37.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.137+0000 7f3ab150d640 1 Processor -- start 2024-03-28T23:31:37.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.137+0000 7f3ab150d640 1 -- start start 2024-03-28T23:31:37.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.138+0000 7f3ab150d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3aac152970 0x7f3aac0c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:37.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.138+0000 7f3ab150d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3aac0c7f90 0x7f3aac0c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:37.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.138+0000 7f3ab150d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3aac1554a0 con 0x7f3aac0c7930 2024-03-28T23:31:37.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.138+0000 7f3ab150d640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f3aac0c9040 con 0x7f3aac152970 2024-03-28T23:31:37.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.138+0000 7f3ab150d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3aac155dd0 con 0x7f3aac0c7f90 2024-03-28T23:31:37.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.139+0000 7f3aaa7fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3aac0c7f90 0x7f3aac0c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:37.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.139+0000 7f3aaaffd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3aac152970 0x7f3aac0c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:37.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.139+0000 7f3aaa7fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3aac0c7f90 0x7f3aac0c73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:36928/0 (socket says 172.21.15.161:36928) 2024-03-28T23:31:37.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.139+0000 7f3aaa7fc640 1 -- 172.21.15.161:0/3057090471 learned_addr learned my addr 172.21.15.161:0/3057090471 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:37.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.139+0000 7f3aab7fe640 1 --1- 172.21.15.161:0/3057090471 >> v1:172.21.15.170:6789/0 conn(0x7f3aac0c7930 0x7f3aac06c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:57928/0 (socket says 172.21.15.161:57928) 2024-03-28T23:31:37.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.139+0000 7f3aaa7fc640 1 -- 172.21.15.161:0/3057090471 >> v2:172.21.15.170:3300/0 conn(0x7f3aac152970 msgr2=0x7f3aac0c8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:37.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.139+0000 7f3aaa7fc640 1 --2- 172.21.15.161:0/3057090471 >> v2:172.21.15.170:3300/0 conn(0x7f3aac152970 0x7f3aac0c8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.139+0000 7f3aaa7fc640 1 -- 172.21.15.161:0/3057090471 >> v1:172.21.15.170:6789/0 conn(0x7f3aac0c7930 legacy=0x7f3aac06c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:37.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.139+0000 7f3aaa7fc640 1 -- 172.21.15.161:0/3057090471 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aac155780 con 0x7f3aac0c7f90 2024-03-28T23:31:37.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.140+0000 7f3aaa7fc640 1 --2- 172.21.15.161:0/3057090471 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3aac0c7f90 0x7f3aac0c73f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0009870 tx=0x7f3aa002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5be2583ee297240f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:37.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.140+0000 7f3aa9ffb640 1 -- 172.21.15.161:0/3057090471 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0035020 con 0x7f3aac0c7f90 2024-03-28T23:31:37.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.140+0000 7f3aa9ffb640 1 -- 172.21.15.161:0/3057090471 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3aa0031640 con 0x7f3aac0c7f90 2024-03-28T23:31:37.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.140+0000 7f3aa9ffb640 1 -- 172.21.15.161:0/3057090471 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa002d040 con 0x7f3aac0c7f90 2024-03-28T23:31:37.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.140+0000 7f3ab150d640 1 -- 172.21.15.161:0/3057090471 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3aac0c7f90 msgr2=0x7f3aac0c73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:37.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.140+0000 7f3ab150d640 1 --2- 172.21.15.161:0/3057090471 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3aac0c7f90 0x7f3aac0c73f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0009870 tx=0x7f3aa002b8f0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.140+0000 7f3ab150d640 1 -- 172.21.15.161:0/3057090471 shutdown_connections 2024-03-28T23:31:37.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.140+0000 7f3ab150d640 1 --2- 172.21.15.161:0/3057090471 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3aac0c7f90 0x7f3aac0c73f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.140+0000 7f3ab150d640 1 --2- 172.21.15.161:0/3057090471 >> v2:172.21.15.170:3300/0 conn(0x7f3aac152970 0x7f3aac0c8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.140+0000 7f3ab150d640 1 -- 172.21.15.161:0/3057090471 >> 172.21.15.161:0/3057090471 conn(0x7f3aac0ac570 msgr2=0x7f3aac0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:37.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.140+0000 7f3ab150d640 1 -- 172.21.15.161:0/3057090471 shutdown_connections 2024-03-28T23:31:37.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.141+0000 7f3ab150d640 1 -- 172.21.15.161:0/3057090471 wait complete. 2024-03-28T23:31:37.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.142+0000 7f3ab150d640 1 Processor -- start 2024-03-28T23:31:37.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.142+0000 7f3ab150d640 1 -- start start 2024-03-28T23:31:37.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.142+0000 7f3ab150d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3aac0c7930 0x7f3aac0c2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:37.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.142+0000 7f3ab150d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3aac0c7f90 0x7f3aac1eb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:37.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.142+0000 7f3ab150d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3aac152970 0x7f3aac1eb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:37.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.142+0000 7f3ab150d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3aac155150 con 0x7f3aac0c7f90 2024-03-28T23:31:37.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.142+0000 7f3ab150d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3aac0c6c90 con 0x7f3aac152970 2024-03-28T23:31:37.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.142+0000 7f3ab150d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3aac06c810 con 0x7f3aac0c7930 2024-03-28T23:31:37.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.143+0000 7f3aaa7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3aac0c7f90 0x7f3aac1eb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:37.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.143+0000 7f3aaa7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3aac0c7f90 0x7f3aac1eb0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59334/0 (socket says 172.21.15.161:59334) 2024-03-28T23:31:37.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.143+0000 7f3aaa7fc640 1 -- 172.21.15.161:0/2736921322 learned_addr learned my addr 172.21.15.161:0/2736921322 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:37.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.143+0000 7f3aaaffd640 1 --2- 172.21.15.161:0/2736921322 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3aac0c7930 0x7f3aac0c2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:37.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.143+0000 7f3aab7fe640 1 --2- 172.21.15.161:0/2736921322 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3aac152970 0x7f3aac1eb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:37.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.143+0000 7f3aaa7fc640 1 -- 172.21.15.161:0/2736921322 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3aac0c7930 msgr2=0x7f3aac0c2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:37.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.143+0000 7f3aaa7fc640 1 --2- 172.21.15.161:0/2736921322 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3aac0c7930 0x7f3aac0c2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.143+0000 7f3aaa7fc640 1 -- 172.21.15.161:0/2736921322 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3aac152970 msgr2=0x7f3aac1eb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:37.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.143+0000 7f3aaa7fc640 1 --2- 172.21.15.161:0/2736921322 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3aac152970 0x7f3aac1eb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.143+0000 7f3aaa7fc640 1 -- 172.21.15.161:0/2736921322 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aac1ebbc0 con 0x7f3aac0c7f90 2024-03-28T23:31:37.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.143+0000 7f3aab7fe640 1 --2- 172.21.15.161:0/2736921322 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3aac152970 0x7f3aac1eb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:37.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.143+0000 7f3aaa7fc640 1 --2- 172.21.15.161:0/2736921322 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3aac0c7f90 0x7f3aac1eb0a0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0000c00 tx=0x7f3aa00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:37.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.143+0000 7f3a937fe640 1 -- 172.21.15.161:0/2736921322 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0040070 con 0x7f3aac0c7f90 2024-03-28T23:31:37.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.143+0000 7f3a937fe640 1 -- 172.21.15.161:0/2736921322 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3aa00399f0 con 0x7f3aac0c7f90 2024-03-28T23:31:37.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.143+0000 7f3ab150d640 1 -- 172.21.15.161:0/2736921322 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3aac0c2020 con 0x7f3aac0c7f90 2024-03-28T23:31:37.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.143+0000 7f3a937fe640 1 -- 172.21.15.161:0/2736921322 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0035040 con 0x7f3aac0c7f90 2024-03-28T23:31:37.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.143+0000 7f3ab150d640 1 -- 172.21.15.161:0/2736921322 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3aac0c1d20 con 0x7f3aac0c7f90 2024-03-28T23:31:37.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.144+0000 7f3ab150d640 1 -- 172.21.15.161:0/2736921322 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3aac04f4b0 con 0x7f3aac0c7f90 2024-03-28T23:31:37.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.145+0000 7f3a937fe640 1 -- 172.21.15.161:0/2736921322 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3aa004b060 con 0x7f3aac0c7f90 2024-03-28T23:31:37.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.145+0000 7f3a937fe640 1 --2- 172.21.15.161:0/2736921322 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3a9806b4d0 0x7f3a9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:37.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.145+0000 7f3a937fe640 1 -- 172.21.15.161:0/2736921322 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5249+0+0 (secure 0 0 0) 0x7f3aa00ab890 con 0x7f3aac0c7f90 2024-03-28T23:31:37.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.145+0000 7f3aaaffd640 1 --2- 172.21.15.161:0/2736921322 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3a9806b4d0 0x7f3a9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:37.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.146+0000 7f3aaaffd640 1 --2- 172.21.15.161:0/2736921322 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3a9806b4d0 0x7f3a9806d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3aac0c1950 tx=0x7f3a9c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:37.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.150+0000 7f3a937fe640 1 -- 172.21.15.161:0/2736921322 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3aa007a960 con 0x7f3aac0c7f90 2024-03-28T23:31:37.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.322+0000 7f3ab150d640 1 -- 172.21.15.161:0/2736921322 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3aac0b0960 con 0x7f3aac0c7f90 2024-03-28T23:31:37.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.322+0000 7f3a937fe640 1 -- 172.21.15.161:0/2736921322 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) ==== 93+0+10 (secure 0 0 0) 0x7f3aa007a260 con 0x7f3aac0c7f90 2024-03-28T23:31:37.323 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:31:37.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.324+0000 7f3ab150d640 1 -- 172.21.15.161:0/2736921322 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3a9806b4d0 msgr2=0x7f3a9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:37.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.324+0000 7f3ab150d640 1 --2- 172.21.15.161:0/2736921322 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3a9806b4d0 0x7f3a9806d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3aac0c1950 tx=0x7f3a9c00b040 comp rx=0 tx=0).stop 2024-03-28T23:31:37.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.324+0000 7f3ab150d640 1 -- 172.21.15.161:0/2736921322 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3aac0c7f90 msgr2=0x7f3aac1eb0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:37.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.324+0000 7f3ab150d640 1 --2- 172.21.15.161:0/2736921322 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3aac0c7f90 0x7f3aac1eb0a0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0000c00 tx=0x7f3aa00046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.324+0000 7f3ab150d640 1 -- 172.21.15.161:0/2736921322 shutdown_connections 2024-03-28T23:31:37.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.324+0000 7f3ab150d640 1 --2- 172.21.15.161:0/2736921322 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3a9806b4d0 0x7f3a9806d990 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.324+0000 7f3ab150d640 1 --2- 172.21.15.161:0/2736921322 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3aac0c7f90 0x7f3aac1eb0a0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.324+0000 7f3ab150d640 1 --2- 172.21.15.161:0/2736921322 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3aac152970 0x7f3aac1eb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.324+0000 7f3ab150d640 1 --2- 172.21.15.161:0/2736921322 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3aac0c7930 0x7f3aac0c2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.324+0000 7f3ab150d640 1 -- 172.21.15.161:0/2736921322 >> 172.21.15.161:0/2736921322 conn(0x7f3aac0ac570 msgr2=0x7f3aac14e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:37.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.324+0000 7f3ab150d640 1 -- 172.21.15.161:0/2736921322 shutdown_connections 2024-03-28T23:31:37.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.324+0000 7f3ab150d640 1 -- 172.21.15.161:0/2736921322 wait complete. 2024-03-28T23:31:37.336 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T23:31:37.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.437+0000 7f722db55640 1 Processor -- start 2024-03-28T23:31:37.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.437+0000 7f722db55640 1 -- start start 2024-03-28T23:31:37.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.438+0000 7f722db55640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f72281504d0 0x7f72281508b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:37.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.438+0000 7f722db55640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7228152f90 0x7f72280c10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:37.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.438+0000 7f722db55640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7228153f60 con 0x7f72280aa290 2024-03-28T23:31:37.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.438+0000 7f722db55640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f72280aa080 con 0x7f72281504d0 2024-03-28T23:31:37.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.438+0000 7f722db55640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f72280c17b0 con 0x7f7228152f90 2024-03-28T23:31:37.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.438+0000 7f7226ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7228152f90 0x7f72280c10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:37.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.438+0000 7f72277fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f72281504d0 0x7f72281508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:37.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.438+0000 7f7226ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7228152f90 0x7f72280c10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:36952/0 (socket says 172.21.15.161:36952) 2024-03-28T23:31:37.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.438+0000 7f7226ffd640 1 -- 172.21.15.161:0/1893975011 learned_addr learned my addr 172.21.15.161:0/1893975011 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:37.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.438+0000 7f7226ffd640 1 -- 172.21.15.161:0/1893975011 >> v2:172.21.15.170:3300/0 conn(0x7f72281504d0 msgr2=0x7f72281508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:37.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.438+0000 7f7226ffd640 1 --2- 172.21.15.161:0/1893975011 >> v2:172.21.15.170:3300/0 conn(0x7f72281504d0 0x7f72281508b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.438+0000 7f7226ffd640 1 -- 172.21.15.161:0/1893975011 >> v1:172.21.15.170:6789/0 conn(0x7f72280aa290 legacy=0x7f72280a8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:37.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.439+0000 7f7226ffd640 1 -- 172.21.15.161:0/1893975011 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72280abfb0 con 0x7f7228152f90 2024-03-28T23:31:37.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.439+0000 7f7226ffd640 1 --2- 172.21.15.161:0/1893975011 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7228152f90 0x7f72280c10d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f721c009870 tx=0x7f721c02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40f5c35c8a041778 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:37.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.439+0000 7f72267fc640 1 -- 172.21.15.161:0/1893975011 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f721c0317c0 con 0x7f7228152f90 2024-03-28T23:31:37.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.439+0000 7f72267fc640 1 -- 172.21.15.161:0/1893975011 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f721c037070 con 0x7f7228152f90 2024-03-28T23:31:37.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.439+0000 7f72267fc640 1 -- 172.21.15.161:0/1893975011 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f721c031b80 con 0x7f7228152f90 2024-03-28T23:31:37.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.440+0000 7f722db55640 1 -- 172.21.15.161:0/1893975011 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7228152f90 msgr2=0x7f72280c10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:37.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.440+0000 7f722db55640 1 --2- 172.21.15.161:0/1893975011 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7228152f90 0x7f72280c10d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f721c009870 tx=0x7f721c02b860 comp rx=0 tx=0).stop 2024-03-28T23:31:37.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.440+0000 7f722db55640 1 -- 172.21.15.161:0/1893975011 shutdown_connections 2024-03-28T23:31:37.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.440+0000 7f722db55640 1 --2- 172.21.15.161:0/1893975011 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7228152f90 0x7f72280c10d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.440+0000 7f722db55640 1 --2- 172.21.15.161:0/1893975011 >> v2:172.21.15.170:3300/0 conn(0x7f72281504d0 0x7f72281508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.440+0000 7f722db55640 1 -- 172.21.15.161:0/1893975011 >> 172.21.15.161:0/1893975011 conn(0x7f72280c66d0 msgr2=0x7f72280c6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:37.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.440+0000 7f722db55640 1 -- 172.21.15.161:0/1893975011 shutdown_connections 2024-03-28T23:31:37.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.440+0000 7f722db55640 1 -- 172.21.15.161:0/1893975011 wait complete. 2024-03-28T23:31:37.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.441+0000 7f722db55640 1 Processor -- start 2024-03-28T23:31:37.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.441+0000 7f722db55640 1 -- start start 2024-03-28T23:31:37.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.442+0000 7f722db55640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72280aa290 0x7f72280bbb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:37.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.442+0000 7f722db55640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72281504d0 0x7f72281eb660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:37.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.442+0000 7f722db55640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7228152f90 0x7f72281ebba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:37.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.442+0000 7f722db55640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f72280a8d80 con 0x7f72280aa290 2024-03-28T23:31:37.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.442+0000 7f722db55640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f72280ac570 con 0x7f72281504d0 2024-03-28T23:31:37.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.442+0000 7f722db55640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f72280a8f00 con 0x7f7228152f90 2024-03-28T23:31:37.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.442+0000 7f72277fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72280aa290 0x7f72280bbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:37.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.442+0000 7f72277fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72280aa290 0x7f72280bbb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59338/0 (socket says 172.21.15.161:59338) 2024-03-28T23:31:37.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.442+0000 7f72277fe640 1 -- 172.21.15.161:0/1846714353 learned_addr learned my addr 172.21.15.161:0/1846714353 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:37.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.442+0000 7f7226ffd640 1 --2- 172.21.15.161:0/1846714353 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72281504d0 0x7f72281eb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:37.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.442+0000 7f7227fff640 1 --2- 172.21.15.161:0/1846714353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7228152f90 0x7f72281ebba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:37.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.442+0000 7f72277fe640 1 -- 172.21.15.161:0/1846714353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7228152f90 msgr2=0x7f72281ebba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:37.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.442+0000 7f72277fe640 1 --2- 172.21.15.161:0/1846714353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7228152f90 0x7f72281ebba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.442+0000 7f72277fe640 1 -- 172.21.15.161:0/1846714353 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72281504d0 msgr2=0x7f72281eb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:37.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.442+0000 7f72277fe640 1 --2- 172.21.15.161:0/1846714353 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72281504d0 0x7f72281eb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.442+0000 7f72277fe640 1 -- 172.21.15.161:0/1846714353 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72281ea5c0 con 0x7f72280aa290 2024-03-28T23:31:37.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.442+0000 7f7226ffd640 1 --2- 172.21.15.161:0/1846714353 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72281504d0 0x7f72281eb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:37.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.443+0000 7f72277fe640 1 --2- 172.21.15.161:0/1846714353 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72280aa290 0x7f72280bbb90 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f721800e9e0 tx=0x7f721800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:37.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.443+0000 7f720ffff640 1 -- 172.21.15.161:0/1846714353 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7218017070 con 0x7f72280aa290 2024-03-28T23:31:37.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.443+0000 7f720ffff640 1 -- 172.21.15.161:0/1846714353 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f721800c9d0 con 0x7f72280aa290 2024-03-28T23:31:37.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.443+0000 7f722db55640 1 -- 172.21.15.161:0/1846714353 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72281ea8a0 con 0x7f72280aa290 2024-03-28T23:31:37.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.443+0000 7f720ffff640 1 -- 172.21.15.161:0/1846714353 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f721800cdc0 con 0x7f72280aa290 2024-03-28T23:31:37.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.443+0000 7f722db55640 1 -- 172.21.15.161:0/1846714353 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72281eae60 con 0x7f72280aa290 2024-03-28T23:31:37.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.444+0000 7f722db55640 1 -- 172.21.15.161:0/1846714353 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f722804f550 con 0x7f72280aa290 2024-03-28T23:31:37.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.444+0000 7f720ffff640 1 -- 172.21.15.161:0/1846714353 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f721801a070 con 0x7f72280aa290 2024-03-28T23:31:37.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.445+0000 7f720ffff640 1 --2- 172.21.15.161:0/1846714353 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f721406b4d0 0x7f721406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:37.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.445+0000 7f720ffff640 1 -- 172.21.15.161:0/1846714353 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5249+0+0 (secure 0 0 0) 0x7f721808a400 con 0x7f72280aa290 2024-03-28T23:31:37.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.445+0000 7f7226ffd640 1 --2- 172.21.15.161:0/1846714353 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f721406b4d0 0x7f721406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:37.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.445+0000 7f7226ffd640 1 --2- 172.21.15.161:0/1846714353 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f721406b4d0 0x7f721406d990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f721c009870 tx=0x7f721c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:37.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.449+0000 7f720ffff640 1 -- 172.21.15.161:0/1846714353 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7218059450 con 0x7f72280aa290 2024-03-28T23:31:37.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.621+0000 7f722db55640 1 -- 172.21.15.161:0/1846714353 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f72280693e0 con 0x7f72280aa290 2024-03-28T23:31:37.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.621+0000 7f720ffff640 1 -- 172.21.15.161:0/1846714353 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v47) ==== 84+0+65 (secure 0 0 0) 0x7f7218058d50 con 0x7f72280aa290 2024-03-28T23:31:37.622 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:37.622 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2024-03-28T23:31:37.622 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:31:37.622 INFO:teuthology.orchestra.run.smithi161.stdout: "poolnum": 1, 2024-03-28T23:31:37.623 INFO:teuthology.orchestra.run.smithi161.stdout: "poolname": ".mgr" 2024-03-28T23:31:37.623 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:31:37.623 INFO:teuthology.orchestra.run.smithi161.stdout:] 2024-03-28T23:31:37.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.623+0000 7f722db55640 1 -- 172.21.15.161:0/1846714353 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f721406b4d0 msgr2=0x7f721406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:37.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.623+0000 7f722db55640 1 --2- 172.21.15.161:0/1846714353 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f721406b4d0 0x7f721406d990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f721c009870 tx=0x7f721c008030 comp rx=0 tx=0).stop 2024-03-28T23:31:37.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.623+0000 7f722db55640 1 -- 172.21.15.161:0/1846714353 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72280aa290 msgr2=0x7f72280bbb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:37.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.623+0000 7f722db55640 1 --2- 172.21.15.161:0/1846714353 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72280aa290 0x7f72280bbb90 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f721800e9e0 tx=0x7f721800eea0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.623+0000 7f722db55640 1 -- 172.21.15.161:0/1846714353 shutdown_connections 2024-03-28T23:31:37.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.623+0000 7f722db55640 1 --2- 172.21.15.161:0/1846714353 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f721406b4d0 0x7f721406d990 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.623+0000 7f722db55640 1 --2- 172.21.15.161:0/1846714353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7228152f90 0x7f72281ebba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.623+0000 7f722db55640 1 --2- 172.21.15.161:0/1846714353 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72281504d0 0x7f72281eb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.623+0000 7f722db55640 1 --2- 172.21.15.161:0/1846714353 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72280aa290 0x7f72280bbb90 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.623+0000 7f722db55640 1 -- 172.21.15.161:0/1846714353 >> 172.21.15.161:0/1846714353 conn(0x7f72280c66d0 msgr2=0x7f72280a8670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:37.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.623+0000 7f722db55640 1 -- 172.21.15.161:0/1846714353 shutdown_connections 2024-03-28T23:31:37.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.623+0000 7f722db55640 1 -- 172.21.15.161:0/1846714353 wait complete. 2024-03-28T23:31:37.634 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-28T23:31:37.635 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-28T23:31:37.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.739+0000 7f5528e81640 1 Processor -- start 2024-03-28T23:31:37.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.739+0000 7f5528e81640 1 -- start start 2024-03-28T23:31:37.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.740+0000 7f5528e81640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f55241567c0 0x7f55240c3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:37.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.740+0000 7f5528e81640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f55240ac180 0x7f5524150460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:37.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.740+0000 7f5528e81640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55240be210 con 0x7f552406beb0 2024-03-28T23:31:37.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.740+0000 7f5528e81640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f55240abfe0 con 0x7f55241567c0 2024-03-28T23:31:37.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.740+0000 7f5528e81640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f552406da60 con 0x7f55240ac180 2024-03-28T23:31:37.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.740+0000 7f551bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f55241567c0 0x7f55240c3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:37.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.740+0000 7f551bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f55241567c0 0x7f55240c3400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59350/0 (socket says 172.21.15.161:59350) 2024-03-28T23:31:37.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.740+0000 7f551bfff640 1 -- 172.21.15.161:0/3258663560 learned_addr learned my addr 172.21.15.161:0/3258663560 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:37.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.740+0000 7f5522d9d640 1 --2- 172.21.15.161:0/3258663560 >> v2:172.21.15.170:3300/0 conn(0x7f55240ac180 0x7f5524150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:37.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.740+0000 7f551bfff640 1 -- 172.21.15.161:0/3258663560 >> v2:172.21.15.170:3300/0 conn(0x7f55240ac180 msgr2=0x7f5524150460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:37.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.740+0000 7f551bfff640 1 --2- 172.21.15.161:0/3258663560 >> v2:172.21.15.170:3300/0 conn(0x7f55240ac180 0x7f5524150460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.740+0000 7f551bfff640 1 -- 172.21.15.161:0/3258663560 >> v1:172.21.15.170:6789/0 conn(0x7f552406beb0 legacy=0x7f552406eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:37.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.740+0000 7f551bfff640 1 -- 172.21.15.161:0/3258663560 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f552406d4a0 con 0x7f55241567c0 2024-03-28T23:31:37.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.740+0000 7f5522d9d640 1 --2- 172.21.15.161:0/3258663560 >> v2:172.21.15.170:3300/0 conn(0x7f55240ac180 0x7f5524150460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:37.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.741+0000 7f551bfff640 1 --2- 172.21.15.161:0/3258663560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55241567c0 0x7f55240c3400 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f55240ac740 tx=0x7f550c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bdc0c6cc7f8d3be server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:37.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.741+0000 7f5521d9b640 1 -- 172.21.15.161:0/3258663560 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550c02e070 con 0x7f55241567c0 2024-03-28T23:31:37.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.741+0000 7f5521d9b640 1 -- 172.21.15.161:0/3258663560 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f550c037070 con 0x7f55241567c0 2024-03-28T23:31:37.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.741+0000 7f5521d9b640 1 -- 172.21.15.161:0/3258663560 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550c031a00 con 0x7f55241567c0 2024-03-28T23:31:37.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.741+0000 7f5528e81640 1 -- 172.21.15.161:0/3258663560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55241567c0 msgr2=0x7f55240c3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:37.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.741+0000 7f5528e81640 1 --2- 172.21.15.161:0/3258663560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55241567c0 0x7f55240c3400 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f55240ac740 tx=0x7f550c02b880 comp rx=0 tx=0).stop 2024-03-28T23:31:37.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.741+0000 7f5528e81640 1 -- 172.21.15.161:0/3258663560 shutdown_connections 2024-03-28T23:31:37.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.741+0000 7f5528e81640 1 --2- 172.21.15.161:0/3258663560 >> v2:172.21.15.170:3300/0 conn(0x7f55240ac180 0x7f5524150460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.741+0000 7f5528e81640 1 --2- 172.21.15.161:0/3258663560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55241567c0 0x7f55240c3400 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.741+0000 7f5528e81640 1 -- 172.21.15.161:0/3258663560 >> 172.21.15.161:0/3258663560 conn(0x7f552406b150 msgr2=0x7f552406b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:37.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.742+0000 7f5528e81640 1 -- 172.21.15.161:0/3258663560 shutdown_connections 2024-03-28T23:31:37.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.742+0000 7f5528e81640 1 -- 172.21.15.161:0/3258663560 wait complete. 2024-03-28T23:31:37.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.743+0000 7f5528e81640 1 Processor -- start 2024-03-28T23:31:37.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.743+0000 7f5528e81640 1 -- start start 2024-03-28T23:31:37.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.743+0000 7f5528e81640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f552406beb0 0x7f55240bf430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:37.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.743+0000 7f5528e81640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55240ac180 0x7f55241e9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:37.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.743+0000 7f5528e81640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55241567c0 0x7f5524153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:37.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.743+0000 7f5528e81640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f552406f700 con 0x7f55240ac180 2024-03-28T23:31:37.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.743+0000 7f5528e81640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f55240bc130 con 0x7f552406beb0 2024-03-28T23:31:37.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.743+0000 7f5528e81640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f552406f880 con 0x7f55241567c0 2024-03-28T23:31:37.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.744+0000 7f551bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55240ac180 0x7f55241e9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:37.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.744+0000 7f551bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55240ac180 0x7f55241e9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59364/0 (socket says 172.21.15.161:59364) 2024-03-28T23:31:37.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.744+0000 7f551bfff640 1 -- 172.21.15.161:0/1958930345 learned_addr learned my addr 172.21.15.161:0/1958930345 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:37.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.744+0000 7f5522d9d640 1 --2- 172.21.15.161:0/1958930345 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55241567c0 0x7f5524153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:37.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.744+0000 7f552259c640 1 --2- 172.21.15.161:0/1958930345 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f552406beb0 0x7f55240bf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:37.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.744+0000 7f551bfff640 1 -- 172.21.15.161:0/1958930345 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55241567c0 msgr2=0x7f5524153480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:37.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.744+0000 7f551bfff640 1 --2- 172.21.15.161:0/1958930345 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55241567c0 0x7f5524153480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.744+0000 7f551bfff640 1 -- 172.21.15.161:0/1958930345 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f552406beb0 msgr2=0x7f55240bf430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:37.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.744+0000 7f551bfff640 1 --2- 172.21.15.161:0/1958930345 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f552406beb0 0x7f55240bf430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:37.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.744+0000 7f551bfff640 1 -- 172.21.15.161:0/1958930345 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55241e9cb0 con 0x7f55240ac180 2024-03-28T23:31:37.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.744+0000 7f551bfff640 1 --2- 172.21.15.161:0/1958930345 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55240ac180 0x7f55241e9770 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f550c02b8c0 tx=0x7f550c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:37.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.744+0000 7f551affd640 1 -- 172.21.15.161:0/1958930345 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550c032880 con 0x7f55240ac180 2024-03-28T23:31:37.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.744+0000 7f551affd640 1 -- 172.21.15.161:0/1958930345 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f550c032a40 con 0x7f55240ac180 2024-03-28T23:31:37.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.744+0000 7f551affd640 1 -- 172.21.15.161:0/1958930345 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550c032e30 con 0x7f55240ac180 2024-03-28T23:31:37.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.744+0000 7f5528e81640 1 -- 172.21.15.161:0/1958930345 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55241449d0 con 0x7f55240ac180 2024-03-28T23:31:37.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.744+0000 7f5528e81640 1 -- 172.21.15.161:0/1958930345 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5524144f90 con 0x7f55240ac180 2024-03-28T23:31:37.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.745+0000 7f5528e81640 1 -- 172.21.15.161:0/1958930345 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55240693e0 con 0x7f55240ac180 2024-03-28T23:31:37.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.746+0000 7f551affd640 1 -- 172.21.15.161:0/1958930345 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f550c0453f0 con 0x7f55240ac180 2024-03-28T23:31:37.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.746+0000 7f551affd640 1 --2- 172.21.15.161:0/1958930345 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f550406b4d0 0x7f550406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:37.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.746+0000 7f551affd640 1 -- 172.21.15.161:0/1958930345 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5249+0+0 (secure 0 0 0) 0x7f550c0ac260 con 0x7f55240ac180 2024-03-28T23:31:37.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.746+0000 7f552259c640 1 --2- 172.21.15.161:0/1958930345 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f550406b4d0 0x7f550406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:37.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.746+0000 7f552259c640 1 --2- 172.21.15.161:0/1958930345 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f550406b4d0 0x7f550406d990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5514006fd0 tx=0x7f5514008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:37.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.750+0000 7f551affd640 1 -- 172.21.15.161:0/1958930345 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f55240693e0 con 0x7f55240ac180 2024-03-28T23:31:37.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:37.922+0000 7f5528e81640 1 -- 172.21.15.161:0/1958930345 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f5524068f90 con 0x7f55240ac180 2024-03-28T23:31:38.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.363+0000 7f551affd640 1 -- 172.21.15.161:0/1958930345 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v48) ==== 138+0+0 (secure 0 0 0) 0x7f55241e9cb0 con 0x7f55240ac180 2024-03-28T23:31:38.364 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_metadata' created 2024-03-28T23:31:38.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.366+0000 7f5528e81640 1 -- 172.21.15.161:0/1958930345 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f550406b4d0 msgr2=0x7f550406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:38.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.366+0000 7f5528e81640 1 --2- 172.21.15.161:0/1958930345 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f550406b4d0 0x7f550406d990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5514006fd0 tx=0x7f5514008040 comp rx=0 tx=0).stop 2024-03-28T23:31:38.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.366+0000 7f5528e81640 1 -- 172.21.15.161:0/1958930345 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55240ac180 msgr2=0x7f55241e9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:38.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.366+0000 7f5528e81640 1 --2- 172.21.15.161:0/1958930345 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55240ac180 0x7f55241e9770 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f550c02b8c0 tx=0x7f550c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:38.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.366+0000 7f5528e81640 1 -- 172.21.15.161:0/1958930345 shutdown_connections 2024-03-28T23:31:38.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.366+0000 7f5528e81640 1 --2- 172.21.15.161:0/1958930345 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f550406b4d0 0x7f550406d990 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:38.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.366+0000 7f5528e81640 1 --2- 172.21.15.161:0/1958930345 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55240ac180 0x7f55241e9770 secure :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f550c02b8c0 tx=0x7f550c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:38.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.366+0000 7f5528e81640 1 --2- 172.21.15.161:0/1958930345 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55241567c0 0x7f5524153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:38.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.366+0000 7f5528e81640 1 --2- 172.21.15.161:0/1958930345 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f552406beb0 0x7f55240bf430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:38.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.366+0000 7f5528e81640 1 -- 172.21.15.161:0/1958930345 >> 172.21.15.161:0/1958930345 conn(0x7f552406b150 msgr2=0x7f552406b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:38.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.367+0000 7f5528e81640 1 -- 172.21.15.161:0/1958930345 shutdown_connections 2024-03-28T23:31:38.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.367+0000 7f5528e81640 1 -- 172.21.15.161:0/1958930345 wait complete. 2024-03-28T23:31:38.381 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T23:31:38.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.488+0000 7f79cba0e640 1 Processor -- start 2024-03-28T23:31:38.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.489+0000 7f79cba0e640 1 -- start start 2024-03-28T23:31:38.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.489+0000 7f79cba0e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79c4152bd0 0x7f79c415d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:38.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.489+0000 7f79cba0e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79c40c2f10 con 0x7f79c406cdd0 2024-03-28T23:31:38.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.489+0000 7f79cba0e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79c414f9a0 con 0x7f79c414f5c0 2024-03-28T23:31:38.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.489+0000 7f79cba0e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f79c40c74d0 con 0x7f79c4152bd0 2024-03-28T23:31:38.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.490+0000 7f79c8fa9640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f79c406cdd0 0x7f79c41524c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50588/0 (socket says 172.21.15.161:50588) 2024-03-28T23:31:38.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.490+0000 7f79c8fa9640 1 -- 172.21.15.161:0/3519878833 learned_addr learned my addr 172.21.15.161:0/3519878833 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:38.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.490+0000 7f79c9fab640 1 --2- 172.21.15.161:0/3519878833 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79c4152bd0 0x7f79c415d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:38.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.490+0000 7f79bbfff640 1 -- 172.21.15.161:0/3519878833 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2065743389 0 0) 0x7f79c0001110 con 0x7f79c406cdd0 2024-03-28T23:31:38.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.490+0000 7f79bbfff640 1 -- 172.21.15.161:0/3519878833 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79b0003520 con 0x7f79c406cdd0 2024-03-28T23:31:38.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.490+0000 7f79bbfff640 1 -- 172.21.15.161:0/3519878833 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1656301880 0 0) 0x7f79b0003520 con 0x7f79c406cdd0 2024-03-28T23:31:38.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.490+0000 7f79bbfff640 1 -- 172.21.15.161:0/3519878833 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79c4152bd0 msgr2=0x7f79c415d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:38.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.490+0000 7f79bbfff640 1 --2- 172.21.15.161:0/3519878833 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79c4152bd0 0x7f79c415d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:38.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.490+0000 7f79bbfff640 1 -- 172.21.15.161:0/3519878833 >> v1:172.21.15.170:6789/0 conn(0x7f79c414f5c0 legacy=0x7f79c40c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:38.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.490+0000 7f79bbfff640 1 -- 172.21.15.161:0/3519878833 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79c406c140 con 0x7f79c406cdd0 2024-03-28T23:31:38.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.490+0000 7f79bbfff640 1 -- 172.21.15.161:0/3519878833 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f79c0001840 con 0x7f79c406cdd0 2024-03-28T23:31:38.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.490+0000 7f79bbfff640 1 -- 172.21.15.161:0/3519878833 >> v1:172.21.15.161:6789/0 conn(0x7f79c406cdd0 legacy=0x7f79c41524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:38.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.490+0000 7f79bbfff640 1 --2- 172.21.15.161:0/3519878833 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79b0003ac0 0x7f79b0003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:38.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.490+0000 7f79bbfff640 1 --2- 172.21.15.161:0/3519878833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79b0004800 0x7f79b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:38.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.490+0000 7f79bbfff640 1 --2- 172.21.15.161:0/3519878833 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79c4152bd0 0x7f79b0005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:38.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.490+0000 7f79bbfff640 1 -- 172.21.15.161:0/3519878833 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f79c40c74d0 con 0x7f79b0004800 2024-03-28T23:31:38.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.490+0000 7f79bbfff640 1 -- 172.21.15.161:0/3519878833 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f79b000bb50 con 0x7f79b0003ac0 2024-03-28T23:31:38.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.490+0000 7f79bbfff640 1 -- 172.21.15.161:0/3519878833 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f79b000bd20 con 0x7f79c4152bd0 2024-03-28T23:31:38.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.490+0000 7f79c8fa9640 1 --2- 172.21.15.161:0/3519878833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79b0004800 0x7f79b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:38.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.490+0000 7f79c97aa640 1 --2- 172.21.15.161:0/3519878833 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79b0003ac0 0x7f79b0003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:38.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.490+0000 7f79c9fab640 1 --2- 172.21.15.161:0/3519878833 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79c4152bd0 0x7f79b0005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:38.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.491+0000 7f79c8fa9640 1 -- 172.21.15.161:0/3519878833 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79c4152bd0 msgr2=0x7f79b0005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:38.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.491+0000 7f79c8fa9640 1 --2- 172.21.15.161:0/3519878833 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79c4152bd0 0x7f79b0005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:38.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.491+0000 7f79c8fa9640 1 -- 172.21.15.161:0/3519878833 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79b0003ac0 msgr2=0x7f79b0003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:38.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.491+0000 7f79c8fa9640 1 --2- 172.21.15.161:0/3519878833 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79b0003ac0 0x7f79b0003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:38.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.491+0000 7f79c8fa9640 1 -- 172.21.15.161:0/3519878833 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b0003520 con 0x7f79b0004800 2024-03-28T23:31:38.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.491+0000 7f79c8fa9640 1 --2- 172.21.15.161:0/3519878833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79b0004800 0x7f79b0004c80 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f79c000c490 tx=0x7f79b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=179e08d1b2325c95 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:38.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.491+0000 7f79bbfff640 1 -- 172.21.15.161:0/3519878833 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c000c0e0 con 0x7f79b0004800 2024-03-28T23:31:38.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.491+0000 7f79bbfff640 1 -- 172.21.15.161:0/3519878833 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79c0030070 con 0x7f79b0004800 2024-03-28T23:31:38.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.491+0000 7f79bbfff640 1 -- 172.21.15.161:0/3519878833 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c00058d0 con 0x7f79b0004800 2024-03-28T23:31:38.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.492+0000 7f79cba0e640 1 -- 172.21.15.161:0/3519878833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79b0004800 msgr2=0x7f79b0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:38.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.492+0000 7f79cba0e640 1 --2- 172.21.15.161:0/3519878833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79b0004800 0x7f79b0004c80 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f79c000c490 tx=0x7f79b0005830 comp rx=0 tx=0).stop 2024-03-28T23:31:38.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.492+0000 7f79c9fab640 1 -- 172.21.15.161:0/3519878833 reap_dead start 2024-03-28T23:31:38.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.492+0000 7f79cba0e640 1 -- 172.21.15.161:0/3519878833 shutdown_connections 2024-03-28T23:31:38.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.492+0000 7f79cba0e640 1 -- 172.21.15.161:0/3519878833 >> 172.21.15.161:0/3519878833 conn(0x7f79c40a83f0 msgr2=0x7f79c40a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:38.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.492+0000 7f79cba0e640 1 -- 172.21.15.161:0/3519878833 shutdown_connections 2024-03-28T23:31:38.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.492+0000 7f79cba0e640 1 -- 172.21.15.161:0/3519878833 wait complete. 2024-03-28T23:31:38.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.493+0000 7f79cba0e640 1 Processor -- start 2024-03-28T23:31:38.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.493+0000 7f79cba0e640 1 -- start start 2024-03-28T23:31:38.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.494+0000 7f79cba0e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79b0004800 0x7f79c4155bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:38.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.494+0000 7f79cba0e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79c4156130 0x7f79c40728b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:38.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.494+0000 7f79cba0e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79c4149030 0x7f79c41e9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:38.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.494+0000 7f79cba0e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f79c4152030 con 0x7f79c4149030 2024-03-28T23:31:38.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.494+0000 7f79cba0e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f79c40c6fa0 con 0x7f79b0004800 2024-03-28T23:31:38.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.494+0000 7f79cba0e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f79c406c3e0 con 0x7f79c4156130 2024-03-28T23:31:38.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.494+0000 7f79c9fab640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79c4149030 0x7f79c41e9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:38.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.494+0000 7f79c9fab640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79c4149030 0x7f79c41e9180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59380/0 (socket says 172.21.15.161:59380) 2024-03-28T23:31:38.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.494+0000 7f79c9fab640 1 -- 172.21.15.161:0/2973801027 learned_addr learned my addr 172.21.15.161:0/2973801027 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:38.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.494+0000 7f79c97aa640 1 --2- 172.21.15.161:0/2973801027 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79b0004800 0x7f79c4155bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:38.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.494+0000 7f79c8fa9640 1 --2- 172.21.15.161:0/2973801027 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79c4156130 0x7f79c40728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:38.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.494+0000 7f79c9fab640 1 -- 172.21.15.161:0/2973801027 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79c4156130 msgr2=0x7f79c40728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:38.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.494+0000 7f79c9fab640 1 --2- 172.21.15.161:0/2973801027 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79c4156130 0x7f79c40728b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:38.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.494+0000 7f79c9fab640 1 -- 172.21.15.161:0/2973801027 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79b0004800 msgr2=0x7f79c4155bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:38.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.494+0000 7f79c9fab640 1 --2- 172.21.15.161:0/2973801027 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79b0004800 0x7f79c4155bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:38.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.494+0000 7f79c9fab640 1 -- 172.21.15.161:0/2973801027 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79c41ecec0 con 0x7f79c4149030 2024-03-28T23:31:38.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.494+0000 7f79c9fab640 1 --2- 172.21.15.161:0/2973801027 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79c4149030 0x7f79c41e9180 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f79c4070250 tx=0x7f79bc00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:38.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.495+0000 7f79b9ffb640 1 -- 172.21.15.161:0/2973801027 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79bc00fde0 con 0x7f79c4149030 2024-03-28T23:31:38.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.495+0000 7f79b9ffb640 1 -- 172.21.15.161:0/2973801027 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79bc015430 con 0x7f79c4149030 2024-03-28T23:31:38.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.495+0000 7f79cba0e640 1 -- 172.21.15.161:0/2973801027 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79c41537b0 con 0x7f79c4149030 2024-03-28T23:31:38.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.495+0000 7f79b9ffb640 1 -- 172.21.15.161:0/2973801027 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79bc015820 con 0x7f79c4149030 2024-03-28T23:31:38.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.495+0000 7f79cba0e640 1 -- 172.21.15.161:0/2973801027 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79c4073390 con 0x7f79c4149030 2024-03-28T23:31:38.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.496+0000 7f79cba0e640 1 -- 172.21.15.161:0/2973801027 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79c40c2150 con 0x7f79c4149030 2024-03-28T23:31:38.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.496+0000 7f79b9ffb640 1 -- 172.21.15.161:0/2973801027 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f79bc0159c0 con 0x7f79c4149030 2024-03-28T23:31:38.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.497+0000 7f79b9ffb640 1 --2- 172.21.15.161:0/2973801027 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f79a806b4d0 0x7f79a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:38.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.497+0000 7f79b9ffb640 1 -- 172.21.15.161:0/2973801027 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5619+0+0 (secure 0 0 0) 0x7f79bc08aed0 con 0x7f79c4149030 2024-03-28T23:31:38.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.497+0000 7f79c97aa640 1 --2- 172.21.15.161:0/2973801027 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f79a806b4d0 0x7f79a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:38.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.497+0000 7f79c97aa640 1 --2- 172.21.15.161:0/2973801027 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f79a806b4d0 0x7f79a806d990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f79b4001760 tx=0x7f79b4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:38.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.501+0000 7f79b9ffb640 1 -- 172.21.15.161:0/2973801027 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f79c40c2150 con 0x7f79c4149030 2024-03-28T23:31:38.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:38.674+0000 7f79cba0e640 1 -- 172.21.15.161:0/2973801027 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f79c404f4b0 con 0x7f79c4149030 2024-03-28T23:31:39.032 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-03-28T23:31:39.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.370+0000 7f79b9ffb640 1 -- 172.21.15.161:0/2973801027 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v49) ==== 170+0+0 (secure 0 0 0) 0x7f79bc019030 con 0x7f79c4149030 2024-03-28T23:31:39.371 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data' created 2024-03-28T23:31:39.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.374+0000 7f79cba0e640 1 -- 172.21.15.161:0/2973801027 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f79a806b4d0 msgr2=0x7f79a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:39.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.374+0000 7f79cba0e640 1 --2- 172.21.15.161:0/2973801027 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f79a806b4d0 0x7f79a806d990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f79b4001760 tx=0x7f79b4009040 comp rx=0 tx=0).stop 2024-03-28T23:31:39.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.374+0000 7f79cba0e640 1 -- 172.21.15.161:0/2973801027 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79c4149030 msgr2=0x7f79c41e9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:39.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.374+0000 7f79cba0e640 1 --2- 172.21.15.161:0/2973801027 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79c4149030 0x7f79c41e9180 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f79c4070250 tx=0x7f79bc00f580 comp rx=0 tx=0).stop 2024-03-28T23:31:39.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.374+0000 7f79cba0e640 1 -- 172.21.15.161:0/2973801027 shutdown_connections 2024-03-28T23:31:39.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.374+0000 7f79cba0e640 1 --2- 172.21.15.161:0/2973801027 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f79a806b4d0 0x7f79a806d990 secure :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f79b4001760 tx=0x7f79b4009040 comp rx=0 tx=0).stop 2024-03-28T23:31:39.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.374+0000 7f79cba0e640 1 --2- 172.21.15.161:0/2973801027 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79c4149030 0x7f79c41e9180 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:39.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.374+0000 7f79cba0e640 1 --2- 172.21.15.161:0/2973801027 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79c4156130 0x7f79c40728b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:39.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.374+0000 7f79cba0e640 1 --2- 172.21.15.161:0/2973801027 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79b0004800 0x7f79c4155bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:39.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.374+0000 7f79cba0e640 1 -- 172.21.15.161:0/2973801027 >> 172.21.15.161:0/2973801027 conn(0x7f79c40a83f0 msgr2=0x7f79c40c6590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:39.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.375+0000 7f79cba0e640 1 -- 172.21.15.161:0/2973801027 shutdown_connections 2024-03-28T23:31:39.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.375+0000 7f79cba0e640 1 -- 172.21.15.161:0/2973801027 wait complete. 2024-03-28T23:31:39.400 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-28T23:31:39.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.504+0000 7f6474809640 1 Processor -- start 2024-03-28T23:31:39.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.504+0000 7f6474809640 1 -- start start 2024-03-28T23:31:39.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.505+0000 7f6474809640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6470152bc0 0x7f64700c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:39.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.505+0000 7f6474809640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64700c6f60 con 0x7f64701589b0 2024-03-28T23:31:39.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.505+0000 7f6474809640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64700c6120 con 0x7f64700c8a80 2024-03-28T23:31:39.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.505+0000 7f6474809640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f64700c6400 con 0x7f6470152bc0 2024-03-28T23:31:39.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.505+0000 7f646ed9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f64701589b0 0x7f6470158e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50604/0 (socket says 172.21.15.161:50604) 2024-03-28T23:31:39.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.505+0000 7f646ed9d640 1 -- 172.21.15.161:0/2526336785 learned_addr learned my addr 172.21.15.161:0/2526336785 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:39.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646e59c640 1 --2- 172.21.15.161:0/2526336785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6470152bc0 0x7f64700c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:39.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646d59a640 1 -- 172.21.15.161:0/2526336785 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3178964999 0 0) 0x7f64700c6f60 con 0x7f64701589b0 2024-03-28T23:31:39.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646d59a640 1 -- 172.21.15.161:0/2526336785 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6458003520 con 0x7f64701589b0 2024-03-28T23:31:39.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646d59a640 1 -- 172.21.15.161:0/2526336785 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2900064633 0 0) 0x7f6458003520 con 0x7f64701589b0 2024-03-28T23:31:39.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646d59a640 1 -- 172.21.15.161:0/2526336785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6470152bc0 msgr2=0x7f64700c0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:39.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646d59a640 1 --2- 172.21.15.161:0/2526336785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6470152bc0 0x7f64700c0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:39.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646d59a640 1 -- 172.21.15.161:0/2526336785 >> v1:172.21.15.170:6789/0 conn(0x7f64700c8a80 legacy=0x7f64700c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:39.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646d59a640 1 -- 172.21.15.161:0/2526336785 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64700c7240 con 0x7f64701589b0 2024-03-28T23:31:39.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646d59a640 1 -- 172.21.15.161:0/2526336785 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f6468001670 con 0x7f64701589b0 2024-03-28T23:31:39.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646d59a640 1 -- 172.21.15.161:0/2526336785 >> v1:172.21.15.161:6789/0 conn(0x7f64701589b0 legacy=0x7f6470158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:39.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646d59a640 1 --2- 172.21.15.161:0/2526336785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6458003ac0 0x7f6458003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:39.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646d59a640 1 --2- 172.21.15.161:0/2526336785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6458004800 0x7f6458004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:39.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646d59a640 1 --2- 172.21.15.161:0/2526336785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6470152bc0 0x7f6458005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:39.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646d59a640 1 -- 172.21.15.161:0/2526336785 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f64700c6400 con 0x7f6458003ac0 2024-03-28T23:31:39.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646d59a640 1 -- 172.21.15.161:0/2526336785 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f645800bb50 con 0x7f6458004800 2024-03-28T23:31:39.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646d59a640 1 -- 172.21.15.161:0/2526336785 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f645800bd20 con 0x7f6470152bc0 2024-03-28T23:31:39.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646e59c640 1 --2- 172.21.15.161:0/2526336785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6458003ac0 0x7f6458003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:39.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646ed9d640 1 --2- 172.21.15.161:0/2526336785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6470152bc0 0x7f6458005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:39.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646dd9b640 1 --2- 172.21.15.161:0/2526336785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6458004800 0x7f6458004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:39.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646e59c640 1 -- 172.21.15.161:0/2526336785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6470152bc0 msgr2=0x7f6458005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:39.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646e59c640 1 --2- 172.21.15.161:0/2526336785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6470152bc0 0x7f6458005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:39.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646e59c640 1 -- 172.21.15.161:0/2526336785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6458004800 msgr2=0x7f6458004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:39.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646e59c640 1 --2- 172.21.15.161:0/2526336785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6458004800 0x7f6458004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:39.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646e59c640 1 -- 172.21.15.161:0/2526336785 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6458003520 con 0x7f6458003ac0 2024-03-28T23:31:39.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.506+0000 7f646dd9b640 1 --2- 172.21.15.161:0/2526336785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6458004800 0x7f6458004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:39.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.507+0000 7f646e59c640 1 --2- 172.21.15.161:0/2526336785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6458003ac0 0x7f6458003ee0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f6460009fc0 tx=0x7f6458005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3413e68dbcbc07c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:39.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.507+0000 7f646d59a640 1 -- 172.21.15.161:0/2526336785 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6460039070 con 0x7f6458003ac0 2024-03-28T23:31:39.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.507+0000 7f646d59a640 1 -- 172.21.15.161:0/2526336785 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f646002cc20 con 0x7f6458003ac0 2024-03-28T23:31:39.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.507+0000 7f646d59a640 1 -- 172.21.15.161:0/2526336785 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6460009c10 con 0x7f6458003ac0 2024-03-28T23:31:39.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.508+0000 7f6474809640 1 -- 172.21.15.161:0/2526336785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6458003ac0 msgr2=0x7f6458003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:39.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.508+0000 7f6474809640 1 --2- 172.21.15.161:0/2526336785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6458003ac0 0x7f6458003ee0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f6460009fc0 tx=0x7f6458005830 comp rx=0 tx=0).stop 2024-03-28T23:31:39.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.508+0000 7f646ed9d640 1 -- 172.21.15.161:0/2526336785 reap_dead start 2024-03-28T23:31:39.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.508+0000 7f6474809640 1 -- 172.21.15.161:0/2526336785 shutdown_connections 2024-03-28T23:31:39.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.508+0000 7f6474809640 1 -- 172.21.15.161:0/2526336785 >> 172.21.15.161:0/2526336785 conn(0x7f64700caa80 msgr2=0x7f64700cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:39.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.508+0000 7f6474809640 1 -- 172.21.15.161:0/2526336785 shutdown_connections 2024-03-28T23:31:39.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.508+0000 7f6474809640 1 -- 172.21.15.161:0/2526336785 wait complete. 2024-03-28T23:31:39.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.509+0000 7f6474809640 1 Processor -- start 2024-03-28T23:31:39.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.509+0000 7f6474809640 1 -- start start 2024-03-28T23:31:39.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.510+0000 7f6474809640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6458003ac0 0x7f647014a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:39.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.510+0000 7f6474809640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6470150d80 0x7f6470154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:39.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.510+0000 7f6474809640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6470154ff0 0x7f64701ede10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:39.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.510+0000 7f6474809640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6470152fa0 con 0x7f6470150d80 2024-03-28T23:31:39.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.510+0000 7f6474809640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f64700c95f0 con 0x7f6470154ff0 2024-03-28T23:31:39.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.510+0000 7f6474809640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6470154640 con 0x7f6458003ac0 2024-03-28T23:31:39.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.510+0000 7f646dd9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6470150d80 0x7f6470154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:39.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.510+0000 7f646dd9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6470150d80 0x7f6470154ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59396/0 (socket says 172.21.15.161:59396) 2024-03-28T23:31:39.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.510+0000 7f646dd9b640 1 -- 172.21.15.161:0/3546906055 learned_addr learned my addr 172.21.15.161:0/3546906055 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:39.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.510+0000 7f646e59c640 1 --2- 172.21.15.161:0/3546906055 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6458003ac0 0x7f647014a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:39.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.510+0000 7f646ed9d640 1 --2- 172.21.15.161:0/3546906055 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6470154ff0 0x7f64701ede10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:39.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.510+0000 7f646dd9b640 1 -- 172.21.15.161:0/3546906055 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6458003ac0 msgr2=0x7f647014a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:39.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.510+0000 7f646dd9b640 1 --2- 172.21.15.161:0/3546906055 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6458003ac0 0x7f647014a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:39.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.510+0000 7f646dd9b640 1 -- 172.21.15.161:0/3546906055 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6470154ff0 msgr2=0x7f64701ede10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:39.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.510+0000 7f646dd9b640 1 --2- 172.21.15.161:0/3546906055 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6470154ff0 0x7f64701ede10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:39.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.510+0000 7f646dd9b640 1 -- 172.21.15.161:0/3546906055 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6470151730 con 0x7f6470150d80 2024-03-28T23:31:39.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.510+0000 7f646dd9b640 1 --2- 172.21.15.161:0/3546906055 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6470150d80 0x7f6470154ab0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f646400e4d0 tx=0x7f646400e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:39.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.510+0000 7f645effd640 1 -- 172.21.15.161:0/3546906055 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64640052e0 con 0x7f6470150d80 2024-03-28T23:31:39.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.510+0000 7f645effd640 1 -- 172.21.15.161:0/3546906055 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6464018070 con 0x7f6470150d80 2024-03-28T23:31:39.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.510+0000 7f6474809640 1 -- 172.21.15.161:0/3546906055 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6470151240 con 0x7f6470150d80 2024-03-28T23:31:39.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.510+0000 7f645effd640 1 -- 172.21.15.161:0/3546906055 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6464003810 con 0x7f6470150d80 2024-03-28T23:31:39.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.511+0000 7f6474809640 1 -- 172.21.15.161:0/3546906055 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6470151cd0 con 0x7f6470150d80 2024-03-28T23:31:39.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.511+0000 7f6474809640 1 -- 172.21.15.161:0/3546906055 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6470068f90 con 0x7f6470150d80 2024-03-28T23:31:39.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.512+0000 7f645effd640 1 -- 172.21.15.161:0/3546906055 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f64640039b0 con 0x7f6470150d80 2024-03-28T23:31:39.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.512+0000 7f645effd640 1 --2- 172.21.15.161:0/3546906055 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f645006b500 0x7f645006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:39.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.512+0000 7f645effd640 1 -- 172.21.15.161:0/3546906055 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 6001+0+0 (secure 0 0 0) 0x7f6464089e80 con 0x7f6470150d80 2024-03-28T23:31:39.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.512+0000 7f646e59c640 1 --2- 172.21.15.161:0/3546906055 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f645006b500 0x7f645006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:39.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.513+0000 7f646e59c640 1 --2- 172.21.15.161:0/3546906055 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f645006b500 0x7f645006d9c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6460036000 tx=0x7f646002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:39.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.517+0000 7f645effd640 1 -- 172.21.15.161:0/3546906055 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6470068f90 con 0x7f6470150d80 2024-03-28T23:31:39.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:39.724+0000 7f6474809640 1 -- 172.21.15.161:0/3546906055 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f647004f4b0 con 0x7f6470150d80 2024-03-28T23:31:40.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.371+0000 7f645effd640 1 -- 172.21.15.161:0/3546906055 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v28) ==== 172+0+0 (secure 0 0 0) 0x7f64640584e0 con 0x7f6470150d80 2024-03-28T23:31:40.372 INFO:teuthology.orchestra.run.smithi161.stderr:new fs with metadata pool 7 and data pool 8 2024-03-28T23:31:40.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.377+0000 7f6474809640 1 -- 172.21.15.161:0/3546906055 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f645006b500 msgr2=0x7f645006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:40.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.377+0000 7f6474809640 1 --2- 172.21.15.161:0/3546906055 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f645006b500 0x7f645006d9c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6460036000 tx=0x7f646002e000 comp rx=0 tx=0).stop 2024-03-28T23:31:40.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.377+0000 7f6474809640 1 -- 172.21.15.161:0/3546906055 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6470150d80 msgr2=0x7f6470154ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:40.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.377+0000 7f6474809640 1 --2- 172.21.15.161:0/3546906055 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6470150d80 0x7f6470154ab0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f646400e4d0 tx=0x7f646400e990 comp rx=0 tx=0).stop 2024-03-28T23:31:40.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.377+0000 7f6474809640 1 -- 172.21.15.161:0/3546906055 shutdown_connections 2024-03-28T23:31:40.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.377+0000 7f6474809640 1 --2- 172.21.15.161:0/3546906055 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f645006b500 0x7f645006d9c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:40.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.377+0000 7f6474809640 1 --2- 172.21.15.161:0/3546906055 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6470150d80 0x7f6470154ab0 secure :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f646400e4d0 tx=0x7f646400e990 comp rx=0 tx=0).stop 2024-03-28T23:31:40.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.377+0000 7f6474809640 1 --2- 172.21.15.161:0/3546906055 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6470154ff0 0x7f64701ede10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:40.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.377+0000 7f6474809640 1 --2- 172.21.15.161:0/3546906055 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6458003ac0 0x7f647014a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:40.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.377+0000 7f6474809640 1 -- 172.21.15.161:0/3546906055 >> 172.21.15.161:0/3546906055 conn(0x7f64700caa80 msgr2=0x7f647014e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:40.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.385+0000 7f6474809640 1 -- 172.21.15.161:0/3546906055 shutdown_connections 2024-03-28T23:31:40.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.390+0000 7f6474809640 1 -- 172.21.15.161:0/3546906055 wait complete. 2024-03-28T23:31:40.412 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:31:40.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.513+0000 7fc59a769640 1 Processor -- start 2024-03-28T23:31:40.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.514+0000 7fc59a769640 1 -- start start 2024-03-28T23:31:40.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.514+0000 7fc59a769640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc594152890 0x7fc5940c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:40.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.514+0000 7fc59a769640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc5940c8ed0 0x7fc5940c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:40.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.514+0000 7fc59a769640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc594158a30 con 0x7fc5940c5950 2024-03-28T23:31:40.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.514+0000 7fc59a769640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc5940c5db0 con 0x7fc5940c8ed0 2024-03-28T23:31:40.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.514+0000 7fc59a769640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc5940c8480 con 0x7fc594152890 2024-03-28T23:31:40.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.514+0000 7fc5937fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc5940c8ed0 0x7fc5940c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:40.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.514+0000 7fc598d06640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc5940c5950 0x7fc5941553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:43934/0 (socket says 172.21.15.161:43934) 2024-03-28T23:31:40.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.514+0000 7fc598d06640 1 -- 172.21.15.161:0/1042278326 learned_addr learned my addr 172.21.15.161:0/1042278326 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:40.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.515+0000 7fc5937fe640 1 -- 172.21.15.161:0/1042278326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc594152890 msgr2=0x7fc5940c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:40.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.515+0000 7fc593fff640 1 --2- 172.21.15.161:0/1042278326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc594152890 0x7fc5940c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:40.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.515+0000 7fc5937fe640 1 --2- 172.21.15.161:0/1042278326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc594152890 0x7fc5940c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:40.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.515+0000 7fc592ffd640 1 -- 172.21.15.161:0/1042278326 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3482814745 0 0) 0x7fc594158a30 con 0x7fc5940c5950 2024-03-28T23:31:40.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.515+0000 7fc5937fe640 1 -- 172.21.15.161:0/1042278326 >> v1:172.21.15.161:6789/0 conn(0x7fc5940c5950 legacy=0x7fc5941553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:40.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.515+0000 7fc5937fe640 1 -- 172.21.15.161:0/1042278326 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc594158d10 con 0x7fc5940c8ed0 2024-03-28T23:31:40.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.515+0000 7fc593fff640 1 --2- 172.21.15.161:0/1042278326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc594152890 0x7fc5940c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:40.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.515+0000 7fc5937fe640 1 --2- 172.21.15.161:0/1042278326 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5940c8ed0 0x7fc5940c5410 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc584009790 tx=0x7fc58402b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ec92cc8f8128064 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:40.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.516+0000 7fc592ffd640 1 -- 172.21.15.161:0/1042278326 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc584039070 con 0x7fc5940c8ed0 2024-03-28T23:31:40.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.516+0000 7fc592ffd640 1 -- 172.21.15.161:0/1042278326 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5840316f0 con 0x7fc5940c8ed0 2024-03-28T23:31:40.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.516+0000 7fc592ffd640 1 -- 172.21.15.161:0/1042278326 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc584031ac0 con 0x7fc5940c8ed0 2024-03-28T23:31:40.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.516+0000 7fc59a769640 1 -- 172.21.15.161:0/1042278326 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5940c8ed0 msgr2=0x7fc5940c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:40.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.516+0000 7fc59a769640 1 --2- 172.21.15.161:0/1042278326 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5940c8ed0 0x7fc5940c5410 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc584009790 tx=0x7fc58402b790 comp rx=0 tx=0).stop 2024-03-28T23:31:40.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.516+0000 7fc59a769640 1 -- 172.21.15.161:0/1042278326 shutdown_connections 2024-03-28T23:31:40.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.516+0000 7fc59a769640 1 --2- 172.21.15.161:0/1042278326 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5940c8ed0 0x7fc5940c5410 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:40.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.516+0000 7fc59a769640 1 --2- 172.21.15.161:0/1042278326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc594152890 0x7fc5940c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:40.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.516+0000 7fc59a769640 1 -- 172.21.15.161:0/1042278326 >> 172.21.15.161:0/1042278326 conn(0x7fc5940ac620 msgr2=0x7fc5940aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:40.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.516+0000 7fc59a769640 1 -- 172.21.15.161:0/1042278326 shutdown_connections 2024-03-28T23:31:40.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.516+0000 7fc59a769640 1 -- 172.21.15.161:0/1042278326 wait complete. 2024-03-28T23:31:40.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.517+0000 7fc59a769640 1 Processor -- start 2024-03-28T23:31:40.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.517+0000 7fc59a769640 1 -- start start 2024-03-28T23:31:40.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.518+0000 7fc59a769640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5940c5950 0x7fc59414cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:40.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.518+0000 7fc59a769640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5940c8ed0 0x7fc5941ee5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:40.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.518+0000 7fc59a769640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc594152890 0x7fc594149b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:40.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.518+0000 7fc59a769640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc5941592d0 con 0x7fc5940c8ed0 2024-03-28T23:31:40.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.518+0000 7fc59a769640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc594159640 con 0x7fc5940c5950 2024-03-28T23:31:40.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.518+0000 7fc59a769640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc594152c70 con 0x7fc594152890 2024-03-28T23:31:40.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.518+0000 7fc5937fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5940c8ed0 0x7fc5941ee5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:40.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.518+0000 7fc5937fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5940c8ed0 0x7fc5941ee5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58292/0 (socket says 172.21.15.161:58292) 2024-03-28T23:31:40.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.518+0000 7fc5937fe640 1 -- 172.21.15.161:0/3778064974 learned_addr learned my addr 172.21.15.161:0/3778064974 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:40.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.518+0000 7fc598d06640 1 --2- 172.21.15.161:0/3778064974 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc594152890 0x7fc594149b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:40.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.518+0000 7fc593fff640 1 --2- 172.21.15.161:0/3778064974 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5940c5950 0x7fc59414cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:40.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.518+0000 7fc5937fe640 1 -- 172.21.15.161:0/3778064974 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc594152890 msgr2=0x7fc594149b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:40.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.518+0000 7fc5937fe640 1 --2- 172.21.15.161:0/3778064974 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc594152890 0x7fc594149b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:40.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.518+0000 7fc5937fe640 1 -- 172.21.15.161:0/3778064974 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5940c5950 msgr2=0x7fc59414cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:40.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.518+0000 7fc5937fe640 1 --2- 172.21.15.161:0/3778064974 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5940c5950 0x7fc59414cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:40.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.518+0000 7fc5937fe640 1 -- 172.21.15.161:0/3778064974 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc594153980 con 0x7fc5940c8ed0 2024-03-28T23:31:40.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.518+0000 7fc598d06640 1 --2- 172.21.15.161:0/3778064974 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc594152890 0x7fc594149b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:40.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.519+0000 7fc593fff640 1 --2- 172.21.15.161:0/3778064974 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5940c5950 0x7fc59414cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:40.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.519+0000 7fc5937fe640 1 --2- 172.21.15.161:0/3778064974 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5940c8ed0 0x7fc5941ee5b0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fc584000c00 tx=0x7fc5840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:40.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.519+0000 7fc590ff9640 1 -- 172.21.15.161:0/3778064974 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc584043070 con 0x7fc5940c8ed0 2024-03-28T23:31:40.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.519+0000 7fc590ff9640 1 -- 172.21.15.161:0/3778064974 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc584005bb0 con 0x7fc5940c8ed0 2024-03-28T23:31:40.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.519+0000 7fc59a769640 1 -- 172.21.15.161:0/3778064974 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc594153c60 con 0x7fc5940c8ed0 2024-03-28T23:31:40.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.519+0000 7fc590ff9640 1 -- 172.21.15.161:0/3778064974 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc584039040 con 0x7fc5940c8ed0 2024-03-28T23:31:40.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.519+0000 7fc59a769640 1 -- 172.21.15.161:0/3778064974 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc59414a0b0 con 0x7fc5940c8ed0 2024-03-28T23:31:40.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.520+0000 7fc59a769640 1 -- 172.21.15.161:0/3778064974 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc59404f4b0 con 0x7fc5940c8ed0 2024-03-28T23:31:40.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.520+0000 7fc590ff9640 1 -- 172.21.15.161:0/3778064974 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc58403c070 con 0x7fc5940c8ed0 2024-03-28T23:31:40.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.521+0000 7fc590ff9640 1 --2- 172.21.15.161:0/3778064974 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc58006b500 0x7fc58006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:40.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.521+0000 7fc590ff9640 1 -- 172.21.15.161:0/3778064974 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6101+0+0 (secure 0 0 0) 0x7fc5840abc90 con 0x7fc5940c8ed0 2024-03-28T23:31:40.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.521+0000 7fc593fff640 1 --2- 172.21.15.161:0/3778064974 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc58006b500 0x7fc58006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:40.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.521+0000 7fc593fff640 1 --2- 172.21.15.161:0/3778064974 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc58006b500 0x7fc58006d9c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc588006fd0 tx=0x7fc588008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:40.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.525+0000 7fc590ff9640 1 -- 172.21.15.161:0/3778064974 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc58407a980 con 0x7fc5940c8ed0 2024-03-28T23:31:40.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.697+0000 7fc59a769640 1 -- 172.21.15.161:0/3778064974 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc59414b4c0 con 0x7fc5940c8ed0 2024-03-28T23:31:40.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.699+0000 7fc590ff9640 1 -- 172.21.15.161:0/3778064974 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+16991 (secure 0 0 0) 0x7fc58407a280 con 0x7fc5940c8ed0 2024-03-28T23:31:40.699 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:40.699 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":50,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:31:39.725310+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:31:37.940838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-28T23:31:38.692659+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:31:40.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.701+0000 7fc59a769640 1 -- 172.21.15.161:0/3778064974 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc58006b500 msgr2=0x7fc58006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:40.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.701+0000 7fc59a769640 1 --2- 172.21.15.161:0/3778064974 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc58006b500 0x7fc58006d9c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc588006fd0 tx=0x7fc588008040 comp rx=0 tx=0).stop 2024-03-28T23:31:40.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.701+0000 7fc59a769640 1 -- 172.21.15.161:0/3778064974 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5940c8ed0 msgr2=0x7fc5941ee5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:40.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.701+0000 7fc59a769640 1 --2- 172.21.15.161:0/3778064974 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5940c8ed0 0x7fc5941ee5b0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fc584000c00 tx=0x7fc5840046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:40.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.701+0000 7fc59a769640 1 -- 172.21.15.161:0/3778064974 shutdown_connections 2024-03-28T23:31:40.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.701+0000 7fc59a769640 1 --2- 172.21.15.161:0/3778064974 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc58006b500 0x7fc58006d9c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:40.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.701+0000 7fc59a769640 1 --2- 172.21.15.161:0/3778064974 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5940c8ed0 0x7fc5941ee5b0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:40.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.701+0000 7fc59a769640 1 --2- 172.21.15.161:0/3778064974 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc594152890 0x7fc594149b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:40.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.701+0000 7fc59a769640 1 --2- 172.21.15.161:0/3778064974 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5940c5950 0x7fc59414cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:40.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.701+0000 7fc59a769640 1 -- 172.21.15.161:0/3778064974 >> 172.21.15.161:0/3778064974 conn(0x7fc5940ac620 msgr2=0x7fc594151a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:40.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.701+0000 7fc59a769640 1 -- 172.21.15.161:0/3778064974 shutdown_connections 2024-03-28T23:31:40.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.701+0000 7fc59a769640 1 -- 172.21.15.161:0/3778064974 wait complete. 2024-03-28T23:31:40.713 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:31:40.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.814+0000 7fc1606ae640 1 Processor -- start 2024-03-28T23:31:40.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.814+0000 7fc1606ae640 1 -- start start 2024-03-28T23:31:40.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.815+0000 7fc1606ae640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc15814be70 0x7fc1580ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:40.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.815+0000 7fc1606ae640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc1580aacd0 0x7fc15815c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:40.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.815+0000 7fc1606ae640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc15814c250 con 0x7fc1580ab6e0 2024-03-28T23:31:40.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.815+0000 7fc1606ae640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc1581529c0 con 0x7fc1580aacd0 2024-03-28T23:31:40.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.815+0000 7fc1606ae640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc1580a9dc0 con 0x7fc15814be70 2024-03-28T23:31:40.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.815+0000 7fc15ec4b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc1580aacd0 0x7fc15815c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:40.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.815+0000 7fc15ec4b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc1580aacd0 0x7fc15815c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58308/0 (socket says 172.21.15.161:58308) 2024-03-28T23:31:40.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.815+0000 7fc15ec4b640 1 -- 172.21.15.161:0/4173791182 learned_addr learned my addr 172.21.15.161:0/4173791182 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:40.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.815+0000 7fc15e44a640 1 --2- 172.21.15.161:0/4173791182 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc15814be70 0x7fc1580ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:40.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.815+0000 7fc15ec4b640 1 -- 172.21.15.161:0/4173791182 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc15814be70 msgr2=0x7fc1580ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:40.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.815+0000 7fc15ec4b640 1 --2- 172.21.15.161:0/4173791182 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc15814be70 0x7fc1580ab1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:40.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.815+0000 7fc15ec4b640 1 -- 172.21.15.161:0/4173791182 >> v1:172.21.15.170:6789/0 conn(0x7fc1580ab6e0 legacy=0x7fc1580aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:40.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.815+0000 7fc15ec4b640 1 -- 172.21.15.161:0/4173791182 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1580aa0a0 con 0x7fc1580aacd0 2024-03-28T23:31:40.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.816+0000 7fc15ec4b640 1 --2- 172.21.15.161:0/4173791182 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1580aacd0 0x7fc15815c740 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fc150009e50 tx=0x7fc15002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61f63183cc82473d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:40.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.816+0000 7fc15d448640 1 -- 172.21.15.161:0/4173791182 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc15002ca70 con 0x7fc1580aacd0 2024-03-28T23:31:40.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.816+0000 7fc15d448640 1 -- 172.21.15.161:0/4173791182 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc15002cc10 con 0x7fc1580aacd0 2024-03-28T23:31:40.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.816+0000 7fc15d448640 1 -- 172.21.15.161:0/4173791182 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1500343f0 con 0x7fc1580aacd0 2024-03-28T23:31:40.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.817+0000 7fc1606ae640 1 -- 172.21.15.161:0/4173791182 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1580aacd0 msgr2=0x7fc15815c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:40.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.817+0000 7fc1606ae640 1 --2- 172.21.15.161:0/4173791182 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1580aacd0 0x7fc15815c740 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fc150009e50 tx=0x7fc15002bed0 comp rx=0 tx=0).stop 2024-03-28T23:31:40.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.817+0000 7fc1606ae640 1 -- 172.21.15.161:0/4173791182 shutdown_connections 2024-03-28T23:31:40.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.817+0000 7fc1606ae640 1 --2- 172.21.15.161:0/4173791182 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1580aacd0 0x7fc15815c740 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:40.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.817+0000 7fc1606ae640 1 --2- 172.21.15.161:0/4173791182 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc15814be70 0x7fc1580ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:40.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.817+0000 7fc1606ae640 1 -- 172.21.15.161:0/4173791182 >> 172.21.15.161:0/4173791182 conn(0x7fc1580c6670 msgr2=0x7fc1580c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:40.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.817+0000 7fc1606ae640 1 -- 172.21.15.161:0/4173791182 shutdown_connections 2024-03-28T23:31:40.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.817+0000 7fc1606ae640 1 -- 172.21.15.161:0/4173791182 wait complete. 2024-03-28T23:31:40.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.818+0000 7fc1606ae640 1 Processor -- start 2024-03-28T23:31:40.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.818+0000 7fc1606ae640 1 -- start start 2024-03-28T23:31:40.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.818+0000 7fc1606ae640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1580aacd0 0x7fc1581ddb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:40.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.818+0000 7fc1606ae640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1580ab6e0 0x7fc1581dca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:40.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.818+0000 7fc1606ae640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc15814be70 0x7fc1581dcf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:40.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.818+0000 7fc1606ae640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc15806b2d0 con 0x7fc1580ab6e0 2024-03-28T23:31:40.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.818+0000 7fc1606ae640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc15806b150 con 0x7fc1580aacd0 2024-03-28T23:31:40.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.818+0000 7fc1606ae640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc1580c1520 con 0x7fc15814be70 2024-03-28T23:31:40.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.818+0000 7fc15dc49640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1580ab6e0 0x7fc1581dca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:40.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.819+0000 7fc15dc49640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1580ab6e0 0x7fc1581dca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58316/0 (socket says 172.21.15.161:58316) 2024-03-28T23:31:40.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.819+0000 7fc15dc49640 1 -- 172.21.15.161:0/1632545878 learned_addr learned my addr 172.21.15.161:0/1632545878 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:40.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.819+0000 7fc15e44a640 1 --2- 172.21.15.161:0/1632545878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1580aacd0 0x7fc1581ddb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:40.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.819+0000 7fc15ec4b640 1 --2- 172.21.15.161:0/1632545878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc15814be70 0x7fc1581dcf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:40.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.819+0000 7fc15dc49640 1 -- 172.21.15.161:0/1632545878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc15814be70 msgr2=0x7fc1581dcf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:40.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.819+0000 7fc15dc49640 1 --2- 172.21.15.161:0/1632545878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc15814be70 0x7fc1581dcf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:40.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.819+0000 7fc15dc49640 1 -- 172.21.15.161:0/1632545878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1580aacd0 msgr2=0x7fc1581ddb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:40.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.819+0000 7fc15dc49640 1 --2- 172.21.15.161:0/1632545878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1580aacd0 0x7fc1581ddb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:40.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.819+0000 7fc15dc49640 1 -- 172.21.15.161:0/1632545878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1581e9860 con 0x7fc1580ab6e0 2024-03-28T23:31:40.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.819+0000 7fc15dc49640 1 --2- 172.21.15.161:0/1632545878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1580ab6e0 0x7fc1581dca20 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fc14800c370 tx=0x7fc14800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:40.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.819+0000 7fc14effd640 1 -- 172.21.15.161:0/1632545878 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc148017070 con 0x7fc1580ab6e0 2024-03-28T23:31:40.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.819+0000 7fc14effd640 1 -- 172.21.15.161:0/1632545878 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1480052d0 con 0x7fc1580ab6e0 2024-03-28T23:31:40.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.819+0000 7fc1606ae640 1 -- 172.21.15.161:0/1632545878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1581dd4a0 con 0x7fc1580ab6e0 2024-03-28T23:31:40.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.819+0000 7fc14effd640 1 -- 172.21.15.161:0/1632545878 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc148003850 con 0x7fc1580ab6e0 2024-03-28T23:31:40.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.819+0000 7fc1606ae640 1 -- 172.21.15.161:0/1632545878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1581e9b00 con 0x7fc1580ab6e0 2024-03-28T23:31:40.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.820+0000 7fc1606ae640 1 -- 172.21.15.161:0/1632545878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc15804f4b0 con 0x7fc1580ab6e0 2024-03-28T23:31:40.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.821+0000 7fc14effd640 1 -- 172.21.15.161:0/1632545878 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc14801a070 con 0x7fc1580ab6e0 2024-03-28T23:31:40.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.821+0000 7fc14effd640 1 --2- 172.21.15.161:0/1632545878 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc13c06b4d0 0x7fc13c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:40.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.821+0000 7fc14effd640 1 -- 172.21.15.161:0/1632545878 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6101+0+0 (secure 0 0 0) 0x7fc148089570 con 0x7fc1580ab6e0 2024-03-28T23:31:40.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.821+0000 7fc15e44a640 1 --2- 172.21.15.161:0/1632545878 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc13c06b4d0 0x7fc13c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:40.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.821+0000 7fc15e44a640 1 --2- 172.21.15.161:0/1632545878 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc13c06b4d0 0x7fc13c06d990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc154009aa0 tx=0x7fc154009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:40.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.825+0000 7fc14effd640 1 -- 172.21.15.161:0/1632545878 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc148057ca0 con 0x7fc1580ab6e0 2024-03-28T23:31:40.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.998+0000 7fc1606ae640 1 -- 172.21.15.161:0/1632545878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc1580c0ff0 con 0x7fc1580ab6e0 2024-03-28T23:31:41.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:40.999+0000 7fc14effd640 1 -- 172.21.15.161:0/1632545878 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+16991 (secure 0 0 0) 0x7fc14805bd40 con 0x7fc1580ab6e0 2024-03-28T23:31:41.000 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:41.000 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":50,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:31:39.725310+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:31:37.940838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-28T23:31:38.692659+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:31:41.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.001+0000 7fc1606ae640 1 -- 172.21.15.161:0/1632545878 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc13c06b4d0 msgr2=0x7fc13c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:41.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.001+0000 7fc1606ae640 1 --2- 172.21.15.161:0/1632545878 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc13c06b4d0 0x7fc13c06d990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc154009aa0 tx=0x7fc154009190 comp rx=0 tx=0).stop 2024-03-28T23:31:41.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.001+0000 7fc1606ae640 1 -- 172.21.15.161:0/1632545878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1580ab6e0 msgr2=0x7fc1581dca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:41.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.001+0000 7fc1606ae640 1 --2- 172.21.15.161:0/1632545878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1580ab6e0 0x7fc1581dca20 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fc14800c370 tx=0x7fc14800c830 comp rx=0 tx=0).stop 2024-03-28T23:31:41.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.001+0000 7fc1606ae640 1 -- 172.21.15.161:0/1632545878 shutdown_connections 2024-03-28T23:31:41.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.001+0000 7fc1606ae640 1 --2- 172.21.15.161:0/1632545878 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc13c06b4d0 0x7fc13c06d990 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:41.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.001+0000 7fc1606ae640 1 --2- 172.21.15.161:0/1632545878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1580ab6e0 0x7fc1581dca20 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:41.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.001+0000 7fc1606ae640 1 --2- 172.21.15.161:0/1632545878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc15814be70 0x7fc1581dcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:41.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.001+0000 7fc1606ae640 1 --2- 172.21.15.161:0/1632545878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1580aacd0 0x7fc1581ddb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:41.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.001+0000 7fc1606ae640 1 -- 172.21.15.161:0/1632545878 >> 172.21.15.161:0/1632545878 conn(0x7fc1580c6670 msgr2=0x7fc1581551a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:41.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.001+0000 7fc1606ae640 1 -- 172.21.15.161:0/1632545878 shutdown_connections 2024-03-28T23:31:41.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.001+0000 7fc1606ae640 1 -- 172.21.15.161:0/1632545878 wait complete. 2024-03-28T23:31:41.013 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-28T23:31:41.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.117+0000 7fb901181640 1 Processor -- start 2024-03-28T23:31:41.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.117+0000 7fb901181640 1 -- start start 2024-03-28T23:31:41.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.118+0000 7fb901181640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb8fc150520 0x7fb8fc0ab280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:41.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.118+0000 7fb901181640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8fc0c2f50 con 0x7fb8fc0a96f0 2024-03-28T23:31:41.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.118+0000 7fb901181640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8fc1530d0 con 0x7fb8fc0c1040 2024-03-28T23:31:41.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.118+0000 7fb901181640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb8fc0a9c20 con 0x7fb8fc150520 2024-03-28T23:31:41.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.118+0000 7fb8fad9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb8fc150520 0x7fb8fc0ab280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:41.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.118+0000 7fb8fb59e640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb8fc0a96f0 0x7fb8fc15d8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:43948/0 (socket says 172.21.15.161:43948) 2024-03-28T23:31:41.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.118+0000 7fb8fb59e640 1 -- 172.21.15.161:0/2789760004 learned_addr learned my addr 172.21.15.161:0/2789760004 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:41.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.118+0000 7fb8fad9d640 1 -- 172.21.15.161:0/2789760004 >> v1:172.21.15.170:6789/0 conn(0x7fb8fc0c1040 legacy=0x7fb8fc1535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:41.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.118+0000 7fb8fad9d640 1 -- 172.21.15.161:0/2789760004 >> v1:172.21.15.161:6789/0 conn(0x7fb8fc0a96f0 legacy=0x7fb8fc15d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:41.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.118+0000 7fb8fa59c640 1 -- 172.21.15.161:0/2789760004 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2581968747 0 0) 0x7fb8fc0c2f50 con 0x7fb8fc0a96f0 2024-03-28T23:31:41.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.118+0000 7fb8fad9d640 1 -- 172.21.15.161:0/2789760004 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8fc0c2990 con 0x7fb8fc150520 2024-03-28T23:31:41.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.119+0000 7fb8fad9d640 1 --2- 172.21.15.161:0/2789760004 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8fc150520 0x7fb8fc0ab280 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec0097a0 tx=0x7fb8ec02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=143fb80775975f17 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:41.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.119+0000 7fb8fa59c640 1 -- 172.21.15.161:0/2789760004 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec0315a0 con 0x7fb8fc150520 2024-03-28T23:31:41.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.119+0000 7fb8fa59c640 1 -- 172.21.15.161:0/2789760004 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8ec037070 con 0x7fb8fc150520 2024-03-28T23:31:41.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.119+0000 7fb8fa59c640 1 -- 172.21.15.161:0/2789760004 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec031960 con 0x7fb8fc150520 2024-03-28T23:31:41.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.120+0000 7fb901181640 1 -- 172.21.15.161:0/2789760004 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8fc150520 msgr2=0x7fb8fc0ab280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:41.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.120+0000 7fb901181640 1 --2- 172.21.15.161:0/2789760004 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8fc150520 0x7fb8fc0ab280 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec0097a0 tx=0x7fb8ec02b840 comp rx=0 tx=0).stop 2024-03-28T23:31:41.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.120+0000 7fb901181640 1 -- 172.21.15.161:0/2789760004 shutdown_connections 2024-03-28T23:31:41.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.120+0000 7fb901181640 1 --2- 172.21.15.161:0/2789760004 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8fc150520 0x7fb8fc0ab280 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:41.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.120+0000 7fb901181640 1 -- 172.21.15.161:0/2789760004 >> 172.21.15.161:0/2789760004 conn(0x7fb8fc0a83f0 msgr2=0x7fb8fc0a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:41.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.120+0000 7fb901181640 1 -- 172.21.15.161:0/2789760004 shutdown_connections 2024-03-28T23:31:41.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.120+0000 7fb901181640 1 -- 172.21.15.161:0/2789760004 wait complete. 2024-03-28T23:31:41.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.121+0000 7fb901181640 1 Processor -- start 2024-03-28T23:31:41.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.121+0000 7fb901181640 1 -- start start 2024-03-28T23:31:41.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.121+0000 7fb901181640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8fc0a96f0 0x7fb8fc0bfd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:41.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.121+0000 7fb901181640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8fc0c1040 0x7fb8fc1dc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:41.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.122+0000 7fb901181640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8fc150520 0x7fb8fc1dcad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:41.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.122+0000 7fb901181640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb8fc0c0c00 con 0x7fb8fc0a96f0 2024-03-28T23:31:41.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.122+0000 7fb901181640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb8fc150900 con 0x7fb8fc0c1040 2024-03-28T23:31:41.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.122+0000 7fb901181640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb8fc0c0d80 con 0x7fb8fc150520 2024-03-28T23:31:41.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.122+0000 7fb8fad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8fc0a96f0 0x7fb8fc0bfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:41.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.122+0000 7fb8fad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8fc0a96f0 0x7fb8fc0bfd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58336/0 (socket says 172.21.15.161:58336) 2024-03-28T23:31:41.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.122+0000 7fb8fad9d640 1 -- 172.21.15.161:0/2438693710 learned_addr learned my addr 172.21.15.161:0/2438693710 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:41.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.122+0000 7fb8fad9d640 1 -- 172.21.15.161:0/2438693710 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8fc150520 msgr2=0x7fb8fc1dcad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:41.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.122+0000 7fb8fad9d640 1 --2- 172.21.15.161:0/2438693710 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8fc150520 0x7fb8fc1dcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:41.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.122+0000 7fb8f3fff640 1 --2- 172.21.15.161:0/2438693710 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8fc0c1040 0x7fb8fc1dc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:41.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.122+0000 7fb8fad9d640 1 -- 172.21.15.161:0/2438693710 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8fc0c1040 msgr2=0x7fb8fc1dc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:41.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.122+0000 7fb8fad9d640 1 --2- 172.21.15.161:0/2438693710 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8fc0c1040 0x7fb8fc1dc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:41.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.122+0000 7fb8fad9d640 1 -- 172.21.15.161:0/2438693710 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8fc06c780 con 0x7fb8fc0a96f0 2024-03-28T23:31:41.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.122+0000 7fb8f3fff640 1 --2- 172.21.15.161:0/2438693710 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8fc0c1040 0x7fb8fc1dc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:41.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.122+0000 7fb8fad9d640 1 --2- 172.21.15.161:0/2438693710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8fc0a96f0 0x7fb8fc0bfd10 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec034040 tx=0x7fb8ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:41.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.122+0000 7fb8f37fe640 1 -- 172.21.15.161:0/2438693710 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec004230 con 0x7fb8fc0a96f0 2024-03-28T23:31:41.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.122+0000 7fb8f37fe640 1 -- 172.21.15.161:0/2438693710 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8ec0043d0 con 0x7fb8fc0a96f0 2024-03-28T23:31:41.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.122+0000 7fb901181640 1 -- 172.21.15.161:0/2438693710 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8fc06ca60 con 0x7fb8fc0a96f0 2024-03-28T23:31:41.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.122+0000 7fb8f37fe640 1 -- 172.21.15.161:0/2438693710 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec032880 con 0x7fb8fc0a96f0 2024-03-28T23:31:41.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.122+0000 7fb901181640 1 -- 172.21.15.161:0/2438693710 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8fc06d020 con 0x7fb8fc0a96f0 2024-03-28T23:31:41.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.123+0000 7fb901181640 1 -- 172.21.15.161:0/2438693710 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8fc04f4b0 con 0x7fb8fc0a96f0 2024-03-28T23:31:41.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.124+0000 7fb8f37fe640 1 -- 172.21.15.161:0/2438693710 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb8ec032a20 con 0x7fb8fc0a96f0 2024-03-28T23:31:41.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.124+0000 7fb8f37fe640 1 --2- 172.21.15.161:0/2438693710 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb8dc06b500 0x7fb8dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:41.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.124+0000 7fb8f37fe640 1 -- 172.21.15.161:0/2438693710 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6101+0+0 (secure 0 0 0) 0x7fb8ec02f030 con 0x7fb8fc0a96f0 2024-03-28T23:31:41.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.124+0000 7fb8f3fff640 1 --2- 172.21.15.161:0/2438693710 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb8dc06b500 0x7fb8dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:41.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.125+0000 7fb8f3fff640 1 --2- 172.21.15.161:0/2438693710 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb8dc06b500 0x7fb8dc06d9c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc151b30 tx=0x7fb8e4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:41.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.129+0000 7fb8f37fe640 1 -- 172.21.15.161:0/2438693710 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb8ec041070 con 0x7fb8fc0a96f0 2024-03-28T23:31:41.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.336+0000 7fb901181640 1 -- 172.21.15.161:0/2438693710 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fb8fc1513b0 con 0x7fb8fc0a96f0 2024-03-28T23:31:41.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.387+0000 7fb8f37fe640 1 -- 172.21.15.161:0/2438693710 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v29) ==== 118+0+0 (secure 0 0 0) 0x7fb8ec044340 con 0x7fb8fc0a96f0 2024-03-28T23:31:41.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.390+0000 7fb901181640 1 -- 172.21.15.161:0/2438693710 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb8dc06b500 msgr2=0x7fb8dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:41.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.390+0000 7fb901181640 1 --2- 172.21.15.161:0/2438693710 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb8dc06b500 0x7fb8dc06d9c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc151b30 tx=0x7fb8e4009040 comp rx=0 tx=0).stop 2024-03-28T23:31:41.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.390+0000 7fb901181640 1 -- 172.21.15.161:0/2438693710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8fc0a96f0 msgr2=0x7fb8fc0bfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:41.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.390+0000 7fb901181640 1 --2- 172.21.15.161:0/2438693710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8fc0a96f0 0x7fb8fc0bfd10 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec034040 tx=0x7fb8ec0046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:41.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.390+0000 7fb901181640 1 -- 172.21.15.161:0/2438693710 shutdown_connections 2024-03-28T23:31:41.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.390+0000 7fb901181640 1 --2- 172.21.15.161:0/2438693710 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb8dc06b500 0x7fb8dc06d9c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:41.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.390+0000 7fb901181640 1 --2- 172.21.15.161:0/2438693710 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8fc150520 0x7fb8fc1dcad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:41.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.390+0000 7fb901181640 1 --2- 172.21.15.161:0/2438693710 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8fc0c1040 0x7fb8fc1dc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:41.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.390+0000 7fb901181640 1 --2- 172.21.15.161:0/2438693710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8fc0a96f0 0x7fb8fc0bfd10 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:41.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.390+0000 7fb901181640 1 -- 172.21.15.161:0/2438693710 >> 172.21.15.161:0/2438693710 conn(0x7fb8fc0a83f0 msgr2=0x7fb8fc145120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:41.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.390+0000 7fb901181640 1 -- 172.21.15.161:0/2438693710 shutdown_connections 2024-03-28T23:31:41.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.390+0000 7fb901181640 1 -- 172.21.15.161:0/2438693710 wait complete. 2024-03-28T23:31:41.402 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-28T23:31:41.402 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-28T23:31:41.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.506+0000 7fc8912cd640 1 Processor -- start 2024-03-28T23:31:41.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.506+0000 7fc8912cd640 1 -- start start 2024-03-28T23:31:41.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.507+0000 7fc8912cd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc88c152bd0 0x7fc88c15d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:41.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.507+0000 7fc8912cd640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc88c0c2f10 con 0x7fc88c06cdd0 2024-03-28T23:31:41.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.507+0000 7fc8912cd640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc88c14f9a0 con 0x7fc88c14f5c0 2024-03-28T23:31:41.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.507+0000 7fc8912cd640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc88c0c74d0 con 0x7fc88c152bd0 2024-03-28T23:31:41.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.507+0000 7fc88b7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc88c152bd0 0x7fc88c15d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:41.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.507+0000 7fc88a7fc640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc88c06cdd0 0x7fc88c1524c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:43952/0 (socket says 172.21.15.161:43952) 2024-03-28T23:31:41.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.507+0000 7fc88a7fc640 1 -- 172.21.15.161:0/978874811 learned_addr learned my addr 172.21.15.161:0/978874811 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:41.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.507+0000 7fc88b7fe640 1 -- 172.21.15.161:0/978874811 >> v1:172.21.15.170:6789/0 conn(0x7fc88c14f5c0 legacy=0x7fc88c0c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:41.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.508+0000 7fc88b7fe640 1 -- 172.21.15.161:0/978874811 >> v1:172.21.15.161:6789/0 conn(0x7fc88c06cdd0 legacy=0x7fc88c1524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:41.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.508+0000 7fc889ffb640 1 -- 172.21.15.161:0/978874811 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1504578258 0 0) 0x7fc880001110 con 0x7fc88c06cdd0 2024-03-28T23:31:41.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.508+0000 7fc88b7fe640 1 -- 172.21.15.161:0/978874811 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc88c06c140 con 0x7fc88c152bd0 2024-03-28T23:31:41.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.508+0000 7fc88b7fe640 1 --2- 172.21.15.161:0/978874811 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc88c152bd0 0x7fc88c15d840 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fc884009d90 tx=0x7fc88402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=958eb318c0e7f52e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:41.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.508+0000 7fc889ffb640 1 -- 172.21.15.161:0/978874811 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc884039070 con 0x7fc88c152bd0 2024-03-28T23:31:41.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.508+0000 7fc889ffb640 1 -- 172.21.15.161:0/978874811 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc88402cd00 con 0x7fc88c152bd0 2024-03-28T23:31:41.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.508+0000 7fc889ffb640 1 -- 172.21.15.161:0/978874811 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc884034660 con 0x7fc88c152bd0 2024-03-28T23:31:41.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.509+0000 7fc8912cd640 1 -- 172.21.15.161:0/978874811 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc88c152bd0 msgr2=0x7fc88c15d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:41.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.509+0000 7fc8912cd640 1 --2- 172.21.15.161:0/978874811 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc88c152bd0 0x7fc88c15d840 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fc884009d90 tx=0x7fc88402bf50 comp rx=0 tx=0).stop 2024-03-28T23:31:41.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.509+0000 7fc8912cd640 1 -- 172.21.15.161:0/978874811 shutdown_connections 2024-03-28T23:31:41.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.509+0000 7fc8912cd640 1 --2- 172.21.15.161:0/978874811 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc88c152bd0 0x7fc88c15d840 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:41.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.509+0000 7fc8912cd640 1 -- 172.21.15.161:0/978874811 >> 172.21.15.161:0/978874811 conn(0x7fc88c0a83f0 msgr2=0x7fc88c0a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:41.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.509+0000 7fc8912cd640 1 -- 172.21.15.161:0/978874811 shutdown_connections 2024-03-28T23:31:41.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.509+0000 7fc8912cd640 1 -- 172.21.15.161:0/978874811 wait complete. 2024-03-28T23:31:41.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.510+0000 7fc8912cd640 1 Processor -- start 2024-03-28T23:31:41.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.511+0000 7fc8912cd640 1 -- start start 2024-03-28T23:31:41.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.511+0000 7fc8912cd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc88c06cdd0 0x7fc88c1de8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:41.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.511+0000 7fc8912cd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc88c14f5c0 0x7fc88c0720c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:41.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.511+0000 7fc8912cd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc88c152bd0 0x7fc88c072600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:41.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.511+0000 7fc8912cd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c152030 con 0x7fc88c152bd0 2024-03-28T23:31:41.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.511+0000 7fc8912cd640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c0c6fa0 con 0x7fc88c14f5c0 2024-03-28T23:31:41.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.511+0000 7fc8912cd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc88c06c3e0 con 0x7fc88c06cdd0 2024-03-28T23:31:41.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.511+0000 7fc88b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc88c152bd0 0x7fc88c072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:41.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.511+0000 7fc88b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc88c152bd0 0x7fc88c072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58352/0 (socket says 172.21.15.161:58352) 2024-03-28T23:31:41.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.511+0000 7fc88b7fe640 1 -- 172.21.15.161:0/855136664 learned_addr learned my addr 172.21.15.161:0/855136664 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:41.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.512+0000 7fc88affd640 1 --2- 172.21.15.161:0/855136664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc88c06cdd0 0x7fc88c1de8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:41.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.512+0000 7fc88a7fc640 1 --2- 172.21.15.161:0/855136664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc88c14f5c0 0x7fc88c0720c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:41.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.512+0000 7fc88b7fe640 1 -- 172.21.15.161:0/855136664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc88c06cdd0 msgr2=0x7fc88c1de8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:41.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.512+0000 7fc88b7fe640 1 --2- 172.21.15.161:0/855136664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc88c06cdd0 0x7fc88c1de8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:41.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.512+0000 7fc88b7fe640 1 -- 172.21.15.161:0/855136664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc88c14f5c0 msgr2=0x7fc88c0720c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:41.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.512+0000 7fc88b7fe640 1 --2- 172.21.15.161:0/855136664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc88c14f5c0 0x7fc88c0720c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:41.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.512+0000 7fc88b7fe640 1 -- 172.21.15.161:0/855136664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc88c14d990 con 0x7fc88c152bd0 2024-03-28T23:31:41.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.512+0000 7fc88a7fc640 1 --2- 172.21.15.161:0/855136664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc88c14f5c0 0x7fc88c0720c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:41.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.512+0000 7fc88b7fe640 1 --2- 172.21.15.161:0/855136664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc88c152bd0 0x7fc88c072600 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc884009d90 tx=0x7fc884004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:41.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.512+0000 7fc8737fe640 1 -- 172.21.15.161:0/855136664 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc884039050 con 0x7fc88c152bd0 2024-03-28T23:31:41.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.512+0000 7fc8737fe640 1 -- 172.21.15.161:0/855136664 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc884045070 con 0x7fc88c152bd0 2024-03-28T23:31:41.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.512+0000 7fc8912cd640 1 -- 172.21.15.161:0/855136664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc88c14dc30 con 0x7fc88c152bd0 2024-03-28T23:31:41.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.512+0000 7fc8737fe640 1 -- 172.21.15.161:0/855136664 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88403e850 con 0x7fc88c152bd0 2024-03-28T23:31:41.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.512+0000 7fc8912cd640 1 -- 172.21.15.161:0/855136664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc88c0718c0 con 0x7fc88c152bd0 2024-03-28T23:31:41.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.513+0000 7fc8912cd640 1 -- 172.21.15.161:0/855136664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc88c04f4b0 con 0x7fc88c152bd0 2024-03-28T23:31:41.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.514+0000 7fc8737fe640 1 -- 172.21.15.161:0/855136664 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc88403e9f0 con 0x7fc88c152bd0 2024-03-28T23:31:41.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.514+0000 7fc8737fe640 1 --2- 172.21.15.161:0/855136664 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc87806b4d0 0x7fc87806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:41.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.514+0000 7fc8737fe640 1 -- 172.21.15.161:0/855136664 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7fc8840ad1a0 con 0x7fc88c152bd0 2024-03-28T23:31:41.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.514+0000 7fc88affd640 1 --2- 172.21.15.161:0/855136664 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc87806b4d0 0x7fc87806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:41.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.514+0000 7fc88affd640 1 --2- 172.21.15.161:0/855136664 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc87806b4d0 0x7fc87806d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc87c000f60 tx=0x7fc87c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:41.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.518+0000 7fc8737fe640 1 -- 172.21.15.161:0/855136664 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc88407bfe0 con 0x7fc88c152bd0 2024-03-28T23:31:41.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:41.816+0000 7fc8912cd640 1 -- 172.21.15.161:0/855136664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fc88c06b830 con 0x7fc88c152bd0 2024-03-28T23:31:42.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.391+0000 7fc8737fe640 1 -- 172.21.15.161:0/855136664 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v30) ==== 122+0+0 (secure 0 0 0) 0x7fc88407b8e0 con 0x7fc88c152bd0 2024-03-28T23:31:42.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.394+0000 7fc8912cd640 1 -- 172.21.15.161:0/855136664 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc87806b4d0 msgr2=0x7fc87806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:42.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.394+0000 7fc8912cd640 1 --2- 172.21.15.161:0/855136664 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc87806b4d0 0x7fc87806d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc87c000f60 tx=0x7fc87c009040 comp rx=0 tx=0).stop 2024-03-28T23:31:42.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.394+0000 7fc8912cd640 1 -- 172.21.15.161:0/855136664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc88c152bd0 msgr2=0x7fc88c072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:42.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.394+0000 7fc8912cd640 1 --2- 172.21.15.161:0/855136664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc88c152bd0 0x7fc88c072600 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc884009d90 tx=0x7fc884004a40 comp rx=0 tx=0).stop 2024-03-28T23:31:42.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.394+0000 7fc8912cd640 1 -- 172.21.15.161:0/855136664 shutdown_connections 2024-03-28T23:31:42.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.394+0000 7fc8912cd640 1 --2- 172.21.15.161:0/855136664 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc87806b4d0 0x7fc87806d990 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:42.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.394+0000 7fc8912cd640 1 --2- 172.21.15.161:0/855136664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc88c152bd0 0x7fc88c072600 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:42.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.394+0000 7fc8912cd640 1 --2- 172.21.15.161:0/855136664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc88c14f5c0 0x7fc88c0720c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:42.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.394+0000 7fc8912cd640 1 --2- 172.21.15.161:0/855136664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc88c06cdd0 0x7fc88c1de8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:42.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.394+0000 7fc8912cd640 1 -- 172.21.15.161:0/855136664 >> 172.21.15.161:0/855136664 conn(0x7fc88c0a83f0 msgr2=0x7fc88c150a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:42.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.394+0000 7fc8912cd640 1 -- 172.21.15.161:0/855136664 shutdown_connections 2024-03-28T23:31:42.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.394+0000 7fc8912cd640 1 -- 172.21.15.161:0/855136664 wait complete. 2024-03-28T23:31:42.409 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-28T23:31:42.409 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:31:42.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.509+0000 7fe40370d640 1 Processor -- start 2024-03-28T23:31:42.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.509+0000 7fe40370d640 1 -- start start 2024-03-28T23:31:42.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.510+0000 7fe40370d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3fc152890 0x7fe3fc0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:42.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.510+0000 7fe40370d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe3fc0c5b20 0x7fe3fc1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:42.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.510+0000 7fe40370d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3fc158a30 con 0x7fe3fc0c8ed0 2024-03-28T23:31:42.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.510+0000 7fe40370d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe3fc0c8480 con 0x7fe3fc0c5b20 2024-03-28T23:31:42.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.510+0000 7fe40370d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe3fc159640 con 0x7fe3fc152890 2024-03-28T23:31:42.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.510+0000 7fe401caa640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe3fc0c5b20 0x7fe3fc1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:42.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.510+0000 7fe401caa640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe3fc0c5b20 0x7fe3fc1553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58358/0 (socket says 172.21.15.161:58358) 2024-03-28T23:31:42.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.510+0000 7fe401caa640 1 -- 172.21.15.161:0/3988550249 learned_addr learned my addr 172.21.15.161:0/3988550249 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:42.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.510+0000 7fe400ca8640 1 --1- 172.21.15.161:0/3988550249 >> v1:172.21.15.161:6789/0 conn(0x7fe3fc0c8ed0 0x7fe3fc0c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:43960/0 (socket says 172.21.15.161:43960) 2024-03-28T23:31:42.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.510+0000 7fe4014a9640 1 --2- 172.21.15.161:0/3988550249 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3fc152890 0x7fe3fc0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:42.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.510+0000 7fe401caa640 1 -- 172.21.15.161:0/3988550249 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3fc152890 msgr2=0x7fe3fc0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:42.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.510+0000 7fe401caa640 1 --2- 172.21.15.161:0/3988550249 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3fc152890 0x7fe3fc0c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:42.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.510+0000 7fe401caa640 1 -- 172.21.15.161:0/3988550249 >> v1:172.21.15.161:6789/0 conn(0x7fe3fc0c8ed0 legacy=0x7fe3fc0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:42.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.510+0000 7fe401caa640 1 -- 172.21.15.161:0/3988550249 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3fc158d10 con 0x7fe3fc0c5b20 2024-03-28T23:31:42.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.510+0000 7fe3f3fff640 1 -- 172.21.15.161:0/3988550249 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 653882199 0 0) 0x7fe3fc158a30 con 0x7fe3fc0c8ed0 2024-03-28T23:31:42.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.511+0000 7fe401caa640 1 --2- 172.21.15.161:0/3988550249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3fc0c5b20 0x7fe3fc1553a0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc1597c0 tx=0x7fe3f402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60994e61df6e3803 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:42.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.511+0000 7fe3f3fff640 1 -- 172.21.15.161:0/3988550249 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f402c9c0 con 0x7fe3fc0c5b20 2024-03-28T23:31:42.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.511+0000 7fe3f3fff640 1 -- 172.21.15.161:0/3988550249 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3f402cb60 con 0x7fe3fc0c5b20 2024-03-28T23:31:42.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.511+0000 7fe3f3fff640 1 -- 172.21.15.161:0/3988550249 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f40343f0 con 0x7fe3fc0c5b20 2024-03-28T23:31:42.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.511+0000 7fe40370d640 1 -- 172.21.15.161:0/3988550249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3fc0c5b20 msgr2=0x7fe3fc1553a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:42.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.512+0000 7fe40370d640 1 --2- 172.21.15.161:0/3988550249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3fc0c5b20 0x7fe3fc1553a0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc1597c0 tx=0x7fe3f402be20 comp rx=0 tx=0).stop 2024-03-28T23:31:42.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.512+0000 7fe40370d640 1 -- 172.21.15.161:0/3988550249 shutdown_connections 2024-03-28T23:31:42.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.512+0000 7fe40370d640 1 --2- 172.21.15.161:0/3988550249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3fc0c5b20 0x7fe3fc1553a0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:42.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.512+0000 7fe40370d640 1 --2- 172.21.15.161:0/3988550249 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3fc152890 0x7fe3fc0c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:42.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.512+0000 7fe40370d640 1 -- 172.21.15.161:0/3988550249 >> 172.21.15.161:0/3988550249 conn(0x7fe3fc0ac620 msgr2=0x7fe3fc0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:42.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.512+0000 7fe40370d640 1 -- 172.21.15.161:0/3988550249 shutdown_connections 2024-03-28T23:31:42.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.512+0000 7fe40370d640 1 -- 172.21.15.161:0/3988550249 wait complete. 2024-03-28T23:31:42.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.513+0000 7fe40370d640 1 Processor -- start 2024-03-28T23:31:42.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.513+0000 7fe40370d640 1 -- start start 2024-03-28T23:31:42.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.514+0000 7fe40370d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3fc0c5b20 0x7fe3fc147ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:42.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.514+0000 7fe40370d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3fc0c8ed0 0x7fe3fc0ab370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:42.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.514+0000 7fe40370d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3fc152890 0x7fe3fc0ab8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:42.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.514+0000 7fe40370d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe3fc152c70 con 0x7fe3fc0c5b20 2024-03-28T23:31:42.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.514+0000 7fe40370d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe3fc1592d0 con 0x7fe3fc0c8ed0 2024-03-28T23:31:42.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.514+0000 7fe40370d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe3fc152df0 con 0x7fe3fc152890 2024-03-28T23:31:42.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.515+0000 7fe4014a9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3fc0c5b20 0x7fe3fc147ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:42.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.515+0000 7fe4014a9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3fc0c5b20 0x7fe3fc147ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58370/0 (socket says 172.21.15.161:58370) 2024-03-28T23:31:42.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.515+0000 7fe4014a9640 1 -- 172.21.15.161:0/2022677113 learned_addr learned my addr 172.21.15.161:0/2022677113 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:42.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.515+0000 7fe401caa640 1 --2- 172.21.15.161:0/2022677113 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3fc152890 0x7fe3fc0ab8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:42.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.515+0000 7fe4014a9640 1 -- 172.21.15.161:0/2022677113 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3fc152890 msgr2=0x7fe3fc0ab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:42.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.515+0000 7fe400ca8640 1 --2- 172.21.15.161:0/2022677113 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3fc0c8ed0 0x7fe3fc0ab370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:42.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.515+0000 7fe4014a9640 1 --2- 172.21.15.161:0/2022677113 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3fc152890 0x7fe3fc0ab8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:42.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.515+0000 7fe4014a9640 1 -- 172.21.15.161:0/2022677113 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3fc0c8ed0 msgr2=0x7fe3fc0ab370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:42.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.515+0000 7fe4014a9640 1 --2- 172.21.15.161:0/2022677113 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3fc0c8ed0 0x7fe3fc0ab370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:42.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.515+0000 7fe4014a9640 1 -- 172.21.15.161:0/2022677113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3fc0c24a0 con 0x7fe3fc0c5b20 2024-03-28T23:31:42.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.515+0000 7fe401caa640 1 --2- 172.21.15.161:0/2022677113 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3fc152890 0x7fe3fc0ab8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:42.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.515+0000 7fe400ca8640 1 --2- 172.21.15.161:0/2022677113 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3fc0c8ed0 0x7fe3fc0ab370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:42.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.515+0000 7fe4014a9640 1 --2- 172.21.15.161:0/2022677113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3fc0c5b20 0x7fe3fc147ab0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fe3f800d800 tx=0x7fe3f800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:42.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.515+0000 7fe3f1ffb640 1 -- 172.21.15.161:0/2022677113 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f800b6c0 con 0x7fe3fc0c5b20 2024-03-28T23:31:42.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.515+0000 7fe3f1ffb640 1 -- 172.21.15.161:0/2022677113 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3f8019070 con 0x7fe3fc0c5b20 2024-03-28T23:31:42.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.515+0000 7fe40370d640 1 -- 172.21.15.161:0/2022677113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3fc147ff0 con 0x7fe3fc0c5b20 2024-03-28T23:31:42.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.515+0000 7fe3f1ffb640 1 -- 172.21.15.161:0/2022677113 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f800ba80 con 0x7fe3fc0c5b20 2024-03-28T23:31:42.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.515+0000 7fe40370d640 1 -- 172.21.15.161:0/2022677113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3fc0c29f0 con 0x7fe3fc0c5b20 2024-03-28T23:31:42.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.517+0000 7fe40370d640 1 -- 172.21.15.161:0/2022677113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3fc04f4b0 con 0x7fe3fc0c5b20 2024-03-28T23:31:42.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.517+0000 7fe3f1ffb640 1 -- 172.21.15.161:0/2022677113 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe3f800bc20 con 0x7fe3fc0c5b20 2024-03-28T23:31:42.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.518+0000 7fe3f1ffb640 1 --2- 172.21.15.161:0/2022677113 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe3e006b4d0 0x7fe3e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:42.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.518+0000 7fe400ca8640 1 --2- 172.21.15.161:0/2022677113 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe3e006b4d0 0x7fe3e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:42.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.518+0000 7fe3f1ffb640 1 -- 172.21.15.161:0/2022677113 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7fe3f808a7b0 con 0x7fe3fc0c5b20 2024-03-28T23:31:42.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.518+0000 7fe400ca8640 1 --2- 172.21.15.161:0/2022677113 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe3e006b4d0 0x7fe3e006d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc1584b0 tx=0x7fe3ec009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:42.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.522+0000 7fe3f1ffb640 1 -- 172.21.15.161:0/2022677113 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe3f8059530 con 0x7fe3fc0c5b20 2024-03-28T23:31:42.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.756+0000 7fe40370d640 1 -- 172.21.15.161:0/2022677113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3fc0c1da0 con 0x7fe3fc0c5b20 2024-03-28T23:31:42.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.757+0000 7fe3f1ffb640 1 -- 172.21.15.161:0/2022677113 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5190 (secure 0 0 0) 0x7fe3f8058e30 con 0x7fe3fc0c5b20 2024-03-28T23:31:42.758 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:42.758 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5186,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/1945122204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1945122204},{"type":"v1","addr":"172.21.15.170:6835","nonce":1945122204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":5542,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/3666899516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3666899516},{"type":"v1","addr":"172.21.15.170:6833","nonce":3666899516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5817,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/1300583128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1300583128},{"type":"v1","addr":"172.21.15.161:6836","nonce":1300583128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:31:39.725625+0000","modified":"2024-03-28T23:31:41.818149+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5820},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5820":{"gid":5820,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":4,"addr":"172.21.15.161:6837/544971001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":544971001},{"type":"v1","addr":"172.21.15.161:6837","nonce":544971001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5820,"qdb_cluster":[5820]},"id":3}]} 2024-03-28T23:31:42.758 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 30 2024-03-28T23:31:42.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.759+0000 7fe40370d640 1 -- 172.21.15.161:0/2022677113 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe3e006b4d0 msgr2=0x7fe3e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:42.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.759+0000 7fe40370d640 1 --2- 172.21.15.161:0/2022677113 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe3e006b4d0 0x7fe3e006d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc1584b0 tx=0x7fe3ec009040 comp rx=0 tx=0).stop 2024-03-28T23:31:42.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.759+0000 7fe40370d640 1 -- 172.21.15.161:0/2022677113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3fc0c5b20 msgr2=0x7fe3fc147ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:42.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.759+0000 7fe40370d640 1 --2- 172.21.15.161:0/2022677113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3fc0c5b20 0x7fe3fc147ab0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fe3f800d800 tx=0x7fe3f800dcc0 comp rx=0 tx=0).stop 2024-03-28T23:31:42.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.759+0000 7fe40370d640 1 -- 172.21.15.161:0/2022677113 shutdown_connections 2024-03-28T23:31:42.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.759+0000 7fe40370d640 1 --2- 172.21.15.161:0/2022677113 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe3e006b4d0 0x7fe3e006d990 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:42.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.759+0000 7fe40370d640 1 --2- 172.21.15.161:0/2022677113 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3fc152890 0x7fe3fc0ab8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:42.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.759+0000 7fe40370d640 1 --2- 172.21.15.161:0/2022677113 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3fc0c8ed0 0x7fe3fc0ab370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:42.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.759+0000 7fe40370d640 1 --2- 172.21.15.161:0/2022677113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3fc0c5b20 0x7fe3fc147ab0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:42.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.759+0000 7fe40370d640 1 -- 172.21.15.161:0/2022677113 >> 172.21.15.161:0/2022677113 conn(0x7fe3fc0ac620 msgr2=0x7fe3fc0ca070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:42.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.759+0000 7fe40370d640 1 -- 172.21.15.161:0/2022677113 shutdown_connections 2024-03-28T23:31:42.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.759+0000 7fe40370d640 1 -- 172.21.15.161:0/2022677113 wait complete. 2024-03-28T23:31:42.770 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:31:42.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.870+0000 7f9209202640 1 Processor -- start 2024-03-28T23:31:42.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.870+0000 7f9209202640 1 -- start start 2024-03-28T23:31:42.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.871+0000 7f9209202640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92040c7350 0x7f92040c7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:42.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.871+0000 7f9209202640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9204155460 con 0x7f9204159050 2024-03-28T23:31:42.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.871+0000 7f9209202640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9204155700 con 0x7f92040c8f90 2024-03-28T23:31:42.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.871+0000 7f9209202640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9204156030 con 0x7f92040c7350 2024-03-28T23:31:42.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.871+0000 7f920359e640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f9204159050 0x7f920415fbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:43970/0 (socket says 172.21.15.161:43970) 2024-03-28T23:31:42.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.871+0000 7f920359e640 1 -- 172.21.15.161:0/378394140 learned_addr learned my addr 172.21.15.161:0/378394140 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:42.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.871+0000 7f9202d9d640 1 --2- 172.21.15.161:0/378394140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92040c7350 0x7f92040c7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:42.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.872+0000 7f920259c640 1 -- 172.21.15.161:0/378394140 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1509835653 0 0) 0x7f9204155460 con 0x7f9204159050 2024-03-28T23:31:42.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.872+0000 7f920259c640 1 -- 172.21.15.161:0/378394140 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91f0003520 con 0x7f9204159050 2024-03-28T23:31:42.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.872+0000 7f920259c640 1 -- 172.21.15.161:0/378394140 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1772778021 0 0) 0x7f9204155700 con 0x7f92040c8f90 2024-03-28T23:31:42.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.872+0000 7f920259c640 1 -- 172.21.15.161:0/378394140 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9204155460 con 0x7f92040c8f90 2024-03-28T23:31:42.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.872+0000 7f9202d9d640 1 -- 172.21.15.161:0/378394140 >> v1:172.21.15.170:6789/0 conn(0x7f92040c8f90 legacy=0x7f9204158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:42.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.872+0000 7f9202d9d640 1 -- 172.21.15.161:0/378394140 >> v1:172.21.15.161:6789/0 conn(0x7f9204159050 legacy=0x7f920415fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:42.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.872+0000 7f920259c640 1 -- 172.21.15.161:0/378394140 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3610979176 0 0) 0x7f91f0003520 con 0x7f9204159050 2024-03-28T23:31:42.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.872+0000 7f9202d9d640 1 -- 172.21.15.161:0/378394140 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92041559e0 con 0x7f92040c7350 2024-03-28T23:31:42.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.873+0000 7f9202d9d640 1 --2- 172.21.15.161:0/378394140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92040c7350 0x7f92040c7730 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f91f40097d0 tx=0x7f91f402b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f436e1b52d60a9f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:42.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.873+0000 7f920259c640 1 -- 172.21.15.161:0/378394140 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91f4037070 con 0x7f92040c7350 2024-03-28T23:31:42.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.873+0000 7f920259c640 1 -- 172.21.15.161:0/378394140 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91f4030510 con 0x7f92040c7350 2024-03-28T23:31:42.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.873+0000 7f920259c640 1 -- 172.21.15.161:0/378394140 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91fc0019f0 con 0x7f92040c7350 2024-03-28T23:31:42.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.873+0000 7f9209202640 1 -- 172.21.15.161:0/378394140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92040c7350 msgr2=0x7f92040c7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:42.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.873+0000 7f9209202640 1 --2- 172.21.15.161:0/378394140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92040c7350 0x7f92040c7730 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f91f40097d0 tx=0x7f91f402b770 comp rx=0 tx=0).stop 2024-03-28T23:31:42.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.873+0000 7f9209202640 1 -- 172.21.15.161:0/378394140 shutdown_connections 2024-03-28T23:31:42.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.873+0000 7f9209202640 1 --2- 172.21.15.161:0/378394140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92040c7350 0x7f92040c7730 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:42.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.873+0000 7f9209202640 1 -- 172.21.15.161:0/378394140 >> 172.21.15.161:0/378394140 conn(0x7f92040ac4e0 msgr2=0x7f92040ac8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:42.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.873+0000 7f9209202640 1 -- 172.21.15.161:0/378394140 shutdown_connections 2024-03-28T23:31:42.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.873+0000 7f9209202640 1 -- 172.21.15.161:0/378394140 wait complete. 2024-03-28T23:31:42.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.875+0000 7f9209202640 1 Processor -- start 2024-03-28T23:31:42.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.875+0000 7f9209202640 1 -- start start 2024-03-28T23:31:42.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.875+0000 7f9209202640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92040c7350 0x7f92041ef4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:42.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.875+0000 7f9209202640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92040c8f90 0x7f920414a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:42.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.875+0000 7f9209202640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9204159050 0x7f9204144c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:42.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.875+0000 7f9209202640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9204152c30 con 0x7f9204159050 2024-03-28T23:31:42.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.875+0000 7f9209202640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f92040c5d00 con 0x7f92040c7350 2024-03-28T23:31:42.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.876+0000 7f9209202640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f92041594e0 con 0x7f92040c8f90 2024-03-28T23:31:42.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.876+0000 7f920359e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9204159050 0x7f9204144c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:42.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.876+0000 7f920359e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9204159050 0x7f9204144c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58384/0 (socket says 172.21.15.161:58384) 2024-03-28T23:31:42.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.876+0000 7f920359e640 1 -- 172.21.15.161:0/910855564 learned_addr learned my addr 172.21.15.161:0/910855564 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:42.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.876+0000 7f9202d9d640 1 --2- 172.21.15.161:0/910855564 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92040c7350 0x7f92041ef4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:42.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.876+0000 7f91fa59c640 1 --2- 172.21.15.161:0/910855564 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92040c8f90 0x7f920414a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:42.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.876+0000 7f920359e640 1 -- 172.21.15.161:0/910855564 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92040c8f90 msgr2=0x7f920414a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:42.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.876+0000 7f920359e640 1 --2- 172.21.15.161:0/910855564 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92040c8f90 0x7f920414a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:42.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.876+0000 7f920359e640 1 -- 172.21.15.161:0/910855564 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92040c7350 msgr2=0x7f92041ef4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:42.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.876+0000 7f920359e640 1 --2- 172.21.15.161:0/910855564 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92040c7350 0x7f92041ef4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:42.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.876+0000 7f920359e640 1 -- 172.21.15.161:0/910855564 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f920414ec50 con 0x7f9204159050 2024-03-28T23:31:42.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.876+0000 7f920359e640 1 --2- 172.21.15.161:0/910855564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9204159050 0x7f9204144c20 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f91fc00cc30 tx=0x7f91fc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:42.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.876+0000 7f91fbfff640 1 -- 172.21.15.161:0/910855564 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91fc017070 con 0x7f9204159050 2024-03-28T23:31:42.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.876+0000 7f91fbfff640 1 -- 172.21.15.161:0/910855564 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91fc00ce40 con 0x7f9204159050 2024-03-28T23:31:42.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.876+0000 7f91fbfff640 1 -- 172.21.15.161:0/910855564 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91fc012620 con 0x7f9204159050 2024-03-28T23:31:42.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.876+0000 7f9209202640 1 -- 172.21.15.161:0/910855564 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f920414ef30 con 0x7f9204159050 2024-03-28T23:31:42.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.876+0000 7f9209202640 1 -- 172.21.15.161:0/910855564 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92040aa650 con 0x7f9204159050 2024-03-28T23:31:42.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.877+0000 7f9209202640 1 -- 172.21.15.161:0/910855564 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9204068640 con 0x7f9204159050 2024-03-28T23:31:42.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.878+0000 7f91fbfff640 1 -- 172.21.15.161:0/910855564 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f91fc01a070 con 0x7f9204159050 2024-03-28T23:31:42.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.879+0000 7f91fbfff640 1 --2- 172.21.15.161:0/910855564 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f91e406b500 0x7f91e406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:42.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.879+0000 7f91fbfff640 1 -- 172.21.15.161:0/910855564 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f91fc08a200 con 0x7f9204159050 2024-03-28T23:31:42.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.879+0000 7f9202d9d640 1 --2- 172.21.15.161:0/910855564 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f91e406b500 0x7f91e406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:42.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.879+0000 7f9202d9d640 1 --2- 172.21.15.161:0/910855564 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f91e406b500 0x7f91e406d9c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f91f4034000 tx=0x7f91f4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:42.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:42.883+0000 7f91fbfff640 1 -- 172.21.15.161:0/910855564 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f91fc0588a0 con 0x7f9204159050 2024-03-28T23:31:43.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.061+0000 7f9209202640 1 -- 172.21.15.161:0/910855564 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f920404f4b0 con 0x7f9204159050 2024-03-28T23:31:43.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.062+0000 7f91fbfff640 1 -- 172.21.15.161:0/910855564 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) ==== 74+0+16978 (secure 0 0 0) 0x7f91fc05c940 con 0x7f9204159050 2024-03-28T23:31:43.063 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:43.064 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":51,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:31:41.382740+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:31:37.940838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-28T23:31:38.692659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:31:43.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.064+0000 7f9209202640 1 -- 172.21.15.161:0/910855564 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f91e406b500 msgr2=0x7f91e406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:43.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.064+0000 7f9209202640 1 --2- 172.21.15.161:0/910855564 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f91e406b500 0x7f91e406d9c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f91f4034000 tx=0x7f91f4008030 comp rx=0 tx=0).stop 2024-03-28T23:31:43.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.064+0000 7f9209202640 1 -- 172.21.15.161:0/910855564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9204159050 msgr2=0x7f9204144c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:43.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.064+0000 7f9209202640 1 --2- 172.21.15.161:0/910855564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9204159050 0x7f9204144c20 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f91fc00cc30 tx=0x7f91fc00a430 comp rx=0 tx=0).stop 2024-03-28T23:31:43.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.064+0000 7f9209202640 1 -- 172.21.15.161:0/910855564 shutdown_connections 2024-03-28T23:31:43.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.065+0000 7f9209202640 1 --2- 172.21.15.161:0/910855564 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f91e406b500 0x7f91e406d9c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.065+0000 7f9209202640 1 --2- 172.21.15.161:0/910855564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9204159050 0x7f9204144c20 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.065+0000 7f9209202640 1 --2- 172.21.15.161:0/910855564 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92040c8f90 0x7f920414a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.065+0000 7f9209202640 1 --2- 172.21.15.161:0/910855564 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92040c7350 0x7f92041ef4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.065+0000 7f9209202640 1 -- 172.21.15.161:0/910855564 >> 172.21.15.161:0/910855564 conn(0x7f92040ac4e0 msgr2=0x7f9204158940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:43.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.065+0000 7f9209202640 1 -- 172.21.15.161:0/910855564 shutdown_connections 2024-03-28T23:31:43.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.065+0000 7f9209202640 1 -- 172.21.15.161:0/910855564 wait complete. 2024-03-28T23:31:43.077 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T23:31:43.078 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:31:43.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.194+0000 7f49d972c640 1 Processor -- start 2024-03-28T23:31:43.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.194+0000 7f49d972c640 1 -- start start 2024-03-28T23:31:43.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.195+0000 7f49d972c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49d4152890 0x7f49d40c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:43.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.195+0000 7f49d972c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49d4158a30 con 0x7f49d40c8ed0 2024-03-28T23:31:43.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.195+0000 7f49d972c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49d4158d10 con 0x7f49d40c5b20 2024-03-28T23:31:43.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.195+0000 7f49d972c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f49d40c8480 con 0x7f49d4152890 2024-03-28T23:31:43.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.195+0000 7f49d27fc640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f49d40c8ed0 0x7f49d40c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:43980/0 (socket says 172.21.15.161:43980) 2024-03-28T23:31:43.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.195+0000 7f49d27fc640 1 -- 172.21.15.161:0/1246786909 learned_addr learned my addr 172.21.15.161:0/1246786909 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:43.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.195+0000 7f49d1ffb640 1 -- 172.21.15.161:0/1246786909 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1642873836 0 0) 0x7f49d4158a30 con 0x7f49d40c8ed0 2024-03-28T23:31:43.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.195+0000 7f49d2ffd640 1 --2- 172.21.15.161:0/1246786909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49d4152890 0x7f49d40c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:43.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.195+0000 7f49d1ffb640 1 -- 172.21.15.161:0/1246786909 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49bc003520 con 0x7f49d40c8ed0 2024-03-28T23:31:43.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.195+0000 7f49d1ffb640 1 -- 172.21.15.161:0/1246786909 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 814468443 0 0) 0x7f49bc003520 con 0x7f49d40c8ed0 2024-03-28T23:31:43.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.195+0000 7f49d1ffb640 1 -- 172.21.15.161:0/1246786909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49d4152890 msgr2=0x7f49d40c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:43.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.195+0000 7f49d1ffb640 1 --2- 172.21.15.161:0/1246786909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49d4152890 0x7f49d40c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.195+0000 7f49d1ffb640 1 -- 172.21.15.161:0/1246786909 >> v1:172.21.15.170:6789/0 conn(0x7f49d40c5b20 legacy=0x7f49d41553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:43.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.195+0000 7f49d1ffb640 1 -- 172.21.15.161:0/1246786909 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d4158ff0 con 0x7f49d40c8ed0 2024-03-28T23:31:43.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.195+0000 7f49d1ffb640 1 -- 172.21.15.161:0/1246786909 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f49c80017f0 con 0x7f49d40c8ed0 2024-03-28T23:31:43.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.195+0000 7f49d1ffb640 1 -- 172.21.15.161:0/1246786909 >> v1:172.21.15.161:6789/0 conn(0x7f49d40c8ed0 legacy=0x7f49d40c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:43.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.196+0000 7f49d1ffb640 1 --2- 172.21.15.161:0/1246786909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49bc003ac0 0x7f49bc003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:43.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.196+0000 7f49d1ffb640 1 --2- 172.21.15.161:0/1246786909 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49bc004800 0x7f49bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:43.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.196+0000 7f49d1ffb640 1 --2- 172.21.15.161:0/1246786909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49d4152890 0x7f49bc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:43.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.196+0000 7f49d1ffb640 1 -- 172.21.15.161:0/1246786909 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f49bc005540 con 0x7f49bc004800 2024-03-28T23:31:43.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.196+0000 7f49d1ffb640 1 -- 172.21.15.161:0/1246786909 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f49bc00bf00 con 0x7f49bc003ac0 2024-03-28T23:31:43.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.196+0000 7f49d1ffb640 1 -- 172.21.15.161:0/1246786909 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f49bc00c080 con 0x7f49d4152890 2024-03-28T23:31:43.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.196+0000 7f49d27fc640 1 --2- 172.21.15.161:0/1246786909 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49bc004800 0x7f49bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:43.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.196+0000 7f49d2ffd640 1 --2- 172.21.15.161:0/1246786909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49bc003ac0 0x7f49bc003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:43.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.196+0000 7f49d37fe640 1 --2- 172.21.15.161:0/1246786909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49d4152890 0x7f49bc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:43.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.196+0000 7f49d27fc640 1 -- 172.21.15.161:0/1246786909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49d4152890 msgr2=0x7f49bc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:43.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.196+0000 7f49d27fc640 1 --2- 172.21.15.161:0/1246786909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49d4152890 0x7f49bc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.196+0000 7f49d27fc640 1 -- 172.21.15.161:0/1246786909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49bc003ac0 msgr2=0x7f49bc003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:43.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.196+0000 7f49d27fc640 1 --2- 172.21.15.161:0/1246786909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49bc003ac0 0x7f49bc003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.196+0000 7f49d27fc640 1 -- 172.21.15.161:0/1246786909 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49bc003520 con 0x7f49bc004800 2024-03-28T23:31:43.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.197+0000 7f49d27fc640 1 --2- 172.21.15.161:0/1246786909 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49bc004800 0x7f49bc004c80 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f49c800c1c0 tx=0x7f49bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f86a89da542179 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:43.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.197+0000 7f49d1ffb640 1 -- 172.21.15.161:0/1246786909 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c8007d00 con 0x7f49bc004800 2024-03-28T23:31:43.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.197+0000 7f49d1ffb640 1 -- 172.21.15.161:0/1246786909 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49c8030070 con 0x7f49bc004800 2024-03-28T23:31:43.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.197+0000 7f49d1ffb640 1 -- 172.21.15.161:0/1246786909 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c802cd30 con 0x7f49bc004800 2024-03-28T23:31:43.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.198+0000 7f49d972c640 1 -- 172.21.15.161:0/1246786909 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49bc004800 msgr2=0x7f49bc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:43.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.198+0000 7f49d972c640 1 --2- 172.21.15.161:0/1246786909 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49bc004800 0x7f49bc004c80 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f49c800c1c0 tx=0x7f49bc005830 comp rx=0 tx=0).stop 2024-03-28T23:31:43.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.198+0000 7f49d37fe640 1 -- 172.21.15.161:0/1246786909 reap_dead start 2024-03-28T23:31:43.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.198+0000 7f49d972c640 1 -- 172.21.15.161:0/1246786909 shutdown_connections 2024-03-28T23:31:43.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.198+0000 7f49d972c640 1 -- 172.21.15.161:0/1246786909 >> 172.21.15.161:0/1246786909 conn(0x7f49d40ac620 msgr2=0x7f49d40aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:43.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.198+0000 7f49d972c640 1 -- 172.21.15.161:0/1246786909 shutdown_connections 2024-03-28T23:31:43.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.198+0000 7f49d972c640 1 -- 172.21.15.161:0/1246786909 wait complete. 2024-03-28T23:31:43.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.199+0000 7f49d972c640 1 Processor -- start 2024-03-28T23:31:43.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.199+0000 7f49d972c640 1 -- start start 2024-03-28T23:31:43.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.200+0000 7f49d972c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49bc004800 0x7f49d40ab720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:43.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.200+0000 7f49d972c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49d40c3760 0x7f49d41ebe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:43.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.200+0000 7f49d972c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49d414f320 0x7f49d40c2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:43.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.200+0000 7f49d972c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f49d4152c70 con 0x7f49bc004800 2024-03-28T23:31:43.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.200+0000 7f49d972c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f49d4159640 con 0x7f49d414f320 2024-03-28T23:31:43.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.200+0000 7f49d972c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f49d4152df0 con 0x7f49d40c3760 2024-03-28T23:31:43.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.200+0000 7f49d2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49bc004800 0x7f49d40ab720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:43.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.200+0000 7f49d2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49bc004800 0x7f49d40ab720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58406/0 (socket says 172.21.15.161:58406) 2024-03-28T23:31:43.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.200+0000 7f49d2ffd640 1 -- 172.21.15.161:0/3280155324 learned_addr learned my addr 172.21.15.161:0/3280155324 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:43.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.200+0000 7f49d27fc640 1 --2- 172.21.15.161:0/3280155324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49d40c3760 0x7f49d41ebe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:43.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.200+0000 7f49d37fe640 1 --2- 172.21.15.161:0/3280155324 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49d414f320 0x7f49d40c2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:43.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.200+0000 7f49d2ffd640 1 -- 172.21.15.161:0/3280155324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49d40c3760 msgr2=0x7f49d41ebe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:43.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.200+0000 7f49d2ffd640 1 --2- 172.21.15.161:0/3280155324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49d40c3760 0x7f49d41ebe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.200+0000 7f49d2ffd640 1 -- 172.21.15.161:0/3280155324 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49d414f320 msgr2=0x7f49d40c2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:43.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.200+0000 7f49d2ffd640 1 --2- 172.21.15.161:0/3280155324 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49d414f320 0x7f49d40c2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.200+0000 7f49d2ffd640 1 -- 172.21.15.161:0/3280155324 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d40abc60 con 0x7f49bc004800 2024-03-28T23:31:43.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.200+0000 7f49d37fe640 1 --2- 172.21.15.161:0/3280155324 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49d414f320 0x7f49d40c2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:43.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.200+0000 7f49d2ffd640 1 --2- 172.21.15.161:0/3280155324 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49bc004800 0x7f49d40ab720 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f49cc011a00 tx=0x7f49cc011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:43.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.200+0000 7f49bb7fe640 1 -- 172.21.15.161:0/3280155324 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49cc00fb50 con 0x7f49bc004800 2024-03-28T23:31:43.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.201+0000 7f49bb7fe640 1 -- 172.21.15.161:0/3280155324 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49cc00fcf0 con 0x7f49bc004800 2024-03-28T23:31:43.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.201+0000 7f49bb7fe640 1 -- 172.21.15.161:0/3280155324 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49cc00fea0 con 0x7f49bc004800 2024-03-28T23:31:43.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.201+0000 7f49d972c640 1 -- 172.21.15.161:0/3280155324 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49d40aa680 con 0x7f49bc004800 2024-03-28T23:31:43.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.201+0000 7f49d972c640 1 -- 172.21.15.161:0/3280155324 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49d40aac40 con 0x7f49bc004800 2024-03-28T23:31:43.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.202+0000 7f49d972c640 1 -- 172.21.15.161:0/3280155324 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49d404f4b0 con 0x7f49bc004800 2024-03-28T23:31:43.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.202+0000 7f49bb7fe640 1 -- 172.21.15.161:0/3280155324 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f49cc012040 con 0x7f49bc004800 2024-03-28T23:31:43.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.203+0000 7f49bb7fe640 1 --2- 172.21.15.161:0/3280155324 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f49c006b500 0x7f49c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:43.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.203+0000 7f49d27fc640 1 --2- 172.21.15.161:0/3280155324 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f49c006b500 0x7f49c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:43.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.203+0000 7f49bb7fe640 1 -- 172.21.15.161:0/3280155324 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f49cc08a8c0 con 0x7f49bc004800 2024-03-28T23:31:43.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.203+0000 7f49d27fc640 1 --2- 172.21.15.161:0/3280155324 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f49c006b500 0x7f49c006d9c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f49c8034000 tx=0x7f49c8009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:43.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.207+0000 7f49bb7fe640 1 -- 172.21.15.161:0/3280155324 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f49cc019030 con 0x7f49bc004800 2024-03-28T23:31:43.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.381+0000 7f49d972c640 1 -- 172.21.15.161:0/3280155324 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f49d40aa430 con 0x7f49c006b500 2024-03-28T23:31:43.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.387+0000 7f49bb7fe640 1 -- 172.21.15.161:0/3280155324 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+240127 (secure 0 0 0) 0x7f49d40aa430 con 0x7f49c006b500 2024-03-28T23:31:43.390 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:43.392 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:31:43.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.390+0000 7f49d972c640 1 -- 172.21.15.161:0/3280155324 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f49c006b500 msgr2=0x7f49c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:43.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.390+0000 7f49d972c640 1 --2- 172.21.15.161:0/3280155324 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f49c006b500 0x7f49c006d9c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f49c8034000 tx=0x7f49c8009030 comp rx=0 tx=0).stop 2024-03-28T23:31:43.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.390+0000 7f49d972c640 1 -- 172.21.15.161:0/3280155324 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49bc004800 msgr2=0x7f49d40ab720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:43.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.390+0000 7f49d972c640 1 --2- 172.21.15.161:0/3280155324 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49bc004800 0x7f49d40ab720 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f49cc011a00 tx=0x7f49cc011ec0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.390+0000 7f49d972c640 1 -- 172.21.15.161:0/3280155324 shutdown_connections 2024-03-28T23:31:43.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.390+0000 7f49d972c640 1 --2- 172.21.15.161:0/3280155324 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f49c006b500 0x7f49c006d9c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.390+0000 7f49d972c640 1 --2- 172.21.15.161:0/3280155324 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49d414f320 0x7f49d40c2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.390+0000 7f49d972c640 1 --2- 172.21.15.161:0/3280155324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49d40c3760 0x7f49d41ebe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.390+0000 7f49d972c640 1 --2- 172.21.15.161:0/3280155324 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49bc004800 0x7f49d40ab720 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.390+0000 7f49d972c640 1 -- 172.21.15.161:0/3280155324 >> 172.21.15.161:0/3280155324 conn(0x7f49d40ac620 msgr2=0x7f49d4150c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:43.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.390+0000 7f49d972c640 1 -- 172.21.15.161:0/3280155324 shutdown_connections 2024-03-28T23:31:43.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.390+0000 7f49d972c640 1 -- 172.21.15.161:0/3280155324 wait complete. 2024-03-28T23:31:43.403 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2024-03-28T23:31:43.333903+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":233216,"kb_used_data":4576,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754741504,"statfs":{"total":773094113280,"available":772855300096,"internally_reserved":0,"allocated":4685824,"data_stored":3179728,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.968830"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.369099+0000","last_change":"2024-03-28T23:31:38.369099+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.369099+0000","last_undegraded":"2024-03-28T23:31:38.369099+0000","last_fullsized":"2024-03-28T23:31:38.369099+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370329+0000","last_change":"2024-03-28T23:31:38.370329+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370329+0000","last_undegraded":"2024-03-28T23:31:38.370329+0000","last_fullsized":"2024-03-28T23:31:38.370329+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370328+0000","last_change":"2024-03-28T23:31:38.370328+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370328+0000","last_undegraded":"2024-03-28T23:31:38.370328+0000","last_fullsized":"2024-03-28T23:31:38.370328+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370985+0000","last_change":"2024-03-28T23:31:38.370985+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370985+0000","last_undegraded":"2024-03-28T23:31:38.370985+0000","last_fullsized":"2024-03-28T23:31:38.370985+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.372009+0000","last_change":"2024-03-28T23:31:38.372009+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.372009+0000","last_undegraded":"2024-03-28T23:31:38.372009+0000","last_fullsized":"2024-03-28T23:31:38.372009+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366496+0000","last_change":"2024-03-28T23:31:38.366496+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366496+0000","last_undegraded":"2024-03-28T23:31:38.366496+0000","last_fullsized":"2024-03-28T23:31:38.366496+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368502+0000","last_change":"2024-03-28T23:31:38.368502+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368502+0000","last_undegraded":"2024-03-28T23:31:38.368502+0000","last_fullsized":"2024-03-28T23:31:38.368502+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370603+0000","last_change":"2024-03-28T23:31:38.370603+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370603+0000","last_undegraded":"2024-03-28T23:31:38.370603+0000","last_fullsized":"2024-03-28T23:31:38.370603+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370731+0000","last_change":"2024-03-28T23:31:38.370731+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370731+0000","last_undegraded":"2024-03-28T23:31:38.370731+0000","last_fullsized":"2024-03-28T23:31:38.370731+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366578+0000","last_change":"2024-03-28T23:31:38.366578+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366578+0000","last_undegraded":"2024-03-28T23:31:38.366578+0000","last_fullsized":"2024-03-28T23:31:38.366578+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.371431+0000","last_change":"2024-03-28T23:31:38.371431+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.371431+0000","last_undegraded":"2024-03-28T23:31:38.371431+0000","last_fullsized":"2024-03-28T23:31:38.371431+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366398+0000","last_change":"2024-03-28T23:31:38.366398+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366398+0000","last_undegraded":"2024-03-28T23:31:38.366398+0000","last_fullsized":"2024-03-28T23:31:38.366398+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.371592+0000","last_change":"2024-03-28T23:31:38.371592+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.371592+0000","last_undegraded":"2024-03-28T23:31:38.371592+0000","last_fullsized":"2024-03-28T23:31:38.371592+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368491+0000","last_change":"2024-03-28T23:31:38.368491+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368491+0000","last_undegraded":"2024-03-28T23:31:38.368491+0000","last_fullsized":"2024-03-28T23:31:38.368491+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370684+0000","last_change":"2024-03-28T23:31:38.370684+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370684+0000","last_undegraded":"2024-03-28T23:31:38.370684+0000","last_fullsized":"2024-03-28T23:31:38.370684+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366432+0000","last_change":"2024-03-28T23:31:38.366432+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366432+0000","last_undegraded":"2024-03-28T23:31:38.366432+0000","last_fullsized":"2024-03-28T23:31:38.366432+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370630+0000","last_change":"2024-03-28T23:31:38.370630+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370630+0000","last_undegraded":"2024-03-28T23:31:38.370630+0000","last_fullsized":"2024-03-28T23:31:38.370630+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368798+0000","last_change":"2024-03-28T23:31:38.368798+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368798+0000","last_undegraded":"2024-03-28T23:31:38.368798+0000","last_fullsized":"2024-03-28T23:31:38.368798+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368596+0000","last_change":"2024-03-28T23:31:38.368596+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368596+0000","last_undegraded":"2024-03-28T23:31:38.368596+0000","last_fullsized":"2024-03-28T23:31:38.368596+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368582+0000","last_change":"2024-03-28T23:31:38.368582+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368582+0000","last_undegraded":"2024-03-28T23:31:38.368582+0000","last_fullsized":"2024-03-28T23:31:38.368582+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368410+0000","last_change":"2024-03-28T23:31:38.368410+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368410+0000","last_undegraded":"2024-03-28T23:31:38.368410+0000","last_fullsized":"2024-03-28T23:31:38.368410+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366810+0000","last_change":"2024-03-28T23:31:38.366810+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366810+0000","last_undegraded":"2024-03-28T23:31:38.366810+0000","last_fullsized":"2024-03-28T23:31:38.366810+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370519+0000","last_change":"2024-03-28T23:31:38.370519+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370519+0000","last_undegraded":"2024-03-28T23:31:38.370519+0000","last_fullsized":"2024-03-28T23:31:38.370519+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368438+0000","last_change":"2024-03-28T23:31:38.368438+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368438+0000","last_undegraded":"2024-03-28T23:31:38.368438+0000","last_fullsized":"2024-03-28T23:31:38.368438+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368581+0000","last_change":"2024-03-28T23:31:38.368581+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368581+0000","last_undegraded":"2024-03-28T23:31:38.368581+0000","last_fullsized":"2024-03-28T23:31:38.368581+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.371001+0000","last_change":"2024-03-28T23:31:38.371001+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.371001+0000","last_undegraded":"2024-03-28T23:31:38.371001+0000","last_fullsized":"2024-03-28T23:31:38.371001+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370719+0000","last_change":"2024-03-28T23:31:38.370719+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370719+0000","last_undegraded":"2024-03-28T23:31:38.370719+0000","last_fullsized":"2024-03-28T23:31:38.370719+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366340+0000","last_change":"2024-03-28T23:31:38.366340+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366340+0000","last_undegraded":"2024-03-28T23:31:38.366340+0000","last_fullsized":"2024-03-28T23:31:38.366340+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366341+0000","last_change":"2024-03-28T23:31:38.366341+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366341+0000","last_undegraded":"2024-03-28T23:31:38.366341+0000","last_fullsized":"2024-03-28T23:31:38.366341+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366564+0000","last_change":"2024-03-28T23:31:38.366564+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366564+0000","last_undegraded":"2024-03-28T23:31:38.366564+0000","last_fullsized":"2024-03-28T23:31:38.366564+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.371193+0000","last_change":"2024-03-28T23:31:38.371193+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.371193+0000","last_undegraded":"2024-03-28T23:31:38.371193+0000","last_fullsized":"2024-03-28T23:31:38.371193+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.371758+0000","last_change":"2024-03-28T23:31:38.371758+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.371758+0000","last_undegraded":"2024-03-28T23:31:38.371758+0000","last_fullsized":"2024-03-28T23:31:38.371758+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":222,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-28T23:31:38.371895+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:31:38.371895+0000","last_peered":"2024-03-28T23:31:38.371895+0000","last_clean":"2024-03-28T23:31:38.371895+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:31:38.371895+0000","last_undegraded":"2024-03-28T23:31:38.371895+0000","last_fullsized":"2024-03-28T23:31:38.371895+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":634880,"data_stored":840242,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31736,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340104,"statfs":{"total":96636764160,"available":96604266496,"internally_reserved":0,"allocated":569344,"data_stored":249874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":249874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31736,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340104,"statfs":{"total":96636764160,"available":96604266496,"internally_reserved":0,"allocated":569344,"data_stored":249874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":249874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":249874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":249874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":634880,"data_stored":840242,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:31:43.408 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:31:43.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.546+0000 7f8fb6ad4640 1 Processor -- start 2024-03-28T23:31:43.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.546+0000 7f8fb6ad4640 1 -- start start 2024-03-28T23:31:43.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.546+0000 7f8fb6ad4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8fb0152890 0x7f8fb00c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:43.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.546+0000 7f8fb6ad4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8fb00c8ed0 0x7f8fb00c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:43.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.546+0000 7f8fb6ad4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fb0158a30 con 0x7f8fb00c5950 2024-03-28T23:31:43.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.546+0000 7f8fb6ad4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8fb00c8480 con 0x7f8fb0152890 2024-03-28T23:31:43.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.546+0000 7f8fb6ad4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f8fb0159640 con 0x7f8fb00c8ed0 2024-03-28T23:31:43.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.547+0000 7f8fb4870640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8fb0152890 0x7f8fb00c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:43.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.547+0000 7f8fb5071640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f8fb00c5950 0x7f8fb01553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:43994/0 (socket says 172.21.15.161:43994) 2024-03-28T23:31:43.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.547+0000 7f8fb5071640 1 -- 172.21.15.161:0/3275549218 learned_addr learned my addr 172.21.15.161:0/3275549218 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:43.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.547+0000 7f8faffff640 1 --2- 172.21.15.161:0/3275549218 >> v2:172.21.15.170:3300/0 conn(0x7f8fb00c8ed0 0x7f8fb00c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:43.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.547+0000 7f8fb4870640 1 -- 172.21.15.161:0/3275549218 >> v2:172.21.15.170:3300/0 conn(0x7f8fb00c8ed0 msgr2=0x7f8fb00c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:43.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.547+0000 7f8fb4870640 1 --2- 172.21.15.161:0/3275549218 >> v2:172.21.15.170:3300/0 conn(0x7f8fb00c8ed0 0x7f8fb00c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.547+0000 7f8fb4870640 1 -- 172.21.15.161:0/3275549218 >> v1:172.21.15.161:6789/0 conn(0x7f8fb00c5950 legacy=0x7f8fb01553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:43.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.547+0000 7f8faf7fe640 1 -- 172.21.15.161:0/3275549218 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2711626675 0 0) 0x7f8fb0158a30 con 0x7f8fb00c5950 2024-03-28T23:31:43.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.547+0000 7f8fb4870640 1 -- 172.21.15.161:0/3275549218 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fb0158d10 con 0x7f8fb0152890 2024-03-28T23:31:43.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.548+0000 7f8fb4870640 1 --2- 172.21.15.161:0/3275549218 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fb0152890 0x7f8fb00c8990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f8fb00ad6e0 tx=0x7f8fa802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c58424b7b707d1c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:43.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.548+0000 7f8faf7fe640 1 -- 172.21.15.161:0/3275549218 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa8039070 con 0x7f8fb0152890 2024-03-28T23:31:43.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.548+0000 7f8faf7fe640 1 -- 172.21.15.161:0/3275549218 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fa802cb60 con 0x7f8fb0152890 2024-03-28T23:31:43.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.548+0000 7f8faf7fe640 1 -- 172.21.15.161:0/3275549218 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa8034440 con 0x7f8fb0152890 2024-03-28T23:31:43.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.548+0000 7f8fb6ad4640 1 -- 172.21.15.161:0/3275549218 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fb0152890 msgr2=0x7f8fb00c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:43.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.548+0000 7f8fb6ad4640 1 --2- 172.21.15.161:0/3275549218 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fb0152890 0x7f8fb00c8990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f8fb00ad6e0 tx=0x7f8fa802bdb0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.548+0000 7f8fb6ad4640 1 -- 172.21.15.161:0/3275549218 shutdown_connections 2024-03-28T23:31:43.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.548+0000 7f8fb6ad4640 1 --2- 172.21.15.161:0/3275549218 >> v2:172.21.15.170:3300/0 conn(0x7f8fb00c8ed0 0x7f8fb00c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.548+0000 7f8fb6ad4640 1 --2- 172.21.15.161:0/3275549218 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fb0152890 0x7f8fb00c8990 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.548+0000 7f8fb6ad4640 1 -- 172.21.15.161:0/3275549218 >> 172.21.15.161:0/3275549218 conn(0x7f8fb00ac620 msgr2=0x7f8fb00aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:43.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.548+0000 7f8fb6ad4640 1 -- 172.21.15.161:0/3275549218 shutdown_connections 2024-03-28T23:31:43.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.548+0000 7f8fb6ad4640 1 -- 172.21.15.161:0/3275549218 wait complete. 2024-03-28T23:31:43.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.549+0000 7f8fb6ad4640 1 Processor -- start 2024-03-28T23:31:43.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.549+0000 7f8fb6ad4640 1 -- start start 2024-03-28T23:31:43.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.550+0000 7f8fb6ad4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8fb00c5950 0x7f8fb01e7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:43.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.550+0000 7f8fb6ad4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fb00c8ed0 0x7f8fb00c3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:43.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.550+0000 7f8fb6ad4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8fb0152890 0x7f8fb00c4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:43.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.550+0000 7f8fb6ad4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb0152c70 con 0x7f8fb00c8ed0 2024-03-28T23:31:43.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.550+0000 7f8fb6ad4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb01592d0 con 0x7f8fb00c5950 2024-03-28T23:31:43.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.551+0000 7f8fb6ad4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8fb0152df0 con 0x7f8fb0152890 2024-03-28T23:31:43.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.551+0000 7f8faffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fb00c8ed0 0x7f8fb00c3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:43.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.551+0000 7f8faffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fb00c8ed0 0x7f8fb00c3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58412/0 (socket says 172.21.15.161:58412) 2024-03-28T23:31:43.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.551+0000 7f8faffff640 1 -- 172.21.15.161:0/2025811261 learned_addr learned my addr 172.21.15.161:0/2025811261 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:43.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.551+0000 7f8fb4870640 1 --2- 172.21.15.161:0/2025811261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8fb00c5950 0x7f8fb01e7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:43.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.551+0000 7f8fb5071640 1 --2- 172.21.15.161:0/2025811261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8fb0152890 0x7f8fb00c4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:43.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.551+0000 7f8faffff640 1 -- 172.21.15.161:0/2025811261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8fb0152890 msgr2=0x7f8fb00c4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:43.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.551+0000 7f8faffff640 1 --2- 172.21.15.161:0/2025811261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8fb0152890 0x7f8fb00c4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.551+0000 7f8faffff640 1 -- 172.21.15.161:0/2025811261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8fb00c5950 msgr2=0x7f8fb01e7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:43.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.551+0000 7f8faffff640 1 --2- 172.21.15.161:0/2025811261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8fb00c5950 0x7f8fb01e7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.551+0000 7f8faffff640 1 -- 172.21.15.161:0/2025811261 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fb00c49c0 con 0x7f8fb00c8ed0 2024-03-28T23:31:43.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.551+0000 7f8fb4870640 1 --2- 172.21.15.161:0/2025811261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8fb00c5950 0x7f8fb01e7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:43.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.551+0000 7f8faffff640 1 --2- 172.21.15.161:0/2025811261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fb00c8ed0 0x7f8fb00c3b00 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800d800 tx=0x7f8f9800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:43.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.551+0000 7f8fad7fa640 1 -- 172.21.15.161:0/2025811261 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f98004030 con 0x7f8fb00c8ed0 2024-03-28T23:31:43.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.551+0000 7f8fad7fa640 1 -- 172.21.15.161:0/2025811261 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f980041d0 con 0x7f8fb00c8ed0 2024-03-28T23:31:43.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.551+0000 7f8fb6ad4640 1 -- 172.21.15.161:0/2025811261 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fb00c4ca0 con 0x7f8fb00c8ed0 2024-03-28T23:31:43.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.551+0000 7f8fad7fa640 1 -- 172.21.15.161:0/2025811261 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f9800b6c0 con 0x7f8fb00c8ed0 2024-03-28T23:31:43.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.551+0000 7f8fb6ad4640 1 -- 172.21.15.161:0/2025811261 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fb01eebe0 con 0x7f8fb00c8ed0 2024-03-28T23:31:43.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.552+0000 7f8fb6ad4640 1 -- 172.21.15.161:0/2025811261 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fb004f4b0 con 0x7f8fb00c8ed0 2024-03-28T23:31:43.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.553+0000 7f8fad7fa640 1 -- 172.21.15.161:0/2025811261 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8f9800b860 con 0x7f8fb00c8ed0 2024-03-28T23:31:43.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.553+0000 7f8fad7fa640 1 --2- 172.21.15.161:0/2025811261 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8f9006b4d0 0x7f8f9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:43.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.553+0000 7f8fad7fa640 1 -- 172.21.15.161:0/2025811261 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f8f9808a630 con 0x7f8fb00c8ed0 2024-03-28T23:31:43.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.553+0000 7f8fb4870640 1 --2- 172.21.15.161:0/2025811261 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8f9006b4d0 0x7f8f9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:43.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.554+0000 7f8fb4870640 1 --2- 172.21.15.161:0/2025811261 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8f9006b4d0 0x7f8f9006d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8036000 tx=0x7f8fa8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:43.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.558+0000 7f8fad7fa640 1 -- 172.21.15.161:0/2025811261 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8f9801c360 con 0x7f8fb00c8ed0 2024-03-28T23:31:43.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.733+0000 7f8fb6ad4640 1 -- 172.21.15.161:0/2025811261 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8fb00c8140 con 0x7f8f9006b4d0 2024-03-28T23:31:43.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.738+0000 7f8fad7fa640 1 -- 172.21.15.161:0/2025811261 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+240127 (secure 0 0 0) 0x7f8fb00c8140 con 0x7f8f9006b4d0 2024-03-28T23:31:43.742 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:43.743 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:31:43.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.741+0000 7f8fb6ad4640 1 -- 172.21.15.161:0/2025811261 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8f9006b4d0 msgr2=0x7f8f9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:43.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.741+0000 7f8fb6ad4640 1 --2- 172.21.15.161:0/2025811261 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8f9006b4d0 0x7f8f9006d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8036000 tx=0x7f8fa8030000 comp rx=0 tx=0).stop 2024-03-28T23:31:43.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.741+0000 7f8fb6ad4640 1 -- 172.21.15.161:0/2025811261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fb00c8ed0 msgr2=0x7f8fb00c3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:43.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.741+0000 7f8fb6ad4640 1 --2- 172.21.15.161:0/2025811261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fb00c8ed0 0x7f8fb00c3b00 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800d800 tx=0x7f8f9800dcc0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.741+0000 7f8fb6ad4640 1 -- 172.21.15.161:0/2025811261 shutdown_connections 2024-03-28T23:31:43.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.741+0000 7f8fb6ad4640 1 --2- 172.21.15.161:0/2025811261 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8f9006b4d0 0x7f8f9006d990 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.741+0000 7f8fb6ad4640 1 --2- 172.21.15.161:0/2025811261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fb00c8ed0 0x7f8fb00c3b00 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.741+0000 7f8fb6ad4640 1 --2- 172.21.15.161:0/2025811261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8fb0152890 0x7f8fb00c4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.741+0000 7f8fb6ad4640 1 --2- 172.21.15.161:0/2025811261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8fb00c5950 0x7f8fb01e7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.741+0000 7f8fb6ad4640 1 -- 172.21.15.161:0/2025811261 >> 172.21.15.161:0/2025811261 conn(0x7f8fb00ac620 msgr2=0x7f8fb006c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:43.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.742+0000 7f8fb6ad4640 1 -- 172.21.15.161:0/2025811261 shutdown_connections 2024-03-28T23:31:43.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.742+0000 7f8fb6ad4640 1 -- 172.21.15.161:0/2025811261 wait complete. 2024-03-28T23:31:43.753 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2024-03-28T23:31:43.333903+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":233216,"kb_used_data":4576,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754741504,"statfs":{"total":773094113280,"available":772855300096,"internally_reserved":0,"allocated":4685824,"data_stored":3179728,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.968830"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.369099+0000","last_change":"2024-03-28T23:31:38.369099+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.369099+0000","last_undegraded":"2024-03-28T23:31:38.369099+0000","last_fullsized":"2024-03-28T23:31:38.369099+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370329+0000","last_change":"2024-03-28T23:31:38.370329+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370329+0000","last_undegraded":"2024-03-28T23:31:38.370329+0000","last_fullsized":"2024-03-28T23:31:38.370329+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370328+0000","last_change":"2024-03-28T23:31:38.370328+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370328+0000","last_undegraded":"2024-03-28T23:31:38.370328+0000","last_fullsized":"2024-03-28T23:31:38.370328+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370985+0000","last_change":"2024-03-28T23:31:38.370985+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370985+0000","last_undegraded":"2024-03-28T23:31:38.370985+0000","last_fullsized":"2024-03-28T23:31:38.370985+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.372009+0000","last_change":"2024-03-28T23:31:38.372009+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.372009+0000","last_undegraded":"2024-03-28T23:31:38.372009+0000","last_fullsized":"2024-03-28T23:31:38.372009+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366496+0000","last_change":"2024-03-28T23:31:38.366496+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366496+0000","last_undegraded":"2024-03-28T23:31:38.366496+0000","last_fullsized":"2024-03-28T23:31:38.366496+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368502+0000","last_change":"2024-03-28T23:31:38.368502+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368502+0000","last_undegraded":"2024-03-28T23:31:38.368502+0000","last_fullsized":"2024-03-28T23:31:38.368502+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370603+0000","last_change":"2024-03-28T23:31:38.370603+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370603+0000","last_undegraded":"2024-03-28T23:31:38.370603+0000","last_fullsized":"2024-03-28T23:31:38.370603+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370731+0000","last_change":"2024-03-28T23:31:38.370731+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370731+0000","last_undegraded":"2024-03-28T23:31:38.370731+0000","last_fullsized":"2024-03-28T23:31:38.370731+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366578+0000","last_change":"2024-03-28T23:31:38.366578+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366578+0000","last_undegraded":"2024-03-28T23:31:38.366578+0000","last_fullsized":"2024-03-28T23:31:38.366578+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.371431+0000","last_change":"2024-03-28T23:31:38.371431+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.371431+0000","last_undegraded":"2024-03-28T23:31:38.371431+0000","last_fullsized":"2024-03-28T23:31:38.371431+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366398+0000","last_change":"2024-03-28T23:31:38.366398+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366398+0000","last_undegraded":"2024-03-28T23:31:38.366398+0000","last_fullsized":"2024-03-28T23:31:38.366398+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.371592+0000","last_change":"2024-03-28T23:31:38.371592+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.371592+0000","last_undegraded":"2024-03-28T23:31:38.371592+0000","last_fullsized":"2024-03-28T23:31:38.371592+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368491+0000","last_change":"2024-03-28T23:31:38.368491+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368491+0000","last_undegraded":"2024-03-28T23:31:38.368491+0000","last_fullsized":"2024-03-28T23:31:38.368491+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370684+0000","last_change":"2024-03-28T23:31:38.370684+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370684+0000","last_undegraded":"2024-03-28T23:31:38.370684+0000","last_fullsized":"2024-03-28T23:31:38.370684+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366432+0000","last_change":"2024-03-28T23:31:38.366432+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366432+0000","last_undegraded":"2024-03-28T23:31:38.366432+0000","last_fullsized":"2024-03-28T23:31:38.366432+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370630+0000","last_change":"2024-03-28T23:31:38.370630+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370630+0000","last_undegraded":"2024-03-28T23:31:38.370630+0000","last_fullsized":"2024-03-28T23:31:38.370630+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368798+0000","last_change":"2024-03-28T23:31:38.368798+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368798+0000","last_undegraded":"2024-03-28T23:31:38.368798+0000","last_fullsized":"2024-03-28T23:31:38.368798+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368596+0000","last_change":"2024-03-28T23:31:38.368596+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368596+0000","last_undegraded":"2024-03-28T23:31:38.368596+0000","last_fullsized":"2024-03-28T23:31:38.368596+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368582+0000","last_change":"2024-03-28T23:31:38.368582+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368582+0000","last_undegraded":"2024-03-28T23:31:38.368582+0000","last_fullsized":"2024-03-28T23:31:38.368582+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368410+0000","last_change":"2024-03-28T23:31:38.368410+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368410+0000","last_undegraded":"2024-03-28T23:31:38.368410+0000","last_fullsized":"2024-03-28T23:31:38.368410+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366810+0000","last_change":"2024-03-28T23:31:38.366810+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366810+0000","last_undegraded":"2024-03-28T23:31:38.366810+0000","last_fullsized":"2024-03-28T23:31:38.366810+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370519+0000","last_change":"2024-03-28T23:31:38.370519+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370519+0000","last_undegraded":"2024-03-28T23:31:38.370519+0000","last_fullsized":"2024-03-28T23:31:38.370519+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368438+0000","last_change":"2024-03-28T23:31:38.368438+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368438+0000","last_undegraded":"2024-03-28T23:31:38.368438+0000","last_fullsized":"2024-03-28T23:31:38.368438+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368581+0000","last_change":"2024-03-28T23:31:38.368581+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368581+0000","last_undegraded":"2024-03-28T23:31:38.368581+0000","last_fullsized":"2024-03-28T23:31:38.368581+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.371001+0000","last_change":"2024-03-28T23:31:38.371001+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.371001+0000","last_undegraded":"2024-03-28T23:31:38.371001+0000","last_fullsized":"2024-03-28T23:31:38.371001+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370719+0000","last_change":"2024-03-28T23:31:38.370719+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370719+0000","last_undegraded":"2024-03-28T23:31:38.370719+0000","last_fullsized":"2024-03-28T23:31:38.370719+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366340+0000","last_change":"2024-03-28T23:31:38.366340+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366340+0000","last_undegraded":"2024-03-28T23:31:38.366340+0000","last_fullsized":"2024-03-28T23:31:38.366340+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366341+0000","last_change":"2024-03-28T23:31:38.366341+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366341+0000","last_undegraded":"2024-03-28T23:31:38.366341+0000","last_fullsized":"2024-03-28T23:31:38.366341+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366564+0000","last_change":"2024-03-28T23:31:38.366564+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366564+0000","last_undegraded":"2024-03-28T23:31:38.366564+0000","last_fullsized":"2024-03-28T23:31:38.366564+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.371193+0000","last_change":"2024-03-28T23:31:38.371193+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.371193+0000","last_undegraded":"2024-03-28T23:31:38.371193+0000","last_fullsized":"2024-03-28T23:31:38.371193+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.371758+0000","last_change":"2024-03-28T23:31:38.371758+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.371758+0000","last_undegraded":"2024-03-28T23:31:38.371758+0000","last_fullsized":"2024-03-28T23:31:38.371758+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":222,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-28T23:31:38.371895+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:31:38.371895+0000","last_peered":"2024-03-28T23:31:38.371895+0000","last_clean":"2024-03-28T23:31:38.371895+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:31:38.371895+0000","last_undegraded":"2024-03-28T23:31:38.371895+0000","last_fullsized":"2024-03-28T23:31:38.371895+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":634880,"data_stored":840242,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31736,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340104,"statfs":{"total":96636764160,"available":96604266496,"internally_reserved":0,"allocated":569344,"data_stored":249874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":249874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31736,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340104,"statfs":{"total":96636764160,"available":96604266496,"internally_reserved":0,"allocated":569344,"data_stored":249874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":249874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":249874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":249874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":634880,"data_stored":840242,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:31:43.757 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:31:43.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.895+0000 7f0b2cb7a640 1 Processor -- start 2024-03-28T23:31:43.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.895+0000 7f0b2cb7a640 1 -- start start 2024-03-28T23:31:43.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.896+0000 7f0b2cb7a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0b280c5410 0x7f0b280c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:43.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.896+0000 7f0b2cb7a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b28158a30 con 0x7f0b281553a0 2024-03-28T23:31:43.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.896+0000 7f0b2cb7a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b28158d10 con 0x7f0b28152890 2024-03-28T23:31:43.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.896+0000 7f0b2cb7a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0b280c8480 con 0x7f0b280c5410 2024-03-28T23:31:43.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.896+0000 7f0b1dd9b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0b280c5410 0x7f0b280c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:43.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.896+0000 7f0b1dd9b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0b280c5410 0x7f0b280c5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58424/0 (socket says 172.21.15.161:58424) 2024-03-28T23:31:43.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.896+0000 7f0b26d9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f0b281553a0 0x7f0b28155850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44008/0 (socket says 172.21.15.161:44008) 2024-03-28T23:31:43.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.896+0000 7f0b1dd9b640 1 -- 172.21.15.161:0/3625734735 learned_addr learned my addr 172.21.15.161:0/3625734735 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:43.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.896+0000 7f0b1dd9b640 1 -- 172.21.15.161:0/3625734735 >> v1:172.21.15.170:6789/0 conn(0x7f0b28152890 legacy=0x7f0b280c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:43.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.897+0000 7f0b1dd9b640 1 -- 172.21.15.161:0/3625734735 >> v1:172.21.15.161:6789/0 conn(0x7f0b281553a0 legacy=0x7f0b28155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:43.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.897+0000 7f0b1dd9b640 1 -- 172.21.15.161:0/3625734735 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b28158ff0 con 0x7f0b280c5410 2024-03-28T23:31:43.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.897+0000 7f0b25d9b640 1 -- 172.21.15.161:0/3625734735 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 864188843 0 0) 0x7f0b28158a30 con 0x7f0b281553a0 2024-03-28T23:31:43.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.898+0000 7f0b1dd9b640 1 --2- 172.21.15.161:0/3625734735 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b280c5410 0x7f0b280c5850 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f0b10009850 tx=0x7f0b1002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=221700f6a0251105 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:43.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.898+0000 7f0b25d9b640 1 -- 172.21.15.161:0/3625734735 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b10004230 con 0x7f0b280c5410 2024-03-28T23:31:43.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.898+0000 7f0b25d9b640 1 -- 172.21.15.161:0/3625734735 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b100043d0 con 0x7f0b280c5410 2024-03-28T23:31:43.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.898+0000 7f0b25d9b640 1 -- 172.21.15.161:0/3625734735 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b10030840 con 0x7f0b280c5410 2024-03-28T23:31:43.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.898+0000 7f0b2cb7a640 1 -- 172.21.15.161:0/3625734735 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b280c5410 msgr2=0x7f0b280c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:43.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.898+0000 7f0b2cb7a640 1 --2- 172.21.15.161:0/3625734735 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b280c5410 0x7f0b280c5850 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f0b10009850 tx=0x7f0b1002b980 comp rx=0 tx=0).stop 2024-03-28T23:31:43.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.898+0000 7f0b2cb7a640 1 -- 172.21.15.161:0/3625734735 shutdown_connections 2024-03-28T23:31:43.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.898+0000 7f0b2cb7a640 1 --2- 172.21.15.161:0/3625734735 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b280c5410 0x7f0b280c5850 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.898+0000 7f0b2cb7a640 1 -- 172.21.15.161:0/3625734735 >> 172.21.15.161:0/3625734735 conn(0x7f0b280ac620 msgr2=0x7f0b280aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:43.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.898+0000 7f0b2cb7a640 1 -- 172.21.15.161:0/3625734735 shutdown_connections 2024-03-28T23:31:43.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.898+0000 7f0b2cb7a640 1 -- 172.21.15.161:0/3625734735 wait complete. 2024-03-28T23:31:43.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.900+0000 7f0b2cb7a640 1 Processor -- start 2024-03-28T23:31:43.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.900+0000 7f0b2cb7a640 1 -- start start 2024-03-28T23:31:43.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.900+0000 7f0b2cb7a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b280c5410 0x7f0b2814cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:43.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.900+0000 7f0b2cb7a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0b28152890 0x7f0b28157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:43.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.900+0000 7f0b2cb7a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b281553a0 0x7f0b28148a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:43.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.900+0000 7f0b2cb7a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0b28159640 con 0x7f0b281553a0 2024-03-28T23:31:43.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.900+0000 7f0b2cb7a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0b280c5d90 con 0x7f0b280c5410 2024-03-28T23:31:43.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.900+0000 7f0b2cb7a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0b280c90a0 con 0x7f0b28152890 2024-03-28T23:31:43.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.901+0000 7f0b26d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b281553a0 0x7f0b28148a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:43.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.901+0000 7f0b26d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b281553a0 0x7f0b28148a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58436/0 (socket says 172.21.15.161:58436) 2024-03-28T23:31:43.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.901+0000 7f0b26d9d640 1 -- 172.21.15.161:0/690931120 learned_addr learned my addr 172.21.15.161:0/690931120 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:43.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.901+0000 7f0b2659c640 1 --2- 172.21.15.161:0/690931120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b280c5410 0x7f0b2814cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:43.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.901+0000 7f0b1dd9b640 1 --2- 172.21.15.161:0/690931120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0b28152890 0x7f0b28157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:43.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.901+0000 7f0b26d9d640 1 -- 172.21.15.161:0/690931120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0b28152890 msgr2=0x7f0b28157a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:43.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.901+0000 7f0b26d9d640 1 --2- 172.21.15.161:0/690931120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0b28152890 0x7f0b28157a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.901+0000 7f0b26d9d640 1 -- 172.21.15.161:0/690931120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b280c5410 msgr2=0x7f0b2814cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:43.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.901+0000 7f0b26d9d640 1 --2- 172.21.15.161:0/690931120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b280c5410 0x7f0b2814cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:43.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.901+0000 7f0b26d9d640 1 -- 172.21.15.161:0/690931120 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b281ef000 con 0x7f0b281553a0 2024-03-28T23:31:43.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.901+0000 7f0b1dd9b640 1 --2- 172.21.15.161:0/690931120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0b28152890 0x7f0b28157a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:43.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.901+0000 7f0b26d9d640 1 --2- 172.21.15.161:0/690931120 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b281553a0 0x7f0b28148a60 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000c7a0 tx=0x7f0b2000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:43.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.901+0000 7f0b1f7fe640 1 -- 172.21.15.161:0/690931120 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b20018070 con 0x7f0b281553a0 2024-03-28T23:31:43.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.901+0000 7f0b1f7fe640 1 -- 172.21.15.161:0/690931120 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b2000a5e0 con 0x7f0b281553a0 2024-03-28T23:31:43.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.901+0000 7f0b1f7fe640 1 -- 172.21.15.161:0/690931120 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2000aa50 con 0x7f0b281553a0 2024-03-28T23:31:43.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.901+0000 7f0b2cb7a640 1 -- 172.21.15.161:0/690931120 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b281ee480 con 0x7f0b281553a0 2024-03-28T23:31:43.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.901+0000 7f0b2cb7a640 1 -- 172.21.15.161:0/690931120 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b281eea40 con 0x7f0b281553a0 2024-03-28T23:31:43.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.902+0000 7f0b2cb7a640 1 -- 172.21.15.161:0/690931120 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b280c6660 con 0x7f0b281553a0 2024-03-28T23:31:43.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.903+0000 7f0b1f7fe640 1 -- 172.21.15.161:0/690931120 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0b2001b070 con 0x7f0b281553a0 2024-03-28T23:31:43.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.903+0000 7f0b1f7fe640 1 --2- 172.21.15.161:0/690931120 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0b0806b500 0x7f0b0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:43.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.903+0000 7f0b1f7fe640 1 -- 172.21.15.161:0/690931120 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f0b200897b0 con 0x7f0b281553a0 2024-03-28T23:31:43.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.903+0000 7f0b2659c640 1 --2- 172.21.15.161:0/690931120 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0b0806b500 0x7f0b0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:43.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.904+0000 7f0b2659c640 1 --2- 172.21.15.161:0/690931120 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0b0806b500 0x7f0b0806d9c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0b18000f60 tx=0x7f0b18009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:43.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:43.908+0000 7f0b1f7fe640 1 -- 172.21.15.161:0/690931120 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0b280c6660 con 0x7f0b281553a0 2024-03-28T23:31:44.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.082+0000 7f0b2cb7a640 1 -- 172.21.15.161:0/690931120 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0b2806d0a0 con 0x7f0b0806b500 2024-03-28T23:31:44.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.092+0000 7f0b1f7fe640 1 -- 172.21.15.161:0/690931120 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+240127 (secure 0 0 0) 0x7f0b2806d0a0 con 0x7f0b0806b500 2024-03-28T23:31:44.095 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:44.096 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:31:44.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.095+0000 7f0b2cb7a640 1 -- 172.21.15.161:0/690931120 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0b0806b500 msgr2=0x7f0b0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:44.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.095+0000 7f0b2cb7a640 1 --2- 172.21.15.161:0/690931120 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0b0806b500 0x7f0b0806d9c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0b18000f60 tx=0x7f0b18009040 comp rx=0 tx=0).stop 2024-03-28T23:31:44.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.095+0000 7f0b2cb7a640 1 -- 172.21.15.161:0/690931120 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b281553a0 msgr2=0x7f0b28148a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:44.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.095+0000 7f0b2cb7a640 1 --2- 172.21.15.161:0/690931120 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b281553a0 0x7f0b28148a60 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000c7a0 tx=0x7f0b2000cc60 comp rx=0 tx=0).stop 2024-03-28T23:31:44.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.095+0000 7f0b2cb7a640 1 -- 172.21.15.161:0/690931120 shutdown_connections 2024-03-28T23:31:44.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.095+0000 7f0b2cb7a640 1 --2- 172.21.15.161:0/690931120 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0b0806b500 0x7f0b0806d9c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.095+0000 7f0b2cb7a640 1 --2- 172.21.15.161:0/690931120 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b281553a0 0x7f0b28148a60 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.095+0000 7f0b2cb7a640 1 --2- 172.21.15.161:0/690931120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0b28152890 0x7f0b28157a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.095+0000 7f0b2cb7a640 1 --2- 172.21.15.161:0/690931120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b280c5410 0x7f0b2814cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.095+0000 7f0b2cb7a640 1 -- 172.21.15.161:0/690931120 >> 172.21.15.161:0/690931120 conn(0x7f0b280ac620 msgr2=0x7f0b280ca240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:44.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.095+0000 7f0b2cb7a640 1 -- 172.21.15.161:0/690931120 shutdown_connections 2024-03-28T23:31:44.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.095+0000 7f0b2cb7a640 1 -- 172.21.15.161:0/690931120 wait complete. 2024-03-28T23:31:44.107 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2024-03-28T23:31:43.333903+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":233216,"kb_used_data":4576,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754741504,"statfs":{"total":773094113280,"available":772855300096,"internally_reserved":0,"allocated":4685824,"data_stored":3179728,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.968830"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.369099+0000","last_change":"2024-03-28T23:31:38.369099+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.369099+0000","last_undegraded":"2024-03-28T23:31:38.369099+0000","last_fullsized":"2024-03-28T23:31:38.369099+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370329+0000","last_change":"2024-03-28T23:31:38.370329+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370329+0000","last_undegraded":"2024-03-28T23:31:38.370329+0000","last_fullsized":"2024-03-28T23:31:38.370329+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370328+0000","last_change":"2024-03-28T23:31:38.370328+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370328+0000","last_undegraded":"2024-03-28T23:31:38.370328+0000","last_fullsized":"2024-03-28T23:31:38.370328+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370985+0000","last_change":"2024-03-28T23:31:38.370985+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370985+0000","last_undegraded":"2024-03-28T23:31:38.370985+0000","last_fullsized":"2024-03-28T23:31:38.370985+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.372009+0000","last_change":"2024-03-28T23:31:38.372009+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.372009+0000","last_undegraded":"2024-03-28T23:31:38.372009+0000","last_fullsized":"2024-03-28T23:31:38.372009+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366496+0000","last_change":"2024-03-28T23:31:38.366496+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366496+0000","last_undegraded":"2024-03-28T23:31:38.366496+0000","last_fullsized":"2024-03-28T23:31:38.366496+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368502+0000","last_change":"2024-03-28T23:31:38.368502+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368502+0000","last_undegraded":"2024-03-28T23:31:38.368502+0000","last_fullsized":"2024-03-28T23:31:38.368502+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370603+0000","last_change":"2024-03-28T23:31:38.370603+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370603+0000","last_undegraded":"2024-03-28T23:31:38.370603+0000","last_fullsized":"2024-03-28T23:31:38.370603+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370731+0000","last_change":"2024-03-28T23:31:38.370731+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370731+0000","last_undegraded":"2024-03-28T23:31:38.370731+0000","last_fullsized":"2024-03-28T23:31:38.370731+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366578+0000","last_change":"2024-03-28T23:31:38.366578+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366578+0000","last_undegraded":"2024-03-28T23:31:38.366578+0000","last_fullsized":"2024-03-28T23:31:38.366578+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.371431+0000","last_change":"2024-03-28T23:31:38.371431+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.371431+0000","last_undegraded":"2024-03-28T23:31:38.371431+0000","last_fullsized":"2024-03-28T23:31:38.371431+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366398+0000","last_change":"2024-03-28T23:31:38.366398+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366398+0000","last_undegraded":"2024-03-28T23:31:38.366398+0000","last_fullsized":"2024-03-28T23:31:38.366398+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.371592+0000","last_change":"2024-03-28T23:31:38.371592+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.371592+0000","last_undegraded":"2024-03-28T23:31:38.371592+0000","last_fullsized":"2024-03-28T23:31:38.371592+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368491+0000","last_change":"2024-03-28T23:31:38.368491+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368491+0000","last_undegraded":"2024-03-28T23:31:38.368491+0000","last_fullsized":"2024-03-28T23:31:38.368491+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370684+0000","last_change":"2024-03-28T23:31:38.370684+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370684+0000","last_undegraded":"2024-03-28T23:31:38.370684+0000","last_fullsized":"2024-03-28T23:31:38.370684+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366432+0000","last_change":"2024-03-28T23:31:38.366432+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366432+0000","last_undegraded":"2024-03-28T23:31:38.366432+0000","last_fullsized":"2024-03-28T23:31:38.366432+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370630+0000","last_change":"2024-03-28T23:31:38.370630+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370630+0000","last_undegraded":"2024-03-28T23:31:38.370630+0000","last_fullsized":"2024-03-28T23:31:38.370630+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368798+0000","last_change":"2024-03-28T23:31:38.368798+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368798+0000","last_undegraded":"2024-03-28T23:31:38.368798+0000","last_fullsized":"2024-03-28T23:31:38.368798+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368596+0000","last_change":"2024-03-28T23:31:38.368596+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368596+0000","last_undegraded":"2024-03-28T23:31:38.368596+0000","last_fullsized":"2024-03-28T23:31:38.368596+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368582+0000","last_change":"2024-03-28T23:31:38.368582+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368582+0000","last_undegraded":"2024-03-28T23:31:38.368582+0000","last_fullsized":"2024-03-28T23:31:38.368582+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368410+0000","last_change":"2024-03-28T23:31:38.368410+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368410+0000","last_undegraded":"2024-03-28T23:31:38.368410+0000","last_fullsized":"2024-03-28T23:31:38.368410+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366810+0000","last_change":"2024-03-28T23:31:38.366810+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366810+0000","last_undegraded":"2024-03-28T23:31:38.366810+0000","last_fullsized":"2024-03-28T23:31:38.366810+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370519+0000","last_change":"2024-03-28T23:31:38.370519+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370519+0000","last_undegraded":"2024-03-28T23:31:38.370519+0000","last_fullsized":"2024-03-28T23:31:38.370519+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368438+0000","last_change":"2024-03-28T23:31:38.368438+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368438+0000","last_undegraded":"2024-03-28T23:31:38.368438+0000","last_fullsized":"2024-03-28T23:31:38.368438+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368581+0000","last_change":"2024-03-28T23:31:38.368581+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368581+0000","last_undegraded":"2024-03-28T23:31:38.368581+0000","last_fullsized":"2024-03-28T23:31:38.368581+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.371001+0000","last_change":"2024-03-28T23:31:38.371001+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.371001+0000","last_undegraded":"2024-03-28T23:31:38.371001+0000","last_fullsized":"2024-03-28T23:31:38.371001+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370719+0000","last_change":"2024-03-28T23:31:38.370719+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370719+0000","last_undegraded":"2024-03-28T23:31:38.370719+0000","last_fullsized":"2024-03-28T23:31:38.370719+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366340+0000","last_change":"2024-03-28T23:31:38.366340+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366340+0000","last_undegraded":"2024-03-28T23:31:38.366340+0000","last_fullsized":"2024-03-28T23:31:38.366340+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366341+0000","last_change":"2024-03-28T23:31:38.366341+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366341+0000","last_undegraded":"2024-03-28T23:31:38.366341+0000","last_fullsized":"2024-03-28T23:31:38.366341+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366564+0000","last_change":"2024-03-28T23:31:38.366564+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366564+0000","last_undegraded":"2024-03-28T23:31:38.366564+0000","last_fullsized":"2024-03-28T23:31:38.366564+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.371193+0000","last_change":"2024-03-28T23:31:38.371193+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.371193+0000","last_undegraded":"2024-03-28T23:31:38.371193+0000","last_fullsized":"2024-03-28T23:31:38.371193+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.371758+0000","last_change":"2024-03-28T23:31:38.371758+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.371758+0000","last_undegraded":"2024-03-28T23:31:38.371758+0000","last_fullsized":"2024-03-28T23:31:38.371758+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":222,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-28T23:31:38.371895+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:31:38.371895+0000","last_peered":"2024-03-28T23:31:38.371895+0000","last_clean":"2024-03-28T23:31:38.371895+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:31:38.371895+0000","last_undegraded":"2024-03-28T23:31:38.371895+0000","last_fullsized":"2024-03-28T23:31:38.371895+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":634880,"data_stored":840242,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31736,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340104,"statfs":{"total":96636764160,"available":96604266496,"internally_reserved":0,"allocated":569344,"data_stored":249874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":249874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31736,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340104,"statfs":{"total":96636764160,"available":96604266496,"internally_reserved":0,"allocated":569344,"data_stored":249874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":249874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":249874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":249874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":634880,"data_stored":840242,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:31:44.111 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3f is not active+clean 2024-03-28T23:31:44.111 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.111 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3e is not active+clean 2024-03-28T23:31:44.111 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.111 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3d is not active+clean 2024-03-28T23:31:44.112 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.112 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3c is not active+clean 2024-03-28T23:31:44.112 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.112 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3b is not active+clean 2024-03-28T23:31:44.112 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.112 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3a is not active+clean 2024-03-28T23:31:44.112 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.112 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.39 is not active+clean 2024-03-28T23:31:44.112 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.112 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.38 is not active+clean 2024-03-28T23:31:44.112 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.113 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.37 is not active+clean 2024-03-28T23:31:44.113 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.113 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.36 is not active+clean 2024-03-28T23:31:44.113 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.113 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.35 is not active+clean 2024-03-28T23:31:44.113 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.113 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.34 is not active+clean 2024-03-28T23:31:44.113 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.113 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.33 is not active+clean 2024-03-28T23:31:44.113 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.113 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.32 is not active+clean 2024-03-28T23:31:44.113 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.114 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.31 is not active+clean 2024-03-28T23:31:44.114 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.114 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.30 is not active+clean 2024-03-28T23:31:44.114 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.114 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2f is not active+clean 2024-03-28T23:31:44.114 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.114 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2e is not active+clean 2024-03-28T23:31:44.114 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.114 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2d is not active+clean 2024-03-28T23:31:44.114 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.115 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2c is not active+clean 2024-03-28T23:31:44.115 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.115 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2b is not active+clean 2024-03-28T23:31:44.115 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.115 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2a is not active+clean 2024-03-28T23:31:44.115 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.115 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.29 is not active+clean 2024-03-28T23:31:44.115 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.115 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.28 is not active+clean 2024-03-28T23:31:44.115 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.115 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.27 is not active+clean 2024-03-28T23:31:44.115 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.116 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.26 is not active+clean 2024-03-28T23:31:44.116 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.116 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.25 is not active+clean 2024-03-28T23:31:44.116 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.116 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.24 is not active+clean 2024-03-28T23:31:44.116 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.116 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.23 is not active+clean 2024-03-28T23:31:44.116 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.116 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.22 is not active+clean 2024-03-28T23:31:44.116 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.116 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.21 is not active+clean 2024-03-28T23:31:44.117 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.117 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.20 is not active+clean 2024-03-28T23:31:44.117 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.117 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.10 is not active+clean 2024-03-28T23:31:44.117 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.117 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1f is not active+clean 2024-03-28T23:31:44.117 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.369099+0000', 'last_change': '2024-03-28T23:31:38.369099+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.369099+0000', 'last_undegraded': '2024-03-28T23:31:38.369099+0000', 'last_fullsized': '2024-03-28T23:31:38.369099+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:31:44.117 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.11 is not active+clean 2024-03-28T23:31:44.117 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.117 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1e is not active+clean 2024-03-28T23:31:44.117 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.370329+0000', 'last_change': '2024-03-28T23:31:38.370329+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.370329+0000', 'last_undegraded': '2024-03-28T23:31:38.370329+0000', 'last_fullsized': '2024-03-28T23:31:38.370329+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-28T23:31:44.118 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.12 is not active+clean 2024-03-28T23:31:44.118 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.118 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1d is not active+clean 2024-03-28T23:31:44.118 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.370328+0000', 'last_change': '2024-03-28T23:31:38.370328+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.370328+0000', 'last_undegraded': '2024-03-28T23:31:38.370328+0000', 'last_fullsized': '2024-03-28T23:31:38.370328+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-28T23:31:44.118 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.13 is not active+clean 2024-03-28T23:31:44.118 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.118 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1c is not active+clean 2024-03-28T23:31:44.118 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.370985+0000', 'last_change': '2024-03-28T23:31:38.370985+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.370985+0000', 'last_undegraded': '2024-03-28T23:31:38.370985+0000', 'last_fullsized': '2024-03-28T23:31:38.370985+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-28T23:31:44.118 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.14 is not active+clean 2024-03-28T23:31:44.118 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.118 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1b is not active+clean 2024-03-28T23:31:44.119 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.372009+0000', 'last_change': '2024-03-28T23:31:38.372009+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.372009+0000', 'last_undegraded': '2024-03-28T23:31:38.372009+0000', 'last_fullsized': '2024-03-28T23:31:38.372009+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:31:44.119 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.15 is not active+clean 2024-03-28T23:31:44.119 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.119 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1a is not active+clean 2024-03-28T23:31:44.119 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.366496+0000', 'last_change': '2024-03-28T23:31:38.366496+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.366496+0000', 'last_undegraded': '2024-03-28T23:31:38.366496+0000', 'last_fullsized': '2024-03-28T23:31:38.366496+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:31:44.119 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.16 is not active+clean 2024-03-28T23:31:44.119 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.119 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.19 is not active+clean 2024-03-28T23:31:44.119 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.368502+0000', 'last_change': '2024-03-28T23:31:38.368502+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.368502+0000', 'last_undegraded': '2024-03-28T23:31:38.368502+0000', 'last_fullsized': '2024-03-28T23:31:38.368502+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:31:44.119 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.17 is not active+clean 2024-03-28T23:31:44.120 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.120 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.18 is not active+clean 2024-03-28T23:31:44.120 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.370603+0000', 'last_change': '2024-03-28T23:31:38.370603+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.370603+0000', 'last_undegraded': '2024-03-28T23:31:38.370603+0000', 'last_fullsized': '2024-03-28T23:31:38.370603+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:31:44.120 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.18 is not active+clean 2024-03-28T23:31:44.120 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.120 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.17 is not active+clean 2024-03-28T23:31:44.120 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.370731+0000', 'last_change': '2024-03-28T23:31:38.370731+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.370731+0000', 'last_undegraded': '2024-03-28T23:31:38.370731+0000', 'last_fullsized': '2024-03-28T23:31:38.370731+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-28T23:31:44.120 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.19 is not active+clean 2024-03-28T23:31:44.120 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.120 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.16 is not active+clean 2024-03-28T23:31:44.120 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.366578+0000', 'last_change': '2024-03-28T23:31:38.366578+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.366578+0000', 'last_undegraded': '2024-03-28T23:31:38.366578+0000', 'last_fullsized': '2024-03-28T23:31:38.366578+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:31:44.120 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1a is not active+clean 2024-03-28T23:31:44.121 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.121 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.15 is not active+clean 2024-03-28T23:31:44.121 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.371431+0000', 'last_change': '2024-03-28T23:31:38.371431+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.371431+0000', 'last_undegraded': '2024-03-28T23:31:38.371431+0000', 'last_fullsized': '2024-03-28T23:31:38.371431+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-28T23:31:44.121 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1b is not active+clean 2024-03-28T23:31:44.121 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.121 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.14 is not active+clean 2024-03-28T23:31:44.121 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.366398+0000', 'last_change': '2024-03-28T23:31:38.366398+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.366398+0000', 'last_undegraded': '2024-03-28T23:31:38.366398+0000', 'last_fullsized': '2024-03-28T23:31:38.366398+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-28T23:31:44.121 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1c is not active+clean 2024-03-28T23:31:44.121 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.121 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.13 is not active+clean 2024-03-28T23:31:44.121 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.371592+0000', 'last_change': '2024-03-28T23:31:38.371592+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.371592+0000', 'last_undegraded': '2024-03-28T23:31:38.371592+0000', 'last_fullsized': '2024-03-28T23:31:38.371592+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:31:44.122 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1d is not active+clean 2024-03-28T23:31:44.122 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.122 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.12 is not active+clean 2024-03-28T23:31:44.122 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.368491+0000', 'last_change': '2024-03-28T23:31:38.368491+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.368491+0000', 'last_undegraded': '2024-03-28T23:31:38.368491+0000', 'last_fullsized': '2024-03-28T23:31:38.368491+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:31:44.122 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1e is not active+clean 2024-03-28T23:31:44.122 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.122 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.11 is not active+clean 2024-03-28T23:31:44.122 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.370684+0000', 'last_change': '2024-03-28T23:31:38.370684+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.370684+0000', 'last_undegraded': '2024-03-28T23:31:38.370684+0000', 'last_fullsized': '2024-03-28T23:31:38.370684+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-28T23:31:44.122 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1f is not active+clean 2024-03-28T23:31:44.122 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.122 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.10 is not active+clean 2024-03-28T23:31:44.123 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.366432+0000', 'last_change': '2024-03-28T23:31:38.366432+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.366432+0000', 'last_undegraded': '2024-03-28T23:31:38.366432+0000', 'last_fullsized': '2024-03-28T23:31:38.366432+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-28T23:31:44.123 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.0 is not active+clean 2024-03-28T23:31:44.123 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.123 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.f is not active+clean 2024-03-28T23:31:44.123 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.370630+0000', 'last_change': '2024-03-28T23:31:38.370630+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.370630+0000', 'last_undegraded': '2024-03-28T23:31:38.370630+0000', 'last_fullsized': '2024-03-28T23:31:38.370630+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:31:44.123 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1 is not active+clean 2024-03-28T23:31:44.123 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.123 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.e is not active+clean 2024-03-28T23:31:44.123 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.368798+0000', 'last_change': '2024-03-28T23:31:38.368798+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.368798+0000', 'last_undegraded': '2024-03-28T23:31:38.368798+0000', 'last_fullsized': '2024-03-28T23:31:38.368798+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-28T23:31:44.123 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2 is not active+clean 2024-03-28T23:31:44.123 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.123 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.d is not active+clean 2024-03-28T23:31:44.124 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.368596+0000', 'last_change': '2024-03-28T23:31:38.368596+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.368596+0000', 'last_undegraded': '2024-03-28T23:31:38.368596+0000', 'last_fullsized': '2024-03-28T23:31:38.368596+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:31:44.124 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3 is not active+clean 2024-03-28T23:31:44.124 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.124 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.c is not active+clean 2024-03-28T23:31:44.124 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.368582+0000', 'last_change': '2024-03-28T23:31:38.368582+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.368582+0000', 'last_undegraded': '2024-03-28T23:31:38.368582+0000', 'last_fullsized': '2024-03-28T23:31:38.368582+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:31:44.124 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.4 is not active+clean 2024-03-28T23:31:44.124 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.124 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.b is not active+clean 2024-03-28T23:31:44.124 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.368410+0000', 'last_change': '2024-03-28T23:31:38.368410+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.368410+0000', 'last_undegraded': '2024-03-28T23:31:38.368410+0000', 'last_fullsized': '2024-03-28T23:31:38.368410+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-28T23:31:44.124 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.5 is not active+clean 2024-03-28T23:31:44.124 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.124 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.a is not active+clean 2024-03-28T23:31:44.124 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.366810+0000', 'last_change': '2024-03-28T23:31:38.366810+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.366810+0000', 'last_undegraded': '2024-03-28T23:31:38.366810+0000', 'last_fullsized': '2024-03-28T23:31:38.366810+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-28T23:31:44.125 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.6 is not active+clean 2024-03-28T23:31:44.125 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.125 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.9 is not active+clean 2024-03-28T23:31:44.125 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.370519+0000', 'last_change': '2024-03-28T23:31:38.370519+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.370519+0000', 'last_undegraded': '2024-03-28T23:31:38.370519+0000', 'last_fullsized': '2024-03-28T23:31:38.370519+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:31:44.125 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.7 is not active+clean 2024-03-28T23:31:44.125 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.125 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.8 is not active+clean 2024-03-28T23:31:44.125 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.368438+0000', 'last_change': '2024-03-28T23:31:38.368438+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.368438+0000', 'last_undegraded': '2024-03-28T23:31:38.368438+0000', 'last_fullsized': '2024-03-28T23:31:38.368438+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-28T23:31:44.125 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.8 is not active+clean 2024-03-28T23:31:44.125 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.125 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.7 is not active+clean 2024-03-28T23:31:44.126 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.368581+0000', 'last_change': '2024-03-28T23:31:38.368581+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.368581+0000', 'last_undegraded': '2024-03-28T23:31:38.368581+0000', 'last_fullsized': '2024-03-28T23:31:38.368581+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:31:44.126 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.a is not active+clean 2024-03-28T23:31:44.126 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.126 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.5 is not active+clean 2024-03-28T23:31:44.126 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.371001+0000', 'last_change': '2024-03-28T23:31:38.371001+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.371001+0000', 'last_undegraded': '2024-03-28T23:31:38.371001+0000', 'last_fullsized': '2024-03-28T23:31:38.371001+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-28T23:31:44.126 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.b is not active+clean 2024-03-28T23:31:44.126 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.126 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.4 is not active+clean 2024-03-28T23:31:44.126 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.370719+0000', 'last_change': '2024-03-28T23:31:38.370719+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.370719+0000', 'last_undegraded': '2024-03-28T23:31:38.370719+0000', 'last_fullsized': '2024-03-28T23:31:38.370719+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:31:44.126 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.c is not active+clean 2024-03-28T23:31:44.126 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.127 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3 is not active+clean 2024-03-28T23:31:44.127 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.366340+0000', 'last_change': '2024-03-28T23:31:38.366340+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.366340+0000', 'last_undegraded': '2024-03-28T23:31:38.366340+0000', 'last_fullsized': '2024-03-28T23:31:38.366340+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-28T23:31:44.127 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.d is not active+clean 2024-03-28T23:31:44.127 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.127 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2 is not active+clean 2024-03-28T23:31:44.127 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.366341+0000', 'last_change': '2024-03-28T23:31:38.366341+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.366341+0000', 'last_undegraded': '2024-03-28T23:31:38.366341+0000', 'last_fullsized': '2024-03-28T23:31:38.366341+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-28T23:31:44.127 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.e is not active+clean 2024-03-28T23:31:44.127 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.127 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1 is not active+clean 2024-03-28T23:31:44.127 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.366564+0000', 'last_change': '2024-03-28T23:31:38.366564+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.366564+0000', 'last_undegraded': '2024-03-28T23:31:38.366564+0000', 'last_fullsized': '2024-03-28T23:31:38.366564+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:31:44.128 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.f is not active+clean 2024-03-28T23:31:44.128 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.128 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.0 is not active+clean 2024-03-28T23:31:44.128 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.371193+0000', 'last_change': '2024-03-28T23:31:38.371193+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.371193+0000', 'last_undegraded': '2024-03-28T23:31:38.371193+0000', 'last_fullsized': '2024-03-28T23:31:38.371193+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-28T23:31:44.128 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.9 is not active+clean 2024-03-28T23:31:44.128 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:31:39.364460+0000', 'last_change': '2024-03-28T23:31:39.364460+0000', 'last_active': '2024-03-28T23:31:39.364460+0000', 'last_peered': '2024-03-28T23:31:39.364460+0000', 'last_clean': '2024-03-28T23:31:39.364460+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:39.364460+0000', 'last_undegraded': '2024-03-28T23:31:39.364460+0000', 'last_fullsized': '2024-03-28T23:31:39.364460+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:39.364460+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:31:44.128 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.6 is not active+clean 2024-03-28T23:31:44.128 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:38.371758+0000', 'last_change': '2024-03-28T23:31:38.371758+0000', 'last_active': '2024-03-28T23:31:38.359664+0000', 'last_peered': '2024-03-28T23:31:38.359664+0000', 'last_clean': '2024-03-28T23:31:38.359664+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:38.371758+0000', 'last_undegraded': '2024-03-28T23:31:38.371758+0000', 'last_fullsized': '2024-03-28T23:31:38.371758+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:38.359664+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:31:44.129 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-28T23:31:44.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.246+0000 7fa1c91b6640 1 Processor -- start 2024-03-28T23:31:44.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.246+0000 7fa1c91b6640 1 -- start start 2024-03-28T23:31:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.247+0000 7fa1c91b6640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa1c40caf60 0x7fa1c41592e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.247+0000 7fa1c91b6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1c40c8970 0x7fa1c40c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.247+0000 7fa1c91b6640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1c40cb4e0 con 0x7fa1c414f260 2024-03-28T23:31:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.247+0000 7fa1c91b6640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa1c4158e50 con 0x7fa1c40caf60 2024-03-28T23:31:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.247+0000 7fa1c91b6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa1c40cb7c0 con 0x7fa1c40c8970 2024-03-28T23:31:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.247+0000 7fa1c259c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa1c40caf60 0x7fa1c41592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.247+0000 7fa1c259c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa1c40caf60 0x7fa1c41592e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58444/0 (socket says 172.21.15.161:58444) 2024-03-28T23:31:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.247+0000 7fa1c259c640 1 -- 172.21.15.161:0/610337307 learned_addr learned my addr 172.21.15.161:0/610337307 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.247+0000 7fa1c359e640 1 --2- 172.21.15.161:0/610337307 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1c40c8970 0x7fa1c40c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.247+0000 7fa1c259c640 1 -- 172.21.15.161:0/610337307 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1c40c8970 msgr2=0x7fa1c40c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.247+0000 7fa1c259c640 1 --2- 172.21.15.161:0/610337307 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1c40c8970 0x7fa1c40c53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.247+0000 7fa1c259c640 1 -- 172.21.15.161:0/610337307 >> v1:172.21.15.170:6789/0 conn(0x7fa1c414f260 legacy=0x7fa1c4155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.247+0000 7fa1c259c640 1 -- 172.21.15.161:0/610337307 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1c4158800 con 0x7fa1c40caf60 2024-03-28T23:31:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.247+0000 7fa1c359e640 1 --2- 172.21.15.161:0/610337307 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1c40c8970 0x7fa1c40c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.248+0000 7fa1c259c640 1 --2- 172.21.15.161:0/610337307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1c40caf60 0x7fa1c41592e0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac009870 tx=0x7fa1ac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a0e79b5707871f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.248+0000 7fa1c1d9b640 1 -- 172.21.15.161:0/610337307 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac035020 con 0x7fa1c40caf60 2024-03-28T23:31:44.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.248+0000 7fa1c1d9b640 1 -- 172.21.15.161:0/610337307 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1ac031640 con 0x7fa1c40caf60 2024-03-28T23:31:44.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.248+0000 7fa1c1d9b640 1 -- 172.21.15.161:0/610337307 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac02d040 con 0x7fa1c40caf60 2024-03-28T23:31:44.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.249+0000 7fa1c91b6640 1 -- 172.21.15.161:0/610337307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1c40caf60 msgr2=0x7fa1c41592e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:44.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.249+0000 7fa1c91b6640 1 --2- 172.21.15.161:0/610337307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1c40caf60 0x7fa1c41592e0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac009870 tx=0x7fa1ac02b8f0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.249+0000 7fa1c91b6640 1 -- 172.21.15.161:0/610337307 shutdown_connections 2024-03-28T23:31:44.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.249+0000 7fa1c91b6640 1 --2- 172.21.15.161:0/610337307 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1c40c8970 0x7fa1c40c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.249+0000 7fa1c91b6640 1 --2- 172.21.15.161:0/610337307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1c40caf60 0x7fa1c41592e0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.249+0000 7fa1c91b6640 1 -- 172.21.15.161:0/610337307 >> 172.21.15.161:0/610337307 conn(0x7fa1c4072ca0 msgr2=0x7fa1c40730b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:44.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.249+0000 7fa1c91b6640 1 -- 172.21.15.161:0/610337307 shutdown_connections 2024-03-28T23:31:44.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.249+0000 7fa1c91b6640 1 -- 172.21.15.161:0/610337307 wait complete. 2024-03-28T23:31:44.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.250+0000 7fa1c91b6640 1 Processor -- start 2024-03-28T23:31:44.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.250+0000 7fa1c91b6640 1 -- start start 2024-03-28T23:31:44.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.251+0000 7fa1c91b6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1c40c8970 0x7fa1c40ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:44.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.251+0000 7fa1c91b6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1c40caf60 0x7fa1c41ebd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:44.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.251+0000 7fa1c91b6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1c414f260 0x7fa1c4154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:44.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.251+0000 7fa1c91b6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c40c8e00 con 0x7fa1c40c8970 2024-03-28T23:31:44.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.251+0000 7fa1c91b6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c414f640 con 0x7fa1c40caf60 2024-03-28T23:31:44.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.251+0000 7fa1c91b6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa1c40c8f80 con 0x7fa1c414f260 2024-03-28T23:31:44.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.251+0000 7fa1c2d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1c40c8970 0x7fa1c40ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:44.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.251+0000 7fa1c2d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1c40c8970 0x7fa1c40ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58458/0 (socket says 172.21.15.161:58458) 2024-03-28T23:31:44.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.251+0000 7fa1c2d9d640 1 -- 172.21.15.161:0/3657027242 learned_addr learned my addr 172.21.15.161:0/3657027242 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:44.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.251+0000 7fa1c359e640 1 --2- 172.21.15.161:0/3657027242 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1c414f260 0x7fa1c4154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:44.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.251+0000 7fa1c259c640 1 --2- 172.21.15.161:0/3657027242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1c40caf60 0x7fa1c41ebd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:44.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.251+0000 7fa1c2d9d640 1 -- 172.21.15.161:0/3657027242 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1c414f260 msgr2=0x7fa1c4154470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:44.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.251+0000 7fa1c2d9d640 1 --2- 172.21.15.161:0/3657027242 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1c414f260 0x7fa1c4154470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.251+0000 7fa1c2d9d640 1 -- 172.21.15.161:0/3657027242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1c40caf60 msgr2=0x7fa1c41ebd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:44.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.251+0000 7fa1c2d9d640 1 --2- 172.21.15.161:0/3657027242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1c40caf60 0x7fa1c41ebd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.251+0000 7fa1c2d9d640 1 -- 172.21.15.161:0/3657027242 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1c4154ed0 con 0x7fa1c40c8970 2024-03-28T23:31:44.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.252+0000 7fa1c2d9d640 1 --2- 172.21.15.161:0/3657027242 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1c40c8970 0x7fa1c40ae3f0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc00c430 tx=0x7fa1bc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:44.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.252+0000 7fa1bb7fe640 1 -- 172.21.15.161:0/3657027242 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc0052d0 con 0x7fa1c40c8970 2024-03-28T23:31:44.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.252+0000 7fa1bb7fe640 1 -- 172.21.15.161:0/3657027242 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1bc005490 con 0x7fa1c40c8970 2024-03-28T23:31:44.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.252+0000 7fa1c91b6640 1 -- 172.21.15.161:0/3657027242 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1c41509b0 con 0x7fa1c40c8970 2024-03-28T23:31:44.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.252+0000 7fa1bb7fe640 1 -- 172.21.15.161:0/3657027242 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc007260 con 0x7fa1c40c8970 2024-03-28T23:31:44.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.252+0000 7fa1c91b6640 1 -- 172.21.15.161:0/3657027242 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1c4152500 con 0x7fa1c40c8970 2024-03-28T23:31:44.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.253+0000 7fa1c91b6640 1 -- 172.21.15.161:0/3657027242 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1c406cdd0 con 0x7fa1c40c8970 2024-03-28T23:31:44.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.253+0000 7fa1bb7fe640 1 -- 172.21.15.161:0/3657027242 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa1bc007400 con 0x7fa1c40c8970 2024-03-28T23:31:44.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.254+0000 7fa1bb7fe640 1 --2- 172.21.15.161:0/3657027242 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa1a406b4d0 0x7fa1a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:44.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.254+0000 7fa1c259c640 1 --2- 172.21.15.161:0/3657027242 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa1a406b4d0 0x7fa1a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:44.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.254+0000 7fa1bb7fe640 1 -- 172.21.15.161:0/3657027242 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7fa1bc089740 con 0x7fa1c40c8970 2024-03-28T23:31:44.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.254+0000 7fa1c259c640 1 --2- 172.21.15.161:0/3657027242 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa1a406b4d0 0x7fa1a406d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac036000 tx=0x7fa1ac03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:44.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.259+0000 7fa1bb7fe640 1 -- 172.21.15.161:0/3657027242 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa1c406cdd0 con 0x7fa1c40c8970 2024-03-28T23:31:44.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.509+0000 7fa1c91b6640 1 -- 172.21.15.161:0/3657027242 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7fa1c4068f90 con 0x7fa1c40c8970 2024-03-28T23:31:44.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.509+0000 7fa1bb7fe640 1 -- 172.21.15.161:0/3657027242 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1109 (secure 0 0 0) 0x7fa1c4068f90 con 0x7fa1c40c8970 2024-03-28T23:31:44.510 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:44.510 INFO:teuthology.orchestra.run.smithi161.stdout:{"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":135,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":51,"num_osds":8,"num_up_osds":8,"osd_up_since":1711668579,"num_in_osds":8,"osd_in_since":1711668571,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":238813184,"bytes_avail":772855300096,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":30,"id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"a","status":"up:active","gid":5820}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-03-28T23:30:47.324341+0000","services":{}},"progress_events":{}} 2024-03-28T23:31:44.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.511+0000 7fa1c91b6640 1 -- 172.21.15.161:0/3657027242 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa1a406b4d0 msgr2=0x7fa1a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:44.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.511+0000 7fa1c91b6640 1 --2- 172.21.15.161:0/3657027242 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa1a406b4d0 0x7fa1a406d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac036000 tx=0x7fa1ac03c040 comp rx=0 tx=0).stop 2024-03-28T23:31:44.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.511+0000 7fa1c91b6640 1 -- 172.21.15.161:0/3657027242 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1c40c8970 msgr2=0x7fa1c40ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:44.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.511+0000 7fa1c91b6640 1 --2- 172.21.15.161:0/3657027242 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1c40c8970 0x7fa1c40ae3f0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc00c430 tx=0x7fa1bc00c8f0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.511+0000 7fa1c91b6640 1 -- 172.21.15.161:0/3657027242 shutdown_connections 2024-03-28T23:31:44.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.511+0000 7fa1c91b6640 1 --2- 172.21.15.161:0/3657027242 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa1a406b4d0 0x7fa1a406d990 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.511+0000 7fa1c91b6640 1 --2- 172.21.15.161:0/3657027242 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1c414f260 0x7fa1c4154470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.511+0000 7fa1c91b6640 1 --2- 172.21.15.161:0/3657027242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1c40caf60 0x7fa1c41ebd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.511+0000 7fa1c91b6640 1 --2- 172.21.15.161:0/3657027242 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1c40c8970 0x7fa1c40ae3f0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.511+0000 7fa1c91b6640 1 -- 172.21.15.161:0/3657027242 >> 172.21.15.161:0/3657027242 conn(0x7fa1c4072ca0 msgr2=0x7fa1c40c7480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:44.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.512+0000 7fa1c91b6640 1 -- 172.21.15.161:0/3657027242 shutdown_connections 2024-03-28T23:31:44.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.512+0000 7fa1c91b6640 1 -- 172.21.15.161:0/3657027242 wait complete. 2024-03-28T23:31:44.523 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-28T23:31:44.523 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:31:44.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.624+0000 7f2b3726b640 1 Processor -- start 2024-03-28T23:31:44.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.624+0000 7f2b3726b640 1 -- start start 2024-03-28T23:31:44.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.625+0000 7f2b3726b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2b30152890 0x7f2b300c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:44.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.625+0000 7f2b3726b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2b300c5b20 0x7f2b301553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:44.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.625+0000 7f2b3726b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b30158a30 con 0x7f2b300c8ed0 2024-03-28T23:31:44.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.625+0000 7f2b3726b640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2b300c8480 con 0x7f2b30152890 2024-03-28T23:31:44.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.625+0000 7f2b3726b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f2b30159640 con 0x7f2b300c5b20 2024-03-28T23:31:44.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.625+0000 7f2b35007640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2b30152890 0x7f2b300c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:44.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.625+0000 7f2b34806640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f2b300c8ed0 0x7f2b300c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44024/0 (socket says 172.21.15.161:44024) 2024-03-28T23:31:44.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.625+0000 7f2b34806640 1 -- 172.21.15.161:0/3708660131 learned_addr learned my addr 172.21.15.161:0/3708660131 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:44.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.626+0000 7f2b35808640 1 --2- 172.21.15.161:0/3708660131 >> v2:172.21.15.170:3300/0 conn(0x7f2b300c5b20 0x7f2b301553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:44.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.626+0000 7f2b35007640 1 -- 172.21.15.161:0/3708660131 >> v2:172.21.15.170:3300/0 conn(0x7f2b300c5b20 msgr2=0x7f2b301553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:44.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.626+0000 7f2b35007640 1 --2- 172.21.15.161:0/3708660131 >> v2:172.21.15.170:3300/0 conn(0x7f2b300c5b20 0x7f2b301553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.626+0000 7f2b35007640 1 -- 172.21.15.161:0/3708660131 >> v1:172.21.15.161:6789/0 conn(0x7f2b300c8ed0 legacy=0x7f2b300c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:44.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.626+0000 7f2b23fff640 1 -- 172.21.15.161:0/3708660131 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 416180334 0 0) 0x7f2b30158a30 con 0x7f2b300c8ed0 2024-03-28T23:31:44.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.626+0000 7f2b35007640 1 -- 172.21.15.161:0/3708660131 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b30158d10 con 0x7f2b30152890 2024-03-28T23:31:44.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.626+0000 7f2b35808640 1 --2- 172.21.15.161:0/3708660131 >> v2:172.21.15.170:3300/0 conn(0x7f2b300c5b20 0x7f2b301553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:44.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.626+0000 7f2b35007640 1 --2- 172.21.15.161:0/3708660131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b30152890 0x7f2b300c8990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f2b300ad480 tx=0x7f2b2c02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1debf77be5da5e30 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:44.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.627+0000 7f2b23fff640 1 -- 172.21.15.161:0/3708660131 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c038070 con 0x7f2b30152890 2024-03-28T23:31:44.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.627+0000 7f2b23fff640 1 -- 172.21.15.161:0/3708660131 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b2c031710 con 0x7f2b30152890 2024-03-28T23:31:44.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.627+0000 7f2b23fff640 1 -- 172.21.15.161:0/3708660131 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c031ae0 con 0x7f2b30152890 2024-03-28T23:31:44.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.627+0000 7f2b3726b640 1 -- 172.21.15.161:0/3708660131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b30152890 msgr2=0x7f2b300c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:44.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.627+0000 7f2b3726b640 1 --2- 172.21.15.161:0/3708660131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b30152890 0x7f2b300c8990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f2b300ad480 tx=0x7f2b2c02b7b0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.627+0000 7f2b3726b640 1 -- 172.21.15.161:0/3708660131 shutdown_connections 2024-03-28T23:31:44.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.627+0000 7f2b3726b640 1 --2- 172.21.15.161:0/3708660131 >> v2:172.21.15.170:3300/0 conn(0x7f2b300c5b20 0x7f2b301553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.627+0000 7f2b3726b640 1 --2- 172.21.15.161:0/3708660131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b30152890 0x7f2b300c8990 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.627+0000 7f2b3726b640 1 -- 172.21.15.161:0/3708660131 >> 172.21.15.161:0/3708660131 conn(0x7f2b300ac620 msgr2=0x7f2b300aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:44.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.627+0000 7f2b3726b640 1 -- 172.21.15.161:0/3708660131 shutdown_connections 2024-03-28T23:31:44.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.627+0000 7f2b3726b640 1 -- 172.21.15.161:0/3708660131 wait complete. 2024-03-28T23:31:44.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.628+0000 7f2b3726b640 1 Processor -- start 2024-03-28T23:31:44.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.628+0000 7f2b3726b640 1 -- start start 2024-03-28T23:31:44.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.629+0000 7f2b3726b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b300c5b20 0x7f2b300bed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:44.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.629+0000 7f2b3726b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b300c8ed0 0x7f2b300ab480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:44.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.629+0000 7f2b3726b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b30152890 0x7f2b30147d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:44.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.629+0000 7f2b3726b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2b30152c70 con 0x7f2b300c8ed0 2024-03-28T23:31:44.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.629+0000 7f2b3726b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2b301592d0 con 0x7f2b300c5b20 2024-03-28T23:31:44.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.629+0000 7f2b3726b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2b30152df0 con 0x7f2b30152890 2024-03-28T23:31:44.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.629+0000 7f2b34806640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b300c8ed0 0x7f2b300ab480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:44.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.629+0000 7f2b34806640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b300c8ed0 0x7f2b300ab480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58478/0 (socket says 172.21.15.161:58478) 2024-03-28T23:31:44.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.629+0000 7f2b34806640 1 -- 172.21.15.161:0/1947535489 learned_addr learned my addr 172.21.15.161:0/1947535489 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:44.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.629+0000 7f2b35808640 1 --2- 172.21.15.161:0/1947535489 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b30152890 0x7f2b30147d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:44.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.629+0000 7f2b35007640 1 --2- 172.21.15.161:0/1947535489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b300c5b20 0x7f2b300bed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:44.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.630+0000 7f2b34806640 1 -- 172.21.15.161:0/1947535489 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b30152890 msgr2=0x7f2b30147d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:44.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.630+0000 7f2b34806640 1 --2- 172.21.15.161:0/1947535489 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b30152890 0x7f2b30147d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.630+0000 7f2b34806640 1 -- 172.21.15.161:0/1947535489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b300c5b20 msgr2=0x7f2b300bed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:44.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.630+0000 7f2b34806640 1 --2- 172.21.15.161:0/1947535489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b300c5b20 0x7f2b300bed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.630+0000 7f2b34806640 1 -- 172.21.15.161:0/1947535489 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b301edcc0 con 0x7f2b300c8ed0 2024-03-28T23:31:44.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.630+0000 7f2b34806640 1 --2- 172.21.15.161:0/1947535489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b300c8ed0 0x7f2b300ab480 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f2b2400c730 tx=0x7f2b2400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:44.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.630+0000 7f2b21ffb640 1 -- 172.21.15.161:0/1947535489 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b24007270 con 0x7f2b300c8ed0 2024-03-28T23:31:44.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.630+0000 7f2b21ffb640 1 -- 172.21.15.161:0/1947535489 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b24007410 con 0x7f2b300c8ed0 2024-03-28T23:31:44.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.630+0000 7f2b3726b640 1 -- 172.21.15.161:0/1947535489 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b301edfa0 con 0x7f2b300c8ed0 2024-03-28T23:31:44.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.630+0000 7f2b21ffb640 1 -- 172.21.15.161:0/1947535489 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b24001d00 con 0x7f2b300c8ed0 2024-03-28T23:31:44.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.630+0000 7f2b3726b640 1 -- 172.21.15.161:0/1947535489 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b301ee560 con 0x7f2b300c8ed0 2024-03-28T23:31:44.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.631+0000 7f2b3726b640 1 -- 172.21.15.161:0/1947535489 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b3004f4b0 con 0x7f2b300c8ed0 2024-03-28T23:31:44.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.631+0000 7f2b21ffb640 1 -- 172.21.15.161:0/1947535489 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2b240057a0 con 0x7f2b300c8ed0 2024-03-28T23:31:44.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.632+0000 7f2b21ffb640 1 --2- 172.21.15.161:0/1947535489 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2b1406b500 0x7f2b1406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:44.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.632+0000 7f2b21ffb640 1 -- 172.21.15.161:0/1947535489 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f2b24089e60 con 0x7f2b300c8ed0 2024-03-28T23:31:44.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.632+0000 7f2b35007640 1 --2- 172.21.15.161:0/1947535489 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2b1406b500 0x7f2b1406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:44.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.632+0000 7f2b35007640 1 --2- 172.21.15.161:0/1947535489 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2b1406b500 0x7f2b1406d9c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c035000 tx=0x7f2b2c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:44.636 INFO:tasks.ceph.mds.a.smithi161.stderr:2024-03-28T23:31:44.636+0000 7fdac8bea640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:31:44.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.636+0000 7f2b21ffb640 1 -- 172.21.15.161:0/1947535489 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2b24058590 con 0x7f2b300c8ed0 2024-03-28T23:31:44.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.811+0000 7f2b3726b640 1 -- 172.21.15.161:0/1947535489 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2b30074690 con 0x7f2b1406b500 2024-03-28T23:31:44.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.816+0000 7f2b21ffb640 1 -- 172.21.15.161:0/1947535489 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+240127 (secure 0 0 0) 0x7f2b30074690 con 0x7f2b1406b500 2024-03-28T23:31:44.819 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:44.820 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:31:44.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.819+0000 7f2b3726b640 1 -- 172.21.15.161:0/1947535489 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2b1406b500 msgr2=0x7f2b1406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:44.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.819+0000 7f2b3726b640 1 --2- 172.21.15.161:0/1947535489 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2b1406b500 0x7f2b1406d9c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c035000 tx=0x7f2b2c008030 comp rx=0 tx=0).stop 2024-03-28T23:31:44.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.819+0000 7f2b3726b640 1 -- 172.21.15.161:0/1947535489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b300c8ed0 msgr2=0x7f2b300ab480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:44.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.819+0000 7f2b3726b640 1 --2- 172.21.15.161:0/1947535489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b300c8ed0 0x7f2b300ab480 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f2b2400c730 tx=0x7f2b2400cbf0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.820+0000 7f2b3726b640 1 -- 172.21.15.161:0/1947535489 shutdown_connections 2024-03-28T23:31:44.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.820+0000 7f2b3726b640 1 --2- 172.21.15.161:0/1947535489 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2b1406b500 0x7f2b1406d9c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.820+0000 7f2b3726b640 1 --2- 172.21.15.161:0/1947535489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b300c8ed0 0x7f2b300ab480 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.820+0000 7f2b3726b640 1 --2- 172.21.15.161:0/1947535489 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b30152890 0x7f2b30147d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.820+0000 7f2b3726b640 1 --2- 172.21.15.161:0/1947535489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b300c5b20 0x7f2b300bed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:44.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.820+0000 7f2b3726b640 1 -- 172.21.15.161:0/1947535489 >> 172.21.15.161:0/1947535489 conn(0x7f2b300ac620 msgr2=0x7f2b301507d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:44.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.820+0000 7f2b3726b640 1 -- 172.21.15.161:0/1947535489 shutdown_connections 2024-03-28T23:31:44.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:44.820+0000 7f2b3726b640 1 -- 172.21.15.161:0/1947535489 wait complete. 2024-03-28T23:31:44.831 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2024-03-28T23:31:43.333903+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":233216,"kb_used_data":4576,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754741504,"statfs":{"total":773094113280,"available":772855300096,"internally_reserved":0,"allocated":4685824,"data_stored":3179728,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.968830"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.369099+0000","last_change":"2024-03-28T23:31:38.369099+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.369099+0000","last_undegraded":"2024-03-28T23:31:38.369099+0000","last_fullsized":"2024-03-28T23:31:38.369099+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370329+0000","last_change":"2024-03-28T23:31:38.370329+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370329+0000","last_undegraded":"2024-03-28T23:31:38.370329+0000","last_fullsized":"2024-03-28T23:31:38.370329+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370328+0000","last_change":"2024-03-28T23:31:38.370328+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370328+0000","last_undegraded":"2024-03-28T23:31:38.370328+0000","last_fullsized":"2024-03-28T23:31:38.370328+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370985+0000","last_change":"2024-03-28T23:31:38.370985+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370985+0000","last_undegraded":"2024-03-28T23:31:38.370985+0000","last_fullsized":"2024-03-28T23:31:38.370985+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.372009+0000","last_change":"2024-03-28T23:31:38.372009+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.372009+0000","last_undegraded":"2024-03-28T23:31:38.372009+0000","last_fullsized":"2024-03-28T23:31:38.372009+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366496+0000","last_change":"2024-03-28T23:31:38.366496+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366496+0000","last_undegraded":"2024-03-28T23:31:38.366496+0000","last_fullsized":"2024-03-28T23:31:38.366496+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368502+0000","last_change":"2024-03-28T23:31:38.368502+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368502+0000","last_undegraded":"2024-03-28T23:31:38.368502+0000","last_fullsized":"2024-03-28T23:31:38.368502+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370603+0000","last_change":"2024-03-28T23:31:38.370603+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370603+0000","last_undegraded":"2024-03-28T23:31:38.370603+0000","last_fullsized":"2024-03-28T23:31:38.370603+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370731+0000","last_change":"2024-03-28T23:31:38.370731+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370731+0000","last_undegraded":"2024-03-28T23:31:38.370731+0000","last_fullsized":"2024-03-28T23:31:38.370731+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366578+0000","last_change":"2024-03-28T23:31:38.366578+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366578+0000","last_undegraded":"2024-03-28T23:31:38.366578+0000","last_fullsized":"2024-03-28T23:31:38.366578+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.371431+0000","last_change":"2024-03-28T23:31:38.371431+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.371431+0000","last_undegraded":"2024-03-28T23:31:38.371431+0000","last_fullsized":"2024-03-28T23:31:38.371431+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366398+0000","last_change":"2024-03-28T23:31:38.366398+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366398+0000","last_undegraded":"2024-03-28T23:31:38.366398+0000","last_fullsized":"2024-03-28T23:31:38.366398+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.371592+0000","last_change":"2024-03-28T23:31:38.371592+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.371592+0000","last_undegraded":"2024-03-28T23:31:38.371592+0000","last_fullsized":"2024-03-28T23:31:38.371592+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368491+0000","last_change":"2024-03-28T23:31:38.368491+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368491+0000","last_undegraded":"2024-03-28T23:31:38.368491+0000","last_fullsized":"2024-03-28T23:31:38.368491+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370684+0000","last_change":"2024-03-28T23:31:38.370684+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370684+0000","last_undegraded":"2024-03-28T23:31:38.370684+0000","last_fullsized":"2024-03-28T23:31:38.370684+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366432+0000","last_change":"2024-03-28T23:31:38.366432+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366432+0000","last_undegraded":"2024-03-28T23:31:38.366432+0000","last_fullsized":"2024-03-28T23:31:38.366432+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370630+0000","last_change":"2024-03-28T23:31:38.370630+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370630+0000","last_undegraded":"2024-03-28T23:31:38.370630+0000","last_fullsized":"2024-03-28T23:31:38.370630+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368798+0000","last_change":"2024-03-28T23:31:38.368798+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368798+0000","last_undegraded":"2024-03-28T23:31:38.368798+0000","last_fullsized":"2024-03-28T23:31:38.368798+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368596+0000","last_change":"2024-03-28T23:31:38.368596+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368596+0000","last_undegraded":"2024-03-28T23:31:38.368596+0000","last_fullsized":"2024-03-28T23:31:38.368596+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368582+0000","last_change":"2024-03-28T23:31:38.368582+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368582+0000","last_undegraded":"2024-03-28T23:31:38.368582+0000","last_fullsized":"2024-03-28T23:31:38.368582+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368410+0000","last_change":"2024-03-28T23:31:38.368410+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368410+0000","last_undegraded":"2024-03-28T23:31:38.368410+0000","last_fullsized":"2024-03-28T23:31:38.368410+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366810+0000","last_change":"2024-03-28T23:31:38.366810+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366810+0000","last_undegraded":"2024-03-28T23:31:38.366810+0000","last_fullsized":"2024-03-28T23:31:38.366810+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370519+0000","last_change":"2024-03-28T23:31:38.370519+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370519+0000","last_undegraded":"2024-03-28T23:31:38.370519+0000","last_fullsized":"2024-03-28T23:31:38.370519+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368438+0000","last_change":"2024-03-28T23:31:38.368438+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368438+0000","last_undegraded":"2024-03-28T23:31:38.368438+0000","last_fullsized":"2024-03-28T23:31:38.368438+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.368581+0000","last_change":"2024-03-28T23:31:38.368581+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.368581+0000","last_undegraded":"2024-03-28T23:31:38.368581+0000","last_fullsized":"2024-03-28T23:31:38.368581+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.371001+0000","last_change":"2024-03-28T23:31:38.371001+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.371001+0000","last_undegraded":"2024-03-28T23:31:38.371001+0000","last_fullsized":"2024-03-28T23:31:38.371001+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.370719+0000","last_change":"2024-03-28T23:31:38.370719+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.370719+0000","last_undegraded":"2024-03-28T23:31:38.370719+0000","last_fullsized":"2024-03-28T23:31:38.370719+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366340+0000","last_change":"2024-03-28T23:31:38.366340+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366340+0000","last_undegraded":"2024-03-28T23:31:38.366340+0000","last_fullsized":"2024-03-28T23:31:38.366340+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366341+0000","last_change":"2024-03-28T23:31:38.366341+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366341+0000","last_undegraded":"2024-03-28T23:31:38.366341+0000","last_fullsized":"2024-03-28T23:31:38.366341+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.366564+0000","last_change":"2024-03-28T23:31:38.366564+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.366564+0000","last_undegraded":"2024-03-28T23:31:38.366564+0000","last_fullsized":"2024-03-28T23:31:38.366564+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.371193+0000","last_change":"2024-03-28T23:31:38.371193+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.371193+0000","last_undegraded":"2024-03-28T23:31:38.371193+0000","last_fullsized":"2024-03-28T23:31:38.371193+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:31:39.364460+0000","last_change":"2024-03-28T23:31:39.364460+0000","last_active":"2024-03-28T23:31:39.364460+0000","last_peered":"2024-03-28T23:31:39.364460+0000","last_clean":"2024-03-28T23:31:39.364460+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:39.364460+0000","last_undegraded":"2024-03-28T23:31:39.364460+0000","last_fullsized":"2024-03-28T23:31:39.364460+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-28T23:31:38.371758+0000","last_change":"2024-03-28T23:31:38.371758+0000","last_active":"2024-03-28T23:31:38.359664+0000","last_peered":"2024-03-28T23:31:38.359664+0000","last_clean":"2024-03-28T23:31:38.359664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:38.371758+0000","last_undegraded":"2024-03-28T23:31:38.371758+0000","last_fullsized":"2024-03-28T23:31:38.371758+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":222,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-28T23:31:38.371895+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:31:38.371895+0000","last_peered":"2024-03-28T23:31:38.371895+0000","last_clean":"2024-03-28T23:31:38.371895+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:31:38.371895+0000","last_undegraded":"2024-03-28T23:31:38.371895+0000","last_fullsized":"2024-03-28T23:31:38.371895+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":634880,"data_stored":840242,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31736,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340104,"statfs":{"total":96636764160,"available":96604266496,"internally_reserved":0,"allocated":569344,"data_stored":249874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":249874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31736,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340104,"statfs":{"total":96636764160,"available":96604266496,"internally_reserved":0,"allocated":569344,"data_stored":249874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":249874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":249874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":249874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":634880,"data_stored":840242,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:31:47.836 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:31:47.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.938+0000 7f56c2e15640 1 Processor -- start 2024-03-28T23:31:47.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.938+0000 7f56c2e15640 1 -- start start 2024-03-28T23:31:47.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.939+0000 7f56c2e15640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56bc152890 0x7f56bc0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:47.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.939+0000 7f56c2e15640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f56bc0c8ed0 0x7f56bc0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:47.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.939+0000 7f56c2e15640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56bc158a30 con 0x7f56bc0c5950 2024-03-28T23:31:47.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.939+0000 7f56c2e15640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f56bc0c5db0 con 0x7f56bc0c8ed0 2024-03-28T23:31:47.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.939+0000 7f56c2e15640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f56bc0c8480 con 0x7f56bc152890 2024-03-28T23:31:47.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.939+0000 7f56c0bb1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56bc152890 0x7f56bc0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:47.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.939+0000 7f56c0bb1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56bc152890 0x7f56bc0c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:58054/0 (socket says 172.21.15.161:58054) 2024-03-28T23:31:47.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.939+0000 7f56bbfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f56bc0c8ed0 0x7f56bc0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:47.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.939+0000 7f56c0bb1640 1 -- 172.21.15.161:0/230258485 learned_addr learned my addr 172.21.15.161:0/230258485 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:47.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.939+0000 7f56c0bb1640 1 -- 172.21.15.161:0/230258485 >> v2:172.21.15.170:3300/0 conn(0x7f56bc0c8ed0 msgr2=0x7f56bc0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:47.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.939+0000 7f56c0bb1640 1 --2- 172.21.15.161:0/230258485 >> v2:172.21.15.170:3300/0 conn(0x7f56bc0c8ed0 0x7f56bc0c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:47.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.939+0000 7f56c0bb1640 1 -- 172.21.15.161:0/230258485 >> v1:172.21.15.170:6789/0 conn(0x7f56bc0c5950 legacy=0x7f56bc1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:47.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.939+0000 7f56c0bb1640 1 -- 172.21.15.161:0/230258485 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56bc158d10 con 0x7f56bc152890 2024-03-28T23:31:47.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.939+0000 7f56bbfff640 1 --2- 172.21.15.161:0/230258485 >> v2:172.21.15.170:3300/0 conn(0x7f56bc0c8ed0 0x7f56bc0c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:47.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.940+0000 7f56c0bb1640 1 --2- 172.21.15.161:0/230258485 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56bc152890 0x7f56bc0c8990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009870 tx=0x7f56b002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ed95f440696ad22 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:47.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.940+0000 7f56bb7fe640 1 -- 172.21.15.161:0/230258485 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b00317c0 con 0x7f56bc152890 2024-03-28T23:31:47.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.940+0000 7f56bb7fe640 1 -- 172.21.15.161:0/230258485 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56b0037070 con 0x7f56bc152890 2024-03-28T23:31:47.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.940+0000 7f56bb7fe640 1 -- 172.21.15.161:0/230258485 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b0031b80 con 0x7f56bc152890 2024-03-28T23:31:47.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.941+0000 7f56c2e15640 1 -- 172.21.15.161:0/230258485 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56bc152890 msgr2=0x7f56bc0c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:47.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.941+0000 7f56c2e15640 1 --2- 172.21.15.161:0/230258485 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56bc152890 0x7f56bc0c8990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009870 tx=0x7f56b002b860 comp rx=0 tx=0).stop 2024-03-28T23:31:47.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.941+0000 7f56c2e15640 1 -- 172.21.15.161:0/230258485 shutdown_connections 2024-03-28T23:31:47.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.941+0000 7f56c2e15640 1 --2- 172.21.15.161:0/230258485 >> v2:172.21.15.170:3300/0 conn(0x7f56bc0c8ed0 0x7f56bc0c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:47.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.941+0000 7f56c2e15640 1 --2- 172.21.15.161:0/230258485 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56bc152890 0x7f56bc0c8990 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:47.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.941+0000 7f56c2e15640 1 -- 172.21.15.161:0/230258485 >> 172.21.15.161:0/230258485 conn(0x7f56bc0ac620 msgr2=0x7f56bc0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:47.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.941+0000 7f56c2e15640 1 -- 172.21.15.161:0/230258485 shutdown_connections 2024-03-28T23:31:47.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.941+0000 7f56c2e15640 1 -- 172.21.15.161:0/230258485 wait complete. 2024-03-28T23:31:47.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.942+0000 7f56c2e15640 1 Processor -- start 2024-03-28T23:31:47.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.942+0000 7f56c2e15640 1 -- start start 2024-03-28T23:31:47.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.943+0000 7f56c2e15640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56bc0c5950 0x7f56bc1eb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:47.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.943+0000 7f56c2e15640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56bc0c8ed0 0x7f56bc14d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:47.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.943+0000 7f56c2e15640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56bc152890 0x7f56bc1e7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:47.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.943+0000 7f56c2e15640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f56bc1592d0 con 0x7f56bc0c5950 2024-03-28T23:31:47.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.943+0000 7f56c2e15640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f56bc159640 con 0x7f56bc0c8ed0 2024-03-28T23:31:47.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.943+0000 7f56c2e15640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f56bc152c70 con 0x7f56bc152890 2024-03-28T23:31:47.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.943+0000 7f56c0bb1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56bc0c5950 0x7f56bc1eb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:47.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.943+0000 7f56c0bb1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56bc0c5950 0x7f56bc1eb270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58486/0 (socket says 172.21.15.161:58486) 2024-03-28T23:31:47.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.943+0000 7f56c0bb1640 1 -- 172.21.15.161:0/1983622136 learned_addr learned my addr 172.21.15.161:0/1983622136 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:47.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.943+0000 7f56bbfff640 1 --2- 172.21.15.161:0/1983622136 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56bc0c8ed0 0x7f56bc14d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:47.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.943+0000 7f56c13b2640 1 --2- 172.21.15.161:0/1983622136 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56bc152890 0x7f56bc1e7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:47.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.943+0000 7f56c0bb1640 1 -- 172.21.15.161:0/1983622136 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56bc152890 msgr2=0x7f56bc1e7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:47.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.943+0000 7f56c0bb1640 1 --2- 172.21.15.161:0/1983622136 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56bc152890 0x7f56bc1e7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:47.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.943+0000 7f56c0bb1640 1 -- 172.21.15.161:0/1983622136 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56bc0c8ed0 msgr2=0x7f56bc14d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:47.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.943+0000 7f56c0bb1640 1 --2- 172.21.15.161:0/1983622136 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56bc0c8ed0 0x7f56bc14d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:47.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.943+0000 7f56c0bb1640 1 -- 172.21.15.161:0/1983622136 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56bc14cba0 con 0x7f56bc0c5950 2024-03-28T23:31:47.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.943+0000 7f56c0bb1640 1 --2- 172.21.15.161:0/1983622136 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56bc0c5950 0x7f56bc1eb270 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f56b0000c00 tx=0x7f56b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:47.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.943+0000 7f56b97fa640 1 -- 172.21.15.161:0/1983622136 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b0032650 con 0x7f56bc0c5950 2024-03-28T23:31:47.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.943+0000 7f56b97fa640 1 -- 172.21.15.161:0/1983622136 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56b00327f0 con 0x7f56bc0c5950 2024-03-28T23:31:47.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.943+0000 7f56c2e15640 1 -- 172.21.15.161:0/1983622136 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56bc143b00 con 0x7f56bc0c5950 2024-03-28T23:31:47.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.944+0000 7f56b97fa640 1 -- 172.21.15.161:0/1983622136 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b0032be0 con 0x7f56bc0c5950 2024-03-28T23:31:47.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.944+0000 7f56c2e15640 1 -- 172.21.15.161:0/1983622136 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56bc142100 con 0x7f56bc0c5950 2024-03-28T23:31:47.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.944+0000 7f56c2e15640 1 -- 172.21.15.161:0/1983622136 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56bc04f4b0 con 0x7f56bc0c5950 2024-03-28T23:31:47.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.945+0000 7f56b97fa640 1 -- 172.21.15.161:0/1983622136 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f56b004c060 con 0x7f56bc0c5950 2024-03-28T23:31:47.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.945+0000 7f56b97fa640 1 --2- 172.21.15.161:0/1983622136 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f56a006b500 0x7f56a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:47.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.946+0000 7f56bbfff640 1 --2- 172.21.15.161:0/1983622136 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f56a006b500 0x7f56a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:47.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.946+0000 7f56b97fa640 1 -- 172.21.15.161:0/1983622136 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f56b00abd70 con 0x7f56bc0c5950 2024-03-28T23:31:47.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.946+0000 7f56bbfff640 1 --2- 172.21.15.161:0/1983622136 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f56a006b500 0x7f56a006d9c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f56a8009500 tx=0x7f56a8009110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:47.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:47.950+0000 7f56b97fa640 1 -- 172.21.15.161:0/1983622136 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f56bc14cba0 con 0x7f56bc0c5950 2024-03-28T23:31:48.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.123+0000 7f56c2e15640 1 -- 172.21.15.161:0/1983622136 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f56bc0c8140 con 0x7f56a006b500 2024-03-28T23:31:48.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.129+0000 7f56b97fa640 1 -- 172.21.15.161:0/1983622136 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252030 (secure 0 0 0) 0x7f56bc0c8140 con 0x7f56a006b500 2024-03-28T23:31:48.133 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:48.134 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:31:48.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.132+0000 7f56c2e15640 1 -- 172.21.15.161:0/1983622136 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f56a006b500 msgr2=0x7f56a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:48.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.132+0000 7f56c2e15640 1 --2- 172.21.15.161:0/1983622136 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f56a006b500 0x7f56a006d9c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f56a8009500 tx=0x7f56a8009110 comp rx=0 tx=0).stop 2024-03-28T23:31:48.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.132+0000 7f56c2e15640 1 -- 172.21.15.161:0/1983622136 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56bc0c5950 msgr2=0x7f56bc1eb270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:48.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.132+0000 7f56c2e15640 1 --2- 172.21.15.161:0/1983622136 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56bc0c5950 0x7f56bc1eb270 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f56b0000c00 tx=0x7f56b00046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:48.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.132+0000 7f56c2e15640 1 -- 172.21.15.161:0/1983622136 shutdown_connections 2024-03-28T23:31:48.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.132+0000 7f56c2e15640 1 --2- 172.21.15.161:0/1983622136 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f56a006b500 0x7f56a006d9c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:48.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.132+0000 7f56c2e15640 1 --2- 172.21.15.161:0/1983622136 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56bc152890 0x7f56bc1e7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:48.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.132+0000 7f56c2e15640 1 --2- 172.21.15.161:0/1983622136 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56bc0c8ed0 0x7f56bc14d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:48.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.132+0000 7f56c2e15640 1 --2- 172.21.15.161:0/1983622136 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56bc0c5950 0x7f56bc1eb270 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:48.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.132+0000 7f56c2e15640 1 -- 172.21.15.161:0/1983622136 >> 172.21.15.161:0/1983622136 conn(0x7f56bc0ac620 msgr2=0x7f56bc06be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:48.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.132+0000 7f56c2e15640 1 -- 172.21.15.161:0/1983622136 shutdown_connections 2024-03-28T23:31:48.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.132+0000 7f56c2e15640 1 -- 172.21.15.161:0/1983622136 wait complete. 2024-03-28T23:31:48.145 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2024-03-28T23:31:47.334700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237736,"kb_used_data":4928,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736984,"statfs":{"total":773094113280,"available":772850671616,"internally_reserved":0,"allocated":5046272,"data_stored":3348756,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.961149"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554139+0000","last_change":"2024-03-28T23:31:40.388090+0000","last_active":"2024-03-28T23:31:41.554139+0000","last_peered":"2024-03-28T23:31:41.554139+0000","last_clean":"2024-03-28T23:31:41.554139+0000","last_became_active":"2024-03-28T23:31:40.387805+0000","last_became_peered":"2024-03-28T23:31:40.387805+0000","last_unstale":"2024-03-28T23:31:41.554139+0000","last_undegraded":"2024-03-28T23:31:41.554139+0000","last_fullsized":"2024-03-28T23:31:41.554139+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:19:47.987753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.555067+0000","last_change":"2024-03-28T23:31:40.395246+0000","last_active":"2024-03-28T23:31:41.555067+0000","last_peered":"2024-03-28T23:31:41.555067+0000","last_clean":"2024-03-28T23:31:41.555067+0000","last_became_active":"2024-03-28T23:31:40.394583+0000","last_became_peered":"2024-03-28T23:31:40.394583+0000","last_unstale":"2024-03-28T23:31:41.555067+0000","last_undegraded":"2024-03-28T23:31:41.555067+0000","last_fullsized":"2024-03-28T23:31:41.555067+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:18.654331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.555214+0000","last_change":"2024-03-28T23:31:40.397914+0000","last_active":"2024-03-28T23:31:41.555214+0000","last_peered":"2024-03-28T23:31:41.555214+0000","last_clean":"2024-03-28T23:31:41.555214+0000","last_became_active":"2024-03-28T23:31:40.394904+0000","last_became_peered":"2024-03-28T23:31:40.394904+0000","last_unstale":"2024-03-28T23:31:41.555214+0000","last_undegraded":"2024-03-28T23:31:41.555214+0000","last_fullsized":"2024-03-28T23:31:41.555214+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:26:03.062546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554729+0000","last_change":"2024-03-28T23:31:40.400594+0000","last_active":"2024-03-28T23:31:41.554729+0000","last_peered":"2024-03-28T23:31:41.554729+0000","last_clean":"2024-03-28T23:31:41.554729+0000","last_became_active":"2024-03-28T23:31:40.400255+0000","last_became_peered":"2024-03-28T23:31:40.400255+0000","last_unstale":"2024-03-28T23:31:41.554729+0000","last_undegraded":"2024-03-28T23:31:41.554729+0000","last_fullsized":"2024-03-28T23:31:41.554729+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:14.515069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.391700+0000","last_change":"2024-03-28T23:31:40.388531+0000","last_active":"2024-03-28T23:31:41.391700+0000","last_peered":"2024-03-28T23:31:41.391700+0000","last_clean":"2024-03-28T23:31:41.391700+0000","last_became_active":"2024-03-28T23:31:40.388206+0000","last_became_peered":"2024-03-28T23:31:40.388206+0000","last_unstale":"2024-03-28T23:31:41.391700+0000","last_undegraded":"2024-03-28T23:31:41.391700+0000","last_fullsized":"2024-03-28T23:31:41.391700+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:23.215366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554088+0000","last_change":"2024-03-28T23:31:40.393907+0000","last_active":"2024-03-28T23:31:41.554088+0000","last_peered":"2024-03-28T23:31:41.554088+0000","last_clean":"2024-03-28T23:31:41.554088+0000","last_became_active":"2024-03-28T23:31:40.393443+0000","last_became_peered":"2024-03-28T23:31:40.393443+0000","last_unstale":"2024-03-28T23:31:41.554088+0000","last_undegraded":"2024-03-28T23:31:41.554088+0000","last_fullsized":"2024-03-28T23:31:41.554088+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:28:29.180775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554785+0000","last_change":"2024-03-28T23:31:40.398876+0000","last_active":"2024-03-28T23:31:41.554785+0000","last_peered":"2024-03-28T23:31:41.554785+0000","last_clean":"2024-03-28T23:31:41.554785+0000","last_became_active":"2024-03-28T23:31:40.398595+0000","last_became_peered":"2024-03-28T23:31:40.398595+0000","last_unstale":"2024-03-28T23:31:41.554785+0000","last_undegraded":"2024-03-28T23:31:41.554785+0000","last_fullsized":"2024-03-28T23:31:41.554785+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:21.725238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.390545+0000","last_change":"2024-03-28T23:31:40.392421+0000","last_active":"2024-03-28T23:31:41.390545+0000","last_peered":"2024-03-28T23:31:41.390545+0000","last_clean":"2024-03-28T23:31:41.390545+0000","last_became_active":"2024-03-28T23:31:40.391463+0000","last_became_peered":"2024-03-28T23:31:40.391463+0000","last_unstale":"2024-03-28T23:31:41.390545+0000","last_undegraded":"2024-03-28T23:31:41.390545+0000","last_fullsized":"2024-03-28T23:31:41.390545+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:18:16.851435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.390958+0000","last_change":"2024-03-28T23:31:40.403340+0000","last_active":"2024-03-28T23:31:41.390958+0000","last_peered":"2024-03-28T23:31:41.390958+0000","last_clean":"2024-03-28T23:31:41.390958+0000","last_became_active":"2024-03-28T23:31:40.403089+0000","last_became_peered":"2024-03-28T23:31:40.403089+0000","last_unstale":"2024-03-28T23:31:41.390958+0000","last_undegraded":"2024-03-28T23:31:41.390958+0000","last_fullsized":"2024-03-28T23:31:41.390958+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:12:38.242108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554781+0000","last_change":"2024-03-28T23:31:40.382204+0000","last_active":"2024-03-28T23:31:41.554781+0000","last_peered":"2024-03-28T23:31:41.554781+0000","last_clean":"2024-03-28T23:31:41.554781+0000","last_became_active":"2024-03-28T23:31:40.381944+0000","last_became_peered":"2024-03-28T23:31:40.381944+0000","last_unstale":"2024-03-28T23:31:41.554781+0000","last_undegraded":"2024-03-28T23:31:41.554781+0000","last_fullsized":"2024-03-28T23:31:41.554781+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:14.515069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.555034+0000","last_change":"2024-03-28T23:31:40.394578+0000","last_active":"2024-03-28T23:31:41.555034+0000","last_peered":"2024-03-28T23:31:41.555034+0000","last_clean":"2024-03-28T23:31:41.555034+0000","last_became_active":"2024-03-28T23:31:40.394272+0000","last_became_peered":"2024-03-28T23:31:40.394272+0000","last_unstale":"2024-03-28T23:31:41.555034+0000","last_undegraded":"2024-03-28T23:31:41.555034+0000","last_fullsized":"2024-03-28T23:31:41.555034+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:24:35.708796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.390682+0000","last_change":"2024-03-28T23:31:40.396702+0000","last_active":"2024-03-28T23:31:41.390682+0000","last_peered":"2024-03-28T23:31:41.390682+0000","last_clean":"2024-03-28T23:31:41.390682+0000","last_became_active":"2024-03-28T23:31:40.395941+0000","last_became_peered":"2024-03-28T23:31:40.395941+0000","last_unstale":"2024-03-28T23:31:41.390682+0000","last_undegraded":"2024-03-28T23:31:41.390682+0000","last_fullsized":"2024-03-28T23:31:41.390682+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:07:13.690791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554432+0000","last_change":"2024-03-28T23:31:40.382947+0000","last_active":"2024-03-28T23:31:41.554432+0000","last_peered":"2024-03-28T23:31:41.554432+0000","last_clean":"2024-03-28T23:31:41.554432+0000","last_became_active":"2024-03-28T23:31:40.381893+0000","last_became_peered":"2024-03-28T23:31:40.381893+0000","last_unstale":"2024-03-28T23:31:41.554432+0000","last_undegraded":"2024-03-28T23:31:41.554432+0000","last_fullsized":"2024-03-28T23:31:41.554432+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:50.872886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554580+0000","last_change":"2024-03-28T23:31:40.405627+0000","last_active":"2024-03-28T23:31:41.554580+0000","last_peered":"2024-03-28T23:31:41.554580+0000","last_clean":"2024-03-28T23:31:41.554580+0000","last_became_active":"2024-03-28T23:31:40.405360+0000","last_became_peered":"2024-03-28T23:31:40.405360+0000","last_unstale":"2024-03-28T23:31:41.554580+0000","last_undegraded":"2024-03-28T23:31:41.554580+0000","last_fullsized":"2024-03-28T23:31:41.554580+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:06.567694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554801+0000","last_change":"2024-03-28T23:31:40.396605+0000","last_active":"2024-03-28T23:31:41.554801+0000","last_peered":"2024-03-28T23:31:41.554801+0000","last_clean":"2024-03-28T23:31:41.554801+0000","last_became_active":"2024-03-28T23:31:40.396237+0000","last_became_peered":"2024-03-28T23:31:40.396237+0000","last_unstale":"2024-03-28T23:31:41.554801+0000","last_undegraded":"2024-03-28T23:31:41.554801+0000","last_fullsized":"2024-03-28T23:31:41.554801+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:14.144440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.553825+0000","last_change":"2024-03-28T23:31:40.382823+0000","last_active":"2024-03-28T23:31:41.553825+0000","last_peered":"2024-03-28T23:31:41.553825+0000","last_clean":"2024-03-28T23:31:41.553825+0000","last_became_active":"2024-03-28T23:31:40.382574+0000","last_became_peered":"2024-03-28T23:31:40.382574+0000","last_unstale":"2024-03-28T23:31:41.553825+0000","last_undegraded":"2024-03-28T23:31:41.553825+0000","last_fullsized":"2024-03-28T23:31:41.553825+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:12:27.477720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.555577+0000","last_change":"2024-03-28T23:31:40.403462+0000","last_active":"2024-03-28T23:31:41.555577+0000","last_peered":"2024-03-28T23:31:41.555577+0000","last_clean":"2024-03-28T23:31:41.555577+0000","last_became_active":"2024-03-28T23:31:40.402885+0000","last_became_peered":"2024-03-28T23:31:40.402885+0000","last_unstale":"2024-03-28T23:31:41.555577+0000","last_undegraded":"2024-03-28T23:31:41.555577+0000","last_fullsized":"2024-03-28T23:31:41.555577+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:42:29.351941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.555115+0000","last_change":"2024-03-28T23:31:40.395478+0000","last_active":"2024-03-28T23:31:41.555115+0000","last_peered":"2024-03-28T23:31:41.555115+0000","last_clean":"2024-03-28T23:31:41.555115+0000","last_became_active":"2024-03-28T23:31:40.394981+0000","last_became_peered":"2024-03-28T23:31:40.394981+0000","last_unstale":"2024-03-28T23:31:41.555115+0000","last_undegraded":"2024-03-28T23:31:41.555115+0000","last_fullsized":"2024-03-28T23:31:41.555115+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:06.567694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.555338+0000","last_change":"2024-03-28T23:31:40.394475+0000","last_active":"2024-03-28T23:31:41.555338+0000","last_peered":"2024-03-28T23:31:41.555338+0000","last_clean":"2024-03-28T23:31:41.555338+0000","last_became_active":"2024-03-28T23:31:40.390205+0000","last_became_peered":"2024-03-28T23:31:40.390205+0000","last_unstale":"2024-03-28T23:31:41.555338+0000","last_undegraded":"2024-03-28T23:31:41.555338+0000","last_fullsized":"2024-03-28T23:31:41.555338+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:04:48.226221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554814+0000","last_change":"2024-03-28T23:31:40.388386+0000","last_active":"2024-03-28T23:31:41.554814+0000","last_peered":"2024-03-28T23:31:41.554814+0000","last_clean":"2024-03-28T23:31:41.554814+0000","last_became_active":"2024-03-28T23:31:40.388043+0000","last_became_peered":"2024-03-28T23:31:40.388043+0000","last_unstale":"2024-03-28T23:31:41.554814+0000","last_undegraded":"2024-03-28T23:31:41.554814+0000","last_fullsized":"2024-03-28T23:31:41.554814+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:07.738674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554356+0000","last_change":"2024-03-28T23:31:40.395811+0000","last_active":"2024-03-28T23:31:41.554356+0000","last_peered":"2024-03-28T23:31:41.554356+0000","last_clean":"2024-03-28T23:31:41.554356+0000","last_became_active":"2024-03-28T23:31:40.395540+0000","last_became_peered":"2024-03-28T23:31:40.395540+0000","last_unstale":"2024-03-28T23:31:41.554356+0000","last_undegraded":"2024-03-28T23:31:41.554356+0000","last_fullsized":"2024-03-28T23:31:41.554356+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:51.864986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.390510+0000","last_change":"2024-03-28T23:31:40.403129+0000","last_active":"2024-03-28T23:31:41.390510+0000","last_peered":"2024-03-28T23:31:41.390510+0000","last_clean":"2024-03-28T23:31:41.390510+0000","last_became_active":"2024-03-28T23:31:40.402879+0000","last_became_peered":"2024-03-28T23:31:40.402879+0000","last_unstale":"2024-03-28T23:31:41.390510+0000","last_undegraded":"2024-03-28T23:31:41.390510+0000","last_fullsized":"2024-03-28T23:31:41.390510+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:47.861300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554355+0000","last_change":"2024-03-28T23:31:40.389854+0000","last_active":"2024-03-28T23:31:41.554355+0000","last_peered":"2024-03-28T23:31:41.554355+0000","last_clean":"2024-03-28T23:31:41.554355+0000","last_became_active":"2024-03-28T23:31:40.389599+0000","last_became_peered":"2024-03-28T23:31:40.389599+0000","last_unstale":"2024-03-28T23:31:41.554355+0000","last_undegraded":"2024-03-28T23:31:41.554355+0000","last_fullsized":"2024-03-28T23:31:41.554355+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:43:37.883841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.555112+0000","last_change":"2024-03-28T23:31:40.406317+0000","last_active":"2024-03-28T23:31:41.555112+0000","last_peered":"2024-03-28T23:31:41.555112+0000","last_clean":"2024-03-28T23:31:41.555112+0000","last_became_active":"2024-03-28T23:31:40.405217+0000","last_became_peered":"2024-03-28T23:31:40.405217+0000","last_unstale":"2024-03-28T23:31:41.555112+0000","last_undegraded":"2024-03-28T23:31:41.555112+0000","last_fullsized":"2024-03-28T23:31:41.555112+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:37:55.870956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.555577+0000","last_change":"2024-03-28T23:31:40.404249+0000","last_active":"2024-03-28T23:31:41.555577+0000","last_peered":"2024-03-28T23:31:41.555577+0000","last_clean":"2024-03-28T23:31:41.555577+0000","last_became_active":"2024-03-28T23:31:40.403966+0000","last_became_peered":"2024-03-28T23:31:40.403966+0000","last_unstale":"2024-03-28T23:31:41.555577+0000","last_undegraded":"2024-03-28T23:31:41.555577+0000","last_fullsized":"2024-03-28T23:31:41.555577+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:40:57.541676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.391076+0000","last_change":"2024-03-28T23:31:40.397170+0000","last_active":"2024-03-28T23:31:41.391076+0000","last_peered":"2024-03-28T23:31:41.391076+0000","last_clean":"2024-03-28T23:31:41.391076+0000","last_became_active":"2024-03-28T23:31:40.396898+0000","last_became_peered":"2024-03-28T23:31:40.396898+0000","last_unstale":"2024-03-28T23:31:41.391076+0000","last_undegraded":"2024-03-28T23:31:41.391076+0000","last_fullsized":"2024-03-28T23:31:41.391076+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:53:26.355348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554214+0000","last_change":"2024-03-28T23:31:40.404955+0000","last_active":"2024-03-28T23:31:41.554214+0000","last_peered":"2024-03-28T23:31:41.554214+0000","last_clean":"2024-03-28T23:31:41.554214+0000","last_became_active":"2024-03-28T23:31:40.404689+0000","last_became_peered":"2024-03-28T23:31:40.404689+0000","last_unstale":"2024-03-28T23:31:41.554214+0000","last_undegraded":"2024-03-28T23:31:41.554214+0000","last_fullsized":"2024-03-28T23:31:41.554214+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:50:20.574648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554598+0000","last_change":"2024-03-28T23:31:40.389261+0000","last_active":"2024-03-28T23:31:41.554598+0000","last_peered":"2024-03-28T23:31:41.554598+0000","last_clean":"2024-03-28T23:31:41.554598+0000","last_became_active":"2024-03-28T23:31:40.388930+0000","last_became_peered":"2024-03-28T23:31:40.388930+0000","last_unstale":"2024-03-28T23:31:41.554598+0000","last_undegraded":"2024-03-28T23:31:41.554598+0000","last_fullsized":"2024-03-28T23:31:41.554598+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:23.215366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.555095+0000","last_change":"2024-03-28T23:31:40.401673+0000","last_active":"2024-03-28T23:31:41.555095+0000","last_peered":"2024-03-28T23:31:41.555095+0000","last_clean":"2024-03-28T23:31:41.555095+0000","last_became_active":"2024-03-28T23:31:40.398923+0000","last_became_peered":"2024-03-28T23:31:40.398923+0000","last_unstale":"2024-03-28T23:31:41.555095+0000","last_undegraded":"2024-03-28T23:31:41.555095+0000","last_fullsized":"2024-03-28T23:31:41.555095+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:18.783120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554942+0000","last_change":"2024-03-28T23:31:40.387082+0000","last_active":"2024-03-28T23:31:41.554942+0000","last_peered":"2024-03-28T23:31:41.554942+0000","last_clean":"2024-03-28T23:31:41.554942+0000","last_became_active":"2024-03-28T23:31:40.386808+0000","last_became_peered":"2024-03-28T23:31:40.386808+0000","last_unstale":"2024-03-28T23:31:41.554942+0000","last_undegraded":"2024-03-28T23:31:41.554942+0000","last_fullsized":"2024-03-28T23:31:41.554942+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:43.111058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554925+0000","last_change":"2024-03-28T23:31:40.397815+0000","last_active":"2024-03-28T23:31:41.554925+0000","last_peered":"2024-03-28T23:31:41.554925+0000","last_clean":"2024-03-28T23:31:41.554925+0000","last_became_active":"2024-03-28T23:31:40.397279+0000","last_became_peered":"2024-03-28T23:31:40.397279+0000","last_unstale":"2024-03-28T23:31:41.554925+0000","last_undegraded":"2024-03-28T23:31:41.554925+0000","last_fullsized":"2024-03-28T23:31:41.554925+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:25:57.449014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.391247+0000","last_change":"2024-03-28T23:31:40.394484+0000","last_active":"2024-03-28T23:31:41.391247+0000","last_peered":"2024-03-28T23:31:41.391247+0000","last_clean":"2024-03-28T23:31:41.391247+0000","last_became_active":"2024-03-28T23:31:40.394219+0000","last_became_peered":"2024-03-28T23:31:40.394219+0000","last_unstale":"2024-03-28T23:31:41.391247+0000","last_undegraded":"2024-03-28T23:31:41.391247+0000","last_fullsized":"2024-03-28T23:31:41.391247+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:18.654331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554191+0000","last_change":"2024-03-28T23:31:40.398232+0000","last_active":"2024-03-28T23:31:41.554191+0000","last_peered":"2024-03-28T23:31:41.554191+0000","last_clean":"2024-03-28T23:31:41.554191+0000","last_became_active":"2024-03-28T23:31:40.397888+0000","last_became_peered":"2024-03-28T23:31:40.397888+0000","last_unstale":"2024-03-28T23:31:41.554191+0000","last_undegraded":"2024-03-28T23:31:41.554191+0000","last_fullsized":"2024-03-28T23:31:41.554191+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:50.872886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"50'4","reported_seq":18,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554282+0000","last_change":"2024-03-28T23:31:39.385488+0000","last_active":"2024-03-28T23:31:41.554282+0000","last_peered":"2024-03-28T23:31:41.554282+0000","last_clean":"2024-03-28T23:31:41.554282+0000","last_became_active":"2024-03-28T23:31:39.385211+0000","last_became_peered":"2024-03-28T23:31:39.385211+0000","last_unstale":"2024-03-28T23:31:41.554282+0000","last_undegraded":"2024-03-28T23:31:41.554282+0000","last_fullsized":"2024-03-28T23:31:41.554282+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:50:10.559743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.390395+0000","last_change":"2024-03-28T23:31:40.390218+0000","last_active":"2024-03-28T23:31:41.390395+0000","last_peered":"2024-03-28T23:31:41.390395+0000","last_clean":"2024-03-28T23:31:41.390395+0000","last_became_active":"2024-03-28T23:31:40.389922+0000","last_became_peered":"2024-03-28T23:31:40.389922+0000","last_unstale":"2024-03-28T23:31:41.390395+0000","last_undegraded":"2024-03-28T23:31:41.390395+0000","last_fullsized":"2024-03-28T23:31:41.390395+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:14.515069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"50'1","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554834+0000","last_change":"2024-03-28T23:31:39.393513+0000","last_active":"2024-03-28T23:31:41.554834+0000","last_peered":"2024-03-28T23:31:41.554834+0000","last_clean":"2024-03-28T23:31:41.554834+0000","last_became_active":"2024-03-28T23:31:39.393176+0000","last_became_peered":"2024-03-28T23:31:39.393176+0000","last_unstale":"2024-03-28T23:31:41.554834+0000","last_undegraded":"2024-03-28T23:31:41.554834+0000","last_fullsized":"2024-03-28T23:31:41.554834+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:29.240985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554161+0000","last_change":"2024-03-28T23:31:40.397077+0000","last_active":"2024-03-28T23:31:41.554161+0000","last_peered":"2024-03-28T23:31:41.554161+0000","last_clean":"2024-03-28T23:31:41.554161+0000","last_became_active":"2024-03-28T23:31:40.395265+0000","last_became_peered":"2024-03-28T23:31:40.395265+0000","last_unstale":"2024-03-28T23:31:41.554161+0000","last_undegraded":"2024-03-28T23:31:41.554161+0000","last_fullsized":"2024-03-28T23:31:41.554161+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:04.829030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"50'2","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554396+0000","last_change":"2024-03-28T23:31:39.391743+0000","last_active":"2024-03-28T23:31:41.554396+0000","last_peered":"2024-03-28T23:31:41.554396+0000","last_clean":"2024-03-28T23:31:41.554396+0000","last_became_active":"2024-03-28T23:31:39.391300+0000","last_became_peered":"2024-03-28T23:31:39.391300+0000","last_unstale":"2024-03-28T23:31:41.554396+0000","last_undegraded":"2024-03-28T23:31:41.554396+0000","last_fullsized":"2024-03-28T23:31:41.554396+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:12.505382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.391002+0000","last_change":"2024-03-28T23:31:40.392391+0000","last_active":"2024-03-28T23:31:41.391002+0000","last_peered":"2024-03-28T23:31:41.391002+0000","last_clean":"2024-03-28T23:31:41.391002+0000","last_became_active":"2024-03-28T23:31:40.392100+0000","last_became_peered":"2024-03-28T23:31:40.392100+0000","last_unstale":"2024-03-28T23:31:41.391002+0000","last_undegraded":"2024-03-28T23:31:41.391002+0000","last_fullsized":"2024-03-28T23:31:41.391002+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:34:38.237729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"50'2","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.391005+0000","last_change":"2024-03-28T23:31:39.385430+0000","last_active":"2024-03-28T23:31:41.391005+0000","last_peered":"2024-03-28T23:31:41.391005+0000","last_clean":"2024-03-28T23:31:41.391005+0000","last_became_active":"2024-03-28T23:31:39.385134+0000","last_became_peered":"2024-03-28T23:31:39.385134+0000","last_unstale":"2024-03-28T23:31:41.391005+0000","last_undegraded":"2024-03-28T23:31:41.391005+0000","last_fullsized":"2024-03-28T23:31:41.391005+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:12:26.472924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.391818+0000","last_change":"2024-03-28T23:31:40.392757+0000","last_active":"2024-03-28T23:31:41.391818+0000","last_peered":"2024-03-28T23:31:41.391818+0000","last_clean":"2024-03-28T23:31:41.391818+0000","last_became_active":"2024-03-28T23:31:40.392460+0000","last_became_peered":"2024-03-28T23:31:40.392460+0000","last_unstale":"2024-03-28T23:31:41.391818+0000","last_undegraded":"2024-03-28T23:31:41.391818+0000","last_fullsized":"2024-03-28T23:31:41.391818+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:07.738674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"50'1","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.555139+0000","last_change":"2024-03-28T23:31:39.381607+0000","last_active":"2024-03-28T23:31:41.555139+0000","last_peered":"2024-03-28T23:31:41.555139+0000","last_clean":"2024-03-28T23:31:41.555139+0000","last_became_active":"2024-03-28T23:31:39.381326+0000","last_became_peered":"2024-03-28T23:31:39.381326+0000","last_unstale":"2024-03-28T23:31:41.555139+0000","last_undegraded":"2024-03-28T23:31:41.555139+0000","last_fullsized":"2024-03-28T23:31:41.555139+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:50.860190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554846+0000","last_change":"2024-03-28T23:31:40.398712+0000","last_active":"2024-03-28T23:31:41.554846+0000","last_peered":"2024-03-28T23:31:41.554846+0000","last_clean":"2024-03-28T23:31:41.554846+0000","last_became_active":"2024-03-28T23:31:40.398392+0000","last_became_peered":"2024-03-28T23:31:40.398392+0000","last_unstale":"2024-03-28T23:31:41.554846+0000","last_undegraded":"2024-03-28T23:31:41.554846+0000","last_fullsized":"2024-03-28T23:31:41.554846+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:54.431044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"50'1","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.391026+0000","last_change":"2024-03-28T23:31:39.384026+0000","last_active":"2024-03-28T23:31:41.391026+0000","last_peered":"2024-03-28T23:31:41.391026+0000","last_clean":"2024-03-28T23:31:41.391026+0000","last_became_active":"2024-03-28T23:31:39.383161+0000","last_became_peered":"2024-03-28T23:31:39.383161+0000","last_unstale":"2024-03-28T23:31:41.391026+0000","last_undegraded":"2024-03-28T23:31:41.391026+0000","last_fullsized":"2024-03-28T23:31:41.391026+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:21:00.890266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.390997+0000","last_change":"2024-03-28T23:31:40.395728+0000","last_active":"2024-03-28T23:31:41.390997+0000","last_peered":"2024-03-28T23:31:41.390997+0000","last_clean":"2024-03-28T23:31:41.390997+0000","last_became_active":"2024-03-28T23:31:40.395474+0000","last_became_peered":"2024-03-28T23:31:40.395474+0000","last_unstale":"2024-03-28T23:31:41.390997+0000","last_undegraded":"2024-03-28T23:31:41.390997+0000","last_fullsized":"2024-03-28T23:31:41.390997+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:15:19.985376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554132+0000","last_change":"2024-03-28T23:31:39.385649+0000","last_active":"2024-03-28T23:31:41.554132+0000","last_peered":"2024-03-28T23:31:41.554132+0000","last_clean":"2024-03-28T23:31:41.554132+0000","last_became_active":"2024-03-28T23:31:39.385341+0000","last_became_peered":"2024-03-28T23:31:39.385341+0000","last_unstale":"2024-03-28T23:31:41.554132+0000","last_undegraded":"2024-03-28T23:31:41.554132+0000","last_fullsized":"2024-03-28T23:31:41.554132+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:52:06.955653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554726+0000","last_change":"2024-03-28T23:31:40.400091+0000","last_active":"2024-03-28T23:31:41.554726+0000","last_peered":"2024-03-28T23:31:41.554726+0000","last_clean":"2024-03-28T23:31:41.554726+0000","last_became_active":"2024-03-28T23:31:40.399190+0000","last_became_peered":"2024-03-28T23:31:40.399190+0000","last_unstale":"2024-03-28T23:31:41.554726+0000","last_undegraded":"2024-03-28T23:31:41.554726+0000","last_fullsized":"2024-03-28T23:31:41.554726+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:22.230986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554998+0000","last_change":"2024-03-28T23:31:39.390198+0000","last_active":"2024-03-28T23:31:41.554998+0000","last_peered":"2024-03-28T23:31:41.554998+0000","last_clean":"2024-03-28T23:31:41.554998+0000","last_became_active":"2024-03-28T23:31:39.389939+0000","last_became_peered":"2024-03-28T23:31:39.389939+0000","last_unstale":"2024-03-28T23:31:41.554998+0000","last_undegraded":"2024-03-28T23:31:41.554998+0000","last_fullsized":"2024-03-28T23:31:41.554998+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:46:30.350009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.390920+0000","last_change":"2024-03-28T23:31:40.384948+0000","last_active":"2024-03-28T23:31:41.390920+0000","last_peered":"2024-03-28T23:31:41.390920+0000","last_clean":"2024-03-28T23:31:41.390920+0000","last_became_active":"2024-03-28T23:31:40.384434+0000","last_became_peered":"2024-03-28T23:31:40.384434+0000","last_unstale":"2024-03-28T23:31:41.390920+0000","last_undegraded":"2024-03-28T23:31:41.390920+0000","last_fullsized":"2024-03-28T23:31:41.390920+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:04.829030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554582+0000","last_change":"2024-03-28T23:31:39.388914+0000","last_active":"2024-03-28T23:31:41.554582+0000","last_peered":"2024-03-28T23:31:41.554582+0000","last_clean":"2024-03-28T23:31:41.554582+0000","last_became_active":"2024-03-28T23:31:39.388639+0000","last_became_peered":"2024-03-28T23:31:39.388639+0000","last_unstale":"2024-03-28T23:31:41.554582+0000","last_undegraded":"2024-03-28T23:31:41.554582+0000","last_fullsized":"2024-03-28T23:31:41.554582+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:00.366278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.391294+0000","last_change":"2024-03-28T23:31:40.386749+0000","last_active":"2024-03-28T23:31:41.391294+0000","last_peered":"2024-03-28T23:31:41.391294+0000","last_clean":"2024-03-28T23:31:41.391294+0000","last_became_active":"2024-03-28T23:31:40.386466+0000","last_became_peered":"2024-03-28T23:31:40.386466+0000","last_unstale":"2024-03-28T23:31:41.391294+0000","last_undegraded":"2024-03-28T23:31:41.391294+0000","last_fullsized":"2024-03-28T23:31:41.391294+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:06.567694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.391523+0000","last_change":"2024-03-28T23:31:39.381050+0000","last_active":"2024-03-28T23:31:41.391523+0000","last_peered":"2024-03-28T23:31:41.391523+0000","last_clean":"2024-03-28T23:31:41.391523+0000","last_became_active":"2024-03-28T23:31:39.380761+0000","last_became_peered":"2024-03-28T23:31:39.380761+0000","last_unstale":"2024-03-28T23:31:41.391523+0000","last_undegraded":"2024-03-28T23:31:41.391523+0000","last_fullsized":"2024-03-28T23:31:41.391523+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:29.240985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554611+0000","last_change":"2024-03-28T23:31:40.400388+0000","last_active":"2024-03-28T23:31:41.554611+0000","last_peered":"2024-03-28T23:31:41.554611+0000","last_clean":"2024-03-28T23:31:41.554611+0000","last_became_active":"2024-03-28T23:31:40.400133+0000","last_became_peered":"2024-03-28T23:31:40.400133+0000","last_unstale":"2024-03-28T23:31:41.554611+0000","last_undegraded":"2024-03-28T23:31:41.554611+0000","last_fullsized":"2024-03-28T23:31:41.554611+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:03:56.252539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"50'3","reported_seq":17,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554898+0000","last_change":"2024-03-28T23:31:39.386478+0000","last_active":"2024-03-28T23:31:41.554898+0000","last_peered":"2024-03-28T23:31:41.554898+0000","last_clean":"2024-03-28T23:31:41.554898+0000","last_became_active":"2024-03-28T23:31:39.385868+0000","last_became_peered":"2024-03-28T23:31:39.385868+0000","last_unstale":"2024-03-28T23:31:41.554898+0000","last_undegraded":"2024-03-28T23:31:41.554898+0000","last_fullsized":"2024-03-28T23:31:41.554898+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:06.733879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.391492+0000","last_change":"2024-03-28T23:31:40.387795+0000","last_active":"2024-03-28T23:31:41.391492+0000","last_peered":"2024-03-28T23:31:41.391492+0000","last_clean":"2024-03-28T23:31:41.391492+0000","last_became_active":"2024-03-28T23:31:40.387529+0000","last_became_peered":"2024-03-28T23:31:40.387529+0000","last_unstale":"2024-03-28T23:31:41.391492+0000","last_undegraded":"2024-03-28T23:31:41.391492+0000","last_fullsized":"2024-03-28T23:31:41.391492+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:51.864986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"50'6","reported_seq":20,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.390609+0000","last_change":"2024-03-28T23:31:39.393411+0000","last_active":"2024-03-28T23:31:41.390609+0000","last_peered":"2024-03-28T23:31:41.390609+0000","last_clean":"2024-03-28T23:31:41.390609+0000","last_became_active":"2024-03-28T23:31:39.393166+0000","last_became_peered":"2024-03-28T23:31:39.393166+0000","last_unstale":"2024-03-28T23:31:41.390609+0000","last_undegraded":"2024-03-28T23:31:41.390609+0000","last_fullsized":"2024-03-28T23:31:41.390609+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:13.510273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.390578+0000","last_change":"2024-03-28T23:31:40.390497+0000","last_active":"2024-03-28T23:31:41.390578+0000","last_peered":"2024-03-28T23:31:41.390578+0000","last_clean":"2024-03-28T23:31:41.390578+0000","last_became_active":"2024-03-28T23:31:40.390167+0000","last_became_peered":"2024-03-28T23:31:40.390167+0000","last_unstale":"2024-03-28T23:31:41.390578+0000","last_undegraded":"2024-03-28T23:31:41.390578+0000","last_fullsized":"2024-03-28T23:31:41.390578+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:54.431044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"50'2","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.555229+0000","last_change":"2024-03-28T23:31:39.380369+0000","last_active":"2024-03-28T23:31:41.555229+0000","last_peered":"2024-03-28T23:31:41.555229+0000","last_clean":"2024-03-28T23:31:41.555229+0000","last_became_active":"2024-03-28T23:31:39.380092+0000","last_became_peered":"2024-03-28T23:31:39.380092+0000","last_unstale":"2024-03-28T23:31:41.555229+0000","last_undegraded":"2024-03-28T23:31:41.555229+0000","last_fullsized":"2024-03-28T23:31:41.555229+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:00.366278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.555424+0000","last_change":"2024-03-28T23:31:40.390579+0000","last_active":"2024-03-28T23:31:41.555424+0000","last_peered":"2024-03-28T23:31:41.555424+0000","last_clean":"2024-03-28T23:31:41.555424+0000","last_became_active":"2024-03-28T23:31:40.390248+0000","last_became_peered":"2024-03-28T23:31:40.390248+0000","last_unstale":"2024-03-28T23:31:41.555424+0000","last_undegraded":"2024-03-28T23:31:41.555424+0000","last_fullsized":"2024-03-28T23:31:41.555424+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:47.861300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554134+0000","last_change":"2024-03-28T23:31:39.389592+0000","last_active":"2024-03-28T23:31:41.554134+0000","last_peered":"2024-03-28T23:31:41.554134+0000","last_clean":"2024-03-28T23:31:41.554134+0000","last_became_active":"2024-03-28T23:31:39.389302+0000","last_became_peered":"2024-03-28T23:31:39.389302+0000","last_unstale":"2024-03-28T23:31:41.554134+0000","last_undegraded":"2024-03-28T23:31:41.554134+0000","last_fullsized":"2024-03-28T23:31:41.554134+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:59:51.501145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.390871+0000","last_change":"2024-03-28T23:31:40.387085+0000","last_active":"2024-03-28T23:31:41.390871+0000","last_peered":"2024-03-28T23:31:41.390871+0000","last_clean":"2024-03-28T23:31:41.390871+0000","last_became_active":"2024-03-28T23:31:40.386695+0000","last_became_peered":"2024-03-28T23:31:40.386695+0000","last_unstale":"2024-03-28T23:31:41.390871+0000","last_undegraded":"2024-03-28T23:31:41.390871+0000","last_fullsized":"2024-03-28T23:31:41.390871+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:40:57.541676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554769+0000","last_change":"2024-03-28T23:31:39.384653+0000","last_active":"2024-03-28T23:31:41.554769+0000","last_peered":"2024-03-28T23:31:41.554769+0000","last_clean":"2024-03-28T23:31:41.554769+0000","last_became_active":"2024-03-28T23:31:39.383495+0000","last_became_peered":"2024-03-28T23:31:39.383495+0000","last_unstale":"2024-03-28T23:31:41.554769+0000","last_undegraded":"2024-03-28T23:31:41.554769+0000","last_fullsized":"2024-03-28T23:31:41.554769+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:50.860190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.555488+0000","last_change":"2024-03-28T23:31:40.403701+0000","last_active":"2024-03-28T23:31:41.555488+0000","last_peered":"2024-03-28T23:31:41.555488+0000","last_clean":"2024-03-28T23:31:41.555488+0000","last_became_active":"2024-03-28T23:31:40.403222+0000","last_became_peered":"2024-03-28T23:31:40.403222+0000","last_unstale":"2024-03-28T23:31:41.555488+0000","last_undegraded":"2024-03-28T23:31:41.555488+0000","last_fullsized":"2024-03-28T23:31:41.555488+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:40:09.867914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"50'8","reported_seq":22,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.390722+0000","last_change":"2024-03-28T23:31:39.383726+0000","last_active":"2024-03-28T23:31:41.390722+0000","last_peered":"2024-03-28T23:31:41.390722+0000","last_clean":"2024-03-28T23:31:41.390722+0000","last_became_active":"2024-03-28T23:31:39.383292+0000","last_became_peered":"2024-03-28T23:31:39.383292+0000","last_unstale":"2024-03-28T23:31:41.390722+0000","last_undegraded":"2024-03-28T23:31:41.390722+0000","last_fullsized":"2024-03-28T23:31:41.390722+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:25:38.113638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.390839+0000","last_change":"2024-03-28T23:31:40.391219+0000","last_active":"2024-03-28T23:31:41.390839+0000","last_peered":"2024-03-28T23:31:41.390839+0000","last_clean":"2024-03-28T23:31:41.390839+0000","last_became_active":"2024-03-28T23:31:40.389552+0000","last_became_peered":"2024-03-28T23:31:40.389552+0000","last_unstale":"2024-03-28T23:31:41.390839+0000","last_undegraded":"2024-03-28T23:31:41.390839+0000","last_fullsized":"2024-03-28T23:31:41.390839+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:42:29.351941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"50'1","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.555610+0000","last_change":"2024-03-28T23:31:39.390253+0000","last_active":"2024-03-28T23:31:41.555610+0000","last_peered":"2024-03-28T23:31:41.555610+0000","last_clean":"2024-03-28T23:31:41.555610+0000","last_became_active":"2024-03-28T23:31:39.389980+0000","last_became_peered":"2024-03-28T23:31:39.389980+0000","last_unstale":"2024-03-28T23:31:41.555610+0000","last_undegraded":"2024-03-28T23:31:41.555610+0000","last_fullsized":"2024-03-28T23:31:41.555610+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:24.520296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.553920+0000","last_change":"2024-03-28T23:31:40.391898+0000","last_active":"2024-03-28T23:31:41.553920+0000","last_peered":"2024-03-28T23:31:41.553920+0000","last_clean":"2024-03-28T23:31:41.553920+0000","last_became_active":"2024-03-28T23:31:40.391622+0000","last_became_peered":"2024-03-28T23:31:40.391622+0000","last_unstale":"2024-03-28T23:31:41.553920+0000","last_undegraded":"2024-03-28T23:31:41.553920+0000","last_fullsized":"2024-03-28T23:31:41.553920+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:38:22.055287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"50'2","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.555210+0000","last_change":"2024-03-28T23:31:39.385339+0000","last_active":"2024-03-28T23:31:41.555210+0000","last_peered":"2024-03-28T23:31:41.555210+0000","last_clean":"2024-03-28T23:31:41.555210+0000","last_became_active":"2024-03-28T23:31:39.385075+0000","last_became_peered":"2024-03-28T23:31:39.385075+0000","last_unstale":"2024-03-28T23:31:41.555210+0000","last_undegraded":"2024-03-28T23:31:41.555210+0000","last_fullsized":"2024-03-28T23:31:41.555210+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:58.098710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.390956+0000","last_change":"2024-03-28T23:31:40.399291+0000","last_active":"2024-03-28T23:31:41.390956+0000","last_peered":"2024-03-28T23:31:41.390956+0000","last_clean":"2024-03-28T23:31:41.390956+0000","last_became_active":"2024-03-28T23:31:40.399005+0000","last_became_peered":"2024-03-28T23:31:40.399005+0000","last_unstale":"2024-03-28T23:31:41.390956+0000","last_undegraded":"2024-03-28T23:31:41.390956+0000","last_fullsized":"2024-03-28T23:31:41.390956+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:01.371074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"50'1","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554231+0000","last_change":"2024-03-28T23:31:39.389465+0000","last_active":"2024-03-28T23:31:41.554231+0000","last_peered":"2024-03-28T23:31:41.554231+0000","last_clean":"2024-03-28T23:31:41.554231+0000","last_became_active":"2024-03-28T23:31:39.389005+0000","last_became_peered":"2024-03-28T23:31:39.389005+0000","last_unstale":"2024-03-28T23:31:41.554231+0000","last_undegraded":"2024-03-28T23:31:41.554231+0000","last_fullsized":"2024-03-28T23:31:41.554231+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:41.430571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554568+0000","last_change":"2024-03-28T23:31:40.388141+0000","last_active":"2024-03-28T23:31:41.554568+0000","last_peered":"2024-03-28T23:31:41.554568+0000","last_clean":"2024-03-28T23:31:41.554568+0000","last_became_active":"2024-03-28T23:31:40.387852+0000","last_became_peered":"2024-03-28T23:31:40.387852+0000","last_unstale":"2024-03-28T23:31:41.554568+0000","last_undegraded":"2024-03-28T23:31:41.554568+0000","last_fullsized":"2024-03-28T23:31:41.554568+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:22.230986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554649+0000","last_change":"2024-03-28T23:31:39.390465+0000","last_active":"2024-03-28T23:31:41.554649+0000","last_peered":"2024-03-28T23:31:41.554649+0000","last_clean":"2024-03-28T23:31:41.554649+0000","last_became_active":"2024-03-28T23:31:39.390203+0000","last_became_peered":"2024-03-28T23:31:39.390203+0000","last_unstale":"2024-03-28T23:31:41.554649+0000","last_undegraded":"2024-03-28T23:31:41.554649+0000","last_fullsized":"2024-03-28T23:31:41.554649+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:50:50.496294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.390723+0000","last_change":"2024-03-28T23:31:40.387376+0000","last_active":"2024-03-28T23:31:41.390723+0000","last_peered":"2024-03-28T23:31:41.390723+0000","last_clean":"2024-03-28T23:31:41.390723+0000","last_became_active":"2024-03-28T23:31:40.387080+0000","last_became_peered":"2024-03-28T23:31:40.387080+0000","last_unstale":"2024-03-28T23:31:41.390723+0000","last_undegraded":"2024-03-28T23:31:41.390723+0000","last_fullsized":"2024-03-28T23:31:41.390723+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:43.111058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"50'2","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554616+0000","last_change":"2024-03-28T23:31:39.388867+0000","last_active":"2024-03-28T23:31:41.554616+0000","last_peered":"2024-03-28T23:31:41.554616+0000","last_clean":"2024-03-28T23:31:41.554616+0000","last_became_active":"2024-03-28T23:31:39.388593+0000","last_became_peered":"2024-03-28T23:31:39.388593+0000","last_unstale":"2024-03-28T23:31:41.554616+0000","last_undegraded":"2024-03-28T23:31:41.554616+0000","last_fullsized":"2024-03-28T23:31:41.554616+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:21:00.890266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554358+0000","last_change":"2024-03-28T23:31:40.393257+0000","last_active":"2024-03-28T23:31:41.554358+0000","last_peered":"2024-03-28T23:31:41.554358+0000","last_clean":"2024-03-28T23:31:41.554358+0000","last_became_active":"2024-03-28T23:31:40.391684+0000","last_became_peered":"2024-03-28T23:31:40.391684+0000","last_unstale":"2024-03-28T23:31:41.554358+0000","last_undegraded":"2024-03-28T23:31:41.554358+0000","last_fullsized":"2024-03-28T23:31:41.554358+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:01.371074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.390562+0000","last_change":"2024-03-28T23:31:39.388198+0000","last_active":"2024-03-28T23:31:41.390562+0000","last_peered":"2024-03-28T23:31:41.390562+0000","last_clean":"2024-03-28T23:31:41.390562+0000","last_became_active":"2024-03-28T23:31:39.387727+0000","last_became_peered":"2024-03-28T23:31:39.387727+0000","last_unstale":"2024-03-28T23:31:41.390562+0000","last_undegraded":"2024-03-28T23:31:41.390562+0000","last_fullsized":"2024-03-28T23:31:41.390562+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:49.868091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.555476+0000","last_change":"2024-03-28T23:31:40.394047+0000","last_active":"2024-03-28T23:31:41.555476+0000","last_peered":"2024-03-28T23:31:41.555476+0000","last_clean":"2024-03-28T23:31:41.555476+0000","last_became_active":"2024-03-28T23:31:40.393772+0000","last_became_peered":"2024-03-28T23:31:40.393772+0000","last_unstale":"2024-03-28T23:31:41.555476+0000","last_undegraded":"2024-03-28T23:31:41.555476+0000","last_fullsized":"2024-03-28T23:31:41.555476+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:07:13.690791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554972+0000","last_change":"2024-03-28T23:31:39.388829+0000","last_active":"2024-03-28T23:31:41.554972+0000","last_peered":"2024-03-28T23:31:41.554972+0000","last_clean":"2024-03-28T23:31:41.554972+0000","last_became_active":"2024-03-28T23:31:39.388521+0000","last_became_peered":"2024-03-28T23:31:39.388521+0000","last_unstale":"2024-03-28T23:31:41.554972+0000","last_undegraded":"2024-03-28T23:31:41.554972+0000","last_fullsized":"2024-03-28T23:31:41.554972+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:23:52.148574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554283+0000","last_change":"2024-03-28T23:31:40.384709+0000","last_active":"2024-03-28T23:31:41.554283+0000","last_peered":"2024-03-28T23:31:41.554283+0000","last_clean":"2024-03-28T23:31:41.554283+0000","last_became_active":"2024-03-28T23:31:40.384412+0000","last_became_peered":"2024-03-28T23:31:40.384412+0000","last_unstale":"2024-03-28T23:31:41.554283+0000","last_undegraded":"2024-03-28T23:31:41.554283+0000","last_fullsized":"2024-03-28T23:31:41.554283+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:25:39.118434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554021+0000","last_change":"2024-03-28T23:31:39.385937+0000","last_active":"2024-03-28T23:31:41.554021+0000","last_peered":"2024-03-28T23:31:41.554021+0000","last_clean":"2024-03-28T23:31:41.554021+0000","last_became_active":"2024-03-28T23:31:39.385675+0000","last_became_peered":"2024-03-28T23:31:39.385675+0000","last_unstale":"2024-03-28T23:31:41.554021+0000","last_undegraded":"2024-03-28T23:31:41.554021+0000","last_fullsized":"2024-03-28T23:31:41.554021+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:35.204851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.390351+0000","last_change":"2024-03-28T23:31:40.387783+0000","last_active":"2024-03-28T23:31:41.390351+0000","last_peered":"2024-03-28T23:31:41.390351+0000","last_clean":"2024-03-28T23:31:41.390351+0000","last_became_active":"2024-03-28T23:31:40.387487+0000","last_became_peered":"2024-03-28T23:31:40.387487+0000","last_unstale":"2024-03-28T23:31:41.390351+0000","last_undegraded":"2024-03-28T23:31:41.390351+0000","last_fullsized":"2024-03-28T23:31:41.390351+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:22.230986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"50'2","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554434+0000","last_change":"2024-03-28T23:31:39.391741+0000","last_active":"2024-03-28T23:31:41.554434+0000","last_peered":"2024-03-28T23:31:41.554434+0000","last_clean":"2024-03-28T23:31:41.554434+0000","last_became_active":"2024-03-28T23:31:39.391520+0000","last_became_peered":"2024-03-28T23:31:39.391520+0000","last_unstale":"2024-03-28T23:31:41.554434+0000","last_undegraded":"2024-03-28T23:31:41.554434+0000","last_fullsized":"2024-03-28T23:31:41.554434+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:16:42.869555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554224+0000","last_change":"2024-03-28T23:31:40.387330+0000","last_active":"2024-03-28T23:31:41.554224+0000","last_peered":"2024-03-28T23:31:41.554224+0000","last_clean":"2024-03-28T23:31:41.554224+0000","last_became_active":"2024-03-28T23:31:40.387076+0000","last_became_peered":"2024-03-28T23:31:40.387076+0000","last_unstale":"2024-03-28T23:31:41.554224+0000","last_undegraded":"2024-03-28T23:31:41.554224+0000","last_fullsized":"2024-03-28T23:31:41.554224+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:25.525091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554904+0000","last_change":"2024-03-28T23:31:39.386686+0000","last_active":"2024-03-28T23:31:41.554904+0000","last_peered":"2024-03-28T23:31:41.554904+0000","last_clean":"2024-03-28T23:31:41.554904+0000","last_became_active":"2024-03-28T23:31:39.386208+0000","last_became_peered":"2024-03-28T23:31:39.386208+0000","last_unstale":"2024-03-28T23:31:41.554904+0000","last_undegraded":"2024-03-28T23:31:41.554904+0000","last_fullsized":"2024-03-28T23:31:41.554904+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:22.210570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.390836+0000","last_change":"2024-03-28T23:31:40.386143+0000","last_active":"2024-03-28T23:31:41.390836+0000","last_peered":"2024-03-28T23:31:41.390836+0000","last_clean":"2024-03-28T23:31:41.390836+0000","last_became_active":"2024-03-28T23:31:40.385606+0000","last_became_peered":"2024-03-28T23:31:40.385606+0000","last_unstale":"2024-03-28T23:31:41.390836+0000","last_undegraded":"2024-03-28T23:31:41.390836+0000","last_fullsized":"2024-03-28T23:31:41.390836+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:40:09.867914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554204+0000","last_change":"2024-03-28T23:31:39.380624+0000","last_active":"2024-03-28T23:31:41.554204+0000","last_peered":"2024-03-28T23:31:41.554204+0000","last_clean":"2024-03-28T23:31:41.554204+0000","last_became_active":"2024-03-28T23:31:39.380340+0000","last_became_peered":"2024-03-28T23:31:39.380340+0000","last_unstale":"2024-03-28T23:31:41.554204+0000","last_undegraded":"2024-03-28T23:31:41.554204+0000","last_fullsized":"2024-03-28T23:31:41.554204+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:22.210570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554151+0000","last_change":"2024-03-28T23:31:40.404359+0000","last_active":"2024-03-28T23:31:41.554151+0000","last_peered":"2024-03-28T23:31:41.554151+0000","last_clean":"2024-03-28T23:31:41.554151+0000","last_became_active":"2024-03-28T23:31:40.404078+0000","last_became_peered":"2024-03-28T23:31:40.404078+0000","last_unstale":"2024-03-28T23:31:41.554151+0000","last_undegraded":"2024-03-28T23:31:41.554151+0000","last_fullsized":"2024-03-28T23:31:41.554151+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:18.654331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.390889+0000","last_change":"2024-03-28T23:31:39.393283+0000","last_active":"2024-03-28T23:31:41.390889+0000","last_peered":"2024-03-28T23:31:41.390889+0000","last_clean":"2024-03-28T23:31:41.390889+0000","last_became_active":"2024-03-28T23:31:39.393035+0000","last_became_peered":"2024-03-28T23:31:39.393035+0000","last_unstale":"2024-03-28T23:31:41.390889+0000","last_undegraded":"2024-03-28T23:31:41.390889+0000","last_fullsized":"2024-03-28T23:31:41.390889+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:13.139644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.390487+0000","last_change":"2024-03-28T23:31:40.392819+0000","last_active":"2024-03-28T23:31:41.390487+0000","last_peered":"2024-03-28T23:31:41.390487+0000","last_clean":"2024-03-28T23:31:41.390487+0000","last_became_active":"2024-03-28T23:31:40.392501+0000","last_became_peered":"2024-03-28T23:31:40.392501+0000","last_unstale":"2024-03-28T23:31:41.390487+0000","last_undegraded":"2024-03-28T23:31:41.390487+0000","last_fullsized":"2024-03-28T23:31:41.390487+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:14.144440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.390648+0000","last_change":"2024-03-28T23:31:39.387993+0000","last_active":"2024-03-28T23:31:41.390648+0000","last_peered":"2024-03-28T23:31:41.390648+0000","last_clean":"2024-03-28T23:31:41.390648+0000","last_became_active":"2024-03-28T23:31:39.387393+0000","last_became_peered":"2024-03-28T23:31:39.387393+0000","last_unstale":"2024-03-28T23:31:41.390648+0000","last_undegraded":"2024-03-28T23:31:41.390648+0000","last_fullsized":"2024-03-28T23:31:41.390648+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:21.226190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.555445+0000","last_change":"2024-03-28T23:31:40.405766+0000","last_active":"2024-03-28T23:31:41.555445+0000","last_peered":"2024-03-28T23:31:41.555445+0000","last_clean":"2024-03-28T23:31:41.555445+0000","last_became_active":"2024-03-28T23:31:40.405517+0000","last_became_peered":"2024-03-28T23:31:40.405517+0000","last_unstale":"2024-03-28T23:31:41.555445+0000","last_undegraded":"2024-03-28T23:31:41.555445+0000","last_fullsized":"2024-03-28T23:31:41.555445+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:18:16.851435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.390792+0000","last_change":"2024-03-28T23:31:39.383748+0000","last_active":"2024-03-28T23:31:41.390792+0000","last_peered":"2024-03-28T23:31:41.390792+0000","last_clean":"2024-03-28T23:31:41.390792+0000","last_became_active":"2024-03-28T23:31:39.383070+0000","last_became_peered":"2024-03-28T23:31:39.383070+0000","last_unstale":"2024-03-28T23:31:41.390792+0000","last_undegraded":"2024-03-28T23:31:41.390792+0000","last_fullsized":"2024-03-28T23:31:41.390792+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:41.430571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.391088+0000","last_change":"2024-03-28T23:31:40.392625+0000","last_active":"2024-03-28T23:31:41.391088+0000","last_peered":"2024-03-28T23:31:41.391088+0000","last_clean":"2024-03-28T23:31:41.391088+0000","last_became_active":"2024-03-28T23:31:40.392359+0000","last_became_peered":"2024-03-28T23:31:40.392359+0000","last_unstale":"2024-03-28T23:31:41.391088+0000","last_undegraded":"2024-03-28T23:31:41.391088+0000","last_fullsized":"2024-03-28T23:31:41.391088+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:02.676183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.390413+0000","last_change":"2024-03-28T23:31:39.389478+0000","last_active":"2024-03-28T23:31:41.390413+0000","last_peered":"2024-03-28T23:31:41.390413+0000","last_clean":"2024-03-28T23:31:41.390413+0000","last_became_active":"2024-03-28T23:31:39.389154+0000","last_became_peered":"2024-03-28T23:31:39.389154+0000","last_unstale":"2024-03-28T23:31:41.390413+0000","last_undegraded":"2024-03-28T23:31:41.390413+0000","last_fullsized":"2024-03-28T23:31:41.390413+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:25:19.055737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554046+0000","last_change":"2024-03-28T23:31:40.389210+0000","last_active":"2024-03-28T23:31:41.554046+0000","last_peered":"2024-03-28T23:31:41.554046+0000","last_clean":"2024-03-28T23:31:41.554046+0000","last_became_active":"2024-03-28T23:31:40.388573+0000","last_became_peered":"2024-03-28T23:31:40.388573+0000","last_unstale":"2024-03-28T23:31:41.554046+0000","last_undegraded":"2024-03-28T23:31:41.554046+0000","last_fullsized":"2024-03-28T23:31:41.554046+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:23:53.153369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"51'1","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.555532+0000","last_change":"2024-03-28T23:31:39.381202+0000","last_active":"2024-03-28T23:31:41.555532+0000","last_peered":"2024-03-28T23:31:41.555532+0000","last_clean":"2024-03-28T23:31:41.555532+0000","last_became_active":"2024-03-28T23:31:39.380913+0000","last_became_peered":"2024-03-28T23:31:39.380913+0000","last_unstale":"2024-03-28T23:31:41.555532+0000","last_undegraded":"2024-03-28T23:31:41.555532+0000","last_fullsized":"2024-03-28T23:31:41.555532+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:38:21.050492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":229,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-28T23:31:41.554299+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:31:41.554299+0000","last_peered":"2024-03-28T23:31:41.554299+0000","last_clean":"2024-03-28T23:31:41.554299+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:31:41.554299+0000","last_undegraded":"2024-03-28T23:31:41.554299+0000","last_fullsized":"2024-03-28T23:31:41.554299+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574875,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":679936,"data_stored":860892,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31780,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340060,"statfs":{"total":96636764160,"available":96604221440,"internally_reserved":0,"allocated":614400,"data_stored":271052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31772,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340068,"statfs":{"total":96636764160,"available":96604229632,"internally_reserved":0,"allocated":606208,"data_stored":270400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31776,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340064,"statfs":{"total":96636764160,"available":96604225536,"internally_reserved":0,"allocated":610304,"data_stored":270982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574875,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":622592,"data_stored":272034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":618496,"data_stored":272012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":614400,"data_stored":270560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574875,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":679936,"data_stored":860824,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:31:48.149 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T23:31:48.149 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-28T23:31:48.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.308+0000 7f937429d640 1 Processor -- start 2024-03-28T23:31:48.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.308+0000 7f937429d640 1 -- start start 2024-03-28T23:31:48.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.309+0000 7f937429d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f936c06bb60 0x7f936c0c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:48.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.309+0000 7f937429d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f936c152db0 con 0x7f936c0c6750 2024-03-28T23:31:48.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.309+0000 7f937429d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f936c0c7210 con 0x7f936c1529d0 2024-03-28T23:31:48.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.309+0000 7f937429d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f936c0c6c50 con 0x7f936c06bb60 2024-03-28T23:31:48.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.309+0000 7f937283a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f936c06bb60 0x7f936c0c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:48.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.309+0000 7f9371838640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f936c0c6750 0x7f936c0c54f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44032/0 (socket says 172.21.15.161:44032) 2024-03-28T23:31:48.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.309+0000 7f9371838640 1 -- 172.21.15.161:0/2395142890 learned_addr learned my addr 172.21.15.161:0/2395142890 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:48.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.310+0000 7f937283a640 1 -- 172.21.15.161:0/2395142890 >> v1:172.21.15.170:6789/0 conn(0x7f936c1529d0 legacy=0x7f936c155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:48.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.310+0000 7f937283a640 1 -- 172.21.15.161:0/2395142890 >> v1:172.21.15.161:6789/0 conn(0x7f936c0c6750 legacy=0x7f936c0c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:48.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.310+0000 7f937283a640 1 -- 172.21.15.161:0/2395142890 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f936c0c6f30 con 0x7f936c06bb60 2024-03-28T23:31:48.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.310+0000 7f9371037640 1 -- 172.21.15.161:0/2395142890 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2350884322 0 0) 0x7f936c152db0 con 0x7f936c0c6750 2024-03-28T23:31:48.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.310+0000 7f937283a640 1 --2- 172.21.15.161:0/2395142890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f936c06bb60 0x7f936c0c88b0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f9364009d90 tx=0x7f936402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6498e38b50a3969c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:48.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.311+0000 7f9371037640 1 -- 172.21.15.161:0/2395142890 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9364039070 con 0x7f936c06bb60 2024-03-28T23:31:48.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.311+0000 7f9371037640 1 -- 172.21.15.161:0/2395142890 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f936402cc60 con 0x7f936c06bb60 2024-03-28T23:31:48.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.311+0000 7f9371037640 1 -- 172.21.15.161:0/2395142890 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9364034660 con 0x7f936c06bb60 2024-03-28T23:31:48.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.311+0000 7f937429d640 1 -- 172.21.15.161:0/2395142890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f936c06bb60 msgr2=0x7f936c0c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:48.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.311+0000 7f937429d640 1 --2- 172.21.15.161:0/2395142890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f936c06bb60 0x7f936c0c88b0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f9364009d90 tx=0x7f936402beb0 comp rx=0 tx=0).stop 2024-03-28T23:31:48.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.311+0000 7f937429d640 1 -- 172.21.15.161:0/2395142890 shutdown_connections 2024-03-28T23:31:48.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.311+0000 7f937429d640 1 --2- 172.21.15.161:0/2395142890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f936c06bb60 0x7f936c0c88b0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:48.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.311+0000 7f937429d640 1 -- 172.21.15.161:0/2395142890 >> 172.21.15.161:0/2395142890 conn(0x7f936c0c7430 msgr2=0x7f936c0c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:48.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.311+0000 7f937429d640 1 -- 172.21.15.161:0/2395142890 shutdown_connections 2024-03-28T23:31:48.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.311+0000 7f937429d640 1 -- 172.21.15.161:0/2395142890 wait complete. 2024-03-28T23:31:48.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.312+0000 7f937429d640 1 Processor -- start 2024-03-28T23:31:48.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.312+0000 7f937429d640 1 -- start start 2024-03-28T23:31:48.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.313+0000 7f937429d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f936c06bb60 0x7f936c1eada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:48.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.313+0000 7f937429d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f936c0c6750 0x7f936c1eead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:48.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.313+0000 7f937429d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f936c1529d0 0x7f936c14b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:48.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.313+0000 7f937429d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f936c0aa160 con 0x7f936c06bb60 2024-03-28T23:31:48.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.313+0000 7f937429d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f936c155710 con 0x7f936c1529d0 2024-03-28T23:31:48.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.313+0000 7f937429d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f936c0c5c00 con 0x7f936c0c6750 2024-03-28T23:31:48.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.313+0000 7f9372039640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f936c06bb60 0x7f936c1eada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:48.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.313+0000 7f9372039640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f936c06bb60 0x7f936c1eada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58510/0 (socket says 172.21.15.161:58510) 2024-03-28T23:31:48.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.313+0000 7f9372039640 1 -- 172.21.15.161:0/3415320162 learned_addr learned my addr 172.21.15.161:0/3415320162 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:48.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.313+0000 7f9371838640 1 --2- 172.21.15.161:0/3415320162 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f936c0c6750 0x7f936c1eead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:48.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.313+0000 7f937283a640 1 --2- 172.21.15.161:0/3415320162 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f936c1529d0 0x7f936c14b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:48.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.313+0000 7f9372039640 1 -- 172.21.15.161:0/3415320162 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f936c0c6750 msgr2=0x7f936c1eead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:48.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.313+0000 7f9372039640 1 --2- 172.21.15.161:0/3415320162 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f936c0c6750 0x7f936c1eead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:48.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.313+0000 7f9372039640 1 -- 172.21.15.161:0/3415320162 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f936c1529d0 msgr2=0x7f936c14b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:48.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.313+0000 7f9372039640 1 --2- 172.21.15.161:0/3415320162 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f936c1529d0 0x7f936c14b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:48.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.313+0000 7f9372039640 1 -- 172.21.15.161:0/3415320162 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f936c154a20 con 0x7f936c06bb60 2024-03-28T23:31:48.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.313+0000 7f937283a640 1 --2- 172.21.15.161:0/3415320162 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f936c1529d0 0x7f936c14b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:48.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.313+0000 7f9372039640 1 --2- 172.21.15.161:0/3415320162 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f936c06bb60 0x7f936c1eada0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f936800b3f0 tx=0x7f936800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:48.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.314+0000 7f9362ffd640 1 -- 172.21.15.161:0/3415320162 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9368015020 con 0x7f936c06bb60 2024-03-28T23:31:48.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.314+0000 7f9362ffd640 1 -- 172.21.15.161:0/3415320162 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9368011830 con 0x7f936c06bb60 2024-03-28T23:31:48.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.314+0000 7f9362ffd640 1 -- 172.21.15.161:0/3415320162 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9368011c20 con 0x7f936c06bb60 2024-03-28T23:31:48.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.314+0000 7f937429d640 1 -- 172.21.15.161:0/3415320162 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f936c154d00 con 0x7f936c06bb60 2024-03-28T23:31:48.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.314+0000 7f937429d640 1 -- 172.21.15.161:0/3415320162 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f936c1ee5b0 con 0x7f936c06bb60 2024-03-28T23:31:48.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.315+0000 7f937429d640 1 -- 172.21.15.161:0/3415320162 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f936c04f4b0 con 0x7f936c06bb60 2024-03-28T23:31:48.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.315+0000 7f9362ffd640 1 -- 172.21.15.161:0/3415320162 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9368018020 con 0x7f936c06bb60 2024-03-28T23:31:48.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.315+0000 7f9362ffd640 1 --2- 172.21.15.161:0/3415320162 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f935006b4d0 0x7f935006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:48.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.315+0000 7f9362ffd640 1 -- 172.21.15.161:0/3415320162 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f9368089ef0 con 0x7f936c06bb60 2024-03-28T23:31:48.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.316+0000 7f9371838640 1 --2- 172.21.15.161:0/3415320162 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f935006b4d0 0x7f935006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:48.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.316+0000 7f9371838640 1 --2- 172.21.15.161:0/3415320162 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f935006b4d0 0x7f935006d990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f936c0740b0 tx=0x7f935c007b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:48.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.320+0000 7f9362ffd640 1 -- 172.21.15.161:0/3415320162 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f936801f030 con 0x7f936c06bb60 2024-03-28T23:31:48.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.580+0000 7f937429d640 1 -- 172.21.15.161:0/3415320162 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f936c0c8180 con 0x7f936c06bb60 2024-03-28T23:31:48.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.582+0000 7f9362ffd640 1 -- 172.21.15.161:0/3415320162 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v10) ==== 190+0+147 (secure 0 0 0) 0x7f93680172c0 con 0x7f936c06bb60 2024-03-28T23:31:48.584 INFO:teuthology.orchestra.run.smithi161.stdout:[client.0] 2024-03-28T23:31:48.584 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQA6/QVmc1krAxAAEKsn1Tuuv9OlFZpB4PFhDg== 2024-03-28T23:31:48.584 INFO:teuthology.orchestra.run.smithi161.stdout: caps mds = "allow" 2024-03-28T23:31:48.584 INFO:teuthology.orchestra.run.smithi161.stdout: caps mon = "allow r" 2024-03-28T23:31:48.584 INFO:teuthology.orchestra.run.smithi161.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-28T23:31:48.584 INFO:teuthology.orchestra.run.smithi161.stderr:updated caps for client.0 2024-03-28T23:31:48.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.584+0000 7f937429d640 1 -- 172.21.15.161:0/3415320162 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f935006b4d0 msgr2=0x7f935006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:48.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.584+0000 7f937429d640 1 --2- 172.21.15.161:0/3415320162 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f935006b4d0 0x7f935006d990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f936c0740b0 tx=0x7f935c007b50 comp rx=0 tx=0).stop 2024-03-28T23:31:48.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.584+0000 7f937429d640 1 -- 172.21.15.161:0/3415320162 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f936c06bb60 msgr2=0x7f936c1eada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:48.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.584+0000 7f937429d640 1 --2- 172.21.15.161:0/3415320162 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f936c06bb60 0x7f936c1eada0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f936800b3f0 tx=0x7f936800b8b0 comp rx=0 tx=0).stop 2024-03-28T23:31:48.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.584+0000 7f937429d640 1 -- 172.21.15.161:0/3415320162 shutdown_connections 2024-03-28T23:31:48.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.584+0000 7f937429d640 1 --2- 172.21.15.161:0/3415320162 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f935006b4d0 0x7f935006d990 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:48.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.584+0000 7f937429d640 1 --2- 172.21.15.161:0/3415320162 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f936c1529d0 0x7f936c14b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:48.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.584+0000 7f937429d640 1 --2- 172.21.15.161:0/3415320162 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f936c0c6750 0x7f936c1eead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:48.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.584+0000 7f937429d640 1 --2- 172.21.15.161:0/3415320162 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f936c06bb60 0x7f936c1eada0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:48.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.584+0000 7f937429d640 1 -- 172.21.15.161:0/3415320162 >> 172.21.15.161:0/3415320162 conn(0x7f936c0c7430 msgr2=0x7f936c155c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:48.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.584+0000 7f937429d640 1 -- 172.21.15.161:0/3415320162 shutdown_connections 2024-03-28T23:31:48.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.584+0000 7f937429d640 1 -- 172.21.15.161:0/3415320162 wait complete. 2024-03-28T23:31:48.596 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:31:48.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.697+0000 7f67ae08f640 1 Processor -- start 2024-03-28T23:31:48.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.697+0000 7f67ae08f640 1 -- start start 2024-03-28T23:31:48.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.698+0000 7f67ae08f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67a8152890 0x7f67a80c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:48.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.698+0000 7f67ae08f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f67a80c5b20 0x7f67a81553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:48.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.698+0000 7f67ae08f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67a8158a30 con 0x7f67a80c8ed0 2024-03-28T23:31:48.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.698+0000 7f67ae08f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f67a80c8480 con 0x7f67a80c5b20 2024-03-28T23:31:48.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.698+0000 7f67ae08f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f67a8159640 con 0x7f67a8152890 2024-03-28T23:31:48.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.698+0000 7f67a77fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67a8152890 0x7f67a80c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:48.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.698+0000 7f67a7fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f67a80c5b20 0x7f67a81553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:48.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.698+0000 7f67a77fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67a8152890 0x7f67a80c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:58090/0 (socket says 172.21.15.161:58090) 2024-03-28T23:31:48.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.698+0000 7f67a6ffd640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f67a80c8ed0 0x7f67a80c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:37218/0 (socket says 172.21.15.161:37218) 2024-03-28T23:31:48.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.698+0000 7f67a77fe640 1 -- 172.21.15.161:0/1023348061 learned_addr learned my addr 172.21.15.161:0/1023348061 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:48.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.699+0000 7f67a77fe640 1 -- 172.21.15.161:0/1023348061 >> v2:172.21.15.170:3300/0 conn(0x7f67a80c5b20 msgr2=0x7f67a81553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:48.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.699+0000 7f67a77fe640 1 --2- 172.21.15.161:0/1023348061 >> v2:172.21.15.170:3300/0 conn(0x7f67a80c5b20 0x7f67a81553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:48.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.699+0000 7f67a77fe640 1 -- 172.21.15.161:0/1023348061 >> v1:172.21.15.170:6789/0 conn(0x7f67a80c8ed0 legacy=0x7f67a80c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:48.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.699+0000 7f67a77fe640 1 -- 172.21.15.161:0/1023348061 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a8158d10 con 0x7f67a8152890 2024-03-28T23:31:48.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.699+0000 7f67a7fff640 1 --2- 172.21.15.161:0/1023348061 >> v2:172.21.15.170:3300/0 conn(0x7f67a80c5b20 0x7f67a81553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:48.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.699+0000 7f67a77fe640 1 --2- 172.21.15.161:0/1023348061 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67a8152890 0x7f67a80c8990 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f67a81597c0 tx=0x7f67a002be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9c935102a539408 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:48.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.700+0000 7f67a67fc640 1 -- 172.21.15.161:0/1023348061 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a002c9c0 con 0x7f67a8152890 2024-03-28T23:31:48.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.700+0000 7f67a67fc640 1 -- 172.21.15.161:0/1023348061 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67a002cb60 con 0x7f67a8152890 2024-03-28T23:31:48.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.700+0000 7f67a67fc640 1 -- 172.21.15.161:0/1023348061 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a00343f0 con 0x7f67a8152890 2024-03-28T23:31:48.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.700+0000 7f67ae08f640 1 -- 172.21.15.161:0/1023348061 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67a8152890 msgr2=0x7f67a80c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:48.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.700+0000 7f67ae08f640 1 --2- 172.21.15.161:0/1023348061 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67a8152890 0x7f67a80c8990 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f67a81597c0 tx=0x7f67a002be20 comp rx=0 tx=0).stop 2024-03-28T23:31:48.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.700+0000 7f67ae08f640 1 -- 172.21.15.161:0/1023348061 shutdown_connections 2024-03-28T23:31:48.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.700+0000 7f67ae08f640 1 --2- 172.21.15.161:0/1023348061 >> v2:172.21.15.170:3300/0 conn(0x7f67a80c5b20 0x7f67a81553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:48.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.700+0000 7f67ae08f640 1 --2- 172.21.15.161:0/1023348061 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67a8152890 0x7f67a80c8990 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:48.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.700+0000 7f67ae08f640 1 -- 172.21.15.161:0/1023348061 >> 172.21.15.161:0/1023348061 conn(0x7f67a80ac620 msgr2=0x7f67a80aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:48.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.700+0000 7f67ae08f640 1 -- 172.21.15.161:0/1023348061 shutdown_connections 2024-03-28T23:31:48.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.700+0000 7f67ae08f640 1 -- 172.21.15.161:0/1023348061 wait complete. 2024-03-28T23:31:48.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.702+0000 7f67ae08f640 1 Processor -- start 2024-03-28T23:31:48.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.702+0000 7f67ae08f640 1 -- start start 2024-03-28T23:31:48.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.703+0000 7f67ae08f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67a80c5b20 0x7f67a8155040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:48.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.703+0000 7f67ae08f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67a80c8ed0 0x7f67a8144aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:48.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.703+0000 7f67ae08f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67a8152890 0x7f67a8155af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:48.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.703+0000 7f67ae08f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f67a8152c70 con 0x7f67a80c8ed0 2024-03-28T23:31:48.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.703+0000 7f67ae08f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f67a81592d0 con 0x7f67a80c5b20 2024-03-28T23:31:48.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.703+0000 7f67ae08f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f67a8152df0 con 0x7f67a8152890 2024-03-28T23:31:48.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.703+0000 7f67a6ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67a80c8ed0 0x7f67a8144aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:48.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.703+0000 7f67a6ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67a80c8ed0 0x7f67a8144aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58518/0 (socket says 172.21.15.161:58518) 2024-03-28T23:31:48.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.703+0000 7f67a6ffd640 1 -- 172.21.15.161:0/1131076061 learned_addr learned my addr 172.21.15.161:0/1131076061 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:48.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.703+0000 7f67a77fe640 1 --2- 172.21.15.161:0/1131076061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67a80c5b20 0x7f67a8155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:48.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.703+0000 7f67a7fff640 1 --2- 172.21.15.161:0/1131076061 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67a8152890 0x7f67a8155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:48.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.703+0000 7f67a6ffd640 1 -- 172.21.15.161:0/1131076061 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67a8152890 msgr2=0x7f67a8155af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:48.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.703+0000 7f67a6ffd640 1 --2- 172.21.15.161:0/1131076061 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67a8152890 0x7f67a8155af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:48.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.703+0000 7f67a6ffd640 1 -- 172.21.15.161:0/1131076061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67a80c5b20 msgr2=0x7f67a8155040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:48.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.703+0000 7f67a6ffd640 1 --2- 172.21.15.161:0/1131076061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67a80c5b20 0x7f67a8155040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:48.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.703+0000 7f67a6ffd640 1 -- 172.21.15.161:0/1131076061 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a8147a60 con 0x7f67a80c8ed0 2024-03-28T23:31:48.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.703+0000 7f67a6ffd640 1 --2- 172.21.15.161:0/1131076061 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67a80c8ed0 0x7f67a8144aa0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f679000c430 tx=0x7f679000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:48.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.704+0000 7f679ffff640 1 -- 172.21.15.161:0/1131076061 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6790017070 con 0x7f67a80c8ed0 2024-03-28T23:31:48.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.704+0000 7f679ffff640 1 -- 172.21.15.161:0/1131076061 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6790005c70 con 0x7f67a80c8ed0 2024-03-28T23:31:48.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.704+0000 7f67ae08f640 1 -- 172.21.15.161:0/1131076061 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67a80ab7c0 con 0x7f67a80c8ed0 2024-03-28T23:31:48.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.704+0000 7f679ffff640 1 -- 172.21.15.161:0/1131076061 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67900072d0 con 0x7f67a80c8ed0 2024-03-28T23:31:48.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.704+0000 7f67ae08f640 1 -- 172.21.15.161:0/1131076061 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67a8155580 con 0x7f67a80c8ed0 2024-03-28T23:31:48.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.705+0000 7f67ae08f640 1 -- 172.21.15.161:0/1131076061 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67a804f4b0 con 0x7f67a80c8ed0 2024-03-28T23:31:48.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.705+0000 7f679ffff640 1 -- 172.21.15.161:0/1131076061 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f679001a070 con 0x7f67a80c8ed0 2024-03-28T23:31:48.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.706+0000 7f679ffff640 1 --2- 172.21.15.161:0/1131076061 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f678806b4d0 0x7f678806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:48.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.706+0000 7f67a77fe640 1 --2- 172.21.15.161:0/1131076061 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f678806b4d0 0x7f678806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:48.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.706+0000 7f679ffff640 1 -- 172.21.15.161:0/1131076061 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f6790089590 con 0x7f67a80c8ed0 2024-03-28T23:31:48.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.706+0000 7f67a77fe640 1 --2- 172.21.15.161:0/1131076061 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f678806b4d0 0x7f678806d990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f67a0035000 tx=0x7f67a0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:48.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:48.710+0000 7f679ffff640 1 -- 172.21.15.161:0/1131076061 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6790005e20 con 0x7f67a80c8ed0 2024-03-28T23:31:49.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.017+0000 7f67ae08f640 1 -- 172.21.15.161:0/1131076061 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67a80c8140 con 0x7f67a80c8ed0 2024-03-28T23:31:49.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.017+0000 7f679ffff640 1 -- 172.21.15.161:0/1131076061 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5190 (secure 0 0 0) 0x7f6790057cc0 con 0x7f67a80c8ed0 2024-03-28T23:31:49.019 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:49.019 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5186,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/1945122204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1945122204},{"type":"v1","addr":"172.21.15.170:6835","nonce":1945122204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":5542,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/3666899516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3666899516},{"type":"v1","addr":"172.21.15.170:6833","nonce":3666899516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5817,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/1300583128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1300583128},{"type":"v1","addr":"172.21.15.161:6836","nonce":1300583128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:31:39.725625+0000","modified":"2024-03-28T23:31:41.818149+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5820},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5820":{"gid":5820,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":4,"addr":"172.21.15.161:6837/544971001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":544971001},{"type":"v1","addr":"172.21.15.161:6837","nonce":544971001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5820,"qdb_cluster":[5820]},"id":3}]} 2024-03-28T23:31:49.020 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 30 2024-03-28T23:31:49.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.019+0000 7f67ae08f640 1 -- 172.21.15.161:0/1131076061 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f678806b4d0 msgr2=0x7f678806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:49.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.019+0000 7f67ae08f640 1 --2- 172.21.15.161:0/1131076061 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f678806b4d0 0x7f678806d990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f67a0035000 tx=0x7f67a0030000 comp rx=0 tx=0).stop 2024-03-28T23:31:49.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.019+0000 7f67ae08f640 1 -- 172.21.15.161:0/1131076061 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67a80c8ed0 msgr2=0x7f67a8144aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:49.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.019+0000 7f67ae08f640 1 --2- 172.21.15.161:0/1131076061 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67a80c8ed0 0x7f67a8144aa0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f679000c430 tx=0x7f679000c8f0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.019+0000 7f67ae08f640 1 -- 172.21.15.161:0/1131076061 shutdown_connections 2024-03-28T23:31:49.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.019+0000 7f67ae08f640 1 --2- 172.21.15.161:0/1131076061 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f678806b4d0 0x7f678806d990 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.019+0000 7f67ae08f640 1 --2- 172.21.15.161:0/1131076061 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67a80c8ed0 0x7f67a8144aa0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.019+0000 7f67ae08f640 1 --2- 172.21.15.161:0/1131076061 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67a8152890 0x7f67a8155af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.019+0000 7f67ae08f640 1 --2- 172.21.15.161:0/1131076061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67a80c5b20 0x7f67a8155040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.019+0000 7f67ae08f640 1 -- 172.21.15.161:0/1131076061 >> 172.21.15.161:0/1131076061 conn(0x7f67a80ac620 msgr2=0x7f67a814ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:49.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.020+0000 7f67ae08f640 1 -- 172.21.15.161:0/1131076061 shutdown_connections 2024-03-28T23:31:49.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.020+0000 7f67ae08f640 1 -- 172.21.15.161:0/1131076061 wait complete. 2024-03-28T23:31:49.434 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:31:39.725625+0000', 'modified': '2024-03-28T23:31:41.818149+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5820}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5820': {'gid': 5820, 'name': 'a', 'rank': 0, 'incarnation': 28, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.161:6837/544971001', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 544971001}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 544971001}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5820, 'qdb_cluster': [5820]} 2024-03-28T23:31:49.435 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:31:49.435 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-28T23:31:49.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.540+0000 7fcd29240640 1 Processor -- start 2024-03-28T23:31:49.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.540+0000 7fcd29240640 1 -- start start 2024-03-28T23:31:49.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.541+0000 7fcd29240640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fcd240c5410 0x7fcd240c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.541+0000 7fcd29240640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd241553a0 0x7fcd24155850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.541+0000 7fcd29240640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd24158a30 con 0x7fcd24152890 2024-03-28T23:31:49.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.541+0000 7fcd29240640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fcd240c8480 con 0x7fcd240c5410 2024-03-28T23:31:49.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.541+0000 7fcd29240640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fcd240c5d90 con 0x7fcd241553a0 2024-03-28T23:31:49.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.541+0000 7fcd22d9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fcd24152890 0x7fcd240c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44042/0 (socket says 172.21.15.161:44042) 2024-03-28T23:31:49.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.541+0000 7fcd22d9d640 1 -- 172.21.15.161:0/1297903067 learned_addr learned my addr 172.21.15.161:0/1297903067 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:49.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.541+0000 7fcd2359e640 1 --2- 172.21.15.161:0/1297903067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd241553a0 0x7fcd24155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.541+0000 7fcd1bfff640 1 --2- 172.21.15.161:0/1297903067 >> v2:172.21.15.170:3300/0 conn(0x7fcd240c5410 0x7fcd240c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.541+0000 7fcd2259c640 1 -- 172.21.15.161:0/1297903067 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1347519237 0 0) 0x7fcd24158a30 con 0x7fcd24152890 2024-03-28T23:31:49.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.541+0000 7fcd2259c640 1 -- 172.21.15.161:0/1297903067 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd10003520 con 0x7fcd24152890 2024-03-28T23:31:49.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.542+0000 7fcd2359e640 1 -- 172.21.15.161:0/1297903067 >> v2:172.21.15.170:3300/0 conn(0x7fcd240c5410 msgr2=0x7fcd240c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:49.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.542+0000 7fcd2359e640 1 --2- 172.21.15.161:0/1297903067 >> v2:172.21.15.170:3300/0 conn(0x7fcd240c5410 0x7fcd240c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.542+0000 7fcd2359e640 1 -- 172.21.15.161:0/1297903067 >> v1:172.21.15.161:6789/0 conn(0x7fcd24152890 legacy=0x7fcd240c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:49.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.542+0000 7fcd2259c640 1 -- 172.21.15.161:0/1297903067 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3535403988 0 0) 0x7fcd10003520 con 0x7fcd24152890 2024-03-28T23:31:49.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.542+0000 7fcd2359e640 1 -- 172.21.15.161:0/1297903067 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd24158d10 con 0x7fcd241553a0 2024-03-28T23:31:49.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.542+0000 7fcd1bfff640 1 --2- 172.21.15.161:0/1297903067 >> v2:172.21.15.170:3300/0 conn(0x7fcd240c5410 0x7fcd240c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:49.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.542+0000 7fcd2359e640 1 --2- 172.21.15.161:0/1297903067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd241553a0 0x7fcd24155850 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009d90 tx=0x7fcd1c02bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f226a3da4507f72f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:49.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.543+0000 7fcd2259c640 1 -- 172.21.15.161:0/1297903067 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c039070 con 0x7fcd241553a0 2024-03-28T23:31:49.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.543+0000 7fcd2259c640 1 -- 172.21.15.161:0/1297903067 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd1c02c810 con 0x7fcd241553a0 2024-03-28T23:31:49.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.543+0000 7fcd2259c640 1 -- 172.21.15.161:0/1297903067 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd14001670 con 0x7fcd241553a0 2024-03-28T23:31:49.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.543+0000 7fcd29240640 1 -- 172.21.15.161:0/1297903067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd241553a0 msgr2=0x7fcd24155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:49.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.543+0000 7fcd29240640 1 --2- 172.21.15.161:0/1297903067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd241553a0 0x7fcd24155850 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009d90 tx=0x7fcd1c02bce0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.543+0000 7fcd29240640 1 -- 172.21.15.161:0/1297903067 shutdown_connections 2024-03-28T23:31:49.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.543+0000 7fcd29240640 1 --2- 172.21.15.161:0/1297903067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd241553a0 0x7fcd24155850 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.543+0000 7fcd29240640 1 --2- 172.21.15.161:0/1297903067 >> v2:172.21.15.170:3300/0 conn(0x7fcd240c5410 0x7fcd240c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.543+0000 7fcd29240640 1 -- 172.21.15.161:0/1297903067 >> 172.21.15.161:0/1297903067 conn(0x7fcd240ac620 msgr2=0x7fcd240aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:49.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.543+0000 7fcd29240640 1 -- 172.21.15.161:0/1297903067 shutdown_connections 2024-03-28T23:31:49.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.543+0000 7fcd29240640 1 -- 172.21.15.161:0/1297903067 wait complete. 2024-03-28T23:31:49.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.545+0000 7fcd29240640 1 Processor -- start 2024-03-28T23:31:49.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.545+0000 7fcd29240640 1 -- start start 2024-03-28T23:31:49.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.546+0000 7fcd29240640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd240c5410 0x7fcd241e8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.546+0000 7fcd29240640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd24152890 0x7fcd240ab430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.546+0000 7fcd29240640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd241553a0 0x7fcd241500b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.546+0000 7fcd29240640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcd241592d0 con 0x7fcd240c5410 2024-03-28T23:31:49.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.546+0000 7fcd29240640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fcd24159640 con 0x7fcd241553a0 2024-03-28T23:31:49.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.546+0000 7fcd29240640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fcd240c90a0 con 0x7fcd24152890 2024-03-28T23:31:49.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.546+0000 7fcd22d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd240c5410 0x7fcd241e8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.547+0000 7fcd22d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd240c5410 0x7fcd241e8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58530/0 (socket says 172.21.15.161:58530) 2024-03-28T23:31:49.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.547+0000 7fcd22d9d640 1 -- 172.21.15.161:0/3307086793 learned_addr learned my addr 172.21.15.161:0/3307086793 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:49.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.547+0000 7fcd1bfff640 1 --2- 172.21.15.161:0/3307086793 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd24152890 0x7fcd240ab430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.547+0000 7fcd2359e640 1 --2- 172.21.15.161:0/3307086793 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd241553a0 0x7fcd241500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.547+0000 7fcd22d9d640 1 -- 172.21.15.161:0/3307086793 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd24152890 msgr2=0x7fcd240ab430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:49.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.547+0000 7fcd22d9d640 1 --2- 172.21.15.161:0/3307086793 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd24152890 0x7fcd240ab430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.547+0000 7fcd22d9d640 1 -- 172.21.15.161:0/3307086793 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd241553a0 msgr2=0x7fcd241500b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:49.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.547+0000 7fcd22d9d640 1 --2- 172.21.15.161:0/3307086793 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd241553a0 0x7fcd241500b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.547+0000 7fcd22d9d640 1 -- 172.21.15.161:0/3307086793 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd241eba60 con 0x7fcd240c5410 2024-03-28T23:31:49.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.547+0000 7fcd2359e640 1 --2- 172.21.15.161:0/3307086793 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd241553a0 0x7fcd241500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:49.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.547+0000 7fcd22d9d640 1 --2- 172.21.15.161:0/3307086793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd240c5410 0x7fcd241e8260 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fcd1400c780 tx=0x7fcd1400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:49.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.547+0000 7fcd1b7fe640 1 -- 172.21.15.161:0/3307086793 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd14005ad0 con 0x7fcd240c5410 2024-03-28T23:31:49.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.547+0000 7fcd1b7fe640 1 -- 172.21.15.161:0/3307086793 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd14005c90 con 0x7fcd240c5410 2024-03-28T23:31:49.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.547+0000 7fcd1b7fe640 1 -- 172.21.15.161:0/3307086793 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd140072b0 con 0x7fcd240c5410 2024-03-28T23:31:49.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.547+0000 7fcd29240640 1 -- 172.21.15.161:0/3307086793 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd241eb780 con 0x7fcd240c5410 2024-03-28T23:31:49.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.547+0000 7fcd29240640 1 -- 172.21.15.161:0/3307086793 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd241516a0 con 0x7fcd240c5410 2024-03-28T23:31:49.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.549+0000 7fcd1b7fe640 1 -- 172.21.15.161:0/3307086793 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcd14007450 con 0x7fcd240c5410 2024-03-28T23:31:49.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.549+0000 7fcd1b7fe640 1 --2- 172.21.15.161:0/3307086793 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcd0406b4d0 0x7fcd0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.549+0000 7fcd1b7fe640 1 -- 172.21.15.161:0/3307086793 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7fcd1408a070 con 0x7fcd240c5410 2024-03-28T23:31:49.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.549+0000 7fcd1bfff640 1 --2- 172.21.15.161:0/3307086793 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcd0406b4d0 0x7fcd0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.550+0000 7fcd1bfff640 1 --2- 172.21.15.161:0/3307086793 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcd0406b4d0 0x7fcd0406d990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c009500 tx=0x7fcd0c009110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:49.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.550+0000 7fcd2be26740 1 Processor -- start 2024-03-28T23:31:49.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.550+0000 7fcd2be26740 1 -- start start 2024-03-28T23:31:49.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.551+0000 7fcd2be26740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559a8a8ffdb0 0x559a8a8fd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.551+0000 7fcd2be26740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a900190 0x559a8a8fddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.551+0000 7fcd2be26740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559a8a8fe310 0x559a8a8fa4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.551+0000 7fcd2be26740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x559a8a863af0 con 0x559a8a900190 2024-03-28T23:31:49.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.551+0000 7fcd2be26740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x559a8a80c9d0 con 0x559a8a8fe310 2024-03-28T23:31:49.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.551+0000 7fcd2be26740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x559a8a85fb90 con 0x559a8a8ffdb0 2024-03-28T23:31:49.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.551+0000 7fcd1bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a900190 0x559a8a8fddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.551+0000 7fcd1bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a900190 0x559a8a8fddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58544/0 (socket says 172.21.15.161:58544) 2024-03-28T23:31:49.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.551+0000 7fcd1bfff640 1 -- 172.21.15.161:0/2258209022 learned_addr learned my addr 172.21.15.161:0/2258209022 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:49.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.551+0000 7fcd22d9d640 1 --2- 172.21.15.161:0/2258209022 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559a8a8ffdb0 0x559a8a8fd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.551+0000 7fcd2359e640 1 --2- 172.21.15.161:0/2258209022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559a8a8fe310 0x559a8a8fa4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.551+0000 7fcd1bfff640 1 -- 172.21.15.161:0/2258209022 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559a8a8ffdb0 msgr2=0x559a8a8fd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:49.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.551+0000 7fcd1bfff640 1 --2- 172.21.15.161:0/2258209022 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559a8a8ffdb0 0x559a8a8fd890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.551+0000 7fcd1bfff640 1 -- 172.21.15.161:0/2258209022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559a8a8fe310 msgr2=0x559a8a8fa4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:49.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.551+0000 7fcd1bfff640 1 --2- 172.21.15.161:0/2258209022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559a8a8fe310 0x559a8a8fa4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.551+0000 7fcd1bfff640 1 -- 172.21.15.161:0/2258209022 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a8a5782d0 con 0x559a8a900190 2024-03-28T23:31:49.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.551+0000 7fcd22d9d640 1 --2- 172.21.15.161:0/2258209022 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559a8a8ffdb0 0x559a8a8fd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:49.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.551+0000 7fcd1bfff640 1 --2- 172.21.15.161:0/2258209022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a900190 0x559a8a8fddd0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c0043f0 tx=0x7fcd0c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=264110535a29db87 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:49.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.552+0000 7fcd18ff9640 1 -- 172.21.15.161:0/2258209022 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c01d050 con 0x559a8a900190 2024-03-28T23:31:49.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.552+0000 7fcd18ff9640 1 -- 172.21.15.161:0/2258209022 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd0c027070 con 0x559a8a900190 2024-03-28T23:31:49.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.552+0000 7fcd18ff9640 1 -- 172.21.15.161:0/2258209022 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c003a20 con 0x559a8a900190 2024-03-28T23:31:49.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.552+0000 7fcd2be26740 1 -- 172.21.15.161:0/2258209022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a900190 msgr2=0x559a8a8fddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:49.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.552+0000 7fcd2be26740 1 --2- 172.21.15.161:0/2258209022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a900190 0x559a8a8fddd0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c0043f0 tx=0x7fcd0c014000 comp rx=0 tx=0).stop 2024-03-28T23:31:49.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.552+0000 7fcd2be26740 1 -- 172.21.15.161:0/2258209022 shutdown_connections 2024-03-28T23:31:49.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.552+0000 7fcd2be26740 1 --2- 172.21.15.161:0/2258209022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a900190 0x559a8a8fddd0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.552+0000 7fcd2be26740 1 --2- 172.21.15.161:0/2258209022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559a8a8fe310 0x559a8a8fa4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.552+0000 7fcd2be26740 1 --2- 172.21.15.161:0/2258209022 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559a8a8ffdb0 0x559a8a8fd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.552+0000 7fcd2be26740 1 -- 172.21.15.161:0/2258209022 >> 172.21.15.161:0/2258209022 conn(0x559a8a8fc630 msgr2=0x559a8a921220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:49.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.552+0000 7fcd2be26740 1 -- 172.21.15.161:0/2258209022 shutdown_connections 2024-03-28T23:31:49.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.552+0000 7fcd2be26740 1 -- 172.21.15.161:0/2258209022 wait complete. 2024-03-28T23:31:49.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.553+0000 7fcd2be26740 1 -- start start 2024-03-28T23:31:49.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.553+0000 7fcd2be26740 1 Processor -- start 2024-03-28T23:31:49.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.554+0000 7fcd2be26740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a8ffdb0 0x559a8a93f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.554+0000 7fcd2be26740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559a8a900190 0x559a8a94c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.554+0000 7fcd2be26740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559a8a9355f0 0x559a8a922460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.554+0000 7fcd2be26740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x559a8a922090 con 0x559a8a8ffdb0 2024-03-28T23:31:49.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.554+0000 7fcd2be26740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x559a8a81ff70 con 0x559a8a900190 2024-03-28T23:31:49.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.554+0000 7fcd2be26740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x559a8a921ad0 con 0x559a8a9355f0 2024-03-28T23:31:49.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.554+0000 7fcd2be26740 10 client.0 _ll_register_callbacks cb 0x559a8a577ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:31:49.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.554+0000 7fcd22d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a8ffdb0 0x559a8a93f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.554+0000 7fcd22d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a8ffdb0 0x559a8a93f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58560/0 (socket says 172.21.15.161:58560) 2024-03-28T23:31:49.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.554+0000 7fcd22d9d640 1 -- 172.21.15.161:0/302430635 learned_addr learned my addr 172.21.15.161:0/302430635 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:49.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.554+0000 7fcd2359e640 1 --2- 172.21.15.161:0/302430635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559a8a9355f0 0x559a8a922460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.554+0000 7fcd1bfff640 1 --2- 172.21.15.161:0/302430635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559a8a900190 0x559a8a94c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.554+0000 7fcd22d9d640 1 -- 172.21.15.161:0/302430635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559a8a9355f0 msgr2=0x559a8a922460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:49.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.554+0000 7fcd22d9d640 1 --2- 172.21.15.161:0/302430635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559a8a9355f0 0x559a8a922460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.554+0000 7fcd22d9d640 1 -- 172.21.15.161:0/302430635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559a8a900190 msgr2=0x559a8a94c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:49.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.554+0000 7fcd22d9d640 1 --2- 172.21.15.161:0/302430635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559a8a900190 0x559a8a94c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.554+0000 7fcd22d9d640 1 -- 172.21.15.161:0/302430635 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a8a6426b0 con 0x559a8a8ffdb0 2024-03-28T23:31:49.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.554+0000 7fcd22d9d640 1 --2- 172.21.15.161:0/302430635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a8ffdb0 0x559a8a93f8d0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fcd14054b50 tx=0x7fcd14058df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62e63fa86925de7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:49.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.554+0000 7fcd01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:31:49.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.554+0000 7fcd01ffb640 1 -- 172.21.15.161:0/302430635 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd14060e00 con 0x559a8a8ffdb0 2024-03-28T23:31:49.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.554+0000 7fcd01ffb640 1 -- 172.21.15.161:0/302430635 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7fcd140700d0 con 0x559a8a8ffdb0 2024-03-28T23:31:49.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.554+0000 7fcd2be26740 1 -- 172.21.15.161:0/302430635 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd0c027070 con 0x559a8a8ffdb0 2024-03-28T23:31:49.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.555+0000 7fcd01ffb640 1 -- 172.21.15.161:0/302430635 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcd14060970 con 0x559a8a8ffdb0 2024-03-28T23:31:49.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.555+0000 7fcd2be26740 10 client.5991 fetch_fsmap learned FSMap version 30 2024-03-28T23:31:49.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.555+0000 7fcd2be26740 1 -- 172.21.15.161:0/302430635 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) -- 0x559a8a940060 con 0x559a8a8ffdb0 2024-03-28T23:31:49.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.555+0000 7fcd01ffb640 1 -- 172.21.15.161:0/302430635 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd0c027070 con 0x559a8a8ffdb0 2024-03-28T23:31:49.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.555+0000 7fcd01ffb640 1 -- 172.21.15.161:0/302430635 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 30) ==== 3494+0+0 (secure 0 0 0) 0x7fcd14057b40 con 0x559a8a8ffdb0 2024-03-28T23:31:49.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.555+0000 7fcd2be26740 10 client.5991 fetch_fsmap finished waiting for FSMap version 30 2024-03-28T23:31:49.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.555+0000 7fcd2be26740 10 client.5991 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T23:31:49.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.555+0000 7fcd2be26740 20 client.5991 populate_metadata read hostname 'smithi161' 2024-03-28T23:31:49.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.555+0000 7fcd2be26740 1 --2- 172.21.15.161:0/302430635 >> [v2:172.21.15.161:6835/544971001,v1:172.21.15.161:6837/544971001] conn(0x559a8a91e0a0 0x559a8a95b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.555+0000 7fcd2be26740 4 client.5991 mds_command: new command op to 5820 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:31:49.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.555+0000 7fcd2be26740 1 -- 172.21.15.161:0/302430635 --> [v2:172.21.15.161:6835/544971001,v1:172.21.15.161:6837/544971001] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a8a871030 con 0x559a8a91e0a0 2024-03-28T23:31:49.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.555+0000 7fcd1bfff640 1 --2- 172.21.15.161:0/302430635 >> [v2:172.21.15.161:6835/544971001,v1:172.21.15.161:6837/544971001] conn(0x559a8a91e0a0 0x559a8a95b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.555+0000 7fcd1bfff640 1 --2- 172.21.15.161:0/302430635 >> [v2:172.21.15.161:6835/544971001,v1:172.21.15.161:6837/544971001] conn(0x559a8a91e0a0 0x559a8a95b0d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb3d51ace088e671 server_cookie=c79c409ca83e1ffa in_seq=0 out_seq=0 2024-03-28T23:31:49.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.555+0000 7fcd01ffb640 10 client.5991 ms_handle_connect on v2:172.21.15.161:6835/544971001 2024-03-28T23:31:49.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.556+0000 7fcd01ffb640 1 -- 172.21.15.161:0/302430635 <== mds.0 v2:172.21.15.161:6835/544971001 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559a8a871030 con 0x559a8a91e0a0 2024-03-28T23:31:49.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.556+0000 7fcd01ffb640 10 client.5991 handle_command_reply: tid=0 2024-03-28T23:31:49.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.556+0000 7fcd2be26740 1 client.5991 shutdown 2024-03-28T23:31:49.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.556+0000 7fcd2be26740 1 -- 172.21.15.161:0/302430635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a8ffdb0 msgr2=0x559a8a93f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:49.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.556+0000 7fcd2be26740 1 --2- 172.21.15.161:0/302430635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a8ffdb0 0x559a8a93f8d0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fcd14054b50 tx=0x7fcd14058df0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.556+0000 7fcd2be26740 1 -- 172.21.15.161:0/302430635 shutdown_connections 2024-03-28T23:31:49.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.556+0000 7fcd2be26740 1 --2- 172.21.15.161:0/302430635 >> [v2:172.21.15.161:6835/544971001,v1:172.21.15.161:6837/544971001] conn(0x559a8a91e0a0 0x559a8a95b0d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.556+0000 7fcd2be26740 1 --2- 172.21.15.161:0/302430635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559a8a9355f0 0x559a8a922460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.556+0000 7fcd2be26740 1 --2- 172.21.15.161:0/302430635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559a8a900190 0x559a8a94c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.556+0000 7fcd01ffb640 0 client.5991 ms_handle_reset on v2:172.21.15.161:6835/544971001 2024-03-28T23:31:49.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.556+0000 7fcd2be26740 1 --2- 172.21.15.161:0/302430635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a8ffdb0 0x559a8a93f8d0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.556+0000 7fcd2be26740 1 -- 172.21.15.161:0/302430635 >> 172.21.15.161:0/302430635 conn(0x559a8a8fe310 msgr2=0x559a8a8fa3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:49.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.556+0000 7fcd2be26740 1 -- 172.21.15.161:0/302430635 shutdown_connections 2024-03-28T23:31:49.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.556+0000 7fcd2be26740 1 -- 172.21.15.161:0/302430635 wait complete. 2024-03-28T23:31:49.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.556+0000 7fcd2be26740 20 client.5991 trim_cache size 0 max 16384 2024-03-28T23:31:49.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.574+0000 7fcd2be26740 1 Processor -- start 2024-03-28T23:31:49.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.574+0000 7fcd2be26740 1 -- start start 2024-03-28T23:31:49.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.575+0000 7fcd2be26740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559a8a8ffdb0 0x559a8a93f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.575+0000 7fcd2be26740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559a8a900190 0x559a8a8fd400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.575+0000 7fcd2be26740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a9355f0 0x559a8a8fd940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.575+0000 7fcd2be26740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcd241f6ed0 con 0x559a8a9355f0 2024-03-28T23:31:49.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.575+0000 7fcd2be26740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fcd241eece0 con 0x559a8a8ffdb0 2024-03-28T23:31:49.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.575+0000 7fcd2be26740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fcd241f6880 con 0x559a8a900190 2024-03-28T23:31:49.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.575+0000 7fcd2359e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a9355f0 0x559a8a8fd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.575+0000 7fcd2359e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a9355f0 0x559a8a8fd940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58566/0 (socket says 172.21.15.161:58566) 2024-03-28T23:31:49.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.575+0000 7fcd2359e640 1 -- 172.21.15.161:0/3388631074 learned_addr learned my addr 172.21.15.161:0/3388631074 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:49.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.575+0000 7fcd1bfff640 1 --2- 172.21.15.161:0/3388631074 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559a8a900190 0x559a8a8fd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.575+0000 7fcd22d9d640 1 --2- 172.21.15.161:0/3388631074 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559a8a8ffdb0 0x559a8a93f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.575+0000 7fcd2359e640 1 -- 172.21.15.161:0/3388631074 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559a8a900190 msgr2=0x559a8a8fd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:49.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.575+0000 7fcd2359e640 1 --2- 172.21.15.161:0/3388631074 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559a8a900190 0x559a8a8fd400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.575+0000 7fcd2359e640 1 -- 172.21.15.161:0/3388631074 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559a8a8ffdb0 msgr2=0x559a8a93f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:49.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.575+0000 7fcd2359e640 1 --2- 172.21.15.161:0/3388631074 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559a8a8ffdb0 0x559a8a93f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.575+0000 7fcd2359e640 1 -- 172.21.15.161:0/3388631074 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a8a9227d0 con 0x559a8a9355f0 2024-03-28T23:31:49.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.575+0000 7fcd22d9d640 1 --2- 172.21.15.161:0/3388631074 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559a8a8ffdb0 0x559a8a93f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:49.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.575+0000 7fcd2359e640 1 --2- 172.21.15.161:0/3388631074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a9355f0 0x559a8a8fd940 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009d90 tx=0x7fcd1c02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=763f715d255b1d6b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:49.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.575+0000 7fcd03fff640 1 -- 172.21.15.161:0/3388631074 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c039040 con 0x559a8a9355f0 2024-03-28T23:31:49.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.575+0000 7fcd03fff640 1 -- 172.21.15.161:0/3388631074 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd1c02c890 con 0x559a8a9355f0 2024-03-28T23:31:49.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.575+0000 7fcd03fff640 1 -- 172.21.15.161:0/3388631074 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c02cc80 con 0x559a8a9355f0 2024-03-28T23:31:49.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.576+0000 7fcd2be26740 1 -- 172.21.15.161:0/3388631074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a9355f0 msgr2=0x559a8a8fd940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:49.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.576+0000 7fcd2be26740 1 --2- 172.21.15.161:0/3388631074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a9355f0 0x559a8a8fd940 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009d90 tx=0x7fcd1c02e000 comp rx=0 tx=0).stop 2024-03-28T23:31:49.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.576+0000 7fcd2be26740 1 -- 172.21.15.161:0/3388631074 shutdown_connections 2024-03-28T23:31:49.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.576+0000 7fcd2be26740 1 --2- 172.21.15.161:0/3388631074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a9355f0 0x559a8a8fd940 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.576+0000 7fcd2be26740 1 --2- 172.21.15.161:0/3388631074 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559a8a900190 0x559a8a8fd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.576+0000 7fcd2be26740 1 --2- 172.21.15.161:0/3388631074 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559a8a8ffdb0 0x559a8a93f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.576+0000 7fcd2be26740 1 -- 172.21.15.161:0/3388631074 >> 172.21.15.161:0/3388631074 conn(0x559a8a8fe310 msgr2=0x559a8a8f5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:49.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.576+0000 7fcd2be26740 1 -- 172.21.15.161:0/3388631074 shutdown_connections 2024-03-28T23:31:49.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.576+0000 7fcd2be26740 1 -- 172.21.15.161:0/3388631074 wait complete. 2024-03-28T23:31:49.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.576+0000 7fcd2be26740 1 -- start start 2024-03-28T23:31:49.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.577+0000 7fcd2be26740 1 Processor -- start 2024-03-28T23:31:49.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.577+0000 7fcd2be26740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559a8a8ffdb0 0x559a8a95f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.577+0000 7fcd2be26740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559a8a900190 0x559a8a9237f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.577+0000 7fcd2be26740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a9355f0 0x559a8a960050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.577+0000 7fcd2be26740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x559a8a933770 con 0x559a8a9355f0 2024-03-28T23:31:49.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.577+0000 7fcd2be26740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x559a8a94f950 con 0x559a8a900190 2024-03-28T23:31:49.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.577+0000 7fcd2be26740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x559a8a922e70 con 0x559a8a8ffdb0 2024-03-28T23:31:49.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.577+0000 7fcd2be26740 10 client.0 _ll_register_callbacks cb 0x7fcd140549e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:31:49.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.577+0000 7fcd2359e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a9355f0 0x559a8a960050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.577+0000 7fcd2359e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a9355f0 0x559a8a960050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58580/0 (socket says 172.21.15.161:58580) 2024-03-28T23:31:49.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.577+0000 7fcd2359e640 1 -- 172.21.15.161:0/1962409781 learned_addr learned my addr 172.21.15.161:0/1962409781 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:49.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.577+0000 7fcd22d9d640 1 --2- 172.21.15.161:0/1962409781 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559a8a8ffdb0 0x559a8a95f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.577+0000 7fcd1bfff640 1 --2- 172.21.15.161:0/1962409781 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559a8a900190 0x559a8a9237f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.578+0000 7fcd2359e640 1 -- 172.21.15.161:0/1962409781 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559a8a8ffdb0 msgr2=0x559a8a95f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:49.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.578+0000 7fcd2359e640 1 --2- 172.21.15.161:0/1962409781 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559a8a8ffdb0 0x559a8a95f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.578+0000 7fcd2359e640 1 -- 172.21.15.161:0/1962409781 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559a8a900190 msgr2=0x559a8a9237f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:49.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.578+0000 7fcd2359e640 1 --2- 172.21.15.161:0/1962409781 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559a8a900190 0x559a8a9237f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.578+0000 7fcd2359e640 1 -- 172.21.15.161:0/1962409781 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a8a93e600 con 0x559a8a9355f0 2024-03-28T23:31:49.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.578+0000 7fcd2359e640 1 --2- 172.21.15.161:0/1962409781 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a9355f0 0x559a8a960050 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009eb0 tx=0x7fcd1c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dda28f22c03fbcf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:49.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.578+0000 7fcd01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:31:49.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.578+0000 7fcd01ffb640 1 -- 172.21.15.161:0/1962409781 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c039070 con 0x559a8a9355f0 2024-03-28T23:31:49.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.578+0000 7fcd01ffb640 1 -- 172.21.15.161:0/1962409781 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7fcd1c003ba0 con 0x559a8a9355f0 2024-03-28T23:31:49.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.578+0000 7fcd2be26740 1 -- 172.21.15.161:0/1962409781 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd1c02c890 con 0x559a8a9355f0 2024-03-28T23:31:49.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.578+0000 7fcd01ffb640 1 -- 172.21.15.161:0/1962409781 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcd1c05d050 con 0x559a8a9355f0 2024-03-28T23:31:49.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.578+0000 7fcd2be26740 10 client.5997 fetch_fsmap learned FSMap version 30 2024-03-28T23:31:49.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.578+0000 7fcd2be26740 1 -- 172.21.15.161:0/1962409781 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) -- 0x559a8a924fd0 con 0x559a8a9355f0 2024-03-28T23:31:49.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.578+0000 7fcd01ffb640 1 -- 172.21.15.161:0/1962409781 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd1c02c890 con 0x559a8a9355f0 2024-03-28T23:31:49.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.578+0000 7fcd01ffb640 1 -- 172.21.15.161:0/1962409781 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 30) ==== 3494+0+0 (secure 0 0 0) 0x7fcd1c060020 con 0x559a8a9355f0 2024-03-28T23:31:49.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.578+0000 7fcd2be26740 10 client.5997 fetch_fsmap finished waiting for FSMap version 30 2024-03-28T23:31:49.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.578+0000 7fcd2be26740 10 client.5997 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T23:31:49.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.578+0000 7fcd2be26740 20 client.5997 populate_metadata read hostname 'smithi161' 2024-03-28T23:31:49.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.578+0000 7fcd2be26740 1 --2- 172.21.15.161:0/1962409781 >> [v2:172.21.15.161:6835/544971001,v1:172.21.15.161:6837/544971001] conn(0x559a8a9b0480 0x559a8a8fe6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.578+0000 7fcd2be26740 4 client.5997 mds_command: new command op to 5820 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:31:49.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.578+0000 7fcd2be26740 1 -- 172.21.15.161:0/1962409781 --> [v2:172.21.15.161:6835/544971001,v1:172.21.15.161:6837/544971001] -- command(tid 0: {"prefix": "status"}) -- 0x559a8a86b380 con 0x559a8a9b0480 2024-03-28T23:31:49.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.578+0000 7fcd22d9d640 1 --2- 172.21.15.161:0/1962409781 >> [v2:172.21.15.161:6835/544971001,v1:172.21.15.161:6837/544971001] conn(0x559a8a9b0480 0x559a8a8fe6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.579+0000 7fcd01ffb640 10 client.5997 ms_handle_connect on v2:172.21.15.161:6835/544971001 2024-03-28T23:31:49.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.579+0000 7fcd22d9d640 1 --2- 172.21.15.161:0/1962409781 >> [v2:172.21.15.161:6835/544971001,v1:172.21.15.161:6837/544971001] conn(0x559a8a9b0480 0x559a8a8fe6f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a29e54af071cbb95 server_cookie=9256a504a37f628 in_seq=0 out_seq=0 2024-03-28T23:31:49.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.579+0000 7fcd01ffb640 1 -- 172.21.15.161:0/1962409781 <== mds.0 v2:172.21.15.161:6835/544971001 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x559a8a86b380 con 0x559a8a9b0480 2024-03-28T23:31:49.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.579+0000 7fcd01ffb640 10 client.5997 handle_command_reply: tid=0 2024-03-28T23:31:49.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.579+0000 7fcd2be26740 1 client.5997 shutdown 2024-03-28T23:31:49.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.579+0000 7fcd2be26740 1 -- 172.21.15.161:0/1962409781 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a9355f0 msgr2=0x559a8a960050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:49.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.579+0000 7fcd2be26740 1 --2- 172.21.15.161:0/1962409781 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a9355f0 0x559a8a960050 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009eb0 tx=0x7fcd1c004ab0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.579+0000 7fcd2be26740 1 -- 172.21.15.161:0/1962409781 shutdown_connections 2024-03-28T23:31:49.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.579+0000 7fcd2be26740 1 --2- 172.21.15.161:0/1962409781 >> [v2:172.21.15.161:6835/544971001,v1:172.21.15.161:6837/544971001] conn(0x559a8a9b0480 0x559a8a8fe6f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.579+0000 7fcd2be26740 1 --2- 172.21.15.161:0/1962409781 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559a8a9355f0 0x559a8a960050 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.579+0000 7fcd2be26740 1 --2- 172.21.15.161:0/1962409781 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559a8a900190 0x559a8a9237f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.579+0000 7fcd2be26740 1 --2- 172.21.15.161:0/1962409781 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559a8a8ffdb0 0x559a8a95f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.579+0000 7fcd2be26740 1 -- 172.21.15.161:0/1962409781 >> 172.21.15.161:0/1962409781 conn(0x559a8a8fe310 msgr2=0x559a8a8fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:49.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.579+0000 7fcd01ffb640 0 client.5997 ms_handle_reset on v2:172.21.15.161:6835/544971001 2024-03-28T23:31:49.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.579+0000 7fcd2be26740 1 -- 172.21.15.161:0/1962409781 shutdown_connections 2024-03-28T23:31:49.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.579+0000 7fcd2be26740 1 -- 172.21.15.161:0/1962409781 wait complete. 2024-03-28T23:31:49.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.579+0000 7fcd2be26740 20 client.5997 trim_cache size 0 max 16384 2024-03-28T23:31:49.589 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:31:49.589 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:31:49.589 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:31:49.589 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 5820, 2024-03-28T23:31:49.589 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:31:49.589 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:31:49.589 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:31:49.589 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 9.2083496730000007, 2024-03-28T23:31:49.589 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 30, 2024-03-28T23:31:49.589 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 51, 2024-03-28T23:31:49.590 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 51, 2024-03-28T23:31:49.590 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 20.327238672 2024-03-28T23:31:49.590 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:31:49.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.580+0000 7fcd29240640 1 -- 172.21.15.161:0/3307086793 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcd0406b4d0 msgr2=0x7fcd0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:49.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.580+0000 7fcd29240640 1 --2- 172.21.15.161:0/3307086793 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcd0406b4d0 0x7fcd0406d990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c009500 tx=0x7fcd0c009110 comp rx=0 tx=0).stop 2024-03-28T23:31:49.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.580+0000 7fcd29240640 1 -- 172.21.15.161:0/3307086793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd240c5410 msgr2=0x7fcd241e8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:49.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.580+0000 7fcd29240640 1 --2- 172.21.15.161:0/3307086793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd240c5410 0x7fcd241e8260 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fcd1400c780 tx=0x7fcd1400cc40 comp rx=0 tx=0).stop 2024-03-28T23:31:49.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.580+0000 7fcd29240640 1 -- 172.21.15.161:0/3307086793 shutdown_connections 2024-03-28T23:31:49.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.580+0000 7fcd29240640 1 --2- 172.21.15.161:0/3307086793 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcd0406b4d0 0x7fcd0406d990 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.580+0000 7fcd29240640 1 --2- 172.21.15.161:0/3307086793 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd241553a0 0x7fcd241500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.580+0000 7fcd29240640 1 --2- 172.21.15.161:0/3307086793 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd24152890 0x7fcd240ab430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.580+0000 7fcd29240640 1 --2- 172.21.15.161:0/3307086793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd240c5410 0x7fcd241e8260 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.580+0000 7fcd29240640 1 -- 172.21.15.161:0/3307086793 >> 172.21.15.161:0/3307086793 conn(0x7fcd240ac620 msgr2=0x7fcd240ca090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:49.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.580+0000 7fcd29240640 1 -- 172.21.15.161:0/3307086793 shutdown_connections 2024-03-28T23:31:49.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.580+0000 7fcd29240640 1 -- 172.21.15.161:0/3307086793 wait complete. 2024-03-28T23:31:49.594 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:31:49.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.696+0000 7f07b048a640 1 Processor -- start 2024-03-28T23:31:49.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.696+0000 7f07b048a640 1 -- start start 2024-03-28T23:31:49.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.697+0000 7f07b048a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f07a806ce80 0x7f07a80aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.697+0000 7f07b048a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f07a80a9680 0x7f07a80aab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.697+0000 7f07b048a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07a814c250 con 0x7f07a814be70 2024-03-28T23:31:49.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.697+0000 7f07b048a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f07a81529c0 con 0x7f07a80a9680 2024-03-28T23:31:49.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.697+0000 7f07b048a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f07a80a9dc0 con 0x7f07a806ce80 2024-03-28T23:31:49.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.697+0000 7f07aea27640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f07a80a9680 0x7f07a80aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.697+0000 7f07aea27640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f07a80a9680 0x7f07a80aab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58594/0 (socket says 172.21.15.161:58594) 2024-03-28T23:31:49.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.697+0000 7f07aea27640 1 -- 172.21.15.161:0/165941590 learned_addr learned my addr 172.21.15.161:0/165941590 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:49.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.698+0000 7f07aea27640 1 -- 172.21.15.161:0/165941590 >> v2:172.21.15.170:3300/0 conn(0x7f07a806ce80 msgr2=0x7f07a80aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:49.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.698+0000 7f07ada25640 1 --2- 172.21.15.161:0/165941590 >> v2:172.21.15.170:3300/0 conn(0x7f07a806ce80 0x7f07a80aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.698+0000 7f07aea27640 1 --2- 172.21.15.161:0/165941590 >> v2:172.21.15.170:3300/0 conn(0x7f07a806ce80 0x7f07a80aa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.698+0000 7f07aea27640 1 -- 172.21.15.161:0/165941590 >> v1:172.21.15.170:6789/0 conn(0x7f07a814be70 legacy=0x7f07a80ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:49.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.698+0000 7f07aea27640 1 -- 172.21.15.161:0/165941590 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07a80aa0a0 con 0x7f07a80a9680 2024-03-28T23:31:49.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.698+0000 7f07ada25640 1 --2- 172.21.15.161:0/165941590 >> v2:172.21.15.170:3300/0 conn(0x7f07a806ce80 0x7f07a80aa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:49.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.698+0000 7f07aea27640 1 --2- 172.21.15.161:0/165941590 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07a80a9680 0x7f07a80aab00 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f07a806f540 tx=0x7f07a002dfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=598ef074085e3ffe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:49.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.699+0000 7f07ad224640 1 -- 172.21.15.161:0/165941590 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a002e970 con 0x7f07a80a9680 2024-03-28T23:31:49.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.699+0000 7f07ad224640 1 -- 172.21.15.161:0/165941590 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07a002eb10 con 0x7f07a80a9680 2024-03-28T23:31:49.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.699+0000 7f07ad224640 1 -- 172.21.15.161:0/165941590 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a0009600 con 0x7f07a80a9680 2024-03-28T23:31:49.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.699+0000 7f07b048a640 1 -- 172.21.15.161:0/165941590 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07a80a9680 msgr2=0x7f07a80aab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:49.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.699+0000 7f07b048a640 1 --2- 172.21.15.161:0/165941590 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07a80a9680 0x7f07a80aab00 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f07a806f540 tx=0x7f07a002dfd0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.699+0000 7f07b048a640 1 -- 172.21.15.161:0/165941590 shutdown_connections 2024-03-28T23:31:49.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.699+0000 7f07b048a640 1 --2- 172.21.15.161:0/165941590 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07a80a9680 0x7f07a80aab00 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.699+0000 7f07b048a640 1 --2- 172.21.15.161:0/165941590 >> v2:172.21.15.170:3300/0 conn(0x7f07a806ce80 0x7f07a80aa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.699+0000 7f07b048a640 1 -- 172.21.15.161:0/165941590 >> 172.21.15.161:0/165941590 conn(0x7f07a80c6670 msgr2=0x7f07a80c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:49.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.699+0000 7f07b048a640 1 -- 172.21.15.161:0/165941590 shutdown_connections 2024-03-28T23:31:49.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.699+0000 7f07b048a640 1 -- 172.21.15.161:0/165941590 wait complete. 2024-03-28T23:31:49.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.700+0000 7f07b048a640 1 Processor -- start 2024-03-28T23:31:49.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.700+0000 7f07b048a640 1 -- start start 2024-03-28T23:31:49.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.701+0000 7f07b048a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07a806ce80 0x7f07a8148a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.701+0000 7f07b048a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07a80a9680 0x7f07a81e9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.701+0000 7f07b048a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f07a814be70 0x7f07a81dbde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.701+0000 7f07b048a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f07a80aba30 con 0x7f07a806ce80 2024-03-28T23:31:49.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.701+0000 7f07b048a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f07a80ab8b0 con 0x7f07a80a9680 2024-03-28T23:31:49.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.701+0000 7f07b048a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f07a806b150 con 0x7f07a814be70 2024-03-28T23:31:49.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.701+0000 7f07ae226640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07a806ce80 0x7f07a8148a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.701+0000 7f07ae226640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07a806ce80 0x7f07a8148a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58600/0 (socket says 172.21.15.161:58600) 2024-03-28T23:31:49.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.701+0000 7f07ae226640 1 -- 172.21.15.161:0/2837861796 learned_addr learned my addr 172.21.15.161:0/2837861796 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:49.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.701+0000 7f07aea27640 1 --2- 172.21.15.161:0/2837861796 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f07a814be70 0x7f07a81dbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.701+0000 7f07ae226640 1 -- 172.21.15.161:0/2837861796 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f07a814be70 msgr2=0x7f07a81dbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:49.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.701+0000 7f07ae226640 1 --2- 172.21.15.161:0/2837861796 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f07a814be70 0x7f07a81dbde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.701+0000 7f07ada25640 1 --2- 172.21.15.161:0/2837861796 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07a80a9680 0x7f07a81e9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.701+0000 7f07ae226640 1 -- 172.21.15.161:0/2837861796 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07a80a9680 msgr2=0x7f07a81e9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:49.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.701+0000 7f07ae226640 1 --2- 172.21.15.161:0/2837861796 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07a80a9680 0x7f07a81e9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.701+0000 7f07ae226640 1 -- 172.21.15.161:0/2837861796 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07a8148760 con 0x7f07a806ce80 2024-03-28T23:31:49.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.701+0000 7f07aea27640 1 --2- 172.21.15.161:0/2837861796 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f07a814be70 0x7f07a81dbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:49.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.701+0000 7f07ada25640 1 --2- 172.21.15.161:0/2837861796 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07a80a9680 0x7f07a81e9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:49.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.702+0000 7f07ae226640 1 --2- 172.21.15.161:0/2837861796 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07a806ce80 0x7f07a8148a40 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f079800b420 tx=0x7f079800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:49.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.702+0000 7f079effd640 1 -- 172.21.15.161:0/2837861796 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0798011830 con 0x7f07a806ce80 2024-03-28T23:31:49.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.702+0000 7f079effd640 1 -- 172.21.15.161:0/2837861796 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0798016070 con 0x7f07a806ce80 2024-03-28T23:31:49.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.702+0000 7f079effd640 1 -- 172.21.15.161:0/2837861796 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0798011c10 con 0x7f07a806ce80 2024-03-28T23:31:49.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.702+0000 7f07b048a640 1 -- 172.21.15.161:0/2837861796 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07a81ea870 con 0x7f07a806ce80 2024-03-28T23:31:49.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.702+0000 7f07b048a640 1 -- 172.21.15.161:0/2837861796 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07a81eab50 con 0x7f07a806ce80 2024-03-28T23:31:49.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.703+0000 7f07b048a640 1 -- 172.21.15.161:0/2837861796 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07a8072830 con 0x7f07a806ce80 2024-03-28T23:31:49.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.703+0000 7f079effd640 1 -- 172.21.15.161:0/2837861796 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f079801b460 con 0x7f07a806ce80 2024-03-28T23:31:49.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.704+0000 7f079effd640 1 --2- 172.21.15.161:0/2837861796 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f078c06b4d0 0x7f078c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:49.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.704+0000 7f07ada25640 1 --2- 172.21.15.161:0/2837861796 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f078c06b4d0 0x7f078c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:49.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.704+0000 7f079effd640 1 -- 172.21.15.161:0/2837861796 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f0798089e50 con 0x7f07a806ce80 2024-03-28T23:31:49.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.704+0000 7f07ada25640 1 --2- 172.21.15.161:0/2837861796 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f078c06b4d0 0x7f078c06d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f07a4006fd0 tx=0x7f07a4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:49.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.708+0000 7f079effd640 1 -- 172.21.15.161:0/2837861796 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f07a8072830 con 0x7f07a806ce80 2024-03-28T23:31:49.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.924+0000 7f07b048a640 1 -- 172.21.15.161:0/2837861796 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07a804f4b0 con 0x7f07a806ce80 2024-03-28T23:31:49.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.925+0000 7f079effd640 1 -- 172.21.15.161:0/2837861796 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5190 (secure 0 0 0) 0x7f0798020020 con 0x7f07a806ce80 2024-03-28T23:31:49.927 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:49.927 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5186,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/1945122204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1945122204},{"type":"v1","addr":"172.21.15.170:6835","nonce":1945122204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":5542,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/3666899516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3666899516},{"type":"v1","addr":"172.21.15.170:6833","nonce":3666899516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5817,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/1300583128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1300583128},{"type":"v1","addr":"172.21.15.161:6836","nonce":1300583128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:31:39.725625+0000","modified":"2024-03-28T23:31:41.818149+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5820},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5820":{"gid":5820,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":4,"addr":"172.21.15.161:6837/544971001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":544971001},{"type":"v1","addr":"172.21.15.161:6837","nonce":544971001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5820,"qdb_cluster":[5820]},"id":3}]} 2024-03-28T23:31:49.927 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 30 2024-03-28T23:31:49.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.927+0000 7f07b048a640 1 -- 172.21.15.161:0/2837861796 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f078c06b4d0 msgr2=0x7f078c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:49.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.927+0000 7f07b048a640 1 --2- 172.21.15.161:0/2837861796 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f078c06b4d0 0x7f078c06d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f07a4006fd0 tx=0x7f07a4008040 comp rx=0 tx=0).stop 2024-03-28T23:31:49.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.927+0000 7f07b048a640 1 -- 172.21.15.161:0/2837861796 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07a806ce80 msgr2=0x7f07a8148a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:49.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.927+0000 7f07b048a640 1 --2- 172.21.15.161:0/2837861796 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07a806ce80 0x7f07a8148a40 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f079800b420 tx=0x7f079800b8e0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.927+0000 7f07b048a640 1 -- 172.21.15.161:0/2837861796 shutdown_connections 2024-03-28T23:31:49.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.927+0000 7f07b048a640 1 --2- 172.21.15.161:0/2837861796 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f078c06b4d0 0x7f078c06d990 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.927+0000 7f07b048a640 1 --2- 172.21.15.161:0/2837861796 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f07a814be70 0x7f07a81dbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.927+0000 7f07b048a640 1 --2- 172.21.15.161:0/2837861796 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07a80a9680 0x7f07a81e9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.927+0000 7f07b048a640 1 --2- 172.21.15.161:0/2837861796 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07a806ce80 0x7f07a8148a40 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:49.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.927+0000 7f07b048a640 1 -- 172.21.15.161:0/2837861796 >> 172.21.15.161:0/2837861796 conn(0x7f07a80c6670 msgr2=0x7f07a80ab1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:49.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.927+0000 7f07b048a640 1 -- 172.21.15.161:0/2837861796 shutdown_connections 2024-03-28T23:31:49.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:49.927+0000 7f07b048a640 1 -- 172.21.15.161:0/2837861796 wait complete. 2024-03-28T23:31:49.940 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 3, "epoch": 30, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 3, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:31:39.725625+0000", "damaged": [], "data_pools": [ 8 ], "enabled": true, "epoch": 30, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5820": { "addr": "172.21.15.161:6837/544971001", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6835", "nonce": 544971001, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 544971001, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5820, "incarnation": 28, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 7, "modified": "2024-03-28T23:31:41.818149+0000", "qdb_cluster": [ 5820 ], "qdb_leader": 5820, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5820 } } } ], "standbys": [ { "addr": "172.21.15.170:6835/1945122204", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6834", "nonce": 1945122204, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 1945122204, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 27, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5186, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.170:6833/3666899516", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 3666899516, "type": "v2" }, { "addr": "172.21.15.170:6833", "nonce": 3666899516, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 26, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5542, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6836/1300583128", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 1300583128, "type": "v2" }, { "addr": "172.21.15.161:6836", "nonce": 1300583128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 26, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5817, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:31:49.940 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-28T23:31:49.940 INFO:tasks.cephfs.test_failover:current = [{'gid': 5820, 'name': 'a', 'rank': 0, 'incarnation': 28, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.161:6837/544971001', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 544971001}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 544971001}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 5186, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/1945122204', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6834', 'nonce': 1945122204}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 1945122204}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 27}, {'gid': 5817, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6836/1300583128', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 1300583128}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 1300583128}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 26}, {'gid': 5542, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6833/3666899516', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 3666899516}, {'type': 'v1', 'addr': '172.21.15.170:6833', 'nonce': 3666899516}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 26}] 2024-03-28T23:31:49.940 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.b mds_join_fs cephfs2 2024-03-28T23:31:50.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.042+0000 7f482748a640 1 Processor -- start 2024-03-28T23:31:50.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.042+0000 7f482748a640 1 -- start start 2024-03-28T23:31:50.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.043+0000 7f482748a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f48200c0330 0x7f48200c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:50.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.043+0000 7f482748a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48200c6f60 con 0x7f48201589b0 2024-03-28T23:31:50.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.043+0000 7f482748a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48200c6120 con 0x7f4820152bc0 2024-03-28T23:31:50.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.043+0000 7f482748a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f48200c6400 con 0x7f48200c0330 2024-03-28T23:31:50.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.043+0000 7f4824a25640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f48200c0330 0x7f48200c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:50.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.043+0000 7f4825a27640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f48201589b0 0x7f4820158e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44050/0 (socket says 172.21.15.161:44050) 2024-03-28T23:31:50.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.043+0000 7f4825a27640 1 -- 172.21.15.161:0/4202241809 learned_addr learned my addr 172.21.15.161:0/4202241809 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:50.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.044+0000 7f4824a25640 1 -- 172.21.15.161:0/4202241809 >> v1:172.21.15.170:6789/0 conn(0x7f4820152bc0 legacy=0x7f48200c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:50.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.044+0000 7f4824a25640 1 -- 172.21.15.161:0/4202241809 >> v1:172.21.15.161:6789/0 conn(0x7f48201589b0 legacy=0x7f4820158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:50.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.044+0000 7f4817fff640 1 -- 172.21.15.161:0/4202241809 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 385598321 0 0) 0x7f48200c6f60 con 0x7f48201589b0 2024-03-28T23:31:50.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.044+0000 7f4824a25640 1 -- 172.21.15.161:0/4202241809 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48200c96d0 con 0x7f48200c0330 2024-03-28T23:31:50.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.045+0000 7f4824a25640 1 --2- 172.21.15.161:0/4202241809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48200c0330 0x7f48200c9190 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f48100097a0 tx=0x7f481002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=975c35e8f0403727 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:50.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.045+0000 7f4817fff640 1 -- 172.21.15.161:0/4202241809 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4810034070 con 0x7f48200c0330 2024-03-28T23:31:50.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.045+0000 7f4817fff640 1 -- 172.21.15.161:0/4202241809 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4810030610 con 0x7f48200c0330 2024-03-28T23:31:50.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.045+0000 7f4817fff640 1 -- 172.21.15.161:0/4202241809 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48100309e0 con 0x7f48200c0330 2024-03-28T23:31:50.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.045+0000 7f482748a640 1 -- 172.21.15.161:0/4202241809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48200c0330 msgr2=0x7f48200c9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:50.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.045+0000 7f482748a640 1 --2- 172.21.15.161:0/4202241809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48200c0330 0x7f48200c9190 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f48100097a0 tx=0x7f481002b960 comp rx=0 tx=0).stop 2024-03-28T23:31:50.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.046+0000 7f482748a640 1 -- 172.21.15.161:0/4202241809 shutdown_connections 2024-03-28T23:31:50.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.046+0000 7f482748a640 1 --2- 172.21.15.161:0/4202241809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48200c0330 0x7f48200c9190 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.046+0000 7f482748a640 1 -- 172.21.15.161:0/4202241809 >> 172.21.15.161:0/4202241809 conn(0x7f48200caa80 msgr2=0x7f48200cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:50.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.046+0000 7f482748a640 1 -- 172.21.15.161:0/4202241809 shutdown_connections 2024-03-28T23:31:50.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.046+0000 7f482748a640 1 -- 172.21.15.161:0/4202241809 wait complete. 2024-03-28T23:31:50.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.047+0000 7f482748a640 1 Processor -- start 2024-03-28T23:31:50.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.047+0000 7f482748a640 1 -- start start 2024-03-28T23:31:50.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.047+0000 7f482748a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48200c0330 0x7f4820146820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:50.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.047+0000 7f482748a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4820152bc0 0x7f4820146d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:50.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.047+0000 7f482748a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48201589b0 0x7f48201ee200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:50.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.047+0000 7f482748a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4820152fa0 con 0x7f48200c0330 2024-03-28T23:31:50.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.047+0000 7f482748a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f48200c6bf0 con 0x7f48201589b0 2024-03-28T23:31:50.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.047+0000 7f482748a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f48201474b0 con 0x7f4820152bc0 2024-03-28T23:31:50.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.047+0000 7f4825226640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48200c0330 0x7f4820146820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:50.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.047+0000 7f4825226640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48200c0330 0x7f4820146820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58628/0 (socket says 172.21.15.161:58628) 2024-03-28T23:31:50.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.047+0000 7f4825226640 1 -- 172.21.15.161:0/2980229533 learned_addr learned my addr 172.21.15.161:0/2980229533 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:50.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.048+0000 7f4824a25640 1 --2- 172.21.15.161:0/2980229533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4820152bc0 0x7f4820146d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:50.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.048+0000 7f4825a27640 1 --2- 172.21.15.161:0/2980229533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48201589b0 0x7f48201ee200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:50.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.048+0000 7f4825226640 1 -- 172.21.15.161:0/2980229533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4820152bc0 msgr2=0x7f4820146d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:50.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.048+0000 7f4825226640 1 --2- 172.21.15.161:0/2980229533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4820152bc0 0x7f4820146d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.048+0000 7f4825226640 1 -- 172.21.15.161:0/2980229533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48201589b0 msgr2=0x7f48201ee200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:50.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.048+0000 7f4825226640 1 --2- 172.21.15.161:0/2980229533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48201589b0 0x7f48201ee200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.048+0000 7f4825226640 1 -- 172.21.15.161:0/2980229533 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48201eeff0 con 0x7f48200c0330 2024-03-28T23:31:50.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.048+0000 7f4824a25640 1 --2- 172.21.15.161:0/2980229533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4820152bc0 0x7f4820146d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:50.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.048+0000 7f4825a27640 1 --2- 172.21.15.161:0/2980229533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48201589b0 0x7f48201ee200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:50.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.048+0000 7f4825226640 1 --2- 172.21.15.161:0/2980229533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48200c0330 0x7f4820146820 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f481c00c450 tx=0x7f481c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:50.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.048+0000 7f4815ffb640 1 -- 172.21.15.161:0/2980229533 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c0052d0 con 0x7f48200c0330 2024-03-28T23:31:50.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.048+0000 7f4815ffb640 1 -- 172.21.15.161:0/2980229533 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f481c005470 con 0x7f48200c0330 2024-03-28T23:31:50.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.048+0000 7f482748a640 1 -- 172.21.15.161:0/2980229533 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4820151d60 con 0x7f48200c0330 2024-03-28T23:31:50.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.048+0000 7f4815ffb640 1 -- 172.21.15.161:0/2980229533 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c012870 con 0x7f48200c0330 2024-03-28T23:31:50.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.048+0000 7f482748a640 1 -- 172.21.15.161:0/2980229533 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4820152320 con 0x7f48200c0330 2024-03-28T23:31:50.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.049+0000 7f482748a640 1 -- 172.21.15.161:0/2980229533 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4820158e00 con 0x7f48200c0330 2024-03-28T23:31:50.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.050+0000 7f4815ffb640 1 -- 172.21.15.161:0/2980229533 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f481c012a10 con 0x7f48200c0330 2024-03-28T23:31:50.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.050+0000 7f4815ffb640 1 --2- 172.21.15.161:0/2980229533 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f480406b4d0 0x7f480406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:50.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.050+0000 7f4824a25640 1 --2- 172.21.15.161:0/2980229533 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f480406b4d0 0x7f480406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:50.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.050+0000 7f4815ffb640 1 -- 172.21.15.161:0/2980229533 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f481c089f40 con 0x7f48200c0330 2024-03-28T23:31:50.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.051+0000 7f4824a25640 1 --2- 172.21.15.161:0/2980229533 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f480406b4d0 0x7f480406d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4810036000 tx=0x7f4810008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:50.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.055+0000 7f4815ffb640 1 -- 172.21.15.161:0/2980229533 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4820158e00 con 0x7f48200c0330 2024-03-28T23:31:50.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.228+0000 7f482748a640 1 -- 172.21.15.161:0/2980229533 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) -- 0x7f482004f4b0 con 0x7f48200c0330 2024-03-28T23:31:50.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.231+0000 7f4815ffb640 1 -- 172.21.15.161:0/2980229533 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v16)=0 v16) ==== 117+0+0 (secure 0 0 0) 0x7f481c058cc0 con 0x7f48200c0330 2024-03-28T23:31:50.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.233+0000 7f482748a640 1 -- 172.21.15.161:0/2980229533 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f480406b4d0 msgr2=0x7f480406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:50.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.233+0000 7f482748a640 1 --2- 172.21.15.161:0/2980229533 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f480406b4d0 0x7f480406d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4810036000 tx=0x7f4810008030 comp rx=0 tx=0).stop 2024-03-28T23:31:50.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.233+0000 7f482748a640 1 -- 172.21.15.161:0/2980229533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48200c0330 msgr2=0x7f4820146820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:50.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.233+0000 7f482748a640 1 --2- 172.21.15.161:0/2980229533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48200c0330 0x7f4820146820 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f481c00c450 tx=0x7f481c00c910 comp rx=0 tx=0).stop 2024-03-28T23:31:50.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.233+0000 7f482748a640 1 -- 172.21.15.161:0/2980229533 shutdown_connections 2024-03-28T23:31:50.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.233+0000 7f482748a640 1 --2- 172.21.15.161:0/2980229533 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f480406b4d0 0x7f480406d990 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.233+0000 7f482748a640 1 --2- 172.21.15.161:0/2980229533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48201589b0 0x7f48201ee200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.233+0000 7f482748a640 1 --2- 172.21.15.161:0/2980229533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4820152bc0 0x7f4820146d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.233+0000 7f482748a640 1 --2- 172.21.15.161:0/2980229533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48200c0330 0x7f4820146820 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.233+0000 7f482748a640 1 -- 172.21.15.161:0/2980229533 >> 172.21.15.161:0/2980229533 conn(0x7f48200caa80 msgr2=0x7f482014ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:50.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.233+0000 7f482748a640 1 -- 172.21.15.161:0/2980229533 shutdown_connections 2024-03-28T23:31:50.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.233+0000 7f482748a640 1 -- 172.21.15.161:0/2980229533 wait complete. 2024-03-28T23:31:50.246 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs2 2024-03-28T23:31:50.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.349+0000 7fe7b9343640 1 Processor -- start 2024-03-28T23:31:50.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.349+0000 7fe7b9343640 1 -- start start 2024-03-28T23:31:50.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.350+0000 7fe7b9343640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe7b40c0330 0x7fe7b40c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:50.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.350+0000 7fe7b9343640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7b41589b0 0x7fe7b4158e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:50.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.350+0000 7fe7b9343640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7b40c6f60 con 0x7fe7b4152bc0 2024-03-28T23:31:50.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.350+0000 7fe7b9343640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe7b40c6120 con 0x7fe7b40c0330 2024-03-28T23:31:50.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.350+0000 7fe7b9343640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe7b40c6400 con 0x7fe7b41589b0 2024-03-28T23:31:50.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.350+0000 7fe7b27fc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe7b40c0330 0x7fe7b40c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:50.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.350+0000 7fe7b2ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe7b4152bc0 0x7fe7b40c8a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44058/0 (socket says 172.21.15.161:44058) 2024-03-28T23:31:50.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.350+0000 7fe7b2ffd640 1 -- 172.21.15.161:0/1298130767 learned_addr learned my addr 172.21.15.161:0/1298130767 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:50.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.350+0000 7fe7b37fe640 1 --2- 172.21.15.161:0/1298130767 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7b41589b0 0x7fe7b4158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:50.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.351+0000 7fe7b27fc640 1 -- 172.21.15.161:0/1298130767 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7b41589b0 msgr2=0x7fe7b4158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:50.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.351+0000 7fe7b27fc640 1 --2- 172.21.15.161:0/1298130767 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7b41589b0 0x7fe7b4158e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.351+0000 7fe7b27fc640 1 -- 172.21.15.161:0/1298130767 >> v1:172.21.15.161:6789/0 conn(0x7fe7b4152bc0 legacy=0x7fe7b40c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:50.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.351+0000 7fe7b27fc640 1 -- 172.21.15.161:0/1298130767 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7b40c96d0 con 0x7fe7b40c0330 2024-03-28T23:31:50.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.351+0000 7fe7b1ffb640 1 -- 172.21.15.161:0/1298130767 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3737105008 0 0) 0x7fe7b40c6f60 con 0x7fe7b4152bc0 2024-03-28T23:31:50.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.351+0000 7fe7b27fc640 1 --2- 172.21.15.161:0/1298130767 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7b40c0330 0x7fe7b40c9190 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fe7a80097a0 tx=0x7fe7a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3460865308a40604 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:50.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.351+0000 7fe7b1ffb640 1 -- 172.21.15.161:0/1298130767 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a8035020 con 0x7fe7b40c0330 2024-03-28T23:31:50.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.351+0000 7fe7b1ffb640 1 -- 172.21.15.161:0/1298130767 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7a8031830 con 0x7fe7b40c0330 2024-03-28T23:31:50.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.352+0000 7fe7b1ffb640 1 -- 172.21.15.161:0/1298130767 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a8031c00 con 0x7fe7b40c0330 2024-03-28T23:31:50.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.352+0000 7fe7b9343640 1 -- 172.21.15.161:0/1298130767 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7b40c0330 msgr2=0x7fe7b40c9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:50.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.352+0000 7fe7b9343640 1 --2- 172.21.15.161:0/1298130767 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7b40c0330 0x7fe7b40c9190 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fe7a80097a0 tx=0x7fe7a802b8d0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.352+0000 7fe7b9343640 1 -- 172.21.15.161:0/1298130767 shutdown_connections 2024-03-28T23:31:50.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.352+0000 7fe7b9343640 1 --2- 172.21.15.161:0/1298130767 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7b41589b0 0x7fe7b4158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.352+0000 7fe7b9343640 1 --2- 172.21.15.161:0/1298130767 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7b40c0330 0x7fe7b40c9190 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.352+0000 7fe7b9343640 1 -- 172.21.15.161:0/1298130767 >> 172.21.15.161:0/1298130767 conn(0x7fe7b40caa80 msgr2=0x7fe7b40cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:50.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.352+0000 7fe7b9343640 1 -- 172.21.15.161:0/1298130767 shutdown_connections 2024-03-28T23:31:50.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.352+0000 7fe7b9343640 1 -- 172.21.15.161:0/1298130767 wait complete. 2024-03-28T23:31:50.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.353+0000 7fe7b9343640 1 Processor -- start 2024-03-28T23:31:50.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.353+0000 7fe7b9343640 1 -- start start 2024-03-28T23:31:50.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.354+0000 7fe7b9343640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7b40c0330 0x7fe7b41570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:50.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.354+0000 7fe7b9343640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7b4152bc0 0x7fe7b406c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:50.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.354+0000 7fe7b9343640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7b41589b0 0x7fe7b4154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:50.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.354+0000 7fe7b9343640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b4152fa0 con 0x7fe7b4152bc0 2024-03-28T23:31:50.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.354+0000 7fe7b9343640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b40c6bf0 con 0x7fe7b41589b0 2024-03-28T23:31:50.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.354+0000 7fe7b9343640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe7b4158630 con 0x7fe7b40c0330 2024-03-28T23:31:50.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.354+0000 7fe7b27fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7b4152bc0 0x7fe7b406c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:50.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.354+0000 7fe7b27fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7b4152bc0 0x7fe7b406c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58650/0 (socket says 172.21.15.161:58650) 2024-03-28T23:31:50.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.354+0000 7fe7b27fc640 1 -- 172.21.15.161:0/4116057594 learned_addr learned my addr 172.21.15.161:0/4116057594 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:50.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.354+0000 7fe7b2ffd640 1 --2- 172.21.15.161:0/4116057594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7b40c0330 0x7fe7b41570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:50.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.354+0000 7fe7b37fe640 1 --2- 172.21.15.161:0/4116057594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7b41589b0 0x7fe7b4154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:50.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.354+0000 7fe7b27fc640 1 -- 172.21.15.161:0/4116057594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7b40c0330 msgr2=0x7fe7b41570d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:50.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.354+0000 7fe7b27fc640 1 --2- 172.21.15.161:0/4116057594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7b40c0330 0x7fe7b41570d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.354+0000 7fe7b27fc640 1 -- 172.21.15.161:0/4116057594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7b41589b0 msgr2=0x7fe7b4154f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:50.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.354+0000 7fe7b27fc640 1 --2- 172.21.15.161:0/4116057594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7b41589b0 0x7fe7b4154f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.354+0000 7fe7b27fc640 1 -- 172.21.15.161:0/4116057594 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7b4157d90 con 0x7fe7b4152bc0 2024-03-28T23:31:50.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.355+0000 7fe7b27fc640 1 --2- 172.21.15.161:0/4116057594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7b4152bc0 0x7fe7b406c610 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe7a802bee0 tx=0x7fe7a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:50.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.355+0000 7fe79b7fe640 1 -- 172.21.15.161:0/4116057594 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a8041070 con 0x7fe7b4152bc0 2024-03-28T23:31:50.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.355+0000 7fe79b7fe640 1 -- 172.21.15.161:0/4116057594 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7a803b920 con 0x7fe7b4152bc0 2024-03-28T23:31:50.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.355+0000 7fe7b9343640 1 -- 172.21.15.161:0/4116057594 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7b4158070 con 0x7fe7b4152bc0 2024-03-28T23:31:50.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.355+0000 7fe79b7fe640 1 -- 172.21.15.161:0/4116057594 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a8035040 con 0x7fe7b4152bc0 2024-03-28T23:31:50.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.355+0000 7fe7b9343640 1 -- 172.21.15.161:0/4116057594 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7b41549e0 con 0x7fe7b4152bc0 2024-03-28T23:31:50.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.356+0000 7fe7b9343640 1 -- 172.21.15.161:0/4116057594 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7b404f4b0 con 0x7fe7b4152bc0 2024-03-28T23:31:50.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.356+0000 7fe79b7fe640 1 -- 172.21.15.161:0/4116057594 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe7a8038020 con 0x7fe7b4152bc0 2024-03-28T23:31:50.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.357+0000 7fe79b7fe640 1 --2- 172.21.15.161:0/4116057594 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe7a006b4d0 0x7fe7a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:50.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.357+0000 7fe79b7fe640 1 -- 172.21.15.161:0/4116057594 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7fe7a80abd20 con 0x7fe7b4152bc0 2024-03-28T23:31:50.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.357+0000 7fe7b2ffd640 1 --2- 172.21.15.161:0/4116057594 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe7a006b4d0 0x7fe7a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:50.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.357+0000 7fe7b2ffd640 1 --2- 172.21.15.161:0/4116057594 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe7a006b4d0 0x7fe7a006d990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4007fd0 tx=0x7fe7a4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:50.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.361+0000 7fe79b7fe640 1 -- 172.21.15.161:0/4116057594 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe7a807aa20 con 0x7fe7b4152bc0 2024-03-28T23:31:50.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.537+0000 7fe7b9343640 1 -- 172.21.15.161:0/4116057594 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) -- 0x7fe7b4154bb0 con 0x7fe7b4152bc0 2024-03-28T23:31:50.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.540+0000 7fe79b7fe640 1 -- 172.21.15.161:0/4116057594 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v17)=0 v17) ==== 117+0+0 (secure 0 0 0) 0x7fe7a807a320 con 0x7fe7b4152bc0 2024-03-28T23:31:50.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.542+0000 7fe7b9343640 1 -- 172.21.15.161:0/4116057594 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe7a006b4d0 msgr2=0x7fe7a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:50.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.542+0000 7fe7b9343640 1 --2- 172.21.15.161:0/4116057594 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe7a006b4d0 0x7fe7a006d990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4007fd0 tx=0x7fe7a4009040 comp rx=0 tx=0).stop 2024-03-28T23:31:50.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.542+0000 7fe7b9343640 1 -- 172.21.15.161:0/4116057594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7b4152bc0 msgr2=0x7fe7b406c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:50.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.542+0000 7fe7b9343640 1 --2- 172.21.15.161:0/4116057594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7b4152bc0 0x7fe7b406c610 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe7a802bee0 tx=0x7fe7a80046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.542+0000 7fe7b9343640 1 -- 172.21.15.161:0/4116057594 shutdown_connections 2024-03-28T23:31:50.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.542+0000 7fe7b9343640 1 --2- 172.21.15.161:0/4116057594 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe7a006b4d0 0x7fe7a006d990 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.542+0000 7fe7b9343640 1 --2- 172.21.15.161:0/4116057594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7b4152bc0 0x7fe7b406c610 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.542+0000 7fe7b9343640 1 --2- 172.21.15.161:0/4116057594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7b41589b0 0x7fe7b4154f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.542+0000 7fe7b9343640 1 --2- 172.21.15.161:0/4116057594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7b40c0330 0x7fe7b41570d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.542+0000 7fe7b9343640 1 -- 172.21.15.161:0/4116057594 >> 172.21.15.161:0/4116057594 conn(0x7fe7b40caa80 msgr2=0x7fe7b40cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:50.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.542+0000 7fe7b9343640 1 -- 172.21.15.161:0/4116057594 shutdown_connections 2024-03-28T23:31:50.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.542+0000 7fe7b9343640 1 -- 172.21.15.161:0/4116057594 wait complete. 2024-03-28T23:31:50.554 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.c mds_join_fs cephfs2 2024-03-28T23:31:50.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.662+0000 7f7a1565c640 1 Processor -- start 2024-03-28T23:31:50.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.662+0000 7f7a1565c640 1 -- start start 2024-03-28T23:31:50.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.663+0000 7f7a1565c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a10152bc0 0x7f7a100c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:50.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.663+0000 7f7a1565c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7a101589b0 0x7f7a10158e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:50.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.663+0000 7f7a1565c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a100c6f60 con 0x7f7a100c8a80 2024-03-28T23:31:50.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.663+0000 7f7a1565c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f7a100c6120 con 0x7f7a101589b0 2024-03-28T23:31:50.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.663+0000 7f7a1565c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7a100c6400 con 0x7f7a10152bc0 2024-03-28T23:31:50.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.663+0000 7f7a0effd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a10152bc0 0x7f7a100c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:50.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.663+0000 7f7a0f7fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7a101589b0 0x7f7a10158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:50.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.663+0000 7f7a0f7fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7a101589b0 0x7f7a10158e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:40816/0 (socket says 172.21.15.161:40816) 2024-03-28T23:31:50.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.663+0000 7f7a0effd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a10152bc0 0x7f7a100c0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:60100/0 (socket says 172.21.15.161:60100) 2024-03-28T23:31:50.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.663+0000 7f7a0f7fe640 1 -- 172.21.15.161:0/2343882090 learned_addr learned my addr 172.21.15.161:0/2343882090 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:50.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.664+0000 7f7a0f7fe640 1 -- 172.21.15.161:0/2343882090 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a10152bc0 msgr2=0x7f7a100c0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:50.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.664+0000 7f7a0f7fe640 1 --2- 172.21.15.161:0/2343882090 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a10152bc0 0x7f7a100c0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.664+0000 7f7a0f7fe640 1 -- 172.21.15.161:0/2343882090 >> v1:172.21.15.170:6789/0 conn(0x7f7a100c8a80 legacy=0x7f7a100c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:50.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.664+0000 7f7a0f7fe640 1 -- 172.21.15.161:0/2343882090 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a100c7240 con 0x7f7a101589b0 2024-03-28T23:31:50.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.664+0000 7f7a0e7fc640 1 -- 172.21.15.161:0/2343882090 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2379373991 0 0) 0x7f7a100c6f60 con 0x7f7a100c8a80 2024-03-28T23:31:50.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.664+0000 7f7a0effd640 1 --2- 172.21.15.161:0/2343882090 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a10152bc0 0x7f7a100c0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:50.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.665+0000 7f7a0f7fe640 1 --2- 172.21.15.161:0/2343882090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a101589b0 0x7f7a10158e10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7a08009e50 tx=0x7f7a0802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86d153dc166e276d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:50.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.665+0000 7f7a0e7fc640 1 -- 172.21.15.161:0/2343882090 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0802cb00 con 0x7f7a101589b0 2024-03-28T23:31:50.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.665+0000 7f7a0e7fc640 1 -- 172.21.15.161:0/2343882090 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a0802cca0 con 0x7f7a101589b0 2024-03-28T23:31:50.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.665+0000 7f7a0e7fc640 1 -- 172.21.15.161:0/2343882090 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a08034610 con 0x7f7a101589b0 2024-03-28T23:31:50.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.665+0000 7f7a1565c640 1 -- 172.21.15.161:0/2343882090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a101589b0 msgr2=0x7f7a10158e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:50.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.665+0000 7f7a1565c640 1 --2- 172.21.15.161:0/2343882090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a101589b0 0x7f7a10158e10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7a08009e50 tx=0x7f7a0802bf60 comp rx=0 tx=0).stop 2024-03-28T23:31:50.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.665+0000 7f7a1565c640 1 -- 172.21.15.161:0/2343882090 shutdown_connections 2024-03-28T23:31:50.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.665+0000 7f7a1565c640 1 --2- 172.21.15.161:0/2343882090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a101589b0 0x7f7a10158e10 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.665+0000 7f7a1565c640 1 --2- 172.21.15.161:0/2343882090 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a10152bc0 0x7f7a100c0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.665+0000 7f7a1565c640 1 -- 172.21.15.161:0/2343882090 >> 172.21.15.161:0/2343882090 conn(0x7f7a100caa80 msgr2=0x7f7a100cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:50.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.665+0000 7f7a1565c640 1 -- 172.21.15.161:0/2343882090 shutdown_connections 2024-03-28T23:31:50.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.665+0000 7f7a1565c640 1 -- 172.21.15.161:0/2343882090 wait complete. 2024-03-28T23:31:50.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.666+0000 7f7a1565c640 1 Processor -- start 2024-03-28T23:31:50.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.666+0000 7f7a1565c640 1 -- start start 2024-03-28T23:31:50.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.667+0000 7f7a1565c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a100c8a80 0x7f7a101510c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:50.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.667+0000 7f7a1565c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a10152bc0 0x7f7a10141310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:50.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.667+0000 7f7a1565c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a101589b0 0x7f7a10141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:50.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.667+0000 7f7a1565c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7a10152fa0 con 0x7f7a100c8a80 2024-03-28T23:31:50.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.667+0000 7f7a1565c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7a100c95f0 con 0x7f7a101589b0 2024-03-28T23:31:50.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.667+0000 7f7a1565c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7a10150b50 con 0x7f7a10152bc0 2024-03-28T23:31:50.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.667+0000 7f7a0effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a100c8a80 0x7f7a101510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:50.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.667+0000 7f7a0effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a100c8a80 0x7f7a101510c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34980/0 (socket says 172.21.15.161:34980) 2024-03-28T23:31:50.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.667+0000 7f7a0effd640 1 -- 172.21.15.161:0/3151513939 learned_addr learned my addr 172.21.15.161:0/3151513939 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:50.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.667+0000 7f7a07fff640 1 --2- 172.21.15.161:0/3151513939 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a10152bc0 0x7f7a10141310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:50.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.667+0000 7f7a0f7fe640 1 --2- 172.21.15.161:0/3151513939 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a101589b0 0x7f7a10141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:50.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.668+0000 7f7a0effd640 1 -- 172.21.15.161:0/3151513939 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a10152bc0 msgr2=0x7f7a10141310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:50.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.668+0000 7f7a0effd640 1 --2- 172.21.15.161:0/3151513939 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a10152bc0 0x7f7a10141310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.668+0000 7f7a0effd640 1 -- 172.21.15.161:0/3151513939 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a101589b0 msgr2=0x7f7a10141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:50.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.668+0000 7f7a0effd640 1 --2- 172.21.15.161:0/3151513939 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a101589b0 0x7f7a10141850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.668+0000 7f7a0effd640 1 -- 172.21.15.161:0/3151513939 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a1006cd20 con 0x7f7a100c8a80 2024-03-28T23:31:50.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.668+0000 7f7a0effd640 1 --2- 172.21.15.161:0/3151513939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a100c8a80 0x7f7a101510c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f7a0000ebe0 tx=0x7f7a0000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:50.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.668+0000 7f7a077fe640 1 -- 172.21.15.161:0/3151513939 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a00017070 con 0x7f7a100c8a80 2024-03-28T23:31:50.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.668+0000 7f7a077fe640 1 -- 172.21.15.161:0/3151513939 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a0000cc30 con 0x7f7a100c8a80 2024-03-28T23:31:50.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.668+0000 7f7a1565c640 1 -- 172.21.15.161:0/3151513939 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a101502b0 con 0x7f7a100c8a80 2024-03-28T23:31:50.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.668+0000 7f7a077fe640 1 -- 172.21.15.161:0/3151513939 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a00013470 con 0x7f7a100c8a80 2024-03-28T23:31:50.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.668+0000 7f7a1565c640 1 -- 172.21.15.161:0/3151513939 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a10150870 con 0x7f7a100c8a80 2024-03-28T23:31:50.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.669+0000 7f7a1565c640 1 -- 172.21.15.161:0/3151513939 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a1004f4b0 con 0x7f7a100c8a80 2024-03-28T23:31:50.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.669+0000 7f7a077fe640 1 -- 172.21.15.161:0/3151513939 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7a0001a070 con 0x7f7a100c8a80 2024-03-28T23:31:50.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.670+0000 7f7a077fe640 1 --2- 172.21.15.161:0/3151513939 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f79f006b4d0 0x7f79f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:50.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.670+0000 7f7a077fe640 1 -- 172.21.15.161:0/3151513939 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f7a0008a610 con 0x7f7a100c8a80 2024-03-28T23:31:50.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.670+0000 7f7a07fff640 1 --2- 172.21.15.161:0/3151513939 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f79f006b4d0 0x7f79f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:50.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.670+0000 7f7a07fff640 1 --2- 172.21.15.161:0/3151513939 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f79f006b4d0 0x7f79f006d990 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7a101525d0 tx=0x7f79f8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:50.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.675+0000 7f7a077fe640 1 -- 172.21.15.161:0/3151513939 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7a00058ca0 con 0x7f7a100c8a80 2024-03-28T23:31:50.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.855+0000 7f7a1565c640 1 -- 172.21.15.161:0/3151513939 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) -- 0x7f7a1014ca70 con 0x7f7a100c8a80 2024-03-28T23:31:50.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.858+0000 7f7a077fe640 1 -- 172.21.15.161:0/3151513939 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v18)=0 v18) ==== 117+0+0 (secure 0 0 0) 0x7f7a0005cd40 con 0x7f7a100c8a80 2024-03-28T23:31:50.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.860+0000 7f7a1565c640 1 -- 172.21.15.161:0/3151513939 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f79f006b4d0 msgr2=0x7f79f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:50.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.860+0000 7f7a1565c640 1 --2- 172.21.15.161:0/3151513939 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f79f006b4d0 0x7f79f006d990 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7a101525d0 tx=0x7f79f8009040 comp rx=0 tx=0).stop 2024-03-28T23:31:50.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.860+0000 7f7a1565c640 1 -- 172.21.15.161:0/3151513939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a100c8a80 msgr2=0x7f7a101510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:50.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.860+0000 7f7a1565c640 1 --2- 172.21.15.161:0/3151513939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a100c8a80 0x7f7a101510c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f7a0000ebe0 tx=0x7f7a0000c520 comp rx=0 tx=0).stop 2024-03-28T23:31:50.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.860+0000 7f7a1565c640 1 -- 172.21.15.161:0/3151513939 shutdown_connections 2024-03-28T23:31:50.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.860+0000 7f7a1565c640 1 --2- 172.21.15.161:0/3151513939 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f79f006b4d0 0x7f79f006d990 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.860+0000 7f7a1565c640 1 --2- 172.21.15.161:0/3151513939 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a101589b0 0x7f7a10141850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.860+0000 7f7a1565c640 1 --2- 172.21.15.161:0/3151513939 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a10152bc0 0x7f7a10141310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.860+0000 7f7a1565c640 1 --2- 172.21.15.161:0/3151513939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a100c8a80 0x7f7a101510c0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.860+0000 7f7a1565c640 1 -- 172.21.15.161:0/3151513939 >> 172.21.15.161:0/3151513939 conn(0x7f7a100caa80 msgr2=0x7f7a100c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:50.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.860+0000 7f7a1565c640 1 -- 172.21.15.161:0/3151513939 shutdown_connections 2024-03-28T23:31:50.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.860+0000 7f7a1565c640 1 -- 172.21.15.161:0/3151513939 wait complete. 2024-03-28T23:31:50.877 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:31:50.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.978+0000 7febe52c2640 1 Processor -- start 2024-03-28T23:31:50.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.978+0000 7febe52c2640 1 -- start start 2024-03-28T23:31:50.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.979+0000 7febe52c2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7febe0152890 0x7febe00c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:50.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.979+0000 7febe52c2640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7febe00c8ed0 0x7febe00c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:50.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.979+0000 7febe52c2640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febe0158a30 con 0x7febe00c5950 2024-03-28T23:31:50.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.979+0000 7febe52c2640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7febe00c8480 con 0x7febe0152890 2024-03-28T23:31:50.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.979+0000 7febe52c2640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7febe0159640 con 0x7febe00c8ed0 2024-03-28T23:31:50.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.979+0000 7febdeffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7febe0152890 0x7febe00c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:50.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.979+0000 7febdeffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7febe0152890 0x7febe00c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34994/0 (socket says 172.21.15.161:34994) 2024-03-28T23:31:50.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.979+0000 7febdeffd640 1 -- 172.21.15.161:0/1923729092 learned_addr learned my addr 172.21.15.161:0/1923729092 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:50.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.980+0000 7febde7fc640 1 --2- 172.21.15.161:0/1923729092 >> v2:172.21.15.170:3300/0 conn(0x7febe00c8ed0 0x7febe00c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:50.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.980+0000 7febdeffd640 1 -- 172.21.15.161:0/1923729092 >> v2:172.21.15.170:3300/0 conn(0x7febe00c8ed0 msgr2=0x7febe00c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:50.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.980+0000 7febdeffd640 1 --2- 172.21.15.161:0/1923729092 >> v2:172.21.15.170:3300/0 conn(0x7febe00c8ed0 0x7febe00c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.980+0000 7febdeffd640 1 -- 172.21.15.161:0/1923729092 >> v1:172.21.15.170:6789/0 conn(0x7febe00c5950 legacy=0x7febe01553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:50.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.980+0000 7febdeffd640 1 -- 172.21.15.161:0/1923729092 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe0158d10 con 0x7febe0152890 2024-03-28T23:31:50.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.980+0000 7febde7fc640 1 --2- 172.21.15.161:0/1923729092 >> v2:172.21.15.170:3300/0 conn(0x7febe00c8ed0 0x7febe00c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:50.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.980+0000 7febdeffd640 1 --2- 172.21.15.161:0/1923729092 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febe0152890 0x7febe00c8990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7febe00ad6e0 tx=0x7febd002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=877e49deca508c57 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:50.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.980+0000 7febddffb640 1 -- 172.21.15.161:0/1923729092 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd0038070 con 0x7febe0152890 2024-03-28T23:31:50.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.980+0000 7febddffb640 1 -- 172.21.15.161:0/1923729092 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febd002f050 con 0x7febe0152890 2024-03-28T23:31:50.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.980+0000 7febddffb640 1 -- 172.21.15.161:0/1923729092 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd0031970 con 0x7febe0152890 2024-03-28T23:31:50.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.981+0000 7febe52c2640 1 -- 172.21.15.161:0/1923729092 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febe0152890 msgr2=0x7febe00c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:50.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.981+0000 7febe52c2640 1 --2- 172.21.15.161:0/1923729092 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febe0152890 0x7febe00c8990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7febe00ad6e0 tx=0x7febd002b7f0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.981+0000 7febe52c2640 1 -- 172.21.15.161:0/1923729092 shutdown_connections 2024-03-28T23:31:50.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.981+0000 7febe52c2640 1 --2- 172.21.15.161:0/1923729092 >> v2:172.21.15.170:3300/0 conn(0x7febe00c8ed0 0x7febe00c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.981+0000 7febe52c2640 1 --2- 172.21.15.161:0/1923729092 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febe0152890 0x7febe00c8990 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:50.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.981+0000 7febe52c2640 1 -- 172.21.15.161:0/1923729092 >> 172.21.15.161:0/1923729092 conn(0x7febe00ac620 msgr2=0x7febe00aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:50.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.981+0000 7febe52c2640 1 -- 172.21.15.161:0/1923729092 shutdown_connections 2024-03-28T23:31:50.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.981+0000 7febe52c2640 1 -- 172.21.15.161:0/1923729092 wait complete. 2024-03-28T23:31:50.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.982+0000 7febe52c2640 1 Processor -- start 2024-03-28T23:31:50.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.982+0000 7febe52c2640 1 -- start start 2024-03-28T23:31:50.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.983+0000 7febe52c2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7febe00c5950 0x7febe0158040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:50.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.983+0000 7febe52c2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7febe00c8ed0 0x7febe00c3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:50.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.983+0000 7febe52c2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febe0152890 0x7febe00c44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:50.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.983+0000 7febe52c2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7febe0152c70 con 0x7febe0152890 2024-03-28T23:31:50.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.983+0000 7febe52c2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7febe01592d0 con 0x7febe00c5950 2024-03-28T23:31:50.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.983+0000 7febe52c2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7febe0152df0 con 0x7febe00c8ed0 2024-03-28T23:31:50.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.983+0000 7febdf7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febe0152890 0x7febe00c44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:50.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.983+0000 7febdf7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febe0152890 0x7febe00c44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34998/0 (socket says 172.21.15.161:34998) 2024-03-28T23:31:50.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.983+0000 7febdf7fe640 1 -- 172.21.15.161:0/3384801071 learned_addr learned my addr 172.21.15.161:0/3384801071 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:50.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.983+0000 7febdeffd640 1 --2- 172.21.15.161:0/3384801071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7febe00c5950 0x7febe0158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:50.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.983+0000 7febde7fc640 1 --2- 172.21.15.161:0/3384801071 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7febe00c8ed0 0x7febe00c3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:50.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.983+0000 7febdf7fe640 1 -- 172.21.15.161:0/3384801071 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7febe00c8ed0 msgr2=0x7febe00c3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:51.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.983+0000 7febdf7fe640 1 --2- 172.21.15.161:0/3384801071 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7febe00c8ed0 0x7febe00c3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.983+0000 7febdf7fe640 1 -- 172.21.15.161:0/3384801071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7febe00c5950 msgr2=0x7febe0158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:51.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.983+0000 7febdf7fe640 1 --2- 172.21.15.161:0/3384801071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7febe00c5950 0x7febe0158040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.983+0000 7febdf7fe640 1 -- 172.21.15.161:0/3384801071 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe01ee120 con 0x7febe0152890 2024-03-28T23:31:51.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.983+0000 7febdeffd640 1 --2- 172.21.15.161:0/3384801071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7febe00c5950 0x7febe0158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:51.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.984+0000 7febdf7fe640 1 --2- 172.21.15.161:0/3384801071 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febe0152890 0x7febe00c44a0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7febd800b970 tx=0x7febd800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:51.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.984+0000 7febc77fe640 1 -- 172.21.15.161:0/3384801071 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd8019070 con 0x7febe0152890 2024-03-28T23:31:51.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.984+0000 7febc77fe640 1 -- 172.21.15.161:0/3384801071 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febd800cb40 con 0x7febe0152890 2024-03-28T23:31:51.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.984+0000 7febe52c2640 1 -- 172.21.15.161:0/3384801071 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febe01ee400 con 0x7febe0152890 2024-03-28T23:31:51.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.984+0000 7febc77fe640 1 -- 172.21.15.161:0/3384801071 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd8014440 con 0x7febe0152890 2024-03-28T23:31:51.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.984+0000 7febe52c2640 1 -- 172.21.15.161:0/3384801071 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febe01ee9c0 con 0x7febe0152890 2024-03-28T23:31:51.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.985+0000 7febe52c2640 1 -- 172.21.15.161:0/3384801071 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febe004f4b0 con 0x7febe0152890 2024-03-28T23:31:51.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.985+0000 7febc77fe640 1 -- 172.21.15.161:0/3384801071 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7febd801c070 con 0x7febe0152890 2024-03-28T23:31:51.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.986+0000 7febc77fe640 1 --2- 172.21.15.161:0/3384801071 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7febcc06b4d0 0x7febcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:51.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.986+0000 7febc77fe640 1 -- 172.21.15.161:0/3384801071 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7febd8089f60 con 0x7febe0152890 2024-03-28T23:31:51.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.986+0000 7febdeffd640 1 --2- 172.21.15.161:0/3384801071 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7febcc06b4d0 0x7febcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:51.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.986+0000 7febdeffd640 1 --2- 172.21.15.161:0/3384801071 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7febcc06b4d0 0x7febcc06d990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7febd0035000 tx=0x7febd0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:51.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:50.990+0000 7febc77fe640 1 -- 172.21.15.161:0/3384801071 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7febd8058ce0 con 0x7febe0152890 2024-03-28T23:31:51.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.170+0000 7febe52c2640 1 -- 172.21.15.161:0/3384801071 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7febe00c8140 con 0x7febe0152890 2024-03-28T23:31:51.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.171+0000 7febc77fe640 1 -- 172.21.15.161:0/3384801071 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) ==== 74+0+16978 (secure 0 0 0) 0x7febd80585e0 con 0x7febe0152890 2024-03-28T23:31:51.173 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:51.173 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":51,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:31:41.382740+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:31:37.940838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-28T23:31:38.692659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:31:51.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.173+0000 7febe52c2640 1 -- 172.21.15.161:0/3384801071 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7febcc06b4d0 msgr2=0x7febcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:51.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.173+0000 7febe52c2640 1 --2- 172.21.15.161:0/3384801071 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7febcc06b4d0 0x7febcc06d990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7febd0035000 tx=0x7febd0008030 comp rx=0 tx=0).stop 2024-03-28T23:31:51.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.173+0000 7febe52c2640 1 -- 172.21.15.161:0/3384801071 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febe0152890 msgr2=0x7febe00c44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:51.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.173+0000 7febe52c2640 1 --2- 172.21.15.161:0/3384801071 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febe0152890 0x7febe00c44a0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7febd800b970 tx=0x7febd800be30 comp rx=0 tx=0).stop 2024-03-28T23:31:51.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.173+0000 7febe52c2640 1 -- 172.21.15.161:0/3384801071 shutdown_connections 2024-03-28T23:31:51.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.173+0000 7febe52c2640 1 --2- 172.21.15.161:0/3384801071 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7febcc06b4d0 0x7febcc06d990 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.173+0000 7febe52c2640 1 --2- 172.21.15.161:0/3384801071 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febe0152890 0x7febe00c44a0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.173+0000 7febe52c2640 1 --2- 172.21.15.161:0/3384801071 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7febe00c8ed0 0x7febe00c3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.173+0000 7febe52c2640 1 --2- 172.21.15.161:0/3384801071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7febe00c5950 0x7febe0158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.173+0000 7febe52c2640 1 -- 172.21.15.161:0/3384801071 >> 172.21.15.161:0/3384801071 conn(0x7febe00ac620 msgr2=0x7febe006be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:51.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.173+0000 7febe52c2640 1 -- 172.21.15.161:0/3384801071 shutdown_connections 2024-03-28T23:31:51.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.173+0000 7febe52c2640 1 -- 172.21.15.161:0/3384801071 wait complete. 2024-03-28T23:31:51.186 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:31:37.940838+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:31:38.692659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:31:51.186 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:31:51.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.290+0000 7f765bd1e640 1 Processor -- start 2024-03-28T23:31:51.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.290+0000 7f765bd1e640 1 -- start start 2024-03-28T23:31:51.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.291+0000 7f765bd1e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7654152990 0x7f76540c52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:51.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.291+0000 7f765bd1e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f76540c8870 0x7f7654155480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:51.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.291+0000 7f765bd1e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76540c8d80 0x7f76541559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:51.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.291+0000 7f765bd1e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f7654152d70 con 0x7f7654152990 2024-03-28T23:31:51.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.291+0000 7f765bd1e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f7654158aa0 con 0x7f76540c8870 2024-03-28T23:31:51.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.291+0000 7f765bd1e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7654158d80 con 0x7f76540c8d80 2024-03-28T23:31:51.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.291+0000 7f7659aba640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7654152990 0x7f76540c52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:51.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.291+0000 7f7659aba640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7654152990 0x7f76540c52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35008/0 (socket says 172.21.15.161:35008) 2024-03-28T23:31:51.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.291+0000 7f7659aba640 1 -- 172.21.15.161:0/1739510724 learned_addr learned my addr 172.21.15.161:0/1739510724 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:51.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.291+0000 7f765a2bb640 1 --2- 172.21.15.161:0/1739510724 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76540c8d80 0x7f76541559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:51.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.291+0000 7f76592b9640 1 --2- 172.21.15.161:0/1739510724 >> v2:172.21.15.170:3300/0 conn(0x7f76540c8870 0x7f7654155480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:51.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.291+0000 7f7659aba640 1 -- 172.21.15.161:0/1739510724 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76540c8d80 msgr2=0x7f76541559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:51.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.291+0000 7f7659aba640 1 --2- 172.21.15.161:0/1739510724 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76540c8d80 0x7f76541559c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.291+0000 7f7659aba640 1 -- 172.21.15.161:0/1739510724 >> v2:172.21.15.170:3300/0 conn(0x7f76540c8870 msgr2=0x7f7654155480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:51.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.291+0000 7f7659aba640 1 --2- 172.21.15.161:0/1739510724 >> v2:172.21.15.170:3300/0 conn(0x7f76540c8870 0x7f7654155480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.291+0000 7f7659aba640 1 -- 172.21.15.161:0/1739510724 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7654159060 con 0x7f7654152990 2024-03-28T23:31:51.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.292+0000 7f7659aba640 1 --2- 172.21.15.161:0/1739510724 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7654152990 0x7f76540c52f0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f76540cb990 tx=0x7f765002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bde83967ba83a00d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:51.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.292+0000 7f7658ab8640 1 -- 172.21.15.161:0/1739510724 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f765002e070 con 0x7f7654152990 2024-03-28T23:31:51.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.292+0000 7f7658ab8640 1 -- 172.21.15.161:0/1739510724 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7650037070 con 0x7f7654152990 2024-03-28T23:31:51.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.292+0000 7f7658ab8640 1 -- 172.21.15.161:0/1739510724 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7650031a00 con 0x7f7654152990 2024-03-28T23:31:51.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.293+0000 7f765bd1e640 1 -- 172.21.15.161:0/1739510724 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7654152990 msgr2=0x7f76540c52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:51.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.293+0000 7f765bd1e640 1 --2- 172.21.15.161:0/1739510724 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7654152990 0x7f76540c52f0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f76540cb990 tx=0x7f765002b880 comp rx=0 tx=0).stop 2024-03-28T23:31:51.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.293+0000 7f765bd1e640 1 -- 172.21.15.161:0/1739510724 shutdown_connections 2024-03-28T23:31:51.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.293+0000 7f765bd1e640 1 --2- 172.21.15.161:0/1739510724 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76540c8d80 0x7f76541559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.293+0000 7f765bd1e640 1 --2- 172.21.15.161:0/1739510724 >> v2:172.21.15.170:3300/0 conn(0x7f76540c8870 0x7f7654155480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.293+0000 7f765bd1e640 1 --2- 172.21.15.161:0/1739510724 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7654152990 0x7f76540c52f0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.293+0000 7f765bd1e640 1 -- 172.21.15.161:0/1739510724 >> 172.21.15.161:0/1739510724 conn(0x7f76540ca970 msgr2=0x7f76540cad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:51.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.293+0000 7f765bd1e640 1 -- 172.21.15.161:0/1739510724 shutdown_connections 2024-03-28T23:31:51.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.293+0000 7f765bd1e640 1 -- 172.21.15.161:0/1739510724 wait complete. 2024-03-28T23:31:51.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.294+0000 7f765bd1e640 1 Processor -- start 2024-03-28T23:31:51.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.294+0000 7f765bd1e640 1 -- start start 2024-03-28T23:31:51.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.294+0000 7f765bd1e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76540c8870 0x7f76541ed9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:51.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.294+0000 7f765bd1e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76540c8d80 0x7f76541ee220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:51.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.294+0000 7f765bd1e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7654152990 0x7f76541ee760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:51.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.294+0000 7f765bd1e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7654073760 con 0x7f7654152990 2024-03-28T23:31:51.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.294+0000 7f765bd1e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7654159620 con 0x7f76540c8d80 2024-03-28T23:31:51.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.294+0000 7f765bd1e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7654155f00 con 0x7f76540c8870 2024-03-28T23:31:51.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.295+0000 7f765a2bb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7654152990 0x7f76541ee760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:51.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.295+0000 7f765a2bb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7654152990 0x7f76541ee760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35012/0 (socket says 172.21.15.161:35012) 2024-03-28T23:31:51.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.295+0000 7f765a2bb640 1 -- 172.21.15.161:0/1103124334 learned_addr learned my addr 172.21.15.161:0/1103124334 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:51.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.295+0000 7f7659aba640 1 --2- 172.21.15.161:0/1103124334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76540c8870 0x7f76541ed9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:51.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.295+0000 7f76592b9640 1 --2- 172.21.15.161:0/1103124334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76540c8d80 0x7f76541ee220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:51.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.295+0000 7f765a2bb640 1 -- 172.21.15.161:0/1103124334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76540c8870 msgr2=0x7f76541ed9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:51.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.295+0000 7f765a2bb640 1 --2- 172.21.15.161:0/1103124334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76540c8870 0x7f76541ed9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.295+0000 7f765a2bb640 1 -- 172.21.15.161:0/1103124334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76540c8d80 msgr2=0x7f76541ee220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:51.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.295+0000 7f765a2bb640 1 --2- 172.21.15.161:0/1103124334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76540c8d80 0x7f76541ee220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.295+0000 7f765a2bb640 1 -- 172.21.15.161:0/1103124334 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76541ef140 con 0x7f7654152990 2024-03-28T23:31:51.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.295+0000 7f76592b9640 1 --2- 172.21.15.161:0/1103124334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76540c8d80 0x7f76541ee220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:51.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.295+0000 7f7659aba640 1 --2- 172.21.15.161:0/1103124334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76540c8870 0x7f76541ed9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:51.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.295+0000 7f765a2bb640 1 --2- 172.21.15.161:0/1103124334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7654152990 0x7f76541ee760 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f764c007e40 tx=0x7f764c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:51.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.295+0000 7f764a7fc640 1 -- 172.21.15.161:0/1103124334 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764c014620 con 0x7f7654152990 2024-03-28T23:31:51.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.295+0000 7f764a7fc640 1 -- 172.21.15.161:0/1103124334 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f764c019070 con 0x7f7654152990 2024-03-28T23:31:51.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.295+0000 7f765bd1e640 1 -- 172.21.15.161:0/1103124334 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76540be4a0 con 0x7f7654152990 2024-03-28T23:31:51.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.295+0000 7f764a7fc640 1 -- 172.21.15.161:0/1103124334 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764c014a20 con 0x7f7654152990 2024-03-28T23:31:51.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.295+0000 7f765bd1e640 1 -- 172.21.15.161:0/1103124334 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76540b0110 con 0x7f7654152990 2024-03-28T23:31:51.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.296+0000 7f765bd1e640 1 -- 172.21.15.161:0/1103124334 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f765404f4b0 con 0x7f7654152990 2024-03-28T23:31:51.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.297+0000 7f764a7fc640 1 -- 172.21.15.161:0/1103124334 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f764c014bc0 con 0x7f7654152990 2024-03-28T23:31:51.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.297+0000 7f764a7fc640 1 --2- 172.21.15.161:0/1103124334 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f763806b500 0x7f763806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:51.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.297+0000 7f7659aba640 1 --2- 172.21.15.161:0/1103124334 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f763806b500 0x7f763806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:51.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.297+0000 7f764a7fc640 1 -- 172.21.15.161:0/1103124334 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f764c08abd0 con 0x7f7654152990 2024-03-28T23:31:51.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.298+0000 7f7659aba640 1 --2- 172.21.15.161:0/1103124334 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f763806b500 0x7f763806d9c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7650036000 tx=0x7f765003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:51.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.305+0000 7f764a7fc640 1 -- 172.21.15.161:0/1103124334 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f764c059950 con 0x7f7654152990 2024-03-28T23:31:51.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.484+0000 7f765bd1e640 1 -- 172.21.15.161:0/1103124334 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f76540ae060 con 0x7f7654152990 2024-03-28T23:31:51.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.484+0000 7f764a7fc640 1 -- 172.21.15.161:0/1103124334 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v51) ==== 93+0+10 (secure 0 0 0) 0x7f764c059250 con 0x7f7654152990 2024-03-28T23:31:51.486 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:31:51.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.486+0000 7f765bd1e640 1 -- 172.21.15.161:0/1103124334 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f763806b500 msgr2=0x7f763806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:51.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.486+0000 7f765bd1e640 1 --2- 172.21.15.161:0/1103124334 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f763806b500 0x7f763806d9c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7650036000 tx=0x7f765003d040 comp rx=0 tx=0).stop 2024-03-28T23:31:51.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.486+0000 7f765bd1e640 1 -- 172.21.15.161:0/1103124334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7654152990 msgr2=0x7f76541ee760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:51.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.486+0000 7f765bd1e640 1 --2- 172.21.15.161:0/1103124334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7654152990 0x7f76541ee760 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f764c007e40 tx=0x7f764c00e4b0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.486+0000 7f765bd1e640 1 -- 172.21.15.161:0/1103124334 shutdown_connections 2024-03-28T23:31:51.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.486+0000 7f765bd1e640 1 --2- 172.21.15.161:0/1103124334 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f763806b500 0x7f763806d9c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.486+0000 7f765bd1e640 1 --2- 172.21.15.161:0/1103124334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7654152990 0x7f76541ee760 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.486+0000 7f765bd1e640 1 --2- 172.21.15.161:0/1103124334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76540c8d80 0x7f76541ee220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.486+0000 7f765bd1e640 1 --2- 172.21.15.161:0/1103124334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76540c8870 0x7f76541ed9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.486+0000 7f765bd1e640 1 -- 172.21.15.161:0/1103124334 >> 172.21.15.161:0/1103124334 conn(0x7f76540ca970 msgr2=0x7f765406c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:51.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.486+0000 7f765bd1e640 1 -- 172.21.15.161:0/1103124334 shutdown_connections 2024-03-28T23:31:51.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.486+0000 7f765bd1e640 1 -- 172.21.15.161:0/1103124334 wait complete. 2024-03-28T23:31:51.502 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:31:51.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.606+0000 7f6fed249640 1 Processor -- start 2024-03-28T23:31:51.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.606+0000 7f6fed249640 1 -- start start 2024-03-28T23:31:51.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.607+0000 7f6fed249640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fe8156080 0x7f6fe80c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:51.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.607+0000 7f6fed249640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6fe806ff90 0x7f6fe8156b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:51.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.607+0000 7f6fed249640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fe8072790 con 0x7f6fe8156460 2024-03-28T23:31:51.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.607+0000 7f6fed249640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6fe80c67b0 con 0x7f6fe806ff90 2024-03-28T23:31:51.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.607+0000 7f6fed249640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6fe80a7ea0 con 0x7f6fe8156080 2024-03-28T23:31:51.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.607+0000 7f6fe659c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6fe806ff90 0x7f6fe8156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:51.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.607+0000 7f6fe759e640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6fe8156460 0x7f6fe81570c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37762/0 (socket says 172.21.15.161:37762) 2024-03-28T23:31:51.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.607+0000 7f6fe759e640 1 -- 172.21.15.161:0/307573613 learned_addr learned my addr 172.21.15.161:0/307573613 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:51.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.607+0000 7f6fe6d9d640 1 --2- 172.21.15.161:0/307573613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fe8156080 0x7f6fe80c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:51.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.607+0000 7f6fe659c640 1 -- 172.21.15.161:0/307573613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fe8156080 msgr2=0x7f6fe80c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:51.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.607+0000 7f6fe659c640 1 --2- 172.21.15.161:0/307573613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fe8156080 0x7f6fe80c5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.607+0000 7f6fe659c640 1 -- 172.21.15.161:0/307573613 >> v1:172.21.15.161:6789/0 conn(0x7f6fe8156460 legacy=0x7f6fe81570c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:51.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.607+0000 7f6fe659c640 1 -- 172.21.15.161:0/307573613 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe80a8180 con 0x7f6fe806ff90 2024-03-28T23:31:51.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.608+0000 7f6fe659c640 1 --2- 172.21.15.161:0/307573613 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fe806ff90 0x7f6fe8156b80 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc0097a0 tx=0x7f6fdc02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc440fe55fd26adb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:51.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.608+0000 7f6fe5d9b640 1 -- 172.21.15.161:0/307573613 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc038070 con 0x7f6fe806ff90 2024-03-28T23:31:51.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.608+0000 7f6fe5d9b640 1 -- 172.21.15.161:0/307573613 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fdc031700 con 0x7f6fe806ff90 2024-03-28T23:31:51.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.608+0000 7f6fe5d9b640 1 -- 172.21.15.161:0/307573613 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc031ad0 con 0x7f6fe806ff90 2024-03-28T23:31:51.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.609+0000 7f6fed249640 1 -- 172.21.15.161:0/307573613 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fe806ff90 msgr2=0x7f6fe8156b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:51.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.609+0000 7f6fed249640 1 --2- 172.21.15.161:0/307573613 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fe806ff90 0x7f6fe8156b80 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc0097a0 tx=0x7f6fdc02b7a0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.609+0000 7f6fed249640 1 -- 172.21.15.161:0/307573613 shutdown_connections 2024-03-28T23:31:51.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.609+0000 7f6fed249640 1 --2- 172.21.15.161:0/307573613 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fe806ff90 0x7f6fe8156b80 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.609+0000 7f6fed249640 1 --2- 172.21.15.161:0/307573613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fe8156080 0x7f6fe80c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.609+0000 7f6fed249640 1 -- 172.21.15.161:0/307573613 >> 172.21.15.161:0/307573613 conn(0x7f6fe806eba0 msgr2=0x7f6fe806b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:51.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.609+0000 7f6fed249640 1 -- 172.21.15.161:0/307573613 shutdown_connections 2024-03-28T23:31:51.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.609+0000 7f6fed249640 1 -- 172.21.15.161:0/307573613 wait complete. 2024-03-28T23:31:51.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.610+0000 7f6fed249640 1 Processor -- start 2024-03-28T23:31:51.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.610+0000 7f6fed249640 1 -- start start 2024-03-28T23:31:51.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.611+0000 7f6fed249640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fe806ff90 0x7f6fe80c1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:51.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.611+0000 7f6fed249640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fe8156080 0x7f6fe8147a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:51.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.611+0000 7f6fed249640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fe8156460 0x7f6fe8147f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:51.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.611+0000 7f6fed249640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe8070620 con 0x7f6fe806ff90 2024-03-28T23:31:51.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.611+0000 7f6fed249640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe80704a0 con 0x7f6fe8156080 2024-03-28T23:31:51.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.611+0000 7f6fed249640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6fe80707a0 con 0x7f6fe8156460 2024-03-28T23:31:51.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.611+0000 7f6fe6d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fe806ff90 0x7f6fe80c1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:51.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.611+0000 7f6fe6d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fe806ff90 0x7f6fe80c1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35024/0 (socket says 172.21.15.161:35024) 2024-03-28T23:31:51.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.611+0000 7f6fe6d9d640 1 -- 172.21.15.161:0/941338334 learned_addr learned my addr 172.21.15.161:0/941338334 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:51.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.611+0000 7f6fe759e640 1 --2- 172.21.15.161:0/941338334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fe8156460 0x7f6fe8147f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:51.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.611+0000 7f6fe659c640 1 --2- 172.21.15.161:0/941338334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fe8156080 0x7f6fe8147a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:51.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.611+0000 7f6fe6d9d640 1 -- 172.21.15.161:0/941338334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fe8156460 msgr2=0x7f6fe8147f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:51.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.611+0000 7f6fe6d9d640 1 --2- 172.21.15.161:0/941338334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fe8156460 0x7f6fe8147f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.611+0000 7f6fe6d9d640 1 -- 172.21.15.161:0/941338334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fe8156080 msgr2=0x7f6fe8147a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:51.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.611+0000 7f6fe6d9d640 1 --2- 172.21.15.161:0/941338334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fe8156080 0x7f6fe8147a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.611+0000 7f6fe6d9d640 1 -- 172.21.15.161:0/941338334 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe81ee430 con 0x7f6fe806ff90 2024-03-28T23:31:51.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.611+0000 7f6fe659c640 1 --2- 172.21.15.161:0/941338334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fe8156080 0x7f6fe8147a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:51.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.611+0000 7f6fe759e640 1 --2- 172.21.15.161:0/941338334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fe8156460 0x7f6fe8147f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:51.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.612+0000 7f6fe6d9d640 1 --2- 172.21.15.161:0/941338334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fe806ff90 0x7f6fe80c1f80 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f6fd800c7f0 tx=0x7f6fd800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:51.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.612+0000 7f6fcf7fe640 1 -- 172.21.15.161:0/941338334 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd8015070 con 0x7f6fe806ff90 2024-03-28T23:31:51.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.612+0000 7f6fcf7fe640 1 -- 172.21.15.161:0/941338334 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fd8004240 con 0x7f6fe806ff90 2024-03-28T23:31:51.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.612+0000 7f6fcf7fe640 1 -- 172.21.15.161:0/941338334 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd8007620 con 0x7f6fe806ff90 2024-03-28T23:31:51.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.612+0000 7f6fed249640 1 -- 172.21.15.161:0/941338334 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fe81ee710 con 0x7f6fe806ff90 2024-03-28T23:31:51.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.612+0000 7f6fed249640 1 -- 172.21.15.161:0/941338334 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fe81eecd0 con 0x7f6fe806ff90 2024-03-28T23:31:51.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.613+0000 7f6fed249640 1 -- 172.21.15.161:0/941338334 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fe804f4b0 con 0x7f6fe806ff90 2024-03-28T23:31:51.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.613+0000 7f6fcf7fe640 1 -- 172.21.15.161:0/941338334 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6fd80079d0 con 0x7f6fe806ff90 2024-03-28T23:31:51.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.614+0000 7f6fcf7fe640 1 --2- 172.21.15.161:0/941338334 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6fd406b4d0 0x7f6fd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:51.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.614+0000 7f6fcf7fe640 1 -- 172.21.15.161:0/941338334 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f6fd808a400 con 0x7f6fe806ff90 2024-03-28T23:31:51.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.614+0000 7f6fe659c640 1 --2- 172.21.15.161:0/941338334 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6fd406b4d0 0x7f6fd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:51.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.614+0000 7f6fe659c640 1 --2- 172.21.15.161:0/941338334 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6fd406b4d0 0x7f6fd406d990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc035000 tx=0x7f6fdc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:51.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.618+0000 7f6fcf7fe640 1 -- 172.21.15.161:0/941338334 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6fd8059180 con 0x7f6fe806ff90 2024-03-28T23:31:51.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.819+0000 7f6fed249640 1 -- 172.21.15.161:0/941338334 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6fe806fd30 con 0x7f6fe806ff90 2024-03-28T23:31:51.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.820+0000 7f6fcf7fe640 1 -- 172.21.15.161:0/941338334 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v51) ==== 104+0+11 (secure 0 0 0) 0x7f6fd8058a80 con 0x7f6fe806ff90 2024-03-28T23:31:51.822 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:31:51.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.822+0000 7f6fed249640 1 -- 172.21.15.161:0/941338334 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6fd406b4d0 msgr2=0x7f6fd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:51.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.822+0000 7f6fed249640 1 --2- 172.21.15.161:0/941338334 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6fd406b4d0 0x7f6fd406d990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc035000 tx=0x7f6fdc008030 comp rx=0 tx=0).stop 2024-03-28T23:31:51.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.822+0000 7f6fed249640 1 -- 172.21.15.161:0/941338334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fe806ff90 msgr2=0x7f6fe80c1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:51.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.822+0000 7f6fed249640 1 --2- 172.21.15.161:0/941338334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fe806ff90 0x7f6fe80c1f80 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f6fd800c7f0 tx=0x7f6fd800ccb0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.822+0000 7f6fed249640 1 -- 172.21.15.161:0/941338334 shutdown_connections 2024-03-28T23:31:51.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.822+0000 7f6fed249640 1 --2- 172.21.15.161:0/941338334 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6fd406b4d0 0x7f6fd406d990 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.822+0000 7f6fed249640 1 --2- 172.21.15.161:0/941338334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fe8156460 0x7f6fe8147f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.822+0000 7f6fed249640 1 --2- 172.21.15.161:0/941338334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fe8156080 0x7f6fe8147a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.822+0000 7f6fed249640 1 --2- 172.21.15.161:0/941338334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fe806ff90 0x7f6fe80c1f80 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.822+0000 7f6fed249640 1 -- 172.21.15.161:0/941338334 >> 172.21.15.161:0/941338334 conn(0x7f6fe806eba0 msgr2=0x7f6fe8156a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:51.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.822+0000 7f6fed249640 1 -- 172.21.15.161:0/941338334 shutdown_connections 2024-03-28T23:31:51.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.822+0000 7f6fed249640 1 -- 172.21.15.161:0/941338334 wait complete. 2024-03-28T23:31:51.834 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:31:51.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.950+0000 7fcfa85c7640 1 Processor -- start 2024-03-28T23:31:51.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.950+0000 7fcfa85c7640 1 -- start start 2024-03-28T23:31:51.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.951+0000 7fcfa85c7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfa00c5dd0 0x7fcfa00a9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:51.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.951+0000 7fcfa85c7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcfa006ea30 0x7fcfa00c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:51.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.951+0000 7fcfa85c7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfa00a6290 con 0x7fcfa00c1290 2024-03-28T23:31:51.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.951+0000 7fcfa85c7640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fcfa00c61b0 con 0x7fcfa006ea30 2024-03-28T23:31:51.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.951+0000 7fcfa85c7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fcfa01532a0 con 0x7fcfa00c5dd0 2024-03-28T23:31:51.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.951+0000 7fcfa5b62640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcfa006ea30 0x7fcfa00c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:51.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.951+0000 7fcfa5b62640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcfa006ea30 0x7fcfa00c1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35030/0 (socket says 172.21.15.161:35030) 2024-03-28T23:31:51.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.951+0000 7fcfa5b62640 1 -- 172.21.15.161:0/2231167416 learned_addr learned my addr 172.21.15.161:0/2231167416 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:51.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.951+0000 7fcfa6363640 1 --2- 172.21.15.161:0/2231167416 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfa00c5dd0 0x7fcfa00a9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:51.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.951+0000 7fcfa5b62640 1 -- 172.21.15.161:0/2231167416 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfa00c5dd0 msgr2=0x7fcfa00a9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:51.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.951+0000 7fcfa5b62640 1 --2- 172.21.15.161:0/2231167416 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfa00c5dd0 0x7fcfa00a9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.951+0000 7fcfa5b62640 1 -- 172.21.15.161:0/2231167416 >> v1:172.21.15.170:6789/0 conn(0x7fcfa00c1290 legacy=0x7fcfa0156540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:51.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.951+0000 7fcfa5b62640 1 -- 172.21.15.161:0/2231167416 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa00abaa0 con 0x7fcfa006ea30 2024-03-28T23:31:51.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.952+0000 7fcfa5b62640 1 --2- 172.21.15.161:0/2231167416 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfa006ea30 0x7fcfa00c1eb0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fcf90009870 tx=0x7fcf9002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7382380c06541632 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:51.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.953+0000 7fcfa5361640 1 -- 172.21.15.161:0/2231167416 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf90034070 con 0x7fcfa006ea30 2024-03-28T23:31:51.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.953+0000 7fcfa5361640 1 -- 172.21.15.161:0/2231167416 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf90030840 con 0x7fcfa006ea30 2024-03-28T23:31:51.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.953+0000 7fcfa5361640 1 -- 172.21.15.161:0/2231167416 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf90030c10 con 0x7fcfa006ea30 2024-03-28T23:31:51.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.953+0000 7fcfa85c7640 1 -- 172.21.15.161:0/2231167416 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfa006ea30 msgr2=0x7fcfa00c1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:51.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.953+0000 7fcfa85c7640 1 --2- 172.21.15.161:0/2231167416 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfa006ea30 0x7fcfa00c1eb0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fcf90009870 tx=0x7fcf9002b980 comp rx=0 tx=0).stop 2024-03-28T23:31:51.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.953+0000 7fcfa85c7640 1 -- 172.21.15.161:0/2231167416 shutdown_connections 2024-03-28T23:31:51.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.953+0000 7fcfa85c7640 1 --2- 172.21.15.161:0/2231167416 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfa006ea30 0x7fcfa00c1eb0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.953+0000 7fcfa85c7640 1 --2- 172.21.15.161:0/2231167416 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfa00c5dd0 0x7fcfa00a9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.953+0000 7fcfa85c7640 1 -- 172.21.15.161:0/2231167416 >> 172.21.15.161:0/2231167416 conn(0x7fcfa00c3390 msgr2=0x7fcfa00c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:51.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.953+0000 7fcfa85c7640 1 -- 172.21.15.161:0/2231167416 shutdown_connections 2024-03-28T23:31:51.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.953+0000 7fcfa85c7640 1 -- 172.21.15.161:0/2231167416 wait complete. 2024-03-28T23:31:51.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.955+0000 7fcfa85c7640 1 Processor -- start 2024-03-28T23:31:51.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.955+0000 7fcfa85c7640 1 -- start start 2024-03-28T23:31:51.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.955+0000 7fcfa85c7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfa006ea30 0x7fcfa01551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:51.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.955+0000 7fcfa85c7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfa00c1290 0x7fcfa00bff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:51.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.955+0000 7fcfa85c7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfa00c5dd0 0x7fcfa00705d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:51.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.955+0000 7fcfa85c7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa00ac340 con 0x7fcfa006ea30 2024-03-28T23:31:51.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.956+0000 7fcfa85c7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa00ac060 con 0x7fcfa00c5dd0 2024-03-28T23:31:51.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.956+0000 7fcfa85c7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fcfa0152f50 con 0x7fcfa00c1290 2024-03-28T23:31:51.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.956+0000 7fcfa6363640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfa006ea30 0x7fcfa01551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:51.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.956+0000 7fcfa6363640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfa006ea30 0x7fcfa01551d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35034/0 (socket says 172.21.15.161:35034) 2024-03-28T23:31:51.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.956+0000 7fcfa6363640 1 -- 172.21.15.161:0/3659543642 learned_addr learned my addr 172.21.15.161:0/3659543642 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:51.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.956+0000 7fcfa5b62640 1 --2- 172.21.15.161:0/3659543642 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfa00c1290 0x7fcfa00bff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:51.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.956+0000 7fcfa6b64640 1 --2- 172.21.15.161:0/3659543642 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfa00c5dd0 0x7fcfa00705d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:51.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.956+0000 7fcfa6363640 1 -- 172.21.15.161:0/3659543642 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfa00c1290 msgr2=0x7fcfa00bff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:51.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.956+0000 7fcfa6363640 1 --2- 172.21.15.161:0/3659543642 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfa00c1290 0x7fcfa00bff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.956+0000 7fcfa6363640 1 -- 172.21.15.161:0/3659543642 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfa00c5dd0 msgr2=0x7fcfa00705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:51.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.956+0000 7fcfa6363640 1 --2- 172.21.15.161:0/3659543642 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfa00c5dd0 0x7fcfa00705d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:51.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.956+0000 7fcfa6363640 1 -- 172.21.15.161:0/3659543642 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa01eb0a0 con 0x7fcfa006ea30 2024-03-28T23:31:51.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.956+0000 7fcfa6363640 1 --2- 172.21.15.161:0/3659543642 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfa006ea30 0x7fcfa01551d0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800d830 tx=0x7fcf9800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:51.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.956+0000 7fcf96ffd640 1 -- 172.21.15.161:0/3659543642 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9800b6c0 con 0x7fcfa006ea30 2024-03-28T23:31:51.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.956+0000 7fcf96ffd640 1 -- 172.21.15.161:0/3659543642 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf98019070 con 0x7fcfa006ea30 2024-03-28T23:31:51.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.956+0000 7fcfa85c7640 1 -- 172.21.15.161:0/3659543642 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa0156ab0 con 0x7fcfa006ea30 2024-03-28T23:31:51.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.956+0000 7fcf96ffd640 1 -- 172.21.15.161:0/3659543642 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9800ba80 con 0x7fcfa006ea30 2024-03-28T23:31:51.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.956+0000 7fcfa85c7640 1 -- 172.21.15.161:0/3659543642 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfa0157070 con 0x7fcfa006ea30 2024-03-28T23:31:51.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.957+0000 7fcfa85c7640 1 -- 172.21.15.161:0/3659543642 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfa004f4b0 con 0x7fcfa006ea30 2024-03-28T23:31:51.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.958+0000 7fcf96ffd640 1 -- 172.21.15.161:0/3659543642 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcf9801e460 con 0x7fcfa006ea30 2024-03-28T23:31:51.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.958+0000 7fcf96ffd640 1 --2- 172.21.15.161:0/3659543642 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcf8406b4d0 0x7fcf8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:51.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.958+0000 7fcf96ffd640 1 -- 172.21.15.161:0/3659543642 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7fcf9808a9b0 con 0x7fcfa006ea30 2024-03-28T23:31:51.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.958+0000 7fcfa5b62640 1 --2- 172.21.15.161:0/3659543642 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcf8406b4d0 0x7fcf8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:51.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.959+0000 7fcfa5b62640 1 --2- 172.21.15.161:0/3659543642 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcf8406b4d0 0x7fcf8406d990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcf90036000 tx=0x7fcf9003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:51.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:51.963+0000 7fcf96ffd640 1 -- 172.21.15.161:0/3659543642 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcf98059730 con 0x7fcfa006ea30 2024-03-28T23:31:52.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.139+0000 7fcfa85c7640 1 -- 172.21.15.161:0/3659543642 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcfa00c6360 con 0x7fcfa006ea30 2024-03-28T23:31:52.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.140+0000 7fcf96ffd640 1 -- 172.21.15.161:0/3659543642 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v51) ==== 100+0+11 (secure 0 0 0) 0x7fcf98059030 con 0x7fcfa006ea30 2024-03-28T23:31:52.141 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:31:52.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.143+0000 7fcfa85c7640 1 -- 172.21.15.161:0/3659543642 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcf8406b4d0 msgr2=0x7fcf8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:52.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.143+0000 7fcfa85c7640 1 --2- 172.21.15.161:0/3659543642 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcf8406b4d0 0x7fcf8406d990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcf90036000 tx=0x7fcf9003b040 comp rx=0 tx=0).stop 2024-03-28T23:31:52.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.143+0000 7fcfa85c7640 1 -- 172.21.15.161:0/3659543642 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfa006ea30 msgr2=0x7fcfa01551d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:52.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.143+0000 7fcfa85c7640 1 --2- 172.21.15.161:0/3659543642 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfa006ea30 0x7fcfa01551d0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800d830 tx=0x7fcf9800dcf0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.143+0000 7fcfa85c7640 1 -- 172.21.15.161:0/3659543642 shutdown_connections 2024-03-28T23:31:52.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.143+0000 7fcfa85c7640 1 --2- 172.21.15.161:0/3659543642 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcf8406b4d0 0x7fcf8406d990 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.143+0000 7fcfa85c7640 1 --2- 172.21.15.161:0/3659543642 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfa00c5dd0 0x7fcfa00705d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.143+0000 7fcfa85c7640 1 --2- 172.21.15.161:0/3659543642 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfa00c1290 0x7fcfa00bff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.143+0000 7fcfa85c7640 1 --2- 172.21.15.161:0/3659543642 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfa006ea30 0x7fcfa01551d0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.143+0000 7fcfa85c7640 1 -- 172.21.15.161:0/3659543642 >> 172.21.15.161:0/3659543642 conn(0x7fcfa00c3390 msgr2=0x7fcfa00a9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:52.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.143+0000 7fcfa85c7640 1 -- 172.21.15.161:0/3659543642 shutdown_connections 2024-03-28T23:31:52.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.143+0000 7fcfa85c7640 1 -- 172.21.15.161:0/3659543642 wait complete. 2024-03-28T23:31:52.154 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T23:31:52.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.259+0000 7f7165d81640 1 Processor -- start 2024-03-28T23:31:52.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.259+0000 7f7165d81640 1 -- start start 2024-03-28T23:31:52.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.260+0000 7f7165d81640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f71601592e0 0x7f71600c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:52.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.260+0000 7f7165d81640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7160159730 con 0x7f71600caf60 2024-03-28T23:31:52.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.260+0000 7f7165d81640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71600cb4e0 con 0x7f716014f260 2024-03-28T23:31:52.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.260+0000 7f7165d81640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f71600c9080 con 0x7f71601592e0 2024-03-28T23:31:52.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.260+0000 7f715ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f71601592e0 0x7f71600c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:52.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.260+0000 7f715effd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f71600caf60 0x7f71600c8970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37774/0 (socket says 172.21.15.161:37774) 2024-03-28T23:31:52.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.260+0000 7f715effd640 1 -- 172.21.15.161:0/1078063522 learned_addr learned my addr 172.21.15.161:0/1078063522 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:52.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.261+0000 7f715ffff640 1 -- 172.21.15.161:0/1078063522 >> v1:172.21.15.170:6789/0 conn(0x7f716014f260 legacy=0x7f7160155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:52.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.261+0000 7f715ffff640 1 -- 172.21.15.161:0/1078063522 >> v1:172.21.15.161:6789/0 conn(0x7f71600caf60 legacy=0x7f71600c8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:52.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.261+0000 7f715ffff640 1 -- 172.21.15.161:0/1078063522 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71600cb7c0 con 0x7f71601592e0 2024-03-28T23:31:52.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.261+0000 7f715e7fc640 1 -- 172.21.15.161:0/1078063522 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 700055075 0 0) 0x7f7160159730 con 0x7f71600caf60 2024-03-28T23:31:52.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.261+0000 7f715ffff640 1 --2- 172.21.15.161:0/1078063522 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71601592e0 0x7f71600c53f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f7158009d90 tx=0x7f715802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f305133bdb9021 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:52.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.261+0000 7f715e7fc640 1 -- 172.21.15.161:0/1078063522 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7158039070 con 0x7f71601592e0 2024-03-28T23:31:52.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.261+0000 7f715e7fc640 1 -- 172.21.15.161:0/1078063522 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f715802cc60 con 0x7f71601592e0 2024-03-28T23:31:52.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.262+0000 7f715e7fc640 1 -- 172.21.15.161:0/1078063522 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7158034660 con 0x7f71601592e0 2024-03-28T23:31:52.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.262+0000 7f7165d81640 1 -- 172.21.15.161:0/1078063522 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71601592e0 msgr2=0x7f71600c53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:52.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.262+0000 7f7165d81640 1 --2- 172.21.15.161:0/1078063522 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71601592e0 0x7f71600c53f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f7158009d90 tx=0x7f715802beb0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.262+0000 7f7165d81640 1 -- 172.21.15.161:0/1078063522 shutdown_connections 2024-03-28T23:31:52.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.262+0000 7f7165d81640 1 --2- 172.21.15.161:0/1078063522 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71601592e0 0x7f71600c53f0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.262+0000 7f7165d81640 1 -- 172.21.15.161:0/1078063522 >> 172.21.15.161:0/1078063522 conn(0x7f7160072ca0 msgr2=0x7f71600730b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:52.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.262+0000 7f7165d81640 1 -- 172.21.15.161:0/1078063522 shutdown_connections 2024-03-28T23:31:52.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.262+0000 7f7165d81640 1 -- 172.21.15.161:0/1078063522 wait complete. 2024-03-28T23:31:52.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.263+0000 7f7165d81640 1 Processor -- start 2024-03-28T23:31:52.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.263+0000 7f7165d81640 1 -- start start 2024-03-28T23:31:52.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.264+0000 7f7165d81640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f71600caf60 0x7f716014d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:52.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.264+0000 7f7165d81640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f716014f260 0x7f71600be7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:52.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.264+0000 7f7165d81640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71601592e0 0x7f716014adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:52.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.264+0000 7f7165d81640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7160158ae0 con 0x7f71601592e0 2024-03-28T23:31:52.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.264+0000 7f7165d81640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7160158e50 con 0x7f71600caf60 2024-03-28T23:31:52.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.264+0000 7f7165d81640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f716014f640 con 0x7f716014f260 2024-03-28T23:31:52.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.264+0000 7f715ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71601592e0 0x7f716014adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:52.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.264+0000 7f715ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71601592e0 0x7f716014adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35054/0 (socket says 172.21.15.161:35054) 2024-03-28T23:31:52.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.264+0000 7f715ffff640 1 -- 172.21.15.161:0/2527332843 learned_addr learned my addr 172.21.15.161:0/2527332843 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:52.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.264+0000 7f715effd640 1 --2- 172.21.15.161:0/2527332843 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f716014f260 0x7f71600be7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:52.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.264+0000 7f715ffff640 1 -- 172.21.15.161:0/2527332843 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f716014f260 msgr2=0x7f71600be7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:52.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.264+0000 7f715ffff640 1 --2- 172.21.15.161:0/2527332843 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f716014f260 0x7f71600be7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.264+0000 7f715f7fe640 1 --2- 172.21.15.161:0/2527332843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f71600caf60 0x7f716014d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:52.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.264+0000 7f715ffff640 1 -- 172.21.15.161:0/2527332843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f71600caf60 msgr2=0x7f716014d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:52.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.264+0000 7f715ffff640 1 --2- 172.21.15.161:0/2527332843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f71600caf60 0x7f716014d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.264+0000 7f715ffff640 1 -- 172.21.15.161:0/2527332843 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71601edc50 con 0x7f71601592e0 2024-03-28T23:31:52.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.265+0000 7f715ffff640 1 --2- 172.21.15.161:0/2527332843 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71601592e0 0x7f716014adb0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f7158037040 tx=0x7f7158004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:52.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.265+0000 7f7147fff640 1 -- 172.21.15.161:0/2527332843 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7158039050 con 0x7f71601592e0 2024-03-28T23:31:52.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.265+0000 7f7147fff640 1 -- 172.21.15.161:0/2527332843 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f715803a040 con 0x7f71601592e0 2024-03-28T23:31:52.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.265+0000 7f7165d81640 1 -- 172.21.15.161:0/2527332843 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71601edef0 con 0x7f71601592e0 2024-03-28T23:31:52.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.265+0000 7f7147fff640 1 -- 172.21.15.161:0/2527332843 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715803e610 con 0x7f71601592e0 2024-03-28T23:31:52.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.265+0000 7f7165d81640 1 -- 172.21.15.161:0/2527332843 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71601ee4b0 con 0x7f71601592e0 2024-03-28T23:31:52.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.266+0000 7f7165d81640 1 -- 172.21.15.161:0/2527332843 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f716004f550 con 0x7f71601592e0 2024-03-28T23:31:52.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.266+0000 7f7147fff640 1 -- 172.21.15.161:0/2527332843 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7158047020 con 0x7f71601592e0 2024-03-28T23:31:52.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.267+0000 7f7147fff640 1 --2- 172.21.15.161:0/2527332843 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f714c06b4d0 0x7f714c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:52.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.267+0000 7f715f7fe640 1 --2- 172.21.15.161:0/2527332843 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f714c06b4d0 0x7f714c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:52.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.267+0000 7f7147fff640 1 -- 172.21.15.161:0/2527332843 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f71580ac6e0 con 0x7f71601592e0 2024-03-28T23:31:52.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.267+0000 7f715f7fe640 1 --2- 172.21.15.161:0/2527332843 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f714c06b4d0 0x7f714c06d990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7150000f60 tx=0x7f7150009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:52.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.271+0000 7f7147fff640 1 -- 172.21.15.161:0/2527332843 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f715804e020 con 0x7f71601592e0 2024-03-28T23:31:52.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.451+0000 7f7165d81640 1 -- 172.21.15.161:0/2527332843 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f716006cfe0 con 0x7f71601592e0 2024-03-28T23:31:52.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.451+0000 7f7147fff640 1 -- 172.21.15.161:0/2527332843 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v51) ==== 84+0+207 (secure 0 0 0) 0x7f715807b520 con 0x7f71601592e0 2024-03-28T23:31:52.452 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:52.452 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2024-03-28T23:31:52.452 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:31:52.452 INFO:teuthology.orchestra.run.smithi161.stdout: "poolnum": 1, 2024-03-28T23:31:52.453 INFO:teuthology.orchestra.run.smithi161.stdout: "poolname": ".mgr" 2024-03-28T23:31:52.453 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:31:52.453 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:31:52.453 INFO:teuthology.orchestra.run.smithi161.stdout: "poolnum": 7, 2024-03-28T23:31:52.453 INFO:teuthology.orchestra.run.smithi161.stdout: "poolname": "cephfs_metadata" 2024-03-28T23:31:52.453 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:31:52.453 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:31:52.453 INFO:teuthology.orchestra.run.smithi161.stdout: "poolnum": 8, 2024-03-28T23:31:52.454 INFO:teuthology.orchestra.run.smithi161.stdout: "poolname": "cephfs_data" 2024-03-28T23:31:52.454 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:31:52.454 INFO:teuthology.orchestra.run.smithi161.stdout:] 2024-03-28T23:31:52.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.453+0000 7f7165d81640 1 -- 172.21.15.161:0/2527332843 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f714c06b4d0 msgr2=0x7f714c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:52.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.453+0000 7f7165d81640 1 --2- 172.21.15.161:0/2527332843 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f714c06b4d0 0x7f714c06d990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7150000f60 tx=0x7f7150009040 comp rx=0 tx=0).stop 2024-03-28T23:31:52.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.453+0000 7f7165d81640 1 -- 172.21.15.161:0/2527332843 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71601592e0 msgr2=0x7f716014adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:52.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.453+0000 7f7165d81640 1 --2- 172.21.15.161:0/2527332843 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71601592e0 0x7f716014adb0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f7158037040 tx=0x7f7158004a40 comp rx=0 tx=0).stop 2024-03-28T23:31:52.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.453+0000 7f7165d81640 1 -- 172.21.15.161:0/2527332843 shutdown_connections 2024-03-28T23:31:52.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.453+0000 7f7165d81640 1 --2- 172.21.15.161:0/2527332843 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f714c06b4d0 0x7f714c06d990 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.453+0000 7f7165d81640 1 --2- 172.21.15.161:0/2527332843 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71601592e0 0x7f716014adb0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.453+0000 7f7165d81640 1 --2- 172.21.15.161:0/2527332843 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f716014f260 0x7f71600be7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.453+0000 7f7165d81640 1 --2- 172.21.15.161:0/2527332843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f71600caf60 0x7f716014d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.453+0000 7f7165d81640 1 -- 172.21.15.161:0/2527332843 >> 172.21.15.161:0/2527332843 conn(0x7f7160072ca0 msgr2=0x7f71600c7480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:52.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.453+0000 7f7165d81640 1 -- 172.21.15.161:0/2527332843 shutdown_connections 2024-03-28T23:31:52.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.454+0000 7f7165d81640 1 -- 172.21.15.161:0/2527332843 wait complete. 2024-03-28T23:31:52.465 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs2' 2024-03-28T23:31:52.465 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_metadata --pg_num_min 64 2024-03-28T23:31:52.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.569+0000 7f47d18ed640 1 Processor -- start 2024-03-28T23:31:52.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.569+0000 7f47d18ed640 1 -- start start 2024-03-28T23:31:52.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.570+0000 7f47d18ed640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f47cc1527f0 0x7f47cc0c5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:52.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.570+0000 7f47d18ed640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47cc152bd0 con 0x7f47cc0be110 2024-03-28T23:31:52.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.570+0000 7f47d18ed640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47cc159220 con 0x7f47cc0c8980 2024-03-28T23:31:52.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.570+0000 7f47d18ed640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f47cc0cafb0 con 0x7f47cc1527f0 2024-03-28T23:31:52.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.570+0000 7f47cb7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f47cc0be110 0x7f47cc155890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37784/0 (socket says 172.21.15.161:37784) 2024-03-28T23:31:52.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.570+0000 7f47cb7fe640 1 -- 172.21.15.161:0/4149304965 learned_addr learned my addr 172.21.15.161:0/4149304965 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:52.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.570+0000 7f47caffd640 1 --2- 172.21.15.161:0/4149304965 >> v2:172.21.15.170:3300/0 conn(0x7f47cc1527f0 0x7f47cc0c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:52.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.570+0000 7f47c9ffb640 1 -- 172.21.15.161:0/4149304965 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3413680280 0 0) 0x7f47cc152bd0 con 0x7f47cc0be110 2024-03-28T23:31:52.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.570+0000 7f47c9ffb640 1 -- 172.21.15.161:0/4149304965 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47b8003520 con 0x7f47cc0be110 2024-03-28T23:31:52.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.570+0000 7f47c9ffb640 1 -- 172.21.15.161:0/4149304965 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1732978615 0 0) 0x7f47cc159220 con 0x7f47cc0c8980 2024-03-28T23:31:52.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.570+0000 7f47c9ffb640 1 -- 172.21.15.161:0/4149304965 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47cc152bd0 con 0x7f47cc0c8980 2024-03-28T23:31:52.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.570+0000 7f47c9ffb640 1 -- 172.21.15.161:0/4149304965 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4052548170 0 0) 0x7f47b8003520 con 0x7f47cc0be110 2024-03-28T23:31:52.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.570+0000 7f47c9ffb640 1 -- 172.21.15.161:0/4149304965 >> v2:172.21.15.170:3300/0 conn(0x7f47cc1527f0 msgr2=0x7f47cc0c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:52.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.570+0000 7f47c9ffb640 1 --2- 172.21.15.161:0/4149304965 >> v2:172.21.15.170:3300/0 conn(0x7f47cc1527f0 0x7f47cc0c5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.570+0000 7f47c9ffb640 1 -- 172.21.15.161:0/4149304965 >> v1:172.21.15.170:6789/0 conn(0x7f47cc0c8980 legacy=0x7f47cc155180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:52.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.570+0000 7f47c9ffb640 1 -- 172.21.15.161:0/4149304965 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47cc159500 con 0x7f47cc0be110 2024-03-28T23:31:52.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.570+0000 7f47c9ffb640 1 -- 172.21.15.161:0/4149304965 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f47bc001670 con 0x7f47cc0be110 2024-03-28T23:31:52.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.571+0000 7f47c9ffb640 1 -- 172.21.15.161:0/4149304965 >> v1:172.21.15.161:6789/0 conn(0x7f47cc0be110 legacy=0x7f47cc155890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:52.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.571+0000 7f47c9ffb640 1 --2- 172.21.15.161:0/4149304965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47b8003b40 0x7f47b8003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:52.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.571+0000 7f47c9ffb640 1 --2- 172.21.15.161:0/4149304965 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47b8004800 0x7f47b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:52.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.571+0000 7f47c9ffb640 1 --2- 172.21.15.161:0/4149304965 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47b8005230 0x7f47b800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:52.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.571+0000 7f47c9ffb640 1 -- 172.21.15.161:0/4149304965 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f47cc0cafb0 con 0x7f47b8005230 2024-03-28T23:31:52.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.571+0000 7f47c9ffb640 1 -- 172.21.15.161:0/4149304965 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f47b800c190 con 0x7f47b8003b40 2024-03-28T23:31:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.571+0000 7f47c9ffb640 1 -- 172.21.15.161:0/4149304965 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f47b800c390 con 0x7f47b8004800 2024-03-28T23:31:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.571+0000 7f47cb7fe640 1 --2- 172.21.15.161:0/4149304965 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47b8005230 0x7f47b800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.571+0000 7f47caffd640 1 --2- 172.21.15.161:0/4149304965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47b8003b40 0x7f47b8003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.571+0000 7f47ca7fc640 1 --2- 172.21.15.161:0/4149304965 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47b8004800 0x7f47b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.571+0000 7f47cb7fe640 1 -- 172.21.15.161:0/4149304965 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47b8004800 msgr2=0x7f47b8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.571+0000 7f47cb7fe640 1 --2- 172.21.15.161:0/4149304965 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47b8004800 0x7f47b8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.571+0000 7f47cb7fe640 1 -- 172.21.15.161:0/4149304965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47b8003b40 msgr2=0x7f47b8003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.571+0000 7f47cb7fe640 1 --2- 172.21.15.161:0/4149304965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47b8003b40 0x7f47b8003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.571+0000 7f47cb7fe640 1 -- 172.21.15.161:0/4149304965 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47b8003520 con 0x7f47b8005230 2024-03-28T23:31:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.572+0000 7f47cb7fe640 1 -- 172.21.15.161:0/4149304965 reap_dead start 2024-03-28T23:31:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.572+0000 7f47cb7fe640 1 --2- 172.21.15.161:0/4149304965 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47b8005230 0x7f47b800b9c0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f47cc0be580 tx=0x7f47b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78391705f74a7d4c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.572+0000 7f47c9ffb640 1 -- 172.21.15.161:0/4149304965 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47bc007ca0 con 0x7f47b8005230 2024-03-28T23:31:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.572+0000 7f47c9ffb640 1 -- 172.21.15.161:0/4149304965 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47bc030070 con 0x7f47b8005230 2024-03-28T23:31:52.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.572+0000 7f47c9ffb640 1 -- 172.21.15.161:0/4149304965 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47bc005d30 con 0x7f47b8005230 2024-03-28T23:31:52.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.573+0000 7f47d18ed640 1 -- 172.21.15.161:0/4149304965 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47b8005230 msgr2=0x7f47b800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:52.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.573+0000 7f47d18ed640 1 --2- 172.21.15.161:0/4149304965 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47b8005230 0x7f47b800b9c0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f47cc0be580 tx=0x7f47b8005830 comp rx=0 tx=0).stop 2024-03-28T23:31:52.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.573+0000 7f47d18ed640 1 -- 172.21.15.161:0/4149304965 shutdown_connections 2024-03-28T23:31:52.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.573+0000 7f47d18ed640 1 --2- 172.21.15.161:0/4149304965 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47b8005230 0x7f47b800b9c0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.573+0000 7f47d18ed640 1 -- 172.21.15.161:0/4149304965 >> 172.21.15.161:0/4149304965 conn(0x7f47cc072ca0 msgr2=0x7f47cc0730b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:52.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.573+0000 7f47d18ed640 1 -- 172.21.15.161:0/4149304965 shutdown_connections 2024-03-28T23:31:52.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.573+0000 7f47d18ed640 1 -- 172.21.15.161:0/4149304965 wait complete. 2024-03-28T23:31:52.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.574+0000 7f47d18ed640 1 Processor -- start 2024-03-28T23:31:52.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.574+0000 7f47d18ed640 1 -- start start 2024-03-28T23:31:52.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.575+0000 7f47d18ed640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47b8005230 0x7f47cc0c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:52.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.575+0000 7f47d18ed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47cc0b06f0 0x7f47cc0bf000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:52.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.575+0000 7f47d18ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47cc0c3920 0x7f47cc0bf540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:52.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.575+0000 7f47d18ed640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f47cc06d440 con 0x7f47b8005230 2024-03-28T23:31:52.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.575+0000 7f47d18ed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f47cc0cb4e0 con 0x7f47cc0b06f0 2024-03-28T23:31:52.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.575+0000 7f47d18ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f47cc06d5c0 con 0x7f47cc0c3920 2024-03-28T23:31:52.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.575+0000 7f47caffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47b8005230 0x7f47cc0c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:52.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.575+0000 7f47caffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47b8005230 0x7f47cc0c33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35080/0 (socket says 172.21.15.161:35080) 2024-03-28T23:31:52.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.575+0000 7f47caffd640 1 -- 172.21.15.161:0/1926319048 learned_addr learned my addr 172.21.15.161:0/1926319048 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:52.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.575+0000 7f47ca7fc640 1 --2- 172.21.15.161:0/1926319048 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47cc0b06f0 0x7f47cc0bf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:52.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.575+0000 7f47cb7fe640 1 --2- 172.21.15.161:0/1926319048 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47cc0c3920 0x7f47cc0bf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:52.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.575+0000 7f47caffd640 1 -- 172.21.15.161:0/1926319048 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47cc0c3920 msgr2=0x7f47cc0bf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:52.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.575+0000 7f47caffd640 1 --2- 172.21.15.161:0/1926319048 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47cc0c3920 0x7f47cc0bf540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.575+0000 7f47caffd640 1 -- 172.21.15.161:0/1926319048 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47cc0b06f0 msgr2=0x7f47cc0bf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:52.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.575+0000 7f47caffd640 1 --2- 172.21.15.161:0/1926319048 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47cc0b06f0 0x7f47cc0bf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.575+0000 7f47caffd640 1 -- 172.21.15.161:0/1926319048 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47cc14c0f0 con 0x7f47b8005230 2024-03-28T23:31:52.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.575+0000 7f47ca7fc640 1 --2- 172.21.15.161:0/1926319048 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47cc0b06f0 0x7f47cc0bf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:52.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.576+0000 7f47caffd640 1 --2- 172.21.15.161:0/1926319048 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47b8005230 0x7f47cc0c33e0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f47c4011e30 tx=0x7f47c400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:52.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.576+0000 7f47c3fff640 1 -- 172.21.15.161:0/1926319048 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c4015430 con 0x7f47b8005230 2024-03-28T23:31:52.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.576+0000 7f47c3fff640 1 -- 172.21.15.161:0/1926319048 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47c40155d0 con 0x7f47b8005230 2024-03-28T23:31:52.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.576+0000 7f47d18ed640 1 -- 172.21.15.161:0/1926319048 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47cc14c3d0 con 0x7f47b8005230 2024-03-28T23:31:52.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.576+0000 7f47c3fff640 1 -- 172.21.15.161:0/1926319048 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c40159c0 con 0x7f47b8005230 2024-03-28T23:31:52.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.576+0000 7f47d18ed640 1 -- 172.21.15.161:0/1926319048 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47cc0c1350 con 0x7f47b8005230 2024-03-28T23:31:52.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.577+0000 7f47d18ed640 1 -- 172.21.15.161:0/1926319048 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47cc0c65a0 con 0x7f47b8005230 2024-03-28T23:31:52.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.577+0000 7f47c3fff640 1 -- 172.21.15.161:0/1926319048 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f47c4015b60 con 0x7f47b8005230 2024-03-28T23:31:52.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.578+0000 7f47c3fff640 1 --2- 172.21.15.161:0/1926319048 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f47ac06b4d0 0x7f47ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:52.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.578+0000 7f47c3fff640 1 -- 172.21.15.161:0/1926319048 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f47c408b390 con 0x7f47b8005230 2024-03-28T23:31:52.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.578+0000 7f47ca7fc640 1 --2- 172.21.15.161:0/1926319048 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f47ac06b4d0 0x7f47ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:52.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.578+0000 7f47ca7fc640 1 --2- 172.21.15.161:0/1926319048 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f47ac06b4d0 0x7f47ac06d990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f47b40010e0 tx=0x7f47b4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:52.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.582+0000 7f47c3fff640 1 -- 172.21.15.161:0/1926319048 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f47cc0c65a0 con 0x7f47b8005230 2024-03-28T23:31:52.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.754+0000 7f47d18ed640 1 -- 172.21.15.161:0/1926319048 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64} v 0) -- 0x7f47cc068f90 con 0x7f47b8005230 2024-03-28T23:31:52.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.864+0000 7f47c3fff640 1 -- 172.21.15.161:0/1926319048 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64}]=0 pool 'cephfs2_metadata' created v52) ==== 140+0+0 (secure 0 0 0) 0x7f47c4019030 con 0x7f47b8005230 2024-03-28T23:31:52.865 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs2_metadata' created 2024-03-28T23:31:52.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.867+0000 7f47d18ed640 1 -- 172.21.15.161:0/1926319048 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f47ac06b4d0 msgr2=0x7f47ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:52.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.867+0000 7f47d18ed640 1 --2- 172.21.15.161:0/1926319048 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f47ac06b4d0 0x7f47ac06d990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f47b40010e0 tx=0x7f47b4009040 comp rx=0 tx=0).stop 2024-03-28T23:31:52.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.867+0000 7f47d18ed640 1 -- 172.21.15.161:0/1926319048 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47b8005230 msgr2=0x7f47cc0c33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:52.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.867+0000 7f47d18ed640 1 --2- 172.21.15.161:0/1926319048 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47b8005230 0x7f47cc0c33e0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f47c4011e30 tx=0x7f47c400f580 comp rx=0 tx=0).stop 2024-03-28T23:31:52.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.869+0000 7f47d18ed640 1 -- 172.21.15.161:0/1926319048 shutdown_connections 2024-03-28T23:31:52.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.869+0000 7f47d18ed640 1 --2- 172.21.15.161:0/1926319048 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f47ac06b4d0 0x7f47ac06d990 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.869+0000 7f47d18ed640 1 --2- 172.21.15.161:0/1926319048 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47cc0c3920 0x7f47cc0bf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.869+0000 7f47d18ed640 1 --2- 172.21.15.161:0/1926319048 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47cc0b06f0 0x7f47cc0bf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.869+0000 7f47d18ed640 1 --2- 172.21.15.161:0/1926319048 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47b8005230 0x7f47cc0c33e0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.869+0000 7f47d18ed640 1 -- 172.21.15.161:0/1926319048 >> 172.21.15.161:0/1926319048 conn(0x7f47cc072ca0 msgr2=0x7f47cc1580f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:52.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.870+0000 7f47d18ed640 1 -- 172.21.15.161:0/1926319048 shutdown_connections 2024-03-28T23:31:52.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.871+0000 7f47d18ed640 1 -- 172.21.15.161:0/1926319048 wait complete. 2024-03-28T23:31:52.885 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T23:31:52.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.993+0000 7f5d88a15640 1 Processor -- start 2024-03-28T23:31:52.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.993+0000 7f5d88a15640 1 -- start start 2024-03-28T23:31:52.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.994+0000 7f5d88a15640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5d841589b0 0x7f5d84158e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:52.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.994+0000 7f5d88a15640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d840c6f60 con 0x7f5d840c0330 2024-03-28T23:31:52.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.994+0000 7f5d88a15640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d840c6120 con 0x7f5d84152bc0 2024-03-28T23:31:52.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.994+0000 7f5d88a15640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f5d840c6400 con 0x7f5d841589b0 2024-03-28T23:31:52.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.994+0000 7f5d7bfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f5d840c0330 0x7f5d840c9190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37800/0 (socket says 172.21.15.161:37800) 2024-03-28T23:31:52.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.994+0000 7f5d7bfff640 1 -- 172.21.15.161:0/1770920476 learned_addr learned my addr 172.21.15.161:0/1770920476 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:52.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.994+0000 7f5d82d9d640 1 --2- 172.21.15.161:0/1770920476 >> v2:172.21.15.170:3300/0 conn(0x7f5d841589b0 0x7f5d84158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:52.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.994+0000 7f5d81d9b640 1 -- 172.21.15.161:0/1770920476 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 266839310 0 0) 0x7f5d840c6f60 con 0x7f5d840c0330 2024-03-28T23:31:52.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.994+0000 7f5d81d9b640 1 -- 172.21.15.161:0/1770920476 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d70003520 con 0x7f5d840c0330 2024-03-28T23:31:52.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.995+0000 7f5d81d9b640 1 -- 172.21.15.161:0/1770920476 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3370171421 0 0) 0x7f5d70003520 con 0x7f5d840c0330 2024-03-28T23:31:52.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.995+0000 7f5d81d9b640 1 -- 172.21.15.161:0/1770920476 >> v2:172.21.15.170:3300/0 conn(0x7f5d841589b0 msgr2=0x7f5d84158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:52.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.995+0000 7f5d81d9b640 1 --2- 172.21.15.161:0/1770920476 >> v2:172.21.15.170:3300/0 conn(0x7f5d841589b0 0x7f5d84158e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.995+0000 7f5d81d9b640 1 -- 172.21.15.161:0/1770920476 >> v1:172.21.15.170:6789/0 conn(0x7f5d84152bc0 legacy=0x7f5d840c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:52.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.995+0000 7f5d81d9b640 1 -- 172.21.15.161:0/1770920476 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d840c7240 con 0x7f5d840c0330 2024-03-28T23:31:52.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.995+0000 7f5d81d9b640 1 -- 172.21.15.161:0/1770920476 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f5d6c001670 con 0x7f5d840c0330 2024-03-28T23:31:52.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.995+0000 7f5d81d9b640 1 -- 172.21.15.161:0/1770920476 >> v1:172.21.15.161:6789/0 conn(0x7f5d840c0330 legacy=0x7f5d840c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:52.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.995+0000 7f5d81d9b640 1 --2- 172.21.15.161:0/1770920476 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d70003a90 0x7f5d70003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:52.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.995+0000 7f5d81d9b640 1 --2- 172.21.15.161:0/1770920476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d70004800 0x7f5d70004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:52.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.995+0000 7f5d81d9b640 1 --2- 172.21.15.161:0/1770920476 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d70005290 0x7f5d7000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:52.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.995+0000 7f5d81d9b640 1 -- 172.21.15.161:0/1770920476 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5d840c6400 con 0x7f5d70004800 2024-03-28T23:31:52.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.995+0000 7f5d81d9b640 1 -- 172.21.15.161:0/1770920476 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7000c1e0 con 0x7f5d70005290 2024-03-28T23:31:52.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.995+0000 7f5d81d9b640 1 -- 172.21.15.161:0/1770920476 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5d7000c3e0 con 0x7f5d70003a90 2024-03-28T23:31:52.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.995+0000 7f5d7bfff640 1 --2- 172.21.15.161:0/1770920476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d70004800 0x7f5d70004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:52.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.995+0000 7f5d82d9d640 1 --2- 172.21.15.161:0/1770920476 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d70005290 0x7f5d7000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:52.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.995+0000 7f5d8259c640 1 --2- 172.21.15.161:0/1770920476 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d70003a90 0x7f5d70003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:52.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.995+0000 7f5d7bfff640 1 -- 172.21.15.161:0/1770920476 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d70003a90 msgr2=0x7f5d70003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:52.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.995+0000 7f5d7bfff640 1 --2- 172.21.15.161:0/1770920476 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d70003a90 0x7f5d70003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.995+0000 7f5d7bfff640 1 -- 172.21.15.161:0/1770920476 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d70005290 msgr2=0x7f5d7000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:52.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.995+0000 7f5d7bfff640 1 --2- 172.21.15.161:0/1770920476 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d70005290 0x7f5d7000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:52.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.995+0000 7f5d82d9d640 1 -- 172.21.15.161:0/1770920476 reap_dead start 2024-03-28T23:31:53.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.995+0000 7f5d7bfff640 1 -- 172.21.15.161:0/1770920476 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d70003520 con 0x7f5d70004800 2024-03-28T23:31:53.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.996+0000 7f5d7bfff640 1 --2- 172.21.15.161:0/1770920476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d70004800 0x7f5d70004c80 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c00c210 tx=0x7f5d70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=689a069b0de8ed58 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:53.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.996+0000 7f5d81d9b640 1 -- 172.21.15.161:0/1770920476 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c007d30 con 0x7f5d70004800 2024-03-28T23:31:53.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.996+0000 7f5d81d9b640 1 -- 172.21.15.161:0/1770920476 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d6c030070 con 0x7f5d70004800 2024-03-28T23:31:53.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.996+0000 7f5d81d9b640 1 -- 172.21.15.161:0/1770920476 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c005d70 con 0x7f5d70004800 2024-03-28T23:31:53.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.997+0000 7f5d88a15640 1 -- 172.21.15.161:0/1770920476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d70004800 msgr2=0x7f5d70004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:53.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.997+0000 7f5d88a15640 1 --2- 172.21.15.161:0/1770920476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d70004800 0x7f5d70004c80 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c00c210 tx=0x7f5d70005830 comp rx=0 tx=0).stop 2024-03-28T23:31:53.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.997+0000 7f5d88a15640 1 -- 172.21.15.161:0/1770920476 shutdown_connections 2024-03-28T23:31:53.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.997+0000 7f5d88a15640 1 --2- 172.21.15.161:0/1770920476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d70004800 0x7f5d70004c80 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:53.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.997+0000 7f5d88a15640 1 -- 172.21.15.161:0/1770920476 >> 172.21.15.161:0/1770920476 conn(0x7f5d840caa80 msgr2=0x7f5d840cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:53.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.997+0000 7f5d88a15640 1 -- 172.21.15.161:0/1770920476 shutdown_connections 2024-03-28T23:31:53.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.997+0000 7f5d88a15640 1 -- 172.21.15.161:0/1770920476 wait complete. 2024-03-28T23:31:53.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.998+0000 7f5d88a15640 1 Processor -- start 2024-03-28T23:31:53.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.998+0000 7f5d88a15640 1 -- start start 2024-03-28T23:31:53.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.999+0000 7f5d88a15640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d70004800 0x7f5d84147cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:53.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.999+0000 7f5d88a15640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d84156970 0x7f5d84149830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:53.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.999+0000 7f5d88a15640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d84149d70 0x7f5d84150aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:53.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.999+0000 7f5d88a15640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5d84152fa0 con 0x7f5d84156970 2024-03-28T23:31:53.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.999+0000 7f5d88a15640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5d84158e00 con 0x7f5d84149d70 2024-03-28T23:31:53.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.999+0000 7f5d88a15640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5d841ede80 con 0x7f5d70004800 2024-03-28T23:31:53.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.999+0000 7f5d7bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d84156970 0x7f5d84149830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:53.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.999+0000 7f5d7bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d84156970 0x7f5d84149830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35096/0 (socket says 172.21.15.161:35096) 2024-03-28T23:31:53.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.999+0000 7f5d7bfff640 1 -- 172.21.15.161:0/2826366959 learned_addr learned my addr 172.21.15.161:0/2826366959 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:53.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.999+0000 7f5d8259c640 1 --2- 172.21.15.161:0/2826366959 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d70004800 0x7f5d84147cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:53.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.999+0000 7f5d82d9d640 1 --2- 172.21.15.161:0/2826366959 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d84149d70 0x7f5d84150aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:53.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.999+0000 7f5d7bfff640 1 -- 172.21.15.161:0/2826366959 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d70004800 msgr2=0x7f5d84147cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:53.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.999+0000 7f5d7bfff640 1 --2- 172.21.15.161:0/2826366959 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d70004800 0x7f5d84147cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:53.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.999+0000 7f5d7bfff640 1 -- 172.21.15.161:0/2826366959 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d84149d70 msgr2=0x7f5d84150aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:53.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.999+0000 7f5d7bfff640 1 --2- 172.21.15.161:0/2826366959 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d84149d70 0x7f5d84150aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:53.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.999+0000 7f5d7bfff640 1 -- 172.21.15.161:0/2826366959 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d84155f30 con 0x7f5d84156970 2024-03-28T23:31:53.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.999+0000 7f5d7bfff640 1 --2- 172.21.15.161:0/2826366959 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d84156970 0x7f5d84149830 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c001970 tx=0x7f5d6c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:53.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:52.999+0000 7f5d7affd640 1 -- 172.21.15.161:0/2826366959 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c045070 con 0x7f5d84156970 2024-03-28T23:31:53.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.000+0000 7f5d7affd640 1 -- 172.21.15.161:0/2826366959 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d6c000ed0 con 0x7f5d84156970 2024-03-28T23:31:53.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.000+0000 7f5d88a15640 1 -- 172.21.15.161:0/2826366959 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d841ee0d0 con 0x7f5d84156970 2024-03-28T23:31:53.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.000+0000 7f5d7affd640 1 -- 172.21.15.161:0/2826366959 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c004e50 con 0x7f5d84156970 2024-03-28T23:31:53.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.000+0000 7f5d88a15640 1 -- 172.21.15.161:0/2826366959 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d841ee690 con 0x7f5d84156970 2024-03-28T23:31:53.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.001+0000 7f5d88a15640 1 -- 172.21.15.161:0/2826366959 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d8404f4b0 con 0x7f5d84156970 2024-03-28T23:31:53.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.002+0000 7f5d7affd640 1 -- 172.21.15.161:0/2826366959 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5d6c007070 con 0x7f5d84156970 2024-03-28T23:31:53.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.002+0000 7f5d7affd640 1 --2- 172.21.15.161:0/2826366959 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5d6406b6a0 0x7f5d6406db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:53.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.002+0000 7f5d8259c640 1 --2- 172.21.15.161:0/2826366959 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5d6406b6a0 0x7f5d6406db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:53.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.002+0000 7f5d7affd640 1 -- 172.21.15.161:0/2826366959 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6472+0+0 (secure 0 0 0) 0x7f5d6c0ab220 con 0x7f5d84156970 2024-03-28T23:31:53.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.003+0000 7f5d8259c640 1 --2- 172.21.15.161:0/2826366959 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5d6406b6a0 0x7f5d6406db60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5d74000c00 tx=0x7f5d74009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:53.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.007+0000 7f5d7affd640 1 -- 172.21.15.161:0/2826366959 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5d6c0797e0 con 0x7f5d84156970 2024-03-28T23:31:53.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.182+0000 7f5d88a15640 1 -- 172.21.15.161:0/2826366959 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f5d84156f50 con 0x7f5d84156970 2024-03-28T23:31:53.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.866+0000 7f5d7affd640 1 -- 172.21.15.161:0/2826366959 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs2_data' created v53) ==== 172+0+0 (secure 0 0 0) 0x7f5d6c07d880 con 0x7f5d84156970 2024-03-28T23:31:53.867 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs2_data' created 2024-03-28T23:31:53.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.870+0000 7f5d88a15640 1 -- 172.21.15.161:0/2826366959 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5d6406b6a0 msgr2=0x7f5d6406db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.870+0000 7f5d88a15640 1 --2- 172.21.15.161:0/2826366959 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5d6406b6a0 0x7f5d6406db60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5d74000c00 tx=0x7f5d74009040 comp rx=0 tx=0).stop 2024-03-28T23:31:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.870+0000 7f5d88a15640 1 -- 172.21.15.161:0/2826366959 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d84156970 msgr2=0x7f5d84149830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.870+0000 7f5d88a15640 1 --2- 172.21.15.161:0/2826366959 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d84156970 0x7f5d84149830 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c001970 tx=0x7f5d6c02e070 comp rx=0 tx=0).stop 2024-03-28T23:31:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.870+0000 7f5d88a15640 1 -- 172.21.15.161:0/2826366959 shutdown_connections 2024-03-28T23:31:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.870+0000 7f5d88a15640 1 --2- 172.21.15.161:0/2826366959 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5d6406b6a0 0x7f5d6406db60 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.870+0000 7f5d88a15640 1 --2- 172.21.15.161:0/2826366959 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d84156970 0x7f5d84149830 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.870+0000 7f5d88a15640 1 --2- 172.21.15.161:0/2826366959 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d84149d70 0x7f5d84150aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.870+0000 7f5d88a15640 1 --2- 172.21.15.161:0/2826366959 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d70004800 0x7f5d84147cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.870+0000 7f5d88a15640 1 -- 172.21.15.161:0/2826366959 >> 172.21.15.161:0/2826366959 conn(0x7f5d840caa80 msgr2=0x7f5d8414df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.870+0000 7f5d88a15640 1 -- 172.21.15.161:0/2826366959 shutdown_connections 2024-03-28T23:31:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.870+0000 7f5d88a15640 1 -- 172.21.15.161:0/2826366959 wait complete. 2024-03-28T23:31:53.893 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs2 cephfs2_metadata cephfs2_data 2024-03-28T23:31:53.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.997+0000 7f1bfd1f0640 1 Processor -- start 2024-03-28T23:31:53.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.997+0000 7f1bfd1f0640 1 -- start start 2024-03-28T23:31:54.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.998+0000 7f1bfd1f0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1bf80c0330 0x7f1bf80c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:54.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.998+0000 7f1bfd1f0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1bf81589b0 0x7f1bf8158e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:54.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.998+0000 7f1bfd1f0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bf80c6f60 con 0x7f1bf8152bc0 2024-03-28T23:31:54.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.998+0000 7f1bfd1f0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1bf80c6120 con 0x7f1bf81589b0 2024-03-28T23:31:54.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.998+0000 7f1bfd1f0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1bf80c6400 con 0x7f1bf80c0330 2024-03-28T23:31:54.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.998+0000 7f1bf759e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1bf81589b0 0x7f1bf8158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:54.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.998+0000 7f1bf759e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1bf81589b0 0x7f1bf8158e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35102/0 (socket says 172.21.15.161:35102) 2024-03-28T23:31:54.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.998+0000 7f1bf759e640 1 -- 172.21.15.161:0/953565610 learned_addr learned my addr 172.21.15.161:0/953565610 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:54.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.998+0000 7f1bf659c640 1 --2- 172.21.15.161:0/953565610 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1bf80c0330 0x7f1bf80c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:54.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.998+0000 7f1bf759e640 1 -- 172.21.15.161:0/953565610 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1bf80c0330 msgr2=0x7f1bf80c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:54.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.998+0000 7f1bf759e640 1 --2- 172.21.15.161:0/953565610 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1bf80c0330 0x7f1bf80c9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:54.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.998+0000 7f1bf759e640 1 -- 172.21.15.161:0/953565610 >> v1:172.21.15.170:6789/0 conn(0x7f1bf8152bc0 legacy=0x7f1bf80c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:54.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.998+0000 7f1bf759e640 1 -- 172.21.15.161:0/953565610 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf80c96d0 con 0x7f1bf81589b0 2024-03-28T23:31:54.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.998+0000 7f1bf659c640 1 --2- 172.21.15.161:0/953565610 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1bf80c0330 0x7f1bf80c9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:54.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.999+0000 7f1bf759e640 1 --2- 172.21.15.161:0/953565610 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bf81589b0 0x7f1bf8158e10 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009870 tx=0x7f1be802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67832ded8c6d8193 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:54.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.999+0000 7f1bf5d9b640 1 -- 172.21.15.161:0/953565610 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be8034070 con 0x7f1bf81589b0 2024-03-28T23:31:54.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.999+0000 7f1bf5d9b640 1 -- 172.21.15.161:0/953565610 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1be8030840 con 0x7f1bf81589b0 2024-03-28T23:31:54.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:53.999+0000 7f1bf5d9b640 1 -- 172.21.15.161:0/953565610 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be8030c10 con 0x7f1bf81589b0 2024-03-28T23:31:54.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.000+0000 7f1bfd1f0640 1 -- 172.21.15.161:0/953565610 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bf81589b0 msgr2=0x7f1bf8158e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:54.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.000+0000 7f1bfd1f0640 1 --2- 172.21.15.161:0/953565610 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bf81589b0 0x7f1bf8158e10 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009870 tx=0x7f1be802b980 comp rx=0 tx=0).stop 2024-03-28T23:31:54.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.000+0000 7f1bfd1f0640 1 -- 172.21.15.161:0/953565610 shutdown_connections 2024-03-28T23:31:54.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.000+0000 7f1bfd1f0640 1 --2- 172.21.15.161:0/953565610 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bf81589b0 0x7f1bf8158e10 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:54.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.000+0000 7f1bfd1f0640 1 --2- 172.21.15.161:0/953565610 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1bf80c0330 0x7f1bf80c9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:54.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.000+0000 7f1bfd1f0640 1 -- 172.21.15.161:0/953565610 >> 172.21.15.161:0/953565610 conn(0x7f1bf80caa80 msgr2=0x7f1bf80cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:54.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.000+0000 7f1bfd1f0640 1 -- 172.21.15.161:0/953565610 shutdown_connections 2024-03-28T23:31:54.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.000+0000 7f1bfd1f0640 1 -- 172.21.15.161:0/953565610 wait complete. 2024-03-28T23:31:54.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.001+0000 7f1bfd1f0640 1 Processor -- start 2024-03-28T23:31:54.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.001+0000 7f1bfd1f0640 1 -- start start 2024-03-28T23:31:54.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.002+0000 7f1bfd1f0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bf80c0330 0x7f1bf8157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:54.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.002+0000 7f1bfd1f0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1bf8152bc0 0x7f1bf81515c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:54.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.002+0000 7f1bfd1f0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bf81589b0 0x7f1bf8151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:54.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.002+0000 7f1bfd1f0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf8152fa0 con 0x7f1bf81589b0 2024-03-28T23:31:54.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.002+0000 7f1bfd1f0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf80c6bf0 con 0x7f1bf80c0330 2024-03-28T23:31:54.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.002+0000 7f1bfd1f0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1bf814ada0 con 0x7f1bf8152bc0 2024-03-28T23:31:54.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.002+0000 7f1bf759e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bf81589b0 0x7f1bf8151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:54.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.002+0000 7f1bf759e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bf81589b0 0x7f1bf8151b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35112/0 (socket says 172.21.15.161:35112) 2024-03-28T23:31:54.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.002+0000 7f1bf759e640 1 -- 172.21.15.161:0/3102086232 learned_addr learned my addr 172.21.15.161:0/3102086232 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:54.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.002+0000 7f1bf6d9d640 1 --2- 172.21.15.161:0/3102086232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bf80c0330 0x7f1bf8157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:54.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.002+0000 7f1bf659c640 1 --2- 172.21.15.161:0/3102086232 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1bf8152bc0 0x7f1bf81515c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:54.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.003+0000 7f1bf759e640 1 -- 172.21.15.161:0/3102086232 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1bf8152bc0 msgr2=0x7f1bf81515c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:54.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.003+0000 7f1bf759e640 1 --2- 172.21.15.161:0/3102086232 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1bf8152bc0 0x7f1bf81515c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:54.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.003+0000 7f1bf759e640 1 -- 172.21.15.161:0/3102086232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bf80c0330 msgr2=0x7f1bf8157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:54.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.003+0000 7f1bf759e640 1 --2- 172.21.15.161:0/3102086232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bf80c0330 0x7f1bf8157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:54.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.003+0000 7f1bf759e640 1 -- 172.21.15.161:0/3102086232 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf8145fb0 con 0x7f1bf81589b0 2024-03-28T23:31:54.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.003+0000 7f1bf759e640 1 --2- 172.21.15.161:0/3102086232 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bf81589b0 0x7f1bf8151b00 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009870 tx=0x7f1be80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:54.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.003+0000 7f1bdf7fe640 1 -- 172.21.15.161:0/3102086232 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be8034040 con 0x7f1bf81589b0 2024-03-28T23:31:54.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.003+0000 7f1bdf7fe640 1 -- 172.21.15.161:0/3102086232 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1be8039d90 con 0x7f1bf81589b0 2024-03-28T23:31:54.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.003+0000 7f1bfd1f0640 1 -- 172.21.15.161:0/3102086232 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bf8150760 con 0x7f1bf81589b0 2024-03-28T23:31:54.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.003+0000 7f1bdf7fe640 1 -- 172.21.15.161:0/3102086232 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be8043610 con 0x7f1bf81589b0 2024-03-28T23:31:54.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.003+0000 7f1bfd1f0640 1 -- 172.21.15.161:0/3102086232 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bf8150e00 con 0x7f1bf81589b0 2024-03-28T23:31:54.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.004+0000 7f1bfd1f0640 1 -- 172.21.15.161:0/3102086232 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bf8068f90 con 0x7f1bf81589b0 2024-03-28T23:31:54.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.004+0000 7f1bdf7fe640 1 -- 172.21.15.161:0/3102086232 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1be804a060 con 0x7f1bf81589b0 2024-03-28T23:31:54.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.005+0000 7f1bdf7fe640 1 --2- 172.21.15.161:0/3102086232 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1be406b500 0x7f1be406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:54.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.005+0000 7f1bf6d9d640 1 --2- 172.21.15.161:0/3102086232 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1be406b500 0x7f1be406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:54.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.005+0000 7f1bdf7fe640 1 -- 172.21.15.161:0/3102086232 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6855+0+0 (secure 0 0 0) 0x7f1be80ac490 con 0x7f1bf81589b0 2024-03-28T23:31:54.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.005+0000 7f1bf6d9d640 1 --2- 172.21.15.161:0/3102086232 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1be406b500 0x7f1be406d9c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0001f10 tx=0x7f1bf0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:54.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.010+0000 7f1bdf7fe640 1 -- 172.21.15.161:0/3102086232 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1bf8068f90 con 0x7f1bf81589b0 2024-03-28T23:31:54.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.224+0000 7f1bfd1f0640 1 -- 172.21.15.161:0/3102086232 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"} v 0) -- 0x7f1bf804f4b0 con 0x7f1bf81589b0 2024-03-28T23:31:54.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.888+0000 7f1bdf7fe640 1 -- 172.21.15.161:0/3102086232 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"}]=0 new fs with metadata pool 9 and data pool 10 v31) ==== 176+0+0 (secure 0 0 0) 0x7f1be803e070 con 0x7f1bf81589b0 2024-03-28T23:31:54.889 INFO:teuthology.orchestra.run.smithi161.stderr:new fs with metadata pool 9 and data pool 10 2024-03-28T23:31:54.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.892+0000 7f1bfd1f0640 1 -- 172.21.15.161:0/3102086232 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1be406b500 msgr2=0x7f1be406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:54.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.892+0000 7f1bfd1f0640 1 --2- 172.21.15.161:0/3102086232 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1be406b500 0x7f1be406d9c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0001f10 tx=0x7f1bf0009040 comp rx=0 tx=0).stop 2024-03-28T23:31:54.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.892+0000 7f1bfd1f0640 1 -- 172.21.15.161:0/3102086232 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bf81589b0 msgr2=0x7f1bf8151b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:54.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.892+0000 7f1bfd1f0640 1 --2- 172.21.15.161:0/3102086232 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bf81589b0 0x7f1bf8151b00 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009870 tx=0x7f1be80046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:54.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.892+0000 7f1bfd1f0640 1 -- 172.21.15.161:0/3102086232 shutdown_connections 2024-03-28T23:31:54.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.892+0000 7f1bfd1f0640 1 --2- 172.21.15.161:0/3102086232 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1be406b500 0x7f1be406d9c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:54.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.892+0000 7f1bfd1f0640 1 --2- 172.21.15.161:0/3102086232 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bf81589b0 0x7f1bf8151b00 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:54.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.892+0000 7f1bfd1f0640 1 --2- 172.21.15.161:0/3102086232 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1bf8152bc0 0x7f1bf81515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:54.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.892+0000 7f1bfd1f0640 1 --2- 172.21.15.161:0/3102086232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bf80c0330 0x7f1bf8157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:54.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.892+0000 7f1bfd1f0640 1 -- 172.21.15.161:0/3102086232 >> 172.21.15.161:0/3102086232 conn(0x7f1bf80caa80 msgr2=0x7f1bf80c5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:54.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.895+0000 7f1bfd1f0640 1 -- 172.21.15.161:0/3102086232 shutdown_connections 2024-03-28T23:31:54.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:54.898+0000 7f1bfd1f0640 1 -- 172.21.15.161:0/3102086232 wait complete. 2024-03-28T23:31:54.921 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:31:55.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.053+0000 7f74dfafb640 1 Processor -- start 2024-03-28T23:31:55.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.053+0000 7f74dfafb640 1 -- start start 2024-03-28T23:31:55.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.054+0000 7f74dfafb640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f74d806ca20 0x7f74d806ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:55.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.054+0000 7f74dfafb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d80a98c0 0x7f74d80a9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:55.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.055+0000 7f74dfafb640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74d8151850 con 0x7f74d814eca0 2024-03-28T23:31:55.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.055+0000 7f74dfafb640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f74d8152460 con 0x7f74d806ca20 2024-03-28T23:31:55.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.055+0000 7f74dfafb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f74d8151b30 con 0x7f74d80a98c0 2024-03-28T23:31:55.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.055+0000 7f74de098640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d80a98c0 0x7f74d80a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:55.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.055+0000 7f74dd096640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f74d806ca20 0x7f74d806ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:55.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.055+0000 7f74de098640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d80a98c0 0x7f74d80a9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:60234/0 (socket says 172.21.15.161:60234) 2024-03-28T23:31:55.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.055+0000 7f74de098640 1 -- 172.21.15.161:0/58590779 learned_addr learned my addr 172.21.15.161:0/58590779 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:55.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.055+0000 7f74de098640 1 -- 172.21.15.161:0/58590779 >> v2:172.21.15.170:3300/0 conn(0x7f74d806ca20 msgr2=0x7f74d806ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:55.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.055+0000 7f74de098640 1 --2- 172.21.15.161:0/58590779 >> v2:172.21.15.170:3300/0 conn(0x7f74d806ca20 0x7f74d806ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.055+0000 7f74de098640 1 -- 172.21.15.161:0/58590779 >> v1:172.21.15.170:6789/0 conn(0x7f74d814eca0 legacy=0x7f74d80c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:55.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.055+0000 7f74de098640 1 -- 172.21.15.161:0/58590779 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74d8151e10 con 0x7f74d80a98c0 2024-03-28T23:31:55.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.055+0000 7f74dd096640 1 --2- 172.21.15.161:0/58590779 >> v2:172.21.15.170:3300/0 conn(0x7f74d806ca20 0x7f74d806ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:55.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.056+0000 7f74de098640 1 --2- 172.21.15.161:0/58590779 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d80a98c0 0x7f74d80a9d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f74c8009870 tx=0x7f74c802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53df8667a0652d0c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:55.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.056+0000 7f74dc895640 1 -- 172.21.15.161:0/58590779 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c8035020 con 0x7f74d80a98c0 2024-03-28T23:31:55.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.056+0000 7f74dc895640 1 -- 172.21.15.161:0/58590779 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74c8031640 con 0x7f74d80a98c0 2024-03-28T23:31:55.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.056+0000 7f74dc895640 1 -- 172.21.15.161:0/58590779 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c802d040 con 0x7f74d80a98c0 2024-03-28T23:31:55.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.057+0000 7f74dfafb640 1 -- 172.21.15.161:0/58590779 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d80a98c0 msgr2=0x7f74d80a9d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:55.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.057+0000 7f74dfafb640 1 --2- 172.21.15.161:0/58590779 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d80a98c0 0x7f74d80a9d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f74c8009870 tx=0x7f74c802b8f0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.057+0000 7f74dfafb640 1 -- 172.21.15.161:0/58590779 shutdown_connections 2024-03-28T23:31:55.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.057+0000 7f74dfafb640 1 --2- 172.21.15.161:0/58590779 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d80a98c0 0x7f74d80a9d50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.057+0000 7f74dfafb640 1 --2- 172.21.15.161:0/58590779 >> v2:172.21.15.170:3300/0 conn(0x7f74d806ca20 0x7f74d806ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.057+0000 7f74dfafb640 1 -- 172.21.15.161:0/58590779 >> 172.21.15.161:0/58590779 conn(0x7f74d80c6670 msgr2=0x7f74d80c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:55.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.057+0000 7f74dfafb640 1 -- 172.21.15.161:0/58590779 shutdown_connections 2024-03-28T23:31:55.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.057+0000 7f74dfafb640 1 -- 172.21.15.161:0/58590779 wait complete. 2024-03-28T23:31:55.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.059+0000 7f74dfafb640 1 Processor -- start 2024-03-28T23:31:55.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.059+0000 7f74dfafb640 1 -- start start 2024-03-28T23:31:55.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.060+0000 7f74dfafb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74d806ca20 0x7f74d81ecd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:55.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.060+0000 7f74dfafb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74d80a98c0 0x7f74d8154d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:55.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.060+0000 7f74dfafb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d814eca0 0x7f74d81e9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:55.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.060+0000 7f74dfafb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f74d81514b0 con 0x7f74d806ca20 2024-03-28T23:31:55.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.060+0000 7f74dfafb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f74d814f080 con 0x7f74d80a98c0 2024-03-28T23:31:55.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.060+0000 7f74dfafb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f74d8151630 con 0x7f74d814eca0 2024-03-28T23:31:55.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.060+0000 7f74dd897640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74d806ca20 0x7f74d81ecd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:55.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.060+0000 7f74dd897640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74d806ca20 0x7f74d81ecd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35124/0 (socket says 172.21.15.161:35124) 2024-03-28T23:31:55.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.060+0000 7f74dd897640 1 -- 172.21.15.161:0/859256461 learned_addr learned my addr 172.21.15.161:0/859256461 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:55.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.060+0000 7f74dd096640 1 --2- 172.21.15.161:0/859256461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74d80a98c0 0x7f74d8154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:55.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.060+0000 7f74de098640 1 --2- 172.21.15.161:0/859256461 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d814eca0 0x7f74d81e9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:55.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.060+0000 7f74dd897640 1 -- 172.21.15.161:0/859256461 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d814eca0 msgr2=0x7f74d81e9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:55.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.060+0000 7f74dd897640 1 --2- 172.21.15.161:0/859256461 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d814eca0 0x7f74d81e9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.060+0000 7f74dd897640 1 -- 172.21.15.161:0/859256461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74d80a98c0 msgr2=0x7f74d8154d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:55.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.060+0000 7f74dd897640 1 --2- 172.21.15.161:0/859256461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74d80a98c0 0x7f74d8154d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.060+0000 7f74dd897640 1 -- 172.21.15.161:0/859256461 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74d80c5790 con 0x7f74d806ca20 2024-03-28T23:31:55.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.061+0000 7f74dd897640 1 --2- 172.21.15.161:0/859256461 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74d806ca20 0x7f74d81ecd10 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f74d000c760 tx=0x7f74d000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:55.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.061+0000 7f74d67fc640 1 -- 172.21.15.161:0/859256461 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d0016020 con 0x7f74d806ca20 2024-03-28T23:31:55.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.061+0000 7f74d67fc640 1 -- 172.21.15.161:0/859256461 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74d0006d50 con 0x7f74d806ca20 2024-03-28T23:31:55.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.061+0000 7f74d67fc640 1 -- 172.21.15.161:0/859256461 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d0003bb0 con 0x7f74d806ca20 2024-03-28T23:31:55.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.061+0000 7f74dfafb640 1 -- 172.21.15.161:0/859256461 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74d80c51d0 con 0x7f74d806ca20 2024-03-28T23:31:55.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.061+0000 7f74dfafb640 1 -- 172.21.15.161:0/859256461 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74d81e8e90 con 0x7f74d806ca20 2024-03-28T23:31:55.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.062+0000 7f74dfafb640 1 -- 172.21.15.161:0/859256461 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74d804f4b0 con 0x7f74d806ca20 2024-03-28T23:31:55.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.063+0000 7f74d67fc640 1 -- 172.21.15.161:0/859256461 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f74d000a850 con 0x7f74d806ca20 2024-03-28T23:31:55.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.063+0000 7f74d67fc640 1 --2- 172.21.15.161:0/859256461 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f74c406b4d0 0x7f74c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:55.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.063+0000 7f74d67fc640 1 -- 172.21.15.161:0/859256461 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6957+0+0 (secure 0 0 0) 0x7f74d008a1f0 con 0x7f74d806ca20 2024-03-28T23:31:55.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.063+0000 7f74dd096640 1 --2- 172.21.15.161:0/859256461 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f74c406b4d0 0x7f74c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:55.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.064+0000 7f74dd096640 1 --2- 172.21.15.161:0/859256461 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f74c406b4d0 0x7f74c406d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f74d806c6b0 tx=0x7f74cc00a2a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:55.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.068+0000 7f74d67fc640 1 -- 172.21.15.161:0/859256461 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f74d0020020 con 0x7f74d806ca20 2024-03-28T23:31:55.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.241+0000 7f74dfafb640 1 -- 172.21.15.161:0/859256461 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f74d80c63a0 con 0x7f74d806ca20 2024-03-28T23:31:55.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.243+0000 7f74d67fc640 1 -- 172.21.15.161:0/859256461 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) ==== 74+0+20715 (secure 0 0 0) 0x7f74d0058550 con 0x7f74d806ca20 2024-03-28T23:31:55.245 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:55.245 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":54,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:31:54.226115+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:31:37.940838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-28T23:31:38.692659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:31:52.774315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:31:53.201664+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:31:55.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.245+0000 7f74dfafb640 1 -- 172.21.15.161:0/859256461 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f74c406b4d0 msgr2=0x7f74c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:55.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.245+0000 7f74dfafb640 1 --2- 172.21.15.161:0/859256461 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f74c406b4d0 0x7f74c406d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f74d806c6b0 tx=0x7f74cc00a2a0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.245+0000 7f74dfafb640 1 -- 172.21.15.161:0/859256461 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74d806ca20 msgr2=0x7f74d81ecd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:55.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.245+0000 7f74dfafb640 1 --2- 172.21.15.161:0/859256461 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74d806ca20 0x7f74d81ecd10 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f74d000c760 tx=0x7f74d000cc20 comp rx=0 tx=0).stop 2024-03-28T23:31:55.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.245+0000 7f74dfafb640 1 -- 172.21.15.161:0/859256461 shutdown_connections 2024-03-28T23:31:55.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.245+0000 7f74dfafb640 1 --2- 172.21.15.161:0/859256461 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f74c406b4d0 0x7f74c406d990 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.245+0000 7f74dfafb640 1 --2- 172.21.15.161:0/859256461 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d814eca0 0x7f74d81e9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.245+0000 7f74dfafb640 1 --2- 172.21.15.161:0/859256461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74d80a98c0 0x7f74d8154d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.245+0000 7f74dfafb640 1 --2- 172.21.15.161:0/859256461 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74d806ca20 0x7f74d81ecd10 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.245+0000 7f74dfafb640 1 -- 172.21.15.161:0/859256461 >> 172.21.15.161:0/859256461 conn(0x7f74d80c6670 msgr2=0x7f74d80aa8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:55.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.245+0000 7f74dfafb640 1 -- 172.21.15.161:0/859256461 shutdown_connections 2024-03-28T23:31:55.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.245+0000 7f74dfafb640 1 -- 172.21.15.161:0/859256461 wait complete. 2024-03-28T23:31:55.258 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:31:55.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.360+0000 7f2e40023640 1 Processor -- start 2024-03-28T23:31:55.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.360+0000 7f2e40023640 1 -- start start 2024-03-28T23:31:55.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.361+0000 7f2e40023640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e38152890 0x7f2e380c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:55.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.361+0000 7f2e40023640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2e380c8ed0 0x7f2e380c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:55.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.361+0000 7f2e40023640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e38158a30 con 0x7f2e380c5950 2024-03-28T23:31:55.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.361+0000 7f2e40023640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f2e380c5db0 con 0x7f2e380c8ed0 2024-03-28T23:31:55.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.361+0000 7f2e40023640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2e380c8480 con 0x7f2e38152890 2024-03-28T23:31:55.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.361+0000 7f2e3ddbf640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e38152890 0x7f2e380c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:55.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.361+0000 7f2e3d5be640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2e380c8ed0 0x7f2e380c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:55.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.361+0000 7f2e3ddbf640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e38152890 0x7f2e380c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:60252/0 (socket says 172.21.15.161:60252) 2024-03-28T23:31:55.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.361+0000 7f2e3d5be640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2e380c8ed0 0x7f2e380c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:40990/0 (socket says 172.21.15.161:40990) 2024-03-28T23:31:55.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.361+0000 7f2e3ddbf640 1 -- 172.21.15.161:0/2305742465 learned_addr learned my addr 172.21.15.161:0/2305742465 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:55.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.361+0000 7f2e3e5c0640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f2e380c5950 0x7f2e381553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:53344/0 (socket says 172.21.15.161:53344) 2024-03-28T23:31:55.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.361+0000 7f2e3ddbf640 1 -- 172.21.15.161:0/2305742465 >> v2:172.21.15.170:3300/0 conn(0x7f2e380c8ed0 msgr2=0x7f2e380c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:55.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.361+0000 7f2e3ddbf640 1 --2- 172.21.15.161:0/2305742465 >> v2:172.21.15.170:3300/0 conn(0x7f2e380c8ed0 0x7f2e380c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.361+0000 7f2e3ddbf640 1 -- 172.21.15.161:0/2305742465 >> v1:172.21.15.170:6789/0 conn(0x7f2e380c5950 legacy=0x7f2e381553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:55.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.361+0000 7f2e3ddbf640 1 -- 172.21.15.161:0/2305742465 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e38158d10 con 0x7f2e38152890 2024-03-28T23:31:55.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.362+0000 7f2e3ddbf640 1 --2- 172.21.15.161:0/2305742465 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e38152890 0x7f2e380c8990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2e34009870 tx=0x7f2e3402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bd1641130bfd6e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:55.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.362+0000 7f2e3cdbd640 1 -- 172.21.15.161:0/2305742465 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e340317c0 con 0x7f2e38152890 2024-03-28T23:31:55.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.362+0000 7f2e3cdbd640 1 -- 172.21.15.161:0/2305742465 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e34037070 con 0x7f2e38152890 2024-03-28T23:31:55.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.362+0000 7f2e3cdbd640 1 -- 172.21.15.161:0/2305742465 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e34031b80 con 0x7f2e38152890 2024-03-28T23:31:55.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.363+0000 7f2e40023640 1 -- 172.21.15.161:0/2305742465 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e38152890 msgr2=0x7f2e380c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:55.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.363+0000 7f2e40023640 1 --2- 172.21.15.161:0/2305742465 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e38152890 0x7f2e380c8990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2e34009870 tx=0x7f2e3402b860 comp rx=0 tx=0).stop 2024-03-28T23:31:55.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.363+0000 7f2e40023640 1 -- 172.21.15.161:0/2305742465 shutdown_connections 2024-03-28T23:31:55.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.363+0000 7f2e40023640 1 --2- 172.21.15.161:0/2305742465 >> v2:172.21.15.170:3300/0 conn(0x7f2e380c8ed0 0x7f2e380c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.363+0000 7f2e40023640 1 --2- 172.21.15.161:0/2305742465 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e38152890 0x7f2e380c8990 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.363+0000 7f2e40023640 1 -- 172.21.15.161:0/2305742465 >> 172.21.15.161:0/2305742465 conn(0x7f2e380ac620 msgr2=0x7f2e380aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:55.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.363+0000 7f2e40023640 1 -- 172.21.15.161:0/2305742465 shutdown_connections 2024-03-28T23:31:55.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.363+0000 7f2e40023640 1 -- 172.21.15.161:0/2305742465 wait complete. 2024-03-28T23:31:55.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.364+0000 7f2e40023640 1 Processor -- start 2024-03-28T23:31:55.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.364+0000 7f2e40023640 1 -- start start 2024-03-28T23:31:55.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.364+0000 7f2e40023640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e380c5950 0x7f2e3814cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:55.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.364+0000 7f2e40023640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e380c8ed0 0x7f2e380ab6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:55.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.364+0000 7f2e40023640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e38152890 0x7f2e381ee7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:55.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.364+0000 7f2e40023640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2e381592d0 con 0x7f2e380c5950 2024-03-28T23:31:55.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.364+0000 7f2e40023640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2e38159640 con 0x7f2e38152890 2024-03-28T23:31:55.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.364+0000 7f2e40023640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2e38152c70 con 0x7f2e380c8ed0 2024-03-28T23:31:55.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.365+0000 7f2e3ddbf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e380c5950 0x7f2e3814cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.365+0000 7f2e3ddbf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e380c5950 0x7f2e3814cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35138/0 (socket says 172.21.15.161:35138) 2024-03-28T23:31:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.365+0000 7f2e3ddbf640 1 -- 172.21.15.161:0/4076523460 learned_addr learned my addr 172.21.15.161:0/4076523460 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.365+0000 7f2e3d5be640 1 --2- 172.21.15.161:0/4076523460 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e380c8ed0 0x7f2e380ab6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.365+0000 7f2e3e5c0640 1 --2- 172.21.15.161:0/4076523460 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e38152890 0x7f2e381ee7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.365+0000 7f2e3ddbf640 1 -- 172.21.15.161:0/4076523460 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e380c8ed0 msgr2=0x7f2e380ab6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.365+0000 7f2e3ddbf640 1 --2- 172.21.15.161:0/4076523460 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e380c8ed0 0x7f2e380ab6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.365+0000 7f2e3ddbf640 1 -- 172.21.15.161:0/4076523460 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e38152890 msgr2=0x7f2e381ee7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.365+0000 7f2e3ddbf640 1 --2- 172.21.15.161:0/4076523460 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e38152890 0x7f2e381ee7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.365+0000 7f2e3ddbf640 1 -- 172.21.15.161:0/4076523460 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e3814fca0 con 0x7f2e380c5950 2024-03-28T23:31:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.365+0000 7f2e3ddbf640 1 --2- 172.21.15.161:0/4076523460 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e380c5950 0x7f2e3814cb30 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f2e34000c00 tx=0x7f2e340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.365+0000 7f2e2e7fc640 1 -- 172.21.15.161:0/4076523460 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e34032650 con 0x7f2e380c5950 2024-03-28T23:31:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.365+0000 7f2e40023640 1 -- 172.21.15.161:0/4076523460 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e3814d1c0 con 0x7f2e380c5950 2024-03-28T23:31:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.365+0000 7f2e2e7fc640 1 -- 172.21.15.161:0/4076523460 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e340327f0 con 0x7f2e380c5950 2024-03-28T23:31:55.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.365+0000 7f2e2e7fc640 1 -- 172.21.15.161:0/4076523460 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e34032be0 con 0x7f2e380c5950 2024-03-28T23:31:55.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.365+0000 7f2e40023640 1 -- 172.21.15.161:0/4076523460 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e3814d780 con 0x7f2e380c5950 2024-03-28T23:31:55.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.366+0000 7f2e40023640 1 -- 172.21.15.161:0/4076523460 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e3804f4b0 con 0x7f2e380c5950 2024-03-28T23:31:55.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.367+0000 7f2e2e7fc640 1 -- 172.21.15.161:0/4076523460 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2e3404c060 con 0x7f2e380c5950 2024-03-28T23:31:55.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.367+0000 7f2e2e7fc640 1 --2- 172.21.15.161:0/4076523460 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e1c06b4d0 0x7f2e1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:55.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.367+0000 7f2e2e7fc640 1 -- 172.21.15.161:0/4076523460 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6957+0+0 (secure 0 0 0) 0x7f2e340ac190 con 0x7f2e380c5950 2024-03-28T23:31:55.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.367+0000 7f2e3d5be640 1 --2- 172.21.15.161:0/4076523460 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e1c06b4d0 0x7f2e1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:55.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.368+0000 7f2e3d5be640 1 --2- 172.21.15.161:0/4076523460 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e1c06b4d0 0x7f2e1c06d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2e38158960 tx=0x7f2e28009110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:55.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.372+0000 7f2e2e7fc640 1 -- 172.21.15.161:0/4076523460 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2e34040070 con 0x7f2e380c5950 2024-03-28T23:31:55.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.547+0000 7f2e40023640 1 -- 172.21.15.161:0/4076523460 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2e380c8140 con 0x7f2e380c5950 2024-03-28T23:31:55.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.549+0000 7f2e2e7fc640 1 -- 172.21.15.161:0/4076523460 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) ==== 74+0+20715 (secure 0 0 0) 0x7f2e3407ab40 con 0x7f2e380c5950 2024-03-28T23:31:55.550 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:55.550 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":54,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:31:54.226115+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:31:37.940838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-28T23:31:38.692659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:31:52.774315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:31:53.201664+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:31:55.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.551+0000 7f2e40023640 1 -- 172.21.15.161:0/4076523460 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e1c06b4d0 msgr2=0x7f2e1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:55.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.551+0000 7f2e40023640 1 --2- 172.21.15.161:0/4076523460 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e1c06b4d0 0x7f2e1c06d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2e38158960 tx=0x7f2e28009110 comp rx=0 tx=0).stop 2024-03-28T23:31:55.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.551+0000 7f2e40023640 1 -- 172.21.15.161:0/4076523460 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e380c5950 msgr2=0x7f2e3814cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:55.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.551+0000 7f2e40023640 1 --2- 172.21.15.161:0/4076523460 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e380c5950 0x7f2e3814cb30 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f2e34000c00 tx=0x7f2e340046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.551+0000 7f2e40023640 1 -- 172.21.15.161:0/4076523460 shutdown_connections 2024-03-28T23:31:55.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.551+0000 7f2e40023640 1 --2- 172.21.15.161:0/4076523460 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e1c06b4d0 0x7f2e1c06d990 secure :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2e38158960 tx=0x7f2e28009110 comp rx=0 tx=0).stop 2024-03-28T23:31:55.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.551+0000 7f2e40023640 1 --2- 172.21.15.161:0/4076523460 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e38152890 0x7f2e381ee7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.551+0000 7f2e40023640 1 --2- 172.21.15.161:0/4076523460 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e380c8ed0 0x7f2e380ab6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.551+0000 7f2e40023640 1 --2- 172.21.15.161:0/4076523460 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e380c5950 0x7f2e3814cb30 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.551+0000 7f2e40023640 1 -- 172.21.15.161:0/4076523460 >> 172.21.15.161:0/4076523460 conn(0x7f2e380ac620 msgr2=0x7f2e3814ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:55.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.551+0000 7f2e40023640 1 -- 172.21.15.161:0/4076523460 shutdown_connections 2024-03-28T23:31:55.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.551+0000 7f2e40023640 1 -- 172.21.15.161:0/4076523460 wait complete. 2024-03-28T23:31:55.563 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 standby_count_wanted 0 2024-03-28T23:31:55.589 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:31:55.609 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:31:55.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.669+0000 7f56830aa640 1 Processor -- start 2024-03-28T23:31:55.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.669+0000 7f56830aa640 1 -- start start 2024-03-28T23:31:55.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.670+0000 7f56830aa640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f567c1503e0 0x7f567c150850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:55.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.671+0000 7f56830aa640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f567c0abd40 con 0x7f567c0c3190 2024-03-28T23:31:55.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.671+0000 7f56830aa640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f567c0ac020 con 0x7f567c0a9560 2024-03-28T23:31:55.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.671+0000 7f56830aa640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f567c0abb10 con 0x7f567c1503e0 2024-03-28T23:31:55.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.671+0000 7f5680e46640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f567c0c3190 0x7f567c066eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37802/0 (socket says 172.21.15.161:37802) 2024-03-28T23:31:55.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.671+0000 7f5680e46640 1 -- 172.21.15.161:0/2950540362 learned_addr learned my addr 172.21.15.161:0/2950540362 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:55.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.671+0000 7f567bfff640 1 --2- 172.21.15.161:0/2950540362 >> v2:172.21.15.170:3300/0 conn(0x7f567c1503e0 0x7f567c150850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:55.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.671+0000 7f567b7fe640 1 -- 172.21.15.161:0/2950540362 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4236623862 0 0) 0x7f567c0abd40 con 0x7f567c0c3190 2024-03-28T23:31:55.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.671+0000 7f567b7fe640 1 -- 172.21.15.161:0/2950540362 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5664003520 con 0x7f567c0c3190 2024-03-28T23:31:55.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.671+0000 7f567bfff640 1 -- 172.21.15.161:0/2950540362 >> v1:172.21.15.170:6789/0 conn(0x7f567c0a9560 legacy=0x7f567c0a9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:55.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.671+0000 7f567bfff640 1 -- 172.21.15.161:0/2950540362 >> v1:172.21.15.161:6789/0 conn(0x7f567c0c3190 legacy=0x7f567c066eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:55.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.671+0000 7f567bfff640 1 -- 172.21.15.161:0/2950540362 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f567c0ac300 con 0x7f567c1503e0 2024-03-28T23:31:55.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.672+0000 7f567bfff640 1 --2- 172.21.15.161:0/2950540362 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f567c1503e0 0x7f567c150850 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f56700097a0 tx=0x7f567002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a90339148126b7fb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:55.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.672+0000 7f567b7fe640 1 -- 172.21.15.161:0/2950540362 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5670035020 con 0x7f567c1503e0 2024-03-28T23:31:55.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.672+0000 7f567b7fe640 1 -- 172.21.15.161:0/2950540362 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5670031830 con 0x7f567c1503e0 2024-03-28T23:31:55.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.672+0000 7f567b7fe640 1 -- 172.21.15.161:0/2950540362 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5670031be0 con 0x7f567c1503e0 2024-03-28T23:31:55.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.673+0000 7f56830aa640 1 -- 172.21.15.161:0/2950540362 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f567c1503e0 msgr2=0x7f567c150850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:55.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.673+0000 7f56830aa640 1 --2- 172.21.15.161:0/2950540362 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f567c1503e0 0x7f567c150850 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f56700097a0 tx=0x7f567002b8d0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.673+0000 7f56830aa640 1 -- 172.21.15.161:0/2950540362 shutdown_connections 2024-03-28T23:31:55.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.673+0000 7f56830aa640 1 --2- 172.21.15.161:0/2950540362 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f567c1503e0 0x7f567c150850 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.673+0000 7f56830aa640 1 -- 172.21.15.161:0/2950540362 >> 172.21.15.161:0/2950540362 conn(0x7f567c06ddc0 msgr2=0x7f567c06e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:55.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.673+0000 7f56830aa640 1 -- 172.21.15.161:0/2950540362 shutdown_connections 2024-03-28T23:31:55.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.673+0000 7f56830aa640 1 -- 172.21.15.161:0/2950540362 wait complete. 2024-03-28T23:31:55.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.674+0000 7f56830aa640 1 Processor -- start 2024-03-28T23:31:55.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.674+0000 7f56830aa640 1 -- start start 2024-03-28T23:31:55.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.675+0000 7f56830aa640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f567c0a9560 0x7f567c14bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:55.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.675+0000 7f56830aa640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f567c0c3190 0x7f567c153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:55.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.675+0000 7f56830aa640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f567c1503e0 0x7f567c153c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:55.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.675+0000 7f56830aa640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f567c0ad0d0 con 0x7f567c0c3190 2024-03-28T23:31:55.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.675+0000 7f56830aa640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f567c0ac670 con 0x7f567c1503e0 2024-03-28T23:31:55.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.675+0000 7f56830aa640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f567c0aa080 con 0x7f567c0a9560 2024-03-28T23:31:55.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.675+0000 7f567bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f567c0c3190 0x7f567c153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:55.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.675+0000 7f567bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f567c0c3190 0x7f567c153700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35148/0 (socket says 172.21.15.161:35148) 2024-03-28T23:31:55.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.675+0000 7f567bfff640 1 -- 172.21.15.161:0/3316531106 learned_addr learned my addr 172.21.15.161:0/3316531106 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:55.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.675+0000 7f5680e46640 1 --2- 172.21.15.161:0/3316531106 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f567c0a9560 0x7f567c14bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:55.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.675+0000 7f5681647640 1 --2- 172.21.15.161:0/3316531106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f567c1503e0 0x7f567c153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:55.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.675+0000 7f567bfff640 1 -- 172.21.15.161:0/3316531106 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f567c0a9560 msgr2=0x7f567c14bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:55.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.675+0000 7f567bfff640 1 --2- 172.21.15.161:0/3316531106 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f567c0a9560 0x7f567c14bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.675+0000 7f567bfff640 1 -- 172.21.15.161:0/3316531106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f567c1503e0 msgr2=0x7f567c153c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:55.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.675+0000 7f567bfff640 1 --2- 172.21.15.161:0/3316531106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f567c1503e0 0x7f567c153c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.675+0000 7f567bfff640 1 -- 172.21.15.161:0/3316531106 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f567c143510 con 0x7f567c0c3190 2024-03-28T23:31:55.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.676+0000 7f567bfff640 1 --2- 172.21.15.161:0/3316531106 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f567c0c3190 0x7f567c153700 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f56700097a0 tx=0x7f56700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:55.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.676+0000 7f56797fa640 1 -- 172.21.15.161:0/3316531106 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5670035040 con 0x7f567c0c3190 2024-03-28T23:31:55.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.676+0000 7f56797fa640 1 -- 172.21.15.161:0/3316531106 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5670004250 con 0x7f567c0c3190 2024-03-28T23:31:55.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.676+0000 7f56830aa640 1 -- 172.21.15.161:0/3316531106 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f567c1e83d0 con 0x7f567c0c3190 2024-03-28T23:31:55.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.676+0000 7f56797fa640 1 -- 172.21.15.161:0/3316531106 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567003a640 con 0x7f567c0c3190 2024-03-28T23:31:55.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.676+0000 7f56830aa640 1 -- 172.21.15.161:0/3316531106 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f567c0c6260 con 0x7f567c0c3190 2024-03-28T23:31:55.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.677+0000 7f56830aa640 1 -- 172.21.15.161:0/3316531106 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f567c0ab310 con 0x7f567c0c3190 2024-03-28T23:31:55.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.677+0000 7f56797fa640 1 -- 172.21.15.161:0/3316531106 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5670038020 con 0x7f567c0c3190 2024-03-28T23:31:55.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.678+0000 7f56797fa640 1 --2- 172.21.15.161:0/3316531106 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f566806b500 0x7f566806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:55.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.678+0000 7f56797fa640 1 -- 172.21.15.161:0/3316531106 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6957+0+0 (secure 0 0 0) 0x7f56700ac1c0 con 0x7f567c0c3190 2024-03-28T23:31:55.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.678+0000 7f5680e46640 1 --2- 172.21.15.161:0/3316531106 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f566806b500 0x7f566806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:55.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.678+0000 7f5680e46640 1 --2- 172.21.15.161:0/3316531106 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f566806b500 0x7f566806d9c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f566c0011e0 tx=0x7f566c007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:55.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.682+0000 7f56797fa640 1 -- 172.21.15.161:0/3316531106 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f567c0ab310 con 0x7f567c0c3190 2024-03-28T23:31:55.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.897+0000 7f56830aa640 1 -- 172.21.15.161:0/3316531106 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f567c04f4b0 con 0x7f567c0c3190 2024-03-28T23:31:55.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.902+0000 7f56797fa640 1 -- 172.21.15.161:0/3316531106 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"}]=0 v32) ==== 119+0+0 (secure 0 0 0) 0x7f567c1e83d0 con 0x7f567c0c3190 2024-03-28T23:31:55.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.905+0000 7f56830aa640 1 -- 172.21.15.161:0/3316531106 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f566806b500 msgr2=0x7f566806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:55.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.905+0000 7f56830aa640 1 --2- 172.21.15.161:0/3316531106 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f566806b500 0x7f566806d9c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f566c0011e0 tx=0x7f566c007d30 comp rx=0 tx=0).stop 2024-03-28T23:31:55.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.905+0000 7f56830aa640 1 -- 172.21.15.161:0/3316531106 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f567c0c3190 msgr2=0x7f567c153700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:55.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.905+0000 7f56830aa640 1 --2- 172.21.15.161:0/3316531106 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f567c0c3190 0x7f567c153700 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f56700097a0 tx=0x7f56700046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.905+0000 7f56830aa640 1 -- 172.21.15.161:0/3316531106 shutdown_connections 2024-03-28T23:31:55.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.905+0000 7f56830aa640 1 --2- 172.21.15.161:0/3316531106 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f566806b500 0x7f566806d9c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.905+0000 7f56830aa640 1 --2- 172.21.15.161:0/3316531106 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f567c0c3190 0x7f567c153700 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.905+0000 7f56830aa640 1 --2- 172.21.15.161:0/3316531106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f567c1503e0 0x7f567c153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.905+0000 7f56830aa640 1 --2- 172.21.15.161:0/3316531106 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f567c0a9560 0x7f567c14bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:55.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.905+0000 7f56830aa640 1 -- 172.21.15.161:0/3316531106 >> 172.21.15.161:0/3316531106 conn(0x7f567c06ddc0 msgr2=0x7f567c14a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:55.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.905+0000 7f56830aa640 1 -- 172.21.15.161:0/3316531106 shutdown_connections 2024-03-28T23:31:55.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:55.905+0000 7f56830aa640 1 -- 172.21.15.161:0/3316531106 wait complete. 2024-03-28T23:31:55.920 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-28T23:31:55.921 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 allow_standby_replay false 2024-03-28T23:31:56.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.065+0000 7f8f7c13e640 1 Processor -- start 2024-03-28T23:31:56.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.065+0000 7f8f7c13e640 1 -- start start 2024-03-28T23:31:56.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.066+0000 7f8f7c13e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f74152bc0 0x7f8f740c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:56.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.066+0000 7f8f7c13e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8f740c8a80 0x7f8f740c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:56.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.066+0000 7f8f7c13e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f740c6f60 con 0x7f8f740c9420 2024-03-28T23:31:56.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.066+0000 7f8f7c13e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f8f740c6120 con 0x7f8f740c8a80 2024-03-28T23:31:56.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.066+0000 7f8f7c13e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8f740c6400 con 0x7f8f74152bc0 2024-03-28T23:31:56.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.066+0000 7f8f79eda640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f74152bc0 0x7f8f740c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:56.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.066+0000 7f8f796d9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8f740c8a80 0x7f8f740c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:56.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.066+0000 7f8f79eda640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f74152bc0 0x7f8f740c0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:60292/0 (socket says 172.21.15.161:60292) 2024-03-28T23:31:56.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.066+0000 7f8f79eda640 1 -- 172.21.15.161:0/2538797539 learned_addr learned my addr 172.21.15.161:0/2538797539 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:56.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.066+0000 7f8f796d9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8f740c8a80 0x7f8f740c8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:41010/0 (socket says 172.21.15.161:41010) 2024-03-28T23:31:56.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.067+0000 7f8f79eda640 1 -- 172.21.15.161:0/2538797539 >> v2:172.21.15.170:3300/0 conn(0x7f8f740c8a80 msgr2=0x7f8f740c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:56.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.067+0000 7f8f79eda640 1 --2- 172.21.15.161:0/2538797539 >> v2:172.21.15.170:3300/0 conn(0x7f8f740c8a80 0x7f8f740c8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:56.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.067+0000 7f8f79eda640 1 -- 172.21.15.161:0/2538797539 >> v1:172.21.15.170:6789/0 conn(0x7f8f740c9420 legacy=0x7f8f741589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:56.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.067+0000 7f8f79eda640 1 -- 172.21.15.161:0/2538797539 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f740c7240 con 0x7f8f74152bc0 2024-03-28T23:31:56.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.067+0000 7f8f78ed8640 1 -- 172.21.15.161:0/2538797539 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3073700807 0 0) 0x7f8f740c6f60 con 0x7f8f740c9420 2024-03-28T23:31:56.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.067+0000 7f8f796d9640 1 --2- 172.21.15.161:0/2538797539 >> v2:172.21.15.170:3300/0 conn(0x7f8f740c8a80 0x7f8f740c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:31:56.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.067+0000 7f8f79eda640 1 --2- 172.21.15.161:0/2538797539 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f74152bc0 0x7f8f740c0330 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8f70009870 tx=0x7f8f7002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f25528a2206c6f89 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:56.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.068+0000 7f8f78ed8640 1 -- 172.21.15.161:0/2538797539 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f70034070 con 0x7f8f74152bc0 2024-03-28T23:31:56.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.068+0000 7f8f78ed8640 1 -- 172.21.15.161:0/2538797539 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f70030840 con 0x7f8f74152bc0 2024-03-28T23:31:56.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.068+0000 7f8f78ed8640 1 -- 172.21.15.161:0/2538797539 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f70030c10 con 0x7f8f74152bc0 2024-03-28T23:31:56.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.068+0000 7f8f7c13e640 1 -- 172.21.15.161:0/2538797539 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f74152bc0 msgr2=0x7f8f740c0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:56.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.068+0000 7f8f7c13e640 1 --2- 172.21.15.161:0/2538797539 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f74152bc0 0x7f8f740c0330 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8f70009870 tx=0x7f8f7002b980 comp rx=0 tx=0).stop 2024-03-28T23:31:56.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.068+0000 7f8f7c13e640 1 -- 172.21.15.161:0/2538797539 shutdown_connections 2024-03-28T23:31:56.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.068+0000 7f8f7c13e640 1 --2- 172.21.15.161:0/2538797539 >> v2:172.21.15.170:3300/0 conn(0x7f8f740c8a80 0x7f8f740c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:56.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.068+0000 7f8f7c13e640 1 --2- 172.21.15.161:0/2538797539 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f74152bc0 0x7f8f740c0330 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:56.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.068+0000 7f8f7c13e640 1 -- 172.21.15.161:0/2538797539 >> 172.21.15.161:0/2538797539 conn(0x7f8f740caa80 msgr2=0x7f8f740cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:56.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.068+0000 7f8f7c13e640 1 -- 172.21.15.161:0/2538797539 shutdown_connections 2024-03-28T23:31:56.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.068+0000 7f8f7c13e640 1 -- 172.21.15.161:0/2538797539 wait complete. 2024-03-28T23:31:56.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.070+0000 7f8f7c13e640 1 Processor -- start 2024-03-28T23:31:56.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.070+0000 7f8f7c13e640 1 -- start start 2024-03-28T23:31:56.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.071+0000 7f8f7c13e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f740c8a80 0x7f8f74155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:56.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.071+0000 7f8f7c13e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f740c9420 0x7f8f741e7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:56.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.071+0000 7f8f7c13e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f74152bc0 0x7f8f741e7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:56.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.071+0000 7f8f7c13e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8f74152fa0 con 0x7f8f740c8a80 2024-03-28T23:31:56.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.071+0000 7f8f7c13e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8f741590c0 con 0x7f8f74152bc0 2024-03-28T23:31:56.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.071+0000 7f8f7c13e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8f74158140 con 0x7f8f740c9420 2024-03-28T23:31:56.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.071+0000 7f8f79eda640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f740c8a80 0x7f8f74155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:56.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.071+0000 7f8f79eda640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f740c8a80 0x7f8f74155ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35152/0 (socket says 172.21.15.161:35152) 2024-03-28T23:31:56.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.071+0000 7f8f79eda640 1 -- 172.21.15.161:0/2956630948 learned_addr learned my addr 172.21.15.161:0/2956630948 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:56.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.072+0000 7f8f796d9640 1 --2- 172.21.15.161:0/2956630948 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f740c9420 0x7f8f741e7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:56.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.072+0000 7f8f7a6db640 1 --2- 172.21.15.161:0/2956630948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f74152bc0 0x7f8f741e7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:56.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.072+0000 7f8f79eda640 1 -- 172.21.15.161:0/2956630948 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f740c9420 msgr2=0x7f8f741e7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:56.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.072+0000 7f8f79eda640 1 --2- 172.21.15.161:0/2956630948 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f740c9420 0x7f8f741e7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:56.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.072+0000 7f8f79eda640 1 -- 172.21.15.161:0/2956630948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f74152bc0 msgr2=0x7f8f741e7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:56.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.072+0000 7f8f79eda640 1 --2- 172.21.15.161:0/2956630948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f74152bc0 0x7f8f741e7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:56.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.072+0000 7f8f79eda640 1 -- 172.21.15.161:0/2956630948 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f74158b20 con 0x7f8f740c8a80 2024-03-28T23:31:56.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.072+0000 7f8f79eda640 1 --2- 172.21.15.161:0/2956630948 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f740c8a80 0x7f8f74155ba0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f8f700090b0 tx=0x7f8f700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:56.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.072+0000 7f8f6a7fc640 1 -- 172.21.15.161:0/2956630948 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f70034040 con 0x7f8f740c8a80 2024-03-28T23:31:56.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.072+0000 7f8f6a7fc640 1 -- 172.21.15.161:0/2956630948 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f7003ab90 con 0x7f8f740c8a80 2024-03-28T23:31:56.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.072+0000 7f8f6a7fc640 1 -- 172.21.15.161:0/2956630948 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f70004030 con 0x7f8f740c8a80 2024-03-28T23:31:56.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.072+0000 7f8f7c13e640 1 -- 172.21.15.161:0/2956630948 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f74158e00 con 0x7f8f740c8a80 2024-03-28T23:31:56.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.072+0000 7f8f7c13e640 1 -- 172.21.15.161:0/2956630948 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f7414c4e0 con 0x7f8f740c8a80 2024-03-28T23:31:56.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.074+0000 7f8f7c13e640 1 -- 172.21.15.161:0/2956630948 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f7404f4b0 con 0x7f8f740c8a80 2024-03-28T23:31:56.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.074+0000 7f8f6a7fc640 1 -- 172.21.15.161:0/2956630948 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8f7004b060 con 0x7f8f740c8a80 2024-03-28T23:31:56.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.074+0000 7f8f6a7fc640 1 --2- 172.21.15.161:0/2956630948 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8f5806b4d0 0x7f8f5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:56.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.074+0000 7f8f6a7fc640 1 -- 172.21.15.161:0/2956630948 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6957+0+0 (secure 0 0 0) 0x7f8f700ac300 con 0x7f8f740c8a80 2024-03-28T23:31:56.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.074+0000 7f8f796d9640 1 --2- 172.21.15.161:0/2956630948 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8f5806b4d0 0x7f8f5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:56.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.075+0000 7f8f796d9640 1 --2- 172.21.15.161:0/2956630948 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8f5806b4d0 0x7f8f5806d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8f64009560 tx=0x7f8f64009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:56.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.078+0000 7f8f6a7fc640 1 -- 172.21.15.161:0/2956630948 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8f7003f070 con 0x7f8f740c8a80 2024-03-28T23:31:56.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.346+0000 7f8f7c13e640 1 -- 172.21.15.161:0/2956630948 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f8f74153ea0 con 0x7f8f740c8a80 2024-03-28T23:31:56.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.907+0000 7f8f6a7fc640 1 -- 172.21.15.161:0/2956630948 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"}]=0 v33) ==== 123+0+0 (secure 0 0 0) 0x7f8f7007a800 con 0x7f8f740c8a80 2024-03-28T23:31:56.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.910+0000 7f8f7c13e640 1 -- 172.21.15.161:0/2956630948 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8f5806b4d0 msgr2=0x7f8f5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:56.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.910+0000 7f8f7c13e640 1 --2- 172.21.15.161:0/2956630948 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8f5806b4d0 0x7f8f5806d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8f64009560 tx=0x7f8f64009190 comp rx=0 tx=0).stop 2024-03-28T23:31:56.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.910+0000 7f8f7c13e640 1 -- 172.21.15.161:0/2956630948 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f740c8a80 msgr2=0x7f8f74155ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:56.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.910+0000 7f8f7c13e640 1 --2- 172.21.15.161:0/2956630948 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f740c8a80 0x7f8f74155ba0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f8f700090b0 tx=0x7f8f700046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:56.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.910+0000 7f8f7c13e640 1 -- 172.21.15.161:0/2956630948 shutdown_connections 2024-03-28T23:31:56.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.910+0000 7f8f7c13e640 1 --2- 172.21.15.161:0/2956630948 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8f5806b4d0 0x7f8f5806d990 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:56.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.910+0000 7f8f7c13e640 1 --2- 172.21.15.161:0/2956630948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f74152bc0 0x7f8f741e7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:56.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.910+0000 7f8f7c13e640 1 --2- 172.21.15.161:0/2956630948 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f740c9420 0x7f8f741e7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:56.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.910+0000 7f8f7c13e640 1 --2- 172.21.15.161:0/2956630948 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f740c8a80 0x7f8f74155ba0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:56.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.910+0000 7f8f7c13e640 1 -- 172.21.15.161:0/2956630948 >> 172.21.15.161:0/2956630948 conn(0x7f8f740caa80 msgr2=0x7f8f740cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:56.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.910+0000 7f8f7c13e640 1 -- 172.21.15.161:0/2956630948 shutdown_connections 2024-03-28T23:31:56.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:56.910+0000 7f8f7c13e640 1 -- 172.21.15.161:0/2956630948 wait complete. 2024-03-28T23:31:56.922 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs2' 2024-03-28T23:31:56.922 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:31:57.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.022+0000 7fec68447640 1 Processor -- start 2024-03-28T23:31:57.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.022+0000 7fec68447640 1 -- start start 2024-03-28T23:31:57.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.023+0000 7fec68447640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fec60152890 0x7fec600c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:57.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.023+0000 7fec68447640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fec600c8ed0 0x7fec600c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:57.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.023+0000 7fec68447640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec60158a30 con 0x7fec600c5950 2024-03-28T23:31:57.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.023+0000 7fec68447640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fec600c8480 con 0x7fec60152890 2024-03-28T23:31:57.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.023+0000 7fec68447640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fec60159640 con 0x7fec600c8ed0 2024-03-28T23:31:57.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.023+0000 7fec661e3640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fec60152890 0x7fec600c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:57.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.023+0000 7fec661e3640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fec60152890 0x7fec600c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35166/0 (socket says 172.21.15.161:35166) 2024-03-28T23:31:57.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.023+0000 7fec661e3640 1 -- 172.21.15.161:0/3584713645 learned_addr learned my addr 172.21.15.161:0/3584713645 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:57.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.023+0000 7fec659e2640 1 --2- 172.21.15.161:0/3584713645 >> v2:172.21.15.170:3300/0 conn(0x7fec600c8ed0 0x7fec600c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:57.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.023+0000 7fec661e3640 1 -- 172.21.15.161:0/3584713645 >> v2:172.21.15.170:3300/0 conn(0x7fec600c8ed0 msgr2=0x7fec600c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:57.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.023+0000 7fec661e3640 1 --2- 172.21.15.161:0/3584713645 >> v2:172.21.15.170:3300/0 conn(0x7fec600c8ed0 0x7fec600c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.023+0000 7fec661e3640 1 -- 172.21.15.161:0/3584713645 >> v1:172.21.15.170:6789/0 conn(0x7fec600c5950 legacy=0x7fec601553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:57.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.023+0000 7fec661e3640 1 -- 172.21.15.161:0/3584713645 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec60158d10 con 0x7fec60152890 2024-03-28T23:31:57.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.023+0000 7fec659e2640 1 --2- 172.21.15.161:0/3584713645 >> v2:172.21.15.170:3300/0 conn(0x7fec600c8ed0 0x7fec600c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:57.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.024+0000 7fec661e3640 1 --2- 172.21.15.161:0/3584713645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec60152890 0x7fec600c8990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fec600ad6e0 tx=0x7fec5c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd52f498cd286254 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:57.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.024+0000 7fec651e1640 1 -- 172.21.15.161:0/3584713645 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5c02e070 con 0x7fec60152890 2024-03-28T23:31:57.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.024+0000 7fec651e1640 1 -- 172.21.15.161:0/3584713645 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec5c037070 con 0x7fec60152890 2024-03-28T23:31:57.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.024+0000 7fec651e1640 1 -- 172.21.15.161:0/3584713645 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5c031a00 con 0x7fec60152890 2024-03-28T23:31:57.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.025+0000 7fec68447640 1 -- 172.21.15.161:0/3584713645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec60152890 msgr2=0x7fec600c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:57.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.025+0000 7fec68447640 1 --2- 172.21.15.161:0/3584713645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec60152890 0x7fec600c8990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fec600ad6e0 tx=0x7fec5c02b880 comp rx=0 tx=0).stop 2024-03-28T23:31:57.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.025+0000 7fec68447640 1 -- 172.21.15.161:0/3584713645 shutdown_connections 2024-03-28T23:31:57.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.025+0000 7fec68447640 1 --2- 172.21.15.161:0/3584713645 >> v2:172.21.15.170:3300/0 conn(0x7fec600c8ed0 0x7fec600c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.025+0000 7fec68447640 1 --2- 172.21.15.161:0/3584713645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec60152890 0x7fec600c8990 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.025+0000 7fec68447640 1 -- 172.21.15.161:0/3584713645 >> 172.21.15.161:0/3584713645 conn(0x7fec600ac620 msgr2=0x7fec600aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:57.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.025+0000 7fec68447640 1 -- 172.21.15.161:0/3584713645 shutdown_connections 2024-03-28T23:31:57.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.025+0000 7fec68447640 1 -- 172.21.15.161:0/3584713645 wait complete. 2024-03-28T23:31:57.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.026+0000 7fec68447640 1 Processor -- start 2024-03-28T23:31:57.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.026+0000 7fec68447640 1 -- start start 2024-03-28T23:31:57.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.027+0000 7fec68447640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec600c5950 0x7fec601eb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:57.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.027+0000 7fec68447640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec600c8ed0 0x7fec600c43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:57.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.027+0000 7fec68447640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec60152890 0x7fec600befd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:57.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.027+0000 7fec68447640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fec60152c70 con 0x7fec600c8ed0 2024-03-28T23:31:57.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.027+0000 7fec68447640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fec601592d0 con 0x7fec60152890 2024-03-28T23:31:57.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.027+0000 7fec68447640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fec60152df0 con 0x7fec600c5950 2024-03-28T23:31:57.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.027+0000 7fec659e2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec600c8ed0 0x7fec600c43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:57.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.027+0000 7fec659e2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec600c8ed0 0x7fec600c43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35170/0 (socket says 172.21.15.161:35170) 2024-03-28T23:31:57.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.027+0000 7fec659e2640 1 -- 172.21.15.161:0/945625094 learned_addr learned my addr 172.21.15.161:0/945625094 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:57.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.027+0000 7fec661e3640 1 --2- 172.21.15.161:0/945625094 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec600c5950 0x7fec601eb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:57.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.027+0000 7fec669e4640 1 --2- 172.21.15.161:0/945625094 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec60152890 0x7fec600befd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:57.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.027+0000 7fec659e2640 1 -- 172.21.15.161:0/945625094 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec600c5950 msgr2=0x7fec601eb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:57.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.027+0000 7fec659e2640 1 --2- 172.21.15.161:0/945625094 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec600c5950 0x7fec601eb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.027+0000 7fec659e2640 1 -- 172.21.15.161:0/945625094 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec60152890 msgr2=0x7fec600befd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:57.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.027+0000 7fec659e2640 1 --2- 172.21.15.161:0/945625094 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec60152890 0x7fec600befd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.027+0000 7fec659e2640 1 -- 172.21.15.161:0/945625094 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec600c4ac0 con 0x7fec600c8ed0 2024-03-28T23:31:57.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.027+0000 7fec659e2640 1 --2- 172.21.15.161:0/945625094 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec600c8ed0 0x7fec600c43e0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fec5000e810 tx=0x7fec5000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:57.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.027+0000 7fec56ffd640 1 -- 172.21.15.161:0/945625094 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5000f040 con 0x7fec600c8ed0 2024-03-28T23:31:57.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.027+0000 7fec56ffd640 1 -- 172.21.15.161:0/945625094 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec50005990 con 0x7fec600c8ed0 2024-03-28T23:31:57.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.027+0000 7fec56ffd640 1 -- 172.21.15.161:0/945625094 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec50005d60 con 0x7fec600c8ed0 2024-03-28T23:31:57.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.027+0000 7fec68447640 1 -- 172.21.15.161:0/945625094 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec600c4da0 con 0x7fec600c8ed0 2024-03-28T23:31:57.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.028+0000 7fec68447640 1 -- 172.21.15.161:0/945625094 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec601eb0c0 con 0x7fec600c8ed0 2024-03-28T23:31:57.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.029+0000 7fec68447640 1 -- 172.21.15.161:0/945625094 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec60067a50 con 0x7fec600c8ed0 2024-03-28T23:31:57.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.029+0000 7fec56ffd640 1 -- 172.21.15.161:0/945625094 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fec5000c6d0 con 0x7fec600c8ed0 2024-03-28T23:31:57.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.029+0000 7fec56ffd640 1 --2- 172.21.15.161:0/945625094 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fec4406b4d0 0x7fec4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:57.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.029+0000 7fec56ffd640 1 -- 172.21.15.161:0/945625094 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6957+0+0 (secure 0 0 0) 0x7fec5008a390 con 0x7fec600c8ed0 2024-03-28T23:31:57.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.030+0000 7fec661e3640 1 --2- 172.21.15.161:0/945625094 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fec4406b4d0 0x7fec4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:57.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.030+0000 7fec661e3640 1 --2- 172.21.15.161:0/945625094 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fec4406b4d0 0x7fec4406d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fec5c036000 tx=0x7fec5c03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:57.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.034+0000 7fec56ffd640 1 -- 172.21.15.161:0/945625094 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fec60067a50 con 0x7fec600c8ed0 2024-03-28T23:31:57.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.252+0000 7fec68447640 1 -- 172.21.15.161:0/945625094 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec6004f4b0 con 0x7fec600c8ed0 2024-03-28T23:31:57.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.253+0000 7fec56ffd640 1 -- 172.21.15.161:0/945625094 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6504 (secure 0 0 0) 0x7fec50017020 con 0x7fec600c8ed0 2024-03-28T23:31:57.255 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:57.255 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":33,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5186,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.170:6835/1945122204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1945122204},{"type":"v1","addr":"172.21.15.170:6835","nonce":1945122204}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5542,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/3666899516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3666899516},{"type":"v1","addr":"172.21.15.170:6833","nonce":3666899516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:31:39.725625+0000","modified":"2024-03-28T23:31:41.818149+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5820},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5820":{"gid":5820,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":4,"addr":"172.21.15.161:6837/544971001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":544971001},{"type":"v1","addr":"172.21.15.161:6837","nonce":544971001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5820,"qdb_cluster":[5820]},"id":3},{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:31:54.226424+0000","modified":"2024-03-28T23:31:56.347601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5817},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5817":{"gid":5817,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":8,"addr":"172.21.15.161:6836/1300583128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1300583128},{"type":"v1","addr":"172.21.15.161:6836","nonce":1300583128}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5817,"qdb_cluster":[5817]},"id":4}]} 2024-03-28T23:31:57.255 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 33 2024-03-28T23:31:57.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.256+0000 7fec68447640 1 -- 172.21.15.161:0/945625094 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fec4406b4d0 msgr2=0x7fec4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:57.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.256+0000 7fec68447640 1 --2- 172.21.15.161:0/945625094 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fec4406b4d0 0x7fec4406d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fec5c036000 tx=0x7fec5c03d040 comp rx=0 tx=0).stop 2024-03-28T23:31:57.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.256+0000 7fec68447640 1 -- 172.21.15.161:0/945625094 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec600c8ed0 msgr2=0x7fec600c43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:57.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.256+0000 7fec68447640 1 --2- 172.21.15.161:0/945625094 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec600c8ed0 0x7fec600c43e0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fec5000e810 tx=0x7fec5000ecd0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.256+0000 7fec68447640 1 -- 172.21.15.161:0/945625094 shutdown_connections 2024-03-28T23:31:57.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.256+0000 7fec68447640 1 --2- 172.21.15.161:0/945625094 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fec4406b4d0 0x7fec4406d990 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.256+0000 7fec68447640 1 --2- 172.21.15.161:0/945625094 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec600c8ed0 0x7fec600c43e0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.256+0000 7fec68447640 1 --2- 172.21.15.161:0/945625094 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec60152890 0x7fec600befd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.256+0000 7fec68447640 1 --2- 172.21.15.161:0/945625094 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec600c5950 0x7fec601eb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.256+0000 7fec68447640 1 -- 172.21.15.161:0/945625094 >> 172.21.15.161:0/945625094 conn(0x7fec600ac620 msgr2=0x7fec6006c890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:57.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.256+0000 7fec68447640 1 -- 172.21.15.161:0/945625094 shutdown_connections 2024-03-28T23:31:57.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.256+0000 7fec68447640 1 -- 172.21.15.161:0/945625094 wait complete. 2024-03-28T23:31:57.267 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:31:57.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.372+0000 7f3ecce0e640 1 Processor -- start 2024-03-28T23:31:57.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.372+0000 7f3ecce0e640 1 -- start start 2024-03-28T23:31:57.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.373+0000 7f3ecce0e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3ec8152890 0x7f3ec80c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:57.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.373+0000 7f3ecce0e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ec80c5b20 0x7f3ec81553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:57.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.373+0000 7f3ecce0e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ec8158a30 con 0x7f3ec80c8ed0 2024-03-28T23:31:57.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.373+0000 7f3ecce0e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f3ec80c8480 con 0x7f3ec8152890 2024-03-28T23:31:57.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.373+0000 7f3ecce0e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3ec8159640 con 0x7f3ec80c5b20 2024-03-28T23:31:57.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.373+0000 7f3ec659c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3ec8152890 0x7f3ec80c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:57.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.374+0000 7f3ec659c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3ec8152890 0x7f3ec80c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35180/0 (socket says 172.21.15.161:35180) 2024-03-28T23:31:57.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.374+0000 7f3ec659c640 1 -- 172.21.15.161:0/1390511116 learned_addr learned my addr 172.21.15.161:0/1390511116 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:57.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.374+0000 7f3ec659c640 1 -- 172.21.15.161:0/1390511116 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ec80c5b20 msgr2=0x7f3ec81553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:57.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.374+0000 7f3ec6d9d640 1 --2- 172.21.15.161:0/1390511116 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ec80c5b20 0x7f3ec81553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:57.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.374+0000 7f3ec659c640 1 --2- 172.21.15.161:0/1390511116 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ec80c5b20 0x7f3ec81553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.374+0000 7f3ec659c640 1 -- 172.21.15.161:0/1390511116 >> v1:172.21.15.170:6789/0 conn(0x7f3ec80c8ed0 legacy=0x7f3ec80c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:57.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.374+0000 7f3ec659c640 1 -- 172.21.15.161:0/1390511116 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ec8158d10 con 0x7f3ec8152890 2024-03-28T23:31:57.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.374+0000 7f3ec6d9d640 1 --2- 172.21.15.161:0/1390511116 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ec80c5b20 0x7f3ec81553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:57.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.375+0000 7f3ec659c640 1 --2- 172.21.15.161:0/1390511116 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ec8152890 0x7f3ec80c8990 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f3ec81597c0 tx=0x7f3eb802b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=290e71c64266660 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:57.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.375+0000 7f3ec5d9b640 1 -- 172.21.15.161:0/1390511116 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8038070 con 0x7f3ec8152890 2024-03-28T23:31:57.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.375+0000 7f3ec5d9b640 1 -- 172.21.15.161:0/1390511116 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3eb8031520 con 0x7f3ec8152890 2024-03-28T23:31:57.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.375+0000 7f3ec5d9b640 1 -- 172.21.15.161:0/1390511116 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb80318f0 con 0x7f3ec8152890 2024-03-28T23:31:57.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.376+0000 7f3ecce0e640 1 -- 172.21.15.161:0/1390511116 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ec8152890 msgr2=0x7f3ec80c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:57.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.376+0000 7f3ecce0e640 1 --2- 172.21.15.161:0/1390511116 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ec8152890 0x7f3ec80c8990 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f3ec81597c0 tx=0x7f3eb802b7c0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.376+0000 7f3ecce0e640 1 -- 172.21.15.161:0/1390511116 shutdown_connections 2024-03-28T23:31:57.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.376+0000 7f3ecce0e640 1 --2- 172.21.15.161:0/1390511116 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ec80c5b20 0x7f3ec81553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.376+0000 7f3ecce0e640 1 --2- 172.21.15.161:0/1390511116 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ec8152890 0x7f3ec80c8990 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.376+0000 7f3ecce0e640 1 -- 172.21.15.161:0/1390511116 >> 172.21.15.161:0/1390511116 conn(0x7f3ec80ac620 msgr2=0x7f3ec80aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:57.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.376+0000 7f3ecce0e640 1 -- 172.21.15.161:0/1390511116 shutdown_connections 2024-03-28T23:31:57.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.376+0000 7f3ecce0e640 1 -- 172.21.15.161:0/1390511116 wait complete. 2024-03-28T23:31:57.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.377+0000 7f3ecce0e640 1 Processor -- start 2024-03-28T23:31:57.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.377+0000 7f3ecce0e640 1 -- start start 2024-03-28T23:31:57.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.378+0000 7f3ecce0e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ec80c5b20 0x7f3ec80c36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:57.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.378+0000 7f3ecce0e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ec80c8ed0 0x7f3ec8149010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:57.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.378+0000 7f3ecce0e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ec8152890 0x7f3ec80bfe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:57.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.378+0000 7f3ecce0e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec8152c70 con 0x7f3ec80c5b20 2024-03-28T23:31:57.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.378+0000 7f3ecce0e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec81592d0 con 0x7f3ec8152890 2024-03-28T23:31:57.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.378+0000 7f3ecce0e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3ec8152df0 con 0x7f3ec80c8ed0 2024-03-28T23:31:57.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.378+0000 7f3ec659c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ec80c5b20 0x7f3ec80c36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:57.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.378+0000 7f3ec659c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ec80c5b20 0x7f3ec80c36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35194/0 (socket says 172.21.15.161:35194) 2024-03-28T23:31:57.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.378+0000 7f3ec659c640 1 -- 172.21.15.161:0/3241189943 learned_addr learned my addr 172.21.15.161:0/3241189943 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:57.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.378+0000 7f3ec6d9d640 1 --2- 172.21.15.161:0/3241189943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ec8152890 0x7f3ec80bfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:57.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.378+0000 7f3ebdd9b640 1 --2- 172.21.15.161:0/3241189943 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ec80c8ed0 0x7f3ec8149010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:57.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.378+0000 7f3ec659c640 1 -- 172.21.15.161:0/3241189943 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ec80c8ed0 msgr2=0x7f3ec8149010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:57.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.378+0000 7f3ec659c640 1 --2- 172.21.15.161:0/3241189943 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ec80c8ed0 0x7f3ec8149010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.378+0000 7f3ec659c640 1 -- 172.21.15.161:0/3241189943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ec8152890 msgr2=0x7f3ec80bfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:57.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.378+0000 7f3ec659c640 1 --2- 172.21.15.161:0/3241189943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ec8152890 0x7f3ec80bfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.378+0000 7f3ec659c640 1 -- 172.21.15.161:0/3241189943 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ec80c4cf0 con 0x7f3ec80c5b20 2024-03-28T23:31:57.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.378+0000 7f3ec659c640 1 --2- 172.21.15.161:0/3241189943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ec80c5b20 0x7f3ec80c36d0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8000c00 tx=0x7f3eb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:57.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.379+0000 7f3ebf7fe640 1 -- 172.21.15.161:0/3241189943 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8042070 con 0x7f3ec80c5b20 2024-03-28T23:31:57.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.379+0000 7f3ebf7fe640 1 -- 172.21.15.161:0/3241189943 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3eb803d890 con 0x7f3ec80c5b20 2024-03-28T23:31:57.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.379+0000 7f3ebf7fe640 1 -- 172.21.15.161:0/3241189943 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8038040 con 0x7f3ec80c5b20 2024-03-28T23:31:57.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.379+0000 7f3ecce0e640 1 -- 172.21.15.161:0/3241189943 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ec80c2580 con 0x7f3ec80c5b20 2024-03-28T23:31:57.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.379+0000 7f3ecce0e640 1 -- 172.21.15.161:0/3241189943 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ec814f910 con 0x7f3ec80c5b20 2024-03-28T23:31:57.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.380+0000 7f3ecce0e640 1 -- 172.21.15.161:0/3241189943 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ec80c2370 con 0x7f3ec80c5b20 2024-03-28T23:31:57.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.381+0000 7f3ebf7fe640 1 -- 172.21.15.161:0/3241189943 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3eb8045070 con 0x7f3ec80c5b20 2024-03-28T23:31:57.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.381+0000 7f3ebf7fe640 1 --2- 172.21.15.161:0/3241189943 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3ea806b4d0 0x7f3ea806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:57.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.381+0000 7f3ebf7fe640 1 -- 172.21.15.161:0/3241189943 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6957+0+0 (secure 0 0 0) 0x7f3eb80ac1e0 con 0x7f3ec80c5b20 2024-03-28T23:31:57.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.381+0000 7f3ebdd9b640 1 --2- 172.21.15.161:0/3241189943 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3ea806b4d0 0x7f3ea806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:57.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.382+0000 7f3ebdd9b640 1 --2- 172.21.15.161:0/3241189943 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3ea806b4d0 0x7f3ea806d990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80aaec0 tx=0x7f3eb0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:57.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.387+0000 7f3ebf7fe640 1 -- 172.21.15.161:0/3241189943 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3eb807ab80 con 0x7f3ec80c5b20 2024-03-28T23:31:57.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.735+0000 7f3ecce0e640 1 -- 172.21.15.161:0/3241189943 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3ec804f4b0 con 0x7f3ec80c5b20 2024-03-28T23:31:57.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.737+0000 7f3ebf7fe640 1 -- 172.21.15.161:0/3241189943 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+20702 (secure 0 0 0) 0x7f3eb807a480 con 0x7f3ec80c5b20 2024-03-28T23:31:57.738 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:57.738 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":55,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:31:55.899578+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:31:37.940838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-28T23:31:38.692659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:31:52.774315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:31:53.201664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:31:57.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.739+0000 7f3ecce0e640 1 -- 172.21.15.161:0/3241189943 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3ea806b4d0 msgr2=0x7f3ea806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:57.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.739+0000 7f3ecce0e640 1 --2- 172.21.15.161:0/3241189943 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3ea806b4d0 0x7f3ea806d990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80aaec0 tx=0x7f3eb0009040 comp rx=0 tx=0).stop 2024-03-28T23:31:57.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.739+0000 7f3ecce0e640 1 -- 172.21.15.161:0/3241189943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ec80c5b20 msgr2=0x7f3ec80c36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:57.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.739+0000 7f3ecce0e640 1 --2- 172.21.15.161:0/3241189943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ec80c5b20 0x7f3ec80c36d0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8000c00 tx=0x7f3eb80046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.739+0000 7f3ecce0e640 1 -- 172.21.15.161:0/3241189943 shutdown_connections 2024-03-28T23:31:57.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.739+0000 7f3ecce0e640 1 --2- 172.21.15.161:0/3241189943 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3ea806b4d0 0x7f3ea806d990 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.739+0000 7f3ecce0e640 1 --2- 172.21.15.161:0/3241189943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ec8152890 0x7f3ec80bfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.739+0000 7f3ecce0e640 1 --2- 172.21.15.161:0/3241189943 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ec80c8ed0 0x7f3ec8149010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.739+0000 7f3ecce0e640 1 --2- 172.21.15.161:0/3241189943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ec80c5b20 0x7f3ec80c36d0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.739+0000 7f3ecce0e640 1 -- 172.21.15.161:0/3241189943 >> 172.21.15.161:0/3241189943 conn(0x7f3ec80ac620 msgr2=0x7f3ec8150ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:57.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.739+0000 7f3ecce0e640 1 -- 172.21.15.161:0/3241189943 shutdown_connections 2024-03-28T23:31:57.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.739+0000 7f3ecce0e640 1 -- 172.21.15.161:0/3241189943 wait complete. 2024-03-28T23:31:57.751 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T23:31:57.751 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:31:57.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.853+0000 7f791ab7a640 1 Processor -- start 2024-03-28T23:31:57.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.853+0000 7f791ab7a640 1 -- start start 2024-03-28T23:31:57.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.854+0000 7f791ab7a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7914152890 0x7f79140c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:57.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.854+0000 7f791ab7a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f79140c8ed0 0x7f79140c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:57.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.854+0000 7f791ab7a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7914158a30 con 0x7f79140c5950 2024-03-28T23:31:57.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.854+0000 7f791ab7a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f79140c5db0 con 0x7f79140c8ed0 2024-03-28T23:31:57.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.854+0000 7f791ab7a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f79140c8480 con 0x7f7914152890 2024-03-28T23:31:57.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.854+0000 7f7919117640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f79140c5950 0x7f79141553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37810/0 (socket says 172.21.15.161:37810) 2024-03-28T23:31:57.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.854+0000 7f7919117640 1 -- 172.21.15.161:0/2336717383 learned_addr learned my addr 172.21.15.161:0/2336717383 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:57.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.854+0000 7f7918916640 1 --2- 172.21.15.161:0/2336717383 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7914152890 0x7f79140c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:57.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.854+0000 7f7913fff640 1 --2- 172.21.15.161:0/2336717383 >> v2:172.21.15.170:3300/0 conn(0x7f79140c8ed0 0x7f79140c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:57.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.854+0000 7f79137fe640 1 -- 172.21.15.161:0/2336717383 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1290964819 0 0) 0x7f7914158a30 con 0x7f79140c5950 2024-03-28T23:31:57.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.854+0000 7f79137fe640 1 -- 172.21.15.161:0/2336717383 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7904003520 con 0x7f79140c5950 2024-03-28T23:31:57.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.854+0000 7f7913fff640 1 -- 172.21.15.161:0/2336717383 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7914152890 msgr2=0x7f79140c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:57.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.854+0000 7f7913fff640 1 --2- 172.21.15.161:0/2336717383 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7914152890 0x7f79140c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.854+0000 7f7913fff640 1 -- 172.21.15.161:0/2336717383 >> v1:172.21.15.161:6789/0 conn(0x7f79140c5950 legacy=0x7f79141553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:57.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.854+0000 7f7913fff640 1 -- 172.21.15.161:0/2336717383 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7914158d10 con 0x7f79140c8ed0 2024-03-28T23:31:57.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.855+0000 7f7913fff640 1 --2- 172.21.15.161:0/2336717383 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79140c8ed0 0x7f79140c5410 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7900009790 tx=0x7f790002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73bd5b62da81d295 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:57.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.856+0000 7f79137fe640 1 -- 172.21.15.161:0/2336717383 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7900039070 con 0x7f79140c8ed0 2024-03-28T23:31:57.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.856+0000 7f79137fe640 1 -- 172.21.15.161:0/2336717383 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79000316f0 con 0x7f79140c8ed0 2024-03-28T23:31:57.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.856+0000 7f79137fe640 1 -- 172.21.15.161:0/2336717383 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7900031ac0 con 0x7f79140c8ed0 2024-03-28T23:31:57.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.856+0000 7f791ab7a640 1 -- 172.21.15.161:0/2336717383 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79140c8ed0 msgr2=0x7f79140c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:57.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.856+0000 7f791ab7a640 1 --2- 172.21.15.161:0/2336717383 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79140c8ed0 0x7f79140c5410 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7900009790 tx=0x7f790002b790 comp rx=0 tx=0).stop 2024-03-28T23:31:57.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.856+0000 7f791ab7a640 1 -- 172.21.15.161:0/2336717383 shutdown_connections 2024-03-28T23:31:57.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.856+0000 7f791ab7a640 1 --2- 172.21.15.161:0/2336717383 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79140c8ed0 0x7f79140c5410 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.856+0000 7f791ab7a640 1 --2- 172.21.15.161:0/2336717383 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7914152890 0x7f79140c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.856+0000 7f791ab7a640 1 -- 172.21.15.161:0/2336717383 >> 172.21.15.161:0/2336717383 conn(0x7f79140ac620 msgr2=0x7f79140aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:57.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.856+0000 7f791ab7a640 1 -- 172.21.15.161:0/2336717383 shutdown_connections 2024-03-28T23:31:57.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.856+0000 7f791ab7a640 1 -- 172.21.15.161:0/2336717383 wait complete. 2024-03-28T23:31:57.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.857+0000 7f791ab7a640 1 Processor -- start 2024-03-28T23:31:57.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.857+0000 7f791ab7a640 1 -- start start 2024-03-28T23:31:57.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.858+0000 7f791ab7a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79140c5950 0x7f79140c31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:57.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.858+0000 7f791ab7a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79140c8ed0 0x7f79141ee840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:57.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.858+0000 7f791ab7a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7914152890 0x7f7914149b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:57.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.858+0000 7f791ab7a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f79141592d0 con 0x7f79140c8ed0 2024-03-28T23:31:57.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.858+0000 7f791ab7a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7914159640 con 0x7f7914152890 2024-03-28T23:31:57.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.858+0000 7f791ab7a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7914152c70 con 0x7f79140c5950 2024-03-28T23:31:57.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.858+0000 7f7913fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79140c8ed0 0x7f79141ee840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:57.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.858+0000 7f7913fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79140c8ed0 0x7f79141ee840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35204/0 (socket says 172.21.15.161:35204) 2024-03-28T23:31:57.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.858+0000 7f7913fff640 1 -- 172.21.15.161:0/1179482244 learned_addr learned my addr 172.21.15.161:0/1179482244 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:57.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.859+0000 7f7918916640 1 --2- 172.21.15.161:0/1179482244 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79140c5950 0x7f79140c31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:57.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.859+0000 7f7919117640 1 --2- 172.21.15.161:0/1179482244 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7914152890 0x7f7914149b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:57.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.859+0000 7f7913fff640 1 -- 172.21.15.161:0/1179482244 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79140c5950 msgr2=0x7f79140c31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:57.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.859+0000 7f7913fff640 1 --2- 172.21.15.161:0/1179482244 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79140c5950 0x7f79140c31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.859+0000 7f7913fff640 1 -- 172.21.15.161:0/1179482244 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7914152890 msgr2=0x7f7914149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:57.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.859+0000 7f7913fff640 1 --2- 172.21.15.161:0/1179482244 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7914152890 0x7f7914149b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:57.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.859+0000 7f7913fff640 1 -- 172.21.15.161:0/1179482244 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79141eed80 con 0x7f79140c8ed0 2024-03-28T23:31:57.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.859+0000 7f7919117640 1 --2- 172.21.15.161:0/1179482244 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7914152890 0x7f7914149b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:57.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.859+0000 7f7913fff640 1 --2- 172.21.15.161:0/1179482244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79140c8ed0 0x7f79141ee840 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f7900000c00 tx=0x7f79000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:57.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.859+0000 7f79117fa640 1 -- 172.21.15.161:0/1179482244 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7900043070 con 0x7f79140c8ed0 2024-03-28T23:31:57.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.859+0000 7f79117fa640 1 -- 172.21.15.161:0/1179482244 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7900034480 con 0x7f79140c8ed0 2024-03-28T23:31:57.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.859+0000 7f791ab7a640 1 -- 172.21.15.161:0/1179482244 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79140c1bd0 con 0x7f79140c8ed0 2024-03-28T23:31:57.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.859+0000 7f79117fa640 1 -- 172.21.15.161:0/1179482244 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7900039040 con 0x7f79140c8ed0 2024-03-28T23:31:57.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.859+0000 7f791ab7a640 1 -- 172.21.15.161:0/1179482244 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f791414d1d0 con 0x7f79140c8ed0 2024-03-28T23:31:57.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.860+0000 7f791ab7a640 1 -- 172.21.15.161:0/1179482244 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f791404f4b0 con 0x7f79140c8ed0 2024-03-28T23:31:57.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.861+0000 7f79117fa640 1 -- 172.21.15.161:0/1179482244 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7900046070 con 0x7f79140c8ed0 2024-03-28T23:31:57.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.861+0000 7f79117fa640 1 --2- 172.21.15.161:0/1179482244 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f78f806b4d0 0x7f78f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:57.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.861+0000 7f79117fa640 1 -- 172.21.15.161:0/1179482244 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6957+0+0 (secure 0 0 0) 0x7f79000ac150 con 0x7f79140c8ed0 2024-03-28T23:31:57.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.861+0000 7f7918916640 1 --2- 172.21.15.161:0/1179482244 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f78f806b4d0 0x7f78f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:57.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.862+0000 7f7918916640 1 --2- 172.21.15.161:0/1179482244 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f78f806b4d0 0x7f78f806d990 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7908009ad0 tx=0x7f7908008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:57.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:57.866+0000 7f79117fa640 1 -- 172.21.15.161:0/1179482244 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f79141eed80 con 0x7f79140c8ed0 2024-03-28T23:31:58.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.043+0000 7f791ab7a640 1 -- 172.21.15.161:0/1179482244 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f79140c88b0 con 0x7f78f806b4d0 2024-03-28T23:31:58.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.055+0000 7f79117fa640 1 -- 172.21.15.161:0/1179482244 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+485262 (secure 0 0 0) 0x7f79140c88b0 con 0x7f78f806b4d0 2024-03-28T23:31:58.060 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:58.064 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:31:58.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.059+0000 7f791ab7a640 1 -- 172.21.15.161:0/1179482244 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f78f806b4d0 msgr2=0x7f78f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:58.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.059+0000 7f791ab7a640 1 --2- 172.21.15.161:0/1179482244 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f78f806b4d0 0x7f78f806d990 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7908009ad0 tx=0x7f7908008040 comp rx=0 tx=0).stop 2024-03-28T23:31:58.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.059+0000 7f791ab7a640 1 -- 172.21.15.161:0/1179482244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79140c8ed0 msgr2=0x7f79141ee840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:58.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.059+0000 7f791ab7a640 1 --2- 172.21.15.161:0/1179482244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79140c8ed0 0x7f79141ee840 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f7900000c00 tx=0x7f79000046c0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.059+0000 7f791ab7a640 1 -- 172.21.15.161:0/1179482244 shutdown_connections 2024-03-28T23:31:58.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.059+0000 7f791ab7a640 1 --2- 172.21.15.161:0/1179482244 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f78f806b4d0 0x7f78f806d990 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.059+0000 7f791ab7a640 1 --2- 172.21.15.161:0/1179482244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79140c8ed0 0x7f79141ee840 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.059+0000 7f791ab7a640 1 --2- 172.21.15.161:0/1179482244 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7914152890 0x7f7914149b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.059+0000 7f791ab7a640 1 --2- 172.21.15.161:0/1179482244 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79140c5950 0x7f79140c31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.059+0000 7f791ab7a640 1 -- 172.21.15.161:0/1179482244 >> 172.21.15.161:0/1179482244 conn(0x7f79140ac620 msgr2=0x7f7914151a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:58.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.059+0000 7f791ab7a640 1 -- 172.21.15.161:0/1179482244 shutdown_connections 2024-03-28T23:31:58.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.059+0000 7f791ab7a640 1 -- 172.21.15.161:0/1179482244 wait complete. 2024-03-28T23:31:58.074 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2024-03-28T23:31:57.336508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":232,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237868,"kb_used_data":5060,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736852,"statfs":{"total":773094113280,"available":772850536448,"internally_reserved":0,"allocated":5181440,"data_stored":3427802,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.470377"},"pg_stats":[{"pgid":"10.3d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872212+0000","last_change":"2024-03-28T23:31:53.872212+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872212+0000","last_undegraded":"2024-03-28T23:31:53.872212+0000","last_fullsized":"2024-03-28T23:31:53.872212+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3f","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.870419+0000","last_change":"2024-03-28T23:31:40.388090+0000","last_active":"2024-03-28T23:31:53.870419+0000","last_peered":"2024-03-28T23:31:53.870419+0000","last_clean":"2024-03-28T23:31:53.870419+0000","last_became_active":"2024-03-28T23:31:40.387805+0000","last_became_peered":"2024-03-28T23:31:40.387805+0000","last_unstale":"2024-03-28T23:31:53.870419+0000","last_undegraded":"2024-03-28T23:31:53.870419+0000","last_fullsized":"2024-03-28T23:31:53.870419+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:19:47.987753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.881085+0000","last_change":"2024-03-28T23:31:53.881085+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.881085+0000","last_undegraded":"2024-03-28T23:31:53.881085+0000","last_fullsized":"2024-03-28T23:31:53.881085+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875872+0000","last_change":"2024-03-28T23:31:40.395246+0000","last_active":"2024-03-28T23:31:53.875872+0000","last_peered":"2024-03-28T23:31:53.875872+0000","last_clean":"2024-03-28T23:31:53.875872+0000","last_became_active":"2024-03-28T23:31:40.394583+0000","last_became_peered":"2024-03-28T23:31:40.394583+0000","last_unstale":"2024-03-28T23:31:53.875872+0000","last_undegraded":"2024-03-28T23:31:53.875872+0000","last_fullsized":"2024-03-28T23:31:53.875872+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:18.654331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.873340+0000","last_change":"2024-03-28T23:31:53.873340+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.873340+0000","last_undegraded":"2024-03-28T23:31:53.873340+0000","last_fullsized":"2024-03-28T23:31:53.873340+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876107+0000","last_change":"2024-03-28T23:31:40.397914+0000","last_active":"2024-03-28T23:31:53.876107+0000","last_peered":"2024-03-28T23:31:53.876107+0000","last_clean":"2024-03-28T23:31:53.876107+0000","last_became_active":"2024-03-28T23:31:40.394904+0000","last_became_peered":"2024-03-28T23:31:40.394904+0000","last_unstale":"2024-03-28T23:31:53.876107+0000","last_undegraded":"2024-03-28T23:31:53.876107+0000","last_fullsized":"2024-03-28T23:31:53.876107+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:26:03.062546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872399+0000","last_change":"2024-03-28T23:31:53.872399+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872399+0000","last_undegraded":"2024-03-28T23:31:53.872399+0000","last_fullsized":"2024-03-28T23:31:53.872399+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874319+0000","last_change":"2024-03-28T23:31:40.400594+0000","last_active":"2024-03-28T23:31:53.874319+0000","last_peered":"2024-03-28T23:31:53.874319+0000","last_clean":"2024-03-28T23:31:53.874319+0000","last_became_active":"2024-03-28T23:31:40.400255+0000","last_became_peered":"2024-03-28T23:31:40.400255+0000","last_unstale":"2024-03-28T23:31:53.874319+0000","last_undegraded":"2024-03-28T23:31:53.874319+0000","last_fullsized":"2024-03-28T23:31:53.874319+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:14.515069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.877696+0000","last_change":"2024-03-28T23:31:53.877696+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.877696+0000","last_undegraded":"2024-03-28T23:31:53.877696+0000","last_fullsized":"2024-03-28T23:31:53.877696+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.884068+0000","last_change":"2024-03-28T23:31:40.388531+0000","last_active":"2024-03-28T23:31:53.884068+0000","last_peered":"2024-03-28T23:31:53.884068+0000","last_clean":"2024-03-28T23:31:53.884068+0000","last_became_active":"2024-03-28T23:31:40.388206+0000","last_became_peered":"2024-03-28T23:31:40.388206+0000","last_unstale":"2024-03-28T23:31:53.884068+0000","last_undegraded":"2024-03-28T23:31:53.884068+0000","last_fullsized":"2024-03-28T23:31:53.884068+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:23.215366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.881544+0000","last_change":"2024-03-28T23:31:53.881544+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.881544+0000","last_undegraded":"2024-03-28T23:31:53.881544+0000","last_fullsized":"2024-03-28T23:31:53.881544+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869622+0000","last_change":"2024-03-28T23:31:40.393907+0000","last_active":"2024-03-28T23:31:53.869622+0000","last_peered":"2024-03-28T23:31:53.869622+0000","last_clean":"2024-03-28T23:31:53.869622+0000","last_became_active":"2024-03-28T23:31:40.393443+0000","last_became_peered":"2024-03-28T23:31:40.393443+0000","last_unstale":"2024-03-28T23:31:53.869622+0000","last_undegraded":"2024-03-28T23:31:53.869622+0000","last_fullsized":"2024-03-28T23:31:53.869622+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:28:29.180775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872259+0000","last_change":"2024-03-28T23:31:53.872259+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872259+0000","last_undegraded":"2024-03-28T23:31:53.872259+0000","last_fullsized":"2024-03-28T23:31:53.872259+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875554+0000","last_change":"2024-03-28T23:31:40.398876+0000","last_active":"2024-03-28T23:31:53.875554+0000","last_peered":"2024-03-28T23:31:53.875554+0000","last_clean":"2024-03-28T23:31:53.875554+0000","last_became_active":"2024-03-28T23:31:40.398595+0000","last_became_peered":"2024-03-28T23:31:40.398595+0000","last_unstale":"2024-03-28T23:31:53.875554+0000","last_undegraded":"2024-03-28T23:31:53.875554+0000","last_fullsized":"2024-03-28T23:31:53.875554+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:21.725238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874359+0000","last_change":"2024-03-28T23:31:53.874359+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874359+0000","last_undegraded":"2024-03-28T23:31:53.874359+0000","last_fullsized":"2024-03-28T23:31:53.874359+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.880118+0000","last_change":"2024-03-28T23:31:40.392421+0000","last_active":"2024-03-28T23:31:53.880118+0000","last_peered":"2024-03-28T23:31:53.880118+0000","last_clean":"2024-03-28T23:31:53.880118+0000","last_became_active":"2024-03-28T23:31:40.391463+0000","last_became_peered":"2024-03-28T23:31:40.391463+0000","last_unstale":"2024-03-28T23:31:53.880118+0000","last_undegraded":"2024-03-28T23:31:53.880118+0000","last_fullsized":"2024-03-28T23:31:53.880118+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:18:16.851435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878422+0000","last_change":"2024-03-28T23:31:53.878422+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878422+0000","last_undegraded":"2024-03-28T23:31:53.878422+0000","last_fullsized":"2024-03-28T23:31:53.878422+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.878936+0000","last_change":"2024-03-28T23:31:40.403340+0000","last_active":"2024-03-28T23:31:53.878936+0000","last_peered":"2024-03-28T23:31:53.878936+0000","last_clean":"2024-03-28T23:31:53.878936+0000","last_became_active":"2024-03-28T23:31:40.403089+0000","last_became_peered":"2024-03-28T23:31:40.403089+0000","last_unstale":"2024-03-28T23:31:53.878936+0000","last_undegraded":"2024-03-28T23:31:53.878936+0000","last_fullsized":"2024-03-28T23:31:53.878936+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:12:38.242108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.883968+0000","last_change":"2024-03-28T23:31:53.883968+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.883968+0000","last_undegraded":"2024-03-28T23:31:53.883968+0000","last_fullsized":"2024-03-28T23:31:53.883968+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.870177+0000","last_change":"2024-03-28T23:31:40.382204+0000","last_active":"2024-03-28T23:31:53.870177+0000","last_peered":"2024-03-28T23:31:53.870177+0000","last_clean":"2024-03-28T23:31:53.870177+0000","last_became_active":"2024-03-28T23:31:40.381944+0000","last_became_peered":"2024-03-28T23:31:40.381944+0000","last_unstale":"2024-03-28T23:31:53.870177+0000","last_undegraded":"2024-03-28T23:31:53.870177+0000","last_fullsized":"2024-03-28T23:31:53.870177+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:14.515069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871125+0000","last_change":"2024-03-28T23:31:53.871125+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871125+0000","last_undegraded":"2024-03-28T23:31:53.871125+0000","last_fullsized":"2024-03-28T23:31:53.871125+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.883144+0000","last_change":"2024-03-28T23:31:40.394578+0000","last_active":"2024-03-28T23:31:53.883144+0000","last_peered":"2024-03-28T23:31:53.883144+0000","last_clean":"2024-03-28T23:31:53.883144+0000","last_became_active":"2024-03-28T23:31:40.394272+0000","last_became_peered":"2024-03-28T23:31:40.394272+0000","last_unstale":"2024-03-28T23:31:53.883144+0000","last_undegraded":"2024-03-28T23:31:53.883144+0000","last_fullsized":"2024-03-28T23:31:53.883144+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:24:35.708796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871848+0000","last_change":"2024-03-28T23:31:53.871848+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871848+0000","last_undegraded":"2024-03-28T23:31:53.871848+0000","last_fullsized":"2024-03-28T23:31:53.871848+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879847+0000","last_change":"2024-03-28T23:31:40.396702+0000","last_active":"2024-03-28T23:31:53.879847+0000","last_peered":"2024-03-28T23:31:53.879847+0000","last_clean":"2024-03-28T23:31:53.879847+0000","last_became_active":"2024-03-28T23:31:40.395941+0000","last_became_peered":"2024-03-28T23:31:40.395941+0000","last_unstale":"2024-03-28T23:31:53.879847+0000","last_undegraded":"2024-03-28T23:31:53.879847+0000","last_fullsized":"2024-03-28T23:31:53.879847+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:07:13.690791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871018+0000","last_change":"2024-03-28T23:31:53.871018+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871018+0000","last_undegraded":"2024-03-28T23:31:53.871018+0000","last_fullsized":"2024-03-28T23:31:53.871018+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869713+0000","last_change":"2024-03-28T23:31:40.382947+0000","last_active":"2024-03-28T23:31:53.869713+0000","last_peered":"2024-03-28T23:31:53.869713+0000","last_clean":"2024-03-28T23:31:53.869713+0000","last_became_active":"2024-03-28T23:31:40.381893+0000","last_became_peered":"2024-03-28T23:31:40.381893+0000","last_unstale":"2024-03-28T23:31:53.869713+0000","last_undegraded":"2024-03-28T23:31:53.869713+0000","last_fullsized":"2024-03-28T23:31:53.869713+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:50.872886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.877447+0000","last_change":"2024-03-28T23:31:53.877447+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.877447+0000","last_undegraded":"2024-03-28T23:31:53.877447+0000","last_fullsized":"2024-03-28T23:31:53.877447+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875836+0000","last_change":"2024-03-28T23:31:40.405627+0000","last_active":"2024-03-28T23:31:53.875836+0000","last_peered":"2024-03-28T23:31:53.875836+0000","last_clean":"2024-03-28T23:31:53.875836+0000","last_became_active":"2024-03-28T23:31:40.405360+0000","last_became_peered":"2024-03-28T23:31:40.405360+0000","last_unstale":"2024-03-28T23:31:53.875836+0000","last_undegraded":"2024-03-28T23:31:53.875836+0000","last_fullsized":"2024-03-28T23:31:53.875836+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:06.567694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.881048+0000","last_change":"2024-03-28T23:31:53.881048+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.881048+0000","last_undegraded":"2024-03-28T23:31:53.881048+0000","last_fullsized":"2024-03-28T23:31:53.881048+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.873535+0000","last_change":"2024-03-28T23:31:40.396605+0000","last_active":"2024-03-28T23:31:53.873535+0000","last_peered":"2024-03-28T23:31:53.873535+0000","last_clean":"2024-03-28T23:31:53.873535+0000","last_became_active":"2024-03-28T23:31:40.396237+0000","last_became_peered":"2024-03-28T23:31:40.396237+0000","last_unstale":"2024-03-28T23:31:53.873535+0000","last_undegraded":"2024-03-28T23:31:53.873535+0000","last_fullsized":"2024-03-28T23:31:53.873535+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:14.144440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871402+0000","last_change":"2024-03-28T23:31:53.871402+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871402+0000","last_undegraded":"2024-03-28T23:31:53.871402+0000","last_fullsized":"2024-03-28T23:31:53.871402+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869477+0000","last_change":"2024-03-28T23:31:40.382823+0000","last_active":"2024-03-28T23:31:53.869477+0000","last_peered":"2024-03-28T23:31:53.869477+0000","last_clean":"2024-03-28T23:31:53.869477+0000","last_became_active":"2024-03-28T23:31:40.382574+0000","last_became_peered":"2024-03-28T23:31:40.382574+0000","last_unstale":"2024-03-28T23:31:53.869477+0000","last_undegraded":"2024-03-28T23:31:53.869477+0000","last_fullsized":"2024-03-28T23:31:53.869477+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:12:27.477720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872681+0000","last_change":"2024-03-28T23:31:53.872681+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872681+0000","last_undegraded":"2024-03-28T23:31:53.872681+0000","last_fullsized":"2024-03-28T23:31:53.872681+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876715+0000","last_change":"2024-03-28T23:31:40.403462+0000","last_active":"2024-03-28T23:31:53.876715+0000","last_peered":"2024-03-28T23:31:53.876715+0000","last_clean":"2024-03-28T23:31:53.876715+0000","last_became_active":"2024-03-28T23:31:40.402885+0000","last_became_peered":"2024-03-28T23:31:40.402885+0000","last_unstale":"2024-03-28T23:31:53.876715+0000","last_undegraded":"2024-03-28T23:31:53.876715+0000","last_fullsized":"2024-03-28T23:31:53.876715+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:42:29.351941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.879388+0000","last_change":"2024-03-28T23:31:53.879388+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.879388+0000","last_undegraded":"2024-03-28T23:31:53.879388+0000","last_fullsized":"2024-03-28T23:31:53.879388+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875969+0000","last_change":"2024-03-28T23:31:40.395478+0000","last_active":"2024-03-28T23:31:53.875969+0000","last_peered":"2024-03-28T23:31:53.875969+0000","last_clean":"2024-03-28T23:31:53.875969+0000","last_became_active":"2024-03-28T23:31:40.394981+0000","last_became_peered":"2024-03-28T23:31:40.394981+0000","last_unstale":"2024-03-28T23:31:53.875969+0000","last_undegraded":"2024-03-28T23:31:53.875969+0000","last_fullsized":"2024-03-28T23:31:53.875969+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:06.567694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872394+0000","last_change":"2024-03-28T23:31:53.872394+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872394+0000","last_undegraded":"2024-03-28T23:31:53.872394+0000","last_fullsized":"2024-03-28T23:31:53.872394+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876119+0000","last_change":"2024-03-28T23:31:40.394475+0000","last_active":"2024-03-28T23:31:53.876119+0000","last_peered":"2024-03-28T23:31:53.876119+0000","last_clean":"2024-03-28T23:31:53.876119+0000","last_became_active":"2024-03-28T23:31:40.390205+0000","last_became_peered":"2024-03-28T23:31:40.390205+0000","last_unstale":"2024-03-28T23:31:53.876119+0000","last_undegraded":"2024-03-28T23:31:53.876119+0000","last_fullsized":"2024-03-28T23:31:53.876119+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:04:48.226221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.882562+0000","last_change":"2024-03-28T23:31:53.882562+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.882562+0000","last_undegraded":"2024-03-28T23:31:53.882562+0000","last_fullsized":"2024-03-28T23:31:53.882562+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.877115+0000","last_change":"2024-03-28T23:31:40.388386+0000","last_active":"2024-03-28T23:31:53.877115+0000","last_peered":"2024-03-28T23:31:53.877115+0000","last_clean":"2024-03-28T23:31:53.877115+0000","last_became_active":"2024-03-28T23:31:40.388043+0000","last_became_peered":"2024-03-28T23:31:40.388043+0000","last_unstale":"2024-03-28T23:31:53.877115+0000","last_undegraded":"2024-03-28T23:31:53.877115+0000","last_fullsized":"2024-03-28T23:31:53.877115+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:07.738674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.880460+0000","last_change":"2024-03-28T23:31:53.880460+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.880460+0000","last_undegraded":"2024-03-28T23:31:53.880460+0000","last_fullsized":"2024-03-28T23:31:53.880460+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875638+0000","last_change":"2024-03-28T23:31:40.395811+0000","last_active":"2024-03-28T23:31:53.875638+0000","last_peered":"2024-03-28T23:31:53.875638+0000","last_clean":"2024-03-28T23:31:53.875638+0000","last_became_active":"2024-03-28T23:31:40.395540+0000","last_became_peered":"2024-03-28T23:31:40.395540+0000","last_unstale":"2024-03-28T23:31:53.875638+0000","last_undegraded":"2024-03-28T23:31:53.875638+0000","last_fullsized":"2024-03-28T23:31:53.875638+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:51.864986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.870958+0000","last_change":"2024-03-28T23:31:53.870958+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.870958+0000","last_undegraded":"2024-03-28T23:31:53.870958+0000","last_fullsized":"2024-03-28T23:31:53.870958+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.881134+0000","last_change":"2024-03-28T23:31:40.403129+0000","last_active":"2024-03-28T23:31:53.881134+0000","last_peered":"2024-03-28T23:31:53.881134+0000","last_clean":"2024-03-28T23:31:53.881134+0000","last_became_active":"2024-03-28T23:31:40.402879+0000","last_became_peered":"2024-03-28T23:31:40.402879+0000","last_unstale":"2024-03-28T23:31:53.881134+0000","last_undegraded":"2024-03-28T23:31:53.881134+0000","last_fullsized":"2024-03-28T23:31:53.881134+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:47.861300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.879206+0000","last_change":"2024-03-28T23:31:53.879206+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.879206+0000","last_undegraded":"2024-03-28T23:31:53.879206+0000","last_fullsized":"2024-03-28T23:31:53.879206+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879397+0000","last_change":"2024-03-28T23:31:40.389854+0000","last_active":"2024-03-28T23:31:53.879397+0000","last_peered":"2024-03-28T23:31:53.879397+0000","last_clean":"2024-03-28T23:31:53.879397+0000","last_became_active":"2024-03-28T23:31:40.389599+0000","last_became_peered":"2024-03-28T23:31:40.389599+0000","last_unstale":"2024-03-28T23:31:53.879397+0000","last_undegraded":"2024-03-28T23:31:53.879397+0000","last_fullsized":"2024-03-28T23:31:53.879397+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:43:37.883841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.880913+0000","last_change":"2024-03-28T23:31:53.880913+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.880913+0000","last_undegraded":"2024-03-28T23:31:53.880913+0000","last_fullsized":"2024-03-28T23:31:53.880913+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875417+0000","last_change":"2024-03-28T23:31:40.406317+0000","last_active":"2024-03-28T23:31:53.875417+0000","last_peered":"2024-03-28T23:31:53.875417+0000","last_clean":"2024-03-28T23:31:53.875417+0000","last_became_active":"2024-03-28T23:31:40.405217+0000","last_became_peered":"2024-03-28T23:31:40.405217+0000","last_unstale":"2024-03-28T23:31:53.875417+0000","last_undegraded":"2024-03-28T23:31:53.875417+0000","last_fullsized":"2024-03-28T23:31:53.875417+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:37:55.870956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.879370+0000","last_change":"2024-03-28T23:31:53.879370+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.879370+0000","last_undegraded":"2024-03-28T23:31:53.879370+0000","last_fullsized":"2024-03-28T23:31:53.879370+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876750+0000","last_change":"2024-03-28T23:31:40.404249+0000","last_active":"2024-03-28T23:31:53.876750+0000","last_peered":"2024-03-28T23:31:53.876750+0000","last_clean":"2024-03-28T23:31:53.876750+0000","last_became_active":"2024-03-28T23:31:40.403966+0000","last_became_peered":"2024-03-28T23:31:40.403966+0000","last_unstale":"2024-03-28T23:31:53.876750+0000","last_undegraded":"2024-03-28T23:31:53.876750+0000","last_fullsized":"2024-03-28T23:31:53.876750+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:40:57.541676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872876+0000","last_change":"2024-03-28T23:31:53.872876+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872876+0000","last_undegraded":"2024-03-28T23:31:53.872876+0000","last_fullsized":"2024-03-28T23:31:53.872876+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.878912+0000","last_change":"2024-03-28T23:31:40.397170+0000","last_active":"2024-03-28T23:31:53.878912+0000","last_peered":"2024-03-28T23:31:53.878912+0000","last_clean":"2024-03-28T23:31:53.878912+0000","last_became_active":"2024-03-28T23:31:40.396898+0000","last_became_peered":"2024-03-28T23:31:40.396898+0000","last_unstale":"2024-03-28T23:31:53.878912+0000","last_undegraded":"2024-03-28T23:31:53.878912+0000","last_fullsized":"2024-03-28T23:31:53.878912+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:53:26.355348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878874+0000","last_change":"2024-03-28T23:31:53.878874+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878874+0000","last_undegraded":"2024-03-28T23:31:53.878874+0000","last_fullsized":"2024-03-28T23:31:53.878874+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.870080+0000","last_change":"2024-03-28T23:31:40.404955+0000","last_active":"2024-03-28T23:31:53.870080+0000","last_peered":"2024-03-28T23:31:53.870080+0000","last_clean":"2024-03-28T23:31:53.870080+0000","last_became_active":"2024-03-28T23:31:40.404689+0000","last_became_peered":"2024-03-28T23:31:40.404689+0000","last_unstale":"2024-03-28T23:31:53.870080+0000","last_undegraded":"2024-03-28T23:31:53.870080+0000","last_fullsized":"2024-03-28T23:31:53.870080+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:50:20.574648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878430+0000","last_change":"2024-03-28T23:31:53.878430+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878430+0000","last_undegraded":"2024-03-28T23:31:53.878430+0000","last_fullsized":"2024-03-28T23:31:53.878430+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.877218+0000","last_change":"2024-03-28T23:31:40.389261+0000","last_active":"2024-03-28T23:31:53.877218+0000","last_peered":"2024-03-28T23:31:53.877218+0000","last_clean":"2024-03-28T23:31:53.877218+0000","last_became_active":"2024-03-28T23:31:40.388930+0000","last_became_peered":"2024-03-28T23:31:40.388930+0000","last_unstale":"2024-03-28T23:31:53.877218+0000","last_undegraded":"2024-03-28T23:31:53.877218+0000","last_fullsized":"2024-03-28T23:31:53.877218+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:23.215366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878783+0000","last_change":"2024-03-28T23:31:53.878783+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878783+0000","last_undegraded":"2024-03-28T23:31:53.878783+0000","last_fullsized":"2024-03-28T23:31:53.878783+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875578+0000","last_change":"2024-03-28T23:31:40.401673+0000","last_active":"2024-03-28T23:31:53.875578+0000","last_peered":"2024-03-28T23:31:53.875578+0000","last_clean":"2024-03-28T23:31:53.875578+0000","last_became_active":"2024-03-28T23:31:40.398923+0000","last_became_peered":"2024-03-28T23:31:40.398923+0000","last_unstale":"2024-03-28T23:31:53.875578+0000","last_undegraded":"2024-03-28T23:31:53.875578+0000","last_fullsized":"2024-03-28T23:31:53.875578+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:18.783120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874991+0000","last_change":"2024-03-28T23:31:53.874991+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874991+0000","last_undegraded":"2024-03-28T23:31:53.874991+0000","last_fullsized":"2024-03-28T23:31:53.874991+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.873933+0000","last_change":"2024-03-28T23:31:40.387082+0000","last_active":"2024-03-28T23:31:53.873933+0000","last_peered":"2024-03-28T23:31:53.873933+0000","last_clean":"2024-03-28T23:31:53.873933+0000","last_became_active":"2024-03-28T23:31:40.386808+0000","last_became_peered":"2024-03-28T23:31:40.386808+0000","last_unstale":"2024-03-28T23:31:53.873933+0000","last_undegraded":"2024-03-28T23:31:53.873933+0000","last_fullsized":"2024-03-28T23:31:53.873933+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:43.111058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.870965+0000","last_change":"2024-03-28T23:31:53.870965+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.870965+0000","last_undegraded":"2024-03-28T23:31:53.870965+0000","last_fullsized":"2024-03-28T23:31:53.870965+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875611+0000","last_change":"2024-03-28T23:31:40.397815+0000","last_active":"2024-03-28T23:31:53.875611+0000","last_peered":"2024-03-28T23:31:53.875611+0000","last_clean":"2024-03-28T23:31:53.875611+0000","last_became_active":"2024-03-28T23:31:40.397279+0000","last_became_peered":"2024-03-28T23:31:40.397279+0000","last_unstale":"2024-03-28T23:31:53.875611+0000","last_undegraded":"2024-03-28T23:31:53.875611+0000","last_fullsized":"2024-03-28T23:31:53.875611+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:25:57.449014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878422+0000","last_change":"2024-03-28T23:31:53.878422+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878422+0000","last_undegraded":"2024-03-28T23:31:53.878422+0000","last_fullsized":"2024-03-28T23:31:53.878422+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874691+0000","last_change":"2024-03-28T23:31:40.394484+0000","last_active":"2024-03-28T23:31:53.874691+0000","last_peered":"2024-03-28T23:31:53.874691+0000","last_clean":"2024-03-28T23:31:53.874691+0000","last_became_active":"2024-03-28T23:31:40.394219+0000","last_became_peered":"2024-03-28T23:31:40.394219+0000","last_unstale":"2024-03-28T23:31:53.874691+0000","last_undegraded":"2024-03-28T23:31:53.874691+0000","last_fullsized":"2024-03-28T23:31:53.874691+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:18.654331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874735+0000","last_change":"2024-03-28T23:31:53.874735+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874735+0000","last_undegraded":"2024-03-28T23:31:53.874735+0000","last_fullsized":"2024-03-28T23:31:53.874735+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891397+0000","last_change":"2024-03-28T23:31:53.890105+0000","last_active":"2024-03-28T23:31:53.891397+0000","last_peered":"2024-03-28T23:31:53.891397+0000","last_clean":"2024-03-28T23:31:53.891397+0000","last_became_active":"2024-03-28T23:31:53.889841+0000","last_became_peered":"2024-03-28T23:31:53.889841+0000","last_unstale":"2024-03-28T23:31:53.891397+0000","last_undegraded":"2024-03-28T23:31:53.891397+0000","last_fullsized":"2024-03-28T23:31:53.891397+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:01.360626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.872604+0000","last_change":"2024-03-28T23:31:40.398232+0000","last_active":"2024-03-28T23:31:53.872604+0000","last_peered":"2024-03-28T23:31:53.872604+0000","last_clean":"2024-03-28T23:31:53.872604+0000","last_became_active":"2024-03-28T23:31:40.397888+0000","last_became_peered":"2024-03-28T23:31:40.397888+0000","last_unstale":"2024-03-28T23:31:53.872604+0000","last_undegraded":"2024-03-28T23:31:53.872604+0000","last_fullsized":"2024-03-28T23:31:53.872604+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:50.872886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"50'4","reported_seq":22,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879785+0000","last_change":"2024-03-28T23:31:39.385488+0000","last_active":"2024-03-28T23:31:53.879785+0000","last_peered":"2024-03-28T23:31:53.879785+0000","last_clean":"2024-03-28T23:31:53.879785+0000","last_became_active":"2024-03-28T23:31:39.385211+0000","last_became_peered":"2024-03-28T23:31:39.385211+0000","last_unstale":"2024-03-28T23:31:53.879785+0000","last_undegraded":"2024-03-28T23:31:53.879785+0000","last_fullsized":"2024-03-28T23:31:53.879785+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:50:10.559743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872455+0000","last_change":"2024-03-28T23:31:53.872455+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872455+0000","last_undegraded":"2024-03-28T23:31:53.872455+0000","last_fullsized":"2024-03-28T23:31:53.872455+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891253+0000","last_change":"2024-03-28T23:31:53.887853+0000","last_active":"2024-03-28T23:31:53.891253+0000","last_peered":"2024-03-28T23:31:53.891253+0000","last_clean":"2024-03-28T23:31:53.891253+0000","last_became_active":"2024-03-28T23:31:53.887588+0000","last_became_peered":"2024-03-28T23:31:53.887588+0000","last_unstale":"2024-03-28T23:31:53.891253+0000","last_undegraded":"2024-03-28T23:31:53.891253+0000","last_fullsized":"2024-03-28T23:31:53.891253+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:28:42.680101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.873999+0000","last_change":"2024-03-28T23:31:40.390218+0000","last_active":"2024-03-28T23:31:53.873999+0000","last_peered":"2024-03-28T23:31:53.873999+0000","last_clean":"2024-03-28T23:31:53.873999+0000","last_became_active":"2024-03-28T23:31:40.389922+0000","last_became_peered":"2024-03-28T23:31:40.389922+0000","last_unstale":"2024-03-28T23:31:53.873999+0000","last_undegraded":"2024-03-28T23:31:53.873999+0000","last_fullsized":"2024-03-28T23:31:53.873999+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:14.515069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"50'1","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.870270+0000","last_change":"2024-03-28T23:31:39.393513+0000","last_active":"2024-03-28T23:31:53.870270+0000","last_peered":"2024-03-28T23:31:53.870270+0000","last_clean":"2024-03-28T23:31:53.870270+0000","last_became_active":"2024-03-28T23:31:39.393176+0000","last_became_peered":"2024-03-28T23:31:39.393176+0000","last_unstale":"2024-03-28T23:31:53.870270+0000","last_undegraded":"2024-03-28T23:31:53.870270+0000","last_fullsized":"2024-03-28T23:31:53.870270+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:29.240985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.880671+0000","last_change":"2024-03-28T23:31:53.880671+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.880671+0000","last_undegraded":"2024-03-28T23:31:53.880671+0000","last_fullsized":"2024-03-28T23:31:53.880671+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891931+0000","last_change":"2024-03-28T23:31:53.890699+0000","last_active":"2024-03-28T23:31:53.891931+0000","last_peered":"2024-03-28T23:31:53.891931+0000","last_clean":"2024-03-28T23:31:53.891931+0000","last_became_active":"2024-03-28T23:31:53.890300+0000","last_became_peered":"2024-03-28T23:31:53.890300+0000","last_unstale":"2024-03-28T23:31:53.891931+0000","last_undegraded":"2024-03-28T23:31:53.891931+0000","last_fullsized":"2024-03-28T23:31:53.891931+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:18:30.350761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875566+0000","last_change":"2024-03-28T23:31:40.397077+0000","last_active":"2024-03-28T23:31:53.875566+0000","last_peered":"2024-03-28T23:31:53.875566+0000","last_clean":"2024-03-28T23:31:53.875566+0000","last_became_active":"2024-03-28T23:31:40.395265+0000","last_became_peered":"2024-03-28T23:31:40.395265+0000","last_unstale":"2024-03-28T23:31:53.875566+0000","last_undegraded":"2024-03-28T23:31:53.875566+0000","last_fullsized":"2024-03-28T23:31:53.875566+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:04.829030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"50'2","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.870290+0000","last_change":"2024-03-28T23:31:39.391743+0000","last_active":"2024-03-28T23:31:53.870290+0000","last_peered":"2024-03-28T23:31:53.870290+0000","last_clean":"2024-03-28T23:31:53.870290+0000","last_became_active":"2024-03-28T23:31:39.391300+0000","last_became_peered":"2024-03-28T23:31:39.391300+0000","last_unstale":"2024-03-28T23:31:53.870290+0000","last_undegraded":"2024-03-28T23:31:53.870290+0000","last_fullsized":"2024-03-28T23:31:53.870290+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:12.505382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.877711+0000","last_change":"2024-03-28T23:31:53.877711+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.877711+0000","last_undegraded":"2024-03-28T23:31:53.877711+0000","last_fullsized":"2024-03-28T23:31:53.877711+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.889315+0000","last_change":"2024-03-28T23:31:53.886833+0000","last_active":"2024-03-28T23:31:53.889315+0000","last_peered":"2024-03-28T23:31:53.889315+0000","last_clean":"2024-03-28T23:31:53.889315+0000","last_became_active":"2024-03-28T23:31:53.886498+0000","last_became_peered":"2024-03-28T23:31:53.886498+0000","last_unstale":"2024-03-28T23:31:53.889315+0000","last_undegraded":"2024-03-28T23:31:53.889315+0000","last_fullsized":"2024-03-28T23:31:53.889315+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:26:16.561871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.881876+0000","last_change":"2024-03-28T23:31:40.392391+0000","last_active":"2024-03-28T23:31:53.881876+0000","last_peered":"2024-03-28T23:31:53.881876+0000","last_clean":"2024-03-28T23:31:53.881876+0000","last_became_active":"2024-03-28T23:31:40.392100+0000","last_became_peered":"2024-03-28T23:31:40.392100+0000","last_unstale":"2024-03-28T23:31:53.881876+0000","last_undegraded":"2024-03-28T23:31:53.881876+0000","last_fullsized":"2024-03-28T23:31:53.881876+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:34:38.237729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"50'2","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874448+0000","last_change":"2024-03-28T23:31:39.385430+0000","last_active":"2024-03-28T23:31:53.874448+0000","last_peered":"2024-03-28T23:31:53.874448+0000","last_clean":"2024-03-28T23:31:53.874448+0000","last_became_active":"2024-03-28T23:31:39.385134+0000","last_became_peered":"2024-03-28T23:31:39.385134+0000","last_unstale":"2024-03-28T23:31:53.874448+0000","last_undegraded":"2024-03-28T23:31:53.874448+0000","last_fullsized":"2024-03-28T23:31:53.874448+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:12:26.472924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.873288+0000","last_change":"2024-03-28T23:31:53.873288+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.873288+0000","last_undegraded":"2024-03-28T23:31:53.873288+0000","last_fullsized":"2024-03-28T23:31:53.873288+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891323+0000","last_change":"2024-03-28T23:31:53.889140+0000","last_active":"2024-03-28T23:31:53.891323+0000","last_peered":"2024-03-28T23:31:53.891323+0000","last_clean":"2024-03-28T23:31:53.891323+0000","last_became_active":"2024-03-28T23:31:53.888738+0000","last_became_peered":"2024-03-28T23:31:53.888738+0000","last_unstale":"2024-03-28T23:31:53.891323+0000","last_undegraded":"2024-03-28T23:31:53.891323+0000","last_fullsized":"2024-03-28T23:31:53.891323+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:05:01.725546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.885385+0000","last_change":"2024-03-28T23:31:40.392757+0000","last_active":"2024-03-28T23:31:53.885385+0000","last_peered":"2024-03-28T23:31:53.885385+0000","last_clean":"2024-03-28T23:31:53.885385+0000","last_became_active":"2024-03-28T23:31:40.392460+0000","last_became_peered":"2024-03-28T23:31:40.392460+0000","last_unstale":"2024-03-28T23:31:53.885385+0000","last_undegraded":"2024-03-28T23:31:53.885385+0000","last_fullsized":"2024-03-28T23:31:53.885385+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:07.738674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"50'1","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875684+0000","last_change":"2024-03-28T23:31:39.381607+0000","last_active":"2024-03-28T23:31:53.875684+0000","last_peered":"2024-03-28T23:31:53.875684+0000","last_clean":"2024-03-28T23:31:53.875684+0000","last_became_active":"2024-03-28T23:31:39.381326+0000","last_became_peered":"2024-03-28T23:31:39.381326+0000","last_unstale":"2024-03-28T23:31:53.875684+0000","last_undegraded":"2024-03-28T23:31:53.875684+0000","last_fullsized":"2024-03-28T23:31:53.875684+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:50.860190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.880191+0000","last_change":"2024-03-28T23:31:53.880191+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.880191+0000","last_undegraded":"2024-03-28T23:31:53.880191+0000","last_fullsized":"2024-03-28T23:31:53.880191+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.888447+0000","last_change":"2024-03-28T23:31:53.886424+0000","last_active":"2024-03-28T23:31:53.888447+0000","last_peered":"2024-03-28T23:31:53.888447+0000","last_clean":"2024-03-28T23:31:53.888447+0000","last_became_active":"2024-03-28T23:31:53.886207+0000","last_became_peered":"2024-03-28T23:31:53.886207+0000","last_unstale":"2024-03-28T23:31:53.888447+0000","last_undegraded":"2024-03-28T23:31:53.888447+0000","last_fullsized":"2024-03-28T23:31:53.888447+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:26:10.948340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876571+0000","last_change":"2024-03-28T23:31:40.398712+0000","last_active":"2024-03-28T23:31:53.876571+0000","last_peered":"2024-03-28T23:31:53.876571+0000","last_clean":"2024-03-28T23:31:53.876571+0000","last_became_active":"2024-03-28T23:31:40.398392+0000","last_became_peered":"2024-03-28T23:31:40.398392+0000","last_unstale":"2024-03-28T23:31:53.876571+0000","last_undegraded":"2024-03-28T23:31:53.876571+0000","last_fullsized":"2024-03-28T23:31:53.876571+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:54.431044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"50'1","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.883791+0000","last_change":"2024-03-28T23:31:39.384026+0000","last_active":"2024-03-28T23:31:53.883791+0000","last_peered":"2024-03-28T23:31:53.883791+0000","last_clean":"2024-03-28T23:31:53.883791+0000","last_became_active":"2024-03-28T23:31:39.383161+0000","last_became_peered":"2024-03-28T23:31:39.383161+0000","last_unstale":"2024-03-28T23:31:53.883791+0000","last_undegraded":"2024-03-28T23:31:53.883791+0000","last_fullsized":"2024-03-28T23:31:53.883791+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:21:00.890266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872423+0000","last_change":"2024-03-28T23:31:53.872423+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872423+0000","last_undegraded":"2024-03-28T23:31:53.872423+0000","last_fullsized":"2024-03-28T23:31:53.872423+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.888932+0000","last_change":"2024-03-28T23:31:53.885914+0000","last_active":"2024-03-28T23:31:53.888932+0000","last_peered":"2024-03-28T23:31:53.888932+0000","last_clean":"2024-03-28T23:31:53.888932+0000","last_became_active":"2024-03-28T23:31:53.885637+0000","last_became_peered":"2024-03-28T23:31:53.885637+0000","last_unstale":"2024-03-28T23:31:53.888932+0000","last_undegraded":"2024-03-28T23:31:53.888932+0000","last_fullsized":"2024-03-28T23:31:53.888932+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:32.282446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879100+0000","last_change":"2024-03-28T23:31:40.395728+0000","last_active":"2024-03-28T23:31:53.879100+0000","last_peered":"2024-03-28T23:31:53.879100+0000","last_clean":"2024-03-28T23:31:53.879100+0000","last_became_active":"2024-03-28T23:31:40.395474+0000","last_became_peered":"2024-03-28T23:31:40.395474+0000","last_unstale":"2024-03-28T23:31:53.879100+0000","last_undegraded":"2024-03-28T23:31:53.879100+0000","last_fullsized":"2024-03-28T23:31:53.879100+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:15:19.985376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876912+0000","last_change":"2024-03-28T23:31:39.385649+0000","last_active":"2024-03-28T23:31:53.876912+0000","last_peered":"2024-03-28T23:31:53.876912+0000","last_clean":"2024-03-28T23:31:53.876912+0000","last_became_active":"2024-03-28T23:31:39.385341+0000","last_became_peered":"2024-03-28T23:31:39.385341+0000","last_unstale":"2024-03-28T23:31:53.876912+0000","last_undegraded":"2024-03-28T23:31:53.876912+0000","last_fullsized":"2024-03-28T23:31:53.876912+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:52:06.955653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878702+0000","last_change":"2024-03-28T23:31:53.878702+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878702+0000","last_undegraded":"2024-03-28T23:31:53.878702+0000","last_fullsized":"2024-03-28T23:31:53.878702+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891992+0000","last_change":"2024-03-28T23:31:53.889258+0000","last_active":"2024-03-28T23:31:53.891992+0000","last_peered":"2024-03-28T23:31:53.891992+0000","last_clean":"2024-03-28T23:31:53.891992+0000","last_became_active":"2024-03-28T23:31:53.888970+0000","last_became_peered":"2024-03-28T23:31:53.888970+0000","last_unstale":"2024-03-28T23:31:53.891992+0000","last_undegraded":"2024-03-28T23:31:53.891992+0000","last_fullsized":"2024-03-28T23:31:53.891992+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:20:01.487078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875176+0000","last_change":"2024-03-28T23:31:40.400091+0000","last_active":"2024-03-28T23:31:53.875176+0000","last_peered":"2024-03-28T23:31:53.875176+0000","last_clean":"2024-03-28T23:31:53.875176+0000","last_became_active":"2024-03-28T23:31:40.399190+0000","last_became_peered":"2024-03-28T23:31:40.399190+0000","last_unstale":"2024-03-28T23:31:53.875176+0000","last_undegraded":"2024-03-28T23:31:53.875176+0000","last_fullsized":"2024-03-28T23:31:53.875176+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:22.230986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875772+0000","last_change":"2024-03-28T23:31:39.390198+0000","last_active":"2024-03-28T23:31:53.875772+0000","last_peered":"2024-03-28T23:31:53.875772+0000","last_clean":"2024-03-28T23:31:53.875772+0000","last_became_active":"2024-03-28T23:31:39.389939+0000","last_became_peered":"2024-03-28T23:31:39.389939+0000","last_unstale":"2024-03-28T23:31:53.875772+0000","last_undegraded":"2024-03-28T23:31:53.875772+0000","last_fullsized":"2024-03-28T23:31:53.875772+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:46:30.350009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872121+0000","last_change":"2024-03-28T23:31:53.872121+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872121+0000","last_undegraded":"2024-03-28T23:31:53.872121+0000","last_fullsized":"2024-03-28T23:31:53.872121+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.896042+0000","last_change":"2024-03-28T23:31:53.892143+0000","last_active":"2024-03-28T23:31:53.896042+0000","last_peered":"2024-03-28T23:31:53.896042+0000","last_clean":"2024-03-28T23:31:53.896042+0000","last_became_active":"2024-03-28T23:31:53.891989+0000","last_became_peered":"2024-03-28T23:31:53.891989+0000","last_unstale":"2024-03-28T23:31:53.896042+0000","last_undegraded":"2024-03-28T23:31:53.896042+0000","last_fullsized":"2024-03-28T23:31:53.896042+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:23:27.129147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.883626+0000","last_change":"2024-03-28T23:31:40.384948+0000","last_active":"2024-03-28T23:31:53.883626+0000","last_peered":"2024-03-28T23:31:53.883626+0000","last_clean":"2024-03-28T23:31:53.883626+0000","last_became_active":"2024-03-28T23:31:40.384434+0000","last_became_peered":"2024-03-28T23:31:40.384434+0000","last_unstale":"2024-03-28T23:31:53.883626+0000","last_undegraded":"2024-03-28T23:31:53.883626+0000","last_fullsized":"2024-03-28T23:31:53.883626+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:04.829030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875277+0000","last_change":"2024-03-28T23:31:39.388914+0000","last_active":"2024-03-28T23:31:53.875277+0000","last_peered":"2024-03-28T23:31:53.875277+0000","last_clean":"2024-03-28T23:31:53.875277+0000","last_became_active":"2024-03-28T23:31:39.388639+0000","last_became_peered":"2024-03-28T23:31:39.388639+0000","last_unstale":"2024-03-28T23:31:53.875277+0000","last_undegraded":"2024-03-28T23:31:53.875277+0000","last_fullsized":"2024-03-28T23:31:53.875277+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:00.366278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.880894+0000","last_change":"2024-03-28T23:31:53.880894+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.880894+0000","last_undegraded":"2024-03-28T23:31:53.880894+0000","last_fullsized":"2024-03-28T23:31:53.880894+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.884484+0000","last_change":"2024-03-28T23:31:53.879950+0000","last_active":"2024-03-28T23:31:53.884484+0000","last_peered":"2024-03-28T23:31:53.884484+0000","last_clean":"2024-03-28T23:31:53.884484+0000","last_became_active":"2024-03-28T23:31:53.879620+0000","last_became_peered":"2024-03-28T23:31:53.879620+0000","last_unstale":"2024-03-28T23:31:53.884484+0000","last_undegraded":"2024-03-28T23:31:53.884484+0000","last_fullsized":"2024-03-28T23:31:53.884484+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:28:42.680101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.873900+0000","last_change":"2024-03-28T23:31:40.386749+0000","last_active":"2024-03-28T23:31:53.873900+0000","last_peered":"2024-03-28T23:31:53.873900+0000","last_clean":"2024-03-28T23:31:53.873900+0000","last_became_active":"2024-03-28T23:31:40.386466+0000","last_became_peered":"2024-03-28T23:31:40.386466+0000","last_unstale":"2024-03-28T23:31:53.873900+0000","last_undegraded":"2024-03-28T23:31:53.873900+0000","last_fullsized":"2024-03-28T23:31:53.873900+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:06.567694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.884919+0000","last_change":"2024-03-28T23:31:39.381050+0000","last_active":"2024-03-28T23:31:53.884919+0000","last_peered":"2024-03-28T23:31:53.884919+0000","last_clean":"2024-03-28T23:31:53.884919+0000","last_became_active":"2024-03-28T23:31:39.380761+0000","last_became_peered":"2024-03-28T23:31:39.380761+0000","last_unstale":"2024-03-28T23:31:53.884919+0000","last_undegraded":"2024-03-28T23:31:53.884919+0000","last_fullsized":"2024-03-28T23:31:53.884919+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:29.240985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878283+0000","last_change":"2024-03-28T23:31:53.878283+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878283+0000","last_undegraded":"2024-03-28T23:31:53.878283+0000","last_fullsized":"2024-03-28T23:31:53.878283+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.893113+0000","last_change":"2024-03-28T23:31:53.889906+0000","last_active":"2024-03-28T23:31:53.893113+0000","last_peered":"2024-03-28T23:31:53.893113+0000","last_clean":"2024-03-28T23:31:53.893113+0000","last_became_active":"2024-03-28T23:31:53.889589+0000","last_became_peered":"2024-03-28T23:31:53.889589+0000","last_unstale":"2024-03-28T23:31:53.893113+0000","last_undegraded":"2024-03-28T23:31:53.893113+0000","last_fullsized":"2024-03-28T23:31:53.893113+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:17:43.674083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875914+0000","last_change":"2024-03-28T23:31:40.400388+0000","last_active":"2024-03-28T23:31:53.875914+0000","last_peered":"2024-03-28T23:31:53.875914+0000","last_clean":"2024-03-28T23:31:53.875914+0000","last_became_active":"2024-03-28T23:31:40.400133+0000","last_became_peered":"2024-03-28T23:31:40.400133+0000","last_unstale":"2024-03-28T23:31:53.875914+0000","last_undegraded":"2024-03-28T23:31:53.875914+0000","last_fullsized":"2024-03-28T23:31:53.875914+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:03:56.252539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"50'3","reported_seq":21,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876731+0000","last_change":"2024-03-28T23:31:39.386478+0000","last_active":"2024-03-28T23:31:53.876731+0000","last_peered":"2024-03-28T23:31:53.876731+0000","last_clean":"2024-03-28T23:31:53.876731+0000","last_became_active":"2024-03-28T23:31:39.385868+0000","last_became_peered":"2024-03-28T23:31:39.385868+0000","last_unstale":"2024-03-28T23:31:53.876731+0000","last_undegraded":"2024-03-28T23:31:53.876731+0000","last_fullsized":"2024-03-28T23:31:53.876731+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:06.733879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872413+0000","last_change":"2024-03-28T23:31:53.872413+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872413+0000","last_undegraded":"2024-03-28T23:31:53.872413+0000","last_fullsized":"2024-03-28T23:31:53.872413+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.893021+0000","last_change":"2024-03-28T23:31:53.889887+0000","last_active":"2024-03-28T23:31:53.893021+0000","last_peered":"2024-03-28T23:31:53.893021+0000","last_clean":"2024-03-28T23:31:53.893021+0000","last_became_active":"2024-03-28T23:31:53.889635+0000","last_became_peered":"2024-03-28T23:31:53.889635+0000","last_unstale":"2024-03-28T23:31:53.893021+0000","last_undegraded":"2024-03-28T23:31:53.893021+0000","last_fullsized":"2024-03-28T23:31:53.893021+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:19:23.382549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.884161+0000","last_change":"2024-03-28T23:31:40.387795+0000","last_active":"2024-03-28T23:31:53.884161+0000","last_peered":"2024-03-28T23:31:53.884161+0000","last_clean":"2024-03-28T23:31:53.884161+0000","last_became_active":"2024-03-28T23:31:40.387529+0000","last_became_peered":"2024-03-28T23:31:40.387529+0000","last_unstale":"2024-03-28T23:31:53.884161+0000","last_undegraded":"2024-03-28T23:31:53.884161+0000","last_fullsized":"2024-03-28T23:31:53.884161+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:51.864986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"50'6","reported_seq":24,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879943+0000","last_change":"2024-03-28T23:31:39.393411+0000","last_active":"2024-03-28T23:31:53.879943+0000","last_peered":"2024-03-28T23:31:53.879943+0000","last_clean":"2024-03-28T23:31:53.879943+0000","last_became_active":"2024-03-28T23:31:39.393166+0000","last_became_peered":"2024-03-28T23:31:39.393166+0000","last_unstale":"2024-03-28T23:31:53.879943+0000","last_undegraded":"2024-03-28T23:31:53.879943+0000","last_fullsized":"2024-03-28T23:31:53.879943+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:13.510273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874577+0000","last_change":"2024-03-28T23:31:53.874577+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874577+0000","last_undegraded":"2024-03-28T23:31:53.874577+0000","last_fullsized":"2024-03-28T23:31:53.874577+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.890522+0000","last_change":"2024-03-28T23:31:53.889399+0000","last_active":"2024-03-28T23:31:53.890522+0000","last_peered":"2024-03-28T23:31:53.890522+0000","last_clean":"2024-03-28T23:31:53.890522+0000","last_became_active":"2024-03-28T23:31:53.889110+0000","last_became_peered":"2024-03-28T23:31:53.889110+0000","last_unstale":"2024-03-28T23:31:53.890522+0000","last_undegraded":"2024-03-28T23:31:53.890522+0000","last_fullsized":"2024-03-28T23:31:53.890522+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:35:48.231940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874249+0000","last_change":"2024-03-28T23:31:40.390497+0000","last_active":"2024-03-28T23:31:53.874249+0000","last_peered":"2024-03-28T23:31:53.874249+0000","last_clean":"2024-03-28T23:31:53.874249+0000","last_became_active":"2024-03-28T23:31:40.390167+0000","last_became_peered":"2024-03-28T23:31:40.390167+0000","last_unstale":"2024-03-28T23:31:53.874249+0000","last_undegraded":"2024-03-28T23:31:53.874249+0000","last_fullsized":"2024-03-28T23:31:53.874249+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:54.431044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"50'2","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875733+0000","last_change":"2024-03-28T23:31:39.380369+0000","last_active":"2024-03-28T23:31:53.875733+0000","last_peered":"2024-03-28T23:31:53.875733+0000","last_clean":"2024-03-28T23:31:53.875733+0000","last_became_active":"2024-03-28T23:31:39.380092+0000","last_became_peered":"2024-03-28T23:31:39.380092+0000","last_unstale":"2024-03-28T23:31:53.875733+0000","last_undegraded":"2024-03-28T23:31:53.875733+0000","last_fullsized":"2024-03-28T23:31:53.875733+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:00.366278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871676+0000","last_change":"2024-03-28T23:31:53.871676+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871676+0000","last_undegraded":"2024-03-28T23:31:53.871676+0000","last_fullsized":"2024-03-28T23:31:53.871676+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.885629+0000","last_change":"2024-03-28T23:31:53.883362+0000","last_active":"2024-03-28T23:31:53.885629+0000","last_peered":"2024-03-28T23:31:53.885629+0000","last_clean":"2024-03-28T23:31:53.885629+0000","last_became_active":"2024-03-28T23:31:53.883123+0000","last_became_peered":"2024-03-28T23:31:53.883123+0000","last_unstale":"2024-03-28T23:31:53.885629+0000","last_undegraded":"2024-03-28T23:31:53.885629+0000","last_fullsized":"2024-03-28T23:31:53.885629+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:16.175508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876210+0000","last_change":"2024-03-28T23:31:40.390579+0000","last_active":"2024-03-28T23:31:53.876210+0000","last_peered":"2024-03-28T23:31:53.876210+0000","last_clean":"2024-03-28T23:31:53.876210+0000","last_became_active":"2024-03-28T23:31:40.390248+0000","last_became_peered":"2024-03-28T23:31:40.390248+0000","last_unstale":"2024-03-28T23:31:53.876210+0000","last_undegraded":"2024-03-28T23:31:53.876210+0000","last_fullsized":"2024-03-28T23:31:53.876210+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:47.861300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875729+0000","last_change":"2024-03-28T23:31:39.389592+0000","last_active":"2024-03-28T23:31:53.875729+0000","last_peered":"2024-03-28T23:31:53.875729+0000","last_clean":"2024-03-28T23:31:53.875729+0000","last_became_active":"2024-03-28T23:31:39.389302+0000","last_became_peered":"2024-03-28T23:31:39.389302+0000","last_unstale":"2024-03-28T23:31:53.875729+0000","last_undegraded":"2024-03-28T23:31:53.875729+0000","last_fullsized":"2024-03-28T23:31:53.875729+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:59:51.501145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872186+0000","last_change":"2024-03-28T23:31:53.872186+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872186+0000","last_undegraded":"2024-03-28T23:31:53.872186+0000","last_fullsized":"2024-03-28T23:31:53.872186+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.884324+0000","last_change":"2024-03-28T23:31:53.881795+0000","last_active":"2024-03-28T23:31:53.884324+0000","last_peered":"2024-03-28T23:31:53.884324+0000","last_clean":"2024-03-28T23:31:53.884324+0000","last_became_active":"2024-03-28T23:31:53.881552+0000","last_became_peered":"2024-03-28T23:31:53.881552+0000","last_unstale":"2024-03-28T23:31:53.884324+0000","last_undegraded":"2024-03-28T23:31:53.884324+0000","last_fullsized":"2024-03-28T23:31:53.884324+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:35.224564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879015+0000","last_change":"2024-03-28T23:31:40.387085+0000","last_active":"2024-03-28T23:31:53.879015+0000","last_peered":"2024-03-28T23:31:53.879015+0000","last_clean":"2024-03-28T23:31:53.879015+0000","last_became_active":"2024-03-28T23:31:40.386695+0000","last_became_peered":"2024-03-28T23:31:40.386695+0000","last_unstale":"2024-03-28T23:31:53.879015+0000","last_undegraded":"2024-03-28T23:31:53.879015+0000","last_fullsized":"2024-03-28T23:31:53.879015+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:40:57.541676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.873632+0000","last_change":"2024-03-28T23:31:39.384653+0000","last_active":"2024-03-28T23:31:53.873632+0000","last_peered":"2024-03-28T23:31:53.873632+0000","last_clean":"2024-03-28T23:31:53.873632+0000","last_became_active":"2024-03-28T23:31:39.383495+0000","last_became_peered":"2024-03-28T23:31:39.383495+0000","last_unstale":"2024-03-28T23:31:53.873632+0000","last_undegraded":"2024-03-28T23:31:53.873632+0000","last_fullsized":"2024-03-28T23:31:53.873632+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:50.860190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878941+0000","last_change":"2024-03-28T23:31:53.878941+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878941+0000","last_undegraded":"2024-03-28T23:31:53.878941+0000","last_fullsized":"2024-03-28T23:31:53.878941+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.886195+0000","last_change":"2024-03-28T23:31:53.884651+0000","last_active":"2024-03-28T23:31:53.886195+0000","last_peered":"2024-03-28T23:31:53.886195+0000","last_clean":"2024-03-28T23:31:53.886195+0000","last_became_active":"2024-03-28T23:31:53.884410+0000","last_became_peered":"2024-03-28T23:31:53.884410+0000","last_unstale":"2024-03-28T23:31:53.886195+0000","last_undegraded":"2024-03-28T23:31:53.886195+0000","last_fullsized":"2024-03-28T23:31:53.886195+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:31:41.553370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876561+0000","last_change":"2024-03-28T23:31:40.403701+0000","last_active":"2024-03-28T23:31:53.876561+0000","last_peered":"2024-03-28T23:31:53.876561+0000","last_clean":"2024-03-28T23:31:53.876561+0000","last_became_active":"2024-03-28T23:31:40.403222+0000","last_became_peered":"2024-03-28T23:31:40.403222+0000","last_unstale":"2024-03-28T23:31:53.876561+0000","last_undegraded":"2024-03-28T23:31:53.876561+0000","last_fullsized":"2024-03-28T23:31:53.876561+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:40:09.867914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"50'8","reported_seq":26,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.880291+0000","last_change":"2024-03-28T23:31:39.383726+0000","last_active":"2024-03-28T23:31:53.880291+0000","last_peered":"2024-03-28T23:31:53.880291+0000","last_clean":"2024-03-28T23:31:53.880291+0000","last_became_active":"2024-03-28T23:31:39.383292+0000","last_became_peered":"2024-03-28T23:31:39.383292+0000","last_unstale":"2024-03-28T23:31:53.880291+0000","last_undegraded":"2024-03-28T23:31:53.880291+0000","last_fullsized":"2024-03-28T23:31:53.880291+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:25:38.113638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872289+0000","last_change":"2024-03-28T23:31:53.872289+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872289+0000","last_undegraded":"2024-03-28T23:31:53.872289+0000","last_fullsized":"2024-03-28T23:31:53.872289+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.896055+0000","last_change":"2024-03-28T23:31:53.892246+0000","last_active":"2024-03-28T23:31:53.896055+0000","last_peered":"2024-03-28T23:31:53.896055+0000","last_clean":"2024-03-28T23:31:53.896055+0000","last_became_active":"2024-03-28T23:31:53.891974+0000","last_became_peered":"2024-03-28T23:31:53.891974+0000","last_unstale":"2024-03-28T23:31:53.896055+0000","last_undegraded":"2024-03-28T23:31:53.896055+0000","last_fullsized":"2024-03-28T23:31:53.896055+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:10:22.963954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.880205+0000","last_change":"2024-03-28T23:31:40.391219+0000","last_active":"2024-03-28T23:31:53.880205+0000","last_peered":"2024-03-28T23:31:53.880205+0000","last_clean":"2024-03-28T23:31:53.880205+0000","last_became_active":"2024-03-28T23:31:40.389552+0000","last_became_peered":"2024-03-28T23:31:40.389552+0000","last_unstale":"2024-03-28T23:31:53.880205+0000","last_undegraded":"2024-03-28T23:31:53.880205+0000","last_fullsized":"2024-03-28T23:31:53.880205+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:42:29.351941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"50'1","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876797+0000","last_change":"2024-03-28T23:31:39.390253+0000","last_active":"2024-03-28T23:31:53.876797+0000","last_peered":"2024-03-28T23:31:53.876797+0000","last_clean":"2024-03-28T23:31:53.876797+0000","last_became_active":"2024-03-28T23:31:39.389980+0000","last_became_peered":"2024-03-28T23:31:39.389980+0000","last_unstale":"2024-03-28T23:31:53.876797+0000","last_undegraded":"2024-03-28T23:31:53.876797+0000","last_fullsized":"2024-03-28T23:31:53.876797+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:24.520296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.877420+0000","last_change":"2024-03-28T23:31:53.877420+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.877420+0000","last_undegraded":"2024-03-28T23:31:53.877420+0000","last_fullsized":"2024-03-28T23:31:53.877420+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.884332+0000","last_change":"2024-03-28T23:31:53.880659+0000","last_active":"2024-03-28T23:31:53.884332+0000","last_peered":"2024-03-28T23:31:53.884332+0000","last_clean":"2024-03-28T23:31:53.884332+0000","last_became_active":"2024-03-28T23:31:53.880265+0000","last_became_peered":"2024-03-28T23:31:53.880265+0000","last_unstale":"2024-03-28T23:31:53.884332+0000","last_undegraded":"2024-03-28T23:31:53.884332+0000","last_fullsized":"2024-03-28T23:31:53.884332+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:04:09.751864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869503+0000","last_change":"2024-03-28T23:31:40.391898+0000","last_active":"2024-03-28T23:31:53.869503+0000","last_peered":"2024-03-28T23:31:53.869503+0000","last_clean":"2024-03-28T23:31:53.869503+0000","last_became_active":"2024-03-28T23:31:40.391622+0000","last_became_peered":"2024-03-28T23:31:40.391622+0000","last_unstale":"2024-03-28T23:31:53.869503+0000","last_undegraded":"2024-03-28T23:31:53.869503+0000","last_fullsized":"2024-03-28T23:31:53.869503+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:38:22.055287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"50'2","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.883203+0000","last_change":"2024-03-28T23:31:39.385339+0000","last_active":"2024-03-28T23:31:53.883203+0000","last_peered":"2024-03-28T23:31:53.883203+0000","last_clean":"2024-03-28T23:31:53.883203+0000","last_became_active":"2024-03-28T23:31:39.385075+0000","last_became_peered":"2024-03-28T23:31:39.385075+0000","last_unstale":"2024-03-28T23:31:53.883203+0000","last_undegraded":"2024-03-28T23:31:53.883203+0000","last_fullsized":"2024-03-28T23:31:53.883203+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:58.098710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.873793+0000","last_change":"2024-03-28T23:31:53.873793+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.873793+0000","last_undegraded":"2024-03-28T23:31:53.873793+0000","last_fullsized":"2024-03-28T23:31:53.873793+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891950+0000","last_change":"2024-03-28T23:31:53.890901+0000","last_active":"2024-03-28T23:31:53.891950+0000","last_peered":"2024-03-28T23:31:53.891950+0000","last_clean":"2024-03-28T23:31:53.891950+0000","last_became_active":"2024-03-28T23:31:53.890643+0000","last_became_peered":"2024-03-28T23:31:53.890643+0000","last_unstale":"2024-03-28T23:31:53.891950+0000","last_undegraded":"2024-03-28T23:31:53.891950+0000","last_fullsized":"2024-03-28T23:31:53.891950+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:07:27.190116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.883208+0000","last_change":"2024-03-28T23:31:40.399291+0000","last_active":"2024-03-28T23:31:53.883208+0000","last_peered":"2024-03-28T23:31:53.883208+0000","last_clean":"2024-03-28T23:31:53.883208+0000","last_became_active":"2024-03-28T23:31:40.399005+0000","last_became_peered":"2024-03-28T23:31:40.399005+0000","last_unstale":"2024-03-28T23:31:53.883208+0000","last_undegraded":"2024-03-28T23:31:53.883208+0000","last_fullsized":"2024-03-28T23:31:53.883208+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:01.371074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"50'1","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876296+0000","last_change":"2024-03-28T23:31:39.389465+0000","last_active":"2024-03-28T23:31:53.876296+0000","last_peered":"2024-03-28T23:31:53.876296+0000","last_clean":"2024-03-28T23:31:53.876296+0000","last_became_active":"2024-03-28T23:31:39.389005+0000","last_became_peered":"2024-03-28T23:31:39.389005+0000","last_unstale":"2024-03-28T23:31:53.876296+0000","last_undegraded":"2024-03-28T23:31:53.876296+0000","last_fullsized":"2024-03-28T23:31:53.876296+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:41.430571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.879255+0000","last_change":"2024-03-28T23:31:53.879255+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.879255+0000","last_undegraded":"2024-03-28T23:31:53.879255+0000","last_fullsized":"2024-03-28T23:31:53.879255+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.890716+0000","last_change":"2024-03-28T23:31:53.888704+0000","last_active":"2024-03-28T23:31:53.890716+0000","last_peered":"2024-03-28T23:31:53.890716+0000","last_clean":"2024-03-28T23:31:53.890716+0000","last_became_active":"2024-03-28T23:31:53.888461+0000","last_became_peered":"2024-03-28T23:31:53.888461+0000","last_unstale":"2024-03-28T23:31:53.890716+0000","last_undegraded":"2024-03-28T23:31:53.890716+0000","last_fullsized":"2024-03-28T23:31:53.890716+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:08.894103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869795+0000","last_change":"2024-03-28T23:31:40.388141+0000","last_active":"2024-03-28T23:31:53.869795+0000","last_peered":"2024-03-28T23:31:53.869795+0000","last_clean":"2024-03-28T23:31:53.869795+0000","last_became_active":"2024-03-28T23:31:40.387852+0000","last_became_peered":"2024-03-28T23:31:40.387852+0000","last_unstale":"2024-03-28T23:31:53.869795+0000","last_undegraded":"2024-03-28T23:31:53.869795+0000","last_fullsized":"2024-03-28T23:31:53.869795+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:22.230986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.877311+0000","last_change":"2024-03-28T23:31:39.390465+0000","last_active":"2024-03-28T23:31:53.877311+0000","last_peered":"2024-03-28T23:31:53.877311+0000","last_clean":"2024-03-28T23:31:53.877311+0000","last_became_active":"2024-03-28T23:31:39.390203+0000","last_became_peered":"2024-03-28T23:31:39.390203+0000","last_unstale":"2024-03-28T23:31:53.877311+0000","last_undegraded":"2024-03-28T23:31:53.877311+0000","last_fullsized":"2024-03-28T23:31:53.877311+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:50:50.496294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.876579+0000","last_change":"2024-03-28T23:31:53.876579+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.876579+0000","last_undegraded":"2024-03-28T23:31:53.876579+0000","last_fullsized":"2024-03-28T23:31:53.876579+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.890502+0000","last_change":"2024-03-28T23:31:53.888267+0000","last_active":"2024-03-28T23:31:53.890502+0000","last_peered":"2024-03-28T23:31:53.890502+0000","last_clean":"2024-03-28T23:31:53.890502+0000","last_became_active":"2024-03-28T23:31:53.887993+0000","last_became_peered":"2024-03-28T23:31:53.887993+0000","last_unstale":"2024-03-28T23:31:53.890502+0000","last_undegraded":"2024-03-28T23:31:53.890502+0000","last_fullsized":"2024-03-28T23:31:53.890502+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:21:27.713981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874357+0000","last_change":"2024-03-28T23:31:40.387376+0000","last_active":"2024-03-28T23:31:53.874357+0000","last_peered":"2024-03-28T23:31:53.874357+0000","last_clean":"2024-03-28T23:31:53.874357+0000","last_became_active":"2024-03-28T23:31:40.387080+0000","last_became_peered":"2024-03-28T23:31:40.387080+0000","last_unstale":"2024-03-28T23:31:53.874357+0000","last_undegraded":"2024-03-28T23:31:53.874357+0000","last_fullsized":"2024-03-28T23:31:53.874357+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:43.111058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"50'2","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869894+0000","last_change":"2024-03-28T23:31:39.388867+0000","last_active":"2024-03-28T23:31:53.869894+0000","last_peered":"2024-03-28T23:31:53.869894+0000","last_clean":"2024-03-28T23:31:53.869894+0000","last_became_active":"2024-03-28T23:31:39.388593+0000","last_became_peered":"2024-03-28T23:31:39.388593+0000","last_unstale":"2024-03-28T23:31:53.869894+0000","last_undegraded":"2024-03-28T23:31:53.869894+0000","last_fullsized":"2024-03-28T23:31:53.869894+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:21:00.890266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872546+0000","last_change":"2024-03-28T23:31:53.872546+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872546+0000","last_undegraded":"2024-03-28T23:31:53.872546+0000","last_fullsized":"2024-03-28T23:31:53.872546+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.888466+0000","last_change":"2024-03-28T23:31:53.886433+0000","last_active":"2024-03-28T23:31:53.888466+0000","last_peered":"2024-03-28T23:31:53.888466+0000","last_clean":"2024-03-28T23:31:53.888466+0000","last_became_active":"2024-03-28T23:31:53.886218+0000","last_became_peered":"2024-03-28T23:31:53.886218+0000","last_unstale":"2024-03-28T23:31:53.888466+0000","last_undegraded":"2024-03-28T23:31:53.888466+0000","last_fullsized":"2024-03-28T23:31:53.888466+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:32.282446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876171+0000","last_change":"2024-03-28T23:31:40.393257+0000","last_active":"2024-03-28T23:31:53.876171+0000","last_peered":"2024-03-28T23:31:53.876171+0000","last_clean":"2024-03-28T23:31:53.876171+0000","last_became_active":"2024-03-28T23:31:40.391684+0000","last_became_peered":"2024-03-28T23:31:40.391684+0000","last_unstale":"2024-03-28T23:31:53.876171+0000","last_undegraded":"2024-03-28T23:31:53.876171+0000","last_fullsized":"2024-03-28T23:31:53.876171+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:01.371074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.881223+0000","last_change":"2024-03-28T23:31:39.388198+0000","last_active":"2024-03-28T23:31:53.881223+0000","last_peered":"2024-03-28T23:31:53.881223+0000","last_clean":"2024-03-28T23:31:53.881223+0000","last_became_active":"2024-03-28T23:31:39.387727+0000","last_became_peered":"2024-03-28T23:31:39.387727+0000","last_unstale":"2024-03-28T23:31:53.881223+0000","last_undegraded":"2024-03-28T23:31:53.881223+0000","last_fullsized":"2024-03-28T23:31:53.881223+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:49.868091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874240+0000","last_change":"2024-03-28T23:31:53.874240+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874240+0000","last_undegraded":"2024-03-28T23:31:53.874240+0000","last_fullsized":"2024-03-28T23:31:53.874240+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.886125+0000","last_change":"2024-03-28T23:31:53.884698+0000","last_active":"2024-03-28T23:31:53.886125+0000","last_peered":"2024-03-28T23:31:53.886125+0000","last_clean":"2024-03-28T23:31:53.886125+0000","last_became_active":"2024-03-28T23:31:53.884413+0000","last_became_peered":"2024-03-28T23:31:53.884413+0000","last_unstale":"2024-03-28T23:31:53.886125+0000","last_undegraded":"2024-03-28T23:31:53.886125+0000","last_fullsized":"2024-03-28T23:31:53.886125+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:32.282446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876556+0000","last_change":"2024-03-28T23:31:40.394047+0000","last_active":"2024-03-28T23:31:53.876556+0000","last_peered":"2024-03-28T23:31:53.876556+0000","last_clean":"2024-03-28T23:31:53.876556+0000","last_became_active":"2024-03-28T23:31:40.393772+0000","last_became_peered":"2024-03-28T23:31:40.393772+0000","last_unstale":"2024-03-28T23:31:53.876556+0000","last_undegraded":"2024-03-28T23:31:53.876556+0000","last_fullsized":"2024-03-28T23:31:53.876556+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:07:13.690791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875624+0000","last_change":"2024-03-28T23:31:39.388829+0000","last_active":"2024-03-28T23:31:53.875624+0000","last_peered":"2024-03-28T23:31:53.875624+0000","last_clean":"2024-03-28T23:31:53.875624+0000","last_became_active":"2024-03-28T23:31:39.388521+0000","last_became_peered":"2024-03-28T23:31:39.388521+0000","last_unstale":"2024-03-28T23:31:53.875624+0000","last_undegraded":"2024-03-28T23:31:53.875624+0000","last_fullsized":"2024-03-28T23:31:53.875624+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:23:52.148574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.877392+0000","last_change":"2024-03-28T23:31:53.877392+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.877392+0000","last_undegraded":"2024-03-28T23:31:53.877392+0000","last_fullsized":"2024-03-28T23:31:53.877392+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.887423+0000","last_change":"2024-03-28T23:31:53.886155+0000","last_active":"2024-03-28T23:31:53.887423+0000","last_peered":"2024-03-28T23:31:53.887423+0000","last_clean":"2024-03-28T23:31:53.887423+0000","last_became_active":"2024-03-28T23:31:53.885843+0000","last_became_peered":"2024-03-28T23:31:53.885843+0000","last_unstale":"2024-03-28T23:31:53.887423+0000","last_undegraded":"2024-03-28T23:31:53.887423+0000","last_fullsized":"2024-03-28T23:31:53.887423+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:38:09.370282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.870442+0000","last_change":"2024-03-28T23:31:40.384709+0000","last_active":"2024-03-28T23:31:53.870442+0000","last_peered":"2024-03-28T23:31:53.870442+0000","last_clean":"2024-03-28T23:31:53.870442+0000","last_became_active":"2024-03-28T23:31:40.384412+0000","last_became_peered":"2024-03-28T23:31:40.384412+0000","last_unstale":"2024-03-28T23:31:53.870442+0000","last_undegraded":"2024-03-28T23:31:53.870442+0000","last_fullsized":"2024-03-28T23:31:53.870442+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:25:39.118434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879185+0000","last_change":"2024-03-28T23:31:39.385937+0000","last_active":"2024-03-28T23:31:53.879185+0000","last_peered":"2024-03-28T23:31:53.879185+0000","last_clean":"2024-03-28T23:31:53.879185+0000","last_became_active":"2024-03-28T23:31:39.385675+0000","last_became_peered":"2024-03-28T23:31:39.385675+0000","last_unstale":"2024-03-28T23:31:53.879185+0000","last_undegraded":"2024-03-28T23:31:53.879185+0000","last_fullsized":"2024-03-28T23:31:53.879185+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:35.204851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874303+0000","last_change":"2024-03-28T23:31:53.874303+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874303+0000","last_undegraded":"2024-03-28T23:31:53.874303+0000","last_fullsized":"2024-03-28T23:31:53.874303+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.886168+0000","last_change":"2024-03-28T23:31:53.884950+0000","last_active":"2024-03-28T23:31:53.886168+0000","last_peered":"2024-03-28T23:31:53.886168+0000","last_clean":"2024-03-28T23:31:53.886168+0000","last_became_active":"2024-03-28T23:31:53.884686+0000","last_became_peered":"2024-03-28T23:31:53.884686+0000","last_unstale":"2024-03-28T23:31:53.886168+0000","last_undegraded":"2024-03-28T23:31:53.886168+0000","last_fullsized":"2024-03-28T23:31:53.886168+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:48.477986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874779+0000","last_change":"2024-03-28T23:31:40.387783+0000","last_active":"2024-03-28T23:31:53.874779+0000","last_peered":"2024-03-28T23:31:53.874779+0000","last_clean":"2024-03-28T23:31:53.874779+0000","last_became_active":"2024-03-28T23:31:40.387487+0000","last_became_peered":"2024-03-28T23:31:40.387487+0000","last_unstale":"2024-03-28T23:31:53.874779+0000","last_undegraded":"2024-03-28T23:31:53.874779+0000","last_fullsized":"2024-03-28T23:31:53.874779+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:22.230986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"50'2","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.880346+0000","last_change":"2024-03-28T23:31:39.391741+0000","last_active":"2024-03-28T23:31:53.880346+0000","last_peered":"2024-03-28T23:31:53.880346+0000","last_clean":"2024-03-28T23:31:53.880346+0000","last_became_active":"2024-03-28T23:31:39.391520+0000","last_became_peered":"2024-03-28T23:31:39.391520+0000","last_unstale":"2024-03-28T23:31:53.880346+0000","last_undegraded":"2024-03-28T23:31:53.880346+0000","last_fullsized":"2024-03-28T23:31:53.880346+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:16:42.869555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874197+0000","last_change":"2024-03-28T23:31:53.874197+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874197+0000","last_undegraded":"2024-03-28T23:31:53.874197+0000","last_fullsized":"2024-03-28T23:31:53.874197+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.890450+0000","last_change":"2024-03-28T23:31:53.887757+0000","last_active":"2024-03-28T23:31:53.890450+0000","last_peered":"2024-03-28T23:31:53.890450+0000","last_clean":"2024-03-28T23:31:53.890450+0000","last_became_active":"2024-03-28T23:31:53.887470+0000","last_became_peered":"2024-03-28T23:31:53.887470+0000","last_unstale":"2024-03-28T23:31:53.890450+0000","last_undegraded":"2024-03-28T23:31:53.890450+0000","last_fullsized":"2024-03-28T23:31:53.890450+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:40:57.385677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869629+0000","last_change":"2024-03-28T23:31:40.387330+0000","last_active":"2024-03-28T23:31:53.869629+0000","last_peered":"2024-03-28T23:31:53.869629+0000","last_clean":"2024-03-28T23:31:53.869629+0000","last_became_active":"2024-03-28T23:31:40.387076+0000","last_became_peered":"2024-03-28T23:31:40.387076+0000","last_unstale":"2024-03-28T23:31:53.869629+0000","last_undegraded":"2024-03-28T23:31:53.869629+0000","last_fullsized":"2024-03-28T23:31:53.869629+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:25.525091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.877434+0000","last_change":"2024-03-28T23:31:39.386686+0000","last_active":"2024-03-28T23:31:53.877434+0000","last_peered":"2024-03-28T23:31:53.877434+0000","last_clean":"2024-03-28T23:31:53.877434+0000","last_became_active":"2024-03-28T23:31:39.386208+0000","last_became_peered":"2024-03-28T23:31:39.386208+0000","last_unstale":"2024-03-28T23:31:53.877434+0000","last_undegraded":"2024-03-28T23:31:53.877434+0000","last_fullsized":"2024-03-28T23:31:53.877434+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:22.210570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.876269+0000","last_change":"2024-03-28T23:31:53.876269+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.876269+0000","last_undegraded":"2024-03-28T23:31:53.876269+0000","last_fullsized":"2024-03-28T23:31:53.876269+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891725+0000","last_change":"2024-03-28T23:31:53.887766+0000","last_active":"2024-03-28T23:31:53.891725+0000","last_peered":"2024-03-28T23:31:53.891725+0000","last_clean":"2024-03-28T23:31:53.891725+0000","last_became_active":"2024-03-28T23:31:53.887471+0000","last_became_peered":"2024-03-28T23:31:53.887471+0000","last_unstale":"2024-03-28T23:31:53.891725+0000","last_undegraded":"2024-03-28T23:31:53.891725+0000","last_fullsized":"2024-03-28T23:31:53.891725+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:05:01.725546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.881956+0000","last_change":"2024-03-28T23:31:40.386143+0000","last_active":"2024-03-28T23:31:53.881956+0000","last_peered":"2024-03-28T23:31:53.881956+0000","last_clean":"2024-03-28T23:31:53.881956+0000","last_became_active":"2024-03-28T23:31:40.385606+0000","last_became_peered":"2024-03-28T23:31:40.385606+0000","last_unstale":"2024-03-28T23:31:53.881956+0000","last_undegraded":"2024-03-28T23:31:53.881956+0000","last_fullsized":"2024-03-28T23:31:53.881956+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:40:09.867914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876026+0000","last_change":"2024-03-28T23:31:39.380624+0000","last_active":"2024-03-28T23:31:53.876026+0000","last_peered":"2024-03-28T23:31:53.876026+0000","last_clean":"2024-03-28T23:31:53.876026+0000","last_became_active":"2024-03-28T23:31:39.380340+0000","last_became_peered":"2024-03-28T23:31:39.380340+0000","last_unstale":"2024-03-28T23:31:53.876026+0000","last_undegraded":"2024-03-28T23:31:53.876026+0000","last_fullsized":"2024-03-28T23:31:53.876026+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:22.210570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.879694+0000","last_change":"2024-03-28T23:31:53.879694+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.879694+0000","last_undegraded":"2024-03-28T23:31:53.879694+0000","last_fullsized":"2024-03-28T23:31:53.879694+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.889286+0000","last_change":"2024-03-28T23:31:53.886660+0000","last_active":"2024-03-28T23:31:53.889286+0000","last_peered":"2024-03-28T23:31:53.889286+0000","last_clean":"2024-03-28T23:31:53.889286+0000","last_became_active":"2024-03-28T23:31:53.886355+0000","last_became_peered":"2024-03-28T23:31:53.886355+0000","last_unstale":"2024-03-28T23:31:53.889286+0000","last_undegraded":"2024-03-28T23:31:53.889286+0000","last_fullsized":"2024-03-28T23:31:53.889286+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:38:09.370282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875942+0000","last_change":"2024-03-28T23:31:40.404359+0000","last_active":"2024-03-28T23:31:53.875942+0000","last_peered":"2024-03-28T23:31:53.875942+0000","last_clean":"2024-03-28T23:31:53.875942+0000","last_became_active":"2024-03-28T23:31:40.404078+0000","last_became_peered":"2024-03-28T23:31:40.404078+0000","last_unstale":"2024-03-28T23:31:53.875942+0000","last_undegraded":"2024-03-28T23:31:53.875942+0000","last_fullsized":"2024-03-28T23:31:53.875942+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:18.654331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.882034+0000","last_change":"2024-03-28T23:31:39.393283+0000","last_active":"2024-03-28T23:31:53.882034+0000","last_peered":"2024-03-28T23:31:53.882034+0000","last_clean":"2024-03-28T23:31:53.882034+0000","last_became_active":"2024-03-28T23:31:39.393035+0000","last_became_peered":"2024-03-28T23:31:39.393035+0000","last_unstale":"2024-03-28T23:31:53.882034+0000","last_undegraded":"2024-03-28T23:31:53.882034+0000","last_fullsized":"2024-03-28T23:31:53.882034+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:13.139644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872993+0000","last_change":"2024-03-28T23:31:53.872993+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872993+0000","last_undegraded":"2024-03-28T23:31:53.872993+0000","last_fullsized":"2024-03-28T23:31:53.872993+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.890751+0000","last_change":"2024-03-28T23:31:53.888742+0000","last_active":"2024-03-28T23:31:53.890751+0000","last_peered":"2024-03-28T23:31:53.890751+0000","last_clean":"2024-03-28T23:31:53.890751+0000","last_became_active":"2024-03-28T23:31:53.888515+0000","last_became_peered":"2024-03-28T23:31:53.888515+0000","last_unstale":"2024-03-28T23:31:53.890751+0000","last_undegraded":"2024-03-28T23:31:53.890751+0000","last_fullsized":"2024-03-28T23:31:53.890751+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:17:10.580507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875207+0000","last_change":"2024-03-28T23:31:40.392819+0000","last_active":"2024-03-28T23:31:53.875207+0000","last_peered":"2024-03-28T23:31:53.875207+0000","last_clean":"2024-03-28T23:31:53.875207+0000","last_became_active":"2024-03-28T23:31:40.392501+0000","last_became_peered":"2024-03-28T23:31:40.392501+0000","last_unstale":"2024-03-28T23:31:53.875207+0000","last_undegraded":"2024-03-28T23:31:53.875207+0000","last_fullsized":"2024-03-28T23:31:53.875207+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:14.144440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.881307+0000","last_change":"2024-03-28T23:31:39.387993+0000","last_active":"2024-03-28T23:31:53.881307+0000","last_peered":"2024-03-28T23:31:53.881307+0000","last_clean":"2024-03-28T23:31:53.881307+0000","last_became_active":"2024-03-28T23:31:39.387393+0000","last_became_peered":"2024-03-28T23:31:39.387393+0000","last_unstale":"2024-03-28T23:31:53.881307+0000","last_undegraded":"2024-03-28T23:31:53.881307+0000","last_fullsized":"2024-03-28T23:31:53.881307+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:21.226190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871482+0000","last_change":"2024-03-28T23:31:53.871482+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871482+0000","last_undegraded":"2024-03-28T23:31:53.871482+0000","last_fullsized":"2024-03-28T23:31:53.871482+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.892224+0000","last_change":"2024-03-28T23:31:53.890146+0000","last_active":"2024-03-28T23:31:53.892224+0000","last_peered":"2024-03-28T23:31:53.892224+0000","last_clean":"2024-03-28T23:31:53.892224+0000","last_became_active":"2024-03-28T23:31:53.888840+0000","last_became_peered":"2024-03-28T23:31:53.888840+0000","last_unstale":"2024-03-28T23:31:53.892224+0000","last_undegraded":"2024-03-28T23:31:53.892224+0000","last_fullsized":"2024-03-28T23:31:53.892224+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:01.360626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876467+0000","last_change":"2024-03-28T23:31:40.405766+0000","last_active":"2024-03-28T23:31:53.876467+0000","last_peered":"2024-03-28T23:31:53.876467+0000","last_clean":"2024-03-28T23:31:53.876467+0000","last_became_active":"2024-03-28T23:31:40.405517+0000","last_became_peered":"2024-03-28T23:31:40.405517+0000","last_unstale":"2024-03-28T23:31:53.876467+0000","last_undegraded":"2024-03-28T23:31:53.876467+0000","last_fullsized":"2024-03-28T23:31:53.876467+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:18:16.851435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.883245+0000","last_change":"2024-03-28T23:31:39.383748+0000","last_active":"2024-03-28T23:31:53.883245+0000","last_peered":"2024-03-28T23:31:53.883245+0000","last_clean":"2024-03-28T23:31:53.883245+0000","last_became_active":"2024-03-28T23:31:39.383070+0000","last_became_peered":"2024-03-28T23:31:39.383070+0000","last_unstale":"2024-03-28T23:31:53.883245+0000","last_undegraded":"2024-03-28T23:31:53.883245+0000","last_fullsized":"2024-03-28T23:31:53.883245+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:41.430571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.873125+0000","last_change":"2024-03-28T23:31:53.873125+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.873125+0000","last_undegraded":"2024-03-28T23:31:53.873125+0000","last_fullsized":"2024-03-28T23:31:53.873125+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.890278+0000","last_change":"2024-03-28T23:31:53.887894+0000","last_active":"2024-03-28T23:31:53.890278+0000","last_peered":"2024-03-28T23:31:53.890278+0000","last_clean":"2024-03-28T23:31:53.890278+0000","last_became_active":"2024-03-28T23:31:53.887619+0000","last_became_peered":"2024-03-28T23:31:53.887619+0000","last_unstale":"2024-03-28T23:31:53.890278+0000","last_undegraded":"2024-03-28T23:31:53.890278+0000","last_fullsized":"2024-03-28T23:31:53.890278+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:26:16.561871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879094+0000","last_change":"2024-03-28T23:31:40.392625+0000","last_active":"2024-03-28T23:31:53.879094+0000","last_peered":"2024-03-28T23:31:53.879094+0000","last_clean":"2024-03-28T23:31:53.879094+0000","last_became_active":"2024-03-28T23:31:40.392359+0000","last_became_peered":"2024-03-28T23:31:40.392359+0000","last_unstale":"2024-03-28T23:31:53.879094+0000","last_undegraded":"2024-03-28T23:31:53.879094+0000","last_fullsized":"2024-03-28T23:31:53.879094+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:02.676183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874872+0000","last_change":"2024-03-28T23:31:39.389478+0000","last_active":"2024-03-28T23:31:53.874872+0000","last_peered":"2024-03-28T23:31:53.874872+0000","last_clean":"2024-03-28T23:31:53.874872+0000","last_became_active":"2024-03-28T23:31:39.389154+0000","last_became_peered":"2024-03-28T23:31:39.389154+0000","last_unstale":"2024-03-28T23:31:53.874872+0000","last_undegraded":"2024-03-28T23:31:53.874872+0000","last_fullsized":"2024-03-28T23:31:53.874872+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:25:19.055737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871716+0000","last_change":"2024-03-28T23:31:53.871716+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871716+0000","last_undegraded":"2024-03-28T23:31:53.871716+0000","last_fullsized":"2024-03-28T23:31:53.871716+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.895972+0000","last_change":"2024-03-28T23:31:53.890575+0000","last_active":"2024-03-28T23:31:53.895972+0000","last_peered":"2024-03-28T23:31:53.895972+0000","last_clean":"2024-03-28T23:31:53.895972+0000","last_became_active":"2024-03-28T23:31:53.890305+0000","last_became_peered":"2024-03-28T23:31:53.890305+0000","last_unstale":"2024-03-28T23:31:53.895972+0000","last_undegraded":"2024-03-28T23:31:53.895972+0000","last_fullsized":"2024-03-28T23:31:53.895972+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:31:25.647111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876751+0000","last_change":"2024-03-28T23:31:40.389210+0000","last_active":"2024-03-28T23:31:53.876751+0000","last_peered":"2024-03-28T23:31:53.876751+0000","last_clean":"2024-03-28T23:31:53.876751+0000","last_became_active":"2024-03-28T23:31:40.388573+0000","last_became_peered":"2024-03-28T23:31:40.388573+0000","last_unstale":"2024-03-28T23:31:53.876751+0000","last_undegraded":"2024-03-28T23:31:53.876751+0000","last_fullsized":"2024-03-28T23:31:53.876751+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:23:53.153369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"51'1","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876646+0000","last_change":"2024-03-28T23:31:39.381202+0000","last_active":"2024-03-28T23:31:53.876646+0000","last_peered":"2024-03-28T23:31:53.876646+0000","last_clean":"2024-03-28T23:31:53.876646+0000","last_became_active":"2024-03-28T23:31:39.380913+0000","last_became_peered":"2024-03-28T23:31:39.380913+0000","last_unstale":"2024-03-28T23:31:53.876646+0000","last_undegraded":"2024-03-28T23:31:53.876646+0000","last_fullsized":"2024-03-28T23:31:53.876646+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:38:21.050492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":233,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.872708+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:31:53.872708+0000","last_peered":"2024-03-28T23:31:53.872708+0000","last_clean":"2024-03-28T23:31:53.872708+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:31:53.872708+0000","last_undegraded":"2024-03-28T23:31:53.872708+0000","last_fullsized":"2024-03-28T23:31:53.872708+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":704512,"data_stored":875469,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574878,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31792,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":626688,"data_stored":278115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574878,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31784,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340056,"statfs":{"total":96636764160,"available":96604217344,"internally_reserved":0,"allocated":618496,"data_stored":277463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574877,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31788,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340052,"statfs":{"total":96636764160,"available":96604213248,"internally_reserved":0,"allocated":622592,"data_stored":278045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":634880,"data_stored":279097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":630784,"data_stored":279075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":285137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":704512,"data_stored":875401,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:31:58.082 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:31:58.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.212+0000 7f2e9fbc9640 1 Processor -- start 2024-03-28T23:31:58.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.212+0000 7f2e9fbc9640 1 -- start start 2024-03-28T23:31:58.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.213+0000 7f2e9fbc9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2e980c5850 0x7f2e98155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:58.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.213+0000 7f2e9fbc9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e980c5d00 con 0x7f2e980c5410 2024-03-28T23:31:58.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.213+0000 7f2e9fbc9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e98158a30 con 0x7f2e98152890 2024-03-28T23:31:58.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.213+0000 7f2e9fbc9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f2e980c8480 con 0x7f2e980c5850 2024-03-28T23:31:58.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.213+0000 7f2e9d164640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f2e980c5410 0x7f2e981553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37814/0 (socket says 172.21.15.161:37814) 2024-03-28T23:31:58.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.213+0000 7f2e9d164640 1 -- 172.21.15.161:0/1847338706 learned_addr learned my addr 172.21.15.161:0/1847338706 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:58.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.213+0000 7f2e9e166640 1 --2- 172.21.15.161:0/1847338706 >> v2:172.21.15.170:3300/0 conn(0x7f2e980c5850 0x7f2e98155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:58.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.213+0000 7f2e9c963640 1 -- 172.21.15.161:0/1847338706 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3455426363 0 0) 0x7f2e980c5d00 con 0x7f2e980c5410 2024-03-28T23:31:58.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.213+0000 7f2e9c963640 1 -- 172.21.15.161:0/1847338706 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e84003520 con 0x7f2e980c5410 2024-03-28T23:31:58.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.214+0000 7f2e9e166640 1 -- 172.21.15.161:0/1847338706 >> v1:172.21.15.170:6789/0 conn(0x7f2e98152890 legacy=0x7f2e980c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:58.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.214+0000 7f2e9e166640 1 -- 172.21.15.161:0/1847338706 >> v1:172.21.15.161:6789/0 conn(0x7f2e980c5410 legacy=0x7f2e981553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:58.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.214+0000 7f2e9c963640 1 -- 172.21.15.161:0/1847338706 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 880156760 0 0) 0x7f2e84003520 con 0x7f2e980c5410 2024-03-28T23:31:58.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.214+0000 7f2e9e166640 1 -- 172.21.15.161:0/1847338706 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e98158d10 con 0x7f2e980c5850 2024-03-28T23:31:58.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.215+0000 7f2e9e166640 1 --2- 172.21.15.161:0/1847338706 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e980c5850 0x7f2e98155ab0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009d90 tx=0x7f2e9002bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2df26f09d6e6e952 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:58.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.215+0000 7f2e9c963640 1 -- 172.21.15.161:0/1847338706 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e90039070 con 0x7f2e980c5850 2024-03-28T23:31:58.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.215+0000 7f2e9c963640 1 -- 172.21.15.161:0/1847338706 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e9002c8b0 con 0x7f2e980c5850 2024-03-28T23:31:58.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.215+0000 7f2e9c963640 1 -- 172.21.15.161:0/1847338706 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e940017f0 con 0x7f2e980c5850 2024-03-28T23:31:58.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.216+0000 7f2e9fbc9640 1 -- 172.21.15.161:0/1847338706 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e980c5850 msgr2=0x7f2e98155ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:58.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.216+0000 7f2e9fbc9640 1 --2- 172.21.15.161:0/1847338706 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e980c5850 0x7f2e98155ab0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009d90 tx=0x7f2e9002bd80 comp rx=0 tx=0).stop 2024-03-28T23:31:58.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.216+0000 7f2e9fbc9640 1 -- 172.21.15.161:0/1847338706 shutdown_connections 2024-03-28T23:31:58.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.216+0000 7f2e9fbc9640 1 --2- 172.21.15.161:0/1847338706 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e980c5850 0x7f2e98155ab0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.216+0000 7f2e9fbc9640 1 -- 172.21.15.161:0/1847338706 >> 172.21.15.161:0/1847338706 conn(0x7f2e980ac620 msgr2=0x7f2e980aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:58.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.216+0000 7f2e9fbc9640 1 -- 172.21.15.161:0/1847338706 shutdown_connections 2024-03-28T23:31:58.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.216+0000 7f2e9fbc9640 1 -- 172.21.15.161:0/1847338706 wait complete. 2024-03-28T23:31:58.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.217+0000 7f2e9fbc9640 1 Processor -- start 2024-03-28T23:31:58.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.217+0000 7f2e9fbc9640 1 -- start start 2024-03-28T23:31:58.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.217+0000 7f2e9fbc9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e980c5410 0x7f2e981eead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:58.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.218+0000 7f2e9fbc9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e980c5850 0x7f2e980ab610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:58.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.218+0000 7f2e9fbc9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e98152890 0x7f2e981537d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:58.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.218+0000 7f2e9fbc9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2e981592d0 con 0x7f2e98152890 2024-03-28T23:31:58.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.218+0000 7f2e9fbc9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2e98159640 con 0x7f2e980c5850 2024-03-28T23:31:58.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.218+0000 7f2e9fbc9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2e98155ff0 con 0x7f2e980c5410 2024-03-28T23:31:58.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.218+0000 7f2e9e166640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e98152890 0x7f2e981537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:58.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.218+0000 7f2e9e166640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e98152890 0x7f2e981537d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35208/0 (socket says 172.21.15.161:35208) 2024-03-28T23:31:58.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.218+0000 7f2e9e166640 1 -- 172.21.15.161:0/4068744936 learned_addr learned my addr 172.21.15.161:0/4068744936 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:58.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.218+0000 7f2e9d164640 1 --2- 172.21.15.161:0/4068744936 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e980c5850 0x7f2e980ab610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:58.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.218+0000 7f2e9d965640 1 --2- 172.21.15.161:0/4068744936 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e980c5410 0x7f2e981eead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:58.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.218+0000 7f2e9e166640 1 -- 172.21.15.161:0/4068744936 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e980c5410 msgr2=0x7f2e981eead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:58.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.218+0000 7f2e9e166640 1 --2- 172.21.15.161:0/4068744936 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e980c5410 0x7f2e981eead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.218+0000 7f2e9e166640 1 -- 172.21.15.161:0/4068744936 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e980c5850 msgr2=0x7f2e980ab610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:58.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.218+0000 7f2e9e166640 1 --2- 172.21.15.161:0/4068744936 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e980c5850 0x7f2e980ab610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.218+0000 7f2e9e166640 1 -- 172.21.15.161:0/4068744936 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e98154a90 con 0x7f2e98152890 2024-03-28T23:31:58.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.218+0000 7f2e9d965640 1 --2- 172.21.15.161:0/4068744936 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e980c5410 0x7f2e981eead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:58.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.218+0000 7f2e9e166640 1 --2- 172.21.15.161:0/4068744936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e98152890 0x7f2e981537d0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f2e90037040 tx=0x7f2e90004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:58.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.218+0000 7f2e8e7fc640 1 -- 172.21.15.161:0/4068744936 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e90044020 con 0x7f2e98152890 2024-03-28T23:31:58.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.218+0000 7f2e8e7fc640 1 -- 172.21.15.161:0/4068744936 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e90034cf0 con 0x7f2e98152890 2024-03-28T23:31:58.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.218+0000 7f2e8e7fc640 1 -- 172.21.15.161:0/4068744936 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e90039040 con 0x7f2e98152890 2024-03-28T23:31:58.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.218+0000 7f2e9fbc9640 1 -- 172.21.15.161:0/4068744936 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e98154d70 con 0x7f2e98152890 2024-03-28T23:31:58.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.218+0000 7f2e9fbc9640 1 -- 172.21.15.161:0/4068744936 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e981532b0 con 0x7f2e98152890 2024-03-28T23:31:58.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.219+0000 7f2e9fbc9640 1 -- 172.21.15.161:0/4068744936 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e9804f4b0 con 0x7f2e98152890 2024-03-28T23:31:58.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.220+0000 7f2e8e7fc640 1 -- 172.21.15.161:0/4068744936 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2e90047020 con 0x7f2e98152890 2024-03-28T23:31:58.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.220+0000 7f2e8e7fc640 1 --2- 172.21.15.161:0/4068744936 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e7c06b4d0 0x7f2e7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:58.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.220+0000 7f2e8e7fc640 1 -- 172.21.15.161:0/4068744936 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6957+0+0 (secure 0 0 0) 0x7f2e900ace80 con 0x7f2e98152890 2024-03-28T23:31:58.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.220+0000 7f2e9d965640 1 --2- 172.21.15.161:0/4068744936 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e7c06b4d0 0x7f2e7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:58.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.221+0000 7f2e9d965640 1 --2- 172.21.15.161:0/4068744936 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e7c06b4d0 0x7f2e7c06d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2e88000f60 tx=0x7f2e88009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:58.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.228+0000 7f2e8e7fc640 1 -- 172.21.15.161:0/4068744936 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2e9004e030 con 0x7f2e98152890 2024-03-28T23:31:58.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.404+0000 7f2e9fbc9640 1 -- 172.21.15.161:0/4068744936 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2e98151bb0 con 0x7f2e7c06b4d0 2024-03-28T23:31:58.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.415+0000 7f2e8e7fc640 1 -- 172.21.15.161:0/4068744936 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+485262 (secure 0 0 0) 0x7f2e98151bb0 con 0x7f2e7c06b4d0 2024-03-28T23:31:58.421 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:58.424 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:31:58.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.419+0000 7f2e9fbc9640 1 -- 172.21.15.161:0/4068744936 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e7c06b4d0 msgr2=0x7f2e7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:58.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.419+0000 7f2e9fbc9640 1 --2- 172.21.15.161:0/4068744936 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e7c06b4d0 0x7f2e7c06d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2e88000f60 tx=0x7f2e88009040 comp rx=0 tx=0).stop 2024-03-28T23:31:58.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.419+0000 7f2e9fbc9640 1 -- 172.21.15.161:0/4068744936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e98152890 msgr2=0x7f2e981537d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:58.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.419+0000 7f2e9fbc9640 1 --2- 172.21.15.161:0/4068744936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e98152890 0x7f2e981537d0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f2e90037040 tx=0x7f2e90004a40 comp rx=0 tx=0).stop 2024-03-28T23:31:58.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.419+0000 7f2e9fbc9640 1 -- 172.21.15.161:0/4068744936 shutdown_connections 2024-03-28T23:31:58.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.419+0000 7f2e9fbc9640 1 --2- 172.21.15.161:0/4068744936 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e7c06b4d0 0x7f2e7c06d990 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.419+0000 7f2e9fbc9640 1 --2- 172.21.15.161:0/4068744936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e98152890 0x7f2e981537d0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.419+0000 7f2e9fbc9640 1 --2- 172.21.15.161:0/4068744936 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e980c5850 0x7f2e980ab610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.419+0000 7f2e9fbc9640 1 --2- 172.21.15.161:0/4068744936 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e980c5410 0x7f2e981eead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.419+0000 7f2e9fbc9640 1 -- 172.21.15.161:0/4068744936 >> 172.21.15.161:0/4068744936 conn(0x7f2e980ac620 msgr2=0x7f2e9814ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:58.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.419+0000 7f2e9fbc9640 1 -- 172.21.15.161:0/4068744936 shutdown_connections 2024-03-28T23:31:58.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.419+0000 7f2e9fbc9640 1 -- 172.21.15.161:0/4068744936 wait complete. 2024-03-28T23:31:58.434 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2024-03-28T23:31:57.336508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":232,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237868,"kb_used_data":5060,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736852,"statfs":{"total":773094113280,"available":772850536448,"internally_reserved":0,"allocated":5181440,"data_stored":3427802,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.470377"},"pg_stats":[{"pgid":"10.3d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872212+0000","last_change":"2024-03-28T23:31:53.872212+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872212+0000","last_undegraded":"2024-03-28T23:31:53.872212+0000","last_fullsized":"2024-03-28T23:31:53.872212+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3f","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.870419+0000","last_change":"2024-03-28T23:31:40.388090+0000","last_active":"2024-03-28T23:31:53.870419+0000","last_peered":"2024-03-28T23:31:53.870419+0000","last_clean":"2024-03-28T23:31:53.870419+0000","last_became_active":"2024-03-28T23:31:40.387805+0000","last_became_peered":"2024-03-28T23:31:40.387805+0000","last_unstale":"2024-03-28T23:31:53.870419+0000","last_undegraded":"2024-03-28T23:31:53.870419+0000","last_fullsized":"2024-03-28T23:31:53.870419+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:19:47.987753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.881085+0000","last_change":"2024-03-28T23:31:53.881085+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.881085+0000","last_undegraded":"2024-03-28T23:31:53.881085+0000","last_fullsized":"2024-03-28T23:31:53.881085+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875872+0000","last_change":"2024-03-28T23:31:40.395246+0000","last_active":"2024-03-28T23:31:53.875872+0000","last_peered":"2024-03-28T23:31:53.875872+0000","last_clean":"2024-03-28T23:31:53.875872+0000","last_became_active":"2024-03-28T23:31:40.394583+0000","last_became_peered":"2024-03-28T23:31:40.394583+0000","last_unstale":"2024-03-28T23:31:53.875872+0000","last_undegraded":"2024-03-28T23:31:53.875872+0000","last_fullsized":"2024-03-28T23:31:53.875872+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:18.654331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.873340+0000","last_change":"2024-03-28T23:31:53.873340+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.873340+0000","last_undegraded":"2024-03-28T23:31:53.873340+0000","last_fullsized":"2024-03-28T23:31:53.873340+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876107+0000","last_change":"2024-03-28T23:31:40.397914+0000","last_active":"2024-03-28T23:31:53.876107+0000","last_peered":"2024-03-28T23:31:53.876107+0000","last_clean":"2024-03-28T23:31:53.876107+0000","last_became_active":"2024-03-28T23:31:40.394904+0000","last_became_peered":"2024-03-28T23:31:40.394904+0000","last_unstale":"2024-03-28T23:31:53.876107+0000","last_undegraded":"2024-03-28T23:31:53.876107+0000","last_fullsized":"2024-03-28T23:31:53.876107+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:26:03.062546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872399+0000","last_change":"2024-03-28T23:31:53.872399+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872399+0000","last_undegraded":"2024-03-28T23:31:53.872399+0000","last_fullsized":"2024-03-28T23:31:53.872399+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874319+0000","last_change":"2024-03-28T23:31:40.400594+0000","last_active":"2024-03-28T23:31:53.874319+0000","last_peered":"2024-03-28T23:31:53.874319+0000","last_clean":"2024-03-28T23:31:53.874319+0000","last_became_active":"2024-03-28T23:31:40.400255+0000","last_became_peered":"2024-03-28T23:31:40.400255+0000","last_unstale":"2024-03-28T23:31:53.874319+0000","last_undegraded":"2024-03-28T23:31:53.874319+0000","last_fullsized":"2024-03-28T23:31:53.874319+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:14.515069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.877696+0000","last_change":"2024-03-28T23:31:53.877696+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.877696+0000","last_undegraded":"2024-03-28T23:31:53.877696+0000","last_fullsized":"2024-03-28T23:31:53.877696+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.884068+0000","last_change":"2024-03-28T23:31:40.388531+0000","last_active":"2024-03-28T23:31:53.884068+0000","last_peered":"2024-03-28T23:31:53.884068+0000","last_clean":"2024-03-28T23:31:53.884068+0000","last_became_active":"2024-03-28T23:31:40.388206+0000","last_became_peered":"2024-03-28T23:31:40.388206+0000","last_unstale":"2024-03-28T23:31:53.884068+0000","last_undegraded":"2024-03-28T23:31:53.884068+0000","last_fullsized":"2024-03-28T23:31:53.884068+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:23.215366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.881544+0000","last_change":"2024-03-28T23:31:53.881544+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.881544+0000","last_undegraded":"2024-03-28T23:31:53.881544+0000","last_fullsized":"2024-03-28T23:31:53.881544+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869622+0000","last_change":"2024-03-28T23:31:40.393907+0000","last_active":"2024-03-28T23:31:53.869622+0000","last_peered":"2024-03-28T23:31:53.869622+0000","last_clean":"2024-03-28T23:31:53.869622+0000","last_became_active":"2024-03-28T23:31:40.393443+0000","last_became_peered":"2024-03-28T23:31:40.393443+0000","last_unstale":"2024-03-28T23:31:53.869622+0000","last_undegraded":"2024-03-28T23:31:53.869622+0000","last_fullsized":"2024-03-28T23:31:53.869622+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:28:29.180775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872259+0000","last_change":"2024-03-28T23:31:53.872259+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872259+0000","last_undegraded":"2024-03-28T23:31:53.872259+0000","last_fullsized":"2024-03-28T23:31:53.872259+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875554+0000","last_change":"2024-03-28T23:31:40.398876+0000","last_active":"2024-03-28T23:31:53.875554+0000","last_peered":"2024-03-28T23:31:53.875554+0000","last_clean":"2024-03-28T23:31:53.875554+0000","last_became_active":"2024-03-28T23:31:40.398595+0000","last_became_peered":"2024-03-28T23:31:40.398595+0000","last_unstale":"2024-03-28T23:31:53.875554+0000","last_undegraded":"2024-03-28T23:31:53.875554+0000","last_fullsized":"2024-03-28T23:31:53.875554+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:21.725238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874359+0000","last_change":"2024-03-28T23:31:53.874359+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874359+0000","last_undegraded":"2024-03-28T23:31:53.874359+0000","last_fullsized":"2024-03-28T23:31:53.874359+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.880118+0000","last_change":"2024-03-28T23:31:40.392421+0000","last_active":"2024-03-28T23:31:53.880118+0000","last_peered":"2024-03-28T23:31:53.880118+0000","last_clean":"2024-03-28T23:31:53.880118+0000","last_became_active":"2024-03-28T23:31:40.391463+0000","last_became_peered":"2024-03-28T23:31:40.391463+0000","last_unstale":"2024-03-28T23:31:53.880118+0000","last_undegraded":"2024-03-28T23:31:53.880118+0000","last_fullsized":"2024-03-28T23:31:53.880118+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:18:16.851435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878422+0000","last_change":"2024-03-28T23:31:53.878422+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878422+0000","last_undegraded":"2024-03-28T23:31:53.878422+0000","last_fullsized":"2024-03-28T23:31:53.878422+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.878936+0000","last_change":"2024-03-28T23:31:40.403340+0000","last_active":"2024-03-28T23:31:53.878936+0000","last_peered":"2024-03-28T23:31:53.878936+0000","last_clean":"2024-03-28T23:31:53.878936+0000","last_became_active":"2024-03-28T23:31:40.403089+0000","last_became_peered":"2024-03-28T23:31:40.403089+0000","last_unstale":"2024-03-28T23:31:53.878936+0000","last_undegraded":"2024-03-28T23:31:53.878936+0000","last_fullsized":"2024-03-28T23:31:53.878936+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:12:38.242108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.883968+0000","last_change":"2024-03-28T23:31:53.883968+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.883968+0000","last_undegraded":"2024-03-28T23:31:53.883968+0000","last_fullsized":"2024-03-28T23:31:53.883968+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.870177+0000","last_change":"2024-03-28T23:31:40.382204+0000","last_active":"2024-03-28T23:31:53.870177+0000","last_peered":"2024-03-28T23:31:53.870177+0000","last_clean":"2024-03-28T23:31:53.870177+0000","last_became_active":"2024-03-28T23:31:40.381944+0000","last_became_peered":"2024-03-28T23:31:40.381944+0000","last_unstale":"2024-03-28T23:31:53.870177+0000","last_undegraded":"2024-03-28T23:31:53.870177+0000","last_fullsized":"2024-03-28T23:31:53.870177+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:14.515069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871125+0000","last_change":"2024-03-28T23:31:53.871125+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871125+0000","last_undegraded":"2024-03-28T23:31:53.871125+0000","last_fullsized":"2024-03-28T23:31:53.871125+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.883144+0000","last_change":"2024-03-28T23:31:40.394578+0000","last_active":"2024-03-28T23:31:53.883144+0000","last_peered":"2024-03-28T23:31:53.883144+0000","last_clean":"2024-03-28T23:31:53.883144+0000","last_became_active":"2024-03-28T23:31:40.394272+0000","last_became_peered":"2024-03-28T23:31:40.394272+0000","last_unstale":"2024-03-28T23:31:53.883144+0000","last_undegraded":"2024-03-28T23:31:53.883144+0000","last_fullsized":"2024-03-28T23:31:53.883144+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:24:35.708796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871848+0000","last_change":"2024-03-28T23:31:53.871848+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871848+0000","last_undegraded":"2024-03-28T23:31:53.871848+0000","last_fullsized":"2024-03-28T23:31:53.871848+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879847+0000","last_change":"2024-03-28T23:31:40.396702+0000","last_active":"2024-03-28T23:31:53.879847+0000","last_peered":"2024-03-28T23:31:53.879847+0000","last_clean":"2024-03-28T23:31:53.879847+0000","last_became_active":"2024-03-28T23:31:40.395941+0000","last_became_peered":"2024-03-28T23:31:40.395941+0000","last_unstale":"2024-03-28T23:31:53.879847+0000","last_undegraded":"2024-03-28T23:31:53.879847+0000","last_fullsized":"2024-03-28T23:31:53.879847+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:07:13.690791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871018+0000","last_change":"2024-03-28T23:31:53.871018+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871018+0000","last_undegraded":"2024-03-28T23:31:53.871018+0000","last_fullsized":"2024-03-28T23:31:53.871018+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869713+0000","last_change":"2024-03-28T23:31:40.382947+0000","last_active":"2024-03-28T23:31:53.869713+0000","last_peered":"2024-03-28T23:31:53.869713+0000","last_clean":"2024-03-28T23:31:53.869713+0000","last_became_active":"2024-03-28T23:31:40.381893+0000","last_became_peered":"2024-03-28T23:31:40.381893+0000","last_unstale":"2024-03-28T23:31:53.869713+0000","last_undegraded":"2024-03-28T23:31:53.869713+0000","last_fullsized":"2024-03-28T23:31:53.869713+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:50.872886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.877447+0000","last_change":"2024-03-28T23:31:53.877447+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.877447+0000","last_undegraded":"2024-03-28T23:31:53.877447+0000","last_fullsized":"2024-03-28T23:31:53.877447+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875836+0000","last_change":"2024-03-28T23:31:40.405627+0000","last_active":"2024-03-28T23:31:53.875836+0000","last_peered":"2024-03-28T23:31:53.875836+0000","last_clean":"2024-03-28T23:31:53.875836+0000","last_became_active":"2024-03-28T23:31:40.405360+0000","last_became_peered":"2024-03-28T23:31:40.405360+0000","last_unstale":"2024-03-28T23:31:53.875836+0000","last_undegraded":"2024-03-28T23:31:53.875836+0000","last_fullsized":"2024-03-28T23:31:53.875836+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:06.567694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.881048+0000","last_change":"2024-03-28T23:31:53.881048+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.881048+0000","last_undegraded":"2024-03-28T23:31:53.881048+0000","last_fullsized":"2024-03-28T23:31:53.881048+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.873535+0000","last_change":"2024-03-28T23:31:40.396605+0000","last_active":"2024-03-28T23:31:53.873535+0000","last_peered":"2024-03-28T23:31:53.873535+0000","last_clean":"2024-03-28T23:31:53.873535+0000","last_became_active":"2024-03-28T23:31:40.396237+0000","last_became_peered":"2024-03-28T23:31:40.396237+0000","last_unstale":"2024-03-28T23:31:53.873535+0000","last_undegraded":"2024-03-28T23:31:53.873535+0000","last_fullsized":"2024-03-28T23:31:53.873535+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:14.144440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871402+0000","last_change":"2024-03-28T23:31:53.871402+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871402+0000","last_undegraded":"2024-03-28T23:31:53.871402+0000","last_fullsized":"2024-03-28T23:31:53.871402+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869477+0000","last_change":"2024-03-28T23:31:40.382823+0000","last_active":"2024-03-28T23:31:53.869477+0000","last_peered":"2024-03-28T23:31:53.869477+0000","last_clean":"2024-03-28T23:31:53.869477+0000","last_became_active":"2024-03-28T23:31:40.382574+0000","last_became_peered":"2024-03-28T23:31:40.382574+0000","last_unstale":"2024-03-28T23:31:53.869477+0000","last_undegraded":"2024-03-28T23:31:53.869477+0000","last_fullsized":"2024-03-28T23:31:53.869477+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:12:27.477720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872681+0000","last_change":"2024-03-28T23:31:53.872681+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872681+0000","last_undegraded":"2024-03-28T23:31:53.872681+0000","last_fullsized":"2024-03-28T23:31:53.872681+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876715+0000","last_change":"2024-03-28T23:31:40.403462+0000","last_active":"2024-03-28T23:31:53.876715+0000","last_peered":"2024-03-28T23:31:53.876715+0000","last_clean":"2024-03-28T23:31:53.876715+0000","last_became_active":"2024-03-28T23:31:40.402885+0000","last_became_peered":"2024-03-28T23:31:40.402885+0000","last_unstale":"2024-03-28T23:31:53.876715+0000","last_undegraded":"2024-03-28T23:31:53.876715+0000","last_fullsized":"2024-03-28T23:31:53.876715+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:42:29.351941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.879388+0000","last_change":"2024-03-28T23:31:53.879388+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.879388+0000","last_undegraded":"2024-03-28T23:31:53.879388+0000","last_fullsized":"2024-03-28T23:31:53.879388+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875969+0000","last_change":"2024-03-28T23:31:40.395478+0000","last_active":"2024-03-28T23:31:53.875969+0000","last_peered":"2024-03-28T23:31:53.875969+0000","last_clean":"2024-03-28T23:31:53.875969+0000","last_became_active":"2024-03-28T23:31:40.394981+0000","last_became_peered":"2024-03-28T23:31:40.394981+0000","last_unstale":"2024-03-28T23:31:53.875969+0000","last_undegraded":"2024-03-28T23:31:53.875969+0000","last_fullsized":"2024-03-28T23:31:53.875969+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:06.567694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872394+0000","last_change":"2024-03-28T23:31:53.872394+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872394+0000","last_undegraded":"2024-03-28T23:31:53.872394+0000","last_fullsized":"2024-03-28T23:31:53.872394+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876119+0000","last_change":"2024-03-28T23:31:40.394475+0000","last_active":"2024-03-28T23:31:53.876119+0000","last_peered":"2024-03-28T23:31:53.876119+0000","last_clean":"2024-03-28T23:31:53.876119+0000","last_became_active":"2024-03-28T23:31:40.390205+0000","last_became_peered":"2024-03-28T23:31:40.390205+0000","last_unstale":"2024-03-28T23:31:53.876119+0000","last_undegraded":"2024-03-28T23:31:53.876119+0000","last_fullsized":"2024-03-28T23:31:53.876119+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:04:48.226221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.882562+0000","last_change":"2024-03-28T23:31:53.882562+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.882562+0000","last_undegraded":"2024-03-28T23:31:53.882562+0000","last_fullsized":"2024-03-28T23:31:53.882562+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.877115+0000","last_change":"2024-03-28T23:31:40.388386+0000","last_active":"2024-03-28T23:31:53.877115+0000","last_peered":"2024-03-28T23:31:53.877115+0000","last_clean":"2024-03-28T23:31:53.877115+0000","last_became_active":"2024-03-28T23:31:40.388043+0000","last_became_peered":"2024-03-28T23:31:40.388043+0000","last_unstale":"2024-03-28T23:31:53.877115+0000","last_undegraded":"2024-03-28T23:31:53.877115+0000","last_fullsized":"2024-03-28T23:31:53.877115+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:07.738674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.880460+0000","last_change":"2024-03-28T23:31:53.880460+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.880460+0000","last_undegraded":"2024-03-28T23:31:53.880460+0000","last_fullsized":"2024-03-28T23:31:53.880460+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875638+0000","last_change":"2024-03-28T23:31:40.395811+0000","last_active":"2024-03-28T23:31:53.875638+0000","last_peered":"2024-03-28T23:31:53.875638+0000","last_clean":"2024-03-28T23:31:53.875638+0000","last_became_active":"2024-03-28T23:31:40.395540+0000","last_became_peered":"2024-03-28T23:31:40.395540+0000","last_unstale":"2024-03-28T23:31:53.875638+0000","last_undegraded":"2024-03-28T23:31:53.875638+0000","last_fullsized":"2024-03-28T23:31:53.875638+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:51.864986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.870958+0000","last_change":"2024-03-28T23:31:53.870958+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.870958+0000","last_undegraded":"2024-03-28T23:31:53.870958+0000","last_fullsized":"2024-03-28T23:31:53.870958+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.881134+0000","last_change":"2024-03-28T23:31:40.403129+0000","last_active":"2024-03-28T23:31:53.881134+0000","last_peered":"2024-03-28T23:31:53.881134+0000","last_clean":"2024-03-28T23:31:53.881134+0000","last_became_active":"2024-03-28T23:31:40.402879+0000","last_became_peered":"2024-03-28T23:31:40.402879+0000","last_unstale":"2024-03-28T23:31:53.881134+0000","last_undegraded":"2024-03-28T23:31:53.881134+0000","last_fullsized":"2024-03-28T23:31:53.881134+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:47.861300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.879206+0000","last_change":"2024-03-28T23:31:53.879206+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.879206+0000","last_undegraded":"2024-03-28T23:31:53.879206+0000","last_fullsized":"2024-03-28T23:31:53.879206+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879397+0000","last_change":"2024-03-28T23:31:40.389854+0000","last_active":"2024-03-28T23:31:53.879397+0000","last_peered":"2024-03-28T23:31:53.879397+0000","last_clean":"2024-03-28T23:31:53.879397+0000","last_became_active":"2024-03-28T23:31:40.389599+0000","last_became_peered":"2024-03-28T23:31:40.389599+0000","last_unstale":"2024-03-28T23:31:53.879397+0000","last_undegraded":"2024-03-28T23:31:53.879397+0000","last_fullsized":"2024-03-28T23:31:53.879397+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:43:37.883841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.880913+0000","last_change":"2024-03-28T23:31:53.880913+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.880913+0000","last_undegraded":"2024-03-28T23:31:53.880913+0000","last_fullsized":"2024-03-28T23:31:53.880913+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875417+0000","last_change":"2024-03-28T23:31:40.406317+0000","last_active":"2024-03-28T23:31:53.875417+0000","last_peered":"2024-03-28T23:31:53.875417+0000","last_clean":"2024-03-28T23:31:53.875417+0000","last_became_active":"2024-03-28T23:31:40.405217+0000","last_became_peered":"2024-03-28T23:31:40.405217+0000","last_unstale":"2024-03-28T23:31:53.875417+0000","last_undegraded":"2024-03-28T23:31:53.875417+0000","last_fullsized":"2024-03-28T23:31:53.875417+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:37:55.870956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.879370+0000","last_change":"2024-03-28T23:31:53.879370+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.879370+0000","last_undegraded":"2024-03-28T23:31:53.879370+0000","last_fullsized":"2024-03-28T23:31:53.879370+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876750+0000","last_change":"2024-03-28T23:31:40.404249+0000","last_active":"2024-03-28T23:31:53.876750+0000","last_peered":"2024-03-28T23:31:53.876750+0000","last_clean":"2024-03-28T23:31:53.876750+0000","last_became_active":"2024-03-28T23:31:40.403966+0000","last_became_peered":"2024-03-28T23:31:40.403966+0000","last_unstale":"2024-03-28T23:31:53.876750+0000","last_undegraded":"2024-03-28T23:31:53.876750+0000","last_fullsized":"2024-03-28T23:31:53.876750+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:40:57.541676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872876+0000","last_change":"2024-03-28T23:31:53.872876+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872876+0000","last_undegraded":"2024-03-28T23:31:53.872876+0000","last_fullsized":"2024-03-28T23:31:53.872876+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.878912+0000","last_change":"2024-03-28T23:31:40.397170+0000","last_active":"2024-03-28T23:31:53.878912+0000","last_peered":"2024-03-28T23:31:53.878912+0000","last_clean":"2024-03-28T23:31:53.878912+0000","last_became_active":"2024-03-28T23:31:40.396898+0000","last_became_peered":"2024-03-28T23:31:40.396898+0000","last_unstale":"2024-03-28T23:31:53.878912+0000","last_undegraded":"2024-03-28T23:31:53.878912+0000","last_fullsized":"2024-03-28T23:31:53.878912+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:53:26.355348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878874+0000","last_change":"2024-03-28T23:31:53.878874+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878874+0000","last_undegraded":"2024-03-28T23:31:53.878874+0000","last_fullsized":"2024-03-28T23:31:53.878874+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.870080+0000","last_change":"2024-03-28T23:31:40.404955+0000","last_active":"2024-03-28T23:31:53.870080+0000","last_peered":"2024-03-28T23:31:53.870080+0000","last_clean":"2024-03-28T23:31:53.870080+0000","last_became_active":"2024-03-28T23:31:40.404689+0000","last_became_peered":"2024-03-28T23:31:40.404689+0000","last_unstale":"2024-03-28T23:31:53.870080+0000","last_undegraded":"2024-03-28T23:31:53.870080+0000","last_fullsized":"2024-03-28T23:31:53.870080+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:50:20.574648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878430+0000","last_change":"2024-03-28T23:31:53.878430+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878430+0000","last_undegraded":"2024-03-28T23:31:53.878430+0000","last_fullsized":"2024-03-28T23:31:53.878430+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.877218+0000","last_change":"2024-03-28T23:31:40.389261+0000","last_active":"2024-03-28T23:31:53.877218+0000","last_peered":"2024-03-28T23:31:53.877218+0000","last_clean":"2024-03-28T23:31:53.877218+0000","last_became_active":"2024-03-28T23:31:40.388930+0000","last_became_peered":"2024-03-28T23:31:40.388930+0000","last_unstale":"2024-03-28T23:31:53.877218+0000","last_undegraded":"2024-03-28T23:31:53.877218+0000","last_fullsized":"2024-03-28T23:31:53.877218+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:23.215366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878783+0000","last_change":"2024-03-28T23:31:53.878783+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878783+0000","last_undegraded":"2024-03-28T23:31:53.878783+0000","last_fullsized":"2024-03-28T23:31:53.878783+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875578+0000","last_change":"2024-03-28T23:31:40.401673+0000","last_active":"2024-03-28T23:31:53.875578+0000","last_peered":"2024-03-28T23:31:53.875578+0000","last_clean":"2024-03-28T23:31:53.875578+0000","last_became_active":"2024-03-28T23:31:40.398923+0000","last_became_peered":"2024-03-28T23:31:40.398923+0000","last_unstale":"2024-03-28T23:31:53.875578+0000","last_undegraded":"2024-03-28T23:31:53.875578+0000","last_fullsized":"2024-03-28T23:31:53.875578+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:18.783120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874991+0000","last_change":"2024-03-28T23:31:53.874991+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874991+0000","last_undegraded":"2024-03-28T23:31:53.874991+0000","last_fullsized":"2024-03-28T23:31:53.874991+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.873933+0000","last_change":"2024-03-28T23:31:40.387082+0000","last_active":"2024-03-28T23:31:53.873933+0000","last_peered":"2024-03-28T23:31:53.873933+0000","last_clean":"2024-03-28T23:31:53.873933+0000","last_became_active":"2024-03-28T23:31:40.386808+0000","last_became_peered":"2024-03-28T23:31:40.386808+0000","last_unstale":"2024-03-28T23:31:53.873933+0000","last_undegraded":"2024-03-28T23:31:53.873933+0000","last_fullsized":"2024-03-28T23:31:53.873933+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:43.111058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.870965+0000","last_change":"2024-03-28T23:31:53.870965+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.870965+0000","last_undegraded":"2024-03-28T23:31:53.870965+0000","last_fullsized":"2024-03-28T23:31:53.870965+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875611+0000","last_change":"2024-03-28T23:31:40.397815+0000","last_active":"2024-03-28T23:31:53.875611+0000","last_peered":"2024-03-28T23:31:53.875611+0000","last_clean":"2024-03-28T23:31:53.875611+0000","last_became_active":"2024-03-28T23:31:40.397279+0000","last_became_peered":"2024-03-28T23:31:40.397279+0000","last_unstale":"2024-03-28T23:31:53.875611+0000","last_undegraded":"2024-03-28T23:31:53.875611+0000","last_fullsized":"2024-03-28T23:31:53.875611+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:25:57.449014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878422+0000","last_change":"2024-03-28T23:31:53.878422+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878422+0000","last_undegraded":"2024-03-28T23:31:53.878422+0000","last_fullsized":"2024-03-28T23:31:53.878422+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874691+0000","last_change":"2024-03-28T23:31:40.394484+0000","last_active":"2024-03-28T23:31:53.874691+0000","last_peered":"2024-03-28T23:31:53.874691+0000","last_clean":"2024-03-28T23:31:53.874691+0000","last_became_active":"2024-03-28T23:31:40.394219+0000","last_became_peered":"2024-03-28T23:31:40.394219+0000","last_unstale":"2024-03-28T23:31:53.874691+0000","last_undegraded":"2024-03-28T23:31:53.874691+0000","last_fullsized":"2024-03-28T23:31:53.874691+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:18.654331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874735+0000","last_change":"2024-03-28T23:31:53.874735+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874735+0000","last_undegraded":"2024-03-28T23:31:53.874735+0000","last_fullsized":"2024-03-28T23:31:53.874735+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891397+0000","last_change":"2024-03-28T23:31:53.890105+0000","last_active":"2024-03-28T23:31:53.891397+0000","last_peered":"2024-03-28T23:31:53.891397+0000","last_clean":"2024-03-28T23:31:53.891397+0000","last_became_active":"2024-03-28T23:31:53.889841+0000","last_became_peered":"2024-03-28T23:31:53.889841+0000","last_unstale":"2024-03-28T23:31:53.891397+0000","last_undegraded":"2024-03-28T23:31:53.891397+0000","last_fullsized":"2024-03-28T23:31:53.891397+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:01.360626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.872604+0000","last_change":"2024-03-28T23:31:40.398232+0000","last_active":"2024-03-28T23:31:53.872604+0000","last_peered":"2024-03-28T23:31:53.872604+0000","last_clean":"2024-03-28T23:31:53.872604+0000","last_became_active":"2024-03-28T23:31:40.397888+0000","last_became_peered":"2024-03-28T23:31:40.397888+0000","last_unstale":"2024-03-28T23:31:53.872604+0000","last_undegraded":"2024-03-28T23:31:53.872604+0000","last_fullsized":"2024-03-28T23:31:53.872604+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:50.872886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"50'4","reported_seq":22,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879785+0000","last_change":"2024-03-28T23:31:39.385488+0000","last_active":"2024-03-28T23:31:53.879785+0000","last_peered":"2024-03-28T23:31:53.879785+0000","last_clean":"2024-03-28T23:31:53.879785+0000","last_became_active":"2024-03-28T23:31:39.385211+0000","last_became_peered":"2024-03-28T23:31:39.385211+0000","last_unstale":"2024-03-28T23:31:53.879785+0000","last_undegraded":"2024-03-28T23:31:53.879785+0000","last_fullsized":"2024-03-28T23:31:53.879785+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:50:10.559743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872455+0000","last_change":"2024-03-28T23:31:53.872455+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872455+0000","last_undegraded":"2024-03-28T23:31:53.872455+0000","last_fullsized":"2024-03-28T23:31:53.872455+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891253+0000","last_change":"2024-03-28T23:31:53.887853+0000","last_active":"2024-03-28T23:31:53.891253+0000","last_peered":"2024-03-28T23:31:53.891253+0000","last_clean":"2024-03-28T23:31:53.891253+0000","last_became_active":"2024-03-28T23:31:53.887588+0000","last_became_peered":"2024-03-28T23:31:53.887588+0000","last_unstale":"2024-03-28T23:31:53.891253+0000","last_undegraded":"2024-03-28T23:31:53.891253+0000","last_fullsized":"2024-03-28T23:31:53.891253+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:28:42.680101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.873999+0000","last_change":"2024-03-28T23:31:40.390218+0000","last_active":"2024-03-28T23:31:53.873999+0000","last_peered":"2024-03-28T23:31:53.873999+0000","last_clean":"2024-03-28T23:31:53.873999+0000","last_became_active":"2024-03-28T23:31:40.389922+0000","last_became_peered":"2024-03-28T23:31:40.389922+0000","last_unstale":"2024-03-28T23:31:53.873999+0000","last_undegraded":"2024-03-28T23:31:53.873999+0000","last_fullsized":"2024-03-28T23:31:53.873999+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:14.515069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"50'1","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.870270+0000","last_change":"2024-03-28T23:31:39.393513+0000","last_active":"2024-03-28T23:31:53.870270+0000","last_peered":"2024-03-28T23:31:53.870270+0000","last_clean":"2024-03-28T23:31:53.870270+0000","last_became_active":"2024-03-28T23:31:39.393176+0000","last_became_peered":"2024-03-28T23:31:39.393176+0000","last_unstale":"2024-03-28T23:31:53.870270+0000","last_undegraded":"2024-03-28T23:31:53.870270+0000","last_fullsized":"2024-03-28T23:31:53.870270+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:29.240985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.880671+0000","last_change":"2024-03-28T23:31:53.880671+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.880671+0000","last_undegraded":"2024-03-28T23:31:53.880671+0000","last_fullsized":"2024-03-28T23:31:53.880671+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891931+0000","last_change":"2024-03-28T23:31:53.890699+0000","last_active":"2024-03-28T23:31:53.891931+0000","last_peered":"2024-03-28T23:31:53.891931+0000","last_clean":"2024-03-28T23:31:53.891931+0000","last_became_active":"2024-03-28T23:31:53.890300+0000","last_became_peered":"2024-03-28T23:31:53.890300+0000","last_unstale":"2024-03-28T23:31:53.891931+0000","last_undegraded":"2024-03-28T23:31:53.891931+0000","last_fullsized":"2024-03-28T23:31:53.891931+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:18:30.350761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875566+0000","last_change":"2024-03-28T23:31:40.397077+0000","last_active":"2024-03-28T23:31:53.875566+0000","last_peered":"2024-03-28T23:31:53.875566+0000","last_clean":"2024-03-28T23:31:53.875566+0000","last_became_active":"2024-03-28T23:31:40.395265+0000","last_became_peered":"2024-03-28T23:31:40.395265+0000","last_unstale":"2024-03-28T23:31:53.875566+0000","last_undegraded":"2024-03-28T23:31:53.875566+0000","last_fullsized":"2024-03-28T23:31:53.875566+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:04.829030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"50'2","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.870290+0000","last_change":"2024-03-28T23:31:39.391743+0000","last_active":"2024-03-28T23:31:53.870290+0000","last_peered":"2024-03-28T23:31:53.870290+0000","last_clean":"2024-03-28T23:31:53.870290+0000","last_became_active":"2024-03-28T23:31:39.391300+0000","last_became_peered":"2024-03-28T23:31:39.391300+0000","last_unstale":"2024-03-28T23:31:53.870290+0000","last_undegraded":"2024-03-28T23:31:53.870290+0000","last_fullsized":"2024-03-28T23:31:53.870290+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:12.505382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.877711+0000","last_change":"2024-03-28T23:31:53.877711+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.877711+0000","last_undegraded":"2024-03-28T23:31:53.877711+0000","last_fullsized":"2024-03-28T23:31:53.877711+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.889315+0000","last_change":"2024-03-28T23:31:53.886833+0000","last_active":"2024-03-28T23:31:53.889315+0000","last_peered":"2024-03-28T23:31:53.889315+0000","last_clean":"2024-03-28T23:31:53.889315+0000","last_became_active":"2024-03-28T23:31:53.886498+0000","last_became_peered":"2024-03-28T23:31:53.886498+0000","last_unstale":"2024-03-28T23:31:53.889315+0000","last_undegraded":"2024-03-28T23:31:53.889315+0000","last_fullsized":"2024-03-28T23:31:53.889315+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:26:16.561871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.881876+0000","last_change":"2024-03-28T23:31:40.392391+0000","last_active":"2024-03-28T23:31:53.881876+0000","last_peered":"2024-03-28T23:31:53.881876+0000","last_clean":"2024-03-28T23:31:53.881876+0000","last_became_active":"2024-03-28T23:31:40.392100+0000","last_became_peered":"2024-03-28T23:31:40.392100+0000","last_unstale":"2024-03-28T23:31:53.881876+0000","last_undegraded":"2024-03-28T23:31:53.881876+0000","last_fullsized":"2024-03-28T23:31:53.881876+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:34:38.237729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"50'2","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874448+0000","last_change":"2024-03-28T23:31:39.385430+0000","last_active":"2024-03-28T23:31:53.874448+0000","last_peered":"2024-03-28T23:31:53.874448+0000","last_clean":"2024-03-28T23:31:53.874448+0000","last_became_active":"2024-03-28T23:31:39.385134+0000","last_became_peered":"2024-03-28T23:31:39.385134+0000","last_unstale":"2024-03-28T23:31:53.874448+0000","last_undegraded":"2024-03-28T23:31:53.874448+0000","last_fullsized":"2024-03-28T23:31:53.874448+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:12:26.472924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.873288+0000","last_change":"2024-03-28T23:31:53.873288+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.873288+0000","last_undegraded":"2024-03-28T23:31:53.873288+0000","last_fullsized":"2024-03-28T23:31:53.873288+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891323+0000","last_change":"2024-03-28T23:31:53.889140+0000","last_active":"2024-03-28T23:31:53.891323+0000","last_peered":"2024-03-28T23:31:53.891323+0000","last_clean":"2024-03-28T23:31:53.891323+0000","last_became_active":"2024-03-28T23:31:53.888738+0000","last_became_peered":"2024-03-28T23:31:53.888738+0000","last_unstale":"2024-03-28T23:31:53.891323+0000","last_undegraded":"2024-03-28T23:31:53.891323+0000","last_fullsized":"2024-03-28T23:31:53.891323+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:05:01.725546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.885385+0000","last_change":"2024-03-28T23:31:40.392757+0000","last_active":"2024-03-28T23:31:53.885385+0000","last_peered":"2024-03-28T23:31:53.885385+0000","last_clean":"2024-03-28T23:31:53.885385+0000","last_became_active":"2024-03-28T23:31:40.392460+0000","last_became_peered":"2024-03-28T23:31:40.392460+0000","last_unstale":"2024-03-28T23:31:53.885385+0000","last_undegraded":"2024-03-28T23:31:53.885385+0000","last_fullsized":"2024-03-28T23:31:53.885385+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:07.738674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"50'1","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875684+0000","last_change":"2024-03-28T23:31:39.381607+0000","last_active":"2024-03-28T23:31:53.875684+0000","last_peered":"2024-03-28T23:31:53.875684+0000","last_clean":"2024-03-28T23:31:53.875684+0000","last_became_active":"2024-03-28T23:31:39.381326+0000","last_became_peered":"2024-03-28T23:31:39.381326+0000","last_unstale":"2024-03-28T23:31:53.875684+0000","last_undegraded":"2024-03-28T23:31:53.875684+0000","last_fullsized":"2024-03-28T23:31:53.875684+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:50.860190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.880191+0000","last_change":"2024-03-28T23:31:53.880191+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.880191+0000","last_undegraded":"2024-03-28T23:31:53.880191+0000","last_fullsized":"2024-03-28T23:31:53.880191+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.888447+0000","last_change":"2024-03-28T23:31:53.886424+0000","last_active":"2024-03-28T23:31:53.888447+0000","last_peered":"2024-03-28T23:31:53.888447+0000","last_clean":"2024-03-28T23:31:53.888447+0000","last_became_active":"2024-03-28T23:31:53.886207+0000","last_became_peered":"2024-03-28T23:31:53.886207+0000","last_unstale":"2024-03-28T23:31:53.888447+0000","last_undegraded":"2024-03-28T23:31:53.888447+0000","last_fullsized":"2024-03-28T23:31:53.888447+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:26:10.948340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876571+0000","last_change":"2024-03-28T23:31:40.398712+0000","last_active":"2024-03-28T23:31:53.876571+0000","last_peered":"2024-03-28T23:31:53.876571+0000","last_clean":"2024-03-28T23:31:53.876571+0000","last_became_active":"2024-03-28T23:31:40.398392+0000","last_became_peered":"2024-03-28T23:31:40.398392+0000","last_unstale":"2024-03-28T23:31:53.876571+0000","last_undegraded":"2024-03-28T23:31:53.876571+0000","last_fullsized":"2024-03-28T23:31:53.876571+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:54.431044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"50'1","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.883791+0000","last_change":"2024-03-28T23:31:39.384026+0000","last_active":"2024-03-28T23:31:53.883791+0000","last_peered":"2024-03-28T23:31:53.883791+0000","last_clean":"2024-03-28T23:31:53.883791+0000","last_became_active":"2024-03-28T23:31:39.383161+0000","last_became_peered":"2024-03-28T23:31:39.383161+0000","last_unstale":"2024-03-28T23:31:53.883791+0000","last_undegraded":"2024-03-28T23:31:53.883791+0000","last_fullsized":"2024-03-28T23:31:53.883791+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:21:00.890266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872423+0000","last_change":"2024-03-28T23:31:53.872423+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872423+0000","last_undegraded":"2024-03-28T23:31:53.872423+0000","last_fullsized":"2024-03-28T23:31:53.872423+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.888932+0000","last_change":"2024-03-28T23:31:53.885914+0000","last_active":"2024-03-28T23:31:53.888932+0000","last_peered":"2024-03-28T23:31:53.888932+0000","last_clean":"2024-03-28T23:31:53.888932+0000","last_became_active":"2024-03-28T23:31:53.885637+0000","last_became_peered":"2024-03-28T23:31:53.885637+0000","last_unstale":"2024-03-28T23:31:53.888932+0000","last_undegraded":"2024-03-28T23:31:53.888932+0000","last_fullsized":"2024-03-28T23:31:53.888932+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:32.282446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879100+0000","last_change":"2024-03-28T23:31:40.395728+0000","last_active":"2024-03-28T23:31:53.879100+0000","last_peered":"2024-03-28T23:31:53.879100+0000","last_clean":"2024-03-28T23:31:53.879100+0000","last_became_active":"2024-03-28T23:31:40.395474+0000","last_became_peered":"2024-03-28T23:31:40.395474+0000","last_unstale":"2024-03-28T23:31:53.879100+0000","last_undegraded":"2024-03-28T23:31:53.879100+0000","last_fullsized":"2024-03-28T23:31:53.879100+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:15:19.985376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876912+0000","last_change":"2024-03-28T23:31:39.385649+0000","last_active":"2024-03-28T23:31:53.876912+0000","last_peered":"2024-03-28T23:31:53.876912+0000","last_clean":"2024-03-28T23:31:53.876912+0000","last_became_active":"2024-03-28T23:31:39.385341+0000","last_became_peered":"2024-03-28T23:31:39.385341+0000","last_unstale":"2024-03-28T23:31:53.876912+0000","last_undegraded":"2024-03-28T23:31:53.876912+0000","last_fullsized":"2024-03-28T23:31:53.876912+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:52:06.955653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878702+0000","last_change":"2024-03-28T23:31:53.878702+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878702+0000","last_undegraded":"2024-03-28T23:31:53.878702+0000","last_fullsized":"2024-03-28T23:31:53.878702+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891992+0000","last_change":"2024-03-28T23:31:53.889258+0000","last_active":"2024-03-28T23:31:53.891992+0000","last_peered":"2024-03-28T23:31:53.891992+0000","last_clean":"2024-03-28T23:31:53.891992+0000","last_became_active":"2024-03-28T23:31:53.888970+0000","last_became_peered":"2024-03-28T23:31:53.888970+0000","last_unstale":"2024-03-28T23:31:53.891992+0000","last_undegraded":"2024-03-28T23:31:53.891992+0000","last_fullsized":"2024-03-28T23:31:53.891992+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:20:01.487078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875176+0000","last_change":"2024-03-28T23:31:40.400091+0000","last_active":"2024-03-28T23:31:53.875176+0000","last_peered":"2024-03-28T23:31:53.875176+0000","last_clean":"2024-03-28T23:31:53.875176+0000","last_became_active":"2024-03-28T23:31:40.399190+0000","last_became_peered":"2024-03-28T23:31:40.399190+0000","last_unstale":"2024-03-28T23:31:53.875176+0000","last_undegraded":"2024-03-28T23:31:53.875176+0000","last_fullsized":"2024-03-28T23:31:53.875176+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:22.230986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875772+0000","last_change":"2024-03-28T23:31:39.390198+0000","last_active":"2024-03-28T23:31:53.875772+0000","last_peered":"2024-03-28T23:31:53.875772+0000","last_clean":"2024-03-28T23:31:53.875772+0000","last_became_active":"2024-03-28T23:31:39.389939+0000","last_became_peered":"2024-03-28T23:31:39.389939+0000","last_unstale":"2024-03-28T23:31:53.875772+0000","last_undegraded":"2024-03-28T23:31:53.875772+0000","last_fullsized":"2024-03-28T23:31:53.875772+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:46:30.350009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872121+0000","last_change":"2024-03-28T23:31:53.872121+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872121+0000","last_undegraded":"2024-03-28T23:31:53.872121+0000","last_fullsized":"2024-03-28T23:31:53.872121+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.896042+0000","last_change":"2024-03-28T23:31:53.892143+0000","last_active":"2024-03-28T23:31:53.896042+0000","last_peered":"2024-03-28T23:31:53.896042+0000","last_clean":"2024-03-28T23:31:53.896042+0000","last_became_active":"2024-03-28T23:31:53.891989+0000","last_became_peered":"2024-03-28T23:31:53.891989+0000","last_unstale":"2024-03-28T23:31:53.896042+0000","last_undegraded":"2024-03-28T23:31:53.896042+0000","last_fullsized":"2024-03-28T23:31:53.896042+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:23:27.129147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.883626+0000","last_change":"2024-03-28T23:31:40.384948+0000","last_active":"2024-03-28T23:31:53.883626+0000","last_peered":"2024-03-28T23:31:53.883626+0000","last_clean":"2024-03-28T23:31:53.883626+0000","last_became_active":"2024-03-28T23:31:40.384434+0000","last_became_peered":"2024-03-28T23:31:40.384434+0000","last_unstale":"2024-03-28T23:31:53.883626+0000","last_undegraded":"2024-03-28T23:31:53.883626+0000","last_fullsized":"2024-03-28T23:31:53.883626+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:04.829030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875277+0000","last_change":"2024-03-28T23:31:39.388914+0000","last_active":"2024-03-28T23:31:53.875277+0000","last_peered":"2024-03-28T23:31:53.875277+0000","last_clean":"2024-03-28T23:31:53.875277+0000","last_became_active":"2024-03-28T23:31:39.388639+0000","last_became_peered":"2024-03-28T23:31:39.388639+0000","last_unstale":"2024-03-28T23:31:53.875277+0000","last_undegraded":"2024-03-28T23:31:53.875277+0000","last_fullsized":"2024-03-28T23:31:53.875277+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:00.366278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.880894+0000","last_change":"2024-03-28T23:31:53.880894+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.880894+0000","last_undegraded":"2024-03-28T23:31:53.880894+0000","last_fullsized":"2024-03-28T23:31:53.880894+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.884484+0000","last_change":"2024-03-28T23:31:53.879950+0000","last_active":"2024-03-28T23:31:53.884484+0000","last_peered":"2024-03-28T23:31:53.884484+0000","last_clean":"2024-03-28T23:31:53.884484+0000","last_became_active":"2024-03-28T23:31:53.879620+0000","last_became_peered":"2024-03-28T23:31:53.879620+0000","last_unstale":"2024-03-28T23:31:53.884484+0000","last_undegraded":"2024-03-28T23:31:53.884484+0000","last_fullsized":"2024-03-28T23:31:53.884484+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:28:42.680101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.873900+0000","last_change":"2024-03-28T23:31:40.386749+0000","last_active":"2024-03-28T23:31:53.873900+0000","last_peered":"2024-03-28T23:31:53.873900+0000","last_clean":"2024-03-28T23:31:53.873900+0000","last_became_active":"2024-03-28T23:31:40.386466+0000","last_became_peered":"2024-03-28T23:31:40.386466+0000","last_unstale":"2024-03-28T23:31:53.873900+0000","last_undegraded":"2024-03-28T23:31:53.873900+0000","last_fullsized":"2024-03-28T23:31:53.873900+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:06.567694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.884919+0000","last_change":"2024-03-28T23:31:39.381050+0000","last_active":"2024-03-28T23:31:53.884919+0000","last_peered":"2024-03-28T23:31:53.884919+0000","last_clean":"2024-03-28T23:31:53.884919+0000","last_became_active":"2024-03-28T23:31:39.380761+0000","last_became_peered":"2024-03-28T23:31:39.380761+0000","last_unstale":"2024-03-28T23:31:53.884919+0000","last_undegraded":"2024-03-28T23:31:53.884919+0000","last_fullsized":"2024-03-28T23:31:53.884919+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:29.240985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878283+0000","last_change":"2024-03-28T23:31:53.878283+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878283+0000","last_undegraded":"2024-03-28T23:31:53.878283+0000","last_fullsized":"2024-03-28T23:31:53.878283+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.893113+0000","last_change":"2024-03-28T23:31:53.889906+0000","last_active":"2024-03-28T23:31:53.893113+0000","last_peered":"2024-03-28T23:31:53.893113+0000","last_clean":"2024-03-28T23:31:53.893113+0000","last_became_active":"2024-03-28T23:31:53.889589+0000","last_became_peered":"2024-03-28T23:31:53.889589+0000","last_unstale":"2024-03-28T23:31:53.893113+0000","last_undegraded":"2024-03-28T23:31:53.893113+0000","last_fullsized":"2024-03-28T23:31:53.893113+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:17:43.674083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875914+0000","last_change":"2024-03-28T23:31:40.400388+0000","last_active":"2024-03-28T23:31:53.875914+0000","last_peered":"2024-03-28T23:31:53.875914+0000","last_clean":"2024-03-28T23:31:53.875914+0000","last_became_active":"2024-03-28T23:31:40.400133+0000","last_became_peered":"2024-03-28T23:31:40.400133+0000","last_unstale":"2024-03-28T23:31:53.875914+0000","last_undegraded":"2024-03-28T23:31:53.875914+0000","last_fullsized":"2024-03-28T23:31:53.875914+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:03:56.252539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"50'3","reported_seq":21,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876731+0000","last_change":"2024-03-28T23:31:39.386478+0000","last_active":"2024-03-28T23:31:53.876731+0000","last_peered":"2024-03-28T23:31:53.876731+0000","last_clean":"2024-03-28T23:31:53.876731+0000","last_became_active":"2024-03-28T23:31:39.385868+0000","last_became_peered":"2024-03-28T23:31:39.385868+0000","last_unstale":"2024-03-28T23:31:53.876731+0000","last_undegraded":"2024-03-28T23:31:53.876731+0000","last_fullsized":"2024-03-28T23:31:53.876731+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:06.733879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872413+0000","last_change":"2024-03-28T23:31:53.872413+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872413+0000","last_undegraded":"2024-03-28T23:31:53.872413+0000","last_fullsized":"2024-03-28T23:31:53.872413+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.893021+0000","last_change":"2024-03-28T23:31:53.889887+0000","last_active":"2024-03-28T23:31:53.893021+0000","last_peered":"2024-03-28T23:31:53.893021+0000","last_clean":"2024-03-28T23:31:53.893021+0000","last_became_active":"2024-03-28T23:31:53.889635+0000","last_became_peered":"2024-03-28T23:31:53.889635+0000","last_unstale":"2024-03-28T23:31:53.893021+0000","last_undegraded":"2024-03-28T23:31:53.893021+0000","last_fullsized":"2024-03-28T23:31:53.893021+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:19:23.382549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.884161+0000","last_change":"2024-03-28T23:31:40.387795+0000","last_active":"2024-03-28T23:31:53.884161+0000","last_peered":"2024-03-28T23:31:53.884161+0000","last_clean":"2024-03-28T23:31:53.884161+0000","last_became_active":"2024-03-28T23:31:40.387529+0000","last_became_peered":"2024-03-28T23:31:40.387529+0000","last_unstale":"2024-03-28T23:31:53.884161+0000","last_undegraded":"2024-03-28T23:31:53.884161+0000","last_fullsized":"2024-03-28T23:31:53.884161+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:51.864986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"50'6","reported_seq":24,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879943+0000","last_change":"2024-03-28T23:31:39.393411+0000","last_active":"2024-03-28T23:31:53.879943+0000","last_peered":"2024-03-28T23:31:53.879943+0000","last_clean":"2024-03-28T23:31:53.879943+0000","last_became_active":"2024-03-28T23:31:39.393166+0000","last_became_peered":"2024-03-28T23:31:39.393166+0000","last_unstale":"2024-03-28T23:31:53.879943+0000","last_undegraded":"2024-03-28T23:31:53.879943+0000","last_fullsized":"2024-03-28T23:31:53.879943+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:13.510273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874577+0000","last_change":"2024-03-28T23:31:53.874577+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874577+0000","last_undegraded":"2024-03-28T23:31:53.874577+0000","last_fullsized":"2024-03-28T23:31:53.874577+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.890522+0000","last_change":"2024-03-28T23:31:53.889399+0000","last_active":"2024-03-28T23:31:53.890522+0000","last_peered":"2024-03-28T23:31:53.890522+0000","last_clean":"2024-03-28T23:31:53.890522+0000","last_became_active":"2024-03-28T23:31:53.889110+0000","last_became_peered":"2024-03-28T23:31:53.889110+0000","last_unstale":"2024-03-28T23:31:53.890522+0000","last_undegraded":"2024-03-28T23:31:53.890522+0000","last_fullsized":"2024-03-28T23:31:53.890522+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:35:48.231940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874249+0000","last_change":"2024-03-28T23:31:40.390497+0000","last_active":"2024-03-28T23:31:53.874249+0000","last_peered":"2024-03-28T23:31:53.874249+0000","last_clean":"2024-03-28T23:31:53.874249+0000","last_became_active":"2024-03-28T23:31:40.390167+0000","last_became_peered":"2024-03-28T23:31:40.390167+0000","last_unstale":"2024-03-28T23:31:53.874249+0000","last_undegraded":"2024-03-28T23:31:53.874249+0000","last_fullsized":"2024-03-28T23:31:53.874249+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:54.431044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"50'2","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875733+0000","last_change":"2024-03-28T23:31:39.380369+0000","last_active":"2024-03-28T23:31:53.875733+0000","last_peered":"2024-03-28T23:31:53.875733+0000","last_clean":"2024-03-28T23:31:53.875733+0000","last_became_active":"2024-03-28T23:31:39.380092+0000","last_became_peered":"2024-03-28T23:31:39.380092+0000","last_unstale":"2024-03-28T23:31:53.875733+0000","last_undegraded":"2024-03-28T23:31:53.875733+0000","last_fullsized":"2024-03-28T23:31:53.875733+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:00.366278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871676+0000","last_change":"2024-03-28T23:31:53.871676+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871676+0000","last_undegraded":"2024-03-28T23:31:53.871676+0000","last_fullsized":"2024-03-28T23:31:53.871676+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.885629+0000","last_change":"2024-03-28T23:31:53.883362+0000","last_active":"2024-03-28T23:31:53.885629+0000","last_peered":"2024-03-28T23:31:53.885629+0000","last_clean":"2024-03-28T23:31:53.885629+0000","last_became_active":"2024-03-28T23:31:53.883123+0000","last_became_peered":"2024-03-28T23:31:53.883123+0000","last_unstale":"2024-03-28T23:31:53.885629+0000","last_undegraded":"2024-03-28T23:31:53.885629+0000","last_fullsized":"2024-03-28T23:31:53.885629+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:16.175508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876210+0000","last_change":"2024-03-28T23:31:40.390579+0000","last_active":"2024-03-28T23:31:53.876210+0000","last_peered":"2024-03-28T23:31:53.876210+0000","last_clean":"2024-03-28T23:31:53.876210+0000","last_became_active":"2024-03-28T23:31:40.390248+0000","last_became_peered":"2024-03-28T23:31:40.390248+0000","last_unstale":"2024-03-28T23:31:53.876210+0000","last_undegraded":"2024-03-28T23:31:53.876210+0000","last_fullsized":"2024-03-28T23:31:53.876210+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:47.861300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875729+0000","last_change":"2024-03-28T23:31:39.389592+0000","last_active":"2024-03-28T23:31:53.875729+0000","last_peered":"2024-03-28T23:31:53.875729+0000","last_clean":"2024-03-28T23:31:53.875729+0000","last_became_active":"2024-03-28T23:31:39.389302+0000","last_became_peered":"2024-03-28T23:31:39.389302+0000","last_unstale":"2024-03-28T23:31:53.875729+0000","last_undegraded":"2024-03-28T23:31:53.875729+0000","last_fullsized":"2024-03-28T23:31:53.875729+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:59:51.501145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872186+0000","last_change":"2024-03-28T23:31:53.872186+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872186+0000","last_undegraded":"2024-03-28T23:31:53.872186+0000","last_fullsized":"2024-03-28T23:31:53.872186+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.884324+0000","last_change":"2024-03-28T23:31:53.881795+0000","last_active":"2024-03-28T23:31:53.884324+0000","last_peered":"2024-03-28T23:31:53.884324+0000","last_clean":"2024-03-28T23:31:53.884324+0000","last_became_active":"2024-03-28T23:31:53.881552+0000","last_became_peered":"2024-03-28T23:31:53.881552+0000","last_unstale":"2024-03-28T23:31:53.884324+0000","last_undegraded":"2024-03-28T23:31:53.884324+0000","last_fullsized":"2024-03-28T23:31:53.884324+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:35.224564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879015+0000","last_change":"2024-03-28T23:31:40.387085+0000","last_active":"2024-03-28T23:31:53.879015+0000","last_peered":"2024-03-28T23:31:53.879015+0000","last_clean":"2024-03-28T23:31:53.879015+0000","last_became_active":"2024-03-28T23:31:40.386695+0000","last_became_peered":"2024-03-28T23:31:40.386695+0000","last_unstale":"2024-03-28T23:31:53.879015+0000","last_undegraded":"2024-03-28T23:31:53.879015+0000","last_fullsized":"2024-03-28T23:31:53.879015+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:40:57.541676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.873632+0000","last_change":"2024-03-28T23:31:39.384653+0000","last_active":"2024-03-28T23:31:53.873632+0000","last_peered":"2024-03-28T23:31:53.873632+0000","last_clean":"2024-03-28T23:31:53.873632+0000","last_became_active":"2024-03-28T23:31:39.383495+0000","last_became_peered":"2024-03-28T23:31:39.383495+0000","last_unstale":"2024-03-28T23:31:53.873632+0000","last_undegraded":"2024-03-28T23:31:53.873632+0000","last_fullsized":"2024-03-28T23:31:53.873632+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:50.860190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878941+0000","last_change":"2024-03-28T23:31:53.878941+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878941+0000","last_undegraded":"2024-03-28T23:31:53.878941+0000","last_fullsized":"2024-03-28T23:31:53.878941+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.886195+0000","last_change":"2024-03-28T23:31:53.884651+0000","last_active":"2024-03-28T23:31:53.886195+0000","last_peered":"2024-03-28T23:31:53.886195+0000","last_clean":"2024-03-28T23:31:53.886195+0000","last_became_active":"2024-03-28T23:31:53.884410+0000","last_became_peered":"2024-03-28T23:31:53.884410+0000","last_unstale":"2024-03-28T23:31:53.886195+0000","last_undegraded":"2024-03-28T23:31:53.886195+0000","last_fullsized":"2024-03-28T23:31:53.886195+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:31:41.553370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876561+0000","last_change":"2024-03-28T23:31:40.403701+0000","last_active":"2024-03-28T23:31:53.876561+0000","last_peered":"2024-03-28T23:31:53.876561+0000","last_clean":"2024-03-28T23:31:53.876561+0000","last_became_active":"2024-03-28T23:31:40.403222+0000","last_became_peered":"2024-03-28T23:31:40.403222+0000","last_unstale":"2024-03-28T23:31:53.876561+0000","last_undegraded":"2024-03-28T23:31:53.876561+0000","last_fullsized":"2024-03-28T23:31:53.876561+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:40:09.867914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"50'8","reported_seq":26,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.880291+0000","last_change":"2024-03-28T23:31:39.383726+0000","last_active":"2024-03-28T23:31:53.880291+0000","last_peered":"2024-03-28T23:31:53.880291+0000","last_clean":"2024-03-28T23:31:53.880291+0000","last_became_active":"2024-03-28T23:31:39.383292+0000","last_became_peered":"2024-03-28T23:31:39.383292+0000","last_unstale":"2024-03-28T23:31:53.880291+0000","last_undegraded":"2024-03-28T23:31:53.880291+0000","last_fullsized":"2024-03-28T23:31:53.880291+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:25:38.113638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872289+0000","last_change":"2024-03-28T23:31:53.872289+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872289+0000","last_undegraded":"2024-03-28T23:31:53.872289+0000","last_fullsized":"2024-03-28T23:31:53.872289+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.896055+0000","last_change":"2024-03-28T23:31:53.892246+0000","last_active":"2024-03-28T23:31:53.896055+0000","last_peered":"2024-03-28T23:31:53.896055+0000","last_clean":"2024-03-28T23:31:53.896055+0000","last_became_active":"2024-03-28T23:31:53.891974+0000","last_became_peered":"2024-03-28T23:31:53.891974+0000","last_unstale":"2024-03-28T23:31:53.896055+0000","last_undegraded":"2024-03-28T23:31:53.896055+0000","last_fullsized":"2024-03-28T23:31:53.896055+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:10:22.963954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.880205+0000","last_change":"2024-03-28T23:31:40.391219+0000","last_active":"2024-03-28T23:31:53.880205+0000","last_peered":"2024-03-28T23:31:53.880205+0000","last_clean":"2024-03-28T23:31:53.880205+0000","last_became_active":"2024-03-28T23:31:40.389552+0000","last_became_peered":"2024-03-28T23:31:40.389552+0000","last_unstale":"2024-03-28T23:31:53.880205+0000","last_undegraded":"2024-03-28T23:31:53.880205+0000","last_fullsized":"2024-03-28T23:31:53.880205+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:42:29.351941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"50'1","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876797+0000","last_change":"2024-03-28T23:31:39.390253+0000","last_active":"2024-03-28T23:31:53.876797+0000","last_peered":"2024-03-28T23:31:53.876797+0000","last_clean":"2024-03-28T23:31:53.876797+0000","last_became_active":"2024-03-28T23:31:39.389980+0000","last_became_peered":"2024-03-28T23:31:39.389980+0000","last_unstale":"2024-03-28T23:31:53.876797+0000","last_undegraded":"2024-03-28T23:31:53.876797+0000","last_fullsized":"2024-03-28T23:31:53.876797+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:24.520296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.877420+0000","last_change":"2024-03-28T23:31:53.877420+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.877420+0000","last_undegraded":"2024-03-28T23:31:53.877420+0000","last_fullsized":"2024-03-28T23:31:53.877420+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.884332+0000","last_change":"2024-03-28T23:31:53.880659+0000","last_active":"2024-03-28T23:31:53.884332+0000","last_peered":"2024-03-28T23:31:53.884332+0000","last_clean":"2024-03-28T23:31:53.884332+0000","last_became_active":"2024-03-28T23:31:53.880265+0000","last_became_peered":"2024-03-28T23:31:53.880265+0000","last_unstale":"2024-03-28T23:31:53.884332+0000","last_undegraded":"2024-03-28T23:31:53.884332+0000","last_fullsized":"2024-03-28T23:31:53.884332+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:04:09.751864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869503+0000","last_change":"2024-03-28T23:31:40.391898+0000","last_active":"2024-03-28T23:31:53.869503+0000","last_peered":"2024-03-28T23:31:53.869503+0000","last_clean":"2024-03-28T23:31:53.869503+0000","last_became_active":"2024-03-28T23:31:40.391622+0000","last_became_peered":"2024-03-28T23:31:40.391622+0000","last_unstale":"2024-03-28T23:31:53.869503+0000","last_undegraded":"2024-03-28T23:31:53.869503+0000","last_fullsized":"2024-03-28T23:31:53.869503+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:38:22.055287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"50'2","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.883203+0000","last_change":"2024-03-28T23:31:39.385339+0000","last_active":"2024-03-28T23:31:53.883203+0000","last_peered":"2024-03-28T23:31:53.883203+0000","last_clean":"2024-03-28T23:31:53.883203+0000","last_became_active":"2024-03-28T23:31:39.385075+0000","last_became_peered":"2024-03-28T23:31:39.385075+0000","last_unstale":"2024-03-28T23:31:53.883203+0000","last_undegraded":"2024-03-28T23:31:53.883203+0000","last_fullsized":"2024-03-28T23:31:53.883203+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:58.098710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.873793+0000","last_change":"2024-03-28T23:31:53.873793+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.873793+0000","last_undegraded":"2024-03-28T23:31:53.873793+0000","last_fullsized":"2024-03-28T23:31:53.873793+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891950+0000","last_change":"2024-03-28T23:31:53.890901+0000","last_active":"2024-03-28T23:31:53.891950+0000","last_peered":"2024-03-28T23:31:53.891950+0000","last_clean":"2024-03-28T23:31:53.891950+0000","last_became_active":"2024-03-28T23:31:53.890643+0000","last_became_peered":"2024-03-28T23:31:53.890643+0000","last_unstale":"2024-03-28T23:31:53.891950+0000","last_undegraded":"2024-03-28T23:31:53.891950+0000","last_fullsized":"2024-03-28T23:31:53.891950+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:07:27.190116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.883208+0000","last_change":"2024-03-28T23:31:40.399291+0000","last_active":"2024-03-28T23:31:53.883208+0000","last_peered":"2024-03-28T23:31:53.883208+0000","last_clean":"2024-03-28T23:31:53.883208+0000","last_became_active":"2024-03-28T23:31:40.399005+0000","last_became_peered":"2024-03-28T23:31:40.399005+0000","last_unstale":"2024-03-28T23:31:53.883208+0000","last_undegraded":"2024-03-28T23:31:53.883208+0000","last_fullsized":"2024-03-28T23:31:53.883208+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:01.371074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"50'1","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876296+0000","last_change":"2024-03-28T23:31:39.389465+0000","last_active":"2024-03-28T23:31:53.876296+0000","last_peered":"2024-03-28T23:31:53.876296+0000","last_clean":"2024-03-28T23:31:53.876296+0000","last_became_active":"2024-03-28T23:31:39.389005+0000","last_became_peered":"2024-03-28T23:31:39.389005+0000","last_unstale":"2024-03-28T23:31:53.876296+0000","last_undegraded":"2024-03-28T23:31:53.876296+0000","last_fullsized":"2024-03-28T23:31:53.876296+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:41.430571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.879255+0000","last_change":"2024-03-28T23:31:53.879255+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.879255+0000","last_undegraded":"2024-03-28T23:31:53.879255+0000","last_fullsized":"2024-03-28T23:31:53.879255+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.890716+0000","last_change":"2024-03-28T23:31:53.888704+0000","last_active":"2024-03-28T23:31:53.890716+0000","last_peered":"2024-03-28T23:31:53.890716+0000","last_clean":"2024-03-28T23:31:53.890716+0000","last_became_active":"2024-03-28T23:31:53.888461+0000","last_became_peered":"2024-03-28T23:31:53.888461+0000","last_unstale":"2024-03-28T23:31:53.890716+0000","last_undegraded":"2024-03-28T23:31:53.890716+0000","last_fullsized":"2024-03-28T23:31:53.890716+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:08.894103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869795+0000","last_change":"2024-03-28T23:31:40.388141+0000","last_active":"2024-03-28T23:31:53.869795+0000","last_peered":"2024-03-28T23:31:53.869795+0000","last_clean":"2024-03-28T23:31:53.869795+0000","last_became_active":"2024-03-28T23:31:40.387852+0000","last_became_peered":"2024-03-28T23:31:40.387852+0000","last_unstale":"2024-03-28T23:31:53.869795+0000","last_undegraded":"2024-03-28T23:31:53.869795+0000","last_fullsized":"2024-03-28T23:31:53.869795+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:22.230986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.877311+0000","last_change":"2024-03-28T23:31:39.390465+0000","last_active":"2024-03-28T23:31:53.877311+0000","last_peered":"2024-03-28T23:31:53.877311+0000","last_clean":"2024-03-28T23:31:53.877311+0000","last_became_active":"2024-03-28T23:31:39.390203+0000","last_became_peered":"2024-03-28T23:31:39.390203+0000","last_unstale":"2024-03-28T23:31:53.877311+0000","last_undegraded":"2024-03-28T23:31:53.877311+0000","last_fullsized":"2024-03-28T23:31:53.877311+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:50:50.496294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.876579+0000","last_change":"2024-03-28T23:31:53.876579+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.876579+0000","last_undegraded":"2024-03-28T23:31:53.876579+0000","last_fullsized":"2024-03-28T23:31:53.876579+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.890502+0000","last_change":"2024-03-28T23:31:53.888267+0000","last_active":"2024-03-28T23:31:53.890502+0000","last_peered":"2024-03-28T23:31:53.890502+0000","last_clean":"2024-03-28T23:31:53.890502+0000","last_became_active":"2024-03-28T23:31:53.887993+0000","last_became_peered":"2024-03-28T23:31:53.887993+0000","last_unstale":"2024-03-28T23:31:53.890502+0000","last_undegraded":"2024-03-28T23:31:53.890502+0000","last_fullsized":"2024-03-28T23:31:53.890502+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:21:27.713981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874357+0000","last_change":"2024-03-28T23:31:40.387376+0000","last_active":"2024-03-28T23:31:53.874357+0000","last_peered":"2024-03-28T23:31:53.874357+0000","last_clean":"2024-03-28T23:31:53.874357+0000","last_became_active":"2024-03-28T23:31:40.387080+0000","last_became_peered":"2024-03-28T23:31:40.387080+0000","last_unstale":"2024-03-28T23:31:53.874357+0000","last_undegraded":"2024-03-28T23:31:53.874357+0000","last_fullsized":"2024-03-28T23:31:53.874357+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:43.111058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"50'2","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869894+0000","last_change":"2024-03-28T23:31:39.388867+0000","last_active":"2024-03-28T23:31:53.869894+0000","last_peered":"2024-03-28T23:31:53.869894+0000","last_clean":"2024-03-28T23:31:53.869894+0000","last_became_active":"2024-03-28T23:31:39.388593+0000","last_became_peered":"2024-03-28T23:31:39.388593+0000","last_unstale":"2024-03-28T23:31:53.869894+0000","last_undegraded":"2024-03-28T23:31:53.869894+0000","last_fullsized":"2024-03-28T23:31:53.869894+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:21:00.890266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872546+0000","last_change":"2024-03-28T23:31:53.872546+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872546+0000","last_undegraded":"2024-03-28T23:31:53.872546+0000","last_fullsized":"2024-03-28T23:31:53.872546+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.888466+0000","last_change":"2024-03-28T23:31:53.886433+0000","last_active":"2024-03-28T23:31:53.888466+0000","last_peered":"2024-03-28T23:31:53.888466+0000","last_clean":"2024-03-28T23:31:53.888466+0000","last_became_active":"2024-03-28T23:31:53.886218+0000","last_became_peered":"2024-03-28T23:31:53.886218+0000","last_unstale":"2024-03-28T23:31:53.888466+0000","last_undegraded":"2024-03-28T23:31:53.888466+0000","last_fullsized":"2024-03-28T23:31:53.888466+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:32.282446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876171+0000","last_change":"2024-03-28T23:31:40.393257+0000","last_active":"2024-03-28T23:31:53.876171+0000","last_peered":"2024-03-28T23:31:53.876171+0000","last_clean":"2024-03-28T23:31:53.876171+0000","last_became_active":"2024-03-28T23:31:40.391684+0000","last_became_peered":"2024-03-28T23:31:40.391684+0000","last_unstale":"2024-03-28T23:31:53.876171+0000","last_undegraded":"2024-03-28T23:31:53.876171+0000","last_fullsized":"2024-03-28T23:31:53.876171+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:01.371074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.881223+0000","last_change":"2024-03-28T23:31:39.388198+0000","last_active":"2024-03-28T23:31:53.881223+0000","last_peered":"2024-03-28T23:31:53.881223+0000","last_clean":"2024-03-28T23:31:53.881223+0000","last_became_active":"2024-03-28T23:31:39.387727+0000","last_became_peered":"2024-03-28T23:31:39.387727+0000","last_unstale":"2024-03-28T23:31:53.881223+0000","last_undegraded":"2024-03-28T23:31:53.881223+0000","last_fullsized":"2024-03-28T23:31:53.881223+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:49.868091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874240+0000","last_change":"2024-03-28T23:31:53.874240+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874240+0000","last_undegraded":"2024-03-28T23:31:53.874240+0000","last_fullsized":"2024-03-28T23:31:53.874240+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.886125+0000","last_change":"2024-03-28T23:31:53.884698+0000","last_active":"2024-03-28T23:31:53.886125+0000","last_peered":"2024-03-28T23:31:53.886125+0000","last_clean":"2024-03-28T23:31:53.886125+0000","last_became_active":"2024-03-28T23:31:53.884413+0000","last_became_peered":"2024-03-28T23:31:53.884413+0000","last_unstale":"2024-03-28T23:31:53.886125+0000","last_undegraded":"2024-03-28T23:31:53.886125+0000","last_fullsized":"2024-03-28T23:31:53.886125+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:32.282446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876556+0000","last_change":"2024-03-28T23:31:40.394047+0000","last_active":"2024-03-28T23:31:53.876556+0000","last_peered":"2024-03-28T23:31:53.876556+0000","last_clean":"2024-03-28T23:31:53.876556+0000","last_became_active":"2024-03-28T23:31:40.393772+0000","last_became_peered":"2024-03-28T23:31:40.393772+0000","last_unstale":"2024-03-28T23:31:53.876556+0000","last_undegraded":"2024-03-28T23:31:53.876556+0000","last_fullsized":"2024-03-28T23:31:53.876556+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:07:13.690791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875624+0000","last_change":"2024-03-28T23:31:39.388829+0000","last_active":"2024-03-28T23:31:53.875624+0000","last_peered":"2024-03-28T23:31:53.875624+0000","last_clean":"2024-03-28T23:31:53.875624+0000","last_became_active":"2024-03-28T23:31:39.388521+0000","last_became_peered":"2024-03-28T23:31:39.388521+0000","last_unstale":"2024-03-28T23:31:53.875624+0000","last_undegraded":"2024-03-28T23:31:53.875624+0000","last_fullsized":"2024-03-28T23:31:53.875624+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:23:52.148574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.877392+0000","last_change":"2024-03-28T23:31:53.877392+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.877392+0000","last_undegraded":"2024-03-28T23:31:53.877392+0000","last_fullsized":"2024-03-28T23:31:53.877392+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.887423+0000","last_change":"2024-03-28T23:31:53.886155+0000","last_active":"2024-03-28T23:31:53.887423+0000","last_peered":"2024-03-28T23:31:53.887423+0000","last_clean":"2024-03-28T23:31:53.887423+0000","last_became_active":"2024-03-28T23:31:53.885843+0000","last_became_peered":"2024-03-28T23:31:53.885843+0000","last_unstale":"2024-03-28T23:31:53.887423+0000","last_undegraded":"2024-03-28T23:31:53.887423+0000","last_fullsized":"2024-03-28T23:31:53.887423+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:38:09.370282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.870442+0000","last_change":"2024-03-28T23:31:40.384709+0000","last_active":"2024-03-28T23:31:53.870442+0000","last_peered":"2024-03-28T23:31:53.870442+0000","last_clean":"2024-03-28T23:31:53.870442+0000","last_became_active":"2024-03-28T23:31:40.384412+0000","last_became_peered":"2024-03-28T23:31:40.384412+0000","last_unstale":"2024-03-28T23:31:53.870442+0000","last_undegraded":"2024-03-28T23:31:53.870442+0000","last_fullsized":"2024-03-28T23:31:53.870442+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:25:39.118434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879185+0000","last_change":"2024-03-28T23:31:39.385937+0000","last_active":"2024-03-28T23:31:53.879185+0000","last_peered":"2024-03-28T23:31:53.879185+0000","last_clean":"2024-03-28T23:31:53.879185+0000","last_became_active":"2024-03-28T23:31:39.385675+0000","last_became_peered":"2024-03-28T23:31:39.385675+0000","last_unstale":"2024-03-28T23:31:53.879185+0000","last_undegraded":"2024-03-28T23:31:53.879185+0000","last_fullsized":"2024-03-28T23:31:53.879185+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:35.204851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874303+0000","last_change":"2024-03-28T23:31:53.874303+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874303+0000","last_undegraded":"2024-03-28T23:31:53.874303+0000","last_fullsized":"2024-03-28T23:31:53.874303+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.886168+0000","last_change":"2024-03-28T23:31:53.884950+0000","last_active":"2024-03-28T23:31:53.886168+0000","last_peered":"2024-03-28T23:31:53.886168+0000","last_clean":"2024-03-28T23:31:53.886168+0000","last_became_active":"2024-03-28T23:31:53.884686+0000","last_became_peered":"2024-03-28T23:31:53.884686+0000","last_unstale":"2024-03-28T23:31:53.886168+0000","last_undegraded":"2024-03-28T23:31:53.886168+0000","last_fullsized":"2024-03-28T23:31:53.886168+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:48.477986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874779+0000","last_change":"2024-03-28T23:31:40.387783+0000","last_active":"2024-03-28T23:31:53.874779+0000","last_peered":"2024-03-28T23:31:53.874779+0000","last_clean":"2024-03-28T23:31:53.874779+0000","last_became_active":"2024-03-28T23:31:40.387487+0000","last_became_peered":"2024-03-28T23:31:40.387487+0000","last_unstale":"2024-03-28T23:31:53.874779+0000","last_undegraded":"2024-03-28T23:31:53.874779+0000","last_fullsized":"2024-03-28T23:31:53.874779+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:22.230986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"50'2","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.880346+0000","last_change":"2024-03-28T23:31:39.391741+0000","last_active":"2024-03-28T23:31:53.880346+0000","last_peered":"2024-03-28T23:31:53.880346+0000","last_clean":"2024-03-28T23:31:53.880346+0000","last_became_active":"2024-03-28T23:31:39.391520+0000","last_became_peered":"2024-03-28T23:31:39.391520+0000","last_unstale":"2024-03-28T23:31:53.880346+0000","last_undegraded":"2024-03-28T23:31:53.880346+0000","last_fullsized":"2024-03-28T23:31:53.880346+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:16:42.869555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874197+0000","last_change":"2024-03-28T23:31:53.874197+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874197+0000","last_undegraded":"2024-03-28T23:31:53.874197+0000","last_fullsized":"2024-03-28T23:31:53.874197+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.890450+0000","last_change":"2024-03-28T23:31:53.887757+0000","last_active":"2024-03-28T23:31:53.890450+0000","last_peered":"2024-03-28T23:31:53.890450+0000","last_clean":"2024-03-28T23:31:53.890450+0000","last_became_active":"2024-03-28T23:31:53.887470+0000","last_became_peered":"2024-03-28T23:31:53.887470+0000","last_unstale":"2024-03-28T23:31:53.890450+0000","last_undegraded":"2024-03-28T23:31:53.890450+0000","last_fullsized":"2024-03-28T23:31:53.890450+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:40:57.385677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869629+0000","last_change":"2024-03-28T23:31:40.387330+0000","last_active":"2024-03-28T23:31:53.869629+0000","last_peered":"2024-03-28T23:31:53.869629+0000","last_clean":"2024-03-28T23:31:53.869629+0000","last_became_active":"2024-03-28T23:31:40.387076+0000","last_became_peered":"2024-03-28T23:31:40.387076+0000","last_unstale":"2024-03-28T23:31:53.869629+0000","last_undegraded":"2024-03-28T23:31:53.869629+0000","last_fullsized":"2024-03-28T23:31:53.869629+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:25.525091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.877434+0000","last_change":"2024-03-28T23:31:39.386686+0000","last_active":"2024-03-28T23:31:53.877434+0000","last_peered":"2024-03-28T23:31:53.877434+0000","last_clean":"2024-03-28T23:31:53.877434+0000","last_became_active":"2024-03-28T23:31:39.386208+0000","last_became_peered":"2024-03-28T23:31:39.386208+0000","last_unstale":"2024-03-28T23:31:53.877434+0000","last_undegraded":"2024-03-28T23:31:53.877434+0000","last_fullsized":"2024-03-28T23:31:53.877434+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:22.210570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.876269+0000","last_change":"2024-03-28T23:31:53.876269+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.876269+0000","last_undegraded":"2024-03-28T23:31:53.876269+0000","last_fullsized":"2024-03-28T23:31:53.876269+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891725+0000","last_change":"2024-03-28T23:31:53.887766+0000","last_active":"2024-03-28T23:31:53.891725+0000","last_peered":"2024-03-28T23:31:53.891725+0000","last_clean":"2024-03-28T23:31:53.891725+0000","last_became_active":"2024-03-28T23:31:53.887471+0000","last_became_peered":"2024-03-28T23:31:53.887471+0000","last_unstale":"2024-03-28T23:31:53.891725+0000","last_undegraded":"2024-03-28T23:31:53.891725+0000","last_fullsized":"2024-03-28T23:31:53.891725+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:05:01.725546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.881956+0000","last_change":"2024-03-28T23:31:40.386143+0000","last_active":"2024-03-28T23:31:53.881956+0000","last_peered":"2024-03-28T23:31:53.881956+0000","last_clean":"2024-03-28T23:31:53.881956+0000","last_became_active":"2024-03-28T23:31:40.385606+0000","last_became_peered":"2024-03-28T23:31:40.385606+0000","last_unstale":"2024-03-28T23:31:53.881956+0000","last_undegraded":"2024-03-28T23:31:53.881956+0000","last_fullsized":"2024-03-28T23:31:53.881956+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:40:09.867914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876026+0000","last_change":"2024-03-28T23:31:39.380624+0000","last_active":"2024-03-28T23:31:53.876026+0000","last_peered":"2024-03-28T23:31:53.876026+0000","last_clean":"2024-03-28T23:31:53.876026+0000","last_became_active":"2024-03-28T23:31:39.380340+0000","last_became_peered":"2024-03-28T23:31:39.380340+0000","last_unstale":"2024-03-28T23:31:53.876026+0000","last_undegraded":"2024-03-28T23:31:53.876026+0000","last_fullsized":"2024-03-28T23:31:53.876026+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:22.210570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.879694+0000","last_change":"2024-03-28T23:31:53.879694+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.879694+0000","last_undegraded":"2024-03-28T23:31:53.879694+0000","last_fullsized":"2024-03-28T23:31:53.879694+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.889286+0000","last_change":"2024-03-28T23:31:53.886660+0000","last_active":"2024-03-28T23:31:53.889286+0000","last_peered":"2024-03-28T23:31:53.889286+0000","last_clean":"2024-03-28T23:31:53.889286+0000","last_became_active":"2024-03-28T23:31:53.886355+0000","last_became_peered":"2024-03-28T23:31:53.886355+0000","last_unstale":"2024-03-28T23:31:53.889286+0000","last_undegraded":"2024-03-28T23:31:53.889286+0000","last_fullsized":"2024-03-28T23:31:53.889286+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:38:09.370282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875942+0000","last_change":"2024-03-28T23:31:40.404359+0000","last_active":"2024-03-28T23:31:53.875942+0000","last_peered":"2024-03-28T23:31:53.875942+0000","last_clean":"2024-03-28T23:31:53.875942+0000","last_became_active":"2024-03-28T23:31:40.404078+0000","last_became_peered":"2024-03-28T23:31:40.404078+0000","last_unstale":"2024-03-28T23:31:53.875942+0000","last_undegraded":"2024-03-28T23:31:53.875942+0000","last_fullsized":"2024-03-28T23:31:53.875942+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:18.654331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.882034+0000","last_change":"2024-03-28T23:31:39.393283+0000","last_active":"2024-03-28T23:31:53.882034+0000","last_peered":"2024-03-28T23:31:53.882034+0000","last_clean":"2024-03-28T23:31:53.882034+0000","last_became_active":"2024-03-28T23:31:39.393035+0000","last_became_peered":"2024-03-28T23:31:39.393035+0000","last_unstale":"2024-03-28T23:31:53.882034+0000","last_undegraded":"2024-03-28T23:31:53.882034+0000","last_fullsized":"2024-03-28T23:31:53.882034+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:13.139644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872993+0000","last_change":"2024-03-28T23:31:53.872993+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872993+0000","last_undegraded":"2024-03-28T23:31:53.872993+0000","last_fullsized":"2024-03-28T23:31:53.872993+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.890751+0000","last_change":"2024-03-28T23:31:53.888742+0000","last_active":"2024-03-28T23:31:53.890751+0000","last_peered":"2024-03-28T23:31:53.890751+0000","last_clean":"2024-03-28T23:31:53.890751+0000","last_became_active":"2024-03-28T23:31:53.888515+0000","last_became_peered":"2024-03-28T23:31:53.888515+0000","last_unstale":"2024-03-28T23:31:53.890751+0000","last_undegraded":"2024-03-28T23:31:53.890751+0000","last_fullsized":"2024-03-28T23:31:53.890751+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:17:10.580507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875207+0000","last_change":"2024-03-28T23:31:40.392819+0000","last_active":"2024-03-28T23:31:53.875207+0000","last_peered":"2024-03-28T23:31:53.875207+0000","last_clean":"2024-03-28T23:31:53.875207+0000","last_became_active":"2024-03-28T23:31:40.392501+0000","last_became_peered":"2024-03-28T23:31:40.392501+0000","last_unstale":"2024-03-28T23:31:53.875207+0000","last_undegraded":"2024-03-28T23:31:53.875207+0000","last_fullsized":"2024-03-28T23:31:53.875207+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:14.144440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.881307+0000","last_change":"2024-03-28T23:31:39.387993+0000","last_active":"2024-03-28T23:31:53.881307+0000","last_peered":"2024-03-28T23:31:53.881307+0000","last_clean":"2024-03-28T23:31:53.881307+0000","last_became_active":"2024-03-28T23:31:39.387393+0000","last_became_peered":"2024-03-28T23:31:39.387393+0000","last_unstale":"2024-03-28T23:31:53.881307+0000","last_undegraded":"2024-03-28T23:31:53.881307+0000","last_fullsized":"2024-03-28T23:31:53.881307+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:21.226190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871482+0000","last_change":"2024-03-28T23:31:53.871482+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871482+0000","last_undegraded":"2024-03-28T23:31:53.871482+0000","last_fullsized":"2024-03-28T23:31:53.871482+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.892224+0000","last_change":"2024-03-28T23:31:53.890146+0000","last_active":"2024-03-28T23:31:53.892224+0000","last_peered":"2024-03-28T23:31:53.892224+0000","last_clean":"2024-03-28T23:31:53.892224+0000","last_became_active":"2024-03-28T23:31:53.888840+0000","last_became_peered":"2024-03-28T23:31:53.888840+0000","last_unstale":"2024-03-28T23:31:53.892224+0000","last_undegraded":"2024-03-28T23:31:53.892224+0000","last_fullsized":"2024-03-28T23:31:53.892224+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:01.360626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876467+0000","last_change":"2024-03-28T23:31:40.405766+0000","last_active":"2024-03-28T23:31:53.876467+0000","last_peered":"2024-03-28T23:31:53.876467+0000","last_clean":"2024-03-28T23:31:53.876467+0000","last_became_active":"2024-03-28T23:31:40.405517+0000","last_became_peered":"2024-03-28T23:31:40.405517+0000","last_unstale":"2024-03-28T23:31:53.876467+0000","last_undegraded":"2024-03-28T23:31:53.876467+0000","last_fullsized":"2024-03-28T23:31:53.876467+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:18:16.851435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.883245+0000","last_change":"2024-03-28T23:31:39.383748+0000","last_active":"2024-03-28T23:31:53.883245+0000","last_peered":"2024-03-28T23:31:53.883245+0000","last_clean":"2024-03-28T23:31:53.883245+0000","last_became_active":"2024-03-28T23:31:39.383070+0000","last_became_peered":"2024-03-28T23:31:39.383070+0000","last_unstale":"2024-03-28T23:31:53.883245+0000","last_undegraded":"2024-03-28T23:31:53.883245+0000","last_fullsized":"2024-03-28T23:31:53.883245+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:41.430571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.873125+0000","last_change":"2024-03-28T23:31:53.873125+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.873125+0000","last_undegraded":"2024-03-28T23:31:53.873125+0000","last_fullsized":"2024-03-28T23:31:53.873125+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.890278+0000","last_change":"2024-03-28T23:31:53.887894+0000","last_active":"2024-03-28T23:31:53.890278+0000","last_peered":"2024-03-28T23:31:53.890278+0000","last_clean":"2024-03-28T23:31:53.890278+0000","last_became_active":"2024-03-28T23:31:53.887619+0000","last_became_peered":"2024-03-28T23:31:53.887619+0000","last_unstale":"2024-03-28T23:31:53.890278+0000","last_undegraded":"2024-03-28T23:31:53.890278+0000","last_fullsized":"2024-03-28T23:31:53.890278+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:26:16.561871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879094+0000","last_change":"2024-03-28T23:31:40.392625+0000","last_active":"2024-03-28T23:31:53.879094+0000","last_peered":"2024-03-28T23:31:53.879094+0000","last_clean":"2024-03-28T23:31:53.879094+0000","last_became_active":"2024-03-28T23:31:40.392359+0000","last_became_peered":"2024-03-28T23:31:40.392359+0000","last_unstale":"2024-03-28T23:31:53.879094+0000","last_undegraded":"2024-03-28T23:31:53.879094+0000","last_fullsized":"2024-03-28T23:31:53.879094+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:02.676183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874872+0000","last_change":"2024-03-28T23:31:39.389478+0000","last_active":"2024-03-28T23:31:53.874872+0000","last_peered":"2024-03-28T23:31:53.874872+0000","last_clean":"2024-03-28T23:31:53.874872+0000","last_became_active":"2024-03-28T23:31:39.389154+0000","last_became_peered":"2024-03-28T23:31:39.389154+0000","last_unstale":"2024-03-28T23:31:53.874872+0000","last_undegraded":"2024-03-28T23:31:53.874872+0000","last_fullsized":"2024-03-28T23:31:53.874872+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:25:19.055737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871716+0000","last_change":"2024-03-28T23:31:53.871716+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871716+0000","last_undegraded":"2024-03-28T23:31:53.871716+0000","last_fullsized":"2024-03-28T23:31:53.871716+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.895972+0000","last_change":"2024-03-28T23:31:53.890575+0000","last_active":"2024-03-28T23:31:53.895972+0000","last_peered":"2024-03-28T23:31:53.895972+0000","last_clean":"2024-03-28T23:31:53.895972+0000","last_became_active":"2024-03-28T23:31:53.890305+0000","last_became_peered":"2024-03-28T23:31:53.890305+0000","last_unstale":"2024-03-28T23:31:53.895972+0000","last_undegraded":"2024-03-28T23:31:53.895972+0000","last_fullsized":"2024-03-28T23:31:53.895972+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:31:25.647111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876751+0000","last_change":"2024-03-28T23:31:40.389210+0000","last_active":"2024-03-28T23:31:53.876751+0000","last_peered":"2024-03-28T23:31:53.876751+0000","last_clean":"2024-03-28T23:31:53.876751+0000","last_became_active":"2024-03-28T23:31:40.388573+0000","last_became_peered":"2024-03-28T23:31:40.388573+0000","last_unstale":"2024-03-28T23:31:53.876751+0000","last_undegraded":"2024-03-28T23:31:53.876751+0000","last_fullsized":"2024-03-28T23:31:53.876751+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:23:53.153369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"51'1","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876646+0000","last_change":"2024-03-28T23:31:39.381202+0000","last_active":"2024-03-28T23:31:53.876646+0000","last_peered":"2024-03-28T23:31:53.876646+0000","last_clean":"2024-03-28T23:31:53.876646+0000","last_became_active":"2024-03-28T23:31:39.380913+0000","last_became_peered":"2024-03-28T23:31:39.380913+0000","last_unstale":"2024-03-28T23:31:53.876646+0000","last_undegraded":"2024-03-28T23:31:53.876646+0000","last_fullsized":"2024-03-28T23:31:53.876646+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:38:21.050492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":233,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.872708+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:31:53.872708+0000","last_peered":"2024-03-28T23:31:53.872708+0000","last_clean":"2024-03-28T23:31:53.872708+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:31:53.872708+0000","last_undegraded":"2024-03-28T23:31:53.872708+0000","last_fullsized":"2024-03-28T23:31:53.872708+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":704512,"data_stored":875469,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574878,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31792,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":626688,"data_stored":278115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574878,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31784,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340056,"statfs":{"total":96636764160,"available":96604217344,"internally_reserved":0,"allocated":618496,"data_stored":277463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574877,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31788,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340052,"statfs":{"total":96636764160,"available":96604213248,"internally_reserved":0,"allocated":622592,"data_stored":278045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":634880,"data_stored":279097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":630784,"data_stored":279075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":285137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":704512,"data_stored":875401,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:31:58.442 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:31:58.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.576+0000 7ff769823640 1 Processor -- start 2024-03-28T23:31:58.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.576+0000 7ff769823640 1 -- start start 2024-03-28T23:31:58.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.576+0000 7ff769823640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff76406ca20 0x7ff76406ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:58.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.576+0000 7ff769823640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff7640a98c0 0x7ff7640a9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:58.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.576+0000 7ff769823640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff764151850 con 0x7ff76414eca0 2024-03-28T23:31:58.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.576+0000 7ff769823640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff764152460 con 0x7ff7640a98c0 2024-03-28T23:31:58.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.576+0000 7ff769823640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff764151b30 con 0x7ff76406ca20 2024-03-28T23:31:58.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.576+0000 7ff7637fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff7640a98c0 0x7ff7640a9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:58.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.577+0000 7ff7637fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff7640a98c0 0x7ff7640a9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35210/0 (socket says 172.21.15.161:35210) 2024-03-28T23:31:58.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.577+0000 7ff762ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ff76414eca0 0x7ff7640c1880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37820/0 (socket says 172.21.15.161:37820) 2024-03-28T23:31:58.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.577+0000 7ff7637fe640 1 -- 172.21.15.161:0/3148127593 learned_addr learned my addr 172.21.15.161:0/3148127593 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:58.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.577+0000 7ff7627fc640 1 --2- 172.21.15.161:0/3148127593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff76406ca20 0x7ff76406ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:58.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.577+0000 7ff7637fe640 1 -- 172.21.15.161:0/3148127593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff76406ca20 msgr2=0x7ff76406ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:58.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.577+0000 7ff7637fe640 1 --2- 172.21.15.161:0/3148127593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff76406ca20 0x7ff76406ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.577+0000 7ff7637fe640 1 -- 172.21.15.161:0/3148127593 >> v1:172.21.15.161:6789/0 conn(0x7ff76414eca0 legacy=0x7ff7640c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:58.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.577+0000 7ff7637fe640 1 -- 172.21.15.161:0/3148127593 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff764151e10 con 0x7ff7640a98c0 2024-03-28T23:31:58.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.577+0000 7ff761ffb640 1 -- 172.21.15.161:0/3148127593 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 138279448 0 0) 0x7ff764151850 con 0x7ff76414eca0 2024-03-28T23:31:58.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.577+0000 7ff7627fc640 1 --2- 172.21.15.161:0/3148127593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff76406ca20 0x7ff76406ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:31:58.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.578+0000 7ff7637fe640 1 --2- 172.21.15.161:0/3148127593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7640a98c0 0x7ff7640a9d20 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7ff75c00d150 tx=0x7ff75c02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d70c2a4a632bc61c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:58.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.578+0000 7ff761ffb640 1 -- 172.21.15.161:0/3148127593 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75c02ea80 con 0x7ff7640a98c0 2024-03-28T23:31:58.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.578+0000 7ff761ffb640 1 -- 172.21.15.161:0/3148127593 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff75c02ec20 con 0x7ff7640a98c0 2024-03-28T23:31:58.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.578+0000 7ff761ffb640 1 -- 172.21.15.161:0/3148127593 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75c009600 con 0x7ff7640a98c0 2024-03-28T23:31:58.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.579+0000 7ff769823640 1 -- 172.21.15.161:0/3148127593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7640a98c0 msgr2=0x7ff7640a9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:58.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.579+0000 7ff769823640 1 --2- 172.21.15.161:0/3148127593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7640a98c0 0x7ff7640a9d20 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7ff75c00d150 tx=0x7ff75c02e0e0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.579+0000 7ff769823640 1 -- 172.21.15.161:0/3148127593 shutdown_connections 2024-03-28T23:31:58.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.579+0000 7ff769823640 1 --2- 172.21.15.161:0/3148127593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7640a98c0 0x7ff7640a9d20 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.579+0000 7ff769823640 1 --2- 172.21.15.161:0/3148127593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff76406ca20 0x7ff76406ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.579+0000 7ff769823640 1 -- 172.21.15.161:0/3148127593 >> 172.21.15.161:0/3148127593 conn(0x7ff7640c6670 msgr2=0x7ff7640c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:58.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.579+0000 7ff769823640 1 -- 172.21.15.161:0/3148127593 shutdown_connections 2024-03-28T23:31:58.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.579+0000 7ff769823640 1 -- 172.21.15.161:0/3148127593 wait complete. 2024-03-28T23:31:58.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.580+0000 7ff769823640 1 Processor -- start 2024-03-28T23:31:58.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.580+0000 7ff769823640 1 -- start start 2024-03-28T23:31:58.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.581+0000 7ff769823640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff76406ca20 0x7ff76414d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:58.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.581+0000 7ff769823640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7640a98c0 0x7ff7641529d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:58.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.581+0000 7ff769823640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff76414eca0 0x7ff7641548f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:58.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.581+0000 7ff769823640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff7641514b0 con 0x7ff76414eca0 2024-03-28T23:31:58.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.581+0000 7ff769823640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff76414f080 con 0x7ff76406ca20 2024-03-28T23:31:58.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.581+0000 7ff769823640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff764151630 con 0x7ff7640a98c0 2024-03-28T23:31:58.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.581+0000 7ff7637fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff76414eca0 0x7ff7641548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:58.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.581+0000 7ff7637fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff76414eca0 0x7ff7641548f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35220/0 (socket says 172.21.15.161:35220) 2024-03-28T23:31:58.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.581+0000 7ff7637fe640 1 -- 172.21.15.161:0/2178109302 learned_addr learned my addr 172.21.15.161:0/2178109302 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:58.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.581+0000 7ff7627fc640 1 --2- 172.21.15.161:0/2178109302 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7640a98c0 0x7ff7641529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:58.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.581+0000 7ff762ffd640 1 --2- 172.21.15.161:0/2178109302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff76406ca20 0x7ff76414d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:58.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.581+0000 7ff7637fe640 1 -- 172.21.15.161:0/2178109302 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7640a98c0 msgr2=0x7ff7641529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:58.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.581+0000 7ff7637fe640 1 --2- 172.21.15.161:0/2178109302 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7640a98c0 0x7ff7641529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.581+0000 7ff7637fe640 1 -- 172.21.15.161:0/2178109302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff76406ca20 msgr2=0x7ff76414d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:58.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.581+0000 7ff7637fe640 1 --2- 172.21.15.161:0/2178109302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff76406ca20 0x7ff76414d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.581+0000 7ff7637fe640 1 -- 172.21.15.161:0/2178109302 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7641eeae0 con 0x7ff76414eca0 2024-03-28T23:31:58.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.581+0000 7ff7637fe640 1 --2- 172.21.15.161:0/2178109302 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff76414eca0 0x7ff7641548f0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7ff75c038040 tx=0x7ff75c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:58.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.581+0000 7ff74bfff640 1 -- 172.21.15.161:0/2178109302 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75c02ea80 con 0x7ff76414eca0 2024-03-28T23:31:58.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.581+0000 7ff74bfff640 1 -- 172.21.15.161:0/2178109302 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff75c045070 con 0x7ff76414eca0 2024-03-28T23:31:58.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.581+0000 7ff769823640 1 -- 172.21.15.161:0/2178109302 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7640c0280 con 0x7ff76414eca0 2024-03-28T23:31:58.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.582+0000 7ff74bfff640 1 -- 172.21.15.161:0/2178109302 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75c00b720 con 0x7ff76414eca0 2024-03-28T23:31:58.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.582+0000 7ff769823640 1 -- 172.21.15.161:0/2178109302 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7640c0840 con 0x7ff76414eca0 2024-03-28T23:31:58.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.582+0000 7ff769823640 1 -- 172.21.15.161:0/2178109302 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff76414db00 con 0x7ff76414eca0 2024-03-28T23:31:58.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.583+0000 7ff74bfff640 1 -- 172.21.15.161:0/2178109302 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff75c036a60 con 0x7ff76414eca0 2024-03-28T23:31:58.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.583+0000 7ff74bfff640 1 --2- 172.21.15.161:0/2178109302 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff75006b4d0 0x7ff75006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:58.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.584+0000 7ff74bfff640 1 -- 172.21.15.161:0/2178109302 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6957+0+0 (secure 0 0 0) 0x7ff75c0ad410 con 0x7ff76414eca0 2024-03-28T23:31:58.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.584+0000 7ff762ffd640 1 --2- 172.21.15.161:0/2178109302 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff75006b4d0 0x7ff75006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:58.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.584+0000 7ff762ffd640 1 --2- 172.21.15.161:0/2178109302 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff75006b4d0 0x7ff75006d990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff754007fd0 tx=0x7ff754007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:58.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.588+0000 7ff74bfff640 1 -- 172.21.15.161:0/2178109302 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff75c07b890 con 0x7ff76414eca0 2024-03-28T23:31:58.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.762+0000 7ff769823640 1 -- 172.21.15.161:0/2178109302 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff76404f4b0 con 0x7ff75006b4d0 2024-03-28T23:31:58.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.783+0000 7ff74bfff640 1 -- 172.21.15.161:0/2178109302 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+485262 (secure 0 0 0) 0x7ff76404f4b0 con 0x7ff75006b4d0 2024-03-28T23:31:58.790 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:58.793 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:31:58.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.787+0000 7ff769823640 1 -- 172.21.15.161:0/2178109302 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff75006b4d0 msgr2=0x7ff75006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:58.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.787+0000 7ff769823640 1 --2- 172.21.15.161:0/2178109302 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff75006b4d0 0x7ff75006d990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff754007fd0 tx=0x7ff754007de0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.787+0000 7ff769823640 1 -- 172.21.15.161:0/2178109302 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff76414eca0 msgr2=0x7ff7641548f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:58.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.787+0000 7ff769823640 1 --2- 172.21.15.161:0/2178109302 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff76414eca0 0x7ff7641548f0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7ff75c038040 tx=0x7ff75c004a40 comp rx=0 tx=0).stop 2024-03-28T23:31:58.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.788+0000 7ff769823640 1 -- 172.21.15.161:0/2178109302 shutdown_connections 2024-03-28T23:31:58.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.788+0000 7ff769823640 1 --2- 172.21.15.161:0/2178109302 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff75006b4d0 0x7ff75006d990 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.788+0000 7ff769823640 1 --2- 172.21.15.161:0/2178109302 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff76414eca0 0x7ff7641548f0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.788+0000 7ff769823640 1 --2- 172.21.15.161:0/2178109302 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7640a98c0 0x7ff7641529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.788+0000 7ff769823640 1 --2- 172.21.15.161:0/2178109302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff76406ca20 0x7ff76414d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.788+0000 7ff769823640 1 -- 172.21.15.161:0/2178109302 >> 172.21.15.161:0/2178109302 conn(0x7ff7640c6670 msgr2=0x7ff764150d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:58.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.788+0000 7ff769823640 1 -- 172.21.15.161:0/2178109302 shutdown_connections 2024-03-28T23:31:58.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.788+0000 7ff769823640 1 -- 172.21.15.161:0/2178109302 wait complete. 2024-03-28T23:31:58.799 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2024-03-28T23:31:57.336508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":232,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237868,"kb_used_data":5060,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736852,"statfs":{"total":773094113280,"available":772850536448,"internally_reserved":0,"allocated":5181440,"data_stored":3427802,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.470377"},"pg_stats":[{"pgid":"10.3d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872212+0000","last_change":"2024-03-28T23:31:53.872212+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872212+0000","last_undegraded":"2024-03-28T23:31:53.872212+0000","last_fullsized":"2024-03-28T23:31:53.872212+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3f","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.870419+0000","last_change":"2024-03-28T23:31:40.388090+0000","last_active":"2024-03-28T23:31:53.870419+0000","last_peered":"2024-03-28T23:31:53.870419+0000","last_clean":"2024-03-28T23:31:53.870419+0000","last_became_active":"2024-03-28T23:31:40.387805+0000","last_became_peered":"2024-03-28T23:31:40.387805+0000","last_unstale":"2024-03-28T23:31:53.870419+0000","last_undegraded":"2024-03-28T23:31:53.870419+0000","last_fullsized":"2024-03-28T23:31:53.870419+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:19:47.987753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.881085+0000","last_change":"2024-03-28T23:31:53.881085+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.881085+0000","last_undegraded":"2024-03-28T23:31:53.881085+0000","last_fullsized":"2024-03-28T23:31:53.881085+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875872+0000","last_change":"2024-03-28T23:31:40.395246+0000","last_active":"2024-03-28T23:31:53.875872+0000","last_peered":"2024-03-28T23:31:53.875872+0000","last_clean":"2024-03-28T23:31:53.875872+0000","last_became_active":"2024-03-28T23:31:40.394583+0000","last_became_peered":"2024-03-28T23:31:40.394583+0000","last_unstale":"2024-03-28T23:31:53.875872+0000","last_undegraded":"2024-03-28T23:31:53.875872+0000","last_fullsized":"2024-03-28T23:31:53.875872+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:18.654331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.873340+0000","last_change":"2024-03-28T23:31:53.873340+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.873340+0000","last_undegraded":"2024-03-28T23:31:53.873340+0000","last_fullsized":"2024-03-28T23:31:53.873340+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876107+0000","last_change":"2024-03-28T23:31:40.397914+0000","last_active":"2024-03-28T23:31:53.876107+0000","last_peered":"2024-03-28T23:31:53.876107+0000","last_clean":"2024-03-28T23:31:53.876107+0000","last_became_active":"2024-03-28T23:31:40.394904+0000","last_became_peered":"2024-03-28T23:31:40.394904+0000","last_unstale":"2024-03-28T23:31:53.876107+0000","last_undegraded":"2024-03-28T23:31:53.876107+0000","last_fullsized":"2024-03-28T23:31:53.876107+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:26:03.062546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872399+0000","last_change":"2024-03-28T23:31:53.872399+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872399+0000","last_undegraded":"2024-03-28T23:31:53.872399+0000","last_fullsized":"2024-03-28T23:31:53.872399+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874319+0000","last_change":"2024-03-28T23:31:40.400594+0000","last_active":"2024-03-28T23:31:53.874319+0000","last_peered":"2024-03-28T23:31:53.874319+0000","last_clean":"2024-03-28T23:31:53.874319+0000","last_became_active":"2024-03-28T23:31:40.400255+0000","last_became_peered":"2024-03-28T23:31:40.400255+0000","last_unstale":"2024-03-28T23:31:53.874319+0000","last_undegraded":"2024-03-28T23:31:53.874319+0000","last_fullsized":"2024-03-28T23:31:53.874319+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:14.515069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.877696+0000","last_change":"2024-03-28T23:31:53.877696+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.877696+0000","last_undegraded":"2024-03-28T23:31:53.877696+0000","last_fullsized":"2024-03-28T23:31:53.877696+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.884068+0000","last_change":"2024-03-28T23:31:40.388531+0000","last_active":"2024-03-28T23:31:53.884068+0000","last_peered":"2024-03-28T23:31:53.884068+0000","last_clean":"2024-03-28T23:31:53.884068+0000","last_became_active":"2024-03-28T23:31:40.388206+0000","last_became_peered":"2024-03-28T23:31:40.388206+0000","last_unstale":"2024-03-28T23:31:53.884068+0000","last_undegraded":"2024-03-28T23:31:53.884068+0000","last_fullsized":"2024-03-28T23:31:53.884068+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:23.215366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.881544+0000","last_change":"2024-03-28T23:31:53.881544+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.881544+0000","last_undegraded":"2024-03-28T23:31:53.881544+0000","last_fullsized":"2024-03-28T23:31:53.881544+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869622+0000","last_change":"2024-03-28T23:31:40.393907+0000","last_active":"2024-03-28T23:31:53.869622+0000","last_peered":"2024-03-28T23:31:53.869622+0000","last_clean":"2024-03-28T23:31:53.869622+0000","last_became_active":"2024-03-28T23:31:40.393443+0000","last_became_peered":"2024-03-28T23:31:40.393443+0000","last_unstale":"2024-03-28T23:31:53.869622+0000","last_undegraded":"2024-03-28T23:31:53.869622+0000","last_fullsized":"2024-03-28T23:31:53.869622+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:28:29.180775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872259+0000","last_change":"2024-03-28T23:31:53.872259+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872259+0000","last_undegraded":"2024-03-28T23:31:53.872259+0000","last_fullsized":"2024-03-28T23:31:53.872259+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875554+0000","last_change":"2024-03-28T23:31:40.398876+0000","last_active":"2024-03-28T23:31:53.875554+0000","last_peered":"2024-03-28T23:31:53.875554+0000","last_clean":"2024-03-28T23:31:53.875554+0000","last_became_active":"2024-03-28T23:31:40.398595+0000","last_became_peered":"2024-03-28T23:31:40.398595+0000","last_unstale":"2024-03-28T23:31:53.875554+0000","last_undegraded":"2024-03-28T23:31:53.875554+0000","last_fullsized":"2024-03-28T23:31:53.875554+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:21.725238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874359+0000","last_change":"2024-03-28T23:31:53.874359+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874359+0000","last_undegraded":"2024-03-28T23:31:53.874359+0000","last_fullsized":"2024-03-28T23:31:53.874359+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.880118+0000","last_change":"2024-03-28T23:31:40.392421+0000","last_active":"2024-03-28T23:31:53.880118+0000","last_peered":"2024-03-28T23:31:53.880118+0000","last_clean":"2024-03-28T23:31:53.880118+0000","last_became_active":"2024-03-28T23:31:40.391463+0000","last_became_peered":"2024-03-28T23:31:40.391463+0000","last_unstale":"2024-03-28T23:31:53.880118+0000","last_undegraded":"2024-03-28T23:31:53.880118+0000","last_fullsized":"2024-03-28T23:31:53.880118+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:18:16.851435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878422+0000","last_change":"2024-03-28T23:31:53.878422+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878422+0000","last_undegraded":"2024-03-28T23:31:53.878422+0000","last_fullsized":"2024-03-28T23:31:53.878422+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.878936+0000","last_change":"2024-03-28T23:31:40.403340+0000","last_active":"2024-03-28T23:31:53.878936+0000","last_peered":"2024-03-28T23:31:53.878936+0000","last_clean":"2024-03-28T23:31:53.878936+0000","last_became_active":"2024-03-28T23:31:40.403089+0000","last_became_peered":"2024-03-28T23:31:40.403089+0000","last_unstale":"2024-03-28T23:31:53.878936+0000","last_undegraded":"2024-03-28T23:31:53.878936+0000","last_fullsized":"2024-03-28T23:31:53.878936+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:12:38.242108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.883968+0000","last_change":"2024-03-28T23:31:53.883968+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.883968+0000","last_undegraded":"2024-03-28T23:31:53.883968+0000","last_fullsized":"2024-03-28T23:31:53.883968+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.870177+0000","last_change":"2024-03-28T23:31:40.382204+0000","last_active":"2024-03-28T23:31:53.870177+0000","last_peered":"2024-03-28T23:31:53.870177+0000","last_clean":"2024-03-28T23:31:53.870177+0000","last_became_active":"2024-03-28T23:31:40.381944+0000","last_became_peered":"2024-03-28T23:31:40.381944+0000","last_unstale":"2024-03-28T23:31:53.870177+0000","last_undegraded":"2024-03-28T23:31:53.870177+0000","last_fullsized":"2024-03-28T23:31:53.870177+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:14.515069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871125+0000","last_change":"2024-03-28T23:31:53.871125+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871125+0000","last_undegraded":"2024-03-28T23:31:53.871125+0000","last_fullsized":"2024-03-28T23:31:53.871125+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.883144+0000","last_change":"2024-03-28T23:31:40.394578+0000","last_active":"2024-03-28T23:31:53.883144+0000","last_peered":"2024-03-28T23:31:53.883144+0000","last_clean":"2024-03-28T23:31:53.883144+0000","last_became_active":"2024-03-28T23:31:40.394272+0000","last_became_peered":"2024-03-28T23:31:40.394272+0000","last_unstale":"2024-03-28T23:31:53.883144+0000","last_undegraded":"2024-03-28T23:31:53.883144+0000","last_fullsized":"2024-03-28T23:31:53.883144+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:24:35.708796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871848+0000","last_change":"2024-03-28T23:31:53.871848+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871848+0000","last_undegraded":"2024-03-28T23:31:53.871848+0000","last_fullsized":"2024-03-28T23:31:53.871848+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879847+0000","last_change":"2024-03-28T23:31:40.396702+0000","last_active":"2024-03-28T23:31:53.879847+0000","last_peered":"2024-03-28T23:31:53.879847+0000","last_clean":"2024-03-28T23:31:53.879847+0000","last_became_active":"2024-03-28T23:31:40.395941+0000","last_became_peered":"2024-03-28T23:31:40.395941+0000","last_unstale":"2024-03-28T23:31:53.879847+0000","last_undegraded":"2024-03-28T23:31:53.879847+0000","last_fullsized":"2024-03-28T23:31:53.879847+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:07:13.690791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871018+0000","last_change":"2024-03-28T23:31:53.871018+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871018+0000","last_undegraded":"2024-03-28T23:31:53.871018+0000","last_fullsized":"2024-03-28T23:31:53.871018+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869713+0000","last_change":"2024-03-28T23:31:40.382947+0000","last_active":"2024-03-28T23:31:53.869713+0000","last_peered":"2024-03-28T23:31:53.869713+0000","last_clean":"2024-03-28T23:31:53.869713+0000","last_became_active":"2024-03-28T23:31:40.381893+0000","last_became_peered":"2024-03-28T23:31:40.381893+0000","last_unstale":"2024-03-28T23:31:53.869713+0000","last_undegraded":"2024-03-28T23:31:53.869713+0000","last_fullsized":"2024-03-28T23:31:53.869713+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:50.872886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.877447+0000","last_change":"2024-03-28T23:31:53.877447+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.877447+0000","last_undegraded":"2024-03-28T23:31:53.877447+0000","last_fullsized":"2024-03-28T23:31:53.877447+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875836+0000","last_change":"2024-03-28T23:31:40.405627+0000","last_active":"2024-03-28T23:31:53.875836+0000","last_peered":"2024-03-28T23:31:53.875836+0000","last_clean":"2024-03-28T23:31:53.875836+0000","last_became_active":"2024-03-28T23:31:40.405360+0000","last_became_peered":"2024-03-28T23:31:40.405360+0000","last_unstale":"2024-03-28T23:31:53.875836+0000","last_undegraded":"2024-03-28T23:31:53.875836+0000","last_fullsized":"2024-03-28T23:31:53.875836+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:06.567694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.881048+0000","last_change":"2024-03-28T23:31:53.881048+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.881048+0000","last_undegraded":"2024-03-28T23:31:53.881048+0000","last_fullsized":"2024-03-28T23:31:53.881048+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.873535+0000","last_change":"2024-03-28T23:31:40.396605+0000","last_active":"2024-03-28T23:31:53.873535+0000","last_peered":"2024-03-28T23:31:53.873535+0000","last_clean":"2024-03-28T23:31:53.873535+0000","last_became_active":"2024-03-28T23:31:40.396237+0000","last_became_peered":"2024-03-28T23:31:40.396237+0000","last_unstale":"2024-03-28T23:31:53.873535+0000","last_undegraded":"2024-03-28T23:31:53.873535+0000","last_fullsized":"2024-03-28T23:31:53.873535+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:14.144440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871402+0000","last_change":"2024-03-28T23:31:53.871402+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871402+0000","last_undegraded":"2024-03-28T23:31:53.871402+0000","last_fullsized":"2024-03-28T23:31:53.871402+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869477+0000","last_change":"2024-03-28T23:31:40.382823+0000","last_active":"2024-03-28T23:31:53.869477+0000","last_peered":"2024-03-28T23:31:53.869477+0000","last_clean":"2024-03-28T23:31:53.869477+0000","last_became_active":"2024-03-28T23:31:40.382574+0000","last_became_peered":"2024-03-28T23:31:40.382574+0000","last_unstale":"2024-03-28T23:31:53.869477+0000","last_undegraded":"2024-03-28T23:31:53.869477+0000","last_fullsized":"2024-03-28T23:31:53.869477+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:12:27.477720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872681+0000","last_change":"2024-03-28T23:31:53.872681+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872681+0000","last_undegraded":"2024-03-28T23:31:53.872681+0000","last_fullsized":"2024-03-28T23:31:53.872681+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876715+0000","last_change":"2024-03-28T23:31:40.403462+0000","last_active":"2024-03-28T23:31:53.876715+0000","last_peered":"2024-03-28T23:31:53.876715+0000","last_clean":"2024-03-28T23:31:53.876715+0000","last_became_active":"2024-03-28T23:31:40.402885+0000","last_became_peered":"2024-03-28T23:31:40.402885+0000","last_unstale":"2024-03-28T23:31:53.876715+0000","last_undegraded":"2024-03-28T23:31:53.876715+0000","last_fullsized":"2024-03-28T23:31:53.876715+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:42:29.351941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.879388+0000","last_change":"2024-03-28T23:31:53.879388+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.879388+0000","last_undegraded":"2024-03-28T23:31:53.879388+0000","last_fullsized":"2024-03-28T23:31:53.879388+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875969+0000","last_change":"2024-03-28T23:31:40.395478+0000","last_active":"2024-03-28T23:31:53.875969+0000","last_peered":"2024-03-28T23:31:53.875969+0000","last_clean":"2024-03-28T23:31:53.875969+0000","last_became_active":"2024-03-28T23:31:40.394981+0000","last_became_peered":"2024-03-28T23:31:40.394981+0000","last_unstale":"2024-03-28T23:31:53.875969+0000","last_undegraded":"2024-03-28T23:31:53.875969+0000","last_fullsized":"2024-03-28T23:31:53.875969+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:06.567694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872394+0000","last_change":"2024-03-28T23:31:53.872394+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872394+0000","last_undegraded":"2024-03-28T23:31:53.872394+0000","last_fullsized":"2024-03-28T23:31:53.872394+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876119+0000","last_change":"2024-03-28T23:31:40.394475+0000","last_active":"2024-03-28T23:31:53.876119+0000","last_peered":"2024-03-28T23:31:53.876119+0000","last_clean":"2024-03-28T23:31:53.876119+0000","last_became_active":"2024-03-28T23:31:40.390205+0000","last_became_peered":"2024-03-28T23:31:40.390205+0000","last_unstale":"2024-03-28T23:31:53.876119+0000","last_undegraded":"2024-03-28T23:31:53.876119+0000","last_fullsized":"2024-03-28T23:31:53.876119+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:04:48.226221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.882562+0000","last_change":"2024-03-28T23:31:53.882562+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.882562+0000","last_undegraded":"2024-03-28T23:31:53.882562+0000","last_fullsized":"2024-03-28T23:31:53.882562+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.877115+0000","last_change":"2024-03-28T23:31:40.388386+0000","last_active":"2024-03-28T23:31:53.877115+0000","last_peered":"2024-03-28T23:31:53.877115+0000","last_clean":"2024-03-28T23:31:53.877115+0000","last_became_active":"2024-03-28T23:31:40.388043+0000","last_became_peered":"2024-03-28T23:31:40.388043+0000","last_unstale":"2024-03-28T23:31:53.877115+0000","last_undegraded":"2024-03-28T23:31:53.877115+0000","last_fullsized":"2024-03-28T23:31:53.877115+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:07.738674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.880460+0000","last_change":"2024-03-28T23:31:53.880460+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.880460+0000","last_undegraded":"2024-03-28T23:31:53.880460+0000","last_fullsized":"2024-03-28T23:31:53.880460+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875638+0000","last_change":"2024-03-28T23:31:40.395811+0000","last_active":"2024-03-28T23:31:53.875638+0000","last_peered":"2024-03-28T23:31:53.875638+0000","last_clean":"2024-03-28T23:31:53.875638+0000","last_became_active":"2024-03-28T23:31:40.395540+0000","last_became_peered":"2024-03-28T23:31:40.395540+0000","last_unstale":"2024-03-28T23:31:53.875638+0000","last_undegraded":"2024-03-28T23:31:53.875638+0000","last_fullsized":"2024-03-28T23:31:53.875638+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:51.864986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.870958+0000","last_change":"2024-03-28T23:31:53.870958+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.870958+0000","last_undegraded":"2024-03-28T23:31:53.870958+0000","last_fullsized":"2024-03-28T23:31:53.870958+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.881134+0000","last_change":"2024-03-28T23:31:40.403129+0000","last_active":"2024-03-28T23:31:53.881134+0000","last_peered":"2024-03-28T23:31:53.881134+0000","last_clean":"2024-03-28T23:31:53.881134+0000","last_became_active":"2024-03-28T23:31:40.402879+0000","last_became_peered":"2024-03-28T23:31:40.402879+0000","last_unstale":"2024-03-28T23:31:53.881134+0000","last_undegraded":"2024-03-28T23:31:53.881134+0000","last_fullsized":"2024-03-28T23:31:53.881134+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:47.861300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.879206+0000","last_change":"2024-03-28T23:31:53.879206+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.879206+0000","last_undegraded":"2024-03-28T23:31:53.879206+0000","last_fullsized":"2024-03-28T23:31:53.879206+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879397+0000","last_change":"2024-03-28T23:31:40.389854+0000","last_active":"2024-03-28T23:31:53.879397+0000","last_peered":"2024-03-28T23:31:53.879397+0000","last_clean":"2024-03-28T23:31:53.879397+0000","last_became_active":"2024-03-28T23:31:40.389599+0000","last_became_peered":"2024-03-28T23:31:40.389599+0000","last_unstale":"2024-03-28T23:31:53.879397+0000","last_undegraded":"2024-03-28T23:31:53.879397+0000","last_fullsized":"2024-03-28T23:31:53.879397+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:43:37.883841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.880913+0000","last_change":"2024-03-28T23:31:53.880913+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.880913+0000","last_undegraded":"2024-03-28T23:31:53.880913+0000","last_fullsized":"2024-03-28T23:31:53.880913+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875417+0000","last_change":"2024-03-28T23:31:40.406317+0000","last_active":"2024-03-28T23:31:53.875417+0000","last_peered":"2024-03-28T23:31:53.875417+0000","last_clean":"2024-03-28T23:31:53.875417+0000","last_became_active":"2024-03-28T23:31:40.405217+0000","last_became_peered":"2024-03-28T23:31:40.405217+0000","last_unstale":"2024-03-28T23:31:53.875417+0000","last_undegraded":"2024-03-28T23:31:53.875417+0000","last_fullsized":"2024-03-28T23:31:53.875417+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:37:55.870956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.879370+0000","last_change":"2024-03-28T23:31:53.879370+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.879370+0000","last_undegraded":"2024-03-28T23:31:53.879370+0000","last_fullsized":"2024-03-28T23:31:53.879370+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876750+0000","last_change":"2024-03-28T23:31:40.404249+0000","last_active":"2024-03-28T23:31:53.876750+0000","last_peered":"2024-03-28T23:31:53.876750+0000","last_clean":"2024-03-28T23:31:53.876750+0000","last_became_active":"2024-03-28T23:31:40.403966+0000","last_became_peered":"2024-03-28T23:31:40.403966+0000","last_unstale":"2024-03-28T23:31:53.876750+0000","last_undegraded":"2024-03-28T23:31:53.876750+0000","last_fullsized":"2024-03-28T23:31:53.876750+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:40:57.541676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872876+0000","last_change":"2024-03-28T23:31:53.872876+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872876+0000","last_undegraded":"2024-03-28T23:31:53.872876+0000","last_fullsized":"2024-03-28T23:31:53.872876+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.878912+0000","last_change":"2024-03-28T23:31:40.397170+0000","last_active":"2024-03-28T23:31:53.878912+0000","last_peered":"2024-03-28T23:31:53.878912+0000","last_clean":"2024-03-28T23:31:53.878912+0000","last_became_active":"2024-03-28T23:31:40.396898+0000","last_became_peered":"2024-03-28T23:31:40.396898+0000","last_unstale":"2024-03-28T23:31:53.878912+0000","last_undegraded":"2024-03-28T23:31:53.878912+0000","last_fullsized":"2024-03-28T23:31:53.878912+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:53:26.355348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878874+0000","last_change":"2024-03-28T23:31:53.878874+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878874+0000","last_undegraded":"2024-03-28T23:31:53.878874+0000","last_fullsized":"2024-03-28T23:31:53.878874+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.870080+0000","last_change":"2024-03-28T23:31:40.404955+0000","last_active":"2024-03-28T23:31:53.870080+0000","last_peered":"2024-03-28T23:31:53.870080+0000","last_clean":"2024-03-28T23:31:53.870080+0000","last_became_active":"2024-03-28T23:31:40.404689+0000","last_became_peered":"2024-03-28T23:31:40.404689+0000","last_unstale":"2024-03-28T23:31:53.870080+0000","last_undegraded":"2024-03-28T23:31:53.870080+0000","last_fullsized":"2024-03-28T23:31:53.870080+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:50:20.574648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878430+0000","last_change":"2024-03-28T23:31:53.878430+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878430+0000","last_undegraded":"2024-03-28T23:31:53.878430+0000","last_fullsized":"2024-03-28T23:31:53.878430+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.877218+0000","last_change":"2024-03-28T23:31:40.389261+0000","last_active":"2024-03-28T23:31:53.877218+0000","last_peered":"2024-03-28T23:31:53.877218+0000","last_clean":"2024-03-28T23:31:53.877218+0000","last_became_active":"2024-03-28T23:31:40.388930+0000","last_became_peered":"2024-03-28T23:31:40.388930+0000","last_unstale":"2024-03-28T23:31:53.877218+0000","last_undegraded":"2024-03-28T23:31:53.877218+0000","last_fullsized":"2024-03-28T23:31:53.877218+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:23.215366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878783+0000","last_change":"2024-03-28T23:31:53.878783+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878783+0000","last_undegraded":"2024-03-28T23:31:53.878783+0000","last_fullsized":"2024-03-28T23:31:53.878783+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875578+0000","last_change":"2024-03-28T23:31:40.401673+0000","last_active":"2024-03-28T23:31:53.875578+0000","last_peered":"2024-03-28T23:31:53.875578+0000","last_clean":"2024-03-28T23:31:53.875578+0000","last_became_active":"2024-03-28T23:31:40.398923+0000","last_became_peered":"2024-03-28T23:31:40.398923+0000","last_unstale":"2024-03-28T23:31:53.875578+0000","last_undegraded":"2024-03-28T23:31:53.875578+0000","last_fullsized":"2024-03-28T23:31:53.875578+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:18.783120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874991+0000","last_change":"2024-03-28T23:31:53.874991+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874991+0000","last_undegraded":"2024-03-28T23:31:53.874991+0000","last_fullsized":"2024-03-28T23:31:53.874991+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.873933+0000","last_change":"2024-03-28T23:31:40.387082+0000","last_active":"2024-03-28T23:31:53.873933+0000","last_peered":"2024-03-28T23:31:53.873933+0000","last_clean":"2024-03-28T23:31:53.873933+0000","last_became_active":"2024-03-28T23:31:40.386808+0000","last_became_peered":"2024-03-28T23:31:40.386808+0000","last_unstale":"2024-03-28T23:31:53.873933+0000","last_undegraded":"2024-03-28T23:31:53.873933+0000","last_fullsized":"2024-03-28T23:31:53.873933+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:43.111058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.870965+0000","last_change":"2024-03-28T23:31:53.870965+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.870965+0000","last_undegraded":"2024-03-28T23:31:53.870965+0000","last_fullsized":"2024-03-28T23:31:53.870965+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875611+0000","last_change":"2024-03-28T23:31:40.397815+0000","last_active":"2024-03-28T23:31:53.875611+0000","last_peered":"2024-03-28T23:31:53.875611+0000","last_clean":"2024-03-28T23:31:53.875611+0000","last_became_active":"2024-03-28T23:31:40.397279+0000","last_became_peered":"2024-03-28T23:31:40.397279+0000","last_unstale":"2024-03-28T23:31:53.875611+0000","last_undegraded":"2024-03-28T23:31:53.875611+0000","last_fullsized":"2024-03-28T23:31:53.875611+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:25:57.449014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878422+0000","last_change":"2024-03-28T23:31:53.878422+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878422+0000","last_undegraded":"2024-03-28T23:31:53.878422+0000","last_fullsized":"2024-03-28T23:31:53.878422+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874691+0000","last_change":"2024-03-28T23:31:40.394484+0000","last_active":"2024-03-28T23:31:53.874691+0000","last_peered":"2024-03-28T23:31:53.874691+0000","last_clean":"2024-03-28T23:31:53.874691+0000","last_became_active":"2024-03-28T23:31:40.394219+0000","last_became_peered":"2024-03-28T23:31:40.394219+0000","last_unstale":"2024-03-28T23:31:53.874691+0000","last_undegraded":"2024-03-28T23:31:53.874691+0000","last_fullsized":"2024-03-28T23:31:53.874691+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:18.654331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874735+0000","last_change":"2024-03-28T23:31:53.874735+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874735+0000","last_undegraded":"2024-03-28T23:31:53.874735+0000","last_fullsized":"2024-03-28T23:31:53.874735+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891397+0000","last_change":"2024-03-28T23:31:53.890105+0000","last_active":"2024-03-28T23:31:53.891397+0000","last_peered":"2024-03-28T23:31:53.891397+0000","last_clean":"2024-03-28T23:31:53.891397+0000","last_became_active":"2024-03-28T23:31:53.889841+0000","last_became_peered":"2024-03-28T23:31:53.889841+0000","last_unstale":"2024-03-28T23:31:53.891397+0000","last_undegraded":"2024-03-28T23:31:53.891397+0000","last_fullsized":"2024-03-28T23:31:53.891397+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:01.360626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.872604+0000","last_change":"2024-03-28T23:31:40.398232+0000","last_active":"2024-03-28T23:31:53.872604+0000","last_peered":"2024-03-28T23:31:53.872604+0000","last_clean":"2024-03-28T23:31:53.872604+0000","last_became_active":"2024-03-28T23:31:40.397888+0000","last_became_peered":"2024-03-28T23:31:40.397888+0000","last_unstale":"2024-03-28T23:31:53.872604+0000","last_undegraded":"2024-03-28T23:31:53.872604+0000","last_fullsized":"2024-03-28T23:31:53.872604+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:50.872886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"50'4","reported_seq":22,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879785+0000","last_change":"2024-03-28T23:31:39.385488+0000","last_active":"2024-03-28T23:31:53.879785+0000","last_peered":"2024-03-28T23:31:53.879785+0000","last_clean":"2024-03-28T23:31:53.879785+0000","last_became_active":"2024-03-28T23:31:39.385211+0000","last_became_peered":"2024-03-28T23:31:39.385211+0000","last_unstale":"2024-03-28T23:31:53.879785+0000","last_undegraded":"2024-03-28T23:31:53.879785+0000","last_fullsized":"2024-03-28T23:31:53.879785+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:50:10.559743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872455+0000","last_change":"2024-03-28T23:31:53.872455+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872455+0000","last_undegraded":"2024-03-28T23:31:53.872455+0000","last_fullsized":"2024-03-28T23:31:53.872455+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891253+0000","last_change":"2024-03-28T23:31:53.887853+0000","last_active":"2024-03-28T23:31:53.891253+0000","last_peered":"2024-03-28T23:31:53.891253+0000","last_clean":"2024-03-28T23:31:53.891253+0000","last_became_active":"2024-03-28T23:31:53.887588+0000","last_became_peered":"2024-03-28T23:31:53.887588+0000","last_unstale":"2024-03-28T23:31:53.891253+0000","last_undegraded":"2024-03-28T23:31:53.891253+0000","last_fullsized":"2024-03-28T23:31:53.891253+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:28:42.680101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.873999+0000","last_change":"2024-03-28T23:31:40.390218+0000","last_active":"2024-03-28T23:31:53.873999+0000","last_peered":"2024-03-28T23:31:53.873999+0000","last_clean":"2024-03-28T23:31:53.873999+0000","last_became_active":"2024-03-28T23:31:40.389922+0000","last_became_peered":"2024-03-28T23:31:40.389922+0000","last_unstale":"2024-03-28T23:31:53.873999+0000","last_undegraded":"2024-03-28T23:31:53.873999+0000","last_fullsized":"2024-03-28T23:31:53.873999+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:14.515069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"50'1","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.870270+0000","last_change":"2024-03-28T23:31:39.393513+0000","last_active":"2024-03-28T23:31:53.870270+0000","last_peered":"2024-03-28T23:31:53.870270+0000","last_clean":"2024-03-28T23:31:53.870270+0000","last_became_active":"2024-03-28T23:31:39.393176+0000","last_became_peered":"2024-03-28T23:31:39.393176+0000","last_unstale":"2024-03-28T23:31:53.870270+0000","last_undegraded":"2024-03-28T23:31:53.870270+0000","last_fullsized":"2024-03-28T23:31:53.870270+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:29.240985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.880671+0000","last_change":"2024-03-28T23:31:53.880671+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.880671+0000","last_undegraded":"2024-03-28T23:31:53.880671+0000","last_fullsized":"2024-03-28T23:31:53.880671+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891931+0000","last_change":"2024-03-28T23:31:53.890699+0000","last_active":"2024-03-28T23:31:53.891931+0000","last_peered":"2024-03-28T23:31:53.891931+0000","last_clean":"2024-03-28T23:31:53.891931+0000","last_became_active":"2024-03-28T23:31:53.890300+0000","last_became_peered":"2024-03-28T23:31:53.890300+0000","last_unstale":"2024-03-28T23:31:53.891931+0000","last_undegraded":"2024-03-28T23:31:53.891931+0000","last_fullsized":"2024-03-28T23:31:53.891931+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:18:30.350761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875566+0000","last_change":"2024-03-28T23:31:40.397077+0000","last_active":"2024-03-28T23:31:53.875566+0000","last_peered":"2024-03-28T23:31:53.875566+0000","last_clean":"2024-03-28T23:31:53.875566+0000","last_became_active":"2024-03-28T23:31:40.395265+0000","last_became_peered":"2024-03-28T23:31:40.395265+0000","last_unstale":"2024-03-28T23:31:53.875566+0000","last_undegraded":"2024-03-28T23:31:53.875566+0000","last_fullsized":"2024-03-28T23:31:53.875566+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:04.829030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"50'2","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.870290+0000","last_change":"2024-03-28T23:31:39.391743+0000","last_active":"2024-03-28T23:31:53.870290+0000","last_peered":"2024-03-28T23:31:53.870290+0000","last_clean":"2024-03-28T23:31:53.870290+0000","last_became_active":"2024-03-28T23:31:39.391300+0000","last_became_peered":"2024-03-28T23:31:39.391300+0000","last_unstale":"2024-03-28T23:31:53.870290+0000","last_undegraded":"2024-03-28T23:31:53.870290+0000","last_fullsized":"2024-03-28T23:31:53.870290+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:12.505382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.877711+0000","last_change":"2024-03-28T23:31:53.877711+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.877711+0000","last_undegraded":"2024-03-28T23:31:53.877711+0000","last_fullsized":"2024-03-28T23:31:53.877711+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.889315+0000","last_change":"2024-03-28T23:31:53.886833+0000","last_active":"2024-03-28T23:31:53.889315+0000","last_peered":"2024-03-28T23:31:53.889315+0000","last_clean":"2024-03-28T23:31:53.889315+0000","last_became_active":"2024-03-28T23:31:53.886498+0000","last_became_peered":"2024-03-28T23:31:53.886498+0000","last_unstale":"2024-03-28T23:31:53.889315+0000","last_undegraded":"2024-03-28T23:31:53.889315+0000","last_fullsized":"2024-03-28T23:31:53.889315+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:26:16.561871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.881876+0000","last_change":"2024-03-28T23:31:40.392391+0000","last_active":"2024-03-28T23:31:53.881876+0000","last_peered":"2024-03-28T23:31:53.881876+0000","last_clean":"2024-03-28T23:31:53.881876+0000","last_became_active":"2024-03-28T23:31:40.392100+0000","last_became_peered":"2024-03-28T23:31:40.392100+0000","last_unstale":"2024-03-28T23:31:53.881876+0000","last_undegraded":"2024-03-28T23:31:53.881876+0000","last_fullsized":"2024-03-28T23:31:53.881876+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:34:38.237729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"50'2","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874448+0000","last_change":"2024-03-28T23:31:39.385430+0000","last_active":"2024-03-28T23:31:53.874448+0000","last_peered":"2024-03-28T23:31:53.874448+0000","last_clean":"2024-03-28T23:31:53.874448+0000","last_became_active":"2024-03-28T23:31:39.385134+0000","last_became_peered":"2024-03-28T23:31:39.385134+0000","last_unstale":"2024-03-28T23:31:53.874448+0000","last_undegraded":"2024-03-28T23:31:53.874448+0000","last_fullsized":"2024-03-28T23:31:53.874448+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:12:26.472924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.873288+0000","last_change":"2024-03-28T23:31:53.873288+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.873288+0000","last_undegraded":"2024-03-28T23:31:53.873288+0000","last_fullsized":"2024-03-28T23:31:53.873288+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891323+0000","last_change":"2024-03-28T23:31:53.889140+0000","last_active":"2024-03-28T23:31:53.891323+0000","last_peered":"2024-03-28T23:31:53.891323+0000","last_clean":"2024-03-28T23:31:53.891323+0000","last_became_active":"2024-03-28T23:31:53.888738+0000","last_became_peered":"2024-03-28T23:31:53.888738+0000","last_unstale":"2024-03-28T23:31:53.891323+0000","last_undegraded":"2024-03-28T23:31:53.891323+0000","last_fullsized":"2024-03-28T23:31:53.891323+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:05:01.725546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.885385+0000","last_change":"2024-03-28T23:31:40.392757+0000","last_active":"2024-03-28T23:31:53.885385+0000","last_peered":"2024-03-28T23:31:53.885385+0000","last_clean":"2024-03-28T23:31:53.885385+0000","last_became_active":"2024-03-28T23:31:40.392460+0000","last_became_peered":"2024-03-28T23:31:40.392460+0000","last_unstale":"2024-03-28T23:31:53.885385+0000","last_undegraded":"2024-03-28T23:31:53.885385+0000","last_fullsized":"2024-03-28T23:31:53.885385+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:07.738674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"50'1","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875684+0000","last_change":"2024-03-28T23:31:39.381607+0000","last_active":"2024-03-28T23:31:53.875684+0000","last_peered":"2024-03-28T23:31:53.875684+0000","last_clean":"2024-03-28T23:31:53.875684+0000","last_became_active":"2024-03-28T23:31:39.381326+0000","last_became_peered":"2024-03-28T23:31:39.381326+0000","last_unstale":"2024-03-28T23:31:53.875684+0000","last_undegraded":"2024-03-28T23:31:53.875684+0000","last_fullsized":"2024-03-28T23:31:53.875684+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:50.860190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.880191+0000","last_change":"2024-03-28T23:31:53.880191+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.880191+0000","last_undegraded":"2024-03-28T23:31:53.880191+0000","last_fullsized":"2024-03-28T23:31:53.880191+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.888447+0000","last_change":"2024-03-28T23:31:53.886424+0000","last_active":"2024-03-28T23:31:53.888447+0000","last_peered":"2024-03-28T23:31:53.888447+0000","last_clean":"2024-03-28T23:31:53.888447+0000","last_became_active":"2024-03-28T23:31:53.886207+0000","last_became_peered":"2024-03-28T23:31:53.886207+0000","last_unstale":"2024-03-28T23:31:53.888447+0000","last_undegraded":"2024-03-28T23:31:53.888447+0000","last_fullsized":"2024-03-28T23:31:53.888447+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:26:10.948340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876571+0000","last_change":"2024-03-28T23:31:40.398712+0000","last_active":"2024-03-28T23:31:53.876571+0000","last_peered":"2024-03-28T23:31:53.876571+0000","last_clean":"2024-03-28T23:31:53.876571+0000","last_became_active":"2024-03-28T23:31:40.398392+0000","last_became_peered":"2024-03-28T23:31:40.398392+0000","last_unstale":"2024-03-28T23:31:53.876571+0000","last_undegraded":"2024-03-28T23:31:53.876571+0000","last_fullsized":"2024-03-28T23:31:53.876571+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:54.431044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"50'1","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.883791+0000","last_change":"2024-03-28T23:31:39.384026+0000","last_active":"2024-03-28T23:31:53.883791+0000","last_peered":"2024-03-28T23:31:53.883791+0000","last_clean":"2024-03-28T23:31:53.883791+0000","last_became_active":"2024-03-28T23:31:39.383161+0000","last_became_peered":"2024-03-28T23:31:39.383161+0000","last_unstale":"2024-03-28T23:31:53.883791+0000","last_undegraded":"2024-03-28T23:31:53.883791+0000","last_fullsized":"2024-03-28T23:31:53.883791+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:21:00.890266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872423+0000","last_change":"2024-03-28T23:31:53.872423+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872423+0000","last_undegraded":"2024-03-28T23:31:53.872423+0000","last_fullsized":"2024-03-28T23:31:53.872423+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.888932+0000","last_change":"2024-03-28T23:31:53.885914+0000","last_active":"2024-03-28T23:31:53.888932+0000","last_peered":"2024-03-28T23:31:53.888932+0000","last_clean":"2024-03-28T23:31:53.888932+0000","last_became_active":"2024-03-28T23:31:53.885637+0000","last_became_peered":"2024-03-28T23:31:53.885637+0000","last_unstale":"2024-03-28T23:31:53.888932+0000","last_undegraded":"2024-03-28T23:31:53.888932+0000","last_fullsized":"2024-03-28T23:31:53.888932+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:32.282446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879100+0000","last_change":"2024-03-28T23:31:40.395728+0000","last_active":"2024-03-28T23:31:53.879100+0000","last_peered":"2024-03-28T23:31:53.879100+0000","last_clean":"2024-03-28T23:31:53.879100+0000","last_became_active":"2024-03-28T23:31:40.395474+0000","last_became_peered":"2024-03-28T23:31:40.395474+0000","last_unstale":"2024-03-28T23:31:53.879100+0000","last_undegraded":"2024-03-28T23:31:53.879100+0000","last_fullsized":"2024-03-28T23:31:53.879100+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:15:19.985376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876912+0000","last_change":"2024-03-28T23:31:39.385649+0000","last_active":"2024-03-28T23:31:53.876912+0000","last_peered":"2024-03-28T23:31:53.876912+0000","last_clean":"2024-03-28T23:31:53.876912+0000","last_became_active":"2024-03-28T23:31:39.385341+0000","last_became_peered":"2024-03-28T23:31:39.385341+0000","last_unstale":"2024-03-28T23:31:53.876912+0000","last_undegraded":"2024-03-28T23:31:53.876912+0000","last_fullsized":"2024-03-28T23:31:53.876912+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:52:06.955653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878702+0000","last_change":"2024-03-28T23:31:53.878702+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878702+0000","last_undegraded":"2024-03-28T23:31:53.878702+0000","last_fullsized":"2024-03-28T23:31:53.878702+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891992+0000","last_change":"2024-03-28T23:31:53.889258+0000","last_active":"2024-03-28T23:31:53.891992+0000","last_peered":"2024-03-28T23:31:53.891992+0000","last_clean":"2024-03-28T23:31:53.891992+0000","last_became_active":"2024-03-28T23:31:53.888970+0000","last_became_peered":"2024-03-28T23:31:53.888970+0000","last_unstale":"2024-03-28T23:31:53.891992+0000","last_undegraded":"2024-03-28T23:31:53.891992+0000","last_fullsized":"2024-03-28T23:31:53.891992+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:20:01.487078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875176+0000","last_change":"2024-03-28T23:31:40.400091+0000","last_active":"2024-03-28T23:31:53.875176+0000","last_peered":"2024-03-28T23:31:53.875176+0000","last_clean":"2024-03-28T23:31:53.875176+0000","last_became_active":"2024-03-28T23:31:40.399190+0000","last_became_peered":"2024-03-28T23:31:40.399190+0000","last_unstale":"2024-03-28T23:31:53.875176+0000","last_undegraded":"2024-03-28T23:31:53.875176+0000","last_fullsized":"2024-03-28T23:31:53.875176+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:22.230986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875772+0000","last_change":"2024-03-28T23:31:39.390198+0000","last_active":"2024-03-28T23:31:53.875772+0000","last_peered":"2024-03-28T23:31:53.875772+0000","last_clean":"2024-03-28T23:31:53.875772+0000","last_became_active":"2024-03-28T23:31:39.389939+0000","last_became_peered":"2024-03-28T23:31:39.389939+0000","last_unstale":"2024-03-28T23:31:53.875772+0000","last_undegraded":"2024-03-28T23:31:53.875772+0000","last_fullsized":"2024-03-28T23:31:53.875772+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:46:30.350009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872121+0000","last_change":"2024-03-28T23:31:53.872121+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872121+0000","last_undegraded":"2024-03-28T23:31:53.872121+0000","last_fullsized":"2024-03-28T23:31:53.872121+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.896042+0000","last_change":"2024-03-28T23:31:53.892143+0000","last_active":"2024-03-28T23:31:53.896042+0000","last_peered":"2024-03-28T23:31:53.896042+0000","last_clean":"2024-03-28T23:31:53.896042+0000","last_became_active":"2024-03-28T23:31:53.891989+0000","last_became_peered":"2024-03-28T23:31:53.891989+0000","last_unstale":"2024-03-28T23:31:53.896042+0000","last_undegraded":"2024-03-28T23:31:53.896042+0000","last_fullsized":"2024-03-28T23:31:53.896042+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:23:27.129147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.883626+0000","last_change":"2024-03-28T23:31:40.384948+0000","last_active":"2024-03-28T23:31:53.883626+0000","last_peered":"2024-03-28T23:31:53.883626+0000","last_clean":"2024-03-28T23:31:53.883626+0000","last_became_active":"2024-03-28T23:31:40.384434+0000","last_became_peered":"2024-03-28T23:31:40.384434+0000","last_unstale":"2024-03-28T23:31:53.883626+0000","last_undegraded":"2024-03-28T23:31:53.883626+0000","last_fullsized":"2024-03-28T23:31:53.883626+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:04.829030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875277+0000","last_change":"2024-03-28T23:31:39.388914+0000","last_active":"2024-03-28T23:31:53.875277+0000","last_peered":"2024-03-28T23:31:53.875277+0000","last_clean":"2024-03-28T23:31:53.875277+0000","last_became_active":"2024-03-28T23:31:39.388639+0000","last_became_peered":"2024-03-28T23:31:39.388639+0000","last_unstale":"2024-03-28T23:31:53.875277+0000","last_undegraded":"2024-03-28T23:31:53.875277+0000","last_fullsized":"2024-03-28T23:31:53.875277+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:00.366278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.880894+0000","last_change":"2024-03-28T23:31:53.880894+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.880894+0000","last_undegraded":"2024-03-28T23:31:53.880894+0000","last_fullsized":"2024-03-28T23:31:53.880894+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.884484+0000","last_change":"2024-03-28T23:31:53.879950+0000","last_active":"2024-03-28T23:31:53.884484+0000","last_peered":"2024-03-28T23:31:53.884484+0000","last_clean":"2024-03-28T23:31:53.884484+0000","last_became_active":"2024-03-28T23:31:53.879620+0000","last_became_peered":"2024-03-28T23:31:53.879620+0000","last_unstale":"2024-03-28T23:31:53.884484+0000","last_undegraded":"2024-03-28T23:31:53.884484+0000","last_fullsized":"2024-03-28T23:31:53.884484+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:28:42.680101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.873900+0000","last_change":"2024-03-28T23:31:40.386749+0000","last_active":"2024-03-28T23:31:53.873900+0000","last_peered":"2024-03-28T23:31:53.873900+0000","last_clean":"2024-03-28T23:31:53.873900+0000","last_became_active":"2024-03-28T23:31:40.386466+0000","last_became_peered":"2024-03-28T23:31:40.386466+0000","last_unstale":"2024-03-28T23:31:53.873900+0000","last_undegraded":"2024-03-28T23:31:53.873900+0000","last_fullsized":"2024-03-28T23:31:53.873900+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:06.567694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.884919+0000","last_change":"2024-03-28T23:31:39.381050+0000","last_active":"2024-03-28T23:31:53.884919+0000","last_peered":"2024-03-28T23:31:53.884919+0000","last_clean":"2024-03-28T23:31:53.884919+0000","last_became_active":"2024-03-28T23:31:39.380761+0000","last_became_peered":"2024-03-28T23:31:39.380761+0000","last_unstale":"2024-03-28T23:31:53.884919+0000","last_undegraded":"2024-03-28T23:31:53.884919+0000","last_fullsized":"2024-03-28T23:31:53.884919+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:29.240985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878283+0000","last_change":"2024-03-28T23:31:53.878283+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878283+0000","last_undegraded":"2024-03-28T23:31:53.878283+0000","last_fullsized":"2024-03-28T23:31:53.878283+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.893113+0000","last_change":"2024-03-28T23:31:53.889906+0000","last_active":"2024-03-28T23:31:53.893113+0000","last_peered":"2024-03-28T23:31:53.893113+0000","last_clean":"2024-03-28T23:31:53.893113+0000","last_became_active":"2024-03-28T23:31:53.889589+0000","last_became_peered":"2024-03-28T23:31:53.889589+0000","last_unstale":"2024-03-28T23:31:53.893113+0000","last_undegraded":"2024-03-28T23:31:53.893113+0000","last_fullsized":"2024-03-28T23:31:53.893113+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:17:43.674083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875914+0000","last_change":"2024-03-28T23:31:40.400388+0000","last_active":"2024-03-28T23:31:53.875914+0000","last_peered":"2024-03-28T23:31:53.875914+0000","last_clean":"2024-03-28T23:31:53.875914+0000","last_became_active":"2024-03-28T23:31:40.400133+0000","last_became_peered":"2024-03-28T23:31:40.400133+0000","last_unstale":"2024-03-28T23:31:53.875914+0000","last_undegraded":"2024-03-28T23:31:53.875914+0000","last_fullsized":"2024-03-28T23:31:53.875914+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:03:56.252539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"50'3","reported_seq":21,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876731+0000","last_change":"2024-03-28T23:31:39.386478+0000","last_active":"2024-03-28T23:31:53.876731+0000","last_peered":"2024-03-28T23:31:53.876731+0000","last_clean":"2024-03-28T23:31:53.876731+0000","last_became_active":"2024-03-28T23:31:39.385868+0000","last_became_peered":"2024-03-28T23:31:39.385868+0000","last_unstale":"2024-03-28T23:31:53.876731+0000","last_undegraded":"2024-03-28T23:31:53.876731+0000","last_fullsized":"2024-03-28T23:31:53.876731+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:06.733879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872413+0000","last_change":"2024-03-28T23:31:53.872413+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872413+0000","last_undegraded":"2024-03-28T23:31:53.872413+0000","last_fullsized":"2024-03-28T23:31:53.872413+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.893021+0000","last_change":"2024-03-28T23:31:53.889887+0000","last_active":"2024-03-28T23:31:53.893021+0000","last_peered":"2024-03-28T23:31:53.893021+0000","last_clean":"2024-03-28T23:31:53.893021+0000","last_became_active":"2024-03-28T23:31:53.889635+0000","last_became_peered":"2024-03-28T23:31:53.889635+0000","last_unstale":"2024-03-28T23:31:53.893021+0000","last_undegraded":"2024-03-28T23:31:53.893021+0000","last_fullsized":"2024-03-28T23:31:53.893021+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:19:23.382549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.884161+0000","last_change":"2024-03-28T23:31:40.387795+0000","last_active":"2024-03-28T23:31:53.884161+0000","last_peered":"2024-03-28T23:31:53.884161+0000","last_clean":"2024-03-28T23:31:53.884161+0000","last_became_active":"2024-03-28T23:31:40.387529+0000","last_became_peered":"2024-03-28T23:31:40.387529+0000","last_unstale":"2024-03-28T23:31:53.884161+0000","last_undegraded":"2024-03-28T23:31:53.884161+0000","last_fullsized":"2024-03-28T23:31:53.884161+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:51.864986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"50'6","reported_seq":24,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879943+0000","last_change":"2024-03-28T23:31:39.393411+0000","last_active":"2024-03-28T23:31:53.879943+0000","last_peered":"2024-03-28T23:31:53.879943+0000","last_clean":"2024-03-28T23:31:53.879943+0000","last_became_active":"2024-03-28T23:31:39.393166+0000","last_became_peered":"2024-03-28T23:31:39.393166+0000","last_unstale":"2024-03-28T23:31:53.879943+0000","last_undegraded":"2024-03-28T23:31:53.879943+0000","last_fullsized":"2024-03-28T23:31:53.879943+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:13.510273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874577+0000","last_change":"2024-03-28T23:31:53.874577+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874577+0000","last_undegraded":"2024-03-28T23:31:53.874577+0000","last_fullsized":"2024-03-28T23:31:53.874577+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.890522+0000","last_change":"2024-03-28T23:31:53.889399+0000","last_active":"2024-03-28T23:31:53.890522+0000","last_peered":"2024-03-28T23:31:53.890522+0000","last_clean":"2024-03-28T23:31:53.890522+0000","last_became_active":"2024-03-28T23:31:53.889110+0000","last_became_peered":"2024-03-28T23:31:53.889110+0000","last_unstale":"2024-03-28T23:31:53.890522+0000","last_undegraded":"2024-03-28T23:31:53.890522+0000","last_fullsized":"2024-03-28T23:31:53.890522+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:35:48.231940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874249+0000","last_change":"2024-03-28T23:31:40.390497+0000","last_active":"2024-03-28T23:31:53.874249+0000","last_peered":"2024-03-28T23:31:53.874249+0000","last_clean":"2024-03-28T23:31:53.874249+0000","last_became_active":"2024-03-28T23:31:40.390167+0000","last_became_peered":"2024-03-28T23:31:40.390167+0000","last_unstale":"2024-03-28T23:31:53.874249+0000","last_undegraded":"2024-03-28T23:31:53.874249+0000","last_fullsized":"2024-03-28T23:31:53.874249+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:54.431044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"50'2","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875733+0000","last_change":"2024-03-28T23:31:39.380369+0000","last_active":"2024-03-28T23:31:53.875733+0000","last_peered":"2024-03-28T23:31:53.875733+0000","last_clean":"2024-03-28T23:31:53.875733+0000","last_became_active":"2024-03-28T23:31:39.380092+0000","last_became_peered":"2024-03-28T23:31:39.380092+0000","last_unstale":"2024-03-28T23:31:53.875733+0000","last_undegraded":"2024-03-28T23:31:53.875733+0000","last_fullsized":"2024-03-28T23:31:53.875733+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:00.366278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871676+0000","last_change":"2024-03-28T23:31:53.871676+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871676+0000","last_undegraded":"2024-03-28T23:31:53.871676+0000","last_fullsized":"2024-03-28T23:31:53.871676+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.885629+0000","last_change":"2024-03-28T23:31:53.883362+0000","last_active":"2024-03-28T23:31:53.885629+0000","last_peered":"2024-03-28T23:31:53.885629+0000","last_clean":"2024-03-28T23:31:53.885629+0000","last_became_active":"2024-03-28T23:31:53.883123+0000","last_became_peered":"2024-03-28T23:31:53.883123+0000","last_unstale":"2024-03-28T23:31:53.885629+0000","last_undegraded":"2024-03-28T23:31:53.885629+0000","last_fullsized":"2024-03-28T23:31:53.885629+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:16.175508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876210+0000","last_change":"2024-03-28T23:31:40.390579+0000","last_active":"2024-03-28T23:31:53.876210+0000","last_peered":"2024-03-28T23:31:53.876210+0000","last_clean":"2024-03-28T23:31:53.876210+0000","last_became_active":"2024-03-28T23:31:40.390248+0000","last_became_peered":"2024-03-28T23:31:40.390248+0000","last_unstale":"2024-03-28T23:31:53.876210+0000","last_undegraded":"2024-03-28T23:31:53.876210+0000","last_fullsized":"2024-03-28T23:31:53.876210+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:47.861300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875729+0000","last_change":"2024-03-28T23:31:39.389592+0000","last_active":"2024-03-28T23:31:53.875729+0000","last_peered":"2024-03-28T23:31:53.875729+0000","last_clean":"2024-03-28T23:31:53.875729+0000","last_became_active":"2024-03-28T23:31:39.389302+0000","last_became_peered":"2024-03-28T23:31:39.389302+0000","last_unstale":"2024-03-28T23:31:53.875729+0000","last_undegraded":"2024-03-28T23:31:53.875729+0000","last_fullsized":"2024-03-28T23:31:53.875729+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:59:51.501145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872186+0000","last_change":"2024-03-28T23:31:53.872186+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872186+0000","last_undegraded":"2024-03-28T23:31:53.872186+0000","last_fullsized":"2024-03-28T23:31:53.872186+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.884324+0000","last_change":"2024-03-28T23:31:53.881795+0000","last_active":"2024-03-28T23:31:53.884324+0000","last_peered":"2024-03-28T23:31:53.884324+0000","last_clean":"2024-03-28T23:31:53.884324+0000","last_became_active":"2024-03-28T23:31:53.881552+0000","last_became_peered":"2024-03-28T23:31:53.881552+0000","last_unstale":"2024-03-28T23:31:53.884324+0000","last_undegraded":"2024-03-28T23:31:53.884324+0000","last_fullsized":"2024-03-28T23:31:53.884324+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:35.224564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879015+0000","last_change":"2024-03-28T23:31:40.387085+0000","last_active":"2024-03-28T23:31:53.879015+0000","last_peered":"2024-03-28T23:31:53.879015+0000","last_clean":"2024-03-28T23:31:53.879015+0000","last_became_active":"2024-03-28T23:31:40.386695+0000","last_became_peered":"2024-03-28T23:31:40.386695+0000","last_unstale":"2024-03-28T23:31:53.879015+0000","last_undegraded":"2024-03-28T23:31:53.879015+0000","last_fullsized":"2024-03-28T23:31:53.879015+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:40:57.541676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.873632+0000","last_change":"2024-03-28T23:31:39.384653+0000","last_active":"2024-03-28T23:31:53.873632+0000","last_peered":"2024-03-28T23:31:53.873632+0000","last_clean":"2024-03-28T23:31:53.873632+0000","last_became_active":"2024-03-28T23:31:39.383495+0000","last_became_peered":"2024-03-28T23:31:39.383495+0000","last_unstale":"2024-03-28T23:31:53.873632+0000","last_undegraded":"2024-03-28T23:31:53.873632+0000","last_fullsized":"2024-03-28T23:31:53.873632+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:50.860190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878941+0000","last_change":"2024-03-28T23:31:53.878941+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878941+0000","last_undegraded":"2024-03-28T23:31:53.878941+0000","last_fullsized":"2024-03-28T23:31:53.878941+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.886195+0000","last_change":"2024-03-28T23:31:53.884651+0000","last_active":"2024-03-28T23:31:53.886195+0000","last_peered":"2024-03-28T23:31:53.886195+0000","last_clean":"2024-03-28T23:31:53.886195+0000","last_became_active":"2024-03-28T23:31:53.884410+0000","last_became_peered":"2024-03-28T23:31:53.884410+0000","last_unstale":"2024-03-28T23:31:53.886195+0000","last_undegraded":"2024-03-28T23:31:53.886195+0000","last_fullsized":"2024-03-28T23:31:53.886195+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:31:41.553370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876561+0000","last_change":"2024-03-28T23:31:40.403701+0000","last_active":"2024-03-28T23:31:53.876561+0000","last_peered":"2024-03-28T23:31:53.876561+0000","last_clean":"2024-03-28T23:31:53.876561+0000","last_became_active":"2024-03-28T23:31:40.403222+0000","last_became_peered":"2024-03-28T23:31:40.403222+0000","last_unstale":"2024-03-28T23:31:53.876561+0000","last_undegraded":"2024-03-28T23:31:53.876561+0000","last_fullsized":"2024-03-28T23:31:53.876561+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:40:09.867914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"50'8","reported_seq":26,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.880291+0000","last_change":"2024-03-28T23:31:39.383726+0000","last_active":"2024-03-28T23:31:53.880291+0000","last_peered":"2024-03-28T23:31:53.880291+0000","last_clean":"2024-03-28T23:31:53.880291+0000","last_became_active":"2024-03-28T23:31:39.383292+0000","last_became_peered":"2024-03-28T23:31:39.383292+0000","last_unstale":"2024-03-28T23:31:53.880291+0000","last_undegraded":"2024-03-28T23:31:53.880291+0000","last_fullsized":"2024-03-28T23:31:53.880291+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:25:38.113638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872289+0000","last_change":"2024-03-28T23:31:53.872289+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872289+0000","last_undegraded":"2024-03-28T23:31:53.872289+0000","last_fullsized":"2024-03-28T23:31:53.872289+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.896055+0000","last_change":"2024-03-28T23:31:53.892246+0000","last_active":"2024-03-28T23:31:53.896055+0000","last_peered":"2024-03-28T23:31:53.896055+0000","last_clean":"2024-03-28T23:31:53.896055+0000","last_became_active":"2024-03-28T23:31:53.891974+0000","last_became_peered":"2024-03-28T23:31:53.891974+0000","last_unstale":"2024-03-28T23:31:53.896055+0000","last_undegraded":"2024-03-28T23:31:53.896055+0000","last_fullsized":"2024-03-28T23:31:53.896055+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:10:22.963954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.880205+0000","last_change":"2024-03-28T23:31:40.391219+0000","last_active":"2024-03-28T23:31:53.880205+0000","last_peered":"2024-03-28T23:31:53.880205+0000","last_clean":"2024-03-28T23:31:53.880205+0000","last_became_active":"2024-03-28T23:31:40.389552+0000","last_became_peered":"2024-03-28T23:31:40.389552+0000","last_unstale":"2024-03-28T23:31:53.880205+0000","last_undegraded":"2024-03-28T23:31:53.880205+0000","last_fullsized":"2024-03-28T23:31:53.880205+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:42:29.351941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"50'1","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876797+0000","last_change":"2024-03-28T23:31:39.390253+0000","last_active":"2024-03-28T23:31:53.876797+0000","last_peered":"2024-03-28T23:31:53.876797+0000","last_clean":"2024-03-28T23:31:53.876797+0000","last_became_active":"2024-03-28T23:31:39.389980+0000","last_became_peered":"2024-03-28T23:31:39.389980+0000","last_unstale":"2024-03-28T23:31:53.876797+0000","last_undegraded":"2024-03-28T23:31:53.876797+0000","last_fullsized":"2024-03-28T23:31:53.876797+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:24.520296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.877420+0000","last_change":"2024-03-28T23:31:53.877420+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.877420+0000","last_undegraded":"2024-03-28T23:31:53.877420+0000","last_fullsized":"2024-03-28T23:31:53.877420+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.884332+0000","last_change":"2024-03-28T23:31:53.880659+0000","last_active":"2024-03-28T23:31:53.884332+0000","last_peered":"2024-03-28T23:31:53.884332+0000","last_clean":"2024-03-28T23:31:53.884332+0000","last_became_active":"2024-03-28T23:31:53.880265+0000","last_became_peered":"2024-03-28T23:31:53.880265+0000","last_unstale":"2024-03-28T23:31:53.884332+0000","last_undegraded":"2024-03-28T23:31:53.884332+0000","last_fullsized":"2024-03-28T23:31:53.884332+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:04:09.751864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869503+0000","last_change":"2024-03-28T23:31:40.391898+0000","last_active":"2024-03-28T23:31:53.869503+0000","last_peered":"2024-03-28T23:31:53.869503+0000","last_clean":"2024-03-28T23:31:53.869503+0000","last_became_active":"2024-03-28T23:31:40.391622+0000","last_became_peered":"2024-03-28T23:31:40.391622+0000","last_unstale":"2024-03-28T23:31:53.869503+0000","last_undegraded":"2024-03-28T23:31:53.869503+0000","last_fullsized":"2024-03-28T23:31:53.869503+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:38:22.055287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"50'2","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.883203+0000","last_change":"2024-03-28T23:31:39.385339+0000","last_active":"2024-03-28T23:31:53.883203+0000","last_peered":"2024-03-28T23:31:53.883203+0000","last_clean":"2024-03-28T23:31:53.883203+0000","last_became_active":"2024-03-28T23:31:39.385075+0000","last_became_peered":"2024-03-28T23:31:39.385075+0000","last_unstale":"2024-03-28T23:31:53.883203+0000","last_undegraded":"2024-03-28T23:31:53.883203+0000","last_fullsized":"2024-03-28T23:31:53.883203+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:58.098710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.873793+0000","last_change":"2024-03-28T23:31:53.873793+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.873793+0000","last_undegraded":"2024-03-28T23:31:53.873793+0000","last_fullsized":"2024-03-28T23:31:53.873793+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891950+0000","last_change":"2024-03-28T23:31:53.890901+0000","last_active":"2024-03-28T23:31:53.891950+0000","last_peered":"2024-03-28T23:31:53.891950+0000","last_clean":"2024-03-28T23:31:53.891950+0000","last_became_active":"2024-03-28T23:31:53.890643+0000","last_became_peered":"2024-03-28T23:31:53.890643+0000","last_unstale":"2024-03-28T23:31:53.891950+0000","last_undegraded":"2024-03-28T23:31:53.891950+0000","last_fullsized":"2024-03-28T23:31:53.891950+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:07:27.190116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.883208+0000","last_change":"2024-03-28T23:31:40.399291+0000","last_active":"2024-03-28T23:31:53.883208+0000","last_peered":"2024-03-28T23:31:53.883208+0000","last_clean":"2024-03-28T23:31:53.883208+0000","last_became_active":"2024-03-28T23:31:40.399005+0000","last_became_peered":"2024-03-28T23:31:40.399005+0000","last_unstale":"2024-03-28T23:31:53.883208+0000","last_undegraded":"2024-03-28T23:31:53.883208+0000","last_fullsized":"2024-03-28T23:31:53.883208+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:01.371074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"50'1","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876296+0000","last_change":"2024-03-28T23:31:39.389465+0000","last_active":"2024-03-28T23:31:53.876296+0000","last_peered":"2024-03-28T23:31:53.876296+0000","last_clean":"2024-03-28T23:31:53.876296+0000","last_became_active":"2024-03-28T23:31:39.389005+0000","last_became_peered":"2024-03-28T23:31:39.389005+0000","last_unstale":"2024-03-28T23:31:53.876296+0000","last_undegraded":"2024-03-28T23:31:53.876296+0000","last_fullsized":"2024-03-28T23:31:53.876296+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:41.430571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.879255+0000","last_change":"2024-03-28T23:31:53.879255+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.879255+0000","last_undegraded":"2024-03-28T23:31:53.879255+0000","last_fullsized":"2024-03-28T23:31:53.879255+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.890716+0000","last_change":"2024-03-28T23:31:53.888704+0000","last_active":"2024-03-28T23:31:53.890716+0000","last_peered":"2024-03-28T23:31:53.890716+0000","last_clean":"2024-03-28T23:31:53.890716+0000","last_became_active":"2024-03-28T23:31:53.888461+0000","last_became_peered":"2024-03-28T23:31:53.888461+0000","last_unstale":"2024-03-28T23:31:53.890716+0000","last_undegraded":"2024-03-28T23:31:53.890716+0000","last_fullsized":"2024-03-28T23:31:53.890716+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:08.894103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869795+0000","last_change":"2024-03-28T23:31:40.388141+0000","last_active":"2024-03-28T23:31:53.869795+0000","last_peered":"2024-03-28T23:31:53.869795+0000","last_clean":"2024-03-28T23:31:53.869795+0000","last_became_active":"2024-03-28T23:31:40.387852+0000","last_became_peered":"2024-03-28T23:31:40.387852+0000","last_unstale":"2024-03-28T23:31:53.869795+0000","last_undegraded":"2024-03-28T23:31:53.869795+0000","last_fullsized":"2024-03-28T23:31:53.869795+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:22.230986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.877311+0000","last_change":"2024-03-28T23:31:39.390465+0000","last_active":"2024-03-28T23:31:53.877311+0000","last_peered":"2024-03-28T23:31:53.877311+0000","last_clean":"2024-03-28T23:31:53.877311+0000","last_became_active":"2024-03-28T23:31:39.390203+0000","last_became_peered":"2024-03-28T23:31:39.390203+0000","last_unstale":"2024-03-28T23:31:53.877311+0000","last_undegraded":"2024-03-28T23:31:53.877311+0000","last_fullsized":"2024-03-28T23:31:53.877311+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:50:50.496294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.876579+0000","last_change":"2024-03-28T23:31:53.876579+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.876579+0000","last_undegraded":"2024-03-28T23:31:53.876579+0000","last_fullsized":"2024-03-28T23:31:53.876579+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.890502+0000","last_change":"2024-03-28T23:31:53.888267+0000","last_active":"2024-03-28T23:31:53.890502+0000","last_peered":"2024-03-28T23:31:53.890502+0000","last_clean":"2024-03-28T23:31:53.890502+0000","last_became_active":"2024-03-28T23:31:53.887993+0000","last_became_peered":"2024-03-28T23:31:53.887993+0000","last_unstale":"2024-03-28T23:31:53.890502+0000","last_undegraded":"2024-03-28T23:31:53.890502+0000","last_fullsized":"2024-03-28T23:31:53.890502+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:21:27.713981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874357+0000","last_change":"2024-03-28T23:31:40.387376+0000","last_active":"2024-03-28T23:31:53.874357+0000","last_peered":"2024-03-28T23:31:53.874357+0000","last_clean":"2024-03-28T23:31:53.874357+0000","last_became_active":"2024-03-28T23:31:40.387080+0000","last_became_peered":"2024-03-28T23:31:40.387080+0000","last_unstale":"2024-03-28T23:31:53.874357+0000","last_undegraded":"2024-03-28T23:31:53.874357+0000","last_fullsized":"2024-03-28T23:31:53.874357+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:43.111058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"50'2","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869894+0000","last_change":"2024-03-28T23:31:39.388867+0000","last_active":"2024-03-28T23:31:53.869894+0000","last_peered":"2024-03-28T23:31:53.869894+0000","last_clean":"2024-03-28T23:31:53.869894+0000","last_became_active":"2024-03-28T23:31:39.388593+0000","last_became_peered":"2024-03-28T23:31:39.388593+0000","last_unstale":"2024-03-28T23:31:53.869894+0000","last_undegraded":"2024-03-28T23:31:53.869894+0000","last_fullsized":"2024-03-28T23:31:53.869894+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:21:00.890266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872546+0000","last_change":"2024-03-28T23:31:53.872546+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872546+0000","last_undegraded":"2024-03-28T23:31:53.872546+0000","last_fullsized":"2024-03-28T23:31:53.872546+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.888466+0000","last_change":"2024-03-28T23:31:53.886433+0000","last_active":"2024-03-28T23:31:53.888466+0000","last_peered":"2024-03-28T23:31:53.888466+0000","last_clean":"2024-03-28T23:31:53.888466+0000","last_became_active":"2024-03-28T23:31:53.886218+0000","last_became_peered":"2024-03-28T23:31:53.886218+0000","last_unstale":"2024-03-28T23:31:53.888466+0000","last_undegraded":"2024-03-28T23:31:53.888466+0000","last_fullsized":"2024-03-28T23:31:53.888466+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:32.282446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876171+0000","last_change":"2024-03-28T23:31:40.393257+0000","last_active":"2024-03-28T23:31:53.876171+0000","last_peered":"2024-03-28T23:31:53.876171+0000","last_clean":"2024-03-28T23:31:53.876171+0000","last_became_active":"2024-03-28T23:31:40.391684+0000","last_became_peered":"2024-03-28T23:31:40.391684+0000","last_unstale":"2024-03-28T23:31:53.876171+0000","last_undegraded":"2024-03-28T23:31:53.876171+0000","last_fullsized":"2024-03-28T23:31:53.876171+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:01.371074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.881223+0000","last_change":"2024-03-28T23:31:39.388198+0000","last_active":"2024-03-28T23:31:53.881223+0000","last_peered":"2024-03-28T23:31:53.881223+0000","last_clean":"2024-03-28T23:31:53.881223+0000","last_became_active":"2024-03-28T23:31:39.387727+0000","last_became_peered":"2024-03-28T23:31:39.387727+0000","last_unstale":"2024-03-28T23:31:53.881223+0000","last_undegraded":"2024-03-28T23:31:53.881223+0000","last_fullsized":"2024-03-28T23:31:53.881223+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:49.868091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874240+0000","last_change":"2024-03-28T23:31:53.874240+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874240+0000","last_undegraded":"2024-03-28T23:31:53.874240+0000","last_fullsized":"2024-03-28T23:31:53.874240+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.886125+0000","last_change":"2024-03-28T23:31:53.884698+0000","last_active":"2024-03-28T23:31:53.886125+0000","last_peered":"2024-03-28T23:31:53.886125+0000","last_clean":"2024-03-28T23:31:53.886125+0000","last_became_active":"2024-03-28T23:31:53.884413+0000","last_became_peered":"2024-03-28T23:31:53.884413+0000","last_unstale":"2024-03-28T23:31:53.886125+0000","last_undegraded":"2024-03-28T23:31:53.886125+0000","last_fullsized":"2024-03-28T23:31:53.886125+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:32.282446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876556+0000","last_change":"2024-03-28T23:31:40.394047+0000","last_active":"2024-03-28T23:31:53.876556+0000","last_peered":"2024-03-28T23:31:53.876556+0000","last_clean":"2024-03-28T23:31:53.876556+0000","last_became_active":"2024-03-28T23:31:40.393772+0000","last_became_peered":"2024-03-28T23:31:40.393772+0000","last_unstale":"2024-03-28T23:31:53.876556+0000","last_undegraded":"2024-03-28T23:31:53.876556+0000","last_fullsized":"2024-03-28T23:31:53.876556+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:07:13.690791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875624+0000","last_change":"2024-03-28T23:31:39.388829+0000","last_active":"2024-03-28T23:31:53.875624+0000","last_peered":"2024-03-28T23:31:53.875624+0000","last_clean":"2024-03-28T23:31:53.875624+0000","last_became_active":"2024-03-28T23:31:39.388521+0000","last_became_peered":"2024-03-28T23:31:39.388521+0000","last_unstale":"2024-03-28T23:31:53.875624+0000","last_undegraded":"2024-03-28T23:31:53.875624+0000","last_fullsized":"2024-03-28T23:31:53.875624+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:23:52.148574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.877392+0000","last_change":"2024-03-28T23:31:53.877392+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.877392+0000","last_undegraded":"2024-03-28T23:31:53.877392+0000","last_fullsized":"2024-03-28T23:31:53.877392+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.887423+0000","last_change":"2024-03-28T23:31:53.886155+0000","last_active":"2024-03-28T23:31:53.887423+0000","last_peered":"2024-03-28T23:31:53.887423+0000","last_clean":"2024-03-28T23:31:53.887423+0000","last_became_active":"2024-03-28T23:31:53.885843+0000","last_became_peered":"2024-03-28T23:31:53.885843+0000","last_unstale":"2024-03-28T23:31:53.887423+0000","last_undegraded":"2024-03-28T23:31:53.887423+0000","last_fullsized":"2024-03-28T23:31:53.887423+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:38:09.370282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.870442+0000","last_change":"2024-03-28T23:31:40.384709+0000","last_active":"2024-03-28T23:31:53.870442+0000","last_peered":"2024-03-28T23:31:53.870442+0000","last_clean":"2024-03-28T23:31:53.870442+0000","last_became_active":"2024-03-28T23:31:40.384412+0000","last_became_peered":"2024-03-28T23:31:40.384412+0000","last_unstale":"2024-03-28T23:31:53.870442+0000","last_undegraded":"2024-03-28T23:31:53.870442+0000","last_fullsized":"2024-03-28T23:31:53.870442+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:25:39.118434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879185+0000","last_change":"2024-03-28T23:31:39.385937+0000","last_active":"2024-03-28T23:31:53.879185+0000","last_peered":"2024-03-28T23:31:53.879185+0000","last_clean":"2024-03-28T23:31:53.879185+0000","last_became_active":"2024-03-28T23:31:39.385675+0000","last_became_peered":"2024-03-28T23:31:39.385675+0000","last_unstale":"2024-03-28T23:31:53.879185+0000","last_undegraded":"2024-03-28T23:31:53.879185+0000","last_fullsized":"2024-03-28T23:31:53.879185+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:35.204851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874303+0000","last_change":"2024-03-28T23:31:53.874303+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874303+0000","last_undegraded":"2024-03-28T23:31:53.874303+0000","last_fullsized":"2024-03-28T23:31:53.874303+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.886168+0000","last_change":"2024-03-28T23:31:53.884950+0000","last_active":"2024-03-28T23:31:53.886168+0000","last_peered":"2024-03-28T23:31:53.886168+0000","last_clean":"2024-03-28T23:31:53.886168+0000","last_became_active":"2024-03-28T23:31:53.884686+0000","last_became_peered":"2024-03-28T23:31:53.884686+0000","last_unstale":"2024-03-28T23:31:53.886168+0000","last_undegraded":"2024-03-28T23:31:53.886168+0000","last_fullsized":"2024-03-28T23:31:53.886168+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:48.477986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874779+0000","last_change":"2024-03-28T23:31:40.387783+0000","last_active":"2024-03-28T23:31:53.874779+0000","last_peered":"2024-03-28T23:31:53.874779+0000","last_clean":"2024-03-28T23:31:53.874779+0000","last_became_active":"2024-03-28T23:31:40.387487+0000","last_became_peered":"2024-03-28T23:31:40.387487+0000","last_unstale":"2024-03-28T23:31:53.874779+0000","last_undegraded":"2024-03-28T23:31:53.874779+0000","last_fullsized":"2024-03-28T23:31:53.874779+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:22.230986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"50'2","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.880346+0000","last_change":"2024-03-28T23:31:39.391741+0000","last_active":"2024-03-28T23:31:53.880346+0000","last_peered":"2024-03-28T23:31:53.880346+0000","last_clean":"2024-03-28T23:31:53.880346+0000","last_became_active":"2024-03-28T23:31:39.391520+0000","last_became_peered":"2024-03-28T23:31:39.391520+0000","last_unstale":"2024-03-28T23:31:53.880346+0000","last_undegraded":"2024-03-28T23:31:53.880346+0000","last_fullsized":"2024-03-28T23:31:53.880346+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:16:42.869555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874197+0000","last_change":"2024-03-28T23:31:53.874197+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874197+0000","last_undegraded":"2024-03-28T23:31:53.874197+0000","last_fullsized":"2024-03-28T23:31:53.874197+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.890450+0000","last_change":"2024-03-28T23:31:53.887757+0000","last_active":"2024-03-28T23:31:53.890450+0000","last_peered":"2024-03-28T23:31:53.890450+0000","last_clean":"2024-03-28T23:31:53.890450+0000","last_became_active":"2024-03-28T23:31:53.887470+0000","last_became_peered":"2024-03-28T23:31:53.887470+0000","last_unstale":"2024-03-28T23:31:53.890450+0000","last_undegraded":"2024-03-28T23:31:53.890450+0000","last_fullsized":"2024-03-28T23:31:53.890450+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:40:57.385677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869629+0000","last_change":"2024-03-28T23:31:40.387330+0000","last_active":"2024-03-28T23:31:53.869629+0000","last_peered":"2024-03-28T23:31:53.869629+0000","last_clean":"2024-03-28T23:31:53.869629+0000","last_became_active":"2024-03-28T23:31:40.387076+0000","last_became_peered":"2024-03-28T23:31:40.387076+0000","last_unstale":"2024-03-28T23:31:53.869629+0000","last_undegraded":"2024-03-28T23:31:53.869629+0000","last_fullsized":"2024-03-28T23:31:53.869629+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:25.525091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.877434+0000","last_change":"2024-03-28T23:31:39.386686+0000","last_active":"2024-03-28T23:31:53.877434+0000","last_peered":"2024-03-28T23:31:53.877434+0000","last_clean":"2024-03-28T23:31:53.877434+0000","last_became_active":"2024-03-28T23:31:39.386208+0000","last_became_peered":"2024-03-28T23:31:39.386208+0000","last_unstale":"2024-03-28T23:31:53.877434+0000","last_undegraded":"2024-03-28T23:31:53.877434+0000","last_fullsized":"2024-03-28T23:31:53.877434+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:22.210570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.876269+0000","last_change":"2024-03-28T23:31:53.876269+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.876269+0000","last_undegraded":"2024-03-28T23:31:53.876269+0000","last_fullsized":"2024-03-28T23:31:53.876269+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891725+0000","last_change":"2024-03-28T23:31:53.887766+0000","last_active":"2024-03-28T23:31:53.891725+0000","last_peered":"2024-03-28T23:31:53.891725+0000","last_clean":"2024-03-28T23:31:53.891725+0000","last_became_active":"2024-03-28T23:31:53.887471+0000","last_became_peered":"2024-03-28T23:31:53.887471+0000","last_unstale":"2024-03-28T23:31:53.891725+0000","last_undegraded":"2024-03-28T23:31:53.891725+0000","last_fullsized":"2024-03-28T23:31:53.891725+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:05:01.725546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.881956+0000","last_change":"2024-03-28T23:31:40.386143+0000","last_active":"2024-03-28T23:31:53.881956+0000","last_peered":"2024-03-28T23:31:53.881956+0000","last_clean":"2024-03-28T23:31:53.881956+0000","last_became_active":"2024-03-28T23:31:40.385606+0000","last_became_peered":"2024-03-28T23:31:40.385606+0000","last_unstale":"2024-03-28T23:31:53.881956+0000","last_undegraded":"2024-03-28T23:31:53.881956+0000","last_fullsized":"2024-03-28T23:31:53.881956+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:40:09.867914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876026+0000","last_change":"2024-03-28T23:31:39.380624+0000","last_active":"2024-03-28T23:31:53.876026+0000","last_peered":"2024-03-28T23:31:53.876026+0000","last_clean":"2024-03-28T23:31:53.876026+0000","last_became_active":"2024-03-28T23:31:39.380340+0000","last_became_peered":"2024-03-28T23:31:39.380340+0000","last_unstale":"2024-03-28T23:31:53.876026+0000","last_undegraded":"2024-03-28T23:31:53.876026+0000","last_fullsized":"2024-03-28T23:31:53.876026+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:22.210570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.879694+0000","last_change":"2024-03-28T23:31:53.879694+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.879694+0000","last_undegraded":"2024-03-28T23:31:53.879694+0000","last_fullsized":"2024-03-28T23:31:53.879694+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.889286+0000","last_change":"2024-03-28T23:31:53.886660+0000","last_active":"2024-03-28T23:31:53.889286+0000","last_peered":"2024-03-28T23:31:53.889286+0000","last_clean":"2024-03-28T23:31:53.889286+0000","last_became_active":"2024-03-28T23:31:53.886355+0000","last_became_peered":"2024-03-28T23:31:53.886355+0000","last_unstale":"2024-03-28T23:31:53.889286+0000","last_undegraded":"2024-03-28T23:31:53.889286+0000","last_fullsized":"2024-03-28T23:31:53.889286+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:38:09.370282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875942+0000","last_change":"2024-03-28T23:31:40.404359+0000","last_active":"2024-03-28T23:31:53.875942+0000","last_peered":"2024-03-28T23:31:53.875942+0000","last_clean":"2024-03-28T23:31:53.875942+0000","last_became_active":"2024-03-28T23:31:40.404078+0000","last_became_peered":"2024-03-28T23:31:40.404078+0000","last_unstale":"2024-03-28T23:31:53.875942+0000","last_undegraded":"2024-03-28T23:31:53.875942+0000","last_fullsized":"2024-03-28T23:31:53.875942+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:18.654331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.882034+0000","last_change":"2024-03-28T23:31:39.393283+0000","last_active":"2024-03-28T23:31:53.882034+0000","last_peered":"2024-03-28T23:31:53.882034+0000","last_clean":"2024-03-28T23:31:53.882034+0000","last_became_active":"2024-03-28T23:31:39.393035+0000","last_became_peered":"2024-03-28T23:31:39.393035+0000","last_unstale":"2024-03-28T23:31:53.882034+0000","last_undegraded":"2024-03-28T23:31:53.882034+0000","last_fullsized":"2024-03-28T23:31:53.882034+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:13.139644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872993+0000","last_change":"2024-03-28T23:31:53.872993+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872993+0000","last_undegraded":"2024-03-28T23:31:53.872993+0000","last_fullsized":"2024-03-28T23:31:53.872993+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.890751+0000","last_change":"2024-03-28T23:31:53.888742+0000","last_active":"2024-03-28T23:31:53.890751+0000","last_peered":"2024-03-28T23:31:53.890751+0000","last_clean":"2024-03-28T23:31:53.890751+0000","last_became_active":"2024-03-28T23:31:53.888515+0000","last_became_peered":"2024-03-28T23:31:53.888515+0000","last_unstale":"2024-03-28T23:31:53.890751+0000","last_undegraded":"2024-03-28T23:31:53.890751+0000","last_fullsized":"2024-03-28T23:31:53.890751+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:17:10.580507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875207+0000","last_change":"2024-03-28T23:31:40.392819+0000","last_active":"2024-03-28T23:31:53.875207+0000","last_peered":"2024-03-28T23:31:53.875207+0000","last_clean":"2024-03-28T23:31:53.875207+0000","last_became_active":"2024-03-28T23:31:40.392501+0000","last_became_peered":"2024-03-28T23:31:40.392501+0000","last_unstale":"2024-03-28T23:31:53.875207+0000","last_undegraded":"2024-03-28T23:31:53.875207+0000","last_fullsized":"2024-03-28T23:31:53.875207+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:14.144440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.881307+0000","last_change":"2024-03-28T23:31:39.387993+0000","last_active":"2024-03-28T23:31:53.881307+0000","last_peered":"2024-03-28T23:31:53.881307+0000","last_clean":"2024-03-28T23:31:53.881307+0000","last_became_active":"2024-03-28T23:31:39.387393+0000","last_became_peered":"2024-03-28T23:31:39.387393+0000","last_unstale":"2024-03-28T23:31:53.881307+0000","last_undegraded":"2024-03-28T23:31:53.881307+0000","last_fullsized":"2024-03-28T23:31:53.881307+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:21.226190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871482+0000","last_change":"2024-03-28T23:31:53.871482+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871482+0000","last_undegraded":"2024-03-28T23:31:53.871482+0000","last_fullsized":"2024-03-28T23:31:53.871482+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.892224+0000","last_change":"2024-03-28T23:31:53.890146+0000","last_active":"2024-03-28T23:31:53.892224+0000","last_peered":"2024-03-28T23:31:53.892224+0000","last_clean":"2024-03-28T23:31:53.892224+0000","last_became_active":"2024-03-28T23:31:53.888840+0000","last_became_peered":"2024-03-28T23:31:53.888840+0000","last_unstale":"2024-03-28T23:31:53.892224+0000","last_undegraded":"2024-03-28T23:31:53.892224+0000","last_fullsized":"2024-03-28T23:31:53.892224+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:01.360626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876467+0000","last_change":"2024-03-28T23:31:40.405766+0000","last_active":"2024-03-28T23:31:53.876467+0000","last_peered":"2024-03-28T23:31:53.876467+0000","last_clean":"2024-03-28T23:31:53.876467+0000","last_became_active":"2024-03-28T23:31:40.405517+0000","last_became_peered":"2024-03-28T23:31:40.405517+0000","last_unstale":"2024-03-28T23:31:53.876467+0000","last_undegraded":"2024-03-28T23:31:53.876467+0000","last_fullsized":"2024-03-28T23:31:53.876467+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:18:16.851435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.883245+0000","last_change":"2024-03-28T23:31:39.383748+0000","last_active":"2024-03-28T23:31:53.883245+0000","last_peered":"2024-03-28T23:31:53.883245+0000","last_clean":"2024-03-28T23:31:53.883245+0000","last_became_active":"2024-03-28T23:31:39.383070+0000","last_became_peered":"2024-03-28T23:31:39.383070+0000","last_unstale":"2024-03-28T23:31:53.883245+0000","last_undegraded":"2024-03-28T23:31:53.883245+0000","last_fullsized":"2024-03-28T23:31:53.883245+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:41.430571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.873125+0000","last_change":"2024-03-28T23:31:53.873125+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.873125+0000","last_undegraded":"2024-03-28T23:31:53.873125+0000","last_fullsized":"2024-03-28T23:31:53.873125+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.890278+0000","last_change":"2024-03-28T23:31:53.887894+0000","last_active":"2024-03-28T23:31:53.890278+0000","last_peered":"2024-03-28T23:31:53.890278+0000","last_clean":"2024-03-28T23:31:53.890278+0000","last_became_active":"2024-03-28T23:31:53.887619+0000","last_became_peered":"2024-03-28T23:31:53.887619+0000","last_unstale":"2024-03-28T23:31:53.890278+0000","last_undegraded":"2024-03-28T23:31:53.890278+0000","last_fullsized":"2024-03-28T23:31:53.890278+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:26:16.561871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879094+0000","last_change":"2024-03-28T23:31:40.392625+0000","last_active":"2024-03-28T23:31:53.879094+0000","last_peered":"2024-03-28T23:31:53.879094+0000","last_clean":"2024-03-28T23:31:53.879094+0000","last_became_active":"2024-03-28T23:31:40.392359+0000","last_became_peered":"2024-03-28T23:31:40.392359+0000","last_unstale":"2024-03-28T23:31:53.879094+0000","last_undegraded":"2024-03-28T23:31:53.879094+0000","last_fullsized":"2024-03-28T23:31:53.879094+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:02.676183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874872+0000","last_change":"2024-03-28T23:31:39.389478+0000","last_active":"2024-03-28T23:31:53.874872+0000","last_peered":"2024-03-28T23:31:53.874872+0000","last_clean":"2024-03-28T23:31:53.874872+0000","last_became_active":"2024-03-28T23:31:39.389154+0000","last_became_peered":"2024-03-28T23:31:39.389154+0000","last_unstale":"2024-03-28T23:31:53.874872+0000","last_undegraded":"2024-03-28T23:31:53.874872+0000","last_fullsized":"2024-03-28T23:31:53.874872+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:25:19.055737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871716+0000","last_change":"2024-03-28T23:31:53.871716+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871716+0000","last_undegraded":"2024-03-28T23:31:53.871716+0000","last_fullsized":"2024-03-28T23:31:53.871716+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.895972+0000","last_change":"2024-03-28T23:31:53.890575+0000","last_active":"2024-03-28T23:31:53.895972+0000","last_peered":"2024-03-28T23:31:53.895972+0000","last_clean":"2024-03-28T23:31:53.895972+0000","last_became_active":"2024-03-28T23:31:53.890305+0000","last_became_peered":"2024-03-28T23:31:53.890305+0000","last_unstale":"2024-03-28T23:31:53.895972+0000","last_undegraded":"2024-03-28T23:31:53.895972+0000","last_fullsized":"2024-03-28T23:31:53.895972+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:31:25.647111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876751+0000","last_change":"2024-03-28T23:31:40.389210+0000","last_active":"2024-03-28T23:31:53.876751+0000","last_peered":"2024-03-28T23:31:53.876751+0000","last_clean":"2024-03-28T23:31:53.876751+0000","last_became_active":"2024-03-28T23:31:40.388573+0000","last_became_peered":"2024-03-28T23:31:40.388573+0000","last_unstale":"2024-03-28T23:31:53.876751+0000","last_undegraded":"2024-03-28T23:31:53.876751+0000","last_fullsized":"2024-03-28T23:31:53.876751+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:23:53.153369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"51'1","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876646+0000","last_change":"2024-03-28T23:31:39.381202+0000","last_active":"2024-03-28T23:31:53.876646+0000","last_peered":"2024-03-28T23:31:53.876646+0000","last_clean":"2024-03-28T23:31:53.876646+0000","last_became_active":"2024-03-28T23:31:39.380913+0000","last_became_peered":"2024-03-28T23:31:39.380913+0000","last_unstale":"2024-03-28T23:31:53.876646+0000","last_undegraded":"2024-03-28T23:31:53.876646+0000","last_fullsized":"2024-03-28T23:31:53.876646+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:38:21.050492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":233,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.872708+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:31:53.872708+0000","last_peered":"2024-03-28T23:31:53.872708+0000","last_clean":"2024-03-28T23:31:53.872708+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:31:53.872708+0000","last_undegraded":"2024-03-28T23:31:53.872708+0000","last_fullsized":"2024-03-28T23:31:53.872708+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":704512,"data_stored":875469,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574878,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31792,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":626688,"data_stored":278115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574878,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31784,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340056,"statfs":{"total":96636764160,"available":96604217344,"internally_reserved":0,"allocated":618496,"data_stored":277463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574877,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31788,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340052,"statfs":{"total":96636764160,"available":96604213248,"internally_reserved":0,"allocated":622592,"data_stored":278045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":634880,"data_stored":279097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":630784,"data_stored":279075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":285137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":704512,"data_stored":875401,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:31:58.806 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3d is not active+clean 2024-03-28T23:31:58.806 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.872212+0000', 'last_change': '2024-03-28T23:31:53.872212+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.872212+0000', 'last_undegraded': '2024-03-28T23:31:53.872212+0000', 'last_fullsized': '2024-03-28T23:31:53.872212+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-28T23:31:58.806 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3c is not active+clean 2024-03-28T23:31:58.806 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.881085+0000', 'last_change': '2024-03-28T23:31:53.881085+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.881085+0000', 'last_undegraded': '2024-03-28T23:31:53.881085+0000', 'last_fullsized': '2024-03-28T23:31:53.881085+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:31:58.807 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3f is not active+clean 2024-03-28T23:31:58.807 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.873340+0000', 'last_change': '2024-03-28T23:31:53.873340+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.873340+0000', 'last_undegraded': '2024-03-28T23:31:53.873340+0000', 'last_fullsized': '2024-03-28T23:31:53.873340+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-28T23:31:58.807 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3e is not active+clean 2024-03-28T23:31:58.807 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.872399+0000', 'last_change': '2024-03-28T23:31:53.872399+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.872399+0000', 'last_undegraded': '2024-03-28T23:31:53.872399+0000', 'last_fullsized': '2024-03-28T23:31:53.872399+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-28T23:31:58.807 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.39 is not active+clean 2024-03-28T23:31:58.807 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.877696+0000', 'last_change': '2024-03-28T23:31:53.877696+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.877696+0000', 'last_undegraded': '2024-03-28T23:31:53.877696+0000', 'last_fullsized': '2024-03-28T23:31:53.877696+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-28T23:31:58.807 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.38 is not active+clean 2024-03-28T23:31:58.807 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.881544+0000', 'last_change': '2024-03-28T23:31:53.881544+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.881544+0000', 'last_undegraded': '2024-03-28T23:31:53.881544+0000', 'last_fullsized': '2024-03-28T23:31:53.881544+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-28T23:31:58.807 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3b is not active+clean 2024-03-28T23:31:58.807 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.872259+0000', 'last_change': '2024-03-28T23:31:53.872259+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.872259+0000', 'last_undegraded': '2024-03-28T23:31:53.872259+0000', 'last_fullsized': '2024-03-28T23:31:53.872259+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:31:58.808 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3a is not active+clean 2024-03-28T23:31:58.808 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.874359+0000', 'last_change': '2024-03-28T23:31:53.874359+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.874359+0000', 'last_undegraded': '2024-03-28T23:31:53.874359+0000', 'last_fullsized': '2024-03-28T23:31:53.874359+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:31:58.808 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.35 is not active+clean 2024-03-28T23:31:58.808 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.878422+0000', 'last_change': '2024-03-28T23:31:53.878422+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.878422+0000', 'last_undegraded': '2024-03-28T23:31:53.878422+0000', 'last_fullsized': '2024-03-28T23:31:53.878422+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:31:58.808 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.34 is not active+clean 2024-03-28T23:31:58.808 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.883968+0000', 'last_change': '2024-03-28T23:31:53.883968+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.883968+0000', 'last_undegraded': '2024-03-28T23:31:53.883968+0000', 'last_fullsized': '2024-03-28T23:31:53.883968+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-28T23:31:58.808 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.37 is not active+clean 2024-03-28T23:31:58.808 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.871125+0000', 'last_change': '2024-03-28T23:31:53.871125+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.871125+0000', 'last_undegraded': '2024-03-28T23:31:53.871125+0000', 'last_fullsized': '2024-03-28T23:31:53.871125+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-28T23:31:58.808 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.36 is not active+clean 2024-03-28T23:31:58.808 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.871848+0000', 'last_change': '2024-03-28T23:31:53.871848+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.871848+0000', 'last_undegraded': '2024-03-28T23:31:53.871848+0000', 'last_fullsized': '2024-03-28T23:31:53.871848+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-28T23:31:58.808 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.31 is not active+clean 2024-03-28T23:31:58.809 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.871018+0000', 'last_change': '2024-03-28T23:31:53.871018+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.871018+0000', 'last_undegraded': '2024-03-28T23:31:53.871018+0000', 'last_fullsized': '2024-03-28T23:31:53.871018+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-28T23:31:58.809 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.30 is not active+clean 2024-03-28T23:31:58.809 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.877447+0000', 'last_change': '2024-03-28T23:31:53.877447+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.877447+0000', 'last_undegraded': '2024-03-28T23:31:53.877447+0000', 'last_fullsized': '2024-03-28T23:31:53.877447+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-28T23:31:58.809 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.33 is not active+clean 2024-03-28T23:31:58.809 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.881048+0000', 'last_change': '2024-03-28T23:31:53.881048+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.881048+0000', 'last_undegraded': '2024-03-28T23:31:53.881048+0000', 'last_fullsized': '2024-03-28T23:31:53.881048+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-28T23:31:58.809 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.32 is not active+clean 2024-03-28T23:31:58.809 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.871402+0000', 'last_change': '2024-03-28T23:31:53.871402+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.871402+0000', 'last_undegraded': '2024-03-28T23:31:53.871402+0000', 'last_fullsized': '2024-03-28T23:31:53.871402+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-28T23:31:58.809 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2d is not active+clean 2024-03-28T23:31:58.809 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.872681+0000', 'last_change': '2024-03-28T23:31:53.872681+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.872681+0000', 'last_undegraded': '2024-03-28T23:31:53.872681+0000', 'last_fullsized': '2024-03-28T23:31:53.872681+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:31:58.809 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2c is not active+clean 2024-03-28T23:31:58.809 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.879388+0000', 'last_change': '2024-03-28T23:31:53.879388+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.879388+0000', 'last_undegraded': '2024-03-28T23:31:53.879388+0000', 'last_fullsized': '2024-03-28T23:31:53.879388+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-28T23:31:58.810 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2f is not active+clean 2024-03-28T23:31:58.810 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.872394+0000', 'last_change': '2024-03-28T23:31:53.872394+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.872394+0000', 'last_undegraded': '2024-03-28T23:31:53.872394+0000', 'last_fullsized': '2024-03-28T23:31:53.872394+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-28T23:31:58.810 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2e is not active+clean 2024-03-28T23:31:58.810 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.882562+0000', 'last_change': '2024-03-28T23:31:53.882562+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.882562+0000', 'last_undegraded': '2024-03-28T23:31:53.882562+0000', 'last_fullsized': '2024-03-28T23:31:53.882562+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:31:58.810 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.29 is not active+clean 2024-03-28T23:31:58.810 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.880460+0000', 'last_change': '2024-03-28T23:31:53.880460+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.880460+0000', 'last_undegraded': '2024-03-28T23:31:53.880460+0000', 'last_fullsized': '2024-03-28T23:31:53.880460+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-28T23:31:58.810 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.28 is not active+clean 2024-03-28T23:31:58.810 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.870958+0000', 'last_change': '2024-03-28T23:31:53.870958+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.870958+0000', 'last_undegraded': '2024-03-28T23:31:53.870958+0000', 'last_fullsized': '2024-03-28T23:31:53.870958+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-28T23:31:58.810 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2b is not active+clean 2024-03-28T23:31:58.810 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.879206+0000', 'last_change': '2024-03-28T23:31:53.879206+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.879206+0000', 'last_undegraded': '2024-03-28T23:31:53.879206+0000', 'last_fullsized': '2024-03-28T23:31:53.879206+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-28T23:31:58.811 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2a is not active+clean 2024-03-28T23:31:58.811 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.880913+0000', 'last_change': '2024-03-28T23:31:53.880913+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.880913+0000', 'last_undegraded': '2024-03-28T23:31:53.880913+0000', 'last_fullsized': '2024-03-28T23:31:53.880913+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-28T23:31:58.811 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.25 is not active+clean 2024-03-28T23:31:58.811 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.879370+0000', 'last_change': '2024-03-28T23:31:53.879370+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.879370+0000', 'last_undegraded': '2024-03-28T23:31:53.879370+0000', 'last_fullsized': '2024-03-28T23:31:53.879370+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:31:58.811 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.24 is not active+clean 2024-03-28T23:31:58.811 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.872876+0000', 'last_change': '2024-03-28T23:31:53.872876+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.872876+0000', 'last_undegraded': '2024-03-28T23:31:53.872876+0000', 'last_fullsized': '2024-03-28T23:31:53.872876+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-28T23:31:58.811 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.27 is not active+clean 2024-03-28T23:31:58.811 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.878874+0000', 'last_change': '2024-03-28T23:31:53.878874+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.878874+0000', 'last_undegraded': '2024-03-28T23:31:53.878874+0000', 'last_fullsized': '2024-03-28T23:31:53.878874+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-28T23:31:58.811 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.26 is not active+clean 2024-03-28T23:31:58.811 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.878430+0000', 'last_change': '2024-03-28T23:31:53.878430+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.878430+0000', 'last_undegraded': '2024-03-28T23:31:53.878430+0000', 'last_fullsized': '2024-03-28T23:31:53.878430+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-28T23:31:58.811 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.21 is not active+clean 2024-03-28T23:31:58.812 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.878783+0000', 'last_change': '2024-03-28T23:31:53.878783+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.878783+0000', 'last_undegraded': '2024-03-28T23:31:53.878783+0000', 'last_fullsized': '2024-03-28T23:31:53.878783+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:31:58.812 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.20 is not active+clean 2024-03-28T23:31:58.812 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.874991+0000', 'last_change': '2024-03-28T23:31:53.874991+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.874991+0000', 'last_undegraded': '2024-03-28T23:31:53.874991+0000', 'last_fullsized': '2024-03-28T23:31:53.874991+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:31:58.812 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.23 is not active+clean 2024-03-28T23:31:58.812 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.870965+0000', 'last_change': '2024-03-28T23:31:53.870965+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.870965+0000', 'last_undegraded': '2024-03-28T23:31:53.870965+0000', 'last_fullsized': '2024-03-28T23:31:53.870965+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-28T23:31:58.812 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.22 is not active+clean 2024-03-28T23:31:58.812 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.878422+0000', 'last_change': '2024-03-28T23:31:53.878422+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.878422+0000', 'last_undegraded': '2024-03-28T23:31:53.878422+0000', 'last_fullsized': '2024-03-28T23:31:53.878422+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-28T23:31:58.812 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.12 is not active+clean 2024-03-28T23:31:58.812 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.874735+0000', 'last_change': '2024-03-28T23:31:53.874735+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.874735+0000', 'last_undegraded': '2024-03-28T23:31:53.874735+0000', 'last_fullsized': '2024-03-28T23:31:53.874735+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:31:58.812 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.13 is not active+clean 2024-03-28T23:31:58.812 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.872455+0000', 'last_change': '2024-03-28T23:31:53.872455+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.872455+0000', 'last_undegraded': '2024-03-28T23:31:53.872455+0000', 'last_fullsized': '2024-03-28T23:31:53.872455+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-28T23:31:58.813 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.10 is not active+clean 2024-03-28T23:31:58.813 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.880671+0000', 'last_change': '2024-03-28T23:31:53.880671+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.880671+0000', 'last_undegraded': '2024-03-28T23:31:53.880671+0000', 'last_fullsized': '2024-03-28T23:31:53.880671+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:31:58.813 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.11 is not active+clean 2024-03-28T23:31:58.813 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.877711+0000', 'last_change': '2024-03-28T23:31:53.877711+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.877711+0000', 'last_undegraded': '2024-03-28T23:31:53.877711+0000', 'last_fullsized': '2024-03-28T23:31:53.877711+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-28T23:31:58.813 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.16 is not active+clean 2024-03-28T23:31:58.813 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.873288+0000', 'last_change': '2024-03-28T23:31:53.873288+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.873288+0000', 'last_undegraded': '2024-03-28T23:31:53.873288+0000', 'last_fullsized': '2024-03-28T23:31:53.873288+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:31:58.813 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.17 is not active+clean 2024-03-28T23:31:58.813 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.880191+0000', 'last_change': '2024-03-28T23:31:53.880191+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.880191+0000', 'last_undegraded': '2024-03-28T23:31:53.880191+0000', 'last_fullsized': '2024-03-28T23:31:53.880191+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:31:58.813 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.14 is not active+clean 2024-03-28T23:31:58.813 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.872423+0000', 'last_change': '2024-03-28T23:31:53.872423+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.872423+0000', 'last_undegraded': '2024-03-28T23:31:53.872423+0000', 'last_fullsized': '2024-03-28T23:31:53.872423+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-28T23:31:58.813 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.15 is not active+clean 2024-03-28T23:31:58.814 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.878702+0000', 'last_change': '2024-03-28T23:31:53.878702+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.878702+0000', 'last_undegraded': '2024-03-28T23:31:53.878702+0000', 'last_fullsized': '2024-03-28T23:31:53.878702+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-28T23:31:58.814 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1a is not active+clean 2024-03-28T23:31:58.814 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.872121+0000', 'last_change': '2024-03-28T23:31:53.872121+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.872121+0000', 'last_undegraded': '2024-03-28T23:31:53.872121+0000', 'last_fullsized': '2024-03-28T23:31:53.872121+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:31:58.814 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1b is not active+clean 2024-03-28T23:31:58.814 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.880894+0000', 'last_change': '2024-03-28T23:31:53.880894+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.880894+0000', 'last_undegraded': '2024-03-28T23:31:53.880894+0000', 'last_fullsized': '2024-03-28T23:31:53.880894+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-28T23:31:58.814 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.18 is not active+clean 2024-03-28T23:31:58.814 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.878283+0000', 'last_change': '2024-03-28T23:31:53.878283+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.878283+0000', 'last_undegraded': '2024-03-28T23:31:53.878283+0000', 'last_fullsized': '2024-03-28T23:31:53.878283+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-28T23:31:58.814 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.19 is not active+clean 2024-03-28T23:31:58.814 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.872413+0000', 'last_change': '2024-03-28T23:31:53.872413+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.872413+0000', 'last_undegraded': '2024-03-28T23:31:53.872413+0000', 'last_fullsized': '2024-03-28T23:31:53.872413+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-28T23:31:58.814 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1e is not active+clean 2024-03-28T23:31:58.814 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.874577+0000', 'last_change': '2024-03-28T23:31:53.874577+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.874577+0000', 'last_undegraded': '2024-03-28T23:31:53.874577+0000', 'last_fullsized': '2024-03-28T23:31:53.874577+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:31:58.815 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1f is not active+clean 2024-03-28T23:31:58.815 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.871676+0000', 'last_change': '2024-03-28T23:31:53.871676+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.871676+0000', 'last_undegraded': '2024-03-28T23:31:53.871676+0000', 'last_fullsized': '2024-03-28T23:31:53.871676+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-28T23:31:58.815 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1c is not active+clean 2024-03-28T23:31:58.815 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.872186+0000', 'last_change': '2024-03-28T23:31:53.872186+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.872186+0000', 'last_undegraded': '2024-03-28T23:31:53.872186+0000', 'last_fullsized': '2024-03-28T23:31:53.872186+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:31:58.815 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1d is not active+clean 2024-03-28T23:31:58.815 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.878941+0000', 'last_change': '2024-03-28T23:31:53.878941+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.878941+0000', 'last_undegraded': '2024-03-28T23:31:53.878941+0000', 'last_fullsized': '2024-03-28T23:31:53.878941+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:31:58.815 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2 is not active+clean 2024-03-28T23:31:58.815 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.872289+0000', 'last_change': '2024-03-28T23:31:53.872289+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.872289+0000', 'last_undegraded': '2024-03-28T23:31:53.872289+0000', 'last_fullsized': '2024-03-28T23:31:53.872289+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-28T23:31:58.815 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3 is not active+clean 2024-03-28T23:31:58.815 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.877420+0000', 'last_change': '2024-03-28T23:31:53.877420+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.877420+0000', 'last_undegraded': '2024-03-28T23:31:53.877420+0000', 'last_fullsized': '2024-03-28T23:31:53.877420+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-28T23:31:58.815 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.0 is not active+clean 2024-03-28T23:31:58.815 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.873793+0000', 'last_change': '2024-03-28T23:31:53.873793+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.873793+0000', 'last_undegraded': '2024-03-28T23:31:53.873793+0000', 'last_fullsized': '2024-03-28T23:31:53.873793+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:31:58.816 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1 is not active+clean 2024-03-28T23:31:58.816 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.879255+0000', 'last_change': '2024-03-28T23:31:53.879255+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.879255+0000', 'last_undegraded': '2024-03-28T23:31:53.879255+0000', 'last_fullsized': '2024-03-28T23:31:53.879255+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:31:58.816 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.6 is not active+clean 2024-03-28T23:31:58.816 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.876579+0000', 'last_change': '2024-03-28T23:31:53.876579+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.876579+0000', 'last_undegraded': '2024-03-28T23:31:53.876579+0000', 'last_fullsized': '2024-03-28T23:31:53.876579+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-28T23:31:58.816 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.7 is not active+clean 2024-03-28T23:31:58.816 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.872546+0000', 'last_change': '2024-03-28T23:31:53.872546+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.872546+0000', 'last_undegraded': '2024-03-28T23:31:53.872546+0000', 'last_fullsized': '2024-03-28T23:31:53.872546+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:31:58.816 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.4 is not active+clean 2024-03-28T23:31:58.816 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.874240+0000', 'last_change': '2024-03-28T23:31:53.874240+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.874240+0000', 'last_undegraded': '2024-03-28T23:31:53.874240+0000', 'last_fullsized': '2024-03-28T23:31:53.874240+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:31:58.816 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.5 is not active+clean 2024-03-28T23:31:58.816 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.877392+0000', 'last_change': '2024-03-28T23:31:53.877392+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.877392+0000', 'last_undegraded': '2024-03-28T23:31:53.877392+0000', 'last_fullsized': '2024-03-28T23:31:53.877392+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-28T23:31:58.816 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.a is not active+clean 2024-03-28T23:31:58.817 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.874303+0000', 'last_change': '2024-03-28T23:31:53.874303+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.874303+0000', 'last_undegraded': '2024-03-28T23:31:53.874303+0000', 'last_fullsized': '2024-03-28T23:31:53.874303+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:31:58.817 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.8 is not active+clean 2024-03-28T23:31:58.817 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.874197+0000', 'last_change': '2024-03-28T23:31:53.874197+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.874197+0000', 'last_undegraded': '2024-03-28T23:31:53.874197+0000', 'last_fullsized': '2024-03-28T23:31:53.874197+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:31:58.817 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.9 is not active+clean 2024-03-28T23:31:58.817 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.876269+0000', 'last_change': '2024-03-28T23:31:53.876269+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.876269+0000', 'last_undegraded': '2024-03-28T23:31:53.876269+0000', 'last_fullsized': '2024-03-28T23:31:53.876269+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-28T23:31:58.817 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.e is not active+clean 2024-03-28T23:31:58.817 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.879694+0000', 'last_change': '2024-03-28T23:31:53.879694+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.879694+0000', 'last_undegraded': '2024-03-28T23:31:53.879694+0000', 'last_fullsized': '2024-03-28T23:31:53.879694+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:31:58.817 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.f is not active+clean 2024-03-28T23:31:58.817 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.872993+0000', 'last_change': '2024-03-28T23:31:53.872993+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.872993+0000', 'last_undegraded': '2024-03-28T23:31:53.872993+0000', 'last_fullsized': '2024-03-28T23:31:53.872993+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:31:58.817 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.c is not active+clean 2024-03-28T23:31:58.817 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.871482+0000', 'last_change': '2024-03-28T23:31:53.871482+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.871482+0000', 'last_undegraded': '2024-03-28T23:31:53.871482+0000', 'last_fullsized': '2024-03-28T23:31:53.871482+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-28T23:31:58.818 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.d is not active+clean 2024-03-28T23:31:58.818 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.873125+0000', 'last_change': '2024-03-28T23:31:53.873125+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.873125+0000', 'last_undegraded': '2024-03-28T23:31:53.873125+0000', 'last_fullsized': '2024-03-28T23:31:53.873125+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:31:58.818 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.b is not active+clean 2024-03-28T23:31:58.818 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:31:53.871716+0000', 'last_change': '2024-03-28T23:31:53.871716+0000', 'last_active': '2024-03-28T23:31:53.865658+0000', 'last_peered': '2024-03-28T23:31:53.865658+0000', 'last_clean': '2024-03-28T23:31:53.865658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:31:53.871716+0000', 'last_undegraded': '2024-03-28T23:31:53.871716+0000', 'last_fullsized': '2024-03-28T23:31:53.871716+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'last_clean_scrub_stamp': '2024-03-28T23:31:53.865658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:31:58.819 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-28T23:31:58.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.939+0000 7f6f53a8f640 1 Processor -- start 2024-03-28T23:31:58.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.940+0000 7f6f53a8f640 1 -- start start 2024-03-28T23:31:58.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.941+0000 7f6f53a8f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6f4c1592e0 0x7f6f4c0c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:58.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.941+0000 7f6f53a8f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f4c159730 con 0x7f6f4c14f260 2024-03-28T23:31:58.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.941+0000 7f6f53a8f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f4c0cb4e0 con 0x7f6f4c0caf60 2024-03-28T23:31:58.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.941+0000 7f6f53a8f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6f4c0c9080 con 0x7f6f4c1592e0 2024-03-28T23:31:58.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.941+0000 7f6f5202c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6f4c1592e0 0x7f6f4c0c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:58.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.941+0000 7f6f5202c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6f4c1592e0 0x7f6f4c0c53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35232/0 (socket says 172.21.15.161:35232) 2024-03-28T23:31:58.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.941+0000 7f6f5202c640 1 -- 172.21.15.161:0/4064374505 learned_addr learned my addr 172.21.15.161:0/4064374505 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:58.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.941+0000 7f6f5182b640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6f4c14f260 0x7f6f4c155c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37826/0 (socket says 172.21.15.161:37826) 2024-03-28T23:31:58.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.941+0000 7f6f5202c640 1 -- 172.21.15.161:0/4064374505 >> v1:172.21.15.170:6789/0 conn(0x7f6f4c0caf60 legacy=0x7f6f4c0c8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:58.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.941+0000 7f6f5202c640 1 -- 172.21.15.161:0/4064374505 >> v1:172.21.15.161:6789/0 conn(0x7f6f4c14f260 legacy=0x7f6f4c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:58.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.941+0000 7f6f5202c640 1 -- 172.21.15.161:0/4064374505 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f4c0cb7c0 con 0x7f6f4c1592e0 2024-03-28T23:31:58.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.941+0000 7f6f50829640 1 -- 172.21.15.161:0/4064374505 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 969194599 0 0) 0x7f6f4c159730 con 0x7f6f4c14f260 2024-03-28T23:31:58.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.942+0000 7f6f5202c640 1 --2- 172.21.15.161:0/4064374505 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f4c1592e0 0x7f6f4c0c53f0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f6f44009870 tx=0x7f6f4402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=286f751739234a6c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:58.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.942+0000 7f6f50829640 1 -- 172.21.15.161:0/4064374505 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f44034070 con 0x7f6f4c1592e0 2024-03-28T23:31:58.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.942+0000 7f6f50829640 1 -- 172.21.15.161:0/4064374505 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f44030860 con 0x7f6f4c1592e0 2024-03-28T23:31:58.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.942+0000 7f6f50829640 1 -- 172.21.15.161:0/4064374505 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f44030c30 con 0x7f6f4c1592e0 2024-03-28T23:31:58.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.943+0000 7f6f53a8f640 1 -- 172.21.15.161:0/4064374505 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f4c1592e0 msgr2=0x7f6f4c0c53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:58.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.943+0000 7f6f53a8f640 1 --2- 172.21.15.161:0/4064374505 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f4c1592e0 0x7f6f4c0c53f0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f6f44009870 tx=0x7f6f4402b9a0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.943+0000 7f6f53a8f640 1 -- 172.21.15.161:0/4064374505 shutdown_connections 2024-03-28T23:31:58.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.943+0000 7f6f53a8f640 1 --2- 172.21.15.161:0/4064374505 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f4c1592e0 0x7f6f4c0c53f0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.943+0000 7f6f53a8f640 1 -- 172.21.15.161:0/4064374505 >> 172.21.15.161:0/4064374505 conn(0x7f6f4c072ca0 msgr2=0x7f6f4c0730b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:58.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.943+0000 7f6f53a8f640 1 -- 172.21.15.161:0/4064374505 shutdown_connections 2024-03-28T23:31:58.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.943+0000 7f6f53a8f640 1 -- 172.21.15.161:0/4064374505 wait complete. 2024-03-28T23:31:58.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.945+0000 7f6f53a8f640 1 Processor -- start 2024-03-28T23:31:58.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.945+0000 7f6f53a8f640 1 -- start start 2024-03-28T23:31:58.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.946+0000 7f6f53a8f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f4c0caf60 0x7f6f4c0c0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:58.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.946+0000 7f6f53a8f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f4c14f260 0x7f6f4c151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:58.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.946+0000 7f6f53a8f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f4c1592e0 0x7f6f4c1ede50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:58.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.946+0000 7f6f53a8f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6f4c158ae0 con 0x7f6f4c14f260 2024-03-28T23:31:58.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.946+0000 7f6f53a8f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6f4c158e50 con 0x7f6f4c1592e0 2024-03-28T23:31:58.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.946+0000 7f6f53a8f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6f4c14f640 con 0x7f6f4c0caf60 2024-03-28T23:31:58.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.946+0000 7f6f5102a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f4c14f260 0x7f6f4c151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:58.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.946+0000 7f6f5102a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f4c14f260 0x7f6f4c151b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35246/0 (socket says 172.21.15.161:35246) 2024-03-28T23:31:58.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.946+0000 7f6f5102a640 1 -- 172.21.15.161:0/750292986 learned_addr learned my addr 172.21.15.161:0/750292986 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:58.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.946+0000 7f6f5182b640 1 --2- 172.21.15.161:0/750292986 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f4c0caf60 0x7f6f4c0c0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:58.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.946+0000 7f6f5202c640 1 --2- 172.21.15.161:0/750292986 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f4c1592e0 0x7f6f4c1ede50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:58.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.946+0000 7f6f5102a640 1 -- 172.21.15.161:0/750292986 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f4c0caf60 msgr2=0x7f6f4c0c0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:58.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.946+0000 7f6f5102a640 1 --2- 172.21.15.161:0/750292986 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f4c0caf60 0x7f6f4c0c0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.946+0000 7f6f5102a640 1 -- 172.21.15.161:0/750292986 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f4c1592e0 msgr2=0x7f6f4c1ede50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:58.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.946+0000 7f6f5102a640 1 --2- 172.21.15.161:0/750292986 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f4c1592e0 0x7f6f4c1ede50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:58.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.946+0000 7f6f5102a640 1 -- 172.21.15.161:0/750292986 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f4c14ea50 con 0x7f6f4c14f260 2024-03-28T23:31:58.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.946+0000 7f6f5182b640 1 --2- 172.21.15.161:0/750292986 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f4c0caf60 0x7f6f4c0c0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:58.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.946+0000 7f6f5202c640 1 --2- 172.21.15.161:0/750292986 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f4c1592e0 0x7f6f4c1ede50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:31:58.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.946+0000 7f6f5102a640 1 --2- 172.21.15.161:0/750292986 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f4c14f260 0x7f6f4c151b00 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000c450 tx=0x7f6f4000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:58.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.946+0000 7f6f4a7fc640 1 -- 172.21.15.161:0/750292986 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f400052d0 con 0x7f6f4c14f260 2024-03-28T23:31:58.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.946+0000 7f6f4a7fc640 1 -- 172.21.15.161:0/750292986 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f40005470 con 0x7f6f4c14f260 2024-03-28T23:31:58.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.946+0000 7f6f4a7fc640 1 -- 172.21.15.161:0/750292986 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f40012870 con 0x7f6f4c14f260 2024-03-28T23:31:58.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.946+0000 7f6f53a8f640 1 -- 172.21.15.161:0/750292986 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f4c1eeb40 con 0x7f6f4c14f260 2024-03-28T23:31:58.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.947+0000 7f6f53a8f640 1 -- 172.21.15.161:0/750292986 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f4c14e770 con 0x7f6f4c14f260 2024-03-28T23:31:58.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.948+0000 7f6f33fff640 1 -- 172.21.15.161:0/750292986 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f4c04f550 con 0x7f6f4c14f260 2024-03-28T23:31:58.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.948+0000 7f6f4a7fc640 1 -- 172.21.15.161:0/750292986 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6f40012a10 con 0x7f6f4c14f260 2024-03-28T23:31:58.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.948+0000 7f6f4a7fc640 1 --2- 172.21.15.161:0/750292986 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6f3806b500 0x7f6f3806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:58.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.949+0000 7f6f4a7fc640 1 -- 172.21.15.161:0/750292986 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6957+0+0 (secure 0 0 0) 0x7f6f4008a240 con 0x7f6f4c14f260 2024-03-28T23:31:58.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.949+0000 7f6f5182b640 1 --2- 172.21.15.161:0/750292986 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6f3806b500 0x7f6f3806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:58.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.949+0000 7f6f5182b640 1 --2- 172.21.15.161:0/750292986 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6f3806b500 0x7f6f3806d9c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c0012e0 tx=0x7f6f3c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:58.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:58.957+0000 7f6f4a7fc640 1 -- 172.21.15.161:0/750292986 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6f40058c70 con 0x7f6f4c14f260 2024-03-28T23:31:59.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.336+0000 7f6f33fff640 1 -- 172.21.15.161:0/750292986 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f6f4c06cfe0 con 0x7f6f4c14f260 2024-03-28T23:31:59.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.336+0000 7f6f4a7fc640 1 -- 172.21.15.161:0/750292986 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1138 (secure 0 0 0) 0x7f6f40058570 con 0x7f6f4c14f260 2024-03-28T23:31:59.337 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:59.337 INFO:teuthology.orchestra.run.smithi161.stdout:{"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":150,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1711668579,"num_in_osds":8,"osd_in_since":1711668571,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":129},{"state_name":"creating+peering","count":64}],"num_pgs":193,"num_pools":5,"num_objects":24,"data_bytes":592778,"bytes_used":243576832,"bytes_avail":772850536448,"bytes_total":773094113280,"inactive_pgs_ratio":0.33160620927810669},"fsmap":{"epoch":34,"id":3,"up":1,"in":1,"max":1,"id":4,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":4,"rank":0,"name":"c","status":"up:active","gid":5817},{"filesystem_id":3,"rank":0,"name":"a","status":"up:active","gid":5820}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-03-28T23:30:47.324341+0000","services":{}},"progress_events":{}} 2024-03-28T23:31:59.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.339+0000 7f6f33fff640 1 -- 172.21.15.161:0/750292986 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6f3806b500 msgr2=0x7f6f3806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:59.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.339+0000 7f6f33fff640 1 --2- 172.21.15.161:0/750292986 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6f3806b500 0x7f6f3806d9c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c0012e0 tx=0x7f6f3c009040 comp rx=0 tx=0).stop 2024-03-28T23:31:59.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.339+0000 7f6f33fff640 1 -- 172.21.15.161:0/750292986 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f4c14f260 msgr2=0x7f6f4c151b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:59.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.339+0000 7f6f33fff640 1 --2- 172.21.15.161:0/750292986 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f4c14f260 0x7f6f4c151b00 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000c450 tx=0x7f6f4000c910 comp rx=0 tx=0).stop 2024-03-28T23:31:59.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.339+0000 7f6f33fff640 1 -- 172.21.15.161:0/750292986 shutdown_connections 2024-03-28T23:31:59.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.339+0000 7f6f33fff640 1 --2- 172.21.15.161:0/750292986 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6f3806b500 0x7f6f3806d9c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:59.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.339+0000 7f6f33fff640 1 --2- 172.21.15.161:0/750292986 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f4c14f260 0x7f6f4c151b00 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:59.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.339+0000 7f6f33fff640 1 --2- 172.21.15.161:0/750292986 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f4c1592e0 0x7f6f4c1ede50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:59.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.339+0000 7f6f33fff640 1 --2- 172.21.15.161:0/750292986 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f4c0caf60 0x7f6f4c0c0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:59.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.339+0000 7f6f33fff640 1 -- 172.21.15.161:0/750292986 >> 172.21.15.161:0/750292986 conn(0x7f6f4c072ca0 msgr2=0x7f6f4c073080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:59.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.339+0000 7f6f33fff640 1 -- 172.21.15.161:0/750292986 shutdown_connections 2024-03-28T23:31:59.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.339+0000 7f6f33fff640 1 -- 172.21.15.161:0/750292986 wait complete. 2024-03-28T23:31:59.351 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-28T23:31:59.351 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:31:59.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.462+0000 7faac2525640 1 Processor -- start 2024-03-28T23:31:59.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.463+0000 7faac2525640 1 -- start start 2024-03-28T23:31:59.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.464+0000 7faac2525640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7faabc0c5410 0x7faabc0c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:59.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.464+0000 7faac2525640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7faabc1553a0 0x7faabc155850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:59.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.464+0000 7faac2525640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faabc158a30 con 0x7faabc152890 2024-03-28T23:31:59.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.464+0000 7faac2525640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7faabc0c8480 con 0x7faabc0c5410 2024-03-28T23:31:59.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.464+0000 7faac2525640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7faabc0c5d90 con 0x7faabc1553a0 2024-03-28T23:31:59.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.464+0000 7faabb7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7faabc0c5410 0x7faabc0c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:59.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.464+0000 7faabbfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7faabc152890 0x7faabc0c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37836/0 (socket says 172.21.15.161:37836) 2024-03-28T23:31:59.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.464+0000 7faabb7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7faabc0c5410 0x7faabc0c5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35252/0 (socket says 172.21.15.161:35252) 2024-03-28T23:31:59.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.464+0000 7faabbfff640 1 -- 172.21.15.161:0/3933100705 learned_addr learned my addr 172.21.15.161:0/3933100705 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:31:59.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.464+0000 7faac0ac2640 1 --2- 172.21.15.161:0/3933100705 >> v2:172.21.15.170:3300/0 conn(0x7faabc1553a0 0x7faabc155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:59.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.464+0000 7faabb7fe640 1 -- 172.21.15.161:0/3933100705 >> v2:172.21.15.170:3300/0 conn(0x7faabc1553a0 msgr2=0x7faabc155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:31:59.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.464+0000 7faabb7fe640 1 --2- 172.21.15.161:0/3933100705 >> v2:172.21.15.170:3300/0 conn(0x7faabc1553a0 0x7faabc155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:59.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.464+0000 7faabb7fe640 1 -- 172.21.15.161:0/3933100705 >> v1:172.21.15.161:6789/0 conn(0x7faabc152890 legacy=0x7faabc0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:59.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.464+0000 7faabaffd640 1 -- 172.21.15.161:0/3933100705 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4228769761 0 0) 0x7faabc158a30 con 0x7faabc152890 2024-03-28T23:31:59.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.464+0000 7faabb7fe640 1 -- 172.21.15.161:0/3933100705 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faabc158d10 con 0x7faabc0c5410 2024-03-28T23:31:59.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.465+0000 7faabb7fe640 1 --2- 172.21.15.161:0/3933100705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faabc0c5410 0x7faabc0c5850 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7faabc0aa310 tx=0x7faab002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8baebbd54aa2a564 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:59.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.465+0000 7faabaffd640 1 -- 172.21.15.161:0/3933100705 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab00315a0 con 0x7faabc0c5410 2024-03-28T23:31:59.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.465+0000 7faabaffd640 1 -- 172.21.15.161:0/3933100705 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faab0037070 con 0x7faabc0c5410 2024-03-28T23:31:59.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.465+0000 7faabaffd640 1 -- 172.21.15.161:0/3933100705 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab0031960 con 0x7faabc0c5410 2024-03-28T23:31:59.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.466+0000 7faac2525640 1 -- 172.21.15.161:0/3933100705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faabc0c5410 msgr2=0x7faabc0c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:59.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.466+0000 7faac2525640 1 --2- 172.21.15.161:0/3933100705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faabc0c5410 0x7faabc0c5850 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7faabc0aa310 tx=0x7faab002b840 comp rx=0 tx=0).stop 2024-03-28T23:31:59.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.466+0000 7faac2525640 1 -- 172.21.15.161:0/3933100705 shutdown_connections 2024-03-28T23:31:59.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.466+0000 7faac2525640 1 --2- 172.21.15.161:0/3933100705 >> v2:172.21.15.170:3300/0 conn(0x7faabc1553a0 0x7faabc155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:59.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.466+0000 7faac2525640 1 --2- 172.21.15.161:0/3933100705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faabc0c5410 0x7faabc0c5850 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:59.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.466+0000 7faac2525640 1 -- 172.21.15.161:0/3933100705 >> 172.21.15.161:0/3933100705 conn(0x7faabc0ac620 msgr2=0x7faabc0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:59.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.466+0000 7faac2525640 1 -- 172.21.15.161:0/3933100705 shutdown_connections 2024-03-28T23:31:59.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.466+0000 7faac2525640 1 -- 172.21.15.161:0/3933100705 wait complete. 2024-03-28T23:31:59.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.467+0000 7faac2525640 1 Processor -- start 2024-03-28T23:31:59.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.467+0000 7faac2525640 1 -- start start 2024-03-28T23:31:59.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.468+0000 7faac2525640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faabc0c5410 0x7faabc1482a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:59.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.468+0000 7faac2525640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faabc152890 0x7faabc14d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:59.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.468+0000 7faac2525640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faabc1553a0 0x7faabc157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:59.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.468+0000 7faac2525640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7faabc1592d0 con 0x7faabc0c5410 2024-03-28T23:31:59.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.468+0000 7faac2525640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7faabc159640 con 0x7faabc1553a0 2024-03-28T23:31:59.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.468+0000 7faac2525640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7faabc0c90a0 con 0x7faabc152890 2024-03-28T23:31:59.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.468+0000 7faabbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faabc0c5410 0x7faabc1482a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:59.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.468+0000 7faabbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faabc0c5410 0x7faabc1482a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35258/0 (socket says 172.21.15.161:35258) 2024-03-28T23:31:59.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.468+0000 7faabbfff640 1 -- 172.21.15.161:0/868906345 learned_addr learned my addr 172.21.15.161:0/868906345 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:31:59.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.468+0000 7faabb7fe640 1 --2- 172.21.15.161:0/868906345 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faabc152890 0x7faabc14d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:59.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.468+0000 7faac0ac2640 1 --2- 172.21.15.161:0/868906345 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faabc1553a0 0x7faabc157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:59.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.468+0000 7faabbfff640 1 -- 172.21.15.161:0/868906345 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faabc152890 msgr2=0x7faabc14d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:59.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.468+0000 7faabbfff640 1 --2- 172.21.15.161:0/868906345 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faabc152890 0x7faabc14d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:59.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.468+0000 7faabbfff640 1 -- 172.21.15.161:0/868906345 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faabc1553a0 msgr2=0x7faabc157d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:59.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.468+0000 7faabbfff640 1 --2- 172.21.15.161:0/868906345 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faabc1553a0 0x7faabc157d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:59.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.468+0000 7faabbfff640 1 -- 172.21.15.161:0/868906345 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faabc149ce0 con 0x7faabc0c5410 2024-03-28T23:31:59.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.468+0000 7faabbfff640 1 --2- 172.21.15.161:0/868906345 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faabc0c5410 0x7faabc1482a0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7faaac00c4d0 tx=0x7faaac00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:59.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.469+0000 7faab8ff9640 1 -- 172.21.15.161:0/868906345 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaac018070 con 0x7faabc0c5410 2024-03-28T23:31:59.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.469+0000 7faab8ff9640 1 -- 172.21.15.161:0/868906345 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaac007500 con 0x7faabc0c5410 2024-03-28T23:31:59.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.469+0000 7faac2525640 1 -- 172.21.15.161:0/868906345 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faabc14e870 con 0x7faabc0c5410 2024-03-28T23:31:59.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.469+0000 7faab8ff9640 1 -- 172.21.15.161:0/868906345 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaac004a20 con 0x7faabc0c5410 2024-03-28T23:31:59.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.469+0000 7faac2525640 1 -- 172.21.15.161:0/868906345 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faabc148c90 con 0x7faabc0c5410 2024-03-28T23:31:59.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.470+0000 7faac2525640 1 -- 172.21.15.161:0/868906345 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faabc04f4b0 con 0x7faabc0c5410 2024-03-28T23:31:59.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.470+0000 7faab8ff9640 1 -- 172.21.15.161:0/868906345 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7faaac01b070 con 0x7faabc0c5410 2024-03-28T23:31:59.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.471+0000 7faab8ff9640 1 --2- 172.21.15.161:0/868906345 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faaa806b4d0 0x7faaa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:31:59.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.471+0000 7faab8ff9640 1 -- 172.21.15.161:0/868906345 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6957+0+0 (secure 0 0 0) 0x7faaac0898e0 con 0x7faabc0c5410 2024-03-28T23:31:59.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.471+0000 7faabb7fe640 1 --2- 172.21.15.161:0/868906345 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faaa806b4d0 0x7faaa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:31:59.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.471+0000 7faabb7fe640 1 --2- 172.21.15.161:0/868906345 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faaa806b4d0 0x7faaa806d990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7faab0036000 tx=0x7faab0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:31:59.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.479+0000 7faab8ff9640 1 -- 172.21.15.161:0/868906345 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7faaac057cc0 con 0x7faabc0c5410 2024-03-28T23:31:59.636 INFO:tasks.ceph.mds.c.smithi161.stderr:2024-03-28T23:31:59.636+0000 7f1efe9ea640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:31:59.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.654+0000 7faac2525640 1 -- 172.21.15.161:0/868906345 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faabc14d420 con 0x7faaa806b4d0 2024-03-28T23:31:59.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.668+0000 7faab8ff9640 1 -- 172.21.15.161:0/868906345 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+485262 (secure 0 0 0) 0x7faabc14d420 con 0x7faaa806b4d0 2024-03-28T23:31:59.675 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:31:59.678 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:31:59.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.672+0000 7faac2525640 1 -- 172.21.15.161:0/868906345 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faaa806b4d0 msgr2=0x7faaa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:59.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.672+0000 7faac2525640 1 --2- 172.21.15.161:0/868906345 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faaa806b4d0 0x7faaa806d990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7faab0036000 tx=0x7faab0008030 comp rx=0 tx=0).stop 2024-03-28T23:31:59.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.672+0000 7faac2525640 1 -- 172.21.15.161:0/868906345 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faabc0c5410 msgr2=0x7faabc1482a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:31:59.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.672+0000 7faac2525640 1 --2- 172.21.15.161:0/868906345 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faabc0c5410 0x7faabc1482a0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7faaac00c4d0 tx=0x7faaac00c990 comp rx=0 tx=0).stop 2024-03-28T23:31:59.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.673+0000 7faac2525640 1 -- 172.21.15.161:0/868906345 shutdown_connections 2024-03-28T23:31:59.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.673+0000 7faac2525640 1 --2- 172.21.15.161:0/868906345 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faaa806b4d0 0x7faaa806d990 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:59.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.673+0000 7faac2525640 1 --2- 172.21.15.161:0/868906345 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faabc1553a0 0x7faabc157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:59.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.673+0000 7faac2525640 1 --2- 172.21.15.161:0/868906345 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faabc152890 0x7faabc14d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:59.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.673+0000 7faac2525640 1 --2- 172.21.15.161:0/868906345 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faabc0c5410 0x7faabc1482a0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:31:59.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.673+0000 7faac2525640 1 -- 172.21.15.161:0/868906345 >> 172.21.15.161:0/868906345 conn(0x7faabc0ac620 msgr2=0x7faabc06c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:31:59.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.673+0000 7faac2525640 1 -- 172.21.15.161:0/868906345 shutdown_connections 2024-03-28T23:31:59.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:31:59.673+0000 7faac2525640 1 -- 172.21.15.161:0/868906345 wait complete. 2024-03-28T23:31:59.684 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-03-28T23:31:59.336787+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":232,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237868,"kb_used_data":5060,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736852,"statfs":{"total":773094113280,"available":772850536448,"internally_reserved":0,"allocated":5181440,"data_stored":3427802,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000948"},"pg_stats":[{"pgid":"10.3d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872212+0000","last_change":"2024-03-28T23:31:53.872212+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872212+0000","last_undegraded":"2024-03-28T23:31:53.872212+0000","last_fullsized":"2024-03-28T23:31:53.872212+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3f","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.870419+0000","last_change":"2024-03-28T23:31:40.388090+0000","last_active":"2024-03-28T23:31:53.870419+0000","last_peered":"2024-03-28T23:31:53.870419+0000","last_clean":"2024-03-28T23:31:53.870419+0000","last_became_active":"2024-03-28T23:31:40.387805+0000","last_became_peered":"2024-03-28T23:31:40.387805+0000","last_unstale":"2024-03-28T23:31:53.870419+0000","last_undegraded":"2024-03-28T23:31:53.870419+0000","last_fullsized":"2024-03-28T23:31:53.870419+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:19:47.987753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.881085+0000","last_change":"2024-03-28T23:31:53.881085+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.881085+0000","last_undegraded":"2024-03-28T23:31:53.881085+0000","last_fullsized":"2024-03-28T23:31:53.881085+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875872+0000","last_change":"2024-03-28T23:31:40.395246+0000","last_active":"2024-03-28T23:31:53.875872+0000","last_peered":"2024-03-28T23:31:53.875872+0000","last_clean":"2024-03-28T23:31:53.875872+0000","last_became_active":"2024-03-28T23:31:40.394583+0000","last_became_peered":"2024-03-28T23:31:40.394583+0000","last_unstale":"2024-03-28T23:31:53.875872+0000","last_undegraded":"2024-03-28T23:31:53.875872+0000","last_fullsized":"2024-03-28T23:31:53.875872+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:18.654331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.873340+0000","last_change":"2024-03-28T23:31:53.873340+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.873340+0000","last_undegraded":"2024-03-28T23:31:53.873340+0000","last_fullsized":"2024-03-28T23:31:53.873340+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876107+0000","last_change":"2024-03-28T23:31:40.397914+0000","last_active":"2024-03-28T23:31:53.876107+0000","last_peered":"2024-03-28T23:31:53.876107+0000","last_clean":"2024-03-28T23:31:53.876107+0000","last_became_active":"2024-03-28T23:31:40.394904+0000","last_became_peered":"2024-03-28T23:31:40.394904+0000","last_unstale":"2024-03-28T23:31:53.876107+0000","last_undegraded":"2024-03-28T23:31:53.876107+0000","last_fullsized":"2024-03-28T23:31:53.876107+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:26:03.062546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872399+0000","last_change":"2024-03-28T23:31:53.872399+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872399+0000","last_undegraded":"2024-03-28T23:31:53.872399+0000","last_fullsized":"2024-03-28T23:31:53.872399+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874319+0000","last_change":"2024-03-28T23:31:40.400594+0000","last_active":"2024-03-28T23:31:53.874319+0000","last_peered":"2024-03-28T23:31:53.874319+0000","last_clean":"2024-03-28T23:31:53.874319+0000","last_became_active":"2024-03-28T23:31:40.400255+0000","last_became_peered":"2024-03-28T23:31:40.400255+0000","last_unstale":"2024-03-28T23:31:53.874319+0000","last_undegraded":"2024-03-28T23:31:53.874319+0000","last_fullsized":"2024-03-28T23:31:53.874319+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:14.515069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.877696+0000","last_change":"2024-03-28T23:31:53.877696+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.877696+0000","last_undegraded":"2024-03-28T23:31:53.877696+0000","last_fullsized":"2024-03-28T23:31:53.877696+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.884068+0000","last_change":"2024-03-28T23:31:40.388531+0000","last_active":"2024-03-28T23:31:53.884068+0000","last_peered":"2024-03-28T23:31:53.884068+0000","last_clean":"2024-03-28T23:31:53.884068+0000","last_became_active":"2024-03-28T23:31:40.388206+0000","last_became_peered":"2024-03-28T23:31:40.388206+0000","last_unstale":"2024-03-28T23:31:53.884068+0000","last_undegraded":"2024-03-28T23:31:53.884068+0000","last_fullsized":"2024-03-28T23:31:53.884068+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:23.215366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.881544+0000","last_change":"2024-03-28T23:31:53.881544+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.881544+0000","last_undegraded":"2024-03-28T23:31:53.881544+0000","last_fullsized":"2024-03-28T23:31:53.881544+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869622+0000","last_change":"2024-03-28T23:31:40.393907+0000","last_active":"2024-03-28T23:31:53.869622+0000","last_peered":"2024-03-28T23:31:53.869622+0000","last_clean":"2024-03-28T23:31:53.869622+0000","last_became_active":"2024-03-28T23:31:40.393443+0000","last_became_peered":"2024-03-28T23:31:40.393443+0000","last_unstale":"2024-03-28T23:31:53.869622+0000","last_undegraded":"2024-03-28T23:31:53.869622+0000","last_fullsized":"2024-03-28T23:31:53.869622+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:28:29.180775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872259+0000","last_change":"2024-03-28T23:31:53.872259+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872259+0000","last_undegraded":"2024-03-28T23:31:53.872259+0000","last_fullsized":"2024-03-28T23:31:53.872259+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875554+0000","last_change":"2024-03-28T23:31:40.398876+0000","last_active":"2024-03-28T23:31:53.875554+0000","last_peered":"2024-03-28T23:31:53.875554+0000","last_clean":"2024-03-28T23:31:53.875554+0000","last_became_active":"2024-03-28T23:31:40.398595+0000","last_became_peered":"2024-03-28T23:31:40.398595+0000","last_unstale":"2024-03-28T23:31:53.875554+0000","last_undegraded":"2024-03-28T23:31:53.875554+0000","last_fullsized":"2024-03-28T23:31:53.875554+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:21.725238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874359+0000","last_change":"2024-03-28T23:31:53.874359+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874359+0000","last_undegraded":"2024-03-28T23:31:53.874359+0000","last_fullsized":"2024-03-28T23:31:53.874359+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.880118+0000","last_change":"2024-03-28T23:31:40.392421+0000","last_active":"2024-03-28T23:31:53.880118+0000","last_peered":"2024-03-28T23:31:53.880118+0000","last_clean":"2024-03-28T23:31:53.880118+0000","last_became_active":"2024-03-28T23:31:40.391463+0000","last_became_peered":"2024-03-28T23:31:40.391463+0000","last_unstale":"2024-03-28T23:31:53.880118+0000","last_undegraded":"2024-03-28T23:31:53.880118+0000","last_fullsized":"2024-03-28T23:31:53.880118+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:18:16.851435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878422+0000","last_change":"2024-03-28T23:31:53.878422+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878422+0000","last_undegraded":"2024-03-28T23:31:53.878422+0000","last_fullsized":"2024-03-28T23:31:53.878422+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.878936+0000","last_change":"2024-03-28T23:31:40.403340+0000","last_active":"2024-03-28T23:31:53.878936+0000","last_peered":"2024-03-28T23:31:53.878936+0000","last_clean":"2024-03-28T23:31:53.878936+0000","last_became_active":"2024-03-28T23:31:40.403089+0000","last_became_peered":"2024-03-28T23:31:40.403089+0000","last_unstale":"2024-03-28T23:31:53.878936+0000","last_undegraded":"2024-03-28T23:31:53.878936+0000","last_fullsized":"2024-03-28T23:31:53.878936+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:12:38.242108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.883968+0000","last_change":"2024-03-28T23:31:53.883968+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.883968+0000","last_undegraded":"2024-03-28T23:31:53.883968+0000","last_fullsized":"2024-03-28T23:31:53.883968+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.870177+0000","last_change":"2024-03-28T23:31:40.382204+0000","last_active":"2024-03-28T23:31:53.870177+0000","last_peered":"2024-03-28T23:31:53.870177+0000","last_clean":"2024-03-28T23:31:53.870177+0000","last_became_active":"2024-03-28T23:31:40.381944+0000","last_became_peered":"2024-03-28T23:31:40.381944+0000","last_unstale":"2024-03-28T23:31:53.870177+0000","last_undegraded":"2024-03-28T23:31:53.870177+0000","last_fullsized":"2024-03-28T23:31:53.870177+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:14.515069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871125+0000","last_change":"2024-03-28T23:31:53.871125+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871125+0000","last_undegraded":"2024-03-28T23:31:53.871125+0000","last_fullsized":"2024-03-28T23:31:53.871125+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.883144+0000","last_change":"2024-03-28T23:31:40.394578+0000","last_active":"2024-03-28T23:31:53.883144+0000","last_peered":"2024-03-28T23:31:53.883144+0000","last_clean":"2024-03-28T23:31:53.883144+0000","last_became_active":"2024-03-28T23:31:40.394272+0000","last_became_peered":"2024-03-28T23:31:40.394272+0000","last_unstale":"2024-03-28T23:31:53.883144+0000","last_undegraded":"2024-03-28T23:31:53.883144+0000","last_fullsized":"2024-03-28T23:31:53.883144+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:24:35.708796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871848+0000","last_change":"2024-03-28T23:31:53.871848+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871848+0000","last_undegraded":"2024-03-28T23:31:53.871848+0000","last_fullsized":"2024-03-28T23:31:53.871848+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879847+0000","last_change":"2024-03-28T23:31:40.396702+0000","last_active":"2024-03-28T23:31:53.879847+0000","last_peered":"2024-03-28T23:31:53.879847+0000","last_clean":"2024-03-28T23:31:53.879847+0000","last_became_active":"2024-03-28T23:31:40.395941+0000","last_became_peered":"2024-03-28T23:31:40.395941+0000","last_unstale":"2024-03-28T23:31:53.879847+0000","last_undegraded":"2024-03-28T23:31:53.879847+0000","last_fullsized":"2024-03-28T23:31:53.879847+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:07:13.690791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871018+0000","last_change":"2024-03-28T23:31:53.871018+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871018+0000","last_undegraded":"2024-03-28T23:31:53.871018+0000","last_fullsized":"2024-03-28T23:31:53.871018+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869713+0000","last_change":"2024-03-28T23:31:40.382947+0000","last_active":"2024-03-28T23:31:53.869713+0000","last_peered":"2024-03-28T23:31:53.869713+0000","last_clean":"2024-03-28T23:31:53.869713+0000","last_became_active":"2024-03-28T23:31:40.381893+0000","last_became_peered":"2024-03-28T23:31:40.381893+0000","last_unstale":"2024-03-28T23:31:53.869713+0000","last_undegraded":"2024-03-28T23:31:53.869713+0000","last_fullsized":"2024-03-28T23:31:53.869713+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:50.872886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.877447+0000","last_change":"2024-03-28T23:31:53.877447+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.877447+0000","last_undegraded":"2024-03-28T23:31:53.877447+0000","last_fullsized":"2024-03-28T23:31:53.877447+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875836+0000","last_change":"2024-03-28T23:31:40.405627+0000","last_active":"2024-03-28T23:31:53.875836+0000","last_peered":"2024-03-28T23:31:53.875836+0000","last_clean":"2024-03-28T23:31:53.875836+0000","last_became_active":"2024-03-28T23:31:40.405360+0000","last_became_peered":"2024-03-28T23:31:40.405360+0000","last_unstale":"2024-03-28T23:31:53.875836+0000","last_undegraded":"2024-03-28T23:31:53.875836+0000","last_fullsized":"2024-03-28T23:31:53.875836+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:06.567694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.881048+0000","last_change":"2024-03-28T23:31:53.881048+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.881048+0000","last_undegraded":"2024-03-28T23:31:53.881048+0000","last_fullsized":"2024-03-28T23:31:53.881048+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.873535+0000","last_change":"2024-03-28T23:31:40.396605+0000","last_active":"2024-03-28T23:31:53.873535+0000","last_peered":"2024-03-28T23:31:53.873535+0000","last_clean":"2024-03-28T23:31:53.873535+0000","last_became_active":"2024-03-28T23:31:40.396237+0000","last_became_peered":"2024-03-28T23:31:40.396237+0000","last_unstale":"2024-03-28T23:31:53.873535+0000","last_undegraded":"2024-03-28T23:31:53.873535+0000","last_fullsized":"2024-03-28T23:31:53.873535+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:14.144440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871402+0000","last_change":"2024-03-28T23:31:53.871402+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871402+0000","last_undegraded":"2024-03-28T23:31:53.871402+0000","last_fullsized":"2024-03-28T23:31:53.871402+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869477+0000","last_change":"2024-03-28T23:31:40.382823+0000","last_active":"2024-03-28T23:31:53.869477+0000","last_peered":"2024-03-28T23:31:53.869477+0000","last_clean":"2024-03-28T23:31:53.869477+0000","last_became_active":"2024-03-28T23:31:40.382574+0000","last_became_peered":"2024-03-28T23:31:40.382574+0000","last_unstale":"2024-03-28T23:31:53.869477+0000","last_undegraded":"2024-03-28T23:31:53.869477+0000","last_fullsized":"2024-03-28T23:31:53.869477+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:12:27.477720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872681+0000","last_change":"2024-03-28T23:31:53.872681+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872681+0000","last_undegraded":"2024-03-28T23:31:53.872681+0000","last_fullsized":"2024-03-28T23:31:53.872681+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876715+0000","last_change":"2024-03-28T23:31:40.403462+0000","last_active":"2024-03-28T23:31:53.876715+0000","last_peered":"2024-03-28T23:31:53.876715+0000","last_clean":"2024-03-28T23:31:53.876715+0000","last_became_active":"2024-03-28T23:31:40.402885+0000","last_became_peered":"2024-03-28T23:31:40.402885+0000","last_unstale":"2024-03-28T23:31:53.876715+0000","last_undegraded":"2024-03-28T23:31:53.876715+0000","last_fullsized":"2024-03-28T23:31:53.876715+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:42:29.351941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.879388+0000","last_change":"2024-03-28T23:31:53.879388+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.879388+0000","last_undegraded":"2024-03-28T23:31:53.879388+0000","last_fullsized":"2024-03-28T23:31:53.879388+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875969+0000","last_change":"2024-03-28T23:31:40.395478+0000","last_active":"2024-03-28T23:31:53.875969+0000","last_peered":"2024-03-28T23:31:53.875969+0000","last_clean":"2024-03-28T23:31:53.875969+0000","last_became_active":"2024-03-28T23:31:40.394981+0000","last_became_peered":"2024-03-28T23:31:40.394981+0000","last_unstale":"2024-03-28T23:31:53.875969+0000","last_undegraded":"2024-03-28T23:31:53.875969+0000","last_fullsized":"2024-03-28T23:31:53.875969+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:06.567694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872394+0000","last_change":"2024-03-28T23:31:53.872394+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872394+0000","last_undegraded":"2024-03-28T23:31:53.872394+0000","last_fullsized":"2024-03-28T23:31:53.872394+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876119+0000","last_change":"2024-03-28T23:31:40.394475+0000","last_active":"2024-03-28T23:31:53.876119+0000","last_peered":"2024-03-28T23:31:53.876119+0000","last_clean":"2024-03-28T23:31:53.876119+0000","last_became_active":"2024-03-28T23:31:40.390205+0000","last_became_peered":"2024-03-28T23:31:40.390205+0000","last_unstale":"2024-03-28T23:31:53.876119+0000","last_undegraded":"2024-03-28T23:31:53.876119+0000","last_fullsized":"2024-03-28T23:31:53.876119+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:04:48.226221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.882562+0000","last_change":"2024-03-28T23:31:53.882562+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.882562+0000","last_undegraded":"2024-03-28T23:31:53.882562+0000","last_fullsized":"2024-03-28T23:31:53.882562+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.877115+0000","last_change":"2024-03-28T23:31:40.388386+0000","last_active":"2024-03-28T23:31:53.877115+0000","last_peered":"2024-03-28T23:31:53.877115+0000","last_clean":"2024-03-28T23:31:53.877115+0000","last_became_active":"2024-03-28T23:31:40.388043+0000","last_became_peered":"2024-03-28T23:31:40.388043+0000","last_unstale":"2024-03-28T23:31:53.877115+0000","last_undegraded":"2024-03-28T23:31:53.877115+0000","last_fullsized":"2024-03-28T23:31:53.877115+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:07.738674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.880460+0000","last_change":"2024-03-28T23:31:53.880460+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.880460+0000","last_undegraded":"2024-03-28T23:31:53.880460+0000","last_fullsized":"2024-03-28T23:31:53.880460+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875638+0000","last_change":"2024-03-28T23:31:40.395811+0000","last_active":"2024-03-28T23:31:53.875638+0000","last_peered":"2024-03-28T23:31:53.875638+0000","last_clean":"2024-03-28T23:31:53.875638+0000","last_became_active":"2024-03-28T23:31:40.395540+0000","last_became_peered":"2024-03-28T23:31:40.395540+0000","last_unstale":"2024-03-28T23:31:53.875638+0000","last_undegraded":"2024-03-28T23:31:53.875638+0000","last_fullsized":"2024-03-28T23:31:53.875638+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:51.864986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.870958+0000","last_change":"2024-03-28T23:31:53.870958+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.870958+0000","last_undegraded":"2024-03-28T23:31:53.870958+0000","last_fullsized":"2024-03-28T23:31:53.870958+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.881134+0000","last_change":"2024-03-28T23:31:40.403129+0000","last_active":"2024-03-28T23:31:53.881134+0000","last_peered":"2024-03-28T23:31:53.881134+0000","last_clean":"2024-03-28T23:31:53.881134+0000","last_became_active":"2024-03-28T23:31:40.402879+0000","last_became_peered":"2024-03-28T23:31:40.402879+0000","last_unstale":"2024-03-28T23:31:53.881134+0000","last_undegraded":"2024-03-28T23:31:53.881134+0000","last_fullsized":"2024-03-28T23:31:53.881134+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:47.861300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.879206+0000","last_change":"2024-03-28T23:31:53.879206+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.879206+0000","last_undegraded":"2024-03-28T23:31:53.879206+0000","last_fullsized":"2024-03-28T23:31:53.879206+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879397+0000","last_change":"2024-03-28T23:31:40.389854+0000","last_active":"2024-03-28T23:31:53.879397+0000","last_peered":"2024-03-28T23:31:53.879397+0000","last_clean":"2024-03-28T23:31:53.879397+0000","last_became_active":"2024-03-28T23:31:40.389599+0000","last_became_peered":"2024-03-28T23:31:40.389599+0000","last_unstale":"2024-03-28T23:31:53.879397+0000","last_undegraded":"2024-03-28T23:31:53.879397+0000","last_fullsized":"2024-03-28T23:31:53.879397+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:43:37.883841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.880913+0000","last_change":"2024-03-28T23:31:53.880913+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.880913+0000","last_undegraded":"2024-03-28T23:31:53.880913+0000","last_fullsized":"2024-03-28T23:31:53.880913+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875417+0000","last_change":"2024-03-28T23:31:40.406317+0000","last_active":"2024-03-28T23:31:53.875417+0000","last_peered":"2024-03-28T23:31:53.875417+0000","last_clean":"2024-03-28T23:31:53.875417+0000","last_became_active":"2024-03-28T23:31:40.405217+0000","last_became_peered":"2024-03-28T23:31:40.405217+0000","last_unstale":"2024-03-28T23:31:53.875417+0000","last_undegraded":"2024-03-28T23:31:53.875417+0000","last_fullsized":"2024-03-28T23:31:53.875417+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:37:55.870956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.879370+0000","last_change":"2024-03-28T23:31:53.879370+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.879370+0000","last_undegraded":"2024-03-28T23:31:53.879370+0000","last_fullsized":"2024-03-28T23:31:53.879370+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876750+0000","last_change":"2024-03-28T23:31:40.404249+0000","last_active":"2024-03-28T23:31:53.876750+0000","last_peered":"2024-03-28T23:31:53.876750+0000","last_clean":"2024-03-28T23:31:53.876750+0000","last_became_active":"2024-03-28T23:31:40.403966+0000","last_became_peered":"2024-03-28T23:31:40.403966+0000","last_unstale":"2024-03-28T23:31:53.876750+0000","last_undegraded":"2024-03-28T23:31:53.876750+0000","last_fullsized":"2024-03-28T23:31:53.876750+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:40:57.541676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872876+0000","last_change":"2024-03-28T23:31:53.872876+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872876+0000","last_undegraded":"2024-03-28T23:31:53.872876+0000","last_fullsized":"2024-03-28T23:31:53.872876+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.878912+0000","last_change":"2024-03-28T23:31:40.397170+0000","last_active":"2024-03-28T23:31:53.878912+0000","last_peered":"2024-03-28T23:31:53.878912+0000","last_clean":"2024-03-28T23:31:53.878912+0000","last_became_active":"2024-03-28T23:31:40.396898+0000","last_became_peered":"2024-03-28T23:31:40.396898+0000","last_unstale":"2024-03-28T23:31:53.878912+0000","last_undegraded":"2024-03-28T23:31:53.878912+0000","last_fullsized":"2024-03-28T23:31:53.878912+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:53:26.355348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878874+0000","last_change":"2024-03-28T23:31:53.878874+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878874+0000","last_undegraded":"2024-03-28T23:31:53.878874+0000","last_fullsized":"2024-03-28T23:31:53.878874+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.870080+0000","last_change":"2024-03-28T23:31:40.404955+0000","last_active":"2024-03-28T23:31:53.870080+0000","last_peered":"2024-03-28T23:31:53.870080+0000","last_clean":"2024-03-28T23:31:53.870080+0000","last_became_active":"2024-03-28T23:31:40.404689+0000","last_became_peered":"2024-03-28T23:31:40.404689+0000","last_unstale":"2024-03-28T23:31:53.870080+0000","last_undegraded":"2024-03-28T23:31:53.870080+0000","last_fullsized":"2024-03-28T23:31:53.870080+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:50:20.574648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878430+0000","last_change":"2024-03-28T23:31:53.878430+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878430+0000","last_undegraded":"2024-03-28T23:31:53.878430+0000","last_fullsized":"2024-03-28T23:31:53.878430+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.877218+0000","last_change":"2024-03-28T23:31:40.389261+0000","last_active":"2024-03-28T23:31:53.877218+0000","last_peered":"2024-03-28T23:31:53.877218+0000","last_clean":"2024-03-28T23:31:53.877218+0000","last_became_active":"2024-03-28T23:31:40.388930+0000","last_became_peered":"2024-03-28T23:31:40.388930+0000","last_unstale":"2024-03-28T23:31:53.877218+0000","last_undegraded":"2024-03-28T23:31:53.877218+0000","last_fullsized":"2024-03-28T23:31:53.877218+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:23.215366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878783+0000","last_change":"2024-03-28T23:31:53.878783+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878783+0000","last_undegraded":"2024-03-28T23:31:53.878783+0000","last_fullsized":"2024-03-28T23:31:53.878783+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875578+0000","last_change":"2024-03-28T23:31:40.401673+0000","last_active":"2024-03-28T23:31:53.875578+0000","last_peered":"2024-03-28T23:31:53.875578+0000","last_clean":"2024-03-28T23:31:53.875578+0000","last_became_active":"2024-03-28T23:31:40.398923+0000","last_became_peered":"2024-03-28T23:31:40.398923+0000","last_unstale":"2024-03-28T23:31:53.875578+0000","last_undegraded":"2024-03-28T23:31:53.875578+0000","last_fullsized":"2024-03-28T23:31:53.875578+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:18.783120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874991+0000","last_change":"2024-03-28T23:31:53.874991+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874991+0000","last_undegraded":"2024-03-28T23:31:53.874991+0000","last_fullsized":"2024-03-28T23:31:53.874991+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.873933+0000","last_change":"2024-03-28T23:31:40.387082+0000","last_active":"2024-03-28T23:31:53.873933+0000","last_peered":"2024-03-28T23:31:53.873933+0000","last_clean":"2024-03-28T23:31:53.873933+0000","last_became_active":"2024-03-28T23:31:40.386808+0000","last_became_peered":"2024-03-28T23:31:40.386808+0000","last_unstale":"2024-03-28T23:31:53.873933+0000","last_undegraded":"2024-03-28T23:31:53.873933+0000","last_fullsized":"2024-03-28T23:31:53.873933+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:43.111058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.870965+0000","last_change":"2024-03-28T23:31:53.870965+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.870965+0000","last_undegraded":"2024-03-28T23:31:53.870965+0000","last_fullsized":"2024-03-28T23:31:53.870965+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875611+0000","last_change":"2024-03-28T23:31:40.397815+0000","last_active":"2024-03-28T23:31:53.875611+0000","last_peered":"2024-03-28T23:31:53.875611+0000","last_clean":"2024-03-28T23:31:53.875611+0000","last_became_active":"2024-03-28T23:31:40.397279+0000","last_became_peered":"2024-03-28T23:31:40.397279+0000","last_unstale":"2024-03-28T23:31:53.875611+0000","last_undegraded":"2024-03-28T23:31:53.875611+0000","last_fullsized":"2024-03-28T23:31:53.875611+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:25:57.449014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878422+0000","last_change":"2024-03-28T23:31:53.878422+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878422+0000","last_undegraded":"2024-03-28T23:31:53.878422+0000","last_fullsized":"2024-03-28T23:31:53.878422+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874691+0000","last_change":"2024-03-28T23:31:40.394484+0000","last_active":"2024-03-28T23:31:53.874691+0000","last_peered":"2024-03-28T23:31:53.874691+0000","last_clean":"2024-03-28T23:31:53.874691+0000","last_became_active":"2024-03-28T23:31:40.394219+0000","last_became_peered":"2024-03-28T23:31:40.394219+0000","last_unstale":"2024-03-28T23:31:53.874691+0000","last_undegraded":"2024-03-28T23:31:53.874691+0000","last_fullsized":"2024-03-28T23:31:53.874691+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:18.654331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874735+0000","last_change":"2024-03-28T23:31:53.874735+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874735+0000","last_undegraded":"2024-03-28T23:31:53.874735+0000","last_fullsized":"2024-03-28T23:31:53.874735+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891397+0000","last_change":"2024-03-28T23:31:53.890105+0000","last_active":"2024-03-28T23:31:53.891397+0000","last_peered":"2024-03-28T23:31:53.891397+0000","last_clean":"2024-03-28T23:31:53.891397+0000","last_became_active":"2024-03-28T23:31:53.889841+0000","last_became_peered":"2024-03-28T23:31:53.889841+0000","last_unstale":"2024-03-28T23:31:53.891397+0000","last_undegraded":"2024-03-28T23:31:53.891397+0000","last_fullsized":"2024-03-28T23:31:53.891397+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:01.360626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.872604+0000","last_change":"2024-03-28T23:31:40.398232+0000","last_active":"2024-03-28T23:31:53.872604+0000","last_peered":"2024-03-28T23:31:53.872604+0000","last_clean":"2024-03-28T23:31:53.872604+0000","last_became_active":"2024-03-28T23:31:40.397888+0000","last_became_peered":"2024-03-28T23:31:40.397888+0000","last_unstale":"2024-03-28T23:31:53.872604+0000","last_undegraded":"2024-03-28T23:31:53.872604+0000","last_fullsized":"2024-03-28T23:31:53.872604+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:50.872886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"50'4","reported_seq":22,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879785+0000","last_change":"2024-03-28T23:31:39.385488+0000","last_active":"2024-03-28T23:31:53.879785+0000","last_peered":"2024-03-28T23:31:53.879785+0000","last_clean":"2024-03-28T23:31:53.879785+0000","last_became_active":"2024-03-28T23:31:39.385211+0000","last_became_peered":"2024-03-28T23:31:39.385211+0000","last_unstale":"2024-03-28T23:31:53.879785+0000","last_undegraded":"2024-03-28T23:31:53.879785+0000","last_fullsized":"2024-03-28T23:31:53.879785+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:50:10.559743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872455+0000","last_change":"2024-03-28T23:31:53.872455+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872455+0000","last_undegraded":"2024-03-28T23:31:53.872455+0000","last_fullsized":"2024-03-28T23:31:53.872455+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891253+0000","last_change":"2024-03-28T23:31:53.887853+0000","last_active":"2024-03-28T23:31:53.891253+0000","last_peered":"2024-03-28T23:31:53.891253+0000","last_clean":"2024-03-28T23:31:53.891253+0000","last_became_active":"2024-03-28T23:31:53.887588+0000","last_became_peered":"2024-03-28T23:31:53.887588+0000","last_unstale":"2024-03-28T23:31:53.891253+0000","last_undegraded":"2024-03-28T23:31:53.891253+0000","last_fullsized":"2024-03-28T23:31:53.891253+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:28:42.680101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.873999+0000","last_change":"2024-03-28T23:31:40.390218+0000","last_active":"2024-03-28T23:31:53.873999+0000","last_peered":"2024-03-28T23:31:53.873999+0000","last_clean":"2024-03-28T23:31:53.873999+0000","last_became_active":"2024-03-28T23:31:40.389922+0000","last_became_peered":"2024-03-28T23:31:40.389922+0000","last_unstale":"2024-03-28T23:31:53.873999+0000","last_undegraded":"2024-03-28T23:31:53.873999+0000","last_fullsized":"2024-03-28T23:31:53.873999+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:14.515069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"50'1","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.870270+0000","last_change":"2024-03-28T23:31:39.393513+0000","last_active":"2024-03-28T23:31:53.870270+0000","last_peered":"2024-03-28T23:31:53.870270+0000","last_clean":"2024-03-28T23:31:53.870270+0000","last_became_active":"2024-03-28T23:31:39.393176+0000","last_became_peered":"2024-03-28T23:31:39.393176+0000","last_unstale":"2024-03-28T23:31:53.870270+0000","last_undegraded":"2024-03-28T23:31:53.870270+0000","last_fullsized":"2024-03-28T23:31:53.870270+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:29.240985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.880671+0000","last_change":"2024-03-28T23:31:53.880671+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.880671+0000","last_undegraded":"2024-03-28T23:31:53.880671+0000","last_fullsized":"2024-03-28T23:31:53.880671+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891931+0000","last_change":"2024-03-28T23:31:53.890699+0000","last_active":"2024-03-28T23:31:53.891931+0000","last_peered":"2024-03-28T23:31:53.891931+0000","last_clean":"2024-03-28T23:31:53.891931+0000","last_became_active":"2024-03-28T23:31:53.890300+0000","last_became_peered":"2024-03-28T23:31:53.890300+0000","last_unstale":"2024-03-28T23:31:53.891931+0000","last_undegraded":"2024-03-28T23:31:53.891931+0000","last_fullsized":"2024-03-28T23:31:53.891931+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:18:30.350761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875566+0000","last_change":"2024-03-28T23:31:40.397077+0000","last_active":"2024-03-28T23:31:53.875566+0000","last_peered":"2024-03-28T23:31:53.875566+0000","last_clean":"2024-03-28T23:31:53.875566+0000","last_became_active":"2024-03-28T23:31:40.395265+0000","last_became_peered":"2024-03-28T23:31:40.395265+0000","last_unstale":"2024-03-28T23:31:53.875566+0000","last_undegraded":"2024-03-28T23:31:53.875566+0000","last_fullsized":"2024-03-28T23:31:53.875566+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:04.829030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"50'2","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.870290+0000","last_change":"2024-03-28T23:31:39.391743+0000","last_active":"2024-03-28T23:31:53.870290+0000","last_peered":"2024-03-28T23:31:53.870290+0000","last_clean":"2024-03-28T23:31:53.870290+0000","last_became_active":"2024-03-28T23:31:39.391300+0000","last_became_peered":"2024-03-28T23:31:39.391300+0000","last_unstale":"2024-03-28T23:31:53.870290+0000","last_undegraded":"2024-03-28T23:31:53.870290+0000","last_fullsized":"2024-03-28T23:31:53.870290+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:12.505382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.877711+0000","last_change":"2024-03-28T23:31:53.877711+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.877711+0000","last_undegraded":"2024-03-28T23:31:53.877711+0000","last_fullsized":"2024-03-28T23:31:53.877711+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.889315+0000","last_change":"2024-03-28T23:31:53.886833+0000","last_active":"2024-03-28T23:31:53.889315+0000","last_peered":"2024-03-28T23:31:53.889315+0000","last_clean":"2024-03-28T23:31:53.889315+0000","last_became_active":"2024-03-28T23:31:53.886498+0000","last_became_peered":"2024-03-28T23:31:53.886498+0000","last_unstale":"2024-03-28T23:31:53.889315+0000","last_undegraded":"2024-03-28T23:31:53.889315+0000","last_fullsized":"2024-03-28T23:31:53.889315+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:26:16.561871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.881876+0000","last_change":"2024-03-28T23:31:40.392391+0000","last_active":"2024-03-28T23:31:53.881876+0000","last_peered":"2024-03-28T23:31:53.881876+0000","last_clean":"2024-03-28T23:31:53.881876+0000","last_became_active":"2024-03-28T23:31:40.392100+0000","last_became_peered":"2024-03-28T23:31:40.392100+0000","last_unstale":"2024-03-28T23:31:53.881876+0000","last_undegraded":"2024-03-28T23:31:53.881876+0000","last_fullsized":"2024-03-28T23:31:53.881876+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:34:38.237729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"50'2","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874448+0000","last_change":"2024-03-28T23:31:39.385430+0000","last_active":"2024-03-28T23:31:53.874448+0000","last_peered":"2024-03-28T23:31:53.874448+0000","last_clean":"2024-03-28T23:31:53.874448+0000","last_became_active":"2024-03-28T23:31:39.385134+0000","last_became_peered":"2024-03-28T23:31:39.385134+0000","last_unstale":"2024-03-28T23:31:53.874448+0000","last_undegraded":"2024-03-28T23:31:53.874448+0000","last_fullsized":"2024-03-28T23:31:53.874448+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:12:26.472924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.873288+0000","last_change":"2024-03-28T23:31:53.873288+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.873288+0000","last_undegraded":"2024-03-28T23:31:53.873288+0000","last_fullsized":"2024-03-28T23:31:53.873288+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891323+0000","last_change":"2024-03-28T23:31:53.889140+0000","last_active":"2024-03-28T23:31:53.891323+0000","last_peered":"2024-03-28T23:31:53.891323+0000","last_clean":"2024-03-28T23:31:53.891323+0000","last_became_active":"2024-03-28T23:31:53.888738+0000","last_became_peered":"2024-03-28T23:31:53.888738+0000","last_unstale":"2024-03-28T23:31:53.891323+0000","last_undegraded":"2024-03-28T23:31:53.891323+0000","last_fullsized":"2024-03-28T23:31:53.891323+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:05:01.725546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.885385+0000","last_change":"2024-03-28T23:31:40.392757+0000","last_active":"2024-03-28T23:31:53.885385+0000","last_peered":"2024-03-28T23:31:53.885385+0000","last_clean":"2024-03-28T23:31:53.885385+0000","last_became_active":"2024-03-28T23:31:40.392460+0000","last_became_peered":"2024-03-28T23:31:40.392460+0000","last_unstale":"2024-03-28T23:31:53.885385+0000","last_undegraded":"2024-03-28T23:31:53.885385+0000","last_fullsized":"2024-03-28T23:31:53.885385+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:07.738674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"50'1","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875684+0000","last_change":"2024-03-28T23:31:39.381607+0000","last_active":"2024-03-28T23:31:53.875684+0000","last_peered":"2024-03-28T23:31:53.875684+0000","last_clean":"2024-03-28T23:31:53.875684+0000","last_became_active":"2024-03-28T23:31:39.381326+0000","last_became_peered":"2024-03-28T23:31:39.381326+0000","last_unstale":"2024-03-28T23:31:53.875684+0000","last_undegraded":"2024-03-28T23:31:53.875684+0000","last_fullsized":"2024-03-28T23:31:53.875684+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:50.860190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.880191+0000","last_change":"2024-03-28T23:31:53.880191+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.880191+0000","last_undegraded":"2024-03-28T23:31:53.880191+0000","last_fullsized":"2024-03-28T23:31:53.880191+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.888447+0000","last_change":"2024-03-28T23:31:53.886424+0000","last_active":"2024-03-28T23:31:53.888447+0000","last_peered":"2024-03-28T23:31:53.888447+0000","last_clean":"2024-03-28T23:31:53.888447+0000","last_became_active":"2024-03-28T23:31:53.886207+0000","last_became_peered":"2024-03-28T23:31:53.886207+0000","last_unstale":"2024-03-28T23:31:53.888447+0000","last_undegraded":"2024-03-28T23:31:53.888447+0000","last_fullsized":"2024-03-28T23:31:53.888447+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:26:10.948340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876571+0000","last_change":"2024-03-28T23:31:40.398712+0000","last_active":"2024-03-28T23:31:53.876571+0000","last_peered":"2024-03-28T23:31:53.876571+0000","last_clean":"2024-03-28T23:31:53.876571+0000","last_became_active":"2024-03-28T23:31:40.398392+0000","last_became_peered":"2024-03-28T23:31:40.398392+0000","last_unstale":"2024-03-28T23:31:53.876571+0000","last_undegraded":"2024-03-28T23:31:53.876571+0000","last_fullsized":"2024-03-28T23:31:53.876571+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:54.431044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"50'1","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.883791+0000","last_change":"2024-03-28T23:31:39.384026+0000","last_active":"2024-03-28T23:31:53.883791+0000","last_peered":"2024-03-28T23:31:53.883791+0000","last_clean":"2024-03-28T23:31:53.883791+0000","last_became_active":"2024-03-28T23:31:39.383161+0000","last_became_peered":"2024-03-28T23:31:39.383161+0000","last_unstale":"2024-03-28T23:31:53.883791+0000","last_undegraded":"2024-03-28T23:31:53.883791+0000","last_fullsized":"2024-03-28T23:31:53.883791+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:21:00.890266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872423+0000","last_change":"2024-03-28T23:31:53.872423+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872423+0000","last_undegraded":"2024-03-28T23:31:53.872423+0000","last_fullsized":"2024-03-28T23:31:53.872423+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.888932+0000","last_change":"2024-03-28T23:31:53.885914+0000","last_active":"2024-03-28T23:31:53.888932+0000","last_peered":"2024-03-28T23:31:53.888932+0000","last_clean":"2024-03-28T23:31:53.888932+0000","last_became_active":"2024-03-28T23:31:53.885637+0000","last_became_peered":"2024-03-28T23:31:53.885637+0000","last_unstale":"2024-03-28T23:31:53.888932+0000","last_undegraded":"2024-03-28T23:31:53.888932+0000","last_fullsized":"2024-03-28T23:31:53.888932+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:32.282446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879100+0000","last_change":"2024-03-28T23:31:40.395728+0000","last_active":"2024-03-28T23:31:53.879100+0000","last_peered":"2024-03-28T23:31:53.879100+0000","last_clean":"2024-03-28T23:31:53.879100+0000","last_became_active":"2024-03-28T23:31:40.395474+0000","last_became_peered":"2024-03-28T23:31:40.395474+0000","last_unstale":"2024-03-28T23:31:53.879100+0000","last_undegraded":"2024-03-28T23:31:53.879100+0000","last_fullsized":"2024-03-28T23:31:53.879100+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:15:19.985376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876912+0000","last_change":"2024-03-28T23:31:39.385649+0000","last_active":"2024-03-28T23:31:53.876912+0000","last_peered":"2024-03-28T23:31:53.876912+0000","last_clean":"2024-03-28T23:31:53.876912+0000","last_became_active":"2024-03-28T23:31:39.385341+0000","last_became_peered":"2024-03-28T23:31:39.385341+0000","last_unstale":"2024-03-28T23:31:53.876912+0000","last_undegraded":"2024-03-28T23:31:53.876912+0000","last_fullsized":"2024-03-28T23:31:53.876912+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:52:06.955653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878702+0000","last_change":"2024-03-28T23:31:53.878702+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878702+0000","last_undegraded":"2024-03-28T23:31:53.878702+0000","last_fullsized":"2024-03-28T23:31:53.878702+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891992+0000","last_change":"2024-03-28T23:31:53.889258+0000","last_active":"2024-03-28T23:31:53.891992+0000","last_peered":"2024-03-28T23:31:53.891992+0000","last_clean":"2024-03-28T23:31:53.891992+0000","last_became_active":"2024-03-28T23:31:53.888970+0000","last_became_peered":"2024-03-28T23:31:53.888970+0000","last_unstale":"2024-03-28T23:31:53.891992+0000","last_undegraded":"2024-03-28T23:31:53.891992+0000","last_fullsized":"2024-03-28T23:31:53.891992+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:20:01.487078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875176+0000","last_change":"2024-03-28T23:31:40.400091+0000","last_active":"2024-03-28T23:31:53.875176+0000","last_peered":"2024-03-28T23:31:53.875176+0000","last_clean":"2024-03-28T23:31:53.875176+0000","last_became_active":"2024-03-28T23:31:40.399190+0000","last_became_peered":"2024-03-28T23:31:40.399190+0000","last_unstale":"2024-03-28T23:31:53.875176+0000","last_undegraded":"2024-03-28T23:31:53.875176+0000","last_fullsized":"2024-03-28T23:31:53.875176+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:22.230986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875772+0000","last_change":"2024-03-28T23:31:39.390198+0000","last_active":"2024-03-28T23:31:53.875772+0000","last_peered":"2024-03-28T23:31:53.875772+0000","last_clean":"2024-03-28T23:31:53.875772+0000","last_became_active":"2024-03-28T23:31:39.389939+0000","last_became_peered":"2024-03-28T23:31:39.389939+0000","last_unstale":"2024-03-28T23:31:53.875772+0000","last_undegraded":"2024-03-28T23:31:53.875772+0000","last_fullsized":"2024-03-28T23:31:53.875772+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:46:30.350009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872121+0000","last_change":"2024-03-28T23:31:53.872121+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872121+0000","last_undegraded":"2024-03-28T23:31:53.872121+0000","last_fullsized":"2024-03-28T23:31:53.872121+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.896042+0000","last_change":"2024-03-28T23:31:53.892143+0000","last_active":"2024-03-28T23:31:53.896042+0000","last_peered":"2024-03-28T23:31:53.896042+0000","last_clean":"2024-03-28T23:31:53.896042+0000","last_became_active":"2024-03-28T23:31:53.891989+0000","last_became_peered":"2024-03-28T23:31:53.891989+0000","last_unstale":"2024-03-28T23:31:53.896042+0000","last_undegraded":"2024-03-28T23:31:53.896042+0000","last_fullsized":"2024-03-28T23:31:53.896042+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:23:27.129147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.883626+0000","last_change":"2024-03-28T23:31:40.384948+0000","last_active":"2024-03-28T23:31:53.883626+0000","last_peered":"2024-03-28T23:31:53.883626+0000","last_clean":"2024-03-28T23:31:53.883626+0000","last_became_active":"2024-03-28T23:31:40.384434+0000","last_became_peered":"2024-03-28T23:31:40.384434+0000","last_unstale":"2024-03-28T23:31:53.883626+0000","last_undegraded":"2024-03-28T23:31:53.883626+0000","last_fullsized":"2024-03-28T23:31:53.883626+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:04.829030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875277+0000","last_change":"2024-03-28T23:31:39.388914+0000","last_active":"2024-03-28T23:31:53.875277+0000","last_peered":"2024-03-28T23:31:53.875277+0000","last_clean":"2024-03-28T23:31:53.875277+0000","last_became_active":"2024-03-28T23:31:39.388639+0000","last_became_peered":"2024-03-28T23:31:39.388639+0000","last_unstale":"2024-03-28T23:31:53.875277+0000","last_undegraded":"2024-03-28T23:31:53.875277+0000","last_fullsized":"2024-03-28T23:31:53.875277+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:00.366278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.880894+0000","last_change":"2024-03-28T23:31:53.880894+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.880894+0000","last_undegraded":"2024-03-28T23:31:53.880894+0000","last_fullsized":"2024-03-28T23:31:53.880894+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.884484+0000","last_change":"2024-03-28T23:31:53.879950+0000","last_active":"2024-03-28T23:31:53.884484+0000","last_peered":"2024-03-28T23:31:53.884484+0000","last_clean":"2024-03-28T23:31:53.884484+0000","last_became_active":"2024-03-28T23:31:53.879620+0000","last_became_peered":"2024-03-28T23:31:53.879620+0000","last_unstale":"2024-03-28T23:31:53.884484+0000","last_undegraded":"2024-03-28T23:31:53.884484+0000","last_fullsized":"2024-03-28T23:31:53.884484+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:28:42.680101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.873900+0000","last_change":"2024-03-28T23:31:40.386749+0000","last_active":"2024-03-28T23:31:53.873900+0000","last_peered":"2024-03-28T23:31:53.873900+0000","last_clean":"2024-03-28T23:31:53.873900+0000","last_became_active":"2024-03-28T23:31:40.386466+0000","last_became_peered":"2024-03-28T23:31:40.386466+0000","last_unstale":"2024-03-28T23:31:53.873900+0000","last_undegraded":"2024-03-28T23:31:53.873900+0000","last_fullsized":"2024-03-28T23:31:53.873900+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:06.567694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.884919+0000","last_change":"2024-03-28T23:31:39.381050+0000","last_active":"2024-03-28T23:31:53.884919+0000","last_peered":"2024-03-28T23:31:53.884919+0000","last_clean":"2024-03-28T23:31:53.884919+0000","last_became_active":"2024-03-28T23:31:39.380761+0000","last_became_peered":"2024-03-28T23:31:39.380761+0000","last_unstale":"2024-03-28T23:31:53.884919+0000","last_undegraded":"2024-03-28T23:31:53.884919+0000","last_fullsized":"2024-03-28T23:31:53.884919+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:29.240985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878283+0000","last_change":"2024-03-28T23:31:53.878283+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878283+0000","last_undegraded":"2024-03-28T23:31:53.878283+0000","last_fullsized":"2024-03-28T23:31:53.878283+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.893113+0000","last_change":"2024-03-28T23:31:53.889906+0000","last_active":"2024-03-28T23:31:53.893113+0000","last_peered":"2024-03-28T23:31:53.893113+0000","last_clean":"2024-03-28T23:31:53.893113+0000","last_became_active":"2024-03-28T23:31:53.889589+0000","last_became_peered":"2024-03-28T23:31:53.889589+0000","last_unstale":"2024-03-28T23:31:53.893113+0000","last_undegraded":"2024-03-28T23:31:53.893113+0000","last_fullsized":"2024-03-28T23:31:53.893113+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:17:43.674083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875914+0000","last_change":"2024-03-28T23:31:40.400388+0000","last_active":"2024-03-28T23:31:53.875914+0000","last_peered":"2024-03-28T23:31:53.875914+0000","last_clean":"2024-03-28T23:31:53.875914+0000","last_became_active":"2024-03-28T23:31:40.400133+0000","last_became_peered":"2024-03-28T23:31:40.400133+0000","last_unstale":"2024-03-28T23:31:53.875914+0000","last_undegraded":"2024-03-28T23:31:53.875914+0000","last_fullsized":"2024-03-28T23:31:53.875914+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:03:56.252539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"50'3","reported_seq":21,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876731+0000","last_change":"2024-03-28T23:31:39.386478+0000","last_active":"2024-03-28T23:31:53.876731+0000","last_peered":"2024-03-28T23:31:53.876731+0000","last_clean":"2024-03-28T23:31:53.876731+0000","last_became_active":"2024-03-28T23:31:39.385868+0000","last_became_peered":"2024-03-28T23:31:39.385868+0000","last_unstale":"2024-03-28T23:31:53.876731+0000","last_undegraded":"2024-03-28T23:31:53.876731+0000","last_fullsized":"2024-03-28T23:31:53.876731+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:06.733879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872413+0000","last_change":"2024-03-28T23:31:53.872413+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872413+0000","last_undegraded":"2024-03-28T23:31:53.872413+0000","last_fullsized":"2024-03-28T23:31:53.872413+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.893021+0000","last_change":"2024-03-28T23:31:53.889887+0000","last_active":"2024-03-28T23:31:53.893021+0000","last_peered":"2024-03-28T23:31:53.893021+0000","last_clean":"2024-03-28T23:31:53.893021+0000","last_became_active":"2024-03-28T23:31:53.889635+0000","last_became_peered":"2024-03-28T23:31:53.889635+0000","last_unstale":"2024-03-28T23:31:53.893021+0000","last_undegraded":"2024-03-28T23:31:53.893021+0000","last_fullsized":"2024-03-28T23:31:53.893021+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:19:23.382549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.884161+0000","last_change":"2024-03-28T23:31:40.387795+0000","last_active":"2024-03-28T23:31:53.884161+0000","last_peered":"2024-03-28T23:31:53.884161+0000","last_clean":"2024-03-28T23:31:53.884161+0000","last_became_active":"2024-03-28T23:31:40.387529+0000","last_became_peered":"2024-03-28T23:31:40.387529+0000","last_unstale":"2024-03-28T23:31:53.884161+0000","last_undegraded":"2024-03-28T23:31:53.884161+0000","last_fullsized":"2024-03-28T23:31:53.884161+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:51.864986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"50'6","reported_seq":24,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879943+0000","last_change":"2024-03-28T23:31:39.393411+0000","last_active":"2024-03-28T23:31:53.879943+0000","last_peered":"2024-03-28T23:31:53.879943+0000","last_clean":"2024-03-28T23:31:53.879943+0000","last_became_active":"2024-03-28T23:31:39.393166+0000","last_became_peered":"2024-03-28T23:31:39.393166+0000","last_unstale":"2024-03-28T23:31:53.879943+0000","last_undegraded":"2024-03-28T23:31:53.879943+0000","last_fullsized":"2024-03-28T23:31:53.879943+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:13.510273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874577+0000","last_change":"2024-03-28T23:31:53.874577+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874577+0000","last_undegraded":"2024-03-28T23:31:53.874577+0000","last_fullsized":"2024-03-28T23:31:53.874577+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.890522+0000","last_change":"2024-03-28T23:31:53.889399+0000","last_active":"2024-03-28T23:31:53.890522+0000","last_peered":"2024-03-28T23:31:53.890522+0000","last_clean":"2024-03-28T23:31:53.890522+0000","last_became_active":"2024-03-28T23:31:53.889110+0000","last_became_peered":"2024-03-28T23:31:53.889110+0000","last_unstale":"2024-03-28T23:31:53.890522+0000","last_undegraded":"2024-03-28T23:31:53.890522+0000","last_fullsized":"2024-03-28T23:31:53.890522+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:35:48.231940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874249+0000","last_change":"2024-03-28T23:31:40.390497+0000","last_active":"2024-03-28T23:31:53.874249+0000","last_peered":"2024-03-28T23:31:53.874249+0000","last_clean":"2024-03-28T23:31:53.874249+0000","last_became_active":"2024-03-28T23:31:40.390167+0000","last_became_peered":"2024-03-28T23:31:40.390167+0000","last_unstale":"2024-03-28T23:31:53.874249+0000","last_undegraded":"2024-03-28T23:31:53.874249+0000","last_fullsized":"2024-03-28T23:31:53.874249+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:54.431044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"50'2","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875733+0000","last_change":"2024-03-28T23:31:39.380369+0000","last_active":"2024-03-28T23:31:53.875733+0000","last_peered":"2024-03-28T23:31:53.875733+0000","last_clean":"2024-03-28T23:31:53.875733+0000","last_became_active":"2024-03-28T23:31:39.380092+0000","last_became_peered":"2024-03-28T23:31:39.380092+0000","last_unstale":"2024-03-28T23:31:53.875733+0000","last_undegraded":"2024-03-28T23:31:53.875733+0000","last_fullsized":"2024-03-28T23:31:53.875733+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:00.366278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871676+0000","last_change":"2024-03-28T23:31:53.871676+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871676+0000","last_undegraded":"2024-03-28T23:31:53.871676+0000","last_fullsized":"2024-03-28T23:31:53.871676+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.885629+0000","last_change":"2024-03-28T23:31:53.883362+0000","last_active":"2024-03-28T23:31:53.885629+0000","last_peered":"2024-03-28T23:31:53.885629+0000","last_clean":"2024-03-28T23:31:53.885629+0000","last_became_active":"2024-03-28T23:31:53.883123+0000","last_became_peered":"2024-03-28T23:31:53.883123+0000","last_unstale":"2024-03-28T23:31:53.885629+0000","last_undegraded":"2024-03-28T23:31:53.885629+0000","last_fullsized":"2024-03-28T23:31:53.885629+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:16.175508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876210+0000","last_change":"2024-03-28T23:31:40.390579+0000","last_active":"2024-03-28T23:31:53.876210+0000","last_peered":"2024-03-28T23:31:53.876210+0000","last_clean":"2024-03-28T23:31:53.876210+0000","last_became_active":"2024-03-28T23:31:40.390248+0000","last_became_peered":"2024-03-28T23:31:40.390248+0000","last_unstale":"2024-03-28T23:31:53.876210+0000","last_undegraded":"2024-03-28T23:31:53.876210+0000","last_fullsized":"2024-03-28T23:31:53.876210+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:47.861300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875729+0000","last_change":"2024-03-28T23:31:39.389592+0000","last_active":"2024-03-28T23:31:53.875729+0000","last_peered":"2024-03-28T23:31:53.875729+0000","last_clean":"2024-03-28T23:31:53.875729+0000","last_became_active":"2024-03-28T23:31:39.389302+0000","last_became_peered":"2024-03-28T23:31:39.389302+0000","last_unstale":"2024-03-28T23:31:53.875729+0000","last_undegraded":"2024-03-28T23:31:53.875729+0000","last_fullsized":"2024-03-28T23:31:53.875729+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:59:51.501145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872186+0000","last_change":"2024-03-28T23:31:53.872186+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872186+0000","last_undegraded":"2024-03-28T23:31:53.872186+0000","last_fullsized":"2024-03-28T23:31:53.872186+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.884324+0000","last_change":"2024-03-28T23:31:53.881795+0000","last_active":"2024-03-28T23:31:53.884324+0000","last_peered":"2024-03-28T23:31:53.884324+0000","last_clean":"2024-03-28T23:31:53.884324+0000","last_became_active":"2024-03-28T23:31:53.881552+0000","last_became_peered":"2024-03-28T23:31:53.881552+0000","last_unstale":"2024-03-28T23:31:53.884324+0000","last_undegraded":"2024-03-28T23:31:53.884324+0000","last_fullsized":"2024-03-28T23:31:53.884324+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:35.224564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879015+0000","last_change":"2024-03-28T23:31:40.387085+0000","last_active":"2024-03-28T23:31:53.879015+0000","last_peered":"2024-03-28T23:31:53.879015+0000","last_clean":"2024-03-28T23:31:53.879015+0000","last_became_active":"2024-03-28T23:31:40.386695+0000","last_became_peered":"2024-03-28T23:31:40.386695+0000","last_unstale":"2024-03-28T23:31:53.879015+0000","last_undegraded":"2024-03-28T23:31:53.879015+0000","last_fullsized":"2024-03-28T23:31:53.879015+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:40:57.541676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.873632+0000","last_change":"2024-03-28T23:31:39.384653+0000","last_active":"2024-03-28T23:31:53.873632+0000","last_peered":"2024-03-28T23:31:53.873632+0000","last_clean":"2024-03-28T23:31:53.873632+0000","last_became_active":"2024-03-28T23:31:39.383495+0000","last_became_peered":"2024-03-28T23:31:39.383495+0000","last_unstale":"2024-03-28T23:31:53.873632+0000","last_undegraded":"2024-03-28T23:31:53.873632+0000","last_fullsized":"2024-03-28T23:31:53.873632+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:50.860190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.878941+0000","last_change":"2024-03-28T23:31:53.878941+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.878941+0000","last_undegraded":"2024-03-28T23:31:53.878941+0000","last_fullsized":"2024-03-28T23:31:53.878941+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.886195+0000","last_change":"2024-03-28T23:31:53.884651+0000","last_active":"2024-03-28T23:31:53.886195+0000","last_peered":"2024-03-28T23:31:53.886195+0000","last_clean":"2024-03-28T23:31:53.886195+0000","last_became_active":"2024-03-28T23:31:53.884410+0000","last_became_peered":"2024-03-28T23:31:53.884410+0000","last_unstale":"2024-03-28T23:31:53.886195+0000","last_undegraded":"2024-03-28T23:31:53.886195+0000","last_fullsized":"2024-03-28T23:31:53.886195+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:31:41.553370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876561+0000","last_change":"2024-03-28T23:31:40.403701+0000","last_active":"2024-03-28T23:31:53.876561+0000","last_peered":"2024-03-28T23:31:53.876561+0000","last_clean":"2024-03-28T23:31:53.876561+0000","last_became_active":"2024-03-28T23:31:40.403222+0000","last_became_peered":"2024-03-28T23:31:40.403222+0000","last_unstale":"2024-03-28T23:31:53.876561+0000","last_undegraded":"2024-03-28T23:31:53.876561+0000","last_fullsized":"2024-03-28T23:31:53.876561+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:40:09.867914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"50'8","reported_seq":26,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.880291+0000","last_change":"2024-03-28T23:31:39.383726+0000","last_active":"2024-03-28T23:31:53.880291+0000","last_peered":"2024-03-28T23:31:53.880291+0000","last_clean":"2024-03-28T23:31:53.880291+0000","last_became_active":"2024-03-28T23:31:39.383292+0000","last_became_peered":"2024-03-28T23:31:39.383292+0000","last_unstale":"2024-03-28T23:31:53.880291+0000","last_undegraded":"2024-03-28T23:31:53.880291+0000","last_fullsized":"2024-03-28T23:31:53.880291+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:25:38.113638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872289+0000","last_change":"2024-03-28T23:31:53.872289+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872289+0000","last_undegraded":"2024-03-28T23:31:53.872289+0000","last_fullsized":"2024-03-28T23:31:53.872289+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.896055+0000","last_change":"2024-03-28T23:31:53.892246+0000","last_active":"2024-03-28T23:31:53.896055+0000","last_peered":"2024-03-28T23:31:53.896055+0000","last_clean":"2024-03-28T23:31:53.896055+0000","last_became_active":"2024-03-28T23:31:53.891974+0000","last_became_peered":"2024-03-28T23:31:53.891974+0000","last_unstale":"2024-03-28T23:31:53.896055+0000","last_undegraded":"2024-03-28T23:31:53.896055+0000","last_fullsized":"2024-03-28T23:31:53.896055+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:10:22.963954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.880205+0000","last_change":"2024-03-28T23:31:40.391219+0000","last_active":"2024-03-28T23:31:53.880205+0000","last_peered":"2024-03-28T23:31:53.880205+0000","last_clean":"2024-03-28T23:31:53.880205+0000","last_became_active":"2024-03-28T23:31:40.389552+0000","last_became_peered":"2024-03-28T23:31:40.389552+0000","last_unstale":"2024-03-28T23:31:53.880205+0000","last_undegraded":"2024-03-28T23:31:53.880205+0000","last_fullsized":"2024-03-28T23:31:53.880205+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:42:29.351941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"50'1","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876797+0000","last_change":"2024-03-28T23:31:39.390253+0000","last_active":"2024-03-28T23:31:53.876797+0000","last_peered":"2024-03-28T23:31:53.876797+0000","last_clean":"2024-03-28T23:31:53.876797+0000","last_became_active":"2024-03-28T23:31:39.389980+0000","last_became_peered":"2024-03-28T23:31:39.389980+0000","last_unstale":"2024-03-28T23:31:53.876797+0000","last_undegraded":"2024-03-28T23:31:53.876797+0000","last_fullsized":"2024-03-28T23:31:53.876797+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:24.520296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.877420+0000","last_change":"2024-03-28T23:31:53.877420+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.877420+0000","last_undegraded":"2024-03-28T23:31:53.877420+0000","last_fullsized":"2024-03-28T23:31:53.877420+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.884332+0000","last_change":"2024-03-28T23:31:53.880659+0000","last_active":"2024-03-28T23:31:53.884332+0000","last_peered":"2024-03-28T23:31:53.884332+0000","last_clean":"2024-03-28T23:31:53.884332+0000","last_became_active":"2024-03-28T23:31:53.880265+0000","last_became_peered":"2024-03-28T23:31:53.880265+0000","last_unstale":"2024-03-28T23:31:53.884332+0000","last_undegraded":"2024-03-28T23:31:53.884332+0000","last_fullsized":"2024-03-28T23:31:53.884332+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:04:09.751864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869503+0000","last_change":"2024-03-28T23:31:40.391898+0000","last_active":"2024-03-28T23:31:53.869503+0000","last_peered":"2024-03-28T23:31:53.869503+0000","last_clean":"2024-03-28T23:31:53.869503+0000","last_became_active":"2024-03-28T23:31:40.391622+0000","last_became_peered":"2024-03-28T23:31:40.391622+0000","last_unstale":"2024-03-28T23:31:53.869503+0000","last_undegraded":"2024-03-28T23:31:53.869503+0000","last_fullsized":"2024-03-28T23:31:53.869503+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:38:22.055287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"50'2","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.883203+0000","last_change":"2024-03-28T23:31:39.385339+0000","last_active":"2024-03-28T23:31:53.883203+0000","last_peered":"2024-03-28T23:31:53.883203+0000","last_clean":"2024-03-28T23:31:53.883203+0000","last_became_active":"2024-03-28T23:31:39.385075+0000","last_became_peered":"2024-03-28T23:31:39.385075+0000","last_unstale":"2024-03-28T23:31:53.883203+0000","last_undegraded":"2024-03-28T23:31:53.883203+0000","last_fullsized":"2024-03-28T23:31:53.883203+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:58.098710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.873793+0000","last_change":"2024-03-28T23:31:53.873793+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.873793+0000","last_undegraded":"2024-03-28T23:31:53.873793+0000","last_fullsized":"2024-03-28T23:31:53.873793+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891950+0000","last_change":"2024-03-28T23:31:53.890901+0000","last_active":"2024-03-28T23:31:53.891950+0000","last_peered":"2024-03-28T23:31:53.891950+0000","last_clean":"2024-03-28T23:31:53.891950+0000","last_became_active":"2024-03-28T23:31:53.890643+0000","last_became_peered":"2024-03-28T23:31:53.890643+0000","last_unstale":"2024-03-28T23:31:53.891950+0000","last_undegraded":"2024-03-28T23:31:53.891950+0000","last_fullsized":"2024-03-28T23:31:53.891950+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:07:27.190116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.883208+0000","last_change":"2024-03-28T23:31:40.399291+0000","last_active":"2024-03-28T23:31:53.883208+0000","last_peered":"2024-03-28T23:31:53.883208+0000","last_clean":"2024-03-28T23:31:53.883208+0000","last_became_active":"2024-03-28T23:31:40.399005+0000","last_became_peered":"2024-03-28T23:31:40.399005+0000","last_unstale":"2024-03-28T23:31:53.883208+0000","last_undegraded":"2024-03-28T23:31:53.883208+0000","last_fullsized":"2024-03-28T23:31:53.883208+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:01.371074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"50'1","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876296+0000","last_change":"2024-03-28T23:31:39.389465+0000","last_active":"2024-03-28T23:31:53.876296+0000","last_peered":"2024-03-28T23:31:53.876296+0000","last_clean":"2024-03-28T23:31:53.876296+0000","last_became_active":"2024-03-28T23:31:39.389005+0000","last_became_peered":"2024-03-28T23:31:39.389005+0000","last_unstale":"2024-03-28T23:31:53.876296+0000","last_undegraded":"2024-03-28T23:31:53.876296+0000","last_fullsized":"2024-03-28T23:31:53.876296+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:41.430571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.879255+0000","last_change":"2024-03-28T23:31:53.879255+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.879255+0000","last_undegraded":"2024-03-28T23:31:53.879255+0000","last_fullsized":"2024-03-28T23:31:53.879255+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.890716+0000","last_change":"2024-03-28T23:31:53.888704+0000","last_active":"2024-03-28T23:31:53.890716+0000","last_peered":"2024-03-28T23:31:53.890716+0000","last_clean":"2024-03-28T23:31:53.890716+0000","last_became_active":"2024-03-28T23:31:53.888461+0000","last_became_peered":"2024-03-28T23:31:53.888461+0000","last_unstale":"2024-03-28T23:31:53.890716+0000","last_undegraded":"2024-03-28T23:31:53.890716+0000","last_fullsized":"2024-03-28T23:31:53.890716+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:08.894103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869795+0000","last_change":"2024-03-28T23:31:40.388141+0000","last_active":"2024-03-28T23:31:53.869795+0000","last_peered":"2024-03-28T23:31:53.869795+0000","last_clean":"2024-03-28T23:31:53.869795+0000","last_became_active":"2024-03-28T23:31:40.387852+0000","last_became_peered":"2024-03-28T23:31:40.387852+0000","last_unstale":"2024-03-28T23:31:53.869795+0000","last_undegraded":"2024-03-28T23:31:53.869795+0000","last_fullsized":"2024-03-28T23:31:53.869795+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:22.230986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.877311+0000","last_change":"2024-03-28T23:31:39.390465+0000","last_active":"2024-03-28T23:31:53.877311+0000","last_peered":"2024-03-28T23:31:53.877311+0000","last_clean":"2024-03-28T23:31:53.877311+0000","last_became_active":"2024-03-28T23:31:39.390203+0000","last_became_peered":"2024-03-28T23:31:39.390203+0000","last_unstale":"2024-03-28T23:31:53.877311+0000","last_undegraded":"2024-03-28T23:31:53.877311+0000","last_fullsized":"2024-03-28T23:31:53.877311+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:50:50.496294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.876579+0000","last_change":"2024-03-28T23:31:53.876579+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.876579+0000","last_undegraded":"2024-03-28T23:31:53.876579+0000","last_fullsized":"2024-03-28T23:31:53.876579+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.890502+0000","last_change":"2024-03-28T23:31:53.888267+0000","last_active":"2024-03-28T23:31:53.890502+0000","last_peered":"2024-03-28T23:31:53.890502+0000","last_clean":"2024-03-28T23:31:53.890502+0000","last_became_active":"2024-03-28T23:31:53.887993+0000","last_became_peered":"2024-03-28T23:31:53.887993+0000","last_unstale":"2024-03-28T23:31:53.890502+0000","last_undegraded":"2024-03-28T23:31:53.890502+0000","last_fullsized":"2024-03-28T23:31:53.890502+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:21:27.713981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874357+0000","last_change":"2024-03-28T23:31:40.387376+0000","last_active":"2024-03-28T23:31:53.874357+0000","last_peered":"2024-03-28T23:31:53.874357+0000","last_clean":"2024-03-28T23:31:53.874357+0000","last_became_active":"2024-03-28T23:31:40.387080+0000","last_became_peered":"2024-03-28T23:31:40.387080+0000","last_unstale":"2024-03-28T23:31:53.874357+0000","last_undegraded":"2024-03-28T23:31:53.874357+0000","last_fullsized":"2024-03-28T23:31:53.874357+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:43.111058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"50'2","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869894+0000","last_change":"2024-03-28T23:31:39.388867+0000","last_active":"2024-03-28T23:31:53.869894+0000","last_peered":"2024-03-28T23:31:53.869894+0000","last_clean":"2024-03-28T23:31:53.869894+0000","last_became_active":"2024-03-28T23:31:39.388593+0000","last_became_peered":"2024-03-28T23:31:39.388593+0000","last_unstale":"2024-03-28T23:31:53.869894+0000","last_undegraded":"2024-03-28T23:31:53.869894+0000","last_fullsized":"2024-03-28T23:31:53.869894+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:21:00.890266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872546+0000","last_change":"2024-03-28T23:31:53.872546+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872546+0000","last_undegraded":"2024-03-28T23:31:53.872546+0000","last_fullsized":"2024-03-28T23:31:53.872546+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.888466+0000","last_change":"2024-03-28T23:31:53.886433+0000","last_active":"2024-03-28T23:31:53.888466+0000","last_peered":"2024-03-28T23:31:53.888466+0000","last_clean":"2024-03-28T23:31:53.888466+0000","last_became_active":"2024-03-28T23:31:53.886218+0000","last_became_peered":"2024-03-28T23:31:53.886218+0000","last_unstale":"2024-03-28T23:31:53.888466+0000","last_undegraded":"2024-03-28T23:31:53.888466+0000","last_fullsized":"2024-03-28T23:31:53.888466+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:32.282446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876171+0000","last_change":"2024-03-28T23:31:40.393257+0000","last_active":"2024-03-28T23:31:53.876171+0000","last_peered":"2024-03-28T23:31:53.876171+0000","last_clean":"2024-03-28T23:31:53.876171+0000","last_became_active":"2024-03-28T23:31:40.391684+0000","last_became_peered":"2024-03-28T23:31:40.391684+0000","last_unstale":"2024-03-28T23:31:53.876171+0000","last_undegraded":"2024-03-28T23:31:53.876171+0000","last_fullsized":"2024-03-28T23:31:53.876171+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:01.371074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.881223+0000","last_change":"2024-03-28T23:31:39.388198+0000","last_active":"2024-03-28T23:31:53.881223+0000","last_peered":"2024-03-28T23:31:53.881223+0000","last_clean":"2024-03-28T23:31:53.881223+0000","last_became_active":"2024-03-28T23:31:39.387727+0000","last_became_peered":"2024-03-28T23:31:39.387727+0000","last_unstale":"2024-03-28T23:31:53.881223+0000","last_undegraded":"2024-03-28T23:31:53.881223+0000","last_fullsized":"2024-03-28T23:31:53.881223+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:49.868091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874240+0000","last_change":"2024-03-28T23:31:53.874240+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874240+0000","last_undegraded":"2024-03-28T23:31:53.874240+0000","last_fullsized":"2024-03-28T23:31:53.874240+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.886125+0000","last_change":"2024-03-28T23:31:53.884698+0000","last_active":"2024-03-28T23:31:53.886125+0000","last_peered":"2024-03-28T23:31:53.886125+0000","last_clean":"2024-03-28T23:31:53.886125+0000","last_became_active":"2024-03-28T23:31:53.884413+0000","last_became_peered":"2024-03-28T23:31:53.884413+0000","last_unstale":"2024-03-28T23:31:53.886125+0000","last_undegraded":"2024-03-28T23:31:53.886125+0000","last_fullsized":"2024-03-28T23:31:53.886125+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:32.282446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876556+0000","last_change":"2024-03-28T23:31:40.394047+0000","last_active":"2024-03-28T23:31:53.876556+0000","last_peered":"2024-03-28T23:31:53.876556+0000","last_clean":"2024-03-28T23:31:53.876556+0000","last_became_active":"2024-03-28T23:31:40.393772+0000","last_became_peered":"2024-03-28T23:31:40.393772+0000","last_unstale":"2024-03-28T23:31:53.876556+0000","last_undegraded":"2024-03-28T23:31:53.876556+0000","last_fullsized":"2024-03-28T23:31:53.876556+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:07:13.690791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875624+0000","last_change":"2024-03-28T23:31:39.388829+0000","last_active":"2024-03-28T23:31:53.875624+0000","last_peered":"2024-03-28T23:31:53.875624+0000","last_clean":"2024-03-28T23:31:53.875624+0000","last_became_active":"2024-03-28T23:31:39.388521+0000","last_became_peered":"2024-03-28T23:31:39.388521+0000","last_unstale":"2024-03-28T23:31:53.875624+0000","last_undegraded":"2024-03-28T23:31:53.875624+0000","last_fullsized":"2024-03-28T23:31:53.875624+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:23:52.148574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.877392+0000","last_change":"2024-03-28T23:31:53.877392+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.877392+0000","last_undegraded":"2024-03-28T23:31:53.877392+0000","last_fullsized":"2024-03-28T23:31:53.877392+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.887423+0000","last_change":"2024-03-28T23:31:53.886155+0000","last_active":"2024-03-28T23:31:53.887423+0000","last_peered":"2024-03-28T23:31:53.887423+0000","last_clean":"2024-03-28T23:31:53.887423+0000","last_became_active":"2024-03-28T23:31:53.885843+0000","last_became_peered":"2024-03-28T23:31:53.885843+0000","last_unstale":"2024-03-28T23:31:53.887423+0000","last_undegraded":"2024-03-28T23:31:53.887423+0000","last_fullsized":"2024-03-28T23:31:53.887423+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:38:09.370282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.870442+0000","last_change":"2024-03-28T23:31:40.384709+0000","last_active":"2024-03-28T23:31:53.870442+0000","last_peered":"2024-03-28T23:31:53.870442+0000","last_clean":"2024-03-28T23:31:53.870442+0000","last_became_active":"2024-03-28T23:31:40.384412+0000","last_became_peered":"2024-03-28T23:31:40.384412+0000","last_unstale":"2024-03-28T23:31:53.870442+0000","last_undegraded":"2024-03-28T23:31:53.870442+0000","last_fullsized":"2024-03-28T23:31:53.870442+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:25:39.118434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879185+0000","last_change":"2024-03-28T23:31:39.385937+0000","last_active":"2024-03-28T23:31:53.879185+0000","last_peered":"2024-03-28T23:31:53.879185+0000","last_clean":"2024-03-28T23:31:53.879185+0000","last_became_active":"2024-03-28T23:31:39.385675+0000","last_became_peered":"2024-03-28T23:31:39.385675+0000","last_unstale":"2024-03-28T23:31:53.879185+0000","last_undegraded":"2024-03-28T23:31:53.879185+0000","last_fullsized":"2024-03-28T23:31:53.879185+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:35.204851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874303+0000","last_change":"2024-03-28T23:31:53.874303+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874303+0000","last_undegraded":"2024-03-28T23:31:53.874303+0000","last_fullsized":"2024-03-28T23:31:53.874303+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.886168+0000","last_change":"2024-03-28T23:31:53.884950+0000","last_active":"2024-03-28T23:31:53.886168+0000","last_peered":"2024-03-28T23:31:53.886168+0000","last_clean":"2024-03-28T23:31:53.886168+0000","last_became_active":"2024-03-28T23:31:53.884686+0000","last_became_peered":"2024-03-28T23:31:53.884686+0000","last_unstale":"2024-03-28T23:31:53.886168+0000","last_undegraded":"2024-03-28T23:31:53.886168+0000","last_fullsized":"2024-03-28T23:31:53.886168+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:48.477986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874779+0000","last_change":"2024-03-28T23:31:40.387783+0000","last_active":"2024-03-28T23:31:53.874779+0000","last_peered":"2024-03-28T23:31:53.874779+0000","last_clean":"2024-03-28T23:31:53.874779+0000","last_became_active":"2024-03-28T23:31:40.387487+0000","last_became_peered":"2024-03-28T23:31:40.387487+0000","last_unstale":"2024-03-28T23:31:53.874779+0000","last_undegraded":"2024-03-28T23:31:53.874779+0000","last_fullsized":"2024-03-28T23:31:53.874779+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:22.230986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"50'2","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.880346+0000","last_change":"2024-03-28T23:31:39.391741+0000","last_active":"2024-03-28T23:31:53.880346+0000","last_peered":"2024-03-28T23:31:53.880346+0000","last_clean":"2024-03-28T23:31:53.880346+0000","last_became_active":"2024-03-28T23:31:39.391520+0000","last_became_peered":"2024-03-28T23:31:39.391520+0000","last_unstale":"2024-03-28T23:31:53.880346+0000","last_undegraded":"2024-03-28T23:31:53.880346+0000","last_fullsized":"2024-03-28T23:31:53.880346+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:16:42.869555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.874197+0000","last_change":"2024-03-28T23:31:53.874197+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.874197+0000","last_undegraded":"2024-03-28T23:31:53.874197+0000","last_fullsized":"2024-03-28T23:31:53.874197+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.890450+0000","last_change":"2024-03-28T23:31:53.887757+0000","last_active":"2024-03-28T23:31:53.890450+0000","last_peered":"2024-03-28T23:31:53.890450+0000","last_clean":"2024-03-28T23:31:53.890450+0000","last_became_active":"2024-03-28T23:31:53.887470+0000","last_became_peered":"2024-03-28T23:31:53.887470+0000","last_unstale":"2024-03-28T23:31:53.890450+0000","last_undegraded":"2024-03-28T23:31:53.890450+0000","last_fullsized":"2024-03-28T23:31:53.890450+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:40:57.385677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.869629+0000","last_change":"2024-03-28T23:31:40.387330+0000","last_active":"2024-03-28T23:31:53.869629+0000","last_peered":"2024-03-28T23:31:53.869629+0000","last_clean":"2024-03-28T23:31:53.869629+0000","last_became_active":"2024-03-28T23:31:40.387076+0000","last_became_peered":"2024-03-28T23:31:40.387076+0000","last_unstale":"2024-03-28T23:31:53.869629+0000","last_undegraded":"2024-03-28T23:31:53.869629+0000","last_fullsized":"2024-03-28T23:31:53.869629+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:25.525091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.877434+0000","last_change":"2024-03-28T23:31:39.386686+0000","last_active":"2024-03-28T23:31:53.877434+0000","last_peered":"2024-03-28T23:31:53.877434+0000","last_clean":"2024-03-28T23:31:53.877434+0000","last_became_active":"2024-03-28T23:31:39.386208+0000","last_became_peered":"2024-03-28T23:31:39.386208+0000","last_unstale":"2024-03-28T23:31:53.877434+0000","last_undegraded":"2024-03-28T23:31:53.877434+0000","last_fullsized":"2024-03-28T23:31:53.877434+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:22.210570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.876269+0000","last_change":"2024-03-28T23:31:53.876269+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.876269+0000","last_undegraded":"2024-03-28T23:31:53.876269+0000","last_fullsized":"2024-03-28T23:31:53.876269+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.891725+0000","last_change":"2024-03-28T23:31:53.887766+0000","last_active":"2024-03-28T23:31:53.891725+0000","last_peered":"2024-03-28T23:31:53.891725+0000","last_clean":"2024-03-28T23:31:53.891725+0000","last_became_active":"2024-03-28T23:31:53.887471+0000","last_became_peered":"2024-03-28T23:31:53.887471+0000","last_unstale":"2024-03-28T23:31:53.891725+0000","last_undegraded":"2024-03-28T23:31:53.891725+0000","last_fullsized":"2024-03-28T23:31:53.891725+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:05:01.725546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.881956+0000","last_change":"2024-03-28T23:31:40.386143+0000","last_active":"2024-03-28T23:31:53.881956+0000","last_peered":"2024-03-28T23:31:53.881956+0000","last_clean":"2024-03-28T23:31:53.881956+0000","last_became_active":"2024-03-28T23:31:40.385606+0000","last_became_peered":"2024-03-28T23:31:40.385606+0000","last_unstale":"2024-03-28T23:31:53.881956+0000","last_undegraded":"2024-03-28T23:31:53.881956+0000","last_fullsized":"2024-03-28T23:31:53.881956+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:40:09.867914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876026+0000","last_change":"2024-03-28T23:31:39.380624+0000","last_active":"2024-03-28T23:31:53.876026+0000","last_peered":"2024-03-28T23:31:53.876026+0000","last_clean":"2024-03-28T23:31:53.876026+0000","last_became_active":"2024-03-28T23:31:39.380340+0000","last_became_peered":"2024-03-28T23:31:39.380340+0000","last_unstale":"2024-03-28T23:31:53.876026+0000","last_undegraded":"2024-03-28T23:31:53.876026+0000","last_fullsized":"2024-03-28T23:31:53.876026+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:22.210570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.879694+0000","last_change":"2024-03-28T23:31:53.879694+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.879694+0000","last_undegraded":"2024-03-28T23:31:53.879694+0000","last_fullsized":"2024-03-28T23:31:53.879694+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.889286+0000","last_change":"2024-03-28T23:31:53.886660+0000","last_active":"2024-03-28T23:31:53.889286+0000","last_peered":"2024-03-28T23:31:53.889286+0000","last_clean":"2024-03-28T23:31:53.889286+0000","last_became_active":"2024-03-28T23:31:53.886355+0000","last_became_peered":"2024-03-28T23:31:53.886355+0000","last_unstale":"2024-03-28T23:31:53.889286+0000","last_undegraded":"2024-03-28T23:31:53.889286+0000","last_fullsized":"2024-03-28T23:31:53.889286+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:38:09.370282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875942+0000","last_change":"2024-03-28T23:31:40.404359+0000","last_active":"2024-03-28T23:31:53.875942+0000","last_peered":"2024-03-28T23:31:53.875942+0000","last_clean":"2024-03-28T23:31:53.875942+0000","last_became_active":"2024-03-28T23:31:40.404078+0000","last_became_peered":"2024-03-28T23:31:40.404078+0000","last_unstale":"2024-03-28T23:31:53.875942+0000","last_undegraded":"2024-03-28T23:31:53.875942+0000","last_fullsized":"2024-03-28T23:31:53.875942+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:18.654331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.882034+0000","last_change":"2024-03-28T23:31:39.393283+0000","last_active":"2024-03-28T23:31:53.882034+0000","last_peered":"2024-03-28T23:31:53.882034+0000","last_clean":"2024-03-28T23:31:53.882034+0000","last_became_active":"2024-03-28T23:31:39.393035+0000","last_became_peered":"2024-03-28T23:31:39.393035+0000","last_unstale":"2024-03-28T23:31:53.882034+0000","last_undegraded":"2024-03-28T23:31:53.882034+0000","last_fullsized":"2024-03-28T23:31:53.882034+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:13.139644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.872993+0000","last_change":"2024-03-28T23:31:53.872993+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.872993+0000","last_undegraded":"2024-03-28T23:31:53.872993+0000","last_fullsized":"2024-03-28T23:31:53.872993+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.890751+0000","last_change":"2024-03-28T23:31:53.888742+0000","last_active":"2024-03-28T23:31:53.890751+0000","last_peered":"2024-03-28T23:31:53.890751+0000","last_clean":"2024-03-28T23:31:53.890751+0000","last_became_active":"2024-03-28T23:31:53.888515+0000","last_became_peered":"2024-03-28T23:31:53.888515+0000","last_unstale":"2024-03-28T23:31:53.890751+0000","last_undegraded":"2024-03-28T23:31:53.890751+0000","last_fullsized":"2024-03-28T23:31:53.890751+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:17:10.580507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.875207+0000","last_change":"2024-03-28T23:31:40.392819+0000","last_active":"2024-03-28T23:31:53.875207+0000","last_peered":"2024-03-28T23:31:53.875207+0000","last_clean":"2024-03-28T23:31:53.875207+0000","last_became_active":"2024-03-28T23:31:40.392501+0000","last_became_peered":"2024-03-28T23:31:40.392501+0000","last_unstale":"2024-03-28T23:31:53.875207+0000","last_undegraded":"2024-03-28T23:31:53.875207+0000","last_fullsized":"2024-03-28T23:31:53.875207+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:14.144440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.881307+0000","last_change":"2024-03-28T23:31:39.387993+0000","last_active":"2024-03-28T23:31:53.881307+0000","last_peered":"2024-03-28T23:31:53.881307+0000","last_clean":"2024-03-28T23:31:53.881307+0000","last_became_active":"2024-03-28T23:31:39.387393+0000","last_became_peered":"2024-03-28T23:31:39.387393+0000","last_unstale":"2024-03-28T23:31:53.881307+0000","last_undegraded":"2024-03-28T23:31:53.881307+0000","last_fullsized":"2024-03-28T23:31:53.881307+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:21.226190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871482+0000","last_change":"2024-03-28T23:31:53.871482+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871482+0000","last_undegraded":"2024-03-28T23:31:53.871482+0000","last_fullsized":"2024-03-28T23:31:53.871482+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.892224+0000","last_change":"2024-03-28T23:31:53.890146+0000","last_active":"2024-03-28T23:31:53.892224+0000","last_peered":"2024-03-28T23:31:53.892224+0000","last_clean":"2024-03-28T23:31:53.892224+0000","last_became_active":"2024-03-28T23:31:53.888840+0000","last_became_peered":"2024-03-28T23:31:53.888840+0000","last_unstale":"2024-03-28T23:31:53.892224+0000","last_undegraded":"2024-03-28T23:31:53.892224+0000","last_fullsized":"2024-03-28T23:31:53.892224+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:01.360626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876467+0000","last_change":"2024-03-28T23:31:40.405766+0000","last_active":"2024-03-28T23:31:53.876467+0000","last_peered":"2024-03-28T23:31:53.876467+0000","last_clean":"2024-03-28T23:31:53.876467+0000","last_became_active":"2024-03-28T23:31:40.405517+0000","last_became_peered":"2024-03-28T23:31:40.405517+0000","last_unstale":"2024-03-28T23:31:53.876467+0000","last_undegraded":"2024-03-28T23:31:53.876467+0000","last_fullsized":"2024-03-28T23:31:53.876467+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:18:16.851435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.883245+0000","last_change":"2024-03-28T23:31:39.383748+0000","last_active":"2024-03-28T23:31:53.883245+0000","last_peered":"2024-03-28T23:31:53.883245+0000","last_clean":"2024-03-28T23:31:53.883245+0000","last_became_active":"2024-03-28T23:31:39.383070+0000","last_became_peered":"2024-03-28T23:31:39.383070+0000","last_unstale":"2024-03-28T23:31:53.883245+0000","last_undegraded":"2024-03-28T23:31:53.883245+0000","last_fullsized":"2024-03-28T23:31:53.883245+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:41.430571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.873125+0000","last_change":"2024-03-28T23:31:53.873125+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.873125+0000","last_undegraded":"2024-03-28T23:31:53.873125+0000","last_fullsized":"2024-03-28T23:31:53.873125+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.890278+0000","last_change":"2024-03-28T23:31:53.887894+0000","last_active":"2024-03-28T23:31:53.890278+0000","last_peered":"2024-03-28T23:31:53.890278+0000","last_clean":"2024-03-28T23:31:53.890278+0000","last_became_active":"2024-03-28T23:31:53.887619+0000","last_became_peered":"2024-03-28T23:31:53.887619+0000","last_unstale":"2024-03-28T23:31:53.890278+0000","last_undegraded":"2024-03-28T23:31:53.890278+0000","last_fullsized":"2024-03-28T23:31:53.890278+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:26:16.561871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.879094+0000","last_change":"2024-03-28T23:31:40.392625+0000","last_active":"2024-03-28T23:31:53.879094+0000","last_peered":"2024-03-28T23:31:53.879094+0000","last_clean":"2024-03-28T23:31:53.879094+0000","last_became_active":"2024-03-28T23:31:40.392359+0000","last_became_peered":"2024-03-28T23:31:40.392359+0000","last_unstale":"2024-03-28T23:31:53.879094+0000","last_undegraded":"2024-03-28T23:31:53.879094+0000","last_fullsized":"2024-03-28T23:31:53.879094+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:02.676183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.874872+0000","last_change":"2024-03-28T23:31:39.389478+0000","last_active":"2024-03-28T23:31:53.874872+0000","last_peered":"2024-03-28T23:31:53.874872+0000","last_clean":"2024-03-28T23:31:53.874872+0000","last_became_active":"2024-03-28T23:31:39.389154+0000","last_became_peered":"2024-03-28T23:31:39.389154+0000","last_unstale":"2024-03-28T23:31:53.874872+0000","last_undegraded":"2024-03-28T23:31:53.874872+0000","last_fullsized":"2024-03-28T23:31:53.874872+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:25:19.055737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-28T23:31:53.871716+0000","last_change":"2024-03-28T23:31:53.871716+0000","last_active":"2024-03-28T23:31:53.865658+0000","last_peered":"2024-03-28T23:31:53.865658+0000","last_clean":"2024-03-28T23:31:53.865658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:31:53.871716+0000","last_undegraded":"2024-03-28T23:31:53.871716+0000","last_fullsized":"2024-03-28T23:31:53.871716+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.895972+0000","last_change":"2024-03-28T23:31:53.890575+0000","last_active":"2024-03-28T23:31:53.895972+0000","last_peered":"2024-03-28T23:31:53.895972+0000","last_clean":"2024-03-28T23:31:53.895972+0000","last_became_active":"2024-03-28T23:31:53.890305+0000","last_became_peered":"2024-03-28T23:31:53.890305+0000","last_unstale":"2024-03-28T23:31:53.895972+0000","last_undegraded":"2024-03-28T23:31:53.895972+0000","last_fullsized":"2024-03-28T23:31:53.895972+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:31:25.647111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876751+0000","last_change":"2024-03-28T23:31:40.389210+0000","last_active":"2024-03-28T23:31:53.876751+0000","last_peered":"2024-03-28T23:31:53.876751+0000","last_clean":"2024-03-28T23:31:53.876751+0000","last_became_active":"2024-03-28T23:31:40.388573+0000","last_became_peered":"2024-03-28T23:31:40.388573+0000","last_unstale":"2024-03-28T23:31:53.876751+0000","last_undegraded":"2024-03-28T23:31:53.876751+0000","last_fullsized":"2024-03-28T23:31:53.876751+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:23:53.153369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"51'1","reported_seq":19,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.876646+0000","last_change":"2024-03-28T23:31:39.381202+0000","last_active":"2024-03-28T23:31:53.876646+0000","last_peered":"2024-03-28T23:31:53.876646+0000","last_clean":"2024-03-28T23:31:53.876646+0000","last_became_active":"2024-03-28T23:31:39.380913+0000","last_became_peered":"2024-03-28T23:31:39.380913+0000","last_unstale":"2024-03-28T23:31:53.876646+0000","last_undegraded":"2024-03-28T23:31:53.876646+0000","last_fullsized":"2024-03-28T23:31:53.876646+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:38:21.050492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":233,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-28T23:31:53.872708+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:31:53.872708+0000","last_peered":"2024-03-28T23:31:53.872708+0000","last_clean":"2024-03-28T23:31:53.872708+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:31:53.872708+0000","last_undegraded":"2024-03-28T23:31:53.872708+0000","last_fullsized":"2024-03-28T23:31:53.872708+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":704512,"data_stored":875469,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574878,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31792,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":626688,"data_stored":278115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574878,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31784,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340056,"statfs":{"total":96636764160,"available":96604217344,"internally_reserved":0,"allocated":618496,"data_stored":277463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574877,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31788,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340052,"statfs":{"total":96636764160,"available":96604213248,"internally_reserved":0,"allocated":622592,"data_stored":278045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":634880,"data_stored":279097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":630784,"data_stored":279075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":285137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":704512,"data_stored":875401,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:32:02.692 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:32:02.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.793+0000 7f25fefe9640 1 Processor -- start 2024-03-28T23:32:02.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.793+0000 7f25fefe9640 1 -- start start 2024-03-28T23:32:02.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.794+0000 7f25fefe9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f25f806fc60 0x7f25f8070110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:02.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.794+0000 7f25fefe9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f25f80c3390 0x7f25f80c37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:02.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.794+0000 7f25fefe9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25f8153050 con 0x7f25f8072810 2024-03-28T23:32:02.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.794+0000 7f25fefe9640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f25f80a8520 con 0x7f25f80c3390 2024-03-28T23:32:02.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.794+0000 7f25fefe9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f25f8153310 con 0x7f25f806fc60 2024-03-28T23:32:02.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.794+0000 7f25fd586640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f25f80c3390 0x7f25f80c37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:02.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.794+0000 7f25fcd85640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f25f8072810 0x7f25f80c4840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37756/0 (socket says 172.21.15.161:37756) 2024-03-28T23:32:02.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.794+0000 7f25fcd85640 1 -- 172.21.15.161:0/3229322594 learned_addr learned my addr 172.21.15.161:0/3229322594 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:02.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.794+0000 7f25fd586640 1 -- 172.21.15.161:0/3229322594 >> v2:172.21.15.170:3300/0 conn(0x7f25f806fc60 msgr2=0x7f25f8070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:02.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.794+0000 7f25f7fff640 1 --2- 172.21.15.161:0/3229322594 >> v2:172.21.15.170:3300/0 conn(0x7f25f806fc60 0x7f25f8070110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:02.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.794+0000 7f25fd586640 1 --2- 172.21.15.161:0/3229322594 >> v2:172.21.15.170:3300/0 conn(0x7f25f806fc60 0x7f25f8070110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:02.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.794+0000 7f25f77fe640 1 -- 172.21.15.161:0/3229322594 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 949508969 0 0) 0x7f25f8153050 con 0x7f25f8072810 2024-03-28T23:32:02.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.794+0000 7f25fd586640 1 -- 172.21.15.161:0/3229322594 >> v1:172.21.15.161:6789/0 conn(0x7f25f8072810 legacy=0x7f25f80c4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:02.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.794+0000 7f25fd586640 1 -- 172.21.15.161:0/3229322594 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25f81535f0 con 0x7f25f80c3390 2024-03-28T23:32:02.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.795+0000 7f25fd586640 1 --2- 172.21.15.161:0/3229322594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25f80c3390 0x7f25f80c37c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f25f8068990 tx=0x7f25f002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d0fc8b2fa460964 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:02.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.795+0000 7f25f77fe640 1 -- 172.21.15.161:0/3229322594 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f0039070 con 0x7f25f80c3390 2024-03-28T23:32:02.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.795+0000 7f25f77fe640 1 -- 172.21.15.161:0/3229322594 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25f002cbf0 con 0x7f25f80c3390 2024-03-28T23:32:02.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.795+0000 7f25f77fe640 1 -- 172.21.15.161:0/3229322594 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f0034440 con 0x7f25f80c3390 2024-03-28T23:32:02.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.795+0000 7f25fefe9640 1 -- 172.21.15.161:0/3229322594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25f80c3390 msgr2=0x7f25f80c37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:02.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.795+0000 7f25fefe9640 1 --2- 172.21.15.161:0/3229322594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25f80c3390 0x7f25f80c37c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f25f8068990 tx=0x7f25f002be40 comp rx=0 tx=0).stop 2024-03-28T23:32:02.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.796+0000 7f25fefe9640 1 -- 172.21.15.161:0/3229322594 shutdown_connections 2024-03-28T23:32:02.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.796+0000 7f25fefe9640 1 --2- 172.21.15.161:0/3229322594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25f80c3390 0x7f25f80c37c0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:02.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.796+0000 7f25fefe9640 1 --2- 172.21.15.161:0/3229322594 >> v2:172.21.15.170:3300/0 conn(0x7f25f806fc60 0x7f25f8070110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:02.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.796+0000 7f25fefe9640 1 -- 172.21.15.161:0/3229322594 >> 172.21.15.161:0/3229322594 conn(0x7f25f80aa740 msgr2=0x7f25f80aab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:02.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.796+0000 7f25fefe9640 1 -- 172.21.15.161:0/3229322594 shutdown_connections 2024-03-28T23:32:02.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.796+0000 7f25fefe9640 1 -- 172.21.15.161:0/3229322594 wait complete. 2024-03-28T23:32:02.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.797+0000 7f25fefe9640 1 Processor -- start 2024-03-28T23:32:02.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.797+0000 7f25fefe9640 1 -- start start 2024-03-28T23:32:02.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.798+0000 7f25fefe9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25f806fc60 0x7f25f8146140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:02.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.798+0000 7f25fefe9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25f8072810 0x7f25f8151f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:02.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.798+0000 7f25fefe9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25f80c3390 0x7f25f8152450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:02.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.798+0000 7f25fefe9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f25f80c4f50 con 0x7f25f806fc60 2024-03-28T23:32:02.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.798+0000 7f25fefe9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f25f8153bb0 con 0x7f25f80c3390 2024-03-28T23:32:02.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.798+0000 7f25fefe9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f25f81545c0 con 0x7f25f8072810 2024-03-28T23:32:02.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.798+0000 7f25fcd85640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25f806fc60 0x7f25f8146140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:02.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.798+0000 7f25fcd85640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25f806fc60 0x7f25f8146140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57772/0 (socket says 172.21.15.161:57772) 2024-03-28T23:32:02.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.798+0000 7f25fcd85640 1 -- 172.21.15.161:0/3816058353 learned_addr learned my addr 172.21.15.161:0/3816058353 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:02.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.798+0000 7f25fcd85640 1 -- 172.21.15.161:0/3816058353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25f8072810 msgr2=0x7f25f8151f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:02.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.798+0000 7f25f7fff640 1 --2- 172.21.15.161:0/3816058353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25f8072810 0x7f25f8151f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:02.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.798+0000 7f25fd586640 1 --2- 172.21.15.161:0/3816058353 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25f80c3390 0x7f25f8152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:02.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.798+0000 7f25fcd85640 1 --2- 172.21.15.161:0/3816058353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25f8072810 0x7f25f8151f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:02.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.798+0000 7f25fcd85640 1 -- 172.21.15.161:0/3816058353 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25f80c3390 msgr2=0x7f25f8152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:02.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.798+0000 7f25fcd85640 1 --2- 172.21.15.161:0/3816058353 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25f80c3390 0x7f25f8152450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:02.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.798+0000 7f25fcd85640 1 -- 172.21.15.161:0/3816058353 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25f81e92c0 con 0x7f25f806fc60 2024-03-28T23:32:02.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.798+0000 7f25fd586640 1 --2- 172.21.15.161:0/3816058353 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25f80c3390 0x7f25f8152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:02.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.798+0000 7f25f7fff640 1 --2- 172.21.15.161:0/3816058353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25f8072810 0x7f25f8151f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:02.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.798+0000 7f25fcd85640 1 --2- 172.21.15.161:0/3816058353 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25f806fc60 0x7f25f8146140 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f25ec00c4d0 tx=0x7f25ec00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:02.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.798+0000 7f25f57fa640 1 -- 172.21.15.161:0/3816058353 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ec018070 con 0x7f25f806fc60 2024-03-28T23:32:02.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.798+0000 7f25f57fa640 1 -- 172.21.15.161:0/3816058353 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25ec007500 con 0x7f25f806fc60 2024-03-28T23:32:02.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.798+0000 7f25fefe9640 1 -- 172.21.15.161:0/3816058353 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25f8152990 con 0x7f25f806fc60 2024-03-28T23:32:02.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.798+0000 7f25f57fa640 1 -- 172.21.15.161:0/3816058353 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ec004a20 con 0x7f25f806fc60 2024-03-28T23:32:02.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.798+0000 7f25fefe9640 1 -- 172.21.15.161:0/3816058353 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25f8154250 con 0x7f25f806fc60 2024-03-28T23:32:02.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.800+0000 7f25f57fa640 1 -- 172.21.15.161:0/3816058353 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f25ec004bc0 con 0x7f25f806fc60 2024-03-28T23:32:02.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.800+0000 7f25fefe9640 1 -- 172.21.15.161:0/3816058353 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25f804f4b0 con 0x7f25f806fc60 2024-03-28T23:32:02.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.800+0000 7f25f57fa640 1 --2- 172.21.15.161:0/3816058353 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f25dc06b500 0x7f25dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:02.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.800+0000 7f25f57fa640 1 -- 172.21.15.161:0/3816058353 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6957+0+0 (secure 0 0 0) 0x7f25ec089920 con 0x7f25f806fc60 2024-03-28T23:32:02.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.800+0000 7f25f7fff640 1 --2- 172.21.15.161:0/3816058353 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f25dc06b500 0x7f25dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:02.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.801+0000 7f25f7fff640 1 --2- 172.21.15.161:0/3816058353 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f25dc06b500 0x7f25dc06d9c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f25f80a9210 tx=0x7f25e4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:02.805 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.805+0000 7f25f57fa640 1 -- 172.21.15.161:0/3816058353 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f25ec0582d0 con 0x7f25f806fc60 2024-03-28T23:32:02.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.986+0000 7f25fefe9640 1 -- 172.21.15.161:0/3816058353 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f25f81548a0 con 0x7f25dc06b500 2024-03-28T23:32:02.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:02.997+0000 7f25f57fa640 1 -- 172.21.15.161:0/3816058353 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490559 (secure 0 0 0) 0x7f25f81548a0 con 0x7f25dc06b500 2024-03-28T23:32:03.003 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:03.006 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:32:03.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.001+0000 7f25fefe9640 1 -- 172.21.15.161:0/3816058353 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f25dc06b500 msgr2=0x7f25dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:03.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.001+0000 7f25fefe9640 1 --2- 172.21.15.161:0/3816058353 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f25dc06b500 0x7f25dc06d9c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f25f80a9210 tx=0x7f25e4008040 comp rx=0 tx=0).stop 2024-03-28T23:32:03.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.001+0000 7f25fefe9640 1 -- 172.21.15.161:0/3816058353 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25f806fc60 msgr2=0x7f25f8146140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:03.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.001+0000 7f25fefe9640 1 --2- 172.21.15.161:0/3816058353 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25f806fc60 0x7f25f8146140 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f25ec00c4d0 tx=0x7f25ec00c990 comp rx=0 tx=0).stop 2024-03-28T23:32:03.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.001+0000 7f25fefe9640 1 -- 172.21.15.161:0/3816058353 shutdown_connections 2024-03-28T23:32:03.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.001+0000 7f25fefe9640 1 --2- 172.21.15.161:0/3816058353 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f25dc06b500 0x7f25dc06d9c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:03.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.001+0000 7f25fefe9640 1 --2- 172.21.15.161:0/3816058353 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25f80c3390 0x7f25f8152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:03.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.001+0000 7f25fefe9640 1 --2- 172.21.15.161:0/3816058353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25f8072810 0x7f25f8151f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:03.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.001+0000 7f25fefe9640 1 --2- 172.21.15.161:0/3816058353 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25f806fc60 0x7f25f8146140 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:03.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.001+0000 7f25fefe9640 1 -- 172.21.15.161:0/3816058353 >> 172.21.15.161:0/3816058353 conn(0x7f25f80aa740 msgr2=0x7f25f80c5420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:03.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.001+0000 7f25fefe9640 1 -- 172.21.15.161:0/3816058353 shutdown_connections 2024-03-28T23:32:03.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.001+0000 7f25fefe9640 1 -- 172.21.15.161:0/3816058353 wait complete. 2024-03-28T23:32:03.012 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2024-03-28T23:32:01.337098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":375,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238184,"kb_used_data":5376,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736536,"statfs":{"total":773094113280,"available":772850212864,"internally_reserved":0,"allocated":5505024,"data_stored":3594784,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.468829"},"pg_stats":[{"pgid":"10.3d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908109+0000","last_change":"2024-03-28T23:31:54.917618+0000","last_active":"2024-03-28T23:31:55.908109+0000","last_peered":"2024-03-28T23:31:55.908109+0000","last_clean":"2024-03-28T23:31:55.908109+0000","last_became_active":"2024-03-28T23:31:54.917344+0000","last_became_peered":"2024-03-28T23:31:54.917344+0000","last_unstale":"2024-03-28T23:31:55.908109+0000","last_undegraded":"2024-03-28T23:31:55.908109+0000","last_fullsized":"2024-03-28T23:31:55.908109+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:11.936731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3f","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570539+0000","last_change":"2024-03-28T23:31:40.388090+0000","last_active":"2024-03-28T23:31:56.570539+0000","last_peered":"2024-03-28T23:31:56.570539+0000","last_clean":"2024-03-28T23:31:56.570539+0000","last_became_active":"2024-03-28T23:31:40.387805+0000","last_became_peered":"2024-03-28T23:31:40.387805+0000","last_unstale":"2024-03-28T23:31:56.570539+0000","last_undegraded":"2024-03-28T23:31:56.570539+0000","last_fullsized":"2024-03-28T23:31:56.570539+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:19:47.987753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.469033+0000","last_change":"2024-03-28T23:31:54.914045+0000","last_active":"2024-03-28T23:31:56.469033+0000","last_peered":"2024-03-28T23:31:56.469033+0000","last_clean":"2024-03-28T23:31:56.469033+0000","last_became_active":"2024-03-28T23:31:54.913765+0000","last_became_peered":"2024-03-28T23:31:54.913765+0000","last_unstale":"2024-03-28T23:31:56.469033+0000","last_undegraded":"2024-03-28T23:31:56.469033+0000","last_fullsized":"2024-03-28T23:31:56.469033+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:40:24.369112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.907375+0000","last_change":"2024-03-28T23:31:40.395246+0000","last_active":"2024-03-28T23:31:55.907375+0000","last_peered":"2024-03-28T23:31:55.907375+0000","last_clean":"2024-03-28T23:31:55.907375+0000","last_became_active":"2024-03-28T23:31:40.394583+0000","last_became_peered":"2024-03-28T23:31:40.394583+0000","last_unstale":"2024-03-28T23:31:55.907375+0000","last_undegraded":"2024-03-28T23:31:55.907375+0000","last_fullsized":"2024-03-28T23:31:55.907375+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:18.654331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3f","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570897+0000","last_change":"2024-03-28T23:31:54.905697+0000","last_active":"2024-03-28T23:31:56.570897+0000","last_peered":"2024-03-28T23:31:56.570897+0000","last_clean":"2024-03-28T23:31:56.570897+0000","last_became_active":"2024-03-28T23:31:54.905451+0000","last_became_peered":"2024-03-28T23:31:54.905451+0000","last_unstale":"2024-03-28T23:31:56.570897+0000","last_undegraded":"2024-03-28T23:31:56.570897+0000","last_fullsized":"2024-03-28T23:31:56.570897+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:35:49.233812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.909241+0000","last_change":"2024-03-28T23:31:40.397914+0000","last_active":"2024-03-28T23:31:55.909241+0000","last_peered":"2024-03-28T23:31:55.909241+0000","last_clean":"2024-03-28T23:31:55.909241+0000","last_became_active":"2024-03-28T23:31:40.394904+0000","last_became_peered":"2024-03-28T23:31:40.394904+0000","last_unstale":"2024-03-28T23:31:55.909241+0000","last_undegraded":"2024-03-28T23:31:55.909241+0000","last_fullsized":"2024-03-28T23:31:55.909241+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:26:03.062546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570609+0000","last_change":"2024-03-28T23:31:54.903469+0000","last_active":"2024-03-28T23:31:56.570609+0000","last_peered":"2024-03-28T23:31:56.570609+0000","last_clean":"2024-03-28T23:31:56.570609+0000","last_became_active":"2024-03-28T23:31:54.903161+0000","last_became_peered":"2024-03-28T23:31:54.903161+0000","last_unstale":"2024-03-28T23:31:56.570609+0000","last_undegraded":"2024-03-28T23:31:56.570609+0000","last_fullsized":"2024-03-28T23:31:56.570609+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:21:28.715853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.907104+0000","last_change":"2024-03-28T23:31:40.400594+0000","last_active":"2024-03-28T23:31:55.907104+0000","last_peered":"2024-03-28T23:31:55.907104+0000","last_clean":"2024-03-28T23:31:55.907104+0000","last_became_active":"2024-03-28T23:31:40.400255+0000","last_became_peered":"2024-03-28T23:31:40.400255+0000","last_unstale":"2024-03-28T23:31:55.907104+0000","last_undegraded":"2024-03-28T23:31:55.907104+0000","last_fullsized":"2024-03-28T23:31:55.907104+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:14.515069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569132+0000","last_change":"2024-03-28T23:31:54.940461+0000","last_active":"2024-03-28T23:31:56.569132+0000","last_peered":"2024-03-28T23:31:56.569132+0000","last_clean":"2024-03-28T23:31:56.569132+0000","last_became_active":"2024-03-28T23:31:54.940294+0000","last_became_peered":"2024-03-28T23:31:54.940294+0000","last_unstale":"2024-03-28T23:31:56.569132+0000","last_undegraded":"2024-03-28T23:31:56.569132+0000","last_fullsized":"2024-03-28T23:31:56.569132+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:21:16.396259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.467997+0000","last_change":"2024-03-28T23:31:40.388531+0000","last_active":"2024-03-28T23:31:56.467997+0000","last_peered":"2024-03-28T23:31:56.467997+0000","last_clean":"2024-03-28T23:31:56.467997+0000","last_became_active":"2024-03-28T23:31:40.388206+0000","last_became_peered":"2024-03-28T23:31:40.388206+0000","last_unstale":"2024-03-28T23:31:56.467997+0000","last_undegraded":"2024-03-28T23:31:56.467997+0000","last_fullsized":"2024-03-28T23:31:56.467997+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:23.215366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.906409+0000","last_change":"2024-03-28T23:31:54.906154+0000","last_active":"2024-03-28T23:31:55.906409+0000","last_peered":"2024-03-28T23:31:55.906409+0000","last_clean":"2024-03-28T23:31:55.906409+0000","last_became_active":"2024-03-28T23:31:54.905798+0000","last_became_peered":"2024-03-28T23:31:54.905798+0000","last_unstale":"2024-03-28T23:31:55.906409+0000","last_undegraded":"2024-03-28T23:31:55.906409+0000","last_fullsized":"2024-03-28T23:31:55.906409+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:17:11.582379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569474+0000","last_change":"2024-03-28T23:31:40.393907+0000","last_active":"2024-03-28T23:31:56.569474+0000","last_peered":"2024-03-28T23:31:56.569474+0000","last_clean":"2024-03-28T23:31:56.569474+0000","last_became_active":"2024-03-28T23:31:40.393443+0000","last_became_peered":"2024-03-28T23:31:40.393443+0000","last_unstale":"2024-03-28T23:31:56.569474+0000","last_undegraded":"2024-03-28T23:31:56.569474+0000","last_fullsized":"2024-03-28T23:31:56.569474+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:28:29.180775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570019+0000","last_change":"2024-03-28T23:31:54.913847+0000","last_active":"2024-03-28T23:31:56.570019+0000","last_peered":"2024-03-28T23:31:56.570019+0000","last_clean":"2024-03-28T23:31:56.570019+0000","last_became_active":"2024-03-28T23:31:54.913475+0000","last_became_peered":"2024-03-28T23:31:54.913475+0000","last_unstale":"2024-03-28T23:31:56.570019+0000","last_undegraded":"2024-03-28T23:31:56.570019+0000","last_fullsized":"2024-03-28T23:31:56.570019+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:18:31.352633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908209+0000","last_change":"2024-03-28T23:31:40.398876+0000","last_active":"2024-03-28T23:31:55.908209+0000","last_peered":"2024-03-28T23:31:55.908209+0000","last_clean":"2024-03-28T23:31:55.908209+0000","last_became_active":"2024-03-28T23:31:40.398595+0000","last_became_peered":"2024-03-28T23:31:40.398595+0000","last_unstale":"2024-03-28T23:31:55.908209+0000","last_undegraded":"2024-03-28T23:31:55.908209+0000","last_fullsized":"2024-03-28T23:31:55.908209+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:21.725238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908586+0000","last_change":"2024-03-28T23:31:54.915605+0000","last_active":"2024-03-28T23:31:55.908586+0000","last_peered":"2024-03-28T23:31:55.908586+0000","last_clean":"2024-03-28T23:31:55.908586+0000","last_became_active":"2024-03-28T23:31:54.915179+0000","last_became_peered":"2024-03-28T23:31:54.915179+0000","last_unstale":"2024-03-28T23:31:55.908586+0000","last_undegraded":"2024-03-28T23:31:55.908586+0000","last_fullsized":"2024-03-28T23:31:55.908586+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:50:33.299498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560049+0000","last_change":"2024-03-28T23:31:40.392421+0000","last_active":"2024-03-28T23:31:56.560049+0000","last_peered":"2024-03-28T23:31:56.560049+0000","last_clean":"2024-03-28T23:31:56.560049+0000","last_became_active":"2024-03-28T23:31:40.391463+0000","last_became_peered":"2024-03-28T23:31:40.391463+0000","last_unstale":"2024-03-28T23:31:56.560049+0000","last_undegraded":"2024-03-28T23:31:56.560049+0000","last_fullsized":"2024-03-28T23:31:56.560049+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:18:16.851435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.468986+0000","last_change":"2024-03-28T23:31:54.914953+0000","last_active":"2024-03-28T23:31:56.468986+0000","last_peered":"2024-03-28T23:31:56.468986+0000","last_clean":"2024-03-28T23:31:56.468986+0000","last_became_active":"2024-03-28T23:31:54.914605+0000","last_became_peered":"2024-03-28T23:31:54.914605+0000","last_unstale":"2024-03-28T23:31:56.468986+0000","last_undegraded":"2024-03-28T23:31:56.468986+0000","last_fullsized":"2024-03-28T23:31:56.468986+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:02.362498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560183+0000","last_change":"2024-03-28T23:31:40.403340+0000","last_active":"2024-03-28T23:31:56.560183+0000","last_peered":"2024-03-28T23:31:56.560183+0000","last_clean":"2024-03-28T23:31:56.560183+0000","last_became_active":"2024-03-28T23:31:40.403089+0000","last_became_peered":"2024-03-28T23:31:40.403089+0000","last_unstale":"2024-03-28T23:31:56.560183+0000","last_undegraded":"2024-03-28T23:31:56.560183+0000","last_fullsized":"2024-03-28T23:31:56.560183+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:12:38.242108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.907472+0000","last_change":"2024-03-28T23:31:54.912679+0000","last_active":"2024-03-28T23:31:55.907472+0000","last_peered":"2024-03-28T23:31:55.907472+0000","last_clean":"2024-03-28T23:31:55.907472+0000","last_became_active":"2024-03-28T23:31:54.912412+0000","last_became_peered":"2024-03-28T23:31:54.912412+0000","last_unstale":"2024-03-28T23:31:55.907472+0000","last_undegraded":"2024-03-28T23:31:55.907472+0000","last_fullsized":"2024-03-28T23:31:55.907472+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:17:44.675955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570645+0000","last_change":"2024-03-28T23:31:40.382204+0000","last_active":"2024-03-28T23:31:56.570645+0000","last_peered":"2024-03-28T23:31:56.570645+0000","last_clean":"2024-03-28T23:31:56.570645+0000","last_became_active":"2024-03-28T23:31:40.381944+0000","last_became_peered":"2024-03-28T23:31:40.381944+0000","last_unstale":"2024-03-28T23:31:56.570645+0000","last_undegraded":"2024-03-28T23:31:56.570645+0000","last_fullsized":"2024-03-28T23:31:56.570645+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:14.515069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908037+0000","last_change":"2024-03-28T23:31:54.920318+0000","last_active":"2024-03-28T23:31:55.908037+0000","last_peered":"2024-03-28T23:31:55.908037+0000","last_clean":"2024-03-28T23:31:55.908037+0000","last_became_active":"2024-03-28T23:31:54.920020+0000","last_became_peered":"2024-03-28T23:31:54.920020+0000","last_unstale":"2024-03-28T23:31:55.908037+0000","last_undegraded":"2024-03-28T23:31:55.908037+0000","last_fullsized":"2024-03-28T23:31:55.908037+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:06.326604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.568782+0000","last_change":"2024-03-28T23:31:40.394578+0000","last_active":"2024-03-28T23:31:56.568782+0000","last_peered":"2024-03-28T23:31:56.568782+0000","last_clean":"2024-03-28T23:31:56.568782+0000","last_became_active":"2024-03-28T23:31:40.394272+0000","last_became_peered":"2024-03-28T23:31:40.394272+0000","last_unstale":"2024-03-28T23:31:56.568782+0000","last_undegraded":"2024-03-28T23:31:56.568782+0000","last_fullsized":"2024-03-28T23:31:56.568782+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:24:35.708796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.907165+0000","last_change":"2024-03-28T23:31:54.909981+0000","last_active":"2024-03-28T23:31:55.907165+0000","last_peered":"2024-03-28T23:31:55.907165+0000","last_clean":"2024-03-28T23:31:55.907165+0000","last_became_active":"2024-03-28T23:31:54.909698+0000","last_became_peered":"2024-03-28T23:31:54.909698+0000","last_unstale":"2024-03-28T23:31:55.907165+0000","last_undegraded":"2024-03-28T23:31:55.907165+0000","last_fullsized":"2024-03-28T23:31:55.907165+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:10:23.965826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560636+0000","last_change":"2024-03-28T23:31:40.396702+0000","last_active":"2024-03-28T23:31:56.560636+0000","last_peered":"2024-03-28T23:31:56.560636+0000","last_clean":"2024-03-28T23:31:56.560636+0000","last_became_active":"2024-03-28T23:31:40.395941+0000","last_became_peered":"2024-03-28T23:31:40.395941+0000","last_unstale":"2024-03-28T23:31:56.560636+0000","last_undegraded":"2024-03-28T23:31:56.560636+0000","last_fullsized":"2024-03-28T23:31:56.560636+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:07:13.690791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.907432+0000","last_change":"2024-03-28T23:31:54.908297+0000","last_active":"2024-03-28T23:31:55.907432+0000","last_peered":"2024-03-28T23:31:55.907432+0000","last_clean":"2024-03-28T23:31:55.907432+0000","last_became_active":"2024-03-28T23:31:54.907619+0000","last_became_peered":"2024-03-28T23:31:54.907619+0000","last_unstale":"2024-03-28T23:31:55.907432+0000","last_undegraded":"2024-03-28T23:31:55.907432+0000","last_fullsized":"2024-03-28T23:31:55.907432+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:04:53.852197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569575+0000","last_change":"2024-03-28T23:31:40.382947+0000","last_active":"2024-03-28T23:31:56.569575+0000","last_peered":"2024-03-28T23:31:56.569575+0000","last_clean":"2024-03-28T23:31:56.569575+0000","last_became_active":"2024-03-28T23:31:40.381893+0000","last_became_peered":"2024-03-28T23:31:40.381893+0000","last_unstale":"2024-03-28T23:31:56.569575+0000","last_undegraded":"2024-03-28T23:31:56.569575+0000","last_fullsized":"2024-03-28T23:31:56.569575+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:50.872886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569125+0000","last_change":"2024-03-28T23:31:54.943272+0000","last_active":"2024-03-28T23:31:56.569125+0000","last_peered":"2024-03-28T23:31:56.569125+0000","last_clean":"2024-03-28T23:31:56.569125+0000","last_became_active":"2024-03-28T23:31:54.943024+0000","last_became_peered":"2024-03-28T23:31:54.943024+0000","last_unstale":"2024-03-28T23:31:56.569125+0000","last_undegraded":"2024-03-28T23:31:56.569125+0000","last_fullsized":"2024-03-28T23:31:56.569125+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:44.746978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908725+0000","last_change":"2024-03-28T23:31:40.405627+0000","last_active":"2024-03-28T23:31:55.908725+0000","last_peered":"2024-03-28T23:31:55.908725+0000","last_clean":"2024-03-28T23:31:55.908725+0000","last_became_active":"2024-03-28T23:31:40.405360+0000","last_became_peered":"2024-03-28T23:31:40.405360+0000","last_unstale":"2024-03-28T23:31:55.908725+0000","last_undegraded":"2024-03-28T23:31:55.908725+0000","last_fullsized":"2024-03-28T23:31:55.908725+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:06.567694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908356+0000","last_change":"2024-03-28T23:31:54.915800+0000","last_active":"2024-03-28T23:31:55.908356+0000","last_peered":"2024-03-28T23:31:55.908356+0000","last_clean":"2024-03-28T23:31:55.908356+0000","last_became_active":"2024-03-28T23:31:54.915467+0000","last_became_peered":"2024-03-28T23:31:54.915467+0000","last_unstale":"2024-03-28T23:31:55.908356+0000","last_undegraded":"2024-03-28T23:31:55.908356+0000","last_fullsized":"2024-03-28T23:31:55.908356+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:55:45.019098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.907590+0000","last_change":"2024-03-28T23:31:40.396605+0000","last_active":"2024-03-28T23:31:55.907590+0000","last_peered":"2024-03-28T23:31:55.907590+0000","last_clean":"2024-03-28T23:31:55.907590+0000","last_became_active":"2024-03-28T23:31:40.396237+0000","last_became_peered":"2024-03-28T23:31:40.396237+0000","last_unstale":"2024-03-28T23:31:55.907590+0000","last_undegraded":"2024-03-28T23:31:55.907590+0000","last_fullsized":"2024-03-28T23:31:55.907590+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:14.144440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908748+0000","last_change":"2024-03-28T23:31:54.916963+0000","last_active":"2024-03-28T23:31:55.908748+0000","last_peered":"2024-03-28T23:31:55.908748+0000","last_clean":"2024-03-28T23:31:55.908748+0000","last_became_active":"2024-03-28T23:31:54.916703+0000","last_became_peered":"2024-03-28T23:31:54.916703+0000","last_unstale":"2024-03-28T23:31:55.908748+0000","last_undegraded":"2024-03-28T23:31:55.908748+0000","last_fullsized":"2024-03-28T23:31:55.908748+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:17:24.437737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569973+0000","last_change":"2024-03-28T23:31:40.382823+0000","last_active":"2024-03-28T23:31:56.569973+0000","last_peered":"2024-03-28T23:31:56.569973+0000","last_clean":"2024-03-28T23:31:56.569973+0000","last_became_active":"2024-03-28T23:31:40.382574+0000","last_became_peered":"2024-03-28T23:31:40.382574+0000","last_unstale":"2024-03-28T23:31:56.569973+0000","last_undegraded":"2024-03-28T23:31:56.569973+0000","last_fullsized":"2024-03-28T23:31:56.569973+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:12:27.477720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569988+0000","last_change":"2024-03-28T23:31:54.940632+0000","last_active":"2024-03-28T23:31:56.569988+0000","last_peered":"2024-03-28T23:31:56.569988+0000","last_clean":"2024-03-28T23:31:56.569988+0000","last_became_active":"2024-03-28T23:31:54.940353+0000","last_became_peered":"2024-03-28T23:31:54.940353+0000","last_unstale":"2024-03-28T23:31:56.569988+0000","last_undegraded":"2024-03-28T23:31:56.569988+0000","last_fullsized":"2024-03-28T23:31:56.569988+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:02.362498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.909722+0000","last_change":"2024-03-28T23:31:40.403462+0000","last_active":"2024-03-28T23:31:55.909722+0000","last_peered":"2024-03-28T23:31:55.909722+0000","last_clean":"2024-03-28T23:31:55.909722+0000","last_became_active":"2024-03-28T23:31:40.402885+0000","last_became_peered":"2024-03-28T23:31:40.402885+0000","last_unstale":"2024-03-28T23:31:55.909722+0000","last_undegraded":"2024-03-28T23:31:55.909722+0000","last_fullsized":"2024-03-28T23:31:55.909722+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:42:29.351941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560518+0000","last_change":"2024-03-28T23:31:54.908075+0000","last_active":"2024-03-28T23:31:56.560518+0000","last_peered":"2024-03-28T23:31:56.560518+0000","last_clean":"2024-03-28T23:31:56.560518+0000","last_became_active":"2024-03-28T23:31:54.907734+0000","last_became_peered":"2024-03-28T23:31:54.907734+0000","last_unstale":"2024-03-28T23:31:56.560518+0000","last_undegraded":"2024-03-28T23:31:56.560518+0000","last_fullsized":"2024-03-28T23:31:56.560518+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:34:43.730148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.907488+0000","last_change":"2024-03-28T23:31:40.395478+0000","last_active":"2024-03-28T23:31:55.907488+0000","last_peered":"2024-03-28T23:31:55.907488+0000","last_clean":"2024-03-28T23:31:55.907488+0000","last_became_active":"2024-03-28T23:31:40.394981+0000","last_became_peered":"2024-03-28T23:31:40.394981+0000","last_unstale":"2024-03-28T23:31:55.907488+0000","last_undegraded":"2024-03-28T23:31:55.907488+0000","last_fullsized":"2024-03-28T23:31:55.907488+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:06.567694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.571037+0000","last_change":"2024-03-28T23:31:54.916181+0000","last_active":"2024-03-28T23:31:56.571037+0000","last_peered":"2024-03-28T23:31:56.571037+0000","last_clean":"2024-03-28T23:31:56.571037+0000","last_became_active":"2024-03-28T23:31:54.915914+0000","last_became_peered":"2024-03-28T23:31:54.915914+0000","last_unstale":"2024-03-28T23:31:56.571037+0000","last_undegraded":"2024-03-28T23:31:56.571037+0000","last_fullsized":"2024-03-28T23:31:56.571037+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:09.895975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.909281+0000","last_change":"2024-03-28T23:31:40.394475+0000","last_active":"2024-03-28T23:31:55.909281+0000","last_peered":"2024-03-28T23:31:55.909281+0000","last_clean":"2024-03-28T23:31:55.909281+0000","last_became_active":"2024-03-28T23:31:40.390205+0000","last_became_peered":"2024-03-28T23:31:40.390205+0000","last_unstale":"2024-03-28T23:31:55.909281+0000","last_undegraded":"2024-03-28T23:31:55.909281+0000","last_fullsized":"2024-03-28T23:31:55.909281+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:04:48.226221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.468964+0000","last_change":"2024-03-28T23:31:54.916336+0000","last_active":"2024-03-28T23:31:56.468964+0000","last_peered":"2024-03-28T23:31:56.468964+0000","last_clean":"2024-03-28T23:31:56.468964+0000","last_became_active":"2024-03-28T23:31:54.916074+0000","last_became_peered":"2024-03-28T23:31:54.916074+0000","last_unstale":"2024-03-28T23:31:56.468964+0000","last_undegraded":"2024-03-28T23:31:56.468964+0000","last_fullsized":"2024-03-28T23:31:56.468964+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:18:31.352633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570010+0000","last_change":"2024-03-28T23:31:40.388386+0000","last_active":"2024-03-28T23:31:56.570010+0000","last_peered":"2024-03-28T23:31:56.570010+0000","last_clean":"2024-03-28T23:31:56.570010+0000","last_became_active":"2024-03-28T23:31:40.388043+0000","last_became_peered":"2024-03-28T23:31:40.388043+0000","last_unstale":"2024-03-28T23:31:56.570010+0000","last_undegraded":"2024-03-28T23:31:56.570010+0000","last_fullsized":"2024-03-28T23:31:56.570010+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:07.738674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560220+0000","last_change":"2024-03-28T23:31:54.912125+0000","last_active":"2024-03-28T23:31:56.560220+0000","last_peered":"2024-03-28T23:31:56.560220+0000","last_clean":"2024-03-28T23:31:56.560220+0000","last_became_active":"2024-03-28T23:31:54.911839+0000","last_became_peered":"2024-03-28T23:31:54.911839+0000","last_unstale":"2024-03-28T23:31:56.560220+0000","last_undegraded":"2024-03-28T23:31:56.560220+0000","last_fullsized":"2024-03-28T23:31:56.560220+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:35:25.803045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569311+0000","last_change":"2024-03-28T23:31:40.395811+0000","last_active":"2024-03-28T23:31:56.569311+0000","last_peered":"2024-03-28T23:31:56.569311+0000","last_clean":"2024-03-28T23:31:56.569311+0000","last_became_active":"2024-03-28T23:31:40.395540+0000","last_became_peered":"2024-03-28T23:31:40.395540+0000","last_unstale":"2024-03-28T23:31:56.569311+0000","last_undegraded":"2024-03-28T23:31:56.569311+0000","last_fullsized":"2024-03-28T23:31:56.569311+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:51.864986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.906489+0000","last_change":"2024-03-28T23:31:54.917556+0000","last_active":"2024-03-28T23:31:55.906489+0000","last_peered":"2024-03-28T23:31:55.906489+0000","last_clean":"2024-03-28T23:31:55.906489+0000","last_became_active":"2024-03-28T23:31:54.917293+0000","last_became_peered":"2024-03-28T23:31:54.917293+0000","last_unstale":"2024-03-28T23:31:55.906489+0000","last_undegraded":"2024-03-28T23:31:55.906489+0000","last_fullsized":"2024-03-28T23:31:55.906489+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:17:44.675955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560040+0000","last_change":"2024-03-28T23:31:40.403129+0000","last_active":"2024-03-28T23:31:56.560040+0000","last_peered":"2024-03-28T23:31:56.560040+0000","last_clean":"2024-03-28T23:31:56.560040+0000","last_became_active":"2024-03-28T23:31:40.402879+0000","last_became_peered":"2024-03-28T23:31:40.402879+0000","last_unstale":"2024-03-28T23:31:56.560040+0000","last_undegraded":"2024-03-28T23:31:56.560040+0000","last_fullsized":"2024-03-28T23:31:56.560040+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:47.861300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908392+0000","last_change":"2024-03-28T23:31:54.903995+0000","last_active":"2024-03-28T23:31:55.908392+0000","last_peered":"2024-03-28T23:31:55.908392+0000","last_clean":"2024-03-28T23:31:55.908392+0000","last_became_active":"2024-03-28T23:31:54.903722+0000","last_became_peered":"2024-03-28T23:31:54.903722+0000","last_unstale":"2024-03-28T23:31:55.908392+0000","last_undegraded":"2024-03-28T23:31:55.908392+0000","last_fullsized":"2024-03-28T23:31:55.908392+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:06.326604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.568747+0000","last_change":"2024-03-28T23:31:40.389854+0000","last_active":"2024-03-28T23:31:56.568747+0000","last_peered":"2024-03-28T23:31:56.568747+0000","last_clean":"2024-03-28T23:31:56.568747+0000","last_became_active":"2024-03-28T23:31:40.389599+0000","last_became_peered":"2024-03-28T23:31:40.389599+0000","last_unstale":"2024-03-28T23:31:56.568747+0000","last_undegraded":"2024-03-28T23:31:56.568747+0000","last_fullsized":"2024-03-28T23:31:56.568747+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:43:37.883841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560532+0000","last_change":"2024-03-28T23:31:54.915410+0000","last_active":"2024-03-28T23:31:56.560532+0000","last_peered":"2024-03-28T23:31:56.560532+0000","last_clean":"2024-03-28T23:31:56.560532+0000","last_became_active":"2024-03-28T23:31:54.915167+0000","last_became_peered":"2024-03-28T23:31:54.915167+0000","last_unstale":"2024-03-28T23:31:56.560532+0000","last_undegraded":"2024-03-28T23:31:56.560532+0000","last_fullsized":"2024-03-28T23:31:56.560532+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:38:24.676314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908286+0000","last_change":"2024-03-28T23:31:40.406317+0000","last_active":"2024-03-28T23:31:55.908286+0000","last_peered":"2024-03-28T23:31:55.908286+0000","last_clean":"2024-03-28T23:31:55.908286+0000","last_became_active":"2024-03-28T23:31:40.405217+0000","last_became_peered":"2024-03-28T23:31:40.405217+0000","last_unstale":"2024-03-28T23:31:55.908286+0000","last_undegraded":"2024-03-28T23:31:55.908286+0000","last_fullsized":"2024-03-28T23:31:55.908286+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:37:55.870956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.468784+0000","last_change":"2024-03-28T23:31:54.944549+0000","last_active":"2024-03-28T23:31:56.468784+0000","last_peered":"2024-03-28T23:31:56.468784+0000","last_clean":"2024-03-28T23:31:56.468784+0000","last_became_active":"2024-03-28T23:31:54.944261+0000","last_became_peered":"2024-03-28T23:31:54.944261+0000","last_unstale":"2024-03-28T23:31:56.468784+0000","last_undegraded":"2024-03-28T23:31:56.468784+0000","last_fullsized":"2024-03-28T23:31:56.468784+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:26:17.563744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.909828+0000","last_change":"2024-03-28T23:31:40.404249+0000","last_active":"2024-03-28T23:31:55.909828+0000","last_peered":"2024-03-28T23:31:55.909828+0000","last_clean":"2024-03-28T23:31:55.909828+0000","last_became_active":"2024-03-28T23:31:40.403966+0000","last_became_peered":"2024-03-28T23:31:40.403966+0000","last_unstale":"2024-03-28T23:31:55.909828+0000","last_undegraded":"2024-03-28T23:31:55.909828+0000","last_fullsized":"2024-03-28T23:31:55.909828+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:40:57.541676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569707+0000","last_change":"2024-03-28T23:31:54.904361+0000","last_active":"2024-03-28T23:31:56.569707+0000","last_peered":"2024-03-28T23:31:56.569707+0000","last_clean":"2024-03-28T23:31:56.569707+0000","last_became_active":"2024-03-28T23:31:54.904054+0000","last_became_peered":"2024-03-28T23:31:54.904054+0000","last_unstale":"2024-03-28T23:31:56.569707+0000","last_undegraded":"2024-03-28T23:31:56.569707+0000","last_fullsized":"2024-03-28T23:31:56.569707+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:53:40.856546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560797+0000","last_change":"2024-03-28T23:31:40.397170+0000","last_active":"2024-03-28T23:31:56.560797+0000","last_peered":"2024-03-28T23:31:56.560797+0000","last_clean":"2024-03-28T23:31:56.560797+0000","last_became_active":"2024-03-28T23:31:40.396898+0000","last_became_peered":"2024-03-28T23:31:40.396898+0000","last_unstale":"2024-03-28T23:31:56.560797+0000","last_undegraded":"2024-03-28T23:31:56.560797+0000","last_fullsized":"2024-03-28T23:31:56.560797+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:53:26.355348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.907850+0000","last_change":"2024-03-28T23:31:54.942701+0000","last_active":"2024-03-28T23:31:55.907850+0000","last_peered":"2024-03-28T23:31:55.907850+0000","last_clean":"2024-03-28T23:31:55.907850+0000","last_became_active":"2024-03-28T23:31:54.942392+0000","last_became_peered":"2024-03-28T23:31:54.942392+0000","last_unstale":"2024-03-28T23:31:55.907850+0000","last_undegraded":"2024-03-28T23:31:55.907850+0000","last_fullsized":"2024-03-28T23:31:55.907850+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:23:28.131019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569474+0000","last_change":"2024-03-28T23:31:40.404955+0000","last_active":"2024-03-28T23:31:56.569474+0000","last_peered":"2024-03-28T23:31:56.569474+0000","last_clean":"2024-03-28T23:31:56.569474+0000","last_became_active":"2024-03-28T23:31:40.404689+0000","last_became_peered":"2024-03-28T23:31:40.404689+0000","last_unstale":"2024-03-28T23:31:56.569474+0000","last_undegraded":"2024-03-28T23:31:56.569474+0000","last_fullsized":"2024-03-28T23:31:56.569474+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:50:20.574648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.907937+0000","last_change":"2024-03-28T23:31:54.942690+0000","last_active":"2024-03-28T23:31:55.907937+0000","last_peered":"2024-03-28T23:31:55.907937+0000","last_clean":"2024-03-28T23:31:55.907937+0000","last_became_active":"2024-03-28T23:31:54.942458+0000","last_became_peered":"2024-03-28T23:31:54.942458+0000","last_unstale":"2024-03-28T23:31:55.907937+0000","last_undegraded":"2024-03-28T23:31:55.907937+0000","last_fullsized":"2024-03-28T23:31:55.907937+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:10:23.965826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570101+0000","last_change":"2024-03-28T23:31:40.389261+0000","last_active":"2024-03-28T23:31:56.570101+0000","last_peered":"2024-03-28T23:31:56.570101+0000","last_clean":"2024-03-28T23:31:56.570101+0000","last_became_active":"2024-03-28T23:31:40.388930+0000","last_became_peered":"2024-03-28T23:31:40.388930+0000","last_unstale":"2024-03-28T23:31:56.570101+0000","last_undegraded":"2024-03-28T23:31:56.570101+0000","last_fullsized":"2024-03-28T23:31:56.570101+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:23.215366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.468297+0000","last_change":"2024-03-28T23:31:54.944552+0000","last_active":"2024-03-28T23:31:56.468297+0000","last_peered":"2024-03-28T23:31:56.468297+0000","last_clean":"2024-03-28T23:31:56.468297+0000","last_became_active":"2024-03-28T23:31:54.944257+0000","last_became_peered":"2024-03-28T23:31:54.944257+0000","last_unstale":"2024-03-28T23:31:56.468297+0000","last_undegraded":"2024-03-28T23:31:56.468297+0000","last_fullsized":"2024-03-28T23:31:56.468297+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:38:10.372154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908606+0000","last_change":"2024-03-28T23:31:40.401673+0000","last_active":"2024-03-28T23:31:55.908606+0000","last_peered":"2024-03-28T23:31:55.908606+0000","last_clean":"2024-03-28T23:31:55.908606+0000","last_became_active":"2024-03-28T23:31:40.398923+0000","last_became_peered":"2024-03-28T23:31:40.398923+0000","last_unstale":"2024-03-28T23:31:55.908606+0000","last_undegraded":"2024-03-28T23:31:55.908606+0000","last_fullsized":"2024-03-28T23:31:55.908606+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:18.783120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908128+0000","last_change":"2024-03-28T23:31:54.944546+0000","last_active":"2024-03-28T23:31:55.908128+0000","last_peered":"2024-03-28T23:31:55.908128+0000","last_clean":"2024-03-28T23:31:55.908128+0000","last_became_active":"2024-03-28T23:31:54.944271+0000","last_became_peered":"2024-03-28T23:31:54.944271+0000","last_unstale":"2024-03-28T23:31:55.908128+0000","last_undegraded":"2024-03-28T23:31:55.908128+0000","last_fullsized":"2024-03-28T23:31:55.908128+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:40:22.833100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908821+0000","last_change":"2024-03-28T23:31:40.387082+0000","last_active":"2024-03-28T23:31:55.908821+0000","last_peered":"2024-03-28T23:31:55.908821+0000","last_clean":"2024-03-28T23:31:55.908821+0000","last_became_active":"2024-03-28T23:31:40.386808+0000","last_became_peered":"2024-03-28T23:31:40.386808+0000","last_unstale":"2024-03-28T23:31:55.908821+0000","last_undegraded":"2024-03-28T23:31:55.908821+0000","last_fullsized":"2024-03-28T23:31:55.908821+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:43.111058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.907015+0000","last_change":"2024-03-28T23:31:54.919854+0000","last_active":"2024-03-28T23:31:55.907015+0000","last_peered":"2024-03-28T23:31:55.907015+0000","last_clean":"2024-03-28T23:31:55.907015+0000","last_became_active":"2024-03-28T23:31:54.919343+0000","last_became_peered":"2024-03-28T23:31:54.919343+0000","last_unstale":"2024-03-28T23:31:55.907015+0000","last_undegraded":"2024-03-28T23:31:55.907015+0000","last_fullsized":"2024-03-28T23:31:55.907015+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:23:28.131019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908307+0000","last_change":"2024-03-28T23:31:40.397815+0000","last_active":"2024-03-28T23:31:55.908307+0000","last_peered":"2024-03-28T23:31:55.908307+0000","last_clean":"2024-03-28T23:31:55.908307+0000","last_became_active":"2024-03-28T23:31:40.397279+0000","last_became_peered":"2024-03-28T23:31:40.397279+0000","last_unstale":"2024-03-28T23:31:55.908307+0000","last_undegraded":"2024-03-28T23:31:55.908307+0000","last_fullsized":"2024-03-28T23:31:55.908307+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:25:57.449014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.906782+0000","last_change":"2024-03-28T23:31:54.911951+0000","last_active":"2024-03-28T23:31:55.906782+0000","last_peered":"2024-03-28T23:31:55.906782+0000","last_clean":"2024-03-28T23:31:55.906782+0000","last_became_active":"2024-03-28T23:31:54.911649+0000","last_became_peered":"2024-03-28T23:31:54.911649+0000","last_unstale":"2024-03-28T23:31:55.906782+0000","last_undegraded":"2024-03-28T23:31:55.906782+0000","last_fullsized":"2024-03-28T23:31:55.906782+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:19:24.384421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.906651+0000","last_change":"2024-03-28T23:31:40.394484+0000","last_active":"2024-03-28T23:31:55.906651+0000","last_peered":"2024-03-28T23:31:55.906651+0000","last_clean":"2024-03-28T23:31:55.906651+0000","last_became_active":"2024-03-28T23:31:40.394219+0000","last_became_peered":"2024-03-28T23:31:40.394219+0000","last_unstale":"2024-03-28T23:31:55.906651+0000","last_undegraded":"2024-03-28T23:31:55.906651+0000","last_fullsized":"2024-03-28T23:31:55.906651+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:18.654331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569659+0000","last_change":"2024-03-28T23:31:54.940646+0000","last_active":"2024-03-28T23:31:56.569659+0000","last_peered":"2024-03-28T23:31:56.569659+0000","last_clean":"2024-03-28T23:31:56.569659+0000","last_became_active":"2024-03-28T23:31:54.940340+0000","last_became_peered":"2024-03-28T23:31:54.940340+0000","last_unstale":"2024-03-28T23:31:56.569659+0000","last_undegraded":"2024-03-28T23:31:56.569659+0000","last_fullsized":"2024-03-28T23:31:56.569659+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:26:11.950212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.907350+0000","last_change":"2024-03-28T23:31:53.890105+0000","last_active":"2024-03-28T23:31:55.907350+0000","last_peered":"2024-03-28T23:31:55.907350+0000","last_clean":"2024-03-28T23:31:55.907350+0000","last_became_active":"2024-03-28T23:31:53.889841+0000","last_became_peered":"2024-03-28T23:31:53.889841+0000","last_unstale":"2024-03-28T23:31:55.907350+0000","last_undegraded":"2024-03-28T23:31:55.907350+0000","last_fullsized":"2024-03-28T23:31:55.907350+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:40:23.367240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.906594+0000","last_change":"2024-03-28T23:31:40.398232+0000","last_active":"2024-03-28T23:31:55.906594+0000","last_peered":"2024-03-28T23:31:55.906594+0000","last_clean":"2024-03-28T23:31:55.906594+0000","last_became_active":"2024-03-28T23:31:40.397888+0000","last_became_peered":"2024-03-28T23:31:40.397888+0000","last_unstale":"2024-03-28T23:31:55.906594+0000","last_undegraded":"2024-03-28T23:31:55.906594+0000","last_fullsized":"2024-03-28T23:31:55.906594+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:50.872886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"50'4","reported_seq":26,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570281+0000","last_change":"2024-03-28T23:31:39.385488+0000","last_active":"2024-03-28T23:31:56.570281+0000","last_peered":"2024-03-28T23:31:56.570281+0000","last_clean":"2024-03-28T23:31:56.570281+0000","last_became_active":"2024-03-28T23:31:39.385211+0000","last_became_peered":"2024-03-28T23:31:39.385211+0000","last_unstale":"2024-03-28T23:31:56.570281+0000","last_undegraded":"2024-03-28T23:31:56.570281+0000","last_fullsized":"2024-03-28T23:31:56.570281+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:50:10.559743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569718+0000","last_change":"2024-03-28T23:31:54.910334+0000","last_active":"2024-03-28T23:31:56.569718+0000","last_peered":"2024-03-28T23:31:56.569718+0000","last_clean":"2024-03-28T23:31:56.569718+0000","last_became_active":"2024-03-28T23:31:54.909965+0000","last_became_peered":"2024-03-28T23:31:54.909965+0000","last_unstale":"2024-03-28T23:31:56.569718+0000","last_undegraded":"2024-03-28T23:31:56.569718+0000","last_fullsized":"2024-03-28T23:31:56.569718+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:17.177381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"54'8","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.468173+0000","last_change":"2024-03-28T23:31:53.887853+0000","last_active":"2024-03-28T23:31:56.468173+0000","last_peered":"2024-03-28T23:31:56.468173+0000","last_clean":"2024-03-28T23:31:56.468173+0000","last_became_active":"2024-03-28T23:31:53.887588+0000","last_became_peered":"2024-03-28T23:31:53.887588+0000","last_unstale":"2024-03-28T23:31:56.468173+0000","last_undegraded":"2024-03-28T23:31:56.468173+0000","last_fullsized":"2024-03-28T23:31:56.468173+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:39.024417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908223+0000","last_change":"2024-03-28T23:31:40.390218+0000","last_active":"2024-03-28T23:31:55.908223+0000","last_peered":"2024-03-28T23:31:55.908223+0000","last_clean":"2024-03-28T23:31:55.908223+0000","last_became_active":"2024-03-28T23:31:40.389922+0000","last_became_peered":"2024-03-28T23:31:40.389922+0000","last_unstale":"2024-03-28T23:31:55.908223+0000","last_undegraded":"2024-03-28T23:31:55.908223+0000","last_fullsized":"2024-03-28T23:31:55.908223+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:14.515069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570698+0000","last_change":"2024-03-28T23:31:39.393513+0000","last_active":"2024-03-28T23:31:56.570698+0000","last_peered":"2024-03-28T23:31:56.570698+0000","last_clean":"2024-03-28T23:31:56.570698+0000","last_became_active":"2024-03-28T23:31:39.393176+0000","last_became_peered":"2024-03-28T23:31:39.393176+0000","last_unstale":"2024-03-28T23:31:56.570698+0000","last_undegraded":"2024-03-28T23:31:56.570698+0000","last_fullsized":"2024-03-28T23:31:56.570698+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:29.240985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.468182+0000","last_change":"2024-03-28T23:31:54.915070+0000","last_active":"2024-03-28T23:31:56.468182+0000","last_peered":"2024-03-28T23:31:56.468182+0000","last_clean":"2024-03-28T23:31:56.468182+0000","last_became_active":"2024-03-28T23:31:54.914796+0000","last_became_peered":"2024-03-28T23:31:54.914796+0000","last_unstale":"2024-03-28T23:31:56.468182+0000","last_undegraded":"2024-03-28T23:31:56.468182+0000","last_fullsized":"2024-03-28T23:31:56.468182+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:26:11.950212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.907106+0000","last_change":"2024-03-28T23:31:53.890699+0000","last_active":"2024-03-28T23:31:55.907106+0000","last_peered":"2024-03-28T23:31:55.907106+0000","last_clean":"2024-03-28T23:31:55.907106+0000","last_became_active":"2024-03-28T23:31:53.890300+0000","last_became_peered":"2024-03-28T23:31:53.890300+0000","last_unstale":"2024-03-28T23:31:55.907106+0000","last_undegraded":"2024-03-28T23:31:55.907106+0000","last_fullsized":"2024-03-28T23:31:55.907106+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:42:42.851267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569232+0000","last_change":"2024-03-28T23:31:40.397077+0000","last_active":"2024-03-28T23:31:56.569232+0000","last_peered":"2024-03-28T23:31:56.569232+0000","last_clean":"2024-03-28T23:31:56.569232+0000","last_became_active":"2024-03-28T23:31:40.395265+0000","last_became_peered":"2024-03-28T23:31:40.395265+0000","last_unstale":"2024-03-28T23:31:56.569232+0000","last_undegraded":"2024-03-28T23:31:56.569232+0000","last_fullsized":"2024-03-28T23:31:56.569232+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:04.829030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570109+0000","last_change":"2024-03-28T23:31:39.391743+0000","last_active":"2024-03-28T23:31:56.570109+0000","last_peered":"2024-03-28T23:31:56.570109+0000","last_clean":"2024-03-28T23:31:56.570109+0000","last_became_active":"2024-03-28T23:31:39.391300+0000","last_became_peered":"2024-03-28T23:31:39.391300+0000","last_unstale":"2024-03-28T23:31:56.570109+0000","last_undegraded":"2024-03-28T23:31:56.570109+0000","last_fullsized":"2024-03-28T23:31:56.570109+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:12.505382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.907962+0000","last_change":"2024-03-28T23:31:54.904811+0000","last_active":"2024-03-28T23:31:55.907962+0000","last_peered":"2024-03-28T23:31:55.907962+0000","last_clean":"2024-03-28T23:31:55.907962+0000","last_became_active":"2024-03-28T23:31:54.904551+0000","last_became_peered":"2024-03-28T23:31:54.904551+0000","last_unstale":"2024-03-28T23:31:55.907962+0000","last_undegraded":"2024-03-28T23:31:55.907962+0000","last_fullsized":"2024-03-28T23:31:55.907962+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:04:53.852197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.906881+0000","last_change":"2024-03-28T23:31:53.886833+0000","last_active":"2024-03-28T23:31:55.906881+0000","last_peered":"2024-03-28T23:31:55.906881+0000","last_clean":"2024-03-28T23:31:55.906881+0000","last_became_active":"2024-03-28T23:31:53.886498+0000","last_became_peered":"2024-03-28T23:31:53.886498+0000","last_unstale":"2024-03-28T23:31:55.906881+0000","last_undegraded":"2024-03-28T23:31:55.906881+0000","last_fullsized":"2024-03-28T23:31:55.906881+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:07:27.190116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560629+0000","last_change":"2024-03-28T23:31:40.392391+0000","last_active":"2024-03-28T23:31:56.560629+0000","last_peered":"2024-03-28T23:31:56.560629+0000","last_clean":"2024-03-28T23:31:56.560629+0000","last_became_active":"2024-03-28T23:31:40.392100+0000","last_became_peered":"2024-03-28T23:31:40.392100+0000","last_unstale":"2024-03-28T23:31:56.560629+0000","last_undegraded":"2024-03-28T23:31:56.560629+0000","last_fullsized":"2024-03-28T23:31:56.560629+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:34:38.237729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.907667+0000","last_change":"2024-03-28T23:31:39.385430+0000","last_active":"2024-03-28T23:31:55.907667+0000","last_peered":"2024-03-28T23:31:55.907667+0000","last_clean":"2024-03-28T23:31:55.907667+0000","last_became_active":"2024-03-28T23:31:39.385134+0000","last_became_peered":"2024-03-28T23:31:39.385134+0000","last_unstale":"2024-03-28T23:31:55.907667+0000","last_undegraded":"2024-03-28T23:31:55.907667+0000","last_fullsized":"2024-03-28T23:31:55.907667+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:12:26.472924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570310+0000","last_change":"2024-03-28T23:31:54.901230+0000","last_active":"2024-03-28T23:31:56.570310+0000","last_peered":"2024-03-28T23:31:56.570310+0000","last_clean":"2024-03-28T23:31:56.570310+0000","last_became_active":"2024-03-28T23:31:54.900802+0000","last_became_peered":"2024-03-28T23:31:54.900802+0000","last_unstale":"2024-03-28T23:31:56.570310+0000","last_undegraded":"2024-03-28T23:31:56.570310+0000","last_fullsized":"2024-03-28T23:31:56.570310+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:40:24.369112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"54'3","reported_seq":17,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908379+0000","last_change":"2024-03-28T23:31:53.889140+0000","last_active":"2024-03-28T23:31:55.908379+0000","last_peered":"2024-03-28T23:31:55.908379+0000","last_clean":"2024-03-28T23:31:55.908379+0000","last_became_active":"2024-03-28T23:31:53.888738+0000","last_became_peered":"2024-03-28T23:31:53.888738+0000","last_unstale":"2024-03-28T23:31:55.908379+0000","last_undegraded":"2024-03-28T23:31:55.908379+0000","last_fullsized":"2024-03-28T23:31:55.908379+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:15:33.484701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.467996+0000","last_change":"2024-03-28T23:31:40.392757+0000","last_active":"2024-03-28T23:31:56.467996+0000","last_peered":"2024-03-28T23:31:56.467996+0000","last_clean":"2024-03-28T23:31:56.467996+0000","last_became_active":"2024-03-28T23:31:40.392460+0000","last_became_peered":"2024-03-28T23:31:40.392460+0000","last_unstale":"2024-03-28T23:31:56.467996+0000","last_undegraded":"2024-03-28T23:31:56.467996+0000","last_fullsized":"2024-03-28T23:31:56.467996+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:07.738674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908724+0000","last_change":"2024-03-28T23:31:39.381607+0000","last_active":"2024-03-28T23:31:55.908724+0000","last_peered":"2024-03-28T23:31:55.908724+0000","last_clean":"2024-03-28T23:31:55.908724+0000","last_became_active":"2024-03-28T23:31:39.381326+0000","last_became_peered":"2024-03-28T23:31:39.381326+0000","last_unstale":"2024-03-28T23:31:55.908724+0000","last_undegraded":"2024-03-28T23:31:55.908724+0000","last_fullsized":"2024-03-28T23:31:55.908724+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:50.860190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.469012+0000","last_change":"2024-03-28T23:31:54.917488+0000","last_active":"2024-03-28T23:31:56.469012+0000","last_peered":"2024-03-28T23:31:56.469012+0000","last_clean":"2024-03-28T23:31:56.469012+0000","last_became_active":"2024-03-28T23:31:54.917171+0000","last_became_peered":"2024-03-28T23:31:54.917171+0000","last_unstale":"2024-03-28T23:31:56.469012+0000","last_undegraded":"2024-03-28T23:31:56.469012+0000","last_fullsized":"2024-03-28T23:31:56.469012+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:07:28.191989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"54'6","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570049+0000","last_change":"2024-03-28T23:31:53.886424+0000","last_active":"2024-03-28T23:31:56.570049+0000","last_peered":"2024-03-28T23:31:56.570049+0000","last_clean":"2024-03-28T23:31:56.570049+0000","last_became_active":"2024-03-28T23:31:53.886207+0000","last_became_peered":"2024-03-28T23:31:53.886207+0000","last_unstale":"2024-03-28T23:31:56.570049+0000","last_undegraded":"2024-03-28T23:31:56.570049+0000","last_fullsized":"2024-03-28T23:31:56.570049+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:05:01.725546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908230+0000","last_change":"2024-03-28T23:31:40.398712+0000","last_active":"2024-03-28T23:31:55.908230+0000","last_peered":"2024-03-28T23:31:55.908230+0000","last_clean":"2024-03-28T23:31:55.908230+0000","last_became_active":"2024-03-28T23:31:40.398392+0000","last_became_peered":"2024-03-28T23:31:40.398392+0000","last_unstale":"2024-03-28T23:31:55.908230+0000","last_undegraded":"2024-03-28T23:31:55.908230+0000","last_fullsized":"2024-03-28T23:31:55.908230+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:54.431044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.468284+0000","last_change":"2024-03-28T23:31:39.384026+0000","last_active":"2024-03-28T23:31:56.468284+0000","last_peered":"2024-03-28T23:31:56.468284+0000","last_clean":"2024-03-28T23:31:56.468284+0000","last_became_active":"2024-03-28T23:31:39.383161+0000","last_became_peered":"2024-03-28T23:31:39.383161+0000","last_unstale":"2024-03-28T23:31:56.468284+0000","last_undegraded":"2024-03-28T23:31:56.468284+0000","last_fullsized":"2024-03-28T23:31:56.468284+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:21:00.890266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569847+0000","last_change":"2024-03-28T23:31:54.942374+0000","last_active":"2024-03-28T23:31:56.569847+0000","last_peered":"2024-03-28T23:31:56.569847+0000","last_clean":"2024-03-28T23:31:56.569847+0000","last_became_active":"2024-03-28T23:31:54.941764+0000","last_became_peered":"2024-03-28T23:31:54.941764+0000","last_unstale":"2024-03-28T23:31:56.569847+0000","last_undegraded":"2024-03-28T23:31:56.569847+0000","last_fullsized":"2024-03-28T23:31:56.569847+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:31:42.555242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908035+0000","last_change":"2024-03-28T23:31:53.885914+0000","last_active":"2024-03-28T23:31:55.908035+0000","last_peered":"2024-03-28T23:31:55.908035+0000","last_clean":"2024-03-28T23:31:55.908035+0000","last_became_active":"2024-03-28T23:31:53.885637+0000","last_became_peered":"2024-03-28T23:31:53.885637+0000","last_unstale":"2024-03-28T23:31:55.908035+0000","last_undegraded":"2024-03-28T23:31:55.908035+0000","last_fullsized":"2024-03-28T23:31:55.908035+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:53:39.854674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560866+0000","last_change":"2024-03-28T23:31:40.395728+0000","last_active":"2024-03-28T23:31:56.560866+0000","last_peered":"2024-03-28T23:31:56.560866+0000","last_clean":"2024-03-28T23:31:56.560866+0000","last_became_active":"2024-03-28T23:31:40.395474+0000","last_became_peered":"2024-03-28T23:31:40.395474+0000","last_unstale":"2024-03-28T23:31:56.560866+0000","last_undegraded":"2024-03-28T23:31:56.560866+0000","last_fullsized":"2024-03-28T23:31:56.560866+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:15:19.985376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569666+0000","last_change":"2024-03-28T23:31:39.385649+0000","last_active":"2024-03-28T23:31:56.569666+0000","last_peered":"2024-03-28T23:31:56.569666+0000","last_clean":"2024-03-28T23:31:56.569666+0000","last_became_active":"2024-03-28T23:31:39.385341+0000","last_became_peered":"2024-03-28T23:31:39.385341+0000","last_unstale":"2024-03-28T23:31:56.569666+0000","last_undegraded":"2024-03-28T23:31:56.569666+0000","last_fullsized":"2024-03-28T23:31:56.569666+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:52:06.955653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560499+0000","last_change":"2024-03-28T23:31:54.908138+0000","last_active":"2024-03-28T23:31:56.560499+0000","last_peered":"2024-03-28T23:31:56.560499+0000","last_clean":"2024-03-28T23:31:56.560499+0000","last_became_active":"2024-03-28T23:31:54.907885+0000","last_became_peered":"2024-03-28T23:31:54.907885+0000","last_unstale":"2024-03-28T23:31:56.560499+0000","last_undegraded":"2024-03-28T23:31:56.560499+0000","last_fullsized":"2024-03-28T23:31:56.560499+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:00.711022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.468827+0000","last_change":"2024-03-28T23:31:53.889258+0000","last_active":"2024-03-28T23:31:56.468827+0000","last_peered":"2024-03-28T23:31:56.468827+0000","last_clean":"2024-03-28T23:31:56.468827+0000","last_became_active":"2024-03-28T23:31:53.888970+0000","last_became_peered":"2024-03-28T23:31:53.888970+0000","last_unstale":"2024-03-28T23:31:56.468827+0000","last_undegraded":"2024-03-28T23:31:56.468827+0000","last_fullsized":"2024-03-28T23:31:56.468827+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:50:34.073974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908149+0000","last_change":"2024-03-28T23:31:40.400091+0000","last_active":"2024-03-28T23:31:55.908149+0000","last_peered":"2024-03-28T23:31:55.908149+0000","last_clean":"2024-03-28T23:31:55.908149+0000","last_became_active":"2024-03-28T23:31:40.399190+0000","last_became_peered":"2024-03-28T23:31:40.399190+0000","last_unstale":"2024-03-28T23:31:55.908149+0000","last_undegraded":"2024-03-28T23:31:55.908149+0000","last_fullsized":"2024-03-28T23:31:55.908149+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:22.230986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908319+0000","last_change":"2024-03-28T23:31:39.390198+0000","last_active":"2024-03-28T23:31:55.908319+0000","last_peered":"2024-03-28T23:31:55.908319+0000","last_clean":"2024-03-28T23:31:55.908319+0000","last_became_active":"2024-03-28T23:31:39.389939+0000","last_became_peered":"2024-03-28T23:31:39.389939+0000","last_unstale":"2024-03-28T23:31:55.908319+0000","last_undegraded":"2024-03-28T23:31:55.908319+0000","last_fullsized":"2024-03-28T23:31:55.908319+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:46:30.350009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569860+0000","last_change":"2024-03-28T23:31:54.906355+0000","last_active":"2024-03-28T23:31:56.569860+0000","last_peered":"2024-03-28T23:31:56.569860+0000","last_clean":"2024-03-28T23:31:56.569860+0000","last_became_active":"2024-03-28T23:31:54.906057+0000","last_became_peered":"2024-03-28T23:31:54.906057+0000","last_unstale":"2024-03-28T23:31:56.569860+0000","last_undegraded":"2024-03-28T23:31:56.569860+0000","last_fullsized":"2024-03-28T23:31:56.569860+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:38:10.372154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560317+0000","last_change":"2024-03-28T23:31:53.892143+0000","last_active":"2024-03-28T23:31:56.560317+0000","last_peered":"2024-03-28T23:31:56.560317+0000","last_clean":"2024-03-28T23:31:56.560317+0000","last_became_active":"2024-03-28T23:31:53.891989+0000","last_became_peered":"2024-03-28T23:31:53.891989+0000","last_unstale":"2024-03-28T23:31:56.560317+0000","last_undegraded":"2024-03-28T23:31:56.560317+0000","last_fullsized":"2024-03-28T23:31:56.560317+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:17:23.435865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.468471+0000","last_change":"2024-03-28T23:31:40.384948+0000","last_active":"2024-03-28T23:31:56.468471+0000","last_peered":"2024-03-28T23:31:56.468471+0000","last_clean":"2024-03-28T23:31:56.468471+0000","last_became_active":"2024-03-28T23:31:40.384434+0000","last_became_peered":"2024-03-28T23:31:40.384434+0000","last_unstale":"2024-03-28T23:31:56.468471+0000","last_undegraded":"2024-03-28T23:31:56.468471+0000","last_fullsized":"2024-03-28T23:31:56.468471+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:04.829030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908184+0000","last_change":"2024-03-28T23:31:39.388914+0000","last_active":"2024-03-28T23:31:55.908184+0000","last_peered":"2024-03-28T23:31:55.908184+0000","last_clean":"2024-03-28T23:31:55.908184+0000","last_became_active":"2024-03-28T23:31:39.388639+0000","last_became_peered":"2024-03-28T23:31:39.388639+0000","last_unstale":"2024-03-28T23:31:55.908184+0000","last_undegraded":"2024-03-28T23:31:55.908184+0000","last_fullsized":"2024-03-28T23:31:55.908184+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:00.366278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560876+0000","last_change":"2024-03-28T23:31:54.914109+0000","last_active":"2024-03-28T23:31:56.560876+0000","last_peered":"2024-03-28T23:31:56.560876+0000","last_clean":"2024-03-28T23:31:56.560876+0000","last_became_active":"2024-03-28T23:31:54.913815+0000","last_became_peered":"2024-03-28T23:31:54.913815+0000","last_unstale":"2024-03-28T23:31:56.560876+0000","last_undegraded":"2024-03-28T23:31:56.560876+0000","last_fullsized":"2024-03-28T23:31:56.560876+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:12:47.701910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569538+0000","last_change":"2024-03-28T23:31:53.879950+0000","last_active":"2024-03-28T23:31:56.569538+0000","last_peered":"2024-03-28T23:31:56.569538+0000","last_clean":"2024-03-28T23:31:56.569538+0000","last_became_active":"2024-03-28T23:31:53.879620+0000","last_became_peered":"2024-03-28T23:31:53.879620+0000","last_unstale":"2024-03-28T23:31:56.569538+0000","last_undegraded":"2024-03-28T23:31:56.569538+0000","last_fullsized":"2024-03-28T23:31:56.569538+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:20:01.487078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908113+0000","last_change":"2024-03-28T23:31:40.386749+0000","last_active":"2024-03-28T23:31:55.908113+0000","last_peered":"2024-03-28T23:31:55.908113+0000","last_clean":"2024-03-28T23:31:55.908113+0000","last_became_active":"2024-03-28T23:31:40.386466+0000","last_became_peered":"2024-03-28T23:31:40.386466+0000","last_unstale":"2024-03-28T23:31:55.908113+0000","last_undegraded":"2024-03-28T23:31:55.908113+0000","last_fullsized":"2024-03-28T23:31:55.908113+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:06.567694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.469135+0000","last_change":"2024-03-28T23:31:39.381050+0000","last_active":"2024-03-28T23:31:56.469135+0000","last_peered":"2024-03-28T23:31:56.469135+0000","last_clean":"2024-03-28T23:31:56.469135+0000","last_became_active":"2024-03-28T23:31:39.380761+0000","last_became_peered":"2024-03-28T23:31:39.380761+0000","last_unstale":"2024-03-28T23:31:56.469135+0000","last_undegraded":"2024-03-28T23:31:56.469135+0000","last_fullsized":"2024-03-28T23:31:56.469135+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:29.240985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.568986+0000","last_change":"2024-03-28T23:31:54.942343+0000","last_active":"2024-03-28T23:31:56.568986+0000","last_peered":"2024-03-28T23:31:56.568986+0000","last_clean":"2024-03-28T23:31:56.568986+0000","last_became_active":"2024-03-28T23:31:54.942075+0000","last_became_peered":"2024-03-28T23:31:54.942075+0000","last_unstale":"2024-03-28T23:31:56.568986+0000","last_undegraded":"2024-03-28T23:31:56.568986+0000","last_fullsized":"2024-03-28T23:31:56.568986+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:56.936565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560527+0000","last_change":"2024-03-28T23:31:53.889906+0000","last_active":"2024-03-28T23:31:56.560527+0000","last_peered":"2024-03-28T23:31:56.560527+0000","last_clean":"2024-03-28T23:31:56.560527+0000","last_became_active":"2024-03-28T23:31:53.889589+0000","last_became_peered":"2024-03-28T23:31:53.889589+0000","last_unstale":"2024-03-28T23:31:56.560527+0000","last_undegraded":"2024-03-28T23:31:56.560527+0000","last_fullsized":"2024-03-28T23:31:56.560527+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:55:44.017225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908727+0000","last_change":"2024-03-28T23:31:40.400388+0000","last_active":"2024-03-28T23:31:55.908727+0000","last_peered":"2024-03-28T23:31:55.908727+0000","last_clean":"2024-03-28T23:31:55.908727+0000","last_became_active":"2024-03-28T23:31:40.400133+0000","last_became_peered":"2024-03-28T23:31:40.400133+0000","last_unstale":"2024-03-28T23:31:55.908727+0000","last_undegraded":"2024-03-28T23:31:55.908727+0000","last_fullsized":"2024-03-28T23:31:55.908727+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:03:56.252539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"50'3","reported_seq":25,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908324+0000","last_change":"2024-03-28T23:31:39.386478+0000","last_active":"2024-03-28T23:31:55.908324+0000","last_peered":"2024-03-28T23:31:55.908324+0000","last_clean":"2024-03-28T23:31:55.908324+0000","last_became_active":"2024-03-28T23:31:39.385868+0000","last_became_peered":"2024-03-28T23:31:39.385868+0000","last_unstale":"2024-03-28T23:31:55.908324+0000","last_undegraded":"2024-03-28T23:31:55.908324+0000","last_fullsized":"2024-03-28T23:31:55.908324+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:06.733879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570400+0000","last_change":"2024-03-28T23:31:54.913804+0000","last_active":"2024-03-28T23:31:56.570400+0000","last_peered":"2024-03-28T23:31:56.570400+0000","last_clean":"2024-03-28T23:31:56.570400+0000","last_became_active":"2024-03-28T23:31:54.913546+0000","last_became_peered":"2024-03-28T23:31:54.913546+0000","last_unstale":"2024-03-28T23:31:56.570400+0000","last_undegraded":"2024-03-28T23:31:56.570400+0000","last_fullsized":"2024-03-28T23:31:56.570400+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:40:58.387549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560585+0000","last_change":"2024-03-28T23:31:53.889887+0000","last_active":"2024-03-28T23:31:56.560585+0000","last_peered":"2024-03-28T23:31:56.560585+0000","last_clean":"2024-03-28T23:31:56.560585+0000","last_became_active":"2024-03-28T23:31:53.889635+0000","last_became_peered":"2024-03-28T23:31:53.889635+0000","last_unstale":"2024-03-28T23:31:56.560585+0000","last_undegraded":"2024-03-28T23:31:56.560585+0000","last_fullsized":"2024-03-28T23:31:56.560585+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:05.324732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.468702+0000","last_change":"2024-03-28T23:31:40.387795+0000","last_active":"2024-03-28T23:31:56.468702+0000","last_peered":"2024-03-28T23:31:56.468702+0000","last_clean":"2024-03-28T23:31:56.468702+0000","last_became_active":"2024-03-28T23:31:40.387529+0000","last_became_peered":"2024-03-28T23:31:40.387529+0000","last_unstale":"2024-03-28T23:31:56.468702+0000","last_undegraded":"2024-03-28T23:31:56.468702+0000","last_fullsized":"2024-03-28T23:31:56.468702+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:51.864986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"50'6","reported_seq":28,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560679+0000","last_change":"2024-03-28T23:31:39.393411+0000","last_active":"2024-03-28T23:31:56.560679+0000","last_peered":"2024-03-28T23:31:56.560679+0000","last_clean":"2024-03-28T23:31:56.560679+0000","last_became_active":"2024-03-28T23:31:39.393166+0000","last_became_peered":"2024-03-28T23:31:39.393166+0000","last_unstale":"2024-03-28T23:31:56.560679+0000","last_undegraded":"2024-03-28T23:31:56.560679+0000","last_fullsized":"2024-03-28T23:31:56.560679+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:13.510273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.909490+0000","last_change":"2024-03-28T23:31:54.944748+0000","last_active":"2024-03-28T23:31:55.909490+0000","last_peered":"2024-03-28T23:31:55.909490+0000","last_clean":"2024-03-28T23:31:55.909490+0000","last_became_active":"2024-03-28T23:31:54.944312+0000","last_became_peered":"2024-03-28T23:31:54.944312+0000","last_unstale":"2024-03-28T23:31:55.909490+0000","last_undegraded":"2024-03-28T23:31:55.909490+0000","last_fullsized":"2024-03-28T23:31:55.909490+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:40:13.121751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.909118+0000","last_change":"2024-03-28T23:31:53.889399+0000","last_active":"2024-03-28T23:31:55.909118+0000","last_peered":"2024-03-28T23:31:55.909118+0000","last_clean":"2024-03-28T23:31:55.909118+0000","last_became_active":"2024-03-28T23:31:53.889110+0000","last_became_peered":"2024-03-28T23:31:53.889110+0000","last_unstale":"2024-03-28T23:31:55.909118+0000","last_undegraded":"2024-03-28T23:31:55.909118+0000","last_fullsized":"2024-03-28T23:31:55.909118+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:19:23.382549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.907465+0000","last_change":"2024-03-28T23:31:40.390497+0000","last_active":"2024-03-28T23:31:55.907465+0000","last_peered":"2024-03-28T23:31:55.907465+0000","last_clean":"2024-03-28T23:31:55.907465+0000","last_became_active":"2024-03-28T23:31:40.390167+0000","last_became_peered":"2024-03-28T23:31:40.390167+0000","last_unstale":"2024-03-28T23:31:55.907465+0000","last_undegraded":"2024-03-28T23:31:55.907465+0000","last_fullsized":"2024-03-28T23:31:55.907465+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:54.431044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908830+0000","last_change":"2024-03-28T23:31:39.380369+0000","last_active":"2024-03-28T23:31:55.908830+0000","last_peered":"2024-03-28T23:31:55.908830+0000","last_clean":"2024-03-28T23:31:55.908830+0000","last_became_active":"2024-03-28T23:31:39.380092+0000","last_became_peered":"2024-03-28T23:31:39.380092+0000","last_unstale":"2024-03-28T23:31:55.908830+0000","last_undegraded":"2024-03-28T23:31:55.908830+0000","last_fullsized":"2024-03-28T23:31:55.908830+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:00.366278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.907918+0000","last_change":"2024-03-28T23:31:54.910254+0000","last_active":"2024-03-28T23:31:55.907918+0000","last_peered":"2024-03-28T23:31:55.907918+0000","last_clean":"2024-03-28T23:31:55.907918+0000","last_became_active":"2024-03-28T23:31:54.909956+0000","last_became_peered":"2024-03-28T23:31:54.909956+0000","last_unstale":"2024-03-28T23:31:55.907918+0000","last_undegraded":"2024-03-28T23:31:55.907918+0000","last_fullsized":"2024-03-28T23:31:55.907918+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:55:45.019098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908871+0000","last_change":"2024-03-28T23:31:53.883362+0000","last_active":"2024-03-28T23:31:55.908871+0000","last_peered":"2024-03-28T23:31:55.908871+0000","last_clean":"2024-03-28T23:31:55.908871+0000","last_became_active":"2024-03-28T23:31:53.883123+0000","last_became_peered":"2024-03-28T23:31:53.883123+0000","last_unstale":"2024-03-28T23:31:55.908871+0000","last_undegraded":"2024-03-28T23:31:55.908871+0000","last_fullsized":"2024-03-28T23:31:55.908871+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:10:22.963954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.909324+0000","last_change":"2024-03-28T23:31:40.390579+0000","last_active":"2024-03-28T23:31:55.909324+0000","last_peered":"2024-03-28T23:31:55.909324+0000","last_clean":"2024-03-28T23:31:55.909324+0000","last_became_active":"2024-03-28T23:31:40.390248+0000","last_became_peered":"2024-03-28T23:31:40.390248+0000","last_unstale":"2024-03-28T23:31:55.909324+0000","last_undegraded":"2024-03-28T23:31:55.909324+0000","last_fullsized":"2024-03-28T23:31:55.909324+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:47.861300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570054+0000","last_change":"2024-03-28T23:31:39.389592+0000","last_active":"2024-03-28T23:31:56.570054+0000","last_peered":"2024-03-28T23:31:56.570054+0000","last_clean":"2024-03-28T23:31:56.570054+0000","last_became_active":"2024-03-28T23:31:39.389302+0000","last_became_peered":"2024-03-28T23:31:39.389302+0000","last_unstale":"2024-03-28T23:31:56.570054+0000","last_undegraded":"2024-03-28T23:31:56.570054+0000","last_fullsized":"2024-03-28T23:31:56.570054+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:59:51.501145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569981+0000","last_change":"2024-03-28T23:31:54.908011+0000","last_active":"2024-03-28T23:31:56.569981+0000","last_peered":"2024-03-28T23:31:56.569981+0000","last_clean":"2024-03-28T23:31:56.569981+0000","last_became_active":"2024-03-28T23:31:54.907736+0000","last_became_peered":"2024-03-28T23:31:54.907736+0000","last_unstale":"2024-03-28T23:31:56.569981+0000","last_undegraded":"2024-03-28T23:31:56.569981+0000","last_fullsized":"2024-03-28T23:31:56.569981+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:05:02.727419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"54'4","reported_seq":18,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.571132+0000","last_change":"2024-03-28T23:31:53.881795+0000","last_active":"2024-03-28T23:31:56.571132+0000","last_peered":"2024-03-28T23:31:56.571132+0000","last_clean":"2024-03-28T23:31:56.571132+0000","last_became_active":"2024-03-28T23:31:53.881552+0000","last_became_peered":"2024-03-28T23:31:53.881552+0000","last_unstale":"2024-03-28T23:31:56.571132+0000","last_undegraded":"2024-03-28T23:31:56.571132+0000","last_fullsized":"2024-03-28T23:31:56.571132+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:26:16.561871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560219+0000","last_change":"2024-03-28T23:31:40.387085+0000","last_active":"2024-03-28T23:31:56.560219+0000","last_peered":"2024-03-28T23:31:56.560219+0000","last_clean":"2024-03-28T23:31:56.560219+0000","last_became_active":"2024-03-28T23:31:40.386695+0000","last_became_peered":"2024-03-28T23:31:40.386695+0000","last_unstale":"2024-03-28T23:31:56.560219+0000","last_undegraded":"2024-03-28T23:31:56.560219+0000","last_fullsized":"2024-03-28T23:31:56.560219+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:40:57.541676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.907750+0000","last_change":"2024-03-28T23:31:39.384653+0000","last_active":"2024-03-28T23:31:55.907750+0000","last_peered":"2024-03-28T23:31:55.907750+0000","last_clean":"2024-03-28T23:31:55.907750+0000","last_became_active":"2024-03-28T23:31:39.383495+0000","last_became_peered":"2024-03-28T23:31:39.383495+0000","last_unstale":"2024-03-28T23:31:55.907750+0000","last_undegraded":"2024-03-28T23:31:55.907750+0000","last_fullsized":"2024-03-28T23:31:55.907750+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:50.860190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.468886+0000","last_change":"2024-03-28T23:31:54.918331+0000","last_active":"2024-03-28T23:31:56.468886+0000","last_peered":"2024-03-28T23:31:56.468886+0000","last_clean":"2024-03-28T23:31:56.468886+0000","last_became_active":"2024-03-28T23:31:54.918066+0000","last_became_peered":"2024-03-28T23:31:54.918066+0000","last_unstale":"2024-03-28T23:31:56.468886+0000","last_undegraded":"2024-03-28T23:31:56.468886+0000","last_fullsized":"2024-03-28T23:31:56.468886+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:05:02.727419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.909532+0000","last_change":"2024-03-28T23:31:53.884651+0000","last_active":"2024-03-28T23:31:55.909532+0000","last_peered":"2024-03-28T23:31:55.909532+0000","last_clean":"2024-03-28T23:31:55.909532+0000","last_became_active":"2024-03-28T23:31:53.884410+0000","last_became_peered":"2024-03-28T23:31:53.884410+0000","last_unstale":"2024-03-28T23:31:55.909532+0000","last_undegraded":"2024-03-28T23:31:55.909532+0000","last_fullsized":"2024-03-28T23:31:55.909532+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:23:27.129147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.909617+0000","last_change":"2024-03-28T23:31:40.403701+0000","last_active":"2024-03-28T23:31:55.909617+0000","last_peered":"2024-03-28T23:31:55.909617+0000","last_clean":"2024-03-28T23:31:55.909617+0000","last_became_active":"2024-03-28T23:31:40.403222+0000","last_became_peered":"2024-03-28T23:31:40.403222+0000","last_unstale":"2024-03-28T23:31:55.909617+0000","last_undegraded":"2024-03-28T23:31:55.909617+0000","last_fullsized":"2024-03-28T23:31:55.909617+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:40:09.867914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"50'8","reported_seq":30,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560518+0000","last_change":"2024-03-28T23:31:39.383726+0000","last_active":"2024-03-28T23:31:56.560518+0000","last_peered":"2024-03-28T23:31:56.560518+0000","last_clean":"2024-03-28T23:31:56.560518+0000","last_became_active":"2024-03-28T23:31:39.383292+0000","last_became_peered":"2024-03-28T23:31:39.383292+0000","last_unstale":"2024-03-28T23:31:56.560518+0000","last_undegraded":"2024-03-28T23:31:56.560518+0000","last_fullsized":"2024-03-28T23:31:56.560518+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:25:38.113638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570296+0000","last_change":"2024-03-28T23:31:54.916130+0000","last_active":"2024-03-28T23:31:56.570296+0000","last_peered":"2024-03-28T23:31:56.570296+0000","last_clean":"2024-03-28T23:31:56.570296+0000","last_became_active":"2024-03-28T23:31:54.915866+0000","last_became_peered":"2024-03-28T23:31:54.915866+0000","last_unstale":"2024-03-28T23:31:56.570296+0000","last_undegraded":"2024-03-28T23:31:56.570296+0000","last_fullsized":"2024-03-28T23:31:56.570296+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:49.479859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560359+0000","last_change":"2024-03-28T23:31:53.892246+0000","last_active":"2024-03-28T23:31:56.560359+0000","last_peered":"2024-03-28T23:31:56.560359+0000","last_clean":"2024-03-28T23:31:56.560359+0000","last_became_active":"2024-03-28T23:31:53.891974+0000","last_became_peered":"2024-03-28T23:31:53.891974+0000","last_unstale":"2024-03-28T23:31:56.560359+0000","last_undegraded":"2024-03-28T23:31:56.560359+0000","last_fullsized":"2024-03-28T23:31:56.560359+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:40:21.831227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560040+0000","last_change":"2024-03-28T23:31:40.391219+0000","last_active":"2024-03-28T23:31:56.560040+0000","last_peered":"2024-03-28T23:31:56.560040+0000","last_clean":"2024-03-28T23:31:56.560040+0000","last_became_active":"2024-03-28T23:31:40.389552+0000","last_became_peered":"2024-03-28T23:31:40.389552+0000","last_unstale":"2024-03-28T23:31:56.560040+0000","last_undegraded":"2024-03-28T23:31:56.560040+0000","last_fullsized":"2024-03-28T23:31:56.560040+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:42:29.351941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.909911+0000","last_change":"2024-03-28T23:31:39.390253+0000","last_active":"2024-03-28T23:31:55.909911+0000","last_peered":"2024-03-28T23:31:55.909911+0000","last_clean":"2024-03-28T23:31:55.909911+0000","last_became_active":"2024-03-28T23:31:39.389980+0000","last_became_peered":"2024-03-28T23:31:39.389980+0000","last_unstale":"2024-03-28T23:31:55.909911+0000","last_undegraded":"2024-03-28T23:31:55.909911+0000","last_fullsized":"2024-03-28T23:31:55.909911+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:24.520296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569421+0000","last_change":"2024-03-28T23:31:54.942714+0000","last_active":"2024-03-28T23:31:56.569421+0000","last_peered":"2024-03-28T23:31:56.569421+0000","last_clean":"2024-03-28T23:31:56.569421+0000","last_became_active":"2024-03-28T23:31:54.942437+0000","last_became_peered":"2024-03-28T23:31:54.942437+0000","last_unstale":"2024-03-28T23:31:56.569421+0000","last_undegraded":"2024-03-28T23:31:56.569421+0000","last_fullsized":"2024-03-28T23:31:56.569421+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:28.011376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569876+0000","last_change":"2024-03-28T23:31:53.880659+0000","last_active":"2024-03-28T23:31:56.569876+0000","last_peered":"2024-03-28T23:31:56.569876+0000","last_clean":"2024-03-28T23:31:56.569876+0000","last_became_active":"2024-03-28T23:31:53.880265+0000","last_became_peered":"2024-03-28T23:31:53.880265+0000","last_unstale":"2024-03-28T23:31:56.569876+0000","last_undegraded":"2024-03-28T23:31:56.569876+0000","last_fullsized":"2024-03-28T23:31:56.569876+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:38:09.370282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570205+0000","last_change":"2024-03-28T23:31:40.391898+0000","last_active":"2024-03-28T23:31:56.570205+0000","last_peered":"2024-03-28T23:31:56.570205+0000","last_clean":"2024-03-28T23:31:56.570205+0000","last_became_active":"2024-03-28T23:31:40.391622+0000","last_became_peered":"2024-03-28T23:31:40.391622+0000","last_unstale":"2024-03-28T23:31:56.570205+0000","last_undegraded":"2024-03-28T23:31:56.570205+0000","last_fullsized":"2024-03-28T23:31:56.570205+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:38:22.055287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570412+0000","last_change":"2024-03-28T23:31:39.385339+0000","last_active":"2024-03-28T23:31:56.570412+0000","last_peered":"2024-03-28T23:31:56.570412+0000","last_clean":"2024-03-28T23:31:56.570412+0000","last_became_active":"2024-03-28T23:31:39.385075+0000","last_became_peered":"2024-03-28T23:31:39.385075+0000","last_unstale":"2024-03-28T23:31:56.570412+0000","last_undegraded":"2024-03-28T23:31:56.570412+0000","last_fullsized":"2024-03-28T23:31:56.570412+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:58.098710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569416+0000","last_change":"2024-03-28T23:31:54.907121+0000","last_active":"2024-03-28T23:31:56.569416+0000","last_peered":"2024-03-28T23:31:56.569416+0000","last_clean":"2024-03-28T23:31:56.569416+0000","last_became_active":"2024-03-28T23:31:54.906862+0000","last_became_peered":"2024-03-28T23:31:54.906862+0000","last_unstale":"2024-03-28T23:31:56.569416+0000","last_undegraded":"2024-03-28T23:31:56.569416+0000","last_fullsized":"2024-03-28T23:31:56.569416+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:55:36.226436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.907202+0000","last_change":"2024-03-28T23:31:53.890901+0000","last_active":"2024-03-28T23:31:55.907202+0000","last_peered":"2024-03-28T23:31:55.907202+0000","last_clean":"2024-03-28T23:31:55.907202+0000","last_became_active":"2024-03-28T23:31:53.890643+0000","last_became_peered":"2024-03-28T23:31:53.890643+0000","last_unstale":"2024-03-28T23:31:55.907202+0000","last_undegraded":"2024-03-28T23:31:55.907202+0000","last_fullsized":"2024-03-28T23:31:55.907202+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:41:11.041002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.467946+0000","last_change":"2024-03-28T23:31:40.399291+0000","last_active":"2024-03-28T23:31:56.467946+0000","last_peered":"2024-03-28T23:31:56.467946+0000","last_clean":"2024-03-28T23:31:56.467946+0000","last_became_active":"2024-03-28T23:31:40.399005+0000","last_became_peered":"2024-03-28T23:31:40.399005+0000","last_unstale":"2024-03-28T23:31:56.467946+0000","last_undegraded":"2024-03-28T23:31:56.467946+0000","last_fullsized":"2024-03-28T23:31:56.467946+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:01.371074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570355+0000","last_change":"2024-03-28T23:31:39.389465+0000","last_active":"2024-03-28T23:31:56.570355+0000","last_peered":"2024-03-28T23:31:56.570355+0000","last_clean":"2024-03-28T23:31:56.570355+0000","last_became_active":"2024-03-28T23:31:39.389005+0000","last_became_peered":"2024-03-28T23:31:39.389005+0000","last_unstale":"2024-03-28T23:31:56.570355+0000","last_undegraded":"2024-03-28T23:31:56.570355+0000","last_fullsized":"2024-03-28T23:31:56.570355+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:41.430571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.468180+0000","last_change":"2024-03-28T23:31:54.919753+0000","last_active":"2024-03-28T23:31:56.468180+0000","last_peered":"2024-03-28T23:31:56.468180+0000","last_clean":"2024-03-28T23:31:56.468180+0000","last_became_active":"2024-03-28T23:31:54.919513+0000","last_became_peered":"2024-03-28T23:31:54.919513+0000","last_unstale":"2024-03-28T23:31:56.468180+0000","last_undegraded":"2024-03-28T23:31:56.468180+0000","last_fullsized":"2024-03-28T23:31:56.468180+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:33.284318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560626+0000","last_change":"2024-03-28T23:31:53.888704+0000","last_active":"2024-03-28T23:31:56.560626+0000","last_peered":"2024-03-28T23:31:56.560626+0000","last_clean":"2024-03-28T23:31:56.560626+0000","last_became_active":"2024-03-28T23:31:53.888461+0000","last_became_peered":"2024-03-28T23:31:53.888461+0000","last_unstale":"2024-03-28T23:31:56.560626+0000","last_undegraded":"2024-03-28T23:31:56.560626+0000","last_fullsized":"2024-03-28T23:31:56.560626+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:10.934859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569124+0000","last_change":"2024-03-28T23:31:40.388141+0000","last_active":"2024-03-28T23:31:56.569124+0000","last_peered":"2024-03-28T23:31:56.569124+0000","last_clean":"2024-03-28T23:31:56.569124+0000","last_became_active":"2024-03-28T23:31:40.387852+0000","last_became_peered":"2024-03-28T23:31:40.387852+0000","last_unstale":"2024-03-28T23:31:56.569124+0000","last_undegraded":"2024-03-28T23:31:56.569124+0000","last_fullsized":"2024-03-28T23:31:56.569124+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:22.230986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570281+0000","last_change":"2024-03-28T23:31:39.390465+0000","last_active":"2024-03-28T23:31:56.570281+0000","last_peered":"2024-03-28T23:31:56.570281+0000","last_clean":"2024-03-28T23:31:56.570281+0000","last_became_active":"2024-03-28T23:31:39.390203+0000","last_became_peered":"2024-03-28T23:31:39.390203+0000","last_unstale":"2024-03-28T23:31:56.570281+0000","last_undegraded":"2024-03-28T23:31:56.570281+0000","last_fullsized":"2024-03-28T23:31:56.570281+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:50:50.496294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908485+0000","last_change":"2024-03-28T23:31:54.904775+0000","last_active":"2024-03-28T23:31:55.908485+0000","last_peered":"2024-03-28T23:31:55.908485+0000","last_clean":"2024-03-28T23:31:55.908485+0000","last_became_active":"2024-03-28T23:31:54.904536+0000","last_became_peered":"2024-03-28T23:31:54.904536+0000","last_unstale":"2024-03-28T23:31:55.908485+0000","last_undegraded":"2024-03-28T23:31:55.908485+0000","last_fullsized":"2024-03-28T23:31:55.908485+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:11.936731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.909170+0000","last_change":"2024-03-28T23:31:53.888267+0000","last_active":"2024-03-28T23:31:55.909170+0000","last_peered":"2024-03-28T23:31:55.909170+0000","last_clean":"2024-03-28T23:31:55.909170+0000","last_became_active":"2024-03-28T23:31:53.887993+0000","last_became_peered":"2024-03-28T23:31:53.887993+0000","last_unstale":"2024-03-28T23:31:55.909170+0000","last_undegraded":"2024-03-28T23:31:55.909170+0000","last_fullsized":"2024-03-28T23:31:55.909170+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:31:25.647111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.907574+0000","last_change":"2024-03-28T23:31:40.387376+0000","last_active":"2024-03-28T23:31:55.907574+0000","last_peered":"2024-03-28T23:31:55.907574+0000","last_clean":"2024-03-28T23:31:55.907574+0000","last_became_active":"2024-03-28T23:31:40.387080+0000","last_became_peered":"2024-03-28T23:31:40.387080+0000","last_unstale":"2024-03-28T23:31:55.907574+0000","last_undegraded":"2024-03-28T23:31:55.907574+0000","last_fullsized":"2024-03-28T23:31:55.907574+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:43.111058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569804+0000","last_change":"2024-03-28T23:31:39.388867+0000","last_active":"2024-03-28T23:31:56.569804+0000","last_peered":"2024-03-28T23:31:56.569804+0000","last_clean":"2024-03-28T23:31:56.569804+0000","last_became_active":"2024-03-28T23:31:39.388593+0000","last_became_peered":"2024-03-28T23:31:39.388593+0000","last_unstale":"2024-03-28T23:31:56.569804+0000","last_undegraded":"2024-03-28T23:31:56.569804+0000","last_fullsized":"2024-03-28T23:31:56.569804+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:21:00.890266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570477+0000","last_change":"2024-03-28T23:31:54.908763+0000","last_active":"2024-03-28T23:31:56.570477+0000","last_peered":"2024-03-28T23:31:56.570477+0000","last_clean":"2024-03-28T23:31:56.570477+0000","last_became_active":"2024-03-28T23:31:54.908468+0000","last_became_peered":"2024-03-28T23:31:54.908468+0000","last_unstale":"2024-03-28T23:31:56.570477+0000","last_undegraded":"2024-03-28T23:31:56.570477+0000","last_fullsized":"2024-03-28T23:31:56.570477+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:41:12.042874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570147+0000","last_change":"2024-03-28T23:31:53.886433+0000","last_active":"2024-03-28T23:31:56.570147+0000","last_peered":"2024-03-28T23:31:56.570147+0000","last_clean":"2024-03-28T23:31:56.570147+0000","last_became_active":"2024-03-28T23:31:53.886218+0000","last_became_peered":"2024-03-28T23:31:53.886218+0000","last_unstale":"2024-03-28T23:31:56.570147+0000","last_undegraded":"2024-03-28T23:31:56.570147+0000","last_fullsized":"2024-03-28T23:31:56.570147+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:01.360626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570236+0000","last_change":"2024-03-28T23:31:40.393257+0000","last_active":"2024-03-28T23:31:56.570236+0000","last_peered":"2024-03-28T23:31:56.570236+0000","last_clean":"2024-03-28T23:31:56.570236+0000","last_became_active":"2024-03-28T23:31:40.391684+0000","last_became_peered":"2024-03-28T23:31:40.391684+0000","last_unstale":"2024-03-28T23:31:56.570236+0000","last_undegraded":"2024-03-28T23:31:56.570236+0000","last_fullsized":"2024-03-28T23:31:56.570236+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:01.371074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560713+0000","last_change":"2024-03-28T23:31:39.388198+0000","last_active":"2024-03-28T23:31:56.560713+0000","last_peered":"2024-03-28T23:31:56.560713+0000","last_clean":"2024-03-28T23:31:56.560713+0000","last_became_active":"2024-03-28T23:31:39.387727+0000","last_became_peered":"2024-03-28T23:31:39.387727+0000","last_unstale":"2024-03-28T23:31:56.560713+0000","last_undegraded":"2024-03-28T23:31:56.560713+0000","last_fullsized":"2024-03-28T23:31:56.560713+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:00:49.868091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908913+0000","last_change":"2024-03-28T23:31:54.919652+0000","last_active":"2024-03-28T23:31:55.908913+0000","last_peered":"2024-03-28T23:31:55.908913+0000","last_clean":"2024-03-28T23:31:55.908913+0000","last_became_active":"2024-03-28T23:31:54.919367+0000","last_became_peered":"2024-03-28T23:31:54.919367+0000","last_unstale":"2024-03-28T23:31:55.908913+0000","last_undegraded":"2024-03-28T23:31:55.908913+0000","last_fullsized":"2024-03-28T23:31:55.908913+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:01:31.329920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908179+0000","last_change":"2024-03-28T23:31:53.884698+0000","last_active":"2024-03-28T23:31:55.908179+0000","last_peered":"2024-03-28T23:31:55.908179+0000","last_clean":"2024-03-28T23:31:55.908179+0000","last_became_active":"2024-03-28T23:31:53.884413+0000","last_became_peered":"2024-03-28T23:31:53.884413+0000","last_unstale":"2024-03-28T23:31:55.908179+0000","last_undegraded":"2024-03-28T23:31:55.908179+0000","last_fullsized":"2024-03-28T23:31:55.908179+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:41:11.041002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.909640+0000","last_change":"2024-03-28T23:31:40.394047+0000","last_active":"2024-03-28T23:31:55.909640+0000","last_peered":"2024-03-28T23:31:55.909640+0000","last_clean":"2024-03-28T23:31:55.909640+0000","last_became_active":"2024-03-28T23:31:40.393772+0000","last_became_peered":"2024-03-28T23:31:40.393772+0000","last_unstale":"2024-03-28T23:31:55.909640+0000","last_undegraded":"2024-03-28T23:31:55.909640+0000","last_fullsized":"2024-03-28T23:31:55.909640+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:07:13.690791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908439+0000","last_change":"2024-03-28T23:31:39.388829+0000","last_active":"2024-03-28T23:31:55.908439+0000","last_peered":"2024-03-28T23:31:55.908439+0000","last_clean":"2024-03-28T23:31:55.908439+0000","last_became_active":"2024-03-28T23:31:39.388521+0000","last_became_peered":"2024-03-28T23:31:39.388521+0000","last_unstale":"2024-03-28T23:31:55.908439+0000","last_undegraded":"2024-03-28T23:31:55.908439+0000","last_fullsized":"2024-03-28T23:31:55.908439+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:23:52.148574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569557+0000","last_change":"2024-03-28T23:31:54.944476+0000","last_active":"2024-03-28T23:31:56.569557+0000","last_peered":"2024-03-28T23:31:56.569557+0000","last_clean":"2024-03-28T23:31:56.569557+0000","last_became_active":"2024-03-28T23:31:54.944238+0000","last_became_peered":"2024-03-28T23:31:54.944238+0000","last_unstale":"2024-03-28T23:31:56.569557+0000","last_undegraded":"2024-03-28T23:31:56.569557+0000","last_fullsized":"2024-03-28T23:31:56.569557+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:31:59.479190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"55'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908164+0000","last_change":"2024-03-28T23:31:53.886155+0000","last_active":"2024-03-28T23:31:55.908164+0000","last_peered":"2024-03-28T23:31:55.908164+0000","last_clean":"2024-03-28T23:31:55.908164+0000","last_became_active":"2024-03-28T23:31:53.885843+0000","last_became_peered":"2024-03-28T23:31:53.885843+0000","last_unstale":"2024-03-28T23:31:55.908164+0000","last_undegraded":"2024-03-28T23:31:55.908164+0000","last_fullsized":"2024-03-28T23:31:55.908164+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:12:51.741434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.571253+0000","last_change":"2024-03-28T23:31:40.384709+0000","last_active":"2024-03-28T23:31:56.571253+0000","last_peered":"2024-03-28T23:31:56.571253+0000","last_clean":"2024-03-28T23:31:56.571253+0000","last_became_active":"2024-03-28T23:31:40.384412+0000","last_became_peered":"2024-03-28T23:31:40.384412+0000","last_unstale":"2024-03-28T23:31:56.571253+0000","last_undegraded":"2024-03-28T23:31:56.571253+0000","last_fullsized":"2024-03-28T23:31:56.571253+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:25:39.118434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569288+0000","last_change":"2024-03-28T23:31:39.385937+0000","last_active":"2024-03-28T23:31:56.569288+0000","last_peered":"2024-03-28T23:31:56.569288+0000","last_clean":"2024-03-28T23:31:56.569288+0000","last_became_active":"2024-03-28T23:31:39.385675+0000","last_became_peered":"2024-03-28T23:31:39.385675+0000","last_unstale":"2024-03-28T23:31:56.569288+0000","last_undegraded":"2024-03-28T23:31:56.569288+0000","last_fullsized":"2024-03-28T23:31:56.569288+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:35.204851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569813+0000","last_change":"2024-03-28T23:31:54.909027+0000","last_active":"2024-03-28T23:31:56.569813+0000","last_peered":"2024-03-28T23:31:56.569813+0000","last_clean":"2024-03-28T23:31:56.569813+0000","last_became_active":"2024-03-28T23:31:54.908745+0000","last_became_peered":"2024-03-28T23:31:54.908745+0000","last_unstale":"2024-03-28T23:31:56.569813+0000","last_undegraded":"2024-03-28T23:31:56.569813+0000","last_fullsized":"2024-03-28T23:31:56.569813+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:33.284318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908161+0000","last_change":"2024-03-28T23:31:53.884950+0000","last_active":"2024-03-28T23:31:55.908161+0000","last_peered":"2024-03-28T23:31:55.908161+0000","last_clean":"2024-03-28T23:31:55.908161+0000","last_became_active":"2024-03-28T23:31:53.884686+0000","last_became_peered":"2024-03-28T23:31:53.884686+0000","last_unstale":"2024-03-28T23:31:55.908161+0000","last_undegraded":"2024-03-28T23:31:55.908161+0000","last_fullsized":"2024-03-28T23:31:55.908161+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:17:10.580507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.906746+0000","last_change":"2024-03-28T23:31:40.387783+0000","last_active":"2024-03-28T23:31:55.906746+0000","last_peered":"2024-03-28T23:31:55.906746+0000","last_clean":"2024-03-28T23:31:55.906746+0000","last_became_active":"2024-03-28T23:31:40.387487+0000","last_became_peered":"2024-03-28T23:31:40.387487+0000","last_unstale":"2024-03-28T23:31:55.906746+0000","last_undegraded":"2024-03-28T23:31:55.906746+0000","last_fullsized":"2024-03-28T23:31:55.906746+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:22.230986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570453+0000","last_change":"2024-03-28T23:31:39.391741+0000","last_active":"2024-03-28T23:31:56.570453+0000","last_peered":"2024-03-28T23:31:56.570453+0000","last_clean":"2024-03-28T23:31:56.570453+0000","last_became_active":"2024-03-28T23:31:39.391520+0000","last_became_peered":"2024-03-28T23:31:39.391520+0000","last_unstale":"2024-03-28T23:31:56.570453+0000","last_undegraded":"2024-03-28T23:31:56.570453+0000","last_fullsized":"2024-03-28T23:31:56.570453+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:16:42.869555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908205+0000","last_change":"2024-03-28T23:31:54.944528+0000","last_active":"2024-03-28T23:31:55.908205+0000","last_peered":"2024-03-28T23:31:55.908205+0000","last_clean":"2024-03-28T23:31:55.908205+0000","last_became_active":"2024-03-28T23:31:54.944245+0000","last_became_peered":"2024-03-28T23:31:54.944245+0000","last_unstale":"2024-03-28T23:31:55.908205+0000","last_undegraded":"2024-03-28T23:31:55.908205+0000","last_fullsized":"2024-03-28T23:31:55.908205+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:53:35.634796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908708+0000","last_change":"2024-03-28T23:31:53.887757+0000","last_active":"2024-03-28T23:31:55.908708+0000","last_peered":"2024-03-28T23:31:55.908708+0000","last_clean":"2024-03-28T23:31:55.908708+0000","last_became_active":"2024-03-28T23:31:53.887470+0000","last_became_peered":"2024-03-28T23:31:53.887470+0000","last_unstale":"2024-03-28T23:31:55.908708+0000","last_undegraded":"2024-03-28T23:31:55.908708+0000","last_fullsized":"2024-03-28T23:31:55.908708+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:17:43.674083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570488+0000","last_change":"2024-03-28T23:31:40.387330+0000","last_active":"2024-03-28T23:31:56.570488+0000","last_peered":"2024-03-28T23:31:56.570488+0000","last_clean":"2024-03-28T23:31:56.570488+0000","last_became_active":"2024-03-28T23:31:40.387076+0000","last_became_peered":"2024-03-28T23:31:40.387076+0000","last_unstale":"2024-03-28T23:31:56.570488+0000","last_undegraded":"2024-03-28T23:31:56.570488+0000","last_fullsized":"2024-03-28T23:31:56.570488+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:25.525091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.908430+0000","last_change":"2024-03-28T23:31:39.386686+0000","last_active":"2024-03-28T23:31:55.908430+0000","last_peered":"2024-03-28T23:31:55.908430+0000","last_clean":"2024-03-28T23:31:55.908430+0000","last_became_active":"2024-03-28T23:31:39.386208+0000","last_became_peered":"2024-03-28T23:31:39.386208+0000","last_unstale":"2024-03-28T23:31:55.908430+0000","last_undegraded":"2024-03-28T23:31:55.908430+0000","last_fullsized":"2024-03-28T23:31:55.908430+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:22.210570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.907803+0000","last_change":"2024-03-28T23:31:54.942698+0000","last_active":"2024-03-28T23:31:55.907803+0000","last_peered":"2024-03-28T23:31:55.907803+0000","last_clean":"2024-03-28T23:31:55.907803+0000","last_became_active":"2024-03-28T23:31:54.942400+0000","last_became_peered":"2024-03-28T23:31:54.942400+0000","last_unstale":"2024-03-28T23:31:55.907803+0000","last_undegraded":"2024-03-28T23:31:55.907803+0000","last_fullsized":"2024-03-28T23:31:55.907803+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:31:26.648983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.906981+0000","last_change":"2024-03-28T23:31:53.887766+0000","last_active":"2024-03-28T23:31:55.906981+0000","last_peered":"2024-03-28T23:31:55.906981+0000","last_clean":"2024-03-28T23:31:55.906981+0000","last_became_active":"2024-03-28T23:31:53.887471+0000","last_became_peered":"2024-03-28T23:31:53.887471+0000","last_unstale":"2024-03-28T23:31:55.906981+0000","last_undegraded":"2024-03-28T23:31:55.906981+0000","last_fullsized":"2024-03-28T23:31:55.906981+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:40:23.367240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560655+0000","last_change":"2024-03-28T23:31:40.386143+0000","last_active":"2024-03-28T23:31:56.560655+0000","last_peered":"2024-03-28T23:31:56.560655+0000","last_clean":"2024-03-28T23:31:56.560655+0000","last_became_active":"2024-03-28T23:31:40.385606+0000","last_became_peered":"2024-03-28T23:31:40.385606+0000","last_unstale":"2024-03-28T23:31:56.560655+0000","last_undegraded":"2024-03-28T23:31:56.560655+0000","last_fullsized":"2024-03-28T23:31:56.560655+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:40:09.867914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.909045+0000","last_change":"2024-03-28T23:31:39.380624+0000","last_active":"2024-03-28T23:31:55.909045+0000","last_peered":"2024-03-28T23:31:55.909045+0000","last_clean":"2024-03-28T23:31:55.909045+0000","last_became_active":"2024-03-28T23:31:39.380340+0000","last_became_peered":"2024-03-28T23:31:39.380340+0000","last_unstale":"2024-03-28T23:31:55.909045+0000","last_undegraded":"2024-03-28T23:31:55.909045+0000","last_fullsized":"2024-03-28T23:31:55.909045+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:22.210570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.469117+0000","last_change":"2024-03-28T23:31:54.918094+0000","last_active":"2024-03-28T23:31:56.469117+0000","last_peered":"2024-03-28T23:31:56.469117+0000","last_clean":"2024-03-28T23:31:56.469117+0000","last_became_active":"2024-03-28T23:31:54.917855+0000","last_became_peered":"2024-03-28T23:31:54.917855+0000","last_unstale":"2024-03-28T23:31:56.469117+0000","last_undegraded":"2024-03-28T23:31:56.469117+0000","last_fullsized":"2024-03-28T23:31:56.469117+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:42:43.853139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.907890+0000","last_change":"2024-03-28T23:31:53.886660+0000","last_active":"2024-03-28T23:31:55.907890+0000","last_peered":"2024-03-28T23:31:55.907890+0000","last_clean":"2024-03-28T23:31:55.907890+0000","last_became_active":"2024-03-28T23:31:53.886355+0000","last_became_peered":"2024-03-28T23:31:53.886355+0000","last_unstale":"2024-03-28T23:31:55.907890+0000","last_undegraded":"2024-03-28T23:31:55.907890+0000","last_fullsized":"2024-03-28T23:31:55.907890+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:12:51.741434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.909049+0000","last_change":"2024-03-28T23:31:40.404359+0000","last_active":"2024-03-28T23:31:55.909049+0000","last_peered":"2024-03-28T23:31:55.909049+0000","last_clean":"2024-03-28T23:31:55.909049+0000","last_became_active":"2024-03-28T23:31:40.404078+0000","last_became_peered":"2024-03-28T23:31:40.404078+0000","last_unstale":"2024-03-28T23:31:55.909049+0000","last_undegraded":"2024-03-28T23:31:55.909049+0000","last_fullsized":"2024-03-28T23:31:55.909049+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:18.654331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560755+0000","last_change":"2024-03-28T23:31:39.393283+0000","last_active":"2024-03-28T23:31:56.560755+0000","last_peered":"2024-03-28T23:31:56.560755+0000","last_clean":"2024-03-28T23:31:56.560755+0000","last_became_active":"2024-03-28T23:31:39.393035+0000","last_became_peered":"2024-03-28T23:31:39.393035+0000","last_unstale":"2024-03-28T23:31:56.560755+0000","last_undegraded":"2024-03-28T23:31:56.560755+0000","last_fullsized":"2024-03-28T23:31:56.560755+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:13.139644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570288+0000","last_change":"2024-03-28T23:31:54.910082+0000","last_active":"2024-03-28T23:31:56.570288+0000","last_peered":"2024-03-28T23:31:56.570288+0000","last_clean":"2024-03-28T23:31:56.570288+0000","last_became_active":"2024-03-28T23:31:54.909694+0000","last_became_peered":"2024-03-28T23:31:54.909694+0000","last_unstale":"2024-03-28T23:31:56.570288+0000","last_undegraded":"2024-03-28T23:31:56.570288+0000","last_fullsized":"2024-03-28T23:31:56.570288+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:42:43.853139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560450+0000","last_change":"2024-03-28T23:31:53.888742+0000","last_active":"2024-03-28T23:31:56.560450+0000","last_peered":"2024-03-28T23:31:56.560450+0000","last_clean":"2024-03-28T23:31:56.560450+0000","last_became_active":"2024-03-28T23:31:53.888515+0000","last_became_peered":"2024-03-28T23:31:53.888515+0000","last_unstale":"2024-03-28T23:31:56.560450+0000","last_undegraded":"2024-03-28T23:31:56.560450+0000","last_fullsized":"2024-03-28T23:31:56.560450+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:53:34.632924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.907333+0000","last_change":"2024-03-28T23:31:40.392819+0000","last_active":"2024-03-28T23:31:55.907333+0000","last_peered":"2024-03-28T23:31:55.907333+0000","last_clean":"2024-03-28T23:31:55.907333+0000","last_became_active":"2024-03-28T23:31:40.392501+0000","last_became_peered":"2024-03-28T23:31:40.392501+0000","last_unstale":"2024-03-28T23:31:55.907333+0000","last_undegraded":"2024-03-28T23:31:55.907333+0000","last_fullsized":"2024-03-28T23:31:55.907333+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:14.144440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560803+0000","last_change":"2024-03-28T23:31:39.387993+0000","last_active":"2024-03-28T23:31:56.560803+0000","last_peered":"2024-03-28T23:31:56.560803+0000","last_clean":"2024-03-28T23:31:56.560803+0000","last_became_active":"2024-03-28T23:31:39.387393+0000","last_became_peered":"2024-03-28T23:31:39.387393+0000","last_unstale":"2024-03-28T23:31:56.560803+0000","last_undegraded":"2024-03-28T23:31:56.560803+0000","last_fullsized":"2024-03-28T23:31:56.560803+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:27:21.226190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.906935+0000","last_change":"2024-03-28T23:31:54.920674+0000","last_active":"2024-03-28T23:31:55.906935+0000","last_peered":"2024-03-28T23:31:55.906935+0000","last_clean":"2024-03-28T23:31:55.906935+0000","last_became_active":"2024-03-28T23:31:54.920400+0000","last_became_peered":"2024-03-28T23:31:54.920400+0000","last_unstale":"2024-03-28T23:31:55.906935+0000","last_undegraded":"2024-03-28T23:31:55.906935+0000","last_fullsized":"2024-03-28T23:31:55.906935+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:31:26.648983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.907964+0000","last_change":"2024-03-28T23:31:53.890146+0000","last_active":"2024-03-28T23:31:55.907964+0000","last_peered":"2024-03-28T23:31:55.907964+0000","last_clean":"2024-03-28T23:31:55.907964+0000","last_became_active":"2024-03-28T23:31:53.888840+0000","last_became_peered":"2024-03-28T23:31:53.888840+0000","last_unstale":"2024-03-28T23:31:55.907964+0000","last_undegraded":"2024-03-28T23:31:55.907964+0000","last_fullsized":"2024-03-28T23:31:55.907964+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:42:42.851267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.909635+0000","last_change":"2024-03-28T23:31:40.405766+0000","last_active":"2024-03-28T23:31:55.909635+0000","last_peered":"2024-03-28T23:31:55.909635+0000","last_clean":"2024-03-28T23:31:55.909635+0000","last_became_active":"2024-03-28T23:31:40.405517+0000","last_became_peered":"2024-03-28T23:31:40.405517+0000","last_unstale":"2024-03-28T23:31:55.909635+0000","last_undegraded":"2024-03-28T23:31:55.909635+0000","last_fullsized":"2024-03-28T23:31:55.909635+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:18:16.851435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.468404+0000","last_change":"2024-03-28T23:31:39.383748+0000","last_active":"2024-03-28T23:31:56.468404+0000","last_peered":"2024-03-28T23:31:56.468404+0000","last_clean":"2024-03-28T23:31:56.468404+0000","last_became_active":"2024-03-28T23:31:39.383070+0000","last_became_peered":"2024-03-28T23:31:39.383070+0000","last_unstale":"2024-03-28T23:31:56.468404+0000","last_undegraded":"2024-03-28T23:31:56.468404+0000","last_fullsized":"2024-03-28T23:31:56.468404+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:41.430571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.570021+0000","last_change":"2024-03-28T23:31:54.916446+0000","last_active":"2024-03-28T23:31:56.570021+0000","last_peered":"2024-03-28T23:31:56.570021+0000","last_clean":"2024-03-28T23:31:56.570021+0000","last_became_active":"2024-03-28T23:31:54.916190+0000","last_became_peered":"2024-03-28T23:31:54.916190+0000","last_unstale":"2024-03-28T23:31:56.570021+0000","last_undegraded":"2024-03-28T23:31:56.570021+0000","last_fullsized":"2024-03-28T23:31:56.570021+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:07:28.191989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.907242+0000","last_change":"2024-03-28T23:31:53.887894+0000","last_active":"2024-03-28T23:31:55.907242+0000","last_peered":"2024-03-28T23:31:55.907242+0000","last_clean":"2024-03-28T23:31:55.907242+0000","last_became_active":"2024-03-28T23:31:53.887619+0000","last_became_peered":"2024-03-28T23:31:53.887619+0000","last_unstale":"2024-03-28T23:31:55.907242+0000","last_undegraded":"2024-03-28T23:31:55.907242+0000","last_fullsized":"2024-03-28T23:31:55.907242+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:34:51.737054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.561004+0000","last_change":"2024-03-28T23:31:40.392625+0000","last_active":"2024-03-28T23:31:56.561004+0000","last_peered":"2024-03-28T23:31:56.561004+0000","last_clean":"2024-03-28T23:31:56.561004+0000","last_became_active":"2024-03-28T23:31:40.392359+0000","last_became_peered":"2024-03-28T23:31:40.392359+0000","last_unstale":"2024-03-28T23:31:56.561004+0000","last_undegraded":"2024-03-28T23:31:56.561004+0000","last_fullsized":"2024-03-28T23:31:56.561004+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:02.676183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.906841+0000","last_change":"2024-03-28T23:31:39.389478+0000","last_active":"2024-03-28T23:31:55.906841+0000","last_peered":"2024-03-28T23:31:55.906841+0000","last_clean":"2024-03-28T23:31:55.906841+0000","last_became_active":"2024-03-28T23:31:39.389154+0000","last_became_peered":"2024-03-28T23:31:39.389154+0000","last_unstale":"2024-03-28T23:31:55.906841+0000","last_undegraded":"2024-03-28T23:31:55.906841+0000","last_fullsized":"2024-03-28T23:31:55.906841+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:25:19.055737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569876+0000","last_change":"2024-03-28T23:31:54.906453+0000","last_active":"2024-03-28T23:31:56.569876+0000","last_peered":"2024-03-28T23:31:56.569876+0000","last_clean":"2024-03-28T23:31:56.569876+0000","last_became_active":"2024-03-28T23:31:54.906160+0000","last_became_peered":"2024-03-28T23:31:54.906160+0000","last_unstale":"2024-03-28T23:31:56.569876+0000","last_undegraded":"2024-03-28T23:31:56.569876+0000","last_fullsized":"2024-03-28T23:31:56.569876+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:53.865658+0000","last_clean_scrub_stamp":"2024-03-28T23:31:53.865658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:26:17.563744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.560328+0000","last_change":"2024-03-28T23:31:53.890575+0000","last_active":"2024-03-28T23:31:56.560328+0000","last_peered":"2024-03-28T23:31:56.560328+0000","last_clean":"2024-03-28T23:31:56.560328+0000","last_became_active":"2024-03-28T23:31:53.890305+0000","last_became_peered":"2024-03-28T23:31:53.890305+0000","last_unstale":"2024-03-28T23:31:56.560328+0000","last_undegraded":"2024-03-28T23:31:56.560328+0000","last_fullsized":"2024-03-28T23:31:56.560328+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:52.863786+0000","last_clean_scrub_stamp":"2024-03-28T23:31:52.863786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:04:52.850325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:56.569212+0000","last_change":"2024-03-28T23:31:40.389210+0000","last_active":"2024-03-28T23:31:56.569212+0000","last_peered":"2024-03-28T23:31:56.569212+0000","last_clean":"2024-03-28T23:31:56.569212+0000","last_became_active":"2024-03-28T23:31:40.388573+0000","last_became_peered":"2024-03-28T23:31:40.388573+0000","last_unstale":"2024-03-28T23:31:56.569212+0000","last_undegraded":"2024-03-28T23:31:56.569212+0000","last_fullsized":"2024-03-28T23:31:56.569212+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:39.364460+0000","last_clean_scrub_stamp":"2024-03-28T23:31:39.364460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:23:53.153369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"51'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.909729+0000","last_change":"2024-03-28T23:31:39.381202+0000","last_active":"2024-03-28T23:31:55.909729+0000","last_peered":"2024-03-28T23:31:55.909729+0000","last_clean":"2024-03-28T23:31:55.909729+0000","last_became_active":"2024-03-28T23:31:39.380913+0000","last_became_peered":"2024-03-28T23:31:39.380913+0000","last_unstale":"2024-03-28T23:31:55.909729+0000","last_undegraded":"2024-03-28T23:31:55.909729+0000","last_fullsized":"2024-03-28T23:31:55.909729+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:31:38.359664+0000","last_clean_scrub_stamp":"2024-03-28T23:31:38.359664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:38:21.050492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":237,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-28T23:31:55.906690+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:31:55.906690+0000","last_peered":"2024-03-28T23:31:55.906690+0000","last_clean":"2024-03-28T23:31:55.906690+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:31:55.906690+0000","last_undegraded":"2024-03-28T23:31:55.906690+0000","last_fullsized":"2024-03-28T23:31:55.906690+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574878,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31908,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339932,"statfs":{"total":96636764160,"available":96604090368,"internally_reserved":0,"allocated":745472,"data_stored":891819,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574879,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":667648,"data_stored":301237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574879,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":659456,"data_stored":300573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574878,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31836,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340004,"statfs":{"total":96636764160,"available":96604164096,"internally_reserved":0,"allocated":671744,"data_stored":302745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574878,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":302185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574878,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":667648,"data_stored":302163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574878,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":303087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574878,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27732,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344108,"statfs":{"total":96636764160,"available":96608366592,"internally_reserved":0,"allocated":729088,"data_stored":890975,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:32:03.020 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T23:32:03.020 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail 3:0 2024-03-28T23:32:03.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.154+0000 7fbf82f10640 1 Processor -- start 2024-03-28T23:32:03.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.154+0000 7fbf82f10640 1 -- start start 2024-03-28T23:32:03.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.155+0000 7fbf82f10640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbf7c0c5410 0x7fbf7c0c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:03.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.155+0000 7fbf82f10640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c1553a0 0x7fbf7c155850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:03.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.155+0000 7fbf82f10640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf7c158a30 con 0x7fbf7c152890 2024-03-28T23:32:03.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.155+0000 7fbf82f10640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fbf7c0c8480 con 0x7fbf7c0c5410 2024-03-28T23:32:03.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.155+0000 7fbf82f10640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbf7c0c5d90 con 0x7fbf7c1553a0 2024-03-28T23:32:03.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.155+0000 7fbf73fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbf7c0c5410 0x7fbf7c0c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:03.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.155+0000 7fbf73fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbf7c0c5410 0x7fbf7c0c5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57780/0 (socket says 172.21.15.161:57780) 2024-03-28T23:32:03.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.155+0000 7fbf73fff640 1 -- 172.21.15.161:0/2333120768 learned_addr learned my addr 172.21.15.161:0/2333120768 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:03.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.155+0000 7fbf73fff640 1 -- 172.21.15.161:0/2333120768 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c1553a0 msgr2=0x7fbf7c155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:03.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.155+0000 7fbf814ad640 1 --2- 172.21.15.161:0/2333120768 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c1553a0 0x7fbf7c155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:03.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.155+0000 7fbf73fff640 1 --2- 172.21.15.161:0/2333120768 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c1553a0 0x7fbf7c155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:03.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.155+0000 7fbf73fff640 1 -- 172.21.15.161:0/2333120768 >> v1:172.21.15.170:6789/0 conn(0x7fbf7c152890 legacy=0x7fbf7c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:03.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.155+0000 7fbf73fff640 1 -- 172.21.15.161:0/2333120768 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf7c158d10 con 0x7fbf7c0c5410 2024-03-28T23:32:03.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.156+0000 7fbf73fff640 1 --2- 172.21.15.161:0/2333120768 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c0c5410 0x7fbf7c0c5850 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c009850 tx=0x7fbf6c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48dfd16898dc530e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:03.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.156+0000 7fbf737fe640 1 -- 172.21.15.161:0/2333120768 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c037070 con 0x7fbf7c0c5410 2024-03-28T23:32:03.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.156+0000 7fbf737fe640 1 -- 172.21.15.161:0/2333120768 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf6c004030 con 0x7fbf7c0c5410 2024-03-28T23:32:03.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.156+0000 7fbf737fe640 1 -- 172.21.15.161:0/2333120768 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c004460 con 0x7fbf7c0c5410 2024-03-28T23:32:03.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.157+0000 7fbf82f10640 1 -- 172.21.15.161:0/2333120768 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c0c5410 msgr2=0x7fbf7c0c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:03.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.157+0000 7fbf82f10640 1 --2- 172.21.15.161:0/2333120768 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c0c5410 0x7fbf7c0c5850 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c009850 tx=0x7fbf6c02b840 comp rx=0 tx=0).stop 2024-03-28T23:32:03.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.157+0000 7fbf82f10640 1 -- 172.21.15.161:0/2333120768 shutdown_connections 2024-03-28T23:32:03.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.157+0000 7fbf82f10640 1 --2- 172.21.15.161:0/2333120768 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c1553a0 0x7fbf7c155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:03.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.157+0000 7fbf82f10640 1 --2- 172.21.15.161:0/2333120768 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c0c5410 0x7fbf7c0c5850 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:03.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.157+0000 7fbf82f10640 1 -- 172.21.15.161:0/2333120768 >> 172.21.15.161:0/2333120768 conn(0x7fbf7c0ac620 msgr2=0x7fbf7c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:03.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.157+0000 7fbf82f10640 1 -- 172.21.15.161:0/2333120768 shutdown_connections 2024-03-28T23:32:03.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.157+0000 7fbf82f10640 1 -- 172.21.15.161:0/2333120768 wait complete. 2024-03-28T23:32:03.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.158+0000 7fbf82f10640 1 Processor -- start 2024-03-28T23:32:03.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.158+0000 7fbf82f10640 1 -- start start 2024-03-28T23:32:03.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.159+0000 7fbf82f10640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf7c0c5410 0x7fbf7c14af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:03.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.159+0000 7fbf82f10640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c152890 0x7fbf7c1eef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:03.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.159+0000 7fbf82f10640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c1553a0 0x7fbf7c14d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:03.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.159+0000 7fbf82f10640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbf7c1592d0 con 0x7fbf7c1553a0 2024-03-28T23:32:03.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.159+0000 7fbf82f10640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbf7c159640 con 0x7fbf7c0c5410 2024-03-28T23:32:03.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.159+0000 7fbf82f10640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbf7c0c90a0 con 0x7fbf7c152890 2024-03-28T23:32:03.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.159+0000 7fbf814ad640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c1553a0 0x7fbf7c14d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:03.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.159+0000 7fbf814ad640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c1553a0 0x7fbf7c14d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57790/0 (socket says 172.21.15.161:57790) 2024-03-28T23:32:03.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.159+0000 7fbf814ad640 1 -- 172.21.15.161:0/2294608079 learned_addr learned my addr 172.21.15.161:0/2294608079 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:03.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.159+0000 7fbf80cac640 1 --2- 172.21.15.161:0/2294608079 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf7c0c5410 0x7fbf7c14af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:03.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.159+0000 7fbf73fff640 1 --2- 172.21.15.161:0/2294608079 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c152890 0x7fbf7c1eef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:03.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.159+0000 7fbf814ad640 1 -- 172.21.15.161:0/2294608079 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c152890 msgr2=0x7fbf7c1eef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:03.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.159+0000 7fbf814ad640 1 --2- 172.21.15.161:0/2294608079 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c152890 0x7fbf7c1eef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:03.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.159+0000 7fbf814ad640 1 -- 172.21.15.161:0/2294608079 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf7c0c5410 msgr2=0x7fbf7c14af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:03.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.159+0000 7fbf814ad640 1 --2- 172.21.15.161:0/2294608079 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf7c0c5410 0x7fbf7c14af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:03.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.159+0000 7fbf814ad640 1 -- 172.21.15.161:0/2294608079 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf7c14b610 con 0x7fbf7c1553a0 2024-03-28T23:32:03.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.160+0000 7fbf814ad640 1 --2- 172.21.15.161:0/2294608079 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c1553a0 0x7fbf7c14d3c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fbf7400cd70 tx=0x7fbf7400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:03.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.160+0000 7fbf717fa640 1 -- 172.21.15.161:0/2294608079 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf74003b90 con 0x7fbf7c1553a0 2024-03-28T23:32:03.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.160+0000 7fbf717fa640 1 -- 172.21.15.161:0/2294608079 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf74003d30 con 0x7fbf7c1553a0 2024-03-28T23:32:03.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.160+0000 7fbf82f10640 1 -- 172.21.15.161:0/2294608079 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf7c14b8f0 con 0x7fbf7c1553a0 2024-03-28T23:32:03.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.160+0000 7fbf717fa640 1 -- 172.21.15.161:0/2294608079 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf74004240 con 0x7fbf7c1553a0 2024-03-28T23:32:03.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.160+0000 7fbf82f10640 1 -- 172.21.15.161:0/2294608079 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf7c1518b0 con 0x7fbf7c1553a0 2024-03-28T23:32:03.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.161+0000 7fbf82f10640 1 -- 172.21.15.161:0/2294608079 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf7c04f4b0 con 0x7fbf7c1553a0 2024-03-28T23:32:03.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.162+0000 7fbf717fa640 1 -- 172.21.15.161:0/2294608079 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbf740043e0 con 0x7fbf7c1553a0 2024-03-28T23:32:03.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.162+0000 7fbf717fa640 1 --2- 172.21.15.161:0/2294608079 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf6806b4d0 0x7fbf6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:03.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.162+0000 7fbf717fa640 1 -- 172.21.15.161:0/2294608079 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6957+0+0 (secure 0 0 0) 0x7fbf7408b040 con 0x7fbf7c1553a0 2024-03-28T23:32:03.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.162+0000 7fbf80cac640 1 --2- 172.21.15.161:0/2294608079 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf6806b4d0 0x7fbf6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:03.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.163+0000 7fbf80cac640 1 --2- 172.21.15.161:0/2294608079 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf6806b4d0 0x7fbf6806d990 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fbf78004f60 tx=0x7fbf78007e30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:03.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.167+0000 7fbf717fa640 1 -- 172.21.15.161:0/2294608079 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbf74059420 con 0x7fbf7c1553a0 2024-03-28T23:32:03.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.517+0000 7fbf82f10640 1 -- 172.21.15.161:0/2294608079 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "3:0"} v 0) -- 0x7fbf7c0ac400 con 0x7fbf7c1553a0 2024-03-28T23:32:03.518 INFO:tasks.ceph.mon.a.smithi161.stderr:2024-03-28T23:32:03.517+0000 7fdc7b85b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T23:32:03.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.519+0000 7fbf717fa640 1 -- 172.21.15.161:0/2294608079 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "3:0"}]=0 failed mds gid 5820 v35) ==== 97+0+0 (secure 0 0 0) 0x7fbf7405d4c0 con 0x7fbf7c1553a0 2024-03-28T23:32:03.519 INFO:teuthology.orchestra.run.smithi161.stderr:failed mds gid 5820 2024-03-28T23:32:03.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.523+0000 7fbf82f10640 1 -- 172.21.15.161:0/2294608079 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf6806b4d0 msgr2=0x7fbf6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:03.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.523+0000 7fbf82f10640 1 --2- 172.21.15.161:0/2294608079 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf6806b4d0 0x7fbf6806d990 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fbf78004f60 tx=0x7fbf78007e30 comp rx=0 tx=0).stop 2024-03-28T23:32:03.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.523+0000 7fbf82f10640 1 -- 172.21.15.161:0/2294608079 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c1553a0 msgr2=0x7fbf7c14d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:03.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.523+0000 7fbf82f10640 1 --2- 172.21.15.161:0/2294608079 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c1553a0 0x7fbf7c14d3c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fbf7400cd70 tx=0x7fbf7400a660 comp rx=0 tx=0).stop 2024-03-28T23:32:03.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.523+0000 7fbf82f10640 1 -- 172.21.15.161:0/2294608079 shutdown_connections 2024-03-28T23:32:03.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.523+0000 7fbf82f10640 1 --2- 172.21.15.161:0/2294608079 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf6806b4d0 0x7fbf6806d990 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:03.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.523+0000 7fbf82f10640 1 --2- 172.21.15.161:0/2294608079 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c1553a0 0x7fbf7c14d3c0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:03.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.523+0000 7fbf82f10640 1 --2- 172.21.15.161:0/2294608079 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c152890 0x7fbf7c1eef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:03.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.523+0000 7fbf82f10640 1 --2- 172.21.15.161:0/2294608079 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf7c0c5410 0x7fbf7c14af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:03.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.523+0000 7fbf82f10640 1 -- 172.21.15.161:0/2294608079 >> 172.21.15.161:0/2294608079 conn(0x7fbf7c0ac620 msgr2=0x7fbf7c0ca120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:03.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.523+0000 7fbf82f10640 1 -- 172.21.15.161:0/2294608079 shutdown_connections 2024-03-28T23:32:03.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.523+0000 7fbf82f10640 1 -- 172.21.15.161:0/2294608079 wait complete. 2024-03-28T23:32:03.525 INFO:tasks.ceph.mds.a.smithi161.stderr: -631> 2024-03-28T23:31:44.636+0000 7fdac8bea640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:32:03.539 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:32:03.552 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-28T23:32:03.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.641+0000 7f1b578eb640 1 Processor -- start 2024-03-28T23:32:03.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.641+0000 7f1b578eb640 1 -- start start 2024-03-28T23:32:03.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.643+0000 7f1b578eb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b50150b10 0x7f1b500acfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:03.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.643+0000 7f1b578eb640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1b50072160 0x7f1b50156ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:03.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.643+0000 7f1b578eb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b500c2610 con 0x7f1b50157400 2024-03-28T23:32:03.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.643+0000 7f1b578eb640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f1b50156750 con 0x7f1b50072160 2024-03-28T23:32:03.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.643+0000 7f1b578eb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1b500c28f0 con 0x7f1b50150b10 2024-03-28T23:32:03.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.643+0000 7f1b55e88640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f1b50157400 0x7f1b5006c160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37760/0 (socket says 172.21.15.161:37760) 2024-03-28T23:32:03.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.643+0000 7f1b55e88640 1 -- 172.21.15.161:0/1504708327 learned_addr learned my addr 172.21.15.161:0/1504708327 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:03.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.643+0000 7f1b55687640 1 --2- 172.21.15.161:0/1504708327 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b50150b10 0x7f1b500acfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:03.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.643+0000 7f1b54e86640 1 --2- 172.21.15.161:0/1504708327 >> v2:172.21.15.170:3300/0 conn(0x7f1b50072160 0x7f1b50156ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:03.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.643+0000 7f1b47fff640 1 -- 172.21.15.161:0/1504708327 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3002052654 0 0) 0x7f1b500c2610 con 0x7f1b50157400 2024-03-28T23:32:03.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.643+0000 7f1b55687640 1 -- 172.21.15.161:0/1504708327 >> v2:172.21.15.170:3300/0 conn(0x7f1b50072160 msgr2=0x7f1b50156ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:03.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.643+0000 7f1b55687640 1 --2- 172.21.15.161:0/1504708327 >> v2:172.21.15.170:3300/0 conn(0x7f1b50072160 0x7f1b50156ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:03.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.643+0000 7f1b55687640 1 -- 172.21.15.161:0/1504708327 >> v1:172.21.15.161:6789/0 conn(0x7f1b50157400 legacy=0x7f1b5006c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:03.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.643+0000 7f1b55687640 1 -- 172.21.15.161:0/1504708327 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b500c2bd0 con 0x7f1b50150b10 2024-03-28T23:32:03.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.644+0000 7f1b55687640 1 --2- 172.21.15.161:0/1504708327 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b50150b10 0x7f1b500acfc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c0097a0 tx=0x7f1b4c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dd5fc73600bdd35 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:03.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.644+0000 7f1b47fff640 1 -- 172.21.15.161:0/1504708327 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c0315a0 con 0x7f1b50150b10 2024-03-28T23:32:03.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.644+0000 7f1b47fff640 1 -- 172.21.15.161:0/1504708327 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b4c037070 con 0x7f1b50150b10 2024-03-28T23:32:03.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.644+0000 7f1b47fff640 1 -- 172.21.15.161:0/1504708327 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c031960 con 0x7f1b50150b10 2024-03-28T23:32:03.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.645+0000 7f1b578eb640 1 -- 172.21.15.161:0/1504708327 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b50150b10 msgr2=0x7f1b500acfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:03.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.645+0000 7f1b578eb640 1 --2- 172.21.15.161:0/1504708327 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b50150b10 0x7f1b500acfc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c0097a0 tx=0x7f1b4c02b840 comp rx=0 tx=0).stop 2024-03-28T23:32:03.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.645+0000 7f1b578eb640 1 -- 172.21.15.161:0/1504708327 shutdown_connections 2024-03-28T23:32:03.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.645+0000 7f1b578eb640 1 --2- 172.21.15.161:0/1504708327 >> v2:172.21.15.170:3300/0 conn(0x7f1b50072160 0x7f1b50156ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:03.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.645+0000 7f1b578eb640 1 --2- 172.21.15.161:0/1504708327 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b50150b10 0x7f1b500acfc0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:03.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.645+0000 7f1b578eb640 1 -- 172.21.15.161:0/1504708327 >> 172.21.15.161:0/1504708327 conn(0x7f1b500c3cf0 msgr2=0x7f1b500c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:03.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.645+0000 7f1b578eb640 1 -- 172.21.15.161:0/1504708327 shutdown_connections 2024-03-28T23:32:03.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.645+0000 7f1b578eb640 1 -- 172.21.15.161:0/1504708327 wait complete. 2024-03-28T23:32:03.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.646+0000 7f1b578eb640 1 Processor -- start 2024-03-28T23:32:03.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.646+0000 7f1b578eb640 1 -- start start 2024-03-28T23:32:03.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.646+0000 7f1b578eb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b50072160 0x7f1b500c08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:03.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.646+0000 7f1b578eb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b50150b10 0x7f1b5013f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:03.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.646+0000 7f1b578eb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b500abf20 0x7f1b5013fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:03.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.646+0000 7f1b578eb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5006c9f0 con 0x7f1b50072160 2024-03-28T23:32:03.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.647+0000 7f1b578eb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5006c870 con 0x7f1b500abf20 2024-03-28T23:32:03.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.647+0000 7f1b578eb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1b500c2230 con 0x7f1b50150b10 2024-03-28T23:32:03.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.647+0000 7f1b55687640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b50072160 0x7f1b500c08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:03.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.647+0000 7f1b55687640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b50072160 0x7f1b500c08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57814/0 (socket says 172.21.15.161:57814) 2024-03-28T23:32:03.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.647+0000 7f1b55687640 1 -- 172.21.15.161:0/341889245 learned_addr learned my addr 172.21.15.161:0/341889245 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:03.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.647+0000 7f1b54e86640 1 --2- 172.21.15.161:0/341889245 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b50150b10 0x7f1b5013f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:03.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.647+0000 7f1b55e88640 1 --2- 172.21.15.161:0/341889245 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b500abf20 0x7f1b5013fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:03.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.647+0000 7f1b55687640 1 -- 172.21.15.161:0/341889245 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b50150b10 msgr2=0x7f1b5013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:03.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.647+0000 7f1b55687640 1 --2- 172.21.15.161:0/341889245 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b50150b10 0x7f1b5013f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:03.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.647+0000 7f1b55687640 1 -- 172.21.15.161:0/341889245 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b500abf20 msgr2=0x7f1b5013fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:03.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.647+0000 7f1b55687640 1 --2- 172.21.15.161:0/341889245 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b500abf20 0x7f1b5013fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:03.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.647+0000 7f1b55687640 1 -- 172.21.15.161:0/341889245 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b50151de0 con 0x7f1b50072160 2024-03-28T23:32:03.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.647+0000 7f1b55e88640 1 --2- 172.21.15.161:0/341889245 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b500abf20 0x7f1b5013fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:03.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.647+0000 7f1b55687640 1 --2- 172.21.15.161:0/341889245 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b50072160 0x7f1b500c08c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c0097a0 tx=0x7f1b4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:03.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.647+0000 7f1b45ffb640 1 -- 172.21.15.161:0/341889245 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c004230 con 0x7f1b50072160 2024-03-28T23:32:03.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.647+0000 7f1b45ffb640 1 -- 172.21.15.161:0/341889245 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b4c0043f0 con 0x7f1b50072160 2024-03-28T23:32:03.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.647+0000 7f1b578eb640 1 -- 172.21.15.161:0/341889245 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b501eef70 con 0x7f1b50072160 2024-03-28T23:32:03.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.647+0000 7f1b45ffb640 1 -- 172.21.15.161:0/341889245 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c003850 con 0x7f1b50072160 2024-03-28T23:32:03.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.648+0000 7f1b578eb640 1 -- 172.21.15.161:0/341889245 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b50152360 con 0x7f1b50072160 2024-03-28T23:32:03.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.649+0000 7f1b578eb640 1 -- 172.21.15.161:0/341889245 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b50068f90 con 0x7f1b50072160 2024-03-28T23:32:03.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.649+0000 7f1b45ffb640 1 -- 172.21.15.161:0/341889245 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1b4c0039f0 con 0x7f1b50072160 2024-03-28T23:32:03.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.649+0000 7f1b45ffb640 1 --2- 172.21.15.161:0/341889245 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1b3406b4d0 0x7f1b3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:03.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.650+0000 7f1b54e86640 1 --2- 172.21.15.161:0/341889245 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1b3406b4d0 0x7f1b3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:03.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.650+0000 7f1b45ffb640 1 -- 172.21.15.161:0/341889245 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7043+0+0 (secure 0 0 0) 0x7f1b4c02f030 con 0x7f1b50072160 2024-03-28T23:32:03.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.650+0000 7f1b54e86640 1 --2- 172.21.15.161:0/341889245 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1b3406b4d0 0x7f1b3406d990 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1b500c11e0 tx=0x7f1b40009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:03.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.654+0000 7f1b45ffb640 1 -- 172.21.15.161:0/341889245 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1b50068f90 con 0x7f1b50072160 2024-03-28T23:32:03.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.862+0000 7f1b578eb640 1 -- 172.21.15.161:0/341889245 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b5004f4b0 con 0x7f1b50072160 2024-03-28T23:32:03.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.863+0000 7f1b45ffb640 1 -- 172.21.15.161:0/341889245 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5672 (secure 0 0 0) 0x7f1b4c041070 con 0x7f1b50072160 2024-03-28T23:32:03.864 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:03.864 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":36,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5542,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.170:6833/3666899516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3666899516},{"type":"v1","addr":"172.21.15.170:6833","nonce":3666899516}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:31:39.725625+0000","modified":"2024-03-28T23:32:03.523636+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5186":{"gid":5186,"name":"b","rank":0,"incarnation":36,"state":"up:replay","state_seq":6,"addr":"172.21.15.170:6835/1945122204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1945122204},{"type":"v1","addr":"172.21.15.170:6835","nonce":1945122204}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3},{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:31:54.226424+0000","modified":"2024-03-28T23:31:56.347601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5817},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5817":{"gid":5817,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":8,"addr":"172.21.15.161:6836/1300583128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1300583128},{"type":"v1","addr":"172.21.15.161:6836","nonce":1300583128}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5817,"qdb_cluster":[5817]},"id":4}]} 2024-03-28T23:32:03.865 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 36 2024-03-28T23:32:03.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.864+0000 7f1b578eb640 1 -- 172.21.15.161:0/341889245 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1b3406b4d0 msgr2=0x7f1b3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:03.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.864+0000 7f1b578eb640 1 --2- 172.21.15.161:0/341889245 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1b3406b4d0 0x7f1b3406d990 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1b500c11e0 tx=0x7f1b40009190 comp rx=0 tx=0).stop 2024-03-28T23:32:03.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.864+0000 7f1b578eb640 1 -- 172.21.15.161:0/341889245 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b50072160 msgr2=0x7f1b500c08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:03.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.864+0000 7f1b578eb640 1 --2- 172.21.15.161:0/341889245 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b50072160 0x7f1b500c08c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c0097a0 tx=0x7f1b4c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:32:03.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.864+0000 7f1b578eb640 1 -- 172.21.15.161:0/341889245 shutdown_connections 2024-03-28T23:32:03.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.864+0000 7f1b578eb640 1 --2- 172.21.15.161:0/341889245 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1b3406b4d0 0x7f1b3406d990 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:03.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.864+0000 7f1b578eb640 1 --2- 172.21.15.161:0/341889245 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b500abf20 0x7f1b5013fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:03.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.864+0000 7f1b578eb640 1 --2- 172.21.15.161:0/341889245 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b50150b10 0x7f1b5013f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:03.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.864+0000 7f1b578eb640 1 --2- 172.21.15.161:0/341889245 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b50072160 0x7f1b500c08c0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:03.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.864+0000 7f1b578eb640 1 -- 172.21.15.161:0/341889245 >> 172.21.15.161:0/341889245 conn(0x7f1b500c3cf0 msgr2=0x7f1b5006e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:03.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.865+0000 7f1b578eb640 1 -- 172.21.15.161:0/341889245 shutdown_connections 2024-03-28T23:32:03.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:03.865+0000 7f1b578eb640 1 -- 172.21.15.161:0/341889245 wait complete. 2024-03-28T23:32:03.876 INFO:tasks.cephfs.test_failover:Waiting for former active to reclaim its spot 2024-03-28T23:32:03.877 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_beacon_grace 2024-03-28T23:32:04.025 INFO:teuthology.orchestra.run.smithi161.stdout:{"mds_beacon_grace":"15.000000"} 2024-03-28T23:32:04.026 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_beacon_grace": "15.000000" } 2024-03-28T23:32:04.026 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:32:04.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.126+0000 7f6a4080e640 1 Processor -- start 2024-03-28T23:32:04.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.126+0000 7f6a4080e640 1 -- start start 2024-03-28T23:32:04.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.127+0000 7f6a4080e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6a3c152890 0x7f6a3c0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:04.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.127+0000 7f6a4080e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6a3c0c5b20 0x7f6a3c1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:04.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.127+0000 7f6a4080e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a3c158a30 con 0x7f6a3c0c8ed0 2024-03-28T23:32:04.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.127+0000 7f6a4080e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6a3c0c8480 con 0x7f6a3c0c5b20 2024-03-28T23:32:04.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.127+0000 7f6a4080e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f6a3c159640 con 0x7f6a3c152890 2024-03-28T23:32:04.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.127+0000 7f6a3ad9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6a3c0c5b20 0x7f6a3c1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:04.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.127+0000 7f6a3ad9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6a3c0c5b20 0x7f6a3c1553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57830/0 (socket says 172.21.15.161:57830) 2024-03-28T23:32:04.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.127+0000 7f6a3ad9d640 1 -- 172.21.15.161:0/3347815956 learned_addr learned my addr 172.21.15.161:0/3347815956 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:04.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.127+0000 7f6a3a59c640 1 --2- 172.21.15.161:0/3347815956 >> v2:172.21.15.170:3300/0 conn(0x7f6a3c152890 0x7f6a3c0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:04.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.127+0000 7f6a3ad9d640 1 -- 172.21.15.161:0/3347815956 >> v2:172.21.15.170:3300/0 conn(0x7f6a3c152890 msgr2=0x7f6a3c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:04.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.127+0000 7f6a3ad9d640 1 --2- 172.21.15.161:0/3347815956 >> v2:172.21.15.170:3300/0 conn(0x7f6a3c152890 0x7f6a3c0c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:04.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.127+0000 7f6a3ad9d640 1 -- 172.21.15.161:0/3347815956 >> v1:172.21.15.170:6789/0 conn(0x7f6a3c0c8ed0 legacy=0x7f6a3c0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:04.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.127+0000 7f6a3ad9d640 1 -- 172.21.15.161:0/3347815956 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a3c158d10 con 0x7f6a3c0c5b20 2024-03-28T23:32:04.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.128+0000 7f6a3ad9d640 1 --2- 172.21.15.161:0/3347815956 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a3c0c5b20 0x7f6a3c1553a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c08cc80 tx=0x7f6a3402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a060bbe45c1ead9a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:04.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.128+0000 7f6a3959a640 1 -- 172.21.15.161:0/3347815956 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3402c9f0 con 0x7f6a3c0c5b20 2024-03-28T23:32:04.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.128+0000 7f6a3959a640 1 -- 172.21.15.161:0/3347815956 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a3402cb90 con 0x7f6a3c0c5b20 2024-03-28T23:32:04.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.128+0000 7f6a3959a640 1 -- 172.21.15.161:0/3347815956 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a340343f0 con 0x7f6a3c0c5b20 2024-03-28T23:32:04.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.129+0000 7f6a4080e640 1 -- 172.21.15.161:0/3347815956 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a3c0c5b20 msgr2=0x7f6a3c1553a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:04.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.129+0000 7f6a4080e640 1 --2- 172.21.15.161:0/3347815956 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a3c0c5b20 0x7f6a3c1553a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c08cc80 tx=0x7f6a3402be50 comp rx=0 tx=0).stop 2024-03-28T23:32:04.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.129+0000 7f6a4080e640 1 -- 172.21.15.161:0/3347815956 shutdown_connections 2024-03-28T23:32:04.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.129+0000 7f6a4080e640 1 --2- 172.21.15.161:0/3347815956 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a3c0c5b20 0x7f6a3c1553a0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:04.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.129+0000 7f6a4080e640 1 --2- 172.21.15.161:0/3347815956 >> v2:172.21.15.170:3300/0 conn(0x7f6a3c152890 0x7f6a3c0c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:04.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.129+0000 7f6a4080e640 1 -- 172.21.15.161:0/3347815956 >> 172.21.15.161:0/3347815956 conn(0x7f6a3c0ac620 msgr2=0x7f6a3c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:04.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.129+0000 7f6a4080e640 1 -- 172.21.15.161:0/3347815956 shutdown_connections 2024-03-28T23:32:04.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.129+0000 7f6a4080e640 1 -- 172.21.15.161:0/3347815956 wait complete. 2024-03-28T23:32:04.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.130+0000 7f6a4080e640 1 Processor -- start 2024-03-28T23:32:04.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.130+0000 7f6a4080e640 1 -- start start 2024-03-28T23:32:04.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.131+0000 7f6a4080e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a3c0c5b20 0x7f6a3c0c3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:04.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.131+0000 7f6a4080e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a3c0c8ed0 0x7f6a3c1ee220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:04.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.131+0000 7f6a4080e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a3c152890 0x7f6a3c1ebeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:04.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.131+0000 7f6a4080e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3c152c70 con 0x7f6a3c152890 2024-03-28T23:32:04.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.131+0000 7f6a4080e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3c1592d0 con 0x7f6a3c0c8ed0 2024-03-28T23:32:04.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.131+0000 7f6a4080e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6a3c152df0 con 0x7f6a3c0c5b20 2024-03-28T23:32:04.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.131+0000 7f6a3ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a3c152890 0x7f6a3c1ebeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:04.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.131+0000 7f6a3ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a3c152890 0x7f6a3c1ebeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57846/0 (socket says 172.21.15.161:57846) 2024-03-28T23:32:04.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.131+0000 7f6a3ad9d640 1 -- 172.21.15.161:0/3231058107 learned_addr learned my addr 172.21.15.161:0/3231058107 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:04.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.131+0000 7f6a39d9b640 1 --2- 172.21.15.161:0/3231058107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a3c0c8ed0 0x7f6a3c1ee220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:04.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.131+0000 7f6a3a59c640 1 --2- 172.21.15.161:0/3231058107 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a3c0c5b20 0x7f6a3c0c3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:04.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.131+0000 7f6a3ad9d640 1 -- 172.21.15.161:0/3231058107 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a3c0c5b20 msgr2=0x7f6a3c0c3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:04.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.131+0000 7f6a3ad9d640 1 --2- 172.21.15.161:0/3231058107 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a3c0c5b20 0x7f6a3c0c3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:04.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.131+0000 7f6a3ad9d640 1 -- 172.21.15.161:0/3231058107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a3c0c8ed0 msgr2=0x7f6a3c1ee220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:04.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.131+0000 7f6a3ad9d640 1 --2- 172.21.15.161:0/3231058107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a3c0c8ed0 0x7f6a3c1ee220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:04.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.131+0000 7f6a3ad9d640 1 -- 172.21.15.161:0/3231058107 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a3c0c3540 con 0x7f6a3c152890 2024-03-28T23:32:04.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.131+0000 7f6a3ad9d640 1 --2- 172.21.15.161:0/3231058107 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a3c152890 0x7f6a3c1ebeb0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f6a34009f40 tx=0x7f6a34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:04.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.131+0000 7f6a2affd640 1 -- 172.21.15.161:0/3231058107 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a34044020 con 0x7f6a3c152890 2024-03-28T23:32:04.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.131+0000 7f6a2affd640 1 -- 172.21.15.161:0/3231058107 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a34040470 con 0x7f6a3c152890 2024-03-28T23:32:04.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.132+0000 7f6a4080e640 1 -- 172.21.15.161:0/3231058107 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a3c1eea60 con 0x7f6a3c152890 2024-03-28T23:32:04.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.132+0000 7f6a2affd640 1 -- 172.21.15.161:0/3231058107 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3402c9f0 con 0x7f6a3c152890 2024-03-28T23:32:04.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.132+0000 7f6a4080e640 1 -- 172.21.15.161:0/3231058107 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a3c0c3810 con 0x7f6a3c152890 2024-03-28T23:32:04.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.132+0000 7f6a4080e640 1 -- 172.21.15.161:0/3231058107 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a3c04f4b0 con 0x7f6a3c152890 2024-03-28T23:32:04.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.133+0000 7f6a2affd640 1 -- 172.21.15.161:0/3231058107 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6a34047020 con 0x7f6a3c152890 2024-03-28T23:32:04.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.133+0000 7f6a2affd640 1 --2- 172.21.15.161:0/3231058107 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6a1c06b500 0x7f6a1c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:04.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.133+0000 7f6a2affd640 1 -- 172.21.15.161:0/3231058107 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7043+0+0 (secure 0 0 0) 0x7f6a340ad230 con 0x7f6a3c152890 2024-03-28T23:32:04.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.133+0000 7f6a3a59c640 1 --2- 172.21.15.161:0/3231058107 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6a1c06b500 0x7f6a1c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:04.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.134+0000 7f6a3a59c640 1 --2- 172.21.15.161:0/3231058107 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6a1c06b500 0x7f6a1c06d9c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c0079a0 tx=0x7f6a2c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:04.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.138+0000 7f6a2affd640 1 -- 172.21.15.161:0/3231058107 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6a3404e020 con 0x7f6a3c152890 2024-03-28T23:32:04.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.349+0000 7f6a4080e640 1 -- 172.21.15.161:0/3231058107 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a3c06d2e0 con 0x7f6a3c152890 2024-03-28T23:32:04.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.349+0000 7f6a2affd640 1 -- 172.21.15.161:0/3231058107 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5672 (secure 0 0 0) 0x7f6a3407b530 con 0x7f6a3c152890 2024-03-28T23:32:04.350 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:04.351 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":36,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5542,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.170:6833/3666899516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3666899516},{"type":"v1","addr":"172.21.15.170:6833","nonce":3666899516}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:31:39.725625+0000","modified":"2024-03-28T23:32:03.523636+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5186":{"gid":5186,"name":"b","rank":0,"incarnation":36,"state":"up:replay","state_seq":6,"addr":"172.21.15.170:6835/1945122204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1945122204},{"type":"v1","addr":"172.21.15.170:6835","nonce":1945122204}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3},{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:31:54.226424+0000","modified":"2024-03-28T23:31:56.347601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5817},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5817":{"gid":5817,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":8,"addr":"172.21.15.161:6836/1300583128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1300583128},{"type":"v1","addr":"172.21.15.161:6836","nonce":1300583128}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5817,"qdb_cluster":[5817]},"id":4}]} 2024-03-28T23:32:04.351 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 36 2024-03-28T23:32:04.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.351+0000 7f6a4080e640 1 -- 172.21.15.161:0/3231058107 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6a1c06b500 msgr2=0x7f6a1c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:04.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.351+0000 7f6a4080e640 1 --2- 172.21.15.161:0/3231058107 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6a1c06b500 0x7f6a1c06d9c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c0079a0 tx=0x7f6a2c008040 comp rx=0 tx=0).stop 2024-03-28T23:32:04.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.351+0000 7f6a4080e640 1 -- 172.21.15.161:0/3231058107 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a3c152890 msgr2=0x7f6a3c1ebeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:04.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.351+0000 7f6a4080e640 1 --2- 172.21.15.161:0/3231058107 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a3c152890 0x7f6a3c1ebeb0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f6a34009f40 tx=0x7f6a34004a40 comp rx=0 tx=0).stop 2024-03-28T23:32:04.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.352+0000 7f6a4080e640 1 -- 172.21.15.161:0/3231058107 shutdown_connections 2024-03-28T23:32:04.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.352+0000 7f6a4080e640 1 --2- 172.21.15.161:0/3231058107 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6a1c06b500 0x7f6a1c06d9c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:04.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.352+0000 7f6a4080e640 1 --2- 172.21.15.161:0/3231058107 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a3c152890 0x7f6a3c1ebeb0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:04.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.352+0000 7f6a4080e640 1 --2- 172.21.15.161:0/3231058107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a3c0c8ed0 0x7f6a3c1ee220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:04.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.352+0000 7f6a4080e640 1 --2- 172.21.15.161:0/3231058107 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a3c0c5b20 0x7f6a3c0c3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:04.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.352+0000 7f6a4080e640 1 -- 172.21.15.161:0/3231058107 >> 172.21.15.161:0/3231058107 conn(0x7f6a3c0ac620 msgr2=0x7f6a3c14ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:04.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.352+0000 7f6a4080e640 1 -- 172.21.15.161:0/3231058107 shutdown_connections 2024-03-28T23:32:04.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:04.352+0000 7f6a4080e640 1 -- 172.21.15.161:0/3231058107 wait complete. 2024-03-28T23:32:04.364 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-03-28T23:32:06.566 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2024-03-28T23:32:09.365 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:32:09.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.467+0000 7f448a5e5640 1 Processor -- start 2024-03-28T23:32:09.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.467+0000 7f448a5e5640 1 -- start start 2024-03-28T23:32:09.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.468+0000 7f448a5e5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f44840c5850 0x7f4484155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:09.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.468+0000 7f448a5e5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44840c5d00 con 0x7f4484152890 2024-03-28T23:32:09.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.468+0000 7f448a5e5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4484158a30 con 0x7f44840c5410 2024-03-28T23:32:09.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.468+0000 7f448a5e5640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f44840c8480 con 0x7f44840c5850 2024-03-28T23:32:09.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.468+0000 7f4488b82640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f44840c5850 0x7f4484155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:09.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.468+0000 7f4483fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f4484152890 0x7f44840c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37772/0 (socket says 172.21.15.161:37772) 2024-03-28T23:32:09.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.468+0000 7f4483fff640 1 -- 172.21.15.161:0/3849492271 learned_addr learned my addr 172.21.15.161:0/3849492271 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:09.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.468+0000 7f4488b82640 1 -- 172.21.15.161:0/3849492271 >> v1:172.21.15.170:6789/0 conn(0x7f44840c5410 legacy=0x7f44841553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:09.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.468+0000 7f4488b82640 1 -- 172.21.15.161:0/3849492271 >> v1:172.21.15.161:6789/0 conn(0x7f4484152890 legacy=0x7f44840c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:09.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.468+0000 7f4488b82640 1 -- 172.21.15.161:0/3849492271 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4484158d10 con 0x7f44840c5850 2024-03-28T23:32:09.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.468+0000 7f4482ffd640 1 -- 172.21.15.161:0/3849492271 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3246878351 0 0) 0x7f44840c5d00 con 0x7f4484152890 2024-03-28T23:32:09.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.469+0000 7f4488b82640 1 --2- 172.21.15.161:0/3849492271 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44840c5850 0x7f4484155ab0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f447c009d90 tx=0x7f447c02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b328662975f8d1c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:09.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.469+0000 7f4482ffd640 1 -- 172.21.15.161:0/3849492271 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447c039070 con 0x7f44840c5850 2024-03-28T23:32:09.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.469+0000 7f4482ffd640 1 -- 172.21.15.161:0/3849492271 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f447c02cbd0 con 0x7f44840c5850 2024-03-28T23:32:09.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.469+0000 7f4482ffd640 1 -- 172.21.15.161:0/3849492271 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447c034440 con 0x7f44840c5850 2024-03-28T23:32:09.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.470+0000 7f448a5e5640 1 -- 172.21.15.161:0/3849492271 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44840c5850 msgr2=0x7f4484155ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:09.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.470+0000 7f448a5e5640 1 --2- 172.21.15.161:0/3849492271 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44840c5850 0x7f4484155ab0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f447c009d90 tx=0x7f447c02be20 comp rx=0 tx=0).stop 2024-03-28T23:32:09.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.470+0000 7f448a5e5640 1 -- 172.21.15.161:0/3849492271 shutdown_connections 2024-03-28T23:32:09.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.470+0000 7f448a5e5640 1 --2- 172.21.15.161:0/3849492271 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44840c5850 0x7f4484155ab0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:09.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.470+0000 7f448a5e5640 1 -- 172.21.15.161:0/3849492271 >> 172.21.15.161:0/3849492271 conn(0x7f44840ac620 msgr2=0x7f44840aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:09.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.470+0000 7f448a5e5640 1 -- 172.21.15.161:0/3849492271 shutdown_connections 2024-03-28T23:32:09.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.470+0000 7f448a5e5640 1 -- 172.21.15.161:0/3849492271 wait complete. 2024-03-28T23:32:09.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.471+0000 7f448a5e5640 1 Processor -- start 2024-03-28T23:32:09.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.471+0000 7f448a5e5640 1 -- start start 2024-03-28T23:32:09.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.472+0000 7f448a5e5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44840c5410 0x7f44841e8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:09.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.472+0000 7f448a5e5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44840c5850 0x7f44840beed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:09.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.472+0000 7f448a5e5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4484152890 0x7f44840c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:09.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.472+0000 7f448a5e5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f44841592d0 con 0x7f44840c5850 2024-03-28T23:32:09.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.472+0000 7f448a5e5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4484159640 con 0x7f4484152890 2024-03-28T23:32:09.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.472+0000 7f448a5e5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4484155ff0 con 0x7f44840c5410 2024-03-28T23:32:09.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.472+0000 7f44837fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44840c5850 0x7f44840beed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:09.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.472+0000 7f44837fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44840c5850 0x7f44840beed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57862/0 (socket says 172.21.15.161:57862) 2024-03-28T23:32:09.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.472+0000 7f44837fe640 1 -- 172.21.15.161:0/2968177450 learned_addr learned my addr 172.21.15.161:0/2968177450 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:09.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.472+0000 7f4483fff640 1 --2- 172.21.15.161:0/2968177450 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44840c5410 0x7f44841e8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:09.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.472+0000 7f4488b82640 1 --2- 172.21.15.161:0/2968177450 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4484152890 0x7f44840c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:09.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.472+0000 7f44837fe640 1 -- 172.21.15.161:0/2968177450 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44840c5410 msgr2=0x7f44841e8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:09.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.472+0000 7f44837fe640 1 --2- 172.21.15.161:0/2968177450 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44840c5410 0x7f44841e8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:09.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.472+0000 7f44837fe640 1 -- 172.21.15.161:0/2968177450 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4484152890 msgr2=0x7f44840c9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:09.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.472+0000 7f44837fe640 1 --2- 172.21.15.161:0/2968177450 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4484152890 0x7f44840c9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:09.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.472+0000 7f44837fe640 1 -- 172.21.15.161:0/2968177450 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4484155230 con 0x7f44840c5850 2024-03-28T23:32:09.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.472+0000 7f44837fe640 1 --2- 172.21.15.161:0/2968177450 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44840c5850 0x7f44840beed0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f447800b3e0 tx=0x7f447800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:09.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.473+0000 7f4480ff9640 1 -- 172.21.15.161:0/2968177450 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4478015020 con 0x7f44840c5850 2024-03-28T23:32:09.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.473+0000 7f4480ff9640 1 -- 172.21.15.161:0/2968177450 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4478011830 con 0x7f44840c5850 2024-03-28T23:32:09.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.473+0000 7f448a5e5640 1 -- 172.21.15.161:0/2968177450 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4484154300 con 0x7f44840c5850 2024-03-28T23:32:09.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.473+0000 7f4480ff9640 1 -- 172.21.15.161:0/2968177450 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4478011c20 con 0x7f44840c5850 2024-03-28T23:32:09.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.473+0000 7f448a5e5640 1 -- 172.21.15.161:0/2968177450 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4484153830 con 0x7f44840c5850 2024-03-28T23:32:09.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.474+0000 7f448a5e5640 1 -- 172.21.15.161:0/2968177450 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4484153a30 con 0x7f44840c5850 2024-03-28T23:32:09.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.474+0000 7f4480ff9640 1 -- 172.21.15.161:0/2968177450 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4478018020 con 0x7f44840c5850 2024-03-28T23:32:09.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.474+0000 7f4480ff9640 1 --2- 172.21.15.161:0/2968177450 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f447006b4d0 0x7f447006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:09.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.475+0000 7f4480ff9640 1 -- 172.21.15.161:0/2968177450 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7129+0+0 (secure 0 0 0) 0x7f447808a4b0 con 0x7f44840c5850 2024-03-28T23:32:09.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.475+0000 7f4483fff640 1 --2- 172.21.15.161:0/2968177450 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f447006b4d0 0x7f447006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:09.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.475+0000 7f4483fff640 1 --2- 172.21.15.161:0/2968177450 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f447006b4d0 0x7f447006d990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4474000f60 tx=0x7f4474007b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:09.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.483+0000 7f4480ff9640 1 -- 172.21.15.161:0/2968177450 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f44780202a0 con 0x7f44840c5850 2024-03-28T23:32:09.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.722+0000 7f448a5e5640 1 -- 172.21.15.161:0/2968177450 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f448404f4b0 con 0x7f44840c5850 2024-03-28T23:32:09.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.723+0000 7f4480ff9640 1 -- 172.21.15.161:0/2968177450 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6507 (secure 0 0 0) 0x7f4478058e30 con 0x7f44840c5850 2024-03-28T23:32:09.724 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:09.724 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":43,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5390,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2649192649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2649192649},{"type":"v1","addr":"172.21.15.170:6835","nonce":2649192649}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5542,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.170:6833/3666899516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3666899516},{"type":"v1","addr":"172.21.15.170:6833","nonce":3666899516}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:31:39.725625+0000","modified":"2024-03-28T23:32:09.544987+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":4,"addr":"172.21.15.161:6837/1498528126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":1498528126},{"type":"v1","addr":"172.21.15.161:6837","nonce":1498528126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6165,"qdb_cluster":[6165]},"id":3},{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:31:54.226424+0000","modified":"2024-03-28T23:31:56.347601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5817},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5817":{"gid":5817,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":8,"addr":"172.21.15.161:6836/1300583128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1300583128},{"type":"v1","addr":"172.21.15.161:6836","nonce":1300583128}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5817,"qdb_cluster":[5817]},"id":4}]} 2024-03-28T23:32:09.724 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 43 2024-03-28T23:32:09.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.725+0000 7f448a5e5640 1 -- 172.21.15.161:0/2968177450 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f447006b4d0 msgr2=0x7f447006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:09.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.725+0000 7f448a5e5640 1 --2- 172.21.15.161:0/2968177450 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f447006b4d0 0x7f447006d990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4474000f60 tx=0x7f4474007b50 comp rx=0 tx=0).stop 2024-03-28T23:32:09.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.725+0000 7f448a5e5640 1 -- 172.21.15.161:0/2968177450 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44840c5850 msgr2=0x7f44840beed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:09.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.725+0000 7f448a5e5640 1 --2- 172.21.15.161:0/2968177450 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44840c5850 0x7f44840beed0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f447800b3e0 tx=0x7f447800b8a0 comp rx=0 tx=0).stop 2024-03-28T23:32:09.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.725+0000 7f448a5e5640 1 -- 172.21.15.161:0/2968177450 shutdown_connections 2024-03-28T23:32:09.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.725+0000 7f448a5e5640 1 --2- 172.21.15.161:0/2968177450 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f447006b4d0 0x7f447006d990 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:09.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.725+0000 7f448a5e5640 1 --2- 172.21.15.161:0/2968177450 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44840c5850 0x7f44840beed0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:09.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.725+0000 7f448a5e5640 1 --2- 172.21.15.161:0/2968177450 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4484152890 0x7f44840c9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:09.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.725+0000 7f448a5e5640 1 --2- 172.21.15.161:0/2968177450 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44840c5410 0x7f44841e8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:09.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.725+0000 7f448a5e5640 1 -- 172.21.15.161:0/2968177450 >> 172.21.15.161:0/2968177450 conn(0x7f44840ac620 msgr2=0x7f448414ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:09.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.725+0000 7f448a5e5640 1 -- 172.21.15.161:0/2968177450 shutdown_connections 2024-03-28T23:32:09.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:09.725+0000 7f448a5e5640 1 -- 172.21.15.161:0/2968177450 wait complete. 2024-03-28T23:32:09.736 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-28T23:32:09.737 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-28T23:32:09.737 DEBUG:teuthology.orchestra.run.smithi161:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:32:09.775 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:32:09.818 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-28T23:32:09.818 INFO:tasks.cephfs.mount:Terminating background process 2024-03-28T23:32:09.818 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:32:09.819 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:32:09.834 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20793 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-28T23:32:09.835 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi161.front.sepia.ceph.com 2024-03-28T23:32:09.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:32:09.835 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-28T23:32:09.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:32:09.890 INFO:teuthology.orchestra.run.smithi161.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-28T23:32:09.890 INFO:tasks.cephfs.mount:Terminating background process 2024-03-28T23:32:09.890 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:32:09.891 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:32:09.908 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056374,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19690 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-28T23:32:09.908 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi170.front.sepia.ceph.com 2024-03-28T23:32:09.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:32:09.908 DEBUG:teuthology.orchestra.run.smithi170:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-28T23:32:09.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:32:09.966 INFO:teuthology.orchestra.run.smithi170.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-28T23:32:09.966 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:32:10.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.107+0000 7f75fc18a640 1 Processor -- start 2024-03-28T23:32:10.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.107+0000 7f75fc18a640 1 -- start start 2024-03-28T23:32:10.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.108+0000 7f75fc18a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75f40c5410 0x7f75f40c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:10.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.108+0000 7f75fc18a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f75f41553a0 0x7f75f4155820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:10.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.108+0000 7f75fc18a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75f4158a30 con 0x7f75f4152890 2024-03-28T23:32:10.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.108+0000 7f75fc18a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f75f40c8480 con 0x7f75f41553a0 2024-03-28T23:32:10.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.108+0000 7f75fc18a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f75f40c5d90 con 0x7f75f40c5410 2024-03-28T23:32:10.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.108+0000 7f75f9725640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75f40c5410 0x7f75f40c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:10.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.108+0000 7f75fa727640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f75f41553a0 0x7f75f4155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:10.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.108+0000 7f75f9725640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75f40c5410 0x7f75f40c5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:49692/0 (socket says 172.21.15.161:49692) 2024-03-28T23:32:10.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.108+0000 7f75f9725640 1 -- 172.21.15.161:0/2205069908 learned_addr learned my addr 172.21.15.161:0/2205069908 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:10.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.109+0000 7f75f9725640 1 -- 172.21.15.161:0/2205069908 >> v2:172.21.15.170:3300/0 conn(0x7f75f41553a0 msgr2=0x7f75f4155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:10.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.109+0000 7f75f9725640 1 --2- 172.21.15.161:0/2205069908 >> v2:172.21.15.170:3300/0 conn(0x7f75f41553a0 0x7f75f4155820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.109+0000 7f75f9725640 1 -- 172.21.15.161:0/2205069908 >> v1:172.21.15.170:6789/0 conn(0x7f75f4152890 legacy=0x7f75f40c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:10.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.109+0000 7f75f9725640 1 -- 172.21.15.161:0/2205069908 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f4158d10 con 0x7f75f40c5410 2024-03-28T23:32:10.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.109+0000 7f75fa727640 1 --2- 172.21.15.161:0/2205069908 >> v2:172.21.15.170:3300/0 conn(0x7f75f41553a0 0x7f75f4155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:32:10.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.109+0000 7f75f9725640 1 --2- 172.21.15.161:0/2205069908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75f40c5410 0x7f75f40c5850 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f75e4009850 tx=0x7f75e402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea06b7c395fdeb33 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:10.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.110+0000 7f75f8f24640 1 -- 172.21.15.161:0/2205069908 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e4004230 con 0x7f75f40c5410 2024-03-28T23:32:10.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.110+0000 7f75f8f24640 1 -- 172.21.15.161:0/2205069908 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75e40043d0 con 0x7f75f40c5410 2024-03-28T23:32:10.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.110+0000 7f75f8f24640 1 -- 172.21.15.161:0/2205069908 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e4031830 con 0x7f75f40c5410 2024-03-28T23:32:10.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.110+0000 7f75fc18a640 1 -- 172.21.15.161:0/2205069908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75f40c5410 msgr2=0x7f75f40c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:10.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.110+0000 7f75fc18a640 1 --2- 172.21.15.161:0/2205069908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75f40c5410 0x7f75f40c5850 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f75e4009850 tx=0x7f75e402b8d0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.110+0000 7f75fc18a640 1 -- 172.21.15.161:0/2205069908 shutdown_connections 2024-03-28T23:32:10.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.110+0000 7f75fc18a640 1 --2- 172.21.15.161:0/2205069908 >> v2:172.21.15.170:3300/0 conn(0x7f75f41553a0 0x7f75f4155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.110+0000 7f75fc18a640 1 --2- 172.21.15.161:0/2205069908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75f40c5410 0x7f75f40c5850 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.110+0000 7f75fc18a640 1 -- 172.21.15.161:0/2205069908 >> 172.21.15.161:0/2205069908 conn(0x7f75f40ac620 msgr2=0x7f75f40aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:10.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.110+0000 7f75fc18a640 1 -- 172.21.15.161:0/2205069908 shutdown_connections 2024-03-28T23:32:10.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.110+0000 7f75fc18a640 1 -- 172.21.15.161:0/2205069908 wait complete. 2024-03-28T23:32:10.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.112+0000 7f75fc18a640 1 Processor -- start 2024-03-28T23:32:10.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.112+0000 7f75fc18a640 1 -- start start 2024-03-28T23:32:10.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.113+0000 7f75fc18a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75f40c5410 0x7f75f414b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:10.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.113+0000 7f75fc18a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75f4152890 0x7f75f41eaf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:10.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.113+0000 7f75fc18a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75f41553a0 0x7f75f41eb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:10.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.113+0000 7f75fc18a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f75f41592d0 con 0x7f75f4152890 2024-03-28T23:32:10.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.113+0000 7f75fc18a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f75f4159640 con 0x7f75f40c5410 2024-03-28T23:32:10.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.113+0000 7f75fc18a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f75f40c90a0 con 0x7f75f41553a0 2024-03-28T23:32:10.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.113+0000 7f75f9725640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75f4152890 0x7f75f41eaf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:10.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.113+0000 7f75f9725640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75f4152890 0x7f75f41eaf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57870/0 (socket says 172.21.15.161:57870) 2024-03-28T23:32:10.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.113+0000 7f75f9725640 1 -- 172.21.15.161:0/64631334 learned_addr learned my addr 172.21.15.161:0/64631334 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:10.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.113+0000 7f75fa727640 1 --2- 172.21.15.161:0/64631334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75f41553a0 0x7f75f41eb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:10.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.113+0000 7f75f9f26640 1 --2- 172.21.15.161:0/64631334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75f40c5410 0x7f75f414b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:10.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.114+0000 7f75f9725640 1 -- 172.21.15.161:0/64631334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75f41553a0 msgr2=0x7f75f41eb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:10.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.114+0000 7f75f9725640 1 --2- 172.21.15.161:0/64631334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75f41553a0 0x7f75f41eb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.114+0000 7f75f9725640 1 -- 172.21.15.161:0/64631334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75f40c5410 msgr2=0x7f75f414b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:10.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.114+0000 7f75f9725640 1 --2- 172.21.15.161:0/64631334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75f40c5410 0x7f75f414b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.114+0000 7f75f9725640 1 -- 172.21.15.161:0/64631334 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f41ef360 con 0x7f75f4152890 2024-03-28T23:32:10.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.114+0000 7f75f9725640 1 --2- 172.21.15.161:0/64631334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75f4152890 0x7f75f41eaf90 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f75e4009970 tx=0x7f75e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:10.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.114+0000 7f75ea7fc640 1 -- 172.21.15.161:0/64631334 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e4031830 con 0x7f75f4152890 2024-03-28T23:32:10.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.114+0000 7f75ea7fc640 1 -- 172.21.15.161:0/64631334 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75e40319d0 con 0x7f75f4152890 2024-03-28T23:32:10.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.114+0000 7f75fc18a640 1 -- 172.21.15.161:0/64631334 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f4151d50 con 0x7f75f4152890 2024-03-28T23:32:10.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.114+0000 7f75ea7fc640 1 -- 172.21.15.161:0/64631334 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e4031dc0 con 0x7f75f4152890 2024-03-28T23:32:10.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.114+0000 7f75fc18a640 1 -- 172.21.15.161:0/64631334 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f41ee5a0 con 0x7f75f4152890 2024-03-28T23:32:10.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.115+0000 7f75fc18a640 1 -- 172.21.15.161:0/64631334 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75f404f4b0 con 0x7f75f4152890 2024-03-28T23:32:10.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.116+0000 7f75ea7fc640 1 -- 172.21.15.161:0/64631334 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f75e4004330 con 0x7f75f4152890 2024-03-28T23:32:10.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.116+0000 7f75ea7fc640 1 --2- 172.21.15.161:0/64631334 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f75d806b4d0 0x7f75d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:10.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.116+0000 7f75ea7fc640 1 -- 172.21.15.161:0/64631334 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7129+0+0 (secure 0 0 0) 0x7f75e40ac9f0 con 0x7f75f4152890 2024-03-28T23:32:10.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.116+0000 7f75f9f26640 1 --2- 172.21.15.161:0/64631334 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f75d806b4d0 0x7f75d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:10.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.117+0000 7f75f9f26640 1 --2- 172.21.15.161:0/64631334 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f75d806b4d0 0x7f75d806d990 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f75f0001780 tx=0x7f75f0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:10.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.121+0000 7f75ea7fc640 1 -- 172.21.15.161:0/64631334 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f75e407b1d0 con 0x7f75f4152890 2024-03-28T23:32:10.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.349+0000 7f75fc18a640 1 -- 172.21.15.161:0/64631334 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75f40ac400 con 0x7f75f4152890 2024-03-28T23:32:10.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.350+0000 7f75ea7fc640 1 -- 172.21.15.161:0/64631334 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6507 (secure 0 0 0) 0x7f75e407aad0 con 0x7f75f4152890 2024-03-28T23:32:10.351 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:10.351 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":43,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5390,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2649192649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2649192649},{"type":"v1","addr":"172.21.15.170:6835","nonce":2649192649}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5542,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.170:6833/3666899516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3666899516},{"type":"v1","addr":"172.21.15.170:6833","nonce":3666899516}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:31:39.725625+0000","modified":"2024-03-28T23:32:09.544987+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":4,"addr":"172.21.15.161:6837/1498528126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":1498528126},{"type":"v1","addr":"172.21.15.161:6837","nonce":1498528126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6165,"qdb_cluster":[6165]},"id":3},{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:31:54.226424+0000","modified":"2024-03-28T23:31:56.347601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5817},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5817":{"gid":5817,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":8,"addr":"172.21.15.161:6836/1300583128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1300583128},{"type":"v1","addr":"172.21.15.161:6836","nonce":1300583128}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5817,"qdb_cluster":[5817]},"id":4}]} 2024-03-28T23:32:10.351 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 43 2024-03-28T23:32:10.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.352+0000 7f75fc18a640 1 -- 172.21.15.161:0/64631334 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f75d806b4d0 msgr2=0x7f75d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:10.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.352+0000 7f75fc18a640 1 --2- 172.21.15.161:0/64631334 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f75d806b4d0 0x7f75d806d990 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f75f0001780 tx=0x7f75f0009040 comp rx=0 tx=0).stop 2024-03-28T23:32:10.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.352+0000 7f75fc18a640 1 -- 172.21.15.161:0/64631334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75f4152890 msgr2=0x7f75f41eaf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:10.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.352+0000 7f75fc18a640 1 --2- 172.21.15.161:0/64631334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75f4152890 0x7f75f41eaf90 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f75e4009970 tx=0x7f75e40046c0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.352+0000 7f75fc18a640 1 -- 172.21.15.161:0/64631334 shutdown_connections 2024-03-28T23:32:10.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.352+0000 7f75fc18a640 1 --2- 172.21.15.161:0/64631334 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f75d806b4d0 0x7f75d806d990 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.352+0000 7f75fc18a640 1 --2- 172.21.15.161:0/64631334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75f4152890 0x7f75f41eaf90 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.352+0000 7f75fc18a640 1 --2- 172.21.15.161:0/64631334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75f41553a0 0x7f75f41eb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.352+0000 7f75fc18a640 1 --2- 172.21.15.161:0/64631334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75f40c5410 0x7f75f414b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.352+0000 7f75fc18a640 1 -- 172.21.15.161:0/64631334 >> 172.21.15.161:0/64631334 conn(0x7f75f40ac620 msgr2=0x7f75f40bddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:10.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.352+0000 7f75fc18a640 1 -- 172.21.15.161:0/64631334 shutdown_connections 2024-03-28T23:32:10.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.352+0000 7f75fc18a640 1 -- 172.21.15.161:0/64631334 wait complete. 2024-03-28T23:32:10.368 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:32:10.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.469+0000 7ff39c772640 1 Processor -- start 2024-03-28T23:32:10.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.469+0000 7ff39c772640 1 -- start start 2024-03-28T23:32:10.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.470+0000 7ff39c772640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff394152890 0x7ff3940c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:10.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.470+0000 7ff39c772640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff3940c5b20 0x7ff3941553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:10.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.470+0000 7ff39c772640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff394158a30 con 0x7ff3940c8ed0 2024-03-28T23:32:10.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.470+0000 7ff39c772640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7ff3940c8480 con 0x7ff3940c5b20 2024-03-28T23:32:10.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.470+0000 7ff39c772640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff394159640 con 0x7ff394152890 2024-03-28T23:32:10.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.471+0000 7ff39a50e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff394152890 0x7ff3940c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:10.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.471+0000 7ff39ad0f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff3940c5b20 0x7ff3941553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:10.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.471+0000 7ff39a50e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff394152890 0x7ff3940c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:49722/0 (socket says 172.21.15.161:49722) 2024-03-28T23:32:10.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.471+0000 7ff39a50e640 1 -- 172.21.15.161:0/3532892701 learned_addr learned my addr 172.21.15.161:0/3532892701 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:10.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.471+0000 7ff39ad0f640 1 --2- 172.21.15.161:0/3532892701 >> v2:172.21.15.170:3300/0 conn(0x7ff3940c5b20 0x7ff3941553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:60266/0 (socket says 172.21.15.161:60266) 2024-03-28T23:32:10.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.471+0000 7ff39a50e640 1 -- 172.21.15.161:0/3532892701 >> v2:172.21.15.170:3300/0 conn(0x7ff3940c5b20 msgr2=0x7ff3941553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:10.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.471+0000 7ff39a50e640 1 --2- 172.21.15.161:0/3532892701 >> v2:172.21.15.170:3300/0 conn(0x7ff3940c5b20 0x7ff3941553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.471+0000 7ff39a50e640 1 -- 172.21.15.161:0/3532892701 >> v1:172.21.15.170:6789/0 conn(0x7ff3940c8ed0 legacy=0x7ff3940c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:10.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.471+0000 7ff39a50e640 1 -- 172.21.15.161:0/3532892701 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff394158d10 con 0x7ff394152890 2024-03-28T23:32:10.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.472+0000 7ff39a50e640 1 --2- 172.21.15.161:0/3532892701 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff394152890 0x7ff3940c8990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff3941597c0 tx=0x7ff39002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c3a0671ba144cb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:10.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.472+0000 7ff39950c640 1 -- 172.21.15.161:0/3532892701 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3900317c0 con 0x7ff394152890 2024-03-28T23:32:10.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.472+0000 7ff39950c640 1 -- 172.21.15.161:0/3532892701 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff390037070 con 0x7ff394152890 2024-03-28T23:32:10.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.472+0000 7ff39950c640 1 -- 172.21.15.161:0/3532892701 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff390031b80 con 0x7ff394152890 2024-03-28T23:32:10.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.472+0000 7ff39c772640 1 -- 172.21.15.161:0/3532892701 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff394152890 msgr2=0x7ff3940c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:10.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.472+0000 7ff39c772640 1 --2- 172.21.15.161:0/3532892701 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff394152890 0x7ff3940c8990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff3941597c0 tx=0x7ff39002b860 comp rx=0 tx=0).stop 2024-03-28T23:32:10.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.472+0000 7ff39c772640 1 -- 172.21.15.161:0/3532892701 shutdown_connections 2024-03-28T23:32:10.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.472+0000 7ff39c772640 1 --2- 172.21.15.161:0/3532892701 >> v2:172.21.15.170:3300/0 conn(0x7ff3940c5b20 0x7ff3941553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.472+0000 7ff39c772640 1 --2- 172.21.15.161:0/3532892701 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff394152890 0x7ff3940c8990 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.473+0000 7ff39c772640 1 -- 172.21.15.161:0/3532892701 >> 172.21.15.161:0/3532892701 conn(0x7ff3940ac620 msgr2=0x7ff3940aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:10.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.473+0000 7ff39c772640 1 -- 172.21.15.161:0/3532892701 shutdown_connections 2024-03-28T23:32:10.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.473+0000 7ff39c772640 1 -- 172.21.15.161:0/3532892701 wait complete. 2024-03-28T23:32:10.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.474+0000 7ff39c772640 1 Processor -- start 2024-03-28T23:32:10.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.474+0000 7ff39c772640 1 -- start start 2024-03-28T23:32:10.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.474+0000 7ff39c772640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3940c5b20 0x7ff394155040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:10.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.474+0000 7ff39c772640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3940c8ed0 0x7ff394144aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:10.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.474+0000 7ff39c772640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff394152890 0x7ff394155af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:10.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.474+0000 7ff39c772640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff394152c70 con 0x7ff394152890 2024-03-28T23:32:10.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.474+0000 7ff39c772640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff3941592d0 con 0x7ff3940c8ed0 2024-03-28T23:32:10.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.474+0000 7ff39c772640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff394152df0 con 0x7ff3940c5b20 2024-03-28T23:32:10.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.474+0000 7ff39ad0f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff394152890 0x7ff394155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:10.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.474+0000 7ff39ad0f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff394152890 0x7ff394155af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57884/0 (socket says 172.21.15.161:57884) 2024-03-28T23:32:10.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.475+0000 7ff39ad0f640 1 -- 172.21.15.161:0/3837735948 learned_addr learned my addr 172.21.15.161:0/3837735948 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:10.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.475+0000 7ff39a50e640 1 --2- 172.21.15.161:0/3837735948 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3940c5b20 0x7ff394155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:10.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.475+0000 7ff399d0d640 1 --2- 172.21.15.161:0/3837735948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3940c8ed0 0x7ff394144aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:10.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.475+0000 7ff39ad0f640 1 -- 172.21.15.161:0/3837735948 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3940c5b20 msgr2=0x7ff394155040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:10.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.475+0000 7ff39ad0f640 1 --2- 172.21.15.161:0/3837735948 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3940c5b20 0x7ff394155040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.475+0000 7ff39ad0f640 1 -- 172.21.15.161:0/3837735948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3940c8ed0 msgr2=0x7ff394144aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:10.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.475+0000 7ff39ad0f640 1 --2- 172.21.15.161:0/3837735948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3940c8ed0 0x7ff394144aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.475+0000 7ff39ad0f640 1 -- 172.21.15.161:0/3837735948 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff394147a60 con 0x7ff394152890 2024-03-28T23:32:10.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.475+0000 7ff39ad0f640 1 --2- 172.21.15.161:0/3837735948 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff394152890 0x7ff394155af0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7ff38c00efc0 tx=0x7ff38c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:10.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.475+0000 7ff38affd640 1 -- 172.21.15.161:0/3837735948 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38c018020 con 0x7ff394152890 2024-03-28T23:32:10.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.475+0000 7ff38affd640 1 -- 172.21.15.161:0/3837735948 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff38c014480 con 0x7ff394152890 2024-03-28T23:32:10.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.475+0000 7ff39c772640 1 -- 172.21.15.161:0/3837735948 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3940ab7c0 con 0x7ff394152890 2024-03-28T23:32:10.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.475+0000 7ff38affd640 1 -- 172.21.15.161:0/3837735948 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38c009e60 con 0x7ff394152890 2024-03-28T23:32:10.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.475+0000 7ff39c772640 1 -- 172.21.15.161:0/3837735948 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff394155580 con 0x7ff394152890 2024-03-28T23:32:10.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.476+0000 7ff39c772640 1 -- 172.21.15.161:0/3837735948 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff39404f4b0 con 0x7ff394152890 2024-03-28T23:32:10.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.477+0000 7ff38affd640 1 -- 172.21.15.161:0/3837735948 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff38c005d10 con 0x7ff394152890 2024-03-28T23:32:10.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.477+0000 7ff38affd640 1 --2- 172.21.15.161:0/3837735948 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff37806b4d0 0x7ff37806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:10.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.477+0000 7ff38affd640 1 -- 172.21.15.161:0/3837735948 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7129+0+0 (secure 0 0 0) 0x7ff38c08b330 con 0x7ff394152890 2024-03-28T23:32:10.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.478+0000 7ff39a50e640 1 --2- 172.21.15.161:0/3837735948 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff37806b4d0 0x7ff37806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:10.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.478+0000 7ff39a50e640 1 --2- 172.21.15.161:0/3837735948 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff37806b4d0 0x7ff37806d990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff390036000 tx=0x7ff39003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:10.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.486+0000 7ff38affd640 1 -- 172.21.15.161:0/3837735948 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff38c022030 con 0x7ff394152890 2024-03-28T23:32:10.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.689+0000 7ff39c772640 1 -- 172.21.15.161:0/3837735948 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff3941576f0 con 0x7ff394152890 2024-03-28T23:32:10.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.691+0000 7ff38affd640 1 -- 172.21.15.161:0/3837735948 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+20963 (secure 0 0 0) 0x7ff38c010040 con 0x7ff394152890 2024-03-28T23:32:10.692 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:10.692 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":57,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:32:06.533687+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:31:37.940838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-28T23:31:38.692659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:31:52.774315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:31:53.201664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6835/1945122204":"2024-03-29T23:32:06.533661+0000","172.21.15.170:6834/1945122204":"2024-03-29T23:32:06.533661+0000","172.21.15.161:6837/544971001":"2024-03-29T23:32:03.518509+0000","172.21.15.161:6835/544971001":"2024-03-29T23:32:03.518509+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:32:10.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.693+0000 7ff39c772640 1 -- 172.21.15.161:0/3837735948 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff37806b4d0 msgr2=0x7ff37806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:10.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.693+0000 7ff39c772640 1 --2- 172.21.15.161:0/3837735948 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff37806b4d0 0x7ff37806d990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff390036000 tx=0x7ff39003d040 comp rx=0 tx=0).stop 2024-03-28T23:32:10.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.694+0000 7ff39c772640 1 -- 172.21.15.161:0/3837735948 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff394152890 msgr2=0x7ff394155af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:10.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.694+0000 7ff39c772640 1 --2- 172.21.15.161:0/3837735948 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff394152890 0x7ff394155af0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7ff38c00efc0 tx=0x7ff38c00c490 comp rx=0 tx=0).stop 2024-03-28T23:32:10.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.694+0000 7ff39c772640 1 -- 172.21.15.161:0/3837735948 shutdown_connections 2024-03-28T23:32:10.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.694+0000 7ff39c772640 1 --2- 172.21.15.161:0/3837735948 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff37806b4d0 0x7ff37806d990 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.694+0000 7ff39c772640 1 --2- 172.21.15.161:0/3837735948 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff394152890 0x7ff394155af0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.694+0000 7ff39c772640 1 --2- 172.21.15.161:0/3837735948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3940c8ed0 0x7ff394144aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.694+0000 7ff39c772640 1 --2- 172.21.15.161:0/3837735948 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3940c5b20 0x7ff394155040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.694+0000 7ff39c772640 1 -- 172.21.15.161:0/3837735948 >> 172.21.15.161:0/3837735948 conn(0x7ff3940ac620 msgr2=0x7ff39414ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:10.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.694+0000 7ff39c772640 1 -- 172.21.15.161:0/3837735948 shutdown_connections 2024-03-28T23:32:10.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.694+0000 7ff39c772640 1 -- 172.21.15.161:0/3837735948 wait complete. 2024-03-28T23:32:10.709 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:31:37.940838+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:31:38.692659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-03-28T23:31:52.774315+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs2_data', 'create_time': '2024-03-28T23:31:53.201664+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:32:10.710 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:32:10.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.819+0000 7fa1707d0640 1 Processor -- start 2024-03-28T23:32:10.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.820+0000 7fa1707d0640 1 -- start start 2024-03-28T23:32:10.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.820+0000 7fa1707d0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa1680c6750 0x7fa16806bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:10.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.820+0000 7fa1707d0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa1680c88b0 0x7fa1680c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:10.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.820+0000 7fa1707d0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa168152db0 con 0x7fa1681529d0 2024-03-28T23:32:10.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.820+0000 7fa1707d0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa1680c7210 con 0x7fa1680c88b0 2024-03-28T23:32:10.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.820+0000 7fa1707d0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fa1680c6c50 con 0x7fa1680c6750 2024-03-28T23:32:10.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.821+0000 7fa16ed6d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa1680c88b0 0x7fa1680c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:10.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.821+0000 7fa16ed6d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa1680c88b0 0x7fa1680c54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41358/0 (socket says 172.21.15.161:41358) 2024-03-28T23:32:10.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.821+0000 7fa16ed6d640 1 -- 172.21.15.161:0/1437824538 learned_addr learned my addr 172.21.15.161:0/1437824538 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:10.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.821+0000 7fa16dd6b640 1 --2- 172.21.15.161:0/1437824538 >> v2:172.21.15.170:3300/0 conn(0x7fa1680c6750 0x7fa16806bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:10.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.821+0000 7fa16ed6d640 1 -- 172.21.15.161:0/1437824538 >> v2:172.21.15.170:3300/0 conn(0x7fa1680c6750 msgr2=0x7fa16806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:10.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.821+0000 7fa16ed6d640 1 --2- 172.21.15.161:0/1437824538 >> v2:172.21.15.170:3300/0 conn(0x7fa1680c6750 0x7fa16806bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.821+0000 7fa16ed6d640 1 -- 172.21.15.161:0/1437824538 >> v1:172.21.15.170:6789/0 conn(0x7fa1681529d0 legacy=0x7fa168155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:10.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.821+0000 7fa16ed6d640 1 -- 172.21.15.161:0/1437824538 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1680c8d40 con 0x7fa1680c88b0 2024-03-28T23:32:10.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.822+0000 7fa16ed6d640 1 --2- 172.21.15.161:0/1437824538 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1680c88b0 0x7fa1680c54f0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fa1680739b0 tx=0x7fa16002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1439a3235724a929 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:10.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.822+0000 7fa16d56a640 1 -- 172.21.15.161:0/1437824538 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16002cb20 con 0x7fa1680c88b0 2024-03-28T23:32:10.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.822+0000 7fa16d56a640 1 -- 172.21.15.161:0/1437824538 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa16002ccc0 con 0x7fa1680c88b0 2024-03-28T23:32:10.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.822+0000 7fa16d56a640 1 -- 172.21.15.161:0/1437824538 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa160034610 con 0x7fa1680c88b0 2024-03-28T23:32:10.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.822+0000 7fa1707d0640 1 -- 172.21.15.161:0/1437824538 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1680c88b0 msgr2=0x7fa1680c54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:10.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.822+0000 7fa1707d0640 1 --2- 172.21.15.161:0/1437824538 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1680c88b0 0x7fa1680c54f0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fa1680739b0 tx=0x7fa16002bf80 comp rx=0 tx=0).stop 2024-03-28T23:32:10.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.822+0000 7fa1707d0640 1 -- 172.21.15.161:0/1437824538 shutdown_connections 2024-03-28T23:32:10.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.822+0000 7fa1707d0640 1 --2- 172.21.15.161:0/1437824538 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1680c88b0 0x7fa1680c54f0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.822+0000 7fa1707d0640 1 --2- 172.21.15.161:0/1437824538 >> v2:172.21.15.170:3300/0 conn(0x7fa1680c6750 0x7fa16806bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.822+0000 7fa1707d0640 1 -- 172.21.15.161:0/1437824538 >> 172.21.15.161:0/1437824538 conn(0x7fa1680c7430 msgr2=0x7fa1680c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:10.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.822+0000 7fa1707d0640 1 -- 172.21.15.161:0/1437824538 shutdown_connections 2024-03-28T23:32:10.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.822+0000 7fa1707d0640 1 -- 172.21.15.161:0/1437824538 wait complete. 2024-03-28T23:32:10.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.823+0000 7fa1707d0640 1 Processor -- start 2024-03-28T23:32:10.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.824+0000 7fa1707d0640 1 -- start start 2024-03-28T23:32:10.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.824+0000 7fa1707d0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1680c6750 0x7fa16814b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:10.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.824+0000 7fa1707d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1680c88b0 0x7fa1681eb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:10.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.824+0000 7fa1707d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1681529d0 0x7fa1681eb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:10.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.824+0000 7fa1707d0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa168155710 con 0x7fa1680c6750 2024-03-28T23:32:10.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.824+0000 7fa1707d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa168155430 con 0x7fa1681529d0 2024-03-28T23:32:10.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.824+0000 7fa1707d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa1680aa160 con 0x7fa1680c88b0 2024-03-28T23:32:10.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.824+0000 7fa16e56c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1680c6750 0x7fa16814b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:10.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.824+0000 7fa16e56c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1680c6750 0x7fa16814b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41366/0 (socket says 172.21.15.161:41366) 2024-03-28T23:32:10.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.824+0000 7fa16e56c640 1 -- 172.21.15.161:0/3687101197 learned_addr learned my addr 172.21.15.161:0/3687101197 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:10.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.824+0000 7fa16ed6d640 1 --2- 172.21.15.161:0/3687101197 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1681529d0 0x7fa1681eb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:10.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.824+0000 7fa16dd6b640 1 --2- 172.21.15.161:0/3687101197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1680c88b0 0x7fa1681eb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:10.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.825+0000 7fa16e56c640 1 -- 172.21.15.161:0/3687101197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1680c88b0 msgr2=0x7fa1681eb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:10.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.825+0000 7fa16e56c640 1 --2- 172.21.15.161:0/3687101197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1680c88b0 0x7fa1681eb380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.825+0000 7fa16e56c640 1 -- 172.21.15.161:0/3687101197 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1681529d0 msgr2=0x7fa1681eb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:10.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.825+0000 7fa16e56c640 1 --2- 172.21.15.161:0/3687101197 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1681529d0 0x7fa1681eb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:10.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.825+0000 7fa16e56c640 1 -- 172.21.15.161:0/3687101197 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1681ef1b0 con 0x7fa1680c6750 2024-03-28T23:32:10.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.825+0000 7fa16e56c640 1 --2- 172.21.15.161:0/3687101197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1680c6750 0x7fa16814b2d0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fa16400c400 tx=0x7fa16400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:10.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.825+0000 7fa15effd640 1 -- 172.21.15.161:0/3687101197 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1640052d0 con 0x7fa1680c6750 2024-03-28T23:32:10.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.825+0000 7fa15effd640 1 -- 172.21.15.161:0/3687101197 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa164005490 con 0x7fa1680c6750 2024-03-28T23:32:10.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.825+0000 7fa1707d0640 1 -- 172.21.15.161:0/3687101197 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa168154480 con 0x7fa1680c6750 2024-03-28T23:32:10.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.825+0000 7fa15effd640 1 -- 172.21.15.161:0/3687101197 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa164007260 con 0x7fa1680c6750 2024-03-28T23:32:10.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.825+0000 7fa1707d0640 1 -- 172.21.15.161:0/3687101197 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa168154a40 con 0x7fa1680c6750 2024-03-28T23:32:10.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.826+0000 7fa1707d0640 1 -- 172.21.15.161:0/3687101197 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa16804f4b0 con 0x7fa1680c6750 2024-03-28T23:32:10.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.827+0000 7fa15effd640 1 -- 172.21.15.161:0/3687101197 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa164007400 con 0x7fa1680c6750 2024-03-28T23:32:10.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.827+0000 7fa15effd640 1 --2- 172.21.15.161:0/3687101197 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa14c06b4d0 0x7fa14c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:10.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.827+0000 7fa15effd640 1 -- 172.21.15.161:0/3687101197 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7129+0+0 (secure 0 0 0) 0x7fa164089bd0 con 0x7fa1680c6750 2024-03-28T23:32:10.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.827+0000 7fa16dd6b640 1 --2- 172.21.15.161:0/3687101197 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa14c06b4d0 0x7fa14c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:10.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.828+0000 7fa16dd6b640 1 --2- 172.21.15.161:0/3687101197 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa14c06b4d0 0x7fa14c06d990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa168154ca0 tx=0x7fa158009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:10.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:10.835+0000 7fa15effd640 1 -- 172.21.15.161:0/3687101197 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa164057f00 con 0x7fa1680c6750 2024-03-28T23:32:11.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.018+0000 7fa1707d0640 1 -- 172.21.15.161:0/3687101197 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa1680ca650 con 0x7fa1680c6750 2024-03-28T23:32:11.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.019+0000 7fa15effd640 1 -- 172.21.15.161:0/3687101197 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) ==== 93+0+10 (secure 0 0 0) 0x7fa16405bfa0 con 0x7fa1680c6750 2024-03-28T23:32:11.020 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:32:11.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.021+0000 7fa1707d0640 1 -- 172.21.15.161:0/3687101197 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa14c06b4d0 msgr2=0x7fa14c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:11.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.021+0000 7fa1707d0640 1 --2- 172.21.15.161:0/3687101197 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa14c06b4d0 0x7fa14c06d990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa168154ca0 tx=0x7fa158009190 comp rx=0 tx=0).stop 2024-03-28T23:32:11.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.021+0000 7fa1707d0640 1 -- 172.21.15.161:0/3687101197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1680c6750 msgr2=0x7fa16814b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:11.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.021+0000 7fa1707d0640 1 --2- 172.21.15.161:0/3687101197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1680c6750 0x7fa16814b2d0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fa16400c400 tx=0x7fa16400c8c0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.021+0000 7fa1707d0640 1 -- 172.21.15.161:0/3687101197 shutdown_connections 2024-03-28T23:32:11.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.021+0000 7fa1707d0640 1 --2- 172.21.15.161:0/3687101197 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa14c06b4d0 0x7fa14c06d990 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.021+0000 7fa1707d0640 1 --2- 172.21.15.161:0/3687101197 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1681529d0 0x7fa1681eb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.021+0000 7fa1707d0640 1 --2- 172.21.15.161:0/3687101197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1680c88b0 0x7fa1681eb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.021+0000 7fa1707d0640 1 --2- 172.21.15.161:0/3687101197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1680c6750 0x7fa16814b2d0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.021+0000 7fa1707d0640 1 -- 172.21.15.161:0/3687101197 >> 172.21.15.161:0/3687101197 conn(0x7fa1680c7430 msgr2=0x7fa168150cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:11.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.021+0000 7fa1707d0640 1 -- 172.21.15.161:0/3687101197 shutdown_connections 2024-03-28T23:32:11.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.021+0000 7fa1707d0640 1 -- 172.21.15.161:0/3687101197 wait complete. 2024-03-28T23:32:11.033 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:32:11.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.138+0000 7f5a171ae640 1 Processor -- start 2024-03-28T23:32:11.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.138+0000 7f5a171ae640 1 -- start start 2024-03-28T23:32:11.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.139+0000 7f5a171ae640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5a10156080 0x7f5a100c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:11.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.139+0000 7f5a171ae640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5a1006ff90 0x7f5a10156b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:11.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.139+0000 7f5a171ae640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a10156460 0x7f5a101570c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:11.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.139+0000 7f5a171ae640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5a100c67b0 con 0x7f5a1006ff90 2024-03-28T23:32:11.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.139+0000 7f5a171ae640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f5a10072790 con 0x7f5a10156080 2024-03-28T23:32:11.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.139+0000 7f5a171ae640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5a100a7ea0 con 0x7f5a10156460 2024-03-28T23:32:11.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.139+0000 7f5a0ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5a1006ff90 0x7f5a10156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:11.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.139+0000 7f5a0ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5a1006ff90 0x7f5a10156b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41370/0 (socket says 172.21.15.161:41370) 2024-03-28T23:32:11.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.139+0000 7f5a0ffff640 1 -- 172.21.15.161:0/2821454113 learned_addr learned my addr 172.21.15.161:0/2821454113 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:11.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.139+0000 7f5a14f4a640 1 --2- 172.21.15.161:0/2821454113 >> v2:172.21.15.170:3300/0 conn(0x7f5a10156080 0x7f5a100c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:11.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.139+0000 7f5a1574b640 1 --2- 172.21.15.161:0/2821454113 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a10156460 0x7f5a101570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:11.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.139+0000 7f5a0ffff640 1 -- 172.21.15.161:0/2821454113 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a10156460 msgr2=0x7f5a101570c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:11.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.139+0000 7f5a0ffff640 1 --2- 172.21.15.161:0/2821454113 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a10156460 0x7f5a101570c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.139+0000 7f5a0ffff640 1 -- 172.21.15.161:0/2821454113 >> v2:172.21.15.170:3300/0 conn(0x7f5a10156080 msgr2=0x7f5a100c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:11.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.139+0000 7f5a0ffff640 1 --2- 172.21.15.161:0/2821454113 >> v2:172.21.15.170:3300/0 conn(0x7f5a10156080 0x7f5a100c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.139+0000 7f5a0ffff640 1 -- 172.21.15.161:0/2821454113 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a100a8180 con 0x7f5a1006ff90 2024-03-28T23:32:11.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.139+0000 7f5a1574b640 1 --2- 172.21.15.161:0/2821454113 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a10156460 0x7f5a101570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:11.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.139+0000 7f5a14f4a640 1 --2- 172.21.15.161:0/2821454113 >> v2:172.21.15.170:3300/0 conn(0x7f5a10156080 0x7f5a100c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:11.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.140+0000 7f5a0ffff640 1 --2- 172.21.15.161:0/2821454113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a1006ff90 0x7f5a10156b80 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f5a100ab6a0 tx=0x7f59fc02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57623610840f7280 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:11.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.140+0000 7f5a0f7fe640 1 -- 172.21.15.161:0/2821454113 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc038070 con 0x7f5a1006ff90 2024-03-28T23:32:11.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.140+0000 7f5a0f7fe640 1 -- 172.21.15.161:0/2821454113 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59fc02f050 con 0x7f5a1006ff90 2024-03-28T23:32:11.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.140+0000 7f5a0f7fe640 1 -- 172.21.15.161:0/2821454113 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc031970 con 0x7f5a1006ff90 2024-03-28T23:32:11.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.141+0000 7f5a171ae640 1 -- 172.21.15.161:0/2821454113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a1006ff90 msgr2=0x7f5a10156b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:11.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.141+0000 7f5a171ae640 1 --2- 172.21.15.161:0/2821454113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a1006ff90 0x7f5a10156b80 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f5a100ab6a0 tx=0x7f59fc02b7f0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.141+0000 7f5a171ae640 1 -- 172.21.15.161:0/2821454113 shutdown_connections 2024-03-28T23:32:11.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.141+0000 7f5a171ae640 1 --2- 172.21.15.161:0/2821454113 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a10156460 0x7f5a101570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.141+0000 7f5a171ae640 1 --2- 172.21.15.161:0/2821454113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a1006ff90 0x7f5a10156b80 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.141+0000 7f5a171ae640 1 --2- 172.21.15.161:0/2821454113 >> v2:172.21.15.170:3300/0 conn(0x7f5a10156080 0x7f5a100c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.141+0000 7f5a171ae640 1 -- 172.21.15.161:0/2821454113 >> 172.21.15.161:0/2821454113 conn(0x7f5a1006eba0 msgr2=0x7f5a1006b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:11.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.141+0000 7f5a171ae640 1 -- 172.21.15.161:0/2821454113 shutdown_connections 2024-03-28T23:32:11.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.141+0000 7f5a171ae640 1 -- 172.21.15.161:0/2821454113 wait complete. 2024-03-28T23:32:11.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.142+0000 7f5a171ae640 1 Processor -- start 2024-03-28T23:32:11.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.142+0000 7f5a171ae640 1 -- start start 2024-03-28T23:32:11.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.143+0000 7f5a171ae640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a1006ff90 0x7f5a101ee640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:11.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.143+0000 7f5a171ae640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a10156080 0x7f5a1006cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:11.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.143+0000 7f5a171ae640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a10156460 0x7f5a100c1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:11.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.143+0000 7f5a171ae640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5a100c2fd0 con 0x7f5a1006ff90 2024-03-28T23:32:11.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.143+0000 7f5a171ae640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5a100c2e50 con 0x7f5a10156080 2024-03-28T23:32:11.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.143+0000 7f5a171ae640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5a100704a0 con 0x7f5a10156460 2024-03-28T23:32:11.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.143+0000 7f5a14f4a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a1006ff90 0x7f5a101ee640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:11.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.143+0000 7f5a14f4a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a1006ff90 0x7f5a101ee640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41380/0 (socket says 172.21.15.161:41380) 2024-03-28T23:32:11.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.143+0000 7f5a14f4a640 1 -- 172.21.15.161:0/1193600341 learned_addr learned my addr 172.21.15.161:0/1193600341 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:11.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.143+0000 7f5a1574b640 1 --2- 172.21.15.161:0/1193600341 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a10156460 0x7f5a100c1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:11.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.143+0000 7f5a0ffff640 1 --2- 172.21.15.161:0/1193600341 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a10156080 0x7f5a1006cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:11.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.143+0000 7f5a14f4a640 1 -- 172.21.15.161:0/1193600341 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a10156460 msgr2=0x7f5a100c1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:11.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.143+0000 7f5a14f4a640 1 --2- 172.21.15.161:0/1193600341 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a10156460 0x7f5a100c1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.143+0000 7f5a14f4a640 1 -- 172.21.15.161:0/1193600341 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a10156080 msgr2=0x7f5a1006cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:11.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.143+0000 7f5a14f4a640 1 --2- 172.21.15.161:0/1193600341 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a10156080 0x7f5a1006cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.143+0000 7f5a14f4a640 1 -- 172.21.15.161:0/1193600341 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a101eeb80 con 0x7f5a1006ff90 2024-03-28T23:32:11.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.144+0000 7f5a14f4a640 1 --2- 172.21.15.161:0/1193600341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a1006ff90 0x7f5a101ee640 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f5a0400c7c0 tx=0x7f5a0400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:11.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.144+0000 7f5a0d7fa640 1 -- 172.21.15.161:0/1193600341 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a04015070 con 0x7f5a1006ff90 2024-03-28T23:32:11.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.144+0000 7f5a0d7fa640 1 -- 172.21.15.161:0/1193600341 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a04007a30 con 0x7f5a1006ff90 2024-03-28T23:32:11.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.144+0000 7f5a171ae640 1 -- 172.21.15.161:0/1193600341 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a1006d460 con 0x7f5a1006ff90 2024-03-28T23:32:11.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.144+0000 7f5a0d7fa640 1 -- 172.21.15.161:0/1193600341 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a04007e00 con 0x7f5a1006ff90 2024-03-28T23:32:11.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.144+0000 7f5a171ae640 1 -- 172.21.15.161:0/1193600341 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a1006c440 con 0x7f5a1006ff90 2024-03-28T23:32:11.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.145+0000 7f5a171ae640 1 -- 172.21.15.161:0/1193600341 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a1006c260 con 0x7f5a1006ff90 2024-03-28T23:32:11.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.146+0000 7f5a0d7fa640 1 -- 172.21.15.161:0/1193600341 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5a04018070 con 0x7f5a1006ff90 2024-03-28T23:32:11.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.146+0000 7f5a0d7fa640 1 --2- 172.21.15.161:0/1193600341 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f59f406b4d0 0x7f59f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:11.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.146+0000 7f5a0d7fa640 1 -- 172.21.15.161:0/1193600341 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7129+0+0 (secure 0 0 0) 0x7f5a0408a6d0 con 0x7f5a1006ff90 2024-03-28T23:32:11.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.146+0000 7f5a0ffff640 1 --2- 172.21.15.161:0/1193600341 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f59f406b4d0 0x7f59f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:11.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.147+0000 7f5a0ffff640 1 --2- 172.21.15.161:0/1193600341 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f59f406b4d0 0x7f59f406d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f59fc035000 tx=0x7f59fc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:11.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.151+0000 7f5a0d7fa640 1 -- 172.21.15.161:0/1193600341 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5a1006c260 con 0x7f5a1006ff90 2024-03-28T23:32:11.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.328+0000 7f5a171ae640 1 -- 172.21.15.161:0/1193600341 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5a1006fd30 con 0x7f5a1006ff90 2024-03-28T23:32:11.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.328+0000 7f5a0d7fa640 1 -- 172.21.15.161:0/1193600341 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) ==== 104+0+11 (secure 0 0 0) 0x7f5a04058fb0 con 0x7f5a1006ff90 2024-03-28T23:32:11.329 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:32:11.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.330+0000 7f5a171ae640 1 -- 172.21.15.161:0/1193600341 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f59f406b4d0 msgr2=0x7f59f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:11.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.330+0000 7f5a171ae640 1 --2- 172.21.15.161:0/1193600341 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f59f406b4d0 0x7f59f406d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f59fc035000 tx=0x7f59fc008030 comp rx=0 tx=0).stop 2024-03-28T23:32:11.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.330+0000 7f5a171ae640 1 -- 172.21.15.161:0/1193600341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a1006ff90 msgr2=0x7f5a101ee640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:11.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.330+0000 7f5a171ae640 1 --2- 172.21.15.161:0/1193600341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a1006ff90 0x7f5a101ee640 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f5a0400c7c0 tx=0x7f5a0400cc80 comp rx=0 tx=0).stop 2024-03-28T23:32:11.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.331+0000 7f5a171ae640 1 -- 172.21.15.161:0/1193600341 shutdown_connections 2024-03-28T23:32:11.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.331+0000 7f5a171ae640 1 --2- 172.21.15.161:0/1193600341 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f59f406b4d0 0x7f59f406d990 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.331+0000 7f5a171ae640 1 --2- 172.21.15.161:0/1193600341 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a10156460 0x7f5a100c1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.331+0000 7f5a171ae640 1 --2- 172.21.15.161:0/1193600341 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a10156080 0x7f5a1006cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.331+0000 7f5a171ae640 1 --2- 172.21.15.161:0/1193600341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a1006ff90 0x7f5a101ee640 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.331+0000 7f5a171ae640 1 -- 172.21.15.161:0/1193600341 >> 172.21.15.161:0/1193600341 conn(0x7f5a1006eba0 msgr2=0x7f5a100c8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:11.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.331+0000 7f5a171ae640 1 -- 172.21.15.161:0/1193600341 shutdown_connections 2024-03-28T23:32:11.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.331+0000 7f5a171ae640 1 -- 172.21.15.161:0/1193600341 wait complete. 2024-03-28T23:32:11.344 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:32:11.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.457+0000 7f7775d81640 1 Processor -- start 2024-03-28T23:32:11.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.457+0000 7f7775d81640 1 -- start start 2024-03-28T23:32:11.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.458+0000 7f7775d81640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f77700c6750 0x7f777006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:11.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.458+0000 7f7775d81640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f77700c88b0 0x7f77700c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:11.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.458+0000 7f7775d81640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7770152db0 con 0x7f77701529d0 2024-03-28T23:32:11.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.458+0000 7f7775d81640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f77700c7210 con 0x7f77700c88b0 2024-03-28T23:32:11.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.458+0000 7f7775d81640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f77700c6c50 con 0x7f77700c6750 2024-03-28T23:32:11.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.458+0000 7f776ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f77700c88b0 0x7f77700c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:11.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.458+0000 7f776ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f77700c88b0 0x7f77700c54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41392/0 (socket says 172.21.15.161:41392) 2024-03-28T23:32:11.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.458+0000 7f776ffff640 1 -- 172.21.15.161:0/3221199207 learned_addr learned my addr 172.21.15.161:0/3221199207 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:11.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.458+0000 7f776effd640 1 --2- 172.21.15.161:0/3221199207 >> v2:172.21.15.170:3300/0 conn(0x7f77700c6750 0x7f777006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:11.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.458+0000 7f776ffff640 1 -- 172.21.15.161:0/3221199207 >> v2:172.21.15.170:3300/0 conn(0x7f77700c6750 msgr2=0x7f777006bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:11.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.458+0000 7f776ffff640 1 --2- 172.21.15.161:0/3221199207 >> v2:172.21.15.170:3300/0 conn(0x7f77700c6750 0x7f777006bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.458+0000 7f776ffff640 1 -- 172.21.15.161:0/3221199207 >> v1:172.21.15.170:6789/0 conn(0x7f77701529d0 legacy=0x7f7770155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:11.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.458+0000 7f776ffff640 1 -- 172.21.15.161:0/3221199207 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77700c8d40 con 0x7f77700c88b0 2024-03-28T23:32:11.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.459+0000 7f776ffff640 1 --2- 172.21.15.161:0/3221199207 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77700c88b0 0x7f77700c54f0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f77700739b0 tx=0x7f776802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8908520ede6a8fa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:11.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.459+0000 7f776e7fc640 1 -- 172.21.15.161:0/3221199207 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776802cb20 con 0x7f77700c88b0 2024-03-28T23:32:11.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.459+0000 7f776e7fc640 1 -- 172.21.15.161:0/3221199207 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f776802ccc0 con 0x7f77700c88b0 2024-03-28T23:32:11.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.459+0000 7f776e7fc640 1 -- 172.21.15.161:0/3221199207 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7768034610 con 0x7f77700c88b0 2024-03-28T23:32:11.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.460+0000 7f7775d81640 1 -- 172.21.15.161:0/3221199207 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77700c88b0 msgr2=0x7f77700c54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:11.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.460+0000 7f7775d81640 1 --2- 172.21.15.161:0/3221199207 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77700c88b0 0x7f77700c54f0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f77700739b0 tx=0x7f776802bf80 comp rx=0 tx=0).stop 2024-03-28T23:32:11.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.460+0000 7f7775d81640 1 -- 172.21.15.161:0/3221199207 shutdown_connections 2024-03-28T23:32:11.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.460+0000 7f7775d81640 1 --2- 172.21.15.161:0/3221199207 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77700c88b0 0x7f77700c54f0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.460+0000 7f7775d81640 1 --2- 172.21.15.161:0/3221199207 >> v2:172.21.15.170:3300/0 conn(0x7f77700c6750 0x7f777006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.460+0000 7f7775d81640 1 -- 172.21.15.161:0/3221199207 >> 172.21.15.161:0/3221199207 conn(0x7f77700c7430 msgr2=0x7f77700c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:11.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.460+0000 7f7775d81640 1 -- 172.21.15.161:0/3221199207 shutdown_connections 2024-03-28T23:32:11.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.460+0000 7f7775d81640 1 -- 172.21.15.161:0/3221199207 wait complete. 2024-03-28T23:32:11.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.461+0000 7f7775d81640 1 Processor -- start 2024-03-28T23:32:11.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.461+0000 7f7775d81640 1 -- start start 2024-03-28T23:32:11.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.462+0000 7f7775d81640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77700c6750 0x7f7770154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:11.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.462+0000 7f7775d81640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77700c88b0 0x7f7770147b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:11.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.462+0000 7f7775d81640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77701529d0 0x7f77701480d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:11.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.462+0000 7f7775d81640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7770155710 con 0x7f77701529d0 2024-03-28T23:32:11.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.462+0000 7f7775d81640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7770155430 con 0x7f77700c6750 2024-03-28T23:32:11.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.462+0000 7f7775d81640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f77700aa160 con 0x7f77700c88b0 2024-03-28T23:32:11.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.462+0000 7f776ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77701529d0 0x7f77701480d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:11.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.462+0000 7f776ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77701529d0 0x7f77701480d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41406/0 (socket says 172.21.15.161:41406) 2024-03-28T23:32:11.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.462+0000 7f776ffff640 1 -- 172.21.15.161:0/699093326 learned_addr learned my addr 172.21.15.161:0/699093326 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:11.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.462+0000 7f776f7fe640 1 --2- 172.21.15.161:0/699093326 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77700c6750 0x7f7770154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:11.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.462+0000 7f776effd640 1 --2- 172.21.15.161:0/699093326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77700c88b0 0x7f7770147b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:11.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.462+0000 7f776ffff640 1 -- 172.21.15.161:0/699093326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77700c88b0 msgr2=0x7f7770147b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:11.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.462+0000 7f776ffff640 1 --2- 172.21.15.161:0/699093326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77700c88b0 0x7f7770147b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.462+0000 7f776ffff640 1 -- 172.21.15.161:0/699093326 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77700c6750 msgr2=0x7f7770154450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:11.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.462+0000 7f776ffff640 1 --2- 172.21.15.161:0/699093326 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77700c6750 0x7f7770154450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.462+0000 7f776ffff640 1 -- 172.21.15.161:0/699093326 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f777014d730 con 0x7f77701529d0 2024-03-28T23:32:11.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.462+0000 7f776effd640 1 --2- 172.21.15.161:0/699093326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77700c88b0 0x7f7770147b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:11.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.462+0000 7f776ffff640 1 --2- 172.21.15.161:0/699093326 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77701529d0 0x7f77701480d0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f7768009f40 tx=0x7f7768004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:11.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.463+0000 7f7757fff640 1 -- 172.21.15.161:0/699093326 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7768043020 con 0x7f77701529d0 2024-03-28T23:32:11.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.463+0000 7f7757fff640 1 -- 172.21.15.161:0/699093326 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f776803e450 con 0x7f77701529d0 2024-03-28T23:32:11.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.463+0000 7f7775d81640 1 -- 172.21.15.161:0/699093326 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f777014da10 con 0x7f77701529d0 2024-03-28T23:32:11.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.463+0000 7f7757fff640 1 -- 172.21.15.161:0/699093326 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7768009510 con 0x7f77701529d0 2024-03-28T23:32:11.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.463+0000 7f7775d81640 1 -- 172.21.15.161:0/699093326 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7770154990 con 0x7f77701529d0 2024-03-28T23:32:11.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.464+0000 7f7775d81640 1 -- 172.21.15.161:0/699093326 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77700ca100 con 0x7f77701529d0 2024-03-28T23:32:11.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.465+0000 7f7757fff640 1 -- 172.21.15.161:0/699093326 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7768046020 con 0x7f77701529d0 2024-03-28T23:32:11.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.465+0000 7f7757fff640 1 --2- 172.21.15.161:0/699093326 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f775c06b4d0 0x7f775c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:11.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.465+0000 7f7757fff640 1 -- 172.21.15.161:0/699093326 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7129+0+0 (secure 0 0 0) 0x7f77680acdc0 con 0x7f77701529d0 2024-03-28T23:32:11.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.465+0000 7f776f7fe640 1 --2- 172.21.15.161:0/699093326 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f775c06b4d0 0x7f775c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:11.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.466+0000 7f776f7fe640 1 --2- 172.21.15.161:0/699093326 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f775c06b4d0 0x7f775c06d990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7760001780 tx=0x7f7760009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:11.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.470+0000 7f7757fff640 1 -- 172.21.15.161:0/699093326 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f77700ca100 con 0x7f77701529d0 2024-03-28T23:32:11.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.650+0000 7f7775d81640 1 -- 172.21.15.161:0/699093326 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7770150810 con 0x7f77701529d0 2024-03-28T23:32:11.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.650+0000 7f7757fff640 1 -- 172.21.15.161:0/699093326 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v57) ==== 100+0+11 (secure 0 0 0) 0x7f7770150810 con 0x7f77701529d0 2024-03-28T23:32:11.651 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:32:11.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.653+0000 7f7775d81640 1 -- 172.21.15.161:0/699093326 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f775c06b4d0 msgr2=0x7f775c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:11.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.653+0000 7f7775d81640 1 --2- 172.21.15.161:0/699093326 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f775c06b4d0 0x7f775c06d990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7760001780 tx=0x7f7760009040 comp rx=0 tx=0).stop 2024-03-28T23:32:11.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.653+0000 7f7775d81640 1 -- 172.21.15.161:0/699093326 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77701529d0 msgr2=0x7f77701480d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:11.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.653+0000 7f7775d81640 1 --2- 172.21.15.161:0/699093326 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77701529d0 0x7f77701480d0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f7768009f40 tx=0x7f7768004a40 comp rx=0 tx=0).stop 2024-03-28T23:32:11.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.653+0000 7f7775d81640 1 -- 172.21.15.161:0/699093326 shutdown_connections 2024-03-28T23:32:11.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.653+0000 7f7775d81640 1 --2- 172.21.15.161:0/699093326 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f775c06b4d0 0x7f775c06d990 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.653+0000 7f7775d81640 1 --2- 172.21.15.161:0/699093326 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77701529d0 0x7f77701480d0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.653+0000 7f7775d81640 1 --2- 172.21.15.161:0/699093326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77700c88b0 0x7f7770147b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.653+0000 7f7775d81640 1 --2- 172.21.15.161:0/699093326 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77700c6750 0x7f7770154450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.653+0000 7f7775d81640 1 -- 172.21.15.161:0/699093326 >> 172.21.15.161:0/699093326 conn(0x7f77700c7430 msgr2=0x7f77700c8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:11.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.653+0000 7f7775d81640 1 -- 172.21.15.161:0/699093326 shutdown_connections 2024-03-28T23:32:11.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.653+0000 7f7775d81640 1 -- 172.21.15.161:0/699093326 wait complete. 2024-03-28T23:32:11.665 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-03-28T23:32:11.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.772+0000 7f4062b80640 1 Processor -- start 2024-03-28T23:32:11.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.772+0000 7f4062b80640 1 -- start start 2024-03-28T23:32:11.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.773+0000 7f4062b80640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f405c152bc0 0x7f405c0c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:11.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.773+0000 7f4062b80640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f405c1589b0 0x7f405c158e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:11.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.773+0000 7f4062b80640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f405c0c6f60 con 0x7f405c0c8a80 2024-03-28T23:32:11.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.773+0000 7f4062b80640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f405c0c6120 con 0x7f405c152bc0 2024-03-28T23:32:11.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.773+0000 7f4062b80640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f405c0c6400 con 0x7f405c1589b0 2024-03-28T23:32:11.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.773+0000 7f406091c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f405c152bc0 0x7f405c0c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:11.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.773+0000 7f405bfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f405c0c8a80 0x7f405c0c8ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:53846/0 (socket says 172.21.15.161:53846) 2024-03-28T23:32:11.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.773+0000 7f406091c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f405c152bc0 0x7f405c0c0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41422/0 (socket says 172.21.15.161:41422) 2024-03-28T23:32:11.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.773+0000 7f405bfff640 1 -- 172.21.15.161:0/1726003480 learned_addr learned my addr 172.21.15.161:0/1726003480 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:11.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.774+0000 7f406111d640 1 --2- 172.21.15.161:0/1726003480 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f405c1589b0 0x7f405c158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:11.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.774+0000 7f406091c640 1 -- 172.21.15.161:0/1726003480 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f405c1589b0 msgr2=0x7f405c158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:11.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.774+0000 7f406091c640 1 --2- 172.21.15.161:0/1726003480 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f405c1589b0 0x7f405c158e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.774+0000 7f406091c640 1 -- 172.21.15.161:0/1726003480 >> v1:172.21.15.161:6789/0 conn(0x7f405c0c8a80 legacy=0x7f405c0c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:11.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.774+0000 7f406091c640 1 -- 172.21.15.161:0/1726003480 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f405c0c7240 con 0x7f405c152bc0 2024-03-28T23:32:11.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.774+0000 7f405b7fe640 1 -- 172.21.15.161:0/1726003480 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4219201202 0 0) 0x7f405c0c6f60 con 0x7f405c0c8a80 2024-03-28T23:32:11.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.774+0000 7f406111d640 1 --2- 172.21.15.161:0/1726003480 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f405c1589b0 0x7f405c158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:11.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.774+0000 7f406091c640 1 --2- 172.21.15.161:0/1726003480 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c152bc0 0x7f405c0c0330 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f40500097a0 tx=0x7f405002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef3685360bf0eb1e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:11.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.775+0000 7f405b7fe640 1 -- 172.21.15.161:0/1726003480 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4050035020 con 0x7f405c152bc0 2024-03-28T23:32:11.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.775+0000 7f405b7fe640 1 -- 172.21.15.161:0/1726003480 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4050031830 con 0x7f405c152bc0 2024-03-28T23:32:11.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.775+0000 7f405b7fe640 1 -- 172.21.15.161:0/1726003480 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4050031c00 con 0x7f405c152bc0 2024-03-28T23:32:11.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.775+0000 7f4062b80640 1 -- 172.21.15.161:0/1726003480 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c152bc0 msgr2=0x7f405c0c0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:11.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.775+0000 7f4062b80640 1 --2- 172.21.15.161:0/1726003480 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c152bc0 0x7f405c0c0330 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f40500097a0 tx=0x7f405002b8d0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.775+0000 7f4062b80640 1 -- 172.21.15.161:0/1726003480 shutdown_connections 2024-03-28T23:32:11.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.775+0000 7f4062b80640 1 --2- 172.21.15.161:0/1726003480 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f405c1589b0 0x7f405c158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.775+0000 7f4062b80640 1 --2- 172.21.15.161:0/1726003480 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c152bc0 0x7f405c0c0330 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.775+0000 7f4062b80640 1 -- 172.21.15.161:0/1726003480 >> 172.21.15.161:0/1726003480 conn(0x7f405c0caa80 msgr2=0x7f405c0cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:11.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.775+0000 7f4062b80640 1 -- 172.21.15.161:0/1726003480 shutdown_connections 2024-03-28T23:32:11.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.775+0000 7f4062b80640 1 -- 172.21.15.161:0/1726003480 wait complete. 2024-03-28T23:32:11.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.776+0000 7f4062b80640 1 Processor -- start 2024-03-28T23:32:11.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.776+0000 7f4062b80640 1 -- start start 2024-03-28T23:32:11.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.777+0000 7f4062b80640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f405c0c8a80 0x7f405c06c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:11.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.777+0000 7f4062b80640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c152bc0 0x7f405c14fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:11.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.777+0000 7f4062b80640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f405c1589b0 0x7f405c150dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:11.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.777+0000 7f4062b80640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f405c152fa0 con 0x7f405c152bc0 2024-03-28T23:32:11.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.777+0000 7f4062b80640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f405c0c95f0 con 0x7f405c0c8a80 2024-03-28T23:32:11.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.777+0000 7f4062b80640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f405c06c840 con 0x7f405c1589b0 2024-03-28T23:32:11.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.777+0000 7f405bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c152bc0 0x7f405c14fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:11.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.777+0000 7f405bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c152bc0 0x7f405c14fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41430/0 (socket says 172.21.15.161:41430) 2024-03-28T23:32:11.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.777+0000 7f405bfff640 1 -- 172.21.15.161:0/3169080054 learned_addr learned my addr 172.21.15.161:0/3169080054 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:11.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.778+0000 7f406111d640 1 --2- 172.21.15.161:0/3169080054 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f405c1589b0 0x7f405c150dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:11.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.778+0000 7f406091c640 1 --2- 172.21.15.161:0/3169080054 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f405c0c8a80 0x7f405c06c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:11.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.778+0000 7f405bfff640 1 -- 172.21.15.161:0/3169080054 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f405c1589b0 msgr2=0x7f405c150dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:11.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.778+0000 7f405bfff640 1 --2- 172.21.15.161:0/3169080054 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f405c1589b0 0x7f405c150dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.778+0000 7f405bfff640 1 -- 172.21.15.161:0/3169080054 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f405c0c8a80 msgr2=0x7f405c06c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:11.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.778+0000 7f405bfff640 1 --2- 172.21.15.161:0/3169080054 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f405c0c8a80 0x7f405c06c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:11.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.778+0000 7f405bfff640 1 -- 172.21.15.161:0/3169080054 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f405c1578a0 con 0x7f405c152bc0 2024-03-28T23:32:11.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.778+0000 7f406111d640 1 --2- 172.21.15.161:0/3169080054 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f405c1589b0 0x7f405c150dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:11.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.778+0000 7f405bfff640 1 --2- 172.21.15.161:0/3169080054 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c152bc0 0x7f405c14fb60 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f404800c4d0 tx=0x7f404800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:11.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.778+0000 7f40597fa640 1 -- 172.21.15.161:0/3169080054 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4048018070 con 0x7f405c152bc0 2024-03-28T23:32:11.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.778+0000 7f40597fa640 1 -- 172.21.15.161:0/3169080054 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4048007500 con 0x7f405c152bc0 2024-03-28T23:32:11.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.778+0000 7f4062b80640 1 -- 172.21.15.161:0/3169080054 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f405c157b10 con 0x7f405c152bc0 2024-03-28T23:32:11.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.778+0000 7f40597fa640 1 -- 172.21.15.161:0/3169080054 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4048004a20 con 0x7f405c152bc0 2024-03-28T23:32:11.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.778+0000 7f4062b80640 1 -- 172.21.15.161:0/3169080054 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f405c1500a0 con 0x7f405c152bc0 2024-03-28T23:32:11.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.779+0000 7f4062b80640 1 -- 172.21.15.161:0/3169080054 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f405c068f90 con 0x7f405c152bc0 2024-03-28T23:32:11.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.780+0000 7f40597fa640 1 -- 172.21.15.161:0/3169080054 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f404801b070 con 0x7f405c152bc0 2024-03-28T23:32:11.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.780+0000 7f40597fa640 1 --2- 172.21.15.161:0/3169080054 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f404006b4d0 0x7f404006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:11.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.780+0000 7f40597fa640 1 -- 172.21.15.161:0/3169080054 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7129+0+0 (secure 0 0 0) 0x7f4048089a10 con 0x7f405c152bc0 2024-03-28T23:32:11.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.780+0000 7f406091c640 1 --2- 172.21.15.161:0/3169080054 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f404006b4d0 0x7f404006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:11.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.781+0000 7f406091c640 1 --2- 172.21.15.161:0/3169080054 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f404006b4d0 0x7f404006d990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4050036000 tx=0x7f4050008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:11.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:11.785+0000 7f40597fa640 1 -- 172.21.15.161:0/3169080054 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f405c068f90 con 0x7f405c152bc0 2024-03-28T23:32:12.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.085+0000 7f4062b80640 1 -- 172.21.15.161:0/3169080054 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) -- 0x7f405c04f4b0 con 0x7f405c152bc0 2024-03-28T23:32:12.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.085+0000 7f40597fa640 1 -- 172.21.15.161:0/3169080054 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v57) ==== 105+0+11 (secure 0 0 0) 0x7f4048058390 con 0x7f405c152bc0 2024-03-28T23:32:12.086 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:32:12.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.087+0000 7f4062b80640 1 -- 172.21.15.161:0/3169080054 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f404006b4d0 msgr2=0x7f404006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:12.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.087+0000 7f4062b80640 1 --2- 172.21.15.161:0/3169080054 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f404006b4d0 0x7f404006d990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4050036000 tx=0x7f4050008030 comp rx=0 tx=0).stop 2024-03-28T23:32:12.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.087+0000 7f4062b80640 1 -- 172.21.15.161:0/3169080054 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c152bc0 msgr2=0x7f405c14fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:12.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.087+0000 7f4062b80640 1 --2- 172.21.15.161:0/3169080054 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c152bc0 0x7f405c14fb60 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f404800c4d0 tx=0x7f404800c990 comp rx=0 tx=0).stop 2024-03-28T23:32:12.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.087+0000 7f4062b80640 1 -- 172.21.15.161:0/3169080054 shutdown_connections 2024-03-28T23:32:12.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.087+0000 7f4062b80640 1 --2- 172.21.15.161:0/3169080054 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f404006b4d0 0x7f404006d990 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.087+0000 7f4062b80640 1 --2- 172.21.15.161:0/3169080054 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c152bc0 0x7f405c14fb60 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.087+0000 7f4062b80640 1 --2- 172.21.15.161:0/3169080054 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f405c1589b0 0x7f405c150dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.087+0000 7f4062b80640 1 --2- 172.21.15.161:0/3169080054 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f405c0c8a80 0x7f405c06c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.087+0000 7f4062b80640 1 -- 172.21.15.161:0/3169080054 >> 172.21.15.161:0/3169080054 conn(0x7f405c0caa80 msgr2=0x7f405c14cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:12.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.087+0000 7f4062b80640 1 -- 172.21.15.161:0/3169080054 shutdown_connections 2024-03-28T23:32:12.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.087+0000 7f4062b80640 1 -- 172.21.15.161:0/3169080054 wait complete. 2024-03-28T23:32:12.099 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-03-28T23:32:12.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.203+0000 7fa9ba372640 1 Processor -- start 2024-03-28T23:32:12.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.203+0000 7fa9ba372640 1 -- start start 2024-03-28T23:32:12.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.203+0000 7fa9ba372640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa9b41529d0 0x7fa9b406bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:12.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.203+0000 7fa9ba372640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9b40c6750 0x7fa9b40c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:12.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.203+0000 7fa9ba372640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa9b4155c30 0x7fa9b40c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:12.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.203+0000 7fa9ba372640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa9b4152db0 con 0x7fa9b41529d0 2024-03-28T23:32:12.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.203+0000 7fa9ba372640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fa9b40c7210 con 0x7fa9b4155c30 2024-03-28T23:32:12.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.203+0000 7fa9ba372640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa9b40c6c50 con 0x7fa9b40c6750 2024-03-28T23:32:12.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.204+0000 7fa9b3fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa9b41529d0 0x7fa9b406bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:12.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.204+0000 7fa9b3fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa9b41529d0 0x7fa9b406bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41440/0 (socket says 172.21.15.161:41440) 2024-03-28T23:32:12.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.204+0000 7fa9b3fff640 1 -- 172.21.15.161:0/2886499107 learned_addr learned my addr 172.21.15.161:0/2886499107 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:12.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.204+0000 7fa9b37fe640 1 --2- 172.21.15.161:0/2886499107 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9b40c6750 0x7fa9b40c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:12.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.204+0000 7fa9b890f640 1 --2- 172.21.15.161:0/2886499107 >> v2:172.21.15.170:3300/0 conn(0x7fa9b4155c30 0x7fa9b40c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:12.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.204+0000 7fa9b3fff640 1 -- 172.21.15.161:0/2886499107 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9b40c6750 msgr2=0x7fa9b40c88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:12.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.204+0000 7fa9b3fff640 1 --2- 172.21.15.161:0/2886499107 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9b40c6750 0x7fa9b40c88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.204+0000 7fa9b3fff640 1 -- 172.21.15.161:0/2886499107 >> v2:172.21.15.170:3300/0 conn(0x7fa9b4155c30 msgr2=0x7fa9b40c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:12.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.204+0000 7fa9b3fff640 1 --2- 172.21.15.161:0/2886499107 >> v2:172.21.15.170:3300/0 conn(0x7fa9b4155c30 0x7fa9b40c54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.204+0000 7fa9b3fff640 1 -- 172.21.15.161:0/2886499107 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b40c6f30 con 0x7fa9b41529d0 2024-03-28T23:32:12.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.204+0000 7fa9b890f640 1 --2- 172.21.15.161:0/2886499107 >> v2:172.21.15.170:3300/0 conn(0x7fa9b4155c30 0x7fa9b40c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:12.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.205+0000 7fa9b3fff640 1 --2- 172.21.15.161:0/2886499107 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9b41529d0 0x7fa9b406bb60 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fa9b40749f0 tx=0x7fa9a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4cb9511502486bc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:12.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.205+0000 7fa9b2ffd640 1 -- 172.21.15.161:0/2886499107 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4035020 con 0x7fa9b41529d0 2024-03-28T23:32:12.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.205+0000 7fa9b2ffd640 1 -- 172.21.15.161:0/2886499107 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9a4031870 con 0x7fa9b41529d0 2024-03-28T23:32:12.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.205+0000 7fa9b2ffd640 1 -- 172.21.15.161:0/2886499107 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4031c40 con 0x7fa9b41529d0 2024-03-28T23:32:12.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.205+0000 7fa9ba372640 1 -- 172.21.15.161:0/2886499107 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9b41529d0 msgr2=0x7fa9b406bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:12.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.205+0000 7fa9ba372640 1 --2- 172.21.15.161:0/2886499107 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9b41529d0 0x7fa9b406bb60 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fa9b40749f0 tx=0x7fa9a402b910 comp rx=0 tx=0).stop 2024-03-28T23:32:12.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.205+0000 7fa9ba372640 1 -- 172.21.15.161:0/2886499107 shutdown_connections 2024-03-28T23:32:12.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.205+0000 7fa9ba372640 1 --2- 172.21.15.161:0/2886499107 >> v2:172.21.15.170:3300/0 conn(0x7fa9b4155c30 0x7fa9b40c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.205+0000 7fa9ba372640 1 --2- 172.21.15.161:0/2886499107 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9b40c6750 0x7fa9b40c88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.205+0000 7fa9ba372640 1 --2- 172.21.15.161:0/2886499107 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9b41529d0 0x7fa9b406bb60 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.205+0000 7fa9ba372640 1 -- 172.21.15.161:0/2886499107 >> 172.21.15.161:0/2886499107 conn(0x7fa9b40c7430 msgr2=0x7fa9b40c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:12.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.205+0000 7fa9ba372640 1 -- 172.21.15.161:0/2886499107 shutdown_connections 2024-03-28T23:32:12.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.205+0000 7fa9ba372640 1 -- 172.21.15.161:0/2886499107 wait complete. 2024-03-28T23:32:12.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.207+0000 7fa9ba372640 1 Processor -- start 2024-03-28T23:32:12.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.207+0000 7fa9ba372640 1 -- start start 2024-03-28T23:32:12.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.207+0000 7fa9ba372640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9b40c6750 0x7fa9b40abb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:12.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.207+0000 7fa9ba372640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9b41529d0 0x7fa9b41dff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:12.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.207+0000 7fa9ba372640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9b4155c30 0x7fa9b41e04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:12.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.207+0000 7fa9ba372640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b40aa160 con 0x7fa9b41529d0 2024-03-28T23:32:12.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.207+0000 7fa9ba372640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b4155710 con 0x7fa9b4155c30 2024-03-28T23:32:12.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.207+0000 7fa9ba372640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa9b41560e0 con 0x7fa9b40c6750 2024-03-28T23:32:12.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.207+0000 7fa9b37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9b41529d0 0x7fa9b41dff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:12.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.208+0000 7fa9b37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9b41529d0 0x7fa9b41dff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41446/0 (socket says 172.21.15.161:41446) 2024-03-28T23:32:12.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.208+0000 7fa9b37fe640 1 -- 172.21.15.161:0/487003681 learned_addr learned my addr 172.21.15.161:0/487003681 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:12.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.208+0000 7fa9b3fff640 1 --2- 172.21.15.161:0/487003681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9b40c6750 0x7fa9b40abb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:12.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.208+0000 7fa9b890f640 1 --2- 172.21.15.161:0/487003681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9b4155c30 0x7fa9b41e04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:12.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.208+0000 7fa9b37fe640 1 -- 172.21.15.161:0/487003681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9b40c6750 msgr2=0x7fa9b40abb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:12.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.208+0000 7fa9b37fe640 1 --2- 172.21.15.161:0/487003681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9b40c6750 0x7fa9b40abb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.208+0000 7fa9b37fe640 1 -- 172.21.15.161:0/487003681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9b4155c30 msgr2=0x7fa9b41e04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:12.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.208+0000 7fa9b37fe640 1 --2- 172.21.15.161:0/487003681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9b4155c30 0x7fa9b41e04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.208+0000 7fa9b37fe640 1 -- 172.21.15.161:0/487003681 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b41eef90 con 0x7fa9b41529d0 2024-03-28T23:32:12.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.208+0000 7fa9b3fff640 1 --2- 172.21.15.161:0/487003681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9b40c6750 0x7fa9b40abb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:12.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.208+0000 7fa9b890f640 1 --2- 172.21.15.161:0/487003681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9b4155c30 0x7fa9b41e04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:12.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.208+0000 7fa9b37fe640 1 --2- 172.21.15.161:0/487003681 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9b41529d0 0x7fa9b41dff60 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fa9a800e870 tx=0x7fa9a800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:12.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.208+0000 7fa9b0ff9640 1 -- 172.21.15.161:0/487003681 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8017070 con 0x7fa9b41529d0 2024-03-28T23:32:12.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.208+0000 7fa9b0ff9640 1 -- 172.21.15.161:0/487003681 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9a800ee60 con 0x7fa9b41529d0 2024-03-28T23:32:12.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.208+0000 7fa9ba372640 1 -- 172.21.15.161:0/487003681 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9b41572f0 con 0x7fa9b41529d0 2024-03-28T23:32:12.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.208+0000 7fa9b0ff9640 1 -- 172.21.15.161:0/487003681 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a800cb80 con 0x7fa9b41529d0 2024-03-28T23:32:12.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.208+0000 7fa9ba372640 1 -- 172.21.15.161:0/487003681 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9b40ac090 con 0x7fa9b41529d0 2024-03-28T23:32:12.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.209+0000 7fa9ba372640 1 -- 172.21.15.161:0/487003681 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9b404f4b0 con 0x7fa9b41529d0 2024-03-28T23:32:12.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.210+0000 7fa9b0ff9640 1 -- 172.21.15.161:0/487003681 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa9a801a070 con 0x7fa9b41529d0 2024-03-28T23:32:12.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.210+0000 7fa9b0ff9640 1 --2- 172.21.15.161:0/487003681 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa9a006b500 0x7fa9a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:12.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.210+0000 7fa9b0ff9640 1 -- 172.21.15.161:0/487003681 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7129+0+0 (secure 0 0 0) 0x7fa9a808a580 con 0x7fa9b41529d0 2024-03-28T23:32:12.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.210+0000 7fa9b3fff640 1 --2- 172.21.15.161:0/487003681 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa9a006b500 0x7fa9a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:12.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.211+0000 7fa9b3fff640 1 --2- 172.21.15.161:0/487003681 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa9a006b500 0x7fa9a006d9c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4036000 tx=0x7fa9a403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:12.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.218+0000 7fa9b0ff9640 1 -- 172.21.15.161:0/487003681 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa9a8058f00 con 0x7fa9b41529d0 2024-03-28T23:32:12.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.455+0000 7fa9ba372640 1 -- 172.21.15.161:0/487003681 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) -- 0x7fa9b41dfcc0 con 0x7fa9b41529d0 2024-03-28T23:32:12.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.455+0000 7fa9b0ff9640 1 -- 172.21.15.161:0/487003681 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v57) ==== 101+0+11 (secure 0 0 0) 0x7fa9a8058800 con 0x7fa9b41529d0 2024-03-28T23:32:12.456 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:32:12.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.458+0000 7fa9ba372640 1 -- 172.21.15.161:0/487003681 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa9a006b500 msgr2=0x7fa9a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:12.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.458+0000 7fa9ba372640 1 --2- 172.21.15.161:0/487003681 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa9a006b500 0x7fa9a006d9c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4036000 tx=0x7fa9a403c040 comp rx=0 tx=0).stop 2024-03-28T23:32:12.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.458+0000 7fa9ba372640 1 -- 172.21.15.161:0/487003681 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9b41529d0 msgr2=0x7fa9b41dff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:12.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.458+0000 7fa9ba372640 1 --2- 172.21.15.161:0/487003681 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9b41529d0 0x7fa9b41dff60 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fa9a800e870 tx=0x7fa9a800ed30 comp rx=0 tx=0).stop 2024-03-28T23:32:12.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.458+0000 7fa9ba372640 1 -- 172.21.15.161:0/487003681 shutdown_connections 2024-03-28T23:32:12.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.458+0000 7fa9ba372640 1 --2- 172.21.15.161:0/487003681 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa9a006b500 0x7fa9a006d9c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.458+0000 7fa9ba372640 1 --2- 172.21.15.161:0/487003681 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9b41529d0 0x7fa9b41dff60 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.458+0000 7fa9ba372640 1 --2- 172.21.15.161:0/487003681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9b4155c30 0x7fa9b41e04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.458+0000 7fa9ba372640 1 --2- 172.21.15.161:0/487003681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9b40c6750 0x7fa9b40abb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.458+0000 7fa9ba372640 1 -- 172.21.15.161:0/487003681 >> 172.21.15.161:0/487003681 conn(0x7fa9b40c7430 msgr2=0x7fa9b4074020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:12.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.458+0000 7fa9ba372640 1 -- 172.21.15.161:0/487003681 shutdown_connections 2024-03-28T23:32:12.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.458+0000 7fa9ba372640 1 -- 172.21.15.161:0/487003681 wait complete. 2024-03-28T23:32:12.469 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:32:12.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.577+0000 7f1a822c0640 1 Processor -- start 2024-03-28T23:32:12.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.577+0000 7f1a822c0640 1 -- start start 2024-03-28T23:32:12.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.578+0000 7f1a822c0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1a7c152890 0x7f1a7c0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:12.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.578+0000 7f1a822c0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a7c0c5b20 0x7f1a7c1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:12.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.578+0000 7f1a822c0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a7c158a30 con 0x7f1a7c0c8ed0 2024-03-28T23:32:12.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.578+0000 7f1a822c0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1a7c0c8480 con 0x7f1a7c152890 2024-03-28T23:32:12.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.578+0000 7f1a822c0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1a7c159640 con 0x7f1a7c0c5b20 2024-03-28T23:32:12.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.578+0000 7f1a7bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1a7c152890 0x7f1a7c0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:12.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.578+0000 7f1a7bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1a7c152890 0x7f1a7c0c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41460/0 (socket says 172.21.15.161:41460) 2024-03-28T23:32:12.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.578+0000 7f1a7bfff640 1 -- 172.21.15.161:0/3659369446 learned_addr learned my addr 172.21.15.161:0/3659369446 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:12.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.579+0000 7f1a8085d640 1 --2- 172.21.15.161:0/3659369446 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a7c0c5b20 0x7f1a7c1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:12.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.579+0000 7f1a7bfff640 1 -- 172.21.15.161:0/3659369446 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a7c0c5b20 msgr2=0x7f1a7c1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:12.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.579+0000 7f1a7bfff640 1 --2- 172.21.15.161:0/3659369446 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a7c0c5b20 0x7f1a7c1553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.579+0000 7f1a7bfff640 1 -- 172.21.15.161:0/3659369446 >> v1:172.21.15.161:6789/0 conn(0x7f1a7c0c8ed0 legacy=0x7f1a7c0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:12.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.579+0000 7f1a7affd640 1 -- 172.21.15.161:0/3659369446 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1102977379 0 0) 0x7f1a7c158a30 con 0x7f1a7c0c8ed0 2024-03-28T23:32:12.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.579+0000 7f1a7bfff640 1 -- 172.21.15.161:0/3659369446 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a7c158d10 con 0x7f1a7c152890 2024-03-28T23:32:12.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.580+0000 7f1a7bfff640 1 --2- 172.21.15.161:0/3659369446 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a7c152890 0x7f1a7c0c8990 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c1597c0 tx=0x7f1a6c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab0c3662315e9af8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:12.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.580+0000 7f1a7affd640 1 -- 172.21.15.161:0/3659369446 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c0317c0 con 0x7f1a7c152890 2024-03-28T23:32:12.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.580+0000 7f1a7affd640 1 -- 172.21.15.161:0/3659369446 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a6c037070 con 0x7f1a7c152890 2024-03-28T23:32:12.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.580+0000 7f1a7affd640 1 -- 172.21.15.161:0/3659369446 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c031b80 con 0x7f1a7c152890 2024-03-28T23:32:12.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.580+0000 7f1a822c0640 1 -- 172.21.15.161:0/3659369446 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a7c152890 msgr2=0x7f1a7c0c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:12.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.580+0000 7f1a822c0640 1 --2- 172.21.15.161:0/3659369446 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a7c152890 0x7f1a7c0c8990 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c1597c0 tx=0x7f1a6c02b860 comp rx=0 tx=0).stop 2024-03-28T23:32:12.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.581+0000 7f1a822c0640 1 -- 172.21.15.161:0/3659369446 shutdown_connections 2024-03-28T23:32:12.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.581+0000 7f1a822c0640 1 --2- 172.21.15.161:0/3659369446 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a7c0c5b20 0x7f1a7c1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.581+0000 7f1a822c0640 1 --2- 172.21.15.161:0/3659369446 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a7c152890 0x7f1a7c0c8990 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.581+0000 7f1a822c0640 1 -- 172.21.15.161:0/3659369446 >> 172.21.15.161:0/3659369446 conn(0x7f1a7c0ac620 msgr2=0x7f1a7c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:12.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.581+0000 7f1a822c0640 1 -- 172.21.15.161:0/3659369446 shutdown_connections 2024-03-28T23:32:12.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.581+0000 7f1a822c0640 1 -- 172.21.15.161:0/3659369446 wait complete. 2024-03-28T23:32:12.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.582+0000 7f1a822c0640 1 Processor -- start 2024-03-28T23:32:12.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.582+0000 7f1a822c0640 1 -- start start 2024-03-28T23:32:12.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.582+0000 7f1a822c0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a7c0c5b20 0x7f1a7c155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:12.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.582+0000 7f1a822c0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a7c0c8ed0 0x7f1a7c1ee360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:12.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.583+0000 7f1a822c0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a7c152890 0x7f1a7c153ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:12.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.583+0000 7f1a822c0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1a7c152c70 con 0x7f1a7c0c8ed0 2024-03-28T23:32:12.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.583+0000 7f1a822c0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1a7c1592d0 con 0x7f1a7c152890 2024-03-28T23:32:12.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.583+0000 7f1a822c0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1a7c152df0 con 0x7f1a7c0c5b20 2024-03-28T23:32:12.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.583+0000 7f1a7b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a7c0c8ed0 0x7f1a7c1ee360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:12.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.583+0000 7f1a7b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a7c0c8ed0 0x7f1a7c1ee360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41476/0 (socket says 172.21.15.161:41476) 2024-03-28T23:32:12.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.583+0000 7f1a7b7fe640 1 -- 172.21.15.161:0/2117962177 learned_addr learned my addr 172.21.15.161:0/2117962177 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:12.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.583+0000 7f1a7bfff640 1 --2- 172.21.15.161:0/2117962177 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a7c0c5b20 0x7f1a7c155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:12.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.583+0000 7f1a8085d640 1 --2- 172.21.15.161:0/2117962177 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a7c152890 0x7f1a7c153ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:12.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.583+0000 7f1a7b7fe640 1 -- 172.21.15.161:0/2117962177 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a7c0c5b20 msgr2=0x7f1a7c155ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:12.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.583+0000 7f1a7b7fe640 1 --2- 172.21.15.161:0/2117962177 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a7c0c5b20 0x7f1a7c155ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.583+0000 7f1a7b7fe640 1 -- 172.21.15.161:0/2117962177 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a7c152890 msgr2=0x7f1a7c153ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:12.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.583+0000 7f1a7b7fe640 1 --2- 172.21.15.161:0/2117962177 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a7c152890 0x7f1a7c153ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.583+0000 7f1a7b7fe640 1 -- 172.21.15.161:0/2117962177 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a7c158580 con 0x7f1a7c0c8ed0 2024-03-28T23:32:12.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.583+0000 7f1a7bfff640 1 --2- 172.21.15.161:0/2117962177 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a7c0c5b20 0x7f1a7c155ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:12.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.583+0000 7f1a7b7fe640 1 --2- 172.21.15.161:0/2117962177 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a7c0c8ed0 0x7f1a7c1ee360 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f1a7000c440 tx=0x7f1a7000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:12.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.583+0000 7f1a78ff9640 1 -- 172.21.15.161:0/2117962177 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a70017070 con 0x7f1a7c0c8ed0 2024-03-28T23:32:12.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.583+0000 7f1a78ff9640 1 -- 172.21.15.161:0/2117962177 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a700035f0 con 0x7f1a7c0c8ed0 2024-03-28T23:32:12.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.583+0000 7f1a822c0640 1 -- 172.21.15.161:0/2117962177 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a7c1eed50 con 0x7f1a7c0c8ed0 2024-03-28T23:32:12.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.583+0000 7f1a78ff9640 1 -- 172.21.15.161:0/2117962177 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a700039e0 con 0x7f1a7c0c8ed0 2024-03-28T23:32:12.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.583+0000 7f1a822c0640 1 -- 172.21.15.161:0/2117962177 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a7c0abf70 con 0x7f1a7c0c8ed0 2024-03-28T23:32:12.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.585+0000 7f1a822c0640 1 -- 172.21.15.161:0/2117962177 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a7c04f4b0 con 0x7f1a7c0c8ed0 2024-03-28T23:32:12.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.585+0000 7f1a78ff9640 1 -- 172.21.15.161:0/2117962177 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1a7001a070 con 0x7f1a7c0c8ed0 2024-03-28T23:32:12.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.586+0000 7f1a78ff9640 1 --2- 172.21.15.161:0/2117962177 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1a6806b500 0x7f1a6806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:12.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.586+0000 7f1a78ff9640 1 -- 172.21.15.161:0/2117962177 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7129+0+0 (secure 0 0 0) 0x7f1a70089930 con 0x7f1a7c0c8ed0 2024-03-28T23:32:12.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.586+0000 7f1a7bfff640 1 --2- 172.21.15.161:0/2117962177 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1a6806b500 0x7f1a6806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:12.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.586+0000 7f1a7bfff640 1 --2- 172.21.15.161:0/2117962177 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1a6806b500 0x7f1a6806d9c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c036000 tx=0x7f1a6c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:12.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.590+0000 7f1a78ff9640 1 -- 172.21.15.161:0/2117962177 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1a70057c60 con 0x7f1a7c0c8ed0 2024-03-28T23:32:12.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.839+0000 7f1a822c0640 1 -- 172.21.15.161:0/2117962177 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a7c156570 con 0x7f1a7c0c8ed0 2024-03-28T23:32:12.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.839+0000 7f1a78ff9640 1 -- 172.21.15.161:0/2117962177 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6507 (secure 0 0 0) 0x7f1a7005bd00 con 0x7f1a7c0c8ed0 2024-03-28T23:32:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:12.840 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":43,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5390,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2649192649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2649192649},{"type":"v1","addr":"172.21.15.170:6835","nonce":2649192649}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5542,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.170:6833/3666899516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3666899516},{"type":"v1","addr":"172.21.15.170:6833","nonce":3666899516}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:31:39.725625+0000","modified":"2024-03-28T23:32:09.544987+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":4,"addr":"172.21.15.161:6837/1498528126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":1498528126},{"type":"v1","addr":"172.21.15.161:6837","nonce":1498528126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6165,"qdb_cluster":[6165]},"id":3},{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:31:54.226424+0000","modified":"2024-03-28T23:31:56.347601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5817},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5817":{"gid":5817,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":8,"addr":"172.21.15.161:6836/1300583128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1300583128},{"type":"v1","addr":"172.21.15.161:6836","nonce":1300583128}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5817,"qdb_cluster":[5817]},"id":4}]} 2024-03-28T23:32:12.840 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 43 2024-03-28T23:32:12.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.841+0000 7f1a822c0640 1 -- 172.21.15.161:0/2117962177 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1a6806b500 msgr2=0x7f1a6806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:12.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.841+0000 7f1a822c0640 1 --2- 172.21.15.161:0/2117962177 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1a6806b500 0x7f1a6806d9c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c036000 tx=0x7f1a6c008030 comp rx=0 tx=0).stop 2024-03-28T23:32:12.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.841+0000 7f1a822c0640 1 -- 172.21.15.161:0/2117962177 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a7c0c8ed0 msgr2=0x7f1a7c1ee360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:12.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.841+0000 7f1a822c0640 1 --2- 172.21.15.161:0/2117962177 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a7c0c8ed0 0x7f1a7c1ee360 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f1a7000c440 tx=0x7f1a7000c900 comp rx=0 tx=0).stop 2024-03-28T23:32:12.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.841+0000 7f1a822c0640 1 -- 172.21.15.161:0/2117962177 shutdown_connections 2024-03-28T23:32:12.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.841+0000 7f1a822c0640 1 --2- 172.21.15.161:0/2117962177 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1a6806b500 0x7f1a6806d9c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.841+0000 7f1a822c0640 1 --2- 172.21.15.161:0/2117962177 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a7c0c8ed0 0x7f1a7c1ee360 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.841+0000 7f1a822c0640 1 --2- 172.21.15.161:0/2117962177 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a7c152890 0x7f1a7c153ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.841+0000 7f1a822c0640 1 --2- 172.21.15.161:0/2117962177 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a7c0c5b20 0x7f1a7c155ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.841+0000 7f1a822c0640 1 -- 172.21.15.161:0/2117962177 >> 172.21.15.161:0/2117962177 conn(0x7f1a7c0ac620 msgr2=0x7f1a7c06c900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:12.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.841+0000 7f1a822c0640 1 -- 172.21.15.161:0/2117962177 shutdown_connections 2024-03-28T23:32:12.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.841+0000 7f1a822c0640 1 -- 172.21.15.161:0/2117962177 wait complete. 2024-03-28T23:32:12.857 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:32:12.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.958+0000 7f22a5fe2640 1 Processor -- start 2024-03-28T23:32:12.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.958+0000 7f22a5fe2640 1 -- start start 2024-03-28T23:32:12.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.959+0000 7f22a5fe2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f22a0152890 0x7f22a00c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:12.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.960+0000 7f22a5fe2640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f22a00c8ed0 0x7f22a00c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:12.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.960+0000 7f22a5fe2640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22a0158a30 con 0x7f22a00c5950 2024-03-28T23:32:12.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.960+0000 7f22a5fe2640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f22a00c8480 con 0x7f22a0152890 2024-03-28T23:32:12.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.960+0000 7f22a5fe2640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f22a0159640 con 0x7f22a00c8ed0 2024-03-28T23:32:12.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.960+0000 7f229f7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f22a0152890 0x7f22a00c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:12.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.960+0000 7f229f7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f22a0152890 0x7f22a00c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41486/0 (socket says 172.21.15.161:41486) 2024-03-28T23:32:12.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.960+0000 7f229f7fe640 1 -- 172.21.15.161:0/1506885115 learned_addr learned my addr 172.21.15.161:0/1506885115 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:12.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.960+0000 7f2296ffd640 1 --2- 172.21.15.161:0/1506885115 >> v2:172.21.15.170:3300/0 conn(0x7f22a00c8ed0 0x7f22a00c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:12.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.960+0000 7f229f7fe640 1 -- 172.21.15.161:0/1506885115 >> v2:172.21.15.170:3300/0 conn(0x7f22a00c8ed0 msgr2=0x7f22a00c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:12.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.960+0000 7f229f7fe640 1 --2- 172.21.15.161:0/1506885115 >> v2:172.21.15.170:3300/0 conn(0x7f22a00c8ed0 0x7f22a00c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.960+0000 7f229f7fe640 1 -- 172.21.15.161:0/1506885115 >> v1:172.21.15.170:6789/0 conn(0x7f22a00c5950 legacy=0x7f22a01553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:12.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.960+0000 7f229f7fe640 1 -- 172.21.15.161:0/1506885115 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a0158d10 con 0x7f22a0152890 2024-03-28T23:32:12.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.961+0000 7f229f7fe640 1 --2- 172.21.15.161:0/1506885115 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22a0152890 0x7f22a00c8990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f22a00ad6e0 tx=0x7f229002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d077b769057925ad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:12.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.961+0000 7f229effd640 1 -- 172.21.15.161:0/1506885115 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229002e070 con 0x7f22a0152890 2024-03-28T23:32:12.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.961+0000 7f229effd640 1 -- 172.21.15.161:0/1506885115 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2290037070 con 0x7f22a0152890 2024-03-28T23:32:12.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.961+0000 7f229effd640 1 -- 172.21.15.161:0/1506885115 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2290031a00 con 0x7f22a0152890 2024-03-28T23:32:12.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.961+0000 7f22a5fe2640 1 -- 172.21.15.161:0/1506885115 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22a0152890 msgr2=0x7f22a00c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:12.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.961+0000 7f22a5fe2640 1 --2- 172.21.15.161:0/1506885115 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22a0152890 0x7f22a00c8990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f22a00ad6e0 tx=0x7f229002b880 comp rx=0 tx=0).stop 2024-03-28T23:32:12.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.962+0000 7f22a5fe2640 1 -- 172.21.15.161:0/1506885115 shutdown_connections 2024-03-28T23:32:12.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.962+0000 7f22a5fe2640 1 --2- 172.21.15.161:0/1506885115 >> v2:172.21.15.170:3300/0 conn(0x7f22a00c8ed0 0x7f22a00c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.962+0000 7f22a5fe2640 1 --2- 172.21.15.161:0/1506885115 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22a0152890 0x7f22a00c8990 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.962+0000 7f22a5fe2640 1 -- 172.21.15.161:0/1506885115 >> 172.21.15.161:0/1506885115 conn(0x7f22a00ac620 msgr2=0x7f22a00aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:12.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.962+0000 7f22a5fe2640 1 -- 172.21.15.161:0/1506885115 shutdown_connections 2024-03-28T23:32:12.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.962+0000 7f22a5fe2640 1 -- 172.21.15.161:0/1506885115 wait complete. 2024-03-28T23:32:12.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.963+0000 7f22a5fe2640 1 Processor -- start 2024-03-28T23:32:12.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.963+0000 7f22a5fe2640 1 -- start start 2024-03-28T23:32:12.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.965+0000 7f22a5fe2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22a00c5950 0x7f22a00bff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:12.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.965+0000 7f22a5fe2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22a00c8ed0 0x7f22a00c11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:12.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.965+0000 7f22a5fe2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f22a0152890 0x7f22a00ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:12.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.965+0000 7f22a5fe2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f22a0152c70 con 0x7f22a00c5950 2024-03-28T23:32:12.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.965+0000 7f22a5fe2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f22a01592d0 con 0x7f22a00c8ed0 2024-03-28T23:32:12.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.965+0000 7f22a5fe2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f22a0152df0 con 0x7f22a0152890 2024-03-28T23:32:12.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.965+0000 7f229f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22a00c5950 0x7f22a00bff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:12.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.965+0000 7f229f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22a00c5950 0x7f22a00bff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41500/0 (socket says 172.21.15.161:41500) 2024-03-28T23:32:12.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.965+0000 7f229f7fe640 1 -- 172.21.15.161:0/3196307241 learned_addr learned my addr 172.21.15.161:0/3196307241 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:12.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.965+0000 7f229ffff640 1 --2- 172.21.15.161:0/3196307241 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f22a0152890 0x7f22a00ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:12.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.965+0000 7f2296ffd640 1 --2- 172.21.15.161:0/3196307241 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22a00c8ed0 0x7f22a00c11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:12.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.965+0000 7f229f7fe640 1 -- 172.21.15.161:0/3196307241 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f22a0152890 msgr2=0x7f22a00ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:12.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.965+0000 7f229f7fe640 1 --2- 172.21.15.161:0/3196307241 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f22a0152890 0x7f22a00ab6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.965+0000 7f229f7fe640 1 -- 172.21.15.161:0/3196307241 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22a00c8ed0 msgr2=0x7f22a00c11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:12.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.965+0000 7f229f7fe640 1 --2- 172.21.15.161:0/3196307241 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22a00c8ed0 0x7f22a00c11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:12.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.965+0000 7f229f7fe640 1 -- 172.21.15.161:0/3196307241 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a00c1730 con 0x7f22a00c5950 2024-03-28T23:32:12.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.965+0000 7f229ffff640 1 --2- 172.21.15.161:0/3196307241 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f22a0152890 0x7f22a00ab6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:12.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.965+0000 7f229f7fe640 1 --2- 172.21.15.161:0/3196307241 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22a00c5950 0x7f22a00bff10 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f2290000c00 tx=0x7f22900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:12.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.965+0000 7f229cff9640 1 -- 172.21.15.161:0/3196307241 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2290032880 con 0x7f22a00c5950 2024-03-28T23:32:12.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.965+0000 7f229cff9640 1 -- 172.21.15.161:0/3196307241 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2290032a40 con 0x7f22a00c5950 2024-03-28T23:32:12.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.965+0000 7f22a5fe2640 1 -- 172.21.15.161:0/3196307241 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22a00c0560 con 0x7f22a00c5950 2024-03-28T23:32:12.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.965+0000 7f229cff9640 1 -- 172.21.15.161:0/3196307241 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2290032e30 con 0x7f22a00c5950 2024-03-28T23:32:12.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.965+0000 7f22a5fe2640 1 -- 172.21.15.161:0/3196307241 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22a01ee5d0 con 0x7f22a00c5950 2024-03-28T23:32:12.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.966+0000 7f22a5fe2640 1 -- 172.21.15.161:0/3196307241 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22a004f4b0 con 0x7f22a00c5950 2024-03-28T23:32:12.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.967+0000 7f229cff9640 1 -- 172.21.15.161:0/3196307241 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f22900453f0 con 0x7f22a00c5950 2024-03-28T23:32:12.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.968+0000 7f229cff9640 1 --2- 172.21.15.161:0/3196307241 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f228006b4d0 0x7f228006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:12.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.968+0000 7f229cff9640 1 -- 172.21.15.161:0/3196307241 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7129+0+0 (secure 0 0 0) 0x7f22900ac790 con 0x7f22a00c5950 2024-03-28T23:32:12.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.968+0000 7f2296ffd640 1 --2- 172.21.15.161:0/3196307241 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f228006b4d0 0x7f228006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:12.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.968+0000 7f2296ffd640 1 --2- 172.21.15.161:0/3196307241 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f228006b4d0 0x7f228006d990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f22a01ebcd0 tx=0x7f2288008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:12.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:12.972+0000 7f229cff9640 1 -- 172.21.15.161:0/3196307241 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f22a00c1730 con 0x7f22a00c5950 2024-03-28T23:32:13.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.152+0000 7f22a5fe2640 1 -- 172.21.15.161:0/3196307241 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f22a0074430 con 0x7f22a00c5950 2024-03-28T23:32:13.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.154+0000 7f229cff9640 1 -- 172.21.15.161:0/3196307241 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+20963 (secure 0 0 0) 0x7f2290040070 con 0x7f22a00c5950 2024-03-28T23:32:13.155 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:13.155 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":57,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:32:06.533687+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:31:37.940838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-28T23:31:38.692659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:31:52.774315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:31:53.201664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6835/1945122204":"2024-03-29T23:32:06.533661+0000","172.21.15.170:6834/1945122204":"2024-03-29T23:32:06.533661+0000","172.21.15.161:6837/544971001":"2024-03-29T23:32:03.518509+0000","172.21.15.161:6835/544971001":"2024-03-29T23:32:03.518509+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:32:13.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.157+0000 7f22a5fe2640 1 -- 172.21.15.161:0/3196307241 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f228006b4d0 msgr2=0x7f228006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:13.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.157+0000 7f22a5fe2640 1 --2- 172.21.15.161:0/3196307241 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f228006b4d0 0x7f228006d990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f22a01ebcd0 tx=0x7f2288008040 comp rx=0 tx=0).stop 2024-03-28T23:32:13.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.157+0000 7f22a5fe2640 1 -- 172.21.15.161:0/3196307241 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22a00c5950 msgr2=0x7f22a00bff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:13.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.157+0000 7f22a5fe2640 1 --2- 172.21.15.161:0/3196307241 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22a00c5950 0x7f22a00bff10 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f2290000c00 tx=0x7f22900046c0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.157+0000 7f22a5fe2640 1 -- 172.21.15.161:0/3196307241 shutdown_connections 2024-03-28T23:32:13.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.157+0000 7f22a5fe2640 1 --2- 172.21.15.161:0/3196307241 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f228006b4d0 0x7f228006d990 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.157+0000 7f22a5fe2640 1 --2- 172.21.15.161:0/3196307241 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f22a0152890 0x7f22a00ab6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.157+0000 7f22a5fe2640 1 --2- 172.21.15.161:0/3196307241 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22a00c8ed0 0x7f22a00c11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.157+0000 7f22a5fe2640 1 --2- 172.21.15.161:0/3196307241 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22a00c5950 0x7f22a00bff10 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.157+0000 7f22a5fe2640 1 -- 172.21.15.161:0/3196307241 >> 172.21.15.161:0/3196307241 conn(0x7f22a00ac620 msgr2=0x7f22a014fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:13.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.157+0000 7f22a5fe2640 1 -- 172.21.15.161:0/3196307241 shutdown_connections 2024-03-28T23:32:13.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.157+0000 7f22a5fe2640 1 -- 172.21.15.161:0/3196307241 wait complete. 2024-03-28T23:32:13.171 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-28T23:32:13.171 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:32:13.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.301+0000 7f56c7e6b640 1 Processor -- start 2024-03-28T23:32:13.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.301+0000 7f56c7e6b640 1 -- start start 2024-03-28T23:32:13.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.301+0000 7f56c7e6b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f56c0152890 0x7f56c00c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:13.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.301+0000 7f56c7e6b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f56c00c8ed0 0x7f56c00c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:13.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.301+0000 7f56c7e6b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56c0158a30 con 0x7f56c00c5950 2024-03-28T23:32:13.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.301+0000 7f56c7e6b640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f56c00c8480 con 0x7f56c0152890 2024-03-28T23:32:13.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.301+0000 7f56c7e6b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f56c0159640 con 0x7f56c00c8ed0 2024-03-28T23:32:13.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.302+0000 7f56c5c07640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f56c0152890 0x7f56c00c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:13.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.302+0000 7f56c5c07640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f56c0152890 0x7f56c00c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41514/0 (socket says 172.21.15.161:41514) 2024-03-28T23:32:13.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.302+0000 7f56c5c07640 1 -- 172.21.15.161:0/2784114833 learned_addr learned my addr 172.21.15.161:0/2784114833 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:13.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.302+0000 7f56c5406640 1 --2- 172.21.15.161:0/2784114833 >> v2:172.21.15.170:3300/0 conn(0x7f56c00c8ed0 0x7f56c00c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:13.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.302+0000 7f56c5c07640 1 -- 172.21.15.161:0/2784114833 >> v2:172.21.15.170:3300/0 conn(0x7f56c00c8ed0 msgr2=0x7f56c00c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:13.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.302+0000 7f56c5c07640 1 --2- 172.21.15.161:0/2784114833 >> v2:172.21.15.170:3300/0 conn(0x7f56c00c8ed0 0x7f56c00c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.302+0000 7f56c5c07640 1 -- 172.21.15.161:0/2784114833 >> v1:172.21.15.170:6789/0 conn(0x7f56c00c5950 legacy=0x7f56c01553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:13.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.302+0000 7f56c5c07640 1 -- 172.21.15.161:0/2784114833 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c0158d10 con 0x7f56c0152890 2024-03-28T23:32:13.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.303+0000 7f56c5c07640 1 --2- 172.21.15.161:0/2784114833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56c0152890 0x7f56c00c8990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f56c00ad6e0 tx=0x7f56bc02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4355b2d6d14ac12b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:13.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.303+0000 7f56c4c05640 1 -- 172.21.15.161:0/2784114833 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56bc038070 con 0x7f56c0152890 2024-03-28T23:32:13.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.303+0000 7f56c4c05640 1 -- 172.21.15.161:0/2784114833 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56bc02f050 con 0x7f56c0152890 2024-03-28T23:32:13.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.303+0000 7f56c4c05640 1 -- 172.21.15.161:0/2784114833 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56bc031970 con 0x7f56c0152890 2024-03-28T23:32:13.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.303+0000 7f56c7e6b640 1 -- 172.21.15.161:0/2784114833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56c0152890 msgr2=0x7f56c00c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:13.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.303+0000 7f56c7e6b640 1 --2- 172.21.15.161:0/2784114833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56c0152890 0x7f56c00c8990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f56c00ad6e0 tx=0x7f56bc02b7f0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.303+0000 7f56c7e6b640 1 -- 172.21.15.161:0/2784114833 shutdown_connections 2024-03-28T23:32:13.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.303+0000 7f56c7e6b640 1 --2- 172.21.15.161:0/2784114833 >> v2:172.21.15.170:3300/0 conn(0x7f56c00c8ed0 0x7f56c00c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.303+0000 7f56c7e6b640 1 --2- 172.21.15.161:0/2784114833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56c0152890 0x7f56c00c8990 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.303+0000 7f56c7e6b640 1 -- 172.21.15.161:0/2784114833 >> 172.21.15.161:0/2784114833 conn(0x7f56c00ac620 msgr2=0x7f56c00aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:13.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.303+0000 7f56c7e6b640 1 -- 172.21.15.161:0/2784114833 shutdown_connections 2024-03-28T23:32:13.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.304+0000 7f56c7e6b640 1 -- 172.21.15.161:0/2784114833 wait complete. 2024-03-28T23:32:13.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.305+0000 7f56c7e6b640 1 Processor -- start 2024-03-28T23:32:13.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.305+0000 7f56c7e6b640 1 -- start start 2024-03-28T23:32:13.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.305+0000 7f56c7e6b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56c00c5950 0x7f56c0157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:13.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.305+0000 7f56c7e6b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56c00c8ed0 0x7f56c0148830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:13.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.305+0000 7f56c7e6b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56c0152890 0x7f56c0148d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:13.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.305+0000 7f56c7e6b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f56c0152c70 con 0x7f56c00c8ed0 2024-03-28T23:32:13.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.305+0000 7f56c7e6b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f56c01592d0 con 0x7f56c00c5950 2024-03-28T23:32:13.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.305+0000 7f56c7e6b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f56c0152df0 con 0x7f56c0152890 2024-03-28T23:32:13.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.305+0000 7f56c5406640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56c00c8ed0 0x7f56c0148830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:13.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.305+0000 7f56c5406640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56c00c8ed0 0x7f56c0148830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41524/0 (socket says 172.21.15.161:41524) 2024-03-28T23:32:13.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.305+0000 7f56c5406640 1 -- 172.21.15.161:0/2787546219 learned_addr learned my addr 172.21.15.161:0/2787546219 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:13.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.305+0000 7f56c5c07640 1 --2- 172.21.15.161:0/2787546219 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56c00c5950 0x7f56c0157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:13.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.306+0000 7f56c6408640 1 --2- 172.21.15.161:0/2787546219 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56c0152890 0x7f56c0148d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:13.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.306+0000 7f56c5406640 1 -- 172.21.15.161:0/2787546219 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56c0152890 msgr2=0x7f56c0148d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:13.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.306+0000 7f56c5406640 1 --2- 172.21.15.161:0/2787546219 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56c0152890 0x7f56c0148d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.306+0000 7f56c5406640 1 -- 172.21.15.161:0/2787546219 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56c00c5950 msgr2=0x7f56c0157a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:13.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.306+0000 7f56c5406640 1 --2- 172.21.15.161:0/2787546219 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56c00c5950 0x7f56c0157a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.306+0000 7f56c5406640 1 -- 172.21.15.161:0/2787546219 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c00c1900 con 0x7f56c00c8ed0 2024-03-28T23:32:13.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.306+0000 7f56c5406640 1 --2- 172.21.15.161:0/2787546219 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56c00c8ed0 0x7f56c0148830 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f56b000c7f0 tx=0x7f56b000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:13.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.306+0000 7f56b67fc640 1 -- 172.21.15.161:0/2787546219 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b0004230 con 0x7f56c00c8ed0 2024-03-28T23:32:13.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.306+0000 7f56b67fc640 1 -- 172.21.15.161:0/2787546219 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56b00043d0 con 0x7f56c00c8ed0 2024-03-28T23:32:13.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.306+0000 7f56c7e6b640 1 -- 172.21.15.161:0/2787546219 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56c01eb190 con 0x7f56c00c8ed0 2024-03-28T23:32:13.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.306+0000 7f56b67fc640 1 -- 172.21.15.161:0/2787546219 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b0007860 con 0x7f56c00c8ed0 2024-03-28T23:32:13.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.306+0000 7f56c7e6b640 1 -- 172.21.15.161:0/2787546219 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56c01eb750 con 0x7f56c00c8ed0 2024-03-28T23:32:13.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.307+0000 7f56c7e6b640 1 -- 172.21.15.161:0/2787546219 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56c004f4b0 con 0x7f56c00c8ed0 2024-03-28T23:32:13.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.308+0000 7f56b67fc640 1 -- 172.21.15.161:0/2787546219 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f56b0007c10 con 0x7f56c00c8ed0 2024-03-28T23:32:13.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.308+0000 7f56b67fc640 1 --2- 172.21.15.161:0/2787546219 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f56a406b4d0 0x7f56a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:13.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.309+0000 7f56b67fc640 1 -- 172.21.15.161:0/2787546219 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7129+0+0 (secure 0 0 0) 0x7f56b008a8d0 con 0x7f56c00c8ed0 2024-03-28T23:32:13.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.309+0000 7f56c5c07640 1 --2- 172.21.15.161:0/2787546219 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f56a406b4d0 0x7f56a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:13.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.309+0000 7f56c5c07640 1 --2- 172.21.15.161:0/2787546219 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f56a406b4d0 0x7f56a406d990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f56bc035000 tx=0x7f56bc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:13.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.313+0000 7f56b67fc640 1 -- 172.21.15.161:0/2787546219 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f56b0015020 con 0x7f56c00c8ed0 2024-03-28T23:32:13.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.640+0000 7f56c7e6b640 1 -- 172.21.15.161:0/2787546219 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56c006c750 con 0x7f56c00c8ed0 2024-03-28T23:32:13.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.641+0000 7f56b67fc640 1 -- 172.21.15.161:0/2787546219 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6507 (secure 0 0 0) 0x7f56b0058c00 con 0x7f56c00c8ed0 2024-03-28T23:32:13.641 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:13.642 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":43,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5390,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2649192649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2649192649},{"type":"v1","addr":"172.21.15.170:6835","nonce":2649192649}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5542,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.170:6833/3666899516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3666899516},{"type":"v1","addr":"172.21.15.170:6833","nonce":3666899516}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:31:39.725625+0000","modified":"2024-03-28T23:32:09.544987+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":4,"addr":"172.21.15.161:6837/1498528126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":1498528126},{"type":"v1","addr":"172.21.15.161:6837","nonce":1498528126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6165,"qdb_cluster":[6165]},"id":3},{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:31:54.226424+0000","modified":"2024-03-28T23:31:56.347601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5817},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5817":{"gid":5817,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":8,"addr":"172.21.15.161:6836/1300583128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1300583128},{"type":"v1","addr":"172.21.15.161:6836","nonce":1300583128}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5817,"qdb_cluster":[5817]},"id":4}]} 2024-03-28T23:32:13.642 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 43 2024-03-28T23:32:13.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.643+0000 7f56c7e6b640 1 -- 172.21.15.161:0/2787546219 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f56a406b4d0 msgr2=0x7f56a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:13.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.643+0000 7f56c7e6b640 1 --2- 172.21.15.161:0/2787546219 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f56a406b4d0 0x7f56a406d990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f56bc035000 tx=0x7f56bc008030 comp rx=0 tx=0).stop 2024-03-28T23:32:13.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.643+0000 7f56c7e6b640 1 -- 172.21.15.161:0/2787546219 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56c00c8ed0 msgr2=0x7f56c0148830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:13.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.643+0000 7f56c7e6b640 1 --2- 172.21.15.161:0/2787546219 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56c00c8ed0 0x7f56c0148830 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f56b000c7f0 tx=0x7f56b000ccb0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.643+0000 7f56c7e6b640 1 -- 172.21.15.161:0/2787546219 shutdown_connections 2024-03-28T23:32:13.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.643+0000 7f56c7e6b640 1 --2- 172.21.15.161:0/2787546219 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f56a406b4d0 0x7f56a406d990 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.643+0000 7f56c7e6b640 1 --2- 172.21.15.161:0/2787546219 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56c00c8ed0 0x7f56c0148830 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.643+0000 7f56c7e6b640 1 --2- 172.21.15.161:0/2787546219 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56c0152890 0x7f56c0148d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.643+0000 7f56c7e6b640 1 --2- 172.21.15.161:0/2787546219 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56c00c5950 0x7f56c0157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.643+0000 7f56c7e6b640 1 -- 172.21.15.161:0/2787546219 >> 172.21.15.161:0/2787546219 conn(0x7f56c00ac620 msgr2=0x7f56c006c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:13.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.643+0000 7f56c7e6b640 1 -- 172.21.15.161:0/2787546219 shutdown_connections 2024-03-28T23:32:13.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.644+0000 7f56c7e6b640 1 -- 172.21.15.161:0/2787546219 wait complete. 2024-03-28T23:32:13.655 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:32:13.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.756+0000 7f11aade2640 1 Processor -- start 2024-03-28T23:32:13.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.756+0000 7f11aade2640 1 -- start start 2024-03-28T23:32:13.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.757+0000 7f11aade2640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f11a4152890 0x7f11a40c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:13.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.757+0000 7f11aade2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a40c5b20 0x7f11a41553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:13.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.757+0000 7f11aade2640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11a4158a30 con 0x7f11a40c8ed0 2024-03-28T23:32:13.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.757+0000 7f11aade2640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f11a40c8480 con 0x7f11a4152890 2024-03-28T23:32:13.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.757+0000 7f11aade2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f11a4159640 con 0x7f11a40c5b20 2024-03-28T23:32:13.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.757+0000 7f11a937f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a40c5b20 0x7f11a41553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:13.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.757+0000 7f11a8b7e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f11a4152890 0x7f11a40c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:13.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.757+0000 7f11a937f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a40c5b20 0x7f11a41553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:42566/0 (socket says 172.21.15.161:42566) 2024-03-28T23:32:13.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.757+0000 7f11a937f640 1 -- 172.21.15.161:0/1996939337 learned_addr learned my addr 172.21.15.161:0/1996939337 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:13.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.758+0000 7f11a37fe640 1 -- 172.21.15.161:0/1996939337 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1777481571 0 0) 0x7f11a4158a30 con 0x7f11a40c8ed0 2024-03-28T23:32:13.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.758+0000 7f11a937f640 1 -- 172.21.15.161:0/1996939337 >> v2:172.21.15.170:3300/0 conn(0x7f11a4152890 msgr2=0x7f11a40c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:13.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.758+0000 7f11a937f640 1 --2- 172.21.15.161:0/1996939337 >> v2:172.21.15.170:3300/0 conn(0x7f11a4152890 0x7f11a40c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.758+0000 7f11a937f640 1 -- 172.21.15.161:0/1996939337 >> v1:172.21.15.170:6789/0 conn(0x7f11a40c8ed0 legacy=0x7f11a40c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:13.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.758+0000 7f11a937f640 1 -- 172.21.15.161:0/1996939337 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11a4158d10 con 0x7f11a40c5b20 2024-03-28T23:32:13.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.758+0000 7f11a937f640 1 --2- 172.21.15.161:0/1996939337 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a40c5b20 0x7f11a41553a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f11a41597c0 tx=0x7f119c02be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f52e3b8e87daf052 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:13.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.759+0000 7f11a37fe640 1 -- 172.21.15.161:0/1996939337 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f119c02c9c0 con 0x7f11a40c5b20 2024-03-28T23:32:13.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.759+0000 7f11a37fe640 1 -- 172.21.15.161:0/1996939337 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f119c02cb60 con 0x7f11a40c5b20 2024-03-28T23:32:13.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.759+0000 7f11a37fe640 1 -- 172.21.15.161:0/1996939337 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f119c0343f0 con 0x7f11a40c5b20 2024-03-28T23:32:13.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.759+0000 7f11aade2640 1 -- 172.21.15.161:0/1996939337 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a40c5b20 msgr2=0x7f11a41553a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:13.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.759+0000 7f11aade2640 1 --2- 172.21.15.161:0/1996939337 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a40c5b20 0x7f11a41553a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f11a41597c0 tx=0x7f119c02be20 comp rx=0 tx=0).stop 2024-03-28T23:32:13.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.759+0000 7f11aade2640 1 -- 172.21.15.161:0/1996939337 shutdown_connections 2024-03-28T23:32:13.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.759+0000 7f11aade2640 1 --2- 172.21.15.161:0/1996939337 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a40c5b20 0x7f11a41553a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.759+0000 7f11aade2640 1 --2- 172.21.15.161:0/1996939337 >> v2:172.21.15.170:3300/0 conn(0x7f11a4152890 0x7f11a40c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.759+0000 7f11aade2640 1 -- 172.21.15.161:0/1996939337 >> 172.21.15.161:0/1996939337 conn(0x7f11a40ac620 msgr2=0x7f11a40aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:13.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.759+0000 7f11aade2640 1 -- 172.21.15.161:0/1996939337 shutdown_connections 2024-03-28T23:32:13.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.759+0000 7f11aade2640 1 -- 172.21.15.161:0/1996939337 wait complete. 2024-03-28T23:32:13.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.761+0000 7f11aade2640 1 Processor -- start 2024-03-28T23:32:13.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.761+0000 7f11aade2640 1 -- start start 2024-03-28T23:32:13.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.762+0000 7f11aade2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11a40c5b20 0x7f11a41eb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:13.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.762+0000 7f11aade2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11a40c8ed0 0x7f11a40c2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:13.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.762+0000 7f11aade2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a4152890 0x7f11a40c05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:13.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.762+0000 7f11aade2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f11a4152c70 con 0x7f11a40c8ed0 2024-03-28T23:32:13.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.762+0000 7f11aade2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f11a41592d0 con 0x7f11a40c5b20 2024-03-28T23:32:13.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.762+0000 7f11aade2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f11a4152df0 con 0x7f11a4152890 2024-03-28T23:32:13.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.762+0000 7f11a3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11a40c8ed0 0x7f11a40c2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:13.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.763+0000 7f11a3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11a40c8ed0 0x7f11a40c2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41536/0 (socket says 172.21.15.161:41536) 2024-03-28T23:32:13.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.763+0000 7f11a3fff640 1 -- 172.21.15.161:0/3921654103 learned_addr learned my addr 172.21.15.161:0/3921654103 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:13.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.763+0000 7f11a8b7e640 1 --2- 172.21.15.161:0/3921654103 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11a40c5b20 0x7f11a41eb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:13.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.763+0000 7f11a937f640 1 --2- 172.21.15.161:0/3921654103 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a4152890 0x7f11a40c05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:13.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.763+0000 7f11a3fff640 1 -- 172.21.15.161:0/3921654103 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a4152890 msgr2=0x7f11a40c05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:13.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.763+0000 7f11a3fff640 1 --2- 172.21.15.161:0/3921654103 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a4152890 0x7f11a40c05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.763+0000 7f11a3fff640 1 -- 172.21.15.161:0/3921654103 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11a40c5b20 msgr2=0x7f11a41eb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:13.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.763+0000 7f11a3fff640 1 --2- 172.21.15.161:0/3921654103 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11a40c5b20 0x7f11a41eb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.763+0000 7f11a3fff640 1 -- 172.21.15.161:0/3921654103 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11a40c1770 con 0x7f11a40c8ed0 2024-03-28T23:32:13.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.763+0000 7f11a8b7e640 1 --2- 172.21.15.161:0/3921654103 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11a40c5b20 0x7f11a41eb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:13.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.763+0000 7f11a937f640 1 --2- 172.21.15.161:0/3921654103 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a4152890 0x7f11a40c05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:13.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.763+0000 7f11a3fff640 1 --2- 172.21.15.161:0/3921654103 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11a40c8ed0 0x7f11a40c2630 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f119000c450 tx=0x7f119000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:13.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.763+0000 7f11a17fa640 1 -- 172.21.15.161:0/3921654103 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1190017070 con 0x7f11a40c8ed0 2024-03-28T23:32:13.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.763+0000 7f11aade2640 1 -- 172.21.15.161:0/3921654103 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11a40bf820 con 0x7f11a40c8ed0 2024-03-28T23:32:13.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.763+0000 7f11a17fa640 1 -- 172.21.15.161:0/3921654103 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1190005480 con 0x7f11a40c8ed0 2024-03-28T23:32:13.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.763+0000 7f11a17fa640 1 -- 172.21.15.161:0/3921654103 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1190003880 con 0x7f11a40c8ed0 2024-03-28T23:32:13.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.763+0000 7f11aade2640 1 -- 172.21.15.161:0/3921654103 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11a40bfde0 con 0x7f11a40c8ed0 2024-03-28T23:32:13.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.764+0000 7f11aade2640 1 -- 172.21.15.161:0/3921654103 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11a404f4b0 con 0x7f11a40c8ed0 2024-03-28T23:32:13.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.765+0000 7f11a17fa640 1 -- 172.21.15.161:0/3921654103 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f119001a070 con 0x7f11a40c8ed0 2024-03-28T23:32:13.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.765+0000 7f11a17fa640 1 --2- 172.21.15.161:0/3921654103 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f118806b500 0x7f118806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:13.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.765+0000 7f11a17fa640 1 -- 172.21.15.161:0/3921654103 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7129+0+0 (secure 0 0 0) 0x7f1190089a80 con 0x7f11a40c8ed0 2024-03-28T23:32:13.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.765+0000 7f11a8b7e640 1 --2- 172.21.15.161:0/3921654103 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f118806b500 0x7f118806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:13.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.766+0000 7f11a8b7e640 1 --2- 172.21.15.161:0/3921654103 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f118806b500 0x7f118806d9c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f1198009b90 tx=0x7f1198008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:13.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.771+0000 7f11a17fa640 1 -- 172.21.15.161:0/3921654103 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1190058380 con 0x7f11a40c8ed0 2024-03-28T23:32:13.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.993+0000 7f11aade2640 1 -- 172.21.15.161:0/3921654103 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11a406d2a0 con 0x7f11a40c8ed0 2024-03-28T23:32:13.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.993+0000 7f11a17fa640 1 -- 172.21.15.161:0/3921654103 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6507 (secure 0 0 0) 0x7f119005e080 con 0x7f11a40c8ed0 2024-03-28T23:32:13.994 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:13.994 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":43,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5390,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2649192649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2649192649},{"type":"v1","addr":"172.21.15.170:6835","nonce":2649192649}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5542,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.170:6833/3666899516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3666899516},{"type":"v1","addr":"172.21.15.170:6833","nonce":3666899516}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:31:39.725625+0000","modified":"2024-03-28T23:32:09.544987+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":4,"addr":"172.21.15.161:6837/1498528126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":1498528126},{"type":"v1","addr":"172.21.15.161:6837","nonce":1498528126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6165,"qdb_cluster":[6165]},"id":3},{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:31:54.226424+0000","modified":"2024-03-28T23:31:56.347601+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5817},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5817":{"gid":5817,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":8,"addr":"172.21.15.161:6836/1300583128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1300583128},{"type":"v1","addr":"172.21.15.161:6836","nonce":1300583128}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5817,"qdb_cluster":[5817]},"id":4}]} 2024-03-28T23:32:13.995 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 43 2024-03-28T23:32:13.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.995+0000 7f11aade2640 1 -- 172.21.15.161:0/3921654103 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f118806b500 msgr2=0x7f118806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:13.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.995+0000 7f11aade2640 1 --2- 172.21.15.161:0/3921654103 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f118806b500 0x7f118806d9c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f1198009b90 tx=0x7f1198008040 comp rx=0 tx=0).stop 2024-03-28T23:32:13.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.995+0000 7f11aade2640 1 -- 172.21.15.161:0/3921654103 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11a40c8ed0 msgr2=0x7f11a40c2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:13.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.995+0000 7f11aade2640 1 --2- 172.21.15.161:0/3921654103 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11a40c8ed0 0x7f11a40c2630 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f119000c450 tx=0x7f119000c910 comp rx=0 tx=0).stop 2024-03-28T23:32:13.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.996+0000 7f11aade2640 1 -- 172.21.15.161:0/3921654103 shutdown_connections 2024-03-28T23:32:13.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.996+0000 7f11aade2640 1 --2- 172.21.15.161:0/3921654103 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f118806b500 0x7f118806d9c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.996+0000 7f11aade2640 1 --2- 172.21.15.161:0/3921654103 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11a40c8ed0 0x7f11a40c2630 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.996+0000 7f11aade2640 1 --2- 172.21.15.161:0/3921654103 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a4152890 0x7f11a40c05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.996+0000 7f11aade2640 1 --2- 172.21.15.161:0/3921654103 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11a40c5b20 0x7f11a41eb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:13.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.996+0000 7f11aade2640 1 -- 172.21.15.161:0/3921654103 >> 172.21.15.161:0/3921654103 conn(0x7f11a40ac620 msgr2=0x7f11a406bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:13.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.996+0000 7f11aade2640 1 -- 172.21.15.161:0/3921654103 shutdown_connections 2024-03-28T23:32:13.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:13.996+0000 7f11aade2640 1 -- 172.21.15.161:0/3921654103 wait complete. 2024-03-28T23:32:14.010 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:32:14.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.113+0000 7fcd1c165640 1 Processor -- start 2024-03-28T23:32:14.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.114+0000 7fcd1c165640 1 -- start start 2024-03-28T23:32:14.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.115+0000 7fcd1c165640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fcd14158780 0x7fcd14155720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:14.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.115+0000 7fcd1c165640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcd140a6cf0 0x7fcd14155c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:14.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.115+0000 7fcd1c165640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd14158cf0 con 0x7fcd1414f3d0 2024-03-28T23:32:14.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.115+0000 7fcd1c165640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fcd141561a0 con 0x7fcd140a6cf0 2024-03-28T23:32:14.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.115+0000 7fcd1c165640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fcd14158fd0 con 0x7fcd14158780 2024-03-28T23:32:14.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.115+0000 7fcd1a702640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcd140a6cf0 0x7fcd14155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:14.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.115+0000 7fcd1a702640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcd140a6cf0 0x7fcd14155c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41538/0 (socket says 172.21.15.161:41538) 2024-03-28T23:32:14.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.115+0000 7fcd1a702640 1 -- 172.21.15.161:0/1724485296 learned_addr learned my addr 172.21.15.161:0/1724485296 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:14.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.115+0000 7fcd19700640 1 --2- 172.21.15.161:0/1724485296 >> v2:172.21.15.170:3300/0 conn(0x7fcd14158780 0x7fcd14155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:14.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.115+0000 7fcd1a702640 1 -- 172.21.15.161:0/1724485296 >> v2:172.21.15.170:3300/0 conn(0x7fcd14158780 msgr2=0x7fcd14155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:14.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.115+0000 7fcd1a702640 1 --2- 172.21.15.161:0/1724485296 >> v2:172.21.15.170:3300/0 conn(0x7fcd14158780 0x7fcd14155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.115+0000 7fcd1a702640 1 -- 172.21.15.161:0/1724485296 >> v1:172.21.15.161:6789/0 conn(0x7fcd1414f3d0 legacy=0x7fcd140c6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:14.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.115+0000 7fcd1a702640 1 -- 172.21.15.161:0/1724485296 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd141592b0 con 0x7fcd140a6cf0 2024-03-28T23:32:14.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.115+0000 7fcd18eff640 1 -- 172.21.15.161:0/1724485296 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3673869109 0 0) 0x7fcd14158cf0 con 0x7fcd1414f3d0 2024-03-28T23:32:14.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.115+0000 7fcd19700640 1 --2- 172.21.15.161:0/1724485296 >> v2:172.21.15.170:3300/0 conn(0x7fcd14158780 0x7fcd14155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:14.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.116+0000 7fcd1a702640 1 --2- 172.21.15.161:0/1724485296 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd140a6cf0 0x7fcd14155c60 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fcd140aa970 tx=0x7fcd0c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=720b332f326f95cf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:14.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.116+0000 7fcd18eff640 1 -- 172.21.15.161:0/1724485296 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c02cb20 con 0x7fcd140a6cf0 2024-03-28T23:32:14.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.116+0000 7fcd18eff640 1 -- 172.21.15.161:0/1724485296 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd0c02ccc0 con 0x7fcd140a6cf0 2024-03-28T23:32:14.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.116+0000 7fcd18eff640 1 -- 172.21.15.161:0/1724485296 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c034610 con 0x7fcd140a6cf0 2024-03-28T23:32:14.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.117+0000 7fcd1c165640 1 -- 172.21.15.161:0/1724485296 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd140a6cf0 msgr2=0x7fcd14155c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:14.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.117+0000 7fcd1c165640 1 --2- 172.21.15.161:0/1724485296 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd140a6cf0 0x7fcd14155c60 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fcd140aa970 tx=0x7fcd0c02bf80 comp rx=0 tx=0).stop 2024-03-28T23:32:14.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.117+0000 7fcd1c165640 1 -- 172.21.15.161:0/1724485296 shutdown_connections 2024-03-28T23:32:14.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.117+0000 7fcd1c165640 1 --2- 172.21.15.161:0/1724485296 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd140a6cf0 0x7fcd14155c60 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.117+0000 7fcd1c165640 1 --2- 172.21.15.161:0/1724485296 >> v2:172.21.15.170:3300/0 conn(0x7fcd14158780 0x7fcd14155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.117+0000 7fcd1c165640 1 -- 172.21.15.161:0/1724485296 >> 172.21.15.161:0/1724485296 conn(0x7fcd140c4330 msgr2=0x7fcd140c4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:14.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.117+0000 7fcd1c165640 1 -- 172.21.15.161:0/1724485296 shutdown_connections 2024-03-28T23:32:14.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.117+0000 7fcd1c165640 1 -- 172.21.15.161:0/1724485296 wait complete. 2024-03-28T23:32:14.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.118+0000 7fcd1c165640 1 Processor -- start 2024-03-28T23:32:14.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.118+0000 7fcd1c165640 1 -- start start 2024-03-28T23:32:14.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.119+0000 7fcd1c165640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd140a6cf0 0x7fcd1414dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:14.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.119+0000 7fcd1c165640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd1414f3d0 0x7fcd140715b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:14.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.119+0000 7fcd1c165640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd14158780 0x7fcd14072660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:14.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.119+0000 7fcd1c165640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcd1414f7b0 con 0x7fcd140a6cf0 2024-03-28T23:32:14.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.119+0000 7fcd1c165640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fcd14155190 con 0x7fcd14158780 2024-03-28T23:32:14.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.119+0000 7fcd1c165640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fcd140c6e50 con 0x7fcd1414f3d0 2024-03-28T23:32:14.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.119+0000 7fcd19f01640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd140a6cf0 0x7fcd1414dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:14.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.119+0000 7fcd19f01640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd140a6cf0 0x7fcd1414dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41548/0 (socket says 172.21.15.161:41548) 2024-03-28T23:32:14.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.119+0000 7fcd19f01640 1 -- 172.21.15.161:0/892150593 learned_addr learned my addr 172.21.15.161:0/892150593 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:14.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.119+0000 7fcd1a702640 1 --2- 172.21.15.161:0/892150593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd14158780 0x7fcd14072660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:14.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.119+0000 7fcd19700640 1 --2- 172.21.15.161:0/892150593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd1414f3d0 0x7fcd140715b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:14.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.119+0000 7fcd19f01640 1 -- 172.21.15.161:0/892150593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd1414f3d0 msgr2=0x7fcd140715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:14.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.119+0000 7fcd19f01640 1 --2- 172.21.15.161:0/892150593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd1414f3d0 0x7fcd140715b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.119+0000 7fcd19f01640 1 -- 172.21.15.161:0/892150593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd14158780 msgr2=0x7fcd14072660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:14.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.119+0000 7fcd19f01640 1 --2- 172.21.15.161:0/892150593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd14158780 0x7fcd14072660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.119+0000 7fcd19f01640 1 -- 172.21.15.161:0/892150593 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd14147c10 con 0x7fcd140a6cf0 2024-03-28T23:32:14.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.119+0000 7fcd1a702640 1 --2- 172.21.15.161:0/892150593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd14158780 0x7fcd14072660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:14.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.119+0000 7fcd19700640 1 --2- 172.21.15.161:0/892150593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd1414f3d0 0x7fcd140715b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:14.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.120+0000 7fcd19f01640 1 --2- 172.21.15.161:0/892150593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd140a6cf0 0x7fcd1414dfc0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fcd1000c4e0 tx=0x7fcd1000c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:14.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.120+0000 7fcd0a7fc640 1 -- 172.21.15.161:0/892150593 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd10017070 con 0x7fcd140a6cf0 2024-03-28T23:32:14.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.120+0000 7fcd0a7fc640 1 -- 172.21.15.161:0/892150593 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd100035f0 con 0x7fcd140a6cf0 2024-03-28T23:32:14.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.120+0000 7fcd1c165640 1 -- 172.21.15.161:0/892150593 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd141eb630 con 0x7fcd140a6cf0 2024-03-28T23:32:14.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.120+0000 7fcd0a7fc640 1 -- 172.21.15.161:0/892150593 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd100039e0 con 0x7fcd140a6cf0 2024-03-28T23:32:14.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.120+0000 7fcd1c165640 1 -- 172.21.15.161:0/892150593 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd141eb910 con 0x7fcd140a6cf0 2024-03-28T23:32:14.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.121+0000 7fcd1c165640 1 -- 172.21.15.161:0/892150593 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd1404f4b0 con 0x7fcd140a6cf0 2024-03-28T23:32:14.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.121+0000 7fcd0a7fc640 1 -- 172.21.15.161:0/892150593 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcd1001a070 con 0x7fcd140a6cf0 2024-03-28T23:32:14.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.122+0000 7fcd0a7fc640 1 --2- 172.21.15.161:0/892150593 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fccf806b4d0 0x7fccf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:14.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.122+0000 7fcd0a7fc640 1 -- 172.21.15.161:0/892150593 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7129+0+0 (secure 0 0 0) 0x7fcd100899f0 con 0x7fcd140a6cf0 2024-03-28T23:32:14.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.122+0000 7fcd19700640 1 --2- 172.21.15.161:0/892150593 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fccf806b4d0 0x7fccf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:14.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.123+0000 7fcd19700640 1 --2- 172.21.15.161:0/892150593 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fccf806b4d0 0x7fccf806d990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcd1414ad50 tx=0x7fcd04008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:14.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.126+0000 7fcd0a7fc640 1 -- 172.21.15.161:0/892150593 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcd10057d20 con 0x7fcd140a6cf0 2024-03-28T23:32:14.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.381+0000 7fcd1c165640 1 -- 172.21.15.161:0/892150593 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcd140c3310 con 0x7fcd140a6cf0 2024-03-28T23:32:14.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.384+0000 7fcd0a7fc640 1 -- 172.21.15.161:0/892150593 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+20963 (secure 0 0 0) 0x7fcd1005bdc0 con 0x7fcd140a6cf0 2024-03-28T23:32:14.384 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:14.385 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":57,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:32:06.533687+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:31:37.940838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-28T23:31:38.692659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:31:52.774315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:31:53.201664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6835/1945122204":"2024-03-29T23:32:06.533661+0000","172.21.15.170:6834/1945122204":"2024-03-29T23:32:06.533661+0000","172.21.15.161:6837/544971001":"2024-03-29T23:32:03.518509+0000","172.21.15.161:6835/544971001":"2024-03-29T23:32:03.518509+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:32:14.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.386+0000 7fcd1c165640 1 -- 172.21.15.161:0/892150593 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fccf806b4d0 msgr2=0x7fccf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:14.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.386+0000 7fcd1c165640 1 --2- 172.21.15.161:0/892150593 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fccf806b4d0 0x7fccf806d990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcd1414ad50 tx=0x7fcd04008040 comp rx=0 tx=0).stop 2024-03-28T23:32:14.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.386+0000 7fcd1c165640 1 -- 172.21.15.161:0/892150593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd140a6cf0 msgr2=0x7fcd1414dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:14.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.386+0000 7fcd1c165640 1 --2- 172.21.15.161:0/892150593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd140a6cf0 0x7fcd1414dfc0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fcd1000c4e0 tx=0x7fcd1000c9a0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.386+0000 7fcd1c165640 1 -- 172.21.15.161:0/892150593 shutdown_connections 2024-03-28T23:32:14.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.386+0000 7fcd1c165640 1 --2- 172.21.15.161:0/892150593 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fccf806b4d0 0x7fccf806d990 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.386+0000 7fcd1c165640 1 --2- 172.21.15.161:0/892150593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd14158780 0x7fcd14072660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.386+0000 7fcd1c165640 1 --2- 172.21.15.161:0/892150593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd1414f3d0 0x7fcd140715b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.386+0000 7fcd1c165640 1 --2- 172.21.15.161:0/892150593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd140a6cf0 0x7fcd1414dfc0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.386+0000 7fcd1c165640 1 -- 172.21.15.161:0/892150593 >> 172.21.15.161:0/892150593 conn(0x7fcd140c4330 msgr2=0x7fcd140c5420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:14.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.386+0000 7fcd1c165640 1 -- 172.21.15.161:0/892150593 shutdown_connections 2024-03-28T23:32:14.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.386+0000 7fcd1c165640 1 -- 172.21.15.161:0/892150593 wait complete. 2024-03-28T23:32:14.398 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-28T23:32:14.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.503+0000 7efd76cc9640 1 Processor -- start 2024-03-28T23:32:14.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.504+0000 7efd76cc9640 1 -- start start 2024-03-28T23:32:14.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.505+0000 7efd76cc9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7014f620 0x7efd7006c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:14.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.505+0000 7efd76cc9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7efd7006c6a0 0x7efd7006d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:14.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.505+0000 7efd76cc9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd700c32a0 con 0x7efd7006da50 2024-03-28T23:32:14.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.505+0000 7efd76cc9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7efd700c1bf0 con 0x7efd7006c6a0 2024-03-28T23:32:14.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.505+0000 7efd76cc9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7efd700c3b40 con 0x7efd7014f620 2024-03-28T23:32:14.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.505+0000 7efd74a65640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7014f620 0x7efd7006c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:14.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.505+0000 7efd6ffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7efd7006c6a0 0x7efd7006d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:14.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.505+0000 7efd74a65640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7014f620 0x7efd7006c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:42608/0 (socket says 172.21.15.161:42608) 2024-03-28T23:32:14.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.505+0000 7efd6ffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7efd7006c6a0 0x7efd7006d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:54870/0 (socket says 172.21.15.161:54870) 2024-03-28T23:32:14.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.505+0000 7efd74a65640 1 -- 172.21.15.161:0/3782755725 learned_addr learned my addr 172.21.15.161:0/3782755725 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:14.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.505+0000 7efd74a65640 1 -- 172.21.15.161:0/3782755725 >> v2:172.21.15.170:3300/0 conn(0x7efd7006c6a0 msgr2=0x7efd7006d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:14.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.505+0000 7efd74a65640 1 --2- 172.21.15.161:0/3782755725 >> v2:172.21.15.170:3300/0 conn(0x7efd7006c6a0 0x7efd7006d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.505+0000 7efd74a65640 1 -- 172.21.15.161:0/3782755725 >> v1:172.21.15.170:6789/0 conn(0x7efd7006da50 legacy=0x7efd70152400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:14.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.506+0000 7efd74a65640 1 -- 172.21.15.161:0/3782755725 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd700c3580 con 0x7efd7014f620 2024-03-28T23:32:14.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.506+0000 7efd74a65640 1 --2- 172.21.15.161:0/3782755725 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7014f620 0x7efd7006c160 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efd64009870 tx=0x7efd6402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76473dfd255223b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:14.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.507+0000 7efd6f7fe640 1 -- 172.21.15.161:0/3782755725 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd64035020 con 0x7efd7014f620 2024-03-28T23:32:14.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.507+0000 7efd6f7fe640 1 -- 172.21.15.161:0/3782755725 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd64031640 con 0x7efd7014f620 2024-03-28T23:32:14.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.507+0000 7efd6f7fe640 1 -- 172.21.15.161:0/3782755725 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6402d040 con 0x7efd7014f620 2024-03-28T23:32:14.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.507+0000 7efd76cc9640 1 -- 172.21.15.161:0/3782755725 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7014f620 msgr2=0x7efd7006c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:14.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.507+0000 7efd76cc9640 1 --2- 172.21.15.161:0/3782755725 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7014f620 0x7efd7006c160 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efd64009870 tx=0x7efd6402b8f0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.507+0000 7efd76cc9640 1 -- 172.21.15.161:0/3782755725 shutdown_connections 2024-03-28T23:32:14.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.507+0000 7efd76cc9640 1 --2- 172.21.15.161:0/3782755725 >> v2:172.21.15.170:3300/0 conn(0x7efd7006c6a0 0x7efd7006d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.507+0000 7efd76cc9640 1 --2- 172.21.15.161:0/3782755725 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7014f620 0x7efd7006c160 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.507+0000 7efd76cc9640 1 -- 172.21.15.161:0/3782755725 >> 172.21.15.161:0/3782755725 conn(0x7efd700c3cf0 msgr2=0x7efd700c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:14.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.507+0000 7efd76cc9640 1 -- 172.21.15.161:0/3782755725 shutdown_connections 2024-03-28T23:32:14.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.507+0000 7efd76cc9640 1 -- 172.21.15.161:0/3782755725 wait complete. 2024-03-28T23:32:14.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.509+0000 7efd76cc9640 1 Processor -- start 2024-03-28T23:32:14.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.509+0000 7efd76cc9640 1 -- start start 2024-03-28T23:32:14.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.510+0000 7efd76cc9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7006c6a0 0x7efd700bc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:14.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.510+0000 7efd76cc9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd7006da50 0x7efd701e85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:14.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.510+0000 7efd76cc9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd7014f620 0x7efd700c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:14.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.510+0000 7efd76cc9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efd7014fa00 con 0x7efd7006da50 2024-03-28T23:32:14.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.510+0000 7efd76cc9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7efd7006dee0 con 0x7efd7014f620 2024-03-28T23:32:14.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.510+0000 7efd76cc9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7efd7006e1e0 con 0x7efd7006c6a0 2024-03-28T23:32:14.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.511+0000 7efd6ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd7006da50 0x7efd701e85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:14.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.511+0000 7efd6ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd7006da50 0x7efd701e85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41550/0 (socket says 172.21.15.161:41550) 2024-03-28T23:32:14.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.511+0000 7efd6ffff640 1 -- 172.21.15.161:0/2232634971 learned_addr learned my addr 172.21.15.161:0/2232634971 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:14.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.511+0000 7efd74a65640 1 --2- 172.21.15.161:0/2232634971 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7006c6a0 0x7efd700bc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:14.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.511+0000 7efd75266640 1 --2- 172.21.15.161:0/2232634971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd7014f620 0x7efd700c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:14.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.511+0000 7efd6ffff640 1 -- 172.21.15.161:0/2232634971 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7006c6a0 msgr2=0x7efd700bc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:14.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.511+0000 7efd6ffff640 1 --2- 172.21.15.161:0/2232634971 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7006c6a0 0x7efd700bc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.511+0000 7efd6ffff640 1 -- 172.21.15.161:0/2232634971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd7014f620 msgr2=0x7efd700c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:14.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.511+0000 7efd6ffff640 1 --2- 172.21.15.161:0/2232634971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd7014f620 0x7efd700c66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.511+0000 7efd6ffff640 1 -- 172.21.15.161:0/2232634971 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd701e7b00 con 0x7efd7006da50 2024-03-28T23:32:14.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.511+0000 7efd75266640 1 --2- 172.21.15.161:0/2232634971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd7014f620 0x7efd700c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:14.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.511+0000 7efd6ffff640 1 --2- 172.21.15.161:0/2232634971 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd7006da50 0x7efd701e85c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7efd5c00e9e0 tx=0x7efd5c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:14.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.511+0000 7efd6d7fa640 1 -- 172.21.15.161:0/2232634971 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5c017070 con 0x7efd7006da50 2024-03-28T23:32:14.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.511+0000 7efd6d7fa640 1 -- 172.21.15.161:0/2232634971 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd5c00c970 con 0x7efd7006da50 2024-03-28T23:32:14.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.511+0000 7efd76cc9640 1 -- 172.21.15.161:0/2232634971 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd701eb210 con 0x7efd7006da50 2024-03-28T23:32:14.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.511+0000 7efd6d7fa640 1 -- 172.21.15.161:0/2232634971 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5c00cd60 con 0x7efd7006da50 2024-03-28T23:32:14.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.511+0000 7efd76cc9640 1 -- 172.21.15.161:0/2232634971 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd700c6c30 con 0x7efd7006da50 2024-03-28T23:32:14.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.512+0000 7efd76cc9640 1 -- 172.21.15.161:0/2232634971 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd7004f4b0 con 0x7efd7006da50 2024-03-28T23:32:14.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.513+0000 7efd6d7fa640 1 -- 172.21.15.161:0/2232634971 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efd5c01a070 con 0x7efd7006da50 2024-03-28T23:32:14.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.513+0000 7efd6d7fa640 1 --2- 172.21.15.161:0/2232634971 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efd5406b4d0 0x7efd5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:14.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.514+0000 7efd74a65640 1 --2- 172.21.15.161:0/2232634971 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efd5406b4d0 0x7efd5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:14.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.514+0000 7efd6d7fa640 1 -- 172.21.15.161:0/2232634971 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7129+0+0 (secure 0 0 0) 0x7efd5c08a640 con 0x7efd7006da50 2024-03-28T23:32:14.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.514+0000 7efd74a65640 1 --2- 172.21.15.161:0/2232634971 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efd5406b4d0 0x7efd5406d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7efd64036000 tx=0x7efd6403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:14.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.518+0000 7efd6d7fa640 1 -- 172.21.15.161:0/2232634971 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efd5c058970 con 0x7efd7006da50 2024-03-28T23:32:14.531 INFO:tasks.ceph.mds.a.smithi161.stderr:2024-03-28T23:32:14.531+0000 7fa6ca47d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:32:14.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.766+0000 7efd76cc9640 1 -- 172.21.15.161:0/2232634971 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7efd70071fd0 con 0x7efd7006da50 2024-03-28T23:32:14.767 INFO:tasks.ceph.mon.a.smithi161.stderr:2024-03-28T23:32:14.766+0000 7fdc7b85b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T23:32:14.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.768+0000 7efd6d7fa640 1 -- 172.21.15.161:0/2232634971 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v44) ==== 161+0+0 (secure 0 0 0) 0x7efd5c05ca10 con 0x7efd7006da50 2024-03-28T23:32:14.768 INFO:teuthology.orchestra.run.smithi161.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-28T23:32:14.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.771+0000 7efd76cc9640 1 -- 172.21.15.161:0/2232634971 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efd5406b4d0 msgr2=0x7efd5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:14.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.771+0000 7efd76cc9640 1 --2- 172.21.15.161:0/2232634971 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efd5406b4d0 0x7efd5406d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7efd64036000 tx=0x7efd6403c040 comp rx=0 tx=0).stop 2024-03-28T23:32:14.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.771+0000 7efd76cc9640 1 -- 172.21.15.161:0/2232634971 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd7006da50 msgr2=0x7efd701e85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:14.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.771+0000 7efd76cc9640 1 --2- 172.21.15.161:0/2232634971 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd7006da50 0x7efd701e85c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7efd5c00e9e0 tx=0x7efd5c00eea0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.771+0000 7efd76cc9640 1 -- 172.21.15.161:0/2232634971 shutdown_connections 2024-03-28T23:32:14.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.771+0000 7efd76cc9640 1 --2- 172.21.15.161:0/2232634971 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efd5406b4d0 0x7efd5406d990 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.771+0000 7efd76cc9640 1 --2- 172.21.15.161:0/2232634971 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd7006da50 0x7efd701e85c0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.771+0000 7efd76cc9640 1 --2- 172.21.15.161:0/2232634971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd7014f620 0x7efd700c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.772+0000 7efd76cc9640 1 --2- 172.21.15.161:0/2232634971 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7006c6a0 0x7efd700bc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.772+0000 7efd76cc9640 1 -- 172.21.15.161:0/2232634971 >> 172.21.15.161:0/2232634971 conn(0x7efd700c3cf0 msgr2=0x7efd7014e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:14.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.772+0000 7efd76cc9640 1 -- 172.21.15.161:0/2232634971 shutdown_connections 2024-03-28T23:32:14.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.772+0000 7efd76cc9640 1 -- 172.21.15.161:0/2232634971 wait complete. 2024-03-28T23:32:14.773 INFO:tasks.ceph.mds.a.smithi161.stderr: -40> 2024-03-28T23:32:14.531+0000 7fa6ca47d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:32:14.784 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-28T23:32:14.796 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-28T23:32:14.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.890+0000 7f92656eb640 1 Processor -- start 2024-03-28T23:32:14.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.890+0000 7f92656eb640 1 -- start start 2024-03-28T23:32:14.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.891+0000 7f92656eb640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9260152890 0x7f92600c04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:14.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.891+0000 7f92656eb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f92600c13c0 0x7f926006b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:14.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.891+0000 7f92656eb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9260155670 con 0x7f92600670d0 2024-03-28T23:32:14.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.891+0000 7f92656eb640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f926006bda0 con 0x7f92600c13c0 2024-03-28T23:32:14.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.891+0000 7f92656eb640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f9260155950 con 0x7f9260152890 2024-03-28T23:32:14.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.891+0000 7f925f7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f92600c13c0 0x7f926006b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:14.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.891+0000 7f925e7fc640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f92600670d0 0x7f926006b150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:53876/0 (socket says 172.21.15.161:53876) 2024-03-28T23:32:14.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.891+0000 7f925f7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f92600c13c0 0x7f926006b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41576/0 (socket says 172.21.15.161:41576) 2024-03-28T23:32:14.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.891+0000 7f925f7fe640 1 -- 172.21.15.161:0/4255561006 learned_addr learned my addr 172.21.15.161:0/4255561006 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:14.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.891+0000 7f925effd640 1 --2- 172.21.15.161:0/4255561006 >> v2:172.21.15.170:3300/0 conn(0x7f9260152890 0x7f92600c04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:14.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.891+0000 7f925f7fe640 1 -- 172.21.15.161:0/4255561006 >> v2:172.21.15.170:3300/0 conn(0x7f9260152890 msgr2=0x7f92600c04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:14.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.891+0000 7f925f7fe640 1 --2- 172.21.15.161:0/4255561006 >> v2:172.21.15.170:3300/0 conn(0x7f9260152890 0x7f92600c04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.891+0000 7f925f7fe640 1 -- 172.21.15.161:0/4255561006 >> v1:172.21.15.161:6789/0 conn(0x7f92600670d0 legacy=0x7f926006b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:14.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.891+0000 7f925f7fe640 1 -- 172.21.15.161:0/4255561006 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9260155c30 con 0x7f92600c13c0 2024-03-28T23:32:14.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.891+0000 7f925dffb640 1 -- 172.21.15.161:0/4255561006 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1791497030 0 0) 0x7f9260155670 con 0x7f92600670d0 2024-03-28T23:32:14.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.892+0000 7f925f7fe640 1 --2- 172.21.15.161:0/4255561006 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92600c13c0 0x7f926006b860 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f92600706d0 tx=0x7f925002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9efc5ae9177a329f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:14.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.892+0000 7f925dffb640 1 -- 172.21.15.161:0/4255561006 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9250035020 con 0x7f92600c13c0 2024-03-28T23:32:14.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.892+0000 7f925dffb640 1 -- 172.21.15.161:0/4255561006 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9250031870 con 0x7f92600c13c0 2024-03-28T23:32:14.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.892+0000 7f925dffb640 1 -- 172.21.15.161:0/4255561006 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9250031c40 con 0x7f92600c13c0 2024-03-28T23:32:14.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.893+0000 7f92656eb640 1 -- 172.21.15.161:0/4255561006 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92600c13c0 msgr2=0x7f926006b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:14.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.893+0000 7f92656eb640 1 --2- 172.21.15.161:0/4255561006 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92600c13c0 0x7f926006b860 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f92600706d0 tx=0x7f925002b910 comp rx=0 tx=0).stop 2024-03-28T23:32:14.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.893+0000 7f92656eb640 1 -- 172.21.15.161:0/4255561006 shutdown_connections 2024-03-28T23:32:14.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.893+0000 7f92656eb640 1 --2- 172.21.15.161:0/4255561006 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92600c13c0 0x7f926006b860 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.893+0000 7f92656eb640 1 --2- 172.21.15.161:0/4255561006 >> v2:172.21.15.170:3300/0 conn(0x7f9260152890 0x7f92600c04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.893+0000 7f92656eb640 1 -- 172.21.15.161:0/4255561006 >> 172.21.15.161:0/4255561006 conn(0x7f92600c3b70 msgr2=0x7f92600c3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:14.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.893+0000 7f92656eb640 1 -- 172.21.15.161:0/4255561006 shutdown_connections 2024-03-28T23:32:14.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.893+0000 7f92656eb640 1 -- 172.21.15.161:0/4255561006 wait complete. 2024-03-28T23:32:14.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.894+0000 7f92656eb640 1 Processor -- start 2024-03-28T23:32:14.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.894+0000 7f92656eb640 1 -- start start 2024-03-28T23:32:14.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.895+0000 7f92656eb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92600670d0 0x7f9260154d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:14.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.895+0000 7f92656eb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92600c13c0 0x7f92600acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:14.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.895+0000 7f92656eb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9260152890 0x7f92600c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:14.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.895+0000 7f92656eb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9260152c70 con 0x7f92600670d0 2024-03-28T23:32:14.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.895+0000 7f92656eb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9260155320 con 0x7f92600c13c0 2024-03-28T23:32:14.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.895+0000 7f92656eb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f92600c1b50 con 0x7f9260152890 2024-03-28T23:32:14.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.895+0000 7f925effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92600670d0 0x7f9260154d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:14.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.895+0000 7f925effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92600670d0 0x7f9260154d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41580/0 (socket says 172.21.15.161:41580) 2024-03-28T23:32:14.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.895+0000 7f925effd640 1 -- 172.21.15.161:0/30939122 learned_addr learned my addr 172.21.15.161:0/30939122 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:14.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.895+0000 7f925e7fc640 1 --2- 172.21.15.161:0/30939122 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92600c13c0 0x7f92600acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:14.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.895+0000 7f925f7fe640 1 --2- 172.21.15.161:0/30939122 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9260152890 0x7f92600c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:14.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.895+0000 7f925effd640 1 -- 172.21.15.161:0/30939122 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9260152890 msgr2=0x7f92600c5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:14.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.895+0000 7f925effd640 1 --2- 172.21.15.161:0/30939122 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9260152890 0x7f92600c5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.895+0000 7f925effd640 1 -- 172.21.15.161:0/30939122 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92600c13c0 msgr2=0x7f92600acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:14.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.895+0000 7f925effd640 1 --2- 172.21.15.161:0/30939122 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92600c13c0 0x7f92600acb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:14.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.895+0000 7f925effd640 1 -- 172.21.15.161:0/30939122 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92600ad070 con 0x7f92600670d0 2024-03-28T23:32:14.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.895+0000 7f925e7fc640 1 --2- 172.21.15.161:0/30939122 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92600c13c0 0x7f92600acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:14.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.895+0000 7f925effd640 1 --2- 172.21.15.161:0/30939122 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92600670d0 0x7f9260154d80 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f9258007e70 tx=0x7f925800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:14.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.895+0000 7f92477fe640 1 -- 172.21.15.161:0/30939122 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9258014420 con 0x7f92600670d0 2024-03-28T23:32:14.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.896+0000 7f92477fe640 1 -- 172.21.15.161:0/30939122 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9258019070 con 0x7f92600670d0 2024-03-28T23:32:14.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.896+0000 7f92656eb640 1 -- 172.21.15.161:0/30939122 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92601ee800 con 0x7f92600670d0 2024-03-28T23:32:14.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.896+0000 7f92477fe640 1 -- 172.21.15.161:0/30939122 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9258014800 con 0x7f92600670d0 2024-03-28T23:32:14.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.896+0000 7f92656eb640 1 -- 172.21.15.161:0/30939122 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92601542a0 con 0x7f92600670d0 2024-03-28T23:32:14.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.897+0000 7f92656eb640 1 -- 172.21.15.161:0/30939122 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f926004f4b0 con 0x7f92600670d0 2024-03-28T23:32:14.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.897+0000 7f92477fe640 1 -- 172.21.15.161:0/30939122 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f92580149a0 con 0x7f92600670d0 2024-03-28T23:32:14.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.897+0000 7f92477fe640 1 --2- 172.21.15.161:0/30939122 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f924c06b4d0 0x7f924c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:14.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.897+0000 7f92477fe640 1 -- 172.21.15.161:0/30939122 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 7215+0+0 (secure 0 0 0) 0x7f925808af50 con 0x7f92600670d0 2024-03-28T23:32:14.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.897+0000 7f925e7fc640 1 --2- 172.21.15.161:0/30939122 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f924c06b4d0 0x7f924c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:14.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.898+0000 7f925e7fc640 1 --2- 172.21.15.161:0/30939122 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f924c06b4d0 0x7f924c06d990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f92600c6a70 tx=0x7f9254007c10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:14.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:14.906+0000 7f92477fe640 1 -- 172.21.15.161:0/30939122 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f92580597f0 con 0x7f92600670d0 2024-03-28T23:32:15.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.121+0000 7f92656eb640 1 -- 172.21.15.161:0/30939122 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f9260066eb0 con 0x7f92600670d0 2024-03-28T23:32:15.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.771+0000 7f92477fe640 1 -- 172.21.15.161:0/30939122 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v45) ==== 104+0+0 (secure 0 0 0) 0x7f92580590f0 con 0x7f92600670d0 2024-03-28T23:32:15.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.774+0000 7f92656eb640 1 -- 172.21.15.161:0/30939122 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f924c06b4d0 msgr2=0x7f924c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:15.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.774+0000 7f92656eb640 1 --2- 172.21.15.161:0/30939122 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f924c06b4d0 0x7f924c06d990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f92600c6a70 tx=0x7f9254007c10 comp rx=0 tx=0).stop 2024-03-28T23:32:15.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.774+0000 7f92656eb640 1 -- 172.21.15.161:0/30939122 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92600670d0 msgr2=0x7f9260154d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:15.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.774+0000 7f92656eb640 1 --2- 172.21.15.161:0/30939122 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92600670d0 0x7f9260154d80 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f9258007e70 tx=0x7f925800e4b0 comp rx=0 tx=0).stop 2024-03-28T23:32:15.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.774+0000 7f92656eb640 1 -- 172.21.15.161:0/30939122 shutdown_connections 2024-03-28T23:32:15.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.774+0000 7f92656eb640 1 --2- 172.21.15.161:0/30939122 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f924c06b4d0 0x7f924c06d990 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:15.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.774+0000 7f92656eb640 1 --2- 172.21.15.161:0/30939122 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9260152890 0x7f92600c5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:15.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.774+0000 7f92656eb640 1 --2- 172.21.15.161:0/30939122 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92600c13c0 0x7f92600acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:15.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.774+0000 7f92656eb640 1 --2- 172.21.15.161:0/30939122 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92600670d0 0x7f9260154d80 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:15.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.774+0000 7f92656eb640 1 -- 172.21.15.161:0/30939122 >> 172.21.15.161:0/30939122 conn(0x7f92600c3b70 msgr2=0x7f926006b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:15.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.775+0000 7f92656eb640 1 -- 172.21.15.161:0/30939122 shutdown_connections 2024-03-28T23:32:15.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.775+0000 7f92656eb640 1 -- 172.21.15.161:0/30939122 wait complete. 2024-03-28T23:32:15.781 INFO:tasks.ceph.mds.c.smithi161.stderr: -527> 2024-03-28T23:31:59.636+0000 7f1efe9ea640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:32:15.791 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-28T23:32:15.808 INFO:tasks.ceph.mds.c.smithi161.stdout:starting mds.c at 2024-03-28T23:32:15.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.911+0000 7fbcd231e640 1 Processor -- start 2024-03-28T23:32:15.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.911+0000 7fbcd231e640 1 -- start start 2024-03-28T23:32:15.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.912+0000 7fbcd231e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fbccc14f1d0 0x7fbccc151db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:15.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.912+0000 7fbcd231e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbccc0c15a0 con 0x7fbccc0c3080 2024-03-28T23:32:15.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.912+0000 7fbcd231e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbccc0c6330 con 0x7fbccc06cab0 2024-03-28T23:32:15.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.912+0000 7fbcd231e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fbccc0c1840 con 0x7fbccc14f1d0 2024-03-28T23:32:15.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.912+0000 7fbccb7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fbccc0c3080 0x7fbccc0c3a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:53888/0 (socket says 172.21.15.161:53888) 2024-03-28T23:32:15.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.912+0000 7fbccb7fe640 1 -- 172.21.15.161:0/1898552560 learned_addr learned my addr 172.21.15.161:0/1898552560 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:15.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.912+0000 7fbccbfff640 1 --2- 172.21.15.161:0/1898552560 >> v2:172.21.15.170:3300/0 conn(0x7fbccc14f1d0 0x7fbccc151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:15.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.912+0000 7fbccaffd640 1 -- 172.21.15.161:0/1898552560 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1004539062 0 0) 0x7fbccc0c15a0 con 0x7fbccc0c3080 2024-03-28T23:32:15.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.912+0000 7fbccaffd640 1 -- 172.21.15.161:0/1898552560 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcb4003520 con 0x7fbccc0c3080 2024-03-28T23:32:15.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.913+0000 7fbccaffd640 1 -- 172.21.15.161:0/1898552560 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2247572914 0 0) 0x7fbcb4003520 con 0x7fbccc0c3080 2024-03-28T23:32:15.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.913+0000 7fbccaffd640 1 -- 172.21.15.161:0/1898552560 >> v2:172.21.15.170:3300/0 conn(0x7fbccc14f1d0 msgr2=0x7fbccc151db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:15.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.913+0000 7fbccaffd640 1 --2- 172.21.15.161:0/1898552560 >> v2:172.21.15.170:3300/0 conn(0x7fbccc14f1d0 0x7fbccc151db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:15.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.913+0000 7fbccaffd640 1 -- 172.21.15.161:0/1898552560 >> v1:172.21.15.170:6789/0 conn(0x7fbccc06cab0 legacy=0x7fbccc06cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:15.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.913+0000 7fbccaffd640 1 -- 172.21.15.161:0/1898552560 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbccc0c1b20 con 0x7fbccc0c3080 2024-03-28T23:32:15.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.913+0000 7fbccaffd640 1 -- 172.21.15.161:0/1898552560 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fbcc0001670 con 0x7fbccc0c3080 2024-03-28T23:32:15.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.913+0000 7fbccaffd640 1 -- 172.21.15.161:0/1898552560 >> v1:172.21.15.161:6789/0 conn(0x7fbccc0c3080 legacy=0x7fbccc0c3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:15.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.913+0000 7fbccaffd640 1 --2- 172.21.15.161:0/1898552560 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbcb40039b0 0x7fbcb4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:15.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.913+0000 7fbccaffd640 1 --2- 172.21.15.161:0/1898552560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbcb4004800 0x7fbcb4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:15.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.913+0000 7fbccaffd640 1 --2- 172.21.15.161:0/1898552560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbcb40051c0 0x7fbcb400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:15.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.913+0000 7fbccaffd640 1 -- 172.21.15.161:0/1898552560 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbccc0c1840 con 0x7fbcb40051c0 2024-03-28T23:32:15.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.913+0000 7fbccaffd640 1 -- 172.21.15.161:0/1898552560 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb400c160 con 0x7fbcb4004800 2024-03-28T23:32:15.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.913+0000 7fbccaffd640 1 -- 172.21.15.161:0/1898552560 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbcb400c360 con 0x7fbcb40039b0 2024-03-28T23:32:15.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.913+0000 7fbcd08bb640 1 --2- 172.21.15.161:0/1898552560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbcb40051c0 0x7fbcb400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:15.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.913+0000 7fbccb7fe640 1 --2- 172.21.15.161:0/1898552560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbcb4004800 0x7fbcb4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:15.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.913+0000 7fbccbfff640 1 --2- 172.21.15.161:0/1898552560 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbcb40039b0 0x7fbcb4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:15.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.913+0000 7fbcd08bb640 1 -- 172.21.15.161:0/1898552560 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbcb40039b0 msgr2=0x7fbcb4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:15.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.913+0000 7fbcd08bb640 1 --2- 172.21.15.161:0/1898552560 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbcb40039b0 0x7fbcb4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:15.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.913+0000 7fbcd08bb640 1 -- 172.21.15.161:0/1898552560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbcb4004800 msgr2=0x7fbcb4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:15.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.913+0000 7fbcd08bb640 1 --2- 172.21.15.161:0/1898552560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbcb4004800 0x7fbcb4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:15.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.913+0000 7fbcd08bb640 1 -- 172.21.15.161:0/1898552560 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcb4003520 con 0x7fbcb40051c0 2024-03-28T23:32:15.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.914+0000 7fbcd08bb640 1 -- 172.21.15.161:0/1898552560 reap_dead start 2024-03-28T23:32:15.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.914+0000 7fbcd08bb640 1 --2- 172.21.15.161:0/1898552560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbcb40051c0 0x7fbcb400b9c0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4001ee0 tx=0x7fbcb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6679f39606e79b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:15.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.914+0000 7fbccaffd640 1 -- 172.21.15.161:0/1898552560 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4001b00 con 0x7fbcb40051c0 2024-03-28T23:32:15.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.914+0000 7fbccaffd640 1 -- 172.21.15.161:0/1898552560 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcc402ee10 con 0x7fbcb40051c0 2024-03-28T23:32:15.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.914+0000 7fbccaffd640 1 -- 172.21.15.161:0/1898552560 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4003bb0 con 0x7fbcb40051c0 2024-03-28T23:32:15.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.914+0000 7fbcd231e640 1 -- 172.21.15.161:0/1898552560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbcb40051c0 msgr2=0x7fbcb400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:15.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.914+0000 7fbcd231e640 1 --2- 172.21.15.161:0/1898552560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbcb40051c0 0x7fbcb400b9c0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4001ee0 tx=0x7fbcb4005830 comp rx=0 tx=0).stop 2024-03-28T23:32:15.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.915+0000 7fbcd231e640 1 -- 172.21.15.161:0/1898552560 shutdown_connections 2024-03-28T23:32:15.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.915+0000 7fbcd231e640 1 --2- 172.21.15.161:0/1898552560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbcb40051c0 0x7fbcb400b9c0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:15.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.915+0000 7fbcd231e640 1 -- 172.21.15.161:0/1898552560 >> 172.21.15.161:0/1898552560 conn(0x7fbccc0c6e20 msgr2=0x7fbccc0c7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:15.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.915+0000 7fbcd231e640 1 -- 172.21.15.161:0/1898552560 shutdown_connections 2024-03-28T23:32:15.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.915+0000 7fbcd231e640 1 -- 172.21.15.161:0/1898552560 wait complete. 2024-03-28T23:32:15.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.916+0000 7fbcd231e640 1 Processor -- start 2024-03-28T23:32:15.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.916+0000 7fbcd231e640 1 -- start start 2024-03-28T23:32:15.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.917+0000 7fbcd231e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbcb40051c0 0x7fbccc1dd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:15.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.917+0000 7fbcd231e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbccc1531a0 0x7fbccc1e8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:15.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.917+0000 7fbcd231e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbccc1e87b0 0x7fbccc0a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:15.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.917+0000 7fbcd231e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbccc14f5b0 con 0x7fbccc1531a0 2024-03-28T23:32:15.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.917+0000 7fbcd231e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbccc0c2600 con 0x7fbccc1e87b0 2024-03-28T23:32:15.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.917+0000 7fbcd231e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbccc14f730 con 0x7fbcb40051c0 2024-03-28T23:32:15.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.917+0000 7fbccb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbccc1531a0 0x7fbccc1e8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:15.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.917+0000 7fbccb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbccc1531a0 0x7fbccc1e8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41620/0 (socket says 172.21.15.161:41620) 2024-03-28T23:32:15.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.917+0000 7fbccb7fe640 1 -- 172.21.15.161:0/530325234 learned_addr learned my addr 172.21.15.161:0/530325234 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:15.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.917+0000 7fbccbfff640 1 --2- 172.21.15.161:0/530325234 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbcb40051c0 0x7fbccc1dd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:15.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.917+0000 7fbcd08bb640 1 --2- 172.21.15.161:0/530325234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbccc1e87b0 0x7fbccc0a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:15.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.917+0000 7fbccb7fe640 1 -- 172.21.15.161:0/530325234 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbcb40051c0 msgr2=0x7fbccc1dd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:15.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.917+0000 7fbccb7fe640 1 --2- 172.21.15.161:0/530325234 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbcb40051c0 0x7fbccc1dd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:15.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.917+0000 7fbccb7fe640 1 -- 172.21.15.161:0/530325234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbccc1e87b0 msgr2=0x7fbccc0a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:15.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.917+0000 7fbccb7fe640 1 --2- 172.21.15.161:0/530325234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbccc1e87b0 0x7fbccc0a7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:15.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.917+0000 7fbccb7fe640 1 -- 172.21.15.161:0/530325234 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbccc0a9e90 con 0x7fbccc1531a0 2024-03-28T23:32:15.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.917+0000 7fbccb7fe640 1 --2- 172.21.15.161:0/530325234 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbccc1531a0 0x7fbccc1e8270 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0007310 tx=0x7fbcc00077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:15.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.917+0000 7fbcc8ff9640 1 -- 172.21.15.161:0/530325234 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc000f940 con 0x7fbccc1531a0 2024-03-28T23:32:15.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.917+0000 7fbcc8ff9640 1 -- 172.21.15.161:0/530325234 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcc0018070 con 0x7fbccc1531a0 2024-03-28T23:32:15.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.917+0000 7fbcd231e640 1 -- 172.21.15.161:0/530325234 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbccc0a9010 con 0x7fbccc1531a0 2024-03-28T23:32:15.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.917+0000 7fbcc8ff9640 1 -- 172.21.15.161:0/530325234 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc000fd40 con 0x7fbccc1531a0 2024-03-28T23:32:15.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.917+0000 7fbcd231e640 1 -- 172.21.15.161:0/530325234 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbccc1ea990 con 0x7fbccc1531a0 2024-03-28T23:32:15.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.918+0000 7fbcd231e640 1 -- 172.21.15.161:0/530325234 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbccc0693e0 con 0x7fbccc1531a0 2024-03-28T23:32:15.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.919+0000 7fbcc8ff9640 1 -- 172.21.15.161:0/530325234 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbcc001d460 con 0x7fbccc1531a0 2024-03-28T23:32:15.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.919+0000 7fbcc8ff9640 1 --2- 172.21.15.161:0/530325234 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbcb806b4d0 0x7fbcb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:15.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.919+0000 7fbcc8ff9640 1 -- 172.21.15.161:0/530325234 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 7301+0+0 (secure 0 0 0) 0x7fbcc008a870 con 0x7fbccc1531a0 2024-03-28T23:32:15.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.919+0000 7fbccbfff640 1 --2- 172.21.15.161:0/530325234 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbcb806b4d0 0x7fbcb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:15.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.920+0000 7fbccbfff640 1 --2- 172.21.15.161:0/530325234 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbcb806b4d0 0x7fbcb806d990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbccc0a9d50 tx=0x7fbcbc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:15.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:15.924+0000 7fbcc8ff9640 1 -- 172.21.15.161:0/530325234 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbccc0693e0 con 0x7fbccc1531a0 2024-03-28T23:32:16.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.095+0000 7fbcd231e640 1 -- 172.21.15.161:0/530325234 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fbccc068f90 con 0x7fbccc1531a0 2024-03-28T23:32:16.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.777+0000 7fbcc8ff9640 1 -- 172.21.15.161:0/530325234 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v60) ==== 181+0+0 (secure 0 0 0) 0x7fbcc0059200 con 0x7fbccc1531a0 2024-03-28T23:32:16.778 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_metadata' removed 2024-03-28T23:32:16.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.780+0000 7fbcd231e640 1 -- 172.21.15.161:0/530325234 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbcb806b4d0 msgr2=0x7fbcb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:16.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.780+0000 7fbcd231e640 1 --2- 172.21.15.161:0/530325234 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbcb806b4d0 0x7fbcb806d990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbccc0a9d50 tx=0x7fbcbc008040 comp rx=0 tx=0).stop 2024-03-28T23:32:16.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.780+0000 7fbcd231e640 1 -- 172.21.15.161:0/530325234 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbccc1531a0 msgr2=0x7fbccc1e8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:16.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.780+0000 7fbcd231e640 1 --2- 172.21.15.161:0/530325234 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbccc1531a0 0x7fbccc1e8270 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0007310 tx=0x7fbcc00077d0 comp rx=0 tx=0).stop 2024-03-28T23:32:16.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.780+0000 7fbcd231e640 1 -- 172.21.15.161:0/530325234 shutdown_connections 2024-03-28T23:32:16.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.780+0000 7fbcd231e640 1 --2- 172.21.15.161:0/530325234 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbcb806b4d0 0x7fbcb806d990 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:16.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.780+0000 7fbcd231e640 1 --2- 172.21.15.161:0/530325234 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbccc1531a0 0x7fbccc1e8270 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:16.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.780+0000 7fbcd231e640 1 --2- 172.21.15.161:0/530325234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbccc1e87b0 0x7fbccc0a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:16.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.780+0000 7fbcd231e640 1 --2- 172.21.15.161:0/530325234 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbcb40051c0 0x7fbccc1dd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:16.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.780+0000 7fbcd231e640 1 -- 172.21.15.161:0/530325234 >> 172.21.15.161:0/530325234 conn(0x7fbccc0c6e20 msgr2=0x7fbccc14de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:16.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.780+0000 7fbcd231e640 1 -- 172.21.15.161:0/530325234 shutdown_connections 2024-03-28T23:32:16.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.780+0000 7fbcd231e640 1 -- 172.21.15.161:0/530325234 wait complete. 2024-03-28T23:32:16.797 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-28T23:32:16.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.905+0000 7ff49edbc640 1 Processor -- start 2024-03-28T23:32:16.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.905+0000 7ff49edbc640 1 -- start start 2024-03-28T23:32:16.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.906+0000 7ff49edbc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff4980c30c0 0x7ff498152880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:16.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.906+0000 7ff49edbc640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4980c3540 con 0x7ff4980c19b0 2024-03-28T23:32:16.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.906+0000 7ff49edbc640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4980c3820 con 0x7ff4980ac430 2024-03-28T23:32:16.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.906+0000 7ff49edbc640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff498152dc0 con 0x7ff4980c30c0 2024-03-28T23:32:16.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.906+0000 7ff48ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff4980c30c0 0x7ff498152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:16.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.906+0000 7ff49d359640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ff4980c19b0 0x7ff49815c780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:53890/0 (socket says 172.21.15.161:53890) 2024-03-28T23:32:16.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.906+0000 7ff49d359640 1 -- 172.21.15.161:0/2747412042 learned_addr learned my addr 172.21.15.161:0/2747412042 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:16.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.906+0000 7ff48ffff640 1 -- 172.21.15.161:0/2747412042 >> v1:172.21.15.170:6789/0 conn(0x7ff4980ac430 legacy=0x7ff4980c24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:16.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.906+0000 7ff48ffff640 1 -- 172.21.15.161:0/2747412042 >> v1:172.21.15.161:6789/0 conn(0x7ff4980c19b0 legacy=0x7ff49815c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:16.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.906+0000 7ff48ffff640 1 -- 172.21.15.161:0/2747412042 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4980c3b00 con 0x7ff4980c30c0 2024-03-28T23:32:16.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.906+0000 7ff48f7fe640 1 -- 172.21.15.161:0/2747412042 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3576589750 0 0) 0x7ff4980c3540 con 0x7ff4980c19b0 2024-03-28T23:32:16.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.907+0000 7ff48ffff640 1 --2- 172.21.15.161:0/2747412042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4980c30c0 0x7ff498152880 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7ff4880097a0 tx=0x7ff48802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7ac4c4ae2dbcbfc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:16.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.907+0000 7ff48f7fe640 1 -- 172.21.15.161:0/2747412042 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff488035020 con 0x7ff4980c30c0 2024-03-28T23:32:16.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.907+0000 7ff48f7fe640 1 -- 172.21.15.161:0/2747412042 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff488031830 con 0x7ff4980c30c0 2024-03-28T23:32:16.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.907+0000 7ff48f7fe640 1 -- 172.21.15.161:0/2747412042 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff488031c00 con 0x7ff4980c30c0 2024-03-28T23:32:16.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.907+0000 7ff49edbc640 1 -- 172.21.15.161:0/2747412042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4980c30c0 msgr2=0x7ff498152880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:16.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.907+0000 7ff49edbc640 1 --2- 172.21.15.161:0/2747412042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4980c30c0 0x7ff498152880 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7ff4880097a0 tx=0x7ff48802b8d0 comp rx=0 tx=0).stop 2024-03-28T23:32:16.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.907+0000 7ff49edbc640 1 -- 172.21.15.161:0/2747412042 shutdown_connections 2024-03-28T23:32:16.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.907+0000 7ff49edbc640 1 --2- 172.21.15.161:0/2747412042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4980c30c0 0x7ff498152880 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:16.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.907+0000 7ff49edbc640 1 -- 172.21.15.161:0/2747412042 >> 172.21.15.161:0/2747412042 conn(0x7ff49806b150 msgr2=0x7ff49806b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:16.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.907+0000 7ff49edbc640 1 -- 172.21.15.161:0/2747412042 shutdown_connections 2024-03-28T23:32:16.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.908+0000 7ff49edbc640 1 -- 172.21.15.161:0/2747412042 wait complete. 2024-03-28T23:32:16.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.909+0000 7ff49edbc640 1 Processor -- start 2024-03-28T23:32:16.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.909+0000 7ff49edbc640 1 -- start start 2024-03-28T23:32:16.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.909+0000 7ff49edbc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4980ac430 0x7ff49814dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:16.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.909+0000 7ff49edbc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4980c19b0 0x7ff498150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:16.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.909+0000 7ff49edbc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4980c30c0 0x7ff49814fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:16.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.909+0000 7ff49edbc640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff4980c6440 con 0x7ff4980c19b0 2024-03-28T23:32:16.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.909+0000 7ff49edbc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff4980c1490 con 0x7ff4980ac430 2024-03-28T23:32:16.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.909+0000 7ff49edbc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff4980c2be0 con 0x7ff4980c30c0 2024-03-28T23:32:16.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.910+0000 7ff48ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4980c19b0 0x7ff498150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:16.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.910+0000 7ff48ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4980c19b0 0x7ff498150b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41624/0 (socket says 172.21.15.161:41624) 2024-03-28T23:32:16.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.910+0000 7ff48ffff640 1 -- 172.21.15.161:0/2476223859 learned_addr learned my addr 172.21.15.161:0/2476223859 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:16.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.910+0000 7ff49d359640 1 --2- 172.21.15.161:0/2476223859 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4980c30c0 0x7ff49814fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:16.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.910+0000 7ff49cb58640 1 --2- 172.21.15.161:0/2476223859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4980ac430 0x7ff49814dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:16.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.910+0000 7ff48ffff640 1 -- 172.21.15.161:0/2476223859 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4980c30c0 msgr2=0x7ff49814fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:16.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.910+0000 7ff48ffff640 1 --2- 172.21.15.161:0/2476223859 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4980c30c0 0x7ff49814fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:16.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.910+0000 7ff48ffff640 1 -- 172.21.15.161:0/2476223859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4980ac430 msgr2=0x7ff49814dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:16.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.910+0000 7ff48ffff640 1 --2- 172.21.15.161:0/2476223859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4980ac430 0x7ff49814dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:16.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.910+0000 7ff48ffff640 1 -- 172.21.15.161:0/2476223859 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff498151780 con 0x7ff4980c19b0 2024-03-28T23:32:16.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.910+0000 7ff48ffff640 1 --2- 172.21.15.161:0/2476223859 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4980c19b0 0x7ff498150b30 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7ff488034040 tx=0x7ff4880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:16.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.910+0000 7ff48d7fa640 1 -- 172.21.15.161:0/2476223859 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff488041070 con 0x7ff4980c19b0 2024-03-28T23:32:16.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.910+0000 7ff48d7fa640 1 -- 172.21.15.161:0/2476223859 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff488004230 con 0x7ff4980c19b0 2024-03-28T23:32:16.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.910+0000 7ff49edbc640 1 -- 172.21.15.161:0/2476223859 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4980732e0 con 0x7ff4980c19b0 2024-03-28T23:32:16.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.910+0000 7ff48d7fa640 1 -- 172.21.15.161:0/2476223859 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff488035040 con 0x7ff4980c19b0 2024-03-28T23:32:16.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.910+0000 7ff49edbc640 1 -- 172.21.15.161:0/2476223859 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4980a6730 con 0x7ff4980c19b0 2024-03-28T23:32:16.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.911+0000 7ff49edbc640 1 -- 172.21.15.161:0/2476223859 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff498068f90 con 0x7ff4980c19b0 2024-03-28T23:32:16.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.912+0000 7ff48d7fa640 1 -- 172.21.15.161:0/2476223859 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff488038020 con 0x7ff4980c19b0 2024-03-28T23:32:16.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.912+0000 7ff48d7fa640 1 --2- 172.21.15.161:0/2476223859 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff48406b500 0x7ff48406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:16.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.912+0000 7ff48d7fa640 1 -- 172.21.15.161:0/2476223859 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6863+0+0 (secure 0 0 0) 0x7ff4880ac0e0 con 0x7ff4980c19b0 2024-03-28T23:32:16.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.912+0000 7ff49cb58640 1 --2- 172.21.15.161:0/2476223859 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff48406b500 0x7ff48406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:16.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.912+0000 7ff49cb58640 1 --2- 172.21.15.161:0/2476223859 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff48406b500 0x7ff48406d9c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff494000f60 tx=0x7ff494009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:16.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:16.916+0000 7ff48d7fa640 1 -- 172.21.15.161:0/2476223859 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff48807aae0 con 0x7ff4980c19b0 2024-03-28T23:32:17.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.088+0000 7ff49edbc640 1 -- 172.21.15.161:0/2476223859 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff4980c6220 con 0x7ff4980c19b0 2024-03-28T23:32:17.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.793+0000 7ff48d7fa640 1 -- 172.21.15.161:0/2476223859 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v61) ==== 169+0+0 (secure 0 0 0) 0x7ff48807a3e0 con 0x7ff4980c19b0 2024-03-28T23:32:17.794 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data' removed 2024-03-28T23:32:17.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.801+0000 7ff49edbc640 1 -- 172.21.15.161:0/2476223859 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff48406b500 msgr2=0x7ff48406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:17.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.801+0000 7ff49edbc640 1 --2- 172.21.15.161:0/2476223859 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff48406b500 0x7ff48406d9c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff494000f60 tx=0x7ff494009040 comp rx=0 tx=0).stop 2024-03-28T23:32:17.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.801+0000 7ff49edbc640 1 -- 172.21.15.161:0/2476223859 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4980c19b0 msgr2=0x7ff498150b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:17.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.801+0000 7ff49edbc640 1 --2- 172.21.15.161:0/2476223859 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4980c19b0 0x7ff498150b30 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7ff488034040 tx=0x7ff4880046c0 comp rx=0 tx=0).stop 2024-03-28T23:32:17.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.801+0000 7ff49edbc640 1 -- 172.21.15.161:0/2476223859 shutdown_connections 2024-03-28T23:32:17.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.801+0000 7ff49edbc640 1 --2- 172.21.15.161:0/2476223859 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff48406b500 0x7ff48406d9c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:17.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.801+0000 7ff49edbc640 1 --2- 172.21.15.161:0/2476223859 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4980c19b0 0x7ff498150b30 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:17.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.801+0000 7ff49edbc640 1 --2- 172.21.15.161:0/2476223859 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4980c30c0 0x7ff49814fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:17.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.801+0000 7ff49edbc640 1 --2- 172.21.15.161:0/2476223859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4980ac430 0x7ff49814dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:17.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.801+0000 7ff49edbc640 1 -- 172.21.15.161:0/2476223859 >> 172.21.15.161:0/2476223859 conn(0x7ff49806b150 msgr2=0x7ff49806cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:17.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.802+0000 7ff49edbc640 1 -- 172.21.15.161:0/2476223859 shutdown_connections 2024-03-28T23:32:17.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.802+0000 7ff49edbc640 1 -- 172.21.15.161:0/2476223859 wait complete. 2024-03-28T23:32:17.818 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:32:17.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.924+0000 7f284e3bc640 1 Processor -- start 2024-03-28T23:32:17.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.924+0000 7f284e3bc640 1 -- start start 2024-03-28T23:32:17.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.925+0000 7f284e3bc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f284806c3d0 0x7f2848152580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:17.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.925+0000 7f284e3bc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f28480ac150 0x7f284815fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:17.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.925+0000 7f284e3bc640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2848066fe0 con 0x7f2848155b90 2024-03-28T23:32:17.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.925+0000 7f284e3bc640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2848156070 con 0x7f28480ac150 2024-03-28T23:32:17.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.925+0000 7f284e3bc640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f28480672c0 con 0x7f284806c3d0 2024-03-28T23:32:17.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.925+0000 7f284c959640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f28480ac150 0x7f284815fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:17.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.925+0000 7f284c959640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f28480ac150 0x7f284815fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41634/0 (socket says 172.21.15.161:41634) 2024-03-28T23:32:17.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.925+0000 7f284c959640 1 -- 172.21.15.161:0/1357557390 learned_addr learned my addr 172.21.15.161:0/1357557390 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:17.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.925+0000 7f284c959640 1 -- 172.21.15.161:0/1357557390 >> v2:172.21.15.170:3300/0 conn(0x7f284806c3d0 msgr2=0x7f2848152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:17.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.925+0000 7f284c959640 1 --2- 172.21.15.161:0/1357557390 >> v2:172.21.15.170:3300/0 conn(0x7f284806c3d0 0x7f2848152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:17.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.925+0000 7f284c959640 1 -- 172.21.15.161:0/1357557390 >> v1:172.21.15.170:6789/0 conn(0x7f2848155b90 legacy=0x7f2848158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:17.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.925+0000 7f284c959640 1 -- 172.21.15.161:0/1357557390 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28480c1c80 con 0x7f28480ac150 2024-03-28T23:32:17.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.926+0000 7f284c959640 1 --2- 172.21.15.161:0/1357557390 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28480ac150 0x7f284815fae0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f28480709c0 tx=0x7f284002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a88c8f78a9dd13 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:17.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.926+0000 7f2846ffd640 1 -- 172.21.15.161:0/1357557390 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284002c9f0 con 0x7f28480ac150 2024-03-28T23:32:17.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.926+0000 7f2846ffd640 1 -- 172.21.15.161:0/1357557390 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f284002cb90 con 0x7f28480ac150 2024-03-28T23:32:17.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.926+0000 7f2846ffd640 1 -- 172.21.15.161:0/1357557390 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28400343f0 con 0x7f28480ac150 2024-03-28T23:32:17.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.926+0000 7f284e3bc640 1 -- 172.21.15.161:0/1357557390 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28480ac150 msgr2=0x7f284815fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:17.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.926+0000 7f284e3bc640 1 --2- 172.21.15.161:0/1357557390 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28480ac150 0x7f284815fae0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f28480709c0 tx=0x7f284002be50 comp rx=0 tx=0).stop 2024-03-28T23:32:17.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.926+0000 7f284e3bc640 1 -- 172.21.15.161:0/1357557390 shutdown_connections 2024-03-28T23:32:17.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.926+0000 7f284e3bc640 1 --2- 172.21.15.161:0/1357557390 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28480ac150 0x7f284815fae0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:17.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.926+0000 7f284e3bc640 1 --2- 172.21.15.161:0/1357557390 >> v2:172.21.15.170:3300/0 conn(0x7f284806c3d0 0x7f2848152580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:17.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.926+0000 7f284e3bc640 1 -- 172.21.15.161:0/1357557390 >> 172.21.15.161:0/1357557390 conn(0x7f28480c3be0 msgr2=0x7f28480c3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:17.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.926+0000 7f284e3bc640 1 -- 172.21.15.161:0/1357557390 shutdown_connections 2024-03-28T23:32:17.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.926+0000 7f284e3bc640 1 -- 172.21.15.161:0/1357557390 wait complete. 2024-03-28T23:32:17.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.927+0000 7f284e3bc640 1 Processor -- start 2024-03-28T23:32:17.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.928+0000 7f284e3bc640 1 -- start start 2024-03-28T23:32:17.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.928+0000 7f284e3bc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f284806c3d0 0x7f284814d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:17.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.928+0000 7f284e3bc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28480ac150 0x7f2848147900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:17.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.928+0000 7f284e3bc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2848155b90 0x7f2848147e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:17.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.928+0000 7f284e3bc640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f284806c7b0 con 0x7f2848155b90 2024-03-28T23:32:17.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.928+0000 7f284e3bc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f284806c930 con 0x7f284806c3d0 2024-03-28T23:32:17.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.928+0000 7f284e3bc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f28480c4900 con 0x7f28480ac150 2024-03-28T23:32:17.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.928+0000 7f284c959640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2848155b90 0x7f2848147e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:17.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.928+0000 7f284c959640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2848155b90 0x7f2848147e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41644/0 (socket says 172.21.15.161:41644) 2024-03-28T23:32:17.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.928+0000 7f284c959640 1 -- 172.21.15.161:0/1101970834 learned_addr learned my addr 172.21.15.161:0/1101970834 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:17.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.929+0000 7f284c959640 1 -- 172.21.15.161:0/1101970834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28480ac150 msgr2=0x7f2848147900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:17.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.929+0000 7f284c959640 1 --2- 172.21.15.161:0/1101970834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28480ac150 0x7f2848147900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:17.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.929+0000 7f284c959640 1 -- 172.21.15.161:0/1101970834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f284806c3d0 msgr2=0x7f284814d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:17.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.929+0000 7f2847fff640 1 --2- 172.21.15.161:0/1101970834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f284806c3d0 0x7f284814d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:17.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.929+0000 7f284c959640 1 --2- 172.21.15.161:0/1101970834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f284806c3d0 0x7f284814d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:17.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.929+0000 7f284c959640 1 -- 172.21.15.161:0/1101970834 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28481ee9a0 con 0x7f2848155b90 2024-03-28T23:32:17.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.929+0000 7f284c959640 1 --2- 172.21.15.161:0/1101970834 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2848155b90 0x7f2848147e40 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f2840009f40 tx=0x7f2840004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:17.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.929+0000 7f2844ff9640 1 -- 172.21.15.161:0/1101970834 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2840043020 con 0x7f2848155b90 2024-03-28T23:32:17.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.929+0000 7f2844ff9640 1 -- 172.21.15.161:0/1101970834 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f284003e450 con 0x7f2848155b90 2024-03-28T23:32:17.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.929+0000 7f284e3bc640 1 -- 172.21.15.161:0/1101970834 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28481ebd70 con 0x7f2848155b90 2024-03-28T23:32:17.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.929+0000 7f2844ff9640 1 -- 172.21.15.161:0/1101970834 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2840009510 con 0x7f2848155b90 2024-03-28T23:32:17.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.929+0000 7f284e3bc640 1 -- 172.21.15.161:0/1101970834 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f284814dbf0 con 0x7f2848155b90 2024-03-28T23:32:17.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.930+0000 7f284e3bc640 1 -- 172.21.15.161:0/1101970834 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f284804f4b0 con 0x7f2848155b90 2024-03-28T23:32:17.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.931+0000 7f2844ff9640 1 -- 172.21.15.161:0/1101970834 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2840046020 con 0x7f2848155b90 2024-03-28T23:32:17.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.931+0000 7f2844ff9640 1 --2- 172.21.15.161:0/1101970834 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f283406b4d0 0x7f283406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:17.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.931+0000 7f2844ff9640 1 -- 172.21.15.161:0/1101970834 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6449+0+0 (secure 0 0 0) 0x7f28400acdd0 con 0x7f2848155b90 2024-03-28T23:32:17.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.931+0000 7f2847fff640 1 --2- 172.21.15.161:0/1101970834 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f283406b4d0 0x7f283406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:17.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.931+0000 7f2847fff640 1 --2- 172.21.15.161:0/1101970834 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f283406b4d0 0x7f283406d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f283800a770 tx=0x7f2838008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:17.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:17.935+0000 7f2844ff9640 1 -- 172.21.15.161:0/1101970834 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f284004d030 con 0x7f2848155b90 2024-03-28T23:32:18.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.118+0000 7f284e3bc640 1 -- 172.21.15.161:0/1101970834 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2848067740 con 0x7f2848155b90 2024-03-28T23:32:18.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.120+0000 7f2844ff9640 1 -- 172.21.15.161:0/1101970834 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) ==== 74+0+17449 (secure 0 0 0) 0x7f284007b9f0 con 0x7f2848155b90 2024-03-28T23:32:18.120 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:18.121 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":61,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:32:17.787428+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:31:52.774315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:31:53.201664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6836/1300583128":"2024-03-29T23:32:15.775261+0000","172.21.15.161:6834/1300583128":"2024-03-29T23:32:15.775261+0000","172.21.15.161:6837/1498528126":"2024-03-29T23:32:14.767288+0000","172.21.15.170:6835/1945122204":"2024-03-29T23:32:06.533661+0000","172.21.15.161:6835/1498528126":"2024-03-29T23:32:14.767288+0000","172.21.15.170:6834/1945122204":"2024-03-29T23:32:06.533661+0000","172.21.15.161:6837/544971001":"2024-03-29T23:32:03.518509+0000","172.21.15.161:6835/544971001":"2024-03-29T23:32:03.518509+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:32:18.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.122+0000 7f284e3bc640 1 -- 172.21.15.161:0/1101970834 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f283406b4d0 msgr2=0x7f283406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:18.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.122+0000 7f284e3bc640 1 --2- 172.21.15.161:0/1101970834 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f283406b4d0 0x7f283406d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f283800a770 tx=0x7f2838008040 comp rx=0 tx=0).stop 2024-03-28T23:32:18.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.122+0000 7f284e3bc640 1 -- 172.21.15.161:0/1101970834 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2848155b90 msgr2=0x7f2848147e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:18.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.122+0000 7f284e3bc640 1 --2- 172.21.15.161:0/1101970834 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2848155b90 0x7f2848147e40 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f2840009f40 tx=0x7f2840004a40 comp rx=0 tx=0).stop 2024-03-28T23:32:18.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.122+0000 7f284e3bc640 1 -- 172.21.15.161:0/1101970834 shutdown_connections 2024-03-28T23:32:18.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.122+0000 7f284e3bc640 1 --2- 172.21.15.161:0/1101970834 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f283406b4d0 0x7f283406d990 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.122+0000 7f284e3bc640 1 --2- 172.21.15.161:0/1101970834 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2848155b90 0x7f2848147e40 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.122+0000 7f284e3bc640 1 --2- 172.21.15.161:0/1101970834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28480ac150 0x7f2848147900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.122+0000 7f284e3bc640 1 --2- 172.21.15.161:0/1101970834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f284806c3d0 0x7f284814d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.122+0000 7f284e3bc640 1 -- 172.21.15.161:0/1101970834 >> 172.21.15.161:0/1101970834 conn(0x7f28480c3be0 msgr2=0x7f2848150900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:18.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.122+0000 7f284e3bc640 1 -- 172.21.15.161:0/1101970834 shutdown_connections 2024-03-28T23:32:18.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.122+0000 7f284e3bc640 1 -- 172.21.15.161:0/1101970834 wait complete. 2024-03-28T23:32:18.134 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-03-28T23:31:52.774315+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs2_data', 'create_time': '2024-03-28T23:31:53.201664+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:32:18.134 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:32:18.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.237+0000 7fbefa302640 1 Processor -- start 2024-03-28T23:32:18.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.237+0000 7fbefa302640 1 -- start start 2024-03-28T23:32:18.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.238+0000 7fbefa302640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbef4151e70 0x7fbef4152250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:18.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.238+0000 7fbefa302640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbef4152790 0x7fbef406c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:18.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.238+0000 7fbefa302640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbef40c1e40 con 0x7fbef40c6350 2024-03-28T23:32:18.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.238+0000 7fbefa302640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fbef40c2120 con 0x7fbef4152790 2024-03-28T23:32:18.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.238+0000 7fbefa302640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbef40c2400 con 0x7fbef4151e70 2024-03-28T23:32:18.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.238+0000 7fbef37fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbef4152790 0x7fbef406c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:18.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.238+0000 7fbef889f640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fbef40c6350 0x7fbef41553d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:53900/0 (socket says 172.21.15.161:53900) 2024-03-28T23:32:18.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.238+0000 7fbef889f640 1 -- 172.21.15.161:0/2207865347 learned_addr learned my addr 172.21.15.161:0/2207865347 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:18.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.238+0000 7fbef3fff640 1 --2- 172.21.15.161:0/2207865347 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbef4151e70 0x7fbef4152250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:18.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.238+0000 7fbef37fe640 1 -- 172.21.15.161:0/2207865347 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbef4151e70 msgr2=0x7fbef4152250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:18.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.238+0000 7fbef37fe640 1 --2- 172.21.15.161:0/2207865347 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbef4151e70 0x7fbef4152250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.238+0000 7fbef37fe640 1 -- 172.21.15.161:0/2207865347 >> v1:172.21.15.161:6789/0 conn(0x7fbef40c6350 legacy=0x7fbef41553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:18.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.238+0000 7fbef2ffd640 1 -- 172.21.15.161:0/2207865347 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3387898458 0 0) 0x7fbef40c1e40 con 0x7fbef40c6350 2024-03-28T23:32:18.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.238+0000 7fbef37fe640 1 -- 172.21.15.161:0/2207865347 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbef40c26e0 con 0x7fbef4152790 2024-03-28T23:32:18.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.239+0000 7fbef37fe640 1 --2- 172.21.15.161:0/2207865347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbef4152790 0x7fbef406c6c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fbee80097a0 tx=0x7fbee802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ee2b4eef040102 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:18.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.239+0000 7fbef2ffd640 1 -- 172.21.15.161:0/2207865347 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee80315a0 con 0x7fbef4152790 2024-03-28T23:32:18.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.239+0000 7fbef2ffd640 1 -- 172.21.15.161:0/2207865347 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbee8037070 con 0x7fbef4152790 2024-03-28T23:32:18.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.239+0000 7fbef2ffd640 1 -- 172.21.15.161:0/2207865347 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee8031960 con 0x7fbef4152790 2024-03-28T23:32:18.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.240+0000 7fbefa302640 1 -- 172.21.15.161:0/2207865347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbef4152790 msgr2=0x7fbef406c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:18.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.240+0000 7fbefa302640 1 --2- 172.21.15.161:0/2207865347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbef4152790 0x7fbef406c6c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fbee80097a0 tx=0x7fbee802b840 comp rx=0 tx=0).stop 2024-03-28T23:32:18.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.240+0000 7fbefa302640 1 -- 172.21.15.161:0/2207865347 shutdown_connections 2024-03-28T23:32:18.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.240+0000 7fbefa302640 1 --2- 172.21.15.161:0/2207865347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbef4152790 0x7fbef406c6c0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.240+0000 7fbefa302640 1 --2- 172.21.15.161:0/2207865347 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbef4151e70 0x7fbef4152250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.240+0000 7fbefa302640 1 -- 172.21.15.161:0/2207865347 >> 172.21.15.161:0/2207865347 conn(0x7fbef40c3390 msgr2=0x7fbef40c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:18.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.240+0000 7fbefa302640 1 -- 172.21.15.161:0/2207865347 shutdown_connections 2024-03-28T23:32:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.240+0000 7fbefa302640 1 -- 172.21.15.161:0/2207865347 wait complete. 2024-03-28T23:32:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.241+0000 7fbefa302640 1 Processor -- start 2024-03-28T23:32:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.241+0000 7fbefa302640 1 -- start start 2024-03-28T23:32:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.242+0000 7fbefa302640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbef40c6350 0x7fbef4151260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.242+0000 7fbefa302640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbef4151e70 0x7fbef41415e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.242+0000 7fbefa302640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbef41517a0 0x7fbef4141b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.242+0000 7fbefa302640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbef40c2d80 con 0x7fbef40c6350 2024-03-28T23:32:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.242+0000 7fbefa302640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbef40c2c00 con 0x7fbef41517a0 2024-03-28T23:32:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.242+0000 7fbefa302640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbef40a5fb0 con 0x7fbef4151e70 2024-03-28T23:32:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.242+0000 7fbef3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbef40c6350 0x7fbef4151260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.242+0000 7fbef3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbef40c6350 0x7fbef4151260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41662/0 (socket says 172.21.15.161:41662) 2024-03-28T23:32:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.242+0000 7fbef3fff640 1 -- 172.21.15.161:0/2849991310 learned_addr learned my addr 172.21.15.161:0/2849991310 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.242+0000 7fbef37fe640 1 --2- 172.21.15.161:0/2849991310 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbef4151e70 0x7fbef41415e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.242+0000 7fbef889f640 1 --2- 172.21.15.161:0/2849991310 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbef41517a0 0x7fbef4141b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.242+0000 7fbef3fff640 1 -- 172.21.15.161:0/2849991310 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbef4151e70 msgr2=0x7fbef41415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:18.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.242+0000 7fbef3fff640 1 --2- 172.21.15.161:0/2849991310 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbef4151e70 0x7fbef41415e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.242+0000 7fbef3fff640 1 -- 172.21.15.161:0/2849991310 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbef41517a0 msgr2=0x7fbef4141b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:18.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.242+0000 7fbef3fff640 1 --2- 172.21.15.161:0/2849991310 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbef41517a0 0x7fbef4141b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.242+0000 7fbef3fff640 1 -- 172.21.15.161:0/2849991310 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbef41ebd90 con 0x7fbef40c6350 2024-03-28T23:32:18.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.242+0000 7fbef37fe640 1 --2- 172.21.15.161:0/2849991310 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbef4151e70 0x7fbef41415e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:18.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.242+0000 7fbef3fff640 1 --2- 172.21.15.161:0/2849991310 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbef40c6350 0x7fbef4151260 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fbee4007ec0 tx=0x7fbee400dbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:18.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.242+0000 7fbef0ff9640 1 -- 172.21.15.161:0/2849991310 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee400b860 con 0x7fbef40c6350 2024-03-28T23:32:18.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.242+0000 7fbef0ff9640 1 -- 172.21.15.161:0/2849991310 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbee400ba00 con 0x7fbef40c6350 2024-03-28T23:32:18.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.242+0000 7fbefa302640 1 -- 172.21.15.161:0/2849991310 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbef41ec070 con 0x7fbef40c6350 2024-03-28T23:32:18.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.242+0000 7fbef0ff9640 1 -- 172.21.15.161:0/2849991310 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee400bdd0 con 0x7fbef40c6350 2024-03-28T23:32:18.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.242+0000 7fbefa302640 1 -- 172.21.15.161:0/2849991310 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbef414eec0 con 0x7fbef40c6350 2024-03-28T23:32:18.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.243+0000 7fbefa302640 1 -- 172.21.15.161:0/2849991310 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbef404f4b0 con 0x7fbef40c6350 2024-03-28T23:32:18.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.244+0000 7fbef0ff9640 1 -- 172.21.15.161:0/2849991310 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbee4019020 con 0x7fbef40c6350 2024-03-28T23:32:18.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.244+0000 7fbef0ff9640 1 --2- 172.21.15.161:0/2849991310 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbee006b500 0x7fbee006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:18.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.244+0000 7fbef0ff9640 1 -- 172.21.15.161:0/2849991310 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6449+0+0 (secure 0 0 0) 0x7fbee408c250 con 0x7fbef40c6350 2024-03-28T23:32:18.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.244+0000 7fbef37fe640 1 --2- 172.21.15.161:0/2849991310 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbee006b500 0x7fbee006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:18.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.245+0000 7fbef37fe640 1 --2- 172.21.15.161:0/2849991310 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbee006b500 0x7fbee006d9c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbee8036000 tx=0x7fbee803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:18.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.249+0000 7fbef0ff9640 1 -- 172.21.15.161:0/2849991310 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbee405ae70 con 0x7fbef40c6350 2024-03-28T23:32:18.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.419+0000 7fbefa302640 1 -- 172.21.15.161:0/2849991310 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbef414ec40 con 0x7fbef40c6350 2024-03-28T23:32:18.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.419+0000 7fbef0ff9640 1 -- 172.21.15.161:0/2849991310 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v61) ==== 93+0+10 (secure 0 0 0) 0x7fbee405a770 con 0x7fbef40c6350 2024-03-28T23:32:18.420 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:32:18.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.421+0000 7fbefa302640 1 -- 172.21.15.161:0/2849991310 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbee006b500 msgr2=0x7fbee006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:18.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.421+0000 7fbefa302640 1 --2- 172.21.15.161:0/2849991310 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbee006b500 0x7fbee006d9c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbee8036000 tx=0x7fbee803d040 comp rx=0 tx=0).stop 2024-03-28T23:32:18.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.421+0000 7fbefa302640 1 -- 172.21.15.161:0/2849991310 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbef40c6350 msgr2=0x7fbef4151260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:18.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.421+0000 7fbefa302640 1 --2- 172.21.15.161:0/2849991310 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbef40c6350 0x7fbef4151260 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fbee4007ec0 tx=0x7fbee400dbd0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.421+0000 7fbefa302640 1 -- 172.21.15.161:0/2849991310 shutdown_connections 2024-03-28T23:32:18.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.421+0000 7fbefa302640 1 --2- 172.21.15.161:0/2849991310 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbee006b500 0x7fbee006d9c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.421+0000 7fbefa302640 1 --2- 172.21.15.161:0/2849991310 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbef41517a0 0x7fbef4141b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.421+0000 7fbefa302640 1 --2- 172.21.15.161:0/2849991310 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbef4151e70 0x7fbef41415e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.421+0000 7fbefa302640 1 --2- 172.21.15.161:0/2849991310 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbef40c6350 0x7fbef4151260 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.421+0000 7fbefa302640 1 -- 172.21.15.161:0/2849991310 >> 172.21.15.161:0/2849991310 conn(0x7fbef40c3390 msgr2=0x7fbef40a97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:18.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.422+0000 7fbefa302640 1 -- 172.21.15.161:0/2849991310 shutdown_connections 2024-03-28T23:32:18.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.422+0000 7fbefa302640 1 -- 172.21.15.161:0/2849991310 wait complete. 2024-03-28T23:32:18.433 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-03-28T23:32:18.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.538+0000 7fc1af720640 1 Processor -- start 2024-03-28T23:32:18.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.539+0000 7fc1af720640 1 -- start start 2024-03-28T23:32:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.539+0000 7fc1af720640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1a814f5c0 0x7fc1a80c2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.539+0000 7fc1af720640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc1a8152bd0 0x7fc1a815c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.539+0000 7fc1af720640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1a80c2f10 con 0x7fc1a806cdd0 2024-03-28T23:32:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.539+0000 7fc1af720640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fc1a80c74d0 con 0x7fc1a8152bd0 2024-03-28T23:32:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.539+0000 7fc1af720640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc1a80c6fa0 con 0x7fc1a814f5c0 2024-03-28T23:32:18.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.540+0000 7fc1adcbd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc1a8152bd0 0x7fc1a815c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:18.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.540+0000 7fc1ad4bc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1a814f5c0 0x7fc1a80c2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:18.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.540+0000 7fc1adcbd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc1a8152bd0 0x7fc1a815c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:55014/0 (socket says 172.21.15.161:55014) 2024-03-28T23:32:18.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.540+0000 7fc1ad4bc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1a814f5c0 0x7fc1a80c2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:42744/0 (socket says 172.21.15.161:42744) 2024-03-28T23:32:18.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.540+0000 7fc1adcbd640 1 -- 172.21.15.161:0/2961970760 learned_addr learned my addr 172.21.15.161:0/2961970760 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:18.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.540+0000 7fc1adcbd640 1 -- 172.21.15.161:0/2961970760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1a814f5c0 msgr2=0x7fc1a80c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:18.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.540+0000 7fc1adcbd640 1 --2- 172.21.15.161:0/2961970760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1a814f5c0 0x7fc1a80c2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.540+0000 7fc1adcbd640 1 -- 172.21.15.161:0/2961970760 >> v1:172.21.15.170:6789/0 conn(0x7fc1a806cdd0 legacy=0x7fc1a81524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:18.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.540+0000 7fc1adcbd640 1 -- 172.21.15.161:0/2961970760 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1a814f9a0 con 0x7fc1a8152bd0 2024-03-28T23:32:18.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.540+0000 7fc19ffff640 1 -- 172.21.15.161:0/2961970760 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3870589226 0 0) 0x7fc198001110 con 0x7fc1a806cdd0 2024-03-28T23:32:18.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.541+0000 7fc1adcbd640 1 --2- 172.21.15.161:0/2961970760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1a8152bd0 0x7fc1a815c830 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc1a0009e50 tx=0x7fc1a002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd348745526b1dfb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:18.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.541+0000 7fc19ffff640 1 -- 172.21.15.161:0/2961970760 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a002cb00 con 0x7fc1a8152bd0 2024-03-28T23:32:18.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.541+0000 7fc19ffff640 1 -- 172.21.15.161:0/2961970760 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1a002cca0 con 0x7fc1a8152bd0 2024-03-28T23:32:18.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.541+0000 7fc19ffff640 1 -- 172.21.15.161:0/2961970760 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a0034610 con 0x7fc1a8152bd0 2024-03-28T23:32:18.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.542+0000 7fc1af720640 1 -- 172.21.15.161:0/2961970760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1a8152bd0 msgr2=0x7fc1a815c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:18.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.542+0000 7fc1af720640 1 --2- 172.21.15.161:0/2961970760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1a8152bd0 0x7fc1a815c830 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc1a0009e50 tx=0x7fc1a002bf60 comp rx=0 tx=0).stop 2024-03-28T23:32:18.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.542+0000 7fc1af720640 1 -- 172.21.15.161:0/2961970760 shutdown_connections 2024-03-28T23:32:18.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.542+0000 7fc1af720640 1 --2- 172.21.15.161:0/2961970760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1a8152bd0 0x7fc1a815c830 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.542+0000 7fc1af720640 1 --2- 172.21.15.161:0/2961970760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1a814f5c0 0x7fc1a80c2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.542+0000 7fc1af720640 1 -- 172.21.15.161:0/2961970760 >> 172.21.15.161:0/2961970760 conn(0x7fc1a80a83f0 msgr2=0x7fc1a80a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:18.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.542+0000 7fc1af720640 1 -- 172.21.15.161:0/2961970760 shutdown_connections 2024-03-28T23:32:18.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.542+0000 7fc1af720640 1 -- 172.21.15.161:0/2961970760 wait complete. 2024-03-28T23:32:18.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.543+0000 7fc1af720640 1 Processor -- start 2024-03-28T23:32:18.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.543+0000 7fc1af720640 1 -- start start 2024-03-28T23:32:18.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.544+0000 7fc1af720640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1a806cdd0 0x7fc1a814eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:18.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.544+0000 7fc1af720640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1a814f5c0 0x7fc1a81e9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:18.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.544+0000 7fc1af720640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1a8152bd0 0x7fc1a814add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:18.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.544+0000 7fc1af720640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a806c140 con 0x7fc1a814f5c0 2024-03-28T23:32:18.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.544+0000 7fc1af720640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a8152030 con 0x7fc1a806cdd0 2024-03-28T23:32:18.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.544+0000 7fc1af720640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc1a806c3e0 con 0x7fc1a8152bd0 2024-03-28T23:32:18.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.544+0000 7fc1accbb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1a814f5c0 0x7fc1a81e9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:18.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.544+0000 7fc1accbb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1a814f5c0 0x7fc1a81e9d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41670/0 (socket says 172.21.15.161:41670) 2024-03-28T23:32:18.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.544+0000 7fc1accbb640 1 -- 172.21.15.161:0/2427534612 learned_addr learned my addr 172.21.15.161:0/2427534612 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:18.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.544+0000 7fc1ad4bc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1a806cdd0 0x7fc1a814eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:18.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.544+0000 7fc1adcbd640 1 --2- 172.21.15.161:0/2427534612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1a8152bd0 0x7fc1a814add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:18.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.544+0000 7fc1accbb640 1 -- 172.21.15.161:0/2427534612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1a8152bd0 msgr2=0x7fc1a814add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:18.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.544+0000 7fc1accbb640 1 --2- 172.21.15.161:0/2427534612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1a8152bd0 0x7fc1a814add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.544+0000 7fc1accbb640 1 -- 172.21.15.161:0/2427534612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1a806cdd0 msgr2=0x7fc1a814eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:18.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.544+0000 7fc1accbb640 1 --2- 172.21.15.161:0/2427534612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1a806cdd0 0x7fc1a814eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.544+0000 7fc1accbb640 1 -- 172.21.15.161:0/2427534612 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1a8140110 con 0x7fc1a814f5c0 2024-03-28T23:32:18.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.544+0000 7fc1accbb640 1 --2- 172.21.15.161:0/2427534612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1a814f5c0 0x7fc1a81e9d00 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fc19800c780 tx=0x7fc19800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:18.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.545+0000 7fc19dffb640 1 -- 172.21.15.161:0/2427534612 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc198017070 con 0x7fc1a814f5c0 2024-03-28T23:32:18.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.545+0000 7fc19dffb640 1 -- 172.21.15.161:0/2427534612 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc198005070 con 0x7fc1a814f5c0 2024-03-28T23:32:18.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.545+0000 7fc19dffb640 1 -- 172.21.15.161:0/2427534612 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc198005460 con 0x7fc1a814f5c0 2024-03-28T23:32:18.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.545+0000 7fc1af720640 1 -- 172.21.15.161:0/2427534612 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1a81403f0 con 0x7fc1a814f5c0 2024-03-28T23:32:18.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.545+0000 7fc1af720640 1 -- 172.21.15.161:0/2427534612 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1a81e97e0 con 0x7fc1a814f5c0 2024-03-28T23:32:18.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.546+0000 7fc1af720640 1 -- 172.21.15.161:0/2427534612 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1a806e570 con 0x7fc1a814f5c0 2024-03-28T23:32:18.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.546+0000 7fc19dffb640 1 -- 172.21.15.161:0/2427534612 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc19801a070 con 0x7fc1a814f5c0 2024-03-28T23:32:18.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.546+0000 7fc19dffb640 1 --2- 172.21.15.161:0/2427534612 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc18c06b4d0 0x7fc18c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:18.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.546+0000 7fc19dffb640 1 -- 172.21.15.161:0/2427534612 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6449+0+0 (secure 0 0 0) 0x7fc198089700 con 0x7fc1a814f5c0 2024-03-28T23:32:18.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.546+0000 7fc1ad4bc640 1 --2- 172.21.15.161:0/2427534612 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc18c06b4d0 0x7fc18c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:18.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.547+0000 7fc1ad4bc640 1 --2- 172.21.15.161:0/2427534612 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc18c06b4d0 0x7fc18c06d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8152b10 tx=0x7fc1a4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:18.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.551+0000 7fc19dffb640 1 -- 172.21.15.161:0/2427534612 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc1a806e570 con 0x7fc1a814f5c0 2024-03-28T23:32:18.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.721+0000 7fc1af720640 1 -- 172.21.15.161:0/2427534612 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) -- 0x7fc1a804f4b0 con 0x7fc1a814f5c0 2024-03-28T23:32:18.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.722+0000 7fc19dffb640 1 -- 172.21.15.161:0/2427534612 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v61) ==== 105+0+11 (secure 0 0 0) 0x7fc1a80c2f10 con 0x7fc1a814f5c0 2024-03-28T23:32:18.722 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:32:18.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.723+0000 7fc1af720640 1 -- 172.21.15.161:0/2427534612 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc18c06b4d0 msgr2=0x7fc18c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:18.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.724+0000 7fc1af720640 1 --2- 172.21.15.161:0/2427534612 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc18c06b4d0 0x7fc18c06d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8152b10 tx=0x7fc1a4009190 comp rx=0 tx=0).stop 2024-03-28T23:32:18.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.724+0000 7fc1af720640 1 -- 172.21.15.161:0/2427534612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1a814f5c0 msgr2=0x7fc1a81e9d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:18.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.724+0000 7fc1af720640 1 --2- 172.21.15.161:0/2427534612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1a814f5c0 0x7fc1a81e9d00 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fc19800c780 tx=0x7fc19800cc40 comp rx=0 tx=0).stop 2024-03-28T23:32:18.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.724+0000 7fc1af720640 1 -- 172.21.15.161:0/2427534612 shutdown_connections 2024-03-28T23:32:18.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.724+0000 7fc1af720640 1 --2- 172.21.15.161:0/2427534612 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc18c06b4d0 0x7fc18c06d990 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.724+0000 7fc1af720640 1 --2- 172.21.15.161:0/2427534612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1a814f5c0 0x7fc1a81e9d00 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.724+0000 7fc1af720640 1 --2- 172.21.15.161:0/2427534612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1a8152bd0 0x7fc1a814add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.724+0000 7fc1af720640 1 --2- 172.21.15.161:0/2427534612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1a806cdd0 0x7fc1a814eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.724+0000 7fc1af720640 1 -- 172.21.15.161:0/2427534612 >> 172.21.15.161:0/2427534612 conn(0x7fc1a80a83f0 msgr2=0x7fc1a8151700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:18.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.724+0000 7fc1af720640 1 -- 172.21.15.161:0/2427534612 shutdown_connections 2024-03-28T23:32:18.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.724+0000 7fc1af720640 1 -- 172.21.15.161:0/2427534612 wait complete. 2024-03-28T23:32:18.735 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-03-28T23:32:18.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.840+0000 7f428f78c640 1 Processor -- start 2024-03-28T23:32:18.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.840+0000 7f428f78c640 1 -- start start 2024-03-28T23:32:18.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.841+0000 7f428f78c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4288151e70 0x7f4288152250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:18.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.841+0000 7f428f78c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4288152790 0x7f428806c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:18.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.841+0000 7f428f78c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f42880c6350 0x7f428806cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:18.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.841+0000 7f428f78c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f42880c1e40 con 0x7f4288152790 2024-03-28T23:32:18.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.841+0000 7f428f78c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f42880c2120 con 0x7f42880c6350 2024-03-28T23:32:18.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.841+0000 7f428f78c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f42880c2400 con 0x7f4288151e70 2024-03-28T23:32:18.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.841+0000 7f428cd27640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4288152790 0x7f428806c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:18.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.841+0000 7f428cd27640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4288152790 0x7f428806c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41672/0 (socket says 172.21.15.161:41672) 2024-03-28T23:32:18.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.841+0000 7f428cd27640 1 -- 172.21.15.161:0/1836601265 learned_addr learned my addr 172.21.15.161:0/1836601265 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:18.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.841+0000 7f428dd29640 1 --2- 172.21.15.161:0/1836601265 >> v2:172.21.15.170:3300/0 conn(0x7f42880c6350 0x7f428806cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:18.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.841+0000 7f428d528640 1 --2- 172.21.15.161:0/1836601265 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4288151e70 0x7f4288152250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:18.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.841+0000 7f428cd27640 1 -- 172.21.15.161:0/1836601265 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4288151e70 msgr2=0x7f4288152250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:18.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.841+0000 7f428cd27640 1 --2- 172.21.15.161:0/1836601265 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4288151e70 0x7f4288152250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.841+0000 7f428cd27640 1 -- 172.21.15.161:0/1836601265 >> v2:172.21.15.170:3300/0 conn(0x7f42880c6350 msgr2=0x7f428806cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:18.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.841+0000 7f428cd27640 1 --2- 172.21.15.161:0/1836601265 >> v2:172.21.15.170:3300/0 conn(0x7f42880c6350 0x7f428806cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.841+0000 7f428cd27640 1 -- 172.21.15.161:0/1836601265 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42880c26e0 con 0x7f4288152790 2024-03-28T23:32:18.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.842+0000 7f428cd27640 1 --2- 172.21.15.161:0/1836601265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4288152790 0x7f428806c6c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f42880c4aa0 tx=0x7f427802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa90a8c0743022f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:18.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.842+0000 7f427ffff640 1 -- 172.21.15.161:0/1836601265 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427802e070 con 0x7f4288152790 2024-03-28T23:32:18.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.842+0000 7f427ffff640 1 -- 172.21.15.161:0/1836601265 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4278037070 con 0x7f4288152790 2024-03-28T23:32:18.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.842+0000 7f427ffff640 1 -- 172.21.15.161:0/1836601265 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278031a00 con 0x7f4288152790 2024-03-28T23:32:18.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.843+0000 7f428f78c640 1 -- 172.21.15.161:0/1836601265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4288152790 msgr2=0x7f428806c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:18.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.843+0000 7f428f78c640 1 --2- 172.21.15.161:0/1836601265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4288152790 0x7f428806c6c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f42880c4aa0 tx=0x7f427802b880 comp rx=0 tx=0).stop 2024-03-28T23:32:18.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.843+0000 7f428f78c640 1 -- 172.21.15.161:0/1836601265 shutdown_connections 2024-03-28T23:32:18.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.843+0000 7f428f78c640 1 --2- 172.21.15.161:0/1836601265 >> v2:172.21.15.170:3300/0 conn(0x7f42880c6350 0x7f428806cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.843+0000 7f428f78c640 1 --2- 172.21.15.161:0/1836601265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4288152790 0x7f428806c6c0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.843+0000 7f428f78c640 1 --2- 172.21.15.161:0/1836601265 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4288151e70 0x7f4288152250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.843+0000 7f428f78c640 1 -- 172.21.15.161:0/1836601265 >> 172.21.15.161:0/1836601265 conn(0x7f42880c3390 msgr2=0x7f42880c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:18.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.843+0000 7f428f78c640 1 -- 172.21.15.161:0/1836601265 shutdown_connections 2024-03-28T23:32:18.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.843+0000 7f428f78c640 1 -- 172.21.15.161:0/1836601265 wait complete. 2024-03-28T23:32:18.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.844+0000 7f428f78c640 1 Processor -- start 2024-03-28T23:32:18.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.844+0000 7f428f78c640 1 -- start start 2024-03-28T23:32:18.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.844+0000 7f428f78c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42880c6350 0x7f4288150460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:18.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.844+0000 7f428f78c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4288151e70 0x7f42880a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:18.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.844+0000 7f428f78c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42881509a0 0x7f42881e7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:18.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.844+0000 7f428f78c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f42880c2d80 con 0x7f42880c6350 2024-03-28T23:32:18.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.844+0000 7f428f78c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f42880c2c00 con 0x7f4288151e70 2024-03-28T23:32:18.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.844+0000 7f428f78c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f42880a5fb0 con 0x7f42881509a0 2024-03-28T23:32:18.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.845+0000 7f428d528640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42880c6350 0x7f4288150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:18.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.845+0000 7f428d528640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42880c6350 0x7f4288150460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41674/0 (socket says 172.21.15.161:41674) 2024-03-28T23:32:18.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.845+0000 7f428d528640 1 -- 172.21.15.161:0/2780762293 learned_addr learned my addr 172.21.15.161:0/2780762293 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:18.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.845+0000 7f428cd27640 1 --2- 172.21.15.161:0/2780762293 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4288151e70 0x7f42880a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:18.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.845+0000 7f428dd29640 1 --2- 172.21.15.161:0/2780762293 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42881509a0 0x7f42881e7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:18.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.845+0000 7f428d528640 1 -- 172.21.15.161:0/2780762293 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42881509a0 msgr2=0x7f42881e7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:18.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.845+0000 7f428d528640 1 --2- 172.21.15.161:0/2780762293 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42881509a0 0x7f42881e7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.845+0000 7f428d528640 1 -- 172.21.15.161:0/2780762293 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4288151e70 msgr2=0x7f42880a75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:18.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.845+0000 7f428d528640 1 --2- 172.21.15.161:0/2780762293 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4288151e70 0x7f42880a75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:18.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.845+0000 7f428d528640 1 -- 172.21.15.161:0/2780762293 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42881e6d50 con 0x7f42880c6350 2024-03-28T23:32:18.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.845+0000 7f428d528640 1 --2- 172.21.15.161:0/2780762293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42880c6350 0x7f4288150460 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f428400e9e0 tx=0x7f428400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:18.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.845+0000 7f427dffb640 1 -- 172.21.15.161:0/2780762293 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4284017070 con 0x7f42880c6350 2024-03-28T23:32:18.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.845+0000 7f427dffb640 1 -- 172.21.15.161:0/2780762293 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f428400c970 con 0x7f42880c6350 2024-03-28T23:32:18.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.845+0000 7f428f78c640 1 -- 172.21.15.161:0/2780762293 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4288149ab0 con 0x7f42880c6350 2024-03-28T23:32:18.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.845+0000 7f427dffb640 1 -- 172.21.15.161:0/2780762293 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f428400cd60 con 0x7f42880c6350 2024-03-28T23:32:18.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.845+0000 7f428f78c640 1 -- 172.21.15.161:0/2780762293 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f428814ba00 con 0x7f42880c6350 2024-03-28T23:32:18.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.846+0000 7f428f78c640 1 -- 172.21.15.161:0/2780762293 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f428814fa20 con 0x7f42880c6350 2024-03-28T23:32:18.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.847+0000 7f427dffb640 1 -- 172.21.15.161:0/2780762293 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f428401a070 con 0x7f42880c6350 2024-03-28T23:32:18.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.847+0000 7f427dffb640 1 --2- 172.21.15.161:0/2780762293 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f426c06b4d0 0x7f426c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:18.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.847+0000 7f427dffb640 1 -- 172.21.15.161:0/2780762293 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6449+0+0 (secure 0 0 0) 0x7f428408a430 con 0x7f42880c6350 2024-03-28T23:32:18.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.847+0000 7f428cd27640 1 --2- 172.21.15.161:0/2780762293 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f426c06b4d0 0x7f426c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:18.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.848+0000 7f428cd27640 1 --2- 172.21.15.161:0/2780762293 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f426c06b4d0 0x7f426c06d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4278036000 tx=0x7f427803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:18.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:18.852+0000 7f427dffb640 1 -- 172.21.15.161:0/2780762293 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f428814fa20 con 0x7f42880c6350 2024-03-28T23:32:19.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.027+0000 7f428f78c640 1 -- 172.21.15.161:0/2780762293 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) -- 0x7f428804f4b0 con 0x7f42880c6350 2024-03-28T23:32:19.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.028+0000 7f427dffb640 1 -- 172.21.15.161:0/2780762293 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v61) ==== 101+0+11 (secure 0 0 0) 0x7f4284058970 con 0x7f42880c6350 2024-03-28T23:32:19.028 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:32:19.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.030+0000 7f428f78c640 1 -- 172.21.15.161:0/2780762293 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f426c06b4d0 msgr2=0x7f426c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:19.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.030+0000 7f428f78c640 1 --2- 172.21.15.161:0/2780762293 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f426c06b4d0 0x7f426c06d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4278036000 tx=0x7f427803d040 comp rx=0 tx=0).stop 2024-03-28T23:32:19.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.030+0000 7f428f78c640 1 -- 172.21.15.161:0/2780762293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42880c6350 msgr2=0x7f4288150460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:19.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.030+0000 7f428f78c640 1 --2- 172.21.15.161:0/2780762293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42880c6350 0x7f4288150460 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f428400e9e0 tx=0x7f428400eea0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.030+0000 7f428f78c640 1 -- 172.21.15.161:0/2780762293 shutdown_connections 2024-03-28T23:32:19.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.030+0000 7f428f78c640 1 --2- 172.21.15.161:0/2780762293 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f426c06b4d0 0x7f426c06d990 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.030+0000 7f428f78c640 1 --2- 172.21.15.161:0/2780762293 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42881509a0 0x7f42881e7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.030+0000 7f428f78c640 1 --2- 172.21.15.161:0/2780762293 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4288151e70 0x7f42880a75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.030+0000 7f428f78c640 1 --2- 172.21.15.161:0/2780762293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42880c6350 0x7f4288150460 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.030+0000 7f428f78c640 1 -- 172.21.15.161:0/2780762293 >> 172.21.15.161:0/2780762293 conn(0x7f42880c3390 msgr2=0x7f428806dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:19.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.030+0000 7f428f78c640 1 -- 172.21.15.161:0/2780762293 shutdown_connections 2024-03-28T23:32:19.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.030+0000 7f428f78c640 1 -- 172.21.15.161:0/2780762293 wait complete. 2024-03-28T23:32:19.041 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:32:19.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.141+0000 7f48dfb8c640 1 Processor -- start 2024-03-28T23:32:19.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.141+0000 7f48dfb8c640 1 -- start start 2024-03-28T23:32:19.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.142+0000 7f48dfb8c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f48d8150590 0x7f48d80a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:19.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.142+0000 7f48dfb8c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f48d80a8fc0 0x7f48d8153150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:19.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.142+0000 7f48dfb8c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48d80aacb0 0x7f48d8153690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:19.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.142+0000 7f48dfb8c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f48d81566f0 con 0x7f48d80a8fc0 2024-03-28T23:32:19.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.142+0000 7f48dfb8c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f48d81569d0 con 0x7f48d8150590 2024-03-28T23:32:19.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.142+0000 7f48dfb8c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f48d8156cb0 con 0x7f48d80aacb0 2024-03-28T23:32:19.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.142+0000 7f48dd127640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f48d80a8fc0 0x7f48d8153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:19.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.142+0000 7f48dd127640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f48d80a8fc0 0x7f48d8153150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41684/0 (socket says 172.21.15.161:41684) 2024-03-28T23:32:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.142+0000 7f48dd127640 1 -- 172.21.15.161:0/1028069957 learned_addr learned my addr 172.21.15.161:0/1028069957 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.142+0000 7f48dd928640 1 --2- 172.21.15.161:0/1028069957 >> v2:172.21.15.170:3300/0 conn(0x7f48d8150590 0x7f48d80a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.142+0000 7f48de129640 1 --2- 172.21.15.161:0/1028069957 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48d80aacb0 0x7f48d8153690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.142+0000 7f48dd127640 1 -- 172.21.15.161:0/1028069957 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48d80aacb0 msgr2=0x7f48d8153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.142+0000 7f48dd127640 1 --2- 172.21.15.161:0/1028069957 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48d80aacb0 0x7f48d8153690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.142+0000 7f48dd127640 1 -- 172.21.15.161:0/1028069957 >> v2:172.21.15.170:3300/0 conn(0x7f48d8150590 msgr2=0x7f48d80a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.142+0000 7f48dd127640 1 --2- 172.21.15.161:0/1028069957 >> v2:172.21.15.170:3300/0 conn(0x7f48d8150590 0x7f48d80a9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.142+0000 7f48dd127640 1 -- 172.21.15.161:0/1028069957 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d8156f90 con 0x7f48d80a8fc0 2024-03-28T23:32:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.142+0000 7f48de129640 1 --2- 172.21.15.161:0/1028069957 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48d80aacb0 0x7f48d8153690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:32:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.142+0000 7f48dd928640 1 --2- 172.21.15.161:0/1028069957 >> v2:172.21.15.170:3300/0 conn(0x7f48d8150590 0x7f48d80a9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:32:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.143+0000 7f48dd127640 1 --2- 172.21.15.161:0/1028069957 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48d80a8fc0 0x7f48d8153150 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f48d80c4210 tx=0x7f48c802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfd9722eba4be091 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.143+0000 7f48dc926640 1 -- 172.21.15.161:0/1028069957 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c8038070 con 0x7f48d80a8fc0 2024-03-28T23:32:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.143+0000 7f48dc926640 1 -- 172.21.15.161:0/1028069957 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48c8031740 con 0x7f48d80a8fc0 2024-03-28T23:32:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.143+0000 7f48dc926640 1 -- 172.21.15.161:0/1028069957 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c802f050 con 0x7f48d80a8fc0 2024-03-28T23:32:19.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.143+0000 7f48dfb8c640 1 -- 172.21.15.161:0/1028069957 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48d80a8fc0 msgr2=0x7f48d8153150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:19.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.143+0000 7f48dfb8c640 1 --2- 172.21.15.161:0/1028069957 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48d80a8fc0 0x7f48d8153150 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f48d80c4210 tx=0x7f48c802b7e0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.143+0000 7f48dfb8c640 1 -- 172.21.15.161:0/1028069957 shutdown_connections 2024-03-28T23:32:19.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.143+0000 7f48dfb8c640 1 --2- 172.21.15.161:0/1028069957 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48d80aacb0 0x7f48d8153690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.143+0000 7f48dfb8c640 1 --2- 172.21.15.161:0/1028069957 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48d80a8fc0 0x7f48d8153150 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.143+0000 7f48dfb8c640 1 --2- 172.21.15.161:0/1028069957 >> v2:172.21.15.170:3300/0 conn(0x7f48d8150590 0x7f48d80a9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.143+0000 7f48dfb8c640 1 -- 172.21.15.161:0/1028069957 >> 172.21.15.161:0/1028069957 conn(0x7f48d80c6670 msgr2=0x7f48d80c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:19.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.144+0000 7f48dfb8c640 1 -- 172.21.15.161:0/1028069957 shutdown_connections 2024-03-28T23:32:19.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.144+0000 7f48dfb8c640 1 -- 172.21.15.161:0/1028069957 wait complete. 2024-03-28T23:32:19.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.145+0000 7f48dfb8c640 1 Processor -- start 2024-03-28T23:32:19.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.145+0000 7f48dfb8c640 1 -- start start 2024-03-28T23:32:19.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.145+0000 7f48dfb8c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48d80a8fc0 0x7f48d81e8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:19.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.145+0000 7f48dfb8c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48d80aacb0 0x7f48d81523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:19.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.145+0000 7f48dfb8c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48d8150590 0x7f48d81532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:19.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.145+0000 7f48dfb8c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f48d8153bd0 con 0x7f48d80aacb0 2024-03-28T23:32:19.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.145+0000 7f48dfb8c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f48d8150970 con 0x7f48d8150590 2024-03-28T23:32:19.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.145+0000 7f48dfb8c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f48d8153d50 con 0x7f48d80a8fc0 2024-03-28T23:32:19.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.145+0000 7f48dd127640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48d80aacb0 0x7f48d81523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:19.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.145+0000 7f48dd127640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48d80aacb0 0x7f48d81523a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41686/0 (socket says 172.21.15.161:41686) 2024-03-28T23:32:19.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.145+0000 7f48dd127640 1 -- 172.21.15.161:0/3311346199 learned_addr learned my addr 172.21.15.161:0/3311346199 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:19.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.145+0000 7f48dd928640 1 --2- 172.21.15.161:0/3311346199 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48d80a8fc0 0x7f48d81e8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:19.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.145+0000 7f48de129640 1 --2- 172.21.15.161:0/3311346199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48d8150590 0x7f48d81532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:19.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.146+0000 7f48dd127640 1 -- 172.21.15.161:0/3311346199 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48d80a8fc0 msgr2=0x7f48d81e8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:19.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.146+0000 7f48dd127640 1 --2- 172.21.15.161:0/3311346199 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48d80a8fc0 0x7f48d81e8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.146+0000 7f48dd127640 1 -- 172.21.15.161:0/3311346199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48d8150590 msgr2=0x7f48d81532c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:19.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.146+0000 7f48dd127640 1 --2- 172.21.15.161:0/3311346199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48d8150590 0x7f48d81532c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.146+0000 7f48dd127640 1 -- 172.21.15.161:0/3311346199 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d80c3da0 con 0x7f48d80aacb0 2024-03-28T23:32:19.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.146+0000 7f48dd127640 1 --2- 172.21.15.161:0/3311346199 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48d80aacb0 0x7f48d81523a0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f48c802b820 tx=0x7f48c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:19.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.146+0000 7f48ce7fc640 1 -- 172.21.15.161:0/3311346199 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c8042070 con 0x7f48d80aacb0 2024-03-28T23:32:19.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.146+0000 7f48ce7fc640 1 -- 172.21.15.161:0/3311346199 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48c803d870 con 0x7f48d80aacb0 2024-03-28T23:32:19.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.146+0000 7f48dfb8c640 1 -- 172.21.15.161:0/3311346199 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48d8151300 con 0x7f48d80aacb0 2024-03-28T23:32:19.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.146+0000 7f48ce7fc640 1 -- 172.21.15.161:0/3311346199 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c8038040 con 0x7f48d80aacb0 2024-03-28T23:32:19.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.146+0000 7f48dfb8c640 1 -- 172.21.15.161:0/3311346199 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48d81518c0 con 0x7f48d80aacb0 2024-03-28T23:32:19.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.147+0000 7f48dfb8c640 1 -- 172.21.15.161:0/3311346199 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48d80c0a90 con 0x7f48d80aacb0 2024-03-28T23:32:19.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.148+0000 7f48ce7fc640 1 -- 172.21.15.161:0/3311346199 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f48c8045070 con 0x7f48d80aacb0 2024-03-28T23:32:19.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.148+0000 7f48ce7fc640 1 --2- 172.21.15.161:0/3311346199 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f48bc06b500 0x7f48bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:19.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.148+0000 7f48ce7fc640 1 -- 172.21.15.161:0/3311346199 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6449+0+0 (secure 0 0 0) 0x7f48c80abe40 con 0x7f48d80aacb0 2024-03-28T23:32:19.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.148+0000 7f48dd928640 1 --2- 172.21.15.161:0/3311346199 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f48bc06b500 0x7f48bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:19.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.148+0000 7f48dd928640 1 --2- 172.21.15.161:0/3311346199 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f48bc06b500 0x7f48bc06d9c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f48d4009560 tx=0x7f48d4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:19.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.153+0000 7f48ce7fc640 1 -- 172.21.15.161:0/3311346199 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f48d80c0a90 con 0x7f48d80aacb0 2024-03-28T23:32:19.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.363+0000 7f48dfb8c640 1 -- 172.21.15.161:0/3311346199 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48d804f4b0 con 0x7f48d80aacb0 2024-03-28T23:32:19.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.363+0000 7f48ce7fc640 1 -- 172.21.15.161:0/3311346199 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+5194 (secure 0 0 0) 0x7f48c804c030 con 0x7f48d80aacb0 2024-03-28T23:32:19.364 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:19.364 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":49,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5542,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.170:6833/3666899516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3666899516},{"type":"v1","addr":"172.21.15.170:6833","nonce":3666899516}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":6273,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/3350861737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3350861737},{"type":"v1","addr":"172.21.15.161:6837","nonce":3350861737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":6288,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/2929859479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":2929859479},{"type":"v1","addr":"172.21.15.161:6836","nonce":2929859479}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:31:54.226424+0000","modified":"2024-03-28T23:32:18.788201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5390":{"gid":5390,"name":"b","rank":0,"incarnation":46,"state":"up:active","state_seq":6,"addr":"172.21.15.170:6835/2649192649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2649192649},{"type":"v1","addr":"172.21.15.170:6835","nonce":2649192649}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5390,"qdb_cluster":[5390]},"id":4}]} 2024-03-28T23:32:19.365 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 49 2024-03-28T23:32:19.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.366+0000 7f48dfb8c640 1 -- 172.21.15.161:0/3311346199 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f48bc06b500 msgr2=0x7f48bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:19.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.366+0000 7f48dfb8c640 1 --2- 172.21.15.161:0/3311346199 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f48bc06b500 0x7f48bc06d9c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f48d4009560 tx=0x7f48d4009190 comp rx=0 tx=0).stop 2024-03-28T23:32:19.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.366+0000 7f48dfb8c640 1 -- 172.21.15.161:0/3311346199 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48d80aacb0 msgr2=0x7f48d81523a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:19.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.366+0000 7f48dfb8c640 1 --2- 172.21.15.161:0/3311346199 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48d80aacb0 0x7f48d81523a0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f48c802b820 tx=0x7f48c80046c0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.366+0000 7f48dfb8c640 1 -- 172.21.15.161:0/3311346199 shutdown_connections 2024-03-28T23:32:19.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.366+0000 7f48dfb8c640 1 --2- 172.21.15.161:0/3311346199 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f48bc06b500 0x7f48bc06d9c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.366+0000 7f48dfb8c640 1 --2- 172.21.15.161:0/3311346199 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48d80aacb0 0x7f48d81523a0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.366+0000 7f48dfb8c640 1 --2- 172.21.15.161:0/3311346199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48d8150590 0x7f48d81532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.366+0000 7f48dfb8c640 1 --2- 172.21.15.161:0/3311346199 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48d80a8fc0 0x7f48d81e8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.366+0000 7f48dfb8c640 1 -- 172.21.15.161:0/3311346199 >> 172.21.15.161:0/3311346199 conn(0x7f48d80c6670 msgr2=0x7f48d814ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:19.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.366+0000 7f48dfb8c640 1 -- 172.21.15.161:0/3311346199 shutdown_connections 2024-03-28T23:32:19.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.366+0000 7f48dfb8c640 1 -- 172.21.15.161:0/3311346199 wait complete. 2024-03-28T23:32:19.377 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:32:19.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.477+0000 7f2f1c6d4640 1 Processor -- start 2024-03-28T23:32:19.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.477+0000 7f2f1c6d4640 1 -- start start 2024-03-28T23:32:19.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.478+0000 7f2f1c6d4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f1414be70 0x7f2f140ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:19.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.478+0000 7f2f1c6d4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2f140ab6e0 0x7f2f140aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:19.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.478+0000 7f2f1c6d4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2f1406ce80 0x7f2f140aab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:19.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.478+0000 7f2f1c6d4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2f1414c250 con 0x7f2f140ab6e0 2024-03-28T23:32:19.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.478+0000 7f2f1c6d4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f2f141529c0 con 0x7f2f1406ce80 2024-03-28T23:32:19.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.478+0000 7f2f1c6d4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2f140a9dc0 con 0x7f2f1414be70 2024-03-28T23:32:19.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.478+0000 7f2f19c6f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2f140ab6e0 0x7f2f140aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:19.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.478+0000 7f2f19c6f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2f140ab6e0 0x7f2f140aa5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41698/0 (socket says 172.21.15.161:41698) 2024-03-28T23:32:19.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.478+0000 7f2f19c6f640 1 -- 172.21.15.161:0/3708652661 learned_addr learned my addr 172.21.15.161:0/3708652661 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:19.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.478+0000 7f2f1a470640 1 --2- 172.21.15.161:0/3708652661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f1414be70 0x7f2f140ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:19.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.478+0000 7f2f1ac71640 1 --2- 172.21.15.161:0/3708652661 >> v2:172.21.15.170:3300/0 conn(0x7f2f1406ce80 0x7f2f140aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:19.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.479+0000 7f2f19c6f640 1 -- 172.21.15.161:0/3708652661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f1414be70 msgr2=0x7f2f140ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:19.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.479+0000 7f2f19c6f640 1 --2- 172.21.15.161:0/3708652661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f1414be70 0x7f2f140ab1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.479+0000 7f2f19c6f640 1 -- 172.21.15.161:0/3708652661 >> v2:172.21.15.170:3300/0 conn(0x7f2f1406ce80 msgr2=0x7f2f140aab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:19.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.479+0000 7f2f19c6f640 1 --2- 172.21.15.161:0/3708652661 >> v2:172.21.15.170:3300/0 conn(0x7f2f1406ce80 0x7f2f140aab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.479+0000 7f2f19c6f640 1 -- 172.21.15.161:0/3708652661 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f140aa0a0 con 0x7f2f140ab6e0 2024-03-28T23:32:19.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.479+0000 7f2f1a470640 1 --2- 172.21.15.161:0/3708652661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f1414be70 0x7f2f140ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:32:19.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.479+0000 7f2f19c6f640 1 --2- 172.21.15.161:0/3708652661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f140ab6e0 0x7f2f140aa5c0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f2f140c3020 tx=0x7f2f0402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bd99ced8835956e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:19.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.479+0000 7f2f1946e640 1 -- 172.21.15.161:0/3708652661 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f04038070 con 0x7f2f140ab6e0 2024-03-28T23:32:19.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.479+0000 7f2f1946e640 1 -- 172.21.15.161:0/3708652661 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f0402f050 con 0x7f2f140ab6e0 2024-03-28T23:32:19.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.480+0000 7f2f1946e640 1 -- 172.21.15.161:0/3708652661 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f04031970 con 0x7f2f140ab6e0 2024-03-28T23:32:19.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.480+0000 7f2f1c6d4640 1 -- 172.21.15.161:0/3708652661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f140ab6e0 msgr2=0x7f2f140aa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:19.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.480+0000 7f2f1c6d4640 1 --2- 172.21.15.161:0/3708652661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f140ab6e0 0x7f2f140aa5c0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f2f140c3020 tx=0x7f2f0402b7f0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.480+0000 7f2f1c6d4640 1 -- 172.21.15.161:0/3708652661 shutdown_connections 2024-03-28T23:32:19.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.480+0000 7f2f1c6d4640 1 --2- 172.21.15.161:0/3708652661 >> v2:172.21.15.170:3300/0 conn(0x7f2f1406ce80 0x7f2f140aab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.480+0000 7f2f1c6d4640 1 --2- 172.21.15.161:0/3708652661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f140ab6e0 0x7f2f140aa5c0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.480+0000 7f2f1c6d4640 1 --2- 172.21.15.161:0/3708652661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f1414be70 0x7f2f140ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.480+0000 7f2f1c6d4640 1 -- 172.21.15.161:0/3708652661 >> 172.21.15.161:0/3708652661 conn(0x7f2f140c6670 msgr2=0x7f2f140c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:19.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.480+0000 7f2f1c6d4640 1 -- 172.21.15.161:0/3708652661 shutdown_connections 2024-03-28T23:32:19.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.480+0000 7f2f1c6d4640 1 -- 172.21.15.161:0/3708652661 wait complete. 2024-03-28T23:32:19.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.481+0000 7f2f1c6d4640 1 Processor -- start 2024-03-28T23:32:19.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.481+0000 7f2f1c6d4640 1 -- start start 2024-03-28T23:32:19.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.482+0000 7f2f1c6d4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f1406ce80 0x7f2f14072840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:19.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.482+0000 7f2f1c6d4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f140ab6e0 0x7f2f14153510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:19.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.482+0000 7f2f1c6d4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f1414be70 0x7f2f14153a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:19.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.482+0000 7f2f1c6d4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1406b2d0 con 0x7f2f140ab6e0 2024-03-28T23:32:19.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.482+0000 7f2f1c6d4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1406b150 con 0x7f2f1406ce80 2024-03-28T23:32:19.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.482+0000 7f2f1c6d4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2f140c1520 con 0x7f2f1414be70 2024-03-28T23:32:19.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.482+0000 7f2f19c6f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f140ab6e0 0x7f2f14153510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:19.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.482+0000 7f2f19c6f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f140ab6e0 0x7f2f14153510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41704/0 (socket says 172.21.15.161:41704) 2024-03-28T23:32:19.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.482+0000 7f2f19c6f640 1 -- 172.21.15.161:0/3344826855 learned_addr learned my addr 172.21.15.161:0/3344826855 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:19.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.482+0000 7f2f1a470640 1 --2- 172.21.15.161:0/3344826855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f1406ce80 0x7f2f14072840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:19.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.482+0000 7f2f1ac71640 1 --2- 172.21.15.161:0/3344826855 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f1414be70 0x7f2f14153a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:19.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.482+0000 7f2f19c6f640 1 -- 172.21.15.161:0/3344826855 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f1414be70 msgr2=0x7f2f14153a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:19.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.482+0000 7f2f19c6f640 1 --2- 172.21.15.161:0/3344826855 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f1414be70 0x7f2f14153a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.482+0000 7f2f19c6f640 1 -- 172.21.15.161:0/3344826855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f1406ce80 msgr2=0x7f2f14072840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:19.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.482+0000 7f2f19c6f640 1 --2- 172.21.15.161:0/3344826855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f1406ce80 0x7f2f14072840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.482+0000 7f2f19c6f640 1 -- 172.21.15.161:0/3344826855 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f1414f760 con 0x7f2f140ab6e0 2024-03-28T23:32:19.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.482+0000 7f2f1a470640 1 --2- 172.21.15.161:0/3344826855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f1406ce80 0x7f2f14072840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:32:19.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.482+0000 7f2f19c6f640 1 --2- 172.21.15.161:0/3344826855 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f140ab6e0 0x7f2f14153510 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f2f04009960 tx=0x7f2f040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:19.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.483+0000 7f2f0affd640 1 -- 172.21.15.161:0/3344826855 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f04042020 con 0x7f2f140ab6e0 2024-03-28T23:32:19.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.483+0000 7f2f0affd640 1 -- 172.21.15.161:0/3344826855 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f0403d680 con 0x7f2f140ab6e0 2024-03-28T23:32:19.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.483+0000 7f2f0affd640 1 -- 172.21.15.161:0/3344826855 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f04038040 con 0x7f2f140ab6e0 2024-03-28T23:32:19.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.483+0000 7f2f1c6d4640 1 -- 172.21.15.161:0/3344826855 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f1414d4c0 con 0x7f2f140ab6e0 2024-03-28T23:32:19.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.483+0000 7f2f1c6d4640 1 -- 172.21.15.161:0/3344826855 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f1414da80 con 0x7f2f140ab6e0 2024-03-28T23:32:19.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.484+0000 7f2f1c6d4640 1 -- 172.21.15.161:0/3344826855 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f140ab120 con 0x7f2f140ab6e0 2024-03-28T23:32:19.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.484+0000 7f2f0affd640 1 -- 172.21.15.161:0/3344826855 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2f04045020 con 0x7f2f140ab6e0 2024-03-28T23:32:19.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.484+0000 7f2f0affd640 1 --2- 172.21.15.161:0/3344826855 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2ef806b4d0 0x7f2ef806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:19.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.485+0000 7f2f0affd640 1 -- 172.21.15.161:0/3344826855 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6449+0+0 (secure 0 0 0) 0x7f2f040abe50 con 0x7f2f140ab6e0 2024-03-28T23:32:19.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.485+0000 7f2f1a470640 1 --2- 172.21.15.161:0/3344826855 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2ef806b4d0 0x7f2ef806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:19.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.485+0000 7f2f1a470640 1 --2- 172.21.15.161:0/3344826855 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2ef806b4d0 0x7f2ef806d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2f10009fd0 tx=0x7f2f1000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:19.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.489+0000 7f2f0affd640 1 -- 172.21.15.161:0/3344826855 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2f0404c020 con 0x7f2f140ab6e0 2024-03-28T23:32:19.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.662+0000 7f2f1c6d4640 1 -- 172.21.15.161:0/3344826855 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2f1404f4b0 con 0x7f2f140ab6e0 2024-03-28T23:32:19.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.663+0000 7f2f0affd640 1 -- 172.21.15.161:0/3344826855 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) ==== 74+0+17449 (secure 0 0 0) 0x7f2f0407a390 con 0x7f2f140ab6e0 2024-03-28T23:32:19.664 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:19.664 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":61,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:32:17.787428+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:31:52.774315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:31:53.201664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6836/1300583128":"2024-03-29T23:32:15.775261+0000","172.21.15.161:6834/1300583128":"2024-03-29T23:32:15.775261+0000","172.21.15.161:6837/1498528126":"2024-03-29T23:32:14.767288+0000","172.21.15.170:6835/1945122204":"2024-03-29T23:32:06.533661+0000","172.21.15.161:6835/1498528126":"2024-03-29T23:32:14.767288+0000","172.21.15.170:6834/1945122204":"2024-03-29T23:32:06.533661+0000","172.21.15.161:6837/544971001":"2024-03-29T23:32:03.518509+0000","172.21.15.161:6835/544971001":"2024-03-29T23:32:03.518509+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:32:19.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.665+0000 7f2f1c6d4640 1 -- 172.21.15.161:0/3344826855 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2ef806b4d0 msgr2=0x7f2ef806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:19.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.665+0000 7f2f1c6d4640 1 --2- 172.21.15.161:0/3344826855 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2ef806b4d0 0x7f2ef806d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2f10009fd0 tx=0x7f2f1000b040 comp rx=0 tx=0).stop 2024-03-28T23:32:19.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.665+0000 7f2f1c6d4640 1 -- 172.21.15.161:0/3344826855 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f140ab6e0 msgr2=0x7f2f14153510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:19.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.665+0000 7f2f1c6d4640 1 --2- 172.21.15.161:0/3344826855 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f140ab6e0 0x7f2f14153510 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f2f04009960 tx=0x7f2f040046c0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.665+0000 7f2f1c6d4640 1 -- 172.21.15.161:0/3344826855 shutdown_connections 2024-03-28T23:32:19.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.665+0000 7f2f1c6d4640 1 --2- 172.21.15.161:0/3344826855 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2ef806b4d0 0x7f2ef806d990 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.665+0000 7f2f1c6d4640 1 --2- 172.21.15.161:0/3344826855 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f140ab6e0 0x7f2f14153510 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.665+0000 7f2f1c6d4640 1 --2- 172.21.15.161:0/3344826855 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f1414be70 0x7f2f14153a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.665+0000 7f2f1c6d4640 1 --2- 172.21.15.161:0/3344826855 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f1406ce80 0x7f2f14072840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.665+0000 7f2f1c6d4640 1 -- 172.21.15.161:0/3344826855 >> 172.21.15.161:0/3344826855 conn(0x7f2f140c6670 msgr2=0x7f2f1415bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:19.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.665+0000 7f2f1c6d4640 1 -- 172.21.15.161:0/3344826855 shutdown_connections 2024-03-28T23:32:19.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.665+0000 7f2f1c6d4640 1 -- 172.21.15.161:0/3344826855 wait complete. 2024-03-28T23:32:19.677 INFO:tasks.cephfs.filesystem:Destroying file system cephfs2 and related pools 2024-03-28T23:32:19.677 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:32:19.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.778+0000 7f8b035d0640 1 Processor -- start 2024-03-28T23:32:19.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.778+0000 7f8b035d0640 1 -- start start 2024-03-28T23:32:19.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.779+0000 7f8b035d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8afc150600 0x7f8afc0a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:19.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.779+0000 7f8b035d0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8afc1567c0 con 0x7f8afc0a8fc0 2024-03-28T23:32:19.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.779+0000 7f8b035d0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8afc156aa0 con 0x7f8afc0aacb0 2024-03-28T23:32:19.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.779+0000 7f8b035d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8afc156570 con 0x7f8afc150600 2024-03-28T23:32:19.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.779+0000 7f8b0136c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8afc150600 0x7f8afc0a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:19.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.779+0000 7f8b0136c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8afc150600 0x7f8afc0a9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:42820/0 (socket says 172.21.15.161:42820) 2024-03-28T23:32:19.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.779+0000 7f8b00b6b640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f8afc0a8fc0 0x7f8afc153270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:53914/0 (socket says 172.21.15.161:53914) 2024-03-28T23:32:19.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.779+0000 7f8b0136c640 1 -- 172.21.15.161:0/3241930266 learned_addr learned my addr 172.21.15.161:0/3241930266 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:19.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.780+0000 7f8b0136c640 1 -- 172.21.15.161:0/3241930266 >> v1:172.21.15.170:6789/0 conn(0x7f8afc0aacb0 legacy=0x7f8afc153980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:19.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.780+0000 7f8b0136c640 1 -- 172.21.15.161:0/3241930266 >> v1:172.21.15.161:6789/0 conn(0x7f8afc0a8fc0 legacy=0x7f8afc153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:19.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.780+0000 7f8af3fff640 1 -- 172.21.15.161:0/3241930266 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3901028533 0 0) 0x7f8afc156aa0 con 0x7f8afc0aacb0 2024-03-28T23:32:19.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.780+0000 7f8b0136c640 1 -- 172.21.15.161:0/3241930266 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8afc156d80 con 0x7f8afc150600 2024-03-28T23:32:19.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.780+0000 7f8b0136c640 1 --2- 172.21.15.161:0/3241930266 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8afc150600 0x7f8afc0a9f40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8afc1566f0 tx=0x7f8af802b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18b8329e67261853 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:19.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.781+0000 7f8af3fff640 1 -- 172.21.15.161:0/3241930266 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af8037070 con 0x7f8afc150600 2024-03-28T23:32:19.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.781+0000 7f8af3fff640 1 -- 172.21.15.161:0/3241930266 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8af8030710 con 0x7f8afc150600 2024-03-28T23:32:19.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.781+0000 7f8af3fff640 1 -- 172.21.15.161:0/3241930266 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af8030ae0 con 0x7f8afc150600 2024-03-28T23:32:19.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.781+0000 7f8b035d0640 1 -- 172.21.15.161:0/3241930266 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8afc150600 msgr2=0x7f8afc0a9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:19.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.781+0000 7f8b035d0640 1 --2- 172.21.15.161:0/3241930266 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8afc150600 0x7f8afc0a9f40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8afc1566f0 tx=0x7f8af802b730 comp rx=0 tx=0).stop 2024-03-28T23:32:19.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.781+0000 7f8b035d0640 1 -- 172.21.15.161:0/3241930266 shutdown_connections 2024-03-28T23:32:19.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.781+0000 7f8b035d0640 1 --2- 172.21.15.161:0/3241930266 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8afc150600 0x7f8afc0a9f40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.781+0000 7f8b035d0640 1 -- 172.21.15.161:0/3241930266 >> 172.21.15.161:0/3241930266 conn(0x7f8afc0c6670 msgr2=0x7f8afc0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:19.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.781+0000 7f8b035d0640 1 -- 172.21.15.161:0/3241930266 shutdown_connections 2024-03-28T23:32:19.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.781+0000 7f8b035d0640 1 -- 172.21.15.161:0/3241930266 wait complete. 2024-03-28T23:32:19.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.782+0000 7f8b035d0640 1 Processor -- start 2024-03-28T23:32:19.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.782+0000 7f8b035d0640 1 -- start start 2024-03-28T23:32:19.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.783+0000 7f8b035d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8afc0a8fc0 0x7f8afc0c6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:19.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.783+0000 7f8b035d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8afc0aacb0 0x7f8afc155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:19.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.783+0000 7f8b035d0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8afc150600 0x7f8afc0c4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:19.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.783+0000 7f8b035d0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc1509e0 con 0x7f8afc150600 2024-03-28T23:32:19.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.783+0000 7f8b035d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc157340 con 0x7f8afc0a8fc0 2024-03-28T23:32:19.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.783+0000 7f8b035d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8afc0aa9e0 con 0x7f8afc0aacb0 2024-03-28T23:32:19.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.783+0000 7f8b01b6d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8afc150600 0x7f8afc0c4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:19.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.783+0000 7f8b01b6d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8afc150600 0x7f8afc0c4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41716/0 (socket says 172.21.15.161:41716) 2024-03-28T23:32:19.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.783+0000 7f8b01b6d640 1 -- 172.21.15.161:0/1280512867 learned_addr learned my addr 172.21.15.161:0/1280512867 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:19.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.783+0000 7f8b00b6b640 1 --2- 172.21.15.161:0/1280512867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8afc0aacb0 0x7f8afc155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:19.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.783+0000 7f8b0136c640 1 --2- 172.21.15.161:0/1280512867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8afc0a8fc0 0x7f8afc0c6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:19.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.783+0000 7f8b01b6d640 1 -- 172.21.15.161:0/1280512867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8afc0aacb0 msgr2=0x7f8afc155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:19.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.783+0000 7f8b01b6d640 1 --2- 172.21.15.161:0/1280512867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8afc0aacb0 0x7f8afc155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.783+0000 7f8b01b6d640 1 -- 172.21.15.161:0/1280512867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8afc0a8fc0 msgr2=0x7f8afc0c6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:19.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.783+0000 7f8b01b6d640 1 --2- 172.21.15.161:0/1280512867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8afc0a8fc0 0x7f8afc0c6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:19.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.783+0000 7f8b01b6d640 1 -- 172.21.15.161:0/1280512867 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8afc14cb30 con 0x7f8afc150600 2024-03-28T23:32:19.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.784+0000 7f8b01b6d640 1 --2- 172.21.15.161:0/1280512867 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8afc150600 0x7f8afc0c4720 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f8af400c820 tx=0x7f8af400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:19.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.784+0000 7f8af1ffb640 1 -- 172.21.15.161:0/1280512867 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af4017070 con 0x7f8afc150600 2024-03-28T23:32:19.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.784+0000 7f8af1ffb640 1 -- 172.21.15.161:0/1280512867 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8af4003dc0 con 0x7f8afc150600 2024-03-28T23:32:19.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.784+0000 7f8af1ffb640 1 -- 172.21.15.161:0/1280512867 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af40052b0 con 0x7f8afc150600 2024-03-28T23:32:19.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.784+0000 7f8b035d0640 1 -- 172.21.15.161:0/1280512867 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8afc14ce10 con 0x7f8afc150600 2024-03-28T23:32:19.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.784+0000 7f8b035d0640 1 -- 172.21.15.161:0/1280512867 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8afc1521b0 con 0x7f8afc150600 2024-03-28T23:32:19.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.785+0000 7f8b035d0640 1 -- 172.21.15.161:0/1280512867 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8afc04f4b0 con 0x7f8afc150600 2024-03-28T23:32:19.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.785+0000 7f8af1ffb640 1 -- 172.21.15.161:0/1280512867 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8af401a070 con 0x7f8afc150600 2024-03-28T23:32:19.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.786+0000 7f8af1ffb640 1 --2- 172.21.15.161:0/1280512867 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8ae006b4d0 0x7f8ae006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:19.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.786+0000 7f8af1ffb640 1 -- 172.21.15.161:0/1280512867 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6449+0+0 (secure 0 0 0) 0x7f8af4089a50 con 0x7f8afc150600 2024-03-28T23:32:19.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.786+0000 7f8b0136c640 1 --2- 172.21.15.161:0/1280512867 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8ae006b4d0 0x7f8ae006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:19.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.786+0000 7f8b0136c640 1 --2- 172.21.15.161:0/1280512867 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8ae006b4d0 0x7f8ae006d990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8af8034000 tx=0x7f8af8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:19.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:19.790+0000 7f8af1ffb640 1 -- 172.21.15.161:0/1280512867 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8af4058020 con 0x7f8afc150600 2024-03-28T23:32:20.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.000+0000 7f8b035d0640 1 -- 172.21.15.161:0/1280512867 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8afc0a9d20 con 0x7f8afc150600 2024-03-28T23:32:20.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.000+0000 7f8af1ffb640 1 -- 172.21.15.161:0/1280512867 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+5194 (secure 0 0 0) 0x7f8af405c0c0 con 0x7f8afc150600 2024-03-28T23:32:20.001 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:20.002 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":49,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5542,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.170:6833/3666899516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3666899516},{"type":"v1","addr":"172.21.15.170:6833","nonce":3666899516}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":6273,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/3350861737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3350861737},{"type":"v1","addr":"172.21.15.161:6837","nonce":3350861737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":6288,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/2929859479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":2929859479},{"type":"v1","addr":"172.21.15.161:6836","nonce":2929859479}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:31:54.226424+0000","modified":"2024-03-28T23:32:18.788201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5390":{"gid":5390,"name":"b","rank":0,"incarnation":46,"state":"up:active","state_seq":6,"addr":"172.21.15.170:6835/2649192649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2649192649},{"type":"v1","addr":"172.21.15.170:6835","nonce":2649192649}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5390,"qdb_cluster":[5390]},"id":4}]} 2024-03-28T23:32:20.002 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 49 2024-03-28T23:32:20.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.003+0000 7f8b035d0640 1 -- 172.21.15.161:0/1280512867 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8ae006b4d0 msgr2=0x7f8ae006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:20.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.003+0000 7f8b035d0640 1 --2- 172.21.15.161:0/1280512867 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8ae006b4d0 0x7f8ae006d990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8af8034000 tx=0x7f8af8008030 comp rx=0 tx=0).stop 2024-03-28T23:32:20.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.003+0000 7f8b035d0640 1 -- 172.21.15.161:0/1280512867 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8afc150600 msgr2=0x7f8afc0c4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:20.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.003+0000 7f8b035d0640 1 --2- 172.21.15.161:0/1280512867 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8afc150600 0x7f8afc0c4720 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f8af400c820 tx=0x7f8af400cce0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.003+0000 7f8b035d0640 1 -- 172.21.15.161:0/1280512867 shutdown_connections 2024-03-28T23:32:20.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.003+0000 7f8b035d0640 1 --2- 172.21.15.161:0/1280512867 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8ae006b4d0 0x7f8ae006d990 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.003+0000 7f8b035d0640 1 --2- 172.21.15.161:0/1280512867 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8afc150600 0x7f8afc0c4720 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.003+0000 7f8b035d0640 1 --2- 172.21.15.161:0/1280512867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8afc0aacb0 0x7f8afc155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.003+0000 7f8b035d0640 1 --2- 172.21.15.161:0/1280512867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8afc0a8fc0 0x7f8afc0c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.003+0000 7f8b035d0640 1 -- 172.21.15.161:0/1280512867 >> 172.21.15.161:0/1280512867 conn(0x7f8afc0c6670 msgr2=0x7f8afc074020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:20.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.003+0000 7f8b035d0640 1 -- 172.21.15.161:0/1280512867 shutdown_connections 2024-03-28T23:32:20.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.003+0000 7f8b035d0640 1 -- 172.21.15.161:0/1280512867 wait complete. 2024-03-28T23:32:20.014 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:32:20.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.114+0000 7f11e9e64640 1 Processor -- start 2024-03-28T23:32:20.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.114+0000 7f11e9e64640 1 -- start start 2024-03-28T23:32:20.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.115+0000 7f11e9e64640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11e414eca0 0x7f11e40c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:20.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.115+0000 7f11e9e64640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f11e40c1dc0 0x7f11e406ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:20.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.115+0000 7f11e9e64640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11e4151850 con 0x7f11e406cf60 2024-03-28T23:32:20.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.115+0000 7f11e9e64640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f11e4152460 con 0x7f11e40c1dc0 2024-03-28T23:32:20.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.115+0000 7f11e9e64640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f11e4151b30 con 0x7f11e414eca0 2024-03-28T23:32:20.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.115+0000 7f11e2ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f11e40c1dc0 0x7f11e406ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:20.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.115+0000 7f11e3fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f11e406cf60 0x7f11e40a98c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:53918/0 (socket says 172.21.15.161:53918) 2024-03-28T23:32:20.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.115+0000 7f11e3fff640 1 -- 172.21.15.161:0/2140635343 learned_addr learned my addr 172.21.15.161:0/2140635343 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:20.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.115+0000 7f11e37fe640 1 --2- 172.21.15.161:0/2140635343 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11e414eca0 0x7f11e40c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:20.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.115+0000 7f11e2ffd640 1 -- 172.21.15.161:0/2140635343 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11e414eca0 msgr2=0x7f11e40c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:20.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.115+0000 7f11e2ffd640 1 --2- 172.21.15.161:0/2140635343 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11e414eca0 0x7f11e40c1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.115+0000 7f11e2ffd640 1 -- 172.21.15.161:0/2140635343 >> v1:172.21.15.161:6789/0 conn(0x7f11e406cf60 legacy=0x7f11e40a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:20.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.115+0000 7f11e2ffd640 1 -- 172.21.15.161:0/2140635343 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e4151e10 con 0x7f11e40c1dc0 2024-03-28T23:32:20.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.115+0000 7f11e27fc640 1 -- 172.21.15.161:0/2140635343 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 536694517 0 0) 0x7f11e4151850 con 0x7f11e406cf60 2024-03-28T23:32:20.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.116+0000 7f11e2ffd640 1 --2- 172.21.15.161:0/2140635343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11e40c1dc0 0x7f11e406ca20 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f11d80097a0 tx=0x7f11d802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1136862de0a3c91 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:20.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.116+0000 7f11e27fc640 1 -- 172.21.15.161:0/2140635343 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d80315a0 con 0x7f11e40c1dc0 2024-03-28T23:32:20.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.116+0000 7f11e27fc640 1 -- 172.21.15.161:0/2140635343 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11d8037070 con 0x7f11e40c1dc0 2024-03-28T23:32:20.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.116+0000 7f11e27fc640 1 -- 172.21.15.161:0/2140635343 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d8031960 con 0x7f11e40c1dc0 2024-03-28T23:32:20.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.117+0000 7f11e9e64640 1 -- 172.21.15.161:0/2140635343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11e40c1dc0 msgr2=0x7f11e406ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:20.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.117+0000 7f11e9e64640 1 --2- 172.21.15.161:0/2140635343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11e40c1dc0 0x7f11e406ca20 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f11d80097a0 tx=0x7f11d802b840 comp rx=0 tx=0).stop 2024-03-28T23:32:20.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.117+0000 7f11e9e64640 1 -- 172.21.15.161:0/2140635343 shutdown_connections 2024-03-28T23:32:20.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.117+0000 7f11e9e64640 1 --2- 172.21.15.161:0/2140635343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11e40c1dc0 0x7f11e406ca20 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.117+0000 7f11e9e64640 1 --2- 172.21.15.161:0/2140635343 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11e414eca0 0x7f11e40c1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.117+0000 7f11e9e64640 1 -- 172.21.15.161:0/2140635343 >> 172.21.15.161:0/2140635343 conn(0x7f11e40c6670 msgr2=0x7f11e40c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:20.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.117+0000 7f11e9e64640 1 -- 172.21.15.161:0/2140635343 shutdown_connections 2024-03-28T23:32:20.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.117+0000 7f11e9e64640 1 -- 172.21.15.161:0/2140635343 wait complete. 2024-03-28T23:32:20.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.118+0000 7f11e9e64640 1 Processor -- start 2024-03-28T23:32:20.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.118+0000 7f11e9e64640 1 -- start start 2024-03-28T23:32:20.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.119+0000 7f11e9e64640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11e406cf60 0x7f11e40bd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:20.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.119+0000 7f11e9e64640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11e40c1dc0 0x7f11e4155340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:20.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.119+0000 7f11e9e64640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11e414eca0 0x7f11e414b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:20.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.119+0000 7f11e9e64640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f11e41514b0 con 0x7f11e40c1dc0 2024-03-28T23:32:20.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.119+0000 7f11e9e64640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f11e414f080 con 0x7f11e414eca0 2024-03-28T23:32:20.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.119+0000 7f11e9e64640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f11e4151630 con 0x7f11e406cf60 2024-03-28T23:32:20.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.119+0000 7f11e2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11e40c1dc0 0x7f11e4155340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:20.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.119+0000 7f11e2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11e40c1dc0 0x7f11e4155340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41740/0 (socket says 172.21.15.161:41740) 2024-03-28T23:32:20.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.119+0000 7f11e2ffd640 1 -- 172.21.15.161:0/1437287012 learned_addr learned my addr 172.21.15.161:0/1437287012 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:20.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.119+0000 7f11e37fe640 1 --2- 172.21.15.161:0/1437287012 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11e406cf60 0x7f11e40bd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:20.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.119+0000 7f11e3fff640 1 --2- 172.21.15.161:0/1437287012 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11e414eca0 0x7f11e414b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:20.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.119+0000 7f11e2ffd640 1 -- 172.21.15.161:0/1437287012 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11e406cf60 msgr2=0x7f11e40bd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:20.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.119+0000 7f11e2ffd640 1 --2- 172.21.15.161:0/1437287012 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11e406cf60 0x7f11e40bd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.119+0000 7f11e2ffd640 1 -- 172.21.15.161:0/1437287012 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11e414eca0 msgr2=0x7f11e414b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:20.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.119+0000 7f11e2ffd640 1 --2- 172.21.15.161:0/1437287012 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11e414eca0 0x7f11e414b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.119+0000 7f11e2ffd640 1 -- 172.21.15.161:0/1437287012 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e41e84b0 con 0x7f11e40c1dc0 2024-03-28T23:32:20.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.119+0000 7f11e37fe640 1 --2- 172.21.15.161:0/1437287012 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11e406cf60 0x7f11e40bd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:32:20.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.119+0000 7f11e2ffd640 1 --2- 172.21.15.161:0/1437287012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11e40c1dc0 0x7f11e4155340 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f11d8034040 tx=0x7f11d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:20.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.119+0000 7f11cbfff640 1 -- 172.21.15.161:0/1437287012 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d8004230 con 0x7f11e40c1dc0 2024-03-28T23:32:20.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.120+0000 7f11cbfff640 1 -- 172.21.15.161:0/1437287012 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11d80043f0 con 0x7f11e40c1dc0 2024-03-28T23:32:20.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.120+0000 7f11e9e64640 1 -- 172.21.15.161:0/1437287012 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11e41e8750 con 0x7f11e40c1dc0 2024-03-28T23:32:20.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.120+0000 7f11cbfff640 1 -- 172.21.15.161:0/1437287012 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d8003850 con 0x7f11e40c1dc0 2024-03-28T23:32:20.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.120+0000 7f11e9e64640 1 -- 172.21.15.161:0/1437287012 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11e41e8d10 con 0x7f11e40c1dc0 2024-03-28T23:32:20.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.121+0000 7f11e9e64640 1 -- 172.21.15.161:0/1437287012 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11e40ab860 con 0x7f11e40c1dc0 2024-03-28T23:32:20.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.121+0000 7f11cbfff640 1 -- 172.21.15.161:0/1437287012 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f11d80039f0 con 0x7f11e40c1dc0 2024-03-28T23:32:20.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.121+0000 7f11cbfff640 1 --2- 172.21.15.161:0/1437287012 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f11d006b4d0 0x7f11d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:20.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.121+0000 7f11cbfff640 1 -- 172.21.15.161:0/1437287012 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6449+0+0 (secure 0 0 0) 0x7f11d802f030 con 0x7f11e40c1dc0 2024-03-28T23:32:20.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.122+0000 7f11e37fe640 1 --2- 172.21.15.161:0/1437287012 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f11d006b4d0 0x7f11d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:20.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.122+0000 7f11e37fe640 1 --2- 172.21.15.161:0/1437287012 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f11d006b4d0 0x7f11d006d990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f11e4069630 tx=0x7f11d4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:20.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.126+0000 7f11cbfff640 1 -- 172.21.15.161:0/1437287012 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f11e40ab860 con 0x7f11e40c1dc0 2024-03-28T23:32:20.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.334+0000 7f11e9e64640 1 -- 172.21.15.161:0/1437287012 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11e404f4b0 con 0x7f11e40c1dc0 2024-03-28T23:32:20.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.335+0000 7f11cbfff640 1 -- 172.21.15.161:0/1437287012 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+5194 (secure 0 0 0) 0x7f11d8041070 con 0x7f11e40c1dc0 2024-03-28T23:32:20.335 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:20.335 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":49,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5542,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.170:6833/3666899516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3666899516},{"type":"v1","addr":"172.21.15.170:6833","nonce":3666899516}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":6273,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/3350861737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3350861737},{"type":"v1","addr":"172.21.15.161:6837","nonce":3350861737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":6288,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/2929859479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":2929859479},{"type":"v1","addr":"172.21.15.161:6836","nonce":2929859479}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:31:54.226424+0000","modified":"2024-03-28T23:32:18.788201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5390":{"gid":5390,"name":"b","rank":0,"incarnation":46,"state":"up:active","state_seq":6,"addr":"172.21.15.170:6835/2649192649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2649192649},{"type":"v1","addr":"172.21.15.170:6835","nonce":2649192649}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5390,"qdb_cluster":[5390]},"id":4}]} 2024-03-28T23:32:20.336 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 49 2024-03-28T23:32:20.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.336+0000 7f11e9e64640 1 -- 172.21.15.161:0/1437287012 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f11d006b4d0 msgr2=0x7f11d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:20.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.336+0000 7f11e9e64640 1 --2- 172.21.15.161:0/1437287012 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f11d006b4d0 0x7f11d006d990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f11e4069630 tx=0x7f11d4009190 comp rx=0 tx=0).stop 2024-03-28T23:32:20.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.336+0000 7f11e9e64640 1 -- 172.21.15.161:0/1437287012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11e40c1dc0 msgr2=0x7f11e4155340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:20.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.336+0000 7f11e9e64640 1 --2- 172.21.15.161:0/1437287012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11e40c1dc0 0x7f11e4155340 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f11d8034040 tx=0x7f11d80046c0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.337+0000 7f11e9e64640 1 -- 172.21.15.161:0/1437287012 shutdown_connections 2024-03-28T23:32:20.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.337+0000 7f11e9e64640 1 --2- 172.21.15.161:0/1437287012 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f11d006b4d0 0x7f11d006d990 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.337+0000 7f11e9e64640 1 --2- 172.21.15.161:0/1437287012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11e40c1dc0 0x7f11e4155340 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.337+0000 7f11e9e64640 1 --2- 172.21.15.161:0/1437287012 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11e414eca0 0x7f11e414b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.337+0000 7f11e9e64640 1 --2- 172.21.15.161:0/1437287012 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11e406cf60 0x7f11e40bd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.337+0000 7f11e9e64640 1 -- 172.21.15.161:0/1437287012 >> 172.21.15.161:0/1437287012 conn(0x7f11e40c6670 msgr2=0x7f11e40c4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:20.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.337+0000 7f11e9e64640 1 -- 172.21.15.161:0/1437287012 shutdown_connections 2024-03-28T23:32:20.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.337+0000 7f11e9e64640 1 -- 172.21.15.161:0/1437287012 wait complete. 2024-03-28T23:32:20.348 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:32:20.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.449+0000 7f09f4f63640 1 Processor -- start 2024-03-28T23:32:20.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.449+0000 7f09f4f63640 1 -- start start 2024-03-28T23:32:20.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.450+0000 7f09f4f63640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09f006ce80 0x7f09f00aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:20.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.450+0000 7f09f4f63640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f09f00a9680 0x7f09f00aab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:20.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.450+0000 7f09f4f63640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09f014c250 con 0x7f09f014be70 2024-03-28T23:32:20.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.450+0000 7f09f4f63640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f09f01529c0 con 0x7f09f00a9680 2024-03-28T23:32:20.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.450+0000 7f09f4f63640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f09f00a9dc0 con 0x7f09f006ce80 2024-03-28T23:32:20.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.450+0000 7f09eed9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f09f00a9680 0x7f09f00aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:20.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.450+0000 7f09edd9b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09f006ce80 0x7f09f00aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:20.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.450+0000 7f09eed9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f09f00a9680 0x7f09f00aab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41744/0 (socket says 172.21.15.161:41744) 2024-03-28T23:32:20.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.450+0000 7f09eed9d640 1 -- 172.21.15.161:0/3945467753 learned_addr learned my addr 172.21.15.161:0/3945467753 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:20.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.451+0000 7f09edd9b640 1 -- 172.21.15.161:0/3945467753 >> v2:172.21.15.161:3300/0 conn(0x7f09f00a9680 msgr2=0x7f09f00aab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:20.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.451+0000 7f09edd9b640 1 --2- 172.21.15.161:0/3945467753 >> v2:172.21.15.161:3300/0 conn(0x7f09f00a9680 0x7f09f00aab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.451+0000 7f09edd9b640 1 -- 172.21.15.161:0/3945467753 >> v1:172.21.15.170:6789/0 conn(0x7f09f014be70 legacy=0x7f09f00ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:20.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.451+0000 7f09edd9b640 1 -- 172.21.15.161:0/3945467753 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f00aa0a0 con 0x7f09f006ce80 2024-03-28T23:32:20.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.451+0000 7f09edd9b640 1 --2- 172.21.15.161:0/3945467753 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09f006ce80 0x7f09f00aa5c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f09e00097a0 tx=0x7f09e002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e9797c96a7f645f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:20.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.452+0000 7f09ed59a640 1 -- 172.21.15.161:0/3945467753 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e0038070 con 0x7f09f006ce80 2024-03-28T23:32:20.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.452+0000 7f09ed59a640 1 -- 172.21.15.161:0/3945467753 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09e0031700 con 0x7f09f006ce80 2024-03-28T23:32:20.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.452+0000 7f09ed59a640 1 -- 172.21.15.161:0/3945467753 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e0031ad0 con 0x7f09f006ce80 2024-03-28T23:32:20.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.452+0000 7f09f4f63640 1 -- 172.21.15.161:0/3945467753 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09f006ce80 msgr2=0x7f09f00aa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:20.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.452+0000 7f09f4f63640 1 --2- 172.21.15.161:0/3945467753 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09f006ce80 0x7f09f00aa5c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f09e00097a0 tx=0x7f09e002b7a0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.452+0000 7f09f4f63640 1 -- 172.21.15.161:0/3945467753 shutdown_connections 2024-03-28T23:32:20.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.452+0000 7f09f4f63640 1 --2- 172.21.15.161:0/3945467753 >> v2:172.21.15.161:3300/0 conn(0x7f09f00a9680 0x7f09f00aab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.452+0000 7f09f4f63640 1 --2- 172.21.15.161:0/3945467753 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09f006ce80 0x7f09f00aa5c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.452+0000 7f09f4f63640 1 -- 172.21.15.161:0/3945467753 >> 172.21.15.161:0/3945467753 conn(0x7f09f00c6670 msgr2=0x7f09f00c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:20.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.452+0000 7f09f4f63640 1 -- 172.21.15.161:0/3945467753 shutdown_connections 2024-03-28T23:32:20.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.452+0000 7f09f4f63640 1 -- 172.21.15.161:0/3945467753 wait complete. 2024-03-28T23:32:20.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.453+0000 7f09f4f63640 1 Processor -- start 2024-03-28T23:32:20.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.454+0000 7f09f4f63640 1 -- start start 2024-03-28T23:32:20.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.454+0000 7f09f4f63640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09f006ce80 0x7f09f00bfc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:20.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.454+0000 7f09f4f63640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09f00a9680 0x7f09f0147e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:20.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.454+0000 7f09f4f63640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09f014be70 0x7f09f01483b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:20.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.454+0000 7f09f4f63640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f09f00aba30 con 0x7f09f00a9680 2024-03-28T23:32:20.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.454+0000 7f09f4f63640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f09f00ab8b0 con 0x7f09f014be70 2024-03-28T23:32:20.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.454+0000 7f09f4f63640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f09f006b150 con 0x7f09f006ce80 2024-03-28T23:32:20.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.454+0000 7f09edd9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09f00a9680 0x7f09f0147e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:20.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.454+0000 7f09edd9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09f00a9680 0x7f09f0147e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41752/0 (socket says 172.21.15.161:41752) 2024-03-28T23:32:20.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.454+0000 7f09edd9b640 1 -- 172.21.15.161:0/2966961140 learned_addr learned my addr 172.21.15.161:0/2966961140 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:20.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.454+0000 7f09ee59c640 1 --2- 172.21.15.161:0/2966961140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09f006ce80 0x7f09f00bfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:20.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.454+0000 7f09eed9d640 1 --2- 172.21.15.161:0/2966961140 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09f014be70 0x7f09f01483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:20.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.455+0000 7f09edd9b640 1 -- 172.21.15.161:0/2966961140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09f006ce80 msgr2=0x7f09f00bfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:20.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.455+0000 7f09edd9b640 1 --2- 172.21.15.161:0/2966961140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09f006ce80 0x7f09f00bfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.455+0000 7f09edd9b640 1 -- 172.21.15.161:0/2966961140 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09f014be70 msgr2=0x7f09f01483b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:20.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.455+0000 7f09edd9b640 1 --2- 172.21.15.161:0/2966961140 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09f014be70 0x7f09f01483b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.455+0000 7f09edd9b640 1 -- 172.21.15.161:0/2966961140 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f01535d0 con 0x7f09f00a9680 2024-03-28T23:32:20.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.455+0000 7f09edd9b640 1 --2- 172.21.15.161:0/2966961140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09f00a9680 0x7f09f0147e70 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f09e0036040 tx=0x7f09e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:20.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.455+0000 7f09deffd640 1 -- 172.21.15.161:0/2966961140 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e0042070 con 0x7f09f00a9680 2024-03-28T23:32:20.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.455+0000 7f09deffd640 1 -- 172.21.15.161:0/2966961140 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09e003d890 con 0x7f09f00a9680 2024-03-28T23:32:20.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.455+0000 7f09f4f63640 1 -- 172.21.15.161:0/2966961140 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09f01dcae0 con 0x7f09f00a9680 2024-03-28T23:32:20.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.455+0000 7f09deffd640 1 -- 172.21.15.161:0/2966961140 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e0038040 con 0x7f09f00a9680 2024-03-28T23:32:20.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.455+0000 7f09f4f63640 1 -- 172.21.15.161:0/2966961140 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09f0153870 con 0x7f09f00a9680 2024-03-28T23:32:20.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.456+0000 7f09f4f63640 1 -- 172.21.15.161:0/2966961140 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09f004f4b0 con 0x7f09f00a9680 2024-03-28T23:32:20.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.456+0000 7f09deffd640 1 -- 172.21.15.161:0/2966961140 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f09e003b070 con 0x7f09f00a9680 2024-03-28T23:32:20.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.457+0000 7f09deffd640 1 --2- 172.21.15.161:0/2966961140 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f09d006b4d0 0x7f09d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:20.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.457+0000 7f09deffd640 1 -- 172.21.15.161:0/2966961140 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6449+0+0 (secure 0 0 0) 0x7f09e00abb60 con 0x7f09f00a9680 2024-03-28T23:32:20.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.457+0000 7f09ee59c640 1 --2- 172.21.15.161:0/2966961140 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f09d006b4d0 0x7f09d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:20.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.457+0000 7f09ee59c640 1 --2- 172.21.15.161:0/2966961140 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f09d006b4d0 0x7f09d006d990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f09f00aae40 tx=0x7f09e4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:20.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.461+0000 7f09deffd640 1 -- 172.21.15.161:0/2966961140 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f09e0030070 con 0x7f09f00a9680 2024-03-28T23:32:20.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.632+0000 7f09f4f63640 1 -- 172.21.15.161:0/2966961140 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f09f01488f0 con 0x7f09f00a9680 2024-03-28T23:32:20.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.633+0000 7f09deffd640 1 -- 172.21.15.161:0/2966961140 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) ==== 74+0+17449 (secure 0 0 0) 0x7f09e007a250 con 0x7f09f00a9680 2024-03-28T23:32:20.634 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:20.634 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":61,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:32:17.787428+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:31:52.774315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:31:53.201664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6836/1300583128":"2024-03-29T23:32:15.775261+0000","172.21.15.161:6834/1300583128":"2024-03-29T23:32:15.775261+0000","172.21.15.161:6837/1498528126":"2024-03-29T23:32:14.767288+0000","172.21.15.170:6835/1945122204":"2024-03-29T23:32:06.533661+0000","172.21.15.161:6835/1498528126":"2024-03-29T23:32:14.767288+0000","172.21.15.170:6834/1945122204":"2024-03-29T23:32:06.533661+0000","172.21.15.161:6837/544971001":"2024-03-29T23:32:03.518509+0000","172.21.15.161:6835/544971001":"2024-03-29T23:32:03.518509+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:32:20.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.635+0000 7f09f4f63640 1 -- 172.21.15.161:0/2966961140 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f09d006b4d0 msgr2=0x7f09d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:20.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.635+0000 7f09f4f63640 1 --2- 172.21.15.161:0/2966961140 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f09d006b4d0 0x7f09d006d990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f09f00aae40 tx=0x7f09e4009040 comp rx=0 tx=0).stop 2024-03-28T23:32:20.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.635+0000 7f09f4f63640 1 -- 172.21.15.161:0/2966961140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09f00a9680 msgr2=0x7f09f0147e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:20.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.635+0000 7f09f4f63640 1 --2- 172.21.15.161:0/2966961140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09f00a9680 0x7f09f0147e70 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f09e0036040 tx=0x7f09e00046c0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.635+0000 7f09f4f63640 1 -- 172.21.15.161:0/2966961140 shutdown_connections 2024-03-28T23:32:20.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.636+0000 7f09f4f63640 1 --2- 172.21.15.161:0/2966961140 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f09d006b4d0 0x7f09d006d990 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.636+0000 7f09f4f63640 1 --2- 172.21.15.161:0/2966961140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09f00a9680 0x7f09f0147e70 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.636+0000 7f09f4f63640 1 --2- 172.21.15.161:0/2966961140 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09f014be70 0x7f09f01483b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.636+0000 7f09f4f63640 1 --2- 172.21.15.161:0/2966961140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09f006ce80 0x7f09f00bfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.636+0000 7f09f4f63640 1 -- 172.21.15.161:0/2966961140 >> 172.21.15.161:0/2966961140 conn(0x7f09f00c6670 msgr2=0x7f09f01de530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:20.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.636+0000 7f09f4f63640 1 -- 172.21.15.161:0/2966961140 shutdown_connections 2024-03-28T23:32:20.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.636+0000 7f09f4f63640 1 -- 172.21.15.161:0/2966961140 wait complete. 2024-03-28T23:32:20.647 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs2 2024-03-28T23:32:20.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.748+0000 7efff6449640 1 Processor -- start 2024-03-28T23:32:20.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.748+0000 7efff6449640 1 -- start start 2024-03-28T23:32:20.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.749+0000 7efff6449640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efff006ce80 0x7efff00aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:20.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.749+0000 7efff6449640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7efff00a9680 0x7efff00aab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:20.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.749+0000 7efff6449640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efff014c250 con 0x7efff014be70 2024-03-28T23:32:20.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.749+0000 7efff6449640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7efff01529c0 con 0x7efff006ce80 2024-03-28T23:32:20.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.749+0000 7efff6449640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7efff00a9dc0 con 0x7efff00a9680 2024-03-28T23:32:20.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.749+0000 7effeffff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7efff014be70 0x7efff00ab1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:35452/0 (socket says 172.21.15.161:35452) 2024-03-28T23:32:20.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.749+0000 7effef7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efff006ce80 0x7efff00aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:20.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.749+0000 7effeffff640 1 -- 172.21.15.161:0/859847596 learned_addr learned my addr 172.21.15.161:0/859847596 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:20.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.749+0000 7efff49e6640 1 --2- 172.21.15.161:0/859847596 >> v2:172.21.15.170:3300/0 conn(0x7efff00a9680 0x7efff00aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:20.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.749+0000 7effeeffd640 1 -- 172.21.15.161:0/859847596 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1431928587 0 0) 0x7efff014c250 con 0x7efff014be70 2024-03-28T23:32:20.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.749+0000 7effef7fe640 1 -- 172.21.15.161:0/859847596 >> v2:172.21.15.170:3300/0 conn(0x7efff00a9680 msgr2=0x7efff00aab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:20.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.749+0000 7effef7fe640 1 --2- 172.21.15.161:0/859847596 >> v2:172.21.15.170:3300/0 conn(0x7efff00a9680 0x7efff00aab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.749+0000 7effef7fe640 1 -- 172.21.15.161:0/859847596 >> v1:172.21.15.170:6789/0 conn(0x7efff014be70 legacy=0x7efff00ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:20.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.750+0000 7effef7fe640 1 -- 172.21.15.161:0/859847596 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efff00aa0a0 con 0x7efff006ce80 2024-03-28T23:32:20.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.750+0000 7effef7fe640 1 --2- 172.21.15.161:0/859847596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efff006ce80 0x7efff00aa5c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7efff0073960 tx=0x7effe402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae360a5702ec4209 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:20.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.750+0000 7effeeffd640 1 -- 172.21.15.161:0/859847596 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe4038070 con 0x7efff006ce80 2024-03-28T23:32:20.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.750+0000 7effeeffd640 1 -- 172.21.15.161:0/859847596 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effe4031710 con 0x7efff006ce80 2024-03-28T23:32:20.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.750+0000 7effeeffd640 1 -- 172.21.15.161:0/859847596 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe4031ae0 con 0x7efff006ce80 2024-03-28T23:32:20.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.751+0000 7efff6449640 1 -- 172.21.15.161:0/859847596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efff006ce80 msgr2=0x7efff00aa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:20.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.751+0000 7efff6449640 1 --2- 172.21.15.161:0/859847596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efff006ce80 0x7efff00aa5c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7efff0073960 tx=0x7effe402b7b0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.751+0000 7efff6449640 1 -- 172.21.15.161:0/859847596 shutdown_connections 2024-03-28T23:32:20.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.751+0000 7efff6449640 1 --2- 172.21.15.161:0/859847596 >> v2:172.21.15.170:3300/0 conn(0x7efff00a9680 0x7efff00aab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.751+0000 7efff6449640 1 --2- 172.21.15.161:0/859847596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efff006ce80 0x7efff00aa5c0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.751+0000 7efff6449640 1 -- 172.21.15.161:0/859847596 >> 172.21.15.161:0/859847596 conn(0x7efff00c6670 msgr2=0x7efff00c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:20.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.751+0000 7efff6449640 1 -- 172.21.15.161:0/859847596 shutdown_connections 2024-03-28T23:32:20.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.751+0000 7efff6449640 1 -- 172.21.15.161:0/859847596 wait complete. 2024-03-28T23:32:20.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.752+0000 7efff6449640 1 Processor -- start 2024-03-28T23:32:20.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.752+0000 7efff6449640 1 -- start start 2024-03-28T23:32:20.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.753+0000 7efff6449640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efff006ce80 0x7efff006b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:20.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.753+0000 7efff6449640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efff00a9680 0x7efff014cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:20.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.753+0000 7efff6449640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efff014be70 0x7efff01ebb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:20.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.753+0000 7efff6449640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efff00aba30 con 0x7efff014be70 2024-03-28T23:32:20.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.753+0000 7efff6449640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7efff00ab8b0 con 0x7efff00a9680 2024-03-28T23:32:20.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.753+0000 7efff6449640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7efff006b150 con 0x7efff006ce80 2024-03-28T23:32:20.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.753+0000 7efff49e6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efff014be70 0x7efff01ebb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:20.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.753+0000 7efff49e6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efff014be70 0x7efff01ebb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33592/0 (socket says 172.21.15.161:33592) 2024-03-28T23:32:20.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.753+0000 7efff49e6640 1 -- 172.21.15.161:0/2273324995 learned_addr learned my addr 172.21.15.161:0/2273324995 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:20.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.753+0000 7effef7fe640 1 --2- 172.21.15.161:0/2273324995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efff00a9680 0x7efff014cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:20.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.753+0000 7effeffff640 1 --2- 172.21.15.161:0/2273324995 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efff006ce80 0x7efff006b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:20.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.753+0000 7efff49e6640 1 -- 172.21.15.161:0/2273324995 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efff006ce80 msgr2=0x7efff006b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:20.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.753+0000 7efff49e6640 1 --2- 172.21.15.161:0/2273324995 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efff006ce80 0x7efff006b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.753+0000 7efff49e6640 1 -- 172.21.15.161:0/2273324995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efff00a9680 msgr2=0x7efff014cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:20.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.753+0000 7efff49e6640 1 --2- 172.21.15.161:0/2273324995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efff00a9680 0x7efff014cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.753+0000 7efff49e6640 1 -- 172.21.15.161:0/2273324995 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efff014f310 con 0x7efff014be70 2024-03-28T23:32:20.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.754+0000 7efff49e6640 1 --2- 172.21.15.161:0/2273324995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efff014be70 0x7efff01ebb40 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7effe8009d30 tx=0x7effe800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:20.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.754+0000 7effecff9640 1 -- 172.21.15.161:0/2273324995 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe8018020 con 0x7efff014be70 2024-03-28T23:32:20.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.754+0000 7effecff9640 1 -- 172.21.15.161:0/2273324995 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effe8014480 con 0x7efff014be70 2024-03-28T23:32:20.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.754+0000 7effecff9640 1 -- 172.21.15.161:0/2273324995 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe8014870 con 0x7efff014be70 2024-03-28T23:32:20.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.754+0000 7efff6449640 1 -- 172.21.15.161:0/2273324995 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efff014e130 con 0x7efff014be70 2024-03-28T23:32:20.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.754+0000 7efff6449640 1 -- 172.21.15.161:0/2273324995 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efff014c460 con 0x7efff014be70 2024-03-28T23:32:20.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.755+0000 7efff6449640 1 -- 172.21.15.161:0/2273324995 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efff004f4b0 con 0x7efff014be70 2024-03-28T23:32:20.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.755+0000 7effecff9640 1 -- 172.21.15.161:0/2273324995 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7effe8005d10 con 0x7efff014be70 2024-03-28T23:32:20.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.756+0000 7effecff9640 1 --2- 172.21.15.161:0/2273324995 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7effdc06b500 0x7effdc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:20.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.756+0000 7effecff9640 1 -- 172.21.15.161:0/2273324995 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6449+0+0 (secure 0 0 0) 0x7effe808ade0 con 0x7efff014be70 2024-03-28T23:32:20.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.756+0000 7effeffff640 1 --2- 172.21.15.161:0/2273324995 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7effdc06b500 0x7effdc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:20.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.756+0000 7effeffff640 1 --2- 172.21.15.161:0/2273324995 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7effdc06b500 0x7effdc06d9c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7effe0007fd0 tx=0x7effe0007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:20.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.760+0000 7effecff9640 1 -- 172.21.15.161:0/2273324995 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7effe8022070 con 0x7efff014be70 2024-03-28T23:32:20.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.968+0000 7efff6449640 1 -- 172.21.15.161:0/2273324995 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs2"} v 0) -- 0x7efff00c0ff0 con 0x7efff014be70 2024-03-28T23:32:20.969 INFO:tasks.ceph.mon.a.smithi161.stderr:2024-03-28T23:32:20.969+0000 7fdc7b85b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T23:32:20.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.970+0000 7effecff9640 1 -- 172.21.15.161:0/2273324995 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs2"}]=0 cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v50) ==== 163+0+0 (secure 0 0 0) 0x7effe8010040 con 0x7efff014be70 2024-03-28T23:32:20.971 INFO:teuthology.orchestra.run.smithi161.stderr:cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-28T23:32:20.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.973+0000 7efff6449640 1 -- 172.21.15.161:0/2273324995 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7effdc06b500 msgr2=0x7effdc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:20.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.973+0000 7efff6449640 1 --2- 172.21.15.161:0/2273324995 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7effdc06b500 0x7effdc06d9c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7effe0007fd0 tx=0x7effe0007de0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.973+0000 7efff6449640 1 -- 172.21.15.161:0/2273324995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efff014be70 msgr2=0x7efff01ebb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:20.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.973+0000 7efff6449640 1 --2- 172.21.15.161:0/2273324995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efff014be70 0x7efff01ebb40 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7effe8009d30 tx=0x7effe800c490 comp rx=0 tx=0).stop 2024-03-28T23:32:20.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.973+0000 7efff6449640 1 -- 172.21.15.161:0/2273324995 shutdown_connections 2024-03-28T23:32:20.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.973+0000 7efff6449640 1 --2- 172.21.15.161:0/2273324995 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7effdc06b500 0x7effdc06d9c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.973+0000 7efff6449640 1 --2- 172.21.15.161:0/2273324995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efff014be70 0x7efff01ebb40 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.973+0000 7efff6449640 1 --2- 172.21.15.161:0/2273324995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efff00a9680 0x7efff014cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.973+0000 7efff6449640 1 --2- 172.21.15.161:0/2273324995 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efff006ce80 0x7efff006b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:20.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.973+0000 7efff6449640 1 -- 172.21.15.161:0/2273324995 >> 172.21.15.161:0/2273324995 conn(0x7efff00c6670 msgr2=0x7efff00c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:20.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.973+0000 7efff6449640 1 -- 172.21.15.161:0/2273324995 shutdown_connections 2024-03-28T23:32:20.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:20.973+0000 7efff6449640 1 -- 172.21.15.161:0/2273324995 wait complete. 2024-03-28T23:32:20.985 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs2 --yes-i-really-mean-it 2024-03-28T23:32:20.998 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2024-03-28T23:32:21.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.087+0000 7f6ce0fac640 1 Processor -- start 2024-03-28T23:32:21.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.087+0000 7f6ce0fac640 1 -- start start 2024-03-28T23:32:21.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.088+0000 7f6ce0fac640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6cdc06c490 0x7f6cdc0c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:21.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.088+0000 7f6ce0fac640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6cdc155110 0x7f6cdc155540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:21.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.088+0000 7f6ce0fac640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cdc06eaf0 con 0x7f6cdc152750 2024-03-28T23:32:21.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.088+0000 7f6ce0fac640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6cdc158700 con 0x7f6cdc06c490 2024-03-28T23:32:21.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.088+0000 7f6ce0fac640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f6cdc152b30 con 0x7f6cdc155110 2024-03-28T23:32:21.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.088+0000 7f6cd9d9b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6cdc06c490 0x7f6cdc0c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:21.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.088+0000 7f6cd9d9b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6cdc06c490 0x7f6cdc0c60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33594/0 (socket says 172.21.15.161:33594) 2024-03-28T23:32:21.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.088+0000 7f6cd9d9b640 1 -- 172.21.15.161:0/1618927311 learned_addr learned my addr 172.21.15.161:0/1618927311 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:21.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.088+0000 7f6cdad9d640 1 --2- 172.21.15.161:0/1618927311 >> v2:172.21.15.170:3300/0 conn(0x7f6cdc155110 0x7f6cdc155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:21.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.089+0000 7f6cd9d9b640 1 -- 172.21.15.161:0/1618927311 >> v2:172.21.15.170:3300/0 conn(0x7f6cdc155110 msgr2=0x7f6cdc155540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:21.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.089+0000 7f6cd9d9b640 1 --2- 172.21.15.161:0/1618927311 >> v2:172.21.15.170:3300/0 conn(0x7f6cdc155110 0x7f6cdc155540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:21.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.089+0000 7f6cd9d9b640 1 -- 172.21.15.161:0/1618927311 >> v1:172.21.15.170:6789/0 conn(0x7f6cdc152750 legacy=0x7f6cdc0bfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:21.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.089+0000 7f6cd9d9b640 1 -- 172.21.15.161:0/1618927311 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cdc158950 con 0x7f6cdc06c490 2024-03-28T23:32:21.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.089+0000 7f6cdad9d640 1 --2- 172.21.15.161:0/1618927311 >> v2:172.21.15.170:3300/0 conn(0x7f6cdc155110 0x7f6cdc155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:21.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.089+0000 7f6cd9d9b640 1 --2- 172.21.15.161:0/1618927311 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cdc06c490 0x7f6cdc0c60e0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc06d350 tx=0x7f6cd002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e127acb26461cd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:21.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.090+0000 7f6cd959a640 1 -- 172.21.15.161:0/1618927311 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd002e070 con 0x7f6cdc06c490 2024-03-28T23:32:21.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.090+0000 7f6cd959a640 1 -- 172.21.15.161:0/1618927311 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cd0037070 con 0x7f6cdc06c490 2024-03-28T23:32:21.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.090+0000 7f6cd959a640 1 -- 172.21.15.161:0/1618927311 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0031a00 con 0x7f6cdc06c490 2024-03-28T23:32:21.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.090+0000 7f6ce0fac640 1 -- 172.21.15.161:0/1618927311 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cdc06c490 msgr2=0x7f6cdc0c60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:21.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.090+0000 7f6ce0fac640 1 --2- 172.21.15.161:0/1618927311 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cdc06c490 0x7f6cdc0c60e0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc06d350 tx=0x7f6cd002b880 comp rx=0 tx=0).stop 2024-03-28T23:32:21.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.090+0000 7f6ce0fac640 1 -- 172.21.15.161:0/1618927311 shutdown_connections 2024-03-28T23:32:21.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.090+0000 7f6ce0fac640 1 --2- 172.21.15.161:0/1618927311 >> v2:172.21.15.170:3300/0 conn(0x7f6cdc155110 0x7f6cdc155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:21.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.090+0000 7f6ce0fac640 1 --2- 172.21.15.161:0/1618927311 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cdc06c490 0x7f6cdc0c60e0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:21.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.090+0000 7f6ce0fac640 1 -- 172.21.15.161:0/1618927311 >> 172.21.15.161:0/1618927311 conn(0x7f6cdc0c6b30 msgr2=0x7f6cdc0c6f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:21.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.090+0000 7f6ce0fac640 1 -- 172.21.15.161:0/1618927311 shutdown_connections 2024-03-28T23:32:21.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.090+0000 7f6ce0fac640 1 -- 172.21.15.161:0/1618927311 wait complete. 2024-03-28T23:32:21.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.091+0000 7f6ce0fac640 1 Processor -- start 2024-03-28T23:32:21.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.092+0000 7f6ce0fac640 1 -- start start 2024-03-28T23:32:21.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.092+0000 7f6ce0fac640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6cdc06c490 0x7f6cdc1ee680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:21.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.092+0000 7f6ce0fac640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6cdc152750 0x7f6cdc071400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:21.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.092+0000 7f6ce0fac640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cdc155110 0x7f6cdc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:21.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.092+0000 7f6ce0fac640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc1591f0 con 0x7f6cdc155110 2024-03-28T23:32:21.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.092+0000 7f6ce0fac640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc158f10 con 0x7f6cdc152750 2024-03-28T23:32:21.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.092+0000 7f6ce0fac640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6cdc0c05b0 con 0x7f6cdc06c490 2024-03-28T23:32:21.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.092+0000 7f6cdad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cdc155110 0x7f6cdc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:21.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.092+0000 7f6cdad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cdc155110 0x7f6cdc071940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33608/0 (socket says 172.21.15.161:33608) 2024-03-28T23:32:21.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.092+0000 7f6cdad9d640 1 -- 172.21.15.161:0/2419860214 learned_addr learned my addr 172.21.15.161:0/2419860214 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:21.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.092+0000 7f6cd9d9b640 1 --2- 172.21.15.161:0/2419860214 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6cdc152750 0x7f6cdc071400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:21.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.092+0000 7f6cda59c640 1 --2- 172.21.15.161:0/2419860214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6cdc06c490 0x7f6cdc1ee680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:21.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.093+0000 7f6cdad9d640 1 -- 172.21.15.161:0/2419860214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6cdc06c490 msgr2=0x7f6cdc1ee680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:21.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.093+0000 7f6cdad9d640 1 --2- 172.21.15.161:0/2419860214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6cdc06c490 0x7f6cdc1ee680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:21.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.093+0000 7f6cdad9d640 1 -- 172.21.15.161:0/2419860214 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6cdc152750 msgr2=0x7f6cdc071400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.093+0000 7f6cdad9d640 1 --2- 172.21.15.161:0/2419860214 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6cdc152750 0x7f6cdc071400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.093+0000 7f6cdad9d640 1 -- 172.21.15.161:0/2419860214 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cdc071e80 con 0x7f6cdc155110 2024-03-28T23:32:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.093+0000 7f6cdad9d640 1 --2- 172.21.15.161:0/2419860214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cdc155110 0x7f6cdc071940 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f6cd400e810 tx=0x7f6cd400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.093+0000 7f6ccaffd640 1 -- 172.21.15.161:0/2419860214 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd400f040 con 0x7f6cdc155110 2024-03-28T23:32:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.093+0000 7f6ccaffd640 1 -- 172.21.15.161:0/2419860214 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cd4004030 con 0x7f6cdc155110 2024-03-28T23:32:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.093+0000 7f6ce0fac640 1 -- 172.21.15.161:0/2419860214 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cdc0a9ab0 con 0x7f6cdc155110 2024-03-28T23:32:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.093+0000 7f6ccaffd640 1 -- 172.21.15.161:0/2419860214 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd40043e0 con 0x7f6cdc155110 2024-03-28T23:32:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.093+0000 7f6ce0fac640 1 -- 172.21.15.161:0/2419860214 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cdc073630 con 0x7f6cdc155110 2024-03-28T23:32:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.094+0000 7f6ce0fac640 1 -- 172.21.15.161:0/2419860214 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cdc0749d0 con 0x7f6cdc155110 2024-03-28T23:32:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.094+0000 7f6ccaffd640 1 -- 172.21.15.161:0/2419860214 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6cd400c6d0 con 0x7f6cdc155110 2024-03-28T23:32:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.095+0000 7f6ccaffd640 1 --2- 172.21.15.161:0/2419860214 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6cbc06b500 0x7f6cbc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.095+0000 7f6ccaffd640 1 -- 172.21.15.161:0/2419860214 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6535+0+0 (secure 0 0 0) 0x7f6cd408a0a0 con 0x7f6cdc155110 2024-03-28T23:32:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.095+0000 7f6cda59c640 1 --2- 172.21.15.161:0/2419860214 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6cbc06b500 0x7f6cbc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:21.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.095+0000 7f6cda59c640 1 --2- 172.21.15.161:0/2419860214 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6cbc06b500 0x7f6cbc06d9c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc073210 tx=0x7f6ccc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:21.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.099+0000 7f6ccaffd640 1 -- 172.21.15.161:0/2419860214 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6cdc0749d0 con 0x7f6cdc155110 2024-03-28T23:32:21.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.308+0000 7f6ce0fac640 1 -- 172.21.15.161:0/2419860214 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true} v 0) -- 0x7f6cdc04f4b0 con 0x7f6cdc155110 2024-03-28T23:32:21.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.974+0000 7f6ccaffd640 1 -- 172.21.15.161:0/2419860214 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true}]=0 v51) ==== 105+0+0 (secure 0 0 0) 0x7f6cdc071e80 con 0x7f6cdc155110 2024-03-28T23:32:21.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.976+0000 7f6ce0fac640 1 -- 172.21.15.161:0/2419860214 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6cbc06b500 msgr2=0x7f6cbc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:21.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.976+0000 7f6ce0fac640 1 --2- 172.21.15.161:0/2419860214 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6cbc06b500 0x7f6cbc06d9c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc073210 tx=0x7f6ccc009040 comp rx=0 tx=0).stop 2024-03-28T23:32:21.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.976+0000 7f6ce0fac640 1 -- 172.21.15.161:0/2419860214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cdc155110 msgr2=0x7f6cdc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:21.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.976+0000 7f6ce0fac640 1 --2- 172.21.15.161:0/2419860214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cdc155110 0x7f6cdc071940 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f6cd400e810 tx=0x7f6cd400ecd0 comp rx=0 tx=0).stop 2024-03-28T23:32:21.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.976+0000 7f6ce0fac640 1 -- 172.21.15.161:0/2419860214 shutdown_connections 2024-03-28T23:32:21.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.976+0000 7f6ce0fac640 1 --2- 172.21.15.161:0/2419860214 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6cbc06b500 0x7f6cbc06d9c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:21.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.976+0000 7f6ce0fac640 1 --2- 172.21.15.161:0/2419860214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cdc155110 0x7f6cdc071940 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:21.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.976+0000 7f6ce0fac640 1 --2- 172.21.15.161:0/2419860214 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6cdc152750 0x7f6cdc071400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:21.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.976+0000 7f6ce0fac640 1 --2- 172.21.15.161:0/2419860214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6cdc06c490 0x7f6cdc1ee680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:21.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.976+0000 7f6ce0fac640 1 -- 172.21.15.161:0/2419860214 >> 172.21.15.161:0/2419860214 conn(0x7f6cdc0c6b30 msgr2=0x7f6cdc150430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:21.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.976+0000 7f6ce0fac640 1 -- 172.21.15.161:0/2419860214 shutdown_connections 2024-03-28T23:32:21.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:21.976+0000 7f6ce0fac640 1 -- 172.21.15.161:0/2419860214 wait complete. 2024-03-28T23:32:21.989 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_metadata cephfs2_metadata --yes-i-really-really-mean-it 2024-03-28T23:32:22.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.092+0000 7f388b748640 1 Processor -- start 2024-03-28T23:32:22.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.093+0000 7f388b748640 1 -- start start 2024-03-28T23:32:22.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.093+0000 7f388b748640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38840c3080 0x7f3884151db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:22.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.093+0000 7f388b748640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f388406cab0 0x7f388406cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:22.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.093+0000 7f388b748640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38840c15a0 con 0x7f388414f1d0 2024-03-28T23:32:22.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.093+0000 7f388b748640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f38840c6330 con 0x7f388406cab0 2024-03-28T23:32:22.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.093+0000 7f388b748640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f38840c1840 con 0x7f38840c3080 2024-03-28T23:32:22.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.094+0000 7f3889ce5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f388406cab0 0x7f388406cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:22.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.094+0000 7f3888ce3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38840c3080 0x7f3884151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:22.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.094+0000 7f38894e4640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f388414f1d0 0x7f38840c3a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:35466/0 (socket says 172.21.15.161:35466) 2024-03-28T23:32:22.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.094+0000 7f38894e4640 1 -- 172.21.15.161:0/467466548 learned_addr learned my addr 172.21.15.161:0/467466548 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:22.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.094+0000 7f3889ce5640 1 -- 172.21.15.161:0/467466548 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38840c3080 msgr2=0x7f3884151db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:22.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.094+0000 7f3889ce5640 1 --2- 172.21.15.161:0/467466548 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38840c3080 0x7f3884151db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:22.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.094+0000 7f3889ce5640 1 -- 172.21.15.161:0/467466548 >> v1:172.21.15.170:6789/0 conn(0x7f388414f1d0 legacy=0x7f38840c3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:22.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.094+0000 7f3889ce5640 1 -- 172.21.15.161:0/467466548 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38840c1b20 con 0x7f388406cab0 2024-03-28T23:32:22.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.094+0000 7f3888ce3640 1 --2- 172.21.15.161:0/467466548 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38840c3080 0x7f3884151db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:32:22.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.095+0000 7f3889ce5640 1 --2- 172.21.15.161:0/467466548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f388406cab0 0x7f388406cf20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f387c009d90 tx=0x7f387c02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c713027445d97c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:22.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.095+0000 7f387bfff640 1 -- 172.21.15.161:0/467466548 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c039070 con 0x7f388406cab0 2024-03-28T23:32:22.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.095+0000 7f387bfff640 1 -- 172.21.15.161:0/467466548 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f387c02cbc0 con 0x7f388406cab0 2024-03-28T23:32:22.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.095+0000 7f387bfff640 1 -- 172.21.15.161:0/467466548 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c034440 con 0x7f388406cab0 2024-03-28T23:32:22.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.096+0000 7f388b748640 1 -- 172.21.15.161:0/467466548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f388406cab0 msgr2=0x7f388406cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:22.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.096+0000 7f388b748640 1 --2- 172.21.15.161:0/467466548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f388406cab0 0x7f388406cf20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f387c009d90 tx=0x7f387c02be10 comp rx=0 tx=0).stop 2024-03-28T23:32:22.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.096+0000 7f388b748640 1 -- 172.21.15.161:0/467466548 shutdown_connections 2024-03-28T23:32:22.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.096+0000 7f388b748640 1 --2- 172.21.15.161:0/467466548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f388406cab0 0x7f388406cf20 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:22.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.096+0000 7f388b748640 1 --2- 172.21.15.161:0/467466548 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38840c3080 0x7f3884151db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:22.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.096+0000 7f388b748640 1 -- 172.21.15.161:0/467466548 >> 172.21.15.161:0/467466548 conn(0x7f38840c6e20 msgr2=0x7f38840c7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:22.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.096+0000 7f388b748640 1 -- 172.21.15.161:0/467466548 shutdown_connections 2024-03-28T23:32:22.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.096+0000 7f388b748640 1 -- 172.21.15.161:0/467466548 wait complete. 2024-03-28T23:32:22.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.097+0000 7f388b748640 1 Processor -- start 2024-03-28T23:32:22.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.097+0000 7f388b748640 1 -- start start 2024-03-28T23:32:22.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.098+0000 7f388b748640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f388406cab0 0x7f3884153fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:22.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.098+0000 7f388b748640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38840c3080 0x7f38841494e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:22.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.098+0000 7f388b748640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f388414f1d0 0x7f3884149a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:22.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.098+0000 7f388b748640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f388406d460 con 0x7f38840c3080 2024-03-28T23:32:22.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.098+0000 7f388b748640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f38840c2600 con 0x7f388414f1d0 2024-03-28T23:32:22.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.098+0000 7f388b748640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f38840c41a0 con 0x7f388406cab0 2024-03-28T23:32:22.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.098+0000 7f3888ce3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38840c3080 0x7f38841494e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:22.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.098+0000 7f3888ce3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38840c3080 0x7f38841494e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33616/0 (socket says 172.21.15.161:33616) 2024-03-28T23:32:22.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.098+0000 7f3888ce3640 1 -- 172.21.15.161:0/655586874 learned_addr learned my addr 172.21.15.161:0/655586874 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:22.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.098+0000 7f38894e4640 1 --2- 172.21.15.161:0/655586874 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f388406cab0 0x7f3884153fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:22.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.098+0000 7f3889ce5640 1 --2- 172.21.15.161:0/655586874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f388414f1d0 0x7f3884149a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:22.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.098+0000 7f3888ce3640 1 -- 172.21.15.161:0/655586874 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f388406cab0 msgr2=0x7f3884153fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:22.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.098+0000 7f3888ce3640 1 --2- 172.21.15.161:0/655586874 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f388406cab0 0x7f3884153fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:22.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.098+0000 7f3888ce3640 1 -- 172.21.15.161:0/655586874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f388414f1d0 msgr2=0x7f3884149a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:22.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.098+0000 7f3888ce3640 1 --2- 172.21.15.161:0/655586874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f388414f1d0 0x7f3884149a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:22.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.098+0000 7f3888ce3640 1 -- 172.21.15.161:0/655586874 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3884155800 con 0x7f38840c3080 2024-03-28T23:32:22.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.098+0000 7f3888ce3640 1 --2- 172.21.15.161:0/655586874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38840c3080 0x7f38841494e0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f388000ec10 tx=0x7f388000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:22.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.098+0000 7f3879ffb640 1 -- 172.21.15.161:0/655586874 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3880017070 con 0x7f38840c3080 2024-03-28T23:32:22.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.098+0000 7f3879ffb640 1 -- 172.21.15.161:0/655586874 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f388000cbd0 con 0x7f38840c3080 2024-03-28T23:32:22.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.098+0000 7f3879ffb640 1 -- 172.21.15.161:0/655586874 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3880013470 con 0x7f38840c3080 2024-03-28T23:32:22.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.098+0000 7f388b748640 1 -- 172.21.15.161:0/655586874 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3884150f00 con 0x7f38840c3080 2024-03-28T23:32:22.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.099+0000 7f388b748640 1 -- 172.21.15.161:0/655586874 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38841560c0 con 0x7f38840c3080 2024-03-28T23:32:22.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.100+0000 7f388b748640 1 -- 172.21.15.161:0/655586874 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3884068f90 con 0x7f38840c3080 2024-03-28T23:32:22.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.100+0000 7f3879ffb640 1 -- 172.21.15.161:0/655586874 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f388001a070 con 0x7f38840c3080 2024-03-28T23:32:22.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.100+0000 7f3879ffb640 1 --2- 172.21.15.161:0/655586874 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f386806b4d0 0x7f386806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:22.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.101+0000 7f3879ffb640 1 -- 172.21.15.161:0/655586874 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6535+0+0 (secure 0 0 0) 0x7f388008a3b0 con 0x7f38840c3080 2024-03-28T23:32:22.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.101+0000 7f38894e4640 1 --2- 172.21.15.161:0/655586874 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f386806b4d0 0x7f386806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:22.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.101+0000 7f38894e4640 1 --2- 172.21.15.161:0/655586874 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f386806b4d0 0x7f386806d990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f38841dd3f0 tx=0x7f3874007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:22.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.105+0000 7f3879ffb640 1 -- 172.21.15.161:0/655586874 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f38800588a0 con 0x7f38840c3080 2024-03-28T23:32:22.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.275+0000 7f388b748640 1 -- 172.21.15.161:0/655586874 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f38840c6590 con 0x7f38840c3080 2024-03-28T23:32:22.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.975+0000 7f3879ffb640 1 -- 172.21.15.161:0/655586874 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_metadata' removed v63) ==== 184+0+0 (secure 0 0 0) 0x7f388005c940 con 0x7f38840c3080 2024-03-28T23:32:22.976 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs2_metadata' removed 2024-03-28T23:32:22.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.978+0000 7f388b748640 1 -- 172.21.15.161:0/655586874 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f386806b4d0 msgr2=0x7f386806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:22.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.978+0000 7f388b748640 1 --2- 172.21.15.161:0/655586874 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f386806b4d0 0x7f386806d990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f38841dd3f0 tx=0x7f3874007e80 comp rx=0 tx=0).stop 2024-03-28T23:32:22.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.978+0000 7f388b748640 1 -- 172.21.15.161:0/655586874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38840c3080 msgr2=0x7f38841494e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:22.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.978+0000 7f388b748640 1 --2- 172.21.15.161:0/655586874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38840c3080 0x7f38841494e0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f388000ec10 tx=0x7f388000c520 comp rx=0 tx=0).stop 2024-03-28T23:32:22.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.978+0000 7f388b748640 1 -- 172.21.15.161:0/655586874 shutdown_connections 2024-03-28T23:32:22.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.978+0000 7f388b748640 1 --2- 172.21.15.161:0/655586874 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f386806b4d0 0x7f386806d990 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:22.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.978+0000 7f388b748640 1 --2- 172.21.15.161:0/655586874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38840c3080 0x7f38841494e0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:22.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.978+0000 7f388b748640 1 --2- 172.21.15.161:0/655586874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f388414f1d0 0x7f3884149a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:22.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.978+0000 7f388b748640 1 --2- 172.21.15.161:0/655586874 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f388406cab0 0x7f3884153fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:22.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.978+0000 7f388b748640 1 -- 172.21.15.161:0/655586874 >> 172.21.15.161:0/655586874 conn(0x7f38840c6e20 msgr2=0x7f3884071a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:22.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.978+0000 7f388b748640 1 -- 172.21.15.161:0/655586874 shutdown_connections 2024-03-28T23:32:22.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:22.978+0000 7f388b748640 1 -- 172.21.15.161:0/655586874 wait complete. 2024-03-28T23:32:22.991 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_data cephfs2_data --yes-i-really-really-mean-it 2024-03-28T23:32:23.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.096+0000 7f581b1b5640 1 Processor -- start 2024-03-28T23:32:23.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.096+0000 7f581b1b5640 1 -- start start 2024-03-28T23:32:23.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.097+0000 7f581b1b5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5814152740 0x7f581406b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:23.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.097+0000 7f581b1b5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5814155d70 0x7f58140c9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:23.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.097+0000 7f581b1b5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58140c9980 con 0x7f58140c53e0 2024-03-28T23:32:23.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.097+0000 7f581b1b5640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f58141561d0 con 0x7f5814152740 2024-03-28T23:32:23.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.097+0000 7f581b1b5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f581406bb70 con 0x7f5814155d70 2024-03-28T23:32:23.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.097+0000 7f5818f51640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5814152740 0x7f581406b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:23.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.097+0000 7f5818f51640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5814152740 0x7f581406b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33620/0 (socket says 172.21.15.161:33620) 2024-03-28T23:32:23.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.097+0000 7f5818f51640 1 -- 172.21.15.161:0/3765133317 learned_addr learned my addr 172.21.15.161:0/3765133317 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:23.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.098+0000 7f5813fff640 1 --2- 172.21.15.161:0/3765133317 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5814155d70 0x7f58140c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.098+0000 7f5818f51640 1 -- 172.21.15.161:0/3765133317 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5814155d70 msgr2=0x7f58140c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.098+0000 7f5818f51640 1 --2- 172.21.15.161:0/3765133317 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5814155d70 0x7f58140c9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.098+0000 7f5818f51640 1 -- 172.21.15.161:0/3765133317 >> v1:172.21.15.170:6789/0 conn(0x7f58140c53e0 legacy=0x7f58140c7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.098+0000 7f5818f51640 1 -- 172.21.15.161:0/3765133317 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f581406be50 con 0x7f5814152740 2024-03-28T23:32:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.098+0000 7f5818f51640 1 --2- 172.21.15.161:0/3765133317 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5814152740 0x7f581406b440 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f5804009870 tx=0x7f580402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf0dc0194355f5d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.099+0000 7f58137fe640 1 -- 172.21.15.161:0/3765133317 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58040317c0 con 0x7f5814152740 2024-03-28T23:32:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.099+0000 7f58137fe640 1 -- 172.21.15.161:0/3765133317 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5804037070 con 0x7f5814152740 2024-03-28T23:32:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.099+0000 7f58137fe640 1 -- 172.21.15.161:0/3765133317 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5804031b80 con 0x7f5814152740 2024-03-28T23:32:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.099+0000 7f581b1b5640 1 -- 172.21.15.161:0/3765133317 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5814152740 msgr2=0x7f581406b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.099+0000 7f581b1b5640 1 --2- 172.21.15.161:0/3765133317 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5814152740 0x7f581406b440 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f5804009870 tx=0x7f580402b860 comp rx=0 tx=0).stop 2024-03-28T23:32:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.099+0000 7f581b1b5640 1 -- 172.21.15.161:0/3765133317 shutdown_connections 2024-03-28T23:32:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.099+0000 7f581b1b5640 1 --2- 172.21.15.161:0/3765133317 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5814155d70 0x7f58140c9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.099+0000 7f581b1b5640 1 --2- 172.21.15.161:0/3765133317 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5814152740 0x7f581406b440 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:23.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.099+0000 7f581b1b5640 1 -- 172.21.15.161:0/3765133317 >> 172.21.15.161:0/3765133317 conn(0x7f58140cac40 msgr2=0x7f58140cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:23.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.099+0000 7f581b1b5640 1 -- 172.21.15.161:0/3765133317 shutdown_connections 2024-03-28T23:32:23.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.099+0000 7f581b1b5640 1 -- 172.21.15.161:0/3765133317 wait complete. 2024-03-28T23:32:23.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.100+0000 7f581b1b5640 1 Processor -- start 2024-03-28T23:32:23.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.100+0000 7f581b1b5640 1 -- start start 2024-03-28T23:32:23.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.101+0000 7f581b1b5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58140c53e0 0x7f581414b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:23.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.101+0000 7f581b1b5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5814152740 0x7f58140be570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.101+0000 7f581b1b5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5814155d70 0x7f58140beab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.101+0000 7f581b1b5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f58140c8f20 con 0x7f58140c53e0 2024-03-28T23:32:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.101+0000 7f581b1b5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f58140c8c40 con 0x7f5814155d70 2024-03-28T23:32:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.101+0000 7f581b1b5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f58140c5870 con 0x7f5814152740 2024-03-28T23:32:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.101+0000 7f5818f51640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58140c53e0 0x7f581414b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.101+0000 7f5818f51640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58140c53e0 0x7f581414b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33630/0 (socket says 172.21.15.161:33630) 2024-03-28T23:32:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.101+0000 7f5818f51640 1 -- 172.21.15.161:0/3790221333 learned_addr learned my addr 172.21.15.161:0/3790221333 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.101+0000 7f5818f51640 1 -- 172.21.15.161:0/3790221333 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5814152740 msgr2=0x7f58140be570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.101+0000 7f5818f51640 1 --2- 172.21.15.161:0/3790221333 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5814152740 0x7f58140be570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.101+0000 7f5818f51640 1 -- 172.21.15.161:0/3790221333 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5814155d70 msgr2=0x7f58140beab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.101+0000 7f5819752640 1 --2- 172.21.15.161:0/3790221333 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5814155d70 0x7f58140beab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.102+0000 7f5818f51640 1 --2- 172.21.15.161:0/3790221333 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5814155d70 0x7f58140beab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.102+0000 7f5818f51640 1 -- 172.21.15.161:0/3790221333 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58141eea60 con 0x7f58140c53e0 2024-03-28T23:32:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.102+0000 7f5818f51640 1 --2- 172.21.15.161:0/3790221333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58140c53e0 0x7f581414b1e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f5804000c00 tx=0x7f58040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.102+0000 7f58117fa640 1 -- 172.21.15.161:0/3790221333 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5804032860 con 0x7f58140c53e0 2024-03-28T23:32:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.102+0000 7f58117fa640 1 -- 172.21.15.161:0/3790221333 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5804032a20 con 0x7f58140c53e0 2024-03-28T23:32:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.102+0000 7f581b1b5640 1 -- 172.21.15.161:0/3790221333 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58141eed40 con 0x7f58140c53e0 2024-03-28T23:32:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.102+0000 7f58117fa640 1 -- 172.21.15.161:0/3790221333 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5804032e10 con 0x7f58140c53e0 2024-03-28T23:32:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.102+0000 7f581b1b5640 1 -- 172.21.15.161:0/3790221333 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f581414fc40 con 0x7f58140c53e0 2024-03-28T23:32:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.103+0000 7f581b1b5640 1 -- 172.21.15.161:0/3790221333 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5814068f90 con 0x7f58140c53e0 2024-03-28T23:32:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.103+0000 7f58117fa640 1 -- 172.21.15.161:0/3790221333 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f580404c060 con 0x7f58140c53e0 2024-03-28T23:32:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.104+0000 7f58117fa640 1 --2- 172.21.15.161:0/3790221333 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f580006b4d0 0x7f580006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.104+0000 7f58117fa640 1 -- 172.21.15.161:0/3790221333 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 6095+0+0 (secure 0 0 0) 0x7f58040abba0 con 0x7f58140c53e0 2024-03-28T23:32:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.104+0000 7f5813fff640 1 --2- 172.21.15.161:0/3790221333 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f580006b4d0 0x7f580006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:23.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.104+0000 7f5813fff640 1 --2- 172.21.15.161:0/3790221333 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f580006b4d0 0x7f580006d990 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f58140c4d70 tx=0x7f5808008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:23.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.108+0000 7f58117fa640 1 -- 172.21.15.161:0/3790221333 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5804040070 con 0x7f58140c53e0 2024-03-28T23:32:23.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.285+0000 7f581b1b5640 1 -- 172.21.15.161:0/3790221333 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f58140ca4f0 con 0x7f58140c53e0 2024-03-28T23:32:23.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.977+0000 7f58117fa640 1 -- 172.21.15.161:0/3790221333 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_data' removed v64) ==== 172+0+0 (secure 0 0 0) 0x7f580407a930 con 0x7f58140c53e0 2024-03-28T23:32:23.978 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs2_data' removed 2024-03-28T23:32:23.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.981+0000 7f581b1b5640 1 -- 172.21.15.161:0/3790221333 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f580006b4d0 msgr2=0x7f580006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:23.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.981+0000 7f581b1b5640 1 --2- 172.21.15.161:0/3790221333 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f580006b4d0 0x7f580006d990 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f58140c4d70 tx=0x7f5808008040 comp rx=0 tx=0).stop 2024-03-28T23:32:23.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.981+0000 7f581b1b5640 1 -- 172.21.15.161:0/3790221333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58140c53e0 msgr2=0x7f581414b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:23.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.981+0000 7f581b1b5640 1 --2- 172.21.15.161:0/3790221333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58140c53e0 0x7f581414b1e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f5804000c00 tx=0x7f58040046c0 comp rx=0 tx=0).stop 2024-03-28T23:32:23.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.981+0000 7f581b1b5640 1 -- 172.21.15.161:0/3790221333 shutdown_connections 2024-03-28T23:32:23.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.981+0000 7f581b1b5640 1 --2- 172.21.15.161:0/3790221333 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f580006b4d0 0x7f580006d990 secure :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f58140c4d70 tx=0x7f5808008040 comp rx=0 tx=0).stop 2024-03-28T23:32:23.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.981+0000 7f581b1b5640 1 --2- 172.21.15.161:0/3790221333 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5814155d70 0x7f58140beab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:23.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.981+0000 7f581b1b5640 1 --2- 172.21.15.161:0/3790221333 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5814152740 0x7f58140be570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:23.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.981+0000 7f581b1b5640 1 --2- 172.21.15.161:0/3790221333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58140c53e0 0x7f581414b1e0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:23.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.981+0000 7f581b1b5640 1 -- 172.21.15.161:0/3790221333 >> 172.21.15.161:0/3790221333 conn(0x7f58140cac40 msgr2=0x7f581414de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:23.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.982+0000 7f581b1b5640 1 -- 172.21.15.161:0/3790221333 shutdown_connections 2024-03-28T23:32:23.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:23.982+0000 7f581b1b5640 1 -- 172.21.15.161:0/3790221333 wait complete. 2024-03-28T23:32:23.999 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-28T23:32:24.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.107+0000 7fe7d91a0640 1 Processor -- start 2024-03-28T23:32:24.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.107+0000 7fe7d91a0640 1 -- start start 2024-03-28T23:32:24.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.108+0000 7fe7d91a0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe7d40c4d10 0x7fe7d4153960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:24.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.108+0000 7fe7d91a0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7d406b4a0 0x7fe7d406b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:24.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.108+0000 7fe7d91a0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7d40ad340 con 0x7fe7d4156e10 2024-03-28T23:32:24.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.108+0000 7fe7d91a0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe7d4150890 con 0x7fe7d40c4d10 2024-03-28T23:32:24.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.108+0000 7fe7d91a0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe7d40705e0 con 0x7fe7d406b4a0 2024-03-28T23:32:24.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.108+0000 7fe7d3fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe7d40c4d10 0x7fe7d4153960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:24.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.108+0000 7fe7d3fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe7d40c4d10 0x7fe7d4153960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33646/0 (socket says 172.21.15.161:33646) 2024-03-28T23:32:24.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.108+0000 7fe7d3fff640 1 -- 172.21.15.161:0/2095899904 learned_addr learned my addr 172.21.15.161:0/2095899904 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:24.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.108+0000 7fe7d899f640 1 --2- 172.21.15.161:0/2095899904 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7d406b4a0 0x7fe7d406b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:24.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.108+0000 7fe7d3fff640 1 -- 172.21.15.161:0/2095899904 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7d406b4a0 msgr2=0x7fe7d406b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:24.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.108+0000 7fe7d3fff640 1 --2- 172.21.15.161:0/2095899904 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7d406b4a0 0x7fe7d406b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.108+0000 7fe7d3fff640 1 -- 172.21.15.161:0/2095899904 >> v1:172.21.15.170:6789/0 conn(0x7fe7d4156e10 legacy=0x7fe7d40acc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:24.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.108+0000 7fe7d3fff640 1 -- 172.21.15.161:0/2095899904 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d40708c0 con 0x7fe7d40c4d10 2024-03-28T23:32:24.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.108+0000 7fe7d899f640 1 --2- 172.21.15.161:0/2095899904 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7d406b4a0 0x7fe7d406b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:24.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.109+0000 7fe7d3fff640 1 --2- 172.21.15.161:0/2095899904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7d40c4d10 0x7fe7d4153960 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4009870 tx=0x7fe7c402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eda49d3dee3210 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:24.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.109+0000 7fe7d2ffd640 1 -- 172.21.15.161:0/2095899904 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4035020 con 0x7fe7d40c4d10 2024-03-28T23:32:24.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.109+0000 7fe7d2ffd640 1 -- 172.21.15.161:0/2095899904 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7c4031640 con 0x7fe7d40c4d10 2024-03-28T23:32:24.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.109+0000 7fe7d2ffd640 1 -- 172.21.15.161:0/2095899904 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c402d040 con 0x7fe7d40c4d10 2024-03-28T23:32:24.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.109+0000 7fe7d91a0640 1 -- 172.21.15.161:0/2095899904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7d40c4d10 msgr2=0x7fe7d4153960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:24.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.109+0000 7fe7d91a0640 1 --2- 172.21.15.161:0/2095899904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7d40c4d10 0x7fe7d4153960 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4009870 tx=0x7fe7c402b8f0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.110+0000 7fe7d91a0640 1 -- 172.21.15.161:0/2095899904 shutdown_connections 2024-03-28T23:32:24.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.110+0000 7fe7d91a0640 1 --2- 172.21.15.161:0/2095899904 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7d406b4a0 0x7fe7d406b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.110+0000 7fe7d91a0640 1 --2- 172.21.15.161:0/2095899904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7d40c4d10 0x7fe7d4153960 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.110+0000 7fe7d91a0640 1 -- 172.21.15.161:0/2095899904 >> 172.21.15.161:0/2095899904 conn(0x7fe7d40c8ae0 msgr2=0x7fe7d40c8ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:24.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.110+0000 7fe7d91a0640 1 -- 172.21.15.161:0/2095899904 shutdown_connections 2024-03-28T23:32:24.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.110+0000 7fe7d91a0640 1 -- 172.21.15.161:0/2095899904 wait complete. 2024-03-28T23:32:24.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.111+0000 7fe7d91a0640 1 Processor -- start 2024-03-28T23:32:24.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.111+0000 7fe7d91a0640 1 -- start start 2024-03-28T23:32:24.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.111+0000 7fe7d91a0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7d406b4a0 0x7fe7d406c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:24.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.111+0000 7fe7d91a0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7d4156e10 0x7fe7d40c1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:24.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.111+0000 7fe7d91a0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7d40a9b80 0x7fe7d406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:24.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.111+0000 7fe7d91a0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d4152e10 con 0x7fe7d406b4a0 2024-03-28T23:32:24.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.111+0000 7fe7d91a0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d4153440 con 0x7fe7d4156e10 2024-03-28T23:32:24.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.111+0000 7fe7d91a0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe7d4157270 con 0x7fe7d40a9b80 2024-03-28T23:32:24.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.112+0000 7fe7d3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7d406b4a0 0x7fe7d406c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:24.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.112+0000 7fe7d3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7d406b4a0 0x7fe7d406c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33660/0 (socket says 172.21.15.161:33660) 2024-03-28T23:32:24.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.112+0000 7fe7d3fff640 1 -- 172.21.15.161:0/3735007869 learned_addr learned my addr 172.21.15.161:0/3735007869 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:24.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.112+0000 7fe7d37fe640 1 --2- 172.21.15.161:0/3735007869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7d4156e10 0x7fe7d40c1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:24.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.112+0000 7fe7d899f640 1 --2- 172.21.15.161:0/3735007869 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7d40a9b80 0x7fe7d406d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:24.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.112+0000 7fe7d3fff640 1 -- 172.21.15.161:0/3735007869 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7d40a9b80 msgr2=0x7fe7d406d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:24.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.112+0000 7fe7d3fff640 1 --2- 172.21.15.161:0/3735007869 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7d40a9b80 0x7fe7d406d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.112+0000 7fe7d3fff640 1 -- 172.21.15.161:0/3735007869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7d4156e10 msgr2=0x7fe7d40c1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:24.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.112+0000 7fe7d3fff640 1 --2- 172.21.15.161:0/3735007869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7d4156e10 0x7fe7d40c1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.112+0000 7fe7d3fff640 1 -- 172.21.15.161:0/3735007869 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d40a82b0 con 0x7fe7d406b4a0 2024-03-28T23:32:24.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.112+0000 7fe7d3fff640 1 --2- 172.21.15.161:0/3735007869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7d406b4a0 0x7fe7d406c850 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fe7c402b930 tx=0x7fe7c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:24.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.112+0000 7fe7d0ff9640 1 -- 172.21.15.161:0/3735007869 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4040070 con 0x7fe7d406b4a0 2024-03-28T23:32:24.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.112+0000 7fe7d0ff9640 1 -- 172.21.15.161:0/3735007869 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7c4039c10 con 0x7fe7d406b4a0 2024-03-28T23:32:24.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.112+0000 7fe7d91a0640 1 -- 172.21.15.161:0/3735007869 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7d40a89f0 con 0x7fe7d406b4a0 2024-03-28T23:32:24.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.112+0000 7fe7d0ff9640 1 -- 172.21.15.161:0/3735007869 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4035040 con 0x7fe7d406b4a0 2024-03-28T23:32:24.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.112+0000 7fe7d91a0640 1 -- 172.21.15.161:0/3735007869 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7d4155890 con 0x7fe7d406b4a0 2024-03-28T23:32:24.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.113+0000 7fe7d91a0640 1 -- 172.21.15.161:0/3735007869 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7d40a8f10 con 0x7fe7d406b4a0 2024-03-28T23:32:24.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.114+0000 7fe7d0ff9640 1 -- 172.21.15.161:0/3735007869 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe7c404b060 con 0x7fe7d406b4a0 2024-03-28T23:32:24.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.114+0000 7fe7d0ff9640 1 --2- 172.21.15.161:0/3735007869 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe7c006b4d0 0x7fe7c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:24.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.114+0000 7fe7d0ff9640 1 -- 172.21.15.161:0/3735007869 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5679+0+0 (secure 0 0 0) 0x7fe7c40abc80 con 0x7fe7d406b4a0 2024-03-28T23:32:24.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.114+0000 7fe7d37fe640 1 --2- 172.21.15.161:0/3735007869 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe7c006b4d0 0x7fe7c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:24.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.115+0000 7fe7d37fe640 1 --2- 172.21.15.161:0/3735007869 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe7c006b4d0 0x7fe7c006d990 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8006fd0 tx=0x7fe7c8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:24.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.118+0000 7fe7d0ff9640 1 -- 172.21.15.161:0/3735007869 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe7d40a8f10 con 0x7fe7d406b4a0 2024-03-28T23:32:24.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.290+0000 7fe7d91a0640 1 -- 172.21.15.161:0/3735007869 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fe7d404f4b0 con 0x7fe7d406b4a0 2024-03-28T23:32:24.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.295+0000 7fe7d0ff9640 1 -- 172.21.15.161:0/3735007869 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) ==== 104+0+0 (secure 0 0 0) 0x7fe7c407abb0 con 0x7fe7d406b4a0 2024-03-28T23:32:24.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.297+0000 7fe7d91a0640 1 -- 172.21.15.161:0/3735007869 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe7c006b4d0 msgr2=0x7fe7c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:24.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.297+0000 7fe7d91a0640 1 --2- 172.21.15.161:0/3735007869 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe7c006b4d0 0x7fe7c006d990 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8006fd0 tx=0x7fe7c8008040 comp rx=0 tx=0).stop 2024-03-28T23:32:24.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.297+0000 7fe7d91a0640 1 -- 172.21.15.161:0/3735007869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7d406b4a0 msgr2=0x7fe7d406c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:24.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.297+0000 7fe7d91a0640 1 --2- 172.21.15.161:0/3735007869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7d406b4a0 0x7fe7d406c850 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fe7c402b930 tx=0x7fe7c40046c0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.297+0000 7fe7d91a0640 1 -- 172.21.15.161:0/3735007869 shutdown_connections 2024-03-28T23:32:24.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.297+0000 7fe7d91a0640 1 --2- 172.21.15.161:0/3735007869 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe7c006b4d0 0x7fe7c006d990 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.297+0000 7fe7d91a0640 1 --2- 172.21.15.161:0/3735007869 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7d40a9b80 0x7fe7d406d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.297+0000 7fe7d91a0640 1 --2- 172.21.15.161:0/3735007869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7d4156e10 0x7fe7d40c1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.297+0000 7fe7d91a0640 1 --2- 172.21.15.161:0/3735007869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7d406b4a0 0x7fe7d406c850 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.297+0000 7fe7d91a0640 1 -- 172.21.15.161:0/3735007869 >> 172.21.15.161:0/3735007869 conn(0x7fe7d40c8ae0 msgr2=0x7fe7d40c5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:24.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.297+0000 7fe7d91a0640 1 -- 172.21.15.161:0/3735007869 shutdown_connections 2024-03-28T23:32:24.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.297+0000 7fe7d91a0640 1 -- 172.21.15.161:0/3735007869 wait complete. 2024-03-28T23:32:24.308 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.c mds_join_fs 2024-03-28T23:32:24.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.410+0000 7f7bdb888640 1 Processor -- start 2024-03-28T23:32:24.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.411+0000 7f7bdb888640 1 -- start start 2024-03-28T23:32:24.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.411+0000 7f7bdb888640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7bd4152bc0 0x7f7bd40c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:24.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.411+0000 7f7bdb888640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7bd41589b0 0x7f7bd4158e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:24.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.411+0000 7f7bdb888640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bd40c6f60 con 0x7f7bd40c8a80 2024-03-28T23:32:24.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.411+0000 7f7bdb888640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f7bd40c6120 con 0x7f7bd41589b0 2024-03-28T23:32:24.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.411+0000 7f7bdb888640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f7bd40c6400 con 0x7f7bd4152bc0 2024-03-28T23:32:24.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.411+0000 7f7bd9e25640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7bd41589b0 0x7f7bd4158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:24.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.411+0000 7f7bd8e23640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f7bd40c8a80 0x7f7bd40c8ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:43278/0 (socket says 172.21.15.161:43278) 2024-03-28T23:32:24.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.411+0000 7f7bd8e23640 1 -- 172.21.15.161:0/3754072364 learned_addr learned my addr 172.21.15.161:0/3754072364 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:24.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.412+0000 7f7bd9624640 1 --2- 172.21.15.161:0/3754072364 >> v2:172.21.15.170:3300/0 conn(0x7f7bd4152bc0 0x7f7bd40c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:24.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.412+0000 7f7bd9e25640 1 -- 172.21.15.161:0/3754072364 >> v2:172.21.15.170:3300/0 conn(0x7f7bd4152bc0 msgr2=0x7f7bd40c0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:24.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.412+0000 7f7bd9e25640 1 --2- 172.21.15.161:0/3754072364 >> v2:172.21.15.170:3300/0 conn(0x7f7bd4152bc0 0x7f7bd40c0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.412+0000 7f7bd9e25640 1 -- 172.21.15.161:0/3754072364 >> v1:172.21.15.161:6789/0 conn(0x7f7bd40c8a80 legacy=0x7f7bd40c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:24.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.412+0000 7f7bd9e25640 1 -- 172.21.15.161:0/3754072364 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bd40c7240 con 0x7f7bd41589b0 2024-03-28T23:32:24.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.412+0000 7f7bcbfff640 1 -- 172.21.15.161:0/3754072364 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 510842304 0 0) 0x7f7bd40c6f60 con 0x7f7bd40c8a80 2024-03-28T23:32:24.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.412+0000 7f7bd9e25640 1 --2- 172.21.15.161:0/3754072364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd41589b0 0x7f7bd4158e00 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f7bd40aa410 tx=0x7f7bcc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd83eeef8b6b5516 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:24.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.413+0000 7f7bcbfff640 1 -- 172.21.15.161:0/3754072364 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc035020 con 0x7f7bd41589b0 2024-03-28T23:32:24.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.413+0000 7f7bcbfff640 1 -- 172.21.15.161:0/3754072364 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bcc031830 con 0x7f7bd41589b0 2024-03-28T23:32:24.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.413+0000 7f7bcbfff640 1 -- 172.21.15.161:0/3754072364 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc031c00 con 0x7f7bd41589b0 2024-03-28T23:32:24.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.413+0000 7f7bdb888640 1 -- 172.21.15.161:0/3754072364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd41589b0 msgr2=0x7f7bd4158e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:24.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.413+0000 7f7bdb888640 1 --2- 172.21.15.161:0/3754072364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd41589b0 0x7f7bd4158e00 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f7bd40aa410 tx=0x7f7bcc02b8d0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.413+0000 7f7bdb888640 1 -- 172.21.15.161:0/3754072364 shutdown_connections 2024-03-28T23:32:24.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.413+0000 7f7bdb888640 1 --2- 172.21.15.161:0/3754072364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd41589b0 0x7f7bd4158e00 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.413+0000 7f7bdb888640 1 --2- 172.21.15.161:0/3754072364 >> v2:172.21.15.170:3300/0 conn(0x7f7bd4152bc0 0x7f7bd40c0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.413+0000 7f7bdb888640 1 -- 172.21.15.161:0/3754072364 >> 172.21.15.161:0/3754072364 conn(0x7f7bd40caa80 msgr2=0x7f7bd40cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:24.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.413+0000 7f7bdb888640 1 -- 172.21.15.161:0/3754072364 shutdown_connections 2024-03-28T23:32:24.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.413+0000 7f7bdb888640 1 -- 172.21.15.161:0/3754072364 wait complete. 2024-03-28T23:32:24.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.414+0000 7f7bdb888640 1 Processor -- start 2024-03-28T23:32:24.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.414+0000 7f7bdb888640 1 -- start start 2024-03-28T23:32:24.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.415+0000 7f7bdb888640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd40c8a80 0x7f7bd40ab420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:24.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.415+0000 7f7bdb888640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7bd4152bc0 0x7f7bd40af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:24.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.415+0000 7f7bdb888640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bd41589b0 0x7f7bd40af8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:24.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.415+0000 7f7bdb888640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd4152fa0 con 0x7f7bd40c8a80 2024-03-28T23:32:24.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.415+0000 7f7bdb888640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd40c95f0 con 0x7f7bd4152bc0 2024-03-28T23:32:24.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.415+0000 7f7bdb888640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7bd40aeee0 con 0x7f7bd41589b0 2024-03-28T23:32:24.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.415+0000 7f7bd9624640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd40c8a80 0x7f7bd40ab420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:24.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.415+0000 7f7bd9624640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd40c8a80 0x7f7bd40ab420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33678/0 (socket says 172.21.15.161:33678) 2024-03-28T23:32:24.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.415+0000 7f7bd9624640 1 -- 172.21.15.161:0/1883072771 learned_addr learned my addr 172.21.15.161:0/1883072771 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:24.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.415+0000 7f7bd8e23640 1 --2- 172.21.15.161:0/1883072771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7bd4152bc0 0x7f7bd40af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:24.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.415+0000 7f7bd9e25640 1 --2- 172.21.15.161:0/1883072771 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bd41589b0 0x7f7bd40af8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:24.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.415+0000 7f7bd9624640 1 -- 172.21.15.161:0/1883072771 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bd41589b0 msgr2=0x7f7bd40af8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:24.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.415+0000 7f7bd9624640 1 --2- 172.21.15.161:0/1883072771 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bd41589b0 0x7f7bd40af8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.415+0000 7f7bd9624640 1 -- 172.21.15.161:0/1883072771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7bd4152bc0 msgr2=0x7f7bd40af3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:24.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.415+0000 7f7bd9624640 1 --2- 172.21.15.161:0/1883072771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7bd4152bc0 0x7f7bd40af3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.415+0000 7f7bd9624640 1 -- 172.21.15.161:0/1883072771 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bd414b420 con 0x7f7bd40c8a80 2024-03-28T23:32:24.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.416+0000 7f7bd9624640 1 --2- 172.21.15.161:0/1883072771 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd40c8a80 0x7f7bd40ab420 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0007f20 tx=0x7f7bd000e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:24.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.416+0000 7f7bc9ffb640 1 -- 172.21.15.161:0/1883072771 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd0014620 con 0x7f7bd40c8a80 2024-03-28T23:32:24.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.416+0000 7f7bc9ffb640 1 -- 172.21.15.161:0/1883072771 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bd0019070 con 0x7f7bd40c8a80 2024-03-28T23:32:24.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.416+0000 7f7bc9ffb640 1 -- 172.21.15.161:0/1883072771 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd0014a00 con 0x7f7bd40c8a80 2024-03-28T23:32:24.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.416+0000 7f7bdb888640 1 -- 172.21.15.161:0/1883072771 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bd414b700 con 0x7f7bd40c8a80 2024-03-28T23:32:24.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.416+0000 7f7bdb888640 1 -- 172.21.15.161:0/1883072771 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bd40abff0 con 0x7f7bd40c8a80 2024-03-28T23:32:24.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.417+0000 7f7bdb888640 1 -- 172.21.15.161:0/1883072771 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bd404f4b0 con 0x7f7bd40c8a80 2024-03-28T23:32:24.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.417+0000 7f7bc9ffb640 1 -- 172.21.15.161:0/1883072771 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7bd0014ba0 con 0x7f7bd40c8a80 2024-03-28T23:32:24.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.417+0000 7f7bc9ffb640 1 --2- 172.21.15.161:0/1883072771 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7bb806b4d0 0x7f7bb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:24.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.418+0000 7f7bc9ffb640 1 -- 172.21.15.161:0/1883072771 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5679+0+0 (secure 0 0 0) 0x7f7bd008ab00 con 0x7f7bd40c8a80 2024-03-28T23:32:24.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.418+0000 7f7bd8e23640 1 --2- 172.21.15.161:0/1883072771 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7bb806b4d0 0x7f7bb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:24.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.418+0000 7f7bd8e23640 1 --2- 172.21.15.161:0/1883072771 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7bb806b4d0 0x7f7bb806d990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4152560 tx=0x7f7bc4007b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:24.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.422+0000 7f7bc9ffb640 1 -- 172.21.15.161:0/1883072771 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7bd0059a30 con 0x7f7bd40c8a80 2024-03-28T23:32:24.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.594+0000 7f7bdb888640 1 -- 172.21.15.161:0/1883072771 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"} v 0) -- 0x7f7bd414cac0 con 0x7f7bd40c8a80 2024-03-28T23:32:24.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.600+0000 7f7bc9ffb640 1 -- 172.21.15.161:0/1883072771 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"}]=0 v20) ==== 96+0+0 (secure 0 0 0) 0x7f7bd0059330 con 0x7f7bd40c8a80 2024-03-28T23:32:24.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.602+0000 7f7bdb888640 1 -- 172.21.15.161:0/1883072771 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7bb806b4d0 msgr2=0x7f7bb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:24.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.602+0000 7f7bdb888640 1 --2- 172.21.15.161:0/1883072771 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7bb806b4d0 0x7f7bb806d990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4152560 tx=0x7f7bc4007b50 comp rx=0 tx=0).stop 2024-03-28T23:32:24.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.602+0000 7f7bdb888640 1 -- 172.21.15.161:0/1883072771 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd40c8a80 msgr2=0x7f7bd40ab420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:24.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.602+0000 7f7bdb888640 1 --2- 172.21.15.161:0/1883072771 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd40c8a80 0x7f7bd40ab420 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0007f20 tx=0x7f7bd000e570 comp rx=0 tx=0).stop 2024-03-28T23:32:24.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.602+0000 7f7bdb888640 1 -- 172.21.15.161:0/1883072771 shutdown_connections 2024-03-28T23:32:24.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.602+0000 7f7bdb888640 1 --2- 172.21.15.161:0/1883072771 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7bb806b4d0 0x7f7bb806d990 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.602+0000 7f7bdb888640 1 --2- 172.21.15.161:0/1883072771 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bd41589b0 0x7f7bd40af8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.602+0000 7f7bdb888640 1 --2- 172.21.15.161:0/1883072771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7bd4152bc0 0x7f7bd40af3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.602+0000 7f7bdb888640 1 --2- 172.21.15.161:0/1883072771 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd40c8a80 0x7f7bd40ab420 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.602+0000 7f7bdb888640 1 -- 172.21.15.161:0/1883072771 >> 172.21.15.161:0/1883072771 conn(0x7f7bd40caa80 msgr2=0x7f7bd414cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:24.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.602+0000 7f7bdb888640 1 -- 172.21.15.161:0/1883072771 shutdown_connections 2024-03-28T23:32:24.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.602+0000 7f7bdb888640 1 -- 172.21.15.161:0/1883072771 wait complete. 2024-03-28T23:32:24.614 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-03-28T23:32:24.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.715+0000 7faeac5c2640 1 Processor -- start 2024-03-28T23:32:24.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.715+0000 7faeac5c2640 1 -- start start 2024-03-28T23:32:24.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.716+0000 7faeac5c2640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7faea406cdd0 0x7faea41524c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:24.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.716+0000 7faeac5c2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faea406ba50 0x7faea4152a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:24.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.716+0000 7faeac5c2640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faea40c2f10 con 0x7faea414f5c0 2024-03-28T23:32:24.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.716+0000 7faeac5c2640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7faea40c74d0 con 0x7faea406cdd0 2024-03-28T23:32:24.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.716+0000 7faeac5c2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7faea40c6fa0 con 0x7faea406ba50 2024-03-28T23:32:24.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.716+0000 7faeaa35e640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7faea414f5c0 0x7faea40c2150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:43286/0 (socket says 172.21.15.161:43286) 2024-03-28T23:32:24.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.716+0000 7faeaa35e640 1 -- 172.21.15.161:0/2552338227 learned_addr learned my addr 172.21.15.161:0/2552338227 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:24.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.716+0000 7faea935c640 1 -- 172.21.15.161:0/2552338227 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3827152891 0 0) 0x7fae9c001490 con 0x7faea414f5c0 2024-03-28T23:32:24.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.716+0000 7faea9b5d640 1 --2- 172.21.15.161:0/2552338227 >> v2:172.21.15.170:3300/0 conn(0x7faea406cdd0 0x7faea41524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:24.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.716+0000 7faeaab5f640 1 --2- 172.21.15.161:0/2552338227 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faea406ba50 0x7faea4152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:24.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.716+0000 7faea935c640 1 -- 172.21.15.161:0/2552338227 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae90003520 con 0x7faea414f5c0 2024-03-28T23:32:24.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faea935c640 1 -- 172.21.15.161:0/2552338227 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2628850035 0 0) 0x7fae90003520 con 0x7faea414f5c0 2024-03-28T23:32:24.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faea935c640 1 -- 172.21.15.161:0/2552338227 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faea406ba50 msgr2=0x7faea4152a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:24.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faea935c640 1 --2- 172.21.15.161:0/2552338227 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faea406ba50 0x7faea4152a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faea935c640 1 -- 172.21.15.161:0/2552338227 >> v2:172.21.15.170:3300/0 conn(0x7faea406cdd0 msgr2=0x7faea41524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:24.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faea935c640 1 --2- 172.21.15.161:0/2552338227 >> v2:172.21.15.170:3300/0 conn(0x7faea406cdd0 0x7faea41524c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faea935c640 1 -- 172.21.15.161:0/2552338227 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faea414f9a0 con 0x7faea414f5c0 2024-03-28T23:32:24.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faea935c640 1 -- 172.21.15.161:0/2552338227 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fae9c001bc0 con 0x7faea414f5c0 2024-03-28T23:32:24.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faeaab5f640 1 --2- 172.21.15.161:0/2552338227 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faea406ba50 0x7faea4152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:24.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faea9b5d640 1 --2- 172.21.15.161:0/2552338227 >> v2:172.21.15.170:3300/0 conn(0x7faea406cdd0 0x7faea41524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:24.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faea935c640 1 -- 172.21.15.161:0/2552338227 >> v1:172.21.15.161:6789/0 conn(0x7faea414f5c0 legacy=0x7faea40c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:24.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faea935c640 1 --2- 172.21.15.161:0/2552338227 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faea406ba50 0x7fae90003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:24.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faea935c640 1 --2- 172.21.15.161:0/2552338227 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae90004080 0x7fae90004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:24.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faea935c640 1 --2- 172.21.15.161:0/2552338227 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae90004d40 0x7fae900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:24.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faea935c640 1 -- 172.21.15.161:0/2552338227 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7faea40c74d0 con 0x7fae90004080 2024-03-28T23:32:24.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faea935c640 1 -- 172.21.15.161:0/2552338227 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7faea40c6fa0 con 0x7fae90004d40 2024-03-28T23:32:24.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faea935c640 1 -- 172.21.15.161:0/2552338227 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fae9000dd90 con 0x7faea406ba50 2024-03-28T23:32:24.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faea9b5d640 1 --2- 172.21.15.161:0/2552338227 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae90004080 0x7fae90004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:24.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faeaa35e640 1 --2- 172.21.15.161:0/2552338227 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faea406ba50 0x7fae90003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:24.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faeaab5f640 1 --2- 172.21.15.161:0/2552338227 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae90004d40 0x7fae900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:24.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faea9b5d640 1 -- 172.21.15.161:0/2552338227 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faea406ba50 msgr2=0x7fae90003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:24.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faea9b5d640 1 --2- 172.21.15.161:0/2552338227 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faea406ba50 0x7fae90003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faea9b5d640 1 -- 172.21.15.161:0/2552338227 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae90004d40 msgr2=0x7fae900051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:24.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faea9b5d640 1 --2- 172.21.15.161:0/2552338227 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae90004d40 0x7fae900051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faea9b5d640 1 -- 172.21.15.161:0/2552338227 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae90003520 con 0x7fae90004080 2024-03-28T23:32:24.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.717+0000 7faeaab5f640 1 --2- 172.21.15.161:0/2552338227 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae90004d40 0x7fae900051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:24.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.718+0000 7faea9b5d640 1 --2- 172.21.15.161:0/2552338227 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae90004080 0x7fae90004800 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fae94006fd0 tx=0x7fae90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73131896091b9ba4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:24.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.718+0000 7faea935c640 1 -- 172.21.15.161:0/2552338227 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae94035020 con 0x7fae90004080 2024-03-28T23:32:24.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.718+0000 7faea935c640 1 -- 172.21.15.161:0/2552338227 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae94031640 con 0x7fae90004080 2024-03-28T23:32:24.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.718+0000 7faea935c640 1 -- 172.21.15.161:0/2552338227 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9400f040 con 0x7fae90004080 2024-03-28T23:32:24.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.718+0000 7faeac5c2640 1 -- 172.21.15.161:0/2552338227 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae90004080 msgr2=0x7fae90004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:24.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.718+0000 7faeac5c2640 1 --2- 172.21.15.161:0/2552338227 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae90004080 0x7fae90004800 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fae94006fd0 tx=0x7fae90005830 comp rx=0 tx=0).stop 2024-03-28T23:32:24.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.718+0000 7faeaab5f640 1 -- 172.21.15.161:0/2552338227 reap_dead start 2024-03-28T23:32:24.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.719+0000 7faeac5c2640 1 -- 172.21.15.161:0/2552338227 shutdown_connections 2024-03-28T23:32:24.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.719+0000 7faeac5c2640 1 -- 172.21.15.161:0/2552338227 >> 172.21.15.161:0/2552338227 conn(0x7faea40a83f0 msgr2=0x7faea40a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:24.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.719+0000 7faeac5c2640 1 -- 172.21.15.161:0/2552338227 shutdown_connections 2024-03-28T23:32:24.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.719+0000 7faeac5c2640 1 -- 172.21.15.161:0/2552338227 wait complete. 2024-03-28T23:32:24.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.720+0000 7faeac5c2640 1 Processor -- start 2024-03-28T23:32:24.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.720+0000 7faeac5c2640 1 -- start start 2024-03-28T23:32:24.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.720+0000 7faeac5c2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae90004080 0x7faea40c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:24.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.720+0000 7faeac5c2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faea40c4ee0 0x7faea41488c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:24.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.720+0000 7faeac5c2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faea4155f80 0x7faea4148e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:24.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.720+0000 7faeac5c2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7faea406c140 con 0x7faea4155f80 2024-03-28T23:32:24.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.720+0000 7faeac5c2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7faea4152030 con 0x7fae90004080 2024-03-28T23:32:24.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.720+0000 7faeac5c2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7faea406c3e0 con 0x7faea40c4ee0 2024-03-28T23:32:24.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.720+0000 7faeaab5f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faea4155f80 0x7faea4148e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:24.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.720+0000 7faeaab5f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faea4155f80 0x7faea4148e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33688/0 (socket says 172.21.15.161:33688) 2024-03-28T23:32:24.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.720+0000 7faeaab5f640 1 -- 172.21.15.161:0/62180451 learned_addr learned my addr 172.21.15.161:0/62180451 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:24.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.721+0000 7faea9b5d640 1 --2- 172.21.15.161:0/62180451 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faea40c4ee0 0x7faea41488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:24.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.721+0000 7faeaa35e640 1 --2- 172.21.15.161:0/62180451 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae90004080 0x7faea40c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:24.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.721+0000 7faeaab5f640 1 -- 172.21.15.161:0/62180451 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faea40c4ee0 msgr2=0x7faea41488c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:24.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.721+0000 7faeaab5f640 1 --2- 172.21.15.161:0/62180451 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faea40c4ee0 0x7faea41488c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.721+0000 7faeaab5f640 1 -- 172.21.15.161:0/62180451 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae90004080 msgr2=0x7faea40c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:24.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.721+0000 7faeaab5f640 1 --2- 172.21.15.161:0/62180451 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae90004080 0x7faea40c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.721+0000 7faeaab5f640 1 -- 172.21.15.161:0/62180451 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faea40732a0 con 0x7faea4155f80 2024-03-28T23:32:24.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.721+0000 7faeaa35e640 1 --2- 172.21.15.161:0/62180451 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae90004080 0x7faea40c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:24.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.721+0000 7faeaab5f640 1 --2- 172.21.15.161:0/62180451 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faea4155f80 0x7faea4148e00 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7faea40a7cf0 tx=0x7faea00108b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:24.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.721+0000 7fae9affd640 1 -- 172.21.15.161:0/62180451 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea0018020 con 0x7faea4155f80 2024-03-28T23:32:24.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.721+0000 7fae9affd640 1 -- 172.21.15.161:0/62180451 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faea0004440 con 0x7faea4155f80 2024-03-28T23:32:24.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.721+0000 7fae9affd640 1 -- 172.21.15.161:0/62180451 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea000f040 con 0x7faea4155f80 2024-03-28T23:32:24.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.721+0000 7faeac5c2640 1 -- 172.21.15.161:0/62180451 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faea41eafa0 con 0x7faea4155f80 2024-03-28T23:32:24.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.721+0000 7faeac5c2640 1 -- 172.21.15.161:0/62180451 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faea40c0a70 con 0x7faea4155f80 2024-03-28T23:32:24.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.722+0000 7faeac5c2640 1 -- 172.21.15.161:0/62180451 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faea404f4b0 con 0x7faea4155f80 2024-03-28T23:32:24.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.723+0000 7fae9affd640 1 -- 172.21.15.161:0/62180451 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7faea001b020 con 0x7faea4155f80 2024-03-28T23:32:24.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.723+0000 7fae9affd640 1 --2- 172.21.15.161:0/62180451 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fae8806b500 0x7fae8806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:24.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.723+0000 7fae9affd640 1 -- 172.21.15.161:0/62180451 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5679+0+0 (secure 0 0 0) 0x7faea0089ce0 con 0x7faea4155f80 2024-03-28T23:32:24.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.723+0000 7faeaa35e640 1 --2- 172.21.15.161:0/62180451 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fae8806b500 0x7fae8806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:24.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.723+0000 7faeaa35e640 1 --2- 172.21.15.161:0/62180451 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fae8806b500 0x7fae8806d9c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fae9c001670 tx=0x7fae9c006040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:24.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.727+0000 7fae9affd640 1 -- 172.21.15.161:0/62180451 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7faea0022020 con 0x7faea4155f80 2024-03-28T23:32:24.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.900+0000 7faeac5c2640 1 -- 172.21.15.161:0/62180451 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) -- 0x7faea406b830 con 0x7faea4155f80 2024-03-28T23:32:24.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.905+0000 7fae9affd640 1 -- 172.21.15.161:0/62180451 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v21) ==== 96+0+0 (secure 0 0 0) 0x7faea0058c10 con 0x7faea4155f80 2024-03-28T23:32:24.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.907+0000 7faeac5c2640 1 -- 172.21.15.161:0/62180451 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fae8806b500 msgr2=0x7fae8806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:24.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.907+0000 7faeac5c2640 1 --2- 172.21.15.161:0/62180451 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fae8806b500 0x7fae8806d9c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fae9c001670 tx=0x7fae9c006040 comp rx=0 tx=0).stop 2024-03-28T23:32:24.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.907+0000 7faeac5c2640 1 -- 172.21.15.161:0/62180451 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faea4155f80 msgr2=0x7faea4148e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:24.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.907+0000 7faeac5c2640 1 --2- 172.21.15.161:0/62180451 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faea4155f80 0x7faea4148e00 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7faea40a7cf0 tx=0x7faea00108b0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.908+0000 7faeac5c2640 1 -- 172.21.15.161:0/62180451 shutdown_connections 2024-03-28T23:32:24.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.908+0000 7faeac5c2640 1 --2- 172.21.15.161:0/62180451 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fae8806b500 0x7fae8806d9c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.908+0000 7faeac5c2640 1 --2- 172.21.15.161:0/62180451 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faea4155f80 0x7faea4148e00 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.908+0000 7faeac5c2640 1 --2- 172.21.15.161:0/62180451 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faea40c4ee0 0x7faea41488c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.908+0000 7faeac5c2640 1 --2- 172.21.15.161:0/62180451 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae90004080 0x7faea40c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:24.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.908+0000 7faeac5c2640 1 -- 172.21.15.161:0/62180451 >> 172.21.15.161:0/62180451 conn(0x7faea40a83f0 msgr2=0x7faea414bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:24.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.908+0000 7faeac5c2640 1 -- 172.21.15.161:0/62180451 shutdown_connections 2024-03-28T23:32:24.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:24.908+0000 7faeac5c2640 1 -- 172.21.15.161:0/62180451 wait complete. 2024-03-28T23:32:24.919 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.b mds_join_fs 2024-03-28T23:32:25.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.023+0000 7fce70cc0640 1 Processor -- start 2024-03-28T23:32:25.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.023+0000 7fce70cc0640 1 -- start start 2024-03-28T23:32:25.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.023+0000 7fce70cc0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fce6c150520 0x7fce6c0ab280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:25.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.023+0000 7fce70cc0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fce6c0c1040 0x7fce6c1535b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:25.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.023+0000 7fce70cc0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce6c0c2f50 con 0x7fce6c0a96f0 2024-03-28T23:32:25.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.023+0000 7fce70cc0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fce6c0a9c20 con 0x7fce6c0c1040 2024-03-28T23:32:25.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.023+0000 7fce70cc0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fce6c150900 con 0x7fce6c150520 2024-03-28T23:32:25.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.023+0000 7fce61d9b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fce6c0c1040 0x7fce6c1535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:25.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.023+0000 7fce6ad9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fce6c0a96f0 0x7fce6c156520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:43300/0 (socket says 172.21.15.161:43300) 2024-03-28T23:32:25.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.024+0000 7fce6ad9d640 1 -- 172.21.15.161:0/2113870070 learned_addr learned my addr 172.21.15.161:0/2113870070 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:25.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.024+0000 7fce61d9b640 1 -- 172.21.15.161:0/2113870070 >> v2:172.21.15.170:3300/0 conn(0x7fce6c150520 msgr2=0x7fce6c0ab280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:25.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.024+0000 7fce61d9b640 1 --2- 172.21.15.161:0/2113870070 >> v2:172.21.15.170:3300/0 conn(0x7fce6c150520 0x7fce6c0ab280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:25.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.024+0000 7fce61d9b640 1 -- 172.21.15.161:0/2113870070 >> v1:172.21.15.161:6789/0 conn(0x7fce6c0a96f0 legacy=0x7fce6c156520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:25.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.024+0000 7fce61d9b640 1 -- 172.21.15.161:0/2113870070 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce6c1530d0 con 0x7fce6c0c1040 2024-03-28T23:32:25.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.024+0000 7fce61d9b640 1 --2- 172.21.15.161:0/2113870070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce6c0c1040 0x7fce6c1535b0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fce6c06e9c0 tx=0x7fce5402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c0382bca4ec4916 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:25.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.025+0000 7fce69d9b640 1 -- 172.21.15.161:0/2113870070 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce540315a0 con 0x7fce6c0c1040 2024-03-28T23:32:25.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.025+0000 7fce69d9b640 1 -- 172.21.15.161:0/2113870070 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce54037070 con 0x7fce6c0c1040 2024-03-28T23:32:25.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.025+0000 7fce69d9b640 1 -- 172.21.15.161:0/2113870070 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce54031960 con 0x7fce6c0c1040 2024-03-28T23:32:25.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.025+0000 7fce70cc0640 1 -- 172.21.15.161:0/2113870070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce6c0c1040 msgr2=0x7fce6c1535b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:25.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.025+0000 7fce70cc0640 1 --2- 172.21.15.161:0/2113870070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce6c0c1040 0x7fce6c1535b0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fce6c06e9c0 tx=0x7fce5402b840 comp rx=0 tx=0).stop 2024-03-28T23:32:25.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.025+0000 7fce70cc0640 1 -- 172.21.15.161:0/2113870070 shutdown_connections 2024-03-28T23:32:25.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.025+0000 7fce70cc0640 1 --2- 172.21.15.161:0/2113870070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce6c0c1040 0x7fce6c1535b0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:25.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.025+0000 7fce70cc0640 1 --2- 172.21.15.161:0/2113870070 >> v2:172.21.15.170:3300/0 conn(0x7fce6c150520 0x7fce6c0ab280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:25.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.025+0000 7fce70cc0640 1 -- 172.21.15.161:0/2113870070 >> 172.21.15.161:0/2113870070 conn(0x7fce6c0a83f0 msgr2=0x7fce6c0a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:25.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.025+0000 7fce70cc0640 1 -- 172.21.15.161:0/2113870070 shutdown_connections 2024-03-28T23:32:25.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.025+0000 7fce70cc0640 1 -- 172.21.15.161:0/2113870070 wait complete. 2024-03-28T23:32:25.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.027+0000 7fce70cc0640 1 Processor -- start 2024-03-28T23:32:25.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.027+0000 7fce70cc0640 1 -- start start 2024-03-28T23:32:25.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.027+0000 7fce70cc0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce6c0a96f0 0x7fce6c0aa950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:25.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.027+0000 7fce70cc0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce6c0c1040 0x7fce6c13e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:25.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.027+0000 7fce70cc0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce6c150520 0x7fce6c13edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:25.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.027+0000 7fce70cc0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fce6c0c0c00 con 0x7fce6c150520 2024-03-28T23:32:25.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.027+0000 7fce70cc0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fce6c0c2c70 con 0x7fce6c0c1040 2024-03-28T23:32:25.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.027+0000 7fce70cc0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fce6c0c0d80 con 0x7fce6c0a96f0 2024-03-28T23:32:25.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.027+0000 7fce6ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce6c150520 0x7fce6c13edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:25.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.027+0000 7fce6ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce6c150520 0x7fce6c13edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33708/0 (socket says 172.21.15.161:33708) 2024-03-28T23:32:25.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.027+0000 7fce6ad9d640 1 -- 172.21.15.161:0/3886036583 learned_addr learned my addr 172.21.15.161:0/3886036583 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:25.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.028+0000 7fce61d9b640 1 --2- 172.21.15.161:0/3886036583 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce6c0c1040 0x7fce6c13e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:25.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.028+0000 7fce6ad9d640 1 -- 172.21.15.161:0/3886036583 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce6c0a96f0 msgr2=0x7fce6c0aa950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:25.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.028+0000 7fce6a59c640 1 --2- 172.21.15.161:0/3886036583 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce6c0a96f0 0x7fce6c0aa950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:25.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.028+0000 7fce6ad9d640 1 --2- 172.21.15.161:0/3886036583 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce6c0a96f0 0x7fce6c0aa950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:25.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.028+0000 7fce6ad9d640 1 -- 172.21.15.161:0/3886036583 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce6c0c1040 msgr2=0x7fce6c13e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:25.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.028+0000 7fce6ad9d640 1 --2- 172.21.15.161:0/3886036583 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce6c0c1040 0x7fce6c13e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:25.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.028+0000 7fce6ad9d640 1 -- 172.21.15.161:0/3886036583 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce6c1e88a0 con 0x7fce6c150520 2024-03-28T23:32:25.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.028+0000 7fce61d9b640 1 --2- 172.21.15.161:0/3886036583 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce6c0c1040 0x7fce6c13e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:25.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.028+0000 7fce6ad9d640 1 --2- 172.21.15.161:0/3886036583 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce6c150520 0x7fce6c13edf0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fce6400caa0 tx=0x7fce6400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:25.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.028+0000 7fce637fe640 1 -- 172.21.15.161:0/3886036583 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce64017070 con 0x7fce6c150520 2024-03-28T23:32:25.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.028+0000 7fce637fe640 1 -- 172.21.15.161:0/3886036583 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce6400aba0 con 0x7fce6c150520 2024-03-28T23:32:25.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.028+0000 7fce70cc0640 1 -- 172.21.15.161:0/3886036583 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce6c1e8b60 con 0x7fce6c150520 2024-03-28T23:32:25.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.028+0000 7fce637fe640 1 -- 172.21.15.161:0/3886036583 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce64012400 con 0x7fce6c150520 2024-03-28T23:32:25.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.028+0000 7fce70cc0640 1 -- 172.21.15.161:0/3886036583 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce6c1e9120 con 0x7fce6c150520 2024-03-28T23:32:25.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.029+0000 7fce70cc0640 1 -- 172.21.15.161:0/3886036583 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce6c1e97e0 con 0x7fce6c150520 2024-03-28T23:32:25.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.030+0000 7fce637fe640 1 -- 172.21.15.161:0/3886036583 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fce6401a070 con 0x7fce6c150520 2024-03-28T23:32:25.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.030+0000 7fce637fe640 1 --2- 172.21.15.161:0/3886036583 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fce4c06b4d0 0x7fce4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:25.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.030+0000 7fce6a59c640 1 --2- 172.21.15.161:0/3886036583 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fce4c06b4d0 0x7fce4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:25.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.030+0000 7fce637fe640 1 -- 172.21.15.161:0/3886036583 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5679+0+0 (secure 0 0 0) 0x7fce64089cc0 con 0x7fce6c150520 2024-03-28T23:32:25.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.030+0000 7fce6a59c640 1 --2- 172.21.15.161:0/3886036583 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fce4c06b4d0 0x7fce4c06d990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fce5c00a770 tx=0x7fce5c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:25.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.035+0000 7fce637fe640 1 -- 172.21.15.161:0/3886036583 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fce6c1e97e0 con 0x7fce6c150520 2024-03-28T23:32:25.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.207+0000 7fce70cc0640 1 -- 172.21.15.161:0/3886036583 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"} v 0) -- 0x7fce6c04f4b0 con 0x7fce6c150520 2024-03-28T23:32:25.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.212+0000 7fce637fe640 1 -- 172.21.15.161:0/3886036583 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"}]=0 v22) ==== 96+0+0 (secure 0 0 0) 0x7fce640585a0 con 0x7fce6c150520 2024-03-28T23:32:25.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.214+0000 7fce70cc0640 1 -- 172.21.15.161:0/3886036583 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fce4c06b4d0 msgr2=0x7fce4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:25.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.214+0000 7fce70cc0640 1 --2- 172.21.15.161:0/3886036583 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fce4c06b4d0 0x7fce4c06d990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fce5c00a770 tx=0x7fce5c008040 comp rx=0 tx=0).stop 2024-03-28T23:32:25.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.214+0000 7fce70cc0640 1 -- 172.21.15.161:0/3886036583 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce6c150520 msgr2=0x7fce6c13edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:25.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.214+0000 7fce70cc0640 1 --2- 172.21.15.161:0/3886036583 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce6c150520 0x7fce6c13edf0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fce6400caa0 tx=0x7fce6400cf60 comp rx=0 tx=0).stop 2024-03-28T23:32:25.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.214+0000 7fce70cc0640 1 -- 172.21.15.161:0/3886036583 shutdown_connections 2024-03-28T23:32:25.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.214+0000 7fce70cc0640 1 --2- 172.21.15.161:0/3886036583 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fce4c06b4d0 0x7fce4c06d990 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:25.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.214+0000 7fce70cc0640 1 --2- 172.21.15.161:0/3886036583 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce6c150520 0x7fce6c13edf0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:25.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.214+0000 7fce70cc0640 1 --2- 172.21.15.161:0/3886036583 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce6c0c1040 0x7fce6c13e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:25.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.214+0000 7fce70cc0640 1 --2- 172.21.15.161:0/3886036583 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce6c0a96f0 0x7fce6c0aa950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:25.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.214+0000 7fce70cc0640 1 -- 172.21.15.161:0/3886036583 >> 172.21.15.161:0/3886036583 conn(0x7fce6c0a83f0 msgr2=0x7fce6c0c1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:25.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.214+0000 7fce70cc0640 1 -- 172.21.15.161:0/3886036583 shutdown_connections 2024-03-28T23:32:25.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.214+0000 7fce70cc0640 1 -- 172.21.15.161:0/3886036583 wait complete. 2024-03-28T23:32:25.226 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort' 2024-03-28T23:32:25.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.327+0000 7fd56f59e640 1 Processor -- start 2024-03-28T23:32:25.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.327+0000 7fd56f59e640 1 -- start start 2024-03-28T23:32:25.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.328+0000 7fd56f59e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd570155290 0x7fd5700a9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:25.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.328+0000 7fd56f59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd5700a6070 0x7fd57006c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:25.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.328+0000 7fd56f59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd57006c760 0x7fd570155670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:25.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.328+0000 7fd56f59e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd57006d380 con 0x7fd570155290 2024-03-28T23:32:25.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.328+0000 7fd56f59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd57006ce30 con 0x7fd5700a6070 2024-03-28T23:32:25.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.328+0000 7fd56f59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd57006d0a0 con 0x7fd57006c760 2024-03-28T23:32:25.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.328+0000 7fd56e59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd570155290 0x7fd5700a9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:25.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.328+0000 7fd56e59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd570155290 0x7fd5700a9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33718/0 (socket says 172.21.15.161:33718) 2024-03-28T23:32:25.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.328+0000 7fd56e59c640 1 -- 172.21.15.161:0/3374409105 learned_addr learned my addr 172.21.15.161:0/3374409105 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:25.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.328+0000 7fd56ed9d640 1 --2- 172.21.15.161:0/3374409105 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd57006c760 0x7fd570155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:25.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.328+0000 7fd567fff640 1 --2- 172.21.15.161:0/3374409105 >> v2:172.21.15.170:3300/0 conn(0x7fd5700a6070 0x7fd57006c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:25.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.328+0000 7fd56e59c640 1 -- 172.21.15.161:0/3374409105 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd57006c760 msgr2=0x7fd570155670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:25.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.328+0000 7fd56e59c640 1 --2- 172.21.15.161:0/3374409105 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd57006c760 0x7fd570155670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:25.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.328+0000 7fd56e59c640 1 -- 172.21.15.161:0/3374409105 >> v2:172.21.15.170:3300/0 conn(0x7fd5700a6070 msgr2=0x7fd57006c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:25.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.328+0000 7fd56e59c640 1 --2- 172.21.15.161:0/3374409105 >> v2:172.21.15.170:3300/0 conn(0x7fd5700a6070 0x7fd57006c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:25.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.328+0000 7fd56e59c640 1 -- 172.21.15.161:0/3374409105 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd57006bd00 con 0x7fd570155290 2024-03-28T23:32:25.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.328+0000 7fd567fff640 1 --2- 172.21.15.161:0/3374409105 >> v2:172.21.15.170:3300/0 conn(0x7fd5700a6070 0x7fd57006c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:25.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.329+0000 7fd56e59c640 1 --2- 172.21.15.161:0/3374409105 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd570155290 0x7fd5700a9d20 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fd5700c0f60 tx=0x7fd56002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70d6dcf99131489c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:25.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.329+0000 7fd56dd9b640 1 -- 172.21.15.161:0/3374409105 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5600317c0 con 0x7fd570155290 2024-03-28T23:32:25.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.329+0000 7fd56dd9b640 1 -- 172.21.15.161:0/3374409105 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd560037070 con 0x7fd570155290 2024-03-28T23:32:25.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.329+0000 7fd56dd9b640 1 -- 172.21.15.161:0/3374409105 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd560031b80 con 0x7fd570155290 2024-03-28T23:32:25.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.329+0000 7fd56f59e640 1 -- 172.21.15.161:0/3374409105 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd570155290 msgr2=0x7fd5700a9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:25.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.329+0000 7fd56f59e640 1 --2- 172.21.15.161:0/3374409105 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd570155290 0x7fd5700a9d20 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fd5700c0f60 tx=0x7fd56002b860 comp rx=0 tx=0).stop 2024-03-28T23:32:25.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.329+0000 7fd56f59e640 1 -- 172.21.15.161:0/3374409105 shutdown_connections 2024-03-28T23:32:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.329+0000 7fd56f59e640 1 --2- 172.21.15.161:0/3374409105 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd57006c760 0x7fd570155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.329+0000 7fd56f59e640 1 --2- 172.21.15.161:0/3374409105 >> v2:172.21.15.170:3300/0 conn(0x7fd5700a6070 0x7fd57006c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.330+0000 7fd56f59e640 1 --2- 172.21.15.161:0/3374409105 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd570155290 0x7fd5700a9d20 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.330+0000 7fd56f59e640 1 -- 172.21.15.161:0/3374409105 >> 172.21.15.161:0/3374409105 conn(0x7fd5700c66d0 msgr2=0x7fd5700c6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.330+0000 7fd56f59e640 1 -- 172.21.15.161:0/3374409105 shutdown_connections 2024-03-28T23:32:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.330+0000 7fd56f59e640 1 -- 172.21.15.161:0/3374409105 wait complete. 2024-03-28T23:32:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.331+0000 7fd56f59e640 1 Processor -- start 2024-03-28T23:32:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.331+0000 7fd56f59e640 1 -- start start 2024-03-28T23:32:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.331+0000 7fd56f59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd57006c760 0x7fd5700bfd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.331+0000 7fd56f59e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5700a6070 0x7fd5701ee300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.331+0000 7fd56f59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd570155290 0x7fd5701ee840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.331+0000 7fd56f59e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd5700a9370 con 0x7fd5700a6070 2024-03-28T23:32:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.331+0000 7fd56f59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd5700a91f0 con 0x7fd570155290 2024-03-28T23:32:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.331+0000 7fd56f59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd570155bb0 con 0x7fd57006c760 2024-03-28T23:32:25.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.332+0000 7fd567fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5700a6070 0x7fd5701ee300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:25.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.332+0000 7fd567fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5700a6070 0x7fd5701ee300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33732/0 (socket says 172.21.15.161:33732) 2024-03-28T23:32:25.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.332+0000 7fd567fff640 1 -- 172.21.15.161:0/1529312134 learned_addr learned my addr 172.21.15.161:0/1529312134 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:25.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.332+0000 7fd567fff640 1 -- 172.21.15.161:0/1529312134 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd57006c760 msgr2=0x7fd5700bfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:25.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.332+0000 7fd567fff640 1 --2- 172.21.15.161:0/1529312134 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd57006c760 0x7fd5700bfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:25.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.332+0000 7fd567fff640 1 -- 172.21.15.161:0/1529312134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd570155290 msgr2=0x7fd5701ee840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:25.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.332+0000 7fd567fff640 1 --2- 172.21.15.161:0/1529312134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd570155290 0x7fd5701ee840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:25.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.332+0000 7fd567fff640 1 -- 172.21.15.161:0/1529312134 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5701eb990 con 0x7fd5700a6070 2024-03-28T23:32:25.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.332+0000 7fd567fff640 1 --2- 172.21.15.161:0/1529312134 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5700a6070 0x7fd5701ee300 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fd55800c7f0 tx=0x7fd55800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:25.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.332+0000 7fd566ffd640 1 -- 172.21.15.161:0/1529312134 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd558015070 con 0x7fd5700a6070 2024-03-28T23:32:25.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.332+0000 7fd566ffd640 1 -- 172.21.15.161:0/1529312134 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd558007a30 con 0x7fd5700a6070 2024-03-28T23:32:25.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.332+0000 7fd56f59e640 1 -- 172.21.15.161:0/1529312134 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5701ead80 con 0x7fd5700a6070 2024-03-28T23:32:25.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.332+0000 7fd566ffd640 1 -- 172.21.15.161:0/1529312134 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd558007e00 con 0x7fd5700a6070 2024-03-28T23:32:25.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.332+0000 7fd56f59e640 1 -- 172.21.15.161:0/1529312134 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5701eb340 con 0x7fd5700a6070 2024-03-28T23:32:25.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.333+0000 7fd56f59e640 1 -- 172.21.15.161:0/1529312134 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd57004f4b0 con 0x7fd5700a6070 2024-03-28T23:32:25.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.334+0000 7fd566ffd640 1 -- 172.21.15.161:0/1529312134 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd558018070 con 0x7fd5700a6070 2024-03-28T23:32:25.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.334+0000 7fd566ffd640 1 --2- 172.21.15.161:0/1529312134 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd55006b500 0x7fd55006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:25.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.334+0000 7fd566ffd640 1 -- 172.21.15.161:0/1529312134 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5679+0+0 (secure 0 0 0) 0x7fd55808a0e0 con 0x7fd5700a6070 2024-03-28T23:32:25.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.334+0000 7fd56e59c640 1 --2- 172.21.15.161:0/1529312134 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd55006b500 0x7fd55006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:25.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.335+0000 7fd56e59c640 1 --2- 172.21.15.161:0/1529312134 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd55006b500 0x7fd55006d9c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd560036000 tx=0x7fd560030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:25.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.338+0000 7fd566ffd640 1 -- 172.21.15.161:0/1529312134 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd558058f80 con 0x7fd5700a6070 2024-03-28T23:32:25.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.576+0000 7fd56f59e640 1 -- 172.21.15.161:0/1529312134 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort"]} v 0) -- 0x7fd5700737c0 con 0x7fd5700a6070 2024-03-28T23:32:25.675 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:32:25.677 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:32:25.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.986+0000 7fd566ffd640 1 -- 172.21.15.161:0/1529312134 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort"]}]=0 v162) ==== 150+0+0 (secure 0 0 0) 0x7fd558058880 con 0x7fd5700a6070 2024-03-28T23:32:25.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.988+0000 7fd56f59e640 1 -- 172.21.15.161:0/1529312134 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd55006b500 msgr2=0x7fd55006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:25.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.988+0000 7fd56f59e640 1 --2- 172.21.15.161:0/1529312134 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd55006b500 0x7fd55006d9c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd560036000 tx=0x7fd560030000 comp rx=0 tx=0).stop 2024-03-28T23:32:25.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.988+0000 7fd56f59e640 1 -- 172.21.15.161:0/1529312134 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5700a6070 msgr2=0x7fd5701ee300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:25.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.988+0000 7fd56f59e640 1 --2- 172.21.15.161:0/1529312134 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5700a6070 0x7fd5701ee300 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fd55800c7f0 tx=0x7fd55800cbf0 comp rx=0 tx=0).stop 2024-03-28T23:32:25.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.988+0000 7fd56f59e640 1 -- 172.21.15.161:0/1529312134 shutdown_connections 2024-03-28T23:32:25.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.988+0000 7fd56f59e640 1 --2- 172.21.15.161:0/1529312134 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd55006b500 0x7fd55006d9c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:25.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.988+0000 7fd56f59e640 1 --2- 172.21.15.161:0/1529312134 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5700a6070 0x7fd5701ee300 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:25.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.988+0000 7fd56f59e640 1 --2- 172.21.15.161:0/1529312134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd570155290 0x7fd5701ee840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:25.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.988+0000 7fd56f59e640 1 --2- 172.21.15.161:0/1529312134 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd57006c760 0x7fd5700bfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:25.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.988+0000 7fd56f59e640 1 -- 172.21.15.161:0/1529312134 >> 172.21.15.161:0/1529312134 conn(0x7fd5700c66d0 msgr2=0x7fd5700c6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:25.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.988+0000 7fd56f59e640 1 -- 172.21.15.161:0/1529312134 shutdown_connections 2024-03-28T23:32:25.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:25.988+0000 7fd56f59e640 1 -- 172.21.15.161:0/1529312134 wait complete. 2024-03-28T23:32:26.000 INFO:tasks.cephfs_test_runner:That a standby with mds_join_fs set to another fs is still used if necessary. ... ok 2024-03-28T23:32:26.000 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_last_resort_refused (tasks.cephfs.test_failover.TestClusterAffinity) That a standby with mds_join_fs set to another fs is not used if refuse_standby_for_another_fs is set. 2024-03-28T23:32:26.000 INFO:tasks.cephfs_test_runner:test_join_fs_last_resort_refused (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-28T23:32:26.001 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:32:26.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.102+0000 7fde48ca4640 1 Processor -- start 2024-03-28T23:32:26.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.102+0000 7fde48ca4640 1 -- start start 2024-03-28T23:32:26.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.102+0000 7fde48ca4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fde4406ce80 0x7fde440aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:26.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.102+0000 7fde48ca4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde440a9680 0x7fde440aab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:26.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.102+0000 7fde48ca4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde4414c250 con 0x7fde4414be70 2024-03-28T23:32:26.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.102+0000 7fde48ca4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fde441529c0 con 0x7fde4406ce80 2024-03-28T23:32:26.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.102+0000 7fde48ca4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fde440a9dc0 con 0x7fde440a9680 2024-03-28T23:32:26.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.102+0000 7fde39d9b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fde4406ce80 0x7fde440aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:26.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.103+0000 7fde4259c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fde4414be70 0x7fde440ab1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:43308/0 (socket says 172.21.15.161:43308) 2024-03-28T23:32:26.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.103+0000 7fde4259c640 1 -- 172.21.15.161:0/1090717339 learned_addr learned my addr 172.21.15.161:0/1090717339 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:26.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.103+0000 7fde42d9d640 1 --2- 172.21.15.161:0/1090717339 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde440a9680 0x7fde440aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:26.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.103+0000 7fde39d9b640 1 -- 172.21.15.161:0/1090717339 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde440a9680 msgr2=0x7fde440aab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:26.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.103+0000 7fde39d9b640 1 --2- 172.21.15.161:0/1090717339 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde440a9680 0x7fde440aab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.103+0000 7fde39d9b640 1 -- 172.21.15.161:0/1090717339 >> v1:172.21.15.161:6789/0 conn(0x7fde4414be70 legacy=0x7fde440ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:26.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.103+0000 7fde39d9b640 1 -- 172.21.15.161:0/1090717339 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde440aa0a0 con 0x7fde4406ce80 2024-03-28T23:32:26.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.103+0000 7fde41d9b640 1 -- 172.21.15.161:0/1090717339 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2820757039 0 0) 0x7fde4414c250 con 0x7fde4414be70 2024-03-28T23:32:26.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.104+0000 7fde39d9b640 1 --2- 172.21.15.161:0/1090717339 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde4406ce80 0x7fde440aa5c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fde2c00ba40 tx=0x7fde2c02df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bac7dbb0434d028 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:26.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.104+0000 7fde41d9b640 1 -- 172.21.15.161:0/1090717339 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c03b070 con 0x7fde4406ce80 2024-03-28T23:32:26.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.104+0000 7fde41d9b640 1 -- 172.21.15.161:0/1090717339 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde2c02ed30 con 0x7fde4406ce80 2024-03-28T23:32:26.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.104+0000 7fde41d9b640 1 -- 172.21.15.161:0/1090717339 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c036660 con 0x7fde4406ce80 2024-03-28T23:32:26.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.104+0000 7fde48ca4640 1 -- 172.21.15.161:0/1090717339 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde4406ce80 msgr2=0x7fde440aa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:26.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.104+0000 7fde48ca4640 1 --2- 172.21.15.161:0/1090717339 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde4406ce80 0x7fde440aa5c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fde2c00ba40 tx=0x7fde2c02df80 comp rx=0 tx=0).stop 2024-03-28T23:32:26.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.104+0000 7fde48ca4640 1 -- 172.21.15.161:0/1090717339 shutdown_connections 2024-03-28T23:32:26.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.104+0000 7fde48ca4640 1 --2- 172.21.15.161:0/1090717339 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde440a9680 0x7fde440aab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.104+0000 7fde48ca4640 1 --2- 172.21.15.161:0/1090717339 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde4406ce80 0x7fde440aa5c0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.104+0000 7fde48ca4640 1 -- 172.21.15.161:0/1090717339 >> 172.21.15.161:0/1090717339 conn(0x7fde440c6670 msgr2=0x7fde440c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:26.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.104+0000 7fde48ca4640 1 -- 172.21.15.161:0/1090717339 shutdown_connections 2024-03-28T23:32:26.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.104+0000 7fde48ca4640 1 -- 172.21.15.161:0/1090717339 wait complete. 2024-03-28T23:32:26.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.106+0000 7fde48ca4640 1 Processor -- start 2024-03-28T23:32:26.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.106+0000 7fde48ca4640 1 -- start start 2024-03-28T23:32:26.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.106+0000 7fde48ca4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde4406ce80 0x7fde44150fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:26.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.106+0000 7fde48ca4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde440a9680 0x7fde44145dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:26.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.106+0000 7fde48ca4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde4414be70 0x7fde44144960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:26.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.106+0000 7fde48ca4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fde440aba30 con 0x7fde4414be70 2024-03-28T23:32:26.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.106+0000 7fde48ca4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fde440ab8b0 con 0x7fde440a9680 2024-03-28T23:32:26.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.106+0000 7fde48ca4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fde4406b150 con 0x7fde4406ce80 2024-03-28T23:32:26.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.106+0000 7fde42d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde4414be70 0x7fde44144960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:26.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.106+0000 7fde42d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde4414be70 0x7fde44144960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33750/0 (socket says 172.21.15.161:33750) 2024-03-28T23:32:26.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.106+0000 7fde42d9d640 1 -- 172.21.15.161:0/3919527773 learned_addr learned my addr 172.21.15.161:0/3919527773 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:26.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.107+0000 7fde4259c640 1 --2- 172.21.15.161:0/3919527773 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde4406ce80 0x7fde44150fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:26.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.107+0000 7fde39d9b640 1 --2- 172.21.15.161:0/3919527773 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde440a9680 0x7fde44145dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:26.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.107+0000 7fde42d9d640 1 -- 172.21.15.161:0/3919527773 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde4406ce80 msgr2=0x7fde44150fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:26.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.107+0000 7fde42d9d640 1 --2- 172.21.15.161:0/3919527773 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde4406ce80 0x7fde44150fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.107+0000 7fde42d9d640 1 -- 172.21.15.161:0/3919527773 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde440a9680 msgr2=0x7fde44145dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:26.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.107+0000 7fde42d9d640 1 --2- 172.21.15.161:0/3919527773 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde440a9680 0x7fde44145dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.107+0000 7fde42d9d640 1 -- 172.21.15.161:0/3919527773 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde44148bc0 con 0x7fde4414be70 2024-03-28T23:32:26.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.107+0000 7fde39d9b640 1 --2- 172.21.15.161:0/3919527773 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde440a9680 0x7fde44145dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:26.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.107+0000 7fde42d9d640 1 --2- 172.21.15.161:0/3919527773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde4414be70 0x7fde44144960 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fde3c007e60 tx=0x7fde3c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:26.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.107+0000 7fde3b7fe640 1 -- 172.21.15.161:0/3919527773 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3c014420 con 0x7fde4414be70 2024-03-28T23:32:26.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.107+0000 7fde3b7fe640 1 -- 172.21.15.161:0/3919527773 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde3c019070 con 0x7fde4414be70 2024-03-28T23:32:26.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.107+0000 7fde3b7fe640 1 -- 172.21.15.161:0/3919527773 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3c014820 con 0x7fde4414be70 2024-03-28T23:32:26.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.107+0000 7fde48ca4640 1 -- 172.21.15.161:0/3919527773 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde441ea970 con 0x7fde4414be70 2024-03-28T23:32:26.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.107+0000 7fde48ca4640 1 -- 172.21.15.161:0/3919527773 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde44151510 con 0x7fde4414be70 2024-03-28T23:32:26.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.108+0000 7fde48ca4640 1 -- 172.21.15.161:0/3919527773 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde4404f4b0 con 0x7fde4414be70 2024-03-28T23:32:26.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.109+0000 7fde3b7fe640 1 -- 172.21.15.161:0/3919527773 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fde3c0149c0 con 0x7fde4414be70 2024-03-28T23:32:26.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.109+0000 7fde3b7fe640 1 --2- 172.21.15.161:0/3919527773 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fde2406b4d0 0x7fde2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:26.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.109+0000 7fde3b7fe640 1 -- 172.21.15.161:0/3919527773 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5679+0+0 (secure 0 0 0) 0x7fde3c08a8d0 con 0x7fde4414be70 2024-03-28T23:32:26.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.109+0000 7fde4259c640 1 --2- 172.21.15.161:0/3919527773 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fde2406b4d0 0x7fde2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:26.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.109+0000 7fde4259c640 1 --2- 172.21.15.161:0/3919527773 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fde2406b4d0 0x7fde2406d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fde34000f60 tx=0x7fde34009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:26.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.113+0000 7fde3b7fe640 1 -- 172.21.15.161:0/3919527773 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fde3c059800 con 0x7fde4414be70 2024-03-28T23:32:26.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.286+0000 7fde48ca4640 1 -- 172.21.15.161:0/3919527773 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fde440c17b0 con 0x7fde4414be70 2024-03-28T23:32:26.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.287+0000 7fde3b7fe640 1 -- 172.21.15.161:0/3919527773 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) ==== 74+0+13858 (secure 0 0 0) 0x7fde3c059100 con 0x7fde4414be70 2024-03-28T23:32:26.288 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:26.288 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":64,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:32:23.976604+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6835/2649192649":"2024-03-29T23:32:20.970068+0000","172.21.15.170:6834/2649192649":"2024-03-29T23:32:20.970068+0000","172.21.15.161:6836/1300583128":"2024-03-29T23:32:15.775261+0000","172.21.15.161:6834/1300583128":"2024-03-29T23:32:15.775261+0000","172.21.15.161:6837/1498528126":"2024-03-29T23:32:14.767288+0000","172.21.15.170:6835/1945122204":"2024-03-29T23:32:06.533661+0000","172.21.15.161:6835/1498528126":"2024-03-29T23:32:14.767288+0000","172.21.15.170:6834/1945122204":"2024-03-29T23:32:06.533661+0000","172.21.15.161:6837/544971001":"2024-03-29T23:32:03.518509+0000","172.21.15.161:6835/544971001":"2024-03-29T23:32:03.518509+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:32:26.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.289+0000 7fde48ca4640 1 -- 172.21.15.161:0/3919527773 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fde2406b4d0 msgr2=0x7fde2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:26.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.289+0000 7fde48ca4640 1 --2- 172.21.15.161:0/3919527773 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fde2406b4d0 0x7fde2406d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fde34000f60 tx=0x7fde34009040 comp rx=0 tx=0).stop 2024-03-28T23:32:26.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.289+0000 7fde48ca4640 1 -- 172.21.15.161:0/3919527773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde4414be70 msgr2=0x7fde44144960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:26.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.289+0000 7fde48ca4640 1 --2- 172.21.15.161:0/3919527773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde4414be70 0x7fde44144960 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fde3c007e60 tx=0x7fde3c00e4b0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.289+0000 7fde48ca4640 1 -- 172.21.15.161:0/3919527773 shutdown_connections 2024-03-28T23:32:26.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.289+0000 7fde48ca4640 1 --2- 172.21.15.161:0/3919527773 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fde2406b4d0 0x7fde2406d990 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.289+0000 7fde48ca4640 1 --2- 172.21.15.161:0/3919527773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde4414be70 0x7fde44144960 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.289+0000 7fde48ca4640 1 --2- 172.21.15.161:0/3919527773 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde440a9680 0x7fde44145dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.289+0000 7fde48ca4640 1 --2- 172.21.15.161:0/3919527773 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde4406ce80 0x7fde44150fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.289+0000 7fde48ca4640 1 -- 172.21.15.161:0/3919527773 >> 172.21.15.161:0/3919527773 conn(0x7fde440c6670 msgr2=0x7fde440ab1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:26.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.290+0000 7fde48ca4640 1 -- 172.21.15.161:0/3919527773 shutdown_connections 2024-03-28T23:32:26.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.290+0000 7fde48ca4640 1 -- 172.21.15.161:0/3919527773 wait complete. 2024-03-28T23:32:26.301 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:32:26.302 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:32:26.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.404+0000 7ff9bc9ac640 1 Processor -- start 2024-03-28T23:32:26.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.404+0000 7ff9bc9ac640 1 -- start start 2024-03-28T23:32:26.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.405+0000 7ff9bc9ac640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9b80c6350 0x7ff9b806c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:26.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.405+0000 7ff9bc9ac640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff9b80a5fb0 0x7ff9b806cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:26.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.405+0000 7ff9bc9ac640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9b8152960 con 0x7ff9b8151e70 2024-03-28T23:32:26.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.405+0000 7ff9bc9ac640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff9b80c1e40 con 0x7ff9b80a5fb0 2024-03-28T23:32:26.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.405+0000 7ff9bc9ac640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff9b80c2120 con 0x7ff9b80c6350 2024-03-28T23:32:26.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.405+0000 7ff9b6d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff9b80a5fb0 0x7ff9b806cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:26.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.406+0000 7ff9b6d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff9b80a5fb0 0x7ff9b806cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33758/0 (socket says 172.21.15.161:33758) 2024-03-28T23:32:26.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.406+0000 7ff9b6d9d640 1 -- 172.21.15.161:0/3182059903 learned_addr learned my addr 172.21.15.161:0/3182059903 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:26.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.406+0000 7ff9b659c640 1 --1- 172.21.15.161:0/3182059903 >> v1:172.21.15.161:6789/0 conn(0x7ff9b8151e70 0x7ff9b8152250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:43314/0 (socket says 172.21.15.161:43314) 2024-03-28T23:32:26.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.406+0000 7ff9b5d9b640 1 --2- 172.21.15.161:0/3182059903 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9b80c6350 0x7ff9b806c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:26.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.406+0000 7ff9b6d9d640 1 -- 172.21.15.161:0/3182059903 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9b80c6350 msgr2=0x7ff9b806c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:26.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.406+0000 7ff9b6d9d640 1 --2- 172.21.15.161:0/3182059903 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9b80c6350 0x7ff9b806c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.406+0000 7ff9b6d9d640 1 -- 172.21.15.161:0/3182059903 >> v1:172.21.15.161:6789/0 conn(0x7ff9b8151e70 legacy=0x7ff9b8152250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:26.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.406+0000 7ff9b559a640 1 -- 172.21.15.161:0/3182059903 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 333184954 0 0) 0x7ff9b8152960 con 0x7ff9b8151e70 2024-03-28T23:32:26.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.406+0000 7ff9b6d9d640 1 -- 172.21.15.161:0/3182059903 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9b80c2400 con 0x7ff9b80a5fb0 2024-03-28T23:32:26.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.407+0000 7ff9b6d9d640 1 --2- 172.21.15.161:0/3182059903 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9b80a5fb0 0x7ff9b806cc00 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8009870 tx=0x7ff9a802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81765a667c7ae578 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:26.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.407+0000 7ff9b559a640 1 -- 172.21.15.161:0/3182059903 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8035020 con 0x7ff9b80a5fb0 2024-03-28T23:32:26.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.407+0000 7ff9b559a640 1 -- 172.21.15.161:0/3182059903 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9a8031640 con 0x7ff9b80a5fb0 2024-03-28T23:32:26.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.407+0000 7ff9b559a640 1 -- 172.21.15.161:0/3182059903 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a802d040 con 0x7ff9b80a5fb0 2024-03-28T23:32:26.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.407+0000 7ff9bc9ac640 1 -- 172.21.15.161:0/3182059903 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9b80a5fb0 msgr2=0x7ff9b806cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:26.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.407+0000 7ff9bc9ac640 1 --2- 172.21.15.161:0/3182059903 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9b80a5fb0 0x7ff9b806cc00 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8009870 tx=0x7ff9a802b8f0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.407+0000 7ff9bc9ac640 1 -- 172.21.15.161:0/3182059903 shutdown_connections 2024-03-28T23:32:26.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.407+0000 7ff9bc9ac640 1 --2- 172.21.15.161:0/3182059903 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9b80a5fb0 0x7ff9b806cc00 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.407+0000 7ff9bc9ac640 1 --2- 172.21.15.161:0/3182059903 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9b80c6350 0x7ff9b806c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.407+0000 7ff9bc9ac640 1 -- 172.21.15.161:0/3182059903 >> 172.21.15.161:0/3182059903 conn(0x7ff9b80c3390 msgr2=0x7ff9b80c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:26.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.407+0000 7ff9bc9ac640 1 -- 172.21.15.161:0/3182059903 shutdown_connections 2024-03-28T23:32:26.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.407+0000 7ff9bc9ac640 1 -- 172.21.15.161:0/3182059903 wait complete. 2024-03-28T23:32:26.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.409+0000 7ff9bc9ac640 1 Processor -- start 2024-03-28T23:32:26.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.409+0000 7ff9bc9ac640 1 -- start start 2024-03-28T23:32:26.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.409+0000 7ff9bc9ac640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9b80a5fb0 0x7ff9b81eac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:26.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.409+0000 7ff9bc9ac640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9b80c6350 0x7ff9b8151440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.409+0000 7ff9bc9ac640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9b8151e70 0x7ff9b81ee8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.409+0000 7ff9bc9ac640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b80c2c00 con 0x7ff9b8151e70 2024-03-28T23:32:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.409+0000 7ff9bc9ac640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b814f810 con 0x7ff9b80c6350 2024-03-28T23:32:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.409+0000 7ff9bc9ac640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff9b80c2d80 con 0x7ff9b80a5fb0 2024-03-28T23:32:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.409+0000 7ff9b6d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9b8151e70 0x7ff9b81ee8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.409+0000 7ff9b6d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9b8151e70 0x7ff9b81ee8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33770/0 (socket says 172.21.15.161:33770) 2024-03-28T23:32:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.409+0000 7ff9b6d9d640 1 -- 172.21.15.161:0/2063675798 learned_addr learned my addr 172.21.15.161:0/2063675798 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.410+0000 7ff9b5d9b640 1 --2- 172.21.15.161:0/2063675798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9b80c6350 0x7ff9b8151440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.410+0000 7ff9b659c640 1 --2- 172.21.15.161:0/2063675798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9b80a5fb0 0x7ff9b81eac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.410+0000 7ff9b6d9d640 1 -- 172.21.15.161:0/2063675798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9b80a5fb0 msgr2=0x7ff9b81eac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.410+0000 7ff9b6d9d640 1 --2- 172.21.15.161:0/2063675798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9b80a5fb0 0x7ff9b81eac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.410+0000 7ff9b6d9d640 1 -- 172.21.15.161:0/2063675798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9b80c6350 msgr2=0x7ff9b8151440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.410+0000 7ff9b6d9d640 1 --2- 172.21.15.161:0/2063675798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9b80c6350 0x7ff9b8151440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.410+0000 7ff9b6d9d640 1 -- 172.21.15.161:0/2063675798 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9b8147210 con 0x7ff9b8151e70 2024-03-28T23:32:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.410+0000 7ff9b5d9b640 1 --2- 172.21.15.161:0/2063675798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9b80c6350 0x7ff9b8151440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.410+0000 7ff9b659c640 1 --2- 172.21.15.161:0/2063675798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9b80a5fb0 0x7ff9b81eac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.410+0000 7ff9b6d9d640 1 --2- 172.21.15.161:0/2063675798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9b8151e70 0x7ff9b81ee8f0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff9a802b930 tx=0x7ff9a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.410+0000 7ff9a6ffd640 1 -- 172.21.15.161:0/2063675798 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8040070 con 0x7ff9b8151e70 2024-03-28T23:32:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.410+0000 7ff9a6ffd640 1 -- 172.21.15.161:0/2063675798 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9a8039cd0 con 0x7ff9b8151e70 2024-03-28T23:32:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.410+0000 7ff9bc9ac640 1 -- 172.21.15.161:0/2063675798 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9b80c1080 con 0x7ff9b8151e70 2024-03-28T23:32:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.410+0000 7ff9a6ffd640 1 -- 172.21.15.161:0/2063675798 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8035040 con 0x7ff9b8151e70 2024-03-28T23:32:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.410+0000 7ff9bc9ac640 1 -- 172.21.15.161:0/2063675798 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9b8147790 con 0x7ff9b8151e70 2024-03-28T23:32:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.411+0000 7ff9bc9ac640 1 -- 172.21.15.161:0/2063675798 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9b804f4b0 con 0x7ff9b8151e70 2024-03-28T23:32:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.412+0000 7ff9a6ffd640 1 -- 172.21.15.161:0/2063675798 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff9a8038020 con 0x7ff9b8151e70 2024-03-28T23:32:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.412+0000 7ff9a6ffd640 1 --2- 172.21.15.161:0/2063675798 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff99806b500 0x7ff99806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:26.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.412+0000 7ff9b659c640 1 --2- 172.21.15.161:0/2063675798 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff99806b500 0x7ff99806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:26.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.412+0000 7ff9a6ffd640 1 -- 172.21.15.161:0/2063675798 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5679+0+0 (secure 0 0 0) 0x7ff9a80abc40 con 0x7ff9b8151e70 2024-03-28T23:32:26.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.413+0000 7ff9b659c640 1 --2- 172.21.15.161:0/2063675798 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff99806b500 0x7ff99806d9c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff9b0000f60 tx=0x7ff9b0007cc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:26.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.417+0000 7ff9a6ffd640 1 -- 172.21.15.161:0/2063675798 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff9a807ab70 con 0x7ff9b8151e70 2024-03-28T23:32:26.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.586+0000 7ff9bc9ac640 1 -- 172.21.15.161:0/2063675798 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff9b80c2f80 con 0x7ff9b8151e70 2024-03-28T23:32:26.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.586+0000 7ff9a6ffd640 1 -- 172.21.15.161:0/2063675798 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) ==== 93+0+10 (secure 0 0 0) 0x7ff9a807a470 con 0x7ff9b8151e70 2024-03-28T23:32:26.587 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:32:26.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.588+0000 7ff9bc9ac640 1 -- 172.21.15.161:0/2063675798 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff99806b500 msgr2=0x7ff99806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:26.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.588+0000 7ff9bc9ac640 1 --2- 172.21.15.161:0/2063675798 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff99806b500 0x7ff99806d9c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff9b0000f60 tx=0x7ff9b0007cc0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.588+0000 7ff9bc9ac640 1 -- 172.21.15.161:0/2063675798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9b8151e70 msgr2=0x7ff9b81ee8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:26.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.588+0000 7ff9bc9ac640 1 --2- 172.21.15.161:0/2063675798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9b8151e70 0x7ff9b81ee8f0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff9a802b930 tx=0x7ff9a80046c0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.588+0000 7ff9bc9ac640 1 -- 172.21.15.161:0/2063675798 shutdown_connections 2024-03-28T23:32:26.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.588+0000 7ff9bc9ac640 1 --2- 172.21.15.161:0/2063675798 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff99806b500 0x7ff99806d9c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.588+0000 7ff9bc9ac640 1 --2- 172.21.15.161:0/2063675798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9b8151e70 0x7ff9b81ee8f0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.588+0000 7ff9bc9ac640 1 --2- 172.21.15.161:0/2063675798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9b80c6350 0x7ff9b8151440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.588+0000 7ff9bc9ac640 1 --2- 172.21.15.161:0/2063675798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9b80a5fb0 0x7ff9b81eac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.589+0000 7ff9bc9ac640 1 -- 172.21.15.161:0/2063675798 >> 172.21.15.161:0/2063675798 conn(0x7ff9b80c3390 msgr2=0x7ff9b806d840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:26.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.589+0000 7ff9bc9ac640 1 -- 172.21.15.161:0/2063675798 shutdown_connections 2024-03-28T23:32:26.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.589+0000 7ff9bc9ac640 1 -- 172.21.15.161:0/2063675798 wait complete. 2024-03-28T23:32:26.600 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused' 2024-03-28T23:32:26.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.701+0000 7f0411ce1640 1 Processor -- start 2024-03-28T23:32:26.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.701+0000 7f0411ce1640 1 -- start start 2024-03-28T23:32:26.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.702+0000 7f0411ce1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f040c14be70 0x7f040c0ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:26.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.702+0000 7f0411ce1640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f040c0ab6e0 0x7f040c0aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:26.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.702+0000 7f0411ce1640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f040c14c250 con 0x7f040c06ce80 2024-03-28T23:32:26.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.702+0000 7f0411ce1640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f040c1529c0 con 0x7f040c14be70 2024-03-28T23:32:26.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.702+0000 7f0411ce1640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f040c0a9dc0 con 0x7f040c0ab6e0 2024-03-28T23:32:26.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.702+0000 7f040b7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f040c14be70 0x7f040c0ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:26.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.702+0000 7f040b7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f040c14be70 0x7f040c0ab1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33774/0 (socket says 172.21.15.161:33774) 2024-03-28T23:32:26.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.702+0000 7f040b7fe640 1 -- 172.21.15.161:0/1856278960 learned_addr learned my addr 172.21.15.161:0/1856278960 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:26.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.702+0000 7f040b7fe640 1 -- 172.21.15.161:0/1856278960 >> v2:172.21.15.170:3300/0 conn(0x7f040c0ab6e0 msgr2=0x7f040c0aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:26.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.702+0000 7f040affd640 1 --2- 172.21.15.161:0/1856278960 >> v2:172.21.15.170:3300/0 conn(0x7f040c0ab6e0 0x7f040c0aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:26.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.702+0000 7f040b7fe640 1 --2- 172.21.15.161:0/1856278960 >> v2:172.21.15.170:3300/0 conn(0x7f040c0ab6e0 0x7f040c0aa5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.703+0000 7f040b7fe640 1 -- 172.21.15.161:0/1856278960 >> v1:172.21.15.170:6789/0 conn(0x7f040c06ce80 legacy=0x7f040c1553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:26.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.703+0000 7f040b7fe640 1 -- 172.21.15.161:0/1856278960 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f040c0aa0a0 con 0x7f040c14be70 2024-03-28T23:32:26.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.703+0000 7f040b7fe640 1 --2- 172.21.15.161:0/1856278960 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f040c14be70 0x7f040c0ab1a0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f040c06f540 tx=0x7f03fc02e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c46b1f809819369c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:26.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.703+0000 7f040a7fc640 1 -- 172.21.15.161:0/1856278960 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03fc03b070 con 0x7f040c14be70 2024-03-28T23:32:26.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.703+0000 7f040a7fc640 1 -- 172.21.15.161:0/1856278960 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03fc02ee00 con 0x7f040c14be70 2024-03-28T23:32:26.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.703+0000 7f040a7fc640 1 -- 172.21.15.161:0/1856278960 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03fc036660 con 0x7f040c14be70 2024-03-28T23:32:26.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.704+0000 7f0411ce1640 1 -- 172.21.15.161:0/1856278960 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f040c14be70 msgr2=0x7f040c0ab1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:26.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.704+0000 7f0411ce1640 1 --2- 172.21.15.161:0/1856278960 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f040c14be70 0x7f040c0ab1a0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f040c06f540 tx=0x7f03fc02e050 comp rx=0 tx=0).stop 2024-03-28T23:32:26.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.704+0000 7f0411ce1640 1 -- 172.21.15.161:0/1856278960 shutdown_connections 2024-03-28T23:32:26.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.704+0000 7f0411ce1640 1 --2- 172.21.15.161:0/1856278960 >> v2:172.21.15.170:3300/0 conn(0x7f040c0ab6e0 0x7f040c0aa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.704+0000 7f0411ce1640 1 --2- 172.21.15.161:0/1856278960 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f040c14be70 0x7f040c0ab1a0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.704+0000 7f0411ce1640 1 -- 172.21.15.161:0/1856278960 >> 172.21.15.161:0/1856278960 conn(0x7f040c0c6670 msgr2=0x7f040c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:26.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.704+0000 7f0411ce1640 1 -- 172.21.15.161:0/1856278960 shutdown_connections 2024-03-28T23:32:26.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.704+0000 7f0411ce1640 1 -- 172.21.15.161:0/1856278960 wait complete. 2024-03-28T23:32:26.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.705+0000 7f0411ce1640 1 Processor -- start 2024-03-28T23:32:26.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.705+0000 7f0411ce1640 1 -- start start 2024-03-28T23:32:26.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.706+0000 7f0411ce1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f040c06ce80 0x7f040c14c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:26.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.706+0000 7f0411ce1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f040c0ab6e0 0x7f040c1e7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:26.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.706+0000 7f0411ce1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f040c14be70 0x7f040c148d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:26.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.706+0000 7f0411ce1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f040c06b2d0 con 0x7f040c0ab6e0 2024-03-28T23:32:26.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.706+0000 7f0411ce1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f040c06b150 con 0x7f040c14be70 2024-03-28T23:32:26.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.706+0000 7f0411ce1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f040c0c1520 con 0x7f040c06ce80 2024-03-28T23:32:26.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.706+0000 7f040affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f040c0ab6e0 0x7f040c1e7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:26.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.706+0000 7f040affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f040c0ab6e0 0x7f040c1e7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33788/0 (socket says 172.21.15.161:33788) 2024-03-28T23:32:26.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.706+0000 7f040affd640 1 -- 172.21.15.161:0/2835926888 learned_addr learned my addr 172.21.15.161:0/2835926888 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:26.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.706+0000 7f040b7fe640 1 --2- 172.21.15.161:0/2835926888 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f040c06ce80 0x7f040c14c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:26.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.706+0000 7f040bfff640 1 --2- 172.21.15.161:0/2835926888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f040c14be70 0x7f040c148d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:26.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.706+0000 7f040affd640 1 -- 172.21.15.161:0/2835926888 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f040c06ce80 msgr2=0x7f040c14c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:26.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.706+0000 7f040affd640 1 --2- 172.21.15.161:0/2835926888 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f040c06ce80 0x7f040c14c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.706+0000 7f040affd640 1 -- 172.21.15.161:0/2835926888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f040c14be70 msgr2=0x7f040c148d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:26.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.706+0000 7f040affd640 1 --2- 172.21.15.161:0/2835926888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f040c14be70 0x7f040c148d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.706+0000 7f040affd640 1 -- 172.21.15.161:0/2835926888 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f040c1e7ab0 con 0x7f040c0ab6e0 2024-03-28T23:32:26.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.707+0000 7f040affd640 1 --2- 172.21.15.161:0/2835926888 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f040c0ab6e0 0x7f040c1e7570 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f040000c610 tx=0x7f040000cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:26.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.707+0000 7f03f3fff640 1 -- 172.21.15.161:0/2835926888 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0400015070 con 0x7f040c0ab6e0 2024-03-28T23:32:26.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.707+0000 7f03f3fff640 1 -- 172.21.15.161:0/2835926888 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0400004220 con 0x7f040c0ab6e0 2024-03-28T23:32:26.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.707+0000 7f0411ce1640 1 -- 172.21.15.161:0/2835926888 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f040c150fa0 con 0x7f040c0ab6e0 2024-03-28T23:32:26.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.707+0000 7f03f3fff640 1 -- 172.21.15.161:0/2835926888 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0400003610 con 0x7f040c0ab6e0 2024-03-28T23:32:26.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.707+0000 7f0411ce1640 1 -- 172.21.15.161:0/2835926888 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f040c14fb10 con 0x7f040c0ab6e0 2024-03-28T23:32:26.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.708+0000 7f0411ce1640 1 -- 172.21.15.161:0/2835926888 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f040c04f4b0 con 0x7f040c0ab6e0 2024-03-28T23:32:26.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.708+0000 7f03f3fff640 1 -- 172.21.15.161:0/2835926888 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f04000037d0 con 0x7f040c0ab6e0 2024-03-28T23:32:26.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.709+0000 7f03f3fff640 1 --2- 172.21.15.161:0/2835926888 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f03f806b4d0 0x7f03f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:26.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.709+0000 7f03f3fff640 1 -- 172.21.15.161:0/2835926888 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5679+0+0 (secure 0 0 0) 0x7f0400089f40 con 0x7f040c0ab6e0 2024-03-28T23:32:26.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.709+0000 7f040b7fe640 1 --2- 172.21.15.161:0/2835926888 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f03f806b4d0 0x7f03f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:26.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.709+0000 7f040b7fe640 1 --2- 172.21.15.161:0/2835926888 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f03f806b4d0 0x7f03f806d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f03fc038000 tx=0x7f03fc032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:26.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.713+0000 7f03f3fff640 1 -- 172.21.15.161:0/2835926888 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0400058820 con 0x7f040c0ab6e0 2024-03-28T23:32:26.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.952+0000 7f0411ce1640 1 -- 172.21.15.161:0/2835926888 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused"]} v 0) -- 0x7f040c156280 con 0x7f040c0ab6e0 2024-03-28T23:32:26.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.988+0000 7f03f3fff640 1 -- 172.21.15.161:0/2835926888 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused"]}]=0 v163) ==== 161+0+0 (secure 0 0 0) 0x7f040005c8c0 con 0x7f040c0ab6e0 2024-03-28T23:32:26.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.990+0000 7f0411ce1640 1 -- 172.21.15.161:0/2835926888 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f03f806b4d0 msgr2=0x7f03f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:26.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.990+0000 7f0411ce1640 1 --2- 172.21.15.161:0/2835926888 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f03f806b4d0 0x7f03f806d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f03fc038000 tx=0x7f03fc032000 comp rx=0 tx=0).stop 2024-03-28T23:32:26.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.990+0000 7f0411ce1640 1 -- 172.21.15.161:0/2835926888 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f040c0ab6e0 msgr2=0x7f040c1e7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:26.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.990+0000 7f0411ce1640 1 --2- 172.21.15.161:0/2835926888 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f040c0ab6e0 0x7f040c1e7570 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f040000c610 tx=0x7f040000cad0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.990+0000 7f0411ce1640 1 -- 172.21.15.161:0/2835926888 shutdown_connections 2024-03-28T23:32:26.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.990+0000 7f0411ce1640 1 --2- 172.21.15.161:0/2835926888 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f03f806b4d0 0x7f03f806d990 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.990+0000 7f0411ce1640 1 --2- 172.21.15.161:0/2835926888 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f040c0ab6e0 0x7f040c1e7570 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.990+0000 7f0411ce1640 1 --2- 172.21.15.161:0/2835926888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f040c14be70 0x7f040c148d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.990+0000 7f0411ce1640 1 --2- 172.21.15.161:0/2835926888 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f040c06ce80 0x7f040c14c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:26.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.990+0000 7f0411ce1640 1 -- 172.21.15.161:0/2835926888 >> 172.21.15.161:0/2835926888 conn(0x7f040c0c6670 msgr2=0x7f040c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:26.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.990+0000 7f0411ce1640 1 -- 172.21.15.161:0/2835926888 shutdown_connections 2024-03-28T23:32:26.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:26.990+0000 7f0411ce1640 1 -- 172.21.15.161:0/2835926888 wait complete. 2024-03-28T23:32:27.002 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-28T23:32:27.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.105+0000 7fe363bfd640 1 Processor -- start 2024-03-28T23:32:27.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.105+0000 7fe363bfd640 1 -- start start 2024-03-28T23:32:27.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.106+0000 7fe363bfd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe35c0a7d00 0x7fe35c0c1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:27.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.106+0000 7fe363bfd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe35c153840 0x7fe35c0ab260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:27.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.106+0000 7fe363bfd640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe35c0c28e0 con 0x7fe35c1506b0 2024-03-28T23:32:27.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.106+0000 7fe363bfd640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe35c0c0d80 con 0x7fe35c153840 2024-03-28T23:32:27.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.106+0000 7fe363bfd640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fe35c0c2bc0 con 0x7fe35c0a7d00 2024-03-28T23:32:27.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.106+0000 7fe36219a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe35c153840 0x7fe35c0ab260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:27.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.106+0000 7fe36219a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe35c153840 0x7fe35c0ab260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33802/0 (socket says 172.21.15.161:33802) 2024-03-28T23:32:27.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.106+0000 7fe36219a640 1 -- 172.21.15.161:0/1005025398 learned_addr learned my addr 172.21.15.161:0/1005025398 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:27.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.106+0000 7fe361198640 1 --2- 172.21.15.161:0/1005025398 >> v2:172.21.15.170:3300/0 conn(0x7fe35c0a7d00 0x7fe35c0c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:27.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.106+0000 7fe36219a640 1 -- 172.21.15.161:0/1005025398 >> v2:172.21.15.170:3300/0 conn(0x7fe35c0a7d00 msgr2=0x7fe35c0c1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:27.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.106+0000 7fe36219a640 1 --2- 172.21.15.161:0/1005025398 >> v2:172.21.15.170:3300/0 conn(0x7fe35c0a7d00 0x7fe35c0c1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.106+0000 7fe36219a640 1 -- 172.21.15.161:0/1005025398 >> v1:172.21.15.170:6789/0 conn(0x7fe35c1506b0 legacy=0x7fe35c0a96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:27.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.106+0000 7fe36219a640 1 -- 172.21.15.161:0/1005025398 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe35c153040 con 0x7fe35c153840 2024-03-28T23:32:27.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.107+0000 7fe36219a640 1 --2- 172.21.15.161:0/1005025398 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe35c153840 0x7fe35c0ab260 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fe35c0739b0 tx=0x7fe35402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=942a98fca09bd21 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:27.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.107+0000 7fe360997640 1 -- 172.21.15.161:0/1005025398 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35402c9f0 con 0x7fe35c153840 2024-03-28T23:32:27.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.107+0000 7fe360997640 1 -- 172.21.15.161:0/1005025398 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe35402cb90 con 0x7fe35c153840 2024-03-28T23:32:27.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.107+0000 7fe360997640 1 -- 172.21.15.161:0/1005025398 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3540343f0 con 0x7fe35c153840 2024-03-28T23:32:27.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.108+0000 7fe363bfd640 1 -- 172.21.15.161:0/1005025398 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe35c153840 msgr2=0x7fe35c0ab260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:27.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.108+0000 7fe363bfd640 1 --2- 172.21.15.161:0/1005025398 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe35c153840 0x7fe35c0ab260 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fe35c0739b0 tx=0x7fe35402be50 comp rx=0 tx=0).stop 2024-03-28T23:32:27.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.108+0000 7fe363bfd640 1 -- 172.21.15.161:0/1005025398 shutdown_connections 2024-03-28T23:32:27.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.108+0000 7fe363bfd640 1 --2- 172.21.15.161:0/1005025398 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe35c153840 0x7fe35c0ab260 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.108+0000 7fe363bfd640 1 --2- 172.21.15.161:0/1005025398 >> v2:172.21.15.170:3300/0 conn(0x7fe35c0a7d00 0x7fe35c0c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.108+0000 7fe363bfd640 1 -- 172.21.15.161:0/1005025398 >> 172.21.15.161:0/1005025398 conn(0x7fe35c0a83f0 msgr2=0x7fe35c0a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:27.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.108+0000 7fe363bfd640 1 -- 172.21.15.161:0/1005025398 shutdown_connections 2024-03-28T23:32:27.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.108+0000 7fe363bfd640 1 -- 172.21.15.161:0/1005025398 wait complete. 2024-03-28T23:32:27.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.109+0000 7fe363bfd640 1 Processor -- start 2024-03-28T23:32:27.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.109+0000 7fe363bfd640 1 -- start start 2024-03-28T23:32:27.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.109+0000 7fe363bfd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe35c0a7d00 0x7fe35c143e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:27.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.110+0000 7fe363bfd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe35c1506b0 0x7fe35c0c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:27.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.110+0000 7fe363bfd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe35c153840 0x7fe35c0c6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:27.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.110+0000 7fe363bfd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe35c153c50 con 0x7fe35c0a7d00 2024-03-28T23:32:27.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.110+0000 7fe363bfd640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe35c0c2ea0 con 0x7fe35c153840 2024-03-28T23:32:27.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.110+0000 7fe363bfd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe35c153dd0 con 0x7fe35c1506b0 2024-03-28T23:32:27.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.110+0000 7fe361999640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe35c0a7d00 0x7fe35c143e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:27.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.110+0000 7fe361999640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe35c0a7d00 0x7fe35c143e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33804/0 (socket says 172.21.15.161:33804) 2024-03-28T23:32:27.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.110+0000 7fe361999640 1 -- 172.21.15.161:0/1867309196 learned_addr learned my addr 172.21.15.161:0/1867309196 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:27.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.110+0000 7fe361999640 1 -- 172.21.15.161:0/1867309196 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe35c1506b0 msgr2=0x7fe35c0c64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:27.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.110+0000 7fe361999640 1 --2- 172.21.15.161:0/1867309196 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe35c1506b0 0x7fe35c0c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.110+0000 7fe361999640 1 -- 172.21.15.161:0/1867309196 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe35c153840 msgr2=0x7fe35c0c6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:27.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.110+0000 7fe361999640 1 --2- 172.21.15.161:0/1867309196 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe35c153840 0x7fe35c0c6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.110+0000 7fe361999640 1 -- 172.21.15.161:0/1867309196 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe35c14a9f0 con 0x7fe35c0a7d00 2024-03-28T23:32:27.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.110+0000 7fe361999640 1 --2- 172.21.15.161:0/1867309196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe35c0a7d00 0x7fe35c143e90 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fe35800c430 tx=0x7fe35800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:27.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.110+0000 7fe3527fc640 1 -- 172.21.15.161:0/1867309196 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe358011790 con 0x7fe35c0a7d00 2024-03-28T23:32:27.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.110+0000 7fe3527fc640 1 -- 172.21.15.161:0/1867309196 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe358017070 con 0x7fe35c0a7d00 2024-03-28T23:32:27.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.110+0000 7fe3527fc640 1 -- 172.21.15.161:0/1867309196 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe358011bb0 con 0x7fe35c0a7d00 2024-03-28T23:32:27.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.110+0000 7fe363bfd640 1 -- 172.21.15.161:0/1867309196 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe35c1e97b0 con 0x7fe35c0a7d00 2024-03-28T23:32:27.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.110+0000 7fe363bfd640 1 -- 172.21.15.161:0/1867309196 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe35c14ca20 con 0x7fe35c0a7d00 2024-03-28T23:32:27.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.111+0000 7fe363bfd640 1 -- 172.21.15.161:0/1867309196 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe35c04f4b0 con 0x7fe35c0a7d00 2024-03-28T23:32:27.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.112+0000 7fe3527fc640 1 -- 172.21.15.161:0/1867309196 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe35801c460 con 0x7fe35c0a7d00 2024-03-28T23:32:27.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.112+0000 7fe3527fc640 1 --2- 172.21.15.161:0/1867309196 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe34006b4d0 0x7fe34006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:27.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.112+0000 7fe361198640 1 --2- 172.21.15.161:0/1867309196 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe34006b4d0 0x7fe34006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:27.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.112+0000 7fe3527fc640 1 -- 172.21.15.161:0/1867309196 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5679+0+0 (secure 0 0 0) 0x7fe358089fc0 con 0x7fe35c0a7d00 2024-03-28T23:32:27.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.113+0000 7fe361198640 1 --2- 172.21.15.161:0/1867309196 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe34006b4d0 0x7fe34006d990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe35c14a6b0 tx=0x7fe34c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:27.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.117+0000 7fe3527fc640 1 -- 172.21.15.161:0/1867309196 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe358058ef0 con 0x7fe35c0a7d00 2024-03-28T23:32:27.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.291+0000 7fe363bfd640 1 -- 172.21.15.161:0/1867309196 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fe35c0c2620 con 0x7fe35c0a7d00 2024-03-28T23:32:27.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.294+0000 7fe3527fc640 1 -- 172.21.15.161:0/1867309196 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v23)=0 v23) ==== 122+0+0 (secure 0 0 0) 0x7fe3580587f0 con 0x7fe35c0a7d00 2024-03-28T23:32:27.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.296+0000 7fe363bfd640 1 -- 172.21.15.161:0/1867309196 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe34006b4d0 msgr2=0x7fe34006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:27.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.296+0000 7fe363bfd640 1 --2- 172.21.15.161:0/1867309196 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe34006b4d0 0x7fe34006d990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe35c14a6b0 tx=0x7fe34c008040 comp rx=0 tx=0).stop 2024-03-28T23:32:27.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.296+0000 7fe363bfd640 1 -- 172.21.15.161:0/1867309196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe35c0a7d00 msgr2=0x7fe35c143e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:27.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.296+0000 7fe363bfd640 1 --2- 172.21.15.161:0/1867309196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe35c0a7d00 0x7fe35c143e90 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fe35800c430 tx=0x7fe35800c8f0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.296+0000 7fe363bfd640 1 -- 172.21.15.161:0/1867309196 shutdown_connections 2024-03-28T23:32:27.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.296+0000 7fe363bfd640 1 --2- 172.21.15.161:0/1867309196 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe34006b4d0 0x7fe34006d990 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.296+0000 7fe363bfd640 1 --2- 172.21.15.161:0/1867309196 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe35c153840 0x7fe35c0c6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.296+0000 7fe363bfd640 1 --2- 172.21.15.161:0/1867309196 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe35c1506b0 0x7fe35c0c64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.296+0000 7fe363bfd640 1 --2- 172.21.15.161:0/1867309196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe35c0a7d00 0x7fe35c143e90 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.296+0000 7fe363bfd640 1 -- 172.21.15.161:0/1867309196 >> 172.21.15.161:0/1867309196 conn(0x7fe35c0a83f0 msgr2=0x7fe35c073270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:27.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.297+0000 7fe363bfd640 1 -- 172.21.15.161:0/1867309196 shutdown_connections 2024-03-28T23:32:27.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.297+0000 7fe363bfd640 1 -- 172.21.15.161:0/1867309196 wait complete. 2024-03-28T23:32:27.308 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-28T23:32:27.308 DEBUG:teuthology.orchestra.run.smithi161:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:32:27.349 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:32:27.394 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-28T23:32:27.394 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:32:27.394 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:32:27.409 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20793 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-28T23:32:27.410 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:32:27.410 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:32:27.451 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056374,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19690 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-28T23:32:27.451 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-28T23:32:27.452 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:32:27.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.552+0000 7f57fda98640 1 Processor -- start 2024-03-28T23:32:27.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.552+0000 7f57fda98640 1 -- start start 2024-03-28T23:32:27.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.552+0000 7f57fda98640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f80a8fc0 0x7f57f8153150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:27.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57fda98640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57f81566f0 con 0x7f57f8150590 2024-03-28T23:32:27.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57fda98640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57f81569d0 con 0x7f57f80aacb0 2024-03-28T23:32:27.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57fda98640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f57f8156cb0 con 0x7f57f80a8fc0 2024-03-28T23:32:27.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57f77fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f57f8150590 0x7f57f80a9f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:43326/0 (socket says 172.21.15.161:43326) 2024-03-28T23:32:27.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57f77fe640 1 -- 172.21.15.161:0/1330671597 learned_addr learned my addr 172.21.15.161:0/1330671597 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:27.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57f6ffd640 1 --2- 172.21.15.161:0/1330671597 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f80a8fc0 0x7f57f8153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:27.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57f67fc640 1 -- 172.21.15.161:0/1330671597 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3794488315 0 0) 0x7f57f81566f0 con 0x7f57f8150590 2024-03-28T23:32:27.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57f67fc640 1 -- 172.21.15.161:0/1330671597 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57e0003520 con 0x7f57f8150590 2024-03-28T23:32:27.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57f67fc640 1 -- 172.21.15.161:0/1330671597 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3286979366 0 0) 0x7f57f81569d0 con 0x7f57f80aacb0 2024-03-28T23:32:27.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57f67fc640 1 -- 172.21.15.161:0/1330671597 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57f81566f0 con 0x7f57f80aacb0 2024-03-28T23:32:27.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57f67fc640 1 -- 172.21.15.161:0/1330671597 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2574860186 0 0) 0x7f57e0003520 con 0x7f57f8150590 2024-03-28T23:32:27.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57f67fc640 1 -- 172.21.15.161:0/1330671597 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f80a8fc0 msgr2=0x7f57f8153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:27.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57f67fc640 1 --2- 172.21.15.161:0/1330671597 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f80a8fc0 0x7f57f8153150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57f67fc640 1 -- 172.21.15.161:0/1330671597 >> v1:172.21.15.170:6789/0 conn(0x7f57f80aacb0 legacy=0x7f57f8153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:27.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57f67fc640 1 -- 172.21.15.161:0/1330671597 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f8156f90 con 0x7f57f8150590 2024-03-28T23:32:27.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57f67fc640 1 -- 172.21.15.161:0/1330671597 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f57e8001690 con 0x7f57f8150590 2024-03-28T23:32:27.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57f67fc640 1 -- 172.21.15.161:0/1330671597 >> v1:172.21.15.161:6789/0 conn(0x7f57f8150590 legacy=0x7f57f80a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:27.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57f67fc640 1 --2- 172.21.15.161:0/1330671597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57e0003c50 0x7f57e0004050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:27.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57f67fc640 1 --2- 172.21.15.161:0/1330671597 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57e0004800 0x7f57e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:27.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57f67fc640 1 --2- 172.21.15.161:0/1330671597 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f80a8fc0 0x7f57e000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:27.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57f67fc640 1 -- 172.21.15.161:0/1330671597 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f57f8156cb0 con 0x7f57e0003c50 2024-03-28T23:32:27.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57f67fc640 1 -- 172.21.15.161:0/1330671597 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f57e00055a0 con 0x7f57e0004800 2024-03-28T23:32:27.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57f67fc640 1 -- 172.21.15.161:0/1330671597 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f57e000bf00 con 0x7f57f80a8fc0 2024-03-28T23:32:27.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57f77fe640 1 --2- 172.21.15.161:0/1330671597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57e0003c50 0x7f57e0004050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:27.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57f6ffd640 1 --2- 172.21.15.161:0/1330671597 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57e0004800 0x7f57e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:27.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.553+0000 7f57f7fff640 1 --2- 172.21.15.161:0/1330671597 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f80a8fc0 0x7f57e000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:27.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.554+0000 7f57f77fe640 1 -- 172.21.15.161:0/1330671597 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f80a8fc0 msgr2=0x7f57e000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:27.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.554+0000 7f57f77fe640 1 --2- 172.21.15.161:0/1330671597 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f80a8fc0 0x7f57e000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.554+0000 7f57f77fe640 1 -- 172.21.15.161:0/1330671597 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57e0004800 msgr2=0x7f57e0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:27.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.554+0000 7f57f77fe640 1 --2- 172.21.15.161:0/1330671597 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57e0004800 0x7f57e0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.554+0000 7f57f77fe640 1 -- 172.21.15.161:0/1330671597 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57e0003520 con 0x7f57e0003c50 2024-03-28T23:32:27.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.554+0000 7f57f7fff640 1 --2- 172.21.15.161:0/1330671597 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f80a8fc0 0x7f57e000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:27.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.554+0000 7f57f77fe640 1 --2- 172.21.15.161:0/1330671597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57e0003c50 0x7f57e0004050 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f57e8007fd0 tx=0x7f57e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f847528443db789c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:27.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.555+0000 7f57f67fc640 1 -- 172.21.15.161:0/1330671597 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e8007c20 con 0x7f57e0003c50 2024-03-28T23:32:27.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.555+0000 7f57f67fc640 1 -- 172.21.15.161:0/1330671597 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57e8030070 con 0x7f57e0003c50 2024-03-28T23:32:27.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.555+0000 7f57f67fc640 1 -- 172.21.15.161:0/1330671597 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e8005d10 con 0x7f57e0003c50 2024-03-28T23:32:27.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.555+0000 7f57fda98640 1 -- 172.21.15.161:0/1330671597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57e0003c50 msgr2=0x7f57e0004050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:27.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.555+0000 7f57fda98640 1 --2- 172.21.15.161:0/1330671597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57e0003c50 0x7f57e0004050 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f57e8007fd0 tx=0x7f57e0005830 comp rx=0 tx=0).stop 2024-03-28T23:32:27.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.555+0000 7f57f7fff640 1 -- 172.21.15.161:0/1330671597 reap_dead start 2024-03-28T23:32:27.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.555+0000 7f57fda98640 1 -- 172.21.15.161:0/1330671597 shutdown_connections 2024-03-28T23:32:27.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.555+0000 7f57fda98640 1 -- 172.21.15.161:0/1330671597 >> 172.21.15.161:0/1330671597 conn(0x7f57f80c6670 msgr2=0x7f57f80c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:27.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.555+0000 7f57fda98640 1 -- 172.21.15.161:0/1330671597 shutdown_connections 2024-03-28T23:32:27.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.555+0000 7f57fda98640 1 -- 172.21.15.161:0/1330671597 wait complete. 2024-03-28T23:32:27.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.556+0000 7f57fda98640 1 Processor -- start 2024-03-28T23:32:27.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.556+0000 7f57fda98640 1 -- start start 2024-03-28T23:32:27.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.557+0000 7f57fda98640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57e0003c50 0x7f57f8152240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:27.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.557+0000 7f57fda98640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f81ebc50 0x7f57f81453e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:27.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.557+0000 7f57fda98640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57f8145920 0x7f57f814a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:27.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.557+0000 7f57fda98640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f57f8150970 con 0x7f57e0003c50 2024-03-28T23:32:27.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.557+0000 7f57fda98640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f57f80aa9c0 con 0x7f57f8145920 2024-03-28T23:32:27.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.557+0000 7f57fda98640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f57f80aa650 con 0x7f57f81ebc50 2024-03-28T23:32:27.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.557+0000 7f57f77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57e0003c50 0x7f57f8152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:27.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.557+0000 7f57f77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57e0003c50 0x7f57f8152240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33826/0 (socket says 172.21.15.161:33826) 2024-03-28T23:32:27.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.557+0000 7f57f77fe640 1 -- 172.21.15.161:0/3168195231 learned_addr learned my addr 172.21.15.161:0/3168195231 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:27.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.557+0000 7f57f6ffd640 1 --2- 172.21.15.161:0/3168195231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f81ebc50 0x7f57f81453e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:27.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.557+0000 7f57f7fff640 1 --2- 172.21.15.161:0/3168195231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57f8145920 0x7f57f814a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:27.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.557+0000 7f57f77fe640 1 -- 172.21.15.161:0/3168195231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f81ebc50 msgr2=0x7f57f81453e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:27.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.557+0000 7f57f77fe640 1 --2- 172.21.15.161:0/3168195231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f81ebc50 0x7f57f81453e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.557+0000 7f57f77fe640 1 -- 172.21.15.161:0/3168195231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57f8145920 msgr2=0x7f57f814a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:27.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.557+0000 7f57f77fe640 1 --2- 172.21.15.161:0/3168195231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57f8145920 0x7f57f814a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.557+0000 7f57f77fe640 1 -- 172.21.15.161:0/3168195231 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f8145dd0 con 0x7f57e0003c50 2024-03-28T23:32:27.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.557+0000 7f57f6ffd640 1 --2- 172.21.15.161:0/3168195231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f81ebc50 0x7f57f81453e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:27.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.558+0000 7f57f77fe640 1 --2- 172.21.15.161:0/3168195231 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57e0003c50 0x7f57f8152240 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f57e802c400 tx=0x7f57f81568c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:27.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.558+0000 7f57dffff640 1 -- 172.21.15.161:0/3168195231 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e8045070 con 0x7f57e0003c50 2024-03-28T23:32:27.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.558+0000 7f57dffff640 1 -- 172.21.15.161:0/3168195231 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57e802c9a0 con 0x7f57e0003c50 2024-03-28T23:32:27.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.558+0000 7f57fda98640 1 -- 172.21.15.161:0/3168195231 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57f80c6430 con 0x7f57e0003c50 2024-03-28T23:32:27.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.558+0000 7f57dffff640 1 -- 172.21.15.161:0/3168195231 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e8007c20 con 0x7f57e0003c50 2024-03-28T23:32:27.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.558+0000 7f57fda98640 1 -- 172.21.15.161:0/3168195231 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57f80c5020 con 0x7f57e0003c50 2024-03-28T23:32:27.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.559+0000 7f57fda98640 1 -- 172.21.15.161:0/3168195231 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57f804f4b0 con 0x7f57e0003c50 2024-03-28T23:32:27.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.559+0000 7f57dffff640 1 -- 172.21.15.161:0/3168195231 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f57e8007070 con 0x7f57e0003c50 2024-03-28T23:32:27.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.560+0000 7f57dffff640 1 --2- 172.21.15.161:0/3168195231 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f57e406b4d0 0x7f57e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:27.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.560+0000 7f57dffff640 1 -- 172.21.15.161:0/3168195231 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5679+0+0 (secure 0 0 0) 0x7f57e80aace0 con 0x7f57e0003c50 2024-03-28T23:32:27.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.560+0000 7f57f6ffd640 1 --2- 172.21.15.161:0/3168195231 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f57e406b4d0 0x7f57e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:27.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.560+0000 7f57f6ffd640 1 --2- 172.21.15.161:0/3168195231 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f57e406b4d0 0x7f57e406d990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f57f80c1420 tx=0x7f57ec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:27.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.564+0000 7f57dffff640 1 -- 172.21.15.161:0/3168195231 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f57e80795c0 con 0x7f57e0003c50 2024-03-28T23:32:27.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.773+0000 7f57fda98640 1 -- 172.21.15.161:0/3168195231 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f57f80a9d20 con 0x7f57e0003c50 2024-03-28T23:32:27.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.773+0000 7f57dffff640 1 -- 172.21.15.161:0/3168195231 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+3881 (secure 0 0 0) 0x7f57e807d660 con 0x7f57e0003c50 2024-03-28T23:32:27.774 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:27.774 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":51,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5542,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.170:6833/3666899516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3666899516},{"type":"v1","addr":"172.21.15.170:6833","nonce":3666899516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51},{"gid":5962,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3772646783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3772646783},{"type":"v1","addr":"172.21.15.170:6835","nonce":3772646783}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51},{"gid":6273,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/3350861737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3350861737},{"type":"v1","addr":"172.21.15.161:6837","nonce":3350861737}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":6288,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/2929859479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":2929859479},{"type":"v1","addr":"172.21.15.161:6836","nonce":2929859479}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[]} 2024-03-28T23:32:27.775 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 51 2024-03-28T23:32:27.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.775+0000 7f57fda98640 1 -- 172.21.15.161:0/3168195231 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f57e406b4d0 msgr2=0x7f57e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:27.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.775+0000 7f57fda98640 1 --2- 172.21.15.161:0/3168195231 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f57e406b4d0 0x7f57e406d990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f57f80c1420 tx=0x7f57ec008040 comp rx=0 tx=0).stop 2024-03-28T23:32:27.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.775+0000 7f57fda98640 1 -- 172.21.15.161:0/3168195231 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57e0003c50 msgr2=0x7f57f8152240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:27.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.775+0000 7f57fda98640 1 --2- 172.21.15.161:0/3168195231 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57e0003c50 0x7f57f8152240 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f57e802c400 tx=0x7f57f81568c0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.775+0000 7f57fda98640 1 -- 172.21.15.161:0/3168195231 shutdown_connections 2024-03-28T23:32:27.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.775+0000 7f57fda98640 1 --2- 172.21.15.161:0/3168195231 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f57e406b4d0 0x7f57e406d990 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.775+0000 7f57fda98640 1 --2- 172.21.15.161:0/3168195231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57f8145920 0x7f57f814a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.775+0000 7f57fda98640 1 --2- 172.21.15.161:0/3168195231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f81ebc50 0x7f57f81453e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.775+0000 7f57fda98640 1 --2- 172.21.15.161:0/3168195231 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57e0003c50 0x7f57f8152240 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.775+0000 7f57fda98640 1 -- 172.21.15.161:0/3168195231 >> 172.21.15.161:0/3168195231 conn(0x7f57f80c6670 msgr2=0x7f57f80c2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:27.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.776+0000 7f57fda98640 1 -- 172.21.15.161:0/3168195231 shutdown_connections 2024-03-28T23:32:27.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.776+0000 7f57fda98640 1 -- 172.21.15.161:0/3168195231 wait complete. 2024-03-28T23:32:27.787 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T23:32:27.787 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-28T23:32:27.788 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T23:32:27.788 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:32:27.788 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T23:32:27.788 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-28T23:32:27.788 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T23:32:27.788 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:32:27.789 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-28T23:32:27.789 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-28T23:32:27.789 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-28T23:32:27.789 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:32:27.789 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-28T23:32:27.789 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-28T23:32:27.789 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-28T23:32:27.789 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:32:27.797 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T23:32:27.797 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T23:32:27.798 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T23:32:27.798 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T23:32:27.800 INFO:tasks.ceph.mds.b:Stopped 2024-03-28T23:32:27.800 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-28T23:32:27.801 INFO:tasks.ceph.mds.d:Stopped 2024-03-28T23:32:27.801 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-28T23:32:27.840 INFO:tasks.ceph.mds.c:Started 2024-03-28T23:32:27.840 DEBUG:teuthology.parallel:result is None 2024-03-28T23:32:27.840 INFO:tasks.ceph.mds.a:Started 2024-03-28T23:32:27.840 DEBUG:teuthology.parallel:result is None 2024-03-28T23:32:27.843 INFO:tasks.ceph.mds.b:Started 2024-03-28T23:32:27.843 DEBUG:teuthology.parallel:result is None 2024-03-28T23:32:27.843 INFO:tasks.ceph.mds.d:Started 2024-03-28T23:32:27.844 DEBUG:teuthology.parallel:result is None 2024-03-28T23:32:27.844 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-28T23:32:27.917 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-28T23:32:27.917 INFO:tasks.ceph.mds.c.smithi161.stdout:starting mds.c at 2024-03-28T23:32:27.927 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2024-03-28T23:32:27.927 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2024-03-28T23:32:27.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.950+0000 7fac41623640 1 Processor -- start 2024-03-28T23:32:27.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.951+0000 7fac41623640 1 -- start start 2024-03-28T23:32:27.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.951+0000 7fac41623640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fac3c152890 0x7fac3c0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:27.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.951+0000 7fac41623640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fac3c0c8ed0 0x7fac3c0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:27.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.951+0000 7fac41623640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac3c0c5950 0x7fac3c1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:27.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.951+0000 7fac41623640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fac3c0c8480 con 0x7fac3c152890 2024-03-28T23:32:27.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.951+0000 7fac41623640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fac3c159640 con 0x7fac3c0c8ed0 2024-03-28T23:32:27.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.951+0000 7fac41623640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fac3c158a30 con 0x7fac3c0c5950 2024-03-28T23:32:27.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.951+0000 7fac3affd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fac3c152890 0x7fac3c0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:27.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.951+0000 7fac3affd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fac3c152890 0x7fac3c0c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33858/0 (socket says 172.21.15.161:33858) 2024-03-28T23:32:27.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.951+0000 7fac3affd640 1 -- 172.21.15.161:0/3279126581 learned_addr learned my addr 172.21.15.161:0/3279126581 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:27.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.951+0000 7fac3a7fc640 1 --2- 172.21.15.161:0/3279126581 >> v2:172.21.15.170:3300/0 conn(0x7fac3c0c8ed0 0x7fac3c0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:27.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.951+0000 7fac3b7fe640 1 --2- 172.21.15.161:0/3279126581 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac3c0c5950 0x7fac3c1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:27.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.952+0000 7fac3a7fc640 1 -- 172.21.15.161:0/3279126581 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac3c0c5950 msgr2=0x7fac3c1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:27.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.952+0000 7fac3a7fc640 1 --2- 172.21.15.161:0/3279126581 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac3c0c5950 0x7fac3c1553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.952+0000 7fac3a7fc640 1 -- 172.21.15.161:0/3279126581 >> v2:172.21.15.161:3300/0 conn(0x7fac3c152890 msgr2=0x7fac3c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:27.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.952+0000 7fac3a7fc640 1 --2- 172.21.15.161:0/3279126581 >> v2:172.21.15.161:3300/0 conn(0x7fac3c152890 0x7fac3c0c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.952+0000 7fac3a7fc640 1 -- 172.21.15.161:0/3279126581 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac3c158d10 con 0x7fac3c0c8ed0 2024-03-28T23:32:27.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.952+0000 7fac3affd640 1 --2- 172.21.15.161:0/3279126581 >> v2:172.21.15.161:3300/0 conn(0x7fac3c152890 0x7fac3c0c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:32:27.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.952+0000 7fac3a7fc640 1 --2- 172.21.15.161:0/3279126581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac3c0c8ed0 0x7fac3c0c5410 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0ad6e0 tx=0x7fac3002b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e0681982232f072 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:27.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.953+0000 7fac39ffb640 1 -- 172.21.15.161:0/3279126581 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac30037070 con 0x7fac3c0c8ed0 2024-03-28T23:32:27.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.953+0000 7fac39ffb640 1 -- 172.21.15.161:0/3279126581 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac30030710 con 0x7fac3c0c8ed0 2024-03-28T23:32:27.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.953+0000 7fac39ffb640 1 -- 172.21.15.161:0/3279126581 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac30030ae0 con 0x7fac3c0c8ed0 2024-03-28T23:32:27.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.953+0000 7fac41623640 1 -- 172.21.15.161:0/3279126581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac3c0c8ed0 msgr2=0x7fac3c0c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:27.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.953+0000 7fac41623640 1 --2- 172.21.15.161:0/3279126581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac3c0c8ed0 0x7fac3c0c5410 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0ad6e0 tx=0x7fac3002b710 comp rx=0 tx=0).stop 2024-03-28T23:32:27.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.953+0000 7fac41623640 1 -- 172.21.15.161:0/3279126581 shutdown_connections 2024-03-28T23:32:27.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.953+0000 7fac41623640 1 --2- 172.21.15.161:0/3279126581 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac3c0c5950 0x7fac3c1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.953+0000 7fac41623640 1 --2- 172.21.15.161:0/3279126581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac3c0c8ed0 0x7fac3c0c5410 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.953+0000 7fac41623640 1 --2- 172.21.15.161:0/3279126581 >> v2:172.21.15.161:3300/0 conn(0x7fac3c152890 0x7fac3c0c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.953+0000 7fac41623640 1 -- 172.21.15.161:0/3279126581 >> 172.21.15.161:0/3279126581 conn(0x7fac3c0ac620 msgr2=0x7fac3c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:27.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.953+0000 7fac41623640 1 -- 172.21.15.161:0/3279126581 shutdown_connections 2024-03-28T23:32:27.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.953+0000 7fac41623640 1 -- 172.21.15.161:0/3279126581 wait complete. 2024-03-28T23:32:27.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.955+0000 7fac41623640 1 Processor -- start 2024-03-28T23:32:27.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.955+0000 7fac41623640 1 -- start start 2024-03-28T23:32:27.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.955+0000 7fac41623640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac3c0c5950 0x7fac3c14baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:27.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.955+0000 7fac41623640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac3c0c8ed0 0x7fac3c1eb240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:27.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.955+0000 7fac41623640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac3c152890 0x7fac3c1eb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:27.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.955+0000 7fac41623640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fac3c152c70 con 0x7fac3c152890 2024-03-28T23:32:27.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.955+0000 7fac41623640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fac3c1592d0 con 0x7fac3c0c5950 2024-03-28T23:32:27.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.955+0000 7fac41623640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fac3c152df0 con 0x7fac3c0c8ed0 2024-03-28T23:32:27.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.955+0000 7fac3b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac3c152890 0x7fac3c1eb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:27.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.955+0000 7fac3b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac3c152890 0x7fac3c1eb780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33870/0 (socket says 172.21.15.161:33870) 2024-03-28T23:32:27.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.955+0000 7fac3b7fe640 1 -- 172.21.15.161:0/2050441191 learned_addr learned my addr 172.21.15.161:0/2050441191 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:27.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.956+0000 7fac3affd640 1 --2- 172.21.15.161:0/2050441191 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac3c0c5950 0x7fac3c14baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:27.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.956+0000 7fac3a7fc640 1 --2- 172.21.15.161:0/2050441191 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac3c0c8ed0 0x7fac3c1eb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:27.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.956+0000 7fac3b7fe640 1 -- 172.21.15.161:0/2050441191 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac3c0c8ed0 msgr2=0x7fac3c1eb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:27.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.956+0000 7fac3b7fe640 1 --2- 172.21.15.161:0/2050441191 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac3c0c8ed0 0x7fac3c1eb240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.956+0000 7fac3b7fe640 1 -- 172.21.15.161:0/2050441191 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac3c0c5950 msgr2=0x7fac3c14baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:27.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.956+0000 7fac3b7fe640 1 --2- 172.21.15.161:0/2050441191 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac3c0c5950 0x7fac3c14baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:27.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.956+0000 7fac3b7fe640 1 -- 172.21.15.161:0/2050441191 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac3c1ebcc0 con 0x7fac3c152890 2024-03-28T23:32:27.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.956+0000 7fac3a7fc640 1 --2- 172.21.15.161:0/2050441191 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac3c0c8ed0 0x7fac3c1eb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:27.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.956+0000 7fac3b7fe640 1 --2- 172.21.15.161:0/2050441191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac3c152890 0x7fac3c1eb780 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fac3400eed0 tx=0x7fac3400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:27.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.956+0000 7fac237fe640 1 -- 172.21.15.161:0/2050441191 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac34018020 con 0x7fac3c152890 2024-03-28T23:32:27.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.956+0000 7fac237fe640 1 -- 172.21.15.161:0/2050441191 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac3400bd20 con 0x7fac3c152890 2024-03-28T23:32:27.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.956+0000 7fac41623640 1 -- 172.21.15.161:0/2050441191 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac3c1eec70 con 0x7fac3c152890 2024-03-28T23:32:27.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.956+0000 7fac237fe640 1 -- 172.21.15.161:0/2050441191 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac34014640 con 0x7fac3c152890 2024-03-28T23:32:27.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.956+0000 7fac41623640 1 -- 172.21.15.161:0/2050441191 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac3c1eef10 con 0x7fac3c152890 2024-03-28T23:32:27.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.957+0000 7fac41623640 1 -- 172.21.15.161:0/2050441191 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac3c04f4b0 con 0x7fac3c152890 2024-03-28T23:32:27.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.957+0000 7fac237fe640 1 -- 172.21.15.161:0/2050441191 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fac34005d10 con 0x7fac3c152890 2024-03-28T23:32:27.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.958+0000 7fac237fe640 1 --2- 172.21.15.161:0/2050441191 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fac2806b4d0 0x7fac2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:27.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.958+0000 7fac3affd640 1 --2- 172.21.15.161:0/2050441191 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fac2806b4d0 0x7fac2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:27.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.958+0000 7fac237fe640 1 -- 172.21.15.161:0/2050441191 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5679+0+0 (secure 0 0 0) 0x7fac3408aa50 con 0x7fac3c152890 2024-03-28T23:32:27.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.958+0000 7fac3affd640 1 --2- 172.21.15.161:0/2050441191 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fac2806b4d0 0x7fac2806d990 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fac3c1564c0 tx=0x7fac2c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:27.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:27.962+0000 7fac237fe640 1 -- 172.21.15.161:0/2050441191 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fac34022030 con 0x7fac3c152890 2024-03-28T23:32:28.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:28.148+0000 7fac41623640 1 -- 172.21.15.161:0/2050441191 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fac3c0ac400 con 0x7fac3c152890 2024-03-28T23:32:29.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.000+0000 7fac237fe640 1 -- 172.21.15.161:0/2050441191 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v69) ==== 97+0+0 (secure 0 0 0) 0x7fac3c1ebcc0 con 0x7fac3c152890 2024-03-28T23:32:29.001 INFO:teuthology.orchestra.run.smithi161.stderr: removed all blocklist entries 2024-03-28T23:32:29.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.003+0000 7fac41623640 1 -- 172.21.15.161:0/2050441191 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fac2806b4d0 msgr2=0x7fac2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:29.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.003+0000 7fac41623640 1 --2- 172.21.15.161:0/2050441191 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fac2806b4d0 0x7fac2806d990 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fac3c1564c0 tx=0x7fac2c008040 comp rx=0 tx=0).stop 2024-03-28T23:32:29.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.003+0000 7fac41623640 1 -- 172.21.15.161:0/2050441191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac3c152890 msgr2=0x7fac3c1eb780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:29.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.003+0000 7fac41623640 1 --2- 172.21.15.161:0/2050441191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac3c152890 0x7fac3c1eb780 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fac3400eed0 tx=0x7fac3400b480 comp rx=0 tx=0).stop 2024-03-28T23:32:29.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.003+0000 7fac41623640 1 -- 172.21.15.161:0/2050441191 shutdown_connections 2024-03-28T23:32:29.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.003+0000 7fac41623640 1 --2- 172.21.15.161:0/2050441191 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fac2806b4d0 0x7fac2806d990 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.003+0000 7fac41623640 1 --2- 172.21.15.161:0/2050441191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac3c152890 0x7fac3c1eb780 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.003+0000 7fac41623640 1 --2- 172.21.15.161:0/2050441191 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac3c0c8ed0 0x7fac3c1eb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.003+0000 7fac41623640 1 --2- 172.21.15.161:0/2050441191 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac3c0c5950 0x7fac3c14baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.003+0000 7fac41623640 1 -- 172.21.15.161:0/2050441191 >> 172.21.15.161:0/2050441191 conn(0x7fac3c0ac620 msgr2=0x7fac3c150fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:29.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.003+0000 7fac41623640 1 -- 172.21.15.161:0/2050441191 shutdown_connections 2024-03-28T23:32:29.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.003+0000 7fac41623640 1 -- 172.21.15.161:0/2050441191 wait complete. 2024-03-28T23:32:29.017 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-28T23:32:29.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.123+0000 7f68c61ac640 1 Processor -- start 2024-03-28T23:32:29.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.123+0000 7f68c61ac640 1 -- start start 2024-03-28T23:32:29.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.124+0000 7f68c61ac640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f68c00c3ae0 0x7f68c01591e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:29.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.124+0000 7f68c61ac640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68c0155d30 con 0x7f68c014f240 2024-03-28T23:32:29.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.124+0000 7f68c61ac640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68c00c1e10 con 0x7f68c00c3410 2024-03-28T23:32:29.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.124+0000 7f68c61ac640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f68c0156d10 con 0x7f68c00c3ae0 2024-03-28T23:32:29.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.124+0000 7f68bffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f68c00c3ae0 0x7f68c01591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:29.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.124+0000 7f68bf7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f68c014f240 0x7f68c0156600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:43340/0 (socket says 172.21.15.161:43340) 2024-03-28T23:32:29.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.124+0000 7f68bf7fe640 1 -- 172.21.15.161:0/2713871922 learned_addr learned my addr 172.21.15.161:0/2713871922 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:29.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.124+0000 7f68bffff640 1 -- 172.21.15.161:0/2713871922 >> v1:172.21.15.170:6789/0 conn(0x7f68c00c3410 legacy=0x7f68c00c66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:29.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.124+0000 7f68bffff640 1 -- 172.21.15.161:0/2713871922 >> v1:172.21.15.161:6789/0 conn(0x7f68c014f240 legacy=0x7f68c0156600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:29.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.124+0000 7f68bffff640 1 -- 172.21.15.161:0/2713871922 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68c0155fd0 con 0x7f68c00c3ae0 2024-03-28T23:32:29.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.125+0000 7f68bffff640 1 --2- 172.21.15.161:0/2713871922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68c00c3ae0 0x7f68c01591e0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f68b8009d90 tx=0x7f68b802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c80095329c8755bd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:29.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.125+0000 7f68be7fc640 1 -- 172.21.15.161:0/2713871922 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b8039070 con 0x7f68c00c3ae0 2024-03-28T23:32:29.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.125+0000 7f68be7fc640 1 -- 172.21.15.161:0/2713871922 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68b802cbd0 con 0x7f68c00c3ae0 2024-03-28T23:32:29.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.125+0000 7f68be7fc640 1 -- 172.21.15.161:0/2713871922 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b8034440 con 0x7f68c00c3ae0 2024-03-28T23:32:29.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.126+0000 7f68c61ac640 1 -- 172.21.15.161:0/2713871922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68c00c3ae0 msgr2=0x7f68c01591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:29.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.126+0000 7f68c61ac640 1 --2- 172.21.15.161:0/2713871922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68c00c3ae0 0x7f68c01591e0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f68b8009d90 tx=0x7f68b802be20 comp rx=0 tx=0).stop 2024-03-28T23:32:29.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.126+0000 7f68c61ac640 1 -- 172.21.15.161:0/2713871922 shutdown_connections 2024-03-28T23:32:29.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.126+0000 7f68c61ac640 1 --2- 172.21.15.161:0/2713871922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68c00c3ae0 0x7f68c01591e0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.126+0000 7f68c61ac640 1 -- 172.21.15.161:0/2713871922 >> 172.21.15.161:0/2713871922 conn(0x7f68c00c7790 msgr2=0x7f68c00c7ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:29.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.126+0000 7f68c61ac640 1 -- 172.21.15.161:0/2713871922 shutdown_connections 2024-03-28T23:32:29.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.126+0000 7f68c61ac640 1 -- 172.21.15.161:0/2713871922 wait complete. 2024-03-28T23:32:29.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.127+0000 7f68c61ac640 1 Processor -- start 2024-03-28T23:32:29.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.127+0000 7f68c61ac640 1 -- start start 2024-03-28T23:32:29.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.128+0000 7f68c61ac640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f68c00c3410 0x7f68c0154b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:29.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.128+0000 7f68c61ac640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68c00c3ae0 0x7f68c01480b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:29.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.128+0000 7f68c61ac640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68c014f240 0x7f68c014ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:29.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.128+0000 7f68c61ac640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f68c01589e0 con 0x7f68c00c3ae0 2024-03-28T23:32:29.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.128+0000 7f68c61ac640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f68c014f620 con 0x7f68c00c3410 2024-03-28T23:32:29.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.128+0000 7f68c61ac640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f68c0158cc0 con 0x7f68c014f240 2024-03-28T23:32:29.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.128+0000 7f68beffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68c00c3ae0 0x7f68c01480b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:29.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.128+0000 7f68beffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68c00c3ae0 0x7f68c01480b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33888/0 (socket says 172.21.15.161:33888) 2024-03-28T23:32:29.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.128+0000 7f68beffd640 1 -- 172.21.15.161:0/1566861683 learned_addr learned my addr 172.21.15.161:0/1566861683 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:29.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.128+0000 7f68bf7fe640 1 --2- 172.21.15.161:0/1566861683 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f68c00c3410 0x7f68c0154b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:29.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.128+0000 7f68bffff640 1 --2- 172.21.15.161:0/1566861683 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68c014f240 0x7f68c014ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:29.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.128+0000 7f68beffd640 1 -- 172.21.15.161:0/1566861683 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68c014f240 msgr2=0x7f68c014ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:29.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.128+0000 7f68beffd640 1 --2- 172.21.15.161:0/1566861683 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68c014f240 0x7f68c014ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.128+0000 7f68beffd640 1 -- 172.21.15.161:0/1566861683 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f68c00c3410 msgr2=0x7f68c0154b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:29.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.128+0000 7f68beffd640 1 --2- 172.21.15.161:0/1566861683 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f68c00c3410 0x7f68c0154b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.128+0000 7f68beffd640 1 -- 172.21.15.161:0/1566861683 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68c0153880 con 0x7f68c00c3ae0 2024-03-28T23:32:29.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.128+0000 7f68bffff640 1 --2- 172.21.15.161:0/1566861683 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68c014f240 0x7f68c014ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:29.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.128+0000 7f68beffd640 1 --2- 172.21.15.161:0/1566861683 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68c00c3ae0 0x7f68c01480b0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f68b400c430 tx=0x7f68b400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:29.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.128+0000 7f68a7fff640 1 -- 172.21.15.161:0/1566861683 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b4011790 con 0x7f68c00c3ae0 2024-03-28T23:32:29.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.128+0000 7f68a7fff640 1 -- 172.21.15.161:0/1566861683 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68b4017070 con 0x7f68c00c3ae0 2024-03-28T23:32:29.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.128+0000 7f68c61ac640 1 -- 172.21.15.161:0/1566861683 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68c0153b60 con 0x7f68c00c3ae0 2024-03-28T23:32:29.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.128+0000 7f68a7fff640 1 -- 172.21.15.161:0/1566861683 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b4011b70 con 0x7f68c00c3ae0 2024-03-28T23:32:29.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.128+0000 7f68c61ac640 1 -- 172.21.15.161:0/1566861683 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68c01485f0 con 0x7f68c00c3ae0 2024-03-28T23:32:29.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.130+0000 7f68c61ac640 1 -- 172.21.15.161:0/1566861683 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68c0154080 con 0x7f68c00c3ae0 2024-03-28T23:32:29.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.130+0000 7f68a7fff640 1 -- 172.21.15.161:0/1566861683 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f68b4011d10 con 0x7f68c00c3ae0 2024-03-28T23:32:29.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.130+0000 7f68a7fff640 1 --2- 172.21.15.161:0/1566861683 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f68ac06b500 0x7f68ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:29.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.130+0000 7f68bf7fe640 1 --2- 172.21.15.161:0/1566861683 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f68ac06b500 0x7f68ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:29.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.130+0000 7f68a7fff640 1 -- 172.21.15.161:0/1566861683 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5249+0+0 (secure 0 0 0) 0x7f68b4089ce0 con 0x7f68c00c3ae0 2024-03-28T23:32:29.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.131+0000 7f68bf7fe640 1 --2- 172.21.15.161:0/1566861683 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f68ac06b500 0x7f68ac06d9c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f68b0007fd0 tx=0x7f68b0007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:29.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.135+0000 7f68a7fff640 1 -- 172.21.15.161:0/1566861683 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f68c0154080 con 0x7f68c00c3ae0 2024-03-28T23:32:29.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.372+0000 7f68c61ac640 1 -- 172.21.15.161:0/1566861683 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f68c0154470 con 0x7f68c00c3ae0 2024-03-28T23:32:29.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.373+0000 7f68a7fff640 1 -- 172.21.15.161:0/1566861683 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v10) ==== 80+0+4724 (secure 0 0 0) 0x7f68b4058db0 con 0x7f68c00c3ae0 2024-03-28T23:32:29.452 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:29.453 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:32:29.453 INFO:teuthology.orchestra.run.smithi161.stdout: "auth_dump": [ 2024-03-28T23:32:29.453 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:32:29.453 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.a", 2024-03-28T23:32:29.453 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmggtoJhAASLXbUICLBhjRr19aUAuZHw==", 2024-03-28T23:32:29.453 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:32:29.453 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:32:29.453 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:32:29.453 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:32:29.453 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:32:29.453 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:32:29.453 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:32:29.453 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:32:29.454 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.b", 2024-03-28T23:32:29.454 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmQbEOMxAA74al6+dMzcN/ngwNZK+aHA==", 2024-03-28T23:32:29.454 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:32:29.454 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:32:29.454 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:32:29.454 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:32:29.454 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:32:29.454 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:32:29.454 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:32:29.455 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:32:29.455 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.c", 2024-03-28T23:32:29.455 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmjf/qLRAAosLbpT8aCWvrKAQnj8Uh3g==", 2024-03-28T23:32:29.455 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:32:29.455 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:32:29.455 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:32:29.455 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:32:29.456 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:32:29.456 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:32:29.456 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:32:29.456 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:32:29.456 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.d", 2024-03-28T23:32:29.456 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmpiZ1OhAAjYrFAaDosDS2Y+BvU4c6JQ==", 2024-03-28T23:32:29.456 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:32:29.456 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:32:29.456 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:32:29.456 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:32:29.456 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:32:29.456 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:32:29.456 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:32:29.457 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:32:29.457 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.0", 2024-03-28T23:32:29.457 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA9/QVmAFPmGBAAS0SkRRwsupm6jm1sqFXJ6g==", 2024-03-28T23:32:29.457 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:32:29.457 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:32:29.457 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:32:29.457 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:32:29.457 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:32:29.457 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:32:29.457 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:32:29.457 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.1", 2024-03-28T23:32:29.457 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBA/QVmNBTVFxAA1htU9ilznfLUDqKmJaQwzg==", 2024-03-28T23:32:29.457 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:32:29.457 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:32:29.458 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:32:29.458 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:32:29.458 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:32:29.458 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:32:29.458 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:32:29.458 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.2", 2024-03-28T23:32:29.458 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBD/QVmg+4YFhAA9TvfFVKT3nQT7/QUOf3Spw==", 2024-03-28T23:32:29.458 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:32:29.458 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:32:29.458 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:32:29.458 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:32:29.458 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:32:29.458 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:32:29.458 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:32:29.459 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.3", 2024-03-28T23:32:29.459 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBG/QVmgiHDFBAAQPtbZT6a1ZGmkisBX0NMgA==", 2024-03-28T23:32:29.459 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:32:29.459 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:32:29.459 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:32:29.459 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:32:29.459 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:32:29.459 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:32:29.459 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:32:29.459 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.4", 2024-03-28T23:32:29.459 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBK/QVmahhZIBAAJfotg/dusbB3usreZlRuPA==", 2024-03-28T23:32:29.459 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:32:29.459 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:32:29.460 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:32:29.460 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:32:29.460 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:32:29.460 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:32:29.460 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:32:29.460 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.5", 2024-03-28T23:32:29.460 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBN/QVmzjlDHhAAli7pLyIo+MVdwSBNJYVg5g==", 2024-03-28T23:32:29.460 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:32:29.460 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:32:29.460 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:32:29.460 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:32:29.460 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:32:29.461 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:32:29.461 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:32:29.461 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.6", 2024-03-28T23:32:29.461 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBQ/QVmtCFSHBAASVfV/5mY/3JNiI0YMT7Vww==", 2024-03-28T23:32:29.461 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:32:29.461 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:32:29.461 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:32:29.461 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:32:29.461 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:32:29.461 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:32:29.461 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:32:29.461 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.7", 2024-03-28T23:32:29.461 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBT/QVm2J7eGhAAaIeZxmLwQBBE1glCxjKbhA==", 2024-03-28T23:32:29.461 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:32:29.462 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:32:29.462 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:32:29.462 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:32:29.462 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:32:29.462 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:32:29.462 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:32:29.462 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.0", 2024-03-28T23:32:29.462 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA6/QVmc1krAxAAEKsn1Tuuv9OlFZpB4PFhDg==", 2024-03-28T23:32:29.462 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:32:29.463 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:32:29.463 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow r", 2024-03-28T23:32:29.463 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-28T23:32:29.463 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:32:29.463 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:32:29.463 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:32:29.464 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.1", 2024-03-28T23:32:29.464 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA6/QVm1B2cBhAAmHDeeFwpFlV/y3NIGWUXHQ==", 2024-03-28T23:32:29.464 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:32:29.464 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:32:29.464 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow r", 2024-03-28T23:32:29.464 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow rw", 2024-03-28T23:32:29.465 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow rwx" 2024-03-28T23:32:29.465 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:32:29.465 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:32:29.465 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:32:29.465 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.admin", 2024-03-28T23:32:29.465 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmzzJ7CRAAYqfhnIkfNJJ3O+VWC2LP+Q==", 2024-03-28T23:32:29.465 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:32:29.466 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-28T23:32:29.466 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:32:29.466 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:32:29.466 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:32:29.466 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:32:29.466 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:32:29.467 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:32:29.467 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mgr.x", 2024-03-28T23:32:29.467 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVm7TD3IhAACByGRGpvJWDpF22NuovHWw==", 2024-03-28T23:32:29.467 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:32:29.467 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-28T23:32:29.467 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile mgr", 2024-03-28T23:32:29.467 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:32:29.467 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:32:29.467 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:32:29.468 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:32:29.468 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mgr.y", 2024-03-28T23:32:29.468 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVm0hLlHhAAh8cewPtZ/TbKrU488kxrBg==", 2024-03-28T23:32:29.468 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:32:29.468 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-28T23:32:29.468 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile mgr", 2024-03-28T23:32:29.468 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:32:29.468 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:32:29.468 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:32:29.468 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-03-28T23:32:29.468 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:32:29.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.375+0000 7f68c61ac640 1 -- 172.21.15.161:0/1566861683 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f68ac06b500 msgr2=0x7f68ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:29.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.375+0000 7f68c61ac640 1 --2- 172.21.15.161:0/1566861683 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f68ac06b500 0x7f68ac06d9c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f68b0007fd0 tx=0x7f68b0007e80 comp rx=0 tx=0).stop 2024-03-28T23:32:29.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.375+0000 7f68c61ac640 1 -- 172.21.15.161:0/1566861683 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68c00c3ae0 msgr2=0x7f68c01480b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:29.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.375+0000 7f68c61ac640 1 --2- 172.21.15.161:0/1566861683 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68c00c3ae0 0x7f68c01480b0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f68b400c430 tx=0x7f68b400c8f0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.375+0000 7f68c61ac640 1 -- 172.21.15.161:0/1566861683 shutdown_connections 2024-03-28T23:32:29.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.375+0000 7f68c61ac640 1 --2- 172.21.15.161:0/1566861683 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f68ac06b500 0x7f68ac06d9c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.375+0000 7f68c61ac640 1 --2- 172.21.15.161:0/1566861683 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68c00c3ae0 0x7f68c01480b0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.375+0000 7f68c61ac640 1 --2- 172.21.15.161:0/1566861683 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68c014f240 0x7f68c014ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.375+0000 7f68c61ac640 1 --2- 172.21.15.161:0/1566861683 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f68c00c3410 0x7f68c0154b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.375+0000 7f68c61ac640 1 -- 172.21.15.161:0/1566861683 >> 172.21.15.161:0/1566861683 conn(0x7f68c00c7790 msgr2=0x7f68c0156e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:29.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.375+0000 7f68c61ac640 1 -- 172.21.15.161:0/1566861683 shutdown_connections 2024-03-28T23:32:29.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.375+0000 7f68c61ac640 1 -- 172.21.15.161:0/1566861683 wait complete. 2024-03-28T23:32:29.472 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:32:29.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.573+0000 7f4f40a09640 1 Processor -- start 2024-03-28T23:32:29.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.573+0000 7f4f40a09640 1 -- start start 2024-03-28T23:32:29.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.574+0000 7f4f40a09640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4f3c150600 0x7f4f3c0a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:29.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.574+0000 7f4f40a09640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4f3c0aacb0 0x7f4f3c153980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:29.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.574+0000 7f4f40a09640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f3c1567c0 con 0x7f4f3c0a8fc0 2024-03-28T23:32:29.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.574+0000 7f4f40a09640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f4f3c156570 con 0x7f4f3c0aacb0 2024-03-28T23:32:29.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.574+0000 7f4f40a09640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f4f3c156aa0 con 0x7f4f3c150600 2024-03-28T23:32:29.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.574+0000 7f4f3ad9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4f3c0aacb0 0x7f4f3c153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:29.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.574+0000 7f4f3ad9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4f3c0aacb0 0x7f4f3c153980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33904/0 (socket says 172.21.15.161:33904) 2024-03-28T23:32:29.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.574+0000 7f4f3ad9d640 1 -- 172.21.15.161:0/3972924926 learned_addr learned my addr 172.21.15.161:0/3972924926 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:29.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.574+0000 7f4f3a59c640 1 --2- 172.21.15.161:0/3972924926 >> v2:172.21.15.170:3300/0 conn(0x7f4f3c150600 0x7f4f3c0a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:29.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.574+0000 7f4f3ad9d640 1 -- 172.21.15.161:0/3972924926 >> v2:172.21.15.170:3300/0 conn(0x7f4f3c150600 msgr2=0x7f4f3c0a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:29.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.574+0000 7f4f3ad9d640 1 --2- 172.21.15.161:0/3972924926 >> v2:172.21.15.170:3300/0 conn(0x7f4f3c150600 0x7f4f3c0a9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.574+0000 7f4f3ad9d640 1 -- 172.21.15.161:0/3972924926 >> v1:172.21.15.170:6789/0 conn(0x7f4f3c0a8fc0 legacy=0x7f4f3c153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:29.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.574+0000 7f4f3ad9d640 1 -- 172.21.15.161:0/3972924926 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f3c156d80 con 0x7f4f3c0aacb0 2024-03-28T23:32:29.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.574+0000 7f4f3a59c640 1 --2- 172.21.15.161:0/3972924926 >> v2:172.21.15.170:3300/0 conn(0x7f4f3c150600 0x7f4f3c0a9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:29.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.575+0000 7f4f3ad9d640 1 --2- 172.21.15.161:0/3972924926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f3c0aacb0 0x7f4f3c153980 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c0c4210 tx=0x7f4f3402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d12b25240a7d8f73 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:29.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.575+0000 7f4f39d9b640 1 -- 172.21.15.161:0/3972924926 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3402c9f0 con 0x7f4f3c0aacb0 2024-03-28T23:32:29.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.575+0000 7f4f39d9b640 1 -- 172.21.15.161:0/3972924926 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f3402cb90 con 0x7f4f3c0aacb0 2024-03-28T23:32:29.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.575+0000 7f4f39d9b640 1 -- 172.21.15.161:0/3972924926 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f340343f0 con 0x7f4f3c0aacb0 2024-03-28T23:32:29.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.575+0000 7f4f40a09640 1 -- 172.21.15.161:0/3972924926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f3c0aacb0 msgr2=0x7f4f3c153980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:29.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.575+0000 7f4f40a09640 1 --2- 172.21.15.161:0/3972924926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f3c0aacb0 0x7f4f3c153980 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c0c4210 tx=0x7f4f3402be50 comp rx=0 tx=0).stop 2024-03-28T23:32:29.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.576+0000 7f4f40a09640 1 -- 172.21.15.161:0/3972924926 shutdown_connections 2024-03-28T23:32:29.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.576+0000 7f4f40a09640 1 --2- 172.21.15.161:0/3972924926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f3c0aacb0 0x7f4f3c153980 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.576+0000 7f4f40a09640 1 --2- 172.21.15.161:0/3972924926 >> v2:172.21.15.170:3300/0 conn(0x7f4f3c150600 0x7f4f3c0a9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.576+0000 7f4f40a09640 1 -- 172.21.15.161:0/3972924926 >> 172.21.15.161:0/3972924926 conn(0x7f4f3c0c6670 msgr2=0x7f4f3c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:29.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.576+0000 7f4f40a09640 1 -- 172.21.15.161:0/3972924926 shutdown_connections 2024-03-28T23:32:29.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.576+0000 7f4f40a09640 1 -- 172.21.15.161:0/3972924926 wait complete. 2024-03-28T23:32:29.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.577+0000 7f4f40a09640 1 Processor -- start 2024-03-28T23:32:29.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.577+0000 7f4f40a09640 1 -- start start 2024-03-28T23:32:29.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.577+0000 7f4f40a09640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f3c0a8fc0 0x7f4f3c147a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:29.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.577+0000 7f4f40a09640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f3c0aacb0 0x7f4f3c1e89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:29.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.577+0000 7f4f40a09640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f3c150600 0x7f4f3c1e8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:29.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.577+0000 7f4f40a09640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3c153ec0 con 0x7f4f3c150600 2024-03-28T23:32:29.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.577+0000 7f4f40a09640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3c157340 con 0x7f4f3c0a8fc0 2024-03-28T23:32:29.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.577+0000 7f4f40a09640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4f3c1509e0 con 0x7f4f3c0aacb0 2024-03-28T23:32:29.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.578+0000 7f4f3ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f3c150600 0x7f4f3c1e8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:29.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.578+0000 7f4f3ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f3c150600 0x7f4f3c1e8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33906/0 (socket says 172.21.15.161:33906) 2024-03-28T23:32:29.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.578+0000 7f4f3ad9d640 1 -- 172.21.15.161:0/3173274933 learned_addr learned my addr 172.21.15.161:0/3173274933 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:29.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.578+0000 7f4f3a59c640 1 --2- 172.21.15.161:0/3173274933 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f3c0a8fc0 0x7f4f3c147a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:29.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.578+0000 7f4f33fff640 1 --2- 172.21.15.161:0/3173274933 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f3c0aacb0 0x7f4f3c1e89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:29.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.578+0000 7f4f3ad9d640 1 -- 172.21.15.161:0/3173274933 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f3c0aacb0 msgr2=0x7f4f3c1e89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:29.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.578+0000 7f4f3ad9d640 1 --2- 172.21.15.161:0/3173274933 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f3c0aacb0 0x7f4f3c1e89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.578+0000 7f4f3ad9d640 1 -- 172.21.15.161:0/3173274933 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f3c0a8fc0 msgr2=0x7f4f3c147a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:29.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.578+0000 7f4f3ad9d640 1 --2- 172.21.15.161:0/3173274933 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f3c0a8fc0 0x7f4f3c147a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.578+0000 7f4f3ad9d640 1 -- 172.21.15.161:0/3173274933 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f3c0c4e80 con 0x7f4f3c150600 2024-03-28T23:32:29.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.578+0000 7f4f3ad9d640 1 --2- 172.21.15.161:0/3173274933 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f3c150600 0x7f4f3c1e8f00 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f4f34009f40 tx=0x7f4f34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:29.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.578+0000 7f4f32ffd640 1 -- 172.21.15.161:0/3173274933 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f34043020 con 0x7f4f3c150600 2024-03-28T23:32:29.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.578+0000 7f4f32ffd640 1 -- 172.21.15.161:0/3173274933 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f3403e450 con 0x7f4f3c150600 2024-03-28T23:32:29.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.578+0000 7f4f40a09640 1 -- 172.21.15.161:0/3173274933 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f3c14fe30 con 0x7f4f3c150600 2024-03-28T23:32:29.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.578+0000 7f4f32ffd640 1 -- 172.21.15.161:0/3173274933 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f34009510 con 0x7f4f3c150600 2024-03-28T23:32:29.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.578+0000 7f4f40a09640 1 -- 172.21.15.161:0/3173274933 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f3c0c5120 con 0x7f4f3c150600 2024-03-28T23:32:29.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.579+0000 7f4f40a09640 1 -- 172.21.15.161:0/3173274933 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f3c04f4b0 con 0x7f4f3c150600 2024-03-28T23:32:29.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.580+0000 7f4f32ffd640 1 -- 172.21.15.161:0/3173274933 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4f34046020 con 0x7f4f3c150600 2024-03-28T23:32:29.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.580+0000 7f4f32ffd640 1 --2- 172.21.15.161:0/3173274933 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4f1c06b4d0 0x7f4f1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:29.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.580+0000 7f4f32ffd640 1 -- 172.21.15.161:0/3173274933 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5249+0+0 (secure 0 0 0) 0x7f4f340ac8f0 con 0x7f4f3c150600 2024-03-28T23:32:29.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.580+0000 7f4f3a59c640 1 --2- 172.21.15.161:0/3173274933 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4f1c06b4d0 0x7f4f1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:29.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.580+0000 7f4f3a59c640 1 --2- 172.21.15.161:0/3173274933 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4f1c06b4d0 0x7f4f1c06d990 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00a990 tx=0x7f4f2c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:29.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.584+0000 7f4f32ffd640 1 -- 172.21.15.161:0/3173274933 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4f3404d030 con 0x7f4f3c150600 2024-03-28T23:32:29.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.758+0000 7f4f40a09640 1 -- 172.21.15.161:0/3173274933 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4f3c0c06b0 con 0x7f4f3c150600 2024-03-28T23:32:29.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.759+0000 7f4f32ffd640 1 -- 172.21.15.161:0/3173274933 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) ==== 74+0+13201 (secure 0 0 0) 0x7f4f3407b9c0 con 0x7f4f3c150600 2024-03-28T23:32:29.759 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:29.759 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":69,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:32:28.999497+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:32:29.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.761+0000 7f4f40a09640 1 -- 172.21.15.161:0/3173274933 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4f1c06b4d0 msgr2=0x7f4f1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:29.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.761+0000 7f4f40a09640 1 --2- 172.21.15.161:0/3173274933 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4f1c06b4d0 0x7f4f1c06d990 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00a990 tx=0x7f4f2c008040 comp rx=0 tx=0).stop 2024-03-28T23:32:29.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.761+0000 7f4f40a09640 1 -- 172.21.15.161:0/3173274933 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f3c150600 msgr2=0x7f4f3c1e8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:29.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.761+0000 7f4f40a09640 1 --2- 172.21.15.161:0/3173274933 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f3c150600 0x7f4f3c1e8f00 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f4f34009f40 tx=0x7f4f34004a40 comp rx=0 tx=0).stop 2024-03-28T23:32:29.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.761+0000 7f4f40a09640 1 -- 172.21.15.161:0/3173274933 shutdown_connections 2024-03-28T23:32:29.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.761+0000 7f4f40a09640 1 --2- 172.21.15.161:0/3173274933 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4f1c06b4d0 0x7f4f1c06d990 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.761+0000 7f4f40a09640 1 --2- 172.21.15.161:0/3173274933 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f3c150600 0x7f4f3c1e8f00 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.761+0000 7f4f40a09640 1 --2- 172.21.15.161:0/3173274933 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f3c0aacb0 0x7f4f3c1e89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.761+0000 7f4f40a09640 1 --2- 172.21.15.161:0/3173274933 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f3c0a8fc0 0x7f4f3c147a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.761+0000 7f4f40a09640 1 -- 172.21.15.161:0/3173274933 >> 172.21.15.161:0/3173274933 conn(0x7f4f3c0c6670 msgr2=0x7f4f3c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:29.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.761+0000 7f4f40a09640 1 -- 172.21.15.161:0/3173274933 shutdown_connections 2024-03-28T23:32:29.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.761+0000 7f4f40a09640 1 -- 172.21.15.161:0/3173274933 wait complete. 2024-03-28T23:32:29.773 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:32:29.773 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:32:29.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.877+0000 7f3f02ec0640 1 Processor -- start 2024-03-28T23:32:29.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.878+0000 7f3f02ec0640 1 -- start start 2024-03-28T23:32:29.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.878+0000 7f3f02ec0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3efc1529d0 0x7f3efc06bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:29.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.878+0000 7f3f02ec0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3efc0c88b0 0x7f3efc0c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:29.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.878+0000 7f3f02ec0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3efc152db0 con 0x7f3efc0c6750 2024-03-28T23:32:29.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.878+0000 7f3f02ec0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f3efc0c7210 con 0x7f3efc0c88b0 2024-03-28T23:32:29.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.878+0000 7f3f02ec0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f3efc0c6c50 con 0x7f3efc1529d0 2024-03-28T23:32:29.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.878+0000 7f3f0145d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3efc0c88b0 0x7f3efc0c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:29.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.878+0000 7f3f0145d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3efc0c88b0 0x7f3efc0c54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33912/0 (socket says 172.21.15.161:33912) 2024-03-28T23:32:29.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.878+0000 7f3f0145d640 1 -- 172.21.15.161:0/1828363953 learned_addr learned my addr 172.21.15.161:0/1828363953 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:29.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.879+0000 7f3f00c5c640 1 --2- 172.21.15.161:0/1828363953 >> v2:172.21.15.170:3300/0 conn(0x7f3efc1529d0 0x7f3efc06bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:29.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.879+0000 7f3f0145d640 1 -- 172.21.15.161:0/1828363953 >> v2:172.21.15.170:3300/0 conn(0x7f3efc1529d0 msgr2=0x7f3efc06bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:29.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.879+0000 7f3f0145d640 1 --2- 172.21.15.161:0/1828363953 >> v2:172.21.15.170:3300/0 conn(0x7f3efc1529d0 0x7f3efc06bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.879+0000 7f3f0145d640 1 -- 172.21.15.161:0/1828363953 >> v1:172.21.15.170:6789/0 conn(0x7f3efc0c6750 legacy=0x7f3efc155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:29.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.879+0000 7f3f0145d640 1 -- 172.21.15.161:0/1828363953 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3efc0c8d40 con 0x7f3efc0c88b0 2024-03-28T23:32:29.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.879+0000 7f3f0145d640 1 --2- 172.21.15.161:0/1828363953 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3efc0c88b0 0x7f3efc0c54f0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f3efc0689e0 tx=0x7f3ef402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c33423d39b6aee5b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:29.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.880+0000 7f3efb7fe640 1 -- 172.21.15.161:0/1828363953 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef402ca80 con 0x7f3efc0c88b0 2024-03-28T23:32:29.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.880+0000 7f3efb7fe640 1 -- 172.21.15.161:0/1828363953 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ef402cc20 con 0x7f3efc0c88b0 2024-03-28T23:32:29.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.880+0000 7f3efb7fe640 1 -- 172.21.15.161:0/1828363953 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef40343f0 con 0x7f3efc0c88b0 2024-03-28T23:32:29.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.880+0000 7f3f02ec0640 1 -- 172.21.15.161:0/1828363953 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3efc0c88b0 msgr2=0x7f3efc0c54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:29.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.880+0000 7f3f02ec0640 1 --2- 172.21.15.161:0/1828363953 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3efc0c88b0 0x7f3efc0c54f0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f3efc0689e0 tx=0x7f3ef402bee0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.880+0000 7f3f02ec0640 1 -- 172.21.15.161:0/1828363953 shutdown_connections 2024-03-28T23:32:29.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.880+0000 7f3f02ec0640 1 --2- 172.21.15.161:0/1828363953 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3efc0c88b0 0x7f3efc0c54f0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.880+0000 7f3f02ec0640 1 --2- 172.21.15.161:0/1828363953 >> v2:172.21.15.170:3300/0 conn(0x7f3efc1529d0 0x7f3efc06bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.880+0000 7f3f02ec0640 1 -- 172.21.15.161:0/1828363953 >> 172.21.15.161:0/1828363953 conn(0x7f3efc0c7430 msgr2=0x7f3efc0c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:29.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.880+0000 7f3f02ec0640 1 -- 172.21.15.161:0/1828363953 shutdown_connections 2024-03-28T23:32:29.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.880+0000 7f3f02ec0640 1 -- 172.21.15.161:0/1828363953 wait complete. 2024-03-28T23:32:29.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.881+0000 7f3f02ec0640 1 Processor -- start 2024-03-28T23:32:29.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.881+0000 7f3f02ec0640 1 -- start start 2024-03-28T23:32:29.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.882+0000 7f3f02ec0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3efc0c6750 0x7f3efc14e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:29.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.882+0000 7f3f02ec0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3efc0c88b0 0x7f3efc14cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:29.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.882+0000 7f3f02ec0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3efc1529d0 0x7f3efc0af990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:29.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.882+0000 7f3f02ec0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc155710 con 0x7f3efc1529d0 2024-03-28T23:32:29.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.882+0000 7f3f02ec0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc155430 con 0x7f3efc0c6750 2024-03-28T23:32:29.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.882+0000 7f3f02ec0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3efc0aa160 con 0x7f3efc0c88b0 2024-03-28T23:32:29.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.882+0000 7f3f0145d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3efc1529d0 0x7f3efc0af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:29.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.882+0000 7f3f0145d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3efc1529d0 0x7f3efc0af990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33918/0 (socket says 172.21.15.161:33918) 2024-03-28T23:32:29.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.882+0000 7f3f0145d640 1 -- 172.21.15.161:0/4238016897 learned_addr learned my addr 172.21.15.161:0/4238016897 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:29.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.882+0000 7f3efbfff640 1 --2- 172.21.15.161:0/4238016897 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3efc0c88b0 0x7f3efc14cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:29.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.882+0000 7f3f00c5c640 1 --2- 172.21.15.161:0/4238016897 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3efc0c6750 0x7f3efc14e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:29.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.882+0000 7f3f0145d640 1 -- 172.21.15.161:0/4238016897 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3efc0c88b0 msgr2=0x7f3efc14cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:29.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.882+0000 7f3f0145d640 1 --2- 172.21.15.161:0/4238016897 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3efc0c88b0 0x7f3efc14cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.882+0000 7f3f0145d640 1 -- 172.21.15.161:0/4238016897 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3efc0c6750 msgr2=0x7f3efc14e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:29.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.882+0000 7f3f0145d640 1 --2- 172.21.15.161:0/4238016897 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3efc0c6750 0x7f3efc14e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:29.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.883+0000 7f3f0145d640 1 -- 172.21.15.161:0/4238016897 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3efc1eb0d0 con 0x7f3efc1529d0 2024-03-28T23:32:29.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.883+0000 7f3f00c5c640 1 --2- 172.21.15.161:0/4238016897 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3efc0c6750 0x7f3efc14e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:29.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.883+0000 7f3f0145d640 1 --2- 172.21.15.161:0/4238016897 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3efc1529d0 0x7f3efc0af990 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4036040 tx=0x7f3ef4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:29.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.883+0000 7f3ef97fa640 1 -- 172.21.15.161:0/4238016897 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4043020 con 0x7f3efc1529d0 2024-03-28T23:32:29.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.883+0000 7f3ef97fa640 1 -- 172.21.15.161:0/4238016897 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ef403e450 con 0x7f3efc1529d0 2024-03-28T23:32:29.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.883+0000 7f3ef97fa640 1 -- 172.21.15.161:0/4238016897 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4009510 con 0x7f3efc1529d0 2024-03-28T23:32:29.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.883+0000 7f3f02ec0640 1 -- 172.21.15.161:0/4238016897 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3efc1eb3b0 con 0x7f3efc1529d0 2024-03-28T23:32:29.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.883+0000 7f3f02ec0640 1 -- 172.21.15.161:0/4238016897 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3efc14d420 con 0x7f3efc1529d0 2024-03-28T23:32:29.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.884+0000 7f3f02ec0640 1 -- 172.21.15.161:0/4238016897 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3efc04f4b0 con 0x7f3efc1529d0 2024-03-28T23:32:29.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.884+0000 7f3ef97fa640 1 -- 172.21.15.161:0/4238016897 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3ef4046020 con 0x7f3efc1529d0 2024-03-28T23:32:29.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.885+0000 7f3ef97fa640 1 --2- 172.21.15.161:0/4238016897 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3ee806b500 0x7f3ee806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:29.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.885+0000 7f3ef97fa640 1 -- 172.21.15.161:0/4238016897 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5249+0+0 (secure 0 0 0) 0x7f3ef40ac7d0 con 0x7f3efc1529d0 2024-03-28T23:32:29.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.885+0000 7f3f00c5c640 1 --2- 172.21.15.161:0/4238016897 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3ee806b500 0x7f3ee806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:29.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.885+0000 7f3f00c5c640 1 --2- 172.21.15.161:0/4238016897 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3ee806b500 0x7f3ee806d9c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3eec006fd0 tx=0x7f3eec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:29.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:29.889+0000 7f3ef97fa640 1 -- 172.21.15.161:0/4238016897 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3ef404d030 con 0x7f3efc1529d0 2024-03-28T23:32:30.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.062+0000 7f3f02ec0640 1 -- 172.21.15.161:0/4238016897 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3efc0c11d0 con 0x7f3efc1529d0 2024-03-28T23:32:30.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.063+0000 7f3ef97fa640 1 -- 172.21.15.161:0/4238016897 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) ==== 93+0+10 (secure 0 0 0) 0x7f3ef407b810 con 0x7f3efc1529d0 2024-03-28T23:32:30.063 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:32:30.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.064+0000 7f3f02ec0640 1 -- 172.21.15.161:0/4238016897 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3ee806b500 msgr2=0x7f3ee806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:30.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.065+0000 7f3f02ec0640 1 --2- 172.21.15.161:0/4238016897 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3ee806b500 0x7f3ee806d9c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3eec006fd0 tx=0x7f3eec008040 comp rx=0 tx=0).stop 2024-03-28T23:32:30.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.065+0000 7f3f02ec0640 1 -- 172.21.15.161:0/4238016897 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3efc1529d0 msgr2=0x7f3efc0af990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:30.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.065+0000 7f3f02ec0640 1 --2- 172.21.15.161:0/4238016897 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3efc1529d0 0x7f3efc0af990 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4036040 tx=0x7f3ef4004a40 comp rx=0 tx=0).stop 2024-03-28T23:32:30.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.065+0000 7f3f02ec0640 1 -- 172.21.15.161:0/4238016897 shutdown_connections 2024-03-28T23:32:30.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.065+0000 7f3f02ec0640 1 --2- 172.21.15.161:0/4238016897 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3ee806b500 0x7f3ee806d9c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:30.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.065+0000 7f3f02ec0640 1 --2- 172.21.15.161:0/4238016897 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3efc1529d0 0x7f3efc0af990 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:30.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.065+0000 7f3f02ec0640 1 --2- 172.21.15.161:0/4238016897 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3efc0c88b0 0x7f3efc14cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:30.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.065+0000 7f3f02ec0640 1 --2- 172.21.15.161:0/4238016897 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3efc0c6750 0x7f3efc14e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:30.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.065+0000 7f3f02ec0640 1 -- 172.21.15.161:0/4238016897 >> 172.21.15.161:0/4238016897 conn(0x7f3efc0c7430 msgr2=0x7f3efc0c5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:30.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.065+0000 7f3f02ec0640 1 -- 172.21.15.161:0/4238016897 shutdown_connections 2024-03-28T23:32:30.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.065+0000 7f3f02ec0640 1 -- 172.21.15.161:0/4238016897 wait complete. 2024-03-28T23:32:30.077 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T23:32:30.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.177+0000 7f88f702e640 1 Processor -- start 2024-03-28T23:32:30.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.177+0000 7f88f702e640 1 -- start start 2024-03-28T23:32:30.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.178+0000 7f88f702e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f88f014f260 0x7f88f01592e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:30.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.178+0000 7f88f702e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88f00cb4e0 con 0x7f88f00c8970 2024-03-28T23:32:30.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.178+0000 7f88f702e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88f00cb7c0 con 0x7f88f00caf60 2024-03-28T23:32:30.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.178+0000 7f88f702e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f88f0158e50 con 0x7f88f014f260 2024-03-28T23:32:30.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.178+0000 7f88f55cb640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f88f00c8970 0x7f88f00c53f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:43348/0 (socket says 172.21.15.161:43348) 2024-03-28T23:32:30.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.178+0000 7f88f55cb640 1 -- 172.21.15.161:0/2945142734 learned_addr learned my addr 172.21.15.161:0/2945142734 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:30.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.178+0000 7f88f4dca640 1 --2- 172.21.15.161:0/2945142734 >> v2:172.21.15.170:3300/0 conn(0x7f88f014f260 0x7f88f01592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:30.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.179+0000 7f88ef7fe640 1 -- 172.21.15.161:0/2945142734 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 868643925 0 0) 0x7f88f00cb4e0 con 0x7f88f00c8970 2024-03-28T23:32:30.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.179+0000 7f88ef7fe640 1 -- 172.21.15.161:0/2945142734 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88d8003520 con 0x7f88f00c8970 2024-03-28T23:32:30.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.179+0000 7f88f4dca640 1 -- 172.21.15.161:0/2945142734 >> v1:172.21.15.170:6789/0 conn(0x7f88f00caf60 legacy=0x7f88f0155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:30.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.179+0000 7f88f4dca640 1 -- 172.21.15.161:0/2945142734 >> v1:172.21.15.161:6789/0 conn(0x7f88f00c8970 legacy=0x7f88f00c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:30.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.179+0000 7f88ef7fe640 1 -- 172.21.15.161:0/2945142734 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3076392812 0 0) 0x7f88f00cb7c0 con 0x7f88f00caf60 2024-03-28T23:32:30.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.179+0000 7f88f4dca640 1 -- 172.21.15.161:0/2945142734 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f0158800 con 0x7f88f014f260 2024-03-28T23:32:30.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.180+0000 7f88f4dca640 1 --2- 172.21.15.161:0/2945142734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88f014f260 0x7f88f01592e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f88e00097a0 tx=0x7f88e002b6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e30f2191e090e399 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:30.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.180+0000 7f88ef7fe640 1 -- 172.21.15.161:0/2945142734 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e0037070 con 0x7f88f014f260 2024-03-28T23:32:30.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.180+0000 7f88ef7fe640 1 -- 172.21.15.161:0/2945142734 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88e0030510 con 0x7f88f014f260 2024-03-28T23:32:30.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.180+0000 7f88ef7fe640 1 -- 172.21.15.161:0/2945142734 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e80019f0 con 0x7f88f014f260 2024-03-28T23:32:30.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.180+0000 7f88f702e640 1 -- 172.21.15.161:0/2945142734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88f014f260 msgr2=0x7f88f01592e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:30.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.180+0000 7f88f702e640 1 --2- 172.21.15.161:0/2945142734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88f014f260 0x7f88f01592e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f88e00097a0 tx=0x7f88e002b6b0 comp rx=0 tx=0).stop 2024-03-28T23:32:30.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.180+0000 7f88f702e640 1 -- 172.21.15.161:0/2945142734 shutdown_connections 2024-03-28T23:32:30.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.180+0000 7f88f702e640 1 --2- 172.21.15.161:0/2945142734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88f014f260 0x7f88f01592e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:30.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.180+0000 7f88f702e640 1 -- 172.21.15.161:0/2945142734 >> 172.21.15.161:0/2945142734 conn(0x7f88f0072ca0 msgr2=0x7f88f00730b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:30.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.180+0000 7f88f702e640 1 -- 172.21.15.161:0/2945142734 shutdown_connections 2024-03-28T23:32:30.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.180+0000 7f88f702e640 1 -- 172.21.15.161:0/2945142734 wait complete. 2024-03-28T23:32:30.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.181+0000 7f88f702e640 1 Processor -- start 2024-03-28T23:32:30.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.182+0000 7f88f702e640 1 -- start start 2024-03-28T23:32:30.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.182+0000 7f88f702e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88f00c8970 0x7f88f0157620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:30.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.182+0000 7f88f702e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88f00caf60 0x7f88f01eadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:30.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.182+0000 7f88f702e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88f014f260 0x7f88f014b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:30.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.182+0000 7f88f702e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f88f00c5b00 con 0x7f88f00c8970 2024-03-28T23:32:30.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.182+0000 7f88f702e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f88f014f640 con 0x7f88f00caf60 2024-03-28T23:32:30.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.182+0000 7f88f702e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f88f00c5c80 con 0x7f88f014f260 2024-03-28T23:32:30.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.182+0000 7f88f4dca640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88f00c8970 0x7f88f0157620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:30.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.182+0000 7f88f4dca640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88f00c8970 0x7f88f0157620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33934/0 (socket says 172.21.15.161:33934) 2024-03-28T23:32:30.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.182+0000 7f88f4dca640 1 -- 172.21.15.161:0/1982600713 learned_addr learned my addr 172.21.15.161:0/1982600713 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:30.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.183+0000 7f88f55cb640 1 --2- 172.21.15.161:0/1982600713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88f014f260 0x7f88f014b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:30.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.183+0000 7f88effff640 1 --2- 172.21.15.161:0/1982600713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88f00caf60 0x7f88f01eadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:30.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.183+0000 7f88f4dca640 1 -- 172.21.15.161:0/1982600713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88f014f260 msgr2=0x7f88f014b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:30.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.183+0000 7f88f4dca640 1 --2- 172.21.15.161:0/1982600713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88f014f260 0x7f88f014b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:30.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.183+0000 7f88f4dca640 1 -- 172.21.15.161:0/1982600713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88f00caf60 msgr2=0x7f88f01eadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:30.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.183+0000 7f88f4dca640 1 --2- 172.21.15.161:0/1982600713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88f00caf60 0x7f88f01eadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:30.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.183+0000 7f88f4dca640 1 -- 172.21.15.161:0/1982600713 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f01eb330 con 0x7f88f00c8970 2024-03-28T23:32:30.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.183+0000 7f88f55cb640 1 --2- 172.21.15.161:0/1982600713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88f014f260 0x7f88f014b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:30.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.183+0000 7f88effff640 1 --2- 172.21.15.161:0/1982600713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88f00caf60 0x7f88f01eadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:30.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.183+0000 7f88f4dca640 1 --2- 172.21.15.161:0/1982600713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88f00c8970 0x7f88f0157620 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f88e002b6f0 tx=0x7f88e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:30.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.183+0000 7f88ed7fa640 1 -- 172.21.15.161:0/1982600713 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e0042070 con 0x7f88f00c8970 2024-03-28T23:32:30.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.183+0000 7f88ed7fa640 1 -- 172.21.15.161:0/1982600713 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88e003e870 con 0x7f88f00c8970 2024-03-28T23:32:30.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.183+0000 7f88f702e640 1 -- 172.21.15.161:0/1982600713 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88f01ed9d0 con 0x7f88f00c8970 2024-03-28T23:32:30.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.183+0000 7f88ed7fa640 1 -- 172.21.15.161:0/1982600713 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e0037040 con 0x7f88f00c8970 2024-03-28T23:32:30.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.183+0000 7f88f702e640 1 -- 172.21.15.161:0/1982600713 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88f01512e0 con 0x7f88f00c8970 2024-03-28T23:32:30.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.184+0000 7f88f702e640 1 -- 172.21.15.161:0/1982600713 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88f004f550 con 0x7f88f00c8970 2024-03-28T23:32:30.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.184+0000 7f88ed7fa640 1 -- 172.21.15.161:0/1982600713 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f88e0045070 con 0x7f88f00c8970 2024-03-28T23:32:30.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.185+0000 7f88ed7fa640 1 --2- 172.21.15.161:0/1982600713 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f88dc06b4d0 0x7f88dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:30.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.185+0000 7f88ed7fa640 1 -- 172.21.15.161:0/1982600713 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5249+0+0 (secure 0 0 0) 0x7f88e00abcc0 con 0x7f88f00c8970 2024-03-28T23:32:30.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.185+0000 7f88effff640 1 --2- 172.21.15.161:0/1982600713 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f88dc06b4d0 0x7f88dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:30.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.185+0000 7f88effff640 1 --2- 172.21.15.161:0/1982600713 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f88dc06b4d0 0x7f88dc06d990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f88f00ca520 tx=0x7f88e4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:30.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.189+0000 7f88ed7fa640 1 -- 172.21.15.161:0/1982600713 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f88e007aeb0 con 0x7f88f00c8970 2024-03-28T23:32:30.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.362+0000 7f88f702e640 1 -- 172.21.15.161:0/1982600713 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f88f006cfb0 con 0x7f88f00c8970 2024-03-28T23:32:30.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.362+0000 7f88ed7fa640 1 -- 172.21.15.161:0/1982600713 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v69) ==== 84+0+65 (secure 0 0 0) 0x7f88e007a7b0 con 0x7f88f00c8970 2024-03-28T23:32:30.363 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:30.363 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2024-03-28T23:32:30.363 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:32:30.363 INFO:teuthology.orchestra.run.smithi161.stdout: "poolnum": 1, 2024-03-28T23:32:30.363 INFO:teuthology.orchestra.run.smithi161.stdout: "poolname": ".mgr" 2024-03-28T23:32:30.363 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:32:30.364 INFO:teuthology.orchestra.run.smithi161.stdout:] 2024-03-28T23:32:30.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.364+0000 7f88f702e640 1 -- 172.21.15.161:0/1982600713 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f88dc06b4d0 msgr2=0x7f88dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:30.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.364+0000 7f88f702e640 1 --2- 172.21.15.161:0/1982600713 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f88dc06b4d0 0x7f88dc06d990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f88f00ca520 tx=0x7f88e4009040 comp rx=0 tx=0).stop 2024-03-28T23:32:30.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.364+0000 7f88f702e640 1 -- 172.21.15.161:0/1982600713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88f00c8970 msgr2=0x7f88f0157620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:30.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.364+0000 7f88f702e640 1 --2- 172.21.15.161:0/1982600713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88f00c8970 0x7f88f0157620 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f88e002b6f0 tx=0x7f88e00046c0 comp rx=0 tx=0).stop 2024-03-28T23:32:30.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.364+0000 7f88f702e640 1 -- 172.21.15.161:0/1982600713 shutdown_connections 2024-03-28T23:32:30.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.364+0000 7f88f702e640 1 --2- 172.21.15.161:0/1982600713 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f88dc06b4d0 0x7f88dc06d990 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:30.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.364+0000 7f88f702e640 1 --2- 172.21.15.161:0/1982600713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88f014f260 0x7f88f014b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:30.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.364+0000 7f88f702e640 1 --2- 172.21.15.161:0/1982600713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88f00caf60 0x7f88f01eadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:30.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.364+0000 7f88f702e640 1 --2- 172.21.15.161:0/1982600713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88f00c8970 0x7f88f0157620 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:30.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.364+0000 7f88f702e640 1 -- 172.21.15.161:0/1982600713 >> 172.21.15.161:0/1982600713 conn(0x7f88f0072ca0 msgr2=0x7f88f0073080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:30.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.364+0000 7f88f702e640 1 -- 172.21.15.161:0/1982600713 shutdown_connections 2024-03-28T23:32:30.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.365+0000 7f88f702e640 1 -- 172.21.15.161:0/1982600713 wait complete. 2024-03-28T23:32:30.376 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-28T23:32:30.376 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-28T23:32:30.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.481+0000 7fb9d1c07640 1 Processor -- start 2024-03-28T23:32:30.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.481+0000 7fb9d1c07640 1 -- start start 2024-03-28T23:32:30.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.482+0000 7fb9d1c07640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb9cc0abf50 0x7fb9cc0c4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:30.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.482+0000 7fb9d1c07640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9cc149ec0 con 0x7fb9cc06d390 2024-03-28T23:32:30.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.482+0000 7fb9d1c07640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9cc0ac3a0 con 0x7fb9cc0abb70 2024-03-28T23:32:30.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.482+0000 7fb9d1c07640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fb9cc153ab0 con 0x7fb9cc0abf50 2024-03-28T23:32:30.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.482+0000 7fb9c2ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb9cc06d390 0x7fb9cc0c46b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:43356/0 (socket says 172.21.15.161:43356) 2024-03-28T23:32:30.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.482+0000 7fb9c2ffd640 1 -- 172.21.15.161:0/2237614057 learned_addr learned my addr 172.21.15.161:0/2237614057 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:30.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.482+0000 7fb9cbfff640 1 --2- 172.21.15.161:0/2237614057 >> v2:172.21.15.170:3300/0 conn(0x7fb9cc0abf50 0x7fb9cc0c4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:30.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.482+0000 7fb9caffd640 1 -- 172.21.15.161:0/2237614057 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1520415299 0 0) 0x7fb9cc149ec0 con 0x7fb9cc06d390 2024-03-28T23:32:30.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.482+0000 7fb9caffd640 1 -- 172.21.15.161:0/2237614057 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9b8003520 con 0x7fb9cc06d390 2024-03-28T23:32:30.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.482+0000 7fb9caffd640 1 -- 172.21.15.161:0/2237614057 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4263997778 0 0) 0x7fb9b8003520 con 0x7fb9cc06d390 2024-03-28T23:32:30.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.482+0000 7fb9caffd640 1 -- 172.21.15.161:0/2237614057 >> v2:172.21.15.170:3300/0 conn(0x7fb9cc0abf50 msgr2=0x7fb9cc0c4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:30.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.482+0000 7fb9caffd640 1 --2- 172.21.15.161:0/2237614057 >> v2:172.21.15.170:3300/0 conn(0x7fb9cc0abf50 0x7fb9cc0c4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:30.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.482+0000 7fb9caffd640 1 -- 172.21.15.161:0/2237614057 >> v1:172.21.15.170:6789/0 conn(0x7fb9cc0abb70 legacy=0x7fb9cc06dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:30.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.482+0000 7fb9caffd640 1 -- 172.21.15.161:0/2237614057 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9cc153180 con 0x7fb9cc06d390 2024-03-28T23:32:30.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.482+0000 7fb9caffd640 1 -- 172.21.15.161:0/2237614057 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fb9b4001670 con 0x7fb9cc06d390 2024-03-28T23:32:30.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.482+0000 7fb9caffd640 1 -- 172.21.15.161:0/2237614057 >> v1:172.21.15.161:6789/0 conn(0x7fb9cc06d390 legacy=0x7fb9cc0c46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:30.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.482+0000 7fb9caffd640 1 --2- 172.21.15.161:0/2237614057 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9b80039b0 0x7fb9b8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:30.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.483+0000 7fb9caffd640 1 --2- 172.21.15.161:0/2237614057 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9b8004800 0x7fb9b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:30.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.483+0000 7fb9caffd640 1 --2- 172.21.15.161:0/2237614057 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9b80051c0 0x7fb9b800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:30.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.483+0000 7fb9caffd640 1 -- 172.21.15.161:0/2237614057 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc153ab0 con 0x7fb9b80039b0 2024-03-28T23:32:30.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.483+0000 7fb9caffd640 1 -- 172.21.15.161:0/2237614057 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b800c060 con 0x7fb9b8004800 2024-03-28T23:32:30.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.483+0000 7fb9caffd640 1 -- 172.21.15.161:0/2237614057 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb9b800c260 con 0x7fb9b80051c0 2024-03-28T23:32:30.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.483+0000 7fb9cb7fe640 1 --2- 172.21.15.161:0/2237614057 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9b80039b0 0x7fb9b8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:30.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.483+0000 7fb9c2ffd640 1 --2- 172.21.15.161:0/2237614057 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9b8004800 0x7fb9b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:30.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.483+0000 7fb9cbfff640 1 --2- 172.21.15.161:0/2237614057 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9b80051c0 0x7fb9b800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:30.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.483+0000 7fb9cb7fe640 1 -- 172.21.15.161:0/2237614057 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9b80051c0 msgr2=0x7fb9b800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:30.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.483+0000 7fb9cb7fe640 1 --2- 172.21.15.161:0/2237614057 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9b80051c0 0x7fb9b800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:30.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.483+0000 7fb9cb7fe640 1 -- 172.21.15.161:0/2237614057 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9b8004800 msgr2=0x7fb9b8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:30.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.483+0000 7fb9cb7fe640 1 --2- 172.21.15.161:0/2237614057 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9b8004800 0x7fb9b8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:30.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.483+0000 7fb9cb7fe640 1 -- 172.21.15.161:0/2237614057 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9b8003520 con 0x7fb9b80039b0 2024-03-28T23:32:30.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.483+0000 7fb9c2ffd640 1 --2- 172.21.15.161:0/2237614057 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9b8004800 0x7fb9b8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:30.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.483+0000 7fb9cbfff640 1 -- 172.21.15.161:0/2237614057 reap_dead start 2024-03-28T23:32:30.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.484+0000 7fb9cb7fe640 1 --2- 172.21.15.161:0/2237614057 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9b80039b0 0x7fb9b8003dd0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc001750 tx=0x7fb9b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b165bca2571409a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:30.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.484+0000 7fb9caffd640 1 -- 172.21.15.161:0/2237614057 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc001370 con 0x7fb9b80039b0 2024-03-28T23:32:30.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.484+0000 7fb9caffd640 1 -- 172.21.15.161:0/2237614057 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9bc0052d0 con 0x7fb9b80039b0 2024-03-28T23:32:30.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.484+0000 7fb9caffd640 1 -- 172.21.15.161:0/2237614057 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc003830 con 0x7fb9b80039b0 2024-03-28T23:32:30.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.484+0000 7fb9d1c07640 1 -- 172.21.15.161:0/2237614057 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9b80039b0 msgr2=0x7fb9b8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:30.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.484+0000 7fb9d1c07640 1 --2- 172.21.15.161:0/2237614057 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9b80039b0 0x7fb9b8003dd0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc001750 tx=0x7fb9b8005830 comp rx=0 tx=0).stop 2024-03-28T23:32:30.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.484+0000 7fb9d1c07640 1 -- 172.21.15.161:0/2237614057 shutdown_connections 2024-03-28T23:32:30.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.484+0000 7fb9d1c07640 1 --2- 172.21.15.161:0/2237614057 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9b80039b0 0x7fb9b8003dd0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:30.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.484+0000 7fb9d1c07640 1 -- 172.21.15.161:0/2237614057 >> 172.21.15.161:0/2237614057 conn(0x7fb9cc06b150 msgr2=0x7fb9cc06b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:30.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.484+0000 7fb9d1c07640 1 -- 172.21.15.161:0/2237614057 shutdown_connections 2024-03-28T23:32:30.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.484+0000 7fb9d1c07640 1 -- 172.21.15.161:0/2237614057 wait complete. 2024-03-28T23:32:30.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.485+0000 7fb9d1c07640 1 Processor -- start 2024-03-28T23:32:30.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.486+0000 7fb9d1c07640 1 -- start start 2024-03-28T23:32:30.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.486+0000 7fb9d1c07640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9b80039b0 0x7fb9cc1eb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:30.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.486+0000 7fb9d1c07640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9cc1eee90 0x7fb9cc150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:30.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.486+0000 7fb9d1c07640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9cc1ef2f0 0x7fb9cc1514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:30.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.486+0000 7fb9d1c07640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc06dcf0 con 0x7fb9cc1ef2f0 2024-03-28T23:32:30.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.486+0000 7fb9d1c07640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc153740 con 0x7fb9b80039b0 2024-03-28T23:32:30.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.486+0000 7fb9d1c07640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb9cc06de70 con 0x7fb9cc1eee90 2024-03-28T23:32:30.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.486+0000 7fb9cbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9cc1ef2f0 0x7fb9cc1514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:30.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.486+0000 7fb9cbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9cc1ef2f0 0x7fb9cc1514d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33948/0 (socket says 172.21.15.161:33948) 2024-03-28T23:32:30.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.486+0000 7fb9cbfff640 1 -- 172.21.15.161:0/474174016 learned_addr learned my addr 172.21.15.161:0/474174016 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:30.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.486+0000 7fb9cb7fe640 1 --2- 172.21.15.161:0/474174016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9b80039b0 0x7fb9cc1eb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:30.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.486+0000 7fb9c2ffd640 1 --2- 172.21.15.161:0/474174016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9cc1eee90 0x7fb9cc150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:30.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.487+0000 7fb9cbfff640 1 -- 172.21.15.161:0/474174016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9cc1eee90 msgr2=0x7fb9cc150f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:30.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.487+0000 7fb9cbfff640 1 --2- 172.21.15.161:0/474174016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9cc1eee90 0x7fb9cc150f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:30.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.487+0000 7fb9cbfff640 1 -- 172.21.15.161:0/474174016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9b80039b0 msgr2=0x7fb9cc1eb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:30.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.487+0000 7fb9cbfff640 1 --2- 172.21.15.161:0/474174016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9b80039b0 0x7fb9cc1eb080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:30.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.487+0000 7fb9cbfff640 1 -- 172.21.15.161:0/474174016 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9cc151a10 con 0x7fb9cc1ef2f0 2024-03-28T23:32:30.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.487+0000 7fb9cb7fe640 1 --2- 172.21.15.161:0/474174016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9b80039b0 0x7fb9cc1eb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:30.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.487+0000 7fb9c2ffd640 1 --2- 172.21.15.161:0/474174016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9cc1eee90 0x7fb9cc150f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:30.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.487+0000 7fb9cbfff640 1 --2- 172.21.15.161:0/474174016 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9cc1ef2f0 0x7fb9cc1514d0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc06c0e0 tx=0x7fb9c40115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:30.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.487+0000 7fb9c8ff9640 1 -- 172.21.15.161:0/474174016 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4011e40 con 0x7fb9cc1ef2f0 2024-03-28T23:32:30.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.487+0000 7fb9c8ff9640 1 -- 172.21.15.161:0/474174016 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9c401b070 con 0x7fb9cc1ef2f0 2024-03-28T23:32:30.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.487+0000 7fb9d1c07640 1 -- 172.21.15.161:0/474174016 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9cc070990 con 0x7fb9cc1ef2f0 2024-03-28T23:32:30.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.487+0000 7fb9c8ff9640 1 -- 172.21.15.161:0/474174016 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4003990 con 0x7fb9cc1ef2f0 2024-03-28T23:32:30.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.487+0000 7fb9d1c07640 1 -- 172.21.15.161:0/474174016 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9cc070f50 con 0x7fb9cc1ef2f0 2024-03-28T23:32:30.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.488+0000 7fb9d1c07640 1 -- 172.21.15.161:0/474174016 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9cc068f90 con 0x7fb9cc1ef2f0 2024-03-28T23:32:30.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.488+0000 7fb9c8ff9640 1 -- 172.21.15.161:0/474174016 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb9c4005d10 con 0x7fb9cc1ef2f0 2024-03-28T23:32:30.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.489+0000 7fb9c8ff9640 1 --2- 172.21.15.161:0/474174016 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb9ac06b500 0x7fb9ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:30.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.489+0000 7fb9cb7fe640 1 --2- 172.21.15.161:0/474174016 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb9ac06b500 0x7fb9ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:30.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.489+0000 7fb9c8ff9640 1 -- 172.21.15.161:0/474174016 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5249+0+0 (secure 0 0 0) 0x7fb9c408a250 con 0x7fb9cc1ef2f0 2024-03-28T23:32:30.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.489+0000 7fb9cb7fe640 1 --2- 172.21.15.161:0/474174016 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb9ac06b500 0x7fb9ac06d9c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc033000 tx=0x7fb9bc009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:30.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.494+0000 7fb9c8ff9640 1 -- 172.21.15.161:0/474174016 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb9c4014350 con 0x7fb9cc1ef2f0 2024-03-28T23:32:30.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:30.666+0000 7fb9d1c07640 1 -- 172.21.15.161:0/474174016 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fb9cc06b7c0 con 0x7fb9cc1ef2f0 2024-03-28T23:32:31.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.012+0000 7fb9c8ff9640 1 -- 172.21.15.161:0/474174016 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v70) ==== 138+0+0 (secure 0 0 0) 0x7fb9c4059320 con 0x7fb9cc1ef2f0 2024-03-28T23:32:31.013 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_metadata' created 2024-03-28T23:32:31.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.014+0000 7fb9d1c07640 1 -- 172.21.15.161:0/474174016 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb9ac06b500 msgr2=0x7fb9ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:31.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.014+0000 7fb9d1c07640 1 --2- 172.21.15.161:0/474174016 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb9ac06b500 0x7fb9ac06d9c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc033000 tx=0x7fb9bc009030 comp rx=0 tx=0).stop 2024-03-28T23:32:31.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.014+0000 7fb9d1c07640 1 -- 172.21.15.161:0/474174016 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9cc1ef2f0 msgr2=0x7fb9cc1514d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:31.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.014+0000 7fb9d1c07640 1 --2- 172.21.15.161:0/474174016 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9cc1ef2f0 0x7fb9cc1514d0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc06c0e0 tx=0x7fb9c40115a0 comp rx=0 tx=0).stop 2024-03-28T23:32:31.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.014+0000 7fb9d1c07640 1 -- 172.21.15.161:0/474174016 shutdown_connections 2024-03-28T23:32:31.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.014+0000 7fb9d1c07640 1 --2- 172.21.15.161:0/474174016 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb9ac06b500 0x7fb9ac06d9c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:31.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.014+0000 7fb9d1c07640 1 --2- 172.21.15.161:0/474174016 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9cc1ef2f0 0x7fb9cc1514d0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:31.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.014+0000 7fb9d1c07640 1 --2- 172.21.15.161:0/474174016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9cc1eee90 0x7fb9cc150f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:31.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.014+0000 7fb9d1c07640 1 --2- 172.21.15.161:0/474174016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9b80039b0 0x7fb9cc1eb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:31.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.014+0000 7fb9d1c07640 1 -- 172.21.15.161:0/474174016 >> 172.21.15.161:0/474174016 conn(0x7fb9cc06b150 msgr2=0x7fb9cc0c43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:31.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.014+0000 7fb9d1c07640 1 -- 172.21.15.161:0/474174016 shutdown_connections 2024-03-28T23:32:31.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.014+0000 7fb9d1c07640 1 -- 172.21.15.161:0/474174016 wait complete. 2024-03-28T23:32:31.029 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T23:32:31.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.137+0000 7f05dfd48640 1 Processor -- start 2024-03-28T23:32:31.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.137+0000 7f05dfd48640 1 -- start start 2024-03-28T23:32:31.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.138+0000 7f05dfd48640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f05d80c0330 0x7f05d80c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:31.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.138+0000 7f05dfd48640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f05d81589b0 0x7f05d8158e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:31.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.138+0000 7f05dfd48640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05d80c6f60 con 0x7f05d8152bc0 2024-03-28T23:32:31.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.138+0000 7f05dfd48640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f05d80c6120 con 0x7f05d81589b0 2024-03-28T23:32:31.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.138+0000 7f05dfd48640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f05d80c6400 con 0x7f05d80c0330 2024-03-28T23:32:31.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.138+0000 7f05de2e5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f05d81589b0 0x7f05d8158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:31.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.138+0000 7f05de2e5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f05d81589b0 0x7f05d8158e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40138/0 (socket says 172.21.15.161:40138) 2024-03-28T23:32:31.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.138+0000 7f05de2e5640 1 -- 172.21.15.161:0/699204079 learned_addr learned my addr 172.21.15.161:0/699204079 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:31.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.139+0000 7f05de2e5640 1 -- 172.21.15.161:0/699204079 >> v2:172.21.15.170:3300/0 conn(0x7f05d80c0330 msgr2=0x7f05d80c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:31.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.139+0000 7f05de2e5640 1 --2- 172.21.15.161:0/699204079 >> v2:172.21.15.170:3300/0 conn(0x7f05d80c0330 0x7f05d80c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:31.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.139+0000 7f05de2e5640 1 -- 172.21.15.161:0/699204079 >> v1:172.21.15.170:6789/0 conn(0x7f05d8152bc0 legacy=0x7f05d80c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:31.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.139+0000 7f05de2e5640 1 -- 172.21.15.161:0/699204079 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05d80c96d0 con 0x7f05d81589b0 2024-03-28T23:32:31.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.139+0000 7f05de2e5640 1 --2- 172.21.15.161:0/699204079 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05d81589b0 0x7f05d8158e00 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f05d80b0c70 tx=0x7f05d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e496b5bd32a83e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:31.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.139+0000 7f05dcae2640 1 -- 172.21.15.161:0/699204079 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d0035020 con 0x7f05d81589b0 2024-03-28T23:32:31.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.139+0000 7f05dcae2640 1 -- 172.21.15.161:0/699204079 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05d0031870 con 0x7f05d81589b0 2024-03-28T23:32:31.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.139+0000 7f05dcae2640 1 -- 172.21.15.161:0/699204079 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d0031c40 con 0x7f05d81589b0 2024-03-28T23:32:31.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.140+0000 7f05dfd48640 1 -- 172.21.15.161:0/699204079 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05d81589b0 msgr2=0x7f05d8158e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:31.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.140+0000 7f05dfd48640 1 --2- 172.21.15.161:0/699204079 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05d81589b0 0x7f05d8158e00 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f05d80b0c70 tx=0x7f05d002b910 comp rx=0 tx=0).stop 2024-03-28T23:32:31.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.140+0000 7f05dfd48640 1 -- 172.21.15.161:0/699204079 shutdown_connections 2024-03-28T23:32:31.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.140+0000 7f05dfd48640 1 --2- 172.21.15.161:0/699204079 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05d81589b0 0x7f05d8158e00 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:31.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.140+0000 7f05dfd48640 1 --2- 172.21.15.161:0/699204079 >> v2:172.21.15.170:3300/0 conn(0x7f05d80c0330 0x7f05d80c9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:31.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.140+0000 7f05dfd48640 1 -- 172.21.15.161:0/699204079 >> 172.21.15.161:0/699204079 conn(0x7f05d80caa80 msgr2=0x7f05d80cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:31.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.140+0000 7f05dfd48640 1 -- 172.21.15.161:0/699204079 shutdown_connections 2024-03-28T23:32:31.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.140+0000 7f05dfd48640 1 -- 172.21.15.161:0/699204079 wait complete. 2024-03-28T23:32:31.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.141+0000 7f05dfd48640 1 Processor -- start 2024-03-28T23:32:31.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.141+0000 7f05dfd48640 1 -- start start 2024-03-28T23:32:31.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.142+0000 7f05dfd48640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05d80c0330 0x7f05d806c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:31.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.142+0000 7f05dfd48640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05d8152bc0 0x7f05d81eaad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:31.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.142+0000 7f05dfd48640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05d81589b0 0x7f05d81eb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:31.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.142+0000 7f05dfd48640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f05d8152fa0 con 0x7f05d8152bc0 2024-03-28T23:32:31.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.142+0000 7f05dfd48640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f05d80c6bf0 con 0x7f05d81589b0 2024-03-28T23:32:31.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.142+0000 7f05dfd48640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f05d81debd0 con 0x7f05d80c0330 2024-03-28T23:32:31.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.142+0000 7f05dd2e3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05d8152bc0 0x7f05d81eaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:31.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.142+0000 7f05dd2e3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05d8152bc0 0x7f05d81eaad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40142/0 (socket says 172.21.15.161:40142) 2024-03-28T23:32:31.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.142+0000 7f05dd2e3640 1 -- 172.21.15.161:0/2321897820 learned_addr learned my addr 172.21.15.161:0/2321897820 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:31.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.142+0000 7f05ddae4640 1 --2- 172.21.15.161:0/2321897820 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05d80c0330 0x7f05d806c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:31.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.142+0000 7f05de2e5640 1 --2- 172.21.15.161:0/2321897820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05d81589b0 0x7f05d81eb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:31.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.142+0000 7f05dd2e3640 1 -- 172.21.15.161:0/2321897820 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05d80c0330 msgr2=0x7f05d806c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:31.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.142+0000 7f05dd2e3640 1 --2- 172.21.15.161:0/2321897820 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05d80c0330 0x7f05d806c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:31.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.142+0000 7f05dd2e3640 1 -- 172.21.15.161:0/2321897820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05d81589b0 msgr2=0x7f05d81eb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:31.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.142+0000 7f05dd2e3640 1 --2- 172.21.15.161:0/2321897820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05d81589b0 0x7f05d81eb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:31.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.142+0000 7f05dd2e3640 1 -- 172.21.15.161:0/2321897820 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05d8157920 con 0x7f05d8152bc0 2024-03-28T23:32:31.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.142+0000 7f05de2e5640 1 --2- 172.21.15.161:0/2321897820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05d81589b0 0x7f05d81eb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:31.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.142+0000 7f05ddae4640 1 --2- 172.21.15.161:0/2321897820 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05d80c0330 0x7f05d806c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:31.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.142+0000 7f05dd2e3640 1 --2- 172.21.15.161:0/2321897820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05d8152bc0 0x7f05d81eaad0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f05c800b4d0 tx=0x7f05c800b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:31.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.143+0000 7f05ce7fc640 1 -- 172.21.15.161:0/2321897820 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c8014070 con 0x7f05d8152bc0 2024-03-28T23:32:31.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.143+0000 7f05ce7fc640 1 -- 172.21.15.161:0/2321897820 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05c8010870 con 0x7f05d8152bc0 2024-03-28T23:32:31.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.143+0000 7f05ce7fc640 1 -- 172.21.15.161:0/2321897820 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c8010c60 con 0x7f05d8152bc0 2024-03-28T23:32:31.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.143+0000 7f05dfd48640 1 -- 172.21.15.161:0/2321897820 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05d806ce70 con 0x7f05d8152bc0 2024-03-28T23:32:31.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.143+0000 7f05dfd48640 1 -- 172.21.15.161:0/2321897820 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05d806c9b0 con 0x7f05d8152bc0 2024-03-28T23:32:31.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.144+0000 7f05dfd48640 1 -- 172.21.15.161:0/2321897820 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05d8068f90 con 0x7f05d8152bc0 2024-03-28T23:32:31.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.144+0000 7f05ce7fc640 1 -- 172.21.15.161:0/2321897820 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f05c8017070 con 0x7f05d8152bc0 2024-03-28T23:32:31.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.144+0000 7f05ce7fc640 1 --2- 172.21.15.161:0/2321897820 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f05bc06b4d0 0x7f05bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:31.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.145+0000 7f05ce7fc640 1 -- 172.21.15.161:0/2321897820 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5619+0+0 (secure 0 0 0) 0x7f05c8089ee0 con 0x7f05d8152bc0 2024-03-28T23:32:31.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.145+0000 7f05ddae4640 1 --2- 172.21.15.161:0/2321897820 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f05bc06b4d0 0x7f05bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:31.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.145+0000 7f05ddae4640 1 --2- 172.21.15.161:0/2321897820 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f05bc06b4d0 0x7f05bc06d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f05d4001e50 tx=0x7f05d4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:31.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.149+0000 7f05ce7fc640 1 -- 172.21.15.161:0/2321897820 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f05d8068f90 con 0x7f05d8152bc0 2024-03-28T23:32:31.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:31.321+0000 7f05dfd48640 1 -- 172.21.15.161:0/2321897820 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f05d804f4b0 con 0x7f05d8152bc0 2024-03-28T23:32:32.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.019+0000 7f05ce7fc640 1 -- 172.21.15.161:0/2321897820 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v71) ==== 170+0+0 (secure 0 0 0) 0x7f05c8058e40 con 0x7f05d8152bc0 2024-03-28T23:32:32.020 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data' created 2024-03-28T23:32:32.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.023+0000 7f05dfd48640 1 -- 172.21.15.161:0/2321897820 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f05bc06b4d0 msgr2=0x7f05bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:32.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.023+0000 7f05dfd48640 1 --2- 172.21.15.161:0/2321897820 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f05bc06b4d0 0x7f05bc06d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f05d4001e50 tx=0x7f05d4009040 comp rx=0 tx=0).stop 2024-03-28T23:32:32.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.023+0000 7f05dfd48640 1 -- 172.21.15.161:0/2321897820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05d8152bc0 msgr2=0x7f05d81eaad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:32.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.023+0000 7f05dfd48640 1 --2- 172.21.15.161:0/2321897820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05d8152bc0 0x7f05d81eaad0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f05c800b4d0 tx=0x7f05c800b990 comp rx=0 tx=0).stop 2024-03-28T23:32:32.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.024+0000 7f05dfd48640 1 -- 172.21.15.161:0/2321897820 shutdown_connections 2024-03-28T23:32:32.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.024+0000 7f05dfd48640 1 --2- 172.21.15.161:0/2321897820 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f05bc06b4d0 0x7f05bc06d990 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:32.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.024+0000 7f05dfd48640 1 --2- 172.21.15.161:0/2321897820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05d8152bc0 0x7f05d81eaad0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:32.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.024+0000 7f05dfd48640 1 --2- 172.21.15.161:0/2321897820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05d81589b0 0x7f05d81eb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:32.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.024+0000 7f05dfd48640 1 --2- 172.21.15.161:0/2321897820 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05d80c0330 0x7f05d806c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:32.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.024+0000 7f05dfd48640 1 -- 172.21.15.161:0/2321897820 >> 172.21.15.161:0/2321897820 conn(0x7f05d80caa80 msgr2=0x7f05d80c5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:32.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.025+0000 7f05dfd48640 1 -- 172.21.15.161:0/2321897820 shutdown_connections 2024-03-28T23:32:32.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.025+0000 7f05dfd48640 1 -- 172.21.15.161:0/2321897820 wait complete. 2024-03-28T23:32:32.048 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-28T23:32:32.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.153+0000 7fc7d1cc0640 1 Processor -- start 2024-03-28T23:32:32.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.153+0000 7fc7d1cc0640 1 -- start start 2024-03-28T23:32:32.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.153+0000 7fc7d1cc0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc7cc152bc0 0x7fc7cc0c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:32.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.153+0000 7fc7d1cc0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc7cc1589b0 0x7fc7cc158e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:32.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.153+0000 7fc7d1cc0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7cc0c6f60 con 0x7fc7cc0c8a80 2024-03-28T23:32:32.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.153+0000 7fc7d1cc0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc7cc0c6120 con 0x7fc7cc1589b0 2024-03-28T23:32:32.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.153+0000 7fc7d1cc0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fc7cc0c6400 con 0x7fc7cc152bc0 2024-03-28T23:32:32.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.154+0000 7fc7cbfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc7cc1589b0 0x7fc7cc158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:32.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.154+0000 7fc7cbfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc7cc1589b0 0x7fc7cc158e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40144/0 (socket says 172.21.15.161:40144) 2024-03-28T23:32:32.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.154+0000 7fc7cbfff640 1 -- 172.21.15.161:0/708283347 learned_addr learned my addr 172.21.15.161:0/708283347 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:32.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.154+0000 7fc7cb7fe640 1 --2- 172.21.15.161:0/708283347 >> v2:172.21.15.170:3300/0 conn(0x7fc7cc152bc0 0x7fc7cc0c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:32.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.154+0000 7fc7cbfff640 1 -- 172.21.15.161:0/708283347 >> v2:172.21.15.170:3300/0 conn(0x7fc7cc152bc0 msgr2=0x7fc7cc0c0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:32.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.154+0000 7fc7cbfff640 1 --2- 172.21.15.161:0/708283347 >> v2:172.21.15.170:3300/0 conn(0x7fc7cc152bc0 0x7fc7cc0c0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:32.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.154+0000 7fc7cbfff640 1 -- 172.21.15.161:0/708283347 >> v1:172.21.15.170:6789/0 conn(0x7fc7cc0c8a80 legacy=0x7fc7cc0c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:32.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.154+0000 7fc7cbfff640 1 -- 172.21.15.161:0/708283347 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7cc0c7240 con 0x7fc7cc1589b0 2024-03-28T23:32:32.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.155+0000 7fc7cbfff640 1 --2- 172.21.15.161:0/708283347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7cc1589b0 0x7fc7cc158e00 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc0aa410 tx=0x7fc7c402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eecb7ded28467496 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:32.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.155+0000 7fc7ca7fc640 1 -- 172.21.15.161:0/708283347 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c402ca80 con 0x7fc7cc1589b0 2024-03-28T23:32:32.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.155+0000 7fc7ca7fc640 1 -- 172.21.15.161:0/708283347 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7c402cc20 con 0x7fc7cc1589b0 2024-03-28T23:32:32.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.155+0000 7fc7ca7fc640 1 -- 172.21.15.161:0/708283347 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c40343f0 con 0x7fc7cc1589b0 2024-03-28T23:32:32.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.155+0000 7fc7d1cc0640 1 -- 172.21.15.161:0/708283347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7cc1589b0 msgr2=0x7fc7cc158e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:32.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.155+0000 7fc7d1cc0640 1 --2- 172.21.15.161:0/708283347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7cc1589b0 0x7fc7cc158e00 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc0aa410 tx=0x7fc7c402bee0 comp rx=0 tx=0).stop 2024-03-28T23:32:32.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.155+0000 7fc7d1cc0640 1 -- 172.21.15.161:0/708283347 shutdown_connections 2024-03-28T23:32:32.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.155+0000 7fc7d1cc0640 1 --2- 172.21.15.161:0/708283347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7cc1589b0 0x7fc7cc158e00 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:32.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.155+0000 7fc7d1cc0640 1 --2- 172.21.15.161:0/708283347 >> v2:172.21.15.170:3300/0 conn(0x7fc7cc152bc0 0x7fc7cc0c0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:32.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.155+0000 7fc7d1cc0640 1 -- 172.21.15.161:0/708283347 >> 172.21.15.161:0/708283347 conn(0x7fc7cc0caa80 msgr2=0x7fc7cc0cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:32.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.155+0000 7fc7d1cc0640 1 -- 172.21.15.161:0/708283347 shutdown_connections 2024-03-28T23:32:32.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.155+0000 7fc7d1cc0640 1 -- 172.21.15.161:0/708283347 wait complete. 2024-03-28T23:32:32.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.156+0000 7fc7d1cc0640 1 Processor -- start 2024-03-28T23:32:32.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.157+0000 7fc7d1cc0640 1 -- start start 2024-03-28T23:32:32.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.157+0000 7fc7d1cc0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7cc0c8a80 0x7fc7cc0c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:32.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.157+0000 7fc7d1cc0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7cc152bc0 0x7fc7cc1571c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:32.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.157+0000 7fc7d1cc0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7cc1589b0 0x7fc7cc141440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:32.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.157+0000 7fc7d1cc0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc7cc152fa0 con 0x7fc7cc152bc0 2024-03-28T23:32:32.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.157+0000 7fc7d1cc0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc7cc0c95f0 con 0x7fc7cc0c8a80 2024-03-28T23:32:32.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.157+0000 7fc7d1cc0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc7cc14e840 con 0x7fc7cc1589b0 2024-03-28T23:32:32.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.157+0000 7fc7caffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7cc152bc0 0x7fc7cc1571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:32.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.157+0000 7fc7caffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7cc152bc0 0x7fc7cc1571c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40152/0 (socket says 172.21.15.161:40152) 2024-03-28T23:32:32.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.157+0000 7fc7caffd640 1 -- 172.21.15.161:0/2226590526 learned_addr learned my addr 172.21.15.161:0/2226590526 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:32.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.157+0000 7fc7cb7fe640 1 --2- 172.21.15.161:0/2226590526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7cc0c8a80 0x7fc7cc0c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:32.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.157+0000 7fc7cbfff640 1 --2- 172.21.15.161:0/2226590526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7cc1589b0 0x7fc7cc141440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:32.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.158+0000 7fc7caffd640 1 -- 172.21.15.161:0/2226590526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7cc1589b0 msgr2=0x7fc7cc141440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:32.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.158+0000 7fc7caffd640 1 --2- 172.21.15.161:0/2226590526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7cc1589b0 0x7fc7cc141440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:32.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.158+0000 7fc7caffd640 1 -- 172.21.15.161:0/2226590526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7cc0c8a80 msgr2=0x7fc7cc0c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:32.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.158+0000 7fc7caffd640 1 --2- 172.21.15.161:0/2226590526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7cc0c8a80 0x7fc7cc0c7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:32.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.158+0000 7fc7caffd640 1 -- 172.21.15.161:0/2226590526 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7cc157c40 con 0x7fc7cc152bc0 2024-03-28T23:32:32.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.158+0000 7fc7cbfff640 1 --2- 172.21.15.161:0/2226590526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7cc1589b0 0x7fc7cc141440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:32.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.158+0000 7fc7caffd640 1 --2- 172.21.15.161:0/2226590526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7cc152bc0 0x7fc7cc1571c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fc7c000b3f0 tx=0x7fc7c000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:32.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.158+0000 7fc7b3fff640 1 -- 172.21.15.161:0/2226590526 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c0015020 con 0x7fc7cc152bc0 2024-03-28T23:32:32.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.158+0000 7fc7b3fff640 1 -- 172.21.15.161:0/2226590526 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7c0011830 con 0x7fc7cc152bc0 2024-03-28T23:32:32.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.158+0000 7fc7d1cc0640 1 -- 172.21.15.161:0/2226590526 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7cc157ee0 con 0x7fc7cc152bc0 2024-03-28T23:32:32.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.158+0000 7fc7b3fff640 1 -- 172.21.15.161:0/2226590526 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c0011c20 con 0x7fc7cc152bc0 2024-03-28T23:32:32.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.158+0000 7fc7d1cc0640 1 -- 172.21.15.161:0/2226590526 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7cc1584a0 con 0x7fc7cc152bc0 2024-03-28T23:32:32.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.159+0000 7fc7d1cc0640 1 -- 172.21.15.161:0/2226590526 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7cc14dd90 con 0x7fc7cc152bc0 2024-03-28T23:32:32.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.160+0000 7fc7b3fff640 1 -- 172.21.15.161:0/2226590526 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc7c0018020 con 0x7fc7cc152bc0 2024-03-28T23:32:32.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.160+0000 7fc7b3fff640 1 --2- 172.21.15.161:0/2226590526 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc7b806b4d0 0x7fc7b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:32.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.160+0000 7fc7b3fff640 1 -- 172.21.15.161:0/2226590526 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6001+0+0 (secure 0 0 0) 0x7fc7c0089db0 con 0x7fc7cc152bc0 2024-03-28T23:32:32.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.160+0000 7fc7cb7fe640 1 --2- 172.21.15.161:0/2226590526 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc7b806b4d0 0x7fc7b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:32.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.160+0000 7fc7cb7fe640 1 --2- 172.21.15.161:0/2226590526 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc7b806b4d0 0x7fc7b806d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc006fd0 tx=0x7fc7bc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:32.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.165+0000 7fc7b3fff640 1 -- 172.21.15.161:0/2226590526 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc7cc14dd90 con 0x7fc7cc152bc0 2024-03-28T23:32:32.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:32.371+0000 7fc7d1cc0640 1 -- 172.21.15.161:0/2226590526 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fc7cc04f4b0 con 0x7fc7cc152bc0 2024-03-28T23:32:33.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.021+0000 7fc7b3fff640 1 -- 172.21.15.161:0/2226590526 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 11 and data pool 12 v53) ==== 174+0+0 (secure 0 0 0) 0x7fc7c001f030 con 0x7fc7cc152bc0 2024-03-28T23:32:33.022 INFO:teuthology.orchestra.run.smithi161.stderr:new fs with metadata pool 11 and data pool 12 2024-03-28T23:32:33.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.026+0000 7fc7d1cc0640 1 -- 172.21.15.161:0/2226590526 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc7b806b4d0 msgr2=0x7fc7b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:33.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.026+0000 7fc7d1cc0640 1 --2- 172.21.15.161:0/2226590526 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc7b806b4d0 0x7fc7b806d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc006fd0 tx=0x7fc7bc008040 comp rx=0 tx=0).stop 2024-03-28T23:32:33.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.026+0000 7fc7d1cc0640 1 -- 172.21.15.161:0/2226590526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7cc152bc0 msgr2=0x7fc7cc1571c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:33.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.026+0000 7fc7d1cc0640 1 --2- 172.21.15.161:0/2226590526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7cc152bc0 0x7fc7cc1571c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fc7c000b3f0 tx=0x7fc7c000b8b0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.028+0000 7fc7d1cc0640 1 -- 172.21.15.161:0/2226590526 shutdown_connections 2024-03-28T23:32:33.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.028+0000 7fc7d1cc0640 1 --2- 172.21.15.161:0/2226590526 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc7b806b4d0 0x7fc7b806d990 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.028+0000 7fc7d1cc0640 1 --2- 172.21.15.161:0/2226590526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7cc152bc0 0x7fc7cc1571c0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.028+0000 7fc7d1cc0640 1 --2- 172.21.15.161:0/2226590526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7cc1589b0 0x7fc7cc141440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.028+0000 7fc7d1cc0640 1 --2- 172.21.15.161:0/2226590526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7cc0c8a80 0x7fc7cc0c7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.028+0000 7fc7d1cc0640 1 -- 172.21.15.161:0/2226590526 >> 172.21.15.161:0/2226590526 conn(0x7fc7cc0caa80 msgr2=0x7fc7cc14baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:33.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.032+0000 7fc7d1cc0640 1 -- 172.21.15.161:0/2226590526 shutdown_connections 2024-03-28T23:32:33.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.033+0000 7fc7d1cc0640 1 -- 172.21.15.161:0/2226590526 wait complete. 2024-03-28T23:32:33.062 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:32:33.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.165+0000 7f7075c55640 1 Processor -- start 2024-03-28T23:32:33.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.165+0000 7f7075c55640 1 -- start start 2024-03-28T23:32:33.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.166+0000 7f7075c55640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7070152890 0x7f70700c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:33.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.166+0000 7f7075c55640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f70700c5b20 0x7f70701553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:33.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.166+0000 7f7075c55640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7070158a30 con 0x7f70700c8ed0 2024-03-28T23:32:33.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.166+0000 7f7075c55640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f70700c8480 con 0x7f70700c5b20 2024-03-28T23:32:33.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.166+0000 7f7075c55640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7070159640 con 0x7f7070152890 2024-03-28T23:32:33.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.166+0000 7f706ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f70700c5b20 0x7f70701553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:33.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.166+0000 7f706ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f70700c5b20 0x7f70701553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40162/0 (socket says 172.21.15.161:40162) 2024-03-28T23:32:33.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.166+0000 7f706ffff640 1 -- 172.21.15.161:0/2026937387 learned_addr learned my addr 172.21.15.161:0/2026937387 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:33.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.166+0000 7f706effd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f70700c8ed0 0x7f70700c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:58436/0 (socket says 172.21.15.161:58436) 2024-03-28T23:32:33.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.166+0000 7f706f7fe640 1 --2- 172.21.15.161:0/2026937387 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7070152890 0x7f70700c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:33.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.167+0000 7f706ffff640 1 -- 172.21.15.161:0/2026937387 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7070152890 msgr2=0x7f70700c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:33.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.167+0000 7f706ffff640 1 --2- 172.21.15.161:0/2026937387 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7070152890 0x7f70700c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.167+0000 7f706ffff640 1 -- 172.21.15.161:0/2026937387 >> v1:172.21.15.161:6789/0 conn(0x7f70700c8ed0 legacy=0x7f70700c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:33.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.167+0000 7f706e7fc640 1 -- 172.21.15.161:0/2026937387 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 925750874 0 0) 0x7f7070158a30 con 0x7f70700c8ed0 2024-03-28T23:32:33.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.167+0000 7f706ffff640 1 -- 172.21.15.161:0/2026937387 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7070158d10 con 0x7f70700c5b20 2024-03-28T23:32:33.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.167+0000 7f706ffff640 1 --2- 172.21.15.161:0/2026937387 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70700c5b20 0x7f70701553a0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f70701597c0 tx=0x7f706802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5db0cf080f0e98a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:33.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.168+0000 7f706e7fc640 1 -- 172.21.15.161:0/2026937387 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706802c9c0 con 0x7f70700c5b20 2024-03-28T23:32:33.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.168+0000 7f706e7fc640 1 -- 172.21.15.161:0/2026937387 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f706802cb60 con 0x7f70700c5b20 2024-03-28T23:32:33.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.168+0000 7f706e7fc640 1 -- 172.21.15.161:0/2026937387 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70680343f0 con 0x7f70700c5b20 2024-03-28T23:32:33.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.168+0000 7f7075c55640 1 -- 172.21.15.161:0/2026937387 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70700c5b20 msgr2=0x7f70701553a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:33.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.168+0000 7f7075c55640 1 --2- 172.21.15.161:0/2026937387 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70700c5b20 0x7f70701553a0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f70701597c0 tx=0x7f706802be20 comp rx=0 tx=0).stop 2024-03-28T23:32:33.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.168+0000 7f7075c55640 1 -- 172.21.15.161:0/2026937387 shutdown_connections 2024-03-28T23:32:33.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.168+0000 7f7075c55640 1 --2- 172.21.15.161:0/2026937387 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70700c5b20 0x7f70701553a0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.168+0000 7f7075c55640 1 --2- 172.21.15.161:0/2026937387 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7070152890 0x7f70700c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.168+0000 7f7075c55640 1 -- 172.21.15.161:0/2026937387 >> 172.21.15.161:0/2026937387 conn(0x7f70700ac620 msgr2=0x7f70700aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:33.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.168+0000 7f7075c55640 1 -- 172.21.15.161:0/2026937387 shutdown_connections 2024-03-28T23:32:33.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.168+0000 7f7075c55640 1 -- 172.21.15.161:0/2026937387 wait complete. 2024-03-28T23:32:33.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.169+0000 7f7075c55640 1 Processor -- start 2024-03-28T23:32:33.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.169+0000 7f7075c55640 1 -- start start 2024-03-28T23:32:33.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.170+0000 7f7075c55640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70700c5b20 0x7f707014e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:33.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.170+0000 7f7075c55640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70700c8ed0 0x7f7070146df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:33.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.170+0000 7f7075c55640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7070152890 0x7f70700c1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:33.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.170+0000 7f7075c55640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7070152c70 con 0x7f70700c8ed0 2024-03-28T23:32:33.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.170+0000 7f7075c55640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f70701592d0 con 0x7f7070152890 2024-03-28T23:32:33.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.170+0000 7f7075c55640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7070152df0 con 0x7f70700c5b20 2024-03-28T23:32:33.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.170+0000 7f706effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70700c8ed0 0x7f7070146df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:33.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.170+0000 7f706effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70700c8ed0 0x7f7070146df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40170/0 (socket says 172.21.15.161:40170) 2024-03-28T23:32:33.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.170+0000 7f706effd640 1 -- 172.21.15.161:0/2231694912 learned_addr learned my addr 172.21.15.161:0/2231694912 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:33.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.170+0000 7f706f7fe640 1 --2- 172.21.15.161:0/2231694912 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70700c5b20 0x7f707014e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:33.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.170+0000 7f706ffff640 1 --2- 172.21.15.161:0/2231694912 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7070152890 0x7f70700c1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:33.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.170+0000 7f706effd640 1 -- 172.21.15.161:0/2231694912 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70700c5b20 msgr2=0x7f707014e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:33.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.170+0000 7f706effd640 1 --2- 172.21.15.161:0/2231694912 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70700c5b20 0x7f707014e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.170+0000 7f706effd640 1 -- 172.21.15.161:0/2231694912 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7070152890 msgr2=0x7f70700c1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:33.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.170+0000 7f706effd640 1 --2- 172.21.15.161:0/2231694912 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7070152890 0x7f70700c1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.171+0000 7f706effd640 1 -- 172.21.15.161:0/2231694912 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f707014f8b0 con 0x7f70700c8ed0 2024-03-28T23:32:33.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.171+0000 7f706effd640 1 --2- 172.21.15.161:0/2231694912 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70700c8ed0 0x7f7070146df0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f706400c430 tx=0x7f706400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:33.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.171+0000 7f7057fff640 1 -- 172.21.15.161:0/2231694912 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7064017070 con 0x7f70700c8ed0 2024-03-28T23:32:33.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.171+0000 7f7057fff640 1 -- 172.21.15.161:0/2231694912 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70640074e0 con 0x7f70700c8ed0 2024-03-28T23:32:33.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.171+0000 7f7057fff640 1 -- 172.21.15.161:0/2231694912 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70640059e0 con 0x7f70700c8ed0 2024-03-28T23:32:33.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.171+0000 7f7075c55640 1 -- 172.21.15.161:0/2231694912 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f707014e9f0 con 0x7f70700c8ed0 2024-03-28T23:32:33.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.171+0000 7f7075c55640 1 -- 172.21.15.161:0/2231694912 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70700c2890 con 0x7f70700c8ed0 2024-03-28T23:32:33.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.172+0000 7f7075c55640 1 -- 172.21.15.161:0/2231694912 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f707004f4b0 con 0x7f70700c8ed0 2024-03-28T23:32:33.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.172+0000 7f7057fff640 1 -- 172.21.15.161:0/2231694912 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f706401a070 con 0x7f70700c8ed0 2024-03-28T23:32:33.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.173+0000 7f7057fff640 1 --2- 172.21.15.161:0/2231694912 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f705c06b4d0 0x7f705c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:33.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.173+0000 7f7057fff640 1 -- 172.21.15.161:0/2231694912 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 6101+0+0 (secure 0 0 0) 0x7f7064089640 con 0x7f70700c8ed0 2024-03-28T23:32:33.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.173+0000 7f706f7fe640 1 --2- 172.21.15.161:0/2231694912 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f705c06b4d0 0x7f705c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:33.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.173+0000 7f706f7fe640 1 --2- 172.21.15.161:0/2231694912 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f705c06b4d0 0x7f705c06d990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7060006fd0 tx=0x7f7060008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:33.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.177+0000 7f7057fff640 1 -- 172.21.15.161:0/2231694912 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7064057d70 con 0x7f70700c8ed0 2024-03-28T23:32:33.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.350+0000 7f7075c55640 1 -- 172.21.15.161:0/2231694912 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7070150250 con 0x7f70700c8ed0 2024-03-28T23:32:33.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.351+0000 7f7057fff640 1 -- 172.21.15.161:0/2231694912 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+16994 (secure 0 0 0) 0x7f706405e0c0 con 0x7f70700c8ed0 2024-03-28T23:32:33.352 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:33.352 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":72,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:32:32.372621+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:32:30.684936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-28T23:32:31.339722+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:32:33.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.353+0000 7f7075c55640 1 -- 172.21.15.161:0/2231694912 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f705c06b4d0 msgr2=0x7f705c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:33.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.353+0000 7f7075c55640 1 --2- 172.21.15.161:0/2231694912 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f705c06b4d0 0x7f705c06d990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7060006fd0 tx=0x7f7060008040 comp rx=0 tx=0).stop 2024-03-28T23:32:33.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.353+0000 7f7075c55640 1 -- 172.21.15.161:0/2231694912 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70700c8ed0 msgr2=0x7f7070146df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:33.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.353+0000 7f7075c55640 1 --2- 172.21.15.161:0/2231694912 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70700c8ed0 0x7f7070146df0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f706400c430 tx=0x7f706400c8f0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.353+0000 7f7075c55640 1 -- 172.21.15.161:0/2231694912 shutdown_connections 2024-03-28T23:32:33.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.353+0000 7f7075c55640 1 --2- 172.21.15.161:0/2231694912 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f705c06b4d0 0x7f705c06d990 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.353+0000 7f7075c55640 1 --2- 172.21.15.161:0/2231694912 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70700c8ed0 0x7f7070146df0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.353+0000 7f7075c55640 1 --2- 172.21.15.161:0/2231694912 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7070152890 0x7f70700c1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.353+0000 7f7075c55640 1 --2- 172.21.15.161:0/2231694912 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70700c5b20 0x7f707014e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.353+0000 7f7075c55640 1 -- 172.21.15.161:0/2231694912 >> 172.21.15.161:0/2231694912 conn(0x7f70700ac620 msgr2=0x7f707014b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:33.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.353+0000 7f7075c55640 1 -- 172.21.15.161:0/2231694912 shutdown_connections 2024-03-28T23:32:33.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.353+0000 7f7075c55640 1 -- 172.21.15.161:0/2231694912 wait complete. 2024-03-28T23:32:33.365 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:32:33.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.464+0000 7f4e30f65640 1 Processor -- start 2024-03-28T23:32:33.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.464+0000 7f4e30f65640 1 -- start start 2024-03-28T23:32:33.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.465+0000 7f4e30f65640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4e2c152890 0x7f4e2c0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:33.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.465+0000 7f4e30f65640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4e2c0c8ed0 0x7f4e2c0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:33.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.465+0000 7f4e30f65640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e2c0c5950 0x7f4e2c1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:33.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.465+0000 7f4e30f65640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f4e2c0c8480 con 0x7f4e2c0c8ed0 2024-03-28T23:32:33.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.465+0000 7f4e30f65640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f4e2c159640 con 0x7f4e2c152890 2024-03-28T23:32:33.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.465+0000 7f4e30f65640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4e2c158a30 con 0x7f4e2c0c5950 2024-03-28T23:32:33.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.465+0000 7f4e29d9b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4e2c0c8ed0 0x7f4e2c0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:33.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.465+0000 7f4e29d9b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4e2c0c8ed0 0x7f4e2c0c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40184/0 (socket says 172.21.15.161:40184) 2024-03-28T23:32:33.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.465+0000 7f4e29d9b640 1 -- 172.21.15.161:0/3117463897 learned_addr learned my addr 172.21.15.161:0/3117463897 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:33.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.466+0000 7f4e2ad9d640 1 --2- 172.21.15.161:0/3117463897 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e2c0c5950 0x7f4e2c1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:33.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.466+0000 7f4e2a59c640 1 --2- 172.21.15.161:0/3117463897 >> v2:172.21.15.170:3300/0 conn(0x7f4e2c152890 0x7f4e2c0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:33.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.466+0000 7f4e29d9b640 1 -- 172.21.15.161:0/3117463897 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e2c0c5950 msgr2=0x7f4e2c1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:33.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.466+0000 7f4e29d9b640 1 --2- 172.21.15.161:0/3117463897 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e2c0c5950 0x7f4e2c1553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.466+0000 7f4e29d9b640 1 -- 172.21.15.161:0/3117463897 >> v2:172.21.15.170:3300/0 conn(0x7f4e2c152890 msgr2=0x7f4e2c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:33.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.466+0000 7f4e29d9b640 1 --2- 172.21.15.161:0/3117463897 >> v2:172.21.15.170:3300/0 conn(0x7f4e2c152890 0x7f4e2c0c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.466+0000 7f4e29d9b640 1 -- 172.21.15.161:0/3117463897 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e2c158d10 con 0x7f4e2c0c8ed0 2024-03-28T23:32:33.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.467+0000 7f4e29d9b640 1 --2- 172.21.15.161:0/3117463897 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e2c0c8ed0 0x7f4e2c0c5410 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c08cc80 tx=0x7f4e2002b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9315dce0b45b5e75 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:33.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.467+0000 7f4e2959a640 1 -- 172.21.15.161:0/3117463897 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e20038070 con 0x7f4e2c0c8ed0 2024-03-28T23:32:33.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.467+0000 7f4e2959a640 1 -- 172.21.15.161:0/3117463897 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e20004030 con 0x7f4e2c0c8ed0 2024-03-28T23:32:33.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.467+0000 7f4e2959a640 1 -- 172.21.15.161:0/3117463897 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e20004460 con 0x7f4e2c0c8ed0 2024-03-28T23:32:33.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.467+0000 7f4e30f65640 1 -- 172.21.15.161:0/3117463897 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e2c0c8ed0 msgr2=0x7f4e2c0c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:33.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.467+0000 7f4e30f65640 1 --2- 172.21.15.161:0/3117463897 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e2c0c8ed0 0x7f4e2c0c5410 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c08cc80 tx=0x7f4e2002b7c0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.467+0000 7f4e30f65640 1 -- 172.21.15.161:0/3117463897 shutdown_connections 2024-03-28T23:32:33.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.467+0000 7f4e30f65640 1 --2- 172.21.15.161:0/3117463897 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e2c0c5950 0x7f4e2c1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.467+0000 7f4e30f65640 1 --2- 172.21.15.161:0/3117463897 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e2c0c8ed0 0x7f4e2c0c5410 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.467+0000 7f4e30f65640 1 --2- 172.21.15.161:0/3117463897 >> v2:172.21.15.170:3300/0 conn(0x7f4e2c152890 0x7f4e2c0c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.467+0000 7f4e30f65640 1 -- 172.21.15.161:0/3117463897 >> 172.21.15.161:0/3117463897 conn(0x7f4e2c0ac620 msgr2=0x7f4e2c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:33.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.467+0000 7f4e30f65640 1 -- 172.21.15.161:0/3117463897 shutdown_connections 2024-03-28T23:32:33.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.467+0000 7f4e30f65640 1 -- 172.21.15.161:0/3117463897 wait complete. 2024-03-28T23:32:33.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.469+0000 7f4e30f65640 1 Processor -- start 2024-03-28T23:32:33.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.469+0000 7f4e30f65640 1 -- start start 2024-03-28T23:32:33.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.469+0000 7f4e30f65640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e2c0c5950 0x7f4e2c155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:33.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.469+0000 7f4e30f65640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e2c0c8ed0 0x7f4e2c14eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:33.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.469+0000 7f4e30f65640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e2c152890 0x7f4e2c1472e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:33.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.469+0000 7f4e30f65640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2c152c70 con 0x7f4e2c0c5950 2024-03-28T23:32:33.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.469+0000 7f4e30f65640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2c1592d0 con 0x7f4e2c0c8ed0 2024-03-28T23:32:33.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.469+0000 7f4e30f65640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4e2c152df0 con 0x7f4e2c152890 2024-03-28T23:32:33.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.469+0000 7f4e2a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e2c0c5950 0x7f4e2c155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:33.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.469+0000 7f4e2a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e2c0c5950 0x7f4e2c155aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40186/0 (socket says 172.21.15.161:40186) 2024-03-28T23:32:33.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.469+0000 7f4e2a59c640 1 -- 172.21.15.161:0/3796784713 learned_addr learned my addr 172.21.15.161:0/3796784713 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:33.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.470+0000 7f4e2ad9d640 1 --2- 172.21.15.161:0/3796784713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e2c152890 0x7f4e2c1472e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:33.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.470+0000 7f4e29d9b640 1 --2- 172.21.15.161:0/3796784713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e2c0c8ed0 0x7f4e2c14eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:33.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.470+0000 7f4e2a59c640 1 -- 172.21.15.161:0/3796784713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e2c152890 msgr2=0x7f4e2c1472e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:33.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.470+0000 7f4e2a59c640 1 --2- 172.21.15.161:0/3796784713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e2c152890 0x7f4e2c1472e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.470+0000 7f4e2a59c640 1 -- 172.21.15.161:0/3796784713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e2c0c8ed0 msgr2=0x7f4e2c14eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:33.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.470+0000 7f4e2a59c640 1 --2- 172.21.15.161:0/3796784713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e2c0c8ed0 0x7f4e2c14eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.470+0000 7f4e2a59c640 1 -- 172.21.15.161:0/3796784713 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e2c14f750 con 0x7f4e2c0c5950 2024-03-28T23:32:33.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.470+0000 7f4e2ad9d640 1 --2- 172.21.15.161:0/3796784713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e2c152890 0x7f4e2c1472e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:33.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.470+0000 7f4e29d9b640 1 --2- 172.21.15.161:0/3796784713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e2c0c8ed0 0x7f4e2c14eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:33.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.470+0000 7f4e2a59c640 1 --2- 172.21.15.161:0/3796784713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e2c0c5950 0x7f4e2c155aa0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c00efc0 tx=0x7f4e1c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:33.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.470+0000 7f4e1affd640 1 -- 172.21.15.161:0/3796784713 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c018020 con 0x7f4e2c0c5950 2024-03-28T23:32:33.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.470+0000 7f4e1affd640 1 -- 172.21.15.161:0/3796784713 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e1c014480 con 0x7f4e2c0c5950 2024-03-28T23:32:33.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.470+0000 7f4e1affd640 1 -- 172.21.15.161:0/3796784713 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c014850 con 0x7f4e2c0c5950 2024-03-28T23:32:33.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.470+0000 7f4e30f65640 1 -- 172.21.15.161:0/3796784713 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e2c14fa30 con 0x7f4e2c0c5950 2024-03-28T23:32:33.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.470+0000 7f4e30f65640 1 -- 172.21.15.161:0/3796784713 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e2c14cfb0 con 0x7f4e2c0c5950 2024-03-28T23:32:33.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.471+0000 7f4e30f65640 1 -- 172.21.15.161:0/3796784713 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e2c04f4b0 con 0x7f4e2c0c5950 2024-03-28T23:32:33.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.472+0000 7f4e1affd640 1 -- 172.21.15.161:0/3796784713 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4e1c005d10 con 0x7f4e2c0c5950 2024-03-28T23:32:33.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.472+0000 7f4e1affd640 1 --2- 172.21.15.161:0/3796784713 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4e0c06b4d0 0x7f4e0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:33.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.472+0000 7f4e1affd640 1 -- 172.21.15.161:0/3796784713 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 6101+0+0 (secure 0 0 0) 0x7f4e1c08af40 con 0x7f4e2c0c5950 2024-03-28T23:32:33.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.472+0000 7f4e29d9b640 1 --2- 172.21.15.161:0/3796784713 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4e0c06b4d0 0x7f4e0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:33.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.472+0000 7f4e29d9b640 1 --2- 172.21.15.161:0/3796784713 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4e0c06b4d0 0x7f4e0c06d990 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4e20035000 tx=0x7f4e20030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:33.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.476+0000 7f4e1affd640 1 -- 172.21.15.161:0/3796784713 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4e1c022070 con 0x7f4e2c0c5950 2024-03-28T23:32:33.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.647+0000 7f4e30f65640 1 -- 172.21.15.161:0/3796784713 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4e2c06d250 con 0x7f4e2c0c5950 2024-03-28T23:32:33.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.648+0000 7f4e1affd640 1 -- 172.21.15.161:0/3796784713 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+16994 (secure 0 0 0) 0x7f4e1c010040 con 0x7f4e2c0c5950 2024-03-28T23:32:33.650 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:33.650 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":72,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:32:32.372621+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:32:30.684936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-28T23:32:31.339722+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:32:33.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.650+0000 7f4e30f65640 1 -- 172.21.15.161:0/3796784713 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4e0c06b4d0 msgr2=0x7f4e0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:33.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.650+0000 7f4e30f65640 1 --2- 172.21.15.161:0/3796784713 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4e0c06b4d0 0x7f4e0c06d990 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4e20035000 tx=0x7f4e20030000 comp rx=0 tx=0).stop 2024-03-28T23:32:33.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.650+0000 7f4e30f65640 1 -- 172.21.15.161:0/3796784713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e2c0c5950 msgr2=0x7f4e2c155aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:33.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.650+0000 7f4e30f65640 1 --2- 172.21.15.161:0/3796784713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e2c0c5950 0x7f4e2c155aa0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c00efc0 tx=0x7f4e1c00c490 comp rx=0 tx=0).stop 2024-03-28T23:32:33.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.650+0000 7f4e30f65640 1 -- 172.21.15.161:0/3796784713 shutdown_connections 2024-03-28T23:32:33.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.650+0000 7f4e30f65640 1 --2- 172.21.15.161:0/3796784713 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4e0c06b4d0 0x7f4e0c06d990 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.650+0000 7f4e30f65640 1 --2- 172.21.15.161:0/3796784713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e2c152890 0x7f4e2c1472e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.650+0000 7f4e30f65640 1 --2- 172.21.15.161:0/3796784713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e2c0c8ed0 0x7f4e2c14eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.650+0000 7f4e30f65640 1 --2- 172.21.15.161:0/3796784713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e2c0c5950 0x7f4e2c155aa0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.650+0000 7f4e30f65640 1 -- 172.21.15.161:0/3796784713 >> 172.21.15.161:0/3796784713 conn(0x7f4e2c0ac620 msgr2=0x7f4e2c150b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:33.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.650+0000 7f4e30f65640 1 -- 172.21.15.161:0/3796784713 shutdown_connections 2024-03-28T23:32:33.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.650+0000 7f4e30f65640 1 -- 172.21.15.161:0/3796784713 wait complete. 2024-03-28T23:32:33.663 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-28T23:32:33.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.765+0000 7f8b59eed640 1 Processor -- start 2024-03-28T23:32:33.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.765+0000 7f8b59eed640 1 -- start start 2024-03-28T23:32:33.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.765+0000 7f8b59eed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b5414f5c0 0x7f8b540c2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:33.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.765+0000 7f8b59eed640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8b5406cdd0 0x7f8b541524c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:33.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.765+0000 7f8b59eed640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b540c2f10 con 0x7f8b540c2690 2024-03-28T23:32:33.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.765+0000 7f8b59eed640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f8b540c74d0 con 0x7f8b5406cdd0 2024-03-28T23:32:33.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.765+0000 7f8b59eed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8b540c6fa0 con 0x7f8b5414f5c0 2024-03-28T23:32:33.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.766+0000 7f8b537fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b5414f5c0 0x7f8b540c2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:33.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.766+0000 7f8b52ffd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8b5406cdd0 0x7f8b541524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:33.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.766+0000 7f8b52ffd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8b5406cdd0 0x7f8b541524c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:35004/0 (socket says 172.21.15.161:35004) 2024-03-28T23:32:33.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.766+0000 7f8b52ffd640 1 -- 172.21.15.161:0/1464955484 learned_addr learned my addr 172.21.15.161:0/1464955484 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:33.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.766+0000 7f8b52ffd640 1 -- 172.21.15.161:0/1464955484 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b5414f5c0 msgr2=0x7f8b540c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:33.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.766+0000 7f8b52ffd640 1 --2- 172.21.15.161:0/1464955484 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b5414f5c0 0x7f8b540c2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.766+0000 7f8b52ffd640 1 -- 172.21.15.161:0/1464955484 >> v1:172.21.15.170:6789/0 conn(0x7f8b540c2690 legacy=0x7f8b541554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:33.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.766+0000 7f8b52ffd640 1 -- 172.21.15.161:0/1464955484 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b5414f9a0 con 0x7f8b5406cdd0 2024-03-28T23:32:33.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.766+0000 7f8b537fe640 1 --2- 172.21.15.161:0/1464955484 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b5414f5c0 0x7f8b540c2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:32:33.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.767+0000 7f8b52ffd640 1 --2- 172.21.15.161:0/1464955484 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b5406cdd0 0x7f8b541524c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8b48009870 tx=0x7f8b4802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2396fc3e8ce5ea9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:33.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.767+0000 7f8b527fc640 1 -- 172.21.15.161:0/1464955484 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b48034070 con 0x7f8b5406cdd0 2024-03-28T23:32:33.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.767+0000 7f8b527fc640 1 -- 172.21.15.161:0/1464955484 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b48030840 con 0x7f8b5406cdd0 2024-03-28T23:32:33.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.767+0000 7f8b527fc640 1 -- 172.21.15.161:0/1464955484 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b48030c10 con 0x7f8b5406cdd0 2024-03-28T23:32:33.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.768+0000 7f8b59eed640 1 -- 172.21.15.161:0/1464955484 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b5406cdd0 msgr2=0x7f8b541524c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:33.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.768+0000 7f8b59eed640 1 --2- 172.21.15.161:0/1464955484 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b5406cdd0 0x7f8b541524c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8b48009870 tx=0x7f8b4802b980 comp rx=0 tx=0).stop 2024-03-28T23:32:33.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.768+0000 7f8b59eed640 1 -- 172.21.15.161:0/1464955484 shutdown_connections 2024-03-28T23:32:33.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.768+0000 7f8b59eed640 1 --2- 172.21.15.161:0/1464955484 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b5406cdd0 0x7f8b541524c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.768+0000 7f8b59eed640 1 --2- 172.21.15.161:0/1464955484 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b5414f5c0 0x7f8b540c2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.768+0000 7f8b59eed640 1 -- 172.21.15.161:0/1464955484 >> 172.21.15.161:0/1464955484 conn(0x7f8b540a83f0 msgr2=0x7f8b540a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:33.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.768+0000 7f8b59eed640 1 -- 172.21.15.161:0/1464955484 shutdown_connections 2024-03-28T23:32:33.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.768+0000 7f8b59eed640 1 -- 172.21.15.161:0/1464955484 wait complete. 2024-03-28T23:32:33.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.769+0000 7f8b59eed640 1 Processor -- start 2024-03-28T23:32:33.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.769+0000 7f8b59eed640 1 -- start start 2024-03-28T23:32:33.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.770+0000 7f8b59eed640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b5406cdd0 0x7f8b541e9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:33.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.770+0000 7f8b59eed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b540c2690 0x7f8b541e99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:33.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.770+0000 7f8b59eed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b5414f5c0 0x7f8b541ea640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:33.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.770+0000 7f8b59eed640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8b5406c140 con 0x7f8b5406cdd0 2024-03-28T23:32:33.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.770+0000 7f8b59eed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8b54152030 con 0x7f8b5414f5c0 2024-03-28T23:32:33.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.770+0000 7f8b59eed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8b5406c3e0 con 0x7f8b540c2690 2024-03-28T23:32:33.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.770+0000 7f8b537fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b5406cdd0 0x7f8b541e9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:33.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.770+0000 7f8b537fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b5406cdd0 0x7f8b541e9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40200/0 (socket says 172.21.15.161:40200) 2024-03-28T23:32:33.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.770+0000 7f8b537fe640 1 -- 172.21.15.161:0/3578419983 learned_addr learned my addr 172.21.15.161:0/3578419983 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:33.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.770+0000 7f8b52ffd640 1 --2- 172.21.15.161:0/3578419983 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b540c2690 0x7f8b541e99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:33.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.770+0000 7f8b53fff640 1 --2- 172.21.15.161:0/3578419983 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b5414f5c0 0x7f8b541ea640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:33.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.770+0000 7f8b537fe640 1 -- 172.21.15.161:0/3578419983 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b540c2690 msgr2=0x7f8b541e99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:33.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.770+0000 7f8b537fe640 1 --2- 172.21.15.161:0/3578419983 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b540c2690 0x7f8b541e99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.770+0000 7f8b537fe640 1 -- 172.21.15.161:0/3578419983 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b5414f5c0 msgr2=0x7f8b541ea640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:33.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.770+0000 7f8b537fe640 1 --2- 172.21.15.161:0/3578419983 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b5414f5c0 0x7f8b541ea640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:33.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.770+0000 7f8b537fe640 1 -- 172.21.15.161:0/3578419983 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b540722a0 con 0x7f8b5406cdd0 2024-03-28T23:32:33.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.770+0000 7f8b53fff640 1 --2- 172.21.15.161:0/3578419983 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b5414f5c0 0x7f8b541ea640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:33.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.770+0000 7f8b537fe640 1 --2- 172.21.15.161:0/3578419983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b5406cdd0 0x7f8b541e9470 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f8b4400e990 tx=0x7f8b4400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:33.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.770+0000 7f8b3bfff640 1 -- 172.21.15.161:0/3578419983 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b44017070 con 0x7f8b5406cdd0 2024-03-28T23:32:33.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.771+0000 7f8b3bfff640 1 -- 172.21.15.161:0/3578419983 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b4400c9e0 con 0x7f8b5406cdd0 2024-03-28T23:32:33.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.771+0000 7f8b59eed640 1 -- 172.21.15.161:0/3578419983 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b54072580 con 0x7f8b5406cdd0 2024-03-28T23:32:33.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.771+0000 7f8b3bfff640 1 -- 172.21.15.161:0/3578419983 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b4400cdb0 con 0x7f8b5406cdd0 2024-03-28T23:32:33.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.771+0000 7f8b59eed640 1 -- 172.21.15.161:0/3578419983 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b54072b40 con 0x7f8b5406cdd0 2024-03-28T23:32:33.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.771+0000 7f8b59eed640 1 -- 172.21.15.161:0/3578419983 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b540ab060 con 0x7f8b5406cdd0 2024-03-28T23:32:33.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.772+0000 7f8b3bfff640 1 -- 172.21.15.161:0/3578419983 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8b4401a070 con 0x7f8b5406cdd0 2024-03-28T23:32:33.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.772+0000 7f8b3bfff640 1 --2- 172.21.15.161:0/3578419983 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8b4006b4d0 0x7f8b4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:33.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.772+0000 7f8b3bfff640 1 -- 172.21.15.161:0/3578419983 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 6101+0+0 (secure 0 0 0) 0x7f8b4408a490 con 0x7f8b5406cdd0 2024-03-28T23:32:33.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.772+0000 7f8b52ffd640 1 --2- 172.21.15.161:0/3578419983 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8b4006b4d0 0x7f8b4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:33.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.773+0000 7f8b52ffd640 1 --2- 172.21.15.161:0/3578419983 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8b4006b4d0 0x7f8b4006d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8b48036000 tx=0x7f8b48008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:33.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.777+0000 7f8b3bfff640 1 -- 172.21.15.161:0/3578419983 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8b540ab060 con 0x7f8b5406cdd0 2024-03-28T23:32:33.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:33.986+0000 7f8b59eed640 1 -- 172.21.15.161:0/3578419983 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f8b540a94e0 con 0x7f8b5406cdd0 2024-03-28T23:32:34.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:34.938+0000 7f8b3bfff640 1 -- 172.21.15.161:0/3578419983 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v55) ==== 118+0+0 (secure 0 0 0) 0x7f8b540a94e0 con 0x7f8b5406cdd0 2024-03-28T23:32:34.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:34.941+0000 7f8b59eed640 1 -- 172.21.15.161:0/3578419983 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8b4006b4d0 msgr2=0x7f8b4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:34.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:34.941+0000 7f8b59eed640 1 --2- 172.21.15.161:0/3578419983 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8b4006b4d0 0x7f8b4006d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8b48036000 tx=0x7f8b48008030 comp rx=0 tx=0).stop 2024-03-28T23:32:34.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:34.941+0000 7f8b59eed640 1 -- 172.21.15.161:0/3578419983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b5406cdd0 msgr2=0x7f8b541e9470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:34.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:34.941+0000 7f8b59eed640 1 --2- 172.21.15.161:0/3578419983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b5406cdd0 0x7f8b541e9470 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f8b4400e990 tx=0x7f8b4400ee50 comp rx=0 tx=0).stop 2024-03-28T23:32:34.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:34.941+0000 7f8b59eed640 1 -- 172.21.15.161:0/3578419983 shutdown_connections 2024-03-28T23:32:34.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:34.941+0000 7f8b59eed640 1 --2- 172.21.15.161:0/3578419983 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8b4006b4d0 0x7f8b4006d990 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:34.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:34.941+0000 7f8b59eed640 1 --2- 172.21.15.161:0/3578419983 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b5414f5c0 0x7f8b541ea640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:34.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:34.941+0000 7f8b59eed640 1 --2- 172.21.15.161:0/3578419983 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b540c2690 0x7f8b541e99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:34.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:34.941+0000 7f8b59eed640 1 --2- 172.21.15.161:0/3578419983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b5406cdd0 0x7f8b541e9470 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:34.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:34.941+0000 7f8b59eed640 1 -- 172.21.15.161:0/3578419983 >> 172.21.15.161:0/3578419983 conn(0x7f8b540a83f0 msgr2=0x7f8b540a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:34.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:34.941+0000 7f8b59eed640 1 -- 172.21.15.161:0/3578419983 shutdown_connections 2024-03-28T23:32:34.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:34.941+0000 7f8b59eed640 1 -- 172.21.15.161:0/3578419983 wait complete. 2024-03-28T23:32:34.954 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-28T23:32:34.954 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-28T23:32:35.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.061+0000 7f23e187c640 1 Processor -- start 2024-03-28T23:32:35.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.061+0000 7f23e187c640 1 -- start start 2024-03-28T23:32:35.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.062+0000 7f23e187c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f23dc0c6740 0x7f23dc0c3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:35.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.062+0000 7f23e187c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23dc0c2e90 0x7f23dc0c6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:35.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.062+0000 7f23e187c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23dc0c3c20 con 0x7f23dc1528f0 2024-03-28T23:32:35.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.062+0000 7f23e187c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f23dc0c2a00 con 0x7f23dc0c6740 2024-03-28T23:32:35.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.062+0000 7f23e187c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f23dc0c23b0 con 0x7f23dc0c2e90 2024-03-28T23:32:35.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.062+0000 7f23db7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f23dc1528f0 0x7f23dc0abc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:58452/0 (socket says 172.21.15.161:58452) 2024-03-28T23:32:35.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.062+0000 7f23db7fe640 1 -- 172.21.15.161:0/965891004 learned_addr learned my addr 172.21.15.161:0/965891004 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23d3fff640 1 --2- 172.21.15.161:0/965891004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23dc0c2e90 0x7f23dc0c6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23da7fc640 1 -- 172.21.15.161:0/965891004 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 15766597 0 0) 0x7f23dc0c3c20 con 0x7f23dc1528f0 2024-03-28T23:32:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23da7fc640 1 -- 172.21.15.161:0/965891004 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23c8003520 con 0x7f23dc1528f0 2024-03-28T23:32:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23daffd640 1 --2- 172.21.15.161:0/965891004 >> v2:172.21.15.170:3300/0 conn(0x7f23dc0c6740 0x7f23dc0c3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23da7fc640 1 -- 172.21.15.161:0/965891004 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 773919371 0 0) 0x7f23c8003520 con 0x7f23dc1528f0 2024-03-28T23:32:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23da7fc640 1 -- 172.21.15.161:0/965891004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23dc0c2e90 msgr2=0x7f23dc0c6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23da7fc640 1 --2- 172.21.15.161:0/965891004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23dc0c2e90 0x7f23dc0c6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23da7fc640 1 -- 172.21.15.161:0/965891004 >> v2:172.21.15.170:3300/0 conn(0x7f23dc0c6740 msgr2=0x7f23dc0c3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23da7fc640 1 --2- 172.21.15.161:0/965891004 >> v2:172.21.15.170:3300/0 conn(0x7f23dc0c6740 0x7f23dc0c3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23da7fc640 1 -- 172.21.15.161:0/965891004 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23dc0c2690 con 0x7f23dc1528f0 2024-03-28T23:32:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23da7fc640 1 -- 172.21.15.161:0/965891004 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f23d40019f0 con 0x7f23dc1528f0 2024-03-28T23:32:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23da7fc640 1 -- 172.21.15.161:0/965891004 >> v1:172.21.15.161:6789/0 conn(0x7f23dc1528f0 legacy=0x7f23dc0abc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23da7fc640 1 --2- 172.21.15.161:0/965891004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23dc0c2e90 0x7f23c8003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:35.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23da7fc640 1 --2- 172.21.15.161:0/965891004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23c8004010 0x7f23c8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:35.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23da7fc640 1 --2- 172.21.15.161:0/965891004 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23c8004d40 0x7f23c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:35.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23da7fc640 1 -- 172.21.15.161:0/965891004 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f23dc0c2a00 con 0x7f23c8004d40 2024-03-28T23:32:35.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23da7fc640 1 -- 172.21.15.161:0/965891004 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f23dc0c23b0 con 0x7f23c8004010 2024-03-28T23:32:35.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23da7fc640 1 -- 172.21.15.161:0/965891004 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f23c800ba40 con 0x7f23dc0c2e90 2024-03-28T23:32:35.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23db7fe640 1 --2- 172.21.15.161:0/965891004 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23c8004d40 0x7f23c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:35.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23d3fff640 1 --2- 172.21.15.161:0/965891004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23c8004010 0x7f23c8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:35.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23daffd640 1 --2- 172.21.15.161:0/965891004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23dc0c2e90 0x7f23c8003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:35.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23db7fe640 1 -- 172.21.15.161:0/965891004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23dc0c2e90 msgr2=0x7f23c8003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:35.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23db7fe640 1 --2- 172.21.15.161:0/965891004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23dc0c2e90 0x7f23c8003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:35.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23db7fe640 1 -- 172.21.15.161:0/965891004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23c8004010 msgr2=0x7f23c8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:35.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23db7fe640 1 --2- 172.21.15.161:0/965891004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23c8004010 0x7f23c8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:35.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.063+0000 7f23db7fe640 1 -- 172.21.15.161:0/965891004 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23c8003520 con 0x7f23c8004d40 2024-03-28T23:32:35.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.065+0000 7f23db7fe640 1 --2- 172.21.15.161:0/965891004 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23c8004d40 0x7f23c80051c0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f23d40013d0 tx=0x7f23c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db235fda7249f5ee server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:35.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.065+0000 7f23da7fc640 1 -- 172.21.15.161:0/965891004 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d400a4e0 con 0x7f23c8004d40 2024-03-28T23:32:35.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.065+0000 7f23da7fc640 1 -- 172.21.15.161:0/965891004 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23d4034650 con 0x7f23c8004d40 2024-03-28T23:32:35.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.065+0000 7f23da7fc640 1 -- 172.21.15.161:0/965891004 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d4034a40 con 0x7f23c8004d40 2024-03-28T23:32:35.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.065+0000 7f23e187c640 1 -- 172.21.15.161:0/965891004 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23c8004d40 msgr2=0x7f23c80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:35.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.065+0000 7f23e187c640 1 --2- 172.21.15.161:0/965891004 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23c8004d40 0x7f23c80051c0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f23d40013d0 tx=0x7f23c8005830 comp rx=0 tx=0).stop 2024-03-28T23:32:35.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.065+0000 7f23db7fe640 1 -- 172.21.15.161:0/965891004 reap_dead start 2024-03-28T23:32:35.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.065+0000 7f23e187c640 1 -- 172.21.15.161:0/965891004 shutdown_connections 2024-03-28T23:32:35.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.065+0000 7f23e187c640 1 -- 172.21.15.161:0/965891004 >> 172.21.15.161:0/965891004 conn(0x7f23dc0c4330 msgr2=0x7f23dc0c4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:35.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.065+0000 7f23e187c640 1 -- 172.21.15.161:0/965891004 shutdown_connections 2024-03-28T23:32:35.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.065+0000 7f23e187c640 1 -- 172.21.15.161:0/965891004 wait complete. 2024-03-28T23:32:35.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.066+0000 7f23e187c640 1 Processor -- start 2024-03-28T23:32:35.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.066+0000 7f23e187c640 1 -- start start 2024-03-28T23:32:35.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.067+0000 7f23e187c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23c8004d40 0x7f23dc1ded10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:35.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.067+0000 7f23e187c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23dc0715d0 0x7f23dc14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:35.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.067+0000 7f23e187c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23dc1478b0 0x7f23dc1ee1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:35.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.067+0000 7f23e187c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f23dc14f7d0 con 0x7f23c8004d40 2024-03-28T23:32:35.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.067+0000 7f23e187c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f23dc0ac350 con 0x7f23dc0715d0 2024-03-28T23:32:35.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.067+0000 7f23e187c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f23dc14f950 con 0x7f23dc1478b0 2024-03-28T23:32:35.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.067+0000 7f23daffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23c8004d40 0x7f23dc1ded10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:35.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.067+0000 7f23daffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23c8004d40 0x7f23dc1ded10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40222/0 (socket says 172.21.15.161:40222) 2024-03-28T23:32:35.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.067+0000 7f23daffd640 1 -- 172.21.15.161:0/4232788173 learned_addr learned my addr 172.21.15.161:0/4232788173 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:35.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.067+0000 7f23db7fe640 1 --2- 172.21.15.161:0/4232788173 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23dc1478b0 0x7f23dc1ee1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.067+0000 7f23d3fff640 1 --2- 172.21.15.161:0/4232788173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23dc0715d0 0x7f23dc14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.068+0000 7f23daffd640 1 -- 172.21.15.161:0/4232788173 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23dc1478b0 msgr2=0x7f23dc1ee1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.068+0000 7f23daffd640 1 --2- 172.21.15.161:0/4232788173 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23dc1478b0 0x7f23dc1ee1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.068+0000 7f23daffd640 1 -- 172.21.15.161:0/4232788173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23dc0715d0 msgr2=0x7f23dc14e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.068+0000 7f23daffd640 1 --2- 172.21.15.161:0/4232788173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23dc0715d0 0x7f23dc14e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.068+0000 7f23daffd640 1 -- 172.21.15.161:0/4232788173 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23dc0722b0 con 0x7f23c8004d40 2024-03-28T23:32:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.068+0000 7f23d3fff640 1 --2- 172.21.15.161:0/4232788173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23dc0715d0 0x7f23dc14e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.068+0000 7f23daffd640 1 --2- 172.21.15.161:0/4232788173 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23c8004d40 0x7f23dc1ded10 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f23cc011a70 tx=0x7f23cc011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.068+0000 7f23d8ff9640 1 -- 172.21.15.161:0/4232788173 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23cc00fb50 con 0x7f23c8004d40 2024-03-28T23:32:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.068+0000 7f23d8ff9640 1 -- 172.21.15.161:0/4232788173 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23cc00fcf0 con 0x7f23c8004d40 2024-03-28T23:32:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.068+0000 7f23e187c640 1 -- 172.21.15.161:0/4232788173 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23dc072590 con 0x7f23c8004d40 2024-03-28T23:32:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.068+0000 7f23d8ff9640 1 -- 172.21.15.161:0/4232788173 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23cc00fea0 con 0x7f23c8004d40 2024-03-28T23:32:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.068+0000 7f23e187c640 1 -- 172.21.15.161:0/4232788173 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23dc14e730 con 0x7f23c8004d40 2024-03-28T23:32:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.069+0000 7f23e187c640 1 -- 172.21.15.161:0/4232788173 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23dc04f4b0 con 0x7f23c8004d40 2024-03-28T23:32:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.070+0000 7f23d8ff9640 1 -- 172.21.15.161:0/4232788173 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f23cc012040 con 0x7f23c8004d40 2024-03-28T23:32:35.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.070+0000 7f23d8ff9640 1 --2- 172.21.15.161:0/4232788173 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f23bc06b500 0x7f23bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:35.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.070+0000 7f23d8ff9640 1 -- 172.21.15.161:0/4232788173 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6101+0+0 (secure 0 0 0) 0x7f23cc08a9c0 con 0x7f23c8004d40 2024-03-28T23:32:35.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.070+0000 7f23d3fff640 1 --2- 172.21.15.161:0/4232788173 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f23bc06b500 0x7f23bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:35.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.071+0000 7f23d3fff640 1 --2- 172.21.15.161:0/4232788173 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f23bc06b500 0x7f23bc06d9c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f23c40093d0 tx=0x7f23c4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:35.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.075+0000 7f23d8ff9640 1 -- 172.21.15.161:0/4232788173 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f23cc08ce20 con 0x7f23c8004d40 2024-03-28T23:32:35.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:35.319+0000 7f23e187c640 1 -- 172.21.15.161:0/4232788173 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f23dc0c1f10 con 0x7f23c8004d40 2024-03-28T23:32:36.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.043+0000 7f23d8ff9640 1 -- 172.21.15.161:0/4232788173 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v56) ==== 122+0+0 (secure 0 0 0) 0x7f23cc019030 con 0x7f23c8004d40 2024-03-28T23:32:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.045+0000 7f23e187c640 1 -- 172.21.15.161:0/4232788173 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f23bc06b500 msgr2=0x7f23bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.045+0000 7f23e187c640 1 --2- 172.21.15.161:0/4232788173 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f23bc06b500 0x7f23bc06d9c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f23c40093d0 tx=0x7f23c4008040 comp rx=0 tx=0).stop 2024-03-28T23:32:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.045+0000 7f23e187c640 1 -- 172.21.15.161:0/4232788173 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23c8004d40 msgr2=0x7f23dc1ded10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.045+0000 7f23e187c640 1 --2- 172.21.15.161:0/4232788173 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23c8004d40 0x7f23dc1ded10 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f23cc011a70 tx=0x7f23cc011f30 comp rx=0 tx=0).stop 2024-03-28T23:32:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.045+0000 7f23e187c640 1 -- 172.21.15.161:0/4232788173 shutdown_connections 2024-03-28T23:32:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.045+0000 7f23e187c640 1 --2- 172.21.15.161:0/4232788173 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f23bc06b500 0x7f23bc06d9c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.045+0000 7f23e187c640 1 --2- 172.21.15.161:0/4232788173 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23dc1478b0 0x7f23dc1ee1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.045+0000 7f23e187c640 1 --2- 172.21.15.161:0/4232788173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23dc0715d0 0x7f23dc14e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.045+0000 7f23e187c640 1 --2- 172.21.15.161:0/4232788173 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23c8004d40 0x7f23dc1ded10 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.045+0000 7f23e187c640 1 -- 172.21.15.161:0/4232788173 >> 172.21.15.161:0/4232788173 conn(0x7f23dc0c4330 msgr2=0x7f23dc147fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.045+0000 7f23e187c640 1 -- 172.21.15.161:0/4232788173 shutdown_connections 2024-03-28T23:32:36.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.045+0000 7f23e187c640 1 -- 172.21.15.161:0/4232788173 wait complete. 2024-03-28T23:32:36.057 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-28T23:32:36.058 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:32:36.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.157+0000 7f28ee548640 1 Processor -- start 2024-03-28T23:32:36.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.157+0000 7f28ee548640 1 -- start start 2024-03-28T23:32:36.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.158+0000 7f28ee548640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28e80c5850 0x7f28e8155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:36.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.158+0000 7f28ee548640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28e80c5d00 con 0x7f28e8152890 2024-03-28T23:32:36.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.158+0000 7f28ee548640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28e8158a30 con 0x7f28e80c5410 2024-03-28T23:32:36.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.158+0000 7f28ee548640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f28e80c8480 con 0x7f28e80c5850 2024-03-28T23:32:36.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.158+0000 7f28e7fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f28e8152890 0x7f28e80c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:58464/0 (socket says 172.21.15.161:58464) 2024-03-28T23:32:36.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.158+0000 7f28e7fff640 1 -- 172.21.15.161:0/1811932894 learned_addr learned my addr 172.21.15.161:0/1811932894 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:36.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.158+0000 7f28ecae5640 1 --2- 172.21.15.161:0/1811932894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28e80c5850 0x7f28e8155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:36.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.158+0000 7f28e6ffd640 1 -- 172.21.15.161:0/1811932894 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3473123767 0 0) 0x7f28e80c5d00 con 0x7f28e8152890 2024-03-28T23:32:36.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.158+0000 7f28e6ffd640 1 -- 172.21.15.161:0/1811932894 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28d0003520 con 0x7f28e8152890 2024-03-28T23:32:36.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.158+0000 7f28e6ffd640 1 -- 172.21.15.161:0/1811932894 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3459159025 0 0) 0x7f28d0003520 con 0x7f28e8152890 2024-03-28T23:32:36.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.158+0000 7f28e6ffd640 1 -- 172.21.15.161:0/1811932894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28e80c5850 msgr2=0x7f28e8155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:36.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.158+0000 7f28e6ffd640 1 --2- 172.21.15.161:0/1811932894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28e80c5850 0x7f28e8155ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:36.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.158+0000 7f28e6ffd640 1 -- 172.21.15.161:0/1811932894 >> v1:172.21.15.170:6789/0 conn(0x7f28e80c5410 legacy=0x7f28e81553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:36.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.158+0000 7f28e6ffd640 1 -- 172.21.15.161:0/1811932894 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28e8158d10 con 0x7f28e8152890 2024-03-28T23:32:36.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.158+0000 7f28e6ffd640 1 -- 172.21.15.161:0/1811932894 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f28d8001670 con 0x7f28e8152890 2024-03-28T23:32:36.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.158+0000 7f28e6ffd640 1 -- 172.21.15.161:0/1811932894 >> v1:172.21.15.161:6789/0 conn(0x7f28e8152890 legacy=0x7f28e80c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:36.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.158+0000 7f28e6ffd640 1 --2- 172.21.15.161:0/1811932894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28d0003ac0 0x7f28d0003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:36.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.158+0000 7f28e6ffd640 1 --2- 172.21.15.161:0/1811932894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28e80c5850 0x7f28d0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:36.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.158+0000 7f28e6ffd640 1 --2- 172.21.15.161:0/1811932894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28d0004d40 0x7f28d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:36.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.159+0000 7f28e6ffd640 1 -- 172.21.15.161:0/1811932894 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f28d000bb50 con 0x7f28d0003ac0 2024-03-28T23:32:36.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.159+0000 7f28e6ffd640 1 -- 172.21.15.161:0/1811932894 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f28d000bd00 con 0x7f28d0004d40 2024-03-28T23:32:36.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.159+0000 7f28e6ffd640 1 -- 172.21.15.161:0/1811932894 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f28d000bf00 con 0x7f28e80c5850 2024-03-28T23:32:36.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.159+0000 7f28e7fff640 1 --2- 172.21.15.161:0/1811932894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28d0003ac0 0x7f28d0003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:36.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.159+0000 7f28e77fe640 1 --2- 172.21.15.161:0/1811932894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28e80c5850 0x7f28d0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:36.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.159+0000 7f28ecae5640 1 --2- 172.21.15.161:0/1811932894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28d0004d40 0x7f28d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:36.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.159+0000 7f28e7fff640 1 -- 172.21.15.161:0/1811932894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28e80c5850 msgr2=0x7f28d0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:36.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.159+0000 7f28e7fff640 1 --2- 172.21.15.161:0/1811932894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28e80c5850 0x7f28d0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:36.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.159+0000 7f28e7fff640 1 -- 172.21.15.161:0/1811932894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28d0004d40 msgr2=0x7f28d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:36.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.159+0000 7f28e7fff640 1 --2- 172.21.15.161:0/1811932894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28d0004d40 0x7f28d00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:36.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.159+0000 7f28e7fff640 1 -- 172.21.15.161:0/1811932894 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d0003520 con 0x7f28d0003ac0 2024-03-28T23:32:36.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.159+0000 7f28ecae5640 1 --2- 172.21.15.161:0/1811932894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28d0004d40 0x7f28d00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:36.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.160+0000 7f28e7fff640 1 --2- 172.21.15.161:0/1811932894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28d0003ac0 0x7f28d0003ea0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f28d800c290 tx=0x7f28d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb7a2f8fd7abd0d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:36.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.160+0000 7f28e6ffd640 1 -- 172.21.15.161:0/1811932894 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d8007d50 con 0x7f28d0003ac0 2024-03-28T23:32:36.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.160+0000 7f28e6ffd640 1 -- 172.21.15.161:0/1811932894 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28d8030070 con 0x7f28d0003ac0 2024-03-28T23:32:36.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.160+0000 7f28e6ffd640 1 -- 172.21.15.161:0/1811932894 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d802cd80 con 0x7f28d0003ac0 2024-03-28T23:32:36.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.160+0000 7f28ee548640 1 -- 172.21.15.161:0/1811932894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28d0003ac0 msgr2=0x7f28d0003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:36.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.160+0000 7f28ee548640 1 --2- 172.21.15.161:0/1811932894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28d0003ac0 0x7f28d0003ea0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f28d800c290 tx=0x7f28d0005830 comp rx=0 tx=0).stop 2024-03-28T23:32:36.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.160+0000 7f28ecae5640 1 -- 172.21.15.161:0/1811932894 reap_dead start 2024-03-28T23:32:36.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.160+0000 7f28ee548640 1 -- 172.21.15.161:0/1811932894 shutdown_connections 2024-03-28T23:32:36.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.160+0000 7f28ee548640 1 -- 172.21.15.161:0/1811932894 >> 172.21.15.161:0/1811932894 conn(0x7f28e80ac620 msgr2=0x7f28e80aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:36.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.160+0000 7f28ee548640 1 -- 172.21.15.161:0/1811932894 shutdown_connections 2024-03-28T23:32:36.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.160+0000 7f28ee548640 1 -- 172.21.15.161:0/1811932894 wait complete. 2024-03-28T23:32:36.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.161+0000 7f28ee548640 1 Processor -- start 2024-03-28T23:32:36.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.162+0000 7f28ee548640 1 -- start start 2024-03-28T23:32:36.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.162+0000 7f28ee548640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28d0003ac0 0x7f28e81eb370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:36.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.162+0000 7f28ee548640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28e80c2a80 0x7f28e80c4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:36.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.162+0000 7f28ee548640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28e80c49a0 0x7f28e80c0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:36.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.162+0000 7f28ee548640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f28e81592d0 con 0x7f28e80c2a80 2024-03-28T23:32:36.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.162+0000 7f28ee548640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f28e8159640 con 0x7f28e80c49a0 2024-03-28T23:32:36.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.162+0000 7f28ee548640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f28e80c90a0 con 0x7f28d0003ac0 2024-03-28T23:32:36.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.162+0000 7f28e77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28e80c2a80 0x7f28e80c4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:36.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.162+0000 7f28e77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28e80c2a80 0x7f28e80c4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40248/0 (socket says 172.21.15.161:40248) 2024-03-28T23:32:36.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.162+0000 7f28e77fe640 1 -- 172.21.15.161:0/3506877428 learned_addr learned my addr 172.21.15.161:0/3506877428 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:36.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.163+0000 7f28e7fff640 1 --2- 172.21.15.161:0/3506877428 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28d0003ac0 0x7f28e81eb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:36.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.163+0000 7f28ecae5640 1 --2- 172.21.15.161:0/3506877428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28e80c49a0 0x7f28e80c0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:36.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.163+0000 7f28e77fe640 1 -- 172.21.15.161:0/3506877428 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28d0003ac0 msgr2=0x7f28e81eb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:36.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.163+0000 7f28e77fe640 1 --2- 172.21.15.161:0/3506877428 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28d0003ac0 0x7f28e81eb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:36.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.163+0000 7f28e77fe640 1 -- 172.21.15.161:0/3506877428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28e80c49a0 msgr2=0x7f28e80c0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:36.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.163+0000 7f28e77fe640 1 --2- 172.21.15.161:0/3506877428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28e80c49a0 0x7f28e80c0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:36.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.163+0000 7f28e77fe640 1 -- 172.21.15.161:0/3506877428 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28e81eb8b0 con 0x7f28e80c2a80 2024-03-28T23:32:36.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.163+0000 7f28e77fe640 1 --2- 172.21.15.161:0/3506877428 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28e80c2a80 0x7f28e80c4460 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f28dc00f7f0 tx=0x7f28dc00fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:36.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.163+0000 7f28e4ff9640 1 -- 172.21.15.161:0/3506877428 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28dc00fea0 con 0x7f28e80c2a80 2024-03-28T23:32:36.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.163+0000 7f28e4ff9640 1 -- 172.21.15.161:0/3506877428 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28dc018070 con 0x7f28e80c2a80 2024-03-28T23:32:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.163+0000 7f28ee548640 1 -- 172.21.15.161:0/3506877428 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28e80be6d0 con 0x7f28e80c2a80 2024-03-28T23:32:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.163+0000 7f28e4ff9640 1 -- 172.21.15.161:0/3506877428 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28dc00d6d0 con 0x7f28e80c2a80 2024-03-28T23:32:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.163+0000 7f28ee548640 1 -- 172.21.15.161:0/3506877428 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28e80bec90 con 0x7f28e80c2a80 2024-03-28T23:32:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.164+0000 7f28ee548640 1 -- 172.21.15.161:0/3506877428 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28e804f4b0 con 0x7f28e80c2a80 2024-03-28T23:32:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.165+0000 7f28e4ff9640 1 -- 172.21.15.161:0/3506877428 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f28dc00d870 con 0x7f28e80c2a80 2024-03-28T23:32:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.165+0000 7f28e4ff9640 1 --2- 172.21.15.161:0/3506877428 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f28d406b500 0x7f28d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.165+0000 7f28e4ff9640 1 -- 172.21.15.161:0/3506877428 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6101+0+0 (secure 0 0 0) 0x7f28dc08a2f0 con 0x7f28e80c2a80 2024-03-28T23:32:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.165+0000 7f28e7fff640 1 --2- 172.21.15.161:0/3506877428 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f28d406b500 0x7f28d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.166+0000 7f28e7fff640 1 --2- 172.21.15.161:0/3506877428 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f28d406b500 0x7f28d406d9c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f28d8034000 tx=0x7f28d8039040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:36.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.170+0000 7f28e4ff9640 1 -- 172.21.15.161:0/3506877428 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f28dc01a310 con 0x7f28e80c2a80 2024-03-28T23:32:36.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.386+0000 7f28ee548640 1 -- 172.21.15.161:0/3506877428 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28e80c8140 con 0x7f28e80c2a80 2024-03-28T23:32:36.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.387+0000 7f28e4ff9640 1 -- 172.21.15.161:0/3506877428 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+5192 (secure 0 0 0) 0x7f28dc058fe0 con 0x7f28e80c2a80 2024-03-28T23:32:36.388 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:36.388 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":56,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/737681942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":737681942},{"type":"v1","addr":"172.21.15.161:6837","nonce":737681942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":5579,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/3501962065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3501962065},{"type":"v1","addr":"172.21.15.170:6834","nonce":3501962065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6037,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2691084827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":2691084827},{"type":"v1","addr":"172.21.15.170:6835","nonce":2691084827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:32:32.372964+0000","modified":"2024-03-28T23:32:35.321715+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6468},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6468":{"gid":6468,"name":"a","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6836/3850309502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3850309502},{"type":"v1","addr":"172.21.15.161:6836","nonce":3850309502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6468,"qdb_cluster":[6468]},"id":5}]} 2024-03-28T23:32:36.389 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 56 2024-03-28T23:32:36.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.389+0000 7f28ee548640 1 -- 172.21.15.161:0/3506877428 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f28d406b500 msgr2=0x7f28d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:36.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.389+0000 7f28ee548640 1 --2- 172.21.15.161:0/3506877428 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f28d406b500 0x7f28d406d9c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f28d8034000 tx=0x7f28d8039040 comp rx=0 tx=0).stop 2024-03-28T23:32:36.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.389+0000 7f28ee548640 1 -- 172.21.15.161:0/3506877428 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28e80c2a80 msgr2=0x7f28e80c4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:36.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.389+0000 7f28ee548640 1 --2- 172.21.15.161:0/3506877428 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28e80c2a80 0x7f28e80c4460 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f28dc00f7f0 tx=0x7f28dc00fcb0 comp rx=0 tx=0).stop 2024-03-28T23:32:36.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.389+0000 7f28ee548640 1 -- 172.21.15.161:0/3506877428 shutdown_connections 2024-03-28T23:32:36.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.389+0000 7f28ee548640 1 --2- 172.21.15.161:0/3506877428 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f28d406b500 0x7f28d406d9c0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:36.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.389+0000 7f28ee548640 1 --2- 172.21.15.161:0/3506877428 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28e80c2a80 0x7f28e80c4460 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:36.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.389+0000 7f28ee548640 1 --2- 172.21.15.161:0/3506877428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28e80c49a0 0x7f28e80c0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:36.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.389+0000 7f28ee548640 1 --2- 172.21.15.161:0/3506877428 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28d0003ac0 0x7f28e81eb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:36.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.389+0000 7f28ee548640 1 -- 172.21.15.161:0/3506877428 >> 172.21.15.161:0/3506877428 conn(0x7f28e80ac620 msgr2=0x7f28e81500b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:36.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.389+0000 7f28ee548640 1 -- 172.21.15.161:0/3506877428 shutdown_connections 2024-03-28T23:32:36.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.389+0000 7f28ee548640 1 -- 172.21.15.161:0/3506877428 wait complete. 2024-03-28T23:32:36.401 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:32:36.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.502+0000 7f66eda15640 1 Processor -- start 2024-03-28T23:32:36.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.502+0000 7f66eda15640 1 -- start start 2024-03-28T23:32:36.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.503+0000 7f66eda15640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66e8155d10 0x7f66e80c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:36.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.503+0000 7f66eda15640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66e806c3c0 con 0x7f66e8152a80 2024-03-28T23:32:36.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.503+0000 7f66eda15640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66e80c5b90 con 0x7f66e806d9f0 2024-03-28T23:32:36.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.503+0000 7f66eda15640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f66e806c6f0 con 0x7f66e8155d10 2024-03-28T23:32:36.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.503+0000 7f66e6ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f66e8152a80 0x7f66e80c93a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:58472/0 (socket says 172.21.15.161:58472) 2024-03-28T23:32:36.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.503+0000 7f66e6ffd640 1 -- 172.21.15.161:0/1668728867 learned_addr learned my addr 172.21.15.161:0/1668728867 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:36.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.503+0000 7f66e77fe640 1 --2- 172.21.15.161:0/1668728867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66e8155d10 0x7f66e80c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:36.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.504+0000 7f66e5ffb640 1 -- 172.21.15.161:0/1668728867 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 662471425 0 0) 0x7f66e806c3c0 con 0x7f66e8152a80 2024-03-28T23:32:36.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.504+0000 7f66e5ffb640 1 -- 172.21.15.161:0/1668728867 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66d0003520 con 0x7f66e8152a80 2024-03-28T23:32:36.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.504+0000 7f66e77fe640 1 -- 172.21.15.161:0/1668728867 >> v1:172.21.15.170:6789/0 conn(0x7f66e806d9f0 legacy=0x7f66e806cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:36.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.504+0000 7f66e77fe640 1 -- 172.21.15.161:0/1668728867 >> v1:172.21.15.161:6789/0 conn(0x7f66e8152a80 legacy=0x7f66e80c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:36.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.504+0000 7f66e77fe640 1 -- 172.21.15.161:0/1668728867 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66e80c55d0 con 0x7f66e8155d10 2024-03-28T23:32:36.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.505+0000 7f66e77fe640 1 --2- 172.21.15.161:0/1668728867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66e8155d10 0x7f66e80c74c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f66e806c870 tx=0x7f66d802bc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11eac0114e24cd97 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:36.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.505+0000 7f66e5ffb640 1 -- 172.21.15.161:0/1668728867 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66d8039070 con 0x7f66e8155d10 2024-03-28T23:32:36.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.505+0000 7f66e5ffb640 1 -- 172.21.15.161:0/1668728867 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66d802c980 con 0x7f66e8155d10 2024-03-28T23:32:36.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.505+0000 7f66e5ffb640 1 -- 172.21.15.161:0/1668728867 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66d802cd50 con 0x7f66e8155d10 2024-03-28T23:32:36.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.505+0000 7f66eda15640 1 -- 172.21.15.161:0/1668728867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66e8155d10 msgr2=0x7f66e80c74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:36.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.505+0000 7f66eda15640 1 --2- 172.21.15.161:0/1668728867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66e8155d10 0x7f66e80c74c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f66e806c870 tx=0x7f66d802bc50 comp rx=0 tx=0).stop 2024-03-28T23:32:36.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.505+0000 7f66eda15640 1 -- 172.21.15.161:0/1668728867 shutdown_connections 2024-03-28T23:32:36.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.505+0000 7f66eda15640 1 --2- 172.21.15.161:0/1668728867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66e8155d10 0x7f66e80c74c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:36.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.505+0000 7f66eda15640 1 -- 172.21.15.161:0/1668728867 >> 172.21.15.161:0/1668728867 conn(0x7f66e80ac720 msgr2=0x7f66e80acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:36.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.505+0000 7f66eda15640 1 -- 172.21.15.161:0/1668728867 shutdown_connections 2024-03-28T23:32:36.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.505+0000 7f66eda15640 1 -- 172.21.15.161:0/1668728867 wait complete. 2024-03-28T23:32:36.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.506+0000 7f66eda15640 1 Processor -- start 2024-03-28T23:32:36.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.507+0000 7f66eda15640 1 -- start start 2024-03-28T23:32:36.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.507+0000 7f66eda15640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66e806d9f0 0x7f66e80c1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:36.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.507+0000 7f66eda15640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66e8152a80 0x7f66e8153c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:36.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.507+0000 7f66eda15640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66e8155d10 0x7f66e8154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:36.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.507+0000 7f66eda15640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f66e80c7a00 con 0x7f66e8155d10 2024-03-28T23:32:36.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.507+0000 7f66eda15640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f66e81561a0 con 0x7f66e806d9f0 2024-03-28T23:32:36.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.507+0000 7f66eda15640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f66e80c7b80 con 0x7f66e8152a80 2024-03-28T23:32:36.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.507+0000 7f66e77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66e8155d10 0x7f66e8154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:36.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.507+0000 7f66e77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66e8155d10 0x7f66e8154140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40258/0 (socket says 172.21.15.161:40258) 2024-03-28T23:32:36.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.507+0000 7f66e77fe640 1 -- 172.21.15.161:0/287455264 learned_addr learned my addr 172.21.15.161:0/287455264 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:36.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.508+0000 7f66e77fe640 1 -- 172.21.15.161:0/287455264 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66e8152a80 msgr2=0x7f66e8153c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:36.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.508+0000 7f66e77fe640 1 --2- 172.21.15.161:0/287455264 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66e8152a80 0x7f66e8153c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:36.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.508+0000 7f66e77fe640 1 -- 172.21.15.161:0/287455264 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66e806d9f0 msgr2=0x7f66e80c1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:36.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.508+0000 7f66e77fe640 1 --2- 172.21.15.161:0/287455264 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66e806d9f0 0x7f66e80c1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:36.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.508+0000 7f66e77fe640 1 -- 172.21.15.161:0/287455264 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66e8148f70 con 0x7f66e8155d10 2024-03-28T23:32:36.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.508+0000 7f66e77fe640 1 --2- 172.21.15.161:0/287455264 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66e8155d10 0x7f66e8154140 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f66d8037040 tx=0x7f66d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:36.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.508+0000 7f66cffff640 1 -- 172.21.15.161:0/287455264 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66d8039050 con 0x7f66e8155d10 2024-03-28T23:32:36.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.508+0000 7f66cffff640 1 -- 172.21.15.161:0/287455264 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66d8045070 con 0x7f66e8155d10 2024-03-28T23:32:36.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.508+0000 7f66eda15640 1 -- 172.21.15.161:0/287455264 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66e8152000 con 0x7f66e8155d10 2024-03-28T23:32:36.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.508+0000 7f66cffff640 1 -- 172.21.15.161:0/287455264 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66d8009450 con 0x7f66e8155d10 2024-03-28T23:32:36.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.508+0000 7f66eda15640 1 -- 172.21.15.161:0/287455264 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66e80c15d0 con 0x7f66e8155d10 2024-03-28T23:32:36.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.509+0000 7f66eda15640 1 -- 172.21.15.161:0/287455264 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66e806b3d0 con 0x7f66e8155d10 2024-03-28T23:32:36.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.510+0000 7f66cffff640 1 -- 172.21.15.161:0/287455264 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f66d804a460 con 0x7f66e8155d10 2024-03-28T23:32:36.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.510+0000 7f66cffff640 1 --2- 172.21.15.161:0/287455264 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f66d406b4d0 0x7f66d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:36.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.510+0000 7f66cffff640 1 -- 172.21.15.161:0/287455264 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6101+0+0 (secure 0 0 0) 0x7f66d80acb30 con 0x7f66e8155d10 2024-03-28T23:32:36.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.510+0000 7f66e6ffd640 1 --2- 172.21.15.161:0/287455264 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f66d406b4d0 0x7f66d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:36.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.510+0000 7f66e6ffd640 1 --2- 172.21.15.161:0/287455264 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f66d406b4d0 0x7f66d406d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f66e80c4e50 tx=0x7f66e0007930 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:36.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.514+0000 7f66cffff640 1 -- 172.21.15.161:0/287455264 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f66e806b3d0 con 0x7f66e8155d10 2024-03-28T23:32:36.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.690+0000 7f66eda15640 1 -- 172.21.15.161:0/287455264 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f66e804f4b0 con 0x7f66e8155d10 2024-03-28T23:32:36.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.691+0000 7f66cffff640 1 -- 172.21.15.161:0/287455264 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+16981 (secure 0 0 0) 0x7f66d807b8b0 con 0x7f66e8155d10 2024-03-28T23:32:36.693 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:36.693 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":73,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:32:34.022058+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:32:30.684936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-28T23:32:31.339722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:32:36.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.693+0000 7f66eda15640 1 -- 172.21.15.161:0/287455264 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f66d406b4d0 msgr2=0x7f66d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:36.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.693+0000 7f66eda15640 1 --2- 172.21.15.161:0/287455264 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f66d406b4d0 0x7f66d406d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f66e80c4e50 tx=0x7f66e0007930 comp rx=0 tx=0).stop 2024-03-28T23:32:36.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.693+0000 7f66eda15640 1 -- 172.21.15.161:0/287455264 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66e8155d10 msgr2=0x7f66e8154140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:36.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.693+0000 7f66eda15640 1 --2- 172.21.15.161:0/287455264 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66e8155d10 0x7f66e8154140 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f66d8037040 tx=0x7f66d8004a40 comp rx=0 tx=0).stop 2024-03-28T23:32:36.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.693+0000 7f66eda15640 1 -- 172.21.15.161:0/287455264 shutdown_connections 2024-03-28T23:32:36.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.693+0000 7f66eda15640 1 --2- 172.21.15.161:0/287455264 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f66d406b4d0 0x7f66d406d990 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:36.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.693+0000 7f66eda15640 1 --2- 172.21.15.161:0/287455264 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66e8155d10 0x7f66e8154140 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:36.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.693+0000 7f66eda15640 1 --2- 172.21.15.161:0/287455264 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66e8152a80 0x7f66e8153c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:36.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.693+0000 7f66eda15640 1 --2- 172.21.15.161:0/287455264 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66e806d9f0 0x7f66e80c1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:36.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.693+0000 7f66eda15640 1 -- 172.21.15.161:0/287455264 >> 172.21.15.161:0/287455264 conn(0x7f66e80ac720 msgr2=0x7f66e80aac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:36.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.693+0000 7f66eda15640 1 -- 172.21.15.161:0/287455264 shutdown_connections 2024-03-28T23:32:36.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.693+0000 7f66eda15640 1 -- 172.21.15.161:0/287455264 wait complete. 2024-03-28T23:32:36.706 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T23:32:36.706 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:32:36.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.808+0000 7f0608e40640 1 Processor -- start 2024-03-28T23:32:36.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.808+0000 7f0608e40640 1 -- start start 2024-03-28T23:32:36.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.809+0000 7f0608e40640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06040c5850 0x7f0604155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:36.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.809+0000 7f0608e40640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06040c5d00 con 0x7f06040c5410 2024-03-28T23:32:36.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.809+0000 7f0608e40640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0604158a30 con 0x7f0604152890 2024-03-28T23:32:36.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.809+0000 7f0608e40640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f06040c8480 con 0x7f06040c5850 2024-03-28T23:32:36.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.809+0000 7f05fbfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f06040c5410 0x7f06041553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:58474/0 (socket says 172.21.15.161:58474) 2024-03-28T23:32:36.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.809+0000 7f05fbfff640 1 -- 172.21.15.161:0/2426438510 learned_addr learned my addr 172.21.15.161:0/2426438510 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:36.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.809+0000 7f0602d9d640 1 --2- 172.21.15.161:0/2426438510 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06040c5850 0x7f0604155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:36.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.809+0000 7f0601d9b640 1 -- 172.21.15.161:0/2426438510 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2906715469 0 0) 0x7f06040c5d00 con 0x7f06040c5410 2024-03-28T23:32:36.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.809+0000 7f0601d9b640 1 -- 172.21.15.161:0/2426438510 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05f0003520 con 0x7f06040c5410 2024-03-28T23:32:36.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.809+0000 7f0601d9b640 1 -- 172.21.15.161:0/2426438510 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 355964862 0 0) 0x7f0604158a30 con 0x7f0604152890 2024-03-28T23:32:36.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.809+0000 7f0601d9b640 1 -- 172.21.15.161:0/2426438510 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06040c5d00 con 0x7f0604152890 2024-03-28T23:32:36.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.809+0000 7f0601d9b640 1 -- 172.21.15.161:0/2426438510 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1371702673 0 0) 0x7f05f0003520 con 0x7f06040c5410 2024-03-28T23:32:36.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.809+0000 7f0602d9d640 1 -- 172.21.15.161:0/2426438510 >> v1:172.21.15.170:6789/0 conn(0x7f0604152890 legacy=0x7f06040c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:36.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.809+0000 7f0602d9d640 1 -- 172.21.15.161:0/2426438510 >> v1:172.21.15.161:6789/0 conn(0x7f06040c5410 legacy=0x7f06041553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:36.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.809+0000 7f0602d9d640 1 -- 172.21.15.161:0/2426438510 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0604158d10 con 0x7f06040c5850 2024-03-28T23:32:36.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.810+0000 7f0602d9d640 1 --2- 172.21.15.161:0/2426438510 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06040c5850 0x7f0604155ab0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f05fc009dc0 tx=0x7f05fc02bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c273eb3786e2bc7e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:36.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.810+0000 7f0601d9b640 1 -- 172.21.15.161:0/2426438510 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05fc039070 con 0x7f06040c5850 2024-03-28T23:32:36.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.810+0000 7f0601d9b640 1 -- 172.21.15.161:0/2426438510 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05fc02c880 con 0x7f06040c5850 2024-03-28T23:32:36.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.810+0000 7f0601d9b640 1 -- 172.21.15.161:0/2426438510 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ec0017f0 con 0x7f06040c5850 2024-03-28T23:32:36.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.811+0000 7f0608e40640 1 -- 172.21.15.161:0/2426438510 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06040c5850 msgr2=0x7f0604155ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:36.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.811+0000 7f0608e40640 1 --2- 172.21.15.161:0/2426438510 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06040c5850 0x7f0604155ab0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f05fc009dc0 tx=0x7f05fc02bd50 comp rx=0 tx=0).stop 2024-03-28T23:32:36.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.811+0000 7f0608e40640 1 -- 172.21.15.161:0/2426438510 shutdown_connections 2024-03-28T23:32:36.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.811+0000 7f0608e40640 1 --2- 172.21.15.161:0/2426438510 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06040c5850 0x7f0604155ab0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:36.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.811+0000 7f0608e40640 1 -- 172.21.15.161:0/2426438510 >> 172.21.15.161:0/2426438510 conn(0x7f06040ac620 msgr2=0x7f06040aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:36.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.811+0000 7f0608e40640 1 -- 172.21.15.161:0/2426438510 shutdown_connections 2024-03-28T23:32:36.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.811+0000 7f0608e40640 1 -- 172.21.15.161:0/2426438510 wait complete. 2024-03-28T23:32:36.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.813+0000 7f0608e40640 1 Processor -- start 2024-03-28T23:32:36.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.813+0000 7f0608e40640 1 -- start start 2024-03-28T23:32:36.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.814+0000 7f0608e40640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06040c5410 0x7f060414bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:36.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.814+0000 7f0608e40640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06040c5850 0x7f06041547c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:36.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.814+0000 7f0608e40640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0604152890 0x7f0604153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:36.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.814+0000 7f0608e40640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f06041592d0 con 0x7f06040c5410 2024-03-28T23:32:36.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.814+0000 7f0608e40640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0604159640 con 0x7f0604152890 2024-03-28T23:32:36.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.814+0000 7f0608e40640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0604155ff0 con 0x7f06040c5850 2024-03-28T23:32:36.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.814+0000 7f060259c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06040c5410 0x7f060414bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:36.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.814+0000 7f060259c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06040c5410 0x7f060414bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40274/0 (socket says 172.21.15.161:40274) 2024-03-28T23:32:36.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.814+0000 7f060259c640 1 -- 172.21.15.161:0/1339548149 learned_addr learned my addr 172.21.15.161:0/1339548149 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:36.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.814+0000 7f05fbfff640 1 --2- 172.21.15.161:0/1339548149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06040c5850 0x7f06041547c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:36.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.814+0000 7f0602d9d640 1 --2- 172.21.15.161:0/1339548149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0604152890 0x7f0604153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:36.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.814+0000 7f060259c640 1 -- 172.21.15.161:0/1339548149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06040c5850 msgr2=0x7f06041547c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:36.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.814+0000 7f060259c640 1 --2- 172.21.15.161:0/1339548149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06040c5850 0x7f06041547c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:36.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.814+0000 7f060259c640 1 -- 172.21.15.161:0/1339548149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0604152890 msgr2=0x7f0604153210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:36.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.814+0000 7f060259c640 1 --2- 172.21.15.161:0/1339548149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0604152890 0x7f0604153210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:36.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.814+0000 7f060259c640 1 -- 172.21.15.161:0/1339548149 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0604154d00 con 0x7f06040c5410 2024-03-28T23:32:36.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.814+0000 7f0602d9d640 1 --2- 172.21.15.161:0/1339548149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0604152890 0x7f0604153210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:36.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.814+0000 7f060259c640 1 --2- 172.21.15.161:0/1339548149 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06040c5410 0x7f060414bf90 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f05f400c490 tx=0x7f05f400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:36.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.814+0000 7f05faffd640 1 -- 172.21.15.161:0/1339548149 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f40052d0 con 0x7f06040c5410 2024-03-28T23:32:36.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.814+0000 7f05faffd640 1 -- 172.21.15.161:0/1339548149 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05f4005470 con 0x7f06040c5410 2024-03-28T23:32:36.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.814+0000 7f0608e40640 1 -- 172.21.15.161:0/1339548149 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06041eb9a0 con 0x7f06040c5410 2024-03-28T23:32:36.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.814+0000 7f05faffd640 1 -- 172.21.15.161:0/1339548149 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f4012850 con 0x7f06040c5410 2024-03-28T23:32:36.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.814+0000 7f0608e40640 1 -- 172.21.15.161:0/1339548149 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0604153750 con 0x7f06040c5410 2024-03-28T23:32:36.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.815+0000 7f0608e40640 1 -- 172.21.15.161:0/1339548149 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f060404f4b0 con 0x7f06040c5410 2024-03-28T23:32:36.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.816+0000 7f05faffd640 1 -- 172.21.15.161:0/1339548149 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f05f40129f0 con 0x7f06040c5410 2024-03-28T23:32:36.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.816+0000 7f05faffd640 1 --2- 172.21.15.161:0/1339548149 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f05e406b4d0 0x7f05e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:36.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.816+0000 7f05faffd640 1 -- 172.21.15.161:0/1339548149 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6101+0+0 (secure 0 0 0) 0x7f05f4089fe0 con 0x7f06040c5410 2024-03-28T23:32:36.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.816+0000 7f05fbfff640 1 --2- 172.21.15.161:0/1339548149 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f05e406b4d0 0x7f05e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:36.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.817+0000 7f05fbfff640 1 --2- 172.21.15.161:0/1339548149 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f05e406b4d0 0x7f05e406d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f06040c4e90 tx=0x7f05ec009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:36.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:36.821+0000 7f05faffd640 1 -- 172.21.15.161:0/1339548149 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f05f4058d60 con 0x7f06040c5410 2024-03-28T23:32:37.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.129+0000 7f0608e40640 1 -- 172.21.15.161:0/1339548149 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f06041eb750 con 0x7f05e406b4d0 2024-03-28T23:32:37.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.135+0000 7f05faffd640 1 -- 172.21.15.161:0/1339548149 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252144 (secure 0 0 0) 0x7f06041eb750 con 0x7f05e406b4d0 2024-03-28T23:32:37.139 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:37.141 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:32:37.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.138+0000 7f0608e40640 1 -- 172.21.15.161:0/1339548149 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f05e406b4d0 msgr2=0x7f05e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:37.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.138+0000 7f0608e40640 1 --2- 172.21.15.161:0/1339548149 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f05e406b4d0 0x7f05e406d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f06040c4e90 tx=0x7f05ec009040 comp rx=0 tx=0).stop 2024-03-28T23:32:37.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.138+0000 7f0608e40640 1 -- 172.21.15.161:0/1339548149 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06040c5410 msgr2=0x7f060414bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:37.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.138+0000 7f0608e40640 1 --2- 172.21.15.161:0/1339548149 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06040c5410 0x7f060414bf90 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f05f400c490 tx=0x7f05f400c950 comp rx=0 tx=0).stop 2024-03-28T23:32:37.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.138+0000 7f0608e40640 1 -- 172.21.15.161:0/1339548149 shutdown_connections 2024-03-28T23:32:37.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.138+0000 7f0608e40640 1 --2- 172.21.15.161:0/1339548149 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f05e406b4d0 0x7f05e406d990 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.138+0000 7f0608e40640 1 --2- 172.21.15.161:0/1339548149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0604152890 0x7f0604153210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.138+0000 7f0608e40640 1 --2- 172.21.15.161:0/1339548149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06040c5850 0x7f06041547c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.138+0000 7f0608e40640 1 --2- 172.21.15.161:0/1339548149 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06040c5410 0x7f060414bf90 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.138+0000 7f0608e40640 1 -- 172.21.15.161:0/1339548149 >> 172.21.15.161:0/1339548149 conn(0x7f06040ac620 msgr2=0x7f060414d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:37.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.138+0000 7f0608e40640 1 -- 172.21.15.161:0/1339548149 shutdown_connections 2024-03-28T23:32:37.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.138+0000 7f0608e40640 1 -- 172.21.15.161:0/1339548149 wait complete. 2024-03-28T23:32:37.151 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-03-28T23:32:35.343697+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239916,"kb_used_data":7084,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754734804,"statfs":{"total":773094113280,"available":772848439296,"internally_reserved":0,"allocated":7254016,"data_stored":4505752,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.327546"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.027696+0000","last_change":"2024-03-28T23:32:33.046706+0000","last_active":"2024-03-28T23:32:34.027696+0000","last_peered":"2024-03-28T23:32:34.027696+0000","last_clean":"2024-03-28T23:32:34.027696+0000","last_became_active":"2024-03-28T23:32:33.046222+0000","last_became_peered":"2024-03-28T23:32:33.046222+0000","last_unstale":"2024-03-28T23:32:34.027696+0000","last_undegraded":"2024-03-28T23:32:34.027696+0000","last_fullsized":"2024-03-28T23:32:34.027696+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:02:45.190566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028766+0000","last_change":"2024-03-28T23:32:33.051174+0000","last_active":"2024-03-28T23:32:34.028766+0000","last_peered":"2024-03-28T23:32:34.028766+0000","last_clean":"2024-03-28T23:32:34.028766+0000","last_became_active":"2024-03-28T23:32:33.050345+0000","last_became_peered":"2024-03-28T23:32:33.050345+0000","last_unstale":"2024-03-28T23:32:34.028766+0000","last_undegraded":"2024-03-28T23:32:34.028766+0000","last_fullsized":"2024-03-28T23:32:34.028766+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:41:39.657731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057623+0000","last_change":"2024-03-28T23:32:33.048785+0000","last_active":"2024-03-28T23:32:34.057623+0000","last_peered":"2024-03-28T23:32:34.057623+0000","last_clean":"2024-03-28T23:32:34.057623+0000","last_became_active":"2024-03-28T23:32:33.048519+0000","last_became_peered":"2024-03-28T23:32:33.048519+0000","last_unstale":"2024-03-28T23:32:34.057623+0000","last_undegraded":"2024-03-28T23:32:34.057623+0000","last_fullsized":"2024-03-28T23:32:34.057623+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:21.464120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026644+0000","last_change":"2024-03-28T23:32:33.033535+0000","last_active":"2024-03-28T23:32:34.026644+0000","last_peered":"2024-03-28T23:32:34.026644+0000","last_clean":"2024-03-28T23:32:34.026644+0000","last_became_active":"2024-03-28T23:32:33.033219+0000","last_became_peered":"2024-03-28T23:32:33.033219+0000","last_unstale":"2024-03-28T23:32:34.026644+0000","last_undegraded":"2024-03-28T23:32:34.026644+0000","last_fullsized":"2024-03-28T23:32:34.026644+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:39:39.113457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.030899+0000","last_change":"2024-03-28T23:32:33.042972+0000","last_active":"2024-03-28T23:32:34.030899+0000","last_peered":"2024-03-28T23:32:34.030899+0000","last_clean":"2024-03-28T23:32:34.030899+0000","last_became_active":"2024-03-28T23:32:33.042665+0000","last_became_peered":"2024-03-28T23:32:33.042665+0000","last_unstale":"2024-03-28T23:32:34.030899+0000","last_undegraded":"2024-03-28T23:32:34.030899+0000","last_fullsized":"2024-03-28T23:32:34.030899+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:41:50.190883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057842+0000","last_change":"2024-03-28T23:32:33.030549+0000","last_active":"2024-03-28T23:32:34.057842+0000","last_peered":"2024-03-28T23:32:34.057842+0000","last_clean":"2024-03-28T23:32:34.057842+0000","last_became_active":"2024-03-28T23:32:33.030299+0000","last_became_peered":"2024-03-28T23:32:33.030299+0000","last_unstale":"2024-03-28T23:32:34.057842+0000","last_undegraded":"2024-03-28T23:32:34.057842+0000","last_fullsized":"2024-03-28T23:32:34.057842+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:35.055035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.027067+0000","last_change":"2024-03-28T23:32:33.044477+0000","last_active":"2024-03-28T23:32:34.027067+0000","last_peered":"2024-03-28T23:32:34.027067+0000","last_clean":"2024-03-28T23:32:34.027067+0000","last_became_active":"2024-03-28T23:32:33.043960+0000","last_became_peered":"2024-03-28T23:32:33.043960+0000","last_unstale":"2024-03-28T23:32:34.027067+0000","last_undegraded":"2024-03-28T23:32:34.027067+0000","last_fullsized":"2024-03-28T23:32:34.027067+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:21.464120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.030686+0000","last_change":"2024-03-28T23:32:33.052803+0000","last_active":"2024-03-28T23:32:34.030686+0000","last_peered":"2024-03-28T23:32:34.030686+0000","last_clean":"2024-03-28T23:32:34.030686+0000","last_became_active":"2024-03-28T23:32:33.052524+0000","last_became_peered":"2024-03-28T23:32:33.052524+0000","last_unstale":"2024-03-28T23:32:34.030686+0000","last_undegraded":"2024-03-28T23:32:34.030686+0000","last_fullsized":"2024-03-28T23:32:34.030686+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:19:09.500642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.058056+0000","last_change":"2024-03-28T23:32:33.042587+0000","last_active":"2024-03-28T23:32:34.058056+0000","last_peered":"2024-03-28T23:32:34.058056+0000","last_clean":"2024-03-28T23:32:34.058056+0000","last_became_active":"2024-03-28T23:32:33.037002+0000","last_became_peered":"2024-03-28T23:32:33.037002+0000","last_unstale":"2024-03-28T23:32:34.058056+0000","last_undegraded":"2024-03-28T23:32:34.058056+0000","last_fullsized":"2024-03-28T23:32:34.058056+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:36:03.951054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.025842+0000","last_change":"2024-03-28T23:32:33.039941+0000","last_active":"2024-03-28T23:32:34.025842+0000","last_peered":"2024-03-28T23:32:34.025842+0000","last_clean":"2024-03-28T23:32:34.025842+0000","last_became_active":"2024-03-28T23:32:33.039656+0000","last_became_peered":"2024-03-28T23:32:33.039656+0000","last_unstale":"2024-03-28T23:32:34.025842+0000","last_undegraded":"2024-03-28T23:32:34.025842+0000","last_fullsized":"2024-03-28T23:32:34.025842+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:16.850667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028608+0000","last_change":"2024-03-28T23:32:33.052329+0000","last_active":"2024-03-28T23:32:34.028608+0000","last_peered":"2024-03-28T23:32:34.028608+0000","last_clean":"2024-03-28T23:32:34.028608+0000","last_became_active":"2024-03-28T23:32:33.051833+0000","last_became_peered":"2024-03-28T23:32:33.051833+0000","last_unstale":"2024-03-28T23:32:34.028608+0000","last_undegraded":"2024-03-28T23:32:34.028608+0000","last_fullsized":"2024-03-28T23:32:34.028608+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:05.929331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026794+0000","last_change":"2024-03-28T23:32:33.042337+0000","last_active":"2024-03-28T23:32:34.026794+0000","last_peered":"2024-03-28T23:32:34.026794+0000","last_clean":"2024-03-28T23:32:34.026794+0000","last_became_active":"2024-03-28T23:32:33.042062+0000","last_became_peered":"2024-03-28T23:32:33.042062+0000","last_unstale":"2024-03-28T23:32:34.026794+0000","last_undegraded":"2024-03-28T23:32:34.026794+0000","last_fullsized":"2024-03-28T23:32:34.026794+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:13:34.921740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.030802+0000","last_change":"2024-03-28T23:32:33.046685+0000","last_active":"2024-03-28T23:32:34.030802+0000","last_peered":"2024-03-28T23:32:34.030802+0000","last_clean":"2024-03-28T23:32:34.030802+0000","last_became_active":"2024-03-28T23:32:33.046206+0000","last_became_peered":"2024-03-28T23:32:33.046206+0000","last_unstale":"2024-03-28T23:32:34.030802+0000","last_undegraded":"2024-03-28T23:32:34.030802+0000","last_fullsized":"2024-03-28T23:32:34.030802+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:43:22.001147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057854+0000","last_change":"2024-03-28T23:32:33.029880+0000","last_active":"2024-03-28T23:32:34.057854+0000","last_peered":"2024-03-28T23:32:34.057854+0000","last_clean":"2024-03-28T23:32:34.057854+0000","last_became_active":"2024-03-28T23:32:33.029582+0000","last_became_peered":"2024-03-28T23:32:33.029582+0000","last_unstale":"2024-03-28T23:32:34.057854+0000","last_undegraded":"2024-03-28T23:32:34.057854+0000","last_fullsized":"2024-03-28T23:32:34.057854+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:38:39.283932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028335+0000","last_change":"2024-03-28T23:32:33.040102+0000","last_active":"2024-03-28T23:32:34.028335+0000","last_peered":"2024-03-28T23:32:34.028335+0000","last_clean":"2024-03-28T23:32:34.028335+0000","last_became_active":"2024-03-28T23:32:33.039660+0000","last_became_peered":"2024-03-28T23:32:33.039660+0000","last_unstale":"2024-03-28T23:32:34.028335+0000","last_undegraded":"2024-03-28T23:32:34.028335+0000","last_fullsized":"2024-03-28T23:32:34.028335+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:35.055035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026721+0000","last_change":"2024-03-28T23:32:33.035442+0000","last_active":"2024-03-28T23:32:34.026721+0000","last_peered":"2024-03-28T23:32:34.026721+0000","last_clean":"2024-03-28T23:32:34.026721+0000","last_became_active":"2024-03-28T23:32:33.035154+0000","last_became_peered":"2024-03-28T23:32:33.035154+0000","last_unstale":"2024-03-28T23:32:34.026721+0000","last_undegraded":"2024-03-28T23:32:34.026721+0000","last_fullsized":"2024-03-28T23:32:34.026721+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:34:22.775718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.031125+0000","last_change":"2024-03-28T23:32:33.047782+0000","last_active":"2024-03-28T23:32:34.031125+0000","last_peered":"2024-03-28T23:32:34.031125+0000","last_clean":"2024-03-28T23:32:34.031125+0000","last_became_active":"2024-03-28T23:32:33.047254+0000","last_became_peered":"2024-03-28T23:32:33.047254+0000","last_unstale":"2024-03-28T23:32:34.031125+0000","last_undegraded":"2024-03-28T23:32:34.031125+0000","last_fullsized":"2024-03-28T23:32:34.031125+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:19.004555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057391+0000","last_change":"2024-03-28T23:32:33.037321+0000","last_active":"2024-03-28T23:32:34.057391+0000","last_peered":"2024-03-28T23:32:34.057391+0000","last_clean":"2024-03-28T23:32:34.057391+0000","last_became_active":"2024-03-28T23:32:33.036961+0000","last_became_peered":"2024-03-28T23:32:33.036961+0000","last_unstale":"2024-03-28T23:32:34.057391+0000","last_undegraded":"2024-03-28T23:32:34.057391+0000","last_fullsized":"2024-03-28T23:32:34.057391+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:25.849919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.058056+0000","last_change":"2024-03-28T23:32:33.053071+0000","last_active":"2024-03-28T23:32:34.058056+0000","last_peered":"2024-03-28T23:32:34.058056+0000","last_clean":"2024-03-28T23:32:34.058056+0000","last_became_active":"2024-03-28T23:32:33.052784+0000","last_became_peered":"2024-03-28T23:32:33.052784+0000","last_unstale":"2024-03-28T23:32:34.058056+0000","last_undegraded":"2024-03-28T23:32:34.058056+0000","last_fullsized":"2024-03-28T23:32:34.058056+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:58:48.231248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.030912+0000","last_change":"2024-03-28T23:32:33.052720+0000","last_active":"2024-03-28T23:32:34.030912+0000","last_peered":"2024-03-28T23:32:34.030912+0000","last_clean":"2024-03-28T23:32:34.030912+0000","last_became_active":"2024-03-28T23:32:33.052460+0000","last_became_peered":"2024-03-28T23:32:33.052460+0000","last_unstale":"2024-03-28T23:32:34.030912+0000","last_undegraded":"2024-03-28T23:32:34.030912+0000","last_fullsized":"2024-03-28T23:32:34.030912+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:30.891315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057833+0000","last_change":"2024-03-28T23:32:33.035666+0000","last_active":"2024-03-28T23:32:34.057833+0000","last_peered":"2024-03-28T23:32:34.057833+0000","last_clean":"2024-03-28T23:32:34.057833+0000","last_became_active":"2024-03-28T23:32:33.035403+0000","last_became_peered":"2024-03-28T23:32:33.035403+0000","last_unstale":"2024-03-28T23:32:34.057833+0000","last_undegraded":"2024-03-28T23:32:34.057833+0000","last_fullsized":"2024-03-28T23:32:34.057833+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:40:51.269760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.030982+0000","last_change":"2024-03-28T23:32:33.047992+0000","last_active":"2024-03-28T23:32:34.030982+0000","last_peered":"2024-03-28T23:32:34.030982+0000","last_clean":"2024-03-28T23:32:34.030982+0000","last_became_active":"2024-03-28T23:32:33.047058+0000","last_became_peered":"2024-03-28T23:32:33.047058+0000","last_unstale":"2024-03-28T23:32:34.030982+0000","last_undegraded":"2024-03-28T23:32:34.030982+0000","last_fullsized":"2024-03-28T23:32:34.030982+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:35:30.886935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057683+0000","last_change":"2024-03-28T23:32:33.029536+0000","last_active":"2024-03-28T23:32:34.057683+0000","last_peered":"2024-03-28T23:32:34.057683+0000","last_clean":"2024-03-28T23:32:34.057683+0000","last_became_active":"2024-03-28T23:32:33.029265+0000","last_became_peered":"2024-03-28T23:32:33.029265+0000","last_unstale":"2024-03-28T23:32:34.057683+0000","last_undegraded":"2024-03-28T23:32:34.057683+0000","last_fullsized":"2024-03-28T23:32:34.057683+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:51:11.447507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028300+0000","last_change":"2024-03-28T23:32:33.041328+0000","last_active":"2024-03-28T23:32:34.028300+0000","last_peered":"2024-03-28T23:32:34.028300+0000","last_clean":"2024-03-28T23:32:34.028300+0000","last_became_active":"2024-03-28T23:32:33.041071+0000","last_became_peered":"2024-03-28T23:32:33.041071+0000","last_unstale":"2024-03-28T23:32:34.028300+0000","last_undegraded":"2024-03-28T23:32:34.028300+0000","last_fullsized":"2024-03-28T23:32:34.028300+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:00.358180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028708+0000","last_change":"2024-03-28T23:32:33.043840+0000","last_active":"2024-03-28T23:32:34.028708+0000","last_peered":"2024-03-28T23:32:34.028708+0000","last_clean":"2024-03-28T23:32:34.028708+0000","last_became_active":"2024-03-28T23:32:33.043569+0000","last_became_peered":"2024-03-28T23:32:33.043569+0000","last_unstale":"2024-03-28T23:32:34.028708+0000","last_undegraded":"2024-03-28T23:32:34.028708+0000","last_fullsized":"2024-03-28T23:32:34.028708+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:28.463493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.025993+0000","last_change":"2024-03-28T23:32:33.047594+0000","last_active":"2024-03-28T23:32:34.025993+0000","last_peered":"2024-03-28T23:32:34.025993+0000","last_clean":"2024-03-28T23:32:34.025993+0000","last_became_active":"2024-03-28T23:32:33.047031+0000","last_became_peered":"2024-03-28T23:32:33.047031+0000","last_unstale":"2024-03-28T23:32:34.025993+0000","last_undegraded":"2024-03-28T23:32:34.025993+0000","last_fullsized":"2024-03-28T23:32:34.025993+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:30:12.769818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.027395+0000","last_change":"2024-03-28T23:32:33.043396+0000","last_active":"2024-03-28T23:32:34.027395+0000","last_peered":"2024-03-28T23:32:34.027395+0000","last_clean":"2024-03-28T23:32:34.027395+0000","last_became_active":"2024-03-28T23:32:33.043123+0000","last_became_peered":"2024-03-28T23:32:33.043123+0000","last_unstale":"2024-03-28T23:32:34.027395+0000","last_undegraded":"2024-03-28T23:32:34.027395+0000","last_fullsized":"2024-03-28T23:32:34.027395+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:28:36.535909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.030962+0000","last_change":"2024-03-28T23:32:33.042288+0000","last_active":"2024-03-28T23:32:34.030962+0000","last_peered":"2024-03-28T23:32:34.030962+0000","last_clean":"2024-03-28T23:32:34.030962+0000","last_became_active":"2024-03-28T23:32:33.042007+0000","last_became_peered":"2024-03-28T23:32:33.042007+0000","last_unstale":"2024-03-28T23:32:34.030962+0000","last_undegraded":"2024-03-28T23:32:34.030962+0000","last_fullsized":"2024-03-28T23:32:34.030962+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:16:12.634582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028532+0000","last_change":"2024-03-28T23:32:33.041336+0000","last_active":"2024-03-28T23:32:34.028532+0000","last_peered":"2024-03-28T23:32:34.028532+0000","last_clean":"2024-03-28T23:32:34.028532+0000","last_became_active":"2024-03-28T23:32:33.040865+0000","last_became_peered":"2024-03-28T23:32:33.040865+0000","last_unstale":"2024-03-28T23:32:34.028532+0000","last_undegraded":"2024-03-28T23:32:34.028532+0000","last_fullsized":"2024-03-28T23:32:34.028532+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:44:58.784172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026760+0000","last_change":"2024-03-28T23:32:33.049222+0000","last_active":"2024-03-28T23:32:34.026760+0000","last_peered":"2024-03-28T23:32:34.026760+0000","last_clean":"2024-03-28T23:32:34.026760+0000","last_became_active":"2024-03-28T23:32:33.048933+0000","last_became_peered":"2024-03-28T23:32:33.048933+0000","last_unstale":"2024-03-28T23:32:34.026760+0000","last_undegraded":"2024-03-28T23:32:34.026760+0000","last_fullsized":"2024-03-28T23:32:34.026760+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:17:45.545141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057563+0000","last_change":"2024-03-28T23:32:33.047977+0000","last_active":"2024-03-28T23:32:34.057563+0000","last_peered":"2024-03-28T23:32:34.057563+0000","last_clean":"2024-03-28T23:32:34.057563+0000","last_became_active":"2024-03-28T23:32:33.047699+0000","last_became_peered":"2024-03-28T23:32:33.047699+0000","last_unstale":"2024-03-28T23:32:34.057563+0000","last_undegraded":"2024-03-28T23:32:34.057563+0000","last_fullsized":"2024-03-28T23:32:34.057563+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:38:51.513907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026882+0000","last_change":"2024-03-28T23:32:33.036527+0000","last_active":"2024-03-28T23:32:34.026882+0000","last_peered":"2024-03-28T23:32:34.026882+0000","last_clean":"2024-03-28T23:32:34.026882+0000","last_became_active":"2024-03-28T23:32:33.036269+0000","last_became_peered":"2024-03-28T23:32:33.036269+0000","last_unstale":"2024-03-28T23:32:34.026882+0000","last_undegraded":"2024-03-28T23:32:34.026882+0000","last_fullsized":"2024-03-28T23:32:34.026882+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:43:35.946992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057140+0000","last_change":"2024-03-28T23:32:33.038016+0000","last_active":"2024-03-28T23:32:34.057140+0000","last_peered":"2024-03-28T23:32:34.057140+0000","last_clean":"2024-03-28T23:32:34.057140+0000","last_became_active":"2024-03-28T23:32:33.037754+0000","last_became_peered":"2024-03-28T23:32:33.037754+0000","last_unstale":"2024-03-28T23:32:34.057140+0000","last_undegraded":"2024-03-28T23:32:34.057140+0000","last_fullsized":"2024-03-28T23:32:34.057140+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:21.878156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"72'4","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.031109+0000","last_change":"2024-03-28T23:32:32.040468+0000","last_active":"2024-03-28T23:32:34.031109+0000","last_peered":"2024-03-28T23:32:34.031109+0000","last_clean":"2024-03-28T23:32:34.031109+0000","last_became_active":"2024-03-28T23:32:32.039342+0000","last_became_peered":"2024-03-28T23:32:32.039342+0000","last_unstale":"2024-03-28T23:32:34.031109+0000","last_undegraded":"2024-03-28T23:32:34.031109+0000","last_fullsized":"2024-03-28T23:32:34.031109+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:05.786248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028348+0000","last_change":"2024-03-28T23:32:33.039778+0000","last_active":"2024-03-28T23:32:34.028348+0000","last_peered":"2024-03-28T23:32:34.028348+0000","last_clean":"2024-03-28T23:32:34.028348+0000","last_became_active":"2024-03-28T23:32:33.039506+0000","last_became_peered":"2024-03-28T23:32:33.039506+0000","last_unstale":"2024-03-28T23:32:34.028348+0000","last_undegraded":"2024-03-28T23:32:34.028348+0000","last_fullsized":"2024-03-28T23:32:34.028348+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:38:39.283932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"72'1","reported_seq":15,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026243+0000","last_change":"2024-03-28T23:32:32.041093+0000","last_active":"2024-03-28T23:32:34.026243+0000","last_peered":"2024-03-28T23:32:34.026243+0000","last_clean":"2024-03-28T23:32:34.026243+0000","last_became_active":"2024-03-28T23:32:32.040827+0000","last_became_peered":"2024-03-28T23:32:32.040827+0000","last_unstale":"2024-03-28T23:32:34.026243+0000","last_undegraded":"2024-03-28T23:32:34.026243+0000","last_fullsized":"2024-03-28T23:32:34.026243+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:55.544945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057929+0000","last_change":"2024-03-28T23:32:33.044887+0000","last_active":"2024-03-28T23:32:34.057929+0000","last_peered":"2024-03-28T23:32:34.057929+0000","last_clean":"2024-03-28T23:32:34.057929+0000","last_became_active":"2024-03-28T23:32:33.042005+0000","last_became_peered":"2024-03-28T23:32:33.042005+0000","last_unstale":"2024-03-28T23:32:34.057929+0000","last_undegraded":"2024-03-28T23:32:34.057929+0000","last_fullsized":"2024-03-28T23:32:34.057929+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:41.257241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"72'2","reported_seq":16,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.027508+0000","last_change":"2024-03-28T23:32:32.040527+0000","last_active":"2024-03-28T23:32:34.027508+0000","last_peered":"2024-03-28T23:32:34.027508+0000","last_clean":"2024-03-28T23:32:34.027508+0000","last_became_active":"2024-03-28T23:32:32.040142+0000","last_became_peered":"2024-03-28T23:32:32.040142+0000","last_unstale":"2024-03-28T23:32:34.027508+0000","last_undegraded":"2024-03-28T23:32:34.027508+0000","last_fullsized":"2024-03-28T23:32:34.027508+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:13:15.856750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057887+0000","last_change":"2024-03-28T23:32:33.040988+0000","last_active":"2024-03-28T23:32:34.057887+0000","last_peered":"2024-03-28T23:32:34.057887+0000","last_clean":"2024-03-28T23:32:34.057887+0000","last_became_active":"2024-03-28T23:32:33.036045+0000","last_became_peered":"2024-03-28T23:32:33.036045+0000","last_unstale":"2024-03-28T23:32:34.057887+0000","last_undegraded":"2024-03-28T23:32:34.057887+0000","last_fullsized":"2024-03-28T23:32:34.057887+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:10:39.981362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"72'2","reported_seq":16,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026203+0000","last_change":"2024-03-28T23:32:32.036856+0000","last_active":"2024-03-28T23:32:34.026203+0000","last_peered":"2024-03-28T23:32:34.026203+0000","last_clean":"2024-03-28T23:32:34.026203+0000","last_became_active":"2024-03-28T23:32:32.036565+0000","last_became_peered":"2024-03-28T23:32:32.036565+0000","last_unstale":"2024-03-28T23:32:34.026203+0000","last_undegraded":"2024-03-28T23:32:34.026203+0000","last_fullsized":"2024-03-28T23:32:34.026203+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:11:01.106436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028510+0000","last_change":"2024-03-28T23:32:33.047911+0000","last_active":"2024-03-28T23:32:34.028510+0000","last_peered":"2024-03-28T23:32:34.028510+0000","last_clean":"2024-03-28T23:32:34.028510+0000","last_became_active":"2024-03-28T23:32:33.047616+0000","last_became_peered":"2024-03-28T23:32:33.047616+0000","last_unstale":"2024-03-28T23:32:34.028510+0000","last_undegraded":"2024-03-28T23:32:34.028510+0000","last_fullsized":"2024-03-28T23:32:34.028510+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:58:48.231248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"72'1","reported_seq":15,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057966+0000","last_change":"2024-03-28T23:32:32.036829+0000","last_active":"2024-03-28T23:32:34.057966+0000","last_peered":"2024-03-28T23:32:34.057966+0000","last_clean":"2024-03-28T23:32:34.057966+0000","last_became_active":"2024-03-28T23:32:32.036571+0000","last_became_peered":"2024-03-28T23:32:32.036571+0000","last_unstale":"2024-03-28T23:32:34.057966+0000","last_undegraded":"2024-03-28T23:32:34.057966+0000","last_fullsized":"2024-03-28T23:32:34.057966+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:32:03.789593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028687+0000","last_change":"2024-03-28T23:32:33.052330+0000","last_active":"2024-03-28T23:32:34.028687+0000","last_peered":"2024-03-28T23:32:34.028687+0000","last_clean":"2024-03-28T23:32:34.028687+0000","last_became_active":"2024-03-28T23:32:33.051828+0000","last_became_peered":"2024-03-28T23:32:33.051828+0000","last_unstale":"2024-03-28T23:32:34.028687+0000","last_undegraded":"2024-03-28T23:32:34.028687+0000","last_fullsized":"2024-03-28T23:32:34.028687+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:40:53.134608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"72'1","reported_seq":15,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.025535+0000","last_change":"2024-03-28T23:32:32.036739+0000","last_active":"2024-03-28T23:32:34.025535+0000","last_peered":"2024-03-28T23:32:34.025535+0000","last_clean":"2024-03-28T23:32:34.025535+0000","last_became_active":"2024-03-28T23:32:32.036471+0000","last_became_peered":"2024-03-28T23:32:32.036471+0000","last_unstale":"2024-03-28T23:32:34.025535+0000","last_undegraded":"2024-03-28T23:32:34.025535+0000","last_fullsized":"2024-03-28T23:32:34.025535+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:48.704351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.058076+0000","last_change":"2024-03-28T23:32:33.034126+0000","last_active":"2024-03-28T23:32:34.058076+0000","last_peered":"2024-03-28T23:32:34.058076+0000","last_clean":"2024-03-28T23:32:34.058076+0000","last_became_active":"2024-03-28T23:32:33.033703+0000","last_became_peered":"2024-03-28T23:32:33.033703+0000","last_unstale":"2024-03-28T23:32:34.058076+0000","last_undegraded":"2024-03-28T23:32:34.058076+0000","last_fullsized":"2024-03-28T23:32:34.058076+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:38.859031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028431+0000","last_change":"2024-03-28T23:32:32.039047+0000","last_active":"2024-03-28T23:32:34.028431+0000","last_peered":"2024-03-28T23:32:34.028431+0000","last_clean":"2024-03-28T23:32:34.028431+0000","last_became_active":"2024-03-28T23:32:32.038780+0000","last_became_peered":"2024-03-28T23:32:32.038780+0000","last_unstale":"2024-03-28T23:32:34.028431+0000","last_undegraded":"2024-03-28T23:32:34.028431+0000","last_fullsized":"2024-03-28T23:32:34.028431+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:13:15.856750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026038+0000","last_change":"2024-03-28T23:32:33.051453+0000","last_active":"2024-03-28T23:32:34.026038+0000","last_peered":"2024-03-28T23:32:34.026038+0000","last_clean":"2024-03-28T23:32:34.026038+0000","last_became_active":"2024-03-28T23:32:33.051155+0000","last_became_peered":"2024-03-28T23:32:33.051155+0000","last_unstale":"2024-03-28T23:32:34.026038+0000","last_undegraded":"2024-03-28T23:32:34.026038+0000","last_fullsized":"2024-03-28T23:32:34.026038+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:36:03.951054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.030722+0000","last_change":"2024-03-28T23:32:32.036979+0000","last_active":"2024-03-28T23:32:34.030722+0000","last_peered":"2024-03-28T23:32:34.030722+0000","last_clean":"2024-03-28T23:32:34.030722+0000","last_became_active":"2024-03-28T23:32:32.036710+0000","last_became_peered":"2024-03-28T23:32:32.036710+0000","last_unstale":"2024-03-28T23:32:34.030722+0000","last_undegraded":"2024-03-28T23:32:34.030722+0000","last_fullsized":"2024-03-28T23:32:34.030722+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:46.072852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028295+0000","last_change":"2024-03-28T23:32:33.041923+0000","last_active":"2024-03-28T23:32:34.028295+0000","last_peered":"2024-03-28T23:32:34.028295+0000","last_clean":"2024-03-28T23:32:34.028295+0000","last_became_active":"2024-03-28T23:32:33.041631+0000","last_became_peered":"2024-03-28T23:32:33.041631+0000","last_unstale":"2024-03-28T23:32:34.028295+0000","last_undegraded":"2024-03-28T23:32:34.028295+0000","last_fullsized":"2024-03-28T23:32:34.028295+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:32:08.473276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.027796+0000","last_change":"2024-03-28T23:32:32.041563+0000","last_active":"2024-03-28T23:32:34.027796+0000","last_peered":"2024-03-28T23:32:34.027796+0000","last_clean":"2024-03-28T23:32:34.027796+0000","last_became_active":"2024-03-28T23:32:32.040424+0000","last_became_peered":"2024-03-28T23:32:32.040424+0000","last_unstale":"2024-03-28T23:32:34.027796+0000","last_undegraded":"2024-03-28T23:32:34.027796+0000","last_fullsized":"2024-03-28T23:32:34.027796+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:37:25.645113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.025772+0000","last_change":"2024-03-28T23:32:33.029986+0000","last_active":"2024-03-28T23:32:34.025772+0000","last_peered":"2024-03-28T23:32:34.025772+0000","last_clean":"2024-03-28T23:32:34.025772+0000","last_became_active":"2024-03-28T23:32:33.029413+0000","last_became_peered":"2024-03-28T23:32:33.029413+0000","last_unstale":"2024-03-28T23:32:34.025772+0000","last_undegraded":"2024-03-28T23:32:34.025772+0000","last_fullsized":"2024-03-28T23:32:34.025772+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:37:26.652511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026096+0000","last_change":"2024-03-28T23:32:32.037039+0000","last_active":"2024-03-28T23:32:34.026096+0000","last_peered":"2024-03-28T23:32:34.026096+0000","last_clean":"2024-03-28T23:32:34.026096+0000","last_became_active":"2024-03-28T23:32:32.036687+0000","last_became_peered":"2024-03-28T23:32:32.036687+0000","last_unstale":"2024-03-28T23:32:34.026096+0000","last_undegraded":"2024-03-28T23:32:34.026096+0000","last_fullsized":"2024-03-28T23:32:34.026096+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:27:15.824876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.058028+0000","last_change":"2024-03-28T23:32:33.035809+0000","last_active":"2024-03-28T23:32:34.058028+0000","last_peered":"2024-03-28T23:32:34.058028+0000","last_clean":"2024-03-28T23:32:34.058028+0000","last_became_active":"2024-03-28T23:32:33.035563+0000","last_became_peered":"2024-03-28T23:32:33.035563+0000","last_unstale":"2024-03-28T23:32:34.058028+0000","last_undegraded":"2024-03-28T23:32:34.058028+0000","last_fullsized":"2024-03-28T23:32:34.058028+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:54.194453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"72'3","reported_seq":17,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.031111+0000","last_change":"2024-03-28T23:32:32.038230+0000","last_active":"2024-03-28T23:32:34.031111+0000","last_peered":"2024-03-28T23:32:34.031111+0000","last_clean":"2024-03-28T23:32:34.031111+0000","last_became_active":"2024-03-28T23:32:32.037681+0000","last_became_peered":"2024-03-28T23:32:32.037681+0000","last_unstale":"2024-03-28T23:32:34.031111+0000","last_undegraded":"2024-03-28T23:32:34.031111+0000","last_fullsized":"2024-03-28T23:32:34.031111+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:36:34.752866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057265+0000","last_change":"2024-03-28T23:32:33.049499+0000","last_active":"2024-03-28T23:32:34.057265+0000","last_peered":"2024-03-28T23:32:34.057265+0000","last_clean":"2024-03-28T23:32:34.057265+0000","last_became_active":"2024-03-28T23:32:33.049233+0000","last_became_peered":"2024-03-28T23:32:33.049233+0000","last_unstale":"2024-03-28T23:32:34.057265+0000","last_undegraded":"2024-03-28T23:32:34.057265+0000","last_fullsized":"2024-03-28T23:32:34.057265+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:39:02.824323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"72'6","reported_seq":20,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026323+0000","last_change":"2024-03-28T23:32:32.036835+0000","last_active":"2024-03-28T23:32:34.026323+0000","last_peered":"2024-03-28T23:32:34.026323+0000","last_clean":"2024-03-28T23:32:34.026323+0000","last_became_active":"2024-03-28T23:32:32.036557+0000","last_became_peered":"2024-03-28T23:32:32.036557+0000","last_unstale":"2024-03-28T23:32:34.026323+0000","last_undegraded":"2024-03-28T23:32:34.026323+0000","last_fullsized":"2024-03-28T23:32:34.026323+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:30.992807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026952+0000","last_change":"2024-03-28T23:32:33.033045+0000","last_active":"2024-03-28T23:32:34.026952+0000","last_peered":"2024-03-28T23:32:34.026952+0000","last_clean":"2024-03-28T23:32:34.026952+0000","last_became_active":"2024-03-28T23:32:33.032771+0000","last_became_peered":"2024-03-28T23:32:33.032771+0000","last_unstale":"2024-03-28T23:32:34.026952+0000","last_undegraded":"2024-03-28T23:32:34.026952+0000","last_fullsized":"2024-03-28T23:32:34.026952+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:43.341877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"72'2","reported_seq":16,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.025905+0000","last_change":"2024-03-28T23:32:32.032053+0000","last_active":"2024-03-28T23:32:34.025905+0000","last_peered":"2024-03-28T23:32:34.025905+0000","last_clean":"2024-03-28T23:32:34.025905+0000","last_became_active":"2024-03-28T23:32:32.031741+0000","last_became_peered":"2024-03-28T23:32:32.031741+0000","last_unstale":"2024-03-28T23:32:34.025905+0000","last_undegraded":"2024-03-28T23:32:34.025905+0000","last_fullsized":"2024-03-28T23:32:34.025905+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:20.456721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026720+0000","last_change":"2024-03-28T23:32:33.039022+0000","last_active":"2024-03-28T23:32:34.026720+0000","last_peered":"2024-03-28T23:32:34.026720+0000","last_clean":"2024-03-28T23:32:34.026720+0000","last_became_active":"2024-03-28T23:32:33.038508+0000","last_became_peered":"2024-03-28T23:32:33.038508+0000","last_unstale":"2024-03-28T23:32:34.026720+0000","last_undegraded":"2024-03-28T23:32:34.026720+0000","last_fullsized":"2024-03-28T23:32:34.026720+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:25:45.189859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026863+0000","last_change":"2024-03-28T23:32:32.041042+0000","last_active":"2024-03-28T23:32:34.026863+0000","last_peered":"2024-03-28T23:32:34.026863+0000","last_clean":"2024-03-28T23:32:34.026863+0000","last_became_active":"2024-03-28T23:32:32.040686+0000","last_became_peered":"2024-03-28T23:32:32.040686+0000","last_unstale":"2024-03-28T23:32:34.026863+0000","last_undegraded":"2024-03-28T23:32:34.026863+0000","last_fullsized":"2024-03-28T23:32:34.026863+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:46.343619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.025919+0000","last_change":"2024-03-28T23:32:33.034262+0000","last_active":"2024-03-28T23:32:34.025919+0000","last_peered":"2024-03-28T23:32:34.025919+0000","last_clean":"2024-03-28T23:32:34.025919+0000","last_became_active":"2024-03-28T23:32:33.034003+0000","last_became_peered":"2024-03-28T23:32:33.034003+0000","last_unstale":"2024-03-28T23:32:34.025919+0000","last_undegraded":"2024-03-28T23:32:34.025919+0000","last_fullsized":"2024-03-28T23:32:34.025919+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:41.257241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.030694+0000","last_change":"2024-03-28T23:32:32.039031+0000","last_active":"2024-03-28T23:32:34.030694+0000","last_peered":"2024-03-28T23:32:34.030694+0000","last_clean":"2024-03-28T23:32:34.030694+0000","last_became_active":"2024-03-28T23:32:32.038766+0000","last_became_peered":"2024-03-28T23:32:32.038766+0000","last_unstale":"2024-03-28T23:32:34.030694+0000","last_undegraded":"2024-03-28T23:32:34.030694+0000","last_fullsized":"2024-03-28T23:32:34.030694+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:13.872794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026193+0000","last_change":"2024-03-28T23:32:33.043171+0000","last_active":"2024-03-28T23:32:34.026193+0000","last_peered":"2024-03-28T23:32:34.026193+0000","last_clean":"2024-03-28T23:32:34.026193+0000","last_became_active":"2024-03-28T23:32:33.042821+0000","last_became_peered":"2024-03-28T23:32:33.042821+0000","last_unstale":"2024-03-28T23:32:34.026193+0000","last_undegraded":"2024-03-28T23:32:34.026193+0000","last_fullsized":"2024-03-28T23:32:34.026193+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:21.878156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"72'8","reported_seq":22,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028646+0000","last_change":"2024-03-28T23:32:32.040166+0000","last_active":"2024-03-28T23:32:34.028646+0000","last_peered":"2024-03-28T23:32:34.028646+0000","last_clean":"2024-03-28T23:32:34.028646+0000","last_became_active":"2024-03-28T23:32:32.039560+0000","last_became_peered":"2024-03-28T23:32:32.039560+0000","last_unstale":"2024-03-28T23:32:34.028646+0000","last_undegraded":"2024-03-28T23:32:34.028646+0000","last_fullsized":"2024-03-28T23:32:34.028646+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:55.544945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057158+0000","last_change":"2024-03-28T23:32:33.043192+0000","last_active":"2024-03-28T23:32:34.057158+0000","last_peered":"2024-03-28T23:32:34.057158+0000","last_clean":"2024-03-28T23:32:34.057158+0000","last_became_active":"2024-03-28T23:32:33.042894+0000","last_became_peered":"2024-03-28T23:32:33.042894+0000","last_unstale":"2024-03-28T23:32:34.057158+0000","last_undegraded":"2024-03-28T23:32:34.057158+0000","last_fullsized":"2024-03-28T23:32:34.057158+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:32:08.473276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"72'1","reported_seq":15,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.027885+0000","last_change":"2024-03-28T23:32:32.041892+0000","last_active":"2024-03-28T23:32:34.027885+0000","last_peered":"2024-03-28T23:32:34.027885+0000","last_clean":"2024-03-28T23:32:34.027885+0000","last_became_active":"2024-03-28T23:32:32.041533+0000","last_became_peered":"2024-03-28T23:32:32.041533+0000","last_unstale":"2024-03-28T23:32:34.027885+0000","last_undegraded":"2024-03-28T23:32:34.027885+0000","last_fullsized":"2024-03-28T23:32:34.027885+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:15.843269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.030716+0000","last_change":"2024-03-28T23:32:33.046899+0000","last_active":"2024-03-28T23:32:34.030716+0000","last_peered":"2024-03-28T23:32:34.030716+0000","last_clean":"2024-03-28T23:32:34.030716+0000","last_became_active":"2024-03-28T23:32:33.046579+0000","last_became_peered":"2024-03-28T23:32:33.046579+0000","last_unstale":"2024-03-28T23:32:34.030716+0000","last_undegraded":"2024-03-28T23:32:34.030716+0000","last_fullsized":"2024-03-28T23:32:34.030716+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:08:06.339997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"72'2","reported_seq":16,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.058082+0000","last_change":"2024-03-28T23:32:32.031135+0000","last_active":"2024-03-28T23:32:34.058082+0000","last_peered":"2024-03-28T23:32:34.058082+0000","last_clean":"2024-03-28T23:32:34.058082+0000","last_became_active":"2024-03-28T23:32:32.030796+0000","last_became_peered":"2024-03-28T23:32:32.030796+0000","last_unstale":"2024-03-28T23:32:34.058082+0000","last_undegraded":"2024-03-28T23:32:34.058082+0000","last_fullsized":"2024-03-28T23:32:34.058082+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:18:21.816565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.030751+0000","last_change":"2024-03-28T23:32:33.053549+0000","last_active":"2024-03-28T23:32:34.030751+0000","last_peered":"2024-03-28T23:32:34.030751+0000","last_clean":"2024-03-28T23:32:34.030751+0000","last_became_active":"2024-03-28T23:32:33.053229+0000","last_became_peered":"2024-03-28T23:32:33.053229+0000","last_unstale":"2024-03-28T23:32:34.030751+0000","last_undegraded":"2024-03-28T23:32:34.030751+0000","last_fullsized":"2024-03-28T23:32:34.030751+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:41:02.517121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"72'1","reported_seq":15,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.031169+0000","last_change":"2024-03-28T23:32:32.038261+0000","last_active":"2024-03-28T23:32:34.031169+0000","last_peered":"2024-03-28T23:32:34.031169+0000","last_clean":"2024-03-28T23:32:34.031169+0000","last_became_active":"2024-03-28T23:32:32.037971+0000","last_became_peered":"2024-03-28T23:32:32.037971+0000","last_unstale":"2024-03-28T23:32:34.031169+0000","last_undegraded":"2024-03-28T23:32:34.031169+0000","last_fullsized":"2024-03-28T23:32:34.031169+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:25:10.296139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057837+0000","last_change":"2024-03-28T23:32:33.043954+0000","last_active":"2024-03-28T23:32:34.057837+0000","last_peered":"2024-03-28T23:32:34.057837+0000","last_clean":"2024-03-28T23:32:34.057837+0000","last_became_active":"2024-03-28T23:32:33.043649+0000","last_became_peered":"2024-03-28T23:32:33.043649+0000","last_unstale":"2024-03-28T23:32:34.057837+0000","last_undegraded":"2024-03-28T23:32:34.057837+0000","last_fullsized":"2024-03-28T23:32:34.057837+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:00.358180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057486+0000","last_change":"2024-03-28T23:32:32.033720+0000","last_active":"2024-03-28T23:32:34.057486+0000","last_peered":"2024-03-28T23:32:34.057486+0000","last_clean":"2024-03-28T23:32:34.057486+0000","last_became_active":"2024-03-28T23:32:32.033422+0000","last_became_peered":"2024-03-28T23:32:32.033422+0000","last_unstale":"2024-03-28T23:32:34.057486+0000","last_undegraded":"2024-03-28T23:32:34.057486+0000","last_fullsized":"2024-03-28T23:32:34.057486+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:51:10.440109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028575+0000","last_change":"2024-03-28T23:32:33.045618+0000","last_active":"2024-03-28T23:32:34.028575+0000","last_peered":"2024-03-28T23:32:34.028575+0000","last_clean":"2024-03-28T23:32:34.028575+0000","last_became_active":"2024-03-28T23:32:33.045324+0000","last_became_peered":"2024-03-28T23:32:33.045324+0000","last_unstale":"2024-03-28T23:32:34.028575+0000","last_undegraded":"2024-03-28T23:32:34.028575+0000","last_fullsized":"2024-03-28T23:32:34.028575+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:46:43.644886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"72'2","reported_seq":16,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057846+0000","last_change":"2024-03-28T23:32:32.031574+0000","last_active":"2024-03-28T23:32:34.057846+0000","last_peered":"2024-03-28T23:32:34.057846+0000","last_clean":"2024-03-28T23:32:34.057846+0000","last_became_active":"2024-03-28T23:32:32.031310+0000","last_became_peered":"2024-03-28T23:32:32.031310+0000","last_unstale":"2024-03-28T23:32:34.057846+0000","last_undegraded":"2024-03-28T23:32:34.057846+0000","last_fullsized":"2024-03-28T23:32:34.057846+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:48:07.073481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028782+0000","last_change":"2024-03-28T23:32:33.040983+0000","last_active":"2024-03-28T23:32:34.028782+0000","last_peered":"2024-03-28T23:32:34.028782+0000","last_clean":"2024-03-28T23:32:34.028782+0000","last_became_active":"2024-03-28T23:32:33.040693+0000","last_became_peered":"2024-03-28T23:32:33.040693+0000","last_unstale":"2024-03-28T23:32:34.028782+0000","last_undegraded":"2024-03-28T23:32:34.028782+0000","last_fullsized":"2024-03-28T23:32:34.028782+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:22:05.201764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057575+0000","last_change":"2024-03-28T23:32:32.036998+0000","last_active":"2024-03-28T23:32:34.057575+0000","last_peered":"2024-03-28T23:32:34.057575+0000","last_clean":"2024-03-28T23:32:34.057575+0000","last_became_active":"2024-03-28T23:32:32.036733+0000","last_became_peered":"2024-03-28T23:32:32.036733+0000","last_unstale":"2024-03-28T23:32:34.057575+0000","last_undegraded":"2024-03-28T23:32:34.057575+0000","last_fullsized":"2024-03-28T23:32:34.057575+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:42.334479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026358+0000","last_change":"2024-03-28T23:32:33.041232+0000","last_active":"2024-03-28T23:32:34.026358+0000","last_peered":"2024-03-28T23:32:34.026358+0000","last_clean":"2024-03-28T23:32:34.026358+0000","last_became_active":"2024-03-28T23:32:33.040132+0000","last_became_peered":"2024-03-28T23:32:33.040132+0000","last_unstale":"2024-03-28T23:32:34.026358+0000","last_undegraded":"2024-03-28T23:32:34.026358+0000","last_fullsized":"2024-03-28T23:32:34.026358+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:39:02.824323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057248+0000","last_change":"2024-03-28T23:32:32.036416+0000","last_active":"2024-03-28T23:32:34.057248+0000","last_peered":"2024-03-28T23:32:34.057248+0000","last_clean":"2024-03-28T23:32:34.057248+0000","last_became_active":"2024-03-28T23:32:32.036131+0000","last_became_peered":"2024-03-28T23:32:32.036131+0000","last_unstale":"2024-03-28T23:32:34.057248+0000","last_undegraded":"2024-03-28T23:32:34.057248+0000","last_fullsized":"2024-03-28T23:32:34.057248+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:13:33.914342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057703+0000","last_change":"2024-03-28T23:32:33.035015+0000","last_active":"2024-03-28T23:32:34.057703+0000","last_peered":"2024-03-28T23:32:34.057703+0000","last_clean":"2024-03-28T23:32:34.057703+0000","last_became_active":"2024-03-28T23:32:33.034730+0000","last_became_peered":"2024-03-28T23:32:33.034730+0000","last_unstale":"2024-03-28T23:32:34.057703+0000","last_undegraded":"2024-03-28T23:32:34.057703+0000","last_fullsized":"2024-03-28T23:32:34.057703+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:09.477928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"72'2","reported_seq":16,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026283+0000","last_change":"2024-03-28T23:32:32.032596+0000","last_active":"2024-03-28T23:32:34.026283+0000","last_peered":"2024-03-28T23:32:34.026283+0000","last_clean":"2024-03-28T23:32:34.026283+0000","last_became_active":"2024-03-28T23:32:32.032336+0000","last_became_peered":"2024-03-28T23:32:32.032336+0000","last_unstale":"2024-03-28T23:32:34.026283+0000","last_undegraded":"2024-03-28T23:32:34.026283+0000","last_fullsized":"2024-03-28T23:32:34.026283+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:28:25.356704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026959+0000","last_change":"2024-03-28T23:32:33.044243+0000","last_active":"2024-03-28T23:32:34.026959+0000","last_peered":"2024-03-28T23:32:34.026959+0000","last_clean":"2024-03-28T23:32:34.026959+0000","last_became_active":"2024-03-28T23:32:33.040218+0000","last_became_peered":"2024-03-28T23:32:33.040218+0000","last_unstale":"2024-03-28T23:32:34.026959+0000","last_undegraded":"2024-03-28T23:32:34.026959+0000","last_fullsized":"2024-03-28T23:32:34.026959+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:38:51.513907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"73'1","reported_seq":15,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.029923+0000","last_change":"2024-03-28T23:32:32.041586+0000","last_active":"2024-03-28T23:32:34.029923+0000","last_peered":"2024-03-28T23:32:34.029923+0000","last_clean":"2024-03-28T23:32:34.029923+0000","last_became_active":"2024-03-28T23:32:32.041327+0000","last_became_peered":"2024-03-28T23:32:32.041327+0000","last_unstale":"2024-03-28T23:32:34.029923+0000","last_undegraded":"2024-03-28T23:32:34.029923+0000","last_fullsized":"2024-03-28T23:32:34.029923+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:06:20.608980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057799+0000","last_change":"2024-03-28T23:32:33.042272+0000","last_active":"2024-03-28T23:32:34.057799+0000","last_peered":"2024-03-28T23:32:34.057799+0000","last_clean":"2024-03-28T23:32:34.057799+0000","last_became_active":"2024-03-28T23:32:33.037633+0000","last_became_peered":"2024-03-28T23:32:33.037633+0000","last_unstale":"2024-03-28T23:32:34.057799+0000","last_undegraded":"2024-03-28T23:32:34.057799+0000","last_fullsized":"2024-03-28T23:32:34.057799+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:45.773838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.027762+0000","last_change":"2024-03-28T23:32:32.037001+0000","last_active":"2024-03-28T23:32:34.027762+0000","last_peered":"2024-03-28T23:32:34.027762+0000","last_clean":"2024-03-28T23:32:34.027762+0000","last_became_active":"2024-03-28T23:32:32.036739+0000","last_became_peered":"2024-03-28T23:32:32.036739+0000","last_unstale":"2024-03-28T23:32:34.027762+0000","last_undegraded":"2024-03-28T23:32:34.027762+0000","last_fullsized":"2024-03-28T23:32:34.027762+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:06:20.608980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028491+0000","last_change":"2024-03-28T23:32:33.041123+0000","last_active":"2024-03-28T23:32:34.028491+0000","last_peered":"2024-03-28T23:32:34.028491+0000","last_clean":"2024-03-28T23:32:34.028491+0000","last_became_active":"2024-03-28T23:32:33.040505+0000","last_became_peered":"2024-03-28T23:32:33.040505+0000","last_unstale":"2024-03-28T23:32:34.028491+0000","last_undegraded":"2024-03-28T23:32:34.028491+0000","last_fullsized":"2024-03-28T23:32:34.028491+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:54.194453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.027417+0000","last_change":"2024-03-28T23:32:32.036857+0000","last_active":"2024-03-28T23:32:34.027417+0000","last_peered":"2024-03-28T23:32:34.027417+0000","last_clean":"2024-03-28T23:32:34.027417+0000","last_became_active":"2024-03-28T23:32:32.036576+0000","last_became_peered":"2024-03-28T23:32:32.036576+0000","last_unstale":"2024-03-28T23:32:34.027417+0000","last_undegraded":"2024-03-28T23:32:34.027417+0000","last_fullsized":"2024-03-28T23:32:34.027417+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:43.073425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026102+0000","last_change":"2024-03-28T23:32:33.041769+0000","last_active":"2024-03-28T23:32:34.026102+0000","last_peered":"2024-03-28T23:32:34.026102+0000","last_clean":"2024-03-28T23:32:34.026102+0000","last_became_active":"2024-03-28T23:32:33.041475+0000","last_became_peered":"2024-03-28T23:32:33.041475+0000","last_unstale":"2024-03-28T23:32:34.026102+0000","last_undegraded":"2024-03-28T23:32:34.026102+0000","last_fullsized":"2024-03-28T23:32:34.026102+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:38.859031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026133+0000","last_change":"2024-03-28T23:32:32.037824+0000","last_active":"2024-03-28T23:32:34.026133+0000","last_peered":"2024-03-28T23:32:34.026133+0000","last_clean":"2024-03-28T23:32:34.026133+0000","last_became_active":"2024-03-28T23:32:32.037550+0000","last_became_peered":"2024-03-28T23:32:32.037550+0000","last_unstale":"2024-03-28T23:32:34.026133+0000","last_undegraded":"2024-03-28T23:32:34.026133+0000","last_fullsized":"2024-03-28T23:32:34.026133+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:43.467214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026576+0000","last_change":"2024-03-28T23:32:33.048966+0000","last_active":"2024-03-28T23:32:34.026576+0000","last_peered":"2024-03-28T23:32:34.026576+0000","last_clean":"2024-03-28T23:32:34.026576+0000","last_became_active":"2024-03-28T23:32:33.048362+0000","last_became_peered":"2024-03-28T23:32:33.048362+0000","last_unstale":"2024-03-28T23:32:34.026576+0000","last_undegraded":"2024-03-28T23:32:34.026576+0000","last_fullsized":"2024-03-28T23:32:34.026576+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:25.849919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.030791+0000","last_change":"2024-03-28T23:32:32.038885+0000","last_active":"2024-03-28T23:32:34.030791+0000","last_peered":"2024-03-28T23:32:34.030791+0000","last_clean":"2024-03-28T23:32:34.030791+0000","last_became_active":"2024-03-28T23:32:32.038364+0000","last_became_peered":"2024-03-28T23:32:32.038364+0000","last_unstale":"2024-03-28T23:32:34.030791+0000","last_undegraded":"2024-03-28T23:32:34.030791+0000","last_fullsized":"2024-03-28T23:32:34.030791+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:16:06.156877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057644+0000","last_change":"2024-03-28T23:32:33.049584+0000","last_active":"2024-03-28T23:32:34.057644+0000","last_peered":"2024-03-28T23:32:34.057644+0000","last_clean":"2024-03-28T23:32:34.057644+0000","last_became_active":"2024-03-28T23:32:33.049342+0000","last_became_peered":"2024-03-28T23:32:33.049342+0000","last_unstale":"2024-03-28T23:32:34.057644+0000","last_undegraded":"2024-03-28T23:32:34.057644+0000","last_fullsized":"2024-03-28T23:32:34.057644+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:49.711750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.027122+0000","last_change":"2024-03-28T23:32:32.034044+0000","last_active":"2024-03-28T23:32:34.027122+0000","last_peered":"2024-03-28T23:32:34.027122+0000","last_clean":"2024-03-28T23:32:34.027122+0000","last_became_active":"2024-03-28T23:32:32.033779+0000","last_became_peered":"2024-03-28T23:32:32.033779+0000","last_unstale":"2024-03-28T23:32:34.027122+0000","last_undegraded":"2024-03-28T23:32:34.027122+0000","last_fullsized":"2024-03-28T23:32:34.027122+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:56:22.159708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057558+0000","last_change":"2024-03-28T23:32:33.041180+0000","last_active":"2024-03-28T23:32:34.057558+0000","last_peered":"2024-03-28T23:32:34.057558+0000","last_clean":"2024-03-28T23:32:34.057558+0000","last_became_active":"2024-03-28T23:32:33.040922+0000","last_became_peered":"2024-03-28T23:32:33.040922+0000","last_unstale":"2024-03-28T23:32:34.057558+0000","last_undegraded":"2024-03-28T23:32:34.057558+0000","last_fullsized":"2024-03-28T23:32:34.057558+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:25:45.189859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057186+0000","last_change":"2024-03-28T23:32:32.034328+0000","last_active":"2024-03-28T23:32:34.057186+0000","last_peered":"2024-03-28T23:32:34.057186+0000","last_clean":"2024-03-28T23:32:34.057186+0000","last_became_active":"2024-03-28T23:32:32.034057+0000","last_became_peered":"2024-03-28T23:32:32.034057+0000","last_unstale":"2024-03-28T23:32:34.057186+0000","last_undegraded":"2024-03-28T23:32:34.057186+0000","last_fullsized":"2024-03-28T23:32:34.057186+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:54:12.775407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.027663+0000","last_change":"2024-03-28T23:32:33.038830+0000","last_active":"2024-03-28T23:32:34.027663+0000","last_peered":"2024-03-28T23:32:34.027663+0000","last_clean":"2024-03-28T23:32:34.027663+0000","last_became_active":"2024-03-28T23:32:33.038524+0000","last_became_peered":"2024-03-28T23:32:33.038524+0000","last_unstale":"2024-03-28T23:32:34.027663+0000","last_undegraded":"2024-03-28T23:32:34.027663+0000","last_fullsized":"2024-03-28T23:32:34.027663+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:55:53.512836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.058110+0000","last_change":"2024-03-28T23:32:32.034270+0000","last_active":"2024-03-28T23:32:34.058110+0000","last_peered":"2024-03-28T23:32:34.058110+0000","last_clean":"2024-03-28T23:32:34.058110+0000","last_became_active":"2024-03-28T23:32:32.034022+0000","last_became_peered":"2024-03-28T23:32:32.034022+0000","last_unstale":"2024-03-28T23:32:34.058110+0000","last_undegraded":"2024-03-28T23:32:34.058110+0000","last_fullsized":"2024-03-28T23:32:34.058110+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:15.683500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":269,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026942+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:32:34.026942+0000","last_peered":"2024-03-28T23:32:34.026942+0000","last_clean":"2024-03-28T23:32:34.026942+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:32:34.026942+0000","last_undegraded":"2024-03-28T23:32:34.026942+0000","last_fullsized":"2024-03-28T23:32:34.026942+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574885,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32112,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339728,"statfs":{"total":96636764160,"available":96603881472,"internally_reserved":0,"allocated":950272,"data_stored":1004578,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574886,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32068,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339772,"statfs":{"total":96636764160,"available":96603926528,"internally_reserved":0,"allocated":905216,"data_stored":422492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32044,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339796,"statfs":{"total":96636764160,"available":96603951104,"internally_reserved":0,"allocated":884736,"data_stored":414222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574885,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32064,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339776,"statfs":{"total":96636764160,"available":96603930624,"internally_reserved":0,"allocated":901120,"data_stored":415904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":884736,"data_stored":414770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343944,"statfs":{"total":96636764160,"available":96608198656,"internally_reserved":0,"allocated":892928,"data_stored":414930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574885,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":888832,"data_stored":414300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574885,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":946176,"data_stored":1004556,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:32:37.157 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:32:37.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.292+0000 7f636ae1c640 1 Processor -- start 2024-03-28T23:32:37.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.292+0000 7f636ae1c640 1 -- start start 2024-03-28T23:32:37.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.293+0000 7f636ae1c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6364152890 0x7f63640c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:37.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.293+0000 7f636ae1c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f63640c5b20 0x7f63641553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:37.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.293+0000 7f636ae1c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6364158a30 con 0x7f63640c8ed0 2024-03-28T23:32:37.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.293+0000 7f636ae1c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f63640c8480 con 0x7f63640c5b20 2024-03-28T23:32:37.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.293+0000 7f636ae1c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6364159640 con 0x7f6364152890 2024-03-28T23:32:37.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.293+0000 7f63693b9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f63640c5b20 0x7f63641553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:37.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.293+0000 7f63693b9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f63640c5b20 0x7f63641553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40280/0 (socket says 172.21.15.161:40280) 2024-03-28T23:32:37.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.293+0000 7f63693b9640 1 -- 172.21.15.161:0/2620326307 learned_addr learned my addr 172.21.15.161:0/2620326307 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:37.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.293+0000 7f6368bb8640 1 --2- 172.21.15.161:0/2620326307 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6364152890 0x7f63640c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:37.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.293+0000 7f63693b9640 1 -- 172.21.15.161:0/2620326307 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6364152890 msgr2=0x7f63640c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:37.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.293+0000 7f63693b9640 1 --2- 172.21.15.161:0/2620326307 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6364152890 0x7f63640c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.293+0000 7f63693b9640 1 -- 172.21.15.161:0/2620326307 >> v1:172.21.15.170:6789/0 conn(0x7f63640c8ed0 legacy=0x7f63640c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:37.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.293+0000 7f63693b9640 1 -- 172.21.15.161:0/2620326307 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6364158d10 con 0x7f63640c5b20 2024-03-28T23:32:37.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.294+0000 7f63693b9640 1 --2- 172.21.15.161:0/2620326307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63640c5b20 0x7f63641553a0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f63641597c0 tx=0x7f635c02b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb5d3e24ffc446c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:37.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.294+0000 7f635b7fe640 1 -- 172.21.15.161:0/2620326307 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635c038070 con 0x7f63640c5b20 2024-03-28T23:32:37.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.294+0000 7f635b7fe640 1 -- 172.21.15.161:0/2620326307 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f635c031520 con 0x7f63640c5b20 2024-03-28T23:32:37.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.294+0000 7f635b7fe640 1 -- 172.21.15.161:0/2620326307 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635c0318f0 con 0x7f63640c5b20 2024-03-28T23:32:37.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.295+0000 7f636ae1c640 1 -- 172.21.15.161:0/2620326307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63640c5b20 msgr2=0x7f63641553a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:37.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.295+0000 7f636ae1c640 1 --2- 172.21.15.161:0/2620326307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63640c5b20 0x7f63641553a0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f63641597c0 tx=0x7f635c02b7c0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.295+0000 7f636ae1c640 1 -- 172.21.15.161:0/2620326307 shutdown_connections 2024-03-28T23:32:37.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.295+0000 7f636ae1c640 1 --2- 172.21.15.161:0/2620326307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63640c5b20 0x7f63641553a0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.295+0000 7f636ae1c640 1 --2- 172.21.15.161:0/2620326307 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6364152890 0x7f63640c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.295+0000 7f636ae1c640 1 -- 172.21.15.161:0/2620326307 >> 172.21.15.161:0/2620326307 conn(0x7f63640ac620 msgr2=0x7f63640aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:37.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.295+0000 7f636ae1c640 1 -- 172.21.15.161:0/2620326307 shutdown_connections 2024-03-28T23:32:37.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.295+0000 7f636ae1c640 1 -- 172.21.15.161:0/2620326307 wait complete. 2024-03-28T23:32:37.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.296+0000 7f636ae1c640 1 Processor -- start 2024-03-28T23:32:37.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.296+0000 7f636ae1c640 1 -- start start 2024-03-28T23:32:37.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.297+0000 7f636ae1c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63640c5b20 0x7f63641e7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:37.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.297+0000 7f636ae1c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63640c8ed0 0x7f63641e8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:37.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.297+0000 7f636ae1c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6364152890 0x7f6364146eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:37.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.297+0000 7f636ae1c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6364152c70 con 0x7f6364152890 2024-03-28T23:32:37.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.297+0000 7f636ae1c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f63641592d0 con 0x7f63640c5b20 2024-03-28T23:32:37.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.297+0000 7f636ae1c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6364152df0 con 0x7f63640c8ed0 2024-03-28T23:32:37.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.297+0000 7f63693b9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6364152890 0x7f6364146eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:37.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.297+0000 7f63693b9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6364152890 0x7f6364146eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40296/0 (socket says 172.21.15.161:40296) 2024-03-28T23:32:37.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.297+0000 7f63693b9640 1 -- 172.21.15.161:0/4178242768 learned_addr learned my addr 172.21.15.161:0/4178242768 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:37.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.297+0000 7f6368bb8640 1 --2- 172.21.15.161:0/4178242768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63640c5b20 0x7f63641e7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:37.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.297+0000 7f635bfff640 1 --2- 172.21.15.161:0/4178242768 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63640c8ed0 0x7f63641e8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:37.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.297+0000 7f63693b9640 1 -- 172.21.15.161:0/4178242768 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63640c8ed0 msgr2=0x7f63641e8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:37.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.297+0000 7f63693b9640 1 --2- 172.21.15.161:0/4178242768 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63640c8ed0 0x7f63641e8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.297+0000 7f63693b9640 1 -- 172.21.15.161:0/4178242768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63640c5b20 msgr2=0x7f63641e7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:37.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.297+0000 7f63693b9640 1 --2- 172.21.15.161:0/4178242768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63640c5b20 0x7f63641e7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.297+0000 7f63693b9640 1 -- 172.21.15.161:0/4178242768 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6364148190 con 0x7f6364152890 2024-03-28T23:32:37.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.297+0000 7f63693b9640 1 --2- 172.21.15.161:0/4178242768 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6364152890 0x7f6364146eb0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f635c02b800 tx=0x7f635c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:37.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.298+0000 7f63597fa640 1 -- 172.21.15.161:0/4178242768 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635c042070 con 0x7f6364152890 2024-03-28T23:32:37.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.298+0000 7f63597fa640 1 -- 172.21.15.161:0/4178242768 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f635c03d870 con 0x7f6364152890 2024-03-28T23:32:37.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.298+0000 7f636ae1c640 1 -- 172.21.15.161:0/4178242768 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63641eebe0 con 0x7f6364152890 2024-03-28T23:32:37.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.298+0000 7f63597fa640 1 -- 172.21.15.161:0/4178242768 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635c038040 con 0x7f6364152890 2024-03-28T23:32:37.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.298+0000 7f636ae1c640 1 -- 172.21.15.161:0/4178242768 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6364145120 con 0x7f6364152890 2024-03-28T23:32:37.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.299+0000 7f636ae1c640 1 -- 172.21.15.161:0/4178242768 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6364145320 con 0x7f6364152890 2024-03-28T23:32:37.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.299+0000 7f63597fa640 1 -- 172.21.15.161:0/4178242768 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f635c045070 con 0x7f6364152890 2024-03-28T23:32:37.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.299+0000 7f63597fa640 1 --2- 172.21.15.161:0/4178242768 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f635006b4d0 0x7f635006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:37.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.300+0000 7f63597fa640 1 -- 172.21.15.161:0/4178242768 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6101+0+0 (secure 0 0 0) 0x7f635c0abfd0 con 0x7f6364152890 2024-03-28T23:32:37.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.300+0000 7f6368bb8640 1 --2- 172.21.15.161:0/4178242768 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f635006b4d0 0x7f635006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:37.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.300+0000 7f6368bb8640 1 --2- 172.21.15.161:0/4178242768 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f635006b4d0 0x7f635006d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f63640c5090 tx=0x7f63600098d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:37.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.304+0000 7f63597fa640 1 -- 172.21.15.161:0/4178242768 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6364145320 con 0x7f6364152890 2024-03-28T23:32:37.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.480+0000 7f636ae1c640 1 -- 172.21.15.161:0/4178242768 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f636406cf80 con 0x7f635006b4d0 2024-03-28T23:32:37.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.486+0000 7f63597fa640 1 -- 172.21.15.161:0/4178242768 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252144 (secure 0 0 0) 0x7f636406cf80 con 0x7f635006b4d0 2024-03-28T23:32:37.490 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:37.491 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:32:37.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.489+0000 7f636ae1c640 1 -- 172.21.15.161:0/4178242768 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f635006b4d0 msgr2=0x7f635006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:37.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.489+0000 7f636ae1c640 1 --2- 172.21.15.161:0/4178242768 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f635006b4d0 0x7f635006d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f63640c5090 tx=0x7f63600098d0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.489+0000 7f636ae1c640 1 -- 172.21.15.161:0/4178242768 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6364152890 msgr2=0x7f6364146eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:37.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.489+0000 7f636ae1c640 1 --2- 172.21.15.161:0/4178242768 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6364152890 0x7f6364146eb0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f635c02b800 tx=0x7f635c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.489+0000 7f636ae1c640 1 -- 172.21.15.161:0/4178242768 shutdown_connections 2024-03-28T23:32:37.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.489+0000 7f636ae1c640 1 --2- 172.21.15.161:0/4178242768 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f635006b4d0 0x7f635006d990 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.489+0000 7f636ae1c640 1 --2- 172.21.15.161:0/4178242768 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6364152890 0x7f6364146eb0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.489+0000 7f636ae1c640 1 --2- 172.21.15.161:0/4178242768 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63640c8ed0 0x7f63641e8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.489+0000 7f636ae1c640 1 --2- 172.21.15.161:0/4178242768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63640c5b20 0x7f63641e7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.489+0000 7f636ae1c640 1 -- 172.21.15.161:0/4178242768 >> 172.21.15.161:0/4178242768 conn(0x7f63640ac620 msgr2=0x7f636406be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:37.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.489+0000 7f636ae1c640 1 -- 172.21.15.161:0/4178242768 shutdown_connections 2024-03-28T23:32:37.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.489+0000 7f636ae1c640 1 -- 172.21.15.161:0/4178242768 wait complete. 2024-03-28T23:32:37.505 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-03-28T23:32:37.344238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239916,"kb_used_data":7084,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754734804,"statfs":{"total":773094113280,"available":772848439296,"internally_reserved":0,"allocated":7254016,"data_stored":4505752,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001238"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.027696+0000","last_change":"2024-03-28T23:32:33.046706+0000","last_active":"2024-03-28T23:32:34.027696+0000","last_peered":"2024-03-28T23:32:34.027696+0000","last_clean":"2024-03-28T23:32:34.027696+0000","last_became_active":"2024-03-28T23:32:33.046222+0000","last_became_peered":"2024-03-28T23:32:33.046222+0000","last_unstale":"2024-03-28T23:32:34.027696+0000","last_undegraded":"2024-03-28T23:32:34.027696+0000","last_fullsized":"2024-03-28T23:32:34.027696+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:02:45.190566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028766+0000","last_change":"2024-03-28T23:32:33.051174+0000","last_active":"2024-03-28T23:32:34.028766+0000","last_peered":"2024-03-28T23:32:34.028766+0000","last_clean":"2024-03-28T23:32:34.028766+0000","last_became_active":"2024-03-28T23:32:33.050345+0000","last_became_peered":"2024-03-28T23:32:33.050345+0000","last_unstale":"2024-03-28T23:32:34.028766+0000","last_undegraded":"2024-03-28T23:32:34.028766+0000","last_fullsized":"2024-03-28T23:32:34.028766+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:41:39.657731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057623+0000","last_change":"2024-03-28T23:32:33.048785+0000","last_active":"2024-03-28T23:32:34.057623+0000","last_peered":"2024-03-28T23:32:34.057623+0000","last_clean":"2024-03-28T23:32:34.057623+0000","last_became_active":"2024-03-28T23:32:33.048519+0000","last_became_peered":"2024-03-28T23:32:33.048519+0000","last_unstale":"2024-03-28T23:32:34.057623+0000","last_undegraded":"2024-03-28T23:32:34.057623+0000","last_fullsized":"2024-03-28T23:32:34.057623+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:21.464120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026644+0000","last_change":"2024-03-28T23:32:33.033535+0000","last_active":"2024-03-28T23:32:34.026644+0000","last_peered":"2024-03-28T23:32:34.026644+0000","last_clean":"2024-03-28T23:32:34.026644+0000","last_became_active":"2024-03-28T23:32:33.033219+0000","last_became_peered":"2024-03-28T23:32:33.033219+0000","last_unstale":"2024-03-28T23:32:34.026644+0000","last_undegraded":"2024-03-28T23:32:34.026644+0000","last_fullsized":"2024-03-28T23:32:34.026644+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:39:39.113457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.030899+0000","last_change":"2024-03-28T23:32:33.042972+0000","last_active":"2024-03-28T23:32:34.030899+0000","last_peered":"2024-03-28T23:32:34.030899+0000","last_clean":"2024-03-28T23:32:34.030899+0000","last_became_active":"2024-03-28T23:32:33.042665+0000","last_became_peered":"2024-03-28T23:32:33.042665+0000","last_unstale":"2024-03-28T23:32:34.030899+0000","last_undegraded":"2024-03-28T23:32:34.030899+0000","last_fullsized":"2024-03-28T23:32:34.030899+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:41:50.190883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057842+0000","last_change":"2024-03-28T23:32:33.030549+0000","last_active":"2024-03-28T23:32:34.057842+0000","last_peered":"2024-03-28T23:32:34.057842+0000","last_clean":"2024-03-28T23:32:34.057842+0000","last_became_active":"2024-03-28T23:32:33.030299+0000","last_became_peered":"2024-03-28T23:32:33.030299+0000","last_unstale":"2024-03-28T23:32:34.057842+0000","last_undegraded":"2024-03-28T23:32:34.057842+0000","last_fullsized":"2024-03-28T23:32:34.057842+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:35.055035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.027067+0000","last_change":"2024-03-28T23:32:33.044477+0000","last_active":"2024-03-28T23:32:34.027067+0000","last_peered":"2024-03-28T23:32:34.027067+0000","last_clean":"2024-03-28T23:32:34.027067+0000","last_became_active":"2024-03-28T23:32:33.043960+0000","last_became_peered":"2024-03-28T23:32:33.043960+0000","last_unstale":"2024-03-28T23:32:34.027067+0000","last_undegraded":"2024-03-28T23:32:34.027067+0000","last_fullsized":"2024-03-28T23:32:34.027067+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:21.464120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.030686+0000","last_change":"2024-03-28T23:32:33.052803+0000","last_active":"2024-03-28T23:32:34.030686+0000","last_peered":"2024-03-28T23:32:34.030686+0000","last_clean":"2024-03-28T23:32:34.030686+0000","last_became_active":"2024-03-28T23:32:33.052524+0000","last_became_peered":"2024-03-28T23:32:33.052524+0000","last_unstale":"2024-03-28T23:32:34.030686+0000","last_undegraded":"2024-03-28T23:32:34.030686+0000","last_fullsized":"2024-03-28T23:32:34.030686+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:19:09.500642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.058056+0000","last_change":"2024-03-28T23:32:33.042587+0000","last_active":"2024-03-28T23:32:34.058056+0000","last_peered":"2024-03-28T23:32:34.058056+0000","last_clean":"2024-03-28T23:32:34.058056+0000","last_became_active":"2024-03-28T23:32:33.037002+0000","last_became_peered":"2024-03-28T23:32:33.037002+0000","last_unstale":"2024-03-28T23:32:34.058056+0000","last_undegraded":"2024-03-28T23:32:34.058056+0000","last_fullsized":"2024-03-28T23:32:34.058056+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:36:03.951054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.025842+0000","last_change":"2024-03-28T23:32:33.039941+0000","last_active":"2024-03-28T23:32:34.025842+0000","last_peered":"2024-03-28T23:32:34.025842+0000","last_clean":"2024-03-28T23:32:34.025842+0000","last_became_active":"2024-03-28T23:32:33.039656+0000","last_became_peered":"2024-03-28T23:32:33.039656+0000","last_unstale":"2024-03-28T23:32:34.025842+0000","last_undegraded":"2024-03-28T23:32:34.025842+0000","last_fullsized":"2024-03-28T23:32:34.025842+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:16.850667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028608+0000","last_change":"2024-03-28T23:32:33.052329+0000","last_active":"2024-03-28T23:32:34.028608+0000","last_peered":"2024-03-28T23:32:34.028608+0000","last_clean":"2024-03-28T23:32:34.028608+0000","last_became_active":"2024-03-28T23:32:33.051833+0000","last_became_peered":"2024-03-28T23:32:33.051833+0000","last_unstale":"2024-03-28T23:32:34.028608+0000","last_undegraded":"2024-03-28T23:32:34.028608+0000","last_fullsized":"2024-03-28T23:32:34.028608+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:05.929331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026794+0000","last_change":"2024-03-28T23:32:33.042337+0000","last_active":"2024-03-28T23:32:34.026794+0000","last_peered":"2024-03-28T23:32:34.026794+0000","last_clean":"2024-03-28T23:32:34.026794+0000","last_became_active":"2024-03-28T23:32:33.042062+0000","last_became_peered":"2024-03-28T23:32:33.042062+0000","last_unstale":"2024-03-28T23:32:34.026794+0000","last_undegraded":"2024-03-28T23:32:34.026794+0000","last_fullsized":"2024-03-28T23:32:34.026794+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:13:34.921740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.030802+0000","last_change":"2024-03-28T23:32:33.046685+0000","last_active":"2024-03-28T23:32:34.030802+0000","last_peered":"2024-03-28T23:32:34.030802+0000","last_clean":"2024-03-28T23:32:34.030802+0000","last_became_active":"2024-03-28T23:32:33.046206+0000","last_became_peered":"2024-03-28T23:32:33.046206+0000","last_unstale":"2024-03-28T23:32:34.030802+0000","last_undegraded":"2024-03-28T23:32:34.030802+0000","last_fullsized":"2024-03-28T23:32:34.030802+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:43:22.001147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057854+0000","last_change":"2024-03-28T23:32:33.029880+0000","last_active":"2024-03-28T23:32:34.057854+0000","last_peered":"2024-03-28T23:32:34.057854+0000","last_clean":"2024-03-28T23:32:34.057854+0000","last_became_active":"2024-03-28T23:32:33.029582+0000","last_became_peered":"2024-03-28T23:32:33.029582+0000","last_unstale":"2024-03-28T23:32:34.057854+0000","last_undegraded":"2024-03-28T23:32:34.057854+0000","last_fullsized":"2024-03-28T23:32:34.057854+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:38:39.283932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028335+0000","last_change":"2024-03-28T23:32:33.040102+0000","last_active":"2024-03-28T23:32:34.028335+0000","last_peered":"2024-03-28T23:32:34.028335+0000","last_clean":"2024-03-28T23:32:34.028335+0000","last_became_active":"2024-03-28T23:32:33.039660+0000","last_became_peered":"2024-03-28T23:32:33.039660+0000","last_unstale":"2024-03-28T23:32:34.028335+0000","last_undegraded":"2024-03-28T23:32:34.028335+0000","last_fullsized":"2024-03-28T23:32:34.028335+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:35.055035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026721+0000","last_change":"2024-03-28T23:32:33.035442+0000","last_active":"2024-03-28T23:32:34.026721+0000","last_peered":"2024-03-28T23:32:34.026721+0000","last_clean":"2024-03-28T23:32:34.026721+0000","last_became_active":"2024-03-28T23:32:33.035154+0000","last_became_peered":"2024-03-28T23:32:33.035154+0000","last_unstale":"2024-03-28T23:32:34.026721+0000","last_undegraded":"2024-03-28T23:32:34.026721+0000","last_fullsized":"2024-03-28T23:32:34.026721+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:34:22.775718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.031125+0000","last_change":"2024-03-28T23:32:33.047782+0000","last_active":"2024-03-28T23:32:34.031125+0000","last_peered":"2024-03-28T23:32:34.031125+0000","last_clean":"2024-03-28T23:32:34.031125+0000","last_became_active":"2024-03-28T23:32:33.047254+0000","last_became_peered":"2024-03-28T23:32:33.047254+0000","last_unstale":"2024-03-28T23:32:34.031125+0000","last_undegraded":"2024-03-28T23:32:34.031125+0000","last_fullsized":"2024-03-28T23:32:34.031125+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:19.004555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057391+0000","last_change":"2024-03-28T23:32:33.037321+0000","last_active":"2024-03-28T23:32:34.057391+0000","last_peered":"2024-03-28T23:32:34.057391+0000","last_clean":"2024-03-28T23:32:34.057391+0000","last_became_active":"2024-03-28T23:32:33.036961+0000","last_became_peered":"2024-03-28T23:32:33.036961+0000","last_unstale":"2024-03-28T23:32:34.057391+0000","last_undegraded":"2024-03-28T23:32:34.057391+0000","last_fullsized":"2024-03-28T23:32:34.057391+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:25.849919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.058056+0000","last_change":"2024-03-28T23:32:33.053071+0000","last_active":"2024-03-28T23:32:34.058056+0000","last_peered":"2024-03-28T23:32:34.058056+0000","last_clean":"2024-03-28T23:32:34.058056+0000","last_became_active":"2024-03-28T23:32:33.052784+0000","last_became_peered":"2024-03-28T23:32:33.052784+0000","last_unstale":"2024-03-28T23:32:34.058056+0000","last_undegraded":"2024-03-28T23:32:34.058056+0000","last_fullsized":"2024-03-28T23:32:34.058056+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:58:48.231248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.030912+0000","last_change":"2024-03-28T23:32:33.052720+0000","last_active":"2024-03-28T23:32:34.030912+0000","last_peered":"2024-03-28T23:32:34.030912+0000","last_clean":"2024-03-28T23:32:34.030912+0000","last_became_active":"2024-03-28T23:32:33.052460+0000","last_became_peered":"2024-03-28T23:32:33.052460+0000","last_unstale":"2024-03-28T23:32:34.030912+0000","last_undegraded":"2024-03-28T23:32:34.030912+0000","last_fullsized":"2024-03-28T23:32:34.030912+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:30.891315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057833+0000","last_change":"2024-03-28T23:32:33.035666+0000","last_active":"2024-03-28T23:32:34.057833+0000","last_peered":"2024-03-28T23:32:34.057833+0000","last_clean":"2024-03-28T23:32:34.057833+0000","last_became_active":"2024-03-28T23:32:33.035403+0000","last_became_peered":"2024-03-28T23:32:33.035403+0000","last_unstale":"2024-03-28T23:32:34.057833+0000","last_undegraded":"2024-03-28T23:32:34.057833+0000","last_fullsized":"2024-03-28T23:32:34.057833+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:40:51.269760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.030982+0000","last_change":"2024-03-28T23:32:33.047992+0000","last_active":"2024-03-28T23:32:34.030982+0000","last_peered":"2024-03-28T23:32:34.030982+0000","last_clean":"2024-03-28T23:32:34.030982+0000","last_became_active":"2024-03-28T23:32:33.047058+0000","last_became_peered":"2024-03-28T23:32:33.047058+0000","last_unstale":"2024-03-28T23:32:34.030982+0000","last_undegraded":"2024-03-28T23:32:34.030982+0000","last_fullsized":"2024-03-28T23:32:34.030982+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:35:30.886935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057683+0000","last_change":"2024-03-28T23:32:33.029536+0000","last_active":"2024-03-28T23:32:34.057683+0000","last_peered":"2024-03-28T23:32:34.057683+0000","last_clean":"2024-03-28T23:32:34.057683+0000","last_became_active":"2024-03-28T23:32:33.029265+0000","last_became_peered":"2024-03-28T23:32:33.029265+0000","last_unstale":"2024-03-28T23:32:34.057683+0000","last_undegraded":"2024-03-28T23:32:34.057683+0000","last_fullsized":"2024-03-28T23:32:34.057683+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:51:11.447507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028300+0000","last_change":"2024-03-28T23:32:33.041328+0000","last_active":"2024-03-28T23:32:34.028300+0000","last_peered":"2024-03-28T23:32:34.028300+0000","last_clean":"2024-03-28T23:32:34.028300+0000","last_became_active":"2024-03-28T23:32:33.041071+0000","last_became_peered":"2024-03-28T23:32:33.041071+0000","last_unstale":"2024-03-28T23:32:34.028300+0000","last_undegraded":"2024-03-28T23:32:34.028300+0000","last_fullsized":"2024-03-28T23:32:34.028300+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:00.358180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028708+0000","last_change":"2024-03-28T23:32:33.043840+0000","last_active":"2024-03-28T23:32:34.028708+0000","last_peered":"2024-03-28T23:32:34.028708+0000","last_clean":"2024-03-28T23:32:34.028708+0000","last_became_active":"2024-03-28T23:32:33.043569+0000","last_became_peered":"2024-03-28T23:32:33.043569+0000","last_unstale":"2024-03-28T23:32:34.028708+0000","last_undegraded":"2024-03-28T23:32:34.028708+0000","last_fullsized":"2024-03-28T23:32:34.028708+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:28.463493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.025993+0000","last_change":"2024-03-28T23:32:33.047594+0000","last_active":"2024-03-28T23:32:34.025993+0000","last_peered":"2024-03-28T23:32:34.025993+0000","last_clean":"2024-03-28T23:32:34.025993+0000","last_became_active":"2024-03-28T23:32:33.047031+0000","last_became_peered":"2024-03-28T23:32:33.047031+0000","last_unstale":"2024-03-28T23:32:34.025993+0000","last_undegraded":"2024-03-28T23:32:34.025993+0000","last_fullsized":"2024-03-28T23:32:34.025993+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:30:12.769818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.027395+0000","last_change":"2024-03-28T23:32:33.043396+0000","last_active":"2024-03-28T23:32:34.027395+0000","last_peered":"2024-03-28T23:32:34.027395+0000","last_clean":"2024-03-28T23:32:34.027395+0000","last_became_active":"2024-03-28T23:32:33.043123+0000","last_became_peered":"2024-03-28T23:32:33.043123+0000","last_unstale":"2024-03-28T23:32:34.027395+0000","last_undegraded":"2024-03-28T23:32:34.027395+0000","last_fullsized":"2024-03-28T23:32:34.027395+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:28:36.535909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.030962+0000","last_change":"2024-03-28T23:32:33.042288+0000","last_active":"2024-03-28T23:32:34.030962+0000","last_peered":"2024-03-28T23:32:34.030962+0000","last_clean":"2024-03-28T23:32:34.030962+0000","last_became_active":"2024-03-28T23:32:33.042007+0000","last_became_peered":"2024-03-28T23:32:33.042007+0000","last_unstale":"2024-03-28T23:32:34.030962+0000","last_undegraded":"2024-03-28T23:32:34.030962+0000","last_fullsized":"2024-03-28T23:32:34.030962+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:16:12.634582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028532+0000","last_change":"2024-03-28T23:32:33.041336+0000","last_active":"2024-03-28T23:32:34.028532+0000","last_peered":"2024-03-28T23:32:34.028532+0000","last_clean":"2024-03-28T23:32:34.028532+0000","last_became_active":"2024-03-28T23:32:33.040865+0000","last_became_peered":"2024-03-28T23:32:33.040865+0000","last_unstale":"2024-03-28T23:32:34.028532+0000","last_undegraded":"2024-03-28T23:32:34.028532+0000","last_fullsized":"2024-03-28T23:32:34.028532+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:44:58.784172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026760+0000","last_change":"2024-03-28T23:32:33.049222+0000","last_active":"2024-03-28T23:32:34.026760+0000","last_peered":"2024-03-28T23:32:34.026760+0000","last_clean":"2024-03-28T23:32:34.026760+0000","last_became_active":"2024-03-28T23:32:33.048933+0000","last_became_peered":"2024-03-28T23:32:33.048933+0000","last_unstale":"2024-03-28T23:32:34.026760+0000","last_undegraded":"2024-03-28T23:32:34.026760+0000","last_fullsized":"2024-03-28T23:32:34.026760+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:17:45.545141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057563+0000","last_change":"2024-03-28T23:32:33.047977+0000","last_active":"2024-03-28T23:32:34.057563+0000","last_peered":"2024-03-28T23:32:34.057563+0000","last_clean":"2024-03-28T23:32:34.057563+0000","last_became_active":"2024-03-28T23:32:33.047699+0000","last_became_peered":"2024-03-28T23:32:33.047699+0000","last_unstale":"2024-03-28T23:32:34.057563+0000","last_undegraded":"2024-03-28T23:32:34.057563+0000","last_fullsized":"2024-03-28T23:32:34.057563+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:38:51.513907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026882+0000","last_change":"2024-03-28T23:32:33.036527+0000","last_active":"2024-03-28T23:32:34.026882+0000","last_peered":"2024-03-28T23:32:34.026882+0000","last_clean":"2024-03-28T23:32:34.026882+0000","last_became_active":"2024-03-28T23:32:33.036269+0000","last_became_peered":"2024-03-28T23:32:33.036269+0000","last_unstale":"2024-03-28T23:32:34.026882+0000","last_undegraded":"2024-03-28T23:32:34.026882+0000","last_fullsized":"2024-03-28T23:32:34.026882+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:43:35.946992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057140+0000","last_change":"2024-03-28T23:32:33.038016+0000","last_active":"2024-03-28T23:32:34.057140+0000","last_peered":"2024-03-28T23:32:34.057140+0000","last_clean":"2024-03-28T23:32:34.057140+0000","last_became_active":"2024-03-28T23:32:33.037754+0000","last_became_peered":"2024-03-28T23:32:33.037754+0000","last_unstale":"2024-03-28T23:32:34.057140+0000","last_undegraded":"2024-03-28T23:32:34.057140+0000","last_fullsized":"2024-03-28T23:32:34.057140+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:21.878156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"72'4","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.031109+0000","last_change":"2024-03-28T23:32:32.040468+0000","last_active":"2024-03-28T23:32:34.031109+0000","last_peered":"2024-03-28T23:32:34.031109+0000","last_clean":"2024-03-28T23:32:34.031109+0000","last_became_active":"2024-03-28T23:32:32.039342+0000","last_became_peered":"2024-03-28T23:32:32.039342+0000","last_unstale":"2024-03-28T23:32:34.031109+0000","last_undegraded":"2024-03-28T23:32:34.031109+0000","last_fullsized":"2024-03-28T23:32:34.031109+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:05.786248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028348+0000","last_change":"2024-03-28T23:32:33.039778+0000","last_active":"2024-03-28T23:32:34.028348+0000","last_peered":"2024-03-28T23:32:34.028348+0000","last_clean":"2024-03-28T23:32:34.028348+0000","last_became_active":"2024-03-28T23:32:33.039506+0000","last_became_peered":"2024-03-28T23:32:33.039506+0000","last_unstale":"2024-03-28T23:32:34.028348+0000","last_undegraded":"2024-03-28T23:32:34.028348+0000","last_fullsized":"2024-03-28T23:32:34.028348+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:38:39.283932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"72'1","reported_seq":15,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026243+0000","last_change":"2024-03-28T23:32:32.041093+0000","last_active":"2024-03-28T23:32:34.026243+0000","last_peered":"2024-03-28T23:32:34.026243+0000","last_clean":"2024-03-28T23:32:34.026243+0000","last_became_active":"2024-03-28T23:32:32.040827+0000","last_became_peered":"2024-03-28T23:32:32.040827+0000","last_unstale":"2024-03-28T23:32:34.026243+0000","last_undegraded":"2024-03-28T23:32:34.026243+0000","last_fullsized":"2024-03-28T23:32:34.026243+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:55.544945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057929+0000","last_change":"2024-03-28T23:32:33.044887+0000","last_active":"2024-03-28T23:32:34.057929+0000","last_peered":"2024-03-28T23:32:34.057929+0000","last_clean":"2024-03-28T23:32:34.057929+0000","last_became_active":"2024-03-28T23:32:33.042005+0000","last_became_peered":"2024-03-28T23:32:33.042005+0000","last_unstale":"2024-03-28T23:32:34.057929+0000","last_undegraded":"2024-03-28T23:32:34.057929+0000","last_fullsized":"2024-03-28T23:32:34.057929+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:41.257241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"72'2","reported_seq":16,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.027508+0000","last_change":"2024-03-28T23:32:32.040527+0000","last_active":"2024-03-28T23:32:34.027508+0000","last_peered":"2024-03-28T23:32:34.027508+0000","last_clean":"2024-03-28T23:32:34.027508+0000","last_became_active":"2024-03-28T23:32:32.040142+0000","last_became_peered":"2024-03-28T23:32:32.040142+0000","last_unstale":"2024-03-28T23:32:34.027508+0000","last_undegraded":"2024-03-28T23:32:34.027508+0000","last_fullsized":"2024-03-28T23:32:34.027508+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:13:15.856750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057887+0000","last_change":"2024-03-28T23:32:33.040988+0000","last_active":"2024-03-28T23:32:34.057887+0000","last_peered":"2024-03-28T23:32:34.057887+0000","last_clean":"2024-03-28T23:32:34.057887+0000","last_became_active":"2024-03-28T23:32:33.036045+0000","last_became_peered":"2024-03-28T23:32:33.036045+0000","last_unstale":"2024-03-28T23:32:34.057887+0000","last_undegraded":"2024-03-28T23:32:34.057887+0000","last_fullsized":"2024-03-28T23:32:34.057887+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:10:39.981362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"72'2","reported_seq":16,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026203+0000","last_change":"2024-03-28T23:32:32.036856+0000","last_active":"2024-03-28T23:32:34.026203+0000","last_peered":"2024-03-28T23:32:34.026203+0000","last_clean":"2024-03-28T23:32:34.026203+0000","last_became_active":"2024-03-28T23:32:32.036565+0000","last_became_peered":"2024-03-28T23:32:32.036565+0000","last_unstale":"2024-03-28T23:32:34.026203+0000","last_undegraded":"2024-03-28T23:32:34.026203+0000","last_fullsized":"2024-03-28T23:32:34.026203+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:11:01.106436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028510+0000","last_change":"2024-03-28T23:32:33.047911+0000","last_active":"2024-03-28T23:32:34.028510+0000","last_peered":"2024-03-28T23:32:34.028510+0000","last_clean":"2024-03-28T23:32:34.028510+0000","last_became_active":"2024-03-28T23:32:33.047616+0000","last_became_peered":"2024-03-28T23:32:33.047616+0000","last_unstale":"2024-03-28T23:32:34.028510+0000","last_undegraded":"2024-03-28T23:32:34.028510+0000","last_fullsized":"2024-03-28T23:32:34.028510+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:58:48.231248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"72'1","reported_seq":15,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057966+0000","last_change":"2024-03-28T23:32:32.036829+0000","last_active":"2024-03-28T23:32:34.057966+0000","last_peered":"2024-03-28T23:32:34.057966+0000","last_clean":"2024-03-28T23:32:34.057966+0000","last_became_active":"2024-03-28T23:32:32.036571+0000","last_became_peered":"2024-03-28T23:32:32.036571+0000","last_unstale":"2024-03-28T23:32:34.057966+0000","last_undegraded":"2024-03-28T23:32:34.057966+0000","last_fullsized":"2024-03-28T23:32:34.057966+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:32:03.789593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028687+0000","last_change":"2024-03-28T23:32:33.052330+0000","last_active":"2024-03-28T23:32:34.028687+0000","last_peered":"2024-03-28T23:32:34.028687+0000","last_clean":"2024-03-28T23:32:34.028687+0000","last_became_active":"2024-03-28T23:32:33.051828+0000","last_became_peered":"2024-03-28T23:32:33.051828+0000","last_unstale":"2024-03-28T23:32:34.028687+0000","last_undegraded":"2024-03-28T23:32:34.028687+0000","last_fullsized":"2024-03-28T23:32:34.028687+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:40:53.134608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"72'1","reported_seq":15,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.025535+0000","last_change":"2024-03-28T23:32:32.036739+0000","last_active":"2024-03-28T23:32:34.025535+0000","last_peered":"2024-03-28T23:32:34.025535+0000","last_clean":"2024-03-28T23:32:34.025535+0000","last_became_active":"2024-03-28T23:32:32.036471+0000","last_became_peered":"2024-03-28T23:32:32.036471+0000","last_unstale":"2024-03-28T23:32:34.025535+0000","last_undegraded":"2024-03-28T23:32:34.025535+0000","last_fullsized":"2024-03-28T23:32:34.025535+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:48.704351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.058076+0000","last_change":"2024-03-28T23:32:33.034126+0000","last_active":"2024-03-28T23:32:34.058076+0000","last_peered":"2024-03-28T23:32:34.058076+0000","last_clean":"2024-03-28T23:32:34.058076+0000","last_became_active":"2024-03-28T23:32:33.033703+0000","last_became_peered":"2024-03-28T23:32:33.033703+0000","last_unstale":"2024-03-28T23:32:34.058076+0000","last_undegraded":"2024-03-28T23:32:34.058076+0000","last_fullsized":"2024-03-28T23:32:34.058076+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:38.859031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028431+0000","last_change":"2024-03-28T23:32:32.039047+0000","last_active":"2024-03-28T23:32:34.028431+0000","last_peered":"2024-03-28T23:32:34.028431+0000","last_clean":"2024-03-28T23:32:34.028431+0000","last_became_active":"2024-03-28T23:32:32.038780+0000","last_became_peered":"2024-03-28T23:32:32.038780+0000","last_unstale":"2024-03-28T23:32:34.028431+0000","last_undegraded":"2024-03-28T23:32:34.028431+0000","last_fullsized":"2024-03-28T23:32:34.028431+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:13:15.856750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026038+0000","last_change":"2024-03-28T23:32:33.051453+0000","last_active":"2024-03-28T23:32:34.026038+0000","last_peered":"2024-03-28T23:32:34.026038+0000","last_clean":"2024-03-28T23:32:34.026038+0000","last_became_active":"2024-03-28T23:32:33.051155+0000","last_became_peered":"2024-03-28T23:32:33.051155+0000","last_unstale":"2024-03-28T23:32:34.026038+0000","last_undegraded":"2024-03-28T23:32:34.026038+0000","last_fullsized":"2024-03-28T23:32:34.026038+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:36:03.951054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.030722+0000","last_change":"2024-03-28T23:32:32.036979+0000","last_active":"2024-03-28T23:32:34.030722+0000","last_peered":"2024-03-28T23:32:34.030722+0000","last_clean":"2024-03-28T23:32:34.030722+0000","last_became_active":"2024-03-28T23:32:32.036710+0000","last_became_peered":"2024-03-28T23:32:32.036710+0000","last_unstale":"2024-03-28T23:32:34.030722+0000","last_undegraded":"2024-03-28T23:32:34.030722+0000","last_fullsized":"2024-03-28T23:32:34.030722+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:46.072852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028295+0000","last_change":"2024-03-28T23:32:33.041923+0000","last_active":"2024-03-28T23:32:34.028295+0000","last_peered":"2024-03-28T23:32:34.028295+0000","last_clean":"2024-03-28T23:32:34.028295+0000","last_became_active":"2024-03-28T23:32:33.041631+0000","last_became_peered":"2024-03-28T23:32:33.041631+0000","last_unstale":"2024-03-28T23:32:34.028295+0000","last_undegraded":"2024-03-28T23:32:34.028295+0000","last_fullsized":"2024-03-28T23:32:34.028295+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:32:08.473276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.027796+0000","last_change":"2024-03-28T23:32:32.041563+0000","last_active":"2024-03-28T23:32:34.027796+0000","last_peered":"2024-03-28T23:32:34.027796+0000","last_clean":"2024-03-28T23:32:34.027796+0000","last_became_active":"2024-03-28T23:32:32.040424+0000","last_became_peered":"2024-03-28T23:32:32.040424+0000","last_unstale":"2024-03-28T23:32:34.027796+0000","last_undegraded":"2024-03-28T23:32:34.027796+0000","last_fullsized":"2024-03-28T23:32:34.027796+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:37:25.645113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.025772+0000","last_change":"2024-03-28T23:32:33.029986+0000","last_active":"2024-03-28T23:32:34.025772+0000","last_peered":"2024-03-28T23:32:34.025772+0000","last_clean":"2024-03-28T23:32:34.025772+0000","last_became_active":"2024-03-28T23:32:33.029413+0000","last_became_peered":"2024-03-28T23:32:33.029413+0000","last_unstale":"2024-03-28T23:32:34.025772+0000","last_undegraded":"2024-03-28T23:32:34.025772+0000","last_fullsized":"2024-03-28T23:32:34.025772+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:37:26.652511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026096+0000","last_change":"2024-03-28T23:32:32.037039+0000","last_active":"2024-03-28T23:32:34.026096+0000","last_peered":"2024-03-28T23:32:34.026096+0000","last_clean":"2024-03-28T23:32:34.026096+0000","last_became_active":"2024-03-28T23:32:32.036687+0000","last_became_peered":"2024-03-28T23:32:32.036687+0000","last_unstale":"2024-03-28T23:32:34.026096+0000","last_undegraded":"2024-03-28T23:32:34.026096+0000","last_fullsized":"2024-03-28T23:32:34.026096+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:27:15.824876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.058028+0000","last_change":"2024-03-28T23:32:33.035809+0000","last_active":"2024-03-28T23:32:34.058028+0000","last_peered":"2024-03-28T23:32:34.058028+0000","last_clean":"2024-03-28T23:32:34.058028+0000","last_became_active":"2024-03-28T23:32:33.035563+0000","last_became_peered":"2024-03-28T23:32:33.035563+0000","last_unstale":"2024-03-28T23:32:34.058028+0000","last_undegraded":"2024-03-28T23:32:34.058028+0000","last_fullsized":"2024-03-28T23:32:34.058028+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:54.194453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"72'3","reported_seq":17,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.031111+0000","last_change":"2024-03-28T23:32:32.038230+0000","last_active":"2024-03-28T23:32:34.031111+0000","last_peered":"2024-03-28T23:32:34.031111+0000","last_clean":"2024-03-28T23:32:34.031111+0000","last_became_active":"2024-03-28T23:32:32.037681+0000","last_became_peered":"2024-03-28T23:32:32.037681+0000","last_unstale":"2024-03-28T23:32:34.031111+0000","last_undegraded":"2024-03-28T23:32:34.031111+0000","last_fullsized":"2024-03-28T23:32:34.031111+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:36:34.752866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057265+0000","last_change":"2024-03-28T23:32:33.049499+0000","last_active":"2024-03-28T23:32:34.057265+0000","last_peered":"2024-03-28T23:32:34.057265+0000","last_clean":"2024-03-28T23:32:34.057265+0000","last_became_active":"2024-03-28T23:32:33.049233+0000","last_became_peered":"2024-03-28T23:32:33.049233+0000","last_unstale":"2024-03-28T23:32:34.057265+0000","last_undegraded":"2024-03-28T23:32:34.057265+0000","last_fullsized":"2024-03-28T23:32:34.057265+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:39:02.824323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"72'6","reported_seq":20,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026323+0000","last_change":"2024-03-28T23:32:32.036835+0000","last_active":"2024-03-28T23:32:34.026323+0000","last_peered":"2024-03-28T23:32:34.026323+0000","last_clean":"2024-03-28T23:32:34.026323+0000","last_became_active":"2024-03-28T23:32:32.036557+0000","last_became_peered":"2024-03-28T23:32:32.036557+0000","last_unstale":"2024-03-28T23:32:34.026323+0000","last_undegraded":"2024-03-28T23:32:34.026323+0000","last_fullsized":"2024-03-28T23:32:34.026323+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:30.992807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026952+0000","last_change":"2024-03-28T23:32:33.033045+0000","last_active":"2024-03-28T23:32:34.026952+0000","last_peered":"2024-03-28T23:32:34.026952+0000","last_clean":"2024-03-28T23:32:34.026952+0000","last_became_active":"2024-03-28T23:32:33.032771+0000","last_became_peered":"2024-03-28T23:32:33.032771+0000","last_unstale":"2024-03-28T23:32:34.026952+0000","last_undegraded":"2024-03-28T23:32:34.026952+0000","last_fullsized":"2024-03-28T23:32:34.026952+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:43.341877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"72'2","reported_seq":16,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.025905+0000","last_change":"2024-03-28T23:32:32.032053+0000","last_active":"2024-03-28T23:32:34.025905+0000","last_peered":"2024-03-28T23:32:34.025905+0000","last_clean":"2024-03-28T23:32:34.025905+0000","last_became_active":"2024-03-28T23:32:32.031741+0000","last_became_peered":"2024-03-28T23:32:32.031741+0000","last_unstale":"2024-03-28T23:32:34.025905+0000","last_undegraded":"2024-03-28T23:32:34.025905+0000","last_fullsized":"2024-03-28T23:32:34.025905+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:20.456721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026720+0000","last_change":"2024-03-28T23:32:33.039022+0000","last_active":"2024-03-28T23:32:34.026720+0000","last_peered":"2024-03-28T23:32:34.026720+0000","last_clean":"2024-03-28T23:32:34.026720+0000","last_became_active":"2024-03-28T23:32:33.038508+0000","last_became_peered":"2024-03-28T23:32:33.038508+0000","last_unstale":"2024-03-28T23:32:34.026720+0000","last_undegraded":"2024-03-28T23:32:34.026720+0000","last_fullsized":"2024-03-28T23:32:34.026720+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:25:45.189859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026863+0000","last_change":"2024-03-28T23:32:32.041042+0000","last_active":"2024-03-28T23:32:34.026863+0000","last_peered":"2024-03-28T23:32:34.026863+0000","last_clean":"2024-03-28T23:32:34.026863+0000","last_became_active":"2024-03-28T23:32:32.040686+0000","last_became_peered":"2024-03-28T23:32:32.040686+0000","last_unstale":"2024-03-28T23:32:34.026863+0000","last_undegraded":"2024-03-28T23:32:34.026863+0000","last_fullsized":"2024-03-28T23:32:34.026863+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:46.343619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.025919+0000","last_change":"2024-03-28T23:32:33.034262+0000","last_active":"2024-03-28T23:32:34.025919+0000","last_peered":"2024-03-28T23:32:34.025919+0000","last_clean":"2024-03-28T23:32:34.025919+0000","last_became_active":"2024-03-28T23:32:33.034003+0000","last_became_peered":"2024-03-28T23:32:33.034003+0000","last_unstale":"2024-03-28T23:32:34.025919+0000","last_undegraded":"2024-03-28T23:32:34.025919+0000","last_fullsized":"2024-03-28T23:32:34.025919+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:41.257241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.030694+0000","last_change":"2024-03-28T23:32:32.039031+0000","last_active":"2024-03-28T23:32:34.030694+0000","last_peered":"2024-03-28T23:32:34.030694+0000","last_clean":"2024-03-28T23:32:34.030694+0000","last_became_active":"2024-03-28T23:32:32.038766+0000","last_became_peered":"2024-03-28T23:32:32.038766+0000","last_unstale":"2024-03-28T23:32:34.030694+0000","last_undegraded":"2024-03-28T23:32:34.030694+0000","last_fullsized":"2024-03-28T23:32:34.030694+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:13.872794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026193+0000","last_change":"2024-03-28T23:32:33.043171+0000","last_active":"2024-03-28T23:32:34.026193+0000","last_peered":"2024-03-28T23:32:34.026193+0000","last_clean":"2024-03-28T23:32:34.026193+0000","last_became_active":"2024-03-28T23:32:33.042821+0000","last_became_peered":"2024-03-28T23:32:33.042821+0000","last_unstale":"2024-03-28T23:32:34.026193+0000","last_undegraded":"2024-03-28T23:32:34.026193+0000","last_fullsized":"2024-03-28T23:32:34.026193+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:21.878156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"72'8","reported_seq":22,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028646+0000","last_change":"2024-03-28T23:32:32.040166+0000","last_active":"2024-03-28T23:32:34.028646+0000","last_peered":"2024-03-28T23:32:34.028646+0000","last_clean":"2024-03-28T23:32:34.028646+0000","last_became_active":"2024-03-28T23:32:32.039560+0000","last_became_peered":"2024-03-28T23:32:32.039560+0000","last_unstale":"2024-03-28T23:32:34.028646+0000","last_undegraded":"2024-03-28T23:32:34.028646+0000","last_fullsized":"2024-03-28T23:32:34.028646+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:55.544945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057158+0000","last_change":"2024-03-28T23:32:33.043192+0000","last_active":"2024-03-28T23:32:34.057158+0000","last_peered":"2024-03-28T23:32:34.057158+0000","last_clean":"2024-03-28T23:32:34.057158+0000","last_became_active":"2024-03-28T23:32:33.042894+0000","last_became_peered":"2024-03-28T23:32:33.042894+0000","last_unstale":"2024-03-28T23:32:34.057158+0000","last_undegraded":"2024-03-28T23:32:34.057158+0000","last_fullsized":"2024-03-28T23:32:34.057158+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:32:08.473276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"72'1","reported_seq":15,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.027885+0000","last_change":"2024-03-28T23:32:32.041892+0000","last_active":"2024-03-28T23:32:34.027885+0000","last_peered":"2024-03-28T23:32:34.027885+0000","last_clean":"2024-03-28T23:32:34.027885+0000","last_became_active":"2024-03-28T23:32:32.041533+0000","last_became_peered":"2024-03-28T23:32:32.041533+0000","last_unstale":"2024-03-28T23:32:34.027885+0000","last_undegraded":"2024-03-28T23:32:34.027885+0000","last_fullsized":"2024-03-28T23:32:34.027885+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:15.843269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.030716+0000","last_change":"2024-03-28T23:32:33.046899+0000","last_active":"2024-03-28T23:32:34.030716+0000","last_peered":"2024-03-28T23:32:34.030716+0000","last_clean":"2024-03-28T23:32:34.030716+0000","last_became_active":"2024-03-28T23:32:33.046579+0000","last_became_peered":"2024-03-28T23:32:33.046579+0000","last_unstale":"2024-03-28T23:32:34.030716+0000","last_undegraded":"2024-03-28T23:32:34.030716+0000","last_fullsized":"2024-03-28T23:32:34.030716+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:08:06.339997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"72'2","reported_seq":16,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.058082+0000","last_change":"2024-03-28T23:32:32.031135+0000","last_active":"2024-03-28T23:32:34.058082+0000","last_peered":"2024-03-28T23:32:34.058082+0000","last_clean":"2024-03-28T23:32:34.058082+0000","last_became_active":"2024-03-28T23:32:32.030796+0000","last_became_peered":"2024-03-28T23:32:32.030796+0000","last_unstale":"2024-03-28T23:32:34.058082+0000","last_undegraded":"2024-03-28T23:32:34.058082+0000","last_fullsized":"2024-03-28T23:32:34.058082+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:18:21.816565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.030751+0000","last_change":"2024-03-28T23:32:33.053549+0000","last_active":"2024-03-28T23:32:34.030751+0000","last_peered":"2024-03-28T23:32:34.030751+0000","last_clean":"2024-03-28T23:32:34.030751+0000","last_became_active":"2024-03-28T23:32:33.053229+0000","last_became_peered":"2024-03-28T23:32:33.053229+0000","last_unstale":"2024-03-28T23:32:34.030751+0000","last_undegraded":"2024-03-28T23:32:34.030751+0000","last_fullsized":"2024-03-28T23:32:34.030751+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:41:02.517121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"72'1","reported_seq":15,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.031169+0000","last_change":"2024-03-28T23:32:32.038261+0000","last_active":"2024-03-28T23:32:34.031169+0000","last_peered":"2024-03-28T23:32:34.031169+0000","last_clean":"2024-03-28T23:32:34.031169+0000","last_became_active":"2024-03-28T23:32:32.037971+0000","last_became_peered":"2024-03-28T23:32:32.037971+0000","last_unstale":"2024-03-28T23:32:34.031169+0000","last_undegraded":"2024-03-28T23:32:34.031169+0000","last_fullsized":"2024-03-28T23:32:34.031169+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:25:10.296139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057837+0000","last_change":"2024-03-28T23:32:33.043954+0000","last_active":"2024-03-28T23:32:34.057837+0000","last_peered":"2024-03-28T23:32:34.057837+0000","last_clean":"2024-03-28T23:32:34.057837+0000","last_became_active":"2024-03-28T23:32:33.043649+0000","last_became_peered":"2024-03-28T23:32:33.043649+0000","last_unstale":"2024-03-28T23:32:34.057837+0000","last_undegraded":"2024-03-28T23:32:34.057837+0000","last_fullsized":"2024-03-28T23:32:34.057837+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:00.358180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057486+0000","last_change":"2024-03-28T23:32:32.033720+0000","last_active":"2024-03-28T23:32:34.057486+0000","last_peered":"2024-03-28T23:32:34.057486+0000","last_clean":"2024-03-28T23:32:34.057486+0000","last_became_active":"2024-03-28T23:32:32.033422+0000","last_became_peered":"2024-03-28T23:32:32.033422+0000","last_unstale":"2024-03-28T23:32:34.057486+0000","last_undegraded":"2024-03-28T23:32:34.057486+0000","last_fullsized":"2024-03-28T23:32:34.057486+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:51:10.440109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028575+0000","last_change":"2024-03-28T23:32:33.045618+0000","last_active":"2024-03-28T23:32:34.028575+0000","last_peered":"2024-03-28T23:32:34.028575+0000","last_clean":"2024-03-28T23:32:34.028575+0000","last_became_active":"2024-03-28T23:32:33.045324+0000","last_became_peered":"2024-03-28T23:32:33.045324+0000","last_unstale":"2024-03-28T23:32:34.028575+0000","last_undegraded":"2024-03-28T23:32:34.028575+0000","last_fullsized":"2024-03-28T23:32:34.028575+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:46:43.644886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"72'2","reported_seq":16,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057846+0000","last_change":"2024-03-28T23:32:32.031574+0000","last_active":"2024-03-28T23:32:34.057846+0000","last_peered":"2024-03-28T23:32:34.057846+0000","last_clean":"2024-03-28T23:32:34.057846+0000","last_became_active":"2024-03-28T23:32:32.031310+0000","last_became_peered":"2024-03-28T23:32:32.031310+0000","last_unstale":"2024-03-28T23:32:34.057846+0000","last_undegraded":"2024-03-28T23:32:34.057846+0000","last_fullsized":"2024-03-28T23:32:34.057846+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:48:07.073481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028782+0000","last_change":"2024-03-28T23:32:33.040983+0000","last_active":"2024-03-28T23:32:34.028782+0000","last_peered":"2024-03-28T23:32:34.028782+0000","last_clean":"2024-03-28T23:32:34.028782+0000","last_became_active":"2024-03-28T23:32:33.040693+0000","last_became_peered":"2024-03-28T23:32:33.040693+0000","last_unstale":"2024-03-28T23:32:34.028782+0000","last_undegraded":"2024-03-28T23:32:34.028782+0000","last_fullsized":"2024-03-28T23:32:34.028782+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:22:05.201764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057575+0000","last_change":"2024-03-28T23:32:32.036998+0000","last_active":"2024-03-28T23:32:34.057575+0000","last_peered":"2024-03-28T23:32:34.057575+0000","last_clean":"2024-03-28T23:32:34.057575+0000","last_became_active":"2024-03-28T23:32:32.036733+0000","last_became_peered":"2024-03-28T23:32:32.036733+0000","last_unstale":"2024-03-28T23:32:34.057575+0000","last_undegraded":"2024-03-28T23:32:34.057575+0000","last_fullsized":"2024-03-28T23:32:34.057575+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:42.334479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026358+0000","last_change":"2024-03-28T23:32:33.041232+0000","last_active":"2024-03-28T23:32:34.026358+0000","last_peered":"2024-03-28T23:32:34.026358+0000","last_clean":"2024-03-28T23:32:34.026358+0000","last_became_active":"2024-03-28T23:32:33.040132+0000","last_became_peered":"2024-03-28T23:32:33.040132+0000","last_unstale":"2024-03-28T23:32:34.026358+0000","last_undegraded":"2024-03-28T23:32:34.026358+0000","last_fullsized":"2024-03-28T23:32:34.026358+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:39:02.824323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057248+0000","last_change":"2024-03-28T23:32:32.036416+0000","last_active":"2024-03-28T23:32:34.057248+0000","last_peered":"2024-03-28T23:32:34.057248+0000","last_clean":"2024-03-28T23:32:34.057248+0000","last_became_active":"2024-03-28T23:32:32.036131+0000","last_became_peered":"2024-03-28T23:32:32.036131+0000","last_unstale":"2024-03-28T23:32:34.057248+0000","last_undegraded":"2024-03-28T23:32:34.057248+0000","last_fullsized":"2024-03-28T23:32:34.057248+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:13:33.914342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057703+0000","last_change":"2024-03-28T23:32:33.035015+0000","last_active":"2024-03-28T23:32:34.057703+0000","last_peered":"2024-03-28T23:32:34.057703+0000","last_clean":"2024-03-28T23:32:34.057703+0000","last_became_active":"2024-03-28T23:32:33.034730+0000","last_became_peered":"2024-03-28T23:32:33.034730+0000","last_unstale":"2024-03-28T23:32:34.057703+0000","last_undegraded":"2024-03-28T23:32:34.057703+0000","last_fullsized":"2024-03-28T23:32:34.057703+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:09.477928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"72'2","reported_seq":16,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026283+0000","last_change":"2024-03-28T23:32:32.032596+0000","last_active":"2024-03-28T23:32:34.026283+0000","last_peered":"2024-03-28T23:32:34.026283+0000","last_clean":"2024-03-28T23:32:34.026283+0000","last_became_active":"2024-03-28T23:32:32.032336+0000","last_became_peered":"2024-03-28T23:32:32.032336+0000","last_unstale":"2024-03-28T23:32:34.026283+0000","last_undegraded":"2024-03-28T23:32:34.026283+0000","last_fullsized":"2024-03-28T23:32:34.026283+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:28:25.356704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026959+0000","last_change":"2024-03-28T23:32:33.044243+0000","last_active":"2024-03-28T23:32:34.026959+0000","last_peered":"2024-03-28T23:32:34.026959+0000","last_clean":"2024-03-28T23:32:34.026959+0000","last_became_active":"2024-03-28T23:32:33.040218+0000","last_became_peered":"2024-03-28T23:32:33.040218+0000","last_unstale":"2024-03-28T23:32:34.026959+0000","last_undegraded":"2024-03-28T23:32:34.026959+0000","last_fullsized":"2024-03-28T23:32:34.026959+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:38:51.513907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"73'1","reported_seq":15,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.029923+0000","last_change":"2024-03-28T23:32:32.041586+0000","last_active":"2024-03-28T23:32:34.029923+0000","last_peered":"2024-03-28T23:32:34.029923+0000","last_clean":"2024-03-28T23:32:34.029923+0000","last_became_active":"2024-03-28T23:32:32.041327+0000","last_became_peered":"2024-03-28T23:32:32.041327+0000","last_unstale":"2024-03-28T23:32:34.029923+0000","last_undegraded":"2024-03-28T23:32:34.029923+0000","last_fullsized":"2024-03-28T23:32:34.029923+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:06:20.608980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057799+0000","last_change":"2024-03-28T23:32:33.042272+0000","last_active":"2024-03-28T23:32:34.057799+0000","last_peered":"2024-03-28T23:32:34.057799+0000","last_clean":"2024-03-28T23:32:34.057799+0000","last_became_active":"2024-03-28T23:32:33.037633+0000","last_became_peered":"2024-03-28T23:32:33.037633+0000","last_unstale":"2024-03-28T23:32:34.057799+0000","last_undegraded":"2024-03-28T23:32:34.057799+0000","last_fullsized":"2024-03-28T23:32:34.057799+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:45.773838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.027762+0000","last_change":"2024-03-28T23:32:32.037001+0000","last_active":"2024-03-28T23:32:34.027762+0000","last_peered":"2024-03-28T23:32:34.027762+0000","last_clean":"2024-03-28T23:32:34.027762+0000","last_became_active":"2024-03-28T23:32:32.036739+0000","last_became_peered":"2024-03-28T23:32:32.036739+0000","last_unstale":"2024-03-28T23:32:34.027762+0000","last_undegraded":"2024-03-28T23:32:34.027762+0000","last_fullsized":"2024-03-28T23:32:34.027762+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:06:20.608980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.028491+0000","last_change":"2024-03-28T23:32:33.041123+0000","last_active":"2024-03-28T23:32:34.028491+0000","last_peered":"2024-03-28T23:32:34.028491+0000","last_clean":"2024-03-28T23:32:34.028491+0000","last_became_active":"2024-03-28T23:32:33.040505+0000","last_became_peered":"2024-03-28T23:32:33.040505+0000","last_unstale":"2024-03-28T23:32:34.028491+0000","last_undegraded":"2024-03-28T23:32:34.028491+0000","last_fullsized":"2024-03-28T23:32:34.028491+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:54.194453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.027417+0000","last_change":"2024-03-28T23:32:32.036857+0000","last_active":"2024-03-28T23:32:34.027417+0000","last_peered":"2024-03-28T23:32:34.027417+0000","last_clean":"2024-03-28T23:32:34.027417+0000","last_became_active":"2024-03-28T23:32:32.036576+0000","last_became_peered":"2024-03-28T23:32:32.036576+0000","last_unstale":"2024-03-28T23:32:34.027417+0000","last_undegraded":"2024-03-28T23:32:34.027417+0000","last_fullsized":"2024-03-28T23:32:34.027417+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:43.073425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026102+0000","last_change":"2024-03-28T23:32:33.041769+0000","last_active":"2024-03-28T23:32:34.026102+0000","last_peered":"2024-03-28T23:32:34.026102+0000","last_clean":"2024-03-28T23:32:34.026102+0000","last_became_active":"2024-03-28T23:32:33.041475+0000","last_became_peered":"2024-03-28T23:32:33.041475+0000","last_unstale":"2024-03-28T23:32:34.026102+0000","last_undegraded":"2024-03-28T23:32:34.026102+0000","last_fullsized":"2024-03-28T23:32:34.026102+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:38.859031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026133+0000","last_change":"2024-03-28T23:32:32.037824+0000","last_active":"2024-03-28T23:32:34.026133+0000","last_peered":"2024-03-28T23:32:34.026133+0000","last_clean":"2024-03-28T23:32:34.026133+0000","last_became_active":"2024-03-28T23:32:32.037550+0000","last_became_peered":"2024-03-28T23:32:32.037550+0000","last_unstale":"2024-03-28T23:32:34.026133+0000","last_undegraded":"2024-03-28T23:32:34.026133+0000","last_fullsized":"2024-03-28T23:32:34.026133+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:43.467214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026576+0000","last_change":"2024-03-28T23:32:33.048966+0000","last_active":"2024-03-28T23:32:34.026576+0000","last_peered":"2024-03-28T23:32:34.026576+0000","last_clean":"2024-03-28T23:32:34.026576+0000","last_became_active":"2024-03-28T23:32:33.048362+0000","last_became_peered":"2024-03-28T23:32:33.048362+0000","last_unstale":"2024-03-28T23:32:34.026576+0000","last_undegraded":"2024-03-28T23:32:34.026576+0000","last_fullsized":"2024-03-28T23:32:34.026576+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:25.849919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.030791+0000","last_change":"2024-03-28T23:32:32.038885+0000","last_active":"2024-03-28T23:32:34.030791+0000","last_peered":"2024-03-28T23:32:34.030791+0000","last_clean":"2024-03-28T23:32:34.030791+0000","last_became_active":"2024-03-28T23:32:32.038364+0000","last_became_peered":"2024-03-28T23:32:32.038364+0000","last_unstale":"2024-03-28T23:32:34.030791+0000","last_undegraded":"2024-03-28T23:32:34.030791+0000","last_fullsized":"2024-03-28T23:32:34.030791+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:16:06.156877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057644+0000","last_change":"2024-03-28T23:32:33.049584+0000","last_active":"2024-03-28T23:32:34.057644+0000","last_peered":"2024-03-28T23:32:34.057644+0000","last_clean":"2024-03-28T23:32:34.057644+0000","last_became_active":"2024-03-28T23:32:33.049342+0000","last_became_peered":"2024-03-28T23:32:33.049342+0000","last_unstale":"2024-03-28T23:32:34.057644+0000","last_undegraded":"2024-03-28T23:32:34.057644+0000","last_fullsized":"2024-03-28T23:32:34.057644+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:49.711750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.027122+0000","last_change":"2024-03-28T23:32:32.034044+0000","last_active":"2024-03-28T23:32:34.027122+0000","last_peered":"2024-03-28T23:32:34.027122+0000","last_clean":"2024-03-28T23:32:34.027122+0000","last_became_active":"2024-03-28T23:32:32.033779+0000","last_became_peered":"2024-03-28T23:32:32.033779+0000","last_unstale":"2024-03-28T23:32:34.027122+0000","last_undegraded":"2024-03-28T23:32:34.027122+0000","last_fullsized":"2024-03-28T23:32:34.027122+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:56:22.159708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057558+0000","last_change":"2024-03-28T23:32:33.041180+0000","last_active":"2024-03-28T23:32:34.057558+0000","last_peered":"2024-03-28T23:32:34.057558+0000","last_clean":"2024-03-28T23:32:34.057558+0000","last_became_active":"2024-03-28T23:32:33.040922+0000","last_became_peered":"2024-03-28T23:32:33.040922+0000","last_unstale":"2024-03-28T23:32:34.057558+0000","last_undegraded":"2024-03-28T23:32:34.057558+0000","last_fullsized":"2024-03-28T23:32:34.057558+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:25:45.189859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.057186+0000","last_change":"2024-03-28T23:32:32.034328+0000","last_active":"2024-03-28T23:32:34.057186+0000","last_peered":"2024-03-28T23:32:34.057186+0000","last_clean":"2024-03-28T23:32:34.057186+0000","last_became_active":"2024-03-28T23:32:32.034057+0000","last_became_peered":"2024-03-28T23:32:32.034057+0000","last_unstale":"2024-03-28T23:32:34.057186+0000","last_undegraded":"2024-03-28T23:32:34.057186+0000","last_fullsized":"2024-03-28T23:32:34.057186+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:54:12.775407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.027663+0000","last_change":"2024-03-28T23:32:33.038830+0000","last_active":"2024-03-28T23:32:34.027663+0000","last_peered":"2024-03-28T23:32:34.027663+0000","last_clean":"2024-03-28T23:32:34.027663+0000","last_became_active":"2024-03-28T23:32:33.038524+0000","last_became_peered":"2024-03-28T23:32:33.038524+0000","last_unstale":"2024-03-28T23:32:34.027663+0000","last_undegraded":"2024-03-28T23:32:34.027663+0000","last_fullsized":"2024-03-28T23:32:34.027663+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:55:53.512836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.058110+0000","last_change":"2024-03-28T23:32:32.034270+0000","last_active":"2024-03-28T23:32:34.058110+0000","last_peered":"2024-03-28T23:32:34.058110+0000","last_clean":"2024-03-28T23:32:34.058110+0000","last_became_active":"2024-03-28T23:32:32.034022+0000","last_became_peered":"2024-03-28T23:32:32.034022+0000","last_unstale":"2024-03-28T23:32:34.058110+0000","last_undegraded":"2024-03-28T23:32:34.058110+0000","last_fullsized":"2024-03-28T23:32:34.058110+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:15.683500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":269,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-28T23:32:34.026942+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:32:34.026942+0000","last_peered":"2024-03-28T23:32:34.026942+0000","last_clean":"2024-03-28T23:32:34.026942+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:32:34.026942+0000","last_undegraded":"2024-03-28T23:32:34.026942+0000","last_fullsized":"2024-03-28T23:32:34.026942+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574885,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32112,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339728,"statfs":{"total":96636764160,"available":96603881472,"internally_reserved":0,"allocated":950272,"data_stored":1004578,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574886,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32068,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339772,"statfs":{"total":96636764160,"available":96603926528,"internally_reserved":0,"allocated":905216,"data_stored":422492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32044,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339796,"statfs":{"total":96636764160,"available":96603951104,"internally_reserved":0,"allocated":884736,"data_stored":414222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574885,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32064,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339776,"statfs":{"total":96636764160,"available":96603930624,"internally_reserved":0,"allocated":901120,"data_stored":415904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":884736,"data_stored":414770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343944,"statfs":{"total":96636764160,"available":96608198656,"internally_reserved":0,"allocated":892928,"data_stored":414930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574885,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":888832,"data_stored":414300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574885,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":946176,"data_stored":1004556,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:32:37.509 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T23:32:37.510 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-28T23:32:37.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.654+0000 7fd7bef81640 1 Processor -- start 2024-03-28T23:32:37.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.654+0000 7fd7bef81640 1 -- start start 2024-03-28T23:32:37.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.655+0000 7fd7bef81640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd7b80c6750 0x7fd7b806bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:37.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.655+0000 7fd7bef81640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7b8152db0 con 0x7fd7b81529d0 2024-03-28T23:32:37.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.655+0000 7fd7bef81640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7b80c8d40 con 0x7fd7b80c88b0 2024-03-28T23:32:37.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.655+0000 7fd7bef81640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd7b80c7210 con 0x7fd7b80c6750 2024-03-28T23:32:37.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.655+0000 7fd7bcd1d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd7b81529d0 0x7fd7b8155c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:58488/0 (socket says 172.21.15.161:58488) 2024-03-28T23:32:37.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.655+0000 7fd7bcd1d640 1 -- 172.21.15.161:0/2309578392 learned_addr learned my addr 172.21.15.161:0/2309578392 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:37.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.655+0000 7fd7b7fff640 1 --2- 172.21.15.161:0/2309578392 >> v2:172.21.15.170:3300/0 conn(0x7fd7b80c6750 0x7fd7b806bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:37.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.655+0000 7fd7b77fe640 1 -- 172.21.15.161:0/2309578392 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 776646977 0 0) 0x7fd7b8152db0 con 0x7fd7b81529d0 2024-03-28T23:32:37.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.655+0000 7fd7b77fe640 1 -- 172.21.15.161:0/2309578392 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7a0003520 con 0x7fd7b81529d0 2024-03-28T23:32:37.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.656+0000 7fd7b77fe640 1 -- 172.21.15.161:0/2309578392 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 651467971 0 0) 0x7fd7a0003520 con 0x7fd7b81529d0 2024-03-28T23:32:37.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.656+0000 7fd7b77fe640 1 -- 172.21.15.161:0/2309578392 >> v2:172.21.15.170:3300/0 conn(0x7fd7b80c6750 msgr2=0x7fd7b806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:37.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.656+0000 7fd7b77fe640 1 --2- 172.21.15.161:0/2309578392 >> v2:172.21.15.170:3300/0 conn(0x7fd7b80c6750 0x7fd7b806bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.656+0000 7fd7b77fe640 1 -- 172.21.15.161:0/2309578392 >> v1:172.21.15.170:6789/0 conn(0x7fd7b80c88b0 legacy=0x7fd7b80c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:37.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.656+0000 7fd7b77fe640 1 -- 172.21.15.161:0/2309578392 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b80c6c50 con 0x7fd7b81529d0 2024-03-28T23:32:37.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.656+0000 7fd7b77fe640 1 -- 172.21.15.161:0/2309578392 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fd7b00019f0 con 0x7fd7b81529d0 2024-03-28T23:32:37.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.656+0000 7fd7b77fe640 1 -- 172.21.15.161:0/2309578392 >> v1:172.21.15.161:6789/0 conn(0x7fd7b81529d0 legacy=0x7fd7b8155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:37.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.656+0000 7fd7b77fe640 1 --2- 172.21.15.161:0/2309578392 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7a00039b0 0x7fd7a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:37.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.656+0000 7fd7b77fe640 1 --2- 172.21.15.161:0/2309578392 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7a0004800 0x7fd7a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:37.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.656+0000 7fd7b77fe640 1 --2- 172.21.15.161:0/2309578392 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7a00051c0 0x7fd7a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:37.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.656+0000 7fd7b77fe640 1 -- 172.21.15.161:0/2309578392 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b80c7210 con 0x7fd7a00039b0 2024-03-28T23:32:37.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.656+0000 7fd7b77fe640 1 -- 172.21.15.161:0/2309578392 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a000c060 con 0x7fd7a00051c0 2024-03-28T23:32:37.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.656+0000 7fd7b77fe640 1 -- 172.21.15.161:0/2309578392 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd7a000c260 con 0x7fd7a0004800 2024-03-28T23:32:37.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.656+0000 7fd7bcd1d640 1 --2- 172.21.15.161:0/2309578392 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7a00039b0 0x7fd7a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:37.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.656+0000 7fd7b7fff640 1 --2- 172.21.15.161:0/2309578392 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7a0004800 0x7fd7a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:37.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.656+0000 7fd7bd51e640 1 --2- 172.21.15.161:0/2309578392 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7a00051c0 0x7fd7a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:37.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.656+0000 7fd7bcd1d640 1 -- 172.21.15.161:0/2309578392 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7a0004800 msgr2=0x7fd7a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:37.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.656+0000 7fd7bcd1d640 1 --2- 172.21.15.161:0/2309578392 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7a0004800 0x7fd7a0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.656+0000 7fd7bcd1d640 1 -- 172.21.15.161:0/2309578392 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7a00051c0 msgr2=0x7fd7a000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:37.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.656+0000 7fd7bcd1d640 1 --2- 172.21.15.161:0/2309578392 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7a00051c0 0x7fd7a000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.656+0000 7fd7bd51e640 1 -- 172.21.15.161:0/2309578392 reap_dead start 2024-03-28T23:32:37.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.656+0000 7fd7bcd1d640 1 -- 172.21.15.161:0/2309578392 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7a0003520 con 0x7fd7a00039b0 2024-03-28T23:32:37.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.657+0000 7fd7bcd1d640 1 --2- 172.21.15.161:0/2309578392 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7a00039b0 0x7fd7a0003dd0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fd7b000a850 tx=0x7fd7a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=305a4eb430631ae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:37.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.657+0000 7fd7b77fe640 1 -- 172.21.15.161:0/2309578392 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b000a4a0 con 0x7fd7a00039b0 2024-03-28T23:32:37.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.657+0000 7fd7b77fe640 1 -- 172.21.15.161:0/2309578392 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7b0034650 con 0x7fd7a00039b0 2024-03-28T23:32:37.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.657+0000 7fd7b77fe640 1 -- 172.21.15.161:0/2309578392 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b0034a80 con 0x7fd7a00039b0 2024-03-28T23:32:37.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.658+0000 7fd7bef81640 1 -- 172.21.15.161:0/2309578392 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7a00039b0 msgr2=0x7fd7a0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:37.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.658+0000 7fd7bef81640 1 --2- 172.21.15.161:0/2309578392 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7a00039b0 0x7fd7a0003dd0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fd7b000a850 tx=0x7fd7a0005830 comp rx=0 tx=0).stop 2024-03-28T23:32:37.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.658+0000 7fd7bef81640 1 -- 172.21.15.161:0/2309578392 shutdown_connections 2024-03-28T23:32:37.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.658+0000 7fd7bef81640 1 --2- 172.21.15.161:0/2309578392 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7a00039b0 0x7fd7a0003dd0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.658+0000 7fd7bef81640 1 -- 172.21.15.161:0/2309578392 >> 172.21.15.161:0/2309578392 conn(0x7fd7b80c7430 msgr2=0x7fd7b80c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:37.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.658+0000 7fd7bef81640 1 -- 172.21.15.161:0/2309578392 shutdown_connections 2024-03-28T23:32:37.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.658+0000 7fd7bef81640 1 -- 172.21.15.161:0/2309578392 wait complete. 2024-03-28T23:32:37.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.659+0000 7fd7bef81640 1 Processor -- start 2024-03-28T23:32:37.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.659+0000 7fd7bef81640 1 -- start start 2024-03-28T23:32:37.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.660+0000 7fd7bef81640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7a00039b0 0x7fd7b8154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:37.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.660+0000 7fd7bef81640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7b80c43b0 0x7fd7b81ee2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:37.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.660+0000 7fd7bef81640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7b8159350 0x7fd7b8147b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:37.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.660+0000 7fd7bef81640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b8155710 con 0x7fd7b80c43b0 2024-03-28T23:32:37.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.660+0000 7fd7bef81640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b8155430 con 0x7fd7b8159350 2024-03-28T23:32:37.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.660+0000 7fd7bef81640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd7b80aa160 con 0x7fd7a00039b0 2024-03-28T23:32:37.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.660+0000 7fd7b7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7b80c43b0 0x7fd7b81ee2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:37.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.660+0000 7fd7b7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7b80c43b0 0x7fd7b81ee2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40320/0 (socket says 172.21.15.161:40320) 2024-03-28T23:32:37.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.660+0000 7fd7b7fff640 1 -- 172.21.15.161:0/3624451868 learned_addr learned my addr 172.21.15.161:0/3624451868 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:37.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.660+0000 7fd7b7fff640 1 -- 172.21.15.161:0/3624451868 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7a00039b0 msgr2=0x7fd7b8154450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:37.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.660+0000 7fd7b7fff640 1 --2- 172.21.15.161:0/3624451868 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7a00039b0 0x7fd7b8154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.660+0000 7fd7b7fff640 1 -- 172.21.15.161:0/3624451868 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7b8159350 msgr2=0x7fd7b8147b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:37.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.660+0000 7fd7b7fff640 1 --2- 172.21.15.161:0/3624451868 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7b8159350 0x7fd7b8147b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.660+0000 7fd7b7fff640 1 -- 172.21.15.161:0/3624451868 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b80c3e60 con 0x7fd7b80c43b0 2024-03-28T23:32:37.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.660+0000 7fd7b7fff640 1 --2- 172.21.15.161:0/3624451868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7b80c43b0 0x7fd7b81ee2c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac011a70 tx=0x7fd7ac011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:37.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.661+0000 7fd7b57fa640 1 -- 172.21.15.161:0/3624451868 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac00fad0 con 0x7fd7b80c43b0 2024-03-28T23:32:37.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.661+0000 7fd7b57fa640 1 -- 172.21.15.161:0/3624451868 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7ac00fc70 con 0x7fd7b80c43b0 2024-03-28T23:32:37.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.661+0000 7fd7b57fa640 1 -- 172.21.15.161:0/3624451868 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac014440 con 0x7fd7b80c43b0 2024-03-28T23:32:37.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.661+0000 7fd7bef81640 1 -- 172.21.15.161:0/3624451868 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7b814d7b0 con 0x7fd7b80c43b0 2024-03-28T23:32:37.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.661+0000 7fd7bef81640 1 -- 172.21.15.161:0/3624451868 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7b80c4140 con 0x7fd7b80c43b0 2024-03-28T23:32:37.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.662+0000 7fd7bef81640 1 -- 172.21.15.161:0/3624451868 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7b804f4b0 con 0x7fd7b80c43b0 2024-03-28T23:32:37.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.662+0000 7fd7b57fa640 1 -- 172.21.15.161:0/3624451868 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd7ac012040 con 0x7fd7b80c43b0 2024-03-28T23:32:37.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.662+0000 7fd7b57fa640 1 --2- 172.21.15.161:0/3624451868 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd7a406b4d0 0x7fd7a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:37.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.663+0000 7fd7b57fa640 1 -- 172.21.15.161:0/3624451868 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6101+0+0 (secure 0 0 0) 0x7fd7ac08a940 con 0x7fd7b80c43b0 2024-03-28T23:32:37.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.663+0000 7fd7bcd1d640 1 --2- 172.21.15.161:0/3624451868 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd7a406b4d0 0x7fd7a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:37.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.663+0000 7fd7bcd1d640 1 --2- 172.21.15.161:0/3624451868 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd7a406b4d0 0x7fd7a406d990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0034000 tx=0x7fd7b0009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:37.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.667+0000 7fd7b57fa640 1 -- 172.21.15.161:0/3624451868 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd7ac019030 con 0x7fd7b80c43b0 2024-03-28T23:32:37.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.913+0000 7fd7bef81640 1 -- 172.21.15.161:0/3624451868 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fd7b80c8180 con 0x7fd7b80c43b0 2024-03-28T23:32:37.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.915+0000 7fd7b57fa640 1 -- 172.21.15.161:0/3624451868 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7fd7ac059070 con 0x7fd7b80c43b0 2024-03-28T23:32:37.916 INFO:teuthology.orchestra.run.smithi161.stdout:[client.0] 2024-03-28T23:32:37.917 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQA6/QVmc1krAxAAEKsn1Tuuv9OlFZpB4PFhDg== 2024-03-28T23:32:37.917 INFO:teuthology.orchestra.run.smithi161.stdout: caps mds = "allow" 2024-03-28T23:32:37.917 INFO:teuthology.orchestra.run.smithi161.stdout: caps mon = "allow r" 2024-03-28T23:32:37.917 INFO:teuthology.orchestra.run.smithi161.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-28T23:32:37.917 INFO:teuthology.orchestra.run.smithi161.stderr:updated caps for client.0 2024-03-28T23:32:37.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.917+0000 7fd7bef81640 1 -- 172.21.15.161:0/3624451868 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd7a406b4d0 msgr2=0x7fd7a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:37.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.917+0000 7fd7bef81640 1 --2- 172.21.15.161:0/3624451868 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd7a406b4d0 0x7fd7a406d990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0034000 tx=0x7fd7b0009030 comp rx=0 tx=0).stop 2024-03-28T23:32:37.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.917+0000 7fd7bef81640 1 -- 172.21.15.161:0/3624451868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7b80c43b0 msgr2=0x7fd7b81ee2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:37.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.917+0000 7fd7bef81640 1 --2- 172.21.15.161:0/3624451868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7b80c43b0 0x7fd7b81ee2c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac011a70 tx=0x7fd7ac011f30 comp rx=0 tx=0).stop 2024-03-28T23:32:37.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.917+0000 7fd7bef81640 1 -- 172.21.15.161:0/3624451868 shutdown_connections 2024-03-28T23:32:37.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.917+0000 7fd7bef81640 1 --2- 172.21.15.161:0/3624451868 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd7a406b4d0 0x7fd7a406d990 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.917+0000 7fd7bef81640 1 --2- 172.21.15.161:0/3624451868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7b80c43b0 0x7fd7b81ee2c0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.917+0000 7fd7bef81640 1 --2- 172.21.15.161:0/3624451868 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7b8159350 0x7fd7b8147b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.917+0000 7fd7bef81640 1 --2- 172.21.15.161:0/3624451868 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7a00039b0 0x7fd7b8154450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:37.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.917+0000 7fd7bef81640 1 -- 172.21.15.161:0/3624451868 >> 172.21.15.161:0/3624451868 conn(0x7fd7b80c7430 msgr2=0x7fd7b814ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:37.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.917+0000 7fd7bef81640 1 -- 172.21.15.161:0/3624451868 shutdown_connections 2024-03-28T23:32:37.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:37.917+0000 7fd7bef81640 1 -- 172.21.15.161:0/3624451868 wait complete. 2024-03-28T23:32:37.930 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:32:38.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.031+0000 7f2e3c255640 1 Processor -- start 2024-03-28T23:32:38.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.032+0000 7f2e3c255640 1 -- start start 2024-03-28T23:32:38.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.032+0000 7f2e3c255640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2e3414eca0 0x7f2e340c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.032+0000 7f2e3c255640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2e340c1dc0 0x7f2e3406ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.032+0000 7f2e3c255640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e34151850 con 0x7f2e3406cf60 2024-03-28T23:32:38.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.032+0000 7f2e3c255640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2e34152460 con 0x7f2e3414eca0 2024-03-28T23:32:38.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.032+0000 7f2e3c255640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f2e34151b30 con 0x7f2e340c1dc0 2024-03-28T23:32:38.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.032+0000 7f2e39ff1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2e3414eca0 0x7f2e340c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.032+0000 7f2e39ff1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2e3414eca0 0x7f2e340c1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40326/0 (socket says 172.21.15.161:40326) 2024-03-28T23:32:38.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.032+0000 7f2e39ff1640 1 -- 172.21.15.161:0/740465140 learned_addr learned my addr 172.21.15.161:0/740465140 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:38.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.033+0000 7f2e397f0640 1 --2- 172.21.15.161:0/740465140 >> v2:172.21.15.170:3300/0 conn(0x7f2e340c1dc0 0x7f2e3406ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.033+0000 7f2e39ff1640 1 -- 172.21.15.161:0/740465140 >> v2:172.21.15.170:3300/0 conn(0x7f2e340c1dc0 msgr2=0x7f2e3406ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:38.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.033+0000 7f2e39ff1640 1 --2- 172.21.15.161:0/740465140 >> v2:172.21.15.170:3300/0 conn(0x7f2e340c1dc0 0x7f2e3406ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.033+0000 7f2e39ff1640 1 -- 172.21.15.161:0/740465140 >> v1:172.21.15.170:6789/0 conn(0x7f2e3406cf60 legacy=0x7f2e340a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:38.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.033+0000 7f2e39ff1640 1 -- 172.21.15.161:0/740465140 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e34151e10 con 0x7f2e3414eca0 2024-03-28T23:32:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.033+0000 7f2e39ff1640 1 --2- 172.21.15.161:0/740465140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e3414eca0 0x7f2e340c1880 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f2e340c3070 tx=0x7f2e2402deb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a36a44589db3bfe5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.034+0000 7f2e38fef640 1 -- 172.21.15.161:0/740465140 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e24034070 con 0x7f2e3414eca0 2024-03-28T23:32:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.034+0000 7f2e38fef640 1 -- 172.21.15.161:0/740465140 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e2402ec60 con 0x7f2e3414eca0 2024-03-28T23:32:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.034+0000 7f2e38fef640 1 -- 172.21.15.161:0/740465140 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e24038620 con 0x7f2e3414eca0 2024-03-28T23:32:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.034+0000 7f2e3c255640 1 -- 172.21.15.161:0/740465140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e3414eca0 msgr2=0x7f2e340c1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.034+0000 7f2e3c255640 1 --2- 172.21.15.161:0/740465140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e3414eca0 0x7f2e340c1880 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f2e340c3070 tx=0x7f2e2402deb0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.034+0000 7f2e3c255640 1 -- 172.21.15.161:0/740465140 shutdown_connections 2024-03-28T23:32:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.034+0000 7f2e3c255640 1 --2- 172.21.15.161:0/740465140 >> v2:172.21.15.170:3300/0 conn(0x7f2e340c1dc0 0x7f2e3406ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.034+0000 7f2e3c255640 1 --2- 172.21.15.161:0/740465140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e3414eca0 0x7f2e340c1880 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.034+0000 7f2e3c255640 1 -- 172.21.15.161:0/740465140 >> 172.21.15.161:0/740465140 conn(0x7f2e340c6670 msgr2=0x7f2e340c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:38.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.034+0000 7f2e3c255640 1 -- 172.21.15.161:0/740465140 shutdown_connections 2024-03-28T23:32:38.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.034+0000 7f2e3c255640 1 -- 172.21.15.161:0/740465140 wait complete. 2024-03-28T23:32:38.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.035+0000 7f2e3c255640 1 Processor -- start 2024-03-28T23:32:38.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.035+0000 7f2e3c255640 1 -- start start 2024-03-28T23:32:38.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.036+0000 7f2e3c255640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e3406cf60 0x7f2e34149e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.036+0000 7f2e3c255640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e340c1dc0 0x7f2e341e61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.036+0000 7f2e3c255640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e3414eca0 0x7f2e341e6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.036+0000 7f2e3c255640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2e341514b0 con 0x7f2e340c1dc0 2024-03-28T23:32:38.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.036+0000 7f2e3c255640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3414f080 con 0x7f2e3406cf60 2024-03-28T23:32:38.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.036+0000 7f2e3c255640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2e34151630 con 0x7f2e3414eca0 2024-03-28T23:32:38.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.036+0000 7f2e397f0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e340c1dc0 0x7f2e341e61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.036+0000 7f2e397f0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e340c1dc0 0x7f2e341e61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40332/0 (socket says 172.21.15.161:40332) 2024-03-28T23:32:38.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.036+0000 7f2e397f0640 1 -- 172.21.15.161:0/322677573 learned_addr learned my addr 172.21.15.161:0/322677573 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:38.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.036+0000 7f2e397f0640 1 -- 172.21.15.161:0/322677573 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e3414eca0 msgr2=0x7f2e341e6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:38.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.036+0000 7f2e3a7f2640 1 --2- 172.21.15.161:0/322677573 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e3414eca0 0x7f2e341e6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.036+0000 7f2e39ff1640 1 --2- 172.21.15.161:0/322677573 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e3406cf60 0x7f2e34149e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.036+0000 7f2e397f0640 1 --2- 172.21.15.161:0/322677573 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e3414eca0 0x7f2e341e6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.036+0000 7f2e397f0640 1 -- 172.21.15.161:0/322677573 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e3406cf60 msgr2=0x7f2e34149e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:38.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.036+0000 7f2e397f0640 1 --2- 172.21.15.161:0/322677573 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e3406cf60 0x7f2e34149e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.036+0000 7f2e397f0640 1 -- 172.21.15.161:0/322677573 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e3414a390 con 0x7f2e340c1dc0 2024-03-28T23:32:38.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.037+0000 7f2e39ff1640 1 --2- 172.21.15.161:0/322677573 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e3406cf60 0x7f2e34149e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:38.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.036+0000 7f2e3a7f2640 1 --2- 172.21.15.161:0/322677573 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e3414eca0 0x7f2e341e6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:38.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.037+0000 7f2e397f0640 1 --2- 172.21.15.161:0/322677573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e340c1dc0 0x7f2e341e61c0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f2e28007ef0 tx=0x7f2e2800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:38.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.037+0000 7f2e327fc640 1 -- 172.21.15.161:0/322677573 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e28014620 con 0x7f2e340c1dc0 2024-03-28T23:32:38.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.037+0000 7f2e327fc640 1 -- 172.21.15.161:0/322677573 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e28019070 con 0x7f2e340c1dc0 2024-03-28T23:32:38.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.037+0000 7f2e3c255640 1 -- 172.21.15.161:0/322677573 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e340c4fb0 con 0x7f2e340c1dc0 2024-03-28T23:32:38.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.037+0000 7f2e327fc640 1 -- 172.21.15.161:0/322677573 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e28014a00 con 0x7f2e340c1dc0 2024-03-28T23:32:38.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.037+0000 7f2e3c255640 1 -- 172.21.15.161:0/322677573 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e340c5250 con 0x7f2e340c1dc0 2024-03-28T23:32:38.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.038+0000 7f2e3c255640 1 -- 172.21.15.161:0/322677573 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e3404f4b0 con 0x7f2e340c1dc0 2024-03-28T23:32:38.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.038+0000 7f2e327fc640 1 -- 172.21.15.161:0/322677573 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2e28014ba0 con 0x7f2e340c1dc0 2024-03-28T23:32:38.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.039+0000 7f2e327fc640 1 --2- 172.21.15.161:0/322677573 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e2006b4d0 0x7f2e2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.039+0000 7f2e327fc640 1 -- 172.21.15.161:0/322677573 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6101+0+0 (secure 0 0 0) 0x7f2e2808acb0 con 0x7f2e340c1dc0 2024-03-28T23:32:38.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.039+0000 7f2e39ff1640 1 --2- 172.21.15.161:0/322677573 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e2006b4d0 0x7f2e2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.039+0000 7f2e39ff1640 1 --2- 172.21.15.161:0/322677573 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e2006b4d0 0x7f2e2006d990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f2e24039000 tx=0x7f2e24034000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:38.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.043+0000 7f2e327fc640 1 -- 172.21.15.161:0/322677573 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2e3414a390 con 0x7f2e340c1dc0 2024-03-28T23:32:38.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.252+0000 7f2e3c255640 1 -- 172.21.15.161:0/322677573 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e3406b2c0 con 0x7f2e340c1dc0 2024-03-28T23:32:38.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.253+0000 7f2e327fc640 1 -- 172.21.15.161:0/322677573 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+5192 (secure 0 0 0) 0x7f2e280599b0 con 0x7f2e340c1dc0 2024-03-28T23:32:38.255 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:38.255 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":56,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/737681942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":737681942},{"type":"v1","addr":"172.21.15.161:6837","nonce":737681942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":5579,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/3501962065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3501962065},{"type":"v1","addr":"172.21.15.170:6834","nonce":3501962065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6037,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2691084827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":2691084827},{"type":"v1","addr":"172.21.15.170:6835","nonce":2691084827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:32:32.372964+0000","modified":"2024-03-28T23:32:35.321715+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6468},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6468":{"gid":6468,"name":"a","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6836/3850309502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3850309502},{"type":"v1","addr":"172.21.15.161:6836","nonce":3850309502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6468,"qdb_cluster":[6468]},"id":5}]} 2024-03-28T23:32:38.255 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 56 2024-03-28T23:32:38.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.255+0000 7f2e3c255640 1 -- 172.21.15.161:0/322677573 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e2006b4d0 msgr2=0x7f2e2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:38.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.255+0000 7f2e3c255640 1 --2- 172.21.15.161:0/322677573 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e2006b4d0 0x7f2e2006d990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f2e24039000 tx=0x7f2e24034000 comp rx=0 tx=0).stop 2024-03-28T23:32:38.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.255+0000 7f2e3c255640 1 -- 172.21.15.161:0/322677573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e340c1dc0 msgr2=0x7f2e341e61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:38.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.255+0000 7f2e3c255640 1 --2- 172.21.15.161:0/322677573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e340c1dc0 0x7f2e341e61c0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f2e28007ef0 tx=0x7f2e2800e4b0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.255+0000 7f2e3c255640 1 -- 172.21.15.161:0/322677573 shutdown_connections 2024-03-28T23:32:38.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.255+0000 7f2e3c255640 1 --2- 172.21.15.161:0/322677573 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e2006b4d0 0x7f2e2006d990 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.255+0000 7f2e3c255640 1 --2- 172.21.15.161:0/322677573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e340c1dc0 0x7f2e341e61c0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.255+0000 7f2e3c255640 1 --2- 172.21.15.161:0/322677573 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e3414eca0 0x7f2e341e6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.255+0000 7f2e3c255640 1 --2- 172.21.15.161:0/322677573 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e3406cf60 0x7f2e34149e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.255+0000 7f2e3c255640 1 -- 172.21.15.161:0/322677573 >> 172.21.15.161:0/322677573 conn(0x7f2e340c6670 msgr2=0x7f2e340c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:38.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.255+0000 7f2e3c255640 1 -- 172.21.15.161:0/322677573 shutdown_connections 2024-03-28T23:32:38.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.255+0000 7f2e3c255640 1 -- 172.21.15.161:0/322677573 wait complete. 2024-03-28T23:32:38.302 INFO:tasks.ceph.mds.a.smithi161.stderr:2024-03-28T23:32:38.300+0000 7fcbbd324640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:32:38.671 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:32:32.372964+0000', 'modified': '2024-03-28T23:32:35.321715+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6468}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6468': {'gid': 6468, 'name': 'a', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.161:6836/3850309502', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 3850309502}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 3850309502}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6468, 'qdb_cluster': [6468]} 2024-03-28T23:32:38.671 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:32:38.671 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-28T23:32:38.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.773+0000 7f08c4230640 1 Processor -- start 2024-03-28T23:32:38.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.773+0000 7f08c4230640 1 -- start start 2024-03-28T23:32:38.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.774+0000 7f08c4230640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f08bc152890 0x7f08bc0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.774+0000 7f08c4230640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f08bc0c5b20 0x7f08bc1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.774+0000 7f08c4230640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08bc158a30 con 0x7f08bc0c8ed0 2024-03-28T23:32:38.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.774+0000 7f08c4230640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f08bc0c8480 con 0x7f08bc152890 2024-03-28T23:32:38.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.774+0000 7f08c4230640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f08bc159640 con 0x7f08bc0c5b20 2024-03-28T23:32:38.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.774+0000 7f08c1fcc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f08bc152890 0x7f08bc0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.774+0000 7f08c1fcc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f08bc152890 0x7f08bc0c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40348/0 (socket says 172.21.15.161:40348) 2024-03-28T23:32:38.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.774+0000 7f08c1fcc640 1 -- 172.21.15.161:0/1617874766 learned_addr learned my addr 172.21.15.161:0/1617874766 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:38.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.775+0000 7f08c27cd640 1 --2- 172.21.15.161:0/1617874766 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f08bc0c5b20 0x7f08bc1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.775+0000 7f08c1fcc640 1 -- 172.21.15.161:0/1617874766 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f08bc0c5b20 msgr2=0x7f08bc1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:38.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.775+0000 7f08c1fcc640 1 --2- 172.21.15.161:0/1617874766 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f08bc0c5b20 0x7f08bc1553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.775+0000 7f08c1fcc640 1 -- 172.21.15.161:0/1617874766 >> v1:172.21.15.170:6789/0 conn(0x7f08bc0c8ed0 legacy=0x7f08bc0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:38.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.775+0000 7f08c1fcc640 1 -- 172.21.15.161:0/1617874766 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08bc158d10 con 0x7f08bc152890 2024-03-28T23:32:38.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.775+0000 7f08c27cd640 1 --2- 172.21.15.161:0/1617874766 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f08bc0c5b20 0x7f08bc1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:38.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.775+0000 7f08c1fcc640 1 --2- 172.21.15.161:0/1617874766 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08bc152890 0x7f08bc0c8990 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f08bc1597c0 tx=0x7f08b802b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39dd3b1cef34b2b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:38.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.776+0000 7f08c0fca640 1 -- 172.21.15.161:0/1617874766 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b8038070 con 0x7f08bc152890 2024-03-28T23:32:38.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.776+0000 7f08c0fca640 1 -- 172.21.15.161:0/1617874766 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08b8031520 con 0x7f08bc152890 2024-03-28T23:32:38.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.776+0000 7f08c0fca640 1 -- 172.21.15.161:0/1617874766 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b80318f0 con 0x7f08bc152890 2024-03-28T23:32:38.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.776+0000 7f08c4230640 1 -- 172.21.15.161:0/1617874766 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08bc152890 msgr2=0x7f08bc0c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:38.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.776+0000 7f08c4230640 1 --2- 172.21.15.161:0/1617874766 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08bc152890 0x7f08bc0c8990 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f08bc1597c0 tx=0x7f08b802b7c0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.776+0000 7f08c4230640 1 -- 172.21.15.161:0/1617874766 shutdown_connections 2024-03-28T23:32:38.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.776+0000 7f08c4230640 1 --2- 172.21.15.161:0/1617874766 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f08bc0c5b20 0x7f08bc1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.776+0000 7f08c4230640 1 --2- 172.21.15.161:0/1617874766 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08bc152890 0x7f08bc0c8990 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.776+0000 7f08c4230640 1 -- 172.21.15.161:0/1617874766 >> 172.21.15.161:0/1617874766 conn(0x7f08bc0ac620 msgr2=0x7f08bc0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:38.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.776+0000 7f08c4230640 1 -- 172.21.15.161:0/1617874766 shutdown_connections 2024-03-28T23:32:38.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.776+0000 7f08c4230640 1 -- 172.21.15.161:0/1617874766 wait complete. 2024-03-28T23:32:38.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.778+0000 7f08c4230640 1 Processor -- start 2024-03-28T23:32:38.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.778+0000 7f08c4230640 1 -- start start 2024-03-28T23:32:38.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.778+0000 7f08c4230640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08bc0c5b20 0x7f08bc0c36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.778+0000 7f08c4230640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f08bc0c8ed0 0x7f08bc149010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.778+0000 7f08c4230640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08bc152890 0x7f08bc0bfe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.778+0000 7f08c4230640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc152c70 con 0x7f08bc152890 2024-03-28T23:32:38.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.778+0000 7f08c4230640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc1592d0 con 0x7f08bc0c5b20 2024-03-28T23:32:38.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.778+0000 7f08c4230640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f08bc152df0 con 0x7f08bc0c8ed0 2024-03-28T23:32:38.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.779+0000 7f08c27cd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08bc152890 0x7f08bc0bfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.779+0000 7f08c27cd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08bc152890 0x7f08bc0bfe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40356/0 (socket says 172.21.15.161:40356) 2024-03-28T23:32:38.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.779+0000 7f08c27cd640 1 -- 172.21.15.161:0/25286205 learned_addr learned my addr 172.21.15.161:0/25286205 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:38.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.779+0000 7f08c17cb640 1 --2- 172.21.15.161:0/25286205 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f08bc0c8ed0 0x7f08bc149010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.779+0000 7f08c1fcc640 1 --2- 172.21.15.161:0/25286205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08bc0c5b20 0x7f08bc0c36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.779+0000 7f08c27cd640 1 -- 172.21.15.161:0/25286205 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f08bc0c8ed0 msgr2=0x7f08bc149010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:38.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.779+0000 7f08c27cd640 1 --2- 172.21.15.161:0/25286205 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f08bc0c8ed0 0x7f08bc149010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.779+0000 7f08c27cd640 1 -- 172.21.15.161:0/25286205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08bc0c5b20 msgr2=0x7f08bc0c36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:38.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.779+0000 7f08c27cd640 1 --2- 172.21.15.161:0/25286205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08bc0c5b20 0x7f08bc0c36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.779+0000 7f08c27cd640 1 -- 172.21.15.161:0/25286205 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08bc0c4cf0 con 0x7f08bc152890 2024-03-28T23:32:38.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.779+0000 7f08c27cd640 1 --2- 172.21.15.161:0/25286205 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08bc152890 0x7f08bc0bfe50 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f08b400cd80 tx=0x7f08b400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:38.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.779+0000 7f08b27fc640 1 -- 172.21.15.161:0/25286205 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b4003b90 con 0x7f08bc152890 2024-03-28T23:32:38.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.779+0000 7f08b27fc640 1 -- 172.21.15.161:0/25286205 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08b4003d50 con 0x7f08bc152890 2024-03-28T23:32:38.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.779+0000 7f08c4230640 1 -- 172.21.15.161:0/25286205 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08bc0c2580 con 0x7f08bc152890 2024-03-28T23:32:38.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.779+0000 7f08b27fc640 1 -- 172.21.15.161:0/25286205 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b4004260 con 0x7f08bc152890 2024-03-28T23:32:38.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.779+0000 7f08c4230640 1 -- 172.21.15.161:0/25286205 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08bc14f910 con 0x7f08bc152890 2024-03-28T23:32:38.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.781+0000 7f08b27fc640 1 -- 172.21.15.161:0/25286205 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f08b4004400 con 0x7f08bc152890 2024-03-28T23:32:38.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.781+0000 7f08b27fc640 1 --2- 172.21.15.161:0/25286205 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f08a006b4d0 0x7f08a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.781+0000 7f08b27fc640 1 -- 172.21.15.161:0/25286205 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6101+0+0 (secure 0 0 0) 0x7f08b408acd0 con 0x7f08bc152890 2024-03-28T23:32:38.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.781+0000 7f08c1fcc640 1 --2- 172.21.15.161:0/25286205 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f08a006b4d0 0x7f08a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.782+0000 7f08c1fcc640 1 --2- 172.21.15.161:0/25286205 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f08a006b4d0 0x7f08a006d990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f08b8035000 tx=0x7f08b8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:38.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.782+0000 7f08c6e0c740 1 Processor -- start 2024-03-28T23:32:38.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.782+0000 7f08c6e0c740 1 -- start start 2024-03-28T23:32:38.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.783+0000 7f08c6e0c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026ba81db0 0x56026ba7f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.783+0000 7f08c6e0c740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56026ba82190 0x56026ba7fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.783+0000 7f08c6e0c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56026ba80310 0x56026ba7c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.783+0000 7f08c6e0c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x56026b9e5af0 con 0x56026ba81db0 2024-03-28T23:32:38.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.783+0000 7f08c6e0c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x56026b98e9d0 con 0x56026ba80310 2024-03-28T23:32:38.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.783+0000 7f08c6e0c740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x56026b9e1b90 con 0x56026ba82190 2024-03-28T23:32:38.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.783+0000 7f08c1fcc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026ba81db0 0x56026ba7f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.783+0000 7f08c1fcc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026ba81db0 0x56026ba7f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40368/0 (socket says 172.21.15.161:40368) 2024-03-28T23:32:38.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.783+0000 7f08c1fcc640 1 -- 172.21.15.161:0/1375930234 learned_addr learned my addr 172.21.15.161:0/1375930234 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:38.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.783+0000 7f08c1fcc640 1 -- 172.21.15.161:0/1375930234 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56026ba82190 msgr2=0x56026ba7fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:38.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.783+0000 7f08c1fcc640 1 --2- 172.21.15.161:0/1375930234 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56026ba82190 0x56026ba7fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.783+0000 7f08c1fcc640 1 -- 172.21.15.161:0/1375930234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56026ba80310 msgr2=0x56026ba7c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:38.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.783+0000 7f08c27cd640 1 --2- 172.21.15.161:0/1375930234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56026ba80310 0x56026ba7c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.783+0000 7f08c1fcc640 1 --2- 172.21.15.161:0/1375930234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56026ba80310 0x56026ba7c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.783+0000 7f08c1fcc640 1 -- 172.21.15.161:0/1375930234 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56026b6fa2d0 con 0x56026ba81db0 2024-03-28T23:32:38.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.784+0000 7f08c1fcc640 1 --2- 172.21.15.161:0/1375930234 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026ba81db0 0x56026ba7f890 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f08b802bdd0 tx=0x7f08b802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e016702341e20b1c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:38.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.784+0000 7f089f7fe640 1 -- 172.21.15.161:0/1375930234 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b8038040 con 0x56026ba81db0 2024-03-28T23:32:38.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.784+0000 7f089f7fe640 1 -- 172.21.15.161:0/1375930234 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08b803d610 con 0x56026ba81db0 2024-03-28T23:32:38.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.784+0000 7f089f7fe640 1 -- 172.21.15.161:0/1375930234 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b803d9c0 con 0x56026ba81db0 2024-03-28T23:32:38.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.784+0000 7f08c6e0c740 1 -- 172.21.15.161:0/1375930234 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026ba81db0 msgr2=0x56026ba7f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:38.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.784+0000 7f08c6e0c740 1 --2- 172.21.15.161:0/1375930234 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026ba81db0 0x56026ba7f890 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f08b802bdd0 tx=0x7f08b802e000 comp rx=0 tx=0).stop 2024-03-28T23:32:38.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.784+0000 7f08c6e0c740 1 -- 172.21.15.161:0/1375930234 shutdown_connections 2024-03-28T23:32:38.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.784+0000 7f08c6e0c740 1 --2- 172.21.15.161:0/1375930234 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56026ba80310 0x56026ba7c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.784+0000 7f08c6e0c740 1 --2- 172.21.15.161:0/1375930234 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56026ba82190 0x56026ba7fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.784+0000 7f08c6e0c740 1 --2- 172.21.15.161:0/1375930234 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026ba81db0 0x56026ba7f890 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.785+0000 7f08c6e0c740 1 -- 172.21.15.161:0/1375930234 >> 172.21.15.161:0/1375930234 conn(0x56026ba7e630 msgr2=0x56026baa3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:38.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.785+0000 7f08c6e0c740 1 -- 172.21.15.161:0/1375930234 shutdown_connections 2024-03-28T23:32:38.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.785+0000 7f08c6e0c740 1 -- 172.21.15.161:0/1375930234 wait complete. 2024-03-28T23:32:38.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.785+0000 7f08c6e0c740 1 -- start start 2024-03-28T23:32:38.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.785+0000 7f08c6e0c740 1 Processor -- start 2024-03-28T23:32:38.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.787+0000 7f08c6e0c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026ba81db0 0x56026babe070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.787+0000 7f08c6e0c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56026ba82190 0x56026bace330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.787+0000 7f08c6e0c740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56026bad21d0 0x56026ba9eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.787+0000 7f08c6e0c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x56026baa4090 con 0x56026ba81db0 2024-03-28T23:32:38.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.787+0000 7f08c6e0c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x56026b9a1f70 con 0x56026ba82190 2024-03-28T23:32:38.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.787+0000 7f08c6e0c740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x56026baa3ad0 con 0x56026bad21d0 2024-03-28T23:32:38.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.787+0000 7f08c6e0c740 10 client.0 _ll_register_callbacks cb 0x56026b6f9ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:32:38.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.787+0000 7f08c1fcc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026ba81db0 0x56026babe070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.787+0000 7f08c1fcc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026ba81db0 0x56026babe070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40384/0 (socket says 172.21.15.161:40384) 2024-03-28T23:32:38.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.787+0000 7f08c1fcc640 1 -- 172.21.15.161:0/2102407032 learned_addr learned my addr 172.21.15.161:0/2102407032 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:38.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.787+0000 7f08c27cd640 1 --2- 172.21.15.161:0/2102407032 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56026bad21d0 0x56026ba9eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.787+0000 7f08c17cb640 1 --2- 172.21.15.161:0/2102407032 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56026ba82190 0x56026bace330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.787+0000 7f08c1fcc640 1 -- 172.21.15.161:0/2102407032 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56026bad21d0 msgr2=0x56026ba9eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:38.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.787+0000 7f08c1fcc640 1 --2- 172.21.15.161:0/2102407032 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56026bad21d0 0x56026ba9eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.787+0000 7f08c1fcc640 1 -- 172.21.15.161:0/2102407032 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56026ba82190 msgr2=0x56026bace330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:38.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.787+0000 7f08c1fcc640 1 --2- 172.21.15.161:0/2102407032 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56026ba82190 0x56026bace330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.787+0000 7f08c1fcc640 1 -- 172.21.15.161:0/2102407032 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56026b7c46b0 con 0x56026ba81db0 2024-03-28T23:32:38.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.787+0000 7f08c1fcc640 1 --2- 172.21.15.161:0/2102407032 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026ba81db0 0x56026babe070 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f08b8006b10 tx=0x7f08b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d0e053db2bcadca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:38.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.787+0000 7f089cff9640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:32:38.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.788+0000 7f089cff9640 1 -- 172.21.15.161:0/2102407032 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b8038030 con 0x56026ba81db0 2024-03-28T23:32:38.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.788+0000 7f089cff9640 1 -- 172.21.15.161:0/2102407032 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 6101+0+0 (secure 0 0 0) 0x7f08b803d9d0 con 0x56026ba81db0 2024-03-28T23:32:38.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.788+0000 7f08c6e0c740 1 -- 172.21.15.161:0/2102407032 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08b803d610 con 0x56026ba81db0 2024-03-28T23:32:38.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.788+0000 7f089cff9640 1 -- 172.21.15.161:0/2102407032 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7f08b8063050 con 0x56026ba81db0 2024-03-28T23:32:38.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.788+0000 7f08c6e0c740 10 client.6609 fetch_fsmap learned FSMap version 56 2024-03-28T23:32:38.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.788+0000 7f08c6e0c740 1 -- 172.21.15.161:0/2102407032 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=2+}) -- 0x56026baa9720 con 0x56026ba81db0 2024-03-28T23:32:38.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.788+0000 7f089cff9640 1 -- 172.21.15.161:0/2102407032 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08b803d610 con 0x56026ba81db0 2024-03-28T23:32:38.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.788+0000 7f089cff9640 1 -- 172.21.15.161:0/2102407032 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 56) ==== 3494+0+0 (secure 0 0 0) 0x7f08b8066020 con 0x56026ba81db0 2024-03-28T23:32:38.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.788+0000 7f08c6e0c740 10 client.6609 fetch_fsmap finished waiting for FSMap version 56 2024-03-28T23:32:38.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.788+0000 7f08c6e0c740 10 client.6609 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T23:32:38.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.788+0000 7f08c6e0c740 20 client.6609 populate_metadata read hostname 'smithi161' 2024-03-28T23:32:38.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.788+0000 7f08c6e0c740 1 --2- 172.21.15.161:0/2102407032 >> [v2:172.21.15.161:6834/3850309502,v1:172.21.15.161:6836/3850309502] conn(0x56026baa6d00 0x56026baa70e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.788+0000 7f08c6e0c740 4 client.6609 mds_command: new command op to 6468 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:32:38.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.788+0000 7f08c6e0c740 1 -- 172.21.15.161:0/2102407032 --> [v2:172.21.15.161:6834/3850309502,v1:172.21.15.161:6836/3850309502] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56026b9f3030 con 0x56026baa6d00 2024-03-28T23:32:38.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.788+0000 7f08c17cb640 1 --2- 172.21.15.161:0/2102407032 >> [v2:172.21.15.161:6834/3850309502,v1:172.21.15.161:6836/3850309502] conn(0x56026baa6d00 0x56026baa70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.788+0000 7f089cff9640 10 client.6609 ms_handle_connect on v2:172.21.15.161:6834/3850309502 2024-03-28T23:32:38.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.788+0000 7f08c17cb640 1 --2- 172.21.15.161:0/2102407032 >> [v2:172.21.15.161:6834/3850309502,v1:172.21.15.161:6836/3850309502] conn(0x56026baa6d00 0x56026baa70e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7840de01f45cf62c server_cookie=ed93005165ab91e3 in_seq=0 out_seq=0 2024-03-28T23:32:38.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.789+0000 7f089cff9640 1 -- 172.21.15.161:0/2102407032 <== mds.0 v2:172.21.15.161:6834/3850309502 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56026b9f3030 con 0x56026baa6d00 2024-03-28T23:32:38.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.789+0000 7f089cff9640 10 client.6609 handle_command_reply: tid=0 2024-03-28T23:32:38.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.789+0000 7f08c6e0c740 1 client.6609 shutdown 2024-03-28T23:32:38.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.789+0000 7f08c6e0c740 1 -- 172.21.15.161:0/2102407032 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026ba81db0 msgr2=0x56026babe070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:38.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.789+0000 7f08c6e0c740 1 --2- 172.21.15.161:0/2102407032 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026ba81db0 0x56026babe070 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f08b8006b10 tx=0x7f08b8004730 comp rx=0 tx=0).stop 2024-03-28T23:32:38.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.789+0000 7f08c6e0c740 1 -- 172.21.15.161:0/2102407032 shutdown_connections 2024-03-28T23:32:38.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.789+0000 7f08c6e0c740 1 --2- 172.21.15.161:0/2102407032 >> [v2:172.21.15.161:6834/3850309502,v1:172.21.15.161:6836/3850309502] conn(0x56026baa6d00 0x56026baa70e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.789+0000 7f08c6e0c740 1 --2- 172.21.15.161:0/2102407032 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56026bad21d0 0x56026ba9eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.789+0000 7f089cff9640 0 client.6609 ms_handle_reset on v2:172.21.15.161:6834/3850309502 2024-03-28T23:32:38.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.789+0000 7f08c6e0c740 1 --2- 172.21.15.161:0/2102407032 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56026ba82190 0x56026bace330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.789+0000 7f08c6e0c740 1 --2- 172.21.15.161:0/2102407032 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026ba81db0 0x56026babe070 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.789+0000 7f08c6e0c740 1 -- 172.21.15.161:0/2102407032 >> 172.21.15.161:0/2102407032 conn(0x56026ba80310 msgr2=0x56026ba9fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:38.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.789+0000 7f08c6e0c740 1 -- 172.21.15.161:0/2102407032 shutdown_connections 2024-03-28T23:32:38.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.790+0000 7f08c6e0c740 1 -- 172.21.15.161:0/2102407032 wait complete. 2024-03-28T23:32:38.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.790+0000 7f08c6e0c740 20 client.6609 trim_cache size 0 max 16384 2024-03-28T23:32:38.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.824+0000 7f08c6e0c740 1 Processor -- start 2024-03-28T23:32:38.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.824+0000 7f08c6e0c740 1 -- start start 2024-03-28T23:32:38.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.825+0000 7f08c6e0c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56026ba81db0 0x56026ba7f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.825+0000 7f08c6e0c740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56026ba82190 0x56026babace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.825+0000 7f08c6e0c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026bad21d0 0x56026babde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.825+0000 7f08c6e0c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc1f6e80 con 0x56026bad21d0 2024-03-28T23:32:38.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.825+0000 7f08c6e0c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc148a70 con 0x56026ba81db0 2024-03-28T23:32:38.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.825+0000 7f08c6e0c740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f08bc1f68c0 con 0x56026ba82190 2024-03-28T23:32:38.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.826+0000 7f08c27cd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026bad21d0 0x56026babde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.826+0000 7f08c27cd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026bad21d0 0x56026babde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40400/0 (socket says 172.21.15.161:40400) 2024-03-28T23:32:38.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.826+0000 7f08c27cd640 1 -- 172.21.15.161:0/3243186247 learned_addr learned my addr 172.21.15.161:0/3243186247 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:38.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.826+0000 7f08c1fcc640 1 --2- 172.21.15.161:0/3243186247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56026ba81db0 0x56026ba7f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.826+0000 7f08c17cb640 1 --2- 172.21.15.161:0/3243186247 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56026ba82190 0x56026babace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.826+0000 7f08c27cd640 1 -- 172.21.15.161:0/3243186247 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56026ba82190 msgr2=0x56026babace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.826+0000 7f08c27cd640 1 --2- 172.21.15.161:0/3243186247 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56026ba82190 0x56026babace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.826+0000 7f08c27cd640 1 -- 172.21.15.161:0/3243186247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56026ba81db0 msgr2=0x56026ba7f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.826+0000 7f08c27cd640 1 --2- 172.21.15.161:0/3243186247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56026ba81db0 0x56026ba7f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.826+0000 7f08c27cd640 1 -- 172.21.15.161:0/3243186247 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56026ba7d420 con 0x56026bad21d0 2024-03-28T23:32:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.826+0000 7f08c27cd640 1 --2- 172.21.15.161:0/3243186247 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026bad21d0 0x56026babde80 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f08b40557d0 tx=0x7f08b4007950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=503627c066005805 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.826+0000 7f089effd640 1 -- 172.21.15.161:0/3243186247 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b4087070 con 0x56026bad21d0 2024-03-28T23:32:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.826+0000 7f089effd640 1 -- 172.21.15.161:0/3243186247 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08b4057670 con 0x56026bad21d0 2024-03-28T23:32:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.826+0000 7f089effd640 1 -- 172.21.15.161:0/3243186247 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b401d070 con 0x56026bad21d0 2024-03-28T23:32:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.827+0000 7f08c6e0c740 1 -- 172.21.15.161:0/3243186247 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026bad21d0 msgr2=0x56026babde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.827+0000 7f08c6e0c740 1 --2- 172.21.15.161:0/3243186247 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026bad21d0 0x56026babde80 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f08b40557d0 tx=0x7f08b4007950 comp rx=0 tx=0).stop 2024-03-28T23:32:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.827+0000 7f08c6e0c740 1 -- 172.21.15.161:0/3243186247 shutdown_connections 2024-03-28T23:32:38.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.827+0000 7f08c6e0c740 1 --2- 172.21.15.161:0/3243186247 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026bad21d0 0x56026babde80 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.827+0000 7f08c6e0c740 1 --2- 172.21.15.161:0/3243186247 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56026ba82190 0x56026babace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.827+0000 7f08c6e0c740 1 --2- 172.21.15.161:0/3243186247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56026ba81db0 0x56026ba7f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.827+0000 7f08c6e0c740 1 -- 172.21.15.161:0/3243186247 >> 172.21.15.161:0/3243186247 conn(0x56026ba80310 msgr2=0x56026bad16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:38.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.827+0000 7f08c6e0c740 1 -- 172.21.15.161:0/3243186247 shutdown_connections 2024-03-28T23:32:38.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.827+0000 7f08c6e0c740 1 -- 172.21.15.161:0/3243186247 wait complete. 2024-03-28T23:32:38.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.827+0000 7f08c6e0c740 1 -- start start 2024-03-28T23:32:38.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.828+0000 7f08c6e0c740 1 Processor -- start 2024-03-28T23:32:38.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.828+0000 7f08c6e0c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026ba81db0 0x56026bac16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.828+0000 7f08c6e0c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56026ba82190 0x56026babaea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.828+0000 7f08c6e0c740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56026bad21d0 0x56026baba1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.828+0000 7f08c6e0c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x56026baa58d0 con 0x56026ba81db0 2024-03-28T23:32:38.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.828+0000 7f08c6e0c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x56026baa4090 con 0x56026ba82190 2024-03-28T23:32:38.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.828+0000 7f08c6e0c740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x56026ba9f450 con 0x56026bad21d0 2024-03-28T23:32:38.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.828+0000 7f08c6e0c740 10 client.0 _ll_register_callbacks cb 0x7f08b804c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:32:38.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.828+0000 7f08c1fcc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026ba81db0 0x56026bac16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.828+0000 7f08c1fcc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026ba81db0 0x56026bac16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40408/0 (socket says 172.21.15.161:40408) 2024-03-28T23:32:38.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.828+0000 7f08c1fcc640 1 -- 172.21.15.161:0/1650533960 learned_addr learned my addr 172.21.15.161:0/1650533960 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:38.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.829+0000 7f08c17cb640 1 --2- 172.21.15.161:0/1650533960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56026ba82190 0x56026babaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.829+0000 7f08c27cd640 1 --2- 172.21.15.161:0/1650533960 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56026bad21d0 0x56026baba1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.829+0000 7f08c1fcc640 1 -- 172.21.15.161:0/1650533960 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56026bad21d0 msgr2=0x56026baba1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:38.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.829+0000 7f08c1fcc640 1 --2- 172.21.15.161:0/1650533960 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56026bad21d0 0x56026baba1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.829+0000 7f08c1fcc640 1 -- 172.21.15.161:0/1650533960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56026ba82190 msgr2=0x56026babaea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:38.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.829+0000 7f08c1fcc640 1 --2- 172.21.15.161:0/1650533960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56026ba82190 0x56026babaea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.829+0000 7f08c1fcc640 1 -- 172.21.15.161:0/1650533960 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56026bad8920 con 0x56026ba81db0 2024-03-28T23:32:38.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.829+0000 7f08c1fcc640 1 --2- 172.21.15.161:0/1650533960 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026ba81db0 0x56026bac16d0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f08b8006b10 tx=0x7f08b8008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2bfcc9f31293717 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:38.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.829+0000 7f089cff9640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:32:38.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.829+0000 7f089cff9640 1 -- 172.21.15.161:0/1650533960 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b806b070 con 0x56026ba81db0 2024-03-28T23:32:38.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.829+0000 7f089cff9640 1 -- 172.21.15.161:0/1650533960 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 6101+0+0 (secure 0 0 0) 0x7f08b8055870 con 0x56026ba81db0 2024-03-28T23:32:38.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.829+0000 7f08c6e0c740 1 -- 172.21.15.161:0/1650533960 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08b4057670 con 0x56026ba81db0 2024-03-28T23:32:38.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.829+0000 7f089cff9640 1 -- 172.21.15.161:0/1650533960 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7f08b8041040 con 0x56026ba81db0 2024-03-28T23:32:38.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.829+0000 7f08c6e0c740 10 client.6615 fetch_fsmap learned FSMap version 56 2024-03-28T23:32:38.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.829+0000 7f08c6e0c740 1 -- 172.21.15.161:0/1650533960 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=2+}) -- 0x56026babb510 con 0x56026ba81db0 2024-03-28T23:32:38.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.829+0000 7f089cff9640 1 -- 172.21.15.161:0/1650533960 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08b4057670 con 0x56026ba81db0 2024-03-28T23:32:38.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.829+0000 7f089cff9640 1 -- 172.21.15.161:0/1650533960 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 56) ==== 3494+0+0 (secure 0 0 0) 0x7f08b8066040 con 0x56026ba81db0 2024-03-28T23:32:38.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.829+0000 7f08c6e0c740 10 client.6615 fetch_fsmap finished waiting for FSMap version 56 2024-03-28T23:32:38.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.830+0000 7f08c6e0c740 10 client.6615 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T23:32:38.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.830+0000 7f08c6e0c740 20 client.6615 populate_metadata read hostname 'smithi161' 2024-03-28T23:32:38.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.830+0000 7f08c6e0c740 1 --2- 172.21.15.161:0/1650533960 >> [v2:172.21.15.161:6834/3850309502,v1:172.21.15.161:6836/3850309502] conn(0x56026bae1610 0x56026bb34ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.830+0000 7f08c6e0c740 4 client.6615 mds_command: new command op to 6468 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:32:38.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.830+0000 7f08c6e0c740 1 -- 172.21.15.161:0/1650533960 --> [v2:172.21.15.161:6834/3850309502,v1:172.21.15.161:6836/3850309502] -- command(tid 0: {"prefix": "status"}) -- 0x56026b9ed380 con 0x56026bae1610 2024-03-28T23:32:38.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.830+0000 7f08c17cb640 1 --2- 172.21.15.161:0/1650533960 >> [v2:172.21.15.161:6834/3850309502,v1:172.21.15.161:6836/3850309502] conn(0x56026bae1610 0x56026bb34ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.830+0000 7f08c17cb640 1 --2- 172.21.15.161:0/1650533960 >> [v2:172.21.15.161:6834/3850309502,v1:172.21.15.161:6836/3850309502] conn(0x56026bae1610 0x56026bb34ef0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc2ce9a4608d8543 server_cookie=7789e6f91f0ff1d6 in_seq=0 out_seq=0 2024-03-28T23:32:38.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.830+0000 7f089cff9640 10 client.6615 ms_handle_connect on v2:172.21.15.161:6834/3850309502 2024-03-28T23:32:38.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.830+0000 7f089cff9640 1 -- 172.21.15.161:0/1650533960 <== mds.0 v2:172.21.15.161:6834/3850309502 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x56026b9ed380 con 0x56026bae1610 2024-03-28T23:32:38.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.830+0000 7f089cff9640 10 client.6615 handle_command_reply: tid=0 2024-03-28T23:32:38.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.830+0000 7f08c6e0c740 1 client.6615 shutdown 2024-03-28T23:32:38.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.830+0000 7f08c6e0c740 1 -- 172.21.15.161:0/1650533960 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026ba81db0 msgr2=0x56026bac16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:38.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.830+0000 7f08c6e0c740 1 --2- 172.21.15.161:0/1650533960 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026ba81db0 0x56026bac16d0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f08b8006b10 tx=0x7f08b8008030 comp rx=0 tx=0).stop 2024-03-28T23:32:38.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.830+0000 7f08c6e0c740 1 -- 172.21.15.161:0/1650533960 shutdown_connections 2024-03-28T23:32:38.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.830+0000 7f08c6e0c740 1 --2- 172.21.15.161:0/1650533960 >> [v2:172.21.15.161:6834/3850309502,v1:172.21.15.161:6836/3850309502] conn(0x56026bae1610 0x56026bb34ef0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.830+0000 7f08c6e0c740 1 --2- 172.21.15.161:0/1650533960 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56026bad21d0 0x56026baba1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.830+0000 7f08c6e0c740 1 --2- 172.21.15.161:0/1650533960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56026ba82190 0x56026babaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.830+0000 7f089cff9640 0 client.6615 ms_handle_reset on v2:172.21.15.161:6834/3850309502 2024-03-28T23:32:38.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.830+0000 7f08c6e0c740 1 --2- 172.21.15.161:0/1650533960 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56026ba81db0 0x56026bac16d0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.830+0000 7f08c6e0c740 1 -- 172.21.15.161:0/1650533960 >> 172.21.15.161:0/1650533960 conn(0x56026ba80310 msgr2=0x56026bad16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:38.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.830+0000 7f08c6e0c740 1 -- 172.21.15.161:0/1650533960 shutdown_connections 2024-03-28T23:32:38.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.831+0000 7f08c6e0c740 1 -- 172.21.15.161:0/1650533960 wait complete. 2024-03-28T23:32:38.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.831+0000 7f08c6e0c740 20 client.6615 trim_cache size 0 max 16384 2024-03-28T23:32:38.839 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:32:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:32:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:32:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 6468, 2024-03-28T23:32:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:32:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:32:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:32:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 5.8097484320000001, 2024-03-28T23:32:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 56, 2024-03-28T23:32:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 73, 2024-03-28T23:32:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 73, 2024-03-28T23:32:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 10.913776886000001 2024-03-28T23:32:38.839 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:32:38.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.831+0000 7f08c4230640 1 -- 172.21.15.161:0/25286205 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f08a006b4d0 msgr2=0x7f08a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:38.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.831+0000 7f08c4230640 1 --2- 172.21.15.161:0/25286205 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f08a006b4d0 0x7f08a006d990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f08b8035000 tx=0x7f08b8030000 comp rx=0 tx=0).stop 2024-03-28T23:32:38.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.831+0000 7f08c4230640 1 -- 172.21.15.161:0/25286205 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08bc152890 msgr2=0x7f08bc0bfe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:38.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.831+0000 7f08c4230640 1 --2- 172.21.15.161:0/25286205 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08bc152890 0x7f08bc0bfe50 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f08b400cd80 tx=0x7f08b400a650 comp rx=0 tx=0).stop 2024-03-28T23:32:38.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.831+0000 7f08c4230640 1 -- 172.21.15.161:0/25286205 shutdown_connections 2024-03-28T23:32:38.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.831+0000 7f08c4230640 1 --2- 172.21.15.161:0/25286205 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f08a006b4d0 0x7f08a006d990 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.831+0000 7f08c4230640 1 --2- 172.21.15.161:0/25286205 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08bc152890 0x7f08bc0bfe50 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.831+0000 7f08c4230640 1 --2- 172.21.15.161:0/25286205 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f08bc0c8ed0 0x7f08bc149010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.831+0000 7f08c4230640 1 --2- 172.21.15.161:0/25286205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08bc0c5b20 0x7f08bc0c36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.831+0000 7f08c4230640 1 -- 172.21.15.161:0/25286205 >> 172.21.15.161:0/25286205 conn(0x7f08bc0ac620 msgr2=0x7f08bc150ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:38.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.831+0000 7f08c4230640 1 -- 172.21.15.161:0/25286205 shutdown_connections 2024-03-28T23:32:38.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.831+0000 7f08c4230640 1 -- 172.21.15.161:0/25286205 wait complete. 2024-03-28T23:32:38.843 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:32:38.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.945+0000 7f8affc55640 1 Processor -- start 2024-03-28T23:32:38.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.945+0000 7f8affc55640 1 -- start start 2024-03-28T23:32:38.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.946+0000 7f8affc55640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8af8152890 0x7f8af80c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.946+0000 7f8affc55640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8af80c5b20 0x7f8af81553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.946+0000 7f8affc55640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8af8158a30 con 0x7f8af80c8ed0 2024-03-28T23:32:38.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.946+0000 7f8affc55640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8af80c8480 con 0x7f8af80c5b20 2024-03-28T23:32:38.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.946+0000 7f8affc55640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8af8159640 con 0x7f8af8152890 2024-03-28T23:32:38.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.946+0000 7f8afe1f2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8af80c5b20 0x7f8af81553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.946+0000 7f8afd1f0640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f8af80c8ed0 0x7f8af80c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:58502/0 (socket says 172.21.15.161:58502) 2024-03-28T23:32:38.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.946+0000 7f8afd1f0640 1 -- 172.21.15.161:0/3683200594 learned_addr learned my addr 172.21.15.161:0/3683200594 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:38.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.946+0000 7f8afd9f1640 1 --2- 172.21.15.161:0/3683200594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8af8152890 0x7f8af80c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.946+0000 7f8afc9ef640 1 -- 172.21.15.161:0/3683200594 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1740282198 0 0) 0x7f8af8158a30 con 0x7f8af80c8ed0 2024-03-28T23:32:38.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.946+0000 7f8afc9ef640 1 -- 172.21.15.161:0/3683200594 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ae4003520 con 0x7f8af80c8ed0 2024-03-28T23:32:38.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.946+0000 7f8afe1f2640 1 -- 172.21.15.161:0/3683200594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8af8152890 msgr2=0x7f8af80c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:38.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.946+0000 7f8afe1f2640 1 --2- 172.21.15.161:0/3683200594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8af8152890 0x7f8af80c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.946+0000 7f8afe1f2640 1 -- 172.21.15.161:0/3683200594 >> v1:172.21.15.161:6789/0 conn(0x7f8af80c8ed0 legacy=0x7f8af80c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:38.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.946+0000 7f8afe1f2640 1 -- 172.21.15.161:0/3683200594 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8af8158d10 con 0x7f8af80c5b20 2024-03-28T23:32:38.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.947+0000 7f8afe1f2640 1 --2- 172.21.15.161:0/3683200594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8af80c5b20 0x7f8af81553a0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f8af81597c0 tx=0x7f8af402bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca048102b37606a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:38.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.947+0000 7f8afc9ef640 1 -- 172.21.15.161:0/3683200594 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af4039070 con 0x7f8af80c5b20 2024-03-28T23:32:38.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.947+0000 7f8afc9ef640 1 -- 172.21.15.161:0/3683200594 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8af402cb20 con 0x7f8af80c5b20 2024-03-28T23:32:38.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.947+0000 7f8afc9ef640 1 -- 172.21.15.161:0/3683200594 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af4034440 con 0x7f8af80c5b20 2024-03-28T23:32:38.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.948+0000 7f8affc55640 1 -- 172.21.15.161:0/3683200594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8af80c5b20 msgr2=0x7f8af81553a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:38.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.948+0000 7f8affc55640 1 --2- 172.21.15.161:0/3683200594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8af80c5b20 0x7f8af81553a0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f8af81597c0 tx=0x7f8af402bd70 comp rx=0 tx=0).stop 2024-03-28T23:32:38.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.948+0000 7f8affc55640 1 -- 172.21.15.161:0/3683200594 shutdown_connections 2024-03-28T23:32:38.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.948+0000 7f8affc55640 1 --2- 172.21.15.161:0/3683200594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8af80c5b20 0x7f8af81553a0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.948+0000 7f8affc55640 1 --2- 172.21.15.161:0/3683200594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8af8152890 0x7f8af80c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.948+0000 7f8affc55640 1 -- 172.21.15.161:0/3683200594 >> 172.21.15.161:0/3683200594 conn(0x7f8af80ac620 msgr2=0x7f8af80aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:38.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.948+0000 7f8affc55640 1 -- 172.21.15.161:0/3683200594 shutdown_connections 2024-03-28T23:32:38.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.948+0000 7f8affc55640 1 -- 172.21.15.161:0/3683200594 wait complete. 2024-03-28T23:32:38.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.949+0000 7f8affc55640 1 Processor -- start 2024-03-28T23:32:38.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.949+0000 7f8affc55640 1 -- start start 2024-03-28T23:32:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.950+0000 7f8affc55640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8af80c5b20 0x7f8af81ea4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.950+0000 7f8affc55640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8af80c8ed0 0x7f8af81ee340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.950+0000 7f8affc55640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8af8152890 0x7f8af8144ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.950+0000 7f8affc55640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8af8152c70 con 0x7f8af8152890 2024-03-28T23:32:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.950+0000 7f8affc55640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8af81592d0 con 0x7f8af80c8ed0 2024-03-28T23:32:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.950+0000 7f8affc55640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8af8152df0 con 0x7f8af80c5b20 2024-03-28T23:32:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.950+0000 7f8afe1f2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8af8152890 0x7f8af8144ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.950+0000 7f8afe1f2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8af8152890 0x7f8af8144ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40432/0 (socket says 172.21.15.161:40432) 2024-03-28T23:32:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.950+0000 7f8afe1f2640 1 -- 172.21.15.161:0/438576152 learned_addr learned my addr 172.21.15.161:0/438576152 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.950+0000 7f8afd9f1640 1 --2- 172.21.15.161:0/438576152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8af80c5b20 0x7f8af81ea4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.950+0000 7f8afd1f0640 1 --2- 172.21.15.161:0/438576152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8af80c8ed0 0x7f8af81ee340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.950+0000 7f8afe1f2640 1 -- 172.21.15.161:0/438576152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8af80c5b20 msgr2=0x7f8af81ea4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.950+0000 7f8afe1f2640 1 --2- 172.21.15.161:0/438576152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8af80c5b20 0x7f8af81ea4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.950+0000 7f8afe1f2640 1 -- 172.21.15.161:0/438576152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8af80c8ed0 msgr2=0x7f8af81ee340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.950+0000 7f8afe1f2640 1 --2- 172.21.15.161:0/438576152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8af80c8ed0 0x7f8af81ee340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.950+0000 7f8afe1f2640 1 -- 172.21.15.161:0/438576152 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8af80c16f0 con 0x7f8af8152890 2024-03-28T23:32:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.950+0000 7f8afe1f2640 1 --2- 172.21.15.161:0/438576152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8af8152890 0x7f8af8144ed0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f8af4009e80 tx=0x7f8af4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.950+0000 7f8aee7fc640 1 -- 172.21.15.161:0/438576152 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af4039050 con 0x7f8af8152890 2024-03-28T23:32:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.951+0000 7f8aee7fc640 1 -- 172.21.15.161:0/438576152 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8af4045070 con 0x7f8af8152890 2024-03-28T23:32:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.951+0000 7f8aee7fc640 1 -- 172.21.15.161:0/438576152 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af4009450 con 0x7f8af8152890 2024-03-28T23:32:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.951+0000 7f8affc55640 1 -- 172.21.15.161:0/438576152 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8af81eaa00 con 0x7f8af8152890 2024-03-28T23:32:38.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.951+0000 7f8affc55640 1 -- 172.21.15.161:0/438576152 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8af80c1130 con 0x7f8af8152890 2024-03-28T23:32:38.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.951+0000 7f8affc55640 1 -- 172.21.15.161:0/438576152 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8af804f4b0 con 0x7f8af8152890 2024-03-28T23:32:38.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.952+0000 7f8aee7fc640 1 -- 172.21.15.161:0/438576152 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8af403e830 con 0x7f8af8152890 2024-03-28T23:32:38.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.952+0000 7f8aee7fc640 1 --2- 172.21.15.161:0/438576152 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8adc06b500 0x7f8adc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:38.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.953+0000 7f8afd9f1640 1 --2- 172.21.15.161:0/438576152 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8adc06b500 0x7f8adc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:38.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.953+0000 7f8aee7fc640 1 -- 172.21.15.161:0/438576152 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6101+0+0 (secure 0 0 0) 0x7f8af40ad150 con 0x7f8af8152890 2024-03-28T23:32:38.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.953+0000 7f8afd9f1640 1 --2- 172.21.15.161:0/438576152 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8adc06b500 0x7f8adc06d9c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8af8148400 tx=0x7f8af0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:38.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:38.957+0000 7f8aee7fc640 1 -- 172.21.15.161:0/438576152 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8af4032e20 con 0x7f8af8152890 2024-03-28T23:32:39.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.313+0000 7f8affc55640 1 -- 172.21.15.161:0/438576152 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8af8067cc0 con 0x7f8af8152890 2024-03-28T23:32:39.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.313+0000 7f8aee7fc640 1 -- 172.21.15.161:0/438576152 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+5192 (secure 0 0 0) 0x7f8af407c0b0 con 0x7f8af8152890 2024-03-28T23:32:39.315 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:39.316 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":56,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/737681942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":737681942},{"type":"v1","addr":"172.21.15.161:6837","nonce":737681942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":5579,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/3501962065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3501962065},{"type":"v1","addr":"172.21.15.170:6834","nonce":3501962065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6037,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2691084827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":2691084827},{"type":"v1","addr":"172.21.15.170:6835","nonce":2691084827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:32:32.372964+0000","modified":"2024-03-28T23:32:35.321715+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6468},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6468":{"gid":6468,"name":"a","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6836/3850309502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3850309502},{"type":"v1","addr":"172.21.15.161:6836","nonce":3850309502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6468,"qdb_cluster":[6468]},"id":5}]} 2024-03-28T23:32:39.316 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 56 2024-03-28T23:32:39.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.316+0000 7f8affc55640 1 -- 172.21.15.161:0/438576152 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8adc06b500 msgr2=0x7f8adc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:39.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.316+0000 7f8affc55640 1 --2- 172.21.15.161:0/438576152 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8adc06b500 0x7f8adc06d9c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8af8148400 tx=0x7f8af0008040 comp rx=0 tx=0).stop 2024-03-28T23:32:39.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.316+0000 7f8affc55640 1 -- 172.21.15.161:0/438576152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8af8152890 msgr2=0x7f8af8144ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:39.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.316+0000 7f8affc55640 1 --2- 172.21.15.161:0/438576152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8af8152890 0x7f8af8144ed0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f8af4009e80 tx=0x7f8af4004a40 comp rx=0 tx=0).stop 2024-03-28T23:32:39.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.316+0000 7f8affc55640 1 -- 172.21.15.161:0/438576152 shutdown_connections 2024-03-28T23:32:39.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.316+0000 7f8affc55640 1 --2- 172.21.15.161:0/438576152 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8adc06b500 0x7f8adc06d9c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:39.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.316+0000 7f8affc55640 1 --2- 172.21.15.161:0/438576152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8af8152890 0x7f8af8144ed0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:39.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.316+0000 7f8affc55640 1 --2- 172.21.15.161:0/438576152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8af80c8ed0 0x7f8af81ee340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:39.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.316+0000 7f8affc55640 1 --2- 172.21.15.161:0/438576152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8af80c5b20 0x7f8af81ea4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:39.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.316+0000 7f8affc55640 1 -- 172.21.15.161:0/438576152 >> 172.21.15.161:0/438576152 conn(0x7f8af80ac620 msgr2=0x7f8af81505f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:39.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.316+0000 7f8affc55640 1 -- 172.21.15.161:0/438576152 shutdown_connections 2024-03-28T23:32:39.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.316+0000 7f8affc55640 1 -- 172.21.15.161:0/438576152 wait complete. 2024-03-28T23:32:39.329 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 5, "epoch": 56, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 5, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:32:32.372964+0000", "damaged": [], "data_pools": [ 12 ], "enabled": true, "epoch": 56, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6468": { "addr": "172.21.15.161:6836/3850309502", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 3850309502, "type": "v2" }, { "addr": "172.21.15.161:6836", "nonce": 3850309502, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6468, "incarnation": 53, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 11, "modified": "2024-03-28T23:32:35.321715+0000", "qdb_cluster": [ 6468 ], "qdb_leader": 6468, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6468 } } } ], "standbys": [ { "addr": "172.21.15.161:6837/737681942", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6835", "nonce": 737681942, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 737681942, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 52, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5570, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.170:6834/3501962065", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 3501962065, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 3501962065, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 52, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5579, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.170:6835/2691084827", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 2691084827, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 2691084827, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 52, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6037, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:32:39.329 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-28T23:32:39.329 INFO:tasks.cephfs.test_failover:current = [{'gid': 6468, 'name': 'a', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.161:6836/3850309502', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 3850309502}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 3850309502}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 6037, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/2691084827', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 2691084827}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 2691084827}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 52}, {'gid': 5570, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6837/737681942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 737681942}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 737681942}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 52}, {'gid': 5579, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6834/3501962065', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 3501962065}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 3501962065}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 52}] 2024-03-28T23:32:39.330 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.c mds_join_fs cephfs2 2024-03-28T23:32:39.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.472+0000 7fbc4b372640 1 Processor -- start 2024-03-28T23:32:39.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.472+0000 7fbc4b372640 1 -- start start 2024-03-28T23:32:39.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.474+0000 7fbc4b372640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fbc440c0330 0x7fbc440c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:39.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.474+0000 7fbc4b372640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbc441589b0 0x7fbc44158e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:39.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.474+0000 7fbc4b372640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc440c6f60 con 0x7fbc44152bc0 2024-03-28T23:32:39.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.474+0000 7fbc4b372640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fbc440c6120 con 0x7fbc441589b0 2024-03-28T23:32:39.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.474+0000 7fbc4b372640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fbc440c6400 con 0x7fbc440c0330 2024-03-28T23:32:39.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.474+0000 7fbc4990f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbc441589b0 0x7fbc44158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:39.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.474+0000 7fbc4890d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fbc440c0330 0x7fbc440c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:39.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.474+0000 7fbc4990f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbc441589b0 0x7fbc44158e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40434/0 (socket says 172.21.15.161:40434) 2024-03-28T23:32:39.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.474+0000 7fbc4990f640 1 -- 172.21.15.161:0/2220626261 learned_addr learned my addr 172.21.15.161:0/2220626261 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:39.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.474+0000 7fbc4890d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fbc440c0330 0x7fbc440c9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:35230/0 (socket says 172.21.15.161:35230) 2024-03-28T23:32:39.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.474+0000 7fbc4910e640 1 --1- 172.21.15.161:0/2220626261 >> v1:172.21.15.170:6789/0 conn(0x7fbc44152bc0 0x7fbc440c8a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:53666/0 (socket says 172.21.15.161:53666) 2024-03-28T23:32:39.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.474+0000 7fbc4990f640 1 -- 172.21.15.161:0/2220626261 >> v2:172.21.15.170:3300/0 conn(0x7fbc440c0330 msgr2=0x7fbc440c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:39.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.474+0000 7fbc4990f640 1 --2- 172.21.15.161:0/2220626261 >> v2:172.21.15.170:3300/0 conn(0x7fbc440c0330 0x7fbc440c9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:39.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.474+0000 7fbc4990f640 1 -- 172.21.15.161:0/2220626261 >> v1:172.21.15.170:6789/0 conn(0x7fbc44152bc0 legacy=0x7fbc440c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:39.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.474+0000 7fbc4990f640 1 -- 172.21.15.161:0/2220626261 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc440c96d0 con 0x7fbc441589b0 2024-03-28T23:32:39.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.475+0000 7fbc4990f640 1 --2- 172.21.15.161:0/2220626261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc441589b0 0x7fbc44158e00 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fbc440b0c70 tx=0x7fbc4002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e39fe76a9fb3687 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:39.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.475+0000 7fbc3bfff640 1 -- 172.21.15.161:0/2220626261 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4002cb20 con 0x7fbc441589b0 2024-03-28T23:32:39.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.475+0000 7fbc3bfff640 1 -- 172.21.15.161:0/2220626261 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc4002ccc0 con 0x7fbc441589b0 2024-03-28T23:32:39.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.475+0000 7fbc3bfff640 1 -- 172.21.15.161:0/2220626261 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc40034610 con 0x7fbc441589b0 2024-03-28T23:32:39.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.476+0000 7fbc4b372640 1 -- 172.21.15.161:0/2220626261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc441589b0 msgr2=0x7fbc44158e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:39.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.476+0000 7fbc4b372640 1 --2- 172.21.15.161:0/2220626261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc441589b0 0x7fbc44158e00 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fbc440b0c70 tx=0x7fbc4002bf80 comp rx=0 tx=0).stop 2024-03-28T23:32:39.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.476+0000 7fbc4b372640 1 -- 172.21.15.161:0/2220626261 shutdown_connections 2024-03-28T23:32:39.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.476+0000 7fbc4b372640 1 --2- 172.21.15.161:0/2220626261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc441589b0 0x7fbc44158e00 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:39.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.476+0000 7fbc4b372640 1 --2- 172.21.15.161:0/2220626261 >> v2:172.21.15.170:3300/0 conn(0x7fbc440c0330 0x7fbc440c9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:39.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.476+0000 7fbc4b372640 1 -- 172.21.15.161:0/2220626261 >> 172.21.15.161:0/2220626261 conn(0x7fbc440caa80 msgr2=0x7fbc440cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:39.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.476+0000 7fbc4b372640 1 -- 172.21.15.161:0/2220626261 shutdown_connections 2024-03-28T23:32:39.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.476+0000 7fbc4b372640 1 -- 172.21.15.161:0/2220626261 wait complete. 2024-03-28T23:32:39.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.477+0000 7fbc4b372640 1 Processor -- start 2024-03-28T23:32:39.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.477+0000 7fbc4b372640 1 -- start start 2024-03-28T23:32:39.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.478+0000 7fbc4b372640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc440c0330 0x7fbc44148280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:39.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.478+0000 7fbc4b372640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc44152bc0 0x7fbc441487c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:39.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.478+0000 7fbc4b372640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc441589b0 0x7fbc440ab840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:39.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.478+0000 7fbc4b372640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbc44152fa0 con 0x7fbc440c0330 2024-03-28T23:32:39.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.478+0000 7fbc4b372640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbc440c6bf0 con 0x7fbc441589b0 2024-03-28T23:32:39.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.478+0000 7fbc4b372640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbc441eb050 con 0x7fbc44152bc0 2024-03-28T23:32:39.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.478+0000 7fbc4910e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc440c0330 0x7fbc44148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:39.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.478+0000 7fbc4910e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc440c0330 0x7fbc44148280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40448/0 (socket says 172.21.15.161:40448) 2024-03-28T23:32:39.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.478+0000 7fbc4910e640 1 -- 172.21.15.161:0/2996245772 learned_addr learned my addr 172.21.15.161:0/2996245772 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:39.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.478+0000 7fbc4890d640 1 --2- 172.21.15.161:0/2996245772 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc44152bc0 0x7fbc441487c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:39.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.478+0000 7fbc4990f640 1 --2- 172.21.15.161:0/2996245772 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc441589b0 0x7fbc440ab840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:39.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.478+0000 7fbc4910e640 1 -- 172.21.15.161:0/2996245772 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc44152bc0 msgr2=0x7fbc441487c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:39.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.478+0000 7fbc4910e640 1 --2- 172.21.15.161:0/2996245772 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc44152bc0 0x7fbc441487c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:39.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.478+0000 7fbc4910e640 1 -- 172.21.15.161:0/2996245772 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc441589b0 msgr2=0x7fbc440ab840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:39.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.478+0000 7fbc4910e640 1 --2- 172.21.15.161:0/2996245772 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc441589b0 0x7fbc440ab840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:39.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.478+0000 7fbc4910e640 1 -- 172.21.15.161:0/2996245772 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc441eb580 con 0x7fbc440c0330 2024-03-28T23:32:39.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.478+0000 7fbc4890d640 1 --2- 172.21.15.161:0/2996245772 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc44152bc0 0x7fbc441487c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:39.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.478+0000 7fbc4910e640 1 --2- 172.21.15.161:0/2996245772 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc440c0330 0x7fbc44148280 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c00c4d0 tx=0x7fbc3c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:39.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.478+0000 7fbc39ffb640 1 -- 172.21.15.161:0/2996245772 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c018070 con 0x7fbc440c0330 2024-03-28T23:32:39.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.478+0000 7fbc39ffb640 1 -- 172.21.15.161:0/2996245772 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc3c007480 con 0x7fbc440c0330 2024-03-28T23:32:39.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.478+0000 7fbc39ffb640 1 -- 172.21.15.161:0/2996245772 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c0047e0 con 0x7fbc440c0330 2024-03-28T23:32:39.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.478+0000 7fbc4b372640 1 -- 172.21.15.161:0/2996245772 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc441eb2a0 con 0x7fbc440c0330 2024-03-28T23:32:39.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.479+0000 7fbc4b372640 1 -- 172.21.15.161:0/2996245772 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc44154b70 con 0x7fbc440c0330 2024-03-28T23:32:39.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.480+0000 7fbc4b372640 1 -- 172.21.15.161:0/2996245772 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc440c94f0 con 0x7fbc440c0330 2024-03-28T23:32:39.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.480+0000 7fbc39ffb640 1 -- 172.21.15.161:0/2996245772 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbc3c01b070 con 0x7fbc440c0330 2024-03-28T23:32:39.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.481+0000 7fbc39ffb640 1 --2- 172.21.15.161:0/2996245772 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbc2806b4d0 0x7fbc2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:39.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.481+0000 7fbc39ffb640 1 -- 172.21.15.161:0/2996245772 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6101+0+0 (secure 0 0 0) 0x7fbc3c089670 con 0x7fbc440c0330 2024-03-28T23:32:39.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.481+0000 7fbc4890d640 1 --2- 172.21.15.161:0/2996245772 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbc2806b4d0 0x7fbc2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:39.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.482+0000 7fbc4890d640 1 --2- 172.21.15.161:0/2996245772 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbc2806b4d0 0x7fbc2806d990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fbc4406d4d0 tx=0x7fbc34008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:39.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.485+0000 7fbc39ffb640 1 -- 172.21.15.161:0/2996245772 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbc440c94f0 con 0x7fbc440c0330 2024-03-28T23:32:39.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.741+0000 7fbc4b372640 1 -- 172.21.15.161:0/2996245772 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) -- 0x7fbc4404f4b0 con 0x7fbc440c0330 2024-03-28T23:32:39.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.745+0000 7fbc39ffb640 1 -- 172.21.15.161:0/2996245772 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v24)=0 v24) ==== 117+0+0 (secure 0 0 0) 0x7fbc3c058360 con 0x7fbc440c0330 2024-03-28T23:32:39.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.747+0000 7fbc4b372640 1 -- 172.21.15.161:0/2996245772 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbc2806b4d0 msgr2=0x7fbc2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:39.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.747+0000 7fbc4b372640 1 --2- 172.21.15.161:0/2996245772 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbc2806b4d0 0x7fbc2806d990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fbc4406d4d0 tx=0x7fbc34008040 comp rx=0 tx=0).stop 2024-03-28T23:32:39.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.747+0000 7fbc4b372640 1 -- 172.21.15.161:0/2996245772 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc440c0330 msgr2=0x7fbc44148280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:39.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.747+0000 7fbc4b372640 1 --2- 172.21.15.161:0/2996245772 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc440c0330 0x7fbc44148280 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c00c4d0 tx=0x7fbc3c00c990 comp rx=0 tx=0).stop 2024-03-28T23:32:39.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.747+0000 7fbc4b372640 1 -- 172.21.15.161:0/2996245772 shutdown_connections 2024-03-28T23:32:39.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.747+0000 7fbc4b372640 1 --2- 172.21.15.161:0/2996245772 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbc2806b4d0 0x7fbc2806d990 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:39.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.747+0000 7fbc4b372640 1 --2- 172.21.15.161:0/2996245772 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc441589b0 0x7fbc440ab840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:39.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.747+0000 7fbc4b372640 1 --2- 172.21.15.161:0/2996245772 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc44152bc0 0x7fbc441487c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:39.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.747+0000 7fbc4b372640 1 --2- 172.21.15.161:0/2996245772 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc440c0330 0x7fbc44148280 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:39.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.747+0000 7fbc4b372640 1 -- 172.21.15.161:0/2996245772 >> 172.21.15.161:0/2996245772 conn(0x7fbc440caa80 msgr2=0x7fbc440cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:39.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.747+0000 7fbc4b372640 1 -- 172.21.15.161:0/2996245772 shutdown_connections 2024-03-28T23:32:39.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.748+0000 7fbc4b372640 1 -- 172.21.15.161:0/2996245772 wait complete. 2024-03-28T23:32:39.760 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs2 2024-03-28T23:32:39.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.863+0000 7fa79120b640 1 Processor -- start 2024-03-28T23:32:39.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.863+0000 7fa79120b640 1 -- start start 2024-03-28T23:32:39.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.864+0000 7fa79120b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa78c152940 0x7fa78c0c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:39.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.864+0000 7fa79120b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa78c0ca580 0x7fa78c0c8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:39.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.864+0000 7fa79120b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa78c0c6df0 con 0x7fa78c0c9480 2024-03-28T23:32:39.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.864+0000 7fa79120b640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa78c0c70d0 con 0x7fa78c0ca580 2024-03-28T23:32:39.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.864+0000 7fa79120b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fa78c0c5830 con 0x7fa78c152940 2024-03-28T23:32:39.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.864+0000 7fa78a59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa78c0ca580 0x7fa78c0c8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:39.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.864+0000 7fa78b59e640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa78c0c9480 0x7fa78c155220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:58512/0 (socket says 172.21.15.161:58512) 2024-03-28T23:32:39.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.864+0000 7fa78b59e640 1 -- 172.21.15.161:0/2006379144 learned_addr learned my addr 172.21.15.161:0/2006379144 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:39.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.864+0000 7fa78ad9d640 1 --2- 172.21.15.161:0/2006379144 >> v2:172.21.15.170:3300/0 conn(0x7fa78c152940 0x7fa78c0c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:39.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.864+0000 7fa789d9b640 1 -- 172.21.15.161:0/2006379144 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3236434537 0 0) 0x7fa78c0c6df0 con 0x7fa78c0c9480 2024-03-28T23:32:39.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.864+0000 7fa789d9b640 1 -- 172.21.15.161:0/2006379144 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa778003520 con 0x7fa78c0c9480 2024-03-28T23:32:39.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.865+0000 7fa78a59c640 1 -- 172.21.15.161:0/2006379144 >> v2:172.21.15.170:3300/0 conn(0x7fa78c152940 msgr2=0x7fa78c0c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:39.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.865+0000 7fa78a59c640 1 --2- 172.21.15.161:0/2006379144 >> v2:172.21.15.170:3300/0 conn(0x7fa78c152940 0x7fa78c0c8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:39.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.865+0000 7fa78a59c640 1 -- 172.21.15.161:0/2006379144 >> v1:172.21.15.161:6789/0 conn(0x7fa78c0c9480 legacy=0x7fa78c155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:39.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.865+0000 7fa78a59c640 1 -- 172.21.15.161:0/2006379144 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa78c0c5b10 con 0x7fa78c0ca580 2024-03-28T23:32:39.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.865+0000 7fa78ad9d640 1 --2- 172.21.15.161:0/2006379144 >> v2:172.21.15.170:3300/0 conn(0x7fa78c152940 0x7fa78c0c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:39.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.865+0000 7fa78a59c640 1 --2- 172.21.15.161:0/2006379144 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa78c0ca580 0x7fa78c0c8f40 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fa78c0aa440 tx=0x7fa77402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e60429c41e971cbb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:39.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.865+0000 7fa789d9b640 1 -- 172.21.15.161:0/2006379144 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa774038070 con 0x7fa78c0ca580 2024-03-28T23:32:39.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.865+0000 7fa789d9b640 1 -- 172.21.15.161:0/2006379144 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa774031710 con 0x7fa78c0ca580 2024-03-28T23:32:39.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.865+0000 7fa789d9b640 1 -- 172.21.15.161:0/2006379144 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa774031ac0 con 0x7fa78c0ca580 2024-03-28T23:32:39.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.866+0000 7fa79120b640 1 -- 172.21.15.161:0/2006379144 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa78c0ca580 msgr2=0x7fa78c0c8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:39.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.866+0000 7fa79120b640 1 --2- 172.21.15.161:0/2006379144 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa78c0ca580 0x7fa78c0c8f40 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fa78c0aa440 tx=0x7fa77402b7b0 comp rx=0 tx=0).stop 2024-03-28T23:32:39.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.866+0000 7fa79120b640 1 -- 172.21.15.161:0/2006379144 shutdown_connections 2024-03-28T23:32:39.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.866+0000 7fa79120b640 1 --2- 172.21.15.161:0/2006379144 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa78c0ca580 0x7fa78c0c8f40 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:39.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.866+0000 7fa79120b640 1 --2- 172.21.15.161:0/2006379144 >> v2:172.21.15.170:3300/0 conn(0x7fa78c152940 0x7fa78c0c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:39.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.866+0000 7fa79120b640 1 -- 172.21.15.161:0/2006379144 >> 172.21.15.161:0/2006379144 conn(0x7fa78c0caa00 msgr2=0x7fa78c0cae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:39.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.866+0000 7fa79120b640 1 -- 172.21.15.161:0/2006379144 shutdown_connections 2024-03-28T23:32:39.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.866+0000 7fa79120b640 1 -- 172.21.15.161:0/2006379144 wait complete. 2024-03-28T23:32:39.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.867+0000 7fa79120b640 1 Processor -- start 2024-03-28T23:32:39.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.868+0000 7fa79120b640 1 -- start start 2024-03-28T23:32:39.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.868+0000 7fa79120b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa78c0c9480 0x7fa78c0af540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:39.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.868+0000 7fa79120b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa78c0ca580 0x7fa78c154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:39.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.868+0000 7fa79120b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa78c152940 0x7fa78c14bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:39.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.868+0000 7fa79120b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa78c0c61c0 con 0x7fa78c0ca580 2024-03-28T23:32:39.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.868+0000 7fa79120b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa78c0c6040 con 0x7fa78c0c9480 2024-03-28T23:32:39.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.868+0000 7fa79120b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa78c155930 con 0x7fa78c152940 2024-03-28T23:32:39.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.868+0000 7fa78a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa78c0ca580 0x7fa78c154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:39.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.868+0000 7fa78a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa78c0ca580 0x7fa78c154350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40466/0 (socket says 172.21.15.161:40466) 2024-03-28T23:32:39.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.868+0000 7fa78a59c640 1 -- 172.21.15.161:0/2177297238 learned_addr learned my addr 172.21.15.161:0/2177297238 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:39.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.869+0000 7fa78b59e640 1 --2- 172.21.15.161:0/2177297238 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa78c152940 0x7fa78c14bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:39.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.869+0000 7fa78ad9d640 1 --2- 172.21.15.161:0/2177297238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa78c0c9480 0x7fa78c0af540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:39.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.869+0000 7fa78a59c640 1 -- 172.21.15.161:0/2177297238 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa78c152940 msgr2=0x7fa78c14bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:39.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.869+0000 7fa78a59c640 1 --2- 172.21.15.161:0/2177297238 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa78c152940 0x7fa78c14bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:39.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.869+0000 7fa78a59c640 1 -- 172.21.15.161:0/2177297238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa78c0c9480 msgr2=0x7fa78c0af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:39.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.869+0000 7fa78a59c640 1 --2- 172.21.15.161:0/2177297238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa78c0c9480 0x7fa78c0af540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:39.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.869+0000 7fa78a59c640 1 -- 172.21.15.161:0/2177297238 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa78c0afa80 con 0x7fa78c0ca580 2024-03-28T23:32:39.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.869+0000 7fa78b59e640 1 --2- 172.21.15.161:0/2177297238 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa78c152940 0x7fa78c14bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:39.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.869+0000 7fa78ad9d640 1 --2- 172.21.15.161:0/2177297238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa78c0c9480 0x7fa78c0af540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:39.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.869+0000 7fa78a59c640 1 --2- 172.21.15.161:0/2177297238 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa78c0ca580 0x7fa78c154350 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa774000c00 tx=0x7fa7740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:39.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.869+0000 7fa7837fe640 1 -- 172.21.15.161:0/2177297238 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa774042030 con 0x7fa78c0ca580 2024-03-28T23:32:39.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.869+0000 7fa7837fe640 1 -- 172.21.15.161:0/2177297238 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa77403d670 con 0x7fa78c0ca580 2024-03-28T23:32:39.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.869+0000 7fa7837fe640 1 -- 172.21.15.161:0/2177297238 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa774038040 con 0x7fa78c0ca580 2024-03-28T23:32:39.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.869+0000 7fa79120b640 1 -- 172.21.15.161:0/2177297238 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa78c0b0210 con 0x7fa78c0ca580 2024-03-28T23:32:39.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.869+0000 7fa79120b640 1 -- 172.21.15.161:0/2177297238 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa78c0ae4e0 con 0x7fa78c0ca580 2024-03-28T23:32:39.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.870+0000 7fa79120b640 1 -- 172.21.15.161:0/2177297238 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa78c04f4b0 con 0x7fa78c0ca580 2024-03-28T23:32:39.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.870+0000 7fa7837fe640 1 -- 172.21.15.161:0/2177297238 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa774045070 con 0x7fa78c0ca580 2024-03-28T23:32:39.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.871+0000 7fa7837fe640 1 --2- 172.21.15.161:0/2177297238 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa76c06b4d0 0x7fa76c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:39.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.871+0000 7fa7837fe640 1 -- 172.21.15.161:0/2177297238 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6101+0+0 (secure 0 0 0) 0x7fa7740abe20 con 0x7fa78c0ca580 2024-03-28T23:32:39.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.871+0000 7fa78ad9d640 1 --2- 172.21.15.161:0/2177297238 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa76c06b4d0 0x7fa76c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:39.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.871+0000 7fa78ad9d640 1 --2- 172.21.15.161:0/2177297238 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa76c06b4d0 0x7fa76c06d990 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa784006fd0 tx=0x7fa784008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:39.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:39.877+0000 7fa7837fe640 1 -- 172.21.15.161:0/2177297238 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa774046e50 con 0x7fa78c0ca580 2024-03-28T23:32:40.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.083+0000 7fa79120b640 1 -- 172.21.15.161:0/2177297238 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) -- 0x7fa78c0c68a0 con 0x7fa78c0ca580 2024-03-28T23:32:40.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.086+0000 7fa7837fe640 1 -- 172.21.15.161:0/2177297238 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v25)=0 v25) ==== 117+0+0 (secure 0 0 0) 0x7fa77407aba0 con 0x7fa78c0ca580 2024-03-28T23:32:40.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.088+0000 7fa79120b640 1 -- 172.21.15.161:0/2177297238 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa76c06b4d0 msgr2=0x7fa76c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:40.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.088+0000 7fa79120b640 1 --2- 172.21.15.161:0/2177297238 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa76c06b4d0 0x7fa76c06d990 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa784006fd0 tx=0x7fa784008040 comp rx=0 tx=0).stop 2024-03-28T23:32:40.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.088+0000 7fa79120b640 1 -- 172.21.15.161:0/2177297238 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa78c0ca580 msgr2=0x7fa78c154350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:40.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.088+0000 7fa79120b640 1 --2- 172.21.15.161:0/2177297238 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa78c0ca580 0x7fa78c154350 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa774000c00 tx=0x7fa7740046c0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.088+0000 7fa79120b640 1 -- 172.21.15.161:0/2177297238 shutdown_connections 2024-03-28T23:32:40.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.088+0000 7fa79120b640 1 --2- 172.21.15.161:0/2177297238 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa76c06b4d0 0x7fa76c06d990 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.088+0000 7fa79120b640 1 --2- 172.21.15.161:0/2177297238 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa78c0ca580 0x7fa78c154350 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.088+0000 7fa79120b640 1 --2- 172.21.15.161:0/2177297238 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa78c152940 0x7fa78c14bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.088+0000 7fa79120b640 1 --2- 172.21.15.161:0/2177297238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa78c0c9480 0x7fa78c0af540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.088+0000 7fa79120b640 1 -- 172.21.15.161:0/2177297238 >> 172.21.15.161:0/2177297238 conn(0x7fa78c0caa00 msgr2=0x7fa78c0cade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:40.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.088+0000 7fa79120b640 1 -- 172.21.15.161:0/2177297238 shutdown_connections 2024-03-28T23:32:40.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.088+0000 7fa79120b640 1 -- 172.21.15.161:0/2177297238 wait complete. 2024-03-28T23:32:40.101 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.b mds_join_fs cephfs2 2024-03-28T23:32:40.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.205+0000 7f1c8508f640 1 Processor -- start 2024-03-28T23:32:40.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.205+0000 7f1c8508f640 1 -- start start 2024-03-28T23:32:40.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.206+0000 7f1c8508f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c800c0330 0x7f1c800c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:40.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.206+0000 7f1c8508f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1c801589b0 0x7f1c80158e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:40.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.206+0000 7f1c8508f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c800c6f60 con 0x7f1c80152bc0 2024-03-28T23:32:40.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.206+0000 7f1c8508f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1c800c6120 con 0x7f1c801589b0 2024-03-28T23:32:40.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.206+0000 7f1c8508f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1c800c6400 con 0x7f1c800c0330 2024-03-28T23:32:40.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.206+0000 7f1c7f59e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1c801589b0 0x7f1c80158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:40.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.206+0000 7f1c7f59e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1c801589b0 0x7f1c80158e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40468/0 (socket says 172.21.15.161:40468) 2024-03-28T23:32:40.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.206+0000 7f1c7f59e640 1 -- 172.21.15.161:0/866918696 learned_addr learned my addr 172.21.15.161:0/866918696 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:40.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.206+0000 7f1c7f59e640 1 -- 172.21.15.161:0/866918696 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c800c0330 msgr2=0x7f1c800c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:40.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.206+0000 7f1c77fff640 1 --2- 172.21.15.161:0/866918696 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c800c0330 0x7f1c800c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:40.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.206+0000 7f1c7f59e640 1 --2- 172.21.15.161:0/866918696 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c800c0330 0x7f1c800c9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.206+0000 7f1c7f59e640 1 -- 172.21.15.161:0/866918696 >> v1:172.21.15.170:6789/0 conn(0x7f1c80152bc0 legacy=0x7f1c800c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:40.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.206+0000 7f1c7f59e640 1 -- 172.21.15.161:0/866918696 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c800c96d0 con 0x7f1c801589b0 2024-03-28T23:32:40.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.207+0000 7f1c7f59e640 1 --2- 172.21.15.161:0/866918696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c801589b0 0x7f1c80158e10 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f1c78009e50 tx=0x7f1c7802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84ae345aff2806f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:40.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.207+0000 7f1c7e59c640 1 -- 172.21.15.161:0/866918696 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7802ca70 con 0x7f1c801589b0 2024-03-28T23:32:40.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.207+0000 7f1c7e59c640 1 -- 172.21.15.161:0/866918696 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c7802cc10 con 0x7f1c801589b0 2024-03-28T23:32:40.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.207+0000 7f1c7e59c640 1 -- 172.21.15.161:0/866918696 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c780343f0 con 0x7f1c801589b0 2024-03-28T23:32:40.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.208+0000 7f1c8508f640 1 -- 172.21.15.161:0/866918696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c801589b0 msgr2=0x7f1c80158e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:40.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.208+0000 7f1c8508f640 1 --2- 172.21.15.161:0/866918696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c801589b0 0x7f1c80158e10 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f1c78009e50 tx=0x7f1c7802bed0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.208+0000 7f1c8508f640 1 -- 172.21.15.161:0/866918696 shutdown_connections 2024-03-28T23:32:40.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.208+0000 7f1c8508f640 1 --2- 172.21.15.161:0/866918696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c801589b0 0x7f1c80158e10 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.208+0000 7f1c8508f640 1 --2- 172.21.15.161:0/866918696 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c800c0330 0x7f1c800c9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.208+0000 7f1c8508f640 1 -- 172.21.15.161:0/866918696 >> 172.21.15.161:0/866918696 conn(0x7f1c800caa80 msgr2=0x7f1c800cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:40.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.208+0000 7f1c8508f640 1 -- 172.21.15.161:0/866918696 shutdown_connections 2024-03-28T23:32:40.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.208+0000 7f1c8508f640 1 -- 172.21.15.161:0/866918696 wait complete. 2024-03-28T23:32:40.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.210+0000 7f1c8508f640 1 Processor -- start 2024-03-28T23:32:40.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.210+0000 7f1c8508f640 1 -- start start 2024-03-28T23:32:40.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.211+0000 7f1c8508f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c800c0330 0x7f1c8006c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:40.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.211+0000 7f1c8508f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c80152bc0 0x7f1c801547d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:40.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.211+0000 7f1c8508f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c801589b0 0x7f1c80154d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:40.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.211+0000 7f1c8508f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1c80152fa0 con 0x7f1c800c0330 2024-03-28T23:32:40.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.211+0000 7f1c8508f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1c800c6bf0 con 0x7f1c80152bc0 2024-03-28T23:32:40.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.211+0000 7f1c8508f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1c801eec20 con 0x7f1c801589b0 2024-03-28T23:32:40.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.211+0000 7f1c7ed9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c800c0330 0x7f1c8006c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:40.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.211+0000 7f1c7ed9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c800c0330 0x7f1c8006c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40478/0 (socket says 172.21.15.161:40478) 2024-03-28T23:32:40.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.211+0000 7f1c7ed9d640 1 -- 172.21.15.161:0/2735675624 learned_addr learned my addr 172.21.15.161:0/2735675624 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:40.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.211+0000 7f1c7f59e640 1 --2- 172.21.15.161:0/2735675624 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c801589b0 0x7f1c80154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:40.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.211+0000 7f1c77fff640 1 --2- 172.21.15.161:0/2735675624 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c80152bc0 0x7f1c801547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:40.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.211+0000 7f1c7ed9d640 1 -- 172.21.15.161:0/2735675624 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c801589b0 msgr2=0x7f1c80154d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:40.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.211+0000 7f1c7ed9d640 1 --2- 172.21.15.161:0/2735675624 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c801589b0 0x7f1c80154d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.211+0000 7f1c7ed9d640 1 -- 172.21.15.161:0/2735675624 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c80152bc0 msgr2=0x7f1c801547d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:40.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.211+0000 7f1c7ed9d640 1 --2- 172.21.15.161:0/2735675624 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c80152bc0 0x7f1c801547d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.211+0000 7f1c7ed9d640 1 -- 172.21.15.161:0/2735675624 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c801edd30 con 0x7f1c800c0330 2024-03-28T23:32:40.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.211+0000 7f1c7f59e640 1 --2- 172.21.15.161:0/2735675624 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c801589b0 0x7f1c80154d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:40.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.211+0000 7f1c77fff640 1 --2- 172.21.15.161:0/2735675624 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c80152bc0 0x7f1c801547d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:40.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.211+0000 7f1c7ed9d640 1 --2- 172.21.15.161:0/2735675624 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c800c0330 0x7f1c8006c8e0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f1c7000c410 tx=0x7f1c7000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:40.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.211+0000 7f1c777fe640 1 -- 172.21.15.161:0/2735675624 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c70017070 con 0x7f1c800c0330 2024-03-28T23:32:40.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.211+0000 7f1c777fe640 1 -- 172.21.15.161:0/2735675624 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c700035f0 con 0x7f1c800c0330 2024-03-28T23:32:40.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.211+0000 7f1c777fe640 1 -- 172.21.15.161:0/2735675624 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c700039e0 con 0x7f1c800c0330 2024-03-28T23:32:40.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.211+0000 7f1c8508f640 1 -- 172.21.15.161:0/2735675624 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c801ee010 con 0x7f1c800c0330 2024-03-28T23:32:40.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.211+0000 7f1c8508f640 1 -- 172.21.15.161:0/2735675624 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c801ee5d0 con 0x7f1c800c0330 2024-03-28T23:32:40.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.212+0000 7f1c8508f640 1 -- 172.21.15.161:0/2735675624 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c8004f4b0 con 0x7f1c800c0330 2024-03-28T23:32:40.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.213+0000 7f1c777fe640 1 -- 172.21.15.161:0/2735675624 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1c7001a070 con 0x7f1c800c0330 2024-03-28T23:32:40.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.214+0000 7f1c777fe640 1 --2- 172.21.15.161:0/2735675624 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1c6006b4d0 0x7f1c6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:40.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.214+0000 7f1c77fff640 1 --2- 172.21.15.161:0/2735675624 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1c6006b4d0 0x7f1c6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:40.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.214+0000 7f1c777fe640 1 -- 172.21.15.161:0/2735675624 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6101+0+0 (secure 0 0 0) 0x7f1c70089770 con 0x7f1c800c0330 2024-03-28T23:32:40.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.214+0000 7f1c77fff640 1 --2- 172.21.15.161:0/2735675624 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1c6006b4d0 0x7f1c6006d990 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1c80148a90 tx=0x7f1c68008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:40.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.219+0000 7f1c777fe640 1 -- 172.21.15.161:0/2735675624 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1c700584f0 con 0x7f1c800c0330 2024-03-28T23:32:40.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.395+0000 7f1c8508f640 1 -- 172.21.15.161:0/2735675624 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) -- 0x7f1c800c54a0 con 0x7f1c800c0330 2024-03-28T23:32:40.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.398+0000 7f1c777fe640 1 -- 172.21.15.161:0/2735675624 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v26)=0 v26) ==== 117+0+0 (secure 0 0 0) 0x7f1c700010c0 con 0x7f1c800c0330 2024-03-28T23:32:40.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.400+0000 7f1c8508f640 1 -- 172.21.15.161:0/2735675624 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1c6006b4d0 msgr2=0x7f1c6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:40.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.400+0000 7f1c8508f640 1 --2- 172.21.15.161:0/2735675624 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1c6006b4d0 0x7f1c6006d990 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1c80148a90 tx=0x7f1c68008040 comp rx=0 tx=0).stop 2024-03-28T23:32:40.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.400+0000 7f1c8508f640 1 -- 172.21.15.161:0/2735675624 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c800c0330 msgr2=0x7f1c8006c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:40.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.400+0000 7f1c8508f640 1 --2- 172.21.15.161:0/2735675624 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c800c0330 0x7f1c8006c8e0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f1c7000c410 tx=0x7f1c7000c8d0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.400+0000 7f1c8508f640 1 -- 172.21.15.161:0/2735675624 shutdown_connections 2024-03-28T23:32:40.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.400+0000 7f1c8508f640 1 --2- 172.21.15.161:0/2735675624 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1c6006b4d0 0x7f1c6006d990 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.400+0000 7f1c8508f640 1 --2- 172.21.15.161:0/2735675624 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c801589b0 0x7f1c80154d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.400+0000 7f1c8508f640 1 --2- 172.21.15.161:0/2735675624 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c80152bc0 0x7f1c801547d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.400+0000 7f1c8508f640 1 --2- 172.21.15.161:0/2735675624 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c800c0330 0x7f1c8006c8e0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.400+0000 7f1c8508f640 1 -- 172.21.15.161:0/2735675624 >> 172.21.15.161:0/2735675624 conn(0x7f1c800caa80 msgr2=0x7f1c800c5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:40.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.400+0000 7f1c8508f640 1 -- 172.21.15.161:0/2735675624 shutdown_connections 2024-03-28T23:32:40.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.401+0000 7f1c8508f640 1 -- 172.21.15.161:0/2735675624 wait complete. 2024-03-28T23:32:40.412 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:32:40.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.529+0000 7f525b7d0640 1 Processor -- start 2024-03-28T23:32:40.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.529+0000 7f525b7d0640 1 -- start start 2024-03-28T23:32:40.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.530+0000 7f525b7d0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f52540aacb0 0x7f5254153860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:40.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.530+0000 7f525b7d0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52541566f0 con 0x7f52540a8fc0 2024-03-28T23:32:40.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.530+0000 7f525b7d0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52541569d0 con 0x7f5254150590 2024-03-28T23:32:40.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.530+0000 7f525b7d0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f5254156cb0 con 0x7f52540aacb0 2024-03-28T23:32:40.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.530+0000 7f5258d6b640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f52540a8fc0 0x7f5254153150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59246/0 (socket says 172.21.15.161:59246) 2024-03-28T23:32:40.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.530+0000 7f5258d6b640 1 -- 172.21.15.161:0/2783324029 learned_addr learned my addr 172.21.15.161:0/2783324029 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:40.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.530+0000 7f524bfff640 1 -- 172.21.15.161:0/2783324029 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1122218230 0 0) 0x7f52541566f0 con 0x7f52540a8fc0 2024-03-28T23:32:40.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.530+0000 7f5259d6d640 1 --2- 172.21.15.161:0/2783324029 >> v2:172.21.15.170:3300/0 conn(0x7f52540aacb0 0x7f5254153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:40.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.530+0000 7f524bfff640 1 -- 172.21.15.161:0/2783324029 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5240003520 con 0x7f52540a8fc0 2024-03-28T23:32:40.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.530+0000 7f524bfff640 1 -- 172.21.15.161:0/2783324029 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1606960523 0 0) 0x7f5240003520 con 0x7f52540a8fc0 2024-03-28T23:32:40.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.530+0000 7f524bfff640 1 -- 172.21.15.161:0/2783324029 >> v2:172.21.15.170:3300/0 conn(0x7f52540aacb0 msgr2=0x7f5254153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:40.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.530+0000 7f524bfff640 1 --2- 172.21.15.161:0/2783324029 >> v2:172.21.15.170:3300/0 conn(0x7f52540aacb0 0x7f5254153860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.530+0000 7f524bfff640 1 -- 172.21.15.161:0/2783324029 >> v1:172.21.15.170:6789/0 conn(0x7f5254150590 legacy=0x7f52540a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:40.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.530+0000 7f524bfff640 1 -- 172.21.15.161:0/2783324029 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5254156f90 con 0x7f52540a8fc0 2024-03-28T23:32:40.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.530+0000 7f524bfff640 1 -- 172.21.15.161:0/2783324029 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f5244001670 con 0x7f52540a8fc0 2024-03-28T23:32:40.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.530+0000 7f524bfff640 1 -- 172.21.15.161:0/2783324029 >> v1:172.21.15.161:6789/0 conn(0x7f52540a8fc0 legacy=0x7f5254153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:40.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.530+0000 7f524bfff640 1 --2- 172.21.15.161:0/2783324029 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5240003a90 0x7f5240003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:40.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.530+0000 7f524bfff640 1 --2- 172.21.15.161:0/2783324029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5240004800 0x7f5240004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:40.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.530+0000 7f524bfff640 1 --2- 172.21.15.161:0/2783324029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52400053c0 0x7f524000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:40.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.530+0000 7f524bfff640 1 -- 172.21.15.161:0/2783324029 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5254156cb0 con 0x7f52400053c0 2024-03-28T23:32:40.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.530+0000 7f524bfff640 1 -- 172.21.15.161:0/2783324029 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f524000c280 con 0x7f5240004800 2024-03-28T23:32:40.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.530+0000 7f524bfff640 1 -- 172.21.15.161:0/2783324029 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f524000c480 con 0x7f5240003a90 2024-03-28T23:32:40.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.530+0000 7f5259d6d640 1 --2- 172.21.15.161:0/2783324029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52400053c0 0x7f524000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:40.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.531+0000 7f525956c640 1 --2- 172.21.15.161:0/2783324029 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5240003a90 0x7f5240003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:40.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.531+0000 7f5258d6b640 1 --2- 172.21.15.161:0/2783324029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5240004800 0x7f5240004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:40.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.531+0000 7f5259d6d640 1 -- 172.21.15.161:0/2783324029 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5240003a90 msgr2=0x7f5240003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:40.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.531+0000 7f5259d6d640 1 --2- 172.21.15.161:0/2783324029 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5240003a90 0x7f5240003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.531+0000 7f5259d6d640 1 -- 172.21.15.161:0/2783324029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5240004800 msgr2=0x7f5240004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:40.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.531+0000 7f5259d6d640 1 --2- 172.21.15.161:0/2783324029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5240004800 0x7f5240004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.531+0000 7f5259d6d640 1 -- 172.21.15.161:0/2783324029 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5240003520 con 0x7f52400053c0 2024-03-28T23:32:40.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.531+0000 7f5259d6d640 1 -- 172.21.15.161:0/2783324029 reap_dead start 2024-03-28T23:32:40.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.531+0000 7f5259d6d640 1 --2- 172.21.15.161:0/2783324029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52400053c0 0x7f524000b9c0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f524c009fc0 tx=0x7f5240005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92404c0ead83c940 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:40.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.532+0000 7f524bfff640 1 -- 172.21.15.161:0/2783324029 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524c009c10 con 0x7f52400053c0 2024-03-28T23:32:40.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.532+0000 7f524bfff640 1 -- 172.21.15.161:0/2783324029 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f524c02cca0 con 0x7f52400053c0 2024-03-28T23:32:40.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.532+0000 7f524bfff640 1 -- 172.21.15.161:0/2783324029 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524c003bd0 con 0x7f52400053c0 2024-03-28T23:32:40.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.532+0000 7f525b7d0640 1 -- 172.21.15.161:0/2783324029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52400053c0 msgr2=0x7f524000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:40.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.532+0000 7f525b7d0640 1 --2- 172.21.15.161:0/2783324029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52400053c0 0x7f524000b9c0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f524c009fc0 tx=0x7f5240005830 comp rx=0 tx=0).stop 2024-03-28T23:32:40.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.532+0000 7f525b7d0640 1 -- 172.21.15.161:0/2783324029 shutdown_connections 2024-03-28T23:32:40.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.532+0000 7f525b7d0640 1 --2- 172.21.15.161:0/2783324029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52400053c0 0x7f524000b9c0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.532+0000 7f525b7d0640 1 -- 172.21.15.161:0/2783324029 >> 172.21.15.161:0/2783324029 conn(0x7f52540c6670 msgr2=0x7f52540c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:40.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.532+0000 7f525b7d0640 1 -- 172.21.15.161:0/2783324029 shutdown_connections 2024-03-28T23:32:40.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.532+0000 7f525b7d0640 1 -- 172.21.15.161:0/2783324029 wait complete. 2024-03-28T23:32:40.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.533+0000 7f525b7d0640 1 Processor -- start 2024-03-28T23:32:40.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.533+0000 7f525b7d0640 1 -- start start 2024-03-28T23:32:40.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.534+0000 7f525b7d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52400053c0 0x7f5254145cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:40.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.534+0000 7f525b7d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5254150cd0 0x7f52541ec2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:40.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.534+0000 7f525b7d0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52540c5950 0x7f52540c5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:40.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.534+0000 7f525b7d0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f52540a9f40 con 0x7f52540c5950 2024-03-28T23:32:40.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.534+0000 7f525b7d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5254150970 con 0x7f5254150cd0 2024-03-28T23:32:40.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.534+0000 7f525b7d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f52540aa0c0 con 0x7f52400053c0 2024-03-28T23:32:40.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.534+0000 7f5259d6d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52540c5950 0x7f52540c5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:40.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.534+0000 7f5259d6d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52540c5950 0x7f52540c5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39712/0 (socket says 172.21.15.161:39712) 2024-03-28T23:32:40.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.534+0000 7f5259d6d640 1 -- 172.21.15.161:0/3041879991 learned_addr learned my addr 172.21.15.161:0/3041879991 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:40.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.534+0000 7f525956c640 1 --2- 172.21.15.161:0/3041879991 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52400053c0 0x7f5254145cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:40.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.534+0000 7f5258d6b640 1 --2- 172.21.15.161:0/3041879991 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5254150cd0 0x7f52541ec2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:40.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.534+0000 7f5259d6d640 1 -- 172.21.15.161:0/3041879991 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52400053c0 msgr2=0x7f5254145cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:40.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.534+0000 7f5259d6d640 1 --2- 172.21.15.161:0/3041879991 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52400053c0 0x7f5254145cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.534+0000 7f5259d6d640 1 -- 172.21.15.161:0/3041879991 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5254150cd0 msgr2=0x7f52541ec2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:40.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.534+0000 7f5259d6d640 1 --2- 172.21.15.161:0/3041879991 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5254150cd0 0x7f52541ec2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.534+0000 7f5259d6d640 1 -- 172.21.15.161:0/3041879991 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52541ecbd0 con 0x7f52540c5950 2024-03-28T23:32:40.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.534+0000 7f5258d6b640 1 --2- 172.21.15.161:0/3041879991 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5254150cd0 0x7f52541ec2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:40.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.534+0000 7f525956c640 1 --2- 172.21.15.161:0/3041879991 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52400053c0 0x7f5254145cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:40.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.535+0000 7f5259d6d640 1 --2- 172.21.15.161:0/3041879991 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52540c5950 0x7f52540c5dd0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f524c02c6d0 tx=0x7f524c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:40.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.535+0000 7f5249ffb640 1 -- 172.21.15.161:0/3041879991 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524c042020 con 0x7f52540c5950 2024-03-28T23:32:40.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.535+0000 7f5249ffb640 1 -- 172.21.15.161:0/3041879991 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f524c02ce50 con 0x7f52540c5950 2024-03-28T23:32:40.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.535+0000 7f5249ffb640 1 -- 172.21.15.161:0/3041879991 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524c003b90 con 0x7f52540c5950 2024-03-28T23:32:40.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.535+0000 7f525b7d0640 1 -- 172.21.15.161:0/3041879991 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5254145a10 con 0x7f52540c5950 2024-03-28T23:32:40.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.535+0000 7f525b7d0640 1 -- 172.21.15.161:0/3041879991 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52540c3990 con 0x7f52540c5950 2024-03-28T23:32:40.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.536+0000 7f525b7d0640 1 -- 172.21.15.161:0/3041879991 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f525414d860 con 0x7f52540c5950 2024-03-28T23:32:40.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.537+0000 7f5249ffb640 1 -- 172.21.15.161:0/3041879991 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f524c045020 con 0x7f52540c5950 2024-03-28T23:32:40.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.537+0000 7f5249ffb640 1 --2- 172.21.15.161:0/3041879991 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f523806b4d0 0x7f523806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:40.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.537+0000 7f5249ffb640 1 -- 172.21.15.161:0/3041879991 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6101+0+0 (secure 0 0 0) 0x7f524c0acd50 con 0x7f52540c5950 2024-03-28T23:32:40.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.537+0000 7f525956c640 1 --2- 172.21.15.161:0/3041879991 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f523806b4d0 0x7f523806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:40.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.538+0000 7f525956c640 1 --2- 172.21.15.161:0/3041879991 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f523806b4d0 0x7f523806d990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f52540c0ab0 tx=0x7f5250009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:40.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.542+0000 7f5249ffb640 1 -- 172.21.15.161:0/3041879991 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f525414d860 con 0x7f52540c5950 2024-03-28T23:32:40.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.716+0000 7f525b7d0640 1 -- 172.21.15.161:0/3041879991 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f525404f4b0 con 0x7f52540c5950 2024-03-28T23:32:40.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.717+0000 7f5249ffb640 1 -- 172.21.15.161:0/3041879991 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+16981 (secure 0 0 0) 0x7f524c0b0030 con 0x7f52540c5950 2024-03-28T23:32:40.718 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:40.718 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":73,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:32:34.022058+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:32:30.684936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-28T23:32:31.339722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:32:40.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.719+0000 7f525b7d0640 1 -- 172.21.15.161:0/3041879991 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f523806b4d0 msgr2=0x7f523806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:40.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.719+0000 7f525b7d0640 1 --2- 172.21.15.161:0/3041879991 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f523806b4d0 0x7f523806d990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f52540c0ab0 tx=0x7f5250009040 comp rx=0 tx=0).stop 2024-03-28T23:32:40.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.719+0000 7f525b7d0640 1 -- 172.21.15.161:0/3041879991 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52540c5950 msgr2=0x7f52540c5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:40.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.719+0000 7f525b7d0640 1 --2- 172.21.15.161:0/3041879991 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52540c5950 0x7f52540c5dd0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f524c02c6d0 tx=0x7f524c009750 comp rx=0 tx=0).stop 2024-03-28T23:32:40.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.719+0000 7f525b7d0640 1 -- 172.21.15.161:0/3041879991 shutdown_connections 2024-03-28T23:32:40.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.719+0000 7f525b7d0640 1 --2- 172.21.15.161:0/3041879991 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f523806b4d0 0x7f523806d990 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.719+0000 7f525b7d0640 1 --2- 172.21.15.161:0/3041879991 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52540c5950 0x7f52540c5dd0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.719+0000 7f525b7d0640 1 --2- 172.21.15.161:0/3041879991 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5254150cd0 0x7f52541ec2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.719+0000 7f525b7d0640 1 --2- 172.21.15.161:0/3041879991 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52400053c0 0x7f5254145cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.719+0000 7f525b7d0640 1 -- 172.21.15.161:0/3041879991 >> 172.21.15.161:0/3041879991 conn(0x7f52540c6670 msgr2=0x7f525414c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:40.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.719+0000 7f525b7d0640 1 -- 172.21.15.161:0/3041879991 shutdown_connections 2024-03-28T23:32:40.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.719+0000 7f525b7d0640 1 -- 172.21.15.161:0/3041879991 wait complete. 2024-03-28T23:32:40.734 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:32:30.684936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:32:31.339722+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:32:40.735 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:32:40.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.840+0000 7f7348ed4640 1 Processor -- start 2024-03-28T23:32:40.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.840+0000 7f7348ed4640 1 -- start start 2024-03-28T23:32:40.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.841+0000 7f7348ed4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f73441529d0 0x7f734406bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:40.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.841+0000 7f7348ed4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f73440c88b0 0x7f73440c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:40.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.841+0000 7f7348ed4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7344152db0 con 0x7f73440c6750 2024-03-28T23:32:40.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.841+0000 7f7348ed4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f73440c7210 con 0x7f73440c88b0 2024-03-28T23:32:40.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.841+0000 7f7348ed4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f73440c6c50 con 0x7f73441529d0 2024-03-28T23:32:40.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.841+0000 7f7342d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f73440c88b0 0x7f73440c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:40.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.841+0000 7f7339d9b640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f73440c6750 0x7f7344155c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59260/0 (socket says 172.21.15.161:59260) 2024-03-28T23:32:40.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.841+0000 7f7339d9b640 1 -- 172.21.15.161:0/1195667325 learned_addr learned my addr 172.21.15.161:0/1195667325 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:40.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.841+0000 7f734259c640 1 --2- 172.21.15.161:0/1195667325 >> v2:172.21.15.170:3300/0 conn(0x7f73441529d0 0x7f734406bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:40.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.842+0000 7f7342d9d640 1 -- 172.21.15.161:0/1195667325 >> v2:172.21.15.170:3300/0 conn(0x7f73441529d0 msgr2=0x7f734406bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:40.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.842+0000 7f7342d9d640 1 --2- 172.21.15.161:0/1195667325 >> v2:172.21.15.170:3300/0 conn(0x7f73441529d0 0x7f734406bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.842+0000 7f7342d9d640 1 -- 172.21.15.161:0/1195667325 >> v1:172.21.15.161:6789/0 conn(0x7f73440c6750 legacy=0x7f7344155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:40.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.842+0000 7f7342d9d640 1 -- 172.21.15.161:0/1195667325 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73440c8d40 con 0x7f73440c88b0 2024-03-28T23:32:40.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.842+0000 7f7341d9b640 1 -- 172.21.15.161:0/1195667325 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2067298028 0 0) 0x7f7344152db0 con 0x7f73440c6750 2024-03-28T23:32:40.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.842+0000 7f734259c640 1 --2- 172.21.15.161:0/1195667325 >> v2:172.21.15.170:3300/0 conn(0x7f73441529d0 0x7f734406bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:40.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.842+0000 7f7342d9d640 1 --2- 172.21.15.161:0/1195667325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73440c88b0 0x7f73440c54f0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f73440689e0 tx=0x7f733c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eaef0bb23c24c4c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:40.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.842+0000 7f7341d9b640 1 -- 172.21.15.161:0/1195667325 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c039070 con 0x7f73440c88b0 2024-03-28T23:32:40.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.842+0000 7f7341d9b640 1 -- 172.21.15.161:0/1195667325 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f733c02cc80 con 0x7f73440c88b0 2024-03-28T23:32:40.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.842+0000 7f7341d9b640 1 -- 172.21.15.161:0/1195667325 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c034660 con 0x7f73440c88b0 2024-03-28T23:32:40.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.843+0000 7f7348ed4640 1 -- 172.21.15.161:0/1195667325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73440c88b0 msgr2=0x7f73440c54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:40.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.843+0000 7f7348ed4640 1 --2- 172.21.15.161:0/1195667325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73440c88b0 0x7f73440c54f0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f73440689e0 tx=0x7f733c02bed0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.843+0000 7f7348ed4640 1 -- 172.21.15.161:0/1195667325 shutdown_connections 2024-03-28T23:32:40.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.843+0000 7f7348ed4640 1 --2- 172.21.15.161:0/1195667325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73440c88b0 0x7f73440c54f0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.843+0000 7f7348ed4640 1 --2- 172.21.15.161:0/1195667325 >> v2:172.21.15.170:3300/0 conn(0x7f73441529d0 0x7f734406bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.843+0000 7f7348ed4640 1 -- 172.21.15.161:0/1195667325 >> 172.21.15.161:0/1195667325 conn(0x7f73440c7430 msgr2=0x7f73440c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:40.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.843+0000 7f7348ed4640 1 -- 172.21.15.161:0/1195667325 shutdown_connections 2024-03-28T23:32:40.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.843+0000 7f7348ed4640 1 -- 172.21.15.161:0/1195667325 wait complete. 2024-03-28T23:32:40.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.845+0000 7f7348ed4640 1 Processor -- start 2024-03-28T23:32:40.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.845+0000 7f7348ed4640 1 -- start start 2024-03-28T23:32:40.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.846+0000 7f7348ed4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73440c6750 0x7f7344149210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:40.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.846+0000 7f7348ed4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73440c88b0 0x7f73440aeef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:40.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.846+0000 7f7348ed4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73441529d0 0x7f73440c0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:40.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.846+0000 7f7348ed4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7344155710 con 0x7f73440c88b0 2024-03-28T23:32:40.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.846+0000 7f7348ed4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7344155430 con 0x7f73441529d0 2024-03-28T23:32:40.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.846+0000 7f7348ed4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f73440aa160 con 0x7f73440c6750 2024-03-28T23:32:40.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.846+0000 7f7339d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73440c88b0 0x7f73440aeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:40.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.846+0000 7f7339d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73440c88b0 0x7f73440aeef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39730/0 (socket says 172.21.15.161:39730) 2024-03-28T23:32:40.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.846+0000 7f7339d9b640 1 -- 172.21.15.161:0/3364402281 learned_addr learned my addr 172.21.15.161:0/3364402281 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:40.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.846+0000 7f7342d9d640 1 --2- 172.21.15.161:0/3364402281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73441529d0 0x7f73440c0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:40.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.847+0000 7f734259c640 1 --2- 172.21.15.161:0/3364402281 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73440c6750 0x7f7344149210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:40.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.847+0000 7f7339d9b640 1 -- 172.21.15.161:0/3364402281 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73440c6750 msgr2=0x7f7344149210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:40.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.847+0000 7f7339d9b640 1 --2- 172.21.15.161:0/3364402281 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73440c6750 0x7f7344149210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.847+0000 7f7339d9b640 1 -- 172.21.15.161:0/3364402281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73441529d0 msgr2=0x7f73440c0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:40.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.847+0000 7f7339d9b640 1 --2- 172.21.15.161:0/3364402281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73441529d0 0x7f73440c0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:40.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.847+0000 7f7339d9b640 1 -- 172.21.15.161:0/3364402281 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f734414df40 con 0x7f73440c88b0 2024-03-28T23:32:40.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.847+0000 7f7339d9b640 1 --2- 172.21.15.161:0/3364402281 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73440c88b0 0x7f73440aeef0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f732c00c430 tx=0x7f732c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:40.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.847+0000 7f733b7fe640 1 -- 172.21.15.161:0/3364402281 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f732c0074e0 con 0x7f73440c88b0 2024-03-28T23:32:40.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.847+0000 7f733b7fe640 1 -- 172.21.15.161:0/3364402281 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f732c0047e0 con 0x7f73440c88b0 2024-03-28T23:32:40.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.847+0000 7f733b7fe640 1 -- 172.21.15.161:0/3364402281 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f732c004bd0 con 0x7f73440c88b0 2024-03-28T23:32:40.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.847+0000 7f7348ed4640 1 -- 172.21.15.161:0/3364402281 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f734414ec50 con 0x7f73440c88b0 2024-03-28T23:32:40.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.847+0000 7f7348ed4640 1 -- 172.21.15.161:0/3364402281 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f734414f210 con 0x7f73440c88b0 2024-03-28T23:32:40.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.848+0000 7f7348ed4640 1 -- 172.21.15.161:0/3364402281 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f734404f4b0 con 0x7f73440c88b0 2024-03-28T23:32:40.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.849+0000 7f733b7fe640 1 -- 172.21.15.161:0/3364402281 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f732c011070 con 0x7f73440c88b0 2024-03-28T23:32:40.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.849+0000 7f733b7fe640 1 --2- 172.21.15.161:0/3364402281 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f732406b4d0 0x7f732406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:40.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.849+0000 7f733b7fe640 1 -- 172.21.15.161:0/3364402281 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6101+0+0 (secure 0 0 0) 0x7f732c089eb0 con 0x7f73440c88b0 2024-03-28T23:32:40.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.849+0000 7f734259c640 1 --2- 172.21.15.161:0/3364402281 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f732406b4d0 0x7f732406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:40.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.850+0000 7f734259c640 1 --2- 172.21.15.161:0/3364402281 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f732406b4d0 0x7f732406d990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7334006fd0 tx=0x7f7334008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:40.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:40.853+0000 7f733b7fe640 1 -- 172.21.15.161:0/3364402281 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f734414df40 con 0x7f73440c88b0 2024-03-28T23:32:41.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.030+0000 7f7348ed4640 1 -- 172.21.15.161:0/3364402281 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7344153290 con 0x7f73440c88b0 2024-03-28T23:32:41.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.031+0000 7f733b7fe640 1 -- 172.21.15.161:0/3364402281 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) ==== 93+0+10 (secure 0 0 0) 0x7f732c0584c0 con 0x7f73440c88b0 2024-03-28T23:32:41.031 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:32:41.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.033+0000 7f7348ed4640 1 -- 172.21.15.161:0/3364402281 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f732406b4d0 msgr2=0x7f732406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:41.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.033+0000 7f7348ed4640 1 --2- 172.21.15.161:0/3364402281 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f732406b4d0 0x7f732406d990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7334006fd0 tx=0x7f7334008040 comp rx=0 tx=0).stop 2024-03-28T23:32:41.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.033+0000 7f7348ed4640 1 -- 172.21.15.161:0/3364402281 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73440c88b0 msgr2=0x7f73440aeef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:41.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.033+0000 7f7348ed4640 1 --2- 172.21.15.161:0/3364402281 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73440c88b0 0x7f73440aeef0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f732c00c430 tx=0x7f732c00c8f0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.033+0000 7f7348ed4640 1 -- 172.21.15.161:0/3364402281 shutdown_connections 2024-03-28T23:32:41.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.033+0000 7f7348ed4640 1 --2- 172.21.15.161:0/3364402281 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f732406b4d0 0x7f732406d990 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.033+0000 7f7348ed4640 1 --2- 172.21.15.161:0/3364402281 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73440c88b0 0x7f73440aeef0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.033+0000 7f7348ed4640 1 --2- 172.21.15.161:0/3364402281 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73441529d0 0x7f73440c0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.033+0000 7f7348ed4640 1 --2- 172.21.15.161:0/3364402281 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73440c6750 0x7f7344149210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.033+0000 7f7348ed4640 1 -- 172.21.15.161:0/3364402281 >> 172.21.15.161:0/3364402281 conn(0x7f73440c7430 msgr2=0x7f73440c5490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:41.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.033+0000 7f7348ed4640 1 -- 172.21.15.161:0/3364402281 shutdown_connections 2024-03-28T23:32:41.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.033+0000 7f7348ed4640 1 -- 172.21.15.161:0/3364402281 wait complete. 2024-03-28T23:32:41.047 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:32:41.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.151+0000 7f02a76cd640 1 Processor -- start 2024-03-28T23:32:41.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.151+0000 7f02a76cd640 1 -- start start 2024-03-28T23:32:41.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.152+0000 7f02a76cd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f02a0152bc0 0x7f02a00c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:41.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.152+0000 7f02a76cd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02a00c8a80 0x7f02a00c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:41.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.152+0000 7f02a76cd640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02a00c6f60 con 0x7f02a00c9420 2024-03-28T23:32:41.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.152+0000 7f02a76cd640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f02a00c6120 con 0x7f02a0152bc0 2024-03-28T23:32:41.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.152+0000 7f02a76cd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f02a00c6400 con 0x7f02a00c8a80 2024-03-28T23:32:41.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.152+0000 7f02a5469640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f02a0152bc0 0x7f02a00c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:41.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.152+0000 7f02a5c6a640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f02a00c9420 0x7f02a01589b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59262/0 (socket says 172.21.15.161:59262) 2024-03-28T23:32:41.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.152+0000 7f02a5c6a640 1 -- 172.21.15.161:0/4204523399 learned_addr learned my addr 172.21.15.161:0/4204523399 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:41.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.153+0000 7f02a5469640 1 -- 172.21.15.161:0/4204523399 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02a00c8a80 msgr2=0x7f02a00c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:41.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.153+0000 7f02a4c68640 1 --2- 172.21.15.161:0/4204523399 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02a00c8a80 0x7f02a00c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:41.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.153+0000 7f0297fff640 1 -- 172.21.15.161:0/4204523399 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2528907307 0 0) 0x7f02a00c6f60 con 0x7f02a00c9420 2024-03-28T23:32:41.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.153+0000 7f02a5469640 1 --2- 172.21.15.161:0/4204523399 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02a00c8a80 0x7f02a00c8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.153+0000 7f02a5469640 1 -- 172.21.15.161:0/4204523399 >> v1:172.21.15.161:6789/0 conn(0x7f02a00c9420 legacy=0x7f02a01589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:41.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.153+0000 7f02a5469640 1 -- 172.21.15.161:0/4204523399 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02a00c7240 con 0x7f02a0152bc0 2024-03-28T23:32:41.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.153+0000 7f02a4c68640 1 --2- 172.21.15.161:0/4204523399 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02a00c8a80 0x7f02a00c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:41.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.153+0000 7f02a5469640 1 --2- 172.21.15.161:0/4204523399 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02a0152bc0 0x7f02a00c0330 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f02900097a0 tx=0x7f029002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f71f1f31dbf96a13 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:41.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.153+0000 7f0297fff640 1 -- 172.21.15.161:0/4204523399 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0290035020 con 0x7f02a0152bc0 2024-03-28T23:32:41.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.154+0000 7f0297fff640 1 -- 172.21.15.161:0/4204523399 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0290031830 con 0x7f02a0152bc0 2024-03-28T23:32:41.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.154+0000 7f0297fff640 1 -- 172.21.15.161:0/4204523399 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0290031c00 con 0x7f02a0152bc0 2024-03-28T23:32:41.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.154+0000 7f02a76cd640 1 -- 172.21.15.161:0/4204523399 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02a0152bc0 msgr2=0x7f02a00c0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:41.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.154+0000 7f02a76cd640 1 --2- 172.21.15.161:0/4204523399 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02a0152bc0 0x7f02a00c0330 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f02900097a0 tx=0x7f029002b8d0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.154+0000 7f02a76cd640 1 -- 172.21.15.161:0/4204523399 shutdown_connections 2024-03-28T23:32:41.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.154+0000 7f02a76cd640 1 --2- 172.21.15.161:0/4204523399 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02a00c8a80 0x7f02a00c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.154+0000 7f02a76cd640 1 --2- 172.21.15.161:0/4204523399 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02a0152bc0 0x7f02a00c0330 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.154+0000 7f02a76cd640 1 -- 172.21.15.161:0/4204523399 >> 172.21.15.161:0/4204523399 conn(0x7f02a00caa80 msgr2=0x7f02a00cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:41.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.154+0000 7f02a76cd640 1 -- 172.21.15.161:0/4204523399 shutdown_connections 2024-03-28T23:32:41.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.154+0000 7f02a76cd640 1 -- 172.21.15.161:0/4204523399 wait complete. 2024-03-28T23:32:41.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.156+0000 7f02a76cd640 1 Processor -- start 2024-03-28T23:32:41.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.156+0000 7f02a76cd640 1 -- start start 2024-03-28T23:32:41.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.157+0000 7f02a76cd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02a00c8a80 0x7f02a01576e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:41.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.157+0000 7f02a76cd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02a00c9420 0x7f02a01eaa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:41.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.157+0000 7f02a76cd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f02a0152bc0 0x7f02a0151a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:41.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.157+0000 7f02a76cd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f02a0152fa0 con 0x7f02a00c9420 2024-03-28T23:32:41.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.157+0000 7f02a76cd640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f02a01590c0 con 0x7f02a0152bc0 2024-03-28T23:32:41.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.157+0000 7f02a76cd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f02a01562e0 con 0x7f02a00c8a80 2024-03-28T23:32:41.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.157+0000 7f02a4c68640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02a00c9420 0x7f02a01eaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:41.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.157+0000 7f02a4c68640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02a00c9420 0x7f02a01eaa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39742/0 (socket says 172.21.15.161:39742) 2024-03-28T23:32:41.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.157+0000 7f02a4c68640 1 -- 172.21.15.161:0/2815641576 learned_addr learned my addr 172.21.15.161:0/2815641576 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:41.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.157+0000 7f02a5469640 1 --2- 172.21.15.161:0/2815641576 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02a00c8a80 0x7f02a01576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:41.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.157+0000 7f02a5c6a640 1 --2- 172.21.15.161:0/2815641576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f02a0152bc0 0x7f02a0151a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:41.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.158+0000 7f02a4c68640 1 -- 172.21.15.161:0/2815641576 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02a00c8a80 msgr2=0x7f02a01576e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:41.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.158+0000 7f02a4c68640 1 --2- 172.21.15.161:0/2815641576 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02a00c8a80 0x7f02a01576e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.158+0000 7f02a4c68640 1 -- 172.21.15.161:0/2815641576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f02a0152bc0 msgr2=0x7f02a0151a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:41.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.158+0000 7f02a4c68640 1 --2- 172.21.15.161:0/2815641576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f02a0152bc0 0x7f02a0151a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.158+0000 7f02a4c68640 1 -- 172.21.15.161:0/2815641576 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02a0157c20 con 0x7f02a00c9420 2024-03-28T23:32:41.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.158+0000 7f02a5c6a640 1 --2- 172.21.15.161:0/2815641576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f02a0152bc0 0x7f02a0151a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:41.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.158+0000 7f02a4c68640 1 --2- 172.21.15.161:0/2815641576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02a00c9420 0x7f02a01eaa80 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f029c00c7f0 tx=0x7f029c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:41.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.158+0000 7f0295ffb640 1 -- 172.21.15.161:0/2815641576 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029c015070 con 0x7f02a00c9420 2024-03-28T23:32:41.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.158+0000 7f0295ffb640 1 -- 172.21.15.161:0/2815641576 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f029c007a30 con 0x7f02a00c9420 2024-03-28T23:32:41.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.158+0000 7f02a76cd640 1 -- 172.21.15.161:0/2815641576 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02a01eafc0 con 0x7f02a00c9420 2024-03-28T23:32:41.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.158+0000 7f0295ffb640 1 -- 172.21.15.161:0/2815641576 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029c007e00 con 0x7f02a00c9420 2024-03-28T23:32:41.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.158+0000 7f02a76cd640 1 -- 172.21.15.161:0/2815641576 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02a0155c90 con 0x7f02a00c9420 2024-03-28T23:32:41.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.159+0000 7f02a76cd640 1 -- 172.21.15.161:0/2815641576 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02a0159450 con 0x7f02a00c9420 2024-03-28T23:32:41.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.160+0000 7f0295ffb640 1 -- 172.21.15.161:0/2815641576 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f029c018070 con 0x7f02a00c9420 2024-03-28T23:32:41.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.160+0000 7f0295ffb640 1 --2- 172.21.15.161:0/2815641576 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f028406b4d0 0x7f028406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:41.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.160+0000 7f02a5469640 1 --2- 172.21.15.161:0/2815641576 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f028406b4d0 0x7f028406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:41.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.160+0000 7f0295ffb640 1 -- 172.21.15.161:0/2815641576 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6101+0+0 (secure 0 0 0) 0x7f029c08a250 con 0x7f02a00c9420 2024-03-28T23:32:41.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.161+0000 7f02a5469640 1 --2- 172.21.15.161:0/2815641576 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f028406b4d0 0x7f028406d990 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0290036000 tx=0x7f029003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:41.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.165+0000 7f0295ffb640 1 -- 172.21.15.161:0/2815641576 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f02a0159450 con 0x7f02a00c9420 2024-03-28T23:32:41.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.340+0000 7f02a76cd640 1 -- 172.21.15.161:0/2815641576 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f02a0068f90 con 0x7f02a00c9420 2024-03-28T23:32:41.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.341+0000 7f0295ffb640 1 -- 172.21.15.161:0/2815641576 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v73) ==== 104+0+11 (secure 0 0 0) 0x7f02a0068f90 con 0x7f02a00c9420 2024-03-28T23:32:41.341 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:32:41.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.342+0000 7f02a76cd640 1 -- 172.21.15.161:0/2815641576 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f028406b4d0 msgr2=0x7f028406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:41.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.342+0000 7f02a76cd640 1 --2- 172.21.15.161:0/2815641576 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f028406b4d0 0x7f028406d990 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0290036000 tx=0x7f029003c040 comp rx=0 tx=0).stop 2024-03-28T23:32:41.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.342+0000 7f02a76cd640 1 -- 172.21.15.161:0/2815641576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02a00c9420 msgr2=0x7f02a01eaa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:41.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.342+0000 7f02a76cd640 1 --2- 172.21.15.161:0/2815641576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02a00c9420 0x7f02a01eaa80 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f029c00c7f0 tx=0x7f029c00cbf0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.343+0000 7f02a76cd640 1 -- 172.21.15.161:0/2815641576 shutdown_connections 2024-03-28T23:32:41.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.343+0000 7f02a76cd640 1 --2- 172.21.15.161:0/2815641576 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f028406b4d0 0x7f028406d990 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.343+0000 7f02a76cd640 1 --2- 172.21.15.161:0/2815641576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02a00c9420 0x7f02a01eaa80 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.343+0000 7f02a76cd640 1 --2- 172.21.15.161:0/2815641576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f02a0152bc0 0x7f02a0151a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.343+0000 7f02a76cd640 1 --2- 172.21.15.161:0/2815641576 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02a00c8a80 0x7f02a01576e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.343+0000 7f02a76cd640 1 -- 172.21.15.161:0/2815641576 >> 172.21.15.161:0/2815641576 conn(0x7f02a00caa80 msgr2=0x7f02a014e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:41.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.343+0000 7f02a76cd640 1 -- 172.21.15.161:0/2815641576 shutdown_connections 2024-03-28T23:32:41.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.343+0000 7f02a76cd640 1 -- 172.21.15.161:0/2815641576 wait complete. 2024-03-28T23:32:41.354 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:32:41.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.470+0000 7fd416948640 1 Processor -- start 2024-03-28T23:32:41.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.470+0000 7fd416948640 1 -- start start 2024-03-28T23:32:41.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.471+0000 7fd416948640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd4101529d0 0x7fd41006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:41.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.471+0000 7fd416948640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd4100c6750 0x7fd4100c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:41.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.471+0000 7fd416948640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd410152db0 con 0x7fd410155c30 2024-03-28T23:32:41.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.471+0000 7fd416948640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd4100c7210 con 0x7fd4100c6750 2024-03-28T23:32:41.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.471+0000 7fd416948640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd4100c6c50 con 0x7fd4101529d0 2024-03-28T23:32:41.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.471+0000 7fd40f7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd4100c6750 0x7fd4100c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:41.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.471+0000 7fd414ee5640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd410155c30 0x7fd4100c54f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59276/0 (socket says 172.21.15.161:59276) 2024-03-28T23:32:41.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.471+0000 7fd40f7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd4100c6750 0x7fd4100c88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39744/0 (socket says 172.21.15.161:39744) 2024-03-28T23:32:41.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.471+0000 7fd414ee5640 1 -- 172.21.15.161:0/840846045 learned_addr learned my addr 172.21.15.161:0/840846045 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:41.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.471+0000 7fd40f7fe640 1 -- 172.21.15.161:0/840846045 >> v2:172.21.15.170:3300/0 conn(0x7fd4101529d0 msgr2=0x7fd41006bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:41.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.471+0000 7fd40f7fe640 1 --2- 172.21.15.161:0/840846045 >> v2:172.21.15.170:3300/0 conn(0x7fd4101529d0 0x7fd41006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.471+0000 7fd40f7fe640 1 -- 172.21.15.161:0/840846045 >> v1:172.21.15.161:6789/0 conn(0x7fd410155c30 legacy=0x7fd4100c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:41.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.471+0000 7fd40effd640 1 -- 172.21.15.161:0/840846045 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4166968765 0 0) 0x7fd410152db0 con 0x7fd410155c30 2024-03-28T23:32:41.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.471+0000 7fd40f7fe640 1 -- 172.21.15.161:0/840846045 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4100c6f30 con 0x7fd4100c6750 2024-03-28T23:32:41.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.472+0000 7fd40f7fe640 1 --2- 172.21.15.161:0/840846045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4100c6750 0x7fd4100c88b0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fd4100689e0 tx=0x7fd40002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=432fb6a7d643ce6e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:41.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.472+0000 7fd40effd640 1 -- 172.21.15.161:0/840846045 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd400035020 con 0x7fd4100c6750 2024-03-28T23:32:41.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.472+0000 7fd40effd640 1 -- 172.21.15.161:0/840846045 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd400031830 con 0x7fd4100c6750 2024-03-28T23:32:41.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.472+0000 7fd40effd640 1 -- 172.21.15.161:0/840846045 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd400031c00 con 0x7fd4100c6750 2024-03-28T23:32:41.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.472+0000 7fd416948640 1 -- 172.21.15.161:0/840846045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4100c6750 msgr2=0x7fd4100c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:41.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.472+0000 7fd416948640 1 --2- 172.21.15.161:0/840846045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4100c6750 0x7fd4100c88b0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fd4100689e0 tx=0x7fd40002b8d0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.472+0000 7fd416948640 1 -- 172.21.15.161:0/840846045 shutdown_connections 2024-03-28T23:32:41.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.472+0000 7fd416948640 1 --2- 172.21.15.161:0/840846045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4100c6750 0x7fd4100c88b0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.472+0000 7fd416948640 1 --2- 172.21.15.161:0/840846045 >> v2:172.21.15.170:3300/0 conn(0x7fd4101529d0 0x7fd41006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.472+0000 7fd416948640 1 -- 172.21.15.161:0/840846045 >> 172.21.15.161:0/840846045 conn(0x7fd4100c7430 msgr2=0x7fd4100c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:41.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.472+0000 7fd416948640 1 -- 172.21.15.161:0/840846045 shutdown_connections 2024-03-28T23:32:41.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.473+0000 7fd416948640 1 -- 172.21.15.161:0/840846045 wait complete. 2024-03-28T23:32:41.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.474+0000 7fd416948640 1 Processor -- start 2024-03-28T23:32:41.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.474+0000 7fd416948640 1 -- start start 2024-03-28T23:32:41.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.474+0000 7fd416948640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4100c6750 0x7fd4100c2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:41.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.474+0000 7fd416948640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4101529d0 0x7fd4100c3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:41.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.474+0000 7fd416948640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd410155c30 0x7fd4100ab5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:41.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.474+0000 7fd416948640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd4100aa160 con 0x7fd4100c6750 2024-03-28T23:32:41.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.474+0000 7fd416948640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd410155710 con 0x7fd4101529d0 2024-03-28T23:32:41.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.474+0000 7fd416948640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd4100c5c00 con 0x7fd410155c30 2024-03-28T23:32:41.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.474+0000 7fd40ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4100c6750 0x7fd4100c2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:41.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.474+0000 7fd40ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4100c6750 0x7fd4100c2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39760/0 (socket says 172.21.15.161:39760) 2024-03-28T23:32:41.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.474+0000 7fd40ffff640 1 -- 172.21.15.161:0/2132535208 learned_addr learned my addr 172.21.15.161:0/2132535208 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:41.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.475+0000 7fd40f7fe640 1 --2- 172.21.15.161:0/2132535208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4101529d0 0x7fd4100c3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:41.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.475+0000 7fd40ffff640 1 -- 172.21.15.161:0/2132535208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd410155c30 msgr2=0x7fd4100ab5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:41.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.475+0000 7fd40ffff640 1 --2- 172.21.15.161:0/2132535208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd410155c30 0x7fd4100ab5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.475+0000 7fd40ffff640 1 -- 172.21.15.161:0/2132535208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4101529d0 msgr2=0x7fd4100c3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:41.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.475+0000 7fd40ffff640 1 --2- 172.21.15.161:0/2132535208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4101529d0 0x7fd4100c3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.475+0000 7fd40ffff640 1 -- 172.21.15.161:0/2132535208 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4100c3bc0 con 0x7fd4100c6750 2024-03-28T23:32:41.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.475+0000 7fd40ffff640 1 --2- 172.21.15.161:0/2132535208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4100c6750 0x7fd4100c2490 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fd40400b4d0 tx=0x7fd40400b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:41.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.475+0000 7fd40cff9640 1 -- 172.21.15.161:0/2132535208 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd404014070 con 0x7fd4100c6750 2024-03-28T23:32:41.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.475+0000 7fd40cff9640 1 -- 172.21.15.161:0/2132535208 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd404010870 con 0x7fd4100c6750 2024-03-28T23:32:41.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.475+0000 7fd416948640 1 -- 172.21.15.161:0/2132535208 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4100c3ff0 con 0x7fd4100c6750 2024-03-28T23:32:41.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.475+0000 7fd40cff9640 1 -- 172.21.15.161:0/2132535208 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd404010c60 con 0x7fd4100c6750 2024-03-28T23:32:41.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.475+0000 7fd416948640 1 -- 172.21.15.161:0/2132535208 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4100c45b0 con 0x7fd4100c6750 2024-03-28T23:32:41.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.476+0000 7fd416948640 1 -- 172.21.15.161:0/2132535208 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd41004f4b0 con 0x7fd4100c6750 2024-03-28T23:32:41.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.477+0000 7fd40cff9640 1 -- 172.21.15.161:0/2132535208 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd404017070 con 0x7fd4100c6750 2024-03-28T23:32:41.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.477+0000 7fd40cff9640 1 --2- 172.21.15.161:0/2132535208 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd3fc06b500 0x7fd3fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:41.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.477+0000 7fd40cff9640 1 -- 172.21.15.161:0/2132535208 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6101+0+0 (secure 0 0 0) 0x7fd40408a430 con 0x7fd4100c6750 2024-03-28T23:32:41.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.477+0000 7fd40f7fe640 1 --2- 172.21.15.161:0/2132535208 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd3fc06b500 0x7fd3fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:41.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.478+0000 7fd40f7fe640 1 --2- 172.21.15.161:0/2132535208 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd3fc06b500 0x7fd3fc06d9c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd400036000 tx=0x7fd400008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:41.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.481+0000 7fd40cff9640 1 -- 172.21.15.161:0/2132535208 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd4100c3bc0 con 0x7fd4100c6750 2024-03-28T23:32:41.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.660+0000 7fd416948640 1 -- 172.21.15.161:0/2132535208 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd4100c2270 con 0x7fd4100c6750 2024-03-28T23:32:41.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.660+0000 7fd40cff9640 1 -- 172.21.15.161:0/2132535208 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v73) ==== 100+0+11 (secure 0 0 0) 0x7fd404059290 con 0x7fd4100c6750 2024-03-28T23:32:41.662 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:32:41.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.663+0000 7fd416948640 1 -- 172.21.15.161:0/2132535208 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd3fc06b500 msgr2=0x7fd3fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:41.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.663+0000 7fd416948640 1 --2- 172.21.15.161:0/2132535208 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd3fc06b500 0x7fd3fc06d9c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd400036000 tx=0x7fd400008030 comp rx=0 tx=0).stop 2024-03-28T23:32:41.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.663+0000 7fd416948640 1 -- 172.21.15.161:0/2132535208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4100c6750 msgr2=0x7fd4100c2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:41.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.663+0000 7fd416948640 1 --2- 172.21.15.161:0/2132535208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4100c6750 0x7fd4100c2490 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fd40400b4d0 tx=0x7fd40400b990 comp rx=0 tx=0).stop 2024-03-28T23:32:41.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.663+0000 7fd416948640 1 -- 172.21.15.161:0/2132535208 shutdown_connections 2024-03-28T23:32:41.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.663+0000 7fd416948640 1 --2- 172.21.15.161:0/2132535208 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd3fc06b500 0x7fd3fc06d9c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.663+0000 7fd416948640 1 --2- 172.21.15.161:0/2132535208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd410155c30 0x7fd4100ab5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.663+0000 7fd416948640 1 --2- 172.21.15.161:0/2132535208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4101529d0 0x7fd4100c3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.663+0000 7fd416948640 1 --2- 172.21.15.161:0/2132535208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4100c6750 0x7fd4100c2490 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.663+0000 7fd416948640 1 -- 172.21.15.161:0/2132535208 >> 172.21.15.161:0/2132535208 conn(0x7fd4100c7430 msgr2=0x7fd4101549f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:41.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.663+0000 7fd416948640 1 -- 172.21.15.161:0/2132535208 shutdown_connections 2024-03-28T23:32:41.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.663+0000 7fd416948640 1 -- 172.21.15.161:0/2132535208 wait complete. 2024-03-28T23:32:41.675 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T23:32:41.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.779+0000 7f824628d640 1 Processor -- start 2024-03-28T23:32:41.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.780+0000 7f824628d640 1 -- start start 2024-03-28T23:32:41.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.780+0000 7f824628d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82400caf60 0x7f82401592e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:41.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.780+0000 7f824628d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f82400c8970 0x7f82400c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:41.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.780+0000 7f824628d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82400cb4e0 con 0x7f824014f260 2024-03-28T23:32:41.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.780+0000 7f824628d640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f8240158e50 con 0x7f82400c8970 2024-03-28T23:32:41.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.780+0000 7f824628d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f82400cb7c0 con 0x7f82400caf60 2024-03-28T23:32:41.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.780+0000 7f823ffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f824014f260 0x7f8240155c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59286/0 (socket says 172.21.15.161:59286) 2024-03-28T23:32:41.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.780+0000 7f823ffff640 1 -- 172.21.15.161:0/3899293745 learned_addr learned my addr 172.21.15.161:0/3899293745 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:41.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.780+0000 7f824482a640 1 --2- 172.21.15.161:0/3899293745 >> v2:172.21.15.170:3300/0 conn(0x7f82400c8970 0x7f82400c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:41.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.780+0000 7f823f7fe640 1 --2- 172.21.15.161:0/3899293745 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82400caf60 0x7f82401592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:41.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.781+0000 7f823effd640 1 -- 172.21.15.161:0/3899293745 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 738554325 0 0) 0x7f82400cb4e0 con 0x7f824014f260 2024-03-28T23:32:41.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.781+0000 7f823effd640 1 -- 172.21.15.161:0/3899293745 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8228003520 con 0x7f824014f260 2024-03-28T23:32:41.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.781+0000 7f824482a640 1 -- 172.21.15.161:0/3899293745 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82400caf60 msgr2=0x7f82401592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:41.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.781+0000 7f824482a640 1 --2- 172.21.15.161:0/3899293745 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82400caf60 0x7f82401592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.781+0000 7f824482a640 1 -- 172.21.15.161:0/3899293745 >> v1:172.21.15.161:6789/0 conn(0x7f824014f260 legacy=0x7f8240155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:41.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.781+0000 7f824482a640 1 -- 172.21.15.161:0/3899293745 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8240158800 con 0x7f82400c8970 2024-03-28T23:32:41.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.782+0000 7f824482a640 1 --2- 172.21.15.161:0/3899293745 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82400c8970 0x7f82400c53f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8238009d90 tx=0x7f823802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0c79139b904303c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:41.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.782+0000 7f823effd640 1 -- 172.21.15.161:0/3899293745 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8238039070 con 0x7f82400c8970 2024-03-28T23:32:41.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.782+0000 7f823effd640 1 -- 172.21.15.161:0/3899293745 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f823802cbc0 con 0x7f82400c8970 2024-03-28T23:32:41.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.782+0000 7f823effd640 1 -- 172.21.15.161:0/3899293745 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8238034440 con 0x7f82400c8970 2024-03-28T23:32:41.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.782+0000 7f824628d640 1 -- 172.21.15.161:0/3899293745 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82400c8970 msgr2=0x7f82400c53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:41.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.782+0000 7f824628d640 1 --2- 172.21.15.161:0/3899293745 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82400c8970 0x7f82400c53f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8238009d90 tx=0x7f823802be10 comp rx=0 tx=0).stop 2024-03-28T23:32:41.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.783+0000 7f824628d640 1 -- 172.21.15.161:0/3899293745 shutdown_connections 2024-03-28T23:32:41.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.783+0000 7f824628d640 1 --2- 172.21.15.161:0/3899293745 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82400c8970 0x7f82400c53f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.783+0000 7f824628d640 1 --2- 172.21.15.161:0/3899293745 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82400caf60 0x7f82401592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.783+0000 7f824628d640 1 -- 172.21.15.161:0/3899293745 >> 172.21.15.161:0/3899293745 conn(0x7f8240072ca0 msgr2=0x7f82400730b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:41.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.783+0000 7f824628d640 1 -- 172.21.15.161:0/3899293745 shutdown_connections 2024-03-28T23:32:41.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.783+0000 7f824628d640 1 -- 172.21.15.161:0/3899293745 wait complete. 2024-03-28T23:32:41.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.784+0000 7f824628d640 1 Processor -- start 2024-03-28T23:32:41.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.784+0000 7f824628d640 1 -- start start 2024-03-28T23:32:41.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.784+0000 7f824628d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82400c8970 0x7f82400af410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:41.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.784+0000 7f824628d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82400caf60 0x7f82400c0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:41.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.785+0000 7f824628d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f824014f260 0x7f82400c1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:41.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.785+0000 7f824628d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f82400c8e00 con 0x7f82400c8970 2024-03-28T23:32:41.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.785+0000 7f824628d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f824014f640 con 0x7f82400caf60 2024-03-28T23:32:41.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.785+0000 7f824628d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f82400c8f80 con 0x7f824014f260 2024-03-28T23:32:41.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.785+0000 7f823ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82400c8970 0x7f82400af410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:41.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.785+0000 7f823ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82400c8970 0x7f82400af410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39772/0 (socket says 172.21.15.161:39772) 2024-03-28T23:32:41.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.785+0000 7f823ffff640 1 -- 172.21.15.161:0/4271333337 learned_addr learned my addr 172.21.15.161:0/4271333337 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:41.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.785+0000 7f823f7fe640 1 --2- 172.21.15.161:0/4271333337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82400caf60 0x7f82400c0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:41.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.785+0000 7f824482a640 1 --2- 172.21.15.161:0/4271333337 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f824014f260 0x7f82400c1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:41.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.785+0000 7f823ffff640 1 -- 172.21.15.161:0/4271333337 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f824014f260 msgr2=0x7f82400c1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:41.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.785+0000 7f823ffff640 1 --2- 172.21.15.161:0/4271333337 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f824014f260 0x7f82400c1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.785+0000 7f823ffff640 1 -- 172.21.15.161:0/4271333337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82400caf60 msgr2=0x7f82400c0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:41.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.785+0000 7f823ffff640 1 --2- 172.21.15.161:0/4271333337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82400caf60 0x7f82400c0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.785+0000 7f823ffff640 1 -- 172.21.15.161:0/4271333337 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82400ae4e0 con 0x7f82400c8970 2024-03-28T23:32:41.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.785+0000 7f824482a640 1 --2- 172.21.15.161:0/4271333337 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f824014f260 0x7f82400c1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:32:41.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.785+0000 7f823ffff640 1 --2- 172.21.15.161:0/4271333337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82400c8970 0x7f82400af410 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f823000c390 tx=0x7f823000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:41.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.785+0000 7f823cff9640 1 -- 172.21.15.161:0/4271333337 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8230017070 con 0x7f82400c8970 2024-03-28T23:32:41.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.785+0000 7f823cff9640 1 -- 172.21.15.161:0/4271333337 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8230005ce0 con 0x7f82400c8970 2024-03-28T23:32:41.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.785+0000 7f824628d640 1 -- 172.21.15.161:0/4271333337 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82400ae7c0 con 0x7f82400c8970 2024-03-28T23:32:41.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.785+0000 7f823cff9640 1 -- 172.21.15.161:0/4271333337 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82300072f0 con 0x7f82400c8970 2024-03-28T23:32:41.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.785+0000 7f824628d640 1 -- 172.21.15.161:0/4271333337 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82400af950 con 0x7f82400c8970 2024-03-28T23:32:41.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.787+0000 7f824628d640 1 -- 172.21.15.161:0/4271333337 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f824004f550 con 0x7f82400c8970 2024-03-28T23:32:41.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.787+0000 7f823cff9640 1 -- 172.21.15.161:0/4271333337 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f823001a070 con 0x7f82400c8970 2024-03-28T23:32:41.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.787+0000 7f823cff9640 1 --2- 172.21.15.161:0/4271333337 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f822c06b500 0x7f822c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:41.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.787+0000 7f823cff9640 1 -- 172.21.15.161:0/4271333337 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6101+0+0 (secure 0 0 0) 0x7f82300897c0 con 0x7f82400c8970 2024-03-28T23:32:41.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.787+0000 7f823f7fe640 1 --2- 172.21.15.161:0/4271333337 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f822c06b500 0x7f822c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:41.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.788+0000 7f823f7fe640 1 --2- 172.21.15.161:0/4271333337 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f822c06b500 0x7f822c06d9c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f82401eb7e0 tx=0x7f8234009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:41.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.792+0000 7f823cff9640 1 -- 172.21.15.161:0/4271333337 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8230057d50 con 0x7f82400c8970 2024-03-28T23:32:41.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.969+0000 7f824628d640 1 -- 172.21.15.161:0/4271333337 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f824006cfe0 con 0x7f82400c8970 2024-03-28T23:32:41.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.969+0000 7f823cff9640 1 -- 172.21.15.161:0/4271333337 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v73) ==== 84+0+209 (secure 0 0 0) 0x7f823005bdf0 con 0x7f82400c8970 2024-03-28T23:32:41.970 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:41.971 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2024-03-28T23:32:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:32:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: "poolnum": 1, 2024-03-28T23:32:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: "poolname": ".mgr" 2024-03-28T23:32:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:32:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:32:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: "poolnum": 11, 2024-03-28T23:32:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: "poolname": "cephfs_metadata" 2024-03-28T23:32:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:32:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:32:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: "poolnum": 12, 2024-03-28T23:32:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: "poolname": "cephfs_data" 2024-03-28T23:32:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:32:41.972 INFO:teuthology.orchestra.run.smithi161.stdout:] 2024-03-28T23:32:41.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.971+0000 7f824628d640 1 -- 172.21.15.161:0/4271333337 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f822c06b500 msgr2=0x7f822c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:41.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.971+0000 7f824628d640 1 --2- 172.21.15.161:0/4271333337 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f822c06b500 0x7f822c06d9c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f82401eb7e0 tx=0x7f8234009190 comp rx=0 tx=0).stop 2024-03-28T23:32:41.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.971+0000 7f824628d640 1 -- 172.21.15.161:0/4271333337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82400c8970 msgr2=0x7f82400af410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:41.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.971+0000 7f824628d640 1 --2- 172.21.15.161:0/4271333337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82400c8970 0x7f82400af410 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f823000c390 tx=0x7f823000c850 comp rx=0 tx=0).stop 2024-03-28T23:32:41.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.971+0000 7f824628d640 1 -- 172.21.15.161:0/4271333337 shutdown_connections 2024-03-28T23:32:41.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.971+0000 7f824628d640 1 --2- 172.21.15.161:0/4271333337 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f822c06b500 0x7f822c06d9c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.971+0000 7f824628d640 1 --2- 172.21.15.161:0/4271333337 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f824014f260 0x7f82400c1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.971+0000 7f824628d640 1 --2- 172.21.15.161:0/4271333337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82400caf60 0x7f82400c0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.971+0000 7f824628d640 1 --2- 172.21.15.161:0/4271333337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82400c8970 0x7f82400af410 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:41.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.971+0000 7f824628d640 1 -- 172.21.15.161:0/4271333337 >> 172.21.15.161:0/4271333337 conn(0x7f8240072ca0 msgr2=0x7f82400c52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:41.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.971+0000 7f824628d640 1 -- 172.21.15.161:0/4271333337 shutdown_connections 2024-03-28T23:32:41.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:41.971+0000 7f824628d640 1 -- 172.21.15.161:0/4271333337 wait complete. 2024-03-28T23:32:41.986 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs2' 2024-03-28T23:32:41.986 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_metadata --pg_num_min 64 2024-03-28T23:32:42.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.093+0000 7ff49aaa4640 1 Processor -- start 2024-03-28T23:32:42.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.093+0000 7ff49aaa4640 1 -- start start 2024-03-28T23:32:42.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.094+0000 7ff49aaa4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4941527f0 0x7ff4940c5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:42.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.094+0000 7ff49aaa4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff4940c8980 0x7ff494155180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:42.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.094+0000 7ff49aaa4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff4940be110 0x7ff4941556c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:42.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.094+0000 7ff49aaa4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff4940cafb0 con 0x7ff4940c8980 2024-03-28T23:32:42.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.094+0000 7ff49aaa4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7ff494152bd0 con 0x7ff4940be110 2024-03-28T23:32:42.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.094+0000 7ff49aaa4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff494159220 con 0x7ff4941527f0 2024-03-28T23:32:42.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.094+0000 7ff493fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff4940c8980 0x7ff494155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:42.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.094+0000 7ff493fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff4940c8980 0x7ff494155180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39774/0 (socket says 172.21.15.161:39774) 2024-03-28T23:32:42.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.094+0000 7ff493fff640 1 -- 172.21.15.161:0/1169532180 learned_addr learned my addr 172.21.15.161:0/1169532180 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:42.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.094+0000 7ff499041640 1 --2- 172.21.15.161:0/1169532180 >> v2:172.21.15.170:3300/0 conn(0x7ff4940be110 0x7ff4941556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:42.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.094+0000 7ff498840640 1 --2- 172.21.15.161:0/1169532180 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4941527f0 0x7ff4940c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:42.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.094+0000 7ff493fff640 1 -- 172.21.15.161:0/1169532180 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4941527f0 msgr2=0x7ff4940c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:42.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.094+0000 7ff493fff640 1 --2- 172.21.15.161:0/1169532180 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4941527f0 0x7ff4940c5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:42.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.094+0000 7ff493fff640 1 -- 172.21.15.161:0/1169532180 >> v2:172.21.15.170:3300/0 conn(0x7ff4940be110 msgr2=0x7ff4941556c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:42.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.094+0000 7ff493fff640 1 --2- 172.21.15.161:0/1169532180 >> v2:172.21.15.170:3300/0 conn(0x7ff4940be110 0x7ff4941556c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:42.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.094+0000 7ff493fff640 1 -- 172.21.15.161:0/1169532180 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff494159500 con 0x7ff4940c8980 2024-03-28T23:32:42.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.095+0000 7ff493fff640 1 --2- 172.21.15.161:0/1169532180 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4940c8980 0x7ff494155180 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7ff4940c6a60 tx=0x7ff48802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fa0ac7cbcb6e9c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:42.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.095+0000 7ff4937fe640 1 -- 172.21.15.161:0/1169532180 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff488038070 con 0x7ff4940c8980 2024-03-28T23:32:42.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.095+0000 7ff4937fe640 1 -- 172.21.15.161:0/1169532180 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff48802f050 con 0x7ff4940c8980 2024-03-28T23:32:42.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.095+0000 7ff4937fe640 1 -- 172.21.15.161:0/1169532180 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff488031970 con 0x7ff4940c8980 2024-03-28T23:32:42.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.096+0000 7ff49aaa4640 1 -- 172.21.15.161:0/1169532180 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4940c8980 msgr2=0x7ff494155180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:42.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.096+0000 7ff49aaa4640 1 --2- 172.21.15.161:0/1169532180 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4940c8980 0x7ff494155180 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7ff4940c6a60 tx=0x7ff48802b7f0 comp rx=0 tx=0).stop 2024-03-28T23:32:42.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.096+0000 7ff49aaa4640 1 -- 172.21.15.161:0/1169532180 shutdown_connections 2024-03-28T23:32:42.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.096+0000 7ff49aaa4640 1 --2- 172.21.15.161:0/1169532180 >> v2:172.21.15.170:3300/0 conn(0x7ff4940be110 0x7ff4941556c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:42.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.096+0000 7ff49aaa4640 1 --2- 172.21.15.161:0/1169532180 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4940c8980 0x7ff494155180 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:42.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.096+0000 7ff49aaa4640 1 --2- 172.21.15.161:0/1169532180 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4941527f0 0x7ff4940c5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:42.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.096+0000 7ff49aaa4640 1 -- 172.21.15.161:0/1169532180 >> 172.21.15.161:0/1169532180 conn(0x7ff494072ca0 msgr2=0x7ff4940730b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:42.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.096+0000 7ff49aaa4640 1 -- 172.21.15.161:0/1169532180 shutdown_connections 2024-03-28T23:32:42.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.096+0000 7ff49aaa4640 1 -- 172.21.15.161:0/1169532180 wait complete. 2024-03-28T23:32:42.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.098+0000 7ff49aaa4640 1 Processor -- start 2024-03-28T23:32:42.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.098+0000 7ff49aaa4640 1 -- start start 2024-03-28T23:32:42.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.099+0000 7ff49aaa4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4940be110 0x7ff4941577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:42.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.099+0000 7ff49aaa4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4940c8980 0x7ff4941eb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:42.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.099+0000 7ff49aaa4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4941527f0 0x7ff4941eb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:42.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.099+0000 7ff49aaa4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff49406d440 con 0x7ff4940c8980 2024-03-28T23:32:42.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.099+0000 7ff49aaa4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff4940cb4e0 con 0x7ff4940be110 2024-03-28T23:32:42.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.099+0000 7ff49aaa4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff49406d5c0 con 0x7ff4941527f0 2024-03-28T23:32:42.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.099+0000 7ff493fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4940c8980 0x7ff4941eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:42.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.099+0000 7ff493fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4940c8980 0x7ff4941eb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39786/0 (socket says 172.21.15.161:39786) 2024-03-28T23:32:42.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.099+0000 7ff493fff640 1 -- 172.21.15.161:0/2033691274 learned_addr learned my addr 172.21.15.161:0/2033691274 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:42.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.099+0000 7ff499041640 1 --2- 172.21.15.161:0/2033691274 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4941527f0 0x7ff4941eb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:42.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.099+0000 7ff498840640 1 --2- 172.21.15.161:0/2033691274 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4940be110 0x7ff4941577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:42.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.099+0000 7ff493fff640 1 -- 172.21.15.161:0/2033691274 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4941527f0 msgr2=0x7ff4941eb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:42.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.099+0000 7ff493fff640 1 --2- 172.21.15.161:0/2033691274 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4941527f0 0x7ff4941eb750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:42.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.099+0000 7ff493fff640 1 -- 172.21.15.161:0/2033691274 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4940be110 msgr2=0x7ff4941577e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:42.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.100+0000 7ff493fff640 1 --2- 172.21.15.161:0/2033691274 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4940be110 0x7ff4941577e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:42.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.100+0000 7ff493fff640 1 -- 172.21.15.161:0/2033691274 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4941ead30 con 0x7ff4940c8980 2024-03-28T23:32:42.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.100+0000 7ff493fff640 1 --2- 172.21.15.161:0/2033691274 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4940c8980 0x7ff4941eb210 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7ff48802b830 tx=0x7ff4880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:42.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.100+0000 7ff4917fa640 1 -- 172.21.15.161:0/2033691274 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff488042070 con 0x7ff4940c8980 2024-03-28T23:32:42.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.100+0000 7ff4917fa640 1 -- 172.21.15.161:0/2033691274 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff48803d680 con 0x7ff4940c8980 2024-03-28T23:32:42.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.100+0000 7ff49aaa4640 1 -- 172.21.15.161:0/2033691274 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4940c41f0 con 0x7ff4940c8980 2024-03-28T23:32:42.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.100+0000 7ff4917fa640 1 -- 172.21.15.161:0/2033691274 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff488038040 con 0x7ff4940c8980 2024-03-28T23:32:42.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.100+0000 7ff49aaa4640 1 -- 172.21.15.161:0/2033691274 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff494153c80 con 0x7ff4940c8980 2024-03-28T23:32:42.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.101+0000 7ff49aaa4640 1 -- 172.21.15.161:0/2033691274 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff49414ce30 con 0x7ff4940c8980 2024-03-28T23:32:42.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.101+0000 7ff4917fa640 1 -- 172.21.15.161:0/2033691274 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff488045070 con 0x7ff4940c8980 2024-03-28T23:32:42.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.102+0000 7ff4917fa640 1 --2- 172.21.15.161:0/2033691274 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff48006b4d0 0x7ff48006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:42.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.102+0000 7ff498840640 1 --2- 172.21.15.161:0/2033691274 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff48006b4d0 0x7ff48006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:42.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.102+0000 7ff4917fa640 1 -- 172.21.15.161:0/2033691274 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6101+0+0 (secure 0 0 0) 0x7ff4880abde0 con 0x7ff4940c8980 2024-03-28T23:32:42.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.102+0000 7ff498840640 1 --2- 172.21.15.161:0/2033691274 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff48006b4d0 0x7ff48006d990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff4940bded0 tx=0x7ff48400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:42.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.106+0000 7ff4917fa640 1 -- 172.21.15.161:0/2033691274 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff49414ce30 con 0x7ff4940c8980 2024-03-28T23:32:42.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:42.359+0000 7ff49aaa4640 1 -- 172.21.15.161:0/2033691274 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64} v 0) -- 0x7ff494068f90 con 0x7ff4940c8980 2024-03-28T23:32:43.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.097+0000 7ff4917fa640 1 -- 172.21.15.161:0/2033691274 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64}]=0 pool 'cephfs2_metadata' created v74) ==== 140+0+0 (secure 0 0 0) 0x7ff48807a510 con 0x7ff4940c8980 2024-03-28T23:32:43.097 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs2_metadata' created 2024-03-28T23:32:43.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.099+0000 7ff49aaa4640 1 -- 172.21.15.161:0/2033691274 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff48006b4d0 msgr2=0x7ff48006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:43.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.099+0000 7ff49aaa4640 1 --2- 172.21.15.161:0/2033691274 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff48006b4d0 0x7ff48006d990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff4940bded0 tx=0x7ff48400b040 comp rx=0 tx=0).stop 2024-03-28T23:32:43.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.099+0000 7ff49aaa4640 1 -- 172.21.15.161:0/2033691274 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4940c8980 msgr2=0x7ff4941eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:43.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.099+0000 7ff49aaa4640 1 --2- 172.21.15.161:0/2033691274 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4940c8980 0x7ff4941eb210 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7ff48802b830 tx=0x7ff4880046c0 comp rx=0 tx=0).stop 2024-03-28T23:32:43.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.100+0000 7ff49aaa4640 1 -- 172.21.15.161:0/2033691274 shutdown_connections 2024-03-28T23:32:43.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.100+0000 7ff49aaa4640 1 --2- 172.21.15.161:0/2033691274 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff48006b4d0 0x7ff48006d990 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:43.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.100+0000 7ff49aaa4640 1 --2- 172.21.15.161:0/2033691274 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4940c8980 0x7ff4941eb210 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:43.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.100+0000 7ff49aaa4640 1 --2- 172.21.15.161:0/2033691274 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4941527f0 0x7ff4941eb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:43.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.100+0000 7ff49aaa4640 1 --2- 172.21.15.161:0/2033691274 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4940be110 0x7ff4941577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:43.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.100+0000 7ff49aaa4640 1 -- 172.21.15.161:0/2033691274 >> 172.21.15.161:0/2033691274 conn(0x7ff494072ca0 msgr2=0x7ff49414b500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:43.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.100+0000 7ff49aaa4640 1 -- 172.21.15.161:0/2033691274 shutdown_connections 2024-03-28T23:32:43.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.100+0000 7ff49aaa4640 1 -- 172.21.15.161:0/2033691274 wait complete. 2024-03-28T23:32:43.116 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T23:32:43.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.224+0000 7f700114b640 1 Processor -- start 2024-03-28T23:32:43.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.224+0000 7f700114b640 1 -- start start 2024-03-28T23:32:43.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.225+0000 7f700114b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6ffc0c0330 0x7f6ffc0c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:43.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.225+0000 7f700114b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ffc0c6f60 con 0x7f6ffc152bc0 2024-03-28T23:32:43.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.225+0000 7f700114b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ffc0c6120 con 0x7f6ffc1589b0 2024-03-28T23:32:43.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.225+0000 7f700114b640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6ffc0c6400 con 0x7f6ffc0c0330 2024-03-28T23:32:43.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.226+0000 7f6ffad9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6ffc152bc0 0x7f6ffc0c8a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59294/0 (socket says 172.21.15.161:59294) 2024-03-28T23:32:43.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.226+0000 7f6ffad9d640 1 -- 172.21.15.161:0/262557886 learned_addr learned my addr 172.21.15.161:0/262557886 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:43.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.226+0000 7f6ffa59c640 1 --2- 172.21.15.161:0/262557886 >> v2:172.21.15.161:3300/0 conn(0x7f6ffc0c0330 0x7f6ffc0c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:43.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.226+0000 7f6ff9d9b640 1 -- 172.21.15.161:0/262557886 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 738776559 0 0) 0x7f6ffc0c6f60 con 0x7f6ffc152bc0 2024-03-28T23:32:43.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.226+0000 7f6ff9d9b640 1 -- 172.21.15.161:0/262557886 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fe4003520 con 0x7f6ffc152bc0 2024-03-28T23:32:43.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.226+0000 7f6ffa59c640 1 -- 172.21.15.161:0/262557886 >> v1:172.21.15.170:6789/0 conn(0x7f6ffc1589b0 legacy=0x7f6ffc158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:43.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.226+0000 7f6ffa59c640 1 -- 172.21.15.161:0/262557886 >> v1:172.21.15.161:6789/0 conn(0x7f6ffc152bc0 legacy=0x7f6ffc0c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:43.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.226+0000 7f6ffa59c640 1 -- 172.21.15.161:0/262557886 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ffc0c96d0 con 0x7f6ffc0c0330 2024-03-28T23:32:43.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.227+0000 7f6ffa59c640 1 --2- 172.21.15.161:0/262557886 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ffc0c0330 0x7f6ffc0c9190 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f6ff00097a0 tx=0x7f6ff002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ef985a863bc7a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:43.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.227+0000 7f6ff9d9b640 1 -- 172.21.15.161:0/262557886 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0035020 con 0x7f6ffc0c0330 2024-03-28T23:32:43.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.227+0000 7f6ff9d9b640 1 -- 172.21.15.161:0/262557886 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ff0031830 con 0x7f6ffc0c0330 2024-03-28T23:32:43.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.227+0000 7f6ff9d9b640 1 -- 172.21.15.161:0/262557886 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0031be0 con 0x7f6ffc0c0330 2024-03-28T23:32:43.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.227+0000 7f700114b640 1 -- 172.21.15.161:0/262557886 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ffc0c0330 msgr2=0x7f6ffc0c9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:43.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.227+0000 7f700114b640 1 --2- 172.21.15.161:0/262557886 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ffc0c0330 0x7f6ffc0c9190 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f6ff00097a0 tx=0x7f6ff002b8d0 comp rx=0 tx=0).stop 2024-03-28T23:32:43.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.227+0000 7f700114b640 1 -- 172.21.15.161:0/262557886 shutdown_connections 2024-03-28T23:32:43.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.227+0000 7f700114b640 1 --2- 172.21.15.161:0/262557886 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ffc0c0330 0x7f6ffc0c9190 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:43.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.227+0000 7f700114b640 1 -- 172.21.15.161:0/262557886 >> 172.21.15.161:0/262557886 conn(0x7f6ffc0caa80 msgr2=0x7f6ffc0cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:43.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.227+0000 7f700114b640 1 -- 172.21.15.161:0/262557886 shutdown_connections 2024-03-28T23:32:43.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.227+0000 7f700114b640 1 -- 172.21.15.161:0/262557886 wait complete. 2024-03-28T23:32:43.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.229+0000 7f700114b640 1 Processor -- start 2024-03-28T23:32:43.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.229+0000 7f700114b640 1 -- start start 2024-03-28T23:32:43.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.231+0000 7f700114b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ffc0c0330 0x7f6ffc1eb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:43.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.231+0000 7f700114b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ffc152bc0 0x7f6ffc151960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:43.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.231+0000 7f700114b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ffc1589b0 0x7f6ffc1503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:43.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.231+0000 7f700114b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6ffc152fa0 con 0x7f6ffc0c0330 2024-03-28T23:32:43.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.231+0000 7f700114b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6ffc0c6bf0 con 0x7f6ffc152bc0 2024-03-28T23:32:43.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.231+0000 7f700114b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6ffc06c040 con 0x7f6ffc1589b0 2024-03-28T23:32:43.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.231+0000 7f6ffad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ffc0c0330 0x7f6ffc1eb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:43.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.231+0000 7f6ffad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ffc0c0330 0x7f6ffc1eb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39792/0 (socket says 172.21.15.161:39792) 2024-03-28T23:32:43.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.231+0000 7f6ffad9d640 1 -- 172.21.15.161:0/2118916854 learned_addr learned my addr 172.21.15.161:0/2118916854 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:43.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.231+0000 7f6ffa59c640 1 --2- 172.21.15.161:0/2118916854 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ffc152bc0 0x7f6ffc151960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:43.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.231+0000 7f6ffb59e640 1 --2- 172.21.15.161:0/2118916854 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ffc1589b0 0x7f6ffc1503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:43.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.231+0000 7f6ffad9d640 1 -- 172.21.15.161:0/2118916854 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ffc1589b0 msgr2=0x7f6ffc1503e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:43.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.231+0000 7f6ffad9d640 1 --2- 172.21.15.161:0/2118916854 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ffc1589b0 0x7f6ffc1503e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:43.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.231+0000 7f6ffad9d640 1 -- 172.21.15.161:0/2118916854 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ffc152bc0 msgr2=0x7f6ffc151960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:43.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.231+0000 7f6ffad9d640 1 --2- 172.21.15.161:0/2118916854 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ffc152bc0 0x7f6ffc151960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:43.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.231+0000 7f6ffad9d640 1 -- 172.21.15.161:0/2118916854 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ffc0c4e70 con 0x7f6ffc0c0330 2024-03-28T23:32:43.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.231+0000 7f6ffa59c640 1 --2- 172.21.15.161:0/2118916854 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ffc152bc0 0x7f6ffc151960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:43.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.231+0000 7f6ffad9d640 1 --2- 172.21.15.161:0/2118916854 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ffc0c0330 0x7f6ffc1eb6f0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f6ff400c460 tx=0x7f6ff400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:43.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.231+0000 7f6fe37fe640 1 -- 172.21.15.161:0/2118916854 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4017070 con 0x7f6ffc0c0330 2024-03-28T23:32:43.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.231+0000 7f6fe37fe640 1 -- 172.21.15.161:0/2118916854 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ff4005ce0 con 0x7f6ffc0c0330 2024-03-28T23:32:43.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.231+0000 7f700114b640 1 -- 172.21.15.161:0/2118916854 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ffc06c7b0 con 0x7f6ffc0c0330 2024-03-28T23:32:43.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.231+0000 7f6fe37fe640 1 -- 172.21.15.161:0/2118916854 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4007290 con 0x7f6ffc0c0330 2024-03-28T23:32:43.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.231+0000 7f700114b640 1 -- 172.21.15.161:0/2118916854 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ffc06ceb0 con 0x7f6ffc0c0330 2024-03-28T23:32:43.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.233+0000 7f700114b640 1 -- 172.21.15.161:0/2118916854 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ffc04f4b0 con 0x7f6ffc0c0330 2024-03-28T23:32:43.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.233+0000 7f6fe37fe640 1 -- 172.21.15.161:0/2118916854 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6ff401a070 con 0x7f6ffc0c0330 2024-03-28T23:32:43.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.233+0000 7f6fe37fe640 1 --2- 172.21.15.161:0/2118916854 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6fe806b4d0 0x7f6fe806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:43.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.234+0000 7f6fe37fe640 1 -- 172.21.15.161:0/2118916854 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6472+0+0 (secure 0 0 0) 0x7f6ff4089820 con 0x7f6ffc0c0330 2024-03-28T23:32:43.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.234+0000 7f6ffa59c640 1 --2- 172.21.15.161:0/2118916854 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6fe806b4d0 0x7f6fe806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:43.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.234+0000 7f6ffa59c640 1 --2- 172.21.15.161:0/2118916854 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6fe806b4d0 0x7f6fe806d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0036000 tx=0x7f6ff0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:43.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.242+0000 7f6fe37fe640 1 -- 172.21.15.161:0/2118916854 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6ff4058430 con 0x7f6ffc0c0330 2024-03-28T23:32:43.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:43.417+0000 7f700114b640 1 -- 172.21.15.161:0/2118916854 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f6ffc067a80 con 0x7f6ffc0c0330 2024-03-28T23:32:44.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.101+0000 7f6fe37fe640 1 -- 172.21.15.161:0/2118916854 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs2_data' created v75) ==== 172+0+0 (secure 0 0 0) 0x7f6ff4057d30 con 0x7f6ffc0c0330 2024-03-28T23:32:44.102 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs2_data' created 2024-03-28T23:32:44.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.104+0000 7f700114b640 1 -- 172.21.15.161:0/2118916854 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6fe806b4d0 msgr2=0x7f6fe806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:44.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.104+0000 7f700114b640 1 --2- 172.21.15.161:0/2118916854 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6fe806b4d0 0x7f6fe806d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0036000 tx=0x7f6ff0008030 comp rx=0 tx=0).stop 2024-03-28T23:32:44.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.104+0000 7f700114b640 1 -- 172.21.15.161:0/2118916854 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ffc0c0330 msgr2=0x7f6ffc1eb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:44.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.104+0000 7f700114b640 1 --2- 172.21.15.161:0/2118916854 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ffc0c0330 0x7f6ffc1eb6f0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f6ff400c460 tx=0x7f6ff400c920 comp rx=0 tx=0).stop 2024-03-28T23:32:44.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.104+0000 7f700114b640 1 -- 172.21.15.161:0/2118916854 shutdown_connections 2024-03-28T23:32:44.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.104+0000 7f700114b640 1 --2- 172.21.15.161:0/2118916854 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6fe806b4d0 0x7f6fe806d990 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:44.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.104+0000 7f700114b640 1 --2- 172.21.15.161:0/2118916854 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ffc1589b0 0x7f6ffc1503e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:44.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.104+0000 7f700114b640 1 --2- 172.21.15.161:0/2118916854 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ffc152bc0 0x7f6ffc151960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:44.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.104+0000 7f700114b640 1 --2- 172.21.15.161:0/2118916854 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ffc0c0330 0x7f6ffc1eb6f0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:44.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.104+0000 7f700114b640 1 -- 172.21.15.161:0/2118916854 >> 172.21.15.161:0/2118916854 conn(0x7f6ffc0caa80 msgr2=0x7f6ffc14dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:44.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.104+0000 7f700114b640 1 -- 172.21.15.161:0/2118916854 shutdown_connections 2024-03-28T23:32:44.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.104+0000 7f700114b640 1 -- 172.21.15.161:0/2118916854 wait complete. 2024-03-28T23:32:44.131 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs2 cephfs2_metadata cephfs2_data 2024-03-28T23:32:44.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.235+0000 7f598e7e5640 1 Processor -- start 2024-03-28T23:32:44.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.235+0000 7f598e7e5640 1 -- start start 2024-03-28T23:32:44.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.237+0000 7f598e7e5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f59880c0330 0x7f59880c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:44.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.237+0000 7f598e7e5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59880c6f60 con 0x7f59881589b0 2024-03-28T23:32:44.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.237+0000 7f598e7e5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59880c6120 con 0x7f5988152bc0 2024-03-28T23:32:44.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.237+0000 7f598e7e5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f59880c6400 con 0x7f59880c0330 2024-03-28T23:32:44.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.237+0000 7f598cd82640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f59881589b0 0x7f5988158e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59306/0 (socket says 172.21.15.161:59306) 2024-03-28T23:32:44.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.237+0000 7f598cd82640 1 -- 172.21.15.161:0/357029016 learned_addr learned my addr 172.21.15.161:0/357029016 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:44.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.237+0000 7f59877fe640 1 --2- 172.21.15.161:0/357029016 >> v2:172.21.15.170:3300/0 conn(0x7f59880c0330 0x7f59880c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:44.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.237+0000 7f5986ffd640 1 -- 172.21.15.161:0/357029016 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1345608123 0 0) 0x7f59880c6f60 con 0x7f59881589b0 2024-03-28T23:32:44.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.237+0000 7f5986ffd640 1 -- 172.21.15.161:0/357029016 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5970003520 con 0x7f59881589b0 2024-03-28T23:32:44.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.237+0000 7f5986ffd640 1 -- 172.21.15.161:0/357029016 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 316301104 0 0) 0x7f5970003520 con 0x7f59881589b0 2024-03-28T23:32:44.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.237+0000 7f5986ffd640 1 -- 172.21.15.161:0/357029016 >> v2:172.21.15.170:3300/0 conn(0x7f59880c0330 msgr2=0x7f59880c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:44.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.237+0000 7f5986ffd640 1 --2- 172.21.15.161:0/357029016 >> v2:172.21.15.170:3300/0 conn(0x7f59880c0330 0x7f59880c9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:44.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.237+0000 7f5986ffd640 1 -- 172.21.15.161:0/357029016 >> v1:172.21.15.170:6789/0 conn(0x7f5988152bc0 legacy=0x7f59880c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:44.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.237+0000 7f5986ffd640 1 -- 172.21.15.161:0/357029016 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59880c96d0 con 0x7f59881589b0 2024-03-28T23:32:44.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.237+0000 7f5986ffd640 1 -- 172.21.15.161:0/357029016 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f59800019f0 con 0x7f59881589b0 2024-03-28T23:32:44.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.237+0000 7f5986ffd640 1 -- 172.21.15.161:0/357029016 >> v1:172.21.15.161:6789/0 conn(0x7f59881589b0 legacy=0x7f5988158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:44.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.237+0000 7f5986ffd640 1 --2- 172.21.15.161:0/357029016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5970003a90 0x7f5970003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:44.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.238+0000 7f5986ffd640 1 --2- 172.21.15.161:0/357029016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5970004800 0x7f5970004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:44.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.238+0000 7f5986ffd640 1 --2- 172.21.15.161:0/357029016 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5970005210 0x7f597000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:44.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.238+0000 7f5986ffd640 1 -- 172.21.15.161:0/357029016 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f59880c6400 con 0x7f5970005210 2024-03-28T23:32:44.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.238+0000 7f5986ffd640 1 -- 172.21.15.161:0/357029016 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f597000c140 con 0x7f5970003a90 2024-03-28T23:32:44.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.238+0000 7f5986ffd640 1 -- 172.21.15.161:0/357029016 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f597000c340 con 0x7f5970004800 2024-03-28T23:32:44.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.238+0000 7f5987fff640 1 --2- 172.21.15.161:0/357029016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5970003a90 0x7f5970003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:44.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.238+0000 7f598cd82640 1 --2- 172.21.15.161:0/357029016 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5970005210 0x7f597000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:44.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.238+0000 7f59877fe640 1 --2- 172.21.15.161:0/357029016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5970004800 0x7f5970004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:44.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.238+0000 7f5987fff640 1 -- 172.21.15.161:0/357029016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5970004800 msgr2=0x7f5970004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:44.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.238+0000 7f5987fff640 1 --2- 172.21.15.161:0/357029016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5970004800 0x7f5970004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:44.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.238+0000 7f5987fff640 1 -- 172.21.15.161:0/357029016 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5970005210 msgr2=0x7f597000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:44.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.238+0000 7f5987fff640 1 --2- 172.21.15.161:0/357029016 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5970005210 0x7f597000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:44.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.238+0000 7f5987fff640 1 -- 172.21.15.161:0/357029016 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5970003520 con 0x7f5970003a90 2024-03-28T23:32:44.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.238+0000 7f598cd82640 1 --2- 172.21.15.161:0/357029016 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5970005210 0x7f597000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:32:44.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.238+0000 7f598cd82640 1 -- 172.21.15.161:0/357029016 reap_dead start 2024-03-28T23:32:44.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.239+0000 7f5987fff640 1 --2- 172.21.15.161:0/357029016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5970003a90 0x7f5970003eb0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5978001720 tx=0x7f5970005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5176c1c2517be3e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:44.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.239+0000 7f5986ffd640 1 -- 172.21.15.161:0/357029016 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5978001370 con 0x7f5970003a90 2024-03-28T23:32:44.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.239+0000 7f5986ffd640 1 -- 172.21.15.161:0/357029016 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59780052d0 con 0x7f5970003a90 2024-03-28T23:32:44.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.239+0000 7f5986ffd640 1 -- 172.21.15.161:0/357029016 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5978003830 con 0x7f5970003a90 2024-03-28T23:32:44.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.240+0000 7f598e7e5640 1 -- 172.21.15.161:0/357029016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5970003a90 msgr2=0x7f5970003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:44.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.240+0000 7f598e7e5640 1 --2- 172.21.15.161:0/357029016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5970003a90 0x7f5970003eb0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5978001720 tx=0x7f5970005830 comp rx=0 tx=0).stop 2024-03-28T23:32:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.240+0000 7f598e7e5640 1 -- 172.21.15.161:0/357029016 shutdown_connections 2024-03-28T23:32:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.240+0000 7f598e7e5640 1 --2- 172.21.15.161:0/357029016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5970003a90 0x7f5970003eb0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.240+0000 7f598e7e5640 1 -- 172.21.15.161:0/357029016 >> 172.21.15.161:0/357029016 conn(0x7f59880caa80 msgr2=0x7f59880cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.240+0000 7f598e7e5640 1 -- 172.21.15.161:0/357029016 shutdown_connections 2024-03-28T23:32:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.240+0000 7f598e7e5640 1 -- 172.21.15.161:0/357029016 wait complete. 2024-03-28T23:32:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.241+0000 7f598e7e5640 1 Processor -- start 2024-03-28T23:32:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.241+0000 7f598e7e5640 1 -- start start 2024-03-28T23:32:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.242+0000 7f598e7e5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5970003a90 0x7f59881eb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.242+0000 7f598e7e5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f598806c930 0x7f5988151430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.242+0000 7f598e7e5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59881eb840 0x7f59880b03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.242+0000 7f598e7e5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5988152fa0 con 0x7f59881eb840 2024-03-28T23:32:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.242+0000 7f598e7e5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f59880c6bf0 con 0x7f598806c930 2024-03-28T23:32:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.242+0000 7f598e7e5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5988152020 con 0x7f5970003a90 2024-03-28T23:32:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.242+0000 7f598cd82640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59881eb840 0x7f59880b03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.242+0000 7f598cd82640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59881eb840 0x7f59880b03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39800/0 (socket says 172.21.15.161:39800) 2024-03-28T23:32:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.242+0000 7f598cd82640 1 -- 172.21.15.161:0/3961648819 learned_addr learned my addr 172.21.15.161:0/3961648819 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:44.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.242+0000 7f5987fff640 1 --2- 172.21.15.161:0/3961648819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5970003a90 0x7f59881eb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:44.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.242+0000 7f59877fe640 1 --2- 172.21.15.161:0/3961648819 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f598806c930 0x7f5988151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:44.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.242+0000 7f598cd82640 1 -- 172.21.15.161:0/3961648819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5970003a90 msgr2=0x7f59881eb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:44.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.242+0000 7f598cd82640 1 --2- 172.21.15.161:0/3961648819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5970003a90 0x7f59881eb300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:44.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.242+0000 7f598cd82640 1 -- 172.21.15.161:0/3961648819 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f598806c930 msgr2=0x7f5988151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:44.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.242+0000 7f598cd82640 1 --2- 172.21.15.161:0/3961648819 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f598806c930 0x7f5988151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:44.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.242+0000 7f598cd82640 1 -- 172.21.15.161:0/3961648819 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f598814a810 con 0x7f59881eb840 2024-03-28T23:32:44.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.242+0000 7f598cd82640 1 --2- 172.21.15.161:0/3961648819 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59881eb840 0x7f59880b03a0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f59880c0fc0 tx=0x7f5980007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:44.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.242+0000 7f5984ff9640 1 -- 172.21.15.161:0/3961648819 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f598000a4a0 con 0x7f59881eb840 2024-03-28T23:32:44.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.242+0000 7f5984ff9640 1 -- 172.21.15.161:0/3961648819 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5980019070 con 0x7f59881eb840 2024-03-28T23:32:44.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.242+0000 7f598e7e5640 1 -- 172.21.15.161:0/3961648819 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f598814a530 con 0x7f59881eb840 2024-03-28T23:32:44.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.242+0000 7f5984ff9640 1 -- 172.21.15.161:0/3961648819 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5980007ad0 con 0x7f59881eb840 2024-03-28T23:32:44.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.242+0000 7f598e7e5640 1 -- 172.21.15.161:0/3961648819 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f598806d100 con 0x7f59881eb840 2024-03-28T23:32:44.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.243+0000 7f598e7e5640 1 -- 172.21.15.161:0/3961648819 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f598806d340 con 0x7f59881eb840 2024-03-28T23:32:44.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.244+0000 7f5984ff9640 1 -- 172.21.15.161:0/3961648819 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5980013650 con 0x7f59881eb840 2024-03-28T23:32:44.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.244+0000 7f5984ff9640 1 --2- 172.21.15.161:0/3961648819 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f597406b4d0 0x7f597406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:44.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.245+0000 7f5984ff9640 1 -- 172.21.15.161:0/3961648819 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6855+0+0 (secure 0 0 0) 0x7f598008b630 con 0x7f59881eb840 2024-03-28T23:32:44.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.245+0000 7f5987fff640 1 --2- 172.21.15.161:0/3961648819 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f597406b4d0 0x7f597406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:44.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.245+0000 7f5987fff640 1 --2- 172.21.15.161:0/3961648819 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f597406b4d0 0x7f597406d990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f5978033000 tx=0x7f5978009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:44.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.249+0000 7f5984ff9640 1 -- 172.21.15.161:0/3961648819 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f598806d340 con 0x7f59881eb840 2024-03-28T23:32:44.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:44.461+0000 7f598e7e5640 1 -- 172.21.15.161:0/3961648819 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"} v 0) -- 0x7f5988158e00 con 0x7f59881eb840 2024-03-28T23:32:45.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.102+0000 7f5984ff9640 1 -- 172.21.15.161:0/3961648819 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"}]=0 new fs with metadata pool 13 and data pool 14 v57) ==== 177+0+0 (secure 0 0 0) 0x7f5988158e00 con 0x7f59881eb840 2024-03-28T23:32:45.103 INFO:teuthology.orchestra.run.smithi161.stderr:new fs with metadata pool 13 and data pool 14 2024-03-28T23:32:45.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.107+0000 7f598e7e5640 1 -- 172.21.15.161:0/3961648819 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f597406b4d0 msgr2=0x7f597406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:45.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.107+0000 7f598e7e5640 1 --2- 172.21.15.161:0/3961648819 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f597406b4d0 0x7f597406d990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f5978033000 tx=0x7f5978009030 comp rx=0 tx=0).stop 2024-03-28T23:32:45.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.107+0000 7f598e7e5640 1 -- 172.21.15.161:0/3961648819 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59881eb840 msgr2=0x7f59880b03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:45.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.107+0000 7f598e7e5640 1 --2- 172.21.15.161:0/3961648819 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59881eb840 0x7f59880b03a0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f59880c0fc0 tx=0x7f5980007430 comp rx=0 tx=0).stop 2024-03-28T23:32:45.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.108+0000 7f598e7e5640 1 -- 172.21.15.161:0/3961648819 shutdown_connections 2024-03-28T23:32:45.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.108+0000 7f598e7e5640 1 --2- 172.21.15.161:0/3961648819 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f597406b4d0 0x7f597406d990 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.108+0000 7f598e7e5640 1 --2- 172.21.15.161:0/3961648819 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59881eb840 0x7f59880b03a0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.108+0000 7f598e7e5640 1 --2- 172.21.15.161:0/3961648819 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f598806c930 0x7f5988151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.108+0000 7f598e7e5640 1 --2- 172.21.15.161:0/3961648819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5970003a90 0x7f59881eb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.108+0000 7f598e7e5640 1 -- 172.21.15.161:0/3961648819 >> 172.21.15.161:0/3961648819 conn(0x7f59880caa80 msgr2=0x7f59880c8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:45.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.108+0000 7f598e7e5640 1 -- 172.21.15.161:0/3961648819 shutdown_connections 2024-03-28T23:32:45.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.108+0000 7f598e7e5640 1 -- 172.21.15.161:0/3961648819 wait complete. 2024-03-28T23:32:45.139 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:32:45.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.242+0000 7f85fbdc7640 1 Processor -- start 2024-03-28T23:32:45.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.242+0000 7f85fbdc7640 1 -- start start 2024-03-28T23:32:45.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.243+0000 7f85fbdc7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f85f4152890 0x7f85f40c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:45.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.243+0000 7f85fbdc7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85f40c8ed0 0x7f85f40c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:45.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.243+0000 7f85fbdc7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85f4158a30 con 0x7f85f40c5950 2024-03-28T23:32:45.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.243+0000 7f85fbdc7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f85f40c8480 con 0x7f85f4152890 2024-03-28T23:32:45.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.243+0000 7f85fbdc7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f85f4159640 con 0x7f85f40c8ed0 2024-03-28T23:32:45.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.244+0000 7f85f9b63640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f85f4152890 0x7f85f40c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:45.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.244+0000 7f85f9362640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85f40c8ed0 0x7f85f40c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:45.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.244+0000 7f85f9b63640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f85f4152890 0x7f85f40c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:35170/0 (socket says 172.21.15.161:35170) 2024-03-28T23:32:45.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.244+0000 7f85f9b63640 1 -- 172.21.15.161:0/3804888742 learned_addr learned my addr 172.21.15.161:0/3804888742 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:45.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.244+0000 7f85fa364640 1 --1- 172.21.15.161:0/3804888742 >> v1:172.21.15.170:6789/0 conn(0x7f85f40c5950 0x7f85f41553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:34136/0 (socket says 172.21.15.161:34136) 2024-03-28T23:32:45.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.244+0000 7f85f8b61640 1 -- 172.21.15.161:0/3804888742 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1424596571 0 0) 0x7f85f4158a30 con 0x7f85f40c5950 2024-03-28T23:32:45.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.244+0000 7f85f8b61640 1 -- 172.21.15.161:0/3804888742 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85e0003520 con 0x7f85f40c5950 2024-03-28T23:32:45.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.244+0000 7f85f9b63640 1 -- 172.21.15.161:0/3804888742 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85f40c8ed0 msgr2=0x7f85f40c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:45.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.244+0000 7f85f9b63640 1 --2- 172.21.15.161:0/3804888742 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85f40c8ed0 0x7f85f40c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.244+0000 7f85f9b63640 1 -- 172.21.15.161:0/3804888742 >> v1:172.21.15.170:6789/0 conn(0x7f85f40c5950 legacy=0x7f85f41553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:45.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.244+0000 7f85f9b63640 1 -- 172.21.15.161:0/3804888742 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f4158d10 con 0x7f85f4152890 2024-03-28T23:32:45.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.244+0000 7f85f9362640 1 --2- 172.21.15.161:0/3804888742 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85f40c8ed0 0x7f85f40c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:32:45.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.245+0000 7f85f9b63640 1 --2- 172.21.15.161:0/3804888742 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85f4152890 0x7f85f40c8990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f85f0009870 tx=0x7f85f002b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b0dfe565b3422af server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:45.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.245+0000 7f85f8b61640 1 -- 172.21.15.161:0/3804888742 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f00317c0 con 0x7f85f4152890 2024-03-28T23:32:45.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.245+0000 7f85f8b61640 1 -- 172.21.15.161:0/3804888742 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85f0037070 con 0x7f85f4152890 2024-03-28T23:32:45.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.245+0000 7f85f8b61640 1 -- 172.21.15.161:0/3804888742 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f0031b80 con 0x7f85f4152890 2024-03-28T23:32:45.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.245+0000 7f85fbdc7640 1 -- 172.21.15.161:0/3804888742 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85f4152890 msgr2=0x7f85f40c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:45.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.245+0000 7f85fbdc7640 1 --2- 172.21.15.161:0/3804888742 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85f4152890 0x7f85f40c8990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f85f0009870 tx=0x7f85f002b860 comp rx=0 tx=0).stop 2024-03-28T23:32:45.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.245+0000 7f85fbdc7640 1 -- 172.21.15.161:0/3804888742 shutdown_connections 2024-03-28T23:32:45.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.245+0000 7f85fbdc7640 1 --2- 172.21.15.161:0/3804888742 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85f40c8ed0 0x7f85f40c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.245+0000 7f85fbdc7640 1 --2- 172.21.15.161:0/3804888742 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85f4152890 0x7f85f40c8990 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.245+0000 7f85fbdc7640 1 -- 172.21.15.161:0/3804888742 >> 172.21.15.161:0/3804888742 conn(0x7f85f40ac620 msgr2=0x7f85f40aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:45.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.246+0000 7f85fbdc7640 1 -- 172.21.15.161:0/3804888742 shutdown_connections 2024-03-28T23:32:45.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.246+0000 7f85fbdc7640 1 -- 172.21.15.161:0/3804888742 wait complete. 2024-03-28T23:32:45.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.247+0000 7f85fbdc7640 1 Processor -- start 2024-03-28T23:32:45.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.247+0000 7f85fbdc7640 1 -- start start 2024-03-28T23:32:45.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.248+0000 7f85fbdc7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85f40c5950 0x7f85f41ea8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:45.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.248+0000 7f85fbdc7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85f40c8ed0 0x7f85f40ad950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:45.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.248+0000 7f85fbdc7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85f4152890 0x7f85f40afc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:45.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.248+0000 7f85fbdc7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f85f4152c70 con 0x7f85f40c5950 2024-03-28T23:32:45.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.248+0000 7f85fbdc7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f85f41592d0 con 0x7f85f4152890 2024-03-28T23:32:45.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.248+0000 7f85fbdc7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f85f4152df0 con 0x7f85f40c8ed0 2024-03-28T23:32:45.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.249+0000 7f85f9b63640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85f40c5950 0x7f85f41ea8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:45.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.249+0000 7f85f9b63640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85f40c5950 0x7f85f41ea8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39804/0 (socket says 172.21.15.161:39804) 2024-03-28T23:32:45.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.249+0000 7f85f9b63640 1 -- 172.21.15.161:0/3180605442 learned_addr learned my addr 172.21.15.161:0/3180605442 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:45.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.249+0000 7f85f9362640 1 --2- 172.21.15.161:0/3180605442 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85f40c8ed0 0x7f85f40ad950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:45.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.249+0000 7f85fa364640 1 --2- 172.21.15.161:0/3180605442 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85f4152890 0x7f85f40afc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:45.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.249+0000 7f85f9b63640 1 -- 172.21.15.161:0/3180605442 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85f40c8ed0 msgr2=0x7f85f40ad950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:45.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.249+0000 7f85f9b63640 1 --2- 172.21.15.161:0/3180605442 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85f40c8ed0 0x7f85f40ad950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.249+0000 7f85f9b63640 1 -- 172.21.15.161:0/3180605442 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85f4152890 msgr2=0x7f85f40afc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:45.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.249+0000 7f85f9b63640 1 --2- 172.21.15.161:0/3180605442 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85f4152890 0x7f85f40afc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.249+0000 7f85f9b63640 1 -- 172.21.15.161:0/3180605442 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f40ac000 con 0x7f85f40c5950 2024-03-28T23:32:45.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.249+0000 7f85f9362640 1 --2- 172.21.15.161:0/3180605442 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85f40c8ed0 0x7f85f40ad950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:32:45.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.249+0000 7f85fa364640 1 --2- 172.21.15.161:0/3180605442 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85f4152890 0x7f85f40afc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:32:45.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.249+0000 7f85f9b63640 1 --2- 172.21.15.161:0/3180605442 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85f40c5950 0x7f85f41ea8b0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f85f002b8a0 tx=0x7f85f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:45.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.249+0000 7f85ea7fc640 1 -- 172.21.15.161:0/3180605442 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f0032650 con 0x7f85f40c5950 2024-03-28T23:32:45.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.249+0000 7f85ea7fc640 1 -- 172.21.15.161:0/3180605442 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85f00327f0 con 0x7f85f40c5950 2024-03-28T23:32:45.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.249+0000 7f85fbdc7640 1 -- 172.21.15.161:0/3180605442 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85f40b0880 con 0x7f85f40c5950 2024-03-28T23:32:45.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.249+0000 7f85ea7fc640 1 -- 172.21.15.161:0/3180605442 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f0032be0 con 0x7f85f40c5950 2024-03-28T23:32:45.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.249+0000 7f85fbdc7640 1 -- 172.21.15.161:0/3180605442 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85f40ade90 con 0x7f85f40c5950 2024-03-28T23:32:45.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.250+0000 7f85fbdc7640 1 -- 172.21.15.161:0/3180605442 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85f40af4e0 con 0x7f85f40c5950 2024-03-28T23:32:45.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.251+0000 7f85ea7fc640 1 -- 172.21.15.161:0/3180605442 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f85f004c060 con 0x7f85f40c5950 2024-03-28T23:32:45.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.251+0000 7f85ea7fc640 1 --2- 172.21.15.161:0/3180605442 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f85d806b4d0 0x7f85d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:45.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.251+0000 7f85f9362640 1 --2- 172.21.15.161:0/3180605442 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f85d806b4d0 0x7f85d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:45.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.251+0000 7f85ea7fc640 1 -- 172.21.15.161:0/3180605442 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6957+0+0 (secure 0 0 0) 0x7f85f00abef0 con 0x7f85f40c5950 2024-03-28T23:32:45.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.252+0000 7f85f9362640 1 --2- 172.21.15.161:0/3180605442 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f85d806b4d0 0x7f85d806d990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f85f40ac470 tx=0x7f85e4009110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:45.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.256+0000 7f85ea7fc640 1 -- 172.21.15.161:0/3180605442 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f85f0040070 con 0x7f85f40c5950 2024-03-28T23:32:45.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.441+0000 7f85fbdc7640 1 -- 172.21.15.161:0/3180605442 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f85f404f4b0 con 0x7f85f40c5950 2024-03-28T23:32:45.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.442+0000 7f85ea7fc640 1 -- 172.21.15.161:0/3180605442 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) ==== 74+0+20718 (secure 0 0 0) 0x7f85f007a2d0 con 0x7f85f40c5950 2024-03-28T23:32:45.443 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:45.443 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":76,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:32:44.462981+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:32:30.684936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-28T23:32:31.339722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:32:42.378609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:32:43.436985+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:32:45.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.444+0000 7f85fbdc7640 1 -- 172.21.15.161:0/3180605442 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f85d806b4d0 msgr2=0x7f85d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:45.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.444+0000 7f85fbdc7640 1 --2- 172.21.15.161:0/3180605442 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f85d806b4d0 0x7f85d806d990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f85f40ac470 tx=0x7f85e4009110 comp rx=0 tx=0).stop 2024-03-28T23:32:45.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.444+0000 7f85fbdc7640 1 -- 172.21.15.161:0/3180605442 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85f40c5950 msgr2=0x7f85f41ea8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:45.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.444+0000 7f85fbdc7640 1 --2- 172.21.15.161:0/3180605442 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85f40c5950 0x7f85f41ea8b0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f85f002b8a0 tx=0x7f85f00046c0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.444+0000 7f85fbdc7640 1 -- 172.21.15.161:0/3180605442 shutdown_connections 2024-03-28T23:32:45.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.444+0000 7f85fbdc7640 1 --2- 172.21.15.161:0/3180605442 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f85d806b4d0 0x7f85d806d990 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.444+0000 7f85fbdc7640 1 --2- 172.21.15.161:0/3180605442 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85f4152890 0x7f85f40afc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.444+0000 7f85fbdc7640 1 --2- 172.21.15.161:0/3180605442 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85f40c8ed0 0x7f85f40ad950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.444+0000 7f85fbdc7640 1 --2- 172.21.15.161:0/3180605442 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85f40c5950 0x7f85f41ea8b0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.444+0000 7f85fbdc7640 1 -- 172.21.15.161:0/3180605442 >> 172.21.15.161:0/3180605442 conn(0x7f85f40ac620 msgr2=0x7f85f40c9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:45.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.444+0000 7f85fbdc7640 1 -- 172.21.15.161:0/3180605442 shutdown_connections 2024-03-28T23:32:45.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.444+0000 7f85fbdc7640 1 -- 172.21.15.161:0/3180605442 wait complete. 2024-03-28T23:32:45.456 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:32:45.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.558+0000 7f8746f7a640 1 Processor -- start 2024-03-28T23:32:45.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.558+0000 7f8746f7a640 1 -- start start 2024-03-28T23:32:45.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.559+0000 7f8746f7a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f874014eca0 0x7f87400c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:45.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.559+0000 7f8746f7a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f87400c1dc0 0x7f874006ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:45.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.559+0000 7f8746f7a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8740151850 con 0x7f874006cf60 2024-03-28T23:32:45.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.559+0000 7f8746f7a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8740152460 con 0x7f87400c1dc0 2024-03-28T23:32:45.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.559+0000 7f8746f7a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f8740151b30 con 0x7f874014eca0 2024-03-28T23:32:45.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.559+0000 7f873ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f87400c1dc0 0x7f874006ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:45.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.559+0000 7f873ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f87400c1dc0 0x7f874006ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39810/0 (socket says 172.21.15.161:39810) 2024-03-28T23:32:45.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.559+0000 7f873ffff640 1 -- 172.21.15.161:0/2428811639 learned_addr learned my addr 172.21.15.161:0/2428811639 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:45.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.559+0000 7f8744d16640 1 --2- 172.21.15.161:0/2428811639 >> v2:172.21.15.170:3300/0 conn(0x7f874014eca0 0x7f87400c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:45.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.559+0000 7f873ffff640 1 -- 172.21.15.161:0/2428811639 >> v2:172.21.15.170:3300/0 conn(0x7f874014eca0 msgr2=0x7f87400c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:45.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.559+0000 7f873ffff640 1 --2- 172.21.15.161:0/2428811639 >> v2:172.21.15.170:3300/0 conn(0x7f874014eca0 0x7f87400c1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.559+0000 7f873ffff640 1 -- 172.21.15.161:0/2428811639 >> v1:172.21.15.170:6789/0 conn(0x7f874006cf60 legacy=0x7f87400a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:45.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.559+0000 7f873ffff640 1 -- 172.21.15.161:0/2428811639 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8740151e10 con 0x7f87400c1dc0 2024-03-28T23:32:45.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.559+0000 7f8744d16640 1 --2- 172.21.15.161:0/2428811639 >> v2:172.21.15.170:3300/0 conn(0x7f874014eca0 0x7f87400c1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:45.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.560+0000 7f873ffff640 1 --2- 172.21.15.161:0/2428811639 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87400c1dc0 0x7f874006ca20 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f87400c75a0 tx=0x7f873402de20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27401e2128247b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:45.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.560+0000 7f873f7fe640 1 -- 172.21.15.161:0/2428811639 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8734034070 con 0x7f87400c1dc0 2024-03-28T23:32:45.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.560+0000 7f873f7fe640 1 -- 172.21.15.161:0/2428811639 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f873402ebd0 con 0x7f87400c1dc0 2024-03-28T23:32:45.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.560+0000 7f873f7fe640 1 -- 172.21.15.161:0/2428811639 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8734038400 con 0x7f87400c1dc0 2024-03-28T23:32:45.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.561+0000 7f8746f7a640 1 -- 172.21.15.161:0/2428811639 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87400c1dc0 msgr2=0x7f874006ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:45.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.561+0000 7f8746f7a640 1 --2- 172.21.15.161:0/2428811639 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87400c1dc0 0x7f874006ca20 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f87400c75a0 tx=0x7f873402de20 comp rx=0 tx=0).stop 2024-03-28T23:32:45.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.561+0000 7f8746f7a640 1 -- 172.21.15.161:0/2428811639 shutdown_connections 2024-03-28T23:32:45.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.561+0000 7f8746f7a640 1 --2- 172.21.15.161:0/2428811639 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87400c1dc0 0x7f874006ca20 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.561+0000 7f8746f7a640 1 --2- 172.21.15.161:0/2428811639 >> v2:172.21.15.170:3300/0 conn(0x7f874014eca0 0x7f87400c1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.561+0000 7f8746f7a640 1 -- 172.21.15.161:0/2428811639 >> 172.21.15.161:0/2428811639 conn(0x7f87400c6670 msgr2=0x7f87400c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:45.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.561+0000 7f8746f7a640 1 -- 172.21.15.161:0/2428811639 shutdown_connections 2024-03-28T23:32:45.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.561+0000 7f8746f7a640 1 -- 172.21.15.161:0/2428811639 wait complete. 2024-03-28T23:32:45.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.562+0000 7f8746f7a640 1 Processor -- start 2024-03-28T23:32:45.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.562+0000 7f8746f7a640 1 -- start start 2024-03-28T23:32:45.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.563+0000 7f8746f7a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f874006cf60 0x7f8740071b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:45.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.563+0000 7f8746f7a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87400c1dc0 0x7f87400c54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:45.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.563+0000 7f8746f7a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f874014eca0 0x7f87401533a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:45.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.563+0000 7f8746f7a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f87401514b0 con 0x7f874006cf60 2024-03-28T23:32:45.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.563+0000 7f8746f7a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f874014f080 con 0x7f874014eca0 2024-03-28T23:32:45.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.563+0000 7f8746f7a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8740151630 con 0x7f87400c1dc0 2024-03-28T23:32:45.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.563+0000 7f8744d16640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f874006cf60 0x7f8740071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:45.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.563+0000 7f8744d16640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f874006cf60 0x7f8740071b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39826/0 (socket says 172.21.15.161:39826) 2024-03-28T23:32:45.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.563+0000 7f8744d16640 1 -- 172.21.15.161:0/3277985798 learned_addr learned my addr 172.21.15.161:0/3277985798 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:45.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.563+0000 7f8744d16640 1 -- 172.21.15.161:0/3277985798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87400c1dc0 msgr2=0x7f87400c54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:45.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.563+0000 7f8744d16640 1 --2- 172.21.15.161:0/3277985798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87400c1dc0 0x7f87400c54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.563+0000 7f8744d16640 1 -- 172.21.15.161:0/3277985798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f874014eca0 msgr2=0x7f87401533a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:45.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.563+0000 7f8745517640 1 --2- 172.21.15.161:0/3277985798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f874014eca0 0x7f87401533a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:45.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.563+0000 7f8744d16640 1 --2- 172.21.15.161:0/3277985798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f874014eca0 0x7f87401533a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.563+0000 7f8744d16640 1 -- 172.21.15.161:0/3277985798 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8740072a90 con 0x7f874006cf60 2024-03-28T23:32:45.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.563+0000 7f8744d16640 1 --2- 172.21.15.161:0/3277985798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f874006cf60 0x7f8740071b70 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f873000c820 tx=0x7f873000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:45.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.564+0000 7f873d7fa640 1 -- 172.21.15.161:0/3277985798 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8730015070 con 0x7f874006cf60 2024-03-28T23:32:45.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.564+0000 7f873d7fa640 1 -- 172.21.15.161:0/3277985798 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8730007a30 con 0x7f874006cf60 2024-03-28T23:32:45.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.564+0000 7f8746f7a640 1 -- 172.21.15.161:0/3277985798 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8740071090 con 0x7f874006cf60 2024-03-28T23:32:45.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.564+0000 7f873d7fa640 1 -- 172.21.15.161:0/3277985798 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8730007de0 con 0x7f874006cf60 2024-03-28T23:32:45.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.564+0000 7f8746f7a640 1 -- 172.21.15.161:0/3277985798 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8740071650 con 0x7f874006cf60 2024-03-28T23:32:45.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.565+0000 7f8746f7a640 1 -- 172.21.15.161:0/3277985798 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f874004f4b0 con 0x7f874006cf60 2024-03-28T23:32:45.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.565+0000 7f873d7fa640 1 -- 172.21.15.161:0/3277985798 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8730018070 con 0x7f874006cf60 2024-03-28T23:32:45.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.565+0000 7f873d7fa640 1 --2- 172.21.15.161:0/3277985798 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f872c06b4d0 0x7f872c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:45.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.566+0000 7f873ffff640 1 --2- 172.21.15.161:0/3277985798 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f872c06b4d0 0x7f872c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:45.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.566+0000 7f873d7fa640 1 -- 172.21.15.161:0/3277985798 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6957+0+0 (secure 0 0 0) 0x7f873008a580 con 0x7f874006cf60 2024-03-28T23:32:45.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.566+0000 7f873ffff640 1 --2- 172.21.15.161:0/3277985798 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f872c06b4d0 0x7f872c06d990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f8734039000 tx=0x7f8734030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:45.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.570+0000 7f873d7fa640 1 -- 172.21.15.161:0/3277985798 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8730058fb0 con 0x7f874006cf60 2024-03-28T23:32:45.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.747+0000 7f8746f7a640 1 -- 172.21.15.161:0/3277985798 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f87400c2ae0 con 0x7f874006cf60 2024-03-28T23:32:45.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.748+0000 7f873d7fa640 1 -- 172.21.15.161:0/3277985798 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) ==== 74+0+20718 (secure 0 0 0) 0x7f87300588b0 con 0x7f874006cf60 2024-03-28T23:32:45.749 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:45.749 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":76,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:32:44.462981+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:32:30.684936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-28T23:32:31.339722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:32:42.378609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:32:43.436985+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:32:45.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.750+0000 7f8746f7a640 1 -- 172.21.15.161:0/3277985798 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f872c06b4d0 msgr2=0x7f872c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:45.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.750+0000 7f8746f7a640 1 --2- 172.21.15.161:0/3277985798 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f872c06b4d0 0x7f872c06d990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f8734039000 tx=0x7f8734030000 comp rx=0 tx=0).stop 2024-03-28T23:32:45.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.750+0000 7f8746f7a640 1 -- 172.21.15.161:0/3277985798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f874006cf60 msgr2=0x7f8740071b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:45.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.750+0000 7f8746f7a640 1 --2- 172.21.15.161:0/3277985798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f874006cf60 0x7f8740071b70 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f873000c820 tx=0x7f873000cce0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.751+0000 7f8746f7a640 1 -- 172.21.15.161:0/3277985798 shutdown_connections 2024-03-28T23:32:45.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.751+0000 7f8746f7a640 1 --2- 172.21.15.161:0/3277985798 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f872c06b4d0 0x7f872c06d990 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.751+0000 7f8746f7a640 1 --2- 172.21.15.161:0/3277985798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f874014eca0 0x7f87401533a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.751+0000 7f8746f7a640 1 --2- 172.21.15.161:0/3277985798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87400c1dc0 0x7f87400c54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.751+0000 7f8746f7a640 1 --2- 172.21.15.161:0/3277985798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f874006cf60 0x7f8740071b70 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.751+0000 7f8746f7a640 1 -- 172.21.15.161:0/3277985798 >> 172.21.15.161:0/3277985798 conn(0x7f87400c6670 msgr2=0x7f874014cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:45.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.751+0000 7f8746f7a640 1 -- 172.21.15.161:0/3277985798 shutdown_connections 2024-03-28T23:32:45.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.751+0000 7f8746f7a640 1 -- 172.21.15.161:0/3277985798 wait complete. 2024-03-28T23:32:45.763 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 standby_count_wanted 0 2024-03-28T23:32:45.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.868+0000 7f799e28a640 1 Processor -- start 2024-03-28T23:32:45.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.868+0000 7f799e28a640 1 -- start start 2024-03-28T23:32:45.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.869+0000 7f799e28a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7998152bc0 0x7f79980c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:45.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.869+0000 7f799e28a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f79981589b0 0x7f7998158e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:45.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.869+0000 7f799e28a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79980c6f60 con 0x7f79980c8a80 2024-03-28T23:32:45.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.869+0000 7f799e28a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f79980c6120 con 0x7f79981589b0 2024-03-28T23:32:45.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.869+0000 7f799e28a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f79980c6400 con 0x7f7998152bc0 2024-03-28T23:32:45.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.869+0000 7f799c827640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f79981589b0 0x7f7998158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:45.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.869+0000 7f79977fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f79980c8a80 0x7f79980c8ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59322/0 (socket says 172.21.15.161:59322) 2024-03-28T23:32:45.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.869+0000 7f79977fe640 1 -- 172.21.15.161:0/1994441280 learned_addr learned my addr 172.21.15.161:0/1994441280 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:45.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.870+0000 7f7997fff640 1 --2- 172.21.15.161:0/1994441280 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7998152bc0 0x7f79980c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:45.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.870+0000 7f799c827640 1 -- 172.21.15.161:0/1994441280 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7998152bc0 msgr2=0x7f79980c0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:45.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.870+0000 7f799c827640 1 --2- 172.21.15.161:0/1994441280 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7998152bc0 0x7f79980c0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.870+0000 7f799c827640 1 -- 172.21.15.161:0/1994441280 >> v1:172.21.15.161:6789/0 conn(0x7f79980c8a80 legacy=0x7f79980c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:45.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.870+0000 7f799c827640 1 -- 172.21.15.161:0/1994441280 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79980c7240 con 0x7f79981589b0 2024-03-28T23:32:45.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.870+0000 7f7996ffd640 1 -- 172.21.15.161:0/1994441280 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3936613439 0 0) 0x7f79980c6f60 con 0x7f79980c8a80 2024-03-28T23:32:45.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.870+0000 7f7997fff640 1 --2- 172.21.15.161:0/1994441280 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7998152bc0 0x7f79980c0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:32:45.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.870+0000 7f799c827640 1 --2- 172.21.15.161:0/1994441280 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79981589b0 0x7f7998158e10 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f7990009d90 tx=0x7f799002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e68d025b4cfafded server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:45.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.870+0000 7f7996ffd640 1 -- 172.21.15.161:0/1994441280 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7990039070 con 0x7f79981589b0 2024-03-28T23:32:45.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.871+0000 7f7996ffd640 1 -- 172.21.15.161:0/1994441280 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f799002cc50 con 0x7f79981589b0 2024-03-28T23:32:45.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.871+0000 7f7996ffd640 1 -- 172.21.15.161:0/1994441280 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7990034440 con 0x7f79981589b0 2024-03-28T23:32:45.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.871+0000 7f799e28a640 1 -- 172.21.15.161:0/1994441280 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79981589b0 msgr2=0x7f7998158e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:45.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.871+0000 7f799e28a640 1 --2- 172.21.15.161:0/1994441280 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79981589b0 0x7f7998158e10 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f7990009d90 tx=0x7f799002bea0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.871+0000 7f799e28a640 1 -- 172.21.15.161:0/1994441280 shutdown_connections 2024-03-28T23:32:45.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.871+0000 7f799e28a640 1 --2- 172.21.15.161:0/1994441280 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79981589b0 0x7f7998158e10 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.871+0000 7f799e28a640 1 --2- 172.21.15.161:0/1994441280 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7998152bc0 0x7f79980c0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.871+0000 7f799e28a640 1 -- 172.21.15.161:0/1994441280 >> 172.21.15.161:0/1994441280 conn(0x7f79980caa80 msgr2=0x7f79980cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:45.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.871+0000 7f799e28a640 1 -- 172.21.15.161:0/1994441280 shutdown_connections 2024-03-28T23:32:45.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.871+0000 7f799e28a640 1 -- 172.21.15.161:0/1994441280 wait complete. 2024-03-28T23:32:45.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.872+0000 7f799e28a640 1 Processor -- start 2024-03-28T23:32:45.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.872+0000 7f799e28a640 1 -- start start 2024-03-28T23:32:45.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.873+0000 7f799e28a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79980c8a80 0x7f7998151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:45.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.873+0000 7f799e28a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7998152bc0 0x7f79981517c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:45.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.873+0000 7f799e28a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79981589b0 0x7f79981442b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:45.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.873+0000 7f799e28a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7998152fa0 con 0x7f7998152bc0 2024-03-28T23:32:45.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.873+0000 7f799e28a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f79980c95f0 con 0x7f79981589b0 2024-03-28T23:32:45.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.873+0000 7f799e28a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7998157190 con 0x7f79980c8a80 2024-03-28T23:32:45.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.873+0000 7f79977fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7998152bc0 0x7f79981517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:45.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.873+0000 7f79977fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7998152bc0 0x7f79981517c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39844/0 (socket says 172.21.15.161:39844) 2024-03-28T23:32:45.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.873+0000 7f79977fe640 1 -- 172.21.15.161:0/3568722560 learned_addr learned my addr 172.21.15.161:0/3568722560 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:45.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.873+0000 7f7997fff640 1 --2- 172.21.15.161:0/3568722560 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79980c8a80 0x7f7998151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:45.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.873+0000 7f79977fe640 1 -- 172.21.15.161:0/3568722560 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79980c8a80 msgr2=0x7f7998151280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:45.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.873+0000 7f799c827640 1 --2- 172.21.15.161:0/3568722560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79981589b0 0x7f79981442b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:45.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.873+0000 7f79977fe640 1 --2- 172.21.15.161:0/3568722560 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79980c8a80 0x7f7998151280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.873+0000 7f79977fe640 1 -- 172.21.15.161:0/3568722560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79981589b0 msgr2=0x7f79981442b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:45.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.873+0000 7f79977fe640 1 --2- 172.21.15.161:0/3568722560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79981589b0 0x7f79981442b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:45.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.873+0000 7f79977fe640 1 -- 172.21.15.161:0/3568722560 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79981eb390 con 0x7f7998152bc0 2024-03-28T23:32:45.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.874+0000 7f79977fe640 1 --2- 172.21.15.161:0/3568722560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7998152bc0 0x7f79981517c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f798800c4d0 tx=0x7f798800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:45.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.874+0000 7f7994ff9640 1 -- 172.21.15.161:0/3568722560 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7988018070 con 0x7f7998152bc0 2024-03-28T23:32:45.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.874+0000 7f7994ff9640 1 -- 172.21.15.161:0/3568722560 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7988007500 con 0x7f7998152bc0 2024-03-28T23:32:45.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.874+0000 7f799e28a640 1 -- 172.21.15.161:0/3568722560 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7998150510 con 0x7f7998152bc0 2024-03-28T23:32:45.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.874+0000 7f7994ff9640 1 -- 172.21.15.161:0/3568722560 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7988004a20 con 0x7f7998152bc0 2024-03-28T23:32:45.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.874+0000 7f799e28a640 1 -- 172.21.15.161:0/3568722560 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7998153750 con 0x7f7998152bc0 2024-03-28T23:32:45.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.875+0000 7f799e28a640 1 -- 172.21.15.161:0/3568722560 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f799804f4b0 con 0x7f7998152bc0 2024-03-28T23:32:45.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.875+0000 7f7994ff9640 1 -- 172.21.15.161:0/3568722560 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f798801b070 con 0x7f7998152bc0 2024-03-28T23:32:45.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.876+0000 7f7994ff9640 1 --2- 172.21.15.161:0/3568722560 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f798406b4d0 0x7f798406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:45.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.876+0000 7f7997fff640 1 --2- 172.21.15.161:0/3568722560 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f798406b4d0 0x7f798406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:45.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.876+0000 7f7994ff9640 1 -- 172.21.15.161:0/3568722560 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6957+0+0 (secure 0 0 0) 0x7f7988089960 con 0x7f7998152bc0 2024-03-28T23:32:45.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.876+0000 7f7997fff640 1 --2- 172.21.15.161:0/3568722560 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f798406b4d0 0x7f798406d990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f798c008850 tx=0x7f798c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:45.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:45.880+0000 7f7994ff9640 1 -- 172.21.15.161:0/3568722560 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7988058390 con 0x7f7998152bc0 2024-03-28T23:32:46.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.094+0000 7f799e28a640 1 -- 172.21.15.161:0/3568722560 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f79980c92d0 con 0x7f7998152bc0 2024-03-28T23:32:46.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.117+0000 7f7994ff9640 1 -- 172.21.15.161:0/3568722560 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"}]=0 v58) ==== 119+0+0 (secure 0 0 0) 0x7f7988057c90 con 0x7f7998152bc0 2024-03-28T23:32:46.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.119+0000 7f799e28a640 1 -- 172.21.15.161:0/3568722560 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f798406b4d0 msgr2=0x7f798406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:46.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.119+0000 7f799e28a640 1 --2- 172.21.15.161:0/3568722560 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f798406b4d0 0x7f798406d990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f798c008850 tx=0x7f798c008040 comp rx=0 tx=0).stop 2024-03-28T23:32:46.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.119+0000 7f799e28a640 1 -- 172.21.15.161:0/3568722560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7998152bc0 msgr2=0x7f79981517c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:46.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.119+0000 7f799e28a640 1 --2- 172.21.15.161:0/3568722560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7998152bc0 0x7f79981517c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f798800c4d0 tx=0x7f798800c990 comp rx=0 tx=0).stop 2024-03-28T23:32:46.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.119+0000 7f799e28a640 1 -- 172.21.15.161:0/3568722560 shutdown_connections 2024-03-28T23:32:46.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.119+0000 7f799e28a640 1 --2- 172.21.15.161:0/3568722560 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f798406b4d0 0x7f798406d990 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:46.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.119+0000 7f799e28a640 1 --2- 172.21.15.161:0/3568722560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7998152bc0 0x7f79981517c0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:46.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.119+0000 7f799e28a640 1 --2- 172.21.15.161:0/3568722560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79981589b0 0x7f79981442b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:46.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.119+0000 7f799e28a640 1 --2- 172.21.15.161:0/3568722560 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79980c8a80 0x7f7998151280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:46.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.119+0000 7f799e28a640 1 -- 172.21.15.161:0/3568722560 >> 172.21.15.161:0/3568722560 conn(0x7f79980caa80 msgr2=0x7f79980c5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:46.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.120+0000 7f799e28a640 1 -- 172.21.15.161:0/3568722560 shutdown_connections 2024-03-28T23:32:46.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.120+0000 7f799e28a640 1 -- 172.21.15.161:0/3568722560 wait complete. 2024-03-28T23:32:46.132 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-28T23:32:46.133 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 allow_standby_replay false 2024-03-28T23:32:46.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.242+0000 7f92aeced640 1 Processor -- start 2024-03-28T23:32:46.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.242+0000 7f92aeced640 1 -- start start 2024-03-28T23:32:46.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.243+0000 7f92aeced640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f92a8152bc0 0x7f92a80c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:46.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.243+0000 7f92aeced640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a81589b0 0x7f92a8158e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:46.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.243+0000 7f92aeced640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92a80c6f60 con 0x7f92a80c8a80 2024-03-28T23:32:46.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.243+0000 7f92aeced640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f92a80c6120 con 0x7f92a8152bc0 2024-03-28T23:32:46.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.243+0000 7f92aeced640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f92a80c6400 con 0x7f92a81589b0 2024-03-28T23:32:46.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.243+0000 7f92a7fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f92a80c8a80 0x7f92a80c8ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59338/0 (socket says 172.21.15.161:59338) 2024-03-28T23:32:46.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.243+0000 7f92a7fff640 1 -- 172.21.15.161:0/2294812951 learned_addr learned my addr 172.21.15.161:0/2294812951 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:46.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.243+0000 7f92aca89640 1 --2- 172.21.15.161:0/2294812951 >> v2:172.21.15.170:3300/0 conn(0x7f92a8152bc0 0x7f92a80c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:46.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.243+0000 7f92ad28a640 1 --2- 172.21.15.161:0/2294812951 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a81589b0 0x7f92a8158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:46.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.243+0000 7f92a77fe640 1 -- 172.21.15.161:0/2294812951 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2742096672 0 0) 0x7f92a80c6f60 con 0x7f92a80c8a80 2024-03-28T23:32:46.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.243+0000 7f92a77fe640 1 -- 172.21.15.161:0/2294812951 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9298003520 con 0x7f92a80c8a80 2024-03-28T23:32:46.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.243+0000 7f92ad28a640 1 -- 172.21.15.161:0/2294812951 >> v2:172.21.15.170:3300/0 conn(0x7f92a8152bc0 msgr2=0x7f92a80c0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:46.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.243+0000 7f92ad28a640 1 --2- 172.21.15.161:0/2294812951 >> v2:172.21.15.170:3300/0 conn(0x7f92a8152bc0 0x7f92a80c0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:46.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.243+0000 7f92ad28a640 1 -- 172.21.15.161:0/2294812951 >> v1:172.21.15.161:6789/0 conn(0x7f92a80c8a80 legacy=0x7f92a80c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:46.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.243+0000 7f92ad28a640 1 -- 172.21.15.161:0/2294812951 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a80c7240 con 0x7f92a81589b0 2024-03-28T23:32:46.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.244+0000 7f92ad28a640 1 --2- 172.21.15.161:0/2294812951 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a81589b0 0x7f92a8158e40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f92a0009d90 tx=0x7f92a002beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30254b19003948c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:46.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.244+0000 7f92a77fe640 1 -- 172.21.15.161:0/2294812951 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a0039070 con 0x7f92a81589b0 2024-03-28T23:32:46.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.244+0000 7f92a77fe640 1 -- 172.21.15.161:0/2294812951 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92a002cc60 con 0x7f92a81589b0 2024-03-28T23:32:46.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.244+0000 7f92a77fe640 1 -- 172.21.15.161:0/2294812951 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a0034660 con 0x7f92a81589b0 2024-03-28T23:32:46.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.245+0000 7f92aeced640 1 -- 172.21.15.161:0/2294812951 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a81589b0 msgr2=0x7f92a8158e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:46.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.245+0000 7f92aeced640 1 --2- 172.21.15.161:0/2294812951 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a81589b0 0x7f92a8158e40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f92a0009d90 tx=0x7f92a002beb0 comp rx=0 tx=0).stop 2024-03-28T23:32:46.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.245+0000 7f92aeced640 1 -- 172.21.15.161:0/2294812951 shutdown_connections 2024-03-28T23:32:46.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.245+0000 7f92aeced640 1 --2- 172.21.15.161:0/2294812951 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a81589b0 0x7f92a8158e40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:46.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.245+0000 7f92aeced640 1 --2- 172.21.15.161:0/2294812951 >> v2:172.21.15.170:3300/0 conn(0x7f92a8152bc0 0x7f92a80c0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:46.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.245+0000 7f92aeced640 1 -- 172.21.15.161:0/2294812951 >> 172.21.15.161:0/2294812951 conn(0x7f92a80caa80 msgr2=0x7f92a80cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:46.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.245+0000 7f92aeced640 1 -- 172.21.15.161:0/2294812951 shutdown_connections 2024-03-28T23:32:46.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.245+0000 7f92aeced640 1 -- 172.21.15.161:0/2294812951 wait complete. 2024-03-28T23:32:46.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.246+0000 7f92aeced640 1 Processor -- start 2024-03-28T23:32:46.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.246+0000 7f92aeced640 1 -- start start 2024-03-28T23:32:46.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.247+0000 7f92aeced640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80c8a80 0x7f92a814ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:46.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.247+0000 7f92aeced640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92a8152bc0 0x7f92a8150200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:46.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.247+0000 7f92aeced640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92a81589b0 0x7f92a806bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:46.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.247+0000 7f92aeced640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f92a8152fa0 con 0x7f92a81589b0 2024-03-28T23:32:46.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.247+0000 7f92aeced640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f92a80c95f0 con 0x7f92a8152bc0 2024-03-28T23:32:46.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.247+0000 7f92aeced640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f92a81edc00 con 0x7f92a80c8a80 2024-03-28T23:32:46.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.247+0000 7f92ad28a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92a81589b0 0x7f92a806bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:46.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.247+0000 7f92a7fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92a8152bc0 0x7f92a8150200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.247+0000 7f92aca89640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80c8a80 0x7f92a814ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.247+0000 7f92a7fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92a8152bc0 0x7f92a8150200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:35216/0 (socket says 172.21.15.161:35216) 2024-03-28T23:32:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.247+0000 7f92a7fff640 1 -- 172.21.15.161:0/1993837092 learned_addr learned my addr 172.21.15.161:0/1993837092 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.247+0000 7f92ad28a640 1 -- 172.21.15.161:0/1993837092 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80c8a80 msgr2=0x7f92a814ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.247+0000 7f92ad28a640 1 --2- 172.21.15.161:0/1993837092 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80c8a80 0x7f92a814ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.247+0000 7f92ad28a640 1 -- 172.21.15.161:0/1993837092 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92a8152bc0 msgr2=0x7f92a8150200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.247+0000 7f92ad28a640 1 --2- 172.21.15.161:0/1993837092 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92a8152bc0 0x7f92a8150200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.247+0000 7f92ad28a640 1 -- 172.21.15.161:0/1993837092 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a81523b0 con 0x7f92a81589b0 2024-03-28T23:32:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.247+0000 7f92aca89640 1 --2- 172.21.15.161:0/1993837092 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80c8a80 0x7f92a814ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:32:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.248+0000 7f92ad28a640 1 --2- 172.21.15.161:0/1993837092 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92a81589b0 0x7f92a806bb70 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f92a0033000 tx=0x7f92a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.248+0000 7f92a57fa640 1 -- 172.21.15.161:0/1993837092 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a0039050 con 0x7f92a81589b0 2024-03-28T23:32:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.248+0000 7f92a57fa640 1 -- 172.21.15.161:0/1993837092 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92a0045070 con 0x7f92a81589b0 2024-03-28T23:32:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.248+0000 7f92a57fa640 1 -- 172.21.15.161:0/1993837092 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a0009450 con 0x7f92a81589b0 2024-03-28T23:32:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.248+0000 7f92aeced640 1 -- 172.21.15.161:0/1993837092 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92a81ede50 con 0x7f92a81589b0 2024-03-28T23:32:46.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.248+0000 7f92aeced640 1 -- 172.21.15.161:0/1993837092 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92a81ee410 con 0x7f92a81589b0 2024-03-28T23:32:46.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.249+0000 7f92aeced640 1 -- 172.21.15.161:0/1993837092 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92a804f4b0 con 0x7f92a81589b0 2024-03-28T23:32:46.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.249+0000 7f92a57fa640 1 -- 172.21.15.161:0/1993837092 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f92a003e850 con 0x7f92a81589b0 2024-03-28T23:32:46.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.250+0000 7f92a57fa640 1 --2- 172.21.15.161:0/1993837092 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f928c06b500 0x7f928c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:46.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.250+0000 7f92a57fa640 1 -- 172.21.15.161:0/1993837092 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6957+0+0 (secure 0 0 0) 0x7f92a00ad180 con 0x7f92a81589b0 2024-03-28T23:32:46.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.250+0000 7f92aca89640 1 --2- 172.21.15.161:0/1993837092 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f928c06b500 0x7f928c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:46.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.250+0000 7f92aca89640 1 --2- 172.21.15.161:0/1993837092 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f928c06b500 0x7f928c06d9c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f929c009f30 tx=0x7f929c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:46.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.254+0000 7f92a57fa640 1 -- 172.21.15.161:0/1993837092 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f92a81523b0 con 0x7f92a81589b0 2024-03-28T23:32:46.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:46.603+0000 7f92aeced640 1 -- 172.21.15.161:0/1993837092 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f92a80c9060 con 0x7f92a81589b0 2024-03-28T23:32:47.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.121+0000 7f92a57fa640 1 -- 172.21.15.161:0/1993837092 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"}]=0 v59) ==== 123+0+0 (secure 0 0 0) 0x7f92a007b5f0 con 0x7f92a81589b0 2024-03-28T23:32:47.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.123+0000 7f92aeced640 1 -- 172.21.15.161:0/1993837092 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f928c06b500 msgr2=0x7f928c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:47.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.123+0000 7f92aeced640 1 --2- 172.21.15.161:0/1993837092 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f928c06b500 0x7f928c06d9c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f929c009f30 tx=0x7f929c009190 comp rx=0 tx=0).stop 2024-03-28T23:32:47.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.123+0000 7f92aeced640 1 -- 172.21.15.161:0/1993837092 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92a81589b0 msgr2=0x7f92a806bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:47.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.123+0000 7f92aeced640 1 --2- 172.21.15.161:0/1993837092 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92a81589b0 0x7f92a806bb70 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f92a0033000 tx=0x7f92a0004a40 comp rx=0 tx=0).stop 2024-03-28T23:32:47.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.123+0000 7f92aeced640 1 -- 172.21.15.161:0/1993837092 shutdown_connections 2024-03-28T23:32:47.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.123+0000 7f92aeced640 1 --2- 172.21.15.161:0/1993837092 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f928c06b500 0x7f928c06d9c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.123+0000 7f92aeced640 1 --2- 172.21.15.161:0/1993837092 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92a81589b0 0x7f92a806bb70 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.123+0000 7f92aeced640 1 --2- 172.21.15.161:0/1993837092 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92a8152bc0 0x7f92a8150200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.123+0000 7f92aeced640 1 --2- 172.21.15.161:0/1993837092 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92a80c8a80 0x7f92a814ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.123+0000 7f92aeced640 1 -- 172.21.15.161:0/1993837092 >> 172.21.15.161:0/1993837092 conn(0x7f92a80caa80 msgr2=0x7f92a814cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:47.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.123+0000 7f92aeced640 1 -- 172.21.15.161:0/1993837092 shutdown_connections 2024-03-28T23:32:47.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.123+0000 7f92aeced640 1 -- 172.21.15.161:0/1993837092 wait complete. 2024-03-28T23:32:47.137 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs2' 2024-03-28T23:32:47.137 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:32:47.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.240+0000 7f3798510640 1 Processor -- start 2024-03-28T23:32:47.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.240+0000 7f3798510640 1 -- start start 2024-03-28T23:32:47.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.241+0000 7f3798510640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37900c5850 0x7f3790155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:47.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.241+0000 7f3798510640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37900c5d00 con 0x7f3790152890 2024-03-28T23:32:47.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.241+0000 7f3798510640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3790158a30 con 0x7f37900c5410 2024-03-28T23:32:47.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.241+0000 7f3798510640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f37900c8480 con 0x7f37900c5850 2024-03-28T23:32:47.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.241+0000 7f37962ac640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f3790152890 0x7f37900c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59346/0 (socket says 172.21.15.161:59346) 2024-03-28T23:32:47.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.241+0000 7f37962ac640 1 -- 172.21.15.161:0/4065745042 learned_addr learned my addr 172.21.15.161:0/4065745042 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:47.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.241+0000 7f3796aad640 1 --2- 172.21.15.161:0/4065745042 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37900c5850 0x7f3790155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:47.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.241+0000 7f37952aa640 1 -- 172.21.15.161:0/4065745042 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1759396214 0 0) 0x7f37900c5d00 con 0x7f3790152890 2024-03-28T23:32:47.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.241+0000 7f37952aa640 1 -- 172.21.15.161:0/4065745042 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f377c003520 con 0x7f3790152890 2024-03-28T23:32:47.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.241+0000 7f37952aa640 1 -- 172.21.15.161:0/4065745042 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2859512469 0 0) 0x7f3790158a30 con 0x7f37900c5410 2024-03-28T23:32:47.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.242+0000 7f37952aa640 1 -- 172.21.15.161:0/4065745042 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37900c5d00 con 0x7f37900c5410 2024-03-28T23:32:47.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.242+0000 7f3796aad640 1 -- 172.21.15.161:0/4065745042 >> v1:172.21.15.170:6789/0 conn(0x7f37900c5410 legacy=0x7f37901553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:47.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.242+0000 7f3796aad640 1 -- 172.21.15.161:0/4065745042 >> v1:172.21.15.161:6789/0 conn(0x7f3790152890 legacy=0x7f37900c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:47.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.242+0000 7f3796aad640 1 -- 172.21.15.161:0/4065745042 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3790158d10 con 0x7f37900c5850 2024-03-28T23:32:47.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.242+0000 7f3796aad640 1 --2- 172.21.15.161:0/4065745042 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37900c5850 0x7f3790155ab0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3788009dc0 tx=0x7f378802be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71e4e5c78dc13f3d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:47.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.243+0000 7f37952aa640 1 -- 172.21.15.161:0/4065745042 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3788039070 con 0x7f37900c5850 2024-03-28T23:32:47.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.243+0000 7f37952aa640 1 -- 172.21.15.161:0/4065745042 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f378802cc40 con 0x7f37900c5850 2024-03-28T23:32:47.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.243+0000 7f37952aa640 1 -- 172.21.15.161:0/4065745042 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3788034440 con 0x7f37900c5850 2024-03-28T23:32:47.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.243+0000 7f3798510640 1 -- 172.21.15.161:0/4065745042 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37900c5850 msgr2=0x7f3790155ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:47.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.243+0000 7f3798510640 1 --2- 172.21.15.161:0/4065745042 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37900c5850 0x7f3790155ab0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3788009dc0 tx=0x7f378802be90 comp rx=0 tx=0).stop 2024-03-28T23:32:47.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.243+0000 7f3798510640 1 -- 172.21.15.161:0/4065745042 shutdown_connections 2024-03-28T23:32:47.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.243+0000 7f3798510640 1 --2- 172.21.15.161:0/4065745042 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37900c5850 0x7f3790155ab0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.243+0000 7f3798510640 1 -- 172.21.15.161:0/4065745042 >> 172.21.15.161:0/4065745042 conn(0x7f37900ac620 msgr2=0x7f37900aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:47.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.243+0000 7f3798510640 1 -- 172.21.15.161:0/4065745042 shutdown_connections 2024-03-28T23:32:47.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.243+0000 7f3798510640 1 -- 172.21.15.161:0/4065745042 wait complete. 2024-03-28T23:32:47.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.245+0000 7f3798510640 1 Processor -- start 2024-03-28T23:32:47.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.245+0000 7f3798510640 1 -- start start 2024-03-28T23:32:47.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.246+0000 7f3798510640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37900c5410 0x7f37901544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:47.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.246+0000 7f3798510640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37900c5850 0x7f37900c3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:47.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.246+0000 7f3798510640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3790152890 0x7f37900c4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:47.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.246+0000 7f3798510640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f37901592d0 con 0x7f37900c5410 2024-03-28T23:32:47.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.246+0000 7f3798510640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3790159640 con 0x7f3790152890 2024-03-28T23:32:47.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.246+0000 7f3798510640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3790155ff0 con 0x7f37900c5850 2024-03-28T23:32:47.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.246+0000 7f37962ac640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37900c5410 0x7f37901544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:47.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.246+0000 7f37962ac640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37900c5410 0x7f37901544d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39864/0 (socket says 172.21.15.161:39864) 2024-03-28T23:32:47.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.246+0000 7f37962ac640 1 -- 172.21.15.161:0/335980132 learned_addr learned my addr 172.21.15.161:0/335980132 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:47.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.246+0000 7f3795aab640 1 --2- 172.21.15.161:0/335980132 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37900c5850 0x7f37900c3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:47.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.246+0000 7f3796aad640 1 --2- 172.21.15.161:0/335980132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3790152890 0x7f37900c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:47.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.246+0000 7f37962ac640 1 -- 172.21.15.161:0/335980132 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37900c5850 msgr2=0x7f37900c3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:47.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.246+0000 7f37962ac640 1 --2- 172.21.15.161:0/335980132 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37900c5850 0x7f37900c3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.246+0000 7f37962ac640 1 -- 172.21.15.161:0/335980132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3790152890 msgr2=0x7f37900c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:47.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.246+0000 7f37962ac640 1 --2- 172.21.15.161:0/335980132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3790152890 0x7f37900c4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.246+0000 7f37962ac640 1 -- 172.21.15.161:0/335980132 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37900c4950 con 0x7f37900c5410 2024-03-28T23:32:47.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.246+0000 7f37962ac640 1 --2- 172.21.15.161:0/335980132 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37900c5410 0x7f37901544d0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f378c00c410 tx=0x7f378c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:47.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.246+0000 7f3786ffd640 1 -- 172.21.15.161:0/335980132 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f378c017070 con 0x7f37900c5410 2024-03-28T23:32:47.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.246+0000 7f3786ffd640 1 -- 172.21.15.161:0/335980132 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f378c005ce0 con 0x7f37900c5410 2024-03-28T23:32:47.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.247+0000 7f3798510640 1 -- 172.21.15.161:0/335980132 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3790151660 con 0x7f37900c5410 2024-03-28T23:32:47.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.247+0000 7f3786ffd640 1 -- 172.21.15.161:0/335980132 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f378c0072d0 con 0x7f37900c5410 2024-03-28T23:32:47.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.247+0000 7f3798510640 1 -- 172.21.15.161:0/335980132 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37901565b0 con 0x7f37900c5410 2024-03-28T23:32:47.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.248+0000 7f3798510640 1 -- 172.21.15.161:0/335980132 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f379004f4b0 con 0x7f37900c5410 2024-03-28T23:32:47.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.248+0000 7f3786ffd640 1 -- 172.21.15.161:0/335980132 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f378c01a070 con 0x7f37900c5410 2024-03-28T23:32:47.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.248+0000 7f3786ffd640 1 --2- 172.21.15.161:0/335980132 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f377406b4d0 0x7f377406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:47.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.249+0000 7f3786ffd640 1 -- 172.21.15.161:0/335980132 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6957+0+0 (secure 0 0 0) 0x7f378c089990 con 0x7f37900c5410 2024-03-28T23:32:47.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.249+0000 7f3795aab640 1 --2- 172.21.15.161:0/335980132 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f377406b4d0 0x7f377406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:47.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.249+0000 7f3795aab640 1 --2- 172.21.15.161:0/335980132 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f377406b4d0 0x7f377406d990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f37901e8610 tx=0x7f3780009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:47.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.253+0000 7f3786ffd640 1 -- 172.21.15.161:0/335980132 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f378c057da0 con 0x7f37900c5410 2024-03-28T23:32:47.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.469+0000 7f3798510640 1 -- 172.21.15.161:0/335980132 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37900c8140 con 0x7f37900c5410 2024-03-28T23:32:47.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.470+0000 7f3786ffd640 1 -- 172.21.15.161:0/335980132 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+6508 (secure 0 0 0) 0x7f378c05be40 con 0x7f37900c5410 2024-03-28T23:32:47.470 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:47.470 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":59,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/737681942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":737681942},{"type":"v1","addr":"172.21.15.161:6837","nonce":737681942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":5579,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/3501962065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3501962065},{"type":"v1","addr":"172.21.15.170:6834","nonce":3501962065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:32:32.372964+0000","modified":"2024-03-28T23:32:35.321715+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6468},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6468":{"gid":6468,"name":"a","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6836/3850309502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3850309502},{"type":"v1","addr":"172.21.15.161:6836","nonce":3850309502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6468,"qdb_cluster":[6468]},"id":5},{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:32:44.463287+0000","modified":"2024-03-28T23:32:46.605012+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6037":{"gid":6037,"name":"b","rank":0,"incarnation":57,"state":"up:active","state_seq":6,"addr":"172.21.15.170:6835/2691084827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":2691084827},{"type":"v1","addr":"172.21.15.170:6835","nonce":2691084827}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6037,"qdb_cluster":[6037]},"id":6}]} 2024-03-28T23:32:47.471 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 59 2024-03-28T23:32:47.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.472+0000 7f3798510640 1 -- 172.21.15.161:0/335980132 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f377406b4d0 msgr2=0x7f377406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:47.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.472+0000 7f3798510640 1 --2- 172.21.15.161:0/335980132 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f377406b4d0 0x7f377406d990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f37901e8610 tx=0x7f3780009040 comp rx=0 tx=0).stop 2024-03-28T23:32:47.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.472+0000 7f3798510640 1 -- 172.21.15.161:0/335980132 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37900c5410 msgr2=0x7f37901544d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:47.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.472+0000 7f3798510640 1 --2- 172.21.15.161:0/335980132 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37900c5410 0x7f37901544d0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f378c00c410 tx=0x7f378c00c8d0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.472+0000 7f3798510640 1 -- 172.21.15.161:0/335980132 shutdown_connections 2024-03-28T23:32:47.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.472+0000 7f3798510640 1 --2- 172.21.15.161:0/335980132 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f377406b4d0 0x7f377406d990 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.472+0000 7f3798510640 1 --2- 172.21.15.161:0/335980132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3790152890 0x7f37900c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.472+0000 7f3798510640 1 --2- 172.21.15.161:0/335980132 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37900c5850 0x7f37900c3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.472+0000 7f3798510640 1 --2- 172.21.15.161:0/335980132 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37900c5410 0x7f37901544d0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.472+0000 7f3798510640 1 -- 172.21.15.161:0/335980132 >> 172.21.15.161:0/335980132 conn(0x7f37900ac620 msgr2=0x7f37900c9f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:47.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.472+0000 7f3798510640 1 -- 172.21.15.161:0/335980132 shutdown_connections 2024-03-28T23:32:47.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.472+0000 7f3798510640 1 -- 172.21.15.161:0/335980132 wait complete. 2024-03-28T23:32:47.483 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:32:47.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.606+0000 7f445a03b640 1 Processor -- start 2024-03-28T23:32:47.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.606+0000 7f445a03b640 1 -- start start 2024-03-28T23:32:47.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.606+0000 7f445a03b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44540c5410 0x7f44540c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:47.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.606+0000 7f445a03b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f44541553a0 0x7f4454155820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:47.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.606+0000 7f445a03b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4454158a30 con 0x7f4454152890 2024-03-28T23:32:47.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.606+0000 7f445a03b640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f44540c8480 con 0x7f44541553a0 2024-03-28T23:32:47.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.606+0000 7f445a03b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f44540c5d90 con 0x7f44540c5410 2024-03-28T23:32:47.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.606+0000 7f4453fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f44541553a0 0x7f4454155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:47.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.606+0000 7f44537fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f4454152890 0x7f44540c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59352/0 (socket says 172.21.15.161:59352) 2024-03-28T23:32:47.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.606+0000 7f4453fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f44541553a0 0x7f4454155820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39874/0 (socket says 172.21.15.161:39874) 2024-03-28T23:32:47.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.606+0000 7f44537fe640 1 -- 172.21.15.161:0/3935811019 learned_addr learned my addr 172.21.15.161:0/3935811019 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:47.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.607+0000 7f4452ffd640 1 --2- 172.21.15.161:0/3935811019 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44540c5410 0x7f44540c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:47.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.607+0000 7f4453fff640 1 -- 172.21.15.161:0/3935811019 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44540c5410 msgr2=0x7f44540c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:47.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.607+0000 7f4453fff640 1 --2- 172.21.15.161:0/3935811019 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44540c5410 0x7f44540c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.607+0000 7f4453fff640 1 -- 172.21.15.161:0/3935811019 >> v1:172.21.15.161:6789/0 conn(0x7f4454152890 legacy=0x7f44540c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:47.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.607+0000 7f4453fff640 1 -- 172.21.15.161:0/3935811019 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4454158d10 con 0x7f44541553a0 2024-03-28T23:32:47.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.607+0000 7f44527fc640 1 -- 172.21.15.161:0/3935811019 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3932408994 0 0) 0x7f4454158a30 con 0x7f4454152890 2024-03-28T23:32:47.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.607+0000 7f4452ffd640 1 --2- 172.21.15.161:0/3935811019 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44540c5410 0x7f44540c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:32:47.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.608+0000 7f4453fff640 1 --2- 172.21.15.161:0/3935811019 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44541553a0 0x7f4454155820 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f444c0097a0 tx=0x7f444c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91a434952061d301 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:47.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.608+0000 7f44527fc640 1 -- 172.21.15.161:0/3935811019 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444c0315a0 con 0x7f44541553a0 2024-03-28T23:32:47.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.608+0000 7f44527fc640 1 -- 172.21.15.161:0/3935811019 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f444c037070 con 0x7f44541553a0 2024-03-28T23:32:47.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.608+0000 7f44527fc640 1 -- 172.21.15.161:0/3935811019 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444c031960 con 0x7f44541553a0 2024-03-28T23:32:47.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.608+0000 7f445a03b640 1 -- 172.21.15.161:0/3935811019 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44541553a0 msgr2=0x7f4454155820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:47.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.608+0000 7f445a03b640 1 --2- 172.21.15.161:0/3935811019 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44541553a0 0x7f4454155820 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f444c0097a0 tx=0x7f444c02b840 comp rx=0 tx=0).stop 2024-03-28T23:32:47.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.608+0000 7f445a03b640 1 -- 172.21.15.161:0/3935811019 shutdown_connections 2024-03-28T23:32:47.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.608+0000 7f445a03b640 1 --2- 172.21.15.161:0/3935811019 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44541553a0 0x7f4454155820 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.608+0000 7f445a03b640 1 --2- 172.21.15.161:0/3935811019 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44540c5410 0x7f44540c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.608+0000 7f445a03b640 1 -- 172.21.15.161:0/3935811019 >> 172.21.15.161:0/3935811019 conn(0x7f44540ac620 msgr2=0x7f44540aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:47.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.608+0000 7f445a03b640 1 -- 172.21.15.161:0/3935811019 shutdown_connections 2024-03-28T23:32:47.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.608+0000 7f445a03b640 1 -- 172.21.15.161:0/3935811019 wait complete. 2024-03-28T23:32:47.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.609+0000 7f445a03b640 1 Processor -- start 2024-03-28T23:32:47.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.610+0000 7f445a03b640 1 -- start start 2024-03-28T23:32:47.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.610+0000 7f445a03b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44540c5410 0x7f44541ea840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:47.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.610+0000 7f445a03b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4454152890 0x7f445406ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:47.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.610+0000 7f445a03b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44541553a0 0x7f44541ee3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:47.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.610+0000 7f445a03b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f44541592d0 con 0x7f44541553a0 2024-03-28T23:32:47.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.610+0000 7f445a03b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4454159640 con 0x7f44540c5410 2024-03-28T23:32:47.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.610+0000 7f445a03b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f44540c90a0 con 0x7f4454152890 2024-03-28T23:32:47.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.610+0000 7f4453fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44541553a0 0x7f44541ee3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:47.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.610+0000 7f4453fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44541553a0 0x7f44541ee3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39890/0 (socket says 172.21.15.161:39890) 2024-03-28T23:32:47.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.610+0000 7f4453fff640 1 -- 172.21.15.161:0/96054518 learned_addr learned my addr 172.21.15.161:0/96054518 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:47.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.610+0000 7f44537fe640 1 --2- 172.21.15.161:0/96054518 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44540c5410 0x7f44541ea840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:47.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.610+0000 7f4452ffd640 1 --2- 172.21.15.161:0/96054518 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4454152890 0x7f445406ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:47.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.611+0000 7f4453fff640 1 -- 172.21.15.161:0/96054518 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4454152890 msgr2=0x7f445406ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:47.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.611+0000 7f4453fff640 1 --2- 172.21.15.161:0/96054518 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4454152890 0x7f445406ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.611+0000 7f4453fff640 1 -- 172.21.15.161:0/96054518 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44540c5410 msgr2=0x7f44541ea840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:47.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.611+0000 7f4453fff640 1 --2- 172.21.15.161:0/96054518 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44540c5410 0x7f44541ea840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.611+0000 7f4453fff640 1 -- 172.21.15.161:0/96054518 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44541eae90 con 0x7f44541553a0 2024-03-28T23:32:47.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.611+0000 7f4453fff640 1 --2- 172.21.15.161:0/96054518 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44541553a0 0x7f44541ee3b0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f444c034040 tx=0x7f444c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:47.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.611+0000 7f444bfff640 1 -- 172.21.15.161:0/96054518 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444c032630 con 0x7f44541553a0 2024-03-28T23:32:47.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.611+0000 7f444bfff640 1 -- 172.21.15.161:0/96054518 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f444c0327f0 con 0x7f44541553a0 2024-03-28T23:32:47.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.611+0000 7f444bfff640 1 -- 172.21.15.161:0/96054518 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444c032be0 con 0x7f44541553a0 2024-03-28T23:32:47.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.611+0000 7f445a03b640 1 -- 172.21.15.161:0/96054518 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44541eb170 con 0x7f44541553a0 2024-03-28T23:32:47.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.611+0000 7f445a03b640 1 -- 172.21.15.161:0/96054518 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44541eb730 con 0x7f44541553a0 2024-03-28T23:32:47.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.612+0000 7f445a03b640 1 -- 172.21.15.161:0/96054518 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f445404f4b0 con 0x7f44541553a0 2024-03-28T23:32:47.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.613+0000 7f444bfff640 1 -- 172.21.15.161:0/96054518 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f444c03b3f0 con 0x7f44541553a0 2024-03-28T23:32:47.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.613+0000 7f444bfff640 1 --2- 172.21.15.161:0/96054518 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f443406b500 0x7f443406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:47.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.613+0000 7f444bfff640 1 -- 172.21.15.161:0/96054518 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6957+0+0 (secure 0 0 0) 0x7f444c02f030 con 0x7f44541553a0 2024-03-28T23:32:47.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.613+0000 7f44537fe640 1 --2- 172.21.15.161:0/96054518 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f443406b500 0x7f443406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:47.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.613+0000 7f44537fe640 1 --2- 172.21.15.161:0/96054518 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f443406b500 0x7f443406d9c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f44440012e0 tx=0x7f4444001700 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:47.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.617+0000 7f444bfff640 1 -- 172.21.15.161:0/96054518 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f444c040070 con 0x7f44541553a0 2024-03-28T23:32:47.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.814+0000 7f445a03b640 1 -- 172.21.15.161:0/96054518 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f44540c8140 con 0x7f44541553a0 2024-03-28T23:32:47.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.815+0000 7f444bfff640 1 -- 172.21.15.161:0/96054518 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+20705 (secure 0 0 0) 0x7f444c07a630 con 0x7f44541553a0 2024-03-28T23:32:47.816 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:47.816 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":77,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:32:46.111691+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:32:30.684936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-28T23:32:31.339722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:32:42.378609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:32:43.436985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:32:47.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.817+0000 7f445a03b640 1 -- 172.21.15.161:0/96054518 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f443406b500 msgr2=0x7f443406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:47.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.817+0000 7f445a03b640 1 --2- 172.21.15.161:0/96054518 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f443406b500 0x7f443406d9c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f44440012e0 tx=0x7f4444001700 comp rx=0 tx=0).stop 2024-03-28T23:32:47.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.817+0000 7f445a03b640 1 -- 172.21.15.161:0/96054518 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44541553a0 msgr2=0x7f44541ee3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:47.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.817+0000 7f445a03b640 1 --2- 172.21.15.161:0/96054518 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44541553a0 0x7f44541ee3b0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f444c034040 tx=0x7f444c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.817+0000 7f445a03b640 1 -- 172.21.15.161:0/96054518 shutdown_connections 2024-03-28T23:32:47.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.817+0000 7f445a03b640 1 --2- 172.21.15.161:0/96054518 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f443406b500 0x7f443406d9c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.817+0000 7f445a03b640 1 --2- 172.21.15.161:0/96054518 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44541553a0 0x7f44541ee3b0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.817+0000 7f445a03b640 1 --2- 172.21.15.161:0/96054518 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4454152890 0x7f445406ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.817+0000 7f445a03b640 1 --2- 172.21.15.161:0/96054518 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44540c5410 0x7f44541ea840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.817+0000 7f445a03b640 1 -- 172.21.15.161:0/96054518 >> 172.21.15.161:0/96054518 conn(0x7f44540ac620 msgr2=0x7f4454152fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:47.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.817+0000 7f445a03b640 1 -- 172.21.15.161:0/96054518 shutdown_connections 2024-03-28T23:32:47.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.817+0000 7f445a03b640 1 -- 172.21.15.161:0/96054518 wait complete. 2024-03-28T23:32:47.829 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T23:32:47.829 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:32:47.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.931+0000 7f8329f79640 1 Processor -- start 2024-03-28T23:32:47.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.931+0000 7f8329f79640 1 -- start start 2024-03-28T23:32:47.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.931+0000 7f8329f79640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8324152890 0x7f83240c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:47.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.931+0000 7f8329f79640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8324158a30 con 0x7f83240c5b20 2024-03-28T23:32:47.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.931+0000 7f8329f79640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8324158d10 con 0x7f83240c8ed0 2024-03-28T23:32:47.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.931+0000 7f8329f79640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f83240c8480 con 0x7f8324152890 2024-03-28T23:32:47.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.931+0000 7f83237fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8324152890 0x7f83240c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:47.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.931+0000 7f8323fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f83240c5b20 0x7f83241553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59354/0 (socket says 172.21.15.161:59354) 2024-03-28T23:32:47.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.931+0000 7f8323fff640 1 -- 172.21.15.161:0/3478806946 learned_addr learned my addr 172.21.15.161:0/3478806946 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:47.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.932+0000 7f83237fe640 1 -- 172.21.15.161:0/3478806946 >> v1:172.21.15.170:6789/0 conn(0x7f83240c8ed0 legacy=0x7f83240c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:47.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.932+0000 7f83237fe640 1 -- 172.21.15.161:0/3478806946 >> v1:172.21.15.161:6789/0 conn(0x7f83240c5b20 legacy=0x7f83241553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:47.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.932+0000 7f83227fc640 1 -- 172.21.15.161:0/3478806946 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4063868634 0 0) 0x7f8324158d10 con 0x7f83240c8ed0 2024-03-28T23:32:47.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.932+0000 7f83237fe640 1 -- 172.21.15.161:0/3478806946 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8324158ff0 con 0x7f8324152890 2024-03-28T23:32:47.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.933+0000 7f83237fe640 1 --2- 172.21.15.161:0/3478806946 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8324152890 0x7f83240c8990 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f83140097a0 tx=0x7f831402b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef7515ec2213e1fa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:47.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.933+0000 7f83227fc640 1 -- 172.21.15.161:0/3478806946 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8314037070 con 0x7f8324152890 2024-03-28T23:32:47.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.933+0000 7f83227fc640 1 -- 172.21.15.161:0/3478806946 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8314030710 con 0x7f8324152890 2024-03-28T23:32:47.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.933+0000 7f83227fc640 1 -- 172.21.15.161:0/3478806946 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8314030ae0 con 0x7f8324152890 2024-03-28T23:32:47.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.933+0000 7f8329f79640 1 -- 172.21.15.161:0/3478806946 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8324152890 msgr2=0x7f83240c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:47.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.933+0000 7f8329f79640 1 --2- 172.21.15.161:0/3478806946 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8324152890 0x7f83240c8990 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f83140097a0 tx=0x7f831402b680 comp rx=0 tx=0).stop 2024-03-28T23:32:47.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.933+0000 7f8329f79640 1 -- 172.21.15.161:0/3478806946 shutdown_connections 2024-03-28T23:32:47.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.933+0000 7f8329f79640 1 --2- 172.21.15.161:0/3478806946 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8324152890 0x7f83240c8990 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.933+0000 7f8329f79640 1 -- 172.21.15.161:0/3478806946 >> 172.21.15.161:0/3478806946 conn(0x7f83240ac620 msgr2=0x7f83240aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:47.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.933+0000 7f8329f79640 1 -- 172.21.15.161:0/3478806946 shutdown_connections 2024-03-28T23:32:47.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.933+0000 7f8329f79640 1 -- 172.21.15.161:0/3478806946 wait complete. 2024-03-28T23:32:47.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.934+0000 7f8329f79640 1 Processor -- start 2024-03-28T23:32:47.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.935+0000 7f8329f79640 1 -- start start 2024-03-28T23:32:47.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.935+0000 7f8329f79640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83240c5b20 0x7f8324155860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:47.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.935+0000 7f8329f79640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83240c8ed0 0x7f8324149c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:47.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.935+0000 7f8329f79640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8324152890 0x7f832414a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:47.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.935+0000 7f8329f79640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8324152c70 con 0x7f8324152890 2024-03-28T23:32:47.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.935+0000 7f8329f79640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8324159640 con 0x7f83240c5b20 2024-03-28T23:32:47.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.935+0000 7f8329f79640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8324152df0 con 0x7f83240c8ed0 2024-03-28T23:32:47.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.935+0000 7f8323fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8324152890 0x7f832414a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:47.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.935+0000 7f8323fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8324152890 0x7f832414a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39904/0 (socket says 172.21.15.161:39904) 2024-03-28T23:32:47.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.935+0000 7f8323fff640 1 -- 172.21.15.161:0/4241260908 learned_addr learned my addr 172.21.15.161:0/4241260908 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:47.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.935+0000 7f83237fe640 1 --2- 172.21.15.161:0/4241260908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83240c5b20 0x7f8324155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:47.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.935+0000 7f8322ffd640 1 --2- 172.21.15.161:0/4241260908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83240c8ed0 0x7f8324149c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:47.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.936+0000 7f8323fff640 1 -- 172.21.15.161:0/4241260908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83240c8ed0 msgr2=0x7f8324149c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:47.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.936+0000 7f8323fff640 1 --2- 172.21.15.161:0/4241260908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83240c8ed0 0x7f8324149c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.936+0000 7f8323fff640 1 -- 172.21.15.161:0/4241260908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83240c5b20 msgr2=0x7f8324155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:47.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.936+0000 7f8323fff640 1 --2- 172.21.15.161:0/4241260908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83240c5b20 0x7f8324155860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:47.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.936+0000 7f8323fff640 1 -- 172.21.15.161:0/4241260908 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83241eb3a0 con 0x7f8324152890 2024-03-28T23:32:47.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.936+0000 7f8323fff640 1 --2- 172.21.15.161:0/4241260908 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8324152890 0x7f832414a160 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f831c00caf0 tx=0x7f831c002090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:47.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.936+0000 7f830bfff640 1 -- 172.21.15.161:0/4241260908 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831c018070 con 0x7f8324152890 2024-03-28T23:32:47.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.936+0000 7f830bfff640 1 -- 172.21.15.161:0/4241260908 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f831c00a700 con 0x7f8324152890 2024-03-28T23:32:47.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.936+0000 7f8329f79640 1 -- 172.21.15.161:0/4241260908 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8324153cc0 con 0x7f8324152890 2024-03-28T23:32:47.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.936+0000 7f830bfff640 1 -- 172.21.15.161:0/4241260908 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831c00aaf0 con 0x7f8324152890 2024-03-28T23:32:47.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.936+0000 7f8329f79640 1 -- 172.21.15.161:0/4241260908 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8324155da0 con 0x7f8324152890 2024-03-28T23:32:47.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.937+0000 7f8329f79640 1 -- 172.21.15.161:0/4241260908 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f832404f4b0 con 0x7f8324152890 2024-03-28T23:32:47.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.937+0000 7f830bfff640 1 -- 172.21.15.161:0/4241260908 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f831c01b070 con 0x7f8324152890 2024-03-28T23:32:47.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.938+0000 7f830bfff640 1 --2- 172.21.15.161:0/4241260908 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f831006b500 0x7f831006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:47.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.938+0000 7f830bfff640 1 -- 172.21.15.161:0/4241260908 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6957+0+0 (secure 0 0 0) 0x7f831c089cf0 con 0x7f8324152890 2024-03-28T23:32:47.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.938+0000 7f83237fe640 1 --2- 172.21.15.161:0/4241260908 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f831006b500 0x7f831006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:47.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.938+0000 7f83237fe640 1 --2- 172.21.15.161:0/4241260908 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f831006b500 0x7f831006d9c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8314034000 tx=0x7f8314008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:47.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:47.942+0000 7f830bfff640 1 -- 172.21.15.161:0/4241260908 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f831c08e070 con 0x7f8324152890 2024-03-28T23:32:48.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.134+0000 7f8329f79640 1 -- 172.21.15.161:0/4241260908 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f83241eb680 con 0x7f831006b500 2024-03-28T23:32:48.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.145+0000 7f830bfff640 1 -- 172.21.15.161:0/4241260908 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+485419 (secure 0 0 0) 0x7f83241eb680 con 0x7f831006b500 2024-03-28T23:32:48.151 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:48.154 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:32:48.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.150+0000 7f8329f79640 1 -- 172.21.15.161:0/4241260908 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f831006b500 msgr2=0x7f831006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:48.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.150+0000 7f8329f79640 1 --2- 172.21.15.161:0/4241260908 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f831006b500 0x7f831006d9c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8314034000 tx=0x7f8314008030 comp rx=0 tx=0).stop 2024-03-28T23:32:48.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.150+0000 7f8329f79640 1 -- 172.21.15.161:0/4241260908 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8324152890 msgr2=0x7f832414a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:48.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.150+0000 7f8329f79640 1 --2- 172.21.15.161:0/4241260908 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8324152890 0x7f832414a160 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f831c00caf0 tx=0x7f831c002090 comp rx=0 tx=0).stop 2024-03-28T23:32:48.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.150+0000 7f8329f79640 1 -- 172.21.15.161:0/4241260908 shutdown_connections 2024-03-28T23:32:48.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.150+0000 7f8329f79640 1 --2- 172.21.15.161:0/4241260908 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f831006b500 0x7f831006d9c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.150+0000 7f8329f79640 1 --2- 172.21.15.161:0/4241260908 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8324152890 0x7f832414a160 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.150+0000 7f8329f79640 1 --2- 172.21.15.161:0/4241260908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83240c8ed0 0x7f8324149c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.150+0000 7f8329f79640 1 --2- 172.21.15.161:0/4241260908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83240c5b20 0x7f8324155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.150+0000 7f8329f79640 1 -- 172.21.15.161:0/4241260908 >> 172.21.15.161:0/4241260908 conn(0x7f83240ac620 msgr2=0x7f8324150c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:48.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.150+0000 7f8329f79640 1 -- 172.21.15.161:0/4241260908 shutdown_connections 2024-03-28T23:32:48.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.150+0000 7f8329f79640 1 -- 172.21.15.161:0/4241260908 wait complete. 2024-03-28T23:32:48.166 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-03-28T23:32:47.345901+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":202,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240076,"kb_used_data":7276,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754734644,"statfs":{"total":773094113280,"available":772848275456,"internally_reserved":0,"allocated":7450624,"data_stored":4616622,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.247617"},"pg_stats":[{"pgid":"14.3d","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.110397+0000","last_change":"2024-03-28T23:32:44.110397+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.110397+0000","last_undegraded":"2024-03-28T23:32:44.110397+0000","last_fullsized":"2024-03-28T23:32:44.110397+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3f","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108246+0000","last_change":"2024-03-28T23:32:33.046706+0000","last_active":"2024-03-28T23:32:44.108246+0000","last_peered":"2024-03-28T23:32:44.108246+0000","last_clean":"2024-03-28T23:32:44.108246+0000","last_became_active":"2024-03-28T23:32:33.046222+0000","last_became_peered":"2024-03-28T23:32:33.046222+0000","last_unstale":"2024-03-28T23:32:44.108246+0000","last_undegraded":"2024-03-28T23:32:44.108246+0000","last_fullsized":"2024-03-28T23:32:44.108246+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:02:45.190566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.109309+0000","last_change":"2024-03-28T23:32:44.109309+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.109309+0000","last_undegraded":"2024-03-28T23:32:44.109309+0000","last_fullsized":"2024-03-28T23:32:44.109309+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122914+0000","last_change":"2024-03-28T23:32:33.051174+0000","last_active":"2024-03-28T23:32:44.122914+0000","last_peered":"2024-03-28T23:32:44.122914+0000","last_clean":"2024-03-28T23:32:44.122914+0000","last_became_active":"2024-03-28T23:32:33.050345+0000","last_became_peered":"2024-03-28T23:32:33.050345+0000","last_unstale":"2024-03-28T23:32:44.122914+0000","last_undegraded":"2024-03-28T23:32:44.122914+0000","last_fullsized":"2024-03-28T23:32:44.122914+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:41:39.657731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.3f","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111805+0000","last_change":"2024-03-28T23:32:44.111805+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111805+0000","last_undegraded":"2024-03-28T23:32:44.111805+0000","last_fullsized":"2024-03-28T23:32:44.111805+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108168+0000","last_change":"2024-03-28T23:32:33.048785+0000","last_active":"2024-03-28T23:32:44.108168+0000","last_peered":"2024-03-28T23:32:44.108168+0000","last_clean":"2024-03-28T23:32:44.108168+0000","last_became_active":"2024-03-28T23:32:33.048519+0000","last_became_peered":"2024-03-28T23:32:33.048519+0000","last_unstale":"2024-03-28T23:32:44.108168+0000","last_undegraded":"2024-03-28T23:32:44.108168+0000","last_fullsized":"2024-03-28T23:32:44.108168+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:21.464120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111326+0000","last_change":"2024-03-28T23:32:44.111326+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111326+0000","last_undegraded":"2024-03-28T23:32:44.111326+0000","last_fullsized":"2024-03-28T23:32:44.111326+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106875+0000","last_change":"2024-03-28T23:32:33.033535+0000","last_active":"2024-03-28T23:32:44.106875+0000","last_peered":"2024-03-28T23:32:44.106875+0000","last_clean":"2024-03-28T23:32:44.106875+0000","last_became_active":"2024-03-28T23:32:33.033219+0000","last_became_peered":"2024-03-28T23:32:33.033219+0000","last_unstale":"2024-03-28T23:32:44.106875+0000","last_undegraded":"2024-03-28T23:32:44.106875+0000","last_fullsized":"2024-03-28T23:32:44.106875+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:39:39.113457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.106821+0000","last_change":"2024-03-28T23:32:44.106821+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.106821+0000","last_undegraded":"2024-03-28T23:32:44.106821+0000","last_fullsized":"2024-03-28T23:32:44.106821+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109010+0000","last_change":"2024-03-28T23:32:33.042972+0000","last_active":"2024-03-28T23:32:44.109010+0000","last_peered":"2024-03-28T23:32:44.109010+0000","last_clean":"2024-03-28T23:32:44.109010+0000","last_became_active":"2024-03-28T23:32:33.042665+0000","last_became_peered":"2024-03-28T23:32:33.042665+0000","last_unstale":"2024-03-28T23:32:44.109010+0000","last_undegraded":"2024-03-28T23:32:44.109010+0000","last_fullsized":"2024-03-28T23:32:44.109010+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:41:50.190883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.109201+0000","last_change":"2024-03-28T23:32:44.109201+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.109201+0000","last_undegraded":"2024-03-28T23:32:44.109201+0000","last_fullsized":"2024-03-28T23:32:44.109201+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113352+0000","last_change":"2024-03-28T23:32:33.030549+0000","last_active":"2024-03-28T23:32:44.113352+0000","last_peered":"2024-03-28T23:32:44.113352+0000","last_clean":"2024-03-28T23:32:44.113352+0000","last_became_active":"2024-03-28T23:32:33.030299+0000","last_became_peered":"2024-03-28T23:32:33.030299+0000","last_unstale":"2024-03-28T23:32:44.113352+0000","last_undegraded":"2024-03-28T23:32:44.113352+0000","last_fullsized":"2024-03-28T23:32:44.113352+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:35.055035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111404+0000","last_change":"2024-03-28T23:32:44.111404+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111404+0000","last_undegraded":"2024-03-28T23:32:44.111404+0000","last_fullsized":"2024-03-28T23:32:44.111404+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.111244+0000","last_change":"2024-03-28T23:32:33.044477+0000","last_active":"2024-03-28T23:32:44.111244+0000","last_peered":"2024-03-28T23:32:44.111244+0000","last_clean":"2024-03-28T23:32:44.111244+0000","last_became_active":"2024-03-28T23:32:33.043960+0000","last_became_peered":"2024-03-28T23:32:33.043960+0000","last_unstale":"2024-03-28T23:32:44.111244+0000","last_undegraded":"2024-03-28T23:32:44.111244+0000","last_fullsized":"2024-03-28T23:32:44.111244+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:21.464120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.113215+0000","last_change":"2024-03-28T23:32:44.113215+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.113215+0000","last_undegraded":"2024-03-28T23:32:44.113215+0000","last_fullsized":"2024-03-28T23:32:44.113215+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108736+0000","last_change":"2024-03-28T23:32:33.052803+0000","last_active":"2024-03-28T23:32:44.108736+0000","last_peered":"2024-03-28T23:32:44.108736+0000","last_clean":"2024-03-28T23:32:44.108736+0000","last_became_active":"2024-03-28T23:32:33.052524+0000","last_became_peered":"2024-03-28T23:32:33.052524+0000","last_unstale":"2024-03-28T23:32:44.108736+0000","last_undegraded":"2024-03-28T23:32:44.108736+0000","last_fullsized":"2024-03-28T23:32:44.108736+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:19:09.500642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.117756+0000","last_change":"2024-03-28T23:32:44.117756+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.117756+0000","last_undegraded":"2024-03-28T23:32:44.117756+0000","last_fullsized":"2024-03-28T23:32:44.117756+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112756+0000","last_change":"2024-03-28T23:32:33.042587+0000","last_active":"2024-03-28T23:32:44.112756+0000","last_peered":"2024-03-28T23:32:44.112756+0000","last_clean":"2024-03-28T23:32:44.112756+0000","last_became_active":"2024-03-28T23:32:33.037002+0000","last_became_peered":"2024-03-28T23:32:33.037002+0000","last_unstale":"2024-03-28T23:32:44.112756+0000","last_undegraded":"2024-03-28T23:32:44.112756+0000","last_fullsized":"2024-03-28T23:32:44.112756+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:36:03.951054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107890+0000","last_change":"2024-03-28T23:32:44.107890+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107890+0000","last_undegraded":"2024-03-28T23:32:44.107890+0000","last_fullsized":"2024-03-28T23:32:44.107890+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107498+0000","last_change":"2024-03-28T23:32:33.039941+0000","last_active":"2024-03-28T23:32:44.107498+0000","last_peered":"2024-03-28T23:32:44.107498+0000","last_clean":"2024-03-28T23:32:44.107498+0000","last_became_active":"2024-03-28T23:32:33.039656+0000","last_became_peered":"2024-03-28T23:32:33.039656+0000","last_unstale":"2024-03-28T23:32:44.107498+0000","last_undegraded":"2024-03-28T23:32:44.107498+0000","last_fullsized":"2024-03-28T23:32:44.107498+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:16.850667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111858+0000","last_change":"2024-03-28T23:32:44.111858+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111858+0000","last_undegraded":"2024-03-28T23:32:44.111858+0000","last_fullsized":"2024-03-28T23:32:44.111858+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122690+0000","last_change":"2024-03-28T23:32:33.052329+0000","last_active":"2024-03-28T23:32:44.122690+0000","last_peered":"2024-03-28T23:32:44.122690+0000","last_clean":"2024-03-28T23:32:44.122690+0000","last_became_active":"2024-03-28T23:32:33.051833+0000","last_became_peered":"2024-03-28T23:32:33.051833+0000","last_unstale":"2024-03-28T23:32:44.122690+0000","last_undegraded":"2024-03-28T23:32:44.122690+0000","last_fullsized":"2024-03-28T23:32:44.122690+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:05.929331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108222+0000","last_change":"2024-03-28T23:32:44.108222+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108222+0000","last_undegraded":"2024-03-28T23:32:44.108222+0000","last_fullsized":"2024-03-28T23:32:44.108222+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106984+0000","last_change":"2024-03-28T23:32:33.042337+0000","last_active":"2024-03-28T23:32:44.106984+0000","last_peered":"2024-03-28T23:32:44.106984+0000","last_clean":"2024-03-28T23:32:44.106984+0000","last_became_active":"2024-03-28T23:32:33.042062+0000","last_became_peered":"2024-03-28T23:32:33.042062+0000","last_unstale":"2024-03-28T23:32:44.106984+0000","last_undegraded":"2024-03-28T23:32:44.106984+0000","last_fullsized":"2024-03-28T23:32:44.106984+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:13:34.921740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108039+0000","last_change":"2024-03-28T23:32:44.108039+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108039+0000","last_undegraded":"2024-03-28T23:32:44.108039+0000","last_fullsized":"2024-03-28T23:32:44.108039+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109102+0000","last_change":"2024-03-28T23:32:33.046685+0000","last_active":"2024-03-28T23:32:44.109102+0000","last_peered":"2024-03-28T23:32:44.109102+0000","last_clean":"2024-03-28T23:32:44.109102+0000","last_became_active":"2024-03-28T23:32:33.046206+0000","last_became_peered":"2024-03-28T23:32:33.046206+0000","last_unstale":"2024-03-28T23:32:44.109102+0000","last_undegraded":"2024-03-28T23:32:44.109102+0000","last_fullsized":"2024-03-28T23:32:44.109102+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:43:22.001147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.113196+0000","last_change":"2024-03-28T23:32:44.113196+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.113196+0000","last_undegraded":"2024-03-28T23:32:44.113196+0000","last_fullsized":"2024-03-28T23:32:44.113196+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113445+0000","last_change":"2024-03-28T23:32:33.029880+0000","last_active":"2024-03-28T23:32:44.113445+0000","last_peered":"2024-03-28T23:32:44.113445+0000","last_clean":"2024-03-28T23:32:44.113445+0000","last_became_active":"2024-03-28T23:32:33.029582+0000","last_became_peered":"2024-03-28T23:32:33.029582+0000","last_unstale":"2024-03-28T23:32:44.113445+0000","last_undegraded":"2024-03-28T23:32:44.113445+0000","last_fullsized":"2024-03-28T23:32:44.113445+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:38:39.283932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111833+0000","last_change":"2024-03-28T23:32:44.111833+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111833+0000","last_undegraded":"2024-03-28T23:32:44.111833+0000","last_fullsized":"2024-03-28T23:32:44.111833+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121362+0000","last_change":"2024-03-28T23:32:33.040102+0000","last_active":"2024-03-28T23:32:44.121362+0000","last_peered":"2024-03-28T23:32:44.121362+0000","last_clean":"2024-03-28T23:32:44.121362+0000","last_became_active":"2024-03-28T23:32:33.039660+0000","last_became_peered":"2024-03-28T23:32:33.039660+0000","last_unstale":"2024-03-28T23:32:44.121362+0000","last_undegraded":"2024-03-28T23:32:44.121362+0000","last_fullsized":"2024-03-28T23:32:44.121362+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:35.055035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112037+0000","last_change":"2024-03-28T23:32:44.112037+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112037+0000","last_undegraded":"2024-03-28T23:32:44.112037+0000","last_fullsized":"2024-03-28T23:32:44.112037+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108875+0000","last_change":"2024-03-28T23:32:33.035442+0000","last_active":"2024-03-28T23:32:44.108875+0000","last_peered":"2024-03-28T23:32:44.108875+0000","last_clean":"2024-03-28T23:32:44.108875+0000","last_became_active":"2024-03-28T23:32:33.035154+0000","last_became_peered":"2024-03-28T23:32:33.035154+0000","last_unstale":"2024-03-28T23:32:44.108875+0000","last_undegraded":"2024-03-28T23:32:44.108875+0000","last_fullsized":"2024-03-28T23:32:44.108875+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:34:22.775718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.115970+0000","last_change":"2024-03-28T23:32:44.115970+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.115970+0000","last_undegraded":"2024-03-28T23:32:44.115970+0000","last_fullsized":"2024-03-28T23:32:44.115970+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.111211+0000","last_change":"2024-03-28T23:32:33.047782+0000","last_active":"2024-03-28T23:32:44.111211+0000","last_peered":"2024-03-28T23:32:44.111211+0000","last_clean":"2024-03-28T23:32:44.111211+0000","last_became_active":"2024-03-28T23:32:33.047254+0000","last_became_peered":"2024-03-28T23:32:33.047254+0000","last_unstale":"2024-03-28T23:32:44.111211+0000","last_undegraded":"2024-03-28T23:32:44.111211+0000","last_fullsized":"2024-03-28T23:32:44.111211+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:19.004555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108940+0000","last_change":"2024-03-28T23:32:44.108940+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108940+0000","last_undegraded":"2024-03-28T23:32:44.108940+0000","last_fullsized":"2024-03-28T23:32:44.108940+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107847+0000","last_change":"2024-03-28T23:32:33.037321+0000","last_active":"2024-03-28T23:32:44.107847+0000","last_peered":"2024-03-28T23:32:44.107847+0000","last_clean":"2024-03-28T23:32:44.107847+0000","last_became_active":"2024-03-28T23:32:33.036961+0000","last_became_peered":"2024-03-28T23:32:33.036961+0000","last_unstale":"2024-03-28T23:32:44.107847+0000","last_undegraded":"2024-03-28T23:32:44.107847+0000","last_fullsized":"2024-03-28T23:32:44.107847+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:25.849919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.120323+0000","last_change":"2024-03-28T23:32:44.120323+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.120323+0000","last_undegraded":"2024-03-28T23:32:44.120323+0000","last_fullsized":"2024-03-28T23:32:44.120323+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113594+0000","last_change":"2024-03-28T23:32:33.053071+0000","last_active":"2024-03-28T23:32:44.113594+0000","last_peered":"2024-03-28T23:32:44.113594+0000","last_clean":"2024-03-28T23:32:44.113594+0000","last_became_active":"2024-03-28T23:32:33.052784+0000","last_became_peered":"2024-03-28T23:32:33.052784+0000","last_unstale":"2024-03-28T23:32:44.113594+0000","last_undegraded":"2024-03-28T23:32:44.113594+0000","last_fullsized":"2024-03-28T23:32:44.113594+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:58:48.231248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112002+0000","last_change":"2024-03-28T23:32:44.112002+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112002+0000","last_undegraded":"2024-03-28T23:32:44.112002+0000","last_fullsized":"2024-03-28T23:32:44.112002+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109678+0000","last_change":"2024-03-28T23:32:33.052720+0000","last_active":"2024-03-28T23:32:44.109678+0000","last_peered":"2024-03-28T23:32:44.109678+0000","last_clean":"2024-03-28T23:32:44.109678+0000","last_became_active":"2024-03-28T23:32:33.052460+0000","last_became_peered":"2024-03-28T23:32:33.052460+0000","last_unstale":"2024-03-28T23:32:44.109678+0000","last_undegraded":"2024-03-28T23:32:44.109678+0000","last_fullsized":"2024-03-28T23:32:44.109678+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:30.891315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107873+0000","last_change":"2024-03-28T23:32:44.107873+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107873+0000","last_undegraded":"2024-03-28T23:32:44.107873+0000","last_fullsized":"2024-03-28T23:32:44.107873+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112648+0000","last_change":"2024-03-28T23:32:33.035666+0000","last_active":"2024-03-28T23:32:44.112648+0000","last_peered":"2024-03-28T23:32:44.112648+0000","last_clean":"2024-03-28T23:32:44.112648+0000","last_became_active":"2024-03-28T23:32:33.035403+0000","last_became_peered":"2024-03-28T23:32:33.035403+0000","last_unstale":"2024-03-28T23:32:44.112648+0000","last_undegraded":"2024-03-28T23:32:44.112648+0000","last_fullsized":"2024-03-28T23:32:44.112648+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:40:51.269760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111443+0000","last_change":"2024-03-28T23:32:44.111443+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111443+0000","last_undegraded":"2024-03-28T23:32:44.111443+0000","last_fullsized":"2024-03-28T23:32:44.111443+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109244+0000","last_change":"2024-03-28T23:32:33.047992+0000","last_active":"2024-03-28T23:32:44.109244+0000","last_peered":"2024-03-28T23:32:44.109244+0000","last_clean":"2024-03-28T23:32:44.109244+0000","last_became_active":"2024-03-28T23:32:33.047058+0000","last_became_peered":"2024-03-28T23:32:33.047058+0000","last_unstale":"2024-03-28T23:32:44.109244+0000","last_undegraded":"2024-03-28T23:32:44.109244+0000","last_fullsized":"2024-03-28T23:32:44.109244+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:35:30.886935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.110595+0000","last_change":"2024-03-28T23:32:44.110595+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.110595+0000","last_undegraded":"2024-03-28T23:32:44.110595+0000","last_fullsized":"2024-03-28T23:32:44.110595+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112366+0000","last_change":"2024-03-28T23:32:33.029536+0000","last_active":"2024-03-28T23:32:44.112366+0000","last_peered":"2024-03-28T23:32:44.112366+0000","last_clean":"2024-03-28T23:32:44.112366+0000","last_became_active":"2024-03-28T23:32:33.029265+0000","last_became_peered":"2024-03-28T23:32:33.029265+0000","last_unstale":"2024-03-28T23:32:44.112366+0000","last_undegraded":"2024-03-28T23:32:44.112366+0000","last_fullsized":"2024-03-28T23:32:44.112366+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:51:11.447507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.110858+0000","last_change":"2024-03-28T23:32:44.110858+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.110858+0000","last_undegraded":"2024-03-28T23:32:44.110858+0000","last_fullsized":"2024-03-28T23:32:44.110858+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121488+0000","last_change":"2024-03-28T23:32:33.041328+0000","last_active":"2024-03-28T23:32:44.121488+0000","last_peered":"2024-03-28T23:32:44.121488+0000","last_clean":"2024-03-28T23:32:44.121488+0000","last_became_active":"2024-03-28T23:32:33.041071+0000","last_became_peered":"2024-03-28T23:32:33.041071+0000","last_unstale":"2024-03-28T23:32:44.121488+0000","last_undegraded":"2024-03-28T23:32:44.121488+0000","last_fullsized":"2024-03-28T23:32:44.121488+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:00.358180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107392+0000","last_change":"2024-03-28T23:32:44.107392+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107392+0000","last_undegraded":"2024-03-28T23:32:44.107392+0000","last_fullsized":"2024-03-28T23:32:44.107392+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122766+0000","last_change":"2024-03-28T23:32:33.043840+0000","last_active":"2024-03-28T23:32:44.122766+0000","last_peered":"2024-03-28T23:32:44.122766+0000","last_clean":"2024-03-28T23:32:44.122766+0000","last_became_active":"2024-03-28T23:32:33.043569+0000","last_became_peered":"2024-03-28T23:32:33.043569+0000","last_unstale":"2024-03-28T23:32:44.122766+0000","last_undegraded":"2024-03-28T23:32:44.122766+0000","last_fullsized":"2024-03-28T23:32:44.122766+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:28.463493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108552+0000","last_change":"2024-03-28T23:32:44.108552+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108552+0000","last_undegraded":"2024-03-28T23:32:44.108552+0000","last_fullsized":"2024-03-28T23:32:44.108552+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107631+0000","last_change":"2024-03-28T23:32:33.047594+0000","last_active":"2024-03-28T23:32:44.107631+0000","last_peered":"2024-03-28T23:32:44.107631+0000","last_clean":"2024-03-28T23:32:44.107631+0000","last_became_active":"2024-03-28T23:32:33.047031+0000","last_became_peered":"2024-03-28T23:32:33.047031+0000","last_unstale":"2024-03-28T23:32:44.107631+0000","last_undegraded":"2024-03-28T23:32:44.107631+0000","last_fullsized":"2024-03-28T23:32:44.107631+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:30:12.769818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112271+0000","last_change":"2024-03-28T23:32:44.112271+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112271+0000","last_undegraded":"2024-03-28T23:32:44.112271+0000","last_fullsized":"2024-03-28T23:32:44.112271+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107919+0000","last_change":"2024-03-28T23:32:33.043396+0000","last_active":"2024-03-28T23:32:44.107919+0000","last_peered":"2024-03-28T23:32:44.107919+0000","last_clean":"2024-03-28T23:32:44.107919+0000","last_became_active":"2024-03-28T23:32:33.043123+0000","last_became_peered":"2024-03-28T23:32:33.043123+0000","last_unstale":"2024-03-28T23:32:44.107919+0000","last_undegraded":"2024-03-28T23:32:44.107919+0000","last_fullsized":"2024-03-28T23:32:44.107919+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:28:36.535909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.114282+0000","last_change":"2024-03-28T23:32:44.114282+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.114282+0000","last_undegraded":"2024-03-28T23:32:44.114282+0000","last_fullsized":"2024-03-28T23:32:44.114282+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109350+0000","last_change":"2024-03-28T23:32:33.042288+0000","last_active":"2024-03-28T23:32:44.109350+0000","last_peered":"2024-03-28T23:32:44.109350+0000","last_clean":"2024-03-28T23:32:44.109350+0000","last_became_active":"2024-03-28T23:32:33.042007+0000","last_became_peered":"2024-03-28T23:32:33.042007+0000","last_unstale":"2024-03-28T23:32:44.109350+0000","last_undegraded":"2024-03-28T23:32:44.109350+0000","last_fullsized":"2024-03-28T23:32:44.109350+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:16:12.634582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.110873+0000","last_change":"2024-03-28T23:32:44.110873+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.110873+0000","last_undegraded":"2024-03-28T23:32:44.110873+0000","last_fullsized":"2024-03-28T23:32:44.110873+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122615+0000","last_change":"2024-03-28T23:32:33.041336+0000","last_active":"2024-03-28T23:32:44.122615+0000","last_peered":"2024-03-28T23:32:44.122615+0000","last_clean":"2024-03-28T23:32:44.122615+0000","last_became_active":"2024-03-28T23:32:33.040865+0000","last_became_peered":"2024-03-28T23:32:33.040865+0000","last_unstale":"2024-03-28T23:32:44.122615+0000","last_undegraded":"2024-03-28T23:32:44.122615+0000","last_fullsized":"2024-03-28T23:32:44.122615+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:44:58.784172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.117069+0000","last_change":"2024-03-28T23:32:44.117069+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.117069+0000","last_undegraded":"2024-03-28T23:32:44.117069+0000","last_fullsized":"2024-03-28T23:32:44.117069+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106185+0000","last_change":"2024-03-28T23:32:33.049222+0000","last_active":"2024-03-28T23:32:44.106185+0000","last_peered":"2024-03-28T23:32:44.106185+0000","last_clean":"2024-03-28T23:32:44.106185+0000","last_became_active":"2024-03-28T23:32:33.048933+0000","last_became_peered":"2024-03-28T23:32:33.048933+0000","last_unstale":"2024-03-28T23:32:44.106185+0000","last_undegraded":"2024-03-28T23:32:44.106185+0000","last_fullsized":"2024-03-28T23:32:44.106185+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:17:45.545141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107516+0000","last_change":"2024-03-28T23:32:44.107516+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107516+0000","last_undegraded":"2024-03-28T23:32:44.107516+0000","last_fullsized":"2024-03-28T23:32:44.107516+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108405+0000","last_change":"2024-03-28T23:32:33.047977+0000","last_active":"2024-03-28T23:32:44.108405+0000","last_peered":"2024-03-28T23:32:44.108405+0000","last_clean":"2024-03-28T23:32:44.108405+0000","last_became_active":"2024-03-28T23:32:33.047699+0000","last_became_peered":"2024-03-28T23:32:33.047699+0000","last_unstale":"2024-03-28T23:32:44.108405+0000","last_undegraded":"2024-03-28T23:32:44.108405+0000","last_fullsized":"2024-03-28T23:32:44.108405+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:38:51.513907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111342+0000","last_change":"2024-03-28T23:32:44.111342+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111342+0000","last_undegraded":"2024-03-28T23:32:44.111342+0000","last_fullsized":"2024-03-28T23:32:44.111342+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108988+0000","last_change":"2024-03-28T23:32:33.036527+0000","last_active":"2024-03-28T23:32:44.108988+0000","last_peered":"2024-03-28T23:32:44.108988+0000","last_clean":"2024-03-28T23:32:44.108988+0000","last_became_active":"2024-03-28T23:32:33.036269+0000","last_became_peered":"2024-03-28T23:32:33.036269+0000","last_unstale":"2024-03-28T23:32:44.108988+0000","last_undegraded":"2024-03-28T23:32:44.108988+0000","last_fullsized":"2024-03-28T23:32:44.108988+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:43:35.946992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112540+0000","last_change":"2024-03-28T23:32:44.112540+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112540+0000","last_undegraded":"2024-03-28T23:32:44.112540+0000","last_fullsized":"2024-03-28T23:32:44.112540+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121867+0000","last_change":"2024-03-28T23:32:44.118134+0000","last_active":"2024-03-28T23:32:44.121867+0000","last_peered":"2024-03-28T23:32:44.121867+0000","last_clean":"2024-03-28T23:32:44.121867+0000","last_became_active":"2024-03-28T23:32:44.117870+0000","last_became_peered":"2024-03-28T23:32:44.117870+0000","last_unstale":"2024-03-28T23:32:44.121867+0000","last_undegraded":"2024-03-28T23:32:44.121867+0000","last_fullsized":"2024-03-28T23:32:44.121867+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:41:04.213333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109389+0000","last_change":"2024-03-28T23:32:33.038016+0000","last_active":"2024-03-28T23:32:44.109389+0000","last_peered":"2024-03-28T23:32:44.109389+0000","last_clean":"2024-03-28T23:32:44.109389+0000","last_became_active":"2024-03-28T23:32:33.037754+0000","last_became_peered":"2024-03-28T23:32:33.037754+0000","last_unstale":"2024-03-28T23:32:44.109389+0000","last_undegraded":"2024-03-28T23:32:44.109389+0000","last_fullsized":"2024-03-28T23:32:44.109389+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:21.878156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"72'4","reported_seq":22,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.111297+0000","last_change":"2024-03-28T23:32:32.040468+0000","last_active":"2024-03-28T23:32:44.111297+0000","last_peered":"2024-03-28T23:32:44.111297+0000","last_clean":"2024-03-28T23:32:44.111297+0000","last_became_active":"2024-03-28T23:32:32.039342+0000","last_became_peered":"2024-03-28T23:32:32.039342+0000","last_unstale":"2024-03-28T23:32:44.111297+0000","last_undegraded":"2024-03-28T23:32:44.111297+0000","last_fullsized":"2024-03-28T23:32:44.111297+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:05.786248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108504+0000","last_change":"2024-03-28T23:32:44.108504+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108504+0000","last_undegraded":"2024-03-28T23:32:44.108504+0000","last_fullsized":"2024-03-28T23:32:44.108504+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.127347+0000","last_change":"2024-03-28T23:32:44.126084+0000","last_active":"2024-03-28T23:32:44.127347+0000","last_peered":"2024-03-28T23:32:44.127347+0000","last_clean":"2024-03-28T23:32:44.127347+0000","last_became_active":"2024-03-28T23:32:44.125728+0000","last_became_peered":"2024-03-28T23:32:44.125728+0000","last_unstale":"2024-03-28T23:32:44.127347+0000","last_undegraded":"2024-03-28T23:32:44.127347+0000","last_fullsized":"2024-03-28T23:32:44.127347+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:17.008056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121405+0000","last_change":"2024-03-28T23:32:33.039778+0000","last_active":"2024-03-28T23:32:44.121405+0000","last_peered":"2024-03-28T23:32:44.121405+0000","last_clean":"2024-03-28T23:32:44.121405+0000","last_became_active":"2024-03-28T23:32:33.039506+0000","last_became_peered":"2024-03-28T23:32:33.039506+0000","last_unstale":"2024-03-28T23:32:44.121405+0000","last_undegraded":"2024-03-28T23:32:44.121405+0000","last_fullsized":"2024-03-28T23:32:44.121405+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:38:39.283932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"72'1","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107225+0000","last_change":"2024-03-28T23:32:32.041093+0000","last_active":"2024-03-28T23:32:44.107225+0000","last_peered":"2024-03-28T23:32:44.107225+0000","last_clean":"2024-03-28T23:32:44.107225+0000","last_became_active":"2024-03-28T23:32:32.040827+0000","last_became_peered":"2024-03-28T23:32:32.040827+0000","last_unstale":"2024-03-28T23:32:44.107225+0000","last_undegraded":"2024-03-28T23:32:44.107225+0000","last_fullsized":"2024-03-28T23:32:44.107225+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:55.544945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108669+0000","last_change":"2024-03-28T23:32:44.108669+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108669+0000","last_undegraded":"2024-03-28T23:32:44.108669+0000","last_fullsized":"2024-03-28T23:32:44.108669+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.120787+0000","last_change":"2024-03-28T23:32:44.119556+0000","last_active":"2024-03-28T23:32:44.120787+0000","last_peered":"2024-03-28T23:32:44.120787+0000","last_clean":"2024-03-28T23:32:44.120787+0000","last_became_active":"2024-03-28T23:32:44.119246+0000","last_became_peered":"2024-03-28T23:32:44.119246+0000","last_unstale":"2024-03-28T23:32:44.120787+0000","last_undegraded":"2024-03-28T23:32:44.120787+0000","last_fullsized":"2024-03-28T23:32:44.120787+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:36.928644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112355+0000","last_change":"2024-03-28T23:32:33.044887+0000","last_active":"2024-03-28T23:32:44.112355+0000","last_peered":"2024-03-28T23:32:44.112355+0000","last_clean":"2024-03-28T23:32:44.112355+0000","last_became_active":"2024-03-28T23:32:33.042005+0000","last_became_peered":"2024-03-28T23:32:33.042005+0000","last_unstale":"2024-03-28T23:32:44.112355+0000","last_undegraded":"2024-03-28T23:32:44.112355+0000","last_fullsized":"2024-03-28T23:32:44.112355+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:41.257241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"72'2","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108029+0000","last_change":"2024-03-28T23:32:32.040527+0000","last_active":"2024-03-28T23:32:44.108029+0000","last_peered":"2024-03-28T23:32:44.108029+0000","last_clean":"2024-03-28T23:32:44.108029+0000","last_became_active":"2024-03-28T23:32:32.040142+0000","last_became_peered":"2024-03-28T23:32:32.040142+0000","last_unstale":"2024-03-28T23:32:44.108029+0000","last_undegraded":"2024-03-28T23:32:44.108029+0000","last_fullsized":"2024-03-28T23:32:44.108029+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:13:15.856750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107395+0000","last_change":"2024-03-28T23:32:44.107395+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107395+0000","last_undegraded":"2024-03-28T23:32:44.107395+0000","last_fullsized":"2024-03-28T23:32:44.107395+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122155+0000","last_change":"2024-03-28T23:32:44.116203+0000","last_active":"2024-03-28T23:32:44.122155+0000","last_peered":"2024-03-28T23:32:44.122155+0000","last_clean":"2024-03-28T23:32:44.122155+0000","last_became_active":"2024-03-28T23:32:44.115846+0000","last_became_peered":"2024-03-28T23:32:44.115846+0000","last_unstale":"2024-03-28T23:32:44.122155+0000","last_undegraded":"2024-03-28T23:32:44.122155+0000","last_fullsized":"2024-03-28T23:32:44.122155+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:58.429743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112741+0000","last_change":"2024-03-28T23:32:33.040988+0000","last_active":"2024-03-28T23:32:44.112741+0000","last_peered":"2024-03-28T23:32:44.112741+0000","last_clean":"2024-03-28T23:32:44.112741+0000","last_became_active":"2024-03-28T23:32:33.036045+0000","last_became_peered":"2024-03-28T23:32:33.036045+0000","last_unstale":"2024-03-28T23:32:44.112741+0000","last_undegraded":"2024-03-28T23:32:44.112741+0000","last_fullsized":"2024-03-28T23:32:44.112741+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:10:39.981362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"72'2","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112781+0000","last_change":"2024-03-28T23:32:32.036856+0000","last_active":"2024-03-28T23:32:44.112781+0000","last_peered":"2024-03-28T23:32:44.112781+0000","last_clean":"2024-03-28T23:32:44.112781+0000","last_became_active":"2024-03-28T23:32:32.036565+0000","last_became_peered":"2024-03-28T23:32:32.036565+0000","last_unstale":"2024-03-28T23:32:44.112781+0000","last_undegraded":"2024-03-28T23:32:44.112781+0000","last_fullsized":"2024-03-28T23:32:44.112781+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:11:01.106436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.106357+0000","last_change":"2024-03-28T23:32:44.106357+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.106357+0000","last_undegraded":"2024-03-28T23:32:44.106357+0000","last_fullsized":"2024-03-28T23:32:44.106357+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.118200+0000","last_change":"2024-03-28T23:32:44.116097+0000","last_active":"2024-03-28T23:32:44.118200+0000","last_peered":"2024-03-28T23:32:44.118200+0000","last_clean":"2024-03-28T23:32:44.118200+0000","last_became_active":"2024-03-28T23:32:44.115810+0000","last_became_peered":"2024-03-28T23:32:44.115810+0000","last_unstale":"2024-03-28T23:32:44.118200+0000","last_undegraded":"2024-03-28T23:32:44.118200+0000","last_fullsized":"2024-03-28T23:32:44.118200+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:45:09.862897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122437+0000","last_change":"2024-03-28T23:32:33.047911+0000","last_active":"2024-03-28T23:32:44.122437+0000","last_peered":"2024-03-28T23:32:44.122437+0000","last_clean":"2024-03-28T23:32:44.122437+0000","last_became_active":"2024-03-28T23:32:33.047616+0000","last_became_peered":"2024-03-28T23:32:33.047616+0000","last_unstale":"2024-03-28T23:32:44.122437+0000","last_undegraded":"2024-03-28T23:32:44.122437+0000","last_fullsized":"2024-03-28T23:32:44.122437+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:58:48.231248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"72'1","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112850+0000","last_change":"2024-03-28T23:32:32.036829+0000","last_active":"2024-03-28T23:32:44.112850+0000","last_peered":"2024-03-28T23:32:44.112850+0000","last_clean":"2024-03-28T23:32:44.112850+0000","last_became_active":"2024-03-28T23:32:32.036571+0000","last_became_peered":"2024-03-28T23:32:32.036571+0000","last_unstale":"2024-03-28T23:32:44.112850+0000","last_undegraded":"2024-03-28T23:32:44.112850+0000","last_fullsized":"2024-03-28T23:32:44.112850+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:32:03.789593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.116621+0000","last_change":"2024-03-28T23:32:44.116621+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.116621+0000","last_undegraded":"2024-03-28T23:32:44.116621+0000","last_fullsized":"2024-03-28T23:32:44.116621+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.120399+0000","last_change":"2024-03-28T23:32:44.117981+0000","last_active":"2024-03-28T23:32:44.120399+0000","last_peered":"2024-03-28T23:32:44.120399+0000","last_clean":"2024-03-28T23:32:44.120399+0000","last_became_active":"2024-03-28T23:32:44.117709+0000","last_became_peered":"2024-03-28T23:32:44.117709+0000","last_unstale":"2024-03-28T23:32:44.120399+0000","last_undegraded":"2024-03-28T23:32:44.120399+0000","last_fullsized":"2024-03-28T23:32:44.120399+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:13:27.942873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122698+0000","last_change":"2024-03-28T23:32:33.052330+0000","last_active":"2024-03-28T23:32:44.122698+0000","last_peered":"2024-03-28T23:32:44.122698+0000","last_clean":"2024-03-28T23:32:44.122698+0000","last_became_active":"2024-03-28T23:32:33.051828+0000","last_became_peered":"2024-03-28T23:32:33.051828+0000","last_unstale":"2024-03-28T23:32:44.122698+0000","last_undegraded":"2024-03-28T23:32:44.122698+0000","last_fullsized":"2024-03-28T23:32:44.122698+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:40:53.134608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"72'1","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106048+0000","last_change":"2024-03-28T23:32:32.036739+0000","last_active":"2024-03-28T23:32:44.106048+0000","last_peered":"2024-03-28T23:32:44.106048+0000","last_clean":"2024-03-28T23:32:44.106048+0000","last_became_active":"2024-03-28T23:32:32.036471+0000","last_became_peered":"2024-03-28T23:32:32.036471+0000","last_unstale":"2024-03-28T23:32:44.106048+0000","last_undegraded":"2024-03-28T23:32:44.106048+0000","last_fullsized":"2024-03-28T23:32:44.106048+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:48.704351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112770+0000","last_change":"2024-03-28T23:32:44.112770+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112770+0000","last_undegraded":"2024-03-28T23:32:44.112770+0000","last_fullsized":"2024-03-28T23:32:44.112770+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.127627+0000","last_change":"2024-03-28T23:32:44.126818+0000","last_active":"2024-03-28T23:32:44.127627+0000","last_peered":"2024-03-28T23:32:44.127627+0000","last_clean":"2024-03-28T23:32:44.127627+0000","last_became_active":"2024-03-28T23:32:44.126493+0000","last_became_peered":"2024-03-28T23:32:44.126493+0000","last_unstale":"2024-03-28T23:32:44.127627+0000","last_undegraded":"2024-03-28T23:32:44.127627+0000","last_fullsized":"2024-03-28T23:32:44.127627+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:06:32.695104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.115074+0000","last_change":"2024-03-28T23:32:33.034126+0000","last_active":"2024-03-28T23:32:44.115074+0000","last_peered":"2024-03-28T23:32:44.115074+0000","last_clean":"2024-03-28T23:32:44.115074+0000","last_became_active":"2024-03-28T23:32:33.033703+0000","last_became_peered":"2024-03-28T23:32:33.033703+0000","last_unstale":"2024-03-28T23:32:44.115074+0000","last_undegraded":"2024-03-28T23:32:44.115074+0000","last_fullsized":"2024-03-28T23:32:44.115074+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:38.859031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121532+0000","last_change":"2024-03-28T23:32:32.039047+0000","last_active":"2024-03-28T23:32:44.121532+0000","last_peered":"2024-03-28T23:32:44.121532+0000","last_clean":"2024-03-28T23:32:44.121532+0000","last_became_active":"2024-03-28T23:32:32.038780+0000","last_became_peered":"2024-03-28T23:32:32.038780+0000","last_unstale":"2024-03-28T23:32:44.121532+0000","last_undegraded":"2024-03-28T23:32:44.121532+0000","last_fullsized":"2024-03-28T23:32:44.121532+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:13:15.856750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.117228+0000","last_change":"2024-03-28T23:32:44.117228+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.117228+0000","last_undegraded":"2024-03-28T23:32:44.117228+0000","last_fullsized":"2024-03-28T23:32:44.117228+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122300+0000","last_change":"2024-03-28T23:32:44.121416+0000","last_active":"2024-03-28T23:32:44.122300+0000","last_peered":"2024-03-28T23:32:44.122300+0000","last_clean":"2024-03-28T23:32:44.122300+0000","last_became_active":"2024-03-28T23:32:44.121127+0000","last_became_peered":"2024-03-28T23:32:44.121127+0000","last_unstale":"2024-03-28T23:32:44.122300+0000","last_undegraded":"2024-03-28T23:32:44.122300+0000","last_fullsized":"2024-03-28T23:32:44.122300+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:39:02.592632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113417+0000","last_change":"2024-03-28T23:32:33.051453+0000","last_active":"2024-03-28T23:32:44.113417+0000","last_peered":"2024-03-28T23:32:44.113417+0000","last_clean":"2024-03-28T23:32:44.113417+0000","last_became_active":"2024-03-28T23:32:33.051155+0000","last_became_peered":"2024-03-28T23:32:33.051155+0000","last_unstale":"2024-03-28T23:32:44.113417+0000","last_undegraded":"2024-03-28T23:32:44.113417+0000","last_fullsized":"2024-03-28T23:32:44.113417+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:36:03.951054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108887+0000","last_change":"2024-03-28T23:32:32.036979+0000","last_active":"2024-03-28T23:32:44.108887+0000","last_peered":"2024-03-28T23:32:44.108887+0000","last_clean":"2024-03-28T23:32:44.108887+0000","last_became_active":"2024-03-28T23:32:32.036710+0000","last_became_peered":"2024-03-28T23:32:32.036710+0000","last_unstale":"2024-03-28T23:32:44.108887+0000","last_undegraded":"2024-03-28T23:32:44.108887+0000","last_fullsized":"2024-03-28T23:32:44.108887+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:46.072852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107489+0000","last_change":"2024-03-28T23:32:44.107489+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107489+0000","last_undegraded":"2024-03-28T23:32:44.107489+0000","last_fullsized":"2024-03-28T23:32:44.107489+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122075+0000","last_change":"2024-03-28T23:32:44.115740+0000","last_active":"2024-03-28T23:32:44.122075+0000","last_peered":"2024-03-28T23:32:44.122075+0000","last_clean":"2024-03-28T23:32:44.122075+0000","last_became_active":"2024-03-28T23:32:44.115341+0000","last_became_peered":"2024-03-28T23:32:44.115341+0000","last_unstale":"2024-03-28T23:32:44.122075+0000","last_undegraded":"2024-03-28T23:32:44.122075+0000","last_fullsized":"2024-03-28T23:32:44.122075+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:55.159548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121499+0000","last_change":"2024-03-28T23:32:33.041923+0000","last_active":"2024-03-28T23:32:44.121499+0000","last_peered":"2024-03-28T23:32:44.121499+0000","last_clean":"2024-03-28T23:32:44.121499+0000","last_became_active":"2024-03-28T23:32:33.041631+0000","last_became_peered":"2024-03-28T23:32:33.041631+0000","last_unstale":"2024-03-28T23:32:44.121499+0000","last_undegraded":"2024-03-28T23:32:44.121499+0000","last_fullsized":"2024-03-28T23:32:44.121499+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:32:08.473276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108352+0000","last_change":"2024-03-28T23:32:32.041563+0000","last_active":"2024-03-28T23:32:44.108352+0000","last_peered":"2024-03-28T23:32:44.108352+0000","last_clean":"2024-03-28T23:32:44.108352+0000","last_became_active":"2024-03-28T23:32:32.040424+0000","last_became_peered":"2024-03-28T23:32:32.040424+0000","last_unstale":"2024-03-28T23:32:44.108352+0000","last_undegraded":"2024-03-28T23:32:44.108352+0000","last_fullsized":"2024-03-28T23:32:44.108352+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:37:25.645113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107400+0000","last_change":"2024-03-28T23:32:44.107400+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107400+0000","last_undegraded":"2024-03-28T23:32:44.107400+0000","last_fullsized":"2024-03-28T23:32:44.107400+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.123301+0000","last_change":"2024-03-28T23:32:44.118040+0000","last_active":"2024-03-28T23:32:44.123301+0000","last_peered":"2024-03-28T23:32:44.123301+0000","last_clean":"2024-03-28T23:32:44.123301+0000","last_became_active":"2024-03-28T23:32:44.117536+0000","last_became_peered":"2024-03-28T23:32:44.117536+0000","last_unstale":"2024-03-28T23:32:44.123301+0000","last_undegraded":"2024-03-28T23:32:44.123301+0000","last_fullsized":"2024-03-28T23:32:44.123301+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:59:07.631069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106030+0000","last_change":"2024-03-28T23:32:33.029986+0000","last_active":"2024-03-28T23:32:44.106030+0000","last_peered":"2024-03-28T23:32:44.106030+0000","last_clean":"2024-03-28T23:32:44.106030+0000","last_became_active":"2024-03-28T23:32:33.029413+0000","last_became_peered":"2024-03-28T23:32:33.029413+0000","last_unstale":"2024-03-28T23:32:44.106030+0000","last_undegraded":"2024-03-28T23:32:44.106030+0000","last_fullsized":"2024-03-28T23:32:44.106030+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:37:26.652511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107737+0000","last_change":"2024-03-28T23:32:32.037039+0000","last_active":"2024-03-28T23:32:44.107737+0000","last_peered":"2024-03-28T23:32:44.107737+0000","last_clean":"2024-03-28T23:32:44.107737+0000","last_became_active":"2024-03-28T23:32:32.036687+0000","last_became_peered":"2024-03-28T23:32:32.036687+0000","last_unstale":"2024-03-28T23:32:44.107737+0000","last_undegraded":"2024-03-28T23:32:44.107737+0000","last_fullsized":"2024-03-28T23:32:44.107737+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:27:15.824876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107961+0000","last_change":"2024-03-28T23:32:44.107961+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107961+0000","last_undegraded":"2024-03-28T23:32:44.107961+0000","last_fullsized":"2024-03-28T23:32:44.107961+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.127537+0000","last_change":"2024-03-28T23:32:44.126156+0000","last_active":"2024-03-28T23:32:44.127537+0000","last_peered":"2024-03-28T23:32:44.127537+0000","last_clean":"2024-03-28T23:32:44.127537+0000","last_became_active":"2024-03-28T23:32:44.125789+0000","last_became_peered":"2024-03-28T23:32:44.125789+0000","last_unstale":"2024-03-28T23:32:44.127537+0000","last_undegraded":"2024-03-28T23:32:44.127537+0000","last_fullsized":"2024-03-28T23:32:44.127537+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:41:04.213333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113597+0000","last_change":"2024-03-28T23:32:33.035809+0000","last_active":"2024-03-28T23:32:44.113597+0000","last_peered":"2024-03-28T23:32:44.113597+0000","last_clean":"2024-03-28T23:32:44.113597+0000","last_became_active":"2024-03-28T23:32:33.035563+0000","last_became_peered":"2024-03-28T23:32:33.035563+0000","last_unstale":"2024-03-28T23:32:44.113597+0000","last_undegraded":"2024-03-28T23:32:44.113597+0000","last_fullsized":"2024-03-28T23:32:44.113597+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:54.194453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"72'3","reported_seq":21,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.110292+0000","last_change":"2024-03-28T23:32:32.038230+0000","last_active":"2024-03-28T23:32:44.110292+0000","last_peered":"2024-03-28T23:32:44.110292+0000","last_clean":"2024-03-28T23:32:44.110292+0000","last_became_active":"2024-03-28T23:32:32.037681+0000","last_became_peered":"2024-03-28T23:32:32.037681+0000","last_unstale":"2024-03-28T23:32:44.110292+0000","last_undegraded":"2024-03-28T23:32:44.110292+0000","last_fullsized":"2024-03-28T23:32:44.110292+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:36:34.752866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112106+0000","last_change":"2024-03-28T23:32:44.112106+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112106+0000","last_undegraded":"2024-03-28T23:32:44.112106+0000","last_fullsized":"2024-03-28T23:32:44.112106+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.126080+0000","last_change":"2024-03-28T23:32:44.125087+0000","last_active":"2024-03-28T23:32:44.126080+0000","last_peered":"2024-03-28T23:32:44.126080+0000","last_clean":"2024-03-28T23:32:44.126080+0000","last_became_active":"2024-03-28T23:32:44.124784+0000","last_became_peered":"2024-03-28T23:32:44.124784+0000","last_unstale":"2024-03-28T23:32:44.126080+0000","last_undegraded":"2024-03-28T23:32:44.126080+0000","last_fullsized":"2024-03-28T23:32:44.126080+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:16:45.967951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112348+0000","last_change":"2024-03-28T23:32:33.049499+0000","last_active":"2024-03-28T23:32:44.112348+0000","last_peered":"2024-03-28T23:32:44.112348+0000","last_clean":"2024-03-28T23:32:44.112348+0000","last_became_active":"2024-03-28T23:32:33.049233+0000","last_became_peered":"2024-03-28T23:32:33.049233+0000","last_unstale":"2024-03-28T23:32:44.112348+0000","last_undegraded":"2024-03-28T23:32:44.112348+0000","last_fullsized":"2024-03-28T23:32:44.112348+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:39:02.824323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"72'6","reported_seq":24,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112883+0000","last_change":"2024-03-28T23:32:32.036835+0000","last_active":"2024-03-28T23:32:44.112883+0000","last_peered":"2024-03-28T23:32:44.112883+0000","last_clean":"2024-03-28T23:32:44.112883+0000","last_became_active":"2024-03-28T23:32:32.036557+0000","last_became_peered":"2024-03-28T23:32:32.036557+0000","last_unstale":"2024-03-28T23:32:44.112883+0000","last_undegraded":"2024-03-28T23:32:44.112883+0000","last_fullsized":"2024-03-28T23:32:44.112883+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:30.992807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111621+0000","last_change":"2024-03-28T23:32:44.111621+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111621+0000","last_undegraded":"2024-03-28T23:32:44.111621+0000","last_fullsized":"2024-03-28T23:32:44.111621+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.127678+0000","last_change":"2024-03-28T23:32:44.127256+0000","last_active":"2024-03-28T23:32:44.127678+0000","last_peered":"2024-03-28T23:32:44.127678+0000","last_clean":"2024-03-28T23:32:44.127678+0000","last_became_active":"2024-03-28T23:32:44.126894+0000","last_became_peered":"2024-03-28T23:32:44.126894+0000","last_unstale":"2024-03-28T23:32:44.127678+0000","last_undegraded":"2024-03-28T23:32:44.127678+0000","last_fullsized":"2024-03-28T23:32:44.127678+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:28:37.442827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107086+0000","last_change":"2024-03-28T23:32:33.033045+0000","last_active":"2024-03-28T23:32:44.107086+0000","last_peered":"2024-03-28T23:32:44.107086+0000","last_clean":"2024-03-28T23:32:44.107086+0000","last_became_active":"2024-03-28T23:32:33.032771+0000","last_became_peered":"2024-03-28T23:32:33.032771+0000","last_unstale":"2024-03-28T23:32:44.107086+0000","last_undegraded":"2024-03-28T23:32:44.107086+0000","last_fullsized":"2024-03-28T23:32:44.107086+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:43.341877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"72'2","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106619+0000","last_change":"2024-03-28T23:32:32.032053+0000","last_active":"2024-03-28T23:32:44.106619+0000","last_peered":"2024-03-28T23:32:44.106619+0000","last_clean":"2024-03-28T23:32:44.106619+0000","last_became_active":"2024-03-28T23:32:32.031741+0000","last_became_peered":"2024-03-28T23:32:32.031741+0000","last_unstale":"2024-03-28T23:32:44.106619+0000","last_undegraded":"2024-03-28T23:32:44.106619+0000","last_fullsized":"2024-03-28T23:32:44.106619+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:20.456721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108993+0000","last_change":"2024-03-28T23:32:44.108993+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108993+0000","last_undegraded":"2024-03-28T23:32:44.108993+0000","last_fullsized":"2024-03-28T23:32:44.108993+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.125906+0000","last_change":"2024-03-28T23:32:44.124910+0000","last_active":"2024-03-28T23:32:44.125906+0000","last_peered":"2024-03-28T23:32:44.125906+0000","last_clean":"2024-03-28T23:32:44.125906+0000","last_became_active":"2024-03-28T23:32:44.124639+0000","last_became_peered":"2024-03-28T23:32:44.124639+0000","last_unstale":"2024-03-28T23:32:44.125906+0000","last_undegraded":"2024-03-28T23:32:44.125906+0000","last_fullsized":"2024-03-28T23:32:44.125906+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:13:09.913978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112459+0000","last_change":"2024-03-28T23:32:33.039022+0000","last_active":"2024-03-28T23:32:44.112459+0000","last_peered":"2024-03-28T23:32:44.112459+0000","last_clean":"2024-03-28T23:32:44.112459+0000","last_became_active":"2024-03-28T23:32:33.038508+0000","last_became_peered":"2024-03-28T23:32:33.038508+0000","last_unstale":"2024-03-28T23:32:44.112459+0000","last_undegraded":"2024-03-28T23:32:44.112459+0000","last_fullsized":"2024-03-28T23:32:44.112459+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:25:45.189859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106348+0000","last_change":"2024-03-28T23:32:32.041042+0000","last_active":"2024-03-28T23:32:44.106348+0000","last_peered":"2024-03-28T23:32:44.106348+0000","last_clean":"2024-03-28T23:32:44.106348+0000","last_became_active":"2024-03-28T23:32:32.040686+0000","last_became_peered":"2024-03-28T23:32:32.040686+0000","last_unstale":"2024-03-28T23:32:44.106348+0000","last_undegraded":"2024-03-28T23:32:44.106348+0000","last_fullsized":"2024-03-28T23:32:44.106348+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:46.343619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.114552+0000","last_change":"2024-03-28T23:32:44.114552+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.114552+0000","last_undegraded":"2024-03-28T23:32:44.114552+0000","last_fullsized":"2024-03-28T23:32:44.114552+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121906+0000","last_change":"2024-03-28T23:32:44.120268+0000","last_active":"2024-03-28T23:32:44.121906+0000","last_peered":"2024-03-28T23:32:44.121906+0000","last_clean":"2024-03-28T23:32:44.121906+0000","last_became_active":"2024-03-28T23:32:44.119951+0000","last_became_peered":"2024-03-28T23:32:44.119951+0000","last_unstale":"2024-03-28T23:32:44.121906+0000","last_undegraded":"2024-03-28T23:32:44.121906+0000","last_fullsized":"2024-03-28T23:32:44.121906+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:28:47.614634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112006+0000","last_change":"2024-03-28T23:32:33.034262+0000","last_active":"2024-03-28T23:32:44.112006+0000","last_peered":"2024-03-28T23:32:44.112006+0000","last_clean":"2024-03-28T23:32:44.112006+0000","last_became_active":"2024-03-28T23:32:33.034003+0000","last_became_peered":"2024-03-28T23:32:33.034003+0000","last_unstale":"2024-03-28T23:32:44.112006+0000","last_undegraded":"2024-03-28T23:32:44.112006+0000","last_fullsized":"2024-03-28T23:32:44.112006+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:41.257241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108918+0000","last_change":"2024-03-28T23:32:32.039031+0000","last_active":"2024-03-28T23:32:44.108918+0000","last_peered":"2024-03-28T23:32:44.108918+0000","last_clean":"2024-03-28T23:32:44.108918+0000","last_became_active":"2024-03-28T23:32:32.038766+0000","last_became_peered":"2024-03-28T23:32:32.038766+0000","last_unstale":"2024-03-28T23:32:44.108918+0000","last_undegraded":"2024-03-28T23:32:44.108918+0000","last_fullsized":"2024-03-28T23:32:44.108918+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:13.872794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107400+0000","last_change":"2024-03-28T23:32:44.107400+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107400+0000","last_undegraded":"2024-03-28T23:32:44.107400+0000","last_fullsized":"2024-03-28T23:32:44.107400+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.123690+0000","last_change":"2024-03-28T23:32:44.119707+0000","last_active":"2024-03-28T23:32:44.123690+0000","last_peered":"2024-03-28T23:32:44.123690+0000","last_clean":"2024-03-28T23:32:44.123690+0000","last_became_active":"2024-03-28T23:32:44.119438+0000","last_became_peered":"2024-03-28T23:32:44.119438+0000","last_unstale":"2024-03-28T23:32:44.123690+0000","last_undegraded":"2024-03-28T23:32:44.123690+0000","last_fullsized":"2024-03-28T23:32:44.123690+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:39.542218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113512+0000","last_change":"2024-03-28T23:32:33.043171+0000","last_active":"2024-03-28T23:32:44.113512+0000","last_peered":"2024-03-28T23:32:44.113512+0000","last_clean":"2024-03-28T23:32:44.113512+0000","last_became_active":"2024-03-28T23:32:33.042821+0000","last_became_peered":"2024-03-28T23:32:33.042821+0000","last_unstale":"2024-03-28T23:32:44.113512+0000","last_undegraded":"2024-03-28T23:32:44.113512+0000","last_fullsized":"2024-03-28T23:32:44.113512+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:21.878156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"72'8","reported_seq":26,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121583+0000","last_change":"2024-03-28T23:32:32.040166+0000","last_active":"2024-03-28T23:32:44.121583+0000","last_peered":"2024-03-28T23:32:44.121583+0000","last_clean":"2024-03-28T23:32:44.121583+0000","last_became_active":"2024-03-28T23:32:32.039560+0000","last_became_peered":"2024-03-28T23:32:32.039560+0000","last_unstale":"2024-03-28T23:32:44.121583+0000","last_undegraded":"2024-03-28T23:32:44.121583+0000","last_fullsized":"2024-03-28T23:32:44.121583+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:55.544945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.110399+0000","last_change":"2024-03-28T23:32:44.110399+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.110399+0000","last_undegraded":"2024-03-28T23:32:44.110399+0000","last_fullsized":"2024-03-28T23:32:44.110399+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.127360+0000","last_change":"2024-03-28T23:32:44.126116+0000","last_active":"2024-03-28T23:32:44.127360+0000","last_peered":"2024-03-28T23:32:44.127360+0000","last_clean":"2024-03-28T23:32:44.127360+0000","last_became_active":"2024-03-28T23:32:44.125761+0000","last_became_peered":"2024-03-28T23:32:44.125761+0000","last_unstale":"2024-03-28T23:32:44.127360+0000","last_undegraded":"2024-03-28T23:32:44.127360+0000","last_fullsized":"2024-03-28T23:32:44.127360+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:04.591562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113351+0000","last_change":"2024-03-28T23:32:33.043192+0000","last_active":"2024-03-28T23:32:44.113351+0000","last_peered":"2024-03-28T23:32:44.113351+0000","last_clean":"2024-03-28T23:32:44.113351+0000","last_became_active":"2024-03-28T23:32:33.042894+0000","last_became_peered":"2024-03-28T23:32:33.042894+0000","last_unstale":"2024-03-28T23:32:44.113351+0000","last_undegraded":"2024-03-28T23:32:44.113351+0000","last_fullsized":"2024-03-28T23:32:44.113351+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:32:08.473276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"72'1","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108454+0000","last_change":"2024-03-28T23:32:32.041892+0000","last_active":"2024-03-28T23:32:44.108454+0000","last_peered":"2024-03-28T23:32:44.108454+0000","last_clean":"2024-03-28T23:32:44.108454+0000","last_became_active":"2024-03-28T23:32:32.041533+0000","last_became_peered":"2024-03-28T23:32:32.041533+0000","last_unstale":"2024-03-28T23:32:44.108454+0000","last_undegraded":"2024-03-28T23:32:44.108454+0000","last_fullsized":"2024-03-28T23:32:44.108454+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:15.843269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.116304+0000","last_change":"2024-03-28T23:32:44.116304+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.116304+0000","last_undegraded":"2024-03-28T23:32:44.116304+0000","last_fullsized":"2024-03-28T23:32:44.116304+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.118959+0000","last_change":"2024-03-28T23:32:44.115585+0000","last_active":"2024-03-28T23:32:44.118959+0000","last_peered":"2024-03-28T23:32:44.118959+0000","last_clean":"2024-03-28T23:32:44.118959+0000","last_became_active":"2024-03-28T23:32:44.115308+0000","last_became_peered":"2024-03-28T23:32:44.115308+0000","last_unstale":"2024-03-28T23:32:44.118959+0000","last_undegraded":"2024-03-28T23:32:44.118959+0000","last_fullsized":"2024-03-28T23:32:44.118959+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:59.122709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108799+0000","last_change":"2024-03-28T23:32:33.046899+0000","last_active":"2024-03-28T23:32:44.108799+0000","last_peered":"2024-03-28T23:32:44.108799+0000","last_clean":"2024-03-28T23:32:44.108799+0000","last_became_active":"2024-03-28T23:32:33.046579+0000","last_became_peered":"2024-03-28T23:32:33.046579+0000","last_unstale":"2024-03-28T23:32:44.108799+0000","last_undegraded":"2024-03-28T23:32:44.108799+0000","last_fullsized":"2024-03-28T23:32:44.108799+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:08:06.339997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"72'2","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.115368+0000","last_change":"2024-03-28T23:32:32.031135+0000","last_active":"2024-03-28T23:32:44.115368+0000","last_peered":"2024-03-28T23:32:44.115368+0000","last_clean":"2024-03-28T23:32:44.115368+0000","last_became_active":"2024-03-28T23:32:32.030796+0000","last_became_peered":"2024-03-28T23:32:32.030796+0000","last_unstale":"2024-03-28T23:32:44.115368+0000","last_undegraded":"2024-03-28T23:32:44.115368+0000","last_fullsized":"2024-03-28T23:32:44.115368+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:18:21.816565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.116620+0000","last_change":"2024-03-28T23:32:44.116620+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.116620+0000","last_undegraded":"2024-03-28T23:32:44.116620+0000","last_fullsized":"2024-03-28T23:32:44.116620+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.117708+0000","last_change":"2024-03-28T23:32:44.116752+0000","last_active":"2024-03-28T23:32:44.117708+0000","last_peered":"2024-03-28T23:32:44.117708+0000","last_clean":"2024-03-28T23:32:44.117708+0000","last_became_active":"2024-03-28T23:32:44.116381+0000","last_became_peered":"2024-03-28T23:32:44.116381+0000","last_unstale":"2024-03-28T23:32:44.117708+0000","last_undegraded":"2024-03-28T23:32:44.117708+0000","last_fullsized":"2024-03-28T23:32:44.117708+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:17:56.623866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109016+0000","last_change":"2024-03-28T23:32:33.053549+0000","last_active":"2024-03-28T23:32:44.109016+0000","last_peered":"2024-03-28T23:32:44.109016+0000","last_clean":"2024-03-28T23:32:44.109016+0000","last_became_active":"2024-03-28T23:32:33.053229+0000","last_became_peered":"2024-03-28T23:32:33.053229+0000","last_unstale":"2024-03-28T23:32:44.109016+0000","last_undegraded":"2024-03-28T23:32:44.109016+0000","last_fullsized":"2024-03-28T23:32:44.109016+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:41:02.517121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"72'1","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.110391+0000","last_change":"2024-03-28T23:32:32.038261+0000","last_active":"2024-03-28T23:32:44.110391+0000","last_peered":"2024-03-28T23:32:44.110391+0000","last_clean":"2024-03-28T23:32:44.110391+0000","last_became_active":"2024-03-28T23:32:32.037971+0000","last_became_peered":"2024-03-28T23:32:32.037971+0000","last_unstale":"2024-03-28T23:32:44.110391+0000","last_undegraded":"2024-03-28T23:32:44.110391+0000","last_fullsized":"2024-03-28T23:32:44.110391+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:25:10.296139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.109933+0000","last_change":"2024-03-28T23:32:44.109933+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.109933+0000","last_undegraded":"2024-03-28T23:32:44.109933+0000","last_fullsized":"2024-03-28T23:32:44.109933+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.118230+0000","last_change":"2024-03-28T23:32:44.115062+0000","last_active":"2024-03-28T23:32:44.118230+0000","last_peered":"2024-03-28T23:32:44.118230+0000","last_clean":"2024-03-28T23:32:44.118230+0000","last_became_active":"2024-03-28T23:32:44.114795+0000","last_became_peered":"2024-03-28T23:32:44.114795+0000","last_unstale":"2024-03-28T23:32:44.118230+0000","last_undegraded":"2024-03-28T23:32:44.118230+0000","last_fullsized":"2024-03-28T23:32:44.118230+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:38.460546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113026+0000","last_change":"2024-03-28T23:32:33.043954+0000","last_active":"2024-03-28T23:32:44.113026+0000","last_peered":"2024-03-28T23:32:44.113026+0000","last_clean":"2024-03-28T23:32:44.113026+0000","last_became_active":"2024-03-28T23:32:33.043649+0000","last_became_peered":"2024-03-28T23:32:33.043649+0000","last_unstale":"2024-03-28T23:32:44.113026+0000","last_undegraded":"2024-03-28T23:32:44.113026+0000","last_fullsized":"2024-03-28T23:32:44.113026+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:00.358180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109556+0000","last_change":"2024-03-28T23:32:32.033720+0000","last_active":"2024-03-28T23:32:44.109556+0000","last_peered":"2024-03-28T23:32:44.109556+0000","last_clean":"2024-03-28T23:32:44.109556+0000","last_became_active":"2024-03-28T23:32:32.033422+0000","last_became_peered":"2024-03-28T23:32:32.033422+0000","last_unstale":"2024-03-28T23:32:44.109556+0000","last_undegraded":"2024-03-28T23:32:44.109556+0000","last_fullsized":"2024-03-28T23:32:44.109556+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:51:10.440109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.113185+0000","last_change":"2024-03-28T23:32:44.113185+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.113185+0000","last_undegraded":"2024-03-28T23:32:44.113185+0000","last_fullsized":"2024-03-28T23:32:44.113185+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.124927+0000","last_change":"2024-03-28T23:32:44.124559+0000","last_active":"2024-03-28T23:32:44.124927+0000","last_peered":"2024-03-28T23:32:44.124927+0000","last_clean":"2024-03-28T23:32:44.124927+0000","last_became_active":"2024-03-28T23:32:44.124282+0000","last_became_peered":"2024-03-28T23:32:44.124282+0000","last_unstale":"2024-03-28T23:32:44.124927+0000","last_undegraded":"2024-03-28T23:32:44.124927+0000","last_fullsized":"2024-03-28T23:32:44.124927+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:32.542845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122528+0000","last_change":"2024-03-28T23:32:33.045618+0000","last_active":"2024-03-28T23:32:44.122528+0000","last_peered":"2024-03-28T23:32:44.122528+0000","last_clean":"2024-03-28T23:32:44.122528+0000","last_became_active":"2024-03-28T23:32:33.045324+0000","last_became_peered":"2024-03-28T23:32:33.045324+0000","last_unstale":"2024-03-28T23:32:44.122528+0000","last_undegraded":"2024-03-28T23:32:44.122528+0000","last_fullsized":"2024-03-28T23:32:44.122528+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:46:43.644886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"72'2","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113118+0000","last_change":"2024-03-28T23:32:32.031574+0000","last_active":"2024-03-28T23:32:44.113118+0000","last_peered":"2024-03-28T23:32:44.113118+0000","last_clean":"2024-03-28T23:32:44.113118+0000","last_became_active":"2024-03-28T23:32:32.031310+0000","last_became_peered":"2024-03-28T23:32:32.031310+0000","last_unstale":"2024-03-28T23:32:44.113118+0000","last_undegraded":"2024-03-28T23:32:44.113118+0000","last_fullsized":"2024-03-28T23:32:44.113118+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:48:07.073481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112326+0000","last_change":"2024-03-28T23:32:44.112326+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112326+0000","last_undegraded":"2024-03-28T23:32:44.112326+0000","last_fullsized":"2024-03-28T23:32:44.112326+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.115233+0000","last_change":"2024-03-28T23:32:44.114144+0000","last_active":"2024-03-28T23:32:44.115233+0000","last_peered":"2024-03-28T23:32:44.115233+0000","last_clean":"2024-03-28T23:32:44.115233+0000","last_became_active":"2024-03-28T23:32:44.113829+0000","last_became_peered":"2024-03-28T23:32:44.113829+0000","last_unstale":"2024-03-28T23:32:44.115233+0000","last_undegraded":"2024-03-28T23:32:44.115233+0000","last_fullsized":"2024-03-28T23:32:44.115233+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:55.159548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.123006+0000","last_change":"2024-03-28T23:32:33.040983+0000","last_active":"2024-03-28T23:32:44.123006+0000","last_peered":"2024-03-28T23:32:44.123006+0000","last_clean":"2024-03-28T23:32:44.123006+0000","last_became_active":"2024-03-28T23:32:33.040693+0000","last_became_peered":"2024-03-28T23:32:33.040693+0000","last_unstale":"2024-03-28T23:32:44.123006+0000","last_undegraded":"2024-03-28T23:32:44.123006+0000","last_fullsized":"2024-03-28T23:32:44.123006+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:22:05.201764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113759+0000","last_change":"2024-03-28T23:32:32.036998+0000","last_active":"2024-03-28T23:32:44.113759+0000","last_peered":"2024-03-28T23:32:44.113759+0000","last_clean":"2024-03-28T23:32:44.113759+0000","last_became_active":"2024-03-28T23:32:32.036733+0000","last_became_peered":"2024-03-28T23:32:32.036733+0000","last_unstale":"2024-03-28T23:32:44.113759+0000","last_undegraded":"2024-03-28T23:32:44.113759+0000","last_fullsized":"2024-03-28T23:32:44.113759+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:42.334479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.105630+0000","last_change":"2024-03-28T23:32:44.105630+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.105630+0000","last_undegraded":"2024-03-28T23:32:44.105630+0000","last_fullsized":"2024-03-28T23:32:44.105630+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.115890+0000","last_change":"2024-03-28T23:32:44.115154+0000","last_active":"2024-03-28T23:32:44.115890+0000","last_peered":"2024-03-28T23:32:44.115890+0000","last_clean":"2024-03-28T23:32:44.115890+0000","last_became_active":"2024-03-28T23:32:44.114869+0000","last_became_peered":"2024-03-28T23:32:44.114869+0000","last_unstale":"2024-03-28T23:32:44.115890+0000","last_undegraded":"2024-03-28T23:32:44.115890+0000","last_fullsized":"2024-03-28T23:32:44.115890+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:58:59.309973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113614+0000","last_change":"2024-03-28T23:32:33.041232+0000","last_active":"2024-03-28T23:32:44.113614+0000","last_peered":"2024-03-28T23:32:44.113614+0000","last_clean":"2024-03-28T23:32:44.113614+0000","last_became_active":"2024-03-28T23:32:33.040132+0000","last_became_peered":"2024-03-28T23:32:33.040132+0000","last_unstale":"2024-03-28T23:32:44.113614+0000","last_undegraded":"2024-03-28T23:32:44.113614+0000","last_fullsized":"2024-03-28T23:32:44.113614+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:39:02.824323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113450+0000","last_change":"2024-03-28T23:32:32.036416+0000","last_active":"2024-03-28T23:32:44.113450+0000","last_peered":"2024-03-28T23:32:44.113450+0000","last_clean":"2024-03-28T23:32:44.113450+0000","last_became_active":"2024-03-28T23:32:32.036131+0000","last_became_peered":"2024-03-28T23:32:32.036131+0000","last_unstale":"2024-03-28T23:32:44.113450+0000","last_undegraded":"2024-03-28T23:32:44.113450+0000","last_fullsized":"2024-03-28T23:32:44.113450+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:13:33.914342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.109522+0000","last_change":"2024-03-28T23:32:44.109522+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.109522+0000","last_undegraded":"2024-03-28T23:32:44.109522+0000","last_fullsized":"2024-03-28T23:32:44.109522+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.119994+0000","last_change":"2024-03-28T23:32:44.116820+0000","last_active":"2024-03-28T23:32:44.119994+0000","last_peered":"2024-03-28T23:32:44.119994+0000","last_clean":"2024-03-28T23:32:44.119994+0000","last_became_active":"2024-03-28T23:32:44.116540+0000","last_became_peered":"2024-03-28T23:32:44.116540+0000","last_unstale":"2024-03-28T23:32:44.119994+0000","last_undegraded":"2024-03-28T23:32:44.119994+0000","last_fullsized":"2024-03-28T23:32:44.119994+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:58.429743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.110759+0000","last_change":"2024-03-28T23:32:33.035015+0000","last_active":"2024-03-28T23:32:44.110759+0000","last_peered":"2024-03-28T23:32:44.110759+0000","last_clean":"2024-03-28T23:32:44.110759+0000","last_became_active":"2024-03-28T23:32:33.034730+0000","last_became_peered":"2024-03-28T23:32:33.034730+0000","last_unstale":"2024-03-28T23:32:44.110759+0000","last_undegraded":"2024-03-28T23:32:44.110759+0000","last_fullsized":"2024-03-28T23:32:44.110759+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:09.477928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"72'2","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108295+0000","last_change":"2024-03-28T23:32:32.032596+0000","last_active":"2024-03-28T23:32:44.108295+0000","last_peered":"2024-03-28T23:32:44.108295+0000","last_clean":"2024-03-28T23:32:44.108295+0000","last_became_active":"2024-03-28T23:32:32.032336+0000","last_became_peered":"2024-03-28T23:32:32.032336+0000","last_unstale":"2024-03-28T23:32:44.108295+0000","last_undegraded":"2024-03-28T23:32:44.108295+0000","last_fullsized":"2024-03-28T23:32:44.108295+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:28:25.356704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.120871+0000","last_change":"2024-03-28T23:32:44.120871+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.120871+0000","last_undegraded":"2024-03-28T23:32:44.120871+0000","last_fullsized":"2024-03-28T23:32:44.120871+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.117955+0000","last_change":"2024-03-28T23:32:44.116649+0000","last_active":"2024-03-28T23:32:44.117955+0000","last_peered":"2024-03-28T23:32:44.117955+0000","last_clean":"2024-03-28T23:32:44.117955+0000","last_became_active":"2024-03-28T23:32:44.116369+0000","last_became_peered":"2024-03-28T23:32:44.116369+0000","last_unstale":"2024-03-28T23:32:44.117955+0000","last_undegraded":"2024-03-28T23:32:44.117955+0000","last_fullsized":"2024-03-28T23:32:44.117955+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:46:54.723611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.111340+0000","last_change":"2024-03-28T23:32:33.044243+0000","last_active":"2024-03-28T23:32:44.111340+0000","last_peered":"2024-03-28T23:32:44.111340+0000","last_clean":"2024-03-28T23:32:44.111340+0000","last_became_active":"2024-03-28T23:32:33.040218+0000","last_became_peered":"2024-03-28T23:32:33.040218+0000","last_unstale":"2024-03-28T23:32:44.111340+0000","last_undegraded":"2024-03-28T23:32:44.111340+0000","last_fullsized":"2024-03-28T23:32:44.111340+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:38:51.513907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"73'1","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.123099+0000","last_change":"2024-03-28T23:32:32.041586+0000","last_active":"2024-03-28T23:32:44.123099+0000","last_peered":"2024-03-28T23:32:44.123099+0000","last_clean":"2024-03-28T23:32:44.123099+0000","last_became_active":"2024-03-28T23:32:32.041327+0000","last_became_peered":"2024-03-28T23:32:32.041327+0000","last_unstale":"2024-03-28T23:32:44.123099+0000","last_undegraded":"2024-03-28T23:32:44.123099+0000","last_fullsized":"2024-03-28T23:32:44.123099+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:06:20.608980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107219+0000","last_change":"2024-03-28T23:32:44.107219+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107219+0000","last_undegraded":"2024-03-28T23:32:44.107219+0000","last_fullsized":"2024-03-28T23:32:44.107219+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.126567+0000","last_change":"2024-03-28T23:32:44.125883+0000","last_active":"2024-03-28T23:32:44.126567+0000","last_peered":"2024-03-28T23:32:44.126567+0000","last_clean":"2024-03-28T23:32:44.126567+0000","last_became_active":"2024-03-28T23:32:44.125578+0000","last_became_peered":"2024-03-28T23:32:44.125578+0000","last_unstale":"2024-03-28T23:32:44.126567+0000","last_undegraded":"2024-03-28T23:32:44.126567+0000","last_fullsized":"2024-03-28T23:32:44.126567+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:25.238412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112455+0000","last_change":"2024-03-28T23:32:33.042272+0000","last_active":"2024-03-28T23:32:44.112455+0000","last_peered":"2024-03-28T23:32:44.112455+0000","last_clean":"2024-03-28T23:32:44.112455+0000","last_became_active":"2024-03-28T23:32:33.037633+0000","last_became_peered":"2024-03-28T23:32:33.037633+0000","last_unstale":"2024-03-28T23:32:44.112455+0000","last_undegraded":"2024-03-28T23:32:44.112455+0000","last_fullsized":"2024-03-28T23:32:44.112455+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:45.773838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108173+0000","last_change":"2024-03-28T23:32:32.037001+0000","last_active":"2024-03-28T23:32:44.108173+0000","last_peered":"2024-03-28T23:32:44.108173+0000","last_clean":"2024-03-28T23:32:44.108173+0000","last_became_active":"2024-03-28T23:32:32.036739+0000","last_became_peered":"2024-03-28T23:32:32.036739+0000","last_unstale":"2024-03-28T23:32:44.108173+0000","last_undegraded":"2024-03-28T23:32:44.108173+0000","last_fullsized":"2024-03-28T23:32:44.108173+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:06:20.608980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112232+0000","last_change":"2024-03-28T23:32:44.112232+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112232+0000","last_undegraded":"2024-03-28T23:32:44.112232+0000","last_fullsized":"2024-03-28T23:32:44.112232+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.117952+0000","last_change":"2024-03-28T23:32:44.116971+0000","last_active":"2024-03-28T23:32:44.117952+0000","last_peered":"2024-03-28T23:32:44.117952+0000","last_clean":"2024-03-28T23:32:44.117952+0000","last_became_active":"2024-03-28T23:32:44.116371+0000","last_became_peered":"2024-03-28T23:32:44.116371+0000","last_unstale":"2024-03-28T23:32:44.117952+0000","last_undegraded":"2024-03-28T23:32:44.117952+0000","last_fullsized":"2024-03-28T23:32:44.117952+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:43:47.025717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122394+0000","last_change":"2024-03-28T23:32:33.041123+0000","last_active":"2024-03-28T23:32:44.122394+0000","last_peered":"2024-03-28T23:32:44.122394+0000","last_clean":"2024-03-28T23:32:44.122394+0000","last_became_active":"2024-03-28T23:32:33.040505+0000","last_became_peered":"2024-03-28T23:32:33.040505+0000","last_unstale":"2024-03-28T23:32:44.122394+0000","last_undegraded":"2024-03-28T23:32:44.122394+0000","last_fullsized":"2024-03-28T23:32:44.122394+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:54.194453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106365+0000","last_change":"2024-03-28T23:32:32.036857+0000","last_active":"2024-03-28T23:32:44.106365+0000","last_peered":"2024-03-28T23:32:44.106365+0000","last_clean":"2024-03-28T23:32:44.106365+0000","last_became_active":"2024-03-28T23:32:32.036576+0000","last_became_peered":"2024-03-28T23:32:32.036576+0000","last_unstale":"2024-03-28T23:32:44.106365+0000","last_undegraded":"2024-03-28T23:32:44.106365+0000","last_fullsized":"2024-03-28T23:32:44.106365+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:43.073425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.120260+0000","last_change":"2024-03-28T23:32:44.120260+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.120260+0000","last_undegraded":"2024-03-28T23:32:44.120260+0000","last_fullsized":"2024-03-28T23:32:44.120260+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.127553+0000","last_change":"2024-03-28T23:32:44.126706+0000","last_active":"2024-03-28T23:32:44.127553+0000","last_peered":"2024-03-28T23:32:44.127553+0000","last_clean":"2024-03-28T23:32:44.127553+0000","last_became_active":"2024-03-28T23:32:44.126400+0000","last_became_peered":"2024-03-28T23:32:44.126400+0000","last_unstale":"2024-03-28T23:32:44.127553+0000","last_undegraded":"2024-03-28T23:32:44.127553+0000","last_fullsized":"2024-03-28T23:32:44.127553+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:00.790475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112670+0000","last_change":"2024-03-28T23:32:33.041769+0000","last_active":"2024-03-28T23:32:44.112670+0000","last_peered":"2024-03-28T23:32:44.112670+0000","last_clean":"2024-03-28T23:32:44.112670+0000","last_became_active":"2024-03-28T23:32:33.041475+0000","last_became_peered":"2024-03-28T23:32:33.041475+0000","last_unstale":"2024-03-28T23:32:44.112670+0000","last_undegraded":"2024-03-28T23:32:44.112670+0000","last_fullsized":"2024-03-28T23:32:44.112670+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:38.859031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112270+0000","last_change":"2024-03-28T23:32:32.037824+0000","last_active":"2024-03-28T23:32:44.112270+0000","last_peered":"2024-03-28T23:32:44.112270+0000","last_clean":"2024-03-28T23:32:44.112270+0000","last_became_active":"2024-03-28T23:32:32.037550+0000","last_became_peered":"2024-03-28T23:32:32.037550+0000","last_unstale":"2024-03-28T23:32:44.112270+0000","last_undegraded":"2024-03-28T23:32:44.112270+0000","last_fullsized":"2024-03-28T23:32:44.112270+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:43.467214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111817+0000","last_change":"2024-03-28T23:32:44.111817+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111817+0000","last_undegraded":"2024-03-28T23:32:44.111817+0000","last_fullsized":"2024-03-28T23:32:44.111817+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.119913+0000","last_change":"2024-03-28T23:32:44.117095+0000","last_active":"2024-03-28T23:32:44.119913+0000","last_peered":"2024-03-28T23:32:44.119913+0000","last_clean":"2024-03-28T23:32:44.119913+0000","last_became_active":"2024-03-28T23:32:44.116824+0000","last_became_peered":"2024-03-28T23:32:44.116824+0000","last_unstale":"2024-03-28T23:32:44.119913+0000","last_undegraded":"2024-03-28T23:32:44.119913+0000","last_fullsized":"2024-03-28T23:32:44.119913+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:59:07.631069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112978+0000","last_change":"2024-03-28T23:32:33.048966+0000","last_active":"2024-03-28T23:32:44.112978+0000","last_peered":"2024-03-28T23:32:44.112978+0000","last_clean":"2024-03-28T23:32:44.112978+0000","last_became_active":"2024-03-28T23:32:33.048362+0000","last_became_peered":"2024-03-28T23:32:33.048362+0000","last_unstale":"2024-03-28T23:32:44.112978+0000","last_undegraded":"2024-03-28T23:32:44.112978+0000","last_fullsized":"2024-03-28T23:32:44.112978+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:25.849919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109099+0000","last_change":"2024-03-28T23:32:32.038885+0000","last_active":"2024-03-28T23:32:44.109099+0000","last_peered":"2024-03-28T23:32:44.109099+0000","last_clean":"2024-03-28T23:32:44.109099+0000","last_became_active":"2024-03-28T23:32:32.038364+0000","last_became_peered":"2024-03-28T23:32:32.038364+0000","last_unstale":"2024-03-28T23:32:44.109099+0000","last_undegraded":"2024-03-28T23:32:44.109099+0000","last_fullsized":"2024-03-28T23:32:44.109099+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:16:06.156877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.105630+0000","last_change":"2024-03-28T23:32:44.105630+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.105630+0000","last_undegraded":"2024-03-28T23:32:44.105630+0000","last_fullsized":"2024-03-28T23:32:44.105630+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.115613+0000","last_change":"2024-03-28T23:32:44.114483+0000","last_active":"2024-03-28T23:32:44.115613+0000","last_peered":"2024-03-28T23:32:44.115613+0000","last_clean":"2024-03-28T23:32:44.115613+0000","last_became_active":"2024-03-28T23:32:44.114228+0000","last_became_peered":"2024-03-28T23:32:44.114228+0000","last_unstale":"2024-03-28T23:32:44.115613+0000","last_undegraded":"2024-03-28T23:32:44.115613+0000","last_fullsized":"2024-03-28T23:32:44.115613+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:56:05.273178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107950+0000","last_change":"2024-03-28T23:32:33.049584+0000","last_active":"2024-03-28T23:32:44.107950+0000","last_peered":"2024-03-28T23:32:44.107950+0000","last_clean":"2024-03-28T23:32:44.107950+0000","last_became_active":"2024-03-28T23:32:33.049342+0000","last_became_peered":"2024-03-28T23:32:33.049342+0000","last_unstale":"2024-03-28T23:32:44.107950+0000","last_undegraded":"2024-03-28T23:32:44.107950+0000","last_fullsized":"2024-03-28T23:32:44.107950+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:49.711750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.111513+0000","last_change":"2024-03-28T23:32:32.034044+0000","last_active":"2024-03-28T23:32:44.111513+0000","last_peered":"2024-03-28T23:32:44.111513+0000","last_clean":"2024-03-28T23:32:44.111513+0000","last_became_active":"2024-03-28T23:32:32.033779+0000","last_became_peered":"2024-03-28T23:32:32.033779+0000","last_unstale":"2024-03-28T23:32:44.111513+0000","last_undegraded":"2024-03-28T23:32:44.111513+0000","last_fullsized":"2024-03-28T23:32:44.111513+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:56:22.159708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.116670+0000","last_change":"2024-03-28T23:32:44.116670+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.116670+0000","last_undegraded":"2024-03-28T23:32:44.116670+0000","last_fullsized":"2024-03-28T23:32:44.116670+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.120835+0000","last_change":"2024-03-28T23:32:44.119649+0000","last_active":"2024-03-28T23:32:44.120835+0000","last_peered":"2024-03-28T23:32:44.120835+0000","last_clean":"2024-03-28T23:32:44.120835+0000","last_became_active":"2024-03-28T23:32:44.119370+0000","last_became_peered":"2024-03-28T23:32:44.119370+0000","last_unstale":"2024-03-28T23:32:44.120835+0000","last_undegraded":"2024-03-28T23:32:44.120835+0000","last_fullsized":"2024-03-28T23:32:44.120835+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:25:56.268584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109652+0000","last_change":"2024-03-28T23:32:33.041180+0000","last_active":"2024-03-28T23:32:44.109652+0000","last_peered":"2024-03-28T23:32:44.109652+0000","last_clean":"2024-03-28T23:32:44.109652+0000","last_became_active":"2024-03-28T23:32:33.040922+0000","last_became_peered":"2024-03-28T23:32:33.040922+0000","last_unstale":"2024-03-28T23:32:44.109652+0000","last_undegraded":"2024-03-28T23:32:44.109652+0000","last_fullsized":"2024-03-28T23:32:44.109652+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:25:45.189859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109480+0000","last_change":"2024-03-28T23:32:32.034328+0000","last_active":"2024-03-28T23:32:44.109480+0000","last_peered":"2024-03-28T23:32:44.109480+0000","last_clean":"2024-03-28T23:32:44.109480+0000","last_became_active":"2024-03-28T23:32:32.034057+0000","last_became_peered":"2024-03-28T23:32:32.034057+0000","last_unstale":"2024-03-28T23:32:44.109480+0000","last_undegraded":"2024-03-28T23:32:44.109480+0000","last_fullsized":"2024-03-28T23:32:44.109480+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:54:12.775407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107591+0000","last_change":"2024-03-28T23:32:44.107591+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107591+0000","last_undegraded":"2024-03-28T23:32:44.107591+0000","last_fullsized":"2024-03-28T23:32:44.107591+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.119109+0000","last_change":"2024-03-28T23:32:44.115020+0000","last_active":"2024-03-28T23:32:44.119109+0000","last_peered":"2024-03-28T23:32:44.119109+0000","last_clean":"2024-03-28T23:32:44.119109+0000","last_became_active":"2024-03-28T23:32:44.114737+0000","last_became_peered":"2024-03-28T23:32:44.114737+0000","last_unstale":"2024-03-28T23:32:44.119109+0000","last_undegraded":"2024-03-28T23:32:44.119109+0000","last_fullsized":"2024-03-28T23:32:44.119109+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:13:16.820792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108150+0000","last_change":"2024-03-28T23:32:33.038830+0000","last_active":"2024-03-28T23:32:44.108150+0000","last_peered":"2024-03-28T23:32:44.108150+0000","last_clean":"2024-03-28T23:32:44.108150+0000","last_became_active":"2024-03-28T23:32:33.038524+0000","last_became_peered":"2024-03-28T23:32:33.038524+0000","last_unstale":"2024-03-28T23:32:44.108150+0000","last_undegraded":"2024-03-28T23:32:44.108150+0000","last_fullsized":"2024-03-28T23:32:44.108150+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:55:53.512836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113822+0000","last_change":"2024-03-28T23:32:32.034270+0000","last_active":"2024-03-28T23:32:44.113822+0000","last_peered":"2024-03-28T23:32:44.113822+0000","last_clean":"2024-03-28T23:32:44.113822+0000","last_became_active":"2024-03-28T23:32:32.034022+0000","last_became_peered":"2024-03-28T23:32:32.034022+0000","last_unstale":"2024-03-28T23:32:44.113822+0000","last_undegraded":"2024-03-28T23:32:44.113822+0000","last_fullsized":"2024-03-28T23:32:44.113822+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:15.683500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":273,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109089+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:32:44.109089+0000","last_peered":"2024-03-28T23:32:44.109089+0000","last_clean":"2024-03-28T23:32:44.109089+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:32:44.109089+0000","last_undegraded":"2024-03-28T23:32:44.109089+0000","last_fullsized":"2024-03-28T23:32:44.109089+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574887,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":974848,"data_stored":1018333,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574888,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32088,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339752,"statfs":{"total":96636764160,"available":96603906048,"internally_reserved":0,"allocated":929792,"data_stored":437077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32068,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339772,"statfs":{"total":96636764160,"available":96603926528,"internally_reserved":0,"allocated":909312,"data_stored":427977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574887,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32084,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339756,"statfs":{"total":96636764160,"available":96603910144,"internally_reserved":0,"allocated":925696,"data_stored":429659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574887,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27908,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343932,"statfs":{"total":96636764160,"available":96608186368,"internally_reserved":0,"allocated":909312,"data_stored":428525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574887,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343924,"statfs":{"total":96636764160,"available":96608178176,"internally_reserved":0,"allocated":917504,"data_stored":428685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574887,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343928,"statfs":{"total":96636764160,"available":96608182272,"internally_reserved":0,"allocated":913408,"data_stored":428055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574887,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":970752,"data_stored":1018311,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:32:48.174 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:32:48.302 INFO:tasks.ceph.mds.b.smithi170.stderr:2024-03-28T23:32:48.302+0000 7f88193ea640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:32:48.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.309+0000 7f2e6c101640 1 Processor -- start 2024-03-28T23:32:48.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.309+0000 7f2e6c101640 1 -- start start 2024-03-28T23:32:48.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.311+0000 7f2e6c101640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2e64152890 0x7f2e640c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:48.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.311+0000 7f2e6c101640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2e640c8ed0 0x7f2e640c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:48.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.311+0000 7f2e6c101640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e64158a30 con 0x7f2e640c5950 2024-03-28T23:32:48.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.311+0000 7f2e6c101640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2e640c8480 con 0x7f2e64152890 2024-03-28T23:32:48.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.311+0000 7f2e6c101640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f2e64159640 con 0x7f2e640c8ed0 2024-03-28T23:32:48.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.311+0000 7f2e69e9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2e64152890 0x7f2e640c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:48.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.311+0000 7f2e6a69e640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f2e640c5950 0x7f2e641553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59364/0 (socket says 172.21.15.161:59364) 2024-03-28T23:32:48.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.311+0000 7f2e6a69e640 1 -- 172.21.15.161:0/2777788080 learned_addr learned my addr 172.21.15.161:0/2777788080 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:48.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.311+0000 7f2e6969c640 1 --2- 172.21.15.161:0/2777788080 >> v2:172.21.15.170:3300/0 conn(0x7f2e640c8ed0 0x7f2e640c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:48.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.311+0000 7f2e69e9d640 1 -- 172.21.15.161:0/2777788080 >> v2:172.21.15.170:3300/0 conn(0x7f2e640c8ed0 msgr2=0x7f2e640c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:48.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.311+0000 7f2e69e9d640 1 --2- 172.21.15.161:0/2777788080 >> v2:172.21.15.170:3300/0 conn(0x7f2e640c8ed0 0x7f2e640c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.311+0000 7f2e69e9d640 1 -- 172.21.15.161:0/2777788080 >> v1:172.21.15.161:6789/0 conn(0x7f2e640c5950 legacy=0x7f2e641553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:48.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.311+0000 7f2e68e9b640 1 -- 172.21.15.161:0/2777788080 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2495944726 0 0) 0x7f2e64158a30 con 0x7f2e640c5950 2024-03-28T23:32:48.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.311+0000 7f2e69e9d640 1 -- 172.21.15.161:0/2777788080 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e64158d10 con 0x7f2e64152890 2024-03-28T23:32:48.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.312+0000 7f2e69e9d640 1 --2- 172.21.15.161:0/2777788080 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e64152890 0x7f2e640c8990 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f2e640ad6e0 tx=0x7f2e5402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=181dfae0500bc0c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:48.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.312+0000 7f2e68e9b640 1 -- 172.21.15.161:0/2777788080 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e54038070 con 0x7f2e64152890 2024-03-28T23:32:48.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.312+0000 7f2e68e9b640 1 -- 172.21.15.161:0/2777788080 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e54031710 con 0x7f2e64152890 2024-03-28T23:32:48.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.312+0000 7f2e68e9b640 1 -- 172.21.15.161:0/2777788080 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e54031ae0 con 0x7f2e64152890 2024-03-28T23:32:48.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.313+0000 7f2e6c101640 1 -- 172.21.15.161:0/2777788080 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e64152890 msgr2=0x7f2e640c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:48.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.313+0000 7f2e6c101640 1 --2- 172.21.15.161:0/2777788080 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e64152890 0x7f2e640c8990 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f2e640ad6e0 tx=0x7f2e5402b7b0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.313+0000 7f2e6c101640 1 -- 172.21.15.161:0/2777788080 shutdown_connections 2024-03-28T23:32:48.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.313+0000 7f2e6c101640 1 --2- 172.21.15.161:0/2777788080 >> v2:172.21.15.170:3300/0 conn(0x7f2e640c8ed0 0x7f2e640c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.313+0000 7f2e6c101640 1 --2- 172.21.15.161:0/2777788080 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e64152890 0x7f2e640c8990 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.313+0000 7f2e6c101640 1 -- 172.21.15.161:0/2777788080 >> 172.21.15.161:0/2777788080 conn(0x7f2e640ac620 msgr2=0x7f2e640aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:48.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.313+0000 7f2e6c101640 1 -- 172.21.15.161:0/2777788080 shutdown_connections 2024-03-28T23:32:48.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.313+0000 7f2e6c101640 1 -- 172.21.15.161:0/2777788080 wait complete. 2024-03-28T23:32:48.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.315+0000 7f2e6c101640 1 Processor -- start 2024-03-28T23:32:48.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.315+0000 7f2e6c101640 1 -- start start 2024-03-28T23:32:48.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.316+0000 7f2e6c101640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e640c5950 0x7f2e641ee790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:48.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.316+0000 7f2e6c101640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e640c8ed0 0x7f2e640bed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:48.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.316+0000 7f2e6c101640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e64152890 0x7f2e6414db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:48.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.316+0000 7f2e6c101640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2e64152c70 con 0x7f2e640c5950 2024-03-28T23:32:48.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.316+0000 7f2e6c101640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2e641592d0 con 0x7f2e640c8ed0 2024-03-28T23:32:48.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.316+0000 7f2e6c101640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2e64152df0 con 0x7f2e64152890 2024-03-28T23:32:48.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.316+0000 7f2e69e9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e640c5950 0x7f2e641ee790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:48.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.316+0000 7f2e69e9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e640c5950 0x7f2e641ee790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39924/0 (socket says 172.21.15.161:39924) 2024-03-28T23:32:48.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.316+0000 7f2e69e9d640 1 -- 172.21.15.161:0/928827995 learned_addr learned my addr 172.21.15.161:0/928827995 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:48.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.316+0000 7f2e6a69e640 1 --2- 172.21.15.161:0/928827995 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e64152890 0x7f2e6414db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:48.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.316+0000 7f2e6969c640 1 --2- 172.21.15.161:0/928827995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e640c8ed0 0x7f2e640bed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:48.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.316+0000 7f2e69e9d640 1 -- 172.21.15.161:0/928827995 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e64152890 msgr2=0x7f2e6414db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:48.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.316+0000 7f2e69e9d640 1 --2- 172.21.15.161:0/928827995 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e64152890 0x7f2e6414db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.316+0000 7f2e69e9d640 1 -- 172.21.15.161:0/928827995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e640c8ed0 msgr2=0x7f2e640bed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:48.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.316+0000 7f2e69e9d640 1 --2- 172.21.15.161:0/928827995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e640c8ed0 0x7f2e640bed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.316+0000 7f2e69e9d640 1 -- 172.21.15.161:0/928827995 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e6414e7b0 con 0x7f2e640c5950 2024-03-28T23:32:48.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.316+0000 7f2e69e9d640 1 --2- 172.21.15.161:0/928827995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e640c5950 0x7f2e641ee790 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f2e54000c00 tx=0x7f2e540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:48.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.317+0000 7f2e5a7fc640 1 -- 172.21.15.161:0/928827995 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e54042070 con 0x7f2e640c5950 2024-03-28T23:32:48.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.317+0000 7f2e5a7fc640 1 -- 172.21.15.161:0/928827995 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e5403d870 con 0x7f2e640c5950 2024-03-28T23:32:48.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.317+0000 7f2e5a7fc640 1 -- 172.21.15.161:0/928827995 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e54038040 con 0x7f2e640c5950 2024-03-28T23:32:48.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.317+0000 7f2e6c101640 1 -- 172.21.15.161:0/928827995 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e6414cb30 con 0x7f2e640c5950 2024-03-28T23:32:48.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.317+0000 7f2e6c101640 1 -- 172.21.15.161:0/928827995 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e6414d0b0 con 0x7f2e640c5950 2024-03-28T23:32:48.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.318+0000 7f2e6c101640 1 -- 172.21.15.161:0/928827995 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e6404f4b0 con 0x7f2e640c5950 2024-03-28T23:32:48.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.318+0000 7f2e5a7fc640 1 -- 172.21.15.161:0/928827995 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2e5403b070 con 0x7f2e640c5950 2024-03-28T23:32:48.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.319+0000 7f2e5a7fc640 1 --2- 172.21.15.161:0/928827995 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e4806b4d0 0x7f2e4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:48.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.319+0000 7f2e6969c640 1 --2- 172.21.15.161:0/928827995 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e4806b4d0 0x7f2e4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:48.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.319+0000 7f2e5a7fc640 1 -- 172.21.15.161:0/928827995 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6957+0+0 (secure 0 0 0) 0x7f2e540abbd0 con 0x7f2e640c5950 2024-03-28T23:32:48.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.319+0000 7f2e6969c640 1 --2- 172.21.15.161:0/928827995 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e4806b4d0 0x7f2e4806d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f2e6414bdd0 tx=0x7f2e60009110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:48.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.323+0000 7f2e5a7fc640 1 -- 172.21.15.161:0/928827995 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2e6414e7b0 con 0x7f2e640c5950 2024-03-28T23:32:48.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.497+0000 7f2e6c101640 1 -- 172.21.15.161:0/928827995 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2e640c8140 con 0x7f2e4806b4d0 2024-03-28T23:32:48.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.508+0000 7f2e5a7fc640 1 -- 172.21.15.161:0/928827995 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+485419 (secure 0 0 0) 0x7f2e640c8140 con 0x7f2e4806b4d0 2024-03-28T23:32:48.514 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:48.517 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:32:48.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.512+0000 7f2e6c101640 1 -- 172.21.15.161:0/928827995 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e4806b4d0 msgr2=0x7f2e4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:48.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.512+0000 7f2e6c101640 1 --2- 172.21.15.161:0/928827995 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e4806b4d0 0x7f2e4806d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f2e6414bdd0 tx=0x7f2e60009110 comp rx=0 tx=0).stop 2024-03-28T23:32:48.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.512+0000 7f2e6c101640 1 -- 172.21.15.161:0/928827995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e640c5950 msgr2=0x7f2e641ee790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:48.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.512+0000 7f2e6c101640 1 --2- 172.21.15.161:0/928827995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e640c5950 0x7f2e641ee790 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f2e54000c00 tx=0x7f2e540046c0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.512+0000 7f2e6c101640 1 -- 172.21.15.161:0/928827995 shutdown_connections 2024-03-28T23:32:48.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.512+0000 7f2e6c101640 1 --2- 172.21.15.161:0/928827995 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e4806b4d0 0x7f2e4806d990 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.512+0000 7f2e6c101640 1 --2- 172.21.15.161:0/928827995 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e64152890 0x7f2e6414db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.512+0000 7f2e6c101640 1 --2- 172.21.15.161:0/928827995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e640c8ed0 0x7f2e640bed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.512+0000 7f2e6c101640 1 --2- 172.21.15.161:0/928827995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e640c5950 0x7f2e641ee790 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.512+0000 7f2e6c101640 1 -- 172.21.15.161:0/928827995 >> 172.21.15.161:0/928827995 conn(0x7f2e640ac620 msgr2=0x7f2e640c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:48.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.512+0000 7f2e6c101640 1 -- 172.21.15.161:0/928827995 shutdown_connections 2024-03-28T23:32:48.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.512+0000 7f2e6c101640 1 -- 172.21.15.161:0/928827995 wait complete. 2024-03-28T23:32:48.523 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-03-28T23:32:47.345901+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":202,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240076,"kb_used_data":7276,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754734644,"statfs":{"total":773094113280,"available":772848275456,"internally_reserved":0,"allocated":7450624,"data_stored":4616622,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.247617"},"pg_stats":[{"pgid":"14.3d","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.110397+0000","last_change":"2024-03-28T23:32:44.110397+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.110397+0000","last_undegraded":"2024-03-28T23:32:44.110397+0000","last_fullsized":"2024-03-28T23:32:44.110397+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3f","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108246+0000","last_change":"2024-03-28T23:32:33.046706+0000","last_active":"2024-03-28T23:32:44.108246+0000","last_peered":"2024-03-28T23:32:44.108246+0000","last_clean":"2024-03-28T23:32:44.108246+0000","last_became_active":"2024-03-28T23:32:33.046222+0000","last_became_peered":"2024-03-28T23:32:33.046222+0000","last_unstale":"2024-03-28T23:32:44.108246+0000","last_undegraded":"2024-03-28T23:32:44.108246+0000","last_fullsized":"2024-03-28T23:32:44.108246+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:02:45.190566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.109309+0000","last_change":"2024-03-28T23:32:44.109309+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.109309+0000","last_undegraded":"2024-03-28T23:32:44.109309+0000","last_fullsized":"2024-03-28T23:32:44.109309+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122914+0000","last_change":"2024-03-28T23:32:33.051174+0000","last_active":"2024-03-28T23:32:44.122914+0000","last_peered":"2024-03-28T23:32:44.122914+0000","last_clean":"2024-03-28T23:32:44.122914+0000","last_became_active":"2024-03-28T23:32:33.050345+0000","last_became_peered":"2024-03-28T23:32:33.050345+0000","last_unstale":"2024-03-28T23:32:44.122914+0000","last_undegraded":"2024-03-28T23:32:44.122914+0000","last_fullsized":"2024-03-28T23:32:44.122914+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:41:39.657731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.3f","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111805+0000","last_change":"2024-03-28T23:32:44.111805+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111805+0000","last_undegraded":"2024-03-28T23:32:44.111805+0000","last_fullsized":"2024-03-28T23:32:44.111805+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108168+0000","last_change":"2024-03-28T23:32:33.048785+0000","last_active":"2024-03-28T23:32:44.108168+0000","last_peered":"2024-03-28T23:32:44.108168+0000","last_clean":"2024-03-28T23:32:44.108168+0000","last_became_active":"2024-03-28T23:32:33.048519+0000","last_became_peered":"2024-03-28T23:32:33.048519+0000","last_unstale":"2024-03-28T23:32:44.108168+0000","last_undegraded":"2024-03-28T23:32:44.108168+0000","last_fullsized":"2024-03-28T23:32:44.108168+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:21.464120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111326+0000","last_change":"2024-03-28T23:32:44.111326+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111326+0000","last_undegraded":"2024-03-28T23:32:44.111326+0000","last_fullsized":"2024-03-28T23:32:44.111326+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106875+0000","last_change":"2024-03-28T23:32:33.033535+0000","last_active":"2024-03-28T23:32:44.106875+0000","last_peered":"2024-03-28T23:32:44.106875+0000","last_clean":"2024-03-28T23:32:44.106875+0000","last_became_active":"2024-03-28T23:32:33.033219+0000","last_became_peered":"2024-03-28T23:32:33.033219+0000","last_unstale":"2024-03-28T23:32:44.106875+0000","last_undegraded":"2024-03-28T23:32:44.106875+0000","last_fullsized":"2024-03-28T23:32:44.106875+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:39:39.113457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.106821+0000","last_change":"2024-03-28T23:32:44.106821+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.106821+0000","last_undegraded":"2024-03-28T23:32:44.106821+0000","last_fullsized":"2024-03-28T23:32:44.106821+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109010+0000","last_change":"2024-03-28T23:32:33.042972+0000","last_active":"2024-03-28T23:32:44.109010+0000","last_peered":"2024-03-28T23:32:44.109010+0000","last_clean":"2024-03-28T23:32:44.109010+0000","last_became_active":"2024-03-28T23:32:33.042665+0000","last_became_peered":"2024-03-28T23:32:33.042665+0000","last_unstale":"2024-03-28T23:32:44.109010+0000","last_undegraded":"2024-03-28T23:32:44.109010+0000","last_fullsized":"2024-03-28T23:32:44.109010+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:41:50.190883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.109201+0000","last_change":"2024-03-28T23:32:44.109201+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.109201+0000","last_undegraded":"2024-03-28T23:32:44.109201+0000","last_fullsized":"2024-03-28T23:32:44.109201+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113352+0000","last_change":"2024-03-28T23:32:33.030549+0000","last_active":"2024-03-28T23:32:44.113352+0000","last_peered":"2024-03-28T23:32:44.113352+0000","last_clean":"2024-03-28T23:32:44.113352+0000","last_became_active":"2024-03-28T23:32:33.030299+0000","last_became_peered":"2024-03-28T23:32:33.030299+0000","last_unstale":"2024-03-28T23:32:44.113352+0000","last_undegraded":"2024-03-28T23:32:44.113352+0000","last_fullsized":"2024-03-28T23:32:44.113352+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:35.055035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111404+0000","last_change":"2024-03-28T23:32:44.111404+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111404+0000","last_undegraded":"2024-03-28T23:32:44.111404+0000","last_fullsized":"2024-03-28T23:32:44.111404+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.111244+0000","last_change":"2024-03-28T23:32:33.044477+0000","last_active":"2024-03-28T23:32:44.111244+0000","last_peered":"2024-03-28T23:32:44.111244+0000","last_clean":"2024-03-28T23:32:44.111244+0000","last_became_active":"2024-03-28T23:32:33.043960+0000","last_became_peered":"2024-03-28T23:32:33.043960+0000","last_unstale":"2024-03-28T23:32:44.111244+0000","last_undegraded":"2024-03-28T23:32:44.111244+0000","last_fullsized":"2024-03-28T23:32:44.111244+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:21.464120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.113215+0000","last_change":"2024-03-28T23:32:44.113215+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.113215+0000","last_undegraded":"2024-03-28T23:32:44.113215+0000","last_fullsized":"2024-03-28T23:32:44.113215+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108736+0000","last_change":"2024-03-28T23:32:33.052803+0000","last_active":"2024-03-28T23:32:44.108736+0000","last_peered":"2024-03-28T23:32:44.108736+0000","last_clean":"2024-03-28T23:32:44.108736+0000","last_became_active":"2024-03-28T23:32:33.052524+0000","last_became_peered":"2024-03-28T23:32:33.052524+0000","last_unstale":"2024-03-28T23:32:44.108736+0000","last_undegraded":"2024-03-28T23:32:44.108736+0000","last_fullsized":"2024-03-28T23:32:44.108736+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:19:09.500642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.117756+0000","last_change":"2024-03-28T23:32:44.117756+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.117756+0000","last_undegraded":"2024-03-28T23:32:44.117756+0000","last_fullsized":"2024-03-28T23:32:44.117756+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112756+0000","last_change":"2024-03-28T23:32:33.042587+0000","last_active":"2024-03-28T23:32:44.112756+0000","last_peered":"2024-03-28T23:32:44.112756+0000","last_clean":"2024-03-28T23:32:44.112756+0000","last_became_active":"2024-03-28T23:32:33.037002+0000","last_became_peered":"2024-03-28T23:32:33.037002+0000","last_unstale":"2024-03-28T23:32:44.112756+0000","last_undegraded":"2024-03-28T23:32:44.112756+0000","last_fullsized":"2024-03-28T23:32:44.112756+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:36:03.951054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107890+0000","last_change":"2024-03-28T23:32:44.107890+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107890+0000","last_undegraded":"2024-03-28T23:32:44.107890+0000","last_fullsized":"2024-03-28T23:32:44.107890+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107498+0000","last_change":"2024-03-28T23:32:33.039941+0000","last_active":"2024-03-28T23:32:44.107498+0000","last_peered":"2024-03-28T23:32:44.107498+0000","last_clean":"2024-03-28T23:32:44.107498+0000","last_became_active":"2024-03-28T23:32:33.039656+0000","last_became_peered":"2024-03-28T23:32:33.039656+0000","last_unstale":"2024-03-28T23:32:44.107498+0000","last_undegraded":"2024-03-28T23:32:44.107498+0000","last_fullsized":"2024-03-28T23:32:44.107498+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:16.850667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111858+0000","last_change":"2024-03-28T23:32:44.111858+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111858+0000","last_undegraded":"2024-03-28T23:32:44.111858+0000","last_fullsized":"2024-03-28T23:32:44.111858+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122690+0000","last_change":"2024-03-28T23:32:33.052329+0000","last_active":"2024-03-28T23:32:44.122690+0000","last_peered":"2024-03-28T23:32:44.122690+0000","last_clean":"2024-03-28T23:32:44.122690+0000","last_became_active":"2024-03-28T23:32:33.051833+0000","last_became_peered":"2024-03-28T23:32:33.051833+0000","last_unstale":"2024-03-28T23:32:44.122690+0000","last_undegraded":"2024-03-28T23:32:44.122690+0000","last_fullsized":"2024-03-28T23:32:44.122690+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:05.929331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108222+0000","last_change":"2024-03-28T23:32:44.108222+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108222+0000","last_undegraded":"2024-03-28T23:32:44.108222+0000","last_fullsized":"2024-03-28T23:32:44.108222+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106984+0000","last_change":"2024-03-28T23:32:33.042337+0000","last_active":"2024-03-28T23:32:44.106984+0000","last_peered":"2024-03-28T23:32:44.106984+0000","last_clean":"2024-03-28T23:32:44.106984+0000","last_became_active":"2024-03-28T23:32:33.042062+0000","last_became_peered":"2024-03-28T23:32:33.042062+0000","last_unstale":"2024-03-28T23:32:44.106984+0000","last_undegraded":"2024-03-28T23:32:44.106984+0000","last_fullsized":"2024-03-28T23:32:44.106984+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:13:34.921740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108039+0000","last_change":"2024-03-28T23:32:44.108039+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108039+0000","last_undegraded":"2024-03-28T23:32:44.108039+0000","last_fullsized":"2024-03-28T23:32:44.108039+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109102+0000","last_change":"2024-03-28T23:32:33.046685+0000","last_active":"2024-03-28T23:32:44.109102+0000","last_peered":"2024-03-28T23:32:44.109102+0000","last_clean":"2024-03-28T23:32:44.109102+0000","last_became_active":"2024-03-28T23:32:33.046206+0000","last_became_peered":"2024-03-28T23:32:33.046206+0000","last_unstale":"2024-03-28T23:32:44.109102+0000","last_undegraded":"2024-03-28T23:32:44.109102+0000","last_fullsized":"2024-03-28T23:32:44.109102+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:43:22.001147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.113196+0000","last_change":"2024-03-28T23:32:44.113196+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.113196+0000","last_undegraded":"2024-03-28T23:32:44.113196+0000","last_fullsized":"2024-03-28T23:32:44.113196+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113445+0000","last_change":"2024-03-28T23:32:33.029880+0000","last_active":"2024-03-28T23:32:44.113445+0000","last_peered":"2024-03-28T23:32:44.113445+0000","last_clean":"2024-03-28T23:32:44.113445+0000","last_became_active":"2024-03-28T23:32:33.029582+0000","last_became_peered":"2024-03-28T23:32:33.029582+0000","last_unstale":"2024-03-28T23:32:44.113445+0000","last_undegraded":"2024-03-28T23:32:44.113445+0000","last_fullsized":"2024-03-28T23:32:44.113445+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:38:39.283932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111833+0000","last_change":"2024-03-28T23:32:44.111833+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111833+0000","last_undegraded":"2024-03-28T23:32:44.111833+0000","last_fullsized":"2024-03-28T23:32:44.111833+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121362+0000","last_change":"2024-03-28T23:32:33.040102+0000","last_active":"2024-03-28T23:32:44.121362+0000","last_peered":"2024-03-28T23:32:44.121362+0000","last_clean":"2024-03-28T23:32:44.121362+0000","last_became_active":"2024-03-28T23:32:33.039660+0000","last_became_peered":"2024-03-28T23:32:33.039660+0000","last_unstale":"2024-03-28T23:32:44.121362+0000","last_undegraded":"2024-03-28T23:32:44.121362+0000","last_fullsized":"2024-03-28T23:32:44.121362+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:35.055035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112037+0000","last_change":"2024-03-28T23:32:44.112037+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112037+0000","last_undegraded":"2024-03-28T23:32:44.112037+0000","last_fullsized":"2024-03-28T23:32:44.112037+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108875+0000","last_change":"2024-03-28T23:32:33.035442+0000","last_active":"2024-03-28T23:32:44.108875+0000","last_peered":"2024-03-28T23:32:44.108875+0000","last_clean":"2024-03-28T23:32:44.108875+0000","last_became_active":"2024-03-28T23:32:33.035154+0000","last_became_peered":"2024-03-28T23:32:33.035154+0000","last_unstale":"2024-03-28T23:32:44.108875+0000","last_undegraded":"2024-03-28T23:32:44.108875+0000","last_fullsized":"2024-03-28T23:32:44.108875+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:34:22.775718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.115970+0000","last_change":"2024-03-28T23:32:44.115970+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.115970+0000","last_undegraded":"2024-03-28T23:32:44.115970+0000","last_fullsized":"2024-03-28T23:32:44.115970+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.111211+0000","last_change":"2024-03-28T23:32:33.047782+0000","last_active":"2024-03-28T23:32:44.111211+0000","last_peered":"2024-03-28T23:32:44.111211+0000","last_clean":"2024-03-28T23:32:44.111211+0000","last_became_active":"2024-03-28T23:32:33.047254+0000","last_became_peered":"2024-03-28T23:32:33.047254+0000","last_unstale":"2024-03-28T23:32:44.111211+0000","last_undegraded":"2024-03-28T23:32:44.111211+0000","last_fullsized":"2024-03-28T23:32:44.111211+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:19.004555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108940+0000","last_change":"2024-03-28T23:32:44.108940+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108940+0000","last_undegraded":"2024-03-28T23:32:44.108940+0000","last_fullsized":"2024-03-28T23:32:44.108940+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107847+0000","last_change":"2024-03-28T23:32:33.037321+0000","last_active":"2024-03-28T23:32:44.107847+0000","last_peered":"2024-03-28T23:32:44.107847+0000","last_clean":"2024-03-28T23:32:44.107847+0000","last_became_active":"2024-03-28T23:32:33.036961+0000","last_became_peered":"2024-03-28T23:32:33.036961+0000","last_unstale":"2024-03-28T23:32:44.107847+0000","last_undegraded":"2024-03-28T23:32:44.107847+0000","last_fullsized":"2024-03-28T23:32:44.107847+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:25.849919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.120323+0000","last_change":"2024-03-28T23:32:44.120323+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.120323+0000","last_undegraded":"2024-03-28T23:32:44.120323+0000","last_fullsized":"2024-03-28T23:32:44.120323+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113594+0000","last_change":"2024-03-28T23:32:33.053071+0000","last_active":"2024-03-28T23:32:44.113594+0000","last_peered":"2024-03-28T23:32:44.113594+0000","last_clean":"2024-03-28T23:32:44.113594+0000","last_became_active":"2024-03-28T23:32:33.052784+0000","last_became_peered":"2024-03-28T23:32:33.052784+0000","last_unstale":"2024-03-28T23:32:44.113594+0000","last_undegraded":"2024-03-28T23:32:44.113594+0000","last_fullsized":"2024-03-28T23:32:44.113594+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:58:48.231248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112002+0000","last_change":"2024-03-28T23:32:44.112002+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112002+0000","last_undegraded":"2024-03-28T23:32:44.112002+0000","last_fullsized":"2024-03-28T23:32:44.112002+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109678+0000","last_change":"2024-03-28T23:32:33.052720+0000","last_active":"2024-03-28T23:32:44.109678+0000","last_peered":"2024-03-28T23:32:44.109678+0000","last_clean":"2024-03-28T23:32:44.109678+0000","last_became_active":"2024-03-28T23:32:33.052460+0000","last_became_peered":"2024-03-28T23:32:33.052460+0000","last_unstale":"2024-03-28T23:32:44.109678+0000","last_undegraded":"2024-03-28T23:32:44.109678+0000","last_fullsized":"2024-03-28T23:32:44.109678+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:30.891315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107873+0000","last_change":"2024-03-28T23:32:44.107873+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107873+0000","last_undegraded":"2024-03-28T23:32:44.107873+0000","last_fullsized":"2024-03-28T23:32:44.107873+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112648+0000","last_change":"2024-03-28T23:32:33.035666+0000","last_active":"2024-03-28T23:32:44.112648+0000","last_peered":"2024-03-28T23:32:44.112648+0000","last_clean":"2024-03-28T23:32:44.112648+0000","last_became_active":"2024-03-28T23:32:33.035403+0000","last_became_peered":"2024-03-28T23:32:33.035403+0000","last_unstale":"2024-03-28T23:32:44.112648+0000","last_undegraded":"2024-03-28T23:32:44.112648+0000","last_fullsized":"2024-03-28T23:32:44.112648+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:40:51.269760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111443+0000","last_change":"2024-03-28T23:32:44.111443+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111443+0000","last_undegraded":"2024-03-28T23:32:44.111443+0000","last_fullsized":"2024-03-28T23:32:44.111443+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109244+0000","last_change":"2024-03-28T23:32:33.047992+0000","last_active":"2024-03-28T23:32:44.109244+0000","last_peered":"2024-03-28T23:32:44.109244+0000","last_clean":"2024-03-28T23:32:44.109244+0000","last_became_active":"2024-03-28T23:32:33.047058+0000","last_became_peered":"2024-03-28T23:32:33.047058+0000","last_unstale":"2024-03-28T23:32:44.109244+0000","last_undegraded":"2024-03-28T23:32:44.109244+0000","last_fullsized":"2024-03-28T23:32:44.109244+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:35:30.886935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.110595+0000","last_change":"2024-03-28T23:32:44.110595+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.110595+0000","last_undegraded":"2024-03-28T23:32:44.110595+0000","last_fullsized":"2024-03-28T23:32:44.110595+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112366+0000","last_change":"2024-03-28T23:32:33.029536+0000","last_active":"2024-03-28T23:32:44.112366+0000","last_peered":"2024-03-28T23:32:44.112366+0000","last_clean":"2024-03-28T23:32:44.112366+0000","last_became_active":"2024-03-28T23:32:33.029265+0000","last_became_peered":"2024-03-28T23:32:33.029265+0000","last_unstale":"2024-03-28T23:32:44.112366+0000","last_undegraded":"2024-03-28T23:32:44.112366+0000","last_fullsized":"2024-03-28T23:32:44.112366+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:51:11.447507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.110858+0000","last_change":"2024-03-28T23:32:44.110858+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.110858+0000","last_undegraded":"2024-03-28T23:32:44.110858+0000","last_fullsized":"2024-03-28T23:32:44.110858+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121488+0000","last_change":"2024-03-28T23:32:33.041328+0000","last_active":"2024-03-28T23:32:44.121488+0000","last_peered":"2024-03-28T23:32:44.121488+0000","last_clean":"2024-03-28T23:32:44.121488+0000","last_became_active":"2024-03-28T23:32:33.041071+0000","last_became_peered":"2024-03-28T23:32:33.041071+0000","last_unstale":"2024-03-28T23:32:44.121488+0000","last_undegraded":"2024-03-28T23:32:44.121488+0000","last_fullsized":"2024-03-28T23:32:44.121488+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:00.358180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107392+0000","last_change":"2024-03-28T23:32:44.107392+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107392+0000","last_undegraded":"2024-03-28T23:32:44.107392+0000","last_fullsized":"2024-03-28T23:32:44.107392+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122766+0000","last_change":"2024-03-28T23:32:33.043840+0000","last_active":"2024-03-28T23:32:44.122766+0000","last_peered":"2024-03-28T23:32:44.122766+0000","last_clean":"2024-03-28T23:32:44.122766+0000","last_became_active":"2024-03-28T23:32:33.043569+0000","last_became_peered":"2024-03-28T23:32:33.043569+0000","last_unstale":"2024-03-28T23:32:44.122766+0000","last_undegraded":"2024-03-28T23:32:44.122766+0000","last_fullsized":"2024-03-28T23:32:44.122766+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:28.463493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108552+0000","last_change":"2024-03-28T23:32:44.108552+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108552+0000","last_undegraded":"2024-03-28T23:32:44.108552+0000","last_fullsized":"2024-03-28T23:32:44.108552+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107631+0000","last_change":"2024-03-28T23:32:33.047594+0000","last_active":"2024-03-28T23:32:44.107631+0000","last_peered":"2024-03-28T23:32:44.107631+0000","last_clean":"2024-03-28T23:32:44.107631+0000","last_became_active":"2024-03-28T23:32:33.047031+0000","last_became_peered":"2024-03-28T23:32:33.047031+0000","last_unstale":"2024-03-28T23:32:44.107631+0000","last_undegraded":"2024-03-28T23:32:44.107631+0000","last_fullsized":"2024-03-28T23:32:44.107631+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:30:12.769818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112271+0000","last_change":"2024-03-28T23:32:44.112271+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112271+0000","last_undegraded":"2024-03-28T23:32:44.112271+0000","last_fullsized":"2024-03-28T23:32:44.112271+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107919+0000","last_change":"2024-03-28T23:32:33.043396+0000","last_active":"2024-03-28T23:32:44.107919+0000","last_peered":"2024-03-28T23:32:44.107919+0000","last_clean":"2024-03-28T23:32:44.107919+0000","last_became_active":"2024-03-28T23:32:33.043123+0000","last_became_peered":"2024-03-28T23:32:33.043123+0000","last_unstale":"2024-03-28T23:32:44.107919+0000","last_undegraded":"2024-03-28T23:32:44.107919+0000","last_fullsized":"2024-03-28T23:32:44.107919+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:28:36.535909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.114282+0000","last_change":"2024-03-28T23:32:44.114282+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.114282+0000","last_undegraded":"2024-03-28T23:32:44.114282+0000","last_fullsized":"2024-03-28T23:32:44.114282+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109350+0000","last_change":"2024-03-28T23:32:33.042288+0000","last_active":"2024-03-28T23:32:44.109350+0000","last_peered":"2024-03-28T23:32:44.109350+0000","last_clean":"2024-03-28T23:32:44.109350+0000","last_became_active":"2024-03-28T23:32:33.042007+0000","last_became_peered":"2024-03-28T23:32:33.042007+0000","last_unstale":"2024-03-28T23:32:44.109350+0000","last_undegraded":"2024-03-28T23:32:44.109350+0000","last_fullsized":"2024-03-28T23:32:44.109350+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:16:12.634582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.110873+0000","last_change":"2024-03-28T23:32:44.110873+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.110873+0000","last_undegraded":"2024-03-28T23:32:44.110873+0000","last_fullsized":"2024-03-28T23:32:44.110873+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122615+0000","last_change":"2024-03-28T23:32:33.041336+0000","last_active":"2024-03-28T23:32:44.122615+0000","last_peered":"2024-03-28T23:32:44.122615+0000","last_clean":"2024-03-28T23:32:44.122615+0000","last_became_active":"2024-03-28T23:32:33.040865+0000","last_became_peered":"2024-03-28T23:32:33.040865+0000","last_unstale":"2024-03-28T23:32:44.122615+0000","last_undegraded":"2024-03-28T23:32:44.122615+0000","last_fullsized":"2024-03-28T23:32:44.122615+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:44:58.784172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.117069+0000","last_change":"2024-03-28T23:32:44.117069+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.117069+0000","last_undegraded":"2024-03-28T23:32:44.117069+0000","last_fullsized":"2024-03-28T23:32:44.117069+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106185+0000","last_change":"2024-03-28T23:32:33.049222+0000","last_active":"2024-03-28T23:32:44.106185+0000","last_peered":"2024-03-28T23:32:44.106185+0000","last_clean":"2024-03-28T23:32:44.106185+0000","last_became_active":"2024-03-28T23:32:33.048933+0000","last_became_peered":"2024-03-28T23:32:33.048933+0000","last_unstale":"2024-03-28T23:32:44.106185+0000","last_undegraded":"2024-03-28T23:32:44.106185+0000","last_fullsized":"2024-03-28T23:32:44.106185+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:17:45.545141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107516+0000","last_change":"2024-03-28T23:32:44.107516+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107516+0000","last_undegraded":"2024-03-28T23:32:44.107516+0000","last_fullsized":"2024-03-28T23:32:44.107516+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108405+0000","last_change":"2024-03-28T23:32:33.047977+0000","last_active":"2024-03-28T23:32:44.108405+0000","last_peered":"2024-03-28T23:32:44.108405+0000","last_clean":"2024-03-28T23:32:44.108405+0000","last_became_active":"2024-03-28T23:32:33.047699+0000","last_became_peered":"2024-03-28T23:32:33.047699+0000","last_unstale":"2024-03-28T23:32:44.108405+0000","last_undegraded":"2024-03-28T23:32:44.108405+0000","last_fullsized":"2024-03-28T23:32:44.108405+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:38:51.513907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111342+0000","last_change":"2024-03-28T23:32:44.111342+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111342+0000","last_undegraded":"2024-03-28T23:32:44.111342+0000","last_fullsized":"2024-03-28T23:32:44.111342+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108988+0000","last_change":"2024-03-28T23:32:33.036527+0000","last_active":"2024-03-28T23:32:44.108988+0000","last_peered":"2024-03-28T23:32:44.108988+0000","last_clean":"2024-03-28T23:32:44.108988+0000","last_became_active":"2024-03-28T23:32:33.036269+0000","last_became_peered":"2024-03-28T23:32:33.036269+0000","last_unstale":"2024-03-28T23:32:44.108988+0000","last_undegraded":"2024-03-28T23:32:44.108988+0000","last_fullsized":"2024-03-28T23:32:44.108988+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:43:35.946992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112540+0000","last_change":"2024-03-28T23:32:44.112540+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112540+0000","last_undegraded":"2024-03-28T23:32:44.112540+0000","last_fullsized":"2024-03-28T23:32:44.112540+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121867+0000","last_change":"2024-03-28T23:32:44.118134+0000","last_active":"2024-03-28T23:32:44.121867+0000","last_peered":"2024-03-28T23:32:44.121867+0000","last_clean":"2024-03-28T23:32:44.121867+0000","last_became_active":"2024-03-28T23:32:44.117870+0000","last_became_peered":"2024-03-28T23:32:44.117870+0000","last_unstale":"2024-03-28T23:32:44.121867+0000","last_undegraded":"2024-03-28T23:32:44.121867+0000","last_fullsized":"2024-03-28T23:32:44.121867+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:41:04.213333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109389+0000","last_change":"2024-03-28T23:32:33.038016+0000","last_active":"2024-03-28T23:32:44.109389+0000","last_peered":"2024-03-28T23:32:44.109389+0000","last_clean":"2024-03-28T23:32:44.109389+0000","last_became_active":"2024-03-28T23:32:33.037754+0000","last_became_peered":"2024-03-28T23:32:33.037754+0000","last_unstale":"2024-03-28T23:32:44.109389+0000","last_undegraded":"2024-03-28T23:32:44.109389+0000","last_fullsized":"2024-03-28T23:32:44.109389+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:21.878156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"72'4","reported_seq":22,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.111297+0000","last_change":"2024-03-28T23:32:32.040468+0000","last_active":"2024-03-28T23:32:44.111297+0000","last_peered":"2024-03-28T23:32:44.111297+0000","last_clean":"2024-03-28T23:32:44.111297+0000","last_became_active":"2024-03-28T23:32:32.039342+0000","last_became_peered":"2024-03-28T23:32:32.039342+0000","last_unstale":"2024-03-28T23:32:44.111297+0000","last_undegraded":"2024-03-28T23:32:44.111297+0000","last_fullsized":"2024-03-28T23:32:44.111297+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:05.786248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108504+0000","last_change":"2024-03-28T23:32:44.108504+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108504+0000","last_undegraded":"2024-03-28T23:32:44.108504+0000","last_fullsized":"2024-03-28T23:32:44.108504+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.127347+0000","last_change":"2024-03-28T23:32:44.126084+0000","last_active":"2024-03-28T23:32:44.127347+0000","last_peered":"2024-03-28T23:32:44.127347+0000","last_clean":"2024-03-28T23:32:44.127347+0000","last_became_active":"2024-03-28T23:32:44.125728+0000","last_became_peered":"2024-03-28T23:32:44.125728+0000","last_unstale":"2024-03-28T23:32:44.127347+0000","last_undegraded":"2024-03-28T23:32:44.127347+0000","last_fullsized":"2024-03-28T23:32:44.127347+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:17.008056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121405+0000","last_change":"2024-03-28T23:32:33.039778+0000","last_active":"2024-03-28T23:32:44.121405+0000","last_peered":"2024-03-28T23:32:44.121405+0000","last_clean":"2024-03-28T23:32:44.121405+0000","last_became_active":"2024-03-28T23:32:33.039506+0000","last_became_peered":"2024-03-28T23:32:33.039506+0000","last_unstale":"2024-03-28T23:32:44.121405+0000","last_undegraded":"2024-03-28T23:32:44.121405+0000","last_fullsized":"2024-03-28T23:32:44.121405+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:38:39.283932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"72'1","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107225+0000","last_change":"2024-03-28T23:32:32.041093+0000","last_active":"2024-03-28T23:32:44.107225+0000","last_peered":"2024-03-28T23:32:44.107225+0000","last_clean":"2024-03-28T23:32:44.107225+0000","last_became_active":"2024-03-28T23:32:32.040827+0000","last_became_peered":"2024-03-28T23:32:32.040827+0000","last_unstale":"2024-03-28T23:32:44.107225+0000","last_undegraded":"2024-03-28T23:32:44.107225+0000","last_fullsized":"2024-03-28T23:32:44.107225+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:55.544945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108669+0000","last_change":"2024-03-28T23:32:44.108669+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108669+0000","last_undegraded":"2024-03-28T23:32:44.108669+0000","last_fullsized":"2024-03-28T23:32:44.108669+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.120787+0000","last_change":"2024-03-28T23:32:44.119556+0000","last_active":"2024-03-28T23:32:44.120787+0000","last_peered":"2024-03-28T23:32:44.120787+0000","last_clean":"2024-03-28T23:32:44.120787+0000","last_became_active":"2024-03-28T23:32:44.119246+0000","last_became_peered":"2024-03-28T23:32:44.119246+0000","last_unstale":"2024-03-28T23:32:44.120787+0000","last_undegraded":"2024-03-28T23:32:44.120787+0000","last_fullsized":"2024-03-28T23:32:44.120787+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:36.928644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112355+0000","last_change":"2024-03-28T23:32:33.044887+0000","last_active":"2024-03-28T23:32:44.112355+0000","last_peered":"2024-03-28T23:32:44.112355+0000","last_clean":"2024-03-28T23:32:44.112355+0000","last_became_active":"2024-03-28T23:32:33.042005+0000","last_became_peered":"2024-03-28T23:32:33.042005+0000","last_unstale":"2024-03-28T23:32:44.112355+0000","last_undegraded":"2024-03-28T23:32:44.112355+0000","last_fullsized":"2024-03-28T23:32:44.112355+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:41.257241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"72'2","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108029+0000","last_change":"2024-03-28T23:32:32.040527+0000","last_active":"2024-03-28T23:32:44.108029+0000","last_peered":"2024-03-28T23:32:44.108029+0000","last_clean":"2024-03-28T23:32:44.108029+0000","last_became_active":"2024-03-28T23:32:32.040142+0000","last_became_peered":"2024-03-28T23:32:32.040142+0000","last_unstale":"2024-03-28T23:32:44.108029+0000","last_undegraded":"2024-03-28T23:32:44.108029+0000","last_fullsized":"2024-03-28T23:32:44.108029+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:13:15.856750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107395+0000","last_change":"2024-03-28T23:32:44.107395+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107395+0000","last_undegraded":"2024-03-28T23:32:44.107395+0000","last_fullsized":"2024-03-28T23:32:44.107395+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122155+0000","last_change":"2024-03-28T23:32:44.116203+0000","last_active":"2024-03-28T23:32:44.122155+0000","last_peered":"2024-03-28T23:32:44.122155+0000","last_clean":"2024-03-28T23:32:44.122155+0000","last_became_active":"2024-03-28T23:32:44.115846+0000","last_became_peered":"2024-03-28T23:32:44.115846+0000","last_unstale":"2024-03-28T23:32:44.122155+0000","last_undegraded":"2024-03-28T23:32:44.122155+0000","last_fullsized":"2024-03-28T23:32:44.122155+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:58.429743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112741+0000","last_change":"2024-03-28T23:32:33.040988+0000","last_active":"2024-03-28T23:32:44.112741+0000","last_peered":"2024-03-28T23:32:44.112741+0000","last_clean":"2024-03-28T23:32:44.112741+0000","last_became_active":"2024-03-28T23:32:33.036045+0000","last_became_peered":"2024-03-28T23:32:33.036045+0000","last_unstale":"2024-03-28T23:32:44.112741+0000","last_undegraded":"2024-03-28T23:32:44.112741+0000","last_fullsized":"2024-03-28T23:32:44.112741+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:10:39.981362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"72'2","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112781+0000","last_change":"2024-03-28T23:32:32.036856+0000","last_active":"2024-03-28T23:32:44.112781+0000","last_peered":"2024-03-28T23:32:44.112781+0000","last_clean":"2024-03-28T23:32:44.112781+0000","last_became_active":"2024-03-28T23:32:32.036565+0000","last_became_peered":"2024-03-28T23:32:32.036565+0000","last_unstale":"2024-03-28T23:32:44.112781+0000","last_undegraded":"2024-03-28T23:32:44.112781+0000","last_fullsized":"2024-03-28T23:32:44.112781+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:11:01.106436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.106357+0000","last_change":"2024-03-28T23:32:44.106357+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.106357+0000","last_undegraded":"2024-03-28T23:32:44.106357+0000","last_fullsized":"2024-03-28T23:32:44.106357+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.118200+0000","last_change":"2024-03-28T23:32:44.116097+0000","last_active":"2024-03-28T23:32:44.118200+0000","last_peered":"2024-03-28T23:32:44.118200+0000","last_clean":"2024-03-28T23:32:44.118200+0000","last_became_active":"2024-03-28T23:32:44.115810+0000","last_became_peered":"2024-03-28T23:32:44.115810+0000","last_unstale":"2024-03-28T23:32:44.118200+0000","last_undegraded":"2024-03-28T23:32:44.118200+0000","last_fullsized":"2024-03-28T23:32:44.118200+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:45:09.862897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122437+0000","last_change":"2024-03-28T23:32:33.047911+0000","last_active":"2024-03-28T23:32:44.122437+0000","last_peered":"2024-03-28T23:32:44.122437+0000","last_clean":"2024-03-28T23:32:44.122437+0000","last_became_active":"2024-03-28T23:32:33.047616+0000","last_became_peered":"2024-03-28T23:32:33.047616+0000","last_unstale":"2024-03-28T23:32:44.122437+0000","last_undegraded":"2024-03-28T23:32:44.122437+0000","last_fullsized":"2024-03-28T23:32:44.122437+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:58:48.231248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"72'1","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112850+0000","last_change":"2024-03-28T23:32:32.036829+0000","last_active":"2024-03-28T23:32:44.112850+0000","last_peered":"2024-03-28T23:32:44.112850+0000","last_clean":"2024-03-28T23:32:44.112850+0000","last_became_active":"2024-03-28T23:32:32.036571+0000","last_became_peered":"2024-03-28T23:32:32.036571+0000","last_unstale":"2024-03-28T23:32:44.112850+0000","last_undegraded":"2024-03-28T23:32:44.112850+0000","last_fullsized":"2024-03-28T23:32:44.112850+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:32:03.789593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.116621+0000","last_change":"2024-03-28T23:32:44.116621+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.116621+0000","last_undegraded":"2024-03-28T23:32:44.116621+0000","last_fullsized":"2024-03-28T23:32:44.116621+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.120399+0000","last_change":"2024-03-28T23:32:44.117981+0000","last_active":"2024-03-28T23:32:44.120399+0000","last_peered":"2024-03-28T23:32:44.120399+0000","last_clean":"2024-03-28T23:32:44.120399+0000","last_became_active":"2024-03-28T23:32:44.117709+0000","last_became_peered":"2024-03-28T23:32:44.117709+0000","last_unstale":"2024-03-28T23:32:44.120399+0000","last_undegraded":"2024-03-28T23:32:44.120399+0000","last_fullsized":"2024-03-28T23:32:44.120399+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:13:27.942873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122698+0000","last_change":"2024-03-28T23:32:33.052330+0000","last_active":"2024-03-28T23:32:44.122698+0000","last_peered":"2024-03-28T23:32:44.122698+0000","last_clean":"2024-03-28T23:32:44.122698+0000","last_became_active":"2024-03-28T23:32:33.051828+0000","last_became_peered":"2024-03-28T23:32:33.051828+0000","last_unstale":"2024-03-28T23:32:44.122698+0000","last_undegraded":"2024-03-28T23:32:44.122698+0000","last_fullsized":"2024-03-28T23:32:44.122698+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:40:53.134608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"72'1","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106048+0000","last_change":"2024-03-28T23:32:32.036739+0000","last_active":"2024-03-28T23:32:44.106048+0000","last_peered":"2024-03-28T23:32:44.106048+0000","last_clean":"2024-03-28T23:32:44.106048+0000","last_became_active":"2024-03-28T23:32:32.036471+0000","last_became_peered":"2024-03-28T23:32:32.036471+0000","last_unstale":"2024-03-28T23:32:44.106048+0000","last_undegraded":"2024-03-28T23:32:44.106048+0000","last_fullsized":"2024-03-28T23:32:44.106048+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:48.704351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112770+0000","last_change":"2024-03-28T23:32:44.112770+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112770+0000","last_undegraded":"2024-03-28T23:32:44.112770+0000","last_fullsized":"2024-03-28T23:32:44.112770+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.127627+0000","last_change":"2024-03-28T23:32:44.126818+0000","last_active":"2024-03-28T23:32:44.127627+0000","last_peered":"2024-03-28T23:32:44.127627+0000","last_clean":"2024-03-28T23:32:44.127627+0000","last_became_active":"2024-03-28T23:32:44.126493+0000","last_became_peered":"2024-03-28T23:32:44.126493+0000","last_unstale":"2024-03-28T23:32:44.127627+0000","last_undegraded":"2024-03-28T23:32:44.127627+0000","last_fullsized":"2024-03-28T23:32:44.127627+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:06:32.695104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.115074+0000","last_change":"2024-03-28T23:32:33.034126+0000","last_active":"2024-03-28T23:32:44.115074+0000","last_peered":"2024-03-28T23:32:44.115074+0000","last_clean":"2024-03-28T23:32:44.115074+0000","last_became_active":"2024-03-28T23:32:33.033703+0000","last_became_peered":"2024-03-28T23:32:33.033703+0000","last_unstale":"2024-03-28T23:32:44.115074+0000","last_undegraded":"2024-03-28T23:32:44.115074+0000","last_fullsized":"2024-03-28T23:32:44.115074+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:38.859031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121532+0000","last_change":"2024-03-28T23:32:32.039047+0000","last_active":"2024-03-28T23:32:44.121532+0000","last_peered":"2024-03-28T23:32:44.121532+0000","last_clean":"2024-03-28T23:32:44.121532+0000","last_became_active":"2024-03-28T23:32:32.038780+0000","last_became_peered":"2024-03-28T23:32:32.038780+0000","last_unstale":"2024-03-28T23:32:44.121532+0000","last_undegraded":"2024-03-28T23:32:44.121532+0000","last_fullsized":"2024-03-28T23:32:44.121532+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:13:15.856750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.117228+0000","last_change":"2024-03-28T23:32:44.117228+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.117228+0000","last_undegraded":"2024-03-28T23:32:44.117228+0000","last_fullsized":"2024-03-28T23:32:44.117228+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122300+0000","last_change":"2024-03-28T23:32:44.121416+0000","last_active":"2024-03-28T23:32:44.122300+0000","last_peered":"2024-03-28T23:32:44.122300+0000","last_clean":"2024-03-28T23:32:44.122300+0000","last_became_active":"2024-03-28T23:32:44.121127+0000","last_became_peered":"2024-03-28T23:32:44.121127+0000","last_unstale":"2024-03-28T23:32:44.122300+0000","last_undegraded":"2024-03-28T23:32:44.122300+0000","last_fullsized":"2024-03-28T23:32:44.122300+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:39:02.592632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113417+0000","last_change":"2024-03-28T23:32:33.051453+0000","last_active":"2024-03-28T23:32:44.113417+0000","last_peered":"2024-03-28T23:32:44.113417+0000","last_clean":"2024-03-28T23:32:44.113417+0000","last_became_active":"2024-03-28T23:32:33.051155+0000","last_became_peered":"2024-03-28T23:32:33.051155+0000","last_unstale":"2024-03-28T23:32:44.113417+0000","last_undegraded":"2024-03-28T23:32:44.113417+0000","last_fullsized":"2024-03-28T23:32:44.113417+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:36:03.951054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108887+0000","last_change":"2024-03-28T23:32:32.036979+0000","last_active":"2024-03-28T23:32:44.108887+0000","last_peered":"2024-03-28T23:32:44.108887+0000","last_clean":"2024-03-28T23:32:44.108887+0000","last_became_active":"2024-03-28T23:32:32.036710+0000","last_became_peered":"2024-03-28T23:32:32.036710+0000","last_unstale":"2024-03-28T23:32:44.108887+0000","last_undegraded":"2024-03-28T23:32:44.108887+0000","last_fullsized":"2024-03-28T23:32:44.108887+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:46.072852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107489+0000","last_change":"2024-03-28T23:32:44.107489+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107489+0000","last_undegraded":"2024-03-28T23:32:44.107489+0000","last_fullsized":"2024-03-28T23:32:44.107489+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122075+0000","last_change":"2024-03-28T23:32:44.115740+0000","last_active":"2024-03-28T23:32:44.122075+0000","last_peered":"2024-03-28T23:32:44.122075+0000","last_clean":"2024-03-28T23:32:44.122075+0000","last_became_active":"2024-03-28T23:32:44.115341+0000","last_became_peered":"2024-03-28T23:32:44.115341+0000","last_unstale":"2024-03-28T23:32:44.122075+0000","last_undegraded":"2024-03-28T23:32:44.122075+0000","last_fullsized":"2024-03-28T23:32:44.122075+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:55.159548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121499+0000","last_change":"2024-03-28T23:32:33.041923+0000","last_active":"2024-03-28T23:32:44.121499+0000","last_peered":"2024-03-28T23:32:44.121499+0000","last_clean":"2024-03-28T23:32:44.121499+0000","last_became_active":"2024-03-28T23:32:33.041631+0000","last_became_peered":"2024-03-28T23:32:33.041631+0000","last_unstale":"2024-03-28T23:32:44.121499+0000","last_undegraded":"2024-03-28T23:32:44.121499+0000","last_fullsized":"2024-03-28T23:32:44.121499+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:32:08.473276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108352+0000","last_change":"2024-03-28T23:32:32.041563+0000","last_active":"2024-03-28T23:32:44.108352+0000","last_peered":"2024-03-28T23:32:44.108352+0000","last_clean":"2024-03-28T23:32:44.108352+0000","last_became_active":"2024-03-28T23:32:32.040424+0000","last_became_peered":"2024-03-28T23:32:32.040424+0000","last_unstale":"2024-03-28T23:32:44.108352+0000","last_undegraded":"2024-03-28T23:32:44.108352+0000","last_fullsized":"2024-03-28T23:32:44.108352+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:37:25.645113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107400+0000","last_change":"2024-03-28T23:32:44.107400+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107400+0000","last_undegraded":"2024-03-28T23:32:44.107400+0000","last_fullsized":"2024-03-28T23:32:44.107400+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.123301+0000","last_change":"2024-03-28T23:32:44.118040+0000","last_active":"2024-03-28T23:32:44.123301+0000","last_peered":"2024-03-28T23:32:44.123301+0000","last_clean":"2024-03-28T23:32:44.123301+0000","last_became_active":"2024-03-28T23:32:44.117536+0000","last_became_peered":"2024-03-28T23:32:44.117536+0000","last_unstale":"2024-03-28T23:32:44.123301+0000","last_undegraded":"2024-03-28T23:32:44.123301+0000","last_fullsized":"2024-03-28T23:32:44.123301+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:59:07.631069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106030+0000","last_change":"2024-03-28T23:32:33.029986+0000","last_active":"2024-03-28T23:32:44.106030+0000","last_peered":"2024-03-28T23:32:44.106030+0000","last_clean":"2024-03-28T23:32:44.106030+0000","last_became_active":"2024-03-28T23:32:33.029413+0000","last_became_peered":"2024-03-28T23:32:33.029413+0000","last_unstale":"2024-03-28T23:32:44.106030+0000","last_undegraded":"2024-03-28T23:32:44.106030+0000","last_fullsized":"2024-03-28T23:32:44.106030+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:37:26.652511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107737+0000","last_change":"2024-03-28T23:32:32.037039+0000","last_active":"2024-03-28T23:32:44.107737+0000","last_peered":"2024-03-28T23:32:44.107737+0000","last_clean":"2024-03-28T23:32:44.107737+0000","last_became_active":"2024-03-28T23:32:32.036687+0000","last_became_peered":"2024-03-28T23:32:32.036687+0000","last_unstale":"2024-03-28T23:32:44.107737+0000","last_undegraded":"2024-03-28T23:32:44.107737+0000","last_fullsized":"2024-03-28T23:32:44.107737+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:27:15.824876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107961+0000","last_change":"2024-03-28T23:32:44.107961+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107961+0000","last_undegraded":"2024-03-28T23:32:44.107961+0000","last_fullsized":"2024-03-28T23:32:44.107961+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.127537+0000","last_change":"2024-03-28T23:32:44.126156+0000","last_active":"2024-03-28T23:32:44.127537+0000","last_peered":"2024-03-28T23:32:44.127537+0000","last_clean":"2024-03-28T23:32:44.127537+0000","last_became_active":"2024-03-28T23:32:44.125789+0000","last_became_peered":"2024-03-28T23:32:44.125789+0000","last_unstale":"2024-03-28T23:32:44.127537+0000","last_undegraded":"2024-03-28T23:32:44.127537+0000","last_fullsized":"2024-03-28T23:32:44.127537+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:41:04.213333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113597+0000","last_change":"2024-03-28T23:32:33.035809+0000","last_active":"2024-03-28T23:32:44.113597+0000","last_peered":"2024-03-28T23:32:44.113597+0000","last_clean":"2024-03-28T23:32:44.113597+0000","last_became_active":"2024-03-28T23:32:33.035563+0000","last_became_peered":"2024-03-28T23:32:33.035563+0000","last_unstale":"2024-03-28T23:32:44.113597+0000","last_undegraded":"2024-03-28T23:32:44.113597+0000","last_fullsized":"2024-03-28T23:32:44.113597+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:54.194453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"72'3","reported_seq":21,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.110292+0000","last_change":"2024-03-28T23:32:32.038230+0000","last_active":"2024-03-28T23:32:44.110292+0000","last_peered":"2024-03-28T23:32:44.110292+0000","last_clean":"2024-03-28T23:32:44.110292+0000","last_became_active":"2024-03-28T23:32:32.037681+0000","last_became_peered":"2024-03-28T23:32:32.037681+0000","last_unstale":"2024-03-28T23:32:44.110292+0000","last_undegraded":"2024-03-28T23:32:44.110292+0000","last_fullsized":"2024-03-28T23:32:44.110292+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:36:34.752866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112106+0000","last_change":"2024-03-28T23:32:44.112106+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112106+0000","last_undegraded":"2024-03-28T23:32:44.112106+0000","last_fullsized":"2024-03-28T23:32:44.112106+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.126080+0000","last_change":"2024-03-28T23:32:44.125087+0000","last_active":"2024-03-28T23:32:44.126080+0000","last_peered":"2024-03-28T23:32:44.126080+0000","last_clean":"2024-03-28T23:32:44.126080+0000","last_became_active":"2024-03-28T23:32:44.124784+0000","last_became_peered":"2024-03-28T23:32:44.124784+0000","last_unstale":"2024-03-28T23:32:44.126080+0000","last_undegraded":"2024-03-28T23:32:44.126080+0000","last_fullsized":"2024-03-28T23:32:44.126080+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:16:45.967951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112348+0000","last_change":"2024-03-28T23:32:33.049499+0000","last_active":"2024-03-28T23:32:44.112348+0000","last_peered":"2024-03-28T23:32:44.112348+0000","last_clean":"2024-03-28T23:32:44.112348+0000","last_became_active":"2024-03-28T23:32:33.049233+0000","last_became_peered":"2024-03-28T23:32:33.049233+0000","last_unstale":"2024-03-28T23:32:44.112348+0000","last_undegraded":"2024-03-28T23:32:44.112348+0000","last_fullsized":"2024-03-28T23:32:44.112348+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:39:02.824323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"72'6","reported_seq":24,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112883+0000","last_change":"2024-03-28T23:32:32.036835+0000","last_active":"2024-03-28T23:32:44.112883+0000","last_peered":"2024-03-28T23:32:44.112883+0000","last_clean":"2024-03-28T23:32:44.112883+0000","last_became_active":"2024-03-28T23:32:32.036557+0000","last_became_peered":"2024-03-28T23:32:32.036557+0000","last_unstale":"2024-03-28T23:32:44.112883+0000","last_undegraded":"2024-03-28T23:32:44.112883+0000","last_fullsized":"2024-03-28T23:32:44.112883+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:30.992807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111621+0000","last_change":"2024-03-28T23:32:44.111621+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111621+0000","last_undegraded":"2024-03-28T23:32:44.111621+0000","last_fullsized":"2024-03-28T23:32:44.111621+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.127678+0000","last_change":"2024-03-28T23:32:44.127256+0000","last_active":"2024-03-28T23:32:44.127678+0000","last_peered":"2024-03-28T23:32:44.127678+0000","last_clean":"2024-03-28T23:32:44.127678+0000","last_became_active":"2024-03-28T23:32:44.126894+0000","last_became_peered":"2024-03-28T23:32:44.126894+0000","last_unstale":"2024-03-28T23:32:44.127678+0000","last_undegraded":"2024-03-28T23:32:44.127678+0000","last_fullsized":"2024-03-28T23:32:44.127678+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:28:37.442827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107086+0000","last_change":"2024-03-28T23:32:33.033045+0000","last_active":"2024-03-28T23:32:44.107086+0000","last_peered":"2024-03-28T23:32:44.107086+0000","last_clean":"2024-03-28T23:32:44.107086+0000","last_became_active":"2024-03-28T23:32:33.032771+0000","last_became_peered":"2024-03-28T23:32:33.032771+0000","last_unstale":"2024-03-28T23:32:44.107086+0000","last_undegraded":"2024-03-28T23:32:44.107086+0000","last_fullsized":"2024-03-28T23:32:44.107086+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:43.341877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"72'2","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106619+0000","last_change":"2024-03-28T23:32:32.032053+0000","last_active":"2024-03-28T23:32:44.106619+0000","last_peered":"2024-03-28T23:32:44.106619+0000","last_clean":"2024-03-28T23:32:44.106619+0000","last_became_active":"2024-03-28T23:32:32.031741+0000","last_became_peered":"2024-03-28T23:32:32.031741+0000","last_unstale":"2024-03-28T23:32:44.106619+0000","last_undegraded":"2024-03-28T23:32:44.106619+0000","last_fullsized":"2024-03-28T23:32:44.106619+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:20.456721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108993+0000","last_change":"2024-03-28T23:32:44.108993+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108993+0000","last_undegraded":"2024-03-28T23:32:44.108993+0000","last_fullsized":"2024-03-28T23:32:44.108993+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.125906+0000","last_change":"2024-03-28T23:32:44.124910+0000","last_active":"2024-03-28T23:32:44.125906+0000","last_peered":"2024-03-28T23:32:44.125906+0000","last_clean":"2024-03-28T23:32:44.125906+0000","last_became_active":"2024-03-28T23:32:44.124639+0000","last_became_peered":"2024-03-28T23:32:44.124639+0000","last_unstale":"2024-03-28T23:32:44.125906+0000","last_undegraded":"2024-03-28T23:32:44.125906+0000","last_fullsized":"2024-03-28T23:32:44.125906+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:13:09.913978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112459+0000","last_change":"2024-03-28T23:32:33.039022+0000","last_active":"2024-03-28T23:32:44.112459+0000","last_peered":"2024-03-28T23:32:44.112459+0000","last_clean":"2024-03-28T23:32:44.112459+0000","last_became_active":"2024-03-28T23:32:33.038508+0000","last_became_peered":"2024-03-28T23:32:33.038508+0000","last_unstale":"2024-03-28T23:32:44.112459+0000","last_undegraded":"2024-03-28T23:32:44.112459+0000","last_fullsized":"2024-03-28T23:32:44.112459+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:25:45.189859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106348+0000","last_change":"2024-03-28T23:32:32.041042+0000","last_active":"2024-03-28T23:32:44.106348+0000","last_peered":"2024-03-28T23:32:44.106348+0000","last_clean":"2024-03-28T23:32:44.106348+0000","last_became_active":"2024-03-28T23:32:32.040686+0000","last_became_peered":"2024-03-28T23:32:32.040686+0000","last_unstale":"2024-03-28T23:32:44.106348+0000","last_undegraded":"2024-03-28T23:32:44.106348+0000","last_fullsized":"2024-03-28T23:32:44.106348+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:46.343619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.114552+0000","last_change":"2024-03-28T23:32:44.114552+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.114552+0000","last_undegraded":"2024-03-28T23:32:44.114552+0000","last_fullsized":"2024-03-28T23:32:44.114552+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121906+0000","last_change":"2024-03-28T23:32:44.120268+0000","last_active":"2024-03-28T23:32:44.121906+0000","last_peered":"2024-03-28T23:32:44.121906+0000","last_clean":"2024-03-28T23:32:44.121906+0000","last_became_active":"2024-03-28T23:32:44.119951+0000","last_became_peered":"2024-03-28T23:32:44.119951+0000","last_unstale":"2024-03-28T23:32:44.121906+0000","last_undegraded":"2024-03-28T23:32:44.121906+0000","last_fullsized":"2024-03-28T23:32:44.121906+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:28:47.614634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112006+0000","last_change":"2024-03-28T23:32:33.034262+0000","last_active":"2024-03-28T23:32:44.112006+0000","last_peered":"2024-03-28T23:32:44.112006+0000","last_clean":"2024-03-28T23:32:44.112006+0000","last_became_active":"2024-03-28T23:32:33.034003+0000","last_became_peered":"2024-03-28T23:32:33.034003+0000","last_unstale":"2024-03-28T23:32:44.112006+0000","last_undegraded":"2024-03-28T23:32:44.112006+0000","last_fullsized":"2024-03-28T23:32:44.112006+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:41.257241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108918+0000","last_change":"2024-03-28T23:32:32.039031+0000","last_active":"2024-03-28T23:32:44.108918+0000","last_peered":"2024-03-28T23:32:44.108918+0000","last_clean":"2024-03-28T23:32:44.108918+0000","last_became_active":"2024-03-28T23:32:32.038766+0000","last_became_peered":"2024-03-28T23:32:32.038766+0000","last_unstale":"2024-03-28T23:32:44.108918+0000","last_undegraded":"2024-03-28T23:32:44.108918+0000","last_fullsized":"2024-03-28T23:32:44.108918+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:13.872794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107400+0000","last_change":"2024-03-28T23:32:44.107400+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107400+0000","last_undegraded":"2024-03-28T23:32:44.107400+0000","last_fullsized":"2024-03-28T23:32:44.107400+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.123690+0000","last_change":"2024-03-28T23:32:44.119707+0000","last_active":"2024-03-28T23:32:44.123690+0000","last_peered":"2024-03-28T23:32:44.123690+0000","last_clean":"2024-03-28T23:32:44.123690+0000","last_became_active":"2024-03-28T23:32:44.119438+0000","last_became_peered":"2024-03-28T23:32:44.119438+0000","last_unstale":"2024-03-28T23:32:44.123690+0000","last_undegraded":"2024-03-28T23:32:44.123690+0000","last_fullsized":"2024-03-28T23:32:44.123690+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:39.542218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113512+0000","last_change":"2024-03-28T23:32:33.043171+0000","last_active":"2024-03-28T23:32:44.113512+0000","last_peered":"2024-03-28T23:32:44.113512+0000","last_clean":"2024-03-28T23:32:44.113512+0000","last_became_active":"2024-03-28T23:32:33.042821+0000","last_became_peered":"2024-03-28T23:32:33.042821+0000","last_unstale":"2024-03-28T23:32:44.113512+0000","last_undegraded":"2024-03-28T23:32:44.113512+0000","last_fullsized":"2024-03-28T23:32:44.113512+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:21.878156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"72'8","reported_seq":26,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121583+0000","last_change":"2024-03-28T23:32:32.040166+0000","last_active":"2024-03-28T23:32:44.121583+0000","last_peered":"2024-03-28T23:32:44.121583+0000","last_clean":"2024-03-28T23:32:44.121583+0000","last_became_active":"2024-03-28T23:32:32.039560+0000","last_became_peered":"2024-03-28T23:32:32.039560+0000","last_unstale":"2024-03-28T23:32:44.121583+0000","last_undegraded":"2024-03-28T23:32:44.121583+0000","last_fullsized":"2024-03-28T23:32:44.121583+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:55.544945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.110399+0000","last_change":"2024-03-28T23:32:44.110399+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.110399+0000","last_undegraded":"2024-03-28T23:32:44.110399+0000","last_fullsized":"2024-03-28T23:32:44.110399+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.127360+0000","last_change":"2024-03-28T23:32:44.126116+0000","last_active":"2024-03-28T23:32:44.127360+0000","last_peered":"2024-03-28T23:32:44.127360+0000","last_clean":"2024-03-28T23:32:44.127360+0000","last_became_active":"2024-03-28T23:32:44.125761+0000","last_became_peered":"2024-03-28T23:32:44.125761+0000","last_unstale":"2024-03-28T23:32:44.127360+0000","last_undegraded":"2024-03-28T23:32:44.127360+0000","last_fullsized":"2024-03-28T23:32:44.127360+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:04.591562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113351+0000","last_change":"2024-03-28T23:32:33.043192+0000","last_active":"2024-03-28T23:32:44.113351+0000","last_peered":"2024-03-28T23:32:44.113351+0000","last_clean":"2024-03-28T23:32:44.113351+0000","last_became_active":"2024-03-28T23:32:33.042894+0000","last_became_peered":"2024-03-28T23:32:33.042894+0000","last_unstale":"2024-03-28T23:32:44.113351+0000","last_undegraded":"2024-03-28T23:32:44.113351+0000","last_fullsized":"2024-03-28T23:32:44.113351+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:32:08.473276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"72'1","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108454+0000","last_change":"2024-03-28T23:32:32.041892+0000","last_active":"2024-03-28T23:32:44.108454+0000","last_peered":"2024-03-28T23:32:44.108454+0000","last_clean":"2024-03-28T23:32:44.108454+0000","last_became_active":"2024-03-28T23:32:32.041533+0000","last_became_peered":"2024-03-28T23:32:32.041533+0000","last_unstale":"2024-03-28T23:32:44.108454+0000","last_undegraded":"2024-03-28T23:32:44.108454+0000","last_fullsized":"2024-03-28T23:32:44.108454+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:15.843269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.116304+0000","last_change":"2024-03-28T23:32:44.116304+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.116304+0000","last_undegraded":"2024-03-28T23:32:44.116304+0000","last_fullsized":"2024-03-28T23:32:44.116304+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.118959+0000","last_change":"2024-03-28T23:32:44.115585+0000","last_active":"2024-03-28T23:32:44.118959+0000","last_peered":"2024-03-28T23:32:44.118959+0000","last_clean":"2024-03-28T23:32:44.118959+0000","last_became_active":"2024-03-28T23:32:44.115308+0000","last_became_peered":"2024-03-28T23:32:44.115308+0000","last_unstale":"2024-03-28T23:32:44.118959+0000","last_undegraded":"2024-03-28T23:32:44.118959+0000","last_fullsized":"2024-03-28T23:32:44.118959+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:59.122709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108799+0000","last_change":"2024-03-28T23:32:33.046899+0000","last_active":"2024-03-28T23:32:44.108799+0000","last_peered":"2024-03-28T23:32:44.108799+0000","last_clean":"2024-03-28T23:32:44.108799+0000","last_became_active":"2024-03-28T23:32:33.046579+0000","last_became_peered":"2024-03-28T23:32:33.046579+0000","last_unstale":"2024-03-28T23:32:44.108799+0000","last_undegraded":"2024-03-28T23:32:44.108799+0000","last_fullsized":"2024-03-28T23:32:44.108799+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:08:06.339997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"72'2","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.115368+0000","last_change":"2024-03-28T23:32:32.031135+0000","last_active":"2024-03-28T23:32:44.115368+0000","last_peered":"2024-03-28T23:32:44.115368+0000","last_clean":"2024-03-28T23:32:44.115368+0000","last_became_active":"2024-03-28T23:32:32.030796+0000","last_became_peered":"2024-03-28T23:32:32.030796+0000","last_unstale":"2024-03-28T23:32:44.115368+0000","last_undegraded":"2024-03-28T23:32:44.115368+0000","last_fullsized":"2024-03-28T23:32:44.115368+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:18:21.816565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.116620+0000","last_change":"2024-03-28T23:32:44.116620+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.116620+0000","last_undegraded":"2024-03-28T23:32:44.116620+0000","last_fullsized":"2024-03-28T23:32:44.116620+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.117708+0000","last_change":"2024-03-28T23:32:44.116752+0000","last_active":"2024-03-28T23:32:44.117708+0000","last_peered":"2024-03-28T23:32:44.117708+0000","last_clean":"2024-03-28T23:32:44.117708+0000","last_became_active":"2024-03-28T23:32:44.116381+0000","last_became_peered":"2024-03-28T23:32:44.116381+0000","last_unstale":"2024-03-28T23:32:44.117708+0000","last_undegraded":"2024-03-28T23:32:44.117708+0000","last_fullsized":"2024-03-28T23:32:44.117708+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:17:56.623866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109016+0000","last_change":"2024-03-28T23:32:33.053549+0000","last_active":"2024-03-28T23:32:44.109016+0000","last_peered":"2024-03-28T23:32:44.109016+0000","last_clean":"2024-03-28T23:32:44.109016+0000","last_became_active":"2024-03-28T23:32:33.053229+0000","last_became_peered":"2024-03-28T23:32:33.053229+0000","last_unstale":"2024-03-28T23:32:44.109016+0000","last_undegraded":"2024-03-28T23:32:44.109016+0000","last_fullsized":"2024-03-28T23:32:44.109016+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:41:02.517121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"72'1","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.110391+0000","last_change":"2024-03-28T23:32:32.038261+0000","last_active":"2024-03-28T23:32:44.110391+0000","last_peered":"2024-03-28T23:32:44.110391+0000","last_clean":"2024-03-28T23:32:44.110391+0000","last_became_active":"2024-03-28T23:32:32.037971+0000","last_became_peered":"2024-03-28T23:32:32.037971+0000","last_unstale":"2024-03-28T23:32:44.110391+0000","last_undegraded":"2024-03-28T23:32:44.110391+0000","last_fullsized":"2024-03-28T23:32:44.110391+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:25:10.296139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.109933+0000","last_change":"2024-03-28T23:32:44.109933+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.109933+0000","last_undegraded":"2024-03-28T23:32:44.109933+0000","last_fullsized":"2024-03-28T23:32:44.109933+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.118230+0000","last_change":"2024-03-28T23:32:44.115062+0000","last_active":"2024-03-28T23:32:44.118230+0000","last_peered":"2024-03-28T23:32:44.118230+0000","last_clean":"2024-03-28T23:32:44.118230+0000","last_became_active":"2024-03-28T23:32:44.114795+0000","last_became_peered":"2024-03-28T23:32:44.114795+0000","last_unstale":"2024-03-28T23:32:44.118230+0000","last_undegraded":"2024-03-28T23:32:44.118230+0000","last_fullsized":"2024-03-28T23:32:44.118230+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:38.460546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113026+0000","last_change":"2024-03-28T23:32:33.043954+0000","last_active":"2024-03-28T23:32:44.113026+0000","last_peered":"2024-03-28T23:32:44.113026+0000","last_clean":"2024-03-28T23:32:44.113026+0000","last_became_active":"2024-03-28T23:32:33.043649+0000","last_became_peered":"2024-03-28T23:32:33.043649+0000","last_unstale":"2024-03-28T23:32:44.113026+0000","last_undegraded":"2024-03-28T23:32:44.113026+0000","last_fullsized":"2024-03-28T23:32:44.113026+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:00.358180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109556+0000","last_change":"2024-03-28T23:32:32.033720+0000","last_active":"2024-03-28T23:32:44.109556+0000","last_peered":"2024-03-28T23:32:44.109556+0000","last_clean":"2024-03-28T23:32:44.109556+0000","last_became_active":"2024-03-28T23:32:32.033422+0000","last_became_peered":"2024-03-28T23:32:32.033422+0000","last_unstale":"2024-03-28T23:32:44.109556+0000","last_undegraded":"2024-03-28T23:32:44.109556+0000","last_fullsized":"2024-03-28T23:32:44.109556+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:51:10.440109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.113185+0000","last_change":"2024-03-28T23:32:44.113185+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.113185+0000","last_undegraded":"2024-03-28T23:32:44.113185+0000","last_fullsized":"2024-03-28T23:32:44.113185+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.124927+0000","last_change":"2024-03-28T23:32:44.124559+0000","last_active":"2024-03-28T23:32:44.124927+0000","last_peered":"2024-03-28T23:32:44.124927+0000","last_clean":"2024-03-28T23:32:44.124927+0000","last_became_active":"2024-03-28T23:32:44.124282+0000","last_became_peered":"2024-03-28T23:32:44.124282+0000","last_unstale":"2024-03-28T23:32:44.124927+0000","last_undegraded":"2024-03-28T23:32:44.124927+0000","last_fullsized":"2024-03-28T23:32:44.124927+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:32.542845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122528+0000","last_change":"2024-03-28T23:32:33.045618+0000","last_active":"2024-03-28T23:32:44.122528+0000","last_peered":"2024-03-28T23:32:44.122528+0000","last_clean":"2024-03-28T23:32:44.122528+0000","last_became_active":"2024-03-28T23:32:33.045324+0000","last_became_peered":"2024-03-28T23:32:33.045324+0000","last_unstale":"2024-03-28T23:32:44.122528+0000","last_undegraded":"2024-03-28T23:32:44.122528+0000","last_fullsized":"2024-03-28T23:32:44.122528+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:46:43.644886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"72'2","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113118+0000","last_change":"2024-03-28T23:32:32.031574+0000","last_active":"2024-03-28T23:32:44.113118+0000","last_peered":"2024-03-28T23:32:44.113118+0000","last_clean":"2024-03-28T23:32:44.113118+0000","last_became_active":"2024-03-28T23:32:32.031310+0000","last_became_peered":"2024-03-28T23:32:32.031310+0000","last_unstale":"2024-03-28T23:32:44.113118+0000","last_undegraded":"2024-03-28T23:32:44.113118+0000","last_fullsized":"2024-03-28T23:32:44.113118+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:48:07.073481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112326+0000","last_change":"2024-03-28T23:32:44.112326+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112326+0000","last_undegraded":"2024-03-28T23:32:44.112326+0000","last_fullsized":"2024-03-28T23:32:44.112326+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.115233+0000","last_change":"2024-03-28T23:32:44.114144+0000","last_active":"2024-03-28T23:32:44.115233+0000","last_peered":"2024-03-28T23:32:44.115233+0000","last_clean":"2024-03-28T23:32:44.115233+0000","last_became_active":"2024-03-28T23:32:44.113829+0000","last_became_peered":"2024-03-28T23:32:44.113829+0000","last_unstale":"2024-03-28T23:32:44.115233+0000","last_undegraded":"2024-03-28T23:32:44.115233+0000","last_fullsized":"2024-03-28T23:32:44.115233+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:55.159548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.123006+0000","last_change":"2024-03-28T23:32:33.040983+0000","last_active":"2024-03-28T23:32:44.123006+0000","last_peered":"2024-03-28T23:32:44.123006+0000","last_clean":"2024-03-28T23:32:44.123006+0000","last_became_active":"2024-03-28T23:32:33.040693+0000","last_became_peered":"2024-03-28T23:32:33.040693+0000","last_unstale":"2024-03-28T23:32:44.123006+0000","last_undegraded":"2024-03-28T23:32:44.123006+0000","last_fullsized":"2024-03-28T23:32:44.123006+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:22:05.201764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113759+0000","last_change":"2024-03-28T23:32:32.036998+0000","last_active":"2024-03-28T23:32:44.113759+0000","last_peered":"2024-03-28T23:32:44.113759+0000","last_clean":"2024-03-28T23:32:44.113759+0000","last_became_active":"2024-03-28T23:32:32.036733+0000","last_became_peered":"2024-03-28T23:32:32.036733+0000","last_unstale":"2024-03-28T23:32:44.113759+0000","last_undegraded":"2024-03-28T23:32:44.113759+0000","last_fullsized":"2024-03-28T23:32:44.113759+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:42.334479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.105630+0000","last_change":"2024-03-28T23:32:44.105630+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.105630+0000","last_undegraded":"2024-03-28T23:32:44.105630+0000","last_fullsized":"2024-03-28T23:32:44.105630+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.115890+0000","last_change":"2024-03-28T23:32:44.115154+0000","last_active":"2024-03-28T23:32:44.115890+0000","last_peered":"2024-03-28T23:32:44.115890+0000","last_clean":"2024-03-28T23:32:44.115890+0000","last_became_active":"2024-03-28T23:32:44.114869+0000","last_became_peered":"2024-03-28T23:32:44.114869+0000","last_unstale":"2024-03-28T23:32:44.115890+0000","last_undegraded":"2024-03-28T23:32:44.115890+0000","last_fullsized":"2024-03-28T23:32:44.115890+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:58:59.309973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113614+0000","last_change":"2024-03-28T23:32:33.041232+0000","last_active":"2024-03-28T23:32:44.113614+0000","last_peered":"2024-03-28T23:32:44.113614+0000","last_clean":"2024-03-28T23:32:44.113614+0000","last_became_active":"2024-03-28T23:32:33.040132+0000","last_became_peered":"2024-03-28T23:32:33.040132+0000","last_unstale":"2024-03-28T23:32:44.113614+0000","last_undegraded":"2024-03-28T23:32:44.113614+0000","last_fullsized":"2024-03-28T23:32:44.113614+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:39:02.824323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113450+0000","last_change":"2024-03-28T23:32:32.036416+0000","last_active":"2024-03-28T23:32:44.113450+0000","last_peered":"2024-03-28T23:32:44.113450+0000","last_clean":"2024-03-28T23:32:44.113450+0000","last_became_active":"2024-03-28T23:32:32.036131+0000","last_became_peered":"2024-03-28T23:32:32.036131+0000","last_unstale":"2024-03-28T23:32:44.113450+0000","last_undegraded":"2024-03-28T23:32:44.113450+0000","last_fullsized":"2024-03-28T23:32:44.113450+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:13:33.914342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.109522+0000","last_change":"2024-03-28T23:32:44.109522+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.109522+0000","last_undegraded":"2024-03-28T23:32:44.109522+0000","last_fullsized":"2024-03-28T23:32:44.109522+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.119994+0000","last_change":"2024-03-28T23:32:44.116820+0000","last_active":"2024-03-28T23:32:44.119994+0000","last_peered":"2024-03-28T23:32:44.119994+0000","last_clean":"2024-03-28T23:32:44.119994+0000","last_became_active":"2024-03-28T23:32:44.116540+0000","last_became_peered":"2024-03-28T23:32:44.116540+0000","last_unstale":"2024-03-28T23:32:44.119994+0000","last_undegraded":"2024-03-28T23:32:44.119994+0000","last_fullsized":"2024-03-28T23:32:44.119994+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:58.429743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.110759+0000","last_change":"2024-03-28T23:32:33.035015+0000","last_active":"2024-03-28T23:32:44.110759+0000","last_peered":"2024-03-28T23:32:44.110759+0000","last_clean":"2024-03-28T23:32:44.110759+0000","last_became_active":"2024-03-28T23:32:33.034730+0000","last_became_peered":"2024-03-28T23:32:33.034730+0000","last_unstale":"2024-03-28T23:32:44.110759+0000","last_undegraded":"2024-03-28T23:32:44.110759+0000","last_fullsized":"2024-03-28T23:32:44.110759+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:09.477928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"72'2","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108295+0000","last_change":"2024-03-28T23:32:32.032596+0000","last_active":"2024-03-28T23:32:44.108295+0000","last_peered":"2024-03-28T23:32:44.108295+0000","last_clean":"2024-03-28T23:32:44.108295+0000","last_became_active":"2024-03-28T23:32:32.032336+0000","last_became_peered":"2024-03-28T23:32:32.032336+0000","last_unstale":"2024-03-28T23:32:44.108295+0000","last_undegraded":"2024-03-28T23:32:44.108295+0000","last_fullsized":"2024-03-28T23:32:44.108295+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:28:25.356704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.120871+0000","last_change":"2024-03-28T23:32:44.120871+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.120871+0000","last_undegraded":"2024-03-28T23:32:44.120871+0000","last_fullsized":"2024-03-28T23:32:44.120871+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.117955+0000","last_change":"2024-03-28T23:32:44.116649+0000","last_active":"2024-03-28T23:32:44.117955+0000","last_peered":"2024-03-28T23:32:44.117955+0000","last_clean":"2024-03-28T23:32:44.117955+0000","last_became_active":"2024-03-28T23:32:44.116369+0000","last_became_peered":"2024-03-28T23:32:44.116369+0000","last_unstale":"2024-03-28T23:32:44.117955+0000","last_undegraded":"2024-03-28T23:32:44.117955+0000","last_fullsized":"2024-03-28T23:32:44.117955+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:46:54.723611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.111340+0000","last_change":"2024-03-28T23:32:33.044243+0000","last_active":"2024-03-28T23:32:44.111340+0000","last_peered":"2024-03-28T23:32:44.111340+0000","last_clean":"2024-03-28T23:32:44.111340+0000","last_became_active":"2024-03-28T23:32:33.040218+0000","last_became_peered":"2024-03-28T23:32:33.040218+0000","last_unstale":"2024-03-28T23:32:44.111340+0000","last_undegraded":"2024-03-28T23:32:44.111340+0000","last_fullsized":"2024-03-28T23:32:44.111340+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:38:51.513907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"73'1","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.123099+0000","last_change":"2024-03-28T23:32:32.041586+0000","last_active":"2024-03-28T23:32:44.123099+0000","last_peered":"2024-03-28T23:32:44.123099+0000","last_clean":"2024-03-28T23:32:44.123099+0000","last_became_active":"2024-03-28T23:32:32.041327+0000","last_became_peered":"2024-03-28T23:32:32.041327+0000","last_unstale":"2024-03-28T23:32:44.123099+0000","last_undegraded":"2024-03-28T23:32:44.123099+0000","last_fullsized":"2024-03-28T23:32:44.123099+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:06:20.608980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107219+0000","last_change":"2024-03-28T23:32:44.107219+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107219+0000","last_undegraded":"2024-03-28T23:32:44.107219+0000","last_fullsized":"2024-03-28T23:32:44.107219+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.126567+0000","last_change":"2024-03-28T23:32:44.125883+0000","last_active":"2024-03-28T23:32:44.126567+0000","last_peered":"2024-03-28T23:32:44.126567+0000","last_clean":"2024-03-28T23:32:44.126567+0000","last_became_active":"2024-03-28T23:32:44.125578+0000","last_became_peered":"2024-03-28T23:32:44.125578+0000","last_unstale":"2024-03-28T23:32:44.126567+0000","last_undegraded":"2024-03-28T23:32:44.126567+0000","last_fullsized":"2024-03-28T23:32:44.126567+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:25.238412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112455+0000","last_change":"2024-03-28T23:32:33.042272+0000","last_active":"2024-03-28T23:32:44.112455+0000","last_peered":"2024-03-28T23:32:44.112455+0000","last_clean":"2024-03-28T23:32:44.112455+0000","last_became_active":"2024-03-28T23:32:33.037633+0000","last_became_peered":"2024-03-28T23:32:33.037633+0000","last_unstale":"2024-03-28T23:32:44.112455+0000","last_undegraded":"2024-03-28T23:32:44.112455+0000","last_fullsized":"2024-03-28T23:32:44.112455+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:45.773838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108173+0000","last_change":"2024-03-28T23:32:32.037001+0000","last_active":"2024-03-28T23:32:44.108173+0000","last_peered":"2024-03-28T23:32:44.108173+0000","last_clean":"2024-03-28T23:32:44.108173+0000","last_became_active":"2024-03-28T23:32:32.036739+0000","last_became_peered":"2024-03-28T23:32:32.036739+0000","last_unstale":"2024-03-28T23:32:44.108173+0000","last_undegraded":"2024-03-28T23:32:44.108173+0000","last_fullsized":"2024-03-28T23:32:44.108173+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:06:20.608980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112232+0000","last_change":"2024-03-28T23:32:44.112232+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112232+0000","last_undegraded":"2024-03-28T23:32:44.112232+0000","last_fullsized":"2024-03-28T23:32:44.112232+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.117952+0000","last_change":"2024-03-28T23:32:44.116971+0000","last_active":"2024-03-28T23:32:44.117952+0000","last_peered":"2024-03-28T23:32:44.117952+0000","last_clean":"2024-03-28T23:32:44.117952+0000","last_became_active":"2024-03-28T23:32:44.116371+0000","last_became_peered":"2024-03-28T23:32:44.116371+0000","last_unstale":"2024-03-28T23:32:44.117952+0000","last_undegraded":"2024-03-28T23:32:44.117952+0000","last_fullsized":"2024-03-28T23:32:44.117952+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:43:47.025717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122394+0000","last_change":"2024-03-28T23:32:33.041123+0000","last_active":"2024-03-28T23:32:44.122394+0000","last_peered":"2024-03-28T23:32:44.122394+0000","last_clean":"2024-03-28T23:32:44.122394+0000","last_became_active":"2024-03-28T23:32:33.040505+0000","last_became_peered":"2024-03-28T23:32:33.040505+0000","last_unstale":"2024-03-28T23:32:44.122394+0000","last_undegraded":"2024-03-28T23:32:44.122394+0000","last_fullsized":"2024-03-28T23:32:44.122394+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:54.194453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106365+0000","last_change":"2024-03-28T23:32:32.036857+0000","last_active":"2024-03-28T23:32:44.106365+0000","last_peered":"2024-03-28T23:32:44.106365+0000","last_clean":"2024-03-28T23:32:44.106365+0000","last_became_active":"2024-03-28T23:32:32.036576+0000","last_became_peered":"2024-03-28T23:32:32.036576+0000","last_unstale":"2024-03-28T23:32:44.106365+0000","last_undegraded":"2024-03-28T23:32:44.106365+0000","last_fullsized":"2024-03-28T23:32:44.106365+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:43.073425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.120260+0000","last_change":"2024-03-28T23:32:44.120260+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.120260+0000","last_undegraded":"2024-03-28T23:32:44.120260+0000","last_fullsized":"2024-03-28T23:32:44.120260+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.127553+0000","last_change":"2024-03-28T23:32:44.126706+0000","last_active":"2024-03-28T23:32:44.127553+0000","last_peered":"2024-03-28T23:32:44.127553+0000","last_clean":"2024-03-28T23:32:44.127553+0000","last_became_active":"2024-03-28T23:32:44.126400+0000","last_became_peered":"2024-03-28T23:32:44.126400+0000","last_unstale":"2024-03-28T23:32:44.127553+0000","last_undegraded":"2024-03-28T23:32:44.127553+0000","last_fullsized":"2024-03-28T23:32:44.127553+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:00.790475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112670+0000","last_change":"2024-03-28T23:32:33.041769+0000","last_active":"2024-03-28T23:32:44.112670+0000","last_peered":"2024-03-28T23:32:44.112670+0000","last_clean":"2024-03-28T23:32:44.112670+0000","last_became_active":"2024-03-28T23:32:33.041475+0000","last_became_peered":"2024-03-28T23:32:33.041475+0000","last_unstale":"2024-03-28T23:32:44.112670+0000","last_undegraded":"2024-03-28T23:32:44.112670+0000","last_fullsized":"2024-03-28T23:32:44.112670+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:38.859031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112270+0000","last_change":"2024-03-28T23:32:32.037824+0000","last_active":"2024-03-28T23:32:44.112270+0000","last_peered":"2024-03-28T23:32:44.112270+0000","last_clean":"2024-03-28T23:32:44.112270+0000","last_became_active":"2024-03-28T23:32:32.037550+0000","last_became_peered":"2024-03-28T23:32:32.037550+0000","last_unstale":"2024-03-28T23:32:44.112270+0000","last_undegraded":"2024-03-28T23:32:44.112270+0000","last_fullsized":"2024-03-28T23:32:44.112270+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:43.467214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111817+0000","last_change":"2024-03-28T23:32:44.111817+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111817+0000","last_undegraded":"2024-03-28T23:32:44.111817+0000","last_fullsized":"2024-03-28T23:32:44.111817+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.119913+0000","last_change":"2024-03-28T23:32:44.117095+0000","last_active":"2024-03-28T23:32:44.119913+0000","last_peered":"2024-03-28T23:32:44.119913+0000","last_clean":"2024-03-28T23:32:44.119913+0000","last_became_active":"2024-03-28T23:32:44.116824+0000","last_became_peered":"2024-03-28T23:32:44.116824+0000","last_unstale":"2024-03-28T23:32:44.119913+0000","last_undegraded":"2024-03-28T23:32:44.119913+0000","last_fullsized":"2024-03-28T23:32:44.119913+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:59:07.631069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112978+0000","last_change":"2024-03-28T23:32:33.048966+0000","last_active":"2024-03-28T23:32:44.112978+0000","last_peered":"2024-03-28T23:32:44.112978+0000","last_clean":"2024-03-28T23:32:44.112978+0000","last_became_active":"2024-03-28T23:32:33.048362+0000","last_became_peered":"2024-03-28T23:32:33.048362+0000","last_unstale":"2024-03-28T23:32:44.112978+0000","last_undegraded":"2024-03-28T23:32:44.112978+0000","last_fullsized":"2024-03-28T23:32:44.112978+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:25.849919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109099+0000","last_change":"2024-03-28T23:32:32.038885+0000","last_active":"2024-03-28T23:32:44.109099+0000","last_peered":"2024-03-28T23:32:44.109099+0000","last_clean":"2024-03-28T23:32:44.109099+0000","last_became_active":"2024-03-28T23:32:32.038364+0000","last_became_peered":"2024-03-28T23:32:32.038364+0000","last_unstale":"2024-03-28T23:32:44.109099+0000","last_undegraded":"2024-03-28T23:32:44.109099+0000","last_fullsized":"2024-03-28T23:32:44.109099+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:16:06.156877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.105630+0000","last_change":"2024-03-28T23:32:44.105630+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.105630+0000","last_undegraded":"2024-03-28T23:32:44.105630+0000","last_fullsized":"2024-03-28T23:32:44.105630+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.115613+0000","last_change":"2024-03-28T23:32:44.114483+0000","last_active":"2024-03-28T23:32:44.115613+0000","last_peered":"2024-03-28T23:32:44.115613+0000","last_clean":"2024-03-28T23:32:44.115613+0000","last_became_active":"2024-03-28T23:32:44.114228+0000","last_became_peered":"2024-03-28T23:32:44.114228+0000","last_unstale":"2024-03-28T23:32:44.115613+0000","last_undegraded":"2024-03-28T23:32:44.115613+0000","last_fullsized":"2024-03-28T23:32:44.115613+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:56:05.273178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107950+0000","last_change":"2024-03-28T23:32:33.049584+0000","last_active":"2024-03-28T23:32:44.107950+0000","last_peered":"2024-03-28T23:32:44.107950+0000","last_clean":"2024-03-28T23:32:44.107950+0000","last_became_active":"2024-03-28T23:32:33.049342+0000","last_became_peered":"2024-03-28T23:32:33.049342+0000","last_unstale":"2024-03-28T23:32:44.107950+0000","last_undegraded":"2024-03-28T23:32:44.107950+0000","last_fullsized":"2024-03-28T23:32:44.107950+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:49.711750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.111513+0000","last_change":"2024-03-28T23:32:32.034044+0000","last_active":"2024-03-28T23:32:44.111513+0000","last_peered":"2024-03-28T23:32:44.111513+0000","last_clean":"2024-03-28T23:32:44.111513+0000","last_became_active":"2024-03-28T23:32:32.033779+0000","last_became_peered":"2024-03-28T23:32:32.033779+0000","last_unstale":"2024-03-28T23:32:44.111513+0000","last_undegraded":"2024-03-28T23:32:44.111513+0000","last_fullsized":"2024-03-28T23:32:44.111513+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:56:22.159708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.116670+0000","last_change":"2024-03-28T23:32:44.116670+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.116670+0000","last_undegraded":"2024-03-28T23:32:44.116670+0000","last_fullsized":"2024-03-28T23:32:44.116670+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.120835+0000","last_change":"2024-03-28T23:32:44.119649+0000","last_active":"2024-03-28T23:32:44.120835+0000","last_peered":"2024-03-28T23:32:44.120835+0000","last_clean":"2024-03-28T23:32:44.120835+0000","last_became_active":"2024-03-28T23:32:44.119370+0000","last_became_peered":"2024-03-28T23:32:44.119370+0000","last_unstale":"2024-03-28T23:32:44.120835+0000","last_undegraded":"2024-03-28T23:32:44.120835+0000","last_fullsized":"2024-03-28T23:32:44.120835+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:25:56.268584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109652+0000","last_change":"2024-03-28T23:32:33.041180+0000","last_active":"2024-03-28T23:32:44.109652+0000","last_peered":"2024-03-28T23:32:44.109652+0000","last_clean":"2024-03-28T23:32:44.109652+0000","last_became_active":"2024-03-28T23:32:33.040922+0000","last_became_peered":"2024-03-28T23:32:33.040922+0000","last_unstale":"2024-03-28T23:32:44.109652+0000","last_undegraded":"2024-03-28T23:32:44.109652+0000","last_fullsized":"2024-03-28T23:32:44.109652+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:25:45.189859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109480+0000","last_change":"2024-03-28T23:32:32.034328+0000","last_active":"2024-03-28T23:32:44.109480+0000","last_peered":"2024-03-28T23:32:44.109480+0000","last_clean":"2024-03-28T23:32:44.109480+0000","last_became_active":"2024-03-28T23:32:32.034057+0000","last_became_peered":"2024-03-28T23:32:32.034057+0000","last_unstale":"2024-03-28T23:32:44.109480+0000","last_undegraded":"2024-03-28T23:32:44.109480+0000","last_fullsized":"2024-03-28T23:32:44.109480+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:54:12.775407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107591+0000","last_change":"2024-03-28T23:32:44.107591+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107591+0000","last_undegraded":"2024-03-28T23:32:44.107591+0000","last_fullsized":"2024-03-28T23:32:44.107591+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.119109+0000","last_change":"2024-03-28T23:32:44.115020+0000","last_active":"2024-03-28T23:32:44.119109+0000","last_peered":"2024-03-28T23:32:44.119109+0000","last_clean":"2024-03-28T23:32:44.119109+0000","last_became_active":"2024-03-28T23:32:44.114737+0000","last_became_peered":"2024-03-28T23:32:44.114737+0000","last_unstale":"2024-03-28T23:32:44.119109+0000","last_undegraded":"2024-03-28T23:32:44.119109+0000","last_fullsized":"2024-03-28T23:32:44.119109+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:13:16.820792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108150+0000","last_change":"2024-03-28T23:32:33.038830+0000","last_active":"2024-03-28T23:32:44.108150+0000","last_peered":"2024-03-28T23:32:44.108150+0000","last_clean":"2024-03-28T23:32:44.108150+0000","last_became_active":"2024-03-28T23:32:33.038524+0000","last_became_peered":"2024-03-28T23:32:33.038524+0000","last_unstale":"2024-03-28T23:32:44.108150+0000","last_undegraded":"2024-03-28T23:32:44.108150+0000","last_fullsized":"2024-03-28T23:32:44.108150+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:55:53.512836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113822+0000","last_change":"2024-03-28T23:32:32.034270+0000","last_active":"2024-03-28T23:32:44.113822+0000","last_peered":"2024-03-28T23:32:44.113822+0000","last_clean":"2024-03-28T23:32:44.113822+0000","last_became_active":"2024-03-28T23:32:32.034022+0000","last_became_peered":"2024-03-28T23:32:32.034022+0000","last_unstale":"2024-03-28T23:32:44.113822+0000","last_undegraded":"2024-03-28T23:32:44.113822+0000","last_fullsized":"2024-03-28T23:32:44.113822+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:15.683500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":273,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109089+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:32:44.109089+0000","last_peered":"2024-03-28T23:32:44.109089+0000","last_clean":"2024-03-28T23:32:44.109089+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:32:44.109089+0000","last_undegraded":"2024-03-28T23:32:44.109089+0000","last_fullsized":"2024-03-28T23:32:44.109089+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574887,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":974848,"data_stored":1018333,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574888,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32088,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339752,"statfs":{"total":96636764160,"available":96603906048,"internally_reserved":0,"allocated":929792,"data_stored":437077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32068,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339772,"statfs":{"total":96636764160,"available":96603926528,"internally_reserved":0,"allocated":909312,"data_stored":427977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574887,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32084,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339756,"statfs":{"total":96636764160,"available":96603910144,"internally_reserved":0,"allocated":925696,"data_stored":429659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574887,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27908,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343932,"statfs":{"total":96636764160,"available":96608186368,"internally_reserved":0,"allocated":909312,"data_stored":428525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574887,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343924,"statfs":{"total":96636764160,"available":96608178176,"internally_reserved":0,"allocated":917504,"data_stored":428685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574887,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343928,"statfs":{"total":96636764160,"available":96608182272,"internally_reserved":0,"allocated":913408,"data_stored":428055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574887,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":970752,"data_stored":1018311,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:32:48.563 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:32:48.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.664+0000 7fa0c67b5640 1 Processor -- start 2024-03-28T23:32:48.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.664+0000 7fa0c67b5640 1 -- start start 2024-03-28T23:32:48.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.666+0000 7fa0c67b5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa0c0152890 0x7fa0c00c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:48.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.666+0000 7fa0c67b5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0c00c8ed0 0x7fa0c00c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:48.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.666+0000 7fa0c67b5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa0c00c5950 0x7fa0c01553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:48.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.666+0000 7fa0c67b5640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa0c00c8480 con 0x7fa0c0152890 2024-03-28T23:32:48.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.666+0000 7fa0c67b5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fa0c0159640 con 0x7fa0c00c5950 2024-03-28T23:32:48.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.666+0000 7fa0c67b5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa0c0158a30 con 0x7fa0c00c8ed0 2024-03-28T23:32:48.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.666+0000 7fa0bffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa0c0152890 0x7fa0c00c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:48.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.666+0000 7fa0bffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa0c0152890 0x7fa0c00c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39926/0 (socket says 172.21.15.161:39926) 2024-03-28T23:32:48.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.666+0000 7fa0bffff640 1 -- 172.21.15.161:0/3563611444 learned_addr learned my addr 172.21.15.161:0/3563611444 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:48.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.666+0000 7fa0c4d52640 1 --2- 172.21.15.161:0/3563611444 >> v2:172.21.15.170:3300/0 conn(0x7fa0c00c5950 0x7fa0c01553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:48.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.666+0000 7fa0bf7fe640 1 --2- 172.21.15.161:0/3563611444 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0c00c8ed0 0x7fa0c00c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:48.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.666+0000 7fa0bffff640 1 -- 172.21.15.161:0/3563611444 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0c00c8ed0 msgr2=0x7fa0c00c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:48.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.666+0000 7fa0bffff640 1 --2- 172.21.15.161:0/3563611444 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0c00c8ed0 0x7fa0c00c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.666+0000 7fa0bffff640 1 -- 172.21.15.161:0/3563611444 >> v2:172.21.15.170:3300/0 conn(0x7fa0c00c5950 msgr2=0x7fa0c01553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:48.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.666+0000 7fa0bffff640 1 --2- 172.21.15.161:0/3563611444 >> v2:172.21.15.170:3300/0 conn(0x7fa0c00c5950 0x7fa0c01553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.666+0000 7fa0bffff640 1 -- 172.21.15.161:0/3563611444 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c0158d10 con 0x7fa0c0152890 2024-03-28T23:32:48.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.667+0000 7fa0bffff640 1 --2- 172.21.15.161:0/3563611444 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c0152890 0x7fa0c00c8990 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fa0c00b0c00 tx=0x7fa0b002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b703b2bfcb94e93 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:48.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.667+0000 7fa0beffd640 1 -- 172.21.15.161:0/3563611444 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0038070 con 0x7fa0c0152890 2024-03-28T23:32:48.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.667+0000 7fa0beffd640 1 -- 172.21.15.161:0/3563611444 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0b002f050 con 0x7fa0c0152890 2024-03-28T23:32:48.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.667+0000 7fa0beffd640 1 -- 172.21.15.161:0/3563611444 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0031970 con 0x7fa0c0152890 2024-03-28T23:32:48.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.667+0000 7fa0c67b5640 1 -- 172.21.15.161:0/3563611444 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c0152890 msgr2=0x7fa0c00c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:48.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.667+0000 7fa0c67b5640 1 --2- 172.21.15.161:0/3563611444 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c0152890 0x7fa0c00c8990 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fa0c00b0c00 tx=0x7fa0b002b7f0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.668+0000 7fa0c67b5640 1 -- 172.21.15.161:0/3563611444 shutdown_connections 2024-03-28T23:32:48.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.668+0000 7fa0c67b5640 1 --2- 172.21.15.161:0/3563611444 >> v2:172.21.15.170:3300/0 conn(0x7fa0c00c5950 0x7fa0c01553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.668+0000 7fa0c67b5640 1 --2- 172.21.15.161:0/3563611444 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0c00c8ed0 0x7fa0c00c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.668+0000 7fa0c67b5640 1 --2- 172.21.15.161:0/3563611444 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c0152890 0x7fa0c00c8990 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.668+0000 7fa0c67b5640 1 -- 172.21.15.161:0/3563611444 >> 172.21.15.161:0/3563611444 conn(0x7fa0c00ac620 msgr2=0x7fa0c00aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:48.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.668+0000 7fa0c67b5640 1 -- 172.21.15.161:0/3563611444 shutdown_connections 2024-03-28T23:32:48.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.668+0000 7fa0c67b5640 1 -- 172.21.15.161:0/3563611444 wait complete. 2024-03-28T23:32:48.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.669+0000 7fa0c67b5640 1 Processor -- start 2024-03-28T23:32:48.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.669+0000 7fa0c67b5640 1 -- start start 2024-03-28T23:32:48.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.669+0000 7fa0c67b5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0c00c5950 0x7fa0c00c49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:48.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.670+0000 7fa0c67b5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0c00c8ed0 0x7fa0c00ab900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:48.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.670+0000 7fa0c67b5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c0152890 0x7fa0c01e7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:48.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.670+0000 7fa0c67b5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c0152c70 con 0x7fa0c0152890 2024-03-28T23:32:48.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.670+0000 7fa0c67b5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c01592d0 con 0x7fa0c00c8ed0 2024-03-28T23:32:48.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.670+0000 7fa0c67b5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa0c0152df0 con 0x7fa0c00c5950 2024-03-28T23:32:48.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.670+0000 7fa0c4d52640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c0152890 0x7fa0c01e7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:48.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.670+0000 7fa0c4d52640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c0152890 0x7fa0c01e7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39940/0 (socket says 172.21.15.161:39940) 2024-03-28T23:32:48.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.670+0000 7fa0c4d52640 1 -- 172.21.15.161:0/4225854525 learned_addr learned my addr 172.21.15.161:0/4225854525 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:48.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.670+0000 7fa0bf7fe640 1 --2- 172.21.15.161:0/4225854525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0c00c8ed0 0x7fa0c00ab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:48.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.670+0000 7fa0bffff640 1 --2- 172.21.15.161:0/4225854525 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0c00c5950 0x7fa0c00c49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:48.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.670+0000 7fa0c4d52640 1 -- 172.21.15.161:0/4225854525 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0c00c5950 msgr2=0x7fa0c00c49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:48.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.670+0000 7fa0c4d52640 1 --2- 172.21.15.161:0/4225854525 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0c00c5950 0x7fa0c00c49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.670+0000 7fa0c4d52640 1 -- 172.21.15.161:0/4225854525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0c00c8ed0 msgr2=0x7fa0c00ab900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:48.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.670+0000 7fa0c4d52640 1 --2- 172.21.15.161:0/4225854525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0c00c8ed0 0x7fa0c00ab900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.670+0000 7fa0c4d52640 1 -- 172.21.15.161:0/4225854525 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c01eb800 con 0x7fa0c0152890 2024-03-28T23:32:48.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.670+0000 7fa0c4d52640 1 --2- 172.21.15.161:0/4225854525 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c0152890 0x7fa0c01e7f80 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fa0b800eed0 tx=0x7fa0b800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:48.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.670+0000 7fa0bcff9640 1 -- 172.21.15.161:0/4225854525 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8018020 con 0x7fa0c0152890 2024-03-28T23:32:48.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.670+0000 7fa0bcff9640 1 -- 172.21.15.161:0/4225854525 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0b800bd20 con 0x7fa0c0152890 2024-03-28T23:32:48.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.670+0000 7fa0c67b5640 1 -- 172.21.15.161:0/4225854525 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0c014a7e0 con 0x7fa0c0152890 2024-03-28T23:32:48.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.670+0000 7fa0bcff9640 1 -- 172.21.15.161:0/4225854525 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8014640 con 0x7fa0c0152890 2024-03-28T23:32:48.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.670+0000 7fa0c67b5640 1 -- 172.21.15.161:0/4225854525 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0c00c4400 con 0x7fa0c0152890 2024-03-28T23:32:48.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.672+0000 7fa0c67b5640 1 -- 172.21.15.161:0/4225854525 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0c004f4b0 con 0x7fa0c0152890 2024-03-28T23:32:48.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.672+0000 7fa0bcff9640 1 -- 172.21.15.161:0/4225854525 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa0b8005d10 con 0x7fa0c0152890 2024-03-28T23:32:48.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.672+0000 7fa0bcff9640 1 --2- 172.21.15.161:0/4225854525 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa0ac06b4d0 0x7fa0ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:48.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.672+0000 7fa0bffff640 1 --2- 172.21.15.161:0/4225854525 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa0ac06b4d0 0x7fa0ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:48.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.672+0000 7fa0bcff9640 1 -- 172.21.15.161:0/4225854525 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6957+0+0 (secure 0 0 0) 0x7fa0b808ab40 con 0x7fa0c0152890 2024-03-28T23:32:48.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.673+0000 7fa0bffff640 1 --2- 172.21.15.161:0/4225854525 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa0ac06b4d0 0x7fa0ac06d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0035000 tx=0x7fa0b0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:48.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.677+0000 7fa0bcff9640 1 -- 172.21.15.161:0/4225854525 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa0b8022070 con 0x7fa0c0152890 2024-03-28T23:32:48.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.927+0000 7fa0c67b5640 1 -- 172.21.15.161:0/4225854525 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa0c01eb5b0 con 0x7fa0ac06b4d0 2024-03-28T23:32:48.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.938+0000 7fa0bcff9640 1 -- 172.21.15.161:0/4225854525 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+485419 (secure 0 0 0) 0x7fa0c01eb5b0 con 0x7fa0ac06b4d0 2024-03-28T23:32:48.944 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:48.947 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:32:48.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.942+0000 7fa0c67b5640 1 -- 172.21.15.161:0/4225854525 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa0ac06b4d0 msgr2=0x7fa0ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:48.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.942+0000 7fa0c67b5640 1 --2- 172.21.15.161:0/4225854525 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa0ac06b4d0 0x7fa0ac06d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0035000 tx=0x7fa0b0008030 comp rx=0 tx=0).stop 2024-03-28T23:32:48.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.942+0000 7fa0c67b5640 1 -- 172.21.15.161:0/4225854525 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c0152890 msgr2=0x7fa0c01e7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:48.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.942+0000 7fa0c67b5640 1 --2- 172.21.15.161:0/4225854525 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c0152890 0x7fa0c01e7f80 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fa0b800eed0 tx=0x7fa0b800b480 comp rx=0 tx=0).stop 2024-03-28T23:32:48.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.942+0000 7fa0c67b5640 1 -- 172.21.15.161:0/4225854525 shutdown_connections 2024-03-28T23:32:48.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.942+0000 7fa0c67b5640 1 --2- 172.21.15.161:0/4225854525 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa0ac06b4d0 0x7fa0ac06d990 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.942+0000 7fa0c67b5640 1 --2- 172.21.15.161:0/4225854525 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c0152890 0x7fa0c01e7f80 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.942+0000 7fa0c67b5640 1 --2- 172.21.15.161:0/4225854525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0c00c8ed0 0x7fa0c00ab900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.942+0000 7fa0c67b5640 1 --2- 172.21.15.161:0/4225854525 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0c00c5950 0x7fa0c00c49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:48.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.942+0000 7fa0c67b5640 1 -- 172.21.15.161:0/4225854525 >> 172.21.15.161:0/4225854525 conn(0x7fa0c00ac620 msgr2=0x7fa0c0151560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:48.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.943+0000 7fa0c67b5640 1 -- 172.21.15.161:0/4225854525 shutdown_connections 2024-03-28T23:32:48.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:48.943+0000 7fa0c67b5640 1 -- 172.21.15.161:0/4225854525 wait complete. 2024-03-28T23:32:48.958 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-03-28T23:32:47.345901+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":202,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240076,"kb_used_data":7276,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754734644,"statfs":{"total":773094113280,"available":772848275456,"internally_reserved":0,"allocated":7450624,"data_stored":4616622,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.247617"},"pg_stats":[{"pgid":"14.3d","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.110397+0000","last_change":"2024-03-28T23:32:44.110397+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.110397+0000","last_undegraded":"2024-03-28T23:32:44.110397+0000","last_fullsized":"2024-03-28T23:32:44.110397+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3f","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108246+0000","last_change":"2024-03-28T23:32:33.046706+0000","last_active":"2024-03-28T23:32:44.108246+0000","last_peered":"2024-03-28T23:32:44.108246+0000","last_clean":"2024-03-28T23:32:44.108246+0000","last_became_active":"2024-03-28T23:32:33.046222+0000","last_became_peered":"2024-03-28T23:32:33.046222+0000","last_unstale":"2024-03-28T23:32:44.108246+0000","last_undegraded":"2024-03-28T23:32:44.108246+0000","last_fullsized":"2024-03-28T23:32:44.108246+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:02:45.190566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.109309+0000","last_change":"2024-03-28T23:32:44.109309+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.109309+0000","last_undegraded":"2024-03-28T23:32:44.109309+0000","last_fullsized":"2024-03-28T23:32:44.109309+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122914+0000","last_change":"2024-03-28T23:32:33.051174+0000","last_active":"2024-03-28T23:32:44.122914+0000","last_peered":"2024-03-28T23:32:44.122914+0000","last_clean":"2024-03-28T23:32:44.122914+0000","last_became_active":"2024-03-28T23:32:33.050345+0000","last_became_peered":"2024-03-28T23:32:33.050345+0000","last_unstale":"2024-03-28T23:32:44.122914+0000","last_undegraded":"2024-03-28T23:32:44.122914+0000","last_fullsized":"2024-03-28T23:32:44.122914+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:41:39.657731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.3f","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111805+0000","last_change":"2024-03-28T23:32:44.111805+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111805+0000","last_undegraded":"2024-03-28T23:32:44.111805+0000","last_fullsized":"2024-03-28T23:32:44.111805+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108168+0000","last_change":"2024-03-28T23:32:33.048785+0000","last_active":"2024-03-28T23:32:44.108168+0000","last_peered":"2024-03-28T23:32:44.108168+0000","last_clean":"2024-03-28T23:32:44.108168+0000","last_became_active":"2024-03-28T23:32:33.048519+0000","last_became_peered":"2024-03-28T23:32:33.048519+0000","last_unstale":"2024-03-28T23:32:44.108168+0000","last_undegraded":"2024-03-28T23:32:44.108168+0000","last_fullsized":"2024-03-28T23:32:44.108168+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:21.464120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111326+0000","last_change":"2024-03-28T23:32:44.111326+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111326+0000","last_undegraded":"2024-03-28T23:32:44.111326+0000","last_fullsized":"2024-03-28T23:32:44.111326+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106875+0000","last_change":"2024-03-28T23:32:33.033535+0000","last_active":"2024-03-28T23:32:44.106875+0000","last_peered":"2024-03-28T23:32:44.106875+0000","last_clean":"2024-03-28T23:32:44.106875+0000","last_became_active":"2024-03-28T23:32:33.033219+0000","last_became_peered":"2024-03-28T23:32:33.033219+0000","last_unstale":"2024-03-28T23:32:44.106875+0000","last_undegraded":"2024-03-28T23:32:44.106875+0000","last_fullsized":"2024-03-28T23:32:44.106875+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:39:39.113457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.106821+0000","last_change":"2024-03-28T23:32:44.106821+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.106821+0000","last_undegraded":"2024-03-28T23:32:44.106821+0000","last_fullsized":"2024-03-28T23:32:44.106821+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109010+0000","last_change":"2024-03-28T23:32:33.042972+0000","last_active":"2024-03-28T23:32:44.109010+0000","last_peered":"2024-03-28T23:32:44.109010+0000","last_clean":"2024-03-28T23:32:44.109010+0000","last_became_active":"2024-03-28T23:32:33.042665+0000","last_became_peered":"2024-03-28T23:32:33.042665+0000","last_unstale":"2024-03-28T23:32:44.109010+0000","last_undegraded":"2024-03-28T23:32:44.109010+0000","last_fullsized":"2024-03-28T23:32:44.109010+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:41:50.190883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.109201+0000","last_change":"2024-03-28T23:32:44.109201+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.109201+0000","last_undegraded":"2024-03-28T23:32:44.109201+0000","last_fullsized":"2024-03-28T23:32:44.109201+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113352+0000","last_change":"2024-03-28T23:32:33.030549+0000","last_active":"2024-03-28T23:32:44.113352+0000","last_peered":"2024-03-28T23:32:44.113352+0000","last_clean":"2024-03-28T23:32:44.113352+0000","last_became_active":"2024-03-28T23:32:33.030299+0000","last_became_peered":"2024-03-28T23:32:33.030299+0000","last_unstale":"2024-03-28T23:32:44.113352+0000","last_undegraded":"2024-03-28T23:32:44.113352+0000","last_fullsized":"2024-03-28T23:32:44.113352+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:35.055035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111404+0000","last_change":"2024-03-28T23:32:44.111404+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111404+0000","last_undegraded":"2024-03-28T23:32:44.111404+0000","last_fullsized":"2024-03-28T23:32:44.111404+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.111244+0000","last_change":"2024-03-28T23:32:33.044477+0000","last_active":"2024-03-28T23:32:44.111244+0000","last_peered":"2024-03-28T23:32:44.111244+0000","last_clean":"2024-03-28T23:32:44.111244+0000","last_became_active":"2024-03-28T23:32:33.043960+0000","last_became_peered":"2024-03-28T23:32:33.043960+0000","last_unstale":"2024-03-28T23:32:44.111244+0000","last_undegraded":"2024-03-28T23:32:44.111244+0000","last_fullsized":"2024-03-28T23:32:44.111244+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:21.464120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.113215+0000","last_change":"2024-03-28T23:32:44.113215+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.113215+0000","last_undegraded":"2024-03-28T23:32:44.113215+0000","last_fullsized":"2024-03-28T23:32:44.113215+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108736+0000","last_change":"2024-03-28T23:32:33.052803+0000","last_active":"2024-03-28T23:32:44.108736+0000","last_peered":"2024-03-28T23:32:44.108736+0000","last_clean":"2024-03-28T23:32:44.108736+0000","last_became_active":"2024-03-28T23:32:33.052524+0000","last_became_peered":"2024-03-28T23:32:33.052524+0000","last_unstale":"2024-03-28T23:32:44.108736+0000","last_undegraded":"2024-03-28T23:32:44.108736+0000","last_fullsized":"2024-03-28T23:32:44.108736+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:19:09.500642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.117756+0000","last_change":"2024-03-28T23:32:44.117756+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.117756+0000","last_undegraded":"2024-03-28T23:32:44.117756+0000","last_fullsized":"2024-03-28T23:32:44.117756+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112756+0000","last_change":"2024-03-28T23:32:33.042587+0000","last_active":"2024-03-28T23:32:44.112756+0000","last_peered":"2024-03-28T23:32:44.112756+0000","last_clean":"2024-03-28T23:32:44.112756+0000","last_became_active":"2024-03-28T23:32:33.037002+0000","last_became_peered":"2024-03-28T23:32:33.037002+0000","last_unstale":"2024-03-28T23:32:44.112756+0000","last_undegraded":"2024-03-28T23:32:44.112756+0000","last_fullsized":"2024-03-28T23:32:44.112756+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:36:03.951054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107890+0000","last_change":"2024-03-28T23:32:44.107890+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107890+0000","last_undegraded":"2024-03-28T23:32:44.107890+0000","last_fullsized":"2024-03-28T23:32:44.107890+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107498+0000","last_change":"2024-03-28T23:32:33.039941+0000","last_active":"2024-03-28T23:32:44.107498+0000","last_peered":"2024-03-28T23:32:44.107498+0000","last_clean":"2024-03-28T23:32:44.107498+0000","last_became_active":"2024-03-28T23:32:33.039656+0000","last_became_peered":"2024-03-28T23:32:33.039656+0000","last_unstale":"2024-03-28T23:32:44.107498+0000","last_undegraded":"2024-03-28T23:32:44.107498+0000","last_fullsized":"2024-03-28T23:32:44.107498+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:16.850667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111858+0000","last_change":"2024-03-28T23:32:44.111858+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111858+0000","last_undegraded":"2024-03-28T23:32:44.111858+0000","last_fullsized":"2024-03-28T23:32:44.111858+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122690+0000","last_change":"2024-03-28T23:32:33.052329+0000","last_active":"2024-03-28T23:32:44.122690+0000","last_peered":"2024-03-28T23:32:44.122690+0000","last_clean":"2024-03-28T23:32:44.122690+0000","last_became_active":"2024-03-28T23:32:33.051833+0000","last_became_peered":"2024-03-28T23:32:33.051833+0000","last_unstale":"2024-03-28T23:32:44.122690+0000","last_undegraded":"2024-03-28T23:32:44.122690+0000","last_fullsized":"2024-03-28T23:32:44.122690+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:05.929331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108222+0000","last_change":"2024-03-28T23:32:44.108222+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108222+0000","last_undegraded":"2024-03-28T23:32:44.108222+0000","last_fullsized":"2024-03-28T23:32:44.108222+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106984+0000","last_change":"2024-03-28T23:32:33.042337+0000","last_active":"2024-03-28T23:32:44.106984+0000","last_peered":"2024-03-28T23:32:44.106984+0000","last_clean":"2024-03-28T23:32:44.106984+0000","last_became_active":"2024-03-28T23:32:33.042062+0000","last_became_peered":"2024-03-28T23:32:33.042062+0000","last_unstale":"2024-03-28T23:32:44.106984+0000","last_undegraded":"2024-03-28T23:32:44.106984+0000","last_fullsized":"2024-03-28T23:32:44.106984+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:13:34.921740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108039+0000","last_change":"2024-03-28T23:32:44.108039+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108039+0000","last_undegraded":"2024-03-28T23:32:44.108039+0000","last_fullsized":"2024-03-28T23:32:44.108039+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109102+0000","last_change":"2024-03-28T23:32:33.046685+0000","last_active":"2024-03-28T23:32:44.109102+0000","last_peered":"2024-03-28T23:32:44.109102+0000","last_clean":"2024-03-28T23:32:44.109102+0000","last_became_active":"2024-03-28T23:32:33.046206+0000","last_became_peered":"2024-03-28T23:32:33.046206+0000","last_unstale":"2024-03-28T23:32:44.109102+0000","last_undegraded":"2024-03-28T23:32:44.109102+0000","last_fullsized":"2024-03-28T23:32:44.109102+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:43:22.001147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.113196+0000","last_change":"2024-03-28T23:32:44.113196+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.113196+0000","last_undegraded":"2024-03-28T23:32:44.113196+0000","last_fullsized":"2024-03-28T23:32:44.113196+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113445+0000","last_change":"2024-03-28T23:32:33.029880+0000","last_active":"2024-03-28T23:32:44.113445+0000","last_peered":"2024-03-28T23:32:44.113445+0000","last_clean":"2024-03-28T23:32:44.113445+0000","last_became_active":"2024-03-28T23:32:33.029582+0000","last_became_peered":"2024-03-28T23:32:33.029582+0000","last_unstale":"2024-03-28T23:32:44.113445+0000","last_undegraded":"2024-03-28T23:32:44.113445+0000","last_fullsized":"2024-03-28T23:32:44.113445+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:38:39.283932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111833+0000","last_change":"2024-03-28T23:32:44.111833+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111833+0000","last_undegraded":"2024-03-28T23:32:44.111833+0000","last_fullsized":"2024-03-28T23:32:44.111833+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121362+0000","last_change":"2024-03-28T23:32:33.040102+0000","last_active":"2024-03-28T23:32:44.121362+0000","last_peered":"2024-03-28T23:32:44.121362+0000","last_clean":"2024-03-28T23:32:44.121362+0000","last_became_active":"2024-03-28T23:32:33.039660+0000","last_became_peered":"2024-03-28T23:32:33.039660+0000","last_unstale":"2024-03-28T23:32:44.121362+0000","last_undegraded":"2024-03-28T23:32:44.121362+0000","last_fullsized":"2024-03-28T23:32:44.121362+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:35.055035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112037+0000","last_change":"2024-03-28T23:32:44.112037+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112037+0000","last_undegraded":"2024-03-28T23:32:44.112037+0000","last_fullsized":"2024-03-28T23:32:44.112037+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108875+0000","last_change":"2024-03-28T23:32:33.035442+0000","last_active":"2024-03-28T23:32:44.108875+0000","last_peered":"2024-03-28T23:32:44.108875+0000","last_clean":"2024-03-28T23:32:44.108875+0000","last_became_active":"2024-03-28T23:32:33.035154+0000","last_became_peered":"2024-03-28T23:32:33.035154+0000","last_unstale":"2024-03-28T23:32:44.108875+0000","last_undegraded":"2024-03-28T23:32:44.108875+0000","last_fullsized":"2024-03-28T23:32:44.108875+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:34:22.775718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.115970+0000","last_change":"2024-03-28T23:32:44.115970+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.115970+0000","last_undegraded":"2024-03-28T23:32:44.115970+0000","last_fullsized":"2024-03-28T23:32:44.115970+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.111211+0000","last_change":"2024-03-28T23:32:33.047782+0000","last_active":"2024-03-28T23:32:44.111211+0000","last_peered":"2024-03-28T23:32:44.111211+0000","last_clean":"2024-03-28T23:32:44.111211+0000","last_became_active":"2024-03-28T23:32:33.047254+0000","last_became_peered":"2024-03-28T23:32:33.047254+0000","last_unstale":"2024-03-28T23:32:44.111211+0000","last_undegraded":"2024-03-28T23:32:44.111211+0000","last_fullsized":"2024-03-28T23:32:44.111211+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:19.004555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108940+0000","last_change":"2024-03-28T23:32:44.108940+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108940+0000","last_undegraded":"2024-03-28T23:32:44.108940+0000","last_fullsized":"2024-03-28T23:32:44.108940+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107847+0000","last_change":"2024-03-28T23:32:33.037321+0000","last_active":"2024-03-28T23:32:44.107847+0000","last_peered":"2024-03-28T23:32:44.107847+0000","last_clean":"2024-03-28T23:32:44.107847+0000","last_became_active":"2024-03-28T23:32:33.036961+0000","last_became_peered":"2024-03-28T23:32:33.036961+0000","last_unstale":"2024-03-28T23:32:44.107847+0000","last_undegraded":"2024-03-28T23:32:44.107847+0000","last_fullsized":"2024-03-28T23:32:44.107847+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:25.849919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.120323+0000","last_change":"2024-03-28T23:32:44.120323+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.120323+0000","last_undegraded":"2024-03-28T23:32:44.120323+0000","last_fullsized":"2024-03-28T23:32:44.120323+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113594+0000","last_change":"2024-03-28T23:32:33.053071+0000","last_active":"2024-03-28T23:32:44.113594+0000","last_peered":"2024-03-28T23:32:44.113594+0000","last_clean":"2024-03-28T23:32:44.113594+0000","last_became_active":"2024-03-28T23:32:33.052784+0000","last_became_peered":"2024-03-28T23:32:33.052784+0000","last_unstale":"2024-03-28T23:32:44.113594+0000","last_undegraded":"2024-03-28T23:32:44.113594+0000","last_fullsized":"2024-03-28T23:32:44.113594+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:58:48.231248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112002+0000","last_change":"2024-03-28T23:32:44.112002+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112002+0000","last_undegraded":"2024-03-28T23:32:44.112002+0000","last_fullsized":"2024-03-28T23:32:44.112002+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109678+0000","last_change":"2024-03-28T23:32:33.052720+0000","last_active":"2024-03-28T23:32:44.109678+0000","last_peered":"2024-03-28T23:32:44.109678+0000","last_clean":"2024-03-28T23:32:44.109678+0000","last_became_active":"2024-03-28T23:32:33.052460+0000","last_became_peered":"2024-03-28T23:32:33.052460+0000","last_unstale":"2024-03-28T23:32:44.109678+0000","last_undegraded":"2024-03-28T23:32:44.109678+0000","last_fullsized":"2024-03-28T23:32:44.109678+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:30.891315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107873+0000","last_change":"2024-03-28T23:32:44.107873+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107873+0000","last_undegraded":"2024-03-28T23:32:44.107873+0000","last_fullsized":"2024-03-28T23:32:44.107873+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112648+0000","last_change":"2024-03-28T23:32:33.035666+0000","last_active":"2024-03-28T23:32:44.112648+0000","last_peered":"2024-03-28T23:32:44.112648+0000","last_clean":"2024-03-28T23:32:44.112648+0000","last_became_active":"2024-03-28T23:32:33.035403+0000","last_became_peered":"2024-03-28T23:32:33.035403+0000","last_unstale":"2024-03-28T23:32:44.112648+0000","last_undegraded":"2024-03-28T23:32:44.112648+0000","last_fullsized":"2024-03-28T23:32:44.112648+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:40:51.269760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111443+0000","last_change":"2024-03-28T23:32:44.111443+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111443+0000","last_undegraded":"2024-03-28T23:32:44.111443+0000","last_fullsized":"2024-03-28T23:32:44.111443+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109244+0000","last_change":"2024-03-28T23:32:33.047992+0000","last_active":"2024-03-28T23:32:44.109244+0000","last_peered":"2024-03-28T23:32:44.109244+0000","last_clean":"2024-03-28T23:32:44.109244+0000","last_became_active":"2024-03-28T23:32:33.047058+0000","last_became_peered":"2024-03-28T23:32:33.047058+0000","last_unstale":"2024-03-28T23:32:44.109244+0000","last_undegraded":"2024-03-28T23:32:44.109244+0000","last_fullsized":"2024-03-28T23:32:44.109244+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:35:30.886935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.110595+0000","last_change":"2024-03-28T23:32:44.110595+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.110595+0000","last_undegraded":"2024-03-28T23:32:44.110595+0000","last_fullsized":"2024-03-28T23:32:44.110595+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112366+0000","last_change":"2024-03-28T23:32:33.029536+0000","last_active":"2024-03-28T23:32:44.112366+0000","last_peered":"2024-03-28T23:32:44.112366+0000","last_clean":"2024-03-28T23:32:44.112366+0000","last_became_active":"2024-03-28T23:32:33.029265+0000","last_became_peered":"2024-03-28T23:32:33.029265+0000","last_unstale":"2024-03-28T23:32:44.112366+0000","last_undegraded":"2024-03-28T23:32:44.112366+0000","last_fullsized":"2024-03-28T23:32:44.112366+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:51:11.447507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.110858+0000","last_change":"2024-03-28T23:32:44.110858+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.110858+0000","last_undegraded":"2024-03-28T23:32:44.110858+0000","last_fullsized":"2024-03-28T23:32:44.110858+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121488+0000","last_change":"2024-03-28T23:32:33.041328+0000","last_active":"2024-03-28T23:32:44.121488+0000","last_peered":"2024-03-28T23:32:44.121488+0000","last_clean":"2024-03-28T23:32:44.121488+0000","last_became_active":"2024-03-28T23:32:33.041071+0000","last_became_peered":"2024-03-28T23:32:33.041071+0000","last_unstale":"2024-03-28T23:32:44.121488+0000","last_undegraded":"2024-03-28T23:32:44.121488+0000","last_fullsized":"2024-03-28T23:32:44.121488+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:00.358180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107392+0000","last_change":"2024-03-28T23:32:44.107392+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107392+0000","last_undegraded":"2024-03-28T23:32:44.107392+0000","last_fullsized":"2024-03-28T23:32:44.107392+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122766+0000","last_change":"2024-03-28T23:32:33.043840+0000","last_active":"2024-03-28T23:32:44.122766+0000","last_peered":"2024-03-28T23:32:44.122766+0000","last_clean":"2024-03-28T23:32:44.122766+0000","last_became_active":"2024-03-28T23:32:33.043569+0000","last_became_peered":"2024-03-28T23:32:33.043569+0000","last_unstale":"2024-03-28T23:32:44.122766+0000","last_undegraded":"2024-03-28T23:32:44.122766+0000","last_fullsized":"2024-03-28T23:32:44.122766+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:28.463493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108552+0000","last_change":"2024-03-28T23:32:44.108552+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108552+0000","last_undegraded":"2024-03-28T23:32:44.108552+0000","last_fullsized":"2024-03-28T23:32:44.108552+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107631+0000","last_change":"2024-03-28T23:32:33.047594+0000","last_active":"2024-03-28T23:32:44.107631+0000","last_peered":"2024-03-28T23:32:44.107631+0000","last_clean":"2024-03-28T23:32:44.107631+0000","last_became_active":"2024-03-28T23:32:33.047031+0000","last_became_peered":"2024-03-28T23:32:33.047031+0000","last_unstale":"2024-03-28T23:32:44.107631+0000","last_undegraded":"2024-03-28T23:32:44.107631+0000","last_fullsized":"2024-03-28T23:32:44.107631+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:30:12.769818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112271+0000","last_change":"2024-03-28T23:32:44.112271+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112271+0000","last_undegraded":"2024-03-28T23:32:44.112271+0000","last_fullsized":"2024-03-28T23:32:44.112271+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107919+0000","last_change":"2024-03-28T23:32:33.043396+0000","last_active":"2024-03-28T23:32:44.107919+0000","last_peered":"2024-03-28T23:32:44.107919+0000","last_clean":"2024-03-28T23:32:44.107919+0000","last_became_active":"2024-03-28T23:32:33.043123+0000","last_became_peered":"2024-03-28T23:32:33.043123+0000","last_unstale":"2024-03-28T23:32:44.107919+0000","last_undegraded":"2024-03-28T23:32:44.107919+0000","last_fullsized":"2024-03-28T23:32:44.107919+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:28:36.535909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.114282+0000","last_change":"2024-03-28T23:32:44.114282+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.114282+0000","last_undegraded":"2024-03-28T23:32:44.114282+0000","last_fullsized":"2024-03-28T23:32:44.114282+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109350+0000","last_change":"2024-03-28T23:32:33.042288+0000","last_active":"2024-03-28T23:32:44.109350+0000","last_peered":"2024-03-28T23:32:44.109350+0000","last_clean":"2024-03-28T23:32:44.109350+0000","last_became_active":"2024-03-28T23:32:33.042007+0000","last_became_peered":"2024-03-28T23:32:33.042007+0000","last_unstale":"2024-03-28T23:32:44.109350+0000","last_undegraded":"2024-03-28T23:32:44.109350+0000","last_fullsized":"2024-03-28T23:32:44.109350+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:16:12.634582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.110873+0000","last_change":"2024-03-28T23:32:44.110873+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.110873+0000","last_undegraded":"2024-03-28T23:32:44.110873+0000","last_fullsized":"2024-03-28T23:32:44.110873+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122615+0000","last_change":"2024-03-28T23:32:33.041336+0000","last_active":"2024-03-28T23:32:44.122615+0000","last_peered":"2024-03-28T23:32:44.122615+0000","last_clean":"2024-03-28T23:32:44.122615+0000","last_became_active":"2024-03-28T23:32:33.040865+0000","last_became_peered":"2024-03-28T23:32:33.040865+0000","last_unstale":"2024-03-28T23:32:44.122615+0000","last_undegraded":"2024-03-28T23:32:44.122615+0000","last_fullsized":"2024-03-28T23:32:44.122615+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:44:58.784172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.117069+0000","last_change":"2024-03-28T23:32:44.117069+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.117069+0000","last_undegraded":"2024-03-28T23:32:44.117069+0000","last_fullsized":"2024-03-28T23:32:44.117069+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106185+0000","last_change":"2024-03-28T23:32:33.049222+0000","last_active":"2024-03-28T23:32:44.106185+0000","last_peered":"2024-03-28T23:32:44.106185+0000","last_clean":"2024-03-28T23:32:44.106185+0000","last_became_active":"2024-03-28T23:32:33.048933+0000","last_became_peered":"2024-03-28T23:32:33.048933+0000","last_unstale":"2024-03-28T23:32:44.106185+0000","last_undegraded":"2024-03-28T23:32:44.106185+0000","last_fullsized":"2024-03-28T23:32:44.106185+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:17:45.545141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107516+0000","last_change":"2024-03-28T23:32:44.107516+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107516+0000","last_undegraded":"2024-03-28T23:32:44.107516+0000","last_fullsized":"2024-03-28T23:32:44.107516+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108405+0000","last_change":"2024-03-28T23:32:33.047977+0000","last_active":"2024-03-28T23:32:44.108405+0000","last_peered":"2024-03-28T23:32:44.108405+0000","last_clean":"2024-03-28T23:32:44.108405+0000","last_became_active":"2024-03-28T23:32:33.047699+0000","last_became_peered":"2024-03-28T23:32:33.047699+0000","last_unstale":"2024-03-28T23:32:44.108405+0000","last_undegraded":"2024-03-28T23:32:44.108405+0000","last_fullsized":"2024-03-28T23:32:44.108405+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:38:51.513907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111342+0000","last_change":"2024-03-28T23:32:44.111342+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111342+0000","last_undegraded":"2024-03-28T23:32:44.111342+0000","last_fullsized":"2024-03-28T23:32:44.111342+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108988+0000","last_change":"2024-03-28T23:32:33.036527+0000","last_active":"2024-03-28T23:32:44.108988+0000","last_peered":"2024-03-28T23:32:44.108988+0000","last_clean":"2024-03-28T23:32:44.108988+0000","last_became_active":"2024-03-28T23:32:33.036269+0000","last_became_peered":"2024-03-28T23:32:33.036269+0000","last_unstale":"2024-03-28T23:32:44.108988+0000","last_undegraded":"2024-03-28T23:32:44.108988+0000","last_fullsized":"2024-03-28T23:32:44.108988+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:43:35.946992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112540+0000","last_change":"2024-03-28T23:32:44.112540+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112540+0000","last_undegraded":"2024-03-28T23:32:44.112540+0000","last_fullsized":"2024-03-28T23:32:44.112540+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121867+0000","last_change":"2024-03-28T23:32:44.118134+0000","last_active":"2024-03-28T23:32:44.121867+0000","last_peered":"2024-03-28T23:32:44.121867+0000","last_clean":"2024-03-28T23:32:44.121867+0000","last_became_active":"2024-03-28T23:32:44.117870+0000","last_became_peered":"2024-03-28T23:32:44.117870+0000","last_unstale":"2024-03-28T23:32:44.121867+0000","last_undegraded":"2024-03-28T23:32:44.121867+0000","last_fullsized":"2024-03-28T23:32:44.121867+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:41:04.213333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109389+0000","last_change":"2024-03-28T23:32:33.038016+0000","last_active":"2024-03-28T23:32:44.109389+0000","last_peered":"2024-03-28T23:32:44.109389+0000","last_clean":"2024-03-28T23:32:44.109389+0000","last_became_active":"2024-03-28T23:32:33.037754+0000","last_became_peered":"2024-03-28T23:32:33.037754+0000","last_unstale":"2024-03-28T23:32:44.109389+0000","last_undegraded":"2024-03-28T23:32:44.109389+0000","last_fullsized":"2024-03-28T23:32:44.109389+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:21.878156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"72'4","reported_seq":22,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.111297+0000","last_change":"2024-03-28T23:32:32.040468+0000","last_active":"2024-03-28T23:32:44.111297+0000","last_peered":"2024-03-28T23:32:44.111297+0000","last_clean":"2024-03-28T23:32:44.111297+0000","last_became_active":"2024-03-28T23:32:32.039342+0000","last_became_peered":"2024-03-28T23:32:32.039342+0000","last_unstale":"2024-03-28T23:32:44.111297+0000","last_undegraded":"2024-03-28T23:32:44.111297+0000","last_fullsized":"2024-03-28T23:32:44.111297+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:05.786248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108504+0000","last_change":"2024-03-28T23:32:44.108504+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108504+0000","last_undegraded":"2024-03-28T23:32:44.108504+0000","last_fullsized":"2024-03-28T23:32:44.108504+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.127347+0000","last_change":"2024-03-28T23:32:44.126084+0000","last_active":"2024-03-28T23:32:44.127347+0000","last_peered":"2024-03-28T23:32:44.127347+0000","last_clean":"2024-03-28T23:32:44.127347+0000","last_became_active":"2024-03-28T23:32:44.125728+0000","last_became_peered":"2024-03-28T23:32:44.125728+0000","last_unstale":"2024-03-28T23:32:44.127347+0000","last_undegraded":"2024-03-28T23:32:44.127347+0000","last_fullsized":"2024-03-28T23:32:44.127347+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:17.008056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121405+0000","last_change":"2024-03-28T23:32:33.039778+0000","last_active":"2024-03-28T23:32:44.121405+0000","last_peered":"2024-03-28T23:32:44.121405+0000","last_clean":"2024-03-28T23:32:44.121405+0000","last_became_active":"2024-03-28T23:32:33.039506+0000","last_became_peered":"2024-03-28T23:32:33.039506+0000","last_unstale":"2024-03-28T23:32:44.121405+0000","last_undegraded":"2024-03-28T23:32:44.121405+0000","last_fullsized":"2024-03-28T23:32:44.121405+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:38:39.283932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"72'1","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107225+0000","last_change":"2024-03-28T23:32:32.041093+0000","last_active":"2024-03-28T23:32:44.107225+0000","last_peered":"2024-03-28T23:32:44.107225+0000","last_clean":"2024-03-28T23:32:44.107225+0000","last_became_active":"2024-03-28T23:32:32.040827+0000","last_became_peered":"2024-03-28T23:32:32.040827+0000","last_unstale":"2024-03-28T23:32:44.107225+0000","last_undegraded":"2024-03-28T23:32:44.107225+0000","last_fullsized":"2024-03-28T23:32:44.107225+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:55.544945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108669+0000","last_change":"2024-03-28T23:32:44.108669+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108669+0000","last_undegraded":"2024-03-28T23:32:44.108669+0000","last_fullsized":"2024-03-28T23:32:44.108669+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.120787+0000","last_change":"2024-03-28T23:32:44.119556+0000","last_active":"2024-03-28T23:32:44.120787+0000","last_peered":"2024-03-28T23:32:44.120787+0000","last_clean":"2024-03-28T23:32:44.120787+0000","last_became_active":"2024-03-28T23:32:44.119246+0000","last_became_peered":"2024-03-28T23:32:44.119246+0000","last_unstale":"2024-03-28T23:32:44.120787+0000","last_undegraded":"2024-03-28T23:32:44.120787+0000","last_fullsized":"2024-03-28T23:32:44.120787+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:36.928644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112355+0000","last_change":"2024-03-28T23:32:33.044887+0000","last_active":"2024-03-28T23:32:44.112355+0000","last_peered":"2024-03-28T23:32:44.112355+0000","last_clean":"2024-03-28T23:32:44.112355+0000","last_became_active":"2024-03-28T23:32:33.042005+0000","last_became_peered":"2024-03-28T23:32:33.042005+0000","last_unstale":"2024-03-28T23:32:44.112355+0000","last_undegraded":"2024-03-28T23:32:44.112355+0000","last_fullsized":"2024-03-28T23:32:44.112355+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:41.257241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"72'2","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108029+0000","last_change":"2024-03-28T23:32:32.040527+0000","last_active":"2024-03-28T23:32:44.108029+0000","last_peered":"2024-03-28T23:32:44.108029+0000","last_clean":"2024-03-28T23:32:44.108029+0000","last_became_active":"2024-03-28T23:32:32.040142+0000","last_became_peered":"2024-03-28T23:32:32.040142+0000","last_unstale":"2024-03-28T23:32:44.108029+0000","last_undegraded":"2024-03-28T23:32:44.108029+0000","last_fullsized":"2024-03-28T23:32:44.108029+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:13:15.856750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107395+0000","last_change":"2024-03-28T23:32:44.107395+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107395+0000","last_undegraded":"2024-03-28T23:32:44.107395+0000","last_fullsized":"2024-03-28T23:32:44.107395+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122155+0000","last_change":"2024-03-28T23:32:44.116203+0000","last_active":"2024-03-28T23:32:44.122155+0000","last_peered":"2024-03-28T23:32:44.122155+0000","last_clean":"2024-03-28T23:32:44.122155+0000","last_became_active":"2024-03-28T23:32:44.115846+0000","last_became_peered":"2024-03-28T23:32:44.115846+0000","last_unstale":"2024-03-28T23:32:44.122155+0000","last_undegraded":"2024-03-28T23:32:44.122155+0000","last_fullsized":"2024-03-28T23:32:44.122155+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:58.429743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112741+0000","last_change":"2024-03-28T23:32:33.040988+0000","last_active":"2024-03-28T23:32:44.112741+0000","last_peered":"2024-03-28T23:32:44.112741+0000","last_clean":"2024-03-28T23:32:44.112741+0000","last_became_active":"2024-03-28T23:32:33.036045+0000","last_became_peered":"2024-03-28T23:32:33.036045+0000","last_unstale":"2024-03-28T23:32:44.112741+0000","last_undegraded":"2024-03-28T23:32:44.112741+0000","last_fullsized":"2024-03-28T23:32:44.112741+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:10:39.981362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"72'2","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112781+0000","last_change":"2024-03-28T23:32:32.036856+0000","last_active":"2024-03-28T23:32:44.112781+0000","last_peered":"2024-03-28T23:32:44.112781+0000","last_clean":"2024-03-28T23:32:44.112781+0000","last_became_active":"2024-03-28T23:32:32.036565+0000","last_became_peered":"2024-03-28T23:32:32.036565+0000","last_unstale":"2024-03-28T23:32:44.112781+0000","last_undegraded":"2024-03-28T23:32:44.112781+0000","last_fullsized":"2024-03-28T23:32:44.112781+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:11:01.106436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.106357+0000","last_change":"2024-03-28T23:32:44.106357+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.106357+0000","last_undegraded":"2024-03-28T23:32:44.106357+0000","last_fullsized":"2024-03-28T23:32:44.106357+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.118200+0000","last_change":"2024-03-28T23:32:44.116097+0000","last_active":"2024-03-28T23:32:44.118200+0000","last_peered":"2024-03-28T23:32:44.118200+0000","last_clean":"2024-03-28T23:32:44.118200+0000","last_became_active":"2024-03-28T23:32:44.115810+0000","last_became_peered":"2024-03-28T23:32:44.115810+0000","last_unstale":"2024-03-28T23:32:44.118200+0000","last_undegraded":"2024-03-28T23:32:44.118200+0000","last_fullsized":"2024-03-28T23:32:44.118200+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:45:09.862897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122437+0000","last_change":"2024-03-28T23:32:33.047911+0000","last_active":"2024-03-28T23:32:44.122437+0000","last_peered":"2024-03-28T23:32:44.122437+0000","last_clean":"2024-03-28T23:32:44.122437+0000","last_became_active":"2024-03-28T23:32:33.047616+0000","last_became_peered":"2024-03-28T23:32:33.047616+0000","last_unstale":"2024-03-28T23:32:44.122437+0000","last_undegraded":"2024-03-28T23:32:44.122437+0000","last_fullsized":"2024-03-28T23:32:44.122437+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:58:48.231248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"72'1","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112850+0000","last_change":"2024-03-28T23:32:32.036829+0000","last_active":"2024-03-28T23:32:44.112850+0000","last_peered":"2024-03-28T23:32:44.112850+0000","last_clean":"2024-03-28T23:32:44.112850+0000","last_became_active":"2024-03-28T23:32:32.036571+0000","last_became_peered":"2024-03-28T23:32:32.036571+0000","last_unstale":"2024-03-28T23:32:44.112850+0000","last_undegraded":"2024-03-28T23:32:44.112850+0000","last_fullsized":"2024-03-28T23:32:44.112850+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:32:03.789593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.116621+0000","last_change":"2024-03-28T23:32:44.116621+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.116621+0000","last_undegraded":"2024-03-28T23:32:44.116621+0000","last_fullsized":"2024-03-28T23:32:44.116621+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.120399+0000","last_change":"2024-03-28T23:32:44.117981+0000","last_active":"2024-03-28T23:32:44.120399+0000","last_peered":"2024-03-28T23:32:44.120399+0000","last_clean":"2024-03-28T23:32:44.120399+0000","last_became_active":"2024-03-28T23:32:44.117709+0000","last_became_peered":"2024-03-28T23:32:44.117709+0000","last_unstale":"2024-03-28T23:32:44.120399+0000","last_undegraded":"2024-03-28T23:32:44.120399+0000","last_fullsized":"2024-03-28T23:32:44.120399+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:13:27.942873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122698+0000","last_change":"2024-03-28T23:32:33.052330+0000","last_active":"2024-03-28T23:32:44.122698+0000","last_peered":"2024-03-28T23:32:44.122698+0000","last_clean":"2024-03-28T23:32:44.122698+0000","last_became_active":"2024-03-28T23:32:33.051828+0000","last_became_peered":"2024-03-28T23:32:33.051828+0000","last_unstale":"2024-03-28T23:32:44.122698+0000","last_undegraded":"2024-03-28T23:32:44.122698+0000","last_fullsized":"2024-03-28T23:32:44.122698+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:40:53.134608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"72'1","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106048+0000","last_change":"2024-03-28T23:32:32.036739+0000","last_active":"2024-03-28T23:32:44.106048+0000","last_peered":"2024-03-28T23:32:44.106048+0000","last_clean":"2024-03-28T23:32:44.106048+0000","last_became_active":"2024-03-28T23:32:32.036471+0000","last_became_peered":"2024-03-28T23:32:32.036471+0000","last_unstale":"2024-03-28T23:32:44.106048+0000","last_undegraded":"2024-03-28T23:32:44.106048+0000","last_fullsized":"2024-03-28T23:32:44.106048+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:48.704351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112770+0000","last_change":"2024-03-28T23:32:44.112770+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112770+0000","last_undegraded":"2024-03-28T23:32:44.112770+0000","last_fullsized":"2024-03-28T23:32:44.112770+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.127627+0000","last_change":"2024-03-28T23:32:44.126818+0000","last_active":"2024-03-28T23:32:44.127627+0000","last_peered":"2024-03-28T23:32:44.127627+0000","last_clean":"2024-03-28T23:32:44.127627+0000","last_became_active":"2024-03-28T23:32:44.126493+0000","last_became_peered":"2024-03-28T23:32:44.126493+0000","last_unstale":"2024-03-28T23:32:44.127627+0000","last_undegraded":"2024-03-28T23:32:44.127627+0000","last_fullsized":"2024-03-28T23:32:44.127627+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:06:32.695104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.115074+0000","last_change":"2024-03-28T23:32:33.034126+0000","last_active":"2024-03-28T23:32:44.115074+0000","last_peered":"2024-03-28T23:32:44.115074+0000","last_clean":"2024-03-28T23:32:44.115074+0000","last_became_active":"2024-03-28T23:32:33.033703+0000","last_became_peered":"2024-03-28T23:32:33.033703+0000","last_unstale":"2024-03-28T23:32:44.115074+0000","last_undegraded":"2024-03-28T23:32:44.115074+0000","last_fullsized":"2024-03-28T23:32:44.115074+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:38.859031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121532+0000","last_change":"2024-03-28T23:32:32.039047+0000","last_active":"2024-03-28T23:32:44.121532+0000","last_peered":"2024-03-28T23:32:44.121532+0000","last_clean":"2024-03-28T23:32:44.121532+0000","last_became_active":"2024-03-28T23:32:32.038780+0000","last_became_peered":"2024-03-28T23:32:32.038780+0000","last_unstale":"2024-03-28T23:32:44.121532+0000","last_undegraded":"2024-03-28T23:32:44.121532+0000","last_fullsized":"2024-03-28T23:32:44.121532+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:13:15.856750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.117228+0000","last_change":"2024-03-28T23:32:44.117228+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.117228+0000","last_undegraded":"2024-03-28T23:32:44.117228+0000","last_fullsized":"2024-03-28T23:32:44.117228+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122300+0000","last_change":"2024-03-28T23:32:44.121416+0000","last_active":"2024-03-28T23:32:44.122300+0000","last_peered":"2024-03-28T23:32:44.122300+0000","last_clean":"2024-03-28T23:32:44.122300+0000","last_became_active":"2024-03-28T23:32:44.121127+0000","last_became_peered":"2024-03-28T23:32:44.121127+0000","last_unstale":"2024-03-28T23:32:44.122300+0000","last_undegraded":"2024-03-28T23:32:44.122300+0000","last_fullsized":"2024-03-28T23:32:44.122300+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:39:02.592632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113417+0000","last_change":"2024-03-28T23:32:33.051453+0000","last_active":"2024-03-28T23:32:44.113417+0000","last_peered":"2024-03-28T23:32:44.113417+0000","last_clean":"2024-03-28T23:32:44.113417+0000","last_became_active":"2024-03-28T23:32:33.051155+0000","last_became_peered":"2024-03-28T23:32:33.051155+0000","last_unstale":"2024-03-28T23:32:44.113417+0000","last_undegraded":"2024-03-28T23:32:44.113417+0000","last_fullsized":"2024-03-28T23:32:44.113417+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:36:03.951054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108887+0000","last_change":"2024-03-28T23:32:32.036979+0000","last_active":"2024-03-28T23:32:44.108887+0000","last_peered":"2024-03-28T23:32:44.108887+0000","last_clean":"2024-03-28T23:32:44.108887+0000","last_became_active":"2024-03-28T23:32:32.036710+0000","last_became_peered":"2024-03-28T23:32:32.036710+0000","last_unstale":"2024-03-28T23:32:44.108887+0000","last_undegraded":"2024-03-28T23:32:44.108887+0000","last_fullsized":"2024-03-28T23:32:44.108887+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:46.072852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107489+0000","last_change":"2024-03-28T23:32:44.107489+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107489+0000","last_undegraded":"2024-03-28T23:32:44.107489+0000","last_fullsized":"2024-03-28T23:32:44.107489+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122075+0000","last_change":"2024-03-28T23:32:44.115740+0000","last_active":"2024-03-28T23:32:44.122075+0000","last_peered":"2024-03-28T23:32:44.122075+0000","last_clean":"2024-03-28T23:32:44.122075+0000","last_became_active":"2024-03-28T23:32:44.115341+0000","last_became_peered":"2024-03-28T23:32:44.115341+0000","last_unstale":"2024-03-28T23:32:44.122075+0000","last_undegraded":"2024-03-28T23:32:44.122075+0000","last_fullsized":"2024-03-28T23:32:44.122075+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:55.159548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121499+0000","last_change":"2024-03-28T23:32:33.041923+0000","last_active":"2024-03-28T23:32:44.121499+0000","last_peered":"2024-03-28T23:32:44.121499+0000","last_clean":"2024-03-28T23:32:44.121499+0000","last_became_active":"2024-03-28T23:32:33.041631+0000","last_became_peered":"2024-03-28T23:32:33.041631+0000","last_unstale":"2024-03-28T23:32:44.121499+0000","last_undegraded":"2024-03-28T23:32:44.121499+0000","last_fullsized":"2024-03-28T23:32:44.121499+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:32:08.473276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108352+0000","last_change":"2024-03-28T23:32:32.041563+0000","last_active":"2024-03-28T23:32:44.108352+0000","last_peered":"2024-03-28T23:32:44.108352+0000","last_clean":"2024-03-28T23:32:44.108352+0000","last_became_active":"2024-03-28T23:32:32.040424+0000","last_became_peered":"2024-03-28T23:32:32.040424+0000","last_unstale":"2024-03-28T23:32:44.108352+0000","last_undegraded":"2024-03-28T23:32:44.108352+0000","last_fullsized":"2024-03-28T23:32:44.108352+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:37:25.645113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107400+0000","last_change":"2024-03-28T23:32:44.107400+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107400+0000","last_undegraded":"2024-03-28T23:32:44.107400+0000","last_fullsized":"2024-03-28T23:32:44.107400+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.123301+0000","last_change":"2024-03-28T23:32:44.118040+0000","last_active":"2024-03-28T23:32:44.123301+0000","last_peered":"2024-03-28T23:32:44.123301+0000","last_clean":"2024-03-28T23:32:44.123301+0000","last_became_active":"2024-03-28T23:32:44.117536+0000","last_became_peered":"2024-03-28T23:32:44.117536+0000","last_unstale":"2024-03-28T23:32:44.123301+0000","last_undegraded":"2024-03-28T23:32:44.123301+0000","last_fullsized":"2024-03-28T23:32:44.123301+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:59:07.631069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106030+0000","last_change":"2024-03-28T23:32:33.029986+0000","last_active":"2024-03-28T23:32:44.106030+0000","last_peered":"2024-03-28T23:32:44.106030+0000","last_clean":"2024-03-28T23:32:44.106030+0000","last_became_active":"2024-03-28T23:32:33.029413+0000","last_became_peered":"2024-03-28T23:32:33.029413+0000","last_unstale":"2024-03-28T23:32:44.106030+0000","last_undegraded":"2024-03-28T23:32:44.106030+0000","last_fullsized":"2024-03-28T23:32:44.106030+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:37:26.652511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107737+0000","last_change":"2024-03-28T23:32:32.037039+0000","last_active":"2024-03-28T23:32:44.107737+0000","last_peered":"2024-03-28T23:32:44.107737+0000","last_clean":"2024-03-28T23:32:44.107737+0000","last_became_active":"2024-03-28T23:32:32.036687+0000","last_became_peered":"2024-03-28T23:32:32.036687+0000","last_unstale":"2024-03-28T23:32:44.107737+0000","last_undegraded":"2024-03-28T23:32:44.107737+0000","last_fullsized":"2024-03-28T23:32:44.107737+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:27:15.824876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107961+0000","last_change":"2024-03-28T23:32:44.107961+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107961+0000","last_undegraded":"2024-03-28T23:32:44.107961+0000","last_fullsized":"2024-03-28T23:32:44.107961+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.127537+0000","last_change":"2024-03-28T23:32:44.126156+0000","last_active":"2024-03-28T23:32:44.127537+0000","last_peered":"2024-03-28T23:32:44.127537+0000","last_clean":"2024-03-28T23:32:44.127537+0000","last_became_active":"2024-03-28T23:32:44.125789+0000","last_became_peered":"2024-03-28T23:32:44.125789+0000","last_unstale":"2024-03-28T23:32:44.127537+0000","last_undegraded":"2024-03-28T23:32:44.127537+0000","last_fullsized":"2024-03-28T23:32:44.127537+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:41:04.213333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113597+0000","last_change":"2024-03-28T23:32:33.035809+0000","last_active":"2024-03-28T23:32:44.113597+0000","last_peered":"2024-03-28T23:32:44.113597+0000","last_clean":"2024-03-28T23:32:44.113597+0000","last_became_active":"2024-03-28T23:32:33.035563+0000","last_became_peered":"2024-03-28T23:32:33.035563+0000","last_unstale":"2024-03-28T23:32:44.113597+0000","last_undegraded":"2024-03-28T23:32:44.113597+0000","last_fullsized":"2024-03-28T23:32:44.113597+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:54.194453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"72'3","reported_seq":21,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.110292+0000","last_change":"2024-03-28T23:32:32.038230+0000","last_active":"2024-03-28T23:32:44.110292+0000","last_peered":"2024-03-28T23:32:44.110292+0000","last_clean":"2024-03-28T23:32:44.110292+0000","last_became_active":"2024-03-28T23:32:32.037681+0000","last_became_peered":"2024-03-28T23:32:32.037681+0000","last_unstale":"2024-03-28T23:32:44.110292+0000","last_undegraded":"2024-03-28T23:32:44.110292+0000","last_fullsized":"2024-03-28T23:32:44.110292+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:36:34.752866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112106+0000","last_change":"2024-03-28T23:32:44.112106+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112106+0000","last_undegraded":"2024-03-28T23:32:44.112106+0000","last_fullsized":"2024-03-28T23:32:44.112106+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.126080+0000","last_change":"2024-03-28T23:32:44.125087+0000","last_active":"2024-03-28T23:32:44.126080+0000","last_peered":"2024-03-28T23:32:44.126080+0000","last_clean":"2024-03-28T23:32:44.126080+0000","last_became_active":"2024-03-28T23:32:44.124784+0000","last_became_peered":"2024-03-28T23:32:44.124784+0000","last_unstale":"2024-03-28T23:32:44.126080+0000","last_undegraded":"2024-03-28T23:32:44.126080+0000","last_fullsized":"2024-03-28T23:32:44.126080+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:16:45.967951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112348+0000","last_change":"2024-03-28T23:32:33.049499+0000","last_active":"2024-03-28T23:32:44.112348+0000","last_peered":"2024-03-28T23:32:44.112348+0000","last_clean":"2024-03-28T23:32:44.112348+0000","last_became_active":"2024-03-28T23:32:33.049233+0000","last_became_peered":"2024-03-28T23:32:33.049233+0000","last_unstale":"2024-03-28T23:32:44.112348+0000","last_undegraded":"2024-03-28T23:32:44.112348+0000","last_fullsized":"2024-03-28T23:32:44.112348+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:39:02.824323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"72'6","reported_seq":24,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112883+0000","last_change":"2024-03-28T23:32:32.036835+0000","last_active":"2024-03-28T23:32:44.112883+0000","last_peered":"2024-03-28T23:32:44.112883+0000","last_clean":"2024-03-28T23:32:44.112883+0000","last_became_active":"2024-03-28T23:32:32.036557+0000","last_became_peered":"2024-03-28T23:32:32.036557+0000","last_unstale":"2024-03-28T23:32:44.112883+0000","last_undegraded":"2024-03-28T23:32:44.112883+0000","last_fullsized":"2024-03-28T23:32:44.112883+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:30.992807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111621+0000","last_change":"2024-03-28T23:32:44.111621+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111621+0000","last_undegraded":"2024-03-28T23:32:44.111621+0000","last_fullsized":"2024-03-28T23:32:44.111621+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.127678+0000","last_change":"2024-03-28T23:32:44.127256+0000","last_active":"2024-03-28T23:32:44.127678+0000","last_peered":"2024-03-28T23:32:44.127678+0000","last_clean":"2024-03-28T23:32:44.127678+0000","last_became_active":"2024-03-28T23:32:44.126894+0000","last_became_peered":"2024-03-28T23:32:44.126894+0000","last_unstale":"2024-03-28T23:32:44.127678+0000","last_undegraded":"2024-03-28T23:32:44.127678+0000","last_fullsized":"2024-03-28T23:32:44.127678+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:28:37.442827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107086+0000","last_change":"2024-03-28T23:32:33.033045+0000","last_active":"2024-03-28T23:32:44.107086+0000","last_peered":"2024-03-28T23:32:44.107086+0000","last_clean":"2024-03-28T23:32:44.107086+0000","last_became_active":"2024-03-28T23:32:33.032771+0000","last_became_peered":"2024-03-28T23:32:33.032771+0000","last_unstale":"2024-03-28T23:32:44.107086+0000","last_undegraded":"2024-03-28T23:32:44.107086+0000","last_fullsized":"2024-03-28T23:32:44.107086+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:43.341877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"72'2","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106619+0000","last_change":"2024-03-28T23:32:32.032053+0000","last_active":"2024-03-28T23:32:44.106619+0000","last_peered":"2024-03-28T23:32:44.106619+0000","last_clean":"2024-03-28T23:32:44.106619+0000","last_became_active":"2024-03-28T23:32:32.031741+0000","last_became_peered":"2024-03-28T23:32:32.031741+0000","last_unstale":"2024-03-28T23:32:44.106619+0000","last_undegraded":"2024-03-28T23:32:44.106619+0000","last_fullsized":"2024-03-28T23:32:44.106619+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:20.456721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108993+0000","last_change":"2024-03-28T23:32:44.108993+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108993+0000","last_undegraded":"2024-03-28T23:32:44.108993+0000","last_fullsized":"2024-03-28T23:32:44.108993+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.125906+0000","last_change":"2024-03-28T23:32:44.124910+0000","last_active":"2024-03-28T23:32:44.125906+0000","last_peered":"2024-03-28T23:32:44.125906+0000","last_clean":"2024-03-28T23:32:44.125906+0000","last_became_active":"2024-03-28T23:32:44.124639+0000","last_became_peered":"2024-03-28T23:32:44.124639+0000","last_unstale":"2024-03-28T23:32:44.125906+0000","last_undegraded":"2024-03-28T23:32:44.125906+0000","last_fullsized":"2024-03-28T23:32:44.125906+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:13:09.913978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112459+0000","last_change":"2024-03-28T23:32:33.039022+0000","last_active":"2024-03-28T23:32:44.112459+0000","last_peered":"2024-03-28T23:32:44.112459+0000","last_clean":"2024-03-28T23:32:44.112459+0000","last_became_active":"2024-03-28T23:32:33.038508+0000","last_became_peered":"2024-03-28T23:32:33.038508+0000","last_unstale":"2024-03-28T23:32:44.112459+0000","last_undegraded":"2024-03-28T23:32:44.112459+0000","last_fullsized":"2024-03-28T23:32:44.112459+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:25:45.189859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106348+0000","last_change":"2024-03-28T23:32:32.041042+0000","last_active":"2024-03-28T23:32:44.106348+0000","last_peered":"2024-03-28T23:32:44.106348+0000","last_clean":"2024-03-28T23:32:44.106348+0000","last_became_active":"2024-03-28T23:32:32.040686+0000","last_became_peered":"2024-03-28T23:32:32.040686+0000","last_unstale":"2024-03-28T23:32:44.106348+0000","last_undegraded":"2024-03-28T23:32:44.106348+0000","last_fullsized":"2024-03-28T23:32:44.106348+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:46.343619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.114552+0000","last_change":"2024-03-28T23:32:44.114552+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.114552+0000","last_undegraded":"2024-03-28T23:32:44.114552+0000","last_fullsized":"2024-03-28T23:32:44.114552+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121906+0000","last_change":"2024-03-28T23:32:44.120268+0000","last_active":"2024-03-28T23:32:44.121906+0000","last_peered":"2024-03-28T23:32:44.121906+0000","last_clean":"2024-03-28T23:32:44.121906+0000","last_became_active":"2024-03-28T23:32:44.119951+0000","last_became_peered":"2024-03-28T23:32:44.119951+0000","last_unstale":"2024-03-28T23:32:44.121906+0000","last_undegraded":"2024-03-28T23:32:44.121906+0000","last_fullsized":"2024-03-28T23:32:44.121906+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:28:47.614634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112006+0000","last_change":"2024-03-28T23:32:33.034262+0000","last_active":"2024-03-28T23:32:44.112006+0000","last_peered":"2024-03-28T23:32:44.112006+0000","last_clean":"2024-03-28T23:32:44.112006+0000","last_became_active":"2024-03-28T23:32:33.034003+0000","last_became_peered":"2024-03-28T23:32:33.034003+0000","last_unstale":"2024-03-28T23:32:44.112006+0000","last_undegraded":"2024-03-28T23:32:44.112006+0000","last_fullsized":"2024-03-28T23:32:44.112006+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:41.257241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108918+0000","last_change":"2024-03-28T23:32:32.039031+0000","last_active":"2024-03-28T23:32:44.108918+0000","last_peered":"2024-03-28T23:32:44.108918+0000","last_clean":"2024-03-28T23:32:44.108918+0000","last_became_active":"2024-03-28T23:32:32.038766+0000","last_became_peered":"2024-03-28T23:32:32.038766+0000","last_unstale":"2024-03-28T23:32:44.108918+0000","last_undegraded":"2024-03-28T23:32:44.108918+0000","last_fullsized":"2024-03-28T23:32:44.108918+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:13.872794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107400+0000","last_change":"2024-03-28T23:32:44.107400+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107400+0000","last_undegraded":"2024-03-28T23:32:44.107400+0000","last_fullsized":"2024-03-28T23:32:44.107400+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.123690+0000","last_change":"2024-03-28T23:32:44.119707+0000","last_active":"2024-03-28T23:32:44.123690+0000","last_peered":"2024-03-28T23:32:44.123690+0000","last_clean":"2024-03-28T23:32:44.123690+0000","last_became_active":"2024-03-28T23:32:44.119438+0000","last_became_peered":"2024-03-28T23:32:44.119438+0000","last_unstale":"2024-03-28T23:32:44.123690+0000","last_undegraded":"2024-03-28T23:32:44.123690+0000","last_fullsized":"2024-03-28T23:32:44.123690+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:39.542218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113512+0000","last_change":"2024-03-28T23:32:33.043171+0000","last_active":"2024-03-28T23:32:44.113512+0000","last_peered":"2024-03-28T23:32:44.113512+0000","last_clean":"2024-03-28T23:32:44.113512+0000","last_became_active":"2024-03-28T23:32:33.042821+0000","last_became_peered":"2024-03-28T23:32:33.042821+0000","last_unstale":"2024-03-28T23:32:44.113512+0000","last_undegraded":"2024-03-28T23:32:44.113512+0000","last_fullsized":"2024-03-28T23:32:44.113512+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:21.878156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"72'8","reported_seq":26,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121583+0000","last_change":"2024-03-28T23:32:32.040166+0000","last_active":"2024-03-28T23:32:44.121583+0000","last_peered":"2024-03-28T23:32:44.121583+0000","last_clean":"2024-03-28T23:32:44.121583+0000","last_became_active":"2024-03-28T23:32:32.039560+0000","last_became_peered":"2024-03-28T23:32:32.039560+0000","last_unstale":"2024-03-28T23:32:44.121583+0000","last_undegraded":"2024-03-28T23:32:44.121583+0000","last_fullsized":"2024-03-28T23:32:44.121583+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:55.544945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.110399+0000","last_change":"2024-03-28T23:32:44.110399+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.110399+0000","last_undegraded":"2024-03-28T23:32:44.110399+0000","last_fullsized":"2024-03-28T23:32:44.110399+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.127360+0000","last_change":"2024-03-28T23:32:44.126116+0000","last_active":"2024-03-28T23:32:44.127360+0000","last_peered":"2024-03-28T23:32:44.127360+0000","last_clean":"2024-03-28T23:32:44.127360+0000","last_became_active":"2024-03-28T23:32:44.125761+0000","last_became_peered":"2024-03-28T23:32:44.125761+0000","last_unstale":"2024-03-28T23:32:44.127360+0000","last_undegraded":"2024-03-28T23:32:44.127360+0000","last_fullsized":"2024-03-28T23:32:44.127360+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:04.591562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113351+0000","last_change":"2024-03-28T23:32:33.043192+0000","last_active":"2024-03-28T23:32:44.113351+0000","last_peered":"2024-03-28T23:32:44.113351+0000","last_clean":"2024-03-28T23:32:44.113351+0000","last_became_active":"2024-03-28T23:32:33.042894+0000","last_became_peered":"2024-03-28T23:32:33.042894+0000","last_unstale":"2024-03-28T23:32:44.113351+0000","last_undegraded":"2024-03-28T23:32:44.113351+0000","last_fullsized":"2024-03-28T23:32:44.113351+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:32:08.473276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"72'1","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108454+0000","last_change":"2024-03-28T23:32:32.041892+0000","last_active":"2024-03-28T23:32:44.108454+0000","last_peered":"2024-03-28T23:32:44.108454+0000","last_clean":"2024-03-28T23:32:44.108454+0000","last_became_active":"2024-03-28T23:32:32.041533+0000","last_became_peered":"2024-03-28T23:32:32.041533+0000","last_unstale":"2024-03-28T23:32:44.108454+0000","last_undegraded":"2024-03-28T23:32:44.108454+0000","last_fullsized":"2024-03-28T23:32:44.108454+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:15.843269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.116304+0000","last_change":"2024-03-28T23:32:44.116304+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.116304+0000","last_undegraded":"2024-03-28T23:32:44.116304+0000","last_fullsized":"2024-03-28T23:32:44.116304+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.118959+0000","last_change":"2024-03-28T23:32:44.115585+0000","last_active":"2024-03-28T23:32:44.118959+0000","last_peered":"2024-03-28T23:32:44.118959+0000","last_clean":"2024-03-28T23:32:44.118959+0000","last_became_active":"2024-03-28T23:32:44.115308+0000","last_became_peered":"2024-03-28T23:32:44.115308+0000","last_unstale":"2024-03-28T23:32:44.118959+0000","last_undegraded":"2024-03-28T23:32:44.118959+0000","last_fullsized":"2024-03-28T23:32:44.118959+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:59.122709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108799+0000","last_change":"2024-03-28T23:32:33.046899+0000","last_active":"2024-03-28T23:32:44.108799+0000","last_peered":"2024-03-28T23:32:44.108799+0000","last_clean":"2024-03-28T23:32:44.108799+0000","last_became_active":"2024-03-28T23:32:33.046579+0000","last_became_peered":"2024-03-28T23:32:33.046579+0000","last_unstale":"2024-03-28T23:32:44.108799+0000","last_undegraded":"2024-03-28T23:32:44.108799+0000","last_fullsized":"2024-03-28T23:32:44.108799+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:08:06.339997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"72'2","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.115368+0000","last_change":"2024-03-28T23:32:32.031135+0000","last_active":"2024-03-28T23:32:44.115368+0000","last_peered":"2024-03-28T23:32:44.115368+0000","last_clean":"2024-03-28T23:32:44.115368+0000","last_became_active":"2024-03-28T23:32:32.030796+0000","last_became_peered":"2024-03-28T23:32:32.030796+0000","last_unstale":"2024-03-28T23:32:44.115368+0000","last_undegraded":"2024-03-28T23:32:44.115368+0000","last_fullsized":"2024-03-28T23:32:44.115368+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:18:21.816565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.116620+0000","last_change":"2024-03-28T23:32:44.116620+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.116620+0000","last_undegraded":"2024-03-28T23:32:44.116620+0000","last_fullsized":"2024-03-28T23:32:44.116620+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.117708+0000","last_change":"2024-03-28T23:32:44.116752+0000","last_active":"2024-03-28T23:32:44.117708+0000","last_peered":"2024-03-28T23:32:44.117708+0000","last_clean":"2024-03-28T23:32:44.117708+0000","last_became_active":"2024-03-28T23:32:44.116381+0000","last_became_peered":"2024-03-28T23:32:44.116381+0000","last_unstale":"2024-03-28T23:32:44.117708+0000","last_undegraded":"2024-03-28T23:32:44.117708+0000","last_fullsized":"2024-03-28T23:32:44.117708+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:17:56.623866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109016+0000","last_change":"2024-03-28T23:32:33.053549+0000","last_active":"2024-03-28T23:32:44.109016+0000","last_peered":"2024-03-28T23:32:44.109016+0000","last_clean":"2024-03-28T23:32:44.109016+0000","last_became_active":"2024-03-28T23:32:33.053229+0000","last_became_peered":"2024-03-28T23:32:33.053229+0000","last_unstale":"2024-03-28T23:32:44.109016+0000","last_undegraded":"2024-03-28T23:32:44.109016+0000","last_fullsized":"2024-03-28T23:32:44.109016+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:41:02.517121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"72'1","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.110391+0000","last_change":"2024-03-28T23:32:32.038261+0000","last_active":"2024-03-28T23:32:44.110391+0000","last_peered":"2024-03-28T23:32:44.110391+0000","last_clean":"2024-03-28T23:32:44.110391+0000","last_became_active":"2024-03-28T23:32:32.037971+0000","last_became_peered":"2024-03-28T23:32:32.037971+0000","last_unstale":"2024-03-28T23:32:44.110391+0000","last_undegraded":"2024-03-28T23:32:44.110391+0000","last_fullsized":"2024-03-28T23:32:44.110391+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:25:10.296139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.109933+0000","last_change":"2024-03-28T23:32:44.109933+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.109933+0000","last_undegraded":"2024-03-28T23:32:44.109933+0000","last_fullsized":"2024-03-28T23:32:44.109933+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.118230+0000","last_change":"2024-03-28T23:32:44.115062+0000","last_active":"2024-03-28T23:32:44.118230+0000","last_peered":"2024-03-28T23:32:44.118230+0000","last_clean":"2024-03-28T23:32:44.118230+0000","last_became_active":"2024-03-28T23:32:44.114795+0000","last_became_peered":"2024-03-28T23:32:44.114795+0000","last_unstale":"2024-03-28T23:32:44.118230+0000","last_undegraded":"2024-03-28T23:32:44.118230+0000","last_fullsized":"2024-03-28T23:32:44.118230+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:38.460546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113026+0000","last_change":"2024-03-28T23:32:33.043954+0000","last_active":"2024-03-28T23:32:44.113026+0000","last_peered":"2024-03-28T23:32:44.113026+0000","last_clean":"2024-03-28T23:32:44.113026+0000","last_became_active":"2024-03-28T23:32:33.043649+0000","last_became_peered":"2024-03-28T23:32:33.043649+0000","last_unstale":"2024-03-28T23:32:44.113026+0000","last_undegraded":"2024-03-28T23:32:44.113026+0000","last_fullsized":"2024-03-28T23:32:44.113026+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:00.358180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109556+0000","last_change":"2024-03-28T23:32:32.033720+0000","last_active":"2024-03-28T23:32:44.109556+0000","last_peered":"2024-03-28T23:32:44.109556+0000","last_clean":"2024-03-28T23:32:44.109556+0000","last_became_active":"2024-03-28T23:32:32.033422+0000","last_became_peered":"2024-03-28T23:32:32.033422+0000","last_unstale":"2024-03-28T23:32:44.109556+0000","last_undegraded":"2024-03-28T23:32:44.109556+0000","last_fullsized":"2024-03-28T23:32:44.109556+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:51:10.440109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.113185+0000","last_change":"2024-03-28T23:32:44.113185+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.113185+0000","last_undegraded":"2024-03-28T23:32:44.113185+0000","last_fullsized":"2024-03-28T23:32:44.113185+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.124927+0000","last_change":"2024-03-28T23:32:44.124559+0000","last_active":"2024-03-28T23:32:44.124927+0000","last_peered":"2024-03-28T23:32:44.124927+0000","last_clean":"2024-03-28T23:32:44.124927+0000","last_became_active":"2024-03-28T23:32:44.124282+0000","last_became_peered":"2024-03-28T23:32:44.124282+0000","last_unstale":"2024-03-28T23:32:44.124927+0000","last_undegraded":"2024-03-28T23:32:44.124927+0000","last_fullsized":"2024-03-28T23:32:44.124927+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:32.542845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122528+0000","last_change":"2024-03-28T23:32:33.045618+0000","last_active":"2024-03-28T23:32:44.122528+0000","last_peered":"2024-03-28T23:32:44.122528+0000","last_clean":"2024-03-28T23:32:44.122528+0000","last_became_active":"2024-03-28T23:32:33.045324+0000","last_became_peered":"2024-03-28T23:32:33.045324+0000","last_unstale":"2024-03-28T23:32:44.122528+0000","last_undegraded":"2024-03-28T23:32:44.122528+0000","last_fullsized":"2024-03-28T23:32:44.122528+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:46:43.644886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"72'2","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113118+0000","last_change":"2024-03-28T23:32:32.031574+0000","last_active":"2024-03-28T23:32:44.113118+0000","last_peered":"2024-03-28T23:32:44.113118+0000","last_clean":"2024-03-28T23:32:44.113118+0000","last_became_active":"2024-03-28T23:32:32.031310+0000","last_became_peered":"2024-03-28T23:32:32.031310+0000","last_unstale":"2024-03-28T23:32:44.113118+0000","last_undegraded":"2024-03-28T23:32:44.113118+0000","last_fullsized":"2024-03-28T23:32:44.113118+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:48:07.073481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112326+0000","last_change":"2024-03-28T23:32:44.112326+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112326+0000","last_undegraded":"2024-03-28T23:32:44.112326+0000","last_fullsized":"2024-03-28T23:32:44.112326+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.115233+0000","last_change":"2024-03-28T23:32:44.114144+0000","last_active":"2024-03-28T23:32:44.115233+0000","last_peered":"2024-03-28T23:32:44.115233+0000","last_clean":"2024-03-28T23:32:44.115233+0000","last_became_active":"2024-03-28T23:32:44.113829+0000","last_became_peered":"2024-03-28T23:32:44.113829+0000","last_unstale":"2024-03-28T23:32:44.115233+0000","last_undegraded":"2024-03-28T23:32:44.115233+0000","last_fullsized":"2024-03-28T23:32:44.115233+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:55.159548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.123006+0000","last_change":"2024-03-28T23:32:33.040983+0000","last_active":"2024-03-28T23:32:44.123006+0000","last_peered":"2024-03-28T23:32:44.123006+0000","last_clean":"2024-03-28T23:32:44.123006+0000","last_became_active":"2024-03-28T23:32:33.040693+0000","last_became_peered":"2024-03-28T23:32:33.040693+0000","last_unstale":"2024-03-28T23:32:44.123006+0000","last_undegraded":"2024-03-28T23:32:44.123006+0000","last_fullsized":"2024-03-28T23:32:44.123006+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:22:05.201764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113759+0000","last_change":"2024-03-28T23:32:32.036998+0000","last_active":"2024-03-28T23:32:44.113759+0000","last_peered":"2024-03-28T23:32:44.113759+0000","last_clean":"2024-03-28T23:32:44.113759+0000","last_became_active":"2024-03-28T23:32:32.036733+0000","last_became_peered":"2024-03-28T23:32:32.036733+0000","last_unstale":"2024-03-28T23:32:44.113759+0000","last_undegraded":"2024-03-28T23:32:44.113759+0000","last_fullsized":"2024-03-28T23:32:44.113759+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:42.334479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.105630+0000","last_change":"2024-03-28T23:32:44.105630+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.105630+0000","last_undegraded":"2024-03-28T23:32:44.105630+0000","last_fullsized":"2024-03-28T23:32:44.105630+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.115890+0000","last_change":"2024-03-28T23:32:44.115154+0000","last_active":"2024-03-28T23:32:44.115890+0000","last_peered":"2024-03-28T23:32:44.115890+0000","last_clean":"2024-03-28T23:32:44.115890+0000","last_became_active":"2024-03-28T23:32:44.114869+0000","last_became_peered":"2024-03-28T23:32:44.114869+0000","last_unstale":"2024-03-28T23:32:44.115890+0000","last_undegraded":"2024-03-28T23:32:44.115890+0000","last_fullsized":"2024-03-28T23:32:44.115890+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:58:59.309973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113614+0000","last_change":"2024-03-28T23:32:33.041232+0000","last_active":"2024-03-28T23:32:44.113614+0000","last_peered":"2024-03-28T23:32:44.113614+0000","last_clean":"2024-03-28T23:32:44.113614+0000","last_became_active":"2024-03-28T23:32:33.040132+0000","last_became_peered":"2024-03-28T23:32:33.040132+0000","last_unstale":"2024-03-28T23:32:44.113614+0000","last_undegraded":"2024-03-28T23:32:44.113614+0000","last_fullsized":"2024-03-28T23:32:44.113614+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:39:02.824323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113450+0000","last_change":"2024-03-28T23:32:32.036416+0000","last_active":"2024-03-28T23:32:44.113450+0000","last_peered":"2024-03-28T23:32:44.113450+0000","last_clean":"2024-03-28T23:32:44.113450+0000","last_became_active":"2024-03-28T23:32:32.036131+0000","last_became_peered":"2024-03-28T23:32:32.036131+0000","last_unstale":"2024-03-28T23:32:44.113450+0000","last_undegraded":"2024-03-28T23:32:44.113450+0000","last_fullsized":"2024-03-28T23:32:44.113450+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:13:33.914342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.109522+0000","last_change":"2024-03-28T23:32:44.109522+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.109522+0000","last_undegraded":"2024-03-28T23:32:44.109522+0000","last_fullsized":"2024-03-28T23:32:44.109522+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.119994+0000","last_change":"2024-03-28T23:32:44.116820+0000","last_active":"2024-03-28T23:32:44.119994+0000","last_peered":"2024-03-28T23:32:44.119994+0000","last_clean":"2024-03-28T23:32:44.119994+0000","last_became_active":"2024-03-28T23:32:44.116540+0000","last_became_peered":"2024-03-28T23:32:44.116540+0000","last_unstale":"2024-03-28T23:32:44.119994+0000","last_undegraded":"2024-03-28T23:32:44.119994+0000","last_fullsized":"2024-03-28T23:32:44.119994+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:58.429743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.110759+0000","last_change":"2024-03-28T23:32:33.035015+0000","last_active":"2024-03-28T23:32:44.110759+0000","last_peered":"2024-03-28T23:32:44.110759+0000","last_clean":"2024-03-28T23:32:44.110759+0000","last_became_active":"2024-03-28T23:32:33.034730+0000","last_became_peered":"2024-03-28T23:32:33.034730+0000","last_unstale":"2024-03-28T23:32:44.110759+0000","last_undegraded":"2024-03-28T23:32:44.110759+0000","last_fullsized":"2024-03-28T23:32:44.110759+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:09.477928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"72'2","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108295+0000","last_change":"2024-03-28T23:32:32.032596+0000","last_active":"2024-03-28T23:32:44.108295+0000","last_peered":"2024-03-28T23:32:44.108295+0000","last_clean":"2024-03-28T23:32:44.108295+0000","last_became_active":"2024-03-28T23:32:32.032336+0000","last_became_peered":"2024-03-28T23:32:32.032336+0000","last_unstale":"2024-03-28T23:32:44.108295+0000","last_undegraded":"2024-03-28T23:32:44.108295+0000","last_fullsized":"2024-03-28T23:32:44.108295+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:28:25.356704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.120871+0000","last_change":"2024-03-28T23:32:44.120871+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.120871+0000","last_undegraded":"2024-03-28T23:32:44.120871+0000","last_fullsized":"2024-03-28T23:32:44.120871+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.117955+0000","last_change":"2024-03-28T23:32:44.116649+0000","last_active":"2024-03-28T23:32:44.117955+0000","last_peered":"2024-03-28T23:32:44.117955+0000","last_clean":"2024-03-28T23:32:44.117955+0000","last_became_active":"2024-03-28T23:32:44.116369+0000","last_became_peered":"2024-03-28T23:32:44.116369+0000","last_unstale":"2024-03-28T23:32:44.117955+0000","last_undegraded":"2024-03-28T23:32:44.117955+0000","last_fullsized":"2024-03-28T23:32:44.117955+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:46:54.723611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.111340+0000","last_change":"2024-03-28T23:32:33.044243+0000","last_active":"2024-03-28T23:32:44.111340+0000","last_peered":"2024-03-28T23:32:44.111340+0000","last_clean":"2024-03-28T23:32:44.111340+0000","last_became_active":"2024-03-28T23:32:33.040218+0000","last_became_peered":"2024-03-28T23:32:33.040218+0000","last_unstale":"2024-03-28T23:32:44.111340+0000","last_undegraded":"2024-03-28T23:32:44.111340+0000","last_fullsized":"2024-03-28T23:32:44.111340+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:38:51.513907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"73'1","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.123099+0000","last_change":"2024-03-28T23:32:32.041586+0000","last_active":"2024-03-28T23:32:44.123099+0000","last_peered":"2024-03-28T23:32:44.123099+0000","last_clean":"2024-03-28T23:32:44.123099+0000","last_became_active":"2024-03-28T23:32:32.041327+0000","last_became_peered":"2024-03-28T23:32:32.041327+0000","last_unstale":"2024-03-28T23:32:44.123099+0000","last_undegraded":"2024-03-28T23:32:44.123099+0000","last_fullsized":"2024-03-28T23:32:44.123099+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:06:20.608980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107219+0000","last_change":"2024-03-28T23:32:44.107219+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107219+0000","last_undegraded":"2024-03-28T23:32:44.107219+0000","last_fullsized":"2024-03-28T23:32:44.107219+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.126567+0000","last_change":"2024-03-28T23:32:44.125883+0000","last_active":"2024-03-28T23:32:44.126567+0000","last_peered":"2024-03-28T23:32:44.126567+0000","last_clean":"2024-03-28T23:32:44.126567+0000","last_became_active":"2024-03-28T23:32:44.125578+0000","last_became_peered":"2024-03-28T23:32:44.125578+0000","last_unstale":"2024-03-28T23:32:44.126567+0000","last_undegraded":"2024-03-28T23:32:44.126567+0000","last_fullsized":"2024-03-28T23:32:44.126567+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:25.238412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112455+0000","last_change":"2024-03-28T23:32:33.042272+0000","last_active":"2024-03-28T23:32:44.112455+0000","last_peered":"2024-03-28T23:32:44.112455+0000","last_clean":"2024-03-28T23:32:44.112455+0000","last_became_active":"2024-03-28T23:32:33.037633+0000","last_became_peered":"2024-03-28T23:32:33.037633+0000","last_unstale":"2024-03-28T23:32:44.112455+0000","last_undegraded":"2024-03-28T23:32:44.112455+0000","last_fullsized":"2024-03-28T23:32:44.112455+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:45.773838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108173+0000","last_change":"2024-03-28T23:32:32.037001+0000","last_active":"2024-03-28T23:32:44.108173+0000","last_peered":"2024-03-28T23:32:44.108173+0000","last_clean":"2024-03-28T23:32:44.108173+0000","last_became_active":"2024-03-28T23:32:32.036739+0000","last_became_peered":"2024-03-28T23:32:32.036739+0000","last_unstale":"2024-03-28T23:32:44.108173+0000","last_undegraded":"2024-03-28T23:32:44.108173+0000","last_fullsized":"2024-03-28T23:32:44.108173+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:06:20.608980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112232+0000","last_change":"2024-03-28T23:32:44.112232+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112232+0000","last_undegraded":"2024-03-28T23:32:44.112232+0000","last_fullsized":"2024-03-28T23:32:44.112232+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.117952+0000","last_change":"2024-03-28T23:32:44.116971+0000","last_active":"2024-03-28T23:32:44.117952+0000","last_peered":"2024-03-28T23:32:44.117952+0000","last_clean":"2024-03-28T23:32:44.117952+0000","last_became_active":"2024-03-28T23:32:44.116371+0000","last_became_peered":"2024-03-28T23:32:44.116371+0000","last_unstale":"2024-03-28T23:32:44.117952+0000","last_undegraded":"2024-03-28T23:32:44.117952+0000","last_fullsized":"2024-03-28T23:32:44.117952+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:43:47.025717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122394+0000","last_change":"2024-03-28T23:32:33.041123+0000","last_active":"2024-03-28T23:32:44.122394+0000","last_peered":"2024-03-28T23:32:44.122394+0000","last_clean":"2024-03-28T23:32:44.122394+0000","last_became_active":"2024-03-28T23:32:33.040505+0000","last_became_peered":"2024-03-28T23:32:33.040505+0000","last_unstale":"2024-03-28T23:32:44.122394+0000","last_undegraded":"2024-03-28T23:32:44.122394+0000","last_fullsized":"2024-03-28T23:32:44.122394+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:54.194453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106365+0000","last_change":"2024-03-28T23:32:32.036857+0000","last_active":"2024-03-28T23:32:44.106365+0000","last_peered":"2024-03-28T23:32:44.106365+0000","last_clean":"2024-03-28T23:32:44.106365+0000","last_became_active":"2024-03-28T23:32:32.036576+0000","last_became_peered":"2024-03-28T23:32:32.036576+0000","last_unstale":"2024-03-28T23:32:44.106365+0000","last_undegraded":"2024-03-28T23:32:44.106365+0000","last_fullsized":"2024-03-28T23:32:44.106365+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:43.073425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.120260+0000","last_change":"2024-03-28T23:32:44.120260+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.120260+0000","last_undegraded":"2024-03-28T23:32:44.120260+0000","last_fullsized":"2024-03-28T23:32:44.120260+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.127553+0000","last_change":"2024-03-28T23:32:44.126706+0000","last_active":"2024-03-28T23:32:44.127553+0000","last_peered":"2024-03-28T23:32:44.127553+0000","last_clean":"2024-03-28T23:32:44.127553+0000","last_became_active":"2024-03-28T23:32:44.126400+0000","last_became_peered":"2024-03-28T23:32:44.126400+0000","last_unstale":"2024-03-28T23:32:44.127553+0000","last_undegraded":"2024-03-28T23:32:44.127553+0000","last_fullsized":"2024-03-28T23:32:44.127553+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:00.790475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112670+0000","last_change":"2024-03-28T23:32:33.041769+0000","last_active":"2024-03-28T23:32:44.112670+0000","last_peered":"2024-03-28T23:32:44.112670+0000","last_clean":"2024-03-28T23:32:44.112670+0000","last_became_active":"2024-03-28T23:32:33.041475+0000","last_became_peered":"2024-03-28T23:32:33.041475+0000","last_unstale":"2024-03-28T23:32:44.112670+0000","last_undegraded":"2024-03-28T23:32:44.112670+0000","last_fullsized":"2024-03-28T23:32:44.112670+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:38.859031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112270+0000","last_change":"2024-03-28T23:32:32.037824+0000","last_active":"2024-03-28T23:32:44.112270+0000","last_peered":"2024-03-28T23:32:44.112270+0000","last_clean":"2024-03-28T23:32:44.112270+0000","last_became_active":"2024-03-28T23:32:32.037550+0000","last_became_peered":"2024-03-28T23:32:32.037550+0000","last_unstale":"2024-03-28T23:32:44.112270+0000","last_undegraded":"2024-03-28T23:32:44.112270+0000","last_fullsized":"2024-03-28T23:32:44.112270+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:43.467214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111817+0000","last_change":"2024-03-28T23:32:44.111817+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111817+0000","last_undegraded":"2024-03-28T23:32:44.111817+0000","last_fullsized":"2024-03-28T23:32:44.111817+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.119913+0000","last_change":"2024-03-28T23:32:44.117095+0000","last_active":"2024-03-28T23:32:44.119913+0000","last_peered":"2024-03-28T23:32:44.119913+0000","last_clean":"2024-03-28T23:32:44.119913+0000","last_became_active":"2024-03-28T23:32:44.116824+0000","last_became_peered":"2024-03-28T23:32:44.116824+0000","last_unstale":"2024-03-28T23:32:44.119913+0000","last_undegraded":"2024-03-28T23:32:44.119913+0000","last_fullsized":"2024-03-28T23:32:44.119913+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:59:07.631069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112978+0000","last_change":"2024-03-28T23:32:33.048966+0000","last_active":"2024-03-28T23:32:44.112978+0000","last_peered":"2024-03-28T23:32:44.112978+0000","last_clean":"2024-03-28T23:32:44.112978+0000","last_became_active":"2024-03-28T23:32:33.048362+0000","last_became_peered":"2024-03-28T23:32:33.048362+0000","last_unstale":"2024-03-28T23:32:44.112978+0000","last_undegraded":"2024-03-28T23:32:44.112978+0000","last_fullsized":"2024-03-28T23:32:44.112978+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:25.849919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109099+0000","last_change":"2024-03-28T23:32:32.038885+0000","last_active":"2024-03-28T23:32:44.109099+0000","last_peered":"2024-03-28T23:32:44.109099+0000","last_clean":"2024-03-28T23:32:44.109099+0000","last_became_active":"2024-03-28T23:32:32.038364+0000","last_became_peered":"2024-03-28T23:32:32.038364+0000","last_unstale":"2024-03-28T23:32:44.109099+0000","last_undegraded":"2024-03-28T23:32:44.109099+0000","last_fullsized":"2024-03-28T23:32:44.109099+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:16:06.156877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.105630+0000","last_change":"2024-03-28T23:32:44.105630+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.105630+0000","last_undegraded":"2024-03-28T23:32:44.105630+0000","last_fullsized":"2024-03-28T23:32:44.105630+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.115613+0000","last_change":"2024-03-28T23:32:44.114483+0000","last_active":"2024-03-28T23:32:44.115613+0000","last_peered":"2024-03-28T23:32:44.115613+0000","last_clean":"2024-03-28T23:32:44.115613+0000","last_became_active":"2024-03-28T23:32:44.114228+0000","last_became_peered":"2024-03-28T23:32:44.114228+0000","last_unstale":"2024-03-28T23:32:44.115613+0000","last_undegraded":"2024-03-28T23:32:44.115613+0000","last_fullsized":"2024-03-28T23:32:44.115613+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:56:05.273178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107950+0000","last_change":"2024-03-28T23:32:33.049584+0000","last_active":"2024-03-28T23:32:44.107950+0000","last_peered":"2024-03-28T23:32:44.107950+0000","last_clean":"2024-03-28T23:32:44.107950+0000","last_became_active":"2024-03-28T23:32:33.049342+0000","last_became_peered":"2024-03-28T23:32:33.049342+0000","last_unstale":"2024-03-28T23:32:44.107950+0000","last_undegraded":"2024-03-28T23:32:44.107950+0000","last_fullsized":"2024-03-28T23:32:44.107950+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:49.711750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.111513+0000","last_change":"2024-03-28T23:32:32.034044+0000","last_active":"2024-03-28T23:32:44.111513+0000","last_peered":"2024-03-28T23:32:44.111513+0000","last_clean":"2024-03-28T23:32:44.111513+0000","last_became_active":"2024-03-28T23:32:32.033779+0000","last_became_peered":"2024-03-28T23:32:32.033779+0000","last_unstale":"2024-03-28T23:32:44.111513+0000","last_undegraded":"2024-03-28T23:32:44.111513+0000","last_fullsized":"2024-03-28T23:32:44.111513+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:56:22.159708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.116670+0000","last_change":"2024-03-28T23:32:44.116670+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.116670+0000","last_undegraded":"2024-03-28T23:32:44.116670+0000","last_fullsized":"2024-03-28T23:32:44.116670+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.120835+0000","last_change":"2024-03-28T23:32:44.119649+0000","last_active":"2024-03-28T23:32:44.120835+0000","last_peered":"2024-03-28T23:32:44.120835+0000","last_clean":"2024-03-28T23:32:44.120835+0000","last_became_active":"2024-03-28T23:32:44.119370+0000","last_became_peered":"2024-03-28T23:32:44.119370+0000","last_unstale":"2024-03-28T23:32:44.120835+0000","last_undegraded":"2024-03-28T23:32:44.120835+0000","last_fullsized":"2024-03-28T23:32:44.120835+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:25:56.268584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109652+0000","last_change":"2024-03-28T23:32:33.041180+0000","last_active":"2024-03-28T23:32:44.109652+0000","last_peered":"2024-03-28T23:32:44.109652+0000","last_clean":"2024-03-28T23:32:44.109652+0000","last_became_active":"2024-03-28T23:32:33.040922+0000","last_became_peered":"2024-03-28T23:32:33.040922+0000","last_unstale":"2024-03-28T23:32:44.109652+0000","last_undegraded":"2024-03-28T23:32:44.109652+0000","last_fullsized":"2024-03-28T23:32:44.109652+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:25:45.189859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109480+0000","last_change":"2024-03-28T23:32:32.034328+0000","last_active":"2024-03-28T23:32:44.109480+0000","last_peered":"2024-03-28T23:32:44.109480+0000","last_clean":"2024-03-28T23:32:44.109480+0000","last_became_active":"2024-03-28T23:32:32.034057+0000","last_became_peered":"2024-03-28T23:32:32.034057+0000","last_unstale":"2024-03-28T23:32:44.109480+0000","last_undegraded":"2024-03-28T23:32:44.109480+0000","last_fullsized":"2024-03-28T23:32:44.109480+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:54:12.775407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107591+0000","last_change":"2024-03-28T23:32:44.107591+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107591+0000","last_undegraded":"2024-03-28T23:32:44.107591+0000","last_fullsized":"2024-03-28T23:32:44.107591+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.119109+0000","last_change":"2024-03-28T23:32:44.115020+0000","last_active":"2024-03-28T23:32:44.119109+0000","last_peered":"2024-03-28T23:32:44.119109+0000","last_clean":"2024-03-28T23:32:44.119109+0000","last_became_active":"2024-03-28T23:32:44.114737+0000","last_became_peered":"2024-03-28T23:32:44.114737+0000","last_unstale":"2024-03-28T23:32:44.119109+0000","last_undegraded":"2024-03-28T23:32:44.119109+0000","last_fullsized":"2024-03-28T23:32:44.119109+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:13:16.820792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108150+0000","last_change":"2024-03-28T23:32:33.038830+0000","last_active":"2024-03-28T23:32:44.108150+0000","last_peered":"2024-03-28T23:32:44.108150+0000","last_clean":"2024-03-28T23:32:44.108150+0000","last_became_active":"2024-03-28T23:32:33.038524+0000","last_became_peered":"2024-03-28T23:32:33.038524+0000","last_unstale":"2024-03-28T23:32:44.108150+0000","last_undegraded":"2024-03-28T23:32:44.108150+0000","last_fullsized":"2024-03-28T23:32:44.108150+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:55:53.512836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113822+0000","last_change":"2024-03-28T23:32:32.034270+0000","last_active":"2024-03-28T23:32:44.113822+0000","last_peered":"2024-03-28T23:32:44.113822+0000","last_clean":"2024-03-28T23:32:44.113822+0000","last_became_active":"2024-03-28T23:32:32.034022+0000","last_became_peered":"2024-03-28T23:32:32.034022+0000","last_unstale":"2024-03-28T23:32:44.113822+0000","last_undegraded":"2024-03-28T23:32:44.113822+0000","last_fullsized":"2024-03-28T23:32:44.113822+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:15.683500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":273,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109089+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:32:44.109089+0000","last_peered":"2024-03-28T23:32:44.109089+0000","last_clean":"2024-03-28T23:32:44.109089+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:32:44.109089+0000","last_undegraded":"2024-03-28T23:32:44.109089+0000","last_fullsized":"2024-03-28T23:32:44.109089+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574887,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":974848,"data_stored":1018333,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574888,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32088,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339752,"statfs":{"total":96636764160,"available":96603906048,"internally_reserved":0,"allocated":929792,"data_stored":437077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32068,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339772,"statfs":{"total":96636764160,"available":96603926528,"internally_reserved":0,"allocated":909312,"data_stored":427977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574887,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32084,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339756,"statfs":{"total":96636764160,"available":96603910144,"internally_reserved":0,"allocated":925696,"data_stored":429659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574887,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27908,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343932,"statfs":{"total":96636764160,"available":96608186368,"internally_reserved":0,"allocated":909312,"data_stored":428525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574887,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343924,"statfs":{"total":96636764160,"available":96608178176,"internally_reserved":0,"allocated":917504,"data_stored":428685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574887,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343928,"statfs":{"total":96636764160,"available":96608182272,"internally_reserved":0,"allocated":913408,"data_stored":428055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574887,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":970752,"data_stored":1018311,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:32:48.965 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3d is not active+clean 2024-03-28T23:32:48.966 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.110397+0000', 'last_change': '2024-03-28T23:32:44.110397+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.110397+0000', 'last_undegraded': '2024-03-28T23:32:44.110397+0000', 'last_fullsized': '2024-03-28T23:32:44.110397+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-28T23:32:48.966 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3c is not active+clean 2024-03-28T23:32:48.966 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.109309+0000', 'last_change': '2024-03-28T23:32:44.109309+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.109309+0000', 'last_undegraded': '2024-03-28T23:32:44.109309+0000', 'last_fullsized': '2024-03-28T23:32:44.109309+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-28T23:32:48.966 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3f is not active+clean 2024-03-28T23:32:48.966 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.111805+0000', 'last_change': '2024-03-28T23:32:44.111805+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.111805+0000', 'last_undegraded': '2024-03-28T23:32:44.111805+0000', 'last_fullsized': '2024-03-28T23:32:44.111805+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-28T23:32:48.966 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3e is not active+clean 2024-03-28T23:32:48.966 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.111326+0000', 'last_change': '2024-03-28T23:32:44.111326+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.111326+0000', 'last_undegraded': '2024-03-28T23:32:44.111326+0000', 'last_fullsized': '2024-03-28T23:32:44.111326+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-28T23:32:48.966 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.39 is not active+clean 2024-03-28T23:32:48.966 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.106821+0000', 'last_change': '2024-03-28T23:32:44.106821+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.106821+0000', 'last_undegraded': '2024-03-28T23:32:44.106821+0000', 'last_fullsized': '2024-03-28T23:32:44.106821+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-28T23:32:48.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.38 is not active+clean 2024-03-28T23:32:48.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.109201+0000', 'last_change': '2024-03-28T23:32:44.109201+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.109201+0000', 'last_undegraded': '2024-03-28T23:32:44.109201+0000', 'last_fullsized': '2024-03-28T23:32:44.109201+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-28T23:32:48.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3b is not active+clean 2024-03-28T23:32:48.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.111404+0000', 'last_change': '2024-03-28T23:32:44.111404+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.111404+0000', 'last_undegraded': '2024-03-28T23:32:44.111404+0000', 'last_fullsized': '2024-03-28T23:32:44.111404+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:32:48.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3a is not active+clean 2024-03-28T23:32:48.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.113215+0000', 'last_change': '2024-03-28T23:32:44.113215+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.113215+0000', 'last_undegraded': '2024-03-28T23:32:44.113215+0000', 'last_fullsized': '2024-03-28T23:32:44.113215+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:32:48.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.35 is not active+clean 2024-03-28T23:32:48.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.117756+0000', 'last_change': '2024-03-28T23:32:44.117756+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.117756+0000', 'last_undegraded': '2024-03-28T23:32:44.117756+0000', 'last_fullsized': '2024-03-28T23:32:44.117756+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:32:48.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.34 is not active+clean 2024-03-28T23:32:48.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.107890+0000', 'last_change': '2024-03-28T23:32:44.107890+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.107890+0000', 'last_undegraded': '2024-03-28T23:32:44.107890+0000', 'last_fullsized': '2024-03-28T23:32:44.107890+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-28T23:32:48.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.37 is not active+clean 2024-03-28T23:32:48.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.111858+0000', 'last_change': '2024-03-28T23:32:44.111858+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.111858+0000', 'last_undegraded': '2024-03-28T23:32:44.111858+0000', 'last_fullsized': '2024-03-28T23:32:44.111858+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:32:48.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.36 is not active+clean 2024-03-28T23:32:48.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.108222+0000', 'last_change': '2024-03-28T23:32:44.108222+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.108222+0000', 'last_undegraded': '2024-03-28T23:32:44.108222+0000', 'last_fullsized': '2024-03-28T23:32:44.108222+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-28T23:32:48.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.31 is not active+clean 2024-03-28T23:32:48.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.108039+0000', 'last_change': '2024-03-28T23:32:44.108039+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.108039+0000', 'last_undegraded': '2024-03-28T23:32:44.108039+0000', 'last_fullsized': '2024-03-28T23:32:44.108039+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-28T23:32:48.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.30 is not active+clean 2024-03-28T23:32:48.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.113196+0000', 'last_change': '2024-03-28T23:32:44.113196+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.113196+0000', 'last_undegraded': '2024-03-28T23:32:44.113196+0000', 'last_fullsized': '2024-03-28T23:32:44.113196+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:32:48.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.33 is not active+clean 2024-03-28T23:32:48.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.111833+0000', 'last_change': '2024-03-28T23:32:44.111833+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.111833+0000', 'last_undegraded': '2024-03-28T23:32:44.111833+0000', 'last_fullsized': '2024-03-28T23:32:44.111833+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-28T23:32:48.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.32 is not active+clean 2024-03-28T23:32:48.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.112037+0000', 'last_change': '2024-03-28T23:32:44.112037+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.112037+0000', 'last_undegraded': '2024-03-28T23:32:44.112037+0000', 'last_fullsized': '2024-03-28T23:32:44.112037+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:32:48.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2d is not active+clean 2024-03-28T23:32:48.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.115970+0000', 'last_change': '2024-03-28T23:32:44.115970+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.115970+0000', 'last_undegraded': '2024-03-28T23:32:44.115970+0000', 'last_fullsized': '2024-03-28T23:32:44.115970+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:32:48.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2c is not active+clean 2024-03-28T23:32:48.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.108940+0000', 'last_change': '2024-03-28T23:32:44.108940+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.108940+0000', 'last_undegraded': '2024-03-28T23:32:44.108940+0000', 'last_fullsized': '2024-03-28T23:32:44.108940+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-28T23:32:48.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2f is not active+clean 2024-03-28T23:32:48.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.120323+0000', 'last_change': '2024-03-28T23:32:44.120323+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.120323+0000', 'last_undegraded': '2024-03-28T23:32:44.120323+0000', 'last_fullsized': '2024-03-28T23:32:44.120323+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:32:48.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2e is not active+clean 2024-03-28T23:32:48.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.112002+0000', 'last_change': '2024-03-28T23:32:44.112002+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.112002+0000', 'last_undegraded': '2024-03-28T23:32:44.112002+0000', 'last_fullsized': '2024-03-28T23:32:44.112002+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-28T23:32:48.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.29 is not active+clean 2024-03-28T23:32:48.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.107873+0000', 'last_change': '2024-03-28T23:32:44.107873+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.107873+0000', 'last_undegraded': '2024-03-28T23:32:44.107873+0000', 'last_fullsized': '2024-03-28T23:32:44.107873+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:32:48.970 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.28 is not active+clean 2024-03-28T23:32:48.970 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.111443+0000', 'last_change': '2024-03-28T23:32:44.111443+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.111443+0000', 'last_undegraded': '2024-03-28T23:32:44.111443+0000', 'last_fullsized': '2024-03-28T23:32:44.111443+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-28T23:32:48.970 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2b is not active+clean 2024-03-28T23:32:48.970 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.110595+0000', 'last_change': '2024-03-28T23:32:44.110595+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.110595+0000', 'last_undegraded': '2024-03-28T23:32:44.110595+0000', 'last_fullsized': '2024-03-28T23:32:44.110595+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:32:48.970 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2a is not active+clean 2024-03-28T23:32:48.970 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.110858+0000', 'last_change': '2024-03-28T23:32:44.110858+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.110858+0000', 'last_undegraded': '2024-03-28T23:32:44.110858+0000', 'last_fullsized': '2024-03-28T23:32:44.110858+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:32:48.970 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.25 is not active+clean 2024-03-28T23:32:48.970 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.107392+0000', 'last_change': '2024-03-28T23:32:44.107392+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.107392+0000', 'last_undegraded': '2024-03-28T23:32:44.107392+0000', 'last_fullsized': '2024-03-28T23:32:44.107392+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-28T23:32:48.970 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.24 is not active+clean 2024-03-28T23:32:48.970 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.108552+0000', 'last_change': '2024-03-28T23:32:44.108552+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.108552+0000', 'last_undegraded': '2024-03-28T23:32:44.108552+0000', 'last_fullsized': '2024-03-28T23:32:44.108552+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-28T23:32:48.971 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.27 is not active+clean 2024-03-28T23:32:48.971 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.112271+0000', 'last_change': '2024-03-28T23:32:44.112271+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.112271+0000', 'last_undegraded': '2024-03-28T23:32:44.112271+0000', 'last_fullsized': '2024-03-28T23:32:44.112271+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-28T23:32:48.971 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.26 is not active+clean 2024-03-28T23:32:48.971 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.114282+0000', 'last_change': '2024-03-28T23:32:44.114282+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.114282+0000', 'last_undegraded': '2024-03-28T23:32:44.114282+0000', 'last_fullsized': '2024-03-28T23:32:44.114282+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:32:48.971 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.21 is not active+clean 2024-03-28T23:32:48.971 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.110873+0000', 'last_change': '2024-03-28T23:32:44.110873+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.110873+0000', 'last_undegraded': '2024-03-28T23:32:44.110873+0000', 'last_fullsized': '2024-03-28T23:32:44.110873+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-28T23:32:48.971 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.20 is not active+clean 2024-03-28T23:32:48.971 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.117069+0000', 'last_change': '2024-03-28T23:32:44.117069+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.117069+0000', 'last_undegraded': '2024-03-28T23:32:44.117069+0000', 'last_fullsized': '2024-03-28T23:32:44.117069+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:32:48.971 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.23 is not active+clean 2024-03-28T23:32:48.971 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.107516+0000', 'last_change': '2024-03-28T23:32:44.107516+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.107516+0000', 'last_undegraded': '2024-03-28T23:32:44.107516+0000', 'last_fullsized': '2024-03-28T23:32:44.107516+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-28T23:32:48.971 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.22 is not active+clean 2024-03-28T23:32:48.972 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.111342+0000', 'last_change': '2024-03-28T23:32:44.111342+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.111342+0000', 'last_undegraded': '2024-03-28T23:32:44.111342+0000', 'last_fullsized': '2024-03-28T23:32:44.111342+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:32:48.972 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1a is not active+clean 2024-03-28T23:32:48.972 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.112540+0000', 'last_change': '2024-03-28T23:32:44.112540+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.112540+0000', 'last_undegraded': '2024-03-28T23:32:44.112540+0000', 'last_fullsized': '2024-03-28T23:32:44.112540+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:32:48.972 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1b is not active+clean 2024-03-28T23:32:48.972 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.108504+0000', 'last_change': '2024-03-28T23:32:44.108504+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.108504+0000', 'last_undegraded': '2024-03-28T23:32:44.108504+0000', 'last_fullsized': '2024-03-28T23:32:44.108504+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-28T23:32:48.972 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.18 is not active+clean 2024-03-28T23:32:48.972 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.108669+0000', 'last_change': '2024-03-28T23:32:44.108669+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.108669+0000', 'last_undegraded': '2024-03-28T23:32:44.108669+0000', 'last_fullsized': '2024-03-28T23:32:44.108669+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-28T23:32:48.972 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.19 is not active+clean 2024-03-28T23:32:48.972 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.107395+0000', 'last_change': '2024-03-28T23:32:44.107395+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.107395+0000', 'last_undegraded': '2024-03-28T23:32:44.107395+0000', 'last_fullsized': '2024-03-28T23:32:44.107395+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:32:48.972 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1e is not active+clean 2024-03-28T23:32:48.972 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.106357+0000', 'last_change': '2024-03-28T23:32:44.106357+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.106357+0000', 'last_undegraded': '2024-03-28T23:32:44.106357+0000', 'last_fullsized': '2024-03-28T23:32:44.106357+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-28T23:32:48.973 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1f is not active+clean 2024-03-28T23:32:48.973 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.116621+0000', 'last_change': '2024-03-28T23:32:44.116621+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.116621+0000', 'last_undegraded': '2024-03-28T23:32:44.116621+0000', 'last_fullsized': '2024-03-28T23:32:44.116621+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:32:48.973 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1c is not active+clean 2024-03-28T23:32:48.973 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.112770+0000', 'last_change': '2024-03-28T23:32:44.112770+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.112770+0000', 'last_undegraded': '2024-03-28T23:32:44.112770+0000', 'last_fullsized': '2024-03-28T23:32:44.112770+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:32:48.973 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1d is not active+clean 2024-03-28T23:32:48.973 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.117228+0000', 'last_change': '2024-03-28T23:32:44.117228+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.117228+0000', 'last_undegraded': '2024-03-28T23:32:44.117228+0000', 'last_fullsized': '2024-03-28T23:32:44.117228+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:32:48.973 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.12 is not active+clean 2024-03-28T23:32:48.973 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.107489+0000', 'last_change': '2024-03-28T23:32:44.107489+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.107489+0000', 'last_undegraded': '2024-03-28T23:32:44.107489+0000', 'last_fullsized': '2024-03-28T23:32:44.107489+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-28T23:32:48.973 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.13 is not active+clean 2024-03-28T23:32:48.973 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.107400+0000', 'last_change': '2024-03-28T23:32:44.107400+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.107400+0000', 'last_undegraded': '2024-03-28T23:32:44.107400+0000', 'last_fullsized': '2024-03-28T23:32:44.107400+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-28T23:32:48.974 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.10 is not active+clean 2024-03-28T23:32:48.974 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.107961+0000', 'last_change': '2024-03-28T23:32:44.107961+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.107961+0000', 'last_undegraded': '2024-03-28T23:32:44.107961+0000', 'last_fullsized': '2024-03-28T23:32:44.107961+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-28T23:32:48.974 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.11 is not active+clean 2024-03-28T23:32:48.974 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.112106+0000', 'last_change': '2024-03-28T23:32:44.112106+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.112106+0000', 'last_undegraded': '2024-03-28T23:32:44.112106+0000', 'last_fullsized': '2024-03-28T23:32:44.112106+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:32:48.974 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.16 is not active+clean 2024-03-28T23:32:48.974 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.111621+0000', 'last_change': '2024-03-28T23:32:44.111621+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.111621+0000', 'last_undegraded': '2024-03-28T23:32:44.111621+0000', 'last_fullsized': '2024-03-28T23:32:44.111621+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-28T23:32:48.974 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.17 is not active+clean 2024-03-28T23:32:48.974 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.108993+0000', 'last_change': '2024-03-28T23:32:44.108993+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.108993+0000', 'last_undegraded': '2024-03-28T23:32:44.108993+0000', 'last_fullsized': '2024-03-28T23:32:44.108993+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-28T23:32:48.974 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.14 is not active+clean 2024-03-28T23:32:48.974 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.114552+0000', 'last_change': '2024-03-28T23:32:44.114552+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.114552+0000', 'last_undegraded': '2024-03-28T23:32:44.114552+0000', 'last_fullsized': '2024-03-28T23:32:44.114552+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:32:48.974 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.15 is not active+clean 2024-03-28T23:32:48.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.107400+0000', 'last_change': '2024-03-28T23:32:44.107400+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.107400+0000', 'last_undegraded': '2024-03-28T23:32:44.107400+0000', 'last_fullsized': '2024-03-28T23:32:44.107400+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-28T23:32:48.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.a is not active+clean 2024-03-28T23:32:48.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.110399+0000', 'last_change': '2024-03-28T23:32:44.110399+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.110399+0000', 'last_undegraded': '2024-03-28T23:32:44.110399+0000', 'last_fullsized': '2024-03-28T23:32:44.110399+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-28T23:32:48.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.b is not active+clean 2024-03-28T23:32:48.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.116304+0000', 'last_change': '2024-03-28T23:32:44.116304+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.116304+0000', 'last_undegraded': '2024-03-28T23:32:44.116304+0000', 'last_fullsized': '2024-03-28T23:32:44.116304+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:32:48.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.8 is not active+clean 2024-03-28T23:32:48.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.116620+0000', 'last_change': '2024-03-28T23:32:44.116620+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.116620+0000', 'last_undegraded': '2024-03-28T23:32:44.116620+0000', 'last_fullsized': '2024-03-28T23:32:44.116620+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:32:48.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.9 is not active+clean 2024-03-28T23:32:48.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.109933+0000', 'last_change': '2024-03-28T23:32:44.109933+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.109933+0000', 'last_undegraded': '2024-03-28T23:32:44.109933+0000', 'last_fullsized': '2024-03-28T23:32:44.109933+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-28T23:32:48.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.e is not active+clean 2024-03-28T23:32:48.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.113185+0000', 'last_change': '2024-03-28T23:32:44.113185+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.113185+0000', 'last_undegraded': '2024-03-28T23:32:44.113185+0000', 'last_fullsized': '2024-03-28T23:32:44.113185+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:32:48.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.c is not active+clean 2024-03-28T23:32:48.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.112326+0000', 'last_change': '2024-03-28T23:32:44.112326+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.112326+0000', 'last_undegraded': '2024-03-28T23:32:44.112326+0000', 'last_fullsized': '2024-03-28T23:32:44.112326+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:32:48.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.d is not active+clean 2024-03-28T23:32:48.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.105630+0000', 'last_change': '2024-03-28T23:32:44.105630+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.105630+0000', 'last_undegraded': '2024-03-28T23:32:44.105630+0000', 'last_fullsized': '2024-03-28T23:32:44.105630+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:32:48.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2 is not active+clean 2024-03-28T23:32:48.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.109522+0000', 'last_change': '2024-03-28T23:32:44.109522+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.109522+0000', 'last_undegraded': '2024-03-28T23:32:44.109522+0000', 'last_fullsized': '2024-03-28T23:32:44.109522+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:32:48.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3 is not active+clean 2024-03-28T23:32:48.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.120871+0000', 'last_change': '2024-03-28T23:32:44.120871+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.120871+0000', 'last_undegraded': '2024-03-28T23:32:44.120871+0000', 'last_fullsized': '2024-03-28T23:32:44.120871+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:32:48.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.0 is not active+clean 2024-03-28T23:32:48.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.107219+0000', 'last_change': '2024-03-28T23:32:44.107219+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.107219+0000', 'last_undegraded': '2024-03-28T23:32:44.107219+0000', 'last_fullsized': '2024-03-28T23:32:44.107219+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-28T23:32:48.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1 is not active+clean 2024-03-28T23:32:48.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.112232+0000', 'last_change': '2024-03-28T23:32:44.112232+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.112232+0000', 'last_undegraded': '2024-03-28T23:32:44.112232+0000', 'last_fullsized': '2024-03-28T23:32:44.112232+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-28T23:32:48.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.6 is not active+clean 2024-03-28T23:32:48.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.120260+0000', 'last_change': '2024-03-28T23:32:44.120260+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.120260+0000', 'last_undegraded': '2024-03-28T23:32:44.120260+0000', 'last_fullsized': '2024-03-28T23:32:44.120260+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:32:48.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.7 is not active+clean 2024-03-28T23:32:48.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.111817+0000', 'last_change': '2024-03-28T23:32:44.111817+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.111817+0000', 'last_undegraded': '2024-03-28T23:32:44.111817+0000', 'last_fullsized': '2024-03-28T23:32:44.111817+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-28T23:32:48.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.4 is not active+clean 2024-03-28T23:32:48.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.105630+0000', 'last_change': '2024-03-28T23:32:44.105630+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.105630+0000', 'last_undegraded': '2024-03-28T23:32:44.105630+0000', 'last_fullsized': '2024-03-28T23:32:44.105630+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:32:48.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.5 is not active+clean 2024-03-28T23:32:48.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.116670+0000', 'last_change': '2024-03-28T23:32:44.116670+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.116670+0000', 'last_undegraded': '2024-03-28T23:32:44.116670+0000', 'last_fullsized': '2024-03-28T23:32:44.116670+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:32:48.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.f is not active+clean 2024-03-28T23:32:48.978 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 75, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:32:44.107591+0000', 'last_change': '2024-03-28T23:32:44.107591+0000', 'last_active': '2024-03-28T23:32:44.098119+0000', 'last_peered': '2024-03-28T23:32:44.098119+0000', 'last_clean': '2024-03-28T23:32:44.098119+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:32:44.107591+0000', 'last_undegraded': '2024-03-28T23:32:44.107591+0000', 'last_fullsized': '2024-03-28T23:32:44.107591+0000', 'mapping_epoch': 75, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 75, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'last_clean_scrub_stamp': '2024-03-28T23:32:44.098119+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-28T23:32:48.978 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-28T23:32:49.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.107+0000 7fbf82a54640 1 Processor -- start 2024-03-28T23:32:49.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.107+0000 7fbf82a54640 1 -- start start 2024-03-28T23:32:49.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.108+0000 7fbf82a54640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c0caf60 0x7fbf7c1592e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:49.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.108+0000 7fbf82a54640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf7c0cb4e0 con 0x7fbf7c14f260 2024-03-28T23:32:49.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.108+0000 7fbf82a54640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf7c0cb7c0 con 0x7fbf7c0c8970 2024-03-28T23:32:49.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.108+0000 7fbf82a54640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbf7c158e50 con 0x7fbf7c0caf60 2024-03-28T23:32:49.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.108+0000 7fbf7bfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fbf7c14f260 0x7fbf7c155c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59380/0 (socket says 172.21.15.161:59380) 2024-03-28T23:32:49.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.108+0000 7fbf7bfff640 1 -- 172.21.15.161:0/2819551170 learned_addr learned my addr 172.21.15.161:0/2819551170 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:49.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.108+0000 7fbf7b7fe640 1 --2- 172.21.15.161:0/2819551170 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c0caf60 0x7fbf7c1592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:49.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.109+0000 7fbf7affd640 1 -- 172.21.15.161:0/2819551170 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4023975077 0 0) 0x7fbf7c0cb4e0 con 0x7fbf7c14f260 2024-03-28T23:32:49.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.109+0000 7fbf7affd640 1 -- 172.21.15.161:0/2819551170 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf64003520 con 0x7fbf7c14f260 2024-03-28T23:32:49.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.109+0000 7fbf7affd640 1 -- 172.21.15.161:0/2819551170 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3353108293 0 0) 0x7fbf7c0cb7c0 con 0x7fbf7c0c8970 2024-03-28T23:32:49.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.109+0000 7fbf7affd640 1 -- 172.21.15.161:0/2819551170 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf7c0cb4e0 con 0x7fbf7c0c8970 2024-03-28T23:32:49.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.109+0000 7fbf7b7fe640 1 -- 172.21.15.161:0/2819551170 >> v1:172.21.15.170:6789/0 conn(0x7fbf7c0c8970 legacy=0x7fbf7c0c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:49.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.109+0000 7fbf7b7fe640 1 -- 172.21.15.161:0/2819551170 >> v1:172.21.15.161:6789/0 conn(0x7fbf7c14f260 legacy=0x7fbf7c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:49.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.109+0000 7fbf7b7fe640 1 -- 172.21.15.161:0/2819551170 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf7c158800 con 0x7fbf7c0caf60 2024-03-28T23:32:49.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.110+0000 7fbf7b7fe640 1 --2- 172.21.15.161:0/2819551170 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c0caf60 0x7fbf7c1592e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c0097d0 tx=0x7fbf6c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=256b746ccda3d68b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:49.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.110+0000 7fbf7affd640 1 -- 172.21.15.161:0/2819551170 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c035020 con 0x7fbf7c0caf60 2024-03-28T23:32:49.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.110+0000 7fbf7affd640 1 -- 172.21.15.161:0/2819551170 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf6c031800 con 0x7fbf7c0caf60 2024-03-28T23:32:49.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.110+0000 7fbf7affd640 1 -- 172.21.15.161:0/2819551170 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c031bb0 con 0x7fbf7c0caf60 2024-03-28T23:32:49.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.111+0000 7fbf82a54640 1 -- 172.21.15.161:0/2819551170 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c0caf60 msgr2=0x7fbf7c1592e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:49.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.111+0000 7fbf82a54640 1 --2- 172.21.15.161:0/2819551170 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c0caf60 0x7fbf7c1592e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c0097d0 tx=0x7fbf6c02b8a0 comp rx=0 tx=0).stop 2024-03-28T23:32:49.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.111+0000 7fbf82a54640 1 -- 172.21.15.161:0/2819551170 shutdown_connections 2024-03-28T23:32:49.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.111+0000 7fbf82a54640 1 --2- 172.21.15.161:0/2819551170 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c0caf60 0x7fbf7c1592e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:49.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.111+0000 7fbf82a54640 1 -- 172.21.15.161:0/2819551170 >> 172.21.15.161:0/2819551170 conn(0x7fbf7c072ca0 msgr2=0x7fbf7c0730b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:49.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.111+0000 7fbf82a54640 1 -- 172.21.15.161:0/2819551170 shutdown_connections 2024-03-28T23:32:49.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.111+0000 7fbf82a54640 1 -- 172.21.15.161:0/2819551170 wait complete. 2024-03-28T23:32:49.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.112+0000 7fbf82a54640 1 Processor -- start 2024-03-28T23:32:49.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.112+0000 7fbf82a54640 1 -- start start 2024-03-28T23:32:49.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.113+0000 7fbf82a54640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf7c0c8970 0x7fbf7c0bf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:49.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.113+0000 7fbf82a54640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c0caf60 0x7fbf7c1ebd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:49.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.113+0000 7fbf82a54640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c14f260 0x7fbf7c1ee0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:49.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.113+0000 7fbf82a54640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbf7c0c5b00 con 0x7fbf7c0caf60 2024-03-28T23:32:49.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.113+0000 7fbf82a54640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbf7c14f640 con 0x7fbf7c0c8970 2024-03-28T23:32:49.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.113+0000 7fbf82a54640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbf7c0c5c80 con 0x7fbf7c14f260 2024-03-28T23:32:49.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.113+0000 7fbf7b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c0caf60 0x7fbf7c1ebd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:49.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.113+0000 7fbf7b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c0caf60 0x7fbf7c1ebd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39956/0 (socket says 172.21.15.161:39956) 2024-03-28T23:32:49.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.113+0000 7fbf7b7fe640 1 -- 172.21.15.161:0/2643147818 learned_addr learned my addr 172.21.15.161:0/2643147818 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:49.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.113+0000 7fbf80ff1640 1 --2- 172.21.15.161:0/2643147818 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c14f260 0x7fbf7c1ee0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:49.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.113+0000 7fbf7bfff640 1 --2- 172.21.15.161:0/2643147818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf7c0c8970 0x7fbf7c0bf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:49.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.113+0000 7fbf7b7fe640 1 -- 172.21.15.161:0/2643147818 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c14f260 msgr2=0x7fbf7c1ee0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:49.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.113+0000 7fbf7b7fe640 1 --2- 172.21.15.161:0/2643147818 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c14f260 0x7fbf7c1ee0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:49.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.113+0000 7fbf7b7fe640 1 -- 172.21.15.161:0/2643147818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf7c0c8970 msgr2=0x7fbf7c0bf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:49.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.113+0000 7fbf7b7fe640 1 --2- 172.21.15.161:0/2643147818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf7c0c8970 0x7fbf7c0bf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:49.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.113+0000 7fbf7b7fe640 1 -- 172.21.15.161:0/2643147818 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf7c0c0e10 con 0x7fbf7c0caf60 2024-03-28T23:32:49.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.113+0000 7fbf7b7fe640 1 --2- 172.21.15.161:0/2643147818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c0caf60 0x7fbf7c1ebd70 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c0098f0 tx=0x7fbf6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:49.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.113+0000 7fbf78ff9640 1 -- 172.21.15.161:0/2643147818 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c035040 con 0x7fbf7c0caf60 2024-03-28T23:32:49.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.113+0000 7fbf78ff9640 1 -- 172.21.15.161:0/2643147818 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf6c03ad00 con 0x7fbf7c0caf60 2024-03-28T23:32:49.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.113+0000 7fbf78ff9640 1 -- 172.21.15.161:0/2643147818 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c0466c0 con 0x7fbf7c0caf60 2024-03-28T23:32:49.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.113+0000 7fbf82a54640 1 -- 172.21.15.161:0/2643147818 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf7c0c10f0 con 0x7fbf7c0caf60 2024-03-28T23:32:49.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.114+0000 7fbf82a54640 1 -- 172.21.15.161:0/2643147818 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf7c154b20 con 0x7fbf7c0caf60 2024-03-28T23:32:49.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.115+0000 7fbf82a54640 1 -- 172.21.15.161:0/2643147818 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf7c04f550 con 0x7fbf7c0caf60 2024-03-28T23:32:49.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.115+0000 7fbf78ff9640 1 -- 172.21.15.161:0/2643147818 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbf6c038020 con 0x7fbf7c0caf60 2024-03-28T23:32:49.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.115+0000 7fbf78ff9640 1 --2- 172.21.15.161:0/2643147818 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf6806b500 0x7fbf6806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:49.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.116+0000 7fbf78ff9640 1 -- 172.21.15.161:0/2643147818 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6957+0+0 (secure 0 0 0) 0x7fbf6c0ac650 con 0x7fbf7c0caf60 2024-03-28T23:32:49.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.116+0000 7fbf7bfff640 1 --2- 172.21.15.161:0/2643147818 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf6806b500 0x7fbf6806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:49.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.116+0000 7fbf7bfff640 1 --2- 172.21.15.161:0/2643147818 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf6806b500 0x7fbf6806d9c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fbf70000f60 tx=0x7fbf70007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:49.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.120+0000 7fbf78ff9640 1 -- 172.21.15.161:0/2643147818 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbf6c04b060 con 0x7fbf7c0caf60 2024-03-28T23:32:49.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.370+0000 7fbf82a54640 1 -- 172.21.15.161:0/2643147818 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7fbf7c06cfe0 con 0x7fbf7c0caf60 2024-03-28T23:32:49.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.371+0000 7fbf78ff9640 1 -- 172.21.15.161:0/2643147818 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1557 (secure 0 0 0) 0x7fbf6c07aa30 con 0x7fbf7c0caf60 2024-03-28T23:32:49.371 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:49.372 INFO:teuthology.orchestra.run.smithi161.stdout:{"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":200,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":77,"num_osds":8,"num_up_osds":8,"osd_up_since":1711668579,"num_in_osds":8,"osd_in_since":1711668571,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":129},{"state_name":"creating+peering","count":64}],"num_pgs":193,"num_pools":5,"num_objects":24,"data_bytes":592778,"bytes_used":245837824,"bytes_avail":772848275456,"bytes_total":773094113280,"inactive_pgs_ratio":0.33160620927810669},"fsmap":{"epoch":60,"id":5,"up":1,"in":1,"max":1,"id":6,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":6,"rank":0,"name":"b","status":"up:active","gid":6037},{"filesystem_id":5,"rank":0,"name":"a","status":"up:active","gid":6468}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-03-28T23:32:21.341327+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-28T23:32:49.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.373+0000 7fbf82a54640 1 -- 172.21.15.161:0/2643147818 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf6806b500 msgr2=0x7fbf6806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:49.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.373+0000 7fbf82a54640 1 --2- 172.21.15.161:0/2643147818 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf6806b500 0x7fbf6806d9c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fbf70000f60 tx=0x7fbf70007d30 comp rx=0 tx=0).stop 2024-03-28T23:32:49.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.373+0000 7fbf82a54640 1 -- 172.21.15.161:0/2643147818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c0caf60 msgr2=0x7fbf7c1ebd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:49.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.373+0000 7fbf82a54640 1 --2- 172.21.15.161:0/2643147818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c0caf60 0x7fbf7c1ebd70 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c0098f0 tx=0x7fbf6c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:32:49.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.373+0000 7fbf82a54640 1 -- 172.21.15.161:0/2643147818 shutdown_connections 2024-03-28T23:32:49.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.373+0000 7fbf82a54640 1 --2- 172.21.15.161:0/2643147818 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf6806b500 0x7fbf6806d9c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:49.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.373+0000 7fbf82a54640 1 --2- 172.21.15.161:0/2643147818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c0caf60 0x7fbf7c1ebd70 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:49.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.373+0000 7fbf82a54640 1 --2- 172.21.15.161:0/2643147818 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c14f260 0x7fbf7c1ee0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:49.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.373+0000 7fbf82a54640 1 --2- 172.21.15.161:0/2643147818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf7c0c8970 0x7fbf7c0bf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:49.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.373+0000 7fbf82a54640 1 -- 172.21.15.161:0/2643147818 >> 172.21.15.161:0/2643147818 conn(0x7fbf7c072ca0 msgr2=0x7fbf7c153b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:49.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.373+0000 7fbf82a54640 1 -- 172.21.15.161:0/2643147818 shutdown_connections 2024-03-28T23:32:49.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.373+0000 7fbf82a54640 1 -- 172.21.15.161:0/2643147818 wait complete. 2024-03-28T23:32:49.384 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-28T23:32:49.384 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:32:49.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.486+0000 7fb5b341c640 1 Processor -- start 2024-03-28T23:32:49.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.487+0000 7fb5b341c640 1 -- start start 2024-03-28T23:32:49.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.487+0000 7fb5b341c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb5ac152890 0x7fb5ac0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:49.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.487+0000 7fb5b341c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5ac0c8ed0 0x7fb5ac0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:49.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.487+0000 7fb5b341c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb5ac0c5950 0x7fb5ac1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:49.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.487+0000 7fb5b341c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb5ac0c8480 con 0x7fb5ac152890 2024-03-28T23:32:49.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.487+0000 7fb5b341c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fb5ac159640 con 0x7fb5ac0c5950 2024-03-28T23:32:49.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.487+0000 7fb5b341c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb5ac158a30 con 0x7fb5ac0c8ed0 2024-03-28T23:32:49.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.487+0000 7fb5b11b8640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb5ac152890 0x7fb5ac0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:49.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.487+0000 7fb5b11b8640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb5ac152890 0x7fb5ac0c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39970/0 (socket says 172.21.15.161:39970) 2024-03-28T23:32:49.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.487+0000 7fb5b11b8640 1 -- 172.21.15.161:0/3256914631 learned_addr learned my addr 172.21.15.161:0/3256914631 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:49.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.487+0000 7fb5b19b9640 1 --2- 172.21.15.161:0/3256914631 >> v2:172.21.15.170:3300/0 conn(0x7fb5ac0c5950 0x7fb5ac1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:49.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.488+0000 7fb5b09b7640 1 --2- 172.21.15.161:0/3256914631 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5ac0c8ed0 0x7fb5ac0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:49.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.488+0000 7fb5b11b8640 1 -- 172.21.15.161:0/3256914631 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5ac0c8ed0 msgr2=0x7fb5ac0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:49.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.488+0000 7fb5b11b8640 1 --2- 172.21.15.161:0/3256914631 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5ac0c8ed0 0x7fb5ac0c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:49.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.488+0000 7fb5b11b8640 1 -- 172.21.15.161:0/3256914631 >> v2:172.21.15.170:3300/0 conn(0x7fb5ac0c5950 msgr2=0x7fb5ac1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:49.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.488+0000 7fb5b11b8640 1 --2- 172.21.15.161:0/3256914631 >> v2:172.21.15.170:3300/0 conn(0x7fb5ac0c5950 0x7fb5ac1553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:49.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.488+0000 7fb5b11b8640 1 -- 172.21.15.161:0/3256914631 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5ac158d10 con 0x7fb5ac152890 2024-03-28T23:32:49.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.488+0000 7fb5b11b8640 1 --2- 172.21.15.161:0/3256914631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5ac152890 0x7fb5ac0c8990 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac0b0c00 tx=0x7fb59c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac8192362b73b8b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:49.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.489+0000 7fb5abfff640 1 -- 172.21.15.161:0/3256914631 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59c038070 con 0x7fb5ac152890 2024-03-28T23:32:49.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.489+0000 7fb5abfff640 1 -- 172.21.15.161:0/3256914631 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb59c02f050 con 0x7fb5ac152890 2024-03-28T23:32:49.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.489+0000 7fb5abfff640 1 -- 172.21.15.161:0/3256914631 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59c031970 con 0x7fb5ac152890 2024-03-28T23:32:49.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.489+0000 7fb5b341c640 1 -- 172.21.15.161:0/3256914631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5ac152890 msgr2=0x7fb5ac0c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:49.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.489+0000 7fb5b341c640 1 --2- 172.21.15.161:0/3256914631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5ac152890 0x7fb5ac0c8990 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac0b0c00 tx=0x7fb59c02b7f0 comp rx=0 tx=0).stop 2024-03-28T23:32:49.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.489+0000 7fb5b341c640 1 -- 172.21.15.161:0/3256914631 shutdown_connections 2024-03-28T23:32:49.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.489+0000 7fb5b341c640 1 --2- 172.21.15.161:0/3256914631 >> v2:172.21.15.170:3300/0 conn(0x7fb5ac0c5950 0x7fb5ac1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:49.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.489+0000 7fb5b341c640 1 --2- 172.21.15.161:0/3256914631 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5ac0c8ed0 0x7fb5ac0c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:49.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.489+0000 7fb5b341c640 1 --2- 172.21.15.161:0/3256914631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5ac152890 0x7fb5ac0c8990 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:49.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.489+0000 7fb5b341c640 1 -- 172.21.15.161:0/3256914631 >> 172.21.15.161:0/3256914631 conn(0x7fb5ac0ac620 msgr2=0x7fb5ac0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:49.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.489+0000 7fb5b341c640 1 -- 172.21.15.161:0/3256914631 shutdown_connections 2024-03-28T23:32:49.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.489+0000 7fb5b341c640 1 -- 172.21.15.161:0/3256914631 wait complete. 2024-03-28T23:32:49.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.490+0000 7fb5b341c640 1 Processor -- start 2024-03-28T23:32:49.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.490+0000 7fb5b341c640 1 -- start start 2024-03-28T23:32:49.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.491+0000 7fb5b341c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5ac0c5950 0x7fb5ac0c49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:49.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.491+0000 7fb5b341c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5ac0c8ed0 0x7fb5ac0ab900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:49.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.491+0000 7fb5b341c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5ac152890 0x7fb5ac1e7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:49.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.491+0000 7fb5b341c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac152c70 con 0x7fb5ac152890 2024-03-28T23:32:49.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.491+0000 7fb5b341c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac1592d0 con 0x7fb5ac0c5950 2024-03-28T23:32:49.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.491+0000 7fb5b341c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb5ac152df0 con 0x7fb5ac0c8ed0 2024-03-28T23:32:49.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.491+0000 7fb5b19b9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5ac152890 0x7fb5ac1e7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:49.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.491+0000 7fb5b19b9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5ac152890 0x7fb5ac1e7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39986/0 (socket says 172.21.15.161:39986) 2024-03-28T23:32:49.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.491+0000 7fb5b19b9640 1 -- 172.21.15.161:0/1818581067 learned_addr learned my addr 172.21.15.161:0/1818581067 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:49.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.491+0000 7fb5b11b8640 1 --2- 172.21.15.161:0/1818581067 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5ac0c5950 0x7fb5ac0c49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:49.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.491+0000 7fb5b09b7640 1 --2- 172.21.15.161:0/1818581067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5ac0c8ed0 0x7fb5ac0ab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:49.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.492+0000 7fb5b19b9640 1 -- 172.21.15.161:0/1818581067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5ac0c8ed0 msgr2=0x7fb5ac0ab900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:49.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.492+0000 7fb5b19b9640 1 --2- 172.21.15.161:0/1818581067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5ac0c8ed0 0x7fb5ac0ab900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:49.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.492+0000 7fb5b19b9640 1 -- 172.21.15.161:0/1818581067 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5ac0c5950 msgr2=0x7fb5ac0c49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:49.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.492+0000 7fb5b19b9640 1 --2- 172.21.15.161:0/1818581067 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5ac0c5950 0x7fb5ac0c49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:49.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.492+0000 7fb5b19b9640 1 -- 172.21.15.161:0/1818581067 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5ac1eb800 con 0x7fb5ac152890 2024-03-28T23:32:49.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.492+0000 7fb5b09b7640 1 --2- 172.21.15.161:0/1818581067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5ac0c8ed0 0x7fb5ac0ab900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:32:49.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.492+0000 7fb5b19b9640 1 --2- 172.21.15.161:0/1818581067 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5ac152890 0x7fb5ac1e7f80 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fb5a400eed0 tx=0x7fb5a400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:49.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.492+0000 7fb5a9ffb640 1 -- 172.21.15.161:0/1818581067 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a4018020 con 0x7fb5ac152890 2024-03-28T23:32:49.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.492+0000 7fb5a9ffb640 1 -- 172.21.15.161:0/1818581067 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5a400bd20 con 0x7fb5ac152890 2024-03-28T23:32:49.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.492+0000 7fb5b341c640 1 -- 172.21.15.161:0/1818581067 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5ac14a7e0 con 0x7fb5ac152890 2024-03-28T23:32:49.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.492+0000 7fb5a9ffb640 1 -- 172.21.15.161:0/1818581067 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a4014640 con 0x7fb5ac152890 2024-03-28T23:32:49.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.492+0000 7fb5b341c640 1 -- 172.21.15.161:0/1818581067 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5ac0c4400 con 0x7fb5ac152890 2024-03-28T23:32:49.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.493+0000 7fb5b341c640 1 -- 172.21.15.161:0/1818581067 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5ac04f4b0 con 0x7fb5ac152890 2024-03-28T23:32:49.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.494+0000 7fb5a9ffb640 1 -- 172.21.15.161:0/1818581067 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb5a4005d10 con 0x7fb5ac152890 2024-03-28T23:32:49.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.494+0000 7fb5a9ffb640 1 --2- 172.21.15.161:0/1818581067 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb59806b4d0 0x7fb59806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:49.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.494+0000 7fb5b11b8640 1 --2- 172.21.15.161:0/1818581067 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb59806b4d0 0x7fb59806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:49.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.494+0000 7fb5a9ffb640 1 -- 172.21.15.161:0/1818581067 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6957+0+0 (secure 0 0 0) 0x7fb5a408ab50 con 0x7fb5ac152890 2024-03-28T23:32:49.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.495+0000 7fb5b11b8640 1 --2- 172.21.15.161:0/1818581067 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb59806b4d0 0x7fb59806d990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb59c035000 tx=0x7fb59c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:49.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.499+0000 7fb5a9ffb640 1 -- 172.21.15.161:0/1818581067 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb5a4022070 con 0x7fb5ac152890 2024-03-28T23:32:49.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.676+0000 7fb5b341c640 1 -- 172.21.15.161:0/1818581067 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb5ac1eb5b0 con 0x7fb59806b4d0 2024-03-28T23:32:49.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.687+0000 7fb5a9ffb640 1 -- 172.21.15.161:0/1818581067 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+487558 (secure 0 0 0) 0x7fb5ac1eb5b0 con 0x7fb59806b4d0 2024-03-28T23:32:49.693 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:49.696 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:32:49.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.691+0000 7fb5b341c640 1 -- 172.21.15.161:0/1818581067 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb59806b4d0 msgr2=0x7fb59806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:49.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.691+0000 7fb5b341c640 1 --2- 172.21.15.161:0/1818581067 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb59806b4d0 0x7fb59806d990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb59c035000 tx=0x7fb59c008030 comp rx=0 tx=0).stop 2024-03-28T23:32:49.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.691+0000 7fb5b341c640 1 -- 172.21.15.161:0/1818581067 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5ac152890 msgr2=0x7fb5ac1e7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:49.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.691+0000 7fb5b341c640 1 --2- 172.21.15.161:0/1818581067 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5ac152890 0x7fb5ac1e7f80 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fb5a400eed0 tx=0x7fb5a400b480 comp rx=0 tx=0).stop 2024-03-28T23:32:49.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.691+0000 7fb5b341c640 1 -- 172.21.15.161:0/1818581067 shutdown_connections 2024-03-28T23:32:49.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.691+0000 7fb5b341c640 1 --2- 172.21.15.161:0/1818581067 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb59806b4d0 0x7fb59806d990 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:49.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.691+0000 7fb5b341c640 1 --2- 172.21.15.161:0/1818581067 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5ac152890 0x7fb5ac1e7f80 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:49.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.691+0000 7fb5b341c640 1 --2- 172.21.15.161:0/1818581067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5ac0c8ed0 0x7fb5ac0ab900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:49.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.691+0000 7fb5b341c640 1 --2- 172.21.15.161:0/1818581067 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5ac0c5950 0x7fb5ac0c49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:49.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.691+0000 7fb5b341c640 1 -- 172.21.15.161:0/1818581067 >> 172.21.15.161:0/1818581067 conn(0x7fb5ac0ac620 msgr2=0x7fb5ac151560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:49.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.691+0000 7fb5b341c640 1 -- 172.21.15.161:0/1818581067 shutdown_connections 2024-03-28T23:32:49.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:49.691+0000 7fb5b341c640 1 -- 172.21.15.161:0/1818581067 wait complete. 2024-03-28T23:32:49.703 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-03-28T23:32:49.346209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594390,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":210,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162,"ondisk_log_size":162,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":270,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240224,"kb_used_data":7408,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754734496,"statfs":{"total":773094113280,"available":772848123904,"internally_reserved":0,"allocated":7585792,"data_stored":4688228,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001023"},"pg_stats":[{"pgid":"14.3d","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164968+0000","last_change":"2024-03-28T23:32:45.124812+0000","last_active":"2024-03-28T23:32:46.164968+0000","last_peered":"2024-03-28T23:32:46.164968+0000","last_clean":"2024-03-28T23:32:46.164968+0000","last_became_active":"2024-03-28T23:32:45.124589+0000","last_became_peered":"2024-03-28T23:32:45.124589+0000","last_unstale":"2024-03-28T23:32:46.164968+0000","last_undegraded":"2024-03-28T23:32:46.164968+0000","last_fullsized":"2024-03-28T23:32:46.164968+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:35:54.086668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3f","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108246+0000","last_change":"2024-03-28T23:32:33.046706+0000","last_active":"2024-03-28T23:32:44.108246+0000","last_peered":"2024-03-28T23:32:44.108246+0000","last_clean":"2024-03-28T23:32:44.108246+0000","last_became_active":"2024-03-28T23:32:33.046222+0000","last_became_peered":"2024-03-28T23:32:33.046222+0000","last_unstale":"2024-03-28T23:32:44.108246+0000","last_undegraded":"2024-03-28T23:32:44.108246+0000","last_fullsized":"2024-03-28T23:32:44.108246+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:02:45.190566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.109309+0000","last_change":"2024-03-28T23:32:44.109309+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.109309+0000","last_undegraded":"2024-03-28T23:32:44.109309+0000","last_fullsized":"2024-03-28T23:32:44.109309+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165821+0000","last_change":"2024-03-28T23:32:33.051174+0000","last_active":"2024-03-28T23:32:46.165821+0000","last_peered":"2024-03-28T23:32:46.165821+0000","last_clean":"2024-03-28T23:32:46.165821+0000","last_became_active":"2024-03-28T23:32:33.050345+0000","last_became_peered":"2024-03-28T23:32:33.050345+0000","last_unstale":"2024-03-28T23:32:46.165821+0000","last_undegraded":"2024-03-28T23:32:46.165821+0000","last_fullsized":"2024-03-28T23:32:46.165821+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:41:39.657731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.3f","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166718+0000","last_change":"2024-03-28T23:32:45.127552+0000","last_active":"2024-03-28T23:32:46.166718+0000","last_peered":"2024-03-28T23:32:46.166718+0000","last_clean":"2024-03-28T23:32:46.166718+0000","last_became_active":"2024-03-28T23:32:45.127291+0000","last_became_peered":"2024-03-28T23:32:45.127291+0000","last_unstale":"2024-03-28T23:32:46.166718+0000","last_undegraded":"2024-03-28T23:32:46.166718+0000","last_fullsized":"2024-03-28T23:32:46.166718+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:21:07.618142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108168+0000","last_change":"2024-03-28T23:32:33.048785+0000","last_active":"2024-03-28T23:32:44.108168+0000","last_peered":"2024-03-28T23:32:44.108168+0000","last_clean":"2024-03-28T23:32:44.108168+0000","last_became_active":"2024-03-28T23:32:33.048519+0000","last_became_peered":"2024-03-28T23:32:33.048519+0000","last_unstale":"2024-03-28T23:32:44.108168+0000","last_undegraded":"2024-03-28T23:32:44.108168+0000","last_fullsized":"2024-03-28T23:32:44.108168+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:21.464120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111326+0000","last_change":"2024-03-28T23:32:44.111326+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111326+0000","last_undegraded":"2024-03-28T23:32:44.111326+0000","last_fullsized":"2024-03-28T23:32:44.111326+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106875+0000","last_change":"2024-03-28T23:32:33.033535+0000","last_active":"2024-03-28T23:32:44.106875+0000","last_peered":"2024-03-28T23:32:44.106875+0000","last_clean":"2024-03-28T23:32:44.106875+0000","last_became_active":"2024-03-28T23:32:33.033219+0000","last_became_peered":"2024-03-28T23:32:33.033219+0000","last_unstale":"2024-03-28T23:32:44.106875+0000","last_undegraded":"2024-03-28T23:32:44.106875+0000","last_fullsized":"2024-03-28T23:32:44.106875+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:39:39.113457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.106821+0000","last_change":"2024-03-28T23:32:44.106821+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.106821+0000","last_undegraded":"2024-03-28T23:32:44.106821+0000","last_fullsized":"2024-03-28T23:32:44.106821+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166903+0000","last_change":"2024-03-28T23:32:33.042972+0000","last_active":"2024-03-28T23:32:46.166903+0000","last_peered":"2024-03-28T23:32:46.166903+0000","last_clean":"2024-03-28T23:32:46.166903+0000","last_became_active":"2024-03-28T23:32:33.042665+0000","last_became_peered":"2024-03-28T23:32:33.042665+0000","last_unstale":"2024-03-28T23:32:46.166903+0000","last_undegraded":"2024-03-28T23:32:46.166903+0000","last_fullsized":"2024-03-28T23:32:46.166903+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:41:50.190883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.109201+0000","last_change":"2024-03-28T23:32:44.109201+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.109201+0000","last_undegraded":"2024-03-28T23:32:44.109201+0000","last_fullsized":"2024-03-28T23:32:44.109201+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165883+0000","last_change":"2024-03-28T23:32:33.030549+0000","last_active":"2024-03-28T23:32:46.165883+0000","last_peered":"2024-03-28T23:32:46.165883+0000","last_clean":"2024-03-28T23:32:46.165883+0000","last_became_active":"2024-03-28T23:32:33.030299+0000","last_became_peered":"2024-03-28T23:32:33.030299+0000","last_unstale":"2024-03-28T23:32:46.165883+0000","last_undegraded":"2024-03-28T23:32:46.165883+0000","last_fullsized":"2024-03-28T23:32:46.165883+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:35.055035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111404+0000","last_change":"2024-03-28T23:32:44.111404+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111404+0000","last_undegraded":"2024-03-28T23:32:44.111404+0000","last_fullsized":"2024-03-28T23:32:44.111404+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.111244+0000","last_change":"2024-03-28T23:32:33.044477+0000","last_active":"2024-03-28T23:32:44.111244+0000","last_peered":"2024-03-28T23:32:44.111244+0000","last_clean":"2024-03-28T23:32:44.111244+0000","last_became_active":"2024-03-28T23:32:33.043960+0000","last_became_peered":"2024-03-28T23:32:33.043960+0000","last_unstale":"2024-03-28T23:32:44.111244+0000","last_undegraded":"2024-03-28T23:32:44.111244+0000","last_fullsized":"2024-03-28T23:32:44.111244+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:21.464120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163593+0000","last_change":"2024-03-28T23:32:45.124985+0000","last_active":"2024-03-28T23:32:46.163593+0000","last_peered":"2024-03-28T23:32:46.163593+0000","last_clean":"2024-03-28T23:32:46.163593+0000","last_became_active":"2024-03-28T23:32:45.124079+0000","last_became_peered":"2024-03-28T23:32:45.124079+0000","last_unstale":"2024-03-28T23:32:46.163593+0000","last_undegraded":"2024-03-28T23:32:46.163593+0000","last_fullsized":"2024-03-28T23:32:46.163593+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:04:31.273811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165692+0000","last_change":"2024-03-28T23:32:33.052803+0000","last_active":"2024-03-28T23:32:46.165692+0000","last_peered":"2024-03-28T23:32:46.165692+0000","last_clean":"2024-03-28T23:32:46.165692+0000","last_became_active":"2024-03-28T23:32:33.052524+0000","last_became_peered":"2024-03-28T23:32:33.052524+0000","last_unstale":"2024-03-28T23:32:46.165692+0000","last_undegraded":"2024-03-28T23:32:46.165692+0000","last_fullsized":"2024-03-28T23:32:46.165692+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:19:09.500642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.117756+0000","last_change":"2024-03-28T23:32:44.117756+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.117756+0000","last_undegraded":"2024-03-28T23:32:44.117756+0000","last_fullsized":"2024-03-28T23:32:44.117756+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112756+0000","last_change":"2024-03-28T23:32:33.042587+0000","last_active":"2024-03-28T23:32:44.112756+0000","last_peered":"2024-03-28T23:32:44.112756+0000","last_clean":"2024-03-28T23:32:44.112756+0000","last_became_active":"2024-03-28T23:32:33.037002+0000","last_became_peered":"2024-03-28T23:32:33.037002+0000","last_unstale":"2024-03-28T23:32:44.112756+0000","last_undegraded":"2024-03-28T23:32:44.112756+0000","last_fullsized":"2024-03-28T23:32:44.112756+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:36:03.951054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165818+0000","last_change":"2024-03-28T23:32:45.113627+0000","last_active":"2024-03-28T23:32:46.165818+0000","last_peered":"2024-03-28T23:32:46.165818+0000","last_clean":"2024-03-28T23:32:46.165818+0000","last_became_active":"2024-03-28T23:32:45.113364+0000","last_became_peered":"2024-03-28T23:32:45.113364+0000","last_unstale":"2024-03-28T23:32:46.165818+0000","last_undegraded":"2024-03-28T23:32:46.165818+0000","last_fullsized":"2024-03-28T23:32:46.165818+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:54:25.867257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107498+0000","last_change":"2024-03-28T23:32:33.039941+0000","last_active":"2024-03-28T23:32:44.107498+0000","last_peered":"2024-03-28T23:32:44.107498+0000","last_clean":"2024-03-28T23:32:44.107498+0000","last_became_active":"2024-03-28T23:32:33.039656+0000","last_became_peered":"2024-03-28T23:32:33.039656+0000","last_unstale":"2024-03-28T23:32:44.107498+0000","last_undegraded":"2024-03-28T23:32:44.107498+0000","last_fullsized":"2024-03-28T23:32:44.107498+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:16.850667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111858+0000","last_change":"2024-03-28T23:32:44.111858+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111858+0000","last_undegraded":"2024-03-28T23:32:44.111858+0000","last_fullsized":"2024-03-28T23:32:44.111858+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164479+0000","last_change":"2024-03-28T23:32:33.052329+0000","last_active":"2024-03-28T23:32:46.164479+0000","last_peered":"2024-03-28T23:32:46.164479+0000","last_clean":"2024-03-28T23:32:46.164479+0000","last_became_active":"2024-03-28T23:32:33.051833+0000","last_became_peered":"2024-03-28T23:32:33.051833+0000","last_unstale":"2024-03-28T23:32:46.164479+0000","last_undegraded":"2024-03-28T23:32:46.164479+0000","last_fullsized":"2024-03-28T23:32:46.164479+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:05.929331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164594+0000","last_change":"2024-03-28T23:32:45.115847+0000","last_active":"2024-03-28T23:32:46.164594+0000","last_peered":"2024-03-28T23:32:46.164594+0000","last_clean":"2024-03-28T23:32:46.164594+0000","last_became_active":"2024-03-28T23:32:45.115452+0000","last_became_peered":"2024-03-28T23:32:45.115452+0000","last_unstale":"2024-03-28T23:32:46.164594+0000","last_undegraded":"2024-03-28T23:32:46.164594+0000","last_fullsized":"2024-03-28T23:32:46.164594+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:02.169192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106984+0000","last_change":"2024-03-28T23:32:33.042337+0000","last_active":"2024-03-28T23:32:44.106984+0000","last_peered":"2024-03-28T23:32:44.106984+0000","last_clean":"2024-03-28T23:32:44.106984+0000","last_became_active":"2024-03-28T23:32:33.042062+0000","last_became_peered":"2024-03-28T23:32:33.042062+0000","last_unstale":"2024-03-28T23:32:44.106984+0000","last_undegraded":"2024-03-28T23:32:44.106984+0000","last_fullsized":"2024-03-28T23:32:44.106984+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:13:34.921740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108039+0000","last_change":"2024-03-28T23:32:44.108039+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108039+0000","last_undegraded":"2024-03-28T23:32:44.108039+0000","last_fullsized":"2024-03-28T23:32:44.108039+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166985+0000","last_change":"2024-03-28T23:32:33.046685+0000","last_active":"2024-03-28T23:32:46.166985+0000","last_peered":"2024-03-28T23:32:46.166985+0000","last_clean":"2024-03-28T23:32:46.166985+0000","last_became_active":"2024-03-28T23:32:33.046206+0000","last_became_peered":"2024-03-28T23:32:33.046206+0000","last_unstale":"2024-03-28T23:32:46.166985+0000","last_undegraded":"2024-03-28T23:32:46.166985+0000","last_fullsized":"2024-03-28T23:32:46.166985+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:43:22.001147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163804+0000","last_change":"2024-03-28T23:32:45.133232+0000","last_active":"2024-03-28T23:32:46.163804+0000","last_peered":"2024-03-28T23:32:46.163804+0000","last_clean":"2024-03-28T23:32:46.163804+0000","last_became_active":"2024-03-28T23:32:45.132973+0000","last_became_peered":"2024-03-28T23:32:45.132973+0000","last_unstale":"2024-03-28T23:32:46.163804+0000","last_undegraded":"2024-03-28T23:32:46.163804+0000","last_fullsized":"2024-03-28T23:32:46.163804+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:03:04.477944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165972+0000","last_change":"2024-03-28T23:32:33.029880+0000","last_active":"2024-03-28T23:32:46.165972+0000","last_peered":"2024-03-28T23:32:46.165972+0000","last_clean":"2024-03-28T23:32:46.165972+0000","last_became_active":"2024-03-28T23:32:33.029582+0000","last_became_peered":"2024-03-28T23:32:33.029582+0000","last_unstale":"2024-03-28T23:32:46.165972+0000","last_undegraded":"2024-03-28T23:32:46.165972+0000","last_fullsized":"2024-03-28T23:32:46.165972+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:38:39.283932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111833+0000","last_change":"2024-03-28T23:32:44.111833+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111833+0000","last_undegraded":"2024-03-28T23:32:44.111833+0000","last_fullsized":"2024-03-28T23:32:44.111833+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165936+0000","last_change":"2024-03-28T23:32:33.040102+0000","last_active":"2024-03-28T23:32:46.165936+0000","last_peered":"2024-03-28T23:32:46.165936+0000","last_clean":"2024-03-28T23:32:46.165936+0000","last_became_active":"2024-03-28T23:32:33.039660+0000","last_became_peered":"2024-03-28T23:32:33.039660+0000","last_unstale":"2024-03-28T23:32:46.165936+0000","last_undegraded":"2024-03-28T23:32:46.165936+0000","last_fullsized":"2024-03-28T23:32:46.165936+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:35.055035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112037+0000","last_change":"2024-03-28T23:32:44.112037+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112037+0000","last_undegraded":"2024-03-28T23:32:44.112037+0000","last_fullsized":"2024-03-28T23:32:44.112037+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108875+0000","last_change":"2024-03-28T23:32:33.035442+0000","last_active":"2024-03-28T23:32:44.108875+0000","last_peered":"2024-03-28T23:32:44.108875+0000","last_clean":"2024-03-28T23:32:44.108875+0000","last_became_active":"2024-03-28T23:32:33.035154+0000","last_became_peered":"2024-03-28T23:32:33.035154+0000","last_unstale":"2024-03-28T23:32:44.108875+0000","last_undegraded":"2024-03-28T23:32:44.108875+0000","last_fullsized":"2024-03-28T23:32:44.108875+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:34:22.775718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.115970+0000","last_change":"2024-03-28T23:32:44.115970+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.115970+0000","last_undegraded":"2024-03-28T23:32:44.115970+0000","last_fullsized":"2024-03-28T23:32:44.115970+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165475+0000","last_change":"2024-03-28T23:32:33.047782+0000","last_active":"2024-03-28T23:32:46.165475+0000","last_peered":"2024-03-28T23:32:46.165475+0000","last_clean":"2024-03-28T23:32:46.165475+0000","last_became_active":"2024-03-28T23:32:33.047254+0000","last_became_peered":"2024-03-28T23:32:33.047254+0000","last_unstale":"2024-03-28T23:32:46.165475+0000","last_undegraded":"2024-03-28T23:32:46.165475+0000","last_fullsized":"2024-03-28T23:32:46.165475+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:19.004555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108940+0000","last_change":"2024-03-28T23:32:44.108940+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108940+0000","last_undegraded":"2024-03-28T23:32:44.108940+0000","last_fullsized":"2024-03-28T23:32:44.108940+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107847+0000","last_change":"2024-03-28T23:32:33.037321+0000","last_active":"2024-03-28T23:32:44.107847+0000","last_peered":"2024-03-28T23:32:44.107847+0000","last_clean":"2024-03-28T23:32:44.107847+0000","last_became_active":"2024-03-28T23:32:33.036961+0000","last_became_peered":"2024-03-28T23:32:33.036961+0000","last_unstale":"2024-03-28T23:32:44.107847+0000","last_undegraded":"2024-03-28T23:32:44.107847+0000","last_fullsized":"2024-03-28T23:32:44.107847+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:25.849919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164898+0000","last_change":"2024-03-28T23:32:45.124739+0000","last_active":"2024-03-28T23:32:46.164898+0000","last_peered":"2024-03-28T23:32:46.164898+0000","last_clean":"2024-03-28T23:32:46.164898+0000","last_became_active":"2024-03-28T23:32:45.124497+0000","last_became_peered":"2024-03-28T23:32:45.124497+0000","last_unstale":"2024-03-28T23:32:46.164898+0000","last_undegraded":"2024-03-28T23:32:46.164898+0000","last_fullsized":"2024-03-28T23:32:46.164898+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:22:55.704581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165055+0000","last_change":"2024-03-28T23:32:33.053071+0000","last_active":"2024-03-28T23:32:46.165055+0000","last_peered":"2024-03-28T23:32:46.165055+0000","last_clean":"2024-03-28T23:32:46.165055+0000","last_became_active":"2024-03-28T23:32:33.052784+0000","last_became_peered":"2024-03-28T23:32:33.052784+0000","last_unstale":"2024-03-28T23:32:46.165055+0000","last_undegraded":"2024-03-28T23:32:46.165055+0000","last_fullsized":"2024-03-28T23:32:46.165055+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:58:48.231248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164101+0000","last_change":"2024-03-28T23:32:45.123590+0000","last_active":"2024-03-28T23:32:46.164101+0000","last_peered":"2024-03-28T23:32:46.164101+0000","last_clean":"2024-03-28T23:32:46.164101+0000","last_became_active":"2024-03-28T23:32:45.123361+0000","last_became_peered":"2024-03-28T23:32:45.123361+0000","last_unstale":"2024-03-28T23:32:46.164101+0000","last_undegraded":"2024-03-28T23:32:46.164101+0000","last_fullsized":"2024-03-28T23:32:46.164101+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:49.881731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165902+0000","last_change":"2024-03-28T23:32:33.052720+0000","last_active":"2024-03-28T23:32:46.165902+0000","last_peered":"2024-03-28T23:32:46.165902+0000","last_clean":"2024-03-28T23:32:46.165902+0000","last_became_active":"2024-03-28T23:32:33.052460+0000","last_became_peered":"2024-03-28T23:32:33.052460+0000","last_unstale":"2024-03-28T23:32:46.165902+0000","last_undegraded":"2024-03-28T23:32:46.165902+0000","last_fullsized":"2024-03-28T23:32:46.165902+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:30.891315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107873+0000","last_change":"2024-03-28T23:32:44.107873+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107873+0000","last_undegraded":"2024-03-28T23:32:44.107873+0000","last_fullsized":"2024-03-28T23:32:44.107873+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112648+0000","last_change":"2024-03-28T23:32:33.035666+0000","last_active":"2024-03-28T23:32:44.112648+0000","last_peered":"2024-03-28T23:32:44.112648+0000","last_clean":"2024-03-28T23:32:44.112648+0000","last_became_active":"2024-03-28T23:32:33.035403+0000","last_became_peered":"2024-03-28T23:32:33.035403+0000","last_unstale":"2024-03-28T23:32:44.112648+0000","last_undegraded":"2024-03-28T23:32:44.112648+0000","last_fullsized":"2024-03-28T23:32:44.112648+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:40:51.269760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164304+0000","last_change":"2024-03-28T23:32:45.124238+0000","last_active":"2024-03-28T23:32:46.164304+0000","last_peered":"2024-03-28T23:32:46.164304+0000","last_clean":"2024-03-28T23:32:46.164304+0000","last_became_active":"2024-03-28T23:32:45.123997+0000","last_became_peered":"2024-03-28T23:32:45.123997+0000","last_unstale":"2024-03-28T23:32:46.164304+0000","last_undegraded":"2024-03-28T23:32:46.164304+0000","last_fullsized":"2024-03-28T23:32:46.164304+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:38:37.572370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164596+0000","last_change":"2024-03-28T23:32:33.047992+0000","last_active":"2024-03-28T23:32:46.164596+0000","last_peered":"2024-03-28T23:32:46.164596+0000","last_clean":"2024-03-28T23:32:46.164596+0000","last_became_active":"2024-03-28T23:32:33.047058+0000","last_became_peered":"2024-03-28T23:32:33.047058+0000","last_unstale":"2024-03-28T23:32:46.164596+0000","last_undegraded":"2024-03-28T23:32:46.164596+0000","last_fullsized":"2024-03-28T23:32:46.164596+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:35:30.886935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.110595+0000","last_change":"2024-03-28T23:32:44.110595+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.110595+0000","last_undegraded":"2024-03-28T23:32:44.110595+0000","last_fullsized":"2024-03-28T23:32:44.110595+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112366+0000","last_change":"2024-03-28T23:32:33.029536+0000","last_active":"2024-03-28T23:32:44.112366+0000","last_peered":"2024-03-28T23:32:44.112366+0000","last_clean":"2024-03-28T23:32:44.112366+0000","last_became_active":"2024-03-28T23:32:33.029265+0000","last_became_peered":"2024-03-28T23:32:33.029265+0000","last_unstale":"2024-03-28T23:32:44.112366+0000","last_undegraded":"2024-03-28T23:32:44.112366+0000","last_fullsized":"2024-03-28T23:32:44.112366+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:51:11.447507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.110858+0000","last_change":"2024-03-28T23:32:44.110858+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.110858+0000","last_undegraded":"2024-03-28T23:32:44.110858+0000","last_fullsized":"2024-03-28T23:32:44.110858+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163897+0000","last_change":"2024-03-28T23:32:33.041328+0000","last_active":"2024-03-28T23:32:46.163897+0000","last_peered":"2024-03-28T23:32:46.163897+0000","last_clean":"2024-03-28T23:32:46.163897+0000","last_became_active":"2024-03-28T23:32:33.041071+0000","last_became_peered":"2024-03-28T23:32:33.041071+0000","last_unstale":"2024-03-28T23:32:46.163897+0000","last_undegraded":"2024-03-28T23:32:46.163897+0000","last_fullsized":"2024-03-28T23:32:46.163897+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:00.358180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164970+0000","last_change":"2024-03-28T23:32:45.114377+0000","last_active":"2024-03-28T23:32:46.164970+0000","last_peered":"2024-03-28T23:32:46.164970+0000","last_clean":"2024-03-28T23:32:46.164970+0000","last_became_active":"2024-03-28T23:32:45.114137+0000","last_became_peered":"2024-03-28T23:32:45.114137+0000","last_unstale":"2024-03-28T23:32:46.164970+0000","last_undegraded":"2024-03-28T23:32:46.164970+0000","last_fullsized":"2024-03-28T23:32:46.164970+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:18:14.670198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165064+0000","last_change":"2024-03-28T23:32:33.043840+0000","last_active":"2024-03-28T23:32:46.165064+0000","last_peered":"2024-03-28T23:32:46.165064+0000","last_clean":"2024-03-28T23:32:46.165064+0000","last_became_active":"2024-03-28T23:32:33.043569+0000","last_became_peered":"2024-03-28T23:32:33.043569+0000","last_unstale":"2024-03-28T23:32:46.165064+0000","last_undegraded":"2024-03-28T23:32:46.165064+0000","last_fullsized":"2024-03-28T23:32:46.165064+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:28.463493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108552+0000","last_change":"2024-03-28T23:32:44.108552+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108552+0000","last_undegraded":"2024-03-28T23:32:44.108552+0000","last_fullsized":"2024-03-28T23:32:44.108552+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107631+0000","last_change":"2024-03-28T23:32:33.047594+0000","last_active":"2024-03-28T23:32:44.107631+0000","last_peered":"2024-03-28T23:32:44.107631+0000","last_clean":"2024-03-28T23:32:44.107631+0000","last_became_active":"2024-03-28T23:32:33.047031+0000","last_became_peered":"2024-03-28T23:32:33.047031+0000","last_unstale":"2024-03-28T23:32:44.107631+0000","last_undegraded":"2024-03-28T23:32:44.107631+0000","last_fullsized":"2024-03-28T23:32:44.107631+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:30:12.769818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112271+0000","last_change":"2024-03-28T23:32:44.112271+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112271+0000","last_undegraded":"2024-03-28T23:32:44.112271+0000","last_fullsized":"2024-03-28T23:32:44.112271+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107919+0000","last_change":"2024-03-28T23:32:33.043396+0000","last_active":"2024-03-28T23:32:44.107919+0000","last_peered":"2024-03-28T23:32:44.107919+0000","last_clean":"2024-03-28T23:32:44.107919+0000","last_became_active":"2024-03-28T23:32:33.043123+0000","last_became_peered":"2024-03-28T23:32:33.043123+0000","last_unstale":"2024-03-28T23:32:44.107919+0000","last_undegraded":"2024-03-28T23:32:44.107919+0000","last_fullsized":"2024-03-28T23:32:44.107919+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:28:36.535909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.114282+0000","last_change":"2024-03-28T23:32:44.114282+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.114282+0000","last_undegraded":"2024-03-28T23:32:44.114282+0000","last_fullsized":"2024-03-28T23:32:44.114282+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165979+0000","last_change":"2024-03-28T23:32:33.042288+0000","last_active":"2024-03-28T23:32:46.165979+0000","last_peered":"2024-03-28T23:32:46.165979+0000","last_clean":"2024-03-28T23:32:46.165979+0000","last_became_active":"2024-03-28T23:32:33.042007+0000","last_became_peered":"2024-03-28T23:32:33.042007+0000","last_unstale":"2024-03-28T23:32:46.165979+0000","last_undegraded":"2024-03-28T23:32:46.165979+0000","last_fullsized":"2024-03-28T23:32:46.165979+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:16:12.634582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.110873+0000","last_change":"2024-03-28T23:32:44.110873+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.110873+0000","last_undegraded":"2024-03-28T23:32:44.110873+0000","last_fullsized":"2024-03-28T23:32:44.110873+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165206+0000","last_change":"2024-03-28T23:32:33.041336+0000","last_active":"2024-03-28T23:32:46.165206+0000","last_peered":"2024-03-28T23:32:46.165206+0000","last_clean":"2024-03-28T23:32:46.165206+0000","last_became_active":"2024-03-28T23:32:33.040865+0000","last_became_peered":"2024-03-28T23:32:33.040865+0000","last_unstale":"2024-03-28T23:32:46.165206+0000","last_undegraded":"2024-03-28T23:32:46.165206+0000","last_fullsized":"2024-03-28T23:32:46.165206+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:44:58.784172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164344+0000","last_change":"2024-03-28T23:32:45.133607+0000","last_active":"2024-03-28T23:32:46.164344+0000","last_peered":"2024-03-28T23:32:46.164344+0000","last_clean":"2024-03-28T23:32:46.164344+0000","last_became_active":"2024-03-28T23:32:45.133367+0000","last_became_peered":"2024-03-28T23:32:45.133367+0000","last_unstale":"2024-03-28T23:32:46.164344+0000","last_undegraded":"2024-03-28T23:32:46.164344+0000","last_fullsized":"2024-03-28T23:32:46.164344+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:52:37.666022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106185+0000","last_change":"2024-03-28T23:32:33.049222+0000","last_active":"2024-03-28T23:32:44.106185+0000","last_peered":"2024-03-28T23:32:44.106185+0000","last_clean":"2024-03-28T23:32:44.106185+0000","last_became_active":"2024-03-28T23:32:33.048933+0000","last_became_peered":"2024-03-28T23:32:33.048933+0000","last_unstale":"2024-03-28T23:32:44.106185+0000","last_undegraded":"2024-03-28T23:32:44.106185+0000","last_fullsized":"2024-03-28T23:32:44.106185+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:17:45.545141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107516+0000","last_change":"2024-03-28T23:32:44.107516+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107516+0000","last_undegraded":"2024-03-28T23:32:44.107516+0000","last_fullsized":"2024-03-28T23:32:44.107516+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108405+0000","last_change":"2024-03-28T23:32:33.047977+0000","last_active":"2024-03-28T23:32:44.108405+0000","last_peered":"2024-03-28T23:32:44.108405+0000","last_clean":"2024-03-28T23:32:44.108405+0000","last_became_active":"2024-03-28T23:32:33.047699+0000","last_became_peered":"2024-03-28T23:32:33.047699+0000","last_unstale":"2024-03-28T23:32:44.108405+0000","last_undegraded":"2024-03-28T23:32:44.108405+0000","last_fullsized":"2024-03-28T23:32:44.108405+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:38:51.513907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111342+0000","last_change":"2024-03-28T23:32:44.111342+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111342+0000","last_undegraded":"2024-03-28T23:32:44.111342+0000","last_fullsized":"2024-03-28T23:32:44.111342+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108988+0000","last_change":"2024-03-28T23:32:33.036527+0000","last_active":"2024-03-28T23:32:44.108988+0000","last_peered":"2024-03-28T23:32:44.108988+0000","last_clean":"2024-03-28T23:32:44.108988+0000","last_became_active":"2024-03-28T23:32:33.036269+0000","last_became_peered":"2024-03-28T23:32:33.036269+0000","last_unstale":"2024-03-28T23:32:44.108988+0000","last_undegraded":"2024-03-28T23:32:44.108988+0000","last_fullsized":"2024-03-28T23:32:44.108988+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:43:35.946992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164668+0000","last_change":"2024-03-28T23:32:45.125187+0000","last_active":"2024-03-28T23:32:46.164668+0000","last_peered":"2024-03-28T23:32:46.164668+0000","last_clean":"2024-03-28T23:32:46.164668+0000","last_became_active":"2024-03-28T23:32:45.124029+0000","last_became_peered":"2024-03-28T23:32:45.124029+0000","last_unstale":"2024-03-28T23:32:46.164668+0000","last_undegraded":"2024-03-28T23:32:46.164668+0000","last_fullsized":"2024-03-28T23:32:46.164668+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:39.510794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164994+0000","last_change":"2024-03-28T23:32:44.118134+0000","last_active":"2024-03-28T23:32:46.164994+0000","last_peered":"2024-03-28T23:32:46.164994+0000","last_clean":"2024-03-28T23:32:46.164994+0000","last_became_active":"2024-03-28T23:32:44.117870+0000","last_became_peered":"2024-03-28T23:32:44.117870+0000","last_unstale":"2024-03-28T23:32:46.164994+0000","last_undegraded":"2024-03-28T23:32:46.164994+0000","last_fullsized":"2024-03-28T23:32:46.164994+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:41:50.736456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109389+0000","last_change":"2024-03-28T23:32:33.038016+0000","last_active":"2024-03-28T23:32:44.109389+0000","last_peered":"2024-03-28T23:32:44.109389+0000","last_clean":"2024-03-28T23:32:44.109389+0000","last_became_active":"2024-03-28T23:32:33.037754+0000","last_became_peered":"2024-03-28T23:32:33.037754+0000","last_unstale":"2024-03-28T23:32:44.109389+0000","last_undegraded":"2024-03-28T23:32:44.109389+0000","last_fullsized":"2024-03-28T23:32:44.109389+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:21.878156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"72'4","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165561+0000","last_change":"2024-03-28T23:32:32.040468+0000","last_active":"2024-03-28T23:32:46.165561+0000","last_peered":"2024-03-28T23:32:46.165561+0000","last_clean":"2024-03-28T23:32:46.165561+0000","last_became_active":"2024-03-28T23:32:32.039342+0000","last_became_peered":"2024-03-28T23:32:32.039342+0000","last_unstale":"2024-03-28T23:32:46.165561+0000","last_undegraded":"2024-03-28T23:32:46.165561+0000","last_fullsized":"2024-03-28T23:32:46.165561+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:05.786248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108504+0000","last_change":"2024-03-28T23:32:44.108504+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108504+0000","last_undegraded":"2024-03-28T23:32:44.108504+0000","last_fullsized":"2024-03-28T23:32:44.108504+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.127347+0000","last_change":"2024-03-28T23:32:44.126084+0000","last_active":"2024-03-28T23:32:44.127347+0000","last_peered":"2024-03-28T23:32:44.127347+0000","last_clean":"2024-03-28T23:32:44.127347+0000","last_became_active":"2024-03-28T23:32:44.125728+0000","last_became_peered":"2024-03-28T23:32:44.125728+0000","last_unstale":"2024-03-28T23:32:44.127347+0000","last_undegraded":"2024-03-28T23:32:44.127347+0000","last_fullsized":"2024-03-28T23:32:44.127347+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:17.008056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166027+0000","last_change":"2024-03-28T23:32:33.039778+0000","last_active":"2024-03-28T23:32:46.166027+0000","last_peered":"2024-03-28T23:32:46.166027+0000","last_clean":"2024-03-28T23:32:46.166027+0000","last_became_active":"2024-03-28T23:32:33.039506+0000","last_became_peered":"2024-03-28T23:32:33.039506+0000","last_unstale":"2024-03-28T23:32:46.166027+0000","last_undegraded":"2024-03-28T23:32:46.166027+0000","last_fullsized":"2024-03-28T23:32:46.166027+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:38:39.283932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"72'1","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107225+0000","last_change":"2024-03-28T23:32:32.041093+0000","last_active":"2024-03-28T23:32:44.107225+0000","last_peered":"2024-03-28T23:32:44.107225+0000","last_clean":"2024-03-28T23:32:44.107225+0000","last_became_active":"2024-03-28T23:32:32.040827+0000","last_became_peered":"2024-03-28T23:32:32.040827+0000","last_unstale":"2024-03-28T23:32:44.107225+0000","last_undegraded":"2024-03-28T23:32:44.107225+0000","last_fullsized":"2024-03-28T23:32:44.107225+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:55.544945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108669+0000","last_change":"2024-03-28T23:32:44.108669+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108669+0000","last_undegraded":"2024-03-28T23:32:44.108669+0000","last_fullsized":"2024-03-28T23:32:44.108669+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.120787+0000","last_change":"2024-03-28T23:32:44.119556+0000","last_active":"2024-03-28T23:32:44.120787+0000","last_peered":"2024-03-28T23:32:44.120787+0000","last_clean":"2024-03-28T23:32:44.120787+0000","last_became_active":"2024-03-28T23:32:44.119246+0000","last_became_peered":"2024-03-28T23:32:44.119246+0000","last_unstale":"2024-03-28T23:32:44.120787+0000","last_undegraded":"2024-03-28T23:32:44.120787+0000","last_fullsized":"2024-03-28T23:32:44.120787+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:36.928644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112355+0000","last_change":"2024-03-28T23:32:33.044887+0000","last_active":"2024-03-28T23:32:44.112355+0000","last_peered":"2024-03-28T23:32:44.112355+0000","last_clean":"2024-03-28T23:32:44.112355+0000","last_became_active":"2024-03-28T23:32:33.042005+0000","last_became_peered":"2024-03-28T23:32:33.042005+0000","last_unstale":"2024-03-28T23:32:44.112355+0000","last_undegraded":"2024-03-28T23:32:44.112355+0000","last_fullsized":"2024-03-28T23:32:44.112355+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:41.257241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"72'2","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108029+0000","last_change":"2024-03-28T23:32:32.040527+0000","last_active":"2024-03-28T23:32:44.108029+0000","last_peered":"2024-03-28T23:32:44.108029+0000","last_clean":"2024-03-28T23:32:44.108029+0000","last_became_active":"2024-03-28T23:32:32.040142+0000","last_became_peered":"2024-03-28T23:32:32.040142+0000","last_unstale":"2024-03-28T23:32:44.108029+0000","last_undegraded":"2024-03-28T23:32:44.108029+0000","last_fullsized":"2024-03-28T23:32:44.108029+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:13:15.856750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107395+0000","last_change":"2024-03-28T23:32:44.107395+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107395+0000","last_undegraded":"2024-03-28T23:32:44.107395+0000","last_fullsized":"2024-03-28T23:32:44.107395+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122155+0000","last_change":"2024-03-28T23:32:44.116203+0000","last_active":"2024-03-28T23:32:44.122155+0000","last_peered":"2024-03-28T23:32:44.122155+0000","last_clean":"2024-03-28T23:32:44.122155+0000","last_became_active":"2024-03-28T23:32:44.115846+0000","last_became_peered":"2024-03-28T23:32:44.115846+0000","last_unstale":"2024-03-28T23:32:44.122155+0000","last_undegraded":"2024-03-28T23:32:44.122155+0000","last_fullsized":"2024-03-28T23:32:44.122155+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:58.429743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112741+0000","last_change":"2024-03-28T23:32:33.040988+0000","last_active":"2024-03-28T23:32:44.112741+0000","last_peered":"2024-03-28T23:32:44.112741+0000","last_clean":"2024-03-28T23:32:44.112741+0000","last_became_active":"2024-03-28T23:32:33.036045+0000","last_became_peered":"2024-03-28T23:32:33.036045+0000","last_unstale":"2024-03-28T23:32:44.112741+0000","last_undegraded":"2024-03-28T23:32:44.112741+0000","last_fullsized":"2024-03-28T23:32:44.112741+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:10:39.981362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"72'2","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112781+0000","last_change":"2024-03-28T23:32:32.036856+0000","last_active":"2024-03-28T23:32:44.112781+0000","last_peered":"2024-03-28T23:32:44.112781+0000","last_clean":"2024-03-28T23:32:44.112781+0000","last_became_active":"2024-03-28T23:32:32.036565+0000","last_became_peered":"2024-03-28T23:32:32.036565+0000","last_unstale":"2024-03-28T23:32:44.112781+0000","last_undegraded":"2024-03-28T23:32:44.112781+0000","last_fullsized":"2024-03-28T23:32:44.112781+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:11:01.106436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.106357+0000","last_change":"2024-03-28T23:32:44.106357+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.106357+0000","last_undegraded":"2024-03-28T23:32:44.106357+0000","last_fullsized":"2024-03-28T23:32:44.106357+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.118200+0000","last_change":"2024-03-28T23:32:44.116097+0000","last_active":"2024-03-28T23:32:44.118200+0000","last_peered":"2024-03-28T23:32:44.118200+0000","last_clean":"2024-03-28T23:32:44.118200+0000","last_became_active":"2024-03-28T23:32:44.115810+0000","last_became_peered":"2024-03-28T23:32:44.115810+0000","last_unstale":"2024-03-28T23:32:44.118200+0000","last_undegraded":"2024-03-28T23:32:44.118200+0000","last_fullsized":"2024-03-28T23:32:44.118200+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:45:09.862897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166091+0000","last_change":"2024-03-28T23:32:33.047911+0000","last_active":"2024-03-28T23:32:46.166091+0000","last_peered":"2024-03-28T23:32:46.166091+0000","last_clean":"2024-03-28T23:32:46.166091+0000","last_became_active":"2024-03-28T23:32:33.047616+0000","last_became_peered":"2024-03-28T23:32:33.047616+0000","last_unstale":"2024-03-28T23:32:46.166091+0000","last_undegraded":"2024-03-28T23:32:46.166091+0000","last_fullsized":"2024-03-28T23:32:46.166091+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:58:48.231248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"72'1","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112850+0000","last_change":"2024-03-28T23:32:32.036829+0000","last_active":"2024-03-28T23:32:44.112850+0000","last_peered":"2024-03-28T23:32:44.112850+0000","last_clean":"2024-03-28T23:32:44.112850+0000","last_became_active":"2024-03-28T23:32:32.036571+0000","last_became_peered":"2024-03-28T23:32:32.036571+0000","last_unstale":"2024-03-28T23:32:44.112850+0000","last_undegraded":"2024-03-28T23:32:44.112850+0000","last_fullsized":"2024-03-28T23:32:44.112850+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:32:03.789593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165166+0000","last_change":"2024-03-28T23:32:45.127680+0000","last_active":"2024-03-28T23:32:46.165166+0000","last_peered":"2024-03-28T23:32:46.165166+0000","last_clean":"2024-03-28T23:32:46.165166+0000","last_became_active":"2024-03-28T23:32:45.127403+0000","last_became_peered":"2024-03-28T23:32:45.127403+0000","last_unstale":"2024-03-28T23:32:46.165166+0000","last_undegraded":"2024-03-28T23:32:46.165166+0000","last_fullsized":"2024-03-28T23:32:46.165166+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:34:51.492132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.120399+0000","last_change":"2024-03-28T23:32:44.117981+0000","last_active":"2024-03-28T23:32:44.120399+0000","last_peered":"2024-03-28T23:32:44.120399+0000","last_clean":"2024-03-28T23:32:44.120399+0000","last_became_active":"2024-03-28T23:32:44.117709+0000","last_became_peered":"2024-03-28T23:32:44.117709+0000","last_unstale":"2024-03-28T23:32:44.120399+0000","last_undegraded":"2024-03-28T23:32:44.120399+0000","last_fullsized":"2024-03-28T23:32:44.120399+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:13:27.942873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164579+0000","last_change":"2024-03-28T23:32:33.052330+0000","last_active":"2024-03-28T23:32:46.164579+0000","last_peered":"2024-03-28T23:32:46.164579+0000","last_clean":"2024-03-28T23:32:46.164579+0000","last_became_active":"2024-03-28T23:32:33.051828+0000","last_became_peered":"2024-03-28T23:32:33.051828+0000","last_unstale":"2024-03-28T23:32:46.164579+0000","last_undegraded":"2024-03-28T23:32:46.164579+0000","last_fullsized":"2024-03-28T23:32:46.164579+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:40:53.134608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"72'1","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106048+0000","last_change":"2024-03-28T23:32:32.036739+0000","last_active":"2024-03-28T23:32:44.106048+0000","last_peered":"2024-03-28T23:32:44.106048+0000","last_clean":"2024-03-28T23:32:44.106048+0000","last_became_active":"2024-03-28T23:32:32.036471+0000","last_became_peered":"2024-03-28T23:32:32.036471+0000","last_unstale":"2024-03-28T23:32:44.106048+0000","last_undegraded":"2024-03-28T23:32:44.106048+0000","last_fullsized":"2024-03-28T23:32:44.106048+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:48.704351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165297+0000","last_change":"2024-03-28T23:32:45.136600+0000","last_active":"2024-03-28T23:32:46.165297+0000","last_peered":"2024-03-28T23:32:46.165297+0000","last_clean":"2024-03-28T23:32:46.165297+0000","last_became_active":"2024-03-28T23:32:45.136366+0000","last_became_peered":"2024-03-28T23:32:45.136366+0000","last_unstale":"2024-03-28T23:32:46.165297+0000","last_undegraded":"2024-03-28T23:32:46.165297+0000","last_fullsized":"2024-03-28T23:32:46.165297+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:06:58.580120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.127627+0000","last_change":"2024-03-28T23:32:44.126818+0000","last_active":"2024-03-28T23:32:44.127627+0000","last_peered":"2024-03-28T23:32:44.127627+0000","last_clean":"2024-03-28T23:32:44.127627+0000","last_became_active":"2024-03-28T23:32:44.126493+0000","last_became_peered":"2024-03-28T23:32:44.126493+0000","last_unstale":"2024-03-28T23:32:44.127627+0000","last_undegraded":"2024-03-28T23:32:44.127627+0000","last_fullsized":"2024-03-28T23:32:44.127627+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:06:32.695104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.115074+0000","last_change":"2024-03-28T23:32:33.034126+0000","last_active":"2024-03-28T23:32:44.115074+0000","last_peered":"2024-03-28T23:32:44.115074+0000","last_clean":"2024-03-28T23:32:44.115074+0000","last_became_active":"2024-03-28T23:32:33.033703+0000","last_became_peered":"2024-03-28T23:32:33.033703+0000","last_unstale":"2024-03-28T23:32:44.115074+0000","last_undegraded":"2024-03-28T23:32:44.115074+0000","last_fullsized":"2024-03-28T23:32:44.115074+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:38.859031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166116+0000","last_change":"2024-03-28T23:32:32.039047+0000","last_active":"2024-03-28T23:32:46.166116+0000","last_peered":"2024-03-28T23:32:46.166116+0000","last_clean":"2024-03-28T23:32:46.166116+0000","last_became_active":"2024-03-28T23:32:32.038780+0000","last_became_peered":"2024-03-28T23:32:32.038780+0000","last_unstale":"2024-03-28T23:32:46.166116+0000","last_undegraded":"2024-03-28T23:32:46.166116+0000","last_fullsized":"2024-03-28T23:32:46.166116+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:13:15.856750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164099+0000","last_change":"2024-03-28T23:32:45.125579+0000","last_active":"2024-03-28T23:32:46.164099+0000","last_peered":"2024-03-28T23:32:46.164099+0000","last_clean":"2024-03-28T23:32:46.164099+0000","last_became_active":"2024-03-28T23:32:45.125164+0000","last_became_peered":"2024-03-28T23:32:45.125164+0000","last_unstale":"2024-03-28T23:32:46.164099+0000","last_undegraded":"2024-03-28T23:32:46.164099+0000","last_fullsized":"2024-03-28T23:32:46.164099+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:13:27.723637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122300+0000","last_change":"2024-03-28T23:32:44.121416+0000","last_active":"2024-03-28T23:32:44.122300+0000","last_peered":"2024-03-28T23:32:44.122300+0000","last_clean":"2024-03-28T23:32:44.122300+0000","last_became_active":"2024-03-28T23:32:44.121127+0000","last_became_peered":"2024-03-28T23:32:44.121127+0000","last_unstale":"2024-03-28T23:32:44.122300+0000","last_undegraded":"2024-03-28T23:32:44.122300+0000","last_fullsized":"2024-03-28T23:32:44.122300+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:39:02.592632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113417+0000","last_change":"2024-03-28T23:32:33.051453+0000","last_active":"2024-03-28T23:32:44.113417+0000","last_peered":"2024-03-28T23:32:44.113417+0000","last_clean":"2024-03-28T23:32:44.113417+0000","last_became_active":"2024-03-28T23:32:33.051155+0000","last_became_peered":"2024-03-28T23:32:33.051155+0000","last_unstale":"2024-03-28T23:32:44.113417+0000","last_undegraded":"2024-03-28T23:32:44.113417+0000","last_fullsized":"2024-03-28T23:32:44.113417+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:36:03.951054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165820+0000","last_change":"2024-03-28T23:32:32.036979+0000","last_active":"2024-03-28T23:32:46.165820+0000","last_peered":"2024-03-28T23:32:46.165820+0000","last_clean":"2024-03-28T23:32:46.165820+0000","last_became_active":"2024-03-28T23:32:32.036710+0000","last_became_peered":"2024-03-28T23:32:32.036710+0000","last_unstale":"2024-03-28T23:32:46.165820+0000","last_undegraded":"2024-03-28T23:32:46.165820+0000","last_fullsized":"2024-03-28T23:32:46.165820+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:46.072852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164193+0000","last_change":"2024-03-28T23:32:45.115755+0000","last_active":"2024-03-28T23:32:46.164193+0000","last_peered":"2024-03-28T23:32:46.164193+0000","last_clean":"2024-03-28T23:32:46.164193+0000","last_became_active":"2024-03-28T23:32:45.115490+0000","last_became_peered":"2024-03-28T23:32:45.115490+0000","last_unstale":"2024-03-28T23:32:46.164193+0000","last_undegraded":"2024-03-28T23:32:46.164193+0000","last_fullsized":"2024-03-28T23:32:46.164193+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:56.559065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.122075+0000","last_change":"2024-03-28T23:32:44.115740+0000","last_active":"2024-03-28T23:32:44.122075+0000","last_peered":"2024-03-28T23:32:44.122075+0000","last_clean":"2024-03-28T23:32:44.122075+0000","last_became_active":"2024-03-28T23:32:44.115341+0000","last_became_peered":"2024-03-28T23:32:44.115341+0000","last_unstale":"2024-03-28T23:32:44.122075+0000","last_undegraded":"2024-03-28T23:32:44.122075+0000","last_fullsized":"2024-03-28T23:32:44.122075+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:55.159548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163983+0000","last_change":"2024-03-28T23:32:33.041923+0000","last_active":"2024-03-28T23:32:46.163983+0000","last_peered":"2024-03-28T23:32:46.163983+0000","last_clean":"2024-03-28T23:32:46.163983+0000","last_became_active":"2024-03-28T23:32:33.041631+0000","last_became_peered":"2024-03-28T23:32:33.041631+0000","last_unstale":"2024-03-28T23:32:46.163983+0000","last_undegraded":"2024-03-28T23:32:46.163983+0000","last_fullsized":"2024-03-28T23:32:46.163983+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:32:08.473276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108352+0000","last_change":"2024-03-28T23:32:32.041563+0000","last_active":"2024-03-28T23:32:44.108352+0000","last_peered":"2024-03-28T23:32:44.108352+0000","last_clean":"2024-03-28T23:32:44.108352+0000","last_became_active":"2024-03-28T23:32:32.040424+0000","last_became_peered":"2024-03-28T23:32:32.040424+0000","last_unstale":"2024-03-28T23:32:44.108352+0000","last_undegraded":"2024-03-28T23:32:44.108352+0000","last_fullsized":"2024-03-28T23:32:44.108352+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:37:25.645113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166814+0000","last_change":"2024-03-28T23:32:45.127172+0000","last_active":"2024-03-28T23:32:46.166814+0000","last_peered":"2024-03-28T23:32:46.166814+0000","last_clean":"2024-03-28T23:32:46.166814+0000","last_became_active":"2024-03-28T23:32:45.126831+0000","last_became_peered":"2024-03-28T23:32:45.126831+0000","last_unstale":"2024-03-28T23:32:46.166814+0000","last_undegraded":"2024-03-28T23:32:46.166814+0000","last_fullsized":"2024-03-28T23:32:46.166814+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:51:23.531959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.123301+0000","last_change":"2024-03-28T23:32:44.118040+0000","last_active":"2024-03-28T23:32:44.123301+0000","last_peered":"2024-03-28T23:32:44.123301+0000","last_clean":"2024-03-28T23:32:44.123301+0000","last_became_active":"2024-03-28T23:32:44.117536+0000","last_became_peered":"2024-03-28T23:32:44.117536+0000","last_unstale":"2024-03-28T23:32:44.123301+0000","last_undegraded":"2024-03-28T23:32:44.123301+0000","last_fullsized":"2024-03-28T23:32:44.123301+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:59:07.631069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106030+0000","last_change":"2024-03-28T23:32:33.029986+0000","last_active":"2024-03-28T23:32:44.106030+0000","last_peered":"2024-03-28T23:32:44.106030+0000","last_clean":"2024-03-28T23:32:44.106030+0000","last_became_active":"2024-03-28T23:32:33.029413+0000","last_became_peered":"2024-03-28T23:32:33.029413+0000","last_unstale":"2024-03-28T23:32:44.106030+0000","last_undegraded":"2024-03-28T23:32:44.106030+0000","last_fullsized":"2024-03-28T23:32:44.106030+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:37:26.652511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107737+0000","last_change":"2024-03-28T23:32:32.037039+0000","last_active":"2024-03-28T23:32:44.107737+0000","last_peered":"2024-03-28T23:32:44.107737+0000","last_clean":"2024-03-28T23:32:44.107737+0000","last_became_active":"2024-03-28T23:32:32.036687+0000","last_became_peered":"2024-03-28T23:32:32.036687+0000","last_unstale":"2024-03-28T23:32:44.107737+0000","last_undegraded":"2024-03-28T23:32:44.107737+0000","last_fullsized":"2024-03-28T23:32:44.107737+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:27:15.824876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107961+0000","last_change":"2024-03-28T23:32:44.107961+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107961+0000","last_undegraded":"2024-03-28T23:32:44.107961+0000","last_fullsized":"2024-03-28T23:32:44.107961+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.127537+0000","last_change":"2024-03-28T23:32:44.126156+0000","last_active":"2024-03-28T23:32:44.127537+0000","last_peered":"2024-03-28T23:32:44.127537+0000","last_clean":"2024-03-28T23:32:44.127537+0000","last_became_active":"2024-03-28T23:32:44.125789+0000","last_became_peered":"2024-03-28T23:32:44.125789+0000","last_unstale":"2024-03-28T23:32:44.127537+0000","last_undegraded":"2024-03-28T23:32:44.127537+0000","last_fullsized":"2024-03-28T23:32:44.127537+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:41:04.213333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166116+0000","last_change":"2024-03-28T23:32:33.035809+0000","last_active":"2024-03-28T23:32:46.166116+0000","last_peered":"2024-03-28T23:32:46.166116+0000","last_clean":"2024-03-28T23:32:46.166116+0000","last_became_active":"2024-03-28T23:32:33.035563+0000","last_became_peered":"2024-03-28T23:32:33.035563+0000","last_unstale":"2024-03-28T23:32:46.166116+0000","last_undegraded":"2024-03-28T23:32:46.166116+0000","last_fullsized":"2024-03-28T23:32:46.166116+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:54.194453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"72'3","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165106+0000","last_change":"2024-03-28T23:32:32.038230+0000","last_active":"2024-03-28T23:32:46.165106+0000","last_peered":"2024-03-28T23:32:46.165106+0000","last_clean":"2024-03-28T23:32:46.165106+0000","last_became_active":"2024-03-28T23:32:32.037681+0000","last_became_peered":"2024-03-28T23:32:32.037681+0000","last_unstale":"2024-03-28T23:32:46.165106+0000","last_undegraded":"2024-03-28T23:32:46.165106+0000","last_fullsized":"2024-03-28T23:32:46.165106+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:36:34.752866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.112106+0000","last_change":"2024-03-28T23:32:44.112106+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.112106+0000","last_undegraded":"2024-03-28T23:32:44.112106+0000","last_fullsized":"2024-03-28T23:32:44.112106+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164529+0000","last_change":"2024-03-28T23:32:44.125087+0000","last_active":"2024-03-28T23:32:46.164529+0000","last_peered":"2024-03-28T23:32:46.164529+0000","last_clean":"2024-03-28T23:32:46.164529+0000","last_became_active":"2024-03-28T23:32:44.124784+0000","last_became_peered":"2024-03-28T23:32:44.124784+0000","last_unstale":"2024-03-28T23:32:46.164529+0000","last_undegraded":"2024-03-28T23:32:46.164529+0000","last_fullsized":"2024-03-28T23:32:46.164529+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:48.876004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112348+0000","last_change":"2024-03-28T23:32:33.049499+0000","last_active":"2024-03-28T23:32:44.112348+0000","last_peered":"2024-03-28T23:32:44.112348+0000","last_clean":"2024-03-28T23:32:44.112348+0000","last_became_active":"2024-03-28T23:32:33.049233+0000","last_became_peered":"2024-03-28T23:32:33.049233+0000","last_unstale":"2024-03-28T23:32:44.112348+0000","last_undegraded":"2024-03-28T23:32:44.112348+0000","last_fullsized":"2024-03-28T23:32:44.112348+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:39:02.824323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"72'6","reported_seq":24,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112883+0000","last_change":"2024-03-28T23:32:32.036835+0000","last_active":"2024-03-28T23:32:44.112883+0000","last_peered":"2024-03-28T23:32:44.112883+0000","last_clean":"2024-03-28T23:32:44.112883+0000","last_became_active":"2024-03-28T23:32:32.036557+0000","last_became_peered":"2024-03-28T23:32:32.036557+0000","last_unstale":"2024-03-28T23:32:44.112883+0000","last_undegraded":"2024-03-28T23:32:44.112883+0000","last_fullsized":"2024-03-28T23:32:44.112883+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:30.992807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166649+0000","last_change":"2024-03-28T23:32:45.135931+0000","last_active":"2024-03-28T23:32:46.166649+0000","last_peered":"2024-03-28T23:32:46.166649+0000","last_clean":"2024-03-28T23:32:46.166649+0000","last_became_active":"2024-03-28T23:32:45.135656+0000","last_became_peered":"2024-03-28T23:32:45.135656+0000","last_unstale":"2024-03-28T23:32:46.166649+0000","last_undegraded":"2024-03-28T23:32:46.166649+0000","last_fullsized":"2024-03-28T23:32:46.166649+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:40:28.334421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.127678+0000","last_change":"2024-03-28T23:32:44.127256+0000","last_active":"2024-03-28T23:32:44.127678+0000","last_peered":"2024-03-28T23:32:44.127678+0000","last_clean":"2024-03-28T23:32:44.127678+0000","last_became_active":"2024-03-28T23:32:44.126894+0000","last_became_peered":"2024-03-28T23:32:44.126894+0000","last_unstale":"2024-03-28T23:32:44.127678+0000","last_undegraded":"2024-03-28T23:32:44.127678+0000","last_fullsized":"2024-03-28T23:32:44.127678+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:28:37.442827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107086+0000","last_change":"2024-03-28T23:32:33.033045+0000","last_active":"2024-03-28T23:32:44.107086+0000","last_peered":"2024-03-28T23:32:44.107086+0000","last_clean":"2024-03-28T23:32:44.107086+0000","last_became_active":"2024-03-28T23:32:33.032771+0000","last_became_peered":"2024-03-28T23:32:33.032771+0000","last_unstale":"2024-03-28T23:32:44.107086+0000","last_undegraded":"2024-03-28T23:32:44.107086+0000","last_fullsized":"2024-03-28T23:32:44.107086+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:43.341877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"72'2","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106619+0000","last_change":"2024-03-28T23:32:32.032053+0000","last_active":"2024-03-28T23:32:44.106619+0000","last_peered":"2024-03-28T23:32:44.106619+0000","last_clean":"2024-03-28T23:32:44.106619+0000","last_became_active":"2024-03-28T23:32:32.031741+0000","last_became_peered":"2024-03-28T23:32:32.031741+0000","last_unstale":"2024-03-28T23:32:44.106619+0000","last_undegraded":"2024-03-28T23:32:44.106619+0000","last_fullsized":"2024-03-28T23:32:44.106619+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:20.456721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.108993+0000","last_change":"2024-03-28T23:32:44.108993+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.108993+0000","last_undegraded":"2024-03-28T23:32:44.108993+0000","last_fullsized":"2024-03-28T23:32:44.108993+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"76'6","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165387+0000","last_change":"2024-03-28T23:32:44.124910+0000","last_active":"2024-03-28T23:32:46.165387+0000","last_peered":"2024-03-28T23:32:46.165387+0000","last_clean":"2024-03-28T23:32:46.165387+0000","last_became_active":"2024-03-28T23:32:44.124639+0000","last_became_peered":"2024-03-28T23:32:44.124639+0000","last_unstale":"2024-03-28T23:32:46.165387+0000","last_undegraded":"2024-03-28T23:32:46.165387+0000","last_fullsized":"2024-03-28T23:32:46.165387+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:20:17.013749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112459+0000","last_change":"2024-03-28T23:32:33.039022+0000","last_active":"2024-03-28T23:32:44.112459+0000","last_peered":"2024-03-28T23:32:44.112459+0000","last_clean":"2024-03-28T23:32:44.112459+0000","last_became_active":"2024-03-28T23:32:33.038508+0000","last_became_peered":"2024-03-28T23:32:33.038508+0000","last_unstale":"2024-03-28T23:32:44.112459+0000","last_undegraded":"2024-03-28T23:32:44.112459+0000","last_fullsized":"2024-03-28T23:32:44.112459+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:25:45.189859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106348+0000","last_change":"2024-03-28T23:32:32.041042+0000","last_active":"2024-03-28T23:32:44.106348+0000","last_peered":"2024-03-28T23:32:44.106348+0000","last_clean":"2024-03-28T23:32:44.106348+0000","last_became_active":"2024-03-28T23:32:32.040686+0000","last_became_peered":"2024-03-28T23:32:32.040686+0000","last_unstale":"2024-03-28T23:32:44.106348+0000","last_undegraded":"2024-03-28T23:32:44.106348+0000","last_fullsized":"2024-03-28T23:32:44.106348+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:46.343619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.114552+0000","last_change":"2024-03-28T23:32:44.114552+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.114552+0000","last_undegraded":"2024-03-28T23:32:44.114552+0000","last_fullsized":"2024-03-28T23:32:44.114552+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.121906+0000","last_change":"2024-03-28T23:32:44.120268+0000","last_active":"2024-03-28T23:32:44.121906+0000","last_peered":"2024-03-28T23:32:44.121906+0000","last_clean":"2024-03-28T23:32:44.121906+0000","last_became_active":"2024-03-28T23:32:44.119951+0000","last_became_peered":"2024-03-28T23:32:44.119951+0000","last_unstale":"2024-03-28T23:32:44.121906+0000","last_undegraded":"2024-03-28T23:32:44.121906+0000","last_fullsized":"2024-03-28T23:32:44.121906+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:28:47.614634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112006+0000","last_change":"2024-03-28T23:32:33.034262+0000","last_active":"2024-03-28T23:32:44.112006+0000","last_peered":"2024-03-28T23:32:44.112006+0000","last_clean":"2024-03-28T23:32:44.112006+0000","last_became_active":"2024-03-28T23:32:33.034003+0000","last_became_peered":"2024-03-28T23:32:33.034003+0000","last_unstale":"2024-03-28T23:32:44.112006+0000","last_undegraded":"2024-03-28T23:32:44.112006+0000","last_fullsized":"2024-03-28T23:32:44.112006+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:41.257241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163979+0000","last_change":"2024-03-28T23:32:32.039031+0000","last_active":"2024-03-28T23:32:46.163979+0000","last_peered":"2024-03-28T23:32:46.163979+0000","last_clean":"2024-03-28T23:32:46.163979+0000","last_became_active":"2024-03-28T23:32:32.038766+0000","last_became_peered":"2024-03-28T23:32:32.038766+0000","last_unstale":"2024-03-28T23:32:46.163979+0000","last_undegraded":"2024-03-28T23:32:46.163979+0000","last_fullsized":"2024-03-28T23:32:46.163979+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:13.872794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107400+0000","last_change":"2024-03-28T23:32:44.107400+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107400+0000","last_undegraded":"2024-03-28T23:32:44.107400+0000","last_fullsized":"2024-03-28T23:32:44.107400+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166739+0000","last_change":"2024-03-28T23:32:44.119707+0000","last_active":"2024-03-28T23:32:46.166739+0000","last_peered":"2024-03-28T23:32:46.166739+0000","last_clean":"2024-03-28T23:32:46.166739+0000","last_became_active":"2024-03-28T23:32:44.119438+0000","last_became_peered":"2024-03-28T23:32:44.119438+0000","last_unstale":"2024-03-28T23:32:46.166739+0000","last_undegraded":"2024-03-28T23:32:46.166739+0000","last_fullsized":"2024-03-28T23:32:46.166739+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:22:16.280489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113512+0000","last_change":"2024-03-28T23:32:33.043171+0000","last_active":"2024-03-28T23:32:44.113512+0000","last_peered":"2024-03-28T23:32:44.113512+0000","last_clean":"2024-03-28T23:32:44.113512+0000","last_became_active":"2024-03-28T23:32:33.042821+0000","last_became_peered":"2024-03-28T23:32:33.042821+0000","last_unstale":"2024-03-28T23:32:44.113512+0000","last_undegraded":"2024-03-28T23:32:44.113512+0000","last_fullsized":"2024-03-28T23:32:44.113512+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:21.878156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"72'8","reported_seq":30,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164138+0000","last_change":"2024-03-28T23:32:32.040166+0000","last_active":"2024-03-28T23:32:46.164138+0000","last_peered":"2024-03-28T23:32:46.164138+0000","last_clean":"2024-03-28T23:32:46.164138+0000","last_became_active":"2024-03-28T23:32:32.039560+0000","last_became_peered":"2024-03-28T23:32:32.039560+0000","last_unstale":"2024-03-28T23:32:46.164138+0000","last_undegraded":"2024-03-28T23:32:46.164138+0000","last_fullsized":"2024-03-28T23:32:46.164138+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:55.544945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.110399+0000","last_change":"2024-03-28T23:32:44.110399+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.110399+0000","last_undegraded":"2024-03-28T23:32:44.110399+0000","last_fullsized":"2024-03-28T23:32:44.110399+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.127360+0000","last_change":"2024-03-28T23:32:44.126116+0000","last_active":"2024-03-28T23:32:44.127360+0000","last_peered":"2024-03-28T23:32:44.127360+0000","last_clean":"2024-03-28T23:32:44.127360+0000","last_became_active":"2024-03-28T23:32:44.125761+0000","last_became_peered":"2024-03-28T23:32:44.125761+0000","last_unstale":"2024-03-28T23:32:44.127360+0000","last_undegraded":"2024-03-28T23:32:44.127360+0000","last_fullsized":"2024-03-28T23:32:44.127360+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:04.591562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164561+0000","last_change":"2024-03-28T23:32:33.043192+0000","last_active":"2024-03-28T23:32:46.164561+0000","last_peered":"2024-03-28T23:32:46.164561+0000","last_clean":"2024-03-28T23:32:46.164561+0000","last_became_active":"2024-03-28T23:32:33.042894+0000","last_became_peered":"2024-03-28T23:32:33.042894+0000","last_unstale":"2024-03-28T23:32:46.164561+0000","last_undegraded":"2024-03-28T23:32:46.164561+0000","last_fullsized":"2024-03-28T23:32:46.164561+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:32:08.473276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"72'1","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108454+0000","last_change":"2024-03-28T23:32:32.041892+0000","last_active":"2024-03-28T23:32:44.108454+0000","last_peered":"2024-03-28T23:32:44.108454+0000","last_clean":"2024-03-28T23:32:44.108454+0000","last_became_active":"2024-03-28T23:32:32.041533+0000","last_became_peered":"2024-03-28T23:32:32.041533+0000","last_unstale":"2024-03-28T23:32:44.108454+0000","last_undegraded":"2024-03-28T23:32:44.108454+0000","last_fullsized":"2024-03-28T23:32:44.108454+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:15.843269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165083+0000","last_change":"2024-03-28T23:32:45.125611+0000","last_active":"2024-03-28T23:32:46.165083+0000","last_peered":"2024-03-28T23:32:46.165083+0000","last_clean":"2024-03-28T23:32:46.165083+0000","last_became_active":"2024-03-28T23:32:45.125349+0000","last_became_peered":"2024-03-28T23:32:45.125349+0000","last_unstale":"2024-03-28T23:32:46.165083+0000","last_undegraded":"2024-03-28T23:32:46.165083+0000","last_fullsized":"2024-03-28T23:32:46.165083+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:29:40.900683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165617+0000","last_change":"2024-03-28T23:32:44.115585+0000","last_active":"2024-03-28T23:32:46.165617+0000","last_peered":"2024-03-28T23:32:46.165617+0000","last_clean":"2024-03-28T23:32:46.165617+0000","last_became_active":"2024-03-28T23:32:44.115308+0000","last_became_peered":"2024-03-28T23:32:44.115308+0000","last_unstale":"2024-03-28T23:32:46.165617+0000","last_undegraded":"2024-03-28T23:32:46.165617+0000","last_fullsized":"2024-03-28T23:32:46.165617+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:18:00.809113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163885+0000","last_change":"2024-03-28T23:32:33.046899+0000","last_active":"2024-03-28T23:32:46.163885+0000","last_peered":"2024-03-28T23:32:46.163885+0000","last_clean":"2024-03-28T23:32:46.163885+0000","last_became_active":"2024-03-28T23:32:33.046579+0000","last_became_peered":"2024-03-28T23:32:33.046579+0000","last_unstale":"2024-03-28T23:32:46.163885+0000","last_undegraded":"2024-03-28T23:32:46.163885+0000","last_fullsized":"2024-03-28T23:32:46.163885+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:08:06.339997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"72'2","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.115368+0000","last_change":"2024-03-28T23:32:32.031135+0000","last_active":"2024-03-28T23:32:44.115368+0000","last_peered":"2024-03-28T23:32:44.115368+0000","last_clean":"2024-03-28T23:32:44.115368+0000","last_became_active":"2024-03-28T23:32:32.030796+0000","last_became_peered":"2024-03-28T23:32:32.030796+0000","last_unstale":"2024-03-28T23:32:44.115368+0000","last_undegraded":"2024-03-28T23:32:44.115368+0000","last_fullsized":"2024-03-28T23:32:44.115368+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:18:21.816565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163601+0000","last_change":"2024-03-28T23:32:45.134234+0000","last_active":"2024-03-28T23:32:46.163601+0000","last_peered":"2024-03-28T23:32:46.163601+0000","last_clean":"2024-03-28T23:32:46.163601+0000","last_became_active":"2024-03-28T23:32:45.134012+0000","last_became_peered":"2024-03-28T23:32:45.134012+0000","last_unstale":"2024-03-28T23:32:46.163601+0000","last_undegraded":"2024-03-28T23:32:46.163601+0000","last_fullsized":"2024-03-28T23:32:46.163601+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:44.843761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.117708+0000","last_change":"2024-03-28T23:32:44.116752+0000","last_active":"2024-03-28T23:32:44.117708+0000","last_peered":"2024-03-28T23:32:44.117708+0000","last_clean":"2024-03-28T23:32:44.117708+0000","last_became_active":"2024-03-28T23:32:44.116381+0000","last_became_peered":"2024-03-28T23:32:44.116381+0000","last_unstale":"2024-03-28T23:32:44.117708+0000","last_undegraded":"2024-03-28T23:32:44.117708+0000","last_fullsized":"2024-03-28T23:32:44.117708+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:17:56.623866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164362+0000","last_change":"2024-03-28T23:32:33.053549+0000","last_active":"2024-03-28T23:32:46.164362+0000","last_peered":"2024-03-28T23:32:46.164362+0000","last_clean":"2024-03-28T23:32:46.164362+0000","last_became_active":"2024-03-28T23:32:33.053229+0000","last_became_peered":"2024-03-28T23:32:33.053229+0000","last_unstale":"2024-03-28T23:32:46.164362+0000","last_undegraded":"2024-03-28T23:32:46.164362+0000","last_fullsized":"2024-03-28T23:32:46.164362+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:41:02.517121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"72'1","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165264+0000","last_change":"2024-03-28T23:32:32.038261+0000","last_active":"2024-03-28T23:32:46.165264+0000","last_peered":"2024-03-28T23:32:46.165264+0000","last_clean":"2024-03-28T23:32:46.165264+0000","last_became_active":"2024-03-28T23:32:32.037971+0000","last_became_peered":"2024-03-28T23:32:32.037971+0000","last_unstale":"2024-03-28T23:32:46.165264+0000","last_undegraded":"2024-03-28T23:32:46.165264+0000","last_fullsized":"2024-03-28T23:32:46.165264+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:25:10.296139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.109933+0000","last_change":"2024-03-28T23:32:44.109933+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.109933+0000","last_undegraded":"2024-03-28T23:32:44.109933+0000","last_fullsized":"2024-03-28T23:32:44.109933+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164276+0000","last_change":"2024-03-28T23:32:44.115062+0000","last_active":"2024-03-28T23:32:46.164276+0000","last_peered":"2024-03-28T23:32:46.164276+0000","last_clean":"2024-03-28T23:32:46.164276+0000","last_became_active":"2024-03-28T23:32:44.114795+0000","last_became_peered":"2024-03-28T23:32:44.114795+0000","last_unstale":"2024-03-28T23:32:46.164276+0000","last_undegraded":"2024-03-28T23:32:46.164276+0000","last_fullsized":"2024-03-28T23:32:46.164276+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:20:13.611155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164170+0000","last_change":"2024-03-28T23:32:33.043954+0000","last_active":"2024-03-28T23:32:46.164170+0000","last_peered":"2024-03-28T23:32:46.164170+0000","last_clean":"2024-03-28T23:32:46.164170+0000","last_became_active":"2024-03-28T23:32:33.043649+0000","last_became_peered":"2024-03-28T23:32:33.043649+0000","last_unstale":"2024-03-28T23:32:46.164170+0000","last_undegraded":"2024-03-28T23:32:46.164170+0000","last_fullsized":"2024-03-28T23:32:46.164170+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:00.358180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109556+0000","last_change":"2024-03-28T23:32:32.033720+0000","last_active":"2024-03-28T23:32:44.109556+0000","last_peered":"2024-03-28T23:32:44.109556+0000","last_clean":"2024-03-28T23:32:44.109556+0000","last_became_active":"2024-03-28T23:32:32.033422+0000","last_became_peered":"2024-03-28T23:32:32.033422+0000","last_unstale":"2024-03-28T23:32:44.109556+0000","last_undegraded":"2024-03-28T23:32:44.109556+0000","last_fullsized":"2024-03-28T23:32:44.109556+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:51:10.440109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.113185+0000","last_change":"2024-03-28T23:32:44.113185+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.113185+0000","last_undegraded":"2024-03-28T23:32:44.113185+0000","last_fullsized":"2024-03-28T23:32:44.113185+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.124927+0000","last_change":"2024-03-28T23:32:44.124559+0000","last_active":"2024-03-28T23:32:44.124927+0000","last_peered":"2024-03-28T23:32:44.124927+0000","last_clean":"2024-03-28T23:32:44.124927+0000","last_became_active":"2024-03-28T23:32:44.124282+0000","last_became_peered":"2024-03-28T23:32:44.124282+0000","last_unstale":"2024-03-28T23:32:44.124927+0000","last_undegraded":"2024-03-28T23:32:44.124927+0000","last_fullsized":"2024-03-28T23:32:44.124927+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:32.542845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165547+0000","last_change":"2024-03-28T23:32:33.045618+0000","last_active":"2024-03-28T23:32:46.165547+0000","last_peered":"2024-03-28T23:32:46.165547+0000","last_clean":"2024-03-28T23:32:46.165547+0000","last_became_active":"2024-03-28T23:32:33.045324+0000","last_became_peered":"2024-03-28T23:32:33.045324+0000","last_unstale":"2024-03-28T23:32:46.165547+0000","last_undegraded":"2024-03-28T23:32:46.165547+0000","last_fullsized":"2024-03-28T23:32:46.165547+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:46:43.644886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"72'2","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164337+0000","last_change":"2024-03-28T23:32:32.031574+0000","last_active":"2024-03-28T23:32:46.164337+0000","last_peered":"2024-03-28T23:32:46.164337+0000","last_clean":"2024-03-28T23:32:46.164337+0000","last_became_active":"2024-03-28T23:32:32.031310+0000","last_became_peered":"2024-03-28T23:32:32.031310+0000","last_unstale":"2024-03-28T23:32:46.164337+0000","last_undegraded":"2024-03-28T23:32:46.164337+0000","last_fullsized":"2024-03-28T23:32:46.164337+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:48:07.073481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165456+0000","last_change":"2024-03-28T23:32:45.130313+0000","last_active":"2024-03-28T23:32:46.165456+0000","last_peered":"2024-03-28T23:32:46.165456+0000","last_clean":"2024-03-28T23:32:46.165456+0000","last_became_active":"2024-03-28T23:32:45.130067+0000","last_became_peered":"2024-03-28T23:32:45.130067+0000","last_unstale":"2024-03-28T23:32:46.165456+0000","last_undegraded":"2024-03-28T23:32:46.165456+0000","last_fullsized":"2024-03-28T23:32:46.165456+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:52:23.046683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.115233+0000","last_change":"2024-03-28T23:32:44.114144+0000","last_active":"2024-03-28T23:32:44.115233+0000","last_peered":"2024-03-28T23:32:44.115233+0000","last_clean":"2024-03-28T23:32:44.115233+0000","last_became_active":"2024-03-28T23:32:44.113829+0000","last_became_peered":"2024-03-28T23:32:44.113829+0000","last_unstale":"2024-03-28T23:32:44.115233+0000","last_undegraded":"2024-03-28T23:32:44.115233+0000","last_fullsized":"2024-03-28T23:32:44.115233+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:55.159548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165916+0000","last_change":"2024-03-28T23:32:33.040983+0000","last_active":"2024-03-28T23:32:46.165916+0000","last_peered":"2024-03-28T23:32:46.165916+0000","last_clean":"2024-03-28T23:32:46.165916+0000","last_became_active":"2024-03-28T23:32:33.040693+0000","last_became_peered":"2024-03-28T23:32:33.040693+0000","last_unstale":"2024-03-28T23:32:46.165916+0000","last_undegraded":"2024-03-28T23:32:46.165916+0000","last_fullsized":"2024-03-28T23:32:46.165916+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:22:05.201764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165375+0000","last_change":"2024-03-28T23:32:32.036998+0000","last_active":"2024-03-28T23:32:46.165375+0000","last_peered":"2024-03-28T23:32:46.165375+0000","last_clean":"2024-03-28T23:32:46.165375+0000","last_became_active":"2024-03-28T23:32:32.036733+0000","last_became_peered":"2024-03-28T23:32:32.036733+0000","last_unstale":"2024-03-28T23:32:46.165375+0000","last_undegraded":"2024-03-28T23:32:46.165375+0000","last_fullsized":"2024-03-28T23:32:46.165375+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:42.334479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.105630+0000","last_change":"2024-03-28T23:32:44.105630+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.105630+0000","last_undegraded":"2024-03-28T23:32:44.105630+0000","last_fullsized":"2024-03-28T23:32:44.105630+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.115890+0000","last_change":"2024-03-28T23:32:44.115154+0000","last_active":"2024-03-28T23:32:44.115890+0000","last_peered":"2024-03-28T23:32:44.115890+0000","last_clean":"2024-03-28T23:32:44.115890+0000","last_became_active":"2024-03-28T23:32:44.114869+0000","last_became_peered":"2024-03-28T23:32:44.114869+0000","last_unstale":"2024-03-28T23:32:44.115890+0000","last_undegraded":"2024-03-28T23:32:44.115890+0000","last_fullsized":"2024-03-28T23:32:44.115890+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:58:59.309973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.113614+0000","last_change":"2024-03-28T23:32:33.041232+0000","last_active":"2024-03-28T23:32:44.113614+0000","last_peered":"2024-03-28T23:32:44.113614+0000","last_clean":"2024-03-28T23:32:44.113614+0000","last_became_active":"2024-03-28T23:32:33.040132+0000","last_became_peered":"2024-03-28T23:32:33.040132+0000","last_unstale":"2024-03-28T23:32:44.113614+0000","last_undegraded":"2024-03-28T23:32:44.113614+0000","last_fullsized":"2024-03-28T23:32:44.113614+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:39:02.824323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164650+0000","last_change":"2024-03-28T23:32:32.036416+0000","last_active":"2024-03-28T23:32:46.164650+0000","last_peered":"2024-03-28T23:32:46.164650+0000","last_clean":"2024-03-28T23:32:46.164650+0000","last_became_active":"2024-03-28T23:32:32.036131+0000","last_became_peered":"2024-03-28T23:32:32.036131+0000","last_unstale":"2024-03-28T23:32:46.164650+0000","last_undegraded":"2024-03-28T23:32:46.164650+0000","last_fullsized":"2024-03-28T23:32:46.164650+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:13:33.914342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.109522+0000","last_change":"2024-03-28T23:32:44.109522+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.109522+0000","last_undegraded":"2024-03-28T23:32:44.109522+0000","last_fullsized":"2024-03-28T23:32:44.109522+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.119994+0000","last_change":"2024-03-28T23:32:44.116820+0000","last_active":"2024-03-28T23:32:44.119994+0000","last_peered":"2024-03-28T23:32:44.119994+0000","last_clean":"2024-03-28T23:32:44.119994+0000","last_became_active":"2024-03-28T23:32:44.116540+0000","last_became_peered":"2024-03-28T23:32:44.116540+0000","last_unstale":"2024-03-28T23:32:44.119994+0000","last_undegraded":"2024-03-28T23:32:44.119994+0000","last_fullsized":"2024-03-28T23:32:44.119994+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:58.429743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.110759+0000","last_change":"2024-03-28T23:32:33.035015+0000","last_active":"2024-03-28T23:32:44.110759+0000","last_peered":"2024-03-28T23:32:44.110759+0000","last_clean":"2024-03-28T23:32:44.110759+0000","last_became_active":"2024-03-28T23:32:33.034730+0000","last_became_peered":"2024-03-28T23:32:33.034730+0000","last_unstale":"2024-03-28T23:32:44.110759+0000","last_undegraded":"2024-03-28T23:32:44.110759+0000","last_fullsized":"2024-03-28T23:32:44.110759+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:09.477928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"72'2","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108295+0000","last_change":"2024-03-28T23:32:32.032596+0000","last_active":"2024-03-28T23:32:44.108295+0000","last_peered":"2024-03-28T23:32:44.108295+0000","last_clean":"2024-03-28T23:32:44.108295+0000","last_became_active":"2024-03-28T23:32:32.032336+0000","last_became_peered":"2024-03-28T23:32:32.032336+0000","last_unstale":"2024-03-28T23:32:44.108295+0000","last_undegraded":"2024-03-28T23:32:44.108295+0000","last_fullsized":"2024-03-28T23:32:44.108295+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:28:25.356704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164807+0000","last_change":"2024-03-28T23:32:45.123614+0000","last_active":"2024-03-28T23:32:46.164807+0000","last_peered":"2024-03-28T23:32:46.164807+0000","last_clean":"2024-03-28T23:32:46.164807+0000","last_became_active":"2024-03-28T23:32:45.123303+0000","last_became_peered":"2024-03-28T23:32:45.123303+0000","last_unstale":"2024-03-28T23:32:46.164807+0000","last_undegraded":"2024-03-28T23:32:46.164807+0000","last_fullsized":"2024-03-28T23:32:46.164807+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:24.670526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.117955+0000","last_change":"2024-03-28T23:32:44.116649+0000","last_active":"2024-03-28T23:32:44.117955+0000","last_peered":"2024-03-28T23:32:44.117955+0000","last_clean":"2024-03-28T23:32:44.117955+0000","last_became_active":"2024-03-28T23:32:44.116369+0000","last_became_peered":"2024-03-28T23:32:44.116369+0000","last_unstale":"2024-03-28T23:32:44.117955+0000","last_undegraded":"2024-03-28T23:32:44.117955+0000","last_fullsized":"2024-03-28T23:32:44.117955+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:46:54.723611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.111340+0000","last_change":"2024-03-28T23:32:33.044243+0000","last_active":"2024-03-28T23:32:44.111340+0000","last_peered":"2024-03-28T23:32:44.111340+0000","last_clean":"2024-03-28T23:32:44.111340+0000","last_became_active":"2024-03-28T23:32:33.040218+0000","last_became_peered":"2024-03-28T23:32:33.040218+0000","last_unstale":"2024-03-28T23:32:44.111340+0000","last_undegraded":"2024-03-28T23:32:44.111340+0000","last_fullsized":"2024-03-28T23:32:44.111340+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:38:51.513907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"73'1","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166004+0000","last_change":"2024-03-28T23:32:32.041586+0000","last_active":"2024-03-28T23:32:46.166004+0000","last_peered":"2024-03-28T23:32:46.166004+0000","last_clean":"2024-03-28T23:32:46.166004+0000","last_became_active":"2024-03-28T23:32:32.041327+0000","last_became_peered":"2024-03-28T23:32:32.041327+0000","last_unstale":"2024-03-28T23:32:46.166004+0000","last_undegraded":"2024-03-28T23:32:46.166004+0000","last_fullsized":"2024-03-28T23:32:46.166004+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:06:20.608980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165537+0000","last_change":"2024-03-28T23:32:45.134577+0000","last_active":"2024-03-28T23:32:46.165537+0000","last_peered":"2024-03-28T23:32:46.165537+0000","last_clean":"2024-03-28T23:32:46.165537+0000","last_became_active":"2024-03-28T23:32:45.134292+0000","last_became_peered":"2024-03-28T23:32:45.134292+0000","last_unstale":"2024-03-28T23:32:46.165537+0000","last_undegraded":"2024-03-28T23:32:46.165537+0000","last_fullsized":"2024-03-28T23:32:46.165537+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:41:13.065560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164952+0000","last_change":"2024-03-28T23:32:44.125883+0000","last_active":"2024-03-28T23:32:46.164952+0000","last_peered":"2024-03-28T23:32:46.164952+0000","last_clean":"2024-03-28T23:32:46.164952+0000","last_became_active":"2024-03-28T23:32:44.125578+0000","last_became_peered":"2024-03-28T23:32:44.125578+0000","last_unstale":"2024-03-28T23:32:46.164952+0000","last_undegraded":"2024-03-28T23:32:46.164952+0000","last_fullsized":"2024-03-28T23:32:46.164952+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:38:36.566643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112455+0000","last_change":"2024-03-28T23:32:33.042272+0000","last_active":"2024-03-28T23:32:44.112455+0000","last_peered":"2024-03-28T23:32:44.112455+0000","last_clean":"2024-03-28T23:32:44.112455+0000","last_became_active":"2024-03-28T23:32:33.037633+0000","last_became_peered":"2024-03-28T23:32:33.037633+0000","last_unstale":"2024-03-28T23:32:44.112455+0000","last_undegraded":"2024-03-28T23:32:44.112455+0000","last_fullsized":"2024-03-28T23:32:44.112455+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:45.773838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108173+0000","last_change":"2024-03-28T23:32:32.037001+0000","last_active":"2024-03-28T23:32:44.108173+0000","last_peered":"2024-03-28T23:32:44.108173+0000","last_clean":"2024-03-28T23:32:44.108173+0000","last_became_active":"2024-03-28T23:32:32.036739+0000","last_became_peered":"2024-03-28T23:32:32.036739+0000","last_unstale":"2024-03-28T23:32:44.108173+0000","last_undegraded":"2024-03-28T23:32:44.108173+0000","last_fullsized":"2024-03-28T23:32:44.108173+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:06:20.608980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165291+0000","last_change":"2024-03-28T23:32:45.126588+0000","last_active":"2024-03-28T23:32:46.165291+0000","last_peered":"2024-03-28T23:32:46.165291+0000","last_clean":"2024-03-28T23:32:46.165291+0000","last_became_active":"2024-03-28T23:32:45.126348+0000","last_became_peered":"2024-03-28T23:32:45.126348+0000","last_unstale":"2024-03-28T23:32:46.165291+0000","last_undegraded":"2024-03-28T23:32:46.165291+0000","last_fullsized":"2024-03-28T23:32:46.165291+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:00:34.558942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.117952+0000","last_change":"2024-03-28T23:32:44.116971+0000","last_active":"2024-03-28T23:32:44.117952+0000","last_peered":"2024-03-28T23:32:44.117952+0000","last_clean":"2024-03-28T23:32:44.117952+0000","last_became_active":"2024-03-28T23:32:44.116371+0000","last_became_peered":"2024-03-28T23:32:44.116371+0000","last_unstale":"2024-03-28T23:32:44.117952+0000","last_undegraded":"2024-03-28T23:32:44.117952+0000","last_fullsized":"2024-03-28T23:32:44.117952+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:43:47.025717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165286+0000","last_change":"2024-03-28T23:32:33.041123+0000","last_active":"2024-03-28T23:32:46.165286+0000","last_peered":"2024-03-28T23:32:46.165286+0000","last_clean":"2024-03-28T23:32:46.165286+0000","last_became_active":"2024-03-28T23:32:33.040505+0000","last_became_peered":"2024-03-28T23:32:33.040505+0000","last_unstale":"2024-03-28T23:32:46.165286+0000","last_undegraded":"2024-03-28T23:32:46.165286+0000","last_fullsized":"2024-03-28T23:32:46.165286+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:54.194453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.106365+0000","last_change":"2024-03-28T23:32:32.036857+0000","last_active":"2024-03-28T23:32:44.106365+0000","last_peered":"2024-03-28T23:32:44.106365+0000","last_clean":"2024-03-28T23:32:44.106365+0000","last_became_active":"2024-03-28T23:32:32.036576+0000","last_became_peered":"2024-03-28T23:32:32.036576+0000","last_unstale":"2024-03-28T23:32:44.106365+0000","last_undegraded":"2024-03-28T23:32:44.106365+0000","last_fullsized":"2024-03-28T23:32:44.106365+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:43.073425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165569+0000","last_change":"2024-03-28T23:32:45.133481+0000","last_active":"2024-03-28T23:32:46.165569+0000","last_peered":"2024-03-28T23:32:46.165569+0000","last_clean":"2024-03-28T23:32:46.165569+0000","last_became_active":"2024-03-28T23:32:45.132954+0000","last_became_peered":"2024-03-28T23:32:45.132954+0000","last_unstale":"2024-03-28T23:32:46.165569+0000","last_undegraded":"2024-03-28T23:32:46.165569+0000","last_fullsized":"2024-03-28T23:32:46.165569+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:25.413469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.127553+0000","last_change":"2024-03-28T23:32:44.126706+0000","last_active":"2024-03-28T23:32:44.127553+0000","last_peered":"2024-03-28T23:32:44.127553+0000","last_clean":"2024-03-28T23:32:44.127553+0000","last_became_active":"2024-03-28T23:32:44.126400+0000","last_became_peered":"2024-03-28T23:32:44.126400+0000","last_unstale":"2024-03-28T23:32:44.127553+0000","last_undegraded":"2024-03-28T23:32:44.127553+0000","last_fullsized":"2024-03-28T23:32:44.127553+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:00.790475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112670+0000","last_change":"2024-03-28T23:32:33.041769+0000","last_active":"2024-03-28T23:32:44.112670+0000","last_peered":"2024-03-28T23:32:44.112670+0000","last_clean":"2024-03-28T23:32:44.112670+0000","last_became_active":"2024-03-28T23:32:33.041475+0000","last_became_peered":"2024-03-28T23:32:33.041475+0000","last_unstale":"2024-03-28T23:32:44.112670+0000","last_undegraded":"2024-03-28T23:32:44.112670+0000","last_fullsized":"2024-03-28T23:32:44.112670+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:38.859031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112270+0000","last_change":"2024-03-28T23:32:32.037824+0000","last_active":"2024-03-28T23:32:44.112270+0000","last_peered":"2024-03-28T23:32:44.112270+0000","last_clean":"2024-03-28T23:32:44.112270+0000","last_became_active":"2024-03-28T23:32:32.037550+0000","last_became_peered":"2024-03-28T23:32:32.037550+0000","last_unstale":"2024-03-28T23:32:44.112270+0000","last_undegraded":"2024-03-28T23:32:44.112270+0000","last_fullsized":"2024-03-28T23:32:44.112270+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:43.467214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.111817+0000","last_change":"2024-03-28T23:32:44.111817+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.111817+0000","last_undegraded":"2024-03-28T23:32:44.111817+0000","last_fullsized":"2024-03-28T23:32:44.111817+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.119913+0000","last_change":"2024-03-28T23:32:44.117095+0000","last_active":"2024-03-28T23:32:44.119913+0000","last_peered":"2024-03-28T23:32:44.119913+0000","last_clean":"2024-03-28T23:32:44.119913+0000","last_became_active":"2024-03-28T23:32:44.116824+0000","last_became_peered":"2024-03-28T23:32:44.116824+0000","last_unstale":"2024-03-28T23:32:44.119913+0000","last_undegraded":"2024-03-28T23:32:44.119913+0000","last_fullsized":"2024-03-28T23:32:44.119913+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:59:07.631069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.112978+0000","last_change":"2024-03-28T23:32:33.048966+0000","last_active":"2024-03-28T23:32:44.112978+0000","last_peered":"2024-03-28T23:32:44.112978+0000","last_clean":"2024-03-28T23:32:44.112978+0000","last_became_active":"2024-03-28T23:32:33.048362+0000","last_became_peered":"2024-03-28T23:32:33.048362+0000","last_unstale":"2024-03-28T23:32:44.112978+0000","last_undegraded":"2024-03-28T23:32:44.112978+0000","last_fullsized":"2024-03-28T23:32:44.112978+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:25.849919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164445+0000","last_change":"2024-03-28T23:32:32.038885+0000","last_active":"2024-03-28T23:32:46.164445+0000","last_peered":"2024-03-28T23:32:46.164445+0000","last_clean":"2024-03-28T23:32:46.164445+0000","last_became_active":"2024-03-28T23:32:32.038364+0000","last_became_peered":"2024-03-28T23:32:32.038364+0000","last_unstale":"2024-03-28T23:32:46.164445+0000","last_undegraded":"2024-03-28T23:32:46.164445+0000","last_fullsized":"2024-03-28T23:32:46.164445+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:16:06.156877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.105630+0000","last_change":"2024-03-28T23:32:44.105630+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.105630+0000","last_undegraded":"2024-03-28T23:32:44.105630+0000","last_fullsized":"2024-03-28T23:32:44.105630+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.115613+0000","last_change":"2024-03-28T23:32:44.114483+0000","last_active":"2024-03-28T23:32:44.115613+0000","last_peered":"2024-03-28T23:32:44.115613+0000","last_clean":"2024-03-28T23:32:44.115613+0000","last_became_active":"2024-03-28T23:32:44.114228+0000","last_became_peered":"2024-03-28T23:32:44.114228+0000","last_unstale":"2024-03-28T23:32:44.115613+0000","last_undegraded":"2024-03-28T23:32:44.115613+0000","last_fullsized":"2024-03-28T23:32:44.115613+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:56:05.273178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.107950+0000","last_change":"2024-03-28T23:32:33.049584+0000","last_active":"2024-03-28T23:32:44.107950+0000","last_peered":"2024-03-28T23:32:44.107950+0000","last_clean":"2024-03-28T23:32:44.107950+0000","last_became_active":"2024-03-28T23:32:33.049342+0000","last_became_peered":"2024-03-28T23:32:33.049342+0000","last_unstale":"2024-03-28T23:32:44.107950+0000","last_undegraded":"2024-03-28T23:32:44.107950+0000","last_fullsized":"2024-03-28T23:32:44.107950+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:49.711750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.111513+0000","last_change":"2024-03-28T23:32:32.034044+0000","last_active":"2024-03-28T23:32:44.111513+0000","last_peered":"2024-03-28T23:32:44.111513+0000","last_clean":"2024-03-28T23:32:44.111513+0000","last_became_active":"2024-03-28T23:32:32.033779+0000","last_became_peered":"2024-03-28T23:32:32.033779+0000","last_unstale":"2024-03-28T23:32:44.111513+0000","last_undegraded":"2024-03-28T23:32:44.111513+0000","last_fullsized":"2024-03-28T23:32:44.111513+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:56:22.159708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164295+0000","last_change":"2024-03-28T23:32:45.130071+0000","last_active":"2024-03-28T23:32:46.164295+0000","last_peered":"2024-03-28T23:32:46.164295+0000","last_clean":"2024-03-28T23:32:46.164295+0000","last_became_active":"2024-03-28T23:32:45.129682+0000","last_became_peered":"2024-03-28T23:32:45.129682+0000","last_unstale":"2024-03-28T23:32:46.164295+0000","last_undegraded":"2024-03-28T23:32:46.164295+0000","last_fullsized":"2024-03-28T23:32:46.164295+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:18:52.487805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.120835+0000","last_change":"2024-03-28T23:32:44.119649+0000","last_active":"2024-03-28T23:32:44.120835+0000","last_peered":"2024-03-28T23:32:44.120835+0000","last_clean":"2024-03-28T23:32:44.120835+0000","last_became_active":"2024-03-28T23:32:44.119370+0000","last_became_peered":"2024-03-28T23:32:44.119370+0000","last_unstale":"2024-03-28T23:32:44.120835+0000","last_undegraded":"2024-03-28T23:32:44.120835+0000","last_fullsized":"2024-03-28T23:32:44.120835+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:25:56.268584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109652+0000","last_change":"2024-03-28T23:32:33.041180+0000","last_active":"2024-03-28T23:32:44.109652+0000","last_peered":"2024-03-28T23:32:44.109652+0000","last_clean":"2024-03-28T23:32:44.109652+0000","last_became_active":"2024-03-28T23:32:33.040922+0000","last_became_peered":"2024-03-28T23:32:33.040922+0000","last_unstale":"2024-03-28T23:32:44.109652+0000","last_undegraded":"2024-03-28T23:32:44.109652+0000","last_fullsized":"2024-03-28T23:32:44.109652+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:25:45.189859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109480+0000","last_change":"2024-03-28T23:32:32.034328+0000","last_active":"2024-03-28T23:32:44.109480+0000","last_peered":"2024-03-28T23:32:44.109480+0000","last_clean":"2024-03-28T23:32:44.109480+0000","last_became_active":"2024-03-28T23:32:32.034057+0000","last_became_peered":"2024-03-28T23:32:32.034057+0000","last_unstale":"2024-03-28T23:32:44.109480+0000","last_undegraded":"2024-03-28T23:32:44.109480+0000","last_fullsized":"2024-03-28T23:32:44.109480+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:54:12.775407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":75,"state":"creating+peering","last_fresh":"2024-03-28T23:32:44.107591+0000","last_change":"2024-03-28T23:32:44.107591+0000","last_active":"2024-03-28T23:32:44.098119+0000","last_peered":"2024-03-28T23:32:44.098119+0000","last_clean":"2024-03-28T23:32:44.098119+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:32:44.107591+0000","last_undegraded":"2024-03-28T23:32:44.107591+0000","last_fullsized":"2024-03-28T23:32:44.107591+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.119109+0000","last_change":"2024-03-28T23:32:44.115020+0000","last_active":"2024-03-28T23:32:44.119109+0000","last_peered":"2024-03-28T23:32:44.119109+0000","last_clean":"2024-03-28T23:32:44.119109+0000","last_became_active":"2024-03-28T23:32:44.114737+0000","last_became_peered":"2024-03-28T23:32:44.114737+0000","last_unstale":"2024-03-28T23:32:44.119109+0000","last_undegraded":"2024-03-28T23:32:44.119109+0000","last_fullsized":"2024-03-28T23:32:44.119109+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:13:16.820792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.108150+0000","last_change":"2024-03-28T23:32:33.038830+0000","last_active":"2024-03-28T23:32:44.108150+0000","last_peered":"2024-03-28T23:32:44.108150+0000","last_clean":"2024-03-28T23:32:44.108150+0000","last_became_active":"2024-03-28T23:32:33.038524+0000","last_became_peered":"2024-03-28T23:32:33.038524+0000","last_unstale":"2024-03-28T23:32:44.108150+0000","last_undegraded":"2024-03-28T23:32:44.108150+0000","last_fullsized":"2024-03-28T23:32:44.108150+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:55:53.512836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166249+0000","last_change":"2024-03-28T23:32:32.034270+0000","last_active":"2024-03-28T23:32:46.166249+0000","last_peered":"2024-03-28T23:32:46.166249+0000","last_clean":"2024-03-28T23:32:46.166249+0000","last_became_active":"2024-03-28T23:32:32.034022+0000","last_became_peered":"2024-03-28T23:32:32.034022+0000","last_unstale":"2024-03-28T23:32:46.166249+0000","last_undegraded":"2024-03-28T23:32:46.166249+0000","last_fullsized":"2024-03-28T23:32:46.166249+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:15.683500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":273,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-03-28T23:32:44.109089+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:32:44.109089+0000","last_peered":"2024-03-28T23:32:44.109089+0000","last_clean":"2024-03-28T23:32:44.109089+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:32:44.109089+0000","last_undegraded":"2024-03-28T23:32:44.109089+0000","last_fullsized":"2024-03-28T23:32:44.109089+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":64,"acting":64,"num_store_stats":8},{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574887,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":974848,"data_stored":1018333,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574888,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32088,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339752,"statfs":{"total":96636764160,"available":96603906048,"internally_reserved":0,"allocated":929792,"data_stored":437077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32068,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339772,"statfs":{"total":96636764160,"available":96603926528,"internally_reserved":0,"allocated":909312,"data_stored":427977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574887,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32084,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339756,"statfs":{"total":96636764160,"available":96603910144,"internally_reserved":0,"allocated":925696,"data_stored":429659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":946176,"data_stored":451621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574888,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":954368,"data_stored":451781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574887,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343928,"statfs":{"total":96636764160,"available":96608182272,"internally_reserved":0,"allocated":913408,"data_stored":428055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574888,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343812,"statfs":{"total":96636764160,"available":96608063488,"internally_reserved":0,"allocated":1032192,"data_stored":1043725,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:32:52.711 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:32:52.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.812+0000 7f4ae94ed640 1 Processor -- start 2024-03-28T23:32:52.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.812+0000 7f4ae94ed640 1 -- start start 2024-03-28T23:32:52.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.813+0000 7f4ae94ed640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4ae406d9f0 0x7f4ae4155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:52.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.813+0000 7f4ae94ed640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ae406c3c0 con 0x7f4ae406cb80 2024-03-28T23:32:52.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.813+0000 7f4ae94ed640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ae40c5b90 con 0x7f4ae4152a80 2024-03-28T23:32:52.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.813+0000 7f4ae94ed640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f4ae4156250 con 0x7f4ae406d9f0 2024-03-28T23:32:52.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.813+0000 7f4ae37fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f4ae406cb80 0x7f4ae40c74c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:55574/0 (socket says 172.21.15.161:55574) 2024-03-28T23:32:52.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.813+0000 7f4ae37fe640 1 -- 172.21.15.161:0/3264021779 learned_addr learned my addr 172.21.15.161:0/3264021779 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:52.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.813+0000 7f4ae27fc640 1 --2- 172.21.15.161:0/3264021779 >> v2:172.21.15.170:3300/0 conn(0x7f4ae406d9f0 0x7f4ae4155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:52.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.814+0000 7f4ae1ffb640 1 -- 172.21.15.161:0/3264021779 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3905327812 0 0) 0x7f4ae406c3c0 con 0x7f4ae406cb80 2024-03-28T23:32:52.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.814+0000 7f4ae1ffb640 1 -- 172.21.15.161:0/3264021779 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4acc003520 con 0x7f4ae406cb80 2024-03-28T23:32:52.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.814+0000 7f4ae1ffb640 1 -- 172.21.15.161:0/3264021779 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2654693371 0 0) 0x7f4ae40c5b90 con 0x7f4ae4152a80 2024-03-28T23:32:52.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.814+0000 7f4ae1ffb640 1 -- 172.21.15.161:0/3264021779 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ae406c3c0 con 0x7f4ae4152a80 2024-03-28T23:32:52.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.814+0000 7f4ae27fc640 1 -- 172.21.15.161:0/3264021779 >> v1:172.21.15.170:6789/0 conn(0x7f4ae4152a80 legacy=0x7f4ae40c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:52.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.814+0000 7f4ae27fc640 1 -- 172.21.15.161:0/3264021779 >> v1:172.21.15.161:6789/0 conn(0x7f4ae406cb80 legacy=0x7f4ae40c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:52.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.814+0000 7f4ae27fc640 1 -- 172.21.15.161:0/3264021779 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ae40c55d0 con 0x7f4ae406d9f0 2024-03-28T23:32:52.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.815+0000 7f4ae27fc640 1 --2- 172.21.15.161:0/3264021779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ae406d9f0 0x7f4ae4155d10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4ad80097a0 tx=0x7f4ad802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93b428f0e3098bb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:52.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.815+0000 7f4ae1ffb640 1 -- 172.21.15.161:0/3264021779 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad80315a0 con 0x7f4ae406d9f0 2024-03-28T23:32:52.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.815+0000 7f4ae1ffb640 1 -- 172.21.15.161:0/3264021779 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ad8037070 con 0x7f4ae406d9f0 2024-03-28T23:32:52.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.815+0000 7f4ae1ffb640 1 -- 172.21.15.161:0/3264021779 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad8031960 con 0x7f4ae406d9f0 2024-03-28T23:32:52.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.815+0000 7f4ae94ed640 1 -- 172.21.15.161:0/3264021779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ae406d9f0 msgr2=0x7f4ae4155d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:52.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.815+0000 7f4ae94ed640 1 --2- 172.21.15.161:0/3264021779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ae406d9f0 0x7f4ae4155d10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4ad80097a0 tx=0x7f4ad802b840 comp rx=0 tx=0).stop 2024-03-28T23:32:52.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.815+0000 7f4ae94ed640 1 -- 172.21.15.161:0/3264021779 shutdown_connections 2024-03-28T23:32:52.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.815+0000 7f4ae94ed640 1 --2- 172.21.15.161:0/3264021779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ae406d9f0 0x7f4ae4155d10 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:52.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.815+0000 7f4ae94ed640 1 -- 172.21.15.161:0/3264021779 >> 172.21.15.161:0/3264021779 conn(0x7f4ae40ac720 msgr2=0x7f4ae40acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:52.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.815+0000 7f4ae94ed640 1 -- 172.21.15.161:0/3264021779 shutdown_connections 2024-03-28T23:32:52.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.815+0000 7f4ae94ed640 1 -- 172.21.15.161:0/3264021779 wait complete. 2024-03-28T23:32:52.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.817+0000 7f4ae94ed640 1 Processor -- start 2024-03-28T23:32:52.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.817+0000 7f4ae94ed640 1 -- start start 2024-03-28T23:32:52.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.818+0000 7f4ae94ed640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ae406cb80 0x7f4ae414deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:52.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.818+0000 7f4ae94ed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ae406d9f0 0x7f4ae41ebee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:52.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.818+0000 7f4ae94ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ae4152a80 0x7f4ae41ec420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:52.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.818+0000 7f4ae94ed640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae40c7bd0 con 0x7f4ae406cb80 2024-03-28T23:32:52.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.818+0000 7f4ae94ed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae406c6f0 con 0x7f4ae406d9f0 2024-03-28T23:32:52.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.818+0000 7f4ae94ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4ae40c7d50 con 0x7f4ae4152a80 2024-03-28T23:32:52.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.818+0000 7f4ae2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ae406cb80 0x7f4ae414deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:52.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.818+0000 7f4ae2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ae406cb80 0x7f4ae414deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51968/0 (socket says 172.21.15.161:51968) 2024-03-28T23:32:52.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.818+0000 7f4ae2ffd640 1 -- 172.21.15.161:0/3841517612 learned_addr learned my addr 172.21.15.161:0/3841517612 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:52.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.818+0000 7f4ae27fc640 1 --2- 172.21.15.161:0/3841517612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ae406d9f0 0x7f4ae41ebee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:52.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.819+0000 7f4ae37fe640 1 --2- 172.21.15.161:0/3841517612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ae4152a80 0x7f4ae41ec420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:52.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.819+0000 7f4ae2ffd640 1 -- 172.21.15.161:0/3841517612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ae4152a80 msgr2=0x7f4ae41ec420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:52.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.819+0000 7f4ae2ffd640 1 --2- 172.21.15.161:0/3841517612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ae4152a80 0x7f4ae41ec420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:52.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.819+0000 7f4ae2ffd640 1 -- 172.21.15.161:0/3841517612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ae406d9f0 msgr2=0x7f4ae41ebee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:52.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.819+0000 7f4ae2ffd640 1 --2- 172.21.15.161:0/3841517612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ae406d9f0 0x7f4ae41ebee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:52.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.819+0000 7f4ae2ffd640 1 -- 172.21.15.161:0/3841517612 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ae40ae950 con 0x7f4ae406cb80 2024-03-28T23:32:52.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.819+0000 7f4ae37fe640 1 --2- 172.21.15.161:0/3841517612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ae4152a80 0x7f4ae41ec420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:32:52.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.819+0000 7f4ae27fc640 1 --2- 172.21.15.161:0/3841517612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ae406d9f0 0x7f4ae41ebee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:32:52.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.819+0000 7f4ae2ffd640 1 --2- 172.21.15.161:0/3841517612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ae406cb80 0x7f4ae414deb0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f4ad400c7d0 tx=0x7f4ad400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:52.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.819+0000 7f4acb7fe640 1 -- 172.21.15.161:0/3841517612 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad4003b70 con 0x7f4ae406cb80 2024-03-28T23:32:52.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.819+0000 7f4acb7fe640 1 -- 172.21.15.161:0/3841517612 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ad4003d30 con 0x7f4ae406cb80 2024-03-28T23:32:52.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.819+0000 7f4acb7fe640 1 -- 172.21.15.161:0/3841517612 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad400aa30 con 0x7f4ae406cb80 2024-03-28T23:32:52.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.819+0000 7f4ae94ed640 1 -- 172.21.15.161:0/3841517612 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ae40aec30 con 0x7f4ae406cb80 2024-03-28T23:32:52.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.819+0000 7f4ae94ed640 1 -- 172.21.15.161:0/3841517612 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ae41ec960 con 0x7f4ae406cb80 2024-03-28T23:32:52.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.820+0000 7f4ae94ed640 1 -- 172.21.15.161:0/3841517612 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ae404f4b0 con 0x7f4ae406cb80 2024-03-28T23:32:52.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.821+0000 7f4acb7fe640 1 -- 172.21.15.161:0/3841517612 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4ad400abd0 con 0x7f4ae406cb80 2024-03-28T23:32:52.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.821+0000 7f4acb7fe640 1 --2- 172.21.15.161:0/3841517612 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4ad006b4d0 0x7f4ad006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:52.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.821+0000 7f4ae27fc640 1 --2- 172.21.15.161:0/3841517612 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4ad006b4d0 0x7f4ad006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:52.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.821+0000 7f4acb7fe640 1 -- 172.21.15.161:0/3841517612 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6957+0+0 (secure 0 0 0) 0x7f4ad408a420 con 0x7f4ae406cb80 2024-03-28T23:32:52.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.822+0000 7f4ae27fc640 1 --2- 172.21.15.161:0/3841517612 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4ad006b4d0 0x7f4ad006d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f4ad8036000 tx=0x7f4ad803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:52.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.826+0000 7f4acb7fe640 1 -- 172.21.15.161:0/3841517612 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4ad4058800 con 0x7f4ae406cb80 2024-03-28T23:32:52.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:52.993+0000 7f4ae94ed640 1 -- 172.21.15.161:0/3841517612 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4ae40c5b90 con 0x7f4ad006b4d0 2024-03-28T23:32:53.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.004+0000 7f4acb7fe640 1 -- 172.21.15.161:0/3841517612 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490718 (secure 0 0 0) 0x7f4ae40c5b90 con 0x7f4ad006b4d0 2024-03-28T23:32:53.009 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:53.012 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:32:53.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.008+0000 7f4ae94ed640 1 -- 172.21.15.161:0/3841517612 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4ad006b4d0 msgr2=0x7f4ad006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:53.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.008+0000 7f4ae94ed640 1 --2- 172.21.15.161:0/3841517612 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4ad006b4d0 0x7f4ad006d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f4ad8036000 tx=0x7f4ad803d040 comp rx=0 tx=0).stop 2024-03-28T23:32:53.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.008+0000 7f4ae94ed640 1 -- 172.21.15.161:0/3841517612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ae406cb80 msgr2=0x7f4ae414deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:53.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.008+0000 7f4ae94ed640 1 --2- 172.21.15.161:0/3841517612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ae406cb80 0x7f4ae414deb0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f4ad400c7d0 tx=0x7f4ad400cc90 comp rx=0 tx=0).stop 2024-03-28T23:32:53.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.008+0000 7f4ae94ed640 1 -- 172.21.15.161:0/3841517612 shutdown_connections 2024-03-28T23:32:53.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.008+0000 7f4ae94ed640 1 --2- 172.21.15.161:0/3841517612 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4ad006b4d0 0x7f4ad006d990 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:53.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.008+0000 7f4ae94ed640 1 --2- 172.21.15.161:0/3841517612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ae4152a80 0x7f4ae41ec420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:53.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.008+0000 7f4ae94ed640 1 --2- 172.21.15.161:0/3841517612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ae406d9f0 0x7f4ae41ebee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:53.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.008+0000 7f4ae94ed640 1 --2- 172.21.15.161:0/3841517612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ae406cb80 0x7f4ae414deb0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:53.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.008+0000 7f4ae94ed640 1 -- 172.21.15.161:0/3841517612 >> 172.21.15.161:0/3841517612 conn(0x7f4ae40ac720 msgr2=0x7f4ae40ab990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:53.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.008+0000 7f4ae94ed640 1 -- 172.21.15.161:0/3841517612 shutdown_connections 2024-03-28T23:32:53.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.008+0000 7f4ae94ed640 1 -- 172.21.15.161:0/3841517612 wait complete. 2024-03-28T23:32:53.019 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2024-03-28T23:32:51.346702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":370,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240460,"kb_used_data":7616,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754734260,"statfs":{"total":773094113280,"available":772847882240,"internally_reserved":0,"allocated":7798784,"data_stored":4798688,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.243522"},"pg_stats":[{"pgid":"14.3d","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164968+0000","last_change":"2024-03-28T23:32:45.124812+0000","last_active":"2024-03-28T23:32:46.164968+0000","last_peered":"2024-03-28T23:32:46.164968+0000","last_clean":"2024-03-28T23:32:46.164968+0000","last_became_active":"2024-03-28T23:32:45.124589+0000","last_became_peered":"2024-03-28T23:32:45.124589+0000","last_unstale":"2024-03-28T23:32:46.164968+0000","last_undegraded":"2024-03-28T23:32:46.164968+0000","last_fullsized":"2024-03-28T23:32:46.164968+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:35:54.086668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3f","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164164+0000","last_change":"2024-03-28T23:32:33.046706+0000","last_active":"2024-03-28T23:32:46.164164+0000","last_peered":"2024-03-28T23:32:46.164164+0000","last_clean":"2024-03-28T23:32:46.164164+0000","last_became_active":"2024-03-28T23:32:33.046222+0000","last_became_peered":"2024-03-28T23:32:33.046222+0000","last_unstale":"2024-03-28T23:32:46.164164+0000","last_undegraded":"2024-03-28T23:32:46.164164+0000","last_fullsized":"2024-03-28T23:32:46.164164+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:02:45.190566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166728+0000","last_change":"2024-03-28T23:32:45.124966+0000","last_active":"2024-03-28T23:32:46.166728+0000","last_peered":"2024-03-28T23:32:46.166728+0000","last_clean":"2024-03-28T23:32:46.166728+0000","last_became_active":"2024-03-28T23:32:45.124677+0000","last_became_peered":"2024-03-28T23:32:45.124677+0000","last_unstale":"2024-03-28T23:32:46.166728+0000","last_undegraded":"2024-03-28T23:32:46.166728+0000","last_fullsized":"2024-03-28T23:32:46.166728+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:45:10.868624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165821+0000","last_change":"2024-03-28T23:32:33.051174+0000","last_active":"2024-03-28T23:32:46.165821+0000","last_peered":"2024-03-28T23:32:46.165821+0000","last_clean":"2024-03-28T23:32:46.165821+0000","last_became_active":"2024-03-28T23:32:33.050345+0000","last_became_peered":"2024-03-28T23:32:33.050345+0000","last_unstale":"2024-03-28T23:32:46.165821+0000","last_undegraded":"2024-03-28T23:32:46.165821+0000","last_fullsized":"2024-03-28T23:32:46.165821+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:41:39.657731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.3f","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166718+0000","last_change":"2024-03-28T23:32:45.127552+0000","last_active":"2024-03-28T23:32:46.166718+0000","last_peered":"2024-03-28T23:32:46.166718+0000","last_clean":"2024-03-28T23:32:46.166718+0000","last_became_active":"2024-03-28T23:32:45.127291+0000","last_became_peered":"2024-03-28T23:32:45.127291+0000","last_unstale":"2024-03-28T23:32:46.166718+0000","last_undegraded":"2024-03-28T23:32:46.166718+0000","last_fullsized":"2024-03-28T23:32:46.166718+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:21:07.618142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164595+0000","last_change":"2024-03-28T23:32:33.048785+0000","last_active":"2024-03-28T23:32:46.164595+0000","last_peered":"2024-03-28T23:32:46.164595+0000","last_clean":"2024-03-28T23:32:46.164595+0000","last_became_active":"2024-03-28T23:32:33.048519+0000","last_became_peered":"2024-03-28T23:32:33.048519+0000","last_unstale":"2024-03-28T23:32:46.164595+0000","last_undegraded":"2024-03-28T23:32:46.164595+0000","last_fullsized":"2024-03-28T23:32:46.164595+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:21.464120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.123743+0000","last_change":"2024-03-28T23:32:45.131200+0000","last_active":"2024-03-28T23:32:46.123743+0000","last_peered":"2024-03-28T23:32:46.123743+0000","last_clean":"2024-03-28T23:32:46.123743+0000","last_became_active":"2024-03-28T23:32:45.130931+0000","last_became_peered":"2024-03-28T23:32:45.130931+0000","last_unstale":"2024-03-28T23:32:46.123743+0000","last_undegraded":"2024-03-28T23:32:46.123743+0000","last_fullsized":"2024-03-28T23:32:46.123743+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:03:02.136678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.124352+0000","last_change":"2024-03-28T23:32:33.033535+0000","last_active":"2024-03-28T23:32:46.124352+0000","last_peered":"2024-03-28T23:32:46.124352+0000","last_clean":"2024-03-28T23:32:46.124352+0000","last_became_active":"2024-03-28T23:32:33.033219+0000","last_became_peered":"2024-03-28T23:32:33.033219+0000","last_unstale":"2024-03-28T23:32:46.124352+0000","last_undegraded":"2024-03-28T23:32:46.124352+0000","last_fullsized":"2024-03-28T23:32:46.124352+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:39:39.113457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163485+0000","last_change":"2024-03-28T23:32:45.117071+0000","last_active":"2024-03-28T23:32:46.163485+0000","last_peered":"2024-03-28T23:32:46.163485+0000","last_clean":"2024-03-28T23:32:46.163485+0000","last_became_active":"2024-03-28T23:32:45.116811+0000","last_became_peered":"2024-03-28T23:32:45.116811+0000","last_unstale":"2024-03-28T23:32:46.163485+0000","last_undegraded":"2024-03-28T23:32:46.163485+0000","last_fullsized":"2024-03-28T23:32:46.163485+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:49.881731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166903+0000","last_change":"2024-03-28T23:32:33.042972+0000","last_active":"2024-03-28T23:32:46.166903+0000","last_peered":"2024-03-28T23:32:46.166903+0000","last_clean":"2024-03-28T23:32:46.166903+0000","last_became_active":"2024-03-28T23:32:33.042665+0000","last_became_peered":"2024-03-28T23:32:33.042665+0000","last_unstale":"2024-03-28T23:32:46.166903+0000","last_undegraded":"2024-03-28T23:32:46.166903+0000","last_fullsized":"2024-03-28T23:32:46.166903+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:41:50.190883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165253+0000","last_change":"2024-03-28T23:32:45.128743+0000","last_active":"2024-03-28T23:32:46.165253+0000","last_peered":"2024-03-28T23:32:46.165253+0000","last_clean":"2024-03-28T23:32:46.165253+0000","last_became_active":"2024-03-28T23:32:45.123494+0000","last_became_peered":"2024-03-28T23:32:45.123494+0000","last_unstale":"2024-03-28T23:32:46.165253+0000","last_undegraded":"2024-03-28T23:32:46.165253+0000","last_fullsized":"2024-03-28T23:32:46.165253+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:32:20.557728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165883+0000","last_change":"2024-03-28T23:32:33.030549+0000","last_active":"2024-03-28T23:32:46.165883+0000","last_peered":"2024-03-28T23:32:46.165883+0000","last_clean":"2024-03-28T23:32:46.165883+0000","last_became_active":"2024-03-28T23:32:33.030299+0000","last_became_peered":"2024-03-28T23:32:33.030299+0000","last_unstale":"2024-03-28T23:32:46.165883+0000","last_undegraded":"2024-03-28T23:32:46.165883+0000","last_fullsized":"2024-03-28T23:32:46.165883+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:35.055035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.123363+0000","last_change":"2024-03-28T23:32:45.124724+0000","last_active":"2024-03-28T23:32:46.123363+0000","last_peered":"2024-03-28T23:32:46.123363+0000","last_clean":"2024-03-28T23:32:46.123363+0000","last_became_active":"2024-03-28T23:32:45.124359+0000","last_became_peered":"2024-03-28T23:32:45.124359+0000","last_unstale":"2024-03-28T23:32:46.123363+0000","last_undegraded":"2024-03-28T23:32:46.123363+0000","last_fullsized":"2024-03-28T23:32:46.123363+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:05.597289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.121380+0000","last_change":"2024-03-28T23:32:33.044477+0000","last_active":"2024-03-28T23:32:46.121380+0000","last_peered":"2024-03-28T23:32:46.121380+0000","last_clean":"2024-03-28T23:32:46.121380+0000","last_became_active":"2024-03-28T23:32:33.043960+0000","last_became_peered":"2024-03-28T23:32:33.043960+0000","last_unstale":"2024-03-28T23:32:46.121380+0000","last_undegraded":"2024-03-28T23:32:46.121380+0000","last_fullsized":"2024-03-28T23:32:46.121380+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:21.464120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163593+0000","last_change":"2024-03-28T23:32:45.124985+0000","last_active":"2024-03-28T23:32:46.163593+0000","last_peered":"2024-03-28T23:32:46.163593+0000","last_clean":"2024-03-28T23:32:46.163593+0000","last_became_active":"2024-03-28T23:32:45.124079+0000","last_became_peered":"2024-03-28T23:32:45.124079+0000","last_unstale":"2024-03-28T23:32:46.163593+0000","last_undegraded":"2024-03-28T23:32:46.163593+0000","last_fullsized":"2024-03-28T23:32:46.163593+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:04:31.273811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165692+0000","last_change":"2024-03-28T23:32:33.052803+0000","last_active":"2024-03-28T23:32:46.165692+0000","last_peered":"2024-03-28T23:32:46.165692+0000","last_clean":"2024-03-28T23:32:46.165692+0000","last_became_active":"2024-03-28T23:32:33.052524+0000","last_became_peered":"2024-03-28T23:32:33.052524+0000","last_unstale":"2024-03-28T23:32:46.165692+0000","last_undegraded":"2024-03-28T23:32:46.165692+0000","last_fullsized":"2024-03-28T23:32:46.165692+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:19:09.500642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.124649+0000","last_change":"2024-03-28T23:32:45.130606+0000","last_active":"2024-03-28T23:32:46.124649+0000","last_peered":"2024-03-28T23:32:46.124649+0000","last_clean":"2024-03-28T23:32:46.124649+0000","last_became_active":"2024-03-28T23:32:45.130345+0000","last_became_peered":"2024-03-28T23:32:45.130345+0000","last_unstale":"2024-03-28T23:32:46.124649+0000","last_undegraded":"2024-03-28T23:32:46.124649+0000","last_fullsized":"2024-03-28T23:32:46.124649+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:46:55.729338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.124213+0000","last_change":"2024-03-28T23:32:33.042587+0000","last_active":"2024-03-28T23:32:46.124213+0000","last_peered":"2024-03-28T23:32:46.124213+0000","last_clean":"2024-03-28T23:32:46.124213+0000","last_became_active":"2024-03-28T23:32:33.037002+0000","last_became_peered":"2024-03-28T23:32:33.037002+0000","last_unstale":"2024-03-28T23:32:46.124213+0000","last_undegraded":"2024-03-28T23:32:46.124213+0000","last_fullsized":"2024-03-28T23:32:46.124213+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:36:03.951054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165818+0000","last_change":"2024-03-28T23:32:45.113627+0000","last_active":"2024-03-28T23:32:46.165818+0000","last_peered":"2024-03-28T23:32:46.165818+0000","last_clean":"2024-03-28T23:32:46.165818+0000","last_became_active":"2024-03-28T23:32:45.113364+0000","last_became_peered":"2024-03-28T23:32:45.113364+0000","last_unstale":"2024-03-28T23:32:46.165818+0000","last_undegraded":"2024-03-28T23:32:46.165818+0000","last_fullsized":"2024-03-28T23:32:46.165818+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:54:25.867257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.123915+0000","last_change":"2024-03-28T23:32:33.039941+0000","last_active":"2024-03-28T23:32:46.123915+0000","last_peered":"2024-03-28T23:32:46.123915+0000","last_clean":"2024-03-28T23:32:46.123915+0000","last_became_active":"2024-03-28T23:32:33.039656+0000","last_became_peered":"2024-03-28T23:32:33.039656+0000","last_unstale":"2024-03-28T23:32:46.123915+0000","last_undegraded":"2024-03-28T23:32:46.123915+0000","last_fullsized":"2024-03-28T23:32:46.123915+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:16.850667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.124189+0000","last_change":"2024-03-28T23:32:45.133252+0000","last_active":"2024-03-28T23:32:46.124189+0000","last_peered":"2024-03-28T23:32:46.124189+0000","last_clean":"2024-03-28T23:32:46.124189+0000","last_became_active":"2024-03-28T23:32:45.132966+0000","last_became_peered":"2024-03-28T23:32:45.132966+0000","last_unstale":"2024-03-28T23:32:46.124189+0000","last_undegraded":"2024-03-28T23:32:46.124189+0000","last_fullsized":"2024-03-28T23:32:46.124189+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:47.139488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164479+0000","last_change":"2024-03-28T23:32:33.052329+0000","last_active":"2024-03-28T23:32:46.164479+0000","last_peered":"2024-03-28T23:32:46.164479+0000","last_clean":"2024-03-28T23:32:46.164479+0000","last_became_active":"2024-03-28T23:32:33.051833+0000","last_became_peered":"2024-03-28T23:32:33.051833+0000","last_unstale":"2024-03-28T23:32:46.164479+0000","last_undegraded":"2024-03-28T23:32:46.164479+0000","last_fullsized":"2024-03-28T23:32:46.164479+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:05.929331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164594+0000","last_change":"2024-03-28T23:32:45.115847+0000","last_active":"2024-03-28T23:32:46.164594+0000","last_peered":"2024-03-28T23:32:46.164594+0000","last_clean":"2024-03-28T23:32:46.164594+0000","last_became_active":"2024-03-28T23:32:45.115452+0000","last_became_peered":"2024-03-28T23:32:45.115452+0000","last_unstale":"2024-03-28T23:32:46.164594+0000","last_undegraded":"2024-03-28T23:32:46.164594+0000","last_fullsized":"2024-03-28T23:32:46.164594+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:02.169192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.123786+0000","last_change":"2024-03-28T23:32:33.042337+0000","last_active":"2024-03-28T23:32:46.123786+0000","last_peered":"2024-03-28T23:32:46.123786+0000","last_clean":"2024-03-28T23:32:46.123786+0000","last_became_active":"2024-03-28T23:32:33.042062+0000","last_became_peered":"2024-03-28T23:32:33.042062+0000","last_unstale":"2024-03-28T23:32:46.123786+0000","last_undegraded":"2024-03-28T23:32:46.123786+0000","last_fullsized":"2024-03-28T23:32:46.123786+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:13:34.921740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165390+0000","last_change":"2024-03-28T23:32:45.120907+0000","last_active":"2024-03-28T23:32:46.165390+0000","last_peered":"2024-03-28T23:32:46.165390+0000","last_clean":"2024-03-28T23:32:46.165390+0000","last_became_active":"2024-03-28T23:32:45.120614+0000","last_became_peered":"2024-03-28T23:32:45.120614+0000","last_unstale":"2024-03-28T23:32:46.165390+0000","last_undegraded":"2024-03-28T23:32:46.165390+0000","last_fullsized":"2024-03-28T23:32:46.165390+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:12.442633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166985+0000","last_change":"2024-03-28T23:32:33.046685+0000","last_active":"2024-03-28T23:32:46.166985+0000","last_peered":"2024-03-28T23:32:46.166985+0000","last_clean":"2024-03-28T23:32:46.166985+0000","last_became_active":"2024-03-28T23:32:33.046206+0000","last_became_peered":"2024-03-28T23:32:33.046206+0000","last_unstale":"2024-03-28T23:32:46.166985+0000","last_undegraded":"2024-03-28T23:32:46.166985+0000","last_fullsized":"2024-03-28T23:32:46.166985+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:43:22.001147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163804+0000","last_change":"2024-03-28T23:32:45.133232+0000","last_active":"2024-03-28T23:32:46.163804+0000","last_peered":"2024-03-28T23:32:46.163804+0000","last_clean":"2024-03-28T23:32:46.163804+0000","last_became_active":"2024-03-28T23:32:45.132973+0000","last_became_peered":"2024-03-28T23:32:45.132973+0000","last_unstale":"2024-03-28T23:32:46.163804+0000","last_undegraded":"2024-03-28T23:32:46.163804+0000","last_fullsized":"2024-03-28T23:32:46.163804+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:03:04.477944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165972+0000","last_change":"2024-03-28T23:32:33.029880+0000","last_active":"2024-03-28T23:32:46.165972+0000","last_peered":"2024-03-28T23:32:46.165972+0000","last_clean":"2024-03-28T23:32:46.165972+0000","last_became_active":"2024-03-28T23:32:33.029582+0000","last_became_peered":"2024-03-28T23:32:33.029582+0000","last_unstale":"2024-03-28T23:32:46.165972+0000","last_undegraded":"2024-03-28T23:32:46.165972+0000","last_fullsized":"2024-03-28T23:32:46.165972+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:38:39.283932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166217+0000","last_change":"2024-03-28T23:32:45.128053+0000","last_active":"2024-03-28T23:32:46.166217+0000","last_peered":"2024-03-28T23:32:46.166217+0000","last_clean":"2024-03-28T23:32:46.166217+0000","last_became_active":"2024-03-28T23:32:45.127791+0000","last_became_peered":"2024-03-28T23:32:45.127791+0000","last_unstale":"2024-03-28T23:32:46.166217+0000","last_undegraded":"2024-03-28T23:32:46.166217+0000","last_fullsized":"2024-03-28T23:32:46.166217+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:59:00.315700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165936+0000","last_change":"2024-03-28T23:32:33.040102+0000","last_active":"2024-03-28T23:32:46.165936+0000","last_peered":"2024-03-28T23:32:46.165936+0000","last_clean":"2024-03-28T23:32:46.165936+0000","last_became_active":"2024-03-28T23:32:33.039660+0000","last_became_peered":"2024-03-28T23:32:33.039660+0000","last_unstale":"2024-03-28T23:32:46.165936+0000","last_undegraded":"2024-03-28T23:32:46.165936+0000","last_fullsized":"2024-03-28T23:32:46.165936+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:35.055035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.123836+0000","last_change":"2024-03-28T23:32:45.122953+0000","last_active":"2024-03-28T23:32:46.123836+0000","last_peered":"2024-03-28T23:32:46.123836+0000","last_clean":"2024-03-28T23:32:46.123836+0000","last_became_active":"2024-03-28T23:32:45.122441+0000","last_became_peered":"2024-03-28T23:32:45.122441+0000","last_unstale":"2024-03-28T23:32:46.123836+0000","last_undegraded":"2024-03-28T23:32:46.123836+0000","last_fullsized":"2024-03-28T23:32:46.123836+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:32:20.557728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163579+0000","last_change":"2024-03-28T23:32:33.035442+0000","last_active":"2024-03-28T23:32:46.163579+0000","last_peered":"2024-03-28T23:32:46.163579+0000","last_clean":"2024-03-28T23:32:46.163579+0000","last_became_active":"2024-03-28T23:32:33.035154+0000","last_became_peered":"2024-03-28T23:32:33.035154+0000","last_unstale":"2024-03-28T23:32:46.163579+0000","last_undegraded":"2024-03-28T23:32:46.163579+0000","last_fullsized":"2024-03-28T23:32:46.163579+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:34:22.775718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.124724+0000","last_change":"2024-03-28T23:32:45.123371+0000","last_active":"2024-03-28T23:32:46.124724+0000","last_peered":"2024-03-28T23:32:46.124724+0000","last_clean":"2024-03-28T23:32:46.124724+0000","last_became_active":"2024-03-28T23:32:45.123118+0000","last_became_peered":"2024-03-28T23:32:45.123118+0000","last_unstale":"2024-03-28T23:32:46.124724+0000","last_undegraded":"2024-03-28T23:32:46.124724+0000","last_fullsized":"2024-03-28T23:32:46.124724+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:18.013783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165475+0000","last_change":"2024-03-28T23:32:33.047782+0000","last_active":"2024-03-28T23:32:46.165475+0000","last_peered":"2024-03-28T23:32:46.165475+0000","last_clean":"2024-03-28T23:32:46.165475+0000","last_became_active":"2024-03-28T23:32:33.047254+0000","last_became_peered":"2024-03-28T23:32:33.047254+0000","last_unstale":"2024-03-28T23:32:46.165475+0000","last_undegraded":"2024-03-28T23:32:46.165475+0000","last_fullsized":"2024-03-28T23:32:46.165475+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:19.004555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.167162+0000","last_change":"2024-03-28T23:32:45.121113+0000","last_active":"2024-03-28T23:32:46.167162+0000","last_peered":"2024-03-28T23:32:46.167162+0000","last_clean":"2024-03-28T23:32:46.167162+0000","last_became_active":"2024-03-28T23:32:45.120855+0000","last_became_peered":"2024-03-28T23:32:45.120855+0000","last_unstale":"2024-03-28T23:32:46.167162+0000","last_undegraded":"2024-03-28T23:32:46.167162+0000","last_fullsized":"2024-03-28T23:32:46.167162+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:18.013783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164953+0000","last_change":"2024-03-28T23:32:33.037321+0000","last_active":"2024-03-28T23:32:46.164953+0000","last_peered":"2024-03-28T23:32:46.164953+0000","last_clean":"2024-03-28T23:32:46.164953+0000","last_became_active":"2024-03-28T23:32:33.036961+0000","last_became_peered":"2024-03-28T23:32:33.036961+0000","last_unstale":"2024-03-28T23:32:46.164953+0000","last_undegraded":"2024-03-28T23:32:46.164953+0000","last_fullsized":"2024-03-28T23:32:46.164953+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:25.849919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164898+0000","last_change":"2024-03-28T23:32:45.124739+0000","last_active":"2024-03-28T23:32:46.164898+0000","last_peered":"2024-03-28T23:32:46.164898+0000","last_clean":"2024-03-28T23:32:46.164898+0000","last_became_active":"2024-03-28T23:32:45.124497+0000","last_became_peered":"2024-03-28T23:32:45.124497+0000","last_unstale":"2024-03-28T23:32:46.164898+0000","last_undegraded":"2024-03-28T23:32:46.164898+0000","last_fullsized":"2024-03-28T23:32:46.164898+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:22:55.704581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165055+0000","last_change":"2024-03-28T23:32:33.053071+0000","last_active":"2024-03-28T23:32:46.165055+0000","last_peered":"2024-03-28T23:32:46.165055+0000","last_clean":"2024-03-28T23:32:46.165055+0000","last_became_active":"2024-03-28T23:32:33.052784+0000","last_became_peered":"2024-03-28T23:32:33.052784+0000","last_unstale":"2024-03-28T23:32:46.165055+0000","last_undegraded":"2024-03-28T23:32:46.165055+0000","last_fullsized":"2024-03-28T23:32:46.165055+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:58:48.231248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164101+0000","last_change":"2024-03-28T23:32:45.123590+0000","last_active":"2024-03-28T23:32:46.164101+0000","last_peered":"2024-03-28T23:32:46.164101+0000","last_clean":"2024-03-28T23:32:46.164101+0000","last_became_active":"2024-03-28T23:32:45.123361+0000","last_became_peered":"2024-03-28T23:32:45.123361+0000","last_unstale":"2024-03-28T23:32:46.164101+0000","last_undegraded":"2024-03-28T23:32:46.164101+0000","last_fullsized":"2024-03-28T23:32:46.164101+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:49.881731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165902+0000","last_change":"2024-03-28T23:32:33.052720+0000","last_active":"2024-03-28T23:32:46.165902+0000","last_peered":"2024-03-28T23:32:46.165902+0000","last_clean":"2024-03-28T23:32:46.165902+0000","last_became_active":"2024-03-28T23:32:33.052460+0000","last_became_peered":"2024-03-28T23:32:33.052460+0000","last_unstale":"2024-03-28T23:32:46.165902+0000","last_undegraded":"2024-03-28T23:32:46.165902+0000","last_fullsized":"2024-03-28T23:32:46.165902+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:30.891315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.121553+0000","last_change":"2024-03-28T23:32:45.130535+0000","last_active":"2024-03-28T23:32:46.121553+0000","last_peered":"2024-03-28T23:32:46.121553+0000","last_clean":"2024-03-28T23:32:46.121553+0000","last_became_active":"2024-03-28T23:32:45.130195+0000","last_became_peered":"2024-03-28T23:32:45.130195+0000","last_unstale":"2024-03-28T23:32:46.121553+0000","last_undegraded":"2024-03-28T23:32:46.121553+0000","last_fullsized":"2024-03-28T23:32:46.121553+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:28:48.620361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.123789+0000","last_change":"2024-03-28T23:32:33.035666+0000","last_active":"2024-03-28T23:32:46.123789+0000","last_peered":"2024-03-28T23:32:46.123789+0000","last_clean":"2024-03-28T23:32:46.123789+0000","last_became_active":"2024-03-28T23:32:33.035403+0000","last_became_peered":"2024-03-28T23:32:33.035403+0000","last_unstale":"2024-03-28T23:32:46.123789+0000","last_undegraded":"2024-03-28T23:32:46.123789+0000","last_fullsized":"2024-03-28T23:32:46.123789+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:40:51.269760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164304+0000","last_change":"2024-03-28T23:32:45.124238+0000","last_active":"2024-03-28T23:32:46.164304+0000","last_peered":"2024-03-28T23:32:46.164304+0000","last_clean":"2024-03-28T23:32:46.164304+0000","last_became_active":"2024-03-28T23:32:45.123997+0000","last_became_peered":"2024-03-28T23:32:45.123997+0000","last_unstale":"2024-03-28T23:32:46.164304+0000","last_undegraded":"2024-03-28T23:32:46.164304+0000","last_fullsized":"2024-03-28T23:32:46.164304+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:38:37.572370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164596+0000","last_change":"2024-03-28T23:32:33.047992+0000","last_active":"2024-03-28T23:32:46.164596+0000","last_peered":"2024-03-28T23:32:46.164596+0000","last_clean":"2024-03-28T23:32:46.164596+0000","last_became_active":"2024-03-28T23:32:33.047058+0000","last_became_peered":"2024-03-28T23:32:33.047058+0000","last_unstale":"2024-03-28T23:32:46.164596+0000","last_undegraded":"2024-03-28T23:32:46.164596+0000","last_fullsized":"2024-03-28T23:32:46.164596+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:35:30.886935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.123616+0000","last_change":"2024-03-28T23:32:45.125148+0000","last_active":"2024-03-28T23:32:46.123616+0000","last_peered":"2024-03-28T23:32:46.123616+0000","last_clean":"2024-03-28T23:32:46.123616+0000","last_became_active":"2024-03-28T23:32:45.124861+0000","last_became_peered":"2024-03-28T23:32:45.124861+0000","last_unstale":"2024-03-28T23:32:46.123616+0000","last_undegraded":"2024-03-28T23:32:46.123616+0000","last_fullsized":"2024-03-28T23:32:46.123616+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:02:57.275018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.123048+0000","last_change":"2024-03-28T23:32:33.029536+0000","last_active":"2024-03-28T23:32:46.123048+0000","last_peered":"2024-03-28T23:32:46.123048+0000","last_clean":"2024-03-28T23:32:46.123048+0000","last_became_active":"2024-03-28T23:32:33.029265+0000","last_became_peered":"2024-03-28T23:32:33.029265+0000","last_unstale":"2024-03-28T23:32:46.123048+0000","last_undegraded":"2024-03-28T23:32:46.123048+0000","last_fullsized":"2024-03-28T23:32:46.123048+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:51:11.447507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.124395+0000","last_change":"2024-03-28T23:32:45.122715+0000","last_active":"2024-03-28T23:32:46.124395+0000","last_peered":"2024-03-28T23:32:46.124395+0000","last_clean":"2024-03-28T23:32:46.124395+0000","last_became_active":"2024-03-28T23:32:45.122136+0000","last_became_peered":"2024-03-28T23:32:45.122136+0000","last_unstale":"2024-03-28T23:32:46.124395+0000","last_undegraded":"2024-03-28T23:32:46.124395+0000","last_fullsized":"2024-03-28T23:32:46.124395+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:56:06.278905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163897+0000","last_change":"2024-03-28T23:32:33.041328+0000","last_active":"2024-03-28T23:32:46.163897+0000","last_peered":"2024-03-28T23:32:46.163897+0000","last_clean":"2024-03-28T23:32:46.163897+0000","last_became_active":"2024-03-28T23:32:33.041071+0000","last_became_peered":"2024-03-28T23:32:33.041071+0000","last_unstale":"2024-03-28T23:32:46.163897+0000","last_undegraded":"2024-03-28T23:32:46.163897+0000","last_fullsized":"2024-03-28T23:32:46.163897+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:00.358180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164970+0000","last_change":"2024-03-28T23:32:45.114377+0000","last_active":"2024-03-28T23:32:46.164970+0000","last_peered":"2024-03-28T23:32:46.164970+0000","last_clean":"2024-03-28T23:32:46.164970+0000","last_became_active":"2024-03-28T23:32:45.114137+0000","last_became_peered":"2024-03-28T23:32:45.114137+0000","last_unstale":"2024-03-28T23:32:46.164970+0000","last_undegraded":"2024-03-28T23:32:46.164970+0000","last_fullsized":"2024-03-28T23:32:46.164970+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:18:14.670198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165064+0000","last_change":"2024-03-28T23:32:33.043840+0000","last_active":"2024-03-28T23:32:46.165064+0000","last_peered":"2024-03-28T23:32:46.165064+0000","last_clean":"2024-03-28T23:32:46.165064+0000","last_became_active":"2024-03-28T23:32:33.043569+0000","last_became_peered":"2024-03-28T23:32:33.043569+0000","last_unstale":"2024-03-28T23:32:46.165064+0000","last_undegraded":"2024-03-28T23:32:46.165064+0000","last_fullsized":"2024-03-28T23:32:46.165064+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:28.463493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166879+0000","last_change":"2024-03-28T23:32:45.129808+0000","last_active":"2024-03-28T23:32:46.166879+0000","last_peered":"2024-03-28T23:32:46.166879+0000","last_clean":"2024-03-28T23:32:46.166879+0000","last_became_active":"2024-03-28T23:32:45.129552+0000","last_became_peered":"2024-03-28T23:32:45.129552+0000","last_unstale":"2024-03-28T23:32:46.166879+0000","last_undegraded":"2024-03-28T23:32:46.166879+0000","last_fullsized":"2024-03-28T23:32:46.166879+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:46:55.729338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.124048+0000","last_change":"2024-03-28T23:32:33.047594+0000","last_active":"2024-03-28T23:32:46.124048+0000","last_peered":"2024-03-28T23:32:46.124048+0000","last_clean":"2024-03-28T23:32:46.124048+0000","last_became_active":"2024-03-28T23:32:33.047031+0000","last_became_peered":"2024-03-28T23:32:33.047031+0000","last_unstale":"2024-03-28T23:32:46.124048+0000","last_undegraded":"2024-03-28T23:32:46.124048+0000","last_fullsized":"2024-03-28T23:32:46.124048+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:30:12.769818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.123558+0000","last_change":"2024-03-28T23:32:45.132578+0000","last_active":"2024-03-28T23:32:46.123558+0000","last_peered":"2024-03-28T23:32:46.123558+0000","last_clean":"2024-03-28T23:32:46.123558+0000","last_became_active":"2024-03-28T23:32:45.132237+0000","last_became_peered":"2024-03-28T23:32:45.132237+0000","last_unstale":"2024-03-28T23:32:46.123558+0000","last_undegraded":"2024-03-28T23:32:46.123558+0000","last_fullsized":"2024-03-28T23:32:46.123558+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:23:55.885296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163728+0000","last_change":"2024-03-28T23:32:33.043396+0000","last_active":"2024-03-28T23:32:46.163728+0000","last_peered":"2024-03-28T23:32:46.163728+0000","last_clean":"2024-03-28T23:32:46.163728+0000","last_became_active":"2024-03-28T23:32:33.043123+0000","last_became_peered":"2024-03-28T23:32:33.043123+0000","last_unstale":"2024-03-28T23:32:46.163728+0000","last_undegraded":"2024-03-28T23:32:46.163728+0000","last_fullsized":"2024-03-28T23:32:46.163728+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:28:36.535909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.123865+0000","last_change":"2024-03-28T23:32:45.119360+0000","last_active":"2024-03-28T23:32:46.123865+0000","last_peered":"2024-03-28T23:32:46.123865+0000","last_clean":"2024-03-28T23:32:46.123865+0000","last_became_active":"2024-03-28T23:32:45.119096+0000","last_became_peered":"2024-03-28T23:32:45.119096+0000","last_unstale":"2024-03-28T23:32:46.123865+0000","last_undegraded":"2024-03-28T23:32:46.123865+0000","last_fullsized":"2024-03-28T23:32:46.123865+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:12.442633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165979+0000","last_change":"2024-03-28T23:32:33.042288+0000","last_active":"2024-03-28T23:32:46.165979+0000","last_peered":"2024-03-28T23:32:46.165979+0000","last_clean":"2024-03-28T23:32:46.165979+0000","last_became_active":"2024-03-28T23:32:33.042007+0000","last_became_peered":"2024-03-28T23:32:33.042007+0000","last_unstale":"2024-03-28T23:32:46.165979+0000","last_undegraded":"2024-03-28T23:32:46.165979+0000","last_fullsized":"2024-03-28T23:32:46.165979+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:16:12.634582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.123166+0000","last_change":"2024-03-28T23:32:45.129068+0000","last_active":"2024-03-28T23:32:46.123166+0000","last_peered":"2024-03-28T23:32:46.123166+0000","last_clean":"2024-03-28T23:32:46.123166+0000","last_became_active":"2024-03-28T23:32:45.128801+0000","last_became_peered":"2024-03-28T23:32:45.128801+0000","last_unstale":"2024-03-28T23:32:46.123166+0000","last_undegraded":"2024-03-28T23:32:46.123166+0000","last_fullsized":"2024-03-28T23:32:46.123166+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:41:51.742183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165206+0000","last_change":"2024-03-28T23:32:33.041336+0000","last_active":"2024-03-28T23:32:46.165206+0000","last_peered":"2024-03-28T23:32:46.165206+0000","last_clean":"2024-03-28T23:32:46.165206+0000","last_became_active":"2024-03-28T23:32:33.040865+0000","last_became_peered":"2024-03-28T23:32:33.040865+0000","last_unstale":"2024-03-28T23:32:46.165206+0000","last_undegraded":"2024-03-28T23:32:46.165206+0000","last_fullsized":"2024-03-28T23:32:46.165206+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:44:58.784172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164344+0000","last_change":"2024-03-28T23:32:45.133607+0000","last_active":"2024-03-28T23:32:46.164344+0000","last_peered":"2024-03-28T23:32:46.164344+0000","last_clean":"2024-03-28T23:32:46.164344+0000","last_became_active":"2024-03-28T23:32:45.133367+0000","last_became_peered":"2024-03-28T23:32:45.133367+0000","last_unstale":"2024-03-28T23:32:46.164344+0000","last_undegraded":"2024-03-28T23:32:46.164344+0000","last_fullsized":"2024-03-28T23:32:46.164344+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:52:37.666022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163441+0000","last_change":"2024-03-28T23:32:33.049222+0000","last_active":"2024-03-28T23:32:46.163441+0000","last_peered":"2024-03-28T23:32:46.163441+0000","last_clean":"2024-03-28T23:32:46.163441+0000","last_became_active":"2024-03-28T23:32:33.048933+0000","last_became_peered":"2024-03-28T23:32:33.048933+0000","last_unstale":"2024-03-28T23:32:46.163441+0000","last_undegraded":"2024-03-28T23:32:46.163441+0000","last_fullsized":"2024-03-28T23:32:46.163441+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:17:45.545141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163688+0000","last_change":"2024-03-28T23:32:45.115970+0000","last_active":"2024-03-28T23:32:46.163688+0000","last_peered":"2024-03-28T23:32:46.163688+0000","last_clean":"2024-03-28T23:32:46.163688+0000","last_became_active":"2024-03-28T23:32:45.115659+0000","last_became_peered":"2024-03-28T23:32:45.115659+0000","last_unstale":"2024-03-28T23:32:46.163688+0000","last_undegraded":"2024-03-28T23:32:46.163688+0000","last_fullsized":"2024-03-28T23:32:46.163688+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:38:37.572370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165670+0000","last_change":"2024-03-28T23:32:33.047977+0000","last_active":"2024-03-28T23:32:46.165670+0000","last_peered":"2024-03-28T23:32:46.165670+0000","last_clean":"2024-03-28T23:32:46.165670+0000","last_became_active":"2024-03-28T23:32:33.047699+0000","last_became_peered":"2024-03-28T23:32:33.047699+0000","last_unstale":"2024-03-28T23:32:46.165670+0000","last_undegraded":"2024-03-28T23:32:46.165670+0000","last_fullsized":"2024-03-28T23:32:46.165670+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:38:51.513907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.124365+0000","last_change":"2024-03-28T23:32:45.133340+0000","last_active":"2024-03-28T23:32:46.124365+0000","last_peered":"2024-03-28T23:32:46.124365+0000","last_clean":"2024-03-28T23:32:46.124365+0000","last_became_active":"2024-03-28T23:32:45.133052+0000","last_became_peered":"2024-03-28T23:32:45.133052+0000","last_unstale":"2024-03-28T23:32:46.124365+0000","last_undegraded":"2024-03-28T23:32:46.124365+0000","last_fullsized":"2024-03-28T23:32:46.124365+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:38:51.368384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163674+0000","last_change":"2024-03-28T23:32:33.036527+0000","last_active":"2024-03-28T23:32:46.163674+0000","last_peered":"2024-03-28T23:32:46.163674+0000","last_clean":"2024-03-28T23:32:46.163674+0000","last_became_active":"2024-03-28T23:32:33.036269+0000","last_became_peered":"2024-03-28T23:32:33.036269+0000","last_unstale":"2024-03-28T23:32:46.163674+0000","last_undegraded":"2024-03-28T23:32:46.163674+0000","last_fullsized":"2024-03-28T23:32:46.163674+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:43:35.946992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164668+0000","last_change":"2024-03-28T23:32:45.125187+0000","last_active":"2024-03-28T23:32:46.164668+0000","last_peered":"2024-03-28T23:32:46.164668+0000","last_clean":"2024-03-28T23:32:46.164668+0000","last_became_active":"2024-03-28T23:32:45.124029+0000","last_became_peered":"2024-03-28T23:32:45.124029+0000","last_unstale":"2024-03-28T23:32:46.164668+0000","last_undegraded":"2024-03-28T23:32:46.164668+0000","last_fullsized":"2024-03-28T23:32:46.164668+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:39.510794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164994+0000","last_change":"2024-03-28T23:32:44.118134+0000","last_active":"2024-03-28T23:32:46.164994+0000","last_peered":"2024-03-28T23:32:46.164994+0000","last_clean":"2024-03-28T23:32:46.164994+0000","last_became_active":"2024-03-28T23:32:44.117870+0000","last_became_peered":"2024-03-28T23:32:44.117870+0000","last_unstale":"2024-03-28T23:32:46.164994+0000","last_undegraded":"2024-03-28T23:32:46.164994+0000","last_fullsized":"2024-03-28T23:32:46.164994+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:41:50.736456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166443+0000","last_change":"2024-03-28T23:32:33.038016+0000","last_active":"2024-03-28T23:32:46.166443+0000","last_peered":"2024-03-28T23:32:46.166443+0000","last_clean":"2024-03-28T23:32:46.166443+0000","last_became_active":"2024-03-28T23:32:33.037754+0000","last_became_peered":"2024-03-28T23:32:33.037754+0000","last_unstale":"2024-03-28T23:32:46.166443+0000","last_undegraded":"2024-03-28T23:32:46.166443+0000","last_fullsized":"2024-03-28T23:32:46.166443+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:21.878156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"72'4","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165561+0000","last_change":"2024-03-28T23:32:32.040468+0000","last_active":"2024-03-28T23:32:46.165561+0000","last_peered":"2024-03-28T23:32:46.165561+0000","last_clean":"2024-03-28T23:32:46.165561+0000","last_became_active":"2024-03-28T23:32:32.039342+0000","last_became_peered":"2024-03-28T23:32:32.039342+0000","last_unstale":"2024-03-28T23:32:46.165561+0000","last_undegraded":"2024-03-28T23:32:46.165561+0000","last_fullsized":"2024-03-28T23:32:46.165561+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:05.786248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166307+0000","last_change":"2024-03-28T23:32:45.124604+0000","last_active":"2024-03-28T23:32:46.166307+0000","last_peered":"2024-03-28T23:32:46.166307+0000","last_clean":"2024-03-28T23:32:46.166307+0000","last_became_active":"2024-03-28T23:32:45.124367+0000","last_became_peered":"2024-03-28T23:32:45.124367+0000","last_unstale":"2024-03-28T23:32:46.166307+0000","last_undegraded":"2024-03-28T23:32:46.166307+0000","last_fullsized":"2024-03-28T23:32:46.166307+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:38:51.368384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163558+0000","last_change":"2024-03-28T23:32:44.126084+0000","last_active":"2024-03-28T23:32:46.163558+0000","last_peered":"2024-03-28T23:32:46.163558+0000","last_clean":"2024-03-28T23:32:46.163558+0000","last_became_active":"2024-03-28T23:32:44.125728+0000","last_became_peered":"2024-03-28T23:32:44.125728+0000","last_unstale":"2024-03-28T23:32:46.163558+0000","last_undegraded":"2024-03-28T23:32:46.163558+0000","last_fullsized":"2024-03-28T23:32:46.163558+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:41:50.736456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166027+0000","last_change":"2024-03-28T23:32:33.039778+0000","last_active":"2024-03-28T23:32:46.166027+0000","last_peered":"2024-03-28T23:32:46.166027+0000","last_clean":"2024-03-28T23:32:46.166027+0000","last_became_active":"2024-03-28T23:32:33.039506+0000","last_became_peered":"2024-03-28T23:32:33.039506+0000","last_unstale":"2024-03-28T23:32:46.166027+0000","last_undegraded":"2024-03-28T23:32:46.166027+0000","last_fullsized":"2024-03-28T23:32:46.166027+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:38:39.283932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"72'1","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164035+0000","last_change":"2024-03-28T23:32:32.041093+0000","last_active":"2024-03-28T23:32:46.164035+0000","last_peered":"2024-03-28T23:32:46.164035+0000","last_clean":"2024-03-28T23:32:46.164035+0000","last_became_active":"2024-03-28T23:32:32.040827+0000","last_became_peered":"2024-03-28T23:32:32.040827+0000","last_unstale":"2024-03-28T23:32:46.164035+0000","last_undegraded":"2024-03-28T23:32:46.164035+0000","last_fullsized":"2024-03-28T23:32:46.164035+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:55.544945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166216+0000","last_change":"2024-03-28T23:32:45.116642+0000","last_active":"2024-03-28T23:32:46.166216+0000","last_peered":"2024-03-28T23:32:46.166216+0000","last_clean":"2024-03-28T23:32:46.166216+0000","last_became_active":"2024-03-28T23:32:45.116387+0000","last_became_peered":"2024-03-28T23:32:45.116387+0000","last_unstale":"2024-03-28T23:32:46.166216+0000","last_undegraded":"2024-03-28T23:32:46.166216+0000","last_fullsized":"2024-03-28T23:32:46.166216+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:47.139488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"76'1","reported_seq":15,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.123459+0000","last_change":"2024-03-28T23:32:44.119556+0000","last_active":"2024-03-28T23:32:46.123459+0000","last_peered":"2024-03-28T23:32:46.123459+0000","last_clean":"2024-03-28T23:32:46.123459+0000","last_became_active":"2024-03-28T23:32:44.119246+0000","last_became_peered":"2024-03-28T23:32:44.119246+0000","last_unstale":"2024-03-28T23:32:46.123459+0000","last_undegraded":"2024-03-28T23:32:46.123459+0000","last_fullsized":"2024-03-28T23:32:46.123459+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:13:16.820792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.124521+0000","last_change":"2024-03-28T23:32:33.044887+0000","last_active":"2024-03-28T23:32:46.124521+0000","last_peered":"2024-03-28T23:32:46.124521+0000","last_clean":"2024-03-28T23:32:46.124521+0000","last_became_active":"2024-03-28T23:32:33.042005+0000","last_became_peered":"2024-03-28T23:32:33.042005+0000","last_unstale":"2024-03-28T23:32:46.124521+0000","last_undegraded":"2024-03-28T23:32:46.124521+0000","last_fullsized":"2024-03-28T23:32:46.124521+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:41.257241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"72'2","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163840+0000","last_change":"2024-03-28T23:32:32.040527+0000","last_active":"2024-03-28T23:32:46.163840+0000","last_peered":"2024-03-28T23:32:46.163840+0000","last_clean":"2024-03-28T23:32:46.163840+0000","last_became_active":"2024-03-28T23:32:32.040142+0000","last_became_peered":"2024-03-28T23:32:32.040142+0000","last_unstale":"2024-03-28T23:32:46.163840+0000","last_undegraded":"2024-03-28T23:32:46.163840+0000","last_fullsized":"2024-03-28T23:32:46.163840+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:13:15.856750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.121947+0000","last_change":"2024-03-28T23:32:45.130945+0000","last_active":"2024-03-28T23:32:46.121947+0000","last_peered":"2024-03-28T23:32:46.121947+0000","last_clean":"2024-03-28T23:32:46.121947+0000","last_became_active":"2024-03-28T23:32:45.130678+0000","last_became_peered":"2024-03-28T23:32:45.130678+0000","last_unstale":"2024-03-28T23:32:46.121947+0000","last_undegraded":"2024-03-28T23:32:46.121947+0000","last_fullsized":"2024-03-28T23:32:46.121947+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:02:57.275018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"76'1","reported_seq":15,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165806+0000","last_change":"2024-03-28T23:32:44.116203+0000","last_active":"2024-03-28T23:32:46.165806+0000","last_peered":"2024-03-28T23:32:46.165806+0000","last_clean":"2024-03-28T23:32:46.165806+0000","last_became_active":"2024-03-28T23:32:44.115846+0000","last_became_peered":"2024-03-28T23:32:44.115846+0000","last_unstale":"2024-03-28T23:32:46.165806+0000","last_undegraded":"2024-03-28T23:32:46.165806+0000","last_fullsized":"2024-03-28T23:32:46.165806+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:27.929392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.123946+0000","last_change":"2024-03-28T23:32:33.040988+0000","last_active":"2024-03-28T23:32:46.123946+0000","last_peered":"2024-03-28T23:32:46.123946+0000","last_clean":"2024-03-28T23:32:46.123946+0000","last_became_active":"2024-03-28T23:32:33.036045+0000","last_became_peered":"2024-03-28T23:32:33.036045+0000","last_unstale":"2024-03-28T23:32:46.123946+0000","last_undegraded":"2024-03-28T23:32:46.123946+0000","last_fullsized":"2024-03-28T23:32:46.123946+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:10:39.981362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"72'2","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.122175+0000","last_change":"2024-03-28T23:32:32.036856+0000","last_active":"2024-03-28T23:32:46.122175+0000","last_peered":"2024-03-28T23:32:46.122175+0000","last_clean":"2024-03-28T23:32:46.122175+0000","last_became_active":"2024-03-28T23:32:32.036565+0000","last_became_peered":"2024-03-28T23:32:32.036565+0000","last_unstale":"2024-03-28T23:32:46.122175+0000","last_undegraded":"2024-03-28T23:32:46.122175+0000","last_fullsized":"2024-03-28T23:32:46.122175+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:11:01.106436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163855+0000","last_change":"2024-03-28T23:32:45.120134+0000","last_active":"2024-03-28T23:32:46.163855+0000","last_peered":"2024-03-28T23:32:46.163855+0000","last_clean":"2024-03-28T23:32:46.163855+0000","last_became_active":"2024-03-28T23:32:45.119812+0000","last_became_peered":"2024-03-28T23:32:45.119812+0000","last_unstale":"2024-03-28T23:32:46.163855+0000","last_undegraded":"2024-03-28T23:32:46.163855+0000","last_fullsized":"2024-03-28T23:32:46.163855+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:35:54.086668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"76'2","reported_seq":16,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163753+0000","last_change":"2024-03-28T23:32:44.116097+0000","last_active":"2024-03-28T23:32:46.163753+0000","last_peered":"2024-03-28T23:32:46.163753+0000","last_clean":"2024-03-28T23:32:46.163753+0000","last_became_active":"2024-03-28T23:32:44.115810+0000","last_became_peered":"2024-03-28T23:32:44.115810+0000","last_unstale":"2024-03-28T23:32:46.163753+0000","last_undegraded":"2024-03-28T23:32:46.163753+0000","last_fullsized":"2024-03-28T23:32:46.163753+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:23:54.879569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166091+0000","last_change":"2024-03-28T23:32:33.047911+0000","last_active":"2024-03-28T23:32:46.166091+0000","last_peered":"2024-03-28T23:32:46.166091+0000","last_clean":"2024-03-28T23:32:46.166091+0000","last_became_active":"2024-03-28T23:32:33.047616+0000","last_became_peered":"2024-03-28T23:32:33.047616+0000","last_unstale":"2024-03-28T23:32:46.166091+0000","last_undegraded":"2024-03-28T23:32:46.166091+0000","last_fullsized":"2024-03-28T23:32:46.166091+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:58:48.231248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"72'1","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.124222+0000","last_change":"2024-03-28T23:32:32.036829+0000","last_active":"2024-03-28T23:32:46.124222+0000","last_peered":"2024-03-28T23:32:46.124222+0000","last_clean":"2024-03-28T23:32:46.124222+0000","last_became_active":"2024-03-28T23:32:32.036571+0000","last_became_peered":"2024-03-28T23:32:32.036571+0000","last_unstale":"2024-03-28T23:32:46.124222+0000","last_undegraded":"2024-03-28T23:32:46.124222+0000","last_fullsized":"2024-03-28T23:32:46.124222+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:32:03.789593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165166+0000","last_change":"2024-03-28T23:32:45.127680+0000","last_active":"2024-03-28T23:32:46.165166+0000","last_peered":"2024-03-28T23:32:46.165166+0000","last_clean":"2024-03-28T23:32:46.165166+0000","last_became_active":"2024-03-28T23:32:45.127403+0000","last_became_peered":"2024-03-28T23:32:45.127403+0000","last_unstale":"2024-03-28T23:32:46.165166+0000","last_undegraded":"2024-03-28T23:32:46.165166+0000","last_fullsized":"2024-03-28T23:32:46.165166+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:34:51.492132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"76'2","reported_seq":16,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.121647+0000","last_change":"2024-03-28T23:32:44.117981+0000","last_active":"2024-03-28T23:32:46.121647+0000","last_peered":"2024-03-28T23:32:46.121647+0000","last_clean":"2024-03-28T23:32:46.121647+0000","last_became_active":"2024-03-28T23:32:44.117709+0000","last_became_peered":"2024-03-28T23:32:44.117709+0000","last_unstale":"2024-03-28T23:32:46.121647+0000","last_undegraded":"2024-03-28T23:32:46.121647+0000","last_fullsized":"2024-03-28T23:32:46.121647+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:37:37.731237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164579+0000","last_change":"2024-03-28T23:32:33.052330+0000","last_active":"2024-03-28T23:32:46.164579+0000","last_peered":"2024-03-28T23:32:46.164579+0000","last_clean":"2024-03-28T23:32:46.164579+0000","last_became_active":"2024-03-28T23:32:33.051828+0000","last_became_peered":"2024-03-28T23:32:33.051828+0000","last_unstale":"2024-03-28T23:32:46.164579+0000","last_undegraded":"2024-03-28T23:32:46.164579+0000","last_fullsized":"2024-03-28T23:32:46.164579+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:40:53.134608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"72'1","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.123306+0000","last_change":"2024-03-28T23:32:32.036739+0000","last_active":"2024-03-28T23:32:46.123306+0000","last_peered":"2024-03-28T23:32:46.123306+0000","last_clean":"2024-03-28T23:32:46.123306+0000","last_became_active":"2024-03-28T23:32:32.036471+0000","last_became_peered":"2024-03-28T23:32:32.036471+0000","last_unstale":"2024-03-28T23:32:46.123306+0000","last_undegraded":"2024-03-28T23:32:46.123306+0000","last_fullsized":"2024-03-28T23:32:46.123306+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:48.704351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165297+0000","last_change":"2024-03-28T23:32:45.136600+0000","last_active":"2024-03-28T23:32:46.165297+0000","last_peered":"2024-03-28T23:32:46.165297+0000","last_clean":"2024-03-28T23:32:46.165297+0000","last_became_active":"2024-03-28T23:32:45.136366+0000","last_became_peered":"2024-03-28T23:32:45.136366+0000","last_unstale":"2024-03-28T23:32:46.165297+0000","last_undegraded":"2024-03-28T23:32:46.165297+0000","last_fullsized":"2024-03-28T23:32:46.165297+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:06:58.580120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"76'4","reported_seq":18,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.122621+0000","last_change":"2024-03-28T23:32:44.126818+0000","last_active":"2024-03-28T23:32:46.122621+0000","last_peered":"2024-03-28T23:32:46.122621+0000","last_clean":"2024-03-28T23:32:46.122621+0000","last_became_active":"2024-03-28T23:32:44.126493+0000","last_became_peered":"2024-03-28T23:32:44.126493+0000","last_unstale":"2024-03-28T23:32:46.122621+0000","last_undegraded":"2024-03-28T23:32:46.122621+0000","last_fullsized":"2024-03-28T23:32:46.122621+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:30:23.848543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.124615+0000","last_change":"2024-03-28T23:32:33.034126+0000","last_active":"2024-03-28T23:32:46.124615+0000","last_peered":"2024-03-28T23:32:46.124615+0000","last_clean":"2024-03-28T23:32:46.124615+0000","last_became_active":"2024-03-28T23:32:33.033703+0000","last_became_peered":"2024-03-28T23:32:33.033703+0000","last_unstale":"2024-03-28T23:32:46.124615+0000","last_undegraded":"2024-03-28T23:32:46.124615+0000","last_fullsized":"2024-03-28T23:32:46.124615+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:38.859031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166116+0000","last_change":"2024-03-28T23:32:32.039047+0000","last_active":"2024-03-28T23:32:46.166116+0000","last_peered":"2024-03-28T23:32:46.166116+0000","last_clean":"2024-03-28T23:32:46.166116+0000","last_became_active":"2024-03-28T23:32:32.038780+0000","last_became_peered":"2024-03-28T23:32:32.038780+0000","last_unstale":"2024-03-28T23:32:46.166116+0000","last_undegraded":"2024-03-28T23:32:46.166116+0000","last_fullsized":"2024-03-28T23:32:46.166116+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:13:15.856750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164099+0000","last_change":"2024-03-28T23:32:45.125579+0000","last_active":"2024-03-28T23:32:46.164099+0000","last_peered":"2024-03-28T23:32:46.164099+0000","last_clean":"2024-03-28T23:32:46.164099+0000","last_became_active":"2024-03-28T23:32:45.125164+0000","last_became_peered":"2024-03-28T23:32:45.125164+0000","last_unstale":"2024-03-28T23:32:46.164099+0000","last_undegraded":"2024-03-28T23:32:46.164099+0000","last_fullsized":"2024-03-28T23:32:46.164099+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:13:27.723637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"76'1","reported_seq":15,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.124005+0000","last_change":"2024-03-28T23:32:44.121416+0000","last_active":"2024-03-28T23:32:46.124005+0000","last_peered":"2024-03-28T23:32:46.124005+0000","last_clean":"2024-03-28T23:32:46.124005+0000","last_became_active":"2024-03-28T23:32:44.121127+0000","last_became_peered":"2024-03-28T23:32:44.121127+0000","last_unstale":"2024-03-28T23:32:46.124005+0000","last_undegraded":"2024-03-28T23:32:46.124005+0000","last_fullsized":"2024-03-28T23:32:46.124005+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:27:27.910999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.122906+0000","last_change":"2024-03-28T23:32:33.051453+0000","last_active":"2024-03-28T23:32:46.122906+0000","last_peered":"2024-03-28T23:32:46.122906+0000","last_clean":"2024-03-28T23:32:46.122906+0000","last_became_active":"2024-03-28T23:32:33.051155+0000","last_became_peered":"2024-03-28T23:32:33.051155+0000","last_unstale":"2024-03-28T23:32:46.122906+0000","last_undegraded":"2024-03-28T23:32:46.122906+0000","last_fullsized":"2024-03-28T23:32:46.122906+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:36:03.951054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165820+0000","last_change":"2024-03-28T23:32:32.036979+0000","last_active":"2024-03-28T23:32:46.165820+0000","last_peered":"2024-03-28T23:32:46.165820+0000","last_clean":"2024-03-28T23:32:46.165820+0000","last_became_active":"2024-03-28T23:32:32.036710+0000","last_became_peered":"2024-03-28T23:32:32.036710+0000","last_unstale":"2024-03-28T23:32:46.165820+0000","last_undegraded":"2024-03-28T23:32:46.165820+0000","last_fullsized":"2024-03-28T23:32:46.165820+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:46.072852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164193+0000","last_change":"2024-03-28T23:32:45.115755+0000","last_active":"2024-03-28T23:32:46.164193+0000","last_peered":"2024-03-28T23:32:46.164193+0000","last_clean":"2024-03-28T23:32:46.164193+0000","last_became_active":"2024-03-28T23:32:45.115490+0000","last_became_peered":"2024-03-28T23:32:45.115490+0000","last_unstale":"2024-03-28T23:32:46.164193+0000","last_undegraded":"2024-03-28T23:32:46.164193+0000","last_fullsized":"2024-03-28T23:32:46.164193+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:56.559065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165532+0000","last_change":"2024-03-28T23:32:44.115740+0000","last_active":"2024-03-28T23:32:46.165532+0000","last_peered":"2024-03-28T23:32:46.165532+0000","last_clean":"2024-03-28T23:32:46.165532+0000","last_became_active":"2024-03-28T23:32:44.115341+0000","last_became_peered":"2024-03-28T23:32:44.115341+0000","last_unstale":"2024-03-28T23:32:46.165532+0000","last_undegraded":"2024-03-28T23:32:46.165532+0000","last_fullsized":"2024-03-28T23:32:46.165532+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:37:37.731237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163983+0000","last_change":"2024-03-28T23:32:33.041923+0000","last_active":"2024-03-28T23:32:46.163983+0000","last_peered":"2024-03-28T23:32:46.163983+0000","last_clean":"2024-03-28T23:32:46.163983+0000","last_became_active":"2024-03-28T23:32:33.041631+0000","last_became_peered":"2024-03-28T23:32:33.041631+0000","last_unstale":"2024-03-28T23:32:46.163983+0000","last_undegraded":"2024-03-28T23:32:46.163983+0000","last_fullsized":"2024-03-28T23:32:46.163983+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:32:08.473276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164223+0000","last_change":"2024-03-28T23:32:32.041563+0000","last_active":"2024-03-28T23:32:46.164223+0000","last_peered":"2024-03-28T23:32:46.164223+0000","last_clean":"2024-03-28T23:32:46.164223+0000","last_became_active":"2024-03-28T23:32:32.040424+0000","last_became_peered":"2024-03-28T23:32:32.040424+0000","last_unstale":"2024-03-28T23:32:46.164223+0000","last_undegraded":"2024-03-28T23:32:46.164223+0000","last_fullsized":"2024-03-28T23:32:46.164223+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:37:25.645113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166814+0000","last_change":"2024-03-28T23:32:45.127172+0000","last_active":"2024-03-28T23:32:46.166814+0000","last_peered":"2024-03-28T23:32:46.166814+0000","last_clean":"2024-03-28T23:32:46.166814+0000","last_became_active":"2024-03-28T23:32:45.126831+0000","last_became_peered":"2024-03-28T23:32:45.126831+0000","last_unstale":"2024-03-28T23:32:46.166814+0000","last_undegraded":"2024-03-28T23:32:46.166814+0000","last_fullsized":"2024-03-28T23:32:46.166814+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:51:23.531959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"76'8","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166372+0000","last_change":"2024-03-28T23:32:44.118040+0000","last_active":"2024-03-28T23:32:46.166372+0000","last_peered":"2024-03-28T23:32:46.166372+0000","last_clean":"2024-03-28T23:32:46.166372+0000","last_became_active":"2024-03-28T23:32:44.117536+0000","last_became_peered":"2024-03-28T23:32:44.117536+0000","last_unstale":"2024-03-28T23:32:46.166372+0000","last_undegraded":"2024-03-28T23:32:46.166372+0000","last_fullsized":"2024-03-28T23:32:46.166372+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:06:32.695104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.123305+0000","last_change":"2024-03-28T23:32:33.029986+0000","last_active":"2024-03-28T23:32:46.123305+0000","last_peered":"2024-03-28T23:32:46.123305+0000","last_clean":"2024-03-28T23:32:46.123305+0000","last_became_active":"2024-03-28T23:32:33.029413+0000","last_became_peered":"2024-03-28T23:32:33.029413+0000","last_unstale":"2024-03-28T23:32:46.123305+0000","last_undegraded":"2024-03-28T23:32:46.123305+0000","last_fullsized":"2024-03-28T23:32:46.123305+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:37:26.652511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.124150+0000","last_change":"2024-03-28T23:32:32.037039+0000","last_active":"2024-03-28T23:32:46.124150+0000","last_peered":"2024-03-28T23:32:46.124150+0000","last_clean":"2024-03-28T23:32:46.124150+0000","last_became_active":"2024-03-28T23:32:32.036687+0000","last_became_peered":"2024-03-28T23:32:32.036687+0000","last_unstale":"2024-03-28T23:32:46.124150+0000","last_undegraded":"2024-03-28T23:32:46.124150+0000","last_fullsized":"2024-03-28T23:32:46.124150+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:27:15.824876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166361+0000","last_change":"2024-03-28T23:32:45.125227+0000","last_active":"2024-03-28T23:32:46.166361+0000","last_peered":"2024-03-28T23:32:46.166361+0000","last_clean":"2024-03-28T23:32:46.166361+0000","last_became_active":"2024-03-28T23:32:45.123446+0000","last_became_peered":"2024-03-28T23:32:45.123446+0000","last_unstale":"2024-03-28T23:32:46.166361+0000","last_undegraded":"2024-03-28T23:32:46.166361+0000","last_fullsized":"2024-03-28T23:32:46.166361+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:56:06.278905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"76'2","reported_seq":16,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163772+0000","last_change":"2024-03-28T23:32:44.126156+0000","last_active":"2024-03-28T23:32:46.163772+0000","last_peered":"2024-03-28T23:32:46.163772+0000","last_clean":"2024-03-28T23:32:46.163772+0000","last_became_active":"2024-03-28T23:32:44.125789+0000","last_became_peered":"2024-03-28T23:32:44.125789+0000","last_unstale":"2024-03-28T23:32:46.163772+0000","last_undegraded":"2024-03-28T23:32:46.163772+0000","last_fullsized":"2024-03-28T23:32:46.163772+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:22:16.280489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166116+0000","last_change":"2024-03-28T23:32:33.035809+0000","last_active":"2024-03-28T23:32:46.166116+0000","last_peered":"2024-03-28T23:32:46.166116+0000","last_clean":"2024-03-28T23:32:46.166116+0000","last_became_active":"2024-03-28T23:32:33.035563+0000","last_became_peered":"2024-03-28T23:32:33.035563+0000","last_unstale":"2024-03-28T23:32:46.166116+0000","last_undegraded":"2024-03-28T23:32:46.166116+0000","last_fullsized":"2024-03-28T23:32:46.166116+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:54.194453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"72'3","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165106+0000","last_change":"2024-03-28T23:32:32.038230+0000","last_active":"2024-03-28T23:32:46.165106+0000","last_peered":"2024-03-28T23:32:46.165106+0000","last_clean":"2024-03-28T23:32:46.165106+0000","last_became_active":"2024-03-28T23:32:32.037681+0000","last_became_peered":"2024-03-28T23:32:32.037681+0000","last_unstale":"2024-03-28T23:32:46.165106+0000","last_undegraded":"2024-03-28T23:32:46.165106+0000","last_fullsized":"2024-03-28T23:32:46.165106+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:36:34.752866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.122883+0000","last_change":"2024-03-28T23:32:45.125115+0000","last_active":"2024-03-28T23:32:46.122883+0000","last_peered":"2024-03-28T23:32:46.122883+0000","last_clean":"2024-03-28T23:32:46.122883+0000","last_became_active":"2024-03-28T23:32:45.124812+0000","last_became_peered":"2024-03-28T23:32:45.124812+0000","last_unstale":"2024-03-28T23:32:46.122883+0000","last_undegraded":"2024-03-28T23:32:46.122883+0000","last_fullsized":"2024-03-28T23:32:46.122883+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:28:48.620361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164529+0000","last_change":"2024-03-28T23:32:44.125087+0000","last_active":"2024-03-28T23:32:46.164529+0000","last_peered":"2024-03-28T23:32:46.164529+0000","last_clean":"2024-03-28T23:32:46.164529+0000","last_became_active":"2024-03-28T23:32:44.124784+0000","last_became_peered":"2024-03-28T23:32:44.124784+0000","last_unstale":"2024-03-28T23:32:46.164529+0000","last_undegraded":"2024-03-28T23:32:46.164529+0000","last_fullsized":"2024-03-28T23:32:46.164529+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:48.876004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166454+0000","last_change":"2024-03-28T23:32:33.049499+0000","last_active":"2024-03-28T23:32:46.166454+0000","last_peered":"2024-03-28T23:32:46.166454+0000","last_clean":"2024-03-28T23:32:46.166454+0000","last_became_active":"2024-03-28T23:32:33.049233+0000","last_became_peered":"2024-03-28T23:32:33.049233+0000","last_unstale":"2024-03-28T23:32:46.166454+0000","last_undegraded":"2024-03-28T23:32:46.166454+0000","last_fullsized":"2024-03-28T23:32:46.166454+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:39:02.824323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"72'6","reported_seq":28,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.122280+0000","last_change":"2024-03-28T23:32:32.036835+0000","last_active":"2024-03-28T23:32:46.122280+0000","last_peered":"2024-03-28T23:32:46.122280+0000","last_clean":"2024-03-28T23:32:46.122280+0000","last_became_active":"2024-03-28T23:32:32.036557+0000","last_became_peered":"2024-03-28T23:32:32.036557+0000","last_unstale":"2024-03-28T23:32:46.122280+0000","last_undegraded":"2024-03-28T23:32:46.122280+0000","last_fullsized":"2024-03-28T23:32:46.122280+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:30.992807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166649+0000","last_change":"2024-03-28T23:32:45.135931+0000","last_active":"2024-03-28T23:32:46.166649+0000","last_peered":"2024-03-28T23:32:46.166649+0000","last_clean":"2024-03-28T23:32:46.166649+0000","last_became_active":"2024-03-28T23:32:45.135656+0000","last_became_peered":"2024-03-28T23:32:45.135656+0000","last_unstale":"2024-03-28T23:32:46.166649+0000","last_undegraded":"2024-03-28T23:32:46.166649+0000","last_fullsized":"2024-03-28T23:32:46.166649+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:40:28.334421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"76'3","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.124791+0000","last_change":"2024-03-28T23:32:44.127256+0000","last_active":"2024-03-28T23:32:46.124791+0000","last_peered":"2024-03-28T23:32:46.124791+0000","last_clean":"2024-03-28T23:32:46.124791+0000","last_became_active":"2024-03-28T23:32:44.126894+0000","last_became_peered":"2024-03-28T23:32:44.126894+0000","last_unstale":"2024-03-28T23:32:46.124791+0000","last_undegraded":"2024-03-28T23:32:46.124791+0000","last_fullsized":"2024-03-28T23:32:46.124791+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:58.429743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.123883+0000","last_change":"2024-03-28T23:32:33.033045+0000","last_active":"2024-03-28T23:32:46.123883+0000","last_peered":"2024-03-28T23:32:46.123883+0000","last_clean":"2024-03-28T23:32:46.123883+0000","last_became_active":"2024-03-28T23:32:33.032771+0000","last_became_peered":"2024-03-28T23:32:33.032771+0000","last_unstale":"2024-03-28T23:32:46.123883+0000","last_undegraded":"2024-03-28T23:32:46.123883+0000","last_fullsized":"2024-03-28T23:32:46.123883+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:43.341877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"72'2","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.124252+0000","last_change":"2024-03-28T23:32:32.032053+0000","last_active":"2024-03-28T23:32:46.124252+0000","last_peered":"2024-03-28T23:32:46.124252+0000","last_clean":"2024-03-28T23:32:46.124252+0000","last_became_active":"2024-03-28T23:32:32.031741+0000","last_became_peered":"2024-03-28T23:32:32.031741+0000","last_unstale":"2024-03-28T23:32:46.124252+0000","last_undegraded":"2024-03-28T23:32:46.124252+0000","last_fullsized":"2024-03-28T23:32:46.124252+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:20.456721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164046+0000","last_change":"2024-03-28T23:32:45.128166+0000","last_active":"2024-03-28T23:32:46.164046+0000","last_peered":"2024-03-28T23:32:46.164046+0000","last_clean":"2024-03-28T23:32:46.164046+0000","last_became_active":"2024-03-28T23:32:45.127851+0000","last_became_peered":"2024-03-28T23:32:45.127851+0000","last_unstale":"2024-03-28T23:32:46.164046+0000","last_undegraded":"2024-03-28T23:32:46.164046+0000","last_fullsized":"2024-03-28T23:32:46.164046+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:02:57.275018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"76'6","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165387+0000","last_change":"2024-03-28T23:32:44.124910+0000","last_active":"2024-03-28T23:32:46.165387+0000","last_peered":"2024-03-28T23:32:46.165387+0000","last_clean":"2024-03-28T23:32:46.165387+0000","last_became_active":"2024-03-28T23:32:44.124639+0000","last_became_peered":"2024-03-28T23:32:44.124639+0000","last_unstale":"2024-03-28T23:32:46.165387+0000","last_undegraded":"2024-03-28T23:32:46.165387+0000","last_fullsized":"2024-03-28T23:32:46.165387+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:20:17.013749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.122201+0000","last_change":"2024-03-28T23:32:33.039022+0000","last_active":"2024-03-28T23:32:46.122201+0000","last_peered":"2024-03-28T23:32:46.122201+0000","last_clean":"2024-03-28T23:32:46.122201+0000","last_became_active":"2024-03-28T23:32:33.038508+0000","last_became_peered":"2024-03-28T23:32:33.038508+0000","last_unstale":"2024-03-28T23:32:46.122201+0000","last_undegraded":"2024-03-28T23:32:46.122201+0000","last_fullsized":"2024-03-28T23:32:46.122201+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:25:45.189859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163472+0000","last_change":"2024-03-28T23:32:32.041042+0000","last_active":"2024-03-28T23:32:46.163472+0000","last_peered":"2024-03-28T23:32:46.163472+0000","last_clean":"2024-03-28T23:32:46.163472+0000","last_became_active":"2024-03-28T23:32:32.040686+0000","last_became_peered":"2024-03-28T23:32:32.040686+0000","last_unstale":"2024-03-28T23:32:46.163472+0000","last_undegraded":"2024-03-28T23:32:46.163472+0000","last_fullsized":"2024-03-28T23:32:46.163472+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:46.343619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.124428+0000","last_change":"2024-03-28T23:32:45.126834+0000","last_active":"2024-03-28T23:32:46.124428+0000","last_peered":"2024-03-28T23:32:46.124428+0000","last_clean":"2024-03-28T23:32:46.124428+0000","last_became_active":"2024-03-28T23:32:45.126572+0000","last_became_peered":"2024-03-28T23:32:45.126572+0000","last_unstale":"2024-03-28T23:32:46.124428+0000","last_undegraded":"2024-03-28T23:32:46.124428+0000","last_fullsized":"2024-03-28T23:32:46.124428+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:59:00.315700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.123842+0000","last_change":"2024-03-28T23:32:44.120268+0000","last_active":"2024-03-28T23:32:46.123842+0000","last_peered":"2024-03-28T23:32:46.123842+0000","last_clean":"2024-03-28T23:32:46.123842+0000","last_became_active":"2024-03-28T23:32:44.119951+0000","last_became_peered":"2024-03-28T23:32:44.119951+0000","last_unstale":"2024-03-28T23:32:46.123842+0000","last_undegraded":"2024-03-28T23:32:46.123842+0000","last_fullsized":"2024-03-28T23:32:46.123842+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:38:50.362657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.122431+0000","last_change":"2024-03-28T23:32:33.034262+0000","last_active":"2024-03-28T23:32:46.122431+0000","last_peered":"2024-03-28T23:32:46.122431+0000","last_clean":"2024-03-28T23:32:46.122431+0000","last_became_active":"2024-03-28T23:32:33.034003+0000","last_became_peered":"2024-03-28T23:32:33.034003+0000","last_unstale":"2024-03-28T23:32:46.122431+0000","last_undegraded":"2024-03-28T23:32:46.122431+0000","last_fullsized":"2024-03-28T23:32:46.122431+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:41.257241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163979+0000","last_change":"2024-03-28T23:32:32.039031+0000","last_active":"2024-03-28T23:32:46.163979+0000","last_peered":"2024-03-28T23:32:46.163979+0000","last_clean":"2024-03-28T23:32:46.163979+0000","last_became_active":"2024-03-28T23:32:32.038766+0000","last_became_peered":"2024-03-28T23:32:32.038766+0000","last_unstale":"2024-03-28T23:32:46.163979+0000","last_undegraded":"2024-03-28T23:32:46.163979+0000","last_fullsized":"2024-03-28T23:32:46.163979+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:13.872794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163941+0000","last_change":"2024-03-28T23:32:45.124189+0000","last_active":"2024-03-28T23:32:46.163941+0000","last_peered":"2024-03-28T23:32:46.163941+0000","last_clean":"2024-03-28T23:32:46.163941+0000","last_became_active":"2024-03-28T23:32:45.123900+0000","last_became_peered":"2024-03-28T23:32:45.123900+0000","last_unstale":"2024-03-28T23:32:46.163941+0000","last_undegraded":"2024-03-28T23:32:46.163941+0000","last_fullsized":"2024-03-28T23:32:46.163941+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:00:34.558942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166739+0000","last_change":"2024-03-28T23:32:44.119707+0000","last_active":"2024-03-28T23:32:46.166739+0000","last_peered":"2024-03-28T23:32:46.166739+0000","last_clean":"2024-03-28T23:32:46.166739+0000","last_became_active":"2024-03-28T23:32:44.119438+0000","last_became_peered":"2024-03-28T23:32:44.119438+0000","last_unstale":"2024-03-28T23:32:46.166739+0000","last_undegraded":"2024-03-28T23:32:46.166739+0000","last_fullsized":"2024-03-28T23:32:46.166739+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:22:16.280489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.123023+0000","last_change":"2024-03-28T23:32:33.043171+0000","last_active":"2024-03-28T23:32:46.123023+0000","last_peered":"2024-03-28T23:32:46.123023+0000","last_clean":"2024-03-28T23:32:46.123023+0000","last_became_active":"2024-03-28T23:32:33.042821+0000","last_became_peered":"2024-03-28T23:32:33.042821+0000","last_unstale":"2024-03-28T23:32:46.123023+0000","last_undegraded":"2024-03-28T23:32:46.123023+0000","last_fullsized":"2024-03-28T23:32:46.123023+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:21.878156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"72'8","reported_seq":30,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164138+0000","last_change":"2024-03-28T23:32:32.040166+0000","last_active":"2024-03-28T23:32:46.164138+0000","last_peered":"2024-03-28T23:32:46.164138+0000","last_clean":"2024-03-28T23:32:46.164138+0000","last_became_active":"2024-03-28T23:32:32.039560+0000","last_became_peered":"2024-03-28T23:32:32.039560+0000","last_unstale":"2024-03-28T23:32:46.164138+0000","last_undegraded":"2024-03-28T23:32:46.164138+0000","last_fullsized":"2024-03-28T23:32:46.164138+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:55.544945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.123191+0000","last_change":"2024-03-28T23:32:45.124155+0000","last_active":"2024-03-28T23:32:46.123191+0000","last_peered":"2024-03-28T23:32:46.123191+0000","last_clean":"2024-03-28T23:32:46.123191+0000","last_became_active":"2024-03-28T23:32:45.123804+0000","last_became_peered":"2024-03-28T23:32:45.123804+0000","last_unstale":"2024-03-28T23:32:46.123191+0000","last_undegraded":"2024-03-28T23:32:46.123191+0000","last_fullsized":"2024-03-28T23:32:46.123191+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:22:17.286216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.122993+0000","last_change":"2024-03-28T23:32:44.126116+0000","last_active":"2024-03-28T23:32:46.122993+0000","last_peered":"2024-03-28T23:32:46.122993+0000","last_clean":"2024-03-28T23:32:46.122993+0000","last_became_active":"2024-03-28T23:32:44.125761+0000","last_became_peered":"2024-03-28T23:32:44.125761+0000","last_unstale":"2024-03-28T23:32:46.122993+0000","last_undegraded":"2024-03-28T23:32:46.122993+0000","last_fullsized":"2024-03-28T23:32:46.122993+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:11.436906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164561+0000","last_change":"2024-03-28T23:32:33.043192+0000","last_active":"2024-03-28T23:32:46.164561+0000","last_peered":"2024-03-28T23:32:46.164561+0000","last_clean":"2024-03-28T23:32:46.164561+0000","last_became_active":"2024-03-28T23:32:33.042894+0000","last_became_peered":"2024-03-28T23:32:33.042894+0000","last_unstale":"2024-03-28T23:32:46.164561+0000","last_undegraded":"2024-03-28T23:32:46.164561+0000","last_fullsized":"2024-03-28T23:32:46.164561+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:32:08.473276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"72'1","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164229+0000","last_change":"2024-03-28T23:32:32.041892+0000","last_active":"2024-03-28T23:32:46.164229+0000","last_peered":"2024-03-28T23:32:46.164229+0000","last_clean":"2024-03-28T23:32:46.164229+0000","last_became_active":"2024-03-28T23:32:32.041533+0000","last_became_peered":"2024-03-28T23:32:32.041533+0000","last_unstale":"2024-03-28T23:32:46.164229+0000","last_undegraded":"2024-03-28T23:32:46.164229+0000","last_fullsized":"2024-03-28T23:32:46.164229+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:15.843269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165083+0000","last_change":"2024-03-28T23:32:45.125611+0000","last_active":"2024-03-28T23:32:46.165083+0000","last_peered":"2024-03-28T23:32:46.165083+0000","last_clean":"2024-03-28T23:32:46.165083+0000","last_became_active":"2024-03-28T23:32:45.125349+0000","last_became_peered":"2024-03-28T23:32:45.125349+0000","last_unstale":"2024-03-28T23:32:46.165083+0000","last_undegraded":"2024-03-28T23:32:46.165083+0000","last_fullsized":"2024-03-28T23:32:46.165083+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:29:40.900683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165617+0000","last_change":"2024-03-28T23:32:44.115585+0000","last_active":"2024-03-28T23:32:46.165617+0000","last_peered":"2024-03-28T23:32:46.165617+0000","last_clean":"2024-03-28T23:32:46.165617+0000","last_became_active":"2024-03-28T23:32:44.115308+0000","last_became_peered":"2024-03-28T23:32:44.115308+0000","last_unstale":"2024-03-28T23:32:46.165617+0000","last_undegraded":"2024-03-28T23:32:46.165617+0000","last_fullsized":"2024-03-28T23:32:46.165617+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:18:00.809113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163885+0000","last_change":"2024-03-28T23:32:33.046899+0000","last_active":"2024-03-28T23:32:46.163885+0000","last_peered":"2024-03-28T23:32:46.163885+0000","last_clean":"2024-03-28T23:32:46.163885+0000","last_became_active":"2024-03-28T23:32:33.046579+0000","last_became_peered":"2024-03-28T23:32:33.046579+0000","last_unstale":"2024-03-28T23:32:46.163885+0000","last_undegraded":"2024-03-28T23:32:46.163885+0000","last_fullsized":"2024-03-28T23:32:46.163885+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:08:06.339997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"72'2","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.124657+0000","last_change":"2024-03-28T23:32:32.031135+0000","last_active":"2024-03-28T23:32:46.124657+0000","last_peered":"2024-03-28T23:32:46.124657+0000","last_clean":"2024-03-28T23:32:46.124657+0000","last_became_active":"2024-03-28T23:32:32.030796+0000","last_became_peered":"2024-03-28T23:32:32.030796+0000","last_unstale":"2024-03-28T23:32:46.124657+0000","last_undegraded":"2024-03-28T23:32:46.124657+0000","last_fullsized":"2024-03-28T23:32:46.124657+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:18:21.816565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163601+0000","last_change":"2024-03-28T23:32:45.134234+0000","last_active":"2024-03-28T23:32:46.163601+0000","last_peered":"2024-03-28T23:32:46.163601+0000","last_clean":"2024-03-28T23:32:46.163601+0000","last_became_active":"2024-03-28T23:32:45.134012+0000","last_became_peered":"2024-03-28T23:32:45.134012+0000","last_unstale":"2024-03-28T23:32:46.163601+0000","last_undegraded":"2024-03-28T23:32:46.163601+0000","last_fullsized":"2024-03-28T23:32:46.163601+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:44.843761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"76'2","reported_seq":16,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.123453+0000","last_change":"2024-03-28T23:32:44.116752+0000","last_active":"2024-03-28T23:32:46.123453+0000","last_peered":"2024-03-28T23:32:46.123453+0000","last_clean":"2024-03-28T23:32:46.123453+0000","last_became_active":"2024-03-28T23:32:44.116381+0000","last_became_peered":"2024-03-28T23:32:44.116381+0000","last_unstale":"2024-03-28T23:32:46.123453+0000","last_undegraded":"2024-03-28T23:32:46.123453+0000","last_fullsized":"2024-03-28T23:32:46.123453+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:32:19.552001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164362+0000","last_change":"2024-03-28T23:32:33.053549+0000","last_active":"2024-03-28T23:32:46.164362+0000","last_peered":"2024-03-28T23:32:46.164362+0000","last_clean":"2024-03-28T23:32:46.164362+0000","last_became_active":"2024-03-28T23:32:33.053229+0000","last_became_peered":"2024-03-28T23:32:33.053229+0000","last_unstale":"2024-03-28T23:32:46.164362+0000","last_undegraded":"2024-03-28T23:32:46.164362+0000","last_fullsized":"2024-03-28T23:32:46.164362+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:41:02.517121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"72'1","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165264+0000","last_change":"2024-03-28T23:32:32.038261+0000","last_active":"2024-03-28T23:32:46.165264+0000","last_peered":"2024-03-28T23:32:46.165264+0000","last_clean":"2024-03-28T23:32:46.165264+0000","last_became_active":"2024-03-28T23:32:32.037971+0000","last_became_peered":"2024-03-28T23:32:32.037971+0000","last_unstale":"2024-03-28T23:32:46.165264+0000","last_undegraded":"2024-03-28T23:32:46.165264+0000","last_fullsized":"2024-03-28T23:32:46.165264+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:25:10.296139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.122852+0000","last_change":"2024-03-28T23:32:45.130495+0000","last_active":"2024-03-28T23:32:46.122852+0000","last_peered":"2024-03-28T23:32:46.122852+0000","last_clean":"2024-03-28T23:32:46.122852+0000","last_became_active":"2024-03-28T23:32:45.130172+0000","last_became_peered":"2024-03-28T23:32:45.130172+0000","last_unstale":"2024-03-28T23:32:46.122852+0000","last_undegraded":"2024-03-28T23:32:46.122852+0000","last_fullsized":"2024-03-28T23:32:46.122852+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:40.547945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164276+0000","last_change":"2024-03-28T23:32:44.115062+0000","last_active":"2024-03-28T23:32:46.164276+0000","last_peered":"2024-03-28T23:32:46.164276+0000","last_clean":"2024-03-28T23:32:46.164276+0000","last_became_active":"2024-03-28T23:32:44.114795+0000","last_became_peered":"2024-03-28T23:32:44.114795+0000","last_unstale":"2024-03-28T23:32:46.164276+0000","last_undegraded":"2024-03-28T23:32:46.164276+0000","last_fullsized":"2024-03-28T23:32:46.164276+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:20:13.611155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164170+0000","last_change":"2024-03-28T23:32:33.043954+0000","last_active":"2024-03-28T23:32:46.164170+0000","last_peered":"2024-03-28T23:32:46.164170+0000","last_clean":"2024-03-28T23:32:46.164170+0000","last_became_active":"2024-03-28T23:32:33.043649+0000","last_became_peered":"2024-03-28T23:32:33.043649+0000","last_unstale":"2024-03-28T23:32:46.164170+0000","last_undegraded":"2024-03-28T23:32:46.164170+0000","last_fullsized":"2024-03-28T23:32:46.164170+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:00.358180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.167097+0000","last_change":"2024-03-28T23:32:32.033720+0000","last_active":"2024-03-28T23:32:46.167097+0000","last_peered":"2024-03-28T23:32:46.167097+0000","last_clean":"2024-03-28T23:32:46.167097+0000","last_became_active":"2024-03-28T23:32:32.033422+0000","last_became_peered":"2024-03-28T23:32:32.033422+0000","last_unstale":"2024-03-28T23:32:46.167097+0000","last_undegraded":"2024-03-28T23:32:46.167097+0000","last_fullsized":"2024-03-28T23:32:46.167097+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:51:10.440109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.124492+0000","last_change":"2024-03-28T23:32:45.116280+0000","last_active":"2024-03-28T23:32:46.124492+0000","last_peered":"2024-03-28T23:32:46.124492+0000","last_clean":"2024-03-28T23:32:46.124492+0000","last_became_active":"2024-03-28T23:32:45.114472+0000","last_became_peered":"2024-03-28T23:32:45.114472+0000","last_unstale":"2024-03-28T23:32:46.124492+0000","last_undegraded":"2024-03-28T23:32:46.124492+0000","last_fullsized":"2024-03-28T23:32:46.124492+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:45:10.868624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"76'1","reported_seq":15,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.124859+0000","last_change":"2024-03-28T23:32:44.124559+0000","last_active":"2024-03-28T23:32:46.124859+0000","last_peered":"2024-03-28T23:32:46.124859+0000","last_clean":"2024-03-28T23:32:46.124859+0000","last_became_active":"2024-03-28T23:32:44.124282+0000","last_became_peered":"2024-03-28T23:32:44.124282+0000","last_unstale":"2024-03-28T23:32:46.124859+0000","last_undegraded":"2024-03-28T23:32:46.124859+0000","last_fullsized":"2024-03-28T23:32:46.124859+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:59:07.631069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165547+0000","last_change":"2024-03-28T23:32:33.045618+0000","last_active":"2024-03-28T23:32:46.165547+0000","last_peered":"2024-03-28T23:32:46.165547+0000","last_clean":"2024-03-28T23:32:46.165547+0000","last_became_active":"2024-03-28T23:32:33.045324+0000","last_became_peered":"2024-03-28T23:32:33.045324+0000","last_unstale":"2024-03-28T23:32:46.165547+0000","last_undegraded":"2024-03-28T23:32:46.165547+0000","last_fullsized":"2024-03-28T23:32:46.165547+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:46:43.644886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"72'2","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164337+0000","last_change":"2024-03-28T23:32:32.031574+0000","last_active":"2024-03-28T23:32:46.164337+0000","last_peered":"2024-03-28T23:32:46.164337+0000","last_clean":"2024-03-28T23:32:46.164337+0000","last_became_active":"2024-03-28T23:32:32.031310+0000","last_became_peered":"2024-03-28T23:32:32.031310+0000","last_unstale":"2024-03-28T23:32:46.164337+0000","last_undegraded":"2024-03-28T23:32:46.164337+0000","last_fullsized":"2024-03-28T23:32:46.164337+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:48:07.073481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165456+0000","last_change":"2024-03-28T23:32:45.130313+0000","last_active":"2024-03-28T23:32:46.165456+0000","last_peered":"2024-03-28T23:32:46.165456+0000","last_clean":"2024-03-28T23:32:46.165456+0000","last_became_active":"2024-03-28T23:32:45.130067+0000","last_became_peered":"2024-03-28T23:32:45.130067+0000","last_unstale":"2024-03-28T23:32:46.165456+0000","last_undegraded":"2024-03-28T23:32:46.165456+0000","last_fullsized":"2024-03-28T23:32:46.165456+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:52:23.046683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"76'1","reported_seq":15,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.122744+0000","last_change":"2024-03-28T23:32:44.114144+0000","last_active":"2024-03-28T23:32:46.122744+0000","last_peered":"2024-03-28T23:32:46.122744+0000","last_clean":"2024-03-28T23:32:46.122744+0000","last_became_active":"2024-03-28T23:32:44.113829+0000","last_became_peered":"2024-03-28T23:32:44.113829+0000","last_unstale":"2024-03-28T23:32:46.122744+0000","last_undegraded":"2024-03-28T23:32:46.122744+0000","last_fullsized":"2024-03-28T23:32:46.122744+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:39:50.192183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165916+0000","last_change":"2024-03-28T23:32:33.040983+0000","last_active":"2024-03-28T23:32:46.165916+0000","last_peered":"2024-03-28T23:32:46.165916+0000","last_clean":"2024-03-28T23:32:46.165916+0000","last_became_active":"2024-03-28T23:32:33.040693+0000","last_became_peered":"2024-03-28T23:32:33.040693+0000","last_unstale":"2024-03-28T23:32:46.165916+0000","last_undegraded":"2024-03-28T23:32:46.165916+0000","last_fullsized":"2024-03-28T23:32:46.165916+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:22:05.201764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165375+0000","last_change":"2024-03-28T23:32:32.036998+0000","last_active":"2024-03-28T23:32:46.165375+0000","last_peered":"2024-03-28T23:32:46.165375+0000","last_clean":"2024-03-28T23:32:46.165375+0000","last_became_active":"2024-03-28T23:32:32.036733+0000","last_became_peered":"2024-03-28T23:32:32.036733+0000","last_unstale":"2024-03-28T23:32:46.165375+0000","last_undegraded":"2024-03-28T23:32:46.165375+0000","last_fullsized":"2024-03-28T23:32:46.165375+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:15:42.334479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.121684+0000","last_change":"2024-03-28T23:32:45.125756+0000","last_active":"2024-03-28T23:32:46.121684+0000","last_peered":"2024-03-28T23:32:46.121684+0000","last_clean":"2024-03-28T23:32:46.121684+0000","last_became_active":"2024-03-28T23:32:45.125503+0000","last_became_peered":"2024-03-28T23:32:45.125503+0000","last_unstale":"2024-03-28T23:32:46.121684+0000","last_undegraded":"2024-03-28T23:32:46.121684+0000","last_fullsized":"2024-03-28T23:32:46.121684+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:05.597289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"76'2","reported_seq":16,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163961+0000","last_change":"2024-03-28T23:32:44.115154+0000","last_active":"2024-03-28T23:32:46.163961+0000","last_peered":"2024-03-28T23:32:46.163961+0000","last_clean":"2024-03-28T23:32:46.163961+0000","last_became_active":"2024-03-28T23:32:44.114869+0000","last_became_peered":"2024-03-28T23:32:44.114869+0000","last_unstale":"2024-03-28T23:32:46.163961+0000","last_undegraded":"2024-03-28T23:32:46.163961+0000","last_fullsized":"2024-03-28T23:32:46.163961+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:08:15.275133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.123128+0000","last_change":"2024-03-28T23:32:33.041232+0000","last_active":"2024-03-28T23:32:46.123128+0000","last_peered":"2024-03-28T23:32:46.123128+0000","last_clean":"2024-03-28T23:32:46.123128+0000","last_became_active":"2024-03-28T23:32:33.040132+0000","last_became_peered":"2024-03-28T23:32:33.040132+0000","last_unstale":"2024-03-28T23:32:46.123128+0000","last_undegraded":"2024-03-28T23:32:46.123128+0000","last_fullsized":"2024-03-28T23:32:46.123128+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:39:02.824323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164650+0000","last_change":"2024-03-28T23:32:32.036416+0000","last_active":"2024-03-28T23:32:46.164650+0000","last_peered":"2024-03-28T23:32:46.164650+0000","last_clean":"2024-03-28T23:32:46.164650+0000","last_became_active":"2024-03-28T23:32:32.036131+0000","last_became_peered":"2024-03-28T23:32:32.036131+0000","last_unstale":"2024-03-28T23:32:46.164650+0000","last_undegraded":"2024-03-28T23:32:46.164650+0000","last_fullsized":"2024-03-28T23:32:46.164650+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:13:33.914342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.121335+0000","last_change":"2024-03-28T23:32:45.133379+0000","last_active":"2024-03-28T23:32:46.121335+0000","last_peered":"2024-03-28T23:32:46.121335+0000","last_clean":"2024-03-28T23:32:46.121335+0000","last_became_active":"2024-03-28T23:32:45.133062+0000","last_became_peered":"2024-03-28T23:32:45.133062+0000","last_unstale":"2024-03-28T23:32:46.121335+0000","last_undegraded":"2024-03-28T23:32:46.121335+0000","last_fullsized":"2024-03-28T23:32:46.121335+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:43:48.031444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.121005+0000","last_change":"2024-03-28T23:32:44.116820+0000","last_active":"2024-03-28T23:32:46.121005+0000","last_peered":"2024-03-28T23:32:46.121005+0000","last_clean":"2024-03-28T23:32:46.121005+0000","last_became_active":"2024-03-28T23:32:44.116540+0000","last_became_peered":"2024-03-28T23:32:44.116540+0000","last_unstale":"2024-03-28T23:32:46.121005+0000","last_undegraded":"2024-03-28T23:32:46.121005+0000","last_fullsized":"2024-03-28T23:32:46.121005+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:13:46.000465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.123039+0000","last_change":"2024-03-28T23:32:33.035015+0000","last_active":"2024-03-28T23:32:46.123039+0000","last_peered":"2024-03-28T23:32:46.123039+0000","last_clean":"2024-03-28T23:32:46.123039+0000","last_became_active":"2024-03-28T23:32:33.034730+0000","last_became_peered":"2024-03-28T23:32:33.034730+0000","last_unstale":"2024-03-28T23:32:46.123039+0000","last_undegraded":"2024-03-28T23:32:46.123039+0000","last_fullsized":"2024-03-28T23:32:46.123039+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:09.477928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"72'2","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.124032+0000","last_change":"2024-03-28T23:32:32.032596+0000","last_active":"2024-03-28T23:32:46.124032+0000","last_peered":"2024-03-28T23:32:46.124032+0000","last_clean":"2024-03-28T23:32:46.124032+0000","last_became_active":"2024-03-28T23:32:32.032336+0000","last_became_peered":"2024-03-28T23:32:32.032336+0000","last_unstale":"2024-03-28T23:32:46.124032+0000","last_undegraded":"2024-03-28T23:32:46.124032+0000","last_fullsized":"2024-03-28T23:32:46.124032+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:28:25.356704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164807+0000","last_change":"2024-03-28T23:32:45.123614+0000","last_active":"2024-03-28T23:32:46.164807+0000","last_peered":"2024-03-28T23:32:46.164807+0000","last_clean":"2024-03-28T23:32:46.164807+0000","last_became_active":"2024-03-28T23:32:45.123303+0000","last_became_peered":"2024-03-28T23:32:45.123303+0000","last_unstale":"2024-03-28T23:32:46.164807+0000","last_undegraded":"2024-03-28T23:32:46.164807+0000","last_fullsized":"2024-03-28T23:32:46.164807+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:24.670526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163350+0000","last_change":"2024-03-28T23:32:44.116649+0000","last_active":"2024-03-28T23:32:46.163350+0000","last_peered":"2024-03-28T23:32:46.163350+0000","last_clean":"2024-03-28T23:32:46.163350+0000","last_became_active":"2024-03-28T23:32:44.116369+0000","last_became_peered":"2024-03-28T23:32:44.116369+0000","last_unstale":"2024-03-28T23:32:46.163350+0000","last_undegraded":"2024-03-28T23:32:46.163350+0000","last_fullsized":"2024-03-28T23:32:46.163350+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:39.542218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.121655+0000","last_change":"2024-03-28T23:32:33.044243+0000","last_active":"2024-03-28T23:32:46.121655+0000","last_peered":"2024-03-28T23:32:46.121655+0000","last_clean":"2024-03-28T23:32:46.121655+0000","last_became_active":"2024-03-28T23:32:33.040218+0000","last_became_peered":"2024-03-28T23:32:33.040218+0000","last_unstale":"2024-03-28T23:32:46.121655+0000","last_undegraded":"2024-03-28T23:32:46.121655+0000","last_fullsized":"2024-03-28T23:32:46.121655+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:38:51.513907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"73'1","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166004+0000","last_change":"2024-03-28T23:32:32.041586+0000","last_active":"2024-03-28T23:32:46.166004+0000","last_peered":"2024-03-28T23:32:46.166004+0000","last_clean":"2024-03-28T23:32:46.166004+0000","last_became_active":"2024-03-28T23:32:32.041327+0000","last_became_peered":"2024-03-28T23:32:32.041327+0000","last_unstale":"2024-03-28T23:32:46.166004+0000","last_undegraded":"2024-03-28T23:32:46.166004+0000","last_fullsized":"2024-03-28T23:32:46.166004+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:06:20.608980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165537+0000","last_change":"2024-03-28T23:32:45.134577+0000","last_active":"2024-03-28T23:32:46.165537+0000","last_peered":"2024-03-28T23:32:46.165537+0000","last_clean":"2024-03-28T23:32:46.165537+0000","last_became_active":"2024-03-28T23:32:45.134292+0000","last_became_peered":"2024-03-28T23:32:45.134292+0000","last_unstale":"2024-03-28T23:32:46.165537+0000","last_undegraded":"2024-03-28T23:32:46.165537+0000","last_fullsized":"2024-03-28T23:32:46.165537+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:41:13.065560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164952+0000","last_change":"2024-03-28T23:32:44.125883+0000","last_active":"2024-03-28T23:32:46.164952+0000","last_peered":"2024-03-28T23:32:46.164952+0000","last_clean":"2024-03-28T23:32:46.164952+0000","last_became_active":"2024-03-28T23:32:44.125578+0000","last_became_peered":"2024-03-28T23:32:44.125578+0000","last_unstale":"2024-03-28T23:32:46.164952+0000","last_undegraded":"2024-03-28T23:32:46.164952+0000","last_fullsized":"2024-03-28T23:32:46.164952+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:38:36.566643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.124584+0000","last_change":"2024-03-28T23:32:33.042272+0000","last_active":"2024-03-28T23:32:46.124584+0000","last_peered":"2024-03-28T23:32:46.124584+0000","last_clean":"2024-03-28T23:32:46.124584+0000","last_became_active":"2024-03-28T23:32:33.037633+0000","last_became_peered":"2024-03-28T23:32:33.037633+0000","last_unstale":"2024-03-28T23:32:46.124584+0000","last_undegraded":"2024-03-28T23:32:46.124584+0000","last_fullsized":"2024-03-28T23:32:46.124584+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:45.773838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164069+0000","last_change":"2024-03-28T23:32:32.037001+0000","last_active":"2024-03-28T23:32:46.164069+0000","last_peered":"2024-03-28T23:32:46.164069+0000","last_clean":"2024-03-28T23:32:46.164069+0000","last_became_active":"2024-03-28T23:32:32.036739+0000","last_became_peered":"2024-03-28T23:32:32.036739+0000","last_unstale":"2024-03-28T23:32:46.164069+0000","last_undegraded":"2024-03-28T23:32:46.164069+0000","last_fullsized":"2024-03-28T23:32:46.164069+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:06:20.608980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165291+0000","last_change":"2024-03-28T23:32:45.126588+0000","last_active":"2024-03-28T23:32:46.165291+0000","last_peered":"2024-03-28T23:32:46.165291+0000","last_clean":"2024-03-28T23:32:46.165291+0000","last_became_active":"2024-03-28T23:32:45.126348+0000","last_became_peered":"2024-03-28T23:32:45.126348+0000","last_unstale":"2024-03-28T23:32:46.165291+0000","last_undegraded":"2024-03-28T23:32:46.165291+0000","last_fullsized":"2024-03-28T23:32:46.165291+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:00:34.558942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.123030+0000","last_change":"2024-03-28T23:32:44.116971+0000","last_active":"2024-03-28T23:32:46.123030+0000","last_peered":"2024-03-28T23:32:46.123030+0000","last_clean":"2024-03-28T23:32:46.123030+0000","last_became_active":"2024-03-28T23:32:44.116371+0000","last_became_peered":"2024-03-28T23:32:44.116371+0000","last_unstale":"2024-03-28T23:32:46.123030+0000","last_undegraded":"2024-03-28T23:32:46.123030+0000","last_fullsized":"2024-03-28T23:32:46.123030+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:46.133761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165286+0000","last_change":"2024-03-28T23:32:33.041123+0000","last_active":"2024-03-28T23:32:46.165286+0000","last_peered":"2024-03-28T23:32:46.165286+0000","last_clean":"2024-03-28T23:32:46.165286+0000","last_became_active":"2024-03-28T23:32:33.040505+0000","last_became_peered":"2024-03-28T23:32:33.040505+0000","last_unstale":"2024-03-28T23:32:46.165286+0000","last_undegraded":"2024-03-28T23:32:46.165286+0000","last_fullsized":"2024-03-28T23:32:46.165286+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:55:54.194453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163743+0000","last_change":"2024-03-28T23:32:32.036857+0000","last_active":"2024-03-28T23:32:46.163743+0000","last_peered":"2024-03-28T23:32:46.163743+0000","last_clean":"2024-03-28T23:32:46.163743+0000","last_became_active":"2024-03-28T23:32:32.036576+0000","last_became_peered":"2024-03-28T23:32:32.036576+0000","last_unstale":"2024-03-28T23:32:46.163743+0000","last_undegraded":"2024-03-28T23:32:46.163743+0000","last_fullsized":"2024-03-28T23:32:46.163743+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:43.073425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165569+0000","last_change":"2024-03-28T23:32:45.133481+0000","last_active":"2024-03-28T23:32:46.165569+0000","last_peered":"2024-03-28T23:32:46.165569+0000","last_clean":"2024-03-28T23:32:46.165569+0000","last_became_active":"2024-03-28T23:32:45.132954+0000","last_became_peered":"2024-03-28T23:32:45.132954+0000","last_unstale":"2024-03-28T23:32:46.165569+0000","last_undegraded":"2024-03-28T23:32:46.165569+0000","last_fullsized":"2024-03-28T23:32:46.165569+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:25.413469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.124873+0000","last_change":"2024-03-28T23:32:44.126706+0000","last_active":"2024-03-28T23:32:46.124873+0000","last_peered":"2024-03-28T23:32:46.124873+0000","last_clean":"2024-03-28T23:32:46.124873+0000","last_became_active":"2024-03-28T23:32:44.126400+0000","last_became_peered":"2024-03-28T23:32:44.126400+0000","last_unstale":"2024-03-28T23:32:46.124873+0000","last_undegraded":"2024-03-28T23:32:46.124873+0000","last_fullsized":"2024-03-28T23:32:46.124873+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:13:27.942873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.121916+0000","last_change":"2024-03-28T23:32:33.041769+0000","last_active":"2024-03-28T23:32:46.121916+0000","last_peered":"2024-03-28T23:32:46.121916+0000","last_clean":"2024-03-28T23:32:46.121916+0000","last_became_active":"2024-03-28T23:32:33.041475+0000","last_became_peered":"2024-03-28T23:32:33.041475+0000","last_unstale":"2024-03-28T23:32:46.121916+0000","last_undegraded":"2024-03-28T23:32:46.121916+0000","last_fullsized":"2024-03-28T23:32:46.121916+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:38.859031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.121529+0000","last_change":"2024-03-28T23:32:32.037824+0000","last_active":"2024-03-28T23:32:46.121529+0000","last_peered":"2024-03-28T23:32:46.121529+0000","last_clean":"2024-03-28T23:32:46.121529+0000","last_became_active":"2024-03-28T23:32:32.037550+0000","last_became_peered":"2024-03-28T23:32:32.037550+0000","last_unstale":"2024-03-28T23:32:46.121529+0000","last_undegraded":"2024-03-28T23:32:46.121529+0000","last_fullsized":"2024-03-28T23:32:46.121529+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:43.467214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.123730+0000","last_change":"2024-03-28T23:32:45.128584+0000","last_active":"2024-03-28T23:32:46.123730+0000","last_peered":"2024-03-28T23:32:46.123730+0000","last_clean":"2024-03-28T23:32:46.123730+0000","last_became_active":"2024-03-28T23:32:45.128228+0000","last_became_peered":"2024-03-28T23:32:45.128228+0000","last_unstale":"2024-03-28T23:32:46.123730+0000","last_undegraded":"2024-03-28T23:32:46.123730+0000","last_fullsized":"2024-03-28T23:32:46.123730+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:08:16.280860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.121770+0000","last_change":"2024-03-28T23:32:44.117095+0000","last_active":"2024-03-28T23:32:46.121770+0000","last_peered":"2024-03-28T23:32:46.121770+0000","last_clean":"2024-03-28T23:32:46.121770+0000","last_became_active":"2024-03-28T23:32:44.116824+0000","last_became_peered":"2024-03-28T23:32:44.116824+0000","last_unstale":"2024-03-28T23:32:46.121770+0000","last_undegraded":"2024-03-28T23:32:46.121770+0000","last_fullsized":"2024-03-28T23:32:46.121770+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:27.929392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.122235+0000","last_change":"2024-03-28T23:32:33.048966+0000","last_active":"2024-03-28T23:32:46.122235+0000","last_peered":"2024-03-28T23:32:46.122235+0000","last_clean":"2024-03-28T23:32:46.122235+0000","last_became_active":"2024-03-28T23:32:33.048362+0000","last_became_peered":"2024-03-28T23:32:33.048362+0000","last_unstale":"2024-03-28T23:32:46.122235+0000","last_undegraded":"2024-03-28T23:32:46.122235+0000","last_fullsized":"2024-03-28T23:32:46.122235+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:25.849919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164445+0000","last_change":"2024-03-28T23:32:32.038885+0000","last_active":"2024-03-28T23:32:46.164445+0000","last_peered":"2024-03-28T23:32:46.164445+0000","last_clean":"2024-03-28T23:32:46.164445+0000","last_became_active":"2024-03-28T23:32:32.038364+0000","last_became_peered":"2024-03-28T23:32:32.038364+0000","last_unstale":"2024-03-28T23:32:46.164445+0000","last_undegraded":"2024-03-28T23:32:46.164445+0000","last_fullsized":"2024-03-28T23:32:46.164445+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:16:06.156877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.122075+0000","last_change":"2024-03-28T23:32:45.121440+0000","last_active":"2024-03-28T23:32:46.122075+0000","last_peered":"2024-03-28T23:32:46.122075+0000","last_clean":"2024-03-28T23:32:46.122075+0000","last_became_active":"2024-03-28T23:32:45.121152+0000","last_became_peered":"2024-03-28T23:32:45.121152+0000","last_unstale":"2024-03-28T23:32:46.122075+0000","last_undegraded":"2024-03-28T23:32:46.122075+0000","last_fullsized":"2024-03-28T23:32:46.122075+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:32:20.557728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"76'2","reported_seq":16,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164081+0000","last_change":"2024-03-28T23:32:44.114483+0000","last_active":"2024-03-28T23:32:46.164081+0000","last_peered":"2024-03-28T23:32:46.164081+0000","last_clean":"2024-03-28T23:32:46.164081+0000","last_became_active":"2024-03-28T23:32:44.114228+0000","last_became_peered":"2024-03-28T23:32:44.114228+0000","last_unstale":"2024-03-28T23:32:46.164081+0000","last_undegraded":"2024-03-28T23:32:46.164081+0000","last_fullsized":"2024-03-28T23:32:46.164081+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:03:01.130951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.165041+0000","last_change":"2024-03-28T23:32:33.049584+0000","last_active":"2024-03-28T23:32:46.165041+0000","last_peered":"2024-03-28T23:32:46.165041+0000","last_clean":"2024-03-28T23:32:46.165041+0000","last_became_active":"2024-03-28T23:32:33.049342+0000","last_became_peered":"2024-03-28T23:32:33.049342+0000","last_unstale":"2024-03-28T23:32:46.165041+0000","last_undegraded":"2024-03-28T23:32:46.165041+0000","last_fullsized":"2024-03-28T23:32:46.165041+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:49.711750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.122049+0000","last_change":"2024-03-28T23:32:32.034044+0000","last_active":"2024-03-28T23:32:46.122049+0000","last_peered":"2024-03-28T23:32:46.122049+0000","last_clean":"2024-03-28T23:32:46.122049+0000","last_became_active":"2024-03-28T23:32:32.033779+0000","last_became_peered":"2024-03-28T23:32:32.033779+0000","last_unstale":"2024-03-28T23:32:46.122049+0000","last_undegraded":"2024-03-28T23:32:46.122049+0000","last_fullsized":"2024-03-28T23:32:46.122049+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:56:22.159708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164295+0000","last_change":"2024-03-28T23:32:45.130071+0000","last_active":"2024-03-28T23:32:46.164295+0000","last_peered":"2024-03-28T23:32:46.164295+0000","last_clean":"2024-03-28T23:32:46.164295+0000","last_became_active":"2024-03-28T23:32:45.129682+0000","last_became_peered":"2024-03-28T23:32:45.129682+0000","last_unstale":"2024-03-28T23:32:46.164295+0000","last_undegraded":"2024-03-28T23:32:46.164295+0000","last_fullsized":"2024-03-28T23:32:46.164295+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:18:52.487805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"77'1","reported_seq":15,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.124195+0000","last_change":"2024-03-28T23:32:44.119649+0000","last_active":"2024-03-28T23:32:46.124195+0000","last_peered":"2024-03-28T23:32:46.124195+0000","last_clean":"2024-03-28T23:32:46.124195+0000","last_became_active":"2024-03-28T23:32:44.119370+0000","last_became_peered":"2024-03-28T23:32:44.119370+0000","last_unstale":"2024-03-28T23:32:46.124195+0000","last_undegraded":"2024-03-28T23:32:46.124195+0000","last_fullsized":"2024-03-28T23:32:46.124195+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:13:55.159548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164195+0000","last_change":"2024-03-28T23:32:33.041180+0000","last_active":"2024-03-28T23:32:46.164195+0000","last_peered":"2024-03-28T23:32:46.164195+0000","last_clean":"2024-03-28T23:32:46.164195+0000","last_became_active":"2024-03-28T23:32:33.040922+0000","last_became_peered":"2024-03-28T23:32:33.040922+0000","last_unstale":"2024-03-28T23:32:46.164195+0000","last_undegraded":"2024-03-28T23:32:46.164195+0000","last_fullsized":"2024-03-28T23:32:46.164195+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:25:45.189859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166485+0000","last_change":"2024-03-28T23:32:32.034328+0000","last_active":"2024-03-28T23:32:46.166485+0000","last_peered":"2024-03-28T23:32:46.166485+0000","last_clean":"2024-03-28T23:32:46.166485+0000","last_became_active":"2024-03-28T23:32:32.034057+0000","last_became_peered":"2024-03-28T23:32:32.034057+0000","last_unstale":"2024-03-28T23:32:46.166485+0000","last_undegraded":"2024-03-28T23:32:46.166485+0000","last_fullsized":"2024-03-28T23:32:46.166485+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:54:12.775407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.164085+0000","last_change":"2024-03-28T23:32:45.122735+0000","last_active":"2024-03-28T23:32:46.164085+0000","last_peered":"2024-03-28T23:32:46.164085+0000","last_clean":"2024-03-28T23:32:46.164085+0000","last_became_active":"2024-03-28T23:32:45.122406+0000","last_became_peered":"2024-03-28T23:32:45.122406+0000","last_unstale":"2024-03-28T23:32:46.164085+0000","last_undegraded":"2024-03-28T23:32:46.164085+0000","last_fullsized":"2024-03-28T23:32:46.164085+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:44.098119+0000","last_clean_scrub_stamp":"2024-03-28T23:32:44.098119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:40:28.334421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166961+0000","last_change":"2024-03-28T23:32:44.115020+0000","last_active":"2024-03-28T23:32:46.166961+0000","last_peered":"2024-03-28T23:32:46.166961+0000","last_clean":"2024-03-28T23:32:46.166961+0000","last_became_active":"2024-03-28T23:32:44.114737+0000","last_became_peered":"2024-03-28T23:32:44.114737+0000","last_unstale":"2024-03-28T23:32:46.166961+0000","last_undegraded":"2024-03-28T23:32:46.166961+0000","last_fullsized":"2024-03-28T23:32:46.166961+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:43.092392+0000","last_clean_scrub_stamp":"2024-03-28T23:32:43.092392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:30:23.848543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163952+0000","last_change":"2024-03-28T23:32:33.038830+0000","last_active":"2024-03-28T23:32:46.163952+0000","last_peered":"2024-03-28T23:32:46.163952+0000","last_clean":"2024-03-28T23:32:46.163952+0000","last_became_active":"2024-03-28T23:32:33.038524+0000","last_became_peered":"2024-03-28T23:32:33.038524+0000","last_unstale":"2024-03-28T23:32:46.163952+0000","last_undegraded":"2024-03-28T23:32:46.163952+0000","last_fullsized":"2024-03-28T23:32:46.163952+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:32.013667+0000","last_clean_scrub_stamp":"2024-03-28T23:32:32.013667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:55:53.512836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.166249+0000","last_change":"2024-03-28T23:32:32.034270+0000","last_active":"2024-03-28T23:32:46.166249+0000","last_peered":"2024-03-28T23:32:46.166249+0000","last_clean":"2024-03-28T23:32:46.166249+0000","last_became_active":"2024-03-28T23:32:32.034022+0000","last_became_peered":"2024-03-28T23:32:32.034022+0000","last_unstale":"2024-03-28T23:32:46.166249+0000","last_undegraded":"2024-03-28T23:32:46.166249+0000","last_fullsized":"2024-03-28T23:32:46.166249+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:32:31.006268+0000","last_clean_scrub_stamp":"2024-03-28T23:32:31.006268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:15.683500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":277,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-28T23:32:46.163750+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:32:46.163750+0000","last_peered":"2024-03-28T23:32:46.163750+0000","last_clean":"2024-03-28T23:32:46.163750+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:32:46.163750+0000","last_undegraded":"2024-03-28T23:32:46.163750+0000","last_fullsized":"2024-03-28T23:32:46.163750+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574888,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1019904,"data_stored":1041485,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574889,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":974848,"data_stored":454867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574889,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32108,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339732,"statfs":{"total":96636764160,"available":96603885568,"internally_reserved":0,"allocated":946176,"data_stored":451073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":970752,"data_stored":452915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":946176,"data_stored":451621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574888,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":954368,"data_stored":451781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574888,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":954368,"data_stored":451221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574888,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343812,"statfs":{"total":96636764160,"available":96608063488,"internally_reserved":0,"allocated":1032192,"data_stored":1043725,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:32:53.028 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T23:32:53.028 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs refuse_standby_for_another_fs true 2024-03-28T23:32:53.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.169+0000 7f83ea4c0640 1 Processor -- start 2024-03-28T23:32:53.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.169+0000 7f83ea4c0640 1 -- start start 2024-03-28T23:32:53.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.170+0000 7f83ea4c0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f83e4156080 0x7f83e40c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:53.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.170+0000 7f83ea4c0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f83e406ff90 0x7f83e4156b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:53.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.170+0000 7f83ea4c0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83e4156460 0x7f83e41570c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:53.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.170+0000 7f83ea4c0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f83e40c67b0 con 0x7f83e406ff90 2024-03-28T23:32:53.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.170+0000 7f83ea4c0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f83e4072790 con 0x7f83e4156080 2024-03-28T23:32:53.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.170+0000 7f83ea4c0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f83e40a7ea0 con 0x7f83e4156460 2024-03-28T23:32:53.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.170+0000 7f83db7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f83e406ff90 0x7f83e4156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:53.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.170+0000 7f83db7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f83e406ff90 0x7f83e4156b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51976/0 (socket says 172.21.15.161:51976) 2024-03-28T23:32:53.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.170+0000 7f83db7fe640 1 -- 172.21.15.161:0/1683690678 learned_addr learned my addr 172.21.15.161:0/1683690678 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:53.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.170+0000 7f83e3fff640 1 --2- 172.21.15.161:0/1683690678 >> v2:172.21.15.170:3300/0 conn(0x7f83e4156080 0x7f83e40c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:53.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.170+0000 7f83e8a5d640 1 --2- 172.21.15.161:0/1683690678 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83e4156460 0x7f83e41570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:53.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.170+0000 7f83e3fff640 1 -- 172.21.15.161:0/1683690678 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83e4156460 msgr2=0x7f83e41570c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:53.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.170+0000 7f83e3fff640 1 --2- 172.21.15.161:0/1683690678 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83e4156460 0x7f83e41570c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:53.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.170+0000 7f83e3fff640 1 -- 172.21.15.161:0/1683690678 >> v2:172.21.15.161:3300/0 conn(0x7f83e406ff90 msgr2=0x7f83e4156b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:53.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.170+0000 7f83e3fff640 1 --2- 172.21.15.161:0/1683690678 >> v2:172.21.15.161:3300/0 conn(0x7f83e406ff90 0x7f83e4156b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:53.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.170+0000 7f83e3fff640 1 -- 172.21.15.161:0/1683690678 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83e40a8180 con 0x7f83e4156080 2024-03-28T23:32:53.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.171+0000 7f83e3fff640 1 --2- 172.21.15.161:0/1683690678 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83e4156080 0x7f83e40c5340 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f83e40ab6a0 tx=0x7f83d402b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fefd7b236b4c9a74 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:53.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.171+0000 7f83e37fe640 1 -- 172.21.15.161:0/1683690678 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d4037070 con 0x7f83e4156080 2024-03-28T23:32:53.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.171+0000 7f83e37fe640 1 -- 172.21.15.161:0/1683690678 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83d4030710 con 0x7f83e4156080 2024-03-28T23:32:53.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.171+0000 7f83e37fe640 1 -- 172.21.15.161:0/1683690678 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d4030ae0 con 0x7f83e4156080 2024-03-28T23:32:53.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.172+0000 7f83ea4c0640 1 -- 172.21.15.161:0/1683690678 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83e4156080 msgr2=0x7f83e40c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:53.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.172+0000 7f83ea4c0640 1 --2- 172.21.15.161:0/1683690678 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83e4156080 0x7f83e40c5340 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f83e40ab6a0 tx=0x7f83d402b710 comp rx=0 tx=0).stop 2024-03-28T23:32:53.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.172+0000 7f83ea4c0640 1 -- 172.21.15.161:0/1683690678 shutdown_connections 2024-03-28T23:32:53.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.172+0000 7f83ea4c0640 1 --2- 172.21.15.161:0/1683690678 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83e4156460 0x7f83e41570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:53.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.172+0000 7f83ea4c0640 1 --2- 172.21.15.161:0/1683690678 >> v2:172.21.15.161:3300/0 conn(0x7f83e406ff90 0x7f83e4156b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:53.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.172+0000 7f83ea4c0640 1 --2- 172.21.15.161:0/1683690678 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83e4156080 0x7f83e40c5340 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:53.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.172+0000 7f83ea4c0640 1 -- 172.21.15.161:0/1683690678 >> 172.21.15.161:0/1683690678 conn(0x7f83e406eba0 msgr2=0x7f83e406b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:53.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.172+0000 7f83ea4c0640 1 -- 172.21.15.161:0/1683690678 shutdown_connections 2024-03-28T23:32:53.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.172+0000 7f83ea4c0640 1 -- 172.21.15.161:0/1683690678 wait complete. 2024-03-28T23:32:53.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.173+0000 7f83ea4c0640 1 Processor -- start 2024-03-28T23:32:53.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.173+0000 7f83ea4c0640 1 -- start start 2024-03-28T23:32:53.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.174+0000 7f83ea4c0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83e406ff90 0x7f83e40a9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:53.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.174+0000 7f83ea4c0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83e4156080 0x7f83e414e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:53.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.174+0000 7f83ea4c0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83e4156460 0x7f83e414d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:53.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.174+0000 7f83ea4c0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f83e40c2fd0 con 0x7f83e406ff90 2024-03-28T23:32:53.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.174+0000 7f83ea4c0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f83e40c2e50 con 0x7f83e4156460 2024-03-28T23:32:53.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.174+0000 7f83ea4c0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f83e40704a0 con 0x7f83e4156080 2024-03-28T23:32:53.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.174+0000 7f83e3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83e406ff90 0x7f83e40a9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:53.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.174+0000 7f83e3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83e406ff90 0x7f83e40a9f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51992/0 (socket says 172.21.15.161:51992) 2024-03-28T23:32:53.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.174+0000 7f83e3fff640 1 -- 172.21.15.161:0/1237767596 learned_addr learned my addr 172.21.15.161:0/1237767596 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:53.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.174+0000 7f83db7fe640 1 --2- 172.21.15.161:0/1237767596 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83e4156080 0x7f83e414e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:53.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.174+0000 7f83e8a5d640 1 --2- 172.21.15.161:0/1237767596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83e4156460 0x7f83e414d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:53.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.174+0000 7f83e3fff640 1 -- 172.21.15.161:0/1237767596 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83e4156080 msgr2=0x7f83e414e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:53.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.174+0000 7f83e3fff640 1 --2- 172.21.15.161:0/1237767596 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83e4156080 0x7f83e414e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:53.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.174+0000 7f83e3fff640 1 -- 172.21.15.161:0/1237767596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83e4156460 msgr2=0x7f83e414d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:53.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.174+0000 7f83e3fff640 1 --2- 172.21.15.161:0/1237767596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83e4156460 0x7f83e414d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:53.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.174+0000 7f83e3fff640 1 -- 172.21.15.161:0/1237767596 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83e40aa470 con 0x7f83e406ff90 2024-03-28T23:32:53.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.174+0000 7f83db7fe640 1 --2- 172.21.15.161:0/1237767596 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83e4156080 0x7f83e414e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:32:53.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.174+0000 7f83e3fff640 1 --2- 172.21.15.161:0/1237767596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83e406ff90 0x7f83e40a9f30 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f83d4009890 tx=0x7f83d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:53.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.175+0000 7f83e17fa640 1 -- 172.21.15.161:0/1237767596 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d4041020 con 0x7f83e406ff90 2024-03-28T23:32:53.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.175+0000 7f83e17fa640 1 -- 172.21.15.161:0/1237767596 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83d403c890 con 0x7f83e406ff90 2024-03-28T23:32:53.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.175+0000 7f83e17fa640 1 -- 172.21.15.161:0/1237767596 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d4037040 con 0x7f83e406ff90 2024-03-28T23:32:53.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.175+0000 7f83ea4c0640 1 -- 172.21.15.161:0/1237767596 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83e40a9530 con 0x7f83e406ff90 2024-03-28T23:32:53.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.175+0000 7f83ea4c0640 1 -- 172.21.15.161:0/1237767596 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83e40a9af0 con 0x7f83e406ff90 2024-03-28T23:32:53.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.176+0000 7f83ea4c0640 1 -- 172.21.15.161:0/1237767596 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83e404f4b0 con 0x7f83e406ff90 2024-03-28T23:32:53.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.176+0000 7f83e17fa640 1 -- 172.21.15.161:0/1237767596 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f83d4044020 con 0x7f83e406ff90 2024-03-28T23:32:53.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.177+0000 7f83e17fa640 1 --2- 172.21.15.161:0/1237767596 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f83c406b4d0 0x7f83c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:53.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.177+0000 7f83e17fa640 1 -- 172.21.15.161:0/1237767596 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6957+0+0 (secure 0 0 0) 0x7f83d40abff0 con 0x7f83e406ff90 2024-03-28T23:32:53.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.177+0000 7f83db7fe640 1 --2- 172.21.15.161:0/1237767596 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f83c406b4d0 0x7f83c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:53.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.177+0000 7f83db7fe640 1 --2- 172.21.15.161:0/1237767596 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f83c406b4d0 0x7f83c406d990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f83e4153130 tx=0x7f83cc009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:53.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.181+0000 7f83e17fa640 1 -- 172.21.15.161:0/1237767596 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f83d404b030 con 0x7f83e406ff90 2024-03-28T23:32:53.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:53.398+0000 7f83ea4c0640 1 -- 172.21.15.161:0/1237767596 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "refuse_standby_for_another_fs", "val": "true"} v 0) -- 0x7f83e414c190 con 0x7f83e406ff90 2024-03-28T23:32:54.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.142+0000 7f83e17fa640 1 -- 172.21.15.161:0/1237767596 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "refuse_standby_for_another_fs", "val": "true"}]=0 set to refuse standby for another fs v61) ==== 166+0+0 (secure 0 0 0) 0x7f83d407a910 con 0x7f83e406ff90 2024-03-28T23:32:54.143 INFO:teuthology.orchestra.run.smithi161.stderr:set to refuse standby for another fs 2024-03-28T23:32:54.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.144+0000 7f83ea4c0640 1 -- 172.21.15.161:0/1237767596 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f83c406b4d0 msgr2=0x7f83c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:54.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.144+0000 7f83ea4c0640 1 --2- 172.21.15.161:0/1237767596 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f83c406b4d0 0x7f83c406d990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f83e4153130 tx=0x7f83cc009240 comp rx=0 tx=0).stop 2024-03-28T23:32:54.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.144+0000 7f83ea4c0640 1 -- 172.21.15.161:0/1237767596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83e406ff90 msgr2=0x7f83e40a9f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:54.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.144+0000 7f83ea4c0640 1 --2- 172.21.15.161:0/1237767596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83e406ff90 0x7f83e40a9f30 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f83d4009890 tx=0x7f83d40046c0 comp rx=0 tx=0).stop 2024-03-28T23:32:54.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.144+0000 7f83ea4c0640 1 -- 172.21.15.161:0/1237767596 shutdown_connections 2024-03-28T23:32:54.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.144+0000 7f83ea4c0640 1 --2- 172.21.15.161:0/1237767596 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f83c406b4d0 0x7f83c406d990 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:54.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.144+0000 7f83ea4c0640 1 --2- 172.21.15.161:0/1237767596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83e4156460 0x7f83e414d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:54.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.144+0000 7f83ea4c0640 1 --2- 172.21.15.161:0/1237767596 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83e4156080 0x7f83e414e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:54.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.144+0000 7f83ea4c0640 1 --2- 172.21.15.161:0/1237767596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83e406ff90 0x7f83e40a9f30 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:54.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.144+0000 7f83ea4c0640 1 -- 172.21.15.161:0/1237767596 >> 172.21.15.161:0/1237767596 conn(0x7f83e406eba0 msgr2=0x7f83e4156a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:54.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.144+0000 7f83ea4c0640 1 -- 172.21.15.161:0/1237767596 shutdown_connections 2024-03-28T23:32:54.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.144+0000 7f83ea4c0640 1 -- 172.21.15.161:0/1237767596 wait complete. 2024-03-28T23:32:54.157 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail 5:0 2024-03-28T23:32:54.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.259+0000 7f562f956640 1 Processor -- start 2024-03-28T23:32:54.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.260+0000 7f562f956640 1 -- start start 2024-03-28T23:32:54.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.261+0000 7f562f956640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f56280c5410 0x7f56280c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:54.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.261+0000 7f562f956640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5628158a30 con 0x7f56281553a0 2024-03-28T23:32:54.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.261+0000 7f562f956640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5628158d10 con 0x7f5628152890 2024-03-28T23:32:54.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.261+0000 7f562f956640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f56280c8480 con 0x7f56280c5410 2024-03-28T23:32:54.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.261+0000 7f562cef1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f56280c5410 0x7f56280c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:54.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.261+0000 7f562def3640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f56281553a0 0x7f5628155850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:55580/0 (socket says 172.21.15.161:55580) 2024-03-28T23:32:54.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.261+0000 7f562def3640 1 -- 172.21.15.161:0/2826884241 learned_addr learned my addr 172.21.15.161:0/2826884241 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:54.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.261+0000 7f562cef1640 1 -- 172.21.15.161:0/2826884241 >> v1:172.21.15.170:6789/0 conn(0x7f5628152890 legacy=0x7f56280c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:54.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.261+0000 7f562cef1640 1 -- 172.21.15.161:0/2826884241 >> v1:172.21.15.161:6789/0 conn(0x7f56281553a0 legacy=0x7f5628155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:54.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.261+0000 7f562cef1640 1 -- 172.21.15.161:0/2826884241 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5628158ff0 con 0x7f56280c5410 2024-03-28T23:32:54.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.262+0000 7f562cef1640 1 --2- 172.21.15.161:0/2826884241 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56280c5410 0x7f56280c5850 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f5618009790 tx=0x7f561802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4df01fefe7c261eb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:54.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.262+0000 7f561ffff640 1 -- 172.21.15.161:0/2826884241 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5618037070 con 0x7f56280c5410 2024-03-28T23:32:54.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.262+0000 7f561ffff640 1 -- 172.21.15.161:0/2826884241 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5618030710 con 0x7f56280c5410 2024-03-28T23:32:54.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.262+0000 7f561ffff640 1 -- 172.21.15.161:0/2826884241 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5618030ae0 con 0x7f56280c5410 2024-03-28T23:32:54.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.263+0000 7f562f956640 1 -- 172.21.15.161:0/2826884241 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56280c5410 msgr2=0x7f56280c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:54.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.263+0000 7f562f956640 1 --2- 172.21.15.161:0/2826884241 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56280c5410 0x7f56280c5850 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f5618009790 tx=0x7f561802b710 comp rx=0 tx=0).stop 2024-03-28T23:32:54.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.263+0000 7f562f956640 1 -- 172.21.15.161:0/2826884241 shutdown_connections 2024-03-28T23:32:54.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.263+0000 7f562f956640 1 --2- 172.21.15.161:0/2826884241 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56280c5410 0x7f56280c5850 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:54.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.263+0000 7f562f956640 1 -- 172.21.15.161:0/2826884241 >> 172.21.15.161:0/2826884241 conn(0x7f56280ac620 msgr2=0x7f56280aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:54.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.263+0000 7f562f956640 1 -- 172.21.15.161:0/2826884241 shutdown_connections 2024-03-28T23:32:54.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.263+0000 7f562f956640 1 -- 172.21.15.161:0/2826884241 wait complete. 2024-03-28T23:32:54.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.265+0000 7f562f956640 1 Processor -- start 2024-03-28T23:32:54.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.265+0000 7f562f956640 1 -- start start 2024-03-28T23:32:54.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.266+0000 7f562f956640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56280c5410 0x7f5628158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:54.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.266+0000 7f562f956640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5628152890 0x7f5628157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:54.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.266+0000 7f562f956640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56281553a0 0x7f56281ee3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:54.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.266+0000 7f562f956640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5628159640 con 0x7f56280c5410 2024-03-28T23:32:54.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.266+0000 7f562f956640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f56280c5d90 con 0x7f5628152890 2024-03-28T23:32:54.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.266+0000 7f562f956640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f56280c90a0 con 0x7f56281553a0 2024-03-28T23:32:54.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.266+0000 7f562d6f2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56280c5410 0x7f5628158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:54.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.266+0000 7f562d6f2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56280c5410 0x7f5628158320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:52004/0 (socket says 172.21.15.161:52004) 2024-03-28T23:32:54.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.266+0000 7f562d6f2640 1 -- 172.21.15.161:0/3043431901 learned_addr learned my addr 172.21.15.161:0/3043431901 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:54.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.266+0000 7f562cef1640 1 --2- 172.21.15.161:0/3043431901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5628152890 0x7f5628157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:54.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.266+0000 7f562def3640 1 --2- 172.21.15.161:0/3043431901 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56281553a0 0x7f56281ee3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:54.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.266+0000 7f562d6f2640 1 -- 172.21.15.161:0/3043431901 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56281553a0 msgr2=0x7f56281ee3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:54.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.266+0000 7f562d6f2640 1 --2- 172.21.15.161:0/3043431901 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56281553a0 0x7f56281ee3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:54.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.266+0000 7f562d6f2640 1 -- 172.21.15.161:0/3043431901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5628152890 msgr2=0x7f5628157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:54.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.266+0000 7f562d6f2640 1 --2- 172.21.15.161:0/3043431901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5628152890 0x7f5628157970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:54.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.266+0000 7f562d6f2640 1 -- 172.21.15.161:0/3043431901 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56281eaf20 con 0x7f56280c5410 2024-03-28T23:32:54.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.266+0000 7f562d6f2640 1 --2- 172.21.15.161:0/3043431901 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56280c5410 0x7f5628158320 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f562400c400 tx=0x7f562400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:54.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.266+0000 7f561dffb640 1 -- 172.21.15.161:0/3043431901 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5624017070 con 0x7f56280c5410 2024-03-28T23:32:54.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.266+0000 7f561dffb640 1 -- 172.21.15.161:0/3043431901 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56240052d0 con 0x7f56280c5410 2024-03-28T23:32:54.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.266+0000 7f561dffb640 1 -- 172.21.15.161:0/3043431901 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5624003830 con 0x7f56280c5410 2024-03-28T23:32:54.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.266+0000 7f562f956640 1 -- 172.21.15.161:0/3043431901 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56281eb200 con 0x7f56280c5410 2024-03-28T23:32:54.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.267+0000 7f562f956640 1 -- 172.21.15.161:0/3043431901 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56281eb7c0 con 0x7f56280c5410 2024-03-28T23:32:54.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.268+0000 7f562f956640 1 -- 172.21.15.161:0/3043431901 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f562804f4b0 con 0x7f56280c5410 2024-03-28T23:32:54.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.268+0000 7f561dffb640 1 -- 172.21.15.161:0/3043431901 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f562401a070 con 0x7f56280c5410 2024-03-28T23:32:54.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.268+0000 7f561dffb640 1 --2- 172.21.15.161:0/3043431901 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f560c06b4d0 0x7f560c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:54.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.268+0000 7f561dffb640 1 -- 172.21.15.161:0/3043431901 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6957+0+0 (secure 0 0 0) 0x7f5624089900 con 0x7f56280c5410 2024-03-28T23:32:54.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.268+0000 7f562cef1640 1 --2- 172.21.15.161:0/3043431901 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f560c06b4d0 0x7f560c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:54.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.269+0000 7f562cef1640 1 --2- 172.21.15.161:0/3043431901 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f560c06b4d0 0x7f560c06d990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5618034000 tx=0x7f5618008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:54.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.274+0000 7f561dffb640 1 -- 172.21.15.161:0/3043431901 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5624057ce0 con 0x7f56280c5410 2024-03-28T23:32:54.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.517+0000 7f562f956640 1 -- 172.21.15.161:0/3043431901 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "5:0"} v 0) -- 0x7f56280c8140 con 0x7f56280c5410 2024-03-28T23:32:54.518 INFO:tasks.ceph.mon.a.smithi161.stderr:2024-03-28T23:32:54.517+0000 7fdc7b85b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T23:32:54.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.521+0000 7f561dffb640 1 -- 172.21.15.161:0/3043431901 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "5:0"}]=0 failed mds gid 6468 v62) ==== 97+0+0 (secure 0 0 0) 0x7f562405bd80 con 0x7f56280c5410 2024-03-28T23:32:54.521 INFO:teuthology.orchestra.run.smithi161.stderr:failed mds gid 6468 2024-03-28T23:32:54.524 INFO:tasks.ceph.mds.a.smithi161.stderr: -579> 2024-03-28T23:32:38.300+0000 7fcbbd324640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:32:54.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.524+0000 7f562f956640 1 -- 172.21.15.161:0/3043431901 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f560c06b4d0 msgr2=0x7f560c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:54.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.524+0000 7f562f956640 1 --2- 172.21.15.161:0/3043431901 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f560c06b4d0 0x7f560c06d990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5618034000 tx=0x7f5618008030 comp rx=0 tx=0).stop 2024-03-28T23:32:54.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.524+0000 7f562f956640 1 -- 172.21.15.161:0/3043431901 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56280c5410 msgr2=0x7f5628158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:54.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.524+0000 7f562f956640 1 --2- 172.21.15.161:0/3043431901 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56280c5410 0x7f5628158320 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f562400c400 tx=0x7f562400c8c0 comp rx=0 tx=0).stop 2024-03-28T23:32:54.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.524+0000 7f562f956640 1 -- 172.21.15.161:0/3043431901 shutdown_connections 2024-03-28T23:32:54.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.524+0000 7f562f956640 1 --2- 172.21.15.161:0/3043431901 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f560c06b4d0 0x7f560c06d990 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:54.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.524+0000 7f562f956640 1 --2- 172.21.15.161:0/3043431901 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56281553a0 0x7f56281ee3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:54.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.524+0000 7f562f956640 1 --2- 172.21.15.161:0/3043431901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5628152890 0x7f5628157970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:54.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.524+0000 7f562f956640 1 --2- 172.21.15.161:0/3043431901 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56280c5410 0x7f5628158320 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:54.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.524+0000 7f562f956640 1 -- 172.21.15.161:0/3043431901 >> 172.21.15.161:0/3043431901 conn(0x7f56280ac620 msgr2=0x7f56280cb2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:54.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.524+0000 7f562f956640 1 -- 172.21.15.161:0/3043431901 shutdown_connections 2024-03-28T23:32:54.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.524+0000 7f562f956640 1 -- 172.21.15.161:0/3043431901 wait complete. 2024-03-28T23:32:54.536 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:32:54.549 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-28T23:32:54.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.638+0000 7f9bb8765640 1 Processor -- start 2024-03-28T23:32:54.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.638+0000 7f9bb8765640 1 -- start start 2024-03-28T23:32:54.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.639+0000 7f9bb8765640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9bb014eca0 0x7f9bb00c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:54.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.639+0000 7f9bb8765640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9bb00c1dc0 0x7f9bb006ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:54.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.639+0000 7f9bb8765640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bb0151850 con 0x7f9bb006cf60 2024-03-28T23:32:54.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.639+0000 7f9bb8765640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f9bb0152460 con 0x7f9bb00c1dc0 2024-03-28T23:32:54.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.639+0000 7f9bb8765640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9bb0151b30 con 0x7f9bb014eca0 2024-03-28T23:32:54.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.639+0000 7f9bb5d00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9bb00c1dc0 0x7f9bb006ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:54.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.639+0000 7f9bb5d00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9bb00c1dc0 0x7f9bb006ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:52034/0 (socket says 172.21.15.161:52034) 2024-03-28T23:32:54.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.639+0000 7f9bb5d00640 1 -- 172.21.15.161:0/188308929 learned_addr learned my addr 172.21.15.161:0/188308929 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:54.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.639+0000 7f9bb6501640 1 --2- 172.21.15.161:0/188308929 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9bb014eca0 0x7f9bb00c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:54.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.639+0000 7f9bb5d00640 1 -- 172.21.15.161:0/188308929 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9bb014eca0 msgr2=0x7f9bb00c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:54.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.639+0000 7f9bb5d00640 1 --2- 172.21.15.161:0/188308929 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9bb014eca0 0x7f9bb00c1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:54.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.639+0000 7f9bb5d00640 1 -- 172.21.15.161:0/188308929 >> v1:172.21.15.170:6789/0 conn(0x7f9bb006cf60 legacy=0x7f9bb00a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:54.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.639+0000 7f9bb5d00640 1 -- 172.21.15.161:0/188308929 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bb0151e10 con 0x7f9bb00c1dc0 2024-03-28T23:32:54.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.640+0000 7f9bb5d00640 1 --2- 172.21.15.161:0/188308929 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9bb00c1dc0 0x7f9bb006ca20 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0009870 tx=0x7f9ba002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5305c6297413e041 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:54.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.640+0000 7f9bb54ff640 1 -- 172.21.15.161:0/188308929 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba0035020 con 0x7f9bb00c1dc0 2024-03-28T23:32:54.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.640+0000 7f9bb54ff640 1 -- 172.21.15.161:0/188308929 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ba0031640 con 0x7f9bb00c1dc0 2024-03-28T23:32:54.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.640+0000 7f9bb54ff640 1 -- 172.21.15.161:0/188308929 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba002d040 con 0x7f9bb00c1dc0 2024-03-28T23:32:54.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.640+0000 7f9bb8765640 1 -- 172.21.15.161:0/188308929 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9bb00c1dc0 msgr2=0x7f9bb006ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:54.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.640+0000 7f9bb8765640 1 --2- 172.21.15.161:0/188308929 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9bb00c1dc0 0x7f9bb006ca20 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0009870 tx=0x7f9ba002b8f0 comp rx=0 tx=0).stop 2024-03-28T23:32:54.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.641+0000 7f9bb8765640 1 -- 172.21.15.161:0/188308929 shutdown_connections 2024-03-28T23:32:54.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.641+0000 7f9bb8765640 1 --2- 172.21.15.161:0/188308929 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9bb00c1dc0 0x7f9bb006ca20 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:54.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.641+0000 7f9bb8765640 1 --2- 172.21.15.161:0/188308929 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9bb014eca0 0x7f9bb00c1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:54.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.641+0000 7f9bb8765640 1 -- 172.21.15.161:0/188308929 >> 172.21.15.161:0/188308929 conn(0x7f9bb00c6670 msgr2=0x7f9bb00c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:54.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.641+0000 7f9bb8765640 1 -- 172.21.15.161:0/188308929 shutdown_connections 2024-03-28T23:32:54.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.641+0000 7f9bb8765640 1 -- 172.21.15.161:0/188308929 wait complete. 2024-03-28T23:32:54.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.642+0000 7f9bb8765640 1 Processor -- start 2024-03-28T23:32:54.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.642+0000 7f9bb8765640 1 -- start start 2024-03-28T23:32:54.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.642+0000 7f9bb8765640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9bb006cf60 0x7f9bb006bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:54.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.642+0000 7f9bb8765640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9bb00c1dc0 0x7f9bb01dd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:54.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.642+0000 7f9bb8765640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9bb014eca0 0x7f9bb01dd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:54.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.642+0000 7f9bb8765640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb01514b0 con 0x7f9bb00c1dc0 2024-03-28T23:32:54.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.642+0000 7f9bb8765640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb014f080 con 0x7f9bb014eca0 2024-03-28T23:32:54.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.642+0000 7f9bb8765640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9bb0151630 con 0x7f9bb006cf60 2024-03-28T23:32:54.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.642+0000 7f9bb5d00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9bb00c1dc0 0x7f9bb01dd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:54.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.642+0000 7f9bb5d00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9bb00c1dc0 0x7f9bb01dd190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:52042/0 (socket says 172.21.15.161:52042) 2024-03-28T23:32:54.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.642+0000 7f9bb5d00640 1 -- 172.21.15.161:0/3082959113 learned_addr learned my addr 172.21.15.161:0/3082959113 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:54.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.643+0000 7f9bb6d02640 1 --2- 172.21.15.161:0/3082959113 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9bb014eca0 0x7f9bb01dd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:54.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.643+0000 7f9bb6501640 1 --2- 172.21.15.161:0/3082959113 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9bb006cf60 0x7f9bb006bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:54.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.643+0000 7f9bb5d00640 1 -- 172.21.15.161:0/3082959113 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9bb006cf60 msgr2=0x7f9bb006bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:54.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.643+0000 7f9bb5d00640 1 --2- 172.21.15.161:0/3082959113 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9bb006cf60 0x7f9bb006bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:54.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.643+0000 7f9bb5d00640 1 -- 172.21.15.161:0/3082959113 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9bb014eca0 msgr2=0x7f9bb01dd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:54.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.643+0000 7f9bb5d00640 1 --2- 172.21.15.161:0/3082959113 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9bb014eca0 0x7f9bb01dd6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:54.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.643+0000 7f9bb5d00640 1 -- 172.21.15.161:0/3082959113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bb0152dd0 con 0x7f9bb00c1dc0 2024-03-28T23:32:54.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.643+0000 7f9bb5d00640 1 --2- 172.21.15.161:0/3082959113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9bb00c1dc0 0x7f9bb01dd190 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f9ba002b930 tx=0x7f9ba00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:54.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.643+0000 7f9ba6ffd640 1 -- 172.21.15.161:0/3082959113 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba0040070 con 0x7f9bb00c1dc0 2024-03-28T23:32:54.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.643+0000 7f9ba6ffd640 1 -- 172.21.15.161:0/3082959113 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ba003e070 con 0x7f9bb00c1dc0 2024-03-28T23:32:54.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.643+0000 7f9bb8765640 1 -- 172.21.15.161:0/3082959113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bb01ec750 con 0x7f9bb00c1dc0 2024-03-28T23:32:54.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.643+0000 7f9ba6ffd640 1 -- 172.21.15.161:0/3082959113 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba0035040 con 0x7f9bb00c1dc0 2024-03-28T23:32:54.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.643+0000 7f9bb8765640 1 -- 172.21.15.161:0/3082959113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bb006c0c0 con 0x7f9bb00c1dc0 2024-03-28T23:32:54.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.644+0000 7f9bb8765640 1 -- 172.21.15.161:0/3082959113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bb004f4b0 con 0x7f9bb00c1dc0 2024-03-28T23:32:54.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.645+0000 7f9ba6ffd640 1 -- 172.21.15.161:0/3082959113 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9ba0038020 con 0x7f9bb00c1dc0 2024-03-28T23:32:54.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.645+0000 7f9ba6ffd640 1 --2- 172.21.15.161:0/3082959113 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9b9406b4d0 0x7f9b9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:54.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.645+0000 7f9bb6501640 1 --2- 172.21.15.161:0/3082959113 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9b9406b4d0 0x7f9b9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:54.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.645+0000 7f9ba6ffd640 1 -- 172.21.15.161:0/3082959113 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7043+0+0 (secure 0 0 0) 0x7f9ba00abf00 con 0x7f9bb00c1dc0 2024-03-28T23:32:54.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.646+0000 7f9bb6501640 1 --2- 172.21.15.161:0/3082959113 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9b9406b4d0 0x7f9b9406d990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9bac009aa0 tx=0x7f9bac009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:54.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.650+0000 7f9ba6ffd640 1 -- 172.21.15.161:0/3082959113 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9ba007a280 con 0x7f9bb00c1dc0 2024-03-28T23:32:54.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.855+0000 7f9bb8765640 1 -- 172.21.15.161:0/3082959113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9bb00c62b0 con 0x7f9bb00c1dc0 2024-03-28T23:32:54.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.856+0000 7f9ba6ffd640 1 -- 172.21.15.161:0/3082959113 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+5663 (secure 0 0 0) 0x7f9ba007e320 con 0x7f9bb00c1dc0 2024-03-28T23:32:54.857 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:54.857 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":62,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.161:6837/737681942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":737681942},{"type":"v1","addr":"172.21.15.161:6837","nonce":737681942}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":60},{"gid":5579,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.170:6834/3501962065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3501962065},{"type":"v1","addr":"172.21.15.170:6834","nonce":3501962065}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":62,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:32:32.372964+0000","modified":"2024-03-28T23:32:54.518709+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":5},{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:32:44.463287+0000","modified":"2024-03-28T23:32:46.605012+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6037":{"gid":6037,"name":"b","rank":0,"incarnation":57,"state":"up:active","state_seq":6,"addr":"172.21.15.170:6835/2691084827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":2691084827},{"type":"v1","addr":"172.21.15.170:6835","nonce":2691084827}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6037,"qdb_cluster":[6037]},"id":6}]} 2024-03-28T23:32:54.857 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 62 2024-03-28T23:32:54.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.858+0000 7f9bb8765640 1 -- 172.21.15.161:0/3082959113 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9b9406b4d0 msgr2=0x7f9b9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:54.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.858+0000 7f9bb8765640 1 --2- 172.21.15.161:0/3082959113 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9b9406b4d0 0x7f9b9406d990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9bac009aa0 tx=0x7f9bac009190 comp rx=0 tx=0).stop 2024-03-28T23:32:54.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.858+0000 7f9bb8765640 1 -- 172.21.15.161:0/3082959113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9bb00c1dc0 msgr2=0x7f9bb01dd190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:54.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.858+0000 7f9bb8765640 1 --2- 172.21.15.161:0/3082959113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9bb00c1dc0 0x7f9bb01dd190 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f9ba002b930 tx=0x7f9ba00046c0 comp rx=0 tx=0).stop 2024-03-28T23:32:54.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.858+0000 7f9bb8765640 1 -- 172.21.15.161:0/3082959113 shutdown_connections 2024-03-28T23:32:54.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.858+0000 7f9bb8765640 1 --2- 172.21.15.161:0/3082959113 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9b9406b4d0 0x7f9b9406d990 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:54.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.858+0000 7f9bb8765640 1 --2- 172.21.15.161:0/3082959113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9bb00c1dc0 0x7f9bb01dd190 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:54.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.858+0000 7f9bb8765640 1 --2- 172.21.15.161:0/3082959113 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9bb014eca0 0x7f9bb01dd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:54.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.858+0000 7f9bb8765640 1 --2- 172.21.15.161:0/3082959113 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9bb006cf60 0x7f9bb006bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:54.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.858+0000 7f9bb8765640 1 -- 172.21.15.161:0/3082959113 >> 172.21.15.161:0/3082959113 conn(0x7f9bb00c6670 msgr2=0x7f9bb00aa8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:54.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.858+0000 7f9bb8765640 1 -- 172.21.15.161:0/3082959113 shutdown_connections 2024-03-28T23:32:54.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:54.858+0000 7f9bb8765640 1 -- 172.21.15.161:0/3082959113 wait complete. 2024-03-28T23:32:54.870 INFO:tasks.cephfs.test_failover:Waiting for former active to reclaim its spot 2024-03-28T23:32:54.870 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_beacon_grace 2024-03-28T23:32:55.017 INFO:teuthology.orchestra.run.smithi161.stdout:{"mds_beacon_grace":"15.000000"} 2024-03-28T23:32:55.017 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_beacon_grace": "15.000000" } 2024-03-28T23:32:55.018 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:32:55.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.117+0000 7f9cd795c640 1 Processor -- start 2024-03-28T23:32:55.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.118+0000 7f9cd795c640 1 -- start start 2024-03-28T23:32:55.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.118+0000 7f9cd795c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cd006ce80 0x7f9cd00aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:55.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.118+0000 7f9cd795c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9cd00a9680 0x7f9cd00aab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:55.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.118+0000 7f9cd795c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cd014c250 con 0x7f9cd014be70 2024-03-28T23:32:55.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.118+0000 7f9cd795c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f9cd01529c0 con 0x7f9cd00a9680 2024-03-28T23:32:55.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.118+0000 7f9cd795c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9cd00a9dc0 con 0x7f9cd006ce80 2024-03-28T23:32:55.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.119+0000 7f9cd56f8640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f9cd014be70 0x7f9cd00ab1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:55588/0 (socket says 172.21.15.161:55588) 2024-03-28T23:32:55.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.119+0000 7f9cd56f8640 1 -- 172.21.15.161:0/3758857214 learned_addr learned my addr 172.21.15.161:0/3758857214 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:32:55.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.119+0000 7f9cd5ef9640 1 --2- 172.21.15.161:0/3758857214 >> v2:172.21.15.170:3300/0 conn(0x7f9cd00a9680 0x7f9cd00aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:55.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.119+0000 7f9cd4ef7640 1 --2- 172.21.15.161:0/3758857214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cd006ce80 0x7f9cd00aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:55.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.119+0000 7f9cd5ef9640 1 -- 172.21.15.161:0/3758857214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cd006ce80 msgr2=0x7f9cd00aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:32:55.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.119+0000 7f9cd5ef9640 1 --2- 172.21.15.161:0/3758857214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cd006ce80 0x7f9cd00aa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:55.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.119+0000 7f9cd5ef9640 1 -- 172.21.15.161:0/3758857214 >> v1:172.21.15.161:6789/0 conn(0x7f9cd014be70 legacy=0x7f9cd00ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:55.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.119+0000 7f9cc7fff640 1 -- 172.21.15.161:0/3758857214 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3787195237 0 0) 0x7f9cd014c250 con 0x7f9cd014be70 2024-03-28T23:32:55.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.119+0000 7f9cd5ef9640 1 -- 172.21.15.161:0/3758857214 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cd00aa0a0 con 0x7f9cd00a9680 2024-03-28T23:32:55.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.119+0000 7f9cd4ef7640 1 --2- 172.21.15.161:0/3758857214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cd006ce80 0x7f9cd00aa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:32:55.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.120+0000 7f9cd5ef9640 1 --2- 172.21.15.161:0/3758857214 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cd00a9680 0x7f9cd00aab00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8009d90 tx=0x7f9cc802bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a7e9131ece2eb48 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:55.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.120+0000 7f9cc7fff640 1 -- 172.21.15.161:0/3758857214 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc8039070 con 0x7f9cd00a9680 2024-03-28T23:32:55.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.120+0000 7f9cc7fff640 1 -- 172.21.15.161:0/3758857214 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cc802cb30 con 0x7f9cd00a9680 2024-03-28T23:32:55.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.120+0000 7f9cc7fff640 1 -- 172.21.15.161:0/3758857214 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc8034440 con 0x7f9cd00a9680 2024-03-28T23:32:55.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.120+0000 7f9cd795c640 1 -- 172.21.15.161:0/3758857214 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cd00a9680 msgr2=0x7f9cd00aab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:55.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.120+0000 7f9cd795c640 1 --2- 172.21.15.161:0/3758857214 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cd00a9680 0x7f9cd00aab00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8009d90 tx=0x7f9cc802bd80 comp rx=0 tx=0).stop 2024-03-28T23:32:55.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.120+0000 7f9cd795c640 1 -- 172.21.15.161:0/3758857214 shutdown_connections 2024-03-28T23:32:55.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.120+0000 7f9cd795c640 1 --2- 172.21.15.161:0/3758857214 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cd00a9680 0x7f9cd00aab00 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:55.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.120+0000 7f9cd795c640 1 --2- 172.21.15.161:0/3758857214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cd006ce80 0x7f9cd00aa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:55.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.120+0000 7f9cd795c640 1 -- 172.21.15.161:0/3758857214 >> 172.21.15.161:0/3758857214 conn(0x7f9cd00c6670 msgr2=0x7f9cd00c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:55.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.120+0000 7f9cd795c640 1 -- 172.21.15.161:0/3758857214 shutdown_connections 2024-03-28T23:32:55.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.120+0000 7f9cd795c640 1 -- 172.21.15.161:0/3758857214 wait complete. 2024-03-28T23:32:55.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.121+0000 7f9cd795c640 1 Processor -- start 2024-03-28T23:32:55.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.122+0000 7f9cd795c640 1 -- start start 2024-03-28T23:32:55.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.122+0000 7f9cd795c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cd006ce80 0x7f9cd01eb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:55.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.122+0000 7f9cd795c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cd00a9680 0x7f9cd006be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:55.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.122+0000 7f9cd795c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cd014be70 0x7f9cd014e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:55.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.122+0000 7f9cd795c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd00aba30 con 0x7f9cd00a9680 2024-03-28T23:32:55.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.122+0000 7f9cd795c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd00ab8b0 con 0x7f9cd014be70 2024-03-28T23:32:55.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.122+0000 7f9cd795c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9cd006b150 con 0x7f9cd006ce80 2024-03-28T23:32:55.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.122+0000 7f9cd4ef7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cd00a9680 0x7f9cd006be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:55.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.122+0000 7f9cd4ef7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cd00a9680 0x7f9cd006be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:52048/0 (socket says 172.21.15.161:52048) 2024-03-28T23:32:55.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.122+0000 7f9cd4ef7640 1 -- 172.21.15.161:0/2581986379 learned_addr learned my addr 172.21.15.161:0/2581986379 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:32:55.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.122+0000 7f9cd56f8640 1 --2- 172.21.15.161:0/2581986379 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cd006ce80 0x7f9cd01eb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:55.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.122+0000 7f9cd5ef9640 1 --2- 172.21.15.161:0/2581986379 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cd014be70 0x7f9cd014e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:55.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.123+0000 7f9cd4ef7640 1 -- 172.21.15.161:0/2581986379 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cd006ce80 msgr2=0x7f9cd01eb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:55.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.123+0000 7f9cd4ef7640 1 --2- 172.21.15.161:0/2581986379 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cd006ce80 0x7f9cd01eb180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:55.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.123+0000 7f9cd4ef7640 1 -- 172.21.15.161:0/2581986379 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cd014be70 msgr2=0x7f9cd014e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:55.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.123+0000 7f9cd4ef7640 1 --2- 172.21.15.161:0/2581986379 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cd014be70 0x7f9cd014e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:55.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.123+0000 7f9cd4ef7640 1 -- 172.21.15.161:0/2581986379 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cd00c5e60 con 0x7f9cd00a9680 2024-03-28T23:32:55.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.123+0000 7f9cd56f8640 1 --2- 172.21.15.161:0/2581986379 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cd006ce80 0x7f9cd01eb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:32:55.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.123+0000 7f9cd5ef9640 1 --2- 172.21.15.161:0/2581986379 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cd014be70 0x7f9cd014e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:32:55.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.123+0000 7f9cd4ef7640 1 --2- 172.21.15.161:0/2581986379 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cd00a9680 0x7f9cd006be70 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc00e960 tx=0x7f9ccc00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:55.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.123+0000 7f9cc5ffb640 1 -- 172.21.15.161:0/2581986379 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc017070 con 0x7f9cd00a9680 2024-03-28T23:32:55.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.123+0000 7f9cc5ffb640 1 -- 172.21.15.161:0/2581986379 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ccc00ca00 con 0x7f9cd00a9680 2024-03-28T23:32:55.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.123+0000 7f9cd795c640 1 -- 172.21.15.161:0/2581986379 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cd00c6140 con 0x7f9cd00a9680 2024-03-28T23:32:55.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.123+0000 7f9cc5ffb640 1 -- 172.21.15.161:0/2581986379 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc00cdd0 con 0x7f9cd00a9680 2024-03-28T23:32:55.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.123+0000 7f9cd795c640 1 -- 172.21.15.161:0/2581986379 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cd01eac60 con 0x7f9cd00a9680 2024-03-28T23:32:55.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.124+0000 7f9cd795c640 1 -- 172.21.15.161:0/2581986379 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cd004f4b0 con 0x7f9cd00a9680 2024-03-28T23:32:55.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.124+0000 7f9cc5ffb640 1 -- 172.21.15.161:0/2581986379 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9ccc01a070 con 0x7f9cd00a9680 2024-03-28T23:32:55.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.125+0000 7f9cc5ffb640 1 --2- 172.21.15.161:0/2581986379 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9cb406b500 0x7f9cb406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:32:55.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.125+0000 7f9cc5ffb640 1 -- 172.21.15.161:0/2581986379 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7043+0+0 (secure 0 0 0) 0x7f9ccc08a940 con 0x7f9cd00a9680 2024-03-28T23:32:55.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.125+0000 7f9cd56f8640 1 --2- 172.21.15.161:0/2581986379 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9cb406b500 0x7f9cb406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:32:55.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.125+0000 7f9cd56f8640 1 --2- 172.21.15.161:0/2581986379 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9cb406b500 0x7f9cb406d9c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9cc0007fd0 tx=0x7f9cc0007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:32:55.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.129+0000 7f9cc5ffb640 1 -- 172.21.15.161:0/2581986379 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9ccc058bb0 con 0x7f9cd00a9680 2024-03-28T23:32:55.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.344+0000 7f9cd795c640 1 -- 172.21.15.161:0/2581986379 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9cd00c0ff0 con 0x7f9cd00a9680 2024-03-28T23:32:55.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.345+0000 7f9cc5ffb640 1 -- 172.21.15.161:0/2581986379 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+5663 (secure 0 0 0) 0x7f9ccc05cc50 con 0x7f9cd00a9680 2024-03-28T23:32:55.346 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:32:55.346 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":62,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.161:6837/737681942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":737681942},{"type":"v1","addr":"172.21.15.161:6837","nonce":737681942}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":60},{"gid":5579,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.170:6834/3501962065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3501962065},{"type":"v1","addr":"172.21.15.170:6834","nonce":3501962065}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":62,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:32:32.372964+0000","modified":"2024-03-28T23:32:54.518709+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":5},{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:32:44.463287+0000","modified":"2024-03-28T23:32:46.605012+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6037":{"gid":6037,"name":"b","rank":0,"incarnation":57,"state":"up:active","state_seq":6,"addr":"172.21.15.170:6835/2691084827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":2691084827},{"type":"v1","addr":"172.21.15.170:6835","nonce":2691084827}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6037,"qdb_cluster":[6037]},"id":6}]} 2024-03-28T23:32:55.346 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 62 2024-03-28T23:32:55.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.347+0000 7f9cd795c640 1 -- 172.21.15.161:0/2581986379 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9cb406b500 msgr2=0x7f9cb406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:55.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.347+0000 7f9cd795c640 1 --2- 172.21.15.161:0/2581986379 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9cb406b500 0x7f9cb406d9c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9cc0007fd0 tx=0x7f9cc0007de0 comp rx=0 tx=0).stop 2024-03-28T23:32:55.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.347+0000 7f9cd795c640 1 -- 172.21.15.161:0/2581986379 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cd00a9680 msgr2=0x7f9cd006be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:32:55.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.347+0000 7f9cd795c640 1 --2- 172.21.15.161:0/2581986379 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cd00a9680 0x7f9cd006be70 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc00e960 tx=0x7f9ccc00ee20 comp rx=0 tx=0).stop 2024-03-28T23:32:55.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.347+0000 7f9cd795c640 1 -- 172.21.15.161:0/2581986379 shutdown_connections 2024-03-28T23:32:55.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.347+0000 7f9cd795c640 1 --2- 172.21.15.161:0/2581986379 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9cb406b500 0x7f9cb406d9c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:55.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.347+0000 7f9cd795c640 1 --2- 172.21.15.161:0/2581986379 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cd00a9680 0x7f9cd006be70 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:55.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.347+0000 7f9cd795c640 1 --2- 172.21.15.161:0/2581986379 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cd014be70 0x7f9cd014e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:55.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.347+0000 7f9cd795c640 1 --2- 172.21.15.161:0/2581986379 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cd006ce80 0x7f9cd01eb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:32:55.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.347+0000 7f9cd795c640 1 -- 172.21.15.161:0/2581986379 >> 172.21.15.161:0/2581986379 conn(0x7f9cd00c6670 msgr2=0x7f9cd00ab1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:32:55.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.347+0000 7f9cd795c640 1 -- 172.21.15.161:0/2581986379 shutdown_connections 2024-03-28T23:32:55.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:32:55.347+0000 7f9cd795c640 1 -- 172.21.15.161:0/2581986379 wait complete. 2024-03-28T23:32:55.359 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-03-28T23:32:55.733 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:32:55.736 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:33:00.361 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:33:00.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.462+0000 7f91919d6640 1 Processor -- start 2024-03-28T23:33:00.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.462+0000 7f91919d6640 1 -- start start 2024-03-28T23:33:00.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.463+0000 7f91919d6640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f918c14be70 0x7f918c0ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:00.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.463+0000 7f91919d6640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f918c0aacd0 0x7f918c15c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:00.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.463+0000 7f91919d6640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f918c14c250 con 0x7f918c0ab6e0 2024-03-28T23:33:00.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.463+0000 7f91919d6640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f918c1529c0 con 0x7f918c14be70 2024-03-28T23:33:00.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.463+0000 7f91919d6640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f918c0a9dc0 con 0x7f918c0aacd0 2024-03-28T23:33:00.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.463+0000 7f918affd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f918c14be70 0x7f918c0ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:00.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.463+0000 7f918affd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f918c14be70 0x7f918c0ab1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:52056/0 (socket says 172.21.15.161:52056) 2024-03-28T23:33:00.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.463+0000 7f918affd640 1 -- 172.21.15.161:0/1922028270 learned_addr learned my addr 172.21.15.161:0/1922028270 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:00.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.463+0000 7f918b7fe640 1 --2- 172.21.15.161:0/1922028270 >> v2:172.21.15.170:3300/0 conn(0x7f918c0aacd0 0x7f918c15c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:00.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.463+0000 7f9189ffb640 1 -- 172.21.15.161:0/1922028270 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2929884905 0 0) 0x7f918c14c250 con 0x7f918c0ab6e0 2024-03-28T23:33:00.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.463+0000 7f918affd640 1 -- 172.21.15.161:0/1922028270 >> v2:172.21.15.170:3300/0 conn(0x7f918c0aacd0 msgr2=0x7f918c15c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:00.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.463+0000 7f918affd640 1 --2- 172.21.15.161:0/1922028270 >> v2:172.21.15.170:3300/0 conn(0x7f918c0aacd0 0x7f918c15c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:00.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.463+0000 7f918affd640 1 -- 172.21.15.161:0/1922028270 >> v1:172.21.15.170:6789/0 conn(0x7f918c0ab6e0 legacy=0x7f918c0aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:00.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.463+0000 7f918affd640 1 -- 172.21.15.161:0/1922028270 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f918c0aa0a0 con 0x7f918c14be70 2024-03-28T23:33:00.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.464+0000 7f918affd640 1 --2- 172.21.15.161:0/1922028270 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f918c14be70 0x7f918c0ab1a0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f918c073960 tx=0x7f917c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98ef17af0e3dce00 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:00.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.464+0000 7f9189ffb640 1 -- 172.21.15.161:0/1922028270 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917c02ca80 con 0x7f918c14be70 2024-03-28T23:33:00.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.464+0000 7f9189ffb640 1 -- 172.21.15.161:0/1922028270 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f917c02cc20 con 0x7f918c14be70 2024-03-28T23:33:00.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.464+0000 7f9189ffb640 1 -- 172.21.15.161:0/1922028270 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917c0343f0 con 0x7f918c14be70 2024-03-28T23:33:00.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.465+0000 7f91919d6640 1 -- 172.21.15.161:0/1922028270 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f918c14be70 msgr2=0x7f918c0ab1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:00.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.465+0000 7f91919d6640 1 --2- 172.21.15.161:0/1922028270 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f918c14be70 0x7f918c0ab1a0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f918c073960 tx=0x7f917c02bee0 comp rx=0 tx=0).stop 2024-03-28T23:33:00.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.465+0000 7f91919d6640 1 -- 172.21.15.161:0/1922028270 shutdown_connections 2024-03-28T23:33:00.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.465+0000 7f91919d6640 1 --2- 172.21.15.161:0/1922028270 >> v2:172.21.15.170:3300/0 conn(0x7f918c0aacd0 0x7f918c15c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:00.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.465+0000 7f91919d6640 1 --2- 172.21.15.161:0/1922028270 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f918c14be70 0x7f918c0ab1a0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:00.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.465+0000 7f91919d6640 1 -- 172.21.15.161:0/1922028270 >> 172.21.15.161:0/1922028270 conn(0x7f918c0c6670 msgr2=0x7f918c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:00.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.465+0000 7f91919d6640 1 -- 172.21.15.161:0/1922028270 shutdown_connections 2024-03-28T23:33:00.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.465+0000 7f91919d6640 1 -- 172.21.15.161:0/1922028270 wait complete. 2024-03-28T23:33:00.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.466+0000 7f91919d6640 1 Processor -- start 2024-03-28T23:33:00.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.466+0000 7f91919d6640 1 -- start start 2024-03-28T23:33:00.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.467+0000 7f91919d6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f918c0aacd0 0x7f918c1dfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:00.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.467+0000 7f91919d6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f918c0ab6e0 0x7f918c155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:00.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.467+0000 7f91919d6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f918c14be70 0x7f918c148910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:00.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.467+0000 7f91919d6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f918c06b2d0 con 0x7f918c0aacd0 2024-03-28T23:33:00.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.467+0000 7f91919d6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f918c06b150 con 0x7f918c0ab6e0 2024-03-28T23:33:00.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.467+0000 7f91919d6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f918c0c1520 con 0x7f918c14be70 2024-03-28T23:33:00.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.467+0000 7f918affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f918c0aacd0 0x7f918c1dfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:00.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.467+0000 7f918affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f918c0aacd0 0x7f918c1dfaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:52068/0 (socket says 172.21.15.161:52068) 2024-03-28T23:33:00.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.467+0000 7f918affd640 1 -- 172.21.15.161:0/4100950254 learned_addr learned my addr 172.21.15.161:0/4100950254 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:00.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.467+0000 7f918b7fe640 1 --2- 172.21.15.161:0/4100950254 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f918c14be70 0x7f918c148910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:00.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.467+0000 7f918a7fc640 1 --2- 172.21.15.161:0/4100950254 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f918c0ab6e0 0x7f918c155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:00.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.467+0000 7f918affd640 1 -- 172.21.15.161:0/4100950254 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f918c14be70 msgr2=0x7f918c148910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:00.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.467+0000 7f918affd640 1 --2- 172.21.15.161:0/4100950254 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f918c14be70 0x7f918c148910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:00.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.467+0000 7f918affd640 1 -- 172.21.15.161:0/4100950254 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f918c0ab6e0 msgr2=0x7f918c155360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:00.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.467+0000 7f918affd640 1 --2- 172.21.15.161:0/4100950254 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f918c0ab6e0 0x7f918c155360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:00.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.467+0000 7f918affd640 1 -- 172.21.15.161:0/4100950254 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f918c1edcc0 con 0x7f918c0aacd0 2024-03-28T23:33:00.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.468+0000 7f918affd640 1 --2- 172.21.15.161:0/4100950254 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f918c0aacd0 0x7f918c1dfaa0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f917c036040 tx=0x7f917c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:00.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.468+0000 7f9173fff640 1 -- 172.21.15.161:0/4100950254 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917c043020 con 0x7f918c0aacd0 2024-03-28T23:33:00.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.468+0000 7f9173fff640 1 -- 172.21.15.161:0/4100950254 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f917c03e450 con 0x7f918c0aacd0 2024-03-28T23:33:00.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.468+0000 7f91919d6640 1 -- 172.21.15.161:0/4100950254 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f918c1edfa0 con 0x7f918c0aacd0 2024-03-28T23:33:00.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.468+0000 7f9173fff640 1 -- 172.21.15.161:0/4100950254 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917c009510 con 0x7f918c0aacd0 2024-03-28T23:33:00.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.468+0000 7f91919d6640 1 -- 172.21.15.161:0/4100950254 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f918c1ee560 con 0x7f918c0aacd0 2024-03-28T23:33:00.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.469+0000 7f91919d6640 1 -- 172.21.15.161:0/4100950254 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f918c04f4b0 con 0x7f918c0aacd0 2024-03-28T23:33:00.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.469+0000 7f9173fff640 1 -- 172.21.15.161:0/4100950254 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f917c046020 con 0x7f918c0aacd0 2024-03-28T23:33:00.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.470+0000 7f9173fff640 1 --2- 172.21.15.161:0/4100950254 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f917806b4d0 0x7f917806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:00.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.470+0000 7f9173fff640 1 -- 172.21.15.161:0/4100950254 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7043+0+0 (secure 0 0 0) 0x7f917c0acd70 con 0x7f918c0aacd0 2024-03-28T23:33:00.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.470+0000 7f918a7fc640 1 --2- 172.21.15.161:0/4100950254 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f917806b4d0 0x7f917806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:00.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.470+0000 7f918a7fc640 1 --2- 172.21.15.161:0/4100950254 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f917806b4d0 0x7f917806d990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f918c14d4d0 tx=0x7f9180009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:00.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.474+0000 7f9173fff640 1 -- 172.21.15.161:0/4100950254 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f917c04d030 con 0x7f918c0aacd0 2024-03-28T23:33:00.528 INFO:tasks.ceph.mds.a.smithi161.stderr:2024-03-28T23:33:00.528+0000 7fbb026d3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:33:00.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.690+0000 7f91919d6640 1 -- 172.21.15.161:0/4100950254 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f918c0c0ff0 con 0x7f918c0aacd0 2024-03-28T23:33:00.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.691+0000 7f9173fff640 1 -- 172.21.15.161:0/4100950254 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6507 (secure 0 0 0) 0x7f917c07b070 con 0x7f918c0aacd0 2024-03-28T23:33:00.692 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:00.692 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":67,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.161:6837/737681942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":737681942},{"type":"v1","addr":"172.21.15.161:6837","nonce":737681942}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":60},{"gid":5579,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.170:6834/3501962065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3501962065},{"type":"v1","addr":"172.21.15.170:6834","nonce":3501962065}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":67,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:32:32.372964+0000","modified":"2024-03-28T23:32:58.528763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6807},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6807":{"gid":6807,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.161:6836/2378263687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":2378263687},{"type":"v1","addr":"172.21.15.161:6836","nonce":2378263687}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6807,"qdb_cluster":[6807]},"id":5},{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:32:44.463287+0000","modified":"2024-03-28T23:32:46.605012+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6037":{"gid":6037,"name":"b","rank":0,"incarnation":57,"state":"up:active","state_seq":6,"addr":"172.21.15.170:6835/2691084827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":2691084827},{"type":"v1","addr":"172.21.15.170:6835","nonce":2691084827}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6037,"qdb_cluster":[6037]},"id":6}]} 2024-03-28T23:33:00.692 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 67 2024-03-28T23:33:00.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.693+0000 7f91919d6640 1 -- 172.21.15.161:0/4100950254 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f917806b4d0 msgr2=0x7f917806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:00.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.693+0000 7f91919d6640 1 --2- 172.21.15.161:0/4100950254 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f917806b4d0 0x7f917806d990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f918c14d4d0 tx=0x7f9180009040 comp rx=0 tx=0).stop 2024-03-28T23:33:00.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.693+0000 7f91919d6640 1 -- 172.21.15.161:0/4100950254 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f918c0aacd0 msgr2=0x7f918c1dfaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:00.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.693+0000 7f91919d6640 1 --2- 172.21.15.161:0/4100950254 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f918c0aacd0 0x7f918c1dfaa0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f917c036040 tx=0x7f917c004a40 comp rx=0 tx=0).stop 2024-03-28T23:33:00.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.693+0000 7f91919d6640 1 -- 172.21.15.161:0/4100950254 shutdown_connections 2024-03-28T23:33:00.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.693+0000 7f91919d6640 1 --2- 172.21.15.161:0/4100950254 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f917806b4d0 0x7f917806d990 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:00.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.693+0000 7f91919d6640 1 --2- 172.21.15.161:0/4100950254 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f918c14be70 0x7f918c148910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:00.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.693+0000 7f91919d6640 1 --2- 172.21.15.161:0/4100950254 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f918c0ab6e0 0x7f918c155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:00.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.693+0000 7f91919d6640 1 --2- 172.21.15.161:0/4100950254 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f918c0aacd0 0x7f918c1dfaa0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:00.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.693+0000 7f91919d6640 1 -- 172.21.15.161:0/4100950254 >> 172.21.15.161:0/4100950254 conn(0x7f918c0c6670 msgr2=0x7f918c151e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:00.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.693+0000 7f91919d6640 1 -- 172.21.15.161:0/4100950254 shutdown_connections 2024-03-28T23:33:00.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:00.693+0000 7f91919d6640 1 -- 172.21.15.161:0/4100950254 wait complete. 2024-03-28T23:33:00.705 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-28T23:33:00.705 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-28T23:33:00.705 DEBUG:teuthology.orchestra.run.smithi161:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:33:00.742 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:33:00.779 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-28T23:33:00.779 INFO:tasks.cephfs.mount:Terminating background process 2024-03-28T23:33:00.779 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:33:00.779 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:33:00.797 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20793 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-28T23:33:00.798 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi161.front.sepia.ceph.com 2024-03-28T23:33:00.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:33:00.798 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-28T23:33:00.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:33:00.854 INFO:teuthology.orchestra.run.smithi161.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-28T23:33:00.854 INFO:tasks.cephfs.mount:Terminating background process 2024-03-28T23:33:00.854 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:33:00.854 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:33:00.869 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056374,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19690 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-28T23:33:00.870 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi170.front.sepia.ceph.com 2024-03-28T23:33:00.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:33:00.870 DEBUG:teuthology.orchestra.run.smithi170:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-28T23:33:00.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:33:00.924 INFO:teuthology.orchestra.run.smithi170.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-28T23:33:00.925 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:33:01.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.027+0000 7f56a27f0640 1 Processor -- start 2024-03-28T23:33:01.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.027+0000 7f56a27f0640 1 -- start start 2024-03-28T23:33:01.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.027+0000 7f56a27f0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f569c152890 0x7f569c0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:01.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.027+0000 7f56a27f0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f569c0c5b20 0x7f569c1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:01.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.027+0000 7f56a27f0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f569c158a30 con 0x7f569c0c8ed0 2024-03-28T23:33:01.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.027+0000 7f56a27f0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f569c0c8480 con 0x7f569c152890 2024-03-28T23:33:01.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.027+0000 7f56a27f0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f569c159640 con 0x7f569c0c5b20 2024-03-28T23:33:01.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.027+0000 7f569bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f569c152890 0x7f569c0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:01.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.028+0000 7f569bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f569c152890 0x7f569c0c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46018/0 (socket says 172.21.15.161:46018) 2024-03-28T23:33:01.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.028+0000 7f569bfff640 1 -- 172.21.15.161:0/2428590160 learned_addr learned my addr 172.21.15.161:0/2428590160 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:01.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.028+0000 7f56a0d8d640 1 --2- 172.21.15.161:0/2428590160 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f569c0c5b20 0x7f569c1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:01.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.028+0000 7f569bfff640 1 -- 172.21.15.161:0/2428590160 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f569c0c5b20 msgr2=0x7f569c1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:01.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.028+0000 7f569bfff640 1 --2- 172.21.15.161:0/2428590160 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f569c0c5b20 0x7f569c1553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.028+0000 7f569bfff640 1 -- 172.21.15.161:0/2428590160 >> v1:172.21.15.170:6789/0 conn(0x7f569c0c8ed0 legacy=0x7f569c0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:01.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.028+0000 7f569bfff640 1 -- 172.21.15.161:0/2428590160 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f569c158d10 con 0x7f569c152890 2024-03-28T23:33:01.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.028+0000 7f569affd640 1 -- 172.21.15.161:0/2428590160 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2439999385 0 0) 0x7f569c158a30 con 0x7f569c0c8ed0 2024-03-28T23:33:01.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.029+0000 7f569bfff640 1 --2- 172.21.15.161:0/2428590160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f569c152890 0x7f569c0c8990 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f569c1597c0 tx=0x7f568c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bea2a41554fd3f73 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:01.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.029+0000 7f569affd640 1 -- 172.21.15.161:0/2428590160 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568c0317c0 con 0x7f569c152890 2024-03-28T23:33:01.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.029+0000 7f569affd640 1 -- 172.21.15.161:0/2428590160 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f568c037070 con 0x7f569c152890 2024-03-28T23:33:01.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.029+0000 7f569affd640 1 -- 172.21.15.161:0/2428590160 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568c031b80 con 0x7f569c152890 2024-03-28T23:33:01.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.029+0000 7f56a27f0640 1 -- 172.21.15.161:0/2428590160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f569c152890 msgr2=0x7f569c0c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:01.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.029+0000 7f56a27f0640 1 --2- 172.21.15.161:0/2428590160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f569c152890 0x7f569c0c8990 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f569c1597c0 tx=0x7f568c02b860 comp rx=0 tx=0).stop 2024-03-28T23:33:01.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.029+0000 7f56a27f0640 1 -- 172.21.15.161:0/2428590160 shutdown_connections 2024-03-28T23:33:01.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.029+0000 7f56a27f0640 1 --2- 172.21.15.161:0/2428590160 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f569c0c5b20 0x7f569c1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.029+0000 7f56a27f0640 1 --2- 172.21.15.161:0/2428590160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f569c152890 0x7f569c0c8990 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.029+0000 7f56a27f0640 1 -- 172.21.15.161:0/2428590160 >> 172.21.15.161:0/2428590160 conn(0x7f569c0ac620 msgr2=0x7f569c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:01.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.029+0000 7f56a27f0640 1 -- 172.21.15.161:0/2428590160 shutdown_connections 2024-03-28T23:33:01.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.029+0000 7f56a27f0640 1 -- 172.21.15.161:0/2428590160 wait complete. 2024-03-28T23:33:01.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.030+0000 7f56a27f0640 1 Processor -- start 2024-03-28T23:33:01.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.031+0000 7f56a27f0640 1 -- start start 2024-03-28T23:33:01.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.032+0000 7f56a27f0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f569c0c5b20 0x7f569c0ab6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:01.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.032+0000 7f56a27f0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f569c0c8ed0 0x7f569c0c0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:01.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.032+0000 7f56a27f0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f569c152890 0x7f569c1eb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:01.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.032+0000 7f56a27f0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f569c152c70 con 0x7f569c152890 2024-03-28T23:33:01.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.032+0000 7f56a27f0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f569c1592d0 con 0x7f569c0c8ed0 2024-03-28T23:33:01.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.032+0000 7f56a27f0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f569c152df0 con 0x7f569c0c5b20 2024-03-28T23:33:01.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.032+0000 7f56a0d8d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f569c152890 0x7f569c1eb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:01.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.032+0000 7f56a0d8d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f569c152890 0x7f569c1eb290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46034/0 (socket says 172.21.15.161:46034) 2024-03-28T23:33:01.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.032+0000 7f569bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f569c0c5b20 0x7f569c0ab6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:01.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.032+0000 7f56a0d8d640 1 -- 172.21.15.161:0/2179328153 learned_addr learned my addr 172.21.15.161:0/2179328153 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:01.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.032+0000 7f569b7fe640 1 --2- 172.21.15.161:0/2179328153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f569c0c8ed0 0x7f569c0c0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:01.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.032+0000 7f56a0d8d640 1 -- 172.21.15.161:0/2179328153 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f569c0c5b20 msgr2=0x7f569c0ab6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:01.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.032+0000 7f56a0d8d640 1 --2- 172.21.15.161:0/2179328153 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f569c0c5b20 0x7f569c0ab6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.032+0000 7f56a0d8d640 1 -- 172.21.15.161:0/2179328153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f569c0c8ed0 msgr2=0x7f569c0c0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:01.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.032+0000 7f56a0d8d640 1 --2- 172.21.15.161:0/2179328153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f569c0c8ed0 0x7f569c0c0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.032+0000 7f56a0d8d640 1 -- 172.21.15.161:0/2179328153 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f569c0c4030 con 0x7f569c152890 2024-03-28T23:33:01.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.032+0000 7f56a0d8d640 1 --2- 172.21.15.161:0/2179328153 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f569c152890 0x7f569c1eb290 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f569400efc0 tx=0x7f569400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:01.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.032+0000 7f5698ff9640 1 -- 172.21.15.161:0/2179328153 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5694018020 con 0x7f569c152890 2024-03-28T23:33:01.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.033+0000 7f5698ff9640 1 -- 172.21.15.161:0/2179328153 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5694014480 con 0x7f569c152890 2024-03-28T23:33:01.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.033+0000 7f56a27f0640 1 -- 172.21.15.161:0/2179328153 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f569c1eaaf0 con 0x7f569c152890 2024-03-28T23:33:01.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.033+0000 7f5698ff9640 1 -- 172.21.15.161:0/2179328153 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5694009e60 con 0x7f569c152890 2024-03-28T23:33:01.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.033+0000 7f56a27f0640 1 -- 172.21.15.161:0/2179328153 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f569c0c01c0 con 0x7f569c152890 2024-03-28T23:33:01.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.033+0000 7f56a27f0640 1 -- 172.21.15.161:0/2179328153 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f569c04f4b0 con 0x7f569c152890 2024-03-28T23:33:01.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.034+0000 7f5698ff9640 1 -- 172.21.15.161:0/2179328153 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5694005d10 con 0x7f569c152890 2024-03-28T23:33:01.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.035+0000 7f5698ff9640 1 --2- 172.21.15.161:0/2179328153 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f568806b4d0 0x7f568806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:01.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.035+0000 7f5698ff9640 1 -- 172.21.15.161:0/2179328153 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7043+0+0 (secure 0 0 0) 0x7f569408b250 con 0x7f569c152890 2024-03-28T23:33:01.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.035+0000 7f569bfff640 1 --2- 172.21.15.161:0/2179328153 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f568806b4d0 0x7f568806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:01.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.035+0000 7f569bfff640 1 --2- 172.21.15.161:0/2179328153 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f568806b4d0 0x7f568806d990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f568c036000 tx=0x7f568c03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:01.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.039+0000 7f5698ff9640 1 -- 172.21.15.161:0/2179328153 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5694022030 con 0x7f569c152890 2024-03-28T23:33:01.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.258+0000 7f56a27f0640 1 -- 172.21.15.161:0/2179328153 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f569c074660 con 0x7f569c152890 2024-03-28T23:33:01.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.259+0000 7f5698ff9640 1 -- 172.21.15.161:0/2179328153 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6507 (secure 0 0 0) 0x7f569c1eaaf0 con 0x7f569c152890 2024-03-28T23:33:01.260 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:01.260 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":67,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.161:6837/737681942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":737681942},{"type":"v1","addr":"172.21.15.161:6837","nonce":737681942}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":60},{"gid":5579,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.170:6834/3501962065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3501962065},{"type":"v1","addr":"172.21.15.170:6834","nonce":3501962065}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":67,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:32:32.372964+0000","modified":"2024-03-28T23:32:58.528763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6807},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6807":{"gid":6807,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.161:6836/2378263687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":2378263687},{"type":"v1","addr":"172.21.15.161:6836","nonce":2378263687}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6807,"qdb_cluster":[6807]},"id":5},{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:32:44.463287+0000","modified":"2024-03-28T23:32:46.605012+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6037":{"gid":6037,"name":"b","rank":0,"incarnation":57,"state":"up:active","state_seq":6,"addr":"172.21.15.170:6835/2691084827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":2691084827},{"type":"v1","addr":"172.21.15.170:6835","nonce":2691084827}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6037,"qdb_cluster":[6037]},"id":6}]} 2024-03-28T23:33:01.262 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 67 2024-03-28T23:33:01.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.260+0000 7f56a27f0640 1 -- 172.21.15.161:0/2179328153 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f568806b4d0 msgr2=0x7f568806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:01.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.260+0000 7f56a27f0640 1 --2- 172.21.15.161:0/2179328153 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f568806b4d0 0x7f568806d990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f568c036000 tx=0x7f568c03d040 comp rx=0 tx=0).stop 2024-03-28T23:33:01.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.261+0000 7f56a27f0640 1 -- 172.21.15.161:0/2179328153 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f569c152890 msgr2=0x7f569c1eb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:01.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.261+0000 7f56a27f0640 1 --2- 172.21.15.161:0/2179328153 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f569c152890 0x7f569c1eb290 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f569400efc0 tx=0x7f569400c490 comp rx=0 tx=0).stop 2024-03-28T23:33:01.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.261+0000 7f56a27f0640 1 -- 172.21.15.161:0/2179328153 shutdown_connections 2024-03-28T23:33:01.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.261+0000 7f56a27f0640 1 --2- 172.21.15.161:0/2179328153 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f568806b4d0 0x7f568806d990 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.261+0000 7f56a27f0640 1 --2- 172.21.15.161:0/2179328153 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f569c152890 0x7f569c1eb290 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.261+0000 7f56a27f0640 1 --2- 172.21.15.161:0/2179328153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f569c0c8ed0 0x7f569c0c0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.261+0000 7f56a27f0640 1 --2- 172.21.15.161:0/2179328153 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f569c0c5b20 0x7f569c0ab6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.261+0000 7f56a27f0640 1 -- 172.21.15.161:0/2179328153 >> 172.21.15.161:0/2179328153 conn(0x7f569c0ac620 msgr2=0x7f569c0c5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:01.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.261+0000 7f56a27f0640 1 -- 172.21.15.161:0/2179328153 shutdown_connections 2024-03-28T23:33:01.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.261+0000 7f56a27f0640 1 -- 172.21.15.161:0/2179328153 wait complete. 2024-03-28T23:33:01.272 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:33:01.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.374+0000 7fcda4a55640 1 Processor -- start 2024-03-28T23:33:01.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.374+0000 7fcda4a55640 1 -- start start 2024-03-28T23:33:01.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.375+0000 7fcda4a55640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcda0152890 0x7fcda00c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:01.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.375+0000 7fcda4a55640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fcda00c5b20 0x7fcda01553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:01.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.375+0000 7fcda4a55640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcda0158a30 con 0x7fcda00c8ed0 2024-03-28T23:33:01.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.375+0000 7fcda4a55640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fcda00c8480 con 0x7fcda00c5b20 2024-03-28T23:33:01.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.375+0000 7fcda4a55640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fcda0159640 con 0x7fcda0152890 2024-03-28T23:33:01.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.375+0000 7fcd97fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fcda00c8ed0 0x7fcda00c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44858/0 (socket says 172.21.15.161:44858) 2024-03-28T23:33:01.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.375+0000 7fcd97fff640 1 -- 172.21.15.161:0/1362561 learned_addr learned my addr 172.21.15.161:0/1362561 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:01.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.375+0000 7fcd9e59c640 1 --2- 172.21.15.161:0/1362561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcda0152890 0x7fcda00c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:01.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.375+0000 7fcd9ed9d640 1 --2- 172.21.15.161:0/1362561 >> v2:172.21.15.170:3300/0 conn(0x7fcda00c5b20 0x7fcda01553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:01.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.376+0000 7fcd9dd9b640 1 -- 172.21.15.161:0/1362561 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 219155843 0 0) 0x7fcda0158a30 con 0x7fcda00c8ed0 2024-03-28T23:33:01.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.376+0000 7fcd9dd9b640 1 -- 172.21.15.161:0/1362561 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd8c003520 con 0x7fcda00c8ed0 2024-03-28T23:33:01.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.376+0000 7fcd9e59c640 1 -- 172.21.15.161:0/1362561 >> v2:172.21.15.170:3300/0 conn(0x7fcda00c5b20 msgr2=0x7fcda01553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:01.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.376+0000 7fcd9e59c640 1 --2- 172.21.15.161:0/1362561 >> v2:172.21.15.170:3300/0 conn(0x7fcda00c5b20 0x7fcda01553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.376+0000 7fcd9e59c640 1 -- 172.21.15.161:0/1362561 >> v1:172.21.15.161:6789/0 conn(0x7fcda00c8ed0 legacy=0x7fcda00c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:01.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.376+0000 7fcd9e59c640 1 -- 172.21.15.161:0/1362561 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcda0158d10 con 0x7fcda0152890 2024-03-28T23:33:01.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.376+0000 7fcd9ed9d640 1 --2- 172.21.15.161:0/1362561 >> v2:172.21.15.170:3300/0 conn(0x7fcda00c5b20 0x7fcda01553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:01.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.377+0000 7fcd9e59c640 1 --2- 172.21.15.161:0/1362561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcda0152890 0x7fcda00c8990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcda01597c0 tx=0x7fcd9002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=757cb07c445c7a3f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:01.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.377+0000 7fcd9dd9b640 1 -- 172.21.15.161:0/1362561 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd90039070 con 0x7fcda0152890 2024-03-28T23:33:01.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.377+0000 7fcd9dd9b640 1 -- 172.21.15.161:0/1362561 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd900316f0 con 0x7fcda0152890 2024-03-28T23:33:01.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.377+0000 7fcd9dd9b640 1 -- 172.21.15.161:0/1362561 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd90031ac0 con 0x7fcda0152890 2024-03-28T23:33:01.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.377+0000 7fcda4a55640 1 -- 172.21.15.161:0/1362561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcda0152890 msgr2=0x7fcda00c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:01.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.377+0000 7fcda4a55640 1 --2- 172.21.15.161:0/1362561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcda0152890 0x7fcda00c8990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcda01597c0 tx=0x7fcd9002b790 comp rx=0 tx=0).stop 2024-03-28T23:33:01.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.378+0000 7fcda4a55640 1 -- 172.21.15.161:0/1362561 shutdown_connections 2024-03-28T23:33:01.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.378+0000 7fcda4a55640 1 --2- 172.21.15.161:0/1362561 >> v2:172.21.15.170:3300/0 conn(0x7fcda00c5b20 0x7fcda01553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.378+0000 7fcda4a55640 1 --2- 172.21.15.161:0/1362561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcda0152890 0x7fcda00c8990 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.378+0000 7fcda4a55640 1 -- 172.21.15.161:0/1362561 >> 172.21.15.161:0/1362561 conn(0x7fcda00ac620 msgr2=0x7fcda00aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:01.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.378+0000 7fcda4a55640 1 -- 172.21.15.161:0/1362561 shutdown_connections 2024-03-28T23:33:01.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.378+0000 7fcda4a55640 1 -- 172.21.15.161:0/1362561 wait complete. 2024-03-28T23:33:01.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.379+0000 7fcda4a55640 1 Processor -- start 2024-03-28T23:33:01.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.379+0000 7fcda4a55640 1 -- start start 2024-03-28T23:33:01.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.380+0000 7fcda4a55640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcda00c5b20 0x7fcda01eaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:01.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.380+0000 7fcda4a55640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcda00c8ed0 0x7fcda0158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:01.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.380+0000 7fcda4a55640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcda0152890 0x7fcda014d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:01.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.380+0000 7fcda4a55640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcda0152c70 con 0x7fcda00c5b20 2024-03-28T23:33:01.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.380+0000 7fcda4a55640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fcda01592d0 con 0x7fcda0152890 2024-03-28T23:33:01.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.380+0000 7fcda4a55640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fcda0152df0 con 0x7fcda00c8ed0 2024-03-28T23:33:01.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.380+0000 7fcd9e59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcda00c5b20 0x7fcda01eaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:01.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.380+0000 7fcd9e59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcda00c5b20 0x7fcda01eaeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46044/0 (socket says 172.21.15.161:46044) 2024-03-28T23:33:01.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.380+0000 7fcd9e59c640 1 -- 172.21.15.161:0/1705619754 learned_addr learned my addr 172.21.15.161:0/1705619754 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:01.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.380+0000 7fcd97fff640 1 --2- 172.21.15.161:0/1705619754 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcda00c8ed0 0x7fcda0158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:01.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.380+0000 7fcd9ed9d640 1 --2- 172.21.15.161:0/1705619754 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcda0152890 0x7fcda014d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:01.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.380+0000 7fcd9e59c640 1 -- 172.21.15.161:0/1705619754 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcda00c8ed0 msgr2=0x7fcda0158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:01.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.380+0000 7fcd9e59c640 1 --2- 172.21.15.161:0/1705619754 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcda00c8ed0 0x7fcda0158070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.380+0000 7fcd9e59c640 1 -- 172.21.15.161:0/1705619754 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcda0152890 msgr2=0x7fcda014d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:01.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.380+0000 7fcd9e59c640 1 --2- 172.21.15.161:0/1705619754 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcda0152890 0x7fcda014d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.380+0000 7fcd9e59c640 1 -- 172.21.15.161:0/1705619754 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcda01eb3f0 con 0x7fcda00c5b20 2024-03-28T23:33:01.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.380+0000 7fcd97fff640 1 --2- 172.21.15.161:0/1705619754 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcda00c8ed0 0x7fcda0158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:01.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.380+0000 7fcd9e59c640 1 --2- 172.21.15.161:0/1705619754 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcda00c5b20 0x7fcda01eaeb0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fcd90000c00 tx=0x7fcd900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:01.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.380+0000 7fcd96ffd640 1 -- 172.21.15.161:0/1705619754 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd90043070 con 0x7fcda00c5b20 2024-03-28T23:33:01.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.380+0000 7fcd96ffd640 1 -- 172.21.15.161:0/1705619754 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd90034680 con 0x7fcda00c5b20 2024-03-28T23:33:01.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.380+0000 7fcda4a55640 1 -- 172.21.15.161:0/1705619754 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcda01eb670 con 0x7fcda00c5b20 2024-03-28T23:33:01.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.380+0000 7fcd96ffd640 1 -- 172.21.15.161:0/1705619754 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd90039040 con 0x7fcda00c5b20 2024-03-28T23:33:01.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.380+0000 7fcda4a55640 1 -- 172.21.15.161:0/1705619754 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcda01ebc30 con 0x7fcda00c5b20 2024-03-28T23:33:01.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.382+0000 7fcda4a55640 1 -- 172.21.15.161:0/1705619754 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcda01ebe30 con 0x7fcda00c5b20 2024-03-28T23:33:01.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.382+0000 7fcd96ffd640 1 -- 172.21.15.161:0/1705619754 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcd9003c070 con 0x7fcda00c5b20 2024-03-28T23:33:01.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.383+0000 7fcd96ffd640 1 --2- 172.21.15.161:0/1705619754 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcd8006b4d0 0x7fcd8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:01.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.383+0000 7fcd96ffd640 1 -- 172.21.15.161:0/1705619754 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7043+0+0 (secure 0 0 0) 0x7fcd900ac0b0 con 0x7fcda00c5b20 2024-03-28T23:33:01.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.383+0000 7fcd97fff640 1 --2- 172.21.15.161:0/1705619754 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcd8006b4d0 0x7fcd8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:01.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.383+0000 7fcd97fff640 1 --2- 172.21.15.161:0/1705619754 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcd8006b4d0 0x7fcd8006d990 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcda006c8b0 tx=0x7fcd88009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:01.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.387+0000 7fcd96ffd640 1 -- 172.21.15.161:0/1705619754 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcda01ebe30 con 0x7fcda00c5b20 2024-03-28T23:33:01.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.568+0000 7fcda4a55640 1 -- 172.21.15.161:0/1705619754 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcda0067a50 con 0x7fcda00c5b20 2024-03-28T23:33:01.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.569+0000 7fcd96ffd640 1 -- 172.21.15.161:0/1705619754 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+20836 (secure 0 0 0) 0x7fcda0067a50 con 0x7fcda00c5b20 2024-03-28T23:33:01.570 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:01.570 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":78,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:32:54.518544+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:32:30.684936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-28T23:32:31.339722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:32:42.378609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:32:43.436985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6836/3850309502":"2024-03-29T23:32:54.518520+0000","172.21.15.161:6834/3850309502":"2024-03-29T23:32:54.518520+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:33:01.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.571+0000 7fcda4a55640 1 -- 172.21.15.161:0/1705619754 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcd8006b4d0 msgr2=0x7fcd8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:01.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.571+0000 7fcda4a55640 1 --2- 172.21.15.161:0/1705619754 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcd8006b4d0 0x7fcd8006d990 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcda006c8b0 tx=0x7fcd88009040 comp rx=0 tx=0).stop 2024-03-28T23:33:01.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.571+0000 7fcda4a55640 1 -- 172.21.15.161:0/1705619754 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcda00c5b20 msgr2=0x7fcda01eaeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:01.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.571+0000 7fcda4a55640 1 --2- 172.21.15.161:0/1705619754 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcda00c5b20 0x7fcda01eaeb0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fcd90000c00 tx=0x7fcd900046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.571+0000 7fcda4a55640 1 -- 172.21.15.161:0/1705619754 shutdown_connections 2024-03-28T23:33:01.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.571+0000 7fcda4a55640 1 --2- 172.21.15.161:0/1705619754 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcd8006b4d0 0x7fcd8006d990 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.571+0000 7fcda4a55640 1 --2- 172.21.15.161:0/1705619754 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcda0152890 0x7fcda014d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.571+0000 7fcda4a55640 1 --2- 172.21.15.161:0/1705619754 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcda00c8ed0 0x7fcda0158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.571+0000 7fcda4a55640 1 --2- 172.21.15.161:0/1705619754 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcda00c5b20 0x7fcda01eaeb0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.571+0000 7fcda4a55640 1 -- 172.21.15.161:0/1705619754 >> 172.21.15.161:0/1705619754 conn(0x7fcda00ac620 msgr2=0x7fcda00c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:01.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.572+0000 7fcda4a55640 1 -- 172.21.15.161:0/1705619754 shutdown_connections 2024-03-28T23:33:01.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.572+0000 7fcda4a55640 1 -- 172.21.15.161:0/1705619754 wait complete. 2024-03-28T23:33:01.583 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:32:30.684936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:32:31.339722+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-03-28T23:32:42.378609+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs2_data', 'create_time': '2024-03-28T23:32:43.436985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:33:01.584 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:33:01.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.688+0000 7ffac7b81640 1 Processor -- start 2024-03-28T23:33:01.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.688+0000 7ffac7b81640 1 -- start start 2024-03-28T23:33:01.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.688+0000 7ffac7b81640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ffac01529d0 0x7ffac006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:01.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.689+0000 7ffac7b81640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffac00c6750 0x7ffac00c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:01.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.689+0000 7ffac7b81640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffac0152db0 con 0x7ffac0155c30 2024-03-28T23:33:01.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.689+0000 7ffac7b81640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7ffac00c7210 con 0x7ffac01529d0 2024-03-28T23:33:01.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.689+0000 7ffac7b81640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ffac00c6c50 con 0x7ffac00c6750 2024-03-28T23:33:01.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.689+0000 7ffac591d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ffac01529d0 0x7ffac006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:01.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.689+0000 7ffac511c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffac00c6750 0x7ffac00c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:01.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.689+0000 7ffac591d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ffac01529d0 0x7ffac006bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:49434/0 (socket says 172.21.15.161:49434) 2024-03-28T23:33:01.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.689+0000 7ffac611e640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7ffac0155c30 0x7ffac00c54f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:44914/0 (socket says 172.21.15.161:44914) 2024-03-28T23:33:01.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.689+0000 7ffac511c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffac00c6750 0x7ffac00c88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:34292/0 (socket says 172.21.15.161:34292) 2024-03-28T23:33:01.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.689+0000 7ffac591d640 1 -- 172.21.15.161:0/2847445802 learned_addr learned my addr 172.21.15.161:0/2847445802 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:01.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.689+0000 7ffac491b640 1 -- 172.21.15.161:0/2847445802 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1686326619 0 0) 0x7ffac0152db0 con 0x7ffac0155c30 2024-03-28T23:33:01.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.689+0000 7ffac511c640 1 -- 172.21.15.161:0/2847445802 >> v2:172.21.15.170:3300/0 conn(0x7ffac01529d0 msgr2=0x7ffac006bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:01.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.689+0000 7ffac511c640 1 --2- 172.21.15.161:0/2847445802 >> v2:172.21.15.170:3300/0 conn(0x7ffac01529d0 0x7ffac006bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.689+0000 7ffac511c640 1 -- 172.21.15.161:0/2847445802 >> v1:172.21.15.170:6789/0 conn(0x7ffac0155c30 legacy=0x7ffac00c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:01.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.689+0000 7ffac511c640 1 -- 172.21.15.161:0/2847445802 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffac00c6f30 con 0x7ffac00c6750 2024-03-28T23:33:01.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.690+0000 7ffac511c640 1 --2- 172.21.15.161:0/2847445802 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffac00c6750 0x7ffac00c88b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ffab00097a0 tx=0x7ffab002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffa7308af42f018d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:01.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.690+0000 7ffac491b640 1 -- 172.21.15.161:0/2847445802 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab0035020 con 0x7ffac00c6750 2024-03-28T23:33:01.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.690+0000 7ffac491b640 1 -- 172.21.15.161:0/2847445802 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffab0031830 con 0x7ffac00c6750 2024-03-28T23:33:01.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.690+0000 7ffac491b640 1 -- 172.21.15.161:0/2847445802 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab0031c00 con 0x7ffac00c6750 2024-03-28T23:33:01.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.691+0000 7ffac7b81640 1 -- 172.21.15.161:0/2847445802 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffac00c6750 msgr2=0x7ffac00c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:01.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.691+0000 7ffac7b81640 1 --2- 172.21.15.161:0/2847445802 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffac00c6750 0x7ffac00c88b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ffab00097a0 tx=0x7ffab002b8d0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.691+0000 7ffac7b81640 1 -- 172.21.15.161:0/2847445802 shutdown_connections 2024-03-28T23:33:01.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.691+0000 7ffac7b81640 1 --2- 172.21.15.161:0/2847445802 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffac00c6750 0x7ffac00c88b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.691+0000 7ffac7b81640 1 --2- 172.21.15.161:0/2847445802 >> v2:172.21.15.170:3300/0 conn(0x7ffac01529d0 0x7ffac006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.691+0000 7ffac7b81640 1 -- 172.21.15.161:0/2847445802 >> 172.21.15.161:0/2847445802 conn(0x7ffac00c7430 msgr2=0x7ffac00c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:01.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.691+0000 7ffac7b81640 1 -- 172.21.15.161:0/2847445802 shutdown_connections 2024-03-28T23:33:01.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.691+0000 7ffac7b81640 1 -- 172.21.15.161:0/2847445802 wait complete. 2024-03-28T23:33:01.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.692+0000 7ffac7b81640 1 Processor -- start 2024-03-28T23:33:01.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.692+0000 7ffac7b81640 1 -- start start 2024-03-28T23:33:01.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.692+0000 7ffac7b81640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffac00c6750 0x7ffac0157080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:01.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.692+0000 7ffac7b81640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffac01529d0 0x7ffac014bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:01.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.692+0000 7ffac7b81640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffac0155c30 0x7ffac01eb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:01.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.692+0000 7ffac7b81640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ffac00aa160 con 0x7ffac0155c30 2024-03-28T23:33:01.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.692+0000 7ffac7b81640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ffac0155710 con 0x7ffac01529d0 2024-03-28T23:33:01.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.692+0000 7ffac7b81640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ffac00c5c00 con 0x7ffac00c6750 2024-03-28T23:33:01.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.693+0000 7ffac611e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffac0155c30 0x7ffac01eb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:01.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.693+0000 7ffac591d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffac00c6750 0x7ffac0157080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:01.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.693+0000 7ffac611e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffac0155c30 0x7ffac01eb740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46052/0 (socket says 172.21.15.161:46052) 2024-03-28T23:33:01.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.693+0000 7ffac511c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffac01529d0 0x7ffac014bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:01.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.693+0000 7ffac591d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffac00c6750 0x7ffac0157080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:34302/0 (socket says 172.21.15.161:34302) 2024-03-28T23:33:01.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.693+0000 7ffac611e640 1 -- 172.21.15.161:0/829548928 learned_addr learned my addr 172.21.15.161:0/829548928 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:01.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.693+0000 7ffac591d640 1 -- 172.21.15.161:0/829548928 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffac01529d0 msgr2=0x7ffac014bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:01.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.693+0000 7ffac591d640 1 --2- 172.21.15.161:0/829548928 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffac01529d0 0x7ffac014bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.693+0000 7ffac591d640 1 -- 172.21.15.161:0/829548928 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffac0155c30 msgr2=0x7ffac01eb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:01.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.693+0000 7ffac591d640 1 --2- 172.21.15.161:0/829548928 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffac0155c30 0x7ffac01eb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.693+0000 7ffac591d640 1 -- 172.21.15.161:0/829548928 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffac0159770 con 0x7ffac00c6750 2024-03-28T23:33:01.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.693+0000 7ffac591d640 1 --2- 172.21.15.161:0/829548928 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffac00c6750 0x7ffac0157080 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffabc00e8a0 tx=0x7ffabc00eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:01.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.693+0000 7ffab67fc640 1 -- 172.21.15.161:0/829548928 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffabc017070 con 0x7ffac00c6750 2024-03-28T23:33:01.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.693+0000 7ffab67fc640 1 -- 172.21.15.161:0/829548928 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffabc00c730 con 0x7ffac00c6750 2024-03-28T23:33:01.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.693+0000 7ffac7b81640 1 -- 172.21.15.161:0/829548928 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffac01594b0 con 0x7ffac00c6750 2024-03-28T23:33:01.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.693+0000 7ffab67fc640 1 -- 172.21.15.161:0/829548928 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffabc00cb00 con 0x7ffac00c6750 2024-03-28T23:33:01.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.693+0000 7ffac7b81640 1 -- 172.21.15.161:0/829548928 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffac01de970 con 0x7ffac00c6750 2024-03-28T23:33:01.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.694+0000 7ffac7b81640 1 -- 172.21.15.161:0/829548928 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffac004f4b0 con 0x7ffac00c6750 2024-03-28T23:33:01.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.695+0000 7ffab67fc640 1 -- 172.21.15.161:0/829548928 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ffabc01a070 con 0x7ffac00c6750 2024-03-28T23:33:01.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.696+0000 7ffab67fc640 1 --2- 172.21.15.161:0/829548928 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffaa406b4d0 0x7ffaa406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:01.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.696+0000 7ffab67fc640 1 -- 172.21.15.161:0/829548928 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(78..78 src has 1..78) ==== 7043+0+0 (secure 0 0 0) 0x7ffabc08a3c0 con 0x7ffac00c6750 2024-03-28T23:33:01.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.696+0000 7ffac511c640 1 --2- 172.21.15.161:0/829548928 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffaa406b4d0 0x7ffaa406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:01.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.696+0000 7ffac511c640 1 --2- 172.21.15.161:0/829548928 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffaa406b4d0 0x7ffaa406d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ffab0036000 tx=0x7ffab0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:01.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.700+0000 7ffab67fc640 1 -- 172.21.15.161:0/829548928 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ffac0159770 con 0x7ffac00c6750 2024-03-28T23:33:01.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.879+0000 7ffac7b81640 1 -- 172.21.15.161:0/829548928 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ffac00ca650 con 0x7ffac00c6750 2024-03-28T23:33:01.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.880+0000 7ffab67fc640 1 -- 172.21.15.161:0/829548928 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) ==== 93+0+10 (secure 0 0 0) 0x7ffabc058740 con 0x7ffac00c6750 2024-03-28T23:33:01.882 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:33:01.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.882+0000 7ffac7b81640 1 -- 172.21.15.161:0/829548928 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffaa406b4d0 msgr2=0x7ffaa406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:01.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.882+0000 7ffac7b81640 1 --2- 172.21.15.161:0/829548928 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffaa406b4d0 0x7ffaa406d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ffab0036000 tx=0x7ffab0008030 comp rx=0 tx=0).stop 2024-03-28T23:33:01.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.882+0000 7ffac7b81640 1 -- 172.21.15.161:0/829548928 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffac00c6750 msgr2=0x7ffac0157080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:01.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.882+0000 7ffac7b81640 1 --2- 172.21.15.161:0/829548928 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffac00c6750 0x7ffac0157080 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffabc00e8a0 tx=0x7ffabc00eca0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.882+0000 7ffac7b81640 1 -- 172.21.15.161:0/829548928 shutdown_connections 2024-03-28T23:33:01.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.882+0000 7ffac7b81640 1 --2- 172.21.15.161:0/829548928 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffaa406b4d0 0x7ffaa406d990 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.882+0000 7ffac7b81640 1 --2- 172.21.15.161:0/829548928 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffac0155c30 0x7ffac01eb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.882+0000 7ffac7b81640 1 --2- 172.21.15.161:0/829548928 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffac01529d0 0x7ffac014bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.882+0000 7ffac7b81640 1 --2- 172.21.15.161:0/829548928 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffac00c6750 0x7ffac0157080 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:01.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.882+0000 7ffac7b81640 1 -- 172.21.15.161:0/829548928 >> 172.21.15.161:0/829548928 conn(0x7ffac00c7430 msgr2=0x7ffac0151c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:01.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.882+0000 7ffac7b81640 1 -- 172.21.15.161:0/829548928 shutdown_connections 2024-03-28T23:33:01.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.882+0000 7ffac7b81640 1 -- 172.21.15.161:0/829548928 wait complete. 2024-03-28T23:33:01.893 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:33:01.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.996+0000 7f762b08f640 1 Processor -- start 2024-03-28T23:33:01.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.996+0000 7f762b08f640 1 -- start start 2024-03-28T23:33:02.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.998+0000 7f762b08f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7624152bc0 0x7f76240c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.998+0000 7f762b08f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f76240c8a80 0x7f76240c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.998+0000 7f762b08f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f76240c9420 0x7f76241589b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.998+0000 7f762b08f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f76240c6f60 con 0x7f76240c8a80 2024-03-28T23:33:02.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.998+0000 7f762b08f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f76240c6120 con 0x7f76240c9420 2024-03-28T23:33:02.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.998+0000 7f762b08f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f76240c6400 con 0x7f7624152bc0 2024-03-28T23:33:02.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.998+0000 7f7623fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f76240c8a80 0x7f76240c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.998+0000 7f7623fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f76240c8a80 0x7f76240c8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46060/0 (socket says 172.21.15.161:46060) 2024-03-28T23:33:02.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.998+0000 7f7623fff640 1 -- 172.21.15.161:0/4040547522 learned_addr learned my addr 172.21.15.161:0/4040547522 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:02.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.998+0000 7f7628e2b640 1 --2- 172.21.15.161:0/4040547522 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7624152bc0 0x7f76240c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.998+0000 7f762962c640 1 --2- 172.21.15.161:0/4040547522 >> v2:172.21.15.170:3300/0 conn(0x7f76240c9420 0x7f76241589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.998+0000 7f7628e2b640 1 -- 172.21.15.161:0/4040547522 >> v2:172.21.15.170:3300/0 conn(0x7f76240c9420 msgr2=0x7f76241589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:02.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.998+0000 7f7628e2b640 1 --2- 172.21.15.161:0/4040547522 >> v2:172.21.15.170:3300/0 conn(0x7f76240c9420 0x7f76241589b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.998+0000 7f7628e2b640 1 -- 172.21.15.161:0/4040547522 >> v2:172.21.15.161:3300/0 conn(0x7f76240c8a80 msgr2=0x7f76240c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:02.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.998+0000 7f7628e2b640 1 --2- 172.21.15.161:0/4040547522 >> v2:172.21.15.161:3300/0 conn(0x7f76240c8a80 0x7f76240c8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.998+0000 7f7628e2b640 1 -- 172.21.15.161:0/4040547522 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76240c7240 con 0x7f7624152bc0 2024-03-28T23:33:02.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.998+0000 7f7623fff640 1 --2- 172.21.15.161:0/4040547522 >> v2:172.21.15.161:3300/0 conn(0x7f76240c8a80 0x7f76240c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:02.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.999+0000 7f7628e2b640 1 --2- 172.21.15.161:0/4040547522 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7624152bc0 0x7f76240c0330 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f76240c8530 tx=0x7f761802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afc7bc93853ebea2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:02.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.999+0000 7f76237fe640 1 -- 172.21.15.161:0/4040547522 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76180315a0 con 0x7f7624152bc0 2024-03-28T23:33:02.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.999+0000 7f76237fe640 1 -- 172.21.15.161:0/4040547522 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7618037070 con 0x7f7624152bc0 2024-03-28T23:33:02.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.999+0000 7f76237fe640 1 -- 172.21.15.161:0/4040547522 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7618031960 con 0x7f7624152bc0 2024-03-28T23:33:02.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:01.999+0000 7f762b08f640 1 -- 172.21.15.161:0/4040547522 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7624152bc0 msgr2=0x7f76240c0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:02.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.000+0000 7f762b08f640 1 --2- 172.21.15.161:0/4040547522 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7624152bc0 0x7f76240c0330 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f76240c8530 tx=0x7f761802b840 comp rx=0 tx=0).stop 2024-03-28T23:33:02.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.000+0000 7f762b08f640 1 -- 172.21.15.161:0/4040547522 shutdown_connections 2024-03-28T23:33:02.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.000+0000 7f762b08f640 1 --2- 172.21.15.161:0/4040547522 >> v2:172.21.15.170:3300/0 conn(0x7f76240c9420 0x7f76241589b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.000+0000 7f762b08f640 1 --2- 172.21.15.161:0/4040547522 >> v2:172.21.15.161:3300/0 conn(0x7f76240c8a80 0x7f76240c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.000+0000 7f762b08f640 1 --2- 172.21.15.161:0/4040547522 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7624152bc0 0x7f76240c0330 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.000+0000 7f762b08f640 1 -- 172.21.15.161:0/4040547522 >> 172.21.15.161:0/4040547522 conn(0x7f76240caa80 msgr2=0x7f76240cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:02.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.000+0000 7f762b08f640 1 -- 172.21.15.161:0/4040547522 shutdown_connections 2024-03-28T23:33:02.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.000+0000 7f762b08f640 1 -- 172.21.15.161:0/4040547522 wait complete. 2024-03-28T23:33:02.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.001+0000 7f762b08f640 1 Processor -- start 2024-03-28T23:33:02.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.001+0000 7f762b08f640 1 -- start start 2024-03-28T23:33:02.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.002+0000 7f762b08f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76240c8a80 0x7f762406cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.002+0000 7f762b08f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76240c9420 0x7f7624148160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.002+0000 7f762b08f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7624152bc0 0x7f76241486a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.002+0000 7f762b08f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7624152fa0 con 0x7f76240c8a80 2024-03-28T23:33:02.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.002+0000 7f762b08f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7624158ef0 con 0x7f76240c9420 2024-03-28T23:33:02.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.002+0000 7f762b08f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f76241eb830 con 0x7f7624152bc0 2024-03-28T23:33:02.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.002+0000 7f7628e2b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76240c8a80 0x7f762406cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.002+0000 7f7628e2b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76240c8a80 0x7f762406cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46068/0 (socket says 172.21.15.161:46068) 2024-03-28T23:33:02.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.002+0000 7f7628e2b640 1 -- 172.21.15.161:0/3578541286 learned_addr learned my addr 172.21.15.161:0/3578541286 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:02.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.002+0000 7f762962c640 1 --2- 172.21.15.161:0/3578541286 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7624152bc0 0x7f76241486a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.002+0000 7f7623fff640 1 --2- 172.21.15.161:0/3578541286 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76240c9420 0x7f7624148160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.002+0000 7f7628e2b640 1 -- 172.21.15.161:0/3578541286 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7624152bc0 msgr2=0x7f76241486a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:02.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.002+0000 7f7628e2b640 1 --2- 172.21.15.161:0/3578541286 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7624152bc0 0x7f76241486a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.002+0000 7f7628e2b640 1 -- 172.21.15.161:0/3578541286 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76240c9420 msgr2=0x7f7624148160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:02.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.002+0000 7f7628e2b640 1 --2- 172.21.15.161:0/3578541286 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76240c9420 0x7f7624148160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.002+0000 7f7628e2b640 1 -- 172.21.15.161:0/3578541286 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7624151fc0 con 0x7f76240c8a80 2024-03-28T23:33:02.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.002+0000 7f7628e2b640 1 --2- 172.21.15.161:0/3578541286 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76240c8a80 0x7f762406cbb0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f761802be50 tx=0x7f76180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:02.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.002+0000 7f76217fa640 1 -- 172.21.15.161:0/3578541286 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7618032430 con 0x7f76240c8a80 2024-03-28T23:33:02.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.002+0000 7f76217fa640 1 -- 172.21.15.161:0/3578541286 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76180325d0 con 0x7f76240c8a80 2024-03-28T23:33:02.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.002+0000 7f762b08f640 1 -- 172.21.15.161:0/3578541286 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7624152280 con 0x7f76240c8a80 2024-03-28T23:33:02.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.002+0000 7f76217fa640 1 -- 172.21.15.161:0/3578541286 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76180329c0 con 0x7f76240c8a80 2024-03-28T23:33:02.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.002+0000 7f762b08f640 1 -- 172.21.15.161:0/3578541286 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f762406d0f0 con 0x7f76240c8a80 2024-03-28T23:33:02.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.003+0000 7f762b08f640 1 -- 172.21.15.161:0/3578541286 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f762404f4b0 con 0x7f76240c8a80 2024-03-28T23:33:02.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.004+0000 7f76217fa640 1 -- 172.21.15.161:0/3578541286 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f761804c020 con 0x7f76240c8a80 2024-03-28T23:33:02.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.004+0000 7f76217fa640 1 --2- 172.21.15.161:0/3578541286 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f760806b4d0 0x7f760806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.005+0000 7f7623fff640 1 --2- 172.21.15.161:0/3578541286 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f760806b4d0 0x7f760806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.005+0000 7f76217fa640 1 -- 172.21.15.161:0/3578541286 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7043+0+0 (secure 0 0 0) 0x7f761802f020 con 0x7f76240c8a80 2024-03-28T23:33:02.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.005+0000 7f7623fff640 1 --2- 172.21.15.161:0/3578541286 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f760806b4d0 0x7f760806d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7610009630 tx=0x7f7610009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:02.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.009+0000 7f76217fa640 1 -- 172.21.15.161:0/3578541286 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7618040070 con 0x7f76240c8a80 2024-03-28T23:33:02.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.181+0000 7f762b08f640 1 -- 172.21.15.161:0/3578541286 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f76240b0c70 con 0x7f76240c8a80 2024-03-28T23:33:02.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.181+0000 7f76217fa640 1 -- 172.21.15.161:0/3578541286 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) ==== 104+0+11 (secure 0 0 0) 0x7f761807a9a0 con 0x7f76240c8a80 2024-03-28T23:33:02.182 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:33:02.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.183+0000 7f762b08f640 1 -- 172.21.15.161:0/3578541286 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f760806b4d0 msgr2=0x7f760806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:02.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.183+0000 7f762b08f640 1 --2- 172.21.15.161:0/3578541286 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f760806b4d0 0x7f760806d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7610009630 tx=0x7f7610009240 comp rx=0 tx=0).stop 2024-03-28T23:33:02.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.183+0000 7f762b08f640 1 -- 172.21.15.161:0/3578541286 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76240c8a80 msgr2=0x7f762406cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:02.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.183+0000 7f762b08f640 1 --2- 172.21.15.161:0/3578541286 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76240c8a80 0x7f762406cbb0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f761802be50 tx=0x7f76180046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.183+0000 7f762b08f640 1 -- 172.21.15.161:0/3578541286 shutdown_connections 2024-03-28T23:33:02.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.183+0000 7f762b08f640 1 --2- 172.21.15.161:0/3578541286 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f760806b4d0 0x7f760806d990 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.183+0000 7f762b08f640 1 --2- 172.21.15.161:0/3578541286 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7624152bc0 0x7f76241486a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.183+0000 7f762b08f640 1 --2- 172.21.15.161:0/3578541286 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76240c9420 0x7f7624148160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.183+0000 7f762b08f640 1 --2- 172.21.15.161:0/3578541286 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76240c8a80 0x7f762406cbb0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.183+0000 7f762b08f640 1 -- 172.21.15.161:0/3578541286 >> 172.21.15.161:0/3578541286 conn(0x7f76240caa80 msgr2=0x7f76240c0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:02.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.183+0000 7f762b08f640 1 -- 172.21.15.161:0/3578541286 shutdown_connections 2024-03-28T23:33:02.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.183+0000 7f762b08f640 1 -- 172.21.15.161:0/3578541286 wait complete. 2024-03-28T23:33:02.194 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:33:02.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.298+0000 7f49e367c640 1 Processor -- start 2024-03-28T23:33:02.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.298+0000 7f49e367c640 1 -- start start 2024-03-28T23:33:02.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.298+0000 7f49e367c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f49dc1529d0 0x7f49dc06bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.298+0000 7f49e367c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f49dc0c6750 0x7f49dc0c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.298+0000 7f49e367c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49dc155c30 0x7f49dc0c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.298+0000 7f49e367c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f49dc152db0 con 0x7f49dc1529d0 2024-03-28T23:33:02.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.298+0000 7f49e367c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f49dc0c7210 con 0x7f49dc0c6750 2024-03-28T23:33:02.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.298+0000 7f49e367c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f49dc0c6c50 con 0x7f49dc155c30 2024-03-28T23:33:02.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.299+0000 7f49e1418640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f49dc1529d0 0x7f49dc06bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.299+0000 7f49e1c19640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49dc155c30 0x7f49dc0c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.299+0000 7f49e0c17640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f49dc0c6750 0x7f49dc0c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.299+0000 7f49e1418640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f49dc1529d0 0x7f49dc06bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46072/0 (socket says 172.21.15.161:46072) 2024-03-28T23:33:02.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.299+0000 7f49e1418640 1 -- 172.21.15.161:0/3840616777 learned_addr learned my addr 172.21.15.161:0/3840616777 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:02.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.299+0000 7f49e1c19640 1 -- 172.21.15.161:0/3840616777 >> v2:172.21.15.170:3300/0 conn(0x7f49dc0c6750 msgr2=0x7f49dc0c88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:02.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.299+0000 7f49e1c19640 1 --2- 172.21.15.161:0/3840616777 >> v2:172.21.15.170:3300/0 conn(0x7f49dc0c6750 0x7f49dc0c88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.299+0000 7f49e1c19640 1 -- 172.21.15.161:0/3840616777 >> v2:172.21.15.161:3300/0 conn(0x7f49dc1529d0 msgr2=0x7f49dc06bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:02.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.299+0000 7f49e1c19640 1 --2- 172.21.15.161:0/3840616777 >> v2:172.21.15.161:3300/0 conn(0x7f49dc1529d0 0x7f49dc06bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.299+0000 7f49e1c19640 1 -- 172.21.15.161:0/3840616777 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49dc0c6f30 con 0x7f49dc155c30 2024-03-28T23:33:02.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.299+0000 7f49e1418640 1 --2- 172.21.15.161:0/3840616777 >> v2:172.21.15.161:3300/0 conn(0x7f49dc1529d0 0x7f49dc06bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:02.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.300+0000 7f49e1c19640 1 --2- 172.21.15.161:0/3840616777 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49dc155c30 0x7f49dc0c54f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f49dc0739b0 tx=0x7f49d802be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad6576b18b13e2dc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:02.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.300+0000 7f49d3fff640 1 -- 172.21.15.161:0/3840616777 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d8039070 con 0x7f49dc155c30 2024-03-28T23:33:02.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.300+0000 7f49d3fff640 1 -- 172.21.15.161:0/3840616777 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49d802cbd0 con 0x7f49dc155c30 2024-03-28T23:33:02.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.300+0000 7f49d3fff640 1 -- 172.21.15.161:0/3840616777 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d8034440 con 0x7f49dc155c30 2024-03-28T23:33:02.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.300+0000 7f49e367c640 1 -- 172.21.15.161:0/3840616777 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49dc155c30 msgr2=0x7f49dc0c54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:02.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.300+0000 7f49e367c640 1 --2- 172.21.15.161:0/3840616777 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49dc155c30 0x7f49dc0c54f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f49dc0739b0 tx=0x7f49d802be20 comp rx=0 tx=0).stop 2024-03-28T23:33:02.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.301+0000 7f49e367c640 1 -- 172.21.15.161:0/3840616777 shutdown_connections 2024-03-28T23:33:02.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.301+0000 7f49e367c640 1 --2- 172.21.15.161:0/3840616777 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49dc155c30 0x7f49dc0c54f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.301+0000 7f49e367c640 1 --2- 172.21.15.161:0/3840616777 >> v2:172.21.15.170:3300/0 conn(0x7f49dc0c6750 0x7f49dc0c88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.301+0000 7f49e367c640 1 --2- 172.21.15.161:0/3840616777 >> v2:172.21.15.161:3300/0 conn(0x7f49dc1529d0 0x7f49dc06bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.301+0000 7f49e367c640 1 -- 172.21.15.161:0/3840616777 >> 172.21.15.161:0/3840616777 conn(0x7f49dc0c7430 msgr2=0x7f49dc0c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:02.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.301+0000 7f49e367c640 1 -- 172.21.15.161:0/3840616777 shutdown_connections 2024-03-28T23:33:02.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.301+0000 7f49e367c640 1 -- 172.21.15.161:0/3840616777 wait complete. 2024-03-28T23:33:02.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.302+0000 7f49e367c640 1 Processor -- start 2024-03-28T23:33:02.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.302+0000 7f49e367c640 1 -- start start 2024-03-28T23:33:02.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.302+0000 7f49e367c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49dc0c6750 0x7f49dc0ab5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.302+0000 7f49e367c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49dc1529d0 0x7f49dc1eb230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.302+0000 7f49e367c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49dc155c30 0x7f49dc1eb770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.302+0000 7f49e367c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc0aa160 con 0x7f49dc0c6750 2024-03-28T23:33:02.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.302+0000 7f49e367c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc155710 con 0x7f49dc155c30 2024-03-28T23:33:02.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.302+0000 7f49e367c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f49dc1560e0 con 0x7f49dc1529d0 2024-03-28T23:33:02.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.303+0000 7f49e1418640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49dc0c6750 0x7f49dc0ab5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.303+0000 7f49e1418640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49dc0c6750 0x7f49dc0ab5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46080/0 (socket says 172.21.15.161:46080) 2024-03-28T23:33:02.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.303+0000 7f49e1418640 1 -- 172.21.15.161:0/1983196484 learned_addr learned my addr 172.21.15.161:0/1983196484 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:02.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.303+0000 7f49e0c17640 1 --2- 172.21.15.161:0/1983196484 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49dc1529d0 0x7f49dc1eb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.303+0000 7f49e1c19640 1 --2- 172.21.15.161:0/1983196484 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49dc155c30 0x7f49dc1eb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.303+0000 7f49e1418640 1 -- 172.21.15.161:0/1983196484 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49dc1529d0 msgr2=0x7f49dc1eb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:02.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.303+0000 7f49e1418640 1 --2- 172.21.15.161:0/1983196484 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49dc1529d0 0x7f49dc1eb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.303+0000 7f49e1418640 1 -- 172.21.15.161:0/1983196484 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49dc155c30 msgr2=0x7f49dc1eb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:02.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.303+0000 7f49e1418640 1 --2- 172.21.15.161:0/1983196484 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49dc155c30 0x7f49dc1eb770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.303+0000 7f49e1418640 1 -- 172.21.15.161:0/1983196484 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49dc0c4a60 con 0x7f49dc0c6750 2024-03-28T23:33:02.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.303+0000 7f49e1418640 1 --2- 172.21.15.161:0/1983196484 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49dc0c6750 0x7f49dc0ab5a0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f49d400e9d0 tx=0x7f49d400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:02.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.303+0000 7f49d1ffb640 1 -- 172.21.15.161:0/1983196484 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d4017070 con 0x7f49dc0c6750 2024-03-28T23:33:02.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.303+0000 7f49e367c640 1 -- 172.21.15.161:0/1983196484 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49dc148d80 con 0x7f49dc0c6750 2024-03-28T23:33:02.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.303+0000 7f49d1ffb640 1 -- 172.21.15.161:0/1983196484 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49d400ca60 con 0x7f49dc0c6750 2024-03-28T23:33:02.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.303+0000 7f49d1ffb640 1 -- 172.21.15.161:0/1983196484 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d400ce10 con 0x7f49dc0c6750 2024-03-28T23:33:02.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.303+0000 7f49e367c640 1 -- 172.21.15.161:0/1983196484 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49dc1ead10 con 0x7f49dc0c6750 2024-03-28T23:33:02.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.305+0000 7f49e367c640 1 -- 172.21.15.161:0/1983196484 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49dc04f4b0 con 0x7f49dc0c6750 2024-03-28T23:33:02.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.305+0000 7f49d1ffb640 1 -- 172.21.15.161:0/1983196484 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f49d401a070 con 0x7f49dc0c6750 2024-03-28T23:33:02.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.305+0000 7f49d1ffb640 1 --2- 172.21.15.161:0/1983196484 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f49c006b4d0 0x7f49c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.305+0000 7f49d1ffb640 1 -- 172.21.15.161:0/1983196484 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7043+0+0 (secure 0 0 0) 0x7f49d408a6a0 con 0x7f49dc0c6750 2024-03-28T23:33:02.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.305+0000 7f49e0c17640 1 --2- 172.21.15.161:0/1983196484 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f49c006b4d0 0x7f49c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.306+0000 7f49e0c17640 1 --2- 172.21.15.161:0/1983196484 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f49c006b4d0 0x7f49c006d990 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f49dc0abd60 tx=0x7f49cc009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:02.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.310+0000 7f49d1ffb640 1 -- 172.21.15.161:0/1983196484 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f49d4058910 con 0x7f49dc0c6750 2024-03-28T23:33:02.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.488+0000 7f49e367c640 1 -- 172.21.15.161:0/1983196484 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f49dc0ca650 con 0x7f49dc0c6750 2024-03-28T23:33:02.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.488+0000 7f49d1ffb640 1 -- 172.21.15.161:0/1983196484 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) ==== 100+0+11 (secure 0 0 0) 0x7f49d405c9b0 con 0x7f49dc0c6750 2024-03-28T23:33:02.489 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:33:02.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.491+0000 7f49e367c640 1 -- 172.21.15.161:0/1983196484 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f49c006b4d0 msgr2=0x7f49c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:02.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.491+0000 7f49e367c640 1 --2- 172.21.15.161:0/1983196484 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f49c006b4d0 0x7f49c006d990 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f49dc0abd60 tx=0x7f49cc009190 comp rx=0 tx=0).stop 2024-03-28T23:33:02.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.491+0000 7f49e367c640 1 -- 172.21.15.161:0/1983196484 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49dc0c6750 msgr2=0x7f49dc0ab5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:02.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.491+0000 7f49e367c640 1 --2- 172.21.15.161:0/1983196484 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49dc0c6750 0x7f49dc0ab5a0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f49d400e9d0 tx=0x7f49d400ee90 comp rx=0 tx=0).stop 2024-03-28T23:33:02.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.491+0000 7f49e367c640 1 -- 172.21.15.161:0/1983196484 shutdown_connections 2024-03-28T23:33:02.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.491+0000 7f49e367c640 1 --2- 172.21.15.161:0/1983196484 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f49c006b4d0 0x7f49c006d990 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.491+0000 7f49e367c640 1 --2- 172.21.15.161:0/1983196484 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49dc155c30 0x7f49dc1eb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.491+0000 7f49e367c640 1 --2- 172.21.15.161:0/1983196484 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49dc1529d0 0x7f49dc1eb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.491+0000 7f49e367c640 1 --2- 172.21.15.161:0/1983196484 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49dc0c6750 0x7f49dc0ab5a0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.491+0000 7f49e367c640 1 -- 172.21.15.161:0/1983196484 >> 172.21.15.161:0/1983196484 conn(0x7f49dc0c7430 msgr2=0x7f49dc0c5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:02.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.491+0000 7f49e367c640 1 -- 172.21.15.161:0/1983196484 shutdown_connections 2024-03-28T23:33:02.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.491+0000 7f49e367c640 1 -- 172.21.15.161:0/1983196484 wait complete. 2024-03-28T23:33:02.502 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-03-28T23:33:02.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.610+0000 7f24adc9d640 1 Processor -- start 2024-03-28T23:33:02.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.610+0000 7f24adc9d640 1 -- start start 2024-03-28T23:33:02.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.611+0000 7f24adc9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24a81589b0 0x7f24a8158e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.611+0000 7f24adc9d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24a80c6f60 con 0x7f24a8152bc0 2024-03-28T23:33:02.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.611+0000 7f24adc9d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24a80c6120 con 0x7f24a80c0330 2024-03-28T23:33:02.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.611+0000 7f24adc9d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f24a80c6400 con 0x7f24a81589b0 2024-03-28T23:33:02.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.611+0000 7f24a77fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f24a8152bc0 0x7f24a80c8a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44866/0 (socket says 172.21.15.161:44866) 2024-03-28T23:33:02.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.611+0000 7f24a77fe640 1 -- 172.21.15.161:0/4161988231 learned_addr learned my addr 172.21.15.161:0/4161988231 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:02.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.611+0000 7f24a7fff640 1 --2- 172.21.15.161:0/4161988231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24a81589b0 0x7f24a8158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.612+0000 7f24a67fc640 1 -- 172.21.15.161:0/4161988231 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1125249817 0 0) 0x7f24a80c6f60 con 0x7f24a8152bc0 2024-03-28T23:33:02.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.612+0000 7f24a67fc640 1 -- 172.21.15.161:0/4161988231 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2490003520 con 0x7f24a8152bc0 2024-03-28T23:33:02.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.612+0000 7f24a7fff640 1 -- 172.21.15.161:0/4161988231 >> v1:172.21.15.170:6789/0 conn(0x7f24a80c0330 legacy=0x7f24a80c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:02.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.612+0000 7f24a67fc640 1 -- 172.21.15.161:0/4161988231 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3196126261 0 0) 0x7f24a80c6120 con 0x7f24a80c0330 2024-03-28T23:33:02.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.612+0000 7f24a7fff640 1 -- 172.21.15.161:0/4161988231 >> v1:172.21.15.161:6789/0 conn(0x7f24a8152bc0 legacy=0x7f24a80c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:02.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.612+0000 7f24a7fff640 1 -- 172.21.15.161:0/4161988231 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a80c7240 con 0x7f24a81589b0 2024-03-28T23:33:02.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.613+0000 7f24a7fff640 1 --2- 172.21.15.161:0/4161988231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24a81589b0 0x7f24a8158e40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f24a0009dc0 tx=0x7f24a002be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba3e501839027cd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:02.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.613+0000 7f24a67fc640 1 -- 172.21.15.161:0/4161988231 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a0039070 con 0x7f24a81589b0 2024-03-28T23:33:02.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.613+0000 7f24a67fc640 1 -- 172.21.15.161:0/4161988231 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24a002cc40 con 0x7f24a81589b0 2024-03-28T23:33:02.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.613+0000 7f24a67fc640 1 -- 172.21.15.161:0/4161988231 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a0034440 con 0x7f24a81589b0 2024-03-28T23:33:02.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.614+0000 7f24adc9d640 1 -- 172.21.15.161:0/4161988231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24a81589b0 msgr2=0x7f24a8158e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:02.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.614+0000 7f24adc9d640 1 --2- 172.21.15.161:0/4161988231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24a81589b0 0x7f24a8158e40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f24a0009dc0 tx=0x7f24a002be90 comp rx=0 tx=0).stop 2024-03-28T23:33:02.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.614+0000 7f24adc9d640 1 -- 172.21.15.161:0/4161988231 shutdown_connections 2024-03-28T23:33:02.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.614+0000 7f24adc9d640 1 --2- 172.21.15.161:0/4161988231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24a81589b0 0x7f24a8158e40 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.614+0000 7f24adc9d640 1 -- 172.21.15.161:0/4161988231 >> 172.21.15.161:0/4161988231 conn(0x7f24a80caa80 msgr2=0x7f24a80cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:02.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.614+0000 7f24adc9d640 1 -- 172.21.15.161:0/4161988231 shutdown_connections 2024-03-28T23:33:02.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.614+0000 7f24adc9d640 1 -- 172.21.15.161:0/4161988231 wait complete. 2024-03-28T23:33:02.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.615+0000 7f24adc9d640 1 Processor -- start 2024-03-28T23:33:02.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.615+0000 7f24adc9d640 1 -- start start 2024-03-28T23:33:02.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.615+0000 7f24adc9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24a80c0330 0x7f24a806c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.615+0000 7f24adc9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24a8152bc0 0x7f24a8149770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.615+0000 7f24adc9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24a81589b0 0x7f24a8149cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.615+0000 7f24adc9d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f24a8152fa0 con 0x7f24a80c0330 2024-03-28T23:33:02.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.615+0000 7f24adc9d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f24a8159380 con 0x7f24a8152bc0 2024-03-28T23:33:02.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.615+0000 7f24adc9d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f24a814b2b0 con 0x7f24a81589b0 2024-03-28T23:33:02.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.616+0000 7f24a77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24a80c0330 0x7f24a806c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.616+0000 7f24a77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24a80c0330 0x7f24a806c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46084/0 (socket says 172.21.15.161:46084) 2024-03-28T23:33:02.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.616+0000 7f24a77fe640 1 -- 172.21.15.161:0/2602425494 learned_addr learned my addr 172.21.15.161:0/2602425494 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:02.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.616+0000 7f24a7fff640 1 --2- 172.21.15.161:0/2602425494 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24a81589b0 0x7f24a8149cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.616+0000 7f24a6ffd640 1 --2- 172.21.15.161:0/2602425494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24a8152bc0 0x7f24a8149770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.616+0000 7f24a77fe640 1 -- 172.21.15.161:0/2602425494 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24a81589b0 msgr2=0x7f24a8149cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:02.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.616+0000 7f24a77fe640 1 --2- 172.21.15.161:0/2602425494 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24a81589b0 0x7f24a8149cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.616+0000 7f24a77fe640 1 -- 172.21.15.161:0/2602425494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24a8152bc0 msgr2=0x7f24a8149770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:02.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.616+0000 7f24a77fe640 1 --2- 172.21.15.161:0/2602425494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24a8152bc0 0x7f24a8149770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.616+0000 7f24a77fe640 1 -- 172.21.15.161:0/2602425494 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a814a960 con 0x7f24a80c0330 2024-03-28T23:33:02.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.616+0000 7f24a77fe640 1 --2- 172.21.15.161:0/2602425494 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24a80c0330 0x7f24a806c4d0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f249800c460 tx=0x7f249800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:02.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.616+0000 7f248ffff640 1 -- 172.21.15.161:0/2602425494 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2498017070 con 0x7f24a80c0330 2024-03-28T23:33:02.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.616+0000 7f248ffff640 1 -- 172.21.15.161:0/2602425494 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2498005ce0 con 0x7f24a80c0330 2024-03-28T23:33:02.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.616+0000 7f24adc9d640 1 -- 172.21.15.161:0/2602425494 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24a814ac40 con 0x7f24a80c0330 2024-03-28T23:33:02.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.616+0000 7f248ffff640 1 -- 172.21.15.161:0/2602425494 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2498007290 con 0x7f24a80c0330 2024-03-28T23:33:02.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.616+0000 7f24adc9d640 1 -- 172.21.15.161:0/2602425494 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24a814aef0 con 0x7f24a80c0330 2024-03-28T23:33:02.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.617+0000 7f24adc9d640 1 -- 172.21.15.161:0/2602425494 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24a80c8320 con 0x7f24a80c0330 2024-03-28T23:33:02.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.618+0000 7f248ffff640 1 -- 172.21.15.161:0/2602425494 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f249801a070 con 0x7f24a80c0330 2024-03-28T23:33:02.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.618+0000 7f248ffff640 1 --2- 172.21.15.161:0/2602425494 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f249406b4d0 0x7f249406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.618+0000 7f248ffff640 1 -- 172.21.15.161:0/2602425494 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7043+0+0 (secure 0 0 0) 0x7f2498089990 con 0x7f24a80c0330 2024-03-28T23:33:02.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.618+0000 7f24a6ffd640 1 --2- 172.21.15.161:0/2602425494 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f249406b4d0 0x7f249406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.619+0000 7f24a6ffd640 1 --2- 172.21.15.161:0/2602425494 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f249406b4d0 0x7f249406d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f249c000f60 tx=0x7f249c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:02.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.623+0000 7f248ffff640 1 -- 172.21.15.161:0/2602425494 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f24a80c8320 con 0x7f24a80c0330 2024-03-28T23:33:02.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.803+0000 7f24adc9d640 1 -- 172.21.15.161:0/2602425494 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) -- 0x7f24a8068f90 con 0x7f24a80c0330 2024-03-28T23:33:02.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.803+0000 7f248ffff640 1 -- 172.21.15.161:0/2602425494 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v78) ==== 105+0+11 (secure 0 0 0) 0x7f24a8068f90 con 0x7f24a80c0330 2024-03-28T23:33:02.804 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:33:02.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.805+0000 7f24adc9d640 1 -- 172.21.15.161:0/2602425494 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f249406b4d0 msgr2=0x7f249406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:02.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.805+0000 7f24adc9d640 1 --2- 172.21.15.161:0/2602425494 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f249406b4d0 0x7f249406d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f249c000f60 tx=0x7f249c009040 comp rx=0 tx=0).stop 2024-03-28T23:33:02.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.805+0000 7f24adc9d640 1 -- 172.21.15.161:0/2602425494 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24a80c0330 msgr2=0x7f24a806c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:02.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.805+0000 7f24adc9d640 1 --2- 172.21.15.161:0/2602425494 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24a80c0330 0x7f24a806c4d0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f249800c460 tx=0x7f249800c920 comp rx=0 tx=0).stop 2024-03-28T23:33:02.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.805+0000 7f24adc9d640 1 -- 172.21.15.161:0/2602425494 shutdown_connections 2024-03-28T23:33:02.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.805+0000 7f24adc9d640 1 --2- 172.21.15.161:0/2602425494 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f249406b4d0 0x7f249406d990 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.805+0000 7f24adc9d640 1 --2- 172.21.15.161:0/2602425494 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24a81589b0 0x7f24a8149cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.805+0000 7f24adc9d640 1 --2- 172.21.15.161:0/2602425494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24a8152bc0 0x7f24a8149770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.805+0000 7f24adc9d640 1 --2- 172.21.15.161:0/2602425494 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24a80c0330 0x7f24a806c4d0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.805+0000 7f24adc9d640 1 -- 172.21.15.161:0/2602425494 >> 172.21.15.161:0/2602425494 conn(0x7f24a80caa80 msgr2=0x7f24a80cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:02.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.805+0000 7f24adc9d640 1 -- 172.21.15.161:0/2602425494 shutdown_connections 2024-03-28T23:33:02.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.805+0000 7f24adc9d640 1 -- 172.21.15.161:0/2602425494 wait complete. 2024-03-28T23:33:02.820 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-03-28T23:33:02.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.924+0000 7fd217e39640 1 Processor -- start 2024-03-28T23:33:02.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.924+0000 7fd217e39640 1 -- start start 2024-03-28T23:33:02.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.925+0000 7fd217e39640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd2101529d0 0x7fd21006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.925+0000 7fd217e39640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2100c6750 0x7fd2100c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.925+0000 7fd217e39640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd210152db0 con 0x7fd210155c30 2024-03-28T23:33:02.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.925+0000 7fd217e39640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd2100c7210 con 0x7fd2101529d0 2024-03-28T23:33:02.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.925+0000 7fd217e39640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd2100c6c50 con 0x7fd2100c6750 2024-03-28T23:33:02.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.925+0000 7fd215bd5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd2101529d0 0x7fd21006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.925+0000 7fd2153d4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2100c6750 0x7fd2100c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.925+0000 7fd2153d4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2100c6750 0x7fd2100c88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:34392/0 (socket says 172.21.15.161:34392) 2024-03-28T23:33:02.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.925+0000 7fd215bd5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd2101529d0 0x7fd21006bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:49502/0 (socket says 172.21.15.161:49502) 2024-03-28T23:33:02.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.925+0000 7fd2153d4640 1 -- 172.21.15.161:0/1001303290 learned_addr learned my addr 172.21.15.161:0/1001303290 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:02.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.925+0000 7fd2153d4640 1 -- 172.21.15.161:0/1001303290 >> v2:172.21.15.170:3300/0 conn(0x7fd2101529d0 msgr2=0x7fd21006bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:02.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.925+0000 7fd2153d4640 1 --2- 172.21.15.161:0/1001303290 >> v2:172.21.15.170:3300/0 conn(0x7fd2101529d0 0x7fd21006bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.925+0000 7fd214bd3640 1 -- 172.21.15.161:0/1001303290 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1273262387 0 0) 0x7fd210152db0 con 0x7fd210155c30 2024-03-28T23:33:02.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.925+0000 7fd2153d4640 1 -- 172.21.15.161:0/1001303290 >> v1:172.21.15.170:6789/0 conn(0x7fd210155c30 legacy=0x7fd2100c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:02.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.925+0000 7fd2153d4640 1 -- 172.21.15.161:0/1001303290 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2100c6f30 con 0x7fd2100c6750 2024-03-28T23:33:02.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.926+0000 7fd2153d4640 1 --2- 172.21.15.161:0/1001303290 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2100c6750 0x7fd2100c88b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd200009870 tx=0x7fd20002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cb80ec73cc51dc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:02.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.926+0000 7fd214bd3640 1 -- 172.21.15.161:0/1001303290 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd200034070 con 0x7fd2100c6750 2024-03-28T23:33:02.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.926+0000 7fd214bd3640 1 -- 172.21.15.161:0/1001303290 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd200030840 con 0x7fd2100c6750 2024-03-28T23:33:02.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.926+0000 7fd214bd3640 1 -- 172.21.15.161:0/1001303290 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd200030c10 con 0x7fd2100c6750 2024-03-28T23:33:02.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.927+0000 7fd217e39640 1 -- 172.21.15.161:0/1001303290 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2100c6750 msgr2=0x7fd2100c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:02.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.927+0000 7fd217e39640 1 --2- 172.21.15.161:0/1001303290 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2100c6750 0x7fd2100c88b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd200009870 tx=0x7fd20002b980 comp rx=0 tx=0).stop 2024-03-28T23:33:02.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.927+0000 7fd217e39640 1 -- 172.21.15.161:0/1001303290 shutdown_connections 2024-03-28T23:33:02.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.927+0000 7fd217e39640 1 --2- 172.21.15.161:0/1001303290 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2100c6750 0x7fd2100c88b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.927+0000 7fd217e39640 1 --2- 172.21.15.161:0/1001303290 >> v2:172.21.15.170:3300/0 conn(0x7fd2101529d0 0x7fd21006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.927+0000 7fd217e39640 1 -- 172.21.15.161:0/1001303290 >> 172.21.15.161:0/1001303290 conn(0x7fd2100c7430 msgr2=0x7fd2100c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:02.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.927+0000 7fd217e39640 1 -- 172.21.15.161:0/1001303290 shutdown_connections 2024-03-28T23:33:02.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.927+0000 7fd217e39640 1 -- 172.21.15.161:0/1001303290 wait complete. 2024-03-28T23:33:02.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.929+0000 7fd217e39640 1 Processor -- start 2024-03-28T23:33:02.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.929+0000 7fd217e39640 1 -- start start 2024-03-28T23:33:02.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.930+0000 7fd217e39640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2100c6750 0x7fd2100c2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.930+0000 7fd217e39640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2101529d0 0x7fd2100c2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.930+0000 7fd217e39640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd210155c30 0x7fd2101ee250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.930+0000 7fd217e39640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd2100aa160 con 0x7fd210155c30 2024-03-28T23:33:02.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.930+0000 7fd217e39640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd210155710 con 0x7fd2100c6750 2024-03-28T23:33:02.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.930+0000 7fd217e39640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd2100c5c00 con 0x7fd2101529d0 2024-03-28T23:33:02.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.930+0000 7fd2163d6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd210155c30 0x7fd2101ee250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.930+0000 7fd215bd5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2100c6750 0x7fd2100c2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.930+0000 7fd215bd5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2100c6750 0x7fd2100c2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:49516/0 (socket says 172.21.15.161:49516) 2024-03-28T23:33:02.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.930+0000 7fd2163d6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd210155c30 0x7fd2101ee250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46096/0 (socket says 172.21.15.161:46096) 2024-03-28T23:33:02.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.930+0000 7fd2153d4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2101529d0 0x7fd2100c2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.930+0000 7fd2163d6640 1 -- 172.21.15.161:0/3840948078 learned_addr learned my addr 172.21.15.161:0/3840948078 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:02.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.931+0000 7fd215bd5640 1 -- 172.21.15.161:0/3840948078 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2101529d0 msgr2=0x7fd2100c2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:02.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.931+0000 7fd215bd5640 1 --2- 172.21.15.161:0/3840948078 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2101529d0 0x7fd2100c2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.931+0000 7fd215bd5640 1 -- 172.21.15.161:0/3840948078 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd210155c30 msgr2=0x7fd2101ee250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:02.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.931+0000 7fd215bd5640 1 --2- 172.21.15.161:0/3840948078 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd210155c30 0x7fd2101ee250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:02.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.931+0000 7fd215bd5640 1 -- 172.21.15.161:0/3840948078 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2100c3e20 con 0x7fd2100c6750 2024-03-28T23:33:02.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.931+0000 7fd215bd5640 1 --2- 172.21.15.161:0/3840948078 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2100c6750 0x7fd2100c2870 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd20c00e870 tx=0x7fd20c00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:02.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.931+0000 7fd2067fc640 1 -- 172.21.15.161:0/3840948078 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd20c017070 con 0x7fd2100c6750 2024-03-28T23:33:02.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.931+0000 7fd2067fc640 1 -- 172.21.15.161:0/3840948078 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd20c00ee60 con 0x7fd2100c6750 2024-03-28T23:33:02.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.931+0000 7fd217e39640 1 -- 172.21.15.161:0/3840948078 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2101ef040 con 0x7fd2100c6750 2024-03-28T23:33:02.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.931+0000 7fd2067fc640 1 -- 172.21.15.161:0/3840948078 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd20c00cb60 con 0x7fd2100c6750 2024-03-28T23:33:02.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.931+0000 7fd217e39640 1 -- 172.21.15.161:0/3840948078 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2100c43c0 con 0x7fd2100c6750 2024-03-28T23:33:02.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.932+0000 7fd2067fc640 1 -- 172.21.15.161:0/3840948078 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd20c01a070 con 0x7fd2100c6750 2024-03-28T23:33:02.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.932+0000 7fd217e39640 1 -- 172.21.15.161:0/3840948078 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd21004f4b0 con 0x7fd2100c6750 2024-03-28T23:33:02.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.933+0000 7fd2067fc640 1 --2- 172.21.15.161:0/3840948078 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd1f406b4d0 0x7fd1f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:02.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.933+0000 7fd2067fc640 1 -- 172.21.15.161:0/3840948078 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7043+0+0 (secure 0 0 0) 0x7fd20c08a800 con 0x7fd2100c6750 2024-03-28T23:33:02.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.933+0000 7fd2153d4640 1 --2- 172.21.15.161:0/3840948078 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd1f406b4d0 0x7fd1f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:02.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.933+0000 7fd2153d4640 1 --2- 172.21.15.161:0/3840948078 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd1f406b4d0 0x7fd1f406d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd200036000 tx=0x7fd20003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:02.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:02.937+0000 7fd2067fc640 1 -- 172.21.15.161:0/3840948078 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd20c058b20 con 0x7fd2100c6750 2024-03-28T23:33:03.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.173+0000 7fd217e39640 1 -- 172.21.15.161:0/3840948078 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) -- 0x7fd2100c2490 con 0x7fd2100c6750 2024-03-28T23:33:03.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.174+0000 7fd2067fc640 1 -- 172.21.15.161:0/3840948078 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v78) ==== 101+0+11 (secure 0 0 0) 0x7fd20c05cbc0 con 0x7fd2100c6750 2024-03-28T23:33:03.175 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:33:03.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.176+0000 7fd217e39640 1 -- 172.21.15.161:0/3840948078 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd1f406b4d0 msgr2=0x7fd1f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:03.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.176+0000 7fd217e39640 1 --2- 172.21.15.161:0/3840948078 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd1f406b4d0 0x7fd1f406d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd200036000 tx=0x7fd20003c040 comp rx=0 tx=0).stop 2024-03-28T23:33:03.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.176+0000 7fd217e39640 1 -- 172.21.15.161:0/3840948078 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2100c6750 msgr2=0x7fd2100c2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:03.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.176+0000 7fd217e39640 1 --2- 172.21.15.161:0/3840948078 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2100c6750 0x7fd2100c2870 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd20c00e870 tx=0x7fd20c00ed30 comp rx=0 tx=0).stop 2024-03-28T23:33:03.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.176+0000 7fd217e39640 1 -- 172.21.15.161:0/3840948078 shutdown_connections 2024-03-28T23:33:03.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.176+0000 7fd217e39640 1 --2- 172.21.15.161:0/3840948078 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd1f406b4d0 0x7fd1f406d990 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.176+0000 7fd217e39640 1 --2- 172.21.15.161:0/3840948078 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd210155c30 0x7fd2101ee250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.176+0000 7fd217e39640 1 --2- 172.21.15.161:0/3840948078 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2101529d0 0x7fd2100c2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.176+0000 7fd217e39640 1 --2- 172.21.15.161:0/3840948078 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2100c6750 0x7fd2100c2870 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.176+0000 7fd217e39640 1 -- 172.21.15.161:0/3840948078 >> 172.21.15.161:0/3840948078 conn(0x7fd2100c7430 msgr2=0x7fd210153060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:03.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.176+0000 7fd217e39640 1 -- 172.21.15.161:0/3840948078 shutdown_connections 2024-03-28T23:33:03.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.176+0000 7fd217e39640 1 -- 172.21.15.161:0/3840948078 wait complete. 2024-03-28T23:33:03.188 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:33:03.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.289+0000 7f2ac119a640 1 Processor -- start 2024-03-28T23:33:03.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.289+0000 7f2ac119a640 1 -- start start 2024-03-28T23:33:03.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.290+0000 7f2ac119a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2abc152890 0x7f2abc0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:03.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.290+0000 7f2ac119a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2abc0c8ed0 0x7f2abc0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:03.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.290+0000 7f2ac119a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2abc158a30 con 0x7f2abc0c5950 2024-03-28T23:33:03.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.290+0000 7f2ac119a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f2abc0c8480 con 0x7f2abc152890 2024-03-28T23:33:03.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.290+0000 7f2ac119a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2abc159640 con 0x7f2abc0c8ed0 2024-03-28T23:33:03.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.290+0000 7f2abb59e640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f2abc0c5950 0x7f2abc1553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44872/0 (socket says 172.21.15.161:44872) 2024-03-28T23:33:03.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.290+0000 7f2abb59e640 1 -- 172.21.15.161:0/1412827406 learned_addr learned my addr 172.21.15.161:0/1412827406 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:03.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.290+0000 7f2aba59c640 1 --2- 172.21.15.161:0/1412827406 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2abc0c8ed0 0x7f2abc0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:03.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.290+0000 7f2abad9d640 1 --2- 172.21.15.161:0/1412827406 >> v2:172.21.15.170:3300/0 conn(0x7f2abc152890 0x7f2abc0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:03.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.290+0000 7f2ab9d9b640 1 -- 172.21.15.161:0/1412827406 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2960157821 0 0) 0x7f2abc158a30 con 0x7f2abc0c5950 2024-03-28T23:33:03.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.290+0000 7f2aba59c640 1 -- 172.21.15.161:0/1412827406 >> v2:172.21.15.170:3300/0 conn(0x7f2abc152890 msgr2=0x7f2abc0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:03.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.290+0000 7f2aba59c640 1 --2- 172.21.15.161:0/1412827406 >> v2:172.21.15.170:3300/0 conn(0x7f2abc152890 0x7f2abc0c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.290+0000 7f2aba59c640 1 -- 172.21.15.161:0/1412827406 >> v1:172.21.15.161:6789/0 conn(0x7f2abc0c5950 legacy=0x7f2abc1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:03.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.290+0000 7f2aba59c640 1 -- 172.21.15.161:0/1412827406 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2abc158d10 con 0x7f2abc0c8ed0 2024-03-28T23:33:03.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.290+0000 7f2abad9d640 1 --2- 172.21.15.161:0/1412827406 >> v2:172.21.15.170:3300/0 conn(0x7f2abc152890 0x7f2abc0c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:03.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.291+0000 7f2aba59c640 1 --2- 172.21.15.161:0/1412827406 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2abc0c8ed0 0x7f2abc0c5410 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2aac009790 tx=0x7f2aac02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=feaa8942618c8c15 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:03.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.291+0000 7f2ab9d9b640 1 -- 172.21.15.161:0/1412827406 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac039070 con 0x7f2abc0c8ed0 2024-03-28T23:33:03.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.291+0000 7f2ab9d9b640 1 -- 172.21.15.161:0/1412827406 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aac0316f0 con 0x7f2abc0c8ed0 2024-03-28T23:33:03.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.291+0000 7f2ab9d9b640 1 -- 172.21.15.161:0/1412827406 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac031ac0 con 0x7f2abc0c8ed0 2024-03-28T23:33:03.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.292+0000 7f2ac119a640 1 -- 172.21.15.161:0/1412827406 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2abc0c8ed0 msgr2=0x7f2abc0c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:03.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.292+0000 7f2ac119a640 1 --2- 172.21.15.161:0/1412827406 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2abc0c8ed0 0x7f2abc0c5410 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2aac009790 tx=0x7f2aac02b790 comp rx=0 tx=0).stop 2024-03-28T23:33:03.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.292+0000 7f2ac119a640 1 -- 172.21.15.161:0/1412827406 shutdown_connections 2024-03-28T23:33:03.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.292+0000 7f2ac119a640 1 --2- 172.21.15.161:0/1412827406 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2abc0c8ed0 0x7f2abc0c5410 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.292+0000 7f2ac119a640 1 --2- 172.21.15.161:0/1412827406 >> v2:172.21.15.170:3300/0 conn(0x7f2abc152890 0x7f2abc0c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.292+0000 7f2ac119a640 1 -- 172.21.15.161:0/1412827406 >> 172.21.15.161:0/1412827406 conn(0x7f2abc0ac620 msgr2=0x7f2abc0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:03.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.292+0000 7f2ac119a640 1 -- 172.21.15.161:0/1412827406 shutdown_connections 2024-03-28T23:33:03.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.292+0000 7f2ac119a640 1 -- 172.21.15.161:0/1412827406 wait complete. 2024-03-28T23:33:03.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.293+0000 7f2ac119a640 1 Processor -- start 2024-03-28T23:33:03.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.293+0000 7f2ac119a640 1 -- start start 2024-03-28T23:33:03.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.294+0000 7f2ac119a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2abc0c5950 0x7f2abc0b0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:03.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.294+0000 7f2ac119a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2abc0c8ed0 0x7f2abc14b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:03.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.294+0000 7f2ac119a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2abc152890 0x7f2abc156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:03.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.294+0000 7f2ac119a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc152c70 con 0x7f2abc0c5950 2024-03-28T23:33:03.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.294+0000 7f2ac119a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc1592d0 con 0x7f2abc0c8ed0 2024-03-28T23:33:03.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.294+0000 7f2ac119a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2abc152df0 con 0x7f2abc152890 2024-03-28T23:33:03.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.294+0000 7f2abad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2abc0c5950 0x7f2abc0b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:03.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.294+0000 7f2abad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2abc0c5950 0x7f2abc0b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46106/0 (socket says 172.21.15.161:46106) 2024-03-28T23:33:03.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.294+0000 7f2abad9d640 1 -- 172.21.15.161:0/1729566963 learned_addr learned my addr 172.21.15.161:0/1729566963 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:03.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.294+0000 7f2abb59e640 1 --2- 172.21.15.161:0/1729566963 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2abc152890 0x7f2abc156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:03.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.294+0000 7f2aba59c640 1 --2- 172.21.15.161:0/1729566963 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2abc0c8ed0 0x7f2abc14b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:03.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.294+0000 7f2abad9d640 1 -- 172.21.15.161:0/1729566963 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2abc152890 msgr2=0x7f2abc156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:03.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.294+0000 7f2abad9d640 1 --2- 172.21.15.161:0/1729566963 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2abc152890 0x7f2abc156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.294+0000 7f2abad9d640 1 -- 172.21.15.161:0/1729566963 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2abc0c8ed0 msgr2=0x7f2abc14b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:03.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.294+0000 7f2abad9d640 1 --2- 172.21.15.161:0/1729566963 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2abc0c8ed0 0x7f2abc14b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.294+0000 7f2abad9d640 1 -- 172.21.15.161:0/1729566963 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2abc153190 con 0x7f2abc0c5950 2024-03-28T23:33:03.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.294+0000 7f2abad9d640 1 --2- 172.21.15.161:0/1729566963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2abc0c5950 0x7f2abc0b0410 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f2ab000ebe0 tx=0x7f2ab000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:03.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.295+0000 7f2aa37fe640 1 -- 172.21.15.161:0/1729566963 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab0017070 con 0x7f2abc0c5950 2024-03-28T23:33:03.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.295+0000 7f2aa37fe640 1 -- 172.21.15.161:0/1729566963 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ab000cc30 con 0x7f2abc0c5950 2024-03-28T23:33:03.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.295+0000 7f2ac119a640 1 -- 172.21.15.161:0/1729566963 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2abc153470 con 0x7f2abc0c5950 2024-03-28T23:33:03.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.295+0000 7f2aa37fe640 1 -- 172.21.15.161:0/1729566963 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab0013470 con 0x7f2abc0c5950 2024-03-28T23:33:03.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.295+0000 7f2ac119a640 1 -- 172.21.15.161:0/1729566963 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2abc157430 con 0x7f2abc0c5950 2024-03-28T23:33:03.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.296+0000 7f2ac119a640 1 -- 172.21.15.161:0/1729566963 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2abc04f4b0 con 0x7f2abc0c5950 2024-03-28T23:33:03.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.296+0000 7f2aa37fe640 1 -- 172.21.15.161:0/1729566963 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2ab001a070 con 0x7f2abc0c5950 2024-03-28T23:33:03.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.297+0000 7f2aa37fe640 1 --2- 172.21.15.161:0/1729566963 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2aa806b4d0 0x7f2aa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:03.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.297+0000 7f2aba59c640 1 --2- 172.21.15.161:0/1729566963 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2aa806b4d0 0x7f2aa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:03.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.297+0000 7f2aa37fe640 1 -- 172.21.15.161:0/1729566963 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7043+0+0 (secure 0 0 0) 0x7f2ab008a900 con 0x7f2abc0c5950 2024-03-28T23:33:03.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.297+0000 7f2aba59c640 1 --2- 172.21.15.161:0/1729566963 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2aa806b4d0 0x7f2aa806d990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f2aac036000 tx=0x7f2aac008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:03.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.301+0000 7f2aa37fe640 1 -- 172.21.15.161:0/1729566963 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2ab0058c80 con 0x7f2abc0c5950 2024-03-28T23:33:03.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.520+0000 7f2ac119a640 1 -- 172.21.15.161:0/1729566963 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2abc156330 con 0x7f2abc0c5950 2024-03-28T23:33:03.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.520+0000 7f2aa37fe640 1 -- 172.21.15.161:0/1729566963 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6507 (secure 0 0 0) 0x7f2ab005cd20 con 0x7f2abc0c5950 2024-03-28T23:33:03.523 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:03.523 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":67,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.161:6837/737681942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":737681942},{"type":"v1","addr":"172.21.15.161:6837","nonce":737681942}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":60},{"gid":5579,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.170:6834/3501962065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3501962065},{"type":"v1","addr":"172.21.15.170:6834","nonce":3501962065}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":67,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:32:32.372964+0000","modified":"2024-03-28T23:32:58.528763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6807},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6807":{"gid":6807,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.161:6836/2378263687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":2378263687},{"type":"v1","addr":"172.21.15.161:6836","nonce":2378263687}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6807,"qdb_cluster":[6807]},"id":5},{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:32:44.463287+0000","modified":"2024-03-28T23:32:46.605012+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6037":{"gid":6037,"name":"b","rank":0,"incarnation":57,"state":"up:active","state_seq":6,"addr":"172.21.15.170:6835/2691084827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":2691084827},{"type":"v1","addr":"172.21.15.170:6835","nonce":2691084827}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6037,"qdb_cluster":[6037]},"id":6}]} 2024-03-28T23:33:03.524 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 67 2024-03-28T23:33:03.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.522+0000 7f2ac119a640 1 -- 172.21.15.161:0/1729566963 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2aa806b4d0 msgr2=0x7f2aa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:03.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.522+0000 7f2ac119a640 1 --2- 172.21.15.161:0/1729566963 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2aa806b4d0 0x7f2aa806d990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f2aac036000 tx=0x7f2aac008030 comp rx=0 tx=0).stop 2024-03-28T23:33:03.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.522+0000 7f2ac119a640 1 -- 172.21.15.161:0/1729566963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2abc0c5950 msgr2=0x7f2abc0b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:03.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.522+0000 7f2ac119a640 1 --2- 172.21.15.161:0/1729566963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2abc0c5950 0x7f2abc0b0410 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f2ab000ebe0 tx=0x7f2ab000c520 comp rx=0 tx=0).stop 2024-03-28T23:33:03.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.522+0000 7f2ac119a640 1 -- 172.21.15.161:0/1729566963 shutdown_connections 2024-03-28T23:33:03.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.522+0000 7f2ac119a640 1 --2- 172.21.15.161:0/1729566963 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2aa806b4d0 0x7f2aa806d990 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.522+0000 7f2ac119a640 1 --2- 172.21.15.161:0/1729566963 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2abc152890 0x7f2abc156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.522+0000 7f2ac119a640 1 --2- 172.21.15.161:0/1729566963 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2abc0c8ed0 0x7f2abc14b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.522+0000 7f2ac119a640 1 --2- 172.21.15.161:0/1729566963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2abc0c5950 0x7f2abc0b0410 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.522+0000 7f2ac119a640 1 -- 172.21.15.161:0/1729566963 >> 172.21.15.161:0/1729566963 conn(0x7f2abc0ac620 msgr2=0x7f2abc14c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:03.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.522+0000 7f2ac119a640 1 -- 172.21.15.161:0/1729566963 shutdown_connections 2024-03-28T23:33:03.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.522+0000 7f2ac119a640 1 -- 172.21.15.161:0/1729566963 wait complete. 2024-03-28T23:33:03.538 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:33:03.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.676+0000 7f6f2b4af640 1 Processor -- start 2024-03-28T23:33:03.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.676+0000 7f6f2b4af640 1 -- start start 2024-03-28T23:33:03.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.677+0000 7f6f2b4af640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6f240c5850 0x7f6f24155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:03.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.677+0000 7f6f2b4af640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f240c5d00 con 0x7f6f240c5410 2024-03-28T23:33:03.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.677+0000 7f6f2b4af640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f24158a30 con 0x7f6f24152890 2024-03-28T23:33:03.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.677+0000 7f6f2b4af640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f6f240c8480 con 0x7f6f240c5850 2024-03-28T23:33:03.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.677+0000 7f6f28a4a640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6f240c5410 0x7f6f241553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44880/0 (socket says 172.21.15.161:44880) 2024-03-28T23:33:03.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.677+0000 7f6f28a4a640 1 -- 172.21.15.161:0/2077208220 learned_addr learned my addr 172.21.15.161:0/2077208220 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:03.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.677+0000 7f6f29a4c640 1 --2- 172.21.15.161:0/2077208220 >> v2:172.21.15.170:3300/0 conn(0x7f6f240c5850 0x7f6f24155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:03.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.677+0000 7f6f17fff640 1 -- 172.21.15.161:0/2077208220 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1363301740 0 0) 0x7f6f240c5d00 con 0x7f6f240c5410 2024-03-28T23:33:03.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.677+0000 7f6f17fff640 1 -- 172.21.15.161:0/2077208220 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f10003520 con 0x7f6f240c5410 2024-03-28T23:33:03.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.677+0000 7f6f29a4c640 1 -- 172.21.15.161:0/2077208220 >> v1:172.21.15.170:6789/0 conn(0x7f6f24152890 legacy=0x7f6f240c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:03.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.677+0000 7f6f29a4c640 1 -- 172.21.15.161:0/2077208220 >> v1:172.21.15.161:6789/0 conn(0x7f6f240c5410 legacy=0x7f6f241553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:03.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.677+0000 7f6f17fff640 1 -- 172.21.15.161:0/2077208220 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2200818723 0 0) 0x7f6f24158a30 con 0x7f6f24152890 2024-03-28T23:33:03.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.677+0000 7f6f29a4c640 1 -- 172.21.15.161:0/2077208220 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f24158d10 con 0x7f6f240c5850 2024-03-28T23:33:03.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.678+0000 7f6f29a4c640 1 --2- 172.21.15.161:0/2077208220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f240c5850 0x7f6f24155ab0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c009d90 tx=0x7f6f1c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78bf74a9600c3999 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:03.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.678+0000 7f6f17fff640 1 -- 172.21.15.161:0/2077208220 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c039070 con 0x7f6f240c5850 2024-03-28T23:33:03.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.678+0000 7f6f17fff640 1 -- 172.21.15.161:0/2077208220 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f1c02cc60 con 0x7f6f240c5850 2024-03-28T23:33:03.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.678+0000 7f6f17fff640 1 -- 172.21.15.161:0/2077208220 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c034640 con 0x7f6f240c5850 2024-03-28T23:33:03.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.679+0000 7f6f2b4af640 1 -- 172.21.15.161:0/2077208220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f240c5850 msgr2=0x7f6f24155ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:03.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.679+0000 7f6f2b4af640 1 --2- 172.21.15.161:0/2077208220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f240c5850 0x7f6f24155ab0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c009d90 tx=0x7f6f1c02beb0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.679+0000 7f6f2b4af640 1 -- 172.21.15.161:0/2077208220 shutdown_connections 2024-03-28T23:33:03.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.679+0000 7f6f2b4af640 1 --2- 172.21.15.161:0/2077208220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f240c5850 0x7f6f24155ab0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.679+0000 7f6f2b4af640 1 -- 172.21.15.161:0/2077208220 >> 172.21.15.161:0/2077208220 conn(0x7f6f240ac620 msgr2=0x7f6f240aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:03.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.679+0000 7f6f2b4af640 1 -- 172.21.15.161:0/2077208220 shutdown_connections 2024-03-28T23:33:03.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.679+0000 7f6f2b4af640 1 -- 172.21.15.161:0/2077208220 wait complete. 2024-03-28T23:33:03.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.680+0000 7f6f2b4af640 1 Processor -- start 2024-03-28T23:33:03.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.680+0000 7f6f2b4af640 1 -- start start 2024-03-28T23:33:03.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.681+0000 7f6f2b4af640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f240c5410 0x7f6f24154340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:03.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.681+0000 7f6f2b4af640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f240c5850 0x7f6f240bfb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:03.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.681+0000 7f6f2b4af640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f24152890 0x7f6f240be5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:03.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.681+0000 7f6f2b4af640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6f241592d0 con 0x7f6f240c5850 2024-03-28T23:33:03.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.681+0000 7f6f2b4af640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6f24159640 con 0x7f6f240c5410 2024-03-28T23:33:03.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.681+0000 7f6f2b4af640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6f24155ff0 con 0x7f6f24152890 2024-03-28T23:33:03.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.681+0000 7f6f28a4a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f240c5850 0x7f6f240bfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:03.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.681+0000 7f6f28a4a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f240c5850 0x7f6f240bfb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46122/0 (socket says 172.21.15.161:46122) 2024-03-28T23:33:03.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.681+0000 7f6f28a4a640 1 -- 172.21.15.161:0/107447644 learned_addr learned my addr 172.21.15.161:0/107447644 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:03.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.681+0000 7f6f2924b640 1 --2- 172.21.15.161:0/107447644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f240c5410 0x7f6f24154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:03.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.681+0000 7f6f29a4c640 1 --2- 172.21.15.161:0/107447644 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f24152890 0x7f6f240be5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:03.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.681+0000 7f6f2924b640 1 -- 172.21.15.161:0/107447644 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f24152890 msgr2=0x7f6f240be5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:03.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.681+0000 7f6f2924b640 1 --2- 172.21.15.161:0/107447644 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f24152890 0x7f6f240be5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.681+0000 7f6f2924b640 1 -- 172.21.15.161:0/107447644 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f240c5850 msgr2=0x7f6f240bfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:03.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.681+0000 7f6f2924b640 1 --2- 172.21.15.161:0/107447644 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f240c5850 0x7f6f240bfb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.681+0000 7f6f2924b640 1 -- 172.21.15.161:0/107447644 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f24158690 con 0x7f6f240c5410 2024-03-28T23:33:03.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.681+0000 7f6f28a4a640 1 --2- 172.21.15.161:0/107447644 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f240c5850 0x7f6f240bfb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:03.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.681+0000 7f6f2924b640 1 --2- 172.21.15.161:0/107447644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f240c5410 0x7f6f24154340 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800c450 tx=0x7f6f1800c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:03.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.682+0000 7f6f15ffb640 1 -- 172.21.15.161:0/107447644 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f18017070 con 0x7f6f240c5410 2024-03-28T23:33:03.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.682+0000 7f6f15ffb640 1 -- 172.21.15.161:0/107447644 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f180035f0 con 0x7f6f240c5410 2024-03-28T23:33:03.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.682+0000 7f6f2b4af640 1 -- 172.21.15.161:0/107447644 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f24154920 con 0x7f6f240c5410 2024-03-28T23:33:03.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.682+0000 7f6f15ffb640 1 -- 172.21.15.161:0/107447644 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f180039e0 con 0x7f6f240c5410 2024-03-28T23:33:03.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.682+0000 7f6f2b4af640 1 -- 172.21.15.161:0/107447644 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f24153740 con 0x7f6f240c5410 2024-03-28T23:33:03.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.683+0000 7f6f2b4af640 1 -- 172.21.15.161:0/107447644 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f2404f4b0 con 0x7f6f240c5410 2024-03-28T23:33:03.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.684+0000 7f6f15ffb640 1 -- 172.21.15.161:0/107447644 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6f1801a070 con 0x7f6f240c5410 2024-03-28T23:33:03.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.684+0000 7f6f15ffb640 1 --2- 172.21.15.161:0/107447644 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6f0806b4d0 0x7f6f0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:03.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.684+0000 7f6f15ffb640 1 -- 172.21.15.161:0/107447644 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7043+0+0 (secure 0 0 0) 0x7f6f18089b60 con 0x7f6f240c5410 2024-03-28T23:33:03.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.684+0000 7f6f28a4a640 1 --2- 172.21.15.161:0/107447644 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6f0806b4d0 0x7f6f0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:03.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.684+0000 7f6f28a4a640 1 --2- 172.21.15.161:0/107447644 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6f0806b4d0 0x7f6f0806d990 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6f20007fd0 tx=0x7f6f20009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:03.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.688+0000 7f6f15ffb640 1 -- 172.21.15.161:0/107447644 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6f18057dc0 con 0x7f6f240c5410 2024-03-28T23:33:03.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.865+0000 7f6f2b4af640 1 -- 172.21.15.161:0/107447644 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6f240ac400 con 0x7f6f240c5410 2024-03-28T23:33:03.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.867+0000 7f6f15ffb640 1 -- 172.21.15.161:0/107447644 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+20836 (secure 0 0 0) 0x7f6f1805be60 con 0x7f6f240c5410 2024-03-28T23:33:03.868 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:03.868 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":78,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:32:54.518544+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:32:30.684936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-28T23:32:31.339722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:32:42.378609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:32:43.436985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6836/3850309502":"2024-03-29T23:32:54.518520+0000","172.21.15.161:6834/3850309502":"2024-03-29T23:32:54.518520+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:33:03.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.869+0000 7f6f2b4af640 1 -- 172.21.15.161:0/107447644 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6f0806b4d0 msgr2=0x7f6f0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:03.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.869+0000 7f6f2b4af640 1 --2- 172.21.15.161:0/107447644 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6f0806b4d0 0x7f6f0806d990 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6f20007fd0 tx=0x7f6f20009040 comp rx=0 tx=0).stop 2024-03-28T23:33:03.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.869+0000 7f6f2b4af640 1 -- 172.21.15.161:0/107447644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f240c5410 msgr2=0x7f6f24154340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:03.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.869+0000 7f6f2b4af640 1 --2- 172.21.15.161:0/107447644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f240c5410 0x7f6f24154340 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800c450 tx=0x7f6f1800c910 comp rx=0 tx=0).stop 2024-03-28T23:33:03.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.869+0000 7f6f2b4af640 1 -- 172.21.15.161:0/107447644 shutdown_connections 2024-03-28T23:33:03.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.869+0000 7f6f2b4af640 1 --2- 172.21.15.161:0/107447644 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6f0806b4d0 0x7f6f0806d990 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.869+0000 7f6f2b4af640 1 --2- 172.21.15.161:0/107447644 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f240c5850 0x7f6f240bfb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.869+0000 7f6f2b4af640 1 --2- 172.21.15.161:0/107447644 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f24152890 0x7f6f240be5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.869+0000 7f6f2b4af640 1 --2- 172.21.15.161:0/107447644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f240c5410 0x7f6f24154340 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.869+0000 7f6f2b4af640 1 -- 172.21.15.161:0/107447644 >> 172.21.15.161:0/107447644 conn(0x7f6f240ac620 msgr2=0x7f6f240c9f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:03.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.869+0000 7f6f2b4af640 1 -- 172.21.15.161:0/107447644 shutdown_connections 2024-03-28T23:33:03.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.869+0000 7f6f2b4af640 1 -- 172.21.15.161:0/107447644 wait complete. 2024-03-28T23:33:03.880 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-28T23:33:03.881 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:33:03.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.986+0000 7f9141956640 1 Processor -- start 2024-03-28T23:33:03.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.986+0000 7f9141956640 1 -- start start 2024-03-28T23:33:03.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.987+0000 7f9141956640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f913c0c5410 0x7f913c0c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:03.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.987+0000 7f9141956640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f913c1553a0 0x7f913c155850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:03.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.987+0000 7f9141956640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f913c158a30 con 0x7f913c152890 2024-03-28T23:33:03.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.987+0000 7f9141956640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f913c0c8480 con 0x7f913c0c5410 2024-03-28T23:33:03.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.987+0000 7f9141956640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f913c0c5d90 con 0x7f913c1553a0 2024-03-28T23:33:03.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.987+0000 7f9133fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f913c0c5410 0x7f913c0c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:03.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.987+0000 7f913affd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f913c152890 0x7f913c0c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44896/0 (socket says 172.21.15.161:44896) 2024-03-28T23:33:03.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.987+0000 7f913affd640 1 -- 172.21.15.161:0/2482572081 learned_addr learned my addr 172.21.15.161:0/2482572081 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:03.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.987+0000 7f913b7fe640 1 --2- 172.21.15.161:0/2482572081 >> v2:172.21.15.170:3300/0 conn(0x7f913c1553a0 0x7f913c155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:03.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.988+0000 7f9133fff640 1 -- 172.21.15.161:0/2482572081 >> v2:172.21.15.170:3300/0 conn(0x7f913c1553a0 msgr2=0x7f913c155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:03.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.988+0000 7f9133fff640 1 --2- 172.21.15.161:0/2482572081 >> v2:172.21.15.170:3300/0 conn(0x7f913c1553a0 0x7f913c155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.988+0000 7f9133fff640 1 -- 172.21.15.161:0/2482572081 >> v1:172.21.15.161:6789/0 conn(0x7f913c152890 legacy=0x7f913c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:03.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.988+0000 7f9133fff640 1 -- 172.21.15.161:0/2482572081 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f913c158d10 con 0x7f913c0c5410 2024-03-28T23:33:03.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.988+0000 7f913a7fc640 1 -- 172.21.15.161:0/2482572081 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2995046293 0 0) 0x7f913c158a30 con 0x7f913c152890 2024-03-28T23:33:03.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.988+0000 7f9133fff640 1 --2- 172.21.15.161:0/2482572081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f913c0c5410 0x7f913c0c5850 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f913c0aa310 tx=0x7f912402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dab09daa17bf4399 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:03.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.988+0000 7f913a7fc640 1 -- 172.21.15.161:0/2482572081 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91240315a0 con 0x7f913c0c5410 2024-03-28T23:33:03.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.989+0000 7f913a7fc640 1 -- 172.21.15.161:0/2482572081 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9124037070 con 0x7f913c0c5410 2024-03-28T23:33:03.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.989+0000 7f913a7fc640 1 -- 172.21.15.161:0/2482572081 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9124031960 con 0x7f913c0c5410 2024-03-28T23:33:03.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.989+0000 7f9141956640 1 -- 172.21.15.161:0/2482572081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f913c0c5410 msgr2=0x7f913c0c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:03.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.989+0000 7f9141956640 1 --2- 172.21.15.161:0/2482572081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f913c0c5410 0x7f913c0c5850 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f913c0aa310 tx=0x7f912402b840 comp rx=0 tx=0).stop 2024-03-28T23:33:03.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.989+0000 7f9141956640 1 -- 172.21.15.161:0/2482572081 shutdown_connections 2024-03-28T23:33:03.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.989+0000 7f9141956640 1 --2- 172.21.15.161:0/2482572081 >> v2:172.21.15.170:3300/0 conn(0x7f913c1553a0 0x7f913c155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.989+0000 7f9141956640 1 --2- 172.21.15.161:0/2482572081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f913c0c5410 0x7f913c0c5850 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.989+0000 7f9141956640 1 -- 172.21.15.161:0/2482572081 >> 172.21.15.161:0/2482572081 conn(0x7f913c0ac620 msgr2=0x7f913c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:03.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.989+0000 7f9141956640 1 -- 172.21.15.161:0/2482572081 shutdown_connections 2024-03-28T23:33:03.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.989+0000 7f9141956640 1 -- 172.21.15.161:0/2482572081 wait complete. 2024-03-28T23:33:03.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.990+0000 7f9141956640 1 Processor -- start 2024-03-28T23:33:03.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.990+0000 7f9141956640 1 -- start start 2024-03-28T23:33:03.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.991+0000 7f9141956640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f913c0c5410 0x7f913c1482a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:03.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.991+0000 7f9141956640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f913c152890 0x7f913c14d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:03.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.991+0000 7f9141956640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f913c1553a0 0x7f913c157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.991+0000 7f9141956640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f913c1592d0 con 0x7f913c0c5410 2024-03-28T23:33:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.991+0000 7f9141956640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f913c159640 con 0x7f913c1553a0 2024-03-28T23:33:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.991+0000 7f9141956640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f913c0c90a0 con 0x7f913c152890 2024-03-28T23:33:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.991+0000 7f913affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f913c0c5410 0x7f913c1482a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.991+0000 7f913affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f913c0c5410 0x7f913c1482a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46146/0 (socket says 172.21.15.161:46146) 2024-03-28T23:33:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.991+0000 7f913affd640 1 -- 172.21.15.161:0/3891688505 learned_addr learned my addr 172.21.15.161:0/3891688505 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.991+0000 7f913b7fe640 1 --2- 172.21.15.161:0/3891688505 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f913c1553a0 0x7f913c157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.991+0000 7f9133fff640 1 --2- 172.21.15.161:0/3891688505 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f913c152890 0x7f913c14d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.991+0000 7f913affd640 1 -- 172.21.15.161:0/3891688505 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f913c152890 msgr2=0x7f913c14d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.991+0000 7f913affd640 1 --2- 172.21.15.161:0/3891688505 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f913c152890 0x7f913c14d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.991+0000 7f913affd640 1 -- 172.21.15.161:0/3891688505 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f913c1553a0 msgr2=0x7f913c157d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.991+0000 7f913affd640 1 --2- 172.21.15.161:0/3891688505 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f913c1553a0 0x7f913c157d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.991+0000 7f913affd640 1 -- 172.21.15.161:0/3891688505 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f913c149ce0 con 0x7f913c0c5410 2024-03-28T23:33:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.992+0000 7f913affd640 1 --2- 172.21.15.161:0/3891688505 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f913c0c5410 0x7f913c1482a0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f912c00c4d0 tx=0x7f912c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:03.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.992+0000 7f9138ff9640 1 -- 172.21.15.161:0/3891688505 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c018070 con 0x7f913c0c5410 2024-03-28T23:33:03.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.992+0000 7f9138ff9640 1 -- 172.21.15.161:0/3891688505 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f912c007500 con 0x7f913c0c5410 2024-03-28T23:33:03.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.992+0000 7f9138ff9640 1 -- 172.21.15.161:0/3891688505 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c004a20 con 0x7f913c0c5410 2024-03-28T23:33:03.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.992+0000 7f9141956640 1 -- 172.21.15.161:0/3891688505 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f913c14e870 con 0x7f913c0c5410 2024-03-28T23:33:03.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.992+0000 7f9141956640 1 -- 172.21.15.161:0/3891688505 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f913c148c90 con 0x7f913c0c5410 2024-03-28T23:33:03.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.993+0000 7f9141956640 1 -- 172.21.15.161:0/3891688505 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f913c04f4b0 con 0x7f913c0c5410 2024-03-28T23:33:03.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.993+0000 7f9138ff9640 1 -- 172.21.15.161:0/3891688505 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f912c01b070 con 0x7f913c0c5410 2024-03-28T23:33:03.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.994+0000 7f9138ff9640 1 --2- 172.21.15.161:0/3891688505 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f911c06b4d0 0x7f911c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:03.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.994+0000 7f9138ff9640 1 -- 172.21.15.161:0/3891688505 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7043+0+0 (secure 0 0 0) 0x7f912c089940 con 0x7f913c0c5410 2024-03-28T23:33:03.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.994+0000 7f9133fff640 1 --2- 172.21.15.161:0/3891688505 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f911c06b4d0 0x7f911c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:03.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.994+0000 7f9133fff640 1 --2- 172.21.15.161:0/3891688505 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f911c06b4d0 0x7f911c06d990 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9124036000 tx=0x7f9124008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:03.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:03.998+0000 7f9138ff9640 1 -- 172.21.15.161:0/3891688505 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f912c057cc0 con 0x7f913c0c5410 2024-03-28T23:33:04.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.214+0000 7f9141956640 1 -- 172.21.15.161:0/3891688505 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f913c0c18b0 con 0x7f913c0c5410 2024-03-28T23:33:04.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.215+0000 7f9138ff9640 1 -- 172.21.15.161:0/3891688505 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6507 (secure 0 0 0) 0x7f912c05bd60 con 0x7f913c0c5410 2024-03-28T23:33:04.215 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:04.216 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":67,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.161:6837/737681942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":737681942},{"type":"v1","addr":"172.21.15.161:6837","nonce":737681942}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":60},{"gid":5579,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.170:6834/3501962065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3501962065},{"type":"v1","addr":"172.21.15.170:6834","nonce":3501962065}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":67,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:32:32.372964+0000","modified":"2024-03-28T23:32:58.528763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6807},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6807":{"gid":6807,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.161:6836/2378263687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":2378263687},{"type":"v1","addr":"172.21.15.161:6836","nonce":2378263687}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6807,"qdb_cluster":[6807]},"id":5},{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:32:44.463287+0000","modified":"2024-03-28T23:32:46.605012+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6037":{"gid":6037,"name":"b","rank":0,"incarnation":57,"state":"up:active","state_seq":6,"addr":"172.21.15.170:6835/2691084827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":2691084827},{"type":"v1","addr":"172.21.15.170:6835","nonce":2691084827}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6037,"qdb_cluster":[6037]},"id":6}]} 2024-03-28T23:33:04.216 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 67 2024-03-28T23:33:04.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.217+0000 7f9141956640 1 -- 172.21.15.161:0/3891688505 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f911c06b4d0 msgr2=0x7f911c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:04.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.217+0000 7f9141956640 1 --2- 172.21.15.161:0/3891688505 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f911c06b4d0 0x7f911c06d990 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9124036000 tx=0x7f9124008030 comp rx=0 tx=0).stop 2024-03-28T23:33:04.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.217+0000 7f9141956640 1 -- 172.21.15.161:0/3891688505 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f913c0c5410 msgr2=0x7f913c1482a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:04.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.217+0000 7f9141956640 1 --2- 172.21.15.161:0/3891688505 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f913c0c5410 0x7f913c1482a0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f912c00c4d0 tx=0x7f912c00c990 comp rx=0 tx=0).stop 2024-03-28T23:33:04.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.217+0000 7f9141956640 1 -- 172.21.15.161:0/3891688505 shutdown_connections 2024-03-28T23:33:04.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.217+0000 7f9141956640 1 --2- 172.21.15.161:0/3891688505 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f911c06b4d0 0x7f911c06d990 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:04.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.217+0000 7f9141956640 1 --2- 172.21.15.161:0/3891688505 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f913c1553a0 0x7f913c157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:04.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.217+0000 7f9141956640 1 --2- 172.21.15.161:0/3891688505 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f913c152890 0x7f913c14d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:04.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.217+0000 7f9141956640 1 --2- 172.21.15.161:0/3891688505 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f913c0c5410 0x7f913c1482a0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:04.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.217+0000 7f9141956640 1 -- 172.21.15.161:0/3891688505 >> 172.21.15.161:0/3891688505 conn(0x7f913c0ac620 msgr2=0x7f913c06c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:04.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.217+0000 7f9141956640 1 -- 172.21.15.161:0/3891688505 shutdown_connections 2024-03-28T23:33:04.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.217+0000 7f9141956640 1 -- 172.21.15.161:0/3891688505 wait complete. 2024-03-28T23:33:04.228 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:33:04.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.330+0000 7fd13bb7a640 1 Processor -- start 2024-03-28T23:33:04.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.330+0000 7fd13bb7a640 1 -- start start 2024-03-28T23:33:04.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.331+0000 7fd13bb7a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd13414eca0 0x7fd1340c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:04.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.331+0000 7fd13bb7a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd1340a98c0 0x7fd1340a9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:04.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.331+0000 7fd13bb7a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd134151850 con 0x7fd1340c1dc0 2024-03-28T23:33:04.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.331+0000 7fd13bb7a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd134152460 con 0x7fd1340a98c0 2024-03-28T23:33:04.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.331+0000 7fd13bb7a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd134151b30 con 0x7fd13414eca0 2024-03-28T23:33:04.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.331+0000 7fd139916640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd13414eca0 0x7fd1340c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:04.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.331+0000 7fd13a117640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd1340a98c0 0x7fd1340a9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:04.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.331+0000 7fd13a117640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd1340a98c0 0x7fd1340a9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:49582/0 (socket says 172.21.15.161:49582) 2024-03-28T23:33:04.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.331+0000 7fd13a117640 1 -- 172.21.15.161:0/2482121082 learned_addr learned my addr 172.21.15.161:0/2482121082 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:04.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.331+0000 7fd139916640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd13414eca0 0x7fd1340c1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:34454/0 (socket says 172.21.15.161:34454) 2024-03-28T23:33:04.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.331+0000 7fd139115640 1 --1- 172.21.15.161:0/2482121082 >> v1:172.21.15.170:6789/0 conn(0x7fd1340c1dc0 0x7fd13406ca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:44952/0 (socket says 172.21.15.161:44952) 2024-03-28T23:33:04.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.331+0000 7fd13a117640 1 -- 172.21.15.161:0/2482121082 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd13414eca0 msgr2=0x7fd1340c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:04.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.331+0000 7fd13a117640 1 --2- 172.21.15.161:0/2482121082 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd13414eca0 0x7fd1340c1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:04.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.331+0000 7fd13a117640 1 -- 172.21.15.161:0/2482121082 >> v1:172.21.15.170:6789/0 conn(0x7fd1340c1dc0 legacy=0x7fd13406ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:04.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.331+0000 7fd13a117640 1 -- 172.21.15.161:0/2482121082 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd134151e10 con 0x7fd1340a98c0 2024-03-28T23:33:04.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.332+0000 7fd13a117640 1 --2- 172.21.15.161:0/2482121082 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1340a98c0 0x7fd1340a9d20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd12c00d150 tx=0x7fd12c02e0e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70009decb0ff66fe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:04.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.332+0000 7fd138914640 1 -- 172.21.15.161:0/2482121082 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd12c02ea80 con 0x7fd1340a98c0 2024-03-28T23:33:04.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.332+0000 7fd138914640 1 -- 172.21.15.161:0/2482121082 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd12c02ec20 con 0x7fd1340a98c0 2024-03-28T23:33:04.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.332+0000 7fd138914640 1 -- 172.21.15.161:0/2482121082 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd12c009600 con 0x7fd1340a98c0 2024-03-28T23:33:04.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.333+0000 7fd13bb7a640 1 -- 172.21.15.161:0/2482121082 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1340a98c0 msgr2=0x7fd1340a9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:04.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.333+0000 7fd13bb7a640 1 --2- 172.21.15.161:0/2482121082 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1340a98c0 0x7fd1340a9d20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd12c00d150 tx=0x7fd12c02e0e0 comp rx=0 tx=0).stop 2024-03-28T23:33:04.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.333+0000 7fd13bb7a640 1 -- 172.21.15.161:0/2482121082 shutdown_connections 2024-03-28T23:33:04.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.333+0000 7fd13bb7a640 1 --2- 172.21.15.161:0/2482121082 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1340a98c0 0x7fd1340a9d20 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:04.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.333+0000 7fd13bb7a640 1 --2- 172.21.15.161:0/2482121082 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd13414eca0 0x7fd1340c1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:04.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.333+0000 7fd13bb7a640 1 -- 172.21.15.161:0/2482121082 >> 172.21.15.161:0/2482121082 conn(0x7fd1340c6670 msgr2=0x7fd1340c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:04.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.333+0000 7fd13bb7a640 1 -- 172.21.15.161:0/2482121082 shutdown_connections 2024-03-28T23:33:04.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.333+0000 7fd13bb7a640 1 -- 172.21.15.161:0/2482121082 wait complete. 2024-03-28T23:33:04.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.335+0000 7fd13bb7a640 1 Processor -- start 2024-03-28T23:33:04.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.335+0000 7fd13bb7a640 1 -- start start 2024-03-28T23:33:04.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.336+0000 7fd13bb7a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1340a98c0 0x7fd1340c01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:04.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.336+0000 7fd13bb7a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1340c1dc0 0x7fd1341543e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:04.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.336+0000 7fd13bb7a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd13414eca0 0x7fd134154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:04.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.336+0000 7fd13bb7a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd1341514b0 con 0x7fd1340a98c0 2024-03-28T23:33:04.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.336+0000 7fd13bb7a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd13414f080 con 0x7fd1340c1dc0 2024-03-28T23:33:04.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.336+0000 7fd13bb7a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd134151630 con 0x7fd13414eca0 2024-03-28T23:33:04.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.336+0000 7fd139916640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1340a98c0 0x7fd1340c01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:04.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.336+0000 7fd139916640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1340a98c0 0x7fd1340c01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46158/0 (socket says 172.21.15.161:46158) 2024-03-28T23:33:04.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.336+0000 7fd139916640 1 -- 172.21.15.161:0/1293767064 learned_addr learned my addr 172.21.15.161:0/1293767064 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:04.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.336+0000 7fd139115640 1 --2- 172.21.15.161:0/1293767064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1340c1dc0 0x7fd1341543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:04.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.336+0000 7fd13a117640 1 --2- 172.21.15.161:0/1293767064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd13414eca0 0x7fd134154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:04.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.336+0000 7fd139115640 1 -- 172.21.15.161:0/1293767064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd13414eca0 msgr2=0x7fd134154920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:04.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.336+0000 7fd139115640 1 --2- 172.21.15.161:0/1293767064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd13414eca0 0x7fd134154920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:04.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.336+0000 7fd139115640 1 -- 172.21.15.161:0/1293767064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1340a98c0 msgr2=0x7fd1340c01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:04.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.336+0000 7fd139115640 1 --2- 172.21.15.161:0/1293767064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1340a98c0 0x7fd1340c01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:04.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.336+0000 7fd139115640 1 -- 172.21.15.161:0/1293767064 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1341e74b0 con 0x7fd1340c1dc0 2024-03-28T23:33:04.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.336+0000 7fd139916640 1 --2- 172.21.15.161:0/1293767064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1340a98c0 0x7fd1340c01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:04.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.337+0000 7fd139115640 1 --2- 172.21.15.161:0/1293767064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1340c1dc0 0x7fd1341543e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd130000f60 tx=0x7fd13000c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:04.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.337+0000 7fd12a7fc640 1 -- 172.21.15.161:0/1293767064 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1300074e0 con 0x7fd1340c1dc0 2024-03-28T23:33:04.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.337+0000 7fd12a7fc640 1 -- 172.21.15.161:0/1293767064 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1300047e0 con 0x7fd1340c1dc0 2024-03-28T23:33:04.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.337+0000 7fd13bb7a640 1 -- 172.21.15.161:0/1293767064 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd134154e60 con 0x7fd1340c1dc0 2024-03-28T23:33:04.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.337+0000 7fd12a7fc640 1 -- 172.21.15.161:0/1293767064 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd130004bd0 con 0x7fd1340c1dc0 2024-03-28T23:33:04.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.337+0000 7fd13bb7a640 1 -- 172.21.15.161:0/1293767064 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1341e7140 con 0x7fd1340c1dc0 2024-03-28T23:33:04.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.338+0000 7fd13bb7a640 1 -- 172.21.15.161:0/1293767064 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1340731b0 con 0x7fd1340c1dc0 2024-03-28T23:33:04.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.338+0000 7fd12a7fc640 1 -- 172.21.15.161:0/1293767064 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd13001a020 con 0x7fd1340c1dc0 2024-03-28T23:33:04.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.339+0000 7fd12a7fc640 1 --2- 172.21.15.161:0/1293767064 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd11806b4d0 0x7fd11806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:04.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.339+0000 7fd139916640 1 --2- 172.21.15.161:0/1293767064 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd11806b4d0 0x7fd11806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:04.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.339+0000 7fd12a7fc640 1 -- 172.21.15.161:0/1293767064 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7043+0+0 (secure 0 0 0) 0x7fd130089d60 con 0x7fd1340c1dc0 2024-03-28T23:33:04.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.339+0000 7fd139916640 1 --2- 172.21.15.161:0/1293767064 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd11806b4d0 0x7fd11806d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd124009f50 tx=0x7fd124009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:04.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.343+0000 7fd12a7fc640 1 -- 172.21.15.161:0/1293767064 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd1340731b0 con 0x7fd1340c1dc0 2024-03-28T23:33:04.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.559+0000 7fd13bb7a640 1 -- 172.21.15.161:0/1293767064 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd13404f4b0 con 0x7fd1340c1dc0 2024-03-28T23:33:04.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.560+0000 7fd12a7fc640 1 -- 172.21.15.161:0/1293767064 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6507 (secure 0 0 0) 0x7fd130058040 con 0x7fd1340c1dc0 2024-03-28T23:33:04.561 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:04.561 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":67,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.161:6837/737681942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":737681942},{"type":"v1","addr":"172.21.15.161:6837","nonce":737681942}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":60},{"gid":5579,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.170:6834/3501962065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3501962065},{"type":"v1","addr":"172.21.15.170:6834","nonce":3501962065}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":67,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:32:32.372964+0000","modified":"2024-03-28T23:32:58.528763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6807},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6807":{"gid":6807,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":4,"addr":"172.21.15.161:6836/2378263687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":2378263687},{"type":"v1","addr":"172.21.15.161:6836","nonce":2378263687}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6807,"qdb_cluster":[6807]},"id":5},{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:32:44.463287+0000","modified":"2024-03-28T23:32:46.605012+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6037":{"gid":6037,"name":"b","rank":0,"incarnation":57,"state":"up:active","state_seq":6,"addr":"172.21.15.170:6835/2691084827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":2691084827},{"type":"v1","addr":"172.21.15.170:6835","nonce":2691084827}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6037,"qdb_cluster":[6037]},"id":6}]} 2024-03-28T23:33:04.561 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 67 2024-03-28T23:33:04.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.562+0000 7fd13bb7a640 1 -- 172.21.15.161:0/1293767064 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd11806b4d0 msgr2=0x7fd11806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:04.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.562+0000 7fd13bb7a640 1 --2- 172.21.15.161:0/1293767064 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd11806b4d0 0x7fd11806d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd124009f50 tx=0x7fd124009190 comp rx=0 tx=0).stop 2024-03-28T23:33:04.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.562+0000 7fd13bb7a640 1 -- 172.21.15.161:0/1293767064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1340c1dc0 msgr2=0x7fd1341543e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:04.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.562+0000 7fd13bb7a640 1 --2- 172.21.15.161:0/1293767064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1340c1dc0 0x7fd1341543e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd130000f60 tx=0x7fd13000c800 comp rx=0 tx=0).stop 2024-03-28T23:33:04.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.562+0000 7fd13bb7a640 1 -- 172.21.15.161:0/1293767064 shutdown_connections 2024-03-28T23:33:04.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.562+0000 7fd13bb7a640 1 --2- 172.21.15.161:0/1293767064 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd11806b4d0 0x7fd11806d990 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:04.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.562+0000 7fd13bb7a640 1 --2- 172.21.15.161:0/1293767064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd13414eca0 0x7fd134154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:04.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.562+0000 7fd13bb7a640 1 --2- 172.21.15.161:0/1293767064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1340c1dc0 0x7fd1341543e0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:04.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.562+0000 7fd13bb7a640 1 --2- 172.21.15.161:0/1293767064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1340a98c0 0x7fd1340c01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:04.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.562+0000 7fd13bb7a640 1 -- 172.21.15.161:0/1293767064 >> 172.21.15.161:0/1293767064 conn(0x7fd1340c6670 msgr2=0x7fd13414cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:04.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.562+0000 7fd13bb7a640 1 -- 172.21.15.161:0/1293767064 shutdown_connections 2024-03-28T23:33:04.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.562+0000 7fd13bb7a640 1 -- 172.21.15.161:0/1293767064 wait complete. 2024-03-28T23:33:04.577 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:33:04.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.678+0000 7f3092735640 1 Processor -- start 2024-03-28T23:33:04.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.678+0000 7f3092735640 1 -- start start 2024-03-28T23:33:04.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.679+0000 7f3092735640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f308c14eca0 0x7f308c0c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:04.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.679+0000 7f3092735640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f308c0c1dc0 0x7f308c06ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:04.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.679+0000 7f3092735640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f308c151850 con 0x7f308c06cf60 2024-03-28T23:33:04.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.679+0000 7f3092735640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f308c152460 con 0x7f308c14eca0 2024-03-28T23:33:04.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.679+0000 7f3092735640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f308c151b30 con 0x7f308c0c1dc0 2024-03-28T23:33:04.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.679+0000 7f308bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f308c14eca0 0x7f308c0c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:04.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.679+0000 7f3090cd2640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f308c06cf60 0x7f308c0a98c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44904/0 (socket says 172.21.15.161:44904) 2024-03-28T23:33:04.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.679+0000 7f3090cd2640 1 -- 172.21.15.161:0/560004316 learned_addr learned my addr 172.21.15.161:0/560004316 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:04.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.679+0000 7f308b7fe640 1 --2- 172.21.15.161:0/560004316 >> v2:172.21.15.170:3300/0 conn(0x7f308c0c1dc0 0x7f308c06ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:04.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.680+0000 7f308affd640 1 -- 172.21.15.161:0/560004316 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 729052988 0 0) 0x7f308c151850 con 0x7f308c06cf60 2024-03-28T23:33:04.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.680+0000 7f308affd640 1 -- 172.21.15.161:0/560004316 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3074004630 con 0x7f308c06cf60 2024-03-28T23:33:04.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.680+0000 7f308b7fe640 1 -- 172.21.15.161:0/560004316 >> v2:172.21.15.161:3300/0 conn(0x7f308c14eca0 msgr2=0x7f308c0c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:04.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.680+0000 7f308b7fe640 1 --2- 172.21.15.161:0/560004316 >> v2:172.21.15.161:3300/0 conn(0x7f308c14eca0 0x7f308c0c1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:04.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.680+0000 7f308b7fe640 1 -- 172.21.15.161:0/560004316 >> v1:172.21.15.161:6789/0 conn(0x7f308c06cf60 legacy=0x7f308c0a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:04.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.680+0000 7f308b7fe640 1 -- 172.21.15.161:0/560004316 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f308c151e10 con 0x7f308c0c1dc0 2024-03-28T23:33:04.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.680+0000 7f308bfff640 1 --2- 172.21.15.161:0/560004316 >> v2:172.21.15.161:3300/0 conn(0x7f308c14eca0 0x7f308c0c1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:04.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.680+0000 7f308b7fe640 1 --2- 172.21.15.161:0/560004316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f308c0c1dc0 0x7f308c06ca20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f308c0c3070 tx=0x7f307c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c9312c3051e9aec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:04.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.681+0000 7f308affd640 1 -- 172.21.15.161:0/560004316 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f307c0315a0 con 0x7f308c0c1dc0 2024-03-28T23:33:04.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.681+0000 7f308affd640 1 -- 172.21.15.161:0/560004316 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f307c037070 con 0x7f308c0c1dc0 2024-03-28T23:33:04.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.681+0000 7f308affd640 1 -- 172.21.15.161:0/560004316 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f307c031960 con 0x7f308c0c1dc0 2024-03-28T23:33:04.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.681+0000 7f3092735640 1 -- 172.21.15.161:0/560004316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f308c0c1dc0 msgr2=0x7f308c06ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:04.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.681+0000 7f3092735640 1 --2- 172.21.15.161:0/560004316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f308c0c1dc0 0x7f308c06ca20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f308c0c3070 tx=0x7f307c02b840 comp rx=0 tx=0).stop 2024-03-28T23:33:04.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.681+0000 7f3092735640 1 -- 172.21.15.161:0/560004316 shutdown_connections 2024-03-28T23:33:04.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.681+0000 7f3092735640 1 --2- 172.21.15.161:0/560004316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f308c0c1dc0 0x7f308c06ca20 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:04.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.681+0000 7f3092735640 1 --2- 172.21.15.161:0/560004316 >> v2:172.21.15.161:3300/0 conn(0x7f308c14eca0 0x7f308c0c1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:04.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.681+0000 7f3092735640 1 -- 172.21.15.161:0/560004316 >> 172.21.15.161:0/560004316 conn(0x7f308c0c6670 msgr2=0x7f308c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:04.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.681+0000 7f3092735640 1 -- 172.21.15.161:0/560004316 shutdown_connections 2024-03-28T23:33:04.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.681+0000 7f3092735640 1 -- 172.21.15.161:0/560004316 wait complete. 2024-03-28T23:33:04.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.683+0000 7f3092735640 1 Processor -- start 2024-03-28T23:33:04.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.683+0000 7f3092735640 1 -- start start 2024-03-28T23:33:04.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.684+0000 7f3092735640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f308c06cf60 0x7f308c145420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:04.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.684+0000 7f3092735640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f308c0c1dc0 0x7f308c14d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:04.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.684+0000 7f3092735640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f308c14eca0 0x7f308c1ebee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:04.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.684+0000 7f3092735640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f308c1514b0 con 0x7f308c14eca0 2024-03-28T23:33:04.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.684+0000 7f3092735640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f308c14f080 con 0x7f308c0c1dc0 2024-03-28T23:33:04.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.684+0000 7f3092735640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f308c151630 con 0x7f308c06cf60 2024-03-28T23:33:04.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.684+0000 7f3090cd2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f308c14eca0 0x7f308c1ebee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:04.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.684+0000 7f308bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f308c06cf60 0x7f308c145420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:04.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.684+0000 7f308bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f308c06cf60 0x7f308c145420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:34462/0 (socket says 172.21.15.161:34462) 2024-03-28T23:33:04.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.684+0000 7f308bfff640 1 -- 172.21.15.161:0/93017603 learned_addr learned my addr 172.21.15.161:0/93017603 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:04.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.684+0000 7f308b7fe640 1 --2- 172.21.15.161:0/93017603 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f308c0c1dc0 0x7f308c14d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:04.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.684+0000 7f308b7fe640 1 -- 172.21.15.161:0/93017603 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f308c06cf60 msgr2=0x7f308c145420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:04.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.684+0000 7f308b7fe640 1 --2- 172.21.15.161:0/93017603 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f308c06cf60 0x7f308c145420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:04.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.684+0000 7f308b7fe640 1 -- 172.21.15.161:0/93017603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f308c14eca0 msgr2=0x7f308c1ebee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:04.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.684+0000 7f308b7fe640 1 --2- 172.21.15.161:0/93017603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f308c14eca0 0x7f308c1ebee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:04.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.684+0000 7f308b7fe640 1 -- 172.21.15.161:0/93017603 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f308c155070 con 0x7f308c0c1dc0 2024-03-28T23:33:04.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.684+0000 7f3090cd2640 1 --2- 172.21.15.161:0/93017603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f308c14eca0 0x7f308c1ebee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:04.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.685+0000 7f308b7fe640 1 --2- 172.21.15.161:0/93017603 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f308c0c1dc0 0x7f308c14d160 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f308c1525e0 tx=0x7f307c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:04.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.685+0000 7f3088ff9640 1 -- 172.21.15.161:0/93017603 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f307c032630 con 0x7f308c0c1dc0 2024-03-28T23:33:04.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.685+0000 7f3088ff9640 1 -- 172.21.15.161:0/93017603 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f307c0327d0 con 0x7f308c0c1dc0 2024-03-28T23:33:04.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.685+0000 7f3088ff9640 1 -- 172.21.15.161:0/93017603 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f307c032ba0 con 0x7f308c0c1dc0 2024-03-28T23:33:04.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.685+0000 7f3092735640 1 -- 172.21.15.161:0/93017603 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f308c154200 con 0x7f308c0c1dc0 2024-03-28T23:33:04.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.685+0000 7f3092735640 1 -- 172.21.15.161:0/93017603 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f308c0c0720 con 0x7f308c0c1dc0 2024-03-28T23:33:04.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.686+0000 7f3092735640 1 -- 172.21.15.161:0/93017603 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f308c0c2820 con 0x7f308c0c1dc0 2024-03-28T23:33:04.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.686+0000 7f3088ff9640 1 -- 172.21.15.161:0/93017603 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f307c04e430 con 0x7f308c0c1dc0 2024-03-28T23:33:04.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.686+0000 7f3088ff9640 1 --2- 172.21.15.161:0/93017603 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f307806b4d0 0x7f307806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:04.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.686+0000 7f3088ff9640 1 -- 172.21.15.161:0/93017603 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7043+0+0 (secure 0 0 0) 0x7f307c02f030 con 0x7f308c0c1dc0 2024-03-28T23:33:04.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.686+0000 7f308bfff640 1 --2- 172.21.15.161:0/93017603 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f307806b4d0 0x7f307806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:04.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.687+0000 7f308bfff640 1 --2- 172.21.15.161:0/93017603 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f307806b4d0 0x7f307806d990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f30800095e0 tx=0x7f3080009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:04.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.691+0000 7f3088ff9640 1 -- 172.21.15.161:0/93017603 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f308c0c2820 con 0x7f308c0c1dc0 2024-03-28T23:33:04.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.950+0000 7f3092735640 1 -- 172.21.15.161:0/93017603 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f308c04f4b0 con 0x7f308c0c1dc0 2024-03-28T23:33:04.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.952+0000 7f3088ff9640 1 -- 172.21.15.161:0/93017603 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+20836 (secure 0 0 0) 0x7f307c040070 con 0x7f308c0c1dc0 2024-03-28T23:33:04.953 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:04.953 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":78,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:32:54.518544+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:32:30.684936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-28T23:32:31.339722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:32:42.378609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:32:43.436985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6836/3850309502":"2024-03-29T23:32:54.518520+0000","172.21.15.161:6834/3850309502":"2024-03-29T23:32:54.518520+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:33:04.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.954+0000 7f3092735640 1 -- 172.21.15.161:0/93017603 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f307806b4d0 msgr2=0x7f307806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:04.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.954+0000 7f3092735640 1 --2- 172.21.15.161:0/93017603 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f307806b4d0 0x7f307806d990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f30800095e0 tx=0x7f3080009190 comp rx=0 tx=0).stop 2024-03-28T23:33:04.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.954+0000 7f3092735640 1 -- 172.21.15.161:0/93017603 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f308c0c1dc0 msgr2=0x7f308c14d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:04.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.954+0000 7f3092735640 1 --2- 172.21.15.161:0/93017603 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f308c0c1dc0 0x7f308c14d160 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f308c1525e0 tx=0x7f307c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:04.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.954+0000 7f3092735640 1 -- 172.21.15.161:0/93017603 shutdown_connections 2024-03-28T23:33:04.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.954+0000 7f3092735640 1 --2- 172.21.15.161:0/93017603 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f307806b4d0 0x7f307806d990 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:04.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.954+0000 7f3092735640 1 --2- 172.21.15.161:0/93017603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f308c14eca0 0x7f308c1ebee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:04.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.954+0000 7f3092735640 1 --2- 172.21.15.161:0/93017603 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f308c0c1dc0 0x7f308c14d160 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:04.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.954+0000 7f3092735640 1 --2- 172.21.15.161:0/93017603 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f308c06cf60 0x7f308c145420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:04.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.954+0000 7f3092735640 1 -- 172.21.15.161:0/93017603 >> 172.21.15.161:0/93017603 conn(0x7f308c0c6670 msgr2=0x7f308c0aa8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:04.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.954+0000 7f3092735640 1 -- 172.21.15.161:0/93017603 shutdown_connections 2024-03-28T23:33:04.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:04.954+0000 7f3092735640 1 -- 172.21.15.161:0/93017603 wait complete. 2024-03-28T23:33:04.967 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-28T23:33:05.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.073+0000 7f5ca3bdd640 1 Processor -- start 2024-03-28T23:33:05.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.073+0000 7f5ca3bdd640 1 -- start start 2024-03-28T23:33:05.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.074+0000 7f5ca3bdd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5c9c06c3d0 0x7f5c9c14e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:05.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.074+0000 7f5ca3bdd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c9c14ee30 0x7f5c9c151710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:05.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.074+0000 7f5ca3bdd640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c9c155c00 con 0x7f5c9c151c50 2024-03-28T23:33:05.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.074+0000 7f5ca3bdd640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f5c9c06d3f0 con 0x7f5c9c06c3d0 2024-03-28T23:33:05.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.074+0000 7f5ca3bdd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5c9c155920 con 0x7f5c9c14ee30 2024-03-28T23:33:05.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.074+0000 7f5ca217a640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f5c9c151c50 0x7f5c9c070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44906/0 (socket says 172.21.15.161:44906) 2024-03-28T23:33:05.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.075+0000 7f5ca217a640 1 -- 172.21.15.161:0/3506759103 learned_addr learned my addr 172.21.15.161:0/3506759103 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:05.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.075+0000 7f5ca1979640 1 --2- 172.21.15.161:0/3506759103 >> v2:172.21.15.170:3300/0 conn(0x7f5c9c06c3d0 0x7f5c9c14e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:05.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.075+0000 7f5ca1178640 1 --2- 172.21.15.161:0/3506759103 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c9c14ee30 0x7f5c9c151710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:05.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.075+0000 7f5ca0977640 1 -- 172.21.15.161:0/3506759103 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1090424586 0 0) 0x7f5c9c155c00 con 0x7f5c9c151c50 2024-03-28T23:33:05.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.075+0000 7f5ca0977640 1 -- 172.21.15.161:0/3506759103 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c88003440 con 0x7f5c9c151c50 2024-03-28T23:33:05.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.075+0000 7f5ca1178640 1 -- 172.21.15.161:0/3506759103 >> v2:172.21.15.170:3300/0 conn(0x7f5c9c06c3d0 msgr2=0x7f5c9c14e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:05.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.075+0000 7f5ca1178640 1 --2- 172.21.15.161:0/3506759103 >> v2:172.21.15.170:3300/0 conn(0x7f5c9c06c3d0 0x7f5c9c14e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:05.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.075+0000 7f5ca1178640 1 -- 172.21.15.161:0/3506759103 >> v1:172.21.15.161:6789/0 conn(0x7f5c9c151c50 legacy=0x7f5c9c070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:05.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.075+0000 7f5ca1178640 1 -- 172.21.15.161:0/3506759103 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c9c06d080 con 0x7f5c9c14ee30 2024-03-28T23:33:05.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.075+0000 7f5ca1979640 1 --2- 172.21.15.161:0/3506759103 >> v2:172.21.15.170:3300/0 conn(0x7f5c9c06c3d0 0x7f5c9c14e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:05.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.076+0000 7f5ca1178640 1 --2- 172.21.15.161:0/3506759103 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c9c14ee30 0x7f5c9c151710 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c0097a0 tx=0x7f5c8c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5690bf2517da4ef server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:05.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.076+0000 7f5ca0977640 1 -- 172.21.15.161:0/3506759103 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c0315a0 con 0x7f5c9c14ee30 2024-03-28T23:33:05.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.076+0000 7f5ca0977640 1 -- 172.21.15.161:0/3506759103 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c8c037070 con 0x7f5c9c14ee30 2024-03-28T23:33:05.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.076+0000 7f5ca0977640 1 -- 172.21.15.161:0/3506759103 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c031960 con 0x7f5c9c14ee30 2024-03-28T23:33:05.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.076+0000 7f5ca3bdd640 1 -- 172.21.15.161:0/3506759103 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c9c14ee30 msgr2=0x7f5c9c151710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:05.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.076+0000 7f5ca3bdd640 1 --2- 172.21.15.161:0/3506759103 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c9c14ee30 0x7f5c9c151710 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c0097a0 tx=0x7f5c8c02b840 comp rx=0 tx=0).stop 2024-03-28T23:33:05.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.076+0000 7f5ca3bdd640 1 -- 172.21.15.161:0/3506759103 shutdown_connections 2024-03-28T23:33:05.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.076+0000 7f5ca3bdd640 1 --2- 172.21.15.161:0/3506759103 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c9c14ee30 0x7f5c9c151710 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:05.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.076+0000 7f5ca3bdd640 1 --2- 172.21.15.161:0/3506759103 >> v2:172.21.15.170:3300/0 conn(0x7f5c9c06c3d0 0x7f5c9c14e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:05.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.076+0000 7f5ca3bdd640 1 -- 172.21.15.161:0/3506759103 >> 172.21.15.161:0/3506759103 conn(0x7f5c9c0c3cf0 msgr2=0x7f5c9c0c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:05.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.076+0000 7f5ca3bdd640 1 -- 172.21.15.161:0/3506759103 shutdown_connections 2024-03-28T23:33:05.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.076+0000 7f5ca3bdd640 1 -- 172.21.15.161:0/3506759103 wait complete. 2024-03-28T23:33:05.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.077+0000 7f5ca3bdd640 1 Processor -- start 2024-03-28T23:33:05.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.078+0000 7f5ca3bdd640 1 -- start start 2024-03-28T23:33:05.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.078+0000 7f5ca3bdd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c9c06c3d0 0x7f5c9c1545b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:05.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.078+0000 7f5ca3bdd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c9c14ee30 0x7f5c9c1ea790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:05.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.078+0000 7f5ca3bdd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c9c151c50 0x7f5c9c0aa170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:05.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.078+0000 7f5ca3bdd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9c152260 con 0x7f5c9c151c50 2024-03-28T23:33:05.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.078+0000 7f5ca3bdd640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9c1520e0 con 0x7f5c9c14ee30 2024-03-28T23:33:05.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.078+0000 7f5ca3bdd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5c9c0c1bf0 con 0x7f5c9c06c3d0 2024-03-28T23:33:05.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.078+0000 7f5ca217a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c9c151c50 0x7f5c9c0aa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:05.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.078+0000 7f5ca217a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c9c151c50 0x7f5c9c0aa170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46190/0 (socket says 172.21.15.161:46190) 2024-03-28T23:33:05.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.078+0000 7f5ca217a640 1 -- 172.21.15.161:0/1816590597 learned_addr learned my addr 172.21.15.161:0/1816590597 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:05.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.078+0000 7f5ca1979640 1 --2- 172.21.15.161:0/1816590597 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c9c06c3d0 0x7f5c9c1545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:05.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.078+0000 7f5ca1178640 1 --2- 172.21.15.161:0/1816590597 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c9c14ee30 0x7f5c9c1ea790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:05.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.079+0000 7f5ca1979640 1 -- 172.21.15.161:0/1816590597 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c9c14ee30 msgr2=0x7f5c9c1ea790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:05.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.079+0000 7f5ca1979640 1 --2- 172.21.15.161:0/1816590597 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c9c14ee30 0x7f5c9c1ea790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:05.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.079+0000 7f5ca1979640 1 -- 172.21.15.161:0/1816590597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c9c151c50 msgr2=0x7f5c9c0aa170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:05.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.079+0000 7f5ca1979640 1 --2- 172.21.15.161:0/1816590597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c9c151c50 0x7f5c9c0aa170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:05.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.079+0000 7f5ca1979640 1 -- 172.21.15.161:0/1816590597 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c9c153af0 con 0x7f5c9c06c3d0 2024-03-28T23:33:05.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.079+0000 7f5ca1178640 1 --2- 172.21.15.161:0/1816590597 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c9c14ee30 0x7f5c9c1ea790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:05.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.079+0000 7f5ca217a640 1 --2- 172.21.15.161:0/1816590597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c9c151c50 0x7f5c9c0aa170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:05.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.079+0000 7f5ca1979640 1 --2- 172.21.15.161:0/1816590597 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c9c06c3d0 0x7f5c9c1545b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5c9800e870 tx=0x7f5c9800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:05.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.079+0000 7f5c927fc640 1 -- 172.21.15.161:0/1816590597 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c98017070 con 0x7f5c9c06c3d0 2024-03-28T23:33:05.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.079+0000 7f5c927fc640 1 -- 172.21.15.161:0/1816590597 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c9800c940 con 0x7f5c9c06c3d0 2024-03-28T23:33:05.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.079+0000 7f5ca3bdd640 1 -- 172.21.15.161:0/1816590597 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c9c1552f0 con 0x7f5c9c06c3d0 2024-03-28T23:33:05.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.079+0000 7f5c927fc640 1 -- 172.21.15.161:0/1816590597 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c9800cd10 con 0x7f5c9c06c3d0 2024-03-28T23:33:05.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.079+0000 7f5ca3bdd640 1 -- 172.21.15.161:0/1816590597 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c9c153db0 con 0x7f5c9c06c3d0 2024-03-28T23:33:05.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.080+0000 7f5ca3bdd640 1 -- 172.21.15.161:0/1816590597 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c9c04f4b0 con 0x7f5c9c06c3d0 2024-03-28T23:33:05.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.080+0000 7f5c927fc640 1 -- 172.21.15.161:0/1816590597 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5c9801a070 con 0x7f5c9c06c3d0 2024-03-28T23:33:05.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.081+0000 7f5c927fc640 1 --2- 172.21.15.161:0/1816590597 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5c8006b500 0x7f5c8006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:05.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.081+0000 7f5c927fc640 1 -- 172.21.15.161:0/1816590597 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(78..78 src has 1..78) ==== 7043+0+0 (secure 0 0 0) 0x7f5c9808a830 con 0x7f5c9c06c3d0 2024-03-28T23:33:05.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.081+0000 7f5ca1178640 1 --2- 172.21.15.161:0/1816590597 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5c8006b500 0x7f5c8006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:05.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.081+0000 7f5ca1178640 1 --2- 172.21.15.161:0/1816590597 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5c8006b500 0x7f5c8006d9c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c036000 tx=0x7f5c8c03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:05.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.085+0000 7f5c927fc640 1 -- 172.21.15.161:0/1816590597 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5c98058b10 con 0x7f5c9c06c3d0 2024-03-28T23:33:05.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.300+0000 7f5ca3bdd640 1 -- 172.21.15.161:0/1816590597 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f5c9c0682d0 con 0x7f5c9c06c3d0 2024-03-28T23:33:05.302 INFO:tasks.ceph.mon.a.smithi161.stderr:2024-03-28T23:33:05.301+0000 7fdc7b85b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T23:33:05.307 INFO:tasks.ceph.mds.a.smithi161.stderr: -155> 2024-03-28T23:33:00.528+0000 7fbb026d3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:33:05.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.307+0000 7f5c927fc640 1 -- 172.21.15.161:0/1816590597 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v68) ==== 161+0+0 (secure 0 0 0) 0x7f5c9805cbb0 con 0x7f5c9c06c3d0 2024-03-28T23:33:05.307 INFO:teuthology.orchestra.run.smithi161.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-28T23:33:05.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.309+0000 7f5ca3bdd640 1 -- 172.21.15.161:0/1816590597 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5c8006b500 msgr2=0x7f5c8006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:05.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.309+0000 7f5ca3bdd640 1 --2- 172.21.15.161:0/1816590597 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5c8006b500 0x7f5c8006d9c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c036000 tx=0x7f5c8c03d040 comp rx=0 tx=0).stop 2024-03-28T23:33:05.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.309+0000 7f5ca3bdd640 1 -- 172.21.15.161:0/1816590597 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c9c06c3d0 msgr2=0x7f5c9c1545b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:05.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.309+0000 7f5ca3bdd640 1 --2- 172.21.15.161:0/1816590597 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c9c06c3d0 0x7f5c9c1545b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5c9800e870 tx=0x7f5c9800ed30 comp rx=0 tx=0).stop 2024-03-28T23:33:05.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.309+0000 7f5ca3bdd640 1 -- 172.21.15.161:0/1816590597 shutdown_connections 2024-03-28T23:33:05.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.309+0000 7f5ca3bdd640 1 --2- 172.21.15.161:0/1816590597 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5c8006b500 0x7f5c8006d9c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:05.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.309+0000 7f5ca3bdd640 1 --2- 172.21.15.161:0/1816590597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c9c151c50 0x7f5c9c0aa170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:05.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.309+0000 7f5ca3bdd640 1 --2- 172.21.15.161:0/1816590597 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c9c14ee30 0x7f5c9c1ea790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:05.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.309+0000 7f5ca3bdd640 1 --2- 172.21.15.161:0/1816590597 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c9c06c3d0 0x7f5c9c1545b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:05.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.309+0000 7f5ca3bdd640 1 -- 172.21.15.161:0/1816590597 >> 172.21.15.161:0/1816590597 conn(0x7f5c9c0c3cf0 msgr2=0x7f5c9c0c2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:05.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.309+0000 7f5ca3bdd640 1 -- 172.21.15.161:0/1816590597 shutdown_connections 2024-03-28T23:33:05.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.309+0000 7f5ca3bdd640 1 -- 172.21.15.161:0/1816590597 wait complete. 2024-03-28T23:33:05.325 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-28T23:33:05.335 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-28T23:33:05.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.427+0000 7fd8bd5d0640 1 Processor -- start 2024-03-28T23:33:05.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.427+0000 7fd8bd5d0640 1 -- start start 2024-03-28T23:33:05.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.428+0000 7fd8bd5d0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd8b8152bc0 0x7fd8b80c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:05.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.428+0000 7fd8bd5d0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8b80c6f60 con 0x7fd8b80c8a80 2024-03-28T23:33:05.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.428+0000 7fd8bd5d0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8b80c6120 con 0x7fd8b81589b0 2024-03-28T23:33:05.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.428+0000 7fd8bd5d0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd8b80c6400 con 0x7fd8b8152bc0 2024-03-28T23:33:05.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.428+0000 7fd8b6ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd8b8152bc0 0x7fd8b80c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:05.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.428+0000 7fd8b67fc640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd8b80c8a80 0x7fd8b80c8ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44916/0 (socket says 172.21.15.161:44916) 2024-03-28T23:33:05.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.428+0000 7fd8b67fc640 1 -- 172.21.15.161:0/2620721649 learned_addr learned my addr 172.21.15.161:0/2620721649 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:05.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.428+0000 7fd8b6ffd640 1 -- 172.21.15.161:0/2620721649 >> v1:172.21.15.170:6789/0 conn(0x7fd8b81589b0 legacy=0x7fd8b8158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:05.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.428+0000 7fd8b6ffd640 1 -- 172.21.15.161:0/2620721649 >> v1:172.21.15.161:6789/0 conn(0x7fd8b80c8a80 legacy=0x7fd8b80c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:05.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.429+0000 7fd8b6ffd640 1 -- 172.21.15.161:0/2620721649 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8b80c7240 con 0x7fd8b8152bc0 2024-03-28T23:33:05.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.428+0000 7fd8b5ffb640 1 -- 172.21.15.161:0/2620721649 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1308636184 0 0) 0x7fd8b80c6120 con 0x7fd8b81589b0 2024-03-28T23:33:05.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.429+0000 7fd8b6ffd640 1 --2- 172.21.15.161:0/2620721649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8b8152bc0 0x7fd8b80c0330 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009d90 tx=0x7fd8a802bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea7c1b7149ffb2d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:05.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.429+0000 7fd8b5ffb640 1 -- 172.21.15.161:0/2620721649 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8039070 con 0x7fd8b8152bc0 2024-03-28T23:33:05.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.429+0000 7fd8b5ffb640 1 -- 172.21.15.161:0/2620721649 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8a802cab0 con 0x7fd8b8152bc0 2024-03-28T23:33:05.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.429+0000 7fd8b5ffb640 1 -- 172.21.15.161:0/2620721649 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8034440 con 0x7fd8b8152bc0 2024-03-28T23:33:05.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.430+0000 7fd8bd5d0640 1 -- 172.21.15.161:0/2620721649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8b8152bc0 msgr2=0x7fd8b80c0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:05.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.430+0000 7fd8bd5d0640 1 --2- 172.21.15.161:0/2620721649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8b8152bc0 0x7fd8b80c0330 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009d90 tx=0x7fd8a802bd80 comp rx=0 tx=0).stop 2024-03-28T23:33:05.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.430+0000 7fd8bd5d0640 1 -- 172.21.15.161:0/2620721649 shutdown_connections 2024-03-28T23:33:05.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.430+0000 7fd8bd5d0640 1 --2- 172.21.15.161:0/2620721649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8b8152bc0 0x7fd8b80c0330 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:05.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.430+0000 7fd8bd5d0640 1 -- 172.21.15.161:0/2620721649 >> 172.21.15.161:0/2620721649 conn(0x7fd8b80caa80 msgr2=0x7fd8b80cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:05.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.430+0000 7fd8bd5d0640 1 -- 172.21.15.161:0/2620721649 shutdown_connections 2024-03-28T23:33:05.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.430+0000 7fd8bd5d0640 1 -- 172.21.15.161:0/2620721649 wait complete. 2024-03-28T23:33:05.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.431+0000 7fd8bd5d0640 1 Processor -- start 2024-03-28T23:33:05.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.431+0000 7fd8bd5d0640 1 -- start start 2024-03-28T23:33:05.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.432+0000 7fd8bd5d0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8b80c8a80 0x7fd8b8146c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:05.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.432+0000 7fd8bd5d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8b8152bc0 0x7fd8b81eb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:05.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.432+0000 7fd8bd5d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8b81589b0 0x7fd8b80aba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:05.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.432+0000 7fd8bd5d0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b8152fa0 con 0x7fd8b80c8a80 2024-03-28T23:33:05.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.432+0000 7fd8bd5d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b80c95f0 con 0x7fd8b81589b0 2024-03-28T23:33:05.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.432+0000 7fd8bd5d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd8b8156230 con 0x7fd8b8152bc0 2024-03-28T23:33:05.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.432+0000 7fd8b6ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8b80c8a80 0x7fd8b8146c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:05.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.432+0000 7fd8b6ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8b80c8a80 0x7fd8b8146c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46222/0 (socket says 172.21.15.161:46222) 2024-03-28T23:33:05.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.432+0000 7fd8b6ffd640 1 -- 172.21.15.161:0/3526305481 learned_addr learned my addr 172.21.15.161:0/3526305481 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:05.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.432+0000 7fd8b77fe640 1 --2- 172.21.15.161:0/3526305481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8b81589b0 0x7fd8b80aba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:05.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.432+0000 7fd8b67fc640 1 --2- 172.21.15.161:0/3526305481 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8b8152bc0 0x7fd8b81eb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:05.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.432+0000 7fd8b6ffd640 1 -- 172.21.15.161:0/3526305481 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8b8152bc0 msgr2=0x7fd8b81eb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:05.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.432+0000 7fd8b6ffd640 1 --2- 172.21.15.161:0/3526305481 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8b8152bc0 0x7fd8b81eb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:05.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.432+0000 7fd8b6ffd640 1 -- 172.21.15.161:0/3526305481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8b81589b0 msgr2=0x7fd8b80aba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:05.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.432+0000 7fd8b6ffd640 1 --2- 172.21.15.161:0/3526305481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8b81589b0 0x7fd8b80aba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:05.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.432+0000 7fd8b6ffd640 1 -- 172.21.15.161:0/3526305481 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8b81eaef0 con 0x7fd8b80c8a80 2024-03-28T23:33:05.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.432+0000 7fd8b6ffd640 1 --2- 172.21.15.161:0/3526305481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8b80c8a80 0x7fd8b8146c80 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8037040 tx=0x7fd8a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:05.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.433+0000 7fd89f7fe640 1 -- 172.21.15.161:0/3526305481 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8044020 con 0x7fd8b80c8a80 2024-03-28T23:33:05.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.433+0000 7fd89f7fe640 1 -- 172.21.15.161:0/3526305481 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8a803f670 con 0x7fd8b80c8a80 2024-03-28T23:33:05.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.433+0000 7fd8bd5d0640 1 -- 172.21.15.161:0/3526305481 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8b81eef00 con 0x7fd8b80c8a80 2024-03-28T23:33:05.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.433+0000 7fd89f7fe640 1 -- 172.21.15.161:0/3526305481 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8039040 con 0x7fd8b80c8a80 2024-03-28T23:33:05.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.433+0000 7fd8bd5d0640 1 -- 172.21.15.161:0/3526305481 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8b81ef1e0 con 0x7fd8b80c8a80 2024-03-28T23:33:05.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.434+0000 7fd8bd5d0640 1 -- 172.21.15.161:0/3526305481 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8b804f4b0 con 0x7fd8b80c8a80 2024-03-28T23:33:05.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.434+0000 7fd89f7fe640 1 -- 172.21.15.161:0/3526305481 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd8a8047020 con 0x7fd8b80c8a80 2024-03-28T23:33:05.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.435+0000 7fd89f7fe640 1 --2- 172.21.15.161:0/3526305481 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd8a406b4d0 0x7fd8a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:05.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.435+0000 7fd89f7fe640 1 -- 172.21.15.161:0/3526305481 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7129+0+0 (secure 0 0 0) 0x7fd8a80ad2c0 con 0x7fd8b80c8a80 2024-03-28T23:33:05.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.435+0000 7fd8b67fc640 1 --2- 172.21.15.161:0/3526305481 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd8a406b4d0 0x7fd8a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:05.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.435+0000 7fd8b67fc640 1 --2- 172.21.15.161:0/3526305481 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd8a406b4d0 0x7fd8a406d990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8145710 tx=0x7fd8ac007e20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:05.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.439+0000 7fd89f7fe640 1 -- 172.21.15.161:0/3526305481 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd8a804e030 con 0x7fd8b80c8a80 2024-03-28T23:33:05.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:05.651+0000 7fd8bd5d0640 1 -- 172.21.15.161:0/3526305481 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fd8b814b8d0 con 0x7fd8b80c8a80 2024-03-28T23:33:06.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.307+0000 7fd89f7fe640 1 -- 172.21.15.161:0/3526305481 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v69) ==== 104+0+0 (secure 0 0 0) 0x7fd8a807b560 con 0x7fd8b80c8a80 2024-03-28T23:33:06.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.309+0000 7fd8bd5d0640 1 -- 172.21.15.161:0/3526305481 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd8a406b4d0 msgr2=0x7fd8a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:06.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.309+0000 7fd8bd5d0640 1 --2- 172.21.15.161:0/3526305481 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd8a406b4d0 0x7fd8a406d990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8145710 tx=0x7fd8ac007e20 comp rx=0 tx=0).stop 2024-03-28T23:33:06.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.309+0000 7fd8bd5d0640 1 -- 172.21.15.161:0/3526305481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8b80c8a80 msgr2=0x7fd8b8146c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:06.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.309+0000 7fd8bd5d0640 1 --2- 172.21.15.161:0/3526305481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8b80c8a80 0x7fd8b8146c80 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8037040 tx=0x7fd8a8004a40 comp rx=0 tx=0).stop 2024-03-28T23:33:06.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.309+0000 7fd8bd5d0640 1 -- 172.21.15.161:0/3526305481 shutdown_connections 2024-03-28T23:33:06.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.309+0000 7fd8bd5d0640 1 --2- 172.21.15.161:0/3526305481 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd8a406b4d0 0x7fd8a406d990 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:06.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.309+0000 7fd8bd5d0640 1 --2- 172.21.15.161:0/3526305481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8b81589b0 0x7fd8b80aba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:06.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.309+0000 7fd8bd5d0640 1 --2- 172.21.15.161:0/3526305481 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8b8152bc0 0x7fd8b81eb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:06.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.309+0000 7fd8bd5d0640 1 --2- 172.21.15.161:0/3526305481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8b80c8a80 0x7fd8b8146c80 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:06.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.309+0000 7fd8bd5d0640 1 -- 172.21.15.161:0/3526305481 >> 172.21.15.161:0/3526305481 conn(0x7fd8b80caa80 msgr2=0x7fd8b8158e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:06.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.309+0000 7fd8bd5d0640 1 -- 172.21.15.161:0/3526305481 shutdown_connections 2024-03-28T23:33:06.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.310+0000 7fd8bd5d0640 1 -- 172.21.15.161:0/3526305481 wait complete. 2024-03-28T23:33:06.316 INFO:tasks.ceph.mds.b.smithi170.stderr: -585> 2024-03-28T23:32:48.302+0000 7f88193ea640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:33:06.325 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-28T23:33:06.341 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2024-03-28T23:33:06.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.471+0000 7f9af3409640 1 Processor -- start 2024-03-28T23:33:06.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.472+0000 7f9af3409640 1 -- start start 2024-03-28T23:33:06.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.472+0000 7f9af3409640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9aec14f1d0 0x7f9aec151db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:06.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.472+0000 7f9af3409640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9aec06cab0 0x7f9aec06cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:06.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.472+0000 7f9af3409640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9aec0c15a0 con 0x7f9aec0c3080 2024-03-28T23:33:06.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.472+0000 7f9af3409640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f9aec0c6330 con 0x7f9aec14f1d0 2024-03-28T23:33:06.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.472+0000 7f9af3409640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f9aec0c1840 con 0x7f9aec06cab0 2024-03-28T23:33:06.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.472+0000 7f9af11a5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9aec14f1d0 0x7f9aec151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:06.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.472+0000 7f9af11a5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9aec14f1d0 0x7f9aec151db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46224/0 (socket says 172.21.15.161:46224) 2024-03-28T23:33:06.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.472+0000 7f9af11a5640 1 -- 172.21.15.161:0/188079495 learned_addr learned my addr 172.21.15.161:0/188079495 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:06.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.473+0000 7f9af19a6640 1 --2- 172.21.15.161:0/188079495 >> v2:172.21.15.170:3300/0 conn(0x7f9aec06cab0 0x7f9aec06cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:06.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.473+0000 7f9af11a5640 1 -- 172.21.15.161:0/188079495 >> v2:172.21.15.170:3300/0 conn(0x7f9aec06cab0 msgr2=0x7f9aec06cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:06.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.473+0000 7f9af11a5640 1 --2- 172.21.15.161:0/188079495 >> v2:172.21.15.170:3300/0 conn(0x7f9aec06cab0 0x7f9aec06cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:06.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.473+0000 7f9af11a5640 1 -- 172.21.15.161:0/188079495 >> v1:172.21.15.170:6789/0 conn(0x7f9aec0c3080 legacy=0x7f9aec0c3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:06.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.473+0000 7f9af11a5640 1 -- 172.21.15.161:0/188079495 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aec0c1b20 con 0x7f9aec14f1d0 2024-03-28T23:33:06.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.474+0000 7f9af11a5640 1 --2- 172.21.15.161:0/188079495 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9aec14f1d0 0x7f9aec151db0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f9aec0acd20 tx=0x7f9ae802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5562992ea2e4711f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:06.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.474+0000 7f9ae3fff640 1 -- 172.21.15.161:0/188079495 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae802c9f0 con 0x7f9aec14f1d0 2024-03-28T23:33:06.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.474+0000 7f9ae3fff640 1 -- 172.21.15.161:0/188079495 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ae802cb90 con 0x7f9aec14f1d0 2024-03-28T23:33:06.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.474+0000 7f9ae3fff640 1 -- 172.21.15.161:0/188079495 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae80343f0 con 0x7f9aec14f1d0 2024-03-28T23:33:06.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.474+0000 7f9af3409640 1 -- 172.21.15.161:0/188079495 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9aec14f1d0 msgr2=0x7f9aec151db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:06.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.474+0000 7f9af3409640 1 --2- 172.21.15.161:0/188079495 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9aec14f1d0 0x7f9aec151db0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f9aec0acd20 tx=0x7f9ae802be50 comp rx=0 tx=0).stop 2024-03-28T23:33:06.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.474+0000 7f9af3409640 1 -- 172.21.15.161:0/188079495 shutdown_connections 2024-03-28T23:33:06.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.474+0000 7f9af3409640 1 --2- 172.21.15.161:0/188079495 >> v2:172.21.15.170:3300/0 conn(0x7f9aec06cab0 0x7f9aec06cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:06.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.474+0000 7f9af3409640 1 --2- 172.21.15.161:0/188079495 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9aec14f1d0 0x7f9aec151db0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:06.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.474+0000 7f9af3409640 1 -- 172.21.15.161:0/188079495 >> 172.21.15.161:0/188079495 conn(0x7f9aec0c6e20 msgr2=0x7f9aec0c7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:06.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.475+0000 7f9af3409640 1 -- 172.21.15.161:0/188079495 shutdown_connections 2024-03-28T23:33:06.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.475+0000 7f9af3409640 1 -- 172.21.15.161:0/188079495 wait complete. 2024-03-28T23:33:06.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.476+0000 7f9af3409640 1 Processor -- start 2024-03-28T23:33:06.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.476+0000 7f9af3409640 1 -- start start 2024-03-28T23:33:06.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.476+0000 7f9af3409640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9aec06cab0 0x7f9aec0ab400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:06.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.476+0000 7f9af3409640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9aec0c3080 0x7f9aec1e8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:06.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.476+0000 7f9af3409640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9aec14f1d0 0x7f9aec1e8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:06.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.476+0000 7f9af3409640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9aec06d480 con 0x7f9aec14f1d0 2024-03-28T23:33:06.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.476+0000 7f9af3409640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9aec0c2600 con 0x7f9aec06cab0 2024-03-28T23:33:06.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.476+0000 7f9af3409640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9aec0c41a0 con 0x7f9aec0c3080 2024-03-28T23:33:06.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.476+0000 7f9af19a6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9aec14f1d0 0x7f9aec1e8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:06.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.476+0000 7f9af19a6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9aec14f1d0 0x7f9aec1e8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46226/0 (socket says 172.21.15.161:46226) 2024-03-28T23:33:06.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.476+0000 7f9af19a6640 1 -- 172.21.15.161:0/1946787337 learned_addr learned my addr 172.21.15.161:0/1946787337 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:06.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.477+0000 7f9af09a4640 1 --2- 172.21.15.161:0/1946787337 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9aec0c3080 0x7f9aec1e8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:06.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.477+0000 7f9af11a5640 1 --2- 172.21.15.161:0/1946787337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9aec06cab0 0x7f9aec0ab400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:06.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.477+0000 7f9af19a6640 1 -- 172.21.15.161:0/1946787337 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9aec0c3080 msgr2=0x7f9aec1e8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:06.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.477+0000 7f9af19a6640 1 --2- 172.21.15.161:0/1946787337 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9aec0c3080 0x7f9aec1e8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:06.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.477+0000 7f9af19a6640 1 -- 172.21.15.161:0/1946787337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9aec06cab0 msgr2=0x7f9aec0ab400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:06.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.477+0000 7f9af19a6640 1 --2- 172.21.15.161:0/1946787337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9aec06cab0 0x7f9aec0ab400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:06.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.477+0000 7f9af19a6640 1 -- 172.21.15.161:0/1946787337 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aec0a8f60 con 0x7f9aec14f1d0 2024-03-28T23:33:06.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.477+0000 7f9af19a6640 1 --2- 172.21.15.161:0/1946787337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9aec14f1d0 0x7f9aec1e8e50 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f9ae400c7f0 tx=0x7f9ae400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:06.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.477+0000 7f9ae1ffb640 1 -- 172.21.15.161:0/1946787337 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae4015070 con 0x7f9aec14f1d0 2024-03-28T23:33:06.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.477+0000 7f9ae1ffb640 1 -- 172.21.15.161:0/1946787337 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ae4004220 con 0x7f9aec14f1d0 2024-03-28T23:33:06.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.477+0000 7f9ae1ffb640 1 -- 172.21.15.161:0/1946787337 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae4007620 con 0x7f9aec14f1d0 2024-03-28T23:33:06.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.477+0000 7f9af3409640 1 -- 172.21.15.161:0/1946787337 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aec0a9240 con 0x7f9aec14f1d0 2024-03-28T23:33:06.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.477+0000 7f9af3409640 1 -- 172.21.15.161:0/1946787337 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aec156140 con 0x7f9aec14f1d0 2024-03-28T23:33:06.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.478+0000 7f9af3409640 1 -- 172.21.15.161:0/1946787337 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9aec068f90 con 0x7f9aec14f1d0 2024-03-28T23:33:06.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.479+0000 7f9ae1ffb640 1 -- 172.21.15.161:0/1946787337 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9ae40079d0 con 0x7f9aec14f1d0 2024-03-28T23:33:06.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.479+0000 7f9ae1ffb640 1 --2- 172.21.15.161:0/1946787337 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9ad006b4d0 0x7f9ad006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:06.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.479+0000 7f9ae1ffb640 1 -- 172.21.15.161:0/1946787337 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7215+0+0 (secure 0 0 0) 0x7f9ae408a770 con 0x7f9aec14f1d0 2024-03-28T23:33:06.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.479+0000 7f9af11a5640 1 --2- 172.21.15.161:0/1946787337 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9ad006b4d0 0x7f9ad006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:06.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.479+0000 7f9af11a5640 1 --2- 172.21.15.161:0/1946787337 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9ad006b4d0 0x7f9ad006d990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8035000 tx=0x7f9ae8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:06.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.483+0000 7f9ae1ffb640 1 -- 172.21.15.161:0/1946787337 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9ae4058a50 con 0x7f9aec14f1d0 2024-03-28T23:33:06.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:06.656+0000 7f9af3409640 1 -- 172.21.15.161:0/1946787337 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9aec14c960 con 0x7f9aec14f1d0 2024-03-28T23:33:07.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.558+0000 7f9ae1ffb640 1 -- 172.21.15.161:0/1946787337 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v81) ==== 181+0+0 (secure 0 0 0) 0x7f9ae405caf0 con 0x7f9aec14f1d0 2024-03-28T23:33:07.559 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_metadata' removed 2024-03-28T23:33:07.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.561+0000 7f9af3409640 1 -- 172.21.15.161:0/1946787337 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9ad006b4d0 msgr2=0x7f9ad006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:07.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.561+0000 7f9af3409640 1 --2- 172.21.15.161:0/1946787337 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9ad006b4d0 0x7f9ad006d990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8035000 tx=0x7f9ae8030000 comp rx=0 tx=0).stop 2024-03-28T23:33:07.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.561+0000 7f9af3409640 1 -- 172.21.15.161:0/1946787337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9aec14f1d0 msgr2=0x7f9aec1e8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:07.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.561+0000 7f9af3409640 1 --2- 172.21.15.161:0/1946787337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9aec14f1d0 0x7f9aec1e8e50 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f9ae400c7f0 tx=0x7f9ae400ccb0 comp rx=0 tx=0).stop 2024-03-28T23:33:07.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.562+0000 7f9af3409640 1 -- 172.21.15.161:0/1946787337 shutdown_connections 2024-03-28T23:33:07.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.562+0000 7f9af3409640 1 --2- 172.21.15.161:0/1946787337 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9ad006b4d0 0x7f9ad006d990 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:07.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.562+0000 7f9af3409640 1 --2- 172.21.15.161:0/1946787337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9aec14f1d0 0x7f9aec1e8e50 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:07.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.562+0000 7f9af3409640 1 --2- 172.21.15.161:0/1946787337 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9aec0c3080 0x7f9aec1e8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:07.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.562+0000 7f9af3409640 1 --2- 172.21.15.161:0/1946787337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9aec06cab0 0x7f9aec0ab400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:07.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.562+0000 7f9af3409640 1 -- 172.21.15.161:0/1946787337 >> 172.21.15.161:0/1946787337 conn(0x7f9aec0c6e20 msgr2=0x7f9aec151be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:07.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.563+0000 7f9af3409640 1 -- 172.21.15.161:0/1946787337 shutdown_connections 2024-03-28T23:33:07.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.563+0000 7f9af3409640 1 -- 172.21.15.161:0/1946787337 wait complete. 2024-03-28T23:33:07.578 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-28T23:33:07.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.692+0000 7febcadb5640 1 Processor -- start 2024-03-28T23:33:07.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.692+0000 7febcadb5640 1 -- start start 2024-03-28T23:33:07.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.693+0000 7febcadb5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7febc40ac430 0x7febc4152880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:07.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.693+0000 7febcadb5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7febc40c19b0 0x7febc40c1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:07.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.693+0000 7febcadb5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febc40c3540 con 0x7febc40c30c0 2024-03-28T23:33:07.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.693+0000 7febcadb5640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7febc4152dc0 con 0x7febc40ac430 2024-03-28T23:33:07.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.693+0000 7febcadb5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7febc40c2300 con 0x7febc40c19b0 2024-03-28T23:33:07.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.693+0000 7febc8b51640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7febc40ac430 0x7febc4152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:07.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.693+0000 7febc8b51640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7febc40ac430 0x7febc4152880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46240/0 (socket says 172.21.15.161:46240) 2024-03-28T23:33:07.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.693+0000 7febc8b51640 1 -- 172.21.15.161:0/4273179161 learned_addr learned my addr 172.21.15.161:0/4273179161 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:07.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.693+0000 7febc9352640 1 --2- 172.21.15.161:0/4273179161 >> v2:172.21.15.170:3300/0 conn(0x7febc40c19b0 0x7febc40c1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:07.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.693+0000 7febc8b51640 1 -- 172.21.15.161:0/4273179161 >> v2:172.21.15.170:3300/0 conn(0x7febc40c19b0 msgr2=0x7febc40c1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:07.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.693+0000 7febc8b51640 1 --2- 172.21.15.161:0/4273179161 >> v2:172.21.15.170:3300/0 conn(0x7febc40c19b0 0x7febc40c1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:07.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.693+0000 7febc8b51640 1 -- 172.21.15.161:0/4273179161 >> v1:172.21.15.170:6789/0 conn(0x7febc40c30c0 legacy=0x7febc40c24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:07.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.693+0000 7febc8b51640 1 -- 172.21.15.161:0/4273179161 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febc40c3820 con 0x7febc40ac430 2024-03-28T23:33:07.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.694+0000 7febc8b51640 1 --2- 172.21.15.161:0/4273179161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febc40ac430 0x7febc4152880 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7febc40c61b0 tx=0x7febb402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76ca333fceae5298 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:07.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.694+0000 7febc37fe640 1 -- 172.21.15.161:0/4273179161 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb4035020 con 0x7febc40ac430 2024-03-28T23:33:07.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.694+0000 7febc37fe640 1 -- 172.21.15.161:0/4273179161 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febb4031870 con 0x7febc40ac430 2024-03-28T23:33:07.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.694+0000 7febc37fe640 1 -- 172.21.15.161:0/4273179161 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb4031c40 con 0x7febc40ac430 2024-03-28T23:33:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.695+0000 7febcadb5640 1 -- 172.21.15.161:0/4273179161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febc40ac430 msgr2=0x7febc4152880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.695+0000 7febcadb5640 1 --2- 172.21.15.161:0/4273179161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febc40ac430 0x7febc4152880 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7febc40c61b0 tx=0x7febb402b910 comp rx=0 tx=0).stop 2024-03-28T23:33:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.695+0000 7febcadb5640 1 -- 172.21.15.161:0/4273179161 shutdown_connections 2024-03-28T23:33:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.695+0000 7febcadb5640 1 --2- 172.21.15.161:0/4273179161 >> v2:172.21.15.170:3300/0 conn(0x7febc40c19b0 0x7febc40c1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.695+0000 7febcadb5640 1 --2- 172.21.15.161:0/4273179161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febc40ac430 0x7febc4152880 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.695+0000 7febcadb5640 1 -- 172.21.15.161:0/4273179161 >> 172.21.15.161:0/4273179161 conn(0x7febc406b150 msgr2=0x7febc406b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.695+0000 7febcadb5640 1 -- 172.21.15.161:0/4273179161 shutdown_connections 2024-03-28T23:33:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.695+0000 7febcadb5640 1 -- 172.21.15.161:0/4273179161 wait complete. 2024-03-28T23:33:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.696+0000 7febcadb5640 1 Processor -- start 2024-03-28T23:33:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.696+0000 7febcadb5640 1 -- start start 2024-03-28T23:33:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.697+0000 7febcadb5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7febc40ac430 0x7febc4155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.697+0000 7febcadb5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7febc40c19b0 0x7febc4153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.697+0000 7febcadb5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febc40c30c0 0x7febc41eacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.697+0000 7febcadb5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7febc40c1490 con 0x7febc40c30c0 2024-03-28T23:33:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.697+0000 7febcadb5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7febc40c11b0 con 0x7febc40ac430 2024-03-28T23:33:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.697+0000 7febcadb5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7febc40c6440 con 0x7febc40c19b0 2024-03-28T23:33:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.697+0000 7febc9352640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febc40c30c0 0x7febc41eacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.697+0000 7febc9352640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febc40c30c0 0x7febc41eacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46244/0 (socket says 172.21.15.161:46244) 2024-03-28T23:33:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.697+0000 7febc9352640 1 -- 172.21.15.161:0/1249218134 learned_addr learned my addr 172.21.15.161:0/1249218134 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.697+0000 7febc8b51640 1 --2- 172.21.15.161:0/1249218134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7febc40ac430 0x7febc4155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.697+0000 7febc3fff640 1 --2- 172.21.15.161:0/1249218134 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7febc40c19b0 0x7febc4153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.697+0000 7febc9352640 1 -- 172.21.15.161:0/1249218134 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7febc40c19b0 msgr2=0x7febc4153460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.697+0000 7febc9352640 1 --2- 172.21.15.161:0/1249218134 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7febc40c19b0 0x7febc4153460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.697+0000 7febc9352640 1 -- 172.21.15.161:0/1249218134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7febc40ac430 msgr2=0x7febc4155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.697+0000 7febc9352640 1 --2- 172.21.15.161:0/1249218134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7febc40ac430 0x7febc4155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.697+0000 7febc9352640 1 -- 172.21.15.161:0/1249218134 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febc4153d30 con 0x7febc40c30c0 2024-03-28T23:33:07.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.698+0000 7febc9352640 1 --2- 172.21.15.161:0/1249218134 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febc40c30c0 0x7febc41eacc0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7febbc00efc0 tx=0x7febbc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:07.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.698+0000 7febc17fa640 1 -- 172.21.15.161:0/1249218134 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febbc018020 con 0x7febc40c30c0 2024-03-28T23:33:07.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.698+0000 7febc17fa640 1 -- 172.21.15.161:0/1249218134 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febbc014480 con 0x7febc40c30c0 2024-03-28T23:33:07.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.698+0000 7febcadb5640 1 -- 172.21.15.161:0/1249218134 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febc4154010 con 0x7febc40c30c0 2024-03-28T23:33:07.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.698+0000 7febc17fa640 1 -- 172.21.15.161:0/1249218134 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febbc014870 con 0x7febc40c30c0 2024-03-28T23:33:07.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.698+0000 7febcadb5640 1 -- 172.21.15.161:0/1249218134 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febc41539a0 con 0x7febc40c30c0 2024-03-28T23:33:07.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.699+0000 7febcadb5640 1 -- 172.21.15.161:0/1249218134 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febc4068f90 con 0x7febc40c30c0 2024-03-28T23:33:07.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.699+0000 7febc17fa640 1 -- 172.21.15.161:0/1249218134 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7febbc005d10 con 0x7febc40c30c0 2024-03-28T23:33:07.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.699+0000 7febc17fa640 1 --2- 172.21.15.161:0/1249218134 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7febb006b500 0x7febb006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:07.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.700+0000 7febc17fa640 1 -- 172.21.15.161:0/1249218134 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 6777+0+0 (secure 0 0 0) 0x7febbc08aef0 con 0x7febc40c30c0 2024-03-28T23:33:07.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.700+0000 7febc8b51640 1 --2- 172.21.15.161:0/1249218134 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7febb006b500 0x7febb006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:07.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.700+0000 7febc8b51640 1 --2- 172.21.15.161:0/1249218134 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7febb006b500 0x7febb006d9c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7febb4036000 tx=0x7febb403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:07.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.704+0000 7febc17fa640 1 -- 172.21.15.161:0/1249218134 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7febbc022070 con 0x7febc40c30c0 2024-03-28T23:33:07.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:07.876+0000 7febcadb5640 1 -- 172.21.15.161:0/1249218134 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7febc40c6220 con 0x7febc40c30c0 2024-03-28T23:33:08.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.562+0000 7febc17fa640 1 -- 172.21.15.161:0/1249218134 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v82) ==== 169+0+0 (secure 0 0 0) 0x7febbc010040 con 0x7febc40c30c0 2024-03-28T23:33:08.562 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data' removed 2024-03-28T23:33:08.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.564+0000 7febcadb5640 1 -- 172.21.15.161:0/1249218134 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7febb006b500 msgr2=0x7febb006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:08.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.564+0000 7febcadb5640 1 --2- 172.21.15.161:0/1249218134 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7febb006b500 0x7febb006d9c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7febb4036000 tx=0x7febb403c040 comp rx=0 tx=0).stop 2024-03-28T23:33:08.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.564+0000 7febcadb5640 1 -- 172.21.15.161:0/1249218134 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febc40c30c0 msgr2=0x7febc41eacc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:08.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.564+0000 7febcadb5640 1 --2- 172.21.15.161:0/1249218134 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febc40c30c0 0x7febc41eacc0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7febbc00efc0 tx=0x7febbc00c490 comp rx=0 tx=0).stop 2024-03-28T23:33:08.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.564+0000 7febcadb5640 1 -- 172.21.15.161:0/1249218134 shutdown_connections 2024-03-28T23:33:08.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.564+0000 7febcadb5640 1 --2- 172.21.15.161:0/1249218134 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7febb006b500 0x7febb006d9c0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:08.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.565+0000 7febcadb5640 1 --2- 172.21.15.161:0/1249218134 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febc40c30c0 0x7febc41eacc0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:08.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.565+0000 7febcadb5640 1 --2- 172.21.15.161:0/1249218134 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7febc40c19b0 0x7febc4153460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:08.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.565+0000 7febcadb5640 1 --2- 172.21.15.161:0/1249218134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7febc40ac430 0x7febc4155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:08.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.565+0000 7febcadb5640 1 -- 172.21.15.161:0/1249218134 >> 172.21.15.161:0/1249218134 conn(0x7febc406b150 msgr2=0x7febc406b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:08.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.565+0000 7febcadb5640 1 -- 172.21.15.161:0/1249218134 shutdown_connections 2024-03-28T23:33:08.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.565+0000 7febcadb5640 1 -- 172.21.15.161:0/1249218134 wait complete. 2024-03-28T23:33:08.582 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:33:08.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.687+0000 7f126578a640 1 Processor -- start 2024-03-28T23:33:08.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.687+0000 7f126578a640 1 -- start start 2024-03-28T23:33:08.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.687+0000 7f126578a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1260152a80 0x7f1260155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:08.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.687+0000 7f126578a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f126006cb80 0x7f12600c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:08.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.688+0000 7f126578a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f126006c3c0 con 0x7f126006d9f0 2024-03-28T23:33:08.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.688+0000 7f126578a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1260156250 con 0x7f1260152a80 2024-03-28T23:33:08.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.688+0000 7f126578a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f126006c6f0 con 0x7f126006cb80 2024-03-28T23:33:08.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.688+0000 7f125effd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1260152a80 0x7f1260155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:08.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.688+0000 7f125effd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1260152a80 0x7f1260155d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46256/0 (socket says 172.21.15.161:46256) 2024-03-28T23:33:08.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.688+0000 7f125effd640 1 -- 172.21.15.161:0/2498871597 learned_addr learned my addr 172.21.15.161:0/2498871597 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:08.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.688+0000 7f125effd640 1 -- 172.21.15.161:0/2498871597 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f126006cb80 msgr2=0x7f12600c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:08.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.688+0000 7f125effd640 1 --2- 172.21.15.161:0/2498871597 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f126006cb80 0x7f12600c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:08.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.688+0000 7f125effd640 1 -- 172.21.15.161:0/2498871597 >> v1:172.21.15.170:6789/0 conn(0x7f126006d9f0 legacy=0x7f12600c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:08.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.688+0000 7f125effd640 1 -- 172.21.15.161:0/2498871597 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12600c5b90 con 0x7f1260152a80 2024-03-28T23:33:08.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.689+0000 7f125effd640 1 --2- 172.21.15.161:0/2498871597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1260152a80 0x7f1260155d10 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f1250009870 tx=0x7f125002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=553bbfc7b66a25d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:08.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.689+0000 7f125dffb640 1 -- 172.21.15.161:0/2498871597 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1250038070 con 0x7f1260152a80 2024-03-28T23:33:08.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.689+0000 7f125dffb640 1 -- 172.21.15.161:0/2498871597 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1250031740 con 0x7f1260152a80 2024-03-28T23:33:08.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.689+0000 7f125dffb640 1 -- 172.21.15.161:0/2498871597 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125002f050 con 0x7f1260152a80 2024-03-28T23:33:08.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.689+0000 7f126578a640 1 -- 172.21.15.161:0/2498871597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1260152a80 msgr2=0x7f1260155d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:08.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.689+0000 7f126578a640 1 --2- 172.21.15.161:0/2498871597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1260152a80 0x7f1260155d10 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f1250009870 tx=0x7f125002b7e0 comp rx=0 tx=0).stop 2024-03-28T23:33:08.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.689+0000 7f126578a640 1 -- 172.21.15.161:0/2498871597 shutdown_connections 2024-03-28T23:33:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.689+0000 7f126578a640 1 --2- 172.21.15.161:0/2498871597 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f126006cb80 0x7f12600c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.689+0000 7f126578a640 1 --2- 172.21.15.161:0/2498871597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1260152a80 0x7f1260155d10 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.689+0000 7f126578a640 1 -- 172.21.15.161:0/2498871597 >> 172.21.15.161:0/2498871597 conn(0x7f12600ac720 msgr2=0x7f12600acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.689+0000 7f126578a640 1 -- 172.21.15.161:0/2498871597 shutdown_connections 2024-03-28T23:33:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.689+0000 7f126578a640 1 -- 172.21.15.161:0/2498871597 wait complete. 2024-03-28T23:33:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.691+0000 7f126578a640 1 Processor -- start 2024-03-28T23:33:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.691+0000 7f126578a640 1 -- start start 2024-03-28T23:33:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.691+0000 7f126578a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f126006cb80 0x7f1260156e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.691+0000 7f126578a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f126006d9f0 0x7f12600c0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.691+0000 7f126578a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1260152a80 0x7f12601549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.691+0000 7f126578a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f126006d010 con 0x7f126006d9f0 2024-03-28T23:33:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.691+0000 7f126578a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f12600c58b0 con 0x7f126006cb80 2024-03-28T23:33:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.691+0000 7f126578a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f126006d190 con 0x7f1260152a80 2024-03-28T23:33:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.691+0000 7f125e7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f126006d9f0 0x7f12600c0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.691+0000 7f125e7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f126006d9f0 0x7f12600c0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46260/0 (socket says 172.21.15.161:46260) 2024-03-28T23:33:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.691+0000 7f125e7fc640 1 -- 172.21.15.161:0/3547760643 learned_addr learned my addr 172.21.15.161:0/3547760643 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.692+0000 7f125effd640 1 --2- 172.21.15.161:0/3547760643 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f126006cb80 0x7f1260156e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.692+0000 7f125f7fe640 1 --2- 172.21.15.161:0/3547760643 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1260152a80 0x7f12601549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.692+0000 7f125e7fc640 1 -- 172.21.15.161:0/3547760643 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1260152a80 msgr2=0x7f12601549b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.692+0000 7f125e7fc640 1 --2- 172.21.15.161:0/3547760643 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1260152a80 0x7f12601549b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.692+0000 7f125e7fc640 1 -- 172.21.15.161:0/3547760643 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f126006cb80 msgr2=0x7f1260156e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.692+0000 7f125e7fc640 1 --2- 172.21.15.161:0/3547760643 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f126006cb80 0x7f1260156e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.692+0000 7f125e7fc640 1 -- 172.21.15.161:0/3547760643 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12601541f0 con 0x7f126006d9f0 2024-03-28T23:33:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.692+0000 7f125effd640 1 --2- 172.21.15.161:0/3547760643 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f126006cb80 0x7f1260156e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.692+0000 7f125e7fc640 1 --2- 172.21.15.161:0/3547760643 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f126006d9f0 0x7f12600c0d80 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f125400c410 tx=0x7f125400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.692+0000 7f12477fe640 1 -- 172.21.15.161:0/3547760643 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1254017070 con 0x7f126006d9f0 2024-03-28T23:33:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.692+0000 7f12477fe640 1 -- 172.21.15.161:0/3547760643 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12540035f0 con 0x7f126006d9f0 2024-03-28T23:33:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.692+0000 7f12477fe640 1 -- 172.21.15.161:0/3547760643 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1254003a20 con 0x7f126006d9f0 2024-03-28T23:33:08.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.692+0000 7f126578a640 1 -- 172.21.15.161:0/3547760643 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12601eb980 con 0x7f126006d9f0 2024-03-28T23:33:08.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.692+0000 7f126578a640 1 -- 172.21.15.161:0/3547760643 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12601ebc60 con 0x7f126006d9f0 2024-03-28T23:33:08.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.693+0000 7f126578a640 1 -- 172.21.15.161:0/3547760643 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f126004f4b0 con 0x7f126006d9f0 2024-03-28T23:33:08.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.693+0000 7f12477fe640 1 -- 172.21.15.161:0/3547760643 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f125401a070 con 0x7f126006d9f0 2024-03-28T23:33:08.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.694+0000 7f12477fe640 1 --2- 172.21.15.161:0/3547760643 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f124c06b4d0 0x7f124c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:08.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.694+0000 7f12477fe640 1 -- 172.21.15.161:0/3547760643 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 6363+0+0 (secure 0 0 0) 0x7f1254089890 con 0x7f126006d9f0 2024-03-28T23:33:08.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.694+0000 7f125effd640 1 --2- 172.21.15.161:0/3547760643 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f124c06b4d0 0x7f124c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:08.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.694+0000 7f125effd640 1 --2- 172.21.15.161:0/3547760643 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f124c06b4d0 0x7f124c06d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f1250035000 tx=0x7f1250008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:08.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.698+0000 7f12477fe640 1 -- 172.21.15.161:0/3547760643 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1254058510 con 0x7f126006d9f0 2024-03-28T23:33:08.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.869+0000 7f126578a640 1 -- 172.21.15.161:0/3547760643 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f12600c4bb0 con 0x7f126006d9f0 2024-03-28T23:33:08.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.870+0000 7f12477fe640 1 -- 172.21.15.161:0/3547760643 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+17320 (secure 0 0 0) 0x7f12540010c0 con 0x7f126006d9f0 2024-03-28T23:33:08.871 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:08.871 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":82,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:33:08.559271+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:32:42.378609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:32:43.436985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6835/2691084827":"2024-03-29T23:33:06.309901+0000","172.21.15.161:6836/2378263687":"2024-03-29T23:33:05.302530+0000","172.21.15.170:6833/2691084827":"2024-03-29T23:33:06.309901+0000","172.21.15.161:6834/2378263687":"2024-03-29T23:33:05.302530+0000","172.21.15.161:6836/3850309502":"2024-03-29T23:32:54.518520+0000","172.21.15.161:6834/3850309502":"2024-03-29T23:32:54.518520+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:33:08.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.872+0000 7f126578a640 1 -- 172.21.15.161:0/3547760643 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f124c06b4d0 msgr2=0x7f124c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:08.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.872+0000 7f126578a640 1 --2- 172.21.15.161:0/3547760643 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f124c06b4d0 0x7f124c06d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f1250035000 tx=0x7f1250008030 comp rx=0 tx=0).stop 2024-03-28T23:33:08.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.872+0000 7f126578a640 1 -- 172.21.15.161:0/3547760643 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f126006d9f0 msgr2=0x7f12600c0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:08.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.872+0000 7f126578a640 1 --2- 172.21.15.161:0/3547760643 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f126006d9f0 0x7f12600c0d80 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f125400c410 tx=0x7f125400c8d0 comp rx=0 tx=0).stop 2024-03-28T23:33:08.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.872+0000 7f126578a640 1 -- 172.21.15.161:0/3547760643 shutdown_connections 2024-03-28T23:33:08.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.872+0000 7f126578a640 1 --2- 172.21.15.161:0/3547760643 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f124c06b4d0 0x7f124c06d990 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:08.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.872+0000 7f126578a640 1 --2- 172.21.15.161:0/3547760643 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f126006d9f0 0x7f12600c0d80 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:08.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.872+0000 7f126578a640 1 --2- 172.21.15.161:0/3547760643 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1260152a80 0x7f12601549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:08.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.872+0000 7f126578a640 1 --2- 172.21.15.161:0/3547760643 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f126006cb80 0x7f1260156e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:08.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.872+0000 7f126578a640 1 -- 172.21.15.161:0/3547760643 >> 172.21.15.161:0/3547760643 conn(0x7f12600ac720 msgr2=0x7f126014f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:08.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.872+0000 7f126578a640 1 -- 172.21.15.161:0/3547760643 shutdown_connections 2024-03-28T23:33:08.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.872+0000 7f126578a640 1 -- 172.21.15.161:0/3547760643 wait complete. 2024-03-28T23:33:08.884 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-03-28T23:32:42.378609+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs2_data', 'create_time': '2024-03-28T23:32:43.436985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:33:08.884 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:33:08.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.987+0000 7fd7dec40640 1 Processor -- start 2024-03-28T23:33:08.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.987+0000 7fd7dec40640 1 -- start start 2024-03-28T23:33:08.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.988+0000 7fd7dec40640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd7d81529d0 0x7fd7d806bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:08.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.988+0000 7fd7dec40640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd7d80c6750 0x7fd7d80c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:08.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.988+0000 7fd7dec40640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7d8152db0 con 0x7fd7d8155c30 2024-03-28T23:33:08.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.988+0000 7fd7dec40640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd7d80c7210 con 0x7fd7d80c6750 2024-03-28T23:33:08.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.988+0000 7fd7dec40640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd7d80c6c50 con 0x7fd7d81529d0 2024-03-28T23:33:08.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.988+0000 7fd7d7fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd7d80c6750 0x7fd7d80c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:08.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.988+0000 7fd7d7fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd7d80c6750 0x7fd7d80c88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46268/0 (socket says 172.21.15.161:46268) 2024-03-28T23:33:08.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.988+0000 7fd7d7fff640 1 -- 172.21.15.161:0/2829319484 learned_addr learned my addr 172.21.15.161:0/2829319484 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:08.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.989+0000 7fd7dc9dc640 1 --2- 172.21.15.161:0/2829319484 >> v2:172.21.15.170:3300/0 conn(0x7fd7d81529d0 0x7fd7d806bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:08.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.989+0000 7fd7d7fff640 1 -- 172.21.15.161:0/2829319484 >> v2:172.21.15.170:3300/0 conn(0x7fd7d81529d0 msgr2=0x7fd7d806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:08.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.989+0000 7fd7d7fff640 1 --2- 172.21.15.161:0/2829319484 >> v2:172.21.15.170:3300/0 conn(0x7fd7d81529d0 0x7fd7d806bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:08.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.989+0000 7fd7d7fff640 1 -- 172.21.15.161:0/2829319484 >> v1:172.21.15.170:6789/0 conn(0x7fd7d8155c30 legacy=0x7fd7d80c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:08.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.989+0000 7fd7d7fff640 1 -- 172.21.15.161:0/2829319484 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7d80c6f30 con 0x7fd7d80c6750 2024-03-28T23:33:08.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.989+0000 7fd7dc9dc640 1 --2- 172.21.15.161:0/2829319484 >> v2:172.21.15.170:3300/0 conn(0x7fd7d81529d0 0x7fd7d806bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:08.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.989+0000 7fd7d7fff640 1 --2- 172.21.15.161:0/2829319484 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7d80c6750 0x7fd7d80c88b0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fd7d80689e0 tx=0x7fd7c802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddca3b9798f775ee server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:08.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.990+0000 7fd7d77fe640 1 -- 172.21.15.161:0/2829319484 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c8034070 con 0x7fd7d80c6750 2024-03-28T23:33:08.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.990+0000 7fd7d77fe640 1 -- 172.21.15.161:0/2829319484 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7c8030860 con 0x7fd7d80c6750 2024-03-28T23:33:08.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.990+0000 7fd7d77fe640 1 -- 172.21.15.161:0/2829319484 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c8030c30 con 0x7fd7d80c6750 2024-03-28T23:33:08.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.990+0000 7fd7dec40640 1 -- 172.21.15.161:0/2829319484 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7d80c6750 msgr2=0x7fd7d80c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:08.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.990+0000 7fd7dec40640 1 --2- 172.21.15.161:0/2829319484 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7d80c6750 0x7fd7d80c88b0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fd7d80689e0 tx=0x7fd7c802b9a0 comp rx=0 tx=0).stop 2024-03-28T23:33:08.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.990+0000 7fd7dec40640 1 -- 172.21.15.161:0/2829319484 shutdown_connections 2024-03-28T23:33:08.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.990+0000 7fd7dec40640 1 --2- 172.21.15.161:0/2829319484 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7d80c6750 0x7fd7d80c88b0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:08.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.990+0000 7fd7dec40640 1 --2- 172.21.15.161:0/2829319484 >> v2:172.21.15.170:3300/0 conn(0x7fd7d81529d0 0x7fd7d806bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:08.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.990+0000 7fd7dec40640 1 -- 172.21.15.161:0/2829319484 >> 172.21.15.161:0/2829319484 conn(0x7fd7d80c7430 msgr2=0x7fd7d80c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:08.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.990+0000 7fd7dec40640 1 -- 172.21.15.161:0/2829319484 shutdown_connections 2024-03-28T23:33:08.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.990+0000 7fd7dec40640 1 -- 172.21.15.161:0/2829319484 wait complete. 2024-03-28T23:33:08.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.991+0000 7fd7dec40640 1 Processor -- start 2024-03-28T23:33:08.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.991+0000 7fd7dec40640 1 -- start start 2024-03-28T23:33:08.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.992+0000 7fd7dec40640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7d80c6750 0x7fd7d8147cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:08.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.992+0000 7fd7dec40640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7d81529d0 0x7fd7d81e0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:08.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.992+0000 7fd7dec40640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7d8155c30 0x7fd7d81e10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:08.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.992+0000 7fd7dec40640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd7d80aa160 con 0x7fd7d8155c30 2024-03-28T23:33:08.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.992+0000 7fd7dec40640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd7d8155710 con 0x7fd7d80c6750 2024-03-28T23:33:08.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.992+0000 7fd7dec40640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd7d80c5c00 con 0x7fd7d81529d0 2024-03-28T23:33:08.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.992+0000 7fd7dd1dd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7d8155c30 0x7fd7d81e10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:08.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.992+0000 7fd7dd1dd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7d8155c30 0x7fd7d81e10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46272/0 (socket says 172.21.15.161:46272) 2024-03-28T23:33:08.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.992+0000 7fd7dd1dd640 1 -- 172.21.15.161:0/3259307776 learned_addr learned my addr 172.21.15.161:0/3259307776 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:08.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.992+0000 7fd7dc9dc640 1 --2- 172.21.15.161:0/3259307776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7d80c6750 0x7fd7d8147cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:08.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.992+0000 7fd7d7fff640 1 --2- 172.21.15.161:0/3259307776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7d81529d0 0x7fd7d81e0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:08.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.992+0000 7fd7dd1dd640 1 -- 172.21.15.161:0/3259307776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7d81529d0 msgr2=0x7fd7d81e0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:08.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.992+0000 7fd7dd1dd640 1 --2- 172.21.15.161:0/3259307776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7d81529d0 0x7fd7d81e0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:08.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.992+0000 7fd7dd1dd640 1 -- 172.21.15.161:0/3259307776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7d80c6750 msgr2=0x7fd7d8147cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:08.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.992+0000 7fd7dd1dd640 1 --2- 172.21.15.161:0/3259307776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7d80c6750 0x7fd7d8147cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:08.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.993+0000 7fd7dd1dd640 1 -- 172.21.15.161:0/3259307776 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7d81e05c0 con 0x7fd7d8155c30 2024-03-28T23:33:08.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.993+0000 7fd7dd1dd640 1 --2- 172.21.15.161:0/3259307776 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7d8155c30 0x7fd7d81e10b0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fd7c000c3f0 tx=0x7fd7c000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:08.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.993+0000 7fd7d57fa640 1 -- 172.21.15.161:0/3259307776 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c00052d0 con 0x7fd7d8155c30 2024-03-28T23:33:08.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.993+0000 7fd7d57fa640 1 -- 172.21.15.161:0/3259307776 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7c0005490 con 0x7fd7d8155c30 2024-03-28T23:33:08.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.993+0000 7fd7d57fa640 1 -- 172.21.15.161:0/3259307776 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c0007260 con 0x7fd7d8155c30 2024-03-28T23:33:08.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.993+0000 7fd7dec40640 1 -- 172.21.15.161:0/3259307776 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7d8145c20 con 0x7fd7d8155c30 2024-03-28T23:33:08.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.993+0000 7fd7dec40640 1 -- 172.21.15.161:0/3259307776 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7d81dec20 con 0x7fd7d8155c30 2024-03-28T23:33:08.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.994+0000 7fd7dec40640 1 -- 172.21.15.161:0/3259307776 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7d804f4b0 con 0x7fd7d8155c30 2024-03-28T23:33:08.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.995+0000 7fd7d57fa640 1 -- 172.21.15.161:0/3259307776 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd7c0007400 con 0x7fd7d8155c30 2024-03-28T23:33:08.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.995+0000 7fd7d57fa640 1 --2- 172.21.15.161:0/3259307776 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd7b806b4d0 0x7fd7b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:08.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.995+0000 7fd7d57fa640 1 -- 172.21.15.161:0/3259307776 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 6363+0+0 (secure 0 0 0) 0x7fd7c00897e0 con 0x7fd7d8155c30 2024-03-28T23:33:08.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.995+0000 7fd7dc9dc640 1 --2- 172.21.15.161:0/3259307776 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd7b806b4d0 0x7fd7b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:08.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:08.995+0000 7fd7dc9dc640 1 --2- 172.21.15.161:0/3259307776 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd7b806b4d0 0x7fd7b806d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8150fb0 tx=0x7fd7d0009890 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:09.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.000+0000 7fd7d57fa640 1 -- 172.21.15.161:0/3259307776 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd7c0057e10 con 0x7fd7d8155c30 2024-03-28T23:33:09.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.176+0000 7fd7dec40640 1 -- 172.21.15.161:0/3259307776 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd7d80ca650 con 0x7fd7d8155c30 2024-03-28T23:33:09.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.176+0000 7fd7d57fa640 1 -- 172.21.15.161:0/3259307776 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7fd7c005beb0 con 0x7fd7d8155c30 2024-03-28T23:33:09.177 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:33:09.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.178+0000 7fd7dec40640 1 -- 172.21.15.161:0/3259307776 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd7b806b4d0 msgr2=0x7fd7b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:09.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.178+0000 7fd7dec40640 1 --2- 172.21.15.161:0/3259307776 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd7b806b4d0 0x7fd7b806d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8150fb0 tx=0x7fd7d0009890 comp rx=0 tx=0).stop 2024-03-28T23:33:09.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.178+0000 7fd7dec40640 1 -- 172.21.15.161:0/3259307776 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7d8155c30 msgr2=0x7fd7d81e10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:09.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.178+0000 7fd7dec40640 1 --2- 172.21.15.161:0/3259307776 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7d8155c30 0x7fd7d81e10b0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fd7c000c3f0 tx=0x7fd7c000c8b0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.178+0000 7fd7dec40640 1 -- 172.21.15.161:0/3259307776 shutdown_connections 2024-03-28T23:33:09.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.178+0000 7fd7dec40640 1 --2- 172.21.15.161:0/3259307776 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd7b806b4d0 0x7fd7b806d990 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.178+0000 7fd7dec40640 1 --2- 172.21.15.161:0/3259307776 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7d8155c30 0x7fd7d81e10b0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.178+0000 7fd7dec40640 1 --2- 172.21.15.161:0/3259307776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7d81529d0 0x7fd7d81e0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.178+0000 7fd7dec40640 1 --2- 172.21.15.161:0/3259307776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7d80c6750 0x7fd7d8147cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.178+0000 7fd7dec40640 1 -- 172.21.15.161:0/3259307776 >> 172.21.15.161:0/3259307776 conn(0x7fd7d80c7430 msgr2=0x7fd7d8151d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:09.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.178+0000 7fd7dec40640 1 -- 172.21.15.161:0/3259307776 shutdown_connections 2024-03-28T23:33:09.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.178+0000 7fd7dec40640 1 -- 172.21.15.161:0/3259307776 wait complete. 2024-03-28T23:33:09.190 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-03-28T23:33:09.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.294+0000 7f2b0b00b640 1 Processor -- start 2024-03-28T23:33:09.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.294+0000 7f2b0b00b640 1 -- start start 2024-03-28T23:33:09.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.295+0000 7f2b0b00b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2b041506b0 0x7f2b040c1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:09.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.295+0000 7f2b0b00b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b040a7d00 0x7f2b040a96f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:09.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.295+0000 7f2b0b00b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b040a9c30 con 0x7f2b04153840 2024-03-28T23:33:09.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.295+0000 7f2b0b00b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f2b040c0d80 con 0x7f2b041506b0 2024-03-28T23:33:09.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.295+0000 7f2b0b00b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2b040c28e0 con 0x7f2b040a7d00 2024-03-28T23:33:09.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.295+0000 7f2b095a8640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f2b04153840 0x7f2b040ab260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44922/0 (socket says 172.21.15.161:44922) 2024-03-28T23:33:09.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.295+0000 7f2b095a8640 1 -- 172.21.15.161:0/1086424993 learned_addr learned my addr 172.21.15.161:0/1086424993 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:09.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.295+0000 7f2b08da7640 1 --2- 172.21.15.161:0/1086424993 >> v2:172.21.15.170:3300/0 conn(0x7f2b041506b0 0x7f2b040c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:09.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.295+0000 7f2b03fff640 1 --2- 172.21.15.161:0/1086424993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b040a7d00 0x7f2b040a96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:09.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.295+0000 7f2b037fe640 1 -- 172.21.15.161:0/1086424993 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1855281690 0 0) 0x7f2b040a9c30 con 0x7f2b04153840 2024-03-28T23:33:09.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.295+0000 7f2b037fe640 1 -- 172.21.15.161:0/1086424993 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2af4003520 con 0x7f2b04153840 2024-03-28T23:33:09.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.295+0000 7f2b08da7640 1 -- 172.21.15.161:0/1086424993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b040a7d00 msgr2=0x7f2b040a96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:09.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.295+0000 7f2b08da7640 1 --2- 172.21.15.161:0/1086424993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b040a7d00 0x7f2b040a96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.295+0000 7f2b037fe640 1 -- 172.21.15.161:0/1086424993 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2507060793 0 0) 0x7f2af4003520 con 0x7f2b04153840 2024-03-28T23:33:09.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.295+0000 7f2b08da7640 1 -- 172.21.15.161:0/1086424993 >> v1:172.21.15.161:6789/0 conn(0x7f2b04153840 legacy=0x7f2b040ab260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:09.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.295+0000 7f2b08da7640 1 -- 172.21.15.161:0/1086424993 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b040c2bc0 con 0x7f2b041506b0 2024-03-28T23:33:09.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.296+0000 7f2b08da7640 1 --2- 172.21.15.161:0/1086424993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b041506b0 0x7f2b040c1040 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2af80097a0 tx=0x7f2af802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36a568ce7072b28b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:09.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.296+0000 7f2b037fe640 1 -- 172.21.15.161:0/1086424993 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af8037070 con 0x7f2b041506b0 2024-03-28T23:33:09.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.296+0000 7f2b037fe640 1 -- 172.21.15.161:0/1086424993 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2af8030510 con 0x7f2b041506b0 2024-03-28T23:33:09.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.296+0000 7f2b037fe640 1 -- 172.21.15.161:0/1086424993 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc0019f0 con 0x7f2b041506b0 2024-03-28T23:33:09.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.297+0000 7f2b0b00b640 1 -- 172.21.15.161:0/1086424993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b041506b0 msgr2=0x7f2b040c1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:09.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.297+0000 7f2b0b00b640 1 --2- 172.21.15.161:0/1086424993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b041506b0 0x7f2b040c1040 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2af80097a0 tx=0x7f2af802b710 comp rx=0 tx=0).stop 2024-03-28T23:33:09.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.297+0000 7f2b0b00b640 1 -- 172.21.15.161:0/1086424993 shutdown_connections 2024-03-28T23:33:09.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.297+0000 7f2b0b00b640 1 --2- 172.21.15.161:0/1086424993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b040a7d00 0x7f2b040a96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.297+0000 7f2b0b00b640 1 --2- 172.21.15.161:0/1086424993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b041506b0 0x7f2b040c1040 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.297+0000 7f2b0b00b640 1 -- 172.21.15.161:0/1086424993 >> 172.21.15.161:0/1086424993 conn(0x7f2b040a83f0 msgr2=0x7f2b040a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:09.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.297+0000 7f2b0b00b640 1 -- 172.21.15.161:0/1086424993 shutdown_connections 2024-03-28T23:33:09.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.297+0000 7f2b0b00b640 1 -- 172.21.15.161:0/1086424993 wait complete. 2024-03-28T23:33:09.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.298+0000 7f2b0b00b640 1 Processor -- start 2024-03-28T23:33:09.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.298+0000 7f2b0b00b640 1 -- start start 2024-03-28T23:33:09.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.299+0000 7f2b0b00b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b040a7d00 0x7f2b04152140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:09.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.299+0000 7f2b0b00b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b041506b0 0x7f2b04144280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:09.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.299+0000 7f2b0b00b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b04153840 0x7f2b040c6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:09.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.299+0000 7f2b0b00b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2b040c2ea0 con 0x7f2b040a7d00 2024-03-28T23:33:09.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.299+0000 7f2b0b00b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2b04153320 con 0x7f2b041506b0 2024-03-28T23:33:09.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.299+0000 7f2b0b00b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2b040ab970 con 0x7f2b04153840 2024-03-28T23:33:09.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.299+0000 7f2b08da7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b040a7d00 0x7f2b04152140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:09.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.299+0000 7f2b08da7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b040a7d00 0x7f2b04152140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46282/0 (socket says 172.21.15.161:46282) 2024-03-28T23:33:09.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.299+0000 7f2b08da7640 1 -- 172.21.15.161:0/2787492208 learned_addr learned my addr 172.21.15.161:0/2787492208 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:09.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.299+0000 7f2b03fff640 1 --2- 172.21.15.161:0/2787492208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b041506b0 0x7f2b04144280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:09.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.299+0000 7f2b095a8640 1 --2- 172.21.15.161:0/2787492208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b04153840 0x7f2b040c6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:09.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.299+0000 7f2b08da7640 1 -- 172.21.15.161:0/2787492208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b04153840 msgr2=0x7f2b040c6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:09.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.299+0000 7f2b08da7640 1 --2- 172.21.15.161:0/2787492208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b04153840 0x7f2b040c6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.299+0000 7f2b08da7640 1 -- 172.21.15.161:0/2787492208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b041506b0 msgr2=0x7f2b04144280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:09.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.299+0000 7f2b08da7640 1 --2- 172.21.15.161:0/2787492208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b041506b0 0x7f2b04144280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.299+0000 7f2b08da7640 1 -- 172.21.15.161:0/2787492208 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b04152870 con 0x7f2b040a7d00 2024-03-28T23:33:09.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.299+0000 7f2b08da7640 1 --2- 172.21.15.161:0/2787492208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b040a7d00 0x7f2b04152140 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f2af802b750 tx=0x7f2af80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:09.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.299+0000 7f2b017fa640 1 -- 172.21.15.161:0/2787492208 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af8041020 con 0x7f2b040a7d00 2024-03-28T23:33:09.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.299+0000 7f2b017fa640 1 -- 172.21.15.161:0/2787492208 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2af803c870 con 0x7f2b040a7d00 2024-03-28T23:33:09.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.299+0000 7f2b0b00b640 1 -- 172.21.15.161:0/2787492208 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b04152b50 con 0x7f2b040a7d00 2024-03-28T23:33:09.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.299+0000 7f2b017fa640 1 -- 172.21.15.161:0/2787492208 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af8037040 con 0x7f2b040a7d00 2024-03-28T23:33:09.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.299+0000 7f2b0b00b640 1 -- 172.21.15.161:0/2787492208 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b04151940 con 0x7f2b040a7d00 2024-03-28T23:33:09.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.300+0000 7f2b0b00b640 1 -- 172.21.15.161:0/2787492208 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b0404f4b0 con 0x7f2b040a7d00 2024-03-28T23:33:09.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.301+0000 7f2b017fa640 1 -- 172.21.15.161:0/2787492208 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2af8044020 con 0x7f2b040a7d00 2024-03-28T23:33:09.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.301+0000 7f2b017fa640 1 --2- 172.21.15.161:0/2787492208 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2ae806b4d0 0x7f2ae806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:09.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.301+0000 7f2b03fff640 1 --2- 172.21.15.161:0/2787492208 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2ae806b4d0 0x7f2ae806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:09.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.302+0000 7f2b017fa640 1 -- 172.21.15.161:0/2787492208 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 6363+0+0 (secure 0 0 0) 0x7f2af80abd30 con 0x7f2b040a7d00 2024-03-28T23:33:09.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.302+0000 7f2b03fff640 1 --2- 172.21.15.161:0/2787492208 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2ae806b4d0 0x7f2ae806d990 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2b041ecf00 tx=0x7f2af0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:09.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.306+0000 7f2b017fa640 1 -- 172.21.15.161:0/2787492208 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2af804b030 con 0x7f2b040a7d00 2024-03-28T23:33:09.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.576+0000 7f2b0b00b640 1 -- 172.21.15.161:0/2787492208 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) -- 0x7f2b040c2620 con 0x7f2b040a7d00 2024-03-28T23:33:09.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.577+0000 7f2b017fa640 1 -- 172.21.15.161:0/2787492208 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v82) ==== 105+0+11 (secure 0 0 0) 0x7f2af807a360 con 0x7f2b040a7d00 2024-03-28T23:33:09.577 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:33:09.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.579+0000 7f2b0b00b640 1 -- 172.21.15.161:0/2787492208 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2ae806b4d0 msgr2=0x7f2ae806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:09.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.579+0000 7f2b0b00b640 1 --2- 172.21.15.161:0/2787492208 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2ae806b4d0 0x7f2ae806d990 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2b041ecf00 tx=0x7f2af0009190 comp rx=0 tx=0).stop 2024-03-28T23:33:09.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.579+0000 7f2b0b00b640 1 -- 172.21.15.161:0/2787492208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b040a7d00 msgr2=0x7f2b04152140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:09.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.579+0000 7f2b0b00b640 1 --2- 172.21.15.161:0/2787492208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b040a7d00 0x7f2b04152140 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f2af802b750 tx=0x7f2af80046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.579+0000 7f2b0b00b640 1 -- 172.21.15.161:0/2787492208 shutdown_connections 2024-03-28T23:33:09.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.579+0000 7f2b0b00b640 1 --2- 172.21.15.161:0/2787492208 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2ae806b4d0 0x7f2ae806d990 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.579+0000 7f2b0b00b640 1 --2- 172.21.15.161:0/2787492208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b04153840 0x7f2b040c6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.579+0000 7f2b0b00b640 1 --2- 172.21.15.161:0/2787492208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b041506b0 0x7f2b04144280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.579+0000 7f2b0b00b640 1 --2- 172.21.15.161:0/2787492208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b040a7d00 0x7f2b04152140 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.579+0000 7f2b0b00b640 1 -- 172.21.15.161:0/2787492208 >> 172.21.15.161:0/2787492208 conn(0x7f2b040a83f0 msgr2=0x7f2b0414f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:09.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.579+0000 7f2b0b00b640 1 -- 172.21.15.161:0/2787492208 shutdown_connections 2024-03-28T23:33:09.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.579+0000 7f2b0b00b640 1 -- 172.21.15.161:0/2787492208 wait complete. 2024-03-28T23:33:09.590 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-03-28T23:33:09.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.695+0000 7fe52439a640 1 Processor -- start 2024-03-28T23:33:09.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.695+0000 7fe52439a640 1 -- start start 2024-03-28T23:33:09.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.696+0000 7fe52439a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe51c06f5b0 0x7fe51c0c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:09.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.696+0000 7fe52439a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe51c06e7f0 0x7fe51c06ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:09.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.696+0000 7fe52439a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe51c0c7490 con 0x7fe51c152a70 2024-03-28T23:33:09.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.696+0000 7fe52439a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe51c0c7260 con 0x7fe51c06f5b0 2024-03-28T23:33:09.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.696+0000 7fe52439a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe51c0c5cf0 con 0x7fe51c06e7f0 2024-03-28T23:33:09.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.696+0000 7fe521935640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe51c06f5b0 0x7fe51c0c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:09.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.696+0000 7fe521935640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe51c06f5b0 0x7fe51c0c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46286/0 (socket says 172.21.15.161:46286) 2024-03-28T23:33:09.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.696+0000 7fe521935640 1 -- 172.21.15.161:0/1788677327 learned_addr learned my addr 172.21.15.161:0/1788677327 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:09.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.696+0000 7fe522937640 1 --2- 172.21.15.161:0/1788677327 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe51c06e7f0 0x7fe51c06ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:09.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.696+0000 7fe521935640 1 -- 172.21.15.161:0/1788677327 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe51c06e7f0 msgr2=0x7fe51c06ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:09.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.696+0000 7fe521935640 1 --2- 172.21.15.161:0/1788677327 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe51c06e7f0 0x7fe51c06ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.696+0000 7fe521935640 1 -- 172.21.15.161:0/1788677327 >> v1:172.21.15.170:6789/0 conn(0x7fe51c152a70 legacy=0x7fe51c06b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:09.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.696+0000 7fe521935640 1 -- 172.21.15.161:0/1788677327 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51c0c5a10 con 0x7fe51c06f5b0 2024-03-28T23:33:09.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.696+0000 7fe522937640 1 --2- 172.21.15.161:0/1788677327 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe51c06e7f0 0x7fe51c06ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:09.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.697+0000 7fe521935640 1 --2- 172.21.15.161:0/1788677327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe51c06f5b0 0x7fe51c0c1710 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fe518009870 tx=0x7fe51802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5fbc7e1d213e9b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:09.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.697+0000 7fe521134640 1 -- 172.21.15.161:0/1788677327 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe518038070 con 0x7fe51c06f5b0 2024-03-28T23:33:09.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.697+0000 7fe521134640 1 -- 172.21.15.161:0/1788677327 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe518031740 con 0x7fe51c06f5b0 2024-03-28T23:33:09.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.697+0000 7fe521134640 1 -- 172.21.15.161:0/1788677327 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51802f050 con 0x7fe51c06f5b0 2024-03-28T23:33:09.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.698+0000 7fe52439a640 1 -- 172.21.15.161:0/1788677327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe51c06f5b0 msgr2=0x7fe51c0c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:09.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.698+0000 7fe52439a640 1 --2- 172.21.15.161:0/1788677327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe51c06f5b0 0x7fe51c0c1710 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fe518009870 tx=0x7fe51802b7e0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.698+0000 7fe52439a640 1 -- 172.21.15.161:0/1788677327 shutdown_connections 2024-03-28T23:33:09.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.698+0000 7fe52439a640 1 --2- 172.21.15.161:0/1788677327 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe51c06e7f0 0x7fe51c06ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.698+0000 7fe52439a640 1 --2- 172.21.15.161:0/1788677327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe51c06f5b0 0x7fe51c0c1710 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.698+0000 7fe52439a640 1 -- 172.21.15.161:0/1788677327 >> 172.21.15.161:0/1788677327 conn(0x7fe51c0c7990 msgr2=0x7fe51c0c7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:09.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.698+0000 7fe52439a640 1 -- 172.21.15.161:0/1788677327 shutdown_connections 2024-03-28T23:33:09.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.698+0000 7fe52439a640 1 -- 172.21.15.161:0/1788677327 wait complete. 2024-03-28T23:33:09.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.699+0000 7fe52439a640 1 Processor -- start 2024-03-28T23:33:09.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.699+0000 7fe52439a640 1 -- start start 2024-03-28T23:33:09.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.700+0000 7fe52439a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe51c06e7f0 0x7fe51c1566a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:09.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.700+0000 7fe52439a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe51c06f5b0 0x7fe51c0aca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:09.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.700+0000 7fe52439a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe51c152a70 0x7fe51c0acf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:09.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.700+0000 7fe52439a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c06b860 con 0x7fe51c152a70 2024-03-28T23:33:09.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.700+0000 7fe52439a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c0c0ef0 con 0x7fe51c06e7f0 2024-03-28T23:33:09.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.700+0000 7fe52439a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe51c06b9e0 con 0x7fe51c06f5b0 2024-03-28T23:33:09.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.701+0000 7fe522937640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe51c152a70 0x7fe51c0acf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:09.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.701+0000 7fe522937640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe51c152a70 0x7fe51c0acf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46290/0 (socket says 172.21.15.161:46290) 2024-03-28T23:33:09.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.701+0000 7fe522937640 1 -- 172.21.15.161:0/3865257689 learned_addr learned my addr 172.21.15.161:0/3865257689 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:09.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.701+0000 7fe521935640 1 --2- 172.21.15.161:0/3865257689 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe51c06f5b0 0x7fe51c0aca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:09.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.701+0000 7fe522136640 1 --2- 172.21.15.161:0/3865257689 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe51c06e7f0 0x7fe51c1566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:09.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.701+0000 7fe522937640 1 -- 172.21.15.161:0/3865257689 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe51c06f5b0 msgr2=0x7fe51c0aca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:09.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.701+0000 7fe522937640 1 --2- 172.21.15.161:0/3865257689 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe51c06f5b0 0x7fe51c0aca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.701+0000 7fe522937640 1 -- 172.21.15.161:0/3865257689 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe51c06e7f0 msgr2=0x7fe51c1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:09.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.701+0000 7fe522937640 1 --2- 172.21.15.161:0/3865257689 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe51c06e7f0 0x7fe51c1566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.701+0000 7fe522937640 1 -- 172.21.15.161:0/3865257689 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51c14e130 con 0x7fe51c152a70 2024-03-28T23:33:09.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.701+0000 7fe522136640 1 --2- 172.21.15.161:0/3865257689 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe51c06e7f0 0x7fe51c1566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:09.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.701+0000 7fe522937640 1 --2- 172.21.15.161:0/3865257689 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe51c152a70 0x7fe51c0acf90 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fe51400cd50 tx=0x7fe51400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:09.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.701+0000 7fe512ffd640 1 -- 172.21.15.161:0/3865257689 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51400f680 con 0x7fe51c152a70 2024-03-28T23:33:09.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.701+0000 7fe512ffd640 1 -- 172.21.15.161:0/3865257689 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe51400f840 con 0x7fe51c152a70 2024-03-28T23:33:09.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.701+0000 7fe52439a640 1 -- 172.21.15.161:0/3865257689 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe51c1e1770 con 0x7fe51c152a70 2024-03-28T23:33:09.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.701+0000 7fe512ffd640 1 -- 172.21.15.161:0/3865257689 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51400fc30 con 0x7fe51c152a70 2024-03-28T23:33:09.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.701+0000 7fe52439a640 1 -- 172.21.15.161:0/3865257689 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe51c14e4e0 con 0x7fe51c152a70 2024-03-28T23:33:09.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.702+0000 7fe52439a640 1 -- 172.21.15.161:0/3865257689 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe51c04f4b0 con 0x7fe51c152a70 2024-03-28T23:33:09.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.703+0000 7fe512ffd640 1 -- 172.21.15.161:0/3865257689 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe5140213f0 con 0x7fe51c152a70 2024-03-28T23:33:09.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.703+0000 7fe512ffd640 1 --2- 172.21.15.161:0/3865257689 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe50006b4d0 0x7fe50006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:09.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.703+0000 7fe512ffd640 1 -- 172.21.15.161:0/3865257689 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 6363+0+0 (secure 0 0 0) 0x7fe51408b1d0 con 0x7fe51c152a70 2024-03-28T23:33:09.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.703+0000 7fe522136640 1 --2- 172.21.15.161:0/3865257689 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe50006b4d0 0x7fe50006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:09.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.704+0000 7fe522136640 1 --2- 172.21.15.161:0/3865257689 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe50006b4d0 0x7fe50006d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fe51c154c10 tx=0x7fe50c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:09.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.708+0000 7fe512ffd640 1 -- 172.21.15.161:0/3865257689 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe514014030 con 0x7fe51c152a70 2024-03-28T23:33:09.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.960+0000 7fe52439a640 1 -- 172.21.15.161:0/3865257689 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) -- 0x7fe51c0c16f0 con 0x7fe51c152a70 2024-03-28T23:33:09.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.960+0000 7fe512ffd640 1 -- 172.21.15.161:0/3865257689 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v82) ==== 101+0+11 (secure 0 0 0) 0x7fe514059e50 con 0x7fe51c152a70 2024-03-28T23:33:09.961 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:33:09.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.962+0000 7fe52439a640 1 -- 172.21.15.161:0/3865257689 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe50006b4d0 msgr2=0x7fe50006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:09.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.962+0000 7fe52439a640 1 --2- 172.21.15.161:0/3865257689 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe50006b4d0 0x7fe50006d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fe51c154c10 tx=0x7fe50c009040 comp rx=0 tx=0).stop 2024-03-28T23:33:09.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.962+0000 7fe52439a640 1 -- 172.21.15.161:0/3865257689 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe51c152a70 msgr2=0x7fe51c0acf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:09.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.962+0000 7fe52439a640 1 --2- 172.21.15.161:0/3865257689 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe51c152a70 0x7fe51c0acf90 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fe51400cd50 tx=0x7fe51400a650 comp rx=0 tx=0).stop 2024-03-28T23:33:09.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.962+0000 7fe52439a640 1 -- 172.21.15.161:0/3865257689 shutdown_connections 2024-03-28T23:33:09.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.962+0000 7fe52439a640 1 --2- 172.21.15.161:0/3865257689 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe50006b4d0 0x7fe50006d990 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.962+0000 7fe52439a640 1 --2- 172.21.15.161:0/3865257689 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe51c152a70 0x7fe51c0acf90 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.962+0000 7fe52439a640 1 --2- 172.21.15.161:0/3865257689 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe51c06f5b0 0x7fe51c0aca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.962+0000 7fe52439a640 1 --2- 172.21.15.161:0/3865257689 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe51c06e7f0 0x7fe51c1566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:09.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.962+0000 7fe52439a640 1 -- 172.21.15.161:0/3865257689 >> 172.21.15.161:0/3865257689 conn(0x7fe51c0c7990 msgr2=0x7fe51c150e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:09.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.962+0000 7fe52439a640 1 -- 172.21.15.161:0/3865257689 shutdown_connections 2024-03-28T23:33:09.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:09.962+0000 7fe52439a640 1 -- 172.21.15.161:0/3865257689 wait complete. 2024-03-28T23:33:09.974 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:33:10.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.075+0000 7fd168d8a640 1 Processor -- start 2024-03-28T23:33:10.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.075+0000 7fd168d8a640 1 -- start start 2024-03-28T23:33:10.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.076+0000 7fd168d8a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd1640c5410 0x7fd1640c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:10.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.076+0000 7fd168d8a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd164158a30 con 0x7fd1641553a0 2024-03-28T23:33:10.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.076+0000 7fd168d8a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd164158d10 con 0x7fd164152890 2024-03-28T23:33:10.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.076+0000 7fd168d8a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd1640c8480 con 0x7fd1640c5410 2024-03-28T23:33:10.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.076+0000 7fd162d9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd1641553a0 0x7fd164155850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44924/0 (socket says 172.21.15.161:44924) 2024-03-28T23:33:10.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.076+0000 7fd162d9d640 1 -- 172.21.15.161:0/354879080 learned_addr learned my addr 172.21.15.161:0/354879080 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:10.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.076+0000 7fd161d9b640 1 --2- 172.21.15.161:0/354879080 >> v2:172.21.15.170:3300/0 conn(0x7fd1640c5410 0x7fd1640c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:10.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.076+0000 7fd16159a640 1 -- 172.21.15.161:0/354879080 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3279641448 0 0) 0x7fd164158a30 con 0x7fd1641553a0 2024-03-28T23:33:10.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.076+0000 7fd16159a640 1 -- 172.21.15.161:0/354879080 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd14c003520 con 0x7fd1641553a0 2024-03-28T23:33:10.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.076+0000 7fd161d9b640 1 -- 172.21.15.161:0/354879080 >> v1:172.21.15.170:6789/0 conn(0x7fd164152890 legacy=0x7fd1640c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:10.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.076+0000 7fd161d9b640 1 -- 172.21.15.161:0/354879080 >> v1:172.21.15.161:6789/0 conn(0x7fd1641553a0 legacy=0x7fd164155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:10.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.076+0000 7fd161d9b640 1 -- 172.21.15.161:0/354879080 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd164158ff0 con 0x7fd1640c5410 2024-03-28T23:33:10.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.077+0000 7fd161d9b640 1 --2- 172.21.15.161:0/354879080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1640c5410 0x7fd1640c5850 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd158009790 tx=0x7fd15802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fd432ef4969b738 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:10.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.077+0000 7fd16159a640 1 -- 172.21.15.161:0/354879080 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd158035020 con 0x7fd1640c5410 2024-03-28T23:33:10.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.077+0000 7fd16159a640 1 -- 172.21.15.161:0/354879080 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd158031830 con 0x7fd1640c5410 2024-03-28T23:33:10.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.077+0000 7fd16159a640 1 -- 172.21.15.161:0/354879080 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd158031c00 con 0x7fd1640c5410 2024-03-28T23:33:10.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.078+0000 7fd168d8a640 1 -- 172.21.15.161:0/354879080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1640c5410 msgr2=0x7fd1640c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:10.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.078+0000 7fd168d8a640 1 --2- 172.21.15.161:0/354879080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1640c5410 0x7fd1640c5850 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd158009790 tx=0x7fd15802b8d0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.078+0000 7fd168d8a640 1 -- 172.21.15.161:0/354879080 shutdown_connections 2024-03-28T23:33:10.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.078+0000 7fd168d8a640 1 --2- 172.21.15.161:0/354879080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1640c5410 0x7fd1640c5850 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.078+0000 7fd168d8a640 1 -- 172.21.15.161:0/354879080 >> 172.21.15.161:0/354879080 conn(0x7fd1640ac620 msgr2=0x7fd1640aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:10.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.078+0000 7fd168d8a640 1 -- 172.21.15.161:0/354879080 shutdown_connections 2024-03-28T23:33:10.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.078+0000 7fd168d8a640 1 -- 172.21.15.161:0/354879080 wait complete. 2024-03-28T23:33:10.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.079+0000 7fd168d8a640 1 Processor -- start 2024-03-28T23:33:10.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.079+0000 7fd168d8a640 1 -- start start 2024-03-28T23:33:10.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.080+0000 7fd168d8a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1640c5410 0x7fd16414afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:10.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.080+0000 7fd168d8a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd164152890 0x7fd1640ab700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:10.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.080+0000 7fd168d8a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1641553a0 0x7fd1641e7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:10.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.080+0000 7fd168d8a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd164159640 con 0x7fd164152890 2024-03-28T23:33:10.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.080+0000 7fd168d8a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd1640c5d90 con 0x7fd1640c5410 2024-03-28T23:33:10.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.080+0000 7fd168d8a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd1640c90a0 con 0x7fd1641553a0 2024-03-28T23:33:10.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.080+0000 7fd161d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd164152890 0x7fd1640ab700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:10.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.080+0000 7fd161d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd164152890 0x7fd1640ab700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46294/0 (socket says 172.21.15.161:46294) 2024-03-28T23:33:10.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.080+0000 7fd161d9b640 1 -- 172.21.15.161:0/155620265 learned_addr learned my addr 172.21.15.161:0/155620265 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:10.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.080+0000 7fd16259c640 1 --2- 172.21.15.161:0/155620265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1640c5410 0x7fd16414afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:10.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.080+0000 7fd162d9d640 1 --2- 172.21.15.161:0/155620265 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1641553a0 0x7fd1641e7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:10.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.080+0000 7fd161d9b640 1 -- 172.21.15.161:0/155620265 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1641553a0 msgr2=0x7fd1641e7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:10.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.080+0000 7fd161d9b640 1 --2- 172.21.15.161:0/155620265 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1641553a0 0x7fd1641e7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.080+0000 7fd161d9b640 1 -- 172.21.15.161:0/155620265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1640c5410 msgr2=0x7fd16414afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:10.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.080+0000 7fd161d9b640 1 --2- 172.21.15.161:0/155620265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1640c5410 0x7fd16414afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.081+0000 7fd161d9b640 1 -- 172.21.15.161:0/155620265 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1641eadc0 con 0x7fd164152890 2024-03-28T23:33:10.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.081+0000 7fd161d9b640 1 --2- 172.21.15.161:0/155620265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd164152890 0x7fd1640ab700 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fd158034040 tx=0x7fd1580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:10.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.081+0000 7fd152ffd640 1 -- 172.21.15.161:0/155620265 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd158040070 con 0x7fd164152890 2024-03-28T23:33:10.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.081+0000 7fd152ffd640 1 -- 172.21.15.161:0/155620265 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd15803e070 con 0x7fd164152890 2024-03-28T23:33:10.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.081+0000 7fd168d8a640 1 -- 172.21.15.161:0/155620265 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1641eb0a0 con 0x7fd164152890 2024-03-28T23:33:10.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.081+0000 7fd152ffd640 1 -- 172.21.15.161:0/155620265 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd158035040 con 0x7fd164152890 2024-03-28T23:33:10.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.081+0000 7fd168d8a640 1 -- 172.21.15.161:0/155620265 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1641eb660 con 0x7fd164152890 2024-03-28T23:33:10.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.082+0000 7fd168d8a640 1 -- 172.21.15.161:0/155620265 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd16404f4b0 con 0x7fd164152890 2024-03-28T23:33:10.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.082+0000 7fd152ffd640 1 -- 172.21.15.161:0/155620265 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd158038020 con 0x7fd164152890 2024-03-28T23:33:10.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.083+0000 7fd152ffd640 1 --2- 172.21.15.161:0/155620265 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd14406b500 0x7fd14406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:10.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.083+0000 7fd16259c640 1 --2- 172.21.15.161:0/155620265 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd14406b500 0x7fd14406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:10.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.083+0000 7fd152ffd640 1 -- 172.21.15.161:0/155620265 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 6363+0+0 (secure 0 0 0) 0x7fd1580abb00 con 0x7fd164152890 2024-03-28T23:33:10.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.083+0000 7fd16259c640 1 --2- 172.21.15.161:0/155620265 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd14406b500 0x7fd14406d9c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd1540016e0 tx=0x7fd154009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:10.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.087+0000 7fd152ffd640 1 -- 172.21.15.161:0/155620265 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd158031800 con 0x7fd164152890 2024-03-28T23:33:10.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.296+0000 7fd168d8a640 1 -- 172.21.15.161:0/155620265 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1640cb710 con 0x7fd164152890 2024-03-28T23:33:10.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.297+0000 7fd152ffd640 1 -- 172.21.15.161:0/155620265 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+5193 (secure 0 0 0) 0x7fd15807a130 con 0x7fd164152890 2024-03-28T23:33:10.298 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:10.298 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":73,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5579,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.170:6834/3501962065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3501962065},{"type":"v1","addr":"172.21.15.170:6834","nonce":3501962065}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":60},{"gid":6397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2706065827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":2706065827},{"type":"v1","addr":"172.21.15.170:6835","nonce":2706065827}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":6903,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/3336491043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3336491043},{"type":"v1","addr":"172.21.15.161:6836","nonce":3336491043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:32:44.463287+0000","modified":"2024-03-28T23:33:09.314639+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5570":{"gid":5570,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":13,"addr":"172.21.15.161:6837/737681942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":737681942},{"type":"v1","addr":"172.21.15.161:6837","nonce":737681942}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5570,"qdb_cluster":[5570]},"id":6}]} 2024-03-28T23:33:10.298 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 73 2024-03-28T23:33:10.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.299+0000 7fd168d8a640 1 -- 172.21.15.161:0/155620265 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd14406b500 msgr2=0x7fd14406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:10.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.299+0000 7fd168d8a640 1 --2- 172.21.15.161:0/155620265 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd14406b500 0x7fd14406d9c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd1540016e0 tx=0x7fd154009040 comp rx=0 tx=0).stop 2024-03-28T23:33:10.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.299+0000 7fd168d8a640 1 -- 172.21.15.161:0/155620265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd164152890 msgr2=0x7fd1640ab700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:10.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.299+0000 7fd168d8a640 1 --2- 172.21.15.161:0/155620265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd164152890 0x7fd1640ab700 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fd158034040 tx=0x7fd1580046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.299+0000 7fd168d8a640 1 -- 172.21.15.161:0/155620265 shutdown_connections 2024-03-28T23:33:10.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.299+0000 7fd168d8a640 1 --2- 172.21.15.161:0/155620265 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd14406b500 0x7fd14406d9c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.299+0000 7fd168d8a640 1 --2- 172.21.15.161:0/155620265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd164152890 0x7fd1640ab700 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.299+0000 7fd168d8a640 1 --2- 172.21.15.161:0/155620265 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1641553a0 0x7fd1641e7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.299+0000 7fd168d8a640 1 --2- 172.21.15.161:0/155620265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1640c5410 0x7fd16414afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.299+0000 7fd168d8a640 1 -- 172.21.15.161:0/155620265 >> 172.21.15.161:0/155620265 conn(0x7fd1640ac620 msgr2=0x7fd1640bddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:10.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.299+0000 7fd168d8a640 1 -- 172.21.15.161:0/155620265 shutdown_connections 2024-03-28T23:33:10.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.299+0000 7fd168d8a640 1 -- 172.21.15.161:0/155620265 wait complete. 2024-03-28T23:33:10.311 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:33:10.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.436+0000 7f98bf9e9640 1 Processor -- start 2024-03-28T23:33:10.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.436+0000 7f98bf9e9640 1 -- start start 2024-03-28T23:33:10.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.437+0000 7f98bf9e9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f98b8152890 0x7f98b80c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:10.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.437+0000 7f98bf9e9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98b8158a30 con 0x7f98b80c8ed0 2024-03-28T23:33:10.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.437+0000 7f98bf9e9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98b8158d10 con 0x7f98b80c5b20 2024-03-28T23:33:10.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.437+0000 7f98bf9e9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f98b80c8480 con 0x7f98b8152890 2024-03-28T23:33:10.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.437+0000 7f98bcf84640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f98b80c8ed0 0x7f98b80c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44928/0 (socket says 172.21.15.161:44928) 2024-03-28T23:33:10.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.437+0000 7f98bcf84640 1 -- 172.21.15.161:0/4131734533 learned_addr learned my addr 172.21.15.161:0/4131734533 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:10.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.437+0000 7f98bd785640 1 --2- 172.21.15.161:0/4131734533 >> v2:172.21.15.170:3300/0 conn(0x7f98b8152890 0x7f98b80c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:10.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.437+0000 7f98affff640 1 -- 172.21.15.161:0/4131734533 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2770144788 0 0) 0x7f98b8158a30 con 0x7f98b80c8ed0 2024-03-28T23:33:10.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.437+0000 7f98affff640 1 -- 172.21.15.161:0/4131734533 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98a4003520 con 0x7f98b80c8ed0 2024-03-28T23:33:10.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.438+0000 7f98affff640 1 -- 172.21.15.161:0/4131734533 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3813525411 0 0) 0x7f98a4003520 con 0x7f98b80c8ed0 2024-03-28T23:33:10.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.438+0000 7f98affff640 1 -- 172.21.15.161:0/4131734533 >> v2:172.21.15.170:3300/0 conn(0x7f98b8152890 msgr2=0x7f98b80c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:10.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.438+0000 7f98affff640 1 --2- 172.21.15.161:0/4131734533 >> v2:172.21.15.170:3300/0 conn(0x7f98b8152890 0x7f98b80c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.438+0000 7f98affff640 1 -- 172.21.15.161:0/4131734533 >> v1:172.21.15.170:6789/0 conn(0x7f98b80c5b20 legacy=0x7f98b81553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:10.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.438+0000 7f98affff640 1 -- 172.21.15.161:0/4131734533 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b8158ff0 con 0x7f98b80c8ed0 2024-03-28T23:33:10.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.438+0000 7f98affff640 1 -- 172.21.15.161:0/4131734533 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f98b40017f0 con 0x7f98b80c8ed0 2024-03-28T23:33:10.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.438+0000 7f98affff640 1 -- 172.21.15.161:0/4131734533 >> v1:172.21.15.161:6789/0 conn(0x7f98b80c8ed0 legacy=0x7f98b80c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:10.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.438+0000 7f98affff640 1 --2- 172.21.15.161:0/4131734533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98a4003a90 0x7f98a4003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:10.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.438+0000 7f98affff640 1 --2- 172.21.15.161:0/4131734533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98a4004800 0x7f98a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:10.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.438+0000 7f98affff640 1 --2- 172.21.15.161:0/4131734533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f98a40053c0 0x7f98a400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:10.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.438+0000 7f98affff640 1 -- 172.21.15.161:0/4131734533 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f98a400c250 con 0x7f98a40053c0 2024-03-28T23:33:10.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.438+0000 7f98affff640 1 -- 172.21.15.161:0/4131734533 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f98a400c400 con 0x7f98a4004800 2024-03-28T23:33:10.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.438+0000 7f98affff640 1 -- 172.21.15.161:0/4131734533 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f98a400c600 con 0x7f98a4003a90 2024-03-28T23:33:10.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.438+0000 7f98bdf86640 1 --2- 172.21.15.161:0/4131734533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f98a40053c0 0x7f98a400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:10.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.438+0000 7f98bd785640 1 --2- 172.21.15.161:0/4131734533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98a4003a90 0x7f98a4003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:10.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.438+0000 7f98bcf84640 1 --2- 172.21.15.161:0/4131734533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98a4004800 0x7f98a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:10.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.438+0000 7f98bdf86640 1 -- 172.21.15.161:0/4131734533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98a4003a90 msgr2=0x7f98a4003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:10.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.438+0000 7f98bdf86640 1 --2- 172.21.15.161:0/4131734533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98a4003a90 0x7f98a4003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.438+0000 7f98bdf86640 1 -- 172.21.15.161:0/4131734533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98a4004800 msgr2=0x7f98a4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:10.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.438+0000 7f98bdf86640 1 --2- 172.21.15.161:0/4131734533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98a4004800 0x7f98a4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.438+0000 7f98bdf86640 1 -- 172.21.15.161:0/4131734533 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98a4003520 con 0x7f98a40053c0 2024-03-28T23:33:10.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.439+0000 7f98bdf86640 1 -- 172.21.15.161:0/4131734533 reap_dead start 2024-03-28T23:33:10.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.439+0000 7f98bdf86640 1 --2- 172.21.15.161:0/4131734533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f98a40053c0 0x7f98a400b9c0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f98b80caa40 tx=0x7f98a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=281a46db89db7a62 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:10.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.439+0000 7f98affff640 1 -- 172.21.15.161:0/4131734533 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b0001a70 con 0x7f98a40053c0 2024-03-28T23:33:10.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.439+0000 7f98affff640 1 -- 172.21.15.161:0/4131734533 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98b0003b70 con 0x7f98a40053c0 2024-03-28T23:33:10.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.439+0000 7f98affff640 1 -- 172.21.15.161:0/4131734533 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b00050d0 con 0x7f98a40053c0 2024-03-28T23:33:10.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.439+0000 7f98bf9e9640 1 -- 172.21.15.161:0/4131734533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f98a40053c0 msgr2=0x7f98a400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:10.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.439+0000 7f98bf9e9640 1 --2- 172.21.15.161:0/4131734533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f98a40053c0 0x7f98a400b9c0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f98b80caa40 tx=0x7f98a4005830 comp rx=0 tx=0).stop 2024-03-28T23:33:10.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.440+0000 7f98bf9e9640 1 -- 172.21.15.161:0/4131734533 shutdown_connections 2024-03-28T23:33:10.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.440+0000 7f98bf9e9640 1 --2- 172.21.15.161:0/4131734533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f98a40053c0 0x7f98a400b9c0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.440+0000 7f98bf9e9640 1 -- 172.21.15.161:0/4131734533 >> 172.21.15.161:0/4131734533 conn(0x7f98b80ac620 msgr2=0x7f98b80aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:10.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.440+0000 7f98bf9e9640 1 -- 172.21.15.161:0/4131734533 shutdown_connections 2024-03-28T23:33:10.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.440+0000 7f98bf9e9640 1 -- 172.21.15.161:0/4131734533 wait complete. 2024-03-28T23:33:10.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.441+0000 7f98bf9e9640 1 Processor -- start 2024-03-28T23:33:10.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.441+0000 7f98bf9e9640 1 -- start start 2024-03-28T23:33:10.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.441+0000 7f98bf9e9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98a40053c0 0x7f98b806c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:10.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.441+0000 7f98bf9e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98b814f520 0x7f98b81eabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:10.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.441+0000 7f98bf9e9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f98b81eb110 0x7f98b81ee0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:10.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.441+0000 7f98bf9e9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f98b8152c70 con 0x7f98b81eb110 2024-03-28T23:33:10.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.441+0000 7f98bf9e9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f98b8159640 con 0x7f98a40053c0 2024-03-28T23:33:10.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.441+0000 7f98bf9e9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f98b8152df0 con 0x7f98b814f520 2024-03-28T23:33:10.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.441+0000 7f98bdf86640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f98b81eb110 0x7f98b81ee0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:10.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.441+0000 7f98bdf86640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f98b81eb110 0x7f98b81ee0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46314/0 (socket says 172.21.15.161:46314) 2024-03-28T23:33:10.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.441+0000 7f98bdf86640 1 -- 172.21.15.161:0/1498364443 learned_addr learned my addr 172.21.15.161:0/1498364443 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:10.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.442+0000 7f98bcf84640 1 --2- 172.21.15.161:0/1498364443 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98b814f520 0x7f98b81eabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:10.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.442+0000 7f98bd785640 1 --2- 172.21.15.161:0/1498364443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98a40053c0 0x7f98b806c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:10.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.442+0000 7f98bdf86640 1 -- 172.21.15.161:0/1498364443 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98b814f520 msgr2=0x7f98b81eabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:10.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.442+0000 7f98bdf86640 1 --2- 172.21.15.161:0/1498364443 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98b814f520 0x7f98b81eabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.442+0000 7f98bdf86640 1 -- 172.21.15.161:0/1498364443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98a40053c0 msgr2=0x7f98b806c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:10.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.442+0000 7f98bdf86640 1 --2- 172.21.15.161:0/1498364443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98a40053c0 0x7f98b806c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.442+0000 7f98bdf86640 1 -- 172.21.15.161:0/1498364443 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b80be5c0 con 0x7f98b81eb110 2024-03-28T23:33:10.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.442+0000 7f98bdf86640 1 --2- 172.21.15.161:0/1498364443 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f98b81eb110 0x7f98b81ee0b0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f98b002c020 tx=0x7f98b00014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:10.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.442+0000 7f98adffb640 1 -- 172.21.15.161:0/1498364443 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b0006b50 con 0x7f98b81eb110 2024-03-28T23:33:10.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.442+0000 7f98adffb640 1 -- 172.21.15.161:0/1498364443 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98b0006cf0 con 0x7f98b81eb110 2024-03-28T23:33:10.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.442+0000 7f98bf9e9640 1 -- 172.21.15.161:0/1498364443 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98b80c0c70 con 0x7f98b81eb110 2024-03-28T23:33:10.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.442+0000 7f98adffb640 1 -- 172.21.15.161:0/1498364443 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b0007250 con 0x7f98b81eb110 2024-03-28T23:33:10.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.442+0000 7f98bf9e9640 1 -- 172.21.15.161:0/1498364443 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98b80c1230 con 0x7f98b81eb110 2024-03-28T23:33:10.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.443+0000 7f98bf9e9640 1 -- 172.21.15.161:0/1498364443 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98b804f4b0 con 0x7f98b81eb110 2024-03-28T23:33:10.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.444+0000 7f98adffb640 1 -- 172.21.15.161:0/1498364443 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f98b00073f0 con 0x7f98b81eb110 2024-03-28T23:33:10.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.444+0000 7f98adffb640 1 --2- 172.21.15.161:0/1498364443 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f989c06b4d0 0x7f989c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:10.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.444+0000 7f98bd785640 1 --2- 172.21.15.161:0/1498364443 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f989c06b4d0 0x7f989c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:10.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.444+0000 7f98adffb640 1 -- 172.21.15.161:0/1498364443 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 6363+0+0 (secure 0 0 0) 0x7f98b00ab840 con 0x7f98b81eb110 2024-03-28T23:33:10.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.445+0000 7f98bd785640 1 --2- 172.21.15.161:0/1498364443 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f989c06b4d0 0x7f989c06d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f98a8009980 tx=0x7f98a8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:10.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.449+0000 7f98adffb640 1 -- 172.21.15.161:0/1498364443 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f98b0049020 con 0x7f98b81eb110 2024-03-28T23:33:10.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.629+0000 7f98bf9e9640 1 -- 172.21.15.161:0/1498364443 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f98b80c8140 con 0x7f98b81eb110 2024-03-28T23:33:10.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.630+0000 7f98adffb640 1 -- 172.21.15.161:0/1498364443 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+17320 (secure 0 0 0) 0x7f98b003f070 con 0x7f98b81eb110 2024-03-28T23:33:10.631 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:10.631 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":82,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:33:08.559271+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:32:42.378609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:32:43.436985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6835/2691084827":"2024-03-29T23:33:06.309901+0000","172.21.15.161:6836/2378263687":"2024-03-29T23:33:05.302530+0000","172.21.15.170:6833/2691084827":"2024-03-29T23:33:06.309901+0000","172.21.15.161:6834/2378263687":"2024-03-29T23:33:05.302530+0000","172.21.15.161:6836/3850309502":"2024-03-29T23:32:54.518520+0000","172.21.15.161:6834/3850309502":"2024-03-29T23:32:54.518520+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:33:10.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.632+0000 7f98bf9e9640 1 -- 172.21.15.161:0/1498364443 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f989c06b4d0 msgr2=0x7f989c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:10.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.632+0000 7f98bf9e9640 1 --2- 172.21.15.161:0/1498364443 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f989c06b4d0 0x7f989c06d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f98a8009980 tx=0x7f98a8008040 comp rx=0 tx=0).stop 2024-03-28T23:33:10.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.632+0000 7f98bf9e9640 1 -- 172.21.15.161:0/1498364443 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f98b81eb110 msgr2=0x7f98b81ee0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:10.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.632+0000 7f98bf9e9640 1 --2- 172.21.15.161:0/1498364443 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f98b81eb110 0x7f98b81ee0b0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f98b002c020 tx=0x7f98b00014a0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.632+0000 7f98bf9e9640 1 -- 172.21.15.161:0/1498364443 shutdown_connections 2024-03-28T23:33:10.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.632+0000 7f98bf9e9640 1 --2- 172.21.15.161:0/1498364443 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f989c06b4d0 0x7f989c06d990 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.632+0000 7f98bf9e9640 1 --2- 172.21.15.161:0/1498364443 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f98b81eb110 0x7f98b81ee0b0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.632+0000 7f98bf9e9640 1 --2- 172.21.15.161:0/1498364443 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98b814f520 0x7f98b81eabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.632+0000 7f98bf9e9640 1 --2- 172.21.15.161:0/1498364443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98a40053c0 0x7f98b806c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.632+0000 7f98bf9e9640 1 -- 172.21.15.161:0/1498364443 >> 172.21.15.161:0/1498364443 conn(0x7f98b80ac620 msgr2=0x7f98b806cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:10.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.632+0000 7f98bf9e9640 1 -- 172.21.15.161:0/1498364443 shutdown_connections 2024-03-28T23:33:10.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.632+0000 7f98bf9e9640 1 -- 172.21.15.161:0/1498364443 wait complete. 2024-03-28T23:33:10.645 INFO:tasks.cephfs.filesystem:Destroying file system cephfs2 and related pools 2024-03-28T23:33:10.645 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:33:10.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.747+0000 7f566e654640 1 Processor -- start 2024-03-28T23:33:10.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.747+0000 7f566e654640 1 -- start start 2024-03-28T23:33:10.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.748+0000 7f566e654640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5668152a80 0x7f5668155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:10.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.748+0000 7f566e654640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f566806d9f0 0x7f56680c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:10.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.748+0000 7f566e654640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f566806cb80 0x7f56680c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:10.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.748+0000 7f566e654640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5668156250 con 0x7f566806cb80 2024-03-28T23:33:10.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.748+0000 7f566e654640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f566806c6f0 con 0x7f5668152a80 2024-03-28T23:33:10.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.748+0000 7f566e654640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f56680c98e0 con 0x7f566806d9f0 2024-03-28T23:33:10.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.748+0000 7f566cbf1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f566806cb80 0x7f56680c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:10.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.748+0000 7f566cbf1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f566806cb80 0x7f56680c74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48740/0 (socket says 172.21.15.161:48740) 2024-03-28T23:33:10.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.748+0000 7f566cbf1640 1 -- 172.21.15.161:0/3192361153 learned_addr learned my addr 172.21.15.161:0/3192361153 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:10.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.748+0000 7f56677fe640 1 --2- 172.21.15.161:0/3192361153 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f566806d9f0 0x7f56680c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:10.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.748+0000 7f5667fff640 1 --2- 172.21.15.161:0/3192361153 >> v2:172.21.15.170:3300/0 conn(0x7f5668152a80 0x7f5668155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:10.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.748+0000 7f566cbf1640 1 -- 172.21.15.161:0/3192361153 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f566806d9f0 msgr2=0x7f56680c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:10.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.748+0000 7f566cbf1640 1 --2- 172.21.15.161:0/3192361153 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f566806d9f0 0x7f56680c93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.748+0000 7f566cbf1640 1 -- 172.21.15.161:0/3192361153 >> v2:172.21.15.170:3300/0 conn(0x7f5668152a80 msgr2=0x7f5668155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:10.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.748+0000 7f566cbf1640 1 --2- 172.21.15.161:0/3192361153 >> v2:172.21.15.170:3300/0 conn(0x7f5668152a80 0x7f5668155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.748+0000 7f566cbf1640 1 -- 172.21.15.161:0/3192361153 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f566806c3c0 con 0x7f566806cb80 2024-03-28T23:33:10.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.749+0000 7f566cbf1640 1 --2- 172.21.15.161:0/3192361153 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f566806cb80 0x7f56680c74c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f56680c7ff0 tx=0x7f566002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd7ba1f709a105f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:10.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.749+0000 7f5666ffd640 1 -- 172.21.15.161:0/3192361153 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566002c950 con 0x7f566806cb80 2024-03-28T23:33:10.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.749+0000 7f5666ffd640 1 -- 172.21.15.161:0/3192361153 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f566002caf0 con 0x7f566806cb80 2024-03-28T23:33:10.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.749+0000 7f5666ffd640 1 -- 172.21.15.161:0/3192361153 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56600343f0 con 0x7f566806cb80 2024-03-28T23:33:10.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.750+0000 7f566e654640 1 -- 172.21.15.161:0/3192361153 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f566806cb80 msgr2=0x7f56680c74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:10.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.750+0000 7f566e654640 1 --2- 172.21.15.161:0/3192361153 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f566806cb80 0x7f56680c74c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f56680c7ff0 tx=0x7f566002bdb0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.750+0000 7f566e654640 1 -- 172.21.15.161:0/3192361153 shutdown_connections 2024-03-28T23:33:10.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.750+0000 7f566e654640 1 --2- 172.21.15.161:0/3192361153 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f566806cb80 0x7f56680c74c0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.750+0000 7f566e654640 1 --2- 172.21.15.161:0/3192361153 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f566806d9f0 0x7f56680c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.750+0000 7f566e654640 1 --2- 172.21.15.161:0/3192361153 >> v2:172.21.15.170:3300/0 conn(0x7f5668152a80 0x7f5668155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.750+0000 7f566e654640 1 -- 172.21.15.161:0/3192361153 >> 172.21.15.161:0/3192361153 conn(0x7f56680ac720 msgr2=0x7f56680acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:10.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.750+0000 7f566e654640 1 -- 172.21.15.161:0/3192361153 shutdown_connections 2024-03-28T23:33:10.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.750+0000 7f566e654640 1 -- 172.21.15.161:0/3192361153 wait complete. 2024-03-28T23:33:10.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.751+0000 7f566e654640 1 Processor -- start 2024-03-28T23:33:10.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.751+0000 7f566e654640 1 -- start start 2024-03-28T23:33:10.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.752+0000 7f566e654640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f566806cb80 0x7f56681543c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:10.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.752+0000 7f566e654640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f566806d9f0 0x7f566814c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:10.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.752+0000 7f566e654640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5668152a80 0x7f566814cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:10.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.752+0000 7f566e654640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f56680c58b0 con 0x7f5668152a80 2024-03-28T23:33:10.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.752+0000 7f566e654640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f56680c55d0 con 0x7f566806d9f0 2024-03-28T23:33:10.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.752+0000 7f566e654640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f566806d010 con 0x7f566806cb80 2024-03-28T23:33:10.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.752+0000 7f566cbf1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5668152a80 0x7f566814cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:10.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.752+0000 7f566cbf1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5668152a80 0x7f566814cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48746/0 (socket says 172.21.15.161:48746) 2024-03-28T23:33:10.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.752+0000 7f566cbf1640 1 -- 172.21.15.161:0/518387373 learned_addr learned my addr 172.21.15.161:0/518387373 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:10.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.752+0000 7f5667fff640 1 --2- 172.21.15.161:0/518387373 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f566806cb80 0x7f56681543c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:10.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.752+0000 7f566cbf1640 1 -- 172.21.15.161:0/518387373 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f566806cb80 msgr2=0x7f56681543c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:10.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.752+0000 7f566cbf1640 1 --2- 172.21.15.161:0/518387373 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f566806cb80 0x7f56681543c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.752+0000 7f566cbf1640 1 -- 172.21.15.161:0/518387373 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f566806d9f0 msgr2=0x7f566814c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:10.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.752+0000 7f566cbf1640 1 --2- 172.21.15.161:0/518387373 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f566806d9f0 0x7f566814c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.752+0000 7f566cbf1640 1 -- 172.21.15.161:0/518387373 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56680b0680 con 0x7f5668152a80 2024-03-28T23:33:10.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.752+0000 7f566cbf1640 1 --2- 172.21.15.161:0/518387373 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5668152a80 0x7f566814cbd0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f5660036040 tx=0x7f5660004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:10.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.752+0000 7f5664ff9640 1 -- 172.21.15.161:0/518387373 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5660043020 con 0x7f5668152a80 2024-03-28T23:33:10.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.752+0000 7f5664ff9640 1 -- 172.21.15.161:0/518387373 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f566003e450 con 0x7f5668152a80 2024-03-28T23:33:10.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.752+0000 7f566e654640 1 -- 172.21.15.161:0/518387373 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5668156fb0 con 0x7f5668152a80 2024-03-28T23:33:10.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.752+0000 7f5664ff9640 1 -- 172.21.15.161:0/518387373 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566003e820 con 0x7f5668152a80 2024-03-28T23:33:10.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.752+0000 7f566e654640 1 -- 172.21.15.161:0/518387373 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5668157570 con 0x7f5668152a80 2024-03-28T23:33:10.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.754+0000 7f566e654640 1 -- 172.21.15.161:0/518387373 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f566804f4b0 con 0x7f5668152a80 2024-03-28T23:33:10.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.754+0000 7f5664ff9640 1 -- 172.21.15.161:0/518387373 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5660046020 con 0x7f5668152a80 2024-03-28T23:33:10.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.754+0000 7f5664ff9640 1 --2- 172.21.15.161:0/518387373 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f564806b4d0 0x7f564806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:10.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.754+0000 7f5667fff640 1 --2- 172.21.15.161:0/518387373 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f564806b4d0 0x7f564806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:10.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.754+0000 7f5664ff9640 1 -- 172.21.15.161:0/518387373 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 6363+0+0 (secure 0 0 0) 0x7f56600acbc0 con 0x7f5668152a80 2024-03-28T23:33:10.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.755+0000 7f5667fff640 1 --2- 172.21.15.161:0/518387373 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f564806b4d0 0x7f564806d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5658009a30 tx=0x7f5658008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:10.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.759+0000 7f5664ff9640 1 -- 172.21.15.161:0/518387373 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f566004d030 con 0x7f5668152a80 2024-03-28T23:33:10.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.976+0000 7f566e654640 1 -- 172.21.15.161:0/518387373 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56680ca030 con 0x7f5668152a80 2024-03-28T23:33:10.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.976+0000 7f5664ff9640 1 -- 172.21.15.161:0/518387373 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+5193 (secure 0 0 0) 0x7f566007b8d0 con 0x7f5668152a80 2024-03-28T23:33:10.977 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:10.977 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":73,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5579,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.170:6834/3501962065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3501962065},{"type":"v1","addr":"172.21.15.170:6834","nonce":3501962065}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":60},{"gid":6397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2706065827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":2706065827},{"type":"v1","addr":"172.21.15.170:6835","nonce":2706065827}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":6903,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/3336491043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3336491043},{"type":"v1","addr":"172.21.15.161:6836","nonce":3336491043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:32:44.463287+0000","modified":"2024-03-28T23:33:09.314639+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5570":{"gid":5570,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":13,"addr":"172.21.15.161:6837/737681942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":737681942},{"type":"v1","addr":"172.21.15.161:6837","nonce":737681942}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5570,"qdb_cluster":[5570]},"id":6}]} 2024-03-28T23:33:10.978 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 73 2024-03-28T23:33:10.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.979+0000 7f566e654640 1 -- 172.21.15.161:0/518387373 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f564806b4d0 msgr2=0x7f564806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:10.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.979+0000 7f566e654640 1 --2- 172.21.15.161:0/518387373 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f564806b4d0 0x7f564806d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5658009a30 tx=0x7f5658008040 comp rx=0 tx=0).stop 2024-03-28T23:33:10.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.979+0000 7f566e654640 1 -- 172.21.15.161:0/518387373 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5668152a80 msgr2=0x7f566814cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:10.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.979+0000 7f566e654640 1 --2- 172.21.15.161:0/518387373 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5668152a80 0x7f566814cbd0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f5660036040 tx=0x7f5660004a40 comp rx=0 tx=0).stop 2024-03-28T23:33:10.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.979+0000 7f566e654640 1 -- 172.21.15.161:0/518387373 shutdown_connections 2024-03-28T23:33:10.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.979+0000 7f566e654640 1 --2- 172.21.15.161:0/518387373 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f564806b4d0 0x7f564806d990 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.979+0000 7f566e654640 1 --2- 172.21.15.161:0/518387373 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5668152a80 0x7f566814cbd0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.979+0000 7f566e654640 1 --2- 172.21.15.161:0/518387373 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f566806d9f0 0x7f566814c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.979+0000 7f566e654640 1 --2- 172.21.15.161:0/518387373 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f566806cb80 0x7f56681543c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:10.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.979+0000 7f566e654640 1 -- 172.21.15.161:0/518387373 >> 172.21.15.161:0/518387373 conn(0x7f56680ac720 msgr2=0x7f56680acb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:10.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.979+0000 7f566e654640 1 -- 172.21.15.161:0/518387373 shutdown_connections 2024-03-28T23:33:10.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:10.979+0000 7f566e654640 1 -- 172.21.15.161:0/518387373 wait complete. 2024-03-28T23:33:10.991 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:33:11.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.092+0000 7f3b31db6640 1 Processor -- start 2024-03-28T23:33:11.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.092+0000 7f3b31db6640 1 -- start start 2024-03-28T23:33:11.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.093+0000 7f3b31db6640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3b2c0c5410 0x7f3b2c0c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:11.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.093+0000 7f3b31db6640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3b2c1553a0 0x7f3b2c155850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:11.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.093+0000 7f3b31db6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b2c158a30 con 0x7f3b2c152890 2024-03-28T23:33:11.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.093+0000 7f3b31db6640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f3b2c0c8480 con 0x7f3b2c1553a0 2024-03-28T23:33:11.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.093+0000 7f3b31db6640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f3b2c0c5d90 con 0x7f3b2c0c5410 2024-03-28T23:33:11.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.093+0000 7f3b2bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3b2c1553a0 0x7f3b2c155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:11.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.093+0000 7f3b2b7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f3b2c152890 0x7f3b2c0c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:39456/0 (socket says 172.21.15.161:39456) 2024-03-28T23:33:11.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.093+0000 7f3b2b7fe640 1 -- 172.21.15.161:0/1344051418 learned_addr learned my addr 172.21.15.161:0/1344051418 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:11.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.093+0000 7f3b23fff640 1 --2- 172.21.15.161:0/1344051418 >> v2:172.21.15.170:3300/0 conn(0x7f3b2c0c5410 0x7f3b2c0c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:11.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.094+0000 7f3b2bfff640 1 -- 172.21.15.161:0/1344051418 >> v2:172.21.15.170:3300/0 conn(0x7f3b2c0c5410 msgr2=0x7f3b2c0c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:11.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.094+0000 7f3b2bfff640 1 --2- 172.21.15.161:0/1344051418 >> v2:172.21.15.170:3300/0 conn(0x7f3b2c0c5410 0x7f3b2c0c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:11.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.094+0000 7f3b2bfff640 1 -- 172.21.15.161:0/1344051418 >> v1:172.21.15.161:6789/0 conn(0x7f3b2c152890 legacy=0x7f3b2c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:11.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.094+0000 7f3b2bfff640 1 -- 172.21.15.161:0/1344051418 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b2c158d10 con 0x7f3b2c1553a0 2024-03-28T23:33:11.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.094+0000 7f3b2bfff640 1 --2- 172.21.15.161:0/1344051418 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b2c1553a0 0x7f3b2c155850 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c0ad6e0 tx=0x7f3b2402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5d24c050bb7188 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:11.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.094+0000 7f3b2affd640 1 -- 172.21.15.161:0/1344051418 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b24039070 con 0x7f3b2c1553a0 2024-03-28T23:33:11.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.094+0000 7f3b2affd640 1 -- 172.21.15.161:0/1344051418 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b2402cbf0 con 0x7f3b2c1553a0 2024-03-28T23:33:11.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.094+0000 7f3b2affd640 1 -- 172.21.15.161:0/1344051418 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b24034440 con 0x7f3b2c1553a0 2024-03-28T23:33:11.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.095+0000 7f3b31db6640 1 -- 172.21.15.161:0/1344051418 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b2c1553a0 msgr2=0x7f3b2c155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:11.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.095+0000 7f3b31db6640 1 --2- 172.21.15.161:0/1344051418 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b2c1553a0 0x7f3b2c155850 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c0ad6e0 tx=0x7f3b2402be40 comp rx=0 tx=0).stop 2024-03-28T23:33:11.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.095+0000 7f3b31db6640 1 -- 172.21.15.161:0/1344051418 shutdown_connections 2024-03-28T23:33:11.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.095+0000 7f3b31db6640 1 --2- 172.21.15.161:0/1344051418 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b2c1553a0 0x7f3b2c155850 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:11.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.095+0000 7f3b31db6640 1 --2- 172.21.15.161:0/1344051418 >> v2:172.21.15.170:3300/0 conn(0x7f3b2c0c5410 0x7f3b2c0c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:11.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.095+0000 7f3b31db6640 1 -- 172.21.15.161:0/1344051418 >> 172.21.15.161:0/1344051418 conn(0x7f3b2c0ac620 msgr2=0x7f3b2c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:11.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.095+0000 7f3b31db6640 1 -- 172.21.15.161:0/1344051418 shutdown_connections 2024-03-28T23:33:11.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.095+0000 7f3b31db6640 1 -- 172.21.15.161:0/1344051418 wait complete. 2024-03-28T23:33:11.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.096+0000 7f3b31db6640 1 Processor -- start 2024-03-28T23:33:11.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.096+0000 7f3b31db6640 1 -- start start 2024-03-28T23:33:11.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.097+0000 7f3b31db6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b2c0c5410 0x7f3b2c14a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:11.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.097+0000 7f3b31db6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b2c152890 0x7f3b2c147e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:11.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.097+0000 7f3b31db6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b2c1553a0 0x7f3b2c148350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:11.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.097+0000 7f3b31db6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2c1592d0 con 0x7f3b2c1553a0 2024-03-28T23:33:11.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.097+0000 7f3b31db6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2c159640 con 0x7f3b2c152890 2024-03-28T23:33:11.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.097+0000 7f3b31db6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3b2c0c90a0 con 0x7f3b2c0c5410 2024-03-28T23:33:11.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.097+0000 7f3b2bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b2c1553a0 0x7f3b2c148350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:11.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.097+0000 7f3b2bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b2c1553a0 0x7f3b2c148350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48772/0 (socket says 172.21.15.161:48772) 2024-03-28T23:33:11.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.097+0000 7f3b2bfff640 1 -- 172.21.15.161:0/2552344575 learned_addr learned my addr 172.21.15.161:0/2552344575 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:11.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.097+0000 7f3b23fff640 1 --2- 172.21.15.161:0/2552344575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b2c152890 0x7f3b2c147e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:11.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.097+0000 7f3b2b7fe640 1 --2- 172.21.15.161:0/2552344575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b2c0c5410 0x7f3b2c14a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:11.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.097+0000 7f3b2bfff640 1 -- 172.21.15.161:0/2552344575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b2c0c5410 msgr2=0x7f3b2c14a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:11.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.097+0000 7f3b2bfff640 1 --2- 172.21.15.161:0/2552344575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b2c0c5410 0x7f3b2c14a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:11.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.097+0000 7f3b2bfff640 1 -- 172.21.15.161:0/2552344575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b2c152890 msgr2=0x7f3b2c147e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:11.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.097+0000 7f3b2bfff640 1 --2- 172.21.15.161:0/2552344575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b2c152890 0x7f3b2c147e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:11.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.097+0000 7f3b2bfff640 1 -- 172.21.15.161:0/2552344575 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b2c0be900 con 0x7f3b2c1553a0 2024-03-28T23:33:11.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.097+0000 7f3b2bfff640 1 --2- 172.21.15.161:0/2552344575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b2c1553a0 0x7f3b2c148350 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f3b2402c450 tx=0x7f3b24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:11.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.098+0000 7f3b28ff9640 1 -- 172.21.15.161:0/2552344575 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b24039050 con 0x7f3b2c1553a0 2024-03-28T23:33:11.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.098+0000 7f3b28ff9640 1 -- 172.21.15.161:0/2552344575 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b24045070 con 0x7f3b2c1553a0 2024-03-28T23:33:11.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.098+0000 7f3b28ff9640 1 -- 172.21.15.161:0/2552344575 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b2403e830 con 0x7f3b2c1553a0 2024-03-28T23:33:11.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.098+0000 7f3b31db6640 1 -- 172.21.15.161:0/2552344575 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b2c0bebe0 con 0x7f3b2c1553a0 2024-03-28T23:33:11.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.098+0000 7f3b31db6640 1 -- 172.21.15.161:0/2552344575 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b2c14a640 con 0x7f3b2c1553a0 2024-03-28T23:33:11.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.099+0000 7f3b31db6640 1 -- 172.21.15.161:0/2552344575 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b2c151a50 con 0x7f3b2c1553a0 2024-03-28T23:33:11.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.099+0000 7f3b28ff9640 1 -- 172.21.15.161:0/2552344575 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3b2403e9d0 con 0x7f3b2c1553a0 2024-03-28T23:33:11.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.100+0000 7f3b28ff9640 1 --2- 172.21.15.161:0/2552344575 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3b0c06b4d0 0x7f3b0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:11.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.100+0000 7f3b28ff9640 1 -- 172.21.15.161:0/2552344575 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 6363+0+0 (secure 0 0 0) 0x7f3b240ad150 con 0x7f3b2c1553a0 2024-03-28T23:33:11.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.100+0000 7f3b2b7fe640 1 --2- 172.21.15.161:0/2552344575 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3b0c06b4d0 0x7f3b0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:11.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.100+0000 7f3b2b7fe640 1 --2- 172.21.15.161:0/2552344575 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3b0c06b4d0 0x7f3b0c06d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c0c4eb0 tx=0x7f3b1c007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:11.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.104+0000 7f3b28ff9640 1 -- 172.21.15.161:0/2552344575 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3b2407b8a0 con 0x7f3b2c1553a0 2024-03-28T23:33:11.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.317+0000 7f3b31db6640 1 -- 172.21.15.161:0/2552344575 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b2c0bedb0 con 0x7f3b2c1553a0 2024-03-28T23:33:11.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.318+0000 7f3b28ff9640 1 -- 172.21.15.161:0/2552344575 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+5193 (secure 0 0 0) 0x7f3b2c0bedb0 con 0x7f3b2c1553a0 2024-03-28T23:33:11.319 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:11.319 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":73,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5579,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.170:6834/3501962065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3501962065},{"type":"v1","addr":"172.21.15.170:6834","nonce":3501962065}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":60},{"gid":6397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2706065827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":2706065827},{"type":"v1","addr":"172.21.15.170:6835","nonce":2706065827}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":6903,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/3336491043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3336491043},{"type":"v1","addr":"172.21.15.161:6836","nonce":3336491043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:32:44.463287+0000","modified":"2024-03-28T23:33:09.314639+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5570":{"gid":5570,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":13,"addr":"172.21.15.161:6837/737681942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":737681942},{"type":"v1","addr":"172.21.15.161:6837","nonce":737681942}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5570,"qdb_cluster":[5570]},"id":6}]} 2024-03-28T23:33:11.319 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 73 2024-03-28T23:33:11.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.320+0000 7f3b31db6640 1 -- 172.21.15.161:0/2552344575 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3b0c06b4d0 msgr2=0x7f3b0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:11.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.320+0000 7f3b31db6640 1 --2- 172.21.15.161:0/2552344575 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3b0c06b4d0 0x7f3b0c06d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c0c4eb0 tx=0x7f3b1c007e80 comp rx=0 tx=0).stop 2024-03-28T23:33:11.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.320+0000 7f3b31db6640 1 -- 172.21.15.161:0/2552344575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b2c1553a0 msgr2=0x7f3b2c148350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:11.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.320+0000 7f3b31db6640 1 --2- 172.21.15.161:0/2552344575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b2c1553a0 0x7f3b2c148350 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f3b2402c450 tx=0x7f3b24004a40 comp rx=0 tx=0).stop 2024-03-28T23:33:11.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.320+0000 7f3b31db6640 1 -- 172.21.15.161:0/2552344575 shutdown_connections 2024-03-28T23:33:11.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.320+0000 7f3b31db6640 1 --2- 172.21.15.161:0/2552344575 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3b0c06b4d0 0x7f3b0c06d990 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:11.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.320+0000 7f3b31db6640 1 --2- 172.21.15.161:0/2552344575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b2c1553a0 0x7f3b2c148350 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:11.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.320+0000 7f3b31db6640 1 --2- 172.21.15.161:0/2552344575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b2c152890 0x7f3b2c147e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:11.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.320+0000 7f3b31db6640 1 --2- 172.21.15.161:0/2552344575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b2c0c5410 0x7f3b2c14a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:11.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.320+0000 7f3b31db6640 1 -- 172.21.15.161:0/2552344575 >> 172.21.15.161:0/2552344575 conn(0x7f3b2c0ac620 msgr2=0x7f3b2c06bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:11.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.320+0000 7f3b31db6640 1 -- 172.21.15.161:0/2552344575 shutdown_connections 2024-03-28T23:33:11.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.320+0000 7f3b31db6640 1 -- 172.21.15.161:0/2552344575 wait complete. 2024-03-28T23:33:11.332 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:33:11.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.434+0000 7f39947dd640 1 Processor -- start 2024-03-28T23:33:11.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.434+0000 7f39947dd640 1 -- start start 2024-03-28T23:33:11.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.435+0000 7f39947dd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f398c14eca0 0x7f398c0c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:11.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.435+0000 7f39947dd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f398c0c1dc0 0x7f398c06ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:11.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.435+0000 7f39947dd640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f398c151850 con 0x7f398c06cf60 2024-03-28T23:33:11.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.435+0000 7f39947dd640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f398c152460 con 0x7f398c0c1dc0 2024-03-28T23:33:11.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.435+0000 7f39947dd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f398c151b30 con 0x7f398c14eca0 2024-03-28T23:33:11.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.435+0000 7f3992579640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f398c14eca0 0x7f398c0c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:11.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.435+0000 7f3991d78640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f398c0c1dc0 0x7f398c06ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:11.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.435+0000 7f3992579640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f398c14eca0 0x7f398c0c1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:45544/0 (socket says 172.21.15.161:45544) 2024-03-28T23:33:11.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.435+0000 7f3992579640 1 -- 172.21.15.161:0/643939755 learned_addr learned my addr 172.21.15.161:0/643939755 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:11.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.435+0000 7f3991d78640 1 -- 172.21.15.161:0/643939755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f398c14eca0 msgr2=0x7f398c0c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:11.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.435+0000 7f3991d78640 1 --2- 172.21.15.161:0/643939755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f398c14eca0 0x7f398c0c1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:11.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.435+0000 7f3991d78640 1 -- 172.21.15.161:0/643939755 >> v1:172.21.15.170:6789/0 conn(0x7f398c06cf60 legacy=0x7f398c0a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:11.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.435+0000 7f3991d78640 1 -- 172.21.15.161:0/643939755 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f398c151e10 con 0x7f398c0c1dc0 2024-03-28T23:33:11.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.435+0000 7f3992579640 1 --2- 172.21.15.161:0/643939755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f398c14eca0 0x7f398c0c1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:11.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.436+0000 7f3991d78640 1 --2- 172.21.15.161:0/643939755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f398c0c1dc0 0x7f398c06ca20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f397c0097a0 tx=0x7f397c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cda98aff356cd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:11.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.436+0000 7f3991577640 1 -- 172.21.15.161:0/643939755 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f397c0315a0 con 0x7f398c0c1dc0 2024-03-28T23:33:11.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.436+0000 7f3991577640 1 -- 172.21.15.161:0/643939755 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f397c037070 con 0x7f398c0c1dc0 2024-03-28T23:33:11.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.436+0000 7f3991577640 1 -- 172.21.15.161:0/643939755 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f397c031960 con 0x7f398c0c1dc0 2024-03-28T23:33:11.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.437+0000 7f39947dd640 1 -- 172.21.15.161:0/643939755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f398c0c1dc0 msgr2=0x7f398c06ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:11.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.437+0000 7f39947dd640 1 --2- 172.21.15.161:0/643939755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f398c0c1dc0 0x7f398c06ca20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f397c0097a0 tx=0x7f397c02b840 comp rx=0 tx=0).stop 2024-03-28T23:33:11.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.437+0000 7f39947dd640 1 -- 172.21.15.161:0/643939755 shutdown_connections 2024-03-28T23:33:11.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.437+0000 7f39947dd640 1 --2- 172.21.15.161:0/643939755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f398c0c1dc0 0x7f398c06ca20 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:11.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.437+0000 7f39947dd640 1 --2- 172.21.15.161:0/643939755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f398c14eca0 0x7f398c0c1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:11.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.437+0000 7f39947dd640 1 -- 172.21.15.161:0/643939755 >> 172.21.15.161:0/643939755 conn(0x7f398c0c6670 msgr2=0x7f398c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:11.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.437+0000 7f39947dd640 1 -- 172.21.15.161:0/643939755 shutdown_connections 2024-03-28T23:33:11.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.437+0000 7f39947dd640 1 -- 172.21.15.161:0/643939755 wait complete. 2024-03-28T23:33:11.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.438+0000 7f39947dd640 1 Processor -- start 2024-03-28T23:33:11.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.438+0000 7f39947dd640 1 -- start start 2024-03-28T23:33:11.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.439+0000 7f39947dd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f398c06cf60 0x7f398c071a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:11.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.439+0000 7f39947dd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f398c0c1dc0 0x7f398c071f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:11.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.440+0000 7f39947dd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f398c14eca0 0x7f398c1547c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:11.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.440+0000 7f39947dd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f398c14f080 con 0x7f398c06cf60 2024-03-28T23:33:11.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.440+0000 7f39947dd640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f398c06d340 con 0x7f398c0c1dc0 2024-03-28T23:33:11.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.440+0000 7f39947dd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f398c1514b0 con 0x7f398c14eca0 2024-03-28T23:33:11.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.440+0000 7f3992579640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f398c06cf60 0x7f398c071a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:11.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.440+0000 7f3992579640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f398c06cf60 0x7f398c071a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48784/0 (socket says 172.21.15.161:48784) 2024-03-28T23:33:11.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.440+0000 7f3992579640 1 -- 172.21.15.161:0/1159115294 learned_addr learned my addr 172.21.15.161:0/1159115294 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:11.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.440+0000 7f3992d7a640 1 --2- 172.21.15.161:0/1159115294 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f398c14eca0 0x7f398c1547c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:11.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.440+0000 7f3991d78640 1 --2- 172.21.15.161:0/1159115294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f398c0c1dc0 0x7f398c071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:11.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.440+0000 7f3992579640 1 -- 172.21.15.161:0/1159115294 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f398c14eca0 msgr2=0x7f398c1547c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:11.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.440+0000 7f3992579640 1 --2- 172.21.15.161:0/1159115294 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f398c14eca0 0x7f398c1547c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:11.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.440+0000 7f3992579640 1 -- 172.21.15.161:0/1159115294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f398c0c1dc0 msgr2=0x7f398c071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:11.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.440+0000 7f3992579640 1 --2- 172.21.15.161:0/1159115294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f398c0c1dc0 0x7f398c071f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:11.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.440+0000 7f3992579640 1 -- 172.21.15.161:0/1159115294 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f398c14fb90 con 0x7f398c06cf60 2024-03-28T23:33:11.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.440+0000 7f3992579640 1 --2- 172.21.15.161:0/1159115294 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f398c06cf60 0x7f398c071a00 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f398800e9e0 tx=0x7f398800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:11.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.440+0000 7f3982ffd640 1 -- 172.21.15.161:0/1159115294 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3988017070 con 0x7f398c06cf60 2024-03-28T23:33:11.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.440+0000 7f3982ffd640 1 -- 172.21.15.161:0/1159115294 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f398800c9d0 con 0x7f398c06cf60 2024-03-28T23:33:11.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.440+0000 7f39947dd640 1 -- 172.21.15.161:0/1159115294 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f398c140870 con 0x7f398c06cf60 2024-03-28T23:33:11.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.440+0000 7f3982ffd640 1 -- 172.21.15.161:0/1159115294 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398800cd80 con 0x7f398c06cf60 2024-03-28T23:33:11.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.440+0000 7f39947dd640 1 -- 172.21.15.161:0/1159115294 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f398c1453f0 con 0x7f398c06cf60 2024-03-28T23:33:11.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.441+0000 7f39947dd640 1 -- 172.21.15.161:0/1159115294 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f398c04f4b0 con 0x7f398c06cf60 2024-03-28T23:33:11.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.442+0000 7f3982ffd640 1 -- 172.21.15.161:0/1159115294 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f398801a070 con 0x7f398c06cf60 2024-03-28T23:33:11.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.442+0000 7f3982ffd640 1 --2- 172.21.15.161:0/1159115294 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f397006b500 0x7f397006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:11.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.442+0000 7f3991d78640 1 --2- 172.21.15.161:0/1159115294 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f397006b500 0x7f397006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:11.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.442+0000 7f3982ffd640 1 -- 172.21.15.161:0/1159115294 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 6363+0+0 (secure 0 0 0) 0x7f398808a1d0 con 0x7f398c06cf60 2024-03-28T23:33:11.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.443+0000 7f3991d78640 1 --2- 172.21.15.161:0/1159115294 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f397006b500 0x7f397006d9c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f397c036000 tx=0x7f397c03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:11.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.447+0000 7f3982ffd640 1 -- 172.21.15.161:0/1159115294 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3988058800 con 0x7f398c06cf60 2024-03-28T23:33:11.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.743+0000 7f39947dd640 1 -- 172.21.15.161:0/1159115294 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f398c154030 con 0x7f398c06cf60 2024-03-28T23:33:11.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.744+0000 7f3982ffd640 1 -- 172.21.15.161:0/1159115294 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+17320 (secure 0 0 0) 0x7f398805c8a0 con 0x7f398c06cf60 2024-03-28T23:33:11.745 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:11.745 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":82,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:33:08.559271+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:32:42.378609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:32:43.436985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6835/2691084827":"2024-03-29T23:33:06.309901+0000","172.21.15.161:6836/2378263687":"2024-03-29T23:33:05.302530+0000","172.21.15.170:6833/2691084827":"2024-03-29T23:33:06.309901+0000","172.21.15.161:6834/2378263687":"2024-03-29T23:33:05.302530+0000","172.21.15.161:6836/3850309502":"2024-03-29T23:32:54.518520+0000","172.21.15.161:6834/3850309502":"2024-03-29T23:32:54.518520+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:33:11.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.746+0000 7f39947dd640 1 -- 172.21.15.161:0/1159115294 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f397006b500 msgr2=0x7f397006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:11.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.746+0000 7f39947dd640 1 --2- 172.21.15.161:0/1159115294 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f397006b500 0x7f397006d9c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f397c036000 tx=0x7f397c03d040 comp rx=0 tx=0).stop 2024-03-28T23:33:11.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.746+0000 7f39947dd640 1 -- 172.21.15.161:0/1159115294 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f398c06cf60 msgr2=0x7f398c071a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:11.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.746+0000 7f39947dd640 1 --2- 172.21.15.161:0/1159115294 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f398c06cf60 0x7f398c071a00 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f398800e9e0 tx=0x7f398800eea0 comp rx=0 tx=0).stop 2024-03-28T23:33:11.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.746+0000 7f39947dd640 1 -- 172.21.15.161:0/1159115294 shutdown_connections 2024-03-28T23:33:11.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.746+0000 7f39947dd640 1 --2- 172.21.15.161:0/1159115294 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f397006b500 0x7f397006d9c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:11.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.746+0000 7f39947dd640 1 --2- 172.21.15.161:0/1159115294 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f398c14eca0 0x7f398c1547c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:11.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.746+0000 7f39947dd640 1 --2- 172.21.15.161:0/1159115294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f398c0c1dc0 0x7f398c071f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:11.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.746+0000 7f39947dd640 1 --2- 172.21.15.161:0/1159115294 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f398c06cf60 0x7f398c071a00 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:11.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.746+0000 7f39947dd640 1 -- 172.21.15.161:0/1159115294 >> 172.21.15.161:0/1159115294 conn(0x7f398c0c6670 msgr2=0x7f398c0aa8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:11.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.746+0000 7f39947dd640 1 -- 172.21.15.161:0/1159115294 shutdown_connections 2024-03-28T23:33:11.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.746+0000 7f39947dd640 1 -- 172.21.15.161:0/1159115294 wait complete. 2024-03-28T23:33:11.758 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs2 2024-03-28T23:33:11.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.864+0000 7fd00d08f640 1 Processor -- start 2024-03-28T23:33:11.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.864+0000 7fd00d08f640 1 -- start start 2024-03-28T23:33:11.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.864+0000 7fd00d08f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd008152890 0x7fd0080c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:11.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.864+0000 7fd00d08f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0080c8ed0 0x7fd0080c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:11.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.864+0000 7fd00d08f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd008158a30 con 0x7fd0080c5950 2024-03-28T23:33:11.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.864+0000 7fd00d08f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd0080c8480 con 0x7fd008152890 2024-03-28T23:33:11.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.864+0000 7fd00d08f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd008159640 con 0x7fd0080c8ed0 2024-03-28T23:33:11.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.865+0000 7fd00659c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0080c8ed0 0x7fd0080c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:11.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.865+0000 7fd00659c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0080c8ed0 0x7fd0080c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:45556/0 (socket says 172.21.15.161:45556) 2024-03-28T23:33:11.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.865+0000 7fd006d9d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd008152890 0x7fd0080c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:11.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.865+0000 7fd00659c640 1 -- 172.21.15.161:0/2979853679 learned_addr learned my addr 172.21.15.161:0/2979853679 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.865+0000 7fd00659c640 1 -- 172.21.15.161:0/2979853679 >> v2:172.21.15.170:3300/0 conn(0x7fd008152890 msgr2=0x7fd0080c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.865+0000 7fd00659c640 1 --2- 172.21.15.161:0/2979853679 >> v2:172.21.15.170:3300/0 conn(0x7fd008152890 0x7fd0080c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.865+0000 7fd00659c640 1 -- 172.21.15.161:0/2979853679 >> v1:172.21.15.170:6789/0 conn(0x7fd0080c5950 legacy=0x7fd0081553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.865+0000 7fd00659c640 1 -- 172.21.15.161:0/2979853679 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd008158d10 con 0x7fd0080c8ed0 2024-03-28T23:33:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.865+0000 7fd006d9d640 1 --2- 172.21.15.161:0/2979853679 >> v2:172.21.15.170:3300/0 conn(0x7fd008152890 0x7fd0080c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:11.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.866+0000 7fd00659c640 1 --2- 172.21.15.161:0/2979853679 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0080c8ed0 0x7fd0080c5410 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009850 tx=0x7fcffc02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=251219867a4bddb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:11.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.866+0000 7fd005d9b640 1 -- 172.21.15.161:0/2979853679 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcffc037070 con 0x7fd0080c8ed0 2024-03-28T23:33:11.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.866+0000 7fd005d9b640 1 -- 172.21.15.161:0/2979853679 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcffc004030 con 0x7fd0080c8ed0 2024-03-28T23:33:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.866+0000 7fd005d9b640 1 -- 172.21.15.161:0/2979853679 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcffc004460 con 0x7fd0080c8ed0 2024-03-28T23:33:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.867+0000 7fd00d08f640 1 -- 172.21.15.161:0/2979853679 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0080c8ed0 msgr2=0x7fd0080c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.867+0000 7fd00d08f640 1 --2- 172.21.15.161:0/2979853679 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0080c8ed0 0x7fd0080c5410 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009850 tx=0x7fcffc02b840 comp rx=0 tx=0).stop 2024-03-28T23:33:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.867+0000 7fd00d08f640 1 -- 172.21.15.161:0/2979853679 shutdown_connections 2024-03-28T23:33:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.867+0000 7fd00d08f640 1 --2- 172.21.15.161:0/2979853679 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0080c8ed0 0x7fd0080c5410 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.867+0000 7fd00d08f640 1 --2- 172.21.15.161:0/2979853679 >> v2:172.21.15.170:3300/0 conn(0x7fd008152890 0x7fd0080c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.867+0000 7fd00d08f640 1 -- 172.21.15.161:0/2979853679 >> 172.21.15.161:0/2979853679 conn(0x7fd0080ac620 msgr2=0x7fd0080aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.867+0000 7fd00d08f640 1 -- 172.21.15.161:0/2979853679 shutdown_connections 2024-03-28T23:33:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.867+0000 7fd00d08f640 1 -- 172.21.15.161:0/2979853679 wait complete. 2024-03-28T23:33:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.868+0000 7fd00d08f640 1 Processor -- start 2024-03-28T23:33:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.868+0000 7fd00d08f640 1 -- start start 2024-03-28T23:33:11.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.869+0000 7fd00d08f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0080c5950 0x7fd008149ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:11.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.869+0000 7fd00d08f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0080c8ed0 0x7fd0080aafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:11.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.869+0000 7fd00d08f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd008152890 0x7fd0080ab4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:11.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.869+0000 7fd00d08f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd008152c70 con 0x7fd0080c8ed0 2024-03-28T23:33:11.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.869+0000 7fd00d08f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd0081592d0 con 0x7fd0080c5950 2024-03-28T23:33:11.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.869+0000 7fd00d08f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd008152df0 con 0x7fd008152890 2024-03-28T23:33:11.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.869+0000 7fd00659c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0080c8ed0 0x7fd0080aafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:11.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.869+0000 7fd00659c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0080c8ed0 0x7fd0080aafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48790/0 (socket says 172.21.15.161:48790) 2024-03-28T23:33:11.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.869+0000 7fd00659c640 1 -- 172.21.15.161:0/4087072785 learned_addr learned my addr 172.21.15.161:0/4087072785 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:11.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.869+0000 7fd006d9d640 1 --2- 172.21.15.161:0/4087072785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0080c5950 0x7fd008149ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.869+0000 7fd00759e640 1 --2- 172.21.15.161:0/4087072785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd008152890 0x7fd0080ab4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.869+0000 7fd00659c640 1 -- 172.21.15.161:0/4087072785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd008152890 msgr2=0x7fd0080ab4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.869+0000 7fd00659c640 1 --2- 172.21.15.161:0/4087072785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd008152890 0x7fd0080ab4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.869+0000 7fd00659c640 1 -- 172.21.15.161:0/4087072785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0080c5950 msgr2=0x7fd008149ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.869+0000 7fd00659c640 1 --2- 172.21.15.161:0/4087072785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0080c5950 0x7fd008149ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.869+0000 7fd00659c640 1 -- 172.21.15.161:0/4087072785 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0081ec070 con 0x7fd0080c8ed0 2024-03-28T23:33:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.869+0000 7fd00659c640 1 --2- 172.21.15.161:0/4087072785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0080c8ed0 0x7fd0080aafb0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fcffc035040 tx=0x7fcffc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.870+0000 7fcfef7fe640 1 -- 172.21.15.161:0/4087072785 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcffc042020 con 0x7fd0080c8ed0 2024-03-28T23:33:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.870+0000 7fcfef7fe640 1 -- 172.21.15.161:0/4087072785 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcffc038040 con 0x7fd0080c8ed0 2024-03-28T23:33:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.870+0000 7fd00d08f640 1 -- 172.21.15.161:0/4087072785 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0081eb2b0 con 0x7fd0080c8ed0 2024-03-28T23:33:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.870+0000 7fcfef7fe640 1 -- 172.21.15.161:0/4087072785 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcffc037050 con 0x7fd0080c8ed0 2024-03-28T23:33:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.870+0000 7fd00d08f640 1 -- 172.21.15.161:0/4087072785 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0081eb870 con 0x7fd0080c8ed0 2024-03-28T23:33:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.871+0000 7fd00d08f640 1 -- 172.21.15.161:0/4087072785 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0081ebd60 con 0x7fd0080c8ed0 2024-03-28T23:33:11.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.871+0000 7fcfef7fe640 1 -- 172.21.15.161:0/4087072785 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcffc003cc0 con 0x7fd0080c8ed0 2024-03-28T23:33:11.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.871+0000 7fcfef7fe640 1 --2- 172.21.15.161:0/4087072785 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcff406b4d0 0x7fcff406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:11.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.871+0000 7fcfef7fe640 1 -- 172.21.15.161:0/4087072785 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 6363+0+0 (secure 0 0 0) 0x7fcffc02f030 con 0x7fd0080c8ed0 2024-03-28T23:33:11.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.872+0000 7fd006d9d640 1 --2- 172.21.15.161:0/4087072785 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcff406b4d0 0x7fcff406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:11.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.872+0000 7fd006d9d640 1 --2- 172.21.15.161:0/4087072785 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcff406b4d0 0x7fcff406d990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fcff8009560 tx=0x7fcff8009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:11.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:11.880+0000 7fcfef7fe640 1 -- 172.21.15.161:0/4087072785 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcffc079e20 con 0x7fd0080c8ed0 2024-03-28T23:33:12.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.098+0000 7fd00d08f640 1 -- 172.21.15.161:0/4087072785 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs2"} v 0) -- 0x7fd008067a50 con 0x7fd0080c8ed0 2024-03-28T23:33:12.099 INFO:tasks.ceph.mon.a.smithi161.stderr:2024-03-28T23:33:12.099+0000 7fdc7b85b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T23:33:12.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.100+0000 7fcfef7fe640 1 -- 172.21.15.161:0/4087072785 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs2"}]=0 cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v74) ==== 163+0+0 (secure 0 0 0) 0x7fd008067a50 con 0x7fd0080c8ed0 2024-03-28T23:33:12.100 INFO:teuthology.orchestra.run.smithi161.stderr:cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-28T23:33:12.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.103+0000 7fd00d08f640 1 -- 172.21.15.161:0/4087072785 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcff406b4d0 msgr2=0x7fcff406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:12.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.103+0000 7fd00d08f640 1 --2- 172.21.15.161:0/4087072785 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcff406b4d0 0x7fcff406d990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fcff8009560 tx=0x7fcff8009190 comp rx=0 tx=0).stop 2024-03-28T23:33:12.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.103+0000 7fd00d08f640 1 -- 172.21.15.161:0/4087072785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0080c8ed0 msgr2=0x7fd0080aafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:12.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.103+0000 7fd00d08f640 1 --2- 172.21.15.161:0/4087072785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0080c8ed0 0x7fd0080aafb0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fcffc035040 tx=0x7fcffc0046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:12.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.103+0000 7fd00d08f640 1 -- 172.21.15.161:0/4087072785 shutdown_connections 2024-03-28T23:33:12.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.103+0000 7fd00d08f640 1 --2- 172.21.15.161:0/4087072785 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcff406b4d0 0x7fcff406d990 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:12.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.103+0000 7fd00d08f640 1 --2- 172.21.15.161:0/4087072785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0080c8ed0 0x7fd0080aafb0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:12.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.103+0000 7fd00d08f640 1 --2- 172.21.15.161:0/4087072785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd008152890 0x7fd0080ab4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:12.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.103+0000 7fd00d08f640 1 --2- 172.21.15.161:0/4087072785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0080c5950 0x7fd008149ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:12.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.103+0000 7fd00d08f640 1 -- 172.21.15.161:0/4087072785 >> 172.21.15.161:0/4087072785 conn(0x7fd0080ac620 msgr2=0x7fd00814ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:12.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.103+0000 7fd00d08f640 1 -- 172.21.15.161:0/4087072785 shutdown_connections 2024-03-28T23:33:12.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.103+0000 7fd00d08f640 1 -- 172.21.15.161:0/4087072785 wait complete. 2024-03-28T23:33:12.115 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs2 --yes-i-really-mean-it 2024-03-28T23:33:12.129 INFO:tasks.ceph.mds.c.smithi161.stdout:starting mds.c at 2024-03-28T23:33:12.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.218+0000 7fd8c8d1e640 1 Processor -- start 2024-03-28T23:33:12.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.218+0000 7fd8c8d1e640 1 -- start start 2024-03-28T23:33:12.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.219+0000 7fd8c8d1e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd8c40a9960 0x7fd8c4072970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:12.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.219+0000 7fd8c8d1e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8c40c7330 0x7fd8c414e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:12.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.219+0000 7fd8c8d1e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8c40abc00 con 0x7fd8c40733e0 2024-03-28T23:33:12.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.219+0000 7fd8c8d1e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd8c40aa3a0 con 0x7fd8c40a9960 2024-03-28T23:33:12.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.219+0000 7fd8c8d1e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd8c406bf70 con 0x7fd8c40c7330 2024-03-28T23:33:12.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.219+0000 7fd8c259c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd8c40a9960 0x7fd8c4072970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:12.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.219+0000 7fd8c2d9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd8c40733e0 0x7fd8c414e9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:39480/0 (socket says 172.21.15.161:39480) 2024-03-28T23:33:12.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.219+0000 7fd8c2d9d640 1 -- 172.21.15.161:0/1884444428 learned_addr learned my addr 172.21.15.161:0/1884444428 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:12.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.219+0000 7fd8c1d9b640 1 --2- 172.21.15.161:0/1884444428 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8c40c7330 0x7fd8c414e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:12.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.220+0000 7fd8c259c640 1 -- 172.21.15.161:0/1884444428 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8c40c7330 msgr2=0x7fd8c414e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:12.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.220+0000 7fd8c259c640 1 --2- 172.21.15.161:0/1884444428 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8c40c7330 0x7fd8c414e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:12.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.220+0000 7fd8c259c640 1 -- 172.21.15.161:0/1884444428 >> v1:172.21.15.161:6789/0 conn(0x7fd8c40733e0 legacy=0x7fd8c414e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:12.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.220+0000 7fd8c159a640 1 -- 172.21.15.161:0/1884444428 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3516779839 0 0) 0x7fd8c40abc00 con 0x7fd8c40733e0 2024-03-28T23:33:12.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.220+0000 7fd8c259c640 1 -- 172.21.15.161:0/1884444428 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c406bcd0 con 0x7fd8c40a9960 2024-03-28T23:33:12.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.220+0000 7fd8c259c640 1 --2- 172.21.15.161:0/1884444428 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8c40a9960 0x7fd8c4072970 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fd8b40097a0 tx=0x7fd8b402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6b318085536e46 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:12.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.220+0000 7fd8c159a640 1 -- 172.21.15.161:0/1884444428 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4038070 con 0x7fd8c40a9960 2024-03-28T23:33:12.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.220+0000 7fd8c159a640 1 -- 172.21.15.161:0/1884444428 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8b4031700 con 0x7fd8c40a9960 2024-03-28T23:33:12.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.220+0000 7fd8c159a640 1 -- 172.21.15.161:0/1884444428 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4031ad0 con 0x7fd8c40a9960 2024-03-28T23:33:12.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.221+0000 7fd8c8d1e640 1 -- 172.21.15.161:0/1884444428 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8c40a9960 msgr2=0x7fd8c4072970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:12.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.221+0000 7fd8c8d1e640 1 --2- 172.21.15.161:0/1884444428 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8c40a9960 0x7fd8c4072970 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fd8b40097a0 tx=0x7fd8b402b7a0 comp rx=0 tx=0).stop 2024-03-28T23:33:12.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.221+0000 7fd8c8d1e640 1 -- 172.21.15.161:0/1884444428 shutdown_connections 2024-03-28T23:33:12.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.221+0000 7fd8c8d1e640 1 --2- 172.21.15.161:0/1884444428 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8c40c7330 0x7fd8c414e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:12.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.221+0000 7fd8c8d1e640 1 --2- 172.21.15.161:0/1884444428 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8c40a9960 0x7fd8c4072970 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:12.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.221+0000 7fd8c8d1e640 1 -- 172.21.15.161:0/1884444428 >> 172.21.15.161:0/1884444428 conn(0x7fd8c40a83f0 msgr2=0x7fd8c40a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:12.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.221+0000 7fd8c8d1e640 1 -- 172.21.15.161:0/1884444428 shutdown_connections 2024-03-28T23:33:12.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.221+0000 7fd8c8d1e640 1 -- 172.21.15.161:0/1884444428 wait complete. 2024-03-28T23:33:12.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.222+0000 7fd8c8d1e640 1 Processor -- start 2024-03-28T23:33:12.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.222+0000 7fd8c8d1e640 1 -- start start 2024-03-28T23:33:12.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.223+0000 7fd8c8d1e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8c40733e0 0x7fd8c4154770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:12.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.223+0000 7fd8c8d1e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8c40a9960 0x7fd8c40bea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:12.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.223+0000 7fd8c8d1e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8c40c7330 0x7fd8c40bef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:12.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.223+0000 7fd8c8d1e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c40ab820 con 0x7fd8c40c7330 2024-03-28T23:33:12.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.223+0000 7fd8c8d1e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c4152250 con 0x7fd8c40a9960 2024-03-28T23:33:12.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.223+0000 7fd8c8d1e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd8c40ab9a0 con 0x7fd8c40733e0 2024-03-28T23:33:12.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.223+0000 7fd8c2d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8c40c7330 0x7fd8c40bef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:12.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.223+0000 7fd8c2d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8c40c7330 0x7fd8c40bef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48822/0 (socket says 172.21.15.161:48822) 2024-03-28T23:33:12.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.223+0000 7fd8c2d9d640 1 -- 172.21.15.161:0/3482191708 learned_addr learned my addr 172.21.15.161:0/3482191708 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:12.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.223+0000 7fd8c259c640 1 --2- 172.21.15.161:0/3482191708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8c40733e0 0x7fd8c4154770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:12.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.223+0000 7fd8c1d9b640 1 --2- 172.21.15.161:0/3482191708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8c40a9960 0x7fd8c40bea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:12.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.223+0000 7fd8c2d9d640 1 -- 172.21.15.161:0/3482191708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8c40733e0 msgr2=0x7fd8c4154770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:12.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.223+0000 7fd8c2d9d640 1 --2- 172.21.15.161:0/3482191708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8c40733e0 0x7fd8c4154770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:12.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.223+0000 7fd8c2d9d640 1 -- 172.21.15.161:0/3482191708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8c40a9960 msgr2=0x7fd8c40bea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:12.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.223+0000 7fd8c2d9d640 1 --2- 172.21.15.161:0/3482191708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8c40a9960 0x7fd8c40bea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:12.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.223+0000 7fd8c2d9d640 1 -- 172.21.15.161:0/3482191708 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c41491b0 con 0x7fd8c40c7330 2024-03-28T23:33:12.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.223+0000 7fd8c1d9b640 1 --2- 172.21.15.161:0/3482191708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8c40a9960 0x7fd8c40bea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:12.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.224+0000 7fd8c2d9d640 1 --2- 172.21.15.161:0/3482191708 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8c40c7330 0x7fd8c40bef70 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc00c830 tx=0x7fd8bc00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:12.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.224+0000 7fd8b2ffd640 1 -- 172.21.15.161:0/3482191708 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc018070 con 0x7fd8c40c7330 2024-03-28T23:33:12.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.224+0000 7fd8b2ffd640 1 -- 172.21.15.161:0/3482191708 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8bc00ce20 con 0x7fd8c40c7330 2024-03-28T23:33:12.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.224+0000 7fd8c8d1e640 1 -- 172.21.15.161:0/3482191708 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c40c0470 con 0x7fd8c40c7330 2024-03-28T23:33:12.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.224+0000 7fd8b2ffd640 1 -- 172.21.15.161:0/3482191708 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc00a880 con 0x7fd8c40c7330 2024-03-28T23:33:12.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.224+0000 7fd8c8d1e640 1 -- 172.21.15.161:0/3482191708 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c40bc810 con 0x7fd8c40c7330 2024-03-28T23:33:12.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.225+0000 7fd8c8d1e640 1 -- 172.21.15.161:0/3482191708 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8c404f4b0 con 0x7fd8c40c7330 2024-03-28T23:33:12.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.225+0000 7fd8b2ffd640 1 -- 172.21.15.161:0/3482191708 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd8bc01b070 con 0x7fd8c40c7330 2024-03-28T23:33:12.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.226+0000 7fd8b2ffd640 1 --2- 172.21.15.161:0/3482191708 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd8a406b4d0 0x7fd8a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:12.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.226+0000 7fd8b2ffd640 1 -- 172.21.15.161:0/3482191708 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 6449+0+0 (secure 0 0 0) 0x7fd8bc089930 con 0x7fd8c40c7330 2024-03-28T23:33:12.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.226+0000 7fd8c259c640 1 --2- 172.21.15.161:0/3482191708 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd8a406b4d0 0x7fd8a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:12.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.226+0000 7fd8c259c640 1 --2- 172.21.15.161:0/3482191708 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd8a406b4d0 0x7fd8a406d990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4035000 tx=0x7fd8b4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:12.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.230+0000 7fd8b2ffd640 1 -- 172.21.15.161:0/3482191708 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd8bc058610 con 0x7fd8c40c7330 2024-03-28T23:33:12.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:12.437+0000 7fd8c8d1e640 1 -- 172.21.15.161:0/3482191708 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true} v 0) -- 0x7fd8c406b1b0 con 0x7fd8c40c7330 2024-03-28T23:33:13.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.102+0000 7fd8b2ffd640 1 -- 172.21.15.161:0/3482191708 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true}]=0 v75) ==== 105+0+0 (secure 0 0 0) 0x7fd8bc057f10 con 0x7fd8c40c7330 2024-03-28T23:33:13.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.104+0000 7fd8c8d1e640 1 -- 172.21.15.161:0/3482191708 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd8a406b4d0 msgr2=0x7fd8a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:13.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.104+0000 7fd8c8d1e640 1 --2- 172.21.15.161:0/3482191708 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd8a406b4d0 0x7fd8a406d990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4035000 tx=0x7fd8b4008030 comp rx=0 tx=0).stop 2024-03-28T23:33:13.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.104+0000 7fd8c8d1e640 1 -- 172.21.15.161:0/3482191708 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8c40c7330 msgr2=0x7fd8c40bef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:13.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.104+0000 7fd8c8d1e640 1 --2- 172.21.15.161:0/3482191708 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8c40c7330 0x7fd8c40bef70 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc00c830 tx=0x7fd8bc00ccf0 comp rx=0 tx=0).stop 2024-03-28T23:33:13.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.104+0000 7fd8c8d1e640 1 -- 172.21.15.161:0/3482191708 shutdown_connections 2024-03-28T23:33:13.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.104+0000 7fd8c8d1e640 1 --2- 172.21.15.161:0/3482191708 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd8a406b4d0 0x7fd8a406d990 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:13.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.104+0000 7fd8c8d1e640 1 --2- 172.21.15.161:0/3482191708 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8c40c7330 0x7fd8c40bef70 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:13.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.104+0000 7fd8c8d1e640 1 --2- 172.21.15.161:0/3482191708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8c40a9960 0x7fd8c40bea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:13.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.104+0000 7fd8c8d1e640 1 --2- 172.21.15.161:0/3482191708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8c40733e0 0x7fd8c4154770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:13.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.104+0000 7fd8c8d1e640 1 -- 172.21.15.161:0/3482191708 >> 172.21.15.161:0/3482191708 conn(0x7fd8c40a83f0 msgr2=0x7fd8c40c3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:13.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.105+0000 7fd8c8d1e640 1 -- 172.21.15.161:0/3482191708 shutdown_connections 2024-03-28T23:33:13.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.105+0000 7fd8c8d1e640 1 -- 172.21.15.161:0/3482191708 wait complete. 2024-03-28T23:33:13.118 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_metadata cephfs2_metadata --yes-i-really-really-mean-it 2024-03-28T23:33:13.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.269+0000 7efd77f16640 1 Processor -- start 2024-03-28T23:33:13.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.269+0000 7efd77f16640 1 -- start start 2024-03-28T23:33:13.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.269+0000 7efd77f16640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efd7006d1c0 0x7efd70151db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:13.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.269+0000 7efd77f16640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd700c15a0 con 0x7efd7014f1d0 2024-03-28T23:33:13.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.270+0000 7efd77f16640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd700c6330 con 0x7efd700c3080 2024-03-28T23:33:13.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.270+0000 7efd77f16640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7efd700c1840 con 0x7efd7006d1c0 2024-03-28T23:33:13.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.270+0000 7efd764b3640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efd7006d1c0 0x7efd70151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:13.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.270+0000 7efd75cb2640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7efd7014f1d0 0x7efd700c3a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:39482/0 (socket says 172.21.15.161:39482) 2024-03-28T23:33:13.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.270+0000 7efd75cb2640 1 -- 172.21.15.161:0/1182981945 learned_addr learned my addr 172.21.15.161:0/1182981945 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:13.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.270+0000 7efd764b3640 1 -- 172.21.15.161:0/1182981945 >> v1:172.21.15.170:6789/0 conn(0x7efd700c3080 legacy=0x7efd7006cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:13.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.270+0000 7efd764b3640 1 -- 172.21.15.161:0/1182981945 >> v1:172.21.15.161:6789/0 conn(0x7efd7014f1d0 legacy=0x7efd700c3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:13.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.270+0000 7efd74cb0640 1 -- 172.21.15.161:0/1182981945 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 693809982 0 0) 0x7efd700c15a0 con 0x7efd7014f1d0 2024-03-28T23:33:13.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.270+0000 7efd764b3640 1 -- 172.21.15.161:0/1182981945 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd700c1b20 con 0x7efd7006d1c0 2024-03-28T23:33:13.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.271+0000 7efd764b3640 1 --2- 172.21.15.161:0/1182981945 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd7006d1c0 0x7efd70151db0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7efd680097d0 tx=0x7efd6802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a290008eadee9ad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:13.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.271+0000 7efd74cb0640 1 -- 172.21.15.161:0/1182981945 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd68035020 con 0x7efd7006d1c0 2024-03-28T23:33:13.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.271+0000 7efd74cb0640 1 -- 172.21.15.161:0/1182981945 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd68031830 con 0x7efd7006d1c0 2024-03-28T23:33:13.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.271+0000 7efd74cb0640 1 -- 172.21.15.161:0/1182981945 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd68031c00 con 0x7efd7006d1c0 2024-03-28T23:33:13.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.271+0000 7efd77f16640 1 -- 172.21.15.161:0/1182981945 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd7006d1c0 msgr2=0x7efd70151db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:13.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.271+0000 7efd77f16640 1 --2- 172.21.15.161:0/1182981945 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd7006d1c0 0x7efd70151db0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7efd680097d0 tx=0x7efd6802b8d0 comp rx=0 tx=0).stop 2024-03-28T23:33:13.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.271+0000 7efd77f16640 1 -- 172.21.15.161:0/1182981945 shutdown_connections 2024-03-28T23:33:13.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.271+0000 7efd77f16640 1 --2- 172.21.15.161:0/1182981945 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd7006d1c0 0x7efd70151db0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:13.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.271+0000 7efd77f16640 1 -- 172.21.15.161:0/1182981945 >> 172.21.15.161:0/1182981945 conn(0x7efd700c6e20 msgr2=0x7efd700c7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:13.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.272+0000 7efd77f16640 1 -- 172.21.15.161:0/1182981945 shutdown_connections 2024-03-28T23:33:13.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.272+0000 7efd77f16640 1 -- 172.21.15.161:0/1182981945 wait complete. 2024-03-28T23:33:13.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.273+0000 7efd77f16640 1 Processor -- start 2024-03-28T23:33:13.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.273+0000 7efd77f16640 1 -- start start 2024-03-28T23:33:13.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.273+0000 7efd77f16640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7006d1c0 0x7efd70072200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:13.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.273+0000 7efd77f16640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd700c3080 0x7efd701e7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:13.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.273+0000 7efd77f16640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd7014f1d0 0x7efd701e7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:13.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.273+0000 7efd77f16640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efd700c41a0 con 0x7efd700c3080 2024-03-28T23:33:13.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.273+0000 7efd77f16640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7efd700c2600 con 0x7efd7014f1d0 2024-03-28T23:33:13.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.273+0000 7efd77f16640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7efd700c4320 con 0x7efd7006d1c0 2024-03-28T23:33:13.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.273+0000 7efd754b1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd700c3080 0x7efd701e7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:13.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.273+0000 7efd754b1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd700c3080 0x7efd701e7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48844/0 (socket says 172.21.15.161:48844) 2024-03-28T23:33:13.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.273+0000 7efd754b1640 1 -- 172.21.15.161:0/2414110005 learned_addr learned my addr 172.21.15.161:0/2414110005 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:13.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.274+0000 7efd75cb2640 1 --2- 172.21.15.161:0/2414110005 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7006d1c0 0x7efd70072200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:13.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.274+0000 7efd764b3640 1 --2- 172.21.15.161:0/2414110005 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd7014f1d0 0x7efd701e7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:13.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.274+0000 7efd754b1640 1 -- 172.21.15.161:0/2414110005 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7006d1c0 msgr2=0x7efd70072200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:13.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.274+0000 7efd754b1640 1 --2- 172.21.15.161:0/2414110005 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7006d1c0 0x7efd70072200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:13.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.274+0000 7efd754b1640 1 -- 172.21.15.161:0/2414110005 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd7014f1d0 msgr2=0x7efd701e7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:13.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.274+0000 7efd754b1640 1 --2- 172.21.15.161:0/2414110005 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd7014f1d0 0x7efd701e7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:13.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.274+0000 7efd754b1640 1 -- 172.21.15.161:0/2414110005 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd70156100 con 0x7efd700c3080 2024-03-28T23:33:13.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.274+0000 7efd754b1640 1 --2- 172.21.15.161:0/2414110005 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd700c3080 0x7efd701e7900 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7efd6000c400 tx=0x7efd6000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:13.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.274+0000 7efd667fc640 1 -- 172.21.15.161:0/2414110005 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60017070 con 0x7efd700c3080 2024-03-28T23:33:13.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.274+0000 7efd667fc640 1 -- 172.21.15.161:0/2414110005 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd600052d0 con 0x7efd700c3080 2024-03-28T23:33:13.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.274+0000 7efd77f16640 1 -- 172.21.15.161:0/2414110005 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd701eadf0 con 0x7efd700c3080 2024-03-28T23:33:13.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.274+0000 7efd667fc640 1 -- 172.21.15.161:0/2414110005 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60003830 con 0x7efd700c3080 2024-03-28T23:33:13.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.274+0000 7efd77f16640 1 -- 172.21.15.161:0/2414110005 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd701eb3b0 con 0x7efd700c3080 2024-03-28T23:33:13.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.275+0000 7efd77f16640 1 -- 172.21.15.161:0/2414110005 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd70068f90 con 0x7efd700c3080 2024-03-28T23:33:13.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.275+0000 7efd667fc640 1 -- 172.21.15.161:0/2414110005 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efd6001a070 con 0x7efd700c3080 2024-03-28T23:33:13.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.276+0000 7efd667fc640 1 --2- 172.21.15.161:0/2414110005 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efd5406b4d0 0x7efd5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:13.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.276+0000 7efd667fc640 1 -- 172.21.15.161:0/2414110005 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 6449+0+0 (secure 0 0 0) 0x7efd600897e0 con 0x7efd700c3080 2024-03-28T23:33:13.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.276+0000 7efd75cb2640 1 --2- 172.21.15.161:0/2414110005 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efd5406b4d0 0x7efd5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:13.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.276+0000 7efd75cb2640 1 --2- 172.21.15.161:0/2414110005 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efd5406b4d0 0x7efd5406d990 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7efd6c0012e0 tx=0x7efd6c007f30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:13.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.280+0000 7efd667fc640 1 -- 172.21.15.161:0/2414110005 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efd60058400 con 0x7efd700c3080 2024-03-28T23:33:13.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:13.454+0000 7efd77f16640 1 -- 172.21.15.161:0/2414110005 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7efd700c6590 con 0x7efd700c3080 2024-03-28T23:33:14.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.108+0000 7efd667fc640 1 -- 172.21.15.161:0/2414110005 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_metadata' removed v84) ==== 184+0+0 (secure 0 0 0) 0x7efd60057d00 con 0x7efd700c3080 2024-03-28T23:33:14.109 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs2_metadata' removed 2024-03-28T23:33:14.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.112+0000 7efd77f16640 1 -- 172.21.15.161:0/2414110005 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efd5406b4d0 msgr2=0x7efd5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:14.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.112+0000 7efd77f16640 1 --2- 172.21.15.161:0/2414110005 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efd5406b4d0 0x7efd5406d990 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7efd6c0012e0 tx=0x7efd6c007f30 comp rx=0 tx=0).stop 2024-03-28T23:33:14.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.112+0000 7efd77f16640 1 -- 172.21.15.161:0/2414110005 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd700c3080 msgr2=0x7efd701e7900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:14.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.112+0000 7efd77f16640 1 --2- 172.21.15.161:0/2414110005 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd700c3080 0x7efd701e7900 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7efd6000c400 tx=0x7efd6000c8c0 comp rx=0 tx=0).stop 2024-03-28T23:33:14.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.113+0000 7efd77f16640 1 -- 172.21.15.161:0/2414110005 shutdown_connections 2024-03-28T23:33:14.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.113+0000 7efd77f16640 1 --2- 172.21.15.161:0/2414110005 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efd5406b4d0 0x7efd5406d990 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:14.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.113+0000 7efd77f16640 1 --2- 172.21.15.161:0/2414110005 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd700c3080 0x7efd701e7900 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:14.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.113+0000 7efd77f16640 1 --2- 172.21.15.161:0/2414110005 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd7014f1d0 0x7efd701e7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:14.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.113+0000 7efd77f16640 1 --2- 172.21.15.161:0/2414110005 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7006d1c0 0x7efd70072200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:14.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.113+0000 7efd77f16640 1 -- 172.21.15.161:0/2414110005 >> 172.21.15.161:0/2414110005 conn(0x7efd700c6e20 msgr2=0x7efd700c7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:14.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.113+0000 7efd77f16640 1 -- 172.21.15.161:0/2414110005 shutdown_connections 2024-03-28T23:33:14.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.113+0000 7efd77f16640 1 -- 172.21.15.161:0/2414110005 wait complete. 2024-03-28T23:33:14.128 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_data cephfs2_data --yes-i-really-really-mean-it 2024-03-28T23:33:14.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.236+0000 7f4556d81640 1 Processor -- start 2024-03-28T23:33:14.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.236+0000 7f4556d81640 1 -- start start 2024-03-28T23:33:14.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.236+0000 7f4556d81640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f45500ac430 0x7f4550152880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:14.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.236+0000 7f4556d81640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f45500c30c0 0x7f45500c24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:14.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.236+0000 7f4556d81640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f45500c2a10 0x7f45500c19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:14.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.236+0000 7f4556d81640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f4550152dc0 con 0x7f45500ac430 2024-03-28T23:33:14.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.237+0000 7f4556d81640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f45500c3540 con 0x7f45500c2a10 2024-03-28T23:33:14.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.237+0000 7f4556d81640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f45500c3820 con 0x7f45500c30c0 2024-03-28T23:33:14.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.237+0000 7f4554b1d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f45500ac430 0x7f4550152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:14.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.237+0000 7f4554b1d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f45500ac430 0x7f4550152880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48856/0 (socket says 172.21.15.161:48856) 2024-03-28T23:33:14.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.237+0000 7f4554b1d640 1 -- 172.21.15.161:0/2685836753 learned_addr learned my addr 172.21.15.161:0/2685836753 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:14.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.237+0000 7f454ffff640 1 --2- 172.21.15.161:0/2685836753 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f45500c30c0 0x7f45500c24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:14.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.237+0000 7f455531e640 1 --2- 172.21.15.161:0/2685836753 >> v2:172.21.15.170:3300/0 conn(0x7f45500c2a10 0x7f45500c19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:14.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.237+0000 7f4554b1d640 1 -- 172.21.15.161:0/2685836753 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f45500c30c0 msgr2=0x7f45500c24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:14.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.237+0000 7f4554b1d640 1 --2- 172.21.15.161:0/2685836753 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f45500c30c0 0x7f45500c24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:14.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.237+0000 7f4554b1d640 1 -- 172.21.15.161:0/2685836753 >> v2:172.21.15.170:3300/0 conn(0x7f45500c2a10 msgr2=0x7f45500c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:14.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.237+0000 7f4554b1d640 1 --2- 172.21.15.161:0/2685836753 >> v2:172.21.15.170:3300/0 conn(0x7f45500c2a10 0x7f45500c19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:14.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.237+0000 7f4554b1d640 1 -- 172.21.15.161:0/2685836753 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45500c3b00 con 0x7f45500ac430 2024-03-28T23:33:14.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.237+0000 7f454ffff640 1 --2- 172.21.15.161:0/2685836753 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f45500c30c0 0x7f45500c24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:14.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.237+0000 7f455531e640 1 --2- 172.21.15.161:0/2685836753 >> v2:172.21.15.170:3300/0 conn(0x7f45500c2a10 0x7f45500c19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:14.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.238+0000 7f4554b1d640 1 --2- 172.21.15.161:0/2685836753 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45500ac430 0x7f4550152880 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f45500aa850 tx=0x7f454402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f35331f4e8a09b8c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:14.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.238+0000 7f454f7fe640 1 -- 172.21.15.161:0/2685836753 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45440317d0 con 0x7f45500ac430 2024-03-28T23:33:14.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.238+0000 7f454f7fe640 1 -- 172.21.15.161:0/2685836753 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4544037070 con 0x7f45500ac430 2024-03-28T23:33:14.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.238+0000 7f454f7fe640 1 -- 172.21.15.161:0/2685836753 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4544031b90 con 0x7f45500ac430 2024-03-28T23:33:14.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.238+0000 7f4556d81640 1 -- 172.21.15.161:0/2685836753 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45500ac430 msgr2=0x7f4550152880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:14.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.238+0000 7f4556d81640 1 --2- 172.21.15.161:0/2685836753 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45500ac430 0x7f4550152880 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f45500aa850 tx=0x7f454402b870 comp rx=0 tx=0).stop 2024-03-28T23:33:14.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.238+0000 7f4556d81640 1 -- 172.21.15.161:0/2685836753 shutdown_connections 2024-03-28T23:33:14.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.238+0000 7f4556d81640 1 --2- 172.21.15.161:0/2685836753 >> v2:172.21.15.170:3300/0 conn(0x7f45500c2a10 0x7f45500c19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:14.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.238+0000 7f4556d81640 1 --2- 172.21.15.161:0/2685836753 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f45500c30c0 0x7f45500c24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:14.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.238+0000 7f4556d81640 1 --2- 172.21.15.161:0/2685836753 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45500ac430 0x7f4550152880 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:14.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.238+0000 7f4556d81640 1 -- 172.21.15.161:0/2685836753 >> 172.21.15.161:0/2685836753 conn(0x7f455006b150 msgr2=0x7f455006b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:14.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.238+0000 7f4556d81640 1 -- 172.21.15.161:0/2685836753 shutdown_connections 2024-03-28T23:33:14.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.238+0000 7f4556d81640 1 -- 172.21.15.161:0/2685836753 wait complete. 2024-03-28T23:33:14.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.239+0000 7f4556d81640 1 Processor -- start 2024-03-28T23:33:14.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.240+0000 7f4556d81640 1 -- start start 2024-03-28T23:33:14.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.240+0000 7f4556d81640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f45500ac430 0x7f4550149760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:14.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.240+0000 7f4556d81640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45500c2a10 0x7f45500a6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:14.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.240+0000 7f4556d81640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45500c30c0 0x7f45501e6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:14.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.240+0000 7f4556d81640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f45500c6440 con 0x7f45500c30c0 2024-03-28T23:33:14.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.240+0000 7f4556d81640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f45500c1490 con 0x7f45500c2a10 2024-03-28T23:33:14.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.240+0000 7f4556d81640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f45500ac810 con 0x7f45500ac430 2024-03-28T23:33:14.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.240+0000 7f455531e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45500c30c0 0x7f45501e6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:14.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.240+0000 7f455531e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45500c30c0 0x7f45501e6a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48864/0 (socket says 172.21.15.161:48864) 2024-03-28T23:33:14.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.241+0000 7f455531e640 1 -- 172.21.15.161:0/1818942086 learned_addr learned my addr 172.21.15.161:0/1818942086 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:14.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.241+0000 7f4554b1d640 1 --2- 172.21.15.161:0/1818942086 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f45500ac430 0x7f4550149760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:14.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.241+0000 7f455531e640 1 -- 172.21.15.161:0/1818942086 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f45500ac430 msgr2=0x7f4550149760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:14.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.241+0000 7f455531e640 1 --2- 172.21.15.161:0/1818942086 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f45500ac430 0x7f4550149760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:14.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.241+0000 7f454ffff640 1 --2- 172.21.15.161:0/1818942086 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45500c2a10 0x7f45500a6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:14.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.241+0000 7f455531e640 1 -- 172.21.15.161:0/1818942086 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45500c2a10 msgr2=0x7f45500a6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:14.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.241+0000 7f455531e640 1 --2- 172.21.15.161:0/1818942086 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45500c2a10 0x7f45500a6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:14.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.241+0000 7f455531e640 1 -- 172.21.15.161:0/1818942086 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4550149eb0 con 0x7f45500c30c0 2024-03-28T23:33:14.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.241+0000 7f455531e640 1 --2- 172.21.15.161:0/1818942086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45500c30c0 0x7f45501e6a50 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f454800cd10 tx=0x7f454800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:14.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.241+0000 7f454d7fa640 1 -- 172.21.15.161:0/1818942086 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454800f680 con 0x7f45500c30c0 2024-03-28T23:33:14.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.241+0000 7f454d7fa640 1 -- 172.21.15.161:0/1818942086 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f454800f840 con 0x7f45500c30c0 2024-03-28T23:33:14.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.241+0000 7f4556d81640 1 -- 172.21.15.161:0/1818942086 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45501e9fc0 con 0x7f45500c30c0 2024-03-28T23:33:14.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.241+0000 7f454d7fa640 1 -- 172.21.15.161:0/1818942086 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454800fc30 con 0x7f45500c30c0 2024-03-28T23:33:14.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.241+0000 7f4556d81640 1 -- 172.21.15.161:0/1818942086 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45501ea580 con 0x7f45500c30c0 2024-03-28T23:33:14.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.242+0000 7f4556d81640 1 -- 172.21.15.161:0/1818942086 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45500c5dd0 con 0x7f45500c30c0 2024-03-28T23:33:14.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.243+0000 7f454d7fa640 1 -- 172.21.15.161:0/1818942086 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f45480213f0 con 0x7f45500c30c0 2024-03-28T23:33:14.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.243+0000 7f454d7fa640 1 --2- 172.21.15.161:0/1818942086 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f453406b4d0 0x7f453406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:14.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.243+0000 7f454d7fa640 1 -- 172.21.15.161:0/1818942086 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 6009+0+0 (secure 0 0 0) 0x7f454808b030 con 0x7f45500c30c0 2024-03-28T23:33:14.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.243+0000 7f4554b1d640 1 --2- 172.21.15.161:0/1818942086 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f453406b4d0 0x7f453406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:14.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.244+0000 7f4554b1d640 1 --2- 172.21.15.161:0/1818942086 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f453406b4d0 0x7f453406d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f4544036000 tx=0x7f4544008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:14.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.248+0000 7f454d7fa640 1 -- 172.21.15.161:0/1818942086 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f45500c5dd0 con 0x7f45500c30c0 2024-03-28T23:33:14.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:14.425+0000 7f4556d81640 1 -- 172.21.15.161:0/1818942086 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4550068f90 con 0x7f45500c30c0 2024-03-28T23:33:15.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.110+0000 7f454d7fa640 1 -- 172.21.15.161:0/1818942086 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_data' removed v85) ==== 172+0+0 (secure 0 0 0) 0x7f4548014030 con 0x7f45500c30c0 2024-03-28T23:33:15.111 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs2_data' removed 2024-03-28T23:33:15.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.114+0000 7f4556d81640 1 -- 172.21.15.161:0/1818942086 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f453406b4d0 msgr2=0x7f453406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:15.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.114+0000 7f4556d81640 1 --2- 172.21.15.161:0/1818942086 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f453406b4d0 0x7f453406d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f4544036000 tx=0x7f4544008030 comp rx=0 tx=0).stop 2024-03-28T23:33:15.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.114+0000 7f4556d81640 1 -- 172.21.15.161:0/1818942086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45500c30c0 msgr2=0x7f45501e6a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:15.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.114+0000 7f4556d81640 1 --2- 172.21.15.161:0/1818942086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45500c30c0 0x7f45501e6a50 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f454800cd10 tx=0x7f454800a650 comp rx=0 tx=0).stop 2024-03-28T23:33:15.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.114+0000 7f4556d81640 1 -- 172.21.15.161:0/1818942086 shutdown_connections 2024-03-28T23:33:15.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.114+0000 7f4556d81640 1 --2- 172.21.15.161:0/1818942086 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f453406b4d0 0x7f453406d990 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.114+0000 7f4556d81640 1 --2- 172.21.15.161:0/1818942086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45500c30c0 0x7f45501e6a50 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.114+0000 7f4556d81640 1 --2- 172.21.15.161:0/1818942086 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45500c2a10 0x7f45500a6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.114+0000 7f4556d81640 1 --2- 172.21.15.161:0/1818942086 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f45500ac430 0x7f4550149760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.114+0000 7f4556d81640 1 -- 172.21.15.161:0/1818942086 >> 172.21.15.161:0/1818942086 conn(0x7f455006b150 msgr2=0x7f455006b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:15.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.114+0000 7f4556d81640 1 -- 172.21.15.161:0/1818942086 shutdown_connections 2024-03-28T23:33:15.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.114+0000 7f4556d81640 1 -- 172.21.15.161:0/1818942086 wait complete. 2024-03-28T23:33:15.129 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-28T23:33:15.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.238+0000 7fbbe028f640 1 Processor -- start 2024-03-28T23:33:15.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.238+0000 7fbbe028f640 1 -- start start 2024-03-28T23:33:15.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.239+0000 7fbbe028f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fbbd814f5c0 0x7fbbd80c2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:15.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.239+0000 7fbbe028f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbbd806cdd0 0x7fbbd81524c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:15.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.239+0000 7fbbe028f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbbd80c2690 0x7fbbd8152a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:15.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.239+0000 7fbbe028f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fbbd80c74d0 con 0x7fbbd806cdd0 2024-03-28T23:33:15.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.239+0000 7fbbe028f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fbbd80c6fa0 con 0x7fbbd814f5c0 2024-03-28T23:33:15.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.239+0000 7fbbe028f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbbd8152030 con 0x7fbbd80c2690 2024-03-28T23:33:15.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.239+0000 7fbbdd82a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbbd806cdd0 0x7fbbd81524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:15.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.239+0000 7fbbdd82a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbbd806cdd0 0x7fbbd81524c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48874/0 (socket says 172.21.15.161:48874) 2024-03-28T23:33:15.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.239+0000 7fbbdd82a640 1 -- 172.21.15.161:0/3525092557 learned_addr learned my addr 172.21.15.161:0/3525092557 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:15.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.239+0000 7fbbde02b640 1 --2- 172.21.15.161:0/3525092557 >> v2:172.21.15.170:3300/0 conn(0x7fbbd814f5c0 0x7fbbd80c2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:15.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.239+0000 7fbbde82c640 1 --2- 172.21.15.161:0/3525092557 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbbd80c2690 0x7fbbd8152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:15.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.239+0000 7fbbdd82a640 1 -- 172.21.15.161:0/3525092557 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbbd80c2690 msgr2=0x7fbbd8152a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:15.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.239+0000 7fbbdd82a640 1 --2- 172.21.15.161:0/3525092557 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbbd80c2690 0x7fbbd8152a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.239+0000 7fbbdd82a640 1 -- 172.21.15.161:0/3525092557 >> v2:172.21.15.170:3300/0 conn(0x7fbbd814f5c0 msgr2=0x7fbbd80c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:15.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.239+0000 7fbbdd82a640 1 --2- 172.21.15.161:0/3525092557 >> v2:172.21.15.170:3300/0 conn(0x7fbbd814f5c0 0x7fbbd80c2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.239+0000 7fbbdd82a640 1 -- 172.21.15.161:0/3525092557 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbd80c2f10 con 0x7fbbd806cdd0 2024-03-28T23:33:15.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.240+0000 7fbbdd82a640 1 --2- 172.21.15.161:0/3525092557 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbd806cdd0 0x7fbbd81524c0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fbbd80c76e0 tx=0x7fbbc802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a453d16c6472cb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:15.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.240+0000 7fbbdd029640 1 -- 172.21.15.161:0/3525092557 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc8035020 con 0x7fbbd806cdd0 2024-03-28T23:33:15.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.240+0000 7fbbdd029640 1 -- 172.21.15.161:0/3525092557 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbc8031870 con 0x7fbbd806cdd0 2024-03-28T23:33:15.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.240+0000 7fbbdd029640 1 -- 172.21.15.161:0/3525092557 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc8031c40 con 0x7fbbd806cdd0 2024-03-28T23:33:15.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.241+0000 7fbbe028f640 1 -- 172.21.15.161:0/3525092557 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbd806cdd0 msgr2=0x7fbbd81524c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:15.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.241+0000 7fbbe028f640 1 --2- 172.21.15.161:0/3525092557 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbd806cdd0 0x7fbbd81524c0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fbbd80c76e0 tx=0x7fbbc802b910 comp rx=0 tx=0).stop 2024-03-28T23:33:15.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.241+0000 7fbbe028f640 1 -- 172.21.15.161:0/3525092557 shutdown_connections 2024-03-28T23:33:15.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.241+0000 7fbbe028f640 1 --2- 172.21.15.161:0/3525092557 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbbd80c2690 0x7fbbd8152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.241+0000 7fbbe028f640 1 --2- 172.21.15.161:0/3525092557 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbd806cdd0 0x7fbbd81524c0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.241+0000 7fbbe028f640 1 --2- 172.21.15.161:0/3525092557 >> v2:172.21.15.170:3300/0 conn(0x7fbbd814f5c0 0x7fbbd80c2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.241+0000 7fbbe028f640 1 -- 172.21.15.161:0/3525092557 >> 172.21.15.161:0/3525092557 conn(0x7fbbd80a83f0 msgr2=0x7fbbd80a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:15.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.241+0000 7fbbe028f640 1 -- 172.21.15.161:0/3525092557 shutdown_connections 2024-03-28T23:33:15.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.241+0000 7fbbe028f640 1 -- 172.21.15.161:0/3525092557 wait complete. 2024-03-28T23:33:15.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.242+0000 7fbbe028f640 1 Processor -- start 2024-03-28T23:33:15.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.242+0000 7fbbe028f640 1 -- start start 2024-03-28T23:33:15.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.243+0000 7fbbe028f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbbd806cdd0 0x7fbbd81eac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:15.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.243+0000 7fbbe028f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbd80c2690 0x7fbbd815ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:15.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.243+0000 7fbbe028f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbbd814f5c0 0x7fbbd814b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:15.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.243+0000 7fbbe028f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd806c140 con 0x7fbbd80c2690 2024-03-28T23:33:15.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.243+0000 7fbbe028f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd814f9a0 con 0x7fbbd814f5c0 2024-03-28T23:33:15.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.243+0000 7fbbe028f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbbd806c3e0 con 0x7fbbd806cdd0 2024-03-28T23:33:15.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.243+0000 7fbbdd82a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbd80c2690 0x7fbbd815ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:15.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.243+0000 7fbbdd82a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbd80c2690 0x7fbbd815ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48880/0 (socket says 172.21.15.161:48880) 2024-03-28T23:33:15.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.243+0000 7fbbdd82a640 1 -- 172.21.15.161:0/2217239427 learned_addr learned my addr 172.21.15.161:0/2217239427 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:15.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.243+0000 7fbbde82c640 1 --2- 172.21.15.161:0/2217239427 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbbd814f5c0 0x7fbbd814b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:15.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.243+0000 7fbbde02b640 1 --2- 172.21.15.161:0/2217239427 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbbd806cdd0 0x7fbbd81eac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:15.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.243+0000 7fbbdd82a640 1 -- 172.21.15.161:0/2217239427 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbbd806cdd0 msgr2=0x7fbbd81eac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:15.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.243+0000 7fbbdd82a640 1 --2- 172.21.15.161:0/2217239427 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbbd806cdd0 0x7fbbd81eac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.243+0000 7fbbdd82a640 1 -- 172.21.15.161:0/2217239427 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbbd814f5c0 msgr2=0x7fbbd814b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:15.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.243+0000 7fbbdd82a640 1 --2- 172.21.15.161:0/2217239427 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbbd814f5c0 0x7fbbd814b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.243+0000 7fbbdd82a640 1 -- 172.21.15.161:0/2217239427 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbd814c0b0 con 0x7fbbd80c2690 2024-03-28T23:33:15.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.243+0000 7fbbdd82a640 1 --2- 172.21.15.161:0/2217239427 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbd80c2690 0x7fbbd815ae80 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fbbc802b950 tx=0x7fbbc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:15.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.243+0000 7fbbceffd640 1 -- 172.21.15.161:0/2217239427 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc8040070 con 0x7fbbd80c2690 2024-03-28T23:33:15.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.243+0000 7fbbceffd640 1 -- 172.21.15.161:0/2217239427 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbc8031e10 con 0x7fbbd80c2690 2024-03-28T23:33:15.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.243+0000 7fbbe028f640 1 -- 172.21.15.161:0/2217239427 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbd814c390 con 0x7fbbd80c2690 2024-03-28T23:33:15.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.243+0000 7fbbceffd640 1 -- 172.21.15.161:0/2217239427 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc8035040 con 0x7fbbd80c2690 2024-03-28T23:33:15.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.243+0000 7fbbe028f640 1 -- 172.21.15.161:0/2217239427 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbd81ee580 con 0x7fbbd80c2690 2024-03-28T23:33:15.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.244+0000 7fbbe028f640 1 -- 172.21.15.161:0/2217239427 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbd804f4b0 con 0x7fbbd80c2690 2024-03-28T23:33:15.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.245+0000 7fbbceffd640 1 -- 172.21.15.161:0/2217239427 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbbc804b060 con 0x7fbbd80c2690 2024-03-28T23:33:15.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.245+0000 7fbbceffd640 1 --2- 172.21.15.161:0/2217239427 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbbbc06b500 0x7fbbbc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:15.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.245+0000 7fbbceffd640 1 -- 172.21.15.161:0/2217239427 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5593+0+0 (secure 0 0 0) 0x7fbbc80abce0 con 0x7fbbd80c2690 2024-03-28T23:33:15.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.245+0000 7fbbde02b640 1 --2- 172.21.15.161:0/2217239427 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbbbc06b500 0x7fbbbc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:15.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.246+0000 7fbbde02b640 1 --2- 172.21.15.161:0/2217239427 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbbbc06b500 0x7fbbbc06d9c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fbbd8151e40 tx=0x7fbbd4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:15.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.250+0000 7fbbceffd640 1 -- 172.21.15.161:0/2217239427 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbbd80c2f10 con 0x7fbbd80c2690 2024-03-28T23:33:15.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.425+0000 7fbbe028f640 1 -- 172.21.15.161:0/2217239427 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fbbd80736d0 con 0x7fbbd80c2690 2024-03-28T23:33:15.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.428+0000 7fbbceffd640 1 -- 172.21.15.161:0/2217239427 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) ==== 104+0+0 (secure 0 0 0) 0x7fbbc802f320 con 0x7fbbd80c2690 2024-03-28T23:33:15.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.430+0000 7fbbe028f640 1 -- 172.21.15.161:0/2217239427 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbbbc06b500 msgr2=0x7fbbbc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:15.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.430+0000 7fbbe028f640 1 --2- 172.21.15.161:0/2217239427 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbbbc06b500 0x7fbbbc06d9c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fbbd8151e40 tx=0x7fbbd4008040 comp rx=0 tx=0).stop 2024-03-28T23:33:15.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.430+0000 7fbbe028f640 1 -- 172.21.15.161:0/2217239427 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbd80c2690 msgr2=0x7fbbd815ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:15.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.430+0000 7fbbe028f640 1 --2- 172.21.15.161:0/2217239427 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbd80c2690 0x7fbbd815ae80 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fbbc802b950 tx=0x7fbbc80046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.431+0000 7fbbe028f640 1 -- 172.21.15.161:0/2217239427 shutdown_connections 2024-03-28T23:33:15.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.431+0000 7fbbe028f640 1 --2- 172.21.15.161:0/2217239427 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbbbc06b500 0x7fbbbc06d9c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.431+0000 7fbbe028f640 1 --2- 172.21.15.161:0/2217239427 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbd80c2690 0x7fbbd815ae80 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.431+0000 7fbbe028f640 1 --2- 172.21.15.161:0/2217239427 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbbd814f5c0 0x7fbbd814b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.431+0000 7fbbe028f640 1 --2- 172.21.15.161:0/2217239427 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbbd806cdd0 0x7fbbd81eac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.431+0000 7fbbe028f640 1 -- 172.21.15.161:0/2217239427 >> 172.21.15.161:0/2217239427 conn(0x7fbbd80a83f0 msgr2=0x7fbbd815bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:15.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.431+0000 7fbbe028f640 1 -- 172.21.15.161:0/2217239427 shutdown_connections 2024-03-28T23:33:15.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.431+0000 7fbbe028f640 1 -- 172.21.15.161:0/2217239427 wait complete. 2024-03-28T23:33:15.442 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-03-28T23:33:15.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.545+0000 7f2e56bbc640 1 Processor -- start 2024-03-28T23:33:15.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.545+0000 7f2e56bbc640 1 -- start start 2024-03-28T23:33:15.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.546+0000 7f2e56bbc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2e50156520 0x7f2e500ab730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:15.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.546+0000 7f2e56bbc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e500c27a0 0x7f2e50156900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:15.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.546+0000 7f2e56bbc640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e500c3560 con 0x7f2e500c3c90 2024-03-28T23:33:15.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.546+0000 7f2e56bbc640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f2e500abc70 con 0x7f2e50156520 2024-03-28T23:33:15.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.546+0000 7f2e56bbc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2e50157180 con 0x7f2e500c27a0 2024-03-28T23:33:15.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.546+0000 7f2e54958640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2e50156520 0x7f2e500ab730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:15.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.546+0000 7f2e47fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e500c27a0 0x7f2e50156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:15.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.546+0000 7f2e54958640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2e50156520 0x7f2e500ab730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:35568/0 (socket says 172.21.15.161:35568) 2024-03-28T23:33:15.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.546+0000 7f2e54958640 1 -- 172.21.15.161:0/3439119094 learned_addr learned my addr 172.21.15.161:0/3439119094 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:15.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.546+0000 7f2e54958640 1 -- 172.21.15.161:0/3439119094 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e500c27a0 msgr2=0x7f2e50156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:15.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.546+0000 7f2e54958640 1 --2- 172.21.15.161:0/3439119094 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e500c27a0 0x7f2e50156900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.546+0000 7f2e54958640 1 -- 172.21.15.161:0/3439119094 >> v1:172.21.15.170:6789/0 conn(0x7f2e500c3c90 legacy=0x7f2e500c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:15.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.546+0000 7f2e54958640 1 -- 172.21.15.161:0/3439119094 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e500c2e70 con 0x7f2e50156520 2024-03-28T23:33:15.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.546+0000 7f2e47fff640 1 --2- 172.21.15.161:0/3439119094 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e500c27a0 0x7f2e50156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:15.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.547+0000 7f2e54958640 1 --2- 172.21.15.161:0/3439119094 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e50156520 0x7f2e500ab730 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c009870 tx=0x7f2e4c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cca6bea0fbc2dff3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:15.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.547+0000 7f2e477fe640 1 -- 172.21.15.161:0/3439119094 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c034070 con 0x7f2e50156520 2024-03-28T23:33:15.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.547+0000 7f2e477fe640 1 -- 172.21.15.161:0/3439119094 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e4c030840 con 0x7f2e50156520 2024-03-28T23:33:15.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.547+0000 7f2e477fe640 1 -- 172.21.15.161:0/3439119094 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c030c10 con 0x7f2e50156520 2024-03-28T23:33:15.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.548+0000 7f2e56bbc640 1 -- 172.21.15.161:0/3439119094 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e50156520 msgr2=0x7f2e500ab730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:15.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.548+0000 7f2e56bbc640 1 --2- 172.21.15.161:0/3439119094 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e50156520 0x7f2e500ab730 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c009870 tx=0x7f2e4c02b980 comp rx=0 tx=0).stop 2024-03-28T23:33:15.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.548+0000 7f2e56bbc640 1 -- 172.21.15.161:0/3439119094 shutdown_connections 2024-03-28T23:33:15.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.548+0000 7f2e56bbc640 1 --2- 172.21.15.161:0/3439119094 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e500c27a0 0x7f2e50156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.548+0000 7f2e56bbc640 1 --2- 172.21.15.161:0/3439119094 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e50156520 0x7f2e500ab730 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.548+0000 7f2e56bbc640 1 -- 172.21.15.161:0/3439119094 >> 172.21.15.161:0/3439119094 conn(0x7f2e500c4250 msgr2=0x7f2e500c4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:15.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.548+0000 7f2e56bbc640 1 -- 172.21.15.161:0/3439119094 shutdown_connections 2024-03-28T23:33:15.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.548+0000 7f2e56bbc640 1 -- 172.21.15.161:0/3439119094 wait complete. 2024-03-28T23:33:15.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.549+0000 7f2e56bbc640 1 Processor -- start 2024-03-28T23:33:15.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.549+0000 7f2e56bbc640 1 -- start start 2024-03-28T23:33:15.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.550+0000 7f2e56bbc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e500c27a0 0x7f2e50146e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:15.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.550+0000 7f2e56bbc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e500c3c90 0x7f2e50153070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:15.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.550+0000 7f2e56bbc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e50156520 0x7f2e501535b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:15.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.550+0000 7f2e56bbc640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2e50156e40 con 0x7f2e500c3c90 2024-03-28T23:33:15.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.550+0000 7f2e56bbc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5014d6a0 con 0x7f2e500c27a0 2024-03-28T23:33:15.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.550+0000 7f2e56bbc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2e5015d8b0 con 0x7f2e50156520 2024-03-28T23:33:15.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.550+0000 7f2e47fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e500c3c90 0x7f2e50153070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:15.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.550+0000 7f2e54958640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e500c27a0 0x7f2e50146e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:15.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.550+0000 7f2e47fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e500c3c90 0x7f2e50153070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48890/0 (socket says 172.21.15.161:48890) 2024-03-28T23:33:15.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.550+0000 7f2e47fff640 1 -- 172.21.15.161:0/1717393170 learned_addr learned my addr 172.21.15.161:0/1717393170 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:15.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.550+0000 7f2e55159640 1 --2- 172.21.15.161:0/1717393170 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e50156520 0x7f2e501535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:15.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.550+0000 7f2e47fff640 1 -- 172.21.15.161:0/1717393170 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e50156520 msgr2=0x7f2e501535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:15.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.550+0000 7f2e47fff640 1 --2- 172.21.15.161:0/1717393170 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e50156520 0x7f2e501535b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.550+0000 7f2e47fff640 1 -- 172.21.15.161:0/1717393170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e500c27a0 msgr2=0x7f2e50146e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:15.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.550+0000 7f2e47fff640 1 --2- 172.21.15.161:0/1717393170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e500c27a0 0x7f2e50146e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.550+0000 7f2e47fff640 1 -- 172.21.15.161:0/1717393170 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e501491f0 con 0x7f2e500c3c90 2024-03-28T23:33:15.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.550+0000 7f2e54958640 1 --2- 172.21.15.161:0/1717393170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e500c27a0 0x7f2e50146e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:15.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.550+0000 7f2e47fff640 1 --2- 172.21.15.161:0/1717393170 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e500c3c90 0x7f2e50153070 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f2e4000e900 tx=0x7f2e4000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:15.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.551+0000 7f2e457fa640 1 -- 172.21.15.161:0/1717393170 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e40017070 con 0x7f2e500c3c90 2024-03-28T23:33:15.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.551+0000 7f2e457fa640 1 -- 172.21.15.161:0/1717393170 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e4000ca00 con 0x7f2e500c3c90 2024-03-28T23:33:15.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.551+0000 7f2e457fa640 1 -- 172.21.15.161:0/1717393170 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4000cdf0 con 0x7f2e500c3c90 2024-03-28T23:33:15.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.551+0000 7f2e56bbc640 1 -- 172.21.15.161:0/1717393170 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e500c6d90 con 0x7f2e500c3c90 2024-03-28T23:33:15.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.551+0000 7f2e56bbc640 1 -- 172.21.15.161:0/1717393170 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e501489f0 con 0x7f2e500c3c90 2024-03-28T23:33:15.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.552+0000 7f2e56bbc640 1 -- 172.21.15.161:0/1717393170 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e5004f4b0 con 0x7f2e500c3c90 2024-03-28T23:33:15.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.552+0000 7f2e457fa640 1 -- 172.21.15.161:0/1717393170 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2e4001a070 con 0x7f2e500c3c90 2024-03-28T23:33:15.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.552+0000 7f2e457fa640 1 --2- 172.21.15.161:0/1717393170 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e3c06b4d0 0x7f2e3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:15.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.552+0000 7f2e457fa640 1 -- 172.21.15.161:0/1717393170 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5593+0+0 (secure 0 0 0) 0x7f2e40089be0 con 0x7f2e500c3c90 2024-03-28T23:33:15.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.553+0000 7f2e54958640 1 --2- 172.21.15.161:0/1717393170 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e3c06b4d0 0x7f2e3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:15.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.553+0000 7f2e54958640 1 --2- 172.21.15.161:0/1717393170 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e3c06b4d0 0x7f2e3c06d990 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c036000 tx=0x7f2e4c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:15.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.557+0000 7f2e457fa640 1 -- 172.21.15.161:0/1717393170 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2e40058490 con 0x7f2e500c3c90 2024-03-28T23:33:15.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.735+0000 7f2e56bbc640 1 -- 172.21.15.161:0/1717393170 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) -- 0x7f2e5014aa70 con 0x7f2e500c3c90 2024-03-28T23:33:15.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.738+0000 7f2e457fa640 1 -- 172.21.15.161:0/1717393170 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v28) ==== 96+0+0 (secure 0 0 0) 0x7f2e4005c530 con 0x7f2e500c3c90 2024-03-28T23:33:15.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.740+0000 7f2e56bbc640 1 -- 172.21.15.161:0/1717393170 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e3c06b4d0 msgr2=0x7f2e3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:15.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.740+0000 7f2e56bbc640 1 --2- 172.21.15.161:0/1717393170 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e3c06b4d0 0x7f2e3c06d990 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c036000 tx=0x7f2e4c008030 comp rx=0 tx=0).stop 2024-03-28T23:33:15.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.740+0000 7f2e56bbc640 1 -- 172.21.15.161:0/1717393170 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e500c3c90 msgr2=0x7f2e50153070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:15.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.740+0000 7f2e56bbc640 1 --2- 172.21.15.161:0/1717393170 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e500c3c90 0x7f2e50153070 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f2e4000e900 tx=0x7f2e4000edc0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.740+0000 7f2e56bbc640 1 -- 172.21.15.161:0/1717393170 shutdown_connections 2024-03-28T23:33:15.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.740+0000 7f2e56bbc640 1 --2- 172.21.15.161:0/1717393170 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2e3c06b4d0 0x7f2e3c06d990 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.740+0000 7f2e56bbc640 1 --2- 172.21.15.161:0/1717393170 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e500c3c90 0x7f2e50153070 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.740+0000 7f2e56bbc640 1 --2- 172.21.15.161:0/1717393170 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e50156520 0x7f2e501535b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.740+0000 7f2e56bbc640 1 --2- 172.21.15.161:0/1717393170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e500c27a0 0x7f2e50146e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.740+0000 7f2e56bbc640 1 -- 172.21.15.161:0/1717393170 >> 172.21.15.161:0/1717393170 conn(0x7f2e500c4250 msgr2=0x7f2e5015cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:15.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.740+0000 7f2e56bbc640 1 -- 172.21.15.161:0/1717393170 shutdown_connections 2024-03-28T23:33:15.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.740+0000 7f2e56bbc640 1 -- 172.21.15.161:0/1717393170 wait complete. 2024-03-28T23:33:15.752 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.c mds_join_fs 2024-03-28T23:33:15.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.853+0000 7f3422e23640 1 Processor -- start 2024-03-28T23:33:15.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.854+0000 7f3422e23640 1 -- start start 2024-03-28T23:33:15.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.854+0000 7f3422e23640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f341c14f5c0 0x7f341c0c2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:15.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.854+0000 7f3422e23640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f341c152bd0 0x7f341c15c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:15.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.854+0000 7f3422e23640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f341c0c2f10 con 0x7f341c06cdd0 2024-03-28T23:33:15.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.854+0000 7f3422e23640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f341c0c74d0 con 0x7f341c14f5c0 2024-03-28T23:33:15.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.854+0000 7f3422e23640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f341c0c6fa0 con 0x7f341c152bd0 2024-03-28T23:33:15.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.854+0000 7f341bfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f341c06cdd0 0x7f341c1524c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:39486/0 (socket says 172.21.15.161:39486) 2024-03-28T23:33:15.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.854+0000 7f341bfff640 1 -- 172.21.15.161:0/1218910874 learned_addr learned my addr 172.21.15.161:0/1218910874 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:15.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f34213c0640 1 --2- 172.21.15.161:0/1218910874 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f341c152bd0 0x7f341c15c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:15.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f3420bbf640 1 --2- 172.21.15.161:0/1218910874 >> v2:172.21.15.170:3300/0 conn(0x7f341c14f5c0 0x7f341c0c2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:15.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f341b7fe640 1 -- 172.21.15.161:0/1218910874 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2257265961 0 0) 0x7f3408001110 con 0x7f341c06cdd0 2024-03-28T23:33:15.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f341b7fe640 1 -- 172.21.15.161:0/1218910874 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f340c003520 con 0x7f341c06cdd0 2024-03-28T23:33:15.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f341b7fe640 1 -- 172.21.15.161:0/1218910874 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1847230045 0 0) 0x7f340c003520 con 0x7f341c06cdd0 2024-03-28T23:33:15.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f341b7fe640 1 -- 172.21.15.161:0/1218910874 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f341c152bd0 msgr2=0x7f341c15c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:15.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f341b7fe640 1 --2- 172.21.15.161:0/1218910874 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f341c152bd0 0x7f341c15c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f341b7fe640 1 -- 172.21.15.161:0/1218910874 >> v2:172.21.15.170:3300/0 conn(0x7f341c14f5c0 msgr2=0x7f341c0c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:15.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f341b7fe640 1 --2- 172.21.15.161:0/1218910874 >> v2:172.21.15.170:3300/0 conn(0x7f341c14f5c0 0x7f341c0c2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f341b7fe640 1 -- 172.21.15.161:0/1218910874 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f341c14f9a0 con 0x7f341c06cdd0 2024-03-28T23:33:15.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f341b7fe640 1 -- 172.21.15.161:0/1218910874 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f3408001840 con 0x7f341c06cdd0 2024-03-28T23:33:15.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f3420bbf640 1 --2- 172.21.15.161:0/1218910874 >> v2:172.21.15.170:3300/0 conn(0x7f341c14f5c0 0x7f341c0c2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:15.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f341b7fe640 1 -- 172.21.15.161:0/1218910874 >> v1:172.21.15.161:6789/0 conn(0x7f341c06cdd0 legacy=0x7f341c1524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:15.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f341b7fe640 1 --2- 172.21.15.161:0/1218910874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f340c003a90 0x7f340c003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:15.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f341b7fe640 1 --2- 172.21.15.161:0/1218910874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f340c004800 0x7f340c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:15.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f341b7fe640 1 --2- 172.21.15.161:0/1218910874 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f340c0051c0 0x7f340c00dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:15.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f341b7fe640 1 -- 172.21.15.161:0/1218910874 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f341c0c74d0 con 0x7f340c004800 2024-03-28T23:33:15.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f341b7fe640 1 -- 172.21.15.161:0/1218910874 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f341c0c6fa0 con 0x7f340c003a90 2024-03-28T23:33:15.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f341b7fe640 1 -- 172.21.15.161:0/1218910874 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f340c00e190 con 0x7f340c0051c0 2024-03-28T23:33:15.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f34213c0640 1 --2- 172.21.15.161:0/1218910874 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f341c152bd0 0x7f341c15c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:15.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f341bfff640 1 --2- 172.21.15.161:0/1218910874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f340c004800 0x7f340c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:15.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f3420bbf640 1 --2- 172.21.15.161:0/1218910874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f340c003a90 0x7f340c003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:15.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f34213c0640 1 --2- 172.21.15.161:0/1218910874 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f340c0051c0 0x7f340c00dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:15.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f341bfff640 1 -- 172.21.15.161:0/1218910874 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f340c0051c0 msgr2=0x7f340c00dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:15.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f341bfff640 1 --2- 172.21.15.161:0/1218910874 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f340c0051c0 0x7f340c00dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f341bfff640 1 -- 172.21.15.161:0/1218910874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f340c003a90 msgr2=0x7f340c003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:15.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f341bfff640 1 --2- 172.21.15.161:0/1218910874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f340c003a90 0x7f340c003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.855+0000 7f341bfff640 1 -- 172.21.15.161:0/1218910874 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f340c003520 con 0x7f340c004800 2024-03-28T23:33:15.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.856+0000 7f341bfff640 1 --2- 172.21.15.161:0/1218910874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f340c004800 0x7f340c004c80 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f340800c3b0 tx=0x7f340c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d50862b15ef0c6d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:15.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.856+0000 7f341b7fe640 1 -- 172.21.15.161:0/1218910874 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f340800c0e0 con 0x7f340c004800 2024-03-28T23:33:15.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.856+0000 7f341b7fe640 1 -- 172.21.15.161:0/1218910874 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3408030070 con 0x7f340c004800 2024-03-28T23:33:15.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.856+0000 7f341b7fe640 1 -- 172.21.15.161:0/1218910874 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34080058d0 con 0x7f340c004800 2024-03-28T23:33:15.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.857+0000 7f3422e23640 1 -- 172.21.15.161:0/1218910874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f340c004800 msgr2=0x7f340c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:15.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.857+0000 7f3422e23640 1 --2- 172.21.15.161:0/1218910874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f340c004800 0x7f340c004c80 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f340800c3b0 tx=0x7f340c005830 comp rx=0 tx=0).stop 2024-03-28T23:33:15.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.857+0000 7f34213c0640 1 -- 172.21.15.161:0/1218910874 reap_dead start 2024-03-28T23:33:15.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.857+0000 7f3422e23640 1 -- 172.21.15.161:0/1218910874 shutdown_connections 2024-03-28T23:33:15.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.857+0000 7f3422e23640 1 -- 172.21.15.161:0/1218910874 >> 172.21.15.161:0/1218910874 conn(0x7f341c0a83f0 msgr2=0x7f341c0a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:15.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.857+0000 7f3422e23640 1 -- 172.21.15.161:0/1218910874 shutdown_connections 2024-03-28T23:33:15.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.857+0000 7f3422e23640 1 -- 172.21.15.161:0/1218910874 wait complete. 2024-03-28T23:33:15.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.858+0000 7f3422e23640 1 Processor -- start 2024-03-28T23:33:15.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.858+0000 7f3422e23640 1 -- start start 2024-03-28T23:33:15.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.859+0000 7f3422e23640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f340c004800 0x7f341c1dd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:15.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.859+0000 7f3422e23640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f341c071f00 0x7f341c149860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:15.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.859+0000 7f3422e23640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f341c14c5e0 0x7f341c149da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:15.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.859+0000 7f3422e23640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f341c06c140 con 0x7f341c14c5e0 2024-03-28T23:33:15.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.859+0000 7f3422e23640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f341c152030 con 0x7f340c004800 2024-03-28T23:33:15.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.859+0000 7f3422e23640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f341c06c3e0 con 0x7f341c071f00 2024-03-28T23:33:15.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.859+0000 7f34213c0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f341c14c5e0 0x7f341c149da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:15.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.859+0000 7f34213c0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f341c14c5e0 0x7f341c149da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48908/0 (socket says 172.21.15.161:48908) 2024-03-28T23:33:15.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.859+0000 7f34213c0640 1 -- 172.21.15.161:0/2572100333 learned_addr learned my addr 172.21.15.161:0/2572100333 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:15.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.859+0000 7f341bfff640 1 --2- 172.21.15.161:0/2572100333 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f341c071f00 0x7f341c149860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:15.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.859+0000 7f3420bbf640 1 --2- 172.21.15.161:0/2572100333 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f340c004800 0x7f341c1dd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:15.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.859+0000 7f34213c0640 1 -- 172.21.15.161:0/2572100333 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f341c071f00 msgr2=0x7f341c149860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:15.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.859+0000 7f34213c0640 1 --2- 172.21.15.161:0/2572100333 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f341c071f00 0x7f341c149860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.859+0000 7f34213c0640 1 -- 172.21.15.161:0/2572100333 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f340c004800 msgr2=0x7f341c1dd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:15.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.859+0000 7f34213c0640 1 --2- 172.21.15.161:0/2572100333 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f340c004800 0x7f341c1dd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:15.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.859+0000 7f34213c0640 1 -- 172.21.15.161:0/2572100333 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f341c072400 con 0x7f341c14c5e0 2024-03-28T23:33:15.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.859+0000 7f34213c0640 1 --2- 172.21.15.161:0/2572100333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f341c14c5e0 0x7f341c149da0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f341c152fb0 tx=0x7f34140115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:15.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.859+0000 7f34197fa640 1 -- 172.21.15.161:0/2572100333 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3414011ea0 con 0x7f341c14c5e0 2024-03-28T23:33:15.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.859+0000 7f34197fa640 1 -- 172.21.15.161:0/2572100333 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f341401b070 con 0x7f341c14c5e0 2024-03-28T23:33:15.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.859+0000 7f34197fa640 1 -- 172.21.15.161:0/2572100333 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3414003990 con 0x7f341c14c5e0 2024-03-28T23:33:15.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.859+0000 7f3422e23640 1 -- 172.21.15.161:0/2572100333 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f341c070740 con 0x7f341c14c5e0 2024-03-28T23:33:15.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.860+0000 7f3422e23640 1 -- 172.21.15.161:0/2572100333 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f341c070d00 con 0x7f341c14c5e0 2024-03-28T23:33:15.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.861+0000 7f3422e23640 1 -- 172.21.15.161:0/2572100333 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f341c0ab060 con 0x7f341c14c5e0 2024-03-28T23:33:15.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.861+0000 7f34197fa640 1 -- 172.21.15.161:0/2572100333 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3414005d10 con 0x7f341c14c5e0 2024-03-28T23:33:15.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.861+0000 7f34197fa640 1 --2- 172.21.15.161:0/2572100333 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f340006b500 0x7f340006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:15.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.861+0000 7f34197fa640 1 -- 172.21.15.161:0/2572100333 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5593+0+0 (secure 0 0 0) 0x7f341408a790 con 0x7f341c14c5e0 2024-03-28T23:33:15.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.862+0000 7f3420bbf640 1 --2- 172.21.15.161:0/2572100333 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f340006b500 0x7f340006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:15.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.862+0000 7f3420bbf640 1 --2- 172.21.15.161:0/2572100333 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f340006b500 0x7f340006d9c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f3410009c10 tx=0x7f3410008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:15.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:15.866+0000 7f34197fa640 1 -- 172.21.15.161:0/2572100333 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f341c0ab060 con 0x7f341c14c5e0 2024-03-28T23:33:16.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.037+0000 7f3422e23640 1 -- 172.21.15.161:0/2572100333 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"} v 0) -- 0x7f341c04f4b0 con 0x7f341c14c5e0 2024-03-28T23:33:16.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.041+0000 7f34197fa640 1 -- 172.21.15.161:0/2572100333 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"}]=0 v29) ==== 96+0+0 (secure 0 0 0) 0x7f3414025020 con 0x7f341c14c5e0 2024-03-28T23:33:16.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.043+0000 7f3422e23640 1 -- 172.21.15.161:0/2572100333 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f340006b500 msgr2=0x7f340006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:16.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.043+0000 7f3422e23640 1 --2- 172.21.15.161:0/2572100333 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f340006b500 0x7f340006d9c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f3410009c10 tx=0x7f3410008040 comp rx=0 tx=0).stop 2024-03-28T23:33:16.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.043+0000 7f3422e23640 1 -- 172.21.15.161:0/2572100333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f341c14c5e0 msgr2=0x7f341c149da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:16.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.043+0000 7f3422e23640 1 --2- 172.21.15.161:0/2572100333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f341c14c5e0 0x7f341c149da0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f341c152fb0 tx=0x7f34140115a0 comp rx=0 tx=0).stop 2024-03-28T23:33:16.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.044+0000 7f3422e23640 1 -- 172.21.15.161:0/2572100333 shutdown_connections 2024-03-28T23:33:16.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.044+0000 7f3422e23640 1 --2- 172.21.15.161:0/2572100333 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f340006b500 0x7f340006d9c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:16.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.044+0000 7f3422e23640 1 --2- 172.21.15.161:0/2572100333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f341c14c5e0 0x7f341c149da0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:16.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.044+0000 7f3422e23640 1 --2- 172.21.15.161:0/2572100333 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f341c071f00 0x7f341c149860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:16.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.044+0000 7f3422e23640 1 --2- 172.21.15.161:0/2572100333 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f340c004800 0x7f341c1dd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:16.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.044+0000 7f3422e23640 1 -- 172.21.15.161:0/2572100333 >> 172.21.15.161:0/2572100333 conn(0x7f341c0a83f0 msgr2=0x7f341c0aa430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:16.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.044+0000 7f3422e23640 1 -- 172.21.15.161:0/2572100333 shutdown_connections 2024-03-28T23:33:16.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.044+0000 7f3422e23640 1 -- 172.21.15.161:0/2572100333 wait complete. 2024-03-28T23:33:16.055 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.b mds_join_fs 2024-03-28T23:33:16.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.159+0000 7fbc60a0e640 1 Processor -- start 2024-03-28T23:33:16.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.159+0000 7fbc60a0e640 1 -- start start 2024-03-28T23:33:16.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.160+0000 7fbc60a0e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc5c156080 0x7fbc5c0c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:16.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.160+0000 7fbc60a0e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc5c072790 con 0x7fbc5c06ff90 2024-03-28T23:33:16.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.160+0000 7fbc60a0e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc5c0a7ea0 con 0x7fbc5c156460 2024-03-28T23:33:16.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.160+0000 7fbc60a0e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbc5c0c67b0 con 0x7fbc5c156080 2024-03-28T23:33:16.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.160+0000 7fbc59d9b640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fbc5c06ff90 0x7fbc5c156b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:39494/0 (socket says 172.21.15.161:39494) 2024-03-28T23:33:16.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.160+0000 7fbc59d9b640 1 -- 172.21.15.161:0/674227785 learned_addr learned my addr 172.21.15.161:0/674227785 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:16.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.161+0000 7fbc5959a640 1 -- 172.21.15.161:0/674227785 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4175865535 0 0) 0x7fbc5c072790 con 0x7fbc5c06ff90 2024-03-28T23:33:16.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.161+0000 7fbc5959a640 1 -- 172.21.15.161:0/674227785 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc44003520 con 0x7fbc5c06ff90 2024-03-28T23:33:16.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.161+0000 7fbc5a59c640 1 --2- 172.21.15.161:0/674227785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc5c156080 0x7fbc5c0c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:16.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.161+0000 7fbc5959a640 1 -- 172.21.15.161:0/674227785 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 463310222 0 0) 0x7fbc44003520 con 0x7fbc5c06ff90 2024-03-28T23:33:16.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.161+0000 7fbc5959a640 1 -- 172.21.15.161:0/674227785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc5c156080 msgr2=0x7fbc5c0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:16.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.161+0000 7fbc5959a640 1 --2- 172.21.15.161:0/674227785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc5c156080 0x7fbc5c0c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:16.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.161+0000 7fbc5959a640 1 -- 172.21.15.161:0/674227785 >> v1:172.21.15.170:6789/0 conn(0x7fbc5c156460 legacy=0x7fbc5c15dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:16.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.161+0000 7fbc5959a640 1 -- 172.21.15.161:0/674227785 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc5c0a8180 con 0x7fbc5c06ff90 2024-03-28T23:33:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.161+0000 7fbc5959a640 1 -- 172.21.15.161:0/674227785 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fbc50001670 con 0x7fbc5c06ff90 2024-03-28T23:33:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.161+0000 7fbc5959a640 1 -- 172.21.15.161:0/674227785 >> v1:172.21.15.161:6789/0 conn(0x7fbc5c06ff90 legacy=0x7fbc5c156b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.161+0000 7fbc5959a640 1 --2- 172.21.15.161:0/674227785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc44003ac0 0x7fbc44003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.161+0000 7fbc5959a640 1 --2- 172.21.15.161:0/674227785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc44004800 0x7fbc44004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.161+0000 7fbc5959a640 1 --2- 172.21.15.161:0/674227785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc5c156080 0x7fbc4400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.161+0000 7fbc5959a640 1 -- 172.21.15.161:0/674227785 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbc44005540 con 0x7fbc44003ac0 2024-03-28T23:33:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.161+0000 7fbc5959a640 1 -- 172.21.15.161:0/674227785 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4400bf00 con 0x7fbc44004800 2024-03-28T23:33:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.161+0000 7fbc5959a640 1 -- 172.21.15.161:0/674227785 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbc4400c080 con 0x7fbc5c156080 2024-03-28T23:33:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.161+0000 7fbc5a59c640 1 --2- 172.21.15.161:0/674227785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc44003ac0 0x7fbc44003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.161+0000 7fbc59d9b640 1 --2- 172.21.15.161:0/674227785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc44004800 0x7fbc44004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.161+0000 7fbc5ad9d640 1 --2- 172.21.15.161:0/674227785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc5c156080 0x7fbc4400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.161+0000 7fbc5a59c640 1 -- 172.21.15.161:0/674227785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc5c156080 msgr2=0x7fbc4400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.161+0000 7fbc5a59c640 1 --2- 172.21.15.161:0/674227785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc5c156080 0x7fbc4400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.161+0000 7fbc5a59c640 1 -- 172.21.15.161:0/674227785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc44004800 msgr2=0x7fbc44004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.161+0000 7fbc5a59c640 1 --2- 172.21.15.161:0/674227785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc44004800 0x7fbc44004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.161+0000 7fbc5a59c640 1 -- 172.21.15.161:0/674227785 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc44003520 con 0x7fbc44003ac0 2024-03-28T23:33:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.162+0000 7fbc5a59c640 1 --2- 172.21.15.161:0/674227785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc44003ac0 0x7fbc44003ee0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c009ac0 tx=0x7fbc44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68c8c86d889babe1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.162+0000 7fbc5959a640 1 -- 172.21.15.161:0/674227785 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c039070 con 0x7fbc44003ac0 2024-03-28T23:33:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.162+0000 7fbc5959a640 1 -- 172.21.15.161:0/674227785 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc4c02c6b0 con 0x7fbc44003ac0 2024-03-28T23:33:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.162+0000 7fbc5959a640 1 -- 172.21.15.161:0/674227785 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c009690 con 0x7fbc44003ac0 2024-03-28T23:33:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.163+0000 7fbc60a0e640 1 -- 172.21.15.161:0/674227785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc44003ac0 msgr2=0x7fbc44003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.163+0000 7fbc60a0e640 1 --2- 172.21.15.161:0/674227785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc44003ac0 0x7fbc44003ee0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c009ac0 tx=0x7fbc44005830 comp rx=0 tx=0).stop 2024-03-28T23:33:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.163+0000 7fbc5ad9d640 1 -- 172.21.15.161:0/674227785 reap_dead start 2024-03-28T23:33:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.163+0000 7fbc60a0e640 1 -- 172.21.15.161:0/674227785 shutdown_connections 2024-03-28T23:33:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.163+0000 7fbc60a0e640 1 -- 172.21.15.161:0/674227785 >> 172.21.15.161:0/674227785 conn(0x7fbc5c06eba0 msgr2=0x7fbc5c06b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.163+0000 7fbc60a0e640 1 -- 172.21.15.161:0/674227785 shutdown_connections 2024-03-28T23:33:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.163+0000 7fbc60a0e640 1 -- 172.21.15.161:0/674227785 wait complete. 2024-03-28T23:33:16.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.164+0000 7fbc60a0e640 1 Processor -- start 2024-03-28T23:33:16.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.164+0000 7fbc60a0e640 1 -- start start 2024-03-28T23:33:16.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.165+0000 7fbc60a0e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc44003ac0 0x7fbc5c1e8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:16.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.165+0000 7fbc60a0e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc5c0a8f80 0x7fbc5c0aa850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:16.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.165+0000 7fbc60a0e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc5c149210 0x7fbc5c0aad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.165+0000 7fbc60a0e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c070620 con 0x7fbc44003ac0 2024-03-28T23:33:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.165+0000 7fbc60a0e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c0704a0 con 0x7fbc5c149210 2024-03-28T23:33:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.165+0000 7fbc60a0e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbc5c0707a0 con 0x7fbc5c0a8f80 2024-03-28T23:33:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.165+0000 7fbc5a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc44003ac0 0x7fbc5c1e8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.165+0000 7fbc5a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc44003ac0 0x7fbc5c1e8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48922/0 (socket says 172.21.15.161:48922) 2024-03-28T23:33:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.165+0000 7fbc5a59c640 1 -- 172.21.15.161:0/2446615417 learned_addr learned my addr 172.21.15.161:0/2446615417 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.165+0000 7fbc59d9b640 1 --2- 172.21.15.161:0/2446615417 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc5c0a8f80 0x7fbc5c0aa850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.165+0000 7fbc5ad9d640 1 --2- 172.21.15.161:0/2446615417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc5c149210 0x7fbc5c0aad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.165+0000 7fbc5a59c640 1 -- 172.21.15.161:0/2446615417 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc5c0a8f80 msgr2=0x7fbc5c0aa850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.165+0000 7fbc5a59c640 1 --2- 172.21.15.161:0/2446615417 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc5c0a8f80 0x7fbc5c0aa850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.165+0000 7fbc5a59c640 1 -- 172.21.15.161:0/2446615417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc5c149210 msgr2=0x7fbc5c0aad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.165+0000 7fbc5a59c640 1 --2- 172.21.15.161:0/2446615417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc5c149210 0x7fbc5c0aad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.165+0000 7fbc5a59c640 1 -- 172.21.15.161:0/2446615417 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc5c1e9210 con 0x7fbc44003ac0 2024-03-28T23:33:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.165+0000 7fbc5a59c640 1 --2- 172.21.15.161:0/2446615417 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc44003ac0 0x7fbc5c1e8940 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c009be0 tx=0x7fbc4c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.165+0000 7fbc4affd640 1 -- 172.21.15.161:0/2446615417 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c042020 con 0x7fbc44003ac0 2024-03-28T23:33:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.165+0000 7fbc4affd640 1 -- 172.21.15.161:0/2446615417 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc4c03a040 con 0x7fbc44003ac0 2024-03-28T23:33:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.165+0000 7fbc60a0e640 1 -- 172.21.15.161:0/2446615417 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc5c0a8460 con 0x7fbc44003ac0 2024-03-28T23:33:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.165+0000 7fbc4affd640 1 -- 172.21.15.161:0/2446615417 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c039070 con 0x7fbc44003ac0 2024-03-28T23:33:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.166+0000 7fbc60a0e640 1 -- 172.21.15.161:0/2446615417 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc5c0adcf0 con 0x7fbc44003ac0 2024-03-28T23:33:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.166+0000 7fbc60a0e640 1 -- 172.21.15.161:0/2446615417 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc5c04f4b0 con 0x7fbc44003ac0 2024-03-28T23:33:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.167+0000 7fbc4affd640 1 -- 172.21.15.161:0/2446615417 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbc4c045020 con 0x7fbc44003ac0 2024-03-28T23:33:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.167+0000 7fbc4affd640 1 --2- 172.21.15.161:0/2446615417 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbc3c06b4d0 0x7fbc3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.167+0000 7fbc4affd640 1 -- 172.21.15.161:0/2446615417 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5593+0+0 (secure 0 0 0) 0x7fbc4c0ab4b0 con 0x7fbc44003ac0 2024-03-28T23:33:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.167+0000 7fbc59d9b640 1 --2- 172.21.15.161:0/2446615417 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbc3c06b4d0 0x7fbc3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.168+0000 7fbc59d9b640 1 --2- 172.21.15.161:0/2446615417 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbc3c06b4d0 0x7fbc3c06d990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c0aee10 tx=0x7fbc50009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:16.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.172+0000 7fbc4affd640 1 -- 172.21.15.161:0/2446615417 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbc4c04c070 con 0x7fbc44003ac0 2024-03-28T23:33:16.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.345+0000 7fbc60a0e640 1 -- 172.21.15.161:0/2446615417 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"} v 0) -- 0x7fbc5c06fd30 con 0x7fbc44003ac0 2024-03-28T23:33:16.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.350+0000 7fbc4affd640 1 -- 172.21.15.161:0/2446615417 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"}]=0 v30) ==== 96+0+0 (secure 0 0 0) 0x7fbc4c07a430 con 0x7fbc44003ac0 2024-03-28T23:33:16.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.352+0000 7fbc60a0e640 1 -- 172.21.15.161:0/2446615417 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbc3c06b4d0 msgr2=0x7fbc3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:16.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.352+0000 7fbc60a0e640 1 --2- 172.21.15.161:0/2446615417 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbc3c06b4d0 0x7fbc3c06d990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c0aee10 tx=0x7fbc50009040 comp rx=0 tx=0).stop 2024-03-28T23:33:16.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.352+0000 7fbc60a0e640 1 -- 172.21.15.161:0/2446615417 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc44003ac0 msgr2=0x7fbc5c1e8940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:16.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.352+0000 7fbc60a0e640 1 --2- 172.21.15.161:0/2446615417 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc44003ac0 0x7fbc5c1e8940 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c009be0 tx=0x7fbc4c004730 comp rx=0 tx=0).stop 2024-03-28T23:33:16.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.352+0000 7fbc60a0e640 1 -- 172.21.15.161:0/2446615417 shutdown_connections 2024-03-28T23:33:16.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.352+0000 7fbc60a0e640 1 --2- 172.21.15.161:0/2446615417 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbc3c06b4d0 0x7fbc3c06d990 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:16.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.352+0000 7fbc60a0e640 1 --2- 172.21.15.161:0/2446615417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc5c149210 0x7fbc5c0aad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:16.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.352+0000 7fbc60a0e640 1 --2- 172.21.15.161:0/2446615417 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc5c0a8f80 0x7fbc5c0aa850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:16.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.352+0000 7fbc60a0e640 1 --2- 172.21.15.161:0/2446615417 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc44003ac0 0x7fbc5c1e8940 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:16.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.352+0000 7fbc60a0e640 1 -- 172.21.15.161:0/2446615417 >> 172.21.15.161:0/2446615417 conn(0x7fbc5c06eba0 msgr2=0x7fbc5c156a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:16.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.352+0000 7fbc60a0e640 1 -- 172.21.15.161:0/2446615417 shutdown_connections 2024-03-28T23:33:16.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.352+0000 7fbc60a0e640 1 -- 172.21.15.161:0/2446615417 wait complete. 2024-03-28T23:33:16.363 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused' 2024-03-28T23:33:16.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.464+0000 7f95f25e2640 1 Processor -- start 2024-03-28T23:33:16.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.464+0000 7f95f25e2640 1 -- start start 2024-03-28T23:33:16.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.465+0000 7f95f25e2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f95ec150600 0x7f95ec0a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:16.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.465+0000 7f95f25e2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95ec0aacb0 0x7f95ec153980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:16.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.465+0000 7f95f25e2640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95ec1567c0 con 0x7f95ec0a8fc0 2024-03-28T23:33:16.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.465+0000 7f95f25e2640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f95ec156570 con 0x7f95ec150600 2024-03-28T23:33:16.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.465+0000 7f95f25e2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f95ec156aa0 con 0x7f95ec0aacb0 2024-03-28T23:33:16.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.465+0000 7f95ebfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f95ec150600 0x7f95ec0a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:16.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.465+0000 7f95ebfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f95ec150600 0x7f95ec0a9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48930/0 (socket says 172.21.15.161:48930) 2024-03-28T23:33:16.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.465+0000 7f95ebfff640 1 -- 172.21.15.161:0/2699927812 learned_addr learned my addr 172.21.15.161:0/2699927812 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:16.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.465+0000 7f95ebfff640 1 -- 172.21.15.161:0/2699927812 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95ec0aacb0 msgr2=0x7f95ec153980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:16.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.465+0000 7f95ebfff640 1 --2- 172.21.15.161:0/2699927812 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95ec0aacb0 0x7f95ec153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:16.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.465+0000 7f95ebfff640 1 -- 172.21.15.161:0/2699927812 >> v1:172.21.15.170:6789/0 conn(0x7f95ec0a8fc0 legacy=0x7f95ec153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:16.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.465+0000 7f95ebfff640 1 -- 172.21.15.161:0/2699927812 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95ec156d80 con 0x7f95ec150600 2024-03-28T23:33:16.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.466+0000 7f95ebfff640 1 --2- 172.21.15.161:0/2699927812 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95ec150600 0x7f95ec0a9f40 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f95dc009870 tx=0x7f95dc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34fbe24ab59c8454 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:16.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.466+0000 7f95eaffd640 1 -- 172.21.15.161:0/2699927812 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95dc0317c0 con 0x7f95ec150600 2024-03-28T23:33:16.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.466+0000 7f95eaffd640 1 -- 172.21.15.161:0/2699927812 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95dc037070 con 0x7f95ec150600 2024-03-28T23:33:16.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.466+0000 7f95eaffd640 1 -- 172.21.15.161:0/2699927812 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95dc031b80 con 0x7f95ec150600 2024-03-28T23:33:16.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.467+0000 7f95f25e2640 1 -- 172.21.15.161:0/2699927812 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95ec150600 msgr2=0x7f95ec0a9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:16.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.467+0000 7f95f25e2640 1 --2- 172.21.15.161:0/2699927812 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95ec150600 0x7f95ec0a9f40 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f95dc009870 tx=0x7f95dc02b860 comp rx=0 tx=0).stop 2024-03-28T23:33:16.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.467+0000 7f95f25e2640 1 -- 172.21.15.161:0/2699927812 shutdown_connections 2024-03-28T23:33:16.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.467+0000 7f95f25e2640 1 --2- 172.21.15.161:0/2699927812 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95ec0aacb0 0x7f95ec153980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:16.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.467+0000 7f95f25e2640 1 --2- 172.21.15.161:0/2699927812 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95ec150600 0x7f95ec0a9f40 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:16.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.467+0000 7f95f25e2640 1 -- 172.21.15.161:0/2699927812 >> 172.21.15.161:0/2699927812 conn(0x7f95ec0c6670 msgr2=0x7f95ec0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:16.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.467+0000 7f95f25e2640 1 -- 172.21.15.161:0/2699927812 shutdown_connections 2024-03-28T23:33:16.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.467+0000 7f95f25e2640 1 -- 172.21.15.161:0/2699927812 wait complete. 2024-03-28T23:33:16.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.468+0000 7f95f25e2640 1 Processor -- start 2024-03-28T23:33:16.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.468+0000 7f95f25e2640 1 -- start start 2024-03-28T23:33:16.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.469+0000 7f95f25e2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95ec0a8fc0 0x7f95ec147f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:16.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.469+0000 7f95f25e2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95ec0aacb0 0x7f95ec1e8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:16.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.469+0000 7f95f25e2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95ec150600 0x7f95ec1e9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.469+0000 7f95f25e2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f95ec153ec0 con 0x7f95ec0aacb0 2024-03-28T23:33:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.469+0000 7f95f25e2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f95ec157340 con 0x7f95ec150600 2024-03-28T23:33:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.469+0000 7f95f25e2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f95ec1509e0 con 0x7f95ec0a8fc0 2024-03-28T23:33:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.469+0000 7f95eb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95ec0aacb0 0x7f95ec1e8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.469+0000 7f95eb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95ec0aacb0 0x7f95ec1e8ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48942/0 (socket says 172.21.15.161:48942) 2024-03-28T23:33:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.469+0000 7f95eb7fe640 1 -- 172.21.15.161:0/187361062 learned_addr learned my addr 172.21.15.161:0/187361062 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.469+0000 7f95ebfff640 1 --2- 172.21.15.161:0/187361062 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95ec0a8fc0 0x7f95ec147f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.469+0000 7f95f0b7f640 1 --2- 172.21.15.161:0/187361062 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95ec150600 0x7f95ec1e9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.469+0000 7f95eb7fe640 1 -- 172.21.15.161:0/187361062 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95ec0a8fc0 msgr2=0x7f95ec147f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.469+0000 7f95eb7fe640 1 --2- 172.21.15.161:0/187361062 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95ec0a8fc0 0x7f95ec147f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.469+0000 7f95eb7fe640 1 -- 172.21.15.161:0/187361062 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95ec150600 msgr2=0x7f95ec1e9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.469+0000 7f95eb7fe640 1 --2- 172.21.15.161:0/187361062 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95ec150600 0x7f95ec1e9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.469+0000 7f95eb7fe640 1 -- 172.21.15.161:0/187361062 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95ec1ebff0 con 0x7f95ec0aacb0 2024-03-28T23:33:16.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.469+0000 7f95f0b7f640 1 --2- 172.21.15.161:0/187361062 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95ec150600 0x7f95ec1e9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:16.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.469+0000 7f95eb7fe640 1 --2- 172.21.15.161:0/187361062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95ec0aacb0 0x7f95ec1e8ac0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f95e000b3e0 tx=0x7f95e000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:16.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.469+0000 7f95e8ff9640 1 -- 172.21.15.161:0/187361062 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e0015020 con 0x7f95ec0aacb0 2024-03-28T23:33:16.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.469+0000 7f95e8ff9640 1 -- 172.21.15.161:0/187361062 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95e0011850 con 0x7f95ec0aacb0 2024-03-28T23:33:16.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.469+0000 7f95f25e2640 1 -- 172.21.15.161:0/187361062 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95ec1ec2d0 con 0x7f95ec0aacb0 2024-03-28T23:33:16.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.469+0000 7f95e8ff9640 1 -- 172.21.15.161:0/187361062 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e0011c40 con 0x7f95ec0aacb0 2024-03-28T23:33:16.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.469+0000 7f95f25e2640 1 -- 172.21.15.161:0/187361062 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95ec1ec890 con 0x7f95ec0aacb0 2024-03-28T23:33:16.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.470+0000 7f95f25e2640 1 -- 172.21.15.161:0/187361062 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95ec04f4b0 con 0x7f95ec0aacb0 2024-03-28T23:33:16.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.471+0000 7f95e8ff9640 1 -- 172.21.15.161:0/187361062 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f95e0018020 con 0x7f95ec0aacb0 2024-03-28T23:33:16.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.471+0000 7f95e8ff9640 1 --2- 172.21.15.161:0/187361062 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f95d806b500 0x7f95d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:16.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.471+0000 7f95e8ff9640 1 -- 172.21.15.161:0/187361062 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5593+0+0 (secure 0 0 0) 0x7f95e008a000 con 0x7f95ec0aacb0 2024-03-28T23:33:16.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.471+0000 7f95ebfff640 1 --2- 172.21.15.161:0/187361062 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f95d806b500 0x7f95d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:16.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.472+0000 7f95ebfff640 1 --2- 172.21.15.161:0/187361062 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f95d806b500 0x7f95d806d9c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f95dc036000 tx=0x7f95dc03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:16.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.476+0000 7f95e8ff9640 1 -- 172.21.15.161:0/187361062 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f95e001f020 con 0x7f95ec0aacb0 2024-03-28T23:33:16.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:16.714+0000 7f95f25e2640 1 -- 172.21.15.161:0/187361062 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused"]} v 0) -- 0x7f95ec0aa4e0 con 0x7f95ec0aacb0 2024-03-28T23:33:17.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.123+0000 7f95e8ff9640 1 -- 172.21.15.161:0/187361062 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused"]}]=0 v211) ==== 158+0+0 (secure 0 0 0) 0x7f95e0058ef0 con 0x7f95ec0aacb0 2024-03-28T23:33:17.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.126+0000 7f95f25e2640 1 -- 172.21.15.161:0/187361062 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f95d806b500 msgr2=0x7f95d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:17.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.126+0000 7f95f25e2640 1 --2- 172.21.15.161:0/187361062 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f95d806b500 0x7f95d806d9c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f95dc036000 tx=0x7f95dc03d040 comp rx=0 tx=0).stop 2024-03-28T23:33:17.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.126+0000 7f95f25e2640 1 -- 172.21.15.161:0/187361062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95ec0aacb0 msgr2=0x7f95ec1e8ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:17.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.126+0000 7f95f25e2640 1 --2- 172.21.15.161:0/187361062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95ec0aacb0 0x7f95ec1e8ac0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f95e000b3e0 tx=0x7f95e000b8a0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.126+0000 7f95f25e2640 1 -- 172.21.15.161:0/187361062 shutdown_connections 2024-03-28T23:33:17.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.126+0000 7f95f25e2640 1 --2- 172.21.15.161:0/187361062 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f95d806b500 0x7f95d806d9c0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.126+0000 7f95f25e2640 1 --2- 172.21.15.161:0/187361062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95ec0aacb0 0x7f95ec1e8ac0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.126+0000 7f95f25e2640 1 --2- 172.21.15.161:0/187361062 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95ec150600 0x7f95ec1e9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.126+0000 7f95f25e2640 1 --2- 172.21.15.161:0/187361062 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95ec0a8fc0 0x7f95ec147f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.126+0000 7f95f25e2640 1 -- 172.21.15.161:0/187361062 >> 172.21.15.161:0/187361062 conn(0x7f95ec0c6670 msgr2=0x7f95ec14cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:17.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.126+0000 7f95f25e2640 1 -- 172.21.15.161:0/187361062 shutdown_connections 2024-03-28T23:33:17.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.126+0000 7f95f25e2640 1 -- 172.21.15.161:0/187361062 wait complete. 2024-03-28T23:33:17.139 INFO:tasks.cephfs_test_runner:That a standby with mds_join_fs set to another fs is not used if refuse_standby_for_another_fs is set. ... ok 2024-03-28T23:33:17.139 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_runtime (tasks.cephfs.test_failover.TestClusterAffinity) That setting mds_join_fs at runtime affects the cluster layout. 2024-03-28T23:33:17.139 INFO:tasks.cephfs_test_runner:test_join_fs_runtime (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-28T23:33:17.140 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:33:17.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.241+0000 7f0cd08df640 1 Processor -- start 2024-03-28T23:33:17.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.241+0000 7f0cd08df640 1 -- start start 2024-03-28T23:33:17.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.242+0000 7f0cd08df640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ccc150600 0x7f0ccc0a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:17.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.242+0000 7f0cd08df640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0ccc0a8fc0 0x7f0ccc153270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:17.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.242+0000 7f0cd08df640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ccc1567c0 con 0x7f0ccc0aacb0 2024-03-28T23:33:17.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.242+0000 7f0cd08df640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f0ccc156570 con 0x7f0ccc0a8fc0 2024-03-28T23:33:17.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.242+0000 7f0cd08df640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0ccc156aa0 con 0x7f0ccc150600 2024-03-28T23:33:17.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.242+0000 7f0cc9d9b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0ccc0a8fc0 0x7f0ccc153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:17.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.242+0000 7f0cca59c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ccc150600 0x7f0ccc0a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:17.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.242+0000 7f0cc9d9b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0ccc0a8fc0 0x7f0ccc153270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:35624/0 (socket says 172.21.15.161:35624) 2024-03-28T23:33:17.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.242+0000 7f0cca59c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ccc150600 0x7f0ccc0a9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:45766/0 (socket says 172.21.15.161:45766) 2024-03-28T23:33:17.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.242+0000 7f0ccad9d640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f0ccc0aacb0 0x7f0ccc1537b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:46880/0 (socket says 172.21.15.161:46880) 2024-03-28T23:33:17.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.242+0000 7f0cc9d9b640 1 -- 172.21.15.161:0/2430004212 learned_addr learned my addr 172.21.15.161:0/2430004212 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:17.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.243+0000 7f0cca59c640 1 -- 172.21.15.161:0/2430004212 >> v2:172.21.15.170:3300/0 conn(0x7f0ccc0a8fc0 msgr2=0x7f0ccc153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:17.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.243+0000 7f0cca59c640 1 --2- 172.21.15.161:0/2430004212 >> v2:172.21.15.170:3300/0 conn(0x7f0ccc0a8fc0 0x7f0ccc153270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.243+0000 7f0cca59c640 1 -- 172.21.15.161:0/2430004212 >> v1:172.21.15.170:6789/0 conn(0x7f0ccc0aacb0 legacy=0x7f0ccc1537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:17.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.243+0000 7f0cca59c640 1 -- 172.21.15.161:0/2430004212 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ccc156d80 con 0x7f0ccc150600 2024-03-28T23:33:17.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.243+0000 7f0cc9d9b640 1 --2- 172.21.15.161:0/2430004212 >> v2:172.21.15.170:3300/0 conn(0x7f0ccc0a8fc0 0x7f0ccc153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:17.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.244+0000 7f0cca59c640 1 --2- 172.21.15.161:0/2430004212 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ccc150600 0x7f0ccc0a9f40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc0097a0 tx=0x7f0cbc02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=431774dce595e8a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:17.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.244+0000 7f0cc959a640 1 -- 172.21.15.161:0/2430004212 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc0315a0 con 0x7f0ccc150600 2024-03-28T23:33:17.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.244+0000 7f0cc959a640 1 -- 172.21.15.161:0/2430004212 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cbc037070 con 0x7f0ccc150600 2024-03-28T23:33:17.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.244+0000 7f0cc959a640 1 -- 172.21.15.161:0/2430004212 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc031960 con 0x7f0ccc150600 2024-03-28T23:33:17.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.244+0000 7f0cd08df640 1 -- 172.21.15.161:0/2430004212 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ccc150600 msgr2=0x7f0ccc0a9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:17.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.244+0000 7f0cd08df640 1 --2- 172.21.15.161:0/2430004212 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ccc150600 0x7f0ccc0a9f40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc0097a0 tx=0x7f0cbc02b840 comp rx=0 tx=0).stop 2024-03-28T23:33:17.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.244+0000 7f0cd08df640 1 -- 172.21.15.161:0/2430004212 shutdown_connections 2024-03-28T23:33:17.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.244+0000 7f0cd08df640 1 --2- 172.21.15.161:0/2430004212 >> v2:172.21.15.170:3300/0 conn(0x7f0ccc0a8fc0 0x7f0ccc153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.244+0000 7f0cd08df640 1 --2- 172.21.15.161:0/2430004212 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ccc150600 0x7f0ccc0a9f40 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.244+0000 7f0cd08df640 1 -- 172.21.15.161:0/2430004212 >> 172.21.15.161:0/2430004212 conn(0x7f0ccc0c6670 msgr2=0x7f0ccc0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:17.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.244+0000 7f0cd08df640 1 -- 172.21.15.161:0/2430004212 shutdown_connections 2024-03-28T23:33:17.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.244+0000 7f0cd08df640 1 -- 172.21.15.161:0/2430004212 wait complete. 2024-03-28T23:33:17.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.246+0000 7f0cd08df640 1 Processor -- start 2024-03-28T23:33:17.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.246+0000 7f0cd08df640 1 -- start start 2024-03-28T23:33:17.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.246+0000 7f0cd08df640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ccc0a8fc0 0x7f0ccc0c3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:17.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.246+0000 7f0cd08df640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ccc0aacb0 0x7f0ccc149460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:17.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.246+0000 7f0cd08df640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ccc150600 0x7f0ccc14bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:17.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.246+0000 7f0cd08df640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc153ec0 con 0x7f0ccc150600 2024-03-28T23:33:17.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.246+0000 7f0cd08df640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc157340 con 0x7f0ccc0aacb0 2024-03-28T23:33:17.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.246+0000 7f0cd08df640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0ccc1509e0 con 0x7f0ccc0a8fc0 2024-03-28T23:33:17.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.246+0000 7f0ccad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ccc150600 0x7f0ccc14bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:17.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.246+0000 7f0ccad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ccc150600 0x7f0ccc14bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48952/0 (socket says 172.21.15.161:48952) 2024-03-28T23:33:17.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.246+0000 7f0ccad9d640 1 -- 172.21.15.161:0/4268112176 learned_addr learned my addr 172.21.15.161:0/4268112176 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:17.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.247+0000 7f0cca59c640 1 --2- 172.21.15.161:0/4268112176 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ccc0a8fc0 0x7f0ccc0c3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:17.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.247+0000 7f0cc9d9b640 1 --2- 172.21.15.161:0/4268112176 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ccc0aacb0 0x7f0ccc149460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:17.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.247+0000 7f0ccad9d640 1 -- 172.21.15.161:0/4268112176 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ccc0a8fc0 msgr2=0x7f0ccc0c3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:17.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.247+0000 7f0ccad9d640 1 --2- 172.21.15.161:0/4268112176 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ccc0a8fc0 0x7f0ccc0c3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.247+0000 7f0ccad9d640 1 -- 172.21.15.161:0/4268112176 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ccc0aacb0 msgr2=0x7f0ccc149460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:17.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.247+0000 7f0ccad9d640 1 --2- 172.21.15.161:0/4268112176 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ccc0aacb0 0x7f0ccc149460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.247+0000 7f0ccad9d640 1 -- 172.21.15.161:0/4268112176 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ccc1e8d90 con 0x7f0ccc150600 2024-03-28T23:33:17.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.247+0000 7f0cca59c640 1 --2- 172.21.15.161:0/4268112176 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ccc0a8fc0 0x7f0ccc0c3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:17.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.247+0000 7f0ccad9d640 1 --2- 172.21.15.161:0/4268112176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ccc150600 0x7f0ccc14bf90 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f0cc400c710 tx=0x7f0cc400cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:17.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.247+0000 7f0cbaffd640 1 -- 172.21.15.161:0/4268112176 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc4017070 con 0x7f0ccc150600 2024-03-28T23:33:17.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.247+0000 7f0cbaffd640 1 -- 172.21.15.161:0/4268112176 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cc400ce20 con 0x7f0ccc150600 2024-03-28T23:33:17.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.247+0000 7f0cd08df640 1 -- 172.21.15.161:0/4268112176 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ccc0c3ce0 con 0x7f0ccc150600 2024-03-28T23:33:17.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.247+0000 7f0cbaffd640 1 -- 172.21.15.161:0/4268112176 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc400a860 con 0x7f0ccc150600 2024-03-28T23:33:17.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.247+0000 7f0cd08df640 1 -- 172.21.15.161:0/4268112176 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ccc1e9330 con 0x7f0ccc150600 2024-03-28T23:33:17.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.248+0000 7f0cd08df640 1 -- 172.21.15.161:0/4268112176 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ccc04f4b0 con 0x7f0ccc150600 2024-03-28T23:33:17.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.249+0000 7f0cbaffd640 1 -- 172.21.15.161:0/4268112176 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0cc401a070 con 0x7f0ccc150600 2024-03-28T23:33:17.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.249+0000 7f0cbaffd640 1 --2- 172.21.15.161:0/4268112176 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0cac06b4d0 0x7f0cac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:17.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.249+0000 7f0cbaffd640 1 -- 172.21.15.161:0/4268112176 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5593+0+0 (secure 0 0 0) 0x7f0cc40896d0 con 0x7f0ccc150600 2024-03-28T23:33:17.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.249+0000 7f0cca59c640 1 --2- 172.21.15.161:0/4268112176 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0cac06b4d0 0x7f0cac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:17.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.249+0000 7f0cca59c640 1 --2- 172.21.15.161:0/4268112176 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0cac06b4d0 0x7f0cac06d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc036000 tx=0x7f0cbc03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:17.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.253+0000 7f0cbaffd640 1 -- 172.21.15.161:0/4268112176 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0cc4057f80 con 0x7f0ccc150600 2024-03-28T23:33:17.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.428+0000 7f0cd08df640 1 -- 172.21.15.161:0/4268112176 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0ccc150fe0 con 0x7f0ccc150600 2024-03-28T23:33:17.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.428+0000 7f0cbaffd640 1 -- 172.21.15.161:0/4268112176 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+13726 (secure 0 0 0) 0x7f0cc405c020 con 0x7f0ccc150600 2024-03-28T23:33:17.429 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:17.429 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":85,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:33:15.109333+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6835/737681942":"2024-03-29T23:33:12.099877+0000","172.21.15.170:6835/2691084827":"2024-03-29T23:33:06.309901+0000","172.21.15.161:6836/2378263687":"2024-03-29T23:33:05.302530+0000","172.21.15.170:6833/2691084827":"2024-03-29T23:33:06.309901+0000","172.21.15.161:6834/2378263687":"2024-03-29T23:33:05.302530+0000","172.21.15.161:6836/3850309502":"2024-03-29T23:32:54.518520+0000","172.21.15.161:6837/737681942":"2024-03-29T23:33:12.099877+0000","172.21.15.161:6834/3850309502":"2024-03-29T23:32:54.518520+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:33:17.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.430+0000 7f0cd08df640 1 -- 172.21.15.161:0/4268112176 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0cac06b4d0 msgr2=0x7f0cac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:17.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.430+0000 7f0cd08df640 1 --2- 172.21.15.161:0/4268112176 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0cac06b4d0 0x7f0cac06d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc036000 tx=0x7f0cbc03d040 comp rx=0 tx=0).stop 2024-03-28T23:33:17.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.430+0000 7f0cd08df640 1 -- 172.21.15.161:0/4268112176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ccc150600 msgr2=0x7f0ccc14bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:17.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.430+0000 7f0cd08df640 1 --2- 172.21.15.161:0/4268112176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ccc150600 0x7f0ccc14bf90 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f0cc400c710 tx=0x7f0cc400cbd0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.430+0000 7f0cd08df640 1 -- 172.21.15.161:0/4268112176 shutdown_connections 2024-03-28T23:33:17.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.430+0000 7f0cd08df640 1 --2- 172.21.15.161:0/4268112176 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0cac06b4d0 0x7f0cac06d990 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.430+0000 7f0cd08df640 1 --2- 172.21.15.161:0/4268112176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ccc150600 0x7f0ccc14bf90 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.430+0000 7f0cd08df640 1 --2- 172.21.15.161:0/4268112176 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ccc0aacb0 0x7f0ccc149460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.430+0000 7f0cd08df640 1 --2- 172.21.15.161:0/4268112176 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ccc0a8fc0 0x7f0ccc0c3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.430+0000 7f0cd08df640 1 -- 172.21.15.161:0/4268112176 >> 172.21.15.161:0/4268112176 conn(0x7f0ccc0c6670 msgr2=0x7f0ccc14eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:17.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.430+0000 7f0cd08df640 1 -- 172.21.15.161:0/4268112176 shutdown_connections 2024-03-28T23:33:17.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.431+0000 7f0cd08df640 1 -- 172.21.15.161:0/4268112176 wait complete. 2024-03-28T23:33:17.442 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:33:17.442 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:33:17.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.545+0000 7f477cf0d640 1 Processor -- start 2024-03-28T23:33:17.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.545+0000 7f477cf0d640 1 -- start start 2024-03-28T23:33:17.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.546+0000 7f477cf0d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f47780c6750 0x7f477806bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:17.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.546+0000 7f477cf0d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4778152db0 con 0x7f47780c88b0 2024-03-28T23:33:17.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.546+0000 7f477cf0d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47780c8d40 con 0x7f47781529d0 2024-03-28T23:33:17.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.546+0000 7f477cf0d640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f47780c7210 con 0x7f47780c6750 2024-03-28T23:33:17.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.546+0000 7f4776d9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f47780c88b0 0x7f47780c54f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:39498/0 (socket says 172.21.15.161:39498) 2024-03-28T23:33:17.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.546+0000 7f4776d9d640 1 -- 172.21.15.161:0/883212368 learned_addr learned my addr 172.21.15.161:0/883212368 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:17.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f4775d9b640 1 --2- 172.21.15.161:0/883212368 >> v2:172.21.15.170:3300/0 conn(0x7f47780c6750 0x7f477806bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:17.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f477559a640 1 -- 172.21.15.161:0/883212368 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3721100230 0 0) 0x7f4778152db0 con 0x7f47780c88b0 2024-03-28T23:33:17.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f477559a640 1 -- 172.21.15.161:0/883212368 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4760003520 con 0x7f47780c88b0 2024-03-28T23:33:17.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f477559a640 1 -- 172.21.15.161:0/883212368 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1749842183 0 0) 0x7f4760003520 con 0x7f47780c88b0 2024-03-28T23:33:17.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f477559a640 1 -- 172.21.15.161:0/883212368 >> v2:172.21.15.170:3300/0 conn(0x7f47780c6750 msgr2=0x7f477806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:17.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f477559a640 1 --2- 172.21.15.161:0/883212368 >> v2:172.21.15.170:3300/0 conn(0x7f47780c6750 0x7f477806bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f477559a640 1 -- 172.21.15.161:0/883212368 >> v1:172.21.15.170:6789/0 conn(0x7f47781529d0 legacy=0x7f4778155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:17.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f477559a640 1 -- 172.21.15.161:0/883212368 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47780c6c50 con 0x7f47780c88b0 2024-03-28T23:33:17.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f477559a640 1 -- 172.21.15.161:0/883212368 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f4770001670 con 0x7f47780c88b0 2024-03-28T23:33:17.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f477559a640 1 -- 172.21.15.161:0/883212368 >> v1:172.21.15.161:6789/0 conn(0x7f47780c88b0 legacy=0x7f47780c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:17.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f477559a640 1 --2- 172.21.15.161:0/883212368 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47600039b0 0x7f4760003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:17.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f477559a640 1 --2- 172.21.15.161:0/883212368 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4760004800 0x7f4760004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:17.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f477559a640 1 --2- 172.21.15.161:0/883212368 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47600051c0 0x7f476000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:17.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f477559a640 1 -- 172.21.15.161:0/883212368 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f47780c7210 con 0x7f4760004800 2024-03-28T23:33:17.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f477559a640 1 -- 172.21.15.161:0/883212368 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f476000c060 con 0x7f47600051c0 2024-03-28T23:33:17.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f477559a640 1 -- 172.21.15.161:0/883212368 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f476000c260 con 0x7f47600039b0 2024-03-28T23:33:17.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f4775d9b640 1 --2- 172.21.15.161:0/883212368 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4760004800 0x7f4760004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:17.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f4776d9d640 1 --2- 172.21.15.161:0/883212368 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47600051c0 0x7f476000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:17.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f477659c640 1 --2- 172.21.15.161:0/883212368 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47600039b0 0x7f4760003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:17.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f4775d9b640 1 -- 172.21.15.161:0/883212368 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47600039b0 msgr2=0x7f4760003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:17.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f4775d9b640 1 --2- 172.21.15.161:0/883212368 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47600039b0 0x7f4760003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f4775d9b640 1 -- 172.21.15.161:0/883212368 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47600051c0 msgr2=0x7f476000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:17.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f4775d9b640 1 --2- 172.21.15.161:0/883212368 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47600051c0 0x7f476000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f4776d9d640 1 -- 172.21.15.161:0/883212368 reap_dead start 2024-03-28T23:33:17.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.547+0000 7f4775d9b640 1 -- 172.21.15.161:0/883212368 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4760003520 con 0x7f4760004800 2024-03-28T23:33:17.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.548+0000 7f4775d9b640 1 --2- 172.21.15.161:0/883212368 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4760004800 0x7f4760004c80 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f476c009ac0 tx=0x7f4760005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88b5e8cd9ab8cbad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:17.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.548+0000 7f477559a640 1 -- 172.21.15.161:0/883212368 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c039070 con 0x7f4760004800 2024-03-28T23:33:17.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.548+0000 7f477559a640 1 -- 172.21.15.161:0/883212368 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f476c02c6b0 con 0x7f4760004800 2024-03-28T23:33:17.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.548+0000 7f477559a640 1 -- 172.21.15.161:0/883212368 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c009690 con 0x7f4760004800 2024-03-28T23:33:17.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.549+0000 7f477cf0d640 1 -- 172.21.15.161:0/883212368 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4760004800 msgr2=0x7f4760004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:17.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.549+0000 7f477cf0d640 1 --2- 172.21.15.161:0/883212368 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4760004800 0x7f4760004c80 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f476c009ac0 tx=0x7f4760005830 comp rx=0 tx=0).stop 2024-03-28T23:33:17.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.549+0000 7f477cf0d640 1 -- 172.21.15.161:0/883212368 shutdown_connections 2024-03-28T23:33:17.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.549+0000 7f477cf0d640 1 --2- 172.21.15.161:0/883212368 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4760004800 0x7f4760004c80 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.549+0000 7f477cf0d640 1 -- 172.21.15.161:0/883212368 >> 172.21.15.161:0/883212368 conn(0x7f47780c7430 msgr2=0x7f47780c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:17.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.549+0000 7f477cf0d640 1 -- 172.21.15.161:0/883212368 shutdown_connections 2024-03-28T23:33:17.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.549+0000 7f477cf0d640 1 -- 172.21.15.161:0/883212368 wait complete. 2024-03-28T23:33:17.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.550+0000 7f477cf0d640 1 Processor -- start 2024-03-28T23:33:17.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.550+0000 7f477cf0d640 1 -- start start 2024-03-28T23:33:17.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.551+0000 7f477cf0d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4760004800 0x7f477814cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:17.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.551+0000 7f477cf0d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47780c4920 0x7f47781eb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:17.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.551+0000 7f477cf0d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4778154450 0x7f47781571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:17.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.551+0000 7f477cf0d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4778155710 con 0x7f47780c4920 2024-03-28T23:33:17.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.551+0000 7f477cf0d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4778155430 con 0x7f4778154450 2024-03-28T23:33:17.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.551+0000 7f477cf0d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f47780aa160 con 0x7f4760004800 2024-03-28T23:33:17.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.551+0000 7f4775d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47780c4920 0x7f47781eb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:17.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.551+0000 7f4775d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47780c4920 0x7f47781eb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48974/0 (socket says 172.21.15.161:48974) 2024-03-28T23:33:17.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.551+0000 7f4775d9b640 1 -- 172.21.15.161:0/811894024 learned_addr learned my addr 172.21.15.161:0/811894024 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:17.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.551+0000 7f477659c640 1 --2- 172.21.15.161:0/811894024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4760004800 0x7f477814cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:17.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.551+0000 7f4776d9d640 1 --2- 172.21.15.161:0/811894024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4778154450 0x7f47781571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:17.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.551+0000 7f4775d9b640 1 -- 172.21.15.161:0/811894024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4760004800 msgr2=0x7f477814cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:17.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.551+0000 7f4775d9b640 1 --2- 172.21.15.161:0/811894024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4760004800 0x7f477814cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.551+0000 7f4775d9b640 1 -- 172.21.15.161:0/811894024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4778154450 msgr2=0x7f47781571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:17.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.551+0000 7f4775d9b640 1 --2- 172.21.15.161:0/811894024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4778154450 0x7f47781571d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.551+0000 7f4775d9b640 1 -- 172.21.15.161:0/811894024 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4778158df0 con 0x7f47780c4920 2024-03-28T23:33:17.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.551+0000 7f4775d9b640 1 --2- 172.21.15.161:0/811894024 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47780c4920 0x7f47781eb9e0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f476c037040 tx=0x7f476c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:17.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.552+0000 7f4766ffd640 1 -- 172.21.15.161:0/811894024 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c039050 con 0x7f47780c4920 2024-03-28T23:33:17.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.552+0000 7f4766ffd640 1 -- 172.21.15.161:0/811894024 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f476c045070 con 0x7f47780c4920 2024-03-28T23:33:17.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.552+0000 7f477cf0d640 1 -- 172.21.15.161:0/811894024 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47781590d0 con 0x7f47780c4920 2024-03-28T23:33:17.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.552+0000 7f4766ffd640 1 -- 172.21.15.161:0/811894024 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c0044e0 con 0x7f47780c4920 2024-03-28T23:33:17.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.552+0000 7f477cf0d640 1 -- 172.21.15.161:0/811894024 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47781569d0 con 0x7f47780c4920 2024-03-28T23:33:17.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.553+0000 7f477cf0d640 1 -- 172.21.15.161:0/811894024 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f477804f4b0 con 0x7f47780c4920 2024-03-28T23:33:17.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.553+0000 7f4766ffd640 1 -- 172.21.15.161:0/811894024 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f476c003a50 con 0x7f47780c4920 2024-03-28T23:33:17.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.553+0000 7f4766ffd640 1 --2- 172.21.15.161:0/811894024 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f475806b500 0x7f475806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:17.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.553+0000 7f4766ffd640 1 -- 172.21.15.161:0/811894024 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5593+0+0 (secure 0 0 0) 0x7f476c0abc30 con 0x7f47780c4920 2024-03-28T23:33:17.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.554+0000 7f477659c640 1 --2- 172.21.15.161:0/811894024 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f475806b500 0x7f475806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:17.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.554+0000 7f477659c640 1 --2- 172.21.15.161:0/811894024 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f475806b500 0x7f475806d9c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f477814f680 tx=0x7f4768009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:17.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.558+0000 7f4766ffd640 1 -- 172.21.15.161:0/811894024 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f476c04e020 con 0x7f47780c4920 2024-03-28T23:33:17.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.728+0000 7f477cf0d640 1 -- 172.21.15.161:0/811894024 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f477814fda0 con 0x7f47780c4920 2024-03-28T23:33:17.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.728+0000 7f4766ffd640 1 -- 172.21.15.161:0/811894024 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f476c07a560 con 0x7f47780c4920 2024-03-28T23:33:17.729 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:33:17.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.730+0000 7f477cf0d640 1 -- 172.21.15.161:0/811894024 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f475806b500 msgr2=0x7f475806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:17.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.730+0000 7f477cf0d640 1 --2- 172.21.15.161:0/811894024 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f475806b500 0x7f475806d9c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f477814f680 tx=0x7f4768009040 comp rx=0 tx=0).stop 2024-03-28T23:33:17.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.730+0000 7f477cf0d640 1 -- 172.21.15.161:0/811894024 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47780c4920 msgr2=0x7f47781eb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:17.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.730+0000 7f477cf0d640 1 --2- 172.21.15.161:0/811894024 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47780c4920 0x7f47781eb9e0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f476c037040 tx=0x7f476c004730 comp rx=0 tx=0).stop 2024-03-28T23:33:17.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.730+0000 7f477cf0d640 1 -- 172.21.15.161:0/811894024 shutdown_connections 2024-03-28T23:33:17.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.730+0000 7f477cf0d640 1 --2- 172.21.15.161:0/811894024 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f475806b500 0x7f475806d9c0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.730+0000 7f477cf0d640 1 --2- 172.21.15.161:0/811894024 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47780c4920 0x7f47781eb9e0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.730+0000 7f477cf0d640 1 --2- 172.21.15.161:0/811894024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4778154450 0x7f47781571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.730+0000 7f477cf0d640 1 --2- 172.21.15.161:0/811894024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4760004800 0x7f477814cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.730+0000 7f477cf0d640 1 -- 172.21.15.161:0/811894024 >> 172.21.15.161:0/811894024 conn(0x7f47780c7430 msgr2=0x7f47781509d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:17.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.730+0000 7f477cf0d640 1 -- 172.21.15.161:0/811894024 shutdown_connections 2024-03-28T23:33:17.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.730+0000 7f477cf0d640 1 -- 172.21.15.161:0/811894024 wait complete. 2024-03-28T23:33:17.742 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime' 2024-03-28T23:33:17.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.842+0000 7fe7a441c640 1 Processor -- start 2024-03-28T23:33:17.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.842+0000 7fe7a441c640 1 -- start start 2024-03-28T23:33:17.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.843+0000 7fe7a441c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe79c14be70 0x7fe79c0ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:17.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.843+0000 7fe7a441c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe79c0ab6e0 0x7fe79c0aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:17.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.843+0000 7fe7a441c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe79c06ce80 0x7fe79c0aab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:17.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.843+0000 7fe7a441c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe79c14c250 con 0x7fe79c0ab6e0 2024-03-28T23:33:17.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.843+0000 7fe7a441c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fe79c1529c0 con 0x7fe79c06ce80 2024-03-28T23:33:17.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.843+0000 7fe7a441c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe79c0a9dc0 con 0x7fe79c14be70 2024-03-28T23:33:17.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.843+0000 7fe7a19b7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe79c0ab6e0 0x7fe79c0aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:17.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.843+0000 7fe7a19b7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe79c0ab6e0 0x7fe79c0aa5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48984/0 (socket says 172.21.15.161:48984) 2024-03-28T23:33:17.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.843+0000 7fe7a19b7640 1 -- 172.21.15.161:0/4065028326 learned_addr learned my addr 172.21.15.161:0/4065028326 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:17.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.843+0000 7fe7a21b8640 1 --2- 172.21.15.161:0/4065028326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe79c14be70 0x7fe79c0ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:17.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.843+0000 7fe7a29b9640 1 --2- 172.21.15.161:0/4065028326 >> v2:172.21.15.170:3300/0 conn(0x7fe79c06ce80 0x7fe79c0aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:17.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.843+0000 7fe7a19b7640 1 -- 172.21.15.161:0/4065028326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe79c14be70 msgr2=0x7fe79c0ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:17.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.843+0000 7fe7a19b7640 1 --2- 172.21.15.161:0/4065028326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe79c14be70 0x7fe79c0ab1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.843+0000 7fe7a19b7640 1 -- 172.21.15.161:0/4065028326 >> v2:172.21.15.170:3300/0 conn(0x7fe79c06ce80 msgr2=0x7fe79c0aab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:17.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.843+0000 7fe7a19b7640 1 --2- 172.21.15.161:0/4065028326 >> v2:172.21.15.170:3300/0 conn(0x7fe79c06ce80 0x7fe79c0aab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.843+0000 7fe7a19b7640 1 -- 172.21.15.161:0/4065028326 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe79c0aa0a0 con 0x7fe79c0ab6e0 2024-03-28T23:33:17.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.843+0000 7fe7a29b9640 1 --2- 172.21.15.161:0/4065028326 >> v2:172.21.15.170:3300/0 conn(0x7fe79c06ce80 0x7fe79c0aab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:17.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.844+0000 7fe7a19b7640 1 --2- 172.21.15.161:0/4065028326 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe79c0ab6e0 0x7fe79c0aa5c0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fe79c0c3020 tx=0x7fe78c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd61abecf9396d14 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:17.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.844+0000 7fe7a11b6640 1 -- 172.21.15.161:0/4065028326 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78c038070 con 0x7fe79c0ab6e0 2024-03-28T23:33:17.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.844+0000 7fe7a11b6640 1 -- 172.21.15.161:0/4065028326 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe78c02f050 con 0x7fe79c0ab6e0 2024-03-28T23:33:17.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.844+0000 7fe7a11b6640 1 -- 172.21.15.161:0/4065028326 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78c031970 con 0x7fe79c0ab6e0 2024-03-28T23:33:17.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.844+0000 7fe7a441c640 1 -- 172.21.15.161:0/4065028326 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe79c0ab6e0 msgr2=0x7fe79c0aa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:17.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.844+0000 7fe7a441c640 1 --2- 172.21.15.161:0/4065028326 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe79c0ab6e0 0x7fe79c0aa5c0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fe79c0c3020 tx=0x7fe78c02b7f0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.845+0000 7fe7a441c640 1 -- 172.21.15.161:0/4065028326 shutdown_connections 2024-03-28T23:33:17.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.845+0000 7fe7a441c640 1 --2- 172.21.15.161:0/4065028326 >> v2:172.21.15.170:3300/0 conn(0x7fe79c06ce80 0x7fe79c0aab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.845+0000 7fe7a441c640 1 --2- 172.21.15.161:0/4065028326 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe79c0ab6e0 0x7fe79c0aa5c0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.845+0000 7fe7a441c640 1 --2- 172.21.15.161:0/4065028326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe79c14be70 0x7fe79c0ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.845+0000 7fe7a441c640 1 -- 172.21.15.161:0/4065028326 >> 172.21.15.161:0/4065028326 conn(0x7fe79c0c6670 msgr2=0x7fe79c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:17.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.845+0000 7fe7a441c640 1 -- 172.21.15.161:0/4065028326 shutdown_connections 2024-03-28T23:33:17.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.845+0000 7fe7a441c640 1 -- 172.21.15.161:0/4065028326 wait complete. 2024-03-28T23:33:17.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.846+0000 7fe7a441c640 1 Processor -- start 2024-03-28T23:33:17.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.846+0000 7fe7a441c640 1 -- start start 2024-03-28T23:33:17.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.846+0000 7fe7a441c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe79c06ce80 0x7fe79c0bfc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:17.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.846+0000 7fe7a441c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe79c0ab6e0 0x7fe79c1487f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:17.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.846+0000 7fe7a441c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe79c14be70 0x7fe79c148d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:17.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.846+0000 7fe7a441c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe79c06b2d0 con 0x7fe79c0ab6e0 2024-03-28T23:33:17.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.846+0000 7fe7a441c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe79c06b150 con 0x7fe79c14be70 2024-03-28T23:33:17.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.846+0000 7fe7a441c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe79c0c1520 con 0x7fe79c06ce80 2024-03-28T23:33:17.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.846+0000 7fe7a19b7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe79c0ab6e0 0x7fe79c1487f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:17.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.846+0000 7fe7a19b7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe79c0ab6e0 0x7fe79c1487f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48998/0 (socket says 172.21.15.161:48998) 2024-03-28T23:33:17.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.846+0000 7fe7a19b7640 1 -- 172.21.15.161:0/3672045244 learned_addr learned my addr 172.21.15.161:0/3672045244 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:17.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.847+0000 7fe7a21b8640 1 --2- 172.21.15.161:0/3672045244 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe79c06ce80 0x7fe79c0bfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:17.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.847+0000 7fe7a29b9640 1 --2- 172.21.15.161:0/3672045244 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe79c14be70 0x7fe79c148d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:17.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.847+0000 7fe7a19b7640 1 -- 172.21.15.161:0/3672045244 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe79c06ce80 msgr2=0x7fe79c0bfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:17.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.847+0000 7fe7a19b7640 1 --2- 172.21.15.161:0/3672045244 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe79c06ce80 0x7fe79c0bfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.847+0000 7fe7a19b7640 1 -- 172.21.15.161:0/3672045244 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe79c14be70 msgr2=0x7fe79c148d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:17.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.847+0000 7fe7a19b7640 1 --2- 172.21.15.161:0/3672045244 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe79c14be70 0x7fe79c148d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:17.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.847+0000 7fe7a19b7640 1 -- 172.21.15.161:0/3672045244 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe79c149270 con 0x7fe79c0ab6e0 2024-03-28T23:33:17.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.847+0000 7fe7a19b7640 1 --2- 172.21.15.161:0/3672045244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe79c0ab6e0 0x7fe79c1487f0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fe78c009960 tx=0x7fe78c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:17.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.847+0000 7fe792ffd640 1 -- 172.21.15.161:0/3672045244 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78c042020 con 0x7fe79c0ab6e0 2024-03-28T23:33:17.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.847+0000 7fe792ffd640 1 -- 172.21.15.161:0/3672045244 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe78c03d680 con 0x7fe79c0ab6e0 2024-03-28T23:33:17.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.847+0000 7fe7a441c640 1 -- 172.21.15.161:0/3672045244 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe79c150a80 con 0x7fe79c0ab6e0 2024-03-28T23:33:17.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.847+0000 7fe792ffd640 1 -- 172.21.15.161:0/3672045244 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78c038040 con 0x7fe79c0ab6e0 2024-03-28T23:33:17.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.847+0000 7fe7a441c640 1 -- 172.21.15.161:0/3672045244 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe79c0c5bb0 con 0x7fe79c0ab6e0 2024-03-28T23:33:17.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.848+0000 7fe7a441c640 1 -- 172.21.15.161:0/3672045244 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe79c04f4b0 con 0x7fe79c0ab6e0 2024-03-28T23:33:17.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.848+0000 7fe792ffd640 1 -- 172.21.15.161:0/3672045244 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe78c045020 con 0x7fe79c0ab6e0 2024-03-28T23:33:17.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.849+0000 7fe792ffd640 1 --2- 172.21.15.161:0/3672045244 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe78006b4d0 0x7fe78006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:17.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.849+0000 7fe792ffd640 1 -- 172.21.15.161:0/3672045244 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5593+0+0 (secure 0 0 0) 0x7fe78c0ab9e0 con 0x7fe79c0ab6e0 2024-03-28T23:33:17.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.849+0000 7fe7a21b8640 1 --2- 172.21.15.161:0/3672045244 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe78006b4d0 0x7fe78006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:17.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.849+0000 7fe7a21b8640 1 --2- 172.21.15.161:0/3672045244 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe78006b4d0 0x7fe78006d990 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe798009aa0 tx=0x7fe798009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:17.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:17.853+0000 7fe792ffd640 1 -- 172.21.15.161:0/3672045244 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe78c04c030 con 0x7fe79c0ab6e0 2024-03-28T23:33:18.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.090+0000 7fe7a441c640 1 -- 172.21.15.161:0/3672045244 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime"]} v 0) -- 0x7fe79c0c0ff0 con 0x7fe79c0ab6e0 2024-03-28T23:33:18.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.127+0000 7fe792ffd640 1 -- 172.21.15.161:0/3672045244 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime"]}]=0 v212) ==== 149+0+0 (secure 0 0 0) 0x7fe78c07a310 con 0x7fe79c0ab6e0 2024-03-28T23:33:18.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.129+0000 7fe7a441c640 1 -- 172.21.15.161:0/3672045244 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe78006b4d0 msgr2=0x7fe78006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:18.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.129+0000 7fe7a441c640 1 --2- 172.21.15.161:0/3672045244 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe78006b4d0 0x7fe78006d990 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe798009aa0 tx=0x7fe798009190 comp rx=0 tx=0).stop 2024-03-28T23:33:18.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.129+0000 7fe7a441c640 1 -- 172.21.15.161:0/3672045244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe79c0ab6e0 msgr2=0x7fe79c1487f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:18.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.129+0000 7fe7a441c640 1 --2- 172.21.15.161:0/3672045244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe79c0ab6e0 0x7fe79c1487f0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fe78c009960 tx=0x7fe78c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.129+0000 7fe7a441c640 1 -- 172.21.15.161:0/3672045244 shutdown_connections 2024-03-28T23:33:18.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.129+0000 7fe7a441c640 1 --2- 172.21.15.161:0/3672045244 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe78006b4d0 0x7fe78006d990 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.129+0000 7fe7a441c640 1 --2- 172.21.15.161:0/3672045244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe79c0ab6e0 0x7fe79c1487f0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.129+0000 7fe7a441c640 1 --2- 172.21.15.161:0/3672045244 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe79c14be70 0x7fe79c148d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.129+0000 7fe7a441c640 1 --2- 172.21.15.161:0/3672045244 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe79c06ce80 0x7fe79c0bfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.129+0000 7fe7a441c640 1 -- 172.21.15.161:0/3672045244 >> 172.21.15.161:0/3672045244 conn(0x7fe79c0c6670 msgr2=0x7fe79c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:18.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.130+0000 7fe7a441c640 1 -- 172.21.15.161:0/3672045244 shutdown_connections 2024-03-28T23:33:18.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.130+0000 7fe7a441c640 1 -- 172.21.15.161:0/3672045244 wait complete. 2024-03-28T23:33:18.141 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-28T23:33:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.244+0000 7fb623f56640 1 Processor -- start 2024-03-28T23:33:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.245+0000 7fb623f56640 1 -- start start 2024-03-28T23:33:18.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.245+0000 7fb623f56640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb61c152bc0 0x7fb61c0c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:18.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.245+0000 7fb623f56640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb61c0c8a80 0x7fb61c0c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:18.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.245+0000 7fb623f56640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb61c0c6f60 con 0x7fb61c0c9420 2024-03-28T23:33:18.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.245+0000 7fb623f56640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb61c0c6120 con 0x7fb61c0c8a80 2024-03-28T23:33:18.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.245+0000 7fb623f56640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fb61c0c6400 con 0x7fb61c152bc0 2024-03-28T23:33:18.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.245+0000 7fb6224f3640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb61c0c9420 0x7fb61c1589b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:39502/0 (socket says 172.21.15.161:39502) 2024-03-28T23:33:18.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.245+0000 7fb6224f3640 1 -- 172.21.15.161:0/4133047361 learned_addr learned my addr 172.21.15.161:0/4133047361 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:18.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.245+0000 7fb6214f1640 1 --2- 172.21.15.161:0/4133047361 >> v2:172.21.15.161:3300/0 conn(0x7fb61c0c8a80 0x7fb61c0c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:18.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.246+0000 7fb621cf2640 1 --2- 172.21.15.161:0/4133047361 >> v2:172.21.15.170:3300/0 conn(0x7fb61c152bc0 0x7fb61c0c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:18.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.246+0000 7fb6214f1640 1 -- 172.21.15.161:0/4133047361 >> v2:172.21.15.170:3300/0 conn(0x7fb61c152bc0 msgr2=0x7fb61c0c0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:18.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.246+0000 7fb6214f1640 1 --2- 172.21.15.161:0/4133047361 >> v2:172.21.15.170:3300/0 conn(0x7fb61c152bc0 0x7fb61c0c0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.246+0000 7fb6214f1640 1 -- 172.21.15.161:0/4133047361 >> v1:172.21.15.161:6789/0 conn(0x7fb61c0c9420 legacy=0x7fb61c1589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:18.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.246+0000 7fb6214f1640 1 -- 172.21.15.161:0/4133047361 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb61c0c7240 con 0x7fb61c0c8a80 2024-03-28T23:33:18.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.246+0000 7fb620cf0640 1 -- 172.21.15.161:0/4133047361 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1058174617 0 0) 0x7fb61c0c6f60 con 0x7fb61c0c9420 2024-03-28T23:33:18.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.246+0000 7fb6214f1640 1 --2- 172.21.15.161:0/4133047361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb61c0c8a80 0x7fb61c0c8ee0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fb61c0aa410 tx=0x7fb61802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be675bd46de1196c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:18.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.247+0000 7fb620cf0640 1 -- 172.21.15.161:0/4133047361 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb618035020 con 0x7fb61c0c8a80 2024-03-28T23:33:18.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.247+0000 7fb620cf0640 1 -- 172.21.15.161:0/4133047361 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb618031830 con 0x7fb61c0c8a80 2024-03-28T23:33:18.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.247+0000 7fb620cf0640 1 -- 172.21.15.161:0/4133047361 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb618031c00 con 0x7fb61c0c8a80 2024-03-28T23:33:18.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.247+0000 7fb623f56640 1 -- 172.21.15.161:0/4133047361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb61c0c8a80 msgr2=0x7fb61c0c8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:18.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.247+0000 7fb623f56640 1 --2- 172.21.15.161:0/4133047361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb61c0c8a80 0x7fb61c0c8ee0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fb61c0aa410 tx=0x7fb61802b8d0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.247+0000 7fb623f56640 1 -- 172.21.15.161:0/4133047361 shutdown_connections 2024-03-28T23:33:18.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.247+0000 7fb623f56640 1 --2- 172.21.15.161:0/4133047361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb61c0c8a80 0x7fb61c0c8ee0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.247+0000 7fb623f56640 1 --2- 172.21.15.161:0/4133047361 >> v2:172.21.15.170:3300/0 conn(0x7fb61c152bc0 0x7fb61c0c0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.247+0000 7fb623f56640 1 -- 172.21.15.161:0/4133047361 >> 172.21.15.161:0/4133047361 conn(0x7fb61c0caa80 msgr2=0x7fb61c0cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:18.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.247+0000 7fb623f56640 1 -- 172.21.15.161:0/4133047361 shutdown_connections 2024-03-28T23:33:18.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.247+0000 7fb623f56640 1 -- 172.21.15.161:0/4133047361 wait complete. 2024-03-28T23:33:18.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.248+0000 7fb623f56640 1 Processor -- start 2024-03-28T23:33:18.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.248+0000 7fb623f56640 1 -- start start 2024-03-28T23:33:18.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.249+0000 7fb623f56640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb61c0c8a80 0x7fb61c14bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:18.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.249+0000 7fb623f56640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb61c0c9420 0x7fb61c1e0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:18.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.249+0000 7fb623f56640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb61c152bc0 0x7fb61c1e0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:18.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.249+0000 7fb623f56640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb61c152fa0 con 0x7fb61c0c9420 2024-03-28T23:33:18.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.249+0000 7fb623f56640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb61c1590c0 con 0x7fb61c152bc0 2024-03-28T23:33:18.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.249+0000 7fb623f56640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb61c146660 con 0x7fb61c0c8a80 2024-03-28T23:33:18.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.249+0000 7fb6214f1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb61c0c9420 0x7fb61c1e0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:18.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.249+0000 7fb6214f1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb61c0c9420 0x7fb61c1e0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49016/0 (socket says 172.21.15.161:49016) 2024-03-28T23:33:18.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.249+0000 7fb6214f1640 1 -- 172.21.15.161:0/1789531453 learned_addr learned my addr 172.21.15.161:0/1789531453 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:18.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.249+0000 7fb6214f1640 1 -- 172.21.15.161:0/1789531453 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb61c0c8a80 msgr2=0x7fb61c14bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:18.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.249+0000 7fb6214f1640 1 --2- 172.21.15.161:0/1789531453 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb61c0c8a80 0x7fb61c14bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.249+0000 7fb6214f1640 1 -- 172.21.15.161:0/1789531453 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb61c152bc0 msgr2=0x7fb61c1e0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:18.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.249+0000 7fb6214f1640 1 --2- 172.21.15.161:0/1789531453 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb61c152bc0 0x7fb61c1e0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.249+0000 7fb6214f1640 1 -- 172.21.15.161:0/1789531453 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb61c1461b0 con 0x7fb61c0c9420 2024-03-28T23:33:18.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.250+0000 7fb6214f1640 1 --2- 172.21.15.161:0/1789531453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb61c0c9420 0x7fb61c1e0020 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fb618009890 tx=0x7fb6180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:18.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.250+0000 7fb6127fc640 1 -- 172.21.15.161:0/1789531453 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb618035070 con 0x7fb61c0c9420 2024-03-28T23:33:18.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.250+0000 7fb6127fc640 1 -- 172.21.15.161:0/1789531453 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb618004250 con 0x7fb61c0c9420 2024-03-28T23:33:18.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.250+0000 7fb6127fc640 1 -- 172.21.15.161:0/1789531453 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61803a6f0 con 0x7fb61c0c9420 2024-03-28T23:33:18.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.250+0000 7fb623f56640 1 -- 172.21.15.161:0/1789531453 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb61c14c4a0 con 0x7fb61c0c9420 2024-03-28T23:33:18.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.250+0000 7fb623f56640 1 -- 172.21.15.161:0/1789531453 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb61c1515f0 con 0x7fb61c0c9420 2024-03-28T23:33:18.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.251+0000 7fb623f56640 1 -- 172.21.15.161:0/1789531453 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb61c04f4b0 con 0x7fb61c0c9420 2024-03-28T23:33:18.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.251+0000 7fb6127fc640 1 -- 172.21.15.161:0/1789531453 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb618038020 con 0x7fb61c0c9420 2024-03-28T23:33:18.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.252+0000 7fb6127fc640 1 --2- 172.21.15.161:0/1789531453 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb60006b4d0 0x7fb60006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:18.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.252+0000 7fb6127fc640 1 -- 172.21.15.161:0/1789531453 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5593+0+0 (secure 0 0 0) 0x7fb6180abeb0 con 0x7fb61c0c9420 2024-03-28T23:33:18.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.252+0000 7fb621cf2640 1 --2- 172.21.15.161:0/1789531453 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb60006b4d0 0x7fb60006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:18.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.252+0000 7fb621cf2640 1 --2- 172.21.15.161:0/1789531453 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb60006b4d0 0x7fb60006d990 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb60c006fd0 tx=0x7fb60c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:18.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.256+0000 7fb6127fc640 1 -- 172.21.15.161:0/1789531453 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb618040070 con 0x7fb61c0c9420 2024-03-28T23:33:18.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.429+0000 7fb623f56640 1 -- 172.21.15.161:0/1789531453 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fb61c14f8f0 con 0x7fb61c0c9420 2024-03-28T23:33:18.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.432+0000 7fb6127fc640 1 -- 172.21.15.161:0/1789531453 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v31)=0 v31) ==== 122+0+0 (secure 0 0 0) 0x7fb61807ada0 con 0x7fb61c0c9420 2024-03-28T23:33:18.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.434+0000 7fb623f56640 1 -- 172.21.15.161:0/1789531453 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb60006b4d0 msgr2=0x7fb60006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:18.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.434+0000 7fb623f56640 1 --2- 172.21.15.161:0/1789531453 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb60006b4d0 0x7fb60006d990 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb60c006fd0 tx=0x7fb60c008040 comp rx=0 tx=0).stop 2024-03-28T23:33:18.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.434+0000 7fb623f56640 1 -- 172.21.15.161:0/1789531453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb61c0c9420 msgr2=0x7fb61c1e0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:18.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.434+0000 7fb623f56640 1 --2- 172.21.15.161:0/1789531453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb61c0c9420 0x7fb61c1e0020 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fb618009890 tx=0x7fb6180046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.434+0000 7fb623f56640 1 -- 172.21.15.161:0/1789531453 shutdown_connections 2024-03-28T23:33:18.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.434+0000 7fb623f56640 1 --2- 172.21.15.161:0/1789531453 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb60006b4d0 0x7fb60006d990 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.434+0000 7fb623f56640 1 --2- 172.21.15.161:0/1789531453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb61c0c9420 0x7fb61c1e0020 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.434+0000 7fb623f56640 1 --2- 172.21.15.161:0/1789531453 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb61c152bc0 0x7fb61c1e0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.434+0000 7fb623f56640 1 --2- 172.21.15.161:0/1789531453 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb61c0c8a80 0x7fb61c14bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.434+0000 7fb623f56640 1 -- 172.21.15.161:0/1789531453 >> 172.21.15.161:0/1789531453 conn(0x7fb61c0caa80 msgr2=0x7fb61c14d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:18.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.434+0000 7fb623f56640 1 -- 172.21.15.161:0/1789531453 shutdown_connections 2024-03-28T23:33:18.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.434+0000 7fb623f56640 1 -- 172.21.15.161:0/1789531453 wait complete. 2024-03-28T23:33:18.445 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-28T23:33:18.446 DEBUG:teuthology.orchestra.run.smithi161:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:33:18.499 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:33:18.550 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-28T23:33:18.550 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:33:18.550 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:33:18.566 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20793 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-28T23:33:18.566 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:33:18.566 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:33:18.584 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056374,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19690 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-28T23:33:18.584 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-28T23:33:18.584 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:33:18.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.707+0000 7fd69f36c640 1 Processor -- start 2024-03-28T23:33:18.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.708+0000 7fd69f36c640 1 -- start start 2024-03-28T23:33:18.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.708+0000 7fd69f36c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd698152890 0x7fd6980c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:18.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.708+0000 7fd69f36c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd6980c8ed0 0x7fd6980c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:18.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.708+0000 7fd69f36c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6980c5950 0x7fd6981553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:18.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.708+0000 7fd69f36c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd6980c8480 con 0x7fd698152890 2024-03-28T23:33:18.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.708+0000 7fd69f36c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd698159640 con 0x7fd6980c8ed0 2024-03-28T23:33:18.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.708+0000 7fd69f36c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd698158a30 con 0x7fd6980c5950 2024-03-28T23:33:18.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.708+0000 7fd69d108640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd698152890 0x7fd6980c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:18.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.708+0000 7fd69d108640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd698152890 0x7fd6980c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49024/0 (socket says 172.21.15.161:49024) 2024-03-28T23:33:18.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.708+0000 7fd69d108640 1 -- 172.21.15.161:0/891057853 learned_addr learned my addr 172.21.15.161:0/891057853 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:18.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.709+0000 7fd69d108640 1 -- 172.21.15.161:0/891057853 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6980c5950 msgr2=0x7fd6981553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:18.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.709+0000 7fd69d108640 1 --2- 172.21.15.161:0/891057853 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6980c5950 0x7fd6981553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.709+0000 7fd69d108640 1 -- 172.21.15.161:0/891057853 >> v2:172.21.15.170:3300/0 conn(0x7fd6980c8ed0 msgr2=0x7fd6980c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:18.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.709+0000 7fd69d108640 1 --2- 172.21.15.161:0/891057853 >> v2:172.21.15.170:3300/0 conn(0x7fd6980c8ed0 0x7fd6980c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.709+0000 7fd69d108640 1 -- 172.21.15.161:0/891057853 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd698158d10 con 0x7fd698152890 2024-03-28T23:33:18.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.709+0000 7fd69d108640 1 --2- 172.21.15.161:0/891057853 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd698152890 0x7fd6980c8990 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fd6980ad6e0 tx=0x7fd69002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c282c6eea59ce473 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:18.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.710+0000 7fd68ffff640 1 -- 172.21.15.161:0/891057853 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd69002c950 con 0x7fd698152890 2024-03-28T23:33:18.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.710+0000 7fd68ffff640 1 -- 172.21.15.161:0/891057853 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd69002caf0 con 0x7fd698152890 2024-03-28T23:33:18.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.710+0000 7fd68ffff640 1 -- 172.21.15.161:0/891057853 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6900343f0 con 0x7fd698152890 2024-03-28T23:33:18.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.710+0000 7fd69f36c640 1 -- 172.21.15.161:0/891057853 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd698152890 msgr2=0x7fd6980c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:18.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.710+0000 7fd69f36c640 1 --2- 172.21.15.161:0/891057853 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd698152890 0x7fd6980c8990 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fd6980ad6e0 tx=0x7fd69002bdb0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.710+0000 7fd69f36c640 1 -- 172.21.15.161:0/891057853 shutdown_connections 2024-03-28T23:33:18.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.710+0000 7fd69f36c640 1 --2- 172.21.15.161:0/891057853 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6980c5950 0x7fd6981553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.710+0000 7fd69f36c640 1 --2- 172.21.15.161:0/891057853 >> v2:172.21.15.170:3300/0 conn(0x7fd6980c8ed0 0x7fd6980c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.710+0000 7fd69f36c640 1 --2- 172.21.15.161:0/891057853 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd698152890 0x7fd6980c8990 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.710+0000 7fd69f36c640 1 -- 172.21.15.161:0/891057853 >> 172.21.15.161:0/891057853 conn(0x7fd6980ac620 msgr2=0x7fd6980aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:18.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.710+0000 7fd69f36c640 1 -- 172.21.15.161:0/891057853 shutdown_connections 2024-03-28T23:33:18.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.710+0000 7fd69f36c640 1 -- 172.21.15.161:0/891057853 wait complete. 2024-03-28T23:33:18.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.711+0000 7fd69f36c640 1 Processor -- start 2024-03-28T23:33:18.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.711+0000 7fd69f36c640 1 -- start start 2024-03-28T23:33:18.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.712+0000 7fd69f36c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6980c5950 0x7fd6981eb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:18.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.712+0000 7fd69f36c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6980c8ed0 0x7fd6980be5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:18.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.712+0000 7fd69f36c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd698152890 0x7fd6980ab430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:18.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.712+0000 7fd69f36c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd698152c70 con 0x7fd698152890 2024-03-28T23:33:18.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.712+0000 7fd69f36c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd6981592d0 con 0x7fd6980c5950 2024-03-28T23:33:18.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.712+0000 7fd69f36c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd698152df0 con 0x7fd6980c8ed0 2024-03-28T23:33:18.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.712+0000 7fd69d909640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd698152890 0x7fd6980ab430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:18.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.712+0000 7fd69d909640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd698152890 0x7fd6980ab430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49036/0 (socket says 172.21.15.161:49036) 2024-03-28T23:33:18.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.712+0000 7fd69d909640 1 -- 172.21.15.161:0/2378786931 learned_addr learned my addr 172.21.15.161:0/2378786931 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:18.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.712+0000 7fd69d108640 1 --2- 172.21.15.161:0/2378786931 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6980c5950 0x7fd6981eb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:18.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.712+0000 7fd69c907640 1 --2- 172.21.15.161:0/2378786931 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6980c8ed0 0x7fd6980be5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:18.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.712+0000 7fd69d909640 1 -- 172.21.15.161:0/2378786931 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6980c8ed0 msgr2=0x7fd6980be5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:18.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.712+0000 7fd69d909640 1 --2- 172.21.15.161:0/2378786931 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6980c8ed0 0x7fd6980be5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.712+0000 7fd69d909640 1 -- 172.21.15.161:0/2378786931 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6980c5950 msgr2=0x7fd6981eb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:18.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.712+0000 7fd69d909640 1 --2- 172.21.15.161:0/2378786931 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6980c5950 0x7fd6981eb790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.712+0000 7fd69d909640 1 -- 172.21.15.161:0/2378786931 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6981eacd0 con 0x7fd698152890 2024-03-28T23:33:18.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.712+0000 7fd69d108640 1 --2- 172.21.15.161:0/2378786931 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6980c5950 0x7fd6981eb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:18.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.712+0000 7fd69c907640 1 --2- 172.21.15.161:0/2378786931 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6980c8ed0 0x7fd6980be5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:18.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.713+0000 7fd69d909640 1 --2- 172.21.15.161:0/2378786931 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd698152890 0x7fd6980ab430 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fd69400b440 tx=0x7fd69400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:18.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.713+0000 7fd68dffb640 1 -- 172.21.15.161:0/2378786931 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd694015020 con 0x7fd698152890 2024-03-28T23:33:18.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.713+0000 7fd68dffb640 1 -- 172.21.15.161:0/2378786931 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd694011830 con 0x7fd698152890 2024-03-28T23:33:18.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.713+0000 7fd69f36c640 1 -- 172.21.15.161:0/2378786931 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6980c00f0 con 0x7fd698152890 2024-03-28T23:33:18.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.713+0000 7fd68dffb640 1 -- 172.21.15.161:0/2378786931 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd694011c20 con 0x7fd698152890 2024-03-28T23:33:18.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.713+0000 7fd69f36c640 1 -- 172.21.15.161:0/2378786931 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6981eaf90 con 0x7fd698152890 2024-03-28T23:33:18.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.714+0000 7fd69f36c640 1 -- 172.21.15.161:0/2378786931 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd69804f4b0 con 0x7fd698152890 2024-03-28T23:33:18.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.714+0000 7fd68dffb640 1 -- 172.21.15.161:0/2378786931 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd694018020 con 0x7fd698152890 2024-03-28T23:33:18.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.715+0000 7fd68dffb640 1 --2- 172.21.15.161:0/2378786931 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd67c06b500 0x7fd67c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:18.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.715+0000 7fd69d108640 1 --2- 172.21.15.161:0/2378786931 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd67c06b500 0x7fd67c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:18.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.715+0000 7fd68dffb640 1 -- 172.21.15.161:0/2378786931 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5593+0+0 (secure 0 0 0) 0x7fd694089f90 con 0x7fd698152890 2024-03-28T23:33:18.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.715+0000 7fd69d108640 1 --2- 172.21.15.161:0/2378786931 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd67c06b500 0x7fd67c06d9c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd690035000 tx=0x7fd690030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:18.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.719+0000 7fd68dffb640 1 -- 172.21.15.161:0/2378786931 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd69401f020 con 0x7fd698152890 2024-03-28T23:33:18.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.925+0000 7fd69f36c640 1 -- 172.21.15.161:0/2378786931 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6980c8140 con 0x7fd698152890 2024-03-28T23:33:18.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.926+0000 7fd68dffb640 1 -- 172.21.15.161:0/2378786931 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) ==== 94+0+3881 (secure 0 0 0) 0x7fd6940172c0 con 0x7fd698152890 2024-03-28T23:33:18.927 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:18.927 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":75,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5579,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.170:6834/3501962065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3501962065},{"type":"v1","addr":"172.21.15.170:6834","nonce":3501962065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":75},{"gid":6397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2706065827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":2706065827},{"type":"v1","addr":"172.21.15.170:6835","nonce":2706065827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":75},{"gid":6903,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/3336491043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3336491043},{"type":"v1","addr":"172.21.15.161:6836","nonce":3336491043}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69},{"gid":6993,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1463250942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":1463250942},{"type":"v1","addr":"172.21.15.161:6837","nonce":1463250942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":75}],"filesystems":[]} 2024-03-28T23:33:18.927 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 75 2024-03-28T23:33:18.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.928+0000 7fd69f36c640 1 -- 172.21.15.161:0/2378786931 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd67c06b500 msgr2=0x7fd67c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:18.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.928+0000 7fd69f36c640 1 --2- 172.21.15.161:0/2378786931 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd67c06b500 0x7fd67c06d9c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd690035000 tx=0x7fd690030000 comp rx=0 tx=0).stop 2024-03-28T23:33:18.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.928+0000 7fd69f36c640 1 -- 172.21.15.161:0/2378786931 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd698152890 msgr2=0x7fd6980ab430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:18.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.928+0000 7fd69f36c640 1 --2- 172.21.15.161:0/2378786931 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd698152890 0x7fd6980ab430 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fd69400b440 tx=0x7fd69400b900 comp rx=0 tx=0).stop 2024-03-28T23:33:18.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.928+0000 7fd69f36c640 1 -- 172.21.15.161:0/2378786931 shutdown_connections 2024-03-28T23:33:18.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.928+0000 7fd69f36c640 1 --2- 172.21.15.161:0/2378786931 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd67c06b500 0x7fd67c06d9c0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.928+0000 7fd69f36c640 1 --2- 172.21.15.161:0/2378786931 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd698152890 0x7fd6980ab430 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.928+0000 7fd69f36c640 1 --2- 172.21.15.161:0/2378786931 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6980c8ed0 0x7fd6980be5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.928+0000 7fd69f36c640 1 --2- 172.21.15.161:0/2378786931 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6980c5950 0x7fd6981eb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:18.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.928+0000 7fd69f36c640 1 -- 172.21.15.161:0/2378786931 >> 172.21.15.161:0/2378786931 conn(0x7fd6980ac620 msgr2=0x7fd69806cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:18.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.928+0000 7fd69f36c640 1 -- 172.21.15.161:0/2378786931 shutdown_connections 2024-03-28T23:33:18.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:18.928+0000 7fd69f36c640 1 -- 172.21.15.161:0/2378786931 wait complete. 2024-03-28T23:33:18.939 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T23:33:18.940 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-28T23:33:18.940 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T23:33:18.940 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:33:18.940 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T23:33:18.940 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-28T23:33:18.940 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T23:33:18.940 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:33:18.941 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-28T23:33:18.941 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-28T23:33:18.941 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-28T23:33:18.941 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:33:18.941 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-28T23:33:18.941 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-28T23:33:18.941 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-28T23:33:18.942 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:33:18.949 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T23:33:18.950 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T23:33:18.950 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T23:33:18.950 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T23:33:18.952 INFO:tasks.ceph.mds.b:Stopped 2024-03-28T23:33:18.952 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-28T23:33:18.953 INFO:tasks.ceph.mds.d:Stopped 2024-03-28T23:33:18.953 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-28T23:33:18.992 INFO:tasks.ceph.mds.c:Started 2024-03-28T23:33:18.992 DEBUG:teuthology.parallel:result is None 2024-03-28T23:33:18.992 INFO:tasks.ceph.mds.a:Started 2024-03-28T23:33:18.993 DEBUG:teuthology.parallel:result is None 2024-03-28T23:33:18.995 INFO:tasks.ceph.mds.b:Started 2024-03-28T23:33:18.995 DEBUG:teuthology.parallel:result is None 2024-03-28T23:33:18.995 INFO:tasks.ceph.mds.d:Started 2024-03-28T23:33:18.996 DEBUG:teuthology.parallel:result is None 2024-03-28T23:33:18.996 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-28T23:33:19.069 INFO:tasks.ceph.mds.c.smithi161.stdout:starting mds.c at 2024-03-28T23:33:19.070 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-28T23:33:19.078 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2024-03-28T23:33:19.079 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2024-03-28T23:33:19.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.102+0000 7f2abb874640 1 Processor -- start 2024-03-28T23:33:19.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.103+0000 7f2abb874640 1 -- start start 2024-03-28T23:33:19.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.103+0000 7f2abb874640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2ab4152890 0x7f2ab40c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:19.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.103+0000 7f2abb874640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ab40c8ed0 0x7f2ab40c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:19.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.103+0000 7f2abb874640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ab4158a30 con 0x7f2ab40c5950 2024-03-28T23:33:19.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.104+0000 7f2abb874640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2ab40c8480 con 0x7f2ab4152890 2024-03-28T23:33:19.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.104+0000 7f2abb874640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2ab4159640 con 0x7f2ab40c8ed0 2024-03-28T23:33:19.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.104+0000 7f2ab9610640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2ab4152890 0x7f2ab40c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:19.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.104+0000 7f2ab9610640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2ab4152890 0x7f2ab40c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49090/0 (socket says 172.21.15.161:49090) 2024-03-28T23:33:19.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.104+0000 7f2ab9610640 1 -- 172.21.15.161:0/76946977 learned_addr learned my addr 172.21.15.161:0/76946977 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:19.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.104+0000 7f2ab9610640 1 -- 172.21.15.161:0/76946977 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ab40c8ed0 msgr2=0x7f2ab40c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:19.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.104+0000 7f2ab8e0f640 1 --2- 172.21.15.161:0/76946977 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ab40c8ed0 0x7f2ab40c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:19.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.104+0000 7f2ab9610640 1 --2- 172.21.15.161:0/76946977 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ab40c8ed0 0x7f2ab40c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:19.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.104+0000 7f2ab9610640 1 -- 172.21.15.161:0/76946977 >> v1:172.21.15.170:6789/0 conn(0x7f2ab40c5950 legacy=0x7f2ab41553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:19.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.104+0000 7f2ab9610640 1 -- 172.21.15.161:0/76946977 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab4158d10 con 0x7f2ab4152890 2024-03-28T23:33:19.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.104+0000 7f2ab8e0f640 1 --2- 172.21.15.161:0/76946977 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ab40c8ed0 0x7f2ab40c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:19.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.105+0000 7f2ab9610640 1 --2- 172.21.15.161:0/76946977 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ab4152890 0x7f2ab40c8990 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0009870 tx=0x7f2ab002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caa01aac7ff15bb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:19.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.105+0000 7f2aabfff640 1 -- 172.21.15.161:0/76946977 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab0038070 con 0x7f2ab4152890 2024-03-28T23:33:19.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.105+0000 7f2aabfff640 1 -- 172.21.15.161:0/76946977 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ab0031740 con 0x7f2ab4152890 2024-03-28T23:33:19.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.105+0000 7f2aabfff640 1 -- 172.21.15.161:0/76946977 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab002f050 con 0x7f2ab4152890 2024-03-28T23:33:19.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.105+0000 7f2abb874640 1 -- 172.21.15.161:0/76946977 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ab4152890 msgr2=0x7f2ab40c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:19.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.105+0000 7f2abb874640 1 --2- 172.21.15.161:0/76946977 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ab4152890 0x7f2ab40c8990 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0009870 tx=0x7f2ab002b7e0 comp rx=0 tx=0).stop 2024-03-28T23:33:19.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.105+0000 7f2abb874640 1 -- 172.21.15.161:0/76946977 shutdown_connections 2024-03-28T23:33:19.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.105+0000 7f2abb874640 1 --2- 172.21.15.161:0/76946977 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ab40c8ed0 0x7f2ab40c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:19.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.105+0000 7f2abb874640 1 --2- 172.21.15.161:0/76946977 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ab4152890 0x7f2ab40c8990 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:19.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.105+0000 7f2abb874640 1 -- 172.21.15.161:0/76946977 >> 172.21.15.161:0/76946977 conn(0x7f2ab40ac620 msgr2=0x7f2ab40aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:19.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.105+0000 7f2abb874640 1 -- 172.21.15.161:0/76946977 shutdown_connections 2024-03-28T23:33:19.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.105+0000 7f2abb874640 1 -- 172.21.15.161:0/76946977 wait complete. 2024-03-28T23:33:19.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.106+0000 7f2abb874640 1 Processor -- start 2024-03-28T23:33:19.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.107+0000 7f2abb874640 1 -- start start 2024-03-28T23:33:19.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.107+0000 7f2abb874640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ab40c5950 0x7f2ab41e2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:19.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.107+0000 7f2abb874640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ab40c8ed0 0x7f2ab414ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:19.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.107+0000 7f2abb874640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ab4152890 0x7f2ab40ad6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:19.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.107+0000 7f2abb874640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab4152c70 con 0x7f2ab40c8ed0 2024-03-28T23:33:19.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.107+0000 7f2abb874640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab41592d0 con 0x7f2ab4152890 2024-03-28T23:33:19.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.107+0000 7f2abb874640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2ab4152df0 con 0x7f2ab40c5950 2024-03-28T23:33:19.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.107+0000 7f2ab8e0f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ab40c8ed0 0x7f2ab414ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:19.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.107+0000 7f2ab8e0f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ab40c8ed0 0x7f2ab414ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49098/0 (socket says 172.21.15.161:49098) 2024-03-28T23:33:19.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.107+0000 7f2ab8e0f640 1 -- 172.21.15.161:0/3944515513 learned_addr learned my addr 172.21.15.161:0/3944515513 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:19.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.108+0000 7f2ab9610640 1 --2- 172.21.15.161:0/3944515513 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ab40c5950 0x7f2ab41e2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:19.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.108+0000 7f2ab9e11640 1 --2- 172.21.15.161:0/3944515513 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ab4152890 0x7f2ab40ad6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:19.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.108+0000 7f2ab8e0f640 1 -- 172.21.15.161:0/3944515513 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ab40c5950 msgr2=0x7f2ab41e2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:19.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.108+0000 7f2ab8e0f640 1 --2- 172.21.15.161:0/3944515513 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ab40c5950 0x7f2ab41e2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:19.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.108+0000 7f2ab8e0f640 1 -- 172.21.15.161:0/3944515513 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ab4152890 msgr2=0x7f2ab40ad6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:19.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.108+0000 7f2ab8e0f640 1 --2- 172.21.15.161:0/3944515513 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ab4152890 0x7f2ab40ad6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:19.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.108+0000 7f2ab8e0f640 1 -- 172.21.15.161:0/3944515513 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab414e7a0 con 0x7f2ab40c8ed0 2024-03-28T23:33:19.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.108+0000 7f2ab8e0f640 1 --2- 172.21.15.161:0/3944515513 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ab40c8ed0 0x7f2ab414ddf0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f2aa400c7f0 tx=0x7f2aa400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:19.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.108+0000 7f2aa9ffb640 1 -- 172.21.15.161:0/3944515513 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa4005b90 con 0x7f2ab40c8ed0 2024-03-28T23:33:19.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.108+0000 7f2aa9ffb640 1 -- 172.21.15.161:0/3944515513 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aa4005d30 con 0x7f2ab40c8ed0 2024-03-28T23:33:19.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.108+0000 7f2abb874640 1 -- 172.21.15.161:0/3944515513 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ab40af0c0 con 0x7f2ab40c8ed0 2024-03-28T23:33:19.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.108+0000 7f2aa9ffb640 1 -- 172.21.15.161:0/3944515513 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa4007a10 con 0x7f2ab40c8ed0 2024-03-28T23:33:19.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.108+0000 7f2abb874640 1 -- 172.21.15.161:0/3944515513 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ab40af950 con 0x7f2ab40c8ed0 2024-03-28T23:33:19.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.109+0000 7f2abb874640 1 -- 172.21.15.161:0/3944515513 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ab404f4b0 con 0x7f2ab40c8ed0 2024-03-28T23:33:19.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.109+0000 7f2aa9ffb640 1 -- 172.21.15.161:0/3944515513 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2aa4007bb0 con 0x7f2ab40c8ed0 2024-03-28T23:33:19.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.110+0000 7f2aa9ffb640 1 --2- 172.21.15.161:0/3944515513 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2a9806b500 0x7f2a9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:19.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.110+0000 7f2aa9ffb640 1 -- 172.21.15.161:0/3944515513 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5593+0+0 (secure 0 0 0) 0x7f2aa408a180 con 0x7f2ab40c8ed0 2024-03-28T23:33:19.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.110+0000 7f2ab9610640 1 --2- 172.21.15.161:0/3944515513 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2a9806b500 0x7f2a9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:19.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.110+0000 7f2ab9610640 1 --2- 172.21.15.161:0/3944515513 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2a9806b500 0x7f2a9806d9c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0035000 tx=0x7f2ab0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:19.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.114+0000 7f2aa9ffb640 1 -- 172.21.15.161:0/3944515513 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2aa4015020 con 0x7f2ab40c8ed0 2024-03-28T23:33:19.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:19.302+0000 7f2abb874640 1 -- 172.21.15.161:0/3944515513 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f2ab40c18b0 con 0x7f2ab40c8ed0 2024-03-28T23:33:20.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.143+0000 7f2aa9ffb640 1 -- 172.21.15.161:0/3944515513 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v90) ==== 97+0+0 (secure 0 0 0) 0x7f2aa4059060 con 0x7f2ab40c8ed0 2024-03-28T23:33:20.144 INFO:teuthology.orchestra.run.smithi161.stderr: removed all blocklist entries 2024-03-28T23:33:20.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.146+0000 7f2abb874640 1 -- 172.21.15.161:0/3944515513 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2a9806b500 msgr2=0x7f2a9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:20.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.146+0000 7f2abb874640 1 --2- 172.21.15.161:0/3944515513 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2a9806b500 0x7f2a9806d9c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0035000 tx=0x7f2ab0008030 comp rx=0 tx=0).stop 2024-03-28T23:33:20.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.146+0000 7f2abb874640 1 -- 172.21.15.161:0/3944515513 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ab40c8ed0 msgr2=0x7f2ab414ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:20.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.146+0000 7f2abb874640 1 --2- 172.21.15.161:0/3944515513 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ab40c8ed0 0x7f2ab414ddf0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f2aa400c7f0 tx=0x7f2aa400ccb0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.146+0000 7f2abb874640 1 -- 172.21.15.161:0/3944515513 shutdown_connections 2024-03-28T23:33:20.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.146+0000 7f2abb874640 1 --2- 172.21.15.161:0/3944515513 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2a9806b500 0x7f2a9806d9c0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.146+0000 7f2abb874640 1 --2- 172.21.15.161:0/3944515513 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ab40c8ed0 0x7f2ab414ddf0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.146+0000 7f2abb874640 1 --2- 172.21.15.161:0/3944515513 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ab4152890 0x7f2ab40ad6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.146+0000 7f2abb874640 1 --2- 172.21.15.161:0/3944515513 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ab40c5950 0x7f2ab41e2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.146+0000 7f2abb874640 1 -- 172.21.15.161:0/3944515513 >> 172.21.15.161:0/3944515513 conn(0x7f2ab40ac620 msgr2=0x7f2ab414ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:20.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.146+0000 7f2abb874640 1 -- 172.21.15.161:0/3944515513 shutdown_connections 2024-03-28T23:33:20.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.146+0000 7f2abb874640 1 -- 172.21.15.161:0/3944515513 wait complete. 2024-03-28T23:33:20.159 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-28T23:33:20.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.261+0000 7f8c037e5640 1 Processor -- start 2024-03-28T23:33:20.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.261+0000 7f8c037e5640 1 -- start start 2024-03-28T23:33:20.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.261+0000 7f8c037e5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8bfc0a8fc0 0x7f8bfc153150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:20.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.261+0000 7f8c037e5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8bfc0aacb0 0x7f8bfc153690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:20.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.261+0000 7f8c037e5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bfc1566f0 con 0x7f8bfc150590 2024-03-28T23:33:20.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.261+0000 7f8c037e5640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8bfc1569d0 con 0x7f8bfc0a8fc0 2024-03-28T23:33:20.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.261+0000 7f8c037e5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f8bfc156cb0 con 0x7f8bfc0aacb0 2024-03-28T23:33:20.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.262+0000 7f8c00d80640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8bfc0a8fc0 0x7f8bfc153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:20.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.262+0000 7f8c00d80640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8bfc0a8fc0 0x7f8bfc153150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49110/0 (socket says 172.21.15.161:49110) 2024-03-28T23:33:20.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.262+0000 7f8c00d80640 1 -- 172.21.15.161:0/532463938 learned_addr learned my addr 172.21.15.161:0/532463938 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:20.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.262+0000 7f8c01d82640 1 --2- 172.21.15.161:0/532463938 >> v2:172.21.15.170:3300/0 conn(0x7f8bfc0aacb0 0x7f8bfc153690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:20.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.262+0000 7f8c00d80640 1 -- 172.21.15.161:0/532463938 >> v2:172.21.15.170:3300/0 conn(0x7f8bfc0aacb0 msgr2=0x7f8bfc153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:20.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.262+0000 7f8c00d80640 1 --2- 172.21.15.161:0/532463938 >> v2:172.21.15.170:3300/0 conn(0x7f8bfc0aacb0 0x7f8bfc153690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.262+0000 7f8c00d80640 1 -- 172.21.15.161:0/532463938 >> v1:172.21.15.170:6789/0 conn(0x7f8bfc150590 legacy=0x7f8bfc0a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:20.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.262+0000 7f8c00d80640 1 -- 172.21.15.161:0/532463938 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bfc156f90 con 0x7f8bfc0a8fc0 2024-03-28T23:33:20.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.262+0000 7f8c01d82640 1 --2- 172.21.15.161:0/532463938 >> v2:172.21.15.170:3300/0 conn(0x7f8bfc0aacb0 0x7f8bfc153690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:20.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.263+0000 7f8c00d80640 1 --2- 172.21.15.161:0/532463938 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8bfc0a8fc0 0x7f8bfc153150 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc0a98b0 tx=0x7f8bf802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d64aa69624d8e63 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:20.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.263+0000 7f8bf3fff640 1 -- 172.21.15.161:0/532463938 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf8038070 con 0x7f8bfc0a8fc0 2024-03-28T23:33:20.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.263+0000 7f8bf3fff640 1 -- 172.21.15.161:0/532463938 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bf802f050 con 0x7f8bfc0a8fc0 2024-03-28T23:33:20.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.263+0000 7f8bf3fff640 1 -- 172.21.15.161:0/532463938 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf8031970 con 0x7f8bfc0a8fc0 2024-03-28T23:33:20.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.263+0000 7f8c037e5640 1 -- 172.21.15.161:0/532463938 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8bfc0a8fc0 msgr2=0x7f8bfc153150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:20.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.263+0000 7f8c037e5640 1 --2- 172.21.15.161:0/532463938 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8bfc0a8fc0 0x7f8bfc153150 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc0a98b0 tx=0x7f8bf802b7f0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.263+0000 7f8c037e5640 1 -- 172.21.15.161:0/532463938 shutdown_connections 2024-03-28T23:33:20.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.263+0000 7f8c037e5640 1 --2- 172.21.15.161:0/532463938 >> v2:172.21.15.170:3300/0 conn(0x7f8bfc0aacb0 0x7f8bfc153690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.263+0000 7f8c037e5640 1 --2- 172.21.15.161:0/532463938 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8bfc0a8fc0 0x7f8bfc153150 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.263+0000 7f8c037e5640 1 -- 172.21.15.161:0/532463938 >> 172.21.15.161:0/532463938 conn(0x7f8bfc0c6670 msgr2=0x7f8bfc0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:20.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.263+0000 7f8c037e5640 1 -- 172.21.15.161:0/532463938 shutdown_connections 2024-03-28T23:33:20.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.263+0000 7f8c037e5640 1 -- 172.21.15.161:0/532463938 wait complete. 2024-03-28T23:33:20.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.264+0000 7f8c037e5640 1 Processor -- start 2024-03-28T23:33:20.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.265+0000 7f8c037e5640 1 -- start start 2024-03-28T23:33:20.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.265+0000 7f8c037e5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8bfc0a8fc0 0x7f8bfc150d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:20.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.265+0000 7f8c037e5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8bfc0aacb0 0x7f8bfc147d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:20.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.265+0000 7f8c037e5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8bfc150590 0x7f8bfc0c57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:20.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.265+0000 7f8c037e5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8bfc0aa650 con 0x7f8bfc0aacb0 2024-03-28T23:33:20.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.265+0000 7f8c037e5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8bfc150970 con 0x7f8bfc0a8fc0 2024-03-28T23:33:20.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.265+0000 7f8c037e5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8bfc0aa7d0 con 0x7f8bfc150590 2024-03-28T23:33:20.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.265+0000 7f8c00d80640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8bfc0aacb0 0x7f8bfc147d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:20.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.265+0000 7f8c00d80640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8bfc0aacb0 0x7f8bfc147d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49120/0 (socket says 172.21.15.161:49120) 2024-03-28T23:33:20.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.265+0000 7f8c00d80640 1 -- 172.21.15.161:0/2853997691 learned_addr learned my addr 172.21.15.161:0/2853997691 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:20.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.265+0000 7f8c01581640 1 --2- 172.21.15.161:0/2853997691 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8bfc0a8fc0 0x7f8bfc150d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:20.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.265+0000 7f8c01d82640 1 --2- 172.21.15.161:0/2853997691 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8bfc150590 0x7f8bfc0c57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:20.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.266+0000 7f8c00d80640 1 -- 172.21.15.161:0/2853997691 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8bfc150590 msgr2=0x7f8bfc0c57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:20.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.266+0000 7f8c00d80640 1 --2- 172.21.15.161:0/2853997691 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8bfc150590 0x7f8bfc0c57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.266+0000 7f8c00d80640 1 -- 172.21.15.161:0/2853997691 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8bfc0a8fc0 msgr2=0x7f8bfc150d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:20.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.266+0000 7f8c00d80640 1 --2- 172.21.15.161:0/2853997691 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8bfc0a8fc0 0x7f8bfc150d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.266+0000 7f8c00d80640 1 -- 172.21.15.161:0/2853997691 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bfc1556b0 con 0x7f8bfc0aacb0 2024-03-28T23:33:20.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.266+0000 7f8c01d82640 1 --2- 172.21.15.161:0/2853997691 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8bfc150590 0x7f8bfc0c57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:20.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.266+0000 7f8c00d80640 1 --2- 172.21.15.161:0/2853997691 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8bfc0aacb0 0x7f8bfc147d70 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8000c00 tx=0x7f8bf80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:20.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.266+0000 7f8bf1ffb640 1 -- 172.21.15.161:0/2853997691 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf8042070 con 0x7f8bfc0aacb0 2024-03-28T23:33:20.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.266+0000 7f8bf1ffb640 1 -- 172.21.15.161:0/2853997691 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bf803d870 con 0x7f8bfc0aacb0 2024-03-28T23:33:20.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.266+0000 7f8c037e5640 1 -- 172.21.15.161:0/2853997691 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bfc0c49f0 con 0x7f8bfc0aacb0 2024-03-28T23:33:20.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.266+0000 7f8bf1ffb640 1 -- 172.21.15.161:0/2853997691 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf8038040 con 0x7f8bfc0aacb0 2024-03-28T23:33:20.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.266+0000 7f8c037e5640 1 -- 172.21.15.161:0/2853997691 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bfc1482b0 con 0x7f8bfc0aacb0 2024-03-28T23:33:20.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.267+0000 7f8c037e5640 1 -- 172.21.15.161:0/2853997691 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bfc04f4b0 con 0x7f8bfc0aacb0 2024-03-28T23:33:20.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.267+0000 7f8bf1ffb640 1 -- 172.21.15.161:0/2853997691 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8bf8045070 con 0x7f8bfc0aacb0 2024-03-28T23:33:20.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.268+0000 7f8bf1ffb640 1 --2- 172.21.15.161:0/2853997691 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8be006b4d0 0x7f8be006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:20.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.268+0000 7f8bf1ffb640 1 -- 172.21.15.161:0/2853997691 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5249+0+0 (secure 0 0 0) 0x7f8bf80ac030 con 0x7f8bfc0aacb0 2024-03-28T23:33:20.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.268+0000 7f8c01581640 1 --2- 172.21.15.161:0/2853997691 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8be006b4d0 0x7f8be006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:20.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.268+0000 7f8c01581640 1 --2- 172.21.15.161:0/2853997691 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8be006b4d0 0x7f8be006d990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8bec006fd0 tx=0x7f8bec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:20.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.272+0000 7f8bf1ffb640 1 -- 172.21.15.161:0/2853997691 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8bf807b1a0 con 0x7f8bfc0aacb0 2024-03-28T23:33:20.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.508+0000 7f8c037e5640 1 -- 172.21.15.161:0/2853997691 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f8bfc0a9d20 con 0x7f8bfc0aacb0 2024-03-28T23:33:20.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.509+0000 7f8bf1ffb640 1 -- 172.21.15.161:0/2853997691 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) ==== 80+0+4724 (secure 0 0 0) 0x7f8bf807aaa0 con 0x7f8bfc0aacb0 2024-03-28T23:33:20.510 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:20.511 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:33:20.511 INFO:teuthology.orchestra.run.smithi161.stdout: "auth_dump": [ 2024-03-28T23:33:20.511 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:20.511 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.a", 2024-03-28T23:33:20.511 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmggtoJhAASLXbUICLBhjRr19aUAuZHw==", 2024-03-28T23:33:20.511 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:20.511 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:33:20.511 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:33:20.511 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:33:20.511 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:20.511 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:20.511 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:20.511 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:20.512 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.b", 2024-03-28T23:33:20.512 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmQbEOMxAA74al6+dMzcN/ngwNZK+aHA==", 2024-03-28T23:33:20.512 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:20.512 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:33:20.512 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:33:20.512 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:33:20.512 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:20.512 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:20.512 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:20.512 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:20.512 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.c", 2024-03-28T23:33:20.512 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmjf/qLRAAosLbpT8aCWvrKAQnj8Uh3g==", 2024-03-28T23:33:20.512 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:20.512 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:33:20.512 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:33:20.513 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:33:20.513 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:20.513 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:20.513 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:20.513 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:20.513 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.d", 2024-03-28T23:33:20.513 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmpiZ1OhAAjYrFAaDosDS2Y+BvU4c6JQ==", 2024-03-28T23:33:20.513 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:20.513 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:33:20.513 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:33:20.513 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:33:20.513 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:20.513 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:20.513 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:20.513 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:20.513 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.0", 2024-03-28T23:33:20.514 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA9/QVmAFPmGBAAS0SkRRwsupm6jm1sqFXJ6g==", 2024-03-28T23:33:20.514 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:20.514 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:33:20.514 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:33:20.514 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:20.514 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:20.514 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:20.514 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:20.514 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.1", 2024-03-28T23:33:20.514 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBA/QVmNBTVFxAA1htU9ilznfLUDqKmJaQwzg==", 2024-03-28T23:33:20.514 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:20.514 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:33:20.514 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:33:20.514 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.2", 2024-03-28T23:33:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBD/QVmg+4YFhAA9TvfFVKT3nQT7/QUOf3Spw==", 2024-03-28T23:33:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:33:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:33:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.3", 2024-03-28T23:33:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBG/QVmgiHDFBAAQPtbZT6a1ZGmkisBX0NMgA==", 2024-03-28T23:33:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:20.516 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:33:20.516 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:33:20.516 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:20.516 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:20.516 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:20.516 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:20.516 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.4", 2024-03-28T23:33:20.516 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBK/QVmahhZIBAAJfotg/dusbB3usreZlRuPA==", 2024-03-28T23:33:20.516 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:20.516 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:33:20.516 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:33:20.516 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:20.516 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:20.516 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:20.516 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:20.516 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.5", 2024-03-28T23:33:20.516 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBN/QVmzjlDHhAAli7pLyIo+MVdwSBNJYVg5g==", 2024-03-28T23:33:20.517 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:20.517 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:33:20.517 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:33:20.517 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:20.517 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:20.517 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:20.517 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:20.517 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.6", 2024-03-28T23:33:20.517 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBQ/QVmtCFSHBAASVfV/5mY/3JNiI0YMT7Vww==", 2024-03-28T23:33:20.517 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:20.517 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:33:20.517 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:33:20.517 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:20.517 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:20.517 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:20.518 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:20.518 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.7", 2024-03-28T23:33:20.518 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBT/QVm2J7eGhAAaIeZxmLwQBBE1glCxjKbhA==", 2024-03-28T23:33:20.518 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:20.518 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:33:20.518 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:33:20.518 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:20.518 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:20.518 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:20.518 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:20.518 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.0", 2024-03-28T23:33:20.518 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA6/QVmc1krAxAAEKsn1Tuuv9OlFZpB4PFhDg==", 2024-03-28T23:33:20.518 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:20.518 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:33:20.519 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow r", 2024-03-28T23:33:20.519 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-28T23:33:20.519 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:20.519 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:20.519 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:20.519 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.1", 2024-03-28T23:33:20.519 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA6/QVm1B2cBhAAmHDeeFwpFlV/y3NIGWUXHQ==", 2024-03-28T23:33:20.519 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:20.519 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:33:20.519 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow r", 2024-03-28T23:33:20.519 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow rw", 2024-03-28T23:33:20.519 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow rwx" 2024-03-28T23:33:20.519 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:20.519 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:20.519 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:20.519 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.admin", 2024-03-28T23:33:20.520 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmzzJ7CRAAYqfhnIkfNJJ3O+VWC2LP+Q==", 2024-03-28T23:33:20.520 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:20.520 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-28T23:33:20.520 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:33:20.520 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:33:20.520 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:20.520 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:20.521 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:20.521 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:20.521 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mgr.x", 2024-03-28T23:33:20.521 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVm7TD3IhAACByGRGpvJWDpF22NuovHWw==", 2024-03-28T23:33:20.521 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:20.522 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-28T23:33:20.522 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile mgr", 2024-03-28T23:33:20.522 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:20.522 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:20.522 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:20.522 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:20.523 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mgr.y", 2024-03-28T23:33:20.523 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVm0hLlHhAAh8cewPtZ/TbKrU488kxrBg==", 2024-03-28T23:33:20.523 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:20.523 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-28T23:33:20.523 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile mgr", 2024-03-28T23:33:20.523 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:20.523 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:20.523 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:20.523 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-03-28T23:33:20.523 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:33:20.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.511+0000 7f8c037e5640 1 -- 172.21.15.161:0/2853997691 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8be006b4d0 msgr2=0x7f8be006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:20.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.511+0000 7f8c037e5640 1 --2- 172.21.15.161:0/2853997691 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8be006b4d0 0x7f8be006d990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8bec006fd0 tx=0x7f8bec008040 comp rx=0 tx=0).stop 2024-03-28T23:33:20.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.511+0000 7f8c037e5640 1 -- 172.21.15.161:0/2853997691 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8bfc0aacb0 msgr2=0x7f8bfc147d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:20.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.511+0000 7f8c037e5640 1 --2- 172.21.15.161:0/2853997691 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8bfc0aacb0 0x7f8bfc147d70 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8000c00 tx=0x7f8bf80046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.511+0000 7f8c037e5640 1 -- 172.21.15.161:0/2853997691 shutdown_connections 2024-03-28T23:33:20.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.511+0000 7f8c037e5640 1 --2- 172.21.15.161:0/2853997691 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8be006b4d0 0x7f8be006d990 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.511+0000 7f8c037e5640 1 --2- 172.21.15.161:0/2853997691 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8bfc0aacb0 0x7f8bfc147d70 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.511+0000 7f8c037e5640 1 --2- 172.21.15.161:0/2853997691 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8bfc150590 0x7f8bfc0c57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.511+0000 7f8c037e5640 1 --2- 172.21.15.161:0/2853997691 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8bfc0a8fc0 0x7f8bfc150d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.511+0000 7f8c037e5640 1 -- 172.21.15.161:0/2853997691 >> 172.21.15.161:0/2853997691 conn(0x7f8bfc0c6670 msgr2=0x7f8bfc0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:20.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.511+0000 7f8c037e5640 1 -- 172.21.15.161:0/2853997691 shutdown_connections 2024-03-28T23:33:20.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.511+0000 7f8c037e5640 1 -- 172.21.15.161:0/2853997691 wait complete. 2024-03-28T23:33:20.526 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:33:20.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.627+0000 7f1576177640 1 Processor -- start 2024-03-28T23:33:20.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.627+0000 7f1576177640 1 -- start start 2024-03-28T23:33:20.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.628+0000 7f1576177640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15700a8fc0 0x7f1570153270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:20.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.628+0000 7f1576177640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f15700aacb0 0x7f15701537b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:20.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.628+0000 7f1576177640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15701567c0 con 0x7f1570150600 2024-03-28T23:33:20.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.628+0000 7f1576177640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1570156570 con 0x7f15700aacb0 2024-03-28T23:33:20.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.628+0000 7f1576177640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1570156aa0 con 0x7f15700a8fc0 2024-03-28T23:33:20.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.628+0000 7f156ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f15700aacb0 0x7f15701537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:20.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.628+0000 7f156ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f15700aacb0 0x7f15701537b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40946/0 (socket says 172.21.15.161:40946) 2024-03-28T23:33:20.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.628+0000 7f156ffff640 1 -- 172.21.15.161:0/1420672845 learned_addr learned my addr 172.21.15.161:0/1420672845 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:20.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.628+0000 7f156effd640 1 --2- 172.21.15.161:0/1420672845 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15700a8fc0 0x7f1570153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:20.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.628+0000 7f156ffff640 1 -- 172.21.15.161:0/1420672845 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15700a8fc0 msgr2=0x7f1570153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:20.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.628+0000 7f156ffff640 1 --2- 172.21.15.161:0/1420672845 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15700a8fc0 0x7f1570153270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.628+0000 7f156ffff640 1 -- 172.21.15.161:0/1420672845 >> v1:172.21.15.170:6789/0 conn(0x7f1570150600 legacy=0x7f15700a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:20.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.628+0000 7f156ffff640 1 -- 172.21.15.161:0/1420672845 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1570156d80 con 0x7f15700aacb0 2024-03-28T23:33:20.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.628+0000 7f156effd640 1 --2- 172.21.15.161:0/1420672845 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15700a8fc0 0x7f1570153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:20.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.629+0000 7f156ffff640 1 --2- 172.21.15.161:0/1420672845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15700aacb0 0x7f15701537b0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f1568009e50 tx=0x7f156802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b629ecfc9c667e88 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:20.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.629+0000 7f156e7fc640 1 -- 172.21.15.161:0/1420672845 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f156802c9e0 con 0x7f15700aacb0 2024-03-28T23:33:20.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.629+0000 7f156e7fc640 1 -- 172.21.15.161:0/1420672845 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f156802cb80 con 0x7f15700aacb0 2024-03-28T23:33:20.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.629+0000 7f156e7fc640 1 -- 172.21.15.161:0/1420672845 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15680343f0 con 0x7f15700aacb0 2024-03-28T23:33:20.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.630+0000 7f1576177640 1 -- 172.21.15.161:0/1420672845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15700aacb0 msgr2=0x7f15701537b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:20.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.630+0000 7f1576177640 1 --2- 172.21.15.161:0/1420672845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15700aacb0 0x7f15701537b0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f1568009e50 tx=0x7f156802be40 comp rx=0 tx=0).stop 2024-03-28T23:33:20.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.630+0000 7f1576177640 1 -- 172.21.15.161:0/1420672845 shutdown_connections 2024-03-28T23:33:20.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.630+0000 7f1576177640 1 --2- 172.21.15.161:0/1420672845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15700aacb0 0x7f15701537b0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.630+0000 7f1576177640 1 --2- 172.21.15.161:0/1420672845 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15700a8fc0 0x7f1570153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.630+0000 7f1576177640 1 -- 172.21.15.161:0/1420672845 >> 172.21.15.161:0/1420672845 conn(0x7f15700c6670 msgr2=0x7f15700c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:20.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.630+0000 7f1576177640 1 -- 172.21.15.161:0/1420672845 shutdown_connections 2024-03-28T23:33:20.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.630+0000 7f1576177640 1 -- 172.21.15.161:0/1420672845 wait complete. 2024-03-28T23:33:20.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.631+0000 7f1576177640 1 Processor -- start 2024-03-28T23:33:20.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.631+0000 7f1576177640 1 -- start start 2024-03-28T23:33:20.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.632+0000 7f1576177640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15700a8fc0 0x7f1570149020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:20.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.632+0000 7f1576177640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15700aacb0 0x7f15700c5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:20.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.632+0000 7f1576177640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1570150600 0x7f15700c58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:20.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.632+0000 7f1576177640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f15701509e0 con 0x7f15700a8fc0 2024-03-28T23:33:20.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.632+0000 7f1576177640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1570157340 con 0x7f1570150600 2024-03-28T23:33:20.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.632+0000 7f1576177640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f15700aa9e0 con 0x7f15700aacb0 2024-03-28T23:33:20.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.632+0000 7f156f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15700a8fc0 0x7f1570149020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:20.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.632+0000 7f156f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15700a8fc0 0x7f1570149020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40948/0 (socket says 172.21.15.161:40948) 2024-03-28T23:33:20.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.632+0000 7f156f7fe640 1 -- 172.21.15.161:0/2628318720 learned_addr learned my addr 172.21.15.161:0/2628318720 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:20.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.632+0000 7f156ffff640 1 --2- 172.21.15.161:0/2628318720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1570150600 0x7f15700c58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:20.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.632+0000 7f156effd640 1 --2- 172.21.15.161:0/2628318720 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15700aacb0 0x7f15700c5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:20.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.632+0000 7f156f7fe640 1 -- 172.21.15.161:0/2628318720 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15700aacb0 msgr2=0x7f15700c5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:20.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.632+0000 7f156f7fe640 1 --2- 172.21.15.161:0/2628318720 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15700aacb0 0x7f15700c5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.632+0000 7f156f7fe640 1 -- 172.21.15.161:0/2628318720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1570150600 msgr2=0x7f15700c58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:20.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.632+0000 7f156f7fe640 1 --2- 172.21.15.161:0/2628318720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1570150600 0x7f15700c58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.632+0000 7f156f7fe640 1 -- 172.21.15.161:0/2628318720 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15700c0ef0 con 0x7f15700a8fc0 2024-03-28T23:33:20.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.632+0000 7f156ffff640 1 --2- 172.21.15.161:0/2628318720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1570150600 0x7f15700c58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:20.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.632+0000 7f156effd640 1 --2- 172.21.15.161:0/2628318720 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15700aacb0 0x7f15700c5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:20.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.632+0000 7f156f7fe640 1 --2- 172.21.15.161:0/2628318720 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15700a8fc0 0x7f1570149020 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f156000b420 tx=0x7f156000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:20.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.632+0000 7f1557fff640 1 -- 172.21.15.161:0/2628318720 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1560011830 con 0x7f15700a8fc0 2024-03-28T23:33:20.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.632+0000 7f1557fff640 1 -- 172.21.15.161:0/2628318720 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1560016070 con 0x7f15700a8fc0 2024-03-28T23:33:20.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.632+0000 7f1576177640 1 -- 172.21.15.161:0/2628318720 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15700c3fb0 con 0x7f15700a8fc0 2024-03-28T23:33:20.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.632+0000 7f1557fff640 1 -- 172.21.15.161:0/2628318720 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1560011c10 con 0x7f15700a8fc0 2024-03-28T23:33:20.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.632+0000 7f1576177640 1 -- 172.21.15.161:0/2628318720 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15700c4d40 con 0x7f15700a8fc0 2024-03-28T23:33:20.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.633+0000 7f1576177640 1 -- 172.21.15.161:0/2628318720 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f157004f4b0 con 0x7f15700a8fc0 2024-03-28T23:33:20.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.634+0000 7f1557fff640 1 -- 172.21.15.161:0/2628318720 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f156001b460 con 0x7f15700a8fc0 2024-03-28T23:33:20.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.634+0000 7f1557fff640 1 --2- 172.21.15.161:0/2628318720 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f155c06b500 0x7f155c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:20.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.634+0000 7f1557fff640 1 -- 172.21.15.161:0/2628318720 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5249+0+0 (secure 0 0 0) 0x7f1560089d30 con 0x7f15700a8fc0 2024-03-28T23:33:20.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.634+0000 7f156effd640 1 --2- 172.21.15.161:0/2628318720 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f155c06b500 0x7f155c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:20.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.635+0000 7f156effd640 1 --2- 172.21.15.161:0/2628318720 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f155c06b500 0x7f155c06d9c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1570152f60 tx=0x7f1564008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:20.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.639+0000 7f1557fff640 1 -- 172.21.15.161:0/2628318720 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1560058ce0 con 0x7f15700a8fc0 2024-03-28T23:33:20.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.810+0000 7f1576177640 1 -- 172.21.15.161:0/2628318720 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f15700c10c0 con 0x7f15700a8fc0 2024-03-28T23:33:20.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.811+0000 7f1557fff640 1 -- 172.21.15.161:0/2628318720 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+13201 (secure 0 0 0) 0x7f15600585e0 con 0x7f15700a8fc0 2024-03-28T23:33:20.812 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:20.812 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":90,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:33:20.139958+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:33:20.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.813+0000 7f1576177640 1 -- 172.21.15.161:0/2628318720 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f155c06b500 msgr2=0x7f155c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:20.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.813+0000 7f1576177640 1 --2- 172.21.15.161:0/2628318720 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f155c06b500 0x7f155c06d9c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1570152f60 tx=0x7f1564008040 comp rx=0 tx=0).stop 2024-03-28T23:33:20.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.813+0000 7f1576177640 1 -- 172.21.15.161:0/2628318720 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15700a8fc0 msgr2=0x7f1570149020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:20.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.813+0000 7f1576177640 1 --2- 172.21.15.161:0/2628318720 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15700a8fc0 0x7f1570149020 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f156000b420 tx=0x7f156000b8e0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.813+0000 7f1576177640 1 -- 172.21.15.161:0/2628318720 shutdown_connections 2024-03-28T23:33:20.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.813+0000 7f1576177640 1 --2- 172.21.15.161:0/2628318720 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f155c06b500 0x7f155c06d9c0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.813+0000 7f1576177640 1 --2- 172.21.15.161:0/2628318720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1570150600 0x7f15700c58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.813+0000 7f1576177640 1 --2- 172.21.15.161:0/2628318720 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15700aacb0 0x7f15700c5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.813+0000 7f1576177640 1 --2- 172.21.15.161:0/2628318720 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15700a8fc0 0x7f1570149020 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.813+0000 7f1576177640 1 -- 172.21.15.161:0/2628318720 >> 172.21.15.161:0/2628318720 conn(0x7f15700c6670 msgr2=0x7f157014de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:20.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.813+0000 7f1576177640 1 -- 172.21.15.161:0/2628318720 shutdown_connections 2024-03-28T23:33:20.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.813+0000 7f1576177640 1 -- 172.21.15.161:0/2628318720 wait complete. 2024-03-28T23:33:20.825 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:33:20.826 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:33:20.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.929+0000 7efe7e2ae640 1 Processor -- start 2024-03-28T23:33:20.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.929+0000 7efe7e2ae640 1 -- start start 2024-03-28T23:33:20.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.930+0000 7efe7e2ae640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efe780a9bb0 0x7efe78153860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:20.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.930+0000 7efe7e2ae640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7efe780c1290 0x7efe78149660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:20.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.930+0000 7efe7e2ae640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe780abb60 con 0x7efe78149ba0 2024-03-28T23:33:20.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.930+0000 7efe7e2ae640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7efe780ac770 con 0x7efe780a9bb0 2024-03-28T23:33:20.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.930+0000 7efe7e2ae640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7efe780abe40 con 0x7efe780c1290 2024-03-28T23:33:20.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.930+0000 7efe77fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efe780a9bb0 0x7efe78153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:20.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.930+0000 7efe77fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efe780a9bb0 0x7efe78153860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40954/0 (socket says 172.21.15.161:40954) 2024-03-28T23:33:20.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.930+0000 7efe77fff640 1 -- 172.21.15.161:0/3313431842 learned_addr learned my addr 172.21.15.161:0/3313431842 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:20.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.930+0000 7efe777fe640 1 --2- 172.21.15.161:0/3313431842 >> v2:172.21.15.170:3300/0 conn(0x7efe780c1290 0x7efe78149660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:20.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.930+0000 7efe77fff640 1 -- 172.21.15.161:0/3313431842 >> v2:172.21.15.170:3300/0 conn(0x7efe780c1290 msgr2=0x7efe78149660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:20.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.930+0000 7efe77fff640 1 --2- 172.21.15.161:0/3313431842 >> v2:172.21.15.170:3300/0 conn(0x7efe780c1290 0x7efe78149660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.930+0000 7efe77fff640 1 -- 172.21.15.161:0/3313431842 >> v1:172.21.15.170:6789/0 conn(0x7efe78149ba0 legacy=0x7efe780c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:20.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.930+0000 7efe77fff640 1 -- 172.21.15.161:0/3313431842 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe780ac120 con 0x7efe780a9bb0 2024-03-28T23:33:20.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.931+0000 7efe77fff640 1 --2- 172.21.15.161:0/3313431842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efe780a9bb0 0x7efe78153860 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7efe7806fc60 tx=0x7efe6802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af751ec6a69d3a0b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:20.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.931+0000 7efe76ffd640 1 -- 172.21.15.161:0/3313431842 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe68035020 con 0x7efe780a9bb0 2024-03-28T23:33:20.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.931+0000 7efe76ffd640 1 -- 172.21.15.161:0/3313431842 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe68031870 con 0x7efe780a9bb0 2024-03-28T23:33:20.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.931+0000 7efe76ffd640 1 -- 172.21.15.161:0/3313431842 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe68031c40 con 0x7efe780a9bb0 2024-03-28T23:33:20.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.932+0000 7efe7e2ae640 1 -- 172.21.15.161:0/3313431842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efe780a9bb0 msgr2=0x7efe78153860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:20.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.932+0000 7efe7e2ae640 1 --2- 172.21.15.161:0/3313431842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efe780a9bb0 0x7efe78153860 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7efe7806fc60 tx=0x7efe6802b910 comp rx=0 tx=0).stop 2024-03-28T23:33:20.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.932+0000 7efe7e2ae640 1 -- 172.21.15.161:0/3313431842 shutdown_connections 2024-03-28T23:33:20.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.932+0000 7efe7e2ae640 1 --2- 172.21.15.161:0/3313431842 >> v2:172.21.15.170:3300/0 conn(0x7efe780c1290 0x7efe78149660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.932+0000 7efe7e2ae640 1 --2- 172.21.15.161:0/3313431842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efe780a9bb0 0x7efe78153860 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.932+0000 7efe7e2ae640 1 -- 172.21.15.161:0/3313431842 >> 172.21.15.161:0/3313431842 conn(0x7efe780c3390 msgr2=0x7efe780c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:20.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.932+0000 7efe7e2ae640 1 -- 172.21.15.161:0/3313431842 shutdown_connections 2024-03-28T23:33:20.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.932+0000 7efe7e2ae640 1 -- 172.21.15.161:0/3313431842 wait complete. 2024-03-28T23:33:20.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.933+0000 7efe7e2ae640 1 Processor -- start 2024-03-28T23:33:20.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.933+0000 7efe7e2ae640 1 -- start start 2024-03-28T23:33:20.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.934+0000 7efe7e2ae640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe780a9bb0 0x7efe780bff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:20.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.934+0000 7efe7e2ae640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efe780c1290 0x7efe7814fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:20.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.934+0000 7efe7e2ae640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe78149ba0 0x7efe781502d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:20.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.934+0000 7efe7e2ae640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efe780a9f90 con 0x7efe780c1290 2024-03-28T23:33:20.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.934+0000 7efe7e2ae640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7efe78153da0 con 0x7efe78149ba0 2024-03-28T23:33:20.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.934+0000 7efe7e2ae640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7efe780aa110 con 0x7efe780a9bb0 2024-03-28T23:33:20.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.934+0000 7efe777fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efe780c1290 0x7efe7814fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:20.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.934+0000 7efe777fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efe780c1290 0x7efe7814fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40970/0 (socket says 172.21.15.161:40970) 2024-03-28T23:33:20.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.934+0000 7efe777fe640 1 -- 172.21.15.161:0/2077084669 learned_addr learned my addr 172.21.15.161:0/2077084669 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:20.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.934+0000 7efe77fff640 1 --2- 172.21.15.161:0/2077084669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe780a9bb0 0x7efe780bff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:20.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.934+0000 7efe7c84b640 1 --2- 172.21.15.161:0/2077084669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe78149ba0 0x7efe781502d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:20.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.934+0000 7efe777fe640 1 -- 172.21.15.161:0/2077084669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe780a9bb0 msgr2=0x7efe780bff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:20.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.934+0000 7efe777fe640 1 --2- 172.21.15.161:0/2077084669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe780a9bb0 0x7efe780bff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.934+0000 7efe777fe640 1 -- 172.21.15.161:0/2077084669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe78149ba0 msgr2=0x7efe781502d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:20.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.934+0000 7efe777fe640 1 --2- 172.21.15.161:0/2077084669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe78149ba0 0x7efe781502d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:20.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.934+0000 7efe777fe640 1 -- 172.21.15.161:0/2077084669 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe781eb840 con 0x7efe780c1290 2024-03-28T23:33:20.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.934+0000 7efe77fff640 1 --2- 172.21.15.161:0/2077084669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe780a9bb0 0x7efe780bff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:20.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.934+0000 7efe777fe640 1 --2- 172.21.15.161:0/2077084669 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efe780c1290 0x7efe7814fd90 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7efe6c00e9b0 tx=0x7efe6c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:20.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.934+0000 7efe74ff9640 1 -- 172.21.15.161:0/2077084669 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe6c017070 con 0x7efe780c1290 2024-03-28T23:33:20.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.934+0000 7efe74ff9640 1 -- 172.21.15.161:0/2077084669 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe6c00c9a0 con 0x7efe780c1290 2024-03-28T23:33:20.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.934+0000 7efe74ff9640 1 -- 172.21.15.161:0/2077084669 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe6c00cd90 con 0x7efe780c1290 2024-03-28T23:33:20.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.934+0000 7efe7e2ae640 1 -- 172.21.15.161:0/2077084669 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe781ebb20 con 0x7efe780c1290 2024-03-28T23:33:20.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.934+0000 7efe7e2ae640 1 -- 172.21.15.161:0/2077084669 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe7814a3d0 con 0x7efe780c1290 2024-03-28T23:33:20.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.935+0000 7efe7e2ae640 1 -- 172.21.15.161:0/2077084669 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe78069530 con 0x7efe780c1290 2024-03-28T23:33:20.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.936+0000 7efe74ff9640 1 -- 172.21.15.161:0/2077084669 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efe6c01a070 con 0x7efe780c1290 2024-03-28T23:33:20.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.936+0000 7efe74ff9640 1 --2- 172.21.15.161:0/2077084669 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efe6406b4d0 0x7efe6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:20.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.936+0000 7efe74ff9640 1 -- 172.21.15.161:0/2077084669 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5249+0+0 (secure 0 0 0) 0x7efe6c08a290 con 0x7efe780c1290 2024-03-28T23:33:20.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.936+0000 7efe77fff640 1 --2- 172.21.15.161:0/2077084669 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efe6406b4d0 0x7efe6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:20.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.937+0000 7efe77fff640 1 --2- 172.21.15.161:0/2077084669 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efe6406b4d0 0x7efe6406d990 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7efe68036000 tx=0x7efe6803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:20.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:20.941+0000 7efe74ff9640 1 -- 172.21.15.161:0/2077084669 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efe78069530 con 0x7efe780c1290 2024-03-28T23:33:21.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.111+0000 7efe7e2ae640 1 -- 172.21.15.161:0/2077084669 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efe7804f4b0 con 0x7efe780c1290 2024-03-28T23:33:21.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.112+0000 7efe74ff9640 1 -- 172.21.15.161:0/2077084669 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7efe6c058d10 con 0x7efe780c1290 2024-03-28T23:33:21.112 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:33:21.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.113+0000 7efe7e2ae640 1 -- 172.21.15.161:0/2077084669 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efe6406b4d0 msgr2=0x7efe6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:21.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.113+0000 7efe7e2ae640 1 --2- 172.21.15.161:0/2077084669 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efe6406b4d0 0x7efe6406d990 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7efe68036000 tx=0x7efe6803c040 comp rx=0 tx=0).stop 2024-03-28T23:33:21.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.114+0000 7efe7e2ae640 1 -- 172.21.15.161:0/2077084669 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efe780c1290 msgr2=0x7efe7814fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:21.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.114+0000 7efe7e2ae640 1 --2- 172.21.15.161:0/2077084669 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efe780c1290 0x7efe7814fd90 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7efe6c00e9b0 tx=0x7efe6c00ee70 comp rx=0 tx=0).stop 2024-03-28T23:33:21.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.114+0000 7efe7e2ae640 1 -- 172.21.15.161:0/2077084669 shutdown_connections 2024-03-28T23:33:21.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.114+0000 7efe7e2ae640 1 --2- 172.21.15.161:0/2077084669 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efe6406b4d0 0x7efe6406d990 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:21.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.114+0000 7efe7e2ae640 1 --2- 172.21.15.161:0/2077084669 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efe780c1290 0x7efe7814fd90 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:21.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.114+0000 7efe7e2ae640 1 --2- 172.21.15.161:0/2077084669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe78149ba0 0x7efe781502d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:21.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.114+0000 7efe7e2ae640 1 --2- 172.21.15.161:0/2077084669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe780a9bb0 0x7efe780bff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:21.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.114+0000 7efe7e2ae640 1 -- 172.21.15.161:0/2077084669 >> 172.21.15.161:0/2077084669 conn(0x7efe780c3390 msgr2=0x7efe780c25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:21.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.114+0000 7efe7e2ae640 1 -- 172.21.15.161:0/2077084669 shutdown_connections 2024-03-28T23:33:21.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.114+0000 7efe7e2ae640 1 -- 172.21.15.161:0/2077084669 wait complete. 2024-03-28T23:33:21.126 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T23:33:21.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.227+0000 7f5fac3e9640 1 Processor -- start 2024-03-28T23:33:21.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.227+0000 7f5fac3e9640 1 -- start start 2024-03-28T23:33:21.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.228+0000 7f5fac3e9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5fa4151f80 0x7f5fa40c21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:21.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.228+0000 7f5fac3e9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fa4152940 con 0x7f5fa40ac310 2024-03-28T23:33:21.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.228+0000 7f5fac3e9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fa4152c20 con 0x7f5fa414f730 2024-03-28T23:33:21.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.228+0000 7f5fac3e9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f5fa4152f00 con 0x7f5fa4151f80 2024-03-28T23:33:21.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.228+0000 7f5fa9984640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f5fa40ac310 0x7f5fa4155e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:55312/0 (socket says 172.21.15.161:55312) 2024-03-28T23:33:21.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.228+0000 7f5fa9984640 1 -- 172.21.15.161:0/3774443628 learned_addr learned my addr 172.21.15.161:0/3774443628 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:21.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.229+0000 7f5faa986640 1 --2- 172.21.15.161:0/3774443628 >> v2:172.21.15.170:3300/0 conn(0x7f5fa4151f80 0x7f5fa40c21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:21.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.229+0000 7f5fa9183640 1 -- 172.21.15.161:0/3774443628 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1872996380 0 0) 0x7f5fa4152940 con 0x7f5fa40ac310 2024-03-28T23:33:21.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.229+0000 7f5fa9183640 1 -- 172.21.15.161:0/3774443628 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f90003520 con 0x7f5fa40ac310 2024-03-28T23:33:21.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.229+0000 7f5fa9183640 1 -- 172.21.15.161:0/3774443628 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 369890022 0 0) 0x7f5f90003520 con 0x7f5fa40ac310 2024-03-28T23:33:21.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.229+0000 7f5faa986640 1 -- 172.21.15.161:0/3774443628 >> v1:172.21.15.170:6789/0 conn(0x7f5fa414f730 legacy=0x7f5fa40aad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:21.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.229+0000 7f5faa986640 1 -- 172.21.15.161:0/3774443628 >> v1:172.21.15.161:6789/0 conn(0x7f5fa40ac310 legacy=0x7f5fa4155e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:21.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.229+0000 7f5faa986640 1 -- 172.21.15.161:0/3774443628 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa4155620 con 0x7f5fa4151f80 2024-03-28T23:33:21.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.230+0000 7f5faa986640 1 --2- 172.21.15.161:0/3774443628 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5fa4151f80 0x7f5fa40c21e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c009d90 tx=0x7f5f9c02bbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9bf647e01c34853 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:21.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.230+0000 7f5fa9183640 1 -- 172.21.15.161:0/3774443628 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f94001670 con 0x7f5fa4151f80 2024-03-28T23:33:21.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.230+0000 7f5fa9183640 1 -- 172.21.15.161:0/3774443628 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f9c02d720 con 0x7f5fa4151f80 2024-03-28T23:33:21.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.230+0000 7f5fa9183640 1 -- 172.21.15.161:0/3774443628 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c02daf0 con 0x7f5fa4151f80 2024-03-28T23:33:21.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.230+0000 7f5fac3e9640 1 -- 172.21.15.161:0/3774443628 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5fa4151f80 msgr2=0x7f5fa40c21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:21.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.230+0000 7f5fac3e9640 1 --2- 172.21.15.161:0/3774443628 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5fa4151f80 0x7f5fa40c21e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c009d90 tx=0x7f5f9c02bbe0 comp rx=0 tx=0).stop 2024-03-28T23:33:21.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.231+0000 7f5fac3e9640 1 -- 172.21.15.161:0/3774443628 shutdown_connections 2024-03-28T23:33:21.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.231+0000 7f5fac3e9640 1 --2- 172.21.15.161:0/3774443628 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5fa4151f80 0x7f5fa40c21e0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:21.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.231+0000 7f5fac3e9640 1 -- 172.21.15.161:0/3774443628 >> 172.21.15.161:0/3774443628 conn(0x7f5fa40c66d0 msgr2=0x7f5fa40c6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:21.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.231+0000 7f5fac3e9640 1 -- 172.21.15.161:0/3774443628 shutdown_connections 2024-03-28T23:33:21.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.231+0000 7f5fac3e9640 1 -- 172.21.15.161:0/3774443628 wait complete. 2024-03-28T23:33:21.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.232+0000 7f5fac3e9640 1 Processor -- start 2024-03-28T23:33:21.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.232+0000 7f5fac3e9640 1 -- start start 2024-03-28T23:33:21.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.232+0000 7f5fac3e9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5fa40ac310 0x7f5fa41eb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:21.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.232+0000 7f5fac3e9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5fa414f730 0x7f5fa40c56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:21.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.232+0000 7f5fac3e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5fa4151f80 0x7f5fa4072400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:21.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.232+0000 7f5fac3e9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa4152430 con 0x7f5fa40ac310 2024-03-28T23:33:21.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.232+0000 7f5fac3e9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa4155990 con 0x7f5fa414f730 2024-03-28T23:33:21.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.232+0000 7f5fac3e9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5fa41525b0 con 0x7f5fa4151f80 2024-03-28T23:33:21.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.232+0000 7f5faa185640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5fa40ac310 0x7f5fa41eb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:21.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.232+0000 7f5faa185640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5fa40ac310 0x7f5fa41eb520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40986/0 (socket says 172.21.15.161:40986) 2024-03-28T23:33:21.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.232+0000 7f5faa185640 1 -- 172.21.15.161:0/2953035202 learned_addr learned my addr 172.21.15.161:0/2953035202 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:21.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.233+0000 7f5fa9984640 1 --2- 172.21.15.161:0/2953035202 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5fa414f730 0x7f5fa40c56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:21.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.233+0000 7f5faa986640 1 --2- 172.21.15.161:0/2953035202 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5fa4151f80 0x7f5fa4072400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:21.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.233+0000 7f5faa185640 1 -- 172.21.15.161:0/2953035202 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5fa4151f80 msgr2=0x7f5fa4072400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:21.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.233+0000 7f5faa185640 1 --2- 172.21.15.161:0/2953035202 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5fa4151f80 0x7f5fa4072400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:21.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.233+0000 7f5faa185640 1 -- 172.21.15.161:0/2953035202 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5fa414f730 msgr2=0x7f5fa40c56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:21.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.233+0000 7f5faa185640 1 --2- 172.21.15.161:0/2953035202 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5fa414f730 0x7f5fa40c56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:21.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.233+0000 7f5faa185640 1 -- 172.21.15.161:0/2953035202 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa4073200 con 0x7f5fa40ac310 2024-03-28T23:33:21.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.233+0000 7f5faa185640 1 --2- 172.21.15.161:0/2953035202 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5fa40ac310 0x7f5fa41eb520 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f5fa000c780 tx=0x7f5fa000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:21.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.233+0000 7f5f9affd640 1 -- 172.21.15.161:0/2953035202 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0017070 con 0x7f5fa40ac310 2024-03-28T23:33:21.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.233+0000 7f5f9affd640 1 -- 172.21.15.161:0/2953035202 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa0005290 con 0x7f5fa40ac310 2024-03-28T23:33:21.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.233+0000 7f5fac3e9640 1 -- 172.21.15.161:0/2953035202 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fa40734c0 con 0x7f5fa40ac310 2024-03-28T23:33:21.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.233+0000 7f5f9affd640 1 -- 172.21.15.161:0/2953035202 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa000a880 con 0x7f5fa40ac310 2024-03-28T23:33:21.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.233+0000 7f5fac3e9640 1 -- 172.21.15.161:0/2953035202 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fa40c03e0 con 0x7f5fa40ac310 2024-03-28T23:33:21.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.234+0000 7f5fac3e9640 1 -- 172.21.15.161:0/2953035202 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fa404f550 con 0x7f5fa40ac310 2024-03-28T23:33:21.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.234+0000 7f5f9affd640 1 -- 172.21.15.161:0/2953035202 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5fa001a070 con 0x7f5fa40ac310 2024-03-28T23:33:21.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.235+0000 7f5f9affd640 1 --2- 172.21.15.161:0/2953035202 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5f8806b4d0 0x7f5f8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:21.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.235+0000 7f5f9affd640 1 -- 172.21.15.161:0/2953035202 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5249+0+0 (secure 0 0 0) 0x7f5fa0089430 con 0x7f5fa40ac310 2024-03-28T23:33:21.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.235+0000 7f5fa9984640 1 --2- 172.21.15.161:0/2953035202 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5f8806b4d0 0x7f5f8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:21.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.235+0000 7f5fa9984640 1 --2- 172.21.15.161:0/2953035202 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5f8806b4d0 0x7f5f8806d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5fa40c6520 tx=0x7f5f940057f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:21.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.239+0000 7f5f9affd640 1 -- 172.21.15.161:0/2953035202 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5fa0057eb0 con 0x7f5fa40ac310 2024-03-28T23:33:21.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.412+0000 7f5fac3e9640 1 -- 172.21.15.161:0/2953035202 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f5fa406e910 con 0x7f5fa40ac310 2024-03-28T23:33:21.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.412+0000 7f5f9affd640 1 -- 172.21.15.161:0/2953035202 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v90) ==== 84+0+65 (secure 0 0 0) 0x7f5fa406e910 con 0x7f5fa40ac310 2024-03-28T23:33:21.413 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:21.413 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2024-03-28T23:33:21.413 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:21.413 INFO:teuthology.orchestra.run.smithi161.stdout: "poolnum": 1, 2024-03-28T23:33:21.414 INFO:teuthology.orchestra.run.smithi161.stdout: "poolname": ".mgr" 2024-03-28T23:33:21.414 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:21.414 INFO:teuthology.orchestra.run.smithi161.stdout:] 2024-03-28T23:33:21.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.414+0000 7f5fac3e9640 1 -- 172.21.15.161:0/2953035202 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5f8806b4d0 msgr2=0x7f5f8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:21.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.414+0000 7f5fac3e9640 1 --2- 172.21.15.161:0/2953035202 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5f8806b4d0 0x7f5f8806d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5fa40c6520 tx=0x7f5f940057f0 comp rx=0 tx=0).stop 2024-03-28T23:33:21.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.414+0000 7f5fac3e9640 1 -- 172.21.15.161:0/2953035202 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5fa40ac310 msgr2=0x7f5fa41eb520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:21.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.414+0000 7f5fac3e9640 1 --2- 172.21.15.161:0/2953035202 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5fa40ac310 0x7f5fa41eb520 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f5fa000c780 tx=0x7f5fa000cc40 comp rx=0 tx=0).stop 2024-03-28T23:33:21.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.414+0000 7f5fac3e9640 1 -- 172.21.15.161:0/2953035202 shutdown_connections 2024-03-28T23:33:21.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.414+0000 7f5fac3e9640 1 --2- 172.21.15.161:0/2953035202 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5f8806b4d0 0x7f5f8806d990 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:21.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.414+0000 7f5fac3e9640 1 --2- 172.21.15.161:0/2953035202 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5fa4151f80 0x7f5fa4072400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:21.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.414+0000 7f5fac3e9640 1 --2- 172.21.15.161:0/2953035202 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5fa414f730 0x7f5fa40c56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:21.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.414+0000 7f5fac3e9640 1 --2- 172.21.15.161:0/2953035202 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5fa40ac310 0x7f5fa41eb520 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:21.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.414+0000 7f5fac3e9640 1 -- 172.21.15.161:0/2953035202 >> 172.21.15.161:0/2953035202 conn(0x7f5fa40c66d0 msgr2=0x7f5fa414eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:21.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.415+0000 7f5fac3e9640 1 -- 172.21.15.161:0/2953035202 shutdown_connections 2024-03-28T23:33:21.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.415+0000 7f5fac3e9640 1 -- 172.21.15.161:0/2953035202 wait complete. 2024-03-28T23:33:21.426 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-28T23:33:21.426 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-28T23:33:21.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.530+0000 7f43fab3e640 1 Processor -- start 2024-03-28T23:33:21.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.530+0000 7f43fab3e640 1 -- start start 2024-03-28T23:33:21.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.531+0000 7f43fab3e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f43f40ac430 0x7f43f4152880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:21.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.531+0000 7f43fab3e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f43f40c30c0 0x7f43f40c24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:21.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.531+0000 7f43fab3e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43f40c2a10 0x7f43f40c19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:21.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.531+0000 7f43fab3e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f43f4152dc0 con 0x7f43f40c30c0 2024-03-28T23:33:21.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.531+0000 7f43fab3e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f43f40c3540 con 0x7f43f40ac430 2024-03-28T23:33:21.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.531+0000 7f43fab3e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f43f40c3820 con 0x7f43f40c2a10 2024-03-28T23:33:21.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.531+0000 7f43f3fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f43f40c30c0 0x7f43f40c24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:21.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.531+0000 7f43f3fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f43f40c30c0 0x7f43f40c24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40994/0 (socket says 172.21.15.161:40994) 2024-03-28T23:33:21.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.531+0000 7f43f3fff640 1 -- 172.21.15.161:0/2089884103 learned_addr learned my addr 172.21.15.161:0/2089884103 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:21.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.531+0000 7f43f88da640 1 --2- 172.21.15.161:0/2089884103 >> v2:172.21.15.170:3300/0 conn(0x7f43f40ac430 0x7f43f4152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:21.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.531+0000 7f43f90db640 1 --2- 172.21.15.161:0/2089884103 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43f40c2a10 0x7f43f40c19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:21.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.531+0000 7f43f3fff640 1 -- 172.21.15.161:0/2089884103 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43f40c2a10 msgr2=0x7f43f40c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:21.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.531+0000 7f43f3fff640 1 --2- 172.21.15.161:0/2089884103 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43f40c2a10 0x7f43f40c19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:21.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.531+0000 7f43f3fff640 1 -- 172.21.15.161:0/2089884103 >> v2:172.21.15.170:3300/0 conn(0x7f43f40ac430 msgr2=0x7f43f4152880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:21.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.531+0000 7f43f3fff640 1 --2- 172.21.15.161:0/2089884103 >> v2:172.21.15.170:3300/0 conn(0x7f43f40ac430 0x7f43f4152880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:21.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.531+0000 7f43f3fff640 1 -- 172.21.15.161:0/2089884103 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f40c3b00 con 0x7f43f40c30c0 2024-03-28T23:33:21.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.531+0000 7f43f88da640 1 --2- 172.21.15.161:0/2089884103 >> v2:172.21.15.170:3300/0 conn(0x7f43f40ac430 0x7f43f4152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:21.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.531+0000 7f43f90db640 1 --2- 172.21.15.161:0/2089884103 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43f40c2a10 0x7f43f40c19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:21.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.532+0000 7f43f3fff640 1 --2- 172.21.15.161:0/2089884103 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43f40c30c0 0x7f43f40c24d0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f43f40a95f0 tx=0x7f43e802e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9eb1a026c488e4d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:21.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.532+0000 7f43f37fe640 1 -- 172.21.15.161:0/2089884103 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e803b070 con 0x7f43f40c30c0 2024-03-28T23:33:21.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.532+0000 7f43f37fe640 1 -- 172.21.15.161:0/2089884103 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43e802ee00 con 0x7f43f40c30c0 2024-03-28T23:33:21.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.532+0000 7f43f37fe640 1 -- 172.21.15.161:0/2089884103 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e8036660 con 0x7f43f40c30c0 2024-03-28T23:33:21.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.533+0000 7f43fab3e640 1 -- 172.21.15.161:0/2089884103 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43f40c30c0 msgr2=0x7f43f40c24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:21.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.533+0000 7f43fab3e640 1 --2- 172.21.15.161:0/2089884103 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43f40c30c0 0x7f43f40c24d0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f43f40a95f0 tx=0x7f43e802e050 comp rx=0 tx=0).stop 2024-03-28T23:33:21.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.533+0000 7f43fab3e640 1 -- 172.21.15.161:0/2089884103 shutdown_connections 2024-03-28T23:33:21.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.533+0000 7f43fab3e640 1 --2- 172.21.15.161:0/2089884103 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43f40c2a10 0x7f43f40c19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:21.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.533+0000 7f43fab3e640 1 --2- 172.21.15.161:0/2089884103 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43f40c30c0 0x7f43f40c24d0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:21.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.533+0000 7f43fab3e640 1 --2- 172.21.15.161:0/2089884103 >> v2:172.21.15.170:3300/0 conn(0x7f43f40ac430 0x7f43f4152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:21.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.533+0000 7f43fab3e640 1 -- 172.21.15.161:0/2089884103 >> 172.21.15.161:0/2089884103 conn(0x7f43f406b150 msgr2=0x7f43f406b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:21.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.533+0000 7f43fab3e640 1 -- 172.21.15.161:0/2089884103 shutdown_connections 2024-03-28T23:33:21.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.533+0000 7f43fab3e640 1 -- 172.21.15.161:0/2089884103 wait complete. 2024-03-28T23:33:21.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.534+0000 7f43fab3e640 1 Processor -- start 2024-03-28T23:33:21.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.534+0000 7f43fab3e640 1 -- start start 2024-03-28T23:33:21.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.535+0000 7f43fab3e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43f40ac430 0x7f43f4072a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:21.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.535+0000 7f43fab3e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43f40c2a10 0x7f43f4142820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:21.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.535+0000 7f43fab3e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43f40c30c0 0x7f43f4142d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:21.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.535+0000 7f43fab3e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f43f40c6440 con 0x7f43f40ac430 2024-03-28T23:33:21.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.535+0000 7f43fab3e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f43f40c1490 con 0x7f43f40c2a10 2024-03-28T23:33:21.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.535+0000 7f43fab3e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f43f40ac810 con 0x7f43f40c30c0 2024-03-28T23:33:21.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.535+0000 7f43f88da640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43f40ac430 0x7f43f4072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:21.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.535+0000 7f43f88da640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43f40ac430 0x7f43f4072a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41004/0 (socket says 172.21.15.161:41004) 2024-03-28T23:33:21.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.535+0000 7f43f88da640 1 -- 172.21.15.161:0/3668666046 learned_addr learned my addr 172.21.15.161:0/3668666046 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:21.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.535+0000 7f43f90db640 1 --2- 172.21.15.161:0/3668666046 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43f40c30c0 0x7f43f4142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:21.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.535+0000 7f43f3fff640 1 --2- 172.21.15.161:0/3668666046 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43f40c2a10 0x7f43f4142820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:21.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.535+0000 7f43f88da640 1 -- 172.21.15.161:0/3668666046 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43f40c30c0 msgr2=0x7f43f4142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:21.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.535+0000 7f43f88da640 1 --2- 172.21.15.161:0/3668666046 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43f40c30c0 0x7f43f4142d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:21.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.535+0000 7f43f88da640 1 -- 172.21.15.161:0/3668666046 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43f40c2a10 msgr2=0x7f43f4142820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:21.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.535+0000 7f43f88da640 1 --2- 172.21.15.161:0/3668666046 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43f40c2a10 0x7f43f4142820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:21.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.535+0000 7f43f88da640 1 -- 172.21.15.161:0/3668666046 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f4149d80 con 0x7f43f40ac430 2024-03-28T23:33:21.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.535+0000 7f43f88da640 1 --2- 172.21.15.161:0/3668666046 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43f40ac430 0x7f43f4072a00 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f43ec00ce60 tx=0x7f43ec00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:21.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.535+0000 7f43f17fa640 1 -- 172.21.15.161:0/3668666046 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ec00f890 con 0x7f43f40ac430 2024-03-28T23:33:21.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.535+0000 7f43f17fa640 1 -- 172.21.15.161:0/3668666046 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43ec00fa30 con 0x7f43f40ac430 2024-03-28T23:33:21.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.535+0000 7f43fab3e640 1 -- 172.21.15.161:0/3668666046 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43f40a9400 con 0x7f43f40ac430 2024-03-28T23:33:21.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.535+0000 7f43f17fa640 1 -- 172.21.15.161:0/3668666046 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ec00fe20 con 0x7f43f40ac430 2024-03-28T23:33:21.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.535+0000 7f43fab3e640 1 -- 172.21.15.161:0/3668666046 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43f40a9120 con 0x7f43f40ac430 2024-03-28T23:33:21.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.536+0000 7f43fab3e640 1 -- 172.21.15.161:0/3668666046 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43f4068f90 con 0x7f43f40ac430 2024-03-28T23:33:21.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.537+0000 7f43f17fa640 1 -- 172.21.15.161:0/3668666046 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f43ec018400 con 0x7f43f40ac430 2024-03-28T23:33:21.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.537+0000 7f43f17fa640 1 --2- 172.21.15.161:0/3668666046 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f43e006b500 0x7f43e006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:21.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.537+0000 7f43f3fff640 1 --2- 172.21.15.161:0/3668666046 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f43e006b500 0x7f43e006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:21.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.537+0000 7f43f17fa640 1 -- 172.21.15.161:0/3668666046 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5249+0+0 (secure 0 0 0) 0x7f43ec08ae50 con 0x7f43f40ac430 2024-03-28T23:33:21.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.538+0000 7f43f3fff640 1 --2- 172.21.15.161:0/3668666046 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f43e006b500 0x7f43e006d9c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f43e8038000 tx=0x7f43e8032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:21.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.542+0000 7f43f17fa640 1 -- 172.21.15.161:0/3668666046 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f43ec014030 con 0x7f43f40ac430 2024-03-28T23:33:21.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:21.713+0000 7f43fab3e640 1 -- 172.21.15.161:0/3668666046 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f43f40c6220 con 0x7f43f40ac430 2024-03-28T23:33:22.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.147+0000 7f43f17fa640 1 -- 172.21.15.161:0/3668666046 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v91) ==== 138+0+0 (secure 0 0 0) 0x7f43ec00ae20 con 0x7f43f40ac430 2024-03-28T23:33:22.149 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_metadata' created 2024-03-28T23:33:22.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.150+0000 7f43fab3e640 1 -- 172.21.15.161:0/3668666046 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f43e006b500 msgr2=0x7f43e006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:22.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.150+0000 7f43fab3e640 1 --2- 172.21.15.161:0/3668666046 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f43e006b500 0x7f43e006d9c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f43e8038000 tx=0x7f43e8032000 comp rx=0 tx=0).stop 2024-03-28T23:33:22.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.150+0000 7f43fab3e640 1 -- 172.21.15.161:0/3668666046 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43f40ac430 msgr2=0x7f43f4072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:22.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.150+0000 7f43fab3e640 1 --2- 172.21.15.161:0/3668666046 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43f40ac430 0x7f43f4072a00 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f43ec00ce60 tx=0x7f43ec00a650 comp rx=0 tx=0).stop 2024-03-28T23:33:22.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.150+0000 7f43fab3e640 1 -- 172.21.15.161:0/3668666046 shutdown_connections 2024-03-28T23:33:22.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.150+0000 7f43fab3e640 1 --2- 172.21.15.161:0/3668666046 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f43e006b500 0x7f43e006d9c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:22.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.150+0000 7f43fab3e640 1 --2- 172.21.15.161:0/3668666046 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43f40c30c0 0x7f43f4142d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:22.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.150+0000 7f43fab3e640 1 --2- 172.21.15.161:0/3668666046 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43f40c2a10 0x7f43f4142820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:22.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.150+0000 7f43fab3e640 1 --2- 172.21.15.161:0/3668666046 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43f40ac430 0x7f43f4072a00 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:22.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.150+0000 7f43fab3e640 1 -- 172.21.15.161:0/3668666046 >> 172.21.15.161:0/3668666046 conn(0x7f43f406b150 msgr2=0x7f43f40ab410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:22.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.150+0000 7f43fab3e640 1 -- 172.21.15.161:0/3668666046 shutdown_connections 2024-03-28T23:33:22.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.150+0000 7f43fab3e640 1 -- 172.21.15.161:0/3668666046 wait complete. 2024-03-28T23:33:22.167 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T23:33:22.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.273+0000 7f29f932c640 1 Processor -- start 2024-03-28T23:33:22.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.273+0000 7f29f932c640 1 -- start start 2024-03-28T23:33:22.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.274+0000 7f29f932c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29f40c1040 0x7f29f41535b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:22.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.274+0000 7f29f932c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f29f40a96f0 0x7f29f415d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:22.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.274+0000 7f29f932c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29f40c2f50 con 0x7f29f4150520 2024-03-28T23:33:22.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.274+0000 7f29f932c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f29f40a9c20 con 0x7f29f40a96f0 2024-03-28T23:33:22.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.274+0000 7f29f932c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f29f4150900 con 0x7f29f40c1040 2024-03-28T23:33:22.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.274+0000 7f29f37fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f29f40a96f0 0x7f29f415d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:22.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.274+0000 7f29f37fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f29f40a96f0 0x7f29f415d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41014/0 (socket says 172.21.15.161:41014) 2024-03-28T23:33:22.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.274+0000 7f29f37fe640 1 -- 172.21.15.161:0/1901862151 learned_addr learned my addr 172.21.15.161:0/1901862151 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:22.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.274+0000 7f29f27fc640 1 --2- 172.21.15.161:0/1901862151 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29f40c1040 0x7f29f41535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:22.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.274+0000 7f29f37fe640 1 -- 172.21.15.161:0/1901862151 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29f40c1040 msgr2=0x7f29f41535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:22.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.274+0000 7f29f37fe640 1 --2- 172.21.15.161:0/1901862151 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29f40c1040 0x7f29f41535b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:22.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.274+0000 7f29f37fe640 1 -- 172.21.15.161:0/1901862151 >> v1:172.21.15.170:6789/0 conn(0x7f29f4150520 legacy=0x7f29f40ab280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:22.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.274+0000 7f29f37fe640 1 -- 172.21.15.161:0/1901862151 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29f41530d0 con 0x7f29f40a96f0 2024-03-28T23:33:22.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.275+0000 7f29f37fe640 1 --2- 172.21.15.161:0/1901862151 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29f40a96f0 0x7f29f415d8b0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f29ec009870 tx=0x7f29ec02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b078c49acf8c8c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:22.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.275+0000 7f29f1ffb640 1 -- 172.21.15.161:0/1901862151 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ec035020 con 0x7f29f40a96f0 2024-03-28T23:33:22.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.275+0000 7f29f1ffb640 1 -- 172.21.15.161:0/1901862151 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29ec031640 con 0x7f29f40a96f0 2024-03-28T23:33:22.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.275+0000 7f29f1ffb640 1 -- 172.21.15.161:0/1901862151 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ec02d040 con 0x7f29f40a96f0 2024-03-28T23:33:22.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.276+0000 7f29f932c640 1 -- 172.21.15.161:0/1901862151 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29f40a96f0 msgr2=0x7f29f415d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:22.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.276+0000 7f29f932c640 1 --2- 172.21.15.161:0/1901862151 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29f40a96f0 0x7f29f415d8b0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f29ec009870 tx=0x7f29ec02b8f0 comp rx=0 tx=0).stop 2024-03-28T23:33:22.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.276+0000 7f29f932c640 1 -- 172.21.15.161:0/1901862151 shutdown_connections 2024-03-28T23:33:22.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.276+0000 7f29f932c640 1 --2- 172.21.15.161:0/1901862151 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29f40a96f0 0x7f29f415d8b0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:22.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.276+0000 7f29f932c640 1 --2- 172.21.15.161:0/1901862151 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29f40c1040 0x7f29f41535b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:22.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.276+0000 7f29f932c640 1 -- 172.21.15.161:0/1901862151 >> 172.21.15.161:0/1901862151 conn(0x7f29f40a83f0 msgr2=0x7f29f40a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:22.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.276+0000 7f29f932c640 1 -- 172.21.15.161:0/1901862151 shutdown_connections 2024-03-28T23:33:22.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.276+0000 7f29f932c640 1 -- 172.21.15.161:0/1901862151 wait complete. 2024-03-28T23:33:22.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.277+0000 7f29f932c640 1 Processor -- start 2024-03-28T23:33:22.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.277+0000 7f29f932c640 1 -- start start 2024-03-28T23:33:22.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.278+0000 7f29f932c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29f40a96f0 0x7f29f4149ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:22.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.278+0000 7f29f932c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29f40c1040 0x7f29f40bfa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:22.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.278+0000 7f29f932c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29f4150520 0x7f29f40c5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:22.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.278+0000 7f29f932c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f29f40ab990 con 0x7f29f40c1040 2024-03-28T23:33:22.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.278+0000 7f29f932c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f29f40c2c70 con 0x7f29f4150520 2024-03-28T23:33:22.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.278+0000 7f29f932c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f29f40abb10 con 0x7f29f40a96f0 2024-03-28T23:33:22.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.278+0000 7f29f27fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29f40c1040 0x7f29f40bfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:22.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.278+0000 7f29f27fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29f40c1040 0x7f29f40bfa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41024/0 (socket says 172.21.15.161:41024) 2024-03-28T23:33:22.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.278+0000 7f29f27fc640 1 -- 172.21.15.161:0/236390642 learned_addr learned my addr 172.21.15.161:0/236390642 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:22.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.278+0000 7f29f2ffd640 1 --2- 172.21.15.161:0/236390642 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29f40a96f0 0x7f29f4149ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:22.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.278+0000 7f29f37fe640 1 --2- 172.21.15.161:0/236390642 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29f4150520 0x7f29f40c5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:22.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.278+0000 7f29f27fc640 1 -- 172.21.15.161:0/236390642 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29f40a96f0 msgr2=0x7f29f4149ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:22.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.278+0000 7f29f27fc640 1 --2- 172.21.15.161:0/236390642 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29f40a96f0 0x7f29f4149ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:22.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.278+0000 7f29f27fc640 1 -- 172.21.15.161:0/236390642 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29f4150520 msgr2=0x7f29f40c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:22.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.278+0000 7f29f27fc640 1 --2- 172.21.15.161:0/236390642 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29f4150520 0x7f29f40c5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:22.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.278+0000 7f29f27fc640 1 -- 172.21.15.161:0/236390642 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29f41eab00 con 0x7f29f40c1040 2024-03-28T23:33:22.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.278+0000 7f29f2ffd640 1 --2- 172.21.15.161:0/236390642 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29f40a96f0 0x7f29f4149ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:22.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.278+0000 7f29f27fc640 1 --2- 172.21.15.161:0/236390642 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29f40c1040 0x7f29f40bfa70 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f29e800d7d0 tx=0x7f29e800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:22.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.279+0000 7f29db7fe640 1 -- 172.21.15.161:0/236390642 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e8004020 con 0x7f29f40c1040 2024-03-28T23:33:22.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.279+0000 7f29db7fe640 1 -- 172.21.15.161:0/236390642 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29e80041e0 con 0x7f29f40c1040 2024-03-28T23:33:22.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.279+0000 7f29db7fe640 1 -- 172.21.15.161:0/236390642 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e80043d0 con 0x7f29f40c1040 2024-03-28T23:33:22.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.279+0000 7f29f932c640 1 -- 172.21.15.161:0/236390642 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29f41eade0 con 0x7f29f40c1040 2024-03-28T23:33:22.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.279+0000 7f29f932c640 1 -- 172.21.15.161:0/236390642 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29f41eb3a0 con 0x7f29f40c1040 2024-03-28T23:33:22.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.280+0000 7f29f932c640 1 -- 172.21.15.161:0/236390642 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29f404f4b0 con 0x7f29f40c1040 2024-03-28T23:33:22.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.280+0000 7f29db7fe640 1 -- 172.21.15.161:0/236390642 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f29e801b020 con 0x7f29f40c1040 2024-03-28T23:33:22.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.281+0000 7f29db7fe640 1 --2- 172.21.15.161:0/236390642 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f29e006b500 0x7f29e006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:22.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.281+0000 7f29db7fe640 1 -- 172.21.15.161:0/236390642 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5619+0+0 (secure 0 0 0) 0x7f29e8089e60 con 0x7f29f40c1040 2024-03-28T23:33:22.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.281+0000 7f29f2ffd640 1 --2- 172.21.15.161:0/236390642 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f29e006b500 0x7f29e006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:22.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.281+0000 7f29f2ffd640 1 --2- 172.21.15.161:0/236390642 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f29e006b500 0x7f29e006d9c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f29f406d330 tx=0x7f29e4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:22.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.285+0000 7f29db7fe640 1 -- 172.21.15.161:0/236390642 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f29e8058dc0 con 0x7f29f40c1040 2024-03-28T23:33:22.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:22.457+0000 7f29f932c640 1 -- 172.21.15.161:0/236390642 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f29f40c2640 con 0x7f29f40c1040 2024-03-28T23:33:23.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.154+0000 7f29db7fe640 1 -- 172.21.15.161:0/236390642 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v92) ==== 170+0+0 (secure 0 0 0) 0x7f29e80586c0 con 0x7f29f40c1040 2024-03-28T23:33:23.156 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data' created 2024-03-28T23:33:23.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.158+0000 7f29f932c640 1 -- 172.21.15.161:0/236390642 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f29e006b500 msgr2=0x7f29e006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:23.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.158+0000 7f29f932c640 1 --2- 172.21.15.161:0/236390642 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f29e006b500 0x7f29e006d9c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f29f406d330 tx=0x7f29e4009040 comp rx=0 tx=0).stop 2024-03-28T23:33:23.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.158+0000 7f29f932c640 1 -- 172.21.15.161:0/236390642 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29f40c1040 msgr2=0x7f29f40bfa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:23.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.158+0000 7f29f932c640 1 --2- 172.21.15.161:0/236390642 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29f40c1040 0x7f29f40bfa70 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f29e800d7d0 tx=0x7f29e800dc90 comp rx=0 tx=0).stop 2024-03-28T23:33:23.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.159+0000 7f29f932c640 1 -- 172.21.15.161:0/236390642 shutdown_connections 2024-03-28T23:33:23.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.159+0000 7f29f932c640 1 --2- 172.21.15.161:0/236390642 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f29e006b500 0x7f29e006d9c0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:23.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.159+0000 7f29f932c640 1 --2- 172.21.15.161:0/236390642 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29f40c1040 0x7f29f40bfa70 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:23.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.159+0000 7f29f932c640 1 --2- 172.21.15.161:0/236390642 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29f4150520 0x7f29f40c5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:23.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.159+0000 7f29f932c640 1 --2- 172.21.15.161:0/236390642 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29f40a96f0 0x7f29f4149ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:23.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.159+0000 7f29f932c640 1 -- 172.21.15.161:0/236390642 >> 172.21.15.161:0/236390642 conn(0x7f29f40a83f0 msgr2=0x7f29f40ab280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:23.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.161+0000 7f29f932c640 1 -- 172.21.15.161:0/236390642 shutdown_connections 2024-03-28T23:33:23.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.164+0000 7f29f932c640 1 -- 172.21.15.161:0/236390642 wait complete. 2024-03-28T23:33:23.186 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-28T23:33:23.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.290+0000 7fdca6d99640 1 Processor -- start 2024-03-28T23:33:23.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.290+0000 7fdca6d99640 1 -- start start 2024-03-28T23:33:23.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.291+0000 7fdca6d99640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdca006cdd0 0x7fdca01524c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:23.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.291+0000 7fdca6d99640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdca00c2f10 con 0x7fdca014f5c0 2024-03-28T23:33:23.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.291+0000 7fdca6d99640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdca014f9a0 con 0x7fdca006ba50 2024-03-28T23:33:23.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.291+0000 7fdca6d99640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdca00c74d0 con 0x7fdca006cdd0 2024-03-28T23:33:23.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.291+0000 7fdca4b35640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fdca014f5c0 0x7fdca00c2150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:55314/0 (socket says 172.21.15.161:55314) 2024-03-28T23:33:23.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.291+0000 7fdca4b35640 1 -- 172.21.15.161:0/2487710852 learned_addr learned my addr 172.21.15.161:0/2487710852 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:23.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.291+0000 7fdc9f7fe640 1 -- 172.21.15.161:0/2487710852 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2449954281 0 0) 0x7fdc90001110 con 0x7fdca014f5c0 2024-03-28T23:33:23.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.291+0000 7fdc9ffff640 1 --2- 172.21.15.161:0/2487710852 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdca006cdd0 0x7fdca01524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:23.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.292+0000 7fdc9f7fe640 1 -- 172.21.15.161:0/2487710852 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc88003520 con 0x7fdca014f5c0 2024-03-28T23:33:23.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.292+0000 7fdc9f7fe640 1 -- 172.21.15.161:0/2487710852 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3595185312 0 0) 0x7fdc88003520 con 0x7fdca014f5c0 2024-03-28T23:33:23.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.292+0000 7fdc9f7fe640 1 -- 172.21.15.161:0/2487710852 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdca006cdd0 msgr2=0x7fdca01524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:23.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.292+0000 7fdc9f7fe640 1 --2- 172.21.15.161:0/2487710852 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdca006cdd0 0x7fdca01524c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:23.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.292+0000 7fdc9f7fe640 1 -- 172.21.15.161:0/2487710852 >> v1:172.21.15.170:6789/0 conn(0x7fdca006ba50 legacy=0x7fdca015c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:23.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.292+0000 7fdc9f7fe640 1 -- 172.21.15.161:0/2487710852 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdca006c140 con 0x7fdca014f5c0 2024-03-28T23:33:23.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.292+0000 7fdc9f7fe640 1 -- 172.21.15.161:0/2487710852 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fdc90001840 con 0x7fdca014f5c0 2024-03-28T23:33:23.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.292+0000 7fdc9f7fe640 1 -- 172.21.15.161:0/2487710852 >> v1:172.21.15.161:6789/0 conn(0x7fdca014f5c0 legacy=0x7fdca00c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:23.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.292+0000 7fdc9f7fe640 1 --2- 172.21.15.161:0/2487710852 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdca006cdd0 0x7fdc88003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:23.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.292+0000 7fdc9f7fe640 1 --2- 172.21.15.161:0/2487710852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc88004040 0x7fdc88004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:23.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.292+0000 7fdc9f7fe640 1 --2- 172.21.15.161:0/2487710852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc88004d40 0x7fdc880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:23.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.292+0000 7fdc9f7fe640 1 -- 172.21.15.161:0/2487710852 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdca00c74d0 con 0x7fdc88004040 2024-03-28T23:33:23.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.292+0000 7fdc9f7fe640 1 -- 172.21.15.161:0/2487710852 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8800bbb0 con 0x7fdc88004d40 2024-03-28T23:33:23.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.292+0000 7fdc9f7fe640 1 -- 172.21.15.161:0/2487710852 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdc8800bdb0 con 0x7fdca006cdd0 2024-03-28T23:33:23.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.292+0000 7fdc9ffff640 1 --2- 172.21.15.161:0/2487710852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc88004040 0x7fdc88004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:23.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.292+0000 7fdca4b35640 1 --2- 172.21.15.161:0/2487710852 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdca006cdd0 0x7fdc88003ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:23.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.292+0000 7fdca5336640 1 --2- 172.21.15.161:0/2487710852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc88004d40 0x7fdc880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:23.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.292+0000 7fdc9ffff640 1 -- 172.21.15.161:0/2487710852 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdca006cdd0 msgr2=0x7fdc88003ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:23.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.292+0000 7fdc9ffff640 1 --2- 172.21.15.161:0/2487710852 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdca006cdd0 0x7fdc88003ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:23.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.292+0000 7fdc9ffff640 1 -- 172.21.15.161:0/2487710852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc88004d40 msgr2=0x7fdc880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:23.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.292+0000 7fdc9ffff640 1 --2- 172.21.15.161:0/2487710852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc88004d40 0x7fdc880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:23.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.292+0000 7fdc9ffff640 1 -- 172.21.15.161:0/2487710852 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc88003520 con 0x7fdc88004040 2024-03-28T23:33:23.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.292+0000 7fdca5336640 1 --2- 172.21.15.161:0/2487710852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc88004d40 0x7fdc880051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:23.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.293+0000 7fdc9ffff640 1 --2- 172.21.15.161:0/2487710852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc88004040 0x7fdc88004800 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fdc94009a90 tx=0x7fdc88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18da5b42ebd20583 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:23.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.293+0000 7fdc9f7fe640 1 -- 172.21.15.161:0/2487710852 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc94039070 con 0x7fdc88004040 2024-03-28T23:33:23.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.293+0000 7fdc9f7fe640 1 -- 172.21.15.161:0/2487710852 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc9402c680 con 0x7fdc88004040 2024-03-28T23:33:23.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.293+0000 7fdc9f7fe640 1 -- 172.21.15.161:0/2487710852 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc94009690 con 0x7fdc88004040 2024-03-28T23:33:23.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.294+0000 7fdca6d99640 1 -- 172.21.15.161:0/2487710852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc88004040 msgr2=0x7fdc88004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:23.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.294+0000 7fdca6d99640 1 --2- 172.21.15.161:0/2487710852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc88004040 0x7fdc88004800 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fdc94009a90 tx=0x7fdc88005830 comp rx=0 tx=0).stop 2024-03-28T23:33:23.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.294+0000 7fdca5336640 1 -- 172.21.15.161:0/2487710852 reap_dead start 2024-03-28T23:33:23.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.294+0000 7fdca6d99640 1 -- 172.21.15.161:0/2487710852 shutdown_connections 2024-03-28T23:33:23.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.294+0000 7fdca6d99640 1 -- 172.21.15.161:0/2487710852 >> 172.21.15.161:0/2487710852 conn(0x7fdca00a83f0 msgr2=0x7fdca00a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:23.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.294+0000 7fdca6d99640 1 -- 172.21.15.161:0/2487710852 shutdown_connections 2024-03-28T23:33:23.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.294+0000 7fdca6d99640 1 -- 172.21.15.161:0/2487710852 wait complete. 2024-03-28T23:33:23.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.295+0000 7fdca6d99640 1 Processor -- start 2024-03-28T23:33:23.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.295+0000 7fdca6d99640 1 -- start start 2024-03-28T23:33:23.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.296+0000 7fdca6d99640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdc88004040 0x7fdca00bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:23.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.296+0000 7fdca6d99640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdca014a9b0 0x7fdca014ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:23.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.296+0000 7fdca6d99640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdca0155a00 0x7fdca0155e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:23.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.296+0000 7fdca6d99640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdca0152030 con 0x7fdca014a9b0 2024-03-28T23:33:23.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.296+0000 7fdca6d99640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdca00c6fa0 con 0x7fdca0155a00 2024-03-28T23:33:23.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.296+0000 7fdca6d99640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdca006c3e0 con 0x7fdc88004040 2024-03-28T23:33:23.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.296+0000 7fdc9ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdca014a9b0 0x7fdca014ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:23.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.296+0000 7fdc9ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdca014a9b0 0x7fdca014ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41036/0 (socket says 172.21.15.161:41036) 2024-03-28T23:33:23.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.296+0000 7fdc9ffff640 1 -- 172.21.15.161:0/1211832248 learned_addr learned my addr 172.21.15.161:0/1211832248 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:23.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.296+0000 7fdca5336640 1 --2- 172.21.15.161:0/1211832248 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdca0155a00 0x7fdca0155e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:23.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.296+0000 7fdca4b35640 1 --2- 172.21.15.161:0/1211832248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdc88004040 0x7fdca00bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:23.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.296+0000 7fdc9ffff640 1 -- 172.21.15.161:0/1211832248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdc88004040 msgr2=0x7fdca00bfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:23.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.296+0000 7fdc9ffff640 1 --2- 172.21.15.161:0/1211832248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdc88004040 0x7fdca00bfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:23.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.296+0000 7fdc9ffff640 1 -- 172.21.15.161:0/1211832248 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdca0155a00 msgr2=0x7fdca0155e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:23.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.296+0000 7fdc9ffff640 1 --2- 172.21.15.161:0/1211832248 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdca0155a00 0x7fdca0155e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:23.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.296+0000 7fdc9ffff640 1 -- 172.21.15.161:0/1211832248 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdca01e9ab0 con 0x7fdca014a9b0 2024-03-28T23:33:23.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.296+0000 7fdca4b35640 1 --2- 172.21.15.161:0/1211832248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdc88004040 0x7fdca00bfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:23.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.296+0000 7fdca5336640 1 --2- 172.21.15.161:0/1211832248 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdca0155a00 0x7fdca0155e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:23.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.296+0000 7fdc9ffff640 1 --2- 172.21.15.161:0/1211832248 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdca014a9b0 0x7fdca014ae30 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fdc9402c2f0 tx=0x7fdc94004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:23.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.296+0000 7fdc9d7fa640 1 -- 172.21.15.161:0/1211832248 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc94042020 con 0x7fdca014a9b0 2024-03-28T23:33:23.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.296+0000 7fdc9d7fa640 1 -- 172.21.15.161:0/1211832248 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc9403a040 con 0x7fdca014a9b0 2024-03-28T23:33:23.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.296+0000 7fdc9d7fa640 1 -- 172.21.15.161:0/1211832248 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc94039070 con 0x7fdca014a9b0 2024-03-28T23:33:23.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.296+0000 7fdca6d99640 1 -- 172.21.15.161:0/1211832248 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdca00bf940 con 0x7fdca014a9b0 2024-03-28T23:33:23.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.296+0000 7fdca6d99640 1 -- 172.21.15.161:0/1211832248 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdca01e9d70 con 0x7fdca014a9b0 2024-03-28T23:33:23.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.297+0000 7fdca6d99640 1 -- 172.21.15.161:0/1211832248 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdca01e98d0 con 0x7fdca014a9b0 2024-03-28T23:33:23.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.298+0000 7fdc9d7fa640 1 -- 172.21.15.161:0/1211832248 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdc94045020 con 0x7fdca014a9b0 2024-03-28T23:33:23.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.298+0000 7fdc9d7fa640 1 --2- 172.21.15.161:0/1211832248 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdc8c06b500 0x7fdc8c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:23.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.298+0000 7fdc9d7fa640 1 -- 172.21.15.161:0/1211832248 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6001+0+0 (secure 0 0 0) 0x7fdc940aba50 con 0x7fdca014a9b0 2024-03-28T23:33:23.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.298+0000 7fdca4b35640 1 --2- 172.21.15.161:0/1211832248 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdc8c06b500 0x7fdc8c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:23.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.299+0000 7fdca4b35640 1 --2- 172.21.15.161:0/1211832248 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdc8c06b500 0x7fdc8c06d9c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fdc90007fd0 tx=0x7fdc90009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:23.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.303+0000 7fdc9d7fa640 1 -- 172.21.15.161:0/1211832248 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdca01e98d0 con 0x7fdca014a9b0 2024-03-28T23:33:23.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:23.513+0000 7fdca6d99640 1 -- 172.21.15.161:0/1211832248 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fdca004f4b0 con 0x7fdca014a9b0 2024-03-28T23:33:24.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.169+0000 7fdc9d7fa640 1 -- 172.21.15.161:0/1211832248 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 15 and data pool 16 v77) ==== 174+0+0 (secure 0 0 0) 0x7fdc9404c030 con 0x7fdca014a9b0 2024-03-28T23:33:24.171 INFO:teuthology.orchestra.run.smithi161.stderr:new fs with metadata pool 15 and data pool 16 2024-03-28T23:33:24.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.175+0000 7fdca6d99640 1 -- 172.21.15.161:0/1211832248 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdc8c06b500 msgr2=0x7fdc8c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:24.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.175+0000 7fdca6d99640 1 --2- 172.21.15.161:0/1211832248 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdc8c06b500 0x7fdc8c06d9c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fdc90007fd0 tx=0x7fdc90009040 comp rx=0 tx=0).stop 2024-03-28T23:33:24.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.175+0000 7fdca6d99640 1 -- 172.21.15.161:0/1211832248 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdca014a9b0 msgr2=0x7fdca014ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:24.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.175+0000 7fdca6d99640 1 --2- 172.21.15.161:0/1211832248 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdca014a9b0 0x7fdca014ae30 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fdc9402c2f0 tx=0x7fdc94004730 comp rx=0 tx=0).stop 2024-03-28T23:33:24.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.175+0000 7fdca6d99640 1 -- 172.21.15.161:0/1211832248 shutdown_connections 2024-03-28T23:33:24.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.175+0000 7fdca6d99640 1 --2- 172.21.15.161:0/1211832248 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdc8c06b500 0x7fdc8c06d9c0 secure :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fdc90007fd0 tx=0x7fdc90009040 comp rx=0 tx=0).stop 2024-03-28T23:33:24.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.175+0000 7fdca6d99640 1 --2- 172.21.15.161:0/1211832248 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdca014a9b0 0x7fdca014ae30 secure :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fdc9402c2f0 tx=0x7fdc94004730 comp rx=0 tx=0).stop 2024-03-28T23:33:24.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.175+0000 7fdca6d99640 1 --2- 172.21.15.161:0/1211832248 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdca0155a00 0x7fdca0155e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.175+0000 7fdca6d99640 1 --2- 172.21.15.161:0/1211832248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdc88004040 0x7fdca00bfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.175+0000 7fdca6d99640 1 -- 172.21.15.161:0/1211832248 >> 172.21.15.161:0/1211832248 conn(0x7fdca00a83f0 msgr2=0x7fdca0150330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:24.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.179+0000 7fdca6d99640 1 -- 172.21.15.161:0/1211832248 shutdown_connections 2024-03-28T23:33:24.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.179+0000 7fdca6d99640 1 -- 172.21.15.161:0/1211832248 wait complete. 2024-03-28T23:33:24.202 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:33:24.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.305+0000 7f200bc15640 1 Processor -- start 2024-03-28T23:33:24.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.305+0000 7f200bc15640 1 -- start start 2024-03-28T23:33:24.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.306+0000 7f200bc15640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2004152890 0x7f20040c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:24.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.306+0000 7f200bc15640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20040c8ed0 0x7f20040c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:24.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.306+0000 7f200bc15640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2004158a30 con 0x7f20040c5950 2024-03-28T23:33:24.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.306+0000 7f200bc15640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f20040c8480 con 0x7f2004152890 2024-03-28T23:33:24.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.306+0000 7f200bc15640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2004159640 con 0x7f20040c8ed0 2024-03-28T23:33:24.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.307+0000 7f20099b1640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2004152890 0x7f20040c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:24.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.307+0000 7f20091b0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20040c8ed0 0x7f20040c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:24.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.307+0000 7f20099b1640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2004152890 0x7f20040c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:37794/0 (socket says 172.21.15.161:37794) 2024-03-28T23:33:24.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.307+0000 7f20099b1640 1 -- 172.21.15.161:0/4014715498 learned_addr learned my addr 172.21.15.161:0/4014715498 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:24.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.307+0000 7f20099b1640 1 -- 172.21.15.161:0/4014715498 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20040c8ed0 msgr2=0x7f20040c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:24.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.307+0000 7f20099b1640 1 --2- 172.21.15.161:0/4014715498 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20040c8ed0 0x7f20040c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.307+0000 7f20099b1640 1 -- 172.21.15.161:0/4014715498 >> v1:172.21.15.170:6789/0 conn(0x7f20040c5950 legacy=0x7f20041553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:24.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.307+0000 7f20099b1640 1 -- 172.21.15.161:0/4014715498 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2004158d10 con 0x7f2004152890 2024-03-28T23:33:24.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.308+0000 7f20099b1640 1 --2- 172.21.15.161:0/4014715498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2004152890 0x7f20040c8990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2000009870 tx=0x7f200002b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c65c2de08222805a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:24.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.308+0000 7f20089af640 1 -- 172.21.15.161:0/4014715498 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20000317c0 con 0x7f2004152890 2024-03-28T23:33:24.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.308+0000 7f20089af640 1 -- 172.21.15.161:0/4014715498 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2000037070 con 0x7f2004152890 2024-03-28T23:33:24.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.308+0000 7f20089af640 1 -- 172.21.15.161:0/4014715498 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2000031b80 con 0x7f2004152890 2024-03-28T23:33:24.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.308+0000 7f200bc15640 1 -- 172.21.15.161:0/4014715498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2004152890 msgr2=0x7f20040c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:24.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.308+0000 7f200bc15640 1 --2- 172.21.15.161:0/4014715498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2004152890 0x7f20040c8990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2000009870 tx=0x7f200002b860 comp rx=0 tx=0).stop 2024-03-28T23:33:24.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.308+0000 7f200bc15640 1 -- 172.21.15.161:0/4014715498 shutdown_connections 2024-03-28T23:33:24.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.308+0000 7f200bc15640 1 --2- 172.21.15.161:0/4014715498 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20040c8ed0 0x7f20040c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.308+0000 7f200bc15640 1 --2- 172.21.15.161:0/4014715498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2004152890 0x7f20040c8990 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.308+0000 7f200bc15640 1 -- 172.21.15.161:0/4014715498 >> 172.21.15.161:0/4014715498 conn(0x7f20040ac620 msgr2=0x7f20040aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:24.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.308+0000 7f200bc15640 1 -- 172.21.15.161:0/4014715498 shutdown_connections 2024-03-28T23:33:24.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.308+0000 7f200bc15640 1 -- 172.21.15.161:0/4014715498 wait complete. 2024-03-28T23:33:24.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.309+0000 7f200bc15640 1 Processor -- start 2024-03-28T23:33:24.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.310+0000 7f200bc15640 1 -- start start 2024-03-28T23:33:24.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.310+0000 7f200bc15640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20040c5950 0x7f2004157380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:24.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.310+0000 7f200bc15640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20040c8ed0 0x7f20041ed740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:24.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.310+0000 7f200bc15640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2004152890 0x7f20041ec3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:24.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.310+0000 7f200bc15640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2004152c70 con 0x7f2004152890 2024-03-28T23:33:24.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.310+0000 7f200bc15640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f20041592d0 con 0x7f20040c8ed0 2024-03-28T23:33:24.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.310+0000 7f200bc15640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2004152df0 con 0x7f20040c5950 2024-03-28T23:33:24.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.310+0000 7f200a1b2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2004152890 0x7f20041ec3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:24.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.310+0000 7f200a1b2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2004152890 0x7f20041ec3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41052/0 (socket says 172.21.15.161:41052) 2024-03-28T23:33:24.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.310+0000 7f200a1b2640 1 -- 172.21.15.161:0/2879787151 learned_addr learned my addr 172.21.15.161:0/2879787151 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:24.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.310+0000 7f20099b1640 1 --2- 172.21.15.161:0/2879787151 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20040c5950 0x7f2004157380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:24.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.310+0000 7f20091b0640 1 --2- 172.21.15.161:0/2879787151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20040c8ed0 0x7f20041ed740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:24.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.311+0000 7f200a1b2640 1 -- 172.21.15.161:0/2879787151 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20040c5950 msgr2=0x7f2004157380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:24.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.311+0000 7f200a1b2640 1 --2- 172.21.15.161:0/2879787151 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20040c5950 0x7f2004157380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.311+0000 7f200a1b2640 1 -- 172.21.15.161:0/2879787151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20040c8ed0 msgr2=0x7f20041ed740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:24.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.311+0000 7f200a1b2640 1 --2- 172.21.15.161:0/2879787151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20040c8ed0 0x7f20041ed740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.311+0000 7f200a1b2640 1 -- 172.21.15.161:0/2879787151 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2004158780 con 0x7f2004152890 2024-03-28T23:33:24.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.311+0000 7f200a1b2640 1 --2- 172.21.15.161:0/2879787151 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2004152890 0x7f20041ec3b0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc00c6b0 tx=0x7f1ffc00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:24.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.311+0000 7f1ffa7fc640 1 -- 172.21.15.161:0/2879787151 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc003b70 con 0x7f2004152890 2024-03-28T23:33:24.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.311+0000 7f1ffa7fc640 1 -- 172.21.15.161:0/2879787151 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ffc003d30 con 0x7f2004152890 2024-03-28T23:33:24.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.311+0000 7f200bc15640 1 -- 172.21.15.161:0/2879787151 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20040ab320 con 0x7f2004152890 2024-03-28T23:33:24.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.311+0000 7f1ffa7fc640 1 -- 172.21.15.161:0/2879787151 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc005310 con 0x7f2004152890 2024-03-28T23:33:24.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.311+0000 7f200bc15640 1 -- 172.21.15.161:0/2879787151 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20040aaa80 con 0x7f2004152890 2024-03-28T23:33:24.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.312+0000 7f200bc15640 1 -- 172.21.15.161:0/2879787151 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f200406c6c0 con 0x7f2004152890 2024-03-28T23:33:24.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.312+0000 7f1ffa7fc640 1 -- 172.21.15.161:0/2879787151 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1ffc0054b0 con 0x7f2004152890 2024-03-28T23:33:24.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.313+0000 7f1ffa7fc640 1 --2- 172.21.15.161:0/2879787151 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1fe806b4d0 0x7f1fe806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:24.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.313+0000 7f1ffa7fc640 1 -- 172.21.15.161:0/2879787151 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6101+0+0 (secure 0 0 0) 0x7f1ffc08a040 con 0x7f2004152890 2024-03-28T23:33:24.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.313+0000 7f20099b1640 1 --2- 172.21.15.161:0/2879787151 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1fe806b4d0 0x7f1fe806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:24.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.313+0000 7f20099b1640 1 --2- 172.21.15.161:0/2879787151 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1fe806b4d0 0x7f1fe806d990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f2000036000 tx=0x7f200003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:24.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.317+0000 7f1ffa7fc640 1 -- 172.21.15.161:0/2879787151 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f200406c6c0 con 0x7f2004152890 2024-03-28T23:33:24.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.490+0000 7f200bc15640 1 -- 172.21.15.161:0/2879787151 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2004156570 con 0x7f2004152890 2024-03-28T23:33:24.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.491+0000 7f1ffa7fc640 1 -- 172.21.15.161:0/2879787151 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+16922 (secure 0 0 0) 0x7f1ffc058770 con 0x7f2004152890 2024-03-28T23:33:24.493 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:24.493 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":93,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:33:23.515866+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:33:21.732337+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-28T23:33:22.477357+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:33:24.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.493+0000 7f200bc15640 1 -- 172.21.15.161:0/2879787151 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1fe806b4d0 msgr2=0x7f1fe806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:24.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.493+0000 7f200bc15640 1 --2- 172.21.15.161:0/2879787151 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1fe806b4d0 0x7f1fe806d990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f2000036000 tx=0x7f200003d040 comp rx=0 tx=0).stop 2024-03-28T23:33:24.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.493+0000 7f200bc15640 1 -- 172.21.15.161:0/2879787151 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2004152890 msgr2=0x7f20041ec3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:24.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.493+0000 7f200bc15640 1 --2- 172.21.15.161:0/2879787151 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2004152890 0x7f20041ec3b0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc00c6b0 tx=0x7f1ffc00cb70 comp rx=0 tx=0).stop 2024-03-28T23:33:24.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.493+0000 7f200bc15640 1 -- 172.21.15.161:0/2879787151 shutdown_connections 2024-03-28T23:33:24.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.493+0000 7f200bc15640 1 --2- 172.21.15.161:0/2879787151 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1fe806b4d0 0x7f1fe806d990 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.493+0000 7f200bc15640 1 --2- 172.21.15.161:0/2879787151 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2004152890 0x7f20041ec3b0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.493+0000 7f200bc15640 1 --2- 172.21.15.161:0/2879787151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20040c8ed0 0x7f20041ed740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.493+0000 7f200bc15640 1 --2- 172.21.15.161:0/2879787151 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20040c5950 0x7f2004157380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.493+0000 7f200bc15640 1 -- 172.21.15.161:0/2879787151 >> 172.21.15.161:0/2879787151 conn(0x7f20040ac620 msgr2=0x7f200414fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:24.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.493+0000 7f200bc15640 1 -- 172.21.15.161:0/2879787151 shutdown_connections 2024-03-28T23:33:24.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.493+0000 7f200bc15640 1 -- 172.21.15.161:0/2879787151 wait complete. 2024-03-28T23:33:24.507 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:33:24.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.607+0000 7f0ce31e9640 1 Processor -- start 2024-03-28T23:33:24.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.607+0000 7f0ce31e9640 1 -- start start 2024-03-28T23:33:24.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.607+0000 7f0ce31e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cdc06ce80 0x7f0cdc0aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:24.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.607+0000 7f0ce31e9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0cdc0a9680 0x7f0cdc0aab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:24.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.607+0000 7f0ce31e9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cdc14c250 con 0x7f0cdc14be70 2024-03-28T23:33:24.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.607+0000 7f0ce31e9640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0cdc1529c0 con 0x7f0cdc0a9680 2024-03-28T23:33:24.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.607+0000 7f0ce31e9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0cdc0a9dc0 con 0x7f0cdc06ce80 2024-03-28T23:33:24.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.608+0000 7f0ce1786640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0cdc0a9680 0x7f0cdc0aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:24.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.608+0000 7f0ce0f85640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f0cdc14be70 0x7f0cdc0ab1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:55330/0 (socket says 172.21.15.161:55330) 2024-03-28T23:33:24.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.608+0000 7f0ce0f85640 1 -- 172.21.15.161:0/3941378930 learned_addr learned my addr 172.21.15.161:0/3941378930 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:24.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.608+0000 7f0cdbfff640 1 --2- 172.21.15.161:0/3941378930 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cdc06ce80 0x7f0cdc0aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:24.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.608+0000 7f0ce1786640 1 -- 172.21.15.161:0/3941378930 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cdc06ce80 msgr2=0x7f0cdc0aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:24.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.608+0000 7f0ce1786640 1 --2- 172.21.15.161:0/3941378930 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cdc06ce80 0x7f0cdc0aa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.608+0000 7f0ce1786640 1 -- 172.21.15.161:0/3941378930 >> v1:172.21.15.161:6789/0 conn(0x7f0cdc14be70 legacy=0x7f0cdc0ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:24.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.608+0000 7f0cdb7fe640 1 -- 172.21.15.161:0/3941378930 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4050093282 0 0) 0x7f0cdc14c250 con 0x7f0cdc14be70 2024-03-28T23:33:24.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.608+0000 7f0ce1786640 1 -- 172.21.15.161:0/3941378930 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cdc0aa0a0 con 0x7f0cdc0a9680 2024-03-28T23:33:24.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.608+0000 7f0cdbfff640 1 --2- 172.21.15.161:0/3941378930 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cdc06ce80 0x7f0cdc0aa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:24.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.609+0000 7f0ce1786640 1 --2- 172.21.15.161:0/3941378930 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0cdc0a9680 0x7f0cdc0aab00 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4009d90 tx=0x7f0cd402bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c32312215a79c85 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:24.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.609+0000 7f0cdb7fe640 1 -- 172.21.15.161:0/3941378930 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4039070 con 0x7f0cdc0a9680 2024-03-28T23:33:24.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.609+0000 7f0cdb7fe640 1 -- 172.21.15.161:0/3941378930 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cd402cb30 con 0x7f0cdc0a9680 2024-03-28T23:33:24.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.609+0000 7f0cdb7fe640 1 -- 172.21.15.161:0/3941378930 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4034440 con 0x7f0cdc0a9680 2024-03-28T23:33:24.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.609+0000 7f0ce31e9640 1 -- 172.21.15.161:0/3941378930 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0cdc0a9680 msgr2=0x7f0cdc0aab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:24.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.609+0000 7f0ce31e9640 1 --2- 172.21.15.161:0/3941378930 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0cdc0a9680 0x7f0cdc0aab00 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4009d90 tx=0x7f0cd402bd80 comp rx=0 tx=0).stop 2024-03-28T23:33:24.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.609+0000 7f0ce31e9640 1 -- 172.21.15.161:0/3941378930 shutdown_connections 2024-03-28T23:33:24.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.609+0000 7f0ce31e9640 1 --2- 172.21.15.161:0/3941378930 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0cdc0a9680 0x7f0cdc0aab00 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.609+0000 7f0ce31e9640 1 --2- 172.21.15.161:0/3941378930 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cdc06ce80 0x7f0cdc0aa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.609+0000 7f0ce31e9640 1 -- 172.21.15.161:0/3941378930 >> 172.21.15.161:0/3941378930 conn(0x7f0cdc0c6670 msgr2=0x7f0cdc0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:24.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.609+0000 7f0ce31e9640 1 -- 172.21.15.161:0/3941378930 shutdown_connections 2024-03-28T23:33:24.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.609+0000 7f0ce31e9640 1 -- 172.21.15.161:0/3941378930 wait complete. 2024-03-28T23:33:24.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.610+0000 7f0ce31e9640 1 Processor -- start 2024-03-28T23:33:24.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.610+0000 7f0ce31e9640 1 -- start start 2024-03-28T23:33:24.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.611+0000 7f0ce31e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cdc06ce80 0x7f0cdc1e90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:24.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.611+0000 7f0ce31e9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0cdc0a9680 0x7f0cdc154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:24.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.611+0000 7f0ce31e9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0cdc14be70 0x7f0cdc154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:24.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.611+0000 7f0ce31e9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0cdc0aba30 con 0x7f0cdc0a9680 2024-03-28T23:33:24.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.611+0000 7f0ce31e9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0cdc0ab8b0 con 0x7f0cdc14be70 2024-03-28T23:33:24.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.611+0000 7f0ce31e9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0cdc06b150 con 0x7f0cdc06ce80 2024-03-28T23:33:24.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.611+0000 7f0cdbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0cdc0a9680 0x7f0cdc154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:24.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.611+0000 7f0cdbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0cdc0a9680 0x7f0cdc154520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41074/0 (socket says 172.21.15.161:41074) 2024-03-28T23:33:24.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.611+0000 7f0cdbfff640 1 -- 172.21.15.161:0/4012559445 learned_addr learned my addr 172.21.15.161:0/4012559445 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:24.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.612+0000 7f0ce0f85640 1 --2- 172.21.15.161:0/4012559445 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cdc06ce80 0x7f0cdc1e90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:24.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.612+0000 7f0cdbfff640 1 -- 172.21.15.161:0/4012559445 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cdc06ce80 msgr2=0x7f0cdc1e90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:24.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.612+0000 7f0cdbfff640 1 --2- 172.21.15.161:0/4012559445 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cdc06ce80 0x7f0cdc1e90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.612+0000 7f0ce1786640 1 --2- 172.21.15.161:0/4012559445 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0cdc14be70 0x7f0cdc154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:24.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.612+0000 7f0cdbfff640 1 -- 172.21.15.161:0/4012559445 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0cdc14be70 msgr2=0x7f0cdc154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:24.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.612+0000 7f0cdbfff640 1 --2- 172.21.15.161:0/4012559445 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0cdc14be70 0x7f0cdc154a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.612+0000 7f0cdbfff640 1 -- 172.21.15.161:0/4012559445 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cdc1e9d50 con 0x7f0cdc0a9680 2024-03-28T23:33:24.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.612+0000 7f0ce0f85640 1 --2- 172.21.15.161:0/4012559445 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cdc06ce80 0x7f0cdc1e90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:24.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.612+0000 7f0cdbfff640 1 --2- 172.21.15.161:0/4012559445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0cdc0a9680 0x7f0cdc154520 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f0cc800c7f0 tx=0x7f0cc800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:24.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.612+0000 7f0cd97fa640 1 -- 172.21.15.161:0/4012559445 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8015070 con 0x7f0cdc0a9680 2024-03-28T23:33:24.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.612+0000 7f0cd97fa640 1 -- 172.21.15.161:0/4012559445 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cc8007820 con 0x7f0cdc0a9680 2024-03-28T23:33:24.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.612+0000 7f0ce31e9640 1 -- 172.21.15.161:0/4012559445 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cdc1554b0 con 0x7f0cdc0a9680 2024-03-28T23:33:24.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.612+0000 7f0cd97fa640 1 -- 172.21.15.161:0/4012559445 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8007bd0 con 0x7f0cdc0a9680 2024-03-28T23:33:24.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.612+0000 7f0ce31e9640 1 -- 172.21.15.161:0/4012559445 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cdc1e9a70 con 0x7f0cdc0a9680 2024-03-28T23:33:24.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.613+0000 7f0ce31e9640 1 -- 172.21.15.161:0/4012559445 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cdc04f4b0 con 0x7f0cdc0a9680 2024-03-28T23:33:24.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.613+0000 7f0cd97fa640 1 -- 172.21.15.161:0/4012559445 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0cc8018070 con 0x7f0cdc0a9680 2024-03-28T23:33:24.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.614+0000 7f0cd97fa640 1 --2- 172.21.15.161:0/4012559445 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0cc006b4d0 0x7f0cc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:24.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.614+0000 7f0cd97fa640 1 -- 172.21.15.161:0/4012559445 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6101+0+0 (secure 0 0 0) 0x7f0cc808a470 con 0x7f0cdc0a9680 2024-03-28T23:33:24.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.614+0000 7f0ce0f85640 1 --2- 172.21.15.161:0/4012559445 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0cc006b4d0 0x7f0cc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:24.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.614+0000 7f0ce0f85640 1 --2- 172.21.15.161:0/4012559445 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0cc006b4d0 0x7f0cc006d990 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0000f60 tx=0x7f0cd0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:24.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.618+0000 7f0cd97fa640 1 -- 172.21.15.161:0/4012559445 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0cc80591f0 con 0x7f0cdc0a9680 2024-03-28T23:33:24.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.790+0000 7f0ce31e9640 1 -- 172.21.15.161:0/4012559445 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0cdc0c6430 con 0x7f0cdc0a9680 2024-03-28T23:33:24.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.791+0000 7f0cd97fa640 1 -- 172.21.15.161:0/4012559445 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+16922 (secure 0 0 0) 0x7f0cc8058af0 con 0x7f0cdc0a9680 2024-03-28T23:33:24.793 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:24.794 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":93,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:33:23.515866+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:33:21.732337+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-28T23:33:22.477357+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:33:24.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.793+0000 7f0ce31e9640 1 -- 172.21.15.161:0/4012559445 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0cc006b4d0 msgr2=0x7f0cc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:24.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.793+0000 7f0ce31e9640 1 --2- 172.21.15.161:0/4012559445 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0cc006b4d0 0x7f0cc006d990 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0000f60 tx=0x7f0cd0009040 comp rx=0 tx=0).stop 2024-03-28T23:33:24.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.793+0000 7f0ce31e9640 1 -- 172.21.15.161:0/4012559445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0cdc0a9680 msgr2=0x7f0cdc154520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:24.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.793+0000 7f0ce31e9640 1 --2- 172.21.15.161:0/4012559445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0cdc0a9680 0x7f0cdc154520 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f0cc800c7f0 tx=0x7f0cc800ccb0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.793+0000 7f0ce31e9640 1 -- 172.21.15.161:0/4012559445 shutdown_connections 2024-03-28T23:33:24.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.793+0000 7f0ce31e9640 1 --2- 172.21.15.161:0/4012559445 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0cc006b4d0 0x7f0cc006d990 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.793+0000 7f0ce31e9640 1 --2- 172.21.15.161:0/4012559445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0cdc0a9680 0x7f0cdc154520 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.793+0000 7f0ce31e9640 1 --2- 172.21.15.161:0/4012559445 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0cdc14be70 0x7f0cdc154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.793+0000 7f0ce31e9640 1 --2- 172.21.15.161:0/4012559445 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cdc06ce80 0x7f0cdc1e90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.793+0000 7f0ce31e9640 1 -- 172.21.15.161:0/4012559445 >> 172.21.15.161:0/4012559445 conn(0x7f0cdc0c6670 msgr2=0x7f0cdc0ab1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:24.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.793+0000 7f0ce31e9640 1 -- 172.21.15.161:0/4012559445 shutdown_connections 2024-03-28T23:33:24.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.793+0000 7f0ce31e9640 1 -- 172.21.15.161:0/4012559445 wait complete. 2024-03-28T23:33:24.807 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-28T23:33:24.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.910+0000 7f436318a640 1 Processor -- start 2024-03-28T23:33:24.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.910+0000 7f436318a640 1 -- start start 2024-03-28T23:33:24.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f436318a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f435c14f5c0 0x7f435c0c2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:24.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f436318a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f435c152bd0 0x7f435c15c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:24.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f436318a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f435c0c2f10 con 0x7f435c06cdd0 2024-03-28T23:33:24.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f436318a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f435c0c74d0 con 0x7f435c152bd0 2024-03-28T23:33:24.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f436318a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f435c0c6fa0 con 0x7f435c14f5c0 2024-03-28T23:33:24.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f435bfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f435c06cdd0 0x7f435c1524c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:55332/0 (socket says 172.21.15.161:55332) 2024-03-28T23:33:24.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f435bfff640 1 -- 172.21.15.161:0/2963341571 learned_addr learned my addr 172.21.15.161:0/2963341571 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:24.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f4361727640 1 --2- 172.21.15.161:0/2963341571 >> v2:172.21.15.170:3300/0 conn(0x7f435c152bd0 0x7f435c15c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:24.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f4360f26640 1 --2- 172.21.15.161:0/2963341571 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f435c14f5c0 0x7f435c0c2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:24.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f435b7fe640 1 -- 172.21.15.161:0/2963341571 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2736632139 0 0) 0x7f4348001110 con 0x7f435c06cdd0 2024-03-28T23:33:24.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f435b7fe640 1 -- 172.21.15.161:0/2963341571 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f434c005730 con 0x7f435c06cdd0 2024-03-28T23:33:24.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f435b7fe640 1 -- 172.21.15.161:0/2963341571 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3613158224 0 0) 0x7f434c005730 con 0x7f435c06cdd0 2024-03-28T23:33:24.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f435b7fe640 1 -- 172.21.15.161:0/2963341571 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f435c14f5c0 msgr2=0x7f435c0c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:24.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f435b7fe640 1 --2- 172.21.15.161:0/2963341571 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f435c14f5c0 0x7f435c0c2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f435b7fe640 1 -- 172.21.15.161:0/2963341571 >> v2:172.21.15.170:3300/0 conn(0x7f435c152bd0 msgr2=0x7f435c15c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:24.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f435b7fe640 1 --2- 172.21.15.161:0/2963341571 >> v2:172.21.15.170:3300/0 conn(0x7f435c152bd0 0x7f435c15c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f435b7fe640 1 -- 172.21.15.161:0/2963341571 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f435c14f9a0 con 0x7f435c06cdd0 2024-03-28T23:33:24.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f435b7fe640 1 -- 172.21.15.161:0/2963341571 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f4348001840 con 0x7f435c06cdd0 2024-03-28T23:33:24.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f4361727640 1 --2- 172.21.15.161:0/2963341571 >> v2:172.21.15.170:3300/0 conn(0x7f435c152bd0 0x7f435c15c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:24.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f435b7fe640 1 -- 172.21.15.161:0/2963341571 >> v1:172.21.15.161:6789/0 conn(0x7f435c06cdd0 legacy=0x7f435c1524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:24.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f435b7fe640 1 --2- 172.21.15.161:0/2963341571 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f434c005ca0 0x7f434c0060c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:24.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f435b7fe640 1 --2- 172.21.15.161:0/2963341571 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f434c006a10 0x7f434c006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:24.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f435b7fe640 1 --2- 172.21.15.161:0/2963341571 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f435c14f5c0 0x7f434c0073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:24.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f435b7fe640 1 -- 172.21.15.161:0/2963341571 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f435c0c74d0 con 0x7f434c005ca0 2024-03-28T23:33:24.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f435b7fe640 1 -- 172.21.15.161:0/2963341571 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f435c0c6fa0 con 0x7f434c006a10 2024-03-28T23:33:24.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.911+0000 7f435b7fe640 1 -- 172.21.15.161:0/2963341571 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f434c00dc00 con 0x7f435c14f5c0 2024-03-28T23:33:24.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.912+0000 7f4360f26640 1 --2- 172.21.15.161:0/2963341571 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f434c005ca0 0x7f434c0060c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:24.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.912+0000 7f435bfff640 1 --2- 172.21.15.161:0/2963341571 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f434c006a10 0x7f434c006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:24.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.912+0000 7f4361727640 1 --2- 172.21.15.161:0/2963341571 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f435c14f5c0 0x7f434c0073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.912+0000 7f4360f26640 1 -- 172.21.15.161:0/2963341571 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f435c14f5c0 msgr2=0x7f434c0073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.912+0000 7f4360f26640 1 --2- 172.21.15.161:0/2963341571 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f435c14f5c0 0x7f434c0073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.912+0000 7f4360f26640 1 -- 172.21.15.161:0/2963341571 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f434c006a10 msgr2=0x7f434c006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.912+0000 7f4360f26640 1 --2- 172.21.15.161:0/2963341571 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f434c006a10 0x7f434c006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.912+0000 7f4360f26640 1 -- 172.21.15.161:0/2963341571 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f434c005730 con 0x7f434c005ca0 2024-03-28T23:33:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.912+0000 7f4361727640 1 --2- 172.21.15.161:0/2963341571 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f435c14f5c0 0x7f434c0073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.912+0000 7f4360f26640 1 --2- 172.21.15.161:0/2963341571 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f434c005ca0 0x7f434c0060c0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f4350009ac0 tx=0x7f434c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fee982a52d5e4bf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.913+0000 7f435b7fe640 1 -- 172.21.15.161:0/2963341571 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4350039070 con 0x7f434c005ca0 2024-03-28T23:33:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.913+0000 7f435b7fe640 1 -- 172.21.15.161:0/2963341571 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f435002c6b0 con 0x7f434c005ca0 2024-03-28T23:33:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.913+0000 7f435b7fe640 1 -- 172.21.15.161:0/2963341571 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4350009690 con 0x7f434c005ca0 2024-03-28T23:33:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.913+0000 7f436318a640 1 -- 172.21.15.161:0/2963341571 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f434c005ca0 msgr2=0x7f434c0060c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:24.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.913+0000 7f436318a640 1 --2- 172.21.15.161:0/2963341571 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f434c005ca0 0x7f434c0060c0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f4350009ac0 tx=0x7f434c007a40 comp rx=0 tx=0).stop 2024-03-28T23:33:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.913+0000 7f4361727640 1 -- 172.21.15.161:0/2963341571 reap_dead start 2024-03-28T23:33:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.913+0000 7f436318a640 1 -- 172.21.15.161:0/2963341571 shutdown_connections 2024-03-28T23:33:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.913+0000 7f436318a640 1 -- 172.21.15.161:0/2963341571 >> 172.21.15.161:0/2963341571 conn(0x7f435c0a83f0 msgr2=0x7f435c0a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.913+0000 7f436318a640 1 -- 172.21.15.161:0/2963341571 shutdown_connections 2024-03-28T23:33:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.913+0000 7f436318a640 1 -- 172.21.15.161:0/2963341571 wait complete. 2024-03-28T23:33:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.914+0000 7f436318a640 1 Processor -- start 2024-03-28T23:33:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.914+0000 7f436318a640 1 -- start start 2024-03-28T23:33:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.915+0000 7f436318a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f434c005ca0 0x7f435c149b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.915+0000 7f436318a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f435c1556e0 0x7f435c1e6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.915+0000 7f436318a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f435c1e70b0 0x7f435c1e9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.915+0000 7f436318a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f435c06c140 con 0x7f435c1556e0 2024-03-28T23:33:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.915+0000 7f436318a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f435c152030 con 0x7f435c1e70b0 2024-03-28T23:33:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.915+0000 7f436318a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f435c06c3e0 con 0x7f434c005ca0 2024-03-28T23:33:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.915+0000 7f435bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f435c1556e0 0x7f435c1e6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:24.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.915+0000 7f435bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f435c1556e0 0x7f435c1e6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41100/0 (socket says 172.21.15.161:41100) 2024-03-28T23:33:24.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.915+0000 7f435bfff640 1 -- 172.21.15.161:0/4285859824 learned_addr learned my addr 172.21.15.161:0/4285859824 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:24.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.915+0000 7f4360f26640 1 --2- 172.21.15.161:0/4285859824 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f434c005ca0 0x7f435c149b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:24.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.915+0000 7f4361727640 1 --2- 172.21.15.161:0/4285859824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f435c1e70b0 0x7f435c1e9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:24.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.916+0000 7f435bfff640 1 -- 172.21.15.161:0/4285859824 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f434c005ca0 msgr2=0x7f435c149b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:24.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.916+0000 7f435bfff640 1 --2- 172.21.15.161:0/4285859824 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f434c005ca0 0x7f435c149b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.916+0000 7f435bfff640 1 -- 172.21.15.161:0/4285859824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f435c1e70b0 msgr2=0x7f435c1e9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:24.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.916+0000 7f435bfff640 1 --2- 172.21.15.161:0/4285859824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f435c1e70b0 0x7f435c1e9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:24.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.916+0000 7f435bfff640 1 -- 172.21.15.161:0/4285859824 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f435c14cbb0 con 0x7f435c1556e0 2024-03-28T23:33:24.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.916+0000 7f4361727640 1 --2- 172.21.15.161:0/4285859824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f435c1e70b0 0x7f435c1e9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:24.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.916+0000 7f435bfff640 1 --2- 172.21.15.161:0/4285859824 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f435c1556e0 0x7f435c1e6b70 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f4348004f70 tx=0x7f43480074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:24.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.916+0000 7f43597fa640 1 -- 172.21.15.161:0/4285859824 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4348007740 con 0x7f435c1556e0 2024-03-28T23:33:24.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.916+0000 7f43597fa640 1 -- 172.21.15.161:0/4285859824 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4348018070 con 0x7f435c1556e0 2024-03-28T23:33:24.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.916+0000 7f436318a640 1 -- 172.21.15.161:0/4285859824 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f435c14a150 con 0x7f435c1556e0 2024-03-28T23:33:24.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.916+0000 7f43597fa640 1 -- 172.21.15.161:0/4285859824 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f434800fbc0 con 0x7f435c1556e0 2024-03-28T23:33:24.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.916+0000 7f436318a640 1 -- 172.21.15.161:0/4285859824 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f435c14c5f0 con 0x7f435c1556e0 2024-03-28T23:33:24.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.917+0000 7f436318a640 1 -- 172.21.15.161:0/4285859824 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f435c04f4b0 con 0x7f435c1556e0 2024-03-28T23:33:24.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.917+0000 7f43597fa640 1 -- 172.21.15.161:0/4285859824 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f434801d460 con 0x7f435c1556e0 2024-03-28T23:33:24.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.918+0000 7f43597fa640 1 --2- 172.21.15.161:0/4285859824 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f434006b4d0 0x7f434006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:24.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.918+0000 7f43597fa640 1 -- 172.21.15.161:0/4285859824 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6101+0+0 (secure 0 0 0) 0x7f434808a390 con 0x7f435c1556e0 2024-03-28T23:33:24.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.918+0000 7f4360f26640 1 --2- 172.21.15.161:0/4285859824 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f434006b4d0 0x7f434006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:24.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.918+0000 7f4360f26640 1 --2- 172.21.15.161:0/4285859824 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f434006b4d0 0x7f434006d990 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4350036000 tx=0x7f4350030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:24.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:24.922+0000 7f43597fa640 1 -- 172.21.15.161:0/4285859824 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f435c0c2f10 con 0x7f435c1556e0 2024-03-28T23:33:25.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.132+0000 7f436318a640 1 -- 172.21.15.161:0/4285859824 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f435c0c0ae0 con 0x7f435c1556e0 2024-03-28T23:33:25.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.172+0000 7f43597fa640 1 -- 172.21.15.161:0/4285859824 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v78) ==== 118+0+0 (secure 0 0 0) 0x7f4348059110 con 0x7f435c1556e0 2024-03-28T23:33:25.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.173+0000 7f436318a640 1 -- 172.21.15.161:0/4285859824 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f434006b4d0 msgr2=0x7f434006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:25.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.173+0000 7f436318a640 1 --2- 172.21.15.161:0/4285859824 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f434006b4d0 0x7f434006d990 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4350036000 tx=0x7f4350030000 comp rx=0 tx=0).stop 2024-03-28T23:33:25.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.173+0000 7f436318a640 1 -- 172.21.15.161:0/4285859824 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f435c1556e0 msgr2=0x7f435c1e6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:25.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.173+0000 7f436318a640 1 --2- 172.21.15.161:0/4285859824 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f435c1556e0 0x7f435c1e6b70 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f4348004f70 tx=0x7f43480074f0 comp rx=0 tx=0).stop 2024-03-28T23:33:25.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.173+0000 7f436318a640 1 -- 172.21.15.161:0/4285859824 shutdown_connections 2024-03-28T23:33:25.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.173+0000 7f436318a640 1 --2- 172.21.15.161:0/4285859824 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f434006b4d0 0x7f434006d990 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:25.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.173+0000 7f436318a640 1 --2- 172.21.15.161:0/4285859824 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f435c1556e0 0x7f435c1e6b70 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:25.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.173+0000 7f436318a640 1 --2- 172.21.15.161:0/4285859824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f435c1e70b0 0x7f435c1e9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:25.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.173+0000 7f436318a640 1 --2- 172.21.15.161:0/4285859824 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f434c005ca0 0x7f435c149b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:25.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.173+0000 7f436318a640 1 -- 172.21.15.161:0/4285859824 >> 172.21.15.161:0/4285859824 conn(0x7f435c0a83f0 msgr2=0x7f435c0a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:25.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.173+0000 7f436318a640 1 -- 172.21.15.161:0/4285859824 shutdown_connections 2024-03-28T23:33:25.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.174+0000 7f436318a640 1 -- 172.21.15.161:0/4285859824 wait complete. 2024-03-28T23:33:25.188 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-28T23:33:25.188 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-28T23:33:25.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.292+0000 7f3c4add0640 1 Processor -- start 2024-03-28T23:33:25.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.292+0000 7f3c4add0640 1 -- start start 2024-03-28T23:33:25.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.293+0000 7f3c4add0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c4414f5c0 0x7f3c440c2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:25.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.293+0000 7f3c4add0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3c4406cdd0 0x7f3c441524c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:25.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.293+0000 7f3c4add0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3c440c2690 0x7f3c44152a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:25.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.293+0000 7f3c4add0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f3c440c74d0 con 0x7f3c440c2690 2024-03-28T23:33:25.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.293+0000 7f3c4add0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f3c440c6fa0 con 0x7f3c4406cdd0 2024-03-28T23:33:25.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.293+0000 7f3c4add0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3c44152030 con 0x7f3c4414f5c0 2024-03-28T23:33:25.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.293+0000 7f3c4936d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3c440c2690 0x7f3c44152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:25.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.293+0000 7f3c4936d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3c440c2690 0x7f3c44152a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41116/0 (socket says 172.21.15.161:41116) 2024-03-28T23:33:25.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.293+0000 7f3c4936d640 1 -- 172.21.15.161:0/1671783387 learned_addr learned my addr 172.21.15.161:0/1671783387 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:25.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.293+0000 7f3c43fff640 1 --2- 172.21.15.161:0/1671783387 >> v2:172.21.15.170:3300/0 conn(0x7f3c4406cdd0 0x7f3c441524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:25.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.293+0000 7f3c48b6c640 1 --2- 172.21.15.161:0/1671783387 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c4414f5c0 0x7f3c440c2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:25.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.293+0000 7f3c4936d640 1 -- 172.21.15.161:0/1671783387 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c4414f5c0 msgr2=0x7f3c440c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:25.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.293+0000 7f3c4936d640 1 --2- 172.21.15.161:0/1671783387 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c4414f5c0 0x7f3c440c2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:25.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.293+0000 7f3c4936d640 1 -- 172.21.15.161:0/1671783387 >> v2:172.21.15.170:3300/0 conn(0x7f3c4406cdd0 msgr2=0x7f3c441524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:25.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.293+0000 7f3c4936d640 1 --2- 172.21.15.161:0/1671783387 >> v2:172.21.15.170:3300/0 conn(0x7f3c4406cdd0 0x7f3c441524c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:25.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.293+0000 7f3c4936d640 1 -- 172.21.15.161:0/1671783387 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c440c2f10 con 0x7f3c440c2690 2024-03-28T23:33:25.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.294+0000 7f3c4936d640 1 --2- 172.21.15.161:0/1671783387 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c440c2690 0x7f3c44152a00 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f3c440a9390 tx=0x7f3c3c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4da440db7ee40df server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:25.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.294+0000 7f3c437fe640 1 -- 172.21.15.161:0/1671783387 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c02ca70 con 0x7f3c440c2690 2024-03-28T23:33:25.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.294+0000 7f3c437fe640 1 -- 172.21.15.161:0/1671783387 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c3c02cc10 con 0x7f3c440c2690 2024-03-28T23:33:25.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.294+0000 7f3c437fe640 1 -- 172.21.15.161:0/1671783387 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c0343f0 con 0x7f3c440c2690 2024-03-28T23:33:25.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.295+0000 7f3c4add0640 1 -- 172.21.15.161:0/1671783387 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c440c2690 msgr2=0x7f3c44152a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:25.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.295+0000 7f3c4add0640 1 --2- 172.21.15.161:0/1671783387 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c440c2690 0x7f3c44152a00 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f3c440a9390 tx=0x7f3c3c02bed0 comp rx=0 tx=0).stop 2024-03-28T23:33:25.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.295+0000 7f3c4add0640 1 -- 172.21.15.161:0/1671783387 shutdown_connections 2024-03-28T23:33:25.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.295+0000 7f3c4add0640 1 --2- 172.21.15.161:0/1671783387 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c440c2690 0x7f3c44152a00 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:25.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.295+0000 7f3c4add0640 1 --2- 172.21.15.161:0/1671783387 >> v2:172.21.15.170:3300/0 conn(0x7f3c4406cdd0 0x7f3c441524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:25.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.295+0000 7f3c4add0640 1 --2- 172.21.15.161:0/1671783387 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c4414f5c0 0x7f3c440c2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:25.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.295+0000 7f3c4add0640 1 -- 172.21.15.161:0/1671783387 >> 172.21.15.161:0/1671783387 conn(0x7f3c440a83f0 msgr2=0x7f3c440a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:25.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.295+0000 7f3c4add0640 1 -- 172.21.15.161:0/1671783387 shutdown_connections 2024-03-28T23:33:25.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.295+0000 7f3c4add0640 1 -- 172.21.15.161:0/1671783387 wait complete. 2024-03-28T23:33:25.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.296+0000 7f3c4add0640 1 Processor -- start 2024-03-28T23:33:25.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.296+0000 7f3c4add0640 1 -- start start 2024-03-28T23:33:25.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.297+0000 7f3c4add0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c4406cdd0 0x7f3c44153830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:25.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.297+0000 7f3c4add0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c440c2690 0x7f3c44148630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:25.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.297+0000 7f3c4add0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c4414f5c0 0x7f3c44148b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:25.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.297+0000 7f3c4add0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3c4406c140 con 0x7f3c4406cdd0 2024-03-28T23:33:25.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.297+0000 7f3c4add0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3c4414f9a0 con 0x7f3c440c2690 2024-03-28T23:33:25.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.297+0000 7f3c4add0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3c4406c3e0 con 0x7f3c4414f5c0 2024-03-28T23:33:25.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.297+0000 7f3c48b6c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c4406cdd0 0x7f3c44153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:25.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.297+0000 7f3c48b6c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c4406cdd0 0x7f3c44153830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41120/0 (socket says 172.21.15.161:41120) 2024-03-28T23:33:25.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.297+0000 7f3c48b6c640 1 -- 172.21.15.161:0/993813306 learned_addr learned my addr 172.21.15.161:0/993813306 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:25.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.297+0000 7f3c4936d640 1 --2- 172.21.15.161:0/993813306 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c4414f5c0 0x7f3c44148b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:25.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.297+0000 7f3c43fff640 1 --2- 172.21.15.161:0/993813306 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c440c2690 0x7f3c44148630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:25.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.297+0000 7f3c48b6c640 1 -- 172.21.15.161:0/993813306 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c4414f5c0 msgr2=0x7f3c44148b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:25.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.297+0000 7f3c48b6c640 1 --2- 172.21.15.161:0/993813306 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c4414f5c0 0x7f3c44148b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:25.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.297+0000 7f3c48b6c640 1 -- 172.21.15.161:0/993813306 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c440c2690 msgr2=0x7f3c44148630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:25.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.297+0000 7f3c48b6c640 1 --2- 172.21.15.161:0/993813306 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c440c2690 0x7f3c44148630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:25.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.297+0000 7f3c48b6c640 1 -- 172.21.15.161:0/993813306 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c441db6f0 con 0x7f3c4406cdd0 2024-03-28T23:33:25.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.297+0000 7f3c48b6c640 1 --2- 172.21.15.161:0/993813306 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c4406cdd0 0x7f3c44153830 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f3c3400e870 tx=0x7f3c3400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:25.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.297+0000 7f3c417fa640 1 -- 172.21.15.161:0/993813306 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c34017070 con 0x7f3c4406cdd0 2024-03-28T23:33:25.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.297+0000 7f3c417fa640 1 -- 172.21.15.161:0/993813306 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c3400c940 con 0x7f3c4406cdd0 2024-03-28T23:33:25.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.298+0000 7f3c4add0640 1 -- 172.21.15.161:0/993813306 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c441542b0 con 0x7f3c4406cdd0 2024-03-28T23:33:25.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.298+0000 7f3c417fa640 1 -- 172.21.15.161:0/993813306 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3400cd30 con 0x7f3c4406cdd0 2024-03-28T23:33:25.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.298+0000 7f3c4add0640 1 -- 172.21.15.161:0/993813306 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c441e9b90 con 0x7f3c4406cdd0 2024-03-28T23:33:25.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.299+0000 7f3c4add0640 1 -- 172.21.15.161:0/993813306 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c4404f4b0 con 0x7f3c4406cdd0 2024-03-28T23:33:25.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.299+0000 7f3c417fa640 1 -- 172.21.15.161:0/993813306 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3c3401a070 con 0x7f3c4406cdd0 2024-03-28T23:33:25.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.299+0000 7f3c417fa640 1 --2- 172.21.15.161:0/993813306 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3c3006b4d0 0x7f3c3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:25.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.300+0000 7f3c43fff640 1 --2- 172.21.15.161:0/993813306 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3c3006b4d0 0x7f3c3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:25.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.300+0000 7f3c417fa640 1 -- 172.21.15.161:0/993813306 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6101+0+0 (secure 0 0 0) 0x7f3c3408a4d0 con 0x7f3c4406cdd0 2024-03-28T23:33:25.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.300+0000 7f3c43fff640 1 --2- 172.21.15.161:0/993813306 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3c3006b4d0 0x7f3c3006d990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3c44152cc0 tx=0x7f3c38009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:25.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.304+0000 7f3c417fa640 1 -- 172.21.15.161:0/993813306 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3c34058c00 con 0x7f3c4406cdd0 2024-03-28T23:33:25.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:25.513+0000 7f3c4add0640 1 -- 172.21.15.161:0/993813306 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f3c4406b830 con 0x7f3c4406cdd0 2024-03-28T23:33:25.798 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:33:25.801 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:33:26.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.184+0000 7f3c417fa640 1 -- 172.21.15.161:0/993813306 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v79) ==== 122+0+0 (secure 0 0 0) 0x7f3c3405cca0 con 0x7f3c4406cdd0 2024-03-28T23:33:26.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.185+0000 7f3c4add0640 1 -- 172.21.15.161:0/993813306 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3c3006b4d0 msgr2=0x7f3c3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:26.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.185+0000 7f3c4add0640 1 --2- 172.21.15.161:0/993813306 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3c3006b4d0 0x7f3c3006d990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3c44152cc0 tx=0x7f3c38009190 comp rx=0 tx=0).stop 2024-03-28T23:33:26.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.185+0000 7f3c4add0640 1 -- 172.21.15.161:0/993813306 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c4406cdd0 msgr2=0x7f3c44153830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:26.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.185+0000 7f3c4add0640 1 --2- 172.21.15.161:0/993813306 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c4406cdd0 0x7f3c44153830 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f3c3400e870 tx=0x7f3c3400ed30 comp rx=0 tx=0).stop 2024-03-28T23:33:26.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.186+0000 7f3c4add0640 1 -- 172.21.15.161:0/993813306 shutdown_connections 2024-03-28T23:33:26.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.186+0000 7f3c4add0640 1 --2- 172.21.15.161:0/993813306 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3c3006b4d0 0x7f3c3006d990 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.186+0000 7f3c4add0640 1 --2- 172.21.15.161:0/993813306 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c4414f5c0 0x7f3c44148b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.186+0000 7f3c4add0640 1 --2- 172.21.15.161:0/993813306 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c440c2690 0x7f3c44148630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.186+0000 7f3c4add0640 1 --2- 172.21.15.161:0/993813306 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c4406cdd0 0x7f3c44153830 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.186+0000 7f3c4add0640 1 -- 172.21.15.161:0/993813306 >> 172.21.15.161:0/993813306 conn(0x7f3c440a83f0 msgr2=0x7f3c440a94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:26.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.186+0000 7f3c4add0640 1 -- 172.21.15.161:0/993813306 shutdown_connections 2024-03-28T23:33:26.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.186+0000 7f3c4add0640 1 -- 172.21.15.161:0/993813306 wait complete. 2024-03-28T23:33:26.199 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-28T23:33:26.199 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:33:26.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.301+0000 7fd2f4ac2640 1 Processor -- start 2024-03-28T23:33:26.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.301+0000 7fd2f4ac2640 1 -- start start 2024-03-28T23:33:26.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.302+0000 7fd2f4ac2640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd2f00a8fc0 0x7fd2f0153270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:26.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.302+0000 7fd2f4ac2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2f00aacb0 0x7fd2f01537b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:26.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.302+0000 7fd2f4ac2640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2f01567c0 con 0x7fd2f0150600 2024-03-28T23:33:26.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.302+0000 7fd2f4ac2640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd2f0156570 con 0x7fd2f00a8fc0 2024-03-28T23:33:26.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.302+0000 7fd2f4ac2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd2f0156aa0 con 0x7fd2f00aacb0 2024-03-28T23:33:26.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.302+0000 7fd2edd9b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd2f00a8fc0 0x7fd2f0153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:26.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.302+0000 7fd2eed9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2f00aacb0 0x7fd2f01537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:26.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.302+0000 7fd2edd9b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd2f00a8fc0 0x7fd2f0153270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:37862/0 (socket says 172.21.15.161:37862) 2024-03-28T23:33:26.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.302+0000 7fd2edd9b640 1 -- 172.21.15.161:0/3323348525 learned_addr learned my addr 172.21.15.161:0/3323348525 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:26.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.302+0000 7fd2edd9b640 1 -- 172.21.15.161:0/3323348525 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2f00aacb0 msgr2=0x7fd2f01537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:26.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.302+0000 7fd2edd9b640 1 --2- 172.21.15.161:0/3323348525 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2f00aacb0 0x7fd2f01537b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.302+0000 7fd2edd9b640 1 -- 172.21.15.161:0/3323348525 >> v1:172.21.15.170:6789/0 conn(0x7fd2f0150600 legacy=0x7fd2f00a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:26.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.302+0000 7fd2edd9b640 1 -- 172.21.15.161:0/3323348525 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2f0156d80 con 0x7fd2f00a8fc0 2024-03-28T23:33:26.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.302+0000 7fd2eed9d640 1 --2- 172.21.15.161:0/3323348525 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2f00aacb0 0x7fd2f01537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:26.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.303+0000 7fd2edd9b640 1 --2- 172.21.15.161:0/3323348525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2f00a8fc0 0x7fd2f0153270 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4009870 tx=0x7fd2e402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1273b2ef8a029c4f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:26.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.303+0000 7fd2ed59a640 1 -- 172.21.15.161:0/3323348525 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4035020 con 0x7fd2f00a8fc0 2024-03-28T23:33:26.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.303+0000 7fd2ed59a640 1 -- 172.21.15.161:0/3323348525 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2e4031640 con 0x7fd2f00a8fc0 2024-03-28T23:33:26.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.303+0000 7fd2ed59a640 1 -- 172.21.15.161:0/3323348525 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e402d040 con 0x7fd2f00a8fc0 2024-03-28T23:33:26.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.304+0000 7fd2f4ac2640 1 -- 172.21.15.161:0/3323348525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2f00a8fc0 msgr2=0x7fd2f0153270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:26.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.304+0000 7fd2f4ac2640 1 --2- 172.21.15.161:0/3323348525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2f00a8fc0 0x7fd2f0153270 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4009870 tx=0x7fd2e402b8f0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.304+0000 7fd2f4ac2640 1 -- 172.21.15.161:0/3323348525 shutdown_connections 2024-03-28T23:33:26.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.304+0000 7fd2f4ac2640 1 --2- 172.21.15.161:0/3323348525 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2f00aacb0 0x7fd2f01537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.304+0000 7fd2f4ac2640 1 --2- 172.21.15.161:0/3323348525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2f00a8fc0 0x7fd2f0153270 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.304+0000 7fd2f4ac2640 1 -- 172.21.15.161:0/3323348525 >> 172.21.15.161:0/3323348525 conn(0x7fd2f00c6670 msgr2=0x7fd2f00c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:26.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.304+0000 7fd2f4ac2640 1 -- 172.21.15.161:0/3323348525 shutdown_connections 2024-03-28T23:33:26.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.304+0000 7fd2f4ac2640 1 -- 172.21.15.161:0/3323348525 wait complete. 2024-03-28T23:33:26.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.305+0000 7fd2f4ac2640 1 Processor -- start 2024-03-28T23:33:26.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.305+0000 7fd2f4ac2640 1 -- start start 2024-03-28T23:33:26.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.306+0000 7fd2f4ac2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2f00a8fc0 0x7fd2f0147e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:26.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.306+0000 7fd2f4ac2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2f00aacb0 0x7fd2f01e8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:26.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.306+0000 7fd2f4ac2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2f0150600 0x7fd2f01e91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:26.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.306+0000 7fd2f4ac2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f01509e0 con 0x7fd2f00aacb0 2024-03-28T23:33:26.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.306+0000 7fd2f4ac2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f0157340 con 0x7fd2f0150600 2024-03-28T23:33:26.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.306+0000 7fd2f4ac2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd2f00aa9e0 con 0x7fd2f00a8fc0 2024-03-28T23:33:26.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.306+0000 7fd2edd9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2f00aacb0 0x7fd2f01e8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:26.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.306+0000 7fd2edd9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2f00aacb0 0x7fd2f01e8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41124/0 (socket says 172.21.15.161:41124) 2024-03-28T23:33:26.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.306+0000 7fd2edd9b640 1 -- 172.21.15.161:0/3575417698 learned_addr learned my addr 172.21.15.161:0/3575417698 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:26.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.306+0000 7fd2ee59c640 1 --2- 172.21.15.161:0/3575417698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2f00a8fc0 0x7fd2f0147e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:26.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.306+0000 7fd2eed9d640 1 --2- 172.21.15.161:0/3575417698 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2f0150600 0x7fd2f01e91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:26.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.306+0000 7fd2edd9b640 1 -- 172.21.15.161:0/3575417698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2f00a8fc0 msgr2=0x7fd2f0147e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:26.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.306+0000 7fd2edd9b640 1 --2- 172.21.15.161:0/3575417698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2f00a8fc0 0x7fd2f0147e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.306+0000 7fd2edd9b640 1 -- 172.21.15.161:0/3575417698 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2f0150600 msgr2=0x7fd2f01e91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:26.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.306+0000 7fd2edd9b640 1 --2- 172.21.15.161:0/3575417698 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2f0150600 0x7fd2f01e91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.306+0000 7fd2edd9b640 1 -- 172.21.15.161:0/3575417698 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2f01ec880 con 0x7fd2f00aacb0 2024-03-28T23:33:26.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.306+0000 7fd2ee59c640 1 --2- 172.21.15.161:0/3575417698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2f00a8fc0 0x7fd2f0147e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:26.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.306+0000 7fd2edd9b640 1 --2- 172.21.15.161:0/3575417698 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2f00aacb0 0x7fd2f01e8ca0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fd2e402b930 tx=0x7fd2e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:26.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.306+0000 7fd2deffd640 1 -- 172.21.15.161:0/3575417698 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4040070 con 0x7fd2f00aacb0 2024-03-28T23:33:26.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.306+0000 7fd2deffd640 1 -- 172.21.15.161:0/3575417698 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2e403e070 con 0x7fd2f00aacb0 2024-03-28T23:33:26.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.306+0000 7fd2f4ac2640 1 -- 172.21.15.161:0/3575417698 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2f01ecb20 con 0x7fd2f00aacb0 2024-03-28T23:33:26.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.306+0000 7fd2deffd640 1 -- 172.21.15.161:0/3575417698 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4035040 con 0x7fd2f00aacb0 2024-03-28T23:33:26.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.306+0000 7fd2f4ac2640 1 -- 172.21.15.161:0/3575417698 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2f00c1580 con 0x7fd2f00aacb0 2024-03-28T23:33:26.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.308+0000 7fd2f4ac2640 1 -- 172.21.15.161:0/3575417698 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2f01e8980 con 0x7fd2f00aacb0 2024-03-28T23:33:26.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.308+0000 7fd2deffd640 1 -- 172.21.15.161:0/3575417698 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd2e4038020 con 0x7fd2f00aacb0 2024-03-28T23:33:26.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.308+0000 7fd2deffd640 1 --2- 172.21.15.161:0/3575417698 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd2d006b4d0 0x7fd2d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:26.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.308+0000 7fd2ee59c640 1 --2- 172.21.15.161:0/3575417698 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd2d006b4d0 0x7fd2d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:26.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.308+0000 7fd2deffd640 1 -- 172.21.15.161:0/3575417698 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6101+0+0 (secure 0 0 0) 0x7fd2e40aba20 con 0x7fd2f00aacb0 2024-03-28T23:33:26.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.309+0000 7fd2ee59c640 1 --2- 172.21.15.161:0/3575417698 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd2d006b4d0 0x7fd2d006d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fd2f014bcd0 tx=0x7fd2e0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:26.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.313+0000 7fd2deffd640 1 -- 172.21.15.161:0/3575417698 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd2f01e8980 con 0x7fd2f00aacb0 2024-03-28T23:33:26.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.519+0000 7fd2f4ac2640 1 -- 172.21.15.161:0/3575417698 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2f004f4b0 con 0x7fd2f00aacb0 2024-03-28T23:33:26.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.520+0000 7fd2deffd640 1 -- 172.21.15.161:0/3575417698 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+5192 (secure 0 0 0) 0x7fd2e407a7a0 con 0x7fd2f00aacb0 2024-03-28T23:33:26.522 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:26.522 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":79,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5963,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3079114491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3079114491},{"type":"v1","addr":"172.21.15.170:6835","nonce":3079114491}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":6547,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4280217233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4280217233},{"type":"v1","addr":"172.21.15.161:6836","nonce":4280217233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":7086,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/404132405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":404132405},{"type":"v1","addr":"172.21.15.161:6837","nonce":404132405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:33:23.516182+0000","modified":"2024-03-28T23:33:25.515642+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7101},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7101":{"gid":7101,"name":"d","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6834/2176248563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":2176248563},{"type":"v1","addr":"172.21.15.170:6834","nonce":2176248563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7101,"qdb_cluster":[7101]},"id":7}]} 2024-03-28T23:33:26.522 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 79 2024-03-28T23:33:26.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.522+0000 7fd2f4ac2640 1 -- 172.21.15.161:0/3575417698 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd2d006b4d0 msgr2=0x7fd2d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:26.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.522+0000 7fd2f4ac2640 1 --2- 172.21.15.161:0/3575417698 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd2d006b4d0 0x7fd2d006d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fd2f014bcd0 tx=0x7fd2e0009040 comp rx=0 tx=0).stop 2024-03-28T23:33:26.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.522+0000 7fd2f4ac2640 1 -- 172.21.15.161:0/3575417698 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2f00aacb0 msgr2=0x7fd2f01e8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:26.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.522+0000 7fd2f4ac2640 1 --2- 172.21.15.161:0/3575417698 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2f00aacb0 0x7fd2f01e8ca0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fd2e402b930 tx=0x7fd2e40046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.522+0000 7fd2f4ac2640 1 -- 172.21.15.161:0/3575417698 shutdown_connections 2024-03-28T23:33:26.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.522+0000 7fd2f4ac2640 1 --2- 172.21.15.161:0/3575417698 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd2d006b4d0 0x7fd2d006d990 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.522+0000 7fd2f4ac2640 1 --2- 172.21.15.161:0/3575417698 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2f00aacb0 0x7fd2f01e8ca0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.522+0000 7fd2f4ac2640 1 --2- 172.21.15.161:0/3575417698 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2f0150600 0x7fd2f01e91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.522+0000 7fd2f4ac2640 1 --2- 172.21.15.161:0/3575417698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2f00a8fc0 0x7fd2f0147e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.522+0000 7fd2f4ac2640 1 -- 172.21.15.161:0/3575417698 >> 172.21.15.161:0/3575417698 conn(0x7fd2f00c6670 msgr2=0x7fd2f0151f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:26.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.522+0000 7fd2f4ac2640 1 -- 172.21.15.161:0/3575417698 shutdown_connections 2024-03-28T23:33:26.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.522+0000 7fd2f4ac2640 1 -- 172.21.15.161:0/3575417698 wait complete. 2024-03-28T23:33:26.535 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:33:26.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.633+0000 7f78c9fbc640 1 Processor -- start 2024-03-28T23:33:26.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.633+0000 7f78c9fbc640 1 -- start start 2024-03-28T23:33:26.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.634+0000 7f78c9fbc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78c4152890 0x7f78c40c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:26.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.634+0000 7f78c9fbc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f78c40c8ed0 0x7f78c40c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:26.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.634+0000 7f78c9fbc640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78c4158a30 con 0x7f78c40c5950 2024-03-28T23:33:26.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.634+0000 7f78c9fbc640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f78c40c5db0 con 0x7f78c40c8ed0 2024-03-28T23:33:26.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.634+0000 7f78c9fbc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f78c40c8480 con 0x7f78c4152890 2024-03-28T23:33:26.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.634+0000 7f78baffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f78c40c8ed0 0x7f78c40c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:26.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.634+0000 7f78baffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f78c40c8ed0 0x7f78c40c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41128/0 (socket says 172.21.15.161:41128) 2024-03-28T23:33:26.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.634+0000 7f78baffd640 1 -- 172.21.15.161:0/1626553304 learned_addr learned my addr 172.21.15.161:0/1626553304 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:26.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.634+0000 7f78c37fe640 1 --2- 172.21.15.161:0/1626553304 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78c4152890 0x7f78c40c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:26.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.635+0000 7f78baffd640 1 -- 172.21.15.161:0/1626553304 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78c4152890 msgr2=0x7f78c40c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:26.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.635+0000 7f78baffd640 1 --2- 172.21.15.161:0/1626553304 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78c4152890 0x7f78c40c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.635+0000 7f78baffd640 1 -- 172.21.15.161:0/1626553304 >> v1:172.21.15.170:6789/0 conn(0x7f78c40c5950 legacy=0x7f78c41553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:26.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.635+0000 7f78baffd640 1 -- 172.21.15.161:0/1626553304 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c4158d10 con 0x7f78c40c8ed0 2024-03-28T23:33:26.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.635+0000 7f78baffd640 1 --2- 172.21.15.161:0/1626553304 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f78c40c8ed0 0x7f78c40c5410 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f78ac009850 tx=0x7f78ac02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5970e5c2a05f1a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:26.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.635+0000 7f78c2ffd640 1 -- 172.21.15.161:0/1626553304 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ac037070 con 0x7f78c40c8ed0 2024-03-28T23:33:26.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.636+0000 7f78c2ffd640 1 -- 172.21.15.161:0/1626553304 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78ac004030 con 0x7f78c40c8ed0 2024-03-28T23:33:26.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.636+0000 7f78c2ffd640 1 -- 172.21.15.161:0/1626553304 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ac004460 con 0x7f78c40c8ed0 2024-03-28T23:33:26.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.636+0000 7f78c9fbc640 1 -- 172.21.15.161:0/1626553304 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f78c40c8ed0 msgr2=0x7f78c40c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:26.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.636+0000 7f78c9fbc640 1 --2- 172.21.15.161:0/1626553304 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f78c40c8ed0 0x7f78c40c5410 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f78ac009850 tx=0x7f78ac02b840 comp rx=0 tx=0).stop 2024-03-28T23:33:26.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.636+0000 7f78c9fbc640 1 -- 172.21.15.161:0/1626553304 shutdown_connections 2024-03-28T23:33:26.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.636+0000 7f78c9fbc640 1 --2- 172.21.15.161:0/1626553304 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f78c40c8ed0 0x7f78c40c5410 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.636+0000 7f78c9fbc640 1 --2- 172.21.15.161:0/1626553304 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78c4152890 0x7f78c40c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.636+0000 7f78c9fbc640 1 -- 172.21.15.161:0/1626553304 >> 172.21.15.161:0/1626553304 conn(0x7f78c40ac620 msgr2=0x7f78c40aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:26.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.636+0000 7f78c9fbc640 1 -- 172.21.15.161:0/1626553304 shutdown_connections 2024-03-28T23:33:26.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.636+0000 7f78c9fbc640 1 -- 172.21.15.161:0/1626553304 wait complete. 2024-03-28T23:33:26.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.637+0000 7f78c9fbc640 1 Processor -- start 2024-03-28T23:33:26.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.637+0000 7f78c9fbc640 1 -- start start 2024-03-28T23:33:26.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.638+0000 7f78c9fbc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78c40c5950 0x7f78c4157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:26.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.638+0000 7f78c9fbc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78c40c8ed0 0x7f78c40c0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:26.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.638+0000 7f78c9fbc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f78c4152890 0x7f78c40c0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:26.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.638+0000 7f78c9fbc640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f78c41592d0 con 0x7f78c4152890 2024-03-28T23:33:26.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.638+0000 7f78c9fbc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f78c4159640 con 0x7f78c40c5950 2024-03-28T23:33:26.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.638+0000 7f78c9fbc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f78c4152c70 con 0x7f78c40c8ed0 2024-03-28T23:33:26.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.638+0000 7f78c3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f78c4152890 0x7f78c40c0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:26.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.638+0000 7f78c3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f78c4152890 0x7f78c40c0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41132/0 (socket says 172.21.15.161:41132) 2024-03-28T23:33:26.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.638+0000 7f78c3fff640 1 -- 172.21.15.161:0/169893516 learned_addr learned my addr 172.21.15.161:0/169893516 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:26.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.638+0000 7f78c37fe640 1 --2- 172.21.15.161:0/169893516 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78c40c5950 0x7f78c4157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:26.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.638+0000 7f78baffd640 1 --2- 172.21.15.161:0/169893516 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78c40c8ed0 0x7f78c40c0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:26.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.638+0000 7f78c3fff640 1 -- 172.21.15.161:0/169893516 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78c40c8ed0 msgr2=0x7f78c40c0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:26.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.638+0000 7f78c3fff640 1 --2- 172.21.15.161:0/169893516 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78c40c8ed0 0x7f78c40c0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.638+0000 7f78c3fff640 1 -- 172.21.15.161:0/169893516 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78c40c5950 msgr2=0x7f78c4157f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:26.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.638+0000 7f78c3fff640 1 --2- 172.21.15.161:0/169893516 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78c40c5950 0x7f78c4157f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.638+0000 7f78c3fff640 1 -- 172.21.15.161:0/169893516 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c40c1210 con 0x7f78c4152890 2024-03-28T23:33:26.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.638+0000 7f78baffd640 1 --2- 172.21.15.161:0/169893516 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78c40c8ed0 0x7f78c40c0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:26.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.638+0000 7f78c37fe640 1 --2- 172.21.15.161:0/169893516 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78c40c5950 0x7f78c4157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:26.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.639+0000 7f78c3fff640 1 --2- 172.21.15.161:0/169893516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f78c4152890 0x7f78c40c0cd0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f78bc00c6e0 tx=0x7f78bc00cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:26.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.639+0000 7f78c0ff9640 1 -- 172.21.15.161:0/169893516 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78bc003b70 con 0x7f78c4152890 2024-03-28T23:33:26.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.639+0000 7f78c0ff9640 1 -- 172.21.15.161:0/169893516 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78bc003d30 con 0x7f78c4152890 2024-03-28T23:33:26.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.639+0000 7f78c0ff9640 1 -- 172.21.15.161:0/169893516 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78bc005310 con 0x7f78c4152890 2024-03-28T23:33:26.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.639+0000 7f78c9fbc640 1 -- 172.21.15.161:0/169893516 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78c41482a0 con 0x7f78c4152890 2024-03-28T23:33:26.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.639+0000 7f78c9fbc640 1 -- 172.21.15.161:0/169893516 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78c414f280 con 0x7f78c4152890 2024-03-28T23:33:26.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.640+0000 7f78c9fbc640 1 -- 172.21.15.161:0/169893516 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78c414f070 con 0x7f78c4152890 2024-03-28T23:33:26.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.640+0000 7f78c0ff9640 1 -- 172.21.15.161:0/169893516 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f78bc0054b0 con 0x7f78c4152890 2024-03-28T23:33:26.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.641+0000 7f78c0ff9640 1 --2- 172.21.15.161:0/169893516 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f78a406b4d0 0x7f78a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:26.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.641+0000 7f78c37fe640 1 --2- 172.21.15.161:0/169893516 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f78a406b4d0 0x7f78a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:26.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.641+0000 7f78c0ff9640 1 -- 172.21.15.161:0/169893516 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6101+0+0 (secure 0 0 0) 0x7f78bc08a0e0 con 0x7f78c4152890 2024-03-28T23:33:26.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.641+0000 7f78c37fe640 1 --2- 172.21.15.161:0/169893516 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f78a406b4d0 0x7f78a406d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009f30 tx=0x7f78b4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:26.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.645+0000 7f78c0ff9640 1 -- 172.21.15.161:0/169893516 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f78c41482a0 con 0x7f78c4152890 2024-03-28T23:33:26.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.819+0000 7f78c9fbc640 1 -- 172.21.15.161:0/169893516 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f78c40c8cb0 con 0x7f78c4152890 2024-03-28T23:33:26.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.820+0000 7f78c0ff9640 1 -- 172.21.15.161:0/169893516 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+16909 (secure 0 0 0) 0x7f78bc058760 con 0x7f78c4152890 2024-03-28T23:33:26.821 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:26.822 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":94,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:33:25.158028+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:33:21.732337+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-28T23:33:22.477357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:33:26.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.822+0000 7f78c9fbc640 1 -- 172.21.15.161:0/169893516 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f78a406b4d0 msgr2=0x7f78a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:26.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.822+0000 7f78c9fbc640 1 --2- 172.21.15.161:0/169893516 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f78a406b4d0 0x7f78a406d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009f30 tx=0x7f78b4009190 comp rx=0 tx=0).stop 2024-03-28T23:33:26.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.822+0000 7f78c9fbc640 1 -- 172.21.15.161:0/169893516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f78c4152890 msgr2=0x7f78c40c0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:26.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.822+0000 7f78c9fbc640 1 --2- 172.21.15.161:0/169893516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f78c4152890 0x7f78c40c0cd0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f78bc00c6e0 tx=0x7f78bc00cba0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.822+0000 7f78c9fbc640 1 -- 172.21.15.161:0/169893516 shutdown_connections 2024-03-28T23:33:26.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.822+0000 7f78c9fbc640 1 --2- 172.21.15.161:0/169893516 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f78a406b4d0 0x7f78a406d990 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.822+0000 7f78c9fbc640 1 --2- 172.21.15.161:0/169893516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f78c4152890 0x7f78c40c0cd0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.822+0000 7f78c9fbc640 1 --2- 172.21.15.161:0/169893516 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78c40c8ed0 0x7f78c40c0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.822+0000 7f78c9fbc640 1 --2- 172.21.15.161:0/169893516 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78c40c5950 0x7f78c4157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.822+0000 7f78c9fbc640 1 -- 172.21.15.161:0/169893516 >> 172.21.15.161:0/169893516 conn(0x7f78c40ac620 msgr2=0x7f78c414a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:26.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.822+0000 7f78c9fbc640 1 -- 172.21.15.161:0/169893516 shutdown_connections 2024-03-28T23:33:26.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.822+0000 7f78c9fbc640 1 -- 172.21.15.161:0/169893516 wait complete. 2024-03-28T23:33:26.835 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T23:33:26.835 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:33:26.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.934+0000 7fc7e3bd0640 1 Processor -- start 2024-03-28T23:33:26.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.934+0000 7fc7e3bd0640 1 -- start start 2024-03-28T23:33:26.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.935+0000 7fc7e3bd0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc7dc06ce80 0x7fc7dc0aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:26.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.935+0000 7fc7e3bd0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7dc14c250 con 0x7fc7dc0a9680 2024-03-28T23:33:26.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.935+0000 7fc7e3bd0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7dc1529c0 con 0x7fc7dc14be70 2024-03-28T23:33:26.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.935+0000 7fc7e3bd0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc7dc0a9dc0 con 0x7fc7dc06ce80 2024-03-28T23:33:26.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.935+0000 7fc7e116b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc7dc06ce80 0x7fc7dc0aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:26.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.935+0000 7fc7e216d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc7dc0a9680 0x7fc7dc15c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:55336/0 (socket says 172.21.15.161:55336) 2024-03-28T23:33:26.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.935+0000 7fc7e116b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc7dc06ce80 0x7fc7dc0aa5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41140/0 (socket says 172.21.15.161:41140) 2024-03-28T23:33:26.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.935+0000 7fc7e216d640 1 -- 172.21.15.161:0/1133064095 learned_addr learned my addr 172.21.15.161:0/1133064095 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:26.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.935+0000 7fc7e116b640 1 -- 172.21.15.161:0/1133064095 >> v1:172.21.15.170:6789/0 conn(0x7fc7dc14be70 legacy=0x7fc7dc0ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:26.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.935+0000 7fc7e096a640 1 -- 172.21.15.161:0/1133064095 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2081840362 0 0) 0x7fc7dc1529c0 con 0x7fc7dc14be70 2024-03-28T23:33:26.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.935+0000 7fc7e116b640 1 -- 172.21.15.161:0/1133064095 >> v1:172.21.15.161:6789/0 conn(0x7fc7dc0a9680 legacy=0x7fc7dc15c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:26.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.935+0000 7fc7e116b640 1 -- 172.21.15.161:0/1133064095 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7dc0aa0a0 con 0x7fc7dc06ce80 2024-03-28T23:33:26.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.936+0000 7fc7e116b640 1 --2- 172.21.15.161:0/1133064095 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7dc06ce80 0x7fc7dc0aa5c0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc0097a0 tx=0x7fc7cc02b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eecbfaf7686884 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:26.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.936+0000 7fc7e096a640 1 -- 172.21.15.161:0/1133064095 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc037070 con 0x7fc7dc06ce80 2024-03-28T23:33:26.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.936+0000 7fc7e096a640 1 -- 172.21.15.161:0/1133064095 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7cc030710 con 0x7fc7dc06ce80 2024-03-28T23:33:26.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.936+0000 7fc7e096a640 1 -- 172.21.15.161:0/1133064095 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc030ae0 con 0x7fc7dc06ce80 2024-03-28T23:33:26.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.937+0000 7fc7e3bd0640 1 -- 172.21.15.161:0/1133064095 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7dc06ce80 msgr2=0x7fc7dc0aa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:26.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.937+0000 7fc7e3bd0640 1 --2- 172.21.15.161:0/1133064095 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7dc06ce80 0x7fc7dc0aa5c0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc0097a0 tx=0x7fc7cc02b710 comp rx=0 tx=0).stop 2024-03-28T23:33:26.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.937+0000 7fc7e3bd0640 1 -- 172.21.15.161:0/1133064095 shutdown_connections 2024-03-28T23:33:26.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.937+0000 7fc7e3bd0640 1 --2- 172.21.15.161:0/1133064095 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7dc06ce80 0x7fc7dc0aa5c0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.937+0000 7fc7e3bd0640 1 -- 172.21.15.161:0/1133064095 >> 172.21.15.161:0/1133064095 conn(0x7fc7dc0c6670 msgr2=0x7fc7dc0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:26.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.937+0000 7fc7e3bd0640 1 -- 172.21.15.161:0/1133064095 shutdown_connections 2024-03-28T23:33:26.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.937+0000 7fc7e3bd0640 1 -- 172.21.15.161:0/1133064095 wait complete. 2024-03-28T23:33:26.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.938+0000 7fc7e3bd0640 1 Processor -- start 2024-03-28T23:33:26.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.938+0000 7fc7e3bd0640 1 -- start start 2024-03-28T23:33:26.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.938+0000 7fc7e3bd0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7dc06ce80 0x7fc7dc1deda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:26.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.938+0000 7fc7e3bd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7dc0a9680 0x7fc7dc148810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:26.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.938+0000 7fc7e3bd0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7dc14be70 0x7fc7dc148d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:26.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.938+0000 7fc7e3bd0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc7dc0aba30 con 0x7fc7dc06ce80 2024-03-28T23:33:26.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.938+0000 7fc7e3bd0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc7dc0ab8b0 con 0x7fc7dc14be70 2024-03-28T23:33:26.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.938+0000 7fc7e3bd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc7dc06b150 con 0x7fc7dc0a9680 2024-03-28T23:33:26.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.939+0000 7fc7e196c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7dc06ce80 0x7fc7dc1deda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:26.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.939+0000 7fc7e196c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7dc06ce80 0x7fc7dc1deda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41156/0 (socket says 172.21.15.161:41156) 2024-03-28T23:33:26.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.939+0000 7fc7e196c640 1 -- 172.21.15.161:0/2798778452 learned_addr learned my addr 172.21.15.161:0/2798778452 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:26.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.939+0000 7fc7e116b640 1 --2- 172.21.15.161:0/2798778452 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7dc0a9680 0x7fc7dc148810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:26.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.939+0000 7fc7e216d640 1 --2- 172.21.15.161:0/2798778452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7dc14be70 0x7fc7dc148d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:26.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.939+0000 7fc7e196c640 1 -- 172.21.15.161:0/2798778452 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7dc0a9680 msgr2=0x7fc7dc148810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:26.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.939+0000 7fc7e196c640 1 --2- 172.21.15.161:0/2798778452 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7dc0a9680 0x7fc7dc148810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.939+0000 7fc7e196c640 1 -- 172.21.15.161:0/2798778452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7dc14be70 msgr2=0x7fc7dc148d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:26.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.939+0000 7fc7e196c640 1 --2- 172.21.15.161:0/2798778452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7dc14be70 0x7fc7dc148d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:26.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.939+0000 7fc7e196c640 1 -- 172.21.15.161:0/2798778452 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7dc149290 con 0x7fc7dc06ce80 2024-03-28T23:33:26.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.939+0000 7fc7e196c640 1 --2- 172.21.15.161:0/2798778452 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7dc06ce80 0x7fc7dc1deda0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fc7d800c7f0 tx=0x7fc7d800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:26.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.939+0000 7fc7d27fc640 1 -- 172.21.15.161:0/2798778452 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d8017070 con 0x7fc7dc06ce80 2024-03-28T23:33:26.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.939+0000 7fc7d27fc640 1 -- 172.21.15.161:0/2798778452 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7d8003da0 con 0x7fc7dc06ce80 2024-03-28T23:33:26.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.939+0000 7fc7d27fc640 1 -- 172.21.15.161:0/2798778452 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d8005290 con 0x7fc7dc06ce80 2024-03-28T23:33:26.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.939+0000 7fc7e3bd0640 1 -- 172.21.15.161:0/2798778452 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7dc1e8bd0 con 0x7fc7dc06ce80 2024-03-28T23:33:26.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.939+0000 7fc7e3bd0640 1 -- 172.21.15.161:0/2798778452 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7dc1e9190 con 0x7fc7dc06ce80 2024-03-28T23:33:26.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.940+0000 7fc7e3bd0640 1 -- 172.21.15.161:0/2798778452 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7dc04f4b0 con 0x7fc7dc06ce80 2024-03-28T23:33:26.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.941+0000 7fc7d27fc640 1 -- 172.21.15.161:0/2798778452 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc7d801a070 con 0x7fc7dc06ce80 2024-03-28T23:33:26.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.941+0000 7fc7d27fc640 1 --2- 172.21.15.161:0/2798778452 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc7c006b500 0x7fc7c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:26.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.941+0000 7fc7d27fc640 1 -- 172.21.15.161:0/2798778452 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6101+0+0 (secure 0 0 0) 0x7fc7d8089920 con 0x7fc7dc06ce80 2024-03-28T23:33:26.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.941+0000 7fc7e116b640 1 --2- 172.21.15.161:0/2798778452 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc7c006b500 0x7fc7c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:26.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.941+0000 7fc7e116b640 1 --2- 172.21.15.161:0/2798778452 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc7c006b500 0x7fc7c006d9c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc034000 tx=0x7fc7cc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:26.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:26.945+0000 7fc7d27fc640 1 -- 172.21.15.161:0/2798778452 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc7d8058050 con 0x7fc7dc06ce80 2024-03-28T23:33:27.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.115+0000 7fc7e3bd0640 1 -- 172.21.15.161:0/2798778452 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc7dc0c2770 con 0x7fc7c006b500 2024-03-28T23:33:27.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.120+0000 7fc7d27fc640 1 -- 172.21.15.161:0/2798778452 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252090 (secure 0 0 0) 0x7fc7dc0c2770 con 0x7fc7c006b500 2024-03-28T23:33:27.125 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:27.126 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:33:27.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.123+0000 7fc7e3bd0640 1 -- 172.21.15.161:0/2798778452 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc7c006b500 msgr2=0x7fc7c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:27.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.124+0000 7fc7e3bd0640 1 --2- 172.21.15.161:0/2798778452 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc7c006b500 0x7fc7c006d9c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc034000 tx=0x7fc7cc008030 comp rx=0 tx=0).stop 2024-03-28T23:33:27.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.124+0000 7fc7e3bd0640 1 -- 172.21.15.161:0/2798778452 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7dc06ce80 msgr2=0x7fc7dc1deda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:27.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.124+0000 7fc7e3bd0640 1 --2- 172.21.15.161:0/2798778452 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7dc06ce80 0x7fc7dc1deda0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fc7d800c7f0 tx=0x7fc7d800ccb0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.124+0000 7fc7e3bd0640 1 -- 172.21.15.161:0/2798778452 shutdown_connections 2024-03-28T23:33:27.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.124+0000 7fc7e3bd0640 1 --2- 172.21.15.161:0/2798778452 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc7c006b500 0x7fc7c006d9c0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.124+0000 7fc7e3bd0640 1 --2- 172.21.15.161:0/2798778452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7dc14be70 0x7fc7dc148d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.124+0000 7fc7e3bd0640 1 --2- 172.21.15.161:0/2798778452 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7dc0a9680 0x7fc7dc148810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.124+0000 7fc7e3bd0640 1 --2- 172.21.15.161:0/2798778452 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7dc06ce80 0x7fc7dc1deda0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.124+0000 7fc7e3bd0640 1 -- 172.21.15.161:0/2798778452 >> 172.21.15.161:0/2798778452 conn(0x7fc7dc0c6670 msgr2=0x7fc7dc14b140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:27.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.124+0000 7fc7e3bd0640 1 -- 172.21.15.161:0/2798778452 shutdown_connections 2024-03-28T23:33:27.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.124+0000 7fc7e3bd0640 1 -- 172.21.15.161:0/2798778452 wait complete. 2024-03-28T23:33:27.136 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-03-28T23:33:25.353240+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":246056,"kb_used_data":9084,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754728664,"statfs":{"total":773094113280,"available":772842151936,"internally_reserved":0,"allocated":9302016,"data_stored":5582229,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000695"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.184247+0000","last_change":"2024-03-28T23:33:24.181088+0000","last_active":"2024-03-28T23:33:24.184247+0000","last_peered":"2024-03-28T23:33:24.184247+0000","last_clean":"2024-03-28T23:33:24.184247+0000","last_became_active":"2024-03-28T23:33:24.180815+0000","last_became_peered":"2024-03-28T23:33:24.180815+0000","last_unstale":"2024-03-28T23:33:24.184247+0000","last_undegraded":"2024-03-28T23:33:24.184247+0000","last_fullsized":"2024-03-28T23:33:24.184247+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:16:14.029745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.175254+0000","last_change":"2024-03-28T23:33:24.173050+0000","last_active":"2024-03-28T23:33:24.175254+0000","last_peered":"2024-03-28T23:33:24.175254+0000","last_clean":"2024-03-28T23:33:24.175254+0000","last_became_active":"2024-03-28T23:33:24.172761+0000","last_became_peered":"2024-03-28T23:33:24.172761+0000","last_unstale":"2024-03-28T23:33:24.175254+0000","last_undegraded":"2024-03-28T23:33:24.175254+0000","last_fullsized":"2024-03-28T23:33:24.175254+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:56:05.296197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.190747+0000","last_change":"2024-03-28T23:33:24.185523+0000","last_active":"2024-03-28T23:33:24.190747+0000","last_peered":"2024-03-28T23:33:24.190747+0000","last_clean":"2024-03-28T23:33:24.190747+0000","last_became_active":"2024-03-28T23:33:24.184535+0000","last_became_peered":"2024-03-28T23:33:24.184535+0000","last_unstale":"2024-03-28T23:33:24.190747+0000","last_undegraded":"2024-03-28T23:33:24.190747+0000","last_fullsized":"2024-03-28T23:33:24.190747+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:35:39.480945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.179924+0000","last_change":"2024-03-28T23:33:24.177488+0000","last_active":"2024-03-28T23:33:24.179924+0000","last_peered":"2024-03-28T23:33:24.179924+0000","last_clean":"2024-03-28T23:33:24.179924+0000","last_became_active":"2024-03-28T23:33:24.177216+0000","last_became_peered":"2024-03-28T23:33:24.177216+0000","last_unstale":"2024-03-28T23:33:24.179924+0000","last_undegraded":"2024-03-28T23:33:24.179924+0000","last_fullsized":"2024-03-28T23:33:24.179924+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:35:30.544190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.189832+0000","last_change":"2024-03-28T23:33:24.176392+0000","last_active":"2024-03-28T23:33:24.189832+0000","last_peered":"2024-03-28T23:33:24.189832+0000","last_clean":"2024-03-28T23:33:24.189832+0000","last_became_active":"2024-03-28T23:33:24.176017+0000","last_became_peered":"2024-03-28T23:33:24.176017+0000","last_unstale":"2024-03-28T23:33:24.189832+0000","last_undegraded":"2024-03-28T23:33:24.189832+0000","last_fullsized":"2024-03-28T23:33:24.189832+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:22.409061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.182577+0000","last_change":"2024-03-28T23:33:24.179934+0000","last_active":"2024-03-28T23:33:24.182577+0000","last_peered":"2024-03-28T23:33:24.182577+0000","last_clean":"2024-03-28T23:33:24.182577+0000","last_became_active":"2024-03-28T23:33:24.179327+0000","last_became_peered":"2024-03-28T23:33:24.179327+0000","last_unstale":"2024-03-28T23:33:24.182577+0000","last_undegraded":"2024-03-28T23:33:24.182577+0000","last_fullsized":"2024-03-28T23:33:24.182577+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:12:17.544325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.182225+0000","last_change":"2024-03-28T23:33:24.178557+0000","last_active":"2024-03-28T23:33:24.182225+0000","last_peered":"2024-03-28T23:33:24.182225+0000","last_clean":"2024-03-28T23:33:24.182225+0000","last_became_active":"2024-03-28T23:33:24.178255+0000","last_became_peered":"2024-03-28T23:33:24.178255+0000","last_unstale":"2024-03-28T23:33:24.182225+0000","last_undegraded":"2024-03-28T23:33:24.182225+0000","last_fullsized":"2024-03-28T23:33:24.182225+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:01:13.611000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.187775+0000","last_change":"2024-03-28T23:33:24.180649+0000","last_active":"2024-03-28T23:33:24.187775+0000","last_peered":"2024-03-28T23:33:24.187775+0000","last_clean":"2024-03-28T23:33:24.187775+0000","last_became_active":"2024-03-28T23:33:24.180014+0000","last_became_peered":"2024-03-28T23:33:24.180014+0000","last_unstale":"2024-03-28T23:33:24.187775+0000","last_undegraded":"2024-03-28T23:33:24.187775+0000","last_fullsized":"2024-03-28T23:33:24.187775+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:56:45.330962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.175289+0000","last_change":"2024-03-28T23:33:24.173554+0000","last_active":"2024-03-28T23:33:24.175289+0000","last_peered":"2024-03-28T23:33:24.175289+0000","last_clean":"2024-03-28T23:33:24.175289+0000","last_became_active":"2024-03-28T23:33:24.173276+0000","last_became_peered":"2024-03-28T23:33:24.173276+0000","last_unstale":"2024-03-28T23:33:24.175289+0000","last_undegraded":"2024-03-28T23:33:24.175289+0000","last_fullsized":"2024-03-28T23:33:24.175289+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:20:57.071533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.176905+0000","last_change":"2024-03-28T23:33:24.170732+0000","last_active":"2024-03-28T23:33:24.176905+0000","last_peered":"2024-03-28T23:33:24.176905+0000","last_clean":"2024-03-28T23:33:24.176905+0000","last_became_active":"2024-03-28T23:33:24.170369+0000","last_became_peered":"2024-03-28T23:33:24.170369+0000","last_unstale":"2024-03-28T23:33:24.176905+0000","last_undegraded":"2024-03-28T23:33:24.176905+0000","last_fullsized":"2024-03-28T23:33:24.176905+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:50:58.694649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.182473+0000","last_change":"2024-03-28T23:33:24.176950+0000","last_active":"2024-03-28T23:33:24.182473+0000","last_peered":"2024-03-28T23:33:24.182473+0000","last_clean":"2024-03-28T23:33:24.182473+0000","last_became_active":"2024-03-28T23:33:24.176142+0000","last_became_peered":"2024-03-28T23:33:24.176142+0000","last_unstale":"2024-03-28T23:33:24.182473+0000","last_undegraded":"2024-03-28T23:33:24.182473+0000","last_fullsized":"2024-03-28T23:33:24.182473+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:22.409061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.183014+0000","last_change":"2024-03-28T23:33:24.172744+0000","last_active":"2024-03-28T23:33:24.183014+0000","last_peered":"2024-03-28T23:33:24.183014+0000","last_clean":"2024-03-28T23:33:24.183014+0000","last_became_active":"2024-03-28T23:33:24.172470+0000","last_became_peered":"2024-03-28T23:33:24.172470+0000","last_unstale":"2024-03-28T23:33:24.183014+0000","last_undegraded":"2024-03-28T23:33:24.183014+0000","last_fullsized":"2024-03-28T23:33:24.183014+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:10:23.895818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.179879+0000","last_change":"2024-03-28T23:33:24.175934+0000","last_active":"2024-03-28T23:33:24.179879+0000","last_peered":"2024-03-28T23:33:24.179879+0000","last_clean":"2024-03-28T23:33:24.179879+0000","last_became_active":"2024-03-28T23:33:24.175660+0000","last_became_peered":"2024-03-28T23:33:24.175660+0000","last_unstale":"2024-03-28T23:33:24.179879+0000","last_undegraded":"2024-03-28T23:33:24.179879+0000","last_fullsized":"2024-03-28T23:33:24.179879+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:30:19.952741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.178083+0000","last_change":"2024-03-28T23:33:24.177353+0000","last_active":"2024-03-28T23:33:24.178083+0000","last_peered":"2024-03-28T23:33:24.178083+0000","last_clean":"2024-03-28T23:33:24.178083+0000","last_became_active":"2024-03-28T23:33:24.177083+0000","last_became_peered":"2024-03-28T23:33:24.177083+0000","last_unstale":"2024-03-28T23:33:24.178083+0000","last_undegraded":"2024-03-28T23:33:24.178083+0000","last_fullsized":"2024-03-28T23:33:24.178083+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:53:16.718079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.182300+0000","last_change":"2024-03-28T23:33:24.180686+0000","last_active":"2024-03-28T23:33:24.182300+0000","last_peered":"2024-03-28T23:33:24.182300+0000","last_clean":"2024-03-28T23:33:24.182300+0000","last_became_active":"2024-03-28T23:33:24.180430+0000","last_became_peered":"2024-03-28T23:33:24.180430+0000","last_unstale":"2024-03-28T23:33:24.182300+0000","last_undegraded":"2024-03-28T23:33:24.182300+0000","last_fullsized":"2024-03-28T23:33:24.182300+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:55:03.722583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.184111+0000","last_change":"2024-03-28T23:33:24.183301+0000","last_active":"2024-03-28T23:33:24.184111+0000","last_peered":"2024-03-28T23:33:24.184111+0000","last_clean":"2024-03-28T23:33:24.184111+0000","last_became_active":"2024-03-28T23:33:24.183011+0000","last_became_peered":"2024-03-28T23:33:24.183011+0000","last_unstale":"2024-03-28T23:33:24.184111+0000","last_undegraded":"2024-03-28T23:33:24.184111+0000","last_fullsized":"2024-03-28T23:33:24.184111+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:34.756638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.189831+0000","last_change":"2024-03-28T23:33:24.182323+0000","last_active":"2024-03-28T23:33:24.189831+0000","last_peered":"2024-03-28T23:33:24.189831+0000","last_clean":"2024-03-28T23:33:24.189831+0000","last_became_active":"2024-03-28T23:33:24.182063+0000","last_became_peered":"2024-03-28T23:33:24.182063+0000","last_unstale":"2024-03-28T23:33:24.189831+0000","last_undegraded":"2024-03-28T23:33:24.189831+0000","last_fullsized":"2024-03-28T23:33:24.189831+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:45.913062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.181378+0000","last_change":"2024-03-28T23:33:24.178952+0000","last_active":"2024-03-28T23:33:24.181378+0000","last_peered":"2024-03-28T23:33:24.181378+0000","last_clean":"2024-03-28T23:33:24.181378+0000","last_became_active":"2024-03-28T23:33:24.178564+0000","last_became_peered":"2024-03-28T23:33:24.178564+0000","last_unstale":"2024-03-28T23:33:24.181378+0000","last_undegraded":"2024-03-28T23:33:24.181378+0000","last_fullsized":"2024-03-28T23:33:24.181378+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:21:46.670200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.193052+0000","last_change":"2024-03-28T23:33:24.187480+0000","last_active":"2024-03-28T23:33:24.193052+0000","last_peered":"2024-03-28T23:33:24.193052+0000","last_clean":"2024-03-28T23:33:24.193052+0000","last_became_active":"2024-03-28T23:33:24.185232+0000","last_became_peered":"2024-03-28T23:33:24.185232+0000","last_unstale":"2024-03-28T23:33:24.193052+0000","last_undegraded":"2024-03-28T23:33:24.193052+0000","last_fullsized":"2024-03-28T23:33:24.193052+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:19:34.917882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.182990+0000","last_change":"2024-03-28T23:33:24.169680+0000","last_active":"2024-03-28T23:33:24.182990+0000","last_peered":"2024-03-28T23:33:24.182990+0000","last_clean":"2024-03-28T23:33:24.182990+0000","last_became_active":"2024-03-28T23:33:24.169268+0000","last_became_peered":"2024-03-28T23:33:24.169268+0000","last_unstale":"2024-03-28T23:33:24.182990+0000","last_undegraded":"2024-03-28T23:33:24.182990+0000","last_fullsized":"2024-03-28T23:33:24.182990+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:49.980134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.185926+0000","last_change":"2024-03-28T23:33:24.175760+0000","last_active":"2024-03-28T23:33:24.185926+0000","last_peered":"2024-03-28T23:33:24.185926+0000","last_clean":"2024-03-28T23:33:24.185926+0000","last_became_active":"2024-03-28T23:33:24.175477+0000","last_became_peered":"2024-03-28T23:33:24.175477+0000","last_unstale":"2024-03-28T23:33:24.185926+0000","last_undegraded":"2024-03-28T23:33:24.185926+0000","last_fullsized":"2024-03-28T23:33:24.185926+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:44.649346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.190630+0000","last_change":"2024-03-28T23:33:24.178678+0000","last_active":"2024-03-28T23:33:24.190630+0000","last_peered":"2024-03-28T23:33:24.190630+0000","last_clean":"2024-03-28T23:33:24.190630+0000","last_became_active":"2024-03-28T23:33:24.178338+0000","last_became_peered":"2024-03-28T23:33:24.178338+0000","last_unstale":"2024-03-28T23:33:24.190630+0000","last_undegraded":"2024-03-28T23:33:24.190630+0000","last_fullsized":"2024-03-28T23:33:24.190630+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:10.325868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.189913+0000","last_change":"2024-03-28T23:33:24.183063+0000","last_active":"2024-03-28T23:33:24.189913+0000","last_peered":"2024-03-28T23:33:24.189913+0000","last_clean":"2024-03-28T23:33:24.189913+0000","last_became_active":"2024-03-28T23:33:24.182747+0000","last_became_peered":"2024-03-28T23:33:24.182747+0000","last_unstale":"2024-03-28T23:33:24.189913+0000","last_undegraded":"2024-03-28T23:33:24.189913+0000","last_fullsized":"2024-03-28T23:33:24.189913+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:50:42.702642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.194870+0000","last_change":"2024-03-28T23:33:24.184579+0000","last_active":"2024-03-28T23:33:24.194870+0000","last_peered":"2024-03-28T23:33:24.194870+0000","last_clean":"2024-03-28T23:33:24.194870+0000","last_became_active":"2024-03-28T23:33:24.184306+0000","last_became_peered":"2024-03-28T23:33:24.184306+0000","last_unstale":"2024-03-28T23:33:24.194870+0000","last_undegraded":"2024-03-28T23:33:24.194870+0000","last_fullsized":"2024-03-28T23:33:24.194870+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:10:23.895818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.181003+0000","last_change":"2024-03-28T23:33:24.175881+0000","last_active":"2024-03-28T23:33:24.181003+0000","last_peered":"2024-03-28T23:33:24.181003+0000","last_clean":"2024-03-28T23:33:24.181003+0000","last_became_active":"2024-03-28T23:33:24.175608+0000","last_became_peered":"2024-03-28T23:33:24.175608+0000","last_unstale":"2024-03-28T23:33:24.181003+0000","last_undegraded":"2024-03-28T23:33:24.181003+0000","last_fullsized":"2024-03-28T23:33:24.181003+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:03:43.530002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.181377+0000","last_change":"2024-03-28T23:33:24.170191+0000","last_active":"2024-03-28T23:33:24.181377+0000","last_peered":"2024-03-28T23:33:24.181377+0000","last_clean":"2024-03-28T23:33:24.181377+0000","last_became_active":"2024-03-28T23:33:24.169865+0000","last_became_peered":"2024-03-28T23:33:24.169865+0000","last_unstale":"2024-03-28T23:33:24.181377+0000","last_undegraded":"2024-03-28T23:33:24.181377+0000","last_fullsized":"2024-03-28T23:33:24.181377+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:22.409061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.174582+0000","last_change":"2024-03-28T23:33:24.173890+0000","last_active":"2024-03-28T23:33:24.174582+0000","last_peered":"2024-03-28T23:33:24.174582+0000","last_clean":"2024-03-28T23:33:24.174582+0000","last_became_active":"2024-03-28T23:33:24.173627+0000","last_became_peered":"2024-03-28T23:33:24.173627+0000","last_unstale":"2024-03-28T23:33:24.174582+0000","last_undegraded":"2024-03-28T23:33:24.174582+0000","last_fullsized":"2024-03-28T23:33:24.174582+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:50:58.694649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.178840+0000","last_change":"2024-03-28T23:33:24.177638+0000","last_active":"2024-03-28T23:33:24.178840+0000","last_peered":"2024-03-28T23:33:24.178840+0000","last_clean":"2024-03-28T23:33:24.178840+0000","last_became_active":"2024-03-28T23:33:24.177338+0000","last_became_peered":"2024-03-28T23:33:24.177338+0000","last_unstale":"2024-03-28T23:33:24.178840+0000","last_undegraded":"2024-03-28T23:33:24.178840+0000","last_fullsized":"2024-03-28T23:33:24.178840+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:36:33.138725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.173244+0000","last_change":"2024-03-28T23:33:24.170334+0000","last_active":"2024-03-28T23:33:24.173244+0000","last_peered":"2024-03-28T23:33:24.173244+0000","last_clean":"2024-03-28T23:33:24.173244+0000","last_became_active":"2024-03-28T23:33:24.170068+0000","last_became_peered":"2024-03-28T23:33:24.170068+0000","last_unstale":"2024-03-28T23:33:24.173244+0000","last_undegraded":"2024-03-28T23:33:24.173244+0000","last_fullsized":"2024-03-28T23:33:24.173244+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:29:27.672418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.181102+0000","last_change":"2024-03-28T23:33:24.177514+0000","last_active":"2024-03-28T23:33:24.181102+0000","last_peered":"2024-03-28T23:33:24.181102+0000","last_clean":"2024-03-28T23:33:24.181102+0000","last_became_active":"2024-03-28T23:33:24.177239+0000","last_became_peered":"2024-03-28T23:33:24.177239+0000","last_unstale":"2024-03-28T23:33:24.181102+0000","last_undegraded":"2024-03-28T23:33:24.181102+0000","last_fullsized":"2024-03-28T23:33:24.181102+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:14:06.775695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.186383+0000","last_change":"2024-03-28T23:33:24.185100+0000","last_active":"2024-03-28T23:33:24.186383+0000","last_peered":"2024-03-28T23:33:24.186383+0000","last_clean":"2024-03-28T23:33:24.186383+0000","last_became_active":"2024-03-28T23:33:24.184739+0000","last_became_peered":"2024-03-28T23:33:24.184739+0000","last_unstale":"2024-03-28T23:33:24.186383+0000","last_undegraded":"2024-03-28T23:33:24.186383+0000","last_fullsized":"2024-03-28T23:33:24.186383+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:42:30.721204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.187804+0000","last_change":"2024-03-28T23:33:24.180876+0000","last_active":"2024-03-28T23:33:24.187804+0000","last_peered":"2024-03-28T23:33:24.187804+0000","last_clean":"2024-03-28T23:33:24.187804+0000","last_became_active":"2024-03-28T23:33:24.180384+0000","last_became_peered":"2024-03-28T23:33:24.180384+0000","last_unstale":"2024-03-28T23:33:24.187804+0000","last_undegraded":"2024-03-28T23:33:24.187804+0000","last_fullsized":"2024-03-28T23:33:24.187804+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:59:39.367757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.190744+0000","last_change":"2024-03-28T23:33:24.185656+0000","last_active":"2024-03-28T23:33:24.190744+0000","last_peered":"2024-03-28T23:33:24.190744+0000","last_clean":"2024-03-28T23:33:24.190744+0000","last_became_active":"2024-03-28T23:33:24.185096+0000","last_became_peered":"2024-03-28T23:33:24.185096+0000","last_unstale":"2024-03-28T23:33:24.190744+0000","last_undegraded":"2024-03-28T23:33:24.190744+0000","last_fullsized":"2024-03-28T23:33:24.190744+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:38.115317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"93'4","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.196053+0000","last_change":"2024-03-28T23:33:23.170656+0000","last_active":"2024-03-28T23:33:24.196053+0000","last_peered":"2024-03-28T23:33:24.196053+0000","last_clean":"2024-03-28T23:33:24.196053+0000","last_became_active":"2024-03-28T23:33:23.170381+0000","last_became_peered":"2024-03-28T23:33:23.170381+0000","last_unstale":"2024-03-28T23:33:24.196053+0000","last_undegraded":"2024-03-28T23:33:24.196053+0000","last_fullsized":"2024-03-28T23:33:24.196053+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:14:05.771188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.184918+0000","last_change":"2024-03-28T23:33:24.178795+0000","last_active":"2024-03-28T23:33:24.184918+0000","last_peered":"2024-03-28T23:33:24.184918+0000","last_clean":"2024-03-28T23:33:24.184918+0000","last_became_active":"2024-03-28T23:33:24.178513+0000","last_became_peered":"2024-03-28T23:33:24.178513+0000","last_unstale":"2024-03-28T23:33:24.184918+0000","last_undegraded":"2024-03-28T23:33:24.184918+0000","last_fullsized":"2024-03-28T23:33:24.184918+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:06:26.191545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.192098+0000","last_change":"2024-03-28T23:33:23.172712+0000","last_active":"2024-03-28T23:33:24.192098+0000","last_peered":"2024-03-28T23:33:24.192098+0000","last_clean":"2024-03-28T23:33:24.192098+0000","last_became_active":"2024-03-28T23:33:23.171741+0000","last_became_peered":"2024-03-28T23:33:23.171741+0000","last_unstale":"2024-03-28T23:33:24.192098+0000","last_undegraded":"2024-03-28T23:33:24.192098+0000","last_fullsized":"2024-03-28T23:33:24.192098+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:36:06.822902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.174510+0000","last_change":"2024-03-28T23:33:24.165989+0000","last_active":"2024-03-28T23:33:24.174510+0000","last_peered":"2024-03-28T23:33:24.174510+0000","last_clean":"2024-03-28T23:33:24.174510+0000","last_became_active":"2024-03-28T23:33:24.165680+0000","last_became_peered":"2024-03-28T23:33:24.165680+0000","last_unstale":"2024-03-28T23:33:24.174510+0000","last_undegraded":"2024-03-28T23:33:24.174510+0000","last_fullsized":"2024-03-28T23:33:24.174510+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:50:58.694649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.191935+0000","last_change":"2024-03-28T23:33:23.176865+0000","last_active":"2024-03-28T23:33:24.191935+0000","last_peered":"2024-03-28T23:33:24.191935+0000","last_clean":"2024-03-28T23:33:24.191935+0000","last_became_active":"2024-03-28T23:33:23.176423+0000","last_became_peered":"2024-03-28T23:33:23.176423+0000","last_unstale":"2024-03-28T23:33:24.191935+0000","last_undegraded":"2024-03-28T23:33:24.191935+0000","last_fullsized":"2024-03-28T23:33:24.191935+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:27:35.509357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.184196+0000","last_change":"2024-03-28T23:33:24.182219+0000","last_active":"2024-03-28T23:33:24.184196+0000","last_peered":"2024-03-28T23:33:24.184196+0000","last_clean":"2024-03-28T23:33:24.184196+0000","last_became_active":"2024-03-28T23:33:24.181938+0000","last_became_peered":"2024-03-28T23:33:24.181938+0000","last_unstale":"2024-03-28T23:33:24.184196+0000","last_undegraded":"2024-03-28T23:33:24.184196+0000","last_fullsized":"2024-03-28T23:33:24.184196+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:14:06.775695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.199067+0000","last_change":"2024-03-28T23:33:23.175536+0000","last_active":"2024-03-28T23:33:24.199067+0000","last_peered":"2024-03-28T23:33:24.199067+0000","last_clean":"2024-03-28T23:33:24.199067+0000","last_became_active":"2024-03-28T23:33:23.175276+0000","last_became_peered":"2024-03-28T23:33:23.175276+0000","last_unstale":"2024-03-28T23:33:24.199067+0000","last_undegraded":"2024-03-28T23:33:24.199067+0000","last_fullsized":"2024-03-28T23:33:24.199067+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:45:07.164863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.179255+0000","last_change":"2024-03-28T23:33:24.173588+0000","last_active":"2024-03-28T23:33:24.179255+0000","last_peered":"2024-03-28T23:33:24.179255+0000","last_clean":"2024-03-28T23:33:24.179255+0000","last_became_active":"2024-03-28T23:33:24.173299+0000","last_became_peered":"2024-03-28T23:33:24.173299+0000","last_unstale":"2024-03-28T23:33:24.179255+0000","last_undegraded":"2024-03-28T23:33:24.179255+0000","last_fullsized":"2024-03-28T23:33:24.179255+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:34.756638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.190274+0000","last_change":"2024-03-28T23:33:23.168249+0000","last_active":"2024-03-28T23:33:24.190274+0000","last_peered":"2024-03-28T23:33:24.190274+0000","last_clean":"2024-03-28T23:33:24.190274+0000","last_became_active":"2024-03-28T23:33:23.167942+0000","last_became_peered":"2024-03-28T23:33:23.167942+0000","last_unstale":"2024-03-28T23:33:24.190274+0000","last_undegraded":"2024-03-28T23:33:24.190274+0000","last_fullsized":"2024-03-28T23:33:24.190274+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:03:42.525496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.190772+0000","last_change":"2024-03-28T23:33:24.185716+0000","last_active":"2024-03-28T23:33:24.190772+0000","last_peered":"2024-03-28T23:33:24.190772+0000","last_clean":"2024-03-28T23:33:24.190772+0000","last_became_active":"2024-03-28T23:33:24.184900+0000","last_became_peered":"2024-03-28T23:33:24.184900+0000","last_unstale":"2024-03-28T23:33:24.190772+0000","last_undegraded":"2024-03-28T23:33:24.190772+0000","last_fullsized":"2024-03-28T23:33:24.190772+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:24:02.991859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.191388+0000","last_change":"2024-03-28T23:33:23.177878+0000","last_active":"2024-03-28T23:33:24.191388+0000","last_peered":"2024-03-28T23:33:24.191388+0000","last_clean":"2024-03-28T23:33:24.191388+0000","last_became_active":"2024-03-28T23:33:23.177517+0000","last_became_peered":"2024-03-28T23:33:23.177517+0000","last_unstale":"2024-03-28T23:33:24.191388+0000","last_undegraded":"2024-03-28T23:33:24.191388+0000","last_fullsized":"2024-03-28T23:33:24.191388+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:54.634069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.181357+0000","last_change":"2024-03-28T23:33:24.178844+0000","last_active":"2024-03-28T23:33:24.181357+0000","last_peered":"2024-03-28T23:33:24.181357+0000","last_clean":"2024-03-28T23:33:24.181357+0000","last_became_active":"2024-03-28T23:33:24.178515+0000","last_became_peered":"2024-03-28T23:33:24.178515+0000","last_unstale":"2024-03-28T23:33:24.181357+0000","last_undegraded":"2024-03-28T23:33:24.181357+0000","last_fullsized":"2024-03-28T23:33:24.181357+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:41:07.386479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.161260+0000","last_change":"2024-03-28T23:33:23.176695+0000","last_active":"2024-03-28T23:33:24.161260+0000","last_peered":"2024-03-28T23:33:24.161260+0000","last_clean":"2024-03-28T23:33:24.161260+0000","last_became_active":"2024-03-28T23:33:23.176392+0000","last_became_peered":"2024-03-28T23:33:23.176392+0000","last_unstale":"2024-03-28T23:33:24.161260+0000","last_undegraded":"2024-03-28T23:33:24.161260+0000","last_fullsized":"2024-03-28T23:33:24.161260+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:15:32.758066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.194949+0000","last_change":"2024-03-28T23:33:24.189572+0000","last_active":"2024-03-28T23:33:24.194949+0000","last_peered":"2024-03-28T23:33:24.194949+0000","last_clean":"2024-03-28T23:33:24.194949+0000","last_became_active":"2024-03-28T23:33:24.188586+0000","last_became_peered":"2024-03-28T23:33:24.188586+0000","last_unstale":"2024-03-28T23:33:24.194949+0000","last_undegraded":"2024-03-28T23:33:24.194949+0000","last_fullsized":"2024-03-28T23:33:24.194949+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:07:37.632177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.162304+0000","last_change":"2024-03-28T23:33:23.172072+0000","last_active":"2024-03-28T23:33:24.162304+0000","last_peered":"2024-03-28T23:33:24.162304+0000","last_clean":"2024-03-28T23:33:24.162304+0000","last_became_active":"2024-03-28T23:33:23.171595+0000","last_became_peered":"2024-03-28T23:33:23.171595+0000","last_unstale":"2024-03-28T23:33:24.162304+0000","last_undegraded":"2024-03-28T23:33:24.162304+0000","last_fullsized":"2024-03-28T23:33:24.162304+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:19:30.535356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.193043+0000","last_change":"2024-03-28T23:33:24.187457+0000","last_active":"2024-03-28T23:33:24.193043+0000","last_peered":"2024-03-28T23:33:24.193043+0000","last_clean":"2024-03-28T23:33:24.193043+0000","last_became_active":"2024-03-28T23:33:24.185415+0000","last_became_peered":"2024-03-28T23:33:24.185415+0000","last_unstale":"2024-03-28T23:33:24.193043+0000","last_undegraded":"2024-03-28T23:33:24.193043+0000","last_fullsized":"2024-03-28T23:33:24.193043+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:25:51.087406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.161966+0000","last_change":"2024-03-28T23:33:23.176967+0000","last_active":"2024-03-28T23:33:24.161966+0000","last_peered":"2024-03-28T23:33:24.161966+0000","last_clean":"2024-03-28T23:33:24.161966+0000","last_became_active":"2024-03-28T23:33:23.176705+0000","last_became_peered":"2024-03-28T23:33:23.176705+0000","last_unstale":"2024-03-28T23:33:24.161966+0000","last_undegraded":"2024-03-28T23:33:24.161966+0000","last_fullsized":"2024-03-28T23:33:24.161966+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:54.634069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.176828+0000","last_change":"2024-03-28T23:33:24.169083+0000","last_active":"2024-03-28T23:33:24.176828+0000","last_peered":"2024-03-28T23:33:24.176828+0000","last_clean":"2024-03-28T23:33:24.176828+0000","last_became_active":"2024-03-28T23:33:24.168715+0000","last_became_peered":"2024-03-28T23:33:24.168715+0000","last_unstale":"2024-03-28T23:33:24.176828+0000","last_undegraded":"2024-03-28T23:33:24.176828+0000","last_fullsized":"2024-03-28T23:33:24.176828+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:55.638575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.177756+0000","last_change":"2024-03-28T23:33:23.173662+0000","last_active":"2024-03-28T23:33:24.177756+0000","last_peered":"2024-03-28T23:33:24.177756+0000","last_clean":"2024-03-28T23:33:24.177756+0000","last_became_active":"2024-03-28T23:33:23.173401+0000","last_became_peered":"2024-03-28T23:33:23.173401+0000","last_unstale":"2024-03-28T23:33:24.177756+0000","last_undegraded":"2024-03-28T23:33:24.177756+0000","last_fullsized":"2024-03-28T23:33:24.177756+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:38:10.896945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.185521+0000","last_change":"2024-03-28T23:33:24.177215+0000","last_active":"2024-03-28T23:33:24.185521+0000","last_peered":"2024-03-28T23:33:24.185521+0000","last_clean":"2024-03-28T23:33:24.185521+0000","last_became_active":"2024-03-28T23:33:24.176953+0000","last_became_peered":"2024-03-28T23:33:24.176953+0000","last_unstale":"2024-03-28T23:33:24.185521+0000","last_undegraded":"2024-03-28T23:33:24.185521+0000","last_fullsized":"2024-03-28T23:33:24.185521+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:03:36.327075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"93'3","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.193964+0000","last_change":"2024-03-28T23:33:23.170496+0000","last_active":"2024-03-28T23:33:24.193964+0000","last_peered":"2024-03-28T23:33:24.193964+0000","last_clean":"2024-03-28T23:33:24.193964+0000","last_became_active":"2024-03-28T23:33:23.170190+0000","last_became_peered":"2024-03-28T23:33:23.170190+0000","last_unstale":"2024-03-28T23:33:24.193964+0000","last_undegraded":"2024-03-28T23:33:24.193964+0000","last_fullsized":"2024-03-28T23:33:24.193964+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:14:05.771188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.177119+0000","last_change":"2024-03-28T23:33:24.172490+0000","last_active":"2024-03-28T23:33:24.177119+0000","last_peered":"2024-03-28T23:33:24.177119+0000","last_clean":"2024-03-28T23:33:24.177119+0000","last_became_active":"2024-03-28T23:33:24.172141+0000","last_became_peered":"2024-03-28T23:33:24.172141+0000","last_unstale":"2024-03-28T23:33:24.177119+0000","last_undegraded":"2024-03-28T23:33:24.177119+0000","last_fullsized":"2024-03-28T23:33:24.177119+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:49.980134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"93'6","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.199362+0000","last_change":"2024-03-28T23:33:23.170242+0000","last_active":"2024-03-28T23:33:24.199362+0000","last_peered":"2024-03-28T23:33:24.199362+0000","last_clean":"2024-03-28T23:33:24.199362+0000","last_became_active":"2024-03-28T23:33:23.169734+0000","last_became_peered":"2024-03-28T23:33:23.169734+0000","last_unstale":"2024-03-28T23:33:24.199362+0000","last_undegraded":"2024-03-28T23:33:24.199362+0000","last_fullsized":"2024-03-28T23:33:24.199362+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:17:25.021229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.184189+0000","last_change":"2024-03-28T23:33:24.181161+0000","last_active":"2024-03-28T23:33:24.184189+0000","last_peered":"2024-03-28T23:33:24.184189+0000","last_clean":"2024-03-28T23:33:24.184189+0000","last_became_active":"2024-03-28T23:33:24.180787+0000","last_became_peered":"2024-03-28T23:33:24.180787+0000","last_unstale":"2024-03-28T23:33:24.184189+0000","last_undegraded":"2024-03-28T23:33:24.184189+0000","last_fullsized":"2024-03-28T23:33:24.184189+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:03:43.530002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.199172+0000","last_change":"2024-03-28T23:33:23.169282+0000","last_active":"2024-03-28T23:33:24.199172+0000","last_peered":"2024-03-28T23:33:24.199172+0000","last_clean":"2024-03-28T23:33:24.199172+0000","last_became_active":"2024-03-28T23:33:23.168883+0000","last_became_peered":"2024-03-28T23:33:23.168883+0000","last_unstale":"2024-03-28T23:33:24.199172+0000","last_undegraded":"2024-03-28T23:33:24.199172+0000","last_fullsized":"2024-03-28T23:33:24.199172+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:27:35.509357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.185363+0000","last_change":"2024-03-28T23:33:24.183839+0000","last_active":"2024-03-28T23:33:24.185363+0000","last_peered":"2024-03-28T23:33:24.185363+0000","last_clean":"2024-03-28T23:33:24.185363+0000","last_became_active":"2024-03-28T23:33:24.182905+0000","last_became_peered":"2024-03-28T23:33:24.182905+0000","last_unstale":"2024-03-28T23:33:24.185363+0000","last_undegraded":"2024-03-28T23:33:24.185363+0000","last_fullsized":"2024-03-28T23:33:24.185363+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:45:08.169369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.164157+0000","last_change":"2024-03-28T23:33:23.173412+0000","last_active":"2024-03-28T23:33:24.164157+0000","last_peered":"2024-03-28T23:33:24.164157+0000","last_clean":"2024-03-28T23:33:24.164157+0000","last_became_active":"2024-03-28T23:33:23.173156+0000","last_became_peered":"2024-03-28T23:33:23.173156+0000","last_unstale":"2024-03-28T23:33:24.164157+0000","last_undegraded":"2024-03-28T23:33:24.164157+0000","last_fullsized":"2024-03-28T23:33:24.164157+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:53:15.713573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.187238+0000","last_change":"2024-03-28T23:33:24.181647+0000","last_active":"2024-03-28T23:33:24.187238+0000","last_peered":"2024-03-28T23:33:24.187238+0000","last_clean":"2024-03-28T23:33:24.187238+0000","last_became_active":"2024-03-28T23:33:24.181298+0000","last_became_peered":"2024-03-28T23:33:24.181298+0000","last_unstale":"2024-03-28T23:33:24.187238+0000","last_undegraded":"2024-03-28T23:33:24.187238+0000","last_fullsized":"2024-03-28T23:33:24.187238+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:45:49.920682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.167900+0000","last_change":"2024-03-28T23:33:23.177730+0000","last_active":"2024-03-28T23:33:24.167900+0000","last_peered":"2024-03-28T23:33:24.167900+0000","last_clean":"2024-03-28T23:33:24.167900+0000","last_became_active":"2024-03-28T23:33:23.177471+0000","last_became_peered":"2024-03-28T23:33:23.177471+0000","last_unstale":"2024-03-28T23:33:24.167900+0000","last_undegraded":"2024-03-28T23:33:24.167900+0000","last_fullsized":"2024-03-28T23:33:24.167900+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:18:35.677144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.190752+0000","last_change":"2024-03-28T23:33:24.185311+0000","last_active":"2024-03-28T23:33:24.190752+0000","last_peered":"2024-03-28T23:33:24.190752+0000","last_clean":"2024-03-28T23:33:24.190752+0000","last_became_active":"2024-03-28T23:33:24.185050+0000","last_became_peered":"2024-03-28T23:33:24.185050+0000","last_unstale":"2024-03-28T23:33:24.190752+0000","last_undegraded":"2024-03-28T23:33:24.190752+0000","last_fullsized":"2024-03-28T23:33:24.190752+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:02:22.419431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"93'8","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.192595+0000","last_change":"2024-03-28T23:33:23.174388+0000","last_active":"2024-03-28T23:33:24.192595+0000","last_peered":"2024-03-28T23:33:24.192595+0000","last_clean":"2024-03-28T23:33:24.192595+0000","last_became_active":"2024-03-28T23:33:23.174105+0000","last_became_peered":"2024-03-28T23:33:23.174105+0000","last_unstale":"2024-03-28T23:33:24.192595+0000","last_undegraded":"2024-03-28T23:33:24.192595+0000","last_fullsized":"2024-03-28T23:33:24.192595+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:36:47.363804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.192094+0000","last_change":"2024-03-28T23:33:24.187636+0000","last_active":"2024-03-28T23:33:24.192094+0000","last_peered":"2024-03-28T23:33:24.192094+0000","last_clean":"2024-03-28T23:33:24.192094+0000","last_became_active":"2024-03-28T23:33:24.187322+0000","last_became_peered":"2024-03-28T23:33:24.187322+0000","last_unstale":"2024-03-28T23:33:24.192094+0000","last_undegraded":"2024-03-28T23:33:24.192094+0000","last_fullsized":"2024-03-28T23:33:24.192094+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:47:34.781395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.191377+0000","last_change":"2024-03-28T23:33:23.177946+0000","last_active":"2024-03-28T23:33:24.191377+0000","last_peered":"2024-03-28T23:33:24.191377+0000","last_clean":"2024-03-28T23:33:24.191377+0000","last_became_active":"2024-03-28T23:33:23.177631+0000","last_became_peered":"2024-03-28T23:33:23.177631+0000","last_unstale":"2024-03-28T23:33:24.191377+0000","last_undegraded":"2024-03-28T23:33:24.191377+0000","last_fullsized":"2024-03-28T23:33:24.191377+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:58:27.929282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.183946+0000","last_change":"2024-03-28T23:33:24.182377+0000","last_active":"2024-03-28T23:33:24.183946+0000","last_peered":"2024-03-28T23:33:24.183946+0000","last_clean":"2024-03-28T23:33:24.183946+0000","last_became_active":"2024-03-28T23:33:24.182105+0000","last_became_peered":"2024-03-28T23:33:24.182105+0000","last_unstale":"2024-03-28T23:33:24.183946+0000","last_undegraded":"2024-03-28T23:33:24.183946+0000","last_fullsized":"2024-03-28T23:33:24.183946+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:37:11.908701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.191393+0000","last_change":"2024-03-28T23:33:23.173465+0000","last_active":"2024-03-28T23:33:24.191393+0000","last_peered":"2024-03-28T23:33:24.191393+0000","last_clean":"2024-03-28T23:33:24.191393+0000","last_became_active":"2024-03-28T23:33:23.173189+0000","last_became_peered":"2024-03-28T23:33:23.173189+0000","last_unstale":"2024-03-28T23:33:24.191393+0000","last_undegraded":"2024-03-28T23:33:24.191393+0000","last_fullsized":"2024-03-28T23:33:24.191393+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:19:30.535356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.177090+0000","last_change":"2024-03-28T23:33:24.168669+0000","last_active":"2024-03-28T23:33:24.177090+0000","last_peered":"2024-03-28T23:33:24.177090+0000","last_clean":"2024-03-28T23:33:24.177090+0000","last_became_active":"2024-03-28T23:33:24.168389+0000","last_became_peered":"2024-03-28T23:33:24.168389+0000","last_unstale":"2024-03-28T23:33:24.177090+0000","last_undegraded":"2024-03-28T23:33:24.177090+0000","last_fullsized":"2024-03-28T23:33:24.177090+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:15:33.762573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.169857+0000","last_change":"2024-03-28T23:33:23.171658+0000","last_active":"2024-03-28T23:33:24.169857+0000","last_peered":"2024-03-28T23:33:24.169857+0000","last_clean":"2024-03-28T23:33:24.169857+0000","last_became_active":"2024-03-28T23:33:23.171359+0000","last_became_peered":"2024-03-28T23:33:23.171359+0000","last_unstale":"2024-03-28T23:33:24.169857+0000","last_undegraded":"2024-03-28T23:33:24.169857+0000","last_fullsized":"2024-03-28T23:33:24.169857+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:15:32.758066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.184190+0000","last_change":"2024-03-28T23:33:24.180456+0000","last_active":"2024-03-28T23:33:24.184190+0000","last_peered":"2024-03-28T23:33:24.184190+0000","last_clean":"2024-03-28T23:33:24.184190+0000","last_became_active":"2024-03-28T23:33:24.179979+0000","last_became_peered":"2024-03-28T23:33:24.179979+0000","last_unstale":"2024-03-28T23:33:24.184190+0000","last_undegraded":"2024-03-28T23:33:24.184190+0000","last_fullsized":"2024-03-28T23:33:24.184190+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:45:25.533216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.193534+0000","last_change":"2024-03-28T23:33:23.169314+0000","last_active":"2024-03-28T23:33:24.193534+0000","last_peered":"2024-03-28T23:33:24.193534+0000","last_clean":"2024-03-28T23:33:24.193534+0000","last_became_active":"2024-03-28T23:33:23.169060+0000","last_became_peered":"2024-03-28T23:33:23.169060+0000","last_unstale":"2024-03-28T23:33:24.193534+0000","last_undegraded":"2024-03-28T23:33:24.193534+0000","last_fullsized":"2024-03-28T23:33:24.193534+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:44:26.078995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.199313+0000","last_change":"2024-03-28T23:33:24.195736+0000","last_active":"2024-03-28T23:33:24.199313+0000","last_peered":"2024-03-28T23:33:24.199313+0000","last_clean":"2024-03-28T23:33:24.199313+0000","last_became_active":"2024-03-28T23:33:24.195532+0000","last_became_peered":"2024-03-28T23:33:24.195532+0000","last_unstale":"2024-03-28T23:33:24.199313+0000","last_undegraded":"2024-03-28T23:33:24.199313+0000","last_fullsized":"2024-03-28T23:33:24.199313+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:10.385852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.167401+0000","last_change":"2024-03-28T23:33:23.173074+0000","last_active":"2024-03-28T23:33:24.167401+0000","last_peered":"2024-03-28T23:33:24.167401+0000","last_clean":"2024-03-28T23:33:24.167401+0000","last_became_active":"2024-03-28T23:33:23.171817+0000","last_became_peered":"2024-03-28T23:33:23.171817+0000","last_unstale":"2024-03-28T23:33:24.167401+0000","last_undegraded":"2024-03-28T23:33:24.167401+0000","last_fullsized":"2024-03-28T23:33:24.167401+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:14:25.053743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.171427+0000","last_change":"2024-03-28T23:33:24.170189+0000","last_active":"2024-03-28T23:33:24.171427+0000","last_peered":"2024-03-28T23:33:24.171427+0000","last_clean":"2024-03-28T23:33:24.171427+0000","last_became_active":"2024-03-28T23:33:24.169830+0000","last_became_peered":"2024-03-28T23:33:24.169830+0000","last_unstale":"2024-03-28T23:33:24.171427+0000","last_undegraded":"2024-03-28T23:33:24.171427+0000","last_fullsized":"2024-03-28T23:33:24.171427+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:05:18.562852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.168634+0000","last_change":"2024-03-28T23:33:23.172293+0000","last_active":"2024-03-28T23:33:24.168634+0000","last_peered":"2024-03-28T23:33:24.168634+0000","last_clean":"2024-03-28T23:33:24.168634+0000","last_became_active":"2024-03-28T23:33:23.171915+0000","last_became_peered":"2024-03-28T23:33:23.171915+0000","last_unstale":"2024-03-28T23:33:24.168634+0000","last_undegraded":"2024-03-28T23:33:24.168634+0000","last_fullsized":"2024-03-28T23:33:24.168634+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:58:23.535559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.184942+0000","last_change":"2024-03-28T23:33:24.178084+0000","last_active":"2024-03-28T23:33:24.184942+0000","last_peered":"2024-03-28T23:33:24.184942+0000","last_clean":"2024-03-28T23:33:24.184942+0000","last_became_active":"2024-03-28T23:33:24.177784+0000","last_became_peered":"2024-03-28T23:33:24.177784+0000","last_unstale":"2024-03-28T23:33:24.184942+0000","last_undegraded":"2024-03-28T23:33:24.184942+0000","last_fullsized":"2024-03-28T23:33:24.184942+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:32:59.609786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.167342+0000","last_change":"2024-03-28T23:33:23.167218+0000","last_active":"2024-03-28T23:33:24.167342+0000","last_peered":"2024-03-28T23:33:24.167342+0000","last_clean":"2024-03-28T23:33:24.167342+0000","last_became_active":"2024-03-28T23:33:23.166972+0000","last_became_peered":"2024-03-28T23:33:23.166972+0000","last_unstale":"2024-03-28T23:33:24.167342+0000","last_undegraded":"2024-03-28T23:33:24.167342+0000","last_fullsized":"2024-03-28T23:33:24.167342+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:40:29.245461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.184350+0000","last_change":"2024-03-28T23:33:24.180501+0000","last_active":"2024-03-28T23:33:24.184350+0000","last_peered":"2024-03-28T23:33:24.184350+0000","last_clean":"2024-03-28T23:33:24.184350+0000","last_became_active":"2024-03-28T23:33:24.180221+0000","last_became_peered":"2024-03-28T23:33:24.180221+0000","last_unstale":"2024-03-28T23:33:24.184350+0000","last_undegraded":"2024-03-28T23:33:24.184350+0000","last_fullsized":"2024-03-28T23:33:24.184350+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:59.077144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.195639+0000","last_change":"2024-03-28T23:33:23.168646+0000","last_active":"2024-03-28T23:33:24.195639+0000","last_peered":"2024-03-28T23:33:24.195639+0000","last_clean":"2024-03-28T23:33:24.195639+0000","last_became_active":"2024-03-28T23:33:23.168396+0000","last_became_peered":"2024-03-28T23:33:23.168396+0000","last_unstale":"2024-03-28T23:33:24.195639+0000","last_undegraded":"2024-03-28T23:33:24.195639+0000","last_fullsized":"2024-03-28T23:33:24.195639+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:36:47.363804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.181785+0000","last_change":"2024-03-28T23:33:24.170667+0000","last_active":"2024-03-28T23:33:24.181785+0000","last_peered":"2024-03-28T23:33:24.181785+0000","last_clean":"2024-03-28T23:33:24.181785+0000","last_became_active":"2024-03-28T23:33:24.170387+0000","last_became_peered":"2024-03-28T23:33:24.170387+0000","last_unstale":"2024-03-28T23:33:24.181785+0000","last_undegraded":"2024-03-28T23:33:24.181785+0000","last_fullsized":"2024-03-28T23:33:24.181785+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:10.325868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.177929+0000","last_change":"2024-03-28T23:33:23.168777+0000","last_active":"2024-03-28T23:33:24.177929+0000","last_peered":"2024-03-28T23:33:24.177929+0000","last_clean":"2024-03-28T23:33:24.177929+0000","last_became_active":"2024-03-28T23:33:23.168535+0000","last_became_peered":"2024-03-28T23:33:23.168535+0000","last_unstale":"2024-03-28T23:33:24.177929+0000","last_undegraded":"2024-03-28T23:33:24.177929+0000","last_fullsized":"2024-03-28T23:33:24.177929+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:39:12.279617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.179385+0000","last_change":"2024-03-28T23:33:24.171534+0000","last_active":"2024-03-28T23:33:24.179385+0000","last_peered":"2024-03-28T23:33:24.179385+0000","last_clean":"2024-03-28T23:33:24.179385+0000","last_became_active":"2024-03-28T23:33:24.171104+0000","last_became_peered":"2024-03-28T23:33:24.171104+0000","last_unstale":"2024-03-28T23:33:24.179385+0000","last_undegraded":"2024-03-28T23:33:24.179385+0000","last_fullsized":"2024-03-28T23:33:24.179385+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:55:03.722583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.174631+0000","last_change":"2024-03-28T23:33:23.171776+0000","last_active":"2024-03-28T23:33:24.174631+0000","last_peered":"2024-03-28T23:33:24.174631+0000","last_clean":"2024-03-28T23:33:24.174631+0000","last_became_active":"2024-03-28T23:33:23.171500+0000","last_became_peered":"2024-03-28T23:33:23.171500+0000","last_unstale":"2024-03-28T23:33:24.174631+0000","last_undegraded":"2024-03-28T23:33:24.174631+0000","last_fullsized":"2024-03-28T23:33:24.174631+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:39:15.619921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.181820+0000","last_change":"2024-03-28T23:33:24.178644+0000","last_active":"2024-03-28T23:33:24.181820+0000","last_peered":"2024-03-28T23:33:24.181820+0000","last_clean":"2024-03-28T23:33:24.181820+0000","last_became_active":"2024-03-28T23:33:24.178305+0000","last_became_peered":"2024-03-28T23:33:24.178305+0000","last_unstale":"2024-03-28T23:33:24.181820+0000","last_undegraded":"2024-03-28T23:33:24.181820+0000","last_fullsized":"2024-03-28T23:33:24.181820+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:10.325868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.161005+0000","last_change":"2024-03-28T23:33:23.169576+0000","last_active":"2024-03-28T23:33:24.161005+0000","last_peered":"2024-03-28T23:33:24.161005+0000","last_clean":"2024-03-28T23:33:24.161005+0000","last_became_active":"2024-03-28T23:33:23.168602+0000","last_became_peered":"2024-03-28T23:33:23.168602+0000","last_unstale":"2024-03-28T23:33:24.161005+0000","last_undegraded":"2024-03-28T23:33:24.161005+0000","last_fullsized":"2024-03-28T23:33:24.161005+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:42:29.716698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.186175+0000","last_change":"2024-03-28T23:33:24.178407+0000","last_active":"2024-03-28T23:33:24.186175+0000","last_peered":"2024-03-28T23:33:24.186175+0000","last_clean":"2024-03-28T23:33:24.186175+0000","last_became_active":"2024-03-28T23:33:24.178112+0000","last_became_peered":"2024-03-28T23:33:24.178112+0000","last_unstale":"2024-03-28T23:33:24.186175+0000","last_undegraded":"2024-03-28T23:33:24.186175+0000","last_fullsized":"2024-03-28T23:33:24.186175+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:51.494690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.160691+0000","last_change":"2024-03-28T23:33:23.174209+0000","last_active":"2024-03-28T23:33:24.160691+0000","last_peered":"2024-03-28T23:33:24.160691+0000","last_clean":"2024-03-28T23:33:24.160691+0000","last_became_active":"2024-03-28T23:33:23.173897+0000","last_became_peered":"2024-03-28T23:33:23.173897+0000","last_unstale":"2024-03-28T23:33:24.160691+0000","last_undegraded":"2024-03-28T23:33:24.160691+0000","last_fullsized":"2024-03-28T23:33:24.160691+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:42:29.716698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.186212+0000","last_change":"2024-03-28T23:33:24.179077+0000","last_active":"2024-03-28T23:33:24.186212+0000","last_peered":"2024-03-28T23:33:24.186212+0000","last_clean":"2024-03-28T23:33:24.186212+0000","last_became_active":"2024-03-28T23:33:24.178735+0000","last_became_peered":"2024-03-28T23:33:24.178735+0000","last_unstale":"2024-03-28T23:33:24.186212+0000","last_undegraded":"2024-03-28T23:33:24.186212+0000","last_fullsized":"2024-03-28T23:33:24.186212+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:39:30.420442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.167594+0000","last_change":"2024-03-28T23:33:23.175820+0000","last_active":"2024-03-28T23:33:24.167594+0000","last_peered":"2024-03-28T23:33:24.167594+0000","last_clean":"2024-03-28T23:33:24.167594+0000","last_became_active":"2024-03-28T23:33:23.175510+0000","last_became_peered":"2024-03-28T23:33:23.175510+0000","last_unstale":"2024-03-28T23:33:24.167594+0000","last_undegraded":"2024-03-28T23:33:24.167594+0000","last_fullsized":"2024-03-28T23:33:24.167594+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:48:58.212882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.182271+0000","last_change":"2024-03-28T23:33:24.179623+0000","last_active":"2024-03-28T23:33:24.182271+0000","last_peered":"2024-03-28T23:33:24.182271+0000","last_clean":"2024-03-28T23:33:24.182271+0000","last_became_active":"2024-03-28T23:33:24.179359+0000","last_became_peered":"2024-03-28T23:33:24.179359+0000","last_unstale":"2024-03-28T23:33:24.182271+0000","last_undegraded":"2024-03-28T23:33:24.182271+0000","last_fullsized":"2024-03-28T23:33:24.182271+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:05:18.562852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.168810+0000","last_change":"2024-03-28T23:33:23.171390+0000","last_active":"2024-03-28T23:33:24.168810+0000","last_peered":"2024-03-28T23:33:24.168810+0000","last_clean":"2024-03-28T23:33:24.168810+0000","last_became_active":"2024-03-28T23:33:23.171087+0000","last_became_peered":"2024-03-28T23:33:23.171087+0000","last_unstale":"2024-03-28T23:33:24.168810+0000","last_undegraded":"2024-03-28T23:33:24.168810+0000","last_fullsized":"2024-03-28T23:33:24.168810+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:13:48.967256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.181466+0000","last_change":"2024-03-28T23:33:24.175821+0000","last_active":"2024-03-28T23:33:24.181466+0000","last_peered":"2024-03-28T23:33:24.181466+0000","last_clean":"2024-03-28T23:33:24.181466+0000","last_became_active":"2024-03-28T23:33:24.175530+0000","last_became_peered":"2024-03-28T23:33:24.175530+0000","last_unstale":"2024-03-28T23:33:24.181466+0000","last_undegraded":"2024-03-28T23:33:24.181466+0000","last_fullsized":"2024-03-28T23:33:24.181466+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:49.980134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.167501+0000","last_change":"2024-03-28T23:33:23.175531+0000","last_active":"2024-03-28T23:33:24.167501+0000","last_peered":"2024-03-28T23:33:24.167501+0000","last_clean":"2024-03-28T23:33:24.167501+0000","last_became_active":"2024-03-28T23:33:23.175226+0000","last_became_peered":"2024-03-28T23:33:23.175226+0000","last_unstale":"2024-03-28T23:33:24.167501+0000","last_undegraded":"2024-03-28T23:33:24.167501+0000","last_fullsized":"2024-03-28T23:33:24.167501+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:33.473881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.181646+0000","last_change":"2024-03-28T23:33:24.180370+0000","last_active":"2024-03-28T23:33:24.181646+0000","last_peered":"2024-03-28T23:33:24.181646+0000","last_clean":"2024-03-28T23:33:24.181646+0000","last_became_active":"2024-03-28T23:33:24.180073+0000","last_became_peered":"2024-03-28T23:33:24.180073+0000","last_unstale":"2024-03-28T23:33:24.181646+0000","last_undegraded":"2024-03-28T23:33:24.181646+0000","last_fullsized":"2024-03-28T23:33:24.181646+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:10:23.895818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.195121+0000","last_change":"2024-03-28T23:33:23.172658+0000","last_active":"2024-03-28T23:33:24.195121+0000","last_peered":"2024-03-28T23:33:24.195121+0000","last_clean":"2024-03-28T23:33:24.195121+0000","last_became_active":"2024-03-28T23:33:23.172368+0000","last_became_peered":"2024-03-28T23:33:23.172368+0000","last_unstale":"2024-03-28T23:33:24.195121+0000","last_undegraded":"2024-03-28T23:33:24.195121+0000","last_fullsized":"2024-03-28T23:33:24.195121+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:35:12.907721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":308,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.166563+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:33:24.166563+0000","last_peered":"2024-03-28T23:33:24.166563+0000","last_clean":"2024-03-28T23:33:24.166563+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:33:24.166563+0000","last_undegraded":"2024-03-28T23:33:24.166563+0000","last_fullsized":"2024-03-28T23:33:24.166563+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1208320,"data_stored":1139446,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32296,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339544,"statfs":{"total":96636764160,"available":96603693056,"internally_reserved":0,"allocated":1142784,"data_stored":549078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574896,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32300,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339540,"statfs":{"total":96636764160,"available":96603688960,"internally_reserved":0,"allocated":1142784,"data_stored":549078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574895,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32296,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339544,"statfs":{"total":96636764160,"available":96603693056,"internally_reserved":0,"allocated":1142784,"data_stored":549078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574895,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28136,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343704,"statfs":{"total":96636764160,"available":96607952896,"internally_reserved":0,"allocated":1142784,"data_stored":549078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574895,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28136,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343704,"statfs":{"total":96636764160,"available":96607952896,"internally_reserved":0,"allocated":1142784,"data_stored":549078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28136,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343704,"statfs":{"total":96636764160,"available":96607952896,"internally_reserved":0,"allocated":1142784,"data_stored":549078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574895,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32396,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339444,"statfs":{"total":96636764160,"available":96603590656,"internally_reserved":0,"allocated":1236992,"data_stored":1148315,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:33:27.141 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:33:27.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.276+0000 7f9387a4e640 1 Processor -- start 2024-03-28T23:33:27.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.276+0000 7f9387a4e640 1 -- start start 2024-03-28T23:33:27.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.277+0000 7f9387a4e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9380155b90 0x7f9380158750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:27.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.277+0000 7f9387a4e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f93800ac150 0x7f9380158c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:27.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.277+0000 7f9387a4e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9380066fe0 con 0x7f938006c3d0 2024-03-28T23:33:27.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.277+0000 7f9387a4e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f93801591d0 con 0x7f93800ac150 2024-03-28T23:33:27.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.277+0000 7f9387a4e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9380156070 con 0x7f9380155b90 2024-03-28T23:33:27.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.277+0000 7f9385feb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f93800ac150 0x7f9380158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:27.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.277+0000 7f9385feb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f93800ac150 0x7f9380158c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41166/0 (socket says 172.21.15.161:41166) 2024-03-28T23:33:27.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.277+0000 7f9385feb640 1 -- 172.21.15.161:0/2491357076 learned_addr learned my addr 172.21.15.161:0/2491357076 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:27.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.277+0000 7f9384fe9640 1 --2- 172.21.15.161:0/2491357076 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9380155b90 0x7f9380158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:27.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.278+0000 7f9385feb640 1 -- 172.21.15.161:0/2491357076 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9380155b90 msgr2=0x7f9380158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:27.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.278+0000 7f9385feb640 1 --2- 172.21.15.161:0/2491357076 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9380155b90 0x7f9380158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.278+0000 7f9385feb640 1 -- 172.21.15.161:0/2491357076 >> v1:172.21.15.170:6789/0 conn(0x7f938006c3d0 legacy=0x7f9380152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:27.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.278+0000 7f9385feb640 1 -- 172.21.15.161:0/2491357076 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93800672c0 con 0x7f93800ac150 2024-03-28T23:33:27.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.278+0000 7f9385feb640 1 --2- 172.21.15.161:0/2491357076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93800ac150 0x7f9380158c90 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f9378009870 tx=0x7f937802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d122550c0575207a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:27.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.279+0000 7f9377fff640 1 -- 172.21.15.161:0/2491357076 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9378037070 con 0x7f93800ac150 2024-03-28T23:33:27.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.279+0000 7f9377fff640 1 -- 172.21.15.161:0/2491357076 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9378004230 con 0x7f93800ac150 2024-03-28T23:33:27.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.279+0000 7f9377fff640 1 -- 172.21.15.161:0/2491357076 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9378003610 con 0x7f93800ac150 2024-03-28T23:33:27.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.279+0000 7f9387a4e640 1 -- 172.21.15.161:0/2491357076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93800ac150 msgr2=0x7f9380158c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:27.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.279+0000 7f9387a4e640 1 --2- 172.21.15.161:0/2491357076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93800ac150 0x7f9380158c90 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f9378009870 tx=0x7f937802b7e0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.279+0000 7f9387a4e640 1 -- 172.21.15.161:0/2491357076 shutdown_connections 2024-03-28T23:33:27.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.279+0000 7f9387a4e640 1 --2- 172.21.15.161:0/2491357076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93800ac150 0x7f9380158c90 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.279+0000 7f9387a4e640 1 --2- 172.21.15.161:0/2491357076 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9380155b90 0x7f9380158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.279+0000 7f9387a4e640 1 -- 172.21.15.161:0/2491357076 >> 172.21.15.161:0/2491357076 conn(0x7f93800c3be0 msgr2=0x7f93800c3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:27.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.279+0000 7f9387a4e640 1 -- 172.21.15.161:0/2491357076 shutdown_connections 2024-03-28T23:33:27.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.279+0000 7f9387a4e640 1 -- 172.21.15.161:0/2491357076 wait complete. 2024-03-28T23:33:27.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.280+0000 7f9387a4e640 1 Processor -- start 2024-03-28T23:33:27.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.280+0000 7f9387a4e640 1 -- start start 2024-03-28T23:33:27.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.281+0000 7f9387a4e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f938006c3d0 0x7f93800729b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:27.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.281+0000 7f9387a4e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93800ac150 0x7f938014ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:27.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.281+0000 7f9387a4e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9380155b90 0x7f938014f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:27.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.281+0000 7f9387a4e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9380152e10 con 0x7f9380155b90 2024-03-28T23:33:27.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.281+0000 7f9387a4e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9380152c90 con 0x7f938006c3d0 2024-03-28T23:33:27.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.281+0000 7f9387a4e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f93800c21a0 con 0x7f93800ac150 2024-03-28T23:33:27.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.281+0000 7f9385feb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9380155b90 0x7f938014f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:27.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.281+0000 7f9385feb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9380155b90 0x7f938014f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41174/0 (socket says 172.21.15.161:41174) 2024-03-28T23:33:27.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.281+0000 7f9385feb640 1 -- 172.21.15.161:0/3870189316 learned_addr learned my addr 172.21.15.161:0/3870189316 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:27.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.281+0000 7f93857ea640 1 --2- 172.21.15.161:0/3870189316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f938006c3d0 0x7f93800729b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:27.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.281+0000 7f9384fe9640 1 --2- 172.21.15.161:0/3870189316 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93800ac150 0x7f938014ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:27.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.281+0000 7f9385feb640 1 -- 172.21.15.161:0/3870189316 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93800ac150 msgr2=0x7f938014ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:27.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.281+0000 7f9385feb640 1 --2- 172.21.15.161:0/3870189316 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93800ac150 0x7f938014ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.281+0000 7f9385feb640 1 -- 172.21.15.161:0/3870189316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f938006c3d0 msgr2=0x7f93800729b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:27.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.281+0000 7f9385feb640 1 --2- 172.21.15.161:0/3870189316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f938006c3d0 0x7f93800729b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.281+0000 7f9385feb640 1 -- 172.21.15.161:0/3870189316 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9380073270 con 0x7f9380155b90 2024-03-28T23:33:27.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.282+0000 7f9385feb640 1 --2- 172.21.15.161:0/3870189316 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9380155b90 0x7f938014f220 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f93780090b0 tx=0x7f93780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:27.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.282+0000 7f9375ffb640 1 -- 172.21.15.161:0/3870189316 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9378041070 con 0x7f9380155b90 2024-03-28T23:33:27.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.282+0000 7f9375ffb640 1 -- 172.21.15.161:0/3870189316 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9378030510 con 0x7f9380155b90 2024-03-28T23:33:27.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.282+0000 7f9375ffb640 1 -- 172.21.15.161:0/3870189316 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9378037040 con 0x7f9380155b90 2024-03-28T23:33:27.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.282+0000 7f9387a4e640 1 -- 172.21.15.161:0/3870189316 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93801eeee0 con 0x7f9380155b90 2024-03-28T23:33:27.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.282+0000 7f9387a4e640 1 -- 172.21.15.161:0/3870189316 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9380072ef0 con 0x7f9380155b90 2024-03-28T23:33:27.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.283+0000 7f9387a4e640 1 -- 172.21.15.161:0/3870189316 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f938004f4b0 con 0x7f9380155b90 2024-03-28T23:33:27.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.283+0000 7f9375ffb640 1 -- 172.21.15.161:0/3870189316 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9378044070 con 0x7f9380155b90 2024-03-28T23:33:27.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.284+0000 7f9375ffb640 1 --2- 172.21.15.161:0/3870189316 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f936406b4d0 0x7f936406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:27.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.284+0000 7f9375ffb640 1 -- 172.21.15.161:0/3870189316 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6101+0+0 (secure 0 0 0) 0x7f93780abce0 con 0x7f9380155b90 2024-03-28T23:33:27.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.284+0000 7f93857ea640 1 --2- 172.21.15.161:0/3870189316 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f936406b4d0 0x7f936406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:27.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.284+0000 7f93857ea640 1 --2- 172.21.15.161:0/3870189316 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f936406b4d0 0x7f936406d990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f937c0020a0 tx=0x7f937c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:27.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.288+0000 7f9375ffb640 1 -- 172.21.15.161:0/3870189316 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f937807a410 con 0x7f9380155b90 2024-03-28T23:33:27.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.457+0000 7f9387a4e640 1 -- 172.21.15.161:0/3870189316 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f93801eec90 con 0x7f936406b4d0 2024-03-28T23:33:27.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.463+0000 7f9375ffb640 1 -- 172.21.15.161:0/3870189316 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252090 (secure 0 0 0) 0x7f93801eec90 con 0x7f936406b4d0 2024-03-28T23:33:27.469 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:27.471 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:33:27.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.466+0000 7f9387a4e640 1 -- 172.21.15.161:0/3870189316 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f936406b4d0 msgr2=0x7f936406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:27.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.466+0000 7f9387a4e640 1 --2- 172.21.15.161:0/3870189316 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f936406b4d0 0x7f936406d990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f937c0020a0 tx=0x7f937c009040 comp rx=0 tx=0).stop 2024-03-28T23:33:27.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.466+0000 7f9387a4e640 1 -- 172.21.15.161:0/3870189316 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9380155b90 msgr2=0x7f938014f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:27.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.466+0000 7f9387a4e640 1 --2- 172.21.15.161:0/3870189316 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9380155b90 0x7f938014f220 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f93780090b0 tx=0x7f93780046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.467+0000 7f9387a4e640 1 -- 172.21.15.161:0/3870189316 shutdown_connections 2024-03-28T23:33:27.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.467+0000 7f9387a4e640 1 --2- 172.21.15.161:0/3870189316 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f936406b4d0 0x7f936406d990 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.467+0000 7f9387a4e640 1 --2- 172.21.15.161:0/3870189316 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9380155b90 0x7f938014f220 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.467+0000 7f9387a4e640 1 --2- 172.21.15.161:0/3870189316 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93800ac150 0x7f938014ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.467+0000 7f9387a4e640 1 --2- 172.21.15.161:0/3870189316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f938006c3d0 0x7f93800729b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.467+0000 7f9387a4e640 1 -- 172.21.15.161:0/3870189316 >> 172.21.15.161:0/3870189316 conn(0x7f93800c3be0 msgr2=0x7f93800c0510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:27.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.467+0000 7f9387a4e640 1 -- 172.21.15.161:0/3870189316 shutdown_connections 2024-03-28T23:33:27.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.467+0000 7f9387a4e640 1 -- 172.21.15.161:0/3870189316 wait complete. 2024-03-28T23:33:27.479 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-03-28T23:33:27.353559+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":246056,"kb_used_data":9084,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754728664,"statfs":{"total":773094113280,"available":772842151936,"internally_reserved":0,"allocated":9302016,"data_stored":5582229,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.202925"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.184247+0000","last_change":"2024-03-28T23:33:24.181088+0000","last_active":"2024-03-28T23:33:24.184247+0000","last_peered":"2024-03-28T23:33:24.184247+0000","last_clean":"2024-03-28T23:33:24.184247+0000","last_became_active":"2024-03-28T23:33:24.180815+0000","last_became_peered":"2024-03-28T23:33:24.180815+0000","last_unstale":"2024-03-28T23:33:24.184247+0000","last_undegraded":"2024-03-28T23:33:24.184247+0000","last_fullsized":"2024-03-28T23:33:24.184247+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:16:14.029745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.175254+0000","last_change":"2024-03-28T23:33:24.173050+0000","last_active":"2024-03-28T23:33:24.175254+0000","last_peered":"2024-03-28T23:33:24.175254+0000","last_clean":"2024-03-28T23:33:24.175254+0000","last_became_active":"2024-03-28T23:33:24.172761+0000","last_became_peered":"2024-03-28T23:33:24.172761+0000","last_unstale":"2024-03-28T23:33:24.175254+0000","last_undegraded":"2024-03-28T23:33:24.175254+0000","last_fullsized":"2024-03-28T23:33:24.175254+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:56:05.296197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.190747+0000","last_change":"2024-03-28T23:33:24.185523+0000","last_active":"2024-03-28T23:33:24.190747+0000","last_peered":"2024-03-28T23:33:24.190747+0000","last_clean":"2024-03-28T23:33:24.190747+0000","last_became_active":"2024-03-28T23:33:24.184535+0000","last_became_peered":"2024-03-28T23:33:24.184535+0000","last_unstale":"2024-03-28T23:33:24.190747+0000","last_undegraded":"2024-03-28T23:33:24.190747+0000","last_fullsized":"2024-03-28T23:33:24.190747+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:35:39.480945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.179924+0000","last_change":"2024-03-28T23:33:24.177488+0000","last_active":"2024-03-28T23:33:24.179924+0000","last_peered":"2024-03-28T23:33:24.179924+0000","last_clean":"2024-03-28T23:33:24.179924+0000","last_became_active":"2024-03-28T23:33:24.177216+0000","last_became_peered":"2024-03-28T23:33:24.177216+0000","last_unstale":"2024-03-28T23:33:24.179924+0000","last_undegraded":"2024-03-28T23:33:24.179924+0000","last_fullsized":"2024-03-28T23:33:24.179924+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:35:30.544190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.189832+0000","last_change":"2024-03-28T23:33:24.176392+0000","last_active":"2024-03-28T23:33:24.189832+0000","last_peered":"2024-03-28T23:33:24.189832+0000","last_clean":"2024-03-28T23:33:24.189832+0000","last_became_active":"2024-03-28T23:33:24.176017+0000","last_became_peered":"2024-03-28T23:33:24.176017+0000","last_unstale":"2024-03-28T23:33:24.189832+0000","last_undegraded":"2024-03-28T23:33:24.189832+0000","last_fullsized":"2024-03-28T23:33:24.189832+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:22.409061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.182577+0000","last_change":"2024-03-28T23:33:24.179934+0000","last_active":"2024-03-28T23:33:24.182577+0000","last_peered":"2024-03-28T23:33:24.182577+0000","last_clean":"2024-03-28T23:33:24.182577+0000","last_became_active":"2024-03-28T23:33:24.179327+0000","last_became_peered":"2024-03-28T23:33:24.179327+0000","last_unstale":"2024-03-28T23:33:24.182577+0000","last_undegraded":"2024-03-28T23:33:24.182577+0000","last_fullsized":"2024-03-28T23:33:24.182577+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:12:17.544325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.182225+0000","last_change":"2024-03-28T23:33:24.178557+0000","last_active":"2024-03-28T23:33:24.182225+0000","last_peered":"2024-03-28T23:33:24.182225+0000","last_clean":"2024-03-28T23:33:24.182225+0000","last_became_active":"2024-03-28T23:33:24.178255+0000","last_became_peered":"2024-03-28T23:33:24.178255+0000","last_unstale":"2024-03-28T23:33:24.182225+0000","last_undegraded":"2024-03-28T23:33:24.182225+0000","last_fullsized":"2024-03-28T23:33:24.182225+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:01:13.611000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.187775+0000","last_change":"2024-03-28T23:33:24.180649+0000","last_active":"2024-03-28T23:33:24.187775+0000","last_peered":"2024-03-28T23:33:24.187775+0000","last_clean":"2024-03-28T23:33:24.187775+0000","last_became_active":"2024-03-28T23:33:24.180014+0000","last_became_peered":"2024-03-28T23:33:24.180014+0000","last_unstale":"2024-03-28T23:33:24.187775+0000","last_undegraded":"2024-03-28T23:33:24.187775+0000","last_fullsized":"2024-03-28T23:33:24.187775+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:56:45.330962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.175289+0000","last_change":"2024-03-28T23:33:24.173554+0000","last_active":"2024-03-28T23:33:24.175289+0000","last_peered":"2024-03-28T23:33:24.175289+0000","last_clean":"2024-03-28T23:33:24.175289+0000","last_became_active":"2024-03-28T23:33:24.173276+0000","last_became_peered":"2024-03-28T23:33:24.173276+0000","last_unstale":"2024-03-28T23:33:24.175289+0000","last_undegraded":"2024-03-28T23:33:24.175289+0000","last_fullsized":"2024-03-28T23:33:24.175289+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:20:57.071533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.176905+0000","last_change":"2024-03-28T23:33:24.170732+0000","last_active":"2024-03-28T23:33:24.176905+0000","last_peered":"2024-03-28T23:33:24.176905+0000","last_clean":"2024-03-28T23:33:24.176905+0000","last_became_active":"2024-03-28T23:33:24.170369+0000","last_became_peered":"2024-03-28T23:33:24.170369+0000","last_unstale":"2024-03-28T23:33:24.176905+0000","last_undegraded":"2024-03-28T23:33:24.176905+0000","last_fullsized":"2024-03-28T23:33:24.176905+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:50:58.694649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.182473+0000","last_change":"2024-03-28T23:33:24.176950+0000","last_active":"2024-03-28T23:33:24.182473+0000","last_peered":"2024-03-28T23:33:24.182473+0000","last_clean":"2024-03-28T23:33:24.182473+0000","last_became_active":"2024-03-28T23:33:24.176142+0000","last_became_peered":"2024-03-28T23:33:24.176142+0000","last_unstale":"2024-03-28T23:33:24.182473+0000","last_undegraded":"2024-03-28T23:33:24.182473+0000","last_fullsized":"2024-03-28T23:33:24.182473+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:22.409061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.183014+0000","last_change":"2024-03-28T23:33:24.172744+0000","last_active":"2024-03-28T23:33:24.183014+0000","last_peered":"2024-03-28T23:33:24.183014+0000","last_clean":"2024-03-28T23:33:24.183014+0000","last_became_active":"2024-03-28T23:33:24.172470+0000","last_became_peered":"2024-03-28T23:33:24.172470+0000","last_unstale":"2024-03-28T23:33:24.183014+0000","last_undegraded":"2024-03-28T23:33:24.183014+0000","last_fullsized":"2024-03-28T23:33:24.183014+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:10:23.895818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.179879+0000","last_change":"2024-03-28T23:33:24.175934+0000","last_active":"2024-03-28T23:33:24.179879+0000","last_peered":"2024-03-28T23:33:24.179879+0000","last_clean":"2024-03-28T23:33:24.179879+0000","last_became_active":"2024-03-28T23:33:24.175660+0000","last_became_peered":"2024-03-28T23:33:24.175660+0000","last_unstale":"2024-03-28T23:33:24.179879+0000","last_undegraded":"2024-03-28T23:33:24.179879+0000","last_fullsized":"2024-03-28T23:33:24.179879+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:30:19.952741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.178083+0000","last_change":"2024-03-28T23:33:24.177353+0000","last_active":"2024-03-28T23:33:24.178083+0000","last_peered":"2024-03-28T23:33:24.178083+0000","last_clean":"2024-03-28T23:33:24.178083+0000","last_became_active":"2024-03-28T23:33:24.177083+0000","last_became_peered":"2024-03-28T23:33:24.177083+0000","last_unstale":"2024-03-28T23:33:24.178083+0000","last_undegraded":"2024-03-28T23:33:24.178083+0000","last_fullsized":"2024-03-28T23:33:24.178083+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:53:16.718079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.182300+0000","last_change":"2024-03-28T23:33:24.180686+0000","last_active":"2024-03-28T23:33:24.182300+0000","last_peered":"2024-03-28T23:33:24.182300+0000","last_clean":"2024-03-28T23:33:24.182300+0000","last_became_active":"2024-03-28T23:33:24.180430+0000","last_became_peered":"2024-03-28T23:33:24.180430+0000","last_unstale":"2024-03-28T23:33:24.182300+0000","last_undegraded":"2024-03-28T23:33:24.182300+0000","last_fullsized":"2024-03-28T23:33:24.182300+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:55:03.722583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.184111+0000","last_change":"2024-03-28T23:33:24.183301+0000","last_active":"2024-03-28T23:33:24.184111+0000","last_peered":"2024-03-28T23:33:24.184111+0000","last_clean":"2024-03-28T23:33:24.184111+0000","last_became_active":"2024-03-28T23:33:24.183011+0000","last_became_peered":"2024-03-28T23:33:24.183011+0000","last_unstale":"2024-03-28T23:33:24.184111+0000","last_undegraded":"2024-03-28T23:33:24.184111+0000","last_fullsized":"2024-03-28T23:33:24.184111+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:34.756638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.189831+0000","last_change":"2024-03-28T23:33:24.182323+0000","last_active":"2024-03-28T23:33:24.189831+0000","last_peered":"2024-03-28T23:33:24.189831+0000","last_clean":"2024-03-28T23:33:24.189831+0000","last_became_active":"2024-03-28T23:33:24.182063+0000","last_became_peered":"2024-03-28T23:33:24.182063+0000","last_unstale":"2024-03-28T23:33:24.189831+0000","last_undegraded":"2024-03-28T23:33:24.189831+0000","last_fullsized":"2024-03-28T23:33:24.189831+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:15:45.913062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.181378+0000","last_change":"2024-03-28T23:33:24.178952+0000","last_active":"2024-03-28T23:33:24.181378+0000","last_peered":"2024-03-28T23:33:24.181378+0000","last_clean":"2024-03-28T23:33:24.181378+0000","last_became_active":"2024-03-28T23:33:24.178564+0000","last_became_peered":"2024-03-28T23:33:24.178564+0000","last_unstale":"2024-03-28T23:33:24.181378+0000","last_undegraded":"2024-03-28T23:33:24.181378+0000","last_fullsized":"2024-03-28T23:33:24.181378+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:21:46.670200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.193052+0000","last_change":"2024-03-28T23:33:24.187480+0000","last_active":"2024-03-28T23:33:24.193052+0000","last_peered":"2024-03-28T23:33:24.193052+0000","last_clean":"2024-03-28T23:33:24.193052+0000","last_became_active":"2024-03-28T23:33:24.185232+0000","last_became_peered":"2024-03-28T23:33:24.185232+0000","last_unstale":"2024-03-28T23:33:24.193052+0000","last_undegraded":"2024-03-28T23:33:24.193052+0000","last_fullsized":"2024-03-28T23:33:24.193052+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:19:34.917882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.182990+0000","last_change":"2024-03-28T23:33:24.169680+0000","last_active":"2024-03-28T23:33:24.182990+0000","last_peered":"2024-03-28T23:33:24.182990+0000","last_clean":"2024-03-28T23:33:24.182990+0000","last_became_active":"2024-03-28T23:33:24.169268+0000","last_became_peered":"2024-03-28T23:33:24.169268+0000","last_unstale":"2024-03-28T23:33:24.182990+0000","last_undegraded":"2024-03-28T23:33:24.182990+0000","last_fullsized":"2024-03-28T23:33:24.182990+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:49.980134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.185926+0000","last_change":"2024-03-28T23:33:24.175760+0000","last_active":"2024-03-28T23:33:24.185926+0000","last_peered":"2024-03-28T23:33:24.185926+0000","last_clean":"2024-03-28T23:33:24.185926+0000","last_became_active":"2024-03-28T23:33:24.175477+0000","last_became_peered":"2024-03-28T23:33:24.175477+0000","last_unstale":"2024-03-28T23:33:24.185926+0000","last_undegraded":"2024-03-28T23:33:24.185926+0000","last_fullsized":"2024-03-28T23:33:24.185926+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:56:44.649346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.190630+0000","last_change":"2024-03-28T23:33:24.178678+0000","last_active":"2024-03-28T23:33:24.190630+0000","last_peered":"2024-03-28T23:33:24.190630+0000","last_clean":"2024-03-28T23:33:24.190630+0000","last_became_active":"2024-03-28T23:33:24.178338+0000","last_became_peered":"2024-03-28T23:33:24.178338+0000","last_unstale":"2024-03-28T23:33:24.190630+0000","last_undegraded":"2024-03-28T23:33:24.190630+0000","last_fullsized":"2024-03-28T23:33:24.190630+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:10.325868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.189913+0000","last_change":"2024-03-28T23:33:24.183063+0000","last_active":"2024-03-28T23:33:24.189913+0000","last_peered":"2024-03-28T23:33:24.189913+0000","last_clean":"2024-03-28T23:33:24.189913+0000","last_became_active":"2024-03-28T23:33:24.182747+0000","last_became_peered":"2024-03-28T23:33:24.182747+0000","last_unstale":"2024-03-28T23:33:24.189913+0000","last_undegraded":"2024-03-28T23:33:24.189913+0000","last_fullsized":"2024-03-28T23:33:24.189913+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:50:42.702642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.194870+0000","last_change":"2024-03-28T23:33:24.184579+0000","last_active":"2024-03-28T23:33:24.194870+0000","last_peered":"2024-03-28T23:33:24.194870+0000","last_clean":"2024-03-28T23:33:24.194870+0000","last_became_active":"2024-03-28T23:33:24.184306+0000","last_became_peered":"2024-03-28T23:33:24.184306+0000","last_unstale":"2024-03-28T23:33:24.194870+0000","last_undegraded":"2024-03-28T23:33:24.194870+0000","last_fullsized":"2024-03-28T23:33:24.194870+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:10:23.895818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.181003+0000","last_change":"2024-03-28T23:33:24.175881+0000","last_active":"2024-03-28T23:33:24.181003+0000","last_peered":"2024-03-28T23:33:24.181003+0000","last_clean":"2024-03-28T23:33:24.181003+0000","last_became_active":"2024-03-28T23:33:24.175608+0000","last_became_peered":"2024-03-28T23:33:24.175608+0000","last_unstale":"2024-03-28T23:33:24.181003+0000","last_undegraded":"2024-03-28T23:33:24.181003+0000","last_fullsized":"2024-03-28T23:33:24.181003+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:03:43.530002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.181377+0000","last_change":"2024-03-28T23:33:24.170191+0000","last_active":"2024-03-28T23:33:24.181377+0000","last_peered":"2024-03-28T23:33:24.181377+0000","last_clean":"2024-03-28T23:33:24.181377+0000","last_became_active":"2024-03-28T23:33:24.169865+0000","last_became_peered":"2024-03-28T23:33:24.169865+0000","last_unstale":"2024-03-28T23:33:24.181377+0000","last_undegraded":"2024-03-28T23:33:24.181377+0000","last_fullsized":"2024-03-28T23:33:24.181377+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:22.409061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.174582+0000","last_change":"2024-03-28T23:33:24.173890+0000","last_active":"2024-03-28T23:33:24.174582+0000","last_peered":"2024-03-28T23:33:24.174582+0000","last_clean":"2024-03-28T23:33:24.174582+0000","last_became_active":"2024-03-28T23:33:24.173627+0000","last_became_peered":"2024-03-28T23:33:24.173627+0000","last_unstale":"2024-03-28T23:33:24.174582+0000","last_undegraded":"2024-03-28T23:33:24.174582+0000","last_fullsized":"2024-03-28T23:33:24.174582+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:50:58.694649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.178840+0000","last_change":"2024-03-28T23:33:24.177638+0000","last_active":"2024-03-28T23:33:24.178840+0000","last_peered":"2024-03-28T23:33:24.178840+0000","last_clean":"2024-03-28T23:33:24.178840+0000","last_became_active":"2024-03-28T23:33:24.177338+0000","last_became_peered":"2024-03-28T23:33:24.177338+0000","last_unstale":"2024-03-28T23:33:24.178840+0000","last_undegraded":"2024-03-28T23:33:24.178840+0000","last_fullsized":"2024-03-28T23:33:24.178840+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:36:33.138725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.173244+0000","last_change":"2024-03-28T23:33:24.170334+0000","last_active":"2024-03-28T23:33:24.173244+0000","last_peered":"2024-03-28T23:33:24.173244+0000","last_clean":"2024-03-28T23:33:24.173244+0000","last_became_active":"2024-03-28T23:33:24.170068+0000","last_became_peered":"2024-03-28T23:33:24.170068+0000","last_unstale":"2024-03-28T23:33:24.173244+0000","last_undegraded":"2024-03-28T23:33:24.173244+0000","last_fullsized":"2024-03-28T23:33:24.173244+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:29:27.672418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.181102+0000","last_change":"2024-03-28T23:33:24.177514+0000","last_active":"2024-03-28T23:33:24.181102+0000","last_peered":"2024-03-28T23:33:24.181102+0000","last_clean":"2024-03-28T23:33:24.181102+0000","last_became_active":"2024-03-28T23:33:24.177239+0000","last_became_peered":"2024-03-28T23:33:24.177239+0000","last_unstale":"2024-03-28T23:33:24.181102+0000","last_undegraded":"2024-03-28T23:33:24.181102+0000","last_fullsized":"2024-03-28T23:33:24.181102+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:14:06.775695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.186383+0000","last_change":"2024-03-28T23:33:24.185100+0000","last_active":"2024-03-28T23:33:24.186383+0000","last_peered":"2024-03-28T23:33:24.186383+0000","last_clean":"2024-03-28T23:33:24.186383+0000","last_became_active":"2024-03-28T23:33:24.184739+0000","last_became_peered":"2024-03-28T23:33:24.184739+0000","last_unstale":"2024-03-28T23:33:24.186383+0000","last_undegraded":"2024-03-28T23:33:24.186383+0000","last_fullsized":"2024-03-28T23:33:24.186383+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:42:30.721204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.187804+0000","last_change":"2024-03-28T23:33:24.180876+0000","last_active":"2024-03-28T23:33:24.187804+0000","last_peered":"2024-03-28T23:33:24.187804+0000","last_clean":"2024-03-28T23:33:24.187804+0000","last_became_active":"2024-03-28T23:33:24.180384+0000","last_became_peered":"2024-03-28T23:33:24.180384+0000","last_unstale":"2024-03-28T23:33:24.187804+0000","last_undegraded":"2024-03-28T23:33:24.187804+0000","last_fullsized":"2024-03-28T23:33:24.187804+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:59:39.367757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.190744+0000","last_change":"2024-03-28T23:33:24.185656+0000","last_active":"2024-03-28T23:33:24.190744+0000","last_peered":"2024-03-28T23:33:24.190744+0000","last_clean":"2024-03-28T23:33:24.190744+0000","last_became_active":"2024-03-28T23:33:24.185096+0000","last_became_peered":"2024-03-28T23:33:24.185096+0000","last_unstale":"2024-03-28T23:33:24.190744+0000","last_undegraded":"2024-03-28T23:33:24.190744+0000","last_fullsized":"2024-03-28T23:33:24.190744+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:38.115317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"93'4","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.196053+0000","last_change":"2024-03-28T23:33:23.170656+0000","last_active":"2024-03-28T23:33:24.196053+0000","last_peered":"2024-03-28T23:33:24.196053+0000","last_clean":"2024-03-28T23:33:24.196053+0000","last_became_active":"2024-03-28T23:33:23.170381+0000","last_became_peered":"2024-03-28T23:33:23.170381+0000","last_unstale":"2024-03-28T23:33:24.196053+0000","last_undegraded":"2024-03-28T23:33:24.196053+0000","last_fullsized":"2024-03-28T23:33:24.196053+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:14:05.771188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.184918+0000","last_change":"2024-03-28T23:33:24.178795+0000","last_active":"2024-03-28T23:33:24.184918+0000","last_peered":"2024-03-28T23:33:24.184918+0000","last_clean":"2024-03-28T23:33:24.184918+0000","last_became_active":"2024-03-28T23:33:24.178513+0000","last_became_peered":"2024-03-28T23:33:24.178513+0000","last_unstale":"2024-03-28T23:33:24.184918+0000","last_undegraded":"2024-03-28T23:33:24.184918+0000","last_fullsized":"2024-03-28T23:33:24.184918+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:06:26.191545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.192098+0000","last_change":"2024-03-28T23:33:23.172712+0000","last_active":"2024-03-28T23:33:24.192098+0000","last_peered":"2024-03-28T23:33:24.192098+0000","last_clean":"2024-03-28T23:33:24.192098+0000","last_became_active":"2024-03-28T23:33:23.171741+0000","last_became_peered":"2024-03-28T23:33:23.171741+0000","last_unstale":"2024-03-28T23:33:24.192098+0000","last_undegraded":"2024-03-28T23:33:24.192098+0000","last_fullsized":"2024-03-28T23:33:24.192098+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:36:06.822902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.174510+0000","last_change":"2024-03-28T23:33:24.165989+0000","last_active":"2024-03-28T23:33:24.174510+0000","last_peered":"2024-03-28T23:33:24.174510+0000","last_clean":"2024-03-28T23:33:24.174510+0000","last_became_active":"2024-03-28T23:33:24.165680+0000","last_became_peered":"2024-03-28T23:33:24.165680+0000","last_unstale":"2024-03-28T23:33:24.174510+0000","last_undegraded":"2024-03-28T23:33:24.174510+0000","last_fullsized":"2024-03-28T23:33:24.174510+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:50:58.694649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.191935+0000","last_change":"2024-03-28T23:33:23.176865+0000","last_active":"2024-03-28T23:33:24.191935+0000","last_peered":"2024-03-28T23:33:24.191935+0000","last_clean":"2024-03-28T23:33:24.191935+0000","last_became_active":"2024-03-28T23:33:23.176423+0000","last_became_peered":"2024-03-28T23:33:23.176423+0000","last_unstale":"2024-03-28T23:33:24.191935+0000","last_undegraded":"2024-03-28T23:33:24.191935+0000","last_fullsized":"2024-03-28T23:33:24.191935+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:27:35.509357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.184196+0000","last_change":"2024-03-28T23:33:24.182219+0000","last_active":"2024-03-28T23:33:24.184196+0000","last_peered":"2024-03-28T23:33:24.184196+0000","last_clean":"2024-03-28T23:33:24.184196+0000","last_became_active":"2024-03-28T23:33:24.181938+0000","last_became_peered":"2024-03-28T23:33:24.181938+0000","last_unstale":"2024-03-28T23:33:24.184196+0000","last_undegraded":"2024-03-28T23:33:24.184196+0000","last_fullsized":"2024-03-28T23:33:24.184196+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:14:06.775695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.199067+0000","last_change":"2024-03-28T23:33:23.175536+0000","last_active":"2024-03-28T23:33:24.199067+0000","last_peered":"2024-03-28T23:33:24.199067+0000","last_clean":"2024-03-28T23:33:24.199067+0000","last_became_active":"2024-03-28T23:33:23.175276+0000","last_became_peered":"2024-03-28T23:33:23.175276+0000","last_unstale":"2024-03-28T23:33:24.199067+0000","last_undegraded":"2024-03-28T23:33:24.199067+0000","last_fullsized":"2024-03-28T23:33:24.199067+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:45:07.164863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.179255+0000","last_change":"2024-03-28T23:33:24.173588+0000","last_active":"2024-03-28T23:33:24.179255+0000","last_peered":"2024-03-28T23:33:24.179255+0000","last_clean":"2024-03-28T23:33:24.179255+0000","last_became_active":"2024-03-28T23:33:24.173299+0000","last_became_peered":"2024-03-28T23:33:24.173299+0000","last_unstale":"2024-03-28T23:33:24.179255+0000","last_undegraded":"2024-03-28T23:33:24.179255+0000","last_fullsized":"2024-03-28T23:33:24.179255+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:23:34.756638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.190274+0000","last_change":"2024-03-28T23:33:23.168249+0000","last_active":"2024-03-28T23:33:24.190274+0000","last_peered":"2024-03-28T23:33:24.190274+0000","last_clean":"2024-03-28T23:33:24.190274+0000","last_became_active":"2024-03-28T23:33:23.167942+0000","last_became_peered":"2024-03-28T23:33:23.167942+0000","last_unstale":"2024-03-28T23:33:24.190274+0000","last_undegraded":"2024-03-28T23:33:24.190274+0000","last_fullsized":"2024-03-28T23:33:24.190274+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:03:42.525496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.190772+0000","last_change":"2024-03-28T23:33:24.185716+0000","last_active":"2024-03-28T23:33:24.190772+0000","last_peered":"2024-03-28T23:33:24.190772+0000","last_clean":"2024-03-28T23:33:24.190772+0000","last_became_active":"2024-03-28T23:33:24.184900+0000","last_became_peered":"2024-03-28T23:33:24.184900+0000","last_unstale":"2024-03-28T23:33:24.190772+0000","last_undegraded":"2024-03-28T23:33:24.190772+0000","last_fullsized":"2024-03-28T23:33:24.190772+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:24:02.991859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.191388+0000","last_change":"2024-03-28T23:33:23.177878+0000","last_active":"2024-03-28T23:33:24.191388+0000","last_peered":"2024-03-28T23:33:24.191388+0000","last_clean":"2024-03-28T23:33:24.191388+0000","last_became_active":"2024-03-28T23:33:23.177517+0000","last_became_peered":"2024-03-28T23:33:23.177517+0000","last_unstale":"2024-03-28T23:33:24.191388+0000","last_undegraded":"2024-03-28T23:33:24.191388+0000","last_fullsized":"2024-03-28T23:33:24.191388+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:54.634069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.181357+0000","last_change":"2024-03-28T23:33:24.178844+0000","last_active":"2024-03-28T23:33:24.181357+0000","last_peered":"2024-03-28T23:33:24.181357+0000","last_clean":"2024-03-28T23:33:24.181357+0000","last_became_active":"2024-03-28T23:33:24.178515+0000","last_became_peered":"2024-03-28T23:33:24.178515+0000","last_unstale":"2024-03-28T23:33:24.181357+0000","last_undegraded":"2024-03-28T23:33:24.181357+0000","last_fullsized":"2024-03-28T23:33:24.181357+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:41:07.386479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.161260+0000","last_change":"2024-03-28T23:33:23.176695+0000","last_active":"2024-03-28T23:33:24.161260+0000","last_peered":"2024-03-28T23:33:24.161260+0000","last_clean":"2024-03-28T23:33:24.161260+0000","last_became_active":"2024-03-28T23:33:23.176392+0000","last_became_peered":"2024-03-28T23:33:23.176392+0000","last_unstale":"2024-03-28T23:33:24.161260+0000","last_undegraded":"2024-03-28T23:33:24.161260+0000","last_fullsized":"2024-03-28T23:33:24.161260+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:15:32.758066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.194949+0000","last_change":"2024-03-28T23:33:24.189572+0000","last_active":"2024-03-28T23:33:24.194949+0000","last_peered":"2024-03-28T23:33:24.194949+0000","last_clean":"2024-03-28T23:33:24.194949+0000","last_became_active":"2024-03-28T23:33:24.188586+0000","last_became_peered":"2024-03-28T23:33:24.188586+0000","last_unstale":"2024-03-28T23:33:24.194949+0000","last_undegraded":"2024-03-28T23:33:24.194949+0000","last_fullsized":"2024-03-28T23:33:24.194949+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:07:37.632177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.162304+0000","last_change":"2024-03-28T23:33:23.172072+0000","last_active":"2024-03-28T23:33:24.162304+0000","last_peered":"2024-03-28T23:33:24.162304+0000","last_clean":"2024-03-28T23:33:24.162304+0000","last_became_active":"2024-03-28T23:33:23.171595+0000","last_became_peered":"2024-03-28T23:33:23.171595+0000","last_unstale":"2024-03-28T23:33:24.162304+0000","last_undegraded":"2024-03-28T23:33:24.162304+0000","last_fullsized":"2024-03-28T23:33:24.162304+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:19:30.535356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.193043+0000","last_change":"2024-03-28T23:33:24.187457+0000","last_active":"2024-03-28T23:33:24.193043+0000","last_peered":"2024-03-28T23:33:24.193043+0000","last_clean":"2024-03-28T23:33:24.193043+0000","last_became_active":"2024-03-28T23:33:24.185415+0000","last_became_peered":"2024-03-28T23:33:24.185415+0000","last_unstale":"2024-03-28T23:33:24.193043+0000","last_undegraded":"2024-03-28T23:33:24.193043+0000","last_fullsized":"2024-03-28T23:33:24.193043+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:25:51.087406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.161966+0000","last_change":"2024-03-28T23:33:23.176967+0000","last_active":"2024-03-28T23:33:24.161966+0000","last_peered":"2024-03-28T23:33:24.161966+0000","last_clean":"2024-03-28T23:33:24.161966+0000","last_became_active":"2024-03-28T23:33:23.176705+0000","last_became_peered":"2024-03-28T23:33:23.176705+0000","last_unstale":"2024-03-28T23:33:24.161966+0000","last_undegraded":"2024-03-28T23:33:24.161966+0000","last_fullsized":"2024-03-28T23:33:24.161966+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:54.634069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.176828+0000","last_change":"2024-03-28T23:33:24.169083+0000","last_active":"2024-03-28T23:33:24.176828+0000","last_peered":"2024-03-28T23:33:24.176828+0000","last_clean":"2024-03-28T23:33:24.176828+0000","last_became_active":"2024-03-28T23:33:24.168715+0000","last_became_peered":"2024-03-28T23:33:24.168715+0000","last_unstale":"2024-03-28T23:33:24.176828+0000","last_undegraded":"2024-03-28T23:33:24.176828+0000","last_fullsized":"2024-03-28T23:33:24.176828+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:55.638575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.177756+0000","last_change":"2024-03-28T23:33:23.173662+0000","last_active":"2024-03-28T23:33:24.177756+0000","last_peered":"2024-03-28T23:33:24.177756+0000","last_clean":"2024-03-28T23:33:24.177756+0000","last_became_active":"2024-03-28T23:33:23.173401+0000","last_became_peered":"2024-03-28T23:33:23.173401+0000","last_unstale":"2024-03-28T23:33:24.177756+0000","last_undegraded":"2024-03-28T23:33:24.177756+0000","last_fullsized":"2024-03-28T23:33:24.177756+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:38:10.896945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.185521+0000","last_change":"2024-03-28T23:33:24.177215+0000","last_active":"2024-03-28T23:33:24.185521+0000","last_peered":"2024-03-28T23:33:24.185521+0000","last_clean":"2024-03-28T23:33:24.185521+0000","last_became_active":"2024-03-28T23:33:24.176953+0000","last_became_peered":"2024-03-28T23:33:24.176953+0000","last_unstale":"2024-03-28T23:33:24.185521+0000","last_undegraded":"2024-03-28T23:33:24.185521+0000","last_fullsized":"2024-03-28T23:33:24.185521+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:03:36.327075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"93'3","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.193964+0000","last_change":"2024-03-28T23:33:23.170496+0000","last_active":"2024-03-28T23:33:24.193964+0000","last_peered":"2024-03-28T23:33:24.193964+0000","last_clean":"2024-03-28T23:33:24.193964+0000","last_became_active":"2024-03-28T23:33:23.170190+0000","last_became_peered":"2024-03-28T23:33:23.170190+0000","last_unstale":"2024-03-28T23:33:24.193964+0000","last_undegraded":"2024-03-28T23:33:24.193964+0000","last_fullsized":"2024-03-28T23:33:24.193964+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:14:05.771188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.177119+0000","last_change":"2024-03-28T23:33:24.172490+0000","last_active":"2024-03-28T23:33:24.177119+0000","last_peered":"2024-03-28T23:33:24.177119+0000","last_clean":"2024-03-28T23:33:24.177119+0000","last_became_active":"2024-03-28T23:33:24.172141+0000","last_became_peered":"2024-03-28T23:33:24.172141+0000","last_unstale":"2024-03-28T23:33:24.177119+0000","last_undegraded":"2024-03-28T23:33:24.177119+0000","last_fullsized":"2024-03-28T23:33:24.177119+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:49.980134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"93'6","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.199362+0000","last_change":"2024-03-28T23:33:23.170242+0000","last_active":"2024-03-28T23:33:24.199362+0000","last_peered":"2024-03-28T23:33:24.199362+0000","last_clean":"2024-03-28T23:33:24.199362+0000","last_became_active":"2024-03-28T23:33:23.169734+0000","last_became_peered":"2024-03-28T23:33:23.169734+0000","last_unstale":"2024-03-28T23:33:24.199362+0000","last_undegraded":"2024-03-28T23:33:24.199362+0000","last_fullsized":"2024-03-28T23:33:24.199362+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:17:25.021229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.184189+0000","last_change":"2024-03-28T23:33:24.181161+0000","last_active":"2024-03-28T23:33:24.184189+0000","last_peered":"2024-03-28T23:33:24.184189+0000","last_clean":"2024-03-28T23:33:24.184189+0000","last_became_active":"2024-03-28T23:33:24.180787+0000","last_became_peered":"2024-03-28T23:33:24.180787+0000","last_unstale":"2024-03-28T23:33:24.184189+0000","last_undegraded":"2024-03-28T23:33:24.184189+0000","last_fullsized":"2024-03-28T23:33:24.184189+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:03:43.530002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.199172+0000","last_change":"2024-03-28T23:33:23.169282+0000","last_active":"2024-03-28T23:33:24.199172+0000","last_peered":"2024-03-28T23:33:24.199172+0000","last_clean":"2024-03-28T23:33:24.199172+0000","last_became_active":"2024-03-28T23:33:23.168883+0000","last_became_peered":"2024-03-28T23:33:23.168883+0000","last_unstale":"2024-03-28T23:33:24.199172+0000","last_undegraded":"2024-03-28T23:33:24.199172+0000","last_fullsized":"2024-03-28T23:33:24.199172+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:27:35.509357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.185363+0000","last_change":"2024-03-28T23:33:24.183839+0000","last_active":"2024-03-28T23:33:24.185363+0000","last_peered":"2024-03-28T23:33:24.185363+0000","last_clean":"2024-03-28T23:33:24.185363+0000","last_became_active":"2024-03-28T23:33:24.182905+0000","last_became_peered":"2024-03-28T23:33:24.182905+0000","last_unstale":"2024-03-28T23:33:24.185363+0000","last_undegraded":"2024-03-28T23:33:24.185363+0000","last_fullsized":"2024-03-28T23:33:24.185363+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:45:08.169369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.164157+0000","last_change":"2024-03-28T23:33:23.173412+0000","last_active":"2024-03-28T23:33:24.164157+0000","last_peered":"2024-03-28T23:33:24.164157+0000","last_clean":"2024-03-28T23:33:24.164157+0000","last_became_active":"2024-03-28T23:33:23.173156+0000","last_became_peered":"2024-03-28T23:33:23.173156+0000","last_unstale":"2024-03-28T23:33:24.164157+0000","last_undegraded":"2024-03-28T23:33:24.164157+0000","last_fullsized":"2024-03-28T23:33:24.164157+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:53:15.713573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.187238+0000","last_change":"2024-03-28T23:33:24.181647+0000","last_active":"2024-03-28T23:33:24.187238+0000","last_peered":"2024-03-28T23:33:24.187238+0000","last_clean":"2024-03-28T23:33:24.187238+0000","last_became_active":"2024-03-28T23:33:24.181298+0000","last_became_peered":"2024-03-28T23:33:24.181298+0000","last_unstale":"2024-03-28T23:33:24.187238+0000","last_undegraded":"2024-03-28T23:33:24.187238+0000","last_fullsized":"2024-03-28T23:33:24.187238+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:45:49.920682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.167900+0000","last_change":"2024-03-28T23:33:23.177730+0000","last_active":"2024-03-28T23:33:24.167900+0000","last_peered":"2024-03-28T23:33:24.167900+0000","last_clean":"2024-03-28T23:33:24.167900+0000","last_became_active":"2024-03-28T23:33:23.177471+0000","last_became_peered":"2024-03-28T23:33:23.177471+0000","last_unstale":"2024-03-28T23:33:24.167900+0000","last_undegraded":"2024-03-28T23:33:24.167900+0000","last_fullsized":"2024-03-28T23:33:24.167900+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:18:35.677144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.190752+0000","last_change":"2024-03-28T23:33:24.185311+0000","last_active":"2024-03-28T23:33:24.190752+0000","last_peered":"2024-03-28T23:33:24.190752+0000","last_clean":"2024-03-28T23:33:24.190752+0000","last_became_active":"2024-03-28T23:33:24.185050+0000","last_became_peered":"2024-03-28T23:33:24.185050+0000","last_unstale":"2024-03-28T23:33:24.190752+0000","last_undegraded":"2024-03-28T23:33:24.190752+0000","last_fullsized":"2024-03-28T23:33:24.190752+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:02:22.419431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"93'8","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.192595+0000","last_change":"2024-03-28T23:33:23.174388+0000","last_active":"2024-03-28T23:33:24.192595+0000","last_peered":"2024-03-28T23:33:24.192595+0000","last_clean":"2024-03-28T23:33:24.192595+0000","last_became_active":"2024-03-28T23:33:23.174105+0000","last_became_peered":"2024-03-28T23:33:23.174105+0000","last_unstale":"2024-03-28T23:33:24.192595+0000","last_undegraded":"2024-03-28T23:33:24.192595+0000","last_fullsized":"2024-03-28T23:33:24.192595+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:36:47.363804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.192094+0000","last_change":"2024-03-28T23:33:24.187636+0000","last_active":"2024-03-28T23:33:24.192094+0000","last_peered":"2024-03-28T23:33:24.192094+0000","last_clean":"2024-03-28T23:33:24.192094+0000","last_became_active":"2024-03-28T23:33:24.187322+0000","last_became_peered":"2024-03-28T23:33:24.187322+0000","last_unstale":"2024-03-28T23:33:24.192094+0000","last_undegraded":"2024-03-28T23:33:24.192094+0000","last_fullsized":"2024-03-28T23:33:24.192094+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:47:34.781395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.191377+0000","last_change":"2024-03-28T23:33:23.177946+0000","last_active":"2024-03-28T23:33:24.191377+0000","last_peered":"2024-03-28T23:33:24.191377+0000","last_clean":"2024-03-28T23:33:24.191377+0000","last_became_active":"2024-03-28T23:33:23.177631+0000","last_became_peered":"2024-03-28T23:33:23.177631+0000","last_unstale":"2024-03-28T23:33:24.191377+0000","last_undegraded":"2024-03-28T23:33:24.191377+0000","last_fullsized":"2024-03-28T23:33:24.191377+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:58:27.929282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.183946+0000","last_change":"2024-03-28T23:33:24.182377+0000","last_active":"2024-03-28T23:33:24.183946+0000","last_peered":"2024-03-28T23:33:24.183946+0000","last_clean":"2024-03-28T23:33:24.183946+0000","last_became_active":"2024-03-28T23:33:24.182105+0000","last_became_peered":"2024-03-28T23:33:24.182105+0000","last_unstale":"2024-03-28T23:33:24.183946+0000","last_undegraded":"2024-03-28T23:33:24.183946+0000","last_fullsized":"2024-03-28T23:33:24.183946+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:37:11.908701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.191393+0000","last_change":"2024-03-28T23:33:23.173465+0000","last_active":"2024-03-28T23:33:24.191393+0000","last_peered":"2024-03-28T23:33:24.191393+0000","last_clean":"2024-03-28T23:33:24.191393+0000","last_became_active":"2024-03-28T23:33:23.173189+0000","last_became_peered":"2024-03-28T23:33:23.173189+0000","last_unstale":"2024-03-28T23:33:24.191393+0000","last_undegraded":"2024-03-28T23:33:24.191393+0000","last_fullsized":"2024-03-28T23:33:24.191393+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:19:30.535356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.177090+0000","last_change":"2024-03-28T23:33:24.168669+0000","last_active":"2024-03-28T23:33:24.177090+0000","last_peered":"2024-03-28T23:33:24.177090+0000","last_clean":"2024-03-28T23:33:24.177090+0000","last_became_active":"2024-03-28T23:33:24.168389+0000","last_became_peered":"2024-03-28T23:33:24.168389+0000","last_unstale":"2024-03-28T23:33:24.177090+0000","last_undegraded":"2024-03-28T23:33:24.177090+0000","last_fullsized":"2024-03-28T23:33:24.177090+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:15:33.762573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.169857+0000","last_change":"2024-03-28T23:33:23.171658+0000","last_active":"2024-03-28T23:33:24.169857+0000","last_peered":"2024-03-28T23:33:24.169857+0000","last_clean":"2024-03-28T23:33:24.169857+0000","last_became_active":"2024-03-28T23:33:23.171359+0000","last_became_peered":"2024-03-28T23:33:23.171359+0000","last_unstale":"2024-03-28T23:33:24.169857+0000","last_undegraded":"2024-03-28T23:33:24.169857+0000","last_fullsized":"2024-03-28T23:33:24.169857+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:15:32.758066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.184190+0000","last_change":"2024-03-28T23:33:24.180456+0000","last_active":"2024-03-28T23:33:24.184190+0000","last_peered":"2024-03-28T23:33:24.184190+0000","last_clean":"2024-03-28T23:33:24.184190+0000","last_became_active":"2024-03-28T23:33:24.179979+0000","last_became_peered":"2024-03-28T23:33:24.179979+0000","last_unstale":"2024-03-28T23:33:24.184190+0000","last_undegraded":"2024-03-28T23:33:24.184190+0000","last_fullsized":"2024-03-28T23:33:24.184190+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:45:25.533216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.193534+0000","last_change":"2024-03-28T23:33:23.169314+0000","last_active":"2024-03-28T23:33:24.193534+0000","last_peered":"2024-03-28T23:33:24.193534+0000","last_clean":"2024-03-28T23:33:24.193534+0000","last_became_active":"2024-03-28T23:33:23.169060+0000","last_became_peered":"2024-03-28T23:33:23.169060+0000","last_unstale":"2024-03-28T23:33:24.193534+0000","last_undegraded":"2024-03-28T23:33:24.193534+0000","last_fullsized":"2024-03-28T23:33:24.193534+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:44:26.078995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.199313+0000","last_change":"2024-03-28T23:33:24.195736+0000","last_active":"2024-03-28T23:33:24.199313+0000","last_peered":"2024-03-28T23:33:24.199313+0000","last_clean":"2024-03-28T23:33:24.199313+0000","last_became_active":"2024-03-28T23:33:24.195532+0000","last_became_peered":"2024-03-28T23:33:24.195532+0000","last_unstale":"2024-03-28T23:33:24.199313+0000","last_undegraded":"2024-03-28T23:33:24.199313+0000","last_fullsized":"2024-03-28T23:33:24.199313+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:10.385852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.167401+0000","last_change":"2024-03-28T23:33:23.173074+0000","last_active":"2024-03-28T23:33:24.167401+0000","last_peered":"2024-03-28T23:33:24.167401+0000","last_clean":"2024-03-28T23:33:24.167401+0000","last_became_active":"2024-03-28T23:33:23.171817+0000","last_became_peered":"2024-03-28T23:33:23.171817+0000","last_unstale":"2024-03-28T23:33:24.167401+0000","last_undegraded":"2024-03-28T23:33:24.167401+0000","last_fullsized":"2024-03-28T23:33:24.167401+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:14:25.053743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.171427+0000","last_change":"2024-03-28T23:33:24.170189+0000","last_active":"2024-03-28T23:33:24.171427+0000","last_peered":"2024-03-28T23:33:24.171427+0000","last_clean":"2024-03-28T23:33:24.171427+0000","last_became_active":"2024-03-28T23:33:24.169830+0000","last_became_peered":"2024-03-28T23:33:24.169830+0000","last_unstale":"2024-03-28T23:33:24.171427+0000","last_undegraded":"2024-03-28T23:33:24.171427+0000","last_fullsized":"2024-03-28T23:33:24.171427+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:05:18.562852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.168634+0000","last_change":"2024-03-28T23:33:23.172293+0000","last_active":"2024-03-28T23:33:24.168634+0000","last_peered":"2024-03-28T23:33:24.168634+0000","last_clean":"2024-03-28T23:33:24.168634+0000","last_became_active":"2024-03-28T23:33:23.171915+0000","last_became_peered":"2024-03-28T23:33:23.171915+0000","last_unstale":"2024-03-28T23:33:24.168634+0000","last_undegraded":"2024-03-28T23:33:24.168634+0000","last_fullsized":"2024-03-28T23:33:24.168634+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:58:23.535559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.184942+0000","last_change":"2024-03-28T23:33:24.178084+0000","last_active":"2024-03-28T23:33:24.184942+0000","last_peered":"2024-03-28T23:33:24.184942+0000","last_clean":"2024-03-28T23:33:24.184942+0000","last_became_active":"2024-03-28T23:33:24.177784+0000","last_became_peered":"2024-03-28T23:33:24.177784+0000","last_unstale":"2024-03-28T23:33:24.184942+0000","last_undegraded":"2024-03-28T23:33:24.184942+0000","last_fullsized":"2024-03-28T23:33:24.184942+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:32:59.609786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.167342+0000","last_change":"2024-03-28T23:33:23.167218+0000","last_active":"2024-03-28T23:33:24.167342+0000","last_peered":"2024-03-28T23:33:24.167342+0000","last_clean":"2024-03-28T23:33:24.167342+0000","last_became_active":"2024-03-28T23:33:23.166972+0000","last_became_peered":"2024-03-28T23:33:23.166972+0000","last_unstale":"2024-03-28T23:33:24.167342+0000","last_undegraded":"2024-03-28T23:33:24.167342+0000","last_fullsized":"2024-03-28T23:33:24.167342+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:40:29.245461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.184350+0000","last_change":"2024-03-28T23:33:24.180501+0000","last_active":"2024-03-28T23:33:24.184350+0000","last_peered":"2024-03-28T23:33:24.184350+0000","last_clean":"2024-03-28T23:33:24.184350+0000","last_became_active":"2024-03-28T23:33:24.180221+0000","last_became_peered":"2024-03-28T23:33:24.180221+0000","last_unstale":"2024-03-28T23:33:24.184350+0000","last_undegraded":"2024-03-28T23:33:24.184350+0000","last_fullsized":"2024-03-28T23:33:24.184350+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:59.077144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.195639+0000","last_change":"2024-03-28T23:33:23.168646+0000","last_active":"2024-03-28T23:33:24.195639+0000","last_peered":"2024-03-28T23:33:24.195639+0000","last_clean":"2024-03-28T23:33:24.195639+0000","last_became_active":"2024-03-28T23:33:23.168396+0000","last_became_peered":"2024-03-28T23:33:23.168396+0000","last_unstale":"2024-03-28T23:33:24.195639+0000","last_undegraded":"2024-03-28T23:33:24.195639+0000","last_fullsized":"2024-03-28T23:33:24.195639+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:36:47.363804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.181785+0000","last_change":"2024-03-28T23:33:24.170667+0000","last_active":"2024-03-28T23:33:24.181785+0000","last_peered":"2024-03-28T23:33:24.181785+0000","last_clean":"2024-03-28T23:33:24.181785+0000","last_became_active":"2024-03-28T23:33:24.170387+0000","last_became_peered":"2024-03-28T23:33:24.170387+0000","last_unstale":"2024-03-28T23:33:24.181785+0000","last_undegraded":"2024-03-28T23:33:24.181785+0000","last_fullsized":"2024-03-28T23:33:24.181785+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:10.325868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.177929+0000","last_change":"2024-03-28T23:33:23.168777+0000","last_active":"2024-03-28T23:33:24.177929+0000","last_peered":"2024-03-28T23:33:24.177929+0000","last_clean":"2024-03-28T23:33:24.177929+0000","last_became_active":"2024-03-28T23:33:23.168535+0000","last_became_peered":"2024-03-28T23:33:23.168535+0000","last_unstale":"2024-03-28T23:33:24.177929+0000","last_undegraded":"2024-03-28T23:33:24.177929+0000","last_fullsized":"2024-03-28T23:33:24.177929+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:39:12.279617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.179385+0000","last_change":"2024-03-28T23:33:24.171534+0000","last_active":"2024-03-28T23:33:24.179385+0000","last_peered":"2024-03-28T23:33:24.179385+0000","last_clean":"2024-03-28T23:33:24.179385+0000","last_became_active":"2024-03-28T23:33:24.171104+0000","last_became_peered":"2024-03-28T23:33:24.171104+0000","last_unstale":"2024-03-28T23:33:24.179385+0000","last_undegraded":"2024-03-28T23:33:24.179385+0000","last_fullsized":"2024-03-28T23:33:24.179385+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:55:03.722583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.174631+0000","last_change":"2024-03-28T23:33:23.171776+0000","last_active":"2024-03-28T23:33:24.174631+0000","last_peered":"2024-03-28T23:33:24.174631+0000","last_clean":"2024-03-28T23:33:24.174631+0000","last_became_active":"2024-03-28T23:33:23.171500+0000","last_became_peered":"2024-03-28T23:33:23.171500+0000","last_unstale":"2024-03-28T23:33:24.174631+0000","last_undegraded":"2024-03-28T23:33:24.174631+0000","last_fullsized":"2024-03-28T23:33:24.174631+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:39:15.619921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.181820+0000","last_change":"2024-03-28T23:33:24.178644+0000","last_active":"2024-03-28T23:33:24.181820+0000","last_peered":"2024-03-28T23:33:24.181820+0000","last_clean":"2024-03-28T23:33:24.181820+0000","last_became_active":"2024-03-28T23:33:24.178305+0000","last_became_peered":"2024-03-28T23:33:24.178305+0000","last_unstale":"2024-03-28T23:33:24.181820+0000","last_undegraded":"2024-03-28T23:33:24.181820+0000","last_fullsized":"2024-03-28T23:33:24.181820+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:10.325868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.161005+0000","last_change":"2024-03-28T23:33:23.169576+0000","last_active":"2024-03-28T23:33:24.161005+0000","last_peered":"2024-03-28T23:33:24.161005+0000","last_clean":"2024-03-28T23:33:24.161005+0000","last_became_active":"2024-03-28T23:33:23.168602+0000","last_became_peered":"2024-03-28T23:33:23.168602+0000","last_unstale":"2024-03-28T23:33:24.161005+0000","last_undegraded":"2024-03-28T23:33:24.161005+0000","last_fullsized":"2024-03-28T23:33:24.161005+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:42:29.716698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.186175+0000","last_change":"2024-03-28T23:33:24.178407+0000","last_active":"2024-03-28T23:33:24.186175+0000","last_peered":"2024-03-28T23:33:24.186175+0000","last_clean":"2024-03-28T23:33:24.186175+0000","last_became_active":"2024-03-28T23:33:24.178112+0000","last_became_peered":"2024-03-28T23:33:24.178112+0000","last_unstale":"2024-03-28T23:33:24.186175+0000","last_undegraded":"2024-03-28T23:33:24.186175+0000","last_fullsized":"2024-03-28T23:33:24.186175+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:51.494690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.160691+0000","last_change":"2024-03-28T23:33:23.174209+0000","last_active":"2024-03-28T23:33:24.160691+0000","last_peered":"2024-03-28T23:33:24.160691+0000","last_clean":"2024-03-28T23:33:24.160691+0000","last_became_active":"2024-03-28T23:33:23.173897+0000","last_became_peered":"2024-03-28T23:33:23.173897+0000","last_unstale":"2024-03-28T23:33:24.160691+0000","last_undegraded":"2024-03-28T23:33:24.160691+0000","last_fullsized":"2024-03-28T23:33:24.160691+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:42:29.716698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.186212+0000","last_change":"2024-03-28T23:33:24.179077+0000","last_active":"2024-03-28T23:33:24.186212+0000","last_peered":"2024-03-28T23:33:24.186212+0000","last_clean":"2024-03-28T23:33:24.186212+0000","last_became_active":"2024-03-28T23:33:24.178735+0000","last_became_peered":"2024-03-28T23:33:24.178735+0000","last_unstale":"2024-03-28T23:33:24.186212+0000","last_undegraded":"2024-03-28T23:33:24.186212+0000","last_fullsized":"2024-03-28T23:33:24.186212+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:39:30.420442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.167594+0000","last_change":"2024-03-28T23:33:23.175820+0000","last_active":"2024-03-28T23:33:24.167594+0000","last_peered":"2024-03-28T23:33:24.167594+0000","last_clean":"2024-03-28T23:33:24.167594+0000","last_became_active":"2024-03-28T23:33:23.175510+0000","last_became_peered":"2024-03-28T23:33:23.175510+0000","last_unstale":"2024-03-28T23:33:24.167594+0000","last_undegraded":"2024-03-28T23:33:24.167594+0000","last_fullsized":"2024-03-28T23:33:24.167594+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:48:58.212882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.182271+0000","last_change":"2024-03-28T23:33:24.179623+0000","last_active":"2024-03-28T23:33:24.182271+0000","last_peered":"2024-03-28T23:33:24.182271+0000","last_clean":"2024-03-28T23:33:24.182271+0000","last_became_active":"2024-03-28T23:33:24.179359+0000","last_became_peered":"2024-03-28T23:33:24.179359+0000","last_unstale":"2024-03-28T23:33:24.182271+0000","last_undegraded":"2024-03-28T23:33:24.182271+0000","last_fullsized":"2024-03-28T23:33:24.182271+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:05:18.562852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.168810+0000","last_change":"2024-03-28T23:33:23.171390+0000","last_active":"2024-03-28T23:33:24.168810+0000","last_peered":"2024-03-28T23:33:24.168810+0000","last_clean":"2024-03-28T23:33:24.168810+0000","last_became_active":"2024-03-28T23:33:23.171087+0000","last_became_peered":"2024-03-28T23:33:23.171087+0000","last_unstale":"2024-03-28T23:33:24.168810+0000","last_undegraded":"2024-03-28T23:33:24.168810+0000","last_fullsized":"2024-03-28T23:33:24.168810+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:13:48.967256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.181466+0000","last_change":"2024-03-28T23:33:24.175821+0000","last_active":"2024-03-28T23:33:24.181466+0000","last_peered":"2024-03-28T23:33:24.181466+0000","last_clean":"2024-03-28T23:33:24.181466+0000","last_became_active":"2024-03-28T23:33:24.175530+0000","last_became_peered":"2024-03-28T23:33:24.175530+0000","last_unstale":"2024-03-28T23:33:24.181466+0000","last_undegraded":"2024-03-28T23:33:24.181466+0000","last_fullsized":"2024-03-28T23:33:24.181466+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:49.980134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.167501+0000","last_change":"2024-03-28T23:33:23.175531+0000","last_active":"2024-03-28T23:33:24.167501+0000","last_peered":"2024-03-28T23:33:24.167501+0000","last_clean":"2024-03-28T23:33:24.167501+0000","last_became_active":"2024-03-28T23:33:23.175226+0000","last_became_peered":"2024-03-28T23:33:23.175226+0000","last_unstale":"2024-03-28T23:33:24.167501+0000","last_undegraded":"2024-03-28T23:33:24.167501+0000","last_fullsized":"2024-03-28T23:33:24.167501+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:33.473881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.181646+0000","last_change":"2024-03-28T23:33:24.180370+0000","last_active":"2024-03-28T23:33:24.181646+0000","last_peered":"2024-03-28T23:33:24.181646+0000","last_clean":"2024-03-28T23:33:24.181646+0000","last_became_active":"2024-03-28T23:33:24.180073+0000","last_became_peered":"2024-03-28T23:33:24.180073+0000","last_unstale":"2024-03-28T23:33:24.181646+0000","last_undegraded":"2024-03-28T23:33:24.181646+0000","last_fullsized":"2024-03-28T23:33:24.181646+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:23.150176+0000","last_clean_scrub_stamp":"2024-03-28T23:33:23.150176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:10:23.895818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.195121+0000","last_change":"2024-03-28T23:33:23.172658+0000","last_active":"2024-03-28T23:33:24.195121+0000","last_peered":"2024-03-28T23:33:24.195121+0000","last_clean":"2024-03-28T23:33:24.195121+0000","last_became_active":"2024-03-28T23:33:23.172368+0000","last_became_peered":"2024-03-28T23:33:23.172368+0000","last_unstale":"2024-03-28T23:33:24.195121+0000","last_undegraded":"2024-03-28T23:33:24.195121+0000","last_fullsized":"2024-03-28T23:33:24.195121+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:22.145670+0000","last_clean_scrub_stamp":"2024-03-28T23:33:22.145670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:35:12.907721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":308,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T23:33:24.166563+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:33:24.166563+0000","last_peered":"2024-03-28T23:33:24.166563+0000","last_clean":"2024-03-28T23:33:24.166563+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:33:24.166563+0000","last_undegraded":"2024-03-28T23:33:24.166563+0000","last_fullsized":"2024-03-28T23:33:24.166563+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1208320,"data_stored":1139446,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32296,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339544,"statfs":{"total":96636764160,"available":96603693056,"internally_reserved":0,"allocated":1142784,"data_stored":549078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574896,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32300,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339540,"statfs":{"total":96636764160,"available":96603688960,"internally_reserved":0,"allocated":1142784,"data_stored":549078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574895,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32296,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339544,"statfs":{"total":96636764160,"available":96603693056,"internally_reserved":0,"allocated":1142784,"data_stored":549078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574895,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28136,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343704,"statfs":{"total":96636764160,"available":96607952896,"internally_reserved":0,"allocated":1142784,"data_stored":549078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574895,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28136,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343704,"statfs":{"total":96636764160,"available":96607952896,"internally_reserved":0,"allocated":1142784,"data_stored":549078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28136,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343704,"statfs":{"total":96636764160,"available":96607952896,"internally_reserved":0,"allocated":1142784,"data_stored":549078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574895,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32396,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339444,"statfs":{"total":96636764160,"available":96603590656,"internally_reserved":0,"allocated":1236992,"data_stored":1148315,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:33:27.486 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T23:33:27.486 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-28T23:33:27.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.626+0000 7fe6d5101640 1 Processor -- start 2024-03-28T23:33:27.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.626+0000 7fe6d5101640 1 -- start start 2024-03-28T23:33:27.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.627+0000 7fe6d5101640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe6d0151e70 0x7fe6d0152250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:27.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.627+0000 7fe6d5101640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6d006cdd0 0x7fe6d015c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:27.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.627+0000 7fe6d5101640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6d00c1e40 con 0x7fe6d0152790 2024-03-28T23:33:27.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.627+0000 7fe6d5101640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fe6d00c2120 con 0x7fe6d0151e70 2024-03-28T23:33:27.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.627+0000 7fe6d5101640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe6d00c2400 con 0x7fe6d006cdd0 2024-03-28T23:33:27.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.627+0000 7fe6ced9d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe6d0151e70 0x7fe6d0152250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:27.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.627+0000 7fe6ced9d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe6d0151e70 0x7fe6d0152250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:37896/0 (socket says 172.21.15.161:37896) 2024-03-28T23:33:27.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.627+0000 7fe6ced9d640 1 -- 172.21.15.161:0/1811721105 learned_addr learned my addr 172.21.15.161:0/1811721105 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:27.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.627+0000 7fe6ce59c640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fe6d0152790 0x7fe6d006c6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:46528/0 (socket says 172.21.15.161:46528) 2024-03-28T23:33:27.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.627+0000 7fe6cf59e640 1 --2- 172.21.15.161:0/1811721105 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6d006cdd0 0x7fe6d015c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:27.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.628+0000 7fe6ced9d640 1 -- 172.21.15.161:0/1811721105 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6d006cdd0 msgr2=0x7fe6d015c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:27.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.628+0000 7fe6ced9d640 1 --2- 172.21.15.161:0/1811721105 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6d006cdd0 0x7fe6d015c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.628+0000 7fe6ced9d640 1 -- 172.21.15.161:0/1811721105 >> v1:172.21.15.170:6789/0 conn(0x7fe6d0152790 legacy=0x7fe6d006c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:27.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.628+0000 7fe6ced9d640 1 -- 172.21.15.161:0/1811721105 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d00c26e0 con 0x7fe6d0151e70 2024-03-28T23:33:27.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.628+0000 7fe6ced9d640 1 --2- 172.21.15.161:0/1811721105 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d0151e70 0x7fe6d0152250 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe6c000bb10 tx=0x7fe6c002e0c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c34349d7e058543 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:27.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.628+0000 7fe6cdd9b640 1 -- 172.21.15.161:0/1811721105 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c003b070 con 0x7fe6d0151e70 2024-03-28T23:33:27.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.628+0000 7fe6cdd9b640 1 -- 172.21.15.161:0/1811721105 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6c0036440 con 0x7fe6d0151e70 2024-03-28T23:33:27.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.628+0000 7fe6cdd9b640 1 -- 172.21.15.161:0/1811721105 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c0036810 con 0x7fe6d0151e70 2024-03-28T23:33:27.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.629+0000 7fe6d5101640 1 -- 172.21.15.161:0/1811721105 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d0151e70 msgr2=0x7fe6d0152250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:27.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.629+0000 7fe6d5101640 1 --2- 172.21.15.161:0/1811721105 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d0151e70 0x7fe6d0152250 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe6c000bb10 tx=0x7fe6c002e0c0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.629+0000 7fe6d5101640 1 -- 172.21.15.161:0/1811721105 shutdown_connections 2024-03-28T23:33:27.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.629+0000 7fe6d5101640 1 --2- 172.21.15.161:0/1811721105 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6d006cdd0 0x7fe6d015c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.629+0000 7fe6d5101640 1 --2- 172.21.15.161:0/1811721105 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d0151e70 0x7fe6d0152250 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.629+0000 7fe6d5101640 1 -- 172.21.15.161:0/1811721105 >> 172.21.15.161:0/1811721105 conn(0x7fe6d00c3390 msgr2=0x7fe6d00c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:27.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.629+0000 7fe6d5101640 1 -- 172.21.15.161:0/1811721105 shutdown_connections 2024-03-28T23:33:27.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.629+0000 7fe6d5101640 1 -- 172.21.15.161:0/1811721105 wait complete. 2024-03-28T23:33:27.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.630+0000 7fe6d5101640 1 Processor -- start 2024-03-28T23:33:27.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.631+0000 7fe6d5101640 1 -- start start 2024-03-28T23:33:27.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.631+0000 7fe6d5101640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d006cdd0 0x7fe6d01eb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:27.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.631+0000 7fe6d5101640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6d0151e70 0x7fe6d014a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:27.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.631+0000 7fe6d5101640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d00abbb0 0x7fe6d01eef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:27.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.631+0000 7fe6d5101640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d00c2d80 con 0x7fe6d00abbb0 2024-03-28T23:33:27.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.631+0000 7fe6d5101640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d00c2c00 con 0x7fe6d006cdd0 2024-03-28T23:33:27.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.631+0000 7fe6d5101640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe6d00c6350 con 0x7fe6d0151e70 2024-03-28T23:33:27.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.631+0000 7fe6cf59e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d00abbb0 0x7fe6d01eef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:27.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.631+0000 7fe6ce59c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6d0151e70 0x7fe6d014a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:27.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.631+0000 7fe6cf59e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d00abbb0 0x7fe6d01eef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41182/0 (socket says 172.21.15.161:41182) 2024-03-28T23:33:27.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.631+0000 7fe6cf59e640 1 -- 172.21.15.161:0/2060313168 learned_addr learned my addr 172.21.15.161:0/2060313168 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:27.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.631+0000 7fe6ced9d640 1 --2- 172.21.15.161:0/2060313168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d006cdd0 0x7fe6d01eb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:27.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.632+0000 7fe6ced9d640 1 -- 172.21.15.161:0/2060313168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6d0151e70 msgr2=0x7fe6d014a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:27.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.632+0000 7fe6ced9d640 1 --2- 172.21.15.161:0/2060313168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6d0151e70 0x7fe6d014a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.632+0000 7fe6ced9d640 1 -- 172.21.15.161:0/2060313168 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d00abbb0 msgr2=0x7fe6d01eef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:27.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.632+0000 7fe6ced9d640 1 --2- 172.21.15.161:0/2060313168 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d00abbb0 0x7fe6d01eef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.632+0000 7fe6ced9d640 1 -- 172.21.15.161:0/2060313168 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d01ee790 con 0x7fe6d006cdd0 2024-03-28T23:33:27.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.632+0000 7fe6cf59e640 1 --2- 172.21.15.161:0/2060313168 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d00abbb0 0x7fe6d01eef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:27.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.632+0000 7fe6ce59c640 1 --2- 172.21.15.161:0/2060313168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6d0151e70 0x7fe6d014a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:27.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.632+0000 7fe6ced9d640 1 --2- 172.21.15.161:0/2060313168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d006cdd0 0x7fe6d01eb180 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0070060 tx=0x7fe6c00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:27.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.632+0000 7fe6b77fe640 1 -- 172.21.15.161:0/2060313168 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c003b070 con 0x7fe6d006cdd0 2024-03-28T23:33:27.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.632+0000 7fe6b77fe640 1 -- 172.21.15.161:0/2060313168 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6c003d050 con 0x7fe6d006cdd0 2024-03-28T23:33:27.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.632+0000 7fe6d5101640 1 -- 172.21.15.161:0/2060313168 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6d014acf0 con 0x7fe6d006cdd0 2024-03-28T23:33:27.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.632+0000 7fe6b77fe640 1 -- 172.21.15.161:0/2060313168 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c0034b30 con 0x7fe6d006cdd0 2024-03-28T23:33:27.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.632+0000 7fe6d5101640 1 -- 172.21.15.161:0/2060313168 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6d014aa00 con 0x7fe6d006cdd0 2024-03-28T23:33:27.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.633+0000 7fe6d5101640 1 -- 172.21.15.161:0/2060313168 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6d004f4b0 con 0x7fe6d006cdd0 2024-03-28T23:33:27.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.633+0000 7fe6b77fe640 1 -- 172.21.15.161:0/2060313168 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe6c0048060 con 0x7fe6d006cdd0 2024-03-28T23:33:27.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.634+0000 7fe6b77fe640 1 --2- 172.21.15.161:0/2060313168 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe6bc06b4d0 0x7fe6bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:27.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.634+0000 7fe6b77fe640 1 -- 172.21.15.161:0/2060313168 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6101+0+0 (secure 0 0 0) 0x7fe6c00abd10 con 0x7fe6d006cdd0 2024-03-28T23:33:27.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.634+0000 7fe6ce59c640 1 --2- 172.21.15.161:0/2060313168 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe6bc06b4d0 0x7fe6bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:27.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.634+0000 7fe6ce59c640 1 --2- 172.21.15.161:0/2060313168 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe6bc06b4d0 0x7fe6bc06d990 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe6d01500b0 tx=0x7fe6c4007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:27.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.639+0000 7fe6b77fe640 1 -- 172.21.15.161:0/2060313168 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe6c007a3b0 con 0x7fe6d006cdd0 2024-03-28T23:33:27.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.874+0000 7fe6d5101640 1 -- 172.21.15.161:0/2060313168 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fe6d006d1b0 con 0x7fe6d006cdd0 2024-03-28T23:33:27.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.878+0000 7fe6b77fe640 1 -- 172.21.15.161:0/2060313168 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) ==== 190+0+147 (secure 0 0 0) 0x7fe6c007e450 con 0x7fe6d006cdd0 2024-03-28T23:33:27.881 INFO:teuthology.orchestra.run.smithi161.stdout:[client.0] 2024-03-28T23:33:27.881 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQA6/QVmc1krAxAAEKsn1Tuuv9OlFZpB4PFhDg== 2024-03-28T23:33:27.881 INFO:teuthology.orchestra.run.smithi161.stdout: caps mds = "allow" 2024-03-28T23:33:27.881 INFO:teuthology.orchestra.run.smithi161.stdout: caps mon = "allow r" 2024-03-28T23:33:27.882 INFO:teuthology.orchestra.run.smithi161.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-28T23:33:27.883 INFO:teuthology.orchestra.run.smithi161.stderr:updated caps for client.0 2024-03-28T23:33:27.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.880+0000 7fe6d5101640 1 -- 172.21.15.161:0/2060313168 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe6bc06b4d0 msgr2=0x7fe6bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:27.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.880+0000 7fe6d5101640 1 --2- 172.21.15.161:0/2060313168 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe6bc06b4d0 0x7fe6bc06d990 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe6d01500b0 tx=0x7fe6c4007e80 comp rx=0 tx=0).stop 2024-03-28T23:33:27.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.881+0000 7fe6d5101640 1 -- 172.21.15.161:0/2060313168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d006cdd0 msgr2=0x7fe6d01eb180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:27.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.881+0000 7fe6d5101640 1 --2- 172.21.15.161:0/2060313168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d006cdd0 0x7fe6d01eb180 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0070060 tx=0x7fe6c00046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.881+0000 7fe6d5101640 1 -- 172.21.15.161:0/2060313168 shutdown_connections 2024-03-28T23:33:27.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.881+0000 7fe6d5101640 1 --2- 172.21.15.161:0/2060313168 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe6bc06b4d0 0x7fe6bc06d990 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.881+0000 7fe6d5101640 1 --2- 172.21.15.161:0/2060313168 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d00abbb0 0x7fe6d01eef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.881+0000 7fe6d5101640 1 --2- 172.21.15.161:0/2060313168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6d0151e70 0x7fe6d014a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.881+0000 7fe6d5101640 1 --2- 172.21.15.161:0/2060313168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d006cdd0 0x7fe6d01eb180 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.881+0000 7fe6d5101640 1 -- 172.21.15.161:0/2060313168 >> 172.21.15.161:0/2060313168 conn(0x7fe6d00c3390 msgr2=0x7fe6d00a9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:27.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.881+0000 7fe6d5101640 1 -- 172.21.15.161:0/2060313168 shutdown_connections 2024-03-28T23:33:27.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.881+0000 7fe6d5101640 1 -- 172.21.15.161:0/2060313168 wait complete. 2024-03-28T23:33:27.894 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:33:27.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.993+0000 7fe6d7ab5640 1 Processor -- start 2024-03-28T23:33:27.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.993+0000 7fe6d7ab5640 1 -- start start 2024-03-28T23:33:27.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.994+0000 7fe6d7ab5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe6d0150590 0x7fe6d00a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:27.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.994+0000 7fe6d7ab5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe6d00a8fc0 0x7fe6d0153150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:27.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.994+0000 7fe6d7ab5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6d01566f0 con 0x7fe6d00aacb0 2024-03-28T23:33:27.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.994+0000 7fe6d7ab5640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe6d01569d0 con 0x7fe6d0150590 2024-03-28T23:33:27.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.994+0000 7fe6d7ab5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fe6d0156cb0 con 0x7fe6d00a8fc0 2024-03-28T23:33:27.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.994+0000 7fe6d5851640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe6d0150590 0x7fe6d00a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:27.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.994+0000 7fe6d6052640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe6d00aacb0 0x7fe6d0153690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:55348/0 (socket says 172.21.15.161:55348) 2024-03-28T23:33:27.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.994+0000 7fe6d5851640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe6d0150590 0x7fe6d00a9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41192/0 (socket says 172.21.15.161:41192) 2024-03-28T23:33:27.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.994+0000 7fe6d6052640 1 -- 172.21.15.161:0/1153268441 learned_addr learned my addr 172.21.15.161:0/1153268441 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:27.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.994+0000 7fe6d5050640 1 --2- 172.21.15.161:0/1153268441 >> v2:172.21.15.170:3300/0 conn(0x7fe6d00a8fc0 0x7fe6d0153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:27.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.994+0000 7fe6d5851640 1 -- 172.21.15.161:0/1153268441 >> v2:172.21.15.170:3300/0 conn(0x7fe6d00a8fc0 msgr2=0x7fe6d0153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:27.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.994+0000 7fe6d5851640 1 --2- 172.21.15.161:0/1153268441 >> v2:172.21.15.170:3300/0 conn(0x7fe6d00a8fc0 0x7fe6d0153150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.994+0000 7fe6d5851640 1 -- 172.21.15.161:0/1153268441 >> v1:172.21.15.161:6789/0 conn(0x7fe6d00aacb0 legacy=0x7fe6d0153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:27.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.994+0000 7fe6d5851640 1 -- 172.21.15.161:0/1153268441 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d0156f90 con 0x7fe6d0150590 2024-03-28T23:33:27.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.994+0000 7fe6d484f640 1 -- 172.21.15.161:0/1153268441 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1565050401 0 0) 0x7fe6d01566f0 con 0x7fe6d00aacb0 2024-03-28T23:33:27.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.995+0000 7fe6d5851640 1 --2- 172.21.15.161:0/1153268441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d0150590 0x7fe6d00a9f40 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fe6d00c7660 tx=0x7fe6cc02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f37cd09b410e8d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:27.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.995+0000 7fe6d484f640 1 -- 172.21.15.161:0/1153268441 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc038070 con 0x7fe6d0150590 2024-03-28T23:33:27.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.995+0000 7fe6d484f640 1 -- 172.21.15.161:0/1153268441 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6cc031710 con 0x7fe6d0150590 2024-03-28T23:33:27.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.995+0000 7fe6d484f640 1 -- 172.21.15.161:0/1153268441 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc031ae0 con 0x7fe6d0150590 2024-03-28T23:33:27.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.996+0000 7fe6d7ab5640 1 -- 172.21.15.161:0/1153268441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d0150590 msgr2=0x7fe6d00a9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:27.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.996+0000 7fe6d7ab5640 1 --2- 172.21.15.161:0/1153268441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d0150590 0x7fe6d00a9f40 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fe6d00c7660 tx=0x7fe6cc02b7b0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.996+0000 7fe6d7ab5640 1 -- 172.21.15.161:0/1153268441 shutdown_connections 2024-03-28T23:33:27.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.996+0000 7fe6d7ab5640 1 --2- 172.21.15.161:0/1153268441 >> v2:172.21.15.170:3300/0 conn(0x7fe6d00a8fc0 0x7fe6d0153150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.996+0000 7fe6d7ab5640 1 --2- 172.21.15.161:0/1153268441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d0150590 0x7fe6d00a9f40 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:27.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.996+0000 7fe6d7ab5640 1 -- 172.21.15.161:0/1153268441 >> 172.21.15.161:0/1153268441 conn(0x7fe6d00c6670 msgr2=0x7fe6d00c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:27.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.996+0000 7fe6d7ab5640 1 -- 172.21.15.161:0/1153268441 shutdown_connections 2024-03-28T23:33:27.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.996+0000 7fe6d7ab5640 1 -- 172.21.15.161:0/1153268441 wait complete. 2024-03-28T23:33:28.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.997+0000 7fe6d7ab5640 1 Processor -- start 2024-03-28T23:33:28.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.997+0000 7fe6d7ab5640 1 -- start start 2024-03-28T23:33:28.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.997+0000 7fe6d7ab5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d00a8fc0 0x7fe6d01ec2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.997+0000 7fe6d7ab5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d00aacb0 0x7fe6d01ec810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.997+0000 7fe6d7ab5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6d0150590 0x7fe6d00c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.997+0000 7fe6d7ab5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d0150970 con 0x7fe6d00a8fc0 2024-03-28T23:33:28.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.997+0000 7fe6d7ab5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d00aa9c0 con 0x7fe6d00aacb0 2024-03-28T23:33:28.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.997+0000 7fe6d7ab5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe6d00aa480 con 0x7fe6d0150590 2024-03-28T23:33:28.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.998+0000 7fe6d5851640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d00a8fc0 0x7fe6d01ec2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.998+0000 7fe6d5851640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d00a8fc0 0x7fe6d01ec2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41198/0 (socket says 172.21.15.161:41198) 2024-03-28T23:33:28.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.998+0000 7fe6d5851640 1 -- 172.21.15.161:0/774584805 learned_addr learned my addr 172.21.15.161:0/774584805 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:28.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.998+0000 7fe6d6052640 1 --2- 172.21.15.161:0/774584805 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6d0150590 0x7fe6d00c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.998+0000 7fe6d5050640 1 --2- 172.21.15.161:0/774584805 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d00aacb0 0x7fe6d01ec810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.998+0000 7fe6d5851640 1 -- 172.21.15.161:0/774584805 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6d0150590 msgr2=0x7fe6d00c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:28.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.998+0000 7fe6d5851640 1 --2- 172.21.15.161:0/774584805 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6d0150590 0x7fe6d00c58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.998+0000 7fe6d5851640 1 -- 172.21.15.161:0/774584805 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d00aacb0 msgr2=0x7fe6d01ec810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:28.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.998+0000 7fe6d5851640 1 --2- 172.21.15.161:0/774584805 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d00aacb0 0x7fe6d01ec810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.998+0000 7fe6d5851640 1 -- 172.21.15.161:0/774584805 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d0147290 con 0x7fe6d00a8fc0 2024-03-28T23:33:28.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.998+0000 7fe6d5851640 1 --2- 172.21.15.161:0/774584805 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d00a8fc0 0x7fe6d01ec2d0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc009890 tx=0x7fe6cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:28.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.998+0000 7fe6c67fc640 1 -- 172.21.15.161:0/774584805 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc042020 con 0x7fe6d00a8fc0 2024-03-28T23:33:28.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.998+0000 7fe6c67fc640 1 -- 172.21.15.161:0/774584805 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6cc03d870 con 0x7fe6d00a8fc0 2024-03-28T23:33:28.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.998+0000 7fe6d7ab5640 1 -- 172.21.15.161:0/774584805 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6d0147ce0 con 0x7fe6d00a8fc0 2024-03-28T23:33:28.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.998+0000 7fe6c67fc640 1 -- 172.21.15.161:0/774584805 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc038040 con 0x7fe6d00a8fc0 2024-03-28T23:33:28.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.998+0000 7fe6d7ab5640 1 -- 172.21.15.161:0/774584805 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6d01482a0 con 0x7fe6d00a8fc0 2024-03-28T23:33:28.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:27.999+0000 7fe6d7ab5640 1 -- 172.21.15.161:0/774584805 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6d0068f90 con 0x7fe6d00a8fc0 2024-03-28T23:33:28.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.000+0000 7fe6c67fc640 1 -- 172.21.15.161:0/774584805 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe6cc045020 con 0x7fe6d00a8fc0 2024-03-28T23:33:28.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.000+0000 7fe6c67fc640 1 --2- 172.21.15.161:0/774584805 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe6b406b4d0 0x7fe6b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.000+0000 7fe6c67fc640 1 -- 172.21.15.161:0/774584805 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6101+0+0 (secure 0 0 0) 0x7fe6cc0ab9e0 con 0x7fe6d00a8fc0 2024-03-28T23:33:28.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.000+0000 7fe6d5050640 1 --2- 172.21.15.161:0/774584805 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe6b406b4d0 0x7fe6b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.001+0000 7fe6d5050640 1 --2- 172.21.15.161:0/774584805 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe6b406b4d0 0x7fe6b406d990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe6d01ed1b0 tx=0x7fe6c0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:28.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.005+0000 7fe6c67fc640 1 -- 172.21.15.161:0/774584805 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe6d0068f90 con 0x7fe6d00a8fc0 2024-03-28T23:33:28.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.215+0000 7fe6d7ab5640 1 -- 172.21.15.161:0/774584805 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe6d006e570 con 0x7fe6d00a8fc0 2024-03-28T23:33:28.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.216+0000 7fe6c67fc640 1 -- 172.21.15.161:0/774584805 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+5192 (secure 0 0 0) 0x7fe6d006e570 con 0x7fe6d00a8fc0 2024-03-28T23:33:28.218 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:28.218 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":79,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5963,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3079114491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3079114491},{"type":"v1","addr":"172.21.15.170:6835","nonce":3079114491}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":6547,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4280217233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4280217233},{"type":"v1","addr":"172.21.15.161:6836","nonce":4280217233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":7086,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/404132405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":404132405},{"type":"v1","addr":"172.21.15.161:6837","nonce":404132405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:33:23.516182+0000","modified":"2024-03-28T23:33:25.515642+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7101},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7101":{"gid":7101,"name":"d","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6834/2176248563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":2176248563},{"type":"v1","addr":"172.21.15.170:6834","nonce":2176248563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7101,"qdb_cluster":[7101]},"id":7}]} 2024-03-28T23:33:28.219 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 79 2024-03-28T23:33:28.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.219+0000 7fe6d7ab5640 1 -- 172.21.15.161:0/774584805 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe6b406b4d0 msgr2=0x7fe6b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:28.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.219+0000 7fe6d7ab5640 1 --2- 172.21.15.161:0/774584805 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe6b406b4d0 0x7fe6b406d990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe6d01ed1b0 tx=0x7fe6c0009190 comp rx=0 tx=0).stop 2024-03-28T23:33:28.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.219+0000 7fe6d7ab5640 1 -- 172.21.15.161:0/774584805 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d00a8fc0 msgr2=0x7fe6d01ec2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:28.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.219+0000 7fe6d7ab5640 1 --2- 172.21.15.161:0/774584805 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d00a8fc0 0x7fe6d01ec2d0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc009890 tx=0x7fe6cc0046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.219+0000 7fe6d7ab5640 1 -- 172.21.15.161:0/774584805 shutdown_connections 2024-03-28T23:33:28.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.219+0000 7fe6d7ab5640 1 --2- 172.21.15.161:0/774584805 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe6b406b4d0 0x7fe6b406d990 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.219+0000 7fe6d7ab5640 1 --2- 172.21.15.161:0/774584805 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6d0150590 0x7fe6d00c58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.219+0000 7fe6d7ab5640 1 --2- 172.21.15.161:0/774584805 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d00aacb0 0x7fe6d01ec810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.219+0000 7fe6d7ab5640 1 --2- 172.21.15.161:0/774584805 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d00a8fc0 0x7fe6d01ec2d0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.219+0000 7fe6d7ab5640 1 -- 172.21.15.161:0/774584805 >> 172.21.15.161:0/774584805 conn(0x7fe6d00c6670 msgr2=0x7fe6d00c0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:28.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.219+0000 7fe6d7ab5640 1 -- 172.21.15.161:0/774584805 shutdown_connections 2024-03-28T23:33:28.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.219+0000 7fe6d7ab5640 1 -- 172.21.15.161:0/774584805 wait complete. 2024-03-28T23:33:28.634 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:33:23.516182+0000', 'modified': '2024-03-28T23:33:25.515642+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7101}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7101': {'gid': 7101, 'name': 'd', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.170:6834/2176248563', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 2176248563}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 2176248563}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7101, 'qdb_cluster': [7101]} 2024-03-28T23:33:28.634 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:33:28.634 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-28T23:33:28.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.738+0000 7f4646c54640 1 Processor -- start 2024-03-28T23:33:28.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.738+0000 7f4646c54640 1 -- start start 2024-03-28T23:33:28.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.739+0000 7f4646c54640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f464014eca0 0x7f46400c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.739+0000 7f4646c54640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f46400c1dc0 0x7f464006ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.739+0000 7f4646c54640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4640151850 con 0x7f464006cf60 2024-03-28T23:33:28.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.739+0000 7f4646c54640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f4640152460 con 0x7f46400c1dc0 2024-03-28T23:33:28.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.739+0000 7f4646c54640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4640151b30 con 0x7f464014eca0 2024-03-28T23:33:28.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.739+0000 7f463ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f46400c1dc0 0x7f464006ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.739+0000 7f463ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f46400c1dc0 0x7f464006ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41212/0 (socket says 172.21.15.161:41212) 2024-03-28T23:33:28.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.739+0000 7f463ffff640 1 -- 172.21.15.161:0/1217141426 learned_addr learned my addr 172.21.15.161:0/1217141426 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:28.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.739+0000 7f46451f1640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f464006cf60 0x7f46400a98c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:55362/0 (socket says 172.21.15.161:55362) 2024-03-28T23:33:28.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.739+0000 7f46449f0640 1 --2- 172.21.15.161:0/1217141426 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f464014eca0 0x7f46400c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.739+0000 7f463ffff640 1 -- 172.21.15.161:0/1217141426 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f464014eca0 msgr2=0x7f46400c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:28.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.739+0000 7f463ffff640 1 --2- 172.21.15.161:0/1217141426 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f464014eca0 0x7f46400c1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.739+0000 7f463ffff640 1 -- 172.21.15.161:0/1217141426 >> v1:172.21.15.161:6789/0 conn(0x7f464006cf60 legacy=0x7f46400a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:28.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.739+0000 7f463ffff640 1 -- 172.21.15.161:0/1217141426 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4640151e10 con 0x7f46400c1dc0 2024-03-28T23:33:28.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.739+0000 7f463f7fe640 1 -- 172.21.15.161:0/1217141426 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2886257469 0 0) 0x7f4640151850 con 0x7f464006cf60 2024-03-28T23:33:28.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.740+0000 7f463ffff640 1 --2- 172.21.15.161:0/1217141426 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46400c1dc0 0x7f464006ca20 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f4634009870 tx=0x7f463402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=568fd4471f3510e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:28.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.740+0000 7f463f7fe640 1 -- 172.21.15.161:0/1217141426 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4634035020 con 0x7f46400c1dc0 2024-03-28T23:33:28.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.740+0000 7f463f7fe640 1 -- 172.21.15.161:0/1217141426 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4634031640 con 0x7f46400c1dc0 2024-03-28T23:33:28.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.740+0000 7f463f7fe640 1 -- 172.21.15.161:0/1217141426 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463402d040 con 0x7f46400c1dc0 2024-03-28T23:33:28.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.741+0000 7f4646c54640 1 -- 172.21.15.161:0/1217141426 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46400c1dc0 msgr2=0x7f464006ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:28.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.741+0000 7f4646c54640 1 --2- 172.21.15.161:0/1217141426 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46400c1dc0 0x7f464006ca20 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f4634009870 tx=0x7f463402b8f0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.741+0000 7f4646c54640 1 -- 172.21.15.161:0/1217141426 shutdown_connections 2024-03-28T23:33:28.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.741+0000 7f4646c54640 1 --2- 172.21.15.161:0/1217141426 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46400c1dc0 0x7f464006ca20 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.741+0000 7f4646c54640 1 --2- 172.21.15.161:0/1217141426 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f464014eca0 0x7f46400c1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.741+0000 7f4646c54640 1 -- 172.21.15.161:0/1217141426 >> 172.21.15.161:0/1217141426 conn(0x7f46400c6670 msgr2=0x7f46400c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:28.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.741+0000 7f4646c54640 1 -- 172.21.15.161:0/1217141426 shutdown_connections 2024-03-28T23:33:28.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.741+0000 7f4646c54640 1 -- 172.21.15.161:0/1217141426 wait complete. 2024-03-28T23:33:28.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.742+0000 7f4646c54640 1 Processor -- start 2024-03-28T23:33:28.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.742+0000 7f4646c54640 1 -- start start 2024-03-28T23:33:28.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.743+0000 7f4646c54640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f464006cf60 0x7f4640142ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.743+0000 7f4646c54640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f46400c1dc0 0x7f46401453f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.743+0000 7f4646c54640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f464014eca0 0x7f46400c3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.743+0000 7f4646c54640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f46401514b0 con 0x7f464006cf60 2024-03-28T23:33:28.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.743+0000 7f4646c54640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f464014f080 con 0x7f46400c1dc0 2024-03-28T23:33:28.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.743+0000 7f4646c54640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4640151630 con 0x7f464014eca0 2024-03-28T23:33:28.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.743+0000 7f46449f0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f464006cf60 0x7f4640142ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.743+0000 7f46449f0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f464006cf60 0x7f4640142ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41220/0 (socket says 172.21.15.161:41220) 2024-03-28T23:33:28.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.743+0000 7f46449f0640 1 -- 172.21.15.161:0/3160293814 learned_addr learned my addr 172.21.15.161:0/3160293814 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:28.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.743+0000 7f463ffff640 1 --2- 172.21.15.161:0/3160293814 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f46400c1dc0 0x7f46401453f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.743+0000 7f46451f1640 1 --2- 172.21.15.161:0/3160293814 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f464014eca0 0x7f46400c3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.743+0000 7f46449f0640 1 -- 172.21.15.161:0/3160293814 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f464014eca0 msgr2=0x7f46400c3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:28.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.743+0000 7f46449f0640 1 --2- 172.21.15.161:0/3160293814 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f464014eca0 0x7f46400c3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.743+0000 7f46449f0640 1 -- 172.21.15.161:0/3160293814 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f46400c1dc0 msgr2=0x7f46401453f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:28.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.743+0000 7f46449f0640 1 --2- 172.21.15.161:0/3160293814 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f46400c1dc0 0x7f46401453f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.743+0000 7f46449f0640 1 -- 172.21.15.161:0/3160293814 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4640155540 con 0x7f464006cf60 2024-03-28T23:33:28.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.743+0000 7f46449f0640 1 --2- 172.21.15.161:0/3160293814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f464006cf60 0x7f4640142ee0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f463000e8a0 tx=0x7f463000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:28.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.743+0000 7f463d7fa640 1 -- 172.21.15.161:0/3160293814 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4630017070 con 0x7f464006cf60 2024-03-28T23:33:28.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.743+0000 7f463d7fa640 1 -- 172.21.15.161:0/3160293814 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f463000c940 con 0x7f464006cf60 2024-03-28T23:33:28.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.743+0000 7f4646c54640 1 -- 172.21.15.161:0/3160293814 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4640152e80 con 0x7f464006cf60 2024-03-28T23:33:28.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.743+0000 7f463d7fa640 1 -- 172.21.15.161:0/3160293814 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463000cd30 con 0x7f464006cf60 2024-03-28T23:33:28.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.743+0000 7f4646c54640 1 -- 172.21.15.161:0/3160293814 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46401437f0 con 0x7f464006cf60 2024-03-28T23:33:28.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.745+0000 7f463d7fa640 1 -- 172.21.15.161:0/3160293814 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f463001a070 con 0x7f464006cf60 2024-03-28T23:33:28.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.746+0000 7f463d7fa640 1 --2- 172.21.15.161:0/3160293814 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f462c06b4d0 0x7f462c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.746+0000 7f463d7fa640 1 -- 172.21.15.161:0/3160293814 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6101+0+0 (secure 0 0 0) 0x7f463008a480 con 0x7f464006cf60 2024-03-28T23:33:28.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.746+0000 7f463ffff640 1 --2- 172.21.15.161:0/3160293814 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f462c06b4d0 0x7f462c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.746+0000 7f463ffff640 1 --2- 172.21.15.161:0/3160293814 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f462c06b4d0 0x7f462c06d990 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f4634036000 tx=0x7f463403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:28.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.747+0000 7f4649830740 1 Processor -- start 2024-03-28T23:33:28.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.747+0000 7f4649830740 1 -- start start 2024-03-28T23:33:28.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.747+0000 7f4649830740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883812db0 0x556883810890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.747+0000 7f4649830740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556883813190 0x556883810dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.747+0000 7f4649830740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556883811310 0x55688380d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.747+0000 7f4649830740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x556883776af0 con 0x556883812db0 2024-03-28T23:33:28.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.747+0000 7f4649830740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55688371f9d0 con 0x556883811310 2024-03-28T23:33:28.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.747+0000 7f4649830740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x556883772b90 con 0x556883813190 2024-03-28T23:33:28.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.747+0000 7f46449f0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883812db0 0x556883810890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.747+0000 7f46449f0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883812db0 0x556883810890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41228/0 (socket says 172.21.15.161:41228) 2024-03-28T23:33:28.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.747+0000 7f46449f0640 1 -- 172.21.15.161:0/268540955 learned_addr learned my addr 172.21.15.161:0/268540955 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:28.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.747+0000 7f463ffff640 1 --2- 172.21.15.161:0/268540955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556883813190 0x556883810dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.747+0000 7f46451f1640 1 --2- 172.21.15.161:0/268540955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556883811310 0x55688380d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.748+0000 7f46449f0640 1 -- 172.21.15.161:0/268540955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556883813190 msgr2=0x556883810dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:28.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.748+0000 7f46449f0640 1 --2- 172.21.15.161:0/268540955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556883813190 0x556883810dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.748+0000 7f46449f0640 1 -- 172.21.15.161:0/268540955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556883811310 msgr2=0x55688380d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:28.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.748+0000 7f46449f0640 1 --2- 172.21.15.161:0/268540955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556883811310 0x55688380d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.748+0000 7f46449f0640 1 -- 172.21.15.161:0/268540955 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55688348b2d0 con 0x556883812db0 2024-03-28T23:33:28.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.748+0000 7f46449f0640 1 --2- 172.21.15.161:0/268540955 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883812db0 0x556883810890 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f4630060ce0 tx=0x7f4630058510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9f65b4ab1f14528 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:28.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.748+0000 7f46267fc640 1 -- 172.21.15.161:0/268540955 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4630021050 con 0x556883812db0 2024-03-28T23:33:28.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.748+0000 7f46267fc640 1 -- 172.21.15.161:0/268540955 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4630061210 con 0x556883812db0 2024-03-28T23:33:28.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.748+0000 7f46267fc640 1 -- 172.21.15.161:0/268540955 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4630055220 con 0x556883812db0 2024-03-28T23:33:28.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.748+0000 7f4649830740 1 -- 172.21.15.161:0/268540955 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883812db0 msgr2=0x556883810890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:28.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.748+0000 7f4649830740 1 --2- 172.21.15.161:0/268540955 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883812db0 0x556883810890 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f4630060ce0 tx=0x7f4630058510 comp rx=0 tx=0).stop 2024-03-28T23:33:28.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.748+0000 7f4649830740 1 -- 172.21.15.161:0/268540955 shutdown_connections 2024-03-28T23:33:28.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.748+0000 7f4649830740 1 --2- 172.21.15.161:0/268540955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556883811310 0x55688380d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.748+0000 7f4649830740 1 --2- 172.21.15.161:0/268540955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556883813190 0x556883810dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.748+0000 7f4649830740 1 --2- 172.21.15.161:0/268540955 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883812db0 0x556883810890 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.748+0000 7f4649830740 1 -- 172.21.15.161:0/268540955 >> 172.21.15.161:0/268540955 conn(0x55688380f630 msgr2=0x556883834220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:28.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.749+0000 7f4649830740 1 -- 172.21.15.161:0/268540955 shutdown_connections 2024-03-28T23:33:28.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.749+0000 7f4649830740 1 -- 172.21.15.161:0/268540955 wait complete. 2024-03-28T23:33:28.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.749+0000 7f4649830740 1 -- start start 2024-03-28T23:33:28.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.749+0000 7f4649830740 1 Processor -- start 2024-03-28T23:33:28.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.750+0000 7f4649830740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883812db0 0x55688380d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.750+0000 7f4649830740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556883813190 0x556883863100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.750+0000 7f4649830740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556883860540 0x55688385fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.750+0000 7f4649830740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x556883835090 con 0x556883812db0 2024-03-28T23:33:28.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.750+0000 7f4649830740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x556883732f70 con 0x556883860540 2024-03-28T23:33:28.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.750+0000 7f4649830740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x556883834ad0 con 0x556883813190 2024-03-28T23:33:28.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.750+0000 7f46449f0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883812db0 0x55688380d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.750+0000 7f4649830740 10 client.0 _ll_register_callbacks cb 0x55688348aac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:33:28.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.750+0000 7f46449f0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883812db0 0x55688380d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41232/0 (socket says 172.21.15.161:41232) 2024-03-28T23:33:28.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.750+0000 7f46449f0640 1 -- 172.21.15.161:0/2297231121 learned_addr learned my addr 172.21.15.161:0/2297231121 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:28.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.750+0000 7f46451f1640 1 --2- 172.21.15.161:0/2297231121 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556883860540 0x55688385fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.750+0000 7f463ffff640 1 --2- 172.21.15.161:0/2297231121 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556883813190 0x556883863100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.750+0000 7f46449f0640 1 -- 172.21.15.161:0/2297231121 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556883813190 msgr2=0x556883863100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:28.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.750+0000 7f46449f0640 1 --2- 172.21.15.161:0/2297231121 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556883813190 0x556883863100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.750+0000 7f46449f0640 1 -- 172.21.15.161:0/2297231121 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556883860540 msgr2=0x55688385fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:28.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.750+0000 7f46449f0640 1 --2- 172.21.15.161:0/2297231121 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556883860540 0x55688385fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.750+0000 7f46449f0640 1 -- 172.21.15.161:0/2297231121 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568835556b0 con 0x556883812db0 2024-03-28T23:33:28.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.751+0000 7f46449f0640 1 --2- 172.21.15.161:0/2297231121 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883812db0 0x55688380d730 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f4630088900 tx=0x7f4630000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a953e1e5dedab95d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:28.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.751+0000 7f461f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:33:28.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.751+0000 7f461f7fe640 1 -- 172.21.15.161:0/2297231121 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4630075060 con 0x556883812db0 2024-03-28T23:33:28.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.751+0000 7f461f7fe640 1 -- 172.21.15.161:0/2297231121 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 6101+0+0 (secure 0 0 0) 0x7f463005dd70 con 0x556883812db0 2024-03-28T23:33:28.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.751+0000 7f4649830740 1 -- 172.21.15.161:0/2297231121 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4630061210 con 0x556883812db0 2024-03-28T23:33:28.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.751+0000 7f461f7fe640 1 -- 172.21.15.161:0/2297231121 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f463007c050 con 0x556883812db0 2024-03-28T23:33:28.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.751+0000 7f4649830740 10 client.7227 fetch_fsmap learned FSMap version 79 2024-03-28T23:33:28.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.751+0000 7f4649830740 1 -- 172.21.15.161:0/2297231121 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=2+}) -- 0x556883835d20 con 0x556883812db0 2024-03-28T23:33:28.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.751+0000 7f461f7fe640 1 -- 172.21.15.161:0/2297231121 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4630061210 con 0x556883812db0 2024-03-28T23:33:28.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.751+0000 7f461f7fe640 1 -- 172.21.15.161:0/2297231121 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 79) ==== 3494+0+0 (secure 0 0 0) 0x7f4630058000 con 0x556883812db0 2024-03-28T23:33:28.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.751+0000 7f4649830740 10 client.7227 fetch_fsmap finished waiting for FSMap version 79 2024-03-28T23:33:28.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.751+0000 7f4649830740 10 client.7227 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-28T23:33:28.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.751+0000 7f4649830740 20 client.7227 populate_metadata read hostname 'smithi161' 2024-03-28T23:33:28.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.751+0000 7f4649830740 1 --2- 172.21.15.161:0/2297231121 >> [v2:172.21.15.170:6832/2176248563,v1:172.21.15.170:6834/2176248563] conn(0x556883833840 0x556883833c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.751+0000 7f4649830740 4 client.7227 mds_command: new command op to 7101 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:33:28.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.751+0000 7f4649830740 1 -- 172.21.15.161:0/2297231121 --> [v2:172.21.15.170:6832/2176248563,v1:172.21.15.170:6834/2176248563] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556883784030 con 0x556883833840 2024-03-28T23:33:28.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.751+0000 7f463ffff640 1 --2- 172.21.15.161:0/2297231121 >> [v2:172.21.15.170:6832/2176248563,v1:172.21.15.170:6834/2176248563] conn(0x556883833840 0x556883833c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.752+0000 7f461f7fe640 10 client.7227 ms_handle_connect on v2:172.21.15.170:6832/2176248563 2024-03-28T23:33:28.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.752+0000 7f463ffff640 1 --2- 172.21.15.161:0/2297231121 >> [v2:172.21.15.170:6832/2176248563,v1:172.21.15.170:6834/2176248563] conn(0x556883833840 0x556883833c20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b5df81f1f234c56 server_cookie=3ff9157a73fea9d4 in_seq=0 out_seq=0 2024-03-28T23:33:28.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.752+0000 7f461f7fe640 1 -- 172.21.15.161:0/2297231121 <== mds.0 v2:172.21.15.170:6832/2176248563 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556883784030 con 0x556883833840 2024-03-28T23:33:28.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.752+0000 7f461f7fe640 10 client.7227 handle_command_reply: tid=0 2024-03-28T23:33:28.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.752+0000 7f4649830740 1 client.7227 shutdown 2024-03-28T23:33:28.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.753+0000 7f4649830740 1 -- 172.21.15.161:0/2297231121 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883812db0 msgr2=0x55688380d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:28.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.753+0000 7f4649830740 1 --2- 172.21.15.161:0/2297231121 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883812db0 0x55688380d730 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f4630088900 tx=0x7f4630000bc0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.753+0000 7f4649830740 1 -- 172.21.15.161:0/2297231121 shutdown_connections 2024-03-28T23:33:28.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.753+0000 7f4649830740 1 --2- 172.21.15.161:0/2297231121 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556883860540 0x55688385fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.753+0000 7f4649830740 1 --2- 172.21.15.161:0/2297231121 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556883813190 0x556883863100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.753+0000 7f4649830740 1 --2- 172.21.15.161:0/2297231121 >> [v2:172.21.15.170:6832/2176248563,v1:172.21.15.170:6834/2176248563] conn(0x556883833840 0x556883833c20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.753+0000 7f4649830740 1 --2- 172.21.15.161:0/2297231121 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883812db0 0x55688380d730 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.753+0000 7f461f7fe640 0 client.7227 ms_handle_reset on v2:172.21.15.170:6832/2176248563 2024-03-28T23:33:28.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.753+0000 7f4649830740 1 -- 172.21.15.161:0/2297231121 >> 172.21.15.161:0/2297231121 conn(0x556883811310 msgr2=0x55688384f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:28.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.753+0000 7f4649830740 1 -- 172.21.15.161:0/2297231121 shutdown_connections 2024-03-28T23:33:28.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.753+0000 7f4649830740 1 -- 172.21.15.161:0/2297231121 wait complete. 2024-03-28T23:33:28.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.753+0000 7f4649830740 20 client.7227 trim_cache size 0 max 16384 2024-03-28T23:33:28.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.771+0000 7f4649830740 1 Processor -- start 2024-03-28T23:33:28.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.771+0000 7f4649830740 1 -- start start 2024-03-28T23:33:28.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.772+0000 7f4649830740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883812db0 0x55688380d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.772+0000 7f4649830740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556883813190 0x55688382f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.772+0000 7f4649830740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556883860540 0x55688382fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.772+0000 7f4649830740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f46401f4c60 con 0x556883812db0 2024-03-28T23:33:28.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.772+0000 7f4649830740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f46401ecaf0 con 0x556883813190 2024-03-28T23:33:28.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.772+0000 7f4649830740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f46401f4610 con 0x556883860540 2024-03-28T23:33:28.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.772+0000 7f46449f0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883812db0 0x55688380d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.772+0000 7f46449f0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883812db0 0x55688380d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41242/0 (socket says 172.21.15.161:41242) 2024-03-28T23:33:28.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.772+0000 7f46449f0640 1 -- 172.21.15.161:0/1513456848 learned_addr learned my addr 172.21.15.161:0/1513456848 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:28.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.772+0000 7f46451f1640 1 --2- 172.21.15.161:0/1513456848 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556883860540 0x55688382fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.772+0000 7f463ffff640 1 --2- 172.21.15.161:0/1513456848 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556883813190 0x55688382f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.772+0000 7f46449f0640 1 -- 172.21.15.161:0/1513456848 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556883860540 msgr2=0x55688382fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:28.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.772+0000 7f46449f0640 1 --2- 172.21.15.161:0/1513456848 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556883860540 0x55688382fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.772+0000 7f46449f0640 1 -- 172.21.15.161:0/1513456848 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556883813190 msgr2=0x55688382f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:28.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.772+0000 7f46449f0640 1 --2- 172.21.15.161:0/1513456848 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556883813190 0x55688382f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.772+0000 7f46449f0640 1 -- 172.21.15.161:0/1513456848 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556883834760 con 0x556883812db0 2024-03-28T23:33:28.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.772+0000 7f46451f1640 1 --2- 172.21.15.161:0/1513456848 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556883860540 0x55688382fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:28.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.772+0000 7f463ffff640 1 --2- 172.21.15.161:0/1513456848 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556883813190 0x55688382f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:28.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.772+0000 7f46449f0640 1 --2- 172.21.15.161:0/1513456848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883812db0 0x55688380d540 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f4630054f40 tx=0x7f463006e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5970b93d8961669e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:28.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.772+0000 7f4625ffb640 1 -- 172.21.15.161:0/1513456848 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4630021040 con 0x556883812db0 2024-03-28T23:33:28.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.772+0000 7f4625ffb640 1 -- 172.21.15.161:0/1513456848 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f463008a2c0 con 0x556883812db0 2024-03-28T23:33:28.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.772+0000 7f4625ffb640 1 -- 172.21.15.161:0/1513456848 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463005dd70 con 0x556883812db0 2024-03-28T23:33:28.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.773+0000 7f4649830740 1 -- 172.21.15.161:0/1513456848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883812db0 msgr2=0x55688380d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:28.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.773+0000 7f4649830740 1 --2- 172.21.15.161:0/1513456848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883812db0 0x55688380d540 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f4630054f40 tx=0x7f463006e040 comp rx=0 tx=0).stop 2024-03-28T23:33:28.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.773+0000 7f4649830740 1 -- 172.21.15.161:0/1513456848 shutdown_connections 2024-03-28T23:33:28.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.773+0000 7f4649830740 1 --2- 172.21.15.161:0/1513456848 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556883860540 0x55688382fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.773+0000 7f4649830740 1 --2- 172.21.15.161:0/1513456848 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556883813190 0x55688382f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.773+0000 7f4649830740 1 --2- 172.21.15.161:0/1513456848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883812db0 0x55688380d540 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.773+0000 7f4649830740 1 -- 172.21.15.161:0/1513456848 >> 172.21.15.161:0/1513456848 conn(0x556883811310 msgr2=0x5568838087d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:28.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.773+0000 7f4649830740 1 -- 172.21.15.161:0/1513456848 shutdown_connections 2024-03-28T23:33:28.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.773+0000 7f4649830740 1 -- 172.21.15.161:0/1513456848 wait complete. 2024-03-28T23:33:28.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.773+0000 7f4649830740 1 -- start start 2024-03-28T23:33:28.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.774+0000 7f4649830740 1 Processor -- start 2024-03-28T23:33:28.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.774+0000 7f4649830740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556883812db0 0x5568838720e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.774+0000 7f4649830740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883813190 0x556883832870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.774+0000 7f4649830740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556883860540 0x5568838309a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.774+0000 7f4649830740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x556883835090 con 0x556883813190 2024-03-28T23:33:28.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.774+0000 7f4649830740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x556883732f70 con 0x556883812db0 2024-03-28T23:33:28.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.774+0000 7f4649830740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55688384c680 con 0x556883860540 2024-03-28T23:33:28.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.774+0000 7f4649830740 10 client.0 _ll_register_callbacks cb 0x7f463006f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:33:28.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.774+0000 7f463ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883813190 0x556883832870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.774+0000 7f463ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883813190 0x556883832870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41250/0 (socket says 172.21.15.161:41250) 2024-03-28T23:33:28.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.774+0000 7f463ffff640 1 -- 172.21.15.161:0/3758330011 learned_addr learned my addr 172.21.15.161:0/3758330011 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:28.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.774+0000 7f46449f0640 1 --2- 172.21.15.161:0/3758330011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556883812db0 0x5568838720e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.774+0000 7f46451f1640 1 --2- 172.21.15.161:0/3758330011 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556883860540 0x5568838309a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.775+0000 7f463ffff640 1 -- 172.21.15.161:0/3758330011 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556883860540 msgr2=0x5568838309a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:28.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.775+0000 7f463ffff640 1 --2- 172.21.15.161:0/3758330011 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556883860540 0x5568838309a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.775+0000 7f463ffff640 1 -- 172.21.15.161:0/3758330011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556883812db0 msgr2=0x5568838720e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:28.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.775+0000 7f463ffff640 1 --2- 172.21.15.161:0/3758330011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556883812db0 0x5568838720e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.775+0000 7f463ffff640 1 -- 172.21.15.161:0/3758330011 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556883862da0 con 0x556883813190 2024-03-28T23:33:28.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.775+0000 7f46449f0640 1 --2- 172.21.15.161:0/3758330011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556883812db0 0x5568838720e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:28.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.775+0000 7f463ffff640 1 --2- 172.21.15.161:0/3758330011 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883813190 0x556883832870 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f463402b930 tx=0x7f463402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7901af5b573a4751 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:28.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.775+0000 7f461f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:33:28.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.775+0000 7f461f7fe640 1 -- 172.21.15.161:0/3758330011 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4634035040 con 0x556883813190 2024-03-28T23:33:28.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.775+0000 7f461f7fe640 1 -- 172.21.15.161:0/3758330011 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 6101+0+0 (secure 0 0 0) 0x7f4634055320 con 0x556883813190 2024-03-28T23:33:28.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.775+0000 7f4649830740 1 -- 172.21.15.161:0/3758330011 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55688377e380 con 0x556883813190 2024-03-28T23:33:28.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.775+0000 7f461f7fe640 1 -- 172.21.15.161:0/3758330011 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f4634060050 con 0x556883813190 2024-03-28T23:33:28.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.775+0000 7f4649830740 10 client.7233 fetch_fsmap learned FSMap version 79 2024-03-28T23:33:28.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.775+0000 7f4649830740 1 -- 172.21.15.161:0/3758330011 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=2+}) -- 0x5568838105f0 con 0x556883813190 2024-03-28T23:33:28.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.775+0000 7f461f7fe640 1 -- 172.21.15.161:0/3758330011 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55688377e380 con 0x556883813190 2024-03-28T23:33:28.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.775+0000 7f461f7fe640 1 -- 172.21.15.161:0/3758330011 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 79) ==== 3494+0+0 (secure 0 0 0) 0x7f4634063020 con 0x556883813190 2024-03-28T23:33:28.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.775+0000 7f4649830740 10 client.7233 fetch_fsmap finished waiting for FSMap version 79 2024-03-28T23:33:28.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.775+0000 7f4649830740 10 client.7233 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-28T23:33:28.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.775+0000 7f4649830740 20 client.7233 populate_metadata read hostname 'smithi161' 2024-03-28T23:33:28.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.775+0000 7f4649830740 1 --2- 172.21.15.161:0/3758330011 >> [v2:172.21.15.170:6832/2176248563,v1:172.21.15.170:6834/2176248563] conn(0x5568838ca7b0 0x5568838cab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.775+0000 7f4649830740 4 client.7233 mds_command: new command op to 7101 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:33:28.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.775+0000 7f4649830740 1 -- 172.21.15.161:0/3758330011 --> [v2:172.21.15.170:6832/2176248563,v1:172.21.15.170:6834/2176248563] -- command(tid 0: {"prefix": "status"}) -- 0x5568836fca00 con 0x5568838ca7b0 2024-03-28T23:33:28.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.776+0000 7f46449f0640 1 --2- 172.21.15.161:0/3758330011 >> [v2:172.21.15.170:6832/2176248563,v1:172.21.15.170:6834/2176248563] conn(0x5568838ca7b0 0x5568838cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.776+0000 7f46449f0640 1 --2- 172.21.15.161:0/3758330011 >> [v2:172.21.15.170:6832/2176248563,v1:172.21.15.170:6834/2176248563] conn(0x5568838ca7b0 0x5568838cab90 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ae33f48cf1365ce server_cookie=b0314c47284b2371 in_seq=0 out_seq=0 2024-03-28T23:33:28.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.776+0000 7f461f7fe640 10 client.7233 ms_handle_connect on v2:172.21.15.170:6832/2176248563 2024-03-28T23:33:28.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.776+0000 7f461f7fe640 1 -- 172.21.15.161:0/3758330011 <== mds.0 v2:172.21.15.170:6832/2176248563 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x5568836fca00 con 0x5568838ca7b0 2024-03-28T23:33:28.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.776+0000 7f461f7fe640 10 client.7233 handle_command_reply: tid=0 2024-03-28T23:33:28.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.776+0000 7f4649830740 1 client.7233 shutdown 2024-03-28T23:33:28.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.776+0000 7f4649830740 1 -- 172.21.15.161:0/3758330011 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883813190 msgr2=0x556883832870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:28.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.776+0000 7f4649830740 1 --2- 172.21.15.161:0/3758330011 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883813190 0x556883832870 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f463402b930 tx=0x7f463402e000 comp rx=0 tx=0).stop 2024-03-28T23:33:28.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.776+0000 7f4649830740 1 -- 172.21.15.161:0/3758330011 shutdown_connections 2024-03-28T23:33:28.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.776+0000 7f4649830740 1 --2- 172.21.15.161:0/3758330011 >> [v2:172.21.15.170:6832/2176248563,v1:172.21.15.170:6834/2176248563] conn(0x5568838ca7b0 0x5568838cab90 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.776+0000 7f4649830740 1 --2- 172.21.15.161:0/3758330011 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556883813190 0x556883832870 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.776+0000 7f461f7fe640 0 client.7233 ms_handle_reset on v2:172.21.15.170:6832/2176248563 2024-03-28T23:33:28.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.776+0000 7f4649830740 1 --2- 172.21.15.161:0/3758330011 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556883860540 0x5568838309a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.776+0000 7f4649830740 1 --2- 172.21.15.161:0/3758330011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556883812db0 0x5568838720e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.776+0000 7f4649830740 1 -- 172.21.15.161:0/3758330011 >> 172.21.15.161:0/3758330011 conn(0x556883811310 msgr2=0x55688382e880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:28.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.776+0000 7f4649830740 1 -- 172.21.15.161:0/3758330011 shutdown_connections 2024-03-28T23:33:28.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.776+0000 7f4649830740 1 -- 172.21.15.161:0/3758330011 wait complete. 2024-03-28T23:33:28.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.776+0000 7f4649830740 20 client.7233 trim_cache size 0 max 16384 2024-03-28T23:33:28.785 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:33:28.786 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:33:28.786 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:33:28.786 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 7101, 2024-03-28T23:33:28.786 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:33:28.786 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:33:28.786 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:33:28.786 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 4.6070256199999999, 2024-03-28T23:33:28.786 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 79, 2024-03-28T23:33:28.786 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 94, 2024-03-28T23:33:28.786 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 94, 2024-03-28T23:33:28.786 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 9.697545045 2024-03-28T23:33:28.786 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:33:28.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.777+0000 7f4646c54640 1 -- 172.21.15.161:0/3160293814 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f462c06b4d0 msgr2=0x7f462c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:28.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.777+0000 7f4646c54640 1 --2- 172.21.15.161:0/3160293814 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f462c06b4d0 0x7f462c06d990 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f4634036000 tx=0x7f463403c040 comp rx=0 tx=0).stop 2024-03-28T23:33:28.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.777+0000 7f4646c54640 1 -- 172.21.15.161:0/3160293814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f464006cf60 msgr2=0x7f4640142ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:28.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.777+0000 7f4646c54640 1 --2- 172.21.15.161:0/3160293814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f464006cf60 0x7f4640142ee0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f463000e8a0 tx=0x7f463000ed60 comp rx=0 tx=0).stop 2024-03-28T23:33:28.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.777+0000 7f4646c54640 1 -- 172.21.15.161:0/3160293814 shutdown_connections 2024-03-28T23:33:28.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.777+0000 7f4646c54640 1 --2- 172.21.15.161:0/3160293814 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f462c06b4d0 0x7f462c06d990 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.777+0000 7f4646c54640 1 --2- 172.21.15.161:0/3160293814 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f464014eca0 0x7f46400c3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.777+0000 7f4646c54640 1 --2- 172.21.15.161:0/3160293814 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f46400c1dc0 0x7f46401453f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.777+0000 7f4646c54640 1 --2- 172.21.15.161:0/3160293814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f464006cf60 0x7f4640142ee0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.777+0000 7f4646c54640 1 -- 172.21.15.161:0/3160293814 >> 172.21.15.161:0/3160293814 conn(0x7f46400c6670 msgr2=0x7f46400c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:28.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.777+0000 7f4646c54640 1 -- 172.21.15.161:0/3160293814 shutdown_connections 2024-03-28T23:33:28.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.777+0000 7f4646c54640 1 -- 172.21.15.161:0/3160293814 wait complete. 2024-03-28T23:33:28.789 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:33:28.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.889+0000 7f13aa956640 1 Processor -- start 2024-03-28T23:33:28.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.889+0000 7f13aa956640 1 -- start start 2024-03-28T23:33:28.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.890+0000 7f13aa956640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f13a40a8fc0 0x7f13a4153270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.890+0000 7f13aa956640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f13a40aacb0 0x7f13a41537b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.890+0000 7f13aa956640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13a41567c0 con 0x7f13a4150600 2024-03-28T23:33:28.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.890+0000 7f13aa956640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f13a4156570 con 0x7f13a40aacb0 2024-03-28T23:33:28.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.890+0000 7f13aa956640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f13a4156aa0 con 0x7f13a40a8fc0 2024-03-28T23:33:28.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.890+0000 7f13a3fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f13a4150600 0x7f13a40a9f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:55364/0 (socket says 172.21.15.161:55364) 2024-03-28T23:33:28.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.890+0000 7f13a3fff640 1 -- 172.21.15.161:0/161924441 learned_addr learned my addr 172.21.15.161:0/161924441 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:28.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.890+0000 7f13a8ef3640 1 --2- 172.21.15.161:0/161924441 >> v2:172.21.15.161:3300/0 conn(0x7f13a40aacb0 0x7f13a41537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.890+0000 7f13a37fe640 1 --2- 172.21.15.161:0/161924441 >> v2:172.21.15.170:3300/0 conn(0x7f13a40a8fc0 0x7f13a4153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.890+0000 7f13a8ef3640 1 -- 172.21.15.161:0/161924441 >> v2:172.21.15.170:3300/0 conn(0x7f13a40a8fc0 msgr2=0x7f13a4153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:28.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.890+0000 7f13a8ef3640 1 --2- 172.21.15.161:0/161924441 >> v2:172.21.15.170:3300/0 conn(0x7f13a40a8fc0 0x7f13a4153270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.890+0000 7f13a8ef3640 1 -- 172.21.15.161:0/161924441 >> v1:172.21.15.161:6789/0 conn(0x7f13a4150600 legacy=0x7f13a40a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:28.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.890+0000 7f13a8ef3640 1 -- 172.21.15.161:0/161924441 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13a4156d80 con 0x7f13a40aacb0 2024-03-28T23:33:28.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.891+0000 7f13a8ef3640 1 --2- 172.21.15.161:0/161924441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13a40aacb0 0x7f13a41537b0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f13a406f570 tx=0x7f139c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c6402dec26204c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:28.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.891+0000 7f13a2ffd640 1 -- 172.21.15.161:0/161924441 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c039070 con 0x7f13a40aacb0 2024-03-28T23:33:28.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.891+0000 7f13a2ffd640 1 -- 172.21.15.161:0/161924441 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f139c02cbf0 con 0x7f13a40aacb0 2024-03-28T23:33:28.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.891+0000 7f13a2ffd640 1 -- 172.21.15.161:0/161924441 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c034440 con 0x7f13a40aacb0 2024-03-28T23:33:28.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.892+0000 7f13aa956640 1 -- 172.21.15.161:0/161924441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13a40aacb0 msgr2=0x7f13a41537b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:28.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.892+0000 7f13aa956640 1 --2- 172.21.15.161:0/161924441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13a40aacb0 0x7f13a41537b0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f13a406f570 tx=0x7f139c02be40 comp rx=0 tx=0).stop 2024-03-28T23:33:28.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.892+0000 7f13aa956640 1 -- 172.21.15.161:0/161924441 shutdown_connections 2024-03-28T23:33:28.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.892+0000 7f13aa956640 1 --2- 172.21.15.161:0/161924441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13a40aacb0 0x7f13a41537b0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.892+0000 7f13aa956640 1 --2- 172.21.15.161:0/161924441 >> v2:172.21.15.170:3300/0 conn(0x7f13a40a8fc0 0x7f13a4153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.892+0000 7f13aa956640 1 -- 172.21.15.161:0/161924441 >> 172.21.15.161:0/161924441 conn(0x7f13a40c6670 msgr2=0x7f13a40c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:28.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.892+0000 7f13aa956640 1 -- 172.21.15.161:0/161924441 shutdown_connections 2024-03-28T23:33:28.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.892+0000 7f13aa956640 1 -- 172.21.15.161:0/161924441 wait complete. 2024-03-28T23:33:28.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.893+0000 7f13aa956640 1 Processor -- start 2024-03-28T23:33:28.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.893+0000 7f13aa956640 1 -- start start 2024-03-28T23:33:28.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.894+0000 7f13aa956640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13a40a8fc0 0x7f13a40c5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.894+0000 7f13aa956640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13a40aacb0 0x7f13a4148db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.894+0000 7f13aa956640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13a4150600 0x7f13a41492f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.894+0000 7f13aa956640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f13a41509e0 con 0x7f13a40a8fc0 2024-03-28T23:33:28.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.894+0000 7f13aa956640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f13a4157340 con 0x7f13a4150600 2024-03-28T23:33:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.894+0000 7f13aa956640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f13a40aa9e0 con 0x7f13a40aacb0 2024-03-28T23:33:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.894+0000 7f13a3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13a40a8fc0 0x7f13a40c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.894+0000 7f13a3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13a40a8fc0 0x7f13a40c5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41270/0 (socket says 172.21.15.161:41270) 2024-03-28T23:33:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.894+0000 7f13a3fff640 1 -- 172.21.15.161:0/1220441921 learned_addr learned my addr 172.21.15.161:0/1220441921 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.894+0000 7f13a37fe640 1 --2- 172.21.15.161:0/1220441921 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13a40aacb0 0x7f13a4148db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.894+0000 7f13a8ef3640 1 --2- 172.21.15.161:0/1220441921 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13a4150600 0x7f13a41492f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.894+0000 7f13a3fff640 1 -- 172.21.15.161:0/1220441921 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13a40aacb0 msgr2=0x7f13a4148db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.894+0000 7f13a3fff640 1 --2- 172.21.15.161:0/1220441921 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13a40aacb0 0x7f13a4148db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.894+0000 7f13a3fff640 1 -- 172.21.15.161:0/1220441921 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13a4150600 msgr2=0x7f13a41492f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.894+0000 7f13a3fff640 1 --2- 172.21.15.161:0/1220441921 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13a4150600 0x7f13a41492f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.894+0000 7f13a3fff640 1 -- 172.21.15.161:0/1220441921 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13a4145f90 con 0x7f13a40a8fc0 2024-03-28T23:33:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.894+0000 7f13a3fff640 1 --2- 172.21.15.161:0/1220441921 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13a40a8fc0 0x7f13a40c5500 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f139400c430 tx=0x7f139400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:28.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.895+0000 7f13a0ff9640 1 -- 172.21.15.161:0/1220441921 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1394017070 con 0x7f13a40a8fc0 2024-03-28T23:33:28.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.895+0000 7f13aa956640 1 -- 172.21.15.161:0/1220441921 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13a4143450 con 0x7f13a40a8fc0 2024-03-28T23:33:28.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.895+0000 7f13a0ff9640 1 -- 172.21.15.161:0/1220441921 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1394005ce0 con 0x7f13a40a8fc0 2024-03-28T23:33:28.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.895+0000 7f13a0ff9640 1 -- 172.21.15.161:0/1220441921 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13940072f0 con 0x7f13a40a8fc0 2024-03-28T23:33:28.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.895+0000 7f13aa956640 1 -- 172.21.15.161:0/1220441921 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13a40c5a40 con 0x7f13a40a8fc0 2024-03-28T23:33:28.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.896+0000 7f13aa956640 1 -- 172.21.15.161:0/1220441921 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13a4143a60 con 0x7f13a40a8fc0 2024-03-28T23:33:28.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.896+0000 7f13a0ff9640 1 -- 172.21.15.161:0/1220441921 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f139401a070 con 0x7f13a40a8fc0 2024-03-28T23:33:28.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.896+0000 7f13a0ff9640 1 --2- 172.21.15.161:0/1220441921 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f139006b4d0 0x7f139006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:28.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.897+0000 7f13a0ff9640 1 -- 172.21.15.161:0/1220441921 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6101+0+0 (secure 0 0 0) 0x7f13940896f0 con 0x7f13a40a8fc0 2024-03-28T23:33:28.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.897+0000 7f13a37fe640 1 --2- 172.21.15.161:0/1220441921 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f139006b4d0 0x7f139006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:28.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.897+0000 7f13a37fe640 1 --2- 172.21.15.161:0/1220441921 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f139006b4d0 0x7f139006d990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f13a414fd60 tx=0x7f1398009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:28.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:28.901+0000 7f13a0ff9640 1 -- 172.21.15.161:0/1220441921 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1394057da0 con 0x7f13a40a8fc0 2024-03-28T23:33:29.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.108+0000 7f13aa956640 1 -- 172.21.15.161:0/1220441921 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13a404f4b0 con 0x7f13a40a8fc0 2024-03-28T23:33:29.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.109+0000 7f13a0ff9640 1 -- 172.21.15.161:0/1220441921 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+5192 (secure 0 0 0) 0x7f139405be40 con 0x7f13a40a8fc0 2024-03-28T23:33:29.110 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:29.110 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":79,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5963,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3079114491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3079114491},{"type":"v1","addr":"172.21.15.170:6835","nonce":3079114491}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":6547,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4280217233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4280217233},{"type":"v1","addr":"172.21.15.161:6836","nonce":4280217233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":7086,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/404132405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":404132405},{"type":"v1","addr":"172.21.15.161:6837","nonce":404132405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:33:23.516182+0000","modified":"2024-03-28T23:33:25.515642+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7101},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7101":{"gid":7101,"name":"d","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6834/2176248563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":2176248563},{"type":"v1","addr":"172.21.15.170:6834","nonce":2176248563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7101,"qdb_cluster":[7101]},"id":7}]} 2024-03-28T23:33:29.110 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 79 2024-03-28T23:33:29.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.111+0000 7f13aa956640 1 -- 172.21.15.161:0/1220441921 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f139006b4d0 msgr2=0x7f139006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:29.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.111+0000 7f13aa956640 1 --2- 172.21.15.161:0/1220441921 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f139006b4d0 0x7f139006d990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f13a414fd60 tx=0x7f1398009190 comp rx=0 tx=0).stop 2024-03-28T23:33:29.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.111+0000 7f13aa956640 1 -- 172.21.15.161:0/1220441921 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13a40a8fc0 msgr2=0x7f13a40c5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:29.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.111+0000 7f13aa956640 1 --2- 172.21.15.161:0/1220441921 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13a40a8fc0 0x7f13a40c5500 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f139400c430 tx=0x7f139400c8f0 comp rx=0 tx=0).stop 2024-03-28T23:33:29.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.111+0000 7f13aa956640 1 -- 172.21.15.161:0/1220441921 shutdown_connections 2024-03-28T23:33:29.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.111+0000 7f13aa956640 1 --2- 172.21.15.161:0/1220441921 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f139006b4d0 0x7f139006d990 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:29.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.111+0000 7f13aa956640 1 --2- 172.21.15.161:0/1220441921 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13a4150600 0x7f13a41492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:29.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.111+0000 7f13aa956640 1 --2- 172.21.15.161:0/1220441921 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13a40aacb0 0x7f13a4148db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:29.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.111+0000 7f13aa956640 1 --2- 172.21.15.161:0/1220441921 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13a40a8fc0 0x7f13a40c5500 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:29.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.111+0000 7f13aa956640 1 -- 172.21.15.161:0/1220441921 >> 172.21.15.161:0/1220441921 conn(0x7f13a40c6670 msgr2=0x7f13a40c2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:29.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.111+0000 7f13aa956640 1 -- 172.21.15.161:0/1220441921 shutdown_connections 2024-03-28T23:33:29.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.111+0000 7f13aa956640 1 -- 172.21.15.161:0/1220441921 wait complete. 2024-03-28T23:33:29.124 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 7, "epoch": 79, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 7, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:33:23.516182+0000", "damaged": [], "data_pools": [ 16 ], "enabled": true, "epoch": 79, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7101": { "addr": "172.21.15.170:6834/2176248563", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 2176248563, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 2176248563, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7101, "incarnation": 77, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 15, "modified": "2024-03-28T23:33:25.515642+0000", "qdb_cluster": [ 7101 ], "qdb_leader": 7101, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7101 } } } ], "standbys": [ { "addr": "172.21.15.170:6835/3079114491", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 3079114491, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 3079114491, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 76, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5963, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6836/4280217233", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 4280217233, "type": "v2" }, { "addr": "172.21.15.161:6836", "nonce": 4280217233, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 76, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6547, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6837/404132405", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6835", "nonce": 404132405, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 404132405, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 76, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7086, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:33:29.124 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-28T23:33:29.125 INFO:tasks.cephfs.test_failover:current = [{'gid': 7086, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6837/404132405', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 404132405}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 404132405}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 76}, {'gid': 5963, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/3079114491', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 3079114491}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 3079114491}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 76}, {'gid': 6547, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6836/4280217233', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 4280217233}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 4280217233}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 76}, {'gid': 7101, 'name': 'd', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.170:6834/2176248563', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 2176248563}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 2176248563}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}] 2024-03-28T23:33:29.125 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.b mds_join_fs cephfs 2024-03-28T23:33:29.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.230+0000 7f9a7bfff640 1 Processor -- start 2024-03-28T23:33:29.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.230+0000 7f9a7bfff640 1 -- start start 2024-03-28T23:33:29.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.231+0000 7f9a7bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9a7c06c380 0x7f9a7c0a8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:29.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.231+0000 7f9a7bfff640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a7c157220 con 0x7f9a7c156dd0 2024-03-28T23:33:29.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.231+0000 7f9a7bfff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a7c06e9e0 con 0x7f9a7c149900 2024-03-28T23:33:29.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.231+0000 7f9a7bfff640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f9a7c0c1750 con 0x7f9a7c06c380 2024-03-28T23:33:29.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.231+0000 7f9a73fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9a7c06c380 0x7f9a7c0a8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:29.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.231+0000 7f9a7b7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f9a7c156dd0 0x7f9a7c0aba00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:55378/0 (socket says 172.21.15.161:55378) 2024-03-28T23:33:29.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.231+0000 7f9a7b7fe640 1 -- 172.21.15.161:0/1820810801 learned_addr learned my addr 172.21.15.161:0/1820810801 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:29.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.231+0000 7f9a73fff640 1 -- 172.21.15.161:0/1820810801 >> v1:172.21.15.170:6789/0 conn(0x7f9a7c149900 legacy=0x7f9a7c0c1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:29.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.231+0000 7f9a73fff640 1 -- 172.21.15.161:0/1820810801 >> v1:172.21.15.161:6789/0 conn(0x7f9a7c156dd0 legacy=0x7f9a7c0aba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:29.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.231+0000 7f9a7a7fc640 1 -- 172.21.15.161:0/1820810801 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3862060574 0 0) 0x7f9a7c06e9e0 con 0x7f9a7c149900 2024-03-28T23:33:29.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.231+0000 7f9a73fff640 1 -- 172.21.15.161:0/1820810801 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a7c153890 con 0x7f9a7c06c380 2024-03-28T23:33:29.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.232+0000 7f9a73fff640 1 --2- 172.21.15.161:0/1820810801 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a7c06c380 0x7f9a7c0a8f10 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f9a640097a0 tx=0x7f9a6402b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c868f8caa5aae818 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:29.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.232+0000 7f9a7a7fc640 1 -- 172.21.15.161:0/1820810801 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a64037070 con 0x7f9a7c06c380 2024-03-28T23:33:29.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.232+0000 7f9a7a7fc640 1 -- 172.21.15.161:0/1820810801 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a64030710 con 0x7f9a7c06c380 2024-03-28T23:33:29.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.232+0000 7f9a7a7fc640 1 -- 172.21.15.161:0/1820810801 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a64030ae0 con 0x7f9a7c06c380 2024-03-28T23:33:29.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.233+0000 7f9a7bfff640 1 -- 172.21.15.161:0/1820810801 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a7c06c380 msgr2=0x7f9a7c0a8f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:29.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.233+0000 7f9a7bfff640 1 --2- 172.21.15.161:0/1820810801 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a7c06c380 0x7f9a7c0a8f10 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f9a640097a0 tx=0x7f9a6402b680 comp rx=0 tx=0).stop 2024-03-28T23:33:29.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.233+0000 7f9a7bfff640 1 -- 172.21.15.161:0/1820810801 shutdown_connections 2024-03-28T23:33:29.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.233+0000 7f9a7bfff640 1 --2- 172.21.15.161:0/1820810801 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a7c06c380 0x7f9a7c0a8f10 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:29.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.233+0000 7f9a7bfff640 1 -- 172.21.15.161:0/1820810801 >> 172.21.15.161:0/1820810801 conn(0x7f9a7c0a83f0 msgr2=0x7f9a7c0a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:29.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.233+0000 7f9a7bfff640 1 -- 172.21.15.161:0/1820810801 shutdown_connections 2024-03-28T23:33:29.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.233+0000 7f9a7bfff640 1 -- 172.21.15.161:0/1820810801 wait complete. 2024-03-28T23:33:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.234+0000 7f9a7bfff640 1 Processor -- start 2024-03-28T23:33:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.234+0000 7f9a7bfff640 1 -- start start 2024-03-28T23:33:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.234+0000 7f9a7bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a7c06c380 0x7f9a7c149f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.234+0000 7f9a7bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a7c149900 0x7f9a7c14b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.234+0000 7f9a7bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a7c156dd0 0x7f9a7c14ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.234+0000 7f9a7bfff640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7c1562f0 con 0x7f9a7c06c380 2024-03-28T23:33:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.234+0000 7f9a7bfff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7c156940 con 0x7f9a7c149900 2024-03-28T23:33:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.234+0000 7f9a7bfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9a7c1565d0 con 0x7f9a7c156dd0 2024-03-28T23:33:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.234+0000 7f9a7affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a7c06c380 0x7f9a7c149f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.234+0000 7f9a7affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a7c06c380 0x7f9a7c149f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41284/0 (socket says 172.21.15.161:41284) 2024-03-28T23:33:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.234+0000 7f9a7affd640 1 -- 172.21.15.161:0/788828779 learned_addr learned my addr 172.21.15.161:0/788828779 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.235+0000 7f9a73fff640 1 --2- 172.21.15.161:0/788828779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a7c149900 0x7f9a7c14b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:29.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.235+0000 7f9a7b7fe640 1 --2- 172.21.15.161:0/788828779 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a7c156dd0 0x7f9a7c14ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:29.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.235+0000 7f9a7affd640 1 -- 172.21.15.161:0/788828779 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a7c156dd0 msgr2=0x7f9a7c14ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:29.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.235+0000 7f9a7affd640 1 --2- 172.21.15.161:0/788828779 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a7c156dd0 0x7f9a7c14ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:29.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.235+0000 7f9a7affd640 1 -- 172.21.15.161:0/788828779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a7c149900 msgr2=0x7f9a7c14b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:29.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.235+0000 7f9a7affd640 1 --2- 172.21.15.161:0/788828779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a7c149900 0x7f9a7c14b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:29.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.235+0000 7f9a7affd640 1 -- 172.21.15.161:0/788828779 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a7c0c53b0 con 0x7f9a7c06c380 2024-03-28T23:33:29.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.235+0000 7f9a7b7fe640 1 --2- 172.21.15.161:0/788828779 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a7c156dd0 0x7f9a7c14ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:29.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.235+0000 7f9a7affd640 1 --2- 172.21.15.161:0/788828779 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a7c06c380 0x7f9a7c149f80 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00c550 tx=0x7f9a6c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:29.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.235+0000 7f9a737fe640 1 -- 172.21.15.161:0/788828779 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c005480 con 0x7f9a7c06c380 2024-03-28T23:33:29.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.235+0000 7f9a737fe640 1 -- 172.21.15.161:0/788828779 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a6c0035f0 con 0x7f9a7c06c380 2024-03-28T23:33:29.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.235+0000 7f9a7bfff640 1 -- 172.21.15.161:0/788828779 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a7c0c5690 con 0x7f9a7c06c380 2024-03-28T23:33:29.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.235+0000 7f9a737fe640 1 -- 172.21.15.161:0/788828779 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c0039e0 con 0x7f9a7c06c380 2024-03-28T23:33:29.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.235+0000 7f9a7bfff640 1 -- 172.21.15.161:0/788828779 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a7c0c5c50 con 0x7f9a7c06c380 2024-03-28T23:33:29.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.236+0000 7f9a7bfff640 1 -- 172.21.15.161:0/788828779 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a7c149440 con 0x7f9a7c06c380 2024-03-28T23:33:29.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.237+0000 7f9a737fe640 1 -- 172.21.15.161:0/788828779 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9a6c003b80 con 0x7f9a7c06c380 2024-03-28T23:33:29.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.237+0000 7f9a737fe640 1 --2- 172.21.15.161:0/788828779 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9a5c06b670 0x7f9a5c06db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:29.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.237+0000 7f9a737fe640 1 -- 172.21.15.161:0/788828779 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6101+0+0 (secure 0 0 0) 0x7f9a6c08a050 con 0x7f9a7c06c380 2024-03-28T23:33:29.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.237+0000 7f9a73fff640 1 --2- 172.21.15.161:0/788828779 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9a5c06b670 0x7f9a5c06db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:29.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.238+0000 7f9a73fff640 1 --2- 172.21.15.161:0/788828779 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9a5c06b670 0x7f9a5c06db30 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f9a64034000 tx=0x7f9a64008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:29.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.242+0000 7f9a737fe640 1 -- 172.21.15.161:0/788828779 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9a6c058d40 con 0x7f9a7c06c380 2024-03-28T23:33:29.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.414+0000 7f9a7bfff640 1 -- 172.21.15.161:0/788828779 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) -- 0x7f9a7c04f4b0 con 0x7f9a7c06c380 2024-03-28T23:33:29.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.416+0000 7f9a737fe640 1 -- 172.21.15.161:0/788828779 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v32)=0 v32) ==== 116+0+0 (secure 0 0 0) 0x7f9a6c058640 con 0x7f9a7c06c380 2024-03-28T23:33:29.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.418+0000 7f9a7bfff640 1 -- 172.21.15.161:0/788828779 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9a5c06b670 msgr2=0x7f9a5c06db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:29.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.418+0000 7f9a7bfff640 1 --2- 172.21.15.161:0/788828779 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9a5c06b670 0x7f9a5c06db30 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f9a64034000 tx=0x7f9a64008030 comp rx=0 tx=0).stop 2024-03-28T23:33:29.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.418+0000 7f9a7bfff640 1 -- 172.21.15.161:0/788828779 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a7c06c380 msgr2=0x7f9a7c149f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:29.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.418+0000 7f9a7bfff640 1 --2- 172.21.15.161:0/788828779 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a7c06c380 0x7f9a7c149f80 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00c550 tx=0x7f9a6c00ca10 comp rx=0 tx=0).stop 2024-03-28T23:33:29.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.418+0000 7f9a7bfff640 1 -- 172.21.15.161:0/788828779 shutdown_connections 2024-03-28T23:33:29.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.418+0000 7f9a7bfff640 1 --2- 172.21.15.161:0/788828779 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9a5c06b670 0x7f9a5c06db30 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:29.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.418+0000 7f9a7bfff640 1 --2- 172.21.15.161:0/788828779 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a7c156dd0 0x7f9a7c14ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:29.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.419+0000 7f9a7bfff640 1 --2- 172.21.15.161:0/788828779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a7c149900 0x7f9a7c14b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:29.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.419+0000 7f9a7bfff640 1 --2- 172.21.15.161:0/788828779 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a7c06c380 0x7f9a7c149f80 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:29.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.419+0000 7f9a7bfff640 1 -- 172.21.15.161:0/788828779 >> 172.21.15.161:0/788828779 conn(0x7f9a7c0a83f0 msgr2=0x7f9a7c0c30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:29.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.419+0000 7f9a7bfff640 1 -- 172.21.15.161:0/788828779 shutdown_connections 2024-03-28T23:33:29.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.419+0000 7f9a7bfff640 1 -- 172.21.15.161:0/788828779 wait complete. 2024-03-28T23:33:29.430 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:33:29.438 INFO:tasks.ceph.mds.d.smithi170.stderr:2024-03-28T23:33:29.438+0000 7f2bb1daf640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:33:29.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.531+0000 7f32cadc2640 1 Processor -- start 2024-03-28T23:33:29.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.531+0000 7f32cadc2640 1 -- start start 2024-03-28T23:33:29.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.532+0000 7f32cadc2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f32c406ce80 0x7f32c40aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:29.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.532+0000 7f32cadc2640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32c414c250 con 0x7f32c40a9680 2024-03-28T23:33:29.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.532+0000 7f32cadc2640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32c41529c0 con 0x7f32c414be70 2024-03-28T23:33:29.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.532+0000 7f32cadc2640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f32c40a9dc0 con 0x7f32c406ce80 2024-03-28T23:33:29.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.532+0000 7f32c3fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f32c406ce80 0x7f32c40aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:29.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.532+0000 7f32c935f640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f32c40a9680 0x7f32c415c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:55384/0 (socket says 172.21.15.161:55384) 2024-03-28T23:33:29.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.532+0000 7f32c935f640 1 -- 172.21.15.161:0/3495351192 learned_addr learned my addr 172.21.15.161:0/3495351192 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:29.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.532+0000 7f32c3fff640 1 -- 172.21.15.161:0/3495351192 >> v1:172.21.15.170:6789/0 conn(0x7f32c414be70 legacy=0x7f32c40ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:29.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.532+0000 7f32c3fff640 1 -- 172.21.15.161:0/3495351192 >> v1:172.21.15.161:6789/0 conn(0x7f32c40a9680 legacy=0x7f32c415c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:29.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.532+0000 7f32c37fe640 1 -- 172.21.15.161:0/3495351192 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 255068300 0 0) 0x7f32c414c250 con 0x7f32c40a9680 2024-03-28T23:33:29.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.532+0000 7f32c3fff640 1 -- 172.21.15.161:0/3495351192 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32c40aa0a0 con 0x7f32c406ce80 2024-03-28T23:33:29.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.533+0000 7f32c3fff640 1 --2- 172.21.15.161:0/3495351192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32c406ce80 0x7f32c40aa5c0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f32b80097a0 tx=0x7f32b802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc7a9002d3228c1a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:29.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.533+0000 7f32c37fe640 1 -- 172.21.15.161:0/3495351192 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b8035020 con 0x7f32c406ce80 2024-03-28T23:33:29.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.533+0000 7f32c37fe640 1 -- 172.21.15.161:0/3495351192 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32b8031830 con 0x7f32c406ce80 2024-03-28T23:33:29.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.533+0000 7f32c37fe640 1 -- 172.21.15.161:0/3495351192 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b8031c00 con 0x7f32c406ce80 2024-03-28T23:33:29.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.534+0000 7f32cadc2640 1 -- 172.21.15.161:0/3495351192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32c406ce80 msgr2=0x7f32c40aa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:29.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.534+0000 7f32cadc2640 1 --2- 172.21.15.161:0/3495351192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32c406ce80 0x7f32c40aa5c0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f32b80097a0 tx=0x7f32b802b8d0 comp rx=0 tx=0).stop 2024-03-28T23:33:29.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.534+0000 7f32cadc2640 1 -- 172.21.15.161:0/3495351192 shutdown_connections 2024-03-28T23:33:29.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.534+0000 7f32cadc2640 1 --2- 172.21.15.161:0/3495351192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32c406ce80 0x7f32c40aa5c0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:29.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.534+0000 7f32cadc2640 1 -- 172.21.15.161:0/3495351192 >> 172.21.15.161:0/3495351192 conn(0x7f32c40c6670 msgr2=0x7f32c40c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:29.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.534+0000 7f32cadc2640 1 -- 172.21.15.161:0/3495351192 shutdown_connections 2024-03-28T23:33:29.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.534+0000 7f32cadc2640 1 -- 172.21.15.161:0/3495351192 wait complete. 2024-03-28T23:33:29.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.535+0000 7f32cadc2640 1 Processor -- start 2024-03-28T23:33:29.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.535+0000 7f32cadc2640 1 -- start start 2024-03-28T23:33:29.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.536+0000 7f32cadc2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32c406ce80 0x7f32c41eb090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:29.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.536+0000 7f32cadc2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32c40a9680 0x7f32c41edcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:29.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.536+0000 7f32cadc2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32c414be70 0x7f32c41de800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:29.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.536+0000 7f32cadc2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f32c40aba30 con 0x7f32c414be70 2024-03-28T23:33:29.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.536+0000 7f32cadc2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f32c40ab8b0 con 0x7f32c406ce80 2024-03-28T23:33:29.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.536+0000 7f32cadc2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f32c406b150 con 0x7f32c40a9680 2024-03-28T23:33:29.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.536+0000 7f32c935f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32c414be70 0x7f32c41de800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:29.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.536+0000 7f32c935f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32c414be70 0x7f32c41de800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41312/0 (socket says 172.21.15.161:41312) 2024-03-28T23:33:29.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.536+0000 7f32c935f640 1 -- 172.21.15.161:0/605318980 learned_addr learned my addr 172.21.15.161:0/605318980 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:29.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.536+0000 7f32c8b5e640 1 --2- 172.21.15.161:0/605318980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32c406ce80 0x7f32c41eb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:29.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.536+0000 7f32c3fff640 1 --2- 172.21.15.161:0/605318980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32c40a9680 0x7f32c41edcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:29.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.536+0000 7f32c935f640 1 -- 172.21.15.161:0/605318980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32c40a9680 msgr2=0x7f32c41edcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:29.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.536+0000 7f32c935f640 1 --2- 172.21.15.161:0/605318980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32c40a9680 0x7f32c41edcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:29.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.536+0000 7f32c935f640 1 -- 172.21.15.161:0/605318980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32c406ce80 msgr2=0x7f32c41eb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:29.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.536+0000 7f32c935f640 1 --2- 172.21.15.161:0/605318980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32c406ce80 0x7f32c41eb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:29.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.536+0000 7f32c935f640 1 -- 172.21.15.161:0/605318980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32c41df420 con 0x7f32c414be70 2024-03-28T23:33:29.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.536+0000 7f32c935f640 1 --2- 172.21.15.161:0/605318980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32c414be70 0x7f32c41de800 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f32bc00c700 tx=0x7f32bc00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:29.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.537+0000 7f32c17fa640 1 -- 172.21.15.161:0/605318980 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32bc018070 con 0x7f32c414be70 2024-03-28T23:33:29.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.537+0000 7f32c17fa640 1 -- 172.21.15.161:0/605318980 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32bc00ce10 con 0x7f32c414be70 2024-03-28T23:33:29.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.537+0000 7f32cadc2640 1 -- 172.21.15.161:0/605318980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32c40c5bb0 con 0x7f32c414be70 2024-03-28T23:33:29.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.537+0000 7f32c17fa640 1 -- 172.21.15.161:0/605318980 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32bc00a860 con 0x7f32c414be70 2024-03-28T23:33:29.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.537+0000 7f32cadc2640 1 -- 172.21.15.161:0/605318980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32c41eb5d0 con 0x7f32c414be70 2024-03-28T23:33:29.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.538+0000 7f32cadc2640 1 -- 172.21.15.161:0/605318980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32c406e570 con 0x7f32c414be70 2024-03-28T23:33:29.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.538+0000 7f32c17fa640 1 -- 172.21.15.161:0/605318980 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f32bc01b070 con 0x7f32c414be70 2024-03-28T23:33:29.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.538+0000 7f32c17fa640 1 --2- 172.21.15.161:0/605318980 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f32b006b4d0 0x7f32b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:29.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.538+0000 7f32c17fa640 1 -- 172.21.15.161:0/605318980 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6101+0+0 (secure 0 0 0) 0x7f32bc0898e0 con 0x7f32c414be70 2024-03-28T23:33:29.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.538+0000 7f32c8b5e640 1 --2- 172.21.15.161:0/605318980 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f32b006b4d0 0x7f32b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:29.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.539+0000 7f32c8b5e640 1 --2- 172.21.15.161:0/605318980 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f32b006b4d0 0x7f32b006d990 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f32b4000f60 tx=0x7f32b4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:29.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.543+0000 7f32c17fa640 1 -- 172.21.15.161:0/605318980 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f32c406e570 con 0x7f32c414be70 2024-03-28T23:33:29.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.749+0000 7f32cadc2640 1 -- 172.21.15.161:0/605318980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32c404f4b0 con 0x7f32c414be70 2024-03-28T23:33:29.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.750+0000 7f32c17fa640 1 -- 172.21.15.161:0/605318980 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+5192 (secure 0 0 0) 0x7f32bc057ee0 con 0x7f32c414be70 2024-03-28T23:33:29.751 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:29.751 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":79,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5963,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3079114491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3079114491},{"type":"v1","addr":"172.21.15.170:6835","nonce":3079114491}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":6547,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4280217233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4280217233},{"type":"v1","addr":"172.21.15.161:6836","nonce":4280217233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":7086,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/404132405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":404132405},{"type":"v1","addr":"172.21.15.161:6837","nonce":404132405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:33:23.516182+0000","modified":"2024-03-28T23:33:25.515642+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7101},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7101":{"gid":7101,"name":"d","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6834/2176248563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":2176248563},{"type":"v1","addr":"172.21.15.170:6834","nonce":2176248563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7101,"qdb_cluster":[7101]},"id":7}]} 2024-03-28T23:33:29.751 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 79 2024-03-28T23:33:29.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.752+0000 7f32cadc2640 1 -- 172.21.15.161:0/605318980 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f32b006b4d0 msgr2=0x7f32b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:29.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.752+0000 7f32cadc2640 1 --2- 172.21.15.161:0/605318980 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f32b006b4d0 0x7f32b006d990 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f32b4000f60 tx=0x7f32b4009040 comp rx=0 tx=0).stop 2024-03-28T23:33:29.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.752+0000 7f32cadc2640 1 -- 172.21.15.161:0/605318980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32c414be70 msgr2=0x7f32c41de800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:29.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.752+0000 7f32cadc2640 1 --2- 172.21.15.161:0/605318980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32c414be70 0x7f32c41de800 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f32bc00c700 tx=0x7f32bc00cbc0 comp rx=0 tx=0).stop 2024-03-28T23:33:29.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.752+0000 7f32cadc2640 1 -- 172.21.15.161:0/605318980 shutdown_connections 2024-03-28T23:33:29.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.752+0000 7f32cadc2640 1 --2- 172.21.15.161:0/605318980 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f32b006b4d0 0x7f32b006d990 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:29.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.752+0000 7f32cadc2640 1 --2- 172.21.15.161:0/605318980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32c414be70 0x7f32c41de800 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:29.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.752+0000 7f32cadc2640 1 --2- 172.21.15.161:0/605318980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32c40a9680 0x7f32c41edcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:29.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.752+0000 7f32cadc2640 1 --2- 172.21.15.161:0/605318980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32c406ce80 0x7f32c41eb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:29.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.752+0000 7f32cadc2640 1 -- 172.21.15.161:0/605318980 >> 172.21.15.161:0/605318980 conn(0x7f32c40c6670 msgr2=0x7f32c40ab1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:29.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.752+0000 7f32cadc2640 1 -- 172.21.15.161:0/605318980 shutdown_connections 2024-03-28T23:33:29.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:29.752+0000 7f32cadc2640 1 -- 172.21.15.161:0/605318980 wait complete. 2024-03-28T23:33:30.166 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:33:23.516182+0000', 'modified': '2024-03-28T23:33:25.515642+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7101}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7101': {'gid': 7101, 'name': 'd', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.170:6834/2176248563', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 2176248563}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 2176248563}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7101, 'qdb_cluster': [7101]} 2024-03-28T23:33:30.166 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:33:30.166 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-28T23:33:30.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.270+0000 7fd240fa1640 1 Processor -- start 2024-03-28T23:33:30.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.270+0000 7fd240fa1640 1 -- start start 2024-03-28T23:33:30.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.271+0000 7fd240fa1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd23c0c5410 0x7fd23c0c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:30.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.271+0000 7fd240fa1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd23c1553a0 0x7fd23c155820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:30.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.271+0000 7fd240fa1640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd23c158a30 con 0x7fd23c152890 2024-03-28T23:33:30.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.271+0000 7fd240fa1640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd23c0c8480 con 0x7fd23c1553a0 2024-03-28T23:33:30.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.271+0000 7fd240fa1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd23c0c5d90 con 0x7fd23c0c5410 2024-03-28T23:33:30.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.271+0000 7fd23ad9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd23c1553a0 0x7fd23c155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:30.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.271+0000 7fd23a59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd23c152890 0x7fd23c0c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:55390/0 (socket says 172.21.15.161:55390) 2024-03-28T23:33:30.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.271+0000 7fd23a59c640 1 -- 172.21.15.161:0/1440567939 learned_addr learned my addr 172.21.15.161:0/1440567939 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:30.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.271+0000 7fd233fff640 1 --2- 172.21.15.161:0/1440567939 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd23c0c5410 0x7fd23c0c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:30.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.271+0000 7fd23ad9d640 1 -- 172.21.15.161:0/1440567939 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd23c0c5410 msgr2=0x7fd23c0c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:30.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.271+0000 7fd23ad9d640 1 --2- 172.21.15.161:0/1440567939 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd23c0c5410 0x7fd23c0c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.271+0000 7fd23ad9d640 1 -- 172.21.15.161:0/1440567939 >> v1:172.21.15.161:6789/0 conn(0x7fd23c152890 legacy=0x7fd23c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:30.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.271+0000 7fd23ad9d640 1 -- 172.21.15.161:0/1440567939 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd23c158d10 con 0x7fd23c1553a0 2024-03-28T23:33:30.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.271+0000 7fd239d9b640 1 -- 172.21.15.161:0/1440567939 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1922867942 0 0) 0x7fd23c158a30 con 0x7fd23c152890 2024-03-28T23:33:30.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.272+0000 7fd23ad9d640 1 --2- 172.21.15.161:0/1440567939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd23c1553a0 0x7fd23c155820 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fd234009d90 tx=0x7fd23402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb3e0793ddd1ba09 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:30.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.272+0000 7fd239d9b640 1 -- 172.21.15.161:0/1440567939 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd234039070 con 0x7fd23c1553a0 2024-03-28T23:33:30.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.272+0000 7fd239d9b640 1 -- 172.21.15.161:0/1440567939 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd23402cbc0 con 0x7fd23c1553a0 2024-03-28T23:33:30.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.272+0000 7fd239d9b640 1 -- 172.21.15.161:0/1440567939 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd234034440 con 0x7fd23c1553a0 2024-03-28T23:33:30.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.273+0000 7fd240fa1640 1 -- 172.21.15.161:0/1440567939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd23c1553a0 msgr2=0x7fd23c155820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:30.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.273+0000 7fd240fa1640 1 --2- 172.21.15.161:0/1440567939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd23c1553a0 0x7fd23c155820 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fd234009d90 tx=0x7fd23402be10 comp rx=0 tx=0).stop 2024-03-28T23:33:30.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.273+0000 7fd240fa1640 1 -- 172.21.15.161:0/1440567939 shutdown_connections 2024-03-28T23:33:30.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.273+0000 7fd240fa1640 1 --2- 172.21.15.161:0/1440567939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd23c1553a0 0x7fd23c155820 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.273+0000 7fd240fa1640 1 --2- 172.21.15.161:0/1440567939 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd23c0c5410 0x7fd23c0c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.273+0000 7fd240fa1640 1 -- 172.21.15.161:0/1440567939 >> 172.21.15.161:0/1440567939 conn(0x7fd23c0ac620 msgr2=0x7fd23c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:30.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.273+0000 7fd240fa1640 1 -- 172.21.15.161:0/1440567939 shutdown_connections 2024-03-28T23:33:30.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.273+0000 7fd240fa1640 1 -- 172.21.15.161:0/1440567939 wait complete. 2024-03-28T23:33:30.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.274+0000 7fd240fa1640 1 Processor -- start 2024-03-28T23:33:30.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.274+0000 7fd240fa1640 1 -- start start 2024-03-28T23:33:30.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.274+0000 7fd240fa1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd23c0c5410 0x7fd23c148430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:30.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.275+0000 7fd240fa1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd23c152890 0x7fd23c157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:30.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.275+0000 7fd240fa1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd23c1553a0 0x7fd23c1ea980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:30.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.275+0000 7fd240fa1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd23c1592d0 con 0x7fd23c0c5410 2024-03-28T23:33:30.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.275+0000 7fd240fa1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd23c159640 con 0x7fd23c1553a0 2024-03-28T23:33:30.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.275+0000 7fd240fa1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd23c0c90a0 con 0x7fd23c152890 2024-03-28T23:33:30.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.275+0000 7fd23a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd23c0c5410 0x7fd23c148430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:30.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.275+0000 7fd23a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd23c0c5410 0x7fd23c148430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41322/0 (socket says 172.21.15.161:41322) 2024-03-28T23:33:30.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.275+0000 7fd23a59c640 1 -- 172.21.15.161:0/388091815 learned_addr learned my addr 172.21.15.161:0/388091815 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:30.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.275+0000 7fd233fff640 1 --2- 172.21.15.161:0/388091815 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd23c152890 0x7fd23c157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:30.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.275+0000 7fd23ad9d640 1 --2- 172.21.15.161:0/388091815 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd23c1553a0 0x7fd23c1ea980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:30.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.275+0000 7fd23a59c640 1 -- 172.21.15.161:0/388091815 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd23c152890 msgr2=0x7fd23c157d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:30.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.275+0000 7fd23a59c640 1 --2- 172.21.15.161:0/388091815 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd23c152890 0x7fd23c157d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.275+0000 7fd23a59c640 1 -- 172.21.15.161:0/388091815 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd23c1553a0 msgr2=0x7fd23c1ea980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:30.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.275+0000 7fd23a59c640 1 --2- 172.21.15.161:0/388091815 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd23c1553a0 0x7fd23c1ea980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.275+0000 7fd23a59c640 1 -- 172.21.15.161:0/388091815 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd23c14eda0 con 0x7fd23c0c5410 2024-03-28T23:33:30.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.275+0000 7fd23ad9d640 1 --2- 172.21.15.161:0/388091815 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd23c1553a0 0x7fd23c1ea980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:30.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.275+0000 7fd23a59c640 1 --2- 172.21.15.161:0/388091815 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd23c0c5410 0x7fd23c148430 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fd22c00c4d0 tx=0x7fd22c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:30.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.275+0000 7fd232ffd640 1 -- 172.21.15.161:0/388091815 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd22c018070 con 0x7fd23c0c5410 2024-03-28T23:33:30.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.275+0000 7fd232ffd640 1 -- 172.21.15.161:0/388091815 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd22c007500 con 0x7fd23c0c5410 2024-03-28T23:33:30.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.275+0000 7fd240fa1640 1 -- 172.21.15.161:0/388091815 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd23c14f080 con 0x7fd23c0c5410 2024-03-28T23:33:30.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.275+0000 7fd232ffd640 1 -- 172.21.15.161:0/388091815 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd22c004a20 con 0x7fd23c0c5410 2024-03-28T23:33:30.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.275+0000 7fd240fa1640 1 -- 172.21.15.161:0/388091815 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd23c14f640 con 0x7fd23c0c5410 2024-03-28T23:33:30.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.277+0000 7fd232ffd640 1 -- 172.21.15.161:0/388091815 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd22c01b070 con 0x7fd23c0c5410 2024-03-28T23:33:30.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.277+0000 7fd232ffd640 1 --2- 172.21.15.161:0/388091815 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd21c06b4d0 0x7fd21c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:30.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.277+0000 7fd232ffd640 1 -- 172.21.15.161:0/388091815 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6101+0+0 (secure 0 0 0) 0x7fd22c089510 con 0x7fd23c0c5410 2024-03-28T23:33:30.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.277+0000 7fd233fff640 1 --2- 172.21.15.161:0/388091815 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd21c06b4d0 0x7fd21c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:30.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.278+0000 7fd233fff640 1 --2- 172.21.15.161:0/388091815 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd21c06b4d0 0x7fd21c06d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fd23c1532e0 tx=0x7fd224009110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:30.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.278+0000 7fd243b84740 1 Processor -- start 2024-03-28T23:33:30.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.278+0000 7fd243b84740 1 -- start start 2024-03-28T23:33:30.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.279+0000 7fd243b84740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987d4db0 0x5651987d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:30.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.279+0000 7fd243b84740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651987d5190 0x5651987d2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:30.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.279+0000 7fd243b84740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651987d3310 0x5651987cf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:30.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.279+0000 7fd243b84740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x565198738af0 con 0x5651987d4db0 2024-03-28T23:33:30.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.279+0000 7fd243b84740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5651986e19d0 con 0x5651987d5190 2024-03-28T23:33:30.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.279+0000 7fd243b84740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x565198734b90 con 0x5651987d3310 2024-03-28T23:33:30.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.279+0000 7fd23a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987d4db0 0x5651987d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:30.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.279+0000 7fd23a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987d4db0 0x5651987d2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41332/0 (socket says 172.21.15.161:41332) 2024-03-28T23:33:30.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.279+0000 7fd23a59c640 1 -- 172.21.15.161:0/2973952926 learned_addr learned my addr 172.21.15.161:0/2973952926 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:30.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.279+0000 7fd233fff640 1 --2- 172.21.15.161:0/2973952926 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651987d5190 0x5651987d2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:30.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.279+0000 7fd23ad9d640 1 --2- 172.21.15.161:0/2973952926 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651987d3310 0x5651987cf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:30.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.279+0000 7fd23a59c640 1 -- 172.21.15.161:0/2973952926 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651987d3310 msgr2=0x5651987cf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:30.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.279+0000 7fd23a59c640 1 --2- 172.21.15.161:0/2973952926 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651987d3310 0x5651987cf4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.279+0000 7fd23a59c640 1 -- 172.21.15.161:0/2973952926 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651987d5190 msgr2=0x5651987d2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:30.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.279+0000 7fd23a59c640 1 --2- 172.21.15.161:0/2973952926 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651987d5190 0x5651987d2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.279+0000 7fd23a59c640 1 -- 172.21.15.161:0/2973952926 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56519844d2d0 con 0x5651987d4db0 2024-03-28T23:33:30.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.280+0000 7fd23a59c640 1 --2- 172.21.15.161:0/2973952926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987d4db0 0x5651987d2890 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fd22c05fd70 tx=0x7fd22c057d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89db86a1af5be822 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:30.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.280+0000 7fd21bfff640 1 -- 172.21.15.161:0/2973952926 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd22c055f40 con 0x5651987d4db0 2024-03-28T23:33:30.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.280+0000 7fd21bfff640 1 -- 172.21.15.161:0/2973952926 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd22c0542b0 con 0x5651987d4db0 2024-03-28T23:33:30.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.280+0000 7fd21bfff640 1 -- 172.21.15.161:0/2973952926 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd22c004e00 con 0x5651987d4db0 2024-03-28T23:33:30.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.280+0000 7fd243b84740 1 -- 172.21.15.161:0/2973952926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987d4db0 msgr2=0x5651987d2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:30.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.280+0000 7fd243b84740 1 --2- 172.21.15.161:0/2973952926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987d4db0 0x5651987d2890 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fd22c05fd70 tx=0x7fd22c057d80 comp rx=0 tx=0).stop 2024-03-28T23:33:30.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.280+0000 7fd243b84740 1 -- 172.21.15.161:0/2973952926 shutdown_connections 2024-03-28T23:33:30.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.280+0000 7fd243b84740 1 --2- 172.21.15.161:0/2973952926 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651987d3310 0x5651987cf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.280+0000 7fd243b84740 1 --2- 172.21.15.161:0/2973952926 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651987d5190 0x5651987d2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.280+0000 7fd243b84740 1 --2- 172.21.15.161:0/2973952926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987d4db0 0x5651987d2890 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.280+0000 7fd243b84740 1 -- 172.21.15.161:0/2973952926 >> 172.21.15.161:0/2973952926 conn(0x5651987d1630 msgr2=0x5651987f6220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:30.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.280+0000 7fd243b84740 1 -- 172.21.15.161:0/2973952926 shutdown_connections 2024-03-28T23:33:30.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.280+0000 7fd243b84740 1 -- 172.21.15.161:0/2973952926 wait complete. 2024-03-28T23:33:30.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.281+0000 7fd243b84740 1 -- start start 2024-03-28T23:33:30.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.281+0000 7fd243b84740 1 Processor -- start 2024-03-28T23:33:30.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.282+0000 7fd243b84740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651987d4db0 0x5651987f98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:30.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.282+0000 7fd243b84740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987d5190 0x5651987cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:30.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.282+0000 7fd243b84740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651988247e0 0x565198809a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:30.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.282+0000 7fd243b84740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5651987f7090 con 0x5651987d5190 2024-03-28T23:33:30.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.282+0000 7fd243b84740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5651986f4f70 con 0x5651988247e0 2024-03-28T23:33:30.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.282+0000 7fd243b84740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x5651987f6ad0 con 0x5651987d4db0 2024-03-28T23:33:30.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.282+0000 7fd243b84740 10 client.0 _ll_register_callbacks cb 0x56519844cac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:33:30.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.282+0000 7fd233fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987d5190 0x5651987cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:30.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.282+0000 7fd233fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987d5190 0x5651987cfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41346/0 (socket says 172.21.15.161:41346) 2024-03-28T23:33:30.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.282+0000 7fd233fff640 1 -- 172.21.15.161:0/3971742361 learned_addr learned my addr 172.21.15.161:0/3971742361 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:30.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.282+0000 7fd23a59c640 1 --2- 172.21.15.161:0/3971742361 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651987d4db0 0x5651987f98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:30.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.282+0000 7fd23ad9d640 1 --2- 172.21.15.161:0/3971742361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651988247e0 0x565198809a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:30.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.282+0000 7fd233fff640 1 -- 172.21.15.161:0/3971742361 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651987d4db0 msgr2=0x5651987f98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:30.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.282+0000 7fd233fff640 1 --2- 172.21.15.161:0/3971742361 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651987d4db0 0x5651987f98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.282+0000 7fd233fff640 1 -- 172.21.15.161:0/3971742361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651988247e0 msgr2=0x565198809a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:30.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.282+0000 7fd233fff640 1 --2- 172.21.15.161:0/3971742361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651988247e0 0x565198809a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.282+0000 7fd233fff640 1 -- 172.21.15.161:0/3971742361 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651985176b0 con 0x5651987d5190 2024-03-28T23:33:30.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.282+0000 7fd233fff640 1 --2- 172.21.15.161:0/3971742361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987d5190 0x5651987cfa00 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fd224005fa0 tx=0x7fd2240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e836c805284e8a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:30.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.282+0000 7fd2197fa640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:33:30.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.283+0000 7fd2197fa640 1 -- 172.21.15.161:0/3971742361 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd224026070 con 0x5651987d5190 2024-03-28T23:33:30.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.283+0000 7fd2197fa640 1 -- 172.21.15.161:0/3971742361 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 6101+0+0 (secure 0 0 0) 0x7fd224004030 con 0x5651987d5190 2024-03-28T23:33:30.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.283+0000 7fd243b84740 1 -- 172.21.15.161:0/3971742361 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd22c0542b0 con 0x5651987d5190 2024-03-28T23:33:30.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.283+0000 7fd2197fa640 1 -- 172.21.15.161:0/3971742361 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7fd224031050 con 0x5651987d5190 2024-03-28T23:33:30.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.283+0000 7fd243b84740 10 client.7275 fetch_fsmap learned FSMap version 79 2024-03-28T23:33:30.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.283+0000 7fd243b84740 1 -- 172.21.15.161:0/3971742361 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=2+}) -- 0x5651987f88d0 con 0x5651987d5190 2024-03-28T23:33:30.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.283+0000 7fd2197fa640 1 -- 172.21.15.161:0/3971742361 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd22c0542b0 con 0x5651987d5190 2024-03-28T23:33:30.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.283+0000 7fd2197fa640 1 -- 172.21.15.161:0/3971742361 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 79) ==== 3494+0+0 (secure 0 0 0) 0x7fd224034020 con 0x5651987d5190 2024-03-28T23:33:30.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.283+0000 7fd243b84740 10 client.7275 fetch_fsmap finished waiting for FSMap version 79 2024-03-28T23:33:30.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.283+0000 7fd243b84740 10 client.7275 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-28T23:33:30.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.283+0000 7fd243b84740 20 client.7275 populate_metadata read hostname 'smithi161' 2024-03-28T23:33:30.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.283+0000 7fd243b84740 1 --2- 172.21.15.161:0/3971742361 >> [v2:172.21.15.170:6832/2176248563,v1:172.21.15.170:6834/2176248563] conn(0x5651987f7f90 0x56519882fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:30.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.283+0000 7fd243b84740 4 client.7275 mds_command: new command op to 7101 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:33:30.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.283+0000 7fd243b84740 1 -- 172.21.15.161:0/3971742361 --> [v2:172.21.15.170:6832/2176248563,v1:172.21.15.170:6834/2176248563] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565198746030 con 0x5651987f7f90 2024-03-28T23:33:30.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.283+0000 7fd23a59c640 1 --2- 172.21.15.161:0/3971742361 >> [v2:172.21.15.170:6832/2176248563,v1:172.21.15.170:6834/2176248563] conn(0x5651987f7f90 0x56519882fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:30.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.284+0000 7fd2197fa640 10 client.7275 ms_handle_connect on v2:172.21.15.170:6832/2176248563 2024-03-28T23:33:30.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.284+0000 7fd23a59c640 1 --2- 172.21.15.161:0/3971742361 >> [v2:172.21.15.170:6832/2176248563,v1:172.21.15.170:6834/2176248563] conn(0x5651987f7f90 0x56519882fe80 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=265c1ee8ba72935a server_cookie=67cadd314fd5b5df in_seq=0 out_seq=0 2024-03-28T23:33:30.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.284+0000 7fd2197fa640 1 -- 172.21.15.161:0/3971742361 <== mds.0 v2:172.21.15.170:6832/2176248563 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x565198746030 con 0x5651987f7f90 2024-03-28T23:33:30.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.284+0000 7fd2197fa640 10 client.7275 handle_command_reply: tid=0 2024-03-28T23:33:30.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.284+0000 7fd243b84740 1 client.7275 shutdown 2024-03-28T23:33:30.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.284+0000 7fd243b84740 1 -- 172.21.15.161:0/3971742361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987d5190 msgr2=0x5651987cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:30.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.284+0000 7fd243b84740 1 --2- 172.21.15.161:0/3971742361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987d5190 0x5651987cfa00 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fd224005fa0 tx=0x7fd2240046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.284+0000 7fd243b84740 1 -- 172.21.15.161:0/3971742361 shutdown_connections 2024-03-28T23:33:30.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.284+0000 7fd243b84740 1 --2- 172.21.15.161:0/3971742361 >> [v2:172.21.15.170:6832/2176248563,v1:172.21.15.170:6834/2176248563] conn(0x5651987f7f90 0x56519882fe80 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.284+0000 7fd243b84740 1 --2- 172.21.15.161:0/3971742361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987d5190 0x5651987cfa00 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.284+0000 7fd2197fa640 0 client.7275 ms_handle_reset on v2:172.21.15.170:6832/2176248563 2024-03-28T23:33:30.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.284+0000 7fd243b84740 1 --2- 172.21.15.161:0/3971742361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651988247e0 0x565198809a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.284+0000 7fd243b84740 1 --2- 172.21.15.161:0/3971742361 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651987d4db0 0x5651987f98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.285+0000 7fd243b84740 1 -- 172.21.15.161:0/3971742361 >> 172.21.15.161:0/3971742361 conn(0x5651987d3310 msgr2=0x5651987d3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:30.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.285+0000 7fd243b84740 1 -- 172.21.15.161:0/3971742361 shutdown_connections 2024-03-28T23:33:30.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.285+0000 7fd243b84740 1 -- 172.21.15.161:0/3971742361 wait complete. 2024-03-28T23:33:30.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.285+0000 7fd243b84740 20 client.7275 trim_cache size 0 max 16384 2024-03-28T23:33:30.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.302+0000 7fd243b84740 1 Processor -- start 2024-03-28T23:33:30.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.302+0000 7fd243b84740 1 -- start start 2024-03-28T23:33:30.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.303+0000 7fd243b84740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651987d4db0 0x5651987f8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:30.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.303+0000 7fd243b84740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651987d5190 0x5651987d36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:30.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.303+0000 7fd243b84740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987f7f90 0x5651987d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:30.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.303+0000 7fd243b84740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd23c1f6ef0 con 0x5651987f7f90 2024-03-28T23:33:30.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.303+0000 7fd243b84740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd23c1f62b0 con 0x5651987d5190 2024-03-28T23:33:30.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.303+0000 7fd243b84740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd23c1f68a0 con 0x5651987d4db0 2024-03-28T23:33:30.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.303+0000 7fd23ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987f7f90 0x5651987d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:30.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.303+0000 7fd23ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987f7f90 0x5651987d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41352/0 (socket says 172.21.15.161:41352) 2024-03-28T23:33:30.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.303+0000 7fd23ad9d640 1 -- 172.21.15.161:0/722941106 learned_addr learned my addr 172.21.15.161:0/722941106 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:30.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.303+0000 7fd233fff640 1 --2- 172.21.15.161:0/722941106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651987d5190 0x5651987d36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:30.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.303+0000 7fd23a59c640 1 --2- 172.21.15.161:0/722941106 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651987d4db0 0x5651987f8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:30.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.303+0000 7fd23ad9d640 1 -- 172.21.15.161:0/722941106 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651987d4db0 msgr2=0x5651987f8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:30.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.303+0000 7fd23ad9d640 1 --2- 172.21.15.161:0/722941106 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651987d4db0 0x5651987f8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.303+0000 7fd23ad9d640 1 -- 172.21.15.161:0/722941106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651987d5190 msgr2=0x5651987d36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:30.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.303+0000 7fd23ad9d640 1 --2- 172.21.15.161:0/722941106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651987d5190 0x5651987d36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.303+0000 7fd23ad9d640 1 -- 172.21.15.161:0/722941106 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56519880a960 con 0x5651987f7f90 2024-03-28T23:33:30.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.304+0000 7fd23ad9d640 1 --2- 172.21.15.161:0/722941106 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987f7f90 0x5651987d3c30 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fd234009d90 tx=0x7fd23402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3441ddba4f814050 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:30.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.304+0000 7fd21b7fe640 1 -- 172.21.15.161:0/722941106 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd23404e070 con 0x5651987f7f90 2024-03-28T23:33:30.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.304+0000 7fd21b7fe640 1 -- 172.21.15.161:0/722941106 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd23402cba0 con 0x5651987f7f90 2024-03-28T23:33:30.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.304+0000 7fd21b7fe640 1 -- 172.21.15.161:0/722941106 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd234039040 con 0x5651987f7f90 2024-03-28T23:33:30.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.304+0000 7fd243b84740 1 -- 172.21.15.161:0/722941106 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987f7f90 msgr2=0x5651987d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:30.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.304+0000 7fd243b84740 1 --2- 172.21.15.161:0/722941106 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987f7f90 0x5651987d3c30 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fd234009d90 tx=0x7fd23402e000 comp rx=0 tx=0).stop 2024-03-28T23:33:30.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.304+0000 7fd243b84740 1 -- 172.21.15.161:0/722941106 shutdown_connections 2024-03-28T23:33:30.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.304+0000 7fd243b84740 1 --2- 172.21.15.161:0/722941106 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987f7f90 0x5651987d3c30 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.304+0000 7fd243b84740 1 --2- 172.21.15.161:0/722941106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651987d5190 0x5651987d36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.304+0000 7fd243b84740 1 --2- 172.21.15.161:0/722941106 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651987d4db0 0x5651987f8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.304+0000 7fd243b84740 1 -- 172.21.15.161:0/722941106 >> 172.21.15.161:0/722941106 conn(0x5651987d3310 msgr2=0x5651987f6ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:30.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.304+0000 7fd243b84740 1 -- 172.21.15.161:0/722941106 shutdown_connections 2024-03-28T23:33:30.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.304+0000 7fd243b84740 1 -- 172.21.15.161:0/722941106 wait complete. 2024-03-28T23:33:30.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.305+0000 7fd243b84740 1 -- start start 2024-03-28T23:33:30.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.305+0000 7fd243b84740 1 Processor -- start 2024-03-28T23:33:30.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.306+0000 7fd243b84740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651987d4db0 0x565198834280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:30.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.306+0000 7fd243b84740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651987d5190 0x5651987d3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:30.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.306+0000 7fd243b84740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987f7f90 0x565198888790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:30.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.306+0000 7fd243b84740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x565198808220 con 0x5651987f7f90 2024-03-28T23:33:30.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.306+0000 7fd243b84740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x565198824c90 con 0x5651987d4db0 2024-03-28T23:33:30.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.306+0000 7fd243b84740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x56519880a2f0 con 0x5651987d5190 2024-03-28T23:33:30.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.306+0000 7fd243b84740 10 client.0 _ll_register_callbacks cb 0x7fd224011bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:33:30.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.306+0000 7fd23ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987f7f90 0x565198888790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:30.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.306+0000 7fd23ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987f7f90 0x565198888790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41358/0 (socket says 172.21.15.161:41358) 2024-03-28T23:33:30.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.306+0000 7fd23a59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651987d4db0 0x565198834280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:30.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.306+0000 7fd23ad9d640 1 -- 172.21.15.161:0/3851580076 learned_addr learned my addr 172.21.15.161:0/3851580076 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:30.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.306+0000 7fd233fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651987d5190 0x5651987d3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:30.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.306+0000 7fd23ad9d640 1 -- 172.21.15.161:0/3851580076 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651987d5190 msgr2=0x5651987d3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:30.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.306+0000 7fd23ad9d640 1 --2- 172.21.15.161:0/3851580076 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651987d5190 0x5651987d3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.306+0000 7fd23ad9d640 1 -- 172.21.15.161:0/3851580076 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651987d4db0 msgr2=0x565198834280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:30.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.306+0000 7fd23ad9d640 1 --2- 172.21.15.161:0/3851580076 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651987d4db0 0x565198834280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.306+0000 7fd23ad9d640 1 -- 172.21.15.161:0/3851580076 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565198807f00 con 0x5651987f7f90 2024-03-28T23:33:30.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.306+0000 7fd23ad9d640 1 --2- 172.21.15.161:0/3851580076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987f7f90 0x565198888790 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fd234009eb0 tx=0x7fd234004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a358da692f759218 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:30.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.306+0000 7fd2197fa640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:33:30.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.306+0000 7fd2197fa640 1 -- 172.21.15.161:0/3851580076 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd23404e030 con 0x5651987f7f90 2024-03-28T23:33:30.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.306+0000 7fd2197fa640 1 -- 172.21.15.161:0/3851580076 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 6101+0+0 (secure 0 0 0) 0x7fd234046050 con 0x5651987f7f90 2024-03-28T23:33:30.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.306+0000 7fd243b84740 1 -- 172.21.15.161:0/3851580076 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd23402cba0 con 0x5651987f7f90 2024-03-28T23:33:30.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.307+0000 7fd2197fa640 1 -- 172.21.15.161:0/3851580076 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7fd23405c050 con 0x5651987f7f90 2024-03-28T23:33:30.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.307+0000 7fd243b84740 10 client.7281 fetch_fsmap learned FSMap version 79 2024-03-28T23:33:30.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.307+0000 7fd243b84740 1 -- 172.21.15.161:0/3851580076 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=2+}) -- 0x5651987f97c0 con 0x5651987f7f90 2024-03-28T23:33:30.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.307+0000 7fd2197fa640 1 -- 172.21.15.161:0/3851580076 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd23402cba0 con 0x5651987f7f90 2024-03-28T23:33:30.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.307+0000 7fd2197fa640 1 -- 172.21.15.161:0/3851580076 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 79) ==== 3494+0+0 (secure 0 0 0) 0x7fd23405f020 con 0x5651987f7f90 2024-03-28T23:33:30.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.307+0000 7fd243b84740 10 client.7281 fetch_fsmap finished waiting for FSMap version 79 2024-03-28T23:33:30.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.307+0000 7fd243b84740 10 client.7281 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-28T23:33:30.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.307+0000 7fd243b84740 20 client.7281 populate_metadata read hostname 'smithi161' 2024-03-28T23:33:30.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.307+0000 7fd243b84740 1 --2- 172.21.15.161:0/3851580076 >> [v2:172.21.15.170:6832/2176248563,v1:172.21.15.170:6834/2176248563] conn(0x5651988247e0 0x5651988068f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:30.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.307+0000 7fd243b84740 4 client.7281 mds_command: new command op to 7101 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:33:30.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.307+0000 7fd243b84740 1 -- 172.21.15.161:0/3851580076 --> [v2:172.21.15.170:6832/2176248563,v1:172.21.15.170:6834/2176248563] -- command(tid 0: {"prefix": "status"}) -- 0x565198740380 con 0x5651988247e0 2024-03-28T23:33:30.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.307+0000 7fd23a59c640 1 --2- 172.21.15.161:0/3851580076 >> [v2:172.21.15.170:6832/2176248563,v1:172.21.15.170:6834/2176248563] conn(0x5651988247e0 0x5651988068f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:30.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.307+0000 7fd2197fa640 10 client.7281 ms_handle_connect on v2:172.21.15.170:6832/2176248563 2024-03-28T23:33:30.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.307+0000 7fd23a59c640 1 --2- 172.21.15.161:0/3851580076 >> [v2:172.21.15.170:6832/2176248563,v1:172.21.15.170:6834/2176248563] conn(0x5651988247e0 0x5651988068f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e4100c542311357 server_cookie=85d1b0bafc0f1bdd in_seq=0 out_seq=0 2024-03-28T23:33:30.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.308+0000 7fd2197fa640 1 -- 172.21.15.161:0/3851580076 <== mds.0 v2:172.21.15.170:6832/2176248563 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x565198740380 con 0x5651988247e0 2024-03-28T23:33:30.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.308+0000 7fd2197fa640 10 client.7281 handle_command_reply: tid=0 2024-03-28T23:33:30.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.308+0000 7fd243b84740 1 client.7281 shutdown 2024-03-28T23:33:30.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.308+0000 7fd243b84740 1 -- 172.21.15.161:0/3851580076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987f7f90 msgr2=0x565198888790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:30.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.308+0000 7fd243b84740 1 --2- 172.21.15.161:0/3851580076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987f7f90 0x565198888790 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fd234009eb0 tx=0x7fd234004ab0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.308+0000 7fd243b84740 1 -- 172.21.15.161:0/3851580076 shutdown_connections 2024-03-28T23:33:30.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.308+0000 7fd243b84740 1 --2- 172.21.15.161:0/3851580076 >> [v2:172.21.15.170:6832/2176248563,v1:172.21.15.170:6834/2176248563] conn(0x5651988247e0 0x5651988068f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.308+0000 7fd243b84740 1 --2- 172.21.15.161:0/3851580076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651987f7f90 0x565198888790 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.308+0000 7fd243b84740 1 --2- 172.21.15.161:0/3851580076 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5651987d5190 0x5651987d3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.308+0000 7fd2197fa640 0 client.7281 ms_handle_reset on v2:172.21.15.170:6832/2176248563 2024-03-28T23:33:30.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.308+0000 7fd243b84740 1 --2- 172.21.15.161:0/3851580076 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5651987d4db0 0x565198834280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.308+0000 7fd243b84740 1 -- 172.21.15.161:0/3851580076 >> 172.21.15.161:0/3851580076 conn(0x5651987d3310 msgr2=0x5651987f8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:30.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.308+0000 7fd243b84740 1 -- 172.21.15.161:0/3851580076 shutdown_connections 2024-03-28T23:33:30.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.308+0000 7fd243b84740 1 -- 172.21.15.161:0/3851580076 wait complete. 2024-03-28T23:33:30.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.308+0000 7fd243b84740 20 client.7281 trim_cache size 0 max 16384 2024-03-28T23:33:30.315 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:33:30.315 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:33:30.315 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:33:30.315 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 7101, 2024-03-28T23:33:30.315 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:33:30.315 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:33:30.316 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:33:30.316 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 6.1376495110000002, 2024-03-28T23:33:30.316 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 79, 2024-03-28T23:33:30.316 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 94, 2024-03-28T23:33:30.316 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 94, 2024-03-28T23:33:30.316 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 11.228169819 2024-03-28T23:33:30.316 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:33:30.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.308+0000 7fd240fa1640 1 -- 172.21.15.161:0/388091815 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd21c06b4d0 msgr2=0x7fd21c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:30.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.308+0000 7fd240fa1640 1 --2- 172.21.15.161:0/388091815 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd21c06b4d0 0x7fd21c06d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fd23c1532e0 tx=0x7fd224009110 comp rx=0 tx=0).stop 2024-03-28T23:33:30.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.308+0000 7fd240fa1640 1 -- 172.21.15.161:0/388091815 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd23c0c5410 msgr2=0x7fd23c148430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:30.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.308+0000 7fd240fa1640 1 --2- 172.21.15.161:0/388091815 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd23c0c5410 0x7fd23c148430 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fd22c00c4d0 tx=0x7fd22c00c990 comp rx=0 tx=0).stop 2024-03-28T23:33:30.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.308+0000 7fd240fa1640 1 -- 172.21.15.161:0/388091815 shutdown_connections 2024-03-28T23:33:30.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.308+0000 7fd240fa1640 1 --2- 172.21.15.161:0/388091815 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd21c06b4d0 0x7fd21c06d990 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.308+0000 7fd240fa1640 1 --2- 172.21.15.161:0/388091815 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd23c1553a0 0x7fd23c1ea980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.308+0000 7fd240fa1640 1 --2- 172.21.15.161:0/388091815 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd23c152890 0x7fd23c157d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.308+0000 7fd240fa1640 1 --2- 172.21.15.161:0/388091815 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd23c0c5410 0x7fd23c148430 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:30.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.309+0000 7fd240fa1640 1 -- 172.21.15.161:0/388091815 >> 172.21.15.161:0/388091815 conn(0x7fd23c0ac620 msgr2=0x7fd23c14bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:30.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.309+0000 7fd240fa1640 1 -- 172.21.15.161:0/388091815 shutdown_connections 2024-03-28T23:33:30.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:30.309+0000 7fd240fa1640 1 -- 172.21.15.161:0/388091815 wait complete. 2024-03-28T23:33:30.320 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 7, "epoch": 79, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 7, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:33:23.516182+0000", "damaged": [], "data_pools": [ 16 ], "enabled": true, "epoch": 79, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7101": { "addr": "172.21.15.170:6834/2176248563", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 2176248563, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 2176248563, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7101, "incarnation": 77, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 15, "modified": "2024-03-28T23:33:25.515642+0000", "qdb_cluster": [ 7101 ], "qdb_leader": 7101, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7101 } } } ], "standbys": [ { "addr": "172.21.15.170:6835/3079114491", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 3079114491, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 3079114491, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 76, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5963, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6836/4280217233", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 4280217233, "type": "v2" }, { "addr": "172.21.15.161:6836", "nonce": 4280217233, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 76, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6547, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6837/404132405", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6835", "nonce": 404132405, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 404132405, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 76, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7086, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:33:30.321 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 7, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-28T23:33:30.321 INFO:tasks.cephfs.test_failover:current = [{'gid': 7086, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6837/404132405', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 404132405}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 404132405}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 76}, {'gid': 5963, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/3079114491', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 3079114491}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 3079114491}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 76}, {'gid': 6547, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6836/4280217233', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 4280217233}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 4280217233}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 76}, {'gid': 7101, 'name': 'd', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.170:6834/2176248563', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 2176248563}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 2176248563}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}] 2024-03-28T23:33:30.321 DEBUG:tasks.cephfs.test_failover:7 != -1 2024-03-28T23:33:30.321 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-28T23:33:31.428 INFO:tasks.ceph.mds.d.smithi170.stderr: -104> 2024-03-28T23:33:29.438+0000 7f2bb1daf640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:33:31.452 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2024-03-28T23:33:34.438 INFO:tasks.ceph.mds.b.smithi170.stderr:2024-03-28T23:33:34.438+0000 7f55ef4c5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:33:35.322 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:33:35.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.435+0000 7ff3e882e640 1 Processor -- start 2024-03-28T23:33:35.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.435+0000 7ff3e882e640 1 -- start start 2024-03-28T23:33:35.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.436+0000 7ff3e882e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3e406ca20 0x7ff3e406ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:35.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.436+0000 7ff3e882e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff3e40a98c0 0x7ff3e40a9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:35.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.436+0000 7ff3e882e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3e4151850 con 0x7ff3e414eca0 2024-03-28T23:33:35.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.436+0000 7ff3e882e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7ff3e4152460 con 0x7ff3e40a98c0 2024-03-28T23:33:35.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.436+0000 7ff3e882e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff3e4151b30 con 0x7ff3e406ca20 2024-03-28T23:33:35.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.436+0000 7ff3e259c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ff3e414eca0 0x7ff3e40c1880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52978/0 (socket says 172.21.15.161:52978) 2024-03-28T23:33:35.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.436+0000 7ff3e259c640 1 -- 172.21.15.161:0/2288120778 learned_addr learned my addr 172.21.15.161:0/2288120778 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:35.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.436+0000 7ff3dbfff640 1 --2- 172.21.15.161:0/2288120778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3e406ca20 0x7ff3e406ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:35.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.436+0000 7ff3e2d9d640 1 --2- 172.21.15.161:0/2288120778 >> v2:172.21.15.170:3300/0 conn(0x7ff3e40a98c0 0x7ff3e40a9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:35.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.437+0000 7ff3dbfff640 1 -- 172.21.15.161:0/2288120778 >> v2:172.21.15.170:3300/0 conn(0x7ff3e40a98c0 msgr2=0x7ff3e40a9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:35.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.437+0000 7ff3dbfff640 1 --2- 172.21.15.161:0/2288120778 >> v2:172.21.15.170:3300/0 conn(0x7ff3e40a98c0 0x7ff3e40a9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:35.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.437+0000 7ff3dbfff640 1 -- 172.21.15.161:0/2288120778 >> v1:172.21.15.161:6789/0 conn(0x7ff3e414eca0 legacy=0x7ff3e40c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:35.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.437+0000 7ff3dbfff640 1 -- 172.21.15.161:0/2288120778 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e4151e10 con 0x7ff3e406ca20 2024-03-28T23:33:35.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.437+0000 7ff3e2d9d640 1 --2- 172.21.15.161:0/2288120778 >> v2:172.21.15.170:3300/0 conn(0x7ff3e40a98c0 0x7ff3e40a9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:35.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.437+0000 7ff3dbfff640 1 --2- 172.21.15.161:0/2288120778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3e406ca20 0x7ff3e406ce80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc0097a0 tx=0x7ff3cc02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=439c328c7905ce10 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:35.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.438+0000 7ff3e1d9b640 1 -- 172.21.15.161:0/2288120778 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc0315a0 con 0x7ff3e406ca20 2024-03-28T23:33:35.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.438+0000 7ff3e1d9b640 1 -- 172.21.15.161:0/2288120778 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3cc037070 con 0x7ff3e406ca20 2024-03-28T23:33:35.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.438+0000 7ff3e1d9b640 1 -- 172.21.15.161:0/2288120778 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc031960 con 0x7ff3e406ca20 2024-03-28T23:33:35.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.438+0000 7ff3e882e640 1 -- 172.21.15.161:0/2288120778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3e406ca20 msgr2=0x7ff3e406ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:35.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.438+0000 7ff3e882e640 1 --2- 172.21.15.161:0/2288120778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3e406ca20 0x7ff3e406ce80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc0097a0 tx=0x7ff3cc02b840 comp rx=0 tx=0).stop 2024-03-28T23:33:35.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.438+0000 7ff3e882e640 1 -- 172.21.15.161:0/2288120778 shutdown_connections 2024-03-28T23:33:35.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.438+0000 7ff3e882e640 1 --2- 172.21.15.161:0/2288120778 >> v2:172.21.15.170:3300/0 conn(0x7ff3e40a98c0 0x7ff3e40a9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:35.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.438+0000 7ff3e882e640 1 --2- 172.21.15.161:0/2288120778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3e406ca20 0x7ff3e406ce80 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:35.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.438+0000 7ff3e882e640 1 -- 172.21.15.161:0/2288120778 >> 172.21.15.161:0/2288120778 conn(0x7ff3e40c6670 msgr2=0x7ff3e40c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:35.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.438+0000 7ff3e882e640 1 -- 172.21.15.161:0/2288120778 shutdown_connections 2024-03-28T23:33:35.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.438+0000 7ff3e882e640 1 -- 172.21.15.161:0/2288120778 wait complete. 2024-03-28T23:33:35.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.440+0000 7ff3e882e640 1 Processor -- start 2024-03-28T23:33:35.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.440+0000 7ff3e882e640 1 -- start start 2024-03-28T23:33:35.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.441+0000 7ff3e882e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3e406ca20 0x7ff3e40bfc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:35.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.441+0000 7ff3e882e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3e40a98c0 0x7ff3e4072720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:35.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.441+0000 7ff3e882e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3e414eca0 0x7ff3e4072c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:35.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.441+0000 7ff3e882e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e41514b0 con 0x7ff3e40a98c0 2024-03-28T23:33:35.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.441+0000 7ff3e882e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e414f080 con 0x7ff3e414eca0 2024-03-28T23:33:35.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.441+0000 7ff3e882e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff3e4151630 con 0x7ff3e406ca20 2024-03-28T23:33:35.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.441+0000 7ff3dbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3e40a98c0 0x7ff3e4072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:35.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.441+0000 7ff3e259c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3e406ca20 0x7ff3e40bfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:35.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.441+0000 7ff3e259c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3e406ca20 0x7ff3e40bfc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:41226/0 (socket says 172.21.15.161:41226) 2024-03-28T23:33:35.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.441+0000 7ff3e259c640 1 -- 172.21.15.161:0/891257545 learned_addr learned my addr 172.21.15.161:0/891257545 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:35.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.441+0000 7ff3dbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3e40a98c0 0x7ff3e4072720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54252/0 (socket says 172.21.15.161:54252) 2024-03-28T23:33:35.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.441+0000 7ff3e2d9d640 1 --2- 172.21.15.161:0/891257545 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3e414eca0 0x7ff3e4072c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:35.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.442+0000 7ff3e259c640 1 -- 172.21.15.161:0/891257545 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3e414eca0 msgr2=0x7ff3e4072c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:35.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.442+0000 7ff3e259c640 1 --2- 172.21.15.161:0/891257545 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3e414eca0 0x7ff3e4072c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:35.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.442+0000 7ff3e259c640 1 -- 172.21.15.161:0/891257545 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3e40a98c0 msgr2=0x7ff3e4072720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:35.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.442+0000 7ff3e259c640 1 --2- 172.21.15.161:0/891257545 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3e40a98c0 0x7ff3e4072720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:35.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.442+0000 7ff3e259c640 1 -- 172.21.15.161:0/891257545 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e41ea990 con 0x7ff3e406ca20 2024-03-28T23:33:35.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.442+0000 7ff3e2d9d640 1 --2- 172.21.15.161:0/891257545 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3e414eca0 0x7ff3e4072c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:35.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.442+0000 7ff3dbfff640 1 --2- 172.21.15.161:0/891257545 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3e40a98c0 0x7ff3e4072720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:35.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.442+0000 7ff3e259c640 1 --2- 172.21.15.161:0/891257545 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3e406ca20 0x7ff3e40bfc70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff3d400ca70 tx=0x7ff3d400cf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:35.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.442+0000 7ff3daffd640 1 -- 172.21.15.161:0/891257545 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4017070 con 0x7ff3e406ca20 2024-03-28T23:33:35.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.442+0000 7ff3daffd640 1 -- 172.21.15.161:0/891257545 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3d400abd0 con 0x7ff3e406ca20 2024-03-28T23:33:35.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.442+0000 7ff3e882e640 1 -- 172.21.15.161:0/891257545 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3e41eac50 con 0x7ff3e406ca20 2024-03-28T23:33:35.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.442+0000 7ff3daffd640 1 -- 172.21.15.161:0/891257545 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4012400 con 0x7ff3e406ca20 2024-03-28T23:33:35.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.442+0000 7ff3e882e640 1 -- 172.21.15.161:0/891257545 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3e41eb210 con 0x7ff3e406ca20 2024-03-28T23:33:35.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.444+0000 7ff3daffd640 1 -- 172.21.15.161:0/891257545 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff3d401a070 con 0x7ff3e406ca20 2024-03-28T23:33:35.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.444+0000 7ff3e882e640 1 -- 172.21.15.161:0/891257545 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3e404f4b0 con 0x7ff3e406ca20 2024-03-28T23:33:35.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.444+0000 7ff3daffd640 1 --2- 172.21.15.161:0/891257545 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff3c406b4d0 0x7ff3c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:35.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.444+0000 7ff3daffd640 1 -- 172.21.15.161:0/891257545 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(97..97 src has 1..97) ==== 6245+0+0 (secure 0 0 0) 0x7ff3d408a020 con 0x7ff3e406ca20 2024-03-28T23:33:35.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.444+0000 7ff3dbfff640 1 --2- 172.21.15.161:0/891257545 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff3c406b4d0 0x7ff3c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:35.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.445+0000 7ff3dbfff640 1 --2- 172.21.15.161:0/891257545 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff3c406b4d0 0x7ff3c406d990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc036000 tx=0x7ff3cc03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:35.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.449+0000 7ff3daffd640 1 -- 172.21.15.161:0/891257545 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff3d40586c0 con 0x7ff3e406ca20 2024-03-28T23:33:35.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.659+0000 7ff3e882e640 1 -- 172.21.15.161:0/891257545 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3e4150fb0 con 0x7ff3e406ca20 2024-03-28T23:33:35.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.660+0000 7ff3daffd640 1 -- 172.21.15.161:0/891257545 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) ==== 94+0+5192 (secure 0 0 0) 0x7ff3d405c760 con 0x7ff3e406ca20 2024-03-28T23:33:35.660 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:35.660 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":84,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6547,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4280217233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4280217233},{"type":"v1","addr":"172.21.15.161:6836","nonce":4280217233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":6706,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/2647418289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":2647418289},{"type":"v1","addr":"172.21.15.170:6834","nonce":2647418289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":7086,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/404132405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":404132405},{"type":"v1","addr":"172.21.15.161:6837","nonce":404132405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:33:23.516182+0000","modified":"2024-03-28T23:33:34.430703+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5963},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5963":{"gid":5963,"name":"b","rank":0,"incarnation":81,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6835/3079114491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3079114491},{"type":"v1","addr":"172.21.15.170:6835","nonce":3079114491}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5963,"qdb_cluster":[5963]},"id":7}]} 2024-03-28T23:33:35.661 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 84 2024-03-28T23:33:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.662+0000 7ff3e882e640 1 -- 172.21.15.161:0/891257545 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff3c406b4d0 msgr2=0x7ff3c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.662+0000 7ff3e882e640 1 --2- 172.21.15.161:0/891257545 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff3c406b4d0 0x7ff3c406d990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc036000 tx=0x7ff3cc03d040 comp rx=0 tx=0).stop 2024-03-28T23:33:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.662+0000 7ff3e882e640 1 -- 172.21.15.161:0/891257545 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3e406ca20 msgr2=0x7ff3e40bfc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.662+0000 7ff3e882e640 1 --2- 172.21.15.161:0/891257545 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3e406ca20 0x7ff3e40bfc70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff3d400ca70 tx=0x7ff3d400cf30 comp rx=0 tx=0).stop 2024-03-28T23:33:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.662+0000 7ff3e882e640 1 -- 172.21.15.161:0/891257545 shutdown_connections 2024-03-28T23:33:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.662+0000 7ff3e882e640 1 --2- 172.21.15.161:0/891257545 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff3c406b4d0 0x7ff3c406d990 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.662+0000 7ff3e882e640 1 --2- 172.21.15.161:0/891257545 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3e40a98c0 0x7ff3e4072720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.662+0000 7ff3e882e640 1 --2- 172.21.15.161:0/891257545 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3e414eca0 0x7ff3e4072c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.662+0000 7ff3e882e640 1 --2- 172.21.15.161:0/891257545 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3e406ca20 0x7ff3e40bfc70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.662+0000 7ff3e882e640 1 -- 172.21.15.161:0/891257545 >> 172.21.15.161:0/891257545 conn(0x7ff3e40c6670 msgr2=0x7ff3e40c1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.662+0000 7ff3e882e640 1 -- 172.21.15.161:0/891257545 shutdown_connections 2024-03-28T23:33:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:35.662+0000 7ff3e882e640 1 -- 172.21.15.161:0/891257545 wait complete. 2024-03-28T23:33:36.076 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:33:23.516182+0000', 'modified': '2024-03-28T23:33:34.430703+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 95, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5963}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5963': {'gid': 5963, 'name': 'b', 'rank': 0, 'incarnation': 81, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.170:6835/3079114491', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 3079114491}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 3079114491}]}, 'join_fscid': 7, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5963, 'qdb_cluster': [5963]} 2024-03-28T23:33:36.076 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:33:36.076 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T23:33:36.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.180+0000 7f7be0f80640 1 Processor -- start 2024-03-28T23:33:36.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.180+0000 7f7be0f80640 1 -- start start 2024-03-28T23:33:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.180+0000 7f7be0f80640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7bdc06ce80 0x7f7bdc0aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.180+0000 7f7be0f80640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bdc0a9680 0x7f7bdc0aab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.181+0000 7f7be0f80640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bdc14c250 con 0x7f7bdc14be70 2024-03-28T23:33:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.181+0000 7f7be0f80640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f7bdc1529c0 con 0x7f7bdc06ce80 2024-03-28T23:33:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.181+0000 7f7be0f80640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7bdc0a9dc0 con 0x7f7bdc0a9680 2024-03-28T23:33:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.181+0000 7f7bd9d9b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7bdc06ce80 0x7f7bdc0aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.181+0000 7f7bd9d9b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7bdc06ce80 0x7f7bdc0aa5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54264/0 (socket says 172.21.15.161:54264) 2024-03-28T23:33:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.181+0000 7f7bd9d9b640 1 -- 172.21.15.161:0/997143081 learned_addr learned my addr 172.21.15.161:0/997143081 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.181+0000 7f7bdad9d640 1 --2- 172.21.15.161:0/997143081 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bdc0a9680 0x7f7bdc0aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.181+0000 7f7bd9d9b640 1 -- 172.21.15.161:0/997143081 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bdc0a9680 msgr2=0x7f7bdc0aab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.181+0000 7f7bd9d9b640 1 --2- 172.21.15.161:0/997143081 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bdc0a9680 0x7f7bdc0aab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.181+0000 7f7bd9d9b640 1 -- 172.21.15.161:0/997143081 >> v1:172.21.15.170:6789/0 conn(0x7f7bdc14be70 legacy=0x7f7bdc0ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:36.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.181+0000 7f7bd9d9b640 1 -- 172.21.15.161:0/997143081 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bdc0aa0a0 con 0x7f7bdc06ce80 2024-03-28T23:33:36.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.182+0000 7f7bd9d9b640 1 --2- 172.21.15.161:0/997143081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bdc06ce80 0x7f7bdc0aa5c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0009870 tx=0x7f7bd002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba3bb209e15f611c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:36.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.182+0000 7f7bd959a640 1 -- 172.21.15.161:0/997143081 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd00317c0 con 0x7f7bdc06ce80 2024-03-28T23:33:36.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.182+0000 7f7bd959a640 1 -- 172.21.15.161:0/997143081 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bd0037070 con 0x7f7bdc06ce80 2024-03-28T23:33:36.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.182+0000 7f7bd959a640 1 -- 172.21.15.161:0/997143081 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd0031b80 con 0x7f7bdc06ce80 2024-03-28T23:33:36.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.182+0000 7f7be0f80640 1 -- 172.21.15.161:0/997143081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bdc06ce80 msgr2=0x7f7bdc0aa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:36.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.182+0000 7f7be0f80640 1 --2- 172.21.15.161:0/997143081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bdc06ce80 0x7f7bdc0aa5c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0009870 tx=0x7f7bd002b860 comp rx=0 tx=0).stop 2024-03-28T23:33:36.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.182+0000 7f7be0f80640 1 -- 172.21.15.161:0/997143081 shutdown_connections 2024-03-28T23:33:36.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.182+0000 7f7be0f80640 1 --2- 172.21.15.161:0/997143081 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bdc0a9680 0x7f7bdc0aab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.182+0000 7f7be0f80640 1 --2- 172.21.15.161:0/997143081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bdc06ce80 0x7f7bdc0aa5c0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.182+0000 7f7be0f80640 1 -- 172.21.15.161:0/997143081 >> 172.21.15.161:0/997143081 conn(0x7f7bdc0c6670 msgr2=0x7f7bdc0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:36.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.183+0000 7f7be0f80640 1 -- 172.21.15.161:0/997143081 shutdown_connections 2024-03-28T23:33:36.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.183+0000 7f7be0f80640 1 -- 172.21.15.161:0/997143081 wait complete. 2024-03-28T23:33:36.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.184+0000 7f7be0f80640 1 Processor -- start 2024-03-28T23:33:36.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.184+0000 7f7be0f80640 1 -- start start 2024-03-28T23:33:36.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.184+0000 7f7be0f80640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7bdc06ce80 0x7f7bdc1477c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.184+0000 7f7be0f80640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bdc0a9680 0x7f7bdc0c1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.184+0000 7f7be0f80640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bdc14be70 0x7f7bdc1ec840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.184+0000 7f7be0f80640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7bdc0aba30 con 0x7f7bdc14be70 2024-03-28T23:33:36.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.184+0000 7f7be0f80640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7bdc0ab8b0 con 0x7f7bdc06ce80 2024-03-28T23:33:36.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.184+0000 7f7be0f80640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7bdc06b150 con 0x7f7bdc0a9680 2024-03-28T23:33:36.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.185+0000 7f7bdad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bdc14be70 0x7f7bdc1ec840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.185+0000 7f7bdad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bdc14be70 0x7f7bdc1ec840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54268/0 (socket says 172.21.15.161:54268) 2024-03-28T23:33:36.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.185+0000 7f7bdad9d640 1 -- 172.21.15.161:0/518309158 learned_addr learned my addr 172.21.15.161:0/518309158 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:36.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.185+0000 7f7bd9d9b640 1 --2- 172.21.15.161:0/518309158 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bdc0a9680 0x7f7bdc0c1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.185+0000 7f7bda59c640 1 --2- 172.21.15.161:0/518309158 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7bdc06ce80 0x7f7bdc1477c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.185+0000 7f7bdad9d640 1 -- 172.21.15.161:0/518309158 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bdc0a9680 msgr2=0x7f7bdc0c1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:36.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.185+0000 7f7bdad9d640 1 --2- 172.21.15.161:0/518309158 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bdc0a9680 0x7f7bdc0c1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.185+0000 7f7bdad9d640 1 -- 172.21.15.161:0/518309158 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7bdc06ce80 msgr2=0x7f7bdc1477c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:36.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.185+0000 7f7bdad9d640 1 --2- 172.21.15.161:0/518309158 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7bdc06ce80 0x7f7bdc1477c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.185+0000 7f7bdad9d640 1 -- 172.21.15.161:0/518309158 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bdc0c26f0 con 0x7f7bdc14be70 2024-03-28T23:33:36.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.185+0000 7f7bdad9d640 1 --2- 172.21.15.161:0/518309158 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bdc14be70 0x7f7bdc1ec840 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f7bd400e7f0 tx=0x7f7bd400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:36.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.185+0000 7f7bcaffd640 1 -- 172.21.15.161:0/518309158 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4017070 con 0x7f7bdc14be70 2024-03-28T23:33:36.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.185+0000 7f7bcaffd640 1 -- 172.21.15.161:0/518309158 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bd400ee40 con 0x7f7bdc14be70 2024-03-28T23:33:36.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.185+0000 7f7be0f80640 1 -- 172.21.15.161:0/518309158 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bdc153490 con 0x7f7bdc14be70 2024-03-28T23:33:36.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.185+0000 7f7bcaffd640 1 -- 172.21.15.161:0/518309158 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd400cb20 con 0x7f7bdc14be70 2024-03-28T23:33:36.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.185+0000 7f7be0f80640 1 -- 172.21.15.161:0/518309158 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bdc153a50 con 0x7f7bdc14be70 2024-03-28T23:33:36.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.187+0000 7f7bcaffd640 1 -- 172.21.15.161:0/518309158 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7bd401a070 con 0x7f7bdc14be70 2024-03-28T23:33:36.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.187+0000 7f7bcaffd640 1 --2- 172.21.15.161:0/518309158 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7bbc06b4d0 0x7f7bbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.187+0000 7f7bcaffd640 1 -- 172.21.15.161:0/518309158 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6245+0+0 (secure 0 0 0) 0x7f7bd408a120 con 0x7f7bdc14be70 2024-03-28T23:33:36.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.187+0000 7f7bda59c640 1 --2- 172.21.15.161:0/518309158 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7bbc06b4d0 0x7f7bbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.187+0000 7f7bda59c640 1 --2- 172.21.15.161:0/518309158 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7bbc06b4d0 0x7f7bbc06d990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc150dd0 tx=0x7f7bcc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:36.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.188+0000 7f7be3c33740 1 Processor -- start 2024-03-28T23:33:36.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.188+0000 7f7be3c33740 1 -- start start 2024-03-28T23:33:36.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.189+0000 7f7be3c33740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cd0faf660 0x557cd0f65e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.189+0000 7f7be3c33740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557cd0f663b0 0x557cd0fafa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.189+0000 7f7be3c33740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0fa0280 0x557cd0f604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.189+0000 7f7be3c33740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x557cd0ec9af0 con 0x557cd0fa0280 2024-03-28T23:33:36.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.189+0000 7f7be3c33740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x557cd0e729d0 con 0x557cd0faf660 2024-03-28T23:33:36.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.189+0000 7f7be3c33740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x557cd0ec5b90 con 0x557cd0f663b0 2024-03-28T23:33:36.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.189+0000 7f7bdad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0fa0280 0x557cd0f604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.189+0000 7f7bdad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0fa0280 0x557cd0f604e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54284/0 (socket says 172.21.15.161:54284) 2024-03-28T23:33:36.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.189+0000 7f7bdad9d640 1 -- 172.21.15.161:0/489127981 learned_addr learned my addr 172.21.15.161:0/489127981 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:36.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.189+0000 7f7bd9d9b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557cd0f663b0 0x557cd0fafa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.189+0000 7f7bda59c640 1 --2- 172.21.15.161:0/489127981 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cd0faf660 0x557cd0f65e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.189+0000 7f7bdad9d640 1 -- 172.21.15.161:0/489127981 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557cd0f663b0 msgr2=0x557cd0fafa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:36.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.189+0000 7f7bdad9d640 1 --2- 172.21.15.161:0/489127981 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557cd0f663b0 0x557cd0fafa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.189+0000 7f7bdad9d640 1 -- 172.21.15.161:0/489127981 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cd0faf660 msgr2=0x557cd0f65e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:36.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.189+0000 7f7bdad9d640 1 --2- 172.21.15.161:0/489127981 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cd0faf660 0x557cd0f65e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.189+0000 7f7bdad9d640 1 -- 172.21.15.161:0/489127981 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cd0bde2d0 con 0x557cd0fa0280 2024-03-28T23:33:36.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.189+0000 7f7bdad9d640 1 --2- 172.21.15.161:0/489127981 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0fa0280 0x557cd0f604e0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f7bd40608f0 tx=0x7f7bd405c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b41d335781b963bc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:36.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.189+0000 7f7bbbfff640 1 -- 172.21.15.161:0/489127981 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4021040 con 0x557cd0fa0280 2024-03-28T23:33:36.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.189+0000 7f7bbbfff640 1 -- 172.21.15.161:0/489127981 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bd40701c0 con 0x557cd0fa0280 2024-03-28T23:33:36.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.189+0000 7f7bbbfff640 1 -- 172.21.15.161:0/489127981 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd400ccc0 con 0x557cd0fa0280 2024-03-28T23:33:36.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.190+0000 7f7be3c33740 1 -- 172.21.15.161:0/489127981 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0fa0280 msgr2=0x557cd0f604e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:36.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.190+0000 7f7be3c33740 1 --2- 172.21.15.161:0/489127981 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0fa0280 0x557cd0f604e0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f7bd40608f0 tx=0x7f7bd405c920 comp rx=0 tx=0).stop 2024-03-28T23:33:36.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.190+0000 7f7be3c33740 1 -- 172.21.15.161:0/489127981 shutdown_connections 2024-03-28T23:33:36.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.190+0000 7f7be3c33740 1 --2- 172.21.15.161:0/489127981 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0fa0280 0x557cd0f604e0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.190+0000 7f7be3c33740 1 --2- 172.21.15.161:0/489127981 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557cd0f663b0 0x557cd0fafa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.190+0000 7f7be3c33740 1 --2- 172.21.15.161:0/489127981 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cd0faf660 0x557cd0f65e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.190+0000 7f7be3c33740 1 -- 172.21.15.161:0/489127981 >> 172.21.15.161:0/489127981 conn(0x557cd0f62630 msgr2=0x557cd0f87220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:36.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.190+0000 7f7be3c33740 1 -- 172.21.15.161:0/489127981 shutdown_connections 2024-03-28T23:33:36.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.190+0000 7f7be3c33740 1 -- 172.21.15.161:0/489127981 wait complete. 2024-03-28T23:33:36.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.191+0000 7f7be3c33740 1 -- start start 2024-03-28T23:33:36.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.191+0000 7f7be3c33740 1 Processor -- start 2024-03-28T23:33:36.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.192+0000 7f7be3c33740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557cd0fa0280 0x557cd0fa1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.192+0000 7f7be3c33740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cd0faf660 0x557cd0fb8040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.192+0000 7f7be3c33740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0f83d30 0x557cd0fb9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.192+0000 7f7be3c33740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x557cd0fa43f0 con 0x557cd0f83d30 2024-03-28T23:33:36.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.192+0000 7f7be3c33740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x557cd0e85f70 con 0x557cd0faf660 2024-03-28T23:33:36.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.192+0000 7f7be3c33740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x557cd0f87c10 con 0x557cd0fa0280 2024-03-28T23:33:36.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.192+0000 7f7be3c33740 10 client.0 _ll_register_callbacks cb 0x557cd0bddac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:33:36.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.192+0000 7f7bdad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0f83d30 0x557cd0fb9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.192+0000 7f7bdad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0f83d30 0x557cd0fb9d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54300/0 (socket says 172.21.15.161:54300) 2024-03-28T23:33:36.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.192+0000 7f7bdad9d640 1 -- 172.21.15.161:0/2934434508 learned_addr learned my addr 172.21.15.161:0/2934434508 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:36.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.192+0000 7f7bd9d9b640 1 --2- 172.21.15.161:0/2934434508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cd0faf660 0x557cd0fb8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.192+0000 7f7bda59c640 1 --2- 172.21.15.161:0/2934434508 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557cd0fa0280 0x557cd0fa1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.192+0000 7f7bdad9d640 1 -- 172.21.15.161:0/2934434508 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557cd0fa0280 msgr2=0x557cd0fa1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:36.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.192+0000 7f7bdad9d640 1 --2- 172.21.15.161:0/2934434508 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557cd0fa0280 0x557cd0fa1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.192+0000 7f7bdad9d640 1 -- 172.21.15.161:0/2934434508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cd0faf660 msgr2=0x557cd0fb8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:36.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.192+0000 7f7bdad9d640 1 --2- 172.21.15.161:0/2934434508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cd0faf660 0x557cd0fb8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.192+0000 7f7bdad9d640 1 -- 172.21.15.161:0/2934434508 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cd0ca86b0 con 0x557cd0f83d30 2024-03-28T23:33:36.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.192+0000 7f7bd9d9b640 1 --2- 172.21.15.161:0/2934434508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cd0faf660 0x557cd0fb8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:36.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.192+0000 7f7bdad9d640 1 --2- 172.21.15.161:0/2934434508 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0f83d30 0x557cd0fb9d60 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4058980 tx=0x7f7bd40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=722112ee6a4141c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:36.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.192+0000 7f7bb97fa640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:33:36.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.192+0000 7f7bb97fa640 1 -- 172.21.15.161:0/2934434508 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4021040 con 0x557cd0f83d30 2024-03-28T23:33:36.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.192+0000 7f7bb97fa640 1 -- 172.21.15.161:0/2934434508 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 6245+0+0 (secure 0 0 0) 0x7f7bd4071d70 con 0x557cd0f83d30 2024-03-28T23:33:36.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.192+0000 7f7be3c33740 1 -- 172.21.15.161:0/2934434508 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bd40701c0 con 0x557cd0f83d30 2024-03-28T23:33:36.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.192+0000 7f7bb97fa640 1 -- 172.21.15.161:0/2934434508 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7f7bd4057c10 con 0x557cd0f83d30 2024-03-28T23:33:36.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.193+0000 7f7be3c33740 10 client.7299 fetch_fsmap learned FSMap version 84 2024-03-28T23:33:36.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.193+0000 7f7be3c33740 1 -- 172.21.15.161:0/2934434508 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=84,monmap=2+}) -- 0x557cd0fadba0 con 0x557cd0f83d30 2024-03-28T23:33:36.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.193+0000 7f7bb97fa640 1 -- 172.21.15.161:0/2934434508 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bd40701c0 con 0x557cd0f83d30 2024-03-28T23:33:36.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.193+0000 7f7bb97fa640 1 -- 172.21.15.161:0/2934434508 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 84) ==== 3494+0+0 (secure 0 0 0) 0x7f7bd4024060 con 0x557cd0f83d30 2024-03-28T23:33:36.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.193+0000 7f7be3c33740 10 client.7299 fetch_fsmap finished waiting for FSMap version 84 2024-03-28T23:33:36.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.193+0000 7f7be3c33740 10 client.7299 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:33:36.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.193+0000 7f7be3c33740 20 client.7299 populate_metadata read hostname 'smithi161' 2024-03-28T23:33:36.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.193+0000 7f7be3c33740 1 --2- 172.21.15.161:0/2934434508 >> [v2:172.21.15.170:6833/3079114491,v1:172.21.15.170:6835/3079114491] conn(0x557cd0fba780 0x557cd0f82c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.193+0000 7f7be3c33740 4 client.7299 mds_command: new command op to 5963 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:33:36.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.193+0000 7f7be3c33740 1 -- 172.21.15.161:0/2934434508 --> [v2:172.21.15.170:6833/3079114491,v1:172.21.15.170:6835/3079114491] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557cd0ed7030 con 0x557cd0fba780 2024-03-28T23:33:36.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.193+0000 7f7bda59c640 1 --2- 172.21.15.161:0/2934434508 >> [v2:172.21.15.170:6833/3079114491,v1:172.21.15.170:6835/3079114491] conn(0x557cd0fba780 0x557cd0f82c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.193+0000 7f7bb97fa640 10 client.7299 ms_handle_connect on v2:172.21.15.170:6833/3079114491 2024-03-28T23:33:36.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.193+0000 7f7bda59c640 1 --2- 172.21.15.161:0/2934434508 >> [v2:172.21.15.170:6833/3079114491,v1:172.21.15.170:6835/3079114491] conn(0x557cd0fba780 0x557cd0f82c00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea59be5eaa5f2556 server_cookie=5365734c8554d5c1 in_seq=0 out_seq=0 2024-03-28T23:33:36.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.194+0000 7f7bb97fa640 1 -- 172.21.15.161:0/2934434508 <== mds.0 v2:172.21.15.170:6833/3079114491 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557cd0ed7030 con 0x557cd0fba780 2024-03-28T23:33:36.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.194+0000 7f7bb97fa640 10 client.7299 handle_command_reply: tid=0 2024-03-28T23:33:36.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.194+0000 7f7be3c33740 1 client.7299 shutdown 2024-03-28T23:33:36.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.194+0000 7f7be3c33740 1 -- 172.21.15.161:0/2934434508 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0f83d30 msgr2=0x557cd0fb9d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:36.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.194+0000 7f7be3c33740 1 --2- 172.21.15.161:0/2934434508 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0f83d30 0x557cd0fb9d60 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4058980 tx=0x7f7bd40046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.194+0000 7f7be3c33740 1 -- 172.21.15.161:0/2934434508 shutdown_connections 2024-03-28T23:33:36.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.194+0000 7f7be3c33740 1 --2- 172.21.15.161:0/2934434508 >> [v2:172.21.15.170:6833/3079114491,v1:172.21.15.170:6835/3079114491] conn(0x557cd0fba780 0x557cd0f82c00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.194+0000 7f7be3c33740 1 --2- 172.21.15.161:0/2934434508 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0f83d30 0x557cd0fb9d60 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.194+0000 7f7be3c33740 1 --2- 172.21.15.161:0/2934434508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cd0faf660 0x557cd0fb8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.194+0000 7f7be3c33740 1 --2- 172.21.15.161:0/2934434508 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557cd0fa0280 0x557cd0fa1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.194+0000 7f7bb97fa640 0 client.7299 ms_handle_reset on v2:172.21.15.170:6833/3079114491 2024-03-28T23:33:36.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.194+0000 7f7be3c33740 1 -- 172.21.15.161:0/2934434508 >> 172.21.15.161:0/2934434508 conn(0x557cd0f663b0 msgr2=0x557cd0facac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:36.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.194+0000 7f7be3c33740 1 -- 172.21.15.161:0/2934434508 shutdown_connections 2024-03-28T23:33:36.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.194+0000 7f7be3c33740 1 -- 172.21.15.161:0/2934434508 wait complete. 2024-03-28T23:33:36.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.194+0000 7f7be3c33740 20 client.7299 trim_cache size 0 max 16384 2024-03-28T23:33:36.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.212+0000 7f7be3c33740 1 Processor -- start 2024-03-28T23:33:36.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.212+0000 7f7be3c33740 1 -- start start 2024-03-28T23:33:36.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.212+0000 7f7be3c33740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557cd0f83d30 0x557cd0faea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.212+0000 7f7be3c33740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0fa0280 0x557cd0fa8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.212+0000 7f7be3c33740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cd0faf660 0x557cd0fa8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.212+0000 7f7be3c33740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7bdc0c5f20 con 0x557cd0fa0280 2024-03-28T23:33:36.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.212+0000 7f7be3c33740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7bdc151e60 con 0x557cd0faf660 2024-03-28T23:33:36.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.212+0000 7f7be3c33740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7bdc0c58d0 con 0x557cd0f83d30 2024-03-28T23:33:36.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.212+0000 7f7bd9d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0fa0280 0x557cd0fa8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.213+0000 7f7bd9d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0fa0280 0x557cd0fa8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54306/0 (socket says 172.21.15.161:54306) 2024-03-28T23:33:36.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.213+0000 7f7bd9d9b640 1 -- 172.21.15.161:0/690438740 learned_addr learned my addr 172.21.15.161:0/690438740 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:36.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.213+0000 7f7bda59c640 1 --2- 172.21.15.161:0/690438740 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557cd0f83d30 0x557cd0faea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.213+0000 7f7bdad9d640 1 --2- 172.21.15.161:0/690438740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cd0faf660 0x557cd0fa8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.213+0000 7f7bd9d9b640 1 -- 172.21.15.161:0/690438740 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557cd0f83d30 msgr2=0x557cd0faea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:36.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.213+0000 7f7bd9d9b640 1 --2- 172.21.15.161:0/690438740 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557cd0f83d30 0x557cd0faea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.213+0000 7f7bd9d9b640 1 -- 172.21.15.161:0/690438740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cd0faf660 msgr2=0x557cd0fa8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:36.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.213+0000 7f7bd9d9b640 1 --2- 172.21.15.161:0/690438740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cd0faf660 0x557cd0fa8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.213+0000 7f7bd9d9b640 1 -- 172.21.15.161:0/690438740 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cd0f61800 con 0x557cd0fa0280 2024-03-28T23:33:36.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.213+0000 7f7bd9d9b640 1 --2- 172.21.15.161:0/690438740 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0fa0280 0x557cd0fa8850 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f7bd00090b0 tx=0x7f7bd0030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ee7e9dc7bb6c6c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:36.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.213+0000 7f7bbb7fe640 1 -- 172.21.15.161:0/690438740 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd0031d60 con 0x557cd0fa0280 2024-03-28T23:33:36.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.213+0000 7f7bbb7fe640 1 -- 172.21.15.161:0/690438740 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bd0037040 con 0x557cd0fa0280 2024-03-28T23:33:36.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.213+0000 7f7bbb7fe640 1 -- 172.21.15.161:0/690438740 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd0032690 con 0x557cd0fa0280 2024-03-28T23:33:36.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.214+0000 7f7be3c33740 1 -- 172.21.15.161:0/690438740 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0fa0280 msgr2=0x557cd0fa8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:36.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.214+0000 7f7be3c33740 1 --2- 172.21.15.161:0/690438740 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0fa0280 0x557cd0fa8850 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f7bd00090b0 tx=0x7f7bd0030000 comp rx=0 tx=0).stop 2024-03-28T23:33:36.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.214+0000 7f7be3c33740 1 -- 172.21.15.161:0/690438740 shutdown_connections 2024-03-28T23:33:36.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.214+0000 7f7be3c33740 1 --2- 172.21.15.161:0/690438740 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0fa0280 0x557cd0fa8850 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.214+0000 7f7be3c33740 1 --2- 172.21.15.161:0/690438740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cd0faf660 0x557cd0fa8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.214+0000 7f7be3c33740 1 --2- 172.21.15.161:0/690438740 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557cd0f83d30 0x557cd0faea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.214+0000 7f7be3c33740 1 -- 172.21.15.161:0/690438740 >> 172.21.15.161:0/690438740 conn(0x557cd0f663b0 msgr2=0x557cd0fca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:36.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.214+0000 7f7be3c33740 1 -- 172.21.15.161:0/690438740 shutdown_connections 2024-03-28T23:33:36.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.214+0000 7f7be3c33740 1 -- 172.21.15.161:0/690438740 wait complete. 2024-03-28T23:33:36.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.214+0000 7f7be3c33740 1 -- start start 2024-03-28T23:33:36.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.214+0000 7f7be3c33740 1 Processor -- start 2024-03-28T23:33:36.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.215+0000 7f7be3c33740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557cd0f83d30 0x557cd0fa8c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.215+0000 7f7be3c33740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cd0fa0280 0x557cd0f86990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.215+0000 7f7be3c33740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0faf660 0x557cd10218c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.215+0000 7f7be3c33740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x557cd0f87c10 con 0x557cd0faf660 2024-03-28T23:33:36.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.215+0000 7f7be3c33740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x557cd0fba600 con 0x557cd0fa0280 2024-03-28T23:33:36.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.215+0000 7f7be3c33740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x557cd0fa43f0 con 0x557cd0f83d30 2024-03-28T23:33:36.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.215+0000 7f7be3c33740 10 client.0 _ll_register_callbacks cb 0x7f7bcc017040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:33:36.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.215+0000 7f7bdad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0faf660 0x557cd10218c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.215+0000 7f7bdad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0faf660 0x557cd10218c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54314/0 (socket says 172.21.15.161:54314) 2024-03-28T23:33:36.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.215+0000 7f7bdad9d640 1 -- 172.21.15.161:0/1403160787 learned_addr learned my addr 172.21.15.161:0/1403160787 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:36.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.215+0000 7f7bda59c640 1 --2- 172.21.15.161:0/1403160787 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557cd0f83d30 0x557cd0fa8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.215+0000 7f7bd9d9b640 1 --2- 172.21.15.161:0/1403160787 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cd0fa0280 0x557cd0f86990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.216+0000 7f7bdad9d640 1 -- 172.21.15.161:0/1403160787 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557cd0f83d30 msgr2=0x557cd0fa8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:36.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.216+0000 7f7bdad9d640 1 --2- 172.21.15.161:0/1403160787 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557cd0f83d30 0x557cd0fa8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.216+0000 7f7bdad9d640 1 -- 172.21.15.161:0/1403160787 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cd0fa0280 msgr2=0x557cd0f86990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:36.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.216+0000 7f7bdad9d640 1 --2- 172.21.15.161:0/1403160787 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cd0fa0280 0x557cd0f86990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.216+0000 7f7bdad9d640 1 -- 172.21.15.161:0/1403160787 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cd1028b70 con 0x557cd0faf660 2024-03-28T23:33:36.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.216+0000 7f7bdad9d640 1 --2- 172.21.15.161:0/1403160787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0faf660 0x557cd10218c0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f7bd40584c0 tx=0x7f7bd4058590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d45d26100d6a9af2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:36.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.216+0000 7f7bb97fa640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:33:36.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.216+0000 7f7bb97fa640 1 -- 172.21.15.161:0/1403160787 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4021070 con 0x557cd0faf660 2024-03-28T23:33:36.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.216+0000 7f7bb97fa640 1 -- 172.21.15.161:0/1403160787 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 6245+0+0 (secure 0 0 0) 0x7f7bd40771f0 con 0x557cd0faf660 2024-03-28T23:33:36.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.216+0000 7f7be3c33740 1 -- 172.21.15.161:0/1403160787 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bd0037040 con 0x557cd0faf660 2024-03-28T23:33:36.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.216+0000 7f7bb97fa640 1 -- 172.21.15.161:0/1403160787 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7f7bd4058300 con 0x557cd0faf660 2024-03-28T23:33:36.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.216+0000 7f7be3c33740 10 client.7305 fetch_fsmap learned FSMap version 84 2024-03-28T23:33:36.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.216+0000 7f7be3c33740 1 -- 172.21.15.161:0/1403160787 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=84,monmap=2+}) -- 0x557cd1028b70 con 0x557cd0faf660 2024-03-28T23:33:36.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.216+0000 7f7bb97fa640 1 -- 172.21.15.161:0/1403160787 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bd0037040 con 0x557cd0faf660 2024-03-28T23:33:36.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.216+0000 7f7bb97fa640 1 -- 172.21.15.161:0/1403160787 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 84) ==== 3494+0+0 (secure 0 0 0) 0x7f7bd4057db0 con 0x557cd0faf660 2024-03-28T23:33:36.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.216+0000 7f7be3c33740 10 client.7305 fetch_fsmap finished waiting for FSMap version 84 2024-03-28T23:33:36.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.216+0000 7f7be3c33740 10 client.7305 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:33:36.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.216+0000 7f7be3c33740 20 client.7305 populate_metadata read hostname 'smithi161' 2024-03-28T23:33:36.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.216+0000 7f7be3c33740 1 --2- 172.21.15.161:0/1403160787 >> [v2:172.21.15.170:6833/3079114491,v1:172.21.15.170:6835/3079114491] conn(0x557cd1022080 0x557cd0fc95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.216+0000 7f7be3c33740 4 client.7305 mds_command: new command op to 5963 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:33:36.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.216+0000 7f7be3c33740 1 -- 172.21.15.161:0/1403160787 --> [v2:172.21.15.170:6833/3079114491,v1:172.21.15.170:6835/3079114491] -- command(tid 0: {"prefix": "status"}) -- 0x557cd0f66ba0 con 0x557cd1022080 2024-03-28T23:33:36.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.216+0000 7f7bda59c640 1 --2- 172.21.15.161:0/1403160787 >> [v2:172.21.15.170:6833/3079114491,v1:172.21.15.170:6835/3079114491] conn(0x557cd1022080 0x557cd0fc95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.217+0000 7f7bb97fa640 10 client.7305 ms_handle_connect on v2:172.21.15.170:6833/3079114491 2024-03-28T23:33:36.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.217+0000 7f7bda59c640 1 --2- 172.21.15.161:0/1403160787 >> [v2:172.21.15.170:6833/3079114491,v1:172.21.15.170:6835/3079114491] conn(0x557cd1022080 0x557cd0fc95d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8cc250a01fa3159 server_cookie=7698bdaaf1371653 in_seq=0 out_seq=0 2024-03-28T23:33:36.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.217+0000 7f7bb97fa640 1 -- 172.21.15.161:0/1403160787 <== mds.0 v2:172.21.15.170:6833/3079114491 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x557cd0f66ba0 con 0x557cd1022080 2024-03-28T23:33:36.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.217+0000 7f7bb97fa640 10 client.7305 handle_command_reply: tid=0 2024-03-28T23:33:36.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.217+0000 7f7be3c33740 1 client.7305 shutdown 2024-03-28T23:33:36.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.217+0000 7f7be3c33740 1 -- 172.21.15.161:0/1403160787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0faf660 msgr2=0x557cd10218c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:36.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.217+0000 7f7be3c33740 1 --2- 172.21.15.161:0/1403160787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0faf660 0x557cd10218c0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f7bd40584c0 tx=0x7f7bd4058590 comp rx=0 tx=0).stop 2024-03-28T23:33:36.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.217+0000 7f7be3c33740 1 -- 172.21.15.161:0/1403160787 shutdown_connections 2024-03-28T23:33:36.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.217+0000 7f7be3c33740 1 --2- 172.21.15.161:0/1403160787 >> [v2:172.21.15.170:6833/3079114491,v1:172.21.15.170:6835/3079114491] conn(0x557cd1022080 0x557cd0fc95d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.217+0000 7f7be3c33740 1 --2- 172.21.15.161:0/1403160787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557cd0faf660 0x557cd10218c0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.217+0000 7f7be3c33740 1 --2- 172.21.15.161:0/1403160787 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557cd0fa0280 0x557cd0f86990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.217+0000 7f7bb97fa640 0 client.7305 ms_handle_reset on v2:172.21.15.170:6833/3079114491 2024-03-28T23:33:36.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.217+0000 7f7be3c33740 1 --2- 172.21.15.161:0/1403160787 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557cd0f83d30 0x557cd0fa8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.217+0000 7f7be3c33740 1 -- 172.21.15.161:0/1403160787 >> 172.21.15.161:0/1403160787 conn(0x557cd0f663b0 msgr2=0x557cd0fae980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:36.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.217+0000 7f7be3c33740 1 -- 172.21.15.161:0/1403160787 shutdown_connections 2024-03-28T23:33:36.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.217+0000 7f7be3c33740 1 -- 172.21.15.161:0/1403160787 wait complete. 2024-03-28T23:33:36.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.217+0000 7f7be3c33740 20 client.7305 trim_cache size 0 max 16384 2024-03-28T23:33:36.224 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:33:36.224 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:33:36.224 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:33:36.224 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 5963, 2024-03-28T23:33:36.224 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:33:36.224 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:33:36.224 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:33:36.225 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 4.7933790549999999, 2024-03-28T23:33:36.225 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 84, 2024-03-28T23:33:36.225 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 97, 2024-03-28T23:33:36.225 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 95, 2024-03-28T23:33:36.225 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 17.137428294999999 2024-03-28T23:33:36.225 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:33:36.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.218+0000 7f7be0f80640 1 -- 172.21.15.161:0/518309158 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7bbc06b4d0 msgr2=0x7f7bbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:36.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.218+0000 7f7be0f80640 1 --2- 172.21.15.161:0/518309158 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7bbc06b4d0 0x7f7bbc06d990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc150dd0 tx=0x7f7bcc009040 comp rx=0 tx=0).stop 2024-03-28T23:33:36.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.218+0000 7f7be0f80640 1 -- 172.21.15.161:0/518309158 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bdc14be70 msgr2=0x7f7bdc1ec840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:36.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.218+0000 7f7be0f80640 1 --2- 172.21.15.161:0/518309158 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bdc14be70 0x7f7bdc1ec840 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f7bd400e7f0 tx=0x7f7bd400ebf0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.218+0000 7f7be0f80640 1 -- 172.21.15.161:0/518309158 shutdown_connections 2024-03-28T23:33:36.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.218+0000 7f7be0f80640 1 --2- 172.21.15.161:0/518309158 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7bbc06b4d0 0x7f7bbc06d990 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.218+0000 7f7be0f80640 1 --2- 172.21.15.161:0/518309158 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bdc14be70 0x7f7bdc1ec840 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.218+0000 7f7be0f80640 1 --2- 172.21.15.161:0/518309158 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bdc0a9680 0x7f7bdc0c1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.218+0000 7f7be0f80640 1 --2- 172.21.15.161:0/518309158 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7bdc06ce80 0x7f7bdc1477c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.218+0000 7f7be0f80640 1 -- 172.21.15.161:0/518309158 >> 172.21.15.161:0/518309158 conn(0x7f7bdc0c6670 msgr2=0x7f7bdc0ab1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:36.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.218+0000 7f7be0f80640 1 -- 172.21.15.161:0/518309158 shutdown_connections 2024-03-28T23:33:36.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.218+0000 7f7be0f80640 1 -- 172.21.15.161:0/518309158 wait complete. 2024-03-28T23:33:36.229 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 7, "epoch": 84, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 7, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:33:23.516182+0000", "damaged": [], "data_pools": [ 16 ], "enabled": true, "epoch": 84, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5963": { "addr": "172.21.15.170:6835/3079114491", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 3079114491, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 3079114491, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5963, "incarnation": 81, "join_fscid": 7, "name": "b", "rank": 0, "state": "up:active", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 95, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 15, "modified": "2024-03-28T23:33:34.430703+0000", "qdb_cluster": [ 5963 ], "qdb_leader": 5963, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5963 } } } ], "standbys": [ { "addr": "172.21.15.161:6836/4280217233", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 4280217233, "type": "v2" }, { "addr": "172.21.15.161:6836", "nonce": 4280217233, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 76, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6547, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.170:6834/2647418289", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 2647418289, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 2647418289, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 82, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6706, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6837/404132405", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6835", "nonce": 404132405, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 404132405, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 76, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7086, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:33:36.230 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 7, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-28T23:33:36.230 INFO:tasks.cephfs.test_failover:current = [{'gid': 7086, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6837/404132405', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 404132405}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 404132405}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 76}, {'gid': 5963, 'name': 'b', 'rank': 0, 'incarnation': 81, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.170:6835/3079114491', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 3079114491}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 3079114491}]}, 'join_fscid': 7, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 6547, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6836/4280217233', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 4280217233}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 4280217233}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 76}, {'gid': 6706, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6834/2647418289', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 2647418289}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 2647418289}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 82}] 2024-03-28T23:33:36.231 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-28T23:33:36.231 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-28T23:33:36.231 DEBUG:teuthology.orchestra.run.smithi161:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:33:36.271 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:33:36.316 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-28T23:33:36.316 INFO:tasks.cephfs.mount:Terminating background process 2024-03-28T23:33:36.316 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:33:36.316 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:33:36.332 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20793 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-28T23:33:36.332 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi161.front.sepia.ceph.com 2024-03-28T23:33:36.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:33:36.332 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-28T23:33:36.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:33:36.387 INFO:teuthology.orchestra.run.smithi161.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-28T23:33:36.387 INFO:tasks.cephfs.mount:Terminating background process 2024-03-28T23:33:36.387 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:33:36.388 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:33:36.403 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056374,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19690 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-28T23:33:36.403 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi170.front.sepia.ceph.com 2024-03-28T23:33:36.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:33:36.403 DEBUG:teuthology.orchestra.run.smithi170:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-28T23:33:36.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:33:36.458 INFO:teuthology.orchestra.run.smithi170.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-28T23:33:36.459 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:33:36.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.559+0000 7f6d60143640 1 Processor -- start 2024-03-28T23:33:36.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.559+0000 7f6d60143640 1 -- start start 2024-03-28T23:33:36.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.560+0000 7f6d60143640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6d580aa7e0 0x7f6d580c0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.560+0000 7f6d60143640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6d5806c400 0x7f6d580c1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.560+0000 7f6d60143640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d5815a390 con 0x7f6d5806b910 2024-03-28T23:33:36.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.560+0000 7f6d60143640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6d580a9f50 con 0x7f6d5806c400 2024-03-28T23:33:36.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.560+0000 7f6d60143640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f6d580a9990 con 0x7f6d580aa7e0 2024-03-28T23:33:36.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.560+0000 7f6d5dedf640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6d5806b910 0x7f6d5806bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52980/0 (socket says 172.21.15.161:52980) 2024-03-28T23:33:36.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.560+0000 7f6d5dedf640 1 -- 172.21.15.161:0/1112987592 learned_addr learned my addr 172.21.15.161:0/1112987592 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:36.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.560+0000 7f6d5e6e0640 1 --2- 172.21.15.161:0/1112987592 >> v2:172.21.15.161:3300/0 conn(0x7f6d5806c400 0x7f6d580c1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.560+0000 7f6d5d6de640 1 --2- 172.21.15.161:0/1112987592 >> v2:172.21.15.170:3300/0 conn(0x7f6d580aa7e0 0x7f6d580c0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.560+0000 7f6d5e6e0640 1 -- 172.21.15.161:0/1112987592 >> v2:172.21.15.170:3300/0 conn(0x7f6d580aa7e0 msgr2=0x7f6d580c0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:36.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.560+0000 7f6d5e6e0640 1 --2- 172.21.15.161:0/1112987592 >> v2:172.21.15.170:3300/0 conn(0x7f6d580aa7e0 0x7f6d580c0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.560+0000 7f6d5e6e0640 1 -- 172.21.15.161:0/1112987592 >> v1:172.21.15.161:6789/0 conn(0x7f6d5806b910 legacy=0x7f6d5806bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:36.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.560+0000 7f6d5e6e0640 1 -- 172.21.15.161:0/1112987592 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d580a9c70 con 0x7f6d5806c400 2024-03-28T23:33:36.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.561+0000 7f6d5e6e0640 1 --2- 172.21.15.161:0/1112987592 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d5806c400 0x7f6d580c1530 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f6d580c7660 tx=0x7f6d5002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86827f90a90226c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:36.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.561+0000 7f6d5cedd640 1 -- 172.21.15.161:0/1112987592 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d50039070 con 0x7f6d5806c400 2024-03-28T23:33:36.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.561+0000 7f6d5cedd640 1 -- 172.21.15.161:0/1112987592 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d5002cb60 con 0x7f6d5806c400 2024-03-28T23:33:36.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.561+0000 7f6d5cedd640 1 -- 172.21.15.161:0/1112987592 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d50034440 con 0x7f6d5806c400 2024-03-28T23:33:36.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.562+0000 7f6d60143640 1 -- 172.21.15.161:0/1112987592 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d5806c400 msgr2=0x7f6d580c1530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:36.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.562+0000 7f6d60143640 1 --2- 172.21.15.161:0/1112987592 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d5806c400 0x7f6d580c1530 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f6d580c7660 tx=0x7f6d5002bdb0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.562+0000 7f6d60143640 1 -- 172.21.15.161:0/1112987592 shutdown_connections 2024-03-28T23:33:36.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.562+0000 7f6d60143640 1 --2- 172.21.15.161:0/1112987592 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d5806c400 0x7f6d580c1530 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.562+0000 7f6d60143640 1 --2- 172.21.15.161:0/1112987592 >> v2:172.21.15.170:3300/0 conn(0x7f6d580aa7e0 0x7f6d580c0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.562+0000 7f6d60143640 1 -- 172.21.15.161:0/1112987592 >> 172.21.15.161:0/1112987592 conn(0x7f6d580c6670 msgr2=0x7f6d580c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:36.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.562+0000 7f6d60143640 1 -- 172.21.15.161:0/1112987592 shutdown_connections 2024-03-28T23:33:36.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.562+0000 7f6d60143640 1 -- 172.21.15.161:0/1112987592 wait complete. 2024-03-28T23:33:36.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.563+0000 7f6d60143640 1 Processor -- start 2024-03-28T23:33:36.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.563+0000 7f6d60143640 1 -- start start 2024-03-28T23:33:36.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.564+0000 7f6d60143640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d5806b910 0x7f6d58148d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.564+0000 7f6d60143640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d5806c400 0x7f6d580c4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.564+0000 7f6d60143640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d580aa7e0 0x7f6d581eee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.564+0000 7f6d60143640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6d58156f80 con 0x7f6d5806b910 2024-03-28T23:33:36.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.564+0000 7f6d60143640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6d58156e00 con 0x7f6d580aa7e0 2024-03-28T23:33:36.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.564+0000 7f6d60143640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6d5806cf50 con 0x7f6d5806c400 2024-03-28T23:33:36.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.564+0000 7f6d5dedf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d5806b910 0x7f6d58148d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.564+0000 7f6d5dedf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d5806b910 0x7f6d58148d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54346/0 (socket says 172.21.15.161:54346) 2024-03-28T23:33:36.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.564+0000 7f6d5dedf640 1 -- 172.21.15.161:0/1065540507 learned_addr learned my addr 172.21.15.161:0/1065540507 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:36.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.564+0000 7f6d5e6e0640 1 --2- 172.21.15.161:0/1065540507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d580aa7e0 0x7f6d581eee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.564+0000 7f6d5d6de640 1 --2- 172.21.15.161:0/1065540507 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d5806c400 0x7f6d580c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.564+0000 7f6d5dedf640 1 -- 172.21.15.161:0/1065540507 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d5806c400 msgr2=0x7f6d580c4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:36.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.564+0000 7f6d5dedf640 1 --2- 172.21.15.161:0/1065540507 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d5806c400 0x7f6d580c4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.564+0000 7f6d5dedf640 1 -- 172.21.15.161:0/1065540507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d580aa7e0 msgr2=0x7f6d581eee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:36.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.564+0000 7f6d5dedf640 1 --2- 172.21.15.161:0/1065540507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d580aa7e0 0x7f6d581eee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.564+0000 7f6d5dedf640 1 -- 172.21.15.161:0/1065540507 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d581f3350 con 0x7f6d5806b910 2024-03-28T23:33:36.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.564+0000 7f6d5dedf640 1 --2- 172.21.15.161:0/1065540507 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d5806b910 0x7f6d58148d40 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f6d5400c400 tx=0x7f6d5400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:36.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.564+0000 7f6d4e7fc640 1 -- 172.21.15.161:0/1065540507 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d54017070 con 0x7f6d5806b910 2024-03-28T23:33:36.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.564+0000 7f6d4e7fc640 1 -- 172.21.15.161:0/1065540507 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d54005ce0 con 0x7f6d5806b910 2024-03-28T23:33:36.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.564+0000 7f6d4e7fc640 1 -- 172.21.15.161:0/1065540507 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d540072f0 con 0x7f6d5806b910 2024-03-28T23:33:36.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.564+0000 7f6d60143640 1 -- 172.21.15.161:0/1065540507 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d581f2590 con 0x7f6d5806b910 2024-03-28T23:33:36.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.564+0000 7f6d60143640 1 -- 172.21.15.161:0/1065540507 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d581f2b50 con 0x7f6d5806b910 2024-03-28T23:33:36.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.565+0000 7f6d60143640 1 -- 172.21.15.161:0/1065540507 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d5804f4b0 con 0x7f6d5806b910 2024-03-28T23:33:36.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.566+0000 7f6d4e7fc640 1 -- 172.21.15.161:0/1065540507 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6d5401a070 con 0x7f6d5806b910 2024-03-28T23:33:36.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.566+0000 7f6d4e7fc640 1 --2- 172.21.15.161:0/1065540507 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6d3c06b4d0 0x7f6d3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.566+0000 7f6d5d6de640 1 --2- 172.21.15.161:0/1065540507 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6d3c06b4d0 0x7f6d3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.566+0000 7f6d4e7fc640 1 -- 172.21.15.161:0/1065540507 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6245+0+0 (secure 0 0 0) 0x7f6d54089690 con 0x7f6d5806b910 2024-03-28T23:33:36.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.567+0000 7f6d5d6de640 1 --2- 172.21.15.161:0/1065540507 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6d3c06b4d0 0x7f6d3c06d990 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f6d581521a0 tx=0x7f6d48009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:36.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.571+0000 7f6d4e7fc640 1 -- 172.21.15.161:0/1065540507 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6d54057d30 con 0x7f6d5806b910 2024-03-28T23:33:36.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.780+0000 7f6d60143640 1 -- 172.21.15.161:0/1065540507 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d5806b6f0 con 0x7f6d5806b910 2024-03-28T23:33:36.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.781+0000 7f6d4e7fc640 1 -- 172.21.15.161:0/1065540507 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) ==== 94+0+5192 (secure 0 0 0) 0x7f6d5405bdd0 con 0x7f6d5806b910 2024-03-28T23:33:36.781 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:36.781 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":84,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6547,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4280217233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4280217233},{"type":"v1","addr":"172.21.15.161:6836","nonce":4280217233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":6706,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/2647418289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":2647418289},{"type":"v1","addr":"172.21.15.170:6834","nonce":2647418289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":7086,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/404132405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":404132405},{"type":"v1","addr":"172.21.15.161:6837","nonce":404132405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:33:23.516182+0000","modified":"2024-03-28T23:33:34.430703+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5963},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5963":{"gid":5963,"name":"b","rank":0,"incarnation":81,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6835/3079114491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3079114491},{"type":"v1","addr":"172.21.15.170:6835","nonce":3079114491}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5963,"qdb_cluster":[5963]},"id":7}]} 2024-03-28T23:33:36.782 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 84 2024-03-28T23:33:36.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.783+0000 7f6d60143640 1 -- 172.21.15.161:0/1065540507 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6d3c06b4d0 msgr2=0x7f6d3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:36.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.783+0000 7f6d60143640 1 --2- 172.21.15.161:0/1065540507 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6d3c06b4d0 0x7f6d3c06d990 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f6d581521a0 tx=0x7f6d48009190 comp rx=0 tx=0).stop 2024-03-28T23:33:36.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.783+0000 7f6d60143640 1 -- 172.21.15.161:0/1065540507 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d5806b910 msgr2=0x7f6d58148d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:36.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.783+0000 7f6d60143640 1 --2- 172.21.15.161:0/1065540507 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d5806b910 0x7f6d58148d40 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f6d5400c400 tx=0x7f6d5400c8c0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.783+0000 7f6d60143640 1 -- 172.21.15.161:0/1065540507 shutdown_connections 2024-03-28T23:33:36.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.783+0000 7f6d60143640 1 --2- 172.21.15.161:0/1065540507 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6d3c06b4d0 0x7f6d3c06d990 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.783+0000 7f6d60143640 1 --2- 172.21.15.161:0/1065540507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d580aa7e0 0x7f6d581eee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.783+0000 7f6d60143640 1 --2- 172.21.15.161:0/1065540507 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d5806c400 0x7f6d580c4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.783+0000 7f6d60143640 1 --2- 172.21.15.161:0/1065540507 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d5806b910 0x7f6d58148d40 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.783+0000 7f6d60143640 1 -- 172.21.15.161:0/1065540507 >> 172.21.15.161:0/1065540507 conn(0x7f6d580c6670 msgr2=0x7f6d580c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:36.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.783+0000 7f6d60143640 1 -- 172.21.15.161:0/1065540507 shutdown_connections 2024-03-28T23:33:36.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.783+0000 7f6d60143640 1 -- 172.21.15.161:0/1065540507 wait complete. 2024-03-28T23:33:36.794 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:33:36.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.895+0000 7fe523e64640 1 Processor -- start 2024-03-28T23:33:36.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.895+0000 7fe523e64640 1 -- start start 2024-03-28T23:33:36.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.896+0000 7fe523e64640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe51c14be70 0x7fe51c0ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.896+0000 7fe523e64640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe51c0aacd0 0x7fe51c15c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.896+0000 7fe523e64640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe51c14c250 con 0x7fe51c0ab6e0 2024-03-28T23:33:36.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.896+0000 7fe523e64640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fe51c1529c0 con 0x7fe51c0aacd0 2024-03-28T23:33:36.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.896+0000 7fe523e64640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe51c0a9dc0 con 0x7fe51c14be70 2024-03-28T23:33:36.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.896+0000 7fe5213ff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe51c0ab6e0 0x7fe51c0aa5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52990/0 (socket says 172.21.15.161:52990) 2024-03-28T23:33:36.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.896+0000 7fe5213ff640 1 -- 172.21.15.161:0/3043487156 learned_addr learned my addr 172.21.15.161:0/3043487156 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:36.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.896+0000 7fe521c00640 1 --2- 172.21.15.161:0/3043487156 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe51c14be70 0x7fe51c0ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.896+0000 7fe522401640 1 --2- 172.21.15.161:0/3043487156 >> v2:172.21.15.170:3300/0 conn(0x7fe51c0aacd0 0x7fe51c15c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.896+0000 7fe520bfe640 1 -- 172.21.15.161:0/3043487156 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4003495575 0 0) 0x7fe51c14c250 con 0x7fe51c0ab6e0 2024-03-28T23:33:36.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.896+0000 7fe521c00640 1 -- 172.21.15.161:0/3043487156 >> v2:172.21.15.170:3300/0 conn(0x7fe51c0aacd0 msgr2=0x7fe51c15c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:36.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.896+0000 7fe521c00640 1 --2- 172.21.15.161:0/3043487156 >> v2:172.21.15.170:3300/0 conn(0x7fe51c0aacd0 0x7fe51c15c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.896+0000 7fe521c00640 1 -- 172.21.15.161:0/3043487156 >> v1:172.21.15.161:6789/0 conn(0x7fe51c0ab6e0 legacy=0x7fe51c0aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:36.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.896+0000 7fe521c00640 1 -- 172.21.15.161:0/3043487156 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51c0aa0a0 con 0x7fe51c14be70 2024-03-28T23:33:36.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.896+0000 7fe522401640 1 --2- 172.21.15.161:0/3043487156 >> v2:172.21.15.170:3300/0 conn(0x7fe51c0aacd0 0x7fe51c15c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:36.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.897+0000 7fe521c00640 1 --2- 172.21.15.161:0/3043487156 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe51c14be70 0x7fe51c0ab1a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe5180097a0 tx=0x7fe51802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a5f9208c75c975f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:36.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.897+0000 7fe520bfe640 1 -- 172.21.15.161:0/3043487156 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5180315a0 con 0x7fe51c14be70 2024-03-28T23:33:36.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.897+0000 7fe520bfe640 1 -- 172.21.15.161:0/3043487156 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe518037070 con 0x7fe51c14be70 2024-03-28T23:33:36.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.897+0000 7fe520bfe640 1 -- 172.21.15.161:0/3043487156 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe518031960 con 0x7fe51c14be70 2024-03-28T23:33:36.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.898+0000 7fe523e64640 1 -- 172.21.15.161:0/3043487156 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe51c14be70 msgr2=0x7fe51c0ab1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:36.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.898+0000 7fe523e64640 1 --2- 172.21.15.161:0/3043487156 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe51c14be70 0x7fe51c0ab1a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe5180097a0 tx=0x7fe51802b840 comp rx=0 tx=0).stop 2024-03-28T23:33:36.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.898+0000 7fe523e64640 1 -- 172.21.15.161:0/3043487156 shutdown_connections 2024-03-28T23:33:36.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.898+0000 7fe523e64640 1 --2- 172.21.15.161:0/3043487156 >> v2:172.21.15.170:3300/0 conn(0x7fe51c0aacd0 0x7fe51c15c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.898+0000 7fe523e64640 1 --2- 172.21.15.161:0/3043487156 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe51c14be70 0x7fe51c0ab1a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.898+0000 7fe523e64640 1 -- 172.21.15.161:0/3043487156 >> 172.21.15.161:0/3043487156 conn(0x7fe51c0c6670 msgr2=0x7fe51c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:36.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.898+0000 7fe523e64640 1 -- 172.21.15.161:0/3043487156 shutdown_connections 2024-03-28T23:33:36.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.898+0000 7fe523e64640 1 -- 172.21.15.161:0/3043487156 wait complete. 2024-03-28T23:33:36.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.899+0000 7fe523e64640 1 Processor -- start 2024-03-28T23:33:36.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.899+0000 7fe523e64640 1 -- start start 2024-03-28T23:33:36.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.900+0000 7fe523e64640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe51c0aacd0 0x7fe51c1e95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.900+0000 7fe523e64640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe51c0ab6e0 0x7fe51c0bf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.900+0000 7fe523e64640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe51c14be70 0x7fe51c0bf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.900+0000 7fe523e64640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c06b2d0 con 0x7fe51c14be70 2024-03-28T23:33:36.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.900+0000 7fe523e64640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c06b150 con 0x7fe51c0aacd0 2024-03-28T23:33:36.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.900+0000 7fe523e64640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe51c0c1520 con 0x7fe51c0ab6e0 2024-03-28T23:33:36.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.900+0000 7fe522401640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe51c14be70 0x7fe51c0bf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.900+0000 7fe522401640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe51c14be70 0x7fe51c0bf8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54356/0 (socket says 172.21.15.161:54356) 2024-03-28T23:33:36.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.900+0000 7fe522401640 1 -- 172.21.15.161:0/3424409020 learned_addr learned my addr 172.21.15.161:0/3424409020 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:36.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.900+0000 7fe5213ff640 1 --2- 172.21.15.161:0/3424409020 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe51c0ab6e0 0x7fe51c0bf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.900+0000 7fe521c00640 1 --2- 172.21.15.161:0/3424409020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe51c0aacd0 0x7fe51c1e95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.900+0000 7fe522401640 1 -- 172.21.15.161:0/3424409020 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe51c0ab6e0 msgr2=0x7fe51c0bf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:36.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.900+0000 7fe522401640 1 --2- 172.21.15.161:0/3424409020 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe51c0ab6e0 0x7fe51c0bf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.900+0000 7fe522401640 1 -- 172.21.15.161:0/3424409020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe51c0aacd0 msgr2=0x7fe51c1e95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:36.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.900+0000 7fe522401640 1 --2- 172.21.15.161:0/3424409020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe51c0aacd0 0x7fe51c1e95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:36.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.900+0000 7fe522401640 1 -- 172.21.15.161:0/3424409020 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51c0bfe30 con 0x7fe51c14be70 2024-03-28T23:33:36.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.900+0000 7fe521c00640 1 --2- 172.21.15.161:0/3424409020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe51c0aacd0 0x7fe51c1e95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:36.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.900+0000 7fe522401640 1 --2- 172.21.15.161:0/3424409020 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe51c14be70 0x7fe51c0bf8f0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fe514007e20 tx=0x7fe51400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:36.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.901+0000 7fe5127fc640 1 -- 172.21.15.161:0/3424409020 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51401a070 con 0x7fe51c14be70 2024-03-28T23:33:36.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.901+0000 7fe5127fc640 1 -- 172.21.15.161:0/3424409020 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe514003b90 con 0x7fe51c14be70 2024-03-28T23:33:36.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.901+0000 7fe523e64640 1 -- 172.21.15.161:0/3424409020 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe51c0bb690 con 0x7fe51c14be70 2024-03-28T23:33:36.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.901+0000 7fe5127fc640 1 -- 172.21.15.161:0/3424409020 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51400de80 con 0x7fe51c14be70 2024-03-28T23:33:36.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.901+0000 7fe523e64640 1 -- 172.21.15.161:0/3424409020 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe51c1debc0 con 0x7fe51c14be70 2024-03-28T23:33:36.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.902+0000 7fe523e64640 1 -- 172.21.15.161:0/3424409020 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe51c04f4b0 con 0x7fe51c14be70 2024-03-28T23:33:36.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.902+0000 7fe5127fc640 1 -- 172.21.15.161:0/3424409020 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe51401d070 con 0x7fe51c14be70 2024-03-28T23:33:36.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.902+0000 7fe5127fc640 1 --2- 172.21.15.161:0/3424409020 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe50006b4d0 0x7fe50006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:36.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.903+0000 7fe5127fc640 1 -- 172.21.15.161:0/3424409020 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6245+0+0 (secure 0 0 0) 0x7fe51408a560 con 0x7fe51c14be70 2024-03-28T23:33:36.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.903+0000 7fe521c00640 1 --2- 172.21.15.161:0/3424409020 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe50006b4d0 0x7fe50006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:36.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.903+0000 7fe521c00640 1 --2- 172.21.15.161:0/3424409020 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe50006b4d0 0x7fe50006d990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe518036000 tx=0x7fe51803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:36.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:36.907+0000 7fe5127fc640 1 -- 172.21.15.161:0/3424409020 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe514010070 con 0x7fe51c14be70 2024-03-28T23:33:37.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.079+0000 7fe523e64640 1 -- 172.21.15.161:0/3424409020 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe51c0c0ff0 con 0x7fe51c14be70 2024-03-28T23:33:37.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.080+0000 7fe5127fc640 1 -- 172.21.15.161:0/3424409020 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+17138 (secure 0 0 0) 0x7fe514058c00 con 0x7fe51c14be70 2024-03-28T23:33:37.081 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:37.081 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":97,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:33:35.435937+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:33:21.732337+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-28T23:33:22.477357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.1d","mappings":[{"from":1,"to":7}]},{"pgid":"16.25","mappings":[{"from":5,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6834/2176248563":"2024-03-29T23:33:31.422647+0000","172.21.15.170:6832/2176248563":"2024-03-29T23:33:31.422647+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:33:37.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.082+0000 7fe523e64640 1 -- 172.21.15.161:0/3424409020 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe50006b4d0 msgr2=0x7fe50006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:37.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.082+0000 7fe523e64640 1 --2- 172.21.15.161:0/3424409020 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe50006b4d0 0x7fe50006d990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe518036000 tx=0x7fe51803d040 comp rx=0 tx=0).stop 2024-03-28T23:33:37.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.082+0000 7fe523e64640 1 -- 172.21.15.161:0/3424409020 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe51c14be70 msgr2=0x7fe51c0bf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:37.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.082+0000 7fe523e64640 1 --2- 172.21.15.161:0/3424409020 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe51c14be70 0x7fe51c0bf8f0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fe514007e20 tx=0x7fe51400d4a0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.082+0000 7fe523e64640 1 -- 172.21.15.161:0/3424409020 shutdown_connections 2024-03-28T23:33:37.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.082+0000 7fe523e64640 1 --2- 172.21.15.161:0/3424409020 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe50006b4d0 0x7fe50006d990 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.082+0000 7fe523e64640 1 --2- 172.21.15.161:0/3424409020 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe51c14be70 0x7fe51c0bf8f0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.082+0000 7fe523e64640 1 --2- 172.21.15.161:0/3424409020 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe51c0ab6e0 0x7fe51c0bf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.082+0000 7fe523e64640 1 --2- 172.21.15.161:0/3424409020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe51c0aacd0 0x7fe51c1e95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.082+0000 7fe523e64640 1 -- 172.21.15.161:0/3424409020 >> 172.21.15.161:0/3424409020 conn(0x7fe51c0c6670 msgr2=0x7fe51c0c2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:37.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.082+0000 7fe523e64640 1 -- 172.21.15.161:0/3424409020 shutdown_connections 2024-03-28T23:33:37.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.082+0000 7fe523e64640 1 -- 172.21.15.161:0/3424409020 wait complete. 2024-03-28T23:33:37.094 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:33:21.732337+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:33:22.477357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:33:37.094 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:33:37.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.197+0000 7f12be23b640 1 Processor -- start 2024-03-28T23:33:37.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.198+0000 7f12be23b640 1 -- start start 2024-03-28T23:33:37.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.198+0000 7f12be23b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b80c6750 0x7f12b806bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:37.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.198+0000 7f12be23b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f12b80c88b0 0x7f12b80c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:37.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.198+0000 7f12be23b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12b8152db0 con 0x7f12b81529d0 2024-03-28T23:33:37.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.198+0000 7f12be23b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f12b80c8d60 con 0x7f12b80c88b0 2024-03-28T23:33:37.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.198+0000 7f12be23b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f12b80c7210 con 0x7f12b80c6750 2024-03-28T23:33:37.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.198+0000 7f12b77fe640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f12b81529d0 0x7f12b8155c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:58062/0 (socket says 172.21.15.161:58062) 2024-03-28T23:33:37.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.198+0000 7f12b77fe640 1 -- 172.21.15.161:0/3577209635 learned_addr learned my addr 172.21.15.161:0/3577209635 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:37.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.198+0000 7f12b7fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f12b80c88b0 0x7f12b80c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:37.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.198+0000 7f12aeffd640 1 --2- 172.21.15.161:0/3577209635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b80c6750 0x7f12b806bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:37.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.199+0000 7f12b6ffd640 1 -- 172.21.15.161:0/3577209635 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2308397053 0 0) 0x7f12b8152db0 con 0x7f12b81529d0 2024-03-28T23:33:37.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.199+0000 7f12b6ffd640 1 -- 172.21.15.161:0/3577209635 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12a4003520 con 0x7f12b81529d0 2024-03-28T23:33:37.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.199+0000 7f12aeffd640 1 -- 172.21.15.161:0/3577209635 >> v2:172.21.15.170:3300/0 conn(0x7f12b80c88b0 msgr2=0x7f12b80c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:37.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.199+0000 7f12aeffd640 1 --2- 172.21.15.161:0/3577209635 >> v2:172.21.15.170:3300/0 conn(0x7f12b80c88b0 0x7f12b80c54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.199+0000 7f12aeffd640 1 -- 172.21.15.161:0/3577209635 >> v1:172.21.15.170:6789/0 conn(0x7f12b81529d0 legacy=0x7f12b8155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:37.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.199+0000 7f12aeffd640 1 -- 172.21.15.161:0/3577209635 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b80c6c50 con 0x7f12b80c6750 2024-03-28T23:33:37.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.199+0000 7f12b7fff640 1 --2- 172.21.15.161:0/3577209635 >> v2:172.21.15.170:3300/0 conn(0x7f12b80c88b0 0x7f12b80c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:37.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.200+0000 7f12aeffd640 1 --2- 172.21.15.161:0/3577209635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b80c6750 0x7f12b806bb60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f12a00097a0 tx=0x7f12a002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3ac0d579a92fafe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:37.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.200+0000 7f12b6ffd640 1 -- 172.21.15.161:0/3577209635 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a0035020 con 0x7f12b80c6750 2024-03-28T23:33:37.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.200+0000 7f12b6ffd640 1 -- 172.21.15.161:0/3577209635 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12a0031830 con 0x7f12b80c6750 2024-03-28T23:33:37.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.200+0000 7f12b6ffd640 1 -- 172.21.15.161:0/3577209635 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a0031be0 con 0x7f12b80c6750 2024-03-28T23:33:37.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.200+0000 7f12be23b640 1 -- 172.21.15.161:0/3577209635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b80c6750 msgr2=0x7f12b806bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:37.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.200+0000 7f12be23b640 1 --2- 172.21.15.161:0/3577209635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b80c6750 0x7f12b806bb60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f12a00097a0 tx=0x7f12a002b8d0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.200+0000 7f12be23b640 1 -- 172.21.15.161:0/3577209635 shutdown_connections 2024-03-28T23:33:37.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.200+0000 7f12be23b640 1 --2- 172.21.15.161:0/3577209635 >> v2:172.21.15.170:3300/0 conn(0x7f12b80c88b0 0x7f12b80c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.200+0000 7f12be23b640 1 --2- 172.21.15.161:0/3577209635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b80c6750 0x7f12b806bb60 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.200+0000 7f12be23b640 1 -- 172.21.15.161:0/3577209635 >> 172.21.15.161:0/3577209635 conn(0x7f12b80c7430 msgr2=0x7f12b80c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:37.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.200+0000 7f12be23b640 1 -- 172.21.15.161:0/3577209635 shutdown_connections 2024-03-28T23:33:37.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.200+0000 7f12be23b640 1 -- 172.21.15.161:0/3577209635 wait complete. 2024-03-28T23:33:37.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.201+0000 7f12be23b640 1 Processor -- start 2024-03-28T23:33:37.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.202+0000 7f12be23b640 1 -- start start 2024-03-28T23:33:37.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.202+0000 7f12be23b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b80c6750 0x7f12b80c2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:37.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.202+0000 7f12be23b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12b80c88b0 0x7f12b81543b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:37.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.202+0000 7f12be23b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12b81529d0 0x7f12b81548f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:37.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.202+0000 7f12be23b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f12b8155710 con 0x7f12b80c88b0 2024-03-28T23:33:37.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.202+0000 7f12be23b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f12b8155430 con 0x7f12b81529d0 2024-03-28T23:33:37.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.202+0000 7f12be23b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f12b80aa160 con 0x7f12b80c6750 2024-03-28T23:33:37.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.202+0000 7f12aeffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12b80c88b0 0x7f12b81543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:37.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.202+0000 7f12b77fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b80c6750 0x7f12b80c2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:37.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.202+0000 7f12aeffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12b80c88b0 0x7f12b81543b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54360/0 (socket says 172.21.15.161:54360) 2024-03-28T23:33:37.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.202+0000 7f12aeffd640 1 -- 172.21.15.161:0/1602212531 learned_addr learned my addr 172.21.15.161:0/1602212531 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:37.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.202+0000 7f12b77fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b80c6750 0x7f12b80c2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:41326/0 (socket says 172.21.15.161:41326) 2024-03-28T23:33:37.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.202+0000 7f12b7fff640 1 --2- 172.21.15.161:0/1602212531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12b81529d0 0x7f12b81548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:37.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.203+0000 7f12aeffd640 1 -- 172.21.15.161:0/1602212531 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b80c6750 msgr2=0x7f12b80c2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:37.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.203+0000 7f12aeffd640 1 --2- 172.21.15.161:0/1602212531 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b80c6750 0x7f12b80c2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.203+0000 7f12aeffd640 1 -- 172.21.15.161:0/1602212531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12b81529d0 msgr2=0x7f12b81548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:37.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.203+0000 7f12aeffd640 1 --2- 172.21.15.161:0/1602212531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12b81529d0 0x7f12b81548f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.203+0000 7f12aeffd640 1 -- 172.21.15.161:0/1602212531 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b8153cd0 con 0x7f12b80c88b0 2024-03-28T23:33:37.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.203+0000 7f12aeffd640 1 --2- 172.21.15.161:0/1602212531 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12b80c88b0 0x7f12b81543b0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f12a0034040 tx=0x7f12a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:37.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.203+0000 7f12b4ff9640 1 -- 172.21.15.161:0/1602212531 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a0040070 con 0x7f12b80c88b0 2024-03-28T23:33:37.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.203+0000 7f12b4ff9640 1 -- 172.21.15.161:0/1602212531 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12a003e070 con 0x7f12b80c88b0 2024-03-28T23:33:37.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.203+0000 7f12be23b640 1 -- 172.21.15.161:0/1602212531 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12b81539f0 con 0x7f12b80c88b0 2024-03-28T23:33:37.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.203+0000 7f12b4ff9640 1 -- 172.21.15.161:0/1602212531 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a0035040 con 0x7f12b80c88b0 2024-03-28T23:33:37.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.203+0000 7f12be23b640 1 -- 172.21.15.161:0/1602212531 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12b8147380 con 0x7f12b80c88b0 2024-03-28T23:33:37.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.204+0000 7f12be23b640 1 -- 172.21.15.161:0/1602212531 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12b804f4b0 con 0x7f12b80c88b0 2024-03-28T23:33:37.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.205+0000 7f12b4ff9640 1 -- 172.21.15.161:0/1602212531 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f12a0038020 con 0x7f12b80c88b0 2024-03-28T23:33:37.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.205+0000 7f12b4ff9640 1 --2- 172.21.15.161:0/1602212531 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f129806b4d0 0x7f129806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:37.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.205+0000 7f12b4ff9640 1 -- 172.21.15.161:0/1602212531 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6245+0+0 (secure 0 0 0) 0x7f12a00abaa0 con 0x7f12b80c88b0 2024-03-28T23:33:37.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.205+0000 7f12b77fe640 1 --2- 172.21.15.161:0/1602212531 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f129806b4d0 0x7f129806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:37.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.205+0000 7f12b77fe640 1 --2- 172.21.15.161:0/1602212531 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f129806b4d0 0x7f129806d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f12a8005880 tx=0x7f12a8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:37.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.209+0000 7f12b4ff9640 1 -- 172.21.15.161:0/1602212531 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f12a007a700 con 0x7f12b80c88b0 2024-03-28T23:33:37.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.381+0000 7f12be23b640 1 -- 172.21.15.161:0/1602212531 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f12b80ca650 con 0x7f12b80c88b0 2024-03-28T23:33:37.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.381+0000 7f12b4ff9640 1 -- 172.21.15.161:0/1602212531 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) ==== 93+0+10 (secure 0 0 0) 0x7f12a007a000 con 0x7f12b80c88b0 2024-03-28T23:33:37.383 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:33:37.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.383+0000 7f12be23b640 1 -- 172.21.15.161:0/1602212531 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f129806b4d0 msgr2=0x7f129806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:37.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.383+0000 7f12be23b640 1 --2- 172.21.15.161:0/1602212531 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f129806b4d0 0x7f129806d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f12a8005880 tx=0x7f12a8009040 comp rx=0 tx=0).stop 2024-03-28T23:33:37.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.383+0000 7f12be23b640 1 -- 172.21.15.161:0/1602212531 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12b80c88b0 msgr2=0x7f12b81543b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:37.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.383+0000 7f12be23b640 1 --2- 172.21.15.161:0/1602212531 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12b80c88b0 0x7f12b81543b0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f12a0034040 tx=0x7f12a00046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.383+0000 7f12be23b640 1 -- 172.21.15.161:0/1602212531 shutdown_connections 2024-03-28T23:33:37.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.383+0000 7f12be23b640 1 --2- 172.21.15.161:0/1602212531 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f129806b4d0 0x7f129806d990 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.383+0000 7f12be23b640 1 --2- 172.21.15.161:0/1602212531 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12b80c88b0 0x7f12b81543b0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.383+0000 7f12be23b640 1 --2- 172.21.15.161:0/1602212531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12b81529d0 0x7f12b81548f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.383+0000 7f12be23b640 1 --2- 172.21.15.161:0/1602212531 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b80c6750 0x7f12b80c2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.383+0000 7f12be23b640 1 -- 172.21.15.161:0/1602212531 >> 172.21.15.161:0/1602212531 conn(0x7f12b80c7430 msgr2=0x7f12b80ab160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:37.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.384+0000 7f12be23b640 1 -- 172.21.15.161:0/1602212531 shutdown_connections 2024-03-28T23:33:37.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.384+0000 7f12be23b640 1 -- 172.21.15.161:0/1602212531 wait complete. 2024-03-28T23:33:37.395 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:33:37.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.500+0000 7fb5380a4640 1 Processor -- start 2024-03-28T23:33:37.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.500+0000 7fb5380a4640 1 -- start start 2024-03-28T23:33:37.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.501+0000 7fb5380a4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb530152bc0 0x7fb5300c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:37.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.501+0000 7fb5380a4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb5301589b0 0x7fb530158e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:37.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.501+0000 7fb5380a4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5300c6f60 con 0x7fb5300c8a80 2024-03-28T23:33:37.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.501+0000 7fb5380a4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb5300c6120 con 0x7fb5301589b0 2024-03-28T23:33:37.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.501+0000 7fb5380a4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb5300c6400 con 0x7fb530152bc0 2024-03-28T23:33:37.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.501+0000 7fb536641640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb5301589b0 0x7fb530158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:37.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.501+0000 7fb536641640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb5301589b0 0x7fb530158e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54376/0 (socket says 172.21.15.161:54376) 2024-03-28T23:33:37.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.501+0000 7fb536641640 1 -- 172.21.15.161:0/995115596 learned_addr learned my addr 172.21.15.161:0/995115596 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:37.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.501+0000 7fb535e40640 1 --2- 172.21.15.161:0/995115596 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb530152bc0 0x7fb5300c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:37.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.502+0000 7fb536641640 1 -- 172.21.15.161:0/995115596 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb530152bc0 msgr2=0x7fb5300c0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:37.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.502+0000 7fb536641640 1 --2- 172.21.15.161:0/995115596 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb530152bc0 0x7fb5300c0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.502+0000 7fb536641640 1 -- 172.21.15.161:0/995115596 >> v1:172.21.15.170:6789/0 conn(0x7fb5300c8a80 legacy=0x7fb5300c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:37.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.502+0000 7fb536641640 1 -- 172.21.15.161:0/995115596 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5300c7240 con 0x7fb5301589b0 2024-03-28T23:33:37.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.502+0000 7fb536641640 1 --2- 172.21.15.161:0/995115596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5301589b0 0x7fb530158e10 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fb528009870 tx=0x7fb52802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=989d3e6794656881 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:37.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.502+0000 7fb534e3e640 1 -- 172.21.15.161:0/995115596 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb528034070 con 0x7fb5301589b0 2024-03-28T23:33:37.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.502+0000 7fb534e3e640 1 -- 172.21.15.161:0/995115596 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb528030840 con 0x7fb5301589b0 2024-03-28T23:33:37.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.502+0000 7fb534e3e640 1 -- 172.21.15.161:0/995115596 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb528030c10 con 0x7fb5301589b0 2024-03-28T23:33:37.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.503+0000 7fb5380a4640 1 -- 172.21.15.161:0/995115596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5301589b0 msgr2=0x7fb530158e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:37.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.503+0000 7fb5380a4640 1 --2- 172.21.15.161:0/995115596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5301589b0 0x7fb530158e10 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fb528009870 tx=0x7fb52802b980 comp rx=0 tx=0).stop 2024-03-28T23:33:37.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.503+0000 7fb5380a4640 1 -- 172.21.15.161:0/995115596 shutdown_connections 2024-03-28T23:33:37.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.503+0000 7fb5380a4640 1 --2- 172.21.15.161:0/995115596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5301589b0 0x7fb530158e10 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.503+0000 7fb5380a4640 1 --2- 172.21.15.161:0/995115596 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb530152bc0 0x7fb5300c0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.503+0000 7fb5380a4640 1 -- 172.21.15.161:0/995115596 >> 172.21.15.161:0/995115596 conn(0x7fb5300caa80 msgr2=0x7fb5300cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:37.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.503+0000 7fb5380a4640 1 -- 172.21.15.161:0/995115596 shutdown_connections 2024-03-28T23:33:37.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.503+0000 7fb5380a4640 1 -- 172.21.15.161:0/995115596 wait complete. 2024-03-28T23:33:37.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.504+0000 7fb5380a4640 1 Processor -- start 2024-03-28T23:33:37.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.504+0000 7fb5380a4640 1 -- start start 2024-03-28T23:33:37.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.505+0000 7fb5380a4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5300c8a80 0x7fb5301484d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:37.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.505+0000 7fb5380a4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb530152bc0 0x7fb530155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:37.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.505+0000 7fb5380a4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5301589b0 0x7fb53006c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:37.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.505+0000 7fb5380a4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb530152fa0 con 0x7fb5301589b0 2024-03-28T23:33:37.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.505+0000 7fb5380a4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb5300c95f0 con 0x7fb530152bc0 2024-03-28T23:33:37.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.505+0000 7fb5380a4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb530148a10 con 0x7fb5300c8a80 2024-03-28T23:33:37.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.505+0000 7fb536641640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5301589b0 0x7fb53006c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:37.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.505+0000 7fb536641640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5301589b0 0x7fb53006c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54384/0 (socket says 172.21.15.161:54384) 2024-03-28T23:33:37.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.505+0000 7fb535e40640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5300c8a80 0x7fb5301484d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:37.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.505+0000 7fb536641640 1 -- 172.21.15.161:0/6980592 learned_addr learned my addr 172.21.15.161:0/6980592 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:37.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.505+0000 7fb53563f640 1 --2- 172.21.15.161:0/6980592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb530152bc0 0x7fb530155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:37.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.505+0000 7fb536641640 1 -- 172.21.15.161:0/6980592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5300c8a80 msgr2=0x7fb5301484d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:37.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.505+0000 7fb536641640 1 --2- 172.21.15.161:0/6980592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5300c8a80 0x7fb5301484d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.505+0000 7fb536641640 1 -- 172.21.15.161:0/6980592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb530152bc0 msgr2=0x7fb530155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:37.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.505+0000 7fb536641640 1 --2- 172.21.15.161:0/6980592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb530152bc0 0x7fb530155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.505+0000 7fb536641640 1 -- 172.21.15.161:0/6980592 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5301eb5a0 con 0x7fb5301589b0 2024-03-28T23:33:37.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.505+0000 7fb536641640 1 --2- 172.21.15.161:0/6980592 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5301589b0 0x7fb53006c3a0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fb528009870 tx=0x7fb5280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:37.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.506+0000 7fb5267fc640 1 -- 172.21.15.161:0/6980592 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb528034040 con 0x7fb5301589b0 2024-03-28T23:33:37.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.506+0000 7fb5267fc640 1 -- 172.21.15.161:0/6980592 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb528039d90 con 0x7fb5301589b0 2024-03-28T23:33:37.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.506+0000 7fb5267fc640 1 -- 172.21.15.161:0/6980592 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb528043610 con 0x7fb5301589b0 2024-03-28T23:33:37.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.506+0000 7fb5380a4640 1 -- 172.21.15.161:0/6980592 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5301eb880 con 0x7fb5301589b0 2024-03-28T23:33:37.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.506+0000 7fb5380a4640 1 -- 172.21.15.161:0/6980592 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb530158670 con 0x7fb5301589b0 2024-03-28T23:33:37.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.507+0000 7fb5380a4640 1 -- 172.21.15.161:0/6980592 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb53004f4b0 con 0x7fb5301589b0 2024-03-28T23:33:37.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.507+0000 7fb5267fc640 1 -- 172.21.15.161:0/6980592 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb52804a060 con 0x7fb5301589b0 2024-03-28T23:33:37.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.508+0000 7fb5267fc640 1 --2- 172.21.15.161:0/6980592 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb51406b4d0 0x7fb51406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:37.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.508+0000 7fb5267fc640 1 -- 172.21.15.161:0/6980592 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6245+0+0 (secure 0 0 0) 0x7fb5280ac1b0 con 0x7fb5301589b0 2024-03-28T23:33:37.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.508+0000 7fb535e40640 1 --2- 172.21.15.161:0/6980592 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb51406b4d0 0x7fb51406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:37.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.508+0000 7fb535e40640 1 --2- 172.21.15.161:0/6980592 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb51406b4d0 0x7fb51406d990 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb5300c4d80 tx=0x7fb52c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:37.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.512+0000 7fb5267fc640 1 -- 172.21.15.161:0/6980592 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb52803e070 con 0x7fb5301589b0 2024-03-28T23:33:37.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.683+0000 7fb5380a4640 1 -- 172.21.15.161:0/6980592 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb53014c000 con 0x7fb5301589b0 2024-03-28T23:33:37.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.683+0000 7fb5267fc640 1 -- 172.21.15.161:0/6980592 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v97) ==== 104+0+11 (secure 0 0 0) 0x7fb52807a850 con 0x7fb5301589b0 2024-03-28T23:33:37.684 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:33:37.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.685+0000 7fb5380a4640 1 -- 172.21.15.161:0/6980592 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb51406b4d0 msgr2=0x7fb51406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:37.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.685+0000 7fb5380a4640 1 --2- 172.21.15.161:0/6980592 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb51406b4d0 0x7fb51406d990 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb5300c4d80 tx=0x7fb52c00b040 comp rx=0 tx=0).stop 2024-03-28T23:33:37.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.685+0000 7fb5380a4640 1 -- 172.21.15.161:0/6980592 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5301589b0 msgr2=0x7fb53006c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:37.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.685+0000 7fb5380a4640 1 --2- 172.21.15.161:0/6980592 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5301589b0 0x7fb53006c3a0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fb528009870 tx=0x7fb5280046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.685+0000 7fb5380a4640 1 -- 172.21.15.161:0/6980592 shutdown_connections 2024-03-28T23:33:37.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.685+0000 7fb5380a4640 1 --2- 172.21.15.161:0/6980592 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb51406b4d0 0x7fb51406d990 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.685+0000 7fb5380a4640 1 --2- 172.21.15.161:0/6980592 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5301589b0 0x7fb53006c3a0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.685+0000 7fb5380a4640 1 --2- 172.21.15.161:0/6980592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb530152bc0 0x7fb530155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.685+0000 7fb5380a4640 1 --2- 172.21.15.161:0/6980592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5300c8a80 0x7fb5301484d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.685+0000 7fb5380a4640 1 -- 172.21.15.161:0/6980592 >> 172.21.15.161:0/6980592 conn(0x7fb5300caa80 msgr2=0x7fb5300cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:37.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.685+0000 7fb5380a4640 1 -- 172.21.15.161:0/6980592 shutdown_connections 2024-03-28T23:33:37.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.685+0000 7fb5380a4640 1 -- 172.21.15.161:0/6980592 wait complete. 2024-03-28T23:33:37.697 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:33:37.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.800+0000 7f56b91bc640 1 Processor -- start 2024-03-28T23:33:37.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.800+0000 7f56b91bc640 1 -- start start 2024-03-28T23:33:37.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.801+0000 7f56b91bc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56b4151e70 0x7f56b4152250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:37.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.801+0000 7f56b91bc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f56b406cdd0 0x7f56b415c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:37.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.801+0000 7f56b91bc640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56b40c1e40 con 0x7f56b4152790 2024-03-28T23:33:37.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.801+0000 7f56b91bc640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f56b40c2120 con 0x7f56b406cdd0 2024-03-28T23:33:37.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.801+0000 7f56b91bc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f56b40c2400 con 0x7f56b4151e70 2024-03-28T23:33:37.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.801+0000 7f56b2d9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56b4151e70 0x7f56b4152250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:37.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.801+0000 7f56b359e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f56b406cdd0 0x7f56b415c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:37.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.801+0000 7f56abfff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f56b4152790 0x7f56b406c6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:58074/0 (socket says 172.21.15.161:58074) 2024-03-28T23:33:37.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.801+0000 7f56abfff640 1 -- 172.21.15.161:0/3111461657 learned_addr learned my addr 172.21.15.161:0/3111461657 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:37.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.801+0000 7f56b2d9d640 1 -- 172.21.15.161:0/3111461657 >> v2:172.21.15.170:3300/0 conn(0x7f56b406cdd0 msgr2=0x7f56b415c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:37.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.801+0000 7f56b2d9d640 1 --2- 172.21.15.161:0/3111461657 >> v2:172.21.15.170:3300/0 conn(0x7f56b406cdd0 0x7f56b415c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.801+0000 7f56b2d9d640 1 -- 172.21.15.161:0/3111461657 >> v1:172.21.15.170:6789/0 conn(0x7f56b4152790 legacy=0x7f56b406c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:37.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.801+0000 7f56b2d9d640 1 -- 172.21.15.161:0/3111461657 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b40c26e0 con 0x7f56b4151e70 2024-03-28T23:33:37.804 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.802+0000 7f56b2d9d640 1 --2- 172.21.15.161:0/3111461657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56b4151e70 0x7f56b4152250 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f56a40097a0 tx=0x7f56a402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cbebe67ea92a133 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:37.804 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.802+0000 7f56b259c640 1 -- 172.21.15.161:0/3111461657 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a40315a0 con 0x7f56b4151e70 2024-03-28T23:33:37.804 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.802+0000 7f56b259c640 1 -- 172.21.15.161:0/3111461657 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56a4037070 con 0x7f56b4151e70 2024-03-28T23:33:37.804 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.802+0000 7f56b259c640 1 -- 172.21.15.161:0/3111461657 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a4031960 con 0x7f56b4151e70 2024-03-28T23:33:37.804 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.803+0000 7f56b91bc640 1 -- 172.21.15.161:0/3111461657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56b4151e70 msgr2=0x7f56b4152250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:37.805 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.803+0000 7f56b91bc640 1 --2- 172.21.15.161:0/3111461657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56b4151e70 0x7f56b4152250 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f56a40097a0 tx=0x7f56a402b840 comp rx=0 tx=0).stop 2024-03-28T23:33:37.805 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.803+0000 7f56b91bc640 1 -- 172.21.15.161:0/3111461657 shutdown_connections 2024-03-28T23:33:37.805 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.803+0000 7f56b91bc640 1 --2- 172.21.15.161:0/3111461657 >> v2:172.21.15.170:3300/0 conn(0x7f56b406cdd0 0x7f56b415c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.805 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.803+0000 7f56b91bc640 1 --2- 172.21.15.161:0/3111461657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56b4151e70 0x7f56b4152250 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.805 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.803+0000 7f56b91bc640 1 -- 172.21.15.161:0/3111461657 >> 172.21.15.161:0/3111461657 conn(0x7f56b40c3390 msgr2=0x7f56b40c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:37.805 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.803+0000 7f56b91bc640 1 -- 172.21.15.161:0/3111461657 shutdown_connections 2024-03-28T23:33:37.805 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.803+0000 7f56b91bc640 1 -- 172.21.15.161:0/3111461657 wait complete. 2024-03-28T23:33:37.805 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.804+0000 7f56b91bc640 1 Processor -- start 2024-03-28T23:33:37.805 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.804+0000 7f56b91bc640 1 -- start start 2024-03-28T23:33:37.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.805+0000 7f56b91bc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56b406cdd0 0x7f56b40bff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:37.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.805+0000 7f56b91bc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56b4151e70 0x7f56b41489f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:37.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.805+0000 7f56b91bc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56b41eafd0 0x7f56b4148f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:37.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.805+0000 7f56b91bc640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f56b40c2d80 con 0x7f56b406cdd0 2024-03-28T23:33:37.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.805+0000 7f56b91bc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f56b40c2c00 con 0x7f56b4151e70 2024-03-28T23:33:37.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.805+0000 7f56b91bc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f56b40c6350 con 0x7f56b41eafd0 2024-03-28T23:33:37.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.805+0000 7f56b2d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56b406cdd0 0x7f56b40bff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:37.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.805+0000 7f56b359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56b41eafd0 0x7f56b4148f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:37.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.805+0000 7f56b2d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56b406cdd0 0x7f56b40bff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54398/0 (socket says 172.21.15.161:54398) 2024-03-28T23:33:37.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.805+0000 7f56b2d9d640 1 -- 172.21.15.161:0/1220116460 learned_addr learned my addr 172.21.15.161:0/1220116460 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:37.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.805+0000 7f56abfff640 1 --2- 172.21.15.161:0/1220116460 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56b4151e70 0x7f56b41489f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:37.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.805+0000 7f56b2d9d640 1 -- 172.21.15.161:0/1220116460 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56b41eafd0 msgr2=0x7f56b4148f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:37.809 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.805+0000 7f56b2d9d640 1 --2- 172.21.15.161:0/1220116460 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56b41eafd0 0x7f56b4148f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.809 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.805+0000 7f56b2d9d640 1 -- 172.21.15.161:0/1220116460 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56b4151e70 msgr2=0x7f56b41489f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:37.809 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.805+0000 7f56b2d9d640 1 --2- 172.21.15.161:0/1220116460 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56b4151e70 0x7f56b41489f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.809 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.805+0000 7f56b2d9d640 1 -- 172.21.15.161:0/1220116460 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b41eb540 con 0x7f56b406cdd0 2024-03-28T23:33:37.809 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.805+0000 7f56b2d9d640 1 --2- 172.21.15.161:0/1220116460 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56b406cdd0 0x7f56b40bff30 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f56a4034040 tx=0x7f56a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:37.809 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.805+0000 7f56ab7fe640 1 -- 172.21.15.161:0/1220116460 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a4032630 con 0x7f56b406cdd0 2024-03-28T23:33:37.809 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.805+0000 7f56ab7fe640 1 -- 172.21.15.161:0/1220116460 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56a40327f0 con 0x7f56b406cdd0 2024-03-28T23:33:37.809 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.805+0000 7f56b91bc640 1 -- 172.21.15.161:0/1220116460 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56b41eb820 con 0x7f56b406cdd0 2024-03-28T23:33:37.809 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.805+0000 7f56ab7fe640 1 -- 172.21.15.161:0/1220116460 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a4032be0 con 0x7f56b406cdd0 2024-03-28T23:33:37.809 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.805+0000 7f56b91bc640 1 -- 172.21.15.161:0/1220116460 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56b41ebde0 con 0x7f56b406cdd0 2024-03-28T23:33:37.809 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.806+0000 7f56b91bc640 1 -- 172.21.15.161:0/1220116460 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56b41ee670 con 0x7f56b406cdd0 2024-03-28T23:33:37.809 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.807+0000 7f56ab7fe640 1 -- 172.21.15.161:0/1220116460 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f56a404e430 con 0x7f56b406cdd0 2024-03-28T23:33:37.809 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.807+0000 7f56ab7fe640 1 --2- 172.21.15.161:0/1220116460 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f569406b4d0 0x7f569406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:37.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.807+0000 7f56ab7fe640 1 -- 172.21.15.161:0/1220116460 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6245+0+0 (secure 0 0 0) 0x7f56a402f030 con 0x7f56b406cdd0 2024-03-28T23:33:37.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.807+0000 7f56abfff640 1 --2- 172.21.15.161:0/1220116460 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f569406b4d0 0x7f569406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:37.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.808+0000 7f56abfff640 1 --2- 172.21.15.161:0/1220116460 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f569406b4d0 0x7f569406d990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f56b414bca0 tx=0x7f569c007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:37.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.812+0000 7f56ab7fe640 1 -- 172.21.15.161:0/1220116460 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f56a4040070 con 0x7f56b406cdd0 2024-03-28T23:33:37.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.983+0000 7f56b91bc640 1 -- 172.21.15.161:0/1220116460 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f56b404f4b0 con 0x7f56b406cdd0 2024-03-28T23:33:37.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.983+0000 7f56ab7fe640 1 -- 172.21.15.161:0/1220116460 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v97) ==== 100+0+11 (secure 0 0 0) 0x7f56a407a7e0 con 0x7f56b406cdd0 2024-03-28T23:33:37.984 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:33:37.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.985+0000 7f56b91bc640 1 -- 172.21.15.161:0/1220116460 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f569406b4d0 msgr2=0x7f569406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:37.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.985+0000 7f56b91bc640 1 --2- 172.21.15.161:0/1220116460 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f569406b4d0 0x7f569406d990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f56b414bca0 tx=0x7f569c007e80 comp rx=0 tx=0).stop 2024-03-28T23:33:37.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.985+0000 7f56b91bc640 1 -- 172.21.15.161:0/1220116460 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56b406cdd0 msgr2=0x7f56b40bff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:37.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.985+0000 7f56b91bc640 1 --2- 172.21.15.161:0/1220116460 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56b406cdd0 0x7f56b40bff30 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f56a4034040 tx=0x7f56a40046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.985+0000 7f56b91bc640 1 -- 172.21.15.161:0/1220116460 shutdown_connections 2024-03-28T23:33:37.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.985+0000 7f56b91bc640 1 --2- 172.21.15.161:0/1220116460 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f569406b4d0 0x7f569406d990 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.985+0000 7f56b91bc640 1 --2- 172.21.15.161:0/1220116460 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56b41eafd0 0x7f56b4148f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.985+0000 7f56b91bc640 1 --2- 172.21.15.161:0/1220116460 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56b4151e70 0x7f56b41489f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.985+0000 7f56b91bc640 1 --2- 172.21.15.161:0/1220116460 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56b406cdd0 0x7f56b40bff30 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:37.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.985+0000 7f56b91bc640 1 -- 172.21.15.161:0/1220116460 >> 172.21.15.161:0/1220116460 conn(0x7f56b40c3390 msgr2=0x7f56b414c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:37.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.985+0000 7f56b91bc640 1 -- 172.21.15.161:0/1220116460 shutdown_connections 2024-03-28T23:33:37.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:37.985+0000 7f56b91bc640 1 -- 172.21.15.161:0/1220116460 wait complete. 2024-03-28T23:33:37.997 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:33:38.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.096+0000 7fbaf224e640 1 Processor -- start 2024-03-28T23:33:38.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.097+0000 7fbaf224e640 1 -- start start 2024-03-28T23:33:38.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.097+0000 7fbaf224e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbaec06ca20 0x7fbaec06ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:38.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.097+0000 7fbaf224e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbaec151850 con 0x7fbaec0a98c0 2024-03-28T23:33:38.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.097+0000 7fbaf224e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbaec151b30 con 0x7fbaec14eca0 2024-03-28T23:33:38.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.097+0000 7fbaf224e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbaec152460 con 0x7fbaec06ca20 2024-03-28T23:33:38.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.097+0000 7fbaebfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fbaec0a98c0 0x7fbaec0a9d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52996/0 (socket says 172.21.15.161:52996) 2024-03-28T23:33:38.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.097+0000 7fbaebfff640 1 -- 172.21.15.161:0/3178229347 learned_addr learned my addr 172.21.15.161:0/3178229347 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:38.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.097+0000 7fbaeaffd640 1 --2- 172.21.15.161:0/3178229347 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbaec06ca20 0x7fbaec06ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:38.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.098+0000 7fbaea7fc640 1 -- 172.21.15.161:0/3178229347 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 163948127 0 0) 0x7fbaec151850 con 0x7fbaec0a98c0 2024-03-28T23:33:38.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.098+0000 7fbaea7fc640 1 -- 172.21.15.161:0/3178229347 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbad4003520 con 0x7fbaec0a98c0 2024-03-28T23:33:38.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.098+0000 7fbaea7fc640 1 -- 172.21.15.161:0/3178229347 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 322417721 0 0) 0x7fbaec151b30 con 0x7fbaec14eca0 2024-03-28T23:33:38.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.098+0000 7fbaea7fc640 1 -- 172.21.15.161:0/3178229347 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbaec151850 con 0x7fbaec14eca0 2024-03-28T23:33:38.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.098+0000 7fbaeaffd640 1 -- 172.21.15.161:0/3178229347 >> v1:172.21.15.170:6789/0 conn(0x7fbaec14eca0 legacy=0x7fbaec0c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:38.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.098+0000 7fbaeaffd640 1 -- 172.21.15.161:0/3178229347 >> v1:172.21.15.161:6789/0 conn(0x7fbaec0a98c0 legacy=0x7fbaec0a9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:38.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.098+0000 7fbaeaffd640 1 -- 172.21.15.161:0/3178229347 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaec151e10 con 0x7fbaec06ca20 2024-03-28T23:33:38.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.099+0000 7fbaeaffd640 1 --2- 172.21.15.161:0/3178229347 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbaec06ca20 0x7fbaec06ce80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbae00097d0 tx=0x7fbae002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbf973557c23ebd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:38.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.099+0000 7fbaea7fc640 1 -- 172.21.15.161:0/3178229347 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae0035020 con 0x7fbaec06ca20 2024-03-28T23:33:38.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.099+0000 7fbaea7fc640 1 -- 172.21.15.161:0/3178229347 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbae0031800 con 0x7fbaec06ca20 2024-03-28T23:33:38.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.099+0000 7fbaea7fc640 1 -- 172.21.15.161:0/3178229347 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae0031bd0 con 0x7fbaec06ca20 2024-03-28T23:33:38.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.099+0000 7fbaf224e640 1 -- 172.21.15.161:0/3178229347 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbaec06ca20 msgr2=0x7fbaec06ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:38.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.099+0000 7fbaf224e640 1 --2- 172.21.15.161:0/3178229347 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbaec06ca20 0x7fbaec06ce80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbae00097d0 tx=0x7fbae002b8a0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.099+0000 7fbaf224e640 1 -- 172.21.15.161:0/3178229347 shutdown_connections 2024-03-28T23:33:38.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.099+0000 7fbaf224e640 1 --2- 172.21.15.161:0/3178229347 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbaec06ca20 0x7fbaec06ce80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.099+0000 7fbaf224e640 1 -- 172.21.15.161:0/3178229347 >> 172.21.15.161:0/3178229347 conn(0x7fbaec0c6670 msgr2=0x7fbaec0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:38.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.099+0000 7fbaf224e640 1 -- 172.21.15.161:0/3178229347 shutdown_connections 2024-03-28T23:33:38.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.099+0000 7fbaf224e640 1 -- 172.21.15.161:0/3178229347 wait complete. 2024-03-28T23:33:38.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.100+0000 7fbaf224e640 1 Processor -- start 2024-03-28T23:33:38.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.101+0000 7fbaf224e640 1 -- start start 2024-03-28T23:33:38.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.101+0000 7fbaf224e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbaec06ca20 0x7fbaec0c0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:38.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.101+0000 7fbaf224e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbaec0a98c0 0x7fbaec0bfc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:38.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.101+0000 7fbaf224e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbaec14eca0 0x7fbaec1ea6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:38.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.101+0000 7fbaf224e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbaec1514b0 con 0x7fbaec14eca0 2024-03-28T23:33:38.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.101+0000 7fbaf224e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbaec14f080 con 0x7fbaec0a98c0 2024-03-28T23:33:38.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.101+0000 7fbaf224e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbaec151630 con 0x7fbaec06ca20 2024-03-28T23:33:38.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.101+0000 7fbaebfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbaec14eca0 0x7fbaec1ea6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:38.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.101+0000 7fbaebfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbaec14eca0 0x7fbaec1ea6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54406/0 (socket says 172.21.15.161:54406) 2024-03-28T23:33:38.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.101+0000 7fbaebfff640 1 -- 172.21.15.161:0/2563218956 learned_addr learned my addr 172.21.15.161:0/2563218956 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:38.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.101+0000 7fbaeb7fe640 1 --2- 172.21.15.161:0/2563218956 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbaec06ca20 0x7fbaec0c0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:38.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.102+0000 7fbaeaffd640 1 --2- 172.21.15.161:0/2563218956 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbaec0a98c0 0x7fbaec0bfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:38.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.102+0000 7fbaebfff640 1 -- 172.21.15.161:0/2563218956 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbaec06ca20 msgr2=0x7fbaec0c0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:38.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.102+0000 7fbaebfff640 1 --2- 172.21.15.161:0/2563218956 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbaec06ca20 0x7fbaec0c0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.102+0000 7fbaebfff640 1 -- 172.21.15.161:0/2563218956 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbaec0a98c0 msgr2=0x7fbaec0bfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:38.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.102+0000 7fbaebfff640 1 --2- 172.21.15.161:0/2563218956 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbaec0a98c0 0x7fbaec0bfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.102+0000 7fbaebfff640 1 -- 172.21.15.161:0/2563218956 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaec0a8890 con 0x7fbaec14eca0 2024-03-28T23:33:38.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.102+0000 7fbaebfff640 1 --2- 172.21.15.161:0/2563218956 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbaec14eca0 0x7fbaec1ea6d0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fbae400cb40 tx=0x7fbae400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:38.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.102+0000 7fbad3fff640 1 -- 172.21.15.161:0/2563218956 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae4017070 con 0x7fbaec14eca0 2024-03-28T23:33:38.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.102+0000 7fbad3fff640 1 -- 172.21.15.161:0/2563218956 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbae400aca0 con 0x7fbaec14eca0 2024-03-28T23:33:38.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.102+0000 7fbaf224e640 1 -- 172.21.15.161:0/2563218956 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaec0a8b70 con 0x7fbaec14eca0 2024-03-28T23:33:38.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.102+0000 7fbad3fff640 1 -- 172.21.15.161:0/2563218956 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae4012600 con 0x7fbaec14eca0 2024-03-28T23:33:38.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.102+0000 7fbaf224e640 1 -- 172.21.15.161:0/2563218956 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaec1554a0 con 0x7fbaec14eca0 2024-03-28T23:33:38.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.103+0000 7fbaf224e640 1 -- 172.21.15.161:0/2563218956 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaec04f4b0 con 0x7fbaec14eca0 2024-03-28T23:33:38.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.103+0000 7fbad3fff640 1 -- 172.21.15.161:0/2563218956 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbae401a070 con 0x7fbaec14eca0 2024-03-28T23:33:38.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.104+0000 7fbad3fff640 1 --2- 172.21.15.161:0/2563218956 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbad806b4d0 0x7fbad806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:38.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.104+0000 7fbad3fff640 1 -- 172.21.15.161:0/2563218956 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6245+0+0 (secure 0 0 0) 0x7fbae408a0e0 con 0x7fbaec14eca0 2024-03-28T23:33:38.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.104+0000 7fbaeb7fe640 1 --2- 172.21.15.161:0/2563218956 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbad806b4d0 0x7fbad806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:38.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.104+0000 7fbaeb7fe640 1 --2- 172.21.15.161:0/2563218956 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbad806b4d0 0x7fbad806d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fbadc007fd0 tx=0x7fbadc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:38.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.108+0000 7fbad3fff640 1 -- 172.21.15.161:0/2563218956 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbae4058780 con 0x7fbaec14eca0 2024-03-28T23:33:38.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.318+0000 7fbaf224e640 1 -- 172.21.15.161:0/2563218956 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbaec14e050 con 0x7fbaec14eca0 2024-03-28T23:33:38.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.319+0000 7fbad3fff640 1 -- 172.21.15.161:0/2563218956 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) ==== 94+0+5192 (secure 0 0 0) 0x7fbae405c820 con 0x7fbaec14eca0 2024-03-28T23:33:38.320 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:38.320 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":84,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6547,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4280217233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4280217233},{"type":"v1","addr":"172.21.15.161:6836","nonce":4280217233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":6706,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/2647418289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":2647418289},{"type":"v1","addr":"172.21.15.170:6834","nonce":2647418289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":7086,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/404132405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":404132405},{"type":"v1","addr":"172.21.15.161:6837","nonce":404132405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:33:23.516182+0000","modified":"2024-03-28T23:33:34.430703+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5963},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5963":{"gid":5963,"name":"b","rank":0,"incarnation":81,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6835/3079114491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3079114491},{"type":"v1","addr":"172.21.15.170:6835","nonce":3079114491}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5963,"qdb_cluster":[5963]},"id":7}]} 2024-03-28T23:33:38.320 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 84 2024-03-28T23:33:38.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.321+0000 7fbaf224e640 1 -- 172.21.15.161:0/2563218956 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbad806b4d0 msgr2=0x7fbad806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:38.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.321+0000 7fbaf224e640 1 --2- 172.21.15.161:0/2563218956 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbad806b4d0 0x7fbad806d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fbadc007fd0 tx=0x7fbadc009040 comp rx=0 tx=0).stop 2024-03-28T23:33:38.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.321+0000 7fbaf224e640 1 -- 172.21.15.161:0/2563218956 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbaec14eca0 msgr2=0x7fbaec1ea6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:38.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.321+0000 7fbaf224e640 1 --2- 172.21.15.161:0/2563218956 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbaec14eca0 0x7fbaec1ea6d0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fbae400cb40 tx=0x7fbae400a060 comp rx=0 tx=0).stop 2024-03-28T23:33:38.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.321+0000 7fbaf224e640 1 -- 172.21.15.161:0/2563218956 shutdown_connections 2024-03-28T23:33:38.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.321+0000 7fbaf224e640 1 --2- 172.21.15.161:0/2563218956 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbad806b4d0 0x7fbad806d990 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.321+0000 7fbaf224e640 1 --2- 172.21.15.161:0/2563218956 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbaec14eca0 0x7fbaec1ea6d0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.321+0000 7fbaf224e640 1 --2- 172.21.15.161:0/2563218956 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbaec0a98c0 0x7fbaec0bfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.321+0000 7fbaf224e640 1 --2- 172.21.15.161:0/2563218956 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbaec06ca20 0x7fbaec0c0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.321+0000 7fbaf224e640 1 -- 172.21.15.161:0/2563218956 >> 172.21.15.161:0/2563218956 conn(0x7fbaec0c6670 msgr2=0x7fbaec0c1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:38.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.321+0000 7fbaf224e640 1 -- 172.21.15.161:0/2563218956 shutdown_connections 2024-03-28T23:33:38.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.321+0000 7fbaf224e640 1 -- 172.21.15.161:0/2563218956 wait complete. 2024-03-28T23:33:38.333 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:33:38.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.434+0000 7ffad404e640 1 Processor -- start 2024-03-28T23:33:38.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.434+0000 7ffad404e640 1 -- start start 2024-03-28T23:33:38.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.435+0000 7ffad404e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffacc150590 0x7ffacc0a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:38.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.435+0000 7ffad404e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffacc1566f0 con 0x7ffacc0aacb0 2024-03-28T23:33:38.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.435+0000 7ffad404e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffacc1569d0 con 0x7ffacc0a8fc0 2024-03-28T23:33:38.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.435+0000 7ffad404e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ffacc156cb0 con 0x7ffacc150590 2024-03-28T23:33:38.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.435+0000 7ffad25eb640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ffacc0aacb0 0x7ffacc153860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:53004/0 (socket says 172.21.15.161:53004) 2024-03-28T23:33:38.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.435+0000 7ffad25eb640 1 -- 172.21.15.161:0/126589036 learned_addr learned my addr 172.21.15.161:0/126589036 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:38.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.435+0000 7ffad1dea640 1 --2- 172.21.15.161:0/126589036 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffacc150590 0x7ffacc0a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:38.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.435+0000 7ffad0de8640 1 -- 172.21.15.161:0/126589036 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2029584223 0 0) 0x7ffacc1566f0 con 0x7ffacc0aacb0 2024-03-28T23:33:38.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.435+0000 7ffad0de8640 1 -- 172.21.15.161:0/126589036 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffab8003520 con 0x7ffacc0aacb0 2024-03-28T23:33:38.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.435+0000 7ffad0de8640 1 -- 172.21.15.161:0/126589036 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2252170191 0 0) 0x7ffacc1569d0 con 0x7ffacc0a8fc0 2024-03-28T23:33:38.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.435+0000 7ffad1dea640 1 -- 172.21.15.161:0/126589036 >> v1:172.21.15.170:6789/0 conn(0x7ffacc0a8fc0 legacy=0x7ffacc153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:38.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.435+0000 7ffad1dea640 1 -- 172.21.15.161:0/126589036 >> v1:172.21.15.161:6789/0 conn(0x7ffacc0aacb0 legacy=0x7ffacc153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:38.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.435+0000 7ffad1dea640 1 -- 172.21.15.161:0/126589036 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffacc156f90 con 0x7ffacc150590 2024-03-28T23:33:38.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.436+0000 7ffad1dea640 1 --2- 172.21.15.161:0/126589036 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffacc150590 0x7ffacc0a9f40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ffac80097d0 tx=0x7ffac802b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=621ee6952cee92f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:38.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.436+0000 7ffad0de8640 1 -- 172.21.15.161:0/126589036 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac8031770 con 0x7ffacc150590 2024-03-28T23:33:38.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.436+0000 7ffad0de8640 1 -- 172.21.15.161:0/126589036 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffac8037070 con 0x7ffacc150590 2024-03-28T23:33:38.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.436+0000 7ffad0de8640 1 -- 172.21.15.161:0/126589036 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac8031b30 con 0x7ffacc150590 2024-03-28T23:33:38.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.437+0000 7ffad404e640 1 -- 172.21.15.161:0/126589036 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffacc150590 msgr2=0x7ffacc0a9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:38.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.437+0000 7ffad404e640 1 --2- 172.21.15.161:0/126589036 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffacc150590 0x7ffacc0a9f40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ffac80097d0 tx=0x7ffac802b810 comp rx=0 tx=0).stop 2024-03-28T23:33:38.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.437+0000 7ffad404e640 1 -- 172.21.15.161:0/126589036 shutdown_connections 2024-03-28T23:33:38.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.437+0000 7ffad404e640 1 --2- 172.21.15.161:0/126589036 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffacc150590 0x7ffacc0a9f40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.437+0000 7ffad404e640 1 -- 172.21.15.161:0/126589036 >> 172.21.15.161:0/126589036 conn(0x7ffacc0c6670 msgr2=0x7ffacc0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:38.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.437+0000 7ffad404e640 1 -- 172.21.15.161:0/126589036 shutdown_connections 2024-03-28T23:33:38.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.437+0000 7ffad404e640 1 -- 172.21.15.161:0/126589036 wait complete. 2024-03-28T23:33:38.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.438+0000 7ffad404e640 1 Processor -- start 2024-03-28T23:33:38.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.438+0000 7ffad404e640 1 -- start start 2024-03-28T23:33:38.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.439+0000 7ffad404e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffacc0a8fc0 0x7ffacc1442d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:38.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.439+0000 7ffad404e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffacc0aacb0 0x7ffacc0c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:38.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.439+0000 7ffad404e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffacc150590 0x7ffacc154d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:38.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.439+0000 7ffad404e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ffacc0aa480 con 0x7ffacc150590 2024-03-28T23:33:38.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.439+0000 7ffad404e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ffacc150970 con 0x7ffacc0a8fc0 2024-03-28T23:33:38.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.439+0000 7ffad404e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ffacc0aa600 con 0x7ffacc0aacb0 2024-03-28T23:33:38.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.439+0000 7ffad25eb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffacc150590 0x7ffacc154d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:38.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.439+0000 7ffad25eb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffacc150590 0x7ffacc154d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54412/0 (socket says 172.21.15.161:54412) 2024-03-28T23:33:38.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.439+0000 7ffad25eb640 1 -- 172.21.15.161:0/2964297710 learned_addr learned my addr 172.21.15.161:0/2964297710 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:38.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.439+0000 7ffad1dea640 1 --2- 172.21.15.161:0/2964297710 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffacc0a8fc0 0x7ffacc1442d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:38.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.439+0000 7ffad15e9640 1 --2- 172.21.15.161:0/2964297710 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffacc0aacb0 0x7ffacc0c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:38.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.439+0000 7ffad25eb640 1 -- 172.21.15.161:0/2964297710 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffacc0aacb0 msgr2=0x7ffacc0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:38.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.439+0000 7ffad25eb640 1 --2- 172.21.15.161:0/2964297710 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffacc0aacb0 0x7ffacc0c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.439+0000 7ffad25eb640 1 -- 172.21.15.161:0/2964297710 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffacc0a8fc0 msgr2=0x7ffacc1442d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:38.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.439+0000 7ffad25eb640 1 --2- 172.21.15.161:0/2964297710 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffacc0a8fc0 0x7ffacc1442d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.439+0000 7ffad25eb640 1 -- 172.21.15.161:0/2964297710 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffacc0c5380 con 0x7ffacc150590 2024-03-28T23:33:38.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.439+0000 7ffad25eb640 1 --2- 172.21.15.161:0/2964297710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffacc150590 0x7ffacc154d70 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7ffac400cb60 tx=0x7ffac400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:38.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.440+0000 7ffac27fc640 1 -- 172.21.15.161:0/2964297710 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac4017070 con 0x7ffacc150590 2024-03-28T23:33:38.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.440+0000 7ffac27fc640 1 -- 172.21.15.161:0/2964297710 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffac400ad40 con 0x7ffacc150590 2024-03-28T23:33:38.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.440+0000 7ffac27fc640 1 -- 172.21.15.161:0/2964297710 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac4012600 con 0x7ffacc150590 2024-03-28T23:33:38.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.440+0000 7ffad404e640 1 -- 172.21.15.161:0/2964297710 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffacc0c5660 con 0x7ffacc150590 2024-03-28T23:33:38.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.440+0000 7ffad404e640 1 -- 172.21.15.161:0/2964297710 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffacc0c63f0 con 0x7ffacc150590 2024-03-28T23:33:38.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.441+0000 7ffad404e640 1 -- 172.21.15.161:0/2964297710 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffacc04f4b0 con 0x7ffacc150590 2024-03-28T23:33:38.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.441+0000 7ffac27fc640 1 -- 172.21.15.161:0/2964297710 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ffac401a070 con 0x7ffacc150590 2024-03-28T23:33:38.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.441+0000 7ffac27fc640 1 --2- 172.21.15.161:0/2964297710 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffab006b4d0 0x7ffab006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:38.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.442+0000 7ffac27fc640 1 -- 172.21.15.161:0/2964297710 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6245+0+0 (secure 0 0 0) 0x7ffac408a010 con 0x7ffacc150590 2024-03-28T23:33:38.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.442+0000 7ffad1dea640 1 --2- 172.21.15.161:0/2964297710 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffab006b4d0 0x7ffab006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:38.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.442+0000 7ffad1dea640 1 --2- 172.21.15.161:0/2964297710 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffab006b4d0 0x7ffab006d990 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ffac8036000 tx=0x7ffac8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:38.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.446+0000 7ffac27fc640 1 -- 172.21.15.161:0/2964297710 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ffac40586b0 con 0x7ffacc150590 2024-03-28T23:33:38.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.618+0000 7ffad404e640 1 -- 172.21.15.161:0/2964297710 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffacc0a9d20 con 0x7ffacc150590 2024-03-28T23:33:38.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.619+0000 7ffac27fc640 1 -- 172.21.15.161:0/2964297710 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+17138 (secure 0 0 0) 0x7ffac405c750 con 0x7ffacc150590 2024-03-28T23:33:38.620 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:38.620 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":97,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:33:35.435937+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:33:21.732337+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-28T23:33:22.477357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.1d","mappings":[{"from":1,"to":7}]},{"pgid":"16.25","mappings":[{"from":5,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6834/2176248563":"2024-03-29T23:33:31.422647+0000","172.21.15.170:6832/2176248563":"2024-03-29T23:33:31.422647+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:33:38.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.621+0000 7ffad404e640 1 -- 172.21.15.161:0/2964297710 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffab006b4d0 msgr2=0x7ffab006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:38.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.621+0000 7ffad404e640 1 --2- 172.21.15.161:0/2964297710 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffab006b4d0 0x7ffab006d990 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ffac8036000 tx=0x7ffac8008030 comp rx=0 tx=0).stop 2024-03-28T23:33:38.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.621+0000 7ffad404e640 1 -- 172.21.15.161:0/2964297710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffacc150590 msgr2=0x7ffacc154d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:38.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.621+0000 7ffad404e640 1 --2- 172.21.15.161:0/2964297710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffacc150590 0x7ffacc154d70 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7ffac400cb60 tx=0x7ffac400a430 comp rx=0 tx=0).stop 2024-03-28T23:33:38.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.621+0000 7ffad404e640 1 -- 172.21.15.161:0/2964297710 shutdown_connections 2024-03-28T23:33:38.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.621+0000 7ffad404e640 1 --2- 172.21.15.161:0/2964297710 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffab006b4d0 0x7ffab006d990 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.621+0000 7ffad404e640 1 --2- 172.21.15.161:0/2964297710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffacc150590 0x7ffacc154d70 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.621+0000 7ffad404e640 1 --2- 172.21.15.161:0/2964297710 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffacc0aacb0 0x7ffacc0c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.621+0000 7ffad404e640 1 --2- 172.21.15.161:0/2964297710 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffacc0a8fc0 0x7ffacc1442d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.621+0000 7ffad404e640 1 -- 172.21.15.161:0/2964297710 >> 172.21.15.161:0/2964297710 conn(0x7ffacc0c6670 msgr2=0x7ffacc0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:38.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.621+0000 7ffad404e640 1 -- 172.21.15.161:0/2964297710 shutdown_connections 2024-03-28T23:33:38.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.621+0000 7ffad404e640 1 -- 172.21.15.161:0/2964297710 wait complete. 2024-03-28T23:33:38.633 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-28T23:33:38.634 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:33:38.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.734+0000 7fec9a71e640 1 Processor -- start 2024-03-28T23:33:38.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.735+0000 7fec9a71e640 1 -- start start 2024-03-28T23:33:38.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.735+0000 7fec9a71e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fec94152890 0x7fec940c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:38.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.735+0000 7fec9a71e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec940c8ed0 0x7fec940c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:38.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.735+0000 7fec9a71e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fec940c5950 0x7fec941553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:38.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.735+0000 7fec9a71e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fec940c8480 con 0x7fec94152890 2024-03-28T23:33:38.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.735+0000 7fec9a71e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fec94159640 con 0x7fec940c5950 2024-03-28T23:33:38.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.735+0000 7fec9a71e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fec94158a30 con 0x7fec940c8ed0 2024-03-28T23:33:38.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.735+0000 7fec93fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fec94152890 0x7fec940c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:38.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.735+0000 7fec93fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fec94152890 0x7fec940c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54414/0 (socket says 172.21.15.161:54414) 2024-03-28T23:33:38.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.735+0000 7fec93fff640 1 -- 172.21.15.161:0/1178176024 learned_addr learned my addr 172.21.15.161:0/1178176024 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:38.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.735+0000 7fec937fe640 1 --2- 172.21.15.161:0/1178176024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec940c8ed0 0x7fec940c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:38.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.735+0000 7fec98cbb640 1 --2- 172.21.15.161:0/1178176024 >> v2:172.21.15.170:3300/0 conn(0x7fec940c5950 0x7fec941553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:38.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.736+0000 7fec93fff640 1 -- 172.21.15.161:0/1178176024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec940c8ed0 msgr2=0x7fec940c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:38.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.736+0000 7fec93fff640 1 --2- 172.21.15.161:0/1178176024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec940c8ed0 0x7fec940c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.736+0000 7fec93fff640 1 -- 172.21.15.161:0/1178176024 >> v2:172.21.15.170:3300/0 conn(0x7fec940c5950 msgr2=0x7fec941553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:38.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.736+0000 7fec93fff640 1 --2- 172.21.15.161:0/1178176024 >> v2:172.21.15.170:3300/0 conn(0x7fec940c5950 0x7fec941553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.736+0000 7fec93fff640 1 -- 172.21.15.161:0/1178176024 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec94158d10 con 0x7fec94152890 2024-03-28T23:33:38.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.736+0000 7fec93fff640 1 --2- 172.21.15.161:0/1178176024 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec94152890 0x7fec940c8990 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fec940b0c00 tx=0x7fec8802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5315425ae51b1b20 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:38.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.737+0000 7fec92ffd640 1 -- 172.21.15.161:0/1178176024 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec88038070 con 0x7fec94152890 2024-03-28T23:33:38.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.737+0000 7fec92ffd640 1 -- 172.21.15.161:0/1178176024 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec8802f050 con 0x7fec94152890 2024-03-28T23:33:38.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.737+0000 7fec92ffd640 1 -- 172.21.15.161:0/1178176024 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec88031970 con 0x7fec94152890 2024-03-28T23:33:38.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.737+0000 7fec9a71e640 1 -- 172.21.15.161:0/1178176024 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec94152890 msgr2=0x7fec940c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:38.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.737+0000 7fec9a71e640 1 --2- 172.21.15.161:0/1178176024 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec94152890 0x7fec940c8990 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fec940b0c00 tx=0x7fec8802b7f0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.737+0000 7fec9a71e640 1 -- 172.21.15.161:0/1178176024 shutdown_connections 2024-03-28T23:33:38.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.737+0000 7fec9a71e640 1 --2- 172.21.15.161:0/1178176024 >> v2:172.21.15.170:3300/0 conn(0x7fec940c5950 0x7fec941553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.737+0000 7fec9a71e640 1 --2- 172.21.15.161:0/1178176024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec940c8ed0 0x7fec940c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.737+0000 7fec9a71e640 1 --2- 172.21.15.161:0/1178176024 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec94152890 0x7fec940c8990 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.737+0000 7fec9a71e640 1 -- 172.21.15.161:0/1178176024 >> 172.21.15.161:0/1178176024 conn(0x7fec940ac620 msgr2=0x7fec940aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:38.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.737+0000 7fec9a71e640 1 -- 172.21.15.161:0/1178176024 shutdown_connections 2024-03-28T23:33:38.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.737+0000 7fec9a71e640 1 -- 172.21.15.161:0/1178176024 wait complete. 2024-03-28T23:33:38.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.738+0000 7fec9a71e640 1 Processor -- start 2024-03-28T23:33:38.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.738+0000 7fec9a71e640 1 -- start start 2024-03-28T23:33:38.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.739+0000 7fec9a71e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec940c5950 0x7fec940c49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:38.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.739+0000 7fec9a71e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec940c8ed0 0x7fec940ab900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:38.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.739+0000 7fec9a71e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec94152890 0x7fec941e7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:38.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.739+0000 7fec9a71e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fec94152c70 con 0x7fec940c8ed0 2024-03-28T23:33:38.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.739+0000 7fec9a71e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fec941592d0 con 0x7fec94152890 2024-03-28T23:33:38.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.739+0000 7fec9a71e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fec94152df0 con 0x7fec940c5950 2024-03-28T23:33:38.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.739+0000 7fec937fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec940c8ed0 0x7fec940ab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:38.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.739+0000 7fec937fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec940c8ed0 0x7fec940ab900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54420/0 (socket says 172.21.15.161:54420) 2024-03-28T23:33:38.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.739+0000 7fec937fe640 1 -- 172.21.15.161:0/1467485677 learned_addr learned my addr 172.21.15.161:0/1467485677 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:38.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.739+0000 7fec93fff640 1 --2- 172.21.15.161:0/1467485677 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec940c5950 0x7fec940c49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:38.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.739+0000 7fec98cbb640 1 --2- 172.21.15.161:0/1467485677 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec94152890 0x7fec941e7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:38.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.740+0000 7fec937fe640 1 -- 172.21.15.161:0/1467485677 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec940c5950 msgr2=0x7fec940c49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:38.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.740+0000 7fec937fe640 1 --2- 172.21.15.161:0/1467485677 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec940c5950 0x7fec940c49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.740+0000 7fec937fe640 1 -- 172.21.15.161:0/1467485677 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec94152890 msgr2=0x7fec941e7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:38.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.740+0000 7fec937fe640 1 --2- 172.21.15.161:0/1467485677 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec94152890 0x7fec941e7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.740+0000 7fec937fe640 1 -- 172.21.15.161:0/1467485677 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec941eb800 con 0x7fec940c8ed0 2024-03-28T23:33:38.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.740+0000 7fec93fff640 1 --2- 172.21.15.161:0/1467485677 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec940c5950 0x7fec940c49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:38.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.740+0000 7fec937fe640 1 --2- 172.21.15.161:0/1467485677 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec940c8ed0 0x7fec940ab900 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fec8400e840 tx=0x7fec8400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:38.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.740+0000 7fec90ff9640 1 -- 172.21.15.161:0/1467485677 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec84017070 con 0x7fec940c8ed0 2024-03-28T23:33:38.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.740+0000 7fec90ff9640 1 -- 172.21.15.161:0/1467485677 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec8400c940 con 0x7fec940c8ed0 2024-03-28T23:33:38.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.740+0000 7fec90ff9640 1 -- 172.21.15.161:0/1467485677 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec8400cd10 con 0x7fec940c8ed0 2024-03-28T23:33:38.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.740+0000 7fec9a71e640 1 -- 172.21.15.161:0/1467485677 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec9414a7e0 con 0x7fec940c8ed0 2024-03-28T23:33:38.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.740+0000 7fec9a71e640 1 -- 172.21.15.161:0/1467485677 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec940c4400 con 0x7fec940c8ed0 2024-03-28T23:33:38.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.741+0000 7fec9a71e640 1 -- 172.21.15.161:0/1467485677 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec9404f4b0 con 0x7fec940c8ed0 2024-03-28T23:33:38.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.741+0000 7fec90ff9640 1 -- 172.21.15.161:0/1467485677 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fec8401a070 con 0x7fec940c8ed0 2024-03-28T23:33:38.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.742+0000 7fec90ff9640 1 --2- 172.21.15.161:0/1467485677 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fec8006b4d0 0x7fec8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:38.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.742+0000 7fec90ff9640 1 -- 172.21.15.161:0/1467485677 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6245+0+0 (secure 0 0 0) 0x7fec8408a650 con 0x7fec940c8ed0 2024-03-28T23:33:38.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.742+0000 7fec93fff640 1 --2- 172.21.15.161:0/1467485677 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fec8006b4d0 0x7fec8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:38.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.742+0000 7fec93fff640 1 --2- 172.21.15.161:0/1467485677 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fec8006b4d0 0x7fec8006d990 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fec88035000 tx=0x7fec88008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:38.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.746+0000 7fec90ff9640 1 -- 172.21.15.161:0/1467485677 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fec84058cf0 con 0x7fec940c8ed0 2024-03-28T23:33:38.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.953+0000 7fec9a71e640 1 -- 172.21.15.161:0/1467485677 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec940c8140 con 0x7fec940c8ed0 2024-03-28T23:33:38.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.954+0000 7fec90ff9640 1 -- 172.21.15.161:0/1467485677 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) ==== 94+0+5192 (secure 0 0 0) 0x7fec8405cd90 con 0x7fec940c8ed0 2024-03-28T23:33:38.955 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:38.955 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":84,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6547,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4280217233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4280217233},{"type":"v1","addr":"172.21.15.161:6836","nonce":4280217233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":6706,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/2647418289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":2647418289},{"type":"v1","addr":"172.21.15.170:6834","nonce":2647418289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":7086,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/404132405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":404132405},{"type":"v1","addr":"172.21.15.161:6837","nonce":404132405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:33:23.516182+0000","modified":"2024-03-28T23:33:34.430703+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5963},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5963":{"gid":5963,"name":"b","rank":0,"incarnation":81,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6835/3079114491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3079114491},{"type":"v1","addr":"172.21.15.170:6835","nonce":3079114491}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5963,"qdb_cluster":[5963]},"id":7}]} 2024-03-28T23:33:38.955 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 84 2024-03-28T23:33:38.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.955+0000 7fec9a71e640 1 -- 172.21.15.161:0/1467485677 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fec8006b4d0 msgr2=0x7fec8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:38.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.955+0000 7fec9a71e640 1 --2- 172.21.15.161:0/1467485677 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fec8006b4d0 0x7fec8006d990 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fec88035000 tx=0x7fec88008030 comp rx=0 tx=0).stop 2024-03-28T23:33:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.955+0000 7fec9a71e640 1 -- 172.21.15.161:0/1467485677 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec940c8ed0 msgr2=0x7fec940ab900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.955+0000 7fec9a71e640 1 --2- 172.21.15.161:0/1467485677 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec940c8ed0 0x7fec940ab900 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fec8400e840 tx=0x7fec8400ed00 comp rx=0 tx=0).stop 2024-03-28T23:33:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.956+0000 7fec9a71e640 1 -- 172.21.15.161:0/1467485677 shutdown_connections 2024-03-28T23:33:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.956+0000 7fec9a71e640 1 --2- 172.21.15.161:0/1467485677 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fec8006b4d0 0x7fec8006d990 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.956+0000 7fec9a71e640 1 --2- 172.21.15.161:0/1467485677 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec940c8ed0 0x7fec940ab900 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.956+0000 7fec9a71e640 1 --2- 172.21.15.161:0/1467485677 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec94152890 0x7fec941e7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.956+0000 7fec9a71e640 1 --2- 172.21.15.161:0/1467485677 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec940c5950 0x7fec940c49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.956+0000 7fec9a71e640 1 -- 172.21.15.161:0/1467485677 >> 172.21.15.161:0/1467485677 conn(0x7fec940ac620 msgr2=0x7fec94151560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.956+0000 7fec9a71e640 1 -- 172.21.15.161:0/1467485677 shutdown_connections 2024-03-28T23:33:38.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:38.956+0000 7fec9a71e640 1 -- 172.21.15.161:0/1467485677 wait complete. 2024-03-28T23:33:38.968 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:33:39.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.068+0000 7f1b28055640 1 Processor -- start 2024-03-28T23:33:39.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.069+0000 7f1b28055640 1 -- start start 2024-03-28T23:33:39.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.069+0000 7f1b28055640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1b20152890 0x7f1b200c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:39.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.069+0000 7f1b28055640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b20158a30 con 0x7f1b200c8ed0 2024-03-28T23:33:39.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.069+0000 7f1b28055640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b20158d10 con 0x7f1b200c5b20 2024-03-28T23:33:39.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.069+0000 7f1b28055640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f1b200c8480 con 0x7f1b20152890 2024-03-28T23:33:39.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.069+0000 7f1b255f0640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f1b200c8ed0 0x7f1b200c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:53020/0 (socket says 172.21.15.161:53020) 2024-03-28T23:33:39.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.069+0000 7f1b255f0640 1 -- 172.21.15.161:0/1591740068 learned_addr learned my addr 172.21.15.161:0/1591740068 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:39.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.069+0000 7f1b25df1640 1 --2- 172.21.15.161:0/1591740068 >> v2:172.21.15.170:3300/0 conn(0x7f1b20152890 0x7f1b200c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:39.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.070+0000 7f1b24def640 1 -- 172.21.15.161:0/1591740068 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1041453477 0 0) 0x7f1b20158a30 con 0x7f1b200c8ed0 2024-03-28T23:33:39.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.070+0000 7f1b24def640 1 -- 172.21.15.161:0/1591740068 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b08003520 con 0x7f1b200c8ed0 2024-03-28T23:33:39.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.070+0000 7f1b24def640 1 -- 172.21.15.161:0/1591740068 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1779256434 0 0) 0x7f1b20158d10 con 0x7f1b200c5b20 2024-03-28T23:33:39.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.070+0000 7f1b24def640 1 -- 172.21.15.161:0/1591740068 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b20158a30 con 0x7f1b200c5b20 2024-03-28T23:33:39.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.070+0000 7f1b25df1640 1 -- 172.21.15.161:0/1591740068 >> v1:172.21.15.170:6789/0 conn(0x7f1b200c5b20 legacy=0x7f1b201553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:39.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.070+0000 7f1b25df1640 1 -- 172.21.15.161:0/1591740068 >> v1:172.21.15.161:6789/0 conn(0x7f1b200c8ed0 legacy=0x7f1b200c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:39.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.070+0000 7f1b25df1640 1 -- 172.21.15.161:0/1591740068 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b20158ff0 con 0x7f1b20152890 2024-03-28T23:33:39.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.071+0000 7f1b25df1640 1 --2- 172.21.15.161:0/1591740068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b20152890 0x7f1b200c8990 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1b10009d90 tx=0x7f1b1002be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdd84d89b03729e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:39.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.071+0000 7f1b24def640 1 -- 172.21.15.161:0/1591740068 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b10039070 con 0x7f1b20152890 2024-03-28T23:33:39.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.071+0000 7f1b24def640 1 -- 172.21.15.161:0/1591740068 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b1002cbd0 con 0x7f1b20152890 2024-03-28T23:33:39.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.071+0000 7f1b24def640 1 -- 172.21.15.161:0/1591740068 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b10034440 con 0x7f1b20152890 2024-03-28T23:33:39.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.071+0000 7f1b28055640 1 -- 172.21.15.161:0/1591740068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b20152890 msgr2=0x7f1b200c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:39.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.071+0000 7f1b28055640 1 --2- 172.21.15.161:0/1591740068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b20152890 0x7f1b200c8990 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1b10009d90 tx=0x7f1b1002be20 comp rx=0 tx=0).stop 2024-03-28T23:33:39.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.071+0000 7f1b28055640 1 -- 172.21.15.161:0/1591740068 shutdown_connections 2024-03-28T23:33:39.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.071+0000 7f1b28055640 1 --2- 172.21.15.161:0/1591740068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b20152890 0x7f1b200c8990 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.071+0000 7f1b28055640 1 -- 172.21.15.161:0/1591740068 >> 172.21.15.161:0/1591740068 conn(0x7f1b200ac620 msgr2=0x7f1b200aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:39.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.071+0000 7f1b28055640 1 -- 172.21.15.161:0/1591740068 shutdown_connections 2024-03-28T23:33:39.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.071+0000 7f1b28055640 1 -- 172.21.15.161:0/1591740068 wait complete. 2024-03-28T23:33:39.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.072+0000 7f1b28055640 1 Processor -- start 2024-03-28T23:33:39.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.072+0000 7f1b28055640 1 -- start start 2024-03-28T23:33:39.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.073+0000 7f1b28055640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b200c5b20 0x7f1b201480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:39.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.073+0000 7f1b28055640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b200c8ed0 0x7f1b2014ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:39.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.073+0000 7f1b28055640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b20152890 0x7f1b201eb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:39.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.073+0000 7f1b28055640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1b20152c70 con 0x7f1b200c5b20 2024-03-28T23:33:39.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.073+0000 7f1b28055640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1b20159640 con 0x7f1b200c8ed0 2024-03-28T23:33:39.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.073+0000 7f1b28055640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1b20152df0 con 0x7f1b20152890 2024-03-28T23:33:39.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.073+0000 7f1b25df1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b200c5b20 0x7f1b201480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:39.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.073+0000 7f1b25df1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b200c5b20 0x7f1b201480f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54430/0 (socket says 172.21.15.161:54430) 2024-03-28T23:33:39.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.073+0000 7f1b25df1640 1 -- 172.21.15.161:0/1428222952 learned_addr learned my addr 172.21.15.161:0/1428222952 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:39.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.073+0000 7f1b255f0640 1 --2- 172.21.15.161:0/1428222952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b200c8ed0 0x7f1b2014ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:39.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.073+0000 7f1b265f2640 1 --2- 172.21.15.161:0/1428222952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b20152890 0x7f1b201eb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:39.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.074+0000 7f1b25df1640 1 -- 172.21.15.161:0/1428222952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b20152890 msgr2=0x7f1b201eb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:39.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.074+0000 7f1b25df1640 1 --2- 172.21.15.161:0/1428222952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b20152890 0x7f1b201eb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.074+0000 7f1b25df1640 1 -- 172.21.15.161:0/1428222952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b200c8ed0 msgr2=0x7f1b2014ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:39.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.074+0000 7f1b25df1640 1 --2- 172.21.15.161:0/1428222952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b200c8ed0 0x7f1b2014ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.074+0000 7f1b25df1640 1 -- 172.21.15.161:0/1428222952 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b2014eb50 con 0x7f1b200c5b20 2024-03-28T23:33:39.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.074+0000 7f1b25df1640 1 --2- 172.21.15.161:0/1428222952 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b200c5b20 0x7f1b201480f0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f1b10009d90 tx=0x7f1b10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:39.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.074+0000 7f1b1e7fc640 1 -- 172.21.15.161:0/1428222952 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b10039050 con 0x7f1b200c5b20 2024-03-28T23:33:39.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.074+0000 7f1b1e7fc640 1 -- 172.21.15.161:0/1428222952 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b10045070 con 0x7f1b200c5b20 2024-03-28T23:33:39.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.074+0000 7f1b28055640 1 -- 172.21.15.161:0/1428222952 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b20148630 con 0x7f1b200c5b20 2024-03-28T23:33:39.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.074+0000 7f1b1e7fc640 1 -- 172.21.15.161:0/1428222952 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1003e830 con 0x7f1b200c5b20 2024-03-28T23:33:39.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.074+0000 7f1b28055640 1 -- 172.21.15.161:0/1428222952 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b200c1e40 con 0x7f1b200c5b20 2024-03-28T23:33:39.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.075+0000 7f1b28055640 1 -- 172.21.15.161:0/1428222952 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b2004f4b0 con 0x7f1b200c5b20 2024-03-28T23:33:39.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.076+0000 7f1b1e7fc640 1 -- 172.21.15.161:0/1428222952 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1b1003e9d0 con 0x7f1b200c5b20 2024-03-28T23:33:39.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.076+0000 7f1b1e7fc640 1 --2- 172.21.15.161:0/1428222952 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1b0c06b4d0 0x7f1b0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:39.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.076+0000 7f1b1e7fc640 1 -- 172.21.15.161:0/1428222952 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6245+0+0 (secure 0 0 0) 0x7f1b100acfa0 con 0x7f1b200c5b20 2024-03-28T23:33:39.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.076+0000 7f1b255f0640 1 --2- 172.21.15.161:0/1428222952 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1b0c06b4d0 0x7f1b0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:39.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.076+0000 7f1b255f0640 1 --2- 172.21.15.161:0/1428222952 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1b0c06b4d0 0x7f1b0c06d990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1b200ac550 tx=0x7f1b14009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:39.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.080+0000 7f1b1e7fc640 1 -- 172.21.15.161:0/1428222952 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1b2014eb50 con 0x7f1b200c5b20 2024-03-28T23:33:39.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.290+0000 7f1b28055640 1 -- 172.21.15.161:0/1428222952 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b200c16d0 con 0x7f1b200c5b20 2024-03-28T23:33:39.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.290+0000 7f1b1e7fc640 1 -- 172.21.15.161:0/1428222952 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) ==== 94+0+5192 (secure 0 0 0) 0x7f1b1007b760 con 0x7f1b200c5b20 2024-03-28T23:33:39.291 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:39.291 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":84,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6547,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4280217233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4280217233},{"type":"v1","addr":"172.21.15.161:6836","nonce":4280217233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":6706,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/2647418289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":2647418289},{"type":"v1","addr":"172.21.15.170:6834","nonce":2647418289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":7086,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/404132405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":404132405},{"type":"v1","addr":"172.21.15.161:6837","nonce":404132405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:33:23.516182+0000","modified":"2024-03-28T23:33:34.430703+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5963},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5963":{"gid":5963,"name":"b","rank":0,"incarnation":81,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6835/3079114491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3079114491},{"type":"v1","addr":"172.21.15.170:6835","nonce":3079114491}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5963,"qdb_cluster":[5963]},"id":7}]} 2024-03-28T23:33:39.291 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 84 2024-03-28T23:33:39.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.292+0000 7f1b28055640 1 -- 172.21.15.161:0/1428222952 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1b0c06b4d0 msgr2=0x7f1b0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:39.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.292+0000 7f1b28055640 1 --2- 172.21.15.161:0/1428222952 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1b0c06b4d0 0x7f1b0c06d990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1b200ac550 tx=0x7f1b14009040 comp rx=0 tx=0).stop 2024-03-28T23:33:39.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.292+0000 7f1b28055640 1 -- 172.21.15.161:0/1428222952 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b200c5b20 msgr2=0x7f1b201480f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:39.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.292+0000 7f1b28055640 1 --2- 172.21.15.161:0/1428222952 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b200c5b20 0x7f1b201480f0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f1b10009d90 tx=0x7f1b10004a40 comp rx=0 tx=0).stop 2024-03-28T23:33:39.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.292+0000 7f1b28055640 1 -- 172.21.15.161:0/1428222952 shutdown_connections 2024-03-28T23:33:39.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.292+0000 7f1b28055640 1 --2- 172.21.15.161:0/1428222952 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1b0c06b4d0 0x7f1b0c06d990 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.292+0000 7f1b28055640 1 --2- 172.21.15.161:0/1428222952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b20152890 0x7f1b201eb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.292+0000 7f1b28055640 1 --2- 172.21.15.161:0/1428222952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b200c8ed0 0x7f1b2014ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.292+0000 7f1b28055640 1 --2- 172.21.15.161:0/1428222952 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b200c5b20 0x7f1b201480f0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.292+0000 7f1b28055640 1 -- 172.21.15.161:0/1428222952 >> 172.21.15.161:0/1428222952 conn(0x7f1b200ac620 msgr2=0x7f1b2014c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:39.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.292+0000 7f1b28055640 1 -- 172.21.15.161:0/1428222952 shutdown_connections 2024-03-28T23:33:39.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.292+0000 7f1b28055640 1 -- 172.21.15.161:0/1428222952 wait complete. 2024-03-28T23:33:39.304 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:33:39.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.405+0000 7fb36388f640 1 Processor -- start 2024-03-28T23:33:39.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.406+0000 7fb36388f640 1 -- start start 2024-03-28T23:33:39.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.406+0000 7fb36388f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb35c0c5410 0x7fb35c0c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:39.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.406+0000 7fb36388f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb35c1553a0 0x7fb35c155850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:39.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.406+0000 7fb36388f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb35c158a30 con 0x7fb35c152890 2024-03-28T23:33:39.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.406+0000 7fb36388f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb35c0c8480 con 0x7fb35c0c5410 2024-03-28T23:33:39.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.406+0000 7fb36388f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fb35c0c5d90 con 0x7fb35c1553a0 2024-03-28T23:33:39.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.406+0000 7fb360e2a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb35c0c5410 0x7fb35c0c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:39.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.406+0000 7fb360e2a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb35c0c5410 0x7fb35c0c5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54444/0 (socket says 172.21.15.161:54444) 2024-03-28T23:33:39.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.406+0000 7fb360e2a640 1 -- 172.21.15.161:0/2302484025 learned_addr learned my addr 172.21.15.161:0/2302484025 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:39.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.406+0000 7fb361e2c640 1 --2- 172.21.15.161:0/2302484025 >> v2:172.21.15.170:3300/0 conn(0x7fb35c1553a0 0x7fb35c155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:39.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.407+0000 7fb360e2a640 1 -- 172.21.15.161:0/2302484025 >> v2:172.21.15.170:3300/0 conn(0x7fb35c1553a0 msgr2=0x7fb35c155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:39.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.407+0000 7fb360e2a640 1 --2- 172.21.15.161:0/2302484025 >> v2:172.21.15.170:3300/0 conn(0x7fb35c1553a0 0x7fb35c155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.407+0000 7fb360e2a640 1 -- 172.21.15.161:0/2302484025 >> v1:172.21.15.161:6789/0 conn(0x7fb35c152890 legacy=0x7fb35c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:39.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.407+0000 7fb360e2a640 1 -- 172.21.15.161:0/2302484025 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb35c158d10 con 0x7fb35c0c5410 2024-03-28T23:33:39.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.407+0000 7fb360e2a640 1 --2- 172.21.15.161:0/2302484025 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb35c0c5410 0x7fb35c0c5850 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fb35c0aa310 tx=0x7fb34c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82d2cb112edab7a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:39.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.408+0000 7fb353fff640 1 -- 172.21.15.161:0/2302484025 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c02d040 con 0x7fb35c0c5410 2024-03-28T23:33:39.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.408+0000 7fb353fff640 1 -- 172.21.15.161:0/2302484025 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb34c004230 con 0x7fb35c0c5410 2024-03-28T23:33:39.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.408+0000 7fb353fff640 1 -- 172.21.15.161:0/2302484025 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c003610 con 0x7fb35c0c5410 2024-03-28T23:33:39.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.408+0000 7fb36388f640 1 -- 172.21.15.161:0/2302484025 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb35c0c5410 msgr2=0x7fb35c0c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:39.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.408+0000 7fb36388f640 1 --2- 172.21.15.161:0/2302484025 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb35c0c5410 0x7fb35c0c5850 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fb35c0aa310 tx=0x7fb34c02b8f0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.408+0000 7fb36388f640 1 -- 172.21.15.161:0/2302484025 shutdown_connections 2024-03-28T23:33:39.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.408+0000 7fb36388f640 1 --2- 172.21.15.161:0/2302484025 >> v2:172.21.15.170:3300/0 conn(0x7fb35c1553a0 0x7fb35c155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.408+0000 7fb36388f640 1 --2- 172.21.15.161:0/2302484025 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb35c0c5410 0x7fb35c0c5850 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.408+0000 7fb36388f640 1 -- 172.21.15.161:0/2302484025 >> 172.21.15.161:0/2302484025 conn(0x7fb35c0ac620 msgr2=0x7fb35c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:39.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.408+0000 7fb36388f640 1 -- 172.21.15.161:0/2302484025 shutdown_connections 2024-03-28T23:33:39.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.408+0000 7fb36388f640 1 -- 172.21.15.161:0/2302484025 wait complete. 2024-03-28T23:33:39.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.409+0000 7fb36388f640 1 Processor -- start 2024-03-28T23:33:39.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.409+0000 7fb36388f640 1 -- start start 2024-03-28T23:33:39.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.410+0000 7fb36388f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb35c0c5410 0x7fb35c1eb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:39.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.410+0000 7fb36388f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb35c152890 0x7fb35c0ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:39.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.410+0000 7fb36388f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb35c1553a0 0x7fb35c14cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:39.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.410+0000 7fb36388f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c1592d0 con 0x7fb35c0c5410 2024-03-28T23:33:39.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.410+0000 7fb36388f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c159640 con 0x7fb35c1553a0 2024-03-28T23:33:39.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.410+0000 7fb36388f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb35c0c90a0 con 0x7fb35c152890 2024-03-28T23:33:39.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.410+0000 7fb36162b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb35c0c5410 0x7fb35c1eb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:39.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.410+0000 7fb36162b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb35c0c5410 0x7fb35c1eb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54456/0 (socket says 172.21.15.161:54456) 2024-03-28T23:33:39.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.410+0000 7fb36162b640 1 -- 172.21.15.161:0/2105752526 learned_addr learned my addr 172.21.15.161:0/2105752526 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:39.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.410+0000 7fb360e2a640 1 --2- 172.21.15.161:0/2105752526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb35c152890 0x7fb35c0ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:39.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.410+0000 7fb36162b640 1 -- 172.21.15.161:0/2105752526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb35c152890 msgr2=0x7fb35c0ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:39.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.410+0000 7fb36162b640 1 --2- 172.21.15.161:0/2105752526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb35c152890 0x7fb35c0ab6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.410+0000 7fb36162b640 1 -- 172.21.15.161:0/2105752526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb35c1553a0 msgr2=0x7fb35c14cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:39.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.410+0000 7fb36162b640 1 --2- 172.21.15.161:0/2105752526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb35c1553a0 0x7fb35c14cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.410+0000 7fb36162b640 1 -- 172.21.15.161:0/2105752526 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb35c1ebc60 con 0x7fb35c0c5410 2024-03-28T23:33:39.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.411+0000 7fb36162b640 1 --2- 172.21.15.161:0/2105752526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb35c0c5410 0x7fb35c1eb720 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fb35800c400 tx=0x7fb35800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:39.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.411+0000 7fb351ffb640 1 -- 172.21.15.161:0/2105752526 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3580052d0 con 0x7fb35c0c5410 2024-03-28T23:33:39.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.411+0000 7fb351ffb640 1 -- 172.21.15.161:0/2105752526 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb358005490 con 0x7fb35c0c5410 2024-03-28T23:33:39.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.411+0000 7fb351ffb640 1 -- 172.21.15.161:0/2105752526 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb358007260 con 0x7fb35c0c5410 2024-03-28T23:33:39.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.411+0000 7fb36388f640 1 -- 172.21.15.161:0/2105752526 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb35c157f10 con 0x7fb35c0c5410 2024-03-28T23:33:39.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.411+0000 7fb36388f640 1 -- 172.21.15.161:0/2105752526 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb35c156c30 con 0x7fb35c0c5410 2024-03-28T23:33:39.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.412+0000 7fb36388f640 1 -- 172.21.15.161:0/2105752526 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb35c156770 con 0x7fb35c0c5410 2024-03-28T23:33:39.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.413+0000 7fb351ffb640 1 -- 172.21.15.161:0/2105752526 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb358007400 con 0x7fb35c0c5410 2024-03-28T23:33:39.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.413+0000 7fb351ffb640 1 --2- 172.21.15.161:0/2105752526 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb34006b4d0 0x7fb34006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:39.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.413+0000 7fb351ffb640 1 -- 172.21.15.161:0/2105752526 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6245+0+0 (secure 0 0 0) 0x7fb3580897a0 con 0x7fb35c0c5410 2024-03-28T23:33:39.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.413+0000 7fb360e2a640 1 --2- 172.21.15.161:0/2105752526 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb34006b4d0 0x7fb34006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:39.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.414+0000 7fb360e2a640 1 --2- 172.21.15.161:0/2105752526 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb34006b4d0 0x7fb34006d990 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb34c035000 tx=0x7fb34c033040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:39.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.417+0000 7fb351ffb640 1 -- 172.21.15.161:0/2105752526 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb35c156770 con 0x7fb35c0c5410 2024-03-28T23:33:39.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.593+0000 7fb36388f640 1 -- 172.21.15.161:0/2105752526 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb35c04f4b0 con 0x7fb35c0c5410 2024-03-28T23:33:39.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.594+0000 7fb351ffb640 1 -- 172.21.15.161:0/2105752526 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+17138 (secure 0 0 0) 0x7fb35c1ebc60 con 0x7fb35c0c5410 2024-03-28T23:33:39.595 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:39.595 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":97,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:33:35.435937+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:33:21.732337+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-28T23:33:22.477357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.1d","mappings":[{"from":1,"to":7}]},{"pgid":"16.25","mappings":[{"from":5,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6834/2176248563":"2024-03-29T23:33:31.422647+0000","172.21.15.170:6832/2176248563":"2024-03-29T23:33:31.422647+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:33:39.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.596+0000 7fb36388f640 1 -- 172.21.15.161:0/2105752526 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb34006b4d0 msgr2=0x7fb34006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:39.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.596+0000 7fb36388f640 1 --2- 172.21.15.161:0/2105752526 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb34006b4d0 0x7fb34006d990 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb34c035000 tx=0x7fb34c033040 comp rx=0 tx=0).stop 2024-03-28T23:33:39.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.596+0000 7fb36388f640 1 -- 172.21.15.161:0/2105752526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb35c0c5410 msgr2=0x7fb35c1eb720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:39.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.596+0000 7fb36388f640 1 --2- 172.21.15.161:0/2105752526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb35c0c5410 0x7fb35c1eb720 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fb35800c400 tx=0x7fb35800c8c0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.596+0000 7fb36388f640 1 -- 172.21.15.161:0/2105752526 shutdown_connections 2024-03-28T23:33:39.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.596+0000 7fb36388f640 1 --2- 172.21.15.161:0/2105752526 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb34006b4d0 0x7fb34006d990 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.596+0000 7fb36388f640 1 --2- 172.21.15.161:0/2105752526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb35c1553a0 0x7fb35c14cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.596+0000 7fb36388f640 1 --2- 172.21.15.161:0/2105752526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb35c152890 0x7fb35c0ab6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.596+0000 7fb36388f640 1 --2- 172.21.15.161:0/2105752526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb35c0c5410 0x7fb35c1eb720 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.596+0000 7fb36388f640 1 -- 172.21.15.161:0/2105752526 >> 172.21.15.161:0/2105752526 conn(0x7fb35c0ac620 msgr2=0x7fb35c06be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:39.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.596+0000 7fb36388f640 1 -- 172.21.15.161:0/2105752526 shutdown_connections 2024-03-28T23:33:39.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.596+0000 7fb36388f640 1 -- 172.21.15.161:0/2105752526 wait complete. 2024-03-28T23:33:39.608 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-28T23:33:39.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.710+0000 7f1ff638a640 1 Processor -- start 2024-03-28T23:33:39.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.710+0000 7f1ff638a640 1 -- start start 2024-03-28T23:33:39.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.710+0000 7f1ff638a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ff006ce80 0x7f1ff00a9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:39.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.710+0000 7f1ff638a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ff006d310 con 0x7f1ff014eca0 2024-03-28T23:33:39.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.710+0000 7f1ff638a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ff0151850 con 0x7f1ff006ca20 2024-03-28T23:33:39.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.710+0000 7f1ff638a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1ff0152460 con 0x7f1ff006ce80 2024-03-28T23:33:39.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.711+0000 7f1feffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f1ff014eca0 0x7f1ff00c1880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:53036/0 (socket says 172.21.15.161:53036) 2024-03-28T23:33:39.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.711+0000 7f1feffff640 1 -- 172.21.15.161:0/844736216 learned_addr learned my addr 172.21.15.161:0/844736216 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:39.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.711+0000 7f1ff4927640 1 --2- 172.21.15.161:0/844736216 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ff006ce80 0x7f1ff00a9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:39.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.711+0000 7f1feeffd640 1 -- 172.21.15.161:0/844736216 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2938950871 0 0) 0x7f1fe0001110 con 0x7f1ff014eca0 2024-03-28T23:33:39.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.711+0000 7f1feeffd640 1 -- 172.21.15.161:0/844736216 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fd8003520 con 0x7f1ff014eca0 2024-03-28T23:33:39.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.711+0000 7f1feeffd640 1 -- 172.21.15.161:0/844736216 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 88304797 0 0) 0x7f1ff0151850 con 0x7f1ff006ca20 2024-03-28T23:33:39.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.711+0000 7f1feeffd640 1 -- 172.21.15.161:0/844736216 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fe0001110 con 0x7f1ff006ca20 2024-03-28T23:33:39.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.711+0000 7f1ff4927640 1 -- 172.21.15.161:0/844736216 >> v1:172.21.15.170:6789/0 conn(0x7f1ff006ca20 legacy=0x7f1ff00a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:39.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.711+0000 7f1ff4927640 1 -- 172.21.15.161:0/844736216 >> v1:172.21.15.161:6789/0 conn(0x7f1ff014eca0 legacy=0x7f1ff00c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:39.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.711+0000 7f1ff4927640 1 -- 172.21.15.161:0/844736216 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ff0151b30 con 0x7f1ff006ce80 2024-03-28T23:33:39.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.712+0000 7f1ff4927640 1 --2- 172.21.15.161:0/844736216 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ff006ce80 0x7f1ff00a9fd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1fe800aed0 tx=0x7f1fe802cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=559fa56a71ccaeb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:39.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.712+0000 7f1feeffd640 1 -- 172.21.15.161:0/844736216 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe802db30 con 0x7f1ff006ce80 2024-03-28T23:33:39.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.712+0000 7f1feeffd640 1 -- 172.21.15.161:0/844736216 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fe802dcd0 con 0x7f1ff006ce80 2024-03-28T23:33:39.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.712+0000 7f1feeffd640 1 -- 172.21.15.161:0/844736216 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8008660 con 0x7f1ff006ce80 2024-03-28T23:33:39.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.712+0000 7f1ff638a640 1 -- 172.21.15.161:0/844736216 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ff006ce80 msgr2=0x7f1ff00a9fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:39.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.712+0000 7f1ff638a640 1 --2- 172.21.15.161:0/844736216 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ff006ce80 0x7f1ff00a9fd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1fe800aed0 tx=0x7f1fe802cf90 comp rx=0 tx=0).stop 2024-03-28T23:33:39.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.712+0000 7f1ff638a640 1 -- 172.21.15.161:0/844736216 shutdown_connections 2024-03-28T23:33:39.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.712+0000 7f1ff638a640 1 --2- 172.21.15.161:0/844736216 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ff006ce80 0x7f1ff00a9fd0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.712+0000 7f1ff638a640 1 -- 172.21.15.161:0/844736216 >> 172.21.15.161:0/844736216 conn(0x7f1ff00c6670 msgr2=0x7f1ff00c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:39.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.713+0000 7f1ff638a640 1 -- 172.21.15.161:0/844736216 shutdown_connections 2024-03-28T23:33:39.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.713+0000 7f1ff638a640 1 -- 172.21.15.161:0/844736216 wait complete. 2024-03-28T23:33:39.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.714+0000 7f1ff638a640 1 Processor -- start 2024-03-28T23:33:39.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.714+0000 7f1ff638a640 1 -- start start 2024-03-28T23:33:39.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.714+0000 7f1ff638a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ff006ca20 0x7f1ff0145420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:39.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.714+0000 7f1ff638a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ff006ce80 0x7f1ff014b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:39.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.714+0000 7f1ff638a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ff014eca0 0x7f1ff014d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:39.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.714+0000 7f1ff638a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff00aa510 con 0x7f1ff006ce80 2024-03-28T23:33:39.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.714+0000 7f1ff638a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff01520f0 con 0x7f1ff006ca20 2024-03-28T23:33:39.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.714+0000 7f1ff638a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1ff014f080 con 0x7f1ff014eca0 2024-03-28T23:33:39.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.714+0000 7f1fef7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ff006ce80 0x7f1ff014b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:39.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.715+0000 7f1fef7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ff006ce80 0x7f1ff014b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54460/0 (socket says 172.21.15.161:54460) 2024-03-28T23:33:39.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.715+0000 7f1fef7fe640 1 -- 172.21.15.161:0/2468065459 learned_addr learned my addr 172.21.15.161:0/2468065459 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:39.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.715+0000 7f1ff4927640 1 --2- 172.21.15.161:0/2468065459 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ff014eca0 0x7f1ff014d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:39.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.715+0000 7f1feffff640 1 --2- 172.21.15.161:0/2468065459 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ff006ca20 0x7f1ff0145420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:39.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.715+0000 7f1fef7fe640 1 -- 172.21.15.161:0/2468065459 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ff014eca0 msgr2=0x7f1ff014d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:39.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.715+0000 7f1fef7fe640 1 --2- 172.21.15.161:0/2468065459 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ff014eca0 0x7f1ff014d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.715+0000 7f1fef7fe640 1 -- 172.21.15.161:0/2468065459 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ff006ca20 msgr2=0x7f1ff0145420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:39.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.715+0000 7f1fef7fe640 1 --2- 172.21.15.161:0/2468065459 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ff006ca20 0x7f1ff0145420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.715+0000 7f1fef7fe640 1 -- 172.21.15.161:0/2468065459 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ff01e9b90 con 0x7f1ff006ce80 2024-03-28T23:33:39.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.715+0000 7f1fef7fe640 1 --2- 172.21.15.161:0/2468065459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ff006ce80 0x7f1ff014b640 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400c430 tx=0x7f1fe400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:39.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.715+0000 7f1fecff9640 1 -- 172.21.15.161:0/2468065459 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4017070 con 0x7f1ff006ce80 2024-03-28T23:33:39.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.715+0000 7f1fecff9640 1 -- 172.21.15.161:0/2468065459 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fe40052f0 con 0x7f1ff006ce80 2024-03-28T23:33:39.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.715+0000 7f1ff638a640 1 -- 172.21.15.161:0/2468065459 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ff01e8c00 con 0x7f1ff006ce80 2024-03-28T23:33:39.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.715+0000 7f1fecff9640 1 -- 172.21.15.161:0/2468065459 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4003830 con 0x7f1ff006ce80 2024-03-28T23:33:39.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.715+0000 7f1ff638a640 1 -- 172.21.15.161:0/2468065459 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ff01e91c0 con 0x7f1ff006ce80 2024-03-28T23:33:39.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.716+0000 7f1ff638a640 1 -- 172.21.15.161:0/2468065459 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ff004f4b0 con 0x7f1ff006ce80 2024-03-28T23:33:39.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.717+0000 7f1fecff9640 1 -- 172.21.15.161:0/2468065459 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1fe401a070 con 0x7f1ff006ce80 2024-03-28T23:33:39.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.717+0000 7f1fecff9640 1 --2- 172.21.15.161:0/2468065459 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1fdc06b4d0 0x7f1fdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:39.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.717+0000 7f1feffff640 1 --2- 172.21.15.161:0/2468065459 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1fdc06b4d0 0x7f1fdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:39.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.717+0000 7f1fecff9640 1 -- 172.21.15.161:0/2468065459 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6245+0+0 (secure 0 0 0) 0x7f1fe40896c0 con 0x7f1ff006ce80 2024-03-28T23:33:39.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.718+0000 7f1feffff640 1 --2- 172.21.15.161:0/2468065459 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1fdc06b4d0 0x7f1fdc06d990 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f1fe00013b0 tx=0x7f1fe0007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:39.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.722+0000 7f1fecff9640 1 -- 172.21.15.161:0/2468065459 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1fe4057d60 con 0x7f1ff006ce80 2024-03-28T23:33:39.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.931+0000 7f1ff638a640 1 -- 172.21.15.161:0/2468065459 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f1ff00aabf0 con 0x7f1ff006ce80 2024-03-28T23:33:39.932 INFO:tasks.ceph.mon.a.smithi161.stderr:2024-03-28T23:33:39.931+0000 7fdc7b85b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T23:33:39.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.932+0000 7f1fecff9640 1 -- 172.21.15.161:0/2468065459 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v85) ==== 161+0+0 (secure 0 0 0) 0x7f1fe405be00 con 0x7f1ff006ce80 2024-03-28T23:33:39.933 INFO:teuthology.orchestra.run.smithi161.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-28T23:33:39.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.935+0000 7f1ff638a640 1 -- 172.21.15.161:0/2468065459 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1fdc06b4d0 msgr2=0x7f1fdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:39.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.935+0000 7f1ff638a640 1 --2- 172.21.15.161:0/2468065459 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1fdc06b4d0 0x7f1fdc06d990 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f1fe00013b0 tx=0x7f1fe0007d30 comp rx=0 tx=0).stop 2024-03-28T23:33:39.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.935+0000 7f1ff638a640 1 -- 172.21.15.161:0/2468065459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ff006ce80 msgr2=0x7f1ff014b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:39.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.935+0000 7f1ff638a640 1 --2- 172.21.15.161:0/2468065459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ff006ce80 0x7f1ff014b640 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400c430 tx=0x7f1fe400c8f0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.935+0000 7f1ff638a640 1 -- 172.21.15.161:0/2468065459 shutdown_connections 2024-03-28T23:33:39.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.935+0000 7f1ff638a640 1 --2- 172.21.15.161:0/2468065459 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1fdc06b4d0 0x7f1fdc06d990 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.935+0000 7f1ff638a640 1 --2- 172.21.15.161:0/2468065459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ff006ce80 0x7f1ff014b640 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.935+0000 7f1ff638a640 1 --2- 172.21.15.161:0/2468065459 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ff014eca0 0x7f1ff014d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.935+0000 7f1ff638a640 1 --2- 172.21.15.161:0/2468065459 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ff006ca20 0x7f1ff0145420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:39.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.935+0000 7f1ff638a640 1 -- 172.21.15.161:0/2468065459 >> 172.21.15.161:0/2468065459 conn(0x7f1ff00c6670 msgr2=0x7f1ff00a98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:39.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.935+0000 7f1ff638a640 1 -- 172.21.15.161:0/2468065459 shutdown_connections 2024-03-28T23:33:39.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:39.935+0000 7f1ff638a640 1 -- 172.21.15.161:0/2468065459 wait complete. 2024-03-28T23:33:39.937 INFO:tasks.ceph.mds.b.smithi170.stderr: -735> 2024-03-28T23:33:34.438+0000 7f55ef4c5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:33:39.947 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-28T23:33:39.960 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2024-03-28T23:33:40.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.049+0000 7f66bd1c7640 1 Processor -- start 2024-03-28T23:33:40.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.049+0000 7f66bd1c7640 1 -- start start 2024-03-28T23:33:40.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.050+0000 7f66bd1c7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f66b8152bc0 0x7f66b80c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:40.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.050+0000 7f66bd1c7640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66b80c6f60 con 0x7f66b81589b0 2024-03-28T23:33:40.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.050+0000 7f66bd1c7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66b80c6120 con 0x7f66b80c8a80 2024-03-28T23:33:40.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.050+0000 7f66bd1c7640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f66b80c6400 con 0x7f66b8152bc0 2024-03-28T23:33:40.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.050+0000 7f66b6d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f66b8152bc0 0x7f66b80c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:40.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.050+0000 7f66b6d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f66b8152bc0 0x7f66b80c0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54472/0 (socket says 172.21.15.161:54472) 2024-03-28T23:33:40.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.050+0000 7f66b6d9d640 1 -- 172.21.15.161:0/28097536 learned_addr learned my addr 172.21.15.161:0/28097536 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:40.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.051+0000 7f66b6d9d640 1 -- 172.21.15.161:0/28097536 >> v1:172.21.15.170:6789/0 conn(0x7f66b80c8a80 legacy=0x7f66b80c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:40.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.051+0000 7f66b6d9d640 1 -- 172.21.15.161:0/28097536 >> v1:172.21.15.161:6789/0 conn(0x7f66b81589b0 legacy=0x7f66b8158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:40.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.051+0000 7f66b6d9d640 1 -- 172.21.15.161:0/28097536 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b80c7240 con 0x7f66b8152bc0 2024-03-28T23:33:40.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.051+0000 7f66b6d9d640 1 --2- 172.21.15.161:0/28097536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66b8152bc0 0x7f66b80c0330 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009870 tx=0x7f66a802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56e1d8af2f2bd0f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:40.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.051+0000 7f66b5d9b640 1 -- 172.21.15.161:0/28097536 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a802f840 con 0x7f66b8152bc0 2024-03-28T23:33:40.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.052+0000 7f66b5d9b640 1 -- 172.21.15.161:0/28097536 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66a802f9e0 con 0x7f66b8152bc0 2024-03-28T23:33:40.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.052+0000 7f66b5d9b640 1 -- 172.21.15.161:0/28097536 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a802fdb0 con 0x7f66b8152bc0 2024-03-28T23:33:40.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.052+0000 7f66bd1c7640 1 -- 172.21.15.161:0/28097536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66b8152bc0 msgr2=0x7f66b80c0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:40.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.052+0000 7f66bd1c7640 1 --2- 172.21.15.161:0/28097536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66b8152bc0 0x7f66b80c0330 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009870 tx=0x7f66a802ba20 comp rx=0 tx=0).stop 2024-03-28T23:33:40.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.052+0000 7f66bd1c7640 1 -- 172.21.15.161:0/28097536 shutdown_connections 2024-03-28T23:33:40.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.052+0000 7f66bd1c7640 1 --2- 172.21.15.161:0/28097536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66b8152bc0 0x7f66b80c0330 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:40.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.052+0000 7f66bd1c7640 1 -- 172.21.15.161:0/28097536 >> 172.21.15.161:0/28097536 conn(0x7f66b80caa80 msgr2=0x7f66b80cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:40.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.052+0000 7f66bd1c7640 1 -- 172.21.15.161:0/28097536 shutdown_connections 2024-03-28T23:33:40.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.052+0000 7f66bd1c7640 1 -- 172.21.15.161:0/28097536 wait complete. 2024-03-28T23:33:40.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.053+0000 7f66bd1c7640 1 Processor -- start 2024-03-28T23:33:40.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.053+0000 7f66bd1c7640 1 -- start start 2024-03-28T23:33:40.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.054+0000 7f66bd1c7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66b80c8a80 0x7f66b814fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:40.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.054+0000 7f66bd1c7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66b8152bc0 0x7f66b81eaa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:40.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.054+0000 7f66bd1c7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66b81589b0 0x7f66b8145310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:40.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.054+0000 7f66bd1c7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f66b8152fa0 con 0x7f66b80c8a80 2024-03-28T23:33:40.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.054+0000 7f66bd1c7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f66b80c95f0 con 0x7f66b8152bc0 2024-03-28T23:33:40.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.054+0000 7f66bd1c7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f66b81eafa0 con 0x7f66b81589b0 2024-03-28T23:33:40.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.054+0000 7f66b6d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66b80c8a80 0x7f66b814fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:40.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.054+0000 7f66b6d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66b80c8a80 0x7f66b814fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54478/0 (socket says 172.21.15.161:54478) 2024-03-28T23:33:40.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.054+0000 7f66b6d9d640 1 -- 172.21.15.161:0/2022325397 learned_addr learned my addr 172.21.15.161:0/2022325397 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:40.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.054+0000 7f66b659c640 1 --2- 172.21.15.161:0/2022325397 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66b8152bc0 0x7f66b81eaa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:40.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.054+0000 7f66b759e640 1 --2- 172.21.15.161:0/2022325397 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66b81589b0 0x7f66b8145310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:40.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.055+0000 7f66b6d9d640 1 -- 172.21.15.161:0/2022325397 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66b81589b0 msgr2=0x7f66b8145310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:40.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.055+0000 7f66b6d9d640 1 --2- 172.21.15.161:0/2022325397 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66b81589b0 0x7f66b8145310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:40.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.055+0000 7f66b6d9d640 1 -- 172.21.15.161:0/2022325397 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66b8152bc0 msgr2=0x7f66b81eaa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:40.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.055+0000 7f66b6d9d640 1 --2- 172.21.15.161:0/2022325397 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66b8152bc0 0x7f66b81eaa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:40.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.055+0000 7f66b6d9d640 1 -- 172.21.15.161:0/2022325397 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b806cf90 con 0x7f66b80c8a80 2024-03-28T23:33:40.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.055+0000 7f66b759e640 1 --2- 172.21.15.161:0/2022325397 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66b81589b0 0x7f66b8145310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:40.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.055+0000 7f66b6d9d640 1 --2- 172.21.15.161:0/2022325397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66b80c8a80 0x7f66b814fd30 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f66a8000c00 tx=0x7f66a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:40.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.055+0000 7f669f7fe640 1 -- 172.21.15.161:0/2022325397 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8041400 con 0x7f66b80c8a80 2024-03-28T23:33:40.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.055+0000 7f669f7fe640 1 -- 172.21.15.161:0/2022325397 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66a8046070 con 0x7f66b80c8a80 2024-03-28T23:33:40.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.055+0000 7f66bd1c7640 1 -- 172.21.15.161:0/2022325397 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66b81515d0 con 0x7f66b80c8a80 2024-03-28T23:33:40.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.055+0000 7f669f7fe640 1 -- 172.21.15.161:0/2022325397 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8041800 con 0x7f66b80c8a80 2024-03-28T23:33:40.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.055+0000 7f66bd1c7640 1 -- 172.21.15.161:0/2022325397 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66b8145c40 con 0x7f66b80c8a80 2024-03-28T23:33:40.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.056+0000 7f66bd1c7640 1 -- 172.21.15.161:0/2022325397 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66b804f4b0 con 0x7f66b80c8a80 2024-03-28T23:33:40.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.056+0000 7f669f7fe640 1 -- 172.21.15.161:0/2022325397 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f66a8040070 con 0x7f66b80c8a80 2024-03-28T23:33:40.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.057+0000 7f669f7fe640 1 --2- 172.21.15.161:0/2022325397 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f66a406b4d0 0x7f66a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:40.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.057+0000 7f669f7fe640 1 -- 172.21.15.161:0/2022325397 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 6331+0+0 (secure 0 0 0) 0x7f66a80acd60 con 0x7f66b80c8a80 2024-03-28T23:33:40.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.057+0000 7f66b659c640 1 --2- 172.21.15.161:0/2022325397 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f66a406b4d0 0x7f66a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:40.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.057+0000 7f66b659c640 1 --2- 172.21.15.161:0/2022325397 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f66a406b4d0 0x7f66a406d990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f66b81582a0 tx=0x7f66ac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:40.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.061+0000 7f669f7fe640 1 -- 172.21.15.161:0/2022325397 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f66a8033030 con 0x7f66b80c8a80 2024-03-28T23:33:40.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.273+0000 7f66bd1c7640 1 -- 172.21.15.161:0/2022325397 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f66b814f980 con 0x7f66b80c8a80 2024-03-28T23:33:40.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.934+0000 7f669f7fe640 1 -- 172.21.15.161:0/2022325397 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v86) ==== 104+0+0 (secure 0 0 0) 0x7f66a807b970 con 0x7f66b80c8a80 2024-03-28T23:33:40.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.936+0000 7f66bd1c7640 1 -- 172.21.15.161:0/2022325397 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f66a406b4d0 msgr2=0x7f66a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:40.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.936+0000 7f66bd1c7640 1 --2- 172.21.15.161:0/2022325397 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f66a406b4d0 0x7f66a406d990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f66b81582a0 tx=0x7f66ac009040 comp rx=0 tx=0).stop 2024-03-28T23:33:40.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.936+0000 7f66bd1c7640 1 -- 172.21.15.161:0/2022325397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66b80c8a80 msgr2=0x7f66b814fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:40.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.936+0000 7f66bd1c7640 1 --2- 172.21.15.161:0/2022325397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66b80c8a80 0x7f66b814fd30 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f66a8000c00 tx=0x7f66a8004730 comp rx=0 tx=0).stop 2024-03-28T23:33:40.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.936+0000 7f66bd1c7640 1 -- 172.21.15.161:0/2022325397 shutdown_connections 2024-03-28T23:33:40.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.936+0000 7f66bd1c7640 1 --2- 172.21.15.161:0/2022325397 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f66a406b4d0 0x7f66a406d990 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:40.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.936+0000 7f66bd1c7640 1 --2- 172.21.15.161:0/2022325397 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66b81589b0 0x7f66b8145310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:40.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.936+0000 7f66bd1c7640 1 --2- 172.21.15.161:0/2022325397 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66b8152bc0 0x7f66b81eaa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:40.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.936+0000 7f66bd1c7640 1 --2- 172.21.15.161:0/2022325397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66b80c8a80 0x7f66b814fd30 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:40.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.936+0000 7f66bd1c7640 1 -- 172.21.15.161:0/2022325397 >> 172.21.15.161:0/2022325397 conn(0x7f66b80caa80 msgr2=0x7f66b80c8ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:40.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.936+0000 7f66bd1c7640 1 -- 172.21.15.161:0/2022325397 shutdown_connections 2024-03-28T23:33:40.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:40.936+0000 7f66bd1c7640 1 -- 172.21.15.161:0/2022325397 wait complete. 2024-03-28T23:33:40.949 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-28T23:33:41.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.055+0000 7f50b764e640 1 Processor -- start 2024-03-28T23:33:41.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.055+0000 7f50b764e640 1 -- start start 2024-03-28T23:33:41.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.056+0000 7f50b764e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f50b00a6190 0x7f50b015e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:41.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.056+0000 7f50b764e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50b006e4d0 con 0x7f50b00c2ad0 2024-03-28T23:33:41.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.056+0000 7f50b764e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50b006e790 con 0x7f50b00c5b30 2024-03-28T23:33:41.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.056+0000 7f50b764e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f50b0156db0 con 0x7f50b00a6190 2024-03-28T23:33:41.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.056+0000 7f50b5beb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f50b00a6190 0x7f50b015e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:41.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.056+0000 7f50b5beb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f50b00a6190 0x7f50b015e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57072/0 (socket says 172.21.15.161:57072) 2024-03-28T23:33:41.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.056+0000 7f50b5beb640 1 -- 172.21.15.161:0/1119367805 learned_addr learned my addr 172.21.15.161:0/1119367805 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:41.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.056+0000 7f50b5beb640 1 -- 172.21.15.161:0/1119367805 >> v1:172.21.15.170:6789/0 conn(0x7f50b00c5b30 legacy=0x7f50b0152ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:41.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.056+0000 7f50b5beb640 1 -- 172.21.15.161:0/1119367805 >> v1:172.21.15.161:6789/0 conn(0x7f50b00c2ad0 legacy=0x7f50b00c3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:41.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.056+0000 7f50b5beb640 1 -- 172.21.15.161:0/1119367805 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50b0156760 con 0x7f50b00a6190 2024-03-28T23:33:41.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.057+0000 7f50b5beb640 1 --2- 172.21.15.161:0/1119367805 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50b00a6190 0x7f50b015e980 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f50a8009e50 tx=0x7f50a802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=161339e738308bb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:41.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.057+0000 7f50a7fff640 1 -- 172.21.15.161:0/1119367805 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a802cb20 con 0x7f50b00a6190 2024-03-28T23:33:41.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.057+0000 7f50a7fff640 1 -- 172.21.15.161:0/1119367805 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50a802ccc0 con 0x7f50b00a6190 2024-03-28T23:33:41.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.057+0000 7f50a7fff640 1 -- 172.21.15.161:0/1119367805 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a8034610 con 0x7f50b00a6190 2024-03-28T23:33:41.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.057+0000 7f50b764e640 1 -- 172.21.15.161:0/1119367805 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50b00a6190 msgr2=0x7f50b015e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:41.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.057+0000 7f50b764e640 1 --2- 172.21.15.161:0/1119367805 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50b00a6190 0x7f50b015e980 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f50a8009e50 tx=0x7f50a802bf80 comp rx=0 tx=0).stop 2024-03-28T23:33:41.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.058+0000 7f50b764e640 1 -- 172.21.15.161:0/1119367805 shutdown_connections 2024-03-28T23:33:41.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.058+0000 7f50b764e640 1 --2- 172.21.15.161:0/1119367805 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50b00a6190 0x7f50b015e980 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:41.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.058+0000 7f50b764e640 1 -- 172.21.15.161:0/1119367805 >> 172.21.15.161:0/1119367805 conn(0x7f50b00c6e20 msgr2=0x7f50b00c7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:41.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.058+0000 7f50b764e640 1 -- 172.21.15.161:0/1119367805 shutdown_connections 2024-03-28T23:33:41.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.058+0000 7f50b764e640 1 -- 172.21.15.161:0/1119367805 wait complete. 2024-03-28T23:33:41.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.059+0000 7f50b764e640 1 Processor -- start 2024-03-28T23:33:41.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.059+0000 7f50b764e640 1 -- start start 2024-03-28T23:33:41.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.059+0000 7f50b764e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50b00a6190 0x7f50b0149e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:41.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.059+0000 7f50b764e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f50b00c2ad0 0x7f50b0149170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:41.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.059+0000 7f50b764e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f50b014d990 0x7f50b00a9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:41.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.059+0000 7f50b764e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f50b00c2eb0 con 0x7f50b00a6190 2024-03-28T23:33:41.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.059+0000 7f50b764e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f50b00c67c0 con 0x7f50b00c2ad0 2024-03-28T23:33:41.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.059+0000 7f50b764e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f50b00c4150 con 0x7f50b014d990 2024-03-28T23:33:41.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.059+0000 7f50b53ea640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50b00a6190 0x7f50b0149e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:41.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.059+0000 7f50b53ea640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50b00a6190 0x7f50b0149e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57088/0 (socket says 172.21.15.161:57088) 2024-03-28T23:33:41.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.060+0000 7f50b53ea640 1 -- 172.21.15.161:0/897949232 learned_addr learned my addr 172.21.15.161:0/897949232 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:41.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.060+0000 7f50b4be9640 1 --2- 172.21.15.161:0/897949232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f50b00c2ad0 0x7f50b0149170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:41.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.060+0000 7f50b5beb640 1 --2- 172.21.15.161:0/897949232 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f50b014d990 0x7f50b00a9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:41.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.060+0000 7f50b53ea640 1 -- 172.21.15.161:0/897949232 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f50b014d990 msgr2=0x7f50b00a9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:41.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.060+0000 7f50b53ea640 1 --2- 172.21.15.161:0/897949232 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f50b014d990 0x7f50b00a9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:41.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.060+0000 7f50b53ea640 1 -- 172.21.15.161:0/897949232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f50b00c2ad0 msgr2=0x7f50b0149170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:41.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.060+0000 7f50b53ea640 1 --2- 172.21.15.161:0/897949232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f50b00c2ad0 0x7f50b0149170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:41.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.060+0000 7f50b53ea640 1 -- 172.21.15.161:0/897949232 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50b014ca50 con 0x7f50b00a6190 2024-03-28T23:33:41.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.060+0000 7f50b5beb640 1 --2- 172.21.15.161:0/897949232 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f50b014d990 0x7f50b00a9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:41.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.060+0000 7f50b53ea640 1 --2- 172.21.15.161:0/897949232 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50b00a6190 0x7f50b0149e00 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f50a000c480 tx=0x7f50a000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:41.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.060+0000 7f50a5ffb640 1 -- 172.21.15.161:0/897949232 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a00052d0 con 0x7f50b00a6190 2024-03-28T23:33:41.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.060+0000 7f50a5ffb640 1 -- 172.21.15.161:0/897949232 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50a0005470 con 0x7f50b00a6190 2024-03-28T23:33:41.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.060+0000 7f50b764e640 1 -- 172.21.15.161:0/897949232 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50b01ecab0 con 0x7f50b00a6190 2024-03-28T23:33:41.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.060+0000 7f50a5ffb640 1 -- 172.21.15.161:0/897949232 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a0012870 con 0x7f50b00a6190 2024-03-28T23:33:41.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.060+0000 7f50b764e640 1 -- 172.21.15.161:0/897949232 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50b01ecd90 con 0x7f50b00a6190 2024-03-28T23:33:41.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.061+0000 7f50b764e640 1 -- 172.21.15.161:0/897949232 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50b0068f90 con 0x7f50b00a6190 2024-03-28T23:33:41.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.062+0000 7f50a5ffb640 1 -- 172.21.15.161:0/897949232 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f50a0012a10 con 0x7f50b00a6190 2024-03-28T23:33:41.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.062+0000 7f50a5ffb640 1 --2- 172.21.15.161:0/897949232 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f509406b500 0x7f509406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:41.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.062+0000 7f50a5ffb640 1 -- 172.21.15.161:0/897949232 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 6331+0+0 (secure 0 0 0) 0x7f50a0089fc0 con 0x7f50b00a6190 2024-03-28T23:33:41.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.062+0000 7f50b4be9640 1 --2- 172.21.15.161:0/897949232 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f509406b500 0x7f509406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:41.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.062+0000 7f50b4be9640 1 --2- 172.21.15.161:0/897949232 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f509406b500 0x7f509406d9c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f50b014c8e0 tx=0x7f50ac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:41.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.066+0000 7f50a5ffb640 1 -- 172.21.15.161:0/897949232 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f50a0058c60 con 0x7f50b00a6190 2024-03-28T23:33:41.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.237+0000 7f50b764e640 1 -- 172.21.15.161:0/897949232 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f50b0073620 con 0x7f50b00a6190 2024-03-28T23:33:41.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.938+0000 7f50a5ffb640 1 -- 172.21.15.161:0/897949232 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v99) ==== 181+0+0 (secure 0 0 0) 0x7f50a0058560 con 0x7f50b00a6190 2024-03-28T23:33:41.939 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_metadata' removed 2024-03-28T23:33:41.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.942+0000 7f50b764e640 1 -- 172.21.15.161:0/897949232 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f509406b500 msgr2=0x7f509406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:41.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.942+0000 7f50b764e640 1 --2- 172.21.15.161:0/897949232 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f509406b500 0x7f509406d9c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f50b014c8e0 tx=0x7f50ac009040 comp rx=0 tx=0).stop 2024-03-28T23:33:41.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.942+0000 7f50b764e640 1 -- 172.21.15.161:0/897949232 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50b00a6190 msgr2=0x7f50b0149e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:41.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.942+0000 7f50b764e640 1 --2- 172.21.15.161:0/897949232 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50b00a6190 0x7f50b0149e00 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f50a000c480 tx=0x7f50a000c940 comp rx=0 tx=0).stop 2024-03-28T23:33:41.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.942+0000 7f50b764e640 1 -- 172.21.15.161:0/897949232 shutdown_connections 2024-03-28T23:33:41.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.942+0000 7f50b764e640 1 --2- 172.21.15.161:0/897949232 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f509406b500 0x7f509406d9c0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:41.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.942+0000 7f50b764e640 1 --2- 172.21.15.161:0/897949232 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f50b014d990 0x7f50b00a9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:41.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.942+0000 7f50b764e640 1 --2- 172.21.15.161:0/897949232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f50b00c2ad0 0x7f50b0149170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:41.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.942+0000 7f50b764e640 1 --2- 172.21.15.161:0/897949232 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50b00a6190 0x7f50b0149e00 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:41.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.942+0000 7f50b764e640 1 -- 172.21.15.161:0/897949232 >> 172.21.15.161:0/897949232 conn(0x7f50b00c6e20 msgr2=0x7f50b00c1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:41.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.942+0000 7f50b764e640 1 -- 172.21.15.161:0/897949232 shutdown_connections 2024-03-28T23:33:41.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:41.942+0000 7f50b764e640 1 -- 172.21.15.161:0/897949232 wait complete. 2024-03-28T23:33:41.956 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-28T23:33:42.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.065+0000 7f7a48296640 1 Processor -- start 2024-03-28T23:33:42.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.066+0000 7f7a48296640 1 -- start start 2024-03-28T23:33:42.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.066+0000 7f7a48296640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7a400ac430 0x7f7a40152880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:42.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.066+0000 7f7a48296640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7a400c30c0 0x7f7a400c24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:42.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.066+0000 7f7a48296640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a400c2a10 0x7f7a400c19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:42.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.066+0000 7f7a48296640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f7a40152dc0 con 0x7f7a400c30c0 2024-03-28T23:33:42.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.066+0000 7f7a48296640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f7a400c3540 con 0x7f7a400ac430 2024-03-28T23:33:42.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.066+0000 7f7a48296640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7a400c3820 con 0x7f7a400c2a10 2024-03-28T23:33:42.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.067+0000 7f7a45831640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7a400c30c0 0x7f7a400c24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:42.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.067+0000 7f7a45831640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7a400c30c0 0x7f7a400c24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57102/0 (socket says 172.21.15.161:57102) 2024-03-28T23:33:42.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.067+0000 7f7a45831640 1 -- 172.21.15.161:0/381687488 learned_addr learned my addr 172.21.15.161:0/381687488 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:42.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.067+0000 7f7a46032640 1 --2- 172.21.15.161:0/381687488 >> v2:172.21.15.170:3300/0 conn(0x7f7a400ac430 0x7f7a40152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:42.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.067+0000 7f7a46833640 1 --2- 172.21.15.161:0/381687488 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a400c2a10 0x7f7a400c19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:42.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.067+0000 7f7a45831640 1 -- 172.21.15.161:0/381687488 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a400c2a10 msgr2=0x7f7a400c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:42.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.067+0000 7f7a45831640 1 --2- 172.21.15.161:0/381687488 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a400c2a10 0x7f7a400c19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:42.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.067+0000 7f7a45831640 1 -- 172.21.15.161:0/381687488 >> v2:172.21.15.170:3300/0 conn(0x7f7a400ac430 msgr2=0x7f7a40152880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:42.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.067+0000 7f7a45831640 1 --2- 172.21.15.161:0/381687488 >> v2:172.21.15.170:3300/0 conn(0x7f7a400ac430 0x7f7a40152880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:42.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.067+0000 7f7a45831640 1 -- 172.21.15.161:0/381687488 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a400c3b00 con 0x7f7a400c30c0 2024-03-28T23:33:42.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.067+0000 7f7a46032640 1 --2- 172.21.15.161:0/381687488 >> v2:172.21.15.170:3300/0 conn(0x7f7a400ac430 0x7f7a40152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:42.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.068+0000 7f7a45831640 1 --2- 172.21.15.161:0/381687488 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a400c30c0 0x7f7a400c24d0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f7a400a95f0 tx=0x7f7a3c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=938e98561f2e6dc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:42.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.068+0000 7f7a45030640 1 -- 172.21.15.161:0/381687488 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c02e070 con 0x7f7a400c30c0 2024-03-28T23:33:42.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.068+0000 7f7a45030640 1 -- 172.21.15.161:0/381687488 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a3c037070 con 0x7f7a400c30c0 2024-03-28T23:33:42.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.068+0000 7f7a45030640 1 -- 172.21.15.161:0/381687488 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c031a00 con 0x7f7a400c30c0 2024-03-28T23:33:42.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.068+0000 7f7a48296640 1 -- 172.21.15.161:0/381687488 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a400c30c0 msgr2=0x7f7a400c24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:42.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.068+0000 7f7a48296640 1 --2- 172.21.15.161:0/381687488 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a400c30c0 0x7f7a400c24d0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f7a400a95f0 tx=0x7f7a3c02b880 comp rx=0 tx=0).stop 2024-03-28T23:33:42.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.068+0000 7f7a48296640 1 -- 172.21.15.161:0/381687488 shutdown_connections 2024-03-28T23:33:42.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.068+0000 7f7a48296640 1 --2- 172.21.15.161:0/381687488 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a400c2a10 0x7f7a400c19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:42.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.068+0000 7f7a48296640 1 --2- 172.21.15.161:0/381687488 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a400c30c0 0x7f7a400c24d0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:42.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.068+0000 7f7a48296640 1 --2- 172.21.15.161:0/381687488 >> v2:172.21.15.170:3300/0 conn(0x7f7a400ac430 0x7f7a40152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:42.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.068+0000 7f7a48296640 1 -- 172.21.15.161:0/381687488 >> 172.21.15.161:0/381687488 conn(0x7f7a4006b150 msgr2=0x7f7a4006b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:42.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.068+0000 7f7a48296640 1 -- 172.21.15.161:0/381687488 shutdown_connections 2024-03-28T23:33:42.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.068+0000 7f7a48296640 1 -- 172.21.15.161:0/381687488 wait complete. 2024-03-28T23:33:42.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.069+0000 7f7a48296640 1 Processor -- start 2024-03-28T23:33:42.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.070+0000 7f7a48296640 1 -- start start 2024-03-28T23:33:42.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.070+0000 7f7a48296640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a400ac430 0x7f7a40072610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:42.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.070+0000 7f7a48296640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a400c2a10 0x7f7a40072b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:42.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.070+0000 7f7a48296640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a400c30c0 0x7f7a401ec460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:42.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.070+0000 7f7a48296640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7a400c6440 con 0x7f7a400c2a10 2024-03-28T23:33:42.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.070+0000 7f7a48296640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7a400c1490 con 0x7f7a400ac430 2024-03-28T23:33:42.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.070+0000 7f7a48296640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7a400ac810 con 0x7f7a400c30c0 2024-03-28T23:33:42.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.070+0000 7f7a45831640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a400c2a10 0x7f7a40072b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:42.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.070+0000 7f7a45831640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a400c2a10 0x7f7a40072b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57108/0 (socket says 172.21.15.161:57108) 2024-03-28T23:33:42.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.070+0000 7f7a45831640 1 -- 172.21.15.161:0/1272271450 learned_addr learned my addr 172.21.15.161:0/1272271450 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:42.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.070+0000 7f7a46032640 1 --2- 172.21.15.161:0/1272271450 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a400ac430 0x7f7a40072610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:42.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.070+0000 7f7a46833640 1 --2- 172.21.15.161:0/1272271450 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a400c30c0 0x7f7a401ec460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:42.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.071+0000 7f7a45831640 1 -- 172.21.15.161:0/1272271450 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a400c30c0 msgr2=0x7f7a401ec460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:42.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.071+0000 7f7a45831640 1 --2- 172.21.15.161:0/1272271450 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a400c30c0 0x7f7a401ec460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:42.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.071+0000 7f7a45831640 1 -- 172.21.15.161:0/1272271450 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a400ac430 msgr2=0x7f7a40072610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:42.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.071+0000 7f7a45831640 1 --2- 172.21.15.161:0/1272271450 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a400ac430 0x7f7a40072610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:42.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.071+0000 7f7a45831640 1 -- 172.21.15.161:0/1272271450 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a40153c40 con 0x7f7a400c2a10 2024-03-28T23:33:42.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.071+0000 7f7a46032640 1 --2- 172.21.15.161:0/1272271450 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a400ac430 0x7f7a40072610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:42.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.071+0000 7f7a45831640 1 --2- 172.21.15.161:0/1272271450 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a400c2a10 0x7f7a40072b50 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c02b8c0 tx=0x7f7a3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:42.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.071+0000 7f7a36ffd640 1 -- 172.21.15.161:0/1272271450 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c032880 con 0x7f7a400c2a10 2024-03-28T23:33:42.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.071+0000 7f7a36ffd640 1 -- 172.21.15.161:0/1272271450 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a3c032a20 con 0x7f7a400c2a10 2024-03-28T23:33:42.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.071+0000 7f7a48296640 1 -- 172.21.15.161:0/1272271450 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a40153680 con 0x7f7a400c2a10 2024-03-28T23:33:42.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.071+0000 7f7a36ffd640 1 -- 172.21.15.161:0/1272271450 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c032e10 con 0x7f7a400c2a10 2024-03-28T23:33:42.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.071+0000 7f7a48296640 1 -- 172.21.15.161:0/1272271450 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a40073250 con 0x7f7a400c2a10 2024-03-28T23:33:42.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.072+0000 7f7a48296640 1 -- 172.21.15.161:0/1272271450 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a40068f90 con 0x7f7a400c2a10 2024-03-28T23:33:42.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.072+0000 7f7a36ffd640 1 -- 172.21.15.161:0/1272271450 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7a3c0453f0 con 0x7f7a400c2a10 2024-03-28T23:33:42.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.073+0000 7f7a36ffd640 1 --2- 172.21.15.161:0/1272271450 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7a2406b4d0 0x7f7a2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:42.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.073+0000 7f7a36ffd640 1 -- 172.21.15.161:0/1272271450 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5893+0+0 (secure 0 0 0) 0x7f7a3c0ac270 con 0x7f7a400c2a10 2024-03-28T23:33:42.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.073+0000 7f7a46032640 1 --2- 172.21.15.161:0/1272271450 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7a2406b4d0 0x7f7a2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:42.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.073+0000 7f7a46032640 1 --2- 172.21.15.161:0/1272271450 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7a2406b4d0 0x7f7a2406d990 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7a400a88d0 tx=0x7f7a30008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:42.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.077+0000 7f7a36ffd640 1 -- 172.21.15.161:0/1272271450 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7a3c040070 con 0x7f7a400c2a10 2024-03-28T23:33:42.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.255+0000 7f7a48296640 1 -- 172.21.15.161:0/1272271450 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7a4006ba50 con 0x7f7a400c2a10 2024-03-28T23:33:42.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.947+0000 7f7a36ffd640 1 -- 172.21.15.161:0/1272271450 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v100) ==== 169+0+0 (secure 0 0 0) 0x7f7a3c07b1e0 con 0x7f7a400c2a10 2024-03-28T23:33:42.948 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data' removed 2024-03-28T23:33:42.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.950+0000 7f7a48296640 1 -- 172.21.15.161:0/1272271450 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7a2406b4d0 msgr2=0x7f7a2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:42.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.950+0000 7f7a48296640 1 --2- 172.21.15.161:0/1272271450 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7a2406b4d0 0x7f7a2406d990 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7a400a88d0 tx=0x7f7a30008040 comp rx=0 tx=0).stop 2024-03-28T23:33:42.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.950+0000 7f7a48296640 1 -- 172.21.15.161:0/1272271450 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a400c2a10 msgr2=0x7f7a40072b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:42.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.950+0000 7f7a48296640 1 --2- 172.21.15.161:0/1272271450 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a400c2a10 0x7f7a40072b50 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c02b8c0 tx=0x7f7a3c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:42.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.950+0000 7f7a48296640 1 -- 172.21.15.161:0/1272271450 shutdown_connections 2024-03-28T23:33:42.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.950+0000 7f7a48296640 1 --2- 172.21.15.161:0/1272271450 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7a2406b4d0 0x7f7a2406d990 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:42.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.950+0000 7f7a48296640 1 --2- 172.21.15.161:0/1272271450 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a400c2a10 0x7f7a40072b50 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:42.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.950+0000 7f7a48296640 1 --2- 172.21.15.161:0/1272271450 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a400c30c0 0x7f7a401ec460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:42.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.950+0000 7f7a48296640 1 --2- 172.21.15.161:0/1272271450 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a400ac430 0x7f7a40072610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:42.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.950+0000 7f7a48296640 1 -- 172.21.15.161:0/1272271450 >> 172.21.15.161:0/1272271450 conn(0x7f7a4006b150 msgr2=0x7f7a400aab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:42.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.950+0000 7f7a48296640 1 -- 172.21.15.161:0/1272271450 shutdown_connections 2024-03-28T23:33:42.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:42.951+0000 7f7a48296640 1 -- 172.21.15.161:0/1272271450 wait complete. 2024-03-28T23:33:42.968 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-28T23:33:43.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.072+0000 7f8074d02640 1 Processor -- start 2024-03-28T23:33:43.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.072+0000 7f8074d02640 1 -- start start 2024-03-28T23:33:43.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.072+0000 7f8074d02640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f80700a9960 0x7f8070072970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:43.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.072+0000 7f8074d02640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80700c7330 0x7f807014e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:43.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.072+0000 7f8074d02640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f80700733e0 0x7f807014e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:43.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.072+0000 7f8074d02640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f80700aa3a0 con 0x7f80700a9960 2024-03-28T23:33:43.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.072+0000 7f8074d02640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f80700abc00 con 0x7f80700733e0 2024-03-28T23:33:43.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.072+0000 7f8074d02640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f807006bf70 con 0x7f80700c7330 2024-03-28T23:33:43.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.073+0000 7f806e59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f80700a9960 0x7f8070072970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:43.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.073+0000 7f806e59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f80700a9960 0x7f8070072970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57110/0 (socket says 172.21.15.161:57110) 2024-03-28T23:33:43.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.073+0000 7f806e59c640 1 -- 172.21.15.161:0/2143864099 learned_addr learned my addr 172.21.15.161:0/2143864099 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:43.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.073+0000 7f8067fff640 1 --2- 172.21.15.161:0/2143864099 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80700c7330 0x7f807014e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:43.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.073+0000 7f806ed9d640 1 --2- 172.21.15.161:0/2143864099 >> v2:172.21.15.170:3300/0 conn(0x7f80700733e0 0x7f807014e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:43.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.073+0000 7f806e59c640 1 -- 172.21.15.161:0/2143864099 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80700c7330 msgr2=0x7f807014e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:43.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.073+0000 7f806e59c640 1 --2- 172.21.15.161:0/2143864099 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80700c7330 0x7f807014e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.073+0000 7f806e59c640 1 -- 172.21.15.161:0/2143864099 >> v2:172.21.15.170:3300/0 conn(0x7f80700733e0 msgr2=0x7f807014e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:43.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.073+0000 7f806e59c640 1 --2- 172.21.15.161:0/2143864099 >> v2:172.21.15.170:3300/0 conn(0x7f80700733e0 0x7f807014e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.073+0000 7f806e59c640 1 -- 172.21.15.161:0/2143864099 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f807006bcd0 con 0x7f80700a9960 2024-03-28T23:33:43.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.073+0000 7f806ed9d640 1 --2- 172.21.15.161:0/2143864099 >> v2:172.21.15.170:3300/0 conn(0x7f80700733e0 0x7f807014e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:43.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.074+0000 7f806e59c640 1 --2- 172.21.15.161:0/2143864099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80700a9960 0x7f8070072970 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f807006bea0 tx=0x7f806002b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b7e1f8f5ca0378 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:43.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.074+0000 7f806dd9b640 1 -- 172.21.15.161:0/2143864099 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8060038070 con 0x7f80700a9960 2024-03-28T23:33:43.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.074+0000 7f806dd9b640 1 -- 172.21.15.161:0/2143864099 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8060031520 con 0x7f80700a9960 2024-03-28T23:33:43.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.074+0000 7f806dd9b640 1 -- 172.21.15.161:0/2143864099 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80600318f0 con 0x7f80700a9960 2024-03-28T23:33:43.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.074+0000 7f8074d02640 1 -- 172.21.15.161:0/2143864099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80700a9960 msgr2=0x7f8070072970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:43.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.074+0000 7f8074d02640 1 --2- 172.21.15.161:0/2143864099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80700a9960 0x7f8070072970 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f807006bea0 tx=0x7f806002b7c0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.074+0000 7f8074d02640 1 -- 172.21.15.161:0/2143864099 shutdown_connections 2024-03-28T23:33:43.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.074+0000 7f8074d02640 1 --2- 172.21.15.161:0/2143864099 >> v2:172.21.15.170:3300/0 conn(0x7f80700733e0 0x7f807014e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.074+0000 7f8074d02640 1 --2- 172.21.15.161:0/2143864099 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80700c7330 0x7f807014e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.074+0000 7f8074d02640 1 --2- 172.21.15.161:0/2143864099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80700a9960 0x7f8070072970 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.075+0000 7f8074d02640 1 -- 172.21.15.161:0/2143864099 >> 172.21.15.161:0/2143864099 conn(0x7f80700a83f0 msgr2=0x7f80700a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:43.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.075+0000 7f8074d02640 1 -- 172.21.15.161:0/2143864099 shutdown_connections 2024-03-28T23:33:43.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.075+0000 7f8074d02640 1 -- 172.21.15.161:0/2143864099 wait complete. 2024-03-28T23:33:43.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.076+0000 7f8074d02640 1 Processor -- start 2024-03-28T23:33:43.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.076+0000 7f8074d02640 1 -- start start 2024-03-28T23:33:43.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.076+0000 7f8074d02640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80700733e0 0x7f8070155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:43.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.076+0000 7f8074d02640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80700a9960 0x7f8070146030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:43.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.076+0000 7f8074d02640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80700c7330 0x7f80700bfce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:43.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.076+0000 7f8074d02640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f80700a9d40 con 0x7f80700c7330 2024-03-28T23:33:43.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.076+0000 7f8074d02640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8070152250 con 0x7f80700733e0 2024-03-28T23:33:43.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.076+0000 7f8074d02640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f80700a9ec0 con 0x7f80700a9960 2024-03-28T23:33:43.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.077+0000 7f806ed9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80700c7330 0x7f80700bfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:43.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.077+0000 7f806ed9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80700c7330 0x7f80700bfce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57114/0 (socket says 172.21.15.161:57114) 2024-03-28T23:33:43.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.077+0000 7f806ed9d640 1 -- 172.21.15.161:0/1479154755 learned_addr learned my addr 172.21.15.161:0/1479154755 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:43.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.077+0000 7f806e59c640 1 --2- 172.21.15.161:0/1479154755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80700733e0 0x7f8070155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:43.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.077+0000 7f8067fff640 1 --2- 172.21.15.161:0/1479154755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80700a9960 0x7f8070146030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:43.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.077+0000 7f806ed9d640 1 -- 172.21.15.161:0/1479154755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80700a9960 msgr2=0x7f8070146030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:43.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.077+0000 7f806ed9d640 1 --2- 172.21.15.161:0/1479154755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80700a9960 0x7f8070146030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.077+0000 7f806ed9d640 1 -- 172.21.15.161:0/1479154755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80700733e0 msgr2=0x7f8070155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:43.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.077+0000 7f806ed9d640 1 --2- 172.21.15.161:0/1479154755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80700733e0 0x7f8070155820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.077+0000 7f806ed9d640 1 -- 172.21.15.161:0/1479154755 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8070150a60 con 0x7f80700c7330 2024-03-28T23:33:43.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.077+0000 7f806ed9d640 1 --2- 172.21.15.161:0/1479154755 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80700c7330 0x7f80700bfce0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f8068007d90 tx=0x7f806800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:43.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.077+0000 7f8066ffd640 1 -- 172.21.15.161:0/1479154755 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f806801a070 con 0x7f80700c7330 2024-03-28T23:33:43.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.077+0000 7f8066ffd640 1 -- 172.21.15.161:0/1479154755 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8068003990 con 0x7f80700c7330 2024-03-28T23:33:43.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.077+0000 7f8066ffd640 1 -- 172.21.15.161:0/1479154755 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8068003d80 con 0x7f80700c7330 2024-03-28T23:33:43.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.077+0000 7f8074d02640 1 -- 172.21.15.161:0/1479154755 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80701eea90 con 0x7f80700c7330 2024-03-28T23:33:43.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.077+0000 7f8074d02640 1 -- 172.21.15.161:0/1479154755 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80700c0220 con 0x7f80700c7330 2024-03-28T23:33:43.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.078+0000 7f8074d02640 1 -- 172.21.15.161:0/1479154755 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f807004f4b0 con 0x7f80700c7330 2024-03-28T23:33:43.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.079+0000 7f8066ffd640 1 -- 172.21.15.161:0/1479154755 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f806801d070 con 0x7f80700c7330 2024-03-28T23:33:43.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.079+0000 7f8066ffd640 1 --2- 172.21.15.161:0/1479154755 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f805006b500 0x7f805006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:43.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.079+0000 7f8066ffd640 1 -- 172.21.15.161:0/1479154755 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5421+0+0 (secure 0 0 0) 0x7f806808a160 con 0x7f80700c7330 2024-03-28T23:33:43.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.079+0000 7f806e59c640 1 --2- 172.21.15.161:0/1479154755 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f805006b500 0x7f805006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:43.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.080+0000 7f806e59c640 1 --2- 172.21.15.161:0/1479154755 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f805006b500 0x7f805006d9c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8060035000 tx=0x7f8060030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:43.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.083+0000 7f8066ffd640 1 -- 172.21.15.161:0/1479154755 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8068010070 con 0x7f80700c7330 2024-03-28T23:33:43.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.254+0000 7f8074d02640 1 -- 172.21.15.161:0/1479154755 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f807006b1b0 con 0x7f80700c7330 2024-03-28T23:33:43.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.258+0000 7f8066ffd640 1 -- 172.21.15.161:0/1479154755 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v33) ==== 104+0+0 (secure 0 0 0) 0x7f8068058b40 con 0x7f80700c7330 2024-03-28T23:33:43.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.260+0000 7f8074d02640 1 -- 172.21.15.161:0/1479154755 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f805006b500 msgr2=0x7f805006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:43.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.260+0000 7f8074d02640 1 --2- 172.21.15.161:0/1479154755 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f805006b500 0x7f805006d9c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8060035000 tx=0x7f8060030000 comp rx=0 tx=0).stop 2024-03-28T23:33:43.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.260+0000 7f8074d02640 1 -- 172.21.15.161:0/1479154755 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80700c7330 msgr2=0x7f80700bfce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:43.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.260+0000 7f8074d02640 1 --2- 172.21.15.161:0/1479154755 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80700c7330 0x7f80700bfce0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f8068007d90 tx=0x7f806800d4a0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.260+0000 7f8074d02640 1 -- 172.21.15.161:0/1479154755 shutdown_connections 2024-03-28T23:33:43.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.260+0000 7f8074d02640 1 --2- 172.21.15.161:0/1479154755 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f805006b500 0x7f805006d9c0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.260+0000 7f8074d02640 1 --2- 172.21.15.161:0/1479154755 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80700c7330 0x7f80700bfce0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.260+0000 7f8074d02640 1 --2- 172.21.15.161:0/1479154755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80700a9960 0x7f8070146030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.260+0000 7f8074d02640 1 --2- 172.21.15.161:0/1479154755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80700733e0 0x7f8070155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.260+0000 7f8074d02640 1 -- 172.21.15.161:0/1479154755 >> 172.21.15.161:0/1479154755 conn(0x7f80700a83f0 msgr2=0x7f8070070470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:43.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.260+0000 7f8074d02640 1 -- 172.21.15.161:0/1479154755 shutdown_connections 2024-03-28T23:33:43.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.260+0000 7f8074d02640 1 -- 172.21.15.161:0/1479154755 wait complete. 2024-03-28T23:33:43.271 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.b mds_join_fs 2024-03-28T23:33:43.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.374+0000 7f133866f640 1 Processor -- start 2024-03-28T23:33:43.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.374+0000 7f133866f640 1 -- start start 2024-03-28T23:33:43.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.375+0000 7f133866f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1330152bd0 0x7f133015d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:43.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.375+0000 7f133866f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13300c2f10 con 0x7f133014f5c0 2024-03-28T23:33:43.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.375+0000 7f133866f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f133014f9a0 con 0x7f133006cdd0 2024-03-28T23:33:43.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.375+0000 7f133866f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f13300c74d0 con 0x7f1330152bd0 2024-03-28T23:33:43.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.375+0000 7f133640b640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f133014f5c0 0x7f13300c2150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:55904/0 (socket says 172.21.15.161:55904) 2024-03-28T23:33:43.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.375+0000 7f133640b640 1 -- 172.21.15.161:0/2736209980 learned_addr learned my addr 172.21.15.161:0/2736209980 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:43.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.375+0000 7f1335409640 1 -- 172.21.15.161:0/2736209980 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2395781813 0 0) 0x7f132c001110 con 0x7f133014f5c0 2024-03-28T23:33:43.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.375+0000 7f1335409640 1 -- 172.21.15.161:0/2736209980 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f131c003520 con 0x7f133014f5c0 2024-03-28T23:33:43.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.375+0000 7f1336c0c640 1 --2- 172.21.15.161:0/2736209980 >> v2:172.21.15.170:3300/0 conn(0x7f1330152bd0 0x7f133015d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:43.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.375+0000 7f1335409640 1 -- 172.21.15.161:0/2736209980 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 447090405 0 0) 0x7f131c003520 con 0x7f133014f5c0 2024-03-28T23:33:43.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.375+0000 7f1335409640 1 -- 172.21.15.161:0/2736209980 >> v2:172.21.15.170:3300/0 conn(0x7f1330152bd0 msgr2=0x7f133015d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:43.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.375+0000 7f1335409640 1 --2- 172.21.15.161:0/2736209980 >> v2:172.21.15.170:3300/0 conn(0x7f1330152bd0 0x7f133015d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.375+0000 7f1335409640 1 -- 172.21.15.161:0/2736209980 >> v1:172.21.15.170:6789/0 conn(0x7f133006cdd0 legacy=0x7f13301524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:43.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.375+0000 7f1335409640 1 -- 172.21.15.161:0/2736209980 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f133006c140 con 0x7f133014f5c0 2024-03-28T23:33:43.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.375+0000 7f1335409640 1 -- 172.21.15.161:0/2736209980 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f132c001840 con 0x7f133014f5c0 2024-03-28T23:33:43.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.375+0000 7f1335409640 1 -- 172.21.15.161:0/2736209980 >> v1:172.21.15.161:6789/0 conn(0x7f133014f5c0 legacy=0x7f13300c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:43.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.375+0000 7f1335409640 1 --2- 172.21.15.161:0/2736209980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f131c003a90 0x7f131c003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:43.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.375+0000 7f1335409640 1 --2- 172.21.15.161:0/2736209980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f131c004800 0x7f131c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:43.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.375+0000 7f1335409640 1 --2- 172.21.15.161:0/2736209980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f131c005210 0x7f131c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:43.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.375+0000 7f1335409640 1 -- 172.21.15.161:0/2736209980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f13300c74d0 con 0x7f131c004800 2024-03-28T23:33:43.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.375+0000 7f1335409640 1 -- 172.21.15.161:0/2736209980 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f131c00c140 con 0x7f131c005210 2024-03-28T23:33:43.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.375+0000 7f1335409640 1 -- 172.21.15.161:0/2736209980 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f131c00c340 con 0x7f131c003a90 2024-03-28T23:33:43.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.376+0000 7f1335c0a640 1 --2- 172.21.15.161:0/2736209980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f131c004800 0x7f131c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:43.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.376+0000 7f133640b640 1 --2- 172.21.15.161:0/2736209980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f131c003a90 0x7f131c003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:43.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.376+0000 7f1336c0c640 1 --2- 172.21.15.161:0/2736209980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f131c005210 0x7f131c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:43.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.376+0000 7f1335c0a640 1 -- 172.21.15.161:0/2736209980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f131c003a90 msgr2=0x7f131c003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:43.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.376+0000 7f1335c0a640 1 --2- 172.21.15.161:0/2736209980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f131c003a90 0x7f131c003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.376+0000 7f1335c0a640 1 -- 172.21.15.161:0/2736209980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f131c005210 msgr2=0x7f131c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:43.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.376+0000 7f1335c0a640 1 --2- 172.21.15.161:0/2736209980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f131c005210 0x7f131c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.376+0000 7f1335c0a640 1 -- 172.21.15.161:0/2736209980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f131c003520 con 0x7f131c004800 2024-03-28T23:33:43.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.376+0000 7f1336c0c640 1 -- 172.21.15.161:0/2736209980 reap_dead start 2024-03-28T23:33:43.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.377+0000 7f1335c0a640 1 --2- 172.21.15.161:0/2736209980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f131c004800 0x7f131c004c80 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f133006d230 tx=0x7f131c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1444362cc3710673 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:43.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.377+0000 7f1335409640 1 -- 172.21.15.161:0/2736209980 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1320001370 con 0x7f131c004800 2024-03-28T23:33:43.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.377+0000 7f1335409640 1 -- 172.21.15.161:0/2736209980 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13200052d0 con 0x7f131c004800 2024-03-28T23:33:43.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.377+0000 7f1335409640 1 -- 172.21.15.161:0/2736209980 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1320003830 con 0x7f131c004800 2024-03-28T23:33:43.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.377+0000 7f133866f640 1 -- 172.21.15.161:0/2736209980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f131c004800 msgr2=0x7f131c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:43.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.377+0000 7f133866f640 1 --2- 172.21.15.161:0/2736209980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f131c004800 0x7f131c004c80 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f133006d230 tx=0x7f131c005830 comp rx=0 tx=0).stop 2024-03-28T23:33:43.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.377+0000 7f133866f640 1 -- 172.21.15.161:0/2736209980 shutdown_connections 2024-03-28T23:33:43.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.377+0000 7f133866f640 1 --2- 172.21.15.161:0/2736209980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f131c004800 0x7f131c004c80 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.377+0000 7f133866f640 1 -- 172.21.15.161:0/2736209980 >> 172.21.15.161:0/2736209980 conn(0x7f13300a83f0 msgr2=0x7f13300a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:43.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.377+0000 7f133866f640 1 -- 172.21.15.161:0/2736209980 shutdown_connections 2024-03-28T23:33:43.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.377+0000 7f133866f640 1 -- 172.21.15.161:0/2736209980 wait complete. 2024-03-28T23:33:43.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.378+0000 7f133866f640 1 Processor -- start 2024-03-28T23:33:43.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.378+0000 7f133866f640 1 -- start start 2024-03-28T23:33:43.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.379+0000 7f133866f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f131c004800 0x7f1330072580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:43.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.379+0000 7f133866f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f133014a550 0x7f13301487c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:43.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.379+0000 7f133866f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1330148d00 0x7f13300c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:43.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.379+0000 7f133866f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1330152030 con 0x7f133014a550 2024-03-28T23:33:43.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.379+0000 7f133866f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f13300c6fa0 con 0x7f131c004800 2024-03-28T23:33:43.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.379+0000 7f133866f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f133006c3e0 con 0x7f1330148d00 2024-03-28T23:33:43.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.379+0000 7f1335c0a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f133014a550 0x7f13301487c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:43.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.379+0000 7f1335c0a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f133014a550 0x7f13301487c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57134/0 (socket says 172.21.15.161:57134) 2024-03-28T23:33:43.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.379+0000 7f1335c0a640 1 -- 172.21.15.161:0/948071657 learned_addr learned my addr 172.21.15.161:0/948071657 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:43.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.379+0000 7f133640b640 1 --2- 172.21.15.161:0/948071657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f131c004800 0x7f1330072580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:43.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.379+0000 7f1335c0a640 1 -- 172.21.15.161:0/948071657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1330148d00 msgr2=0x7f13300c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:43.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.379+0000 7f1335c0a640 1 --2- 172.21.15.161:0/948071657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1330148d00 0x7f13300c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.379+0000 7f1335c0a640 1 -- 172.21.15.161:0/948071657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f131c004800 msgr2=0x7f1330072580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:43.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.379+0000 7f1335c0a640 1 --2- 172.21.15.161:0/948071657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f131c004800 0x7f1330072580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.379+0000 7f1335c0a640 1 -- 172.21.15.161:0/948071657 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1330073190 con 0x7f133014a550 2024-03-28T23:33:43.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.380+0000 7f1335c0a640 1 --2- 172.21.15.161:0/948071657 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f133014a550 0x7f13301487c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f1320000c00 tx=0x7f1320001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:43.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.380+0000 7f1326ffd640 1 -- 172.21.15.161:0/948071657 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1320041020 con 0x7f133014a550 2024-03-28T23:33:43.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.380+0000 7f1326ffd640 1 -- 172.21.15.161:0/948071657 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1320005480 con 0x7f133014a550 2024-03-28T23:33:43.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.380+0000 7f1326ffd640 1 -- 172.21.15.161:0/948071657 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1320001370 con 0x7f133014a550 2024-03-28T23:33:43.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.380+0000 7f133866f640 1 -- 172.21.15.161:0/948071657 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1330072ac0 con 0x7f133014a550 2024-03-28T23:33:43.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.380+0000 7f133866f640 1 -- 172.21.15.161:0/948071657 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1330071750 con 0x7f133014a550 2024-03-28T23:33:43.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.381+0000 7f133866f640 1 -- 172.21.15.161:0/948071657 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f133004f4b0 con 0x7f133014a550 2024-03-28T23:33:43.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.381+0000 7f1326ffd640 1 -- 172.21.15.161:0/948071657 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1320044020 con 0x7f133014a550 2024-03-28T23:33:43.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.382+0000 7f1326ffd640 1 --2- 172.21.15.161:0/948071657 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f131406b4d0 0x7f131406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:43.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.382+0000 7f1326ffd640 1 -- 172.21.15.161:0/948071657 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5421+0+0 (secure 0 0 0) 0x7f13200ab910 con 0x7f133014a550 2024-03-28T23:33:43.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.382+0000 7f133640b640 1 --2- 172.21.15.161:0/948071657 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f131406b4d0 0x7f131406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:43.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.382+0000 7f133640b640 1 --2- 172.21.15.161:0/948071657 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f131406b4d0 0x7f131406d990 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f132c0012f0 tx=0x7f132c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:43.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.386+0000 7f1326ffd640 1 -- 172.21.15.161:0/948071657 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f132004b030 con 0x7f133014a550 2024-03-28T23:33:43.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.557+0000 7f133866f640 1 -- 172.21.15.161:0/948071657 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"} v 0) -- 0x7f13300c4210 con 0x7f133014a550 2024-03-28T23:33:43.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.562+0000 7f1326ffd640 1 -- 172.21.15.161:0/948071657 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"}]=0 v34) ==== 96+0+0 (secure 0 0 0) 0x7f13200303c0 con 0x7f133014a550 2024-03-28T23:33:43.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.564+0000 7f133866f640 1 -- 172.21.15.161:0/948071657 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f131406b4d0 msgr2=0x7f131406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:43.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.564+0000 7f133866f640 1 --2- 172.21.15.161:0/948071657 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f131406b4d0 0x7f131406d990 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f132c0012f0 tx=0x7f132c009040 comp rx=0 tx=0).stop 2024-03-28T23:33:43.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.564+0000 7f133866f640 1 -- 172.21.15.161:0/948071657 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f133014a550 msgr2=0x7f13301487c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:43.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.564+0000 7f133866f640 1 --2- 172.21.15.161:0/948071657 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f133014a550 0x7f13301487c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f1320000c00 tx=0x7f1320001120 comp rx=0 tx=0).stop 2024-03-28T23:33:43.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.564+0000 7f133866f640 1 -- 172.21.15.161:0/948071657 shutdown_connections 2024-03-28T23:33:43.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.564+0000 7f133866f640 1 --2- 172.21.15.161:0/948071657 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f131406b4d0 0x7f131406d990 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.564+0000 7f133866f640 1 --2- 172.21.15.161:0/948071657 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f133014a550 0x7f13301487c0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.564+0000 7f133866f640 1 --2- 172.21.15.161:0/948071657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1330148d00 0x7f13300c45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.564+0000 7f133866f640 1 --2- 172.21.15.161:0/948071657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f131c004800 0x7f1330072580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.564+0000 7f133866f640 1 -- 172.21.15.161:0/948071657 >> 172.21.15.161:0/948071657 conn(0x7f13300a83f0 msgr2=0x7f13301dd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.564+0000 7f133866f640 1 -- 172.21.15.161:0/948071657 shutdown_connections 2024-03-28T23:33:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.565+0000 7f133866f640 1 -- 172.21.15.161:0/948071657 wait complete. 2024-03-28T23:33:43.576 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime' 2024-03-28T23:33:43.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.677+0000 7fdf96e3b640 1 Processor -- start 2024-03-28T23:33:43.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.677+0000 7fdf96e3b640 1 -- start start 2024-03-28T23:33:43.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.678+0000 7fdf96e3b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fdf900c5850 0x7fdf90155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:43.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.678+0000 7fdf96e3b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf900c5d00 con 0x7fdf90152890 2024-03-28T23:33:43.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.678+0000 7fdf96e3b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf90158a30 con 0x7fdf900c5410 2024-03-28T23:33:43.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.678+0000 7fdf96e3b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fdf900c8480 con 0x7fdf900c5850 2024-03-28T23:33:43.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.678+0000 7fdf94bd7640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fdf90152890 0x7fdf900c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:55918/0 (socket says 172.21.15.161:55918) 2024-03-28T23:33:43.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.678+0000 7fdf94bd7640 1 -- 172.21.15.161:0/1780206499 learned_addr learned my addr 172.21.15.161:0/1780206499 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:43.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.678+0000 7fdf8f7fe640 1 -- 172.21.15.161:0/1780206499 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 373851548 0 0) 0x7fdf900c5d00 con 0x7fdf90152890 2024-03-28T23:33:43.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.679+0000 7fdf8f7fe640 1 -- 172.21.15.161:0/1780206499 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf78003520 con 0x7fdf90152890 2024-03-28T23:33:43.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.679+0000 7fdf953d8640 1 --2- 172.21.15.161:0/1780206499 >> v2:172.21.15.170:3300/0 conn(0x7fdf900c5850 0x7fdf90155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:43.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.679+0000 7fdf8f7fe640 1 -- 172.21.15.161:0/1780206499 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2682440969 0 0) 0x7fdf78003520 con 0x7fdf90152890 2024-03-28T23:33:43.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.679+0000 7fdf8f7fe640 1 -- 172.21.15.161:0/1780206499 >> v2:172.21.15.170:3300/0 conn(0x7fdf900c5850 msgr2=0x7fdf90155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:43.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.679+0000 7fdf8f7fe640 1 --2- 172.21.15.161:0/1780206499 >> v2:172.21.15.170:3300/0 conn(0x7fdf900c5850 0x7fdf90155ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.679+0000 7fdf8f7fe640 1 -- 172.21.15.161:0/1780206499 >> v1:172.21.15.170:6789/0 conn(0x7fdf900c5410 legacy=0x7fdf901553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:43.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.679+0000 7fdf8f7fe640 1 -- 172.21.15.161:0/1780206499 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf90158d10 con 0x7fdf90152890 2024-03-28T23:33:43.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.679+0000 7fdf8f7fe640 1 -- 172.21.15.161:0/1780206499 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fdf80001670 con 0x7fdf90152890 2024-03-28T23:33:43.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.679+0000 7fdf8f7fe640 1 -- 172.21.15.161:0/1780206499 >> v1:172.21.15.161:6789/0 conn(0x7fdf90152890 legacy=0x7fdf900c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:43.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.679+0000 7fdf8f7fe640 1 --2- 172.21.15.161:0/1780206499 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf78003a90 0x7fdf78003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:43.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.679+0000 7fdf8f7fe640 1 --2- 172.21.15.161:0/1780206499 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf78004800 0x7fdf78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:43.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.679+0000 7fdf8f7fe640 1 --2- 172.21.15.161:0/1780206499 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf780053c0 0x7fdf7800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:43.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.679+0000 7fdf8f7fe640 1 -- 172.21.15.161:0/1780206499 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7800c250 con 0x7fdf78004800 2024-03-28T23:33:43.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.679+0000 7fdf8f7fe640 1 -- 172.21.15.161:0/1780206499 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7800c400 con 0x7fdf78003a90 2024-03-28T23:33:43.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.679+0000 7fdf8f7fe640 1 -- 172.21.15.161:0/1780206499 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdf7800c600 con 0x7fdf780053c0 2024-03-28T23:33:43.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.679+0000 7fdf8ffff640 1 --2- 172.21.15.161:0/1780206499 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf78004800 0x7fdf78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:43.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.679+0000 7fdf94bd7640 1 --2- 172.21.15.161:0/1780206499 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf78003a90 0x7fdf78003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:43.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.679+0000 7fdf953d8640 1 --2- 172.21.15.161:0/1780206499 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf780053c0 0x7fdf7800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:43.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.679+0000 7fdf8ffff640 1 -- 172.21.15.161:0/1780206499 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf780053c0 msgr2=0x7fdf7800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:43.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.679+0000 7fdf8ffff640 1 --2- 172.21.15.161:0/1780206499 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf780053c0 0x7fdf7800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.679+0000 7fdf8ffff640 1 -- 172.21.15.161:0/1780206499 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf78003a90 msgr2=0x7fdf78003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:43.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.679+0000 7fdf8ffff640 1 --2- 172.21.15.161:0/1780206499 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf78003a90 0x7fdf78003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.679+0000 7fdf8ffff640 1 -- 172.21.15.161:0/1780206499 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf78003520 con 0x7fdf78004800 2024-03-28T23:33:43.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.679+0000 7fdf953d8640 1 -- 172.21.15.161:0/1780206499 reap_dead start 2024-03-28T23:33:43.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.680+0000 7fdf8ffff640 1 --2- 172.21.15.161:0/1780206499 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf78004800 0x7fdf78004c80 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fdf900ad1f0 tx=0x7fdf78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=998e6e9e45c97e5f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:43.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.680+0000 7fdf8f7fe640 1 -- 172.21.15.161:0/1780206499 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf840013c0 con 0x7fdf78004800 2024-03-28T23:33:43.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.680+0000 7fdf8f7fe640 1 -- 172.21.15.161:0/1780206499 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf840052d0 con 0x7fdf78004800 2024-03-28T23:33:43.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.680+0000 7fdf8f7fe640 1 -- 172.21.15.161:0/1780206499 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf84003640 con 0x7fdf78004800 2024-03-28T23:33:43.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.681+0000 7fdf96e3b640 1 -- 172.21.15.161:0/1780206499 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf78004800 msgr2=0x7fdf78004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:43.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.681+0000 7fdf96e3b640 1 --2- 172.21.15.161:0/1780206499 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf78004800 0x7fdf78004c80 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fdf900ad1f0 tx=0x7fdf78005830 comp rx=0 tx=0).stop 2024-03-28T23:33:43.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.681+0000 7fdf96e3b640 1 -- 172.21.15.161:0/1780206499 shutdown_connections 2024-03-28T23:33:43.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.681+0000 7fdf96e3b640 1 --2- 172.21.15.161:0/1780206499 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf78004800 0x7fdf78004c80 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.681+0000 7fdf96e3b640 1 -- 172.21.15.161:0/1780206499 >> 172.21.15.161:0/1780206499 conn(0x7fdf900ac620 msgr2=0x7fdf900aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:43.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.681+0000 7fdf96e3b640 1 -- 172.21.15.161:0/1780206499 shutdown_connections 2024-03-28T23:33:43.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.681+0000 7fdf96e3b640 1 -- 172.21.15.161:0/1780206499 wait complete. 2024-03-28T23:33:43.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.682+0000 7fdf96e3b640 1 Processor -- start 2024-03-28T23:33:43.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.682+0000 7fdf96e3b640 1 -- start start 2024-03-28T23:33:43.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.683+0000 7fdf96e3b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf78004800 0x7fdf90155b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:43.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.683+0000 7fdf96e3b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf900c3f70 0x7fdf900c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:43.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.683+0000 7fdf96e3b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf900c4930 0x7fdf90157640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:43.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.683+0000 7fdf96e3b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdf901592d0 con 0x7fdf78004800 2024-03-28T23:33:43.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.683+0000 7fdf96e3b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdf90159640 con 0x7fdf900c3f70 2024-03-28T23:33:43.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.683+0000 7fdf96e3b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdf90152c70 con 0x7fdf900c4930 2024-03-28T23:33:43.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.683+0000 7fdf94bd7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf78004800 0x7fdf90155b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:43.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.683+0000 7fdf94bd7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf78004800 0x7fdf90155b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57148/0 (socket says 172.21.15.161:57148) 2024-03-28T23:33:43.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.683+0000 7fdf94bd7640 1 -- 172.21.15.161:0/1443350792 learned_addr learned my addr 172.21.15.161:0/1443350792 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:43.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.683+0000 7fdf8ffff640 1 --2- 172.21.15.161:0/1443350792 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf900c3f70 0x7fdf900c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:43.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.683+0000 7fdf953d8640 1 --2- 172.21.15.161:0/1443350792 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf900c4930 0x7fdf90157640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:43.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.683+0000 7fdf94bd7640 1 -- 172.21.15.161:0/1443350792 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf900c4930 msgr2=0x7fdf90157640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:43.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.683+0000 7fdf94bd7640 1 --2- 172.21.15.161:0/1443350792 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf900c4930 0x7fdf90157640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.683+0000 7fdf94bd7640 1 -- 172.21.15.161:0/1443350792 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf900c3f70 msgr2=0x7fdf900c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:43.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.683+0000 7fdf94bd7640 1 --2- 172.21.15.161:0/1443350792 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf900c3f70 0x7fdf900c43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.683+0000 7fdf94bd7640 1 -- 172.21.15.161:0/1443350792 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf900c1bd0 con 0x7fdf78004800 2024-03-28T23:33:43.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.683+0000 7fdf8ffff640 1 --2- 172.21.15.161:0/1443350792 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf900c3f70 0x7fdf900c43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:43.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.683+0000 7fdf94bd7640 1 --2- 172.21.15.161:0/1443350792 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf78004800 0x7fdf90155b60 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fdf800072c0 tx=0x7fdf80007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:43.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.683+0000 7fdf8d7fa640 1 -- 172.21.15.161:0/1443350792 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8001a070 con 0x7fdf78004800 2024-03-28T23:33:43.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.684+0000 7fdf8d7fa640 1 -- 172.21.15.161:0/1443350792 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf80001660 con 0x7fdf78004800 2024-03-28T23:33:43.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.684+0000 7fdf96e3b640 1 -- 172.21.15.161:0/1443350792 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf90156e80 con 0x7fdf78004800 2024-03-28T23:33:43.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.684+0000 7fdf8d7fa640 1 -- 172.21.15.161:0/1443350792 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8000fb00 con 0x7fdf78004800 2024-03-28T23:33:43.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.684+0000 7fdf96e3b640 1 -- 172.21.15.161:0/1443350792 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf90157120 con 0x7fdf78004800 2024-03-28T23:33:43.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.684+0000 7fdf96e3b640 1 -- 172.21.15.161:0/1443350792 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf9004f4b0 con 0x7fdf78004800 2024-03-28T23:33:43.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.685+0000 7fdf8d7fa640 1 -- 172.21.15.161:0/1443350792 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdf8001d070 con 0x7fdf78004800 2024-03-28T23:33:43.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.685+0000 7fdf8d7fa640 1 --2- 172.21.15.161:0/1443350792 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdf7c06b4d0 0x7fdf7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:43.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.686+0000 7fdf8d7fa640 1 -- 172.21.15.161:0/1443350792 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5421+0+0 (secure 0 0 0) 0x7fdf80089890 con 0x7fdf78004800 2024-03-28T23:33:43.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.686+0000 7fdf8ffff640 1 --2- 172.21.15.161:0/1443350792 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdf7c06b4d0 0x7fdf7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:43.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.686+0000 7fdf8ffff640 1 --2- 172.21.15.161:0/1443350792 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdf7c06b4d0 0x7fdf7c06d990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fdf84033000 tx=0x7fdf84009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:43.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.690+0000 7fdf8d7fa640 1 -- 172.21.15.161:0/1443350792 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdf800588c0 con 0x7fdf78004800 2024-03-28T23:33:43.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.937+0000 7fdf96e3b640 1 -- 172.21.15.161:0/1443350792 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime"]} v 0) -- 0x7fdf900c8140 con 0x7fdf78004800 2024-03-28T23:33:43.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.949+0000 7fdf8d7fa640 1 -- 172.21.15.161:0/1443350792 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime"]}]=0 v236) ==== 146+0+0 (secure 0 0 0) 0x7fdf800581c0 con 0x7fdf78004800 2024-03-28T23:33:43.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.951+0000 7fdf96e3b640 1 -- 172.21.15.161:0/1443350792 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdf7c06b4d0 msgr2=0x7fdf7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:43.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.951+0000 7fdf96e3b640 1 --2- 172.21.15.161:0/1443350792 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdf7c06b4d0 0x7fdf7c06d990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fdf84033000 tx=0x7fdf84009030 comp rx=0 tx=0).stop 2024-03-28T23:33:43.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.951+0000 7fdf96e3b640 1 -- 172.21.15.161:0/1443350792 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf78004800 msgr2=0x7fdf90155b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:43.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.951+0000 7fdf96e3b640 1 --2- 172.21.15.161:0/1443350792 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf78004800 0x7fdf90155b60 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fdf800072c0 tx=0x7fdf80007780 comp rx=0 tx=0).stop 2024-03-28T23:33:43.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.951+0000 7fdf96e3b640 1 -- 172.21.15.161:0/1443350792 shutdown_connections 2024-03-28T23:33:43.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.951+0000 7fdf96e3b640 1 --2- 172.21.15.161:0/1443350792 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdf7c06b4d0 0x7fdf7c06d990 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.951+0000 7fdf96e3b640 1 --2- 172.21.15.161:0/1443350792 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf900c4930 0x7fdf90157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.951+0000 7fdf96e3b640 1 --2- 172.21.15.161:0/1443350792 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf900c3f70 0x7fdf900c43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.951+0000 7fdf96e3b640 1 --2- 172.21.15.161:0/1443350792 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf78004800 0x7fdf90155b60 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:43.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.951+0000 7fdf96e3b640 1 -- 172.21.15.161:0/1443350792 >> 172.21.15.161:0/1443350792 conn(0x7fdf900ac620 msgr2=0x7fdf9014c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:43.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.951+0000 7fdf96e3b640 1 -- 172.21.15.161:0/1443350792 shutdown_connections 2024-03-28T23:33:43.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:43.951+0000 7fdf96e3b640 1 -- 172.21.15.161:0/1443350792 wait complete. 2024-03-28T23:33:43.963 INFO:tasks.cephfs_test_runner:That setting mds_join_fs at runtime affects the cluster layout. ... ok 2024-03-28T23:33:43.963 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_standby_replay (tasks.cephfs.test_failover.TestClusterAffinity) That a standby-replay daemon with weak affinity is replaced by a stronger one. 2024-03-28T23:33:43.963 INFO:tasks.cephfs_test_runner:test_join_fs_standby_replay (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-28T23:33:43.963 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:33:44.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.064+0000 7f3f3e41c640 1 Processor -- start 2024-03-28T23:33:44.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.064+0000 7f3f3e41c640 1 -- start start 2024-03-28T23:33:44.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.064+0000 7f3f3e41c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f38152890 0x7f3f380c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:44.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.064+0000 7f3f3e41c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3f380c8ed0 0x7f3f380c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:44.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.064+0000 7f3f3e41c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3f380c5950 0x7f3f381553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:44.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.064+0000 7f3f3e41c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f3f380c5db0 con 0x7f3f380c8ed0 2024-03-28T23:33:44.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.064+0000 7f3f3e41c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f3f380c8480 con 0x7f3f380c5950 2024-03-28T23:33:44.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.064+0000 7f3f3e41c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3f38159640 con 0x7f3f38152890 2024-03-28T23:33:44.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.064+0000 7f3f377fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3f380c8ed0 0x7f3f380c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.064+0000 7f3f377fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3f380c8ed0 0x7f3f380c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57158/0 (socket says 172.21.15.161:57158) 2024-03-28T23:33:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.065+0000 7f3f377fe640 1 -- 172.21.15.161:0/3874059119 learned_addr learned my addr 172.21.15.161:0/3874059119 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.065+0000 7f3f37fff640 1 --2- 172.21.15.161:0/3874059119 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f38152890 0x7f3f380c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.065+0000 7f3f3c9b9640 1 --2- 172.21.15.161:0/3874059119 >> v2:172.21.15.170:3300/0 conn(0x7f3f380c5950 0x7f3f381553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.065+0000 7f3f377fe640 1 -- 172.21.15.161:0/3874059119 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f38152890 msgr2=0x7f3f380c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.065+0000 7f3f377fe640 1 --2- 172.21.15.161:0/3874059119 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f38152890 0x7f3f380c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.065+0000 7f3f377fe640 1 -- 172.21.15.161:0/3874059119 >> v2:172.21.15.170:3300/0 conn(0x7f3f380c5950 msgr2=0x7f3f381553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.065+0000 7f3f377fe640 1 --2- 172.21.15.161:0/3874059119 >> v2:172.21.15.170:3300/0 conn(0x7f3f380c5950 0x7f3f381553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.065+0000 7f3f377fe640 1 -- 172.21.15.161:0/3874059119 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f38158a30 con 0x7f3f380c8ed0 2024-03-28T23:33:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.066+0000 7f3f377fe640 1 --2- 172.21.15.161:0/3874059119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f380c8ed0 0x7f3f380c5410 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f3f380c4e90 tx=0x7f3f2c02b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e32a88266e146ea server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:44.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.066+0000 7f3f36ffd640 1 -- 172.21.15.161:0/3874059119 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c038070 con 0x7f3f380c8ed0 2024-03-28T23:33:44.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.066+0000 7f3f36ffd640 1 -- 172.21.15.161:0/3874059119 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f2c004030 con 0x7f3f380c8ed0 2024-03-28T23:33:44.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.066+0000 7f3f36ffd640 1 -- 172.21.15.161:0/3874059119 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c004460 con 0x7f3f380c8ed0 2024-03-28T23:33:44.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.066+0000 7f3f3e41c640 1 -- 172.21.15.161:0/3874059119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f380c8ed0 msgr2=0x7f3f380c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:44.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.066+0000 7f3f3e41c640 1 --2- 172.21.15.161:0/3874059119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f380c8ed0 0x7f3f380c5410 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f3f380c4e90 tx=0x7f3f2c02b7c0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.066+0000 7f3f3e41c640 1 -- 172.21.15.161:0/3874059119 shutdown_connections 2024-03-28T23:33:44.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.066+0000 7f3f3e41c640 1 --2- 172.21.15.161:0/3874059119 >> v2:172.21.15.170:3300/0 conn(0x7f3f380c5950 0x7f3f381553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.066+0000 7f3f3e41c640 1 --2- 172.21.15.161:0/3874059119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f380c8ed0 0x7f3f380c5410 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.066+0000 7f3f3e41c640 1 --2- 172.21.15.161:0/3874059119 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f38152890 0x7f3f380c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.066+0000 7f3f3e41c640 1 -- 172.21.15.161:0/3874059119 >> 172.21.15.161:0/3874059119 conn(0x7f3f380ac620 msgr2=0x7f3f380aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:44.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.066+0000 7f3f3e41c640 1 -- 172.21.15.161:0/3874059119 shutdown_connections 2024-03-28T23:33:44.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.066+0000 7f3f3e41c640 1 -- 172.21.15.161:0/3874059119 wait complete. 2024-03-28T23:33:44.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.067+0000 7f3f3e41c640 1 Processor -- start 2024-03-28T23:33:44.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.068+0000 7f3f3e41c640 1 -- start start 2024-03-28T23:33:44.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.068+0000 7f3f3e41c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f380c5950 0x7f3f381eb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:44.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.068+0000 7f3f3e41c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f380c8ed0 0x7f3f3814e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:44.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.068+0000 7f3f3e41c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f38152890 0x7f3f380ab6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:44.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.068+0000 7f3f3e41c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3f381592d0 con 0x7f3f38152890 2024-03-28T23:33:44.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.068+0000 7f3f3e41c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3f38158ff0 con 0x7f3f380c8ed0 2024-03-28T23:33:44.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.068+0000 7f3f3e41c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3f38152c70 con 0x7f3f380c5950 2024-03-28T23:33:44.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.068+0000 7f3f3c9b9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f38152890 0x7f3f380ab6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:44.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.068+0000 7f3f3c9b9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f38152890 0x7f3f380ab6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57172/0 (socket says 172.21.15.161:57172) 2024-03-28T23:33:44.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.068+0000 7f3f3c9b9640 1 -- 172.21.15.161:0/3323844289 learned_addr learned my addr 172.21.15.161:0/3323844289 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:44.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.068+0000 7f3f377fe640 1 --2- 172.21.15.161:0/3323844289 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f380c8ed0 0x7f3f3814e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:44.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.069+0000 7f3f37fff640 1 --2- 172.21.15.161:0/3323844289 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f380c5950 0x7f3f381eb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:44.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.069+0000 7f3f3c9b9640 1 -- 172.21.15.161:0/3323844289 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f380c5950 msgr2=0x7f3f381eb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:44.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.069+0000 7f3f3c9b9640 1 --2- 172.21.15.161:0/3323844289 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f380c5950 0x7f3f381eb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.069+0000 7f3f3c9b9640 1 -- 172.21.15.161:0/3323844289 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f380c8ed0 msgr2=0x7f3f3814e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:44.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.069+0000 7f3f3c9b9640 1 --2- 172.21.15.161:0/3323844289 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f380c8ed0 0x7f3f3814e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.069+0000 7f3f3c9b9640 1 -- 172.21.15.161:0/3323844289 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f3814f0b0 con 0x7f3f38152890 2024-03-28T23:33:44.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.069+0000 7f3f3c9b9640 1 --2- 172.21.15.161:0/3323844289 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f38152890 0x7f3f380ab6d0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f3f30007d20 tx=0x7f3f3000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:44.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.069+0000 7f3f34ff9640 1 -- 172.21.15.161:0/3323844289 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f30014420 con 0x7f3f38152890 2024-03-28T23:33:44.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.069+0000 7f3f34ff9640 1 -- 172.21.15.161:0/3323844289 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f30019070 con 0x7f3f38152890 2024-03-28T23:33:44.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.069+0000 7f3f3e41c640 1 -- 172.21.15.161:0/3323844289 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f3814ccd0 con 0x7f3f38152890 2024-03-28T23:33:44.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.069+0000 7f3f34ff9640 1 -- 172.21.15.161:0/3323844289 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f30014820 con 0x7f3f38152890 2024-03-28T23:33:44.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.069+0000 7f3f3e41c640 1 -- 172.21.15.161:0/3323844289 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f3814d290 con 0x7f3f38152890 2024-03-28T23:33:44.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.070+0000 7f3f3e41c640 1 -- 172.21.15.161:0/3323844289 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f3804f4b0 con 0x7f3f38152890 2024-03-28T23:33:44.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.071+0000 7f3f34ff9640 1 -- 172.21.15.161:0/3323844289 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3f300149c0 con 0x7f3f38152890 2024-03-28T23:33:44.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.071+0000 7f3f34ff9640 1 --2- 172.21.15.161:0/3323844289 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3f2406b500 0x7f3f2406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:44.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.071+0000 7f3f34ff9640 1 -- 172.21.15.161:0/3323844289 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5421+0+0 (secure 0 0 0) 0x7f3f3008a6b0 con 0x7f3f38152890 2024-03-28T23:33:44.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.071+0000 7f3f37fff640 1 --2- 172.21.15.161:0/3323844289 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3f2406b500 0x7f3f2406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:44.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.072+0000 7f3f37fff640 1 --2- 172.21.15.161:0/3323844289 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3f2406b500 0x7f3f2406d9c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3f28006fd0 tx=0x7f3f28008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:44.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.076+0000 7f3f34ff9640 1 -- 172.21.15.161:0/3323844289 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3f30059090 con 0x7f3f38152890 2024-03-28T23:33:44.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.247+0000 7f3f3e41c640 1 -- 172.21.15.161:0/3323844289 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3f38156570 con 0x7f3f38152890 2024-03-28T23:33:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.248+0000 7f3f34ff9640 1 -- 172.21.15.161:0/3323844289 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) ==== 74+0+13465 (secure 0 0 0) 0x7f3f3005d130 con 0x7f3f38152890 2024-03-28T23:33:44.248 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:44.248 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":100,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:33:42.939524+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6835/3079114491":"2024-03-29T23:33:39.932292+0000","172.21.15.170:6833/3079114491":"2024-03-29T23:33:39.932292+0000","172.21.15.170:6834/2176248563":"2024-03-29T23:33:31.422647+0000","172.21.15.170:6832/2176248563":"2024-03-29T23:33:31.422647+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:33:44.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.250+0000 7f3f3e41c640 1 -- 172.21.15.161:0/3323844289 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3f2406b500 msgr2=0x7f3f2406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:44.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.250+0000 7f3f3e41c640 1 --2- 172.21.15.161:0/3323844289 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3f2406b500 0x7f3f2406d9c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3f28006fd0 tx=0x7f3f28008040 comp rx=0 tx=0).stop 2024-03-28T23:33:44.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.250+0000 7f3f3e41c640 1 -- 172.21.15.161:0/3323844289 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f38152890 msgr2=0x7f3f380ab6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:44.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.250+0000 7f3f3e41c640 1 --2- 172.21.15.161:0/3323844289 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f38152890 0x7f3f380ab6d0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f3f30007d20 tx=0x7f3f3000e4b0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.250+0000 7f3f3e41c640 1 -- 172.21.15.161:0/3323844289 shutdown_connections 2024-03-28T23:33:44.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.250+0000 7f3f3e41c640 1 --2- 172.21.15.161:0/3323844289 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3f2406b500 0x7f3f2406d9c0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.250+0000 7f3f3e41c640 1 --2- 172.21.15.161:0/3323844289 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f38152890 0x7f3f380ab6d0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.250+0000 7f3f3e41c640 1 --2- 172.21.15.161:0/3323844289 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f380c8ed0 0x7f3f3814e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.250+0000 7f3f3e41c640 1 --2- 172.21.15.161:0/3323844289 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f380c5950 0x7f3f381eb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.250+0000 7f3f3e41c640 1 -- 172.21.15.161:0/3323844289 >> 172.21.15.161:0/3323844289 conn(0x7f3f380ac620 msgr2=0x7f3f38150ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:44.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.250+0000 7f3f3e41c640 1 -- 172.21.15.161:0/3323844289 shutdown_connections 2024-03-28T23:33:44.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.250+0000 7f3f3e41c640 1 -- 172.21.15.161:0/3323844289 wait complete. 2024-03-28T23:33:44.262 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:33:44.262 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:33:44.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.366+0000 7fcd46113640 1 Processor -- start 2024-03-28T23:33:44.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.366+0000 7fcd46113640 1 -- start start 2024-03-28T23:33:44.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.367+0000 7fcd46113640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fcd400c1290 0x7fcd4015e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:44.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.367+0000 7fcd46113640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd400a6290 con 0x7fcd400c5dd0 2024-03-28T23:33:44.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.367+0000 7fcd46113640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd401532a0 con 0x7fcd4006ea30 2024-03-28T23:33:44.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.367+0000 7fcd46113640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fcd400c61b0 con 0x7fcd400c1290 2024-03-28T23:33:44.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.367+0000 7fcd3f7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fcd400c5dd0 0x7fcd400a9940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:55934/0 (socket says 172.21.15.161:55934) 2024-03-28T23:33:44.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.367+0000 7fcd3f7fe640 1 -- 172.21.15.161:0/3751508980 learned_addr learned my addr 172.21.15.161:0/3751508980 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:44.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.367+0000 7fcd3e7fc640 1 -- 172.21.15.161:0/3751508980 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1159014006 0 0) 0x7fcd400a6290 con 0x7fcd400c5dd0 2024-03-28T23:33:44.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.367+0000 7fcd3ffff640 1 --2- 172.21.15.161:0/3751508980 >> v2:172.21.15.170:3300/0 conn(0x7fcd400c1290 0x7fcd4015e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:44.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.367+0000 7fcd3e7fc640 1 -- 172.21.15.161:0/3751508980 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd28003520 con 0x7fcd400c5dd0 2024-03-28T23:33:44.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.368+0000 7fcd3e7fc640 1 -- 172.21.15.161:0/3751508980 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4057692682 0 0) 0x7fcd28003520 con 0x7fcd400c5dd0 2024-03-28T23:33:44.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.368+0000 7fcd3e7fc640 1 -- 172.21.15.161:0/3751508980 >> v2:172.21.15.170:3300/0 conn(0x7fcd400c1290 msgr2=0x7fcd4015e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:44.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.368+0000 7fcd3e7fc640 1 --2- 172.21.15.161:0/3751508980 >> v2:172.21.15.170:3300/0 conn(0x7fcd400c1290 0x7fcd4015e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.368+0000 7fcd3e7fc640 1 -- 172.21.15.161:0/3751508980 >> v1:172.21.15.170:6789/0 conn(0x7fcd4006ea30 legacy=0x7fcd400c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:44.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.368+0000 7fcd3e7fc640 1 -- 172.21.15.161:0/3751508980 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd400abaa0 con 0x7fcd400c5dd0 2024-03-28T23:33:44.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.368+0000 7fcd3e7fc640 1 -- 172.21.15.161:0/3751508980 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fcd30001670 con 0x7fcd400c5dd0 2024-03-28T23:33:44.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.368+0000 7fcd3e7fc640 1 -- 172.21.15.161:0/3751508980 >> v1:172.21.15.161:6789/0 conn(0x7fcd400c5dd0 legacy=0x7fcd400a9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:44.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.368+0000 7fcd3e7fc640 1 --2- 172.21.15.161:0/3751508980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd280039b0 0x7fcd28003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:44.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.368+0000 7fcd3e7fc640 1 --2- 172.21.15.161:0/3751508980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd28004800 0x7fcd28004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:44.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.368+0000 7fcd3e7fc640 1 --2- 172.21.15.161:0/3751508980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd280051c0 0x7fcd2800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:44.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.368+0000 7fcd3e7fc640 1 -- 172.21.15.161:0/3751508980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcd400c61b0 con 0x7fcd280051c0 2024-03-28T23:33:44.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.368+0000 7fcd3e7fc640 1 -- 172.21.15.161:0/3751508980 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fcd2800c060 con 0x7fcd280039b0 2024-03-28T23:33:44.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.368+0000 7fcd3e7fc640 1 -- 172.21.15.161:0/3751508980 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fcd2800c260 con 0x7fcd28004800 2024-03-28T23:33:44.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.368+0000 7fcd3ffff640 1 --2- 172.21.15.161:0/3751508980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd280051c0 0x7fcd2800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:44.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.368+0000 7fcd3f7fe640 1 --2- 172.21.15.161:0/3751508980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd280039b0 0x7fcd28003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:44.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.368+0000 7fcd3effd640 1 --2- 172.21.15.161:0/3751508980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd28004800 0x7fcd28004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:44.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.368+0000 7fcd3ffff640 1 -- 172.21.15.161:0/3751508980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd28004800 msgr2=0x7fcd28004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:44.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.368+0000 7fcd3ffff640 1 --2- 172.21.15.161:0/3751508980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd28004800 0x7fcd28004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.368+0000 7fcd3ffff640 1 -- 172.21.15.161:0/3751508980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd280039b0 msgr2=0x7fcd28003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:44.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.368+0000 7fcd3ffff640 1 --2- 172.21.15.161:0/3751508980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd280039b0 0x7fcd28003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.368+0000 7fcd3ffff640 1 -- 172.21.15.161:0/3751508980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd28003520 con 0x7fcd280051c0 2024-03-28T23:33:44.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.369+0000 7fcd3ffff640 1 -- 172.21.15.161:0/3751508980 reap_dead start 2024-03-28T23:33:44.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.369+0000 7fcd3ffff640 1 --2- 172.21.15.161:0/3751508980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd280051c0 0x7fcd2800b9c0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fcd38009f90 tx=0x7fcd28005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd37eaac8f43d06a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:44.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.369+0000 7fcd3e7fc640 1 -- 172.21.15.161:0/3751508980 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd38039070 con 0x7fcd280051c0 2024-03-28T23:33:44.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.369+0000 7fcd3e7fc640 1 -- 172.21.15.161:0/3751508980 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd3802cc20 con 0x7fcd280051c0 2024-03-28T23:33:44.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.369+0000 7fcd3e7fc640 1 -- 172.21.15.161:0/3751508980 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd38009c10 con 0x7fcd280051c0 2024-03-28T23:33:44.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.369+0000 7fcd46113640 1 -- 172.21.15.161:0/3751508980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd280051c0 msgr2=0x7fcd2800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:44.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.369+0000 7fcd46113640 1 --2- 172.21.15.161:0/3751508980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd280051c0 0x7fcd2800b9c0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fcd38009f90 tx=0x7fcd28005830 comp rx=0 tx=0).stop 2024-03-28T23:33:44.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.370+0000 7fcd46113640 1 -- 172.21.15.161:0/3751508980 shutdown_connections 2024-03-28T23:33:44.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.370+0000 7fcd46113640 1 --2- 172.21.15.161:0/3751508980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd280051c0 0x7fcd2800b9c0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.370+0000 7fcd46113640 1 -- 172.21.15.161:0/3751508980 >> 172.21.15.161:0/3751508980 conn(0x7fcd400c3390 msgr2=0x7fcd400c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:44.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.370+0000 7fcd46113640 1 -- 172.21.15.161:0/3751508980 shutdown_connections 2024-03-28T23:33:44.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.370+0000 7fcd46113640 1 -- 172.21.15.161:0/3751508980 wait complete. 2024-03-28T23:33:44.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.371+0000 7fcd46113640 1 Processor -- start 2024-03-28T23:33:44.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.371+0000 7fcd46113640 1 -- start start 2024-03-28T23:33:44.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.371+0000 7fcd46113640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd280051c0 0x7fcd400bcee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:44.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.371+0000 7fcd46113640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd401570c0 0x7fcd400bbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:44.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.371+0000 7fcd46113640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd401e9140 0x7fcd400bc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:44.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.371+0000 7fcd46113640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcd400ac340 con 0x7fcd401e9140 2024-03-28T23:33:44.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.371+0000 7fcd46113640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fcd400ac060 con 0x7fcd401570c0 2024-03-28T23:33:44.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.371+0000 7fcd46113640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fcd40152f50 con 0x7fcd280051c0 2024-03-28T23:33:44.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.372+0000 7fcd3ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd401e9140 0x7fcd400bc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:44.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.372+0000 7fcd3ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd401e9140 0x7fcd400bc3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57184/0 (socket says 172.21.15.161:57184) 2024-03-28T23:33:44.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.372+0000 7fcd3ffff640 1 -- 172.21.15.161:0/3281439142 learned_addr learned my addr 172.21.15.161:0/3281439142 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:44.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.372+0000 7fcd3effd640 1 --2- 172.21.15.161:0/3281439142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd401570c0 0x7fcd400bbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:44.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.372+0000 7fcd3ffff640 1 -- 172.21.15.161:0/3281439142 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd280051c0 msgr2=0x7fcd400bcee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:44.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.372+0000 7fcd3f7fe640 1 --2- 172.21.15.161:0/3281439142 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd280051c0 0x7fcd400bcee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:44.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.372+0000 7fcd3ffff640 1 --2- 172.21.15.161:0/3281439142 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd280051c0 0x7fcd400bcee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.372+0000 7fcd3ffff640 1 -- 172.21.15.161:0/3281439142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd401570c0 msgr2=0x7fcd400bbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:44.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.372+0000 7fcd3ffff640 1 --2- 172.21.15.161:0/3281439142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd401570c0 0x7fcd400bbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.372+0000 7fcd3ffff640 1 -- 172.21.15.161:0/3281439142 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd401e6350 con 0x7fcd401e9140 2024-03-28T23:33:44.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.372+0000 7fcd3effd640 1 --2- 172.21.15.161:0/3281439142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd401570c0 0x7fcd400bbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:44.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.372+0000 7fcd3ffff640 1 --2- 172.21.15.161:0/3281439142 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd401e9140 0x7fcd400bc3b0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fcd3802c690 tx=0x7fcd38009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:44.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.372+0000 7fcd27fff640 1 -- 172.21.15.161:0/3281439142 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd38042070 con 0x7fcd401e9140 2024-03-28T23:33:44.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.372+0000 7fcd27fff640 1 -- 172.21.15.161:0/3281439142 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd3803e480 con 0x7fcd401e9140 2024-03-28T23:33:44.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.372+0000 7fcd46113640 1 -- 172.21.15.161:0/3281439142 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd401e95d0 con 0x7fcd401e9140 2024-03-28T23:33:44.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.372+0000 7fcd27fff640 1 -- 172.21.15.161:0/3281439142 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd38039040 con 0x7fcd401e9140 2024-03-28T23:33:44.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.372+0000 7fcd46113640 1 -- 172.21.15.161:0/3281439142 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd40156560 con 0x7fcd401e9140 2024-03-28T23:33:44.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.373+0000 7fcd46113640 1 -- 172.21.15.161:0/3281439142 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd4004f4b0 con 0x7fcd401e9140 2024-03-28T23:33:44.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.374+0000 7fcd27fff640 1 -- 172.21.15.161:0/3281439142 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcd38045070 con 0x7fcd401e9140 2024-03-28T23:33:44.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.374+0000 7fcd27fff640 1 --2- 172.21.15.161:0/3281439142 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcd2c06b4d0 0x7fcd2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:44.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.374+0000 7fcd27fff640 1 -- 172.21.15.161:0/3281439142 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5421+0+0 (secure 0 0 0) 0x7fcd380ac360 con 0x7fcd401e9140 2024-03-28T23:33:44.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.374+0000 7fcd3f7fe640 1 --2- 172.21.15.161:0/3281439142 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcd2c06b4d0 0x7fcd2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:44.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.375+0000 7fcd3f7fe640 1 --2- 172.21.15.161:0/3281439142 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcd2c06b4d0 0x7fcd2c06d990 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fcd400a8510 tx=0x7fcd30009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:44.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.379+0000 7fcd27fff640 1 -- 172.21.15.161:0/3281439142 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcd401e6350 con 0x7fcd401e9140 2024-03-28T23:33:44.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.548+0000 7fcd46113640 1 -- 172.21.15.161:0/3281439142 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcd400ab340 con 0x7fcd401e9140 2024-03-28T23:33:44.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.549+0000 7fcd27fff640 1 -- 172.21.15.161:0/3281439142 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) ==== 93+0+10 (secure 0 0 0) 0x7fcd3807ad40 con 0x7fcd401e9140 2024-03-28T23:33:44.549 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:33:44.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.550+0000 7fcd46113640 1 -- 172.21.15.161:0/3281439142 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcd2c06b4d0 msgr2=0x7fcd2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:44.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.550+0000 7fcd46113640 1 --2- 172.21.15.161:0/3281439142 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcd2c06b4d0 0x7fcd2c06d990 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fcd400a8510 tx=0x7fcd30009040 comp rx=0 tx=0).stop 2024-03-28T23:33:44.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.550+0000 7fcd46113640 1 -- 172.21.15.161:0/3281439142 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd401e9140 msgr2=0x7fcd400bc3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:44.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.550+0000 7fcd46113640 1 --2- 172.21.15.161:0/3281439142 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd401e9140 0x7fcd400bc3b0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fcd3802c690 tx=0x7fcd38009750 comp rx=0 tx=0).stop 2024-03-28T23:33:44.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.550+0000 7fcd46113640 1 -- 172.21.15.161:0/3281439142 shutdown_connections 2024-03-28T23:33:44.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.551+0000 7fcd46113640 1 --2- 172.21.15.161:0/3281439142 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcd2c06b4d0 0x7fcd2c06d990 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.551+0000 7fcd46113640 1 --2- 172.21.15.161:0/3281439142 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd401e9140 0x7fcd400bc3b0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.551+0000 7fcd46113640 1 --2- 172.21.15.161:0/3281439142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd401570c0 0x7fcd400bbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.551+0000 7fcd46113640 1 --2- 172.21.15.161:0/3281439142 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd280051c0 0x7fcd400bcee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.551+0000 7fcd46113640 1 -- 172.21.15.161:0/3281439142 >> 172.21.15.161:0/3281439142 conn(0x7fcd400c3390 msgr2=0x7fcd4014ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:44.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.551+0000 7fcd46113640 1 -- 172.21.15.161:0/3281439142 shutdown_connections 2024-03-28T23:33:44.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.551+0000 7fcd46113640 1 -- 172.21.15.161:0/3281439142 wait complete. 2024-03-28T23:33:44.562 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay' 2024-03-28T23:33:44.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.664+0000 7f145206f640 1 Processor -- start 2024-03-28T23:33:44.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.664+0000 7f145206f640 1 -- start start 2024-03-28T23:33:44.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.665+0000 7f145206f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f144c06e4e0 0x7f144c0bfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:44.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.665+0000 7f145206f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f144c0c03e0 0x7f144c0a5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:44.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.665+0000 7f145206f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f144c0c6040 con 0x7f144c152730 2024-03-28T23:33:44.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.665+0000 7f145206f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f144c0c6320 con 0x7f144c0c03e0 2024-03-28T23:33:44.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.665+0000 7f145206f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f144c0c6600 con 0x7f144c06e4e0 2024-03-28T23:33:44.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.665+0000 7f144bfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f144c0c03e0 0x7f144c0a5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:44.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.665+0000 7f144b7fe640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f144c152730 0x7f144c1551a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:34142/0 (socket says 172.21.15.161:34142) 2024-03-28T23:33:44.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.665+0000 7f144b7fe640 1 -- 172.21.15.161:0/3319702111 learned_addr learned my addr 172.21.15.161:0/3319702111 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:44.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.665+0000 7f144affd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f144c06e4e0 0x7f144c0bfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:44.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.665+0000 7f144bfff640 1 -- 172.21.15.161:0/3319702111 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f144c06e4e0 msgr2=0x7f144c0bfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:44.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.665+0000 7f144bfff640 1 --2- 172.21.15.161:0/3319702111 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f144c06e4e0 0x7f144c0bfea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.665+0000 7f144bfff640 1 -- 172.21.15.161:0/3319702111 >> v1:172.21.15.170:6789/0 conn(0x7f144c152730 legacy=0x7f144c1551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:44.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.665+0000 7f144bfff640 1 -- 172.21.15.161:0/3319702111 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f144c06dce0 con 0x7f144c0c03e0 2024-03-28T23:33:44.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.665+0000 7f144a7fc640 1 -- 172.21.15.161:0/3319702111 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3015965124 0 0) 0x7f144c0c6040 con 0x7f144c152730 2024-03-28T23:33:44.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.665+0000 7f144affd640 1 --2- 172.21.15.161:0/3319702111 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f144c06e4e0 0x7f144c0bfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:44.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.666+0000 7f144bfff640 1 --2- 172.21.15.161:0/3319702111 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f144c0c03e0 0x7f144c0a5d50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1444009d90 tx=0x7f144402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de66380c183e0dbb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:44.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.666+0000 7f144a7fc640 1 -- 172.21.15.161:0/3319702111 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1444039070 con 0x7f144c0c03e0 2024-03-28T23:33:44.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.666+0000 7f144a7fc640 1 -- 172.21.15.161:0/3319702111 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f144402cbc0 con 0x7f144c0c03e0 2024-03-28T23:33:44.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.666+0000 7f144a7fc640 1 -- 172.21.15.161:0/3319702111 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1444034440 con 0x7f144c0c03e0 2024-03-28T23:33:44.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.667+0000 7f145206f640 1 -- 172.21.15.161:0/3319702111 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f144c0c03e0 msgr2=0x7f144c0a5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:44.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.667+0000 7f145206f640 1 --2- 172.21.15.161:0/3319702111 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f144c0c03e0 0x7f144c0a5d50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1444009d90 tx=0x7f144402be10 comp rx=0 tx=0).stop 2024-03-28T23:33:44.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.667+0000 7f145206f640 1 -- 172.21.15.161:0/3319702111 shutdown_connections 2024-03-28T23:33:44.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.667+0000 7f145206f640 1 --2- 172.21.15.161:0/3319702111 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f144c0c03e0 0x7f144c0a5d50 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.667+0000 7f145206f640 1 --2- 172.21.15.161:0/3319702111 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f144c06e4e0 0x7f144c0bfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.667+0000 7f145206f640 1 -- 172.21.15.161:0/3319702111 >> 172.21.15.161:0/3319702111 conn(0x7f144c0c6b10 msgr2=0x7f144c0c6ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:44.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.667+0000 7f145206f640 1 -- 172.21.15.161:0/3319702111 shutdown_connections 2024-03-28T23:33:44.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.667+0000 7f145206f640 1 -- 172.21.15.161:0/3319702111 wait complete. 2024-03-28T23:33:44.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.668+0000 7f145206f640 1 Processor -- start 2024-03-28T23:33:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.668+0000 7f145206f640 1 -- start start 2024-03-28T23:33:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.669+0000 7f145206f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f144c06e4e0 0x7f144c072170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.669+0000 7f145206f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f144c0c03e0 0x7f144c1490f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.669+0000 7f145206f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f144c152730 0x7f144c154250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.669+0000 7f145206f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f144c0c5d10 con 0x7f144c152730 2024-03-28T23:33:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.669+0000 7f145206f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f144c06e960 con 0x7f144c06e4e0 2024-03-28T23:33:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.669+0000 7f145206f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f144c152b10 con 0x7f144c0c03e0 2024-03-28T23:33:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.669+0000 7f144bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f144c152730 0x7f144c154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.669+0000 7f144bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f144c152730 0x7f144c154250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57192/0 (socket says 172.21.15.161:57192) 2024-03-28T23:33:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.669+0000 7f144bfff640 1 -- 172.21.15.161:0/1947490638 learned_addr learned my addr 172.21.15.161:0/1947490638 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.669+0000 7f144affd640 1 --2- 172.21.15.161:0/1947490638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f144c0c03e0 0x7f144c1490f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.669+0000 7f144b7fe640 1 --2- 172.21.15.161:0/1947490638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f144c06e4e0 0x7f144c072170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.669+0000 7f144bfff640 1 -- 172.21.15.161:0/1947490638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f144c0c03e0 msgr2=0x7f144c1490f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.669+0000 7f144bfff640 1 --2- 172.21.15.161:0/1947490638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f144c0c03e0 0x7f144c1490f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.669+0000 7f144bfff640 1 -- 172.21.15.161:0/1947490638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f144c06e4e0 msgr2=0x7f144c072170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.669+0000 7f144bfff640 1 --2- 172.21.15.161:0/1947490638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f144c06e4e0 0x7f144c072170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.669+0000 7f144bfff640 1 -- 172.21.15.161:0/1947490638 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f144c155fa0 con 0x7f144c152730 2024-03-28T23:33:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.669+0000 7f144bfff640 1 --2- 172.21.15.161:0/1947490638 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f144c152730 0x7f144c154250 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f1444009d90 tx=0x7f1444004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.670+0000 7f1433fff640 1 -- 172.21.15.161:0/1947490638 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1444039050 con 0x7f144c152730 2024-03-28T23:33:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.670+0000 7f1433fff640 1 -- 172.21.15.161:0/1947490638 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1444045070 con 0x7f144c152730 2024-03-28T23:33:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.670+0000 7f145206f640 1 -- 172.21.15.161:0/1947490638 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f144c1550c0 con 0x7f144c152730 2024-03-28T23:33:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.670+0000 7f1433fff640 1 -- 172.21.15.161:0/1947490638 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1444009450 con 0x7f144c152730 2024-03-28T23:33:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.670+0000 7f145206f640 1 -- 172.21.15.161:0/1947490638 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f144c071450 con 0x7f144c152730 2024-03-28T23:33:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.671+0000 7f145206f640 1 -- 172.21.15.161:0/1947490638 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f144c1e83c0 con 0x7f144c152730 2024-03-28T23:33:44.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.671+0000 7f1433fff640 1 -- 172.21.15.161:0/1947490638 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f144403e850 con 0x7f144c152730 2024-03-28T23:33:44.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.671+0000 7f1433fff640 1 --2- 172.21.15.161:0/1947490638 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f143806b4d0 0x7f143806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:44.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.672+0000 7f1433fff640 1 -- 172.21.15.161:0/1947490638 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5421+0+0 (secure 0 0 0) 0x7f14440acd30 con 0x7f144c152730 2024-03-28T23:33:44.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.672+0000 7f144b7fe640 1 --2- 172.21.15.161:0/1947490638 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f143806b4d0 0x7f143806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:44.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.672+0000 7f144b7fe640 1 --2- 172.21.15.161:0/1947490638 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f143806b4d0 0x7f143806d990 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f1440000f60 tx=0x7f1440007c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:44.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.676+0000 7f1433fff640 1 -- 172.21.15.161:0/1947490638 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1444032e20 con 0x7f144c152730 2024-03-28T23:33:44.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.914+0000 7f145206f640 1 -- 172.21.15.161:0/1947490638 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay"]} v 0) -- 0x7f144c04f550 con 0x7f144c152730 2024-03-28T23:33:44.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.950+0000 7f1433fff640 1 -- 172.21.15.161:0/1947490638 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay"]}]=0 v237) ==== 156+0+0 (secure 0 0 0) 0x7f144407bd60 con 0x7f144c152730 2024-03-28T23:33:44.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.952+0000 7f145206f640 1 -- 172.21.15.161:0/1947490638 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f143806b4d0 msgr2=0x7f143806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:44.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.952+0000 7f145206f640 1 --2- 172.21.15.161:0/1947490638 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f143806b4d0 0x7f143806d990 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f1440000f60 tx=0x7f1440007c00 comp rx=0 tx=0).stop 2024-03-28T23:33:44.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.952+0000 7f145206f640 1 -- 172.21.15.161:0/1947490638 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f144c152730 msgr2=0x7f144c154250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:44.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.952+0000 7f145206f640 1 --2- 172.21.15.161:0/1947490638 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f144c152730 0x7f144c154250 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f1444009d90 tx=0x7f1444004a40 comp rx=0 tx=0).stop 2024-03-28T23:33:44.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.952+0000 7f145206f640 1 -- 172.21.15.161:0/1947490638 shutdown_connections 2024-03-28T23:33:44.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.952+0000 7f145206f640 1 --2- 172.21.15.161:0/1947490638 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f143806b4d0 0x7f143806d990 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.952+0000 7f145206f640 1 --2- 172.21.15.161:0/1947490638 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f144c152730 0x7f144c154250 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.952+0000 7f145206f640 1 --2- 172.21.15.161:0/1947490638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f144c0c03e0 0x7f144c1490f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.952+0000 7f145206f640 1 --2- 172.21.15.161:0/1947490638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f144c06e4e0 0x7f144c072170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:44.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.952+0000 7f145206f640 1 -- 172.21.15.161:0/1947490638 >> 172.21.15.161:0/1947490638 conn(0x7f144c0c6b10 msgr2=0x7f144c14f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:44.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.953+0000 7f145206f640 1 -- 172.21.15.161:0/1947490638 shutdown_connections 2024-03-28T23:33:44.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:44.953+0000 7f145206f640 1 -- 172.21.15.161:0/1947490638 wait complete. 2024-03-28T23:33:44.964 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-28T23:33:45.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.067+0000 7fa61e9a1640 1 Processor -- start 2024-03-28T23:33:45.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.067+0000 7fa61e9a1640 1 -- start start 2024-03-28T23:33:45.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.068+0000 7fa61e9a1640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa6180c0330 0x7fa6180c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:45.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.068+0000 7fa61e9a1640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6180c6f60 con 0x7fa618152bc0 2024-03-28T23:33:45.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.068+0000 7fa61e9a1640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6180c6120 con 0x7fa6181589b0 2024-03-28T23:33:45.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.068+0000 7fa61e9a1640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fa6180c6400 con 0x7fa6180c0330 2024-03-28T23:33:45.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.068+0000 7fa617fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa618152bc0 0x7fa6180c8a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:55948/0 (socket says 172.21.15.161:55948) 2024-03-28T23:33:45.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.068+0000 7fa617fff640 1 -- 172.21.15.161:0/1776144077 learned_addr learned my addr 172.21.15.161:0/1776144077 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:45.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.069+0000 7fa616ffd640 1 -- 172.21.15.161:0/1776144077 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3061008991 0 0) 0x7fa6180c6f60 con 0x7fa618152bc0 2024-03-28T23:33:45.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.069+0000 7fa616ffd640 1 -- 172.21.15.161:0/1776144077 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa600003520 con 0x7fa618152bc0 2024-03-28T23:33:45.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.069+0000 7fa6177fe640 1 --2- 172.21.15.161:0/1776144077 >> v2:172.21.15.170:3300/0 conn(0x7fa6180c0330 0x7fa6180c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:45.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.069+0000 7fa616ffd640 1 -- 172.21.15.161:0/1776144077 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3875085967 0 0) 0x7fa600003520 con 0x7fa618152bc0 2024-03-28T23:33:45.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.069+0000 7fa616ffd640 1 -- 172.21.15.161:0/1776144077 >> v2:172.21.15.170:3300/0 conn(0x7fa6180c0330 msgr2=0x7fa6180c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:45.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.069+0000 7fa616ffd640 1 --2- 172.21.15.161:0/1776144077 >> v2:172.21.15.170:3300/0 conn(0x7fa6180c0330 0x7fa6180c9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.069+0000 7fa616ffd640 1 -- 172.21.15.161:0/1776144077 >> v1:172.21.15.170:6789/0 conn(0x7fa6181589b0 legacy=0x7fa618158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:45.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.069+0000 7fa616ffd640 1 -- 172.21.15.161:0/1776144077 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6180c96d0 con 0x7fa618152bc0 2024-03-28T23:33:45.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.069+0000 7fa616ffd640 1 -- 172.21.15.161:0/1776144077 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fa608001670 con 0x7fa618152bc0 2024-03-28T23:33:45.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.069+0000 7fa616ffd640 1 -- 172.21.15.161:0/1776144077 >> v1:172.21.15.161:6789/0 conn(0x7fa618152bc0 legacy=0x7fa6180c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:45.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.069+0000 7fa616ffd640 1 --2- 172.21.15.161:0/1776144077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa600003a90 0x7fa600003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:45.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.069+0000 7fa616ffd640 1 --2- 172.21.15.161:0/1776144077 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa600004800 0x7fa600004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:45.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.069+0000 7fa616ffd640 1 --2- 172.21.15.161:0/1776144077 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa600005210 0x7fa60000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:45.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.069+0000 7fa616ffd640 1 -- 172.21.15.161:0/1776144077 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa6180c6400 con 0x7fa600004800 2024-03-28T23:33:45.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.069+0000 7fa616ffd640 1 -- 172.21.15.161:0/1776144077 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa60000c140 con 0x7fa600003a90 2024-03-28T23:33:45.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.069+0000 7fa616ffd640 1 -- 172.21.15.161:0/1776144077 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa60000c340 con 0x7fa600005210 2024-03-28T23:33:45.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.069+0000 7fa6177fe640 1 --2- 172.21.15.161:0/1776144077 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa600004800 0x7fa600004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:45.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.069+0000 7fa617fff640 1 --2- 172.21.15.161:0/1776144077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa600003a90 0x7fa600003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:45.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.069+0000 7fa61cf3e640 1 --2- 172.21.15.161:0/1776144077 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa600005210 0x7fa60000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:45.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.069+0000 7fa6177fe640 1 -- 172.21.15.161:0/1776144077 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa600005210 msgr2=0x7fa60000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:45.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.069+0000 7fa6177fe640 1 --2- 172.21.15.161:0/1776144077 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa600005210 0x7fa60000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.069+0000 7fa6177fe640 1 -- 172.21.15.161:0/1776144077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa600003a90 msgr2=0x7fa600003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:45.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.069+0000 7fa6177fe640 1 --2- 172.21.15.161:0/1776144077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa600003a90 0x7fa600003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.069+0000 7fa6177fe640 1 -- 172.21.15.161:0/1776144077 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa600003520 con 0x7fa600004800 2024-03-28T23:33:45.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.070+0000 7fa61cf3e640 1 -- 172.21.15.161:0/1776144077 reap_dead start 2024-03-28T23:33:45.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.070+0000 7fa6177fe640 1 --2- 172.21.15.161:0/1776144077 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa600004800 0x7fa600004c80 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fa60c009a90 tx=0x7fa600005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2b32d98b822eba9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:45.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.070+0000 7fa616ffd640 1 -- 172.21.15.161:0/1776144077 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60c039070 con 0x7fa600004800 2024-03-28T23:33:45.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.070+0000 7fa616ffd640 1 -- 172.21.15.161:0/1776144077 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa60c02c660 con 0x7fa600004800 2024-03-28T23:33:45.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.070+0000 7fa616ffd640 1 -- 172.21.15.161:0/1776144077 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60c009690 con 0x7fa600004800 2024-03-28T23:33:45.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.071+0000 7fa61e9a1640 1 -- 172.21.15.161:0/1776144077 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa600004800 msgr2=0x7fa600004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:45.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.071+0000 7fa61e9a1640 1 --2- 172.21.15.161:0/1776144077 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa600004800 0x7fa600004c80 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fa60c009a90 tx=0x7fa600005830 comp rx=0 tx=0).stop 2024-03-28T23:33:45.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.071+0000 7fa61e9a1640 1 -- 172.21.15.161:0/1776144077 shutdown_connections 2024-03-28T23:33:45.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.071+0000 7fa61e9a1640 1 --2- 172.21.15.161:0/1776144077 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa600004800 0x7fa600004c80 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.071+0000 7fa61e9a1640 1 -- 172.21.15.161:0/1776144077 >> 172.21.15.161:0/1776144077 conn(0x7fa6180caa80 msgr2=0x7fa6180cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:45.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.071+0000 7fa61e9a1640 1 -- 172.21.15.161:0/1776144077 shutdown_connections 2024-03-28T23:33:45.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.071+0000 7fa61e9a1640 1 -- 172.21.15.161:0/1776144077 wait complete. 2024-03-28T23:33:45.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.072+0000 7fa61e9a1640 1 Processor -- start 2024-03-28T23:33:45.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.072+0000 7fa61e9a1640 1 -- start start 2024-03-28T23:33:45.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.073+0000 7fa61e9a1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa600004800 0x7fa6180afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:45.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.073+0000 7fa61e9a1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa618145880 0x7fa6181edff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:45.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.073+0000 7fa61e9a1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa618151e50 0x7fa6181503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:45.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.073+0000 7fa61e9a1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa618152fa0 con 0x7fa618151e50 2024-03-28T23:33:45.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.073+0000 7fa61e9a1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa6180c6bf0 con 0x7fa618145880 2024-03-28T23:33:45.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.073+0000 7fa61e9a1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa6180af660 con 0x7fa600004800 2024-03-28T23:33:45.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.073+0000 7fa61cf3e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa618151e50 0x7fa6181503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:45.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.073+0000 7fa61cf3e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa618151e50 0x7fa6181503e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57202/0 (socket says 172.21.15.161:57202) 2024-03-28T23:33:45.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.073+0000 7fa61cf3e640 1 -- 172.21.15.161:0/70336482 learned_addr learned my addr 172.21.15.161:0/70336482 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:45.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.073+0000 7fa6177fe640 1 --2- 172.21.15.161:0/70336482 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa618145880 0x7fa6181edff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:45.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.073+0000 7fa617fff640 1 --2- 172.21.15.161:0/70336482 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa600004800 0x7fa6180afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:45.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.073+0000 7fa61cf3e640 1 -- 172.21.15.161:0/70336482 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa600004800 msgr2=0x7fa6180afaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:45.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.073+0000 7fa61cf3e640 1 --2- 172.21.15.161:0/70336482 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa600004800 0x7fa6180afaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.073+0000 7fa61cf3e640 1 -- 172.21.15.161:0/70336482 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa618145880 msgr2=0x7fa6181edff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:45.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.073+0000 7fa61cf3e640 1 --2- 172.21.15.161:0/70336482 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa618145880 0x7fa6181edff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.073+0000 7fa61cf3e640 1 -- 172.21.15.161:0/70336482 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6180b0030 con 0x7fa618151e50 2024-03-28T23:33:45.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.074+0000 7fa61cf3e640 1 --2- 172.21.15.161:0/70336482 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa618151e50 0x7fa6181503e0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fa6180c0fc0 tx=0x7fa61000fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:45.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.074+0000 7fa614ff9640 1 -- 172.21.15.161:0/70336482 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa610003b80 con 0x7fa618151e50 2024-03-28T23:33:45.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.074+0000 7fa614ff9640 1 -- 172.21.15.161:0/70336482 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa610018070 con 0x7fa618151e50 2024-03-28T23:33:45.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.074+0000 7fa61e9a1640 1 -- 172.21.15.161:0/70336482 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6180b0550 con 0x7fa618151e50 2024-03-28T23:33:45.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.074+0000 7fa614ff9640 1 -- 172.21.15.161:0/70336482 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6100050d0 con 0x7fa618151e50 2024-03-28T23:33:45.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.074+0000 7fa61e9a1640 1 -- 172.21.15.161:0/70336482 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6180af2f0 con 0x7fa618151e50 2024-03-28T23:33:45.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.075+0000 7fa61e9a1640 1 -- 172.21.15.161:0/70336482 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa61804f4b0 con 0x7fa618151e50 2024-03-28T23:33:45.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.075+0000 7fa614ff9640 1 -- 172.21.15.161:0/70336482 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa610005270 con 0x7fa618151e50 2024-03-28T23:33:45.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.076+0000 7fa614ff9640 1 --2- 172.21.15.161:0/70336482 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa60406b4d0 0x7fa60406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:45.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.076+0000 7fa614ff9640 1 -- 172.21.15.161:0/70336482 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5421+0+0 (secure 0 0 0) 0x7fa610089d70 con 0x7fa618151e50 2024-03-28T23:33:45.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.076+0000 7fa617fff640 1 --2- 172.21.15.161:0/70336482 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa60406b4d0 0x7fa60406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:45.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.076+0000 7fa617fff640 1 --2- 172.21.15.161:0/70336482 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa60406b4d0 0x7fa60406d990 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa61814cce0 tx=0x7fa608000f80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:45.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.080+0000 7fa614ff9640 1 -- 172.21.15.161:0/70336482 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa610058750 con 0x7fa618151e50 2024-03-28T23:33:45.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.251+0000 7fa61e9a1640 1 -- 172.21.15.161:0/70336482 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fa6181ee530 con 0x7fa618151e50 2024-03-28T23:33:45.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.254+0000 7fa614ff9640 1 -- 172.21.15.161:0/70336482 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v35)=0 v35) ==== 122+0+0 (secure 0 0 0) 0x7fa61005c7f0 con 0x7fa618151e50 2024-03-28T23:33:45.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.256+0000 7fa61e9a1640 1 -- 172.21.15.161:0/70336482 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa60406b4d0 msgr2=0x7fa60406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:45.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.256+0000 7fa61e9a1640 1 --2- 172.21.15.161:0/70336482 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa60406b4d0 0x7fa60406d990 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa61814cce0 tx=0x7fa608000f80 comp rx=0 tx=0).stop 2024-03-28T23:33:45.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.256+0000 7fa61e9a1640 1 -- 172.21.15.161:0/70336482 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa618151e50 msgr2=0x7fa6181503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:45.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.256+0000 7fa61e9a1640 1 --2- 172.21.15.161:0/70336482 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa618151e50 0x7fa6181503e0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fa6180c0fc0 tx=0x7fa61000fd40 comp rx=0 tx=0).stop 2024-03-28T23:33:45.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.256+0000 7fa61e9a1640 1 -- 172.21.15.161:0/70336482 shutdown_connections 2024-03-28T23:33:45.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.256+0000 7fa61e9a1640 1 --2- 172.21.15.161:0/70336482 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa60406b4d0 0x7fa60406d990 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.256+0000 7fa61e9a1640 1 --2- 172.21.15.161:0/70336482 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa618151e50 0x7fa6181503e0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.256+0000 7fa61e9a1640 1 --2- 172.21.15.161:0/70336482 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa618145880 0x7fa6181edff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.256+0000 7fa61e9a1640 1 --2- 172.21.15.161:0/70336482 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa600004800 0x7fa6180afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.256+0000 7fa61e9a1640 1 -- 172.21.15.161:0/70336482 >> 172.21.15.161:0/70336482 conn(0x7fa6180caa80 msgr2=0x7fa6180cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:45.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.256+0000 7fa61e9a1640 1 -- 172.21.15.161:0/70336482 shutdown_connections 2024-03-28T23:33:45.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.256+0000 7fa61e9a1640 1 -- 172.21.15.161:0/70336482 wait complete. 2024-03-28T23:33:45.268 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-28T23:33:45.268 DEBUG:teuthology.orchestra.run.smithi161:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:33:45.313 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:33:45.356 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-28T23:33:45.356 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:33:45.357 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:33:45.372 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20793 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-28T23:33:45.373 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:33:45.373 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:33:45.414 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056374,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19690 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-28T23:33:45.414 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-28T23:33:45.414 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:33:45.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.516+0000 7fea97e8f640 1 Processor -- start 2024-03-28T23:33:45.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.516+0000 7fea97e8f640 1 -- start start 2024-03-28T23:33:45.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.517+0000 7fea97e8f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fea900a8fc0 0x7fea90153150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:45.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.517+0000 7fea97e8f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea901566f0 con 0x7fea90150590 2024-03-28T23:33:45.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.517+0000 7fea97e8f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea901569d0 con 0x7fea900aacb0 2024-03-28T23:33:45.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.517+0000 7fea97e8f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fea90156cb0 con 0x7fea900a8fc0 2024-03-28T23:33:45.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.517+0000 7fea95c2b640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fea90150590 0x7fea900a9f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:55962/0 (socket says 172.21.15.161:55962) 2024-03-28T23:33:45.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.517+0000 7fea95c2b640 1 -- 172.21.15.161:0/1779868198 learned_addr learned my addr 172.21.15.161:0/1779868198 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:45.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.517+0000 7fea9542a640 1 --2- 172.21.15.161:0/1779868198 >> v2:172.21.15.170:3300/0 conn(0x7fea900a8fc0 0x7fea90153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:45.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.517+0000 7fea94c29640 1 -- 172.21.15.161:0/1779868198 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1899374150 0 0) 0x7fea901566f0 con 0x7fea90150590 2024-03-28T23:33:45.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.517+0000 7fea94c29640 1 -- 172.21.15.161:0/1779868198 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea7c003520 con 0x7fea90150590 2024-03-28T23:33:45.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.517+0000 7fea94c29640 1 -- 172.21.15.161:0/1779868198 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2226251395 0 0) 0x7fea901569d0 con 0x7fea900aacb0 2024-03-28T23:33:45.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.517+0000 7fea94c29640 1 -- 172.21.15.161:0/1779868198 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea901566f0 con 0x7fea900aacb0 2024-03-28T23:33:45.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.517+0000 7fea94c29640 1 -- 172.21.15.161:0/1779868198 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 163946918 0 0) 0x7fea7c003520 con 0x7fea90150590 2024-03-28T23:33:45.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.517+0000 7fea94c29640 1 -- 172.21.15.161:0/1779868198 >> v2:172.21.15.170:3300/0 conn(0x7fea900a8fc0 msgr2=0x7fea90153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:45.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.517+0000 7fea94c29640 1 --2- 172.21.15.161:0/1779868198 >> v2:172.21.15.170:3300/0 conn(0x7fea900a8fc0 0x7fea90153150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.517+0000 7fea94c29640 1 -- 172.21.15.161:0/1779868198 >> v1:172.21.15.170:6789/0 conn(0x7fea900aacb0 legacy=0x7fea90153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:45.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.517+0000 7fea94c29640 1 -- 172.21.15.161:0/1779868198 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea90156f90 con 0x7fea90150590 2024-03-28T23:33:45.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.517+0000 7fea94c29640 1 -- 172.21.15.161:0/1779868198 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fea8c001670 con 0x7fea90150590 2024-03-28T23:33:45.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.517+0000 7fea94c29640 1 -- 172.21.15.161:0/1779868198 >> v1:172.21.15.161:6789/0 conn(0x7fea90150590 legacy=0x7fea900a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:45.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.517+0000 7fea94c29640 1 --2- 172.21.15.161:0/1779868198 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea7c003c70 0x7fea7c004070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:45.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.517+0000 7fea94c29640 1 --2- 172.21.15.161:0/1779868198 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea7c004800 0x7fea7c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:45.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.517+0000 7fea94c29640 1 --2- 172.21.15.161:0/1779868198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea7c005390 0x7fea7c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:45.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.518+0000 7fea94c29640 1 -- 172.21.15.161:0/1779868198 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fea90156cb0 con 0x7fea7c004800 2024-03-28T23:33:45.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.518+0000 7fea94c29640 1 -- 172.21.15.161:0/1779868198 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fea7c00c2a0 con 0x7fea7c005390 2024-03-28T23:33:45.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.518+0000 7fea94c29640 1 -- 172.21.15.161:0/1779868198 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fea7c00c4a0 con 0x7fea7c003c70 2024-03-28T23:33:45.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.518+0000 7fea9542a640 1 --2- 172.21.15.161:0/1779868198 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea7c004800 0x7fea7c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:45.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.518+0000 7fea9642c640 1 --2- 172.21.15.161:0/1779868198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea7c005390 0x7fea7c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:45.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.518+0000 7fea95c2b640 1 --2- 172.21.15.161:0/1779868198 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea7c003c70 0x7fea7c004070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:45.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.518+0000 7fea9542a640 1 -- 172.21.15.161:0/1779868198 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea7c003c70 msgr2=0x7fea7c004070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:45.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.518+0000 7fea9542a640 1 --2- 172.21.15.161:0/1779868198 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea7c003c70 0x7fea7c004070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.518+0000 7fea9542a640 1 -- 172.21.15.161:0/1779868198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea7c005390 msgr2=0x7fea7c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:45.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.518+0000 7fea9542a640 1 --2- 172.21.15.161:0/1779868198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea7c005390 0x7fea7c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.518+0000 7fea9642c640 1 -- 172.21.15.161:0/1779868198 reap_dead start 2024-03-28T23:33:45.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.518+0000 7fea9542a640 1 -- 172.21.15.161:0/1779868198 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea7c003520 con 0x7fea7c004800 2024-03-28T23:33:45.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.518+0000 7fea95c2b640 1 --2- 172.21.15.161:0/1779868198 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea7c003c70 0x7fea7c004070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:45.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.519+0000 7fea9542a640 1 --2- 172.21.15.161:0/1779868198 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea7c004800 0x7fea7c004c80 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fea80009a90 tx=0x7fea7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df48fcfae4b1da2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:45.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.519+0000 7fea94c29640 1 -- 172.21.15.161:0/1779868198 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea80039070 con 0x7fea7c004800 2024-03-28T23:33:45.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.519+0000 7fea94c29640 1 -- 172.21.15.161:0/1779868198 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea8002c680 con 0x7fea7c004800 2024-03-28T23:33:45.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.519+0000 7fea94c29640 1 -- 172.21.15.161:0/1779868198 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea80009690 con 0x7fea7c004800 2024-03-28T23:33:45.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.519+0000 7fea97e8f640 1 -- 172.21.15.161:0/1779868198 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea7c004800 msgr2=0x7fea7c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:45.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.519+0000 7fea97e8f640 1 --2- 172.21.15.161:0/1779868198 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea7c004800 0x7fea7c004c80 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fea80009a90 tx=0x7fea7c005830 comp rx=0 tx=0).stop 2024-03-28T23:33:45.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.519+0000 7fea97e8f640 1 -- 172.21.15.161:0/1779868198 shutdown_connections 2024-03-28T23:33:45.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.519+0000 7fea97e8f640 1 --2- 172.21.15.161:0/1779868198 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea7c004800 0x7fea7c004c80 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.519+0000 7fea97e8f640 1 -- 172.21.15.161:0/1779868198 >> 172.21.15.161:0/1779868198 conn(0x7fea900c6670 msgr2=0x7fea900c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:45.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.519+0000 7fea97e8f640 1 -- 172.21.15.161:0/1779868198 shutdown_connections 2024-03-28T23:33:45.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.519+0000 7fea97e8f640 1 -- 172.21.15.161:0/1779868198 wait complete. 2024-03-28T23:33:45.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.520+0000 7fea97e8f640 1 Processor -- start 2024-03-28T23:33:45.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.520+0000 7fea97e8f640 1 -- start start 2024-03-28T23:33:45.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.521+0000 7fea97e8f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea7c004800 0x7fea901ec7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:45.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.521+0000 7fea97e8f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea9013ec00 0x7fea9014bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:45.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.521+0000 7fea97e8f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea90147d20 0x7fea901481b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:45.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.521+0000 7fea97e8f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fea900a9f40 con 0x7fea7c004800 2024-03-28T23:33:45.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.521+0000 7fea97e8f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fea90150970 con 0x7fea9013ec00 2024-03-28T23:33:45.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.521+0000 7fea97e8f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fea900aa0c0 con 0x7fea90147d20 2024-03-28T23:33:45.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.521+0000 7fea95c2b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea7c004800 0x7fea901ec7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:45.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.521+0000 7fea95c2b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea7c004800 0x7fea901ec7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57216/0 (socket says 172.21.15.161:57216) 2024-03-28T23:33:45.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.521+0000 7fea95c2b640 1 -- 172.21.15.161:0/1027707175 learned_addr learned my addr 172.21.15.161:0/1027707175 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:45.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.521+0000 7fea9542a640 1 --2- 172.21.15.161:0/1027707175 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea9013ec00 0x7fea9014bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:45.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.521+0000 7fea9642c640 1 --2- 172.21.15.161:0/1027707175 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea90147d20 0x7fea901481b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:45.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.522+0000 7fea95c2b640 1 -- 172.21.15.161:0/1027707175 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea90147d20 msgr2=0x7fea901481b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:45.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.522+0000 7fea95c2b640 1 --2- 172.21.15.161:0/1027707175 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea90147d20 0x7fea901481b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.522+0000 7fea95c2b640 1 -- 172.21.15.161:0/1027707175 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea9013ec00 msgr2=0x7fea9014bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:45.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.522+0000 7fea95c2b640 1 --2- 172.21.15.161:0/1027707175 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea9013ec00 0x7fea9014bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.522+0000 7fea95c2b640 1 -- 172.21.15.161:0/1027707175 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea901e8ba0 con 0x7fea7c004800 2024-03-28T23:33:45.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.522+0000 7fea95c2b640 1 --2- 172.21.15.161:0/1027707175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea7c004800 0x7fea901ec7d0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fea8c000ff0 tx=0x7fea8c00e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:45.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.522+0000 7fea867fc640 1 -- 172.21.15.161:0/1027707175 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c012590 con 0x7fea7c004800 2024-03-28T23:33:45.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.522+0000 7fea867fc640 1 -- 172.21.15.161:0/1027707175 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea8c018070 con 0x7fea7c004800 2024-03-28T23:33:45.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.522+0000 7fea97e8f640 1 -- 172.21.15.161:0/1027707175 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea901e8e80 con 0x7fea7c004800 2024-03-28T23:33:45.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.522+0000 7fea867fc640 1 -- 172.21.15.161:0/1027707175 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c012930 con 0x7fea7c004800 2024-03-28T23:33:45.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.522+0000 7fea97e8f640 1 -- 172.21.15.161:0/1027707175 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea901e9440 con 0x7fea7c004800 2024-03-28T23:33:45.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.523+0000 7fea97e8f640 1 -- 172.21.15.161:0/1027707175 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea9004f4b0 con 0x7fea7c004800 2024-03-28T23:33:45.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.523+0000 7fea867fc640 1 -- 172.21.15.161:0/1027707175 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fea8c012ad0 con 0x7fea7c004800 2024-03-28T23:33:45.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.524+0000 7fea867fc640 1 --2- 172.21.15.161:0/1027707175 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fea7406b4d0 0x7fea7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:45.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.524+0000 7fea867fc640 1 -- 172.21.15.161:0/1027707175 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5421+0+0 (secure 0 0 0) 0x7fea8c089b10 con 0x7fea7c004800 2024-03-28T23:33:45.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.524+0000 7fea9542a640 1 --2- 172.21.15.161:0/1027707175 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fea7406b4d0 0x7fea7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:45.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.524+0000 7fea9542a640 1 --2- 172.21.15.161:0/1027707175 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fea7406b4d0 0x7fea7406d990 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fea80036000 tx=0x7fea80030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:45.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.528+0000 7fea867fc640 1 -- 172.21.15.161:0/1027707175 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fea8c0584f0 con 0x7fea7c004800 2024-03-28T23:33:45.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.735+0000 7fea97e8f640 1 -- 172.21.15.161:0/1027707175 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea9014d120 con 0x7fea7c004800 2024-03-28T23:33:45.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.735+0000 7fea867fc640 1 -- 172.21.15.161:0/1027707175 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+3875 (secure 0 0 0) 0x7fea8c05c590 con 0x7fea7c004800 2024-03-28T23:33:45.736 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:45.736 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":86,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6547,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4280217233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4280217233},{"type":"v1","addr":"172.21.15.161:6836","nonce":4280217233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":6706,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/2647418289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":2647418289},{"type":"v1","addr":"172.21.15.170:6834","nonce":2647418289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":6811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/646827056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":646827056},{"type":"v1","addr":"172.21.15.170:6835","nonce":646827056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":7086,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/404132405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":404132405},{"type":"v1","addr":"172.21.15.161:6837","nonce":404132405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[]} 2024-03-28T23:33:45.736 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 86 2024-03-28T23:33:45.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.737+0000 7fea97e8f640 1 -- 172.21.15.161:0/1027707175 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fea7406b4d0 msgr2=0x7fea7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:45.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.737+0000 7fea97e8f640 1 --2- 172.21.15.161:0/1027707175 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fea7406b4d0 0x7fea7406d990 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fea80036000 tx=0x7fea80030000 comp rx=0 tx=0).stop 2024-03-28T23:33:45.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.737+0000 7fea97e8f640 1 -- 172.21.15.161:0/1027707175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea7c004800 msgr2=0x7fea901ec7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:45.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.737+0000 7fea97e8f640 1 --2- 172.21.15.161:0/1027707175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea7c004800 0x7fea901ec7d0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fea8c000ff0 tx=0x7fea8c00e9e0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.737+0000 7fea97e8f640 1 -- 172.21.15.161:0/1027707175 shutdown_connections 2024-03-28T23:33:45.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.737+0000 7fea97e8f640 1 --2- 172.21.15.161:0/1027707175 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fea7406b4d0 0x7fea7406d990 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.737+0000 7fea97e8f640 1 --2- 172.21.15.161:0/1027707175 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea90147d20 0x7fea901481b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.737+0000 7fea97e8f640 1 --2- 172.21.15.161:0/1027707175 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea9013ec00 0x7fea9014bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.737+0000 7fea97e8f640 1 --2- 172.21.15.161:0/1027707175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea7c004800 0x7fea901ec7d0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.737+0000 7fea97e8f640 1 -- 172.21.15.161:0/1027707175 >> 172.21.15.161:0/1027707175 conn(0x7fea900c6670 msgr2=0x7fea900c0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:45.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.738+0000 7fea97e8f640 1 -- 172.21.15.161:0/1027707175 shutdown_connections 2024-03-28T23:33:45.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.738+0000 7fea97e8f640 1 -- 172.21.15.161:0/1027707175 wait complete. 2024-03-28T23:33:45.750 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T23:33:45.751 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-28T23:33:45.751 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T23:33:45.751 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:33:45.751 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T23:33:45.751 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-28T23:33:45.751 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T23:33:45.752 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:33:45.752 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-28T23:33:45.752 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-28T23:33:45.752 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-28T23:33:45.752 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:33:45.752 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-28T23:33:45.752 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-28T23:33:45.753 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-28T23:33:45.753 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:33:45.760 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T23:33:45.760 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T23:33:45.761 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T23:33:45.761 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T23:33:45.764 INFO:tasks.ceph.mds.b:Stopped 2024-03-28T23:33:45.764 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-28T23:33:45.765 INFO:tasks.ceph.mds.d:Stopped 2024-03-28T23:33:45.765 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-28T23:33:45.802 INFO:tasks.ceph.mds.a:Started 2024-03-28T23:33:45.803 DEBUG:teuthology.parallel:result is None 2024-03-28T23:33:45.803 INFO:tasks.ceph.mds.c:Started 2024-03-28T23:33:45.803 DEBUG:teuthology.parallel:result is None 2024-03-28T23:33:45.807 INFO:tasks.ceph.mds.b:Started 2024-03-28T23:33:45.807 DEBUG:teuthology.parallel:result is None 2024-03-28T23:33:45.808 INFO:tasks.ceph.mds.d:Started 2024-03-28T23:33:45.808 DEBUG:teuthology.parallel:result is None 2024-03-28T23:33:45.808 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-28T23:33:45.880 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-28T23:33:45.880 INFO:tasks.ceph.mds.c.smithi161.stdout:starting mds.c at 2024-03-28T23:33:45.890 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2024-03-28T23:33:45.890 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2024-03-28T23:33:45.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.914+0000 7f1850455640 1 Processor -- start 2024-03-28T23:33:45.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.914+0000 7f1850455640 1 -- start start 2024-03-28T23:33:45.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.915+0000 7f1850455640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f18480c3770 0x7f18480c1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:45.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.915+0000 7f1850455640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f18480c2300 0x7f184815fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:45.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.915+0000 7f1850455640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18480c3bd0 con 0x7f18480c3390 2024-03-28T23:33:45.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.915+0000 7f1850455640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1848155e80 con 0x7f18480c2300 2024-03-28T23:33:45.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.915+0000 7f1850455640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f18480a8410 con 0x7f18480c3770 2024-03-28T23:33:45.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.915+0000 7f184e1f1640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f18480c3390 0x7f18480a7d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:55988/0 (socket says 172.21.15.161:55988) 2024-03-28T23:33:45.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.915+0000 7f184e1f1640 1 -- 172.21.15.161:0/1808690829 learned_addr learned my addr 172.21.15.161:0/1808690829 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:45.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.915+0000 7f184d9f0640 1 --2- 172.21.15.161:0/1808690829 >> v2:172.21.15.170:3300/0 conn(0x7f18480c3770 0x7f18480c1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:45.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.915+0000 7f184e9f2640 1 --2- 172.21.15.161:0/1808690829 >> v2:172.21.15.161:3300/0 conn(0x7f18480c2300 0x7f184815fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:45.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.915+0000 7f184d1ef640 1 -- 172.21.15.161:0/1808690829 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2843669433 0 0) 0x7f18480c3bd0 con 0x7f18480c3390 2024-03-28T23:33:45.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.915+0000 7f184d1ef640 1 -- 172.21.15.161:0/1808690829 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1834003520 con 0x7f18480c3390 2024-03-28T23:33:45.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.915+0000 7f184d9f0640 1 -- 172.21.15.161:0/1808690829 >> v2:172.21.15.161:3300/0 conn(0x7f18480c2300 msgr2=0x7f184815fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:45.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.915+0000 7f184d9f0640 1 --2- 172.21.15.161:0/1808690829 >> v2:172.21.15.161:3300/0 conn(0x7f18480c2300 0x7f184815fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.915+0000 7f184d9f0640 1 -- 172.21.15.161:0/1808690829 >> v1:172.21.15.161:6789/0 conn(0x7f18480c3390 legacy=0x7f18480a7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:45.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.915+0000 7f184d9f0640 1 -- 172.21.15.161:0/1808690829 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1848155270 con 0x7f18480c3770 2024-03-28T23:33:45.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.916+0000 7f184d9f0640 1 --2- 172.21.15.161:0/1808690829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18480c3770 0x7f18480c1dc0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f18480c3080 tx=0x7f183802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69bdf32dc45a485c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:45.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.916+0000 7f184d1ef640 1 -- 172.21.15.161:0/1808690829 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18380315a0 con 0x7f18480c3770 2024-03-28T23:33:45.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.916+0000 7f184d1ef640 1 -- 172.21.15.161:0/1808690829 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1838037070 con 0x7f18480c3770 2024-03-28T23:33:45.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.916+0000 7f184d1ef640 1 -- 172.21.15.161:0/1808690829 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1838031960 con 0x7f18480c3770 2024-03-28T23:33:45.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.917+0000 7f1850455640 1 -- 172.21.15.161:0/1808690829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18480c3770 msgr2=0x7f18480c1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:45.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.917+0000 7f1850455640 1 --2- 172.21.15.161:0/1808690829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18480c3770 0x7f18480c1dc0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f18480c3080 tx=0x7f183802b840 comp rx=0 tx=0).stop 2024-03-28T23:33:45.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.917+0000 7f1850455640 1 -- 172.21.15.161:0/1808690829 shutdown_connections 2024-03-28T23:33:45.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.917+0000 7f1850455640 1 --2- 172.21.15.161:0/1808690829 >> v2:172.21.15.161:3300/0 conn(0x7f18480c2300 0x7f184815fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.917+0000 7f1850455640 1 --2- 172.21.15.161:0/1808690829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18480c3770 0x7f18480c1dc0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.917+0000 7f1850455640 1 -- 172.21.15.161:0/1808690829 >> 172.21.15.161:0/1808690829 conn(0x7f18480aa740 msgr2=0x7f18480aab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:45.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.917+0000 7f1850455640 1 -- 172.21.15.161:0/1808690829 shutdown_connections 2024-03-28T23:33:45.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.917+0000 7f1850455640 1 -- 172.21.15.161:0/1808690829 wait complete. 2024-03-28T23:33:45.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.918+0000 7f1850455640 1 Processor -- start 2024-03-28T23:33:45.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.918+0000 7f1850455640 1 -- start start 2024-03-28T23:33:45.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.919+0000 7f1850455640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18480c2300 0x7f18480ae390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:45.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.919+0000 7f1850455640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18480c3390 0x7f18481ee1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:45.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.919+0000 7f1850455640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f18480c3770 0x7f18480ac5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:45.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.919+0000 7f1850455640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1848155b10 con 0x7f18480c2300 2024-03-28T23:33:45.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.919+0000 7f1850455640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1848155830 con 0x7f18480c3390 2024-03-28T23:33:45.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.919+0000 7f1850455640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1848152a80 con 0x7f18480c3770 2024-03-28T23:33:45.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.919+0000 7f184e1f1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18480c2300 0x7f18480ae390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:45.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.919+0000 7f184e1f1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18480c2300 0x7f18480ae390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57244/0 (socket says 172.21.15.161:57244) 2024-03-28T23:33:45.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.919+0000 7f184e1f1640 1 -- 172.21.15.161:0/2761853401 learned_addr learned my addr 172.21.15.161:0/2761853401 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:45.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.919+0000 7f184d9f0640 1 --2- 172.21.15.161:0/2761853401 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18480c3390 0x7f18481ee1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:45.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.919+0000 7f184e9f2640 1 --2- 172.21.15.161:0/2761853401 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f18480c3770 0x7f18480ac5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:45.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.919+0000 7f184e1f1640 1 -- 172.21.15.161:0/2761853401 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f18480c3770 msgr2=0x7f18480ac5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:45.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.919+0000 7f184e1f1640 1 --2- 172.21.15.161:0/2761853401 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f18480c3770 0x7f18480ac5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.919+0000 7f184e1f1640 1 -- 172.21.15.161:0/2761853401 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18480c3390 msgr2=0x7f18481ee1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:45.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.919+0000 7f184e1f1640 1 --2- 172.21.15.161:0/2761853401 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18480c3390 0x7f18481ee1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:45.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.919+0000 7f184e1f1640 1 -- 172.21.15.161:0/2761853401 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18481571a0 con 0x7f18480c2300 2024-03-28T23:33:45.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.919+0000 7f184d9f0640 1 --2- 172.21.15.161:0/2761853401 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18480c3390 0x7f18481ee1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:45.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.919+0000 7f184e9f2640 1 --2- 172.21.15.161:0/2761853401 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f18480c3770 0x7f18480ac5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:45.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.919+0000 7f184e1f1640 1 --2- 172.21.15.161:0/2761853401 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18480c2300 0x7f18480ae390 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f184400c430 tx=0x7f184400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:45.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.920+0000 7f183effd640 1 -- 172.21.15.161:0/2761853401 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1844017070 con 0x7f18480c2300 2024-03-28T23:33:45.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.920+0000 7f183effd640 1 -- 172.21.15.161:0/2761853401 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1844005ce0 con 0x7f18480c2300 2024-03-28T23:33:45.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.920+0000 7f1850455640 1 -- 172.21.15.161:0/2761853401 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18481eee00 con 0x7f18480c2300 2024-03-28T23:33:45.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.920+0000 7f183effd640 1 -- 172.21.15.161:0/2761853401 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18440072f0 con 0x7f18480c2300 2024-03-28T23:33:45.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.920+0000 7f1850455640 1 -- 172.21.15.161:0/2761853401 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1848156500 con 0x7f18480c2300 2024-03-28T23:33:45.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.921+0000 7f1850455640 1 -- 172.21.15.161:0/2761853401 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f184804f4b0 con 0x7f18480c2300 2024-03-28T23:33:45.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.921+0000 7f183effd640 1 -- 172.21.15.161:0/2761853401 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f184401a070 con 0x7f18480c2300 2024-03-28T23:33:45.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.922+0000 7f183effd640 1 --2- 172.21.15.161:0/2761853401 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f182c06b4d0 0x7f182c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:45.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.922+0000 7f183effd640 1 -- 172.21.15.161:0/2761853401 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5421+0+0 (secure 0 0 0) 0x7f18440895b0 con 0x7f18480c2300 2024-03-28T23:33:45.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.922+0000 7f184d9f0640 1 --2- 172.21.15.161:0/2761853401 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f182c06b4d0 0x7f182c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:45.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.922+0000 7f184d9f0640 1 --2- 172.21.15.161:0/2761853401 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f182c06b4d0 0x7f182c06d990 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f1838036000 tx=0x7f183803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:45.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:45.926+0000 7f183effd640 1 -- 172.21.15.161:0/2761853401 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1844058560 con 0x7f18480c2300 2024-03-28T23:33:46.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:46.114+0000 7f1850455640 1 -- 172.21.15.161:0/2761853401 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f1848156030 con 0x7f18480c2300 2024-03-28T23:33:46.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:46.964+0000 7f183effd640 1 -- 172.21.15.161:0/2761853401 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v105) ==== 97+0+0 (secure 0 0 0) 0x7f1844057e60 con 0x7f18480c2300 2024-03-28T23:33:46.964 INFO:teuthology.orchestra.run.smithi161.stderr: removed all blocklist entries 2024-03-28T23:33:46.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:46.967+0000 7f1850455640 1 -- 172.21.15.161:0/2761853401 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f182c06b4d0 msgr2=0x7f182c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:46.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:46.967+0000 7f1850455640 1 --2- 172.21.15.161:0/2761853401 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f182c06b4d0 0x7f182c06d990 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f1838036000 tx=0x7f183803d040 comp rx=0 tx=0).stop 2024-03-28T23:33:46.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:46.967+0000 7f1850455640 1 -- 172.21.15.161:0/2761853401 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18480c2300 msgr2=0x7f18480ae390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:46.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:46.967+0000 7f1850455640 1 --2- 172.21.15.161:0/2761853401 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18480c2300 0x7f18480ae390 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f184400c430 tx=0x7f184400c8f0 comp rx=0 tx=0).stop 2024-03-28T23:33:46.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:46.967+0000 7f1850455640 1 -- 172.21.15.161:0/2761853401 shutdown_connections 2024-03-28T23:33:46.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:46.967+0000 7f1850455640 1 --2- 172.21.15.161:0/2761853401 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f182c06b4d0 0x7f182c06d990 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:46.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:46.967+0000 7f1850455640 1 --2- 172.21.15.161:0/2761853401 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f18480c3770 0x7f18480ac5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:46.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:46.967+0000 7f1850455640 1 --2- 172.21.15.161:0/2761853401 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18480c3390 0x7f18481ee1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:46.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:46.967+0000 7f1850455640 1 --2- 172.21.15.161:0/2761853401 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18480c2300 0x7f18480ae390 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:46.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:46.967+0000 7f1850455640 1 -- 172.21.15.161:0/2761853401 >> 172.21.15.161:0/2761853401 conn(0x7f18480aa740 msgr2=0x7f184814f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:46.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:46.967+0000 7f1850455640 1 -- 172.21.15.161:0/2761853401 shutdown_connections 2024-03-28T23:33:46.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:46.967+0000 7f1850455640 1 -- 172.21.15.161:0/2761853401 wait complete. 2024-03-28T23:33:46.981 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-28T23:33:47.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.087+0000 7fcf3226f640 1 Processor -- start 2024-03-28T23:33:47.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.087+0000 7fcf3226f640 1 -- start start 2024-03-28T23:33:47.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.088+0000 7fcf3226f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fcf2c14be70 0x7fcf2c0ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:47.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.088+0000 7fcf3226f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf2c14c250 con 0x7fcf2c0ab6e0 2024-03-28T23:33:47.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.088+0000 7fcf3226f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf2c1529c0 con 0x7fcf2c0aacd0 2024-03-28T23:33:47.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.088+0000 7fcf3226f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fcf2c0a9dc0 con 0x7fcf2c14be70 2024-03-28T23:33:47.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.088+0000 7fcf2b7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fcf2c0ab6e0 0x7fcf2c0aa5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:55990/0 (socket says 172.21.15.161:55990) 2024-03-28T23:33:47.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.088+0000 7fcf2b7fe640 1 -- 172.21.15.161:0/4091909888 learned_addr learned my addr 172.21.15.161:0/4091909888 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:47.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.088+0000 7fcf2bfff640 1 --2- 172.21.15.161:0/4091909888 >> v2:172.21.15.170:3300/0 conn(0x7fcf2c14be70 0x7fcf2c0ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:47.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.088+0000 7fcf2affd640 1 -- 172.21.15.161:0/4091909888 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3430119978 0 0) 0x7fcf2c14c250 con 0x7fcf2c0ab6e0 2024-03-28T23:33:47.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.089+0000 7fcf2affd640 1 -- 172.21.15.161:0/4091909888 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf14003520 con 0x7fcf2c0ab6e0 2024-03-28T23:33:47.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.089+0000 7fcf2affd640 1 -- 172.21.15.161:0/4091909888 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4080890525 0 0) 0x7fcf14003520 con 0x7fcf2c0ab6e0 2024-03-28T23:33:47.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.089+0000 7fcf2bfff640 1 -- 172.21.15.161:0/4091909888 >> v1:172.21.15.170:6789/0 conn(0x7fcf2c0aacd0 legacy=0x7fcf2c15c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:47.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.089+0000 7fcf2bfff640 1 -- 172.21.15.161:0/4091909888 >> v1:172.21.15.161:6789/0 conn(0x7fcf2c0ab6e0 legacy=0x7fcf2c0aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:47.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.089+0000 7fcf2bfff640 1 -- 172.21.15.161:0/4091909888 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf2c0aa0a0 con 0x7fcf2c14be70 2024-03-28T23:33:47.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.090+0000 7fcf2bfff640 1 --2- 172.21.15.161:0/4091909888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf2c14be70 0x7fcf2c0ab1a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0097a0 tx=0x7fcf1c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c66161782d09f0aa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:47.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.090+0000 7fcf2affd640 1 -- 172.21.15.161:0/4091909888 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c037070 con 0x7fcf2c14be70 2024-03-28T23:33:47.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.090+0000 7fcf2affd640 1 -- 172.21.15.161:0/4091909888 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf1c030510 con 0x7fcf2c14be70 2024-03-28T23:33:47.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.090+0000 7fcf2affd640 1 -- 172.21.15.161:0/4091909888 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf20001670 con 0x7fcf2c14be70 2024-03-28T23:33:47.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.090+0000 7fcf3226f640 1 -- 172.21.15.161:0/4091909888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf2c14be70 msgr2=0x7fcf2c0ab1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:47.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.090+0000 7fcf3226f640 1 --2- 172.21.15.161:0/4091909888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf2c14be70 0x7fcf2c0ab1a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0097a0 tx=0x7fcf1c02b790 comp rx=0 tx=0).stop 2024-03-28T23:33:47.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.090+0000 7fcf3226f640 1 -- 172.21.15.161:0/4091909888 shutdown_connections 2024-03-28T23:33:47.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.090+0000 7fcf3226f640 1 --2- 172.21.15.161:0/4091909888 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf2c14be70 0x7fcf2c0ab1a0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.090+0000 7fcf3226f640 1 -- 172.21.15.161:0/4091909888 >> 172.21.15.161:0/4091909888 conn(0x7fcf2c0c6670 msgr2=0x7fcf2c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:47.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.090+0000 7fcf3226f640 1 -- 172.21.15.161:0/4091909888 shutdown_connections 2024-03-28T23:33:47.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.090+0000 7fcf3226f640 1 -- 172.21.15.161:0/4091909888 wait complete. 2024-03-28T23:33:47.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.091+0000 7fcf3226f640 1 Processor -- start 2024-03-28T23:33:47.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.092+0000 7fcf3226f640 1 -- start start 2024-03-28T23:33:47.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.092+0000 7fcf3226f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf2c0aacd0 0x7fcf2c1dbe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:47.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.092+0000 7fcf3226f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf2c0ab6e0 0x7fcf2c1553b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:47.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.092+0000 7fcf3226f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf2c14be70 0x7fcf2c1488a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:47.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.092+0000 7fcf3226f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2c06b2d0 con 0x7fcf2c0aacd0 2024-03-28T23:33:47.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.092+0000 7fcf3226f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2c06b150 con 0x7fcf2c0ab6e0 2024-03-28T23:33:47.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.092+0000 7fcf3226f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fcf2c0c1520 con 0x7fcf2c14be70 2024-03-28T23:33:47.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.092+0000 7fcf2bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf2c0aacd0 0x7fcf2c1dbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:47.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.092+0000 7fcf2bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf2c0aacd0 0x7fcf2c1dbe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57254/0 (socket says 172.21.15.161:57254) 2024-03-28T23:33:47.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.092+0000 7fcf2bfff640 1 -- 172.21.15.161:0/2155265664 learned_addr learned my addr 172.21.15.161:0/2155265664 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:47.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.093+0000 7fcf2b7fe640 1 --2- 172.21.15.161:0/2155265664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf2c0ab6e0 0x7fcf2c1553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:47.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.093+0000 7fcf3080c640 1 --2- 172.21.15.161:0/2155265664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf2c14be70 0x7fcf2c1488a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:47.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.093+0000 7fcf2bfff640 1 -- 172.21.15.161:0/2155265664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf2c14be70 msgr2=0x7fcf2c1488a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:47.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.093+0000 7fcf2bfff640 1 --2- 172.21.15.161:0/2155265664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf2c14be70 0x7fcf2c1488a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.093+0000 7fcf2bfff640 1 -- 172.21.15.161:0/2155265664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf2c0ab6e0 msgr2=0x7fcf2c1553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:47.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.093+0000 7fcf2bfff640 1 --2- 172.21.15.161:0/2155265664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf2c0ab6e0 0x7fcf2c1553b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.093+0000 7fcf2bfff640 1 -- 172.21.15.161:0/2155265664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf2c1e8f20 con 0x7fcf2c0aacd0 2024-03-28T23:33:47.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.093+0000 7fcf3080c640 1 --2- 172.21.15.161:0/2155265664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf2c14be70 0x7fcf2c1488a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:47.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.093+0000 7fcf2bfff640 1 --2- 172.21.15.161:0/2155265664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf2c0aacd0 0x7fcf2c1dbe10 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0098c0 tx=0x7fcf1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:47.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.093+0000 7fcf28ff9640 1 -- 172.21.15.161:0/2155265664 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c042070 con 0x7fcf2c0aacd0 2024-03-28T23:33:47.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.093+0000 7fcf28ff9640 1 -- 172.21.15.161:0/2155265664 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf1c03e870 con 0x7fcf2c0aacd0 2024-03-28T23:33:47.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.093+0000 7fcf3226f640 1 -- 172.21.15.161:0/2155265664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf2c1e9200 con 0x7fcf2c0aacd0 2024-03-28T23:33:47.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.093+0000 7fcf28ff9640 1 -- 172.21.15.161:0/2155265664 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c037040 con 0x7fcf2c0aacd0 2024-03-28T23:33:47.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.093+0000 7fcf3226f640 1 -- 172.21.15.161:0/2155265664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf2c1dc350 con 0x7fcf2c0aacd0 2024-03-28T23:33:47.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.094+0000 7fcf3226f640 1 -- 172.21.15.161:0/2155265664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf2c04f4b0 con 0x7fcf2c0aacd0 2024-03-28T23:33:47.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.095+0000 7fcf28ff9640 1 -- 172.21.15.161:0/2155265664 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcf1c045070 con 0x7fcf2c0aacd0 2024-03-28T23:33:47.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.095+0000 7fcf28ff9640 1 --2- 172.21.15.161:0/2155265664 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcf1806b4d0 0x7fcf1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:47.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.095+0000 7fcf2b7fe640 1 --2- 172.21.15.161:0/2155265664 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcf1806b4d0 0x7fcf1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:47.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.095+0000 7fcf28ff9640 1 -- 172.21.15.161:0/2155265664 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5249+0+0 (secure 0 0 0) 0x7fcf1c0abc50 con 0x7fcf2c0aacd0 2024-03-28T23:33:47.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.096+0000 7fcf2b7fe640 1 --2- 172.21.15.161:0/2155265664 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcf1806b4d0 0x7fcf1806d990 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c155110 tx=0x7fcf20009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:47.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.100+0000 7fcf28ff9640 1 -- 172.21.15.161:0/2155265664 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcf1c07aca0 con 0x7fcf2c0aacd0 2024-03-28T23:33:47.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.336+0000 7fcf3226f640 1 -- 172.21.15.161:0/2155265664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fcf2c0c0ff0 con 0x7fcf2c0aacd0 2024-03-28T23:33:47.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.337+0000 7fcf28ff9640 1 -- 172.21.15.161:0/2155265664 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) ==== 80+0+4724 (secure 0 0 0) 0x7fcf1c07a5a0 con 0x7fcf2c0aacd0 2024-03-28T23:33:47.338 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:47.338 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:33:47.338 INFO:teuthology.orchestra.run.smithi161.stdout: "auth_dump": [ 2024-03-28T23:33:47.338 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:47.338 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.a", 2024-03-28T23:33:47.338 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmggtoJhAASLXbUICLBhjRr19aUAuZHw==", 2024-03-28T23:33:47.338 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:47.338 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:33:47.338 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:33:47.338 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:33:47.338 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:47.339 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:47.339 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:47.339 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:47.339 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.b", 2024-03-28T23:33:47.339 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmQbEOMxAA74al6+dMzcN/ngwNZK+aHA==", 2024-03-28T23:33:47.339 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:47.339 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:33:47.339 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:33:47.339 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:33:47.339 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:47.339 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:47.339 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:47.339 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:47.339 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.c", 2024-03-28T23:33:47.340 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmjf/qLRAAosLbpT8aCWvrKAQnj8Uh3g==", 2024-03-28T23:33:47.340 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:47.340 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:33:47.340 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:33:47.340 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:33:47.340 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:47.340 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:47.340 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:47.340 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:47.340 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.d", 2024-03-28T23:33:47.340 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmpiZ1OhAAjYrFAaDosDS2Y+BvU4c6JQ==", 2024-03-28T23:33:47.340 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:47.340 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:33:47.340 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:33:47.340 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:33:47.341 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:47.341 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:47.341 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:47.341 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:47.341 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.0", 2024-03-28T23:33:47.341 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA9/QVmAFPmGBAAS0SkRRwsupm6jm1sqFXJ6g==", 2024-03-28T23:33:47.341 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:47.341 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:33:47.341 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:33:47.341 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:47.341 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:47.341 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:47.341 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:47.341 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.1", 2024-03-28T23:33:47.342 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBA/QVmNBTVFxAA1htU9ilznfLUDqKmJaQwzg==", 2024-03-28T23:33:47.342 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:47.342 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:33:47.342 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:33:47.342 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:47.342 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:47.342 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:47.342 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:47.342 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.2", 2024-03-28T23:33:47.342 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBD/QVmg+4YFhAA9TvfFVKT3nQT7/QUOf3Spw==", 2024-03-28T23:33:47.342 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:47.342 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:33:47.342 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:33:47.343 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:47.343 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:47.343 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:47.343 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:47.343 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.3", 2024-03-28T23:33:47.343 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBG/QVmgiHDFBAAQPtbZT6a1ZGmkisBX0NMgA==", 2024-03-28T23:33:47.343 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:47.343 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:33:47.343 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:33:47.343 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:47.343 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:47.343 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:47.343 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:47.343 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.4", 2024-03-28T23:33:47.343 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBK/QVmahhZIBAAJfotg/dusbB3usreZlRuPA==", 2024-03-28T23:33:47.344 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:47.344 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:33:47.344 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:33:47.344 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:47.344 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:47.344 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:47.344 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:47.344 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.5", 2024-03-28T23:33:47.344 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBN/QVmzjlDHhAAli7pLyIo+MVdwSBNJYVg5g==", 2024-03-28T23:33:47.344 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:47.344 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:33:47.344 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:33:47.344 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:47.344 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:47.345 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:47.345 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:47.345 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.6", 2024-03-28T23:33:47.345 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBQ/QVmtCFSHBAASVfV/5mY/3JNiI0YMT7Vww==", 2024-03-28T23:33:47.345 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:47.345 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:33:47.345 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:33:47.345 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:47.345 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:47.345 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:47.345 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:47.345 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.7", 2024-03-28T23:33:47.345 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBT/QVm2J7eGhAAaIeZxmLwQBBE1glCxjKbhA==", 2024-03-28T23:33:47.345 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:47.346 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:33:47.346 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:33:47.346 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:47.346 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:47.346 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:47.346 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:47.346 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.0", 2024-03-28T23:33:47.346 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA6/QVmc1krAxAAEKsn1Tuuv9OlFZpB4PFhDg==", 2024-03-28T23:33:47.346 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:47.346 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:33:47.346 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow r", 2024-03-28T23:33:47.346 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-28T23:33:47.346 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:47.347 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:47.347 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:47.347 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.1", 2024-03-28T23:33:47.347 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA6/QVm1B2cBhAAmHDeeFwpFlV/y3NIGWUXHQ==", 2024-03-28T23:33:47.347 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:47.347 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:33:47.347 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow r", 2024-03-28T23:33:47.347 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow rw", 2024-03-28T23:33:47.347 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow rwx" 2024-03-28T23:33:47.347 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:47.347 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:47.347 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:47.347 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.admin", 2024-03-28T23:33:47.347 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmzzJ7CRAAYqfhnIkfNJJ3O+VWC2LP+Q==", 2024-03-28T23:33:47.348 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:47.348 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-28T23:33:47.348 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:33:47.348 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:33:47.348 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:47.348 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:47.348 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:47.348 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:47.348 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mgr.x", 2024-03-28T23:33:47.348 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVm7TD3IhAACByGRGpvJWDpF22NuovHWw==", 2024-03-28T23:33:47.348 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:47.348 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-28T23:33:47.348 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile mgr", 2024-03-28T23:33:47.348 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:47.349 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:47.349 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:33:47.349 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:47.349 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mgr.y", 2024-03-28T23:33:47.349 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVm0hLlHhAAh8cewPtZ/TbKrU488kxrBg==", 2024-03-28T23:33:47.349 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:33:47.349 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-28T23:33:47.349 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile mgr", 2024-03-28T23:33:47.349 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:33:47.349 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:47.349 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:47.349 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-03-28T23:33:47.349 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:33:47.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.339+0000 7fcf3226f640 1 -- 172.21.15.161:0/2155265664 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcf1806b4d0 msgr2=0x7fcf1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:47.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.339+0000 7fcf3226f640 1 --2- 172.21.15.161:0/2155265664 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcf1806b4d0 0x7fcf1806d990 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c155110 tx=0x7fcf20009040 comp rx=0 tx=0).stop 2024-03-28T23:33:47.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.339+0000 7fcf3226f640 1 -- 172.21.15.161:0/2155265664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf2c0aacd0 msgr2=0x7fcf2c1dbe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:47.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.339+0000 7fcf3226f640 1 --2- 172.21.15.161:0/2155265664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf2c0aacd0 0x7fcf2c1dbe10 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0098c0 tx=0x7fcf1c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.339+0000 7fcf3226f640 1 -- 172.21.15.161:0/2155265664 shutdown_connections 2024-03-28T23:33:47.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.339+0000 7fcf3226f640 1 --2- 172.21.15.161:0/2155265664 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcf1806b4d0 0x7fcf1806d990 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.339+0000 7fcf3226f640 1 --2- 172.21.15.161:0/2155265664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf2c14be70 0x7fcf2c1488a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.339+0000 7fcf3226f640 1 --2- 172.21.15.161:0/2155265664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf2c0ab6e0 0x7fcf2c1553b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.339+0000 7fcf3226f640 1 --2- 172.21.15.161:0/2155265664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf2c0aacd0 0x7fcf2c1dbe10 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.339+0000 7fcf3226f640 1 -- 172.21.15.161:0/2155265664 >> 172.21.15.161:0/2155265664 conn(0x7fcf2c0c6670 msgr2=0x7fcf2c0c2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:47.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.339+0000 7fcf3226f640 1 -- 172.21.15.161:0/2155265664 shutdown_connections 2024-03-28T23:33:47.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.339+0000 7fcf3226f640 1 -- 172.21.15.161:0/2155265664 wait complete. 2024-03-28T23:33:47.352 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:33:47.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.453+0000 7fcdbbc14640 1 Processor -- start 2024-03-28T23:33:47.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.453+0000 7fcdbbc14640 1 -- start start 2024-03-28T23:33:47.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.454+0000 7fcdbbc14640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdb414be70 0x7fcdb40ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:47.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.454+0000 7fcdbbc14640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcdb40ab6e0 0x7fcdb40aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:47.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.454+0000 7fcdbbc14640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdb414c250 con 0x7fcdb406ce80 2024-03-28T23:33:47.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.454+0000 7fcdbbc14640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fcdb41529c0 con 0x7fcdb40ab6e0 2024-03-28T23:33:47.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.454+0000 7fcdbbc14640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fcdb40a9dc0 con 0x7fcdb414be70 2024-03-28T23:33:47.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.454+0000 7fcdb91af640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcdb40ab6e0 0x7fcdb40aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:47.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.454+0000 7fcdb91af640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcdb40ab6e0 0x7fcdb40aa5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57260/0 (socket says 172.21.15.161:57260) 2024-03-28T23:33:47.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.454+0000 7fcdb91af640 1 -- 172.21.15.161:0/2407212742 learned_addr learned my addr 172.21.15.161:0/2407212742 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:47.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.454+0000 7fcdb99b0640 1 --2- 172.21.15.161:0/2407212742 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdb414be70 0x7fcdb40ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:47.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.454+0000 7fcdb91af640 1 -- 172.21.15.161:0/2407212742 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdb414be70 msgr2=0x7fcdb40ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:47.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.454+0000 7fcdb91af640 1 --2- 172.21.15.161:0/2407212742 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdb414be70 0x7fcdb40ab1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.454+0000 7fcdb91af640 1 -- 172.21.15.161:0/2407212742 >> v1:172.21.15.170:6789/0 conn(0x7fcdb406ce80 legacy=0x7fcdb41553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:47.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.454+0000 7fcdb91af640 1 -- 172.21.15.161:0/2407212742 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdb40aa0a0 con 0x7fcdb40ab6e0 2024-03-28T23:33:47.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.455+0000 7fcdb91af640 1 --2- 172.21.15.161:0/2407212742 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdb40ab6e0 0x7fcdb40aa5c0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fcda4009870 tx=0x7fcda402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c878f1e5d6dc24c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:47.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.455+0000 7fcdb89ae640 1 -- 172.21.15.161:0/2407212742 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda40317c0 con 0x7fcdb40ab6e0 2024-03-28T23:33:47.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.455+0000 7fcdb89ae640 1 -- 172.21.15.161:0/2407212742 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcda4037070 con 0x7fcdb40ab6e0 2024-03-28T23:33:47.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.455+0000 7fcdb89ae640 1 -- 172.21.15.161:0/2407212742 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda4031b80 con 0x7fcdb40ab6e0 2024-03-28T23:33:47.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.456+0000 7fcdbbc14640 1 -- 172.21.15.161:0/2407212742 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdb40ab6e0 msgr2=0x7fcdb40aa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:47.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.456+0000 7fcdbbc14640 1 --2- 172.21.15.161:0/2407212742 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdb40ab6e0 0x7fcdb40aa5c0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fcda4009870 tx=0x7fcda402b860 comp rx=0 tx=0).stop 2024-03-28T23:33:47.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.456+0000 7fcdbbc14640 1 -- 172.21.15.161:0/2407212742 shutdown_connections 2024-03-28T23:33:47.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.456+0000 7fcdbbc14640 1 --2- 172.21.15.161:0/2407212742 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdb40ab6e0 0x7fcdb40aa5c0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.456+0000 7fcdbbc14640 1 --2- 172.21.15.161:0/2407212742 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdb414be70 0x7fcdb40ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.456+0000 7fcdbbc14640 1 -- 172.21.15.161:0/2407212742 >> 172.21.15.161:0/2407212742 conn(0x7fcdb40c6670 msgr2=0x7fcdb40c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:47.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.456+0000 7fcdbbc14640 1 -- 172.21.15.161:0/2407212742 shutdown_connections 2024-03-28T23:33:47.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.456+0000 7fcdbbc14640 1 -- 172.21.15.161:0/2407212742 wait complete. 2024-03-28T23:33:47.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.457+0000 7fcdbbc14640 1 Processor -- start 2024-03-28T23:33:47.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.457+0000 7fcdbbc14640 1 -- start start 2024-03-28T23:33:47.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.458+0000 7fcdbbc14640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdb406ce80 0x7fcdb414cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:47.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.458+0000 7fcdbbc14640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdb40ab6e0 0x7fcdb40bfed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:47.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.458+0000 7fcdbbc14640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdb414be70 0x7fcdb41541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:47.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.458+0000 7fcdbbc14640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb406b2d0 con 0x7fcdb406ce80 2024-03-28T23:33:47.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.458+0000 7fcdbbc14640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb406b150 con 0x7fcdb40ab6e0 2024-03-28T23:33:47.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.458+0000 7fcdbbc14640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fcdb40c1520 con 0x7fcdb414be70 2024-03-28T23:33:47.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.458+0000 7fcdb99b0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdb406ce80 0x7fcdb414cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:47.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.458+0000 7fcdb99b0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdb406ce80 0x7fcdb414cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57272/0 (socket says 172.21.15.161:57272) 2024-03-28T23:33:47.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.458+0000 7fcdb99b0640 1 -- 172.21.15.161:0/4198000197 learned_addr learned my addr 172.21.15.161:0/4198000197 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:47.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.458+0000 7fcdb99b0640 1 -- 172.21.15.161:0/4198000197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdb414be70 msgr2=0x7fcdb41541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:47.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.458+0000 7fcdb99b0640 1 --2- 172.21.15.161:0/4198000197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdb414be70 0x7fcdb41541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.458+0000 7fcdb99b0640 1 -- 172.21.15.161:0/4198000197 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdb40ab6e0 msgr2=0x7fcdb40bfed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:47.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.458+0000 7fcdb99b0640 1 --2- 172.21.15.161:0/4198000197 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdb40ab6e0 0x7fcdb40bfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.458+0000 7fcdb99b0640 1 -- 172.21.15.161:0/4198000197 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdb4153100 con 0x7fcdb406ce80 2024-03-28T23:33:47.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.458+0000 7fcdb99b0640 1 --2- 172.21.15.161:0/4198000197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdb406ce80 0x7fcdb414cc00 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000c820 tx=0x7fcdb000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:47.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.458+0000 7fcdaa7fc640 1 -- 172.21.15.161:0/4198000197 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0015070 con 0x7fcdb406ce80 2024-03-28T23:33:47.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.458+0000 7fcdaa7fc640 1 -- 172.21.15.161:0/4198000197 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdb0004220 con 0x7fcdb406ce80 2024-03-28T23:33:47.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.458+0000 7fcdbbc14640 1 -- 172.21.15.161:0/4198000197 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdb41533e0 con 0x7fcdb406ce80 2024-03-28T23:33:47.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.458+0000 7fcdaa7fc640 1 -- 172.21.15.161:0/4198000197 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb000ce10 con 0x7fcdb406ce80 2024-03-28T23:33:47.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.458+0000 7fcdbbc14640 1 -- 172.21.15.161:0/4198000197 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdb41539a0 con 0x7fcdb406ce80 2024-03-28T23:33:47.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.459+0000 7fcdbbc14640 1 -- 172.21.15.161:0/4198000197 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdb404f4b0 con 0x7fcdb406ce80 2024-03-28T23:33:47.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.460+0000 7fcdaa7fc640 1 -- 172.21.15.161:0/4198000197 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcdb0007830 con 0x7fcdb406ce80 2024-03-28T23:33:47.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.460+0000 7fcdaa7fc640 1 --2- 172.21.15.161:0/4198000197 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcd9806b4d0 0x7fcd9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:47.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.460+0000 7fcdaa7fc640 1 -- 172.21.15.161:0/4198000197 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5249+0+0 (secure 0 0 0) 0x7fcdb008a270 con 0x7fcdb406ce80 2024-03-28T23:33:47.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.460+0000 7fcdb91af640 1 --2- 172.21.15.161:0/4198000197 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcd9806b4d0 0x7fcd9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:47.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.461+0000 7fcdb91af640 1 --2- 172.21.15.161:0/4198000197 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcd9806b4d0 0x7fcd9806d990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fcda4036000 tx=0x7fcda4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:47.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.464+0000 7fcdaa7fc640 1 -- 172.21.15.161:0/4198000197 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcdb0059340 con 0x7fcdb406ce80 2024-03-28T23:33:47.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.636+0000 7fcdbbc14640 1 -- 172.21.15.161:0/4198000197 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcdb40c0ff0 con 0x7fcdb406ce80 2024-03-28T23:33:47.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.637+0000 7fcdaa7fc640 1 -- 172.21.15.161:0/4198000197 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) ==== 74+0+13202 (secure 0 0 0) 0x7fcdb0058c40 con 0x7fcdb406ce80 2024-03-28T23:33:47.638 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:47.638 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":105,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:33:46.963447+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:33:47.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.639+0000 7fcdbbc14640 1 -- 172.21.15.161:0/4198000197 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcd9806b4d0 msgr2=0x7fcd9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:47.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.639+0000 7fcdbbc14640 1 --2- 172.21.15.161:0/4198000197 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcd9806b4d0 0x7fcd9806d990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fcda4036000 tx=0x7fcda4030000 comp rx=0 tx=0).stop 2024-03-28T23:33:47.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.639+0000 7fcdbbc14640 1 -- 172.21.15.161:0/4198000197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdb406ce80 msgr2=0x7fcdb414cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:47.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.639+0000 7fcdbbc14640 1 --2- 172.21.15.161:0/4198000197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdb406ce80 0x7fcdb414cc00 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000c820 tx=0x7fcdb000cce0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.639+0000 7fcdbbc14640 1 -- 172.21.15.161:0/4198000197 shutdown_connections 2024-03-28T23:33:47.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.639+0000 7fcdbbc14640 1 --2- 172.21.15.161:0/4198000197 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcd9806b4d0 0x7fcd9806d990 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.639+0000 7fcdbbc14640 1 --2- 172.21.15.161:0/4198000197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdb414be70 0x7fcdb41541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.639+0000 7fcdbbc14640 1 --2- 172.21.15.161:0/4198000197 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdb40ab6e0 0x7fcdb40bfed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.639+0000 7fcdbbc14640 1 --2- 172.21.15.161:0/4198000197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdb406ce80 0x7fcdb414cc00 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.639+0000 7fcdbbc14640 1 -- 172.21.15.161:0/4198000197 >> 172.21.15.161:0/4198000197 conn(0x7fcdb40c6670 msgr2=0x7fcdb41512c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:47.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.639+0000 7fcdbbc14640 1 -- 172.21.15.161:0/4198000197 shutdown_connections 2024-03-28T23:33:47.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.639+0000 7fcdbbc14640 1 -- 172.21.15.161:0/4198000197 wait complete. 2024-03-28T23:33:47.650 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:33:47.651 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:33:47.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.754+0000 7f0195cb5640 1 Processor -- start 2024-03-28T23:33:47.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.754+0000 7f0195cb5640 1 -- start start 2024-03-28T23:33:47.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.755+0000 7f0195cb5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f01901511e0 0x7f019006d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:47.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.755+0000 7f0195cb5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f01900c6470 0x7f0190151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:47.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.755+0000 7f0195cb5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01900c2fc0 con 0x7f019006d610 2024-03-28T23:33:47.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.755+0000 7f0195cb5640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f01900c1290 con 0x7f01900c6470 2024-03-28T23:33:47.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.755+0000 7f0195cb5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f019006c8a0 con 0x7f01901511e0 2024-03-28T23:33:47.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.755+0000 7f018effd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f01900c6470 0x7f0190151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:47.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.755+0000 7f018ffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f019006d610 0x7f01901520e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:56002/0 (socket says 172.21.15.161:56002) 2024-03-28T23:33:47.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.755+0000 7f018ffff640 1 -- 172.21.15.161:0/1393506862 learned_addr learned my addr 172.21.15.161:0/1393506862 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:47.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.755+0000 7f018f7fe640 1 --2- 172.21.15.161:0/1393506862 >> v2:172.21.15.170:3300/0 conn(0x7f01901511e0 0x7f019006d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:47.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.756+0000 7f018effd640 1 -- 172.21.15.161:0/1393506862 >> v2:172.21.15.170:3300/0 conn(0x7f01901511e0 msgr2=0x7f019006d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:47.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.756+0000 7f018effd640 1 --2- 172.21.15.161:0/1393506862 >> v2:172.21.15.170:3300/0 conn(0x7f01901511e0 0x7f019006d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.756+0000 7f018effd640 1 -- 172.21.15.161:0/1393506862 >> v1:172.21.15.161:6789/0 conn(0x7f019006d610 legacy=0x7f01901520e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:47.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.756+0000 7f018effd640 1 -- 172.21.15.161:0/1393506862 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f019006cb80 con 0x7f01900c6470 2024-03-28T23:33:47.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.756+0000 7f018e7fc640 1 -- 172.21.15.161:0/1393506862 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2826715019 0 0) 0x7f01900c2fc0 con 0x7f019006d610 2024-03-28T23:33:47.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.756+0000 7f018effd640 1 --2- 172.21.15.161:0/1393506862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01900c6470 0x7f0190151ba0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f019006fde0 tx=0x7f018402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50fdf36c5cfa3c8a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:47.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.756+0000 7f018e7fc640 1 -- 172.21.15.161:0/1393506862 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0184038070 con 0x7f01900c6470 2024-03-28T23:33:47.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.756+0000 7f018e7fc640 1 -- 172.21.15.161:0/1393506862 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0184031710 con 0x7f01900c6470 2024-03-28T23:33:47.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.756+0000 7f018e7fc640 1 -- 172.21.15.161:0/1393506862 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0184031ae0 con 0x7f01900c6470 2024-03-28T23:33:47.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.757+0000 7f0195cb5640 1 -- 172.21.15.161:0/1393506862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01900c6470 msgr2=0x7f0190151ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:47.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.757+0000 7f0195cb5640 1 --2- 172.21.15.161:0/1393506862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01900c6470 0x7f0190151ba0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f019006fde0 tx=0x7f018402b7b0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.757+0000 7f0195cb5640 1 -- 172.21.15.161:0/1393506862 shutdown_connections 2024-03-28T23:33:47.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.757+0000 7f0195cb5640 1 --2- 172.21.15.161:0/1393506862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01900c6470 0x7f0190151ba0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.757+0000 7f0195cb5640 1 --2- 172.21.15.161:0/1393506862 >> v2:172.21.15.170:3300/0 conn(0x7f01901511e0 0x7f019006d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.757+0000 7f0195cb5640 1 -- 172.21.15.161:0/1393506862 >> 172.21.15.161:0/1393506862 conn(0x7f01900c3390 msgr2=0x7f01900c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:47.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.757+0000 7f0195cb5640 1 -- 172.21.15.161:0/1393506862 shutdown_connections 2024-03-28T23:33:47.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.757+0000 7f0195cb5640 1 -- 172.21.15.161:0/1393506862 wait complete. 2024-03-28T23:33:47.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.758+0000 7f0195cb5640 1 Processor -- start 2024-03-28T23:33:47.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.758+0000 7f0195cb5640 1 -- start start 2024-03-28T23:33:47.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.759+0000 7f0195cb5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f019006d610 0x7f01900a76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:47.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.759+0000 7f0195cb5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01901511e0 0x7f0190144ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:47.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.759+0000 7f0195cb5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0190153de0 0x7f0190145010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:47.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.759+0000 7f0195cb5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f019006c2e0 con 0x7f01901511e0 2024-03-28T23:33:47.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.759+0000 7f0195cb5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f019006c160 con 0x7f0190153de0 2024-03-28T23:33:47.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.759+0000 7f0195cb5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f01900c5f20 con 0x7f019006d610 2024-03-28T23:33:47.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.759+0000 7f018effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01901511e0 0x7f0190144ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:47.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.759+0000 7f018effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01901511e0 0x7f0190144ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57294/0 (socket says 172.21.15.161:57294) 2024-03-28T23:33:47.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.759+0000 7f018effd640 1 -- 172.21.15.161:0/3543076793 learned_addr learned my addr 172.21.15.161:0/3543076793 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:47.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.759+0000 7f018ffff640 1 --2- 172.21.15.161:0/3543076793 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0190153de0 0x7f0190145010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:47.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.759+0000 7f018effd640 1 -- 172.21.15.161:0/3543076793 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f019006d610 msgr2=0x7f01900a76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:47.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.759+0000 7f018effd640 1 --2- 172.21.15.161:0/3543076793 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f019006d610 0x7f01900a76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.759+0000 7f018effd640 1 -- 172.21.15.161:0/3543076793 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0190153de0 msgr2=0x7f0190145010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:47.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.759+0000 7f018effd640 1 --2- 172.21.15.161:0/3543076793 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0190153de0 0x7f0190145010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.759+0000 7f018effd640 1 -- 172.21.15.161:0/3543076793 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01901544f0 con 0x7f01901511e0 2024-03-28T23:33:47.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.760+0000 7f018effd640 1 --2- 172.21.15.161:0/3543076793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01901511e0 0x7f0190144ad0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f0184036040 tx=0x7f01840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:47.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.760+0000 7f0177fff640 1 -- 172.21.15.161:0/3543076793 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0184038040 con 0x7f01901511e0 2024-03-28T23:33:47.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.760+0000 7f0177fff640 1 -- 172.21.15.161:0/3543076793 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01840339c0 con 0x7f01901511e0 2024-03-28T23:33:47.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.760+0000 7f0195cb5640 1 -- 172.21.15.161:0/3543076793 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01901547d0 con 0x7f01901511e0 2024-03-28T23:33:47.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.760+0000 7f0177fff640 1 -- 172.21.15.161:0/3543076793 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0184033d90 con 0x7f01901511e0 2024-03-28T23:33:47.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.760+0000 7f0195cb5640 1 -- 172.21.15.161:0/3543076793 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0190145550 con 0x7f01901511e0 2024-03-28T23:33:47.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.761+0000 7f0195cb5640 1 -- 172.21.15.161:0/3543076793 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f019004f4b0 con 0x7f01901511e0 2024-03-28T23:33:47.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.761+0000 7f0177fff640 1 -- 172.21.15.161:0/3543076793 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f018403b070 con 0x7f01901511e0 2024-03-28T23:33:47.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.762+0000 7f0177fff640 1 --2- 172.21.15.161:0/3543076793 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f017c06b500 0x7f017c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:47.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.762+0000 7f0177fff640 1 -- 172.21.15.161:0/3543076793 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5249+0+0 (secure 0 0 0) 0x7f01840abbb0 con 0x7f01901511e0 2024-03-28T23:33:47.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.762+0000 7f018f7fe640 1 --2- 172.21.15.161:0/3543076793 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f017c06b500 0x7f017c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:47.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.762+0000 7f018f7fe640 1 --2- 172.21.15.161:0/3543076793 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f017c06b500 0x7f017c06d9c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f0180006fd0 tx=0x7f0180008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:47.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.766+0000 7f0177fff640 1 -- 172.21.15.161:0/3543076793 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0184042020 con 0x7f01901511e0 2024-03-28T23:33:47.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.939+0000 7f0195cb5640 1 -- 172.21.15.161:0/3543076793 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f01900c5d00 con 0x7f01901511e0 2024-03-28T23:33:47.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.939+0000 7f0177fff640 1 -- 172.21.15.161:0/3543076793 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v105) ==== 93+0+10 (secure 0 0 0) 0x7f018407ada0 con 0x7f01901511e0 2024-03-28T23:33:47.940 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:33:47.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.941+0000 7f0195cb5640 1 -- 172.21.15.161:0/3543076793 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f017c06b500 msgr2=0x7f017c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:47.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.941+0000 7f0195cb5640 1 --2- 172.21.15.161:0/3543076793 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f017c06b500 0x7f017c06d9c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f0180006fd0 tx=0x7f0180008040 comp rx=0 tx=0).stop 2024-03-28T23:33:47.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.941+0000 7f0195cb5640 1 -- 172.21.15.161:0/3543076793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01901511e0 msgr2=0x7f0190144ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:47.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.941+0000 7f0195cb5640 1 --2- 172.21.15.161:0/3543076793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01901511e0 0x7f0190144ad0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f0184036040 tx=0x7f01840046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.941+0000 7f0195cb5640 1 -- 172.21.15.161:0/3543076793 shutdown_connections 2024-03-28T23:33:47.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.941+0000 7f0195cb5640 1 --2- 172.21.15.161:0/3543076793 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f017c06b500 0x7f017c06d9c0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.941+0000 7f0195cb5640 1 --2- 172.21.15.161:0/3543076793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01901511e0 0x7f0190144ad0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.941+0000 7f0195cb5640 1 --2- 172.21.15.161:0/3543076793 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0190153de0 0x7f0190145010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.941+0000 7f0195cb5640 1 --2- 172.21.15.161:0/3543076793 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f019006d610 0x7f01900a76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:47.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.941+0000 7f0195cb5640 1 -- 172.21.15.161:0/3543076793 >> 172.21.15.161:0/3543076793 conn(0x7f01900c3390 msgr2=0x7f0190148180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:47.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.941+0000 7f0195cb5640 1 -- 172.21.15.161:0/3543076793 shutdown_connections 2024-03-28T23:33:47.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:47.941+0000 7f0195cb5640 1 -- 172.21.15.161:0/3543076793 wait complete. 2024-03-28T23:33:47.953 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T23:33:48.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.053+0000 7f958d76c640 1 Processor -- start 2024-03-28T23:33:48.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.053+0000 7f958d76c640 1 -- start start 2024-03-28T23:33:48.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.054+0000 7f958d76c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f95880c7120 0x7f9588155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:48.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.054+0000 7f958d76c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95880c5b70 0x7f95881561f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:48.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.054+0000 7f958d76c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95880bed60 con 0x7f95880befb0 2024-03-28T23:33:48.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.054+0000 7f958d76c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f95880be030 con 0x7f95880c7120 2024-03-28T23:33:48.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.054+0000 7f958d76c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f95880be8d0 con 0x7f95880c5b70 2024-03-28T23:33:48.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.054+0000 7f9586ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f95880c7120 0x7f9588155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:48.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.054+0000 7f9586ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f95880c7120 0x7f9588155cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57300/0 (socket says 172.21.15.161:57300) 2024-03-28T23:33:48.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.054+0000 7f9586ffd640 1 -- 172.21.15.161:0/2497814463 learned_addr learned my addr 172.21.15.161:0/2497814463 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:48.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.054+0000 7f95867fc640 1 --2- 172.21.15.161:0/2497814463 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95880c5b70 0x7f95881561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:48.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.054+0000 7f9586ffd640 1 -- 172.21.15.161:0/2497814463 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95880c5b70 msgr2=0x7f95881561f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:48.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.054+0000 7f9586ffd640 1 --2- 172.21.15.161:0/2497814463 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95880c5b70 0x7f95881561f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:48.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.054+0000 7f9586ffd640 1 -- 172.21.15.161:0/2497814463 >> v1:172.21.15.170:6789/0 conn(0x7f95880befb0 legacy=0x7f9588158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:48.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.054+0000 7f9586ffd640 1 -- 172.21.15.161:0/2497814463 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95880c8540 con 0x7f95880c7120 2024-03-28T23:33:48.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.055+0000 7f9586ffd640 1 --2- 172.21.15.161:0/2497814463 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95880c7120 0x7f9588155cb0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f9578009870 tx=0x7f957802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d149502e033c1e76 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:48.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.055+0000 7f9585ffb640 1 -- 172.21.15.161:0/2497814463 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9578035020 con 0x7f95880c7120 2024-03-28T23:33:48.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.055+0000 7f9585ffb640 1 -- 172.21.15.161:0/2497814463 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9578031640 con 0x7f95880c7120 2024-03-28T23:33:48.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.055+0000 7f9585ffb640 1 -- 172.21.15.161:0/2497814463 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957802d040 con 0x7f95880c7120 2024-03-28T23:33:48.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.056+0000 7f958d76c640 1 -- 172.21.15.161:0/2497814463 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95880c7120 msgr2=0x7f9588155cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:48.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.056+0000 7f958d76c640 1 --2- 172.21.15.161:0/2497814463 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95880c7120 0x7f9588155cb0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f9578009870 tx=0x7f957802b8f0 comp rx=0 tx=0).stop 2024-03-28T23:33:48.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.056+0000 7f958d76c640 1 -- 172.21.15.161:0/2497814463 shutdown_connections 2024-03-28T23:33:48.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.056+0000 7f958d76c640 1 --2- 172.21.15.161:0/2497814463 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95880c5b70 0x7f95881561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:48.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.056+0000 7f958d76c640 1 --2- 172.21.15.161:0/2497814463 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95880c7120 0x7f9588155cb0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:48.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.056+0000 7f958d76c640 1 -- 172.21.15.161:0/2497814463 >> 172.21.15.161:0/2497814463 conn(0x7f958806bc70 msgr2=0x7f95880aa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:48.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.056+0000 7f958d76c640 1 -- 172.21.15.161:0/2497814463 shutdown_connections 2024-03-28T23:33:48.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.056+0000 7f958d76c640 1 -- 172.21.15.161:0/2497814463 wait complete. 2024-03-28T23:33:48.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.057+0000 7f958d76c640 1 Processor -- start 2024-03-28T23:33:48.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.057+0000 7f958d76c640 1 -- start start 2024-03-28T23:33:48.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.058+0000 7f958d76c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95880befb0 0x7f9588152700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:48.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.058+0000 7f958d76c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95880c5b70 0x7f95881e0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:48.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.058+0000 7f958d76c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95880c7120 0x7f95881e11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:48.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.058+0000 7f958d76c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f95880c81a0 con 0x7f95880befb0 2024-03-28T23:33:48.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.058+0000 7f958d76c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f95880be560 con 0x7f95880c7120 2024-03-28T23:33:48.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.058+0000 7f958d76c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f95880c8320 con 0x7f95880c5b70 2024-03-28T23:33:48.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.058+0000 7f9586ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95880befb0 0x7f9588152700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:48.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.058+0000 7f9586ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95880befb0 0x7f9588152700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57302/0 (socket says 172.21.15.161:57302) 2024-03-28T23:33:48.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.058+0000 7f9586ffd640 1 -- 172.21.15.161:0/1628937375 learned_addr learned my addr 172.21.15.161:0/1628937375 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:48.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.058+0000 7f95867fc640 1 --2- 172.21.15.161:0/1628937375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95880c5b70 0x7f95881e0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:48.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.058+0000 7f95877fe640 1 --2- 172.21.15.161:0/1628937375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95880c7120 0x7f95881e11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:48.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.058+0000 7f9586ffd640 1 -- 172.21.15.161:0/1628937375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95880c5b70 msgr2=0x7f95881e0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:48.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.058+0000 7f9586ffd640 1 --2- 172.21.15.161:0/1628937375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95880c5b70 0x7f95881e0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:48.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.058+0000 7f9586ffd640 1 -- 172.21.15.161:0/1628937375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95880c7120 msgr2=0x7f95881e11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:48.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.058+0000 7f9586ffd640 1 --2- 172.21.15.161:0/1628937375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95880c7120 0x7f95881e11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:48.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.058+0000 7f9586ffd640 1 -- 172.21.15.161:0/1628937375 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95881dfc00 con 0x7f95880befb0 2024-03-28T23:33:48.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.058+0000 7f95877fe640 1 --2- 172.21.15.161:0/1628937375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95880c7120 0x7f95881e11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:48.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.058+0000 7f9586ffd640 1 --2- 172.21.15.161:0/1628937375 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95880befb0 0x7f9588152700 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f9578000c00 tx=0x7f95780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:48.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.058+0000 7f956f7fe640 1 -- 172.21.15.161:0/1628937375 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9578040070 con 0x7f95880befb0 2024-03-28T23:33:48.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.058+0000 7f956f7fe640 1 -- 172.21.15.161:0/1628937375 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95780399f0 con 0x7f95880befb0 2024-03-28T23:33:48.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.058+0000 7f958d76c640 1 -- 172.21.15.161:0/1628937375 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95881eeec0 con 0x7f95880befb0 2024-03-28T23:33:48.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.058+0000 7f956f7fe640 1 -- 172.21.15.161:0/1628937375 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9578035040 con 0x7f95880befb0 2024-03-28T23:33:48.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.058+0000 7f958d76c640 1 -- 172.21.15.161:0/1628937375 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f958806ce00 con 0x7f95880befb0 2024-03-28T23:33:48.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.059+0000 7f958d76c640 1 -- 172.21.15.161:0/1628937375 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95880c6c90 con 0x7f95880befb0 2024-03-28T23:33:48.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.060+0000 7f956f7fe640 1 -- 172.21.15.161:0/1628937375 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f957804b060 con 0x7f95880befb0 2024-03-28T23:33:48.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.060+0000 7f956f7fe640 1 --2- 172.21.15.161:0/1628937375 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f957406b4d0 0x7f957406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:48.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.060+0000 7f956f7fe640 1 -- 172.21.15.161:0/1628937375 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5249+0+0 (secure 0 0 0) 0x7f95780ab7f0 con 0x7f95880befb0 2024-03-28T23:33:48.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.060+0000 7f95867fc640 1 --2- 172.21.15.161:0/1628937375 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f957406b4d0 0x7f957406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:48.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.061+0000 7f95867fc640 1 --2- 172.21.15.161:0/1628937375 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f957406b4d0 0x7f957406d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f95881547c0 tx=0x7f957c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:48.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.065+0000 7f956f7fe640 1 -- 172.21.15.161:0/1628937375 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f957807a8c0 con 0x7f95880befb0 2024-03-28T23:33:48.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.239+0000 7f958d76c640 1 -- 172.21.15.161:0/1628937375 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f95880705f0 con 0x7f95880befb0 2024-03-28T23:33:48.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.239+0000 7f956f7fe640 1 -- 172.21.15.161:0/1628937375 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v105) ==== 84+0+65 (secure 0 0 0) 0x7f957807a1c0 con 0x7f95880befb0 2024-03-28T23:33:48.240 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:48.240 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2024-03-28T23:33:48.240 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:33:48.240 INFO:teuthology.orchestra.run.smithi161.stdout: "poolnum": 1, 2024-03-28T23:33:48.240 INFO:teuthology.orchestra.run.smithi161.stdout: "poolname": ".mgr" 2024-03-28T23:33:48.240 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:33:48.240 INFO:teuthology.orchestra.run.smithi161.stdout:] 2024-03-28T23:33:48.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.241+0000 7f958d76c640 1 -- 172.21.15.161:0/1628937375 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f957406b4d0 msgr2=0x7f957406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:48.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.241+0000 7f958d76c640 1 --2- 172.21.15.161:0/1628937375 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f957406b4d0 0x7f957406d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f95881547c0 tx=0x7f957c008040 comp rx=0 tx=0).stop 2024-03-28T23:33:48.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.241+0000 7f958d76c640 1 -- 172.21.15.161:0/1628937375 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95880befb0 msgr2=0x7f9588152700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:48.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.241+0000 7f958d76c640 1 --2- 172.21.15.161:0/1628937375 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95880befb0 0x7f9588152700 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f9578000c00 tx=0x7f95780046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:48.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.241+0000 7f958d76c640 1 -- 172.21.15.161:0/1628937375 shutdown_connections 2024-03-28T23:33:48.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.241+0000 7f958d76c640 1 --2- 172.21.15.161:0/1628937375 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f957406b4d0 0x7f957406d990 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:48.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.241+0000 7f958d76c640 1 --2- 172.21.15.161:0/1628937375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95880c7120 0x7f95881e11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:48.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.241+0000 7f958d76c640 1 --2- 172.21.15.161:0/1628937375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95880c5b70 0x7f95881e0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:48.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.241+0000 7f958d76c640 1 --2- 172.21.15.161:0/1628937375 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95880befb0 0x7f9588152700 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:48.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.241+0000 7f958d76c640 1 -- 172.21.15.161:0/1628937375 >> 172.21.15.161:0/1628937375 conn(0x7f958806bc70 msgr2=0x7f9588158fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:48.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.241+0000 7f958d76c640 1 -- 172.21.15.161:0/1628937375 shutdown_connections 2024-03-28T23:33:48.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.241+0000 7f958d76c640 1 -- 172.21.15.161:0/1628937375 wait complete. 2024-03-28T23:33:48.252 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-28T23:33:48.253 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-28T23:33:48.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.357+0000 7f1392274640 1 Processor -- start 2024-03-28T23:33:48.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.357+0000 7f1392274640 1 -- start start 2024-03-28T23:33:48.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.358+0000 7f1392274640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f138c0abb70 0x7f138c0abf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:48.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.358+0000 7f1392274640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f138c149ec0 con 0x7f138c0c46b0 2024-03-28T23:33:48.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.358+0000 7f1392274640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f138c153180 con 0x7f138c06d390 2024-03-28T23:33:48.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.358+0000 7f1392274640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f138c153ab0 con 0x7f138c0abb70 2024-03-28T23:33:48.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.358+0000 7f1390811640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f138c0c46b0 0x7f138c0c4b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:56018/0 (socket says 172.21.15.161:56018) 2024-03-28T23:33:48.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.358+0000 7f138bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f138c0abb70 0x7f138c0abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:48.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.358+0000 7f1390811640 1 -- 172.21.15.161:0/1470879789 learned_addr learned my addr 172.21.15.161:0/1470879789 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:48.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.358+0000 7f138affd640 1 -- 172.21.15.161:0/1470879789 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 583638528 0 0) 0x7f138c149ec0 con 0x7f138c0c46b0 2024-03-28T23:33:48.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.358+0000 7f138affd640 1 -- 172.21.15.161:0/1470879789 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1374003520 con 0x7f138c0c46b0 2024-03-28T23:33:48.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.358+0000 7f138bfff640 1 -- 172.21.15.161:0/1470879789 >> v1:172.21.15.170:6789/0 conn(0x7f138c06d390 legacy=0x7f138c06dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:48.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.358+0000 7f138bfff640 1 -- 172.21.15.161:0/1470879789 >> v1:172.21.15.161:6789/0 conn(0x7f138c0c46b0 legacy=0x7f138c0c4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:48.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.358+0000 7f138bfff640 1 -- 172.21.15.161:0/1470879789 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f138c153460 con 0x7f138c0abb70 2024-03-28T23:33:48.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.359+0000 7f138bfff640 1 --2- 172.21.15.161:0/1470879789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f138c0abb70 0x7f138c0abf50 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f137c0097a0 tx=0x7f137c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c3f25a4fa84af52 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:48.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.359+0000 7f138affd640 1 -- 172.21.15.161:0/1470879789 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137c035020 con 0x7f138c0abb70 2024-03-28T23:33:48.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.359+0000 7f138affd640 1 -- 172.21.15.161:0/1470879789 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f137c031830 con 0x7f138c0abb70 2024-03-28T23:33:48.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.359+0000 7f138affd640 1 -- 172.21.15.161:0/1470879789 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137c031c00 con 0x7f138c0abb70 2024-03-28T23:33:48.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.360+0000 7f1392274640 1 -- 172.21.15.161:0/1470879789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f138c0abb70 msgr2=0x7f138c0abf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:48.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.360+0000 7f1392274640 1 --2- 172.21.15.161:0/1470879789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f138c0abb70 0x7f138c0abf50 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f137c0097a0 tx=0x7f137c02b8d0 comp rx=0 tx=0).stop 2024-03-28T23:33:48.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.360+0000 7f1392274640 1 -- 172.21.15.161:0/1470879789 shutdown_connections 2024-03-28T23:33:48.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.360+0000 7f1392274640 1 --2- 172.21.15.161:0/1470879789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f138c0abb70 0x7f138c0abf50 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:48.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.360+0000 7f1392274640 1 -- 172.21.15.161:0/1470879789 >> 172.21.15.161:0/1470879789 conn(0x7f138c06b150 msgr2=0x7f138c06b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:48.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.360+0000 7f1392274640 1 -- 172.21.15.161:0/1470879789 shutdown_connections 2024-03-28T23:33:48.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.360+0000 7f1392274640 1 -- 172.21.15.161:0/1470879789 wait complete. 2024-03-28T23:33:48.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.361+0000 7f1392274640 1 Processor -- start 2024-03-28T23:33:48.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.361+0000 7f1392274640 1 -- start start 2024-03-28T23:33:48.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.362+0000 7f1392274640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f138c06d390 0x7f138c13ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:48.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.362+0000 7f1392274640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f138c0abb70 0x7f138c13f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:48.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.362+0000 7f1392274640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f138c0c46b0 0x7f138c154360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:48.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.362+0000 7f1392274640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f138c0c5210 con 0x7f138c0c46b0 2024-03-28T23:33:48.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.362+0000 7f1392274640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f138c06e400 con 0x7f138c06d390 2024-03-28T23:33:48.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.362+0000 7f1392274640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f138c15e900 con 0x7f138c0abb70 2024-03-28T23:33:48.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.362+0000 7f1390811640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f138c0c46b0 0x7f138c154360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:48.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.362+0000 7f1390811640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f138c0c46b0 0x7f138c154360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57320/0 (socket says 172.21.15.161:57320) 2024-03-28T23:33:48.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.362+0000 7f1390811640 1 -- 172.21.15.161:0/2789652090 learned_addr learned my addr 172.21.15.161:0/2789652090 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:48.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.362+0000 7f138bfff640 1 --2- 172.21.15.161:0/2789652090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f138c06d390 0x7f138c13ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:48.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.362+0000 7f138b7fe640 1 --2- 172.21.15.161:0/2789652090 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f138c0abb70 0x7f138c13f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:48.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.362+0000 7f1390811640 1 -- 172.21.15.161:0/2789652090 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f138c0abb70 msgr2=0x7f138c13f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:48.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.362+0000 7f1390811640 1 --2- 172.21.15.161:0/2789652090 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f138c0abb70 0x7f138c13f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:48.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.362+0000 7f1390811640 1 -- 172.21.15.161:0/2789652090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f138c06d390 msgr2=0x7f138c13ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:48.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.362+0000 7f1390811640 1 --2- 172.21.15.161:0/2789652090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f138c06d390 0x7f138c13ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:48.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.362+0000 7f1390811640 1 -- 172.21.15.161:0/2789652090 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f138c13d360 con 0x7f138c0c46b0 2024-03-28T23:33:48.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.362+0000 7f1390811640 1 --2- 172.21.15.161:0/2789652090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f138c0c46b0 0x7f138c154360 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f138400cad0 tx=0x7f138400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:48.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.363+0000 7f1388ff9640 1 -- 172.21.15.161:0/2789652090 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1384017070 con 0x7f138c0c46b0 2024-03-28T23:33:48.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.363+0000 7f1388ff9640 1 -- 172.21.15.161:0/2789652090 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f138400ac30 con 0x7f138c0c46b0 2024-03-28T23:33:48.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.363+0000 7f1388ff9640 1 -- 172.21.15.161:0/2789652090 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1384012400 con 0x7f138c0c46b0 2024-03-28T23:33:48.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.363+0000 7f1392274640 1 -- 172.21.15.161:0/2789652090 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f138c1564f0 con 0x7f138c0c46b0 2024-03-28T23:33:48.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.363+0000 7f1392274640 1 -- 172.21.15.161:0/2789652090 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f138c140cc0 con 0x7f138c0c46b0 2024-03-28T23:33:48.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.364+0000 7f1392274640 1 -- 172.21.15.161:0/2789652090 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f138c068f90 con 0x7f138c0c46b0 2024-03-28T23:33:48.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.364+0000 7f1388ff9640 1 -- 172.21.15.161:0/2789652090 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f138401a070 con 0x7f138c0c46b0 2024-03-28T23:33:48.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.364+0000 7f1388ff9640 1 --2- 172.21.15.161:0/2789652090 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f137806b500 0x7f137806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:48.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.364+0000 7f1388ff9640 1 -- 172.21.15.161:0/2789652090 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5249+0+0 (secure 0 0 0) 0x7f1384089d40 con 0x7f138c0c46b0 2024-03-28T23:33:48.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.365+0000 7f138bfff640 1 --2- 172.21.15.161:0/2789652090 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f137806b500 0x7f137806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:48.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.365+0000 7f138bfff640 1 --2- 172.21.15.161:0/2789652090 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f137806b500 0x7f137806d9c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f137c036000 tx=0x7f137c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:48.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.369+0000 7f1388ff9640 1 -- 172.21.15.161:0/2789652090 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1384058730 con 0x7f138c0c46b0 2024-03-28T23:33:48.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.541+0000 7f1392274640 1 -- 172.21.15.161:0/2789652090 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f138c06d7c0 con 0x7f138c0c46b0 2024-03-28T23:33:48.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.970+0000 7f1388ff9640 1 -- 172.21.15.161:0/2789652090 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v106) ==== 138+0+0 (secure 0 0 0) 0x7f138405c7d0 con 0x7f138c0c46b0 2024-03-28T23:33:48.971 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_metadata' created 2024-03-28T23:33:48.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.974+0000 7f1392274640 1 -- 172.21.15.161:0/2789652090 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f137806b500 msgr2=0x7f137806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:48.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.974+0000 7f1392274640 1 --2- 172.21.15.161:0/2789652090 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f137806b500 0x7f137806d9c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f137c036000 tx=0x7f137c008030 comp rx=0 tx=0).stop 2024-03-28T23:33:48.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.975+0000 7f1392274640 1 -- 172.21.15.161:0/2789652090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f138c0c46b0 msgr2=0x7f138c154360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:48.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.975+0000 7f1392274640 1 --2- 172.21.15.161:0/2789652090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f138c0c46b0 0x7f138c154360 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f138400cad0 tx=0x7f138400cf90 comp rx=0 tx=0).stop 2024-03-28T23:33:48.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.976+0000 7f1392274640 1 -- 172.21.15.161:0/2789652090 shutdown_connections 2024-03-28T23:33:48.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.976+0000 7f1392274640 1 --2- 172.21.15.161:0/2789652090 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f137806b500 0x7f137806d9c0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:48.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.976+0000 7f1392274640 1 --2- 172.21.15.161:0/2789652090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f138c0c46b0 0x7f138c154360 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:48.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.976+0000 7f1392274640 1 --2- 172.21.15.161:0/2789652090 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f138c0abb70 0x7f138c13f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:48.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.976+0000 7f1392274640 1 --2- 172.21.15.161:0/2789652090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f138c06d390 0x7f138c13ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:48.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.976+0000 7f1392274640 1 -- 172.21.15.161:0/2789652090 >> 172.21.15.161:0/2789652090 conn(0x7f138c06b150 msgr2=0x7f138c1436c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:48.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.977+0000 7f1392274640 1 -- 172.21.15.161:0/2789652090 shutdown_connections 2024-03-28T23:33:48.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:48.977+0000 7f1392274640 1 -- 172.21.15.161:0/2789652090 wait complete. 2024-03-28T23:33:48.990 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T23:33:49.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.097+0000 7fa314fe5640 1 Processor -- start 2024-03-28T23:33:49.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.097+0000 7fa314fe5640 1 -- start start 2024-03-28T23:33:49.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.098+0000 7fa314fe5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa310152bc0 0x7fa3100c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:49.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.098+0000 7fa314fe5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3100c6f60 con 0x7fa3100c8a80 2024-03-28T23:33:49.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.098+0000 7fa314fe5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3100c6120 con 0x7fa3101589b0 2024-03-28T23:33:49.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.098+0000 7fa314fe5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fa3100c6400 con 0x7fa310152bc0 2024-03-28T23:33:49.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.098+0000 7fa30dd9b640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa3100c8a80 0x7fa3100c8ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:56030/0 (socket says 172.21.15.161:56030) 2024-03-28T23:33:49.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.098+0000 7fa30dd9b640 1 -- 172.21.15.161:0/4201768006 learned_addr learned my addr 172.21.15.161:0/4201768006 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:49.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.098+0000 7fa30e59c640 1 --2- 172.21.15.161:0/4201768006 >> v2:172.21.15.170:3300/0 conn(0x7fa310152bc0 0x7fa3100c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:49.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.099+0000 7fa30d59a640 1 -- 172.21.15.161:0/4201768006 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1503620478 0 0) 0x7fa3100c6f60 con 0x7fa3100c8a80 2024-03-28T23:33:49.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.099+0000 7fa30d59a640 1 -- 172.21.15.161:0/4201768006 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2f8003520 con 0x7fa3100c8a80 2024-03-28T23:33:49.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.099+0000 7fa30d59a640 1 -- 172.21.15.161:0/4201768006 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2334133833 0 0) 0x7fa2f8003520 con 0x7fa3100c8a80 2024-03-28T23:33:49.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.099+0000 7fa30d59a640 1 -- 172.21.15.161:0/4201768006 >> v2:172.21.15.170:3300/0 conn(0x7fa310152bc0 msgr2=0x7fa3100c0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:49.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.099+0000 7fa30d59a640 1 --2- 172.21.15.161:0/4201768006 >> v2:172.21.15.170:3300/0 conn(0x7fa310152bc0 0x7fa3100c0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:49.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.099+0000 7fa30d59a640 1 -- 172.21.15.161:0/4201768006 >> v1:172.21.15.170:6789/0 conn(0x7fa3101589b0 legacy=0x7fa310158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:49.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.099+0000 7fa30d59a640 1 -- 172.21.15.161:0/4201768006 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3100c7240 con 0x7fa3100c8a80 2024-03-28T23:33:49.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.099+0000 7fa30d59a640 1 -- 172.21.15.161:0/4201768006 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fa304001670 con 0x7fa3100c8a80 2024-03-28T23:33:49.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.099+0000 7fa30d59a640 1 -- 172.21.15.161:0/4201768006 >> v1:172.21.15.161:6789/0 conn(0x7fa3100c8a80 legacy=0x7fa3100c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:49.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.099+0000 7fa30d59a640 1 --2- 172.21.15.161:0/4201768006 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2f8003a90 0x7fa2f8003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:49.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.099+0000 7fa30d59a640 1 --2- 172.21.15.161:0/4201768006 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2f8004800 0x7fa2f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:49.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.099+0000 7fa30d59a640 1 --2- 172.21.15.161:0/4201768006 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2f8005210 0x7fa2f800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:49.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.099+0000 7fa30d59a640 1 -- 172.21.15.161:0/4201768006 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa3100c6400 con 0x7fa2f8004800 2024-03-28T23:33:49.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.099+0000 7fa30d59a640 1 -- 172.21.15.161:0/4201768006 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f800c140 con 0x7fa2f8003a90 2024-03-28T23:33:49.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.099+0000 7fa30d59a640 1 -- 172.21.15.161:0/4201768006 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa2f800c340 con 0x7fa2f8005210 2024-03-28T23:33:49.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.099+0000 7fa30dd9b640 1 --2- 172.21.15.161:0/4201768006 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2f8004800 0x7fa2f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:49.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.099+0000 7fa30e59c640 1 --2- 172.21.15.161:0/4201768006 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2f8003a90 0x7fa2f8003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:49.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.099+0000 7fa30ed9d640 1 --2- 172.21.15.161:0/4201768006 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2f8005210 0x7fa2f800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:49.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.099+0000 7fa30dd9b640 1 -- 172.21.15.161:0/4201768006 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2f8005210 msgr2=0x7fa2f800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:49.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.099+0000 7fa30dd9b640 1 --2- 172.21.15.161:0/4201768006 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2f8005210 0x7fa2f800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:49.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.099+0000 7fa30dd9b640 1 -- 172.21.15.161:0/4201768006 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2f8003a90 msgr2=0x7fa2f8003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:49.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.099+0000 7fa30dd9b640 1 --2- 172.21.15.161:0/4201768006 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2f8003a90 0x7fa2f8003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:49.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.099+0000 7fa30ed9d640 1 -- 172.21.15.161:0/4201768006 reap_dead start 2024-03-28T23:33:49.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.099+0000 7fa30dd9b640 1 -- 172.21.15.161:0/4201768006 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f8003520 con 0x7fa2f8004800 2024-03-28T23:33:49.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.100+0000 7fa30dd9b640 1 --2- 172.21.15.161:0/4201768006 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2f8004800 0x7fa2f8004c80 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fa304001050 tx=0x7fa2f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d3976c03a61e94f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:49.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.100+0000 7fa30d59a640 1 -- 172.21.15.161:0/4201768006 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa304007d30 con 0x7fa2f8004800 2024-03-28T23:33:49.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.100+0000 7fa30d59a640 1 -- 172.21.15.161:0/4201768006 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa304030070 con 0x7fa2f8004800 2024-03-28T23:33:49.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.100+0000 7fa30d59a640 1 -- 172.21.15.161:0/4201768006 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa304005d30 con 0x7fa2f8004800 2024-03-28T23:33:49.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.101+0000 7fa314fe5640 1 -- 172.21.15.161:0/4201768006 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2f8004800 msgr2=0x7fa2f8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:49.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.101+0000 7fa314fe5640 1 --2- 172.21.15.161:0/4201768006 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2f8004800 0x7fa2f8004c80 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fa304001050 tx=0x7fa2f8005830 comp rx=0 tx=0).stop 2024-03-28T23:33:49.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.101+0000 7fa314fe5640 1 -- 172.21.15.161:0/4201768006 shutdown_connections 2024-03-28T23:33:49.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.101+0000 7fa314fe5640 1 --2- 172.21.15.161:0/4201768006 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2f8004800 0x7fa2f8004c80 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:49.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.101+0000 7fa314fe5640 1 -- 172.21.15.161:0/4201768006 >> 172.21.15.161:0/4201768006 conn(0x7fa3100caa80 msgr2=0x7fa3100cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:49.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.101+0000 7fa314fe5640 1 -- 172.21.15.161:0/4201768006 shutdown_connections 2024-03-28T23:33:49.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.101+0000 7fa314fe5640 1 -- 172.21.15.161:0/4201768006 wait complete. 2024-03-28T23:33:49.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.102+0000 7fa314fe5640 1 Processor -- start 2024-03-28T23:33:49.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.102+0000 7fa314fe5640 1 -- start start 2024-03-28T23:33:49.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.103+0000 7fa314fe5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2f8004800 0x7fa31006ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:49.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.103+0000 7fa314fe5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa310150bb0 0x7fa310156760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:49.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.103+0000 7fa314fe5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa310147b00 0x7fa3101512b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:49.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.103+0000 7fa314fe5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa310152fa0 con 0x7fa310147b00 2024-03-28T23:33:49.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.103+0000 7fa314fe5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa3100c0330 con 0x7fa310150bb0 2024-03-28T23:33:49.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.103+0000 7fa314fe5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa3101470b0 con 0x7fa2f8004800 2024-03-28T23:33:49.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.103+0000 7fa30ed9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa310147b00 0x7fa3101512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:49.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.103+0000 7fa30ed9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa310147b00 0x7fa3101512b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57342/0 (socket says 172.21.15.161:57342) 2024-03-28T23:33:49.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.103+0000 7fa30ed9d640 1 -- 172.21.15.161:0/1493781779 learned_addr learned my addr 172.21.15.161:0/1493781779 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:49.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.103+0000 7fa30dd9b640 1 --2- 172.21.15.161:0/1493781779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa310150bb0 0x7fa310156760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:49.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.103+0000 7fa30e59c640 1 --2- 172.21.15.161:0/1493781779 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2f8004800 0x7fa31006ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:49.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.103+0000 7fa30ed9d640 1 -- 172.21.15.161:0/1493781779 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2f8004800 msgr2=0x7fa31006ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:49.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.103+0000 7fa30ed9d640 1 --2- 172.21.15.161:0/1493781779 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2f8004800 0x7fa31006ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:49.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.103+0000 7fa30ed9d640 1 -- 172.21.15.161:0/1493781779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa310150bb0 msgr2=0x7fa310156760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:49.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.103+0000 7fa30ed9d640 1 --2- 172.21.15.161:0/1493781779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa310150bb0 0x7fa310156760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:49.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.103+0000 7fa30ed9d640 1 -- 172.21.15.161:0/1493781779 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa310156ca0 con 0x7fa310147b00 2024-03-28T23:33:49.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.103+0000 7fa30e59c640 1 --2- 172.21.15.161:0/1493781779 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2f8004800 0x7fa31006ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:49.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.103+0000 7fa30dd9b640 1 --2- 172.21.15.161:0/1493781779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa310150bb0 0x7fa310156760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:49.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.103+0000 7fa30ed9d640 1 --2- 172.21.15.161:0/1493781779 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa310147b00 0x7fa3101512b0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fa3100c0fc0 tx=0x7fa30800f960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:49.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.104+0000 7fa2feffd640 1 -- 172.21.15.161:0/1493781779 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3080053a0 con 0x7fa310147b00 2024-03-28T23:33:49.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.104+0000 7fa2feffd640 1 -- 172.21.15.161:0/1493781779 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa308018070 con 0x7fa310147b00 2024-03-28T23:33:49.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.104+0000 7fa314fe5640 1 -- 172.21.15.161:0/1493781779 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3101eabf0 con 0x7fa310147b00 2024-03-28T23:33:49.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.104+0000 7fa2feffd640 1 -- 172.21.15.161:0/1493781779 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa308003830 con 0x7fa310147b00 2024-03-28T23:33:49.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.104+0000 7fa314fe5640 1 -- 172.21.15.161:0/1493781779 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3101586e0 con 0x7fa310147b00 2024-03-28T23:33:49.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.105+0000 7fa314fe5640 1 -- 172.21.15.161:0/1493781779 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa31004f4b0 con 0x7fa310147b00 2024-03-28T23:33:49.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.105+0000 7fa2feffd640 1 -- 172.21.15.161:0/1493781779 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa3080039d0 con 0x7fa310147b00 2024-03-28T23:33:49.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.106+0000 7fa2feffd640 1 --2- 172.21.15.161:0/1493781779 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa2f006b4d0 0x7fa2f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:49.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.106+0000 7fa2feffd640 1 -- 172.21.15.161:0/1493781779 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5619+0+0 (secure 0 0 0) 0x7fa308089cb0 con 0x7fa310147b00 2024-03-28T23:33:49.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.106+0000 7fa30e59c640 1 --2- 172.21.15.161:0/1493781779 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa2f006b4d0 0x7fa2f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:49.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.106+0000 7fa30e59c640 1 --2- 172.21.15.161:0/1493781779 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa2f006b4d0 0x7fa2f006d990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fa310050db0 tx=0x7fa30000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:49.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.110+0000 7fa2feffd640 1 -- 172.21.15.161:0/1493781779 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa308058540 con 0x7fa310147b00 2024-03-28T23:33:49.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.282+0000 7fa314fe5640 1 -- 172.21.15.161:0/1493781779 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fa3100bf9c0 con 0x7fa310147b00 2024-03-28T23:33:49.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.972+0000 7fa2feffd640 1 -- 172.21.15.161:0/1493781779 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v107) ==== 170+0+0 (secure 0 0 0) 0x7fa30805c5e0 con 0x7fa310147b00 2024-03-28T23:33:49.973 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data' created 2024-03-28T23:33:49.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.975+0000 7fa314fe5640 1 -- 172.21.15.161:0/1493781779 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa2f006b4d0 msgr2=0x7fa2f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:49.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.975+0000 7fa314fe5640 1 --2- 172.21.15.161:0/1493781779 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa2f006b4d0 0x7fa2f006d990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fa310050db0 tx=0x7fa30000b040 comp rx=0 tx=0).stop 2024-03-28T23:33:49.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.975+0000 7fa314fe5640 1 -- 172.21.15.161:0/1493781779 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa310147b00 msgr2=0x7fa3101512b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:49.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.975+0000 7fa314fe5640 1 --2- 172.21.15.161:0/1493781779 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa310147b00 0x7fa3101512b0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fa3100c0fc0 tx=0x7fa30800f960 comp rx=0 tx=0).stop 2024-03-28T23:33:49.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.976+0000 7fa314fe5640 1 -- 172.21.15.161:0/1493781779 shutdown_connections 2024-03-28T23:33:49.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.976+0000 7fa314fe5640 1 --2- 172.21.15.161:0/1493781779 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa2f006b4d0 0x7fa2f006d990 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:49.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.976+0000 7fa314fe5640 1 --2- 172.21.15.161:0/1493781779 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa310147b00 0x7fa3101512b0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:49.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.976+0000 7fa314fe5640 1 --2- 172.21.15.161:0/1493781779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa310150bb0 0x7fa310156760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:49.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.976+0000 7fa314fe5640 1 --2- 172.21.15.161:0/1493781779 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2f8004800 0x7fa31006ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:49.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.976+0000 7fa314fe5640 1 -- 172.21.15.161:0/1493781779 >> 172.21.15.161:0/1493781779 conn(0x7fa3100caa80 msgr2=0x7fa31014dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:49.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.977+0000 7fa314fe5640 1 -- 172.21.15.161:0/1493781779 shutdown_connections 2024-03-28T23:33:49.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:49.977+0000 7fa314fe5640 1 -- 172.21.15.161:0/1493781779 wait complete. 2024-03-28T23:33:49.998 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-28T23:33:50.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.103+0000 7fc6d90ae640 1 Processor -- start 2024-03-28T23:33:50.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.103+0000 7fc6d90ae640 1 -- start start 2024-03-28T23:33:50.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.104+0000 7fc6d90ae640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc6d414f5c0 0x7fc6d40c2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:50.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.104+0000 7fc6d90ae640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6d406cdd0 0x7fc6d41524c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:50.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.104+0000 7fc6d90ae640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc6d40c2690 0x7fc6d4152a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:50.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.104+0000 7fc6d90ae640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc6d40c74d0 con 0x7fc6d40c2690 2024-03-28T23:33:50.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.104+0000 7fc6d90ae640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fc6d40c6fa0 con 0x7fc6d414f5c0 2024-03-28T23:33:50.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.104+0000 7fc6d90ae640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc6d4152030 con 0x7fc6d406cdd0 2024-03-28T23:33:50.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.104+0000 7fc6d359e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc6d40c2690 0x7fc6d4152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:50.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.104+0000 7fc6d359e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc6d40c2690 0x7fc6d4152a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57352/0 (socket says 172.21.15.161:57352) 2024-03-28T23:33:50.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.104+0000 7fc6d359e640 1 -- 172.21.15.161:0/1358285297 learned_addr learned my addr 172.21.15.161:0/1358285297 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:50.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.104+0000 7fc6d359e640 1 -- 172.21.15.161:0/1358285297 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6d406cdd0 msgr2=0x7fc6d41524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:50.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.104+0000 7fc6cbfff640 1 --2- 172.21.15.161:0/1358285297 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6d406cdd0 0x7fc6d41524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:50.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.104+0000 7fc6d359e640 1 --2- 172.21.15.161:0/1358285297 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6d406cdd0 0x7fc6d41524c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:50.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.104+0000 7fc6d359e640 1 -- 172.21.15.161:0/1358285297 >> v2:172.21.15.170:3300/0 conn(0x7fc6d414f5c0 msgr2=0x7fc6d40c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:50.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.104+0000 7fc6d2d9d640 1 --2- 172.21.15.161:0/1358285297 >> v2:172.21.15.170:3300/0 conn(0x7fc6d414f5c0 0x7fc6d40c2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.104+0000 7fc6d359e640 1 --2- 172.21.15.161:0/1358285297 >> v2:172.21.15.170:3300/0 conn(0x7fc6d414f5c0 0x7fc6d40c2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.104+0000 7fc6d359e640 1 -- 172.21.15.161:0/1358285297 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d40c2f10 con 0x7fc6d40c2690 2024-03-28T23:33:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.105+0000 7fc6d359e640 1 --2- 172.21.15.161:0/1358285297 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6d40c2690 0x7fc6d4152a00 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fc6d40c76e0 tx=0x7fc6cc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee5a5db9a418cabf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.105+0000 7fc6d259c640 1 -- 172.21.15.161:0/1358285297 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc02ca70 con 0x7fc6d40c2690 2024-03-28T23:33:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.105+0000 7fc6d259c640 1 -- 172.21.15.161:0/1358285297 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6cc02cc10 con 0x7fc6d40c2690 2024-03-28T23:33:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.105+0000 7fc6d259c640 1 -- 172.21.15.161:0/1358285297 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc0343f0 con 0x7fc6d40c2690 2024-03-28T23:33:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.106+0000 7fc6d90ae640 1 -- 172.21.15.161:0/1358285297 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6d40c2690 msgr2=0x7fc6d4152a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.106+0000 7fc6d90ae640 1 --2- 172.21.15.161:0/1358285297 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6d40c2690 0x7fc6d4152a00 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fc6d40c76e0 tx=0x7fc6cc02bed0 comp rx=0 tx=0).stop 2024-03-28T23:33:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.106+0000 7fc6d90ae640 1 -- 172.21.15.161:0/1358285297 shutdown_connections 2024-03-28T23:33:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.106+0000 7fc6d90ae640 1 --2- 172.21.15.161:0/1358285297 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6d40c2690 0x7fc6d4152a00 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.106+0000 7fc6d90ae640 1 --2- 172.21.15.161:0/1358285297 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6d406cdd0 0x7fc6d41524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.106+0000 7fc6d90ae640 1 --2- 172.21.15.161:0/1358285297 >> v2:172.21.15.170:3300/0 conn(0x7fc6d414f5c0 0x7fc6d40c2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:50.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.106+0000 7fc6d90ae640 1 -- 172.21.15.161:0/1358285297 >> 172.21.15.161:0/1358285297 conn(0x7fc6d40a83f0 msgr2=0x7fc6d40a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:50.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.106+0000 7fc6d90ae640 1 -- 172.21.15.161:0/1358285297 shutdown_connections 2024-03-28T23:33:50.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.106+0000 7fc6d90ae640 1 -- 172.21.15.161:0/1358285297 wait complete. 2024-03-28T23:33:50.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.107+0000 7fc6d90ae640 1 Processor -- start 2024-03-28T23:33:50.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.107+0000 7fc6d90ae640 1 -- start start 2024-03-28T23:33:50.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.108+0000 7fc6d90ae640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6d406cdd0 0x7fc6d4153910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:50.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.108+0000 7fc6d90ae640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6d40c2690 0x7fc6d4154530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:50.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.108+0000 7fc6d90ae640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6d414f5c0 0x7fc6d41ec390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:50.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.108+0000 7fc6d90ae640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d406c140 con 0x7fc6d406cdd0 2024-03-28T23:33:50.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.108+0000 7fc6d90ae640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d414f9a0 con 0x7fc6d40c2690 2024-03-28T23:33:50.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.108+0000 7fc6d90ae640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc6d406c3e0 con 0x7fc6d414f5c0 2024-03-28T23:33:50.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.108+0000 7fc6d2d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6d406cdd0 0x7fc6d4153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:50.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.108+0000 7fc6d2d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6d406cdd0 0x7fc6d4153910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57360/0 (socket says 172.21.15.161:57360) 2024-03-28T23:33:50.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.108+0000 7fc6d2d9d640 1 -- 172.21.15.161:0/4211753927 learned_addr learned my addr 172.21.15.161:0/4211753927 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:50.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.108+0000 7fc6d359e640 1 --2- 172.21.15.161:0/4211753927 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6d414f5c0 0x7fc6d41ec390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:50.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.108+0000 7fc6cbfff640 1 --2- 172.21.15.161:0/4211753927 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6d40c2690 0x7fc6d4154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:50.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.108+0000 7fc6d2d9d640 1 -- 172.21.15.161:0/4211753927 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6d414f5c0 msgr2=0x7fc6d41ec390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:50.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.108+0000 7fc6d2d9d640 1 --2- 172.21.15.161:0/4211753927 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6d414f5c0 0x7fc6d41ec390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:50.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.108+0000 7fc6d2d9d640 1 -- 172.21.15.161:0/4211753927 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6d40c2690 msgr2=0x7fc6d4154530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:50.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.108+0000 7fc6d2d9d640 1 --2- 172.21.15.161:0/4211753927 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6d40c2690 0x7fc6d4154530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:50.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.108+0000 7fc6d2d9d640 1 -- 172.21.15.161:0/4211753927 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d41eccd0 con 0x7fc6d406cdd0 2024-03-28T23:33:50.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.108+0000 7fc6d2d9d640 1 --2- 172.21.15.161:0/4211753927 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6d406cdd0 0x7fc6d4153910 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400c820 tx=0x7fc6c400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:50.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.108+0000 7fc6cb7fe640 1 -- 172.21.15.161:0/4211753927 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4015070 con 0x7fc6d406cdd0 2024-03-28T23:33:50.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.108+0000 7fc6cb7fe640 1 -- 172.21.15.161:0/4211753927 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6c4004020 con 0x7fc6d406cdd0 2024-03-28T23:33:50.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.108+0000 7fc6d90ae640 1 -- 172.21.15.161:0/4211753927 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6d4149d70 con 0x7fc6d406cdd0 2024-03-28T23:33:50.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.108+0000 7fc6cb7fe640 1 -- 172.21.15.161:0/4211753927 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c40043f0 con 0x7fc6d406cdd0 2024-03-28T23:33:50.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.108+0000 7fc6d90ae640 1 -- 172.21.15.161:0/4211753927 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6d41491c0 con 0x7fc6d406cdd0 2024-03-28T23:33:50.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.109+0000 7fc6d90ae640 1 -- 172.21.15.161:0/4211753927 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6d404f4b0 con 0x7fc6d406cdd0 2024-03-28T23:33:50.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.110+0000 7fc6cb7fe640 1 -- 172.21.15.161:0/4211753927 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc6c4007830 con 0x7fc6d406cdd0 2024-03-28T23:33:50.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.110+0000 7fc6cb7fe640 1 --2- 172.21.15.161:0/4211753927 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc6b406b500 0x7fc6b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:50.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.110+0000 7fc6cb7fe640 1 -- 172.21.15.161:0/4211753927 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 6001+0+0 (secure 0 0 0) 0x7fc6c408a0a0 con 0x7fc6d406cdd0 2024-03-28T23:33:50.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.110+0000 7fc6cbfff640 1 --2- 172.21.15.161:0/4211753927 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc6b406b500 0x7fc6b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:50.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.111+0000 7fc6cbfff640 1 --2- 172.21.15.161:0/4211753927 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc6b406b500 0x7fc6b406d9c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc006fd0 tx=0x7fc6bc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:50.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.115+0000 7fc6cb7fe640 1 -- 172.21.15.161:0/4211753927 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc6c4058df0 con 0x7fc6d406cdd0 2024-03-28T23:33:50.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:50.324+0000 7fc6d90ae640 1 -- 172.21.15.161:0/4211753927 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fc6d40c1370 con 0x7fc6d406cdd0 2024-03-28T23:33:51.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.006+0000 7fc6cb7fe640 1 -- 172.21.15.161:0/4211753927 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 17 and data pool 18 v88) ==== 174+0+0 (secure 0 0 0) 0x7fc6c40586f0 con 0x7fc6d406cdd0 2024-03-28T23:33:51.007 INFO:teuthology.orchestra.run.smithi161.stderr:new fs with metadata pool 17 and data pool 18 2024-03-28T23:33:51.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.018+0000 7fc6d90ae640 1 -- 172.21.15.161:0/4211753927 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc6b406b500 msgr2=0x7fc6b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:51.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.018+0000 7fc6d90ae640 1 --2- 172.21.15.161:0/4211753927 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc6b406b500 0x7fc6b406d9c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc006fd0 tx=0x7fc6bc008040 comp rx=0 tx=0).stop 2024-03-28T23:33:51.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.018+0000 7fc6d90ae640 1 -- 172.21.15.161:0/4211753927 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6d406cdd0 msgr2=0x7fc6d4153910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:51.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.018+0000 7fc6d90ae640 1 --2- 172.21.15.161:0/4211753927 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6d406cdd0 0x7fc6d4153910 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400c820 tx=0x7fc6c400cce0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.019+0000 7fc6d90ae640 1 -- 172.21.15.161:0/4211753927 shutdown_connections 2024-03-28T23:33:51.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.019+0000 7fc6d90ae640 1 --2- 172.21.15.161:0/4211753927 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc6b406b500 0x7fc6b406d9c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.019+0000 7fc6d90ae640 1 --2- 172.21.15.161:0/4211753927 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc6d414f5c0 0x7fc6d41ec390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.019+0000 7fc6d90ae640 1 --2- 172.21.15.161:0/4211753927 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6d40c2690 0x7fc6d4154530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.019+0000 7fc6d90ae640 1 --2- 172.21.15.161:0/4211753927 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6d406cdd0 0x7fc6d4153910 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.019+0000 7fc6d90ae640 1 -- 172.21.15.161:0/4211753927 >> 172.21.15.161:0/4211753927 conn(0x7fc6d40a83f0 msgr2=0x7fc6d40a94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:51.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.021+0000 7fc6d90ae640 1 -- 172.21.15.161:0/4211753927 shutdown_connections 2024-03-28T23:33:51.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.022+0000 7fc6d90ae640 1 -- 172.21.15.161:0/4211753927 wait complete. 2024-03-28T23:33:51.039 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:33:51.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.141+0000 7f714b1b5640 1 Processor -- start 2024-03-28T23:33:51.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.141+0000 7f714b1b5640 1 -- start start 2024-03-28T23:33:51.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.141+0000 7f714b1b5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7144152890 0x7f71440c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:51.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.141+0000 7f714b1b5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f71440c8ed0 0x7f71440c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:51.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.141+0000 7f714b1b5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7144158a30 con 0x7f71440c5950 2024-03-28T23:33:51.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.141+0000 7f714b1b5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f71440c5db0 con 0x7f71440c8ed0 2024-03-28T23:33:51.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.141+0000 7f714b1b5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f71440c8480 con 0x7f7144152890 2024-03-28T23:33:51.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.142+0000 7f7143fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f71440c8ed0 0x7f71440c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:51.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.142+0000 7f7148f51640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7144152890 0x7f71440c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:51.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.142+0000 7f7148f51640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7144152890 0x7f71440c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:55244/0 (socket says 172.21.15.161:55244) 2024-03-28T23:33:51.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.142+0000 7f7143fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f71440c8ed0 0x7f71440c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:51266/0 (socket says 172.21.15.161:51266) 2024-03-28T23:33:51.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.142+0000 7f7148f51640 1 -- 172.21.15.161:0/1326695655 learned_addr learned my addr 172.21.15.161:0/1326695655 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:51.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.142+0000 7f7149752640 1 --1- 172.21.15.161:0/1326695655 >> v1:172.21.15.170:6789/0 conn(0x7f71440c5950 0x7f71441553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:56776/0 (socket says 172.21.15.161:56776) 2024-03-28T23:33:51.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.142+0000 7f7148f51640 1 -- 172.21.15.161:0/1326695655 >> v2:172.21.15.170:3300/0 conn(0x7f71440c8ed0 msgr2=0x7f71440c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:51.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.142+0000 7f7148f51640 1 --2- 172.21.15.161:0/1326695655 >> v2:172.21.15.170:3300/0 conn(0x7f71440c8ed0 0x7f71440c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.142+0000 7f7148f51640 1 -- 172.21.15.161:0/1326695655 >> v1:172.21.15.170:6789/0 conn(0x7f71440c5950 legacy=0x7f71441553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:51.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.142+0000 7f7148f51640 1 -- 172.21.15.161:0/1326695655 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7144158d10 con 0x7f7144152890 2024-03-28T23:33:51.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.142+0000 7f7143fff640 1 --2- 172.21.15.161:0/1326695655 >> v2:172.21.15.170:3300/0 conn(0x7f71440c8ed0 0x7f71440c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:33:51.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.143+0000 7f7148f51640 1 --2- 172.21.15.161:0/1326695655 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7144152890 0x7f71440c8990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f713c009e50 tx=0x7f713c02be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf6b45e2d9c8c1de server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:51.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.143+0000 7f71437fe640 1 -- 172.21.15.161:0/1326695655 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f713c02c9e0 con 0x7f7144152890 2024-03-28T23:33:51.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.143+0000 7f71437fe640 1 -- 172.21.15.161:0/1326695655 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f713c02cb80 con 0x7f7144152890 2024-03-28T23:33:51.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.143+0000 7f71437fe640 1 -- 172.21.15.161:0/1326695655 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f713c0343f0 con 0x7f7144152890 2024-03-28T23:33:51.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.143+0000 7f714b1b5640 1 -- 172.21.15.161:0/1326695655 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7144152890 msgr2=0x7f71440c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:51.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.143+0000 7f714b1b5640 1 --2- 172.21.15.161:0/1326695655 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7144152890 0x7f71440c8990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f713c009e50 tx=0x7f713c02be40 comp rx=0 tx=0).stop 2024-03-28T23:33:51.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.143+0000 7f714b1b5640 1 -- 172.21.15.161:0/1326695655 shutdown_connections 2024-03-28T23:33:51.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.143+0000 7f714b1b5640 1 --2- 172.21.15.161:0/1326695655 >> v2:172.21.15.170:3300/0 conn(0x7f71440c8ed0 0x7f71440c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.143+0000 7f714b1b5640 1 --2- 172.21.15.161:0/1326695655 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7144152890 0x7f71440c8990 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.143+0000 7f714b1b5640 1 -- 172.21.15.161:0/1326695655 >> 172.21.15.161:0/1326695655 conn(0x7f71440ac620 msgr2=0x7f71440aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:51.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.143+0000 7f714b1b5640 1 -- 172.21.15.161:0/1326695655 shutdown_connections 2024-03-28T23:33:51.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.143+0000 7f714b1b5640 1 -- 172.21.15.161:0/1326695655 wait complete. 2024-03-28T23:33:51.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.145+0000 7f714b1b5640 1 Processor -- start 2024-03-28T23:33:51.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.145+0000 7f714b1b5640 1 -- start start 2024-03-28T23:33:51.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.145+0000 7f714b1b5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71440c5950 0x7f714414cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:51.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.145+0000 7f714b1b5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f71440c8ed0 0x7f71440ab6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:51.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.145+0000 7f714b1b5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7144152890 0x7f71441ee7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:51.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.145+0000 7f714b1b5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f71441592d0 con 0x7f71440c5950 2024-03-28T23:33:51.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.145+0000 7f714b1b5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7144159640 con 0x7f7144152890 2024-03-28T23:33:51.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.145+0000 7f714b1b5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7144152c70 con 0x7f71440c8ed0 2024-03-28T23:33:51.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.145+0000 7f7148f51640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71440c5950 0x7f714414cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:51.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.145+0000 7f7148f51640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71440c5950 0x7f714414cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59978/0 (socket says 172.21.15.161:59978) 2024-03-28T23:33:51.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.145+0000 7f7148f51640 1 -- 172.21.15.161:0/633448461 learned_addr learned my addr 172.21.15.161:0/633448461 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:51.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.146+0000 7f7149752640 1 --2- 172.21.15.161:0/633448461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7144152890 0x7f71441ee7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:51.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.146+0000 7f7143fff640 1 --2- 172.21.15.161:0/633448461 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f71440c8ed0 0x7f71440ab6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:51.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.146+0000 7f7148f51640 1 -- 172.21.15.161:0/633448461 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f71440c8ed0 msgr2=0x7f71440ab6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:51.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.146+0000 7f7148f51640 1 --2- 172.21.15.161:0/633448461 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f71440c8ed0 0x7f71440ab6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.146+0000 7f7148f51640 1 -- 172.21.15.161:0/633448461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7144152890 msgr2=0x7f71441ee7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:51.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.146+0000 7f7148f51640 1 --2- 172.21.15.161:0/633448461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7144152890 0x7f71441ee7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.146+0000 7f7148f51640 1 -- 172.21.15.161:0/633448461 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f714414fca0 con 0x7f71440c5950 2024-03-28T23:33:51.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.146+0000 7f7148f51640 1 --2- 172.21.15.161:0/633448461 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71440c5950 0x7f714414cb30 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f713c000f80 tx=0x7f713c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:51.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.146+0000 7f71417fa640 1 -- 172.21.15.161:0/633448461 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f713c043020 con 0x7f71440c5950 2024-03-28T23:33:51.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.146+0000 7f71417fa640 1 -- 172.21.15.161:0/633448461 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f713c03e450 con 0x7f71440c5950 2024-03-28T23:33:51.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.146+0000 7f714b1b5640 1 -- 172.21.15.161:0/633448461 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f714414d1c0 con 0x7f71440c5950 2024-03-28T23:33:51.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.146+0000 7f71417fa640 1 -- 172.21.15.161:0/633448461 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f713c03e840 con 0x7f71440c5950 2024-03-28T23:33:51.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.146+0000 7f714b1b5640 1 -- 172.21.15.161:0/633448461 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f714414d780 con 0x7f71440c5950 2024-03-28T23:33:51.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.147+0000 7f714b1b5640 1 -- 172.21.15.161:0/633448461 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f714404f4b0 con 0x7f71440c5950 2024-03-28T23:33:51.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.148+0000 7f71417fa640 1 -- 172.21.15.161:0/633448461 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f713c046020 con 0x7f71440c5950 2024-03-28T23:33:51.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.148+0000 7f71417fa640 1 --2- 172.21.15.161:0/633448461 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f712406b4d0 0x7f712406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:51.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.148+0000 7f71417fa640 1 -- 172.21.15.161:0/633448461 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 6101+0+0 (secure 0 0 0) 0x7f713c0acca0 con 0x7f71440c5950 2024-03-28T23:33:51.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.148+0000 7f7143fff640 1 --2- 172.21.15.161:0/633448461 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f712406b4d0 0x7f712406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:51.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.148+0000 7f7143fff640 1 --2- 172.21.15.161:0/633448461 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f712406b4d0 0x7f712406d990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f712c009500 tx=0x7f712c009110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:51.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.152+0000 7f71417fa640 1 -- 172.21.15.161:0/633448461 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f713c04d030 con 0x7f71440c5950 2024-03-28T23:33:51.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.327+0000 7f714b1b5640 1 -- 172.21.15.161:0/633448461 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f71440c8140 con 0x7f71440c5950 2024-03-28T23:33:51.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.328+0000 7f71417fa640 1 -- 172.21.15.161:0/633448461 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) ==== 74+0+17001 (secure 0 0 0) 0x7f713c07b470 con 0x7f71440c5950 2024-03-28T23:33:51.329 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:51.329 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":108,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:33:50.326182+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:33:48.560815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-28T23:33:49.301625+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:33:51.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.330+0000 7f714b1b5640 1 -- 172.21.15.161:0/633448461 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f712406b4d0 msgr2=0x7f712406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:51.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.330+0000 7f714b1b5640 1 --2- 172.21.15.161:0/633448461 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f712406b4d0 0x7f712406d990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f712c009500 tx=0x7f712c009110 comp rx=0 tx=0).stop 2024-03-28T23:33:51.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.330+0000 7f714b1b5640 1 -- 172.21.15.161:0/633448461 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71440c5950 msgr2=0x7f714414cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:51.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.330+0000 7f714b1b5640 1 --2- 172.21.15.161:0/633448461 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71440c5950 0x7f714414cb30 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f713c000f80 tx=0x7f713c004a40 comp rx=0 tx=0).stop 2024-03-28T23:33:51.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.330+0000 7f714b1b5640 1 -- 172.21.15.161:0/633448461 shutdown_connections 2024-03-28T23:33:51.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.330+0000 7f714b1b5640 1 --2- 172.21.15.161:0/633448461 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f712406b4d0 0x7f712406d990 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.330+0000 7f714b1b5640 1 --2- 172.21.15.161:0/633448461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7144152890 0x7f71441ee7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.330+0000 7f714b1b5640 1 --2- 172.21.15.161:0/633448461 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f71440c8ed0 0x7f71440ab6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.330+0000 7f714b1b5640 1 --2- 172.21.15.161:0/633448461 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71440c5950 0x7f714414cb30 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.330+0000 7f714b1b5640 1 -- 172.21.15.161:0/633448461 >> 172.21.15.161:0/633448461 conn(0x7f71440ac620 msgr2=0x7f714414ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:51.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.330+0000 7f714b1b5640 1 -- 172.21.15.161:0/633448461 shutdown_connections 2024-03-28T23:33:51.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.330+0000 7f714b1b5640 1 -- 172.21.15.161:0/633448461 wait complete. 2024-03-28T23:33:51.342 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:33:51.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.442+0000 7f96b698a640 1 Processor -- start 2024-03-28T23:33:51.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.442+0000 7f96b698a640 1 -- start start 2024-03-28T23:33:51.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.442+0000 7f96b698a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f96b0157150 0x7f96b0153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:51.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.443+0000 7f96b698a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96b0154480 0x7f96b00c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:51.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.443+0000 7f96b698a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96b00c4060 con 0x7f96b0150710 2024-03-28T23:33:51.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.443+0000 7f96b698a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f96b0153130 con 0x7f96b0157150 2024-03-28T23:33:51.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.443+0000 7f96b698a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f96b00c3d80 con 0x7f96b0154480 2024-03-28T23:33:51.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.443+0000 7f96af7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f96b0157150 0x7f96b0153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:51.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.443+0000 7f96affff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f96b0150710 0x7f96b00c3530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41430/0 (socket says 172.21.15.161:41430) 2024-03-28T23:33:51.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.443+0000 7f96affff640 1 -- 172.21.15.161:0/1996364683 learned_addr learned my addr 172.21.15.161:0/1996364683 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:51.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.443+0000 7f96b4f27640 1 --2- 172.21.15.161:0/1996364683 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96b0154480 0x7f96b00c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:51.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.443+0000 7f96af7fe640 1 -- 172.21.15.161:0/1996364683 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96b0154480 msgr2=0x7f96b00c66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:51.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.443+0000 7f96af7fe640 1 --2- 172.21.15.161:0/1996364683 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96b0154480 0x7f96b00c66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.443+0000 7f96af7fe640 1 -- 172.21.15.161:0/1996364683 >> v1:172.21.15.161:6789/0 conn(0x7f96b0150710 legacy=0x7f96b00c3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:51.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.443+0000 7f96aeffd640 1 -- 172.21.15.161:0/1996364683 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1951124649 0 0) 0x7f96b00c4060 con 0x7f96b0150710 2024-03-28T23:33:51.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.443+0000 7f96af7fe640 1 -- 172.21.15.161:0/1996364683 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b01533d0 con 0x7f96b0157150 2024-03-28T23:33:51.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.443+0000 7f96b4f27640 1 --2- 172.21.15.161:0/1996364683 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96b0154480 0x7f96b00c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:51.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.444+0000 7f96af7fe640 1 --2- 172.21.15.161:0/1996364683 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96b0157150 0x7f96b0153bd0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f96a40097a0 tx=0x7f96a402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=960af8517c3fe182 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:51.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.444+0000 7f96aeffd640 1 -- 172.21.15.161:0/1996364683 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a40315a0 con 0x7f96b0157150 2024-03-28T23:33:51.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.444+0000 7f96aeffd640 1 -- 172.21.15.161:0/1996364683 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96a4037070 con 0x7f96b0157150 2024-03-28T23:33:51.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.444+0000 7f96aeffd640 1 -- 172.21.15.161:0/1996364683 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a4031960 con 0x7f96b0157150 2024-03-28T23:33:51.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.444+0000 7f96b698a640 1 -- 172.21.15.161:0/1996364683 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96b0157150 msgr2=0x7f96b0153bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:51.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.444+0000 7f96b698a640 1 --2- 172.21.15.161:0/1996364683 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96b0157150 0x7f96b0153bd0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f96a40097a0 tx=0x7f96a402b840 comp rx=0 tx=0).stop 2024-03-28T23:33:51.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.444+0000 7f96b698a640 1 -- 172.21.15.161:0/1996364683 shutdown_connections 2024-03-28T23:33:51.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.444+0000 7f96b698a640 1 --2- 172.21.15.161:0/1996364683 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96b0154480 0x7f96b00c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.444+0000 7f96b698a640 1 --2- 172.21.15.161:0/1996364683 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96b0157150 0x7f96b0153bd0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.444+0000 7f96b698a640 1 -- 172.21.15.161:0/1996364683 >> 172.21.15.161:0/1996364683 conn(0x7f96b00c7790 msgr2=0x7f96b00c7ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:51.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.445+0000 7f96b698a640 1 -- 172.21.15.161:0/1996364683 shutdown_connections 2024-03-28T23:33:51.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.445+0000 7f96b698a640 1 -- 172.21.15.161:0/1996364683 wait complete. 2024-03-28T23:33:51.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.446+0000 7f96b698a640 1 Processor -- start 2024-03-28T23:33:51.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.446+0000 7f96b698a640 1 -- start start 2024-03-28T23:33:51.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.446+0000 7f96b698a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96b0150710 0x7f96b0155070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:51.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.446+0000 7f96b698a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96b0154480 0x7f96b01e5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:51.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.446+0000 7f96b698a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96b0157150 0x7f96b01ebc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:51.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.446+0000 7f96b698a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f96b0154a90 con 0x7f96b0157150 2024-03-28T23:33:51.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.446+0000 7f96b698a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f96b0154910 con 0x7f96b0150710 2024-03-28T23:33:51.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.446+0000 7f96b698a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f96b0154c10 con 0x7f96b0154480 2024-03-28T23:33:51.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.447+0000 7f96b4f27640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96b0157150 0x7f96b01ebc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:51.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.447+0000 7f96b4f27640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96b0157150 0x7f96b01ebc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60000/0 (socket says 172.21.15.161:60000) 2024-03-28T23:33:51.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.447+0000 7f96b4f27640 1 -- 172.21.15.161:0/4211101946 learned_addr learned my addr 172.21.15.161:0/4211101946 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:51.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.447+0000 7f96affff640 1 --2- 172.21.15.161:0/4211101946 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96b0150710 0x7f96b0155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:51.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.447+0000 7f96af7fe640 1 --2- 172.21.15.161:0/4211101946 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96b0154480 0x7f96b01e5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:51.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.447+0000 7f96b4f27640 1 -- 172.21.15.161:0/4211101946 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96b0154480 msgr2=0x7f96b01e5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:51.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.447+0000 7f96b4f27640 1 --2- 172.21.15.161:0/4211101946 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96b0154480 0x7f96b01e5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.447+0000 7f96b4f27640 1 -- 172.21.15.161:0/4211101946 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96b0150710 msgr2=0x7f96b0155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:51.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.447+0000 7f96b4f27640 1 --2- 172.21.15.161:0/4211101946 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96b0150710 0x7f96b0155070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.447+0000 7f96b4f27640 1 -- 172.21.15.161:0/4211101946 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b0156990 con 0x7f96b0157150 2024-03-28T23:33:51.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.447+0000 7f96b4f27640 1 --2- 172.21.15.161:0/4211101946 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96b0157150 0x7f96b01ebc90 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f96a800cd80 tx=0x7f96a800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:51.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.447+0000 7f96acff9640 1 -- 172.21.15.161:0/4211101946 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a800f680 con 0x7f96b0157150 2024-03-28T23:33:51.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.447+0000 7f96acff9640 1 -- 172.21.15.161:0/4211101946 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96a800f840 con 0x7f96b0157150 2024-03-28T23:33:51.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.447+0000 7f96acff9640 1 -- 172.21.15.161:0/4211101946 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a800fc30 con 0x7f96b0157150 2024-03-28T23:33:51.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.447+0000 7f96b698a640 1 -- 172.21.15.161:0/4211101946 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96b01e6390 con 0x7f96b0157150 2024-03-28T23:33:51.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.447+0000 7f96b698a640 1 -- 172.21.15.161:0/4211101946 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96b01e9520 con 0x7f96b0157150 2024-03-28T23:33:51.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.448+0000 7f96b698a640 1 -- 172.21.15.161:0/4211101946 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96b004f4b0 con 0x7f96b0157150 2024-03-28T23:33:51.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.449+0000 7f96acff9640 1 -- 172.21.15.161:0/4211101946 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f96a80213f0 con 0x7f96b0157150 2024-03-28T23:33:51.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.449+0000 7f96acff9640 1 --2- 172.21.15.161:0/4211101946 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f969c06b500 0x7f969c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:51.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.449+0000 7f96acff9640 1 -- 172.21.15.161:0/4211101946 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 6101+0+0 (secure 0 0 0) 0x7f96a808b070 con 0x7f96b0157150 2024-03-28T23:33:51.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.449+0000 7f96affff640 1 --2- 172.21.15.161:0/4211101946 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f969c06b500 0x7f969c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:51.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.450+0000 7f96affff640 1 --2- 172.21.15.161:0/4211101946 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f969c06b500 0x7f969c06d9c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f96a0000f60 tx=0x7f96a0007c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:51.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.454+0000 7f96acff9640 1 -- 172.21.15.161:0/4211101946 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f96a8014030 con 0x7f96b0157150 2024-03-28T23:33:51.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.628+0000 7f96b698a640 1 -- 172.21.15.161:0/4211101946 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f96b00c32f0 con 0x7f96b0157150 2024-03-28T23:33:51.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.630+0000 7f96acff9640 1 -- 172.21.15.161:0/4211101946 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) ==== 74+0+17001 (secure 0 0 0) 0x7f96a80597a0 con 0x7f96b0157150 2024-03-28T23:33:51.630 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:51.631 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":108,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:33:50.326182+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:33:48.560815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-28T23:33:49.301625+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:33:51.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.631+0000 7f96b698a640 1 -- 172.21.15.161:0/4211101946 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f969c06b500 msgr2=0x7f969c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:51.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.631+0000 7f96b698a640 1 --2- 172.21.15.161:0/4211101946 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f969c06b500 0x7f969c06d9c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f96a0000f60 tx=0x7f96a0007c00 comp rx=0 tx=0).stop 2024-03-28T23:33:51.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.631+0000 7f96b698a640 1 -- 172.21.15.161:0/4211101946 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96b0157150 msgr2=0x7f96b01ebc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:51.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.631+0000 7f96b698a640 1 --2- 172.21.15.161:0/4211101946 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96b0157150 0x7f96b01ebc90 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f96a800cd80 tx=0x7f96a800a650 comp rx=0 tx=0).stop 2024-03-28T23:33:51.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.632+0000 7f96b698a640 1 -- 172.21.15.161:0/4211101946 shutdown_connections 2024-03-28T23:33:51.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.632+0000 7f96b698a640 1 --2- 172.21.15.161:0/4211101946 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f969c06b500 0x7f969c06d9c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.632+0000 7f96b698a640 1 --2- 172.21.15.161:0/4211101946 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96b0157150 0x7f96b01ebc90 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.632+0000 7f96b698a640 1 --2- 172.21.15.161:0/4211101946 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96b0154480 0x7f96b01e5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.632+0000 7f96b698a640 1 --2- 172.21.15.161:0/4211101946 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96b0150710 0x7f96b0155070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.632+0000 7f96b698a640 1 -- 172.21.15.161:0/4211101946 >> 172.21.15.161:0/4211101946 conn(0x7f96b00c7790 msgr2=0x7f96b00c5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:51.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.632+0000 7f96b698a640 1 -- 172.21.15.161:0/4211101946 shutdown_connections 2024-03-28T23:33:51.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.632+0000 7f96b698a640 1 -- 172.21.15.161:0/4211101946 wait complete. 2024-03-28T23:33:51.643 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-28T23:33:51.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.747+0000 7ff03ff5e640 1 Processor -- start 2024-03-28T23:33:51.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.747+0000 7ff03ff5e640 1 -- start start 2024-03-28T23:33:51.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.748+0000 7ff03ff5e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff0380733e0 0x7ff03814eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:51.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.748+0000 7ff03ff5e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0380abc00 con 0x7ff0380a9960 2024-03-28T23:33:51.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.748+0000 7ff03ff5e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff03806bf70 con 0x7ff0380c7330 2024-03-28T23:33:51.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.748+0000 7ff03ff5e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7ff0380aa3a0 con 0x7ff0380733e0 2024-03-28T23:33:51.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.748+0000 7ff03dcfa640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ff0380a9960 0x7ff038072970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41442/0 (socket says 172.21.15.161:41442) 2024-03-28T23:33:51.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.748+0000 7ff03dcfa640 1 -- 172.21.15.161:0/3197783343 learned_addr learned my addr 172.21.15.161:0/3197783343 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:51.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.748+0000 7ff03e4fb640 1 --2- 172.21.15.161:0/3197783343 >> v2:172.21.15.170:3300/0 conn(0x7ff0380733e0 0x7ff03814eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:51.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.748+0000 7ff03ccf8640 1 -- 172.21.15.161:0/3197783343 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1362716102 0 0) 0x7ff0380abc00 con 0x7ff0380a9960 2024-03-28T23:33:51.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.748+0000 7ff03ccf8640 1 -- 172.21.15.161:0/3197783343 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff024003520 con 0x7ff0380a9960 2024-03-28T23:33:51.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.748+0000 7ff03e4fb640 1 -- 172.21.15.161:0/3197783343 >> v1:172.21.15.170:6789/0 conn(0x7ff0380c7330 legacy=0x7ff03814e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:51.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.748+0000 7ff03e4fb640 1 -- 172.21.15.161:0/3197783343 >> v1:172.21.15.161:6789/0 conn(0x7ff0380a9960 legacy=0x7ff038072970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:51.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.748+0000 7ff03ccf8640 1 -- 172.21.15.161:0/3197783343 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1016549345 0 0) 0x7ff024003520 con 0x7ff0380a9960 2024-03-28T23:33:51.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.748+0000 7ff03e4fb640 1 -- 172.21.15.161:0/3197783343 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff03806bcd0 con 0x7ff0380733e0 2024-03-28T23:33:51.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.749+0000 7ff03e4fb640 1 --2- 172.21.15.161:0/3197783343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0380733e0 0x7ff03814eb70 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff030009d90 tx=0x7ff03002bcf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78d22a617871e8dc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:51.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.749+0000 7ff03ccf8640 1 -- 172.21.15.161:0/3197783343 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff030039070 con 0x7ff0380733e0 2024-03-28T23:33:51.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.749+0000 7ff03ccf8640 1 -- 172.21.15.161:0/3197783343 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff03002c820 con 0x7ff0380733e0 2024-03-28T23:33:51.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.749+0000 7ff03ccf8640 1 -- 172.21.15.161:0/3197783343 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff034001670 con 0x7ff0380733e0 2024-03-28T23:33:51.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.750+0000 7ff03ff5e640 1 -- 172.21.15.161:0/3197783343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0380733e0 msgr2=0x7ff03814eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:51.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.750+0000 7ff03ff5e640 1 --2- 172.21.15.161:0/3197783343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0380733e0 0x7ff03814eb70 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff030009d90 tx=0x7ff03002bcf0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.750+0000 7ff03ff5e640 1 -- 172.21.15.161:0/3197783343 shutdown_connections 2024-03-28T23:33:51.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.750+0000 7ff03ff5e640 1 --2- 172.21.15.161:0/3197783343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0380733e0 0x7ff03814eb70 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.750+0000 7ff03ff5e640 1 -- 172.21.15.161:0/3197783343 >> 172.21.15.161:0/3197783343 conn(0x7ff0380a83f0 msgr2=0x7ff0380a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:51.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.750+0000 7ff03ff5e640 1 -- 172.21.15.161:0/3197783343 shutdown_connections 2024-03-28T23:33:51.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.750+0000 7ff03ff5e640 1 -- 172.21.15.161:0/3197783343 wait complete. 2024-03-28T23:33:51.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.751+0000 7ff03ff5e640 1 Processor -- start 2024-03-28T23:33:51.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.751+0000 7ff03ff5e640 1 -- start start 2024-03-28T23:33:51.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.752+0000 7ff03ff5e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0380733e0 0x7ff0380c5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:51.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.752+0000 7ff03ff5e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0380a9960 0x7ff0380bfc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:51.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.752+0000 7ff03ff5e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0380c7330 0x7ff0380c0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:51.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.752+0000 7ff03ff5e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff0380ab820 con 0x7ff0380a9960 2024-03-28T23:33:51.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.752+0000 7ff03ff5e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff038152250 con 0x7ff0380c7330 2024-03-28T23:33:51.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.752+0000 7ff03ff5e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff0380ab9a0 con 0x7ff0380733e0 2024-03-28T23:33:51.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.752+0000 7ff03d4f9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0380a9960 0x7ff0380bfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:51.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.752+0000 7ff03d4f9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0380a9960 0x7ff0380bfc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60014/0 (socket says 172.21.15.161:60014) 2024-03-28T23:33:51.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.752+0000 7ff03d4f9640 1 -- 172.21.15.161:0/1920837049 learned_addr learned my addr 172.21.15.161:0/1920837049 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:51.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.752+0000 7ff03e4fb640 1 --2- 172.21.15.161:0/1920837049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0380c7330 0x7ff0380c0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:51.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.752+0000 7ff03dcfa640 1 --2- 172.21.15.161:0/1920837049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0380733e0 0x7ff0380c5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:51.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.752+0000 7ff03d4f9640 1 -- 172.21.15.161:0/1920837049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0380733e0 msgr2=0x7ff0380c5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:51.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.752+0000 7ff03d4f9640 1 --2- 172.21.15.161:0/1920837049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0380733e0 0x7ff0380c5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.752+0000 7ff03d4f9640 1 -- 172.21.15.161:0/1920837049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0380c7330 msgr2=0x7ff0380c0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:51.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.752+0000 7ff03d4f9640 1 --2- 172.21.15.161:0/1920837049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0380c7330 0x7ff0380c0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:51.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.752+0000 7ff03d4f9640 1 -- 172.21.15.161:0/1920837049 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0381511e0 con 0x7ff0380a9960 2024-03-28T23:33:51.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.752+0000 7ff03d4f9640 1 --2- 172.21.15.161:0/1920837049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0380a9960 0x7ff0380bfc30 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7ff02800c3f0 tx=0x7ff02800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:51.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.752+0000 7ff02e7fc640 1 -- 172.21.15.161:0/1920837049 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff028017070 con 0x7ff0380a9960 2024-03-28T23:33:51.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.752+0000 7ff02e7fc640 1 -- 172.21.15.161:0/1920837049 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0280052f0 con 0x7ff0380a9960 2024-03-28T23:33:51.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.752+0000 7ff03ff5e640 1 -- 172.21.15.161:0/1920837049 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0381ee070 con 0x7ff0380a9960 2024-03-28T23:33:51.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.752+0000 7ff02e7fc640 1 -- 172.21.15.161:0/1920837049 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff028003830 con 0x7ff0380a9960 2024-03-28T23:33:51.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.752+0000 7ff03ff5e640 1 -- 172.21.15.161:0/1920837049 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0381ee630 con 0x7ff0380a9960 2024-03-28T23:33:51.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.753+0000 7ff03ff5e640 1 -- 172.21.15.161:0/1920837049 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0381ee940 con 0x7ff0380a9960 2024-03-28T23:33:51.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.754+0000 7ff02e7fc640 1 -- 172.21.15.161:0/1920837049 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff02801a070 con 0x7ff0380a9960 2024-03-28T23:33:51.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.754+0000 7ff02e7fc640 1 --2- 172.21.15.161:0/1920837049 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff01c06b500 0x7ff01c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:51.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.754+0000 7ff02e7fc640 1 -- 172.21.15.161:0/1920837049 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 6101+0+0 (secure 0 0 0) 0x7ff0280895d0 con 0x7ff0380a9960 2024-03-28T23:33:51.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.754+0000 7ff03dcfa640 1 --2- 172.21.15.161:0/1920837049 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff01c06b500 0x7ff01c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:51.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.755+0000 7ff03dcfa640 1 --2- 172.21.15.161:0/1920837049 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff01c06b500 0x7ff01c06d9c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ff034005880 tx=0x7ff0340057f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:51.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.759+0000 7ff02e7fc640 1 -- 172.21.15.161:0/1920837049 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff028057d00 con 0x7ff0380a9960 2024-03-28T23:33:51.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:51.968+0000 7ff03ff5e640 1 -- 172.21.15.161:0/1920837049 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7ff03806b1b0 con 0x7ff0380a9960 2024-03-28T23:33:52.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.009+0000 7ff02e7fc640 1 -- 172.21.15.161:0/1920837049 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v89) ==== 118+0+0 (secure 0 0 0) 0x7ff02805bda0 con 0x7ff0380a9960 2024-03-28T23:33:52.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.011+0000 7ff03ff5e640 1 -- 172.21.15.161:0/1920837049 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff01c06b500 msgr2=0x7ff01c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:52.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.011+0000 7ff03ff5e640 1 --2- 172.21.15.161:0/1920837049 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff01c06b500 0x7ff01c06d9c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ff034005880 tx=0x7ff0340057f0 comp rx=0 tx=0).stop 2024-03-28T23:33:52.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.011+0000 7ff03ff5e640 1 -- 172.21.15.161:0/1920837049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0380a9960 msgr2=0x7ff0380bfc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:52.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.011+0000 7ff03ff5e640 1 --2- 172.21.15.161:0/1920837049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0380a9960 0x7ff0380bfc30 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7ff02800c3f0 tx=0x7ff02800c8b0 comp rx=0 tx=0).stop 2024-03-28T23:33:52.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.011+0000 7ff03ff5e640 1 -- 172.21.15.161:0/1920837049 shutdown_connections 2024-03-28T23:33:52.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.011+0000 7ff03ff5e640 1 --2- 172.21.15.161:0/1920837049 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff01c06b500 0x7ff01c06d9c0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:52.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.011+0000 7ff03ff5e640 1 --2- 172.21.15.161:0/1920837049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0380a9960 0x7ff0380bfc30 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:52.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.011+0000 7ff03ff5e640 1 --2- 172.21.15.161:0/1920837049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0380c7330 0x7ff0380c0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:52.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.011+0000 7ff03ff5e640 1 --2- 172.21.15.161:0/1920837049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0380733e0 0x7ff0380c5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:52.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.011+0000 7ff03ff5e640 1 -- 172.21.15.161:0/1920837049 >> 172.21.15.161:0/1920837049 conn(0x7ff0380a83f0 msgr2=0x7ff038072740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:52.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.011+0000 7ff03ff5e640 1 -- 172.21.15.161:0/1920837049 shutdown_connections 2024-03-28T23:33:52.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.011+0000 7ff03ff5e640 1 -- 172.21.15.161:0/1920837049 wait complete. 2024-03-28T23:33:52.023 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-28T23:33:52.023 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-28T23:33:52.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.127+0000 7f7448a40640 1 Processor -- start 2024-03-28T23:33:52.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.127+0000 7f7448a40640 1 -- start start 2024-03-28T23:33:52.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.128+0000 7f7448a40640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7444152bc0 0x7f74440c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:52.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.128+0000 7f7448a40640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74440c8a80 0x7f74440c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:52.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.128+0000 7f7448a40640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f74440c9420 0x7f74441589b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:52.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.128+0000 7f7448a40640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f74440c6f60 con 0x7f74440c9420 2024-03-28T23:33:52.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.128+0000 7f7448a40640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f74440c6120 con 0x7f7444152bc0 2024-03-28T23:33:52.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.128+0000 7f7448a40640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f74440c6400 con 0x7f74440c8a80 2024-03-28T23:33:52.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.128+0000 7f7442d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f74440c9420 0x7f74441589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:52.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.128+0000 7f7442d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f74440c9420 0x7f74441589b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60016/0 (socket says 172.21.15.161:60016) 2024-03-28T23:33:52.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.128+0000 7f7442d9d640 1 -- 172.21.15.161:0/4218928645 learned_addr learned my addr 172.21.15.161:0/4218928645 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:52.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.129+0000 7f7441d9b640 1 --2- 172.21.15.161:0/4218928645 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74440c8a80 0x7f74440c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:52.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.129+0000 7f744259c640 1 --2- 172.21.15.161:0/4218928645 >> v2:172.21.15.170:3300/0 conn(0x7f7444152bc0 0x7f74440c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:52.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.129+0000 7f7442d9d640 1 -- 172.21.15.161:0/4218928645 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74440c8a80 msgr2=0x7f74440c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:52.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.129+0000 7f7442d9d640 1 --2- 172.21.15.161:0/4218928645 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74440c8a80 0x7f74440c8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:52.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.129+0000 7f7442d9d640 1 -- 172.21.15.161:0/4218928645 >> v2:172.21.15.170:3300/0 conn(0x7f7444152bc0 msgr2=0x7f74440c0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:52.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.129+0000 7f7442d9d640 1 --2- 172.21.15.161:0/4218928645 >> v2:172.21.15.170:3300/0 conn(0x7f7444152bc0 0x7f74440c0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:52.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.129+0000 7f7442d9d640 1 -- 172.21.15.161:0/4218928645 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74440c7240 con 0x7f74440c9420 2024-03-28T23:33:52.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.129+0000 7f744259c640 1 --2- 172.21.15.161:0/4218928645 >> v2:172.21.15.170:3300/0 conn(0x7f7444152bc0 0x7f74440c0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:52.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.129+0000 7f7441d9b640 1 --2- 172.21.15.161:0/4218928645 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74440c8a80 0x7f74440c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:52.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.129+0000 7f7442d9d640 1 --2- 172.21.15.161:0/4218928645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74440c9420 0x7f74441589b0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f74440aa410 tx=0x7f743402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c02ed55184dc6ca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:52.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.130+0000 7f744159a640 1 -- 172.21.15.161:0/4218928645 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743402ca70 con 0x7f74440c9420 2024-03-28T23:33:52.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.130+0000 7f744159a640 1 -- 172.21.15.161:0/4218928645 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f743402cc10 con 0x7f74440c9420 2024-03-28T23:33:52.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.130+0000 7f744159a640 1 -- 172.21.15.161:0/4218928645 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74340343f0 con 0x7f74440c9420 2024-03-28T23:33:52.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.130+0000 7f7448a40640 1 -- 172.21.15.161:0/4218928645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74440c9420 msgr2=0x7f74441589b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:52.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.130+0000 7f7448a40640 1 --2- 172.21.15.161:0/4218928645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74440c9420 0x7f74441589b0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f74440aa410 tx=0x7f743402bed0 comp rx=0 tx=0).stop 2024-03-28T23:33:52.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.130+0000 7f7448a40640 1 -- 172.21.15.161:0/4218928645 shutdown_connections 2024-03-28T23:33:52.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.130+0000 7f7448a40640 1 --2- 172.21.15.161:0/4218928645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74440c9420 0x7f74441589b0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:52.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.130+0000 7f7448a40640 1 --2- 172.21.15.161:0/4218928645 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74440c8a80 0x7f74440c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:52.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.130+0000 7f7448a40640 1 --2- 172.21.15.161:0/4218928645 >> v2:172.21.15.170:3300/0 conn(0x7f7444152bc0 0x7f74440c0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:52.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.130+0000 7f7448a40640 1 -- 172.21.15.161:0/4218928645 >> 172.21.15.161:0/4218928645 conn(0x7f74440caa80 msgr2=0x7f74440cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:52.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.130+0000 7f7448a40640 1 -- 172.21.15.161:0/4218928645 shutdown_connections 2024-03-28T23:33:52.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.130+0000 7f7448a40640 1 -- 172.21.15.161:0/4218928645 wait complete. 2024-03-28T23:33:52.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.132+0000 7f7448a40640 1 Processor -- start 2024-03-28T23:33:52.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.132+0000 7f7448a40640 1 -- start start 2024-03-28T23:33:52.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.133+0000 7f7448a40640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74440c8a80 0x7f744406c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:52.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.133+0000 7f7448a40640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74440c9420 0x7f74441450a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:52.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.133+0000 7f7448a40640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7444152bc0 0x7f74441455e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:52.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.133+0000 7f7448a40640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7444152fa0 con 0x7f74440c8a80 2024-03-28T23:33:52.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.133+0000 7f7448a40640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7444158ef0 con 0x7f74440c9420 2024-03-28T23:33:52.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.133+0000 7f7448a40640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7444156b30 con 0x7f7444152bc0 2024-03-28T23:33:52.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.133+0000 7f744259c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74440c8a80 0x7f744406c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:52.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.133+0000 7f744259c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74440c8a80 0x7f744406c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60024/0 (socket says 172.21.15.161:60024) 2024-03-28T23:33:52.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.133+0000 7f744259c640 1 -- 172.21.15.161:0/3938147233 learned_addr learned my addr 172.21.15.161:0/3938147233 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:52.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.133+0000 7f7442d9d640 1 --2- 172.21.15.161:0/3938147233 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7444152bc0 0x7f74441455e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:52.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.133+0000 7f7441d9b640 1 --2- 172.21.15.161:0/3938147233 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74440c9420 0x7f74441450a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:52.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.133+0000 7f744259c640 1 -- 172.21.15.161:0/3938147233 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7444152bc0 msgr2=0x7f74441455e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:52.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.133+0000 7f744259c640 1 --2- 172.21.15.161:0/3938147233 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7444152bc0 0x7f74441455e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:52.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.133+0000 7f744259c640 1 -- 172.21.15.161:0/3938147233 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74440c9420 msgr2=0x7f74441450a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:52.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.133+0000 7f744259c640 1 --2- 172.21.15.161:0/3938147233 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74440c9420 0x7f74441450a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:52.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.133+0000 7f744259c640 1 -- 172.21.15.161:0/3938147233 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7444145ea0 con 0x7f74440c8a80 2024-03-28T23:33:52.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.133+0000 7f744259c640 1 --2- 172.21.15.161:0/3938147233 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74440c8a80 0x7f744406c680 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f743c00c7f0 tx=0x7f743c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:52.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.133+0000 7f743affd640 1 -- 172.21.15.161:0/3938147233 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c015070 con 0x7f74440c8a80 2024-03-28T23:33:52.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.134+0000 7f743affd640 1 -- 172.21.15.161:0/3938147233 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f743c007820 con 0x7f74440c8a80 2024-03-28T23:33:52.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.134+0000 7f743affd640 1 -- 172.21.15.161:0/3938147233 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c007bd0 con 0x7f74440c8a80 2024-03-28T23:33:52.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.134+0000 7f7448a40640 1 -- 172.21.15.161:0/3938147233 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7444145bc0 con 0x7f74440c8a80 2024-03-28T23:33:52.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.134+0000 7f7448a40640 1 -- 172.21.15.161:0/3938147233 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74440ada50 con 0x7f74440c8a80 2024-03-28T23:33:52.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.135+0000 7f7448a40640 1 -- 172.21.15.161:0/3938147233 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f744404f4b0 con 0x7f74440c8a80 2024-03-28T23:33:52.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.135+0000 7f743affd640 1 -- 172.21.15.161:0/3938147233 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f743c018070 con 0x7f74440c8a80 2024-03-28T23:33:52.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.135+0000 7f743affd640 1 --2- 172.21.15.161:0/3938147233 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f742406b4d0 0x7f742406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:52.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.136+0000 7f743affd640 1 -- 172.21.15.161:0/3938147233 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6101+0+0 (secure 0 0 0) 0x7f743c08a4c0 con 0x7f74440c8a80 2024-03-28T23:33:52.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.136+0000 7f7441d9b640 1 --2- 172.21.15.161:0/3938147233 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f742406b4d0 0x7f742406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:52.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.136+0000 7f7441d9b640 1 --2- 172.21.15.161:0/3938147233 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f742406b4d0 0x7f742406d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f744414cfa0 tx=0x7f742c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:52.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.140+0000 7f743affd640 1 -- 172.21.15.161:0/3938147233 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f743c0591b0 con 0x7f74440c8a80 2024-03-28T23:33:52.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:52.353+0000 7f7448a40640 1 -- 172.21.15.161:0/3938147233 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f7444150190 con 0x7f74440c8a80 2024-03-28T23:33:53.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.010+0000 7f743affd640 1 -- 172.21.15.161:0/3938147233 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v90) ==== 122+0+0 (secure 0 0 0) 0x7f743c058ab0 con 0x7f74440c8a80 2024-03-28T23:33:53.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.012+0000 7f7448a40640 1 -- 172.21.15.161:0/3938147233 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f742406b4d0 msgr2=0x7f742406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:53.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.012+0000 7f7448a40640 1 --2- 172.21.15.161:0/3938147233 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f742406b4d0 0x7f742406d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f744414cfa0 tx=0x7f742c008040 comp rx=0 tx=0).stop 2024-03-28T23:33:53.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.012+0000 7f7448a40640 1 -- 172.21.15.161:0/3938147233 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74440c8a80 msgr2=0x7f744406c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:53.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.012+0000 7f7448a40640 1 --2- 172.21.15.161:0/3938147233 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74440c8a80 0x7f744406c680 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f743c00c7f0 tx=0x7f743c00ccb0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.012+0000 7f7448a40640 1 -- 172.21.15.161:0/3938147233 shutdown_connections 2024-03-28T23:33:53.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.012+0000 7f7448a40640 1 --2- 172.21.15.161:0/3938147233 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f742406b4d0 0x7f742406d990 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.012+0000 7f7448a40640 1 --2- 172.21.15.161:0/3938147233 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7444152bc0 0x7f74441455e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.012+0000 7f7448a40640 1 --2- 172.21.15.161:0/3938147233 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74440c9420 0x7f74441450a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.012+0000 7f7448a40640 1 --2- 172.21.15.161:0/3938147233 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74440c8a80 0x7f744406c680 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.012+0000 7f7448a40640 1 -- 172.21.15.161:0/3938147233 >> 172.21.15.161:0/3938147233 conn(0x7f74440caa80 msgr2=0x7f74440c0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:53.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.012+0000 7f7448a40640 1 -- 172.21.15.161:0/3938147233 shutdown_connections 2024-03-28T23:33:53.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.012+0000 7f7448a40640 1 -- 172.21.15.161:0/3938147233 wait complete. 2024-03-28T23:33:53.023 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-28T23:33:53.024 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:33:53.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.125+0000 7f0400023640 1 Processor -- start 2024-03-28T23:33:53.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.126+0000 7f0400023640 1 -- start start 2024-03-28T23:33:53.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.126+0000 7f0400023640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03f8152890 0x7f03f80c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:53.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.126+0000 7f0400023640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f03f80c5b20 0x7f03f81553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:53.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.126+0000 7f0400023640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03f8158a30 con 0x7f03f80c8ed0 2024-03-28T23:33:53.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.126+0000 7f0400023640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f03f80c8480 con 0x7f03f80c5b20 2024-03-28T23:33:53.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.126+0000 7f0400023640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f03f8159640 con 0x7f03f8152890 2024-03-28T23:33:53.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fd5be640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f03f80c8ed0 0x7f03f80c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41458/0 (socket says 172.21.15.161:41458) 2024-03-28T23:33:53.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fd5be640 1 -- 172.21.15.161:0/979244635 learned_addr learned my addr 172.21.15.161:0/979244635 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:53.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fcdbd640 1 -- 172.21.15.161:0/979244635 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4028349575 0 0) 0x7f03f8158a30 con 0x7f03f80c8ed0 2024-03-28T23:33:53.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fcdbd640 1 -- 172.21.15.161:0/979244635 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03e4003520 con 0x7f03f80c8ed0 2024-03-28T23:33:53.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fddbf640 1 --2- 172.21.15.161:0/979244635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03f8152890 0x7f03f80c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:53.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fe5c0640 1 --2- 172.21.15.161:0/979244635 >> v2:172.21.15.170:3300/0 conn(0x7f03f80c5b20 0x7f03f81553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:53.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fcdbd640 1 -- 172.21.15.161:0/979244635 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4161587451 0 0) 0x7f03e4003520 con 0x7f03f80c8ed0 2024-03-28T23:33:53.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fcdbd640 1 -- 172.21.15.161:0/979244635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03f8152890 msgr2=0x7f03f80c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:53.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fcdbd640 1 --2- 172.21.15.161:0/979244635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03f8152890 0x7f03f80c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fcdbd640 1 -- 172.21.15.161:0/979244635 >> v2:172.21.15.170:3300/0 conn(0x7f03f80c5b20 msgr2=0x7f03f81553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:53.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fcdbd640 1 --2- 172.21.15.161:0/979244635 >> v2:172.21.15.170:3300/0 conn(0x7f03f80c5b20 0x7f03f81553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fcdbd640 1 -- 172.21.15.161:0/979244635 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f8158d10 con 0x7f03f80c8ed0 2024-03-28T23:33:53.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fcdbd640 1 -- 172.21.15.161:0/979244635 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f03e80017f0 con 0x7f03f80c8ed0 2024-03-28T23:33:53.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fe5c0640 1 --2- 172.21.15.161:0/979244635 >> v2:172.21.15.170:3300/0 conn(0x7f03f80c5b20 0x7f03f81553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:53.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fddbf640 1 --2- 172.21.15.161:0/979244635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03f8152890 0x7f03f80c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:53.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fcdbd640 1 -- 172.21.15.161:0/979244635 >> v1:172.21.15.161:6789/0 conn(0x7f03f80c8ed0 legacy=0x7f03f80c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:53.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fcdbd640 1 --2- 172.21.15.161:0/979244635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03e4003a90 0x7f03e4003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:53.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fcdbd640 1 --2- 172.21.15.161:0/979244635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03e4004800 0x7f03e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:53.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fcdbd640 1 --2- 172.21.15.161:0/979244635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03f8152890 0x7f03e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:53.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fcdbd640 1 -- 172.21.15.161:0/979244635 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f03f8159640 con 0x7f03e4003a90 2024-03-28T23:33:53.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fcdbd640 1 -- 172.21.15.161:0/979244635 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f03e4005500 con 0x7f03e4004800 2024-03-28T23:33:53.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fcdbd640 1 -- 172.21.15.161:0/979244635 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f03e400bf00 con 0x7f03f8152890 2024-03-28T23:33:53.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fddbf640 1 --2- 172.21.15.161:0/979244635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03e4003a90 0x7f03e4003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:53.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fd5be640 1 --2- 172.21.15.161:0/979244635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03e4004800 0x7f03e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:53.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.127+0000 7f03fe5c0640 1 --2- 172.21.15.161:0/979244635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03f8152890 0x7f03e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:53.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.128+0000 7f03fddbf640 1 -- 172.21.15.161:0/979244635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03f8152890 msgr2=0x7f03e400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:53.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.128+0000 7f03fddbf640 1 --2- 172.21.15.161:0/979244635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03f8152890 0x7f03e400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.128+0000 7f03fddbf640 1 -- 172.21.15.161:0/979244635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03e4004800 msgr2=0x7f03e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:53.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.128+0000 7f03fddbf640 1 --2- 172.21.15.161:0/979244635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03e4004800 0x7f03e4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.128+0000 7f03fddbf640 1 -- 172.21.15.161:0/979244635 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e4003520 con 0x7f03e4003a90 2024-03-28T23:33:53.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.128+0000 7f03fddbf640 1 --2- 172.21.15.161:0/979244635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03e4003a90 0x7f03e4003ed0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f03f40098a0 tx=0x7f03e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=379e175b79a74f4c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:53.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.128+0000 7f03fcdbd640 1 -- 172.21.15.161:0/979244635 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f400f040 con 0x7f03e4003a90 2024-03-28T23:33:53.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.128+0000 7f03fcdbd640 1 -- 172.21.15.161:0/979244635 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03f4004230 con 0x7f03e4003a90 2024-03-28T23:33:53.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.128+0000 7f03fcdbd640 1 -- 172.21.15.161:0/979244635 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f4003610 con 0x7f03e4003a90 2024-03-28T23:33:53.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.129+0000 7f0400023640 1 -- 172.21.15.161:0/979244635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03e4003a90 msgr2=0x7f03e4003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:53.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.129+0000 7f0400023640 1 --2- 172.21.15.161:0/979244635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03e4003a90 0x7f03e4003ed0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f03f40098a0 tx=0x7f03e4005830 comp rx=0 tx=0).stop 2024-03-28T23:33:53.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.129+0000 7f03fe5c0640 1 -- 172.21.15.161:0/979244635 reap_dead start 2024-03-28T23:33:53.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.129+0000 7f0400023640 1 -- 172.21.15.161:0/979244635 shutdown_connections 2024-03-28T23:33:53.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.129+0000 7f0400023640 1 -- 172.21.15.161:0/979244635 >> 172.21.15.161:0/979244635 conn(0x7f03f80ac620 msgr2=0x7f03f80aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:53.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.129+0000 7f0400023640 1 -- 172.21.15.161:0/979244635 shutdown_connections 2024-03-28T23:33:53.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.129+0000 7f0400023640 1 -- 172.21.15.161:0/979244635 wait complete. 2024-03-28T23:33:53.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.131+0000 7f0400023640 1 Processor -- start 2024-03-28T23:33:53.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.131+0000 7f0400023640 1 -- start start 2024-03-28T23:33:53.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.132+0000 7f0400023640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03e4003a90 0x7f03f81ee620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:53.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.132+0000 7f0400023640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03f8148fb0 0x7f03f8149430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:53.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.132+0000 7f0400023640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03f80c35c0 0x7f03f80c3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:53.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.132+0000 7f0400023640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f03f8152c70 con 0x7f03e4003a90 2024-03-28T23:33:53.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.132+0000 7f0400023640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f03f81592d0 con 0x7f03f8148fb0 2024-03-28T23:33:53.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.132+0000 7f0400023640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f03f8152df0 con 0x7f03f80c35c0 2024-03-28T23:33:53.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.132+0000 7f03fddbf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03e4003a90 0x7f03f81ee620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:53.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.132+0000 7f03fddbf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03e4003a90 0x7f03f81ee620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60046/0 (socket says 172.21.15.161:60046) 2024-03-28T23:33:53.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.132+0000 7f03fddbf640 1 -- 172.21.15.161:0/1294616705 learned_addr learned my addr 172.21.15.161:0/1294616705 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:53.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.132+0000 7f03fd5be640 1 --2- 172.21.15.161:0/1294616705 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03f8148fb0 0x7f03f8149430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:53.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.132+0000 7f03fe5c0640 1 --2- 172.21.15.161:0/1294616705 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03f80c35c0 0x7f03f80c3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:53.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.132+0000 7f03fddbf640 1 -- 172.21.15.161:0/1294616705 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03f80c35c0 msgr2=0x7f03f80c3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:53.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.132+0000 7f03fddbf640 1 --2- 172.21.15.161:0/1294616705 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03f80c35c0 0x7f03f80c3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.132+0000 7f03fddbf640 1 -- 172.21.15.161:0/1294616705 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03f8148fb0 msgr2=0x7f03f8149430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:53.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.132+0000 7f03fddbf640 1 --2- 172.21.15.161:0/1294616705 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03f8148fb0 0x7f03f8149430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.132+0000 7f03fddbf640 1 -- 172.21.15.161:0/1294616705 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f814a230 con 0x7f03e4003a90 2024-03-28T23:33:53.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.133+0000 7f03fddbf640 1 --2- 172.21.15.161:0/1294616705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03e4003a90 0x7f03f81ee620 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f03f402bf90 tx=0x7f03f4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:53.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.133+0000 7f03ee7fc640 1 -- 172.21.15.161:0/1294616705 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f4048070 con 0x7f03e4003a90 2024-03-28T23:33:53.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.133+0000 7f03ee7fc640 1 -- 172.21.15.161:0/1294616705 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03f4030430 con 0x7f03e4003a90 2024-03-28T23:33:53.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.133+0000 7f0400023640 1 -- 172.21.15.161:0/1294616705 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03f8149d90 con 0x7f03e4003a90 2024-03-28T23:33:53.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.133+0000 7f03ee7fc640 1 -- 172.21.15.161:0/1294616705 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f4030820 con 0x7f03e4003a90 2024-03-28T23:33:53.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.133+0000 7f0400023640 1 -- 172.21.15.161:0/1294616705 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03f81dea00 con 0x7f03e4003a90 2024-03-28T23:33:53.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.134+0000 7f0400023640 1 -- 172.21.15.161:0/1294616705 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03f814dc90 con 0x7f03e4003a90 2024-03-28T23:33:53.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.134+0000 7f03ee7fc640 1 -- 172.21.15.161:0/1294616705 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f03f404b070 con 0x7f03e4003a90 2024-03-28T23:33:53.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.135+0000 7f03ee7fc640 1 --2- 172.21.15.161:0/1294616705 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f03dc06b4d0 0x7f03dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:53.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.135+0000 7f03ee7fc640 1 -- 172.21.15.161:0/1294616705 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6101+0+0 (secure 0 0 0) 0x7f03f40abd80 con 0x7f03e4003a90 2024-03-28T23:33:53.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.135+0000 7f03fd5be640 1 --2- 172.21.15.161:0/1294616705 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f03dc06b4d0 0x7f03dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:53.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.135+0000 7f03fd5be640 1 --2- 172.21.15.161:0/1294616705 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f03dc06b4d0 0x7f03dc06d990 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f03f81509e0 tx=0x7f03e8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:53.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.139+0000 7f03ee7fc640 1 -- 172.21.15.161:0/1294616705 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f03f814dc90 con 0x7f03e4003a90 2024-03-28T23:33:53.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.351+0000 7f0400023640 1 -- 172.21.15.161:0/1294616705 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03f804f4b0 con 0x7f03e4003a90 2024-03-28T23:33:53.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.351+0000 7f03ee7fc640 1 -- 172.21.15.161:0/1294616705 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) ==== 94+0+5195 (secure 0 0 0) 0x7f03f4034020 con 0x7f03e4003a90 2024-03-28T23:33:53.352 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:53.352 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":90,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6203,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/1096504848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1096504848},{"type":"v1","addr":"172.21.15.170:6834","nonce":1096504848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":6206,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3351418222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3351418222},{"type":"v1","addr":"172.21.15.170:6835","nonce":3351418222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":6892,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/2684760895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2684760895},{"type":"v1","addr":"172.21.15.161:6837","nonce":2684760895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:33:50.326492+0000","modified":"2024-03-28T23:33:52.354643+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7452},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7452":{"gid":7452,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6835/2544838511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":2544838511},{"type":"v1","addr":"172.21.15.161:6835","nonce":2544838511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7452,"qdb_cluster":[7452]},"id":8}]} 2024-03-28T23:33:53.352 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 90 2024-03-28T23:33:53.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.353+0000 7f0400023640 1 -- 172.21.15.161:0/1294616705 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f03dc06b4d0 msgr2=0x7f03dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:53.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.353+0000 7f0400023640 1 --2- 172.21.15.161:0/1294616705 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f03dc06b4d0 0x7f03dc06d990 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f03f81509e0 tx=0x7f03e8009040 comp rx=0 tx=0).stop 2024-03-28T23:33:53.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.353+0000 7f0400023640 1 -- 172.21.15.161:0/1294616705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03e4003a90 msgr2=0x7f03f81ee620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:53.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.353+0000 7f0400023640 1 --2- 172.21.15.161:0/1294616705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03e4003a90 0x7f03f81ee620 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f03f402bf90 tx=0x7f03f4004730 comp rx=0 tx=0).stop 2024-03-28T23:33:53.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.353+0000 7f0400023640 1 -- 172.21.15.161:0/1294616705 shutdown_connections 2024-03-28T23:33:53.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.353+0000 7f0400023640 1 --2- 172.21.15.161:0/1294616705 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f03dc06b4d0 0x7f03dc06d990 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.353+0000 7f0400023640 1 --2- 172.21.15.161:0/1294616705 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03f80c35c0 0x7f03f80c3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.353+0000 7f0400023640 1 --2- 172.21.15.161:0/1294616705 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03f8148fb0 0x7f03f8149430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.353+0000 7f0400023640 1 --2- 172.21.15.161:0/1294616705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03e4003a90 0x7f03f81ee620 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.353+0000 7f0400023640 1 -- 172.21.15.161:0/1294616705 >> 172.21.15.161:0/1294616705 conn(0x7f03f80ac620 msgr2=0x7f03f806cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:53.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.353+0000 7f0400023640 1 -- 172.21.15.161:0/1294616705 shutdown_connections 2024-03-28T23:33:53.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.353+0000 7f0400023640 1 -- 172.21.15.161:0/1294616705 wait complete. 2024-03-28T23:33:53.365 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:33:53.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.466+0000 7f17139e9640 1 Processor -- start 2024-03-28T23:33:53.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.466+0000 7f17139e9640 1 -- start start 2024-03-28T23:33:53.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.467+0000 7f17139e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f170c14be70 0x7f170c0ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:53.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.467+0000 7f17139e9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f170c0ab6e0 0x7f170c0aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:53.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.467+0000 7f17139e9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f170c14c250 con 0x7f170c06ce80 2024-03-28T23:33:53.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.467+0000 7f17139e9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f170c1529c0 con 0x7f170c0ab6e0 2024-03-28T23:33:53.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.467+0000 7f17139e9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f170c0a9dc0 con 0x7f170c14be70 2024-03-28T23:33:53.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.467+0000 7f1711f86640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f170c06ce80 0x7f170c1553b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41464/0 (socket says 172.21.15.161:41464) 2024-03-28T23:33:53.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.467+0000 7f1711f86640 1 -- 172.21.15.161:0/2851112169 learned_addr learned my addr 172.21.15.161:0/2851112169 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:53.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.468+0000 7f1711785640 1 --2- 172.21.15.161:0/2851112169 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f170c14be70 0x7f170c0ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:53.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.468+0000 7f1710f84640 1 --2- 172.21.15.161:0/2851112169 >> v2:172.21.15.170:3300/0 conn(0x7f170c0ab6e0 0x7f170c0aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:53.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.468+0000 7f1703fff640 1 -- 172.21.15.161:0/2851112169 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 465722221 0 0) 0x7f170c14c250 con 0x7f170c06ce80 2024-03-28T23:33:53.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.468+0000 7f1703fff640 1 -- 172.21.15.161:0/2851112169 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16f8005730 con 0x7f170c06ce80 2024-03-28T23:33:53.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.468+0000 7f1703fff640 1 -- 172.21.15.161:0/2851112169 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3636866180 0 0) 0x7f16f8005730 con 0x7f170c06ce80 2024-03-28T23:33:53.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.468+0000 7f1703fff640 1 -- 172.21.15.161:0/2851112169 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f170c14be70 msgr2=0x7f170c0ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:53.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.468+0000 7f1703fff640 1 --2- 172.21.15.161:0/2851112169 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f170c14be70 0x7f170c0ab1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.468+0000 7f1703fff640 1 -- 172.21.15.161:0/2851112169 >> v2:172.21.15.170:3300/0 conn(0x7f170c0ab6e0 msgr2=0x7f170c0aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:53.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.468+0000 7f1703fff640 1 --2- 172.21.15.161:0/2851112169 >> v2:172.21.15.170:3300/0 conn(0x7f170c0ab6e0 0x7f170c0aa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.468+0000 7f1703fff640 1 -- 172.21.15.161:0/2851112169 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f170c0aa0a0 con 0x7f170c06ce80 2024-03-28T23:33:53.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.468+0000 7f1703fff640 1 -- 172.21.15.161:0/2851112169 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f17040019f0 con 0x7f170c06ce80 2024-03-28T23:33:53.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.468+0000 7f1703fff640 1 -- 172.21.15.161:0/2851112169 >> v1:172.21.15.161:6789/0 conn(0x7f170c06ce80 legacy=0x7f170c1553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:53.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.468+0000 7f1703fff640 1 --2- 172.21.15.161:0/2851112169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16f8005ca0 0x7f16f80060c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:53.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.468+0000 7f1703fff640 1 --2- 172.21.15.161:0/2851112169 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f170c14be70 0x7f16f8006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:53.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.468+0000 7f1703fff640 1 --2- 172.21.15.161:0/2851112169 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16f8006f50 0x7f16f80073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:53.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.468+0000 7f1703fff640 1 -- 172.21.15.161:0/2851112169 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f170c1529c0 con 0x7f16f8005ca0 2024-03-28T23:33:53.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.468+0000 7f1703fff640 1 -- 172.21.15.161:0/2851112169 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f170c0a9dc0 con 0x7f16f8006f50 2024-03-28T23:33:53.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.468+0000 7f1703fff640 1 -- 172.21.15.161:0/2851112169 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f16f800dca0 con 0x7f170c14be70 2024-03-28T23:33:53.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.468+0000 7f1711785640 1 --2- 172.21.15.161:0/2851112169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16f8005ca0 0x7f16f80060c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:53.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.468+0000 7f1710f84640 1 --2- 172.21.15.161:0/2851112169 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f170c14be70 0x7f16f8006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:53.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.468+0000 7f1711f86640 1 --2- 172.21.15.161:0/2851112169 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16f8006f50 0x7f16f80073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:53.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.469+0000 7f1711785640 1 -- 172.21.15.161:0/2851112169 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f170c14be70 msgr2=0x7f16f8006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:53.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.469+0000 7f1711785640 1 --2- 172.21.15.161:0/2851112169 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f170c14be70 0x7f16f8006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.469+0000 7f1711785640 1 -- 172.21.15.161:0/2851112169 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16f8006f50 msgr2=0x7f16f80073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:53.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.469+0000 7f1711785640 1 --2- 172.21.15.161:0/2851112169 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16f8006f50 0x7f16f80073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.469+0000 7f1711785640 1 -- 172.21.15.161:0/2851112169 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f8005730 con 0x7f16f8005ca0 2024-03-28T23:33:53.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.469+0000 7f1710f84640 1 --2- 172.21.15.161:0/2851112169 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f170c14be70 0x7f16f8006a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:53.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.469+0000 7f1711785640 1 --2- 172.21.15.161:0/2851112169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16f8005ca0 0x7f16f80060c0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f16fc009ac0 tx=0x7f16f8007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=582c15898bc4f749 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:53.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.469+0000 7f1703fff640 1 -- 172.21.15.161:0/2851112169 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc039070 con 0x7f16f8005ca0 2024-03-28T23:33:53.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.469+0000 7f1703fff640 1 -- 172.21.15.161:0/2851112169 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16fc02c600 con 0x7f16f8005ca0 2024-03-28T23:33:53.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.469+0000 7f1703fff640 1 -- 172.21.15.161:0/2851112169 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc009690 con 0x7f16f8005ca0 2024-03-28T23:33:53.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.470+0000 7f17139e9640 1 -- 172.21.15.161:0/2851112169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16f8005ca0 msgr2=0x7f16f80060c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:53.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.470+0000 7f17139e9640 1 --2- 172.21.15.161:0/2851112169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16f8005ca0 0x7f16f80060c0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f16fc009ac0 tx=0x7f16f8007a40 comp rx=0 tx=0).stop 2024-03-28T23:33:53.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.470+0000 7f1711f86640 1 -- 172.21.15.161:0/2851112169 reap_dead start 2024-03-28T23:33:53.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.470+0000 7f17139e9640 1 -- 172.21.15.161:0/2851112169 shutdown_connections 2024-03-28T23:33:53.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.470+0000 7f17139e9640 1 -- 172.21.15.161:0/2851112169 >> 172.21.15.161:0/2851112169 conn(0x7f170c0c6670 msgr2=0x7f170c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:53.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.470+0000 7f17139e9640 1 -- 172.21.15.161:0/2851112169 shutdown_connections 2024-03-28T23:33:53.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.470+0000 7f17139e9640 1 -- 172.21.15.161:0/2851112169 wait complete. 2024-03-28T23:33:53.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.472+0000 7f17139e9640 1 Processor -- start 2024-03-28T23:33:53.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.472+0000 7f17139e9640 1 -- start start 2024-03-28T23:33:53.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.473+0000 7f17139e9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16f8005ca0 0x7f170c0c23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:53.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.473+0000 7f17139e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f170c155e20 0x7f170c1e98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:53.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.473+0000 7f17139e9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f170c1e6db0 0x7f170c1e9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:53.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.473+0000 7f17139e9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f170c06b2d0 con 0x7f16f8005ca0 2024-03-28T23:33:53.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.473+0000 7f17139e9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f170c06b150 con 0x7f170c1e6db0 2024-03-28T23:33:53.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.473+0000 7f17139e9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f170c0c1520 con 0x7f170c155e20 2024-03-28T23:33:53.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.473+0000 7f1711785640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16f8005ca0 0x7f170c0c23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:53.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.473+0000 7f1711785640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16f8005ca0 0x7f170c0c23e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60058/0 (socket says 172.21.15.161:60058) 2024-03-28T23:33:53.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.473+0000 7f1711785640 1 -- 172.21.15.161:0/57085374 learned_addr learned my addr 172.21.15.161:0/57085374 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:53.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.473+0000 7f1710f84640 1 --2- 172.21.15.161:0/57085374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f170c155e20 0x7f170c1e98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:53.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.473+0000 7f1711f86640 1 --2- 172.21.15.161:0/57085374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f170c1e6db0 0x7f170c1e9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:53.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.474+0000 7f1711785640 1 -- 172.21.15.161:0/57085374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f170c155e20 msgr2=0x7f170c1e98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:53.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.474+0000 7f1711785640 1 --2- 172.21.15.161:0/57085374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f170c155e20 0x7f170c1e98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.474+0000 7f1711785640 1 -- 172.21.15.161:0/57085374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f170c1e6db0 msgr2=0x7f170c1e9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:53.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.474+0000 7f1711785640 1 --2- 172.21.15.161:0/57085374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f170c1e6db0 0x7f170c1e9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.474+0000 7f1711785640 1 -- 172.21.15.161:0/57085374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f170c0c2920 con 0x7f16f8005ca0 2024-03-28T23:33:53.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.474+0000 7f1711f86640 1 --2- 172.21.15.161:0/57085374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f170c1e6db0 0x7f170c1e9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:53.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.474+0000 7f1710f84640 1 --2- 172.21.15.161:0/57085374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f170c155e20 0x7f170c1e98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:53.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.474+0000 7f1711785640 1 --2- 172.21.15.161:0/57085374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16f8005ca0 0x7f170c0c23e0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f16fc009b20 tx=0x7f16fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:53.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.474+0000 7f1701ffb640 1 -- 172.21.15.161:0/57085374 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc039050 con 0x7f16f8005ca0 2024-03-28T23:33:53.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.474+0000 7f1701ffb640 1 -- 172.21.15.161:0/57085374 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16fc045070 con 0x7f16f8005ca0 2024-03-28T23:33:53.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.474+0000 7f1701ffb640 1 -- 172.21.15.161:0/57085374 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc003850 con 0x7f16f8005ca0 2024-03-28T23:33:53.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.474+0000 7f17139e9640 1 -- 172.21.15.161:0/57085374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f170c1ea320 con 0x7f16f8005ca0 2024-03-28T23:33:53.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.474+0000 7f17139e9640 1 -- 172.21.15.161:0/57085374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f170c14f4f0 con 0x7f16f8005ca0 2024-03-28T23:33:53.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.476+0000 7f1701ffb640 1 -- 172.21.15.161:0/57085374 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f16fc0039f0 con 0x7f16f8005ca0 2024-03-28T23:33:53.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.476+0000 7f17139e9640 1 -- 172.21.15.161:0/57085374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f170c04f4b0 con 0x7f16f8005ca0 2024-03-28T23:33:53.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.476+0000 7f1701ffb640 1 --2- 172.21.15.161:0/57085374 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f16f006b500 0x7f16f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:53.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.476+0000 7f1701ffb640 1 -- 172.21.15.161:0/57085374 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6101+0+0 (secure 0 0 0) 0x7f16fc0abed0 con 0x7f16f8005ca0 2024-03-28T23:33:53.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.476+0000 7f1710f84640 1 --2- 172.21.15.161:0/57085374 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f16f006b500 0x7f16f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:53.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.476+0000 7f1710f84640 1 --2- 172.21.15.161:0/57085374 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f16f006b500 0x7f16f006d9c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f1708006d40 tx=0x7f1708008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:53.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.480+0000 7f1701ffb640 1 -- 172.21.15.161:0/57085374 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f16fc04e020 con 0x7f16f8005ca0 2024-03-28T23:33:53.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.794+0000 7f17139e9640 1 -- 172.21.15.161:0/57085374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f170c0c0ff0 con 0x7f16f8005ca0 2024-03-28T23:33:53.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.795+0000 7f1701ffb640 1 -- 172.21.15.161:0/57085374 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) ==== 74+0+16988 (secure 0 0 0) 0x7f16fc03d360 con 0x7f16f8005ca0 2024-03-28T23:33:53.796 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:53.796 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":109,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:33:51.997581+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:33:48.560815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-28T23:33:49.301625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:33:53.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.797+0000 7f17139e9640 1 -- 172.21.15.161:0/57085374 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f16f006b500 msgr2=0x7f16f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:53.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.797+0000 7f17139e9640 1 --2- 172.21.15.161:0/57085374 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f16f006b500 0x7f16f006d9c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f1708006d40 tx=0x7f1708008040 comp rx=0 tx=0).stop 2024-03-28T23:33:53.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.797+0000 7f17139e9640 1 -- 172.21.15.161:0/57085374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16f8005ca0 msgr2=0x7f170c0c23e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:53.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.797+0000 7f17139e9640 1 --2- 172.21.15.161:0/57085374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16f8005ca0 0x7f170c0c23e0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f16fc009b20 tx=0x7f16fc004730 comp rx=0 tx=0).stop 2024-03-28T23:33:53.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.798+0000 7f17139e9640 1 -- 172.21.15.161:0/57085374 shutdown_connections 2024-03-28T23:33:53.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.798+0000 7f17139e9640 1 --2- 172.21.15.161:0/57085374 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f16f006b500 0x7f16f006d9c0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.798+0000 7f17139e9640 1 --2- 172.21.15.161:0/57085374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f170c1e6db0 0x7f170c1e9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.798+0000 7f17139e9640 1 --2- 172.21.15.161:0/57085374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f170c155e20 0x7f170c1e98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.798+0000 7f17139e9640 1 --2- 172.21.15.161:0/57085374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16f8005ca0 0x7f170c0c23e0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.798+0000 7f17139e9640 1 -- 172.21.15.161:0/57085374 >> 172.21.15.161:0/57085374 conn(0x7f170c0c6670 msgr2=0x7f170c072230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:53.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.798+0000 7f17139e9640 1 -- 172.21.15.161:0/57085374 shutdown_connections 2024-03-28T23:33:53.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.798+0000 7f17139e9640 1 -- 172.21.15.161:0/57085374 wait complete. 2024-03-28T23:33:53.809 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T23:33:53.810 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:33:53.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.911+0000 7fc4e258a640 1 Processor -- start 2024-03-28T23:33:53.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.911+0000 7fc4e258a640 1 -- start start 2024-03-28T23:33:53.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.912+0000 7fc4e258a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc4dc152890 0x7fc4dc0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:53.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.912+0000 7fc4e258a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc4dc0c8ed0 0x7fc4dc0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:53.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.912+0000 7fc4e258a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4dc158a30 con 0x7fc4dc0c5950 2024-03-28T23:33:53.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.912+0000 7fc4e258a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc4dc0c8480 con 0x7fc4dc0c8ed0 2024-03-28T23:33:53.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.912+0000 7fc4e258a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fc4dc159640 con 0x7fc4dc152890 2024-03-28T23:33:53.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.912+0000 7fc4db7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc4dc0c8ed0 0x7fc4dc0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:53.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.912+0000 7fc4db7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc4dc0c8ed0 0x7fc4dc0c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60074/0 (socket says 172.21.15.161:60074) 2024-03-28T23:33:53.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.912+0000 7fc4db7fe640 1 -- 172.21.15.161:0/56198330 learned_addr learned my addr 172.21.15.161:0/56198330 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:53.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.912+0000 7fc4dbfff640 1 --2- 172.21.15.161:0/56198330 >> v2:172.21.15.170:3300/0 conn(0x7fc4dc152890 0x7fc4dc0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:53.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.912+0000 7fc4db7fe640 1 -- 172.21.15.161:0/56198330 >> v2:172.21.15.170:3300/0 conn(0x7fc4dc152890 msgr2=0x7fc4dc0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:53.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.912+0000 7fc4db7fe640 1 --2- 172.21.15.161:0/56198330 >> v2:172.21.15.170:3300/0 conn(0x7fc4dc152890 0x7fc4dc0c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.912+0000 7fc4db7fe640 1 -- 172.21.15.161:0/56198330 >> v1:172.21.15.170:6789/0 conn(0x7fc4dc0c5950 legacy=0x7fc4dc1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:53.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.912+0000 7fc4db7fe640 1 -- 172.21.15.161:0/56198330 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4dc158d10 con 0x7fc4dc0c8ed0 2024-03-28T23:33:53.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.913+0000 7fc4db7fe640 1 --2- 172.21.15.161:0/56198330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4dc0c8ed0 0x7fc4dc0c5410 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc08cc80 tx=0x7fc4d002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb99046b67820d60 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:53.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.913+0000 7fc4daffd640 1 -- 172.21.15.161:0/56198330 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d0038070 con 0x7fc4dc0c8ed0 2024-03-28T23:33:53.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.913+0000 7fc4daffd640 1 -- 172.21.15.161:0/56198330 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4d0004230 con 0x7fc4dc0c8ed0 2024-03-28T23:33:53.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.913+0000 7fc4daffd640 1 -- 172.21.15.161:0/56198330 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d002f050 con 0x7fc4dc0c8ed0 2024-03-28T23:33:53.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.913+0000 7fc4e258a640 1 -- 172.21.15.161:0/56198330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4dc0c8ed0 msgr2=0x7fc4dc0c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:53.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.913+0000 7fc4e258a640 1 --2- 172.21.15.161:0/56198330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4dc0c8ed0 0x7fc4dc0c5410 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc08cc80 tx=0x7fc4d002b7e0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.913+0000 7fc4e258a640 1 -- 172.21.15.161:0/56198330 shutdown_connections 2024-03-28T23:33:53.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.914+0000 7fc4e258a640 1 --2- 172.21.15.161:0/56198330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4dc0c8ed0 0x7fc4dc0c5410 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.914+0000 7fc4e258a640 1 --2- 172.21.15.161:0/56198330 >> v2:172.21.15.170:3300/0 conn(0x7fc4dc152890 0x7fc4dc0c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.914+0000 7fc4e258a640 1 -- 172.21.15.161:0/56198330 >> 172.21.15.161:0/56198330 conn(0x7fc4dc0ac620 msgr2=0x7fc4dc0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:53.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.914+0000 7fc4e258a640 1 -- 172.21.15.161:0/56198330 shutdown_connections 2024-03-28T23:33:53.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.914+0000 7fc4e258a640 1 -- 172.21.15.161:0/56198330 wait complete. 2024-03-28T23:33:53.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.915+0000 7fc4e258a640 1 Processor -- start 2024-03-28T23:33:53.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.915+0000 7fc4e258a640 1 -- start start 2024-03-28T23:33:53.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.915+0000 7fc4e258a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4dc0c5950 0x7fc4dc0c1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:53.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.915+0000 7fc4e258a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4dc0c8ed0 0x7fc4dc1ee590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:53.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.915+0000 7fc4e258a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4dc152890 0x7fc4dc155650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:53.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.915+0000 7fc4e258a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc4dc152c70 con 0x7fc4dc152890 2024-03-28T23:33:53.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.915+0000 7fc4e258a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc4dc1592d0 con 0x7fc4dc0c8ed0 2024-03-28T23:33:53.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.915+0000 7fc4e258a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc4dc152df0 con 0x7fc4dc0c5950 2024-03-28T23:33:53.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.916+0000 7fc4e0b27640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4dc152890 0x7fc4dc155650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:53.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.916+0000 7fc4e0b27640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4dc152890 0x7fc4dc155650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60076/0 (socket says 172.21.15.161:60076) 2024-03-28T23:33:53.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.916+0000 7fc4e0b27640 1 -- 172.21.15.161:0/3201613539 learned_addr learned my addr 172.21.15.161:0/3201613539 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:53.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.916+0000 7fc4db7fe640 1 --2- 172.21.15.161:0/3201613539 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4dc0c8ed0 0x7fc4dc1ee590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:53.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.916+0000 7fc4dbfff640 1 --2- 172.21.15.161:0/3201613539 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4dc0c5950 0x7fc4dc0c1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:53.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.916+0000 7fc4e0b27640 1 -- 172.21.15.161:0/3201613539 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4dc0c5950 msgr2=0x7fc4dc0c1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:53.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.916+0000 7fc4e0b27640 1 --2- 172.21.15.161:0/3201613539 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4dc0c5950 0x7fc4dc0c1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.916+0000 7fc4e0b27640 1 -- 172.21.15.161:0/3201613539 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4dc0c8ed0 msgr2=0x7fc4dc1ee590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:53.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.916+0000 7fc4e0b27640 1 --2- 172.21.15.161:0/3201613539 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4dc0c8ed0 0x7fc4dc1ee590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:53.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.916+0000 7fc4e0b27640 1 -- 172.21.15.161:0/3201613539 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4dc153f00 con 0x7fc4dc152890 2024-03-28T23:33:53.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.916+0000 7fc4e0b27640 1 --2- 172.21.15.161:0/3201613539 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4dc152890 0x7fc4dc155650 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fc4d400b990 tx=0x7fc4d400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:53.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.916+0000 7fc4d8ff9640 1 -- 172.21.15.161:0/3201613539 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d4019070 con 0x7fc4dc152890 2024-03-28T23:33:53.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.916+0000 7fc4d8ff9640 1 -- 172.21.15.161:0/3201613539 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4d400cba0 con 0x7fc4dc152890 2024-03-28T23:33:53.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.916+0000 7fc4e258a640 1 -- 172.21.15.161:0/3201613539 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4dc1ef040 con 0x7fc4dc152890 2024-03-28T23:33:53.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.916+0000 7fc4d8ff9640 1 -- 172.21.15.161:0/3201613539 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d4014440 con 0x7fc4dc152890 2024-03-28T23:33:53.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.916+0000 7fc4e258a640 1 -- 172.21.15.161:0/3201613539 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4dc1539e0 con 0x7fc4dc152890 2024-03-28T23:33:53.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.917+0000 7fc4e258a640 1 -- 172.21.15.161:0/3201613539 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4dc04f4b0 con 0x7fc4dc152890 2024-03-28T23:33:53.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.918+0000 7fc4d8ff9640 1 -- 172.21.15.161:0/3201613539 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc4d401c070 con 0x7fc4dc152890 2024-03-28T23:33:53.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.918+0000 7fc4d8ff9640 1 --2- 172.21.15.161:0/3201613539 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc4c806b500 0x7fc4c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:53.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.918+0000 7fc4d8ff9640 1 -- 172.21.15.161:0/3201613539 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6101+0+0 (secure 0 0 0) 0x7fc4d408a120 con 0x7fc4dc152890 2024-03-28T23:33:53.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.918+0000 7fc4dbfff640 1 --2- 172.21.15.161:0/3201613539 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc4c806b500 0x7fc4c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:53.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.919+0000 7fc4dbfff640 1 --2- 172.21.15.161:0/3201613539 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc4c806b500 0x7fc4c806d9c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc00a970 tx=0x7fc4cc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:53.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:53.926+0000 7fc4d8ff9640 1 -- 172.21.15.161:0/3201613539 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc4d4058e10 con 0x7fc4dc152890 2024-03-28T23:33:54.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.101+0000 7fc4e258a640 1 -- 172.21.15.161:0/3201613539 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc4dc149fb0 con 0x7fc4c806b500 2024-03-28T23:33:54.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.107+0000 7fc4d8ff9640 1 -- 172.21.15.161:0/3201613539 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+240322 (secure 0 0 0) 0x7fc4dc149fb0 con 0x7fc4c806b500 2024-03-28T23:33:54.110 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:54.111 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:33:54.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.110+0000 7fc4e258a640 1 -- 172.21.15.161:0/3201613539 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc4c806b500 msgr2=0x7fc4c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:54.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.110+0000 7fc4e258a640 1 --2- 172.21.15.161:0/3201613539 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc4c806b500 0x7fc4c806d9c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc00a970 tx=0x7fc4cc008040 comp rx=0 tx=0).stop 2024-03-28T23:33:54.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.110+0000 7fc4e258a640 1 -- 172.21.15.161:0/3201613539 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4dc152890 msgr2=0x7fc4dc155650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:54.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.110+0000 7fc4e258a640 1 --2- 172.21.15.161:0/3201613539 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4dc152890 0x7fc4dc155650 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fc4d400b990 tx=0x7fc4d400be50 comp rx=0 tx=0).stop 2024-03-28T23:33:54.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.110+0000 7fc4e258a640 1 -- 172.21.15.161:0/3201613539 shutdown_connections 2024-03-28T23:33:54.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.110+0000 7fc4e258a640 1 --2- 172.21.15.161:0/3201613539 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc4c806b500 0x7fc4c806d9c0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.110+0000 7fc4e258a640 1 --2- 172.21.15.161:0/3201613539 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4dc152890 0x7fc4dc155650 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.110+0000 7fc4e258a640 1 --2- 172.21.15.161:0/3201613539 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4dc0c8ed0 0x7fc4dc1ee590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.110+0000 7fc4e258a640 1 --2- 172.21.15.161:0/3201613539 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4dc0c5950 0x7fc4dc0c1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.110+0000 7fc4e258a640 1 -- 172.21.15.161:0/3201613539 >> 172.21.15.161:0/3201613539 conn(0x7fc4dc0ac620 msgr2=0x7fc4dc14c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:54.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.110+0000 7fc4e258a640 1 -- 172.21.15.161:0/3201613539 shutdown_connections 2024-03-28T23:33:54.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.110+0000 7fc4e258a640 1 -- 172.21.15.161:0/3201613539 wait complete. 2024-03-28T23:33:54.122 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-03-28T23:33:53.358505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255720,"kb_used_data":10392,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754719000,"statfs":{"total":773094113280,"available":772832256000,"internally_reserved":0,"allocated":10641408,"data_stored":6230594,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.386735"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.980110+0000","last_change":"2024-03-28T23:33:48.980110+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.980110+0000","last_undegraded":"2024-03-28T23:33:48.980110+0000","last_fullsized":"2024-03-28T23:33:48.980110+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976936+0000","last_change":"2024-03-28T23:33:48.976936+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976936+0000","last_undegraded":"2024-03-28T23:33:48.976936+0000","last_fullsized":"2024-03-28T23:33:48.976936+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976327+0000","last_change":"2024-03-28T23:33:48.976327+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976327+0000","last_undegraded":"2024-03-28T23:33:48.976327+0000","last_fullsized":"2024-03-28T23:33:48.976327+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976494+0000","last_change":"2024-03-28T23:33:48.976494+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976494+0000","last_undegraded":"2024-03-28T23:33:48.976494+0000","last_fullsized":"2024-03-28T23:33:48.976494+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973933+0000","last_change":"2024-03-28T23:33:48.973933+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973933+0000","last_undegraded":"2024-03-28T23:33:48.973933+0000","last_fullsized":"2024-03-28T23:33:48.973933+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973246+0000","last_change":"2024-03-28T23:33:48.973246+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973246+0000","last_undegraded":"2024-03-28T23:33:48.973246+0000","last_fullsized":"2024-03-28T23:33:48.973246+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976300+0000","last_change":"2024-03-28T23:33:48.976300+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976300+0000","last_undegraded":"2024-03-28T23:33:48.976300+0000","last_fullsized":"2024-03-28T23:33:48.976300+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973252+0000","last_change":"2024-03-28T23:33:48.973252+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973252+0000","last_undegraded":"2024-03-28T23:33:48.973252+0000","last_fullsized":"2024-03-28T23:33:48.973252+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.979663+0000","last_change":"2024-03-28T23:33:48.979663+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.979663+0000","last_undegraded":"2024-03-28T23:33:48.979663+0000","last_fullsized":"2024-03-28T23:33:48.979663+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976398+0000","last_change":"2024-03-28T23:33:48.976398+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976398+0000","last_undegraded":"2024-03-28T23:33:48.976398+0000","last_fullsized":"2024-03-28T23:33:48.976398+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976142+0000","last_change":"2024-03-28T23:33:48.976142+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976142+0000","last_undegraded":"2024-03-28T23:33:48.976142+0000","last_fullsized":"2024-03-28T23:33:48.976142+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975388+0000","last_change":"2024-03-28T23:33:48.975388+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975388+0000","last_undegraded":"2024-03-28T23:33:48.975388+0000","last_fullsized":"2024-03-28T23:33:48.975388+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975171+0000","last_change":"2024-03-28T23:33:48.975171+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975171+0000","last_undegraded":"2024-03-28T23:33:48.975171+0000","last_fullsized":"2024-03-28T23:33:48.975171+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975112+0000","last_change":"2024-03-28T23:33:48.975112+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975112+0000","last_undegraded":"2024-03-28T23:33:48.975112+0000","last_fullsized":"2024-03-28T23:33:48.975112+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975532+0000","last_change":"2024-03-28T23:33:48.975532+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975532+0000","last_undegraded":"2024-03-28T23:33:48.975532+0000","last_fullsized":"2024-03-28T23:33:48.975532+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975984+0000","last_change":"2024-03-28T23:33:48.975984+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975984+0000","last_undegraded":"2024-03-28T23:33:48.975984+0000","last_fullsized":"2024-03-28T23:33:48.975984+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976398+0000","last_change":"2024-03-28T23:33:48.976398+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976398+0000","last_undegraded":"2024-03-28T23:33:48.976398+0000","last_fullsized":"2024-03-28T23:33:48.976398+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975786+0000","last_change":"2024-03-28T23:33:48.975786+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975786+0000","last_undegraded":"2024-03-28T23:33:48.975786+0000","last_fullsized":"2024-03-28T23:33:48.975786+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975738+0000","last_change":"2024-03-28T23:33:48.975738+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975738+0000","last_undegraded":"2024-03-28T23:33:48.975738+0000","last_fullsized":"2024-03-28T23:33:48.975738+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973245+0000","last_change":"2024-03-28T23:33:48.973245+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973245+0000","last_undegraded":"2024-03-28T23:33:48.973245+0000","last_fullsized":"2024-03-28T23:33:48.973245+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976062+0000","last_change":"2024-03-28T23:33:48.976062+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976062+0000","last_undegraded":"2024-03-28T23:33:48.976062+0000","last_fullsized":"2024-03-28T23:33:48.976062+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975100+0000","last_change":"2024-03-28T23:33:48.975100+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975100+0000","last_undegraded":"2024-03-28T23:33:48.975100+0000","last_fullsized":"2024-03-28T23:33:48.975100+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.977053+0000","last_change":"2024-03-28T23:33:48.977053+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.977053+0000","last_undegraded":"2024-03-28T23:33:48.977053+0000","last_fullsized":"2024-03-28T23:33:48.977053+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973198+0000","last_change":"2024-03-28T23:33:48.973198+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973198+0000","last_undegraded":"2024-03-28T23:33:48.973198+0000","last_fullsized":"2024-03-28T23:33:48.973198+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.979374+0000","last_change":"2024-03-28T23:33:48.979374+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.979374+0000","last_undegraded":"2024-03-28T23:33:48.979374+0000","last_fullsized":"2024-03-28T23:33:48.979374+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.979329+0000","last_change":"2024-03-28T23:33:48.979329+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.979329+0000","last_undegraded":"2024-03-28T23:33:48.979329+0000","last_fullsized":"2024-03-28T23:33:48.979329+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973409+0000","last_change":"2024-03-28T23:33:48.973409+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973409+0000","last_undegraded":"2024-03-28T23:33:48.973409+0000","last_fullsized":"2024-03-28T23:33:48.973409+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973253+0000","last_change":"2024-03-28T23:33:48.973253+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973253+0000","last_undegraded":"2024-03-28T23:33:48.973253+0000","last_fullsized":"2024-03-28T23:33:48.973253+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973312+0000","last_change":"2024-03-28T23:33:48.973312+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973312+0000","last_undegraded":"2024-03-28T23:33:48.973312+0000","last_fullsized":"2024-03-28T23:33:48.973312+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973203+0000","last_change":"2024-03-28T23:33:48.973203+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973203+0000","last_undegraded":"2024-03-28T23:33:48.973203+0000","last_fullsized":"2024-03-28T23:33:48.973203+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976032+0000","last_change":"2024-03-28T23:33:48.976032+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976032+0000","last_undegraded":"2024-03-28T23:33:48.976032+0000","last_fullsized":"2024-03-28T23:33:48.976032+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975006+0000","last_change":"2024-03-28T23:33:48.975006+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975006+0000","last_undegraded":"2024-03-28T23:33:48.975006+0000","last_fullsized":"2024-03-28T23:33:48.975006+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":332,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-28T23:33:48.977471+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:33:48.977471+0000","last_peered":"2024-03-28T23:33:48.977471+0000","last_clean":"2024-03-28T23:33:48.977471+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:33:48.977471+0000","last_undegraded":"2024-03-28T23:33:48.977471+0000","last_fullsized":"2024-03-28T23:33:48.977471+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32524,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339316,"statfs":{"total":96636764160,"available":96603459584,"internally_reserved":0,"allocated":1376256,"data_stored":1220048,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1323008,"data_stored":635889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32464,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339376,"statfs":{"total":96636764160,"available":96603521024,"internally_reserved":0,"allocated":1310720,"data_stored":629680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1310720,"data_stored":629680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1310720,"data_stored":629680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28304,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343536,"statfs":{"total":96636764160,"available":96607780864,"internally_reserved":0,"allocated":1310720,"data_stored":629680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1310720,"data_stored":629680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32540,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339300,"statfs":{"total":96636764160,"available":96603443200,"internally_reserved":0,"allocated":1388544,"data_stored":1226257,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:33:54.126 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:33:54.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.263+0000 7fb7a7284640 1 Processor -- start 2024-03-28T23:33:54.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.263+0000 7fb7a7284640 1 -- start start 2024-03-28T23:33:54.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.264+0000 7fb7a7284640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb7a00c5410 0x7fb7a00c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:54.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.264+0000 7fb7a7284640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb7a01553a0 0x7fb7a0155850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:54.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.264+0000 7fb7a7284640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7a0158a30 con 0x7fb7a0152890 2024-03-28T23:33:54.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.264+0000 7fb7a7284640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb7a00c8480 con 0x7fb7a01553a0 2024-03-28T23:33:54.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.264+0000 7fb7a7284640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fb7a00c5d90 con 0x7fb7a00c5410 2024-03-28T23:33:54.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.264+0000 7fb7a5821640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb7a01553a0 0x7fb7a0155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:54.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.264+0000 7fb7a5821640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb7a01553a0 0x7fb7a0155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60078/0 (socket says 172.21.15.161:60078) 2024-03-28T23:33:54.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.264+0000 7fb7a5821640 1 -- 172.21.15.161:0/1320242436 learned_addr learned my addr 172.21.15.161:0/1320242436 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:54.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.265+0000 7fb7a481f640 1 --2- 172.21.15.161:0/1320242436 >> v2:172.21.15.170:3300/0 conn(0x7fb7a00c5410 0x7fb7a00c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:54.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.265+0000 7fb7a5821640 1 -- 172.21.15.161:0/1320242436 >> v2:172.21.15.170:3300/0 conn(0x7fb7a00c5410 msgr2=0x7fb7a00c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:54.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.265+0000 7fb7a5821640 1 --2- 172.21.15.161:0/1320242436 >> v2:172.21.15.170:3300/0 conn(0x7fb7a00c5410 0x7fb7a00c5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.265+0000 7fb7a5821640 1 -- 172.21.15.161:0/1320242436 >> v1:172.21.15.170:6789/0 conn(0x7fb7a0152890 legacy=0x7fb7a00c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:54.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.265+0000 7fb7a5821640 1 -- 172.21.15.161:0/1320242436 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7a0158d10 con 0x7fb7a01553a0 2024-03-28T23:33:54.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.265+0000 7fb7a5821640 1 --2- 172.21.15.161:0/1320242436 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7a01553a0 0x7fb7a0155850 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fb7a00ad6e0 tx=0x7fb79c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c16d2881c54b50d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:54.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.266+0000 7fb797fff640 1 -- 172.21.15.161:0/1320242436 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79c02c9f0 con 0x7fb7a01553a0 2024-03-28T23:33:54.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.266+0000 7fb797fff640 1 -- 172.21.15.161:0/1320242436 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb79c02cb90 con 0x7fb7a01553a0 2024-03-28T23:33:54.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.266+0000 7fb797fff640 1 -- 172.21.15.161:0/1320242436 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79c0343f0 con 0x7fb7a01553a0 2024-03-28T23:33:54.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.266+0000 7fb7a7284640 1 -- 172.21.15.161:0/1320242436 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7a01553a0 msgr2=0x7fb7a0155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:54.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.266+0000 7fb7a7284640 1 --2- 172.21.15.161:0/1320242436 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7a01553a0 0x7fb7a0155850 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fb7a00ad6e0 tx=0x7fb79c02be50 comp rx=0 tx=0).stop 2024-03-28T23:33:54.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.266+0000 7fb7a7284640 1 -- 172.21.15.161:0/1320242436 shutdown_connections 2024-03-28T23:33:54.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.266+0000 7fb7a7284640 1 --2- 172.21.15.161:0/1320242436 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7a01553a0 0x7fb7a0155850 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.266+0000 7fb7a7284640 1 --2- 172.21.15.161:0/1320242436 >> v2:172.21.15.170:3300/0 conn(0x7fb7a00c5410 0x7fb7a00c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.266+0000 7fb7a7284640 1 -- 172.21.15.161:0/1320242436 >> 172.21.15.161:0/1320242436 conn(0x7fb7a00ac620 msgr2=0x7fb7a00aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:54.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.266+0000 7fb7a7284640 1 -- 172.21.15.161:0/1320242436 shutdown_connections 2024-03-28T23:33:54.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.266+0000 7fb7a7284640 1 -- 172.21.15.161:0/1320242436 wait complete. 2024-03-28T23:33:54.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.267+0000 7fb7a7284640 1 Processor -- start 2024-03-28T23:33:54.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.267+0000 7fb7a7284640 1 -- start start 2024-03-28T23:33:54.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.268+0000 7fb7a7284640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7a00c5410 0x7fb7a0148830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:54.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.268+0000 7fb7a7284640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7a0152890 0x7fb7a014aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:54.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.268+0000 7fb7a7284640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7a01553a0 0x7fb7a0150390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:54.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.268+0000 7fb7a7284640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a01592d0 con 0x7fb7a0152890 2024-03-28T23:33:54.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.268+0000 7fb7a7284640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a0159640 con 0x7fb7a01553a0 2024-03-28T23:33:54.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.268+0000 7fb7a7284640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb7a00c90a0 con 0x7fb7a00c5410 2024-03-28T23:33:54.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.268+0000 7fb7a481f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7a0152890 0x7fb7a014aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:54.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.268+0000 7fb7a481f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7a0152890 0x7fb7a014aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60080/0 (socket says 172.21.15.161:60080) 2024-03-28T23:33:54.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.268+0000 7fb7a481f640 1 -- 172.21.15.161:0/1922546991 learned_addr learned my addr 172.21.15.161:0/1922546991 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:54.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.268+0000 7fb7a5821640 1 --2- 172.21.15.161:0/1922546991 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7a01553a0 0x7fb7a0150390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:54.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.268+0000 7fb7a5020640 1 --2- 172.21.15.161:0/1922546991 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7a00c5410 0x7fb7a0148830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:54.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.268+0000 7fb7a481f640 1 -- 172.21.15.161:0/1922546991 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7a00c5410 msgr2=0x7fb7a0148830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:54.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.268+0000 7fb7a481f640 1 --2- 172.21.15.161:0/1922546991 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7a00c5410 0x7fb7a0148830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.268+0000 7fb7a481f640 1 -- 172.21.15.161:0/1922546991 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7a01553a0 msgr2=0x7fb7a0150390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:54.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.268+0000 7fb7a481f640 1 --2- 172.21.15.161:0/1922546991 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7a01553a0 0x7fb7a0150390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.268+0000 7fb7a481f640 1 -- 172.21.15.161:0/1922546991 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7a01eb2e0 con 0x7fb7a0152890 2024-03-28T23:33:54.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.268+0000 7fb7a5821640 1 --2- 172.21.15.161:0/1922546991 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7a01553a0 0x7fb7a0150390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:54.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.268+0000 7fb7a5020640 1 --2- 172.21.15.161:0/1922546991 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7a00c5410 0x7fb7a0148830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:54.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.268+0000 7fb7a481f640 1 --2- 172.21.15.161:0/1922546991 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7a0152890 0x7fb7a014aee0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fb79000c610 tx=0x7fb79000cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:54.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.269+0000 7fb795ffb640 1 -- 172.21.15.161:0/1922546991 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb790004230 con 0x7fb7a0152890 2024-03-28T23:33:54.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.269+0000 7fb795ffb640 1 -- 172.21.15.161:0/1922546991 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7900043d0 con 0x7fb7a0152890 2024-03-28T23:33:54.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.269+0000 7fb7a7284640 1 -- 172.21.15.161:0/1922546991 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7a01eb5c0 con 0x7fb7a0152890 2024-03-28T23:33:54.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.269+0000 7fb795ffb640 1 -- 172.21.15.161:0/1922546991 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb790003830 con 0x7fb7a0152890 2024-03-28T23:33:54.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.269+0000 7fb7a7284640 1 -- 172.21.15.161:0/1922546991 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7a01ebb80 con 0x7fb7a0152890 2024-03-28T23:33:54.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.270+0000 7fb7a7284640 1 -- 172.21.15.161:0/1922546991 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7a01531d0 con 0x7fb7a0152890 2024-03-28T23:33:54.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.271+0000 7fb795ffb640 1 -- 172.21.15.161:0/1922546991 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb7900039d0 con 0x7fb7a0152890 2024-03-28T23:33:54.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.271+0000 7fb795ffb640 1 --2- 172.21.15.161:0/1922546991 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb78406b4d0 0x7fb78406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:54.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.271+0000 7fb795ffb640 1 -- 172.21.15.161:0/1922546991 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6101+0+0 (secure 0 0 0) 0x7fb790089ee0 con 0x7fb7a0152890 2024-03-28T23:33:54.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.271+0000 7fb7a5020640 1 --2- 172.21.15.161:0/1922546991 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb78406b4d0 0x7fb78406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:54.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.272+0000 7fb7a5020640 1 --2- 172.21.15.161:0/1922546991 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb78406b4d0 0x7fb78406d990 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fb798007fd0 tx=0x7fb798009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:54.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.276+0000 7fb795ffb640 1 -- 172.21.15.161:0/1922546991 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb7a01531d0 con 0x7fb7a0152890 2024-03-28T23:33:54.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.450+0000 7fb7a7284640 1 -- 172.21.15.161:0/1922546991 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb7a01ebef0 con 0x7fb78406b4d0 2024-03-28T23:33:54.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.455+0000 7fb795ffb640 1 -- 172.21.15.161:0/1922546991 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+240322 (secure 0 0 0) 0x7fb7a01ebef0 con 0x7fb78406b4d0 2024-03-28T23:33:54.458 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:54.459 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:33:54.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.458+0000 7fb7a7284640 1 -- 172.21.15.161:0/1922546991 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb78406b4d0 msgr2=0x7fb78406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:54.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.458+0000 7fb7a7284640 1 --2- 172.21.15.161:0/1922546991 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb78406b4d0 0x7fb78406d990 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fb798007fd0 tx=0x7fb798009040 comp rx=0 tx=0).stop 2024-03-28T23:33:54.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.458+0000 7fb7a7284640 1 -- 172.21.15.161:0/1922546991 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7a0152890 msgr2=0x7fb7a014aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:54.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.458+0000 7fb7a7284640 1 --2- 172.21.15.161:0/1922546991 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7a0152890 0x7fb7a014aee0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fb79000c610 tx=0x7fb79000cad0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.458+0000 7fb7a7284640 1 -- 172.21.15.161:0/1922546991 shutdown_connections 2024-03-28T23:33:54.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.458+0000 7fb7a7284640 1 --2- 172.21.15.161:0/1922546991 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb78406b4d0 0x7fb78406d990 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.458+0000 7fb7a7284640 1 --2- 172.21.15.161:0/1922546991 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7a0152890 0x7fb7a014aee0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.458+0000 7fb7a7284640 1 --2- 172.21.15.161:0/1922546991 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7a01553a0 0x7fb7a0150390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.458+0000 7fb7a7284640 1 --2- 172.21.15.161:0/1922546991 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7a00c5410 0x7fb7a0148830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.458+0000 7fb7a7284640 1 -- 172.21.15.161:0/1922546991 >> 172.21.15.161:0/1922546991 conn(0x7fb7a00ac620 msgr2=0x7fb7a00caa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:54.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.458+0000 7fb7a7284640 1 -- 172.21.15.161:0/1922546991 shutdown_connections 2024-03-28T23:33:54.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.458+0000 7fb7a7284640 1 -- 172.21.15.161:0/1922546991 wait complete. 2024-03-28T23:33:54.470 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-03-28T23:33:53.358505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255720,"kb_used_data":10392,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754719000,"statfs":{"total":773094113280,"available":772832256000,"internally_reserved":0,"allocated":10641408,"data_stored":6230594,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.386735"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.980110+0000","last_change":"2024-03-28T23:33:48.980110+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.980110+0000","last_undegraded":"2024-03-28T23:33:48.980110+0000","last_fullsized":"2024-03-28T23:33:48.980110+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976936+0000","last_change":"2024-03-28T23:33:48.976936+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976936+0000","last_undegraded":"2024-03-28T23:33:48.976936+0000","last_fullsized":"2024-03-28T23:33:48.976936+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976327+0000","last_change":"2024-03-28T23:33:48.976327+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976327+0000","last_undegraded":"2024-03-28T23:33:48.976327+0000","last_fullsized":"2024-03-28T23:33:48.976327+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976494+0000","last_change":"2024-03-28T23:33:48.976494+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976494+0000","last_undegraded":"2024-03-28T23:33:48.976494+0000","last_fullsized":"2024-03-28T23:33:48.976494+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973933+0000","last_change":"2024-03-28T23:33:48.973933+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973933+0000","last_undegraded":"2024-03-28T23:33:48.973933+0000","last_fullsized":"2024-03-28T23:33:48.973933+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973246+0000","last_change":"2024-03-28T23:33:48.973246+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973246+0000","last_undegraded":"2024-03-28T23:33:48.973246+0000","last_fullsized":"2024-03-28T23:33:48.973246+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976300+0000","last_change":"2024-03-28T23:33:48.976300+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976300+0000","last_undegraded":"2024-03-28T23:33:48.976300+0000","last_fullsized":"2024-03-28T23:33:48.976300+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973252+0000","last_change":"2024-03-28T23:33:48.973252+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973252+0000","last_undegraded":"2024-03-28T23:33:48.973252+0000","last_fullsized":"2024-03-28T23:33:48.973252+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.979663+0000","last_change":"2024-03-28T23:33:48.979663+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.979663+0000","last_undegraded":"2024-03-28T23:33:48.979663+0000","last_fullsized":"2024-03-28T23:33:48.979663+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976398+0000","last_change":"2024-03-28T23:33:48.976398+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976398+0000","last_undegraded":"2024-03-28T23:33:48.976398+0000","last_fullsized":"2024-03-28T23:33:48.976398+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976142+0000","last_change":"2024-03-28T23:33:48.976142+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976142+0000","last_undegraded":"2024-03-28T23:33:48.976142+0000","last_fullsized":"2024-03-28T23:33:48.976142+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975388+0000","last_change":"2024-03-28T23:33:48.975388+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975388+0000","last_undegraded":"2024-03-28T23:33:48.975388+0000","last_fullsized":"2024-03-28T23:33:48.975388+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975171+0000","last_change":"2024-03-28T23:33:48.975171+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975171+0000","last_undegraded":"2024-03-28T23:33:48.975171+0000","last_fullsized":"2024-03-28T23:33:48.975171+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975112+0000","last_change":"2024-03-28T23:33:48.975112+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975112+0000","last_undegraded":"2024-03-28T23:33:48.975112+0000","last_fullsized":"2024-03-28T23:33:48.975112+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975532+0000","last_change":"2024-03-28T23:33:48.975532+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975532+0000","last_undegraded":"2024-03-28T23:33:48.975532+0000","last_fullsized":"2024-03-28T23:33:48.975532+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975984+0000","last_change":"2024-03-28T23:33:48.975984+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975984+0000","last_undegraded":"2024-03-28T23:33:48.975984+0000","last_fullsized":"2024-03-28T23:33:48.975984+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976398+0000","last_change":"2024-03-28T23:33:48.976398+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976398+0000","last_undegraded":"2024-03-28T23:33:48.976398+0000","last_fullsized":"2024-03-28T23:33:48.976398+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975786+0000","last_change":"2024-03-28T23:33:48.975786+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975786+0000","last_undegraded":"2024-03-28T23:33:48.975786+0000","last_fullsized":"2024-03-28T23:33:48.975786+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975738+0000","last_change":"2024-03-28T23:33:48.975738+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975738+0000","last_undegraded":"2024-03-28T23:33:48.975738+0000","last_fullsized":"2024-03-28T23:33:48.975738+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973245+0000","last_change":"2024-03-28T23:33:48.973245+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973245+0000","last_undegraded":"2024-03-28T23:33:48.973245+0000","last_fullsized":"2024-03-28T23:33:48.973245+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976062+0000","last_change":"2024-03-28T23:33:48.976062+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976062+0000","last_undegraded":"2024-03-28T23:33:48.976062+0000","last_fullsized":"2024-03-28T23:33:48.976062+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975100+0000","last_change":"2024-03-28T23:33:48.975100+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975100+0000","last_undegraded":"2024-03-28T23:33:48.975100+0000","last_fullsized":"2024-03-28T23:33:48.975100+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.977053+0000","last_change":"2024-03-28T23:33:48.977053+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.977053+0000","last_undegraded":"2024-03-28T23:33:48.977053+0000","last_fullsized":"2024-03-28T23:33:48.977053+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973198+0000","last_change":"2024-03-28T23:33:48.973198+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973198+0000","last_undegraded":"2024-03-28T23:33:48.973198+0000","last_fullsized":"2024-03-28T23:33:48.973198+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.979374+0000","last_change":"2024-03-28T23:33:48.979374+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.979374+0000","last_undegraded":"2024-03-28T23:33:48.979374+0000","last_fullsized":"2024-03-28T23:33:48.979374+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.979329+0000","last_change":"2024-03-28T23:33:48.979329+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.979329+0000","last_undegraded":"2024-03-28T23:33:48.979329+0000","last_fullsized":"2024-03-28T23:33:48.979329+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973409+0000","last_change":"2024-03-28T23:33:48.973409+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973409+0000","last_undegraded":"2024-03-28T23:33:48.973409+0000","last_fullsized":"2024-03-28T23:33:48.973409+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973253+0000","last_change":"2024-03-28T23:33:48.973253+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973253+0000","last_undegraded":"2024-03-28T23:33:48.973253+0000","last_fullsized":"2024-03-28T23:33:48.973253+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973312+0000","last_change":"2024-03-28T23:33:48.973312+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973312+0000","last_undegraded":"2024-03-28T23:33:48.973312+0000","last_fullsized":"2024-03-28T23:33:48.973312+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973203+0000","last_change":"2024-03-28T23:33:48.973203+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973203+0000","last_undegraded":"2024-03-28T23:33:48.973203+0000","last_fullsized":"2024-03-28T23:33:48.973203+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976032+0000","last_change":"2024-03-28T23:33:48.976032+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976032+0000","last_undegraded":"2024-03-28T23:33:48.976032+0000","last_fullsized":"2024-03-28T23:33:48.976032+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975006+0000","last_change":"2024-03-28T23:33:48.975006+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975006+0000","last_undegraded":"2024-03-28T23:33:48.975006+0000","last_fullsized":"2024-03-28T23:33:48.975006+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":332,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-28T23:33:48.977471+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:33:48.977471+0000","last_peered":"2024-03-28T23:33:48.977471+0000","last_clean":"2024-03-28T23:33:48.977471+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:33:48.977471+0000","last_undegraded":"2024-03-28T23:33:48.977471+0000","last_fullsized":"2024-03-28T23:33:48.977471+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32524,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339316,"statfs":{"total":96636764160,"available":96603459584,"internally_reserved":0,"allocated":1376256,"data_stored":1220048,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1323008,"data_stored":635889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32464,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339376,"statfs":{"total":96636764160,"available":96603521024,"internally_reserved":0,"allocated":1310720,"data_stored":629680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1310720,"data_stored":629680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1310720,"data_stored":629680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28304,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343536,"statfs":{"total":96636764160,"available":96607780864,"internally_reserved":0,"allocated":1310720,"data_stored":629680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1310720,"data_stored":629680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32540,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339300,"statfs":{"total":96636764160,"available":96603443200,"internally_reserved":0,"allocated":1388544,"data_stored":1226257,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:33:54.473 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:33:54.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.613+0000 7f91527c2640 1 Processor -- start 2024-03-28T23:33:54.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.613+0000 7f91527c2640 1 -- start start 2024-03-28T23:33:54.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.614+0000 7f91527c2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f914c155b90 0x7f914c158750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:54.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.614+0000 7f91527c2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f914c0ac150 0x7f914c158c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:54.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.614+0000 7f91527c2640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f914c066fe0 con 0x7f914c06c3d0 2024-03-28T23:33:54.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.614+0000 7f91527c2640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f914c1591d0 con 0x7f914c155b90 2024-03-28T23:33:54.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.614+0000 7f91527c2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f914c156070 con 0x7f914c0ac150 2024-03-28T23:33:54.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.614+0000 7f914b7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f914c155b90 0x7f914c158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:54.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.614+0000 7f914bfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f914c06c3d0 0x7f914c152580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41480/0 (socket says 172.21.15.161:41480) 2024-03-28T23:33:54.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.614+0000 7f914bfff640 1 -- 172.21.15.161:0/2105431212 learned_addr learned my addr 172.21.15.161:0/2105431212 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:54.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.614+0000 7f9150d5f640 1 --2- 172.21.15.161:0/2105431212 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f914c0ac150 0x7f914c158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:54.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.614+0000 7f914b7fe640 1 -- 172.21.15.161:0/2105431212 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f914c0ac150 msgr2=0x7f914c158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:54.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.614+0000 7f914b7fe640 1 --2- 172.21.15.161:0/2105431212 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f914c0ac150 0x7f914c158c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.614+0000 7f914b7fe640 1 -- 172.21.15.161:0/2105431212 >> v1:172.21.15.161:6789/0 conn(0x7f914c06c3d0 legacy=0x7f914c152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:54.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.614+0000 7f914b7fe640 1 -- 172.21.15.161:0/2105431212 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f914c0672c0 con 0x7f914c155b90 2024-03-28T23:33:54.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.615+0000 7f914b7fe640 1 --2- 172.21.15.161:0/2105431212 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f914c155b90 0x7f914c158750 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f913c0097a0 tx=0x7f913c02b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf39c7bdd827402 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:54.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.615+0000 7f914affd640 1 -- 172.21.15.161:0/2105431212 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f913c037070 con 0x7f914c155b90 2024-03-28T23:33:54.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.615+0000 7f914affd640 1 -- 172.21.15.161:0/2105431212 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f913c030710 con 0x7f914c155b90 2024-03-28T23:33:54.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.615+0000 7f914affd640 1 -- 172.21.15.161:0/2105431212 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f913c030ae0 con 0x7f914c155b90 2024-03-28T23:33:54.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.615+0000 7f91527c2640 1 -- 172.21.15.161:0/2105431212 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f914c155b90 msgr2=0x7f914c158750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:54.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.615+0000 7f91527c2640 1 --2- 172.21.15.161:0/2105431212 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f914c155b90 0x7f914c158750 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f913c0097a0 tx=0x7f913c02b710 comp rx=0 tx=0).stop 2024-03-28T23:33:54.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.616+0000 7f91527c2640 1 -- 172.21.15.161:0/2105431212 shutdown_connections 2024-03-28T23:33:54.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.616+0000 7f91527c2640 1 --2- 172.21.15.161:0/2105431212 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f914c0ac150 0x7f914c158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.616+0000 7f91527c2640 1 --2- 172.21.15.161:0/2105431212 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f914c155b90 0x7f914c158750 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.616+0000 7f91527c2640 1 -- 172.21.15.161:0/2105431212 >> 172.21.15.161:0/2105431212 conn(0x7f914c0c3be0 msgr2=0x7f914c0c3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:54.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.616+0000 7f91527c2640 1 -- 172.21.15.161:0/2105431212 shutdown_connections 2024-03-28T23:33:54.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.616+0000 7f91527c2640 1 -- 172.21.15.161:0/2105431212 wait complete. 2024-03-28T23:33:54.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.617+0000 7f91527c2640 1 Processor -- start 2024-03-28T23:33:54.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.617+0000 7f91527c2640 1 -- start start 2024-03-28T23:33:54.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.617+0000 7f91527c2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f914c06c3d0 0x7f914c1577f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:54.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.617+0000 7f91527c2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f914c0ac150 0x7f914c0711a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:54.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.617+0000 7f91527c2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f914c155b90 0x7f914c0716e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:54.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.617+0000 7f91527c2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f914c152e10 con 0x7f914c0ac150 2024-03-28T23:33:54.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.617+0000 7f91527c2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f914c152c90 con 0x7f914c06c3d0 2024-03-28T23:33:54.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.617+0000 7f91527c2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f914c0c21a0 con 0x7f914c155b90 2024-03-28T23:33:54.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.618+0000 7f914b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f914c0ac150 0x7f914c0711a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:54.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.618+0000 7f914b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f914c0ac150 0x7f914c0711a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60094/0 (socket says 172.21.15.161:60094) 2024-03-28T23:33:54.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.618+0000 7f914b7fe640 1 -- 172.21.15.161:0/2128329748 learned_addr learned my addr 172.21.15.161:0/2128329748 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:54.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.618+0000 7f9150d5f640 1 --2- 172.21.15.161:0/2128329748 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f914c155b90 0x7f914c0716e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:54.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.618+0000 7f914bfff640 1 --2- 172.21.15.161:0/2128329748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f914c06c3d0 0x7f914c1577f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:54.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.618+0000 7f914b7fe640 1 -- 172.21.15.161:0/2128329748 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f914c155b90 msgr2=0x7f914c0716e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:54.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.618+0000 7f914b7fe640 1 --2- 172.21.15.161:0/2128329748 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f914c155b90 0x7f914c0716e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.618+0000 7f914b7fe640 1 -- 172.21.15.161:0/2128329748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f914c06c3d0 msgr2=0x7f914c1577f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:54.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.618+0000 7f914b7fe640 1 --2- 172.21.15.161:0/2128329748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f914c06c3d0 0x7f914c1577f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.618+0000 7f914b7fe640 1 -- 172.21.15.161:0/2128329748 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f914c0c6b60 con 0x7f914c0ac150 2024-03-28T23:33:54.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.618+0000 7f914b7fe640 1 --2- 172.21.15.161:0/2128329748 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f914c0ac150 0x7f914c0711a0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f913c000c00 tx=0x7f913c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:54.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.618+0000 7f9148ff9640 1 -- 172.21.15.161:0/2128329748 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f913c041020 con 0x7f914c0ac150 2024-03-28T23:33:54.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.618+0000 7f9148ff9640 1 -- 172.21.15.161:0/2128329748 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f913c032a40 con 0x7f914c0ac150 2024-03-28T23:33:54.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.618+0000 7f91527c2640 1 -- 172.21.15.161:0/2128329748 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f914c0c6120 con 0x7f914c0ac150 2024-03-28T23:33:54.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.618+0000 7f9148ff9640 1 -- 172.21.15.161:0/2128329748 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f913c037040 con 0x7f914c0ac150 2024-03-28T23:33:54.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.618+0000 7f91527c2640 1 -- 172.21.15.161:0/2128329748 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f914c1ee940 con 0x7f914c0ac150 2024-03-28T23:33:54.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.619+0000 7f91527c2640 1 -- 172.21.15.161:0/2128329748 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f914c1eed20 con 0x7f914c0ac150 2024-03-28T23:33:54.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.620+0000 7f9148ff9640 1 -- 172.21.15.161:0/2128329748 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f913c044020 con 0x7f914c0ac150 2024-03-28T23:33:54.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.620+0000 7f9148ff9640 1 --2- 172.21.15.161:0/2128329748 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f913806b4d0 0x7f913806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:54.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.620+0000 7f9148ff9640 1 -- 172.21.15.161:0/2128329748 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6101+0+0 (secure 0 0 0) 0x7f913c0abdd0 con 0x7f914c0ac150 2024-03-28T23:33:54.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.621+0000 7f914bfff640 1 --2- 172.21.15.161:0/2128329748 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f913806b4d0 0x7f913806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:54.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.621+0000 7f914bfff640 1 --2- 172.21.15.161:0/2128329748 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f913806b4d0 0x7f913806d990 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9140007fd0 tx=0x7f9140009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:54.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.625+0000 7f9148ff9640 1 -- 172.21.15.161:0/2128329748 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f914c1eed20 con 0x7f914c0ac150 2024-03-28T23:33:54.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.832+0000 7f91527c2640 1 -- 172.21.15.161:0/2128329748 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f914c04f4b0 con 0x7f913806b4d0 2024-03-28T23:33:54.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.837+0000 7f9148ff9640 1 -- 172.21.15.161:0/2128329748 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+240322 (secure 0 0 0) 0x7f914c04f4b0 con 0x7f913806b4d0 2024-03-28T23:33:54.840 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:54.841 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:33:54.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.840+0000 7f91527c2640 1 -- 172.21.15.161:0/2128329748 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f913806b4d0 msgr2=0x7f913806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:54.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.840+0000 7f91527c2640 1 --2- 172.21.15.161:0/2128329748 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f913806b4d0 0x7f913806d990 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9140007fd0 tx=0x7f9140009040 comp rx=0 tx=0).stop 2024-03-28T23:33:54.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.840+0000 7f91527c2640 1 -- 172.21.15.161:0/2128329748 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f914c0ac150 msgr2=0x7f914c0711a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:54.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.840+0000 7f91527c2640 1 --2- 172.21.15.161:0/2128329748 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f914c0ac150 0x7f914c0711a0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f913c000c00 tx=0x7f913c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.841+0000 7f91527c2640 1 -- 172.21.15.161:0/2128329748 shutdown_connections 2024-03-28T23:33:54.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.841+0000 7f91527c2640 1 --2- 172.21.15.161:0/2128329748 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f913806b4d0 0x7f913806d990 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.841+0000 7f91527c2640 1 --2- 172.21.15.161:0/2128329748 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f914c0ac150 0x7f914c0711a0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.841+0000 7f91527c2640 1 --2- 172.21.15.161:0/2128329748 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f914c155b90 0x7f914c0716e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.841+0000 7f91527c2640 1 --2- 172.21.15.161:0/2128329748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f914c06c3d0 0x7f914c1577f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.841+0000 7f91527c2640 1 -- 172.21.15.161:0/2128329748 >> 172.21.15.161:0/2128329748 conn(0x7f914c0c3be0 msgr2=0x7f914c06e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:54.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.841+0000 7f91527c2640 1 -- 172.21.15.161:0/2128329748 shutdown_connections 2024-03-28T23:33:54.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.841+0000 7f91527c2640 1 -- 172.21.15.161:0/2128329748 wait complete. 2024-03-28T23:33:54.852 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-03-28T23:33:53.358505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255720,"kb_used_data":10392,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754719000,"statfs":{"total":773094113280,"available":772832256000,"internally_reserved":0,"allocated":10641408,"data_stored":6230594,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.386735"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.980110+0000","last_change":"2024-03-28T23:33:48.980110+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.980110+0000","last_undegraded":"2024-03-28T23:33:48.980110+0000","last_fullsized":"2024-03-28T23:33:48.980110+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976936+0000","last_change":"2024-03-28T23:33:48.976936+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976936+0000","last_undegraded":"2024-03-28T23:33:48.976936+0000","last_fullsized":"2024-03-28T23:33:48.976936+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976327+0000","last_change":"2024-03-28T23:33:48.976327+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976327+0000","last_undegraded":"2024-03-28T23:33:48.976327+0000","last_fullsized":"2024-03-28T23:33:48.976327+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976494+0000","last_change":"2024-03-28T23:33:48.976494+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976494+0000","last_undegraded":"2024-03-28T23:33:48.976494+0000","last_fullsized":"2024-03-28T23:33:48.976494+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973933+0000","last_change":"2024-03-28T23:33:48.973933+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973933+0000","last_undegraded":"2024-03-28T23:33:48.973933+0000","last_fullsized":"2024-03-28T23:33:48.973933+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973246+0000","last_change":"2024-03-28T23:33:48.973246+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973246+0000","last_undegraded":"2024-03-28T23:33:48.973246+0000","last_fullsized":"2024-03-28T23:33:48.973246+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976300+0000","last_change":"2024-03-28T23:33:48.976300+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976300+0000","last_undegraded":"2024-03-28T23:33:48.976300+0000","last_fullsized":"2024-03-28T23:33:48.976300+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973252+0000","last_change":"2024-03-28T23:33:48.973252+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973252+0000","last_undegraded":"2024-03-28T23:33:48.973252+0000","last_fullsized":"2024-03-28T23:33:48.973252+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.979663+0000","last_change":"2024-03-28T23:33:48.979663+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.979663+0000","last_undegraded":"2024-03-28T23:33:48.979663+0000","last_fullsized":"2024-03-28T23:33:48.979663+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976398+0000","last_change":"2024-03-28T23:33:48.976398+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976398+0000","last_undegraded":"2024-03-28T23:33:48.976398+0000","last_fullsized":"2024-03-28T23:33:48.976398+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976142+0000","last_change":"2024-03-28T23:33:48.976142+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976142+0000","last_undegraded":"2024-03-28T23:33:48.976142+0000","last_fullsized":"2024-03-28T23:33:48.976142+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975388+0000","last_change":"2024-03-28T23:33:48.975388+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975388+0000","last_undegraded":"2024-03-28T23:33:48.975388+0000","last_fullsized":"2024-03-28T23:33:48.975388+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975171+0000","last_change":"2024-03-28T23:33:48.975171+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975171+0000","last_undegraded":"2024-03-28T23:33:48.975171+0000","last_fullsized":"2024-03-28T23:33:48.975171+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975112+0000","last_change":"2024-03-28T23:33:48.975112+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975112+0000","last_undegraded":"2024-03-28T23:33:48.975112+0000","last_fullsized":"2024-03-28T23:33:48.975112+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975532+0000","last_change":"2024-03-28T23:33:48.975532+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975532+0000","last_undegraded":"2024-03-28T23:33:48.975532+0000","last_fullsized":"2024-03-28T23:33:48.975532+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975984+0000","last_change":"2024-03-28T23:33:48.975984+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975984+0000","last_undegraded":"2024-03-28T23:33:48.975984+0000","last_fullsized":"2024-03-28T23:33:48.975984+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976398+0000","last_change":"2024-03-28T23:33:48.976398+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976398+0000","last_undegraded":"2024-03-28T23:33:48.976398+0000","last_fullsized":"2024-03-28T23:33:48.976398+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975786+0000","last_change":"2024-03-28T23:33:48.975786+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975786+0000","last_undegraded":"2024-03-28T23:33:48.975786+0000","last_fullsized":"2024-03-28T23:33:48.975786+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975738+0000","last_change":"2024-03-28T23:33:48.975738+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975738+0000","last_undegraded":"2024-03-28T23:33:48.975738+0000","last_fullsized":"2024-03-28T23:33:48.975738+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973245+0000","last_change":"2024-03-28T23:33:48.973245+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973245+0000","last_undegraded":"2024-03-28T23:33:48.973245+0000","last_fullsized":"2024-03-28T23:33:48.973245+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976062+0000","last_change":"2024-03-28T23:33:48.976062+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976062+0000","last_undegraded":"2024-03-28T23:33:48.976062+0000","last_fullsized":"2024-03-28T23:33:48.976062+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975100+0000","last_change":"2024-03-28T23:33:48.975100+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975100+0000","last_undegraded":"2024-03-28T23:33:48.975100+0000","last_fullsized":"2024-03-28T23:33:48.975100+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.977053+0000","last_change":"2024-03-28T23:33:48.977053+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.977053+0000","last_undegraded":"2024-03-28T23:33:48.977053+0000","last_fullsized":"2024-03-28T23:33:48.977053+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973198+0000","last_change":"2024-03-28T23:33:48.973198+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973198+0000","last_undegraded":"2024-03-28T23:33:48.973198+0000","last_fullsized":"2024-03-28T23:33:48.973198+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.979374+0000","last_change":"2024-03-28T23:33:48.979374+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.979374+0000","last_undegraded":"2024-03-28T23:33:48.979374+0000","last_fullsized":"2024-03-28T23:33:48.979374+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.979329+0000","last_change":"2024-03-28T23:33:48.979329+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.979329+0000","last_undegraded":"2024-03-28T23:33:48.979329+0000","last_fullsized":"2024-03-28T23:33:48.979329+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973409+0000","last_change":"2024-03-28T23:33:48.973409+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973409+0000","last_undegraded":"2024-03-28T23:33:48.973409+0000","last_fullsized":"2024-03-28T23:33:48.973409+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973253+0000","last_change":"2024-03-28T23:33:48.973253+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973253+0000","last_undegraded":"2024-03-28T23:33:48.973253+0000","last_fullsized":"2024-03-28T23:33:48.973253+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973312+0000","last_change":"2024-03-28T23:33:48.973312+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973312+0000","last_undegraded":"2024-03-28T23:33:48.973312+0000","last_fullsized":"2024-03-28T23:33:48.973312+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.973203+0000","last_change":"2024-03-28T23:33:48.973203+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.973203+0000","last_undegraded":"2024-03-28T23:33:48.973203+0000","last_fullsized":"2024-03-28T23:33:48.973203+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.976032+0000","last_change":"2024-03-28T23:33:48.976032+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.976032+0000","last_undegraded":"2024-03-28T23:33:48.976032+0000","last_fullsized":"2024-03-28T23:33:48.976032+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-28T23:33:49.970798+0000","last_change":"2024-03-28T23:33:49.970798+0000","last_active":"2024-03-28T23:33:49.970798+0000","last_peered":"2024-03-28T23:33:49.970798+0000","last_clean":"2024-03-28T23:33:49.970798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:49.970798+0000","last_undegraded":"2024-03-28T23:33:49.970798+0000","last_fullsized":"2024-03-28T23:33:49.970798+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-03-28T23:33:48.975006+0000","last_change":"2024-03-28T23:33:48.975006+0000","last_active":"2024-03-28T23:33:48.969017+0000","last_peered":"2024-03-28T23:33:48.969017+0000","last_clean":"2024-03-28T23:33:48.969017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T23:33:48.975006+0000","last_undegraded":"2024-03-28T23:33:48.975006+0000","last_fullsized":"2024-03-28T23:33:48.975006+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":332,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-28T23:33:48.977471+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:33:48.977471+0000","last_peered":"2024-03-28T23:33:48.977471+0000","last_clean":"2024-03-28T23:33:48.977471+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:33:48.977471+0000","last_undegraded":"2024-03-28T23:33:48.977471+0000","last_fullsized":"2024-03-28T23:33:48.977471+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32524,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339316,"statfs":{"total":96636764160,"available":96603459584,"internally_reserved":0,"allocated":1376256,"data_stored":1220048,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1323008,"data_stored":635889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32464,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339376,"statfs":{"total":96636764160,"available":96603521024,"internally_reserved":0,"allocated":1310720,"data_stored":629680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1310720,"data_stored":629680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1310720,"data_stored":629680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28304,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343536,"statfs":{"total":96636764160,"available":96607780864,"internally_reserved":0,"allocated":1310720,"data_stored":629680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1310720,"data_stored":629680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32540,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339300,"statfs":{"total":96636764160,"available":96603443200,"internally_reserved":0,"allocated":1388544,"data_stored":1226257,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:33:54.856 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3f is not active+clean 2024-03-28T23:33:54.856 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.856 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3e is not active+clean 2024-03-28T23:33:54.856 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.856 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3d is not active+clean 2024-03-28T23:33:54.856 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.857 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3c is not active+clean 2024-03-28T23:33:54.857 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.857 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3b is not active+clean 2024-03-28T23:33:54.857 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.857 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3a is not active+clean 2024-03-28T23:33:54.857 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.857 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.39 is not active+clean 2024-03-28T23:33:54.857 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.857 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.38 is not active+clean 2024-03-28T23:33:54.857 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.857 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.37 is not active+clean 2024-03-28T23:33:54.858 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.858 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.36 is not active+clean 2024-03-28T23:33:54.858 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.858 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.35 is not active+clean 2024-03-28T23:33:54.858 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.859 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.34 is not active+clean 2024-03-28T23:33:54.859 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.859 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.33 is not active+clean 2024-03-28T23:33:54.859 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.859 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.32 is not active+clean 2024-03-28T23:33:54.859 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.859 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.31 is not active+clean 2024-03-28T23:33:54.859 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.859 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.30 is not active+clean 2024-03-28T23:33:54.859 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.859 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2f is not active+clean 2024-03-28T23:33:54.859 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.860 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2e is not active+clean 2024-03-28T23:33:54.860 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.860 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2d is not active+clean 2024-03-28T23:33:54.860 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.860 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2c is not active+clean 2024-03-28T23:33:54.860 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.860 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2b is not active+clean 2024-03-28T23:33:54.860 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.860 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2a is not active+clean 2024-03-28T23:33:54.860 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.860 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.29 is not active+clean 2024-03-28T23:33:54.860 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.861 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.28 is not active+clean 2024-03-28T23:33:54.861 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.861 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.27 is not active+clean 2024-03-28T23:33:54.861 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.861 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.26 is not active+clean 2024-03-28T23:33:54.861 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.861 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.25 is not active+clean 2024-03-28T23:33:54.861 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.861 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.24 is not active+clean 2024-03-28T23:33:54.861 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.861 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.23 is not active+clean 2024-03-28T23:33:54.861 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.862 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.22 is not active+clean 2024-03-28T23:33:54.862 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.862 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.21 is not active+clean 2024-03-28T23:33:54.862 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.862 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.20 is not active+clean 2024-03-28T23:33:54.862 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.862 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1c is not active+clean 2024-03-28T23:33:54.862 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.862 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1f is not active+clean 2024-03-28T23:33:54.862 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.980110+0000', 'last_change': '2024-03-28T23:33:48.980110+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.980110+0000', 'last_undegraded': '2024-03-28T23:33:48.980110+0000', 'last_fullsized': '2024-03-28T23:33:48.980110+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:33:54.862 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1d is not active+clean 2024-03-28T23:33:54.863 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.863 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1e is not active+clean 2024-03-28T23:33:54.863 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.976936+0000', 'last_change': '2024-03-28T23:33:48.976936+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.976936+0000', 'last_undegraded': '2024-03-28T23:33:48.976936+0000', 'last_fullsized': '2024-03-28T23:33:48.976936+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-28T23:33:54.863 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1e is not active+clean 2024-03-28T23:33:54.863 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.863 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1d is not active+clean 2024-03-28T23:33:54.863 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.976327+0000', 'last_change': '2024-03-28T23:33:48.976327+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.976327+0000', 'last_undegraded': '2024-03-28T23:33:48.976327+0000', 'last_fullsized': '2024-03-28T23:33:48.976327+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-28T23:33:54.863 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1f is not active+clean 2024-03-28T23:33:54.863 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.863 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1c is not active+clean 2024-03-28T23:33:54.863 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.976494+0000', 'last_change': '2024-03-28T23:33:48.976494+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.976494+0000', 'last_undegraded': '2024-03-28T23:33:48.976494+0000', 'last_fullsized': '2024-03-28T23:33:48.976494+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-28T23:33:54.863 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.18 is not active+clean 2024-03-28T23:33:54.864 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.864 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1b is not active+clean 2024-03-28T23:33:54.864 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.973933+0000', 'last_change': '2024-03-28T23:33:48.973933+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.973933+0000', 'last_undegraded': '2024-03-28T23:33:48.973933+0000', 'last_fullsized': '2024-03-28T23:33:48.973933+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:33:54.864 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.19 is not active+clean 2024-03-28T23:33:54.864 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.864 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1a is not active+clean 2024-03-28T23:33:54.864 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.973246+0000', 'last_change': '2024-03-28T23:33:48.973246+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.973246+0000', 'last_undegraded': '2024-03-28T23:33:48.973246+0000', 'last_fullsized': '2024-03-28T23:33:48.973246+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:33:54.864 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1a is not active+clean 2024-03-28T23:33:54.864 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.864 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.19 is not active+clean 2024-03-28T23:33:54.864 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.976300+0000', 'last_change': '2024-03-28T23:33:48.976300+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.976300+0000', 'last_undegraded': '2024-03-28T23:33:48.976300+0000', 'last_fullsized': '2024-03-28T23:33:48.976300+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-28T23:33:54.865 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1b is not active+clean 2024-03-28T23:33:54.865 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.865 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.18 is not active+clean 2024-03-28T23:33:54.865 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.973252+0000', 'last_change': '2024-03-28T23:33:48.973252+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.973252+0000', 'last_undegraded': '2024-03-28T23:33:48.973252+0000', 'last_fullsized': '2024-03-28T23:33:48.973252+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:33:54.865 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.14 is not active+clean 2024-03-28T23:33:54.865 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.865 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.17 is not active+clean 2024-03-28T23:33:54.865 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.979663+0000', 'last_change': '2024-03-28T23:33:48.979663+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.979663+0000', 'last_undegraded': '2024-03-28T23:33:48.979663+0000', 'last_fullsized': '2024-03-28T23:33:48.979663+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:33:54.866 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.15 is not active+clean 2024-03-28T23:33:54.866 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.866 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.16 is not active+clean 2024-03-28T23:33:54.866 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.976398+0000', 'last_change': '2024-03-28T23:33:48.976398+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.976398+0000', 'last_undegraded': '2024-03-28T23:33:48.976398+0000', 'last_fullsized': '2024-03-28T23:33:48.976398+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-28T23:33:54.866 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.16 is not active+clean 2024-03-28T23:33:54.866 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.866 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.15 is not active+clean 2024-03-28T23:33:54.866 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.976142+0000', 'last_change': '2024-03-28T23:33:48.976142+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.976142+0000', 'last_undegraded': '2024-03-28T23:33:48.976142+0000', 'last_fullsized': '2024-03-28T23:33:48.976142+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-28T23:33:54.866 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.17 is not active+clean 2024-03-28T23:33:54.866 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.867 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.14 is not active+clean 2024-03-28T23:33:54.867 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.975388+0000', 'last_change': '2024-03-28T23:33:48.975388+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.975388+0000', 'last_undegraded': '2024-03-28T23:33:48.975388+0000', 'last_fullsized': '2024-03-28T23:33:48.975388+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-28T23:33:54.867 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.10 is not active+clean 2024-03-28T23:33:54.867 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.867 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.13 is not active+clean 2024-03-28T23:33:54.867 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.975171+0000', 'last_change': '2024-03-28T23:33:48.975171+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.975171+0000', 'last_undegraded': '2024-03-28T23:33:48.975171+0000', 'last_fullsized': '2024-03-28T23:33:48.975171+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-28T23:33:54.867 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.11 is not active+clean 2024-03-28T23:33:54.867 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.867 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.12 is not active+clean 2024-03-28T23:33:54.867 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.975112+0000', 'last_change': '2024-03-28T23:33:48.975112+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.975112+0000', 'last_undegraded': '2024-03-28T23:33:48.975112+0000', 'last_fullsized': '2024-03-28T23:33:48.975112+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-28T23:33:54.867 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.12 is not active+clean 2024-03-28T23:33:54.868 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.868 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.11 is not active+clean 2024-03-28T23:33:54.868 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.975532+0000', 'last_change': '2024-03-28T23:33:48.975532+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.975532+0000', 'last_undegraded': '2024-03-28T23:33:48.975532+0000', 'last_fullsized': '2024-03-28T23:33:48.975532+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-28T23:33:54.868 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.c is not active+clean 2024-03-28T23:33:54.868 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.868 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.f is not active+clean 2024-03-28T23:33:54.868 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.975984+0000', 'last_change': '2024-03-28T23:33:48.975984+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.975984+0000', 'last_undegraded': '2024-03-28T23:33:48.975984+0000', 'last_fullsized': '2024-03-28T23:33:48.975984+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-28T23:33:54.868 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.d is not active+clean 2024-03-28T23:33:54.868 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.868 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.e is not active+clean 2024-03-28T23:33:54.869 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.976398+0000', 'last_change': '2024-03-28T23:33:48.976398+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.976398+0000', 'last_undegraded': '2024-03-28T23:33:48.976398+0000', 'last_fullsized': '2024-03-28T23:33:48.976398+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-28T23:33:54.869 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.e is not active+clean 2024-03-28T23:33:54.869 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.869 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.d is not active+clean 2024-03-28T23:33:54.869 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.975786+0000', 'last_change': '2024-03-28T23:33:48.975786+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.975786+0000', 'last_undegraded': '2024-03-28T23:33:48.975786+0000', 'last_fullsized': '2024-03-28T23:33:48.975786+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-28T23:33:54.869 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.f is not active+clean 2024-03-28T23:33:54.869 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.869 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.c is not active+clean 2024-03-28T23:33:54.869 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.975738+0000', 'last_change': '2024-03-28T23:33:48.975738+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.975738+0000', 'last_undegraded': '2024-03-28T23:33:48.975738+0000', 'last_fullsized': '2024-03-28T23:33:48.975738+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-28T23:33:54.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.8 is not active+clean 2024-03-28T23:33:54.870 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.b is not active+clean 2024-03-28T23:33:54.870 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.973245+0000', 'last_change': '2024-03-28T23:33:48.973245+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.973245+0000', 'last_undegraded': '2024-03-28T23:33:48.973245+0000', 'last_fullsized': '2024-03-28T23:33:48.973245+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:33:54.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.9 is not active+clean 2024-03-28T23:33:54.870 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.a is not active+clean 2024-03-28T23:33:54.870 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.976062+0000', 'last_change': '2024-03-28T23:33:48.976062+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.976062+0000', 'last_undegraded': '2024-03-28T23:33:48.976062+0000', 'last_fullsized': '2024-03-28T23:33:48.976062+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-28T23:33:54.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.a is not active+clean 2024-03-28T23:33:54.870 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.9 is not active+clean 2024-03-28T23:33:54.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.975100+0000', 'last_change': '2024-03-28T23:33:48.975100+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.975100+0000', 'last_undegraded': '2024-03-28T23:33:48.975100+0000', 'last_fullsized': '2024-03-28T23:33:48.975100+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-28T23:33:54.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.b is not active+clean 2024-03-28T23:33:54.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.8 is not active+clean 2024-03-28T23:33:54.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.977053+0000', 'last_change': '2024-03-28T23:33:48.977053+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.977053+0000', 'last_undegraded': '2024-03-28T23:33:48.977053+0000', 'last_fullsized': '2024-03-28T23:33:48.977053+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-28T23:33:54.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.4 is not active+clean 2024-03-28T23:33:54.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.7 is not active+clean 2024-03-28T23:33:54.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.973198+0000', 'last_change': '2024-03-28T23:33:48.973198+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.973198+0000', 'last_undegraded': '2024-03-28T23:33:48.973198+0000', 'last_fullsized': '2024-03-28T23:33:48.973198+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:33:54.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.5 is not active+clean 2024-03-28T23:33:54.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.6 is not active+clean 2024-03-28T23:33:54.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.979374+0000', 'last_change': '2024-03-28T23:33:48.979374+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.979374+0000', 'last_undegraded': '2024-03-28T23:33:48.979374+0000', 'last_fullsized': '2024-03-28T23:33:48.979374+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:33:54.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.6 is not active+clean 2024-03-28T23:33:54.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.5 is not active+clean 2024-03-28T23:33:54.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.979329+0000', 'last_change': '2024-03-28T23:33:48.979329+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.979329+0000', 'last_undegraded': '2024-03-28T23:33:48.979329+0000', 'last_fullsized': '2024-03-28T23:33:48.979329+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-28T23:33:54.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.7 is not active+clean 2024-03-28T23:33:54.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.4 is not active+clean 2024-03-28T23:33:54.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.973409+0000', 'last_change': '2024-03-28T23:33:48.973409+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.973409+0000', 'last_undegraded': '2024-03-28T23:33:48.973409+0000', 'last_fullsized': '2024-03-28T23:33:48.973409+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:33:54.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.0 is not active+clean 2024-03-28T23:33:54.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3 is not active+clean 2024-03-28T23:33:54.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.973253+0000', 'last_change': '2024-03-28T23:33:48.973253+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.973253+0000', 'last_undegraded': '2024-03-28T23:33:48.973253+0000', 'last_fullsized': '2024-03-28T23:33:48.973253+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:33:54.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1 is not active+clean 2024-03-28T23:33:54.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2 is not active+clean 2024-03-28T23:33:54.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.973312+0000', 'last_change': '2024-03-28T23:33:48.973312+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.973312+0000', 'last_undegraded': '2024-03-28T23:33:48.973312+0000', 'last_fullsized': '2024-03-28T23:33:48.973312+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-28T23:33:54.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2 is not active+clean 2024-03-28T23:33:54.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1 is not active+clean 2024-03-28T23:33:54.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.973203+0000', 'last_change': '2024-03-28T23:33:48.973203+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.973203+0000', 'last_undegraded': '2024-03-28T23:33:48.973203+0000', 'last_fullsized': '2024-03-28T23:33:48.973203+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-28T23:33:54.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3 is not active+clean 2024-03-28T23:33:54.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.0 is not active+clean 2024-03-28T23:33:54.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.976032+0000', 'last_change': '2024-03-28T23:33:48.976032+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.976032+0000', 'last_undegraded': '2024-03-28T23:33:48.976032+0000', 'last_fullsized': '2024-03-28T23:33:48.976032+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-28T23:33:54.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.13 is not active+clean 2024-03-28T23:33:54.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-28T23:33:49.970798+0000', 'last_change': '2024-03-28T23:33:49.970798+0000', 'last_active': '2024-03-28T23:33:49.970798+0000', 'last_peered': '2024-03-28T23:33:49.970798+0000', 'last_clean': '2024-03-28T23:33:49.970798+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:49.970798+0000', 'last_undegraded': '2024-03-28T23:33:49.970798+0000', 'last_fullsized': '2024-03-28T23:33:49.970798+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:49.970798+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-28T23:33:54.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.10 is not active+clean 2024-03-28T23:33:54.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-03-28T23:33:48.975006+0000', 'last_change': '2024-03-28T23:33:48.975006+0000', 'last_active': '2024-03-28T23:33:48.969017+0000', 'last_peered': '2024-03-28T23:33:48.969017+0000', 'last_clean': '2024-03-28T23:33:48.969017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-28T23:33:48.975006+0000', 'last_undegraded': '2024-03-28T23:33:48.975006+0000', 'last_fullsized': '2024-03-28T23:33:48.975006+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'last_clean_scrub_stamp': '2024-03-28T23:33:48.969017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-28T23:33:54.878 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-28T23:33:54.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.993+0000 7f2005e5e640 1 Processor -- start 2024-03-28T23:33:54.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.993+0000 7f2005e5e640 1 -- start start 2024-03-28T23:33:54.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.994+0000 7f2005e5e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f20000caf60 0x7f20001592e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:54.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.994+0000 7f2005e5e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f20000c8970 0x7f20000c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:54.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.994+0000 7f2005e5e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20000cb4e0 con 0x7f200014f260 2024-03-28T23:33:54.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.994+0000 7f2005e5e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2000158e50 con 0x7f20000caf60 2024-03-28T23:33:54.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.994+0000 7f2005e5e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f20000cb7c0 con 0x7f20000c8970 2024-03-28T23:33:54.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.994+0000 7f1ff7fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f20000caf60 0x7f20001592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:54.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.994+0000 7f1fff7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f200014f260 0x7f2000155c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41490/0 (socket says 172.21.15.161:41490) 2024-03-28T23:33:54.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.994+0000 7f1fff7fe640 1 -- 172.21.15.161:0/417430820 learned_addr learned my addr 172.21.15.161:0/417430820 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:54.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.995+0000 7f1ff7fff640 1 -- 172.21.15.161:0/417430820 >> v2:172.21.15.170:3300/0 conn(0x7f20000c8970 msgr2=0x7f20000c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:54.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.995+0000 7f1ffffff640 1 --2- 172.21.15.161:0/417430820 >> v2:172.21.15.170:3300/0 conn(0x7f20000c8970 0x7f20000c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:54.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.995+0000 7f1ff7fff640 1 --2- 172.21.15.161:0/417430820 >> v2:172.21.15.170:3300/0 conn(0x7f20000c8970 0x7f20000c53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.995+0000 7f1ff7fff640 1 -- 172.21.15.161:0/417430820 >> v1:172.21.15.161:6789/0 conn(0x7f200014f260 legacy=0x7f2000155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:54.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.995+0000 7f1ffeffd640 1 -- 172.21.15.161:0/417430820 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1488575165 0 0) 0x7f20000cb4e0 con 0x7f200014f260 2024-03-28T23:33:54.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.995+0000 7f1ff7fff640 1 -- 172.21.15.161:0/417430820 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2000158800 con 0x7f20000caf60 2024-03-28T23:33:54.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.995+0000 7f1ffffff640 1 --2- 172.21.15.161:0/417430820 >> v2:172.21.15.170:3300/0 conn(0x7f20000c8970 0x7f20000c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:33:54.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.996+0000 7f1ff7fff640 1 --2- 172.21.15.161:0/417430820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20000caf60 0x7f20001592e0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f2000158fd0 tx=0x7f1fe802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff689637591daaaa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:54.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.996+0000 7f1ffeffd640 1 -- 172.21.15.161:0/417430820 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe80315a0 con 0x7f20000caf60 2024-03-28T23:33:54.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.996+0000 7f1ffeffd640 1 -- 172.21.15.161:0/417430820 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fe8037070 con 0x7f20000caf60 2024-03-28T23:33:54.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.996+0000 7f1ffeffd640 1 -- 172.21.15.161:0/417430820 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8031960 con 0x7f20000caf60 2024-03-28T23:33:54.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.996+0000 7f2005e5e640 1 -- 172.21.15.161:0/417430820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20000caf60 msgr2=0x7f20001592e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:54.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.996+0000 7f2005e5e640 1 --2- 172.21.15.161:0/417430820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20000caf60 0x7f20001592e0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f2000158fd0 tx=0x7f1fe802b840 comp rx=0 tx=0).stop 2024-03-28T23:33:54.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.996+0000 7f2005e5e640 1 -- 172.21.15.161:0/417430820 shutdown_connections 2024-03-28T23:33:54.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.996+0000 7f2005e5e640 1 --2- 172.21.15.161:0/417430820 >> v2:172.21.15.170:3300/0 conn(0x7f20000c8970 0x7f20000c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.996+0000 7f2005e5e640 1 --2- 172.21.15.161:0/417430820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20000caf60 0x7f20001592e0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:54.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.996+0000 7f2005e5e640 1 -- 172.21.15.161:0/417430820 >> 172.21.15.161:0/417430820 conn(0x7f2000072ca0 msgr2=0x7f20000730b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:54.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.996+0000 7f2005e5e640 1 -- 172.21.15.161:0/417430820 shutdown_connections 2024-03-28T23:33:54.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.996+0000 7f2005e5e640 1 -- 172.21.15.161:0/417430820 wait complete. 2024-03-28T23:33:54.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.998+0000 7f2005e5e640 1 Processor -- start 2024-03-28T23:33:54.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.998+0000 7f2005e5e640 1 -- start start 2024-03-28T23:33:55.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.999+0000 7f2005e5e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20000c8970 0x7f20001ede00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:55.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.999+0000 7f2005e5e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20000caf60 0x7f20000abcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:55.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.999+0000 7f2005e5e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f200014f260 0x7f20001eafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:55.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.999+0000 7f2005e5e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f20000c8dc0 con 0x7f20000c8970 2024-03-28T23:33:55.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.999+0000 7f2005e5e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f200014f640 con 0x7f200014f260 2024-03-28T23:33:55.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.999+0000 7f2005e5e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f20000c8f40 con 0x7f20000caf60 2024-03-28T23:33:55.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.999+0000 7f1fff7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20000c8970 0x7f20001ede00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:55.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.999+0000 7f1fff7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20000c8970 0x7f20001ede00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60114/0 (socket says 172.21.15.161:60114) 2024-03-28T23:33:55.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:54.999+0000 7f1fff7fe640 1 -- 172.21.15.161:0/3404820412 learned_addr learned my addr 172.21.15.161:0/3404820412 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:55.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.000+0000 7f1ff7fff640 1 --2- 172.21.15.161:0/3404820412 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20000caf60 0x7f20000abcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:55.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.000+0000 7f1ffffff640 1 --2- 172.21.15.161:0/3404820412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f200014f260 0x7f20001eafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:55.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.000+0000 7f1fff7fe640 1 -- 172.21.15.161:0/3404820412 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20000caf60 msgr2=0x7f20000abcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:55.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.000+0000 7f1fff7fe640 1 --2- 172.21.15.161:0/3404820412 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20000caf60 0x7f20000abcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:55.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.000+0000 7f1fff7fe640 1 -- 172.21.15.161:0/3404820412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f200014f260 msgr2=0x7f20001eafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:55.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.000+0000 7f1fff7fe640 1 --2- 172.21.15.161:0/3404820412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f200014f260 0x7f20001eafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:55.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.000+0000 7f1fff7fe640 1 -- 172.21.15.161:0/3404820412 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20001ee500 con 0x7f20000c8970 2024-03-28T23:33:55.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.000+0000 7f1ff7fff640 1 --2- 172.21.15.161:0/3404820412 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20000caf60 0x7f20000abcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:55.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.000+0000 7f1ffffff640 1 --2- 172.21.15.161:0/3404820412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f200014f260 0x7f20001eafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:55.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.000+0000 7f1fff7fe640 1 --2- 172.21.15.161:0/3404820412 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20000c8970 0x7f20001ede00 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f1ff000c430 tx=0x7f1ff000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:55.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.000+0000 7f1ffcff9640 1 -- 172.21.15.161:0/3404820412 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff00074e0 con 0x7f20000c8970 2024-03-28T23:33:55.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.000+0000 7f1ffcff9640 1 -- 172.21.15.161:0/3404820412 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ff00047e0 con 0x7f20000c8970 2024-03-28T23:33:55.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.000+0000 7f2005e5e640 1 -- 172.21.15.161:0/3404820412 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20001ee7e0 con 0x7f20000c8970 2024-03-28T23:33:55.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.000+0000 7f1ffcff9640 1 -- 172.21.15.161:0/3404820412 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0004bd0 con 0x7f20000c8970 2024-03-28T23:33:55.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.000+0000 7f2005e5e640 1 -- 172.21.15.161:0/3404820412 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20001eaaa0 con 0x7f20000c8970 2024-03-28T23:33:55.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.001+0000 7f2005e5e640 1 -- 172.21.15.161:0/3404820412 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20001ea890 con 0x7f20000c8970 2024-03-28T23:33:55.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.001+0000 7f1ffcff9640 1 -- 172.21.15.161:0/3404820412 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1ff0011070 con 0x7f20000c8970 2024-03-28T23:33:55.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.002+0000 7f1ffcff9640 1 --2- 172.21.15.161:0/3404820412 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1fe006b4d0 0x7f1fe006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:55.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.002+0000 7f1ffcff9640 1 -- 172.21.15.161:0/3404820412 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6101+0+0 (secure 0 0 0) 0x7f1ff0089fc0 con 0x7f20000c8970 2024-03-28T23:33:55.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.002+0000 7f1ff7fff640 1 --2- 172.21.15.161:0/3404820412 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1fe006b4d0 0x7f1fe006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:55.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.002+0000 7f1ff7fff640 1 --2- 172.21.15.161:0/3404820412 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1fe006b4d0 0x7f1fe006d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8036000 tx=0x7f1fe803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:55.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.006+0000 7f1ffcff9640 1 -- 172.21.15.161:0/3404820412 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1ff0058d40 con 0x7f20000c8970 2024-03-28T23:33:55.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.249+0000 7f2005e5e640 1 -- 172.21.15.161:0/3404820412 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f200004f550 con 0x7f20000c8970 2024-03-28T23:33:55.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.250+0000 7f1ffcff9640 1 -- 172.21.15.161:0/3404820412 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1529 (secure 0 0 0) 0x7f1ff0058640 con 0x7f20000c8970 2024-03-28T23:33:55.251 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:55.251 INFO:teuthology.orchestra.run.smithi161.stdout:{"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":266,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":109,"num_osds":8,"num_up_osds":8,"osd_up_since":1711668579,"num_in_osds":8,"osd_in_since":1711668571,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":261857280,"bytes_avail":772832256000,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":90,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"a","status":"up:active","gid":7452}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-03-28T23:33:05.349934+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-28T23:33:55.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.252+0000 7f2005e5e640 1 -- 172.21.15.161:0/3404820412 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1fe006b4d0 msgr2=0x7f1fe006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:55.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.252+0000 7f2005e5e640 1 --2- 172.21.15.161:0/3404820412 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1fe006b4d0 0x7f1fe006d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8036000 tx=0x7f1fe803d040 comp rx=0 tx=0).stop 2024-03-28T23:33:55.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.252+0000 7f2005e5e640 1 -- 172.21.15.161:0/3404820412 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20000c8970 msgr2=0x7f20001ede00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:55.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.252+0000 7f2005e5e640 1 --2- 172.21.15.161:0/3404820412 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20000c8970 0x7f20001ede00 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f1ff000c430 tx=0x7f1ff000c8f0 comp rx=0 tx=0).stop 2024-03-28T23:33:55.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.252+0000 7f2005e5e640 1 -- 172.21.15.161:0/3404820412 shutdown_connections 2024-03-28T23:33:55.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.252+0000 7f2005e5e640 1 --2- 172.21.15.161:0/3404820412 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1fe006b4d0 0x7f1fe006d990 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:55.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.252+0000 7f2005e5e640 1 --2- 172.21.15.161:0/3404820412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f200014f260 0x7f20001eafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:55.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.252+0000 7f2005e5e640 1 --2- 172.21.15.161:0/3404820412 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20000caf60 0x7f20000abcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:55.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.252+0000 7f2005e5e640 1 --2- 172.21.15.161:0/3404820412 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20000c8970 0x7f20001ede00 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:55.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.252+0000 7f2005e5e640 1 -- 172.21.15.161:0/3404820412 >> 172.21.15.161:0/3404820412 conn(0x7f2000072ca0 msgr2=0x7f20000c52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:55.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.252+0000 7f2005e5e640 1 -- 172.21.15.161:0/3404820412 shutdown_connections 2024-03-28T23:33:55.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.252+0000 7f2005e5e640 1 -- 172.21.15.161:0/3404820412 wait complete. 2024-03-28T23:33:55.263 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-28T23:33:55.263 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:33:55.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.363+0000 7fcfedf02640 1 Processor -- start 2024-03-28T23:33:55.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.364+0000 7fcfedf02640 1 -- start start 2024-03-28T23:33:55.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.364+0000 7fcfedf02640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fcfe80c5410 0x7fcfe80c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:55.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.364+0000 7fcfedf02640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcfe81553a0 0x7fcfe8155850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:55.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.364+0000 7fcfedf02640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfe8158a30 con 0x7fcfe8152890 2024-03-28T23:33:55.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.364+0000 7fcfedf02640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fcfe80c8480 con 0x7fcfe81553a0 2024-03-28T23:33:55.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.364+0000 7fcfedf02640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fcfe80c5d90 con 0x7fcfe80c5410 2024-03-28T23:33:55.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.364+0000 7fcfe7fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcfe81553a0 0x7fcfe8155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:55.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.364+0000 7fcfe7fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcfe81553a0 0x7fcfe8155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60128/0 (socket says 172.21.15.161:60128) 2024-03-28T23:33:55.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.364+0000 7fcfe7fff640 1 -- 172.21.15.161:0/43082024 learned_addr learned my addr 172.21.15.161:0/43082024 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:55.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.365+0000 7fcfe6ffd640 1 --2- 172.21.15.161:0/43082024 >> v2:172.21.15.170:3300/0 conn(0x7fcfe80c5410 0x7fcfe80c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:55.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.365+0000 7fcfe7fff640 1 -- 172.21.15.161:0/43082024 >> v2:172.21.15.170:3300/0 conn(0x7fcfe80c5410 msgr2=0x7fcfe80c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:55.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.365+0000 7fcfe7fff640 1 --2- 172.21.15.161:0/43082024 >> v2:172.21.15.170:3300/0 conn(0x7fcfe80c5410 0x7fcfe80c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:55.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.365+0000 7fcfe7fff640 1 -- 172.21.15.161:0/43082024 >> v1:172.21.15.170:6789/0 conn(0x7fcfe8152890 legacy=0x7fcfe80c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:55.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.365+0000 7fcfe7fff640 1 -- 172.21.15.161:0/43082024 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfe8158d10 con 0x7fcfe81553a0 2024-03-28T23:33:55.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.365+0000 7fcfe7fff640 1 --2- 172.21.15.161:0/43082024 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfe81553a0 0x7fcfe8155850 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80ad6e0 tx=0x7fcfe002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74504e54b74de320 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:55.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.366+0000 7fcfe67fc640 1 -- 172.21.15.161:0/43082024 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe002ca80 con 0x7fcfe81553a0 2024-03-28T23:33:55.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.366+0000 7fcfe67fc640 1 -- 172.21.15.161:0/43082024 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfe002cc20 con 0x7fcfe81553a0 2024-03-28T23:33:55.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.366+0000 7fcfe67fc640 1 -- 172.21.15.161:0/43082024 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe00343f0 con 0x7fcfe81553a0 2024-03-28T23:33:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.366+0000 7fcfedf02640 1 -- 172.21.15.161:0/43082024 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfe81553a0 msgr2=0x7fcfe8155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.366+0000 7fcfedf02640 1 --2- 172.21.15.161:0/43082024 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfe81553a0 0x7fcfe8155850 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80ad6e0 tx=0x7fcfe002bee0 comp rx=0 tx=0).stop 2024-03-28T23:33:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.366+0000 7fcfedf02640 1 -- 172.21.15.161:0/43082024 shutdown_connections 2024-03-28T23:33:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.366+0000 7fcfedf02640 1 --2- 172.21.15.161:0/43082024 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfe81553a0 0x7fcfe8155850 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.366+0000 7fcfedf02640 1 --2- 172.21.15.161:0/43082024 >> v2:172.21.15.170:3300/0 conn(0x7fcfe80c5410 0x7fcfe80c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.366+0000 7fcfedf02640 1 -- 172.21.15.161:0/43082024 >> 172.21.15.161:0/43082024 conn(0x7fcfe80ac620 msgr2=0x7fcfe80aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.366+0000 7fcfedf02640 1 -- 172.21.15.161:0/43082024 shutdown_connections 2024-03-28T23:33:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.366+0000 7fcfedf02640 1 -- 172.21.15.161:0/43082024 wait complete. 2024-03-28T23:33:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.367+0000 7fcfedf02640 1 Processor -- start 2024-03-28T23:33:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.367+0000 7fcfedf02640 1 -- start start 2024-03-28T23:33:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.368+0000 7fcfedf02640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfe80c5410 0x7fcfe814a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.368+0000 7fcfedf02640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfe8152890 0x7fcfe8147e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:55.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.368+0000 7fcfedf02640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfe81553a0 0x7fcfe8148350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:55.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.368+0000 7fcfedf02640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe81592d0 con 0x7fcfe80c5410 2024-03-28T23:33:55.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.368+0000 7fcfedf02640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe8159640 con 0x7fcfe81553a0 2024-03-28T23:33:55.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.368+0000 7fcfedf02640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fcfe80c90a0 con 0x7fcfe8152890 2024-03-28T23:33:55.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.368+0000 7fcfe77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfe80c5410 0x7fcfe814a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:55.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.368+0000 7fcfe77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfe80c5410 0x7fcfe814a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60140/0 (socket says 172.21.15.161:60140) 2024-03-28T23:33:55.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.368+0000 7fcfe77fe640 1 -- 172.21.15.161:0/2296263330 learned_addr learned my addr 172.21.15.161:0/2296263330 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:55.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.368+0000 7fcfe7fff640 1 --2- 172.21.15.161:0/2296263330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfe81553a0 0x7fcfe8148350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:55.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.368+0000 7fcfe6ffd640 1 --2- 172.21.15.161:0/2296263330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfe8152890 0x7fcfe8147e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:55.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.369+0000 7fcfe77fe640 1 -- 172.21.15.161:0/2296263330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfe8152890 msgr2=0x7fcfe8147e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:55.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.369+0000 7fcfe77fe640 1 --2- 172.21.15.161:0/2296263330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfe8152890 0x7fcfe8147e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:55.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.369+0000 7fcfe77fe640 1 -- 172.21.15.161:0/2296263330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfe81553a0 msgr2=0x7fcfe8148350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:55.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.369+0000 7fcfe77fe640 1 --2- 172.21.15.161:0/2296263330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfe81553a0 0x7fcfe8148350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:55.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.369+0000 7fcfe77fe640 1 -- 172.21.15.161:0/2296263330 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfe80be900 con 0x7fcfe80c5410 2024-03-28T23:33:55.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.369+0000 7fcfe7fff640 1 --2- 172.21.15.161:0/2296263330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfe81553a0 0x7fcfe8148350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:55.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.369+0000 7fcfe77fe640 1 --2- 172.21.15.161:0/2296263330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfe80c5410 0x7fcfe814a100 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800c430 tx=0x7fcfd800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:55.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.369+0000 7fcfcffff640 1 -- 172.21.15.161:0/2296263330 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd80052d0 con 0x7fcfe80c5410 2024-03-28T23:33:55.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.369+0000 7fcfcffff640 1 -- 172.21.15.161:0/2296263330 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfd8005490 con 0x7fcfe80c5410 2024-03-28T23:33:55.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.369+0000 7fcfcffff640 1 -- 172.21.15.161:0/2296263330 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8007260 con 0x7fcfe80c5410 2024-03-28T23:33:55.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.369+0000 7fcfedf02640 1 -- 172.21.15.161:0/2296263330 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfe80bebe0 con 0x7fcfe80c5410 2024-03-28T23:33:55.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.369+0000 7fcfedf02640 1 -- 172.21.15.161:0/2296263330 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfe814a640 con 0x7fcfe80c5410 2024-03-28T23:33:55.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.370+0000 7fcfedf02640 1 -- 172.21.15.161:0/2296263330 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfe80bedb0 con 0x7fcfe80c5410 2024-03-28T23:33:55.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.371+0000 7fcfcffff640 1 -- 172.21.15.161:0/2296263330 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcfd8007400 con 0x7fcfe80c5410 2024-03-28T23:33:55.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.371+0000 7fcfcffff640 1 --2- 172.21.15.161:0/2296263330 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcfd406b4d0 0x7fcfd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:55.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.371+0000 7fcfcffff640 1 -- 172.21.15.161:0/2296263330 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6101+0+0 (secure 0 0 0) 0x7fcfd8089820 con 0x7fcfe80c5410 2024-03-28T23:33:55.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.371+0000 7fcfe6ffd640 1 --2- 172.21.15.161:0/2296263330 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcfd406b4d0 0x7fcfd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:55.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.372+0000 7fcfe6ffd640 1 --2- 172.21.15.161:0/2296263330 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcfd406b4d0 0x7fcfd406d990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8154520 tx=0x7fcfdc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:55.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.379+0000 7fcfcffff640 1 -- 172.21.15.161:0/2296263330 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcfe80bedb0 con 0x7fcfe80c5410 2024-03-28T23:33:55.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.553+0000 7fcfedf02640 1 -- 172.21.15.161:0/2296263330 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcfe8151a50 con 0x7fcfd406b4d0 2024-03-28T23:33:55.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.559+0000 7fcfcffff640 1 -- 172.21.15.161:0/2296263330 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252573 (secure 0 0 0) 0x7fcfe8151a50 con 0x7fcfd406b4d0 2024-03-28T23:33:55.562 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:55.564 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:33:55.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.562+0000 7fcfedf02640 1 -- 172.21.15.161:0/2296263330 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcfd406b4d0 msgr2=0x7fcfd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:55.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.562+0000 7fcfedf02640 1 --2- 172.21.15.161:0/2296263330 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcfd406b4d0 0x7fcfd406d990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8154520 tx=0x7fcfdc008040 comp rx=0 tx=0).stop 2024-03-28T23:33:55.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.562+0000 7fcfedf02640 1 -- 172.21.15.161:0/2296263330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfe80c5410 msgr2=0x7fcfe814a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:55.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.562+0000 7fcfedf02640 1 --2- 172.21.15.161:0/2296263330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfe80c5410 0x7fcfe814a100 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800c430 tx=0x7fcfd800c8f0 comp rx=0 tx=0).stop 2024-03-28T23:33:55.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.562+0000 7fcfedf02640 1 -- 172.21.15.161:0/2296263330 shutdown_connections 2024-03-28T23:33:55.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.562+0000 7fcfedf02640 1 --2- 172.21.15.161:0/2296263330 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcfd406b4d0 0x7fcfd406d990 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:55.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.562+0000 7fcfedf02640 1 --2- 172.21.15.161:0/2296263330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfe81553a0 0x7fcfe8148350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:55.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.562+0000 7fcfedf02640 1 --2- 172.21.15.161:0/2296263330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfe8152890 0x7fcfe8147e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:55.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.562+0000 7fcfedf02640 1 --2- 172.21.15.161:0/2296263330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcfe80c5410 0x7fcfe814a100 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:55.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.562+0000 7fcfedf02640 1 -- 172.21.15.161:0/2296263330 >> 172.21.15.161:0/2296263330 conn(0x7fcfe80ac620 msgr2=0x7fcfe806bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:55.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.563+0000 7fcfedf02640 1 -- 172.21.15.161:0/2296263330 shutdown_connections 2024-03-28T23:33:55.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:55.563+0000 7fcfedf02640 1 -- 172.21.15.161:0/2296263330 wait complete. 2024-03-28T23:33:55.574 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2024-03-28T23:33:55.358815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260288,"kb_used_data":10816,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714432,"statfs":{"total":773094113280,"available":772827578368,"internally_reserved":0,"allocated":11075584,"data_stored":6436876,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5],"upper_bound":8},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001008"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005718+0000","last_change":"2024-03-28T23:33:51.013666+0000","last_active":"2024-03-28T23:33:52.005718+0000","last_peered":"2024-03-28T23:33:52.005718+0000","last_clean":"2024-03-28T23:33:52.005718+0000","last_became_active":"2024-03-28T23:33:51.013355+0000","last_became_peered":"2024-03-28T23:33:51.013355+0000","last_unstale":"2024-03-28T23:33:52.005718+0000","last_undegraded":"2024-03-28T23:33:52.005718+0000","last_fullsized":"2024-03-28T23:33:52.005718+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:32.390009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074441+0000","last_change":"2024-03-28T23:33:51.019220+0000","last_active":"2024-03-28T23:33:53.074441+0000","last_peered":"2024-03-28T23:33:53.074441+0000","last_clean":"2024-03-28T23:33:53.074441+0000","last_became_active":"2024-03-28T23:33:51.018735+0000","last_became_peered":"2024-03-28T23:33:51.018735+0000","last_unstale":"2024-03-28T23:33:53.074441+0000","last_undegraded":"2024-03-28T23:33:53.074441+0000","last_fullsized":"2024-03-28T23:33:53.074441+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:30:37.699564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074731+0000","last_change":"2024-03-28T23:33:51.018423+0000","last_active":"2024-03-28T23:33:53.074731+0000","last_peered":"2024-03-28T23:33:53.074731+0000","last_clean":"2024-03-28T23:33:53.074731+0000","last_became_active":"2024-03-28T23:33:51.015925+0000","last_became_peered":"2024-03-28T23:33:51.015925+0000","last_unstale":"2024-03-28T23:33:53.074731+0000","last_undegraded":"2024-03-28T23:33:53.074731+0000","last_fullsized":"2024-03-28T23:33:53.074731+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:46:08.946536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074663+0000","last_change":"2024-03-28T23:33:51.014367+0000","last_active":"2024-03-28T23:33:53.074663+0000","last_peered":"2024-03-28T23:33:53.074663+0000","last_clean":"2024-03-28T23:33:53.074663+0000","last_became_active":"2024-03-28T23:33:51.013995+0000","last_became_peered":"2024-03-28T23:33:51.013995+0000","last_unstale":"2024-03-28T23:33:53.074663+0000","last_undegraded":"2024-03-28T23:33:53.074663+0000","last_fullsized":"2024-03-28T23:33:53.074663+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:59:52.655622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005564+0000","last_change":"2024-03-28T23:33:51.009473+0000","last_active":"2024-03-28T23:33:52.005564+0000","last_peered":"2024-03-28T23:33:52.005564+0000","last_clean":"2024-03-28T23:33:52.005564+0000","last_became_active":"2024-03-28T23:33:51.009179+0000","last_became_peered":"2024-03-28T23:33:51.009179+0000","last_unstale":"2024-03-28T23:33:52.005564+0000","last_undegraded":"2024-03-28T23:33:52.005564+0000","last_fullsized":"2024-03-28T23:33:52.005564+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:37.206474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.018841+0000","last_change":"2024-03-28T23:33:51.030849+0000","last_active":"2024-03-28T23:33:52.018841+0000","last_peered":"2024-03-28T23:33:52.018841+0000","last_clean":"2024-03-28T23:33:52.018841+0000","last_became_active":"2024-03-28T23:33:51.030557+0000","last_became_peered":"2024-03-28T23:33:51.030557+0000","last_unstale":"2024-03-28T23:33:52.018841+0000","last_undegraded":"2024-03-28T23:33:52.018841+0000","last_fullsized":"2024-03-28T23:33:52.018841+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:04:45.819608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.018622+0000","last_change":"2024-03-28T23:33:51.017932+0000","last_active":"2024-03-28T23:33:52.018622+0000","last_peered":"2024-03-28T23:33:52.018622+0000","last_clean":"2024-03-28T23:33:52.018622+0000","last_became_active":"2024-03-28T23:33:51.017660+0000","last_became_peered":"2024-03-28T23:33:51.017660+0000","last_unstale":"2024-03-28T23:33:52.018622+0000","last_undegraded":"2024-03-28T23:33:52.018622+0000","last_fullsized":"2024-03-28T23:33:52.018622+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:27:37.193802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074077+0000","last_change":"2024-03-28T23:33:51.024956+0000","last_active":"2024-03-28T23:33:53.074077+0000","last_peered":"2024-03-28T23:33:53.074077+0000","last_clean":"2024-03-28T23:33:53.074077+0000","last_became_active":"2024-03-28T23:33:51.019702+0000","last_became_peered":"2024-03-28T23:33:51.019702+0000","last_unstale":"2024-03-28T23:33:53.074077+0000","last_undegraded":"2024-03-28T23:33:53.074077+0000","last_fullsized":"2024-03-28T23:33:53.074077+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:15:35.186625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.075042+0000","last_change":"2024-03-28T23:33:51.025952+0000","last_active":"2024-03-28T23:33:53.075042+0000","last_peered":"2024-03-28T23:33:53.075042+0000","last_clean":"2024-03-28T23:33:53.075042+0000","last_became_active":"2024-03-28T23:33:51.025674+0000","last_became_peered":"2024-03-28T23:33:51.025674+0000","last_unstale":"2024-03-28T23:33:53.075042+0000","last_undegraded":"2024-03-28T23:33:53.075042+0000","last_fullsized":"2024-03-28T23:33:53.075042+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:27:03.382059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.003956+0000","last_change":"2024-03-28T23:33:51.017655+0000","last_active":"2024-03-28T23:33:52.003956+0000","last_peered":"2024-03-28T23:33:52.003956+0000","last_clean":"2024-03-28T23:33:52.003956+0000","last_became_active":"2024-03-28T23:33:51.017399+0000","last_became_peered":"2024-03-28T23:33:51.017399+0000","last_unstale":"2024-03-28T23:33:52.003956+0000","last_undegraded":"2024-03-28T23:33:52.003956+0000","last_fullsized":"2024-03-28T23:33:52.003956+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:24:29.812480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005011+0000","last_change":"2024-03-28T23:33:51.015409+0000","last_active":"2024-03-28T23:33:52.005011+0000","last_peered":"2024-03-28T23:33:52.005011+0000","last_clean":"2024-03-28T23:33:52.005011+0000","last_became_active":"2024-03-28T23:33:51.015114+0000","last_became_peered":"2024-03-28T23:33:51.015114+0000","last_unstale":"2024-03-28T23:33:52.005011+0000","last_undegraded":"2024-03-28T23:33:52.005011+0000","last_fullsized":"2024-03-28T23:33:52.005011+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:36:06.301567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.019071+0000","last_change":"2024-03-28T23:33:51.019199+0000","last_active":"2024-03-28T23:33:52.019071+0000","last_peered":"2024-03-28T23:33:52.019071+0000","last_clean":"2024-03-28T23:33:52.019071+0000","last_became_active":"2024-03-28T23:33:51.018737+0000","last_became_peered":"2024-03-28T23:33:51.018737+0000","last_unstale":"2024-03-28T23:33:52.019071+0000","last_undegraded":"2024-03-28T23:33:52.019071+0000","last_fullsized":"2024-03-28T23:33:52.019071+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:07:26.080830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.004269+0000","last_change":"2024-03-28T23:33:51.008289+0000","last_active":"2024-03-28T23:33:52.004269+0000","last_peered":"2024-03-28T23:33:52.004269+0000","last_clean":"2024-03-28T23:33:52.004269+0000","last_became_active":"2024-03-28T23:33:51.008018+0000","last_became_peered":"2024-03-28T23:33:51.008018+0000","last_unstale":"2024-03-28T23:33:52.004269+0000","last_undegraded":"2024-03-28T23:33:52.004269+0000","last_fullsized":"2024-03-28T23:33:52.004269+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:32.390009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074761+0000","last_change":"2024-03-28T23:33:51.020584+0000","last_active":"2024-03-28T23:33:53.074761+0000","last_peered":"2024-03-28T23:33:53.074761+0000","last_clean":"2024-03-28T23:33:53.074761+0000","last_became_active":"2024-03-28T23:33:51.020321+0000","last_became_peered":"2024-03-28T23:33:51.020321+0000","last_unstale":"2024-03-28T23:33:53.074761+0000","last_undegraded":"2024-03-28T23:33:53.074761+0000","last_fullsized":"2024-03-28T23:33:53.074761+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:26.450839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005300+0000","last_change":"2024-03-28T23:33:51.014905+0000","last_active":"2024-03-28T23:33:52.005300+0000","last_peered":"2024-03-28T23:33:52.005300+0000","last_clean":"2024-03-28T23:33:52.005300+0000","last_became_active":"2024-03-28T23:33:51.014638+0000","last_became_peered":"2024-03-28T23:33:51.014638+0000","last_unstale":"2024-03-28T23:33:52.005300+0000","last_undegraded":"2024-03-28T23:33:52.005300+0000","last_fullsized":"2024-03-28T23:33:52.005300+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:24:29.812480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.018588+0000","last_change":"2024-03-28T23:33:51.017765+0000","last_active":"2024-03-28T23:33:52.018588+0000","last_peered":"2024-03-28T23:33:52.018588+0000","last_clean":"2024-03-28T23:33:52.018588+0000","last_became_active":"2024-03-28T23:33:51.017500+0000","last_became_peered":"2024-03-28T23:33:51.017500+0000","last_unstale":"2024-03-28T23:33:52.018588+0000","last_undegraded":"2024-03-28T23:33:52.018588+0000","last_fullsized":"2024-03-28T23:33:52.018588+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:43.384007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.004038+0000","last_change":"2024-03-28T23:33:51.011874+0000","last_active":"2024-03-28T23:33:52.004038+0000","last_peered":"2024-03-28T23:33:52.004038+0000","last_clean":"2024-03-28T23:33:52.004038+0000","last_became_active":"2024-03-28T23:33:51.011590+0000","last_became_peered":"2024-03-28T23:33:51.011590+0000","last_unstale":"2024-03-28T23:33:52.004038+0000","last_undegraded":"2024-03-28T23:33:52.004038+0000","last_fullsized":"2024-03-28T23:33:52.004038+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:20:01.738504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074424+0000","last_change":"2024-03-28T23:33:51.019461+0000","last_active":"2024-03-28T23:33:53.074424+0000","last_peered":"2024-03-28T23:33:53.074424+0000","last_clean":"2024-03-28T23:33:53.074424+0000","last_became_active":"2024-03-28T23:33:51.018352+0000","last_became_peered":"2024-03-28T23:33:51.018352+0000","last_unstale":"2024-03-28T23:33:53.074424+0000","last_undegraded":"2024-03-28T23:33:53.074424+0000","last_fullsized":"2024-03-28T23:33:53.074424+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:56:05.822381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005614+0000","last_change":"2024-03-28T23:33:51.018848+0000","last_active":"2024-03-28T23:33:52.005614+0000","last_peered":"2024-03-28T23:33:52.005614+0000","last_clean":"2024-03-28T23:33:52.005614+0000","last_became_active":"2024-03-28T23:33:51.018598+0000","last_became_peered":"2024-03-28T23:33:51.018598+0000","last_unstale":"2024-03-28T23:33:52.005614+0000","last_undegraded":"2024-03-28T23:33:52.005614+0000","last_fullsized":"2024-03-28T23:33:52.005614+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:21:03.177601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074352+0000","last_change":"2024-03-28T23:33:51.015079+0000","last_active":"2024-03-28T23:33:53.074352+0000","last_peered":"2024-03-28T23:33:53.074352+0000","last_clean":"2024-03-28T23:33:53.074352+0000","last_became_active":"2024-03-28T23:33:51.014426+0000","last_became_peered":"2024-03-28T23:33:51.014426+0000","last_unstale":"2024-03-28T23:33:53.074352+0000","last_undegraded":"2024-03-28T23:33:53.074352+0000","last_fullsized":"2024-03-28T23:33:53.074352+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:49.229683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074632+0000","last_change":"2024-03-28T23:33:51.020292+0000","last_active":"2024-03-28T23:33:53.074632+0000","last_peered":"2024-03-28T23:33:53.074632+0000","last_clean":"2024-03-28T23:33:53.074632+0000","last_became_active":"2024-03-28T23:33:51.018341+0000","last_became_peered":"2024-03-28T23:33:51.018341+0000","last_unstale":"2024-03-28T23:33:53.074632+0000","last_undegraded":"2024-03-28T23:33:53.074632+0000","last_fullsized":"2024-03-28T23:33:53.074632+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:16:18.521824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005189+0000","last_change":"2024-03-28T23:33:51.025449+0000","last_active":"2024-03-28T23:33:52.005189+0000","last_peered":"2024-03-28T23:33:52.005189+0000","last_clean":"2024-03-28T23:33:52.005189+0000","last_became_active":"2024-03-28T23:33:51.025176+0000","last_became_peered":"2024-03-28T23:33:51.025176+0000","last_unstale":"2024-03-28T23:33:52.005189+0000","last_undegraded":"2024-03-28T23:33:52.005189+0000","last_fullsized":"2024-03-28T23:33:52.005189+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:04.935939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074570+0000","last_change":"2024-03-28T23:33:51.026442+0000","last_active":"2024-03-28T23:33:53.074570+0000","last_peered":"2024-03-28T23:33:53.074570+0000","last_clean":"2024-03-28T23:33:53.074570+0000","last_became_active":"2024-03-28T23:33:51.026161+0000","last_became_peered":"2024-03-28T23:33:51.026161+0000","last_unstale":"2024-03-28T23:33:53.074570+0000","last_undegraded":"2024-03-28T23:33:53.074570+0000","last_fullsized":"2024-03-28T23:33:53.074570+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:39:40.687068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.073966+0000","last_change":"2024-03-28T23:33:51.010612+0000","last_active":"2024-03-28T23:33:53.073966+0000","last_peered":"2024-03-28T23:33:53.073966+0000","last_clean":"2024-03-28T23:33:53.073966+0000","last_became_active":"2024-03-28T23:33:51.010354+0000","last_became_peered":"2024-03-28T23:33:51.010354+0000","last_unstale":"2024-03-28T23:33:53.073966+0000","last_undegraded":"2024-03-28T23:33:53.073966+0000","last_fullsized":"2024-03-28T23:33:53.073966+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:24:29.812480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074183+0000","last_change":"2024-03-28T23:33:51.009461+0000","last_active":"2024-03-28T23:33:53.074183+0000","last_peered":"2024-03-28T23:33:53.074183+0000","last_clean":"2024-03-28T23:33:53.074183+0000","last_became_active":"2024-03-28T23:33:51.009177+0000","last_became_peered":"2024-03-28T23:33:51.009177+0000","last_unstale":"2024-03-28T23:33:53.074183+0000","last_undegraded":"2024-03-28T23:33:53.074183+0000","last_fullsized":"2024-03-28T23:33:53.074183+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:16:18.521824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074284+0000","last_change":"2024-03-28T23:33:51.008394+0000","last_active":"2024-03-28T23:33:53.074284+0000","last_peered":"2024-03-28T23:33:53.074284+0000","last_clean":"2024-03-28T23:33:53.074284+0000","last_became_active":"2024-03-28T23:33:51.008129+0000","last_became_peered":"2024-03-28T23:33:51.008129+0000","last_unstale":"2024-03-28T23:33:53.074284+0000","last_undegraded":"2024-03-28T23:33:53.074284+0000","last_fullsized":"2024-03-28T23:33:53.074284+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:16.800756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005679+0000","last_change":"2024-03-28T23:33:51.013537+0000","last_active":"2024-03-28T23:33:52.005679+0000","last_peered":"2024-03-28T23:33:52.005679+0000","last_clean":"2024-03-28T23:33:52.005679+0000","last_became_active":"2024-03-28T23:33:51.013276+0000","last_became_peered":"2024-03-28T23:33:51.013276+0000","last_unstale":"2024-03-28T23:33:52.005679+0000","last_undegraded":"2024-03-28T23:33:52.005679+0000","last_fullsized":"2024-03-28T23:33:52.005679+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:13.274788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.019130+0000","last_change":"2024-03-28T23:33:51.021129+0000","last_active":"2024-03-28T23:33:52.019130+0000","last_peered":"2024-03-28T23:33:52.019130+0000","last_clean":"2024-03-28T23:33:52.019130+0000","last_became_active":"2024-03-28T23:33:51.020837+0000","last_became_peered":"2024-03-28T23:33:51.020837+0000","last_unstale":"2024-03-28T23:33:52.019130+0000","last_undegraded":"2024-03-28T23:33:52.019130+0000","last_fullsized":"2024-03-28T23:33:52.019130+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:21.360493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074708+0000","last_change":"2024-03-28T23:33:51.019890+0000","last_active":"2024-03-28T23:33:53.074708+0000","last_peered":"2024-03-28T23:33:53.074708+0000","last_clean":"2024-03-28T23:33:53.074708+0000","last_became_active":"2024-03-28T23:33:51.019443+0000","last_became_peered":"2024-03-28T23:33:51.019443+0000","last_unstale":"2024-03-28T23:33:53.074708+0000","last_undegraded":"2024-03-28T23:33:53.074708+0000","last_fullsized":"2024-03-28T23:33:53.074708+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:37.146490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074891+0000","last_change":"2024-03-28T23:33:51.023744+0000","last_active":"2024-03-28T23:33:53.074891+0000","last_peered":"2024-03-28T23:33:53.074891+0000","last_clean":"2024-03-28T23:33:53.074891+0000","last_became_active":"2024-03-28T23:33:51.023477+0000","last_became_peered":"2024-03-28T23:33:51.023477+0000","last_unstale":"2024-03-28T23:33:53.074891+0000","last_undegraded":"2024-03-28T23:33:53.074891+0000","last_fullsized":"2024-03-28T23:33:53.074891+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:34.632115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074051+0000","last_change":"2024-03-28T23:33:51.011502+0000","last_active":"2024-03-28T23:33:53.074051+0000","last_peered":"2024-03-28T23:33:53.074051+0000","last_clean":"2024-03-28T23:33:53.074051+0000","last_became_active":"2024-03-28T23:33:51.011155+0000","last_became_peered":"2024-03-28T23:33:51.011155+0000","last_unstale":"2024-03-28T23:33:53.074051+0000","last_undegraded":"2024-03-28T23:33:53.074051+0000","last_fullsized":"2024-03-28T23:33:53.074051+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:37.206474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.073793+0000","last_change":"2024-03-28T23:33:51.011257+0000","last_active":"2024-03-28T23:33:53.073793+0000","last_peered":"2024-03-28T23:33:53.073793+0000","last_clean":"2024-03-28T23:33:53.073793+0000","last_became_active":"2024-03-28T23:33:51.010958+0000","last_became_peered":"2024-03-28T23:33:51.010958+0000","last_unstale":"2024-03-28T23:33:53.073793+0000","last_undegraded":"2024-03-28T23:33:53.073793+0000","last_fullsized":"2024-03-28T23:33:53.073793+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:04.935939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.004103+0000","last_change":"2024-03-28T23:33:51.021643+0000","last_active":"2024-03-28T23:33:52.004103+0000","last_peered":"2024-03-28T23:33:52.004103+0000","last_clean":"2024-03-28T23:33:52.004103+0000","last_became_active":"2024-03-28T23:33:51.021384+0000","last_became_peered":"2024-03-28T23:33:51.021384+0000","last_unstale":"2024-03-28T23:33:52.004103+0000","last_undegraded":"2024-03-28T23:33:52.004103+0000","last_fullsized":"2024-03-28T23:33:52.004103+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:37.206474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"108'4","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005258+0000","last_change":"2024-03-28T23:33:49.985988+0000","last_active":"2024-03-28T23:33:52.005258+0000","last_peered":"2024-03-28T23:33:52.005258+0000","last_clean":"2024-03-28T23:33:52.005258+0000","last_became_active":"2024-03-28T23:33:49.985665+0000","last_became_peered":"2024-03-28T23:33:49.985665+0000","last_unstale":"2024-03-28T23:33:52.005258+0000","last_undegraded":"2024-03-28T23:33:52.005258+0000","last_fullsized":"2024-03-28T23:33:52.005258+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:40.272936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.018723+0000","last_change":"2024-03-28T23:33:51.030837+0000","last_active":"2024-03-28T23:33:52.018723+0000","last_peered":"2024-03-28T23:33:52.018723+0000","last_clean":"2024-03-28T23:33:52.018723+0000","last_became_active":"2024-03-28T23:33:51.030546+0000","last_became_peered":"2024-03-28T23:33:51.030546+0000","last_unstale":"2024-03-28T23:33:52.018723+0000","last_undegraded":"2024-03-28T23:33:52.018723+0000","last_fullsized":"2024-03-28T23:33:52.018723+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:03:02.359745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.003781+0000","last_change":"2024-03-28T23:33:49.987198+0000","last_active":"2024-03-28T23:33:52.003781+0000","last_peered":"2024-03-28T23:33:52.003781+0000","last_clean":"2024-03-28T23:33:52.003781+0000","last_became_active":"2024-03-28T23:33:49.986675+0000","last_became_peered":"2024-03-28T23:33:49.986675+0000","last_unstale":"2024-03-28T23:33:52.003781+0000","last_undegraded":"2024-03-28T23:33:52.003781+0000","last_fullsized":"2024-03-28T23:33:52.003781+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:10:54.386639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074118+0000","last_change":"2024-03-28T23:33:51.016090+0000","last_active":"2024-03-28T23:33:53.074118+0000","last_peered":"2024-03-28T23:33:53.074118+0000","last_clean":"2024-03-28T23:33:53.074118+0000","last_became_active":"2024-03-28T23:33:51.015781+0000","last_became_peered":"2024-03-28T23:33:51.015781+0000","last_unstale":"2024-03-28T23:33:53.074118+0000","last_undegraded":"2024-03-28T23:33:53.074118+0000","last_fullsized":"2024-03-28T23:33:53.074118+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:32.390009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074738+0000","last_change":"2024-03-28T23:33:49.992114+0000","last_active":"2024-03-28T23:33:53.074738+0000","last_peered":"2024-03-28T23:33:53.074738+0000","last_clean":"2024-03-28T23:33:53.074738+0000","last_became_active":"2024-03-28T23:33:49.991817+0000","last_became_peered":"2024-03-28T23:33:49.991817+0000","last_unstale":"2024-03-28T23:33:53.074738+0000","last_undegraded":"2024-03-28T23:33:53.074738+0000","last_fullsized":"2024-03-28T23:33:53.074738+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:10:24.895984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.075039+0000","last_change":"2024-03-28T23:33:51.017959+0000","last_active":"2024-03-28T23:33:53.075039+0000","last_peered":"2024-03-28T23:33:53.075039+0000","last_clean":"2024-03-28T23:33:53.075039+0000","last_became_active":"2024-03-28T23:33:51.017699+0000","last_became_peered":"2024-03-28T23:33:51.017699+0000","last_unstale":"2024-03-28T23:33:53.075039+0000","last_undegraded":"2024-03-28T23:33:53.075039+0000","last_fullsized":"2024-03-28T23:33:53.075039+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:38:29.769002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.004091+0000","last_change":"2024-03-28T23:33:49.985075+0000","last_active":"2024-03-28T23:33:52.004091+0000","last_peered":"2024-03-28T23:33:52.004091+0000","last_clean":"2024-03-28T23:33:52.004091+0000","last_became_active":"2024-03-28T23:33:49.984788+0000","last_became_peered":"2024-03-28T23:33:49.984788+0000","last_unstale":"2024-03-28T23:33:52.004091+0000","last_undegraded":"2024-03-28T23:33:52.004091+0000","last_fullsized":"2024-03-28T23:33:52.004091+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:49:01.660610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.003684+0000","last_change":"2024-03-28T23:33:51.009658+0000","last_active":"2024-03-28T23:33:52.003684+0000","last_peered":"2024-03-28T23:33:52.003684+0000","last_clean":"2024-03-28T23:33:52.003684+0000","last_became_active":"2024-03-28T23:33:51.009400+0000","last_became_peered":"2024-03-28T23:33:51.009400+0000","last_unstale":"2024-03-28T23:33:52.003684+0000","last_undegraded":"2024-03-28T23:33:52.003684+0000","last_fullsized":"2024-03-28T23:33:52.003684+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:51:09.523264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005426+0000","last_change":"2024-03-28T23:33:49.984472+0000","last_active":"2024-03-28T23:33:52.005426+0000","last_peered":"2024-03-28T23:33:52.005426+0000","last_clean":"2024-03-28T23:33:52.005426+0000","last_became_active":"2024-03-28T23:33:49.984161+0000","last_became_peered":"2024-03-28T23:33:49.984161+0000","last_unstale":"2024-03-28T23:33:52.005426+0000","last_undegraded":"2024-03-28T23:33:52.005426+0000","last_fullsized":"2024-03-28T23:33:52.005426+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:52.093172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005490+0000","last_change":"2024-03-28T23:33:51.012950+0000","last_active":"2024-03-28T23:33:52.005490+0000","last_peered":"2024-03-28T23:33:52.005490+0000","last_clean":"2024-03-28T23:33:52.005490+0000","last_became_active":"2024-03-28T23:33:51.012660+0000","last_became_peered":"2024-03-28T23:33:51.012660+0000","last_unstale":"2024-03-28T23:33:52.005490+0000","last_undegraded":"2024-03-28T23:33:52.005490+0000","last_fullsized":"2024-03-28T23:33:52.005490+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:20:01.738504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.018771+0000","last_change":"2024-03-28T23:33:49.997011+0000","last_active":"2024-03-28T23:33:52.018771+0000","last_peered":"2024-03-28T23:33:52.018771+0000","last_clean":"2024-03-28T23:33:52.018771+0000","last_became_active":"2024-03-28T23:33:49.996748+0000","last_became_peered":"2024-03-28T23:33:49.996748+0000","last_unstale":"2024-03-28T23:33:52.018771+0000","last_undegraded":"2024-03-28T23:33:52.018771+0000","last_fullsized":"2024-03-28T23:33:52.018771+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:29:19.407500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.075006+0000","last_change":"2024-03-28T23:33:51.015284+0000","last_active":"2024-03-28T23:33:53.075006+0000","last_peered":"2024-03-28T23:33:53.075006+0000","last_clean":"2024-03-28T23:33:53.075006+0000","last_became_active":"2024-03-28T23:33:51.015003+0000","last_became_peered":"2024-03-28T23:33:51.015003+0000","last_unstale":"2024-03-28T23:33:53.075006+0000","last_undegraded":"2024-03-28T23:33:53.075006+0000","last_fullsized":"2024-03-28T23:33:53.075006+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:11.329576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.073806+0000","last_change":"2024-03-28T23:33:49.988293+0000","last_active":"2024-03-28T23:33:53.073806+0000","last_peered":"2024-03-28T23:33:53.073806+0000","last_clean":"2024-03-28T23:33:53.073806+0000","last_became_active":"2024-03-28T23:33:49.987865+0000","last_became_peered":"2024-03-28T23:33:49.987865+0000","last_unstale":"2024-03-28T23:33:53.073806+0000","last_undegraded":"2024-03-28T23:33:53.073806+0000","last_fullsized":"2024-03-28T23:33:53.073806+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:21:02.175819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.003834+0000","last_change":"2024-03-28T23:33:51.022417+0000","last_active":"2024-03-28T23:33:52.003834+0000","last_peered":"2024-03-28T23:33:52.003834+0000","last_clean":"2024-03-28T23:33:52.003834+0000","last_became_active":"2024-03-28T23:33:51.022177+0000","last_became_peered":"2024-03-28T23:33:51.022177+0000","last_unstale":"2024-03-28T23:33:52.003834+0000","last_undegraded":"2024-03-28T23:33:52.003834+0000","last_fullsized":"2024-03-28T23:33:52.003834+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:04.935939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.018717+0000","last_change":"2024-03-28T23:33:49.992491+0000","last_active":"2024-03-28T23:33:52.018717+0000","last_peered":"2024-03-28T23:33:52.018717+0000","last_clean":"2024-03-28T23:33:52.018717+0000","last_became_active":"2024-03-28T23:33:49.992224+0000","last_became_peered":"2024-03-28T23:33:49.992224+0000","last_unstale":"2024-03-28T23:33:52.018717+0000","last_undegraded":"2024-03-28T23:33:52.018717+0000","last_fullsized":"2024-03-28T23:33:52.018717+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:38.377566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.004894+0000","last_change":"2024-03-28T23:33:51.014515+0000","last_active":"2024-03-28T23:33:52.004894+0000","last_peered":"2024-03-28T23:33:52.004894+0000","last_clean":"2024-03-28T23:33:52.004894+0000","last_became_active":"2024-03-28T23:33:51.014168+0000","last_became_peered":"2024-03-28T23:33:51.014168+0000","last_unstale":"2024-03-28T23:33:52.004894+0000","last_undegraded":"2024-03-28T23:33:52.004894+0000","last_fullsized":"2024-03-28T23:33:52.004894+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:49:02.662391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005374+0000","last_change":"2024-03-28T23:33:49.997600+0000","last_active":"2024-03-28T23:33:52.005374+0000","last_peered":"2024-03-28T23:33:52.005374+0000","last_clean":"2024-03-28T23:33:52.005374+0000","last_became_active":"2024-03-28T23:33:49.997288+0000","last_became_peered":"2024-03-28T23:33:49.997288+0000","last_unstale":"2024-03-28T23:33:52.005374+0000","last_undegraded":"2024-03-28T23:33:52.005374+0000","last_fullsized":"2024-03-28T23:33:52.005374+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:00:59.695434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074115+0000","last_change":"2024-03-28T23:33:51.025385+0000","last_active":"2024-03-28T23:33:53.074115+0000","last_peered":"2024-03-28T23:33:53.074115+0000","last_clean":"2024-03-28T23:33:53.074115+0000","last_became_active":"2024-03-28T23:33:51.025059+0000","last_became_peered":"2024-03-28T23:33:51.025059+0000","last_unstale":"2024-03-28T23:33:53.074115+0000","last_undegraded":"2024-03-28T23:33:53.074115+0000","last_fullsized":"2024-03-28T23:33:53.074115+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:39:40.687068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.073983+0000","last_change":"2024-03-28T23:33:49.989460+0000","last_active":"2024-03-28T23:33:53.073983+0000","last_peered":"2024-03-28T23:33:53.073983+0000","last_clean":"2024-03-28T23:33:53.073983+0000","last_became_active":"2024-03-28T23:33:49.989123+0000","last_became_peered":"2024-03-28T23:33:49.989123+0000","last_unstale":"2024-03-28T23:33:53.073983+0000","last_undegraded":"2024-03-28T23:33:53.073983+0000","last_fullsized":"2024-03-28T23:33:53.073983+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:14:19.604793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074547+0000","last_change":"2024-03-28T23:33:51.016203+0000","last_active":"2024-03-28T23:33:53.074547+0000","last_peered":"2024-03-28T23:33:53.074547+0000","last_clean":"2024-03-28T23:33:53.074547+0000","last_became_active":"2024-03-28T23:33:51.015936+0000","last_became_peered":"2024-03-28T23:33:51.015936+0000","last_unstale":"2024-03-28T23:33:53.074547+0000","last_undegraded":"2024-03-28T23:33:53.074547+0000","last_fullsized":"2024-03-28T23:33:53.074547+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:45:44.498188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"108'3","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.073981+0000","last_change":"2024-03-28T23:33:49.990570+0000","last_active":"2024-03-28T23:33:53.073981+0000","last_peered":"2024-03-28T23:33:53.073981+0000","last_clean":"2024-03-28T23:33:53.073981+0000","last_became_active":"2024-03-28T23:33:49.987820+0000","last_became_peered":"2024-03-28T23:33:49.987820+0000","last_unstale":"2024-03-28T23:33:53.073981+0000","last_undegraded":"2024-03-28T23:33:53.073981+0000","last_fullsized":"2024-03-28T23:33:53.073981+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:12.273007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074240+0000","last_change":"2024-03-28T23:33:51.012003+0000","last_active":"2024-03-28T23:33:53.074240+0000","last_peered":"2024-03-28T23:33:53.074240+0000","last_clean":"2024-03-28T23:33:53.074240+0000","last_became_active":"2024-03-28T23:33:51.011736+0000","last_became_peered":"2024-03-28T23:33:51.011736+0000","last_unstale":"2024-03-28T23:33:53.074240+0000","last_undegraded":"2024-03-28T23:33:53.074240+0000","last_fullsized":"2024-03-28T23:33:53.074240+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:59:52.655622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"108'6","reported_seq":20,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074912+0000","last_change":"2024-03-28T23:33:49.990203+0000","last_active":"2024-03-28T23:33:53.074912+0000","last_peered":"2024-03-28T23:33:53.074912+0000","last_clean":"2024-03-28T23:33:53.074912+0000","last_became_active":"2024-03-28T23:33:49.989962+0000","last_became_peered":"2024-03-28T23:33:49.989962+0000","last_unstale":"2024-03-28T23:33:53.074912+0000","last_undegraded":"2024-03-28T23:33:53.074912+0000","last_fullsized":"2024-03-28T23:33:53.074912+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:20:00.736722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005141+0000","last_change":"2024-03-28T23:33:51.019810+0000","last_active":"2024-03-28T23:33:52.005141+0000","last_peered":"2024-03-28T23:33:52.005141+0000","last_clean":"2024-03-28T23:33:52.005141+0000","last_became_active":"2024-03-28T23:33:51.019552+0000","last_became_peered":"2024-03-28T23:33:51.019552+0000","last_unstale":"2024-03-28T23:33:52.005141+0000","last_undegraded":"2024-03-28T23:33:52.005141+0000","last_fullsized":"2024-03-28T23:33:52.005141+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:14:20.606574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074741+0000","last_change":"2024-03-28T23:33:49.984539+0000","last_active":"2024-03-28T23:33:53.074741+0000","last_peered":"2024-03-28T23:33:53.074741+0000","last_clean":"2024-03-28T23:33:53.074741+0000","last_became_active":"2024-03-28T23:33:49.982619+0000","last_became_peered":"2024-03-28T23:33:49.982619+0000","last_unstale":"2024-03-28T23:33:53.074741+0000","last_undegraded":"2024-03-28T23:33:53.074741+0000","last_fullsized":"2024-03-28T23:33:53.074741+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:26:16.906247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.003692+0000","last_change":"2024-03-28T23:33:51.014751+0000","last_active":"2024-03-28T23:33:52.003692+0000","last_peered":"2024-03-28T23:33:52.003692+0000","last_clean":"2024-03-28T23:33:52.003692+0000","last_became_active":"2024-03-28T23:33:51.014491+0000","last_became_peered":"2024-03-28T23:33:51.014491+0000","last_unstale":"2024-03-28T23:33:52.003692+0000","last_undegraded":"2024-03-28T23:33:52.003692+0000","last_fullsized":"2024-03-28T23:33:52.003692+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:02:49.240053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.075040+0000","last_change":"2024-03-28T23:33:49.991532+0000","last_active":"2024-03-28T23:33:53.075040+0000","last_peered":"2024-03-28T23:33:53.075040+0000","last_clean":"2024-03-28T23:33:53.075040+0000","last_became_active":"2024-03-28T23:33:49.991276+0000","last_became_peered":"2024-03-28T23:33:49.991276+0000","last_unstale":"2024-03-28T23:33:53.075040+0000","last_undegraded":"2024-03-28T23:33:53.075040+0000","last_fullsized":"2024-03-28T23:33:53.075040+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:03.934158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074205+0000","last_change":"2024-03-28T23:33:51.006212+0000","last_active":"2024-03-28T23:33:53.074205+0000","last_peered":"2024-03-28T23:33:53.074205+0000","last_clean":"2024-03-28T23:33:53.074205+0000","last_became_active":"2024-03-28T23:33:51.005936+0000","last_became_peered":"2024-03-28T23:33:51.005936+0000","last_unstale":"2024-03-28T23:33:53.074205+0000","last_undegraded":"2024-03-28T23:33:53.074205+0000","last_fullsized":"2024-03-28T23:33:53.074205+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:51:25.515271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074294+0000","last_change":"2024-03-28T23:33:49.989507+0000","last_active":"2024-03-28T23:33:53.074294+0000","last_peered":"2024-03-28T23:33:53.074294+0000","last_clean":"2024-03-28T23:33:53.074294+0000","last_became_active":"2024-03-28T23:33:49.988693+0000","last_became_peered":"2024-03-28T23:33:49.988693+0000","last_unstale":"2024-03-28T23:33:53.074294+0000","last_undegraded":"2024-03-28T23:33:53.074294+0000","last_fullsized":"2024-03-28T23:33:53.074294+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:36:05.299785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074046+0000","last_change":"2024-03-28T23:33:51.016137+0000","last_active":"2024-03-28T23:33:53.074046+0000","last_peered":"2024-03-28T23:33:53.074046+0000","last_clean":"2024-03-28T23:33:53.074046+0000","last_became_active":"2024-03-28T23:33:51.015837+0000","last_became_peered":"2024-03-28T23:33:51.015837+0000","last_unstale":"2024-03-28T23:33:53.074046+0000","last_undegraded":"2024-03-28T23:33:53.074046+0000","last_fullsized":"2024-03-28T23:33:53.074046+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:20:01.738504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.004089+0000","last_change":"2024-03-28T23:33:49.997252+0000","last_active":"2024-03-28T23:33:52.004089+0000","last_peered":"2024-03-28T23:33:52.004089+0000","last_clean":"2024-03-28T23:33:52.004089+0000","last_became_active":"2024-03-28T23:33:49.996952+0000","last_became_peered":"2024-03-28T23:33:49.996952+0000","last_unstale":"2024-03-28T23:33:52.004089+0000","last_undegraded":"2024-03-28T23:33:52.004089+0000","last_fullsized":"2024-03-28T23:33:52.004089+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:26:48.812133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005426+0000","last_change":"2024-03-28T23:33:51.025429+0000","last_active":"2024-03-28T23:33:52.005426+0000","last_peered":"2024-03-28T23:33:52.005426+0000","last_clean":"2024-03-28T23:33:52.005426+0000","last_became_active":"2024-03-28T23:33:51.025144+0000","last_became_peered":"2024-03-28T23:33:51.025144+0000","last_unstale":"2024-03-28T23:33:52.005426+0000","last_undegraded":"2024-03-28T23:33:52.005426+0000","last_fullsized":"2024-03-28T23:33:52.005426+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:26:17.908028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074414+0000","last_change":"2024-03-28T23:33:49.997415+0000","last_active":"2024-03-28T23:33:53.074414+0000","last_peered":"2024-03-28T23:33:53.074414+0000","last_clean":"2024-03-28T23:33:53.074414+0000","last_became_active":"2024-03-28T23:33:49.997127+0000","last_became_peered":"2024-03-28T23:33:49.997127+0000","last_unstale":"2024-03-28T23:33:53.074414+0000","last_undegraded":"2024-03-28T23:33:53.074414+0000","last_fullsized":"2024-03-28T23:33:53.074414+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:45:33.988210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.018913+0000","last_change":"2024-03-28T23:33:51.017947+0000","last_active":"2024-03-28T23:33:52.018913+0000","last_peered":"2024-03-28T23:33:52.018913+0000","last_clean":"2024-03-28T23:33:52.018913+0000","last_became_active":"2024-03-28T23:33:51.017357+0000","last_became_peered":"2024-03-28T23:33:51.017357+0000","last_unstale":"2024-03-28T23:33:52.018913+0000","last_undegraded":"2024-03-28T23:33:52.018913+0000","last_fullsized":"2024-03-28T23:33:52.018913+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:27:23.718523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074747+0000","last_change":"2024-03-28T23:33:49.992128+0000","last_active":"2024-03-28T23:33:53.074747+0000","last_peered":"2024-03-28T23:33:53.074747+0000","last_clean":"2024-03-28T23:33:53.074747+0000","last_became_active":"2024-03-28T23:33:49.991464+0000","last_became_peered":"2024-03-28T23:33:49.991464+0000","last_unstale":"2024-03-28T23:33:53.074747+0000","last_undegraded":"2024-03-28T23:33:53.074747+0000","last_fullsized":"2024-03-28T23:33:53.074747+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:16:39.848586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074333+0000","last_change":"2024-03-28T23:33:51.015993+0000","last_active":"2024-03-28T23:33:53.074333+0000","last_peered":"2024-03-28T23:33:53.074333+0000","last_clean":"2024-03-28T23:33:53.074333+0000","last_became_active":"2024-03-28T23:33:51.013143+0000","last_became_peered":"2024-03-28T23:33:51.013143+0000","last_unstale":"2024-03-28T23:33:53.074333+0000","last_undegraded":"2024-03-28T23:33:53.074333+0000","last_fullsized":"2024-03-28T23:33:53.074333+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:13.109884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074525+0000","last_change":"2024-03-28T23:33:49.998016+0000","last_active":"2024-03-28T23:33:53.074525+0000","last_peered":"2024-03-28T23:33:53.074525+0000","last_clean":"2024-03-28T23:33:53.074525+0000","last_became_active":"2024-03-28T23:33:49.997751+0000","last_became_peered":"2024-03-28T23:33:49.997751+0000","last_unstale":"2024-03-28T23:33:53.074525+0000","last_undegraded":"2024-03-28T23:33:53.074525+0000","last_fullsized":"2024-03-28T23:33:53.074525+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:45:51.352057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.018665+0000","last_change":"2024-03-28T23:33:51.016437+0000","last_active":"2024-03-28T23:33:52.018665+0000","last_peered":"2024-03-28T23:33:52.018665+0000","last_clean":"2024-03-28T23:33:52.018665+0000","last_became_active":"2024-03-28T23:33:51.016142+0000","last_became_peered":"2024-03-28T23:33:51.016142+0000","last_unstale":"2024-03-28T23:33:52.018665+0000","last_undegraded":"2024-03-28T23:33:52.018665+0000","last_fullsized":"2024-03-28T23:33:52.018665+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:47:01.158311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.018581+0000","last_change":"2024-03-28T23:33:49.993146+0000","last_active":"2024-03-28T23:33:52.018581+0000","last_peered":"2024-03-28T23:33:52.018581+0000","last_clean":"2024-03-28T23:33:52.018581+0000","last_became_active":"2024-03-28T23:33:49.992913+0000","last_became_peered":"2024-03-28T23:33:49.992913+0000","last_unstale":"2024-03-28T23:33:52.018581+0000","last_undegraded":"2024-03-28T23:33:52.018581+0000","last_fullsized":"2024-03-28T23:33:52.018581+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:04:47.661317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074086+0000","last_change":"2024-03-28T23:33:51.023128+0000","last_active":"2024-03-28T23:33:53.074086+0000","last_peered":"2024-03-28T23:33:53.074086+0000","last_clean":"2024-03-28T23:33:53.074086+0000","last_became_active":"2024-03-28T23:33:51.022835+0000","last_became_peered":"2024-03-28T23:33:51.022835+0000","last_unstale":"2024-03-28T23:33:53.074086+0000","last_undegraded":"2024-03-28T23:33:53.074086+0000","last_fullsized":"2024-03-28T23:33:53.074086+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:43:24.492761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074451+0000","last_change":"2024-03-28T23:33:49.990123+0000","last_active":"2024-03-28T23:33:53.074451+0000","last_peered":"2024-03-28T23:33:53.074451+0000","last_clean":"2024-03-28T23:33:53.074451+0000","last_became_active":"2024-03-28T23:33:49.989834+0000","last_became_peered":"2024-03-28T23:33:49.989834+0000","last_unstale":"2024-03-28T23:33:53.074451+0000","last_undegraded":"2024-03-28T23:33:53.074451+0000","last_fullsized":"2024-03-28T23:33:53.074451+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:52.093172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.075118+0000","last_change":"2024-03-28T23:33:51.015947+0000","last_active":"2024-03-28T23:33:53.075118+0000","last_peered":"2024-03-28T23:33:53.075118+0000","last_clean":"2024-03-28T23:33:53.075118+0000","last_became_active":"2024-03-28T23:33:51.015599+0000","last_became_peered":"2024-03-28T23:33:51.015599+0000","last_unstale":"2024-03-28T23:33:53.075118+0000","last_undegraded":"2024-03-28T23:33:53.075118+0000","last_fullsized":"2024-03-28T23:33:53.075118+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:02.377786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074299+0000","last_change":"2024-03-28T23:33:49.996325+0000","last_active":"2024-03-28T23:33:53.074299+0000","last_peered":"2024-03-28T23:33:53.074299+0000","last_clean":"2024-03-28T23:33:53.074299+0000","last_became_active":"2024-03-28T23:33:49.996067+0000","last_became_peered":"2024-03-28T23:33:49.996067+0000","last_unstale":"2024-03-28T23:33:53.074299+0000","last_undegraded":"2024-03-28T23:33:53.074299+0000","last_fullsized":"2024-03-28T23:33:53.074299+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:24:28.810699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074007+0000","last_change":"2024-03-28T23:33:51.005474+0000","last_active":"2024-03-28T23:33:53.074007+0000","last_peered":"2024-03-28T23:33:53.074007+0000","last_clean":"2024-03-28T23:33:53.074007+0000","last_became_active":"2024-03-28T23:33:51.005209+0000","last_became_peered":"2024-03-28T23:33:51.005209+0000","last_unstale":"2024-03-28T23:33:53.074007+0000","last_undegraded":"2024-03-28T23:33:53.074007+0000","last_fullsized":"2024-03-28T23:33:53.074007+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:26:17.908028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.073795+0000","last_change":"2024-03-28T23:33:49.990740+0000","last_active":"2024-03-28T23:33:53.073795+0000","last_peered":"2024-03-28T23:33:53.073795+0000","last_clean":"2024-03-28T23:33:53.073795+0000","last_became_active":"2024-03-28T23:33:49.990494+0000","last_became_peered":"2024-03-28T23:33:49.990494+0000","last_unstale":"2024-03-28T23:33:53.073795+0000","last_undegraded":"2024-03-28T23:33:53.073795+0000","last_fullsized":"2024-03-28T23:33:53.073795+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:38:37.720292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.019213+0000","last_change":"2024-03-28T23:33:51.027238+0000","last_active":"2024-03-28T23:33:52.019213+0000","last_peered":"2024-03-28T23:33:52.019213+0000","last_clean":"2024-03-28T23:33:52.019213+0000","last_became_active":"2024-03-28T23:33:51.026460+0000","last_became_peered":"2024-03-28T23:33:51.026460+0000","last_unstale":"2024-03-28T23:33:52.019213+0000","last_undegraded":"2024-03-28T23:33:52.019213+0000","last_fullsized":"2024-03-28T23:33:52.019213+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:34:07.840166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.019041+0000","last_change":"2024-03-28T23:33:49.996464+0000","last_active":"2024-03-28T23:33:52.019041+0000","last_peered":"2024-03-28T23:33:52.019041+0000","last_clean":"2024-03-28T23:33:52.019041+0000","last_became_active":"2024-03-28T23:33:49.996122+0000","last_became_peered":"2024-03-28T23:33:49.996122+0000","last_unstale":"2024-03-28T23:33:52.019041+0000","last_undegraded":"2024-03-28T23:33:52.019041+0000","last_fullsized":"2024-03-28T23:33:52.019041+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:46:50.856475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.003796+0000","last_change":"2024-03-28T23:33:51.010596+0000","last_active":"2024-03-28T23:33:52.003796+0000","last_peered":"2024-03-28T23:33:52.003796+0000","last_clean":"2024-03-28T23:33:52.003796+0000","last_became_active":"2024-03-28T23:33:51.010331+0000","last_became_peered":"2024-03-28T23:33:51.010331+0000","last_unstale":"2024-03-28T23:33:52.003796+0000","last_undegraded":"2024-03-28T23:33:52.003796+0000","last_fullsized":"2024-03-28T23:33:52.003796+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:36:06.301567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"109'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005175+0000","last_change":"2024-03-28T23:33:49.997526+0000","last_active":"2024-03-28T23:33:52.005175+0000","last_peered":"2024-03-28T23:33:52.005175+0000","last_clean":"2024-03-28T23:33:52.005175+0000","last_became_active":"2024-03-28T23:33:49.997216+0000","last_became_peered":"2024-03-28T23:33:49.997216+0000","last_unstale":"2024-03-28T23:33:52.005175+0000","last_undegraded":"2024-03-28T23:33:52.005175+0000","last_fullsized":"2024-03-28T23:33:52.005175+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:33:09.234630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074924+0000","last_change":"2024-03-28T23:33:51.012495+0000","last_active":"2024-03-28T23:33:53.074924+0000","last_peered":"2024-03-28T23:33:53.074924+0000","last_clean":"2024-03-28T23:33:53.074924+0000","last_became_active":"2024-03-28T23:33:51.012232+0000","last_became_peered":"2024-03-28T23:33:51.012232+0000","last_unstale":"2024-03-28T23:33:53.074924+0000","last_undegraded":"2024-03-28T23:33:53.074924+0000","last_fullsized":"2024-03-28T23:33:53.074924+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:09.825723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005673+0000","last_change":"2024-03-28T23:33:49.997622+0000","last_active":"2024-03-28T23:33:52.005673+0000","last_peered":"2024-03-28T23:33:52.005673+0000","last_clean":"2024-03-28T23:33:52.005673+0000","last_became_active":"2024-03-28T23:33:49.997247+0000","last_became_peered":"2024-03-28T23:33:49.997247+0000","last_unstale":"2024-03-28T23:33:52.005673+0000","last_undegraded":"2024-03-28T23:33:52.005673+0000","last_fullsized":"2024-03-28T23:33:52.005673+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:59:21.391833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.075123+0000","last_change":"2024-03-28T23:33:51.013400+0000","last_active":"2024-03-28T23:33:53.075123+0000","last_peered":"2024-03-28T23:33:53.075123+0000","last_clean":"2024-03-28T23:33:53.075123+0000","last_became_active":"2024-03-28T23:33:51.013130+0000","last_became_peered":"2024-03-28T23:33:51.013130+0000","last_unstale":"2024-03-28T23:33:53.075123+0000","last_undegraded":"2024-03-28T23:33:53.075123+0000","last_fullsized":"2024-03-28T23:33:53.075123+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:10:50.716440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.019188+0000","last_change":"2024-03-28T23:33:49.996259+0000","last_active":"2024-03-28T23:33:52.019188+0000","last_peered":"2024-03-28T23:33:52.019188+0000","last_clean":"2024-03-28T23:33:52.019188+0000","last_became_active":"2024-03-28T23:33:49.996003+0000","last_became_peered":"2024-03-28T23:33:49.996003+0000","last_unstale":"2024-03-28T23:33:52.019188+0000","last_undegraded":"2024-03-28T23:33:52.019188+0000","last_fullsized":"2024-03-28T23:33:52.019188+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:01:59.777890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.018783+0000","last_change":"2024-03-28T23:33:51.016991+0000","last_active":"2024-03-28T23:33:52.018783+0000","last_peered":"2024-03-28T23:33:52.018783+0000","last_clean":"2024-03-28T23:33:52.018783+0000","last_became_active":"2024-03-28T23:33:51.016731+0000","last_became_peered":"2024-03-28T23:33:51.016731+0000","last_unstale":"2024-03-28T23:33:52.018783+0000","last_undegraded":"2024-03-28T23:33:52.018783+0000","last_fullsized":"2024-03-28T23:33:52.018783+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:29:13.673822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005653+0000","last_change":"2024-03-28T23:33:49.988649+0000","last_active":"2024-03-28T23:33:52.005653+0000","last_peered":"2024-03-28T23:33:52.005653+0000","last_clean":"2024-03-28T23:33:52.005653+0000","last_became_active":"2024-03-28T23:33:49.988374+0000","last_became_peered":"2024-03-28T23:33:49.988374+0000","last_unstale":"2024-03-28T23:33:52.005653+0000","last_undegraded":"2024-03-28T23:33:52.005653+0000","last_fullsized":"2024-03-28T23:33:52.005653+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:38:37.720292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074238+0000","last_change":"2024-03-28T23:33:51.019372+0000","last_active":"2024-03-28T23:33:53.074238+0000","last_peered":"2024-03-28T23:33:53.074238+0000","last_clean":"2024-03-28T23:33:53.074238+0000","last_became_active":"2024-03-28T23:33:51.019096+0000","last_became_peered":"2024-03-28T23:33:51.019096+0000","last_unstale":"2024-03-28T23:33:53.074238+0000","last_undegraded":"2024-03-28T23:33:53.074238+0000","last_fullsized":"2024-03-28T23:33:53.074238+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:48:21.194198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005259+0000","last_change":"2024-03-28T23:33:49.985591+0000","last_active":"2024-03-28T23:33:52.005259+0000","last_peered":"2024-03-28T23:33:52.005259+0000","last_clean":"2024-03-28T23:33:52.005259+0000","last_became_active":"2024-03-28T23:33:49.985344+0000","last_became_peered":"2024-03-28T23:33:49.985344+0000","last_unstale":"2024-03-28T23:33:52.005259+0000","last_undegraded":"2024-03-28T23:33:52.005259+0000","last_fullsized":"2024-03-28T23:33:52.005259+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:12.273007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074555+0000","last_change":"2024-03-28T23:33:51.019705+0000","last_active":"2024-03-28T23:33:53.074555+0000","last_peered":"2024-03-28T23:33:53.074555+0000","last_clean":"2024-03-28T23:33:53.074555+0000","last_became_active":"2024-03-28T23:33:51.018300+0000","last_became_peered":"2024-03-28T23:33:51.018300+0000","last_unstale":"2024-03-28T23:33:53.074555+0000","last_undegraded":"2024-03-28T23:33:53.074555+0000","last_fullsized":"2024-03-28T23:33:53.074555+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:32.390009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.018583+0000","last_change":"2024-03-28T23:33:49.996343+0000","last_active":"2024-03-28T23:33:52.018583+0000","last_peered":"2024-03-28T23:33:52.018583+0000","last_clean":"2024-03-28T23:33:52.018583+0000","last_became_active":"2024-03-28T23:33:49.996095+0000","last_became_peered":"2024-03-28T23:33:49.996095+0000","last_unstale":"2024-03-28T23:33:52.018583+0000","last_undegraded":"2024-03-28T23:33:52.018583+0000","last_fullsized":"2024-03-28T23:33:52.018583+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:35:17.358659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.003969+0000","last_change":"2024-03-28T23:33:51.018684+0000","last_active":"2024-03-28T23:33:52.003969+0000","last_peered":"2024-03-28T23:33:52.003969+0000","last_clean":"2024-03-28T23:33:52.003969+0000","last_became_active":"2024-03-28T23:33:51.015122+0000","last_became_peered":"2024-03-28T23:33:51.015122+0000","last_unstale":"2024-03-28T23:33:52.003969+0000","last_undegraded":"2024-03-28T23:33:52.003969+0000","last_fullsized":"2024-03-28T23:33:52.003969+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:26:17.908028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074214+0000","last_change":"2024-03-28T23:33:49.988606+0000","last_active":"2024-03-28T23:33:53.074214+0000","last_peered":"2024-03-28T23:33:53.074214+0000","last_clean":"2024-03-28T23:33:53.074214+0000","last_became_active":"2024-03-28T23:33:49.988351+0000","last_became_peered":"2024-03-28T23:33:49.988351+0000","last_unstale":"2024-03-28T23:33:53.074214+0000","last_undegraded":"2024-03-28T23:33:53.074214+0000","last_fullsized":"2024-03-28T23:33:53.074214+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:39:39.102964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074755+0000","last_change":"2024-03-28T23:33:51.020599+0000","last_active":"2024-03-28T23:33:53.074755+0000","last_peered":"2024-03-28T23:33:53.074755+0000","last_clean":"2024-03-28T23:33:53.074755+0000","last_became_active":"2024-03-28T23:33:51.020020+0000","last_became_peered":"2024-03-28T23:33:51.020020+0000","last_unstale":"2024-03-28T23:33:53.074755+0000","last_undegraded":"2024-03-28T23:33:53.074755+0000","last_fullsized":"2024-03-28T23:33:53.074755+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:13.109884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"108'8","reported_seq":22,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074034+0000","last_change":"2024-03-28T23:33:49.982727+0000","last_active":"2024-03-28T23:33:53.074034+0000","last_peered":"2024-03-28T23:33:53.074034+0000","last_clean":"2024-03-28T23:33:53.074034+0000","last_became_active":"2024-03-28T23:33:49.982456+0000","last_became_peered":"2024-03-28T23:33:49.982456+0000","last_unstale":"2024-03-28T23:33:53.074034+0000","last_undegraded":"2024-03-28T23:33:53.074034+0000","last_fullsized":"2024-03-28T23:33:53.074034+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:02:48.238272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":338,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074293+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:33:53.074293+0000","last_peered":"2024-03-28T23:33:53.074293+0000","last_clean":"2024-03-28T23:33:53.074293+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:33:53.074293+0000","last_undegraded":"2024-03-28T23:33:53.074293+0000","last_fullsized":"2024-03-28T23:33:53.074293+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574901,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32588,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339252,"statfs":{"total":96636764160,"available":96603394048,"internally_reserved":0,"allocated":1441792,"data_stored":1247559,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32516,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339324,"statfs":{"total":96636764160,"available":96603467776,"internally_reserved":0,"allocated":1363968,"data_stored":656997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574902,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32520,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339320,"statfs":{"total":96636764160,"available":96603463680,"internally_reserved":0,"allocated":1368064,"data_stored":656575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574901,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32524,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339316,"statfs":{"total":96636764160,"available":96603459584,"internally_reserved":0,"allocated":1372160,"data_stored":658027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32528,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339312,"statfs":{"total":96636764160,"available":96603455488,"internally_reserved":0,"allocated":1376256,"data_stored":658061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574901,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1359872,"data_stored":656415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32516,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339324,"statfs":{"total":96636764160,"available":96603467776,"internally_reserved":0,"allocated":1363968,"data_stored":656437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5],"upper_bound":8},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574901,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1429504,"data_stored":1246805,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:33:55.910 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:33:55.912 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:33:56.261 INFO:tasks.ceph.mds.a.smithi161.stderr:2024-03-28T23:33:56.261+0000 7ff84eb1f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:33:58.578 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:33:58.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.680+0000 7f76cd148640 1 Processor -- start 2024-03-28T23:33:58.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.680+0000 7f76cd148640 1 -- start start 2024-03-28T23:33:58.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.681+0000 7f76cd148640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f76c8152890 0x7f76c80c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:58.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.681+0000 7f76cd148640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76c80c8ed0 0x7f76c80c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:58.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.681+0000 7f76cd148640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76c8158a30 con 0x7f76c80c5950 2024-03-28T23:33:58.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.681+0000 7f76cd148640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f76c80c8480 con 0x7f76c8152890 2024-03-28T23:33:58.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.681+0000 7f76cd148640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f76c8159640 con 0x7f76c80c8ed0 2024-03-28T23:33:58.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.681+0000 7f76c6d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f76c8152890 0x7f76c80c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:58.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.681+0000 7f76c6d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f76c8152890 0x7f76c80c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60150/0 (socket says 172.21.15.161:60150) 2024-03-28T23:33:58.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.681+0000 7f76c6d9d640 1 -- 172.21.15.161:0/2601812584 learned_addr learned my addr 172.21.15.161:0/2601812584 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:58.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.681+0000 7f76c659c640 1 --2- 172.21.15.161:0/2601812584 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76c80c8ed0 0x7f76c80c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:58.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.681+0000 7f76c6d9d640 1 -- 172.21.15.161:0/2601812584 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76c80c8ed0 msgr2=0x7f76c80c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:58.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.681+0000 7f76c6d9d640 1 --2- 172.21.15.161:0/2601812584 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76c80c8ed0 0x7f76c80c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:58.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.681+0000 7f76c6d9d640 1 -- 172.21.15.161:0/2601812584 >> v1:172.21.15.170:6789/0 conn(0x7f76c80c5950 legacy=0x7f76c81553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:58.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.681+0000 7f76c6d9d640 1 -- 172.21.15.161:0/2601812584 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76c8158d10 con 0x7f76c8152890 2024-03-28T23:33:58.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.682+0000 7f76c6d9d640 1 --2- 172.21.15.161:0/2601812584 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76c8152890 0x7f76c80c8990 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f76bc009870 tx=0x7f76bc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e79c1885d3c94ef server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:58.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.682+0000 7f76c5d9b640 1 -- 172.21.15.161:0/2601812584 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc0317c0 con 0x7f76c8152890 2024-03-28T23:33:58.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.682+0000 7f76c5d9b640 1 -- 172.21.15.161:0/2601812584 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76bc037070 con 0x7f76c8152890 2024-03-28T23:33:58.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.682+0000 7f76c5d9b640 1 -- 172.21.15.161:0/2601812584 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc031b80 con 0x7f76c8152890 2024-03-28T23:33:58.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.683+0000 7f76cd148640 1 -- 172.21.15.161:0/2601812584 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76c8152890 msgr2=0x7f76c80c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:58.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.683+0000 7f76cd148640 1 --2- 172.21.15.161:0/2601812584 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76c8152890 0x7f76c80c8990 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f76bc009870 tx=0x7f76bc02b860 comp rx=0 tx=0).stop 2024-03-28T23:33:58.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.683+0000 7f76cd148640 1 -- 172.21.15.161:0/2601812584 shutdown_connections 2024-03-28T23:33:58.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.683+0000 7f76cd148640 1 --2- 172.21.15.161:0/2601812584 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76c80c8ed0 0x7f76c80c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:58.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.683+0000 7f76cd148640 1 --2- 172.21.15.161:0/2601812584 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76c8152890 0x7f76c80c8990 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:58.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.683+0000 7f76cd148640 1 -- 172.21.15.161:0/2601812584 >> 172.21.15.161:0/2601812584 conn(0x7f76c80ac620 msgr2=0x7f76c80aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:58.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.683+0000 7f76cd148640 1 -- 172.21.15.161:0/2601812584 shutdown_connections 2024-03-28T23:33:58.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.683+0000 7f76cd148640 1 -- 172.21.15.161:0/2601812584 wait complete. 2024-03-28T23:33:58.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.685+0000 7f76cd148640 1 Processor -- start 2024-03-28T23:33:58.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.685+0000 7f76cd148640 1 -- start start 2024-03-28T23:33:58.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.686+0000 7f76cd148640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76c80c5950 0x7f76c8145da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:58.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.686+0000 7f76cd148640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76c80c8ed0 0x7f76c8157380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:58.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.686+0000 7f76cd148640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76c8152890 0x7f76c81ed5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:58.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.686+0000 7f76cd148640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f76c8152c70 con 0x7f76c80c5950 2024-03-28T23:33:58.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.686+0000 7f76cd148640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f76c81592d0 con 0x7f76c8152890 2024-03-28T23:33:58.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.686+0000 7f76cd148640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f76c8152df0 con 0x7f76c80c8ed0 2024-03-28T23:33:58.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.686+0000 7f76c6d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76c80c5950 0x7f76c8145da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:58.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.686+0000 7f76c6d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76c80c5950 0x7f76c8145da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60166/0 (socket says 172.21.15.161:60166) 2024-03-28T23:33:58.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.686+0000 7f76c6d9d640 1 -- 172.21.15.161:0/2820078350 learned_addr learned my addr 172.21.15.161:0/2820078350 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:58.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.686+0000 7f76c659c640 1 --2- 172.21.15.161:0/2820078350 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76c80c8ed0 0x7f76c8157380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:58.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.686+0000 7f76c759e640 1 --2- 172.21.15.161:0/2820078350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76c8152890 0x7f76c81ed5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:58.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.686+0000 7f76c6d9d640 1 -- 172.21.15.161:0/2820078350 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76c80c8ed0 msgr2=0x7f76c8157380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:58.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.686+0000 7f76c6d9d640 1 --2- 172.21.15.161:0/2820078350 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76c80c8ed0 0x7f76c8157380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:58.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.686+0000 7f76c6d9d640 1 -- 172.21.15.161:0/2820078350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76c8152890 msgr2=0x7f76c81ed5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:58.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.686+0000 7f76c6d9d640 1 --2- 172.21.15.161:0/2820078350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76c8152890 0x7f76c81ed5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:58.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.686+0000 7f76c6d9d640 1 -- 172.21.15.161:0/2820078350 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76c81ee160 con 0x7f76c80c5950 2024-03-28T23:33:58.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.686+0000 7f76c6d9d640 1 --2- 172.21.15.161:0/2820078350 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76c80c5950 0x7f76c8145da0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f76bc02b8a0 tx=0x7f76bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:58.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.686+0000 7f76af7fe640 1 -- 172.21.15.161:0/2820078350 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc032650 con 0x7f76c80c5950 2024-03-28T23:33:58.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.686+0000 7f76af7fe640 1 -- 172.21.15.161:0/2820078350 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76bc0327f0 con 0x7f76c80c5950 2024-03-28T23:33:58.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.686+0000 7f76cd148640 1 -- 172.21.15.161:0/2820078350 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76c81ee440 con 0x7f76c80c5950 2024-03-28T23:33:58.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.686+0000 7f76af7fe640 1 -- 172.21.15.161:0/2820078350 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc032be0 con 0x7f76c80c5950 2024-03-28T23:33:58.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.687+0000 7f76cd148640 1 -- 172.21.15.161:0/2820078350 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76c81eea00 con 0x7f76c80c5950 2024-03-28T23:33:58.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.688+0000 7f76cd148640 1 -- 172.21.15.161:0/2820078350 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76c806c6c0 con 0x7f76c80c5950 2024-03-28T23:33:58.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.688+0000 7f76af7fe640 1 -- 172.21.15.161:0/2820078350 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f76bc04c060 con 0x7f76c80c5950 2024-03-28T23:33:58.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.688+0000 7f76af7fe640 1 --2- 172.21.15.161:0/2820078350 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f76b406b4d0 0x7f76b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:58.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.689+0000 7f76af7fe640 1 -- 172.21.15.161:0/2820078350 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6101+0+0 (secure 0 0 0) 0x7f76bc0abce0 con 0x7f76c80c5950 2024-03-28T23:33:58.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.689+0000 7f76c659c640 1 --2- 172.21.15.161:0/2820078350 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f76b406b4d0 0x7f76b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:58.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.689+0000 7f76c659c640 1 --2- 172.21.15.161:0/2820078350 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f76b406b4d0 0x7f76b406d990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f76c80c1840 tx=0x7f76b8009110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:58.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.693+0000 7f76af7fe640 1 -- 172.21.15.161:0/2820078350 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f76c806c6c0 con 0x7f76c80c5950 2024-03-28T23:33:58.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.860+0000 7f76cd148640 1 -- 172.21.15.161:0/2820078350 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f76c804f4b0 con 0x7f76b406b4d0 2024-03-28T23:33:58.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.867+0000 7f76af7fe640 1 -- 172.21.15.161:0/2820078350 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252573 (secure 0 0 0) 0x7f76c804f4b0 con 0x7f76b406b4d0 2024-03-28T23:33:58.870 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:58.871 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:33:58.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.870+0000 7f76cd148640 1 -- 172.21.15.161:0/2820078350 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f76b406b4d0 msgr2=0x7f76b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:58.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.870+0000 7f76cd148640 1 --2- 172.21.15.161:0/2820078350 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f76b406b4d0 0x7f76b406d990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f76c80c1840 tx=0x7f76b8009110 comp rx=0 tx=0).stop 2024-03-28T23:33:58.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.870+0000 7f76cd148640 1 -- 172.21.15.161:0/2820078350 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76c80c5950 msgr2=0x7f76c8145da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:58.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.870+0000 7f76cd148640 1 --2- 172.21.15.161:0/2820078350 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76c80c5950 0x7f76c8145da0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f76bc02b8a0 tx=0x7f76bc0046c0 comp rx=0 tx=0).stop 2024-03-28T23:33:58.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.870+0000 7f76cd148640 1 -- 172.21.15.161:0/2820078350 shutdown_connections 2024-03-28T23:33:58.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.870+0000 7f76cd148640 1 --2- 172.21.15.161:0/2820078350 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f76b406b4d0 0x7f76b406d990 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:58.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.870+0000 7f76cd148640 1 --2- 172.21.15.161:0/2820078350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76c8152890 0x7f76c81ed5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:58.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.870+0000 7f76cd148640 1 --2- 172.21.15.161:0/2820078350 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76c80c8ed0 0x7f76c8157380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:58.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.870+0000 7f76cd148640 1 --2- 172.21.15.161:0/2820078350 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76c80c5950 0x7f76c8145da0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:58.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.870+0000 7f76cd148640 1 -- 172.21.15.161:0/2820078350 >> 172.21.15.161:0/2820078350 conn(0x7f76c80ac620 msgr2=0x7f76c814fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:58.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.870+0000 7f76cd148640 1 -- 172.21.15.161:0/2820078350 shutdown_connections 2024-03-28T23:33:58.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:58.870+0000 7f76cd148640 1 -- 172.21.15.161:0/2820078350 wait complete. 2024-03-28T23:33:58.881 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2024-03-28T23:33:57.359241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260288,"kb_used_data":10816,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714432,"statfs":{"total":773094113280,"available":772827578368,"internally_reserved":0,"allocated":11075584,"data_stored":6436876,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5],"upper_bound":8},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.385596"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005718+0000","last_change":"2024-03-28T23:33:51.013666+0000","last_active":"2024-03-28T23:33:52.005718+0000","last_peered":"2024-03-28T23:33:52.005718+0000","last_clean":"2024-03-28T23:33:52.005718+0000","last_became_active":"2024-03-28T23:33:51.013355+0000","last_became_peered":"2024-03-28T23:33:51.013355+0000","last_unstale":"2024-03-28T23:33:52.005718+0000","last_undegraded":"2024-03-28T23:33:52.005718+0000","last_fullsized":"2024-03-28T23:33:52.005718+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:32.390009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074441+0000","last_change":"2024-03-28T23:33:51.019220+0000","last_active":"2024-03-28T23:33:53.074441+0000","last_peered":"2024-03-28T23:33:53.074441+0000","last_clean":"2024-03-28T23:33:53.074441+0000","last_became_active":"2024-03-28T23:33:51.018735+0000","last_became_peered":"2024-03-28T23:33:51.018735+0000","last_unstale":"2024-03-28T23:33:53.074441+0000","last_undegraded":"2024-03-28T23:33:53.074441+0000","last_fullsized":"2024-03-28T23:33:53.074441+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:30:37.699564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074731+0000","last_change":"2024-03-28T23:33:51.018423+0000","last_active":"2024-03-28T23:33:53.074731+0000","last_peered":"2024-03-28T23:33:53.074731+0000","last_clean":"2024-03-28T23:33:53.074731+0000","last_became_active":"2024-03-28T23:33:51.015925+0000","last_became_peered":"2024-03-28T23:33:51.015925+0000","last_unstale":"2024-03-28T23:33:53.074731+0000","last_undegraded":"2024-03-28T23:33:53.074731+0000","last_fullsized":"2024-03-28T23:33:53.074731+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:46:08.946536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074663+0000","last_change":"2024-03-28T23:33:51.014367+0000","last_active":"2024-03-28T23:33:53.074663+0000","last_peered":"2024-03-28T23:33:53.074663+0000","last_clean":"2024-03-28T23:33:53.074663+0000","last_became_active":"2024-03-28T23:33:51.013995+0000","last_became_peered":"2024-03-28T23:33:51.013995+0000","last_unstale":"2024-03-28T23:33:53.074663+0000","last_undegraded":"2024-03-28T23:33:53.074663+0000","last_fullsized":"2024-03-28T23:33:53.074663+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:59:52.655622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005564+0000","last_change":"2024-03-28T23:33:51.009473+0000","last_active":"2024-03-28T23:33:52.005564+0000","last_peered":"2024-03-28T23:33:52.005564+0000","last_clean":"2024-03-28T23:33:52.005564+0000","last_became_active":"2024-03-28T23:33:51.009179+0000","last_became_peered":"2024-03-28T23:33:51.009179+0000","last_unstale":"2024-03-28T23:33:52.005564+0000","last_undegraded":"2024-03-28T23:33:52.005564+0000","last_fullsized":"2024-03-28T23:33:52.005564+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:37.206474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.018841+0000","last_change":"2024-03-28T23:33:51.030849+0000","last_active":"2024-03-28T23:33:52.018841+0000","last_peered":"2024-03-28T23:33:52.018841+0000","last_clean":"2024-03-28T23:33:52.018841+0000","last_became_active":"2024-03-28T23:33:51.030557+0000","last_became_peered":"2024-03-28T23:33:51.030557+0000","last_unstale":"2024-03-28T23:33:52.018841+0000","last_undegraded":"2024-03-28T23:33:52.018841+0000","last_fullsized":"2024-03-28T23:33:52.018841+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:04:45.819608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.018622+0000","last_change":"2024-03-28T23:33:51.017932+0000","last_active":"2024-03-28T23:33:52.018622+0000","last_peered":"2024-03-28T23:33:52.018622+0000","last_clean":"2024-03-28T23:33:52.018622+0000","last_became_active":"2024-03-28T23:33:51.017660+0000","last_became_peered":"2024-03-28T23:33:51.017660+0000","last_unstale":"2024-03-28T23:33:52.018622+0000","last_undegraded":"2024-03-28T23:33:52.018622+0000","last_fullsized":"2024-03-28T23:33:52.018622+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:27:37.193802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074077+0000","last_change":"2024-03-28T23:33:51.024956+0000","last_active":"2024-03-28T23:33:53.074077+0000","last_peered":"2024-03-28T23:33:53.074077+0000","last_clean":"2024-03-28T23:33:53.074077+0000","last_became_active":"2024-03-28T23:33:51.019702+0000","last_became_peered":"2024-03-28T23:33:51.019702+0000","last_unstale":"2024-03-28T23:33:53.074077+0000","last_undegraded":"2024-03-28T23:33:53.074077+0000","last_fullsized":"2024-03-28T23:33:53.074077+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:15:35.186625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.075042+0000","last_change":"2024-03-28T23:33:51.025952+0000","last_active":"2024-03-28T23:33:53.075042+0000","last_peered":"2024-03-28T23:33:53.075042+0000","last_clean":"2024-03-28T23:33:53.075042+0000","last_became_active":"2024-03-28T23:33:51.025674+0000","last_became_peered":"2024-03-28T23:33:51.025674+0000","last_unstale":"2024-03-28T23:33:53.075042+0000","last_undegraded":"2024-03-28T23:33:53.075042+0000","last_fullsized":"2024-03-28T23:33:53.075042+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:27:03.382059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.003956+0000","last_change":"2024-03-28T23:33:51.017655+0000","last_active":"2024-03-28T23:33:52.003956+0000","last_peered":"2024-03-28T23:33:52.003956+0000","last_clean":"2024-03-28T23:33:52.003956+0000","last_became_active":"2024-03-28T23:33:51.017399+0000","last_became_peered":"2024-03-28T23:33:51.017399+0000","last_unstale":"2024-03-28T23:33:52.003956+0000","last_undegraded":"2024-03-28T23:33:52.003956+0000","last_fullsized":"2024-03-28T23:33:52.003956+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:24:29.812480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005011+0000","last_change":"2024-03-28T23:33:51.015409+0000","last_active":"2024-03-28T23:33:52.005011+0000","last_peered":"2024-03-28T23:33:52.005011+0000","last_clean":"2024-03-28T23:33:52.005011+0000","last_became_active":"2024-03-28T23:33:51.015114+0000","last_became_peered":"2024-03-28T23:33:51.015114+0000","last_unstale":"2024-03-28T23:33:52.005011+0000","last_undegraded":"2024-03-28T23:33:52.005011+0000","last_fullsized":"2024-03-28T23:33:52.005011+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:36:06.301567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.019071+0000","last_change":"2024-03-28T23:33:51.019199+0000","last_active":"2024-03-28T23:33:52.019071+0000","last_peered":"2024-03-28T23:33:52.019071+0000","last_clean":"2024-03-28T23:33:52.019071+0000","last_became_active":"2024-03-28T23:33:51.018737+0000","last_became_peered":"2024-03-28T23:33:51.018737+0000","last_unstale":"2024-03-28T23:33:52.019071+0000","last_undegraded":"2024-03-28T23:33:52.019071+0000","last_fullsized":"2024-03-28T23:33:52.019071+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:07:26.080830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.004269+0000","last_change":"2024-03-28T23:33:51.008289+0000","last_active":"2024-03-28T23:33:52.004269+0000","last_peered":"2024-03-28T23:33:52.004269+0000","last_clean":"2024-03-28T23:33:52.004269+0000","last_became_active":"2024-03-28T23:33:51.008018+0000","last_became_peered":"2024-03-28T23:33:51.008018+0000","last_unstale":"2024-03-28T23:33:52.004269+0000","last_undegraded":"2024-03-28T23:33:52.004269+0000","last_fullsized":"2024-03-28T23:33:52.004269+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:32.390009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074761+0000","last_change":"2024-03-28T23:33:51.020584+0000","last_active":"2024-03-28T23:33:53.074761+0000","last_peered":"2024-03-28T23:33:53.074761+0000","last_clean":"2024-03-28T23:33:53.074761+0000","last_became_active":"2024-03-28T23:33:51.020321+0000","last_became_peered":"2024-03-28T23:33:51.020321+0000","last_unstale":"2024-03-28T23:33:53.074761+0000","last_undegraded":"2024-03-28T23:33:53.074761+0000","last_fullsized":"2024-03-28T23:33:53.074761+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:26.450839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005300+0000","last_change":"2024-03-28T23:33:51.014905+0000","last_active":"2024-03-28T23:33:52.005300+0000","last_peered":"2024-03-28T23:33:52.005300+0000","last_clean":"2024-03-28T23:33:52.005300+0000","last_became_active":"2024-03-28T23:33:51.014638+0000","last_became_peered":"2024-03-28T23:33:51.014638+0000","last_unstale":"2024-03-28T23:33:52.005300+0000","last_undegraded":"2024-03-28T23:33:52.005300+0000","last_fullsized":"2024-03-28T23:33:52.005300+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:24:29.812480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.018588+0000","last_change":"2024-03-28T23:33:51.017765+0000","last_active":"2024-03-28T23:33:52.018588+0000","last_peered":"2024-03-28T23:33:52.018588+0000","last_clean":"2024-03-28T23:33:52.018588+0000","last_became_active":"2024-03-28T23:33:51.017500+0000","last_became_peered":"2024-03-28T23:33:51.017500+0000","last_unstale":"2024-03-28T23:33:52.018588+0000","last_undegraded":"2024-03-28T23:33:52.018588+0000","last_fullsized":"2024-03-28T23:33:52.018588+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:43.384007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.004038+0000","last_change":"2024-03-28T23:33:51.011874+0000","last_active":"2024-03-28T23:33:52.004038+0000","last_peered":"2024-03-28T23:33:52.004038+0000","last_clean":"2024-03-28T23:33:52.004038+0000","last_became_active":"2024-03-28T23:33:51.011590+0000","last_became_peered":"2024-03-28T23:33:51.011590+0000","last_unstale":"2024-03-28T23:33:52.004038+0000","last_undegraded":"2024-03-28T23:33:52.004038+0000","last_fullsized":"2024-03-28T23:33:52.004038+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:20:01.738504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074424+0000","last_change":"2024-03-28T23:33:51.019461+0000","last_active":"2024-03-28T23:33:53.074424+0000","last_peered":"2024-03-28T23:33:53.074424+0000","last_clean":"2024-03-28T23:33:53.074424+0000","last_became_active":"2024-03-28T23:33:51.018352+0000","last_became_peered":"2024-03-28T23:33:51.018352+0000","last_unstale":"2024-03-28T23:33:53.074424+0000","last_undegraded":"2024-03-28T23:33:53.074424+0000","last_fullsized":"2024-03-28T23:33:53.074424+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:56:05.822381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005614+0000","last_change":"2024-03-28T23:33:51.018848+0000","last_active":"2024-03-28T23:33:52.005614+0000","last_peered":"2024-03-28T23:33:52.005614+0000","last_clean":"2024-03-28T23:33:52.005614+0000","last_became_active":"2024-03-28T23:33:51.018598+0000","last_became_peered":"2024-03-28T23:33:51.018598+0000","last_unstale":"2024-03-28T23:33:52.005614+0000","last_undegraded":"2024-03-28T23:33:52.005614+0000","last_fullsized":"2024-03-28T23:33:52.005614+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:21:03.177601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074352+0000","last_change":"2024-03-28T23:33:51.015079+0000","last_active":"2024-03-28T23:33:53.074352+0000","last_peered":"2024-03-28T23:33:53.074352+0000","last_clean":"2024-03-28T23:33:53.074352+0000","last_became_active":"2024-03-28T23:33:51.014426+0000","last_became_peered":"2024-03-28T23:33:51.014426+0000","last_unstale":"2024-03-28T23:33:53.074352+0000","last_undegraded":"2024-03-28T23:33:53.074352+0000","last_fullsized":"2024-03-28T23:33:53.074352+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:49.229683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074632+0000","last_change":"2024-03-28T23:33:51.020292+0000","last_active":"2024-03-28T23:33:53.074632+0000","last_peered":"2024-03-28T23:33:53.074632+0000","last_clean":"2024-03-28T23:33:53.074632+0000","last_became_active":"2024-03-28T23:33:51.018341+0000","last_became_peered":"2024-03-28T23:33:51.018341+0000","last_unstale":"2024-03-28T23:33:53.074632+0000","last_undegraded":"2024-03-28T23:33:53.074632+0000","last_fullsized":"2024-03-28T23:33:53.074632+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:16:18.521824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005189+0000","last_change":"2024-03-28T23:33:51.025449+0000","last_active":"2024-03-28T23:33:52.005189+0000","last_peered":"2024-03-28T23:33:52.005189+0000","last_clean":"2024-03-28T23:33:52.005189+0000","last_became_active":"2024-03-28T23:33:51.025176+0000","last_became_peered":"2024-03-28T23:33:51.025176+0000","last_unstale":"2024-03-28T23:33:52.005189+0000","last_undegraded":"2024-03-28T23:33:52.005189+0000","last_fullsized":"2024-03-28T23:33:52.005189+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:04.935939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074570+0000","last_change":"2024-03-28T23:33:51.026442+0000","last_active":"2024-03-28T23:33:53.074570+0000","last_peered":"2024-03-28T23:33:53.074570+0000","last_clean":"2024-03-28T23:33:53.074570+0000","last_became_active":"2024-03-28T23:33:51.026161+0000","last_became_peered":"2024-03-28T23:33:51.026161+0000","last_unstale":"2024-03-28T23:33:53.074570+0000","last_undegraded":"2024-03-28T23:33:53.074570+0000","last_fullsized":"2024-03-28T23:33:53.074570+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:39:40.687068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.073966+0000","last_change":"2024-03-28T23:33:51.010612+0000","last_active":"2024-03-28T23:33:53.073966+0000","last_peered":"2024-03-28T23:33:53.073966+0000","last_clean":"2024-03-28T23:33:53.073966+0000","last_became_active":"2024-03-28T23:33:51.010354+0000","last_became_peered":"2024-03-28T23:33:51.010354+0000","last_unstale":"2024-03-28T23:33:53.073966+0000","last_undegraded":"2024-03-28T23:33:53.073966+0000","last_fullsized":"2024-03-28T23:33:53.073966+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:24:29.812480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074183+0000","last_change":"2024-03-28T23:33:51.009461+0000","last_active":"2024-03-28T23:33:53.074183+0000","last_peered":"2024-03-28T23:33:53.074183+0000","last_clean":"2024-03-28T23:33:53.074183+0000","last_became_active":"2024-03-28T23:33:51.009177+0000","last_became_peered":"2024-03-28T23:33:51.009177+0000","last_unstale":"2024-03-28T23:33:53.074183+0000","last_undegraded":"2024-03-28T23:33:53.074183+0000","last_fullsized":"2024-03-28T23:33:53.074183+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:16:18.521824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074284+0000","last_change":"2024-03-28T23:33:51.008394+0000","last_active":"2024-03-28T23:33:53.074284+0000","last_peered":"2024-03-28T23:33:53.074284+0000","last_clean":"2024-03-28T23:33:53.074284+0000","last_became_active":"2024-03-28T23:33:51.008129+0000","last_became_peered":"2024-03-28T23:33:51.008129+0000","last_unstale":"2024-03-28T23:33:53.074284+0000","last_undegraded":"2024-03-28T23:33:53.074284+0000","last_fullsized":"2024-03-28T23:33:53.074284+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:16.800756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005679+0000","last_change":"2024-03-28T23:33:51.013537+0000","last_active":"2024-03-28T23:33:52.005679+0000","last_peered":"2024-03-28T23:33:52.005679+0000","last_clean":"2024-03-28T23:33:52.005679+0000","last_became_active":"2024-03-28T23:33:51.013276+0000","last_became_peered":"2024-03-28T23:33:51.013276+0000","last_unstale":"2024-03-28T23:33:52.005679+0000","last_undegraded":"2024-03-28T23:33:52.005679+0000","last_fullsized":"2024-03-28T23:33:52.005679+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:13.274788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.019130+0000","last_change":"2024-03-28T23:33:51.021129+0000","last_active":"2024-03-28T23:33:52.019130+0000","last_peered":"2024-03-28T23:33:52.019130+0000","last_clean":"2024-03-28T23:33:52.019130+0000","last_became_active":"2024-03-28T23:33:51.020837+0000","last_became_peered":"2024-03-28T23:33:51.020837+0000","last_unstale":"2024-03-28T23:33:52.019130+0000","last_undegraded":"2024-03-28T23:33:52.019130+0000","last_fullsized":"2024-03-28T23:33:52.019130+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:21.360493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074708+0000","last_change":"2024-03-28T23:33:51.019890+0000","last_active":"2024-03-28T23:33:53.074708+0000","last_peered":"2024-03-28T23:33:53.074708+0000","last_clean":"2024-03-28T23:33:53.074708+0000","last_became_active":"2024-03-28T23:33:51.019443+0000","last_became_peered":"2024-03-28T23:33:51.019443+0000","last_unstale":"2024-03-28T23:33:53.074708+0000","last_undegraded":"2024-03-28T23:33:53.074708+0000","last_fullsized":"2024-03-28T23:33:53.074708+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:37.146490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074891+0000","last_change":"2024-03-28T23:33:51.023744+0000","last_active":"2024-03-28T23:33:53.074891+0000","last_peered":"2024-03-28T23:33:53.074891+0000","last_clean":"2024-03-28T23:33:53.074891+0000","last_became_active":"2024-03-28T23:33:51.023477+0000","last_became_peered":"2024-03-28T23:33:51.023477+0000","last_unstale":"2024-03-28T23:33:53.074891+0000","last_undegraded":"2024-03-28T23:33:53.074891+0000","last_fullsized":"2024-03-28T23:33:53.074891+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:34.632115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074051+0000","last_change":"2024-03-28T23:33:51.011502+0000","last_active":"2024-03-28T23:33:53.074051+0000","last_peered":"2024-03-28T23:33:53.074051+0000","last_clean":"2024-03-28T23:33:53.074051+0000","last_became_active":"2024-03-28T23:33:51.011155+0000","last_became_peered":"2024-03-28T23:33:51.011155+0000","last_unstale":"2024-03-28T23:33:53.074051+0000","last_undegraded":"2024-03-28T23:33:53.074051+0000","last_fullsized":"2024-03-28T23:33:53.074051+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:37.206474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.073793+0000","last_change":"2024-03-28T23:33:51.011257+0000","last_active":"2024-03-28T23:33:53.073793+0000","last_peered":"2024-03-28T23:33:53.073793+0000","last_clean":"2024-03-28T23:33:53.073793+0000","last_became_active":"2024-03-28T23:33:51.010958+0000","last_became_peered":"2024-03-28T23:33:51.010958+0000","last_unstale":"2024-03-28T23:33:53.073793+0000","last_undegraded":"2024-03-28T23:33:53.073793+0000","last_fullsized":"2024-03-28T23:33:53.073793+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:04.935939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.004103+0000","last_change":"2024-03-28T23:33:51.021643+0000","last_active":"2024-03-28T23:33:52.004103+0000","last_peered":"2024-03-28T23:33:52.004103+0000","last_clean":"2024-03-28T23:33:52.004103+0000","last_became_active":"2024-03-28T23:33:51.021384+0000","last_became_peered":"2024-03-28T23:33:51.021384+0000","last_unstale":"2024-03-28T23:33:52.004103+0000","last_undegraded":"2024-03-28T23:33:52.004103+0000","last_fullsized":"2024-03-28T23:33:52.004103+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:37.206474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"108'4","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005258+0000","last_change":"2024-03-28T23:33:49.985988+0000","last_active":"2024-03-28T23:33:52.005258+0000","last_peered":"2024-03-28T23:33:52.005258+0000","last_clean":"2024-03-28T23:33:52.005258+0000","last_became_active":"2024-03-28T23:33:49.985665+0000","last_became_peered":"2024-03-28T23:33:49.985665+0000","last_unstale":"2024-03-28T23:33:52.005258+0000","last_undegraded":"2024-03-28T23:33:52.005258+0000","last_fullsized":"2024-03-28T23:33:52.005258+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:40.272936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.018723+0000","last_change":"2024-03-28T23:33:51.030837+0000","last_active":"2024-03-28T23:33:52.018723+0000","last_peered":"2024-03-28T23:33:52.018723+0000","last_clean":"2024-03-28T23:33:52.018723+0000","last_became_active":"2024-03-28T23:33:51.030546+0000","last_became_peered":"2024-03-28T23:33:51.030546+0000","last_unstale":"2024-03-28T23:33:52.018723+0000","last_undegraded":"2024-03-28T23:33:52.018723+0000","last_fullsized":"2024-03-28T23:33:52.018723+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:03:02.359745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.003781+0000","last_change":"2024-03-28T23:33:49.987198+0000","last_active":"2024-03-28T23:33:52.003781+0000","last_peered":"2024-03-28T23:33:52.003781+0000","last_clean":"2024-03-28T23:33:52.003781+0000","last_became_active":"2024-03-28T23:33:49.986675+0000","last_became_peered":"2024-03-28T23:33:49.986675+0000","last_unstale":"2024-03-28T23:33:52.003781+0000","last_undegraded":"2024-03-28T23:33:52.003781+0000","last_fullsized":"2024-03-28T23:33:52.003781+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:10:54.386639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074118+0000","last_change":"2024-03-28T23:33:51.016090+0000","last_active":"2024-03-28T23:33:53.074118+0000","last_peered":"2024-03-28T23:33:53.074118+0000","last_clean":"2024-03-28T23:33:53.074118+0000","last_became_active":"2024-03-28T23:33:51.015781+0000","last_became_peered":"2024-03-28T23:33:51.015781+0000","last_unstale":"2024-03-28T23:33:53.074118+0000","last_undegraded":"2024-03-28T23:33:53.074118+0000","last_fullsized":"2024-03-28T23:33:53.074118+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:32.390009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074738+0000","last_change":"2024-03-28T23:33:49.992114+0000","last_active":"2024-03-28T23:33:53.074738+0000","last_peered":"2024-03-28T23:33:53.074738+0000","last_clean":"2024-03-28T23:33:53.074738+0000","last_became_active":"2024-03-28T23:33:49.991817+0000","last_became_peered":"2024-03-28T23:33:49.991817+0000","last_unstale":"2024-03-28T23:33:53.074738+0000","last_undegraded":"2024-03-28T23:33:53.074738+0000","last_fullsized":"2024-03-28T23:33:53.074738+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:10:24.895984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.075039+0000","last_change":"2024-03-28T23:33:51.017959+0000","last_active":"2024-03-28T23:33:53.075039+0000","last_peered":"2024-03-28T23:33:53.075039+0000","last_clean":"2024-03-28T23:33:53.075039+0000","last_became_active":"2024-03-28T23:33:51.017699+0000","last_became_peered":"2024-03-28T23:33:51.017699+0000","last_unstale":"2024-03-28T23:33:53.075039+0000","last_undegraded":"2024-03-28T23:33:53.075039+0000","last_fullsized":"2024-03-28T23:33:53.075039+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:38:29.769002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.004091+0000","last_change":"2024-03-28T23:33:49.985075+0000","last_active":"2024-03-28T23:33:52.004091+0000","last_peered":"2024-03-28T23:33:52.004091+0000","last_clean":"2024-03-28T23:33:52.004091+0000","last_became_active":"2024-03-28T23:33:49.984788+0000","last_became_peered":"2024-03-28T23:33:49.984788+0000","last_unstale":"2024-03-28T23:33:52.004091+0000","last_undegraded":"2024-03-28T23:33:52.004091+0000","last_fullsized":"2024-03-28T23:33:52.004091+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:49:01.660610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.003684+0000","last_change":"2024-03-28T23:33:51.009658+0000","last_active":"2024-03-28T23:33:52.003684+0000","last_peered":"2024-03-28T23:33:52.003684+0000","last_clean":"2024-03-28T23:33:52.003684+0000","last_became_active":"2024-03-28T23:33:51.009400+0000","last_became_peered":"2024-03-28T23:33:51.009400+0000","last_unstale":"2024-03-28T23:33:52.003684+0000","last_undegraded":"2024-03-28T23:33:52.003684+0000","last_fullsized":"2024-03-28T23:33:52.003684+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:51:09.523264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005426+0000","last_change":"2024-03-28T23:33:49.984472+0000","last_active":"2024-03-28T23:33:52.005426+0000","last_peered":"2024-03-28T23:33:52.005426+0000","last_clean":"2024-03-28T23:33:52.005426+0000","last_became_active":"2024-03-28T23:33:49.984161+0000","last_became_peered":"2024-03-28T23:33:49.984161+0000","last_unstale":"2024-03-28T23:33:52.005426+0000","last_undegraded":"2024-03-28T23:33:52.005426+0000","last_fullsized":"2024-03-28T23:33:52.005426+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:52.093172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005490+0000","last_change":"2024-03-28T23:33:51.012950+0000","last_active":"2024-03-28T23:33:52.005490+0000","last_peered":"2024-03-28T23:33:52.005490+0000","last_clean":"2024-03-28T23:33:52.005490+0000","last_became_active":"2024-03-28T23:33:51.012660+0000","last_became_peered":"2024-03-28T23:33:51.012660+0000","last_unstale":"2024-03-28T23:33:52.005490+0000","last_undegraded":"2024-03-28T23:33:52.005490+0000","last_fullsized":"2024-03-28T23:33:52.005490+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:20:01.738504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.018771+0000","last_change":"2024-03-28T23:33:49.997011+0000","last_active":"2024-03-28T23:33:52.018771+0000","last_peered":"2024-03-28T23:33:52.018771+0000","last_clean":"2024-03-28T23:33:52.018771+0000","last_became_active":"2024-03-28T23:33:49.996748+0000","last_became_peered":"2024-03-28T23:33:49.996748+0000","last_unstale":"2024-03-28T23:33:52.018771+0000","last_undegraded":"2024-03-28T23:33:52.018771+0000","last_fullsized":"2024-03-28T23:33:52.018771+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:29:19.407500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.075006+0000","last_change":"2024-03-28T23:33:51.015284+0000","last_active":"2024-03-28T23:33:53.075006+0000","last_peered":"2024-03-28T23:33:53.075006+0000","last_clean":"2024-03-28T23:33:53.075006+0000","last_became_active":"2024-03-28T23:33:51.015003+0000","last_became_peered":"2024-03-28T23:33:51.015003+0000","last_unstale":"2024-03-28T23:33:53.075006+0000","last_undegraded":"2024-03-28T23:33:53.075006+0000","last_fullsized":"2024-03-28T23:33:53.075006+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:11.329576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.073806+0000","last_change":"2024-03-28T23:33:49.988293+0000","last_active":"2024-03-28T23:33:53.073806+0000","last_peered":"2024-03-28T23:33:53.073806+0000","last_clean":"2024-03-28T23:33:53.073806+0000","last_became_active":"2024-03-28T23:33:49.987865+0000","last_became_peered":"2024-03-28T23:33:49.987865+0000","last_unstale":"2024-03-28T23:33:53.073806+0000","last_undegraded":"2024-03-28T23:33:53.073806+0000","last_fullsized":"2024-03-28T23:33:53.073806+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:21:02.175819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.003834+0000","last_change":"2024-03-28T23:33:51.022417+0000","last_active":"2024-03-28T23:33:52.003834+0000","last_peered":"2024-03-28T23:33:52.003834+0000","last_clean":"2024-03-28T23:33:52.003834+0000","last_became_active":"2024-03-28T23:33:51.022177+0000","last_became_peered":"2024-03-28T23:33:51.022177+0000","last_unstale":"2024-03-28T23:33:52.003834+0000","last_undegraded":"2024-03-28T23:33:52.003834+0000","last_fullsized":"2024-03-28T23:33:52.003834+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:04.935939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.018717+0000","last_change":"2024-03-28T23:33:49.992491+0000","last_active":"2024-03-28T23:33:52.018717+0000","last_peered":"2024-03-28T23:33:52.018717+0000","last_clean":"2024-03-28T23:33:52.018717+0000","last_became_active":"2024-03-28T23:33:49.992224+0000","last_became_peered":"2024-03-28T23:33:49.992224+0000","last_unstale":"2024-03-28T23:33:52.018717+0000","last_undegraded":"2024-03-28T23:33:52.018717+0000","last_fullsized":"2024-03-28T23:33:52.018717+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:38.377566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.004894+0000","last_change":"2024-03-28T23:33:51.014515+0000","last_active":"2024-03-28T23:33:52.004894+0000","last_peered":"2024-03-28T23:33:52.004894+0000","last_clean":"2024-03-28T23:33:52.004894+0000","last_became_active":"2024-03-28T23:33:51.014168+0000","last_became_peered":"2024-03-28T23:33:51.014168+0000","last_unstale":"2024-03-28T23:33:52.004894+0000","last_undegraded":"2024-03-28T23:33:52.004894+0000","last_fullsized":"2024-03-28T23:33:52.004894+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:49:02.662391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005374+0000","last_change":"2024-03-28T23:33:49.997600+0000","last_active":"2024-03-28T23:33:52.005374+0000","last_peered":"2024-03-28T23:33:52.005374+0000","last_clean":"2024-03-28T23:33:52.005374+0000","last_became_active":"2024-03-28T23:33:49.997288+0000","last_became_peered":"2024-03-28T23:33:49.997288+0000","last_unstale":"2024-03-28T23:33:52.005374+0000","last_undegraded":"2024-03-28T23:33:52.005374+0000","last_fullsized":"2024-03-28T23:33:52.005374+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:00:59.695434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074115+0000","last_change":"2024-03-28T23:33:51.025385+0000","last_active":"2024-03-28T23:33:53.074115+0000","last_peered":"2024-03-28T23:33:53.074115+0000","last_clean":"2024-03-28T23:33:53.074115+0000","last_became_active":"2024-03-28T23:33:51.025059+0000","last_became_peered":"2024-03-28T23:33:51.025059+0000","last_unstale":"2024-03-28T23:33:53.074115+0000","last_undegraded":"2024-03-28T23:33:53.074115+0000","last_fullsized":"2024-03-28T23:33:53.074115+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:39:40.687068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.073983+0000","last_change":"2024-03-28T23:33:49.989460+0000","last_active":"2024-03-28T23:33:53.073983+0000","last_peered":"2024-03-28T23:33:53.073983+0000","last_clean":"2024-03-28T23:33:53.073983+0000","last_became_active":"2024-03-28T23:33:49.989123+0000","last_became_peered":"2024-03-28T23:33:49.989123+0000","last_unstale":"2024-03-28T23:33:53.073983+0000","last_undegraded":"2024-03-28T23:33:53.073983+0000","last_fullsized":"2024-03-28T23:33:53.073983+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:14:19.604793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074547+0000","last_change":"2024-03-28T23:33:51.016203+0000","last_active":"2024-03-28T23:33:53.074547+0000","last_peered":"2024-03-28T23:33:53.074547+0000","last_clean":"2024-03-28T23:33:53.074547+0000","last_became_active":"2024-03-28T23:33:51.015936+0000","last_became_peered":"2024-03-28T23:33:51.015936+0000","last_unstale":"2024-03-28T23:33:53.074547+0000","last_undegraded":"2024-03-28T23:33:53.074547+0000","last_fullsized":"2024-03-28T23:33:53.074547+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:45:44.498188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"108'3","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.073981+0000","last_change":"2024-03-28T23:33:49.990570+0000","last_active":"2024-03-28T23:33:53.073981+0000","last_peered":"2024-03-28T23:33:53.073981+0000","last_clean":"2024-03-28T23:33:53.073981+0000","last_became_active":"2024-03-28T23:33:49.987820+0000","last_became_peered":"2024-03-28T23:33:49.987820+0000","last_unstale":"2024-03-28T23:33:53.073981+0000","last_undegraded":"2024-03-28T23:33:53.073981+0000","last_fullsized":"2024-03-28T23:33:53.073981+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:12.273007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074240+0000","last_change":"2024-03-28T23:33:51.012003+0000","last_active":"2024-03-28T23:33:53.074240+0000","last_peered":"2024-03-28T23:33:53.074240+0000","last_clean":"2024-03-28T23:33:53.074240+0000","last_became_active":"2024-03-28T23:33:51.011736+0000","last_became_peered":"2024-03-28T23:33:51.011736+0000","last_unstale":"2024-03-28T23:33:53.074240+0000","last_undegraded":"2024-03-28T23:33:53.074240+0000","last_fullsized":"2024-03-28T23:33:53.074240+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:59:52.655622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"108'6","reported_seq":20,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074912+0000","last_change":"2024-03-28T23:33:49.990203+0000","last_active":"2024-03-28T23:33:53.074912+0000","last_peered":"2024-03-28T23:33:53.074912+0000","last_clean":"2024-03-28T23:33:53.074912+0000","last_became_active":"2024-03-28T23:33:49.989962+0000","last_became_peered":"2024-03-28T23:33:49.989962+0000","last_unstale":"2024-03-28T23:33:53.074912+0000","last_undegraded":"2024-03-28T23:33:53.074912+0000","last_fullsized":"2024-03-28T23:33:53.074912+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:20:00.736722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005141+0000","last_change":"2024-03-28T23:33:51.019810+0000","last_active":"2024-03-28T23:33:52.005141+0000","last_peered":"2024-03-28T23:33:52.005141+0000","last_clean":"2024-03-28T23:33:52.005141+0000","last_became_active":"2024-03-28T23:33:51.019552+0000","last_became_peered":"2024-03-28T23:33:51.019552+0000","last_unstale":"2024-03-28T23:33:52.005141+0000","last_undegraded":"2024-03-28T23:33:52.005141+0000","last_fullsized":"2024-03-28T23:33:52.005141+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:14:20.606574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074741+0000","last_change":"2024-03-28T23:33:49.984539+0000","last_active":"2024-03-28T23:33:53.074741+0000","last_peered":"2024-03-28T23:33:53.074741+0000","last_clean":"2024-03-28T23:33:53.074741+0000","last_became_active":"2024-03-28T23:33:49.982619+0000","last_became_peered":"2024-03-28T23:33:49.982619+0000","last_unstale":"2024-03-28T23:33:53.074741+0000","last_undegraded":"2024-03-28T23:33:53.074741+0000","last_fullsized":"2024-03-28T23:33:53.074741+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:26:16.906247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.003692+0000","last_change":"2024-03-28T23:33:51.014751+0000","last_active":"2024-03-28T23:33:52.003692+0000","last_peered":"2024-03-28T23:33:52.003692+0000","last_clean":"2024-03-28T23:33:52.003692+0000","last_became_active":"2024-03-28T23:33:51.014491+0000","last_became_peered":"2024-03-28T23:33:51.014491+0000","last_unstale":"2024-03-28T23:33:52.003692+0000","last_undegraded":"2024-03-28T23:33:52.003692+0000","last_fullsized":"2024-03-28T23:33:52.003692+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:02:49.240053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.075040+0000","last_change":"2024-03-28T23:33:49.991532+0000","last_active":"2024-03-28T23:33:53.075040+0000","last_peered":"2024-03-28T23:33:53.075040+0000","last_clean":"2024-03-28T23:33:53.075040+0000","last_became_active":"2024-03-28T23:33:49.991276+0000","last_became_peered":"2024-03-28T23:33:49.991276+0000","last_unstale":"2024-03-28T23:33:53.075040+0000","last_undegraded":"2024-03-28T23:33:53.075040+0000","last_fullsized":"2024-03-28T23:33:53.075040+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:03.934158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074205+0000","last_change":"2024-03-28T23:33:51.006212+0000","last_active":"2024-03-28T23:33:53.074205+0000","last_peered":"2024-03-28T23:33:53.074205+0000","last_clean":"2024-03-28T23:33:53.074205+0000","last_became_active":"2024-03-28T23:33:51.005936+0000","last_became_peered":"2024-03-28T23:33:51.005936+0000","last_unstale":"2024-03-28T23:33:53.074205+0000","last_undegraded":"2024-03-28T23:33:53.074205+0000","last_fullsized":"2024-03-28T23:33:53.074205+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:51:25.515271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074294+0000","last_change":"2024-03-28T23:33:49.989507+0000","last_active":"2024-03-28T23:33:53.074294+0000","last_peered":"2024-03-28T23:33:53.074294+0000","last_clean":"2024-03-28T23:33:53.074294+0000","last_became_active":"2024-03-28T23:33:49.988693+0000","last_became_peered":"2024-03-28T23:33:49.988693+0000","last_unstale":"2024-03-28T23:33:53.074294+0000","last_undegraded":"2024-03-28T23:33:53.074294+0000","last_fullsized":"2024-03-28T23:33:53.074294+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:36:05.299785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074046+0000","last_change":"2024-03-28T23:33:51.016137+0000","last_active":"2024-03-28T23:33:53.074046+0000","last_peered":"2024-03-28T23:33:53.074046+0000","last_clean":"2024-03-28T23:33:53.074046+0000","last_became_active":"2024-03-28T23:33:51.015837+0000","last_became_peered":"2024-03-28T23:33:51.015837+0000","last_unstale":"2024-03-28T23:33:53.074046+0000","last_undegraded":"2024-03-28T23:33:53.074046+0000","last_fullsized":"2024-03-28T23:33:53.074046+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:20:01.738504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.004089+0000","last_change":"2024-03-28T23:33:49.997252+0000","last_active":"2024-03-28T23:33:52.004089+0000","last_peered":"2024-03-28T23:33:52.004089+0000","last_clean":"2024-03-28T23:33:52.004089+0000","last_became_active":"2024-03-28T23:33:49.996952+0000","last_became_peered":"2024-03-28T23:33:49.996952+0000","last_unstale":"2024-03-28T23:33:52.004089+0000","last_undegraded":"2024-03-28T23:33:52.004089+0000","last_fullsized":"2024-03-28T23:33:52.004089+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:26:48.812133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005426+0000","last_change":"2024-03-28T23:33:51.025429+0000","last_active":"2024-03-28T23:33:52.005426+0000","last_peered":"2024-03-28T23:33:52.005426+0000","last_clean":"2024-03-28T23:33:52.005426+0000","last_became_active":"2024-03-28T23:33:51.025144+0000","last_became_peered":"2024-03-28T23:33:51.025144+0000","last_unstale":"2024-03-28T23:33:52.005426+0000","last_undegraded":"2024-03-28T23:33:52.005426+0000","last_fullsized":"2024-03-28T23:33:52.005426+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:26:17.908028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074414+0000","last_change":"2024-03-28T23:33:49.997415+0000","last_active":"2024-03-28T23:33:53.074414+0000","last_peered":"2024-03-28T23:33:53.074414+0000","last_clean":"2024-03-28T23:33:53.074414+0000","last_became_active":"2024-03-28T23:33:49.997127+0000","last_became_peered":"2024-03-28T23:33:49.997127+0000","last_unstale":"2024-03-28T23:33:53.074414+0000","last_undegraded":"2024-03-28T23:33:53.074414+0000","last_fullsized":"2024-03-28T23:33:53.074414+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:45:33.988210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.018913+0000","last_change":"2024-03-28T23:33:51.017947+0000","last_active":"2024-03-28T23:33:52.018913+0000","last_peered":"2024-03-28T23:33:52.018913+0000","last_clean":"2024-03-28T23:33:52.018913+0000","last_became_active":"2024-03-28T23:33:51.017357+0000","last_became_peered":"2024-03-28T23:33:51.017357+0000","last_unstale":"2024-03-28T23:33:52.018913+0000","last_undegraded":"2024-03-28T23:33:52.018913+0000","last_fullsized":"2024-03-28T23:33:52.018913+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:27:23.718523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074747+0000","last_change":"2024-03-28T23:33:49.992128+0000","last_active":"2024-03-28T23:33:53.074747+0000","last_peered":"2024-03-28T23:33:53.074747+0000","last_clean":"2024-03-28T23:33:53.074747+0000","last_became_active":"2024-03-28T23:33:49.991464+0000","last_became_peered":"2024-03-28T23:33:49.991464+0000","last_unstale":"2024-03-28T23:33:53.074747+0000","last_undegraded":"2024-03-28T23:33:53.074747+0000","last_fullsized":"2024-03-28T23:33:53.074747+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:16:39.848586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074333+0000","last_change":"2024-03-28T23:33:51.015993+0000","last_active":"2024-03-28T23:33:53.074333+0000","last_peered":"2024-03-28T23:33:53.074333+0000","last_clean":"2024-03-28T23:33:53.074333+0000","last_became_active":"2024-03-28T23:33:51.013143+0000","last_became_peered":"2024-03-28T23:33:51.013143+0000","last_unstale":"2024-03-28T23:33:53.074333+0000","last_undegraded":"2024-03-28T23:33:53.074333+0000","last_fullsized":"2024-03-28T23:33:53.074333+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:13.109884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074525+0000","last_change":"2024-03-28T23:33:49.998016+0000","last_active":"2024-03-28T23:33:53.074525+0000","last_peered":"2024-03-28T23:33:53.074525+0000","last_clean":"2024-03-28T23:33:53.074525+0000","last_became_active":"2024-03-28T23:33:49.997751+0000","last_became_peered":"2024-03-28T23:33:49.997751+0000","last_unstale":"2024-03-28T23:33:53.074525+0000","last_undegraded":"2024-03-28T23:33:53.074525+0000","last_fullsized":"2024-03-28T23:33:53.074525+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:45:51.352057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.018665+0000","last_change":"2024-03-28T23:33:51.016437+0000","last_active":"2024-03-28T23:33:52.018665+0000","last_peered":"2024-03-28T23:33:52.018665+0000","last_clean":"2024-03-28T23:33:52.018665+0000","last_became_active":"2024-03-28T23:33:51.016142+0000","last_became_peered":"2024-03-28T23:33:51.016142+0000","last_unstale":"2024-03-28T23:33:52.018665+0000","last_undegraded":"2024-03-28T23:33:52.018665+0000","last_fullsized":"2024-03-28T23:33:52.018665+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:47:01.158311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.018581+0000","last_change":"2024-03-28T23:33:49.993146+0000","last_active":"2024-03-28T23:33:52.018581+0000","last_peered":"2024-03-28T23:33:52.018581+0000","last_clean":"2024-03-28T23:33:52.018581+0000","last_became_active":"2024-03-28T23:33:49.992913+0000","last_became_peered":"2024-03-28T23:33:49.992913+0000","last_unstale":"2024-03-28T23:33:52.018581+0000","last_undegraded":"2024-03-28T23:33:52.018581+0000","last_fullsized":"2024-03-28T23:33:52.018581+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:04:47.661317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074086+0000","last_change":"2024-03-28T23:33:51.023128+0000","last_active":"2024-03-28T23:33:53.074086+0000","last_peered":"2024-03-28T23:33:53.074086+0000","last_clean":"2024-03-28T23:33:53.074086+0000","last_became_active":"2024-03-28T23:33:51.022835+0000","last_became_peered":"2024-03-28T23:33:51.022835+0000","last_unstale":"2024-03-28T23:33:53.074086+0000","last_undegraded":"2024-03-28T23:33:53.074086+0000","last_fullsized":"2024-03-28T23:33:53.074086+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:43:24.492761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074451+0000","last_change":"2024-03-28T23:33:49.990123+0000","last_active":"2024-03-28T23:33:53.074451+0000","last_peered":"2024-03-28T23:33:53.074451+0000","last_clean":"2024-03-28T23:33:53.074451+0000","last_became_active":"2024-03-28T23:33:49.989834+0000","last_became_peered":"2024-03-28T23:33:49.989834+0000","last_unstale":"2024-03-28T23:33:53.074451+0000","last_undegraded":"2024-03-28T23:33:53.074451+0000","last_fullsized":"2024-03-28T23:33:53.074451+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:52.093172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.075118+0000","last_change":"2024-03-28T23:33:51.015947+0000","last_active":"2024-03-28T23:33:53.075118+0000","last_peered":"2024-03-28T23:33:53.075118+0000","last_clean":"2024-03-28T23:33:53.075118+0000","last_became_active":"2024-03-28T23:33:51.015599+0000","last_became_peered":"2024-03-28T23:33:51.015599+0000","last_unstale":"2024-03-28T23:33:53.075118+0000","last_undegraded":"2024-03-28T23:33:53.075118+0000","last_fullsized":"2024-03-28T23:33:53.075118+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:02.377786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074299+0000","last_change":"2024-03-28T23:33:49.996325+0000","last_active":"2024-03-28T23:33:53.074299+0000","last_peered":"2024-03-28T23:33:53.074299+0000","last_clean":"2024-03-28T23:33:53.074299+0000","last_became_active":"2024-03-28T23:33:49.996067+0000","last_became_peered":"2024-03-28T23:33:49.996067+0000","last_unstale":"2024-03-28T23:33:53.074299+0000","last_undegraded":"2024-03-28T23:33:53.074299+0000","last_fullsized":"2024-03-28T23:33:53.074299+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:24:28.810699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074007+0000","last_change":"2024-03-28T23:33:51.005474+0000","last_active":"2024-03-28T23:33:53.074007+0000","last_peered":"2024-03-28T23:33:53.074007+0000","last_clean":"2024-03-28T23:33:53.074007+0000","last_became_active":"2024-03-28T23:33:51.005209+0000","last_became_peered":"2024-03-28T23:33:51.005209+0000","last_unstale":"2024-03-28T23:33:53.074007+0000","last_undegraded":"2024-03-28T23:33:53.074007+0000","last_fullsized":"2024-03-28T23:33:53.074007+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:26:17.908028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.073795+0000","last_change":"2024-03-28T23:33:49.990740+0000","last_active":"2024-03-28T23:33:53.073795+0000","last_peered":"2024-03-28T23:33:53.073795+0000","last_clean":"2024-03-28T23:33:53.073795+0000","last_became_active":"2024-03-28T23:33:49.990494+0000","last_became_peered":"2024-03-28T23:33:49.990494+0000","last_unstale":"2024-03-28T23:33:53.073795+0000","last_undegraded":"2024-03-28T23:33:53.073795+0000","last_fullsized":"2024-03-28T23:33:53.073795+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:38:37.720292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.019213+0000","last_change":"2024-03-28T23:33:51.027238+0000","last_active":"2024-03-28T23:33:52.019213+0000","last_peered":"2024-03-28T23:33:52.019213+0000","last_clean":"2024-03-28T23:33:52.019213+0000","last_became_active":"2024-03-28T23:33:51.026460+0000","last_became_peered":"2024-03-28T23:33:51.026460+0000","last_unstale":"2024-03-28T23:33:52.019213+0000","last_undegraded":"2024-03-28T23:33:52.019213+0000","last_fullsized":"2024-03-28T23:33:52.019213+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:34:07.840166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.019041+0000","last_change":"2024-03-28T23:33:49.996464+0000","last_active":"2024-03-28T23:33:52.019041+0000","last_peered":"2024-03-28T23:33:52.019041+0000","last_clean":"2024-03-28T23:33:52.019041+0000","last_became_active":"2024-03-28T23:33:49.996122+0000","last_became_peered":"2024-03-28T23:33:49.996122+0000","last_unstale":"2024-03-28T23:33:52.019041+0000","last_undegraded":"2024-03-28T23:33:52.019041+0000","last_fullsized":"2024-03-28T23:33:52.019041+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:46:50.856475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.003796+0000","last_change":"2024-03-28T23:33:51.010596+0000","last_active":"2024-03-28T23:33:52.003796+0000","last_peered":"2024-03-28T23:33:52.003796+0000","last_clean":"2024-03-28T23:33:52.003796+0000","last_became_active":"2024-03-28T23:33:51.010331+0000","last_became_peered":"2024-03-28T23:33:51.010331+0000","last_unstale":"2024-03-28T23:33:52.003796+0000","last_undegraded":"2024-03-28T23:33:52.003796+0000","last_fullsized":"2024-03-28T23:33:52.003796+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:36:06.301567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"109'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005175+0000","last_change":"2024-03-28T23:33:49.997526+0000","last_active":"2024-03-28T23:33:52.005175+0000","last_peered":"2024-03-28T23:33:52.005175+0000","last_clean":"2024-03-28T23:33:52.005175+0000","last_became_active":"2024-03-28T23:33:49.997216+0000","last_became_peered":"2024-03-28T23:33:49.997216+0000","last_unstale":"2024-03-28T23:33:52.005175+0000","last_undegraded":"2024-03-28T23:33:52.005175+0000","last_fullsized":"2024-03-28T23:33:52.005175+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:33:09.234630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074924+0000","last_change":"2024-03-28T23:33:51.012495+0000","last_active":"2024-03-28T23:33:53.074924+0000","last_peered":"2024-03-28T23:33:53.074924+0000","last_clean":"2024-03-28T23:33:53.074924+0000","last_became_active":"2024-03-28T23:33:51.012232+0000","last_became_peered":"2024-03-28T23:33:51.012232+0000","last_unstale":"2024-03-28T23:33:53.074924+0000","last_undegraded":"2024-03-28T23:33:53.074924+0000","last_fullsized":"2024-03-28T23:33:53.074924+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:09.825723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005673+0000","last_change":"2024-03-28T23:33:49.997622+0000","last_active":"2024-03-28T23:33:52.005673+0000","last_peered":"2024-03-28T23:33:52.005673+0000","last_clean":"2024-03-28T23:33:52.005673+0000","last_became_active":"2024-03-28T23:33:49.997247+0000","last_became_peered":"2024-03-28T23:33:49.997247+0000","last_unstale":"2024-03-28T23:33:52.005673+0000","last_undegraded":"2024-03-28T23:33:52.005673+0000","last_fullsized":"2024-03-28T23:33:52.005673+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:59:21.391833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.075123+0000","last_change":"2024-03-28T23:33:51.013400+0000","last_active":"2024-03-28T23:33:53.075123+0000","last_peered":"2024-03-28T23:33:53.075123+0000","last_clean":"2024-03-28T23:33:53.075123+0000","last_became_active":"2024-03-28T23:33:51.013130+0000","last_became_peered":"2024-03-28T23:33:51.013130+0000","last_unstale":"2024-03-28T23:33:53.075123+0000","last_undegraded":"2024-03-28T23:33:53.075123+0000","last_fullsized":"2024-03-28T23:33:53.075123+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:10:50.716440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.019188+0000","last_change":"2024-03-28T23:33:49.996259+0000","last_active":"2024-03-28T23:33:52.019188+0000","last_peered":"2024-03-28T23:33:52.019188+0000","last_clean":"2024-03-28T23:33:52.019188+0000","last_became_active":"2024-03-28T23:33:49.996003+0000","last_became_peered":"2024-03-28T23:33:49.996003+0000","last_unstale":"2024-03-28T23:33:52.019188+0000","last_undegraded":"2024-03-28T23:33:52.019188+0000","last_fullsized":"2024-03-28T23:33:52.019188+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:01:59.777890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.018783+0000","last_change":"2024-03-28T23:33:51.016991+0000","last_active":"2024-03-28T23:33:52.018783+0000","last_peered":"2024-03-28T23:33:52.018783+0000","last_clean":"2024-03-28T23:33:52.018783+0000","last_became_active":"2024-03-28T23:33:51.016731+0000","last_became_peered":"2024-03-28T23:33:51.016731+0000","last_unstale":"2024-03-28T23:33:52.018783+0000","last_undegraded":"2024-03-28T23:33:52.018783+0000","last_fullsized":"2024-03-28T23:33:52.018783+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:29:13.673822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005653+0000","last_change":"2024-03-28T23:33:49.988649+0000","last_active":"2024-03-28T23:33:52.005653+0000","last_peered":"2024-03-28T23:33:52.005653+0000","last_clean":"2024-03-28T23:33:52.005653+0000","last_became_active":"2024-03-28T23:33:49.988374+0000","last_became_peered":"2024-03-28T23:33:49.988374+0000","last_unstale":"2024-03-28T23:33:52.005653+0000","last_undegraded":"2024-03-28T23:33:52.005653+0000","last_fullsized":"2024-03-28T23:33:52.005653+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:38:37.720292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074238+0000","last_change":"2024-03-28T23:33:51.019372+0000","last_active":"2024-03-28T23:33:53.074238+0000","last_peered":"2024-03-28T23:33:53.074238+0000","last_clean":"2024-03-28T23:33:53.074238+0000","last_became_active":"2024-03-28T23:33:51.019096+0000","last_became_peered":"2024-03-28T23:33:51.019096+0000","last_unstale":"2024-03-28T23:33:53.074238+0000","last_undegraded":"2024-03-28T23:33:53.074238+0000","last_fullsized":"2024-03-28T23:33:53.074238+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:48:21.194198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.005259+0000","last_change":"2024-03-28T23:33:49.985591+0000","last_active":"2024-03-28T23:33:52.005259+0000","last_peered":"2024-03-28T23:33:52.005259+0000","last_clean":"2024-03-28T23:33:52.005259+0000","last_became_active":"2024-03-28T23:33:49.985344+0000","last_became_peered":"2024-03-28T23:33:49.985344+0000","last_unstale":"2024-03-28T23:33:52.005259+0000","last_undegraded":"2024-03-28T23:33:52.005259+0000","last_fullsized":"2024-03-28T23:33:52.005259+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:12.273007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074555+0000","last_change":"2024-03-28T23:33:51.019705+0000","last_active":"2024-03-28T23:33:53.074555+0000","last_peered":"2024-03-28T23:33:53.074555+0000","last_clean":"2024-03-28T23:33:53.074555+0000","last_became_active":"2024-03-28T23:33:51.018300+0000","last_became_peered":"2024-03-28T23:33:51.018300+0000","last_unstale":"2024-03-28T23:33:53.074555+0000","last_undegraded":"2024-03-28T23:33:53.074555+0000","last_fullsized":"2024-03-28T23:33:53.074555+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:32.390009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.018583+0000","last_change":"2024-03-28T23:33:49.996343+0000","last_active":"2024-03-28T23:33:52.018583+0000","last_peered":"2024-03-28T23:33:52.018583+0000","last_clean":"2024-03-28T23:33:52.018583+0000","last_became_active":"2024-03-28T23:33:49.996095+0000","last_became_peered":"2024-03-28T23:33:49.996095+0000","last_unstale":"2024-03-28T23:33:52.018583+0000","last_undegraded":"2024-03-28T23:33:52.018583+0000","last_fullsized":"2024-03-28T23:33:52.018583+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:35:17.358659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:52.003969+0000","last_change":"2024-03-28T23:33:51.018684+0000","last_active":"2024-03-28T23:33:52.003969+0000","last_peered":"2024-03-28T23:33:52.003969+0000","last_clean":"2024-03-28T23:33:52.003969+0000","last_became_active":"2024-03-28T23:33:51.015122+0000","last_became_peered":"2024-03-28T23:33:51.015122+0000","last_unstale":"2024-03-28T23:33:52.003969+0000","last_undegraded":"2024-03-28T23:33:52.003969+0000","last_fullsized":"2024-03-28T23:33:52.003969+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:26:17.908028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074214+0000","last_change":"2024-03-28T23:33:49.988606+0000","last_active":"2024-03-28T23:33:53.074214+0000","last_peered":"2024-03-28T23:33:53.074214+0000","last_clean":"2024-03-28T23:33:53.074214+0000","last_became_active":"2024-03-28T23:33:49.988351+0000","last_became_peered":"2024-03-28T23:33:49.988351+0000","last_unstale":"2024-03-28T23:33:53.074214+0000","last_undegraded":"2024-03-28T23:33:53.074214+0000","last_fullsized":"2024-03-28T23:33:53.074214+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:39:39.102964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074755+0000","last_change":"2024-03-28T23:33:51.020599+0000","last_active":"2024-03-28T23:33:53.074755+0000","last_peered":"2024-03-28T23:33:53.074755+0000","last_clean":"2024-03-28T23:33:53.074755+0000","last_became_active":"2024-03-28T23:33:51.020020+0000","last_became_peered":"2024-03-28T23:33:51.020020+0000","last_unstale":"2024-03-28T23:33:53.074755+0000","last_undegraded":"2024-03-28T23:33:53.074755+0000","last_fullsized":"2024-03-28T23:33:53.074755+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:49.970798+0000","last_clean_scrub_stamp":"2024-03-28T23:33:49.970798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:13.109884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"108'8","reported_seq":22,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074034+0000","last_change":"2024-03-28T23:33:49.982727+0000","last_active":"2024-03-28T23:33:53.074034+0000","last_peered":"2024-03-28T23:33:53.074034+0000","last_clean":"2024-03-28T23:33:53.074034+0000","last_became_active":"2024-03-28T23:33:49.982456+0000","last_became_peered":"2024-03-28T23:33:49.982456+0000","last_unstale":"2024-03-28T23:33:53.074034+0000","last_undegraded":"2024-03-28T23:33:53.074034+0000","last_fullsized":"2024-03-28T23:33:53.074034+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:33:48.969017+0000","last_clean_scrub_stamp":"2024-03-28T23:33:48.969017+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:02:48.238272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":338,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-28T23:33:53.074293+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:33:53.074293+0000","last_peered":"2024-03-28T23:33:53.074293+0000","last_clean":"2024-03-28T23:33:53.074293+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:33:53.074293+0000","last_undegraded":"2024-03-28T23:33:53.074293+0000","last_fullsized":"2024-03-28T23:33:53.074293+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574901,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32588,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339252,"statfs":{"total":96636764160,"available":96603394048,"internally_reserved":0,"allocated":1441792,"data_stored":1247559,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32516,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339324,"statfs":{"total":96636764160,"available":96603467776,"internally_reserved":0,"allocated":1363968,"data_stored":656997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574902,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32520,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339320,"statfs":{"total":96636764160,"available":96603463680,"internally_reserved":0,"allocated":1368064,"data_stored":656575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574901,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32524,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339316,"statfs":{"total":96636764160,"available":96603459584,"internally_reserved":0,"allocated":1372160,"data_stored":658027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32528,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339312,"statfs":{"total":96636764160,"available":96603455488,"internally_reserved":0,"allocated":1376256,"data_stored":658061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574901,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1359872,"data_stored":656415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32516,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339324,"statfs":{"total":96636764160,"available":96603467776,"internally_reserved":0,"allocated":1363968,"data_stored":656437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5],"upper_bound":8},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574901,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1429504,"data_stored":1246805,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:33:58.885 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T23:33:58.885 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-28T23:33:59.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.035+0000 7f8e1281c640 1 Processor -- start 2024-03-28T23:33:59.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.035+0000 7f8e1281c640 1 -- start start 2024-03-28T23:33:59.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.036+0000 7f8e1281c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8e0c0c6750 0x7f8e0c06bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:59.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.036+0000 7f8e1281c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e0c152db0 con 0x7f8e0c0c88b0 2024-03-28T23:33:59.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.036+0000 7f8e1281c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e0c0c8d40 con 0x7f8e0c1529d0 2024-03-28T23:33:59.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.036+0000 7f8e1281c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f8e0c0c7210 con 0x7f8e0c0c6750 2024-03-28T23:33:59.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.036+0000 7f8e10db9640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f8e0c0c88b0 0x7f8e0c0c54f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41506/0 (socket says 172.21.15.161:41506) 2024-03-28T23:33:59.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.036+0000 7f8e10db9640 1 -- 172.21.15.161:0/1350744161 learned_addr learned my addr 172.21.15.161:0/1350744161 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:59.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.037+0000 7f8e0b7fe640 1 --2- 172.21.15.161:0/1350744161 >> v2:172.21.15.170:3300/0 conn(0x7f8e0c0c6750 0x7f8e0c06bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:59.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.037+0000 7f8e0affd640 1 -- 172.21.15.161:0/1350744161 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4069721414 0 0) 0x7f8e0c152db0 con 0x7f8e0c0c88b0 2024-03-28T23:33:59.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.037+0000 7f8e0affd640 1 -- 172.21.15.161:0/1350744161 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8df4003520 con 0x7f8e0c0c88b0 2024-03-28T23:33:59.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.037+0000 7f8e0affd640 1 -- 172.21.15.161:0/1350744161 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3606770195 0 0) 0x7f8df4003520 con 0x7f8e0c0c88b0 2024-03-28T23:33:59.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.037+0000 7f8e0affd640 1 -- 172.21.15.161:0/1350744161 >> v2:172.21.15.170:3300/0 conn(0x7f8e0c0c6750 msgr2=0x7f8e0c06bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:59.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.037+0000 7f8e0affd640 1 --2- 172.21.15.161:0/1350744161 >> v2:172.21.15.170:3300/0 conn(0x7f8e0c0c6750 0x7f8e0c06bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:59.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.037+0000 7f8e0affd640 1 -- 172.21.15.161:0/1350744161 >> v1:172.21.15.170:6789/0 conn(0x7f8e0c1529d0 legacy=0x7f8e0c155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:59.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.037+0000 7f8e0affd640 1 -- 172.21.15.161:0/1350744161 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e0c0c6c50 con 0x7f8e0c0c88b0 2024-03-28T23:33:59.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.037+0000 7f8e0affd640 1 -- 172.21.15.161:0/1350744161 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f8e040019f0 con 0x7f8e0c0c88b0 2024-03-28T23:33:59.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.037+0000 7f8e0affd640 1 -- 172.21.15.161:0/1350744161 >> v1:172.21.15.161:6789/0 conn(0x7f8e0c0c88b0 legacy=0x7f8e0c0c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:59.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.037+0000 7f8e0affd640 1 --2- 172.21.15.161:0/1350744161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8df40039b0 0x7f8df4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:59.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.037+0000 7f8e0affd640 1 --2- 172.21.15.161:0/1350744161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8df4004800 0x7f8df4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:59.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.037+0000 7f8e0affd640 1 --2- 172.21.15.161:0/1350744161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8df40051c0 0x7f8df400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:59.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.037+0000 7f8e0affd640 1 -- 172.21.15.161:0/1350744161 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0c0c7210 con 0x7f8df40051c0 2024-03-28T23:33:59.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.037+0000 7f8e0affd640 1 -- 172.21.15.161:0/1350744161 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8df400c060 con 0x7f8df40039b0 2024-03-28T23:33:59.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.037+0000 7f8e0affd640 1 -- 172.21.15.161:0/1350744161 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8df400c260 con 0x7f8df4004800 2024-03-28T23:33:59.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.037+0000 7f8e10db9640 1 --2- 172.21.15.161:0/1350744161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8df40051c0 0x7f8df400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:59.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.037+0000 7f8e0bfff640 1 --2- 172.21.15.161:0/1350744161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8df40039b0 0x7f8df4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:59.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.037+0000 7f8e0b7fe640 1 --2- 172.21.15.161:0/1350744161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8df4004800 0x7f8df4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:59.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.037+0000 7f8e10db9640 1 -- 172.21.15.161:0/1350744161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8df4004800 msgr2=0x7f8df4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:59.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.037+0000 7f8e10db9640 1 --2- 172.21.15.161:0/1350744161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8df4004800 0x7f8df4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:59.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.037+0000 7f8e10db9640 1 -- 172.21.15.161:0/1350744161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8df40039b0 msgr2=0x7f8df4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:59.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.037+0000 7f8e10db9640 1 --2- 172.21.15.161:0/1350744161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8df40039b0 0x7f8df4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:59.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.037+0000 7f8e10db9640 1 -- 172.21.15.161:0/1350744161 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df4003520 con 0x7f8df40051c0 2024-03-28T23:33:59.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.038+0000 7f8e10db9640 1 -- 172.21.15.161:0/1350744161 reap_dead start 2024-03-28T23:33:59.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.038+0000 7f8e10db9640 1 --2- 172.21.15.161:0/1350744161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8df40051c0 0x7f8df400b9c0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f8e0400a850 tx=0x7f8df4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=941b432cc34393e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:59.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.038+0000 7f8e0affd640 1 -- 172.21.15.161:0/1350744161 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e0400a4a0 con 0x7f8df40051c0 2024-03-28T23:33:59.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.038+0000 7f8e0affd640 1 -- 172.21.15.161:0/1350744161 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e04034650 con 0x7f8df40051c0 2024-03-28T23:33:59.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.038+0000 7f8e0affd640 1 -- 172.21.15.161:0/1350744161 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e04034a40 con 0x7f8df40051c0 2024-03-28T23:33:59.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.039+0000 7f8e1281c640 1 -- 172.21.15.161:0/1350744161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8df40051c0 msgr2=0x7f8df400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:59.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.039+0000 7f8e1281c640 1 --2- 172.21.15.161:0/1350744161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8df40051c0 0x7f8df400b9c0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f8e0400a850 tx=0x7f8df4005830 comp rx=0 tx=0).stop 2024-03-28T23:33:59.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.039+0000 7f8e1281c640 1 -- 172.21.15.161:0/1350744161 shutdown_connections 2024-03-28T23:33:59.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.039+0000 7f8e1281c640 1 --2- 172.21.15.161:0/1350744161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8df40051c0 0x7f8df400b9c0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:59.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.039+0000 7f8e1281c640 1 -- 172.21.15.161:0/1350744161 >> 172.21.15.161:0/1350744161 conn(0x7f8e0c0c7430 msgr2=0x7f8e0c0c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:59.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.039+0000 7f8e1281c640 1 -- 172.21.15.161:0/1350744161 shutdown_connections 2024-03-28T23:33:59.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.039+0000 7f8e1281c640 1 -- 172.21.15.161:0/1350744161 wait complete. 2024-03-28T23:33:59.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.040+0000 7f8e1281c640 1 Processor -- start 2024-03-28T23:33:59.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.040+0000 7f8e1281c640 1 -- start start 2024-03-28T23:33:59.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.041+0000 7f8e1281c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8df40051c0 0x7f8e0c159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:59.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.041+0000 7f8e1281c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e0c0c05e0 0x7f8e0c149eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:59.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.041+0000 7f8e1281c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8e0c14a3f0 0x7f8e0c1ee250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:59.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.041+0000 7f8e1281c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0c155710 con 0x7f8df40051c0 2024-03-28T23:33:59.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.041+0000 7f8e1281c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0c155430 con 0x7f8e0c0c05e0 2024-03-28T23:33:59.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.041+0000 7f8e1281c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8e0c0aa160 con 0x7f8e0c14a3f0 2024-03-28T23:33:59.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.041+0000 7f8e0bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8df40051c0 0x7f8e0c159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:59.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.041+0000 7f8e0bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8df40051c0 0x7f8e0c159440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60196/0 (socket says 172.21.15.161:60196) 2024-03-28T23:33:59.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.041+0000 7f8e0bfff640 1 -- 172.21.15.161:0/3572996816 learned_addr learned my addr 172.21.15.161:0/3572996816 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:59.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.041+0000 7f8e10db9640 1 --2- 172.21.15.161:0/3572996816 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8e0c14a3f0 0x7f8e0c1ee250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:59.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.041+0000 7f8e0b7fe640 1 --2- 172.21.15.161:0/3572996816 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e0c0c05e0 0x7f8e0c149eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:59.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.041+0000 7f8e0bfff640 1 -- 172.21.15.161:0/3572996816 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8e0c14a3f0 msgr2=0x7f8e0c1ee250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:59.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.041+0000 7f8e0bfff640 1 --2- 172.21.15.161:0/3572996816 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8e0c14a3f0 0x7f8e0c1ee250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:59.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.041+0000 7f8e0bfff640 1 -- 172.21.15.161:0/3572996816 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e0c0c05e0 msgr2=0x7f8e0c149eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:59.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.041+0000 7f8e0bfff640 1 --2- 172.21.15.161:0/3572996816 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e0c0c05e0 0x7f8e0c149eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:59.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.041+0000 7f8e0bfff640 1 -- 172.21.15.161:0/3572996816 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e0c145b70 con 0x7f8df40051c0 2024-03-28T23:33:59.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.041+0000 7f8e0bfff640 1 --2- 172.21.15.161:0/3572996816 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8df40051c0 0x7f8e0c159440 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00f620 tx=0x7f8dfc00fa20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:59.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.042+0000 7f8e08ff9640 1 -- 172.21.15.161:0/3572996816 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc0052e0 con 0x7f8df40051c0 2024-03-28T23:33:59.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.042+0000 7f8e08ff9640 1 -- 172.21.15.161:0/3572996816 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dfc018070 con 0x7f8df40051c0 2024-03-28T23:33:59.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.042+0000 7f8e08ff9640 1 -- 172.21.15.161:0/3572996816 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc003810 con 0x7f8df40051c0 2024-03-28T23:33:59.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.042+0000 7f8e1281c640 1 -- 172.21.15.161:0/3572996816 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e0c1eadf0 con 0x7f8df40051c0 2024-03-28T23:33:59.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.042+0000 7f8e1281c640 1 -- 172.21.15.161:0/3572996816 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e0c1eee00 con 0x7f8df40051c0 2024-03-28T23:33:59.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.043+0000 7f8e1281c640 1 -- 172.21.15.161:0/3572996816 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e0c04f4b0 con 0x7f8df40051c0 2024-03-28T23:33:59.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.043+0000 7f8e08ff9640 1 -- 172.21.15.161:0/3572996816 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8dfc0039b0 con 0x7f8df40051c0 2024-03-28T23:33:59.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.044+0000 7f8e08ff9640 1 --2- 172.21.15.161:0/3572996816 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8df806b500 0x7f8df806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:59.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.044+0000 7f8e0b7fe640 1 --2- 172.21.15.161:0/3572996816 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8df806b500 0x7f8df806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:59.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.044+0000 7f8e08ff9640 1 -- 172.21.15.161:0/3572996816 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6101+0+0 (secure 0 0 0) 0x7f8dfc089f40 con 0x7f8df40051c0 2024-03-28T23:33:59.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.044+0000 7f8e0b7fe640 1 --2- 172.21.15.161:0/3572996816 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8df806b500 0x7f8df806d9c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c148270 tx=0x7f8e00009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:59.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.048+0000 7f8e08ff9640 1 -- 172.21.15.161:0/3572996816 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8dfc058c30 con 0x7f8df40051c0 2024-03-28T23:33:59.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.293+0000 7f8e1281c640 1 -- 172.21.15.161:0/3572996816 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f8e0c0c0ba0 con 0x7f8df40051c0 2024-03-28T23:33:59.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.295+0000 7f8e08ff9640 1 -- 172.21.15.161:0/3572996816 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) ==== 190+0+147 (secure 0 0 0) 0x7f8dfc058530 con 0x7f8df40051c0 2024-03-28T23:33:59.295 INFO:teuthology.orchestra.run.smithi161.stdout:[client.0] 2024-03-28T23:33:59.296 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQA6/QVmc1krAxAAEKsn1Tuuv9OlFZpB4PFhDg== 2024-03-28T23:33:59.296 INFO:teuthology.orchestra.run.smithi161.stdout: caps mds = "allow" 2024-03-28T23:33:59.296 INFO:teuthology.orchestra.run.smithi161.stdout: caps mon = "allow r" 2024-03-28T23:33:59.296 INFO:teuthology.orchestra.run.smithi161.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-28T23:33:59.296 INFO:teuthology.orchestra.run.smithi161.stderr:updated caps for client.0 2024-03-28T23:33:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.297+0000 7f8e1281c640 1 -- 172.21.15.161:0/3572996816 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8df806b500 msgr2=0x7f8df806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.297+0000 7f8e1281c640 1 --2- 172.21.15.161:0/3572996816 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8df806b500 0x7f8df806d9c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c148270 tx=0x7f8e00009190 comp rx=0 tx=0).stop 2024-03-28T23:33:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.297+0000 7f8e1281c640 1 -- 172.21.15.161:0/3572996816 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8df40051c0 msgr2=0x7f8e0c159440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.297+0000 7f8e1281c640 1 --2- 172.21.15.161:0/3572996816 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8df40051c0 0x7f8e0c159440 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00f620 tx=0x7f8dfc00fa20 comp rx=0 tx=0).stop 2024-03-28T23:33:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.297+0000 7f8e1281c640 1 -- 172.21.15.161:0/3572996816 shutdown_connections 2024-03-28T23:33:59.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.297+0000 7f8e1281c640 1 --2- 172.21.15.161:0/3572996816 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8df806b500 0x7f8df806d9c0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:59.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.297+0000 7f8e1281c640 1 --2- 172.21.15.161:0/3572996816 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8e0c14a3f0 0x7f8e0c1ee250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:59.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.297+0000 7f8e1281c640 1 --2- 172.21.15.161:0/3572996816 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e0c0c05e0 0x7f8e0c149eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:59.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.297+0000 7f8e1281c640 1 --2- 172.21.15.161:0/3572996816 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8df40051c0 0x7f8e0c159440 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:59.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.297+0000 7f8e1281c640 1 -- 172.21.15.161:0/3572996816 >> 172.21.15.161:0/3572996816 conn(0x7f8e0c0c7430 msgr2=0x7f8e0c1501f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:59.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.297+0000 7f8e1281c640 1 -- 172.21.15.161:0/3572996816 shutdown_connections 2024-03-28T23:33:59.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.297+0000 7f8e1281c640 1 -- 172.21.15.161:0/3572996816 wait complete. 2024-03-28T23:33:59.309 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:33:59.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.412+0000 7f153a5a1640 1 Processor -- start 2024-03-28T23:33:59.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.412+0000 7f153a5a1640 1 -- start start 2024-03-28T23:33:59.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.413+0000 7f153a5a1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15340c5410 0x7f15340c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:59.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.413+0000 7f153a5a1640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1534158a30 con 0x7f1534152890 2024-03-28T23:33:59.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.413+0000 7f153a5a1640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1534158d10 con 0x7f15341553a0 2024-03-28T23:33:59.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.413+0000 7f153a5a1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f15340c8480 con 0x7f15340c5410 2024-03-28T23:33:59.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.413+0000 7f1533fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f1534152890 0x7f15340c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41518/0 (socket says 172.21.15.161:41518) 2024-03-28T23:33:59.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.413+0000 7f1533fff640 1 -- 172.21.15.161:0/3940642388 learned_addr learned my addr 172.21.15.161:0/3940642388 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:33:59.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.413+0000 7f1532ffd640 1 -- 172.21.15.161:0/3940642388 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3783342130 0 0) 0x7f1534158a30 con 0x7f1534152890 2024-03-28T23:33:59.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f1532ffd640 1 -- 172.21.15.161:0/3940642388 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f151c003520 con 0x7f1534152890 2024-03-28T23:33:59.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f15337fe640 1 --2- 172.21.15.161:0/3940642388 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15340c5410 0x7f15340c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:59.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f1532ffd640 1 -- 172.21.15.161:0/3940642388 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 411025598 0 0) 0x7f151c003520 con 0x7f1534152890 2024-03-28T23:33:59.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f1532ffd640 1 -- 172.21.15.161:0/3940642388 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15340c5410 msgr2=0x7f15340c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:59.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f1532ffd640 1 --2- 172.21.15.161:0/3940642388 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15340c5410 0x7f15340c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:59.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f1532ffd640 1 -- 172.21.15.161:0/3940642388 >> v1:172.21.15.170:6789/0 conn(0x7f15341553a0 legacy=0x7f1534155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:59.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f1532ffd640 1 -- 172.21.15.161:0/3940642388 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1534158ff0 con 0x7f1534152890 2024-03-28T23:33:59.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f1532ffd640 1 -- 172.21.15.161:0/3940642388 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f1528001670 con 0x7f1534152890 2024-03-28T23:33:59.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f1532ffd640 1 -- 172.21.15.161:0/3940642388 >> v1:172.21.15.161:6789/0 conn(0x7f1534152890 legacy=0x7f15340c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:59.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f1532ffd640 1 --2- 172.21.15.161:0/3940642388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f151c003ac0 0x7f151c003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:59.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f1532ffd640 1 --2- 172.21.15.161:0/3940642388 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f151c004800 0x7f151c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:59.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f1532ffd640 1 --2- 172.21.15.161:0/3940642388 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15340c5410 0x7f151c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:59.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f1532ffd640 1 -- 172.21.15.161:0/3940642388 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f151c0054f0 con 0x7f151c004800 2024-03-28T23:33:59.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f1532ffd640 1 -- 172.21.15.161:0/3940642388 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f151c00bf00 con 0x7f151c003ac0 2024-03-28T23:33:59.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f1532ffd640 1 -- 172.21.15.161:0/3940642388 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f151c00c080 con 0x7f15340c5410 2024-03-28T23:33:59.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f15337fe640 1 --2- 172.21.15.161:0/3940642388 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f151c004800 0x7f151c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:59.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f1538b3e640 1 --2- 172.21.15.161:0/3940642388 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15340c5410 0x7f151c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:59.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f1533fff640 1 --2- 172.21.15.161:0/3940642388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f151c003ac0 0x7f151c003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:59.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f15337fe640 1 -- 172.21.15.161:0/3940642388 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15340c5410 msgr2=0x7f151c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:59.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f15337fe640 1 --2- 172.21.15.161:0/3940642388 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15340c5410 0x7f151c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:59.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f15337fe640 1 -- 172.21.15.161:0/3940642388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f151c003ac0 msgr2=0x7f151c003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:33:59.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f15337fe640 1 --2- 172.21.15.161:0/3940642388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f151c003ac0 0x7f151c003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:59.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f15337fe640 1 -- 172.21.15.161:0/3940642388 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f151c003520 con 0x7f151c004800 2024-03-28T23:33:59.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f1538b3e640 1 --2- 172.21.15.161:0/3940642388 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15340c5410 0x7f151c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:59.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.414+0000 7f1533fff640 1 --2- 172.21.15.161:0/3940642388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f151c003ac0 0x7f151c003ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:33:59.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.415+0000 7f15337fe640 1 --2- 172.21.15.161:0/3940642388 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f151c004800 0x7f151c004c80 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f1524009a80 tx=0x7f151c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27bdc39197eea89f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:59.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.415+0000 7f1532ffd640 1 -- 172.21.15.161:0/3940642388 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1524009680 con 0x7f151c004800 2024-03-28T23:33:59.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.415+0000 7f1532ffd640 1 -- 172.21.15.161:0/3940642388 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1524005990 con 0x7f151c004800 2024-03-28T23:33:59.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.415+0000 7f1532ffd640 1 -- 172.21.15.161:0/3940642388 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1524005d40 con 0x7f151c004800 2024-03-28T23:33:59.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.416+0000 7f153a5a1640 1 -- 172.21.15.161:0/3940642388 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f151c004800 msgr2=0x7f151c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:59.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.416+0000 7f153a5a1640 1 --2- 172.21.15.161:0/3940642388 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f151c004800 0x7f151c004c80 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f1524009a80 tx=0x7f151c005830 comp rx=0 tx=0).stop 2024-03-28T23:33:59.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.416+0000 7f1538b3e640 1 -- 172.21.15.161:0/3940642388 reap_dead start 2024-03-28T23:33:59.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.416+0000 7f153a5a1640 1 -- 172.21.15.161:0/3940642388 shutdown_connections 2024-03-28T23:33:59.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.416+0000 7f153a5a1640 1 -- 172.21.15.161:0/3940642388 >> 172.21.15.161:0/3940642388 conn(0x7f15340ac620 msgr2=0x7f15340aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:59.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.416+0000 7f153a5a1640 1 -- 172.21.15.161:0/3940642388 shutdown_connections 2024-03-28T23:33:59.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.416+0000 7f153a5a1640 1 -- 172.21.15.161:0/3940642388 wait complete. 2024-03-28T23:33:59.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.417+0000 7f153a5a1640 1 Processor -- start 2024-03-28T23:33:59.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.417+0000 7f153a5a1640 1 -- start start 2024-03-28T23:33:59.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.418+0000 7f153a5a1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f151c004800 0x7f15341eebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:59.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.418+0000 7f153a5a1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15341516e0 0x7f153414d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:59.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.418+0000 7f153a5a1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15340ab6f0 0x7f1534150a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:59.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.418+0000 7f153a5a1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f15340c90a0 con 0x7f15340ab6f0 2024-03-28T23:33:59.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.418+0000 7f153a5a1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1534159640 con 0x7f15341516e0 2024-03-28T23:33:59.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.418+0000 7f153a5a1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f15340c9220 con 0x7f151c004800 2024-03-28T23:33:59.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.418+0000 7f1538b3e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15340ab6f0 0x7f1534150a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:59.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.418+0000 7f1538b3e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15340ab6f0 0x7f1534150a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60206/0 (socket says 172.21.15.161:60206) 2024-03-28T23:33:59.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.418+0000 7f1538b3e640 1 -- 172.21.15.161:0/4093244080 learned_addr learned my addr 172.21.15.161:0/4093244080 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:33:59.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.418+0000 7f1533fff640 1 --2- 172.21.15.161:0/4093244080 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f151c004800 0x7f15341eebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:59.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.418+0000 7f15337fe640 1 --2- 172.21.15.161:0/4093244080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15341516e0 0x7f153414d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:59.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.418+0000 7f1538b3e640 1 -- 172.21.15.161:0/4093244080 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f151c004800 msgr2=0x7f15341eebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:59.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.418+0000 7f1538b3e640 1 --2- 172.21.15.161:0/4093244080 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f151c004800 0x7f15341eebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:59.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.418+0000 7f1538b3e640 1 -- 172.21.15.161:0/4093244080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15341516e0 msgr2=0x7f153414d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:59.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.418+0000 7f1538b3e640 1 --2- 172.21.15.161:0/4093244080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15341516e0 0x7f153414d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:59.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.418+0000 7f1538b3e640 1 -- 172.21.15.161:0/4093244080 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f153414d860 con 0x7f15340ab6f0 2024-03-28T23:33:59.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.418+0000 7f1538b3e640 1 --2- 172.21.15.161:0/4093244080 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15340ab6f0 0x7f1534150a10 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f152c000f80 tx=0x7f152c00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:59.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.418+0000 7f1530ff9640 1 -- 172.21.15.161:0/4093244080 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f152c006b50 con 0x7f15340ab6f0 2024-03-28T23:33:59.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.419+0000 7f1530ff9640 1 -- 172.21.15.161:0/4093244080 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f152c006d10 con 0x7f15340ab6f0 2024-03-28T23:33:59.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.419+0000 7f153a5a1640 1 -- 172.21.15.161:0/4093244080 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1534151be0 con 0x7f15340ab6f0 2024-03-28T23:33:59.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.419+0000 7f1530ff9640 1 -- 172.21.15.161:0/4093244080 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f152c003970 con 0x7f15340ab6f0 2024-03-28T23:33:59.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.419+0000 7f153a5a1640 1 -- 172.21.15.161:0/4093244080 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1534150210 con 0x7f15340ab6f0 2024-03-28T23:33:59.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.420+0000 7f153a5a1640 1 -- 172.21.15.161:0/4093244080 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f153404f4b0 con 0x7f15340ab6f0 2024-03-28T23:33:59.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.420+0000 7f1530ff9640 1 -- 172.21.15.161:0/4093244080 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f152c01a020 con 0x7f15340ab6f0 2024-03-28T23:33:59.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.420+0000 7f1530ff9640 1 --2- 172.21.15.161:0/4093244080 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f152006b4d0 0x7f152006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:33:59.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.420+0000 7f1530ff9640 1 -- 172.21.15.161:0/4093244080 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6101+0+0 (secure 0 0 0) 0x7f152c0897b0 con 0x7f15340ab6f0 2024-03-28T23:33:59.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.420+0000 7f1533fff640 1 --2- 172.21.15.161:0/4093244080 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f152006b4d0 0x7f152006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:33:59.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.421+0000 7f1533fff640 1 --2- 172.21.15.161:0/4093244080 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f152006b4d0 0x7f152006d990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f15280017e0 tx=0x7f1528007040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:33:59.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.425+0000 7f1530ff9640 1 -- 172.21.15.161:0/4093244080 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f152c057ee0 con 0x7f15340ab6f0 2024-03-28T23:33:59.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.641+0000 7f153a5a1640 1 -- 172.21.15.161:0/4093244080 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1534074190 con 0x7f15340ab6f0 2024-03-28T23:33:59.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.642+0000 7f1530ff9640 1 -- 172.21.15.161:0/4093244080 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) ==== 94+0+5195 (secure 0 0 0) 0x7f152c05bf80 con 0x7f15340ab6f0 2024-03-28T23:33:59.642 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:33:59.642 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":90,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6203,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/1096504848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1096504848},{"type":"v1","addr":"172.21.15.170:6834","nonce":1096504848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":6206,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3351418222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3351418222},{"type":"v1","addr":"172.21.15.170:6835","nonce":3351418222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":6892,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/2684760895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2684760895},{"type":"v1","addr":"172.21.15.161:6837","nonce":2684760895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:33:50.326492+0000","modified":"2024-03-28T23:33:52.354643+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7452},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7452":{"gid":7452,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6835/2544838511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":2544838511},{"type":"v1","addr":"172.21.15.161:6835","nonce":2544838511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7452,"qdb_cluster":[7452]},"id":8}]} 2024-03-28T23:33:59.643 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 90 2024-03-28T23:33:59.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.643+0000 7f153a5a1640 1 -- 172.21.15.161:0/4093244080 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f152006b4d0 msgr2=0x7f152006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:59.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.643+0000 7f153a5a1640 1 --2- 172.21.15.161:0/4093244080 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f152006b4d0 0x7f152006d990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f15280017e0 tx=0x7f1528007040 comp rx=0 tx=0).stop 2024-03-28T23:33:59.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.643+0000 7f153a5a1640 1 -- 172.21.15.161:0/4093244080 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15340ab6f0 msgr2=0x7f1534150a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:33:59.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.643+0000 7f153a5a1640 1 --2- 172.21.15.161:0/4093244080 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15340ab6f0 0x7f1534150a10 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f152c000f80 tx=0x7f152c00eb90 comp rx=0 tx=0).stop 2024-03-28T23:33:59.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.643+0000 7f153a5a1640 1 -- 172.21.15.161:0/4093244080 shutdown_connections 2024-03-28T23:33:59.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.643+0000 7f153a5a1640 1 --2- 172.21.15.161:0/4093244080 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f152006b4d0 0x7f152006d990 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:59.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.643+0000 7f153a5a1640 1 --2- 172.21.15.161:0/4093244080 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15340ab6f0 0x7f1534150a10 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:59.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.643+0000 7f153a5a1640 1 --2- 172.21.15.161:0/4093244080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15341516e0 0x7f153414d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:59.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.643+0000 7f153a5a1640 1 --2- 172.21.15.161:0/4093244080 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f151c004800 0x7f15341eebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:33:59.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.643+0000 7f153a5a1640 1 -- 172.21.15.161:0/4093244080 >> 172.21.15.161:0/4093244080 conn(0x7f15340ac620 msgr2=0x7f15340ca240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:33:59.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.644+0000 7f153a5a1640 1 -- 172.21.15.161:0/4093244080 shutdown_connections 2024-03-28T23:33:59.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:33:59.644+0000 7f153a5a1640 1 -- 172.21.15.161:0/4093244080 wait complete. 2024-03-28T23:34:00.058 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 90, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:33:50.326492+0000', 'modified': '2024-03-28T23:33:52.354643+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7452}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7452': {'gid': 7452, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.161:6835/2544838511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 2544838511}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 2544838511}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7452, 'qdb_cluster': [7452]} 2024-03-28T23:34:00.059 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:34:00.059 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-28T23:34:00.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.162+0000 7f468cd7a640 1 Processor -- start 2024-03-28T23:34:00.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.162+0000 7f468cd7a640 1 -- start start 2024-03-28T23:34:00.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.164+0000 7f468cd7a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4688152890 0x7f46880c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.164+0000 7f468cd7a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f46880c5b20 0x7f46881553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.164+0000 7f468cd7a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4688158a30 con 0x7f46880c8ed0 2024-03-28T23:34:00.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.164+0000 7f468cd7a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f46880c8480 con 0x7f46880c5b20 2024-03-28T23:34:00.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.164+0000 7f468cd7a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4688159640 con 0x7f4688152890 2024-03-28T23:34:00.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.164+0000 7f467ffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f46880c8ed0 0x7f46880c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41530/0 (socket says 172.21.15.161:41530) 2024-03-28T23:34:00.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.164+0000 7f467ffff640 1 -- 172.21.15.161:0/2304752352 learned_addr learned my addr 172.21.15.161:0/2304752352 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:00.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.164+0000 7f4686d9d640 1 --2- 172.21.15.161:0/2304752352 >> v2:172.21.15.170:3300/0 conn(0x7f46880c5b20 0x7f46881553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.164+0000 7f468659c640 1 --2- 172.21.15.161:0/2304752352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4688152890 0x7f46880c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.164+0000 7f4685d9b640 1 -- 172.21.15.161:0/2304752352 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3306622770 0 0) 0x7f4688158a30 con 0x7f46880c8ed0 2024-03-28T23:34:00.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.164+0000 7f4685d9b640 1 -- 172.21.15.161:0/2304752352 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4674003520 con 0x7f46880c8ed0 2024-03-28T23:34:00.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.164+0000 7f4685d9b640 1 -- 172.21.15.161:0/2304752352 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 18024518 0 0) 0x7f4674003520 con 0x7f46880c8ed0 2024-03-28T23:34:00.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.164+0000 7f4685d9b640 1 -- 172.21.15.161:0/2304752352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4688152890 msgr2=0x7f46880c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:00.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.164+0000 7f4685d9b640 1 --2- 172.21.15.161:0/2304752352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4688152890 0x7f46880c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.164+0000 7f4685d9b640 1 -- 172.21.15.161:0/2304752352 >> v2:172.21.15.170:3300/0 conn(0x7f46880c5b20 msgr2=0x7f46881553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:00.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.164+0000 7f4685d9b640 1 --2- 172.21.15.161:0/2304752352 >> v2:172.21.15.170:3300/0 conn(0x7f46880c5b20 0x7f46881553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.164+0000 7f4685d9b640 1 -- 172.21.15.161:0/2304752352 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4688158d10 con 0x7f46880c8ed0 2024-03-28T23:34:00.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.164+0000 7f4685d9b640 1 -- 172.21.15.161:0/2304752352 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f46700017f0 con 0x7f46880c8ed0 2024-03-28T23:34:00.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.164+0000 7f4685d9b640 1 -- 172.21.15.161:0/2304752352 >> v1:172.21.15.161:6789/0 conn(0x7f46880c8ed0 legacy=0x7f46880c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:00.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.164+0000 7f4685d9b640 1 --2- 172.21.15.161:0/2304752352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4674003a90 0x7f4674003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.164+0000 7f4685d9b640 1 --2- 172.21.15.161:0/2304752352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4674004800 0x7f4674004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.165+0000 7f4685d9b640 1 --2- 172.21.15.161:0/2304752352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4688152890 0x7f46740051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.165+0000 7f4685d9b640 1 -- 172.21.15.161:0/2304752352 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4688159640 con 0x7f4674003a90 2024-03-28T23:34:00.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.165+0000 7f4685d9b640 1 -- 172.21.15.161:0/2304752352 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f467400bad0 con 0x7f4674004800 2024-03-28T23:34:00.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.165+0000 7f4685d9b640 1 -- 172.21.15.161:0/2304752352 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f467400bcd0 con 0x7f4688152890 2024-03-28T23:34:00.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.165+0000 7f468659c640 1 --2- 172.21.15.161:0/2304752352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4674003a90 0x7f4674003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.165+0000 7f467ffff640 1 --2- 172.21.15.161:0/2304752352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4674004800 0x7f4674004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.165+0000 7f4686d9d640 1 --2- 172.21.15.161:0/2304752352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4688152890 0x7f46740051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.165+0000 7f468659c640 1 -- 172.21.15.161:0/2304752352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4688152890 msgr2=0x7f46740051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:00.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.165+0000 7f468659c640 1 --2- 172.21.15.161:0/2304752352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4688152890 0x7f46740051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.165+0000 7f468659c640 1 -- 172.21.15.161:0/2304752352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4674004800 msgr2=0x7f4674004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:00.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.165+0000 7f468659c640 1 --2- 172.21.15.161:0/2304752352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4674004800 0x7f4674004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.165+0000 7f468659c640 1 -- 172.21.15.161:0/2304752352 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4674003520 con 0x7f4674003a90 2024-03-28T23:34:00.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.165+0000 7f467ffff640 1 --2- 172.21.15.161:0/2304752352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4674004800 0x7f4674004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:00.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.166+0000 7f468659c640 1 --2- 172.21.15.161:0/2304752352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4674003a90 0x7f4674003ed0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f4678009ac0 tx=0x7f4674005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b94bede66ab28363 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:00.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.166+0000 7f4685d9b640 1 -- 172.21.15.161:0/2304752352 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4678039070 con 0x7f4674003a90 2024-03-28T23:34:00.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.166+0000 7f4685d9b640 1 -- 172.21.15.161:0/2304752352 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f467802c6d0 con 0x7f4674003a90 2024-03-28T23:34:00.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.166+0000 7f4685d9b640 1 -- 172.21.15.161:0/2304752352 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4678009690 con 0x7f4674003a90 2024-03-28T23:34:00.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.166+0000 7f468cd7a640 1 -- 172.21.15.161:0/2304752352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4674003a90 msgr2=0x7f4674003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:00.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.166+0000 7f468cd7a640 1 --2- 172.21.15.161:0/2304752352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4674003a90 0x7f4674003ed0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f4678009ac0 tx=0x7f4674005830 comp rx=0 tx=0).stop 2024-03-28T23:34:00.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.166+0000 7f4686d9d640 1 -- 172.21.15.161:0/2304752352 reap_dead start 2024-03-28T23:34:00.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.166+0000 7f468cd7a640 1 -- 172.21.15.161:0/2304752352 shutdown_connections 2024-03-28T23:34:00.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.166+0000 7f468cd7a640 1 -- 172.21.15.161:0/2304752352 >> 172.21.15.161:0/2304752352 conn(0x7f46880ac620 msgr2=0x7f46880aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:00.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.166+0000 7f468cd7a640 1 -- 172.21.15.161:0/2304752352 shutdown_connections 2024-03-28T23:34:00.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.166+0000 7f468cd7a640 1 -- 172.21.15.161:0/2304752352 wait complete. 2024-03-28T23:34:00.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.168+0000 7f468cd7a640 1 Processor -- start 2024-03-28T23:34:00.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.168+0000 7f468cd7a640 1 -- start start 2024-03-28T23:34:00.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.169+0000 7f468cd7a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4674003a90 0x7f4688157d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.169+0000 7f468cd7a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4688153c20 0x7f4688146f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.169+0000 7f468cd7a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f468814f6d0 0x7f4688147460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.169+0000 7f468cd7a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4688152c70 con 0x7f4688153c20 2024-03-28T23:34:00.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.169+0000 7f468cd7a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f46881592d0 con 0x7f4674003a90 2024-03-28T23:34:00.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.169+0000 7f468cd7a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4688152df0 con 0x7f468814f6d0 2024-03-28T23:34:00.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.169+0000 7f467ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4688153c20 0x7f4688146f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.169+0000 7f467ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4688153c20 0x7f4688146f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60218/0 (socket says 172.21.15.161:60218) 2024-03-28T23:34:00.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.169+0000 7f467ffff640 1 -- 172.21.15.161:0/3508558341 learned_addr learned my addr 172.21.15.161:0/3508558341 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:00.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.169+0000 7f468659c640 1 --2- 172.21.15.161:0/3508558341 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4674003a90 0x7f4688157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.169+0000 7f4686d9d640 1 --2- 172.21.15.161:0/3508558341 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f468814f6d0 0x7f4688147460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.169+0000 7f467ffff640 1 -- 172.21.15.161:0/3508558341 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f468814f6d0 msgr2=0x7f4688147460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:00.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.169+0000 7f467ffff640 1 --2- 172.21.15.161:0/3508558341 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f468814f6d0 0x7f4688147460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.169+0000 7f467ffff640 1 -- 172.21.15.161:0/3508558341 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4674003a90 msgr2=0x7f4688157d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:00.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.169+0000 7f467ffff640 1 --2- 172.21.15.161:0/3508558341 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4674003a90 0x7f4688157d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.169+0000 7f467ffff640 1 -- 172.21.15.161:0/3508558341 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46881ea8a0 con 0x7f4688153c20 2024-03-28T23:34:00.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.169+0000 7f467ffff640 1 --2- 172.21.15.161:0/3508558341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4688153c20 0x7f4688146f20 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f4670001d60 tx=0x7f467000e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:00.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.170+0000 7f467effd640 1 -- 172.21.15.161:0/3508558341 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46700072e0 con 0x7f4688153c20 2024-03-28T23:34:00.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.170+0000 7f467effd640 1 -- 172.21.15.161:0/3508558341 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4670018070 con 0x7f4688153c20 2024-03-28T23:34:00.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.170+0000 7f468cd7a640 1 -- 172.21.15.161:0/3508558341 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46881eab80 con 0x7f4688153c20 2024-03-28T23:34:00.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.170+0000 7f467effd640 1 -- 172.21.15.161:0/3508558341 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46700057a0 con 0x7f4688153c20 2024-03-28T23:34:00.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.170+0000 7f468cd7a640 1 -- 172.21.15.161:0/3508558341 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46881eb140 con 0x7f4688153c20 2024-03-28T23:34:00.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.171+0000 7f467effd640 1 -- 172.21.15.161:0/3508558341 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4670005940 con 0x7f4688153c20 2024-03-28T23:34:00.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.172+0000 7f467effd640 1 --2- 172.21.15.161:0/3508558341 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f466806b500 0x7f466806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.172+0000 7f467effd640 1 -- 172.21.15.161:0/3508558341 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6101+0+0 (secure 0 0 0) 0x7f4670089ff0 con 0x7f4688153c20 2024-03-28T23:34:00.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.172+0000 7f468659c640 1 --2- 172.21.15.161:0/3508558341 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f466806b500 0x7f466806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.172+0000 7f468659c640 1 --2- 172.21.15.161:0/3508558341 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f466806b500 0x7f466806d9c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f4678036000 tx=0x7f4678030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:00.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.173+0000 7f468f5fd740 1 Processor -- start 2024-03-28T23:34:00.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.173+0000 7f468f5fd740 1 -- start start 2024-03-28T23:34:00.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.173+0000 7f468f5fd740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a366778db0 0x55a366776890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.173+0000 7f468f5fd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a366779190 0x55a366776dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.173+0000 7f468f5fd740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a366777310 0x55a3667734e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.173+0000 7f468f5fd740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55a3666dcaf0 con 0x55a366777310 2024-03-28T23:34:00.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.173+0000 7f468f5fd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55a3666859d0 con 0x55a366779190 2024-03-28T23:34:00.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.173+0000 7f468f5fd740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55a3666d8b90 con 0x55a366778db0 2024-03-28T23:34:00.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.173+0000 7f4686d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a366777310 0x55a3667734e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.174+0000 7f4686d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a366777310 0x55a3667734e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60234/0 (socket says 172.21.15.161:60234) 2024-03-28T23:34:00.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.174+0000 7f4686d9d640 1 -- 172.21.15.161:0/4279845738 learned_addr learned my addr 172.21.15.161:0/4279845738 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:00.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.174+0000 7f467ffff640 1 --2- 172.21.15.161:0/4279845738 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a366779190 0x55a366776dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.174+0000 7f468659c640 1 --2- 172.21.15.161:0/4279845738 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a366778db0 0x55a366776890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.174+0000 7f4686d9d640 1 -- 172.21.15.161:0/4279845738 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a366778db0 msgr2=0x55a366776890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:00.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.174+0000 7f4686d9d640 1 --2- 172.21.15.161:0/4279845738 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a366778db0 0x55a366776890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.174+0000 7f4686d9d640 1 -- 172.21.15.161:0/4279845738 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a366779190 msgr2=0x55a366776dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:00.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.174+0000 7f4686d9d640 1 --2- 172.21.15.161:0/4279845738 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a366779190 0x55a366776dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.174+0000 7f4686d9d640 1 -- 172.21.15.161:0/4279845738 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3663f12d0 con 0x55a366777310 2024-03-28T23:34:00.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.174+0000 7f4686d9d640 1 --2- 172.21.15.161:0/4279845738 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a366777310 0x55a3667734e0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f46880c5f00 tx=0x7f4680009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8da6d545515e936f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:00.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.174+0000 7f4667fff640 1 -- 172.21.15.161:0/4279845738 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468001b070 con 0x55a366777310 2024-03-28T23:34:00.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.174+0000 7f4667fff640 1 -- 172.21.15.161:0/4279845738 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4680017430 con 0x55a366777310 2024-03-28T23:34:00.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.174+0000 7f4667fff640 1 -- 172.21.15.161:0/4279845738 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46800177e0 con 0x55a366777310 2024-03-28T23:34:00.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.175+0000 7f468f5fd740 1 -- 172.21.15.161:0/4279845738 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a366777310 msgr2=0x55a3667734e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:00.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.175+0000 7f468f5fd740 1 --2- 172.21.15.161:0/4279845738 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a366777310 0x55a3667734e0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f46880c5f00 tx=0x7f4680009750 comp rx=0 tx=0).stop 2024-03-28T23:34:00.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.175+0000 7f468f5fd740 1 -- 172.21.15.161:0/4279845738 shutdown_connections 2024-03-28T23:34:00.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.175+0000 7f468f5fd740 1 --2- 172.21.15.161:0/4279845738 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a366777310 0x55a3667734e0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.175+0000 7f468f5fd740 1 --2- 172.21.15.161:0/4279845738 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a366779190 0x55a366776dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.175+0000 7f468f5fd740 1 --2- 172.21.15.161:0/4279845738 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a366778db0 0x55a366776890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.175+0000 7f468f5fd740 1 -- 172.21.15.161:0/4279845738 >> 172.21.15.161:0/4279845738 conn(0x55a366775630 msgr2=0x55a36679a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:00.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.175+0000 7f468f5fd740 1 -- 172.21.15.161:0/4279845738 shutdown_connections 2024-03-28T23:34:00.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.175+0000 7f468f5fd740 1 -- 172.21.15.161:0/4279845738 wait complete. 2024-03-28T23:34:00.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.175+0000 7f468f5fd740 1 -- start start 2024-03-28T23:34:00.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.175+0000 7f468f5fd740 1 Processor -- start 2024-03-28T23:34:00.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.176+0000 7f468f5fd740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a366778db0 0x55a36679f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.176+0000 7f468f5fd740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a366779190 0x55a3667b5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.176+0000 7f468f5fd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a3667c5040 0x55a3667c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.176+0000 7f468f5fd740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55a36679b090 con 0x55a366778db0 2024-03-28T23:34:00.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.176+0000 7f468f5fd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55a366698f70 con 0x55a3667c5040 2024-03-28T23:34:00.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.176+0000 7f468f5fd740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55a36679aad0 con 0x55a366779190 2024-03-28T23:34:00.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.176+0000 7f468f5fd740 10 client.0 _ll_register_callbacks cb 0x55a3663f0ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:34:00.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.176+0000 7f468659c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a366778db0 0x55a36679f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.176+0000 7f468659c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a366778db0 0x55a36679f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60248/0 (socket says 172.21.15.161:60248) 2024-03-28T23:34:00.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.176+0000 7f468659c640 1 -- 172.21.15.161:0/3272908912 learned_addr learned my addr 172.21.15.161:0/3272908912 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:00.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.176+0000 7f4686d9d640 1 --2- 172.21.15.161:0/3272908912 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a3667c5040 0x55a3667c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.176+0000 7f467ffff640 1 --2- 172.21.15.161:0/3272908912 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a366779190 0x55a3667b5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.177+0000 7f468659c640 1 -- 172.21.15.161:0/3272908912 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a366779190 msgr2=0x55a3667b5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:00.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.177+0000 7f468659c640 1 --2- 172.21.15.161:0/3272908912 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a366779190 0x55a3667b5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.177+0000 7f468659c640 1 -- 172.21.15.161:0/3272908912 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a3667c5040 msgr2=0x55a3667c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:00.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.177+0000 7f468659c640 1 --2- 172.21.15.161:0/3272908912 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a3667c5040 0x55a3667c54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.177+0000 7f468659c640 1 -- 172.21.15.161:0/3272908912 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3664bb6b0 con 0x55a366778db0 2024-03-28T23:34:00.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.177+0000 7f467ffff640 1 --2- 172.21.15.161:0/3272908912 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a366779190 0x55a3667b5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:00.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.177+0000 7f468659c640 1 --2- 172.21.15.161:0/3272908912 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a366778db0 0x55a36679f280 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f4678009be0 tx=0x7f467802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dfd18830eb57576 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:00.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.177+0000 7f46657fa640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:34:00.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.177+0000 7f46657fa640 1 -- 172.21.15.161:0/3272908912 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4678053060 con 0x55a366778db0 2024-03-28T23:34:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.177+0000 7f46657fa640 1 -- 172.21.15.161:0/3272908912 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 6101+0+0 (secure 0 0 0) 0x7f4678005990 con 0x55a366778db0 2024-03-28T23:34:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.177+0000 7f468f5fd740 1 -- 172.21.15.161:0/3272908912 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4680017430 con 0x55a366778db0 2024-03-28T23:34:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.177+0000 7f46657fa640 1 -- 172.21.15.161:0/3272908912 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f4678059050 con 0x55a366778db0 2024-03-28T23:34:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.177+0000 7f468f5fd740 10 client.7626 fetch_fsmap learned FSMap version 90 2024-03-28T23:34:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.177+0000 7f468f5fd740 1 -- 172.21.15.161:0/3272908912 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=2+}) -- 0x55a3667c9200 con 0x55a366778db0 2024-03-28T23:34:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.177+0000 7f46657fa640 1 -- 172.21.15.161:0/3272908912 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4680017430 con 0x55a366778db0 2024-03-28T23:34:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.177+0000 7f46657fa640 1 -- 172.21.15.161:0/3272908912 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 90) ==== 3494+0+0 (secure 0 0 0) 0x7f467805c020 con 0x55a366778db0 2024-03-28T23:34:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.177+0000 7f468f5fd740 10 client.7626 fetch_fsmap finished waiting for FSMap version 90 2024-03-28T23:34:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.177+0000 7f468f5fd740 10 client.7626 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T23:34:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.177+0000 7f468f5fd740 20 client.7626 populate_metadata read hostname 'smithi161' 2024-03-28T23:34:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.178+0000 7f468f5fd740 1 --2- 172.21.15.161:0/3272908912 >> [v2:172.21.15.161:6834/2544838511,v1:172.21.15.161:6835/2544838511] conn(0x55a3667b8b80 0x55a3667b8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.178+0000 7f468f5fd740 4 client.7626 mds_command: new command op to 7452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:34:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.178+0000 7f468f5fd740 1 -- 172.21.15.161:0/3272908912 --> [v2:172.21.15.161:6834/2544838511,v1:172.21.15.161:6835/2544838511] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3666ea030 con 0x55a3667b8b80 2024-03-28T23:34:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.178+0000 7f467ffff640 1 --2- 172.21.15.161:0/3272908912 >> [v2:172.21.15.161:6834/2544838511,v1:172.21.15.161:6835/2544838511] conn(0x55a3667b8b80 0x55a3667b8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.178+0000 7f46657fa640 10 client.7626 ms_handle_connect on v2:172.21.15.161:6834/2544838511 2024-03-28T23:34:00.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.178+0000 7f467ffff640 1 --2- 172.21.15.161:0/3272908912 >> [v2:172.21.15.161:6834/2544838511,v1:172.21.15.161:6835/2544838511] conn(0x55a3667b8b80 0x55a3667b8f60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd19ef638f028022 server_cookie=ba2b9ce4a1e7555a in_seq=0 out_seq=0 2024-03-28T23:34:00.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.179+0000 7f46657fa640 1 -- 172.21.15.161:0/3272908912 <== mds.0 v2:172.21.15.161:6834/2544838511 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a3666ea030 con 0x55a3667b8b80 2024-03-28T23:34:00.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.179+0000 7f46657fa640 10 client.7626 handle_command_reply: tid=0 2024-03-28T23:34:00.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.179+0000 7f468f5fd740 1 client.7626 shutdown 2024-03-28T23:34:00.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.179+0000 7f468f5fd740 1 -- 172.21.15.161:0/3272908912 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a366778db0 msgr2=0x55a36679f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:00.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.179+0000 7f468f5fd740 1 --2- 172.21.15.161:0/3272908912 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a366778db0 0x55a36679f280 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f4678009be0 tx=0x7f467802e070 comp rx=0 tx=0).stop 2024-03-28T23:34:00.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.179+0000 7f468f5fd740 1 -- 172.21.15.161:0/3272908912 shutdown_connections 2024-03-28T23:34:00.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.179+0000 7f468f5fd740 1 --2- 172.21.15.161:0/3272908912 >> [v2:172.21.15.161:6834/2544838511,v1:172.21.15.161:6835/2544838511] conn(0x55a3667b8b80 0x55a3667b8f60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.179+0000 7f468f5fd740 1 --2- 172.21.15.161:0/3272908912 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a3667c5040 0x55a3667c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.179+0000 7f468f5fd740 1 --2- 172.21.15.161:0/3272908912 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a366779190 0x55a3667b5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.179+0000 7f46657fa640 0 client.7626 ms_handle_reset on v2:172.21.15.161:6834/2544838511 2024-03-28T23:34:00.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.179+0000 7f468f5fd740 1 --2- 172.21.15.161:0/3272908912 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a366778db0 0x55a36679f280 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.179+0000 7f468f5fd740 1 -- 172.21.15.161:0/3272908912 >> 172.21.15.161:0/3272908912 conn(0x55a366777310 msgr2=0x55a366777740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:00.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.179+0000 7f468f5fd740 1 -- 172.21.15.161:0/3272908912 shutdown_connections 2024-03-28T23:34:00.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.179+0000 7f468f5fd740 1 -- 172.21.15.161:0/3272908912 wait complete. 2024-03-28T23:34:00.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.179+0000 7f468f5fd740 20 client.7626 trim_cache size 0 max 16384 2024-03-28T23:34:00.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.197+0000 7f468f5fd740 1 Processor -- start 2024-03-28T23:34:00.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.197+0000 7f468f5fd740 1 -- start start 2024-03-28T23:34:00.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.198+0000 7f468f5fd740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a366778db0 0x55a366776b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.198+0000 7f468f5fd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a366779190 0x55a366779780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.198+0000 7f468f5fd740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a3667c5040 0x55a3667994c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.198+0000 7f468f5fd740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f46881f73f0 con 0x55a3667c5040 2024-03-28T23:34:00.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.198+0000 7f468f5fd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f46881f6990 con 0x55a366779190 2024-03-28T23:34:00.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.198+0000 7f468f5fd740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f46881f6e30 con 0x55a366778db0 2024-03-28T23:34:00.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.198+0000 7f4686d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a3667c5040 0x55a3667994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.198+0000 7f4686d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a3667c5040 0x55a3667994c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60260/0 (socket says 172.21.15.161:60260) 2024-03-28T23:34:00.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.198+0000 7f4686d9d640 1 -- 172.21.15.161:0/1503109033 learned_addr learned my addr 172.21.15.161:0/1503109033 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:00.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.198+0000 7f4686d9d640 1 -- 172.21.15.161:0/1503109033 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a366778db0 msgr2=0x55a366776b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:00.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.198+0000 7f4686d9d640 1 --2- 172.21.15.161:0/1503109033 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a366778db0 0x55a366776b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.198+0000 7f4686d9d640 1 -- 172.21.15.161:0/1503109033 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a366779190 msgr2=0x55a366779780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:00.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.198+0000 7f467ffff640 1 --2- 172.21.15.161:0/1503109033 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a366779190 0x55a366779780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.198+0000 7f4686d9d640 1 --2- 172.21.15.161:0/1503109033 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a366779190 0x55a366779780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.198+0000 7f4686d9d640 1 -- 172.21.15.161:0/1503109033 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3667b7940 con 0x55a3667c5040 2024-03-28T23:34:00.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.199+0000 7f4686d9d640 1 --2- 172.21.15.161:0/1503109033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a3667c5040 0x55a3667994c0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f468001a000 tx=0x7f4680011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d75ee47dd071f47 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:00.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.199+0000 7f46677fe640 1 -- 172.21.15.161:0/1503109033 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468001b040 con 0x55a3667c5040 2024-03-28T23:34:00.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.199+0000 7f46677fe640 1 -- 172.21.15.161:0/1503109033 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4680020450 con 0x55a3667c5040 2024-03-28T23:34:00.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.199+0000 7f46677fe640 1 -- 172.21.15.161:0/1503109033 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4680020840 con 0x55a3667c5040 2024-03-28T23:34:00.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.199+0000 7f468f5fd740 1 -- 172.21.15.161:0/1503109033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a3667c5040 msgr2=0x55a3667994c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:00.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.199+0000 7f468f5fd740 1 --2- 172.21.15.161:0/1503109033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a3667c5040 0x55a3667994c0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f468001a000 tx=0x7f4680011000 comp rx=0 tx=0).stop 2024-03-28T23:34:00.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.199+0000 7f468f5fd740 1 -- 172.21.15.161:0/1503109033 shutdown_connections 2024-03-28T23:34:00.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.199+0000 7f468f5fd740 1 --2- 172.21.15.161:0/1503109033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a3667c5040 0x55a3667994c0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.199+0000 7f468f5fd740 1 --2- 172.21.15.161:0/1503109033 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a366779190 0x55a366779780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.199+0000 7f468f5fd740 1 --2- 172.21.15.161:0/1503109033 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a366778db0 0x55a366776b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.199+0000 7f468f5fd740 1 -- 172.21.15.161:0/1503109033 >> 172.21.15.161:0/1503109033 conn(0x55a366777310 msgr2=0x55a36679b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:00.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.199+0000 7f468f5fd740 1 -- 172.21.15.161:0/1503109033 shutdown_connections 2024-03-28T23:34:00.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.199+0000 7f468f5fd740 1 -- 172.21.15.161:0/1503109033 wait complete. 2024-03-28T23:34:00.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.200+0000 7f468f5fd740 1 -- start start 2024-03-28T23:34:00.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.200+0000 7f468f5fd740 1 Processor -- start 2024-03-28T23:34:00.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.201+0000 7f468f5fd740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a366778db0 0x55a366774aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.201+0000 7f468f5fd740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a366779190 0x55a36683a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.201+0000 7f468f5fd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a3667c5040 0x55a3668342f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.201+0000 7f468f5fd740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55a3667c5a50 con 0x55a366778db0 2024-03-28T23:34:00.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.201+0000 7f468f5fd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55a36679e040 con 0x55a3667c5040 2024-03-28T23:34:00.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.201+0000 7f468f5fd740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55a366795f70 con 0x55a366779190 2024-03-28T23:34:00.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.201+0000 7f468f5fd740 10 client.0 _ll_register_callbacks cb 0x7f4678037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:34:00.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.201+0000 7f468659c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a366778db0 0x55a366774aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.201+0000 7f468659c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a366778db0 0x55a366774aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60264/0 (socket says 172.21.15.161:60264) 2024-03-28T23:34:00.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.201+0000 7f468659c640 1 -- 172.21.15.161:0/1375673514 learned_addr learned my addr 172.21.15.161:0/1375673514 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:00.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.201+0000 7f467ffff640 1 --2- 172.21.15.161:0/1375673514 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a366779190 0x55a36683a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.201+0000 7f4686d9d640 1 --2- 172.21.15.161:0/1375673514 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a3667c5040 0x55a3668342f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.201+0000 7f468659c640 1 -- 172.21.15.161:0/1375673514 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a366779190 msgr2=0x55a36683a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:00.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.201+0000 7f468659c640 1 --2- 172.21.15.161:0/1375673514 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a366779190 0x55a36683a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.201+0000 7f468659c640 1 -- 172.21.15.161:0/1375673514 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a3667c5040 msgr2=0x55a3668342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:00.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.201+0000 7f468659c640 1 --2- 172.21.15.161:0/1375673514 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a3667c5040 0x55a3668342f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.201+0000 7f468659c640 1 -- 172.21.15.161:0/1375673514 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a36679e350 con 0x55a366778db0 2024-03-28T23:34:00.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.201+0000 7f467ffff640 1 --2- 172.21.15.161:0/1375673514 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a366779190 0x55a36683a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:00.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.201+0000 7f468659c640 1 --2- 172.21.15.161:0/1375673514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a366778db0 0x55a366774aa0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f4678009830 tx=0x7f46780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58e59a513057b1d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:00.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.201+0000 7f46657fa640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:34:00.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.202+0000 7f46657fa640 1 -- 172.21.15.161:0/1375673514 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4678039050 con 0x55a366778db0 2024-03-28T23:34:00.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.202+0000 7f46657fa640 1 -- 172.21.15.161:0/1375673514 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 6101+0+0 (secure 0 0 0) 0x7f467805ec30 con 0x55a366778db0 2024-03-28T23:34:00.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.202+0000 7f468f5fd740 1 -- 172.21.15.161:0/1375673514 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4680020450 con 0x55a366778db0 2024-03-28T23:34:00.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.202+0000 7f46657fa640 1 -- 172.21.15.161:0/1375673514 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f4678053020 con 0x55a366778db0 2024-03-28T23:34:00.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.202+0000 7f468f5fd740 10 client.7632 fetch_fsmap learned FSMap version 90 2024-03-28T23:34:00.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.202+0000 7f468f5fd740 1 -- 172.21.15.161:0/1375673514 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=2+}) -- 0x55a366798640 con 0x55a366778db0 2024-03-28T23:34:00.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.202+0000 7f46657fa640 1 -- 172.21.15.161:0/1375673514 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4680020450 con 0x55a366778db0 2024-03-28T23:34:00.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.202+0000 7f46657fa640 1 -- 172.21.15.161:0/1375673514 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 90) ==== 3494+0+0 (secure 0 0 0) 0x7f4678056020 con 0x55a366778db0 2024-03-28T23:34:00.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.202+0000 7f468f5fd740 10 client.7632 fetch_fsmap finished waiting for FSMap version 90 2024-03-28T23:34:00.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.202+0000 7f468f5fd740 10 client.7632 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T23:34:00.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.202+0000 7f468f5fd740 20 client.7632 populate_metadata read hostname 'smithi161' 2024-03-28T23:34:00.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.202+0000 7f468f5fd740 1 --2- 172.21.15.161:0/1375673514 >> [v2:172.21.15.161:6834/2544838511,v1:172.21.15.161:6835/2544838511] conn(0x55a366792070 0x55a366792450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.202+0000 7f468f5fd740 4 client.7632 mds_command: new command op to 7452 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:34:00.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.202+0000 7f468f5fd740 1 -- 172.21.15.161:0/1375673514 --> [v2:172.21.15.161:6834/2544838511,v1:172.21.15.161:6835/2544838511] -- command(tid 0: {"prefix": "status"}) -- 0x55a3666e4380 con 0x55a366792070 2024-03-28T23:34:00.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.202+0000 7f467ffff640 1 --2- 172.21.15.161:0/1375673514 >> [v2:172.21.15.161:6834/2544838511,v1:172.21.15.161:6835/2544838511] conn(0x55a366792070 0x55a366792450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.202+0000 7f46657fa640 10 client.7632 ms_handle_connect on v2:172.21.15.161:6834/2544838511 2024-03-28T23:34:00.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.203+0000 7f467ffff640 1 --2- 172.21.15.161:0/1375673514 >> [v2:172.21.15.161:6834/2544838511,v1:172.21.15.161:6835/2544838511] conn(0x55a366792070 0x55a366792450 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f39537b2c4a67ebc server_cookie=c718b841b3722425 in_seq=0 out_seq=0 2024-03-28T23:34:00.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.203+0000 7f46657fa640 1 -- 172.21.15.161:0/1375673514 <== mds.0 v2:172.21.15.161:6834/2544838511 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55a3666e4380 con 0x55a366792070 2024-03-28T23:34:00.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.203+0000 7f46657fa640 10 client.7632 handle_command_reply: tid=0 2024-03-28T23:34:00.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.203+0000 7f468f5fd740 1 client.7632 shutdown 2024-03-28T23:34:00.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.203+0000 7f468f5fd740 1 -- 172.21.15.161:0/1375673514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a366778db0 msgr2=0x55a366774aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:00.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.203+0000 7f468f5fd740 1 --2- 172.21.15.161:0/1375673514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a366778db0 0x55a366774aa0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f4678009830 tx=0x7f46780046c0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.203+0000 7f468f5fd740 1 -- 172.21.15.161:0/1375673514 shutdown_connections 2024-03-28T23:34:00.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.203+0000 7f468f5fd740 1 --2- 172.21.15.161:0/1375673514 >> [v2:172.21.15.161:6834/2544838511,v1:172.21.15.161:6835/2544838511] conn(0x55a366792070 0x55a366792450 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.203+0000 7f468f5fd740 1 --2- 172.21.15.161:0/1375673514 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a3667c5040 0x55a3668342f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.203+0000 7f468f5fd740 1 --2- 172.21.15.161:0/1375673514 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55a366779190 0x55a36683a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.203+0000 7f468f5fd740 1 --2- 172.21.15.161:0/1375673514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a366778db0 0x55a366774aa0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.203+0000 7f46657fa640 0 client.7632 ms_handle_reset on v2:172.21.15.161:6834/2544838511 2024-03-28T23:34:00.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.203+0000 7f468f5fd740 1 -- 172.21.15.161:0/1375673514 >> 172.21.15.161:0/1375673514 conn(0x55a366777310 msgr2=0x55a3667c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:00.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.203+0000 7f468f5fd740 1 -- 172.21.15.161:0/1375673514 shutdown_connections 2024-03-28T23:34:00.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.203+0000 7f468f5fd740 1 -- 172.21.15.161:0/1375673514 wait complete. 2024-03-28T23:34:00.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.203+0000 7f468f5fd740 20 client.7632 trim_cache size 0 max 16384 2024-03-28T23:34:00.210 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:34:00.210 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:34:00.211 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:34:00.211 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 7452, 2024-03-28T23:34:00.211 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:34:00.211 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:34:00.211 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:34:00.211 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 9.1962591210000006, 2024-03-28T23:34:00.211 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 90, 2024-03-28T23:34:00.211 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 109, 2024-03-28T23:34:00.211 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 109, 2024-03-28T23:34:00.211 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 14.321908593 2024-03-28T23:34:00.211 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:34:00.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.204+0000 7f468cd7a640 1 -- 172.21.15.161:0/3508558341 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f466806b500 msgr2=0x7f466806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:00.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.204+0000 7f468cd7a640 1 --2- 172.21.15.161:0/3508558341 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f466806b500 0x7f466806d9c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f4678036000 tx=0x7f4678030000 comp rx=0 tx=0).stop 2024-03-28T23:34:00.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.204+0000 7f468cd7a640 1 -- 172.21.15.161:0/3508558341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4688153c20 msgr2=0x7f4688146f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:00.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.204+0000 7f468cd7a640 1 --2- 172.21.15.161:0/3508558341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4688153c20 0x7f4688146f20 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f4670001d60 tx=0x7f467000e9e0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.204+0000 7f468cd7a640 1 -- 172.21.15.161:0/3508558341 shutdown_connections 2024-03-28T23:34:00.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.204+0000 7f468cd7a640 1 --2- 172.21.15.161:0/3508558341 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f466806b500 0x7f466806d9c0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.204+0000 7f468cd7a640 1 --2- 172.21.15.161:0/3508558341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4688153c20 0x7f4688146f20 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.204+0000 7f468cd7a640 1 --2- 172.21.15.161:0/3508558341 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f468814f6d0 0x7f4688147460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.204+0000 7f468cd7a640 1 --2- 172.21.15.161:0/3508558341 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4674003a90 0x7f4688157d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.204+0000 7f468cd7a640 1 -- 172.21.15.161:0/3508558341 >> 172.21.15.161:0/3508558341 conn(0x7f46880ac620 msgr2=0x7f46880ca280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:00.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.204+0000 7f468cd7a640 1 -- 172.21.15.161:0/3508558341 shutdown_connections 2024-03-28T23:34:00.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.204+0000 7f468cd7a640 1 -- 172.21.15.161:0/3508558341 wait complete. 2024-03-28T23:34:00.215 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:34:00.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.343+0000 7fc46b716640 1 Processor -- start 2024-03-28T23:34:00.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.343+0000 7fc46b716640 1 -- start start 2024-03-28T23:34:00.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.343+0000 7fc46b716640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc46414be70 0x7fc4640ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc46b716640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4640aacd0 0x7fc46415c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc46b716640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc46414c250 con 0x7fc4640ab6e0 2024-03-28T23:34:00.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc46b716640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fc4641529c0 con 0x7fc46414be70 2024-03-28T23:34:00.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc46b716640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc4640a9dc0 con 0x7fc4640aacd0 2024-03-28T23:34:00.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc468cb1640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc4640ab6e0 0x7fc4640aa5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41534/0 (socket says 172.21.15.161:41534) 2024-03-28T23:34:00.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc468cb1640 1 -- 172.21.15.161:0/717028902 learned_addr learned my addr 172.21.15.161:0/717028902 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:00.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc45bfff640 1 -- 172.21.15.161:0/717028902 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2580517445 0 0) 0x7fc46414c250 con 0x7fc4640ab6e0 2024-03-28T23:34:00.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc45bfff640 1 -- 172.21.15.161:0/717028902 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc450005730 con 0x7fc4640ab6e0 2024-03-28T23:34:00.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc469cb3640 1 --2- 172.21.15.161:0/717028902 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4640aacd0 0x7fc46415c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc4694b2640 1 --2- 172.21.15.161:0/717028902 >> v2:172.21.15.170:3300/0 conn(0x7fc46414be70 0x7fc4640ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc45bfff640 1 -- 172.21.15.161:0/717028902 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4241884505 0 0) 0x7fc450005730 con 0x7fc4640ab6e0 2024-03-28T23:34:00.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc45bfff640 1 -- 172.21.15.161:0/717028902 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4640aacd0 msgr2=0x7fc46415c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:00.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc45bfff640 1 --2- 172.21.15.161:0/717028902 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4640aacd0 0x7fc46415c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc45bfff640 1 -- 172.21.15.161:0/717028902 >> v2:172.21.15.170:3300/0 conn(0x7fc46414be70 msgr2=0x7fc4640ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:00.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc45bfff640 1 --2- 172.21.15.161:0/717028902 >> v2:172.21.15.170:3300/0 conn(0x7fc46414be70 0x7fc4640ab1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc45bfff640 1 -- 172.21.15.161:0/717028902 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4640aa0a0 con 0x7fc4640ab6e0 2024-03-28T23:34:00.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc45bfff640 1 -- 172.21.15.161:0/717028902 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fc454001670 con 0x7fc4640ab6e0 2024-03-28T23:34:00.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc45bfff640 1 -- 172.21.15.161:0/717028902 >> v1:172.21.15.161:6789/0 conn(0x7fc4640ab6e0 legacy=0x7fc4640aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:00.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc45bfff640 1 --2- 172.21.15.161:0/717028902 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4640aacd0 0x7fc450005ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc45bfff640 1 --2- 172.21.15.161:0/717028902 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc450006220 0x7fc450006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc45bfff640 1 --2- 172.21.15.161:0/717028902 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc450006f50 0x7fc4500073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc45bfff640 1 -- 172.21.15.161:0/717028902 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc4641529c0 con 0x7fc450006f50 2024-03-28T23:34:00.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc45bfff640 1 -- 172.21.15.161:0/717028902 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc4640a9dc0 con 0x7fc450006220 2024-03-28T23:34:00.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc45bfff640 1 -- 172.21.15.161:0/717028902 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc45000dd00 con 0x7fc4640aacd0 2024-03-28T23:34:00.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.344+0000 7fc469cb3640 1 --2- 172.21.15.161:0/717028902 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc450006f50 0x7fc4500073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.345+0000 7fc4694b2640 1 --2- 172.21.15.161:0/717028902 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4640aacd0 0x7fc450005ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.345+0000 7fc468cb1640 1 --2- 172.21.15.161:0/717028902 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc450006220 0x7fc450006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.345+0000 7fc469cb3640 1 -- 172.21.15.161:0/717028902 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4640aacd0 msgr2=0x7fc450005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:00.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.345+0000 7fc469cb3640 1 --2- 172.21.15.161:0/717028902 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4640aacd0 0x7fc450005ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.345+0000 7fc469cb3640 1 -- 172.21.15.161:0/717028902 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc450006220 msgr2=0x7fc450006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:00.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.345+0000 7fc469cb3640 1 --2- 172.21.15.161:0/717028902 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc450006220 0x7fc450006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.345+0000 7fc469cb3640 1 -- 172.21.15.161:0/717028902 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc450005730 con 0x7fc450006f50 2024-03-28T23:34:00.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.345+0000 7fc468cb1640 1 --2- 172.21.15.161:0/717028902 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc450006220 0x7fc450006a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:00.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.345+0000 7fc469cb3640 1 --2- 172.21.15.161:0/717028902 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc450006f50 0x7fc4500073d0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fc45c009fc0 tx=0x7fc450007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1670bfd12ebfffbb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:00.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.346+0000 7fc45bfff640 1 -- 172.21.15.161:0/717028902 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc45c039070 con 0x7fc450006f50 2024-03-28T23:34:00.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.346+0000 7fc45bfff640 1 -- 172.21.15.161:0/717028902 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc45c02cc20 con 0x7fc450006f50 2024-03-28T23:34:00.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.346+0000 7fc45bfff640 1 -- 172.21.15.161:0/717028902 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc45c009c10 con 0x7fc450006f50 2024-03-28T23:34:00.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.346+0000 7fc46b716640 1 -- 172.21.15.161:0/717028902 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc450006f50 msgr2=0x7fc4500073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:00.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.346+0000 7fc46b716640 1 --2- 172.21.15.161:0/717028902 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc450006f50 0x7fc4500073d0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fc45c009fc0 tx=0x7fc450007a40 comp rx=0 tx=0).stop 2024-03-28T23:34:00.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.346+0000 7fc469cb3640 1 -- 172.21.15.161:0/717028902 reap_dead start 2024-03-28T23:34:00.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.346+0000 7fc46b716640 1 -- 172.21.15.161:0/717028902 shutdown_connections 2024-03-28T23:34:00.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.346+0000 7fc46b716640 1 -- 172.21.15.161:0/717028902 >> 172.21.15.161:0/717028902 conn(0x7fc4640c6670 msgr2=0x7fc4640c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:00.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.346+0000 7fc46b716640 1 -- 172.21.15.161:0/717028902 shutdown_connections 2024-03-28T23:34:00.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.346+0000 7fc46b716640 1 -- 172.21.15.161:0/717028902 wait complete. 2024-03-28T23:34:00.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.347+0000 7fc46b716640 1 Processor -- start 2024-03-28T23:34:00.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.347+0000 7fc46b716640 1 -- start start 2024-03-28T23:34:00.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.348+0000 7fc46b716640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc450006f50 0x7fc464150cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.348+0000 7fc46b716640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc46414f0c0 0x7fc464145090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.348+0000 7fc46b716640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4640c4220 0x7fc4641533c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.348+0000 7fc46b716640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc46406b2d0 con 0x7fc46414f0c0 2024-03-28T23:34:00.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.348+0000 7fc46b716640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc46406b150 con 0x7fc4640c4220 2024-03-28T23:34:00.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.348+0000 7fc46b716640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc4640c1520 con 0x7fc450006f50 2024-03-28T23:34:00.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.348+0000 7fc468cb1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc46414f0c0 0x7fc464145090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.348+0000 7fc468cb1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc46414f0c0 0x7fc464145090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60280/0 (socket says 172.21.15.161:60280) 2024-03-28T23:34:00.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.348+0000 7fc468cb1640 1 -- 172.21.15.161:0/4215097412 learned_addr learned my addr 172.21.15.161:0/4215097412 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:00.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.348+0000 7fc468cb1640 1 -- 172.21.15.161:0/4215097412 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc450006f50 msgr2=0x7fc464150cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:00.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.348+0000 7fc468cb1640 1 --2- 172.21.15.161:0/4215097412 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc450006f50 0x7fc464150cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.348+0000 7fc468cb1640 1 -- 172.21.15.161:0/4215097412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4640c4220 msgr2=0x7fc4641533c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:00.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.348+0000 7fc468cb1640 1 --2- 172.21.15.161:0/4215097412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4640c4220 0x7fc4641533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.348+0000 7fc468cb1640 1 -- 172.21.15.161:0/4215097412 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc464155830 con 0x7fc46414f0c0 2024-03-28T23:34:00.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.349+0000 7fc468cb1640 1 --2- 172.21.15.161:0/4215097412 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc46414f0c0 0x7fc464145090 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fc45400e830 tx=0x7fc45400ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:00.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.349+0000 7fc459ffb640 1 -- 172.21.15.161:0/4215097412 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc45400c7f0 con 0x7fc46414f0c0 2024-03-28T23:34:00.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.349+0000 7fc459ffb640 1 -- 172.21.15.161:0/4215097412 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc454018070 con 0x7fc46414f0c0 2024-03-28T23:34:00.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.349+0000 7fc46b716640 1 -- 172.21.15.161:0/4215097412 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4641540b0 con 0x7fc46414f0c0 2024-03-28T23:34:00.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.349+0000 7fc459ffb640 1 -- 172.21.15.161:0/4215097412 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc45400cb90 con 0x7fc46414f0c0 2024-03-28T23:34:00.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.349+0000 7fc46b716640 1 -- 172.21.15.161:0/4215097412 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc464155d80 con 0x7fc46414f0c0 2024-03-28T23:34:00.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.350+0000 7fc46b716640 1 -- 172.21.15.161:0/4215097412 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc46404f4b0 con 0x7fc46414f0c0 2024-03-28T23:34:00.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.350+0000 7fc459ffb640 1 -- 172.21.15.161:0/4215097412 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc45400cd30 con 0x7fc46414f0c0 2024-03-28T23:34:00.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.351+0000 7fc459ffb640 1 --2- 172.21.15.161:0/4215097412 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc44806b500 0x7fc44806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.351+0000 7fc459ffb640 1 -- 172.21.15.161:0/4215097412 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6101+0+0 (secure 0 0 0) 0x7fc454089de0 con 0x7fc46414f0c0 2024-03-28T23:34:00.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.351+0000 7fc4694b2640 1 --2- 172.21.15.161:0/4215097412 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc44806b500 0x7fc44806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.351+0000 7fc4694b2640 1 --2- 172.21.15.161:0/4215097412 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc44806b500 0x7fc44806d9c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc460006e80 tx=0x7fc460006df0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:00.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.359+0000 7fc459ffb640 1 -- 172.21.15.161:0/4215097412 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc454058b60 con 0x7fc46414f0c0 2024-03-28T23:34:00.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.567+0000 7fc46b716640 1 -- 172.21.15.161:0/4215097412 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4640c0ff0 con 0x7fc46414f0c0 2024-03-28T23:34:00.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.568+0000 7fc459ffb640 1 -- 172.21.15.161:0/4215097412 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) ==== 94+0+5195 (secure 0 0 0) 0x7fc454058460 con 0x7fc46414f0c0 2024-03-28T23:34:00.569 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:00.569 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":90,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6203,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/1096504848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1096504848},{"type":"v1","addr":"172.21.15.170:6834","nonce":1096504848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":6206,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3351418222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3351418222},{"type":"v1","addr":"172.21.15.170:6835","nonce":3351418222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":6892,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/2684760895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2684760895},{"type":"v1","addr":"172.21.15.161:6837","nonce":2684760895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:33:50.326492+0000","modified":"2024-03-28T23:33:52.354643+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7452},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7452":{"gid":7452,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6835/2544838511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":2544838511},{"type":"v1","addr":"172.21.15.161:6835","nonce":2544838511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7452,"qdb_cluster":[7452]},"id":8}]} 2024-03-28T23:34:00.569 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 90 2024-03-28T23:34:00.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.570+0000 7fc46b716640 1 -- 172.21.15.161:0/4215097412 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc44806b500 msgr2=0x7fc44806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:00.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.570+0000 7fc46b716640 1 --2- 172.21.15.161:0/4215097412 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc44806b500 0x7fc44806d9c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc460006e80 tx=0x7fc460006df0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.570+0000 7fc46b716640 1 -- 172.21.15.161:0/4215097412 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc46414f0c0 msgr2=0x7fc464145090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:00.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.570+0000 7fc46b716640 1 --2- 172.21.15.161:0/4215097412 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc46414f0c0 0x7fc464145090 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fc45400e830 tx=0x7fc45400ecf0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.570+0000 7fc46b716640 1 -- 172.21.15.161:0/4215097412 shutdown_connections 2024-03-28T23:34:00.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.570+0000 7fc46b716640 1 --2- 172.21.15.161:0/4215097412 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc44806b500 0x7fc44806d9c0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.570+0000 7fc46b716640 1 --2- 172.21.15.161:0/4215097412 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc46414f0c0 0x7fc464145090 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.570+0000 7fc46b716640 1 --2- 172.21.15.161:0/4215097412 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4640c4220 0x7fc4641533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.570+0000 7fc46b716640 1 --2- 172.21.15.161:0/4215097412 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc450006f50 0x7fc464150cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.570+0000 7fc46b716640 1 -- 172.21.15.161:0/4215097412 >> 172.21.15.161:0/4215097412 conn(0x7fc4640c6670 msgr2=0x7fc4640a9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:00.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.570+0000 7fc46b716640 1 -- 172.21.15.161:0/4215097412 shutdown_connections 2024-03-28T23:34:00.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.570+0000 7fc46b716640 1 -- 172.21.15.161:0/4215097412 wait complete. 2024-03-28T23:34:00.582 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 90, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:33:50.326492+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 90, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7452": { "addr": "172.21.15.161:6835/2544838511", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 2544838511, "type": "v2" }, { "addr": "172.21.15.161:6835", "nonce": 2544838511, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7452, "incarnation": 88, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-03-28T23:33:52.354643+0000", "qdb_cluster": [ 7452 ], "qdb_leader": 7452, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7452 } } } ], "standbys": [ { "addr": "172.21.15.170:6834/1096504848", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 1096504848, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 1096504848, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 87, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6203, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.170:6835/3351418222", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 3351418222, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 3351418222, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 87, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6206, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6837/2684760895", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6836", "nonce": 2684760895, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 2684760895, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 87, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6892, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:34:00.582 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-28T23:34:00.582 INFO:tasks.cephfs.test_failover:current = [{'gid': 7452, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.161:6835/2544838511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 2544838511}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 2544838511}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 6203, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6834/1096504848', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 1096504848}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 1096504848}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 87}, {'gid': 6892, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6837/2684760895', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 2684760895}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 2684760895}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 87}, {'gid': 6206, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/3351418222', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 3351418222}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 3351418222}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 87}] 2024-03-28T23:34:00.583 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.b mds_join_fs cephfs 2024-03-28T23:34:00.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.686+0000 7f96d9eed640 1 Processor -- start 2024-03-28T23:34:00.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.686+0000 7f96d9eed640 1 -- start start 2024-03-28T23:34:00.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.687+0000 7f96d9eed640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f96d4152bc0 0x7f96d40c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.687+0000 7f96d9eed640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f96d41589b0 0x7f96d4158e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.687+0000 7f96d9eed640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96d40c6f60 con 0x7f96d40c8a80 2024-03-28T23:34:00.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.687+0000 7f96d9eed640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f96d40c6120 con 0x7f96d4152bc0 2024-03-28T23:34:00.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.687+0000 7f96d9eed640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f96d40c6400 con 0x7f96d41589b0 2024-03-28T23:34:00.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.687+0000 7f96d37fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f96d4152bc0 0x7f96d40c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.687+0000 7f96d37fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f96d4152bc0 0x7f96d40c0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55318/0 (socket says 172.21.15.161:55318) 2024-03-28T23:34:00.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.687+0000 7f96d37fe640 1 -- 172.21.15.161:0/3640664876 learned_addr learned my addr 172.21.15.161:0/3640664876 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:00.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.687+0000 7f96d3fff640 1 --2- 172.21.15.161:0/3640664876 >> v2:172.21.15.170:3300/0 conn(0x7f96d41589b0 0x7f96d4158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.687+0000 7f96d37fe640 1 -- 172.21.15.161:0/3640664876 >> v2:172.21.15.170:3300/0 conn(0x7f96d41589b0 msgr2=0x7f96d4158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:00.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.687+0000 7f96d37fe640 1 --2- 172.21.15.161:0/3640664876 >> v2:172.21.15.170:3300/0 conn(0x7f96d41589b0 0x7f96d4158e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.687+0000 7f96d37fe640 1 -- 172.21.15.161:0/3640664876 >> v1:172.21.15.170:6789/0 conn(0x7f96d40c8a80 legacy=0x7f96d40c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:00.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.687+0000 7f96d37fe640 1 -- 172.21.15.161:0/3640664876 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96d40c7240 con 0x7f96d4152bc0 2024-03-28T23:34:00.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.688+0000 7f96d37fe640 1 --2- 172.21.15.161:0/3640664876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96d4152bc0 0x7f96d40c0330 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f96d40aa5d0 tx=0x7f96c802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6536d915d2f270c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:00.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.688+0000 7f96d27fc640 1 -- 172.21.15.161:0/3640664876 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c8034070 con 0x7f96d4152bc0 2024-03-28T23:34:00.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.688+0000 7f96d27fc640 1 -- 172.21.15.161:0/3640664876 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96c8030860 con 0x7f96d4152bc0 2024-03-28T23:34:00.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.688+0000 7f96d27fc640 1 -- 172.21.15.161:0/3640664876 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c8030c30 con 0x7f96d4152bc0 2024-03-28T23:34:00.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.689+0000 7f96d9eed640 1 -- 172.21.15.161:0/3640664876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96d4152bc0 msgr2=0x7f96d40c0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:00.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.689+0000 7f96d9eed640 1 --2- 172.21.15.161:0/3640664876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96d4152bc0 0x7f96d40c0330 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f96d40aa5d0 tx=0x7f96c802b9a0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.689+0000 7f96d9eed640 1 -- 172.21.15.161:0/3640664876 shutdown_connections 2024-03-28T23:34:00.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.689+0000 7f96d9eed640 1 --2- 172.21.15.161:0/3640664876 >> v2:172.21.15.170:3300/0 conn(0x7f96d41589b0 0x7f96d4158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.689+0000 7f96d9eed640 1 --2- 172.21.15.161:0/3640664876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96d4152bc0 0x7f96d40c0330 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.689+0000 7f96d9eed640 1 -- 172.21.15.161:0/3640664876 >> 172.21.15.161:0/3640664876 conn(0x7f96d40caa80 msgr2=0x7f96d40cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:00.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.689+0000 7f96d9eed640 1 -- 172.21.15.161:0/3640664876 shutdown_connections 2024-03-28T23:34:00.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.689+0000 7f96d9eed640 1 -- 172.21.15.161:0/3640664876 wait complete. 2024-03-28T23:34:00.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.690+0000 7f96d9eed640 1 Processor -- start 2024-03-28T23:34:00.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.690+0000 7f96d9eed640 1 -- start start 2024-03-28T23:34:00.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.691+0000 7f96d9eed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96d40c8a80 0x7f96d40ab800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.691+0000 7f96d9eed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96d4152bc0 0x7f96d4150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.691+0000 7f96d9eed640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96d41589b0 0x7f96d4151030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.691+0000 7f96d9eed640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f96d4152fa0 con 0x7f96d41589b0 2024-03-28T23:34:00.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.691+0000 7f96d9eed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f96d40c95f0 con 0x7f96d40c8a80 2024-03-28T23:34:00.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.691+0000 7f96d9eed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f96d41ede10 con 0x7f96d4152bc0 2024-03-28T23:34:00.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.691+0000 7f96d3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96d41589b0 0x7f96d4151030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.691+0000 7f96d3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96d41589b0 0x7f96d4151030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55334/0 (socket says 172.21.15.161:55334) 2024-03-28T23:34:00.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.691+0000 7f96d3fff640 1 -- 172.21.15.161:0/1265465120 learned_addr learned my addr 172.21.15.161:0/1265465120 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:00.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.692+0000 7f96d37fe640 1 --2- 172.21.15.161:0/1265465120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96d40c8a80 0x7f96d40ab800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.692+0000 7f96d2ffd640 1 --2- 172.21.15.161:0/1265465120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96d4152bc0 0x7f96d4150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.692+0000 7f96d3fff640 1 -- 172.21.15.161:0/1265465120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96d4152bc0 msgr2=0x7f96d4150af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:00.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.692+0000 7f96d3fff640 1 --2- 172.21.15.161:0/1265465120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96d4152bc0 0x7f96d4150af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.692+0000 7f96d3fff640 1 -- 172.21.15.161:0/1265465120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96d40c8a80 msgr2=0x7f96d40ab800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:00.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.692+0000 7f96d3fff640 1 --2- 172.21.15.161:0/1265465120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96d40c8a80 0x7f96d40ab800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.692+0000 7f96d3fff640 1 -- 172.21.15.161:0/1265465120 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96d4147460 con 0x7f96d41589b0 2024-03-28T23:34:00.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.692+0000 7f96d3fff640 1 --2- 172.21.15.161:0/1265465120 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96d41589b0 0x7f96d4151030 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f96cc007e60 tx=0x7f96cc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:00.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.692+0000 7f96bbfff640 1 -- 172.21.15.161:0/1265465120 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96cc014420 con 0x7f96d41589b0 2024-03-28T23:34:00.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.692+0000 7f96bbfff640 1 -- 172.21.15.161:0/1265465120 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96cc019070 con 0x7f96d41589b0 2024-03-28T23:34:00.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.692+0000 7f96bbfff640 1 -- 172.21.15.161:0/1265465120 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96cc014820 con 0x7f96d41589b0 2024-03-28T23:34:00.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.692+0000 7f96d9eed640 1 -- 172.21.15.161:0/1265465120 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96d4151b80 con 0x7f96d41589b0 2024-03-28T23:34:00.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.692+0000 7f96d9eed640 1 -- 172.21.15.161:0/1265465120 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96d4151e60 con 0x7f96d41589b0 2024-03-28T23:34:00.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.693+0000 7f96d9eed640 1 -- 172.21.15.161:0/1265465120 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96d40cb7b0 con 0x7f96d41589b0 2024-03-28T23:34:00.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.694+0000 7f96bbfff640 1 -- 172.21.15.161:0/1265465120 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f96cc0149c0 con 0x7f96d41589b0 2024-03-28T23:34:00.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.694+0000 7f96bbfff640 1 --2- 172.21.15.161:0/1265465120 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f96c006b500 0x7f96c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:00.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.694+0000 7f96bbfff640 1 -- 172.21.15.161:0/1265465120 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6101+0+0 (secure 0 0 0) 0x7f96cc08ab30 con 0x7f96d41589b0 2024-03-28T23:34:00.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.694+0000 7f96d37fe640 1 --2- 172.21.15.161:0/1265465120 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f96c006b500 0x7f96c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:00.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.695+0000 7f96d37fe640 1 --2- 172.21.15.161:0/1265465120 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f96c006b500 0x7f96c006d9c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f96c8036000 tx=0x7f96c803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:00.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.699+0000 7f96bbfff640 1 -- 172.21.15.161:0/1265465120 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f96d40cb7b0 con 0x7f96d41589b0 2024-03-28T23:34:00.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.872+0000 7f96d9eed640 1 -- 172.21.15.161:0/1265465120 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) -- 0x7f96d404f4b0 con 0x7f96d41589b0 2024-03-28T23:34:00.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.875+0000 7f96bbfff640 1 -- 172.21.15.161:0/1265465120 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v36)=0 v36) ==== 116+0+0 (secure 0 0 0) 0x7f96cc0598b0 con 0x7f96d41589b0 2024-03-28T23:34:00.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.877+0000 7f96d9eed640 1 -- 172.21.15.161:0/1265465120 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f96c006b500 msgr2=0x7f96c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:00.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.877+0000 7f96d9eed640 1 --2- 172.21.15.161:0/1265465120 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f96c006b500 0x7f96c006d9c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f96c8036000 tx=0x7f96c803b040 comp rx=0 tx=0).stop 2024-03-28T23:34:00.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.877+0000 7f96d9eed640 1 -- 172.21.15.161:0/1265465120 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96d41589b0 msgr2=0x7f96d4151030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:00.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.877+0000 7f96d9eed640 1 --2- 172.21.15.161:0/1265465120 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96d41589b0 0x7f96d4151030 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f96cc007e60 tx=0x7f96cc00e4b0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.877+0000 7f96d9eed640 1 -- 172.21.15.161:0/1265465120 shutdown_connections 2024-03-28T23:34:00.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.877+0000 7f96d9eed640 1 --2- 172.21.15.161:0/1265465120 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f96c006b500 0x7f96c006d9c0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.877+0000 7f96d9eed640 1 --2- 172.21.15.161:0/1265465120 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96d41589b0 0x7f96d4151030 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.877+0000 7f96d9eed640 1 --2- 172.21.15.161:0/1265465120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96d4152bc0 0x7f96d4150af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.877+0000 7f96d9eed640 1 --2- 172.21.15.161:0/1265465120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96d40c8a80 0x7f96d40ab800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:00.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.877+0000 7f96d9eed640 1 -- 172.21.15.161:0/1265465120 >> 172.21.15.161:0/1265465120 conn(0x7f96d40caa80 msgr2=0x7f96d414f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:00.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.877+0000 7f96d9eed640 1 -- 172.21.15.161:0/1265465120 shutdown_connections 2024-03-28T23:34:00.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:00.877+0000 7f96d9eed640 1 -- 172.21.15.161:0/1265465120 wait complete. 2024-03-28T23:34:00.888 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:34:01.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.000+0000 7f87d50a4640 1 Processor -- start 2024-03-28T23:34:01.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.000+0000 7f87d50a4640 1 -- start start 2024-03-28T23:34:01.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.000+0000 7f87d50a4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f87d0152890 0x7f87d00c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.000+0000 7f87d50a4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f87d00c8ed0 0x7f87d00c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.000+0000 7f87d50a4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87d0158a30 con 0x7f87d00c5950 2024-03-28T23:34:01.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.000+0000 7f87d50a4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f87d00c8480 con 0x7f87d00c8ed0 2024-03-28T23:34:01.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.000+0000 7f87d50a4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f87d0159640 con 0x7f87d0152890 2024-03-28T23:34:01.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.000+0000 7f87ce59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f87d00c8ed0 0x7f87d00c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.000+0000 7f87cf59e640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f87d00c5950 0x7f87d01553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59792/0 (socket says 172.21.15.161:59792) 2024-03-28T23:34:01.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.000+0000 7f87cf59e640 1 -- 172.21.15.161:0/1427013601 learned_addr learned my addr 172.21.15.161:0/1427013601 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:01.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.001+0000 7f87ced9d640 1 --2- 172.21.15.161:0/1427013601 >> v2:172.21.15.170:3300/0 conn(0x7f87d0152890 0x7f87d00c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.001+0000 7f87ce59c640 1 -- 172.21.15.161:0/1427013601 >> v2:172.21.15.170:3300/0 conn(0x7f87d0152890 msgr2=0x7f87d00c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:01.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.001+0000 7f87ce59c640 1 --2- 172.21.15.161:0/1427013601 >> v2:172.21.15.170:3300/0 conn(0x7f87d0152890 0x7f87d00c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.001+0000 7f87ce59c640 1 -- 172.21.15.161:0/1427013601 >> v1:172.21.15.161:6789/0 conn(0x7f87d00c5950 legacy=0x7f87d01553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:01.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.001+0000 7f87ce59c640 1 -- 172.21.15.161:0/1427013601 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87d0158d10 con 0x7f87d00c8ed0 2024-03-28T23:34:01.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.001+0000 7f87cdd9b640 1 -- 172.21.15.161:0/1427013601 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 446032961 0 0) 0x7f87d0158a30 con 0x7f87d00c5950 2024-03-28T23:34:01.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.001+0000 7f87ce59c640 1 --2- 172.21.15.161:0/1427013601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87d00c8ed0 0x7f87d00c5410 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f87d008cc80 tx=0x7f87c402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=120e08eae69f291a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:01.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.002+0000 7f87cdd9b640 1 -- 172.21.15.161:0/1427013601 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c4038070 con 0x7f87d00c8ed0 2024-03-28T23:34:01.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.002+0000 7f87cdd9b640 1 -- 172.21.15.161:0/1427013601 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87c4031700 con 0x7f87d00c8ed0 2024-03-28T23:34:01.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.002+0000 7f87cdd9b640 1 -- 172.21.15.161:0/1427013601 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c4031ad0 con 0x7f87d00c8ed0 2024-03-28T23:34:01.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.002+0000 7f87d50a4640 1 -- 172.21.15.161:0/1427013601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87d00c8ed0 msgr2=0x7f87d00c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:01.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.002+0000 7f87d50a4640 1 --2- 172.21.15.161:0/1427013601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87d00c8ed0 0x7f87d00c5410 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f87d008cc80 tx=0x7f87c402b7a0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.002+0000 7f87d50a4640 1 -- 172.21.15.161:0/1427013601 shutdown_connections 2024-03-28T23:34:01.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.002+0000 7f87d50a4640 1 --2- 172.21.15.161:0/1427013601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87d00c8ed0 0x7f87d00c5410 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.002+0000 7f87d50a4640 1 --2- 172.21.15.161:0/1427013601 >> v2:172.21.15.170:3300/0 conn(0x7f87d0152890 0x7f87d00c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.002+0000 7f87d50a4640 1 -- 172.21.15.161:0/1427013601 >> 172.21.15.161:0/1427013601 conn(0x7f87d00ac620 msgr2=0x7f87d00aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:01.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.002+0000 7f87d50a4640 1 -- 172.21.15.161:0/1427013601 shutdown_connections 2024-03-28T23:34:01.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.002+0000 7f87d50a4640 1 -- 172.21.15.161:0/1427013601 wait complete. 2024-03-28T23:34:01.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.004+0000 7f87d50a4640 1 Processor -- start 2024-03-28T23:34:01.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.004+0000 7f87d50a4640 1 -- start start 2024-03-28T23:34:01.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.004+0000 7f87d50a4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87d00c5950 0x7f87d00c1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.004+0000 7f87d50a4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87d00c8ed0 0x7f87d00bf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.004+0000 7f87d50a4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87d0152890 0x7f87d00bf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.004+0000 7f87d50a4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f87d0152c70 con 0x7f87d00c5950 2024-03-28T23:34:01.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.004+0000 7f87d50a4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f87d01592d0 con 0x7f87d00c8ed0 2024-03-28T23:34:01.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.004+0000 7f87d50a4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f87d0152df0 con 0x7f87d0152890 2024-03-28T23:34:01.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.004+0000 7f87ced9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87d00c5950 0x7f87d00c1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.004+0000 7f87ced9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87d00c5950 0x7f87d00c1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55350/0 (socket says 172.21.15.161:55350) 2024-03-28T23:34:01.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.004+0000 7f87ced9d640 1 -- 172.21.15.161:0/2349179421 learned_addr learned my addr 172.21.15.161:0/2349179421 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:01.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.005+0000 7f87cf59e640 1 --2- 172.21.15.161:0/2349179421 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87d0152890 0x7f87d00bf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.005+0000 7f87ce59c640 1 --2- 172.21.15.161:0/2349179421 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87d00c8ed0 0x7f87d00bf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.005+0000 7f87ced9d640 1 -- 172.21.15.161:0/2349179421 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87d0152890 msgr2=0x7f87d00bf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:01.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.005+0000 7f87ced9d640 1 --2- 172.21.15.161:0/2349179421 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87d0152890 0x7f87d00bf9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.005+0000 7f87ced9d640 1 -- 172.21.15.161:0/2349179421 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87d00c8ed0 msgr2=0x7f87d00bf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:01.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.005+0000 7f87ced9d640 1 --2- 172.21.15.161:0/2349179421 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87d00c8ed0 0x7f87d00bf4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.005+0000 7f87ced9d640 1 -- 172.21.15.161:0/2349179421 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87d01e86c0 con 0x7f87d00c5950 2024-03-28T23:34:01.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.005+0000 7f87cf59e640 1 --2- 172.21.15.161:0/2349179421 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87d0152890 0x7f87d00bf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:01.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.005+0000 7f87ced9d640 1 --2- 172.21.15.161:0/2349179421 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87d00c5950 0x7f87d00c1080 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f87c800d800 tx=0x7f87c800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:01.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.005+0000 7f87b77fe640 1 -- 172.21.15.161:0/2349179421 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c800b6c0 con 0x7f87d00c5950 2024-03-28T23:34:01.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.005+0000 7f87b77fe640 1 -- 172.21.15.161:0/2349179421 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87c8019070 con 0x7f87d00c5950 2024-03-28T23:34:01.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.005+0000 7f87b77fe640 1 -- 172.21.15.161:0/2349179421 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c800ba80 con 0x7f87d00c5950 2024-03-28T23:34:01.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.005+0000 7f87d50a4640 1 -- 172.21.15.161:0/2349179421 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87d00be6f0 con 0x7f87d00c5950 2024-03-28T23:34:01.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.005+0000 7f87d50a4640 1 -- 172.21.15.161:0/2349179421 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87d00becb0 con 0x7f87d00c5950 2024-03-28T23:34:01.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.006+0000 7f87d50a4640 1 -- 172.21.15.161:0/2349179421 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87d00ac440 con 0x7f87d00c5950 2024-03-28T23:34:01.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.006+0000 7f87b77fe640 1 -- 172.21.15.161:0/2349179421 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f87c800bc20 con 0x7f87d00c5950 2024-03-28T23:34:01.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.007+0000 7f87b77fe640 1 --2- 172.21.15.161:0/2349179421 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f87bc06b4d0 0x7f87bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.007+0000 7f87b77fe640 1 -- 172.21.15.161:0/2349179421 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6101+0+0 (secure 0 0 0) 0x7f87c808a7e0 con 0x7f87d00c5950 2024-03-28T23:34:01.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.007+0000 7f87ce59c640 1 --2- 172.21.15.161:0/2349179421 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f87bc06b4d0 0x7f87bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.007+0000 7f87ce59c640 1 --2- 172.21.15.161:0/2349179421 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f87bc06b4d0 0x7f87bc06d990 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f87c4035000 tx=0x7f87c4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:01.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.011+0000 7f87b77fe640 1 -- 172.21.15.161:0/2349179421 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f87d00ac440 con 0x7f87d00c5950 2024-03-28T23:34:01.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.221+0000 7f87d50a4640 1 -- 172.21.15.161:0/2349179421 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87d004f4b0 con 0x7f87d00c5950 2024-03-28T23:34:01.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.222+0000 7f87b77fe640 1 -- 172.21.15.161:0/2349179421 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) ==== 94+0+5195 (secure 0 0 0) 0x7f87c801c360 con 0x7f87d00c5950 2024-03-28T23:34:01.222 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:01.222 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":90,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6203,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/1096504848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1096504848},{"type":"v1","addr":"172.21.15.170:6834","nonce":1096504848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":6206,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3351418222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3351418222},{"type":"v1","addr":"172.21.15.170:6835","nonce":3351418222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":6892,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/2684760895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2684760895},{"type":"v1","addr":"172.21.15.161:6837","nonce":2684760895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:33:50.326492+0000","modified":"2024-03-28T23:33:52.354643+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7452},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7452":{"gid":7452,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6835/2544838511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":2544838511},{"type":"v1","addr":"172.21.15.161:6835","nonce":2544838511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7452,"qdb_cluster":[7452]},"id":8}]} 2024-03-28T23:34:01.223 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 90 2024-03-28T23:34:01.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.224+0000 7f87d50a4640 1 -- 172.21.15.161:0/2349179421 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f87bc06b4d0 msgr2=0x7f87bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:01.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.224+0000 7f87d50a4640 1 --2- 172.21.15.161:0/2349179421 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f87bc06b4d0 0x7f87bc06d990 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f87c4035000 tx=0x7f87c4008030 comp rx=0 tx=0).stop 2024-03-28T23:34:01.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.224+0000 7f87d50a4640 1 -- 172.21.15.161:0/2349179421 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87d00c5950 msgr2=0x7f87d00c1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:01.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.224+0000 7f87d50a4640 1 --2- 172.21.15.161:0/2349179421 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87d00c5950 0x7f87d00c1080 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f87c800d800 tx=0x7f87c800dcc0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.224+0000 7f87d50a4640 1 -- 172.21.15.161:0/2349179421 shutdown_connections 2024-03-28T23:34:01.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.224+0000 7f87d50a4640 1 --2- 172.21.15.161:0/2349179421 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f87bc06b4d0 0x7f87bc06d990 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.224+0000 7f87d50a4640 1 --2- 172.21.15.161:0/2349179421 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87d0152890 0x7f87d00bf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.224+0000 7f87d50a4640 1 --2- 172.21.15.161:0/2349179421 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87d00c8ed0 0x7f87d00bf4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.224+0000 7f87d50a4640 1 --2- 172.21.15.161:0/2349179421 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87d00c5950 0x7f87d00c1080 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.224+0000 7f87d50a4640 1 -- 172.21.15.161:0/2349179421 >> 172.21.15.161:0/2349179421 conn(0x7f87d00ac620 msgr2=0x7f87d00c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:01.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.224+0000 7f87d50a4640 1 -- 172.21.15.161:0/2349179421 shutdown_connections 2024-03-28T23:34:01.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.224+0000 7f87d50a4640 1 -- 172.21.15.161:0/2349179421 wait complete. 2024-03-28T23:34:01.641 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 90, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:33:50.326492+0000', 'modified': '2024-03-28T23:33:52.354643+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7452}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7452': {'gid': 7452, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.161:6835/2544838511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 2544838511}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 2544838511}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7452, 'qdb_cluster': [7452]} 2024-03-28T23:34:01.641 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:34:01.641 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-28T23:34:01.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.742+0000 7fbed631e640 1 Processor -- start 2024-03-28T23:34:01.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.743+0000 7fbed631e640 1 -- start start 2024-03-28T23:34:01.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.743+0000 7fbed631e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fbed00c5410 0x7fbed00c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.743+0000 7fbed631e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbed01553a0 0x7fbed0155850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.743+0000 7fbed631e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbed0158a30 con 0x7fbed0152890 2024-03-28T23:34:01.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.743+0000 7fbed631e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fbed00c8480 con 0x7fbed00c5410 2024-03-28T23:34:01.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.743+0000 7fbed631e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbed00c5d90 con 0x7fbed01553a0 2024-03-28T23:34:01.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.743+0000 7fbecffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fbed0152890 0x7fbed00c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59806/0 (socket says 172.21.15.161:59806) 2024-03-28T23:34:01.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.743+0000 7fbecffff640 1 -- 172.21.15.161:0/3886289987 learned_addr learned my addr 172.21.15.161:0/3886289987 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:01.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.744+0000 7fbecf7fe640 1 --2- 172.21.15.161:0/3886289987 >> v2:172.21.15.170:3300/0 conn(0x7fbed00c5410 0x7fbed00c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.744+0000 7fbed48bb640 1 --2- 172.21.15.161:0/3886289987 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbed01553a0 0x7fbed0155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.744+0000 7fbeceffd640 1 -- 172.21.15.161:0/3886289987 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 465129908 0 0) 0x7fbed0158a30 con 0x7fbed0152890 2024-03-28T23:34:01.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.744+0000 7fbeceffd640 1 -- 172.21.15.161:0/3886289987 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbeb8003520 con 0x7fbed0152890 2024-03-28T23:34:01.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.744+0000 7fbeceffd640 1 -- 172.21.15.161:0/3886289987 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3366542265 0 0) 0x7fbeb8003520 con 0x7fbed0152890 2024-03-28T23:34:01.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.744+0000 7fbecf7fe640 1 -- 172.21.15.161:0/3886289987 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbed01553a0 msgr2=0x7fbed0155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:01.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.744+0000 7fbecf7fe640 1 --2- 172.21.15.161:0/3886289987 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbed01553a0 0x7fbed0155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.744+0000 7fbecf7fe640 1 -- 172.21.15.161:0/3886289987 >> v1:172.21.15.161:6789/0 conn(0x7fbed0152890 legacy=0x7fbed00c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:01.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.744+0000 7fbecf7fe640 1 -- 172.21.15.161:0/3886289987 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed0158d10 con 0x7fbed00c5410 2024-03-28T23:34:01.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.744+0000 7fbed48bb640 1 --2- 172.21.15.161:0/3886289987 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbed01553a0 0x7fbed0155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:34:01.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.745+0000 7fbecf7fe640 1 --2- 172.21.15.161:0/3886289987 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbed00c5410 0x7fbed00c5850 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbec4009790 tx=0x7fbec402b6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d1fd6e2362b6b18 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:01.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.745+0000 7fbeceffd640 1 -- 172.21.15.161:0/3886289987 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec4037070 con 0x7fbed00c5410 2024-03-28T23:34:01.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.745+0000 7fbeceffd640 1 -- 172.21.15.161:0/3886289987 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbec4030510 con 0x7fbed00c5410 2024-03-28T23:34:01.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.745+0000 7fbeceffd640 1 -- 172.21.15.161:0/3886289987 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec0001670 con 0x7fbed00c5410 2024-03-28T23:34:01.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.746+0000 7fbed631e640 1 -- 172.21.15.161:0/3886289987 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbed00c5410 msgr2=0x7fbed00c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:01.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.746+0000 7fbed631e640 1 --2- 172.21.15.161:0/3886289987 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbed00c5410 0x7fbed00c5850 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbec4009790 tx=0x7fbec402b6e0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.746+0000 7fbed631e640 1 -- 172.21.15.161:0/3886289987 shutdown_connections 2024-03-28T23:34:01.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.746+0000 7fbed631e640 1 --2- 172.21.15.161:0/3886289987 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbed01553a0 0x7fbed0155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.746+0000 7fbed631e640 1 --2- 172.21.15.161:0/3886289987 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbed00c5410 0x7fbed00c5850 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.746+0000 7fbed631e640 1 -- 172.21.15.161:0/3886289987 >> 172.21.15.161:0/3886289987 conn(0x7fbed00ac620 msgr2=0x7fbed00aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:01.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.746+0000 7fbed631e640 1 -- 172.21.15.161:0/3886289987 shutdown_connections 2024-03-28T23:34:01.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.746+0000 7fbed631e640 1 -- 172.21.15.161:0/3886289987 wait complete. 2024-03-28T23:34:01.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.747+0000 7fbed631e640 1 Processor -- start 2024-03-28T23:34:01.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.748+0000 7fbed631e640 1 -- start start 2024-03-28T23:34:01.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.749+0000 7fbed631e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbed00c5410 0x7fbed01eebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.749+0000 7fbed631e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbed0152890 0x7fbed01eb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.749+0000 7fbed631e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbed01553a0 0x7fbed00abce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.749+0000 7fbed631e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbed01592d0 con 0x7fbed00c5410 2024-03-28T23:34:01.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.749+0000 7fbed631e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbed0159640 con 0x7fbed0152890 2024-03-28T23:34:01.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.749+0000 7fbed631e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbed00c90a0 con 0x7fbed01553a0 2024-03-28T23:34:01.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.749+0000 7fbecffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbed00c5410 0x7fbed01eebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.749+0000 7fbecffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbed00c5410 0x7fbed01eebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55364/0 (socket says 172.21.15.161:55364) 2024-03-28T23:34:01.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.749+0000 7fbecffff640 1 -- 172.21.15.161:0/460448996 learned_addr learned my addr 172.21.15.161:0/460448996 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:01.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.749+0000 7fbed48bb640 1 --2- 172.21.15.161:0/460448996 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbed01553a0 0x7fbed00abce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.749+0000 7fbecf7fe640 1 --2- 172.21.15.161:0/460448996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbed0152890 0x7fbed01eb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.749+0000 7fbecffff640 1 -- 172.21.15.161:0/460448996 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbed01553a0 msgr2=0x7fbed00abce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:01.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.749+0000 7fbecffff640 1 --2- 172.21.15.161:0/460448996 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbed01553a0 0x7fbed00abce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.749+0000 7fbecffff640 1 -- 172.21.15.161:0/460448996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbed0152890 msgr2=0x7fbed01eb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:01.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.749+0000 7fbecffff640 1 --2- 172.21.15.161:0/460448996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbed0152890 0x7fbed01eb510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.749+0000 7fbecffff640 1 -- 172.21.15.161:0/460448996 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed0157800 con 0x7fbed00c5410 2024-03-28T23:34:01.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.749+0000 7fbed48bb640 1 --2- 172.21.15.161:0/460448996 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbed01553a0 0x7fbed00abce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:01.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.749+0000 7fbecf7fe640 1 --2- 172.21.15.161:0/460448996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbed0152890 0x7fbed01eb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:01.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.749+0000 7fbecffff640 1 --2- 172.21.15.161:0/460448996 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbed00c5410 0x7fbed01eebc0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fbec00017e0 tx=0x7fbec000cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:01.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.749+0000 7fbeccff9640 1 -- 172.21.15.161:0/460448996 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec0005ad0 con 0x7fbed00c5410 2024-03-28T23:34:01.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.750+0000 7fbeccff9640 1 -- 172.21.15.161:0/460448996 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbec0005c90 con 0x7fbed00c5410 2024-03-28T23:34:01.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.750+0000 7fbeccff9640 1 -- 172.21.15.161:0/460448996 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec00072b0 con 0x7fbed00c5410 2024-03-28T23:34:01.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.750+0000 7fbed631e640 1 -- 172.21.15.161:0/460448996 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbed01ee0e0 con 0x7fbed00c5410 2024-03-28T23:34:01.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.750+0000 7fbed631e640 1 -- 172.21.15.161:0/460448996 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbed01ee6a0 con 0x7fbed00c5410 2024-03-28T23:34:01.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.751+0000 7fbeccff9640 1 -- 172.21.15.161:0/460448996 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbec0007450 con 0x7fbed00c5410 2024-03-28T23:34:01.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.752+0000 7fbeccff9640 1 --2- 172.21.15.161:0/460448996 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbebc06b4d0 0x7fbebc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.752+0000 7fbeccff9640 1 -- 172.21.15.161:0/460448996 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6101+0+0 (secure 0 0 0) 0x7fbec008a130 con 0x7fbed00c5410 2024-03-28T23:34:01.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.752+0000 7fbecf7fe640 1 --2- 172.21.15.161:0/460448996 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbebc06b4d0 0x7fbebc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.753+0000 7fbecf7fe640 1 --2- 172.21.15.161:0/460448996 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbebc06b4d0 0x7fbebc06d990 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fbec4034000 tx=0x7fbec4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:01.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.753+0000 7fbed8fdb740 1 Processor -- start 2024-03-28T23:34:01.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.753+0000 7fbed8fdb740 1 -- start start 2024-03-28T23:34:01.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.753+0000 7fbed8fdb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c82c01db0 0x559c82bff890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.753+0000 7fbed8fdb740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559c82c02190 0x559c82bffdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.753+0000 7fbed8fdb740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c00310 0x559c82bfc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.753+0000 7fbed8fdb740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x559c82b65af0 con 0x559c82c00310 2024-03-28T23:34:01.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.753+0000 7fbed8fdb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x559c82b0e9d0 con 0x559c82c01db0 2024-03-28T23:34:01.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.753+0000 7fbed8fdb740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x559c82b61b90 con 0x559c82c02190 2024-03-28T23:34:01.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.753+0000 7fbed48bb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c00310 0x559c82bfc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.754+0000 7fbed48bb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c00310 0x559c82bfc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55374/0 (socket says 172.21.15.161:55374) 2024-03-28T23:34:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.754+0000 7fbed48bb640 1 -- 172.21.15.161:0/3849296012 learned_addr learned my addr 172.21.15.161:0/3849296012 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.754+0000 7fbecf7fe640 1 --2- 172.21.15.161:0/3849296012 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559c82c02190 0x559c82bffdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.754+0000 7fbecffff640 1 --2- 172.21.15.161:0/3849296012 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c82c01db0 0x559c82bff890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.754+0000 7fbed48bb640 1 -- 172.21.15.161:0/3849296012 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559c82c02190 msgr2=0x559c82bffdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.754+0000 7fbed48bb640 1 --2- 172.21.15.161:0/3849296012 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559c82c02190 0x559c82bffdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.754+0000 7fbed48bb640 1 -- 172.21.15.161:0/3849296012 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c82c01db0 msgr2=0x559c82bff890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.754+0000 7fbed48bb640 1 --2- 172.21.15.161:0/3849296012 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c82c01db0 0x559c82bff890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.754+0000 7fbed48bb640 1 -- 172.21.15.161:0/3849296012 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c8287a2d0 con 0x559c82c00310 2024-03-28T23:34:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.754+0000 7fbecf7fe640 1 --2- 172.21.15.161:0/3849296012 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559c82c02190 0x559c82bffdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.754+0000 7fbecffff640 1 --2- 172.21.15.161:0/3849296012 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c82c01db0 0x559c82bff890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.754+0000 7fbed48bb640 1 --2- 172.21.15.161:0/3849296012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c00310 0x559c82bfc4e0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fbec8007680 tx=0x7fbec8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cb23fc8022c3416 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.754+0000 7fbeb5ffb640 1 -- 172.21.15.161:0/3849296012 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec801a020 con 0x559c82c00310 2024-03-28T23:34:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.754+0000 7fbeb5ffb640 1 -- 172.21.15.161:0/3849296012 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbec8015850 con 0x559c82c00310 2024-03-28T23:34:01.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.754+0000 7fbeb5ffb640 1 -- 172.21.15.161:0/3849296012 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec8015c20 con 0x559c82c00310 2024-03-28T23:34:01.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.755+0000 7fbed8fdb740 1 -- 172.21.15.161:0/3849296012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c00310 msgr2=0x559c82bfc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:01.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.755+0000 7fbed8fdb740 1 --2- 172.21.15.161:0/3849296012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c00310 0x559c82bfc4e0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fbec8007680 tx=0x7fbec8009750 comp rx=0 tx=0).stop 2024-03-28T23:34:01.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.755+0000 7fbed8fdb740 1 -- 172.21.15.161:0/3849296012 shutdown_connections 2024-03-28T23:34:01.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.755+0000 7fbed8fdb740 1 --2- 172.21.15.161:0/3849296012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c00310 0x559c82bfc4e0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.755+0000 7fbed8fdb740 1 --2- 172.21.15.161:0/3849296012 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559c82c02190 0x559c82bffdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.755+0000 7fbed8fdb740 1 --2- 172.21.15.161:0/3849296012 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c82c01db0 0x559c82bff890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.755+0000 7fbed8fdb740 1 -- 172.21.15.161:0/3849296012 >> 172.21.15.161:0/3849296012 conn(0x559c82bfe630 msgr2=0x559c82c23220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:01.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.755+0000 7fbed8fdb740 1 -- 172.21.15.161:0/3849296012 shutdown_connections 2024-03-28T23:34:01.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.755+0000 7fbed8fdb740 1 -- 172.21.15.161:0/3849296012 wait complete. 2024-03-28T23:34:01.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.755+0000 7fbed8fdb740 1 -- start start 2024-03-28T23:34:01.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.756+0000 7fbed8fdb740 1 Processor -- start 2024-03-28T23:34:01.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.756+0000 7fbed8fdb740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c01db0 0x559c82c4ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.756+0000 7fbed8fdb740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559c82c02190 0x559c82c418d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.756+0000 7fbed8fdb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c82c4f9a0 0x559c82c4fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.756+0000 7fbed8fdb740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x559c82c24090 con 0x559c82c01db0 2024-03-28T23:34:01.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.756+0000 7fbed8fdb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x559c82b21f70 con 0x559c82c4f9a0 2024-03-28T23:34:01.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.756+0000 7fbed8fdb740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x559c82c23ad0 con 0x559c82c02190 2024-03-28T23:34:01.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.756+0000 7fbed8fdb740 10 client.0 _ll_register_callbacks cb 0x559c82879ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:34:01.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.756+0000 7fbecffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c01db0 0x559c82c4ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.756+0000 7fbecffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c01db0 0x559c82c4ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55376/0 (socket says 172.21.15.161:55376) 2024-03-28T23:34:01.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.756+0000 7fbecffff640 1 -- 172.21.15.161:0/3080523572 learned_addr learned my addr 172.21.15.161:0/3080523572 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:01.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.756+0000 7fbecf7fe640 1 --2- 172.21.15.161:0/3080523572 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559c82c02190 0x559c82c418d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.757+0000 7fbed48bb640 1 --2- 172.21.15.161:0/3080523572 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c82c4f9a0 0x559c82c4fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.757+0000 7fbecffff640 1 -- 172.21.15.161:0/3080523572 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559c82c02190 msgr2=0x559c82c418d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:01.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.757+0000 7fbecffff640 1 --2- 172.21.15.161:0/3080523572 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559c82c02190 0x559c82c418d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.757+0000 7fbecffff640 1 -- 172.21.15.161:0/3080523572 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c82c4f9a0 msgr2=0x559c82c4fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:01.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.757+0000 7fbecffff640 1 --2- 172.21.15.161:0/3080523572 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c82c4f9a0 0x559c82c4fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.757+0000 7fbecffff640 1 -- 172.21.15.161:0/3080523572 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c829446b0 con 0x559c82c01db0 2024-03-28T23:34:01.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.757+0000 7fbecffff640 1 --2- 172.21.15.161:0/3080523572 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c01db0 0x559c82c4ecd0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fbec000cfd0 tx=0x7fbec0005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4785bae11b4a7800 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:01.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.757+0000 7fbeaeffd640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:34:01.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.757+0000 7fbeaeffd640 1 -- 172.21.15.161:0/3080523572 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec0070020 con 0x559c82c01db0 2024-03-28T23:34:01.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.757+0000 7fbeaeffd640 1 -- 172.21.15.161:0/3080523572 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 6101+0+0 (secure 0 0 0) 0x7fbec0060a30 con 0x559c82c01db0 2024-03-28T23:34:01.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.757+0000 7fbed8fdb740 1 -- 172.21.15.161:0/3080523572 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbec8015850 con 0x559c82c01db0 2024-03-28T23:34:01.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.757+0000 7fbeaeffd640 1 -- 172.21.15.161:0/3080523572 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fbec0060ec0 con 0x559c82c01db0 2024-03-28T23:34:01.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.757+0000 7fbed8fdb740 10 client.7665 fetch_fsmap learned FSMap version 90 2024-03-28T23:34:01.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.757+0000 7fbed8fdb740 1 -- 172.21.15.161:0/3080523572 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=2+}) -- 0x559c82c50400 con 0x559c82c01db0 2024-03-28T23:34:01.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.757+0000 7fbeaeffd640 1 -- 172.21.15.161:0/3080523572 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbec8015850 con 0x559c82c01db0 2024-03-28T23:34:01.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.757+0000 7fbeaeffd640 1 -- 172.21.15.161:0/3080523572 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 90) ==== 3494+0+0 (secure 0 0 0) 0x7fbec00880d0 con 0x559c82c01db0 2024-03-28T23:34:01.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.757+0000 7fbed8fdb740 10 client.7665 fetch_fsmap finished waiting for FSMap version 90 2024-03-28T23:34:01.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.757+0000 7fbed8fdb740 10 client.7665 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T23:34:01.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.758+0000 7fbed8fdb740 20 client.7665 populate_metadata read hostname 'smithi161' 2024-03-28T23:34:01.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.758+0000 7fbed8fdb740 1 --2- 172.21.15.161:0/3080523572 >> [v2:172.21.15.161:6834/2544838511,v1:172.21.15.161:6835/2544838511] conn(0x559c82c51e20 0x559c82c52200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.758+0000 7fbed8fdb740 4 client.7665 mds_command: new command op to 7452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:34:01.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.758+0000 7fbed8fdb740 1 -- 172.21.15.161:0/3080523572 --> [v2:172.21.15.161:6834/2544838511,v1:172.21.15.161:6835/2544838511] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c82b73030 con 0x559c82c51e20 2024-03-28T23:34:01.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.758+0000 7fbecf7fe640 1 --2- 172.21.15.161:0/3080523572 >> [v2:172.21.15.161:6834/2544838511,v1:172.21.15.161:6835/2544838511] conn(0x559c82c51e20 0x559c82c52200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.758+0000 7fbeaeffd640 10 client.7665 ms_handle_connect on v2:172.21.15.161:6834/2544838511 2024-03-28T23:34:01.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.758+0000 7fbecf7fe640 1 --2- 172.21.15.161:0/3080523572 >> [v2:172.21.15.161:6834/2544838511,v1:172.21.15.161:6835/2544838511] conn(0x559c82c51e20 0x559c82c52200 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=799d6b2c355a855b server_cookie=60994e27df50793f in_seq=0 out_seq=0 2024-03-28T23:34:01.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.759+0000 7fbeaeffd640 1 -- 172.21.15.161:0/3080523572 <== mds.0 v2:172.21.15.161:6834/2544838511 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559c82b73030 con 0x559c82c51e20 2024-03-28T23:34:01.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.759+0000 7fbeaeffd640 10 client.7665 handle_command_reply: tid=0 2024-03-28T23:34:01.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.759+0000 7fbed8fdb740 1 client.7665 shutdown 2024-03-28T23:34:01.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.759+0000 7fbed8fdb740 1 -- 172.21.15.161:0/3080523572 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c01db0 msgr2=0x559c82c4ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:01.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.759+0000 7fbed8fdb740 1 --2- 172.21.15.161:0/3080523572 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c01db0 0x559c82c4ecd0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fbec000cfd0 tx=0x7fbec0005f80 comp rx=0 tx=0).stop 2024-03-28T23:34:01.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.759+0000 7fbed8fdb740 1 -- 172.21.15.161:0/3080523572 shutdown_connections 2024-03-28T23:34:01.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.759+0000 7fbed8fdb740 1 --2- 172.21.15.161:0/3080523572 >> [v2:172.21.15.161:6834/2544838511,v1:172.21.15.161:6835/2544838511] conn(0x559c82c51e20 0x559c82c52200 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.759+0000 7fbed8fdb740 1 --2- 172.21.15.161:0/3080523572 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c82c4f9a0 0x559c82c4fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.759+0000 7fbeaeffd640 0 client.7665 ms_handle_reset on v2:172.21.15.161:6834/2544838511 2024-03-28T23:34:01.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.759+0000 7fbed8fdb740 1 --2- 172.21.15.161:0/3080523572 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559c82c02190 0x559c82c418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.759+0000 7fbed8fdb740 1 --2- 172.21.15.161:0/3080523572 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c01db0 0x559c82c4ecd0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.759+0000 7fbed8fdb740 1 -- 172.21.15.161:0/3080523572 >> 172.21.15.161:0/3080523572 conn(0x559c82c00310 msgr2=0x559c82c006f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:01.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.759+0000 7fbed8fdb740 1 -- 172.21.15.161:0/3080523572 shutdown_connections 2024-03-28T23:34:01.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.759+0000 7fbed8fdb740 1 -- 172.21.15.161:0/3080523572 wait complete. 2024-03-28T23:34:01.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.759+0000 7fbed8fdb740 20 client.7665 trim_cache size 0 max 16384 2024-03-28T23:34:01.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.777+0000 7fbed8fdb740 1 Processor -- start 2024-03-28T23:34:01.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.777+0000 7fbed8fdb740 1 -- start start 2024-03-28T23:34:01.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.778+0000 7fbed8fdb740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c01db0 0x559c82c4eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.778+0000 7fbed8fdb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c82c02190 0x559c82bff5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.778+0000 7fbed8fdb740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559c82c4f9a0 0x559c82c21600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.778+0000 7fbed8fdb740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbed01f6ef0 con 0x559c82c01db0 2024-03-28T23:34:01.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.778+0000 7fbed8fdb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbed0151de0 con 0x559c82c02190 2024-03-28T23:34:01.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.778+0000 7fbed8fdb740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbed01f68a0 con 0x559c82c4f9a0 2024-03-28T23:34:01.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.778+0000 7fbecffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c01db0 0x559c82c4eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.778+0000 7fbecffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c01db0 0x559c82c4eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55386/0 (socket says 172.21.15.161:55386) 2024-03-28T23:34:01.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.778+0000 7fbecffff640 1 -- 172.21.15.161:0/1836325307 learned_addr learned my addr 172.21.15.161:0/1836325307 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:01.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.778+0000 7fbecf7fe640 1 --2- 172.21.15.161:0/1836325307 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c82c02190 0x559c82bff5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.778+0000 7fbed48bb640 1 --2- 172.21.15.161:0/1836325307 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559c82c4f9a0 0x559c82c21600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.778+0000 7fbecffff640 1 -- 172.21.15.161:0/1836325307 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559c82c4f9a0 msgr2=0x559c82c21600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:01.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.778+0000 7fbecffff640 1 --2- 172.21.15.161:0/1836325307 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559c82c4f9a0 0x559c82c21600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.778+0000 7fbecffff640 1 -- 172.21.15.161:0/1836325307 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c82c02190 msgr2=0x559c82bff5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:01.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.778+0000 7fbecffff640 1 --2- 172.21.15.161:0/1836325307 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c82c02190 0x559c82bff5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.778+0000 7fbecffff640 1 -- 172.21.15.161:0/1836325307 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c82bfb040 con 0x559c82c01db0 2024-03-28T23:34:01.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.778+0000 7fbecffff640 1 --2- 172.21.15.161:0/1836325307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c01db0 0x559c82c4eb80 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fbec0054c10 tx=0x7fbec001d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=869a64b96a2e5fbc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:01.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.778+0000 7fbeb57fa640 1 -- 172.21.15.161:0/1836325307 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec0070040 con 0x559c82c01db0 2024-03-28T23:34:01.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.779+0000 7fbeb57fa640 1 -- 172.21.15.161:0/1836325307 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbec00701e0 con 0x559c82c01db0 2024-03-28T23:34:01.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.779+0000 7fbeb57fa640 1 -- 172.21.15.161:0/1836325307 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec00582f0 con 0x559c82c01db0 2024-03-28T23:34:01.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.779+0000 7fbed8fdb740 1 -- 172.21.15.161:0/1836325307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c01db0 msgr2=0x559c82c4eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:01.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.779+0000 7fbed8fdb740 1 --2- 172.21.15.161:0/1836325307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c01db0 0x559c82c4eb80 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fbec0054c10 tx=0x7fbec001d040 comp rx=0 tx=0).stop 2024-03-28T23:34:01.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.779+0000 7fbed8fdb740 1 -- 172.21.15.161:0/1836325307 shutdown_connections 2024-03-28T23:34:01.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.779+0000 7fbed8fdb740 1 --2- 172.21.15.161:0/1836325307 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559c82c4f9a0 0x559c82c21600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.779+0000 7fbed8fdb740 1 --2- 172.21.15.161:0/1836325307 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c82c02190 0x559c82bff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.779+0000 7fbed8fdb740 1 --2- 172.21.15.161:0/1836325307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c01db0 0x559c82c4eb80 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.779+0000 7fbed8fdb740 1 -- 172.21.15.161:0/1836325307 >> 172.21.15.161:0/1836325307 conn(0x559c82c00310 msgr2=0x559c82c416e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:01.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.779+0000 7fbed8fdb740 1 -- 172.21.15.161:0/1836325307 shutdown_connections 2024-03-28T23:34:01.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.779+0000 7fbed8fdb740 1 -- 172.21.15.161:0/1836325307 wait complete. 2024-03-28T23:34:01.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.780+0000 7fbed8fdb740 1 -- start start 2024-03-28T23:34:01.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.780+0000 7fbed8fdb740 1 Processor -- start 2024-03-28T23:34:01.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.780+0000 7fbed8fdb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c82c01db0 0x559c82cbbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.780+0000 7fbed8fdb740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c02190 0x559c82cbad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.780+0000 7fbed8fdb740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559c82c4f9a0 0x559c82ca1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.781+0000 7fbed8fdb740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x559c82c286d0 con 0x559c82c02190 2024-03-28T23:34:01.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.781+0000 7fbed8fdb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x559c82c27240 con 0x559c82c01db0 2024-03-28T23:34:01.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.781+0000 7fbed8fdb740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x559c82c1f0c0 con 0x559c82c4f9a0 2024-03-28T23:34:01.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.781+0000 7fbed8fdb740 10 client.0 _ll_register_callbacks cb 0x7fbec0001370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:34:01.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.781+0000 7fbecf7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c02190 0x559c82cbad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.781+0000 7fbecf7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c02190 0x559c82cbad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55402/0 (socket says 172.21.15.161:55402) 2024-03-28T23:34:01.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.781+0000 7fbecf7fe640 1 -- 172.21.15.161:0/4260204559 learned_addr learned my addr 172.21.15.161:0/4260204559 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:01.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.781+0000 7fbecffff640 1 --2- 172.21.15.161:0/4260204559 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c82c01db0 0x559c82cbbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.781+0000 7fbed48bb640 1 --2- 172.21.15.161:0/4260204559 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559c82c4f9a0 0x559c82ca1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.781+0000 7fbecf7fe640 1 -- 172.21.15.161:0/4260204559 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559c82c4f9a0 msgr2=0x559c82ca1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:01.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.781+0000 7fbecf7fe640 1 --2- 172.21.15.161:0/4260204559 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559c82c4f9a0 0x559c82ca1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.781+0000 7fbecf7fe640 1 -- 172.21.15.161:0/4260204559 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c82c01db0 msgr2=0x559c82cbbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:01.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.781+0000 7fbecf7fe640 1 --2- 172.21.15.161:0/4260204559 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c82c01db0 0x559c82cbbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.781+0000 7fbecf7fe640 1 -- 172.21.15.161:0/4260204559 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c82c501d0 con 0x559c82c02190 2024-03-28T23:34:01.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.781+0000 7fbecf7fe640 1 --2- 172.21.15.161:0/4260204559 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c02190 0x559c82cbad80 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fbec4030800 tx=0x7fbec4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11fcb56903e17f74 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:01.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.781+0000 7fbeaeffd640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:34:01.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.781+0000 7fbeaeffd640 1 -- 172.21.15.161:0/4260204559 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec4053070 con 0x559c82c02190 2024-03-28T23:34:01.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.781+0000 7fbeaeffd640 1 -- 172.21.15.161:0/4260204559 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 6101+0+0 (secure 0 0 0) 0x7fbec4030b30 con 0x559c82c02190 2024-03-28T23:34:01.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.781+0000 7fbed8fdb740 1 -- 172.21.15.161:0/4260204559 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbec00701e0 con 0x559c82c02190 2024-03-28T23:34:01.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.781+0000 7fbeaeffd640 1 -- 172.21.15.161:0/4260204559 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fbec4037040 con 0x559c82c02190 2024-03-28T23:34:01.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.781+0000 7fbed8fdb740 10 client.7671 fetch_fsmap learned FSMap version 90 2024-03-28T23:34:01.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.781+0000 7fbed8fdb740 1 -- 172.21.15.161:0/4260204559 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=2+}) -- 0x559c82cba5e0 con 0x559c82c02190 2024-03-28T23:34:01.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.782+0000 7fbeaeffd640 1 -- 172.21.15.161:0/4260204559 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbec00701e0 con 0x559c82c02190 2024-03-28T23:34:01.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.782+0000 7fbeaeffd640 1 -- 172.21.15.161:0/4260204559 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 90) ==== 3494+0+0 (secure 0 0 0) 0x7fbec4030d30 con 0x559c82c02190 2024-03-28T23:34:01.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.782+0000 7fbed8fdb740 10 client.7671 fetch_fsmap finished waiting for FSMap version 90 2024-03-28T23:34:01.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.782+0000 7fbed8fdb740 10 client.7671 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T23:34:01.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.782+0000 7fbed8fdb740 20 client.7671 populate_metadata read hostname 'smithi161' 2024-03-28T23:34:01.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.782+0000 7fbed8fdb740 1 --2- 172.21.15.161:0/4260204559 >> [v2:172.21.15.161:6834/2544838511,v1:172.21.15.161:6835/2544838511] conn(0x559c82cb85e0 0x559c82cb89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:01.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.782+0000 7fbed8fdb740 4 client.7671 mds_command: new command op to 7452 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:34:01.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.782+0000 7fbed8fdb740 1 -- 172.21.15.161:0/4260204559 --> [v2:172.21.15.161:6834/2544838511,v1:172.21.15.161:6835/2544838511] -- command(tid 0: {"prefix": "status"}) -- 0x559c82b6d380 con 0x559c82cb85e0 2024-03-28T23:34:01.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.782+0000 7fbecffff640 1 --2- 172.21.15.161:0/4260204559 >> [v2:172.21.15.161:6834/2544838511,v1:172.21.15.161:6835/2544838511] conn(0x559c82cb85e0 0x559c82cb89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:01.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.782+0000 7fbeaeffd640 10 client.7671 ms_handle_connect on v2:172.21.15.161:6834/2544838511 2024-03-28T23:34:01.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.782+0000 7fbecffff640 1 --2- 172.21.15.161:0/4260204559 >> [v2:172.21.15.161:6834/2544838511,v1:172.21.15.161:6835/2544838511] conn(0x559c82cb85e0 0x559c82cb89c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b12c2eea52ca5236 server_cookie=cb7b11d34bb58133 in_seq=0 out_seq=0 2024-03-28T23:34:01.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.782+0000 7fbeaeffd640 1 -- 172.21.15.161:0/4260204559 <== mds.0 v2:172.21.15.161:6834/2544838511 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x559c82b6d380 con 0x559c82cb85e0 2024-03-28T23:34:01.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.782+0000 7fbeaeffd640 10 client.7671 handle_command_reply: tid=0 2024-03-28T23:34:01.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.782+0000 7fbed8fdb740 1 client.7671 shutdown 2024-03-28T23:34:01.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.782+0000 7fbed8fdb740 1 -- 172.21.15.161:0/4260204559 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c02190 msgr2=0x559c82cbad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:01.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.782+0000 7fbed8fdb740 1 --2- 172.21.15.161:0/4260204559 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c02190 0x559c82cbad80 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fbec4030800 tx=0x7fbec4004730 comp rx=0 tx=0).stop 2024-03-28T23:34:01.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.783+0000 7fbed8fdb740 1 -- 172.21.15.161:0/4260204559 shutdown_connections 2024-03-28T23:34:01.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.783+0000 7fbed8fdb740 1 --2- 172.21.15.161:0/4260204559 >> [v2:172.21.15.161:6834/2544838511,v1:172.21.15.161:6835/2544838511] conn(0x559c82cb85e0 0x559c82cb89c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.783+0000 7fbed8fdb740 1 --2- 172.21.15.161:0/4260204559 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c82c02190 0x559c82cbad80 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.783+0000 7fbeaeffd640 0 client.7671 ms_handle_reset on v2:172.21.15.161:6834/2544838511 2024-03-28T23:34:01.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.783+0000 7fbed8fdb740 1 --2- 172.21.15.161:0/4260204559 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559c82c4f9a0 0x559c82ca1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.783+0000 7fbed8fdb740 1 --2- 172.21.15.161:0/4260204559 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559c82c01db0 0x559c82cbbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.783+0000 7fbed8fdb740 1 -- 172.21.15.161:0/4260204559 >> 172.21.15.161:0/4260204559 conn(0x559c82c00310 msgr2=0x559c82bff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:01.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.783+0000 7fbed8fdb740 1 -- 172.21.15.161:0/4260204559 shutdown_connections 2024-03-28T23:34:01.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.783+0000 7fbed8fdb740 1 -- 172.21.15.161:0/4260204559 wait complete. 2024-03-28T23:34:01.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.783+0000 7fbed8fdb740 20 client.7671 trim_cache size 0 max 16384 2024-03-28T23:34:01.791 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:34:01.792 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:34:01.792 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:34:01.792 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 7452, 2024-03-28T23:34:01.792 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:34:01.792 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:34:01.792 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:34:01.792 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 10.775874895999999, 2024-03-28T23:34:01.792 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 90, 2024-03-28T23:34:01.792 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 109, 2024-03-28T23:34:01.792 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 109, 2024-03-28T23:34:01.792 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 15.901524884000001 2024-03-28T23:34:01.792 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:34:01.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.783+0000 7fbed631e640 1 -- 172.21.15.161:0/460448996 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbebc06b4d0 msgr2=0x7fbebc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:01.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.783+0000 7fbed631e640 1 --2- 172.21.15.161:0/460448996 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbebc06b4d0 0x7fbebc06d990 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fbec4034000 tx=0x7fbec4008030 comp rx=0 tx=0).stop 2024-03-28T23:34:01.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.783+0000 7fbed631e640 1 -- 172.21.15.161:0/460448996 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbed00c5410 msgr2=0x7fbed01eebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:01.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.783+0000 7fbed631e640 1 --2- 172.21.15.161:0/460448996 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbed00c5410 0x7fbed01eebc0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fbec00017e0 tx=0x7fbec000cbe0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.783+0000 7fbed631e640 1 -- 172.21.15.161:0/460448996 shutdown_connections 2024-03-28T23:34:01.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.783+0000 7fbed631e640 1 --2- 172.21.15.161:0/460448996 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbebc06b4d0 0x7fbebc06d990 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.783+0000 7fbed631e640 1 --2- 172.21.15.161:0/460448996 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbed01553a0 0x7fbed00abce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.783+0000 7fbed631e640 1 --2- 172.21.15.161:0/460448996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbed0152890 0x7fbed01eb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.783+0000 7fbed631e640 1 --2- 172.21.15.161:0/460448996 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbed00c5410 0x7fbed01eebc0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:01.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.783+0000 7fbed631e640 1 -- 172.21.15.161:0/460448996 >> 172.21.15.161:0/460448996 conn(0x7fbed00ac620 msgr2=0x7fbed0074020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:01.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.783+0000 7fbed631e640 1 -- 172.21.15.161:0/460448996 shutdown_connections 2024-03-28T23:34:01.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:01.783+0000 7fbed631e640 1 -- 172.21.15.161:0/460448996 wait complete. 2024-03-28T23:34:01.795 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 90, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:33:50.326492+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 90, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7452": { "addr": "172.21.15.161:6835/2544838511", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 2544838511, "type": "v2" }, { "addr": "172.21.15.161:6835", "nonce": 2544838511, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7452, "incarnation": 88, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-03-28T23:33:52.354643+0000", "qdb_cluster": [ 7452 ], "qdb_leader": 7452, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7452 } } } ], "standbys": [ { "addr": "172.21.15.170:6834/1096504848", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 1096504848, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 1096504848, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 87, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6203, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.170:6835/3351418222", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 3351418222, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 3351418222, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 87, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6206, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6837/2684760895", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6836", "nonce": 2684760895, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 2684760895, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 87, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6892, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:34:01.796 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 8, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-28T23:34:01.796 INFO:tasks.cephfs.test_failover:current = [{'gid': 7452, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.161:6835/2544838511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 2544838511}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 2544838511}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 6203, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6834/1096504848', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 1096504848}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 1096504848}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 87}, {'gid': 6892, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6837/2684760895', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 2684760895}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 2684760895}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 87}, {'gid': 6206, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/3351418222', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 3351418222}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 3351418222}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 87}] 2024-03-28T23:34:01.796 DEBUG:tasks.cephfs.test_failover:8 != -1 2024-03-28T23:34:01.796 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-28T23:34:02.038 INFO:tasks.ceph.mds.a.smithi161.stderr: -264> 2024-03-28T23:33:56.261+0000 7ff84eb1f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:34:02.062 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-28T23:34:06.263 INFO:tasks.ceph.mds.b.smithi170.stderr:2024-03-28T23:34:06.262+0000 7fadcc9a7640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:34:06.798 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:34:06.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.911+0000 7ff5eef56640 1 Processor -- start 2024-03-28T23:34:06.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.911+0000 7ff5eef56640 1 -- start start 2024-03-28T23:34:06.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.911+0000 7ff5eef56640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff5e814eca0 0x7ff5e80c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:06.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.911+0000 7ff5eef56640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff5e80c1dc0 0x7ff5e806ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:06.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.911+0000 7ff5eef56640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5e8151850 con 0x7ff5e806cf60 2024-03-28T23:34:06.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.911+0000 7ff5eef56640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff5e8152460 con 0x7ff5e814eca0 2024-03-28T23:34:06.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.911+0000 7ff5eef56640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7ff5e8151b30 con 0x7ff5e80c1dc0 2024-03-28T23:34:06.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.912+0000 7ff5eccf2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff5e814eca0 0x7ff5e80c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:06.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.912+0000 7ff5ed4f3640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ff5e806cf60 0x7ff5e80a98c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59816/0 (socket says 172.21.15.161:59816) 2024-03-28T23:34:06.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.912+0000 7ff5ed4f3640 1 -- 172.21.15.161:0/1356753171 learned_addr learned my addr 172.21.15.161:0/1356753171 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:06.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.912+0000 7ff5eccf2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff5e814eca0 0x7ff5e80c1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55428/0 (socket says 172.21.15.161:55428) 2024-03-28T23:34:06.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.912+0000 7ff5e7fff640 1 --2- 172.21.15.161:0/1356753171 >> v2:172.21.15.170:3300/0 conn(0x7ff5e80c1dc0 0x7ff5e806ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:06.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.912+0000 7ff5eccf2640 1 -- 172.21.15.161:0/1356753171 >> v2:172.21.15.170:3300/0 conn(0x7ff5e80c1dc0 msgr2=0x7ff5e806ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:06.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.912+0000 7ff5eccf2640 1 --2- 172.21.15.161:0/1356753171 >> v2:172.21.15.170:3300/0 conn(0x7ff5e80c1dc0 0x7ff5e806ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:06.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.912+0000 7ff5eccf2640 1 -- 172.21.15.161:0/1356753171 >> v1:172.21.15.161:6789/0 conn(0x7ff5e806cf60 legacy=0x7ff5e80a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:06.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.912+0000 7ff5eccf2640 1 -- 172.21.15.161:0/1356753171 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e8151e10 con 0x7ff5e814eca0 2024-03-28T23:34:06.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.912+0000 7ff5e77fe640 1 -- 172.21.15.161:0/1356753171 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 643155310 0 0) 0x7ff5e8151850 con 0x7ff5e806cf60 2024-03-28T23:34:06.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.913+0000 7ff5eccf2640 1 --2- 172.21.15.161:0/1356753171 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5e814eca0 0x7ff5e80c1880 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80c3070 tx=0x7ff5e002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7679e5f41fea164d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:06.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.913+0000 7ff5e77fe640 1 -- 172.21.15.161:0/1356753171 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0039070 con 0x7ff5e814eca0 2024-03-28T23:34:06.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.913+0000 7ff5e77fe640 1 -- 172.21.15.161:0/1356753171 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5e002cbf0 con 0x7ff5e814eca0 2024-03-28T23:34:06.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.913+0000 7ff5e77fe640 1 -- 172.21.15.161:0/1356753171 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0034440 con 0x7ff5e814eca0 2024-03-28T23:34:06.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.913+0000 7ff5eef56640 1 -- 172.21.15.161:0/1356753171 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5e814eca0 msgr2=0x7ff5e80c1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:06.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.913+0000 7ff5eef56640 1 --2- 172.21.15.161:0/1356753171 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5e814eca0 0x7ff5e80c1880 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80c3070 tx=0x7ff5e002be40 comp rx=0 tx=0).stop 2024-03-28T23:34:06.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.914+0000 7ff5eef56640 1 -- 172.21.15.161:0/1356753171 shutdown_connections 2024-03-28T23:34:06.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.914+0000 7ff5eef56640 1 --2- 172.21.15.161:0/1356753171 >> v2:172.21.15.170:3300/0 conn(0x7ff5e80c1dc0 0x7ff5e806ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:06.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.914+0000 7ff5eef56640 1 --2- 172.21.15.161:0/1356753171 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5e814eca0 0x7ff5e80c1880 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:06.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.914+0000 7ff5eef56640 1 -- 172.21.15.161:0/1356753171 >> 172.21.15.161:0/1356753171 conn(0x7ff5e80c6670 msgr2=0x7ff5e80c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:06.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.914+0000 7ff5eef56640 1 -- 172.21.15.161:0/1356753171 shutdown_connections 2024-03-28T23:34:06.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.914+0000 7ff5eef56640 1 -- 172.21.15.161:0/1356753171 wait complete. 2024-03-28T23:34:06.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.915+0000 7ff5eef56640 1 Processor -- start 2024-03-28T23:34:06.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.915+0000 7ff5eef56640 1 -- start start 2024-03-28T23:34:06.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.915+0000 7ff5eef56640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5e806cf60 0x7ff5e8152fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:06.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.915+0000 7ff5eef56640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5e80c1dc0 0x7ff5e80c32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:06.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.915+0000 7ff5eef56640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5e814eca0 0x7ff5e81400d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:06.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.916+0000 7ff5eef56640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e81514b0 con 0x7ff5e80c1dc0 2024-03-28T23:34:06.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.916+0000 7ff5eef56640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e814f080 con 0x7ff5e806cf60 2024-03-28T23:34:06.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.916+0000 7ff5eef56640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff5e8151630 con 0x7ff5e814eca0 2024-03-28T23:34:06.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.916+0000 7ff5e7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5e80c1dc0 0x7ff5e80c32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:06.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.916+0000 7ff5e7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5e80c1dc0 0x7ff5e80c32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55430/0 (socket says 172.21.15.161:55430) 2024-03-28T23:34:06.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.916+0000 7ff5e7fff640 1 -- 172.21.15.161:0/240819062 learned_addr learned my addr 172.21.15.161:0/240819062 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:06.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.916+0000 7ff5eccf2640 1 --2- 172.21.15.161:0/240819062 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5e806cf60 0x7ff5e8152fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:06.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.916+0000 7ff5ed4f3640 1 --2- 172.21.15.161:0/240819062 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5e814eca0 0x7ff5e81400d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:06.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.916+0000 7ff5e7fff640 1 -- 172.21.15.161:0/240819062 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5e814eca0 msgr2=0x7ff5e81400d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:06.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.916+0000 7ff5e7fff640 1 --2- 172.21.15.161:0/240819062 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5e814eca0 0x7ff5e81400d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:06.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.916+0000 7ff5e7fff640 1 -- 172.21.15.161:0/240819062 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5e806cf60 msgr2=0x7ff5e8152fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:06.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.916+0000 7ff5e7fff640 1 --2- 172.21.15.161:0/240819062 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5e806cf60 0x7ff5e8152fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:06.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.916+0000 7ff5e7fff640 1 -- 172.21.15.161:0/240819062 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e81454e0 con 0x7ff5e80c1dc0 2024-03-28T23:34:06.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.916+0000 7ff5e7fff640 1 --2- 172.21.15.161:0/240819062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5e80c1dc0 0x7ff5e80c32e0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7ff5d000e9e0 tx=0x7ff5d000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:06.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.916+0000 7ff5e57fa640 1 -- 172.21.15.161:0/240819062 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0017070 con 0x7ff5e80c1dc0 2024-03-28T23:34:06.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.916+0000 7ff5e57fa640 1 -- 172.21.15.161:0/240819062 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5d000c9d0 con 0x7ff5e80c1dc0 2024-03-28T23:34:06.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.916+0000 7ff5e57fa640 1 -- 172.21.15.161:0/240819062 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d000cdc0 con 0x7ff5e80c1dc0 2024-03-28T23:34:06.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.916+0000 7ff5eef56640 1 -- 172.21.15.161:0/240819062 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5e81457c0 con 0x7ff5e80c1dc0 2024-03-28T23:34:06.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.916+0000 7ff5eef56640 1 -- 172.21.15.161:0/240819062 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5e813f8d0 con 0x7ff5e80c1dc0 2024-03-28T23:34:06.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.917+0000 7ff5eef56640 1 -- 172.21.15.161:0/240819062 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5e804f4b0 con 0x7ff5e80c1dc0 2024-03-28T23:34:06.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.918+0000 7ff5e57fa640 1 -- 172.21.15.161:0/240819062 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff5d001a070 con 0x7ff5e80c1dc0 2024-03-28T23:34:06.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.918+0000 7ff5e57fa640 1 --2- 172.21.15.161:0/240819062 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff5c806b4d0 0x7ff5c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:06.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.918+0000 7ff5e57fa640 1 -- 172.21.15.161:0/240819062 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7ff5d008a290 con 0x7ff5e80c1dc0 2024-03-28T23:34:06.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.918+0000 7ff5eccf2640 1 --2- 172.21.15.161:0/240819062 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff5c806b4d0 0x7ff5c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:06.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.919+0000 7ff5eccf2640 1 --2- 172.21.15.161:0/240819062 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff5c806b4d0 0x7ff5c806d990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0036000 tx=0x7ff5e0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:06.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:06.923+0000 7ff5e57fa640 1 -- 172.21.15.161:0/240819062 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff5d0058970 con 0x7ff5e80c1dc0 2024-03-28T23:34:07.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.137+0000 7ff5eef56640 1 -- 172.21.15.161:0/240819062 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5e8144910 con 0x7ff5e80c1dc0 2024-03-28T23:34:07.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.137+0000 7ff5e57fa640 1 -- 172.21.15.161:0/240819062 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) ==== 94+0+5196 (secure 0 0 0) 0x7ff5d005ca10 con 0x7ff5e80c1dc0 2024-03-28T23:34:07.138 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:07.138 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":95,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6206,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3351418222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3351418222},{"type":"v1","addr":"172.21.15.170:6835","nonce":3351418222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":6892,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/2684760895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2684760895},{"type":"v1","addr":"172.21.15.161:6837","nonce":2684760895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":7680,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/2943346687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":2943346687},{"type":"v1","addr":"172.21.15.161:6835","nonce":2943346687}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":93}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:33:50.326492+0000","modified":"2024-03-28T23:34:05.043006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6203":{"gid":6203,"name":"b","rank":0,"incarnation":92,"state":"up:active","state_seq":8,"addr":"172.21.15.170:6834/1096504848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1096504848},{"type":"v1","addr":"172.21.15.170:6834","nonce":1096504848}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6203,"qdb_cluster":[6203]},"id":8}]} 2024-03-28T23:34:07.138 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 95 2024-03-28T23:34:07.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.139+0000 7ff5eef56640 1 -- 172.21.15.161:0/240819062 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff5c806b4d0 msgr2=0x7ff5c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:07.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.139+0000 7ff5eef56640 1 --2- 172.21.15.161:0/240819062 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff5c806b4d0 0x7ff5c806d990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0036000 tx=0x7ff5e0030000 comp rx=0 tx=0).stop 2024-03-28T23:34:07.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.139+0000 7ff5eef56640 1 -- 172.21.15.161:0/240819062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5e80c1dc0 msgr2=0x7ff5e80c32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:07.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.139+0000 7ff5eef56640 1 --2- 172.21.15.161:0/240819062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5e80c1dc0 0x7ff5e80c32e0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7ff5d000e9e0 tx=0x7ff5d000eea0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.139+0000 7ff5eef56640 1 -- 172.21.15.161:0/240819062 shutdown_connections 2024-03-28T23:34:07.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.139+0000 7ff5eef56640 1 --2- 172.21.15.161:0/240819062 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff5c806b4d0 0x7ff5c806d990 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.139+0000 7ff5eef56640 1 --2- 172.21.15.161:0/240819062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5e80c1dc0 0x7ff5e80c32e0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.139+0000 7ff5eef56640 1 --2- 172.21.15.161:0/240819062 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5e814eca0 0x7ff5e81400d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.139+0000 7ff5eef56640 1 --2- 172.21.15.161:0/240819062 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5e806cf60 0x7ff5e8152fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.139+0000 7ff5eef56640 1 -- 172.21.15.161:0/240819062 >> 172.21.15.161:0/240819062 conn(0x7ff5e80c6670 msgr2=0x7ff5e814f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:07.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.139+0000 7ff5eef56640 1 -- 172.21.15.161:0/240819062 shutdown_connections 2024-03-28T23:34:07.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.139+0000 7ff5eef56640 1 -- 172.21.15.161:0/240819062 wait complete. 2024-03-28T23:34:07.553 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:33:50.326492+0000', 'modified': '2024-03-28T23:34:05.043006+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 110, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6203}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6203': {'gid': 6203, 'name': 'b', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.170:6834/1096504848', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 1096504848}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 1096504848}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6203, 'qdb_cluster': [6203]} 2024-03-28T23:34:07.554 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:34:07.554 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T23:34:07.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.657+0000 7fe1de33e640 1 Processor -- start 2024-03-28T23:34:07.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.657+0000 7fe1de33e640 1 -- start start 2024-03-28T23:34:07.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.658+0000 7fe1de33e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe1d806ce80 0x7fe1d80aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.658+0000 7fe1de33e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1d814c250 con 0x7fe1d80a9680 2024-03-28T23:34:07.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.658+0000 7fe1de33e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1d81529c0 con 0x7fe1d814be70 2024-03-28T23:34:07.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.658+0000 7fe1de33e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe1d80a9dc0 con 0x7fe1d806ce80 2024-03-28T23:34:07.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.658+0000 7fe1d77fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe1d806ce80 0x7fe1d80aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.658+0000 7fe1dc8db640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe1d80a9680 0x7fe1d815c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59818/0 (socket says 172.21.15.161:59818) 2024-03-28T23:34:07.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.658+0000 7fe1d77fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe1d806ce80 0x7fe1d80aa5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55440/0 (socket says 172.21.15.161:55440) 2024-03-28T23:34:07.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.658+0000 7fe1dc8db640 1 -- 172.21.15.161:0/776835805 learned_addr learned my addr 172.21.15.161:0/776835805 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:07.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.658+0000 7fe1d77fe640 1 -- 172.21.15.161:0/776835805 >> v1:172.21.15.170:6789/0 conn(0x7fe1d814be70 legacy=0x7fe1d80ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:07.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.658+0000 7fe1d77fe640 1 -- 172.21.15.161:0/776835805 >> v1:172.21.15.161:6789/0 conn(0x7fe1d80a9680 legacy=0x7fe1d815c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:07.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.658+0000 7fe1d77fe640 1 -- 172.21.15.161:0/776835805 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1d80aa0a0 con 0x7fe1d806ce80 2024-03-28T23:34:07.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.658+0000 7fe1d6ffd640 1 -- 172.21.15.161:0/776835805 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 659836512 0 0) 0x7fe1d81529c0 con 0x7fe1d814be70 2024-03-28T23:34:07.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.659+0000 7fe1d77fe640 1 --2- 172.21.15.161:0/776835805 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1d806ce80 0x7fe1d80aa5c0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fe1c80097a0 tx=0x7fe1c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=764b65b3c6d8439e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:07.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.659+0000 7fe1d6ffd640 1 -- 172.21.15.161:0/776835805 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8035020 con 0x7fe1d806ce80 2024-03-28T23:34:07.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.659+0000 7fe1d6ffd640 1 -- 172.21.15.161:0/776835805 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1c8031830 con 0x7fe1d806ce80 2024-03-28T23:34:07.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.659+0000 7fe1d6ffd640 1 -- 172.21.15.161:0/776835805 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8031c00 con 0x7fe1d806ce80 2024-03-28T23:34:07.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.659+0000 7fe1de33e640 1 -- 172.21.15.161:0/776835805 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1d806ce80 msgr2=0x7fe1d80aa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:07.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.659+0000 7fe1de33e640 1 --2- 172.21.15.161:0/776835805 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1d806ce80 0x7fe1d80aa5c0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fe1c80097a0 tx=0x7fe1c802b8d0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.659+0000 7fe1de33e640 1 -- 172.21.15.161:0/776835805 shutdown_connections 2024-03-28T23:34:07.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.659+0000 7fe1de33e640 1 --2- 172.21.15.161:0/776835805 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1d806ce80 0x7fe1d80aa5c0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.660+0000 7fe1de33e640 1 -- 172.21.15.161:0/776835805 >> 172.21.15.161:0/776835805 conn(0x7fe1d80c6670 msgr2=0x7fe1d80c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:07.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.660+0000 7fe1de33e640 1 -- 172.21.15.161:0/776835805 shutdown_connections 2024-03-28T23:34:07.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.660+0000 7fe1de33e640 1 -- 172.21.15.161:0/776835805 wait complete. 2024-03-28T23:34:07.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.661+0000 7fe1de33e640 1 Processor -- start 2024-03-28T23:34:07.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.661+0000 7fe1de33e640 1 -- start start 2024-03-28T23:34:07.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.661+0000 7fe1de33e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1d806ce80 0x7fe1d81de910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.661+0000 7fe1de33e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1d80a9680 0x7fe1d814f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.661+0000 7fe1de33e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe1d814be70 0x7fe1d8148910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.661+0000 7fe1de33e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d80aba30 con 0x7fe1d806ce80 2024-03-28T23:34:07.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.661+0000 7fe1de33e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d80ab8b0 con 0x7fe1d80a9680 2024-03-28T23:34:07.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.661+0000 7fe1de33e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe1d806b150 con 0x7fe1d814be70 2024-03-28T23:34:07.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.662+0000 7fe1d7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1d806ce80 0x7fe1d81de910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.662+0000 7fe1d7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1d806ce80 0x7fe1d81de910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55450/0 (socket says 172.21.15.161:55450) 2024-03-28T23:34:07.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.662+0000 7fe1d7fff640 1 -- 172.21.15.161:0/469122879 learned_addr learned my addr 172.21.15.161:0/469122879 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:07.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.662+0000 7fe1dc8db640 1 --2- 172.21.15.161:0/469122879 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe1d814be70 0x7fe1d8148910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.662+0000 7fe1d77fe640 1 --2- 172.21.15.161:0/469122879 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1d80a9680 0x7fe1d814f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.662+0000 7fe1d7fff640 1 -- 172.21.15.161:0/469122879 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe1d814be70 msgr2=0x7fe1d8148910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:07.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.662+0000 7fe1d7fff640 1 --2- 172.21.15.161:0/469122879 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe1d814be70 0x7fe1d8148910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.662+0000 7fe1d7fff640 1 -- 172.21.15.161:0/469122879 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1d80a9680 msgr2=0x7fe1d814f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:07.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.662+0000 7fe1d7fff640 1 --2- 172.21.15.161:0/469122879 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1d80a9680 0x7fe1d814f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.662+0000 7fe1d7fff640 1 -- 172.21.15.161:0/469122879 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1d80c09f0 con 0x7fe1d806ce80 2024-03-28T23:34:07.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.662+0000 7fe1d7fff640 1 --2- 172.21.15.161:0/469122879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1d806ce80 0x7fe1d81de910 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000c410 tx=0x7fe1c000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:07.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.662+0000 7fe1d4ff9640 1 -- 172.21.15.161:0/469122879 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0017070 con 0x7fe1d806ce80 2024-03-28T23:34:07.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.662+0000 7fe1d4ff9640 1 -- 172.21.15.161:0/469122879 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1c00035f0 con 0x7fe1d806ce80 2024-03-28T23:34:07.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.662+0000 7fe1d4ff9640 1 -- 172.21.15.161:0/469122879 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c00039e0 con 0x7fe1d806ce80 2024-03-28T23:34:07.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.662+0000 7fe1de33e640 1 -- 172.21.15.161:0/469122879 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1d80c0c90 con 0x7fe1d806ce80 2024-03-28T23:34:07.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.662+0000 7fe1de33e640 1 -- 172.21.15.161:0/469122879 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1d81e91f0 con 0x7fe1d806ce80 2024-03-28T23:34:07.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.664+0000 7fe1d4ff9640 1 -- 172.21.15.161:0/469122879 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe1c001a070 con 0x7fe1d806ce80 2024-03-28T23:34:07.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.664+0000 7fe1d4ff9640 1 --2- 172.21.15.161:0/469122879 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe1b806b4d0 0x7fe1b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.664+0000 7fe1d4ff9640 1 -- 172.21.15.161:0/469122879 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7fe1c0089780 con 0x7fe1d806ce80 2024-03-28T23:34:07.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.664+0000 7fe1d77fe640 1 --2- 172.21.15.161:0/469122879 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe1b806b4d0 0x7fe1b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.664+0000 7fe1d77fe640 1 --2- 172.21.15.161:0/469122879 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe1b806b4d0 0x7fe1b806d990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8036000 tx=0x7fe1c803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:07.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.665+0000 7fe1e0ff6740 1 Processor -- start 2024-03-28T23:34:07.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.665+0000 7fe1e0ff6740 1 -- start start 2024-03-28T23:34:07.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.666+0000 7fe1e0ff6740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2cdb0 0x55b5adb2a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.666+0000 7fe1e0ff6740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b5adb2d190 0x55b5adb2add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.666+0000 7fe1e0ff6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b5adb2b310 0x55b5adb274e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.666+0000 7fe1e0ff6740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55b5ada90af0 con 0x55b5adb2cdb0 2024-03-28T23:34:07.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.666+0000 7fe1e0ff6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55b5ada399d0 con 0x55b5adb2b310 2024-03-28T23:34:07.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.666+0000 7fe1e0ff6740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55b5ada8cb90 con 0x55b5adb2d190 2024-03-28T23:34:07.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.666+0000 7fe1d7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2cdb0 0x55b5adb2a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.666+0000 7fe1d7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2cdb0 0x55b5adb2a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55460/0 (socket says 172.21.15.161:55460) 2024-03-28T23:34:07.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.666+0000 7fe1d7fff640 1 -- 172.21.15.161:0/1421951059 learned_addr learned my addr 172.21.15.161:0/1421951059 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:07.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.666+0000 7fe1d77fe640 1 --2- 172.21.15.161:0/1421951059 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b5adb2d190 0x55b5adb2add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.666+0000 7fe1dc8db640 1 --2- 172.21.15.161:0/1421951059 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b5adb2b310 0x55b5adb274e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.666+0000 7fe1d7fff640 1 -- 172.21.15.161:0/1421951059 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b5adb2d190 msgr2=0x55b5adb2add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:07.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.666+0000 7fe1d7fff640 1 --2- 172.21.15.161:0/1421951059 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b5adb2d190 0x55b5adb2add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.666+0000 7fe1d7fff640 1 -- 172.21.15.161:0/1421951059 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b5adb2b310 msgr2=0x55b5adb274e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:07.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.666+0000 7fe1d7fff640 1 --2- 172.21.15.161:0/1421951059 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b5adb2b310 0x55b5adb274e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.666+0000 7fe1d7fff640 1 -- 172.21.15.161:0/1421951059 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5ad7a52d0 con 0x55b5adb2cdb0 2024-03-28T23:34:07.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.666+0000 7fe1d7fff640 1 --2- 172.21.15.161:0/1421951059 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2cdb0 0x55b5adb2a890 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fe1c005ff00 tx=0x7fe1c005be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8006716ccde99dba server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:07.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.666+0000 7fe1cdffb640 1 -- 172.21.15.161:0/1421951059 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c005e070 con 0x55b5adb2cdb0 2024-03-28T23:34:07.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.666+0000 7fe1cdffb640 1 -- 172.21.15.161:0/1421951059 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1c0054440 con 0x55b5adb2cdb0 2024-03-28T23:34:07.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.666+0000 7fe1cdffb640 1 -- 172.21.15.161:0/1421951059 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0003d80 con 0x55b5adb2cdb0 2024-03-28T23:34:07.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.667+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/1421951059 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2cdb0 msgr2=0x55b5adb2a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:07.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.667+0000 7fe1e0ff6740 1 --2- 172.21.15.161:0/1421951059 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2cdb0 0x55b5adb2a890 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fe1c005ff00 tx=0x7fe1c005be70 comp rx=0 tx=0).stop 2024-03-28T23:34:07.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.667+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/1421951059 shutdown_connections 2024-03-28T23:34:07.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.667+0000 7fe1e0ff6740 1 --2- 172.21.15.161:0/1421951059 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b5adb2b310 0x55b5adb274e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.667+0000 7fe1e0ff6740 1 --2- 172.21.15.161:0/1421951059 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b5adb2d190 0x55b5adb2add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.667+0000 7fe1e0ff6740 1 --2- 172.21.15.161:0/1421951059 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2cdb0 0x55b5adb2a890 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.667+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/1421951059 >> 172.21.15.161:0/1421951059 conn(0x55b5adb29630 msgr2=0x55b5adb4e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:07.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.667+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/1421951059 shutdown_connections 2024-03-28T23:34:07.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.667+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/1421951059 wait complete. 2024-03-28T23:34:07.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.667+0000 7fe1e0ff6740 1 -- start start 2024-03-28T23:34:07.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.668+0000 7fe1e0ff6740 1 Processor -- start 2024-03-28T23:34:07.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.668+0000 7fe1e0ff6740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2cdb0 0x55b5adb4d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.668+0000 7fe1e0ff6740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b5adb2d190 0x55b5adb7c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.668+0000 7fe1e0ff6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b5adb504b0 0x55b5adb49d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.668+0000 7fe1e0ff6740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55b5adb4f090 con 0x55b5adb2cdb0 2024-03-28T23:34:07.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.668+0000 7fe1e0ff6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55b5ada4cf70 con 0x55b5adb504b0 2024-03-28T23:34:07.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.668+0000 7fe1e0ff6740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55b5adb4ead0 con 0x55b5adb2d190 2024-03-28T23:34:07.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.668+0000 7fe1e0ff6740 10 client.0 _ll_register_callbacks cb 0x55b5ad7a4ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:34:07.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.668+0000 7fe1d7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2cdb0 0x55b5adb4d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.668+0000 7fe1d7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2cdb0 0x55b5adb4d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55474/0 (socket says 172.21.15.161:55474) 2024-03-28T23:34:07.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.668+0000 7fe1d77fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b5adb2d190 0x55b5adb7c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.668+0000 7fe1d7fff640 1 -- 172.21.15.161:0/3227215221 learned_addr learned my addr 172.21.15.161:0/3227215221 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:07.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.669+0000 7fe1dc8db640 1 --2- 172.21.15.161:0/3227215221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b5adb504b0 0x55b5adb49d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.669+0000 7fe1d7fff640 1 -- 172.21.15.161:0/3227215221 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b5adb2d190 msgr2=0x55b5adb7c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:07.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.669+0000 7fe1d7fff640 1 --2- 172.21.15.161:0/3227215221 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b5adb2d190 0x55b5adb7c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.669+0000 7fe1d7fff640 1 -- 172.21.15.161:0/3227215221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b5adb504b0 msgr2=0x55b5adb49d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:07.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.669+0000 7fe1d7fff640 1 --2- 172.21.15.161:0/3227215221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b5adb504b0 0x55b5adb49d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.669+0000 7fe1d7fff640 1 -- 172.21.15.161:0/3227215221 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5ad86f6b0 con 0x55b5adb2cdb0 2024-03-28T23:34:07.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.669+0000 7fe1d7fff640 1 --2- 172.21.15.161:0/3227215221 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2cdb0 0x55b5adb4d610 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0000f60 tx=0x7fe1c0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=464a436ed1fa5823 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:07.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.669+0000 7fe1b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:34:07.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.669+0000 7fe1b6ffd640 1 -- 172.21.15.161:0/3227215221 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0057220 con 0x55b5adb2cdb0 2024-03-28T23:34:07.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.669+0000 7fe1b6ffd640 1 -- 172.21.15.161:0/3227215221 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7fe1c00573c0 con 0x55b5adb2cdb0 2024-03-28T23:34:07.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.669+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/3227215221 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1c0054440 con 0x55b5adb2cdb0 2024-03-28T23:34:07.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.669+0000 7fe1b6ffd640 1 -- 172.21.15.161:0/3227215221 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7fe1c007b050 con 0x55b5adb2cdb0 2024-03-28T23:34:07.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.669+0000 7fe1e0ff6740 10 client.7704 fetch_fsmap learned FSMap version 95 2024-03-28T23:34:07.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.669+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/3227215221 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=2+}) -- 0x55b5adb53870 con 0x55b5adb2cdb0 2024-03-28T23:34:07.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.669+0000 7fe1b6ffd640 1 -- 172.21.15.161:0/3227215221 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1c0054440 con 0x55b5adb2cdb0 2024-03-28T23:34:07.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.669+0000 7fe1b6ffd640 1 -- 172.21.15.161:0/3227215221 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 95) ==== 3494+0+0 (secure 0 0 0) 0x7fe1c006bce0 con 0x55b5adb2cdb0 2024-03-28T23:34:07.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.669+0000 7fe1e0ff6740 10 client.7704 fetch_fsmap finished waiting for FSMap version 95 2024-03-28T23:34:07.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.669+0000 7fe1e0ff6740 10 client.7704 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:34:07.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.669+0000 7fe1e0ff6740 20 client.7704 populate_metadata read hostname 'smithi161' 2024-03-28T23:34:07.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.670+0000 7fe1e0ff6740 1 --2- 172.21.15.161:0/3227215221 >> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] conn(0x55b5adb4f460 0x55b5adb4f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.670+0000 7fe1e0ff6740 4 client.7704 mds_command: new command op to 6203 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:34:07.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.670+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/3227215221 --> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5ada9e030 con 0x55b5adb4f460 2024-03-28T23:34:07.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.670+0000 7fe1d77fe640 1 --2- 172.21.15.161:0/3227215221 >> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] conn(0x55b5adb4f460 0x55b5adb4f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.670+0000 7fe1d77fe640 1 --2- 172.21.15.161:0/3227215221 >> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] conn(0x55b5adb4f460 0x55b5adb4f880 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81b73cd34319de97 server_cookie=295a9a87d144af62 in_seq=0 out_seq=0 2024-03-28T23:34:07.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.670+0000 7fe1b6ffd640 10 client.7704 ms_handle_connect on v2:172.21.15.170:6832/1096504848 2024-03-28T23:34:07.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.671+0000 7fe1b6ffd640 1 -- 172.21.15.161:0/3227215221 <== mds.0 v2:172.21.15.170:6832/1096504848 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b5ada9e030 con 0x55b5adb4f460 2024-03-28T23:34:07.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.671+0000 7fe1b6ffd640 10 client.7704 handle_command_reply: tid=0 2024-03-28T23:34:07.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.671+0000 7fe1e0ff6740 1 client.7704 shutdown 2024-03-28T23:34:07.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.671+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/3227215221 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2cdb0 msgr2=0x55b5adb4d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:07.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.671+0000 7fe1e0ff6740 1 --2- 172.21.15.161:0/3227215221 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2cdb0 0x55b5adb4d610 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0000f60 tx=0x7fe1c0000ed0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.671+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/3227215221 shutdown_connections 2024-03-28T23:34:07.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.671+0000 7fe1e0ff6740 1 --2- 172.21.15.161:0/3227215221 >> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] conn(0x55b5adb4f460 0x55b5adb4f880 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.671+0000 7fe1e0ff6740 1 --2- 172.21.15.161:0/3227215221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b5adb504b0 0x55b5adb49d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.671+0000 7fe1b6ffd640 0 client.7704 ms_handle_reset on v2:172.21.15.170:6832/1096504848 2024-03-28T23:34:07.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.671+0000 7fe1e0ff6740 1 --2- 172.21.15.161:0/3227215221 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b5adb2d190 0x55b5adb7c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.671+0000 7fe1e0ff6740 1 --2- 172.21.15.161:0/3227215221 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2cdb0 0x55b5adb4d610 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.671+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/3227215221 >> 172.21.15.161:0/3227215221 conn(0x55b5adb2b310 msgr2=0x55b5adb699b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:07.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.671+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/3227215221 shutdown_connections 2024-03-28T23:34:07.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.671+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/3227215221 wait complete. 2024-03-28T23:34:07.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.671+0000 7fe1e0ff6740 20 client.7704 trim_cache size 0 max 16384 2024-03-28T23:34:07.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.689+0000 7fe1e0ff6740 1 Processor -- start 2024-03-28T23:34:07.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.689+0000 7fe1e0ff6740 1 -- start start 2024-03-28T23:34:07.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.689+0000 7fe1e0ff6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b5adb2cdb0 0x55b5adb50a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.689+0000 7fe1e0ff6740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2d190 0x55b5adb4dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.689+0000 7fe1e0ff6740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b5adb4e1e0 0x55b5adb2a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.689+0000 7fe1e0ff6740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d81f4d40 con 0x55b5adb2d190 2024-03-28T23:34:07.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.689+0000 7fe1e0ff6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d81ecbf0 con 0x55b5adb2cdb0 2024-03-28T23:34:07.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.689+0000 7fe1e0ff6740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe1d81f4820 con 0x55b5adb4e1e0 2024-03-28T23:34:07.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.689+0000 7fe1d77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2d190 0x55b5adb4dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.689+0000 7fe1d77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2d190 0x55b5adb4dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55478/0 (socket says 172.21.15.161:55478) 2024-03-28T23:34:07.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.689+0000 7fe1d77fe640 1 -- 172.21.15.161:0/1742850022 learned_addr learned my addr 172.21.15.161:0/1742850022 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:07.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.690+0000 7fe1d7fff640 1 --2- 172.21.15.161:0/1742850022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b5adb2cdb0 0x55b5adb50a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.690+0000 7fe1dc8db640 1 --2- 172.21.15.161:0/1742850022 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b5adb4e1e0 0x55b5adb2a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.690+0000 7fe1d77fe640 1 -- 172.21.15.161:0/1742850022 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b5adb4e1e0 msgr2=0x55b5adb2a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:07.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.690+0000 7fe1d77fe640 1 --2- 172.21.15.161:0/1742850022 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b5adb4e1e0 0x55b5adb2a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.690+0000 7fe1d77fe640 1 -- 172.21.15.161:0/1742850022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b5adb2cdb0 msgr2=0x55b5adb50a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:07.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.690+0000 7fe1d77fe640 1 --2- 172.21.15.161:0/1742850022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b5adb2cdb0 0x55b5adb50a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.690+0000 7fe1d77fe640 1 -- 172.21.15.161:0/1742850022 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5adb28d90 con 0x55b5adb2d190 2024-03-28T23:34:07.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.690+0000 7fe1d77fe640 1 --2- 172.21.15.161:0/1742850022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2d190 0x55b5adb4dca0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8009890 tx=0x7fe1c8030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5fdb3e20e66844 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:07.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.690+0000 7fe1cd7fa640 1 -- 172.21.15.161:0/1742850022 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8035040 con 0x55b5adb2d190 2024-03-28T23:34:07.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.690+0000 7fe1cd7fa640 1 -- 172.21.15.161:0/1742850022 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1c805a070 con 0x55b5adb2d190 2024-03-28T23:34:07.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.690+0000 7fe1cd7fa640 1 -- 172.21.15.161:0/1742850022 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8055550 con 0x55b5adb2d190 2024-03-28T23:34:07.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.690+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/1742850022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2d190 msgr2=0x55b5adb4dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:07.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.690+0000 7fe1e0ff6740 1 --2- 172.21.15.161:0/1742850022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2d190 0x55b5adb4dca0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8009890 tx=0x7fe1c8030000 comp rx=0 tx=0).stop 2024-03-28T23:34:07.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.690+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/1742850022 shutdown_connections 2024-03-28T23:34:07.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.690+0000 7fe1e0ff6740 1 --2- 172.21.15.161:0/1742850022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2d190 0x55b5adb4dca0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.690+0000 7fe1e0ff6740 1 --2- 172.21.15.161:0/1742850022 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b5adb4e1e0 0x55b5adb2a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.690+0000 7fe1e0ff6740 1 --2- 172.21.15.161:0/1742850022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b5adb2cdb0 0x55b5adb50a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.690+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/1742850022 >> 172.21.15.161:0/1742850022 conn(0x55b5adb2b310 msgr2=0x55b5adb28610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:07.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.691+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/1742850022 shutdown_connections 2024-03-28T23:34:07.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.691+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/1742850022 wait complete. 2024-03-28T23:34:07.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.691+0000 7fe1e0ff6740 1 -- start start 2024-03-28T23:34:07.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.691+0000 7fe1e0ff6740 1 Processor -- start 2024-03-28T23:34:07.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.692+0000 7fe1e0ff6740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2cdb0 0x55b5adb2bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.692+0000 7fe1e0ff6740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b5adb2d190 0x55b5adb262b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.692+0000 7fe1e0ff6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b5adb4e1e0 0x55b5adb4a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.692+0000 7fe1e0ff6740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55b5ada4cf70 con 0x55b5adb2cdb0 2024-03-28T23:34:07.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.692+0000 7fe1e0ff6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55b5adb52400 con 0x55b5adb4e1e0 2024-03-28T23:34:07.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.692+0000 7fe1e0ff6740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55b5adb4f090 con 0x55b5adb2d190 2024-03-28T23:34:07.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.692+0000 7fe1e0ff6740 10 client.0 _ll_register_callbacks cb 0x7fe1c006e070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:34:07.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.692+0000 7fe1d7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2cdb0 0x55b5adb2bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.692+0000 7fe1d7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2cdb0 0x55b5adb2bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55490/0 (socket says 172.21.15.161:55490) 2024-03-28T23:34:07.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.692+0000 7fe1d7fff640 1 -- 172.21.15.161:0/309484150 learned_addr learned my addr 172.21.15.161:0/309484150 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:07.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.692+0000 7fe1dc8db640 1 --2- 172.21.15.161:0/309484150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b5adb4e1e0 0x55b5adb4a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.692+0000 7fe1d77fe640 1 --2- 172.21.15.161:0/309484150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b5adb2d190 0x55b5adb262b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.692+0000 7fe1d7fff640 1 -- 172.21.15.161:0/309484150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b5adb2d190 msgr2=0x55b5adb262b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.692+0000 7fe1d7fff640 1 --2- 172.21.15.161:0/309484150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b5adb2d190 0x55b5adb262b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.692+0000 7fe1d7fff640 1 -- 172.21.15.161:0/309484150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b5adb4e1e0 msgr2=0x55b5adb4a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.692+0000 7fe1d7fff640 1 --2- 172.21.15.161:0/309484150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b5adb4e1e0 0x55b5adb4a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.692+0000 7fe1d7fff640 1 -- 172.21.15.161:0/309484150 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5adb267f0 con 0x55b5adb2cdb0 2024-03-28T23:34:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.692+0000 7fe1d7fff640 1 --2- 172.21.15.161:0/309484150 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2cdb0 0x55b5adb2bb40 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fe1c00050a0 tx=0x7fe1c006d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78a1e68037c226a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.692+0000 7fe1b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:34:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.693+0000 7fe1b6ffd640 1 -- 172.21.15.161:0/309484150 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0057220 con 0x55b5adb2cdb0 2024-03-28T23:34:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.693+0000 7fe1b6ffd640 1 -- 172.21.15.161:0/309484150 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7fe1c00573c0 con 0x55b5adb2cdb0 2024-03-28T23:34:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.693+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/309484150 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1c805a070 con 0x55b5adb2cdb0 2024-03-28T23:34:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.693+0000 7fe1b6ffd640 1 -- 172.21.15.161:0/309484150 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7fe1c0068050 con 0x55b5adb2cdb0 2024-03-28T23:34:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.693+0000 7fe1e0ff6740 10 client.7710 fetch_fsmap learned FSMap version 95 2024-03-28T23:34:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.693+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/309484150 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=2+}) -- 0x55b5adb267f0 con 0x55b5adb2cdb0 2024-03-28T23:34:07.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.693+0000 7fe1b6ffd640 1 -- 172.21.15.161:0/309484150 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1c805a070 con 0x55b5adb2cdb0 2024-03-28T23:34:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.693+0000 7fe1b6ffd640 1 -- 172.21.15.161:0/309484150 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 95) ==== 3494+0+0 (secure 0 0 0) 0x7fe1c006b020 con 0x55b5adb2cdb0 2024-03-28T23:34:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.693+0000 7fe1e0ff6740 10 client.7710 fetch_fsmap finished waiting for FSMap version 95 2024-03-28T23:34:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.693+0000 7fe1e0ff6740 10 client.7710 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:34:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.693+0000 7fe1e0ff6740 20 client.7710 populate_metadata read hostname 'smithi161' 2024-03-28T23:34:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.693+0000 7fe1e0ff6740 1 --2- 172.21.15.161:0/309484150 >> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] conn(0x55b5adb4af70 0x55b5adb4b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.693+0000 7fe1e0ff6740 4 client.7710 mds_command: new command op to 6203 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:34:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.693+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/309484150 --> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] -- command(tid 0: {"prefix": "status"}) -- 0x55b5ada98380 con 0x55b5adb4af70 2024-03-28T23:34:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.693+0000 7fe1d77fe640 1 --2- 172.21.15.161:0/309484150 >> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] conn(0x55b5adb4af70 0x55b5adb4b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.693+0000 7fe1b6ffd640 10 client.7710 ms_handle_connect on v2:172.21.15.170:6832/1096504848 2024-03-28T23:34:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.693+0000 7fe1d77fe640 1 --2- 172.21.15.161:0/309484150 >> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] conn(0x55b5adb4af70 0x55b5adb4b350 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87c3236a1989d171 server_cookie=f4737ef4435d9d0f in_seq=0 out_seq=0 2024-03-28T23:34:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.694+0000 7fe1b6ffd640 1 -- 172.21.15.161:0/309484150 <== mds.0 v2:172.21.15.170:6832/1096504848 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55b5ada98380 con 0x55b5adb4af70 2024-03-28T23:34:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.694+0000 7fe1b6ffd640 10 client.7710 handle_command_reply: tid=0 2024-03-28T23:34:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.694+0000 7fe1e0ff6740 1 client.7710 shutdown 2024-03-28T23:34:07.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.694+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/309484150 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2cdb0 msgr2=0x55b5adb2bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:07.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.694+0000 7fe1e0ff6740 1 --2- 172.21.15.161:0/309484150 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2cdb0 0x55b5adb2bb40 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fe1c00050a0 tx=0x7fe1c006d040 comp rx=0 tx=0).stop 2024-03-28T23:34:07.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.694+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/309484150 shutdown_connections 2024-03-28T23:34:07.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.694+0000 7fe1e0ff6740 1 --2- 172.21.15.161:0/309484150 >> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] conn(0x55b5adb4af70 0x55b5adb4b350 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.694+0000 7fe1e0ff6740 1 --2- 172.21.15.161:0/309484150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b5adb4e1e0 0x55b5adb4a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.694+0000 7fe1b6ffd640 0 client.7710 ms_handle_reset on v2:172.21.15.170:6832/1096504848 2024-03-28T23:34:07.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.694+0000 7fe1e0ff6740 1 --2- 172.21.15.161:0/309484150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b5adb2d190 0x55b5adb262b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.694+0000 7fe1e0ff6740 1 --2- 172.21.15.161:0/309484150 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b5adb2cdb0 0x55b5adb2bb40 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.694+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/309484150 >> 172.21.15.161:0/309484150 conn(0x55b5adb2b310 msgr2=0x55b5adb28610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:07.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.694+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/309484150 shutdown_connections 2024-03-28T23:34:07.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.694+0000 7fe1e0ff6740 1 -- 172.21.15.161:0/309484150 wait complete. 2024-03-28T23:34:07.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.694+0000 7fe1e0ff6740 20 client.7710 trim_cache size 0 max 16384 2024-03-28T23:34:07.702 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:34:07.702 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:34:07.702 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:34:07.702 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 6203, 2024-03-28T23:34:07.702 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:34:07.702 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:34:07.702 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:34:07.702 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 5.6584528860000001, 2024-03-28T23:34:07.702 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 95, 2024-03-28T23:34:07.702 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 110, 2024-03-28T23:34:07.702 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 110, 2024-03-28T23:34:07.702 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 21.802489839 2024-03-28T23:34:07.702 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:34:07.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.694+0000 7fe1de33e640 1 -- 172.21.15.161:0/469122879 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe1b806b4d0 msgr2=0x7fe1b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:07.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.694+0000 7fe1de33e640 1 --2- 172.21.15.161:0/469122879 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe1b806b4d0 0x7fe1b806d990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8036000 tx=0x7fe1c803c040 comp rx=0 tx=0).stop 2024-03-28T23:34:07.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.694+0000 7fe1de33e640 1 -- 172.21.15.161:0/469122879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1d806ce80 msgr2=0x7fe1d81de910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:07.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.694+0000 7fe1de33e640 1 --2- 172.21.15.161:0/469122879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1d806ce80 0x7fe1d81de910 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000c410 tx=0x7fe1c000c8d0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.695+0000 7fe1de33e640 1 -- 172.21.15.161:0/469122879 shutdown_connections 2024-03-28T23:34:07.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.695+0000 7fe1de33e640 1 --2- 172.21.15.161:0/469122879 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe1b806b4d0 0x7fe1b806d990 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.695+0000 7fe1de33e640 1 --2- 172.21.15.161:0/469122879 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe1d814be70 0x7fe1d8148910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.695+0000 7fe1de33e640 1 --2- 172.21.15.161:0/469122879 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1d80a9680 0x7fe1d814f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.695+0000 7fe1de33e640 1 --2- 172.21.15.161:0/469122879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1d806ce80 0x7fe1d81de910 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.695+0000 7fe1de33e640 1 -- 172.21.15.161:0/469122879 >> 172.21.15.161:0/469122879 conn(0x7fe1d80c6670 msgr2=0x7fe1d80c2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:07.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.695+0000 7fe1de33e640 1 -- 172.21.15.161:0/469122879 shutdown_connections 2024-03-28T23:34:07.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.695+0000 7fe1de33e640 1 -- 172.21.15.161:0/469122879 wait complete. 2024-03-28T23:34:07.706 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 95, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:33:50.326492+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 95, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6203": { "addr": "172.21.15.170:6834/1096504848", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 1096504848, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 1096504848, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6203, "incarnation": 92, "join_fscid": 8, "name": "b", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 110, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-03-28T23:34:05.043006+0000", "qdb_cluster": [ 6203 ], "qdb_leader": 6203, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6203 } } } ], "standbys": [ { "addr": "172.21.15.170:6835/3351418222", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 3351418222, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 3351418222, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 87, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6206, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6837/2684760895", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6836", "nonce": 2684760895, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 2684760895, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 87, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6892, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6835/2943346687", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 2943346687, "type": "v2" }, { "addr": "172.21.15.161:6835", "nonce": 2943346687, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 93, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7680, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:34:07.707 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 8, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-28T23:34:07.707 INFO:tasks.cephfs.test_failover:current = [{'gid': 7680, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6835/2943346687', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 2943346687}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 2943346687}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 93}, {'gid': 6203, 'name': 'b', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.170:6834/1096504848', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 1096504848}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 1096504848}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 6892, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6837/2684760895', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 2684760895}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 2684760895}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 87}, {'gid': 6206, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/3351418222', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 3351418222}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 3351418222}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 87}] 2024-03-28T23:34:07.707 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-28T23:34:07.707 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-28T23:34:07.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.813+0000 7f2ccf484640 1 Processor -- start 2024-03-28T23:34:07.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.813+0000 7f2ccf484640 1 -- start start 2024-03-28T23:34:07.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.814+0000 7f2ccf484640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2cc80c7330 0x7f2cc814e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.814+0000 7f2ccf484640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cc80733e0 0x7f2cc814e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.814+0000 7f2ccf484640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cc80abc00 con 0x7f2cc80a9960 2024-03-28T23:34:07.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.814+0000 7f2ccf484640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2cc80aa3a0 con 0x7f2cc80c7330 2024-03-28T23:34:07.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.814+0000 7f2ccf484640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2cc806bf70 con 0x7f2cc80733e0 2024-03-28T23:34:07.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.814+0000 7f2ccca1f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2cc80c7330 0x7f2cc814e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.814+0000 7f2ccda21640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cc80733e0 0x7f2cc814e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.814+0000 7f2ccca1f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2cc80c7330 0x7f2cc814e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55498/0 (socket says 172.21.15.161:55498) 2024-03-28T23:34:07.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.814+0000 7f2ccca1f640 1 -- 172.21.15.161:0/1591613155 learned_addr learned my addr 172.21.15.161:0/1591613155 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:07.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.814+0000 7f2ccd220640 1 --1- 172.21.15.161:0/1591613155 >> v1:172.21.15.170:6789/0 conn(0x7f2cc80a9960 0x7f2cc8072970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:49406/0 (socket says 172.21.15.161:49406) 2024-03-28T23:34:07.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.814+0000 7f2ccda21640 1 -- 172.21.15.161:0/1591613155 >> v2:172.21.15.161:3300/0 conn(0x7f2cc80c7330 msgr2=0x7f2cc814e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:07.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.814+0000 7f2ccda21640 1 --2- 172.21.15.161:0/1591613155 >> v2:172.21.15.161:3300/0 conn(0x7f2cc80c7330 0x7f2cc814e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.814+0000 7f2ccda21640 1 -- 172.21.15.161:0/1591613155 >> v1:172.21.15.170:6789/0 conn(0x7f2cc80a9960 legacy=0x7f2cc8072970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:07.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.814+0000 7f2cbffff640 1 -- 172.21.15.161:0/1591613155 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2686378240 0 0) 0x7f2cc80abc00 con 0x7f2cc80a9960 2024-03-28T23:34:07.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.814+0000 7f2ccda21640 1 -- 172.21.15.161:0/1591613155 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc806bcd0 con 0x7f2cc80733e0 2024-03-28T23:34:07.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.814+0000 7f2ccca1f640 1 --2- 172.21.15.161:0/1591613155 >> v2:172.21.15.161:3300/0 conn(0x7f2cc80c7330 0x7f2cc814e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:34:07.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.815+0000 7f2ccda21640 1 --2- 172.21.15.161:0/1591613155 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cc80733e0 0x7f2cc814e9a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0009d90 tx=0x7f2cc002bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a20fee247f7a377a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:07.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.815+0000 7f2cbffff640 1 -- 172.21.15.161:0/1591613155 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0039070 con 0x7f2cc80733e0 2024-03-28T23:34:07.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.815+0000 7f2cbffff640 1 -- 172.21.15.161:0/1591613155 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cc002cb40 con 0x7f2cc80733e0 2024-03-28T23:34:07.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.815+0000 7f2cbffff640 1 -- 172.21.15.161:0/1591613155 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0034440 con 0x7f2cc80733e0 2024-03-28T23:34:07.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.816+0000 7f2ccf484640 1 -- 172.21.15.161:0/1591613155 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cc80733e0 msgr2=0x7f2cc814e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:07.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.816+0000 7f2ccf484640 1 --2- 172.21.15.161:0/1591613155 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cc80733e0 0x7f2cc814e9a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0009d90 tx=0x7f2cc002bd90 comp rx=0 tx=0).stop 2024-03-28T23:34:07.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.816+0000 7f2ccf484640 1 -- 172.21.15.161:0/1591613155 shutdown_connections 2024-03-28T23:34:07.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.816+0000 7f2ccf484640 1 --2- 172.21.15.161:0/1591613155 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cc80733e0 0x7f2cc814e9a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.816+0000 7f2ccf484640 1 --2- 172.21.15.161:0/1591613155 >> v2:172.21.15.161:3300/0 conn(0x7f2cc80c7330 0x7f2cc814e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.816+0000 7f2ccf484640 1 -- 172.21.15.161:0/1591613155 >> 172.21.15.161:0/1591613155 conn(0x7f2cc80a83f0 msgr2=0x7f2cc80a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:07.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.816+0000 7f2ccf484640 1 -- 172.21.15.161:0/1591613155 shutdown_connections 2024-03-28T23:34:07.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.816+0000 7f2ccf484640 1 -- 172.21.15.161:0/1591613155 wait complete. 2024-03-28T23:34:07.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.817+0000 7f2ccf484640 1 Processor -- start 2024-03-28T23:34:07.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.817+0000 7f2ccf484640 1 -- start start 2024-03-28T23:34:07.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.817+0000 7f2ccf484640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2cc80733e0 0x7f2cc81ea8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.817+0000 7f2ccf484640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cc80a9960 0x7f2cc80c6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.817+0000 7f2ccf484640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cc80c7330 0x7f2cc814b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.817+0000 7f2ccf484640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc8073080 con 0x7f2cc80a9960 2024-03-28T23:34:07.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.817+0000 7f2ccf484640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc8152250 con 0x7f2cc80733e0 2024-03-28T23:34:07.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.817+0000 7f2ccf484640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2cc8073200 con 0x7f2cc80c7330 2024-03-28T23:34:07.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.818+0000 7f2ccca1f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cc80a9960 0x7f2cc80c6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.818+0000 7f2ccca1f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cc80a9960 0x7f2cc80c6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55514/0 (socket says 172.21.15.161:55514) 2024-03-28T23:34:07.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.818+0000 7f2ccd220640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2cc80733e0 0x7f2cc81ea8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.818+0000 7f2ccca1f640 1 -- 172.21.15.161:0/558338084 learned_addr learned my addr 172.21.15.161:0/558338084 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:07.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.818+0000 7f2ccda21640 1 --2- 172.21.15.161:0/558338084 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cc80c7330 0x7f2cc814b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.818+0000 7f2ccca1f640 1 -- 172.21.15.161:0/558338084 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cc80c7330 msgr2=0x7f2cc814b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:07.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.818+0000 7f2ccca1f640 1 --2- 172.21.15.161:0/558338084 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cc80c7330 0x7f2cc814b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.818+0000 7f2ccca1f640 1 -- 172.21.15.161:0/558338084 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2cc80733e0 msgr2=0x7f2cc81ea8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:07.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.818+0000 7f2ccca1f640 1 --2- 172.21.15.161:0/558338084 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2cc80733e0 0x7f2cc81ea8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:07.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.818+0000 7f2ccca1f640 1 -- 172.21.15.161:0/558338084 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc81547f0 con 0x7f2cc80a9960 2024-03-28T23:34:07.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.818+0000 7f2ccd220640 1 --2- 172.21.15.161:0/558338084 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2cc80733e0 0x7f2cc81ea8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:34:07.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.818+0000 7f2ccca1f640 1 --2- 172.21.15.161:0/558338084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cc80a9960 0x7f2cc80c6510 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8010da0 tx=0x7f2cb800e730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:07.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.818+0000 7f2cbdffb640 1 -- 172.21.15.161:0/558338084 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb801e070 con 0x7f2cc80a9960 2024-03-28T23:34:07.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.818+0000 7f2cbdffb640 1 -- 172.21.15.161:0/558338084 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cb8011860 con 0x7f2cc80a9960 2024-03-28T23:34:07.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.818+0000 7f2cbdffb640 1 -- 172.21.15.161:0/558338084 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb8011c10 con 0x7f2cc80a9960 2024-03-28T23:34:07.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.818+0000 7f2ccf484640 1 -- 172.21.15.161:0/558338084 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cc8154ad0 con 0x7f2cc80a9960 2024-03-28T23:34:07.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.818+0000 7f2ccf484640 1 -- 172.21.15.161:0/558338084 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cc80c6a50 con 0x7f2cc80a9960 2024-03-28T23:34:07.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.820+0000 7f2ccf484640 1 -- 172.21.15.161:0/558338084 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cc80ac9e0 con 0x7f2cc80a9960 2024-03-28T23:34:07.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.820+0000 7f2cbdffb640 1 -- 172.21.15.161:0/558338084 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2cb8021070 con 0x7f2cc80a9960 2024-03-28T23:34:07.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.820+0000 7f2cbdffb640 1 --2- 172.21.15.161:0/558338084 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2cac06b4d0 0x7f2cac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:07.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.820+0000 7f2cbdffb640 1 -- 172.21.15.161:0/558338084 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7f2cb808ccb0 con 0x7f2cc80a9960 2024-03-28T23:34:07.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.820+0000 7f2ccd220640 1 --2- 172.21.15.161:0/558338084 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2cac06b4d0 0x7f2cac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:07.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.821+0000 7f2ccd220640 1 --2- 172.21.15.161:0/558338084 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2cac06b4d0 0x7f2cac06d990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4000f60 tx=0x7f2cc4007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:07.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:07.825+0000 7f2cbdffb640 1 -- 172.21.15.161:0/558338084 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2cc80ac9e0 con 0x7f2cc80a9960 2024-03-28T23:34:08.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.033+0000 7f2ccf484640 1 -- 172.21.15.161:0/558338084 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f2cc804f4b0 con 0x7f2cc80a9960 2024-03-28T23:34:08.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.053+0000 7f2cbdffb640 1 -- 172.21.15.161:0/558338084 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v96) ==== 121+0+0 (secure 0 0 0) 0x7f2cb805b390 con 0x7f2cc80a9960 2024-03-28T23:34:08.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.056+0000 7f2ccf484640 1 -- 172.21.15.161:0/558338084 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2cac06b4d0 msgr2=0x7f2cac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:08.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.056+0000 7f2ccf484640 1 --2- 172.21.15.161:0/558338084 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2cac06b4d0 0x7f2cac06d990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4000f60 tx=0x7f2cc4007d50 comp rx=0 tx=0).stop 2024-03-28T23:34:08.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.056+0000 7f2ccf484640 1 -- 172.21.15.161:0/558338084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cc80a9960 msgr2=0x7f2cc80c6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:08.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.056+0000 7f2ccf484640 1 --2- 172.21.15.161:0/558338084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cc80a9960 0x7f2cc80c6510 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8010da0 tx=0x7f2cb800e730 comp rx=0 tx=0).stop 2024-03-28T23:34:08.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.056+0000 7f2ccf484640 1 -- 172.21.15.161:0/558338084 shutdown_connections 2024-03-28T23:34:08.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.056+0000 7f2ccf484640 1 --2- 172.21.15.161:0/558338084 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2cac06b4d0 0x7f2cac06d990 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.056+0000 7f2ccf484640 1 --2- 172.21.15.161:0/558338084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cc80a9960 0x7f2cc80c6510 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.056+0000 7f2ccf484640 1 --2- 172.21.15.161:0/558338084 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cc80c7330 0x7f2cc814b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.056+0000 7f2ccf484640 1 --2- 172.21.15.161:0/558338084 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2cc80733e0 0x7f2cc81ea8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.056+0000 7f2ccf484640 1 -- 172.21.15.161:0/558338084 >> 172.21.15.161:0/558338084 conn(0x7f2cc80a83f0 msgr2=0x7f2cc80c17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:08.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.056+0000 7f2ccf484640 1 -- 172.21.15.161:0/558338084 shutdown_connections 2024-03-28T23:34:08.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.056+0000 7f2ccf484640 1 -- 172.21.15.161:0/558338084 wait complete. 2024-03-28T23:34:08.068 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:34:08.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.168+0000 7f88dd3e9640 1 Processor -- start 2024-03-28T23:34:08.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.168+0000 7f88dd3e9640 1 -- start start 2024-03-28T23:34:08.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.169+0000 7f88dd3e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88d8152a80 0x7f88d8155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:08.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.169+0000 7f88dd3e9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88d806c3c0 con 0x7f88d806cb80 2024-03-28T23:34:08.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.169+0000 7f88dd3e9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88d80c5b90 con 0x7f88d806d9f0 2024-03-28T23:34:08.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.169+0000 7f88dd3e9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f88d8156250 con 0x7f88d8152a80 2024-03-28T23:34:08.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.169+0000 7f88d77fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f88d806cb80 0x7f88d80c74c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59824/0 (socket says 172.21.15.161:59824) 2024-03-28T23:34:08.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.169+0000 7f88d77fe640 1 -- 172.21.15.161:0/1922543455 learned_addr learned my addr 172.21.15.161:0/1922543455 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:08.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.169+0000 7f88d6ffd640 1 --2- 172.21.15.161:0/1922543455 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88d8152a80 0x7f88d8155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:08.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.170+0000 7f88d67fc640 1 -- 172.21.15.161:0/1922543455 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4109104734 0 0) 0x7f88d806c3c0 con 0x7f88d806cb80 2024-03-28T23:34:08.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.170+0000 7f88d67fc640 1 -- 172.21.15.161:0/1922543455 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88c4003520 con 0x7f88d806cb80 2024-03-28T23:34:08.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.170+0000 7f88d67fc640 1 -- 172.21.15.161:0/1922543455 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3751260985 0 0) 0x7f88d80c5b90 con 0x7f88d806d9f0 2024-03-28T23:34:08.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.170+0000 7f88d67fc640 1 -- 172.21.15.161:0/1922543455 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88d806c3c0 con 0x7f88d806d9f0 2024-03-28T23:34:08.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.170+0000 7f88d6ffd640 1 -- 172.21.15.161:0/1922543455 >> v1:172.21.15.170:6789/0 conn(0x7f88d806d9f0 legacy=0x7f88d80c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:08.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.170+0000 7f88d6ffd640 1 -- 172.21.15.161:0/1922543455 >> v1:172.21.15.161:6789/0 conn(0x7f88d806cb80 legacy=0x7f88d80c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:08.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.170+0000 7f88d6ffd640 1 -- 172.21.15.161:0/1922543455 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d80c55d0 con 0x7f88d8152a80 2024-03-28T23:34:08.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.171+0000 7f88d6ffd640 1 --2- 172.21.15.161:0/1922543455 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88d8152a80 0x7f88d8155d10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f88c80097d0 tx=0x7f88c802b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b295ba9a3dc7c489 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:08.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.171+0000 7f88d67fc640 1 -- 172.21.15.161:0/1922543455 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c8031770 con 0x7f88d8152a80 2024-03-28T23:34:08.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.171+0000 7f88d67fc640 1 -- 172.21.15.161:0/1922543455 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88c8037070 con 0x7f88d8152a80 2024-03-28T23:34:08.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.171+0000 7f88d67fc640 1 -- 172.21.15.161:0/1922543455 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c8031b30 con 0x7f88d8152a80 2024-03-28T23:34:08.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.171+0000 7f88dd3e9640 1 -- 172.21.15.161:0/1922543455 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88d8152a80 msgr2=0x7f88d8155d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:08.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.171+0000 7f88dd3e9640 1 --2- 172.21.15.161:0/1922543455 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88d8152a80 0x7f88d8155d10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f88c80097d0 tx=0x7f88c802b810 comp rx=0 tx=0).stop 2024-03-28T23:34:08.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.171+0000 7f88dd3e9640 1 -- 172.21.15.161:0/1922543455 shutdown_connections 2024-03-28T23:34:08.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.171+0000 7f88dd3e9640 1 --2- 172.21.15.161:0/1922543455 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88d8152a80 0x7f88d8155d10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.171+0000 7f88dd3e9640 1 -- 172.21.15.161:0/1922543455 >> 172.21.15.161:0/1922543455 conn(0x7f88d80ac720 msgr2=0x7f88d80acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:08.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.171+0000 7f88dd3e9640 1 -- 172.21.15.161:0/1922543455 shutdown_connections 2024-03-28T23:34:08.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.171+0000 7f88dd3e9640 1 -- 172.21.15.161:0/1922543455 wait complete. 2024-03-28T23:34:08.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.172+0000 7f88dd3e9640 1 Processor -- start 2024-03-28T23:34:08.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.172+0000 7f88dd3e9640 1 -- start start 2024-03-28T23:34:08.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.173+0000 7f88dd3e9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88d806cb80 0x7f88d8154e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:08.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.173+0000 7f88dd3e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88d806d9f0 0x7f88d8155380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:08.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.173+0000 7f88dd3e9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88d8152a80 0x7f88d8145280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:08.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.173+0000 7f88dd3e9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f88d80c7bd0 con 0x7f88d8152a80 2024-03-28T23:34:08.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.173+0000 7f88dd3e9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f88d806c6f0 con 0x7f88d806cb80 2024-03-28T23:34:08.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.173+0000 7f88dd3e9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f88d80c7d50 con 0x7f88d806d9f0 2024-03-28T23:34:08.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.173+0000 7f88d77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88d8152a80 0x7f88d8145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:08.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.173+0000 7f88d6ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88d806cb80 0x7f88d8154e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:08.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.173+0000 7f88d77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88d8152a80 0x7f88d8145280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55518/0 (socket says 172.21.15.161:55518) 2024-03-28T23:34:08.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.173+0000 7f88d77fe640 1 -- 172.21.15.161:0/2514172619 learned_addr learned my addr 172.21.15.161:0/2514172619 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:08.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.173+0000 7f88cffff640 1 --2- 172.21.15.161:0/2514172619 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88d806d9f0 0x7f88d8155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:08.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.173+0000 7f88d77fe640 1 -- 172.21.15.161:0/2514172619 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88d806d9f0 msgr2=0x7f88d8155380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:08.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.173+0000 7f88d77fe640 1 --2- 172.21.15.161:0/2514172619 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88d806d9f0 0x7f88d8155380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.173+0000 7f88d77fe640 1 -- 172.21.15.161:0/2514172619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88d806cb80 msgr2=0x7f88d8154e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:08.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.173+0000 7f88d77fe640 1 --2- 172.21.15.161:0/2514172619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88d806cb80 0x7f88d8154e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.173+0000 7f88d77fe640 1 -- 172.21.15.161:0/2514172619 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d81eb7f0 con 0x7f88d8152a80 2024-03-28T23:34:08.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.174+0000 7f88d77fe640 1 --2- 172.21.15.161:0/2514172619 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88d8152a80 0x7f88d8145280 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f88d000cba0 tx=0x7f88d000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:08.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.174+0000 7f88cf7fe640 1 -- 172.21.15.161:0/2514172619 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d0017070 con 0x7f88d8152a80 2024-03-28T23:34:08.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.174+0000 7f88cf7fe640 1 -- 172.21.15.161:0/2514172619 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88d000ace0 con 0x7f88d8152a80 2024-03-28T23:34:08.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.174+0000 7f88dd3e9640 1 -- 172.21.15.161:0/2514172619 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88d81558c0 con 0x7f88d8152a80 2024-03-28T23:34:08.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.174+0000 7f88cf7fe640 1 -- 172.21.15.161:0/2514172619 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d0012620 con 0x7f88d8152a80 2024-03-28T23:34:08.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.174+0000 7f88dd3e9640 1 -- 172.21.15.161:0/2514172619 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88d81533e0 con 0x7f88d8152a80 2024-03-28T23:34:08.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.175+0000 7f88dd3e9640 1 -- 172.21.15.161:0/2514172619 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88d804f4b0 con 0x7f88d8152a80 2024-03-28T23:34:08.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.175+0000 7f88cf7fe640 1 -- 172.21.15.161:0/2514172619 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f88d001a070 con 0x7f88d8152a80 2024-03-28T23:34:08.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.176+0000 7f88cf7fe640 1 --2- 172.21.15.161:0/2514172619 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f88b806b4d0 0x7f88b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:08.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.176+0000 7f88cf7fe640 1 -- 172.21.15.161:0/2514172619 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7f88d008a0c0 con 0x7f88d8152a80 2024-03-28T23:34:08.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.176+0000 7f88d6ffd640 1 --2- 172.21.15.161:0/2514172619 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f88b806b4d0 0x7f88b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:08.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.176+0000 7f88d6ffd640 1 --2- 172.21.15.161:0/2514172619 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f88b806b4d0 0x7f88b806d990 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f88c8036000 tx=0x7f88c803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:08.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.180+0000 7f88cf7fe640 1 -- 172.21.15.161:0/2514172619 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f88d81eb7f0 con 0x7f88d8152a80 2024-03-28T23:34:08.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.385+0000 7f88dd3e9640 1 -- 172.21.15.161:0/2514172619 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88d806d7d0 con 0x7f88d8152a80 2024-03-28T23:34:08.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.386+0000 7f88cf7fe640 1 -- 172.21.15.161:0/2514172619 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+5203 (secure 0 0 0) 0x7f88d00587a0 con 0x7f88d8152a80 2024-03-28T23:34:08.387 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:08.387 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":97,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6206,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3351418222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3351418222},{"type":"v1","addr":"172.21.15.170:6835","nonce":3351418222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":6892,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/2684760895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2684760895},{"type":"v1","addr":"172.21.15.161:6837","nonce":2684760895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":97,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T23:33:50.326492+0000","modified":"2024-03-28T23:34:08.057065+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6203":{"gid":6203,"name":"b","rank":0,"incarnation":92,"state":"up:active","state_seq":8,"addr":"172.21.15.170:6834/1096504848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1096504848},{"type":"v1","addr":"172.21.15.170:6834","nonce":1096504848}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7680":{"gid":7680,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/2943346687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":2943346687},{"type":"v1","addr":"172.21.15.161:6835","nonce":2943346687}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6203,"qdb_cluster":[6203]},"id":8}]} 2024-03-28T23:34:08.387 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 97 2024-03-28T23:34:08.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.388+0000 7f88dd3e9640 1 -- 172.21.15.161:0/2514172619 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f88b806b4d0 msgr2=0x7f88b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:08.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.388+0000 7f88dd3e9640 1 --2- 172.21.15.161:0/2514172619 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f88b806b4d0 0x7f88b806d990 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f88c8036000 tx=0x7f88c803d040 comp rx=0 tx=0).stop 2024-03-28T23:34:08.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.388+0000 7f88dd3e9640 1 -- 172.21.15.161:0/2514172619 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88d8152a80 msgr2=0x7f88d8145280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:08.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.388+0000 7f88dd3e9640 1 --2- 172.21.15.161:0/2514172619 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88d8152a80 0x7f88d8145280 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f88d000cba0 tx=0x7f88d000a430 comp rx=0 tx=0).stop 2024-03-28T23:34:08.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.388+0000 7f88dd3e9640 1 -- 172.21.15.161:0/2514172619 shutdown_connections 2024-03-28T23:34:08.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.388+0000 7f88dd3e9640 1 --2- 172.21.15.161:0/2514172619 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f88b806b4d0 0x7f88b806d990 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.388+0000 7f88dd3e9640 1 --2- 172.21.15.161:0/2514172619 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88d8152a80 0x7f88d8145280 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.388+0000 7f88dd3e9640 1 --2- 172.21.15.161:0/2514172619 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88d806d9f0 0x7f88d8155380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.388+0000 7f88dd3e9640 1 --2- 172.21.15.161:0/2514172619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88d806cb80 0x7f88d8154e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.388+0000 7f88dd3e9640 1 -- 172.21.15.161:0/2514172619 >> 172.21.15.161:0/2514172619 conn(0x7f88d80ac720 msgr2=0x7f88d814fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:08.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.388+0000 7f88dd3e9640 1 -- 172.21.15.161:0/2514172619 shutdown_connections 2024-03-28T23:34:08.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.388+0000 7f88dd3e9640 1 -- 172.21.15.161:0/2514172619 wait complete. 2024-03-28T23:34:08.399 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs 2024-03-28T23:34:08.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.503+0000 7f7f0998a640 1 Processor -- start 2024-03-28T23:34:08.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.503+0000 7f7f0998a640 1 -- start start 2024-03-28T23:34:08.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.503+0000 7f7f0998a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7f04156520 0x7f7f040ab730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:08.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.503+0000 7f7f0998a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f040c3560 con 0x7f7f040c27a0 2024-03-28T23:34:08.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.503+0000 7f7f0998a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f040abc70 con 0x7f7f040c3c90 2024-03-28T23:34:08.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.503+0000 7f7f0998a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f7f04157180 con 0x7f7f04156520 2024-03-28T23:34:08.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.503+0000 7f7f02ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7f04156520 0x7f7f040ab730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:08.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.503+0000 7f7f027fc640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f7f040c27a0 0x7f7f04156900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:59834/0 (socket says 172.21.15.161:59834) 2024-03-28T23:34:08.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.503+0000 7f7f027fc640 1 -- 172.21.15.161:0/3194899193 learned_addr learned my addr 172.21.15.161:0/3194899193 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:08.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.504+0000 7f7f02ffd640 1 -- 172.21.15.161:0/3194899193 >> v1:172.21.15.170:6789/0 conn(0x7f7f040c3c90 legacy=0x7f7f040c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:08.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.504+0000 7f7f02ffd640 1 -- 172.21.15.161:0/3194899193 >> v1:172.21.15.161:6789/0 conn(0x7f7f040c27a0 legacy=0x7f7f04156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:08.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.504+0000 7f7f01ffb640 1 -- 172.21.15.161:0/3194899193 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 187364050 0 0) 0x7f7f040abc70 con 0x7f7f040c3c90 2024-03-28T23:34:08.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.504+0000 7f7f02ffd640 1 -- 172.21.15.161:0/3194899193 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f040c2e70 con 0x7f7f04156520 2024-03-28T23:34:08.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.505+0000 7f7f02ffd640 1 --2- 172.21.15.161:0/3194899193 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f04156520 0x7f7f040ab730 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f7efc009d90 tx=0x7f7efc02bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b998945cba24743 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:08.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.505+0000 7f7f01ffb640 1 -- 172.21.15.161:0/3194899193 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7efc039070 con 0x7f7f04156520 2024-03-28T23:34:08.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.505+0000 7f7f01ffb640 1 -- 172.21.15.161:0/3194899193 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7efc02cab0 con 0x7f7f04156520 2024-03-28T23:34:08.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.505+0000 7f7f01ffb640 1 -- 172.21.15.161:0/3194899193 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7efc034440 con 0x7f7f04156520 2024-03-28T23:34:08.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.505+0000 7f7f0998a640 1 -- 172.21.15.161:0/3194899193 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f04156520 msgr2=0x7f7f040ab730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:08.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.505+0000 7f7f0998a640 1 --2- 172.21.15.161:0/3194899193 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f04156520 0x7f7f040ab730 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f7efc009d90 tx=0x7f7efc02bd80 comp rx=0 tx=0).stop 2024-03-28T23:34:08.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.505+0000 7f7f0998a640 1 -- 172.21.15.161:0/3194899193 shutdown_connections 2024-03-28T23:34:08.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.505+0000 7f7f0998a640 1 --2- 172.21.15.161:0/3194899193 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f04156520 0x7f7f040ab730 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.505+0000 7f7f0998a640 1 -- 172.21.15.161:0/3194899193 >> 172.21.15.161:0/3194899193 conn(0x7f7f040c4250 msgr2=0x7f7f040c4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:08.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.505+0000 7f7f0998a640 1 -- 172.21.15.161:0/3194899193 shutdown_connections 2024-03-28T23:34:08.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.505+0000 7f7f0998a640 1 -- 172.21.15.161:0/3194899193 wait complete. 2024-03-28T23:34:08.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.506+0000 7f7f0998a640 1 Processor -- start 2024-03-28T23:34:08.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.507+0000 7f7f0998a640 1 -- start start 2024-03-28T23:34:08.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.507+0000 7f7f0998a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f040c27a0 0x7f7f04142280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:08.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.507+0000 7f7f0998a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f040c3c90 0x7f7f041e9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:08.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.507+0000 7f7f0998a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f04156520 0x7f7f041e9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:08.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.507+0000 7f7f0998a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0415e8c0 con 0x7f7f040c3c90 2024-03-28T23:34:08.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.507+0000 7f7f0998a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0414d6a0 con 0x7f7f04156520 2024-03-28T23:34:08.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.507+0000 7f7f0998a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7f0415ea40 con 0x7f7f040c27a0 2024-03-28T23:34:08.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.507+0000 7f7f027fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f040c3c90 0x7f7f041e9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:08.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.507+0000 7f7f027fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f040c3c90 0x7f7f041e9520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55532/0 (socket says 172.21.15.161:55532) 2024-03-28T23:34:08.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.507+0000 7f7f027fc640 1 -- 172.21.15.161:0/409048335 learned_addr learned my addr 172.21.15.161:0/409048335 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:08.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.507+0000 7f7f02ffd640 1 --2- 172.21.15.161:0/409048335 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f040c27a0 0x7f7f04142280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:08.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.507+0000 7f7f037fe640 1 --2- 172.21.15.161:0/409048335 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f04156520 0x7f7f041e9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:08.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.508+0000 7f7f027fc640 1 -- 172.21.15.161:0/409048335 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f040c27a0 msgr2=0x7f7f04142280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:08.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.508+0000 7f7f027fc640 1 --2- 172.21.15.161:0/409048335 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f040c27a0 0x7f7f04142280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.508+0000 7f7f027fc640 1 -- 172.21.15.161:0/409048335 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f04156520 msgr2=0x7f7f041e9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:08.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.508+0000 7f7f027fc640 1 --2- 172.21.15.161:0/409048335 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f04156520 0x7f7f041e9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.508+0000 7f7f027fc640 1 -- 172.21.15.161:0/409048335 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f04151ac0 con 0x7f7f040c3c90 2024-03-28T23:34:08.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.508+0000 7f7f027fc640 1 --2- 172.21.15.161:0/409048335 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f040c3c90 0x7f7f041e9520 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f7ef800c570 tx=0x7f7ef800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:08.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.508+0000 7f7eebfff640 1 -- 172.21.15.161:0/409048335 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8018070 con 0x7f7f040c3c90 2024-03-28T23:34:08.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.508+0000 7f7eebfff640 1 -- 172.21.15.161:0/409048335 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ef80074e0 con 0x7f7f040c3c90 2024-03-28T23:34:08.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.508+0000 7f7eebfff640 1 -- 172.21.15.161:0/409048335 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8007690 con 0x7f7f040c3c90 2024-03-28T23:34:08.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.508+0000 7f7f0998a640 1 -- 172.21.15.161:0/409048335 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f04151d30 con 0x7f7f040c3c90 2024-03-28T23:34:08.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.508+0000 7f7f0998a640 1 -- 172.21.15.161:0/409048335 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f041522f0 con 0x7f7f040c3c90 2024-03-28T23:34:08.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.509+0000 7f7f0998a640 1 -- 172.21.15.161:0/409048335 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f040adaf0 con 0x7f7f040c3c90 2024-03-28T23:34:08.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.510+0000 7f7eebfff640 1 -- 172.21.15.161:0/409048335 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7ef801b070 con 0x7f7f040c3c90 2024-03-28T23:34:08.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.510+0000 7f7eebfff640 1 --2- 172.21.15.161:0/409048335 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7ef006b4d0 0x7f7ef006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:08.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.510+0000 7f7eebfff640 1 -- 172.21.15.161:0/409048335 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7f7ef8089750 con 0x7f7f040c3c90 2024-03-28T23:34:08.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.510+0000 7f7f02ffd640 1 --2- 172.21.15.161:0/409048335 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7ef006b4d0 0x7f7ef006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:08.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.511+0000 7f7f02ffd640 1 --2- 172.21.15.161:0/409048335 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7ef006b4d0 0x7f7ef006d990 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7efc036000 tx=0x7f7efc030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:08.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.515+0000 7f7eebfff640 1 -- 172.21.15.161:0/409048335 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7f040adaf0 con 0x7f7f040c3c90 2024-03-28T23:34:08.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.686+0000 7f7f0998a640 1 -- 172.21.15.161:0/409048335 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) -- 0x7f7f0404f4b0 con 0x7f7f040c3c90 2024-03-28T23:34:08.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.688+0000 7f7eebfff640 1 -- 172.21.15.161:0/409048335 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v37)=0 v37) ==== 116+0+0 (secure 0 0 0) 0x7f7ef8057da0 con 0x7f7f040c3c90 2024-03-28T23:34:08.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.690+0000 7f7f0998a640 1 -- 172.21.15.161:0/409048335 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7ef006b4d0 msgr2=0x7f7ef006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:08.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.690+0000 7f7f0998a640 1 --2- 172.21.15.161:0/409048335 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7ef006b4d0 0x7f7ef006d990 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7efc036000 tx=0x7f7efc030000 comp rx=0 tx=0).stop 2024-03-28T23:34:08.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.691+0000 7f7f0998a640 1 -- 172.21.15.161:0/409048335 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f040c3c90 msgr2=0x7f7f041e9520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:08.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.691+0000 7f7f0998a640 1 --2- 172.21.15.161:0/409048335 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f040c3c90 0x7f7f041e9520 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f7ef800c570 tx=0x7f7ef800ca30 comp rx=0 tx=0).stop 2024-03-28T23:34:08.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.691+0000 7f7f0998a640 1 -- 172.21.15.161:0/409048335 shutdown_connections 2024-03-28T23:34:08.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.691+0000 7f7f0998a640 1 --2- 172.21.15.161:0/409048335 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7ef006b4d0 0x7f7ef006d990 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.691+0000 7f7f0998a640 1 --2- 172.21.15.161:0/409048335 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f040c3c90 0x7f7f041e9520 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.691+0000 7f7f0998a640 1 --2- 172.21.15.161:0/409048335 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f04156520 0x7f7f041e9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.691+0000 7f7f0998a640 1 --2- 172.21.15.161:0/409048335 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f040c27a0 0x7f7f04142280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.691+0000 7f7f0998a640 1 -- 172.21.15.161:0/409048335 >> 172.21.15.161:0/409048335 conn(0x7f7f040c4250 msgr2=0x7f7f040aa5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:08.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.691+0000 7f7f0998a640 1 -- 172.21.15.161:0/409048335 shutdown_connections 2024-03-28T23:34:08.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.691+0000 7f7f0998a640 1 -- 172.21.15.161:0/409048335 wait complete. 2024-03-28T23:34:08.703 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:34:08.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.802+0000 7ff67ac2e640 1 Processor -- start 2024-03-28T23:34:08.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.802+0000 7ff67ac2e640 1 -- start start 2024-03-28T23:34:08.805 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.803+0000 7ff67ac2e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff67414eca0 0x7ff6740c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:08.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.803+0000 7ff67ac2e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff6740c1dc0 0x7ff67406ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:08.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.803+0000 7ff67ac2e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff67406cf60 0x7ff6740a98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:08.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.803+0000 7ff67ac2e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff674152460 con 0x7ff67406cf60 2024-03-28T23:34:08.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.803+0000 7ff67ac2e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7ff674151850 con 0x7ff6740c1dc0 2024-03-28T23:34:08.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.803+0000 7ff67ac2e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff674151b30 con 0x7ff67414eca0 2024-03-28T23:34:08.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.803+0000 7ff6791cb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff67406cf60 0x7ff6740a98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:08.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.803+0000 7ff6791cb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff67406cf60 0x7ff6740a98c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55538/0 (socket says 172.21.15.161:55538) 2024-03-28T23:34:08.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.803+0000 7ff6791cb640 1 -- 172.21.15.161:0/2766973188 learned_addr learned my addr 172.21.15.161:0/2766973188 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:08.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.803+0000 7ff673fff640 1 --2- 172.21.15.161:0/2766973188 >> v2:172.21.15.170:3300/0 conn(0x7ff6740c1dc0 0x7ff67406ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:08.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.803+0000 7ff6789ca640 1 --2- 172.21.15.161:0/2766973188 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff67414eca0 0x7ff6740c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:08.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.804+0000 7ff6791cb640 1 -- 172.21.15.161:0/2766973188 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff67414eca0 msgr2=0x7ff6740c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:08.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.804+0000 7ff6791cb640 1 --2- 172.21.15.161:0/2766973188 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff67414eca0 0x7ff6740c1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.804+0000 7ff6791cb640 1 -- 172.21.15.161:0/2766973188 >> v2:172.21.15.170:3300/0 conn(0x7ff6740c1dc0 msgr2=0x7ff67406ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:08.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.804+0000 7ff6791cb640 1 --2- 172.21.15.161:0/2766973188 >> v2:172.21.15.170:3300/0 conn(0x7ff6740c1dc0 0x7ff67406ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.804+0000 7ff6791cb640 1 -- 172.21.15.161:0/2766973188 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff674151e10 con 0x7ff67406cf60 2024-03-28T23:34:08.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.804+0000 7ff673fff640 1 --2- 172.21.15.161:0/2766973188 >> v2:172.21.15.170:3300/0 conn(0x7ff6740c1dc0 0x7ff67406ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:34:08.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.804+0000 7ff6791cb640 1 --2- 172.21.15.161:0/2766973188 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff67406cf60 0x7ff6740a98c0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7ff6740c7690 tx=0x7ff66c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7088f0659a9b2984 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:08.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.804+0000 7ff6737fe640 1 -- 172.21.15.161:0/2766973188 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66c02c9e0 con 0x7ff67406cf60 2024-03-28T23:34:08.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.804+0000 7ff6737fe640 1 -- 172.21.15.161:0/2766973188 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff66c02cb80 con 0x7ff67406cf60 2024-03-28T23:34:08.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.804+0000 7ff6737fe640 1 -- 172.21.15.161:0/2766973188 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66c0343f0 con 0x7ff67406cf60 2024-03-28T23:34:08.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.805+0000 7ff67ac2e640 1 -- 172.21.15.161:0/2766973188 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff67406cf60 msgr2=0x7ff6740a98c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:08.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.805+0000 7ff67ac2e640 1 --2- 172.21.15.161:0/2766973188 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff67406cf60 0x7ff6740a98c0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7ff6740c7690 tx=0x7ff66c02be40 comp rx=0 tx=0).stop 2024-03-28T23:34:08.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.805+0000 7ff67ac2e640 1 -- 172.21.15.161:0/2766973188 shutdown_connections 2024-03-28T23:34:08.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.805+0000 7ff67ac2e640 1 --2- 172.21.15.161:0/2766973188 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff67406cf60 0x7ff6740a98c0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.805+0000 7ff67ac2e640 1 --2- 172.21.15.161:0/2766973188 >> v2:172.21.15.170:3300/0 conn(0x7ff6740c1dc0 0x7ff67406ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.805+0000 7ff67ac2e640 1 --2- 172.21.15.161:0/2766973188 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff67414eca0 0x7ff6740c1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.805+0000 7ff67ac2e640 1 -- 172.21.15.161:0/2766973188 >> 172.21.15.161:0/2766973188 conn(0x7ff6740c6670 msgr2=0x7ff6740c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:08.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.805+0000 7ff67ac2e640 1 -- 172.21.15.161:0/2766973188 shutdown_connections 2024-03-28T23:34:08.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.805+0000 7ff67ac2e640 1 -- 172.21.15.161:0/2766973188 wait complete. 2024-03-28T23:34:08.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.806+0000 7ff67ac2e640 1 Processor -- start 2024-03-28T23:34:08.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.806+0000 7ff67ac2e640 1 -- start start 2024-03-28T23:34:08.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.807+0000 7ff67ac2e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff67406cf60 0x7ff6740bfc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:08.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.807+0000 7ff67ac2e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6740c1dc0 0x7ff6741e7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:08.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.807+0000 7ff67ac2e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff67414eca0 0x7ff6741e8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:08.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.807+0000 7ff67ac2e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff6741514b0 con 0x7ff67414eca0 2024-03-28T23:34:08.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.807+0000 7ff67ac2e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff67414f080 con 0x7ff6740c1dc0 2024-03-28T23:34:08.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.807+0000 7ff67ac2e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff674151630 con 0x7ff67406cf60 2024-03-28T23:34:08.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.807+0000 7ff6791cb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff67414eca0 0x7ff6741e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:08.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.807+0000 7ff6791cb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff67414eca0 0x7ff6741e8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55542/0 (socket says 172.21.15.161:55542) 2024-03-28T23:34:08.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.807+0000 7ff6791cb640 1 -- 172.21.15.161:0/2948053053 learned_addr learned my addr 172.21.15.161:0/2948053053 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:08.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.807+0000 7ff673fff640 1 --2- 172.21.15.161:0/2948053053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6740c1dc0 0x7ff6741e7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:08.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.807+0000 7ff6789ca640 1 --2- 172.21.15.161:0/2948053053 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff67406cf60 0x7ff6740bfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:08.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.807+0000 7ff6791cb640 1 -- 172.21.15.161:0/2948053053 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff67406cf60 msgr2=0x7ff6740bfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:08.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.807+0000 7ff6791cb640 1 --2- 172.21.15.161:0/2948053053 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff67406cf60 0x7ff6740bfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.807+0000 7ff6791cb640 1 -- 172.21.15.161:0/2948053053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6740c1dc0 msgr2=0x7ff6741e7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:08.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.807+0000 7ff6791cb640 1 --2- 172.21.15.161:0/2948053053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6740c1dc0 0x7ff6741e7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:08.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.807+0000 7ff6791cb640 1 -- 172.21.15.161:0/2948053053 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff674073240 con 0x7ff67414eca0 2024-03-28T23:34:08.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.808+0000 7ff6791cb640 1 --2- 172.21.15.161:0/2948053053 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff67414eca0 0x7ff6741e8170 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7ff66c009f40 tx=0x7ff66c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:08.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.808+0000 7ff6717fa640 1 -- 172.21.15.161:0/2948053053 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66c043020 con 0x7ff67414eca0 2024-03-28T23:34:08.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.808+0000 7ff6717fa640 1 -- 172.21.15.161:0/2948053053 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff66c03e450 con 0x7ff67414eca0 2024-03-28T23:34:08.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.808+0000 7ff67ac2e640 1 -- 172.21.15.161:0/2948053053 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff674071aa0 con 0x7ff67414eca0 2024-03-28T23:34:08.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.808+0000 7ff6717fa640 1 -- 172.21.15.161:0/2948053053 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66c009510 con 0x7ff67414eca0 2024-03-28T23:34:08.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.808+0000 7ff67ac2e640 1 -- 172.21.15.161:0/2948053053 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff674071d40 con 0x7ff67414eca0 2024-03-28T23:34:08.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.809+0000 7ff67ac2e640 1 -- 172.21.15.161:0/2948053053 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6740a8d10 con 0x7ff67414eca0 2024-03-28T23:34:08.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.809+0000 7ff6717fa640 1 -- 172.21.15.161:0/2948053053 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff66c046020 con 0x7ff67414eca0 2024-03-28T23:34:08.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.810+0000 7ff6717fa640 1 --2- 172.21.15.161:0/2948053053 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff66006b4d0 0x7ff66006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:08.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.810+0000 7ff6717fa640 1 -- 172.21.15.161:0/2948053053 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7ff66c0ac930 con 0x7ff67414eca0 2024-03-28T23:34:08.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.810+0000 7ff6789ca640 1 --2- 172.21.15.161:0/2948053053 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff66006b4d0 0x7ff66006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:08.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.810+0000 7ff6789ca640 1 --2- 172.21.15.161:0/2948053053 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff66006b4d0 0x7ff66006d990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff6740c55d0 tx=0x7ff664009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:08.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:08.814+0000 7ff6717fa640 1 -- 172.21.15.161:0/2948053053 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff6740a8d10 con 0x7ff67414eca0 2024-03-28T23:34:09.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.021+0000 7ff67ac2e640 1 -- 172.21.15.161:0/2948053053 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6740aa890 con 0x7ff67414eca0 2024-03-28T23:34:09.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.021+0000 7ff6717fa640 1 -- 172.21.15.161:0/2948053053 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+5203 (secure 0 0 0) 0x7ff66c04d030 con 0x7ff67414eca0 2024-03-28T23:34:09.022 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:09.022 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":97,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6206,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3351418222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3351418222},{"type":"v1","addr":"172.21.15.170:6835","nonce":3351418222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":6892,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/2684760895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2684760895},{"type":"v1","addr":"172.21.15.161:6837","nonce":2684760895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":97,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T23:33:50.326492+0000","modified":"2024-03-28T23:34:08.057065+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6203":{"gid":6203,"name":"b","rank":0,"incarnation":92,"state":"up:active","state_seq":8,"addr":"172.21.15.170:6834/1096504848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1096504848},{"type":"v1","addr":"172.21.15.170:6834","nonce":1096504848}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7680":{"gid":7680,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/2943346687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":2943346687},{"type":"v1","addr":"172.21.15.161:6835","nonce":2943346687}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6203,"qdb_cluster":[6203]},"id":8}]} 2024-03-28T23:34:09.022 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 97 2024-03-28T23:34:09.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.023+0000 7ff67ac2e640 1 -- 172.21.15.161:0/2948053053 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff66006b4d0 msgr2=0x7ff66006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:09.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.023+0000 7ff67ac2e640 1 --2- 172.21.15.161:0/2948053053 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff66006b4d0 0x7ff66006d990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff6740c55d0 tx=0x7ff664009190 comp rx=0 tx=0).stop 2024-03-28T23:34:09.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.023+0000 7ff67ac2e640 1 -- 172.21.15.161:0/2948053053 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff67414eca0 msgr2=0x7ff6741e8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:09.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.023+0000 7ff67ac2e640 1 --2- 172.21.15.161:0/2948053053 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff67414eca0 0x7ff6741e8170 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7ff66c009f40 tx=0x7ff66c004a40 comp rx=0 tx=0).stop 2024-03-28T23:34:09.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.023+0000 7ff67ac2e640 1 -- 172.21.15.161:0/2948053053 shutdown_connections 2024-03-28T23:34:09.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.023+0000 7ff67ac2e640 1 --2- 172.21.15.161:0/2948053053 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff66006b4d0 0x7ff66006d990 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.023+0000 7ff67ac2e640 1 --2- 172.21.15.161:0/2948053053 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff67414eca0 0x7ff6741e8170 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.023+0000 7ff67ac2e640 1 --2- 172.21.15.161:0/2948053053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6740c1dc0 0x7ff6741e7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.023+0000 7ff67ac2e640 1 --2- 172.21.15.161:0/2948053053 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff67406cf60 0x7ff6740bfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.023+0000 7ff67ac2e640 1 -- 172.21.15.161:0/2948053053 >> 172.21.15.161:0/2948053053 conn(0x7ff6740c6670 msgr2=0x7ff674150da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:09.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.023+0000 7ff67ac2e640 1 -- 172.21.15.161:0/2948053053 shutdown_connections 2024-03-28T23:34:09.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.023+0000 7ff67ac2e640 1 -- 172.21.15.161:0/2948053053 wait complete. 2024-03-28T23:34:09.437 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-28T23:33:50.326492+0000', 'modified': '2024-03-28T23:34:08.057065+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 110, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6203}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6203': {'gid': 6203, 'name': 'b', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.170:6834/1096504848', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 1096504848}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 1096504848}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7680': {'gid': 7680, 'name': 'a', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.161:6835/2943346687', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 2943346687}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 2943346687}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6203, 'qdb_cluster': [6203]} 2024-03-28T23:34:09.437 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:34:09.438 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T23:34:09.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.541+0000 7f26f204e640 1 Processor -- start 2024-03-28T23:34:09.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.541+0000 7f26f204e640 1 -- start start 2024-03-28T23:34:09.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.542+0000 7f26f204e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26ec06ca20 0x7f26ec06ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:09.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.542+0000 7f26f204e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f26ec0a98c0 0x7f26ec0a9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:09.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.542+0000 7f26f204e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26ec151850 con 0x7f26ec14eca0 2024-03-28T23:34:09.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.542+0000 7f26f204e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f26ec152460 con 0x7f26ec0a98c0 2024-03-28T23:34:09.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.542+0000 7f26f204e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f26ec151b30 con 0x7f26ec06ca20 2024-03-28T23:34:09.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.542+0000 7f26ebfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f26ec0a98c0 0x7f26ec0a9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:09.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.542+0000 7f26eb7fe640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f26ec14eca0 0x7f26ec0c1880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:49442/0 (socket says 172.21.15.161:49442) 2024-03-28T23:34:09.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.542+0000 7f26eb7fe640 1 -- 172.21.15.161:0/1933407346 learned_addr learned my addr 172.21.15.161:0/1933407346 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:09.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.542+0000 7f26eaffd640 1 --2- 172.21.15.161:0/1933407346 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26ec06ca20 0x7f26ec06ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:09.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.542+0000 7f26ebfff640 1 -- 172.21.15.161:0/1933407346 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26ec06ca20 msgr2=0x7f26ec06ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:09.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.542+0000 7f26ebfff640 1 --2- 172.21.15.161:0/1933407346 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26ec06ca20 0x7f26ec06ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.542+0000 7f26ebfff640 1 -- 172.21.15.161:0/1933407346 >> v1:172.21.15.170:6789/0 conn(0x7f26ec14eca0 legacy=0x7f26ec0c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:09.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.542+0000 7f26ebfff640 1 -- 172.21.15.161:0/1933407346 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26ec151e10 con 0x7f26ec0a98c0 2024-03-28T23:34:09.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.543+0000 7f26ebfff640 1 --2- 172.21.15.161:0/1933407346 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26ec0a98c0 0x7f26ec0a9d20 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009d90 tx=0x7f26e402bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54af5ce7b73cf087 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:09.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.543+0000 7f26ea7fc640 1 -- 172.21.15.161:0/1933407346 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e4039070 con 0x7f26ec0a98c0 2024-03-28T23:34:09.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.543+0000 7f26ea7fc640 1 -- 172.21.15.161:0/1933407346 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26e402c980 con 0x7f26ec0a98c0 2024-03-28T23:34:09.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.543+0000 7f26ea7fc640 1 -- 172.21.15.161:0/1933407346 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e402cd50 con 0x7f26ec0a98c0 2024-03-28T23:34:09.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.544+0000 7f26f204e640 1 -- 172.21.15.161:0/1933407346 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26ec0a98c0 msgr2=0x7f26ec0a9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:09.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.544+0000 7f26f204e640 1 --2- 172.21.15.161:0/1933407346 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26ec0a98c0 0x7f26ec0a9d20 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009d90 tx=0x7f26e402bc50 comp rx=0 tx=0).stop 2024-03-28T23:34:09.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.544+0000 7f26f204e640 1 -- 172.21.15.161:0/1933407346 shutdown_connections 2024-03-28T23:34:09.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.544+0000 7f26f204e640 1 --2- 172.21.15.161:0/1933407346 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26ec0a98c0 0x7f26ec0a9d20 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.544+0000 7f26f204e640 1 --2- 172.21.15.161:0/1933407346 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26ec06ca20 0x7f26ec06ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.544+0000 7f26f204e640 1 -- 172.21.15.161:0/1933407346 >> 172.21.15.161:0/1933407346 conn(0x7f26ec0c6670 msgr2=0x7f26ec0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:09.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.544+0000 7f26f204e640 1 -- 172.21.15.161:0/1933407346 shutdown_connections 2024-03-28T23:34:09.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.544+0000 7f26f204e640 1 -- 172.21.15.161:0/1933407346 wait complete. 2024-03-28T23:34:09.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.545+0000 7f26f204e640 1 Processor -- start 2024-03-28T23:34:09.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.545+0000 7f26f204e640 1 -- start start 2024-03-28T23:34:09.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.546+0000 7f26f204e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26ec06ca20 0x7f26ec0c5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:09.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.546+0000 7f26f204e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26ec0a98c0 0x7f26ec13f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:09.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.546+0000 7f26f204e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26ec14eca0 0x7f26ec13f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:09.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.546+0000 7f26f204e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec1514b0 con 0x7f26ec06ca20 2024-03-28T23:34:09.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.546+0000 7f26f204e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec14f080 con 0x7f26ec0a98c0 2024-03-28T23:34:09.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.546+0000 7f26f204e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f26ec151630 con 0x7f26ec14eca0 2024-03-28T23:34:09.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.546+0000 7f26eb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26ec06ca20 0x7f26ec0c5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:09.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.546+0000 7f26eb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26ec06ca20 0x7f26ec0c5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55560/0 (socket says 172.21.15.161:55560) 2024-03-28T23:34:09.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.546+0000 7f26eb7fe640 1 -- 172.21.15.161:0/2736955574 learned_addr learned my addr 172.21.15.161:0/2736955574 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:09.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.546+0000 7f26ebfff640 1 --2- 172.21.15.161:0/2736955574 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26ec14eca0 0x7f26ec13f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:09.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.546+0000 7f26eaffd640 1 --2- 172.21.15.161:0/2736955574 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26ec0a98c0 0x7f26ec13f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:09.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.546+0000 7f26eb7fe640 1 -- 172.21.15.161:0/2736955574 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26ec14eca0 msgr2=0x7f26ec13f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:09.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.546+0000 7f26eb7fe640 1 --2- 172.21.15.161:0/2736955574 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26ec14eca0 0x7f26ec13f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.546+0000 7f26eb7fe640 1 -- 172.21.15.161:0/2736955574 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26ec0a98c0 msgr2=0x7f26ec13f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:09.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.546+0000 7f26eb7fe640 1 --2- 172.21.15.161:0/2736955574 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26ec0a98c0 0x7f26ec13f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.546+0000 7f26eb7fe640 1 -- 172.21.15.161:0/2736955574 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26ec143910 con 0x7f26ec06ca20 2024-03-28T23:34:09.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.546+0000 7f26eb7fe640 1 --2- 172.21.15.161:0/2736955574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26ec06ca20 0x7f26ec0c5630 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f26dc00c780 tx=0x7f26dc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:09.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.546+0000 7f26d3fff640 1 -- 172.21.15.161:0/2736955574 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26dc018070 con 0x7f26ec06ca20 2024-03-28T23:34:09.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.546+0000 7f26d3fff640 1 -- 172.21.15.161:0/2736955574 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26dc0072d0 con 0x7f26ec06ca20 2024-03-28T23:34:09.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.546+0000 7f26f204e640 1 -- 172.21.15.161:0/2736955574 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26ec14c9b0 con 0x7f26ec06ca20 2024-03-28T23:34:09.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.546+0000 7f26d3fff640 1 -- 172.21.15.161:0/2736955574 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26dc0047e0 con 0x7f26ec06ca20 2024-03-28T23:34:09.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.546+0000 7f26f204e640 1 -- 172.21.15.161:0/2736955574 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26ec141760 con 0x7f26ec06ca20 2024-03-28T23:34:09.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.548+0000 7f26d3fff640 1 -- 172.21.15.161:0/2736955574 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f26dc01b070 con 0x7f26ec06ca20 2024-03-28T23:34:09.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.548+0000 7f26d3fff640 1 --2- 172.21.15.161:0/2736955574 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f26d806b4d0 0x7f26d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:09.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.548+0000 7f26d3fff640 1 -- 172.21.15.161:0/2736955574 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7f26dc0896e0 con 0x7f26ec06ca20 2024-03-28T23:34:09.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.548+0000 7f26eaffd640 1 --2- 172.21.15.161:0/2736955574 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f26d806b4d0 0x7f26d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:09.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.549+0000 7f26eaffd640 1 --2- 172.21.15.161:0/2736955574 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f26d806b4d0 0x7f26d806d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f26ec1504c0 tx=0x7f26e0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:09.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.549+0000 7f26f4c35740 1 Processor -- start 2024-03-28T23:34:09.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.549+0000 7f26f4c35740 1 -- start start 2024-03-28T23:34:09.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.550+0000 7f26f4c35740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852318db0 0x564852316890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:09.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.550+0000 7f26f4c35740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564852319190 0x564852316dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:09.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.550+0000 7f26f4c35740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564852317310 0x5648523134e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:09.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.550+0000 7f26f4c35740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x56485227caf0 con 0x564852318db0 2024-03-28T23:34:09.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.550+0000 7f26f4c35740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5648522259d0 con 0x564852319190 2024-03-28T23:34:09.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.550+0000 7f26f4c35740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x564852278b90 con 0x564852317310 2024-03-28T23:34:09.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.550+0000 7f26eb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852318db0 0x564852316890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:09.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.550+0000 7f26eb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852318db0 0x564852316890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55568/0 (socket says 172.21.15.161:55568) 2024-03-28T23:34:09.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.550+0000 7f26eb7fe640 1 -- 172.21.15.161:0/4149121669 learned_addr learned my addr 172.21.15.161:0/4149121669 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:09.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.550+0000 7f26ebfff640 1 --2- 172.21.15.161:0/4149121669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564852317310 0x5648523134e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:09.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.550+0000 7f26eaffd640 1 --2- 172.21.15.161:0/4149121669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564852319190 0x564852316dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:09.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.550+0000 7f26eb7fe640 1 -- 172.21.15.161:0/4149121669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564852317310 msgr2=0x5648523134e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:09.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.550+0000 7f26eb7fe640 1 --2- 172.21.15.161:0/4149121669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564852317310 0x5648523134e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.550+0000 7f26eb7fe640 1 -- 172.21.15.161:0/4149121669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564852319190 msgr2=0x564852316dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:09.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.550+0000 7f26eb7fe640 1 --2- 172.21.15.161:0/4149121669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564852319190 0x564852316dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.550+0000 7f26eb7fe640 1 -- 172.21.15.161:0/4149121669 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564851f912d0 con 0x564852318db0 2024-03-28T23:34:09.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.551+0000 7f26eb7fe640 1 --2- 172.21.15.161:0/4149121669 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852318db0 0x564852316890 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0077e0 tx=0x7f26dc007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8abc287163181f29 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:09.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.551+0000 7f26d17fa640 1 -- 172.21.15.161:0/4149121669 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26dc087070 con 0x564852318db0 2024-03-28T23:34:09.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.551+0000 7f26d17fa640 1 -- 172.21.15.161:0/4149121669 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26dc087e50 con 0x564852318db0 2024-03-28T23:34:09.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.551+0000 7f26d17fa640 1 -- 172.21.15.161:0/4149121669 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26dc060450 con 0x564852318db0 2024-03-28T23:34:09.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.551+0000 7f26f4c35740 1 -- 172.21.15.161:0/4149121669 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852318db0 msgr2=0x564852316890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:09.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.551+0000 7f26f4c35740 1 --2- 172.21.15.161:0/4149121669 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852318db0 0x564852316890 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0077e0 tx=0x7f26dc007f90 comp rx=0 tx=0).stop 2024-03-28T23:34:09.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.551+0000 7f26f4c35740 1 -- 172.21.15.161:0/4149121669 shutdown_connections 2024-03-28T23:34:09.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.551+0000 7f26f4c35740 1 --2- 172.21.15.161:0/4149121669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564852317310 0x5648523134e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.551+0000 7f26f4c35740 1 --2- 172.21.15.161:0/4149121669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564852319190 0x564852316dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.551+0000 7f26f4c35740 1 --2- 172.21.15.161:0/4149121669 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852318db0 0x564852316890 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.551+0000 7f26f4c35740 1 -- 172.21.15.161:0/4149121669 >> 172.21.15.161:0/4149121669 conn(0x564852315630 msgr2=0x56485233a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:09.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.551+0000 7f26f4c35740 1 -- 172.21.15.161:0/4149121669 shutdown_connections 2024-03-28T23:34:09.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.551+0000 7f26f4c35740 1 -- 172.21.15.161:0/4149121669 wait complete. 2024-03-28T23:34:09.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.552+0000 7f26f4c35740 1 -- start start 2024-03-28T23:34:09.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.552+0000 7f26f4c35740 1 Processor -- start 2024-03-28T23:34:09.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.553+0000 7f26f4c35740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564852318db0 0x5648523690c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:09.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.553+0000 7f26f4c35740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852319190 0x564852339610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:09.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.553+0000 7f26f4c35740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56485233e1e0 0x564852365500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:09.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.553+0000 7f26f4c35740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x56485233b090 con 0x564852319190 2024-03-28T23:34:09.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.553+0000 7f26f4c35740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x564852238f70 con 0x564852318db0 2024-03-28T23:34:09.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.553+0000 7f26f4c35740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x56485233aad0 con 0x56485233e1e0 2024-03-28T23:34:09.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.553+0000 7f26f4c35740 10 client.0 _ll_register_callbacks cb 0x564851f90ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:34:09.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.553+0000 7f26eaffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852319190 0x564852339610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:09.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.553+0000 7f26eaffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852319190 0x564852339610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55570/0 (socket says 172.21.15.161:55570) 2024-03-28T23:34:09.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.553+0000 7f26eaffd640 1 -- 172.21.15.161:0/567629422 learned_addr learned my addr 172.21.15.161:0/567629422 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:09.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.553+0000 7f26ebfff640 1 --2- 172.21.15.161:0/567629422 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56485233e1e0 0x564852365500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:09.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.553+0000 7f26eb7fe640 1 --2- 172.21.15.161:0/567629422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564852318db0 0x5648523690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:09.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.553+0000 7f26eaffd640 1 -- 172.21.15.161:0/567629422 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56485233e1e0 msgr2=0x564852365500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:09.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.553+0000 7f26eaffd640 1 --2- 172.21.15.161:0/567629422 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56485233e1e0 0x564852365500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.553+0000 7f26eaffd640 1 -- 172.21.15.161:0/567629422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564852318db0 msgr2=0x5648523690c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:09.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.553+0000 7f26eaffd640 1 --2- 172.21.15.161:0/567629422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564852318db0 0x5648523690c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.553+0000 7f26eaffd640 1 -- 172.21.15.161:0/567629422 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56485205b6b0 con 0x564852319190 2024-03-28T23:34:09.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.553+0000 7f26eaffd640 1 --2- 172.21.15.161:0/567629422 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852319190 0x564852339610 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f26e0011ff0 tx=0x7f26e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4bd0711b945abe9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:09.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.553+0000 7f26ca7fc640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:34:09.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.553+0000 7f26ca7fc640 1 -- 172.21.15.161:0/567629422 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e0027070 con 0x564852319190 2024-03-28T23:34:09.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.553+0000 7f26ca7fc640 1 -- 172.21.15.161:0/567629422 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7f26e0014440 con 0x564852319190 2024-03-28T23:34:09.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.553+0000 7f26f4c35740 1 -- 172.21.15.161:0/567629422 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26dc087e50 con 0x564852319190 2024-03-28T23:34:09.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.554+0000 7f26ca7fc640 1 -- 172.21.15.161:0/567629422 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f26e0014640 con 0x564852319190 2024-03-28T23:34:09.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.554+0000 7f26f4c35740 10 client.7749 fetch_fsmap learned FSMap version 97 2024-03-28T23:34:09.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.554+0000 7f26f4c35740 1 -- 172.21.15.161:0/567629422 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=2+}) -- 0x564852365f60 con 0x564852319190 2024-03-28T23:34:09.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.554+0000 7f26ca7fc640 1 -- 172.21.15.161:0/567629422 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26dc087e50 con 0x564852319190 2024-03-28T23:34:09.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.554+0000 7f26ca7fc640 1 -- 172.21.15.161:0/567629422 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 97) ==== 3482+0+0 (secure 0 0 0) 0x7f26e0035080 con 0x564852319190 2024-03-28T23:34:09.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.554+0000 7f26f4c35740 10 client.7749 fetch_fsmap finished waiting for FSMap version 97 2024-03-28T23:34:09.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.554+0000 7f26f4c35740 10 client.7749 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:34:09.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.554+0000 7f26f4c35740 20 client.7749 populate_metadata read hostname 'smithi161' 2024-03-28T23:34:09.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.554+0000 7f26f4c35740 1 --2- 172.21.15.161:0/567629422 >> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] conn(0x56485233bd50 0x564852373cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:09.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.554+0000 7f26f4c35740 4 client.7749 mds_command: new command op to 6203 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:34:09.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.554+0000 7f26f4c35740 1 -- 172.21.15.161:0/567629422 --> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56485228a030 con 0x56485233bd50 2024-03-28T23:34:09.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.554+0000 7f26eb7fe640 1 --2- 172.21.15.161:0/567629422 >> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] conn(0x56485233bd50 0x564852373cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:09.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.554+0000 7f26ca7fc640 10 client.7749 ms_handle_connect on v2:172.21.15.170:6832/1096504848 2024-03-28T23:34:09.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.554+0000 7f26eb7fe640 1 --2- 172.21.15.161:0/567629422 >> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] conn(0x56485233bd50 0x564852373cf0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a91d4318dbe70c7 server_cookie=cca200598d0548c3 in_seq=0 out_seq=0 2024-03-28T23:34:09.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.555+0000 7f26ca7fc640 1 -- 172.21.15.161:0/567629422 <== mds.0 v2:172.21.15.170:6832/1096504848 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56485228a030 con 0x56485233bd50 2024-03-28T23:34:09.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.555+0000 7f26ca7fc640 10 client.7749 handle_command_reply: tid=0 2024-03-28T23:34:09.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.555+0000 7f26f4c35740 1 client.7749 shutdown 2024-03-28T23:34:09.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.555+0000 7f26f4c35740 1 -- 172.21.15.161:0/567629422 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852319190 msgr2=0x564852339610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:09.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.555+0000 7f26f4c35740 1 --2- 172.21.15.161:0/567629422 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852319190 0x564852339610 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f26e0011ff0 tx=0x7f26e00046c0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.555+0000 7f26f4c35740 1 -- 172.21.15.161:0/567629422 shutdown_connections 2024-03-28T23:34:09.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.555+0000 7f26f4c35740 1 --2- 172.21.15.161:0/567629422 >> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] conn(0x56485233bd50 0x564852373cf0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.555+0000 7f26f4c35740 1 --2- 172.21.15.161:0/567629422 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852319190 0x564852339610 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.555+0000 7f26ca7fc640 0 client.7749 ms_handle_reset on v2:172.21.15.170:6832/1096504848 2024-03-28T23:34:09.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.555+0000 7f26f4c35740 1 --2- 172.21.15.161:0/567629422 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56485233e1e0 0x564852365500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.555+0000 7f26f4c35740 1 --2- 172.21.15.161:0/567629422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564852318db0 0x5648523690c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.555+0000 7f26f4c35740 1 -- 172.21.15.161:0/567629422 >> 172.21.15.161:0/567629422 conn(0x564852317310 msgr2=0x5648523559b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:09.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.555+0000 7f26f4c35740 1 -- 172.21.15.161:0/567629422 shutdown_connections 2024-03-28T23:34:09.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.555+0000 7f26f4c35740 1 -- 172.21.15.161:0/567629422 wait complete. 2024-03-28T23:34:09.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.555+0000 7f26f4c35740 20 client.7749 trim_cache size 0 max 16384 2024-03-28T23:34:09.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.573+0000 7f26f4c35740 1 Processor -- start 2024-03-28T23:34:09.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.573+0000 7f26f4c35740 1 -- start start 2024-03-28T23:34:09.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.574+0000 7f26f4c35740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564852318db0 0x5648523567b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:09.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.574+0000 7f26f4c35740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852319190 0x564852316610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:09.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.574+0000 7f26f4c35740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56485233e1e0 0x564852316b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:09.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.574+0000 7f26f4c35740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec1f6070 con 0x564852319190 2024-03-28T23:34:09.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.574+0000 7f26f4c35740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec1f43f0 con 0x564852318db0 2024-03-28T23:34:09.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.574+0000 7f26f4c35740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f26ec1f4a40 con 0x56485233e1e0 2024-03-28T23:34:09.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.574+0000 7f26eaffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852319190 0x564852316610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:09.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.574+0000 7f26eaffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852319190 0x564852316610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55572/0 (socket says 172.21.15.161:55572) 2024-03-28T23:34:09.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.574+0000 7f26eaffd640 1 -- 172.21.15.161:0/88686419 learned_addr learned my addr 172.21.15.161:0/88686419 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:09.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.575+0000 7f26eb7fe640 1 --2- 172.21.15.161:0/88686419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564852318db0 0x5648523567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:09.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.575+0000 7f26ebfff640 1 --2- 172.21.15.161:0/88686419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56485233e1e0 0x564852316b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:09.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.575+0000 7f26eaffd640 1 -- 172.21.15.161:0/88686419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56485233e1e0 msgr2=0x564852316b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:09.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.575+0000 7f26eaffd640 1 --2- 172.21.15.161:0/88686419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56485233e1e0 0x564852316b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.575+0000 7f26eaffd640 1 -- 172.21.15.161:0/88686419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564852318db0 msgr2=0x5648523567b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:09.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.575+0000 7f26eaffd640 1 --2- 172.21.15.161:0/88686419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564852318db0 0x5648523567b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.575+0000 7f26eaffd640 1 -- 172.21.15.161:0/88686419 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56485233d350 con 0x564852319190 2024-03-28T23:34:09.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.575+0000 7f26eb7fe640 1 --2- 172.21.15.161:0/88686419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564852318db0 0x5648523567b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:09.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.575+0000 7f26eaffd640 1 --2- 172.21.15.161:0/88686419 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852319190 0x564852316610 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f26e00149c0 tx=0x7f26e0024000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48d5ddfd7e6c3516 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:09.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.575+0000 7f26d0ff9640 1 -- 172.21.15.161:0/88686419 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e0027040 con 0x564852319190 2024-03-28T23:34:09.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.575+0000 7f26d0ff9640 1 -- 172.21.15.161:0/88686419 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26e0014c00 con 0x564852319190 2024-03-28T23:34:09.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.575+0000 7f26d0ff9640 1 -- 172.21.15.161:0/88686419 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e002c440 con 0x564852319190 2024-03-28T23:34:09.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.576+0000 7f26f4c35740 1 -- 172.21.15.161:0/88686419 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852319190 msgr2=0x564852316610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:09.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.576+0000 7f26f4c35740 1 --2- 172.21.15.161:0/88686419 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852319190 0x564852316610 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f26e00149c0 tx=0x7f26e0024000 comp rx=0 tx=0).stop 2024-03-28T23:34:09.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.576+0000 7f26f4c35740 1 -- 172.21.15.161:0/88686419 shutdown_connections 2024-03-28T23:34:09.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.576+0000 7f26f4c35740 1 --2- 172.21.15.161:0/88686419 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852319190 0x564852316610 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.576+0000 7f26f4c35740 1 --2- 172.21.15.161:0/88686419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56485233e1e0 0x564852316b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.576+0000 7f26f4c35740 1 --2- 172.21.15.161:0/88686419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564852318db0 0x5648523567b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.576+0000 7f26f4c35740 1 -- 172.21.15.161:0/88686419 >> 172.21.15.161:0/88686419 conn(0x564852317310 msgr2=0x564852352050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:09.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.576+0000 7f26f4c35740 1 -- 172.21.15.161:0/88686419 shutdown_connections 2024-03-28T23:34:09.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.576+0000 7f26f4c35740 1 -- 172.21.15.161:0/88686419 wait complete. 2024-03-28T23:34:09.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.576+0000 7f26f4c35740 1 -- start start 2024-03-28T23:34:09.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.576+0000 7f26f4c35740 1 Processor -- start 2024-03-28T23:34:09.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.577+0000 7f26f4c35740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852318db0 0x564852338790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:09.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.577+0000 7f26f4c35740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564852319190 0x5648523d3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:09.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.577+0000 7f26f4c35740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56485233e1e0 0x5648523d0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:09.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.577+0000 7f26f4c35740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x56485233f770 con 0x564852318db0 2024-03-28T23:34:09.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.577+0000 7f26f4c35740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x564852336600 con 0x56485233e1e0 2024-03-28T23:34:09.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.577+0000 7f26f4c35740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x564852335fe0 con 0x564852319190 2024-03-28T23:34:09.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.577+0000 7f26f4c35740 10 client.0 _ll_register_callbacks cb 0x7f26e0011b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:34:09.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.577+0000 7f26eb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852318db0 0x564852338790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:09.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.577+0000 7f26eb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852318db0 0x564852338790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55574/0 (socket says 172.21.15.161:55574) 2024-03-28T23:34:09.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.577+0000 7f26eb7fe640 1 -- 172.21.15.161:0/3584551337 learned_addr learned my addr 172.21.15.161:0/3584551337 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:09.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.577+0000 7f26eaffd640 1 --2- 172.21.15.161:0/3584551337 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564852319190 0x5648523d3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:09.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.577+0000 7f26ebfff640 1 --2- 172.21.15.161:0/3584551337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56485233e1e0 0x5648523d0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:09.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.577+0000 7f26eb7fe640 1 -- 172.21.15.161:0/3584551337 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564852319190 msgr2=0x5648523d3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:09.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.577+0000 7f26eb7fe640 1 --2- 172.21.15.161:0/3584551337 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564852319190 0x5648523d3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.577+0000 7f26eb7fe640 1 -- 172.21.15.161:0/3584551337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56485233e1e0 msgr2=0x5648523d0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:09.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.577+0000 7f26eb7fe640 1 --2- 172.21.15.161:0/3584551337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56485233e1e0 0x5648523d0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.577+0000 7f26eb7fe640 1 -- 172.21.15.161:0/3584551337 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564852287690 con 0x564852318db0 2024-03-28T23:34:09.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.578+0000 7f26eb7fe640 1 --2- 172.21.15.161:0/3584551337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852318db0 0x564852338790 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f26dc087a00 tx=0x7f26dc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b0875a104dfb24 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:09.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.578+0000 7f26ca7fc640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:34:09.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.578+0000 7f26ca7fc640 1 -- 172.21.15.161:0/3584551337 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26dc087040 con 0x564852318db0 2024-03-28T23:34:09.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.578+0000 7f26ca7fc640 1 -- 172.21.15.161:0/3584551337 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7f26dc054460 con 0x564852318db0 2024-03-28T23:34:09.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.578+0000 7f26f4c35740 1 -- 172.21.15.161:0/3584551337 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26e0014c00 con 0x564852318db0 2024-03-28T23:34:09.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.578+0000 7f26ca7fc640 1 -- 172.21.15.161:0/3584551337 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7f26dc076b50 con 0x564852318db0 2024-03-28T23:34:09.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.578+0000 7f26f4c35740 10 client.7755 fetch_fsmap learned FSMap version 97 2024-03-28T23:34:09.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.578+0000 7f26f4c35740 1 -- 172.21.15.161:0/3584551337 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=2+}) -- 0x564852316ed0 con 0x564852318db0 2024-03-28T23:34:09.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.578+0000 7f26ca7fc640 1 -- 172.21.15.161:0/3584551337 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26e0014c00 con 0x564852318db0 2024-03-28T23:34:09.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.578+0000 7f26ca7fc640 1 -- 172.21.15.161:0/3584551337 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 97) ==== 3482+0+0 (secure 0 0 0) 0x7f26dc057240 con 0x564852318db0 2024-03-28T23:34:09.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.578+0000 7f26f4c35740 10 client.7755 fetch_fsmap finished waiting for FSMap version 97 2024-03-28T23:34:09.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.578+0000 7f26f4c35740 10 client.7755 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:34:09.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.578+0000 7f26f4c35740 20 client.7755 populate_metadata read hostname 'smithi161' 2024-03-28T23:34:09.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.578+0000 7f26f4c35740 1 --2- 172.21.15.161:0/3584551337 >> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] conn(0x5648523d0f70 0x564852377d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:09.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.578+0000 7f26f4c35740 4 client.7755 mds_command: new command op to 6203 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:34:09.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.578+0000 7f26f4c35740 1 -- 172.21.15.161:0/3584551337 --> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] -- command(tid 0: {"prefix": "status"}) -- 0x564852284380 con 0x5648523d0f70 2024-03-28T23:34:09.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.578+0000 7f26eaffd640 1 --2- 172.21.15.161:0/3584551337 >> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] conn(0x5648523d0f70 0x564852377d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:09.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.579+0000 7f26ca7fc640 10 client.7755 ms_handle_connect on v2:172.21.15.170:6832/1096504848 2024-03-28T23:34:09.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.579+0000 7f26eaffd640 1 --2- 172.21.15.161:0/3584551337 >> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] conn(0x5648523d0f70 0x564852377d10 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc10b1aae7c0aa87 server_cookie=dbbe724cf98de326 in_seq=0 out_seq=0 2024-03-28T23:34:09.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.579+0000 7f26ca7fc640 1 -- 172.21.15.161:0/3584551337 <== mds.0 v2:172.21.15.170:6832/1096504848 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x564852284380 con 0x5648523d0f70 2024-03-28T23:34:09.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.579+0000 7f26ca7fc640 10 client.7755 handle_command_reply: tid=0 2024-03-28T23:34:09.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.579+0000 7f26f4c35740 1 client.7755 shutdown 2024-03-28T23:34:09.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.579+0000 7f26f4c35740 1 -- 172.21.15.161:0/3584551337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852318db0 msgr2=0x564852338790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:09.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.579+0000 7f26f4c35740 1 --2- 172.21.15.161:0/3584551337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852318db0 0x564852338790 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f26dc087a00 tx=0x7f26dc000bc0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.579+0000 7f26f4c35740 1 -- 172.21.15.161:0/3584551337 shutdown_connections 2024-03-28T23:34:09.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.579+0000 7f26f4c35740 1 --2- 172.21.15.161:0/3584551337 >> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] conn(0x5648523d0f70 0x564852377d10 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.579+0000 7f26f4c35740 1 --2- 172.21.15.161:0/3584551337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56485233e1e0 0x5648523d0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.579+0000 7f26f4c35740 1 --2- 172.21.15.161:0/3584551337 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564852319190 0x5648523d3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.579+0000 7f26f4c35740 1 --2- 172.21.15.161:0/3584551337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564852318db0 0x564852338790 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.579+0000 7f26ca7fc640 0 client.7755 ms_handle_reset on v2:172.21.15.170:6832/1096504848 2024-03-28T23:34:09.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.579+0000 7f26f4c35740 1 -- 172.21.15.161:0/3584551337 >> 172.21.15.161:0/3584551337 conn(0x564852317310 msgr2=0x564852339f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:09.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.579+0000 7f26f4c35740 1 -- 172.21.15.161:0/3584551337 shutdown_connections 2024-03-28T23:34:09.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.579+0000 7f26f4c35740 1 -- 172.21.15.161:0/3584551337 wait complete. 2024-03-28T23:34:09.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.579+0000 7f26f4c35740 20 client.7755 trim_cache size 0 max 16384 2024-03-28T23:34:09.587 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:34:09.587 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:34:09.587 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:34:09.587 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 6203, 2024-03-28T23:34:09.587 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:34:09.587 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:34:09.587 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:34:09.587 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 7.5436087250000003, 2024-03-28T23:34:09.587 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 97, 2024-03-28T23:34:09.587 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 110, 2024-03-28T23:34:09.587 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 110, 2024-03-28T23:34:09.587 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 23.687645617000001 2024-03-28T23:34:09.587 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:34:09.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.580+0000 7f26f204e640 1 -- 172.21.15.161:0/2736955574 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f26d806b4d0 msgr2=0x7f26d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:09.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.580+0000 7f26f204e640 1 --2- 172.21.15.161:0/2736955574 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f26d806b4d0 0x7f26d806d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f26ec1504c0 tx=0x7f26e0009190 comp rx=0 tx=0).stop 2024-03-28T23:34:09.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.580+0000 7f26f204e640 1 -- 172.21.15.161:0/2736955574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26ec06ca20 msgr2=0x7f26ec0c5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:09.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.580+0000 7f26f204e640 1 --2- 172.21.15.161:0/2736955574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26ec06ca20 0x7f26ec0c5630 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f26dc00c780 tx=0x7f26dc00cc40 comp rx=0 tx=0).stop 2024-03-28T23:34:09.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.580+0000 7f26f204e640 1 -- 172.21.15.161:0/2736955574 shutdown_connections 2024-03-28T23:34:09.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.580+0000 7f26f204e640 1 --2- 172.21.15.161:0/2736955574 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f26d806b4d0 0x7f26d806d990 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.580+0000 7f26f204e640 1 --2- 172.21.15.161:0/2736955574 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26ec14eca0 0x7f26ec13f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.580+0000 7f26f204e640 1 --2- 172.21.15.161:0/2736955574 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26ec0a98c0 0x7f26ec13f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.580+0000 7f26f204e640 1 --2- 172.21.15.161:0/2736955574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26ec06ca20 0x7f26ec0c5630 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:09.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.580+0000 7f26f204e640 1 -- 172.21.15.161:0/2736955574 >> 172.21.15.161:0/2736955574 conn(0x7f26ec0c6670 msgr2=0x7f26ec0c1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:09.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.580+0000 7f26f204e640 1 -- 172.21.15.161:0/2736955574 shutdown_connections 2024-03-28T23:34:09.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:09.580+0000 7f26f204e640 1 -- 172.21.15.161:0/2736955574 wait complete. 2024-03-28T23:34:09.591 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 97, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:33:50.326492+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 97, "ever_allowed_features": 32, "explicitly_allowed_features": 32, "failed": [], "flags": 50, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": true, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6203": { "addr": "172.21.15.170:6834/1096504848", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 1096504848, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 1096504848, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6203, "incarnation": 92, "join_fscid": 8, "name": "b", "rank": 0, "state": "up:active", "state_seq": 8 }, "gid_7680": { "addr": "172.21.15.161:6835/2943346687", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 2943346687, "type": "v2" }, { "addr": "172.21.15.161:6835", "nonce": 2943346687, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7680, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:standby-replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 110, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-03-28T23:34:08.057065+0000", "qdb_cluster": [ 6203 ], "qdb_leader": 6203, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6203 } } } ], "standbys": [ { "addr": "172.21.15.170:6835/3351418222", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 3351418222, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 3351418222, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 87, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6206, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6837/2684760895", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6836", "nonce": 2684760895, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 2684760895, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 87, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6892, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:34:09.592 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 8, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 8, 'name': 'd', 'state': 'up:standby-replay'}] 2024-03-28T23:34:09.592 INFO:tasks.cephfs.test_failover:current = [{'gid': 7680, 'name': 'a', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.161:6835/2943346687', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 2943346687}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 2943346687}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 6203, 'name': 'b', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.170:6834/1096504848', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 1096504848}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 1096504848}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 6892, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6837/2684760895', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 2684760895}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 2684760895}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 87}, {'gid': 6206, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/3351418222', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 3351418222}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 3351418222}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 87}] 2024-03-28T23:34:09.592 DEBUG:tasks.cephfs.test_failover:8 != -1 2024-03-28T23:34:09.592 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-28T23:34:10.092 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-28T23:34:14.594 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:34:14.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.705+0000 7f44125c9640 1 Processor -- start 2024-03-28T23:34:14.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.705+0000 7f44125c9640 1 -- start start 2024-03-28T23:34:14.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.706+0000 7f44125c9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f440c0c2390 0x7f440c0a8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:14.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.706+0000 7f44125c9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f440c0aad50 0x7f440c0ab1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:14.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.706+0000 7f44125c9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f440c1550d0 con 0x7f440c152a90 2024-03-28T23:34:14.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.706+0000 7f44125c9640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f440c155fa0 con 0x7f440c0c2390 2024-03-28T23:34:14.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.706+0000 7f44125c9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f440c155390 con 0x7f440c0aad50 2024-03-28T23:34:14.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.706+0000 7f440b7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f440c0c2390 0x7f440c0a8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:14.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.706+0000 7f440b7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f440c0c2390 0x7f440c0a8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45112/0 (socket says 172.21.15.161:45112) 2024-03-28T23:34:14.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.706+0000 7f440b7fe640 1 -- 172.21.15.161:0/658939474 learned_addr learned my addr 172.21.15.161:0/658939474 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:14.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.706+0000 7f4410b66640 1 --2- 172.21.15.161:0/658939474 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f440c0aad50 0x7f440c0ab1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:14.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.706+0000 7f440affd640 1 -- 172.21.15.161:0/658939474 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3990728054 0 0) 0x7f440c1550d0 con 0x7f440c152a90 2024-03-28T23:34:14.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.706+0000 7f440b7fe640 1 -- 172.21.15.161:0/658939474 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f440c0aad50 msgr2=0x7f440c0ab1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:14.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.706+0000 7f440b7fe640 1 --2- 172.21.15.161:0/658939474 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f440c0aad50 0x7f440c0ab1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:14.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.706+0000 7f440b7fe640 1 -- 172.21.15.161:0/658939474 >> v1:172.21.15.170:6789/0 conn(0x7f440c152a90 legacy=0x7f440c0bfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:14.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.706+0000 7f440b7fe640 1 -- 172.21.15.161:0/658939474 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f440c155670 con 0x7f440c0c2390 2024-03-28T23:34:14.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.707+0000 7f440b7fe640 1 --2- 172.21.15.161:0/658939474 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f440c0c2390 0x7f440c0a8d10 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f44000098a0 tx=0x7f440002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf2bd53eb107a71 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:14.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.707+0000 7f440affd640 1 -- 172.21.15.161:0/658939474 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4400035020 con 0x7f440c0c2390 2024-03-28T23:34:14.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.707+0000 7f440affd640 1 -- 172.21.15.161:0/658939474 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4400031640 con 0x7f440c0c2390 2024-03-28T23:34:14.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.707+0000 7f440affd640 1 -- 172.21.15.161:0/658939474 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440002d040 con 0x7f440c0c2390 2024-03-28T23:34:14.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.708+0000 7f44125c9640 1 -- 172.21.15.161:0/658939474 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f440c0c2390 msgr2=0x7f440c0a8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:14.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.708+0000 7f44125c9640 1 --2- 172.21.15.161:0/658939474 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f440c0c2390 0x7f440c0a8d10 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f44000098a0 tx=0x7f440002b8f0 comp rx=0 tx=0).stop 2024-03-28T23:34:14.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.708+0000 7f44125c9640 1 -- 172.21.15.161:0/658939474 shutdown_connections 2024-03-28T23:34:14.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.708+0000 7f44125c9640 1 --2- 172.21.15.161:0/658939474 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f440c0aad50 0x7f440c0ab1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:14.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.708+0000 7f44125c9640 1 --2- 172.21.15.161:0/658939474 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f440c0c2390 0x7f440c0a8d10 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:14.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.708+0000 7f44125c9640 1 -- 172.21.15.161:0/658939474 >> 172.21.15.161:0/658939474 conn(0x7f440c0c6670 msgr2=0x7f440c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:14.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.708+0000 7f44125c9640 1 -- 172.21.15.161:0/658939474 shutdown_connections 2024-03-28T23:34:14.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.708+0000 7f44125c9640 1 -- 172.21.15.161:0/658939474 wait complete. 2024-03-28T23:34:14.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.709+0000 7f44125c9640 1 Processor -- start 2024-03-28T23:34:14.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.709+0000 7f44125c9640 1 -- start start 2024-03-28T23:34:14.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.710+0000 7f44125c9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f440c0aad50 0x7f440c1579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:14.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.710+0000 7f44125c9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f440c0c2390 0x7f440c157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:14.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.710+0000 7f44125c9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f440c152a90 0x7f440c147b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:14.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.710+0000 7f44125c9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f440c0c0110 con 0x7f440c0c2390 2024-03-28T23:34:14.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.710+0000 7f44125c9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f440c155c30 con 0x7f440c0aad50 2024-03-28T23:34:14.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.710+0000 7f44125c9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f440c0c0290 con 0x7f440c152a90 2024-03-28T23:34:14.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.710+0000 7f440b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f440c0c2390 0x7f440c157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:14.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.710+0000 7f440b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f440c0c2390 0x7f440c157f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45128/0 (socket says 172.21.15.161:45128) 2024-03-28T23:34:14.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.710+0000 7f440b7fe640 1 -- 172.21.15.161:0/2105223776 learned_addr learned my addr 172.21.15.161:0/2105223776 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:14.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.710+0000 7f4410b66640 1 --2- 172.21.15.161:0/2105223776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f440c152a90 0x7f440c147b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:14.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.710+0000 7f440bfff640 1 --2- 172.21.15.161:0/2105223776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f440c0aad50 0x7f440c1579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:14.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.711+0000 7f440b7fe640 1 -- 172.21.15.161:0/2105223776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f440c152a90 msgr2=0x7f440c147b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:14.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.711+0000 7f440b7fe640 1 --2- 172.21.15.161:0/2105223776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f440c152a90 0x7f440c147b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:14.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.711+0000 7f440b7fe640 1 -- 172.21.15.161:0/2105223776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f440c0aad50 msgr2=0x7f440c1579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:14.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.711+0000 7f440b7fe640 1 --2- 172.21.15.161:0/2105223776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f440c0aad50 0x7f440c1579d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:14.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.711+0000 7f440b7fe640 1 -- 172.21.15.161:0/2105223776 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f440c1566a0 con 0x7f440c0c2390 2024-03-28T23:34:14.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.711+0000 7f440b7fe640 1 --2- 172.21.15.161:0/2105223776 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f440c0c2390 0x7f440c157f10 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f4400000c00 tx=0x7f44000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:14.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.711+0000 7f4408ff9640 1 -- 172.21.15.161:0/2105223776 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4400040070 con 0x7f440c0c2390 2024-03-28T23:34:14.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.711+0000 7f4408ff9640 1 -- 172.21.15.161:0/2105223776 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44000399f0 con 0x7f440c0c2390 2024-03-28T23:34:14.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.711+0000 7f4408ff9640 1 -- 172.21.15.161:0/2105223776 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4400035040 con 0x7f440c0c2390 2024-03-28T23:34:14.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.711+0000 7f44125c9640 1 -- 172.21.15.161:0/2105223776 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f440c156980 con 0x7f440c0c2390 2024-03-28T23:34:14.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.711+0000 7f44125c9640 1 -- 172.21.15.161:0/2105223776 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f440c14efc0 con 0x7f440c0c2390 2024-03-28T23:34:14.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.712+0000 7f44125c9640 1 -- 172.21.15.161:0/2105223776 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f440c06e570 con 0x7f440c0c2390 2024-03-28T23:34:14.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.712+0000 7f4408ff9640 1 -- 172.21.15.161:0/2105223776 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f440004b060 con 0x7f440c0c2390 2024-03-28T23:34:14.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.713+0000 7f4408ff9640 1 --2- 172.21.15.161:0/2105223776 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f43f806b4d0 0x7f43f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:14.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.713+0000 7f4408ff9640 1 -- 172.21.15.161:0/2105223776 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7f44000abaa0 con 0x7f440c0c2390 2024-03-28T23:34:14.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.713+0000 7f440bfff640 1 --2- 172.21.15.161:0/2105223776 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f43f806b4d0 0x7f43f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:14.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.713+0000 7f440bfff640 1 --2- 172.21.15.161:0/2105223776 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f43f806b4d0 0x7f43f806d990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f43fc000f60 tx=0x7f43fc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:14.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.717+0000 7f4408ff9640 1 -- 172.21.15.161:0/2105223776 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f440c06e570 con 0x7f440c0c2390 2024-03-28T23:34:14.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.925+0000 7f44125c9640 1 -- 172.21.15.161:0/2105223776 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f440c0c2b50 con 0x7f440c0c2390 2024-03-28T23:34:14.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.926+0000 7f4408ff9640 1 -- 172.21.15.161:0/2105223776 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+5202 (secure 0 0 0) 0x7f440c0c2b50 con 0x7f440c0c2390 2024-03-28T23:34:14.928 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:14.928 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":100,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/1610475969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1610475969},{"type":"v1","addr":"172.21.15.161:6835","nonce":1610475969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":6892,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/2684760895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2684760895},{"type":"v1","addr":"172.21.15.161:6837","nonce":2684760895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":99,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T23:33:50.326492+0000","modified":"2024-03-28T23:34:10.064154+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6203":{"gid":6203,"name":"b","rank":0,"incarnation":92,"state":"up:active","state_seq":8,"addr":"172.21.15.170:6834/1096504848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1096504848},{"type":"v1","addr":"172.21.15.170:6834","nonce":1096504848}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6206":{"gid":6206,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":7,"addr":"172.21.15.170:6835/3351418222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3351418222},{"type":"v1","addr":"172.21.15.170:6835","nonce":3351418222}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6203,"qdb_cluster":[6203]},"id":8}]} 2024-03-28T23:34:14.928 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 100 2024-03-28T23:34:14.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.928+0000 7f44125c9640 1 -- 172.21.15.161:0/2105223776 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f43f806b4d0 msgr2=0x7f43f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:14.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.928+0000 7f44125c9640 1 --2- 172.21.15.161:0/2105223776 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f43f806b4d0 0x7f43f806d990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f43fc000f60 tx=0x7f43fc009040 comp rx=0 tx=0).stop 2024-03-28T23:34:14.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.928+0000 7f44125c9640 1 -- 172.21.15.161:0/2105223776 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f440c0c2390 msgr2=0x7f440c157f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:14.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.928+0000 7f44125c9640 1 --2- 172.21.15.161:0/2105223776 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f440c0c2390 0x7f440c157f10 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f4400000c00 tx=0x7f44000046c0 comp rx=0 tx=0).stop 2024-03-28T23:34:14.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.928+0000 7f44125c9640 1 -- 172.21.15.161:0/2105223776 shutdown_connections 2024-03-28T23:34:14.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.928+0000 7f44125c9640 1 --2- 172.21.15.161:0/2105223776 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f43f806b4d0 0x7f43f806d990 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:14.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.928+0000 7f44125c9640 1 --2- 172.21.15.161:0/2105223776 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f440c0c2390 0x7f440c157f10 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:14.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.928+0000 7f44125c9640 1 --2- 172.21.15.161:0/2105223776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f440c152a90 0x7f440c147b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:14.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.928+0000 7f44125c9640 1 --2- 172.21.15.161:0/2105223776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f440c0aad50 0x7f440c1579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:14.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.928+0000 7f44125c9640 1 -- 172.21.15.161:0/2105223776 >> 172.21.15.161:0/2105223776 conn(0x7f440c0c6670 msgr2=0x7f440c06b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:14.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.928+0000 7f44125c9640 1 -- 172.21.15.161:0/2105223776 shutdown_connections 2024-03-28T23:34:14.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:14.928+0000 7f44125c9640 1 -- 172.21.15.161:0/2105223776 wait complete. 2024-03-28T23:34:15.343 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-28T23:33:50.326492+0000', 'modified': '2024-03-28T23:34:10.064154+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6203}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6203': {'gid': 6203, 'name': 'b', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.170:6834/1096504848', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 1096504848}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 1096504848}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_6206': {'gid': 6206, 'name': 'd', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 7, 'addr': '172.21.15.170:6835/3351418222', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 3351418222}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 3351418222}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6203, 'qdb_cluster': [6203]} 2024-03-28T23:34:15.343 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:34:15.344 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T23:34:15.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.446+0000 7f2fbd4f4640 1 Processor -- start 2024-03-28T23:34:15.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.446+0000 7f2fbd4f4640 1 -- start start 2024-03-28T23:34:15.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.447+0000 7f2fbd4f4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2fb806ce80 0x7f2fb80aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.447+0000 7f2fbd4f4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2fb80a9680 0x7f2fb80aab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.447+0000 7f2fbd4f4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fb814c250 con 0x7f2fb814be70 2024-03-28T23:34:15.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.447+0000 7f2fbd4f4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2fb81529c0 con 0x7f2fb80a9680 2024-03-28T23:34:15.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.447+0000 7f2fbd4f4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f2fb80a9dc0 con 0x7f2fb806ce80 2024-03-28T23:34:15.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.447+0000 7f2fb77fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2fb80a9680 0x7f2fb80aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.447+0000 7f2fb6ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f2fb814be70 0x7f2fb80ab1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52188/0 (socket says 172.21.15.161:52188) 2024-03-28T23:34:15.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.447+0000 7f2fb6ffd640 1 -- 172.21.15.161:0/2626189176 learned_addr learned my addr 172.21.15.161:0/2626189176 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:15.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.447+0000 7f2fb67fc640 1 --2- 172.21.15.161:0/2626189176 >> v2:172.21.15.170:3300/0 conn(0x7f2fb806ce80 0x7f2fb80aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.447+0000 7f2fb77fe640 1 -- 172.21.15.161:0/2626189176 >> v2:172.21.15.170:3300/0 conn(0x7f2fb806ce80 msgr2=0x7f2fb80aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:15.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.447+0000 7f2fb77fe640 1 --2- 172.21.15.161:0/2626189176 >> v2:172.21.15.170:3300/0 conn(0x7f2fb806ce80 0x7f2fb80aa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.447+0000 7f2fb77fe640 1 -- 172.21.15.161:0/2626189176 >> v1:172.21.15.161:6789/0 conn(0x7f2fb814be70 legacy=0x7f2fb80ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:15.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.447+0000 7f2fb77fe640 1 -- 172.21.15.161:0/2626189176 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb80aa0a0 con 0x7f2fb80a9680 2024-03-28T23:34:15.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.447+0000 7f2fb5ffb640 1 -- 172.21.15.161:0/2626189176 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 206040513 0 0) 0x7f2fb814c250 con 0x7f2fb814be70 2024-03-28T23:34:15.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.448+0000 7f2fb77fe640 1 --2- 172.21.15.161:0/2626189176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fb80a9680 0x7f2fb80aab00 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f2fb806f540 tx=0x7f2fa802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b70389873668e096 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:15.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.448+0000 7f2fb5ffb640 1 -- 172.21.15.161:0/2626189176 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8038070 con 0x7f2fb80a9680 2024-03-28T23:34:15.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.448+0000 7f2fb5ffb640 1 -- 172.21.15.161:0/2626189176 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fa8031710 con 0x7f2fb80a9680 2024-03-28T23:34:15.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.448+0000 7f2fb5ffb640 1 -- 172.21.15.161:0/2626189176 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8031ae0 con 0x7f2fb80a9680 2024-03-28T23:34:15.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.449+0000 7f2fbd4f4640 1 -- 172.21.15.161:0/2626189176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fb80a9680 msgr2=0x7f2fb80aab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:15.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.449+0000 7f2fbd4f4640 1 --2- 172.21.15.161:0/2626189176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fb80a9680 0x7f2fb80aab00 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f2fb806f540 tx=0x7f2fa802b7b0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.449+0000 7f2fbd4f4640 1 -- 172.21.15.161:0/2626189176 shutdown_connections 2024-03-28T23:34:15.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.449+0000 7f2fbd4f4640 1 --2- 172.21.15.161:0/2626189176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fb80a9680 0x7f2fb80aab00 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.449+0000 7f2fbd4f4640 1 --2- 172.21.15.161:0/2626189176 >> v2:172.21.15.170:3300/0 conn(0x7f2fb806ce80 0x7f2fb80aa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.449+0000 7f2fbd4f4640 1 -- 172.21.15.161:0/2626189176 >> 172.21.15.161:0/2626189176 conn(0x7f2fb80c6670 msgr2=0x7f2fb80c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:15.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.449+0000 7f2fbd4f4640 1 -- 172.21.15.161:0/2626189176 shutdown_connections 2024-03-28T23:34:15.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.449+0000 7f2fbd4f4640 1 -- 172.21.15.161:0/2626189176 wait complete. 2024-03-28T23:34:15.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.450+0000 7f2fbd4f4640 1 Processor -- start 2024-03-28T23:34:15.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.450+0000 7f2fbd4f4640 1 -- start start 2024-03-28T23:34:15.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.451+0000 7f2fbd4f4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fb806ce80 0x7f2fb8071a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.451+0000 7f2fbd4f4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fb80a9680 0x7f2fb8143950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.451+0000 7f2fbd4f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fb814be70 0x7f2fb8143e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.451+0000 7f2fbd4f4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb80aba30 con 0x7f2fb806ce80 2024-03-28T23:34:15.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.451+0000 7f2fbd4f4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb80ab8b0 con 0x7f2fb80a9680 2024-03-28T23:34:15.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.451+0000 7f2fbd4f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2fb806b150 con 0x7f2fb814be70 2024-03-28T23:34:15.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.451+0000 7f2fb6ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fb806ce80 0x7f2fb8071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.451+0000 7f2fb6ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fb806ce80 0x7f2fb8071a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45142/0 (socket says 172.21.15.161:45142) 2024-03-28T23:34:15.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.451+0000 7f2fb6ffd640 1 -- 172.21.15.161:0/402334436 learned_addr learned my addr 172.21.15.161:0/402334436 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:15.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.451+0000 7f2fb67fc640 1 --2- 172.21.15.161:0/402334436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fb80a9680 0x7f2fb8143950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.451+0000 7f2fb77fe640 1 --2- 172.21.15.161:0/402334436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fb814be70 0x7f2fb8143e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.451+0000 7f2fb6ffd640 1 -- 172.21.15.161:0/402334436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fb814be70 msgr2=0x7f2fb8143e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:15.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.451+0000 7f2fb6ffd640 1 --2- 172.21.15.161:0/402334436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fb814be70 0x7f2fb8143e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.451+0000 7f2fb6ffd640 1 -- 172.21.15.161:0/402334436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fb80a9680 msgr2=0x7f2fb8143950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:15.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.451+0000 7f2fb6ffd640 1 --2- 172.21.15.161:0/402334436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fb80a9680 0x7f2fb8143950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.451+0000 7f2fb6ffd640 1 -- 172.21.15.161:0/402334436 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb81e9b50 con 0x7f2fb806ce80 2024-03-28T23:34:15.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.451+0000 7f2fb6ffd640 1 --2- 172.21.15.161:0/402334436 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fb806ce80 0x7f2fb8071a90 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f2fb000c7c0 tx=0x7f2fb000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:15.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.451+0000 7f2f9f7fe640 1 -- 172.21.15.161:0/402334436 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0018070 con 0x7f2fb806ce80 2024-03-28T23:34:15.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.451+0000 7f2f9f7fe640 1 -- 172.21.15.161:0/402334436 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fb000a640 con 0x7f2fb806ce80 2024-03-28T23:34:15.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.451+0000 7f2fbd4f4640 1 -- 172.21.15.161:0/402334436 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fb81558f0 con 0x7f2fb806ce80 2024-03-28T23:34:15.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.451+0000 7f2f9f7fe640 1 -- 172.21.15.161:0/402334436 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb000aa30 con 0x7f2fb806ce80 2024-03-28T23:34:15.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.451+0000 7f2fbd4f4640 1 -- 172.21.15.161:0/402334436 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fb814f0a0 con 0x7f2fb806ce80 2024-03-28T23:34:15.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.453+0000 7f2f9f7fe640 1 -- 172.21.15.161:0/402334436 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2fb001b070 con 0x7f2fb806ce80 2024-03-28T23:34:15.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.453+0000 7f2f9f7fe640 1 --2- 172.21.15.161:0/402334436 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2fa406b4d0 0x7f2fa406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.454+0000 7f2f9f7fe640 1 -- 172.21.15.161:0/402334436 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7f2fb00898d0 con 0x7f2fb806ce80 2024-03-28T23:34:15.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.454+0000 7f2fb67fc640 1 --2- 172.21.15.161:0/402334436 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2fa406b4d0 0x7f2fa406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.454+0000 7f2fb67fc640 1 --2- 172.21.15.161:0/402334436 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2fa406b4d0 0x7f2fa406d990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f2fb80c08b0 tx=0x7f2fac009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:15.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.454+0000 7f2fc01b5740 1 Processor -- start 2024-03-28T23:34:15.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.454+0000 7f2fc01b5740 1 -- start start 2024-03-28T23:34:15.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.455+0000 7f2fc01b5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563ef3c6db0 0x5563ef3c4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.455+0000 7f2fc01b5740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5563ef3c7190 0x5563ef3c4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.455+0000 7f2fc01b5740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef3c5310 0x5563ef3c14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.455+0000 7f2fc01b5740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5563ef32aaf0 con 0x5563ef3c5310 2024-03-28T23:34:15.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.455+0000 7f2fc01b5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5563ef2d39d0 con 0x5563ef3c6db0 2024-03-28T23:34:15.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.455+0000 7f2fc01b5740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x5563ef326b90 con 0x5563ef3c7190 2024-03-28T23:34:15.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.455+0000 7f2fb77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef3c5310 0x5563ef3c14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.455+0000 7f2fb67fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5563ef3c7190 0x5563ef3c4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.455+0000 7f2fb77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef3c5310 0x5563ef3c14e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45154/0 (socket says 172.21.15.161:45154) 2024-03-28T23:34:15.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.455+0000 7f2fb77fe640 1 -- 172.21.15.161:0/3440305873 learned_addr learned my addr 172.21.15.161:0/3440305873 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:15.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.455+0000 7f2fb6ffd640 1 --2- 172.21.15.161:0/3440305873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563ef3c6db0 0x5563ef3c4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.455+0000 7f2fb77fe640 1 -- 172.21.15.161:0/3440305873 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5563ef3c7190 msgr2=0x5563ef3c4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:15.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.455+0000 7f2fb77fe640 1 --2- 172.21.15.161:0/3440305873 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5563ef3c7190 0x5563ef3c4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.455+0000 7f2fb77fe640 1 -- 172.21.15.161:0/3440305873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563ef3c6db0 msgr2=0x5563ef3c4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:15.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.455+0000 7f2fb77fe640 1 --2- 172.21.15.161:0/3440305873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563ef3c6db0 0x5563ef3c4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.455+0000 7f2fb77fe640 1 -- 172.21.15.161:0/3440305873 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563ef03f2d0 con 0x5563ef3c5310 2024-03-28T23:34:15.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.455+0000 7f2fb67fc640 1 --2- 172.21.15.161:0/3440305873 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5563ef3c7190 0x5563ef3c4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:34:15.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.455+0000 7f2fb6ffd640 1 --2- 172.21.15.161:0/3440305873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563ef3c6db0 0x5563ef3c4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:34:15.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.456+0000 7f2fb77fe640 1 --2- 172.21.15.161:0/3440305873 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef3c5310 0x5563ef3c14e0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8009890 tx=0x7f2fa8030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=685c8846047ce987 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:15.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.456+0000 7f2f9cff9640 1 -- 172.21.15.161:0/3440305873 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8038040 con 0x5563ef3c5310 2024-03-28T23:34:15.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.456+0000 7f2f9cff9640 1 -- 172.21.15.161:0/3440305873 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fa8031950 con 0x5563ef3c5310 2024-03-28T23:34:15.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.456+0000 7f2f9cff9640 1 -- 172.21.15.161:0/3440305873 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8031d40 con 0x5563ef3c5310 2024-03-28T23:34:15.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.456+0000 7f2fc01b5740 1 -- 172.21.15.161:0/3440305873 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef3c5310 msgr2=0x5563ef3c14e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:15.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.456+0000 7f2fc01b5740 1 --2- 172.21.15.161:0/3440305873 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef3c5310 0x5563ef3c14e0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8009890 tx=0x7f2fa8030000 comp rx=0 tx=0).stop 2024-03-28T23:34:15.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.457+0000 7f2fc01b5740 1 -- 172.21.15.161:0/3440305873 shutdown_connections 2024-03-28T23:34:15.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.457+0000 7f2fc01b5740 1 --2- 172.21.15.161:0/3440305873 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef3c5310 0x5563ef3c14e0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.457+0000 7f2fc01b5740 1 --2- 172.21.15.161:0/3440305873 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5563ef3c7190 0x5563ef3c4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.457+0000 7f2fc01b5740 1 --2- 172.21.15.161:0/3440305873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563ef3c6db0 0x5563ef3c4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.457+0000 7f2fc01b5740 1 -- 172.21.15.161:0/3440305873 >> 172.21.15.161:0/3440305873 conn(0x5563ef3c3630 msgr2=0x5563ef3e8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:15.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.457+0000 7f2fc01b5740 1 -- 172.21.15.161:0/3440305873 shutdown_connections 2024-03-28T23:34:15.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.457+0000 7f2fc01b5740 1 -- 172.21.15.161:0/3440305873 wait complete. 2024-03-28T23:34:15.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.457+0000 7f2fc01b5740 1 -- start start 2024-03-28T23:34:15.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.457+0000 7f2fc01b5740 1 Processor -- start 2024-03-28T23:34:15.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.458+0000 7f2fc01b5740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5563ef3c6db0 0x5563ef3e3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.458+0000 7f2fc01b5740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef3c7190 0x5563ef416db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.458+0000 7f2fc01b5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563ef4172f0 0x5563ef3ecdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.458+0000 7f2fc01b5740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5563ef3e9090 con 0x5563ef3c7190 2024-03-28T23:34:15.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.458+0000 7f2fc01b5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5563ef2e6f70 con 0x5563ef4172f0 2024-03-28T23:34:15.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.458+0000 7f2fc01b5740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x5563ef3e8ad0 con 0x5563ef3c6db0 2024-03-28T23:34:15.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.458+0000 7f2fc01b5740 10 client.0 _ll_register_callbacks cb 0x5563ef03eac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:34:15.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.458+0000 7f2fb67fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef3c7190 0x5563ef416db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.458+0000 7f2fb67fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef3c7190 0x5563ef416db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45164/0 (socket says 172.21.15.161:45164) 2024-03-28T23:34:15.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.458+0000 7f2fb67fc640 1 -- 172.21.15.161:0/2804552486 learned_addr learned my addr 172.21.15.161:0/2804552486 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:15.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.458+0000 7f2fb6ffd640 1 --2- 172.21.15.161:0/2804552486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5563ef3c6db0 0x5563ef3e3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.458+0000 7f2fb77fe640 1 --2- 172.21.15.161:0/2804552486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563ef4172f0 0x5563ef3ecdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.458+0000 7f2fb67fc640 1 -- 172.21.15.161:0/2804552486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5563ef3c6db0 msgr2=0x5563ef3e3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:15.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.458+0000 7f2fb67fc640 1 --2- 172.21.15.161:0/2804552486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5563ef3c6db0 0x5563ef3e3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.458+0000 7f2fb67fc640 1 -- 172.21.15.161:0/2804552486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563ef4172f0 msgr2=0x5563ef3ecdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:15.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.458+0000 7f2fb67fc640 1 --2- 172.21.15.161:0/2804552486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563ef4172f0 0x5563ef3ecdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.458+0000 7f2fb67fc640 1 -- 172.21.15.161:0/2804552486 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563ef1096b0 con 0x5563ef3c7190 2024-03-28T23:34:15.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.458+0000 7f2fb77fe640 1 --2- 172.21.15.161:0/2804552486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563ef4172f0 0x5563ef3ecdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:34:15.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.459+0000 7f2fb67fc640 1 --2- 172.21.15.161:0/2804552486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef3c7190 0x5563ef416db0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8149f10 tx=0x7f2fac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae58abd84593f476 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:15.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.459+0000 7f2f95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:34:15.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.459+0000 7f2f95ffb640 1 -- 172.21.15.161:0/2804552486 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fac027070 con 0x5563ef3c7190 2024-03-28T23:34:15.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.459+0000 7f2f95ffb640 1 -- 172.21.15.161:0/2804552486 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7f2fac014440 con 0x5563ef3c7190 2024-03-28T23:34:15.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.459+0000 7f2fc01b5740 1 -- 172.21.15.161:0/2804552486 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fa8031950 con 0x5563ef3c7190 2024-03-28T23:34:15.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.459+0000 7f2f95ffb640 1 -- 172.21.15.161:0/2804552486 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f2fac014680 con 0x5563ef3c7190 2024-03-28T23:34:15.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.459+0000 7f2fc01b5740 10 client.7779 fetch_fsmap learned FSMap version 100 2024-03-28T23:34:15.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.459+0000 7f2fc01b5740 1 -- 172.21.15.161:0/2804552486 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=2+}) -- 0x5563ef4165b0 con 0x5563ef3c7190 2024-03-28T23:34:15.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.459+0000 7f2f95ffb640 1 -- 172.21.15.161:0/2804552486 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fa8031950 con 0x5563ef3c7190 2024-03-28T23:34:15.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.459+0000 7f2f95ffb640 1 -- 172.21.15.161:0/2804552486 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 100) ==== 3482+0+0 (secure 0 0 0) 0x7f2fac035080 con 0x5563ef3c7190 2024-03-28T23:34:15.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.459+0000 7f2fc01b5740 10 client.7779 fetch_fsmap finished waiting for FSMap version 100 2024-03-28T23:34:15.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.459+0000 7f2fc01b5740 10 client.7779 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:34:15.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.459+0000 7f2fc01b5740 20 client.7779 populate_metadata read hostname 'smithi161' 2024-03-28T23:34:15.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.459+0000 7f2fc01b5740 1 --2- 172.21.15.161:0/2804552486 >> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] conn(0x5563ef413570 0x5563ef413950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.459+0000 7f2fc01b5740 4 client.7779 mds_command: new command op to 6203 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:34:15.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.459+0000 7f2fc01b5740 1 -- 172.21.15.161:0/2804552486 --> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563ef338030 con 0x5563ef413570 2024-03-28T23:34:15.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.459+0000 7f2fb6ffd640 1 --2- 172.21.15.161:0/2804552486 >> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] conn(0x5563ef413570 0x5563ef413950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.460+0000 7f2f95ffb640 10 client.7779 ms_handle_connect on v2:172.21.15.170:6832/1096504848 2024-03-28T23:34:15.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.460+0000 7f2fb6ffd640 1 --2- 172.21.15.161:0/2804552486 >> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] conn(0x5563ef413570 0x5563ef413950 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=259f4d8b64bd425 server_cookie=57e62841af22c9d7 in_seq=0 out_seq=0 2024-03-28T23:34:15.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.460+0000 7f2f95ffb640 1 -- 172.21.15.161:0/2804552486 <== mds.0 v2:172.21.15.170:6832/1096504848 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5563ef338030 con 0x5563ef413570 2024-03-28T23:34:15.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.460+0000 7f2f95ffb640 10 client.7779 handle_command_reply: tid=0 2024-03-28T23:34:15.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.460+0000 7f2fc01b5740 1 client.7779 shutdown 2024-03-28T23:34:15.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.460+0000 7f2fc01b5740 1 -- 172.21.15.161:0/2804552486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef3c7190 msgr2=0x5563ef416db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:15.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.460+0000 7f2fc01b5740 1 --2- 172.21.15.161:0/2804552486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef3c7190 0x5563ef416db0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8149f10 tx=0x7f2fac0046c0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.460+0000 7f2fc01b5740 1 -- 172.21.15.161:0/2804552486 shutdown_connections 2024-03-28T23:34:15.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.460+0000 7f2fc01b5740 1 --2- 172.21.15.161:0/2804552486 >> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] conn(0x5563ef413570 0x5563ef413950 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.460+0000 7f2fc01b5740 1 --2- 172.21.15.161:0/2804552486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef3c7190 0x5563ef416db0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.460+0000 7f2fc01b5740 1 --2- 172.21.15.161:0/2804552486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563ef4172f0 0x5563ef3ecdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.460+0000 7f2f95ffb640 0 client.7779 ms_handle_reset on v2:172.21.15.170:6832/1096504848 2024-03-28T23:34:15.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.460+0000 7f2fc01b5740 1 --2- 172.21.15.161:0/2804552486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5563ef3c6db0 0x5563ef3e3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.460+0000 7f2fc01b5740 1 -- 172.21.15.161:0/2804552486 >> 172.21.15.161:0/2804552486 conn(0x5563ef3c5310 msgr2=0x5563ef3c56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:15.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.461+0000 7f2fc01b5740 1 -- 172.21.15.161:0/2804552486 shutdown_connections 2024-03-28T23:34:15.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.461+0000 7f2fc01b5740 1 -- 172.21.15.161:0/2804552486 wait complete. 2024-03-28T23:34:15.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.461+0000 7f2fc01b5740 20 client.7779 trim_cache size 0 max 16384 2024-03-28T23:34:15.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.478+0000 7f2fc01b5740 1 Processor -- start 2024-03-28T23:34:15.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.478+0000 7f2fc01b5740 1 -- start start 2024-03-28T23:34:15.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.479+0000 7f2fc01b5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563ef3c6db0 0x5563ef4270a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.479+0000 7f2fc01b5740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef413950 0x5563ef413d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.479+0000 7f2fc01b5740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5563ef414800 0x5563ef414c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.479+0000 7f2fc01b5740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb81f5de0 con 0x5563ef413950 2024-03-28T23:34:15.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.479+0000 7f2fc01b5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb81ecd50 con 0x5563ef3c6db0 2024-03-28T23:34:15.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.479+0000 7f2fc01b5740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2fb81f4880 con 0x5563ef414800 2024-03-28T23:34:15.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.479+0000 7f2fb67fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef413950 0x5563ef413d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.479+0000 7f2fb67fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef413950 0x5563ef413d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45176/0 (socket says 172.21.15.161:45176) 2024-03-28T23:34:15.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.479+0000 7f2fb67fc640 1 -- 172.21.15.161:0/565388386 learned_addr learned my addr 172.21.15.161:0/565388386 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:15.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.479+0000 7f2fb77fe640 1 --2- 172.21.15.161:0/565388386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5563ef414800 0x5563ef414c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.479+0000 7f2fb6ffd640 1 --2- 172.21.15.161:0/565388386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563ef3c6db0 0x5563ef4270a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.479+0000 7f2fb67fc640 1 -- 172.21.15.161:0/565388386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5563ef414800 msgr2=0x5563ef414c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:15.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.479+0000 7f2fb67fc640 1 --2- 172.21.15.161:0/565388386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5563ef414800 0x5563ef414c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.479+0000 7f2fb67fc640 1 -- 172.21.15.161:0/565388386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563ef3c6db0 msgr2=0x5563ef4270a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:15.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.479+0000 7f2fb67fc640 1 --2- 172.21.15.161:0/565388386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563ef3c6db0 0x5563ef4270a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.479+0000 7f2fb67fc640 1 -- 172.21.15.161:0/565388386 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563ef3c8070 con 0x5563ef413950 2024-03-28T23:34:15.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.480+0000 7f2fb67fc640 1 --2- 172.21.15.161:0/565388386 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef413950 0x5563ef413d70 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f2fac009880 tx=0x7f2fac024000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=742ccbfd01572d8b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:15.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.480+0000 7f2f97fff640 1 -- 172.21.15.161:0/565388386 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fac027070 con 0x5563ef413950 2024-03-28T23:34:15.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.480+0000 7f2f97fff640 1 -- 172.21.15.161:0/565388386 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fac014820 con 0x5563ef413950 2024-03-28T23:34:15.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.480+0000 7f2f97fff640 1 -- 172.21.15.161:0/565388386 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fac039070 con 0x5563ef413950 2024-03-28T23:34:15.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.480+0000 7f2fc01b5740 1 -- 172.21.15.161:0/565388386 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef413950 msgr2=0x5563ef413d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:15.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.480+0000 7f2fc01b5740 1 --2- 172.21.15.161:0/565388386 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef413950 0x5563ef413d70 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f2fac009880 tx=0x7f2fac024000 comp rx=0 tx=0).stop 2024-03-28T23:34:15.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.480+0000 7f2fc01b5740 1 -- 172.21.15.161:0/565388386 shutdown_connections 2024-03-28T23:34:15.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.480+0000 7f2fc01b5740 1 --2- 172.21.15.161:0/565388386 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef413950 0x5563ef413d70 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.480+0000 7f2fc01b5740 1 --2- 172.21.15.161:0/565388386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5563ef414800 0x5563ef414c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.480+0000 7f2fc01b5740 1 --2- 172.21.15.161:0/565388386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563ef3c6db0 0x5563ef4270a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.480+0000 7f2fc01b5740 1 -- 172.21.15.161:0/565388386 >> 172.21.15.161:0/565388386 conn(0x5563ef3c5310 msgr2=0x5563ef3c16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:15.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.480+0000 7f2fc01b5740 1 -- 172.21.15.161:0/565388386 shutdown_connections 2024-03-28T23:34:15.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.480+0000 7f2fc01b5740 1 -- 172.21.15.161:0/565388386 wait complete. 2024-03-28T23:34:15.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.481+0000 7f2fc01b5740 1 -- start start 2024-03-28T23:34:15.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.481+0000 7f2fc01b5740 1 Processor -- start 2024-03-28T23:34:15.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2fc01b5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563ef3c6db0 0x5563ef3c5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2fc01b5740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5563ef413950 0x5563ef47ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2fc01b5740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef414800 0x5563ef47f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2fc01b5740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5563ef4133f0 con 0x5563ef414800 2024-03-28T23:34:15.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2fc01b5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5563ef3ee6f0 con 0x5563ef3c6db0 2024-03-28T23:34:15.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2fc01b5740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x5563ef4145e0 con 0x5563ef413950 2024-03-28T23:34:15.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2fc01b5740 10 client.0 _ll_register_callbacks cb 0x7f2fac004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:34:15.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2fb6ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563ef3c6db0 0x5563ef3c5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2fb77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef414800 0x5563ef47f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2fb67fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5563ef413950 0x5563ef47ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2fb6ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563ef3c6db0 0x5563ef3c5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:50060/0 (socket says 172.21.15.161:50060) 2024-03-28T23:34:15.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2fb6ffd640 1 -- 172.21.15.161:0/2223134954 learned_addr learned my addr 172.21.15.161:0/2223134954 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:15.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2fb67fc640 1 --2- 172.21.15.161:0/2223134954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5563ef413950 0x5563ef47ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:57920/0 (socket says 172.21.15.161:57920) 2024-03-28T23:34:15.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2fb6ffd640 1 -- 172.21.15.161:0/2223134954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5563ef413950 msgr2=0x5563ef47ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:15.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2fb6ffd640 1 --2- 172.21.15.161:0/2223134954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5563ef413950 0x5563ef47ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2fb6ffd640 1 -- 172.21.15.161:0/2223134954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef414800 msgr2=0x5563ef47f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:15.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2fb6ffd640 1 --2- 172.21.15.161:0/2223134954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef414800 0x5563ef47f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2fb6ffd640 1 -- 172.21.15.161:0/2223134954 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563ef3c4fb0 con 0x5563ef3c6db0 2024-03-28T23:34:15.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2fb77fe640 1 --2- 172.21.15.161:0/2223134954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef414800 0x5563ef47f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:34:15.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2fb6ffd640 1 --2- 172.21.15.161:0/2223134954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563ef3c6db0 0x5563ef3c5af0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0057cb0 tx=0x7f2fb0089040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f312d3ea65e60cc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:15.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2f95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T23:34:15.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2f95ffb640 1 -- 172.21.15.161:0/2223134954 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0060380 con 0x5563ef3c6db0 2024-03-28T23:34:15.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2f95ffb640 1 -- 172.21.15.161:0/2223134954 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7f2fb005e070 con 0x5563ef3c6db0 2024-03-28T23:34:15.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2fc01b5740 1 -- 172.21.15.161:0/2223134954 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fac014820 con 0x5563ef3c6db0 2024-03-28T23:34:15.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2f95ffb640 1 -- 172.21.15.161:0/2223134954 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f2fb0056370 con 0x5563ef3c6db0 2024-03-28T23:34:15.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2fc01b5740 10 client.7159 fetch_fsmap learned FSMap version 100 2024-03-28T23:34:15.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.482+0000 7f2fc01b5740 1 -- 172.21.15.161:0/2223134954 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=2+}) -- 0x5563ef410110 con 0x5563ef3c6db0 2024-03-28T23:34:15.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.483+0000 7f2f95ffb640 1 -- 172.21.15.161:0/2223134954 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fac014820 con 0x5563ef3c6db0 2024-03-28T23:34:15.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.483+0000 7f2f95ffb640 1 -- 172.21.15.161:0/2223134954 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 100) ==== 3482+0+0 (secure 0 0 0) 0x7f2fb0057400 con 0x5563ef3c6db0 2024-03-28T23:34:15.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.483+0000 7f2fc01b5740 10 client.7159 fetch_fsmap finished waiting for FSMap version 100 2024-03-28T23:34:15.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.483+0000 7f2fc01b5740 10 client.7159 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:34:15.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.483+0000 7f2fc01b5740 20 client.7159 populate_metadata read hostname 'smithi161' 2024-03-28T23:34:15.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.483+0000 7f2fc01b5740 1 --2- 172.21.15.161:0/2223134954 >> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] conn(0x5563ef4823a0 0x5563ef482780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.483+0000 7f2fc01b5740 4 client.7159 mds_command: new command op to 6203 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:34:15.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.483+0000 7f2fc01b5740 1 -- 172.21.15.161:0/2223134954 --> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] -- command(tid 0: {"prefix": "status"}) -- 0x5563ef332380 con 0x5563ef4823a0 2024-03-28T23:34:15.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.483+0000 7f2fb67fc640 1 --2- 172.21.15.161:0/2223134954 >> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] conn(0x5563ef4823a0 0x5563ef482780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.483+0000 7f2fb67fc640 1 --2- 172.21.15.161:0/2223134954 >> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] conn(0x5563ef4823a0 0x5563ef482780 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc23f7dfca411b85 server_cookie=702c91943b628c16 in_seq=0 out_seq=0 2024-03-28T23:34:15.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.483+0000 7f2f95ffb640 10 client.7159 ms_handle_connect on v2:172.21.15.170:6832/1096504848 2024-03-28T23:34:15.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.483+0000 7f2f95ffb640 1 -- 172.21.15.161:0/2223134954 <== mds.0 v2:172.21.15.170:6832/1096504848 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x5563ef332380 con 0x5563ef4823a0 2024-03-28T23:34:15.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.483+0000 7f2f95ffb640 10 client.7159 handle_command_reply: tid=0 2024-03-28T23:34:15.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.483+0000 7f2fc01b5740 1 client.7159 shutdown 2024-03-28T23:34:15.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.483+0000 7f2fc01b5740 1 -- 172.21.15.161:0/2223134954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563ef3c6db0 msgr2=0x5563ef3c5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:15.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.483+0000 7f2fc01b5740 1 --2- 172.21.15.161:0/2223134954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563ef3c6db0 0x5563ef3c5af0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0057cb0 tx=0x7f2fb0089040 comp rx=0 tx=0).stop 2024-03-28T23:34:15.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.483+0000 7f2fc01b5740 1 -- 172.21.15.161:0/2223134954 shutdown_connections 2024-03-28T23:34:15.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.483+0000 7f2fc01b5740 1 --2- 172.21.15.161:0/2223134954 >> [v2:172.21.15.170:6832/1096504848,v1:172.21.15.170:6834/1096504848] conn(0x5563ef4823a0 0x5563ef482780 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.483+0000 7f2fc01b5740 1 --2- 172.21.15.161:0/2223134954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5563ef414800 0x5563ef47f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.483+0000 7f2fc01b5740 1 --2- 172.21.15.161:0/2223134954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5563ef413950 0x5563ef47ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.483+0000 7f2fc01b5740 1 --2- 172.21.15.161:0/2223134954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5563ef3c6db0 0x5563ef3c5af0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.483+0000 7f2f95ffb640 0 client.7159 ms_handle_reset on v2:172.21.15.170:6832/1096504848 2024-03-28T23:34:15.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.483+0000 7f2fc01b5740 1 -- 172.21.15.161:0/2223134954 >> 172.21.15.161:0/2223134954 conn(0x5563ef3c5310 msgr2=0x5563ef414be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:15.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.484+0000 7f2fc01b5740 1 -- 172.21.15.161:0/2223134954 shutdown_connections 2024-03-28T23:34:15.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.484+0000 7f2fc01b5740 1 -- 172.21.15.161:0/2223134954 wait complete. 2024-03-28T23:34:15.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.484+0000 7f2fc01b5740 20 client.7159 trim_cache size 0 max 16384 2024-03-28T23:34:15.492 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:34:15.492 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:34:15.492 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:34:15.492 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 6203, 2024-03-28T23:34:15.492 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:34:15.492 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:34:15.492 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:34:15.493 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 13.449173565000001, 2024-03-28T23:34:15.493 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 99, 2024-03-28T23:34:15.493 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 110, 2024-03-28T23:34:15.493 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 110, 2024-03-28T23:34:15.493 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 29.593210979999999 2024-03-28T23:34:15.493 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:34:15.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.484+0000 7f2fbd4f4640 1 -- 172.21.15.161:0/402334436 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2fa406b4d0 msgr2=0x7f2fa406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:15.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.484+0000 7f2fbd4f4640 1 --2- 172.21.15.161:0/402334436 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2fa406b4d0 0x7f2fa406d990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f2fb80c08b0 tx=0x7f2fac009190 comp rx=0 tx=0).stop 2024-03-28T23:34:15.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.484+0000 7f2fbd4f4640 1 -- 172.21.15.161:0/402334436 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fb806ce80 msgr2=0x7f2fb8071a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:15.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.484+0000 7f2fbd4f4640 1 --2- 172.21.15.161:0/402334436 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fb806ce80 0x7f2fb8071a90 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f2fb000c7c0 tx=0x7f2fb000cc80 comp rx=0 tx=0).stop 2024-03-28T23:34:15.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.484+0000 7f2fbd4f4640 1 -- 172.21.15.161:0/402334436 shutdown_connections 2024-03-28T23:34:15.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.484+0000 7f2fbd4f4640 1 --2- 172.21.15.161:0/402334436 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2fa406b4d0 0x7f2fa406d990 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.484+0000 7f2fbd4f4640 1 --2- 172.21.15.161:0/402334436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fb814be70 0x7f2fb8143e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.484+0000 7f2fbd4f4640 1 --2- 172.21.15.161:0/402334436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fb80a9680 0x7f2fb8143950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.484+0000 7f2fbd4f4640 1 --2- 172.21.15.161:0/402334436 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2fb806ce80 0x7f2fb8071a90 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.484+0000 7f2fbd4f4640 1 -- 172.21.15.161:0/402334436 >> 172.21.15.161:0/402334436 conn(0x7f2fb80c6670 msgr2=0x7f2fb80c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:15.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.484+0000 7f2fbd4f4640 1 -- 172.21.15.161:0/402334436 shutdown_connections 2024-03-28T23:34:15.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.484+0000 7f2fbd4f4640 1 -- 172.21.15.161:0/402334436 wait complete. 2024-03-28T23:34:15.496 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 100, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:33:50.326492+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 99, "ever_allowed_features": 32, "explicitly_allowed_features": 32, "failed": [], "flags": 50, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": true, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6203": { "addr": "172.21.15.170:6834/1096504848", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 1096504848, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 1096504848, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6203, "incarnation": 92, "join_fscid": 8, "name": "b", "rank": 0, "state": "up:active", "state_seq": 8 }, "gid_6206": { "addr": "172.21.15.170:6835/3351418222", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 3351418222, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 3351418222, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6206, "incarnation": 0, "join_fscid": 8, "name": "d", "rank": 0, "state": "up:standby-replay", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-03-28T23:34:10.064154+0000", "qdb_cluster": [ 6203 ], "qdb_leader": 6203, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6203 } } } ], "standbys": [ { "addr": "172.21.15.161:6835/1610475969", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 1610475969, "type": "v2" }, { "addr": "172.21.15.161:6835", "nonce": 1610475969, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 100, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6377, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6837/2684760895", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6836", "nonce": 2684760895, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 2684760895, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 87, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6892, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:34:15.497 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 8, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 8, 'name': 'd', 'state': 'up:standby-replay'}] 2024-03-28T23:34:15.497 INFO:tasks.cephfs.test_failover:current = [{'gid': 6377, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6835/1610475969', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 1610475969}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 1610475969}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 100}, {'gid': 6203, 'name': 'b', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.170:6834/1096504848', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 1096504848}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 1096504848}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 6892, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6837/2684760895', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 2684760895}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 2684760895}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 87}, {'gid': 6206, 'name': 'd', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 7, 'addr': '172.21.15.170:6835/3351418222', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 3351418222}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 3351418222}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}] 2024-03-28T23:34:15.497 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-28T23:34:15.498 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-28T23:34:15.498 DEBUG:teuthology.orchestra.run.smithi161:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:34:15.549 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:34:15.587 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-28T23:34:15.588 INFO:tasks.cephfs.mount:Terminating background process 2024-03-28T23:34:15.588 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:34:15.588 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:34:15.603 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20793 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-28T23:34:15.603 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi161.front.sepia.ceph.com 2024-03-28T23:34:15.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:34:15.604 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-28T23:34:15.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:34:15.658 INFO:teuthology.orchestra.run.smithi161.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-28T23:34:15.658 INFO:tasks.cephfs.mount:Terminating background process 2024-03-28T23:34:15.658 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:34:15.658 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:34:15.673 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056374,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19690 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-28T23:34:15.673 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi170.front.sepia.ceph.com 2024-03-28T23:34:15.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:34:15.674 DEBUG:teuthology.orchestra.run.smithi170:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-28T23:34:15.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:34:15.728 INFO:teuthology.orchestra.run.smithi170.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-28T23:34:15.729 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:34:15.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.828+0000 7fa9af80b640 1 Processor -- start 2024-03-28T23:34:15.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.828+0000 7fa9af80b640 1 -- start start 2024-03-28T23:34:15.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.829+0000 7fa9af80b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9a8152890 0x7fa9a80c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.829+0000 7fa9af80b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa9a80c8ed0 0x7fa9a80c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.829+0000 7fa9af80b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa9a80c5950 0x7fa9a81553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.829+0000 7fa9af80b640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa9a80c5db0 con 0x7fa9a80c8ed0 2024-03-28T23:34:15.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.829+0000 7fa9af80b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fa9a80c8480 con 0x7fa9a80c5950 2024-03-28T23:34:15.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.829+0000 7fa9af80b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa9a8159640 con 0x7fa9a8152890 2024-03-28T23:34:15.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.829+0000 7fa9acda6640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa9a80c8ed0 0x7fa9a80c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.829+0000 7fa9acda6640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa9a80c8ed0 0x7fa9a80c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45188/0 (socket says 172.21.15.161:45188) 2024-03-28T23:34:15.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.829+0000 7fa9acda6640 1 -- 172.21.15.161:0/4110925661 learned_addr learned my addr 172.21.15.161:0/4110925661 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:15.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.829+0000 7fa9ad5a7640 1 --2- 172.21.15.161:0/4110925661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9a8152890 0x7fa9a80c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.829+0000 7fa9adda8640 1 --2- 172.21.15.161:0/4110925661 >> v2:172.21.15.170:3300/0 conn(0x7fa9a80c5950 0x7fa9a81553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.830+0000 7fa9acda6640 1 -- 172.21.15.161:0/4110925661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9a8152890 msgr2=0x7fa9a80c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:15.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.830+0000 7fa9acda6640 1 --2- 172.21.15.161:0/4110925661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9a8152890 0x7fa9a80c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.830+0000 7fa9acda6640 1 -- 172.21.15.161:0/4110925661 >> v2:172.21.15.170:3300/0 conn(0x7fa9a80c5950 msgr2=0x7fa9a81553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:15.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.830+0000 7fa9acda6640 1 --2- 172.21.15.161:0/4110925661 >> v2:172.21.15.170:3300/0 conn(0x7fa9a80c5950 0x7fa9a81553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.830+0000 7fa9acda6640 1 -- 172.21.15.161:0/4110925661 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9a8158a30 con 0x7fa9a80c8ed0 2024-03-28T23:34:15.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.830+0000 7fa9acda6640 1 --2- 172.21.15.161:0/4110925661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9a80c8ed0 0x7fa9a80c5410 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80c4e90 tx=0x7fa99802b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3774defd55e4e9f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:15.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.831+0000 7fa99ffff640 1 -- 172.21.15.161:0/4110925661 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa998038070 con 0x7fa9a80c8ed0 2024-03-28T23:34:15.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.831+0000 7fa99ffff640 1 -- 172.21.15.161:0/4110925661 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa998004030 con 0x7fa9a80c8ed0 2024-03-28T23:34:15.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.831+0000 7fa99ffff640 1 -- 172.21.15.161:0/4110925661 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa998004460 con 0x7fa9a80c8ed0 2024-03-28T23:34:15.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.831+0000 7fa9af80b640 1 -- 172.21.15.161:0/4110925661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9a80c8ed0 msgr2=0x7fa9a80c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:15.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.831+0000 7fa9af80b640 1 --2- 172.21.15.161:0/4110925661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9a80c8ed0 0x7fa9a80c5410 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80c4e90 tx=0x7fa99802b7c0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.831+0000 7fa9af80b640 1 -- 172.21.15.161:0/4110925661 shutdown_connections 2024-03-28T23:34:15.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.831+0000 7fa9af80b640 1 --2- 172.21.15.161:0/4110925661 >> v2:172.21.15.170:3300/0 conn(0x7fa9a80c5950 0x7fa9a81553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.831+0000 7fa9af80b640 1 --2- 172.21.15.161:0/4110925661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9a80c8ed0 0x7fa9a80c5410 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.831+0000 7fa9af80b640 1 --2- 172.21.15.161:0/4110925661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9a8152890 0x7fa9a80c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.831+0000 7fa9af80b640 1 -- 172.21.15.161:0/4110925661 >> 172.21.15.161:0/4110925661 conn(0x7fa9a80ac620 msgr2=0x7fa9a80aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:15.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.831+0000 7fa9af80b640 1 -- 172.21.15.161:0/4110925661 shutdown_connections 2024-03-28T23:34:15.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.831+0000 7fa9af80b640 1 -- 172.21.15.161:0/4110925661 wait complete. 2024-03-28T23:34:15.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.832+0000 7fa9af80b640 1 Processor -- start 2024-03-28T23:34:15.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.832+0000 7fa9af80b640 1 -- start start 2024-03-28T23:34:15.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.833+0000 7fa9af80b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9a80c5950 0x7fa9a8142730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.833+0000 7fa9af80b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9a80c8ed0 0x7fa9a8144380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.833+0000 7fa9af80b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9a8152890 0x7fa9a81448c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.833+0000 7fa9af80b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a81592d0 con 0x7fa9a8152890 2024-03-28T23:34:15.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.833+0000 7fa9af80b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a8158ff0 con 0x7fa9a80c5950 2024-03-28T23:34:15.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.833+0000 7fa9af80b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa9a8152c70 con 0x7fa9a80c8ed0 2024-03-28T23:34:15.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.833+0000 7fa9adda8640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9a8152890 0x7fa9a81448c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.833+0000 7fa9adda8640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9a8152890 0x7fa9a81448c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45190/0 (socket says 172.21.15.161:45190) 2024-03-28T23:34:15.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.833+0000 7fa9adda8640 1 -- 172.21.15.161:0/2629720080 learned_addr learned my addr 172.21.15.161:0/2629720080 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:15.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.833+0000 7fa9ad5a7640 1 --2- 172.21.15.161:0/2629720080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9a80c5950 0x7fa9a8142730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.833+0000 7fa9acda6640 1 --2- 172.21.15.161:0/2629720080 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9a80c8ed0 0x7fa9a8144380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.833+0000 7fa9adda8640 1 -- 172.21.15.161:0/2629720080 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9a80c8ed0 msgr2=0x7fa9a8144380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:15.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.833+0000 7fa9adda8640 1 --2- 172.21.15.161:0/2629720080 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9a80c8ed0 0x7fa9a8144380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.833+0000 7fa9adda8640 1 -- 172.21.15.161:0/2629720080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9a80c5950 msgr2=0x7fa9a8142730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:15.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.833+0000 7fa9adda8640 1 --2- 172.21.15.161:0/2629720080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9a80c5950 0x7fa9a8142730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:15.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.833+0000 7fa9adda8640 1 -- 172.21.15.161:0/2629720080 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9a81e8550 con 0x7fa9a8152890 2024-03-28T23:34:15.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.833+0000 7fa9adda8640 1 --2- 172.21.15.161:0/2629720080 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9a8152890 0x7fa9a81448c0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000e7f0 tx=0x7fa9a000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:15.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.834+0000 7fa99dffb640 1 -- 172.21.15.161:0/2629720080 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0017070 con 0x7fa9a8152890 2024-03-28T23:34:15.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.834+0000 7fa99dffb640 1 -- 172.21.15.161:0/2629720080 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9a000c730 con 0x7fa9a8152890 2024-03-28T23:34:15.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.834+0000 7fa9af80b640 1 -- 172.21.15.161:0/2629720080 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9a8147d90 con 0x7fa9a8152890 2024-03-28T23:34:15.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.834+0000 7fa99dffb640 1 -- 172.21.15.161:0/2629720080 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a000cb00 con 0x7fa9a8152890 2024-03-28T23:34:15.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.834+0000 7fa9af80b640 1 -- 172.21.15.161:0/2629720080 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9a8142c70 con 0x7fa9a8152890 2024-03-28T23:34:15.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.835+0000 7fa9af80b640 1 -- 172.21.15.161:0/2629720080 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9a8144f60 con 0x7fa9a8152890 2024-03-28T23:34:15.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.835+0000 7fa99dffb640 1 -- 172.21.15.161:0/2629720080 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa9a001a070 con 0x7fa9a8152890 2024-03-28T23:34:15.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.836+0000 7fa99dffb640 1 --2- 172.21.15.161:0/2629720080 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa98c06b4d0 0x7fa98c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:15.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.836+0000 7fa99dffb640 1 -- 172.21.15.161:0/2629720080 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7fa9a008a480 con 0x7fa9a8152890 2024-03-28T23:34:15.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.836+0000 7fa9ad5a7640 1 --2- 172.21.15.161:0/2629720080 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa98c06b4d0 0x7fa98c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:15.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.836+0000 7fa9ad5a7640 1 --2- 172.21.15.161:0/2629720080 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa98c06b4d0 0x7fa98c06d990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa9a40113e0 tx=0x7fa9a400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:15.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:15.840+0000 7fa99dffb640 1 -- 172.21.15.161:0/2629720080 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa9a0058b60 con 0x7fa9a8152890 2024-03-28T23:34:16.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.046+0000 7fa9af80b640 1 -- 172.21.15.161:0/2629720080 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9a804f4b0 con 0x7fa9a8152890 2024-03-28T23:34:16.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.046+0000 7fa99dffb640 1 -- 172.21.15.161:0/2629720080 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+5202 (secure 0 0 0) 0x7fa9a005cc00 con 0x7fa9a8152890 2024-03-28T23:34:16.048 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:16.048 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":100,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/1610475969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1610475969},{"type":"v1","addr":"172.21.15.161:6835","nonce":1610475969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":6892,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/2684760895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2684760895},{"type":"v1","addr":"172.21.15.161:6837","nonce":2684760895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":99,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T23:33:50.326492+0000","modified":"2024-03-28T23:34:10.064154+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6203":{"gid":6203,"name":"b","rank":0,"incarnation":92,"state":"up:active","state_seq":8,"addr":"172.21.15.170:6834/1096504848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1096504848},{"type":"v1","addr":"172.21.15.170:6834","nonce":1096504848}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6206":{"gid":6206,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":7,"addr":"172.21.15.170:6835/3351418222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3351418222},{"type":"v1","addr":"172.21.15.170:6835","nonce":3351418222}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6203,"qdb_cluster":[6203]},"id":8}]} 2024-03-28T23:34:16.048 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 100 2024-03-28T23:34:16.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.048+0000 7fa9af80b640 1 -- 172.21.15.161:0/2629720080 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa98c06b4d0 msgr2=0x7fa98c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:16.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.048+0000 7fa9af80b640 1 --2- 172.21.15.161:0/2629720080 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa98c06b4d0 0x7fa98c06d990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa9a40113e0 tx=0x7fa9a400b040 comp rx=0 tx=0).stop 2024-03-28T23:34:16.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.048+0000 7fa9af80b640 1 -- 172.21.15.161:0/2629720080 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9a8152890 msgr2=0x7fa9a81448c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:16.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.048+0000 7fa9af80b640 1 --2- 172.21.15.161:0/2629720080 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9a8152890 0x7fa9a81448c0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000e7f0 tx=0x7fa9a000ecb0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.049+0000 7fa9af80b640 1 -- 172.21.15.161:0/2629720080 shutdown_connections 2024-03-28T23:34:16.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.049+0000 7fa9af80b640 1 --2- 172.21.15.161:0/2629720080 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa98c06b4d0 0x7fa98c06d990 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.049+0000 7fa9af80b640 1 --2- 172.21.15.161:0/2629720080 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9a8152890 0x7fa9a81448c0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.049+0000 7fa9af80b640 1 --2- 172.21.15.161:0/2629720080 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9a80c8ed0 0x7fa9a8144380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.049+0000 7fa9af80b640 1 --2- 172.21.15.161:0/2629720080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9a80c5950 0x7fa9a8142730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.049+0000 7fa9af80b640 1 -- 172.21.15.161:0/2629720080 >> 172.21.15.161:0/2629720080 conn(0x7fa9a80ac620 msgr2=0x7fa9a806c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:16.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.049+0000 7fa9af80b640 1 -- 172.21.15.161:0/2629720080 shutdown_connections 2024-03-28T23:34:16.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.049+0000 7fa9af80b640 1 -- 172.21.15.161:0/2629720080 wait complete. 2024-03-28T23:34:16.061 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:34:16.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.160+0000 7f2364adf640 1 Processor -- start 2024-03-28T23:34:16.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.160+0000 7f2364adf640 1 -- start start 2024-03-28T23:34:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.161+0000 7f2364adf640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f236006ce80 0x7f23600aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.161+0000 7f2364adf640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f236014c250 con 0x7f236014be70 2024-03-28T23:34:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.161+0000 7f2364adf640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23601529c0 con 0x7f23600a9680 2024-03-28T23:34:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.161+0000 7f2364adf640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f23600a9dc0 con 0x7f236006ce80 2024-03-28T23:34:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.161+0000 7f235e59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f236014be70 0x7f23600ab1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52194/0 (socket says 172.21.15.161:52194) 2024-03-28T23:34:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.161+0000 7f235e59c640 1 -- 172.21.15.161:0/3209817955 learned_addr learned my addr 172.21.15.161:0/3209817955 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.162+0000 7f235dd9b640 1 --2- 172.21.15.161:0/3209817955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f236006ce80 0x7f23600aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.162+0000 7f235d59a640 1 -- 172.21.15.161:0/3209817955 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2471649079 0 0) 0x7f236014c250 con 0x7f236014be70 2024-03-28T23:34:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.162+0000 7f235d59a640 1 -- 172.21.15.161:0/3209817955 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2348003520 con 0x7f236014be70 2024-03-28T23:34:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.162+0000 7f235d59a640 1 -- 172.21.15.161:0/3209817955 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 313424370 0 0) 0x7f23601529c0 con 0x7f23600a9680 2024-03-28T23:34:16.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.162+0000 7f235d59a640 1 -- 172.21.15.161:0/3209817955 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f236014c250 con 0x7f23600a9680 2024-03-28T23:34:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.162+0000 7f235dd9b640 1 -- 172.21.15.161:0/3209817955 >> v1:172.21.15.170:6789/0 conn(0x7f23600a9680 legacy=0x7f236015c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.162+0000 7f235dd9b640 1 -- 172.21.15.161:0/3209817955 >> v1:172.21.15.161:6789/0 conn(0x7f236014be70 legacy=0x7f23600ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.162+0000 7f235dd9b640 1 -- 172.21.15.161:0/3209817955 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23600aa0a0 con 0x7f236006ce80 2024-03-28T23:34:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.163+0000 7f235dd9b640 1 --2- 172.21.15.161:0/3209817955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f236006ce80 0x7f23600aa5c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2354009dc0 tx=0x7f235402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60ac84dc39dd6c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.163+0000 7f235d59a640 1 -- 172.21.15.161:0/3209817955 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2354039070 con 0x7f236006ce80 2024-03-28T23:34:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.163+0000 7f235d59a640 1 -- 172.21.15.161:0/3209817955 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f235402cc40 con 0x7f236006ce80 2024-03-28T23:34:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.163+0000 7f235d59a640 1 -- 172.21.15.161:0/3209817955 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2354034440 con 0x7f236006ce80 2024-03-28T23:34:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.163+0000 7f2364adf640 1 -- 172.21.15.161:0/3209817955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f236006ce80 msgr2=0x7f23600aa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.163+0000 7f2364adf640 1 --2- 172.21.15.161:0/3209817955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f236006ce80 0x7f23600aa5c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2354009dc0 tx=0x7f235402be90 comp rx=0 tx=0).stop 2024-03-28T23:34:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.163+0000 7f2364adf640 1 -- 172.21.15.161:0/3209817955 shutdown_connections 2024-03-28T23:34:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.163+0000 7f2364adf640 1 --2- 172.21.15.161:0/3209817955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f236006ce80 0x7f23600aa5c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.163+0000 7f2364adf640 1 -- 172.21.15.161:0/3209817955 >> 172.21.15.161:0/3209817955 conn(0x7f23600c6670 msgr2=0x7f23600c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.163+0000 7f2364adf640 1 -- 172.21.15.161:0/3209817955 shutdown_connections 2024-03-28T23:34:16.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.163+0000 7f2364adf640 1 -- 172.21.15.161:0/3209817955 wait complete. 2024-03-28T23:34:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.165+0000 7f2364adf640 1 Processor -- start 2024-03-28T23:34:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.165+0000 7f2364adf640 1 -- start start 2024-03-28T23:34:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.165+0000 7f2364adf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f236006ce80 0x7f23601488c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.165+0000 7f2364adf640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23600a9680 0x7f236006b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.165+0000 7f2364adf640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f236014be70 0x7f23601dbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.165+0000 7f2364adf640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f23600aba30 con 0x7f236006ce80 2024-03-28T23:34:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.165+0000 7f2364adf640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f23600ab8b0 con 0x7f236014be70 2024-03-28T23:34:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.165+0000 7f2364adf640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f236006b150 con 0x7f23600a9680 2024-03-28T23:34:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.165+0000 7f235e59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f236006ce80 0x7f23601488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.166+0000 7f235e59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f236006ce80 0x7f23601488c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45202/0 (socket says 172.21.15.161:45202) 2024-03-28T23:34:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.166+0000 7f235e59c640 1 -- 172.21.15.161:0/2164478405 learned_addr learned my addr 172.21.15.161:0/2164478405 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:16.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.166+0000 7f235dd9b640 1 --2- 172.21.15.161:0/2164478405 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23600a9680 0x7f236006b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.166+0000 7f235ed9d640 1 --2- 172.21.15.161:0/2164478405 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f236014be70 0x7f23601dbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.166+0000 7f235e59c640 1 -- 172.21.15.161:0/2164478405 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23600a9680 msgr2=0x7f236006b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.166+0000 7f235e59c640 1 --2- 172.21.15.161:0/2164478405 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23600a9680 0x7f236006b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.166+0000 7f235e59c640 1 -- 172.21.15.161:0/2164478405 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f236014be70 msgr2=0x7f23601dbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.166+0000 7f235e59c640 1 --2- 172.21.15.161:0/2164478405 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f236014be70 0x7f23601dbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.166+0000 7f235e59c640 1 -- 172.21.15.161:0/2164478405 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23601ddea0 con 0x7f236006ce80 2024-03-28T23:34:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.166+0000 7f235ed9d640 1 --2- 172.21.15.161:0/2164478405 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f236014be70 0x7f23601dbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:34:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.166+0000 7f235e59c640 1 --2- 172.21.15.161:0/2164478405 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f236006ce80 0x7f23601488c0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f235800c410 tx=0x7f235800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.166+0000 7f234effd640 1 -- 172.21.15.161:0/2164478405 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2358017070 con 0x7f236006ce80 2024-03-28T23:34:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.166+0000 7f234effd640 1 -- 172.21.15.161:0/2164478405 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2358005ce0 con 0x7f236006ce80 2024-03-28T23:34:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.166+0000 7f2364adf640 1 -- 172.21.15.161:0/2164478405 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23601de180 con 0x7f236006ce80 2024-03-28T23:34:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.166+0000 7f234effd640 1 -- 172.21.15.161:0/2164478405 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23580072d0 con 0x7f236006ce80 2024-03-28T23:34:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.166+0000 7f2364adf640 1 -- 172.21.15.161:0/2164478405 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23601db700 con 0x7f236006ce80 2024-03-28T23:34:16.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.167+0000 7f2364adf640 1 -- 172.21.15.161:0/2164478405 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f236004f4b0 con 0x7f236006ce80 2024-03-28T23:34:16.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.168+0000 7f234effd640 1 -- 172.21.15.161:0/2164478405 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f235801a070 con 0x7f236006ce80 2024-03-28T23:34:16.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.168+0000 7f234effd640 1 --2- 172.21.15.161:0/2164478405 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f234006b500 0x7f234006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:16.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.168+0000 7f234effd640 1 -- 172.21.15.161:0/2164478405 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7f2358089610 con 0x7f236006ce80 2024-03-28T23:34:16.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.168+0000 7f235dd9b640 1 --2- 172.21.15.161:0/2164478405 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f234006b500 0x7f234006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:16.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.169+0000 7f235dd9b640 1 --2- 172.21.15.161:0/2164478405 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f234006b500 0x7f234006d9c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2354036000 tx=0x7f2354030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:16.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.172+0000 7f234effd640 1 -- 172.21.15.161:0/2164478405 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2358057cf0 con 0x7f236006ce80 2024-03-28T23:34:16.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.345+0000 7f2364adf640 1 -- 172.21.15.161:0/2164478405 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f23600c0ff0 con 0x7f236006ce80 2024-03-28T23:34:16.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.346+0000 7f234effd640 1 -- 172.21.15.161:0/2164478405 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) ==== 74+0+17119 (secure 0 0 0) 0x7f235805bd90 con 0x7f236006ce80 2024-03-28T23:34:16.348 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:16.348 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":110,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:34:02.034225+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:33:48.560815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-28T23:33:49.301625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6835/2544838511":"2024-03-29T23:34:02.034196+0000","172.21.15.161:6834/2544838511":"2024-03-29T23:34:02.034196+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:34:16.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.348+0000 7f2364adf640 1 -- 172.21.15.161:0/2164478405 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f234006b500 msgr2=0x7f234006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:16.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.348+0000 7f2364adf640 1 --2- 172.21.15.161:0/2164478405 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f234006b500 0x7f234006d9c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2354036000 tx=0x7f2354030000 comp rx=0 tx=0).stop 2024-03-28T23:34:16.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.348+0000 7f2364adf640 1 -- 172.21.15.161:0/2164478405 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f236006ce80 msgr2=0x7f23601488c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:16.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.348+0000 7f2364adf640 1 --2- 172.21.15.161:0/2164478405 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f236006ce80 0x7f23601488c0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f235800c410 tx=0x7f235800c8d0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.348+0000 7f2364adf640 1 -- 172.21.15.161:0/2164478405 shutdown_connections 2024-03-28T23:34:16.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.348+0000 7f2364adf640 1 --2- 172.21.15.161:0/2164478405 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f234006b500 0x7f234006d9c0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.348+0000 7f2364adf640 1 --2- 172.21.15.161:0/2164478405 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f236014be70 0x7f23601dbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.348+0000 7f2364adf640 1 --2- 172.21.15.161:0/2164478405 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23600a9680 0x7f236006b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.348+0000 7f2364adf640 1 --2- 172.21.15.161:0/2164478405 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f236006ce80 0x7f23601488c0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.348+0000 7f2364adf640 1 -- 172.21.15.161:0/2164478405 >> 172.21.15.161:0/2164478405 conn(0x7f23600c6670 msgr2=0x7f23600c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:16.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.348+0000 7f2364adf640 1 -- 172.21.15.161:0/2164478405 shutdown_connections 2024-03-28T23:34:16.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.348+0000 7f2364adf640 1 -- 172.21.15.161:0/2164478405 wait complete. 2024-03-28T23:34:16.361 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:33:48.560815+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:33:49.301625+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:34:16.361 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:34:16.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.463+0000 7f2a39ba7640 1 Processor -- start 2024-03-28T23:34:16.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.463+0000 7f2a39ba7640 1 -- start start 2024-03-28T23:34:16.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.464+0000 7f2a39ba7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2a34151e70 0x7f2a34152250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:16.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.464+0000 7f2a39ba7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2a34152790 0x7f2a3406c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:16.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.464+0000 7f2a39ba7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a340c1e40 con 0x7f2a340c6350 2024-03-28T23:34:16.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.464+0000 7f2a39ba7640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2a340c2120 con 0x7f2a34152790 2024-03-28T23:34:16.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.464+0000 7f2a39ba7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f2a340c2400 con 0x7f2a34151e70 2024-03-28T23:34:16.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.464+0000 7f2a32ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2a34152790 0x7f2a3406c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:16.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.464+0000 7f2a32ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2a34152790 0x7f2a3406c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45218/0 (socket says 172.21.15.161:45218) 2024-03-28T23:34:16.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.464+0000 7f2a32ffd640 1 -- 172.21.15.161:0/2871420029 learned_addr learned my addr 172.21.15.161:0/2871420029 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:16.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.464+0000 7f2a337fe640 1 --2- 172.21.15.161:0/2871420029 >> v2:172.21.15.170:3300/0 conn(0x7f2a34151e70 0x7f2a34152250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:16.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.464+0000 7f2a32ffd640 1 -- 172.21.15.161:0/2871420029 >> v2:172.21.15.170:3300/0 conn(0x7f2a34151e70 msgr2=0x7f2a34152250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:16.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.464+0000 7f2a32ffd640 1 --2- 172.21.15.161:0/2871420029 >> v2:172.21.15.170:3300/0 conn(0x7f2a34151e70 0x7f2a34152250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.464+0000 7f2a32ffd640 1 -- 172.21.15.161:0/2871420029 >> v1:172.21.15.170:6789/0 conn(0x7f2a340c6350 legacy=0x7f2a341553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:16.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.464+0000 7f2a32ffd640 1 -- 172.21.15.161:0/2871420029 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a340c26e0 con 0x7f2a34152790 2024-03-28T23:34:16.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.465+0000 7f2a32ffd640 1 --2- 172.21.15.161:0/2871420029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a34152790 0x7f2a3406c6c0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f2a3406fc50 tx=0x7f2a2802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=113bbb550c20f8e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:16.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.465+0000 7f2a327fc640 1 -- 172.21.15.161:0/2871420029 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a28035020 con 0x7f2a34152790 2024-03-28T23:34:16.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.465+0000 7f2a327fc640 1 -- 172.21.15.161:0/2871420029 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a28031870 con 0x7f2a34152790 2024-03-28T23:34:16.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.465+0000 7f2a327fc640 1 -- 172.21.15.161:0/2871420029 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a28031c40 con 0x7f2a34152790 2024-03-28T23:34:16.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.466+0000 7f2a39ba7640 1 -- 172.21.15.161:0/2871420029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a34152790 msgr2=0x7f2a3406c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:16.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.466+0000 7f2a39ba7640 1 --2- 172.21.15.161:0/2871420029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a34152790 0x7f2a3406c6c0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f2a3406fc50 tx=0x7f2a2802b910 comp rx=0 tx=0).stop 2024-03-28T23:34:16.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.466+0000 7f2a39ba7640 1 -- 172.21.15.161:0/2871420029 shutdown_connections 2024-03-28T23:34:16.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.466+0000 7f2a39ba7640 1 --2- 172.21.15.161:0/2871420029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a34152790 0x7f2a3406c6c0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.466+0000 7f2a39ba7640 1 --2- 172.21.15.161:0/2871420029 >> v2:172.21.15.170:3300/0 conn(0x7f2a34151e70 0x7f2a34152250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.466+0000 7f2a39ba7640 1 -- 172.21.15.161:0/2871420029 >> 172.21.15.161:0/2871420029 conn(0x7f2a340c3390 msgr2=0x7f2a340c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:16.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.466+0000 7f2a39ba7640 1 -- 172.21.15.161:0/2871420029 shutdown_connections 2024-03-28T23:34:16.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.466+0000 7f2a39ba7640 1 -- 172.21.15.161:0/2871420029 wait complete. 2024-03-28T23:34:16.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.467+0000 7f2a39ba7640 1 Processor -- start 2024-03-28T23:34:16.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.467+0000 7f2a39ba7640 1 -- start start 2024-03-28T23:34:16.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.467+0000 7f2a39ba7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a340c6350 0x7f2a3414c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:16.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.468+0000 7f2a39ba7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a34151e70 0x7f2a340a6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:16.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.468+0000 7f2a39ba7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a340a5bd0 0x7f2a34073170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:16.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.468+0000 7f2a39ba7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2a340c2d80 con 0x7f2a340c6350 2024-03-28T23:34:16.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.468+0000 7f2a39ba7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2a340c2c00 con 0x7f2a34151e70 2024-03-28T23:34:16.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.468+0000 7f2a39ba7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2a340a5fb0 con 0x7f2a340a5bd0 2024-03-28T23:34:16.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.468+0000 7f2a337fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a340c6350 0x7f2a3414c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:16.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.468+0000 7f2a337fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a340c6350 0x7f2a3414c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45224/0 (socket says 172.21.15.161:45224) 2024-03-28T23:34:16.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.468+0000 7f2a337fe640 1 -- 172.21.15.161:0/3990364167 learned_addr learned my addr 172.21.15.161:0/3990364167 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:16.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.468+0000 7f2a32ffd640 1 --2- 172.21.15.161:0/3990364167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a34151e70 0x7f2a340a6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:16.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.468+0000 7f2a33fff640 1 --2- 172.21.15.161:0/3990364167 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a340a5bd0 0x7f2a34073170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:16.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.468+0000 7f2a337fe640 1 -- 172.21.15.161:0/3990364167 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a340a5bd0 msgr2=0x7f2a34073170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:16.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.468+0000 7f2a337fe640 1 --2- 172.21.15.161:0/3990364167 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a340a5bd0 0x7f2a34073170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.468+0000 7f2a337fe640 1 -- 172.21.15.161:0/3990364167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a34151e70 msgr2=0x7f2a340a6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.468+0000 7f2a337fe640 1 --2- 172.21.15.161:0/3990364167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a34151e70 0x7f2a340a6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.468+0000 7f2a337fe640 1 -- 172.21.15.161:0/3990364167 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a3414af60 con 0x7f2a340c6350 2024-03-28T23:34:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.468+0000 7f2a337fe640 1 --2- 172.21.15.161:0/3990364167 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a340c6350 0x7f2a3414c760 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f2a2400e9b0 tx=0x7f2a2400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.468+0000 7f2a1bfff640 1 -- 172.21.15.161:0/3990364167 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a24017070 con 0x7f2a340c6350 2024-03-28T23:34:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.468+0000 7f2a1bfff640 1 -- 172.21.15.161:0/3990364167 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a2400c9a0 con 0x7f2a340c6350 2024-03-28T23:34:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.468+0000 7f2a1bfff640 1 -- 172.21.15.161:0/3990364167 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2400cd90 con 0x7f2a340c6350 2024-03-28T23:34:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.468+0000 7f2a39ba7640 1 -- 172.21.15.161:0/3990364167 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a340ac2e0 con 0x7f2a340c6350 2024-03-28T23:34:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.468+0000 7f2a39ba7640 1 -- 172.21.15.161:0/3990364167 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a340a8110 con 0x7f2a340c6350 2024-03-28T23:34:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.469+0000 7f2a39ba7640 1 -- 172.21.15.161:0/3990364167 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a3404f4b0 con 0x7f2a340c6350 2024-03-28T23:34:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.470+0000 7f2a1bfff640 1 -- 172.21.15.161:0/3990364167 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2a2401a070 con 0x7f2a340c6350 2024-03-28T23:34:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.470+0000 7f2a1bfff640 1 --2- 172.21.15.161:0/3990364167 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2a2006b4d0 0x7f2a2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.470+0000 7f2a1bfff640 1 -- 172.21.15.161:0/3990364167 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7f2a2408a630 con 0x7f2a340c6350 2024-03-28T23:34:16.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.470+0000 7f2a32ffd640 1 --2- 172.21.15.161:0/3990364167 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2a2006b4d0 0x7f2a2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:16.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.471+0000 7f2a32ffd640 1 --2- 172.21.15.161:0/3990364167 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2a2006b4d0 0x7f2a2006d990 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f2a28036000 tx=0x7f2a2803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:16.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.475+0000 7f2a1bfff640 1 -- 172.21.15.161:0/3990364167 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2a24058d10 con 0x7f2a340c6350 2024-03-28T23:34:16.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.647+0000 7f2a39ba7640 1 -- 172.21.15.161:0/3990364167 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2a340c2f80 con 0x7f2a340c6350 2024-03-28T23:34:16.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.647+0000 7f2a1bfff640 1 -- 172.21.15.161:0/3990364167 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v110) ==== 93+0+10 (secure 0 0 0) 0x7f2a2405cdb0 con 0x7f2a340c6350 2024-03-28T23:34:16.649 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:34:16.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.649+0000 7f2a39ba7640 1 -- 172.21.15.161:0/3990364167 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2a2006b4d0 msgr2=0x7f2a2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:16.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.649+0000 7f2a39ba7640 1 --2- 172.21.15.161:0/3990364167 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2a2006b4d0 0x7f2a2006d990 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f2a28036000 tx=0x7f2a2803c040 comp rx=0 tx=0).stop 2024-03-28T23:34:16.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.649+0000 7f2a39ba7640 1 -- 172.21.15.161:0/3990364167 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a340c6350 msgr2=0x7f2a3414c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:16.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.649+0000 7f2a39ba7640 1 --2- 172.21.15.161:0/3990364167 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a340c6350 0x7f2a3414c760 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f2a2400e9b0 tx=0x7f2a2400ee70 comp rx=0 tx=0).stop 2024-03-28T23:34:16.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.649+0000 7f2a39ba7640 1 -- 172.21.15.161:0/3990364167 shutdown_connections 2024-03-28T23:34:16.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.649+0000 7f2a39ba7640 1 --2- 172.21.15.161:0/3990364167 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2a2006b4d0 0x7f2a2006d990 secure :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f2a28036000 tx=0x7f2a2803c040 comp rx=0 tx=0).stop 2024-03-28T23:34:16.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.649+0000 7f2a39ba7640 1 --2- 172.21.15.161:0/3990364167 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a340a5bd0 0x7f2a34073170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.649+0000 7f2a39ba7640 1 --2- 172.21.15.161:0/3990364167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a34151e70 0x7f2a340a6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.649+0000 7f2a39ba7640 1 --2- 172.21.15.161:0/3990364167 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a340c6350 0x7f2a3414c760 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.649+0000 7f2a39ba7640 1 -- 172.21.15.161:0/3990364167 >> 172.21.15.161:0/3990364167 conn(0x7f2a340c3390 msgr2=0x7f2a340c5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:16.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.649+0000 7f2a39ba7640 1 -- 172.21.15.161:0/3990364167 shutdown_connections 2024-03-28T23:34:16.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.649+0000 7f2a39ba7640 1 -- 172.21.15.161:0/3990364167 wait complete. 2024-03-28T23:34:16.661 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:34:16.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.763+0000 7f35778ed640 1 Processor -- start 2024-03-28T23:34:16.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.763+0000 7f35778ed640 1 -- start start 2024-03-28T23:34:16.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.764+0000 7f35778ed640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3570152bc0 0x7f35700c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:16.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.764+0000 7f35778ed640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f35701589b0 0x7f3570158e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:16.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.764+0000 7f35778ed640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35700c6f60 con 0x7f35700c8a80 2024-03-28T23:34:16.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.764+0000 7f35778ed640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f35700c6120 con 0x7f3570152bc0 2024-03-28T23:34:16.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.764+0000 7f35778ed640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f35700c6400 con 0x7f35701589b0 2024-03-28T23:34:16.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.764+0000 7f3575689640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3570152bc0 0x7f35700c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:16.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.764+0000 7f3574e88640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f35700c8a80 0x7f35700c8ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52200/0 (socket says 172.21.15.161:52200) 2024-03-28T23:34:16.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.764+0000 7f3575689640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3570152bc0 0x7f35700c0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45236/0 (socket says 172.21.15.161:45236) 2024-03-28T23:34:16.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.764+0000 7f3574e88640 1 -- 172.21.15.161:0/3817353278 learned_addr learned my addr 172.21.15.161:0/3817353278 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:16.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.764+0000 7f3575e8a640 1 --2- 172.21.15.161:0/3817353278 >> v2:172.21.15.170:3300/0 conn(0x7f35701589b0 0x7f3570158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:16.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.765+0000 7f3575689640 1 -- 172.21.15.161:0/3817353278 >> v2:172.21.15.170:3300/0 conn(0x7f35701589b0 msgr2=0x7f3570158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:16.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.765+0000 7f3575689640 1 --2- 172.21.15.161:0/3817353278 >> v2:172.21.15.170:3300/0 conn(0x7f35701589b0 0x7f3570158e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.765+0000 7f3575689640 1 -- 172.21.15.161:0/3817353278 >> v1:172.21.15.161:6789/0 conn(0x7f35700c8a80 legacy=0x7f35700c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:16.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.765+0000 7f3575689640 1 -- 172.21.15.161:0/3817353278 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35700c7240 con 0x7f3570152bc0 2024-03-28T23:34:16.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.765+0000 7f3567fff640 1 -- 172.21.15.161:0/3817353278 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1591124926 0 0) 0x7f35700c6f60 con 0x7f35700c8a80 2024-03-28T23:34:16.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.765+0000 7f3575689640 1 --2- 172.21.15.161:0/3817353278 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3570152bc0 0x7f35700c0330 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f35700aa5d0 tx=0x7f356c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=374947327aa0770 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:16.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.765+0000 7f3567fff640 1 -- 172.21.15.161:0/3817353278 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356c039070 con 0x7f3570152bc0 2024-03-28T23:34:16.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.765+0000 7f3567fff640 1 -- 172.21.15.161:0/3817353278 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f356c02cc80 con 0x7f3570152bc0 2024-03-28T23:34:16.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.765+0000 7f3567fff640 1 -- 172.21.15.161:0/3817353278 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356c034660 con 0x7f3570152bc0 2024-03-28T23:34:16.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.766+0000 7f35778ed640 1 -- 172.21.15.161:0/3817353278 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3570152bc0 msgr2=0x7f35700c0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:16.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.766+0000 7f35778ed640 1 --2- 172.21.15.161:0/3817353278 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3570152bc0 0x7f35700c0330 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f35700aa5d0 tx=0x7f356c02bed0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.766+0000 7f35778ed640 1 -- 172.21.15.161:0/3817353278 shutdown_connections 2024-03-28T23:34:16.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.766+0000 7f35778ed640 1 --2- 172.21.15.161:0/3817353278 >> v2:172.21.15.170:3300/0 conn(0x7f35701589b0 0x7f3570158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.766+0000 7f35778ed640 1 --2- 172.21.15.161:0/3817353278 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3570152bc0 0x7f35700c0330 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.766+0000 7f35778ed640 1 -- 172.21.15.161:0/3817353278 >> 172.21.15.161:0/3817353278 conn(0x7f35700caa80 msgr2=0x7f35700cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:16.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.766+0000 7f35778ed640 1 -- 172.21.15.161:0/3817353278 shutdown_connections 2024-03-28T23:34:16.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.766+0000 7f35778ed640 1 -- 172.21.15.161:0/3817353278 wait complete. 2024-03-28T23:34:16.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.767+0000 7f35778ed640 1 Processor -- start 2024-03-28T23:34:16.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.767+0000 7f35778ed640 1 -- start start 2024-03-28T23:34:16.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.768+0000 7f35778ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f35700c8a80 0x7f357013eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:16.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.768+0000 7f35778ed640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3570152bc0 0x7f357006cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:16.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.768+0000 7f35778ed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f35701589b0 0x7f357006b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:16.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.768+0000 7f35778ed640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3570152fa0 con 0x7f3570152bc0 2024-03-28T23:34:16.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.768+0000 7f35778ed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f35700c95f0 con 0x7f35701589b0 2024-03-28T23:34:16.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.768+0000 7f35778ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3570147ff0 con 0x7f35700c8a80 2024-03-28T23:34:16.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.768+0000 7f3574e88640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3570152bc0 0x7f357006cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:16.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.768+0000 7f3574e88640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3570152bc0 0x7f357006cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45238/0 (socket says 172.21.15.161:45238) 2024-03-28T23:34:16.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.768+0000 7f3574e88640 1 -- 172.21.15.161:0/475402238 learned_addr learned my addr 172.21.15.161:0/475402238 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:16.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.768+0000 7f3575689640 1 --2- 172.21.15.161:0/475402238 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f35700c8a80 0x7f357013eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:16.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.768+0000 7f3575e8a640 1 --2- 172.21.15.161:0/475402238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f35701589b0 0x7f357006b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:16.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.768+0000 7f3574e88640 1 -- 172.21.15.161:0/475402238 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f35700c8a80 msgr2=0x7f357013eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:16.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.768+0000 7f3574e88640 1 --2- 172.21.15.161:0/475402238 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f35700c8a80 0x7f357013eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.768+0000 7f3574e88640 1 -- 172.21.15.161:0/475402238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f35701589b0 msgr2=0x7f357006b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:16.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.768+0000 7f3574e88640 1 --2- 172.21.15.161:0/475402238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f35701589b0 0x7f357006b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.768+0000 7f3574e88640 1 -- 172.21.15.161:0/475402238 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35701eaee0 con 0x7f3570152bc0 2024-03-28T23:34:16.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.768+0000 7f3574e88640 1 --2- 172.21.15.161:0/475402238 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3570152bc0 0x7f357006cba0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f356000c4d0 tx=0x7f356000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:16.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.769+0000 7f3565ffb640 1 -- 172.21.15.161:0/475402238 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3560018070 con 0x7f3570152bc0 2024-03-28T23:34:16.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.769+0000 7f3565ffb640 1 -- 172.21.15.161:0/475402238 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3560007500 con 0x7f3570152bc0 2024-03-28T23:34:16.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.769+0000 7f3565ffb640 1 -- 172.21.15.161:0/475402238 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3560004a20 con 0x7f3570152bc0 2024-03-28T23:34:16.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.769+0000 7f35778ed640 1 -- 172.21.15.161:0/475402238 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35701eb1c0 con 0x7f3570152bc0 2024-03-28T23:34:16.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.769+0000 7f35778ed640 1 -- 172.21.15.161:0/475402238 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35701eb780 con 0x7f3570152bc0 2024-03-28T23:34:16.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.770+0000 7f35778ed640 1 -- 172.21.15.161:0/475402238 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f357004f4b0 con 0x7f3570152bc0 2024-03-28T23:34:16.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.770+0000 7f3565ffb640 1 -- 172.21.15.161:0/475402238 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f356001b070 con 0x7f3570152bc0 2024-03-28T23:34:16.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.770+0000 7f3565ffb640 1 --2- 172.21.15.161:0/475402238 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f355406b4d0 0x7f355406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:16.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.770+0000 7f3565ffb640 1 -- 172.21.15.161:0/475402238 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7f35600896e0 con 0x7f3570152bc0 2024-03-28T23:34:16.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.770+0000 7f3575689640 1 --2- 172.21.15.161:0/475402238 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f355406b4d0 0x7f355406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:16.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.771+0000 7f3575689640 1 --2- 172.21.15.161:0/475402238 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f355406b4d0 0x7f355406d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f356c036000 tx=0x7f356c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:16.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.775+0000 7f3565ffb640 1 -- 172.21.15.161:0/475402238 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3560057d30 con 0x7f3570152bc0 2024-03-28T23:34:16.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.945+0000 7f35778ed640 1 -- 172.21.15.161:0/475402238 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f357014bec0 con 0x7f3570152bc0 2024-03-28T23:34:16.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.945+0000 7f3565ffb640 1 -- 172.21.15.161:0/475402238 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v110) ==== 104+0+11 (secure 0 0 0) 0x7f356005bdd0 con 0x7f3570152bc0 2024-03-28T23:34:16.947 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:34:16.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.948+0000 7f35778ed640 1 -- 172.21.15.161:0/475402238 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f355406b4d0 msgr2=0x7f355406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:16.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.948+0000 7f35778ed640 1 --2- 172.21.15.161:0/475402238 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f355406b4d0 0x7f355406d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f356c036000 tx=0x7f356c030000 comp rx=0 tx=0).stop 2024-03-28T23:34:16.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.948+0000 7f35778ed640 1 -- 172.21.15.161:0/475402238 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3570152bc0 msgr2=0x7f357006cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:16.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.948+0000 7f35778ed640 1 --2- 172.21.15.161:0/475402238 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3570152bc0 0x7f357006cba0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f356000c4d0 tx=0x7f356000c990 comp rx=0 tx=0).stop 2024-03-28T23:34:16.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.948+0000 7f35778ed640 1 -- 172.21.15.161:0/475402238 shutdown_connections 2024-03-28T23:34:16.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.948+0000 7f35778ed640 1 --2- 172.21.15.161:0/475402238 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f355406b4d0 0x7f355406d990 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.948+0000 7f35778ed640 1 --2- 172.21.15.161:0/475402238 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3570152bc0 0x7f357006cba0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.948+0000 7f35778ed640 1 --2- 172.21.15.161:0/475402238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f35701589b0 0x7f357006b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.948+0000 7f35778ed640 1 --2- 172.21.15.161:0/475402238 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f35700c8a80 0x7f357013eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:16.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.948+0000 7f35778ed640 1 -- 172.21.15.161:0/475402238 >> 172.21.15.161:0/475402238 conn(0x7f35700caa80 msgr2=0x7f35700cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:16.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.948+0000 7f35778ed640 1 -- 172.21.15.161:0/475402238 shutdown_connections 2024-03-28T23:34:16.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:16.948+0000 7f35778ed640 1 -- 172.21.15.161:0/475402238 wait complete. 2024-03-28T23:34:16.959 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:34:17.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.061+0000 7ff16488f640 1 Processor -- start 2024-03-28T23:34:17.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.061+0000 7ff16488f640 1 -- start start 2024-03-28T23:34:17.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.062+0000 7ff16488f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff1601529d0 0x7ff16006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:17.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.062+0000 7ff16488f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1600c6750 0x7ff1600c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:17.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.062+0000 7ff16488f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff160152db0 con 0x7ff160155c30 2024-03-28T23:34:17.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.062+0000 7ff16488f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7ff1600c7210 con 0x7ff1601529d0 2024-03-28T23:34:17.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.062+0000 7ff16488f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff1600c6c50 con 0x7ff1600c6750 2024-03-28T23:34:17.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.062+0000 7ff15ed9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ff160155c30 0x7ff1600c54f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52216/0 (socket says 172.21.15.161:52216) 2024-03-28T23:34:17.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.062+0000 7ff15ed9d640 1 -- 172.21.15.161:0/3845560698 learned_addr learned my addr 172.21.15.161:0/3845560698 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:17.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.062+0000 7ff15dd9b640 1 --2- 172.21.15.161:0/3845560698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1600c6750 0x7ff1600c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:17.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.062+0000 7ff15e59c640 1 --2- 172.21.15.161:0/3845560698 >> v2:172.21.15.170:3300/0 conn(0x7ff1601529d0 0x7ff16006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:17.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.063+0000 7ff15d59a640 1 -- 172.21.15.161:0/3845560698 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2353365495 0 0) 0x7ff160152db0 con 0x7ff160155c30 2024-03-28T23:34:17.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.063+0000 7ff15dd9b640 1 -- 172.21.15.161:0/3845560698 >> v2:172.21.15.170:3300/0 conn(0x7ff1601529d0 msgr2=0x7ff16006bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:17.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.063+0000 7ff15dd9b640 1 --2- 172.21.15.161:0/3845560698 >> v2:172.21.15.170:3300/0 conn(0x7ff1601529d0 0x7ff16006bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.063+0000 7ff15dd9b640 1 -- 172.21.15.161:0/3845560698 >> v1:172.21.15.161:6789/0 conn(0x7ff160155c30 legacy=0x7ff1600c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:17.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.063+0000 7ff15dd9b640 1 -- 172.21.15.161:0/3845560698 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1600c6f30 con 0x7ff1600c6750 2024-03-28T23:34:17.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.063+0000 7ff15dd9b640 1 --2- 172.21.15.161:0/3845560698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1600c6750 0x7ff1600c88b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff1540097a0 tx=0x7ff15402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78038f0c833c8276 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:17.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.064+0000 7ff15d59a640 1 -- 172.21.15.161:0/3845560698 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff154035020 con 0x7ff1600c6750 2024-03-28T23:34:17.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.064+0000 7ff15d59a640 1 -- 172.21.15.161:0/3845560698 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff154031830 con 0x7ff1600c6750 2024-03-28T23:34:17.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.064+0000 7ff15d59a640 1 -- 172.21.15.161:0/3845560698 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff154031c00 con 0x7ff1600c6750 2024-03-28T23:34:17.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.064+0000 7ff16488f640 1 -- 172.21.15.161:0/3845560698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1600c6750 msgr2=0x7ff1600c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:17.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.064+0000 7ff16488f640 1 --2- 172.21.15.161:0/3845560698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1600c6750 0x7ff1600c88b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff1540097a0 tx=0x7ff15402b8d0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.064+0000 7ff16488f640 1 -- 172.21.15.161:0/3845560698 shutdown_connections 2024-03-28T23:34:17.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.064+0000 7ff16488f640 1 --2- 172.21.15.161:0/3845560698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1600c6750 0x7ff1600c88b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.064+0000 7ff16488f640 1 --2- 172.21.15.161:0/3845560698 >> v2:172.21.15.170:3300/0 conn(0x7ff1601529d0 0x7ff16006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.064+0000 7ff16488f640 1 -- 172.21.15.161:0/3845560698 >> 172.21.15.161:0/3845560698 conn(0x7ff1600c7430 msgr2=0x7ff1600c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:17.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.064+0000 7ff16488f640 1 -- 172.21.15.161:0/3845560698 shutdown_connections 2024-03-28T23:34:17.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.064+0000 7ff16488f640 1 -- 172.21.15.161:0/3845560698 wait complete. 2024-03-28T23:34:17.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.065+0000 7ff16488f640 1 Processor -- start 2024-03-28T23:34:17.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.065+0000 7ff16488f640 1 -- start start 2024-03-28T23:34:17.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.066+0000 7ff16488f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff1600c6750 0x7ff1600c2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:17.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.066+0000 7ff16488f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1601529d0 0x7ff1600c2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:17.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.066+0000 7ff16488f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff160155c30 0x7ff1601ee250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:17.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.066+0000 7ff16488f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff1600aa160 con 0x7ff1601529d0 2024-03-28T23:34:17.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.066+0000 7ff16488f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff160155710 con 0x7ff1600c6750 2024-03-28T23:34:17.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.066+0000 7ff16488f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff1600c5c00 con 0x7ff160155c30 2024-03-28T23:34:17.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.066+0000 7ff15dd9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1601529d0 0x7ff1600c2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:17.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.066+0000 7ff15dd9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1601529d0 0x7ff1600c2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45240/0 (socket says 172.21.15.161:45240) 2024-03-28T23:34:17.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.066+0000 7ff15dd9b640 1 -- 172.21.15.161:0/3813869699 learned_addr learned my addr 172.21.15.161:0/3813869699 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:17.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.066+0000 7ff15ed9d640 1 --2- 172.21.15.161:0/3813869699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff160155c30 0x7ff1601ee250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:17.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.066+0000 7ff15e59c640 1 --2- 172.21.15.161:0/3813869699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff1600c6750 0x7ff1600c2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:17.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.066+0000 7ff15dd9b640 1 -- 172.21.15.161:0/3813869699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff160155c30 msgr2=0x7ff1601ee250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:17.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.066+0000 7ff15dd9b640 1 --2- 172.21.15.161:0/3813869699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff160155c30 0x7ff1601ee250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.066+0000 7ff15dd9b640 1 -- 172.21.15.161:0/3813869699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff1600c6750 msgr2=0x7ff1600c2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:17.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.066+0000 7ff15dd9b640 1 --2- 172.21.15.161:0/3813869699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff1600c6750 0x7ff1600c2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.066+0000 7ff15dd9b640 1 -- 172.21.15.161:0/3813869699 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1600c3e20 con 0x7ff1601529d0 2024-03-28T23:34:17.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.067+0000 7ff15dd9b640 1 --2- 172.21.15.161:0/3813869699 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1601529d0 0x7ff1600c2db0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7ff1540097a0 tx=0x7ff1540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:17.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.067+0000 7ff14effd640 1 -- 172.21.15.161:0/3813869699 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff154035040 con 0x7ff1601529d0 2024-03-28T23:34:17.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.067+0000 7ff14effd640 1 -- 172.21.15.161:0/3813869699 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff154004230 con 0x7ff1601529d0 2024-03-28T23:34:17.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.067+0000 7ff16488f640 1 -- 172.21.15.161:0/3813869699 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1601ef040 con 0x7ff1601529d0 2024-03-28T23:34:17.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.067+0000 7ff14effd640 1 -- 172.21.15.161:0/3813869699 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff15403a640 con 0x7ff1601529d0 2024-03-28T23:34:17.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.067+0000 7ff16488f640 1 -- 172.21.15.161:0/3813869699 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1600c43c0 con 0x7ff1601529d0 2024-03-28T23:34:17.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.068+0000 7ff16488f640 1 -- 172.21.15.161:0/3813869699 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff16004f4b0 con 0x7ff1601529d0 2024-03-28T23:34:17.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.068+0000 7ff14effd640 1 -- 172.21.15.161:0/3813869699 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff154038020 con 0x7ff1601529d0 2024-03-28T23:34:17.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.069+0000 7ff14effd640 1 --2- 172.21.15.161:0/3813869699 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff14006b4d0 0x7ff14006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:17.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.069+0000 7ff14effd640 1 -- 172.21.15.161:0/3813869699 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7ff1540ac010 con 0x7ff1601529d0 2024-03-28T23:34:17.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.069+0000 7ff15e59c640 1 --2- 172.21.15.161:0/3813869699 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff14006b4d0 0x7ff14006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:17.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.069+0000 7ff15e59c640 1 --2- 172.21.15.161:0/3813869699 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff14006b4d0 0x7ff14006d990 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff150011340 tx=0x7ff15000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:17.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.073+0000 7ff14effd640 1 -- 172.21.15.161:0/3813869699 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff15407a670 con 0x7ff1601529d0 2024-03-28T23:34:17.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.248+0000 7ff16488f640 1 -- 172.21.15.161:0/3813869699 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff1600ca650 con 0x7ff1601529d0 2024-03-28T23:34:17.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.248+0000 7ff14effd640 1 -- 172.21.15.161:0/3813869699 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v110) ==== 100+0+11 (secure 0 0 0) 0x7ff15407e710 con 0x7ff1601529d0 2024-03-28T23:34:17.250 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:34:17.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.250+0000 7ff16488f640 1 -- 172.21.15.161:0/3813869699 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff14006b4d0 msgr2=0x7ff14006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:17.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.250+0000 7ff16488f640 1 --2- 172.21.15.161:0/3813869699 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff14006b4d0 0x7ff14006d990 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff150011340 tx=0x7ff15000b040 comp rx=0 tx=0).stop 2024-03-28T23:34:17.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.250+0000 7ff16488f640 1 -- 172.21.15.161:0/3813869699 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1601529d0 msgr2=0x7ff1600c2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:17.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.250+0000 7ff16488f640 1 --2- 172.21.15.161:0/3813869699 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1601529d0 0x7ff1600c2db0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7ff1540097a0 tx=0x7ff1540046c0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.250+0000 7ff16488f640 1 -- 172.21.15.161:0/3813869699 shutdown_connections 2024-03-28T23:34:17.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.250+0000 7ff16488f640 1 --2- 172.21.15.161:0/3813869699 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff14006b4d0 0x7ff14006d990 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.250+0000 7ff16488f640 1 --2- 172.21.15.161:0/3813869699 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1601529d0 0x7ff1600c2db0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.250+0000 7ff16488f640 1 --2- 172.21.15.161:0/3813869699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff160155c30 0x7ff1601ee250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.250+0000 7ff16488f640 1 --2- 172.21.15.161:0/3813869699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff1600c6750 0x7ff1600c2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.250+0000 7ff16488f640 1 -- 172.21.15.161:0/3813869699 >> 172.21.15.161:0/3813869699 conn(0x7ff1600c7430 msgr2=0x7ff160153060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:17.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.250+0000 7ff16488f640 1 -- 172.21.15.161:0/3813869699 shutdown_connections 2024-03-28T23:34:17.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.250+0000 7ff16488f640 1 -- 172.21.15.161:0/3813869699 wait complete. 2024-03-28T23:34:17.263 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:34:17.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.362+0000 7efd5307c640 1 Processor -- start 2024-03-28T23:34:17.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.362+0000 7efd5307c640 1 -- start start 2024-03-28T23:34:17.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.363+0000 7efd5307c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7efd4c152890 0x7efd4c0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:17.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.363+0000 7efd5307c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efd4c0c5b20 0x7efd4c1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:17.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.363+0000 7efd5307c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd4c158a30 con 0x7efd4c0c8ed0 2024-03-28T23:34:17.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.363+0000 7efd5307c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7efd4c0c8480 con 0x7efd4c0c5b20 2024-03-28T23:34:17.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.363+0000 7efd5307c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7efd4c159640 con 0x7efd4c152890 2024-03-28T23:34:17.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.363+0000 7efd51619640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efd4c0c5b20 0x7efd4c1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:17.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.363+0000 7efd4bfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7efd4c0c8ed0 0x7efd4c0c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52222/0 (socket says 172.21.15.161:52222) 2024-03-28T23:34:17.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.363+0000 7efd51619640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efd4c0c5b20 0x7efd4c1553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45246/0 (socket says 172.21.15.161:45246) 2024-03-28T23:34:17.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.363+0000 7efd4bfff640 1 -- 172.21.15.161:0/1195220070 learned_addr learned my addr 172.21.15.161:0/1195220070 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:17.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.363+0000 7efd50e18640 1 --2- 172.21.15.161:0/1195220070 >> v2:172.21.15.170:3300/0 conn(0x7efd4c152890 0x7efd4c0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:17.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.364+0000 7efd51619640 1 -- 172.21.15.161:0/1195220070 >> v2:172.21.15.170:3300/0 conn(0x7efd4c152890 msgr2=0x7efd4c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:17.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.364+0000 7efd51619640 1 --2- 172.21.15.161:0/1195220070 >> v2:172.21.15.170:3300/0 conn(0x7efd4c152890 0x7efd4c0c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.364+0000 7efd51619640 1 -- 172.21.15.161:0/1195220070 >> v1:172.21.15.161:6789/0 conn(0x7efd4c0c8ed0 legacy=0x7efd4c0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:17.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.364+0000 7efd51619640 1 -- 172.21.15.161:0/1195220070 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd4c158d10 con 0x7efd4c0c5b20 2024-03-28T23:34:17.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.364+0000 7efd51619640 1 --2- 172.21.15.161:0/1195220070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd4c0c5b20 0x7efd4c1553a0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7efd4c08cc80 tx=0x7efd4402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37d64103c73493bb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:17.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.364+0000 7efd4b7fe640 1 -- 172.21.15.161:0/1195220070 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd44039070 con 0x7efd4c0c5b20 2024-03-28T23:34:17.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.365+0000 7efd4b7fe640 1 -- 172.21.15.161:0/1195220070 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd4402cb60 con 0x7efd4c0c5b20 2024-03-28T23:34:17.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.365+0000 7efd4b7fe640 1 -- 172.21.15.161:0/1195220070 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd44034440 con 0x7efd4c0c5b20 2024-03-28T23:34:17.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.365+0000 7efd5307c640 1 -- 172.21.15.161:0/1195220070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd4c0c5b20 msgr2=0x7efd4c1553a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:17.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.365+0000 7efd5307c640 1 --2- 172.21.15.161:0/1195220070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd4c0c5b20 0x7efd4c1553a0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7efd4c08cc80 tx=0x7efd4402bdb0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.365+0000 7efd5307c640 1 -- 172.21.15.161:0/1195220070 shutdown_connections 2024-03-28T23:34:17.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.365+0000 7efd5307c640 1 --2- 172.21.15.161:0/1195220070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd4c0c5b20 0x7efd4c1553a0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.365+0000 7efd5307c640 1 --2- 172.21.15.161:0/1195220070 >> v2:172.21.15.170:3300/0 conn(0x7efd4c152890 0x7efd4c0c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.365+0000 7efd5307c640 1 -- 172.21.15.161:0/1195220070 >> 172.21.15.161:0/1195220070 conn(0x7efd4c0ac620 msgr2=0x7efd4c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:17.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.365+0000 7efd5307c640 1 -- 172.21.15.161:0/1195220070 shutdown_connections 2024-03-28T23:34:17.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.365+0000 7efd5307c640 1 -- 172.21.15.161:0/1195220070 wait complete. 2024-03-28T23:34:17.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.366+0000 7efd5307c640 1 Processor -- start 2024-03-28T23:34:17.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.366+0000 7efd5307c640 1 -- start start 2024-03-28T23:34:17.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.367+0000 7efd5307c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd4c0c5b20 0x7efd4c0c1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:17.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.367+0000 7efd5307c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd4c0c8ed0 0x7efd4c1ee570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:17.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.367+0000 7efd5307c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd4c152890 0x7efd4c1555a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:17.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.367+0000 7efd5307c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efd4c152c70 con 0x7efd4c152890 2024-03-28T23:34:17.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.367+0000 7efd5307c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7efd4c1592d0 con 0x7efd4c0c8ed0 2024-03-28T23:34:17.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.367+0000 7efd5307c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7efd4c152df0 con 0x7efd4c0c5b20 2024-03-28T23:34:17.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.367+0000 7efd51619640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd4c152890 0x7efd4c1555a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:17.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.367+0000 7efd51619640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd4c152890 0x7efd4c1555a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45252/0 (socket says 172.21.15.161:45252) 2024-03-28T23:34:17.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.367+0000 7efd51619640 1 -- 172.21.15.161:0/278406928 learned_addr learned my addr 172.21.15.161:0/278406928 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:17.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.367+0000 7efd4bfff640 1 --2- 172.21.15.161:0/278406928 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd4c0c8ed0 0x7efd4c1ee570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:17.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.367+0000 7efd50e18640 1 --2- 172.21.15.161:0/278406928 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd4c0c5b20 0x7efd4c0c1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:17.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.367+0000 7efd51619640 1 -- 172.21.15.161:0/278406928 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd4c0c5b20 msgr2=0x7efd4c0c1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:17.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.367+0000 7efd51619640 1 --2- 172.21.15.161:0/278406928 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd4c0c5b20 0x7efd4c0c1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.367+0000 7efd51619640 1 -- 172.21.15.161:0/278406928 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd4c0c8ed0 msgr2=0x7efd4c1ee570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:17.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.367+0000 7efd51619640 1 --2- 172.21.15.161:0/278406928 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd4c0c8ed0 0x7efd4c1ee570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.367+0000 7efd51619640 1 -- 172.21.15.161:0/278406928 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd4c153ec0 con 0x7efd4c152890 2024-03-28T23:34:17.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.368+0000 7efd51619640 1 --2- 172.21.15.161:0/278406928 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd4c152890 0x7efd4c1555a0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7efd4402c3c0 tx=0x7efd44004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:17.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.368+0000 7efd497fa640 1 -- 172.21.15.161:0/278406928 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd44039050 con 0x7efd4c152890 2024-03-28T23:34:17.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.368+0000 7efd497fa640 1 -- 172.21.15.161:0/278406928 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd44045070 con 0x7efd4c152890 2024-03-28T23:34:17.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.368+0000 7efd5307c640 1 -- 172.21.15.161:0/278406928 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd4c14a290 con 0x7efd4c152890 2024-03-28T23:34:17.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.368+0000 7efd497fa640 1 -- 172.21.15.161:0/278406928 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd4403e850 con 0x7efd4c152890 2024-03-28T23:34:17.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.368+0000 7efd5307c640 1 -- 172.21.15.161:0/278406928 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd4c1ef020 con 0x7efd4c152890 2024-03-28T23:34:17.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.369+0000 7efd5307c640 1 -- 172.21.15.161:0/278406928 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd4c04f4b0 con 0x7efd4c152890 2024-03-28T23:34:17.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.369+0000 7efd497fa640 1 -- 172.21.15.161:0/278406928 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efd4402cd10 con 0x7efd4c152890 2024-03-28T23:34:17.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.370+0000 7efd497fa640 1 --2- 172.21.15.161:0/278406928 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efd3006b500 0x7efd3006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:17.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.370+0000 7efd50e18640 1 --2- 172.21.15.161:0/278406928 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efd3006b500 0x7efd3006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:17.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.370+0000 7efd497fa640 1 -- 172.21.15.161:0/278406928 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7efd440aceb0 con 0x7efd4c152890 2024-03-28T23:34:17.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.370+0000 7efd50e18640 1 --2- 172.21.15.161:0/278406928 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efd3006b500 0x7efd3006d9c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7efd4c0c98a0 tx=0x7efd40009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:17.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.375+0000 7efd497fa640 1 -- 172.21.15.161:0/278406928 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efd4407b6b0 con 0x7efd4c152890 2024-03-28T23:34:17.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.581+0000 7efd5307c640 1 -- 172.21.15.161:0/278406928 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd4c0c5900 con 0x7efd4c152890 2024-03-28T23:34:17.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.583+0000 7efd497fa640 1 -- 172.21.15.161:0/278406928 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+5202 (secure 0 0 0) 0x7efd4407f750 con 0x7efd4c152890 2024-03-28T23:34:17.583 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:17.584 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":100,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/1610475969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1610475969},{"type":"v1","addr":"172.21.15.161:6835","nonce":1610475969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":6892,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/2684760895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2684760895},{"type":"v1","addr":"172.21.15.161:6837","nonce":2684760895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":99,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T23:33:50.326492+0000","modified":"2024-03-28T23:34:10.064154+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6203":{"gid":6203,"name":"b","rank":0,"incarnation":92,"state":"up:active","state_seq":8,"addr":"172.21.15.170:6834/1096504848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1096504848},{"type":"v1","addr":"172.21.15.170:6834","nonce":1096504848}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6206":{"gid":6206,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":7,"addr":"172.21.15.170:6835/3351418222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3351418222},{"type":"v1","addr":"172.21.15.170:6835","nonce":3351418222}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6203,"qdb_cluster":[6203]},"id":8}]} 2024-03-28T23:34:17.584 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 100 2024-03-28T23:34:17.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.585+0000 7efd5307c640 1 -- 172.21.15.161:0/278406928 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efd3006b500 msgr2=0x7efd3006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:17.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.585+0000 7efd5307c640 1 --2- 172.21.15.161:0/278406928 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efd3006b500 0x7efd3006d9c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7efd4c0c98a0 tx=0x7efd40009190 comp rx=0 tx=0).stop 2024-03-28T23:34:17.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.585+0000 7efd5307c640 1 -- 172.21.15.161:0/278406928 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd4c152890 msgr2=0x7efd4c1555a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:17.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.585+0000 7efd5307c640 1 --2- 172.21.15.161:0/278406928 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd4c152890 0x7efd4c1555a0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7efd4402c3c0 tx=0x7efd44004a40 comp rx=0 tx=0).stop 2024-03-28T23:34:17.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.585+0000 7efd5307c640 1 -- 172.21.15.161:0/278406928 shutdown_connections 2024-03-28T23:34:17.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.585+0000 7efd5307c640 1 --2- 172.21.15.161:0/278406928 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efd3006b500 0x7efd3006d9c0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.585+0000 7efd5307c640 1 --2- 172.21.15.161:0/278406928 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd4c152890 0x7efd4c1555a0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.585+0000 7efd5307c640 1 --2- 172.21.15.161:0/278406928 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd4c0c8ed0 0x7efd4c1ee570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.585+0000 7efd5307c640 1 --2- 172.21.15.161:0/278406928 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd4c0c5b20 0x7efd4c0c1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.585+0000 7efd5307c640 1 -- 172.21.15.161:0/278406928 >> 172.21.15.161:0/278406928 conn(0x7efd4c0ac620 msgr2=0x7efd4c0ca090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:17.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.585+0000 7efd5307c640 1 -- 172.21.15.161:0/278406928 shutdown_connections 2024-03-28T23:34:17.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.585+0000 7efd5307c640 1 -- 172.21.15.161:0/278406928 wait complete. 2024-03-28T23:34:17.597 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:34:17.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.697+0000 7ff5163b5640 1 Processor -- start 2024-03-28T23:34:17.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.697+0000 7ff5163b5640 1 -- start start 2024-03-28T23:34:17.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.698+0000 7ff5163b5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff510150600 0x7ff5100a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:17.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.698+0000 7ff5163b5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5100aacb0 0x7ff510153980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:17.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.698+0000 7ff5163b5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5101567c0 con 0x7ff5100a8fc0 2024-03-28T23:34:17.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.698+0000 7ff5163b5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7ff510156570 con 0x7ff510150600 2024-03-28T23:34:17.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.698+0000 7ff5163b5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff510156aa0 con 0x7ff5100aacb0 2024-03-28T23:34:17.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.698+0000 7ff50f7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ff5100a8fc0 0x7ff510153270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52238/0 (socket says 172.21.15.161:52238) 2024-03-28T23:34:17.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.698+0000 7ff50f7fe640 1 -- 172.21.15.161:0/1247850500 learned_addr learned my addr 172.21.15.161:0/1247850500 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:17.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.698+0000 7ff50ffff640 1 --2- 172.21.15.161:0/1247850500 >> v2:172.21.15.170:3300/0 conn(0x7ff510150600 0x7ff5100a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:17.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.698+0000 7ff514952640 1 --2- 172.21.15.161:0/1247850500 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5100aacb0 0x7ff510153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:17.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.698+0000 7ff50effd640 1 -- 172.21.15.161:0/1247850500 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3186304372 0 0) 0x7ff5101567c0 con 0x7ff5100a8fc0 2024-03-28T23:34:17.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.698+0000 7ff50effd640 1 -- 172.21.15.161:0/1247850500 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4f8003520 con 0x7ff5100a8fc0 2024-03-28T23:34:17.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.698+0000 7ff50ffff640 1 -- 172.21.15.161:0/1247850500 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5100aacb0 msgr2=0x7ff510153980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:17.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.698+0000 7ff50ffff640 1 --2- 172.21.15.161:0/1247850500 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5100aacb0 0x7ff510153980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.698+0000 7ff50ffff640 1 -- 172.21.15.161:0/1247850500 >> v1:172.21.15.161:6789/0 conn(0x7ff5100a8fc0 legacy=0x7ff510153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:17.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.698+0000 7ff50ffff640 1 -- 172.21.15.161:0/1247850500 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff510156d80 con 0x7ff510150600 2024-03-28T23:34:17.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.699+0000 7ff50ffff640 1 --2- 172.21.15.161:0/1247850500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff510150600 0x7ff5100a9f40 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff5000097a0 tx=0x7ff50002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=292c1491f54555bc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:17.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.699+0000 7ff50effd640 1 -- 172.21.15.161:0/1247850500 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5000315a0 con 0x7ff510150600 2024-03-28T23:34:17.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.699+0000 7ff50effd640 1 -- 172.21.15.161:0/1247850500 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff500037070 con 0x7ff510150600 2024-03-28T23:34:17.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.699+0000 7ff50effd640 1 -- 172.21.15.161:0/1247850500 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff500031960 con 0x7ff510150600 2024-03-28T23:34:17.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.700+0000 7ff5163b5640 1 -- 172.21.15.161:0/1247850500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff510150600 msgr2=0x7ff5100a9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:17.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.700+0000 7ff5163b5640 1 --2- 172.21.15.161:0/1247850500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff510150600 0x7ff5100a9f40 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff5000097a0 tx=0x7ff50002b840 comp rx=0 tx=0).stop 2024-03-28T23:34:17.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.700+0000 7ff5163b5640 1 -- 172.21.15.161:0/1247850500 shutdown_connections 2024-03-28T23:34:17.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.700+0000 7ff5163b5640 1 --2- 172.21.15.161:0/1247850500 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5100aacb0 0x7ff510153980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.700+0000 7ff5163b5640 1 --2- 172.21.15.161:0/1247850500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff510150600 0x7ff5100a9f40 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.700+0000 7ff5163b5640 1 -- 172.21.15.161:0/1247850500 >> 172.21.15.161:0/1247850500 conn(0x7ff5100c6670 msgr2=0x7ff5100c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:17.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.700+0000 7ff5163b5640 1 -- 172.21.15.161:0/1247850500 shutdown_connections 2024-03-28T23:34:17.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.700+0000 7ff5163b5640 1 -- 172.21.15.161:0/1247850500 wait complete. 2024-03-28T23:34:17.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.701+0000 7ff5163b5640 1 Processor -- start 2024-03-28T23:34:17.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.701+0000 7ff5163b5640 1 -- start start 2024-03-28T23:34:17.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.702+0000 7ff5163b5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5100a8fc0 0x7ff510152260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:17.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.702+0000 7ff5163b5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5100aacb0 0x7ff5101edb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:17.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.702+0000 7ff5163b5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff510150600 0x7ff5101ee070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:17.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.702+0000 7ff5163b5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff510153ec0 con 0x7ff5100a8fc0 2024-03-28T23:34:17.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.702+0000 7ff5163b5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff510157340 con 0x7ff5100aacb0 2024-03-28T23:34:17.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.702+0000 7ff5163b5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff5101509e0 con 0x7ff510150600 2024-03-28T23:34:17.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.702+0000 7ff50ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5100a8fc0 0x7ff510152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:17.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.702+0000 7ff50ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5100a8fc0 0x7ff510152260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45262/0 (socket says 172.21.15.161:45262) 2024-03-28T23:34:17.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.702+0000 7ff50ffff640 1 -- 172.21.15.161:0/1652818262 learned_addr learned my addr 172.21.15.161:0/1652818262 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:17.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.702+0000 7ff50f7fe640 1 --2- 172.21.15.161:0/1652818262 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5100aacb0 0x7ff5101edb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:17.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.702+0000 7ff514952640 1 --2- 172.21.15.161:0/1652818262 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff510150600 0x7ff5101ee070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:17.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.702+0000 7ff50ffff640 1 -- 172.21.15.161:0/1652818262 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff510150600 msgr2=0x7ff5101ee070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:17.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.702+0000 7ff50ffff640 1 --2- 172.21.15.161:0/1652818262 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff510150600 0x7ff5101ee070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.702+0000 7ff50ffff640 1 -- 172.21.15.161:0/1652818262 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5100aacb0 msgr2=0x7ff5101edb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:17.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.702+0000 7ff50ffff640 1 --2- 172.21.15.161:0/1652818262 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5100aacb0 0x7ff5101edb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.702+0000 7ff50ffff640 1 -- 172.21.15.161:0/1652818262 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff51006bb60 con 0x7ff5100a8fc0 2024-03-28T23:34:17.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.702+0000 7ff50ffff640 1 --2- 172.21.15.161:0/1652818262 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5100a8fc0 0x7ff510152260 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7ff5000098c0 tx=0x7ff5000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:17.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.702+0000 7ff50cff9640 1 -- 172.21.15.161:0/1652818262 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff500032630 con 0x7ff5100a8fc0 2024-03-28T23:34:17.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.702+0000 7ff50cff9640 1 -- 172.21.15.161:0/1652818262 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5000327f0 con 0x7ff5100a8fc0 2024-03-28T23:34:17.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.703+0000 7ff5163b5640 1 -- 172.21.15.161:0/1652818262 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5100c0820 con 0x7ff5100a8fc0 2024-03-28T23:34:17.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.703+0000 7ff50cff9640 1 -- 172.21.15.161:0/1652818262 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff500032be0 con 0x7ff5100a8fc0 2024-03-28T23:34:17.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.703+0000 7ff5163b5640 1 -- 172.21.15.161:0/1652818262 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff51006c020 con 0x7ff5100a8fc0 2024-03-28T23:34:17.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.704+0000 7ff5163b5640 1 -- 172.21.15.161:0/1652818262 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff51006be10 con 0x7ff5100a8fc0 2024-03-28T23:34:17.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.704+0000 7ff50cff9640 1 -- 172.21.15.161:0/1652818262 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff50004e430 con 0x7ff5100a8fc0 2024-03-28T23:34:17.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.704+0000 7ff50cff9640 1 --2- 172.21.15.161:0/1652818262 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff4fc06b500 0x7ff4fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:17.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.704+0000 7ff50cff9640 1 -- 172.21.15.161:0/1652818262 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7ff50002f030 con 0x7ff5100a8fc0 2024-03-28T23:34:17.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.704+0000 7ff50f7fe640 1 --2- 172.21.15.161:0/1652818262 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff4fc06b500 0x7ff4fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:17.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.705+0000 7ff50f7fe640 1 --2- 172.21.15.161:0/1652818262 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff4fc06b500 0x7ff4fc06d9c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff5100bfab0 tx=0x7ff504009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:17.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.709+0000 7ff50cff9640 1 -- 172.21.15.161:0/1652818262 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff500040070 con 0x7ff5100a8fc0 2024-03-28T23:34:17.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.880+0000 7ff5163b5640 1 -- 172.21.15.161:0/1652818262 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff51004f4b0 con 0x7ff5100a8fc0 2024-03-28T23:34:17.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.882+0000 7ff50cff9640 1 -- 172.21.15.161:0/1652818262 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) ==== 74+0+17119 (secure 0 0 0) 0x7ff50007a690 con 0x7ff5100a8fc0 2024-03-28T23:34:17.882 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:17.883 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":110,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:34:02.034225+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:33:48.560815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-28T23:33:49.301625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6835/2544838511":"2024-03-29T23:34:02.034196+0000","172.21.15.161:6834/2544838511":"2024-03-29T23:34:02.034196+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:34:17.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.884+0000 7ff5163b5640 1 -- 172.21.15.161:0/1652818262 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff4fc06b500 msgr2=0x7ff4fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:17.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.884+0000 7ff5163b5640 1 --2- 172.21.15.161:0/1652818262 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff4fc06b500 0x7ff4fc06d9c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff5100bfab0 tx=0x7ff504009040 comp rx=0 tx=0).stop 2024-03-28T23:34:17.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.884+0000 7ff5163b5640 1 -- 172.21.15.161:0/1652818262 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5100a8fc0 msgr2=0x7ff510152260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:17.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.884+0000 7ff5163b5640 1 --2- 172.21.15.161:0/1652818262 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5100a8fc0 0x7ff510152260 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7ff5000098c0 tx=0x7ff5000046c0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.884+0000 7ff5163b5640 1 -- 172.21.15.161:0/1652818262 shutdown_connections 2024-03-28T23:34:17.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.884+0000 7ff5163b5640 1 --2- 172.21.15.161:0/1652818262 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff4fc06b500 0x7ff4fc06d9c0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.884+0000 7ff5163b5640 1 --2- 172.21.15.161:0/1652818262 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff510150600 0x7ff5101ee070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.884+0000 7ff5163b5640 1 --2- 172.21.15.161:0/1652818262 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5100aacb0 0x7ff5101edb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.884+0000 7ff5163b5640 1 --2- 172.21.15.161:0/1652818262 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5100a8fc0 0x7ff510152260 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.884+0000 7ff5163b5640 1 -- 172.21.15.161:0/1652818262 >> 172.21.15.161:0/1652818262 conn(0x7ff5100c6670 msgr2=0x7ff5100c2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:17.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.884+0000 7ff5163b5640 1 -- 172.21.15.161:0/1652818262 shutdown_connections 2024-03-28T23:34:17.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.884+0000 7ff5163b5640 1 -- 172.21.15.161:0/1652818262 wait complete. 2024-03-28T23:34:17.895 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-28T23:34:17.896 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:34:17.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.996+0000 7f86138ed640 1 Processor -- start 2024-03-28T23:34:17.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.997+0000 7f86138ed640 1 -- start start 2024-03-28T23:34:17.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.997+0000 7f86138ed640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f860c152890 0x7f860c0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:17.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.997+0000 7f86138ed640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f860c0c8ed0 0x7f860c0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:17.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.997+0000 7f86138ed640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f860c158a30 con 0x7f860c0c5950 2024-03-28T23:34:17.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.997+0000 7f86138ed640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f860c0c8480 con 0x7f860c0c8ed0 2024-03-28T23:34:17.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.997+0000 7f86138ed640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f860c159640 con 0x7f860c152890 2024-03-28T23:34:17.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.997+0000 7f8610e88640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f860c0c8ed0 0x7f860c0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:17.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.997+0000 7f8610e88640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f860c0c8ed0 0x7f860c0c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45268/0 (socket says 172.21.15.161:45268) 2024-03-28T23:34:17.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.997+0000 7f8611689640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f860c152890 0x7f860c0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:17.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.997+0000 7f8610e88640 1 -- 172.21.15.161:0/2768527737 learned_addr learned my addr 172.21.15.161:0/2768527737 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:17.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.998+0000 7f8611689640 1 -- 172.21.15.161:0/2768527737 >> v2:172.21.15.161:3300/0 conn(0x7f860c0c8ed0 msgr2=0x7f860c0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:17.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.998+0000 7f8611689640 1 --2- 172.21.15.161:0/2768527737 >> v2:172.21.15.161:3300/0 conn(0x7f860c0c8ed0 0x7f860c0c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:17.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.998+0000 7f8611689640 1 -- 172.21.15.161:0/2768527737 >> v1:172.21.15.170:6789/0 conn(0x7f860c0c5950 legacy=0x7f860c1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:17.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.998+0000 7f8611689640 1 -- 172.21.15.161:0/2768527737 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860c158d10 con 0x7f860c152890 2024-03-28T23:34:17.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.998+0000 7f8603fff640 1 -- 172.21.15.161:0/2768527737 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 292853787 0 0) 0x7f860c158a30 con 0x7f860c0c5950 2024-03-28T23:34:17.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.998+0000 7f8610e88640 1 --2- 172.21.15.161:0/2768527737 >> v2:172.21.15.161:3300/0 conn(0x7f860c0c8ed0 0x7f860c0c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:34:18.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.999+0000 7f8611689640 1 --2- 172.21.15.161:0/2768527737 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f860c152890 0x7f860c0c8990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f860c08cc80 tx=0x7f85fc02b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16d3bc7e72a99122 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:18.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.999+0000 7f8603fff640 1 -- 172.21.15.161:0/2768527737 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85fc038070 con 0x7f860c152890 2024-03-28T23:34:18.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.999+0000 7f8603fff640 1 -- 172.21.15.161:0/2768527737 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85fc031710 con 0x7f860c152890 2024-03-28T23:34:18.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.999+0000 7f8603fff640 1 -- 172.21.15.161:0/2768527737 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85fc031ae0 con 0x7f860c152890 2024-03-28T23:34:18.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.999+0000 7f86138ed640 1 -- 172.21.15.161:0/2768527737 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f860c152890 msgr2=0x7f860c0c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:18.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.999+0000 7f86138ed640 1 --2- 172.21.15.161:0/2768527737 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f860c152890 0x7f860c0c8990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f860c08cc80 tx=0x7f85fc02b7b0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.999+0000 7f86138ed640 1 -- 172.21.15.161:0/2768527737 shutdown_connections 2024-03-28T23:34:18.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.999+0000 7f86138ed640 1 --2- 172.21.15.161:0/2768527737 >> v2:172.21.15.161:3300/0 conn(0x7f860c0c8ed0 0x7f860c0c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.999+0000 7f86138ed640 1 --2- 172.21.15.161:0/2768527737 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f860c152890 0x7f860c0c8990 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.999+0000 7f86138ed640 1 -- 172.21.15.161:0/2768527737 >> 172.21.15.161:0/2768527737 conn(0x7f860c0ac620 msgr2=0x7f860c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:18.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.999+0000 7f86138ed640 1 -- 172.21.15.161:0/2768527737 shutdown_connections 2024-03-28T23:34:18.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:17.999+0000 7f86138ed640 1 -- 172.21.15.161:0/2768527737 wait complete. 2024-03-28T23:34:18.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.000+0000 7f86138ed640 1 Processor -- start 2024-03-28T23:34:18.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.000+0000 7f86138ed640 1 -- start start 2024-03-28T23:34:18.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.001+0000 7f86138ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f860c0c5950 0x7f860c0c41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:18.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.001+0000 7f86138ed640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f860c0c8ed0 0x7f860c157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:18.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.001+0000 7f86138ed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f860c152890 0x7f860c147c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:18.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.001+0000 7f86138ed640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f860c152c70 con 0x7f860c0c8ed0 2024-03-28T23:34:18.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.001+0000 7f86138ed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f860c1592d0 con 0x7f860c152890 2024-03-28T23:34:18.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.001+0000 7f86138ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f860c152df0 con 0x7f860c0c5950 2024-03-28T23:34:18.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.001+0000 7f8610e88640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f860c0c8ed0 0x7f860c157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:18.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.001+0000 7f8610e88640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f860c0c8ed0 0x7f860c157ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45280/0 (socket says 172.21.15.161:45280) 2024-03-28T23:34:18.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.001+0000 7f8610e88640 1 -- 172.21.15.161:0/307727331 learned_addr learned my addr 172.21.15.161:0/307727331 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:18.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.001+0000 7f8611689640 1 --2- 172.21.15.161:0/307727331 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f860c0c5950 0x7f860c0c41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:18.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.001+0000 7f8611e8a640 1 --2- 172.21.15.161:0/307727331 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f860c152890 0x7f860c147c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:18.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.001+0000 7f8610e88640 1 -- 172.21.15.161:0/307727331 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f860c0c5950 msgr2=0x7f860c0c41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:18.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.001+0000 7f8610e88640 1 --2- 172.21.15.161:0/307727331 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f860c0c5950 0x7f860c0c41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.001+0000 7f8610e88640 1 -- 172.21.15.161:0/307727331 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f860c152890 msgr2=0x7f860c147c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:18.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.001+0000 7f8610e88640 1 --2- 172.21.15.161:0/307727331 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f860c152890 0x7f860c147c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.001+0000 7f8610e88640 1 -- 172.21.15.161:0/307727331 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860c157700 con 0x7f860c0c8ed0 2024-03-28T23:34:18.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.002+0000 7f8610e88640 1 --2- 172.21.15.161:0/307727331 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f860c0c8ed0 0x7f860c157ba0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f860800e990 tx=0x7f860800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:18.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.002+0000 7f8601ffb640 1 -- 172.21.15.161:0/307727331 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860800c9e0 con 0x7f860c0c8ed0 2024-03-28T23:34:18.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.002+0000 7f8601ffb640 1 -- 172.21.15.161:0/307727331 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f860800cb80 con 0x7f860c0c8ed0 2024-03-28T23:34:18.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.002+0000 7f8601ffb640 1 -- 172.21.15.161:0/307727331 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8608005990 con 0x7f860c0c8ed0 2024-03-28T23:34:18.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.002+0000 7f86138ed640 1 -- 172.21.15.161:0/307727331 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f860c0abac0 con 0x7f860c0c8ed0 2024-03-28T23:34:18.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.002+0000 7f86138ed640 1 -- 172.21.15.161:0/307727331 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f860c156c20 con 0x7f860c0c8ed0 2024-03-28T23:34:18.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.003+0000 7f86138ed640 1 -- 172.21.15.161:0/307727331 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f860c156a10 con 0x7f860c0c8ed0 2024-03-28T23:34:18.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.003+0000 7f8601ffb640 1 -- 172.21.15.161:0/307727331 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8608005b30 con 0x7f860c0c8ed0 2024-03-28T23:34:18.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.004+0000 7f8601ffb640 1 --2- 172.21.15.161:0/307727331 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f85f006b4d0 0x7f85f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:18.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.004+0000 7f8601ffb640 1 -- 172.21.15.161:0/307727331 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7f860808a660 con 0x7f860c0c8ed0 2024-03-28T23:34:18.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.004+0000 7f8611689640 1 --2- 172.21.15.161:0/307727331 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f85f006b4d0 0x7f85f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:18.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.004+0000 7f8611689640 1 --2- 172.21.15.161:0/307727331 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f85f006b4d0 0x7f85f006d990 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f85fc035000 tx=0x7f85fc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:18.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.008+0000 7f8601ffb640 1 -- 172.21.15.161:0/307727331 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8608017020 con 0x7f860c0c8ed0 2024-03-28T23:34:18.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.219+0000 7f86138ed640 1 -- 172.21.15.161:0/307727331 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f860c0ac400 con 0x7f860c0c8ed0 2024-03-28T23:34:18.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.220+0000 7f8601ffb640 1 -- 172.21.15.161:0/307727331 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+5202 (secure 0 0 0) 0x7f8608058d40 con 0x7f860c0c8ed0 2024-03-28T23:34:18.221 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:18.221 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":100,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/1610475969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1610475969},{"type":"v1","addr":"172.21.15.161:6835","nonce":1610475969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":6892,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/2684760895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2684760895},{"type":"v1","addr":"172.21.15.161:6837","nonce":2684760895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":99,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T23:33:50.326492+0000","modified":"2024-03-28T23:34:10.064154+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6203":{"gid":6203,"name":"b","rank":0,"incarnation":92,"state":"up:active","state_seq":8,"addr":"172.21.15.170:6834/1096504848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1096504848},{"type":"v1","addr":"172.21.15.170:6834","nonce":1096504848}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6206":{"gid":6206,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":7,"addr":"172.21.15.170:6835/3351418222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3351418222},{"type":"v1","addr":"172.21.15.170:6835","nonce":3351418222}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6203,"qdb_cluster":[6203]},"id":8}]} 2024-03-28T23:34:18.221 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 100 2024-03-28T23:34:18.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.222+0000 7f86138ed640 1 -- 172.21.15.161:0/307727331 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f85f006b4d0 msgr2=0x7f85f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:18.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.222+0000 7f86138ed640 1 --2- 172.21.15.161:0/307727331 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f85f006b4d0 0x7f85f006d990 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f85fc035000 tx=0x7f85fc008030 comp rx=0 tx=0).stop 2024-03-28T23:34:18.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.222+0000 7f86138ed640 1 -- 172.21.15.161:0/307727331 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f860c0c8ed0 msgr2=0x7f860c157ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:18.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.222+0000 7f86138ed640 1 --2- 172.21.15.161:0/307727331 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f860c0c8ed0 0x7f860c157ba0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f860800e990 tx=0x7f860800ee50 comp rx=0 tx=0).stop 2024-03-28T23:34:18.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.222+0000 7f86138ed640 1 -- 172.21.15.161:0/307727331 shutdown_connections 2024-03-28T23:34:18.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.222+0000 7f86138ed640 1 --2- 172.21.15.161:0/307727331 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f85f006b4d0 0x7f85f006d990 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.222+0000 7f86138ed640 1 --2- 172.21.15.161:0/307727331 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f860c0c8ed0 0x7f860c157ba0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.222+0000 7f86138ed640 1 --2- 172.21.15.161:0/307727331 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f860c152890 0x7f860c147c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.222+0000 7f86138ed640 1 --2- 172.21.15.161:0/307727331 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f860c0c5950 0x7f860c0c41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.222+0000 7f86138ed640 1 -- 172.21.15.161:0/307727331 >> 172.21.15.161:0/307727331 conn(0x7f860c0ac620 msgr2=0x7f860c0c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:18.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.222+0000 7f86138ed640 1 -- 172.21.15.161:0/307727331 shutdown_connections 2024-03-28T23:34:18.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.222+0000 7f86138ed640 1 -- 172.21.15.161:0/307727331 wait complete. 2024-03-28T23:34:18.234 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:34:18.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.334+0000 7f1b27c6d640 1 Processor -- start 2024-03-28T23:34:18.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.334+0000 7f1b27c6d640 1 -- start start 2024-03-28T23:34:18.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.335+0000 7f1b27c6d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1b2006ce80 0x7f1b200aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:18.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.335+0000 7f1b27c6d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1b200a9680 0x7f1b200aab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:18.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.335+0000 7f1b27c6d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b2014c250 con 0x7f1b2014be70 2024-03-28T23:34:18.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.335+0000 7f1b27c6d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1b201529c0 con 0x7f1b200a9680 2024-03-28T23:34:18.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.335+0000 7f1b27c6d640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f1b200a9dc0 con 0x7f1b2006ce80 2024-03-28T23:34:18.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.335+0000 7f1b2620a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1b200a9680 0x7f1b200aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:18.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.335+0000 7f1b2620a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1b200a9680 0x7f1b200aab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45282/0 (socket says 172.21.15.161:45282) 2024-03-28T23:34:18.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.335+0000 7f1b2620a640 1 -- 172.21.15.161:0/2375333481 learned_addr learned my addr 172.21.15.161:0/2375333481 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:18.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.335+0000 7f1b25208640 1 --2- 172.21.15.161:0/2375333481 >> v2:172.21.15.170:3300/0 conn(0x7f1b2006ce80 0x7f1b200aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:18.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.335+0000 7f1b2620a640 1 -- 172.21.15.161:0/2375333481 >> v2:172.21.15.170:3300/0 conn(0x7f1b2006ce80 msgr2=0x7f1b200aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:18.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.335+0000 7f1b2620a640 1 --2- 172.21.15.161:0/2375333481 >> v2:172.21.15.170:3300/0 conn(0x7f1b2006ce80 0x7f1b200aa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.335+0000 7f1b2620a640 1 -- 172.21.15.161:0/2375333481 >> v1:172.21.15.170:6789/0 conn(0x7f1b2014be70 legacy=0x7f1b200ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:18.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.335+0000 7f1b2620a640 1 -- 172.21.15.161:0/2375333481 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b200aa0a0 con 0x7f1b200a9680 2024-03-28T23:34:18.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.336+0000 7f1b2620a640 1 --2- 172.21.15.161:0/2375333481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b200a9680 0x7f1b200aab00 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f1b2006f540 tx=0x7f1b1802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2258e0a468a581ef server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:18.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.336+0000 7f1b24a07640 1 -- 172.21.15.161:0/2375333481 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1802c9f0 con 0x7f1b200a9680 2024-03-28T23:34:18.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.336+0000 7f1b24a07640 1 -- 172.21.15.161:0/2375333481 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b1802cb90 con 0x7f1b200a9680 2024-03-28T23:34:18.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.336+0000 7f1b24a07640 1 -- 172.21.15.161:0/2375333481 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b180343f0 con 0x7f1b200a9680 2024-03-28T23:34:18.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.336+0000 7f1b27c6d640 1 -- 172.21.15.161:0/2375333481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b200a9680 msgr2=0x7f1b200aab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:18.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.336+0000 7f1b27c6d640 1 --2- 172.21.15.161:0/2375333481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b200a9680 0x7f1b200aab00 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f1b2006f540 tx=0x7f1b1802be50 comp rx=0 tx=0).stop 2024-03-28T23:34:18.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.336+0000 7f1b27c6d640 1 -- 172.21.15.161:0/2375333481 shutdown_connections 2024-03-28T23:34:18.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.336+0000 7f1b27c6d640 1 --2- 172.21.15.161:0/2375333481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b200a9680 0x7f1b200aab00 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.336+0000 7f1b27c6d640 1 --2- 172.21.15.161:0/2375333481 >> v2:172.21.15.170:3300/0 conn(0x7f1b2006ce80 0x7f1b200aa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.336+0000 7f1b27c6d640 1 -- 172.21.15.161:0/2375333481 >> 172.21.15.161:0/2375333481 conn(0x7f1b200c6670 msgr2=0x7f1b200c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:18.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.337+0000 7f1b27c6d640 1 -- 172.21.15.161:0/2375333481 shutdown_connections 2024-03-28T23:34:18.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.337+0000 7f1b27c6d640 1 -- 172.21.15.161:0/2375333481 wait complete. 2024-03-28T23:34:18.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.338+0000 7f1b27c6d640 1 Processor -- start 2024-03-28T23:34:18.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.338+0000 7f1b27c6d640 1 -- start start 2024-03-28T23:34:18.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.338+0000 7f1b27c6d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b2006ce80 0x7f1b200bfc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:18.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.338+0000 7f1b27c6d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b200a9680 0x7f1b20154cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:18.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.338+0000 7f1b27c6d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b2014be70 0x7f1b201551f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:18.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.338+0000 7f1b27c6d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1b200aba30 con 0x7f1b200a9680 2024-03-28T23:34:18.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.338+0000 7f1b27c6d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1b200ab8b0 con 0x7f1b2014be70 2024-03-28T23:34:18.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.338+0000 7f1b27c6d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1b2006b150 con 0x7f1b2006ce80 2024-03-28T23:34:18.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.338+0000 7f1b25208640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b200a9680 0x7f1b20154cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.338+0000 7f1b25208640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b200a9680 0x7f1b20154cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45288/0 (socket says 172.21.15.161:45288) 2024-03-28T23:34:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.338+0000 7f1b25208640 1 -- 172.21.15.161:0/1450843422 learned_addr learned my addr 172.21.15.161:0/1450843422 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.338+0000 7f1b2620a640 1 --2- 172.21.15.161:0/1450843422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b2014be70 0x7f1b201551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.338+0000 7f1b25a09640 1 --2- 172.21.15.161:0/1450843422 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b2006ce80 0x7f1b200bfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.339+0000 7f1b25208640 1 -- 172.21.15.161:0/1450843422 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b2006ce80 msgr2=0x7f1b200bfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.339+0000 7f1b25208640 1 --2- 172.21.15.161:0/1450843422 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b2006ce80 0x7f1b200bfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.339+0000 7f1b25208640 1 -- 172.21.15.161:0/1450843422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b2014be70 msgr2=0x7f1b201551f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.339+0000 7f1b25208640 1 --2- 172.21.15.161:0/1450843422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b2014be70 0x7f1b201551f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.339+0000 7f1b25208640 1 -- 172.21.15.161:0/1450843422 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b201eb500 con 0x7f1b200a9680 2024-03-28T23:34:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.339+0000 7f1b25208640 1 --2- 172.21.15.161:0/1450843422 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b200a9680 0x7f1b20154cb0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f1b1000e870 tx=0x7f1b1000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.339+0000 7f1b167fc640 1 -- 172.21.15.161:0/1450843422 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b10017070 con 0x7f1b200a9680 2024-03-28T23:34:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.339+0000 7f1b167fc640 1 -- 172.21.15.161:0/1450843422 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b1000c940 con 0x7f1b200a9680 2024-03-28T23:34:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.339+0000 7f1b27c6d640 1 -- 172.21.15.161:0/1450843422 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b201eb7e0 con 0x7f1b200a9680 2024-03-28T23:34:18.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.339+0000 7f1b167fc640 1 -- 172.21.15.161:0/1450843422 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1000cd30 con 0x7f1b200a9680 2024-03-28T23:34:18.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.339+0000 7f1b27c6d640 1 -- 172.21.15.161:0/1450843422 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b20154790 con 0x7f1b200a9680 2024-03-28T23:34:18.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.340+0000 7f1b27c6d640 1 -- 172.21.15.161:0/1450843422 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b2004f4b0 con 0x7f1b200a9680 2024-03-28T23:34:18.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.340+0000 7f1b167fc640 1 -- 172.21.15.161:0/1450843422 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1b1001a070 con 0x7f1b200a9680 2024-03-28T23:34:18.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.341+0000 7f1b167fc640 1 --2- 172.21.15.161:0/1450843422 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1b0406b4d0 0x7f1b0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:18.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.341+0000 7f1b167fc640 1 -- 172.21.15.161:0/1450843422 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7f1b1008a4c0 con 0x7f1b200a9680 2024-03-28T23:34:18.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.341+0000 7f1b25a09640 1 --2- 172.21.15.161:0/1450843422 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1b0406b4d0 0x7f1b0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:18.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.341+0000 7f1b25a09640 1 --2- 172.21.15.161:0/1450843422 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1b0406b4d0 0x7f1b0406d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c0016e0 tx=0x7f1b1c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:18.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.345+0000 7f1b167fc640 1 -- 172.21.15.161:0/1450843422 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1b10058ba0 con 0x7f1b200a9680 2024-03-28T23:34:18.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.552+0000 7f1b27c6d640 1 -- 172.21.15.161:0/1450843422 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b200c0ff0 con 0x7f1b200a9680 2024-03-28T23:34:18.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.553+0000 7f1b167fc640 1 -- 172.21.15.161:0/1450843422 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+5202 (secure 0 0 0) 0x7f1b1005cc40 con 0x7f1b200a9680 2024-03-28T23:34:18.553 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:18.553 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":100,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/1610475969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1610475969},{"type":"v1","addr":"172.21.15.161:6835","nonce":1610475969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":6892,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/2684760895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2684760895},{"type":"v1","addr":"172.21.15.161:6837","nonce":2684760895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":99,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T23:33:50.326492+0000","modified":"2024-03-28T23:34:10.064154+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6203":{"gid":6203,"name":"b","rank":0,"incarnation":92,"state":"up:active","state_seq":8,"addr":"172.21.15.170:6834/1096504848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1096504848},{"type":"v1","addr":"172.21.15.170:6834","nonce":1096504848}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6206":{"gid":6206,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":7,"addr":"172.21.15.170:6835/3351418222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3351418222},{"type":"v1","addr":"172.21.15.170:6835","nonce":3351418222}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6203,"qdb_cluster":[6203]},"id":8}]} 2024-03-28T23:34:18.554 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 100 2024-03-28T23:34:18.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.555+0000 7f1b27c6d640 1 -- 172.21.15.161:0/1450843422 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1b0406b4d0 msgr2=0x7f1b0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.555+0000 7f1b27c6d640 1 --2- 172.21.15.161:0/1450843422 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1b0406b4d0 0x7f1b0406d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c0016e0 tx=0x7f1b1c009040 comp rx=0 tx=0).stop 2024-03-28T23:34:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.555+0000 7f1b27c6d640 1 -- 172.21.15.161:0/1450843422 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b200a9680 msgr2=0x7f1b20154cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.555+0000 7f1b27c6d640 1 --2- 172.21.15.161:0/1450843422 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b200a9680 0x7f1b20154cb0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f1b1000e870 tx=0x7f1b1000ed30 comp rx=0 tx=0).stop 2024-03-28T23:34:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.555+0000 7f1b27c6d640 1 -- 172.21.15.161:0/1450843422 shutdown_connections 2024-03-28T23:34:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.555+0000 7f1b27c6d640 1 --2- 172.21.15.161:0/1450843422 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1b0406b4d0 0x7f1b0406d990 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.555+0000 7f1b27c6d640 1 --2- 172.21.15.161:0/1450843422 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b200a9680 0x7f1b20154cb0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.555+0000 7f1b27c6d640 1 --2- 172.21.15.161:0/1450843422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b2014be70 0x7f1b201551f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.555+0000 7f1b27c6d640 1 --2- 172.21.15.161:0/1450843422 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b2006ce80 0x7f1b200bfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.555+0000 7f1b27c6d640 1 -- 172.21.15.161:0/1450843422 >> 172.21.15.161:0/1450843422 conn(0x7f1b200c6670 msgr2=0x7f1b200731b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.555+0000 7f1b27c6d640 1 -- 172.21.15.161:0/1450843422 shutdown_connections 2024-03-28T23:34:18.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.555+0000 7f1b27c6d640 1 -- 172.21.15.161:0/1450843422 wait complete. 2024-03-28T23:34:18.566 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:34:18.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.669+0000 7f03145d0640 1 Processor -- start 2024-03-28T23:34:18.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.669+0000 7f03145d0640 1 -- start start 2024-03-28T23:34:18.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.670+0000 7f03145d0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f030c0a8fc0 0x7f030c153270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:18.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.670+0000 7f03145d0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f030c1567c0 con 0x7f030c0aacb0 2024-03-28T23:34:18.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.670+0000 7f03145d0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f030c156aa0 con 0x7f030c150600 2024-03-28T23:34:18.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.670+0000 7f03145d0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f030c156570 con 0x7f030c0a8fc0 2024-03-28T23:34:18.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.670+0000 7f0312b6d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f030c0aacb0 0x7f030c1537b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52244/0 (socket says 172.21.15.161:52244) 2024-03-28T23:34:18.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.670+0000 7f0312b6d640 1 -- 172.21.15.161:0/2389432978 learned_addr learned my addr 172.21.15.161:0/2389432978 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:18.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.670+0000 7f0311b6b640 1 --2- 172.21.15.161:0/2389432978 >> v2:172.21.15.170:3300/0 conn(0x7f030c0a8fc0 0x7f030c153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:18.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.670+0000 7f031136a640 1 -- 172.21.15.161:0/2389432978 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 981190343 0 0) 0x7f030c1567c0 con 0x7f030c0aacb0 2024-03-28T23:34:18.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.670+0000 7f031136a640 1 -- 172.21.15.161:0/2389432978 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02f8003520 con 0x7f030c0aacb0 2024-03-28T23:34:18.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.670+0000 7f031136a640 1 -- 172.21.15.161:0/2389432978 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1421446406 0 0) 0x7f030c156aa0 con 0x7f030c150600 2024-03-28T23:34:18.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.670+0000 7f031136a640 1 -- 172.21.15.161:0/2389432978 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f030c1567c0 con 0x7f030c150600 2024-03-28T23:34:18.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.670+0000 7f0311b6b640 1 -- 172.21.15.161:0/2389432978 >> v1:172.21.15.170:6789/0 conn(0x7f030c150600 legacy=0x7f030c0a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:18.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.670+0000 7f0311b6b640 1 -- 172.21.15.161:0/2389432978 >> v1:172.21.15.161:6789/0 conn(0x7f030c0aacb0 legacy=0x7f030c1537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:18.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.670+0000 7f0311b6b640 1 -- 172.21.15.161:0/2389432978 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f030c156d80 con 0x7f030c0a8fc0 2024-03-28T23:34:18.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.671+0000 7f0311b6b640 1 --2- 172.21.15.161:0/2389432978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f030c0a8fc0 0x7f030c153270 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f02fc0097a0 tx=0x7f02fc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5322c26f518c9e9b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:18.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.671+0000 7f031136a640 1 -- 172.21.15.161:0/2389432978 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02fc035020 con 0x7f030c0a8fc0 2024-03-28T23:34:18.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.671+0000 7f031136a640 1 -- 172.21.15.161:0/2389432978 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02fc031830 con 0x7f030c0a8fc0 2024-03-28T23:34:18.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.671+0000 7f031136a640 1 -- 172.21.15.161:0/2389432978 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02fc031be0 con 0x7f030c0a8fc0 2024-03-28T23:34:18.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.672+0000 7f03145d0640 1 -- 172.21.15.161:0/2389432978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f030c0a8fc0 msgr2=0x7f030c153270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:18.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.672+0000 7f03145d0640 1 --2- 172.21.15.161:0/2389432978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f030c0a8fc0 0x7f030c153270 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f02fc0097a0 tx=0x7f02fc02b8d0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.672+0000 7f03145d0640 1 -- 172.21.15.161:0/2389432978 shutdown_connections 2024-03-28T23:34:18.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.672+0000 7f03145d0640 1 --2- 172.21.15.161:0/2389432978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f030c0a8fc0 0x7f030c153270 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.672+0000 7f03145d0640 1 -- 172.21.15.161:0/2389432978 >> 172.21.15.161:0/2389432978 conn(0x7f030c0c6670 msgr2=0x7f030c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:18.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.672+0000 7f03145d0640 1 -- 172.21.15.161:0/2389432978 shutdown_connections 2024-03-28T23:34:18.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.672+0000 7f03145d0640 1 -- 172.21.15.161:0/2389432978 wait complete. 2024-03-28T23:34:18.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.673+0000 7f03145d0640 1 Processor -- start 2024-03-28T23:34:18.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.673+0000 7f03145d0640 1 -- start start 2024-03-28T23:34:18.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.673+0000 7f03145d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f030c0a8fc0 0x7f030c152800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:18.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.673+0000 7f03145d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f030c0aacb0 0x7f030c14c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:18.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.673+0000 7f03145d0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f030c150600 0x7f030c1e90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:18.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.673+0000 7f03145d0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f030c153ec0 con 0x7f030c150600 2024-03-28T23:34:18.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.673+0000 7f03145d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f030c157340 con 0x7f030c0aacb0 2024-03-28T23:34:18.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.673+0000 7f03145d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f030c1509e0 con 0x7f030c0a8fc0 2024-03-28T23:34:18.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.674+0000 7f0312b6d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f030c150600 0x7f030c1e90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:18.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.674+0000 7f0312b6d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f030c150600 0x7f030c1e90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45300/0 (socket says 172.21.15.161:45300) 2024-03-28T23:34:18.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.674+0000 7f0312b6d640 1 -- 172.21.15.161:0/3493211218 learned_addr learned my addr 172.21.15.161:0/3493211218 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:18.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.674+0000 7f0311b6b640 1 --2- 172.21.15.161:0/3493211218 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f030c0aacb0 0x7f030c14c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:18.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.674+0000 7f031236c640 1 --2- 172.21.15.161:0/3493211218 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f030c0a8fc0 0x7f030c152800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:18.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.674+0000 7f0312b6d640 1 -- 172.21.15.161:0/3493211218 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f030c0a8fc0 msgr2=0x7f030c152800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:18.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.674+0000 7f0312b6d640 1 --2- 172.21.15.161:0/3493211218 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f030c0a8fc0 0x7f030c152800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.674+0000 7f0312b6d640 1 -- 172.21.15.161:0/3493211218 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f030c0aacb0 msgr2=0x7f030c14c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:18.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.674+0000 7f0312b6d640 1 --2- 172.21.15.161:0/3493211218 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f030c0aacb0 0x7f030c14c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.674+0000 7f0312b6d640 1 -- 172.21.15.161:0/3493211218 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f030c14ab60 con 0x7f030c150600 2024-03-28T23:34:18.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.674+0000 7f031236c640 1 --2- 172.21.15.161:0/3493211218 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f030c0a8fc0 0x7f030c152800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:34:18.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.674+0000 7f0312b6d640 1 --2- 172.21.15.161:0/3493211218 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f030c150600 0x7f030c1e90d0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f030400cad0 tx=0x7f030400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:18.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.674+0000 7f0302ffd640 1 -- 172.21.15.161:0/3493211218 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0304017070 con 0x7f030c150600 2024-03-28T23:34:18.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.674+0000 7f0302ffd640 1 -- 172.21.15.161:0/3493211218 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f030400ac50 con 0x7f030c150600 2024-03-28T23:34:18.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.674+0000 7f03145d0640 1 -- 172.21.15.161:0/3493211218 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f030c152d40 con 0x7f030c150600 2024-03-28T23:34:18.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.674+0000 7f0302ffd640 1 -- 172.21.15.161:0/3493211218 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0304012400 con 0x7f030c150600 2024-03-28T23:34:18.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.674+0000 7f03145d0640 1 -- 172.21.15.161:0/3493211218 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f030c14b0e0 con 0x7f030c150600 2024-03-28T23:34:18.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.675+0000 7f03145d0640 1 -- 172.21.15.161:0/3493211218 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f030c04f4b0 con 0x7f030c150600 2024-03-28T23:34:18.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.676+0000 7f0302ffd640 1 -- 172.21.15.161:0/3493211218 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f030401a070 con 0x7f030c150600 2024-03-28T23:34:18.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.676+0000 7f0302ffd640 1 --2- 172.21.15.161:0/3493211218 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f02f006b4d0 0x7f02f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:18.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.676+0000 7f0302ffd640 1 -- 172.21.15.161:0/3493211218 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7f030408a050 con 0x7f030c150600 2024-03-28T23:34:18.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.676+0000 7f031236c640 1 --2- 172.21.15.161:0/3493211218 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f02f006b4d0 0x7f02f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:18.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.676+0000 7f031236c640 1 --2- 172.21.15.161:0/3493211218 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f02f006b4d0 0x7f02f006d990 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f0308007fd0 tx=0x7f0308009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:18.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.680+0000 7f0302ffd640 1 -- 172.21.15.161:0/3493211218 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0304058730 con 0x7f030c150600 2024-03-28T23:34:18.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.854+0000 7f03145d0640 1 -- 172.21.15.161:0/3493211218 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f030c0ab1c0 con 0x7f030c150600 2024-03-28T23:34:18.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.856+0000 7f0302ffd640 1 -- 172.21.15.161:0/3493211218 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) ==== 74+0+17119 (secure 0 0 0) 0x7f030405c7d0 con 0x7f030c150600 2024-03-28T23:34:18.857 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:18.857 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":110,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:34:02.034225+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:33:48.560815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-28T23:33:49.301625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6835/2544838511":"2024-03-29T23:34:02.034196+0000","172.21.15.161:6834/2544838511":"2024-03-29T23:34:02.034196+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:34:18.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.858+0000 7f03145d0640 1 -- 172.21.15.161:0/3493211218 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f02f006b4d0 msgr2=0x7f02f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:18.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.858+0000 7f03145d0640 1 --2- 172.21.15.161:0/3493211218 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f02f006b4d0 0x7f02f006d990 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f0308007fd0 tx=0x7f0308009040 comp rx=0 tx=0).stop 2024-03-28T23:34:18.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.858+0000 7f03145d0640 1 -- 172.21.15.161:0/3493211218 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f030c150600 msgr2=0x7f030c1e90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:18.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.858+0000 7f03145d0640 1 --2- 172.21.15.161:0/3493211218 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f030c150600 0x7f030c1e90d0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f030400cad0 tx=0x7f030400cf90 comp rx=0 tx=0).stop 2024-03-28T23:34:18.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.858+0000 7f03145d0640 1 -- 172.21.15.161:0/3493211218 shutdown_connections 2024-03-28T23:34:18.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.858+0000 7f03145d0640 1 --2- 172.21.15.161:0/3493211218 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f02f006b4d0 0x7f02f006d990 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.858+0000 7f03145d0640 1 --2- 172.21.15.161:0/3493211218 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f030c150600 0x7f030c1e90d0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.858+0000 7f03145d0640 1 --2- 172.21.15.161:0/3493211218 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f030c0aacb0 0x7f030c14c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.858+0000 7f03145d0640 1 --2- 172.21.15.161:0/3493211218 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f030c0a8fc0 0x7f030c152800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.858+0000 7f03145d0640 1 -- 172.21.15.161:0/3493211218 >> 172.21.15.161:0/3493211218 conn(0x7f030c0c6670 msgr2=0x7f030c0c0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:18.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.858+0000 7f03145d0640 1 -- 172.21.15.161:0/3493211218 shutdown_connections 2024-03-28T23:34:18.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.858+0000 7f03145d0640 1 -- 172.21.15.161:0/3493211218 wait complete. 2024-03-28T23:34:18.870 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-28T23:34:18.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.971+0000 7fad74313640 1 Processor -- start 2024-03-28T23:34:18.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.971+0000 7fad74313640 1 -- start start 2024-03-28T23:34:18.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.972+0000 7fad74313640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fad6c0c5410 0x7fad6c0c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:18.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.972+0000 7fad74313640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fad6c1553a0 0x7fad6c155850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:18.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.972+0000 7fad74313640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad6c158a30 con 0x7fad6c152890 2024-03-28T23:34:18.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.972+0000 7fad74313640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fad6c0c8480 con 0x7fad6c1553a0 2024-03-28T23:34:18.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.972+0000 7fad74313640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fad6c0c5d90 con 0x7fad6c0c5410 2024-03-28T23:34:18.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.972+0000 7fad728b0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fad6c1553a0 0x7fad6c155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:18.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.972+0000 7fad728b0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fad6c1553a0 0x7fad6c155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45314/0 (socket says 172.21.15.161:45314) 2024-03-28T23:34:18.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.972+0000 7fad728b0640 1 -- 172.21.15.161:0/212468691 learned_addr learned my addr 172.21.15.161:0/212468691 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:18.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.972+0000 7fad720af640 1 --1- 172.21.15.161:0/212468691 >> v1:172.21.15.170:6789/0 conn(0x7fad6c152890 0x7fad6c0c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:40702/0 (socket says 172.21.15.161:40702) 2024-03-28T23:34:18.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.972+0000 7fad718ae640 1 --2- 172.21.15.161:0/212468691 >> v2:172.21.15.170:3300/0 conn(0x7fad6c0c5410 0x7fad6c0c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:18.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.973+0000 7fad728b0640 1 -- 172.21.15.161:0/212468691 >> v2:172.21.15.170:3300/0 conn(0x7fad6c0c5410 msgr2=0x7fad6c0c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:18.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.973+0000 7fad728b0640 1 --2- 172.21.15.161:0/212468691 >> v2:172.21.15.170:3300/0 conn(0x7fad6c0c5410 0x7fad6c0c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.973+0000 7fad728b0640 1 -- 172.21.15.161:0/212468691 >> v1:172.21.15.170:6789/0 conn(0x7fad6c152890 legacy=0x7fad6c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:18.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.973+0000 7fad728b0640 1 -- 172.21.15.161:0/212468691 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad6c158d10 con 0x7fad6c1553a0 2024-03-28T23:34:18.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.973+0000 7fad718ae640 1 --2- 172.21.15.161:0/212468691 >> v2:172.21.15.170:3300/0 conn(0x7fad6c0c5410 0x7fad6c0c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:34:18.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.973+0000 7fad728b0640 1 --2- 172.21.15.161:0/212468691 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad6c1553a0 0x7fad6c155850 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fad6c0ad6e0 tx=0x7fad6402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd3cc965cea438c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:18.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.974+0000 7fad710ad640 1 -- 172.21.15.161:0/212468691 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad64035020 con 0x7fad6c1553a0 2024-03-28T23:34:18.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.974+0000 7fad710ad640 1 -- 172.21.15.161:0/212468691 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad64031870 con 0x7fad6c1553a0 2024-03-28T23:34:18.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.974+0000 7fad710ad640 1 -- 172.21.15.161:0/212468691 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad64031c40 con 0x7fad6c1553a0 2024-03-28T23:34:18.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.974+0000 7fad74313640 1 -- 172.21.15.161:0/212468691 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad6c1553a0 msgr2=0x7fad6c155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:18.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.974+0000 7fad74313640 1 --2- 172.21.15.161:0/212468691 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad6c1553a0 0x7fad6c155850 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fad6c0ad6e0 tx=0x7fad6402b910 comp rx=0 tx=0).stop 2024-03-28T23:34:18.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.974+0000 7fad74313640 1 -- 172.21.15.161:0/212468691 shutdown_connections 2024-03-28T23:34:18.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.974+0000 7fad74313640 1 --2- 172.21.15.161:0/212468691 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad6c1553a0 0x7fad6c155850 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.974+0000 7fad74313640 1 --2- 172.21.15.161:0/212468691 >> v2:172.21.15.170:3300/0 conn(0x7fad6c0c5410 0x7fad6c0c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.974+0000 7fad74313640 1 -- 172.21.15.161:0/212468691 >> 172.21.15.161:0/212468691 conn(0x7fad6c0ac620 msgr2=0x7fad6c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:18.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.974+0000 7fad74313640 1 -- 172.21.15.161:0/212468691 shutdown_connections 2024-03-28T23:34:18.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.974+0000 7fad74313640 1 -- 172.21.15.161:0/212468691 wait complete. 2024-03-28T23:34:18.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.975+0000 7fad74313640 1 Processor -- start 2024-03-28T23:34:18.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.975+0000 7fad74313640 1 -- start start 2024-03-28T23:34:18.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.976+0000 7fad74313640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad6c0c5410 0x7fad6c14d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:18.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.976+0000 7fad74313640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad6c152890 0x7fad6c14e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:18.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.976+0000 7fad74313640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad6c1553a0 0x7fad6c0ab810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:18.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.976+0000 7fad74313640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fad6c1592d0 con 0x7fad6c0c5410 2024-03-28T23:34:18.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.976+0000 7fad74313640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fad6c159640 con 0x7fad6c152890 2024-03-28T23:34:18.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.976+0000 7fad74313640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fad6c0c90a0 con 0x7fad6c1553a0 2024-03-28T23:34:18.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.976+0000 7fad720af640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad6c0c5410 0x7fad6c14d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:18.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.976+0000 7fad720af640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad6c0c5410 0x7fad6c14d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45316/0 (socket says 172.21.15.161:45316) 2024-03-28T23:34:18.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.976+0000 7fad720af640 1 -- 172.21.15.161:0/350810225 learned_addr learned my addr 172.21.15.161:0/350810225 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:18.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.976+0000 7fad718ae640 1 --2- 172.21.15.161:0/350810225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad6c152890 0x7fad6c14e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:18.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.976+0000 7fad728b0640 1 --2- 172.21.15.161:0/350810225 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad6c1553a0 0x7fad6c0ab810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:18.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.976+0000 7fad720af640 1 -- 172.21.15.161:0/350810225 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad6c1553a0 msgr2=0x7fad6c0ab810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:18.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.976+0000 7fad720af640 1 --2- 172.21.15.161:0/350810225 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad6c1553a0 0x7fad6c0ab810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.976+0000 7fad720af640 1 -- 172.21.15.161:0/350810225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad6c152890 msgr2=0x7fad6c14e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:18.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.976+0000 7fad720af640 1 --2- 172.21.15.161:0/350810225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad6c152890 0x7fad6c14e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:18.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.976+0000 7fad720af640 1 -- 172.21.15.161:0/350810225 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad6c151040 con 0x7fad6c0c5410 2024-03-28T23:34:18.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.977+0000 7fad720af640 1 --2- 172.21.15.161:0/350810225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad6c0c5410 0x7fad6c14d4d0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fad6800c790 tx=0x7fad6800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:18.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.977+0000 7fad62ffd640 1 -- 172.21.15.161:0/350810225 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad68017070 con 0x7fad6c0c5410 2024-03-28T23:34:18.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.977+0000 7fad62ffd640 1 -- 172.21.15.161:0/350810225 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad6800a640 con 0x7fad6c0c5410 2024-03-28T23:34:18.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.977+0000 7fad74313640 1 -- 172.21.15.161:0/350810225 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad6c151320 con 0x7fad6c0c5410 2024-03-28T23:34:18.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.977+0000 7fad62ffd640 1 -- 172.21.15.161:0/350810225 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6800a9f0 con 0x7fad6c0c5410 2024-03-28T23:34:18.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.977+0000 7fad74313640 1 -- 172.21.15.161:0/350810225 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad6c14cfb0 con 0x7fad6c0c5410 2024-03-28T23:34:18.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.978+0000 7fad74313640 1 -- 172.21.15.161:0/350810225 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad6c04f4b0 con 0x7fad6c0c5410 2024-03-28T23:34:18.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.978+0000 7fad62ffd640 1 -- 172.21.15.161:0/350810225 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fad6801a070 con 0x7fad6c0c5410 2024-03-28T23:34:18.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.979+0000 7fad62ffd640 1 --2- 172.21.15.161:0/350810225 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fad5006b4d0 0x7fad5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:18.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.979+0000 7fad62ffd640 1 -- 172.21.15.161:0/350810225 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6187+0+0 (secure 0 0 0) 0x7fad680898a0 con 0x7fad6c0c5410 2024-03-28T23:34:18.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.979+0000 7fad718ae640 1 --2- 172.21.15.161:0/350810225 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fad5006b4d0 0x7fad5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:18.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.979+0000 7fad718ae640 1 --2- 172.21.15.161:0/350810225 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fad5006b4d0 0x7fad5006d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fad5c009500 tx=0x7fad5c009110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:18.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:18.983+0000 7fad62ffd640 1 -- 172.21.15.161:0/350810225 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fad68057f80 con 0x7fad6c0c5410 2024-03-28T23:34:19.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.192+0000 7fad74313640 1 -- 172.21.15.161:0/350810225 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fad6c14d1f0 con 0x7fad6c0c5410 2024-03-28T23:34:19.193 INFO:tasks.ceph.mon.a.smithi161.stderr:2024-03-28T23:34:19.192+0000 7fdc7b85b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T23:34:19.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.195+0000 7fad62ffd640 1 -- 172.21.15.161:0/350810225 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v101) ==== 161+0+0 (secure 0 0 0) 0x7fad6c14d1f0 con 0x7fad6c0c5410 2024-03-28T23:34:19.196 INFO:teuthology.orchestra.run.smithi161.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-28T23:34:19.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.198+0000 7fad74313640 1 -- 172.21.15.161:0/350810225 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fad5006b4d0 msgr2=0x7fad5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:19.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.198+0000 7fad74313640 1 --2- 172.21.15.161:0/350810225 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fad5006b4d0 0x7fad5006d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fad5c009500 tx=0x7fad5c009110 comp rx=0 tx=0).stop 2024-03-28T23:34:19.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.198+0000 7fad74313640 1 -- 172.21.15.161:0/350810225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad6c0c5410 msgr2=0x7fad6c14d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:19.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.198+0000 7fad74313640 1 --2- 172.21.15.161:0/350810225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad6c0c5410 0x7fad6c14d4d0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fad6800c790 tx=0x7fad6800cc50 comp rx=0 tx=0).stop 2024-03-28T23:34:19.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.198+0000 7fad74313640 1 -- 172.21.15.161:0/350810225 shutdown_connections 2024-03-28T23:34:19.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.198+0000 7fad74313640 1 --2- 172.21.15.161:0/350810225 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fad5006b4d0 0x7fad5006d990 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:19.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.198+0000 7fad74313640 1 --2- 172.21.15.161:0/350810225 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad6c1553a0 0x7fad6c0ab810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:19.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.198+0000 7fad74313640 1 --2- 172.21.15.161:0/350810225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad6c152890 0x7fad6c14e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:19.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.198+0000 7fad74313640 1 --2- 172.21.15.161:0/350810225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad6c0c5410 0x7fad6c14d4d0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:19.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.198+0000 7fad74313640 1 -- 172.21.15.161:0/350810225 >> 172.21.15.161:0/350810225 conn(0x7fad6c0ac620 msgr2=0x7fad6c153a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:19.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.198+0000 7fad74313640 1 -- 172.21.15.161:0/350810225 shutdown_connections 2024-03-28T23:34:19.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.198+0000 7fad74313640 1 -- 172.21.15.161:0/350810225 wait complete. 2024-03-28T23:34:19.201 INFO:tasks.ceph.mds.b.smithi170.stderr: -569> 2024-03-28T23:34:06.262+0000 7fadcc9a7640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:34:19.210 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-28T23:34:19.223 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2024-03-28T23:34:19.224 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2024-03-28T23:34:19.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.311+0000 7fd0bbacd640 1 Processor -- start 2024-03-28T23:34:19.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.311+0000 7fd0bbacd640 1 -- start start 2024-03-28T23:34:19.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.312+0000 7fd0bbacd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b414f5c0 0x7fd0b40c2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:19.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.312+0000 7fd0bbacd640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0b40c2f10 con 0x7fd0b4152bd0 2024-03-28T23:34:19.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.312+0000 7fd0bbacd640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0b414f9a0 con 0x7fd0b406cdd0 2024-03-28T23:34:19.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.312+0000 7fd0bbacd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd0b40c74d0 con 0x7fd0b414f5c0 2024-03-28T23:34:19.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.312+0000 7fd0ba06a640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd0b4152bd0 0x7fd0b415c830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52258/0 (socket says 172.21.15.161:52258) 2024-03-28T23:34:19.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.312+0000 7fd0ba06a640 1 -- 172.21.15.161:0/1313956917 learned_addr learned my addr 172.21.15.161:0/1313956917 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:19.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.312+0000 7fd0b9869640 1 --2- 172.21.15.161:0/1313956917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b414f5c0 0x7fd0b40c2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:19.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.313+0000 7fd0b8867640 1 -- 172.21.15.161:0/1313956917 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1859302107 0 0) 0x7fd0ac001110 con 0x7fd0b4152bd0 2024-03-28T23:34:19.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.313+0000 7fd0b8867640 1 -- 172.21.15.161:0/1313956917 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0a0003520 con 0x7fd0b4152bd0 2024-03-28T23:34:19.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.313+0000 7fd0b8867640 1 -- 172.21.15.161:0/1313956917 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1703322555 0 0) 0x7fd0b414f9a0 con 0x7fd0b406cdd0 2024-03-28T23:34:19.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.313+0000 7fd0b8867640 1 -- 172.21.15.161:0/1313956917 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0ac001110 con 0x7fd0b406cdd0 2024-03-28T23:34:19.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.313+0000 7fd0b9869640 1 -- 172.21.15.161:0/1313956917 >> v1:172.21.15.170:6789/0 conn(0x7fd0b406cdd0 legacy=0x7fd0b41524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:19.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.313+0000 7fd0b9869640 1 -- 172.21.15.161:0/1313956917 >> v1:172.21.15.161:6789/0 conn(0x7fd0b4152bd0 legacy=0x7fd0b415c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:19.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.313+0000 7fd0b8867640 1 -- 172.21.15.161:0/1313956917 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3479121500 0 0) 0x7fd0a0003520 con 0x7fd0b4152bd0 2024-03-28T23:34:19.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.313+0000 7fd0b9869640 1 -- 172.21.15.161:0/1313956917 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0b406c140 con 0x7fd0b414f5c0 2024-03-28T23:34:19.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.314+0000 7fd0b9869640 1 --2- 172.21.15.161:0/1313956917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b414f5c0 0x7fd0b40c2150 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0009dc0 tx=0x7fd0b002bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47285941b757a105 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:19.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.314+0000 7fd0b8867640 1 -- 172.21.15.161:0/1313956917 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b0039070 con 0x7fd0b414f5c0 2024-03-28T23:34:19.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.314+0000 7fd0b8867640 1 -- 172.21.15.161:0/1313956917 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0b002c910 con 0x7fd0b414f5c0 2024-03-28T23:34:19.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.314+0000 7fd0b8867640 1 -- 172.21.15.161:0/1313956917 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac001840 con 0x7fd0b414f5c0 2024-03-28T23:34:19.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.314+0000 7fd0bbacd640 1 -- 172.21.15.161:0/1313956917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b414f5c0 msgr2=0x7fd0b40c2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:19.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.314+0000 7fd0bbacd640 1 --2- 172.21.15.161:0/1313956917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b414f5c0 0x7fd0b40c2150 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0009dc0 tx=0x7fd0b002bde0 comp rx=0 tx=0).stop 2024-03-28T23:34:19.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.314+0000 7fd0bbacd640 1 -- 172.21.15.161:0/1313956917 shutdown_connections 2024-03-28T23:34:19.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.314+0000 7fd0bbacd640 1 --2- 172.21.15.161:0/1313956917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b414f5c0 0x7fd0b40c2150 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:19.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.314+0000 7fd0bbacd640 1 -- 172.21.15.161:0/1313956917 >> 172.21.15.161:0/1313956917 conn(0x7fd0b40a83f0 msgr2=0x7fd0b40a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:19.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.314+0000 7fd0bbacd640 1 -- 172.21.15.161:0/1313956917 shutdown_connections 2024-03-28T23:34:19.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.314+0000 7fd0bbacd640 1 -- 172.21.15.161:0/1313956917 wait complete. 2024-03-28T23:34:19.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.315+0000 7fd0bbacd640 1 Processor -- start 2024-03-28T23:34:19.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.315+0000 7fd0bbacd640 1 -- start start 2024-03-28T23:34:19.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.316+0000 7fd0bbacd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0b406cdd0 0x7fd0b4148ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:19.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.316+0000 7fd0bbacd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0b414f5c0 0x7fd0b4153020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:19.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.316+0000 7fd0bbacd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b4152bd0 0x7fd0b4072ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:19.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.316+0000 7fd0bbacd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b4152030 con 0x7fd0b414f5c0 2024-03-28T23:34:19.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.316+0000 7fd0bbacd640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b40c6fa0 con 0x7fd0b406cdd0 2024-03-28T23:34:19.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.316+0000 7fd0bbacd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd0b406c3e0 con 0x7fd0b4152bd0 2024-03-28T23:34:19.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.316+0000 7fd0b9068640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0b414f5c0 0x7fd0b4153020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:19.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.316+0000 7fd0b9068640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0b414f5c0 0x7fd0b4153020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45330/0 (socket says 172.21.15.161:45330) 2024-03-28T23:34:19.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.316+0000 7fd0b9068640 1 -- 172.21.15.161:0/2237040206 learned_addr learned my addr 172.21.15.161:0/2237040206 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:19.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.316+0000 7fd0b9869640 1 --2- 172.21.15.161:0/2237040206 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0b406cdd0 0x7fd0b4148ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:19.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.316+0000 7fd0ba06a640 1 --2- 172.21.15.161:0/2237040206 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b4152bd0 0x7fd0b4072ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:19.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.316+0000 7fd0b9068640 1 -- 172.21.15.161:0/2237040206 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b4152bd0 msgr2=0x7fd0b4072ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:19.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.316+0000 7fd0b9068640 1 --2- 172.21.15.161:0/2237040206 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b4152bd0 0x7fd0b4072ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:19.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.316+0000 7fd0b9068640 1 -- 172.21.15.161:0/2237040206 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0b406cdd0 msgr2=0x7fd0b4148ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:19.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.316+0000 7fd0b9068640 1 --2- 172.21.15.161:0/2237040206 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0b406cdd0 0x7fd0b4148ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:19.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.316+0000 7fd0b9068640 1 -- 172.21.15.161:0/2237040206 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0b414c490 con 0x7fd0b414f5c0 2024-03-28T23:34:19.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.317+0000 7fd0b9068640 1 --2- 172.21.15.161:0/2237040206 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0b414f5c0 0x7fd0b4153020 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fd0a400c480 tx=0x7fd0a400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:19.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.317+0000 7fd0aa7fc640 1 -- 172.21.15.161:0/2237040206 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a40052d0 con 0x7fd0b414f5c0 2024-03-28T23:34:19.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.317+0000 7fd0aa7fc640 1 -- 172.21.15.161:0/2237040206 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0a4005470 con 0x7fd0b414f5c0 2024-03-28T23:34:19.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.317+0000 7fd0bbacd640 1 -- 172.21.15.161:0/2237040206 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0b414c1b0 con 0x7fd0b414f5c0 2024-03-28T23:34:19.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.317+0000 7fd0aa7fc640 1 -- 172.21.15.161:0/2237040206 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4012870 con 0x7fd0b414f5c0 2024-03-28T23:34:19.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.317+0000 7fd0bbacd640 1 -- 172.21.15.161:0/2237040206 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0b414b2e0 con 0x7fd0b414f5c0 2024-03-28T23:34:19.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.318+0000 7fd0bbacd640 1 -- 172.21.15.161:0/2237040206 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0b404f4b0 con 0x7fd0b414f5c0 2024-03-28T23:34:19.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.318+0000 7fd0aa7fc640 1 -- 172.21.15.161:0/2237040206 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd0a4012a10 con 0x7fd0b414f5c0 2024-03-28T23:34:19.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.319+0000 7fd0aa7fc640 1 --2- 172.21.15.161:0/2237040206 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd09806b4d0 0x7fd09806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:19.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.319+0000 7fd0aa7fc640 1 -- 172.21.15.161:0/2237040206 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 6273+0+0 (secure 0 0 0) 0x7fd0a4089f60 con 0x7fd0b414f5c0 2024-03-28T23:34:19.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.319+0000 7fd0b9869640 1 --2- 172.21.15.161:0/2237040206 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd09806b4d0 0x7fd09806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:19.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.319+0000 7fd0b9869640 1 --2- 172.21.15.161:0/2237040206 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd09806b4d0 0x7fd09806d990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0036000 tx=0x7fd0b0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:19.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.323+0000 7fd0aa7fc640 1 -- 172.21.15.161:0/2237040206 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd0a40586a0 con 0x7fd0b414f5c0 2024-03-28T23:34:19.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:19.533+0000 7fd0bbacd640 1 -- 172.21.15.161:0/2237040206 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fd0b40736d0 con 0x7fd0b414f5c0 2024-03-28T23:34:20.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.199+0000 7fd0aa7fc640 1 -- 172.21.15.161:0/2237040206 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v102) ==== 104+0+0 (secure 0 0 0) 0x7fd0a405c740 con 0x7fd0b414f5c0 2024-03-28T23:34:20.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.201+0000 7fd0bbacd640 1 -- 172.21.15.161:0/2237040206 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd09806b4d0 msgr2=0x7fd09806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:20.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.201+0000 7fd0bbacd640 1 --2- 172.21.15.161:0/2237040206 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd09806b4d0 0x7fd09806d990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0036000 tx=0x7fd0b0030000 comp rx=0 tx=0).stop 2024-03-28T23:34:20.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.201+0000 7fd0bbacd640 1 -- 172.21.15.161:0/2237040206 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0b414f5c0 msgr2=0x7fd0b4153020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:20.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.201+0000 7fd0bbacd640 1 --2- 172.21.15.161:0/2237040206 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0b414f5c0 0x7fd0b4153020 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fd0a400c480 tx=0x7fd0a400c940 comp rx=0 tx=0).stop 2024-03-28T23:34:20.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.201+0000 7fd0bbacd640 1 -- 172.21.15.161:0/2237040206 shutdown_connections 2024-03-28T23:34:20.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.201+0000 7fd0bbacd640 1 --2- 172.21.15.161:0/2237040206 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd09806b4d0 0x7fd09806d990 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:20.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.201+0000 7fd0bbacd640 1 --2- 172.21.15.161:0/2237040206 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0b414f5c0 0x7fd0b4153020 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:20.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.201+0000 7fd0bbacd640 1 --2- 172.21.15.161:0/2237040206 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b4152bd0 0x7fd0b4072ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:20.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.201+0000 7fd0bbacd640 1 --2- 172.21.15.161:0/2237040206 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0b406cdd0 0x7fd0b4148ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:20.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.201+0000 7fd0bbacd640 1 -- 172.21.15.161:0/2237040206 >> 172.21.15.161:0/2237040206 conn(0x7fd0b40a83f0 msgr2=0x7fd0b4150d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:20.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.201+0000 7fd0bbacd640 1 -- 172.21.15.161:0/2237040206 shutdown_connections 2024-03-28T23:34:20.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.201+0000 7fd0bbacd640 1 -- 172.21.15.161:0/2237040206 wait complete. 2024-03-28T23:34:20.213 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-28T23:34:20.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.318+0000 7ffa62dd0640 1 Processor -- start 2024-03-28T23:34:20.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.318+0000 7ffa62dd0640 1 -- start start 2024-03-28T23:34:20.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.319+0000 7ffa62dd0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ffa5c155ab0 0x7ffa5c158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:20.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.319+0000 7ffa62dd0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ffa5c072810 0x7ffa5c072ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:20.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.319+0000 7ffa62dd0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa5c159530 con 0x7ffa5c14efb0 2024-03-28T23:34:20.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.319+0000 7ffa62dd0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ffa5c0c2ec0 con 0x7ffa5c155ab0 2024-03-28T23:34:20.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.319+0000 7ffa62dd0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7ffa5c158b60 con 0x7ffa5c072810 2024-03-28T23:34:20.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.319+0000 7ffa5bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ffa5c155ab0 0x7ffa5c158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:20.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.319+0000 7ffa5bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ffa5c155ab0 0x7ffa5c158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45342/0 (socket says 172.21.15.161:45342) 2024-03-28T23:34:20.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.319+0000 7ffa5bfff640 1 -- 172.21.15.161:0/2258487857 learned_addr learned my addr 172.21.15.161:0/2258487857 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:20.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.319+0000 7ffa60b6c640 1 --1- 172.21.15.161:0/2258487857 >> v1:172.21.15.161:6789/0 conn(0x7ffa5c14efb0 0x7ffa5c0c70f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52272/0 (socket says 172.21.15.161:52272) 2024-03-28T23:34:20.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.319+0000 7ffa6136d640 1 --2- 172.21.15.161:0/2258487857 >> v2:172.21.15.170:3300/0 conn(0x7ffa5c072810 0x7ffa5c072ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:20.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.320+0000 7ffa5bfff640 1 -- 172.21.15.161:0/2258487857 >> v2:172.21.15.170:3300/0 conn(0x7ffa5c072810 msgr2=0x7ffa5c072ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:20.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.320+0000 7ffa5bfff640 1 --2- 172.21.15.161:0/2258487857 >> v2:172.21.15.170:3300/0 conn(0x7ffa5c072810 0x7ffa5c072ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:20.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.320+0000 7ffa5bfff640 1 -- 172.21.15.161:0/2258487857 >> v1:172.21.15.161:6789/0 conn(0x7ffa5c14efb0 legacy=0x7ffa5c0c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:20.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.320+0000 7ffa5b7fe640 1 -- 172.21.15.161:0/2258487857 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4215840777 0 0) 0x7ffa5c159530 con 0x7ffa5c14efb0 2024-03-28T23:34:20.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.320+0000 7ffa5bfff640 1 -- 172.21.15.161:0/2258487857 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa5c155f40 con 0x7ffa5c155ab0 2024-03-28T23:34:20.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.320+0000 7ffa5bfff640 1 --2- 172.21.15.161:0/2258487857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa5c155ab0 0x7ffa5c158ff0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c0c8960 tx=0x7ffa5002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5be8f131c4c3b354 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:20.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.320+0000 7ffa5b7fe640 1 -- 172.21.15.161:0/2258487857 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa50035020 con 0x7ffa5c155ab0 2024-03-28T23:34:20.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.320+0000 7ffa5b7fe640 1 -- 172.21.15.161:0/2258487857 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa50031870 con 0x7ffa5c155ab0 2024-03-28T23:34:20.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.320+0000 7ffa5b7fe640 1 -- 172.21.15.161:0/2258487857 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa50031c40 con 0x7ffa5c155ab0 2024-03-28T23:34:20.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.321+0000 7ffa62dd0640 1 -- 172.21.15.161:0/2258487857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa5c155ab0 msgr2=0x7ffa5c158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:20.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.321+0000 7ffa62dd0640 1 --2- 172.21.15.161:0/2258487857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa5c155ab0 0x7ffa5c158ff0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c0c8960 tx=0x7ffa5002b910 comp rx=0 tx=0).stop 2024-03-28T23:34:20.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.321+0000 7ffa62dd0640 1 -- 172.21.15.161:0/2258487857 shutdown_connections 2024-03-28T23:34:20.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.321+0000 7ffa62dd0640 1 --2- 172.21.15.161:0/2258487857 >> v2:172.21.15.170:3300/0 conn(0x7ffa5c072810 0x7ffa5c072ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:20.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.321+0000 7ffa62dd0640 1 --2- 172.21.15.161:0/2258487857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa5c155ab0 0x7ffa5c158ff0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:20.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.321+0000 7ffa62dd0640 1 -- 172.21.15.161:0/2258487857 >> 172.21.15.161:0/2258487857 conn(0x7ffa5c0c8990 msgr2=0x7ffa5c0c8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:20.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.321+0000 7ffa62dd0640 1 -- 172.21.15.161:0/2258487857 shutdown_connections 2024-03-28T23:34:20.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.321+0000 7ffa62dd0640 1 -- 172.21.15.161:0/2258487857 wait complete. 2024-03-28T23:34:20.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.322+0000 7ffa62dd0640 1 Processor -- start 2024-03-28T23:34:20.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.322+0000 7ffa62dd0640 1 -- start start 2024-03-28T23:34:20.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.323+0000 7ffa62dd0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa5c072810 0x7ffa5c0ae210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:20.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.323+0000 7ffa62dd0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa5c14efb0 0x7ffa5c06fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:20.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.323+0000 7ffa62dd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffa5c155ab0 0x7ffa5c149ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:20.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.323+0000 7ffa62dd0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ffa5c1587f0 con 0x7ffa5c072810 2024-03-28T23:34:20.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.323+0000 7ffa62dd0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ffa5c0c3650 con 0x7ffa5c14efb0 2024-03-28T23:34:20.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.323+0000 7ffa62dd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ffa5c14f390 con 0x7ffa5c155ab0 2024-03-28T23:34:20.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.323+0000 7ffa60b6c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa5c072810 0x7ffa5c0ae210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:20.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.323+0000 7ffa60b6c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa5c072810 0x7ffa5c0ae210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45350/0 (socket says 172.21.15.161:45350) 2024-03-28T23:34:20.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.323+0000 7ffa60b6c640 1 -- 172.21.15.161:0/3425750022 learned_addr learned my addr 172.21.15.161:0/3425750022 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:20.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.323+0000 7ffa5bfff640 1 --2- 172.21.15.161:0/3425750022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa5c14efb0 0x7ffa5c06fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:20.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.323+0000 7ffa6136d640 1 --2- 172.21.15.161:0/3425750022 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffa5c155ab0 0x7ffa5c149ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:20.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.323+0000 7ffa60b6c640 1 -- 172.21.15.161:0/3425750022 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffa5c155ab0 msgr2=0x7ffa5c149ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:20.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.323+0000 7ffa60b6c640 1 --2- 172.21.15.161:0/3425750022 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffa5c155ab0 0x7ffa5c149ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:20.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.323+0000 7ffa60b6c640 1 -- 172.21.15.161:0/3425750022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa5c14efb0 msgr2=0x7ffa5c06fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:20.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.323+0000 7ffa60b6c640 1 --2- 172.21.15.161:0/3425750022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa5c14efb0 0x7ffa5c06fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:20.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.323+0000 7ffa60b6c640 1 -- 172.21.15.161:0/3425750022 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa5c070730 con 0x7ffa5c072810 2024-03-28T23:34:20.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.323+0000 7ffa60b6c640 1 --2- 172.21.15.161:0/3425750022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa5c072810 0x7ffa5c0ae210 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c00c500 tx=0x7ffa4c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:20.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.324+0000 7ffa597fa640 1 -- 172.21.15.161:0/3425750022 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c018070 con 0x7ffa5c072810 2024-03-28T23:34:20.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.324+0000 7ffa597fa640 1 -- 172.21.15.161:0/3425750022 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa4c007580 con 0x7ffa5c072810 2024-03-28T23:34:20.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.324+0000 7ffa62dd0640 1 -- 172.21.15.161:0/3425750022 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa5c156e30 con 0x7ffa5c072810 2024-03-28T23:34:20.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.324+0000 7ffa597fa640 1 -- 172.21.15.161:0/3425750022 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c004a20 con 0x7ffa5c072810 2024-03-28T23:34:20.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.324+0000 7ffa62dd0640 1 -- 172.21.15.161:0/3425750022 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa5c06e100 con 0x7ffa5c072810 2024-03-28T23:34:20.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.325+0000 7ffa62dd0640 1 -- 172.21.15.161:0/3425750022 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa5c156c60 con 0x7ffa5c072810 2024-03-28T23:34:20.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.325+0000 7ffa597fa640 1 -- 172.21.15.161:0/3425750022 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ffa4c01b070 con 0x7ffa5c072810 2024-03-28T23:34:20.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.326+0000 7ffa597fa640 1 --2- 172.21.15.161:0/3425750022 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffa4806b4d0 0x7ffa4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:20.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.326+0000 7ffa5bfff640 1 --2- 172.21.15.161:0/3425750022 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffa4806b4d0 0x7ffa4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:20.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.326+0000 7ffa597fa640 1 -- 172.21.15.161:0/3425750022 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 6273+0+0 (secure 0 0 0) 0x7ffa4c0896e0 con 0x7ffa5c072810 2024-03-28T23:34:20.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.326+0000 7ffa5bfff640 1 --2- 172.21.15.161:0/3425750022 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffa4806b4d0 0x7ffa4806d990 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ffa50036000 tx=0x7ffa5003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:20.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.330+0000 7ffa597fa640 1 -- 172.21.15.161:0/3425750022 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ffa4c0583b0 con 0x7ffa5c072810 2024-03-28T23:34:20.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:20.501+0000 7ffa62dd0640 1 -- 172.21.15.161:0/3425750022 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7ffa5c0c21a0 con 0x7ffa5c072810 2024-03-28T23:34:21.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.201+0000 7ffa597fa640 1 -- 172.21.15.161:0/3425750022 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v112) ==== 181+0+0 (secure 0 0 0) 0x7ffa4c057cb0 con 0x7ffa5c072810 2024-03-28T23:34:21.202 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_metadata' removed 2024-03-28T23:34:21.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.204+0000 7ffa62dd0640 1 -- 172.21.15.161:0/3425750022 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffa4806b4d0 msgr2=0x7ffa4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:21.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.204+0000 7ffa62dd0640 1 --2- 172.21.15.161:0/3425750022 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffa4806b4d0 0x7ffa4806d990 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ffa50036000 tx=0x7ffa5003c040 comp rx=0 tx=0).stop 2024-03-28T23:34:21.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.204+0000 7ffa62dd0640 1 -- 172.21.15.161:0/3425750022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa5c072810 msgr2=0x7ffa5c0ae210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:21.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.204+0000 7ffa62dd0640 1 --2- 172.21.15.161:0/3425750022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa5c072810 0x7ffa5c0ae210 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c00c500 tx=0x7ffa4c00c9c0 comp rx=0 tx=0).stop 2024-03-28T23:34:21.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.204+0000 7ffa62dd0640 1 -- 172.21.15.161:0/3425750022 shutdown_connections 2024-03-28T23:34:21.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.204+0000 7ffa62dd0640 1 --2- 172.21.15.161:0/3425750022 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffa4806b4d0 0x7ffa4806d990 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:21.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.204+0000 7ffa62dd0640 1 --2- 172.21.15.161:0/3425750022 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffa5c155ab0 0x7ffa5c149ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:21.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.204+0000 7ffa62dd0640 1 --2- 172.21.15.161:0/3425750022 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa5c14efb0 0x7ffa5c06fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:21.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.204+0000 7ffa62dd0640 1 --2- 172.21.15.161:0/3425750022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa5c072810 0x7ffa5c0ae210 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:21.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.204+0000 7ffa62dd0640 1 -- 172.21.15.161:0/3425750022 >> 172.21.15.161:0/3425750022 conn(0x7ffa5c0c8990 msgr2=0x7ffa5c072ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:21.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.204+0000 7ffa62dd0640 1 -- 172.21.15.161:0/3425750022 shutdown_connections 2024-03-28T23:34:21.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.204+0000 7ffa62dd0640 1 -- 172.21.15.161:0/3425750022 wait complete. 2024-03-28T23:34:21.219 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-28T23:34:21.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.332+0000 7f2158a40640 1 Processor -- start 2024-03-28T23:34:21.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.332+0000 7f2158a40640 1 -- start start 2024-03-28T23:34:21.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.333+0000 7f2158a40640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21540abb70 0x7f21540abf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:21.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.333+0000 7f2158a40640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f21540c46b0 0x7f21540c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:21.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.333+0000 7f2158a40640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2154149ec0 con 0x7f215406d390 2024-03-28T23:34:21.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.333+0000 7f2158a40640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f2154153ab0 con 0x7f21540c46b0 2024-03-28T23:34:21.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.333+0000 7f2158a40640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f215406e400 con 0x7f21540abb70 2024-03-28T23:34:21.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.333+0000 7f215259c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21540abb70 0x7f21540abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:21.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.333+0000 7f2152d9d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f21540c46b0 0x7f21540c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:21.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.333+0000 7f2149d9b640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f215406d390 0x7f215406dcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:37604/0 (socket says 172.21.15.161:37604) 2024-03-28T23:34:21.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.333+0000 7f215259c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21540abb70 0x7f21540abf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:38412/0 (socket says 172.21.15.161:38412) 2024-03-28T23:34:21.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.333+0000 7f2149d9b640 1 -- 172.21.15.161:0/3377388013 learned_addr learned my addr 172.21.15.161:0/3377388013 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:21.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.333+0000 7f2151d9b640 1 -- 172.21.15.161:0/3377388013 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3618531941 0 0) 0x7f2154149ec0 con 0x7f215406d390 2024-03-28T23:34:21.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.333+0000 7f2151d9b640 1 -- 172.21.15.161:0/3377388013 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2140003520 con 0x7f215406d390 2024-03-28T23:34:21.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.334+0000 7f215259c640 1 -- 172.21.15.161:0/3377388013 >> v2:172.21.15.170:3300/0 conn(0x7f21540c46b0 msgr2=0x7f21540c4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:21.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.334+0000 7f215259c640 1 --2- 172.21.15.161:0/3377388013 >> v2:172.21.15.170:3300/0 conn(0x7f21540c46b0 0x7f21540c4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:21.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.334+0000 7f215259c640 1 -- 172.21.15.161:0/3377388013 >> v1:172.21.15.170:6789/0 conn(0x7f215406d390 legacy=0x7f215406dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:21.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.334+0000 7f215259c640 1 -- 172.21.15.161:0/3377388013 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2154153180 con 0x7f21540abb70 2024-03-28T23:34:21.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.334+0000 7f2152d9d640 1 --2- 172.21.15.161:0/3377388013 >> v2:172.21.15.170:3300/0 conn(0x7f21540c46b0 0x7f21540c4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:34:21.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.334+0000 7f215259c640 1 --2- 172.21.15.161:0/3377388013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21540abb70 0x7f21540abf50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f21440097a0 tx=0x7f214402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63db76da2b4397b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:21.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.335+0000 7f2151d9b640 1 -- 172.21.15.161:0/3377388013 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2144035020 con 0x7f21540abb70 2024-03-28T23:34:21.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.335+0000 7f2151d9b640 1 -- 172.21.15.161:0/3377388013 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2144031810 con 0x7f21540abb70 2024-03-28T23:34:21.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.335+0000 7f2151d9b640 1 -- 172.21.15.161:0/3377388013 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2144031be0 con 0x7f21540abb70 2024-03-28T23:34:21.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.335+0000 7f2158a40640 1 -- 172.21.15.161:0/3377388013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21540abb70 msgr2=0x7f21540abf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:21.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.335+0000 7f2158a40640 1 --2- 172.21.15.161:0/3377388013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21540abb70 0x7f21540abf50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f21440097a0 tx=0x7f214402b8b0 comp rx=0 tx=0).stop 2024-03-28T23:34:21.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.335+0000 7f2158a40640 1 -- 172.21.15.161:0/3377388013 shutdown_connections 2024-03-28T23:34:21.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.335+0000 7f2158a40640 1 --2- 172.21.15.161:0/3377388013 >> v2:172.21.15.170:3300/0 conn(0x7f21540c46b0 0x7f21540c4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:21.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.335+0000 7f2158a40640 1 --2- 172.21.15.161:0/3377388013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21540abb70 0x7f21540abf50 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:21.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.335+0000 7f2158a40640 1 -- 172.21.15.161:0/3377388013 >> 172.21.15.161:0/3377388013 conn(0x7f215406b150 msgr2=0x7f215406b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:21.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.335+0000 7f2158a40640 1 -- 172.21.15.161:0/3377388013 shutdown_connections 2024-03-28T23:34:21.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.335+0000 7f2158a40640 1 -- 172.21.15.161:0/3377388013 wait complete. 2024-03-28T23:34:21.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.337+0000 7f2158a40640 1 Processor -- start 2024-03-28T23:34:21.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.337+0000 7f2158a40640 1 -- start start 2024-03-28T23:34:21.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.337+0000 7f2158a40640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f215406d390 0x7f2154154360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:21.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.337+0000 7f2158a40640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21540abb70 0x7f21541ec090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:21.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.337+0000 7f2158a40640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21540c46b0 0x7f21541ec5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:21.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.337+0000 7f2158a40640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f21540c50a0 con 0x7f215406d390 2024-03-28T23:34:21.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.337+0000 7f2158a40640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2154153740 con 0x7f21540abb70 2024-03-28T23:34:21.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.337+0000 7f2158a40640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f21540c5220 con 0x7f21540c46b0 2024-03-28T23:34:21.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.337+0000 7f215259c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f215406d390 0x7f2154154360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:21.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.337+0000 7f215259c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f215406d390 0x7f2154154360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43278/0 (socket says 172.21.15.161:43278) 2024-03-28T23:34:21.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.337+0000 7f215259c640 1 -- 172.21.15.161:0/492412481 learned_addr learned my addr 172.21.15.161:0/492412481 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:21.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.338+0000 7f2149d9b640 1 --2- 172.21.15.161:0/492412481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21540abb70 0x7f21541ec090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:21.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.338+0000 7f2152d9d640 1 --2- 172.21.15.161:0/492412481 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21540c46b0 0x7f21541ec5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:21.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.338+0000 7f215259c640 1 -- 172.21.15.161:0/492412481 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21540c46b0 msgr2=0x7f21541ec5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:21.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.338+0000 7f215259c640 1 --2- 172.21.15.161:0/492412481 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21540c46b0 0x7f21541ec5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:21.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.338+0000 7f215259c640 1 -- 172.21.15.161:0/492412481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21540abb70 msgr2=0x7f21541ec090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:21.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.338+0000 7f215259c640 1 --2- 172.21.15.161:0/492412481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21540abb70 0x7f21541ec090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:21.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.338+0000 7f215259c640 1 -- 172.21.15.161:0/492412481 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2154156870 con 0x7f215406d390 2024-03-28T23:34:21.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.338+0000 7f2149d9b640 1 --2- 172.21.15.161:0/492412481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21540abb70 0x7f21541ec090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:21.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.338+0000 7f215259c640 1 --2- 172.21.15.161:0/492412481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f215406d390 0x7f2154154360 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f214402bec0 tx=0x7f21440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:21.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.338+0000 7f214b7fe640 1 -- 172.21.15.161:0/492412481 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2144041070 con 0x7f215406d390 2024-03-28T23:34:21.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.338+0000 7f214b7fe640 1 -- 172.21.15.161:0/492412481 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2144031e40 con 0x7f215406d390 2024-03-28T23:34:21.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.338+0000 7f2158a40640 1 -- 172.21.15.161:0/492412481 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2154157310 con 0x7f215406d390 2024-03-28T23:34:21.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.338+0000 7f214b7fe640 1 -- 172.21.15.161:0/492412481 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2144035040 con 0x7f215406d390 2024-03-28T23:34:21.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.338+0000 7f2158a40640 1 -- 172.21.15.161:0/492412481 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2154156b10 con 0x7f215406d390 2024-03-28T23:34:21.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.339+0000 7f2158a40640 1 -- 172.21.15.161:0/492412481 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2154068f90 con 0x7f215406d390 2024-03-28T23:34:21.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.340+0000 7f214b7fe640 1 -- 172.21.15.161:0/492412481 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f214404c060 con 0x7f215406d390 2024-03-28T23:34:21.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.340+0000 7f214b7fe640 1 --2- 172.21.15.161:0/492412481 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f213406b500 0x7f213406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:21.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.340+0000 7f214b7fe640 1 -- 172.21.15.161:0/492412481 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5835+0+0 (secure 0 0 0) 0x7f21440abce0 con 0x7f215406d390 2024-03-28T23:34:21.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.340+0000 7f2149d9b640 1 --2- 172.21.15.161:0/492412481 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f213406b500 0x7f213406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:21.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.341+0000 7f2149d9b640 1 --2- 172.21.15.161:0/492412481 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f213406b500 0x7f213406d9c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f213c007fd0 tx=0x7f213c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:21.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.345+0000 7f214b7fe640 1 -- 172.21.15.161:0/492412481 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f214404b2c0 con 0x7f215406d390 2024-03-28T23:34:21.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:21.526+0000 7f2158a40640 1 -- 172.21.15.161:0/492412481 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f215406d7c0 con 0x7f215406d390 2024-03-28T23:34:22.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.205+0000 7f214b7fe640 1 -- 172.21.15.161:0/492412481 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v113) ==== 169+0+0 (secure 0 0 0) 0x7f214407ab70 con 0x7f215406d390 2024-03-28T23:34:22.206 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data' removed 2024-03-28T23:34:22.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.208+0000 7f2158a40640 1 -- 172.21.15.161:0/492412481 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f213406b500 msgr2=0x7f213406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:22.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.208+0000 7f2158a40640 1 --2- 172.21.15.161:0/492412481 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f213406b500 0x7f213406d9c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f213c007fd0 tx=0x7f213c009040 comp rx=0 tx=0).stop 2024-03-28T23:34:22.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.208+0000 7f2158a40640 1 -- 172.21.15.161:0/492412481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f215406d390 msgr2=0x7f2154154360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:22.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.208+0000 7f2158a40640 1 --2- 172.21.15.161:0/492412481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f215406d390 0x7f2154154360 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f214402bec0 tx=0x7f21440046c0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.209+0000 7f2158a40640 1 -- 172.21.15.161:0/492412481 shutdown_connections 2024-03-28T23:34:22.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.209+0000 7f2158a40640 1 --2- 172.21.15.161:0/492412481 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f213406b500 0x7f213406d9c0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.209+0000 7f2158a40640 1 --2- 172.21.15.161:0/492412481 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21540c46b0 0x7f21541ec5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.209+0000 7f2158a40640 1 --2- 172.21.15.161:0/492412481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21540abb70 0x7f21541ec090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.209+0000 7f2158a40640 1 --2- 172.21.15.161:0/492412481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f215406d390 0x7f2154154360 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.209+0000 7f2158a40640 1 -- 172.21.15.161:0/492412481 >> 172.21.15.161:0/492412481 conn(0x7f215406b150 msgr2=0x7f21540a7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:22.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.209+0000 7f2158a40640 1 -- 172.21.15.161:0/492412481 shutdown_connections 2024-03-28T23:34:22.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.210+0000 7f2158a40640 1 -- 172.21.15.161:0/492412481 wait complete. 2024-03-28T23:34:22.225 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-28T23:34:22.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.331+0000 7fc821bd0640 1 Processor -- start 2024-03-28T23:34:22.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.331+0000 7fc821bd0640 1 -- start start 2024-03-28T23:34:22.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.332+0000 7fc821bd0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc81c14f5c0 0x7fc81c0c2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:22.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.332+0000 7fc821bd0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc81c06cdd0 0x7fc81c1524c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:22.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.332+0000 7fc821bd0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc81c0c2f10 con 0x7fc81c0c2690 2024-03-28T23:34:22.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.332+0000 7fc821bd0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc81c0c74d0 con 0x7fc81c06cdd0 2024-03-28T23:34:22.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.332+0000 7fc821bd0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fc81c0c6fa0 con 0x7fc81c14f5c0 2024-03-28T23:34:22.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.332+0000 7fc81affd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc81c06cdd0 0x7fc81c1524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:22.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.332+0000 7fc81affd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc81c06cdd0 0x7fc81c1524c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43280/0 (socket says 172.21.15.161:43280) 2024-03-28T23:34:22.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.332+0000 7fc81affd640 1 -- 172.21.15.161:0/4155970608 learned_addr learned my addr 172.21.15.161:0/4155970608 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:22.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.332+0000 7fc81affd640 1 -- 172.21.15.161:0/4155970608 >> v2:172.21.15.170:3300/0 conn(0x7fc81c14f5c0 msgr2=0x7fc81c0c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:22.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.332+0000 7fc81affd640 1 --2- 172.21.15.161:0/4155970608 >> v2:172.21.15.170:3300/0 conn(0x7fc81c14f5c0 0x7fc81c0c2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.332+0000 7fc81affd640 1 -- 172.21.15.161:0/4155970608 >> v1:172.21.15.170:6789/0 conn(0x7fc81c0c2690 legacy=0x7fc81c1554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:22.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.332+0000 7fc81affd640 1 -- 172.21.15.161:0/4155970608 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc81c14f9a0 con 0x7fc81c06cdd0 2024-03-28T23:34:22.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.333+0000 7fc81affd640 1 --2- 172.21.15.161:0/4155970608 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc81c06cdd0 0x7fc81c1524c0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fc81c0c76e0 tx=0x7fc81002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=889fa2a02e5f39ef server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:22.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.333+0000 7fc81a7fc640 1 -- 172.21.15.161:0/4155970608 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc810035020 con 0x7fc81c06cdd0 2024-03-28T23:34:22.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.333+0000 7fc81a7fc640 1 -- 172.21.15.161:0/4155970608 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc810031870 con 0x7fc81c06cdd0 2024-03-28T23:34:22.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.333+0000 7fc81a7fc640 1 -- 172.21.15.161:0/4155970608 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc810031c40 con 0x7fc81c06cdd0 2024-03-28T23:34:22.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.333+0000 7fc821bd0640 1 -- 172.21.15.161:0/4155970608 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc81c06cdd0 msgr2=0x7fc81c1524c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:22.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.333+0000 7fc821bd0640 1 --2- 172.21.15.161:0/4155970608 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc81c06cdd0 0x7fc81c1524c0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fc81c0c76e0 tx=0x7fc81002b910 comp rx=0 tx=0).stop 2024-03-28T23:34:22.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.334+0000 7fc821bd0640 1 -- 172.21.15.161:0/4155970608 shutdown_connections 2024-03-28T23:34:22.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.334+0000 7fc821bd0640 1 --2- 172.21.15.161:0/4155970608 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc81c06cdd0 0x7fc81c1524c0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.334+0000 7fc821bd0640 1 --2- 172.21.15.161:0/4155970608 >> v2:172.21.15.170:3300/0 conn(0x7fc81c14f5c0 0x7fc81c0c2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.334+0000 7fc821bd0640 1 -- 172.21.15.161:0/4155970608 >> 172.21.15.161:0/4155970608 conn(0x7fc81c0a83f0 msgr2=0x7fc81c0a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:22.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.334+0000 7fc821bd0640 1 -- 172.21.15.161:0/4155970608 shutdown_connections 2024-03-28T23:34:22.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.334+0000 7fc821bd0640 1 -- 172.21.15.161:0/4155970608 wait complete. 2024-03-28T23:34:22.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.335+0000 7fc821bd0640 1 Processor -- start 2024-03-28T23:34:22.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.335+0000 7fc821bd0640 1 -- start start 2024-03-28T23:34:22.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.335+0000 7fc821bd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc81c06cdd0 0x7fc81c0bfce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:22.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.335+0000 7fc821bd0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc81c0c2690 0x7fc81c1553b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:22.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.335+0000 7fc821bd0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc81c14f5c0 0x7fc81c1eaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:22.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.335+0000 7fc821bd0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc81c06c140 con 0x7fc81c14f5c0 2024-03-28T23:34:22.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.335+0000 7fc821bd0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc81c152030 con 0x7fc81c0c2690 2024-03-28T23:34:22.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.335+0000 7fc821bd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc81c06c3e0 con 0x7fc81c06cdd0 2024-03-28T23:34:22.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.336+0000 7fc81bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc81c14f5c0 0x7fc81c1eaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:22.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.336+0000 7fc81bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc81c14f5c0 0x7fc81c1eaeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43296/0 (socket says 172.21.15.161:43296) 2024-03-28T23:34:22.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.336+0000 7fc81bfff640 1 -- 172.21.15.161:0/2151114788 learned_addr learned my addr 172.21.15.161:0/2151114788 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:22.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.336+0000 7fc81affd640 1 --2- 172.21.15.161:0/2151114788 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc81c0c2690 0x7fc81c1553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:22.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.336+0000 7fc81b7fe640 1 --2- 172.21.15.161:0/2151114788 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc81c06cdd0 0x7fc81c0bfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:22.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.336+0000 7fc81bfff640 1 -- 172.21.15.161:0/2151114788 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc81c06cdd0 msgr2=0x7fc81c0bfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:22.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.336+0000 7fc81bfff640 1 --2- 172.21.15.161:0/2151114788 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc81c06cdd0 0x7fc81c0bfce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.336+0000 7fc81bfff640 1 -- 172.21.15.161:0/2151114788 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc81c0c2690 msgr2=0x7fc81c1553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:22.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.336+0000 7fc81bfff640 1 --2- 172.21.15.161:0/2151114788 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc81c0c2690 0x7fc81c1553b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.336+0000 7fc81bfff640 1 -- 172.21.15.161:0/2151114788 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc81c1545b0 con 0x7fc81c14f5c0 2024-03-28T23:34:22.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.336+0000 7fc81bfff640 1 --2- 172.21.15.161:0/2151114788 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc81c14f5c0 0x7fc81c1eaeb0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fc81400c780 tx=0x7fc81400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:22.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.336+0000 7fc803fff640 1 -- 172.21.15.161:0/2151114788 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc814017070 con 0x7fc81c14f5c0 2024-03-28T23:34:22.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.336+0000 7fc803fff640 1 -- 172.21.15.161:0/2151114788 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc814003da0 con 0x7fc81c14f5c0 2024-03-28T23:34:22.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.336+0000 7fc821bd0640 1 -- 172.21.15.161:0/2151114788 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc81c154890 con 0x7fc81c14f5c0 2024-03-28T23:34:22.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.336+0000 7fc803fff640 1 -- 172.21.15.161:0/2151114788 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8140052b0 con 0x7fc81c14f5c0 2024-03-28T23:34:22.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.336+0000 7fc821bd0640 1 -- 172.21.15.161:0/2151114788 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc81c0c0220 con 0x7fc81c14f5c0 2024-03-28T23:34:22.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.337+0000 7fc821bd0640 1 -- 172.21.15.161:0/2151114788 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc81c04f4b0 con 0x7fc81c14f5c0 2024-03-28T23:34:22.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.338+0000 7fc803fff640 1 -- 172.21.15.161:0/2151114788 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc81401a070 con 0x7fc81c14f5c0 2024-03-28T23:34:22.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.338+0000 7fc803fff640 1 --2- 172.21.15.161:0/2151114788 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc80806b4d0 0x7fc80806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:22.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.338+0000 7fc81b7fe640 1 --2- 172.21.15.161:0/2151114788 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc80806b4d0 0x7fc80806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:22.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.338+0000 7fc803fff640 1 -- 172.21.15.161:0/2151114788 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5421+0+0 (secure 0 0 0) 0x7fc814089710 con 0x7fc81c14f5c0 2024-03-28T23:34:22.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.338+0000 7fc81b7fe640 1 --2- 172.21.15.161:0/2151114788 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc80806b4d0 0x7fc80806d990 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fc80c009890 tx=0x7fc80c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:22.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.343+0000 7fc803fff640 1 -- 172.21.15.161:0/2151114788 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc8140586b0 con 0x7fc81c14f5c0 2024-03-28T23:34:22.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.519+0000 7fc821bd0640 1 -- 172.21.15.161:0/2151114788 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fc81c0a8cc0 con 0x7fc81c14f5c0 2024-03-28T23:34:22.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.523+0000 7fc803fff640 1 -- 172.21.15.161:0/2151114788 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v38) ==== 104+0+0 (secure 0 0 0) 0x7fc814057fb0 con 0x7fc81c14f5c0 2024-03-28T23:34:22.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.525+0000 7fc821bd0640 1 -- 172.21.15.161:0/2151114788 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc80806b4d0 msgr2=0x7fc80806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:22.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.525+0000 7fc821bd0640 1 --2- 172.21.15.161:0/2151114788 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc80806b4d0 0x7fc80806d990 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fc80c009890 tx=0x7fc80c008040 comp rx=0 tx=0).stop 2024-03-28T23:34:22.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.525+0000 7fc821bd0640 1 -- 172.21.15.161:0/2151114788 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc81c14f5c0 msgr2=0x7fc81c1eaeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:22.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.525+0000 7fc821bd0640 1 --2- 172.21.15.161:0/2151114788 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc81c14f5c0 0x7fc81c1eaeb0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fc81400c780 tx=0x7fc81400cc40 comp rx=0 tx=0).stop 2024-03-28T23:34:22.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.525+0000 7fc821bd0640 1 -- 172.21.15.161:0/2151114788 shutdown_connections 2024-03-28T23:34:22.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.525+0000 7fc821bd0640 1 --2- 172.21.15.161:0/2151114788 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc80806b4d0 0x7fc80806d990 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.525+0000 7fc821bd0640 1 --2- 172.21.15.161:0/2151114788 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc81c14f5c0 0x7fc81c1eaeb0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.525+0000 7fc821bd0640 1 --2- 172.21.15.161:0/2151114788 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc81c0c2690 0x7fc81c1553b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.525+0000 7fc821bd0640 1 --2- 172.21.15.161:0/2151114788 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc81c06cdd0 0x7fc81c0bfce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.525+0000 7fc821bd0640 1 -- 172.21.15.161:0/2151114788 >> 172.21.15.161:0/2151114788 conn(0x7fc81c0a83f0 msgr2=0x7fc81c14fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:22.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.525+0000 7fc821bd0640 1 -- 172.21.15.161:0/2151114788 shutdown_connections 2024-03-28T23:34:22.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.525+0000 7fc821bd0640 1 -- 172.21.15.161:0/2151114788 wait complete. 2024-03-28T23:34:22.536 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-03-28T23:34:22.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.639+0000 7f489ae09640 1 Processor -- start 2024-03-28T23:34:22.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.639+0000 7f489ae09640 1 -- start start 2024-03-28T23:34:22.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.640+0000 7f489ae09640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4894152bc0 0x7f48940c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:22.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.640+0000 7f489ae09640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48941589b0 0x7f4894158e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:22.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.640+0000 7f489ae09640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48940c6f60 con 0x7f48940c8a80 2024-03-28T23:34:22.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.640+0000 7f489ae09640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f48940c6120 con 0x7f4894152bc0 2024-03-28T23:34:22.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.640+0000 7f489ae09640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f48940c6400 con 0x7f48941589b0 2024-03-28T23:34:22.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.640+0000 7f4898ba5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4894152bc0 0x7f48940c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:22.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.640+0000 7f4898ba5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4894152bc0 0x7f48940c0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43310/0 (socket says 172.21.15.161:43310) 2024-03-28T23:34:22.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.640+0000 7f4898ba5640 1 -- 172.21.15.161:0/455730530 learned_addr learned my addr 172.21.15.161:0/455730530 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:22.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.641+0000 7f48993a6640 1 --2- 172.21.15.161:0/455730530 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48941589b0 0x7f4894158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:22.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.641+0000 7f4898ba5640 1 -- 172.21.15.161:0/455730530 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48941589b0 msgr2=0x7f4894158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:22.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.641+0000 7f4898ba5640 1 --2- 172.21.15.161:0/455730530 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48941589b0 0x7f4894158e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.641+0000 7f4898ba5640 1 -- 172.21.15.161:0/455730530 >> v1:172.21.15.170:6789/0 conn(0x7f48940c8a80 legacy=0x7f48940c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:22.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.641+0000 7f4898ba5640 1 -- 172.21.15.161:0/455730530 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48940c7240 con 0x7f4894152bc0 2024-03-28T23:34:22.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.641+0000 7f4898ba5640 1 --2- 172.21.15.161:0/455730530 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4894152bc0 0x7f48940c0330 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f4884009e50 tx=0x7f488402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec4b8aaef5d333b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:22.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.641+0000 7f48937fe640 1 -- 172.21.15.161:0/455730530 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488402ca70 con 0x7f4894152bc0 2024-03-28T23:34:22.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.641+0000 7f48937fe640 1 -- 172.21.15.161:0/455730530 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f488402cc10 con 0x7f4894152bc0 2024-03-28T23:34:22.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.642+0000 7f48937fe640 1 -- 172.21.15.161:0/455730530 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48840343f0 con 0x7f4894152bc0 2024-03-28T23:34:22.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.642+0000 7f489ae09640 1 -- 172.21.15.161:0/455730530 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4894152bc0 msgr2=0x7f48940c0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:22.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.642+0000 7f489ae09640 1 --2- 172.21.15.161:0/455730530 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4894152bc0 0x7f48940c0330 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f4884009e50 tx=0x7f488402bed0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.642+0000 7f489ae09640 1 -- 172.21.15.161:0/455730530 shutdown_connections 2024-03-28T23:34:22.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.642+0000 7f489ae09640 1 --2- 172.21.15.161:0/455730530 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48941589b0 0x7f4894158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.642+0000 7f489ae09640 1 --2- 172.21.15.161:0/455730530 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4894152bc0 0x7f48940c0330 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.642+0000 7f489ae09640 1 -- 172.21.15.161:0/455730530 >> 172.21.15.161:0/455730530 conn(0x7f48940caa80 msgr2=0x7f48940cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:22.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.642+0000 7f489ae09640 1 -- 172.21.15.161:0/455730530 shutdown_connections 2024-03-28T23:34:22.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.642+0000 7f489ae09640 1 -- 172.21.15.161:0/455730530 wait complete. 2024-03-28T23:34:22.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.643+0000 7f489ae09640 1 Processor -- start 2024-03-28T23:34:22.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.643+0000 7f489ae09640 1 -- start start 2024-03-28T23:34:22.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.644+0000 7f489ae09640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48940c8a80 0x7f4894151660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:22.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.644+0000 7f489ae09640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4894152bc0 0x7f48941412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:22.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.644+0000 7f489ae09640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48941589b0 0x7f4894141830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:22.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.644+0000 7f489ae09640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4894152fa0 con 0x7f48940c8a80 2024-03-28T23:34:22.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.644+0000 7f489ae09640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f48940c95f0 con 0x7f48941589b0 2024-03-28T23:34:22.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.644+0000 7f489ae09640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f48941eab60 con 0x7f4894152bc0 2024-03-28T23:34:22.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.644+0000 7f4898ba5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48940c8a80 0x7f4894151660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:22.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.644+0000 7f4898ba5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48940c8a80 0x7f4894151660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43322/0 (socket says 172.21.15.161:43322) 2024-03-28T23:34:22.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.644+0000 7f4898ba5640 1 -- 172.21.15.161:0/3321104127 learned_addr learned my addr 172.21.15.161:0/3321104127 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:22.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.644+0000 7f4898ba5640 1 -- 172.21.15.161:0/3321104127 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4894152bc0 msgr2=0x7f48941412f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:22.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.644+0000 7f4898ba5640 1 --2- 172.21.15.161:0/3321104127 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4894152bc0 0x7f48941412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.644+0000 7f4898ba5640 1 -- 172.21.15.161:0/3321104127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48941589b0 msgr2=0x7f4894141830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:22.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.644+0000 7f4898ba5640 1 --2- 172.21.15.161:0/3321104127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48941589b0 0x7f4894141830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.644+0000 7f4898ba5640 1 -- 172.21.15.161:0/3321104127 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48941eae40 con 0x7f48940c8a80 2024-03-28T23:34:22.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.645+0000 7f4898ba5640 1 --2- 172.21.15.161:0/3321104127 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48940c8a80 0x7f4894151660 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f4884009e50 tx=0x7f4884004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:22.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.645+0000 7f48917fa640 1 -- 172.21.15.161:0/3321104127 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4884043020 con 0x7f48940c8a80 2024-03-28T23:34:22.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.645+0000 7f48917fa640 1 -- 172.21.15.161:0/3321104127 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f488403e450 con 0x7f48940c8a80 2024-03-28T23:34:22.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.645+0000 7f489ae09640 1 -- 172.21.15.161:0/3321104127 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48941eb120 con 0x7f48940c8a80 2024-03-28T23:34:22.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.645+0000 7f48917fa640 1 -- 172.21.15.161:0/3321104127 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488403e840 con 0x7f48940c8a80 2024-03-28T23:34:22.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.645+0000 7f489ae09640 1 -- 172.21.15.161:0/3321104127 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48941eb6e0 con 0x7f48940c8a80 2024-03-28T23:34:22.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.646+0000 7f489ae09640 1 -- 172.21.15.161:0/3321104127 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f489404f4b0 con 0x7f48940c8a80 2024-03-28T23:34:22.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.646+0000 7f48917fa640 1 -- 172.21.15.161:0/3321104127 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4884046020 con 0x7f48940c8a80 2024-03-28T23:34:22.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.646+0000 7f48917fa640 1 --2- 172.21.15.161:0/3321104127 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f487406b4d0 0x7f487406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:22.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.647+0000 7f48917fa640 1 -- 172.21.15.161:0/3321104127 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5421+0+0 (secure 0 0 0) 0x7f48840ac9f0 con 0x7f48940c8a80 2024-03-28T23:34:22.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.647+0000 7f4893fff640 1 --2- 172.21.15.161:0/3321104127 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f487406b4d0 0x7f487406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:22.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.647+0000 7f4893fff640 1 --2- 172.21.15.161:0/3321104127 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f487406b4d0 0x7f487406d990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f487c006fd0 tx=0x7f487c006bc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:22.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.651+0000 7f48917fa640 1 -- 172.21.15.161:0/3321104127 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f488404d030 con 0x7f48940c8a80 2024-03-28T23:34:22.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.823+0000 7f489ae09640 1 -- 172.21.15.161:0/3321104127 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) -- 0x7f489414bf90 con 0x7f48940c8a80 2024-03-28T23:34:22.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.826+0000 7f48917fa640 1 -- 172.21.15.161:0/3321104127 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v39) ==== 96+0+0 (secure 0 0 0) 0x7f488407ba20 con 0x7f48940c8a80 2024-03-28T23:34:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.828+0000 7f489ae09640 1 -- 172.21.15.161:0/3321104127 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f487406b4d0 msgr2=0x7f487406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.828+0000 7f489ae09640 1 --2- 172.21.15.161:0/3321104127 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f487406b4d0 0x7f487406d990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f487c006fd0 tx=0x7f487c006bc0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.828+0000 7f489ae09640 1 -- 172.21.15.161:0/3321104127 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48940c8a80 msgr2=0x7f4894151660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.828+0000 7f489ae09640 1 --2- 172.21.15.161:0/3321104127 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48940c8a80 0x7f4894151660 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f4884009e50 tx=0x7f4884004a40 comp rx=0 tx=0).stop 2024-03-28T23:34:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.828+0000 7f489ae09640 1 -- 172.21.15.161:0/3321104127 shutdown_connections 2024-03-28T23:34:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.828+0000 7f489ae09640 1 --2- 172.21.15.161:0/3321104127 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f487406b4d0 0x7f487406d990 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.828+0000 7f489ae09640 1 --2- 172.21.15.161:0/3321104127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48941589b0 0x7f4894141830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.828+0000 7f489ae09640 1 --2- 172.21.15.161:0/3321104127 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4894152bc0 0x7f48941412f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.828+0000 7f489ae09640 1 --2- 172.21.15.161:0/3321104127 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48940c8a80 0x7f4894151660 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.828+0000 7f489ae09640 1 -- 172.21.15.161:0/3321104127 >> 172.21.15.161:0/3321104127 conn(0x7f48940caa80 msgr2=0x7f489414b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.828+0000 7f489ae09640 1 -- 172.21.15.161:0/3321104127 shutdown_connections 2024-03-28T23:34:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.828+0000 7f489ae09640 1 -- 172.21.15.161:0/3321104127 wait complete. 2024-03-28T23:34:22.839 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.b mds_join_fs 2024-03-28T23:34:22.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.943+0000 7f06a66aa640 1 Processor -- start 2024-03-28T23:34:22.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.944+0000 7f06a66aa640 1 -- start start 2024-03-28T23:34:22.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.944+0000 7f06a66aa640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f06a0150520 0x7f06a00ab280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:22.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.944+0000 7f06a66aa640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06a00a96f0 0x7f06a015d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:22.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.944+0000 7f06a66aa640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06a00c2f50 con 0x7f06a00c1040 2024-03-28T23:34:22.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.944+0000 7f06a66aa640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f06a00a9c20 con 0x7f06a0150520 2024-03-28T23:34:22.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.944+0000 7f06a66aa640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f06a0150900 con 0x7f06a00a96f0 2024-03-28T23:34:22.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.945+0000 7f069ffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f06a0150520 0x7f06a00ab280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:22.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.945+0000 7f06a4c47640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06a00a96f0 0x7f06a015d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:22.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.945+0000 7f069f7fe640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f06a00c1040 0x7f06a01535b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:37628/0 (socket says 172.21.15.161:37628) 2024-03-28T23:34:22.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.945+0000 7f069f7fe640 1 -- 172.21.15.161:0/1680282603 learned_addr learned my addr 172.21.15.161:0/1680282603 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:22.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.945+0000 7f069ffff640 1 -- 172.21.15.161:0/1680282603 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06a00a96f0 msgr2=0x7f06a015d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:22.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.945+0000 7f069ffff640 1 --2- 172.21.15.161:0/1680282603 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06a00a96f0 0x7f06a015d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.945+0000 7f069ffff640 1 -- 172.21.15.161:0/1680282603 >> v1:172.21.15.170:6789/0 conn(0x7f06a00c1040 legacy=0x7f06a01535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:22.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.945+0000 7f069ffff640 1 -- 172.21.15.161:0/1680282603 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a01530d0 con 0x7f06a0150520 2024-03-28T23:34:22.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.946+0000 7f069ffff640 1 --2- 172.21.15.161:0/1680282603 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06a0150520 0x7f06a00ab280 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f06900097a0 tx=0x7f069002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93d464d56b00c965 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:22.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.946+0000 7f069effd640 1 -- 172.21.15.161:0/1680282603 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06900315a0 con 0x7f06a0150520 2024-03-28T23:34:22.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.946+0000 7f069effd640 1 -- 172.21.15.161:0/1680282603 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0690037070 con 0x7f06a0150520 2024-03-28T23:34:22.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.946+0000 7f069effd640 1 -- 172.21.15.161:0/1680282603 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0690031960 con 0x7f06a0150520 2024-03-28T23:34:22.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.946+0000 7f06a66aa640 1 -- 172.21.15.161:0/1680282603 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06a0150520 msgr2=0x7f06a00ab280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:22.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.946+0000 7f06a66aa640 1 --2- 172.21.15.161:0/1680282603 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06a0150520 0x7f06a00ab280 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f06900097a0 tx=0x7f069002b840 comp rx=0 tx=0).stop 2024-03-28T23:34:22.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.947+0000 7f06a66aa640 1 -- 172.21.15.161:0/1680282603 shutdown_connections 2024-03-28T23:34:22.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.947+0000 7f06a66aa640 1 --2- 172.21.15.161:0/1680282603 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06a00a96f0 0x7f06a015d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.947+0000 7f06a66aa640 1 --2- 172.21.15.161:0/1680282603 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06a0150520 0x7f06a00ab280 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.947+0000 7f06a66aa640 1 -- 172.21.15.161:0/1680282603 >> 172.21.15.161:0/1680282603 conn(0x7f06a00a83f0 msgr2=0x7f06a00a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:22.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.947+0000 7f06a66aa640 1 -- 172.21.15.161:0/1680282603 shutdown_connections 2024-03-28T23:34:22.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.947+0000 7f06a66aa640 1 -- 172.21.15.161:0/1680282603 wait complete. 2024-03-28T23:34:22.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.948+0000 7f06a66aa640 1 Processor -- start 2024-03-28T23:34:22.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.948+0000 7f06a66aa640 1 -- start start 2024-03-28T23:34:22.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.948+0000 7f06a66aa640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06a00a96f0 0x7f06a01e8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:22.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.948+0000 7f06a66aa640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06a00c1040 0x7f06a01442e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:22.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.948+0000 7f06a66aa640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06a0150520 0x7f06a0152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:22.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.948+0000 7f06a66aa640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f06a00c0c00 con 0x7f06a00a96f0 2024-03-28T23:34:22.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.948+0000 7f06a66aa640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f06a00c2c70 con 0x7f06a0150520 2024-03-28T23:34:22.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.948+0000 7f06a66aa640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f06a00c0d80 con 0x7f06a00c1040 2024-03-28T23:34:22.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.949+0000 7f069ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06a00a96f0 0x7f06a01e8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:22.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.949+0000 7f069ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06a00a96f0 0x7f06a01e8c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43324/0 (socket says 172.21.15.161:43324) 2024-03-28T23:34:22.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.949+0000 7f069ffff640 1 -- 172.21.15.161:0/2233489497 learned_addr learned my addr 172.21.15.161:0/2233489497 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:22.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.949+0000 7f069f7fe640 1 --2- 172.21.15.161:0/2233489497 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06a00c1040 0x7f06a01442e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:22.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.949+0000 7f06a4c47640 1 --2- 172.21.15.161:0/2233489497 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06a0150520 0x7f06a0152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:22.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.949+0000 7f069ffff640 1 -- 172.21.15.161:0/2233489497 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06a00c1040 msgr2=0x7f06a01442e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:22.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.949+0000 7f069ffff640 1 --2- 172.21.15.161:0/2233489497 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06a00c1040 0x7f06a01442e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.949+0000 7f069ffff640 1 -- 172.21.15.161:0/2233489497 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06a0150520 msgr2=0x7f06a0152220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:22.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.949+0000 7f069ffff640 1 --2- 172.21.15.161:0/2233489497 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06a0150520 0x7f06a0152220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:22.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.949+0000 7f069ffff640 1 -- 172.21.15.161:0/2233489497 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a0151c90 con 0x7f06a00a96f0 2024-03-28T23:34:22.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.949+0000 7f069ffff640 1 --2- 172.21.15.161:0/2233489497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06a00a96f0 0x7f06a01e8c20 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f0690034040 tx=0x7f06900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:22.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.949+0000 7f069cff9640 1 -- 172.21.15.161:0/2233489497 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0690032630 con 0x7f06a00a96f0 2024-03-28T23:34:22.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.949+0000 7f069cff9640 1 -- 172.21.15.161:0/2233489497 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06900327d0 con 0x7f06a00a96f0 2024-03-28T23:34:22.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.949+0000 7f06a66aa640 1 -- 172.21.15.161:0/2233489497 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06a0150da0 con 0x7f06a00a96f0 2024-03-28T23:34:22.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.949+0000 7f069cff9640 1 -- 172.21.15.161:0/2233489497 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0690032bc0 con 0x7f06a00a96f0 2024-03-28T23:34:22.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.949+0000 7f06a66aa640 1 -- 172.21.15.161:0/2233489497 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06a0151360 con 0x7f06a00a96f0 2024-03-28T23:34:22.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.950+0000 7f06a66aa640 1 -- 172.21.15.161:0/2233489497 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06a014fd50 con 0x7f06a00a96f0 2024-03-28T23:34:22.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.951+0000 7f069cff9640 1 -- 172.21.15.161:0/2233489497 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f069004d3f0 con 0x7f06a00a96f0 2024-03-28T23:34:22.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.951+0000 7f069cff9640 1 --2- 172.21.15.161:0/2233489497 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f068c06b4d0 0x7f068c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:22.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.951+0000 7f069cff9640 1 -- 172.21.15.161:0/2233489497 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5421+0+0 (secure 0 0 0) 0x7f069002f030 con 0x7f06a00a96f0 2024-03-28T23:34:22.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.951+0000 7f069f7fe640 1 --2- 172.21.15.161:0/2233489497 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f068c06b4d0 0x7f068c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:22.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.952+0000 7f069f7fe640 1 --2- 172.21.15.161:0/2233489497 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f068c06b4d0 0x7f068c06d990 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f0694007fd0 tx=0x7f0694007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:22.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:22.956+0000 7f069cff9640 1 -- 172.21.15.161:0/2233489497 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f06a014fd50 con 0x7f06a00a96f0 2024-03-28T23:34:23.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.126+0000 7f06a66aa640 1 -- 172.21.15.161:0/2233489497 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"} v 0) -- 0x7f06a00693e0 con 0x7f06a00a96f0 2024-03-28T23:34:23.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.129+0000 7f069cff9640 1 -- 172.21.15.161:0/2233489497 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"}]=0 v40) ==== 96+0+0 (secure 0 0 0) 0x7f06a00693e0 con 0x7f06a00a96f0 2024-03-28T23:34:23.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.132+0000 7f06a66aa640 1 -- 172.21.15.161:0/2233489497 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f068c06b4d0 msgr2=0x7f068c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:23.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.132+0000 7f06a66aa640 1 --2- 172.21.15.161:0/2233489497 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f068c06b4d0 0x7f068c06d990 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f0694007fd0 tx=0x7f0694007e80 comp rx=0 tx=0).stop 2024-03-28T23:34:23.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.132+0000 7f06a66aa640 1 -- 172.21.15.161:0/2233489497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06a00a96f0 msgr2=0x7f06a01e8c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:23.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.132+0000 7f06a66aa640 1 --2- 172.21.15.161:0/2233489497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06a00a96f0 0x7f06a01e8c20 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f0690034040 tx=0x7f06900046c0 comp rx=0 tx=0).stop 2024-03-28T23:34:23.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.132+0000 7f06a66aa640 1 -- 172.21.15.161:0/2233489497 shutdown_connections 2024-03-28T23:34:23.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.132+0000 7f06a66aa640 1 --2- 172.21.15.161:0/2233489497 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f068c06b4d0 0x7f068c06d990 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:23.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.132+0000 7f06a66aa640 1 --2- 172.21.15.161:0/2233489497 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06a0150520 0x7f06a0152220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:23.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.132+0000 7f06a66aa640 1 --2- 172.21.15.161:0/2233489497 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06a00c1040 0x7f06a01442e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:23.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.132+0000 7f06a66aa640 1 --2- 172.21.15.161:0/2233489497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06a00a96f0 0x7f06a01e8c20 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:23.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.132+0000 7f06a66aa640 1 -- 172.21.15.161:0/2233489497 >> 172.21.15.161:0/2233489497 conn(0x7f06a00a83f0 msgr2=0x7f06a01453b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:23.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.132+0000 7f06a66aa640 1 -- 172.21.15.161:0/2233489497 shutdown_connections 2024-03-28T23:34:23.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.132+0000 7f06a66aa640 1 -- 172.21.15.161:0/2233489497 wait complete. 2024-03-28T23:34:23.143 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay' 2024-03-28T23:34:23.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.245+0000 7f062b91e640 1 Processor -- start 2024-03-28T23:34:23.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.246+0000 7f062b91e640 1 -- start start 2024-03-28T23:34:23.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.246+0000 7f062b91e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0624152890 0x7f06240c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:23.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.246+0000 7f062b91e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f06240c8ed0 0x7f06240c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:23.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.246+0000 7f062b91e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0624158a30 con 0x7f06240c5950 2024-03-28T23:34:23.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.246+0000 7f062b91e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f06240c5db0 con 0x7f06240c8ed0 2024-03-28T23:34:23.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.246+0000 7f062b91e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f06240c8480 con 0x7f0624152890 2024-03-28T23:34:23.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.246+0000 7f0628eb9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f06240c8ed0 0x7f06240c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:23.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.246+0000 7f0629ebb640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f06240c5950 0x7f06241553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:58322/0 (socket says 172.21.15.161:58322) 2024-03-28T23:34:23.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.247+0000 7f0629ebb640 1 -- 172.21.15.161:0/345828011 learned_addr learned my addr 172.21.15.161:0/345828011 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:23.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.247+0000 7f06296ba640 1 --2- 172.21.15.161:0/345828011 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0624152890 0x7f06240c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:23.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.247+0000 7f0628eb9640 1 -- 172.21.15.161:0/345828011 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0624152890 msgr2=0x7f06240c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:23.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.247+0000 7f0628eb9640 1 --2- 172.21.15.161:0/345828011 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0624152890 0x7f06240c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:23.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.247+0000 7f0628eb9640 1 -- 172.21.15.161:0/345828011 >> v1:172.21.15.161:6789/0 conn(0x7f06240c5950 legacy=0x7f06241553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:23.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.247+0000 7f0617fff640 1 -- 172.21.15.161:0/345828011 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3281763956 0 0) 0x7f0624158a30 con 0x7f06240c5950 2024-03-28T23:34:23.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.247+0000 7f0628eb9640 1 -- 172.21.15.161:0/345828011 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0624158d10 con 0x7f06240c8ed0 2024-03-28T23:34:23.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.248+0000 7f0628eb9640 1 --2- 172.21.15.161:0/345828011 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06240c8ed0 0x7f06240c5410 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f0618009790 tx=0x7f061802b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf51e1c7b4376d40 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:23.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.248+0000 7f0617fff640 1 -- 172.21.15.161:0/345828011 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0618039070 con 0x7f06240c8ed0 2024-03-28T23:34:23.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.248+0000 7f0617fff640 1 -- 172.21.15.161:0/345828011 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06180316f0 con 0x7f06240c8ed0 2024-03-28T23:34:23.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.248+0000 7f0617fff640 1 -- 172.21.15.161:0/345828011 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0618031ac0 con 0x7f06240c8ed0 2024-03-28T23:34:23.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.248+0000 7f062b91e640 1 -- 172.21.15.161:0/345828011 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06240c8ed0 msgr2=0x7f06240c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:23.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.248+0000 7f062b91e640 1 --2- 172.21.15.161:0/345828011 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06240c8ed0 0x7f06240c5410 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f0618009790 tx=0x7f061802b790 comp rx=0 tx=0).stop 2024-03-28T23:34:23.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.248+0000 7f062b91e640 1 -- 172.21.15.161:0/345828011 shutdown_connections 2024-03-28T23:34:23.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.248+0000 7f062b91e640 1 --2- 172.21.15.161:0/345828011 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06240c8ed0 0x7f06240c5410 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:23.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.248+0000 7f062b91e640 1 --2- 172.21.15.161:0/345828011 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0624152890 0x7f06240c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:23.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.248+0000 7f062b91e640 1 -- 172.21.15.161:0/345828011 >> 172.21.15.161:0/345828011 conn(0x7f06240ac620 msgr2=0x7f06240aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:23.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.248+0000 7f062b91e640 1 -- 172.21.15.161:0/345828011 shutdown_connections 2024-03-28T23:34:23.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.248+0000 7f062b91e640 1 -- 172.21.15.161:0/345828011 wait complete. 2024-03-28T23:34:23.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.249+0000 7f062b91e640 1 Processor -- start 2024-03-28T23:34:23.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.249+0000 7f062b91e640 1 -- start start 2024-03-28T23:34:23.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.250+0000 7f062b91e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06240c5950 0x7f0624147080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:23.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.250+0000 7f062b91e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06240c8ed0 0x7f06241e7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:23.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.250+0000 7f062b91e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0624152890 0x7f06241e84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:23.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.250+0000 7f062b91e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f06241592d0 con 0x7f0624152890 2024-03-28T23:34:23.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.250+0000 7f062b91e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0624159640 con 0x7f06240c5950 2024-03-28T23:34:23.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.250+0000 7f062b91e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0624152c70 con 0x7f06240c8ed0 2024-03-28T23:34:23.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.250+0000 7f0629ebb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0624152890 0x7f06241e84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:23.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.250+0000 7f0629ebb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0624152890 0x7f06241e84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43346/0 (socket says 172.21.15.161:43346) 2024-03-28T23:34:23.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.250+0000 7f0629ebb640 1 -- 172.21.15.161:0/353286515 learned_addr learned my addr 172.21.15.161:0/353286515 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:23.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.250+0000 7f0628eb9640 1 --2- 172.21.15.161:0/353286515 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06240c8ed0 0x7f06241e7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:23.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.250+0000 7f06296ba640 1 --2- 172.21.15.161:0/353286515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06240c5950 0x7f0624147080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:23.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.250+0000 7f0629ebb640 1 -- 172.21.15.161:0/353286515 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06240c8ed0 msgr2=0x7f06241e7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:23.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.250+0000 7f0629ebb640 1 --2- 172.21.15.161:0/353286515 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06240c8ed0 0x7f06241e7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:23.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.250+0000 7f0629ebb640 1 -- 172.21.15.161:0/353286515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06240c5950 msgr2=0x7f0624147080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:23.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.250+0000 7f0629ebb640 1 --2- 172.21.15.161:0/353286515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06240c5950 0x7f0624147080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:23.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.251+0000 7f0629ebb640 1 -- 172.21.15.161:0/353286515 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06241507a0 con 0x7f0624152890 2024-03-28T23:34:23.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.251+0000 7f0628eb9640 1 --2- 172.21.15.161:0/353286515 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06240c8ed0 0x7f06241e7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:23.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.251+0000 7f06296ba640 1 --2- 172.21.15.161:0/353286515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06240c5950 0x7f0624147080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:23.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.251+0000 7f0629ebb640 1 --2- 172.21.15.161:0/353286515 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0624152890 0x7f06241e84b0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f061c00c700 tx=0x7f061c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:23.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.251+0000 7f0615ffb640 1 -- 172.21.15.161:0/353286515 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f061c018070 con 0x7f0624152890 2024-03-28T23:34:23.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.251+0000 7f0615ffb640 1 -- 172.21.15.161:0/353286515 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f061c00ce10 con 0x7f0624152890 2024-03-28T23:34:23.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.251+0000 7f062b91e640 1 -- 172.21.15.161:0/353286515 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06241ee390 con 0x7f0624152890 2024-03-28T23:34:23.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.251+0000 7f0615ffb640 1 -- 172.21.15.161:0/353286515 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f061c00a860 con 0x7f0624152890 2024-03-28T23:34:23.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.251+0000 7f062b91e640 1 -- 172.21.15.161:0/353286515 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06241ee950 con 0x7f0624152890 2024-03-28T23:34:23.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.252+0000 7f062b91e640 1 -- 172.21.15.161:0/353286515 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f062404f4b0 con 0x7f0624152890 2024-03-28T23:34:23.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.252+0000 7f0615ffb640 1 -- 172.21.15.161:0/353286515 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f061c01b070 con 0x7f0624152890 2024-03-28T23:34:23.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.253+0000 7f0615ffb640 1 --2- 172.21.15.161:0/353286515 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f060806b500 0x7f060806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:23.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.253+0000 7f06296ba640 1 --2- 172.21.15.161:0/353286515 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f060806b500 0x7f060806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:23.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.253+0000 7f0615ffb640 1 -- 172.21.15.161:0/353286515 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5421+0+0 (secure 0 0 0) 0x7f061c089600 con 0x7f0624152890 2024-03-28T23:34:23.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.253+0000 7f06296ba640 1 --2- 172.21.15.161:0/353286515 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f060806b500 0x7f060806d9c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0620006fd0 tx=0x7f0620008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:23.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.257+0000 7f0615ffb640 1 -- 172.21.15.161:0/353286515 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f061c058630 con 0x7f0624152890 2024-03-28T23:34:23.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:23.497+0000 7f062b91e640 1 -- 172.21.15.161:0/353286515 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay"]} v 0) -- 0x7f062414eac0 con 0x7f0624152890 2024-03-28T23:34:24.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.210+0000 7f0615ffb640 1 -- 172.21.15.161:0/353286515 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay"]}]=0 v273) ==== 153+0+0 (secure 0 0 0) 0x7f061c057f30 con 0x7f0624152890 2024-03-28T23:34:24.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.213+0000 7f062b91e640 1 -- 172.21.15.161:0/353286515 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f060806b500 msgr2=0x7f060806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:24.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.213+0000 7f062b91e640 1 --2- 172.21.15.161:0/353286515 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f060806b500 0x7f060806d9c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0620006fd0 tx=0x7f0620008040 comp rx=0 tx=0).stop 2024-03-28T23:34:24.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.213+0000 7f062b91e640 1 -- 172.21.15.161:0/353286515 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0624152890 msgr2=0x7f06241e84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:24.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.213+0000 7f062b91e640 1 --2- 172.21.15.161:0/353286515 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0624152890 0x7f06241e84b0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f061c00c700 tx=0x7f061c00cbc0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.213+0000 7f062b91e640 1 -- 172.21.15.161:0/353286515 shutdown_connections 2024-03-28T23:34:24.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.213+0000 7f062b91e640 1 --2- 172.21.15.161:0/353286515 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f060806b500 0x7f060806d9c0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.213+0000 7f062b91e640 1 --2- 172.21.15.161:0/353286515 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0624152890 0x7f06241e84b0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.213+0000 7f062b91e640 1 --2- 172.21.15.161:0/353286515 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06240c8ed0 0x7f06241e7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.213+0000 7f062b91e640 1 --2- 172.21.15.161:0/353286515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06240c5950 0x7f0624147080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.213+0000 7f062b91e640 1 -- 172.21.15.161:0/353286515 >> 172.21.15.161:0/353286515 conn(0x7f06240ac620 msgr2=0x7f06240c8660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:24.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.213+0000 7f062b91e640 1 -- 172.21.15.161:0/353286515 shutdown_connections 2024-03-28T23:34:24.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.213+0000 7f062b91e640 1 -- 172.21.15.161:0/353286515 wait complete. 2024-03-28T23:34:24.225 INFO:tasks.cephfs_test_runner:That a standby-replay daemon with weak affinity is replaced by a stronger one. ... ok 2024-03-28T23:34:24.226 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_steady (tasks.cephfs.test_failover.TestClusterAffinity) That a sole MDS with mds_join_fs set will come back as active eventually even after failover. 2024-03-28T23:34:24.226 INFO:tasks.cephfs_test_runner:test_join_fs_steady (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-28T23:34:24.226 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:34:24.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.329+0000 7fc8d639a640 1 Processor -- start 2024-03-28T23:34:24.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.329+0000 7fc8d639a640 1 -- start start 2024-03-28T23:34:24.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.329+0000 7fc8d639a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc8d014be70 0x7fc8d00ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:24.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.329+0000 7fc8d639a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8d014c250 con 0x7fc8d00ab6e0 2024-03-28T23:34:24.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.329+0000 7fc8d639a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8d01529c0 con 0x7fc8d00aacd0 2024-03-28T23:34:24.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.329+0000 7fc8d639a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc8d00a9dc0 con 0x7fc8d014be70 2024-03-28T23:34:24.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.329+0000 7fc8cffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc8d014be70 0x7fc8d00ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:24.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.329+0000 7fc8cf7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc8d00ab6e0 0x7fc8d00aa5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:58332/0 (socket says 172.21.15.161:58332) 2024-03-28T23:34:24.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.329+0000 7fc8cf7fe640 1 -- 172.21.15.161:0/304231926 learned_addr learned my addr 172.21.15.161:0/304231926 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:24.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.330+0000 7fc8cffff640 1 -- 172.21.15.161:0/304231926 >> v1:172.21.15.170:6789/0 conn(0x7fc8d00aacd0 legacy=0x7fc8d015c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:24.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.330+0000 7fc8cffff640 1 -- 172.21.15.161:0/304231926 >> v1:172.21.15.161:6789/0 conn(0x7fc8d00ab6e0 legacy=0x7fc8d00aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:24.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.330+0000 7fc8cffff640 1 -- 172.21.15.161:0/304231926 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d00aa0a0 con 0x7fc8d014be70 2024-03-28T23:34:24.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.330+0000 7fc8ceffd640 1 -- 172.21.15.161:0/304231926 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1613668399 0 0) 0x7fc8d014c250 con 0x7fc8d00ab6e0 2024-03-28T23:34:24.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.330+0000 7fc8cffff640 1 --2- 172.21.15.161:0/304231926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8d014be70 0x7fc8d00ab1a0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fc8c00097a0 tx=0x7fc8c002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a233749337899f0f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:24.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.331+0000 7fc8ceffd640 1 -- 172.21.15.161:0/304231926 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c00315a0 con 0x7fc8d014be70 2024-03-28T23:34:24.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.331+0000 7fc8ceffd640 1 -- 172.21.15.161:0/304231926 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8c0037070 con 0x7fc8d014be70 2024-03-28T23:34:24.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.331+0000 7fc8ceffd640 1 -- 172.21.15.161:0/304231926 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c0031960 con 0x7fc8d014be70 2024-03-28T23:34:24.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.331+0000 7fc8d639a640 1 -- 172.21.15.161:0/304231926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8d014be70 msgr2=0x7fc8d00ab1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:24.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.331+0000 7fc8d639a640 1 --2- 172.21.15.161:0/304231926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8d014be70 0x7fc8d00ab1a0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fc8c00097a0 tx=0x7fc8c002b840 comp rx=0 tx=0).stop 2024-03-28T23:34:24.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.331+0000 7fc8d639a640 1 -- 172.21.15.161:0/304231926 shutdown_connections 2024-03-28T23:34:24.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.331+0000 7fc8d639a640 1 --2- 172.21.15.161:0/304231926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8d014be70 0x7fc8d00ab1a0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.331+0000 7fc8d639a640 1 -- 172.21.15.161:0/304231926 >> 172.21.15.161:0/304231926 conn(0x7fc8d00c6670 msgr2=0x7fc8d00c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:24.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.331+0000 7fc8d639a640 1 -- 172.21.15.161:0/304231926 shutdown_connections 2024-03-28T23:34:24.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.331+0000 7fc8d639a640 1 -- 172.21.15.161:0/304231926 wait complete. 2024-03-28T23:34:24.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.332+0000 7fc8d639a640 1 Processor -- start 2024-03-28T23:34:24.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.333+0000 7fc8d639a640 1 -- start start 2024-03-28T23:34:24.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.333+0000 7fc8d639a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8d00aacd0 0x7fc8d01eb040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:24.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.333+0000 7fc8d639a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8d00ab6e0 0x7fc8d01eb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:24.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.333+0000 7fc8d639a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8d014be70 0x7fc8d0148560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:24.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.333+0000 7fc8d639a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d006b2d0 con 0x7fc8d00ab6e0 2024-03-28T23:34:24.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.333+0000 7fc8d639a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d006b150 con 0x7fc8d014be70 2024-03-28T23:34:24.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.333+0000 7fc8d639a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc8d00c1520 con 0x7fc8d00aacd0 2024-03-28T23:34:24.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.333+0000 7fc8cf7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8d00ab6e0 0x7fc8d01eb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:24.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.333+0000 7fc8cf7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8d00ab6e0 0x7fc8d01eb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43362/0 (socket says 172.21.15.161:43362) 2024-03-28T23:34:24.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.333+0000 7fc8cf7fe640 1 -- 172.21.15.161:0/1528235661 learned_addr learned my addr 172.21.15.161:0/1528235661 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:24.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.334+0000 7fc8d4937640 1 --2- 172.21.15.161:0/1528235661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8d014be70 0x7fc8d0148560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:24.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.334+0000 7fc8cffff640 1 --2- 172.21.15.161:0/1528235661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8d00aacd0 0x7fc8d01eb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:24.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.334+0000 7fc8cf7fe640 1 -- 172.21.15.161:0/1528235661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8d00aacd0 msgr2=0x7fc8d01eb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:24.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.334+0000 7fc8cf7fe640 1 --2- 172.21.15.161:0/1528235661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8d00aacd0 0x7fc8d01eb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.334+0000 7fc8cf7fe640 1 -- 172.21.15.161:0/1528235661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8d014be70 msgr2=0x7fc8d0148560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:24.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.334+0000 7fc8cf7fe640 1 --2- 172.21.15.161:0/1528235661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8d014be70 0x7fc8d0148560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.334+0000 7fc8cf7fe640 1 -- 172.21.15.161:0/1528235661 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d01531b0 con 0x7fc8d00ab6e0 2024-03-28T23:34:24.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.334+0000 7fc8d4937640 1 --2- 172.21.15.161:0/1528235661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8d014be70 0x7fc8d0148560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:24.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.334+0000 7fc8cffff640 1 --2- 172.21.15.161:0/1528235661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8d00aacd0 0x7fc8d01eb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:24.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.334+0000 7fc8cf7fe640 1 --2- 172.21.15.161:0/1528235661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8d00ab6e0 0x7fc8d01eb580 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fc8c400e730 tx=0x7fc8c400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:24.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.334+0000 7fc8ccff9640 1 -- 172.21.15.161:0/1528235661 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c400ee40 con 0x7fc8d00ab6e0 2024-03-28T23:34:24.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.334+0000 7fc8ccff9640 1 -- 172.21.15.161:0/1528235661 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8c4016020 con 0x7fc8d00ab6e0 2024-03-28T23:34:24.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.334+0000 7fc8d639a640 1 -- 172.21.15.161:0/1528235661 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8d0153490 con 0x7fc8d00ab6e0 2024-03-28T23:34:24.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.334+0000 7fc8ccff9640 1 -- 172.21.15.161:0/1528235661 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4007520 con 0x7fc8d00ab6e0 2024-03-28T23:34:24.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.334+0000 7fc8d639a640 1 -- 172.21.15.161:0/1528235661 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8d0073240 con 0x7fc8d00ab6e0 2024-03-28T23:34:24.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.335+0000 7fc8d639a640 1 -- 172.21.15.161:0/1528235661 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8d004f4b0 con 0x7fc8d00ab6e0 2024-03-28T23:34:24.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.335+0000 7fc8ccff9640 1 -- 172.21.15.161:0/1528235661 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc8c40016d0 con 0x7fc8d00ab6e0 2024-03-28T23:34:24.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.336+0000 7fc8ccff9640 1 --2- 172.21.15.161:0/1528235661 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc8bc06b500 0x7fc8bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:24.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.336+0000 7fc8ccff9640 1 -- 172.21.15.161:0/1528235661 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5421+0+0 (secure 0 0 0) 0x7fc8c408c580 con 0x7fc8d00ab6e0 2024-03-28T23:34:24.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.336+0000 7fc8cffff640 1 --2- 172.21.15.161:0/1528235661 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc8bc06b500 0x7fc8bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:24.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.336+0000 7fc8cffff640 1 --2- 172.21.15.161:0/1528235661 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc8bc06b500 0x7fc8bc06d9c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0036000 tx=0x7fc8c0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:24.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.340+0000 7fc8ccff9640 1 -- 172.21.15.161:0/1528235661 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc8c4020020 con 0x7fc8d00ab6e0 2024-03-28T23:34:24.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.511+0000 7fc8d639a640 1 -- 172.21.15.161:0/1528235661 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc8d00c0ff0 con 0x7fc8d00ab6e0 2024-03-28T23:34:24.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.512+0000 7fc8ccff9640 1 -- 172.21.15.161:0/1528235661 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) ==== 74+0+13473 (secure 0 0 0) 0x7fc8c405b5b0 con 0x7fc8d00ab6e0 2024-03-28T23:34:24.513 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:24.513 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":113,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:34:22.202477+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6834/1096504848":"2024-03-29T23:34:19.193357+0000","172.21.15.161:6835/2544838511":"2024-03-29T23:34:02.034196+0000","172.21.15.170:6832/1096504848":"2024-03-29T23:34:19.193357+0000","172.21.15.161:6834/2544838511":"2024-03-29T23:34:02.034196+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:34:24.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.514+0000 7fc8d639a640 1 -- 172.21.15.161:0/1528235661 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc8bc06b500 msgr2=0x7fc8bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:24.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.514+0000 7fc8d639a640 1 --2- 172.21.15.161:0/1528235661 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc8bc06b500 0x7fc8bc06d9c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0036000 tx=0x7fc8c0008030 comp rx=0 tx=0).stop 2024-03-28T23:34:24.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.514+0000 7fc8d639a640 1 -- 172.21.15.161:0/1528235661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8d00ab6e0 msgr2=0x7fc8d01eb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:24.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.514+0000 7fc8d639a640 1 --2- 172.21.15.161:0/1528235661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8d00ab6e0 0x7fc8d01eb580 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fc8c400e730 tx=0x7fc8c400ebf0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.515 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.514+0000 7fc8d639a640 1 -- 172.21.15.161:0/1528235661 shutdown_connections 2024-03-28T23:34:24.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.514+0000 7fc8d639a640 1 --2- 172.21.15.161:0/1528235661 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc8bc06b500 0x7fc8bc06d9c0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.514+0000 7fc8d639a640 1 --2- 172.21.15.161:0/1528235661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8d00ab6e0 0x7fc8d01eb580 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.514+0000 7fc8d639a640 1 --2- 172.21.15.161:0/1528235661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8d014be70 0x7fc8d0148560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.514+0000 7fc8d639a640 1 --2- 172.21.15.161:0/1528235661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8d00aacd0 0x7fc8d01eb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.514+0000 7fc8d639a640 1 -- 172.21.15.161:0/1528235661 >> 172.21.15.161:0/1528235661 conn(0x7fc8d00c6670 msgr2=0x7fc8d014a130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:24.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.515+0000 7fc8d639a640 1 -- 172.21.15.161:0/1528235661 shutdown_connections 2024-03-28T23:34:24.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.515+0000 7fc8d639a640 1 -- 172.21.15.161:0/1528235661 wait complete. 2024-03-28T23:34:24.526 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:34:24.526 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:34:24.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.630+0000 7fbd080c0640 1 Processor -- start 2024-03-28T23:34:24.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.630+0000 7fbd080c0640 1 -- start start 2024-03-28T23:34:24.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.631+0000 7fbd080c0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbd00151e70 0x7fbd00152250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:24.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.631+0000 7fbd080c0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fbd00152790 0x7fbd0006c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:24.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.631+0000 7fbd080c0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd000c1e40 con 0x7fbd000c6350 2024-03-28T23:34:24.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.631+0000 7fbd080c0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fbd000c2120 con 0x7fbd00151e70 2024-03-28T23:34:24.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.631+0000 7fbd080c0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fbd000c2400 con 0x7fbd00152790 2024-03-28T23:34:24.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.631+0000 7fbd05e5c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbd00151e70 0x7fbd00152250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:24.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.631+0000 7fbd05e5c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbd00151e70 0x7fbd00152250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43364/0 (socket says 172.21.15.161:43364) 2024-03-28T23:34:24.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.631+0000 7fbd05e5c640 1 -- 172.21.15.161:0/4183019514 learned_addr learned my addr 172.21.15.161:0/4183019514 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:24.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.631+0000 7fbd05e5c640 1 -- 172.21.15.161:0/4183019514 >> v2:172.21.15.170:3300/0 conn(0x7fbd00152790 msgr2=0x7fbd0006c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:24.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.631+0000 7fbd05e5c640 1 --2- 172.21.15.161:0/4183019514 >> v2:172.21.15.170:3300/0 conn(0x7fbd00152790 0x7fbd0006c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.631+0000 7fbd05e5c640 1 -- 172.21.15.161:0/4183019514 >> v1:172.21.15.161:6789/0 conn(0x7fbd000c6350 legacy=0x7fbd001553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:24.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.631+0000 7fbd05e5c640 1 -- 172.21.15.161:0/4183019514 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd000c26e0 con 0x7fbd00151e70 2024-03-28T23:34:24.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.631+0000 7fbd04e5a640 1 -- 172.21.15.161:0/4183019514 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 202577468 0 0) 0x7fbd000c1e40 con 0x7fbd000c6350 2024-03-28T23:34:24.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.632+0000 7fbd05e5c640 1 --2- 172.21.15.161:0/4183019514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd00151e70 0x7fbd00152250 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fbd0006fdc0 tx=0x7fbcfc02b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4436f6d7d834ed5c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:24.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.632+0000 7fbd04e5a640 1 -- 172.21.15.161:0/4183019514 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc035020 con 0x7fbd00151e70 2024-03-28T23:34:24.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.632+0000 7fbd04e5a640 1 -- 172.21.15.161:0/4183019514 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcfc031890 con 0x7fbd00151e70 2024-03-28T23:34:24.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.632+0000 7fbd04e5a640 1 -- 172.21.15.161:0/4183019514 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc031c60 con 0x7fbd00151e70 2024-03-28T23:34:24.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.632+0000 7fbd080c0640 1 -- 172.21.15.161:0/4183019514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd00151e70 msgr2=0x7fbd00152250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:24.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.632+0000 7fbd080c0640 1 --2- 172.21.15.161:0/4183019514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd00151e70 0x7fbd00152250 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fbd0006fdc0 tx=0x7fbcfc02b930 comp rx=0 tx=0).stop 2024-03-28T23:34:24.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.632+0000 7fbd080c0640 1 -- 172.21.15.161:0/4183019514 shutdown_connections 2024-03-28T23:34:24.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.632+0000 7fbd080c0640 1 --2- 172.21.15.161:0/4183019514 >> v2:172.21.15.170:3300/0 conn(0x7fbd00152790 0x7fbd0006c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.632+0000 7fbd080c0640 1 --2- 172.21.15.161:0/4183019514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd00151e70 0x7fbd00152250 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.632+0000 7fbd080c0640 1 -- 172.21.15.161:0/4183019514 >> 172.21.15.161:0/4183019514 conn(0x7fbd000c3390 msgr2=0x7fbd000c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:24.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.633+0000 7fbd080c0640 1 -- 172.21.15.161:0/4183019514 shutdown_connections 2024-03-28T23:34:24.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.633+0000 7fbd080c0640 1 -- 172.21.15.161:0/4183019514 wait complete. 2024-03-28T23:34:24.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.634+0000 7fbd080c0640 1 Processor -- start 2024-03-28T23:34:24.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.634+0000 7fbd080c0640 1 -- start start 2024-03-28T23:34:24.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.634+0000 7fbd080c0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd000c6350 0x7fbd000ac160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:24.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.634+0000 7fbd080c0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd00151e70 0x7fbd001eb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:24.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.634+0000 7fbd080c0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd00143000 0x7fbd001eb800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:24.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.634+0000 7fbd080c0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbd000c2d80 con 0x7fbd000c6350 2024-03-28T23:34:24.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.634+0000 7fbd080c0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbd000c2c00 con 0x7fbd00143000 2024-03-28T23:34:24.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.634+0000 7fbd080c0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbd000a5fb0 con 0x7fbd00151e70 2024-03-28T23:34:24.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.634+0000 7fbd05e5c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd000c6350 0x7fbd000ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:24.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.634+0000 7fbd05e5c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd000c6350 0x7fbd000ac160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43378/0 (socket says 172.21.15.161:43378) 2024-03-28T23:34:24.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.634+0000 7fbd05e5c640 1 -- 172.21.15.161:0/2818406428 learned_addr learned my addr 172.21.15.161:0/2818406428 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:24.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.635+0000 7fbd0665d640 1 --2- 172.21.15.161:0/2818406428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd00143000 0x7fbd001eb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:24.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.635+0000 7fbd0565b640 1 --2- 172.21.15.161:0/2818406428 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd00151e70 0x7fbd001eb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:24.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.635+0000 7fbd05e5c640 1 -- 172.21.15.161:0/2818406428 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd00151e70 msgr2=0x7fbd001eb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:24.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.635+0000 7fbd05e5c640 1 --2- 172.21.15.161:0/2818406428 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd00151e70 0x7fbd001eb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.635+0000 7fbd05e5c640 1 -- 172.21.15.161:0/2818406428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd00143000 msgr2=0x7fbd001eb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:24.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.635+0000 7fbd05e5c640 1 --2- 172.21.15.161:0/2818406428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd00143000 0x7fbd001eb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.635+0000 7fbd05e5c640 1 -- 172.21.15.161:0/2818406428 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd0014c690 con 0x7fbd000c6350 2024-03-28T23:34:24.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.635+0000 7fbd0565b640 1 --2- 172.21.15.161:0/2818406428 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd00151e70 0x7fbd001eb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:24.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.635+0000 7fbd05e5c640 1 --2- 172.21.15.161:0/2818406428 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd000c6350 0x7fbd000ac160 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc000c00 tx=0x7fbcfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:24.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.635+0000 7fbcf67fc640 1 -- 172.21.15.161:0/2818406428 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc03ad60 con 0x7fbd000c6350 2024-03-28T23:34:24.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.635+0000 7fbcf67fc640 1 -- 172.21.15.161:0/2818406428 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcfc0443f0 con 0x7fbd000c6350 2024-03-28T23:34:24.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.635+0000 7fbd080c0640 1 -- 172.21.15.161:0/2818406428 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd00071090 con 0x7fbd000c6350 2024-03-28T23:34:24.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.635+0000 7fbcf67fc640 1 -- 172.21.15.161:0/2818406428 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc0447c0 con 0x7fbd000c6350 2024-03-28T23:34:24.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.635+0000 7fbd080c0640 1 -- 172.21.15.161:0/2818406428 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd000ac6a0 con 0x7fbd000c6350 2024-03-28T23:34:24.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.636+0000 7fbd080c0640 1 -- 172.21.15.161:0/2818406428 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd0004f4b0 con 0x7fbd000c6350 2024-03-28T23:34:24.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.636+0000 7fbcf67fc640 1 -- 172.21.15.161:0/2818406428 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbcfc04b060 con 0x7fbd000c6350 2024-03-28T23:34:24.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.637+0000 7fbcf67fc640 1 --2- 172.21.15.161:0/2818406428 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbce406b500 0x7fbce406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:24.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.637+0000 7fbcf67fc640 1 -- 172.21.15.161:0/2818406428 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5421+0+0 (secure 0 0 0) 0x7fbcfc0ac300 con 0x7fbd000c6350 2024-03-28T23:34:24.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.637+0000 7fbd0565b640 1 --2- 172.21.15.161:0/2818406428 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbce406b500 0x7fbce406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:24.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.637+0000 7fbd0565b640 1 --2- 172.21.15.161:0/2818406428 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbce406b500 0x7fbce406d9c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fbd0014c0b0 tx=0x7fbcf0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:24.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.641+0000 7fbcf67fc640 1 -- 172.21.15.161:0/2818406428 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbcfc03f070 con 0x7fbd000c6350 2024-03-28T23:34:24.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.816+0000 7fbd080c0640 1 -- 172.21.15.161:0/2818406428 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbd000c2f80 con 0x7fbd000c6350 2024-03-28T23:34:24.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.817+0000 7fbcf67fc640 1 -- 172.21.15.161:0/2818406428 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v113) ==== 93+0+10 (secure 0 0 0) 0x7fbcfc02d040 con 0x7fbd000c6350 2024-03-28T23:34:24.817 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:34:24.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.819+0000 7fbd080c0640 1 -- 172.21.15.161:0/2818406428 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbce406b500 msgr2=0x7fbce406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:24.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.819+0000 7fbd080c0640 1 --2- 172.21.15.161:0/2818406428 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbce406b500 0x7fbce406d9c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fbd0014c0b0 tx=0x7fbcf0008040 comp rx=0 tx=0).stop 2024-03-28T23:34:24.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.819+0000 7fbd080c0640 1 -- 172.21.15.161:0/2818406428 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd000c6350 msgr2=0x7fbd000ac160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:24.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.819+0000 7fbd080c0640 1 --2- 172.21.15.161:0/2818406428 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd000c6350 0x7fbd000ac160 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc000c00 tx=0x7fbcfc0046c0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.819+0000 7fbd080c0640 1 -- 172.21.15.161:0/2818406428 shutdown_connections 2024-03-28T23:34:24.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.819+0000 7fbd080c0640 1 --2- 172.21.15.161:0/2818406428 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbce406b500 0x7fbce406d9c0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.819+0000 7fbd080c0640 1 --2- 172.21.15.161:0/2818406428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd00143000 0x7fbd001eb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.819+0000 7fbd080c0640 1 --2- 172.21.15.161:0/2818406428 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd00151e70 0x7fbd001eb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.819+0000 7fbd080c0640 1 --2- 172.21.15.161:0/2818406428 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd000c6350 0x7fbd000ac160 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.819+0000 7fbd080c0640 1 -- 172.21.15.161:0/2818406428 >> 172.21.15.161:0/2818406428 conn(0x7fbd000c3390 msgr2=0x7fbd000a7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:24.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.819+0000 7fbd080c0640 1 -- 172.21.15.161:0/2818406428 shutdown_connections 2024-03-28T23:34:24.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.819+0000 7fbd080c0640 1 -- 172.21.15.161:0/2818406428 wait complete. 2024-03-28T23:34:24.830 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady' 2024-03-28T23:34:24.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.930+0000 7fdffd62a640 1 Processor -- start 2024-03-28T23:34:24.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.930+0000 7fdffd62a640 1 -- start start 2024-03-28T23:34:24.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.931+0000 7fdffd62a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fdff8150a70 0x7fdff8072fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:24.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.931+0000 7fdffd62a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdff8072850 0x7fdff81530c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:24.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.931+0000 7fdffd62a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdff80c5120 con 0x7fdff8150690 2024-03-28T23:34:24.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.931+0000 7fdffd62a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fdff8156630 con 0x7fdff8072850 2024-03-28T23:34:24.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.931+0000 7fdffd62a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fdff80736e0 con 0x7fdff8150a70 2024-03-28T23:34:24.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.931+0000 7fdff77fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdff8072850 0x7fdff81530c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:24.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.931+0000 7fdff6ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fdff8150690 0x7fdff80c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:58360/0 (socket says 172.21.15.161:58360) 2024-03-28T23:34:24.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.931+0000 7fdff6ffd640 1 -- 172.21.15.161:0/459115228 learned_addr learned my addr 172.21.15.161:0/459115228 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:24.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.931+0000 7fdff67fc640 1 --2- 172.21.15.161:0/459115228 >> v2:172.21.15.170:3300/0 conn(0x7fdff8150a70 0x7fdff8072fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:24.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.932+0000 7fdff77fe640 1 -- 172.21.15.161:0/459115228 >> v2:172.21.15.170:3300/0 conn(0x7fdff8150a70 msgr2=0x7fdff8072fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:24.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.932+0000 7fdff77fe640 1 --2- 172.21.15.161:0/459115228 >> v2:172.21.15.170:3300/0 conn(0x7fdff8150a70 0x7fdff8072fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.932+0000 7fdff77fe640 1 -- 172.21.15.161:0/459115228 >> v1:172.21.15.161:6789/0 conn(0x7fdff8150690 legacy=0x7fdff80c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:24.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.932+0000 7fdff77fe640 1 -- 172.21.15.161:0/459115228 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdff81568f0 con 0x7fdff8072850 2024-03-28T23:34:24.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.932+0000 7fdff5ffb640 1 -- 172.21.15.161:0/459115228 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3364347845 0 0) 0x7fdff80c5120 con 0x7fdff8150690 2024-03-28T23:34:24.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.932+0000 7fdff77fe640 1 --2- 172.21.15.161:0/459115228 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdff8072850 0x7fdff81530c0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fdff80c3180 tx=0x7fdff002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aa9ee81aa2bd495 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:24.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.932+0000 7fdff5ffb640 1 -- 172.21.15.161:0/459115228 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff00315a0 con 0x7fdff8072850 2024-03-28T23:34:24.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.932+0000 7fdff5ffb640 1 -- 172.21.15.161:0/459115228 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdff0037070 con 0x7fdff8072850 2024-03-28T23:34:24.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.932+0000 7fdff5ffb640 1 -- 172.21.15.161:0/459115228 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff0031960 con 0x7fdff8072850 2024-03-28T23:34:24.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.933+0000 7fdffd62a640 1 -- 172.21.15.161:0/459115228 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdff8072850 msgr2=0x7fdff81530c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:24.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.933+0000 7fdffd62a640 1 --2- 172.21.15.161:0/459115228 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdff8072850 0x7fdff81530c0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fdff80c3180 tx=0x7fdff002b840 comp rx=0 tx=0).stop 2024-03-28T23:34:24.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.933+0000 7fdffd62a640 1 -- 172.21.15.161:0/459115228 shutdown_connections 2024-03-28T23:34:24.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.933+0000 7fdffd62a640 1 --2- 172.21.15.161:0/459115228 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdff8072850 0x7fdff81530c0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.933+0000 7fdffd62a640 1 --2- 172.21.15.161:0/459115228 >> v2:172.21.15.170:3300/0 conn(0x7fdff8150a70 0x7fdff8072fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.933+0000 7fdffd62a640 1 -- 172.21.15.161:0/459115228 >> 172.21.15.161:0/459115228 conn(0x7fdff80c88b0 msgr2=0x7fdff80c8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:24.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.933+0000 7fdffd62a640 1 -- 172.21.15.161:0/459115228 shutdown_connections 2024-03-28T23:34:24.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.933+0000 7fdffd62a640 1 -- 172.21.15.161:0/459115228 wait complete. 2024-03-28T23:34:24.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.934+0000 7fdffd62a640 1 Processor -- start 2024-03-28T23:34:24.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.934+0000 7fdffd62a640 1 -- start start 2024-03-28T23:34:24.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.935+0000 7fdffd62a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdff8072850 0x7fdff80abdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:24.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.935+0000 7fdffd62a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdff8150690 0x7fdff80c1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:24.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.935+0000 7fdffd62a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdff8150a70 0x7fdff80ada80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:24.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.935+0000 7fdffd62a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdff80c5a50 con 0x7fdff8072850 2024-03-28T23:34:24.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.935+0000 7fdffd62a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdff81573d0 con 0x7fdff8150690 2024-03-28T23:34:24.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.935+0000 7fdffd62a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdff80c5bd0 con 0x7fdff8150a70 2024-03-28T23:34:24.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.935+0000 7fdff6ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdff8072850 0x7fdff80abdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:24.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.935+0000 7fdff6ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdff8072850 0x7fdff80abdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43404/0 (socket says 172.21.15.161:43404) 2024-03-28T23:34:24.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.935+0000 7fdff6ffd640 1 -- 172.21.15.161:0/523887673 learned_addr learned my addr 172.21.15.161:0/523887673 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:24.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.935+0000 7fdff6ffd640 1 -- 172.21.15.161:0/523887673 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdff8150a70 msgr2=0x7fdff80ada80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:24.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.935+0000 7fdff6ffd640 1 --2- 172.21.15.161:0/523887673 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdff8150a70 0x7fdff80ada80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.935+0000 7fdff6ffd640 1 -- 172.21.15.161:0/523887673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdff8150690 msgr2=0x7fdff80c1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:24.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.935+0000 7fdff6ffd640 1 --2- 172.21.15.161:0/523887673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdff8150690 0x7fdff80c1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:24.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.935+0000 7fdff6ffd640 1 -- 172.21.15.161:0/523887673 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdff81eb4e0 con 0x7fdff8072850 2024-03-28T23:34:24.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.936+0000 7fdff6ffd640 1 --2- 172.21.15.161:0/523887673 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdff8072850 0x7fdff80abdf0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fdfe800c7f0 tx=0x7fdfe800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:24.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.936+0000 7fdfdf7fe640 1 -- 172.21.15.161:0/523887673 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe8018070 con 0x7fdff8072850 2024-03-28T23:34:24.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.936+0000 7fdfdf7fe640 1 -- 172.21.15.161:0/523887673 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfe800a640 con 0x7fdff8072850 2024-03-28T23:34:24.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.936+0000 7fdffd62a640 1 -- 172.21.15.161:0/523887673 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdff81eb780 con 0x7fdff8072850 2024-03-28T23:34:24.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.936+0000 7fdfdf7fe640 1 -- 172.21.15.161:0/523887673 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe800aa30 con 0x7fdff8072850 2024-03-28T23:34:24.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.936+0000 7fdffd62a640 1 -- 172.21.15.161:0/523887673 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdff80a9550 con 0x7fdff8072850 2024-03-28T23:34:24.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.937+0000 7fdffd62a640 1 -- 172.21.15.161:0/523887673 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdff804f4b0 con 0x7fdff8072850 2024-03-28T23:34:24.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.937+0000 7fdfdf7fe640 1 -- 172.21.15.161:0/523887673 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdfe801b070 con 0x7fdff8072850 2024-03-28T23:34:24.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.937+0000 7fdfdf7fe640 1 --2- 172.21.15.161:0/523887673 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdfe406b4d0 0x7fdfe406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:24.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.938+0000 7fdfdf7fe640 1 -- 172.21.15.161:0/523887673 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5421+0+0 (secure 0 0 0) 0x7fdfe8089640 con 0x7fdff8072850 2024-03-28T23:34:24.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.938+0000 7fdff67fc640 1 --2- 172.21.15.161:0/523887673 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdfe406b4d0 0x7fdfe406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:24.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.938+0000 7fdff67fc640 1 --2- 172.21.15.161:0/523887673 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdfe406b4d0 0x7fdfe406d990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fdfec006a90 tx=0x7fdfec006a20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:24.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:24.942+0000 7fdfdf7fe640 1 -- 172.21.15.161:0/523887673 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdfe8058670 con 0x7fdff8072850 2024-03-28T23:34:25.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.178+0000 7fdffd62a640 1 -- 172.21.15.161:0/523887673 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady"]} v 0) -- 0x7fdff8150380 con 0x7fdff8072850 2024-03-28T23:34:25.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.212+0000 7fdfdf7fe640 1 -- 172.21.15.161:0/523887673 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady"]}]=0 v274) ==== 148+0+0 (secure 0 0 0) 0x7fdfe8057f70 con 0x7fdff8072850 2024-03-28T23:34:25.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.214+0000 7fdffd62a640 1 -- 172.21.15.161:0/523887673 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdfe406b4d0 msgr2=0x7fdfe406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:25.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.214+0000 7fdffd62a640 1 --2- 172.21.15.161:0/523887673 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdfe406b4d0 0x7fdfe406d990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fdfec006a90 tx=0x7fdfec006a20 comp rx=0 tx=0).stop 2024-03-28T23:34:25.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.214+0000 7fdffd62a640 1 -- 172.21.15.161:0/523887673 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdff8072850 msgr2=0x7fdff80abdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:25.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.214+0000 7fdffd62a640 1 --2- 172.21.15.161:0/523887673 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdff8072850 0x7fdff80abdf0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fdfe800c7f0 tx=0x7fdfe800ccb0 comp rx=0 tx=0).stop 2024-03-28T23:34:25.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.214+0000 7fdffd62a640 1 -- 172.21.15.161:0/523887673 shutdown_connections 2024-03-28T23:34:25.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.214+0000 7fdffd62a640 1 --2- 172.21.15.161:0/523887673 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdfe406b4d0 0x7fdfe406d990 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:25.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.214+0000 7fdffd62a640 1 --2- 172.21.15.161:0/523887673 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdff8150a70 0x7fdff80ada80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:25.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.214+0000 7fdffd62a640 1 --2- 172.21.15.161:0/523887673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdff8150690 0x7fdff80c1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:25.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.214+0000 7fdffd62a640 1 --2- 172.21.15.161:0/523887673 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdff8072850 0x7fdff80abdf0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:25.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.214+0000 7fdffd62a640 1 -- 172.21.15.161:0/523887673 >> 172.21.15.161:0/523887673 conn(0x7fdff80c88b0 msgr2=0x7fdff8154860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:25.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.214+0000 7fdffd62a640 1 -- 172.21.15.161:0/523887673 shutdown_connections 2024-03-28T23:34:25.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.214+0000 7fdffd62a640 1 -- 172.21.15.161:0/523887673 wait complete. 2024-03-28T23:34:25.225 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-28T23:34:25.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.329+0000 7f3934beb640 1 Processor -- start 2024-03-28T23:34:25.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.329+0000 7f3934beb640 1 -- start start 2024-03-28T23:34:25.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.330+0000 7f3934beb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39300c0330 0x7f39300c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.330+0000 7f3934beb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39300c6f60 con 0x7f3930152bc0 2024-03-28T23:34:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.330+0000 7f3934beb640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39300c6120 con 0x7f39301589b0 2024-03-28T23:34:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.330+0000 7f3934beb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f39300c6400 con 0x7f39300c0330 2024-03-28T23:34:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.330+0000 7f392e59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f3930152bc0 0x7f39300c8a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:58362/0 (socket says 172.21.15.161:58362) 2024-03-28T23:34:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.330+0000 7f392e59c640 1 -- 172.21.15.161:0/342486852 learned_addr learned my addr 172.21.15.161:0/342486852 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.330+0000 7f392dd9b640 1 --2- 172.21.15.161:0/342486852 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39300c0330 0x7f39300c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.331+0000 7f392d59a640 1 -- 172.21.15.161:0/342486852 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2742369432 0 0) 0x7f39300c6f60 con 0x7f3930152bc0 2024-03-28T23:34:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.331+0000 7f392d59a640 1 -- 172.21.15.161:0/342486852 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3918003520 con 0x7f3930152bc0 2024-03-28T23:34:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.331+0000 7f392d59a640 1 -- 172.21.15.161:0/342486852 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1946402242 0 0) 0x7f3918003520 con 0x7f3930152bc0 2024-03-28T23:34:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.331+0000 7f392d59a640 1 -- 172.21.15.161:0/342486852 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39300c0330 msgr2=0x7f39300c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:25.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.331+0000 7f392d59a640 1 --2- 172.21.15.161:0/342486852 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39300c0330 0x7f39300c9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:25.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.331+0000 7f392d59a640 1 -- 172.21.15.161:0/342486852 >> v1:172.21.15.170:6789/0 conn(0x7f39301589b0 legacy=0x7f3930158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:25.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.331+0000 7f392d59a640 1 -- 172.21.15.161:0/342486852 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39300c96d0 con 0x7f3930152bc0 2024-03-28T23:34:25.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.331+0000 7f392d59a640 1 -- 172.21.15.161:0/342486852 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f39200019f0 con 0x7f3930152bc0 2024-03-28T23:34:25.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.331+0000 7f392d59a640 1 -- 172.21.15.161:0/342486852 >> v1:172.21.15.161:6789/0 conn(0x7f3930152bc0 legacy=0x7f39300c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:25.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.331+0000 7f392d59a640 1 --2- 172.21.15.161:0/342486852 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39300c0330 0x7f3918003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:25.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.331+0000 7f392d59a640 1 --2- 172.21.15.161:0/342486852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3918004000 0x7f3918004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:25.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.331+0000 7f392d59a640 1 --2- 172.21.15.161:0/342486852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3918004d40 0x7f39180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:25.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.331+0000 7f392d59a640 1 -- 172.21.15.161:0/342486852 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f39300c6400 con 0x7f3918004000 2024-03-28T23:34:25.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.331+0000 7f392d59a640 1 -- 172.21.15.161:0/342486852 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f391800bb00 con 0x7f3918004d40 2024-03-28T23:34:25.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.331+0000 7f392d59a640 1 -- 172.21.15.161:0/342486852 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f391800bd00 con 0x7f39300c0330 2024-03-28T23:34:25.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.331+0000 7f392dd9b640 1 --2- 172.21.15.161:0/342486852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3918004000 0x7f3918004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:25.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.331+0000 7f392e59c640 1 --2- 172.21.15.161:0/342486852 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39300c0330 0x7f3918003ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:25.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.331+0000 7f392ed9d640 1 --2- 172.21.15.161:0/342486852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3918004d40 0x7f39180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:25.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.331+0000 7f392dd9b640 1 -- 172.21.15.161:0/342486852 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39300c0330 msgr2=0x7f3918003ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:25.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.331+0000 7f392dd9b640 1 --2- 172.21.15.161:0/342486852 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39300c0330 0x7f3918003ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:25.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.331+0000 7f392dd9b640 1 -- 172.21.15.161:0/342486852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3918004d40 msgr2=0x7f39180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:25.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.331+0000 7f392dd9b640 1 --2- 172.21.15.161:0/342486852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3918004d40 0x7f39180051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:25.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.331+0000 7f392dd9b640 1 -- 172.21.15.161:0/342486852 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3918003520 con 0x7f3918004000 2024-03-28T23:34:25.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.332+0000 7f392dd9b640 1 --2- 172.21.15.161:0/342486852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3918004000 0x7f3918004800 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f3924009ac0 tx=0x7f3918005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd850c33ddcc7d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:25.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.332+0000 7f392d59a640 1 -- 172.21.15.161:0/342486852 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3924039070 con 0x7f3918004000 2024-03-28T23:34:25.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.332+0000 7f392d59a640 1 -- 172.21.15.161:0/342486852 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f392402c600 con 0x7f3918004000 2024-03-28T23:34:25.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.332+0000 7f392d59a640 1 -- 172.21.15.161:0/342486852 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3924009690 con 0x7f3918004000 2024-03-28T23:34:25.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.333+0000 7f3934beb640 1 -- 172.21.15.161:0/342486852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3918004000 msgr2=0x7f3918004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:25.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.333+0000 7f3934beb640 1 --2- 172.21.15.161:0/342486852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3918004000 0x7f3918004800 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f3924009ac0 tx=0x7f3918005830 comp rx=0 tx=0).stop 2024-03-28T23:34:25.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.333+0000 7f392ed9d640 1 -- 172.21.15.161:0/342486852 reap_dead start 2024-03-28T23:34:25.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.333+0000 7f3934beb640 1 -- 172.21.15.161:0/342486852 shutdown_connections 2024-03-28T23:34:25.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.333+0000 7f3934beb640 1 -- 172.21.15.161:0/342486852 >> 172.21.15.161:0/342486852 conn(0x7f39300caa80 msgr2=0x7f39300cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:25.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.333+0000 7f3934beb640 1 -- 172.21.15.161:0/342486852 shutdown_connections 2024-03-28T23:34:25.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.333+0000 7f3934beb640 1 -- 172.21.15.161:0/342486852 wait complete. 2024-03-28T23:34:25.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.334+0000 7f3934beb640 1 Processor -- start 2024-03-28T23:34:25.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.334+0000 7f3934beb640 1 -- start start 2024-03-28T23:34:25.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.335+0000 7f3934beb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3918004000 0x7f393006bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:25.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.335+0000 7f3934beb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3930149fa0 0x7f3930150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:25.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.335+0000 7f3934beb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f393006c5d0 0x7f3930151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:25.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.335+0000 7f3934beb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3930152fa0 con 0x7f393006c5d0 2024-03-28T23:34:25.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.335+0000 7f3934beb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f39300c6bf0 con 0x7f3918004000 2024-03-28T23:34:25.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.335+0000 7f3934beb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f39300b0310 con 0x7f3930149fa0 2024-03-28T23:34:25.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.335+0000 7f392ed9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f393006c5d0 0x7f3930151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:25.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.335+0000 7f392ed9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f393006c5d0 0x7f3930151280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43410/0 (socket says 172.21.15.161:43410) 2024-03-28T23:34:25.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.335+0000 7f392ed9d640 1 -- 172.21.15.161:0/3525764466 learned_addr learned my addr 172.21.15.161:0/3525764466 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:25.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.335+0000 7f392dd9b640 1 --2- 172.21.15.161:0/3525764466 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3930149fa0 0x7f3930150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:25.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.335+0000 7f392e59c640 1 --2- 172.21.15.161:0/3525764466 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3918004000 0x7f393006bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:25.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.335+0000 7f392ed9d640 1 -- 172.21.15.161:0/3525764466 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3930149fa0 msgr2=0x7f3930150d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:25.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.335+0000 7f392ed9d640 1 --2- 172.21.15.161:0/3525764466 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3930149fa0 0x7f3930150d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:25.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.335+0000 7f392ed9d640 1 -- 172.21.15.161:0/3525764466 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3918004000 msgr2=0x7f393006bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:25.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.335+0000 7f392ed9d640 1 --2- 172.21.15.161:0/3525764466 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3918004000 0x7f393006bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:25.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.335+0000 7f392ed9d640 1 -- 172.21.15.161:0/3525764466 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39300b0840 con 0x7f393006c5d0 2024-03-28T23:34:25.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.335+0000 7f392ed9d640 1 --2- 172.21.15.161:0/3525764466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f393006c5d0 0x7f3930151280 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f393008c4b0 tx=0x7f392800e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:25.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.335+0000 7f391effd640 1 -- 172.21.15.161:0/3525764466 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3928019070 con 0x7f393006c5d0 2024-03-28T23:34:25.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.335+0000 7f391effd640 1 -- 172.21.15.161:0/3525764466 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39280052e0 con 0x7f393006c5d0 2024-03-28T23:34:25.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.336+0000 7f3934beb640 1 -- 172.21.15.161:0/3525764466 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39301eb6f0 con 0x7f393006c5d0 2024-03-28T23:34:25.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.336+0000 7f391effd640 1 -- 172.21.15.161:0/3525764466 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3928003830 con 0x7f393006c5d0 2024-03-28T23:34:25.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.336+0000 7f3934beb640 1 -- 172.21.15.161:0/3525764466 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39300b0560 con 0x7f393006c5d0 2024-03-28T23:34:25.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.337+0000 7f3934beb640 1 -- 172.21.15.161:0/3525764466 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f393004f4b0 con 0x7f393006c5d0 2024-03-28T23:34:25.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.337+0000 7f391effd640 1 -- 172.21.15.161:0/3525764466 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f392801c070 con 0x7f393006c5d0 2024-03-28T23:34:25.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.337+0000 7f391effd640 1 --2- 172.21.15.161:0/3525764466 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f391006b500 0x7f391006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:25.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.337+0000 7f392e59c640 1 --2- 172.21.15.161:0/3525764466 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f391006b500 0x7f391006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:25.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.337+0000 7f391effd640 1 -- 172.21.15.161:0/3525764466 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5421+0+0 (secure 0 0 0) 0x7f39280894b0 con 0x7f393006c5d0 2024-03-28T23:34:25.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.338+0000 7f392e59c640 1 --2- 172.21.15.161:0/3525764466 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f391006b500 0x7f391006d9c0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f393014a4f0 tx=0x7f3920006040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:25.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.342+0000 7f391effd640 1 -- 172.21.15.161:0/3525764466 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3928057e90 con 0x7f393006c5d0 2024-03-28T23:34:25.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.514+0000 7f3934beb640 1 -- 172.21.15.161:0/3525764466 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f39300b0a50 con 0x7f393006c5d0 2024-03-28T23:34:25.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.516+0000 7f391effd640 1 -- 172.21.15.161:0/3525764466 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v41)=0 v41) ==== 122+0+0 (secure 0 0 0) 0x7f392805bf30 con 0x7f393006c5d0 2024-03-28T23:34:25.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.519+0000 7f3934beb640 1 -- 172.21.15.161:0/3525764466 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f391006b500 msgr2=0x7f391006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:25.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.519+0000 7f3934beb640 1 --2- 172.21.15.161:0/3525764466 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f391006b500 0x7f391006d9c0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f393014a4f0 tx=0x7f3920006040 comp rx=0 tx=0).stop 2024-03-28T23:34:25.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.519+0000 7f3934beb640 1 -- 172.21.15.161:0/3525764466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f393006c5d0 msgr2=0x7f3930151280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:25.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.519+0000 7f3934beb640 1 --2- 172.21.15.161:0/3525764466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f393006c5d0 0x7f3930151280 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f393008c4b0 tx=0x7f392800e8a0 comp rx=0 tx=0).stop 2024-03-28T23:34:25.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.519+0000 7f3934beb640 1 -- 172.21.15.161:0/3525764466 shutdown_connections 2024-03-28T23:34:25.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.519+0000 7f3934beb640 1 --2- 172.21.15.161:0/3525764466 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f391006b500 0x7f391006d9c0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:25.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.519+0000 7f3934beb640 1 --2- 172.21.15.161:0/3525764466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f393006c5d0 0x7f3930151280 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:25.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.519+0000 7f3934beb640 1 --2- 172.21.15.161:0/3525764466 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3930149fa0 0x7f3930150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:25.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.519+0000 7f3934beb640 1 --2- 172.21.15.161:0/3525764466 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3918004000 0x7f393006bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:25.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.519+0000 7f3934beb640 1 -- 172.21.15.161:0/3525764466 >> 172.21.15.161:0/3525764466 conn(0x7f39300caa80 msgr2=0x7f39300c9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:25.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.519+0000 7f3934beb640 1 -- 172.21.15.161:0/3525764466 shutdown_connections 2024-03-28T23:34:25.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.519+0000 7f3934beb640 1 -- 172.21.15.161:0/3525764466 wait complete. 2024-03-28T23:34:25.530 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-28T23:34:25.530 DEBUG:teuthology.orchestra.run.smithi161:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:34:25.570 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:34:25.620 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-28T23:34:25.621 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:34:25.621 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:34:25.636 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20793 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-28T23:34:25.636 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:34:25.636 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:34:25.653 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056374,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19690 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-28T23:34:25.653 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-28T23:34:25.653 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:34:25.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.777+0000 7f935689d640 1 Processor -- start 2024-03-28T23:34:25.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.777+0000 7f935689d640 1 -- start start 2024-03-28T23:34:25.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.777+0000 7f935689d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f935014be70 0x7f93500ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:25.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.777+0000 7f935689d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93500ab6e0 0x7f93500aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:25.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.777+0000 7f935689d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f935014c250 con 0x7f935006ce80 2024-03-28T23:34:25.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.777+0000 7f935689d640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f93501529c0 con 0x7f935014be70 2024-03-28T23:34:25.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.777+0000 7f935689d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f93500a9dc0 con 0x7f93500ab6e0 2024-03-28T23:34:25.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.778+0000 7f9354e3a640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f935006ce80 0x7f93501553b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:58376/0 (socket says 172.21.15.161:58376) 2024-03-28T23:34:25.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.778+0000 7f9354e3a640 1 -- 172.21.15.161:0/3531815065 learned_addr learned my addr 172.21.15.161:0/3531815065 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:25.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.778+0000 7f934ffff640 1 --2- 172.21.15.161:0/3531815065 >> v2:172.21.15.170:3300/0 conn(0x7f935014be70 0x7f93500ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:25.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.778+0000 7f934f7fe640 1 --2- 172.21.15.161:0/3531815065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93500ab6e0 0x7f93500aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:25.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.778+0000 7f934effd640 1 -- 172.21.15.161:0/3531815065 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 116918828 0 0) 0x7f935014c250 con 0x7f935006ce80 2024-03-28T23:34:25.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.778+0000 7f934effd640 1 -- 172.21.15.161:0/3531815065 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9338003520 con 0x7f935006ce80 2024-03-28T23:34:25.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.778+0000 7f934effd640 1 -- 172.21.15.161:0/3531815065 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 108744866 0 0) 0x7f9338003520 con 0x7f935006ce80 2024-03-28T23:34:25.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.778+0000 7f934effd640 1 -- 172.21.15.161:0/3531815065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93500ab6e0 msgr2=0x7f93500aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:25.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.778+0000 7f934effd640 1 --2- 172.21.15.161:0/3531815065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93500ab6e0 0x7f93500aa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:25.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.778+0000 7f934effd640 1 -- 172.21.15.161:0/3531815065 >> v2:172.21.15.170:3300/0 conn(0x7f935014be70 msgr2=0x7f93500ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:25.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.778+0000 7f934effd640 1 --2- 172.21.15.161:0/3531815065 >> v2:172.21.15.170:3300/0 conn(0x7f935014be70 0x7f93500ab1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:25.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.778+0000 7f934effd640 1 -- 172.21.15.161:0/3531815065 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93500aa0a0 con 0x7f935006ce80 2024-03-28T23:34:25.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.778+0000 7f934effd640 1 -- 172.21.15.161:0/3531815065 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f93480019f0 con 0x7f935006ce80 2024-03-28T23:34:25.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.778+0000 7f934ffff640 1 --2- 172.21.15.161:0/3531815065 >> v2:172.21.15.170:3300/0 conn(0x7f935014be70 0x7f93500ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:34:25.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.778+0000 7f934effd640 1 -- 172.21.15.161:0/3531815065 >> v1:172.21.15.161:6789/0 conn(0x7f935006ce80 legacy=0x7f93501553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:25.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.778+0000 7f934effd640 1 --2- 172.21.15.161:0/3531815065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93500ab6e0 0x7f9338003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:25.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.778+0000 7f934effd640 1 --2- 172.21.15.161:0/3531815065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9338004010 0x7f9338004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:25.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.778+0000 7f934effd640 1 --2- 172.21.15.161:0/3531815065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9338004d40 0x7f93380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:25.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.778+0000 7f934effd640 1 -- 172.21.15.161:0/3531815065 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f93501529c0 con 0x7f9338004010 2024-03-28T23:34:25.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.778+0000 7f934effd640 1 -- 172.21.15.161:0/3531815065 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f93500a9dc0 con 0x7f9338004d40 2024-03-28T23:34:25.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.778+0000 7f934effd640 1 -- 172.21.15.161:0/3531815065 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f933800baf0 con 0x7f93500ab6e0 2024-03-28T23:34:25.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.778+0000 7f934f7fe640 1 --2- 172.21.15.161:0/3531815065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9338004010 0x7f9338004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:25.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.778+0000 7f934ffff640 1 --2- 172.21.15.161:0/3531815065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93500ab6e0 0x7f9338003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:25.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.778+0000 7f9354e3a640 1 --2- 172.21.15.161:0/3531815065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9338004d40 0x7f93380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:25.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.779+0000 7f934f7fe640 1 -- 172.21.15.161:0/3531815065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93500ab6e0 msgr2=0x7f9338003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:25.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.779+0000 7f934f7fe640 1 --2- 172.21.15.161:0/3531815065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93500ab6e0 0x7f9338003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:25.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.779+0000 7f934f7fe640 1 -- 172.21.15.161:0/3531815065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9338004d40 msgr2=0x7f93380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:25.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.779+0000 7f934f7fe640 1 --2- 172.21.15.161:0/3531815065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9338004d40 0x7f93380051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:25.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.779+0000 7f934f7fe640 1 -- 172.21.15.161:0/3531815065 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9338003520 con 0x7f9338004010 2024-03-28T23:34:25.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.779+0000 7f934f7fe640 1 --2- 172.21.15.161:0/3531815065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9338004010 0x7f9338004800 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f9344009ac0 tx=0x7f9338005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae41d2998dce2cce server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:25.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.780+0000 7f934effd640 1 -- 172.21.15.161:0/3531815065 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9344039070 con 0x7f9338004010 2024-03-28T23:34:25.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.780+0000 7f934effd640 1 -- 172.21.15.161:0/3531815065 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f934402c6b0 con 0x7f9338004010 2024-03-28T23:34:25.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.780+0000 7f934effd640 1 -- 172.21.15.161:0/3531815065 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9344009690 con 0x7f9338004010 2024-03-28T23:34:25.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.780+0000 7f935689d640 1 -- 172.21.15.161:0/3531815065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9338004010 msgr2=0x7f9338004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:25.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.780+0000 7f935689d640 1 --2- 172.21.15.161:0/3531815065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9338004010 0x7f9338004800 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f9344009ac0 tx=0x7f9338005830 comp rx=0 tx=0).stop 2024-03-28T23:34:25.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.780+0000 7f9354e3a640 1 -- 172.21.15.161:0/3531815065 reap_dead start 2024-03-28T23:34:25.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.780+0000 7f935689d640 1 -- 172.21.15.161:0/3531815065 shutdown_connections 2024-03-28T23:34:25.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.780+0000 7f935689d640 1 -- 172.21.15.161:0/3531815065 >> 172.21.15.161:0/3531815065 conn(0x7f93500c6670 msgr2=0x7f93500c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:25.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.780+0000 7f935689d640 1 -- 172.21.15.161:0/3531815065 shutdown_connections 2024-03-28T23:34:25.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.780+0000 7f935689d640 1 -- 172.21.15.161:0/3531815065 wait complete. 2024-03-28T23:34:25.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.781+0000 7f935689d640 1 Processor -- start 2024-03-28T23:34:25.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.781+0000 7f935689d640 1 -- start start 2024-03-28T23:34:25.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.782+0000 7f935689d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9338004010 0x7f93501eb770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:25.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.782+0000 7f935689d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9350153a60 0x7f93500bfc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:25.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.782+0000 7f935689d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93501ee920 0x7f93501eedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:25.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.782+0000 7f935689d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f935006b2d0 con 0x7f9338004010 2024-03-28T23:34:25.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.782+0000 7f935689d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f935006b150 con 0x7f93501ee920 2024-03-28T23:34:25.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.782+0000 7f935689d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f93500c1520 con 0x7f9350153a60 2024-03-28T23:34:25.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.782+0000 7f934ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9338004010 0x7f93501eb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:25.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.782+0000 7f934ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9338004010 0x7f93501eb770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43436/0 (socket says 172.21.15.161:43436) 2024-03-28T23:34:25.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.782+0000 7f934ffff640 1 -- 172.21.15.161:0/3805318584 learned_addr learned my addr 172.21.15.161:0/3805318584 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:25.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.782+0000 7f934f7fe640 1 --2- 172.21.15.161:0/3805318584 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9350153a60 0x7f93500bfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:25.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.782+0000 7f9354e3a640 1 --2- 172.21.15.161:0/3805318584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93501ee920 0x7f93501eedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:25.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.782+0000 7f934ffff640 1 -- 172.21.15.161:0/3805318584 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9350153a60 msgr2=0x7f93500bfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:25.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.782+0000 7f934ffff640 1 --2- 172.21.15.161:0/3805318584 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9350153a60 0x7f93500bfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:25.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.782+0000 7f934ffff640 1 -- 172.21.15.161:0/3805318584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93501ee920 msgr2=0x7f93501eedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:25.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.782+0000 7f934ffff640 1 --2- 172.21.15.161:0/3805318584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93501ee920 0x7f93501eedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:25.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.782+0000 7f934ffff640 1 -- 172.21.15.161:0/3805318584 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9350151990 con 0x7f9338004010 2024-03-28T23:34:25.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.783+0000 7f934ffff640 1 --2- 172.21.15.161:0/3805318584 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9338004010 0x7f93501eb770 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f9340011cd0 tx=0x7f934000f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:25.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.783+0000 7f934cff9640 1 -- 172.21.15.161:0/3805318584 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934000fcc0 con 0x7f9338004010 2024-03-28T23:34:25.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.783+0000 7f934cff9640 1 -- 172.21.15.161:0/3805318584 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f934000fe80 con 0x7f9338004010 2024-03-28T23:34:25.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.783+0000 7f935689d640 1 -- 172.21.15.161:0/3805318584 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9350151c70 con 0x7f9338004010 2024-03-28T23:34:25.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.783+0000 7f934cff9640 1 -- 172.21.15.161:0/3805318584 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9340004260 con 0x7f9338004010 2024-03-28T23:34:25.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.783+0000 7f935689d640 1 -- 172.21.15.161:0/3805318584 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9350152230 con 0x7f9338004010 2024-03-28T23:34:25.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.784+0000 7f935689d640 1 -- 172.21.15.161:0/3805318584 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93500737b0 con 0x7f9338004010 2024-03-28T23:34:25.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.784+0000 7f934cff9640 1 -- 172.21.15.161:0/3805318584 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9340012040 con 0x7f9338004010 2024-03-28T23:34:25.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.784+0000 7f934cff9640 1 --2- 172.21.15.161:0/3805318584 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f933c06b500 0x7f933c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:25.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.785+0000 7f934cff9640 1 -- 172.21.15.161:0/3805318584 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5421+0+0 (secure 0 0 0) 0x7f934008a5f0 con 0x7f9338004010 2024-03-28T23:34:25.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.785+0000 7f934f7fe640 1 --2- 172.21.15.161:0/3805318584 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f933c06b500 0x7f933c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:25.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.785+0000 7f934f7fe640 1 --2- 172.21.15.161:0/3805318584 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f933c06b500 0x7f933c06d9c0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9344009ac0 tx=0x7f9344030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:25.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:25.789+0000 7f934cff9640 1 -- 172.21.15.161:0/3805318584 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f93500737b0 con 0x7f9338004010 2024-03-28T23:34:25.958 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:34:25.960 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:34:26.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:26.000+0000 7f935689d640 1 -- 172.21.15.161:0/3805318584 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f935004f4b0 con 0x7f9338004010 2024-03-28T23:34:26.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:26.000+0000 7f934cff9640 1 -- 172.21.15.161:0/3805318584 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+3885 (secure 0 0 0) 0x7f9340019030 con 0x7f9338004010 2024-03-28T23:34:26.001 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:26.001 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":102,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/1610475969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1610475969},{"type":"v1","addr":"172.21.15.161:6835","nonce":1610475969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":6452,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/3906583125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3906583125},{"type":"v1","addr":"172.21.15.170:6833","nonce":3906583125}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":102},{"gid":6892,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/2684760895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2684760895},{"type":"v1","addr":"172.21.15.161:6837","nonce":2684760895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":7255,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/1904398353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1904398353},{"type":"v1","addr":"172.21.15.170:6835","nonce":1904398353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":102}],"filesystems":[]} 2024-03-28T23:34:26.001 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 102 2024-03-28T23:34:26.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:26.002+0000 7f935689d640 1 -- 172.21.15.161:0/3805318584 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f933c06b500 msgr2=0x7f933c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:26.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:26.002+0000 7f935689d640 1 --2- 172.21.15.161:0/3805318584 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f933c06b500 0x7f933c06d9c0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9344009ac0 tx=0x7f9344030000 comp rx=0 tx=0).stop 2024-03-28T23:34:26.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:26.002+0000 7f935689d640 1 -- 172.21.15.161:0/3805318584 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9338004010 msgr2=0x7f93501eb770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:26.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:26.002+0000 7f935689d640 1 --2- 172.21.15.161:0/3805318584 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9338004010 0x7f93501eb770 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f9340011cd0 tx=0x7f934000f610 comp rx=0 tx=0).stop 2024-03-28T23:34:26.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:26.002+0000 7f935689d640 1 -- 172.21.15.161:0/3805318584 shutdown_connections 2024-03-28T23:34:26.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:26.002+0000 7f935689d640 1 --2- 172.21.15.161:0/3805318584 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f933c06b500 0x7f933c06d9c0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:26.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:26.002+0000 7f935689d640 1 --2- 172.21.15.161:0/3805318584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93501ee920 0x7f93501eedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:26.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:26.002+0000 7f935689d640 1 --2- 172.21.15.161:0/3805318584 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9350153a60 0x7f93500bfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:26.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:26.002+0000 7f935689d640 1 --2- 172.21.15.161:0/3805318584 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9338004010 0x7f93501eb770 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:26.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:26.002+0000 7f935689d640 1 -- 172.21.15.161:0/3805318584 >> 172.21.15.161:0/3805318584 conn(0x7f93500c6670 msgr2=0x7f935006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:26.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:26.002+0000 7f935689d640 1 -- 172.21.15.161:0/3805318584 shutdown_connections 2024-03-28T23:34:26.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:26.002+0000 7f935689d640 1 -- 172.21.15.161:0/3805318584 wait complete. 2024-03-28T23:34:26.014 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T23:34:26.014 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-28T23:34:26.014 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T23:34:26.015 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:34:26.015 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T23:34:26.015 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-28T23:34:26.015 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T23:34:26.015 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:34:26.015 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-28T23:34:26.015 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-28T23:34:26.016 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-28T23:34:26.016 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:34:26.016 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-28T23:34:26.016 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-28T23:34:26.016 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-28T23:34:26.016 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:34:26.026 INFO:tasks.ceph.mds.b:Stopped 2024-03-28T23:34:26.027 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-28T23:34:26.027 INFO:tasks.ceph.mds.d:Stopped 2024-03-28T23:34:26.028 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-28T23:34:26.065 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T23:34:26.065 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T23:34:26.070 INFO:tasks.ceph.mds.b:Started 2024-03-28T23:34:26.070 DEBUG:teuthology.parallel:result is None 2024-03-28T23:34:26.071 INFO:tasks.ceph.mds.d:Started 2024-03-28T23:34:26.071 DEBUG:teuthology.parallel:result is None 2024-03-28T23:34:26.108 INFO:tasks.ceph.mds.c:Started 2024-03-28T23:34:26.108 DEBUG:teuthology.parallel:result is None 2024-03-28T23:34:26.154 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2024-03-28T23:34:26.154 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2024-03-28T23:34:26.183 INFO:tasks.ceph.mds.c.smithi161.stdout:starting mds.c at 2024-03-28T23:34:26.758 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-03-28T23:34:32.065 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T23:34:32.065 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T23:34:32.068 INFO:tasks.ceph.mds.a:Started 2024-03-28T23:34:32.069 DEBUG:teuthology.parallel:result is None 2024-03-28T23:34:32.069 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-28T23:34:32.156 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-28T23:34:32.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.183+0000 7f43b226f640 1 Processor -- start 2024-03-28T23:34:32.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.183+0000 7f43b226f640 1 -- start start 2024-03-28T23:34:32.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.184+0000 7f43b226f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f43ac06fc60 0x7f43ac070110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:32.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.184+0000 7f43b226f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0c3390 0x7f43ac0c3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:32.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.184+0000 7f43b226f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43ac153050 con 0x7f43ac072810 2024-03-28T23:34:32.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.184+0000 7f43b226f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f43ac0a8520 con 0x7f43ac06fc60 2024-03-28T23:34:32.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.184+0000 7f43b226f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f43ac153310 con 0x7f43ac0c3390 2024-03-28T23:34:32.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.184+0000 7f43ab7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f43ac06fc60 0x7f43ac070110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:32.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.184+0000 7f43abfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f43ac072810 0x7f43ac0c4840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50988/0 (socket says 172.21.15.161:50988) 2024-03-28T23:34:32.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.184+0000 7f43abfff640 1 -- 172.21.15.161:0/487166244 learned_addr learned my addr 172.21.15.161:0/487166244 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:32.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.184+0000 7f43b080c640 1 --2- 172.21.15.161:0/487166244 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0c3390 0x7f43ac0c3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:32.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.184+0000 7f43ab7fe640 1 -- 172.21.15.161:0/487166244 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0c3390 msgr2=0x7f43ac0c3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:32.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.184+0000 7f43ab7fe640 1 --2- 172.21.15.161:0/487166244 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0c3390 0x7f43ac0c3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:32.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.184+0000 7f43ab7fe640 1 -- 172.21.15.161:0/487166244 >> v1:172.21.15.161:6789/0 conn(0x7f43ac072810 legacy=0x7f43ac0c4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:32.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.184+0000 7f43ab7fe640 1 -- 172.21.15.161:0/487166244 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43ac1535f0 con 0x7f43ac06fc60 2024-03-28T23:34:32.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.184+0000 7f43aaffd640 1 -- 172.21.15.161:0/487166244 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4254964317 0 0) 0x7f43ac153050 con 0x7f43ac072810 2024-03-28T23:34:32.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.184+0000 7f43b080c640 1 --2- 172.21.15.161:0/487166244 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0c3390 0x7f43ac0c3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:32.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.185+0000 7f43ab7fe640 1 --2- 172.21.15.161:0/487166244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43ac06fc60 0x7f43ac070110 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f43a00097a0 tx=0x7f43a002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d224a7db808c46e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:32.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.185+0000 7f43aaffd640 1 -- 172.21.15.161:0/487166244 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a00315a0 con 0x7f43ac06fc60 2024-03-28T23:34:32.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.185+0000 7f43aaffd640 1 -- 172.21.15.161:0/487166244 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43a0037070 con 0x7f43ac06fc60 2024-03-28T23:34:32.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.185+0000 7f43aaffd640 1 -- 172.21.15.161:0/487166244 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a0031960 con 0x7f43ac06fc60 2024-03-28T23:34:32.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.186+0000 7f43b226f640 1 -- 172.21.15.161:0/487166244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43ac06fc60 msgr2=0x7f43ac070110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:32.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.186+0000 7f43b226f640 1 --2- 172.21.15.161:0/487166244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43ac06fc60 0x7f43ac070110 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f43a00097a0 tx=0x7f43a002b840 comp rx=0 tx=0).stop 2024-03-28T23:34:32.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.186+0000 7f43b226f640 1 -- 172.21.15.161:0/487166244 shutdown_connections 2024-03-28T23:34:32.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.186+0000 7f43b226f640 1 --2- 172.21.15.161:0/487166244 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0c3390 0x7f43ac0c3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:32.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.186+0000 7f43b226f640 1 --2- 172.21.15.161:0/487166244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43ac06fc60 0x7f43ac070110 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:32.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.186+0000 7f43b226f640 1 -- 172.21.15.161:0/487166244 >> 172.21.15.161:0/487166244 conn(0x7f43ac0aa740 msgr2=0x7f43ac0aab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:32.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.186+0000 7f43b226f640 1 -- 172.21.15.161:0/487166244 shutdown_connections 2024-03-28T23:34:32.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.186+0000 7f43b226f640 1 -- 172.21.15.161:0/487166244 wait complete. 2024-03-28T23:34:32.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.187+0000 7f43b226f640 1 Processor -- start 2024-03-28T23:34:32.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.187+0000 7f43b226f640 1 -- start start 2024-03-28T23:34:32.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.188+0000 7f43b226f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43ac06fc60 0x7f43ac14b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:32.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.188+0000 7f43b226f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43ac072810 0x7f43ac148de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:32.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.188+0000 7f43b226f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0c3390 0x7f43ac149320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:32.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.188+0000 7f43b226f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f43ac0c4f50 con 0x7f43ac072810 2024-03-28T23:34:32.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.188+0000 7f43b226f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f43ac153bb0 con 0x7f43ac06fc60 2024-03-28T23:34:32.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.188+0000 7f43b226f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f43ac0ac070 con 0x7f43ac0c3390 2024-03-28T23:34:32.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.188+0000 7f43ab7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43ac072810 0x7f43ac148de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:32.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.188+0000 7f43ab7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43ac072810 0x7f43ac148de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34758/0 (socket says 172.21.15.161:34758) 2024-03-28T23:34:32.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.188+0000 7f43ab7fe640 1 -- 172.21.15.161:0/390175580 learned_addr learned my addr 172.21.15.161:0/390175580 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:32.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.188+0000 7f43abfff640 1 --2- 172.21.15.161:0/390175580 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43ac06fc60 0x7f43ac14b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:32.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.188+0000 7f43b080c640 1 --2- 172.21.15.161:0/390175580 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0c3390 0x7f43ac149320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:32.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.189+0000 7f43ab7fe640 1 -- 172.21.15.161:0/390175580 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0c3390 msgr2=0x7f43ac149320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:32.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.189+0000 7f43ab7fe640 1 --2- 172.21.15.161:0/390175580 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0c3390 0x7f43ac149320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:32.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.189+0000 7f43ab7fe640 1 -- 172.21.15.161:0/390175580 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43ac06fc60 msgr2=0x7f43ac14b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:32.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.189+0000 7f43ab7fe640 1 --2- 172.21.15.161:0/390175580 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43ac06fc60 0x7f43ac14b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:32.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.189+0000 7f43ab7fe640 1 -- 172.21.15.161:0/390175580 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43ac1544e0 con 0x7f43ac072810 2024-03-28T23:34:32.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.189+0000 7f43ab7fe640 1 --2- 172.21.15.161:0/390175580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43ac072810 0x7f43ac148de0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f43a0034040 tx=0x7f43a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:32.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.189+0000 7f43a8ff9640 1 -- 172.21.15.161:0/390175580 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a0032840 con 0x7f43ac072810 2024-03-28T23:34:32.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.189+0000 7f43a8ff9640 1 -- 172.21.15.161:0/390175580 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43a0032a00 con 0x7f43ac072810 2024-03-28T23:34:32.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.189+0000 7f43b226f640 1 -- 172.21.15.161:0/390175580 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43ac0ac310 con 0x7f43ac072810 2024-03-28T23:34:32.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.189+0000 7f43a8ff9640 1 -- 172.21.15.161:0/390175580 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a0032df0 con 0x7f43ac072810 2024-03-28T23:34:32.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.189+0000 7f43b226f640 1 -- 172.21.15.161:0/390175580 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43ac14b940 con 0x7f43ac072810 2024-03-28T23:34:32.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.190+0000 7f43b226f640 1 -- 172.21.15.161:0/390175580 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43ac04f4b0 con 0x7f43ac072810 2024-03-28T23:34:32.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.191+0000 7f43a8ff9640 1 -- 172.21.15.161:0/390175580 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f43a003b3f0 con 0x7f43ac072810 2024-03-28T23:34:32.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.191+0000 7f43a8ff9640 1 --2- 172.21.15.161:0/390175580 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f439806b4d0 0x7f439806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:32.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.191+0000 7f43abfff640 1 --2- 172.21.15.161:0/390175580 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f439806b4d0 0x7f439806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:32.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.191+0000 7f43a8ff9640 1 -- 172.21.15.161:0/390175580 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5421+0+0 (secure 0 0 0) 0x7f43a002f030 con 0x7f43ac072810 2024-03-28T23:34:32.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.192+0000 7f43abfff640 1 --2- 172.21.15.161:0/390175580 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f439806b4d0 0x7f439806d990 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f439c000f60 tx=0x7f439c007b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:32.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.196+0000 7f43a8ff9640 1 -- 172.21.15.161:0/390175580 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f43a0040070 con 0x7f43ac072810 2024-03-28T23:34:32.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:32.379+0000 7f43b226f640 1 -- 172.21.15.161:0/390175580 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f43ac0a9e50 con 0x7f43ac072810 2024-03-28T23:34:33.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.428+0000 7f43a8ff9640 1 -- 172.21.15.161:0/390175580 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v118) ==== 97+0+0 (secure 0 0 0) 0x7f43a007b2b0 con 0x7f43ac072810 2024-03-28T23:34:33.428 INFO:teuthology.orchestra.run.smithi161.stderr: removed all blocklist entries 2024-03-28T23:34:33.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.431+0000 7f43b226f640 1 -- 172.21.15.161:0/390175580 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f439806b4d0 msgr2=0x7f439806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:33.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.431+0000 7f43b226f640 1 --2- 172.21.15.161:0/390175580 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f439806b4d0 0x7f439806d990 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f439c000f60 tx=0x7f439c007b50 comp rx=0 tx=0).stop 2024-03-28T23:34:33.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.431+0000 7f43b226f640 1 -- 172.21.15.161:0/390175580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43ac072810 msgr2=0x7f43ac148de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:33.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.431+0000 7f43b226f640 1 --2- 172.21.15.161:0/390175580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43ac072810 0x7f43ac148de0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f43a0034040 tx=0x7f43a00046c0 comp rx=0 tx=0).stop 2024-03-28T23:34:33.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.431+0000 7f43b226f640 1 -- 172.21.15.161:0/390175580 shutdown_connections 2024-03-28T23:34:33.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.431+0000 7f43b226f640 1 --2- 172.21.15.161:0/390175580 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f439806b4d0 0x7f439806d990 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:33.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.431+0000 7f43b226f640 1 --2- 172.21.15.161:0/390175580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43ac072810 0x7f43ac148de0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:33.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.431+0000 7f43b226f640 1 --2- 172.21.15.161:0/390175580 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0c3390 0x7f43ac149320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:33.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.431+0000 7f43b226f640 1 --2- 172.21.15.161:0/390175580 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43ac06fc60 0x7f43ac14b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:33.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.431+0000 7f43b226f640 1 -- 172.21.15.161:0/390175580 >> 172.21.15.161:0/390175580 conn(0x7f43ac0aa740 msgr2=0x7f43ac0c7b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:33.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.431+0000 7f43b226f640 1 -- 172.21.15.161:0/390175580 shutdown_connections 2024-03-28T23:34:33.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.431+0000 7f43b226f640 1 -- 172.21.15.161:0/390175580 wait complete. 2024-03-28T23:34:33.445 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-28T23:34:33.470 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-03-28T23:34:33.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.551+0000 7fe630aed640 1 Processor -- start 2024-03-28T23:34:33.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.551+0000 7fe630aed640 1 -- start start 2024-03-28T23:34:33.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.552+0000 7fe630aed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe62c0aacb0 0x7fe62c153860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:33.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.552+0000 7fe630aed640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe62c1566f0 con 0x7fe62c150590 2024-03-28T23:34:33.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.552+0000 7fe630aed640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe62c1569d0 con 0x7fe62c0a8fc0 2024-03-28T23:34:33.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.552+0000 7fe630aed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe62c156cb0 con 0x7fe62c0aacb0 2024-03-28T23:34:33.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.552+0000 7fe62a59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe62c150590 0x7fe62c0a9f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50992/0 (socket says 172.21.15.161:50992) 2024-03-28T23:34:33.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.552+0000 7fe62a59c640 1 -- 172.21.15.161:0/3137426899 learned_addr learned my addr 172.21.15.161:0/3137426899 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:33.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.552+0000 7fe62ad9d640 1 --2- 172.21.15.161:0/3137426899 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe62c0aacb0 0x7fe62c153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:33.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.552+0000 7fe62959a640 1 -- 172.21.15.161:0/3137426899 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3295826490 0 0) 0x7fe62c1566f0 con 0x7fe62c150590 2024-03-28T23:34:33.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.552+0000 7fe62959a640 1 -- 172.21.15.161:0/3137426899 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe614003520 con 0x7fe62c150590 2024-03-28T23:34:33.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.553+0000 7fe62959a640 1 -- 172.21.15.161:0/3137426899 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 453364512 0 0) 0x7fe62c1569d0 con 0x7fe62c0a8fc0 2024-03-28T23:34:33.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.553+0000 7fe62959a640 1 -- 172.21.15.161:0/3137426899 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe62c1566f0 con 0x7fe62c0a8fc0 2024-03-28T23:34:33.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.553+0000 7fe62959a640 1 -- 172.21.15.161:0/3137426899 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3336177163 0 0) 0x7fe614003520 con 0x7fe62c150590 2024-03-28T23:34:33.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.553+0000 7fe62ad9d640 1 -- 172.21.15.161:0/3137426899 >> v1:172.21.15.170:6789/0 conn(0x7fe62c0a8fc0 legacy=0x7fe62c153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:33.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.553+0000 7fe62ad9d640 1 -- 172.21.15.161:0/3137426899 >> v1:172.21.15.161:6789/0 conn(0x7fe62c150590 legacy=0x7fe62c0a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:33.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.553+0000 7fe62ad9d640 1 -- 172.21.15.161:0/3137426899 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe62c156f90 con 0x7fe62c0aacb0 2024-03-28T23:34:33.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.553+0000 7fe62ad9d640 1 --2- 172.21.15.161:0/3137426899 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe62c0aacb0 0x7fe62c153860 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe624009dc0 tx=0x7fe62402bcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31ee9e5009fd69a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:33.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.554+0000 7fe62959a640 1 -- 172.21.15.161:0/3137426899 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe624039070 con 0x7fe62c0aacb0 2024-03-28T23:34:33.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.554+0000 7fe62959a640 1 -- 172.21.15.161:0/3137426899 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe62402c7f0 con 0x7fe62c0aacb0 2024-03-28T23:34:33.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.554+0000 7fe62959a640 1 -- 172.21.15.161:0/3137426899 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe61c001690 con 0x7fe62c0aacb0 2024-03-28T23:34:33.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.554+0000 7fe630aed640 1 -- 172.21.15.161:0/3137426899 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe62c0aacb0 msgr2=0x7fe62c153860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:33.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.554+0000 7fe630aed640 1 --2- 172.21.15.161:0/3137426899 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe62c0aacb0 0x7fe62c153860 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe624009dc0 tx=0x7fe62402bcc0 comp rx=0 tx=0).stop 2024-03-28T23:34:33.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.554+0000 7fe630aed640 1 -- 172.21.15.161:0/3137426899 shutdown_connections 2024-03-28T23:34:33.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.554+0000 7fe630aed640 1 --2- 172.21.15.161:0/3137426899 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe62c0aacb0 0x7fe62c153860 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:33.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.554+0000 7fe630aed640 1 -- 172.21.15.161:0/3137426899 >> 172.21.15.161:0/3137426899 conn(0x7fe62c0c6670 msgr2=0x7fe62c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:33.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.554+0000 7fe630aed640 1 -- 172.21.15.161:0/3137426899 shutdown_connections 2024-03-28T23:34:33.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.554+0000 7fe630aed640 1 -- 172.21.15.161:0/3137426899 wait complete. 2024-03-28T23:34:33.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.555+0000 7fe630aed640 1 Processor -- start 2024-03-28T23:34:33.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.556+0000 7fe630aed640 1 -- start start 2024-03-28T23:34:33.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.556+0000 7fe630aed640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe62c0a8fc0 0x7fe62c145a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:33.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.556+0000 7fe630aed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe62c0aacb0 0x7fe62c1ebe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:33.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.556+0000 7fe630aed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe62c150590 0x7fe62c14c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:33.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.556+0000 7fe630aed640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe62c150970 con 0x7fe62c0a8fc0 2024-03-28T23:34:33.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.556+0000 7fe630aed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe62c0aa9c0 con 0x7fe62c0aacb0 2024-03-28T23:34:33.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.556+0000 7fe630aed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe62c0aa650 con 0x7fe62c150590 2024-03-28T23:34:33.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.556+0000 7fe62a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe62c0a8fc0 0x7fe62c145a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:33.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.556+0000 7fe62a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe62c0a8fc0 0x7fe62c145a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34768/0 (socket says 172.21.15.161:34768) 2024-03-28T23:34:33.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.556+0000 7fe62a59c640 1 -- 172.21.15.161:0/2944303864 learned_addr learned my addr 172.21.15.161:0/2944303864 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:33.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.556+0000 7fe62ad9d640 1 --2- 172.21.15.161:0/2944303864 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe62c150590 0x7fe62c14c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:33.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.556+0000 7fe629d9b640 1 --2- 172.21.15.161:0/2944303864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe62c0aacb0 0x7fe62c1ebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:33.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.557+0000 7fe62a59c640 1 -- 172.21.15.161:0/2944303864 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe62c150590 msgr2=0x7fe62c14c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:33.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.557+0000 7fe62a59c640 1 --2- 172.21.15.161:0/2944303864 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe62c150590 0x7fe62c14c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:33.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.557+0000 7fe62a59c640 1 -- 172.21.15.161:0/2944303864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe62c0aacb0 msgr2=0x7fe62c1ebe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:33.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.557+0000 7fe62a59c640 1 --2- 172.21.15.161:0/2944303864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe62c0aacb0 0x7fe62c1ebe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:33.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.557+0000 7fe62a59c640 1 -- 172.21.15.161:0/2944303864 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe62c1ec860 con 0x7fe62c0a8fc0 2024-03-28T23:34:33.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.557+0000 7fe629d9b640 1 --2- 172.21.15.161:0/2944303864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe62c0aacb0 0x7fe62c1ebe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:33.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.557+0000 7fe62a59c640 1 --2- 172.21.15.161:0/2944303864 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe62c0a8fc0 0x7fe62c145a00 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fe61c00c700 tx=0x7fe61c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:33.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.557+0000 7fe61affd640 1 -- 172.21.15.161:0/2944303864 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe61c018070 con 0x7fe62c0a8fc0 2024-03-28T23:34:33.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.557+0000 7fe61affd640 1 -- 172.21.15.161:0/2944303864 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe61c0072d0 con 0x7fe62c0a8fc0 2024-03-28T23:34:33.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.557+0000 7fe630aed640 1 -- 172.21.15.161:0/2944303864 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe62c0c1020 con 0x7fe62c0a8fc0 2024-03-28T23:34:33.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.557+0000 7fe61affd640 1 -- 172.21.15.161:0/2944303864 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe61c0047e0 con 0x7fe62c0a8fc0 2024-03-28T23:34:33.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.557+0000 7fe630aed640 1 -- 172.21.15.161:0/2944303864 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe62c156160 con 0x7fe62c0a8fc0 2024-03-28T23:34:33.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.558+0000 7fe630aed640 1 -- 172.21.15.161:0/2944303864 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe62c06e570 con 0x7fe62c0a8fc0 2024-03-28T23:34:33.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.559+0000 7fe61affd640 1 -- 172.21.15.161:0/2944303864 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe61c01b070 con 0x7fe62c0a8fc0 2024-03-28T23:34:33.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.559+0000 7fe61affd640 1 --2- 172.21.15.161:0/2944303864 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe60c06b4d0 0x7fe60c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:33.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.559+0000 7fe61affd640 1 -- 172.21.15.161:0/2944303864 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5249+0+0 (secure 0 0 0) 0x7fe61c0893c0 con 0x7fe62c0a8fc0 2024-03-28T23:34:33.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.559+0000 7fe629d9b640 1 --2- 172.21.15.161:0/2944303864 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe60c06b4d0 0x7fe60c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:33.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.559+0000 7fe629d9b640 1 --2- 172.21.15.161:0/2944303864 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe60c06b4d0 0x7fe60c06d990 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe62c146250 tx=0x7fe620009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:33.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.563+0000 7fe61affd640 1 -- 172.21.15.161:0/2944303864 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe62c06e570 con 0x7fe62c0a8fc0 2024-03-28T23:34:33.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.797+0000 7fe630aed640 1 -- 172.21.15.161:0/2944303864 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fe62c04f4b0 con 0x7fe62c0a8fc0 2024-03-28T23:34:33.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.798+0000 7fe61affd640 1 -- 172.21.15.161:0/2944303864 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) ==== 80+0+4724 (secure 0 0 0) 0x7fe61c057e40 con 0x7fe62c0a8fc0 2024-03-28T23:34:33.799 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:33.799 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:34:33.799 INFO:teuthology.orchestra.run.smithi161.stdout: "auth_dump": [ 2024-03-28T23:34:33.799 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:34:33.799 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.a", 2024-03-28T23:34:33.799 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmggtoJhAASLXbUICLBhjRr19aUAuZHw==", 2024-03-28T23:34:33.799 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:34:33.799 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:34:33.799 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:34:33.799 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:34:33.799 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:34:33.799 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:34:33.799 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:34:33.800 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:34:33.800 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.b", 2024-03-28T23:34:33.800 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmQbEOMxAA74al6+dMzcN/ngwNZK+aHA==", 2024-03-28T23:34:33.800 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:34:33.800 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:34:33.800 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:34:33.800 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:34:33.800 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:34:33.800 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:34:33.800 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:34:33.800 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:34:33.800 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.c", 2024-03-28T23:34:33.800 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmjf/qLRAAosLbpT8aCWvrKAQnj8Uh3g==", 2024-03-28T23:34:33.800 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:34:33.801 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:34:33.801 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:34:33.801 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:34:33.801 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:34:33.801 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:34:33.801 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:34:33.801 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:34:33.801 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.d", 2024-03-28T23:34:33.801 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmpiZ1OhAAjYrFAaDosDS2Y+BvU4c6JQ==", 2024-03-28T23:34:33.801 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:34:33.801 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:34:33.801 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:34:33.801 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:34:33.801 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:34:33.802 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:34:33.802 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:34:33.802 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:34:33.802 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.0", 2024-03-28T23:34:33.802 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA9/QVmAFPmGBAAS0SkRRwsupm6jm1sqFXJ6g==", 2024-03-28T23:34:33.802 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:34:33.802 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:34:33.802 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:34:33.802 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:34:33.802 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:34:33.802 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:34:33.802 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:34:33.802 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.1", 2024-03-28T23:34:33.803 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBA/QVmNBTVFxAA1htU9ilznfLUDqKmJaQwzg==", 2024-03-28T23:34:33.803 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:34:33.803 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:34:33.803 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:34:33.803 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:34:33.803 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:34:33.803 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:34:33.803 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:34:33.803 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.2", 2024-03-28T23:34:33.803 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBD/QVmg+4YFhAA9TvfFVKT3nQT7/QUOf3Spw==", 2024-03-28T23:34:33.803 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:34:33.803 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:34:33.803 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:34:33.803 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:34:33.804 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:34:33.804 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:34:33.804 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:34:33.804 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.3", 2024-03-28T23:34:33.804 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBG/QVmgiHDFBAAQPtbZT6a1ZGmkisBX0NMgA==", 2024-03-28T23:34:33.804 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:34:33.804 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:34:33.804 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:34:33.804 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:34:33.804 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:34:33.804 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:34:33.804 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:34:33.804 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.4", 2024-03-28T23:34:33.804 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBK/QVmahhZIBAAJfotg/dusbB3usreZlRuPA==", 2024-03-28T23:34:33.804 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:34:33.804 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:34:33.805 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:34:33.805 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:34:33.805 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:34:33.805 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:34:33.805 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:34:33.805 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.5", 2024-03-28T23:34:33.805 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBN/QVmzjlDHhAAli7pLyIo+MVdwSBNJYVg5g==", 2024-03-28T23:34:33.805 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:34:33.805 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:34:33.805 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:34:33.805 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:34:33.805 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:34:33.805 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:34:33.805 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:34:33.806 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.6", 2024-03-28T23:34:33.806 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBQ/QVmtCFSHBAASVfV/5mY/3JNiI0YMT7Vww==", 2024-03-28T23:34:33.806 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:34:33.806 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:34:33.806 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:34:33.806 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:34:33.806 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:34:33.806 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:34:33.806 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:34:33.806 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.7", 2024-03-28T23:34:33.806 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBT/QVm2J7eGhAAaIeZxmLwQBBE1glCxjKbhA==", 2024-03-28T23:34:33.806 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:34:33.806 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:34:33.806 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:34:33.807 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:34:33.807 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:34:33.807 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:34:33.807 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:34:33.807 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.0", 2024-03-28T23:34:33.807 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA6/QVmc1krAxAAEKsn1Tuuv9OlFZpB4PFhDg==", 2024-03-28T23:34:33.807 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:34:33.807 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:34:33.807 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow r", 2024-03-28T23:34:33.807 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-28T23:34:33.807 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:34:33.807 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:34:33.807 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:34:33.807 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.1", 2024-03-28T23:34:33.807 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA6/QVm1B2cBhAAmHDeeFwpFlV/y3NIGWUXHQ==", 2024-03-28T23:34:33.808 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:34:33.808 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:34:33.808 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow r", 2024-03-28T23:34:33.808 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow rw", 2024-03-28T23:34:33.808 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow rwx" 2024-03-28T23:34:33.808 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:34:33.808 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:34:33.808 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:34:33.808 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.admin", 2024-03-28T23:34:33.808 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmzzJ7CRAAYqfhnIkfNJJ3O+VWC2LP+Q==", 2024-03-28T23:34:33.808 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:34:33.808 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-28T23:34:33.808 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:34:33.808 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:34:33.808 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:34:33.809 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:34:33.809 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:34:33.809 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:34:33.809 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mgr.x", 2024-03-28T23:34:33.809 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVm7TD3IhAACByGRGpvJWDpF22NuovHWw==", 2024-03-28T23:34:33.809 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:34:33.809 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-28T23:34:33.809 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile mgr", 2024-03-28T23:34:33.809 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:34:33.809 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:34:33.809 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:34:33.809 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:34:33.809 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mgr.y", 2024-03-28T23:34:33.809 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVm0hLlHhAAh8cewPtZ/TbKrU488kxrBg==", 2024-03-28T23:34:33.810 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:34:33.810 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-28T23:34:33.810 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile mgr", 2024-03-28T23:34:33.810 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:34:33.810 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:34:33.810 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:34:33.810 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-03-28T23:34:33.810 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:34:33.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.800+0000 7fe630aed640 1 -- 172.21.15.161:0/2944303864 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe60c06b4d0 msgr2=0x7fe60c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:33.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.800+0000 7fe630aed640 1 --2- 172.21.15.161:0/2944303864 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe60c06b4d0 0x7fe60c06d990 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe62c146250 tx=0x7fe620009040 comp rx=0 tx=0).stop 2024-03-28T23:34:33.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.800+0000 7fe630aed640 1 -- 172.21.15.161:0/2944303864 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe62c0a8fc0 msgr2=0x7fe62c145a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:33.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.800+0000 7fe630aed640 1 --2- 172.21.15.161:0/2944303864 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe62c0a8fc0 0x7fe62c145a00 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fe61c00c700 tx=0x7fe61c00cbc0 comp rx=0 tx=0).stop 2024-03-28T23:34:33.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.800+0000 7fe630aed640 1 -- 172.21.15.161:0/2944303864 shutdown_connections 2024-03-28T23:34:33.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.800+0000 7fe630aed640 1 --2- 172.21.15.161:0/2944303864 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe60c06b4d0 0x7fe60c06d990 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:33.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.800+0000 7fe630aed640 1 --2- 172.21.15.161:0/2944303864 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe62c150590 0x7fe62c14c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:33.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.800+0000 7fe630aed640 1 --2- 172.21.15.161:0/2944303864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe62c0aacb0 0x7fe62c1ebe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:33.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.800+0000 7fe630aed640 1 --2- 172.21.15.161:0/2944303864 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe62c0a8fc0 0x7fe62c145a00 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:33.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.800+0000 7fe630aed640 1 -- 172.21.15.161:0/2944303864 >> 172.21.15.161:0/2944303864 conn(0x7fe62c0c6670 msgr2=0x7fe62c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:33.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.800+0000 7fe630aed640 1 -- 172.21.15.161:0/2944303864 shutdown_connections 2024-03-28T23:34:33.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.800+0000 7fe630aed640 1 -- 172.21.15.161:0/2944303864 wait complete. 2024-03-28T23:34:33.813 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:34:33.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.913+0000 7fd21c93e640 1 Processor -- start 2024-03-28T23:34:33.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.913+0000 7fd21c93e640 1 -- start start 2024-03-28T23:34:33.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.914+0000 7fd21c93e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd21814be70 0x7fd2180ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:33.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.914+0000 7fd21c93e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd21814c250 con 0x7fd2180aacd0 2024-03-28T23:34:33.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.914+0000 7fd21c93e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2181529c0 con 0x7fd2180ab6e0 2024-03-28T23:34:33.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.914+0000 7fd21c93e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd2180a9dc0 con 0x7fd21814be70 2024-03-28T23:34:33.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.915+0000 7fd21659c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd21814be70 0x7fd2180ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:33.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.915+0000 7fd216d9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd2180aacd0 0x7fd21815c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50994/0 (socket says 172.21.15.161:50994) 2024-03-28T23:34:33.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.915+0000 7fd216d9d640 1 -- 172.21.15.161:0/1830269668 learned_addr learned my addr 172.21.15.161:0/1830269668 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:33.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.915+0000 7fd21659c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd21814be70 0x7fd2180ab1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34776/0 (socket says 172.21.15.161:34776) 2024-03-28T23:34:33.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.915+0000 7fd21659c640 1 -- 172.21.15.161:0/1830269668 >> v1:172.21.15.170:6789/0 conn(0x7fd2180ab6e0 legacy=0x7fd2180aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:33.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.915+0000 7fd21659c640 1 -- 172.21.15.161:0/1830269668 >> v1:172.21.15.161:6789/0 conn(0x7fd2180aacd0 legacy=0x7fd21815c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:33.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.915+0000 7fd21559a640 1 -- 172.21.15.161:0/1830269668 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1235669642 0 0) 0x7fd21814c250 con 0x7fd2180aacd0 2024-03-28T23:34:33.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.915+0000 7fd21659c640 1 -- 172.21.15.161:0/1830269668 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2180aa0a0 con 0x7fd21814be70 2024-03-28T23:34:33.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.916+0000 7fd21659c640 1 --2- 172.21.15.161:0/1830269668 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd21814be70 0x7fd2180ab1a0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fd2080097a0 tx=0x7fd20802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf1af2aaa7fac9b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:33.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.916+0000 7fd21559a640 1 -- 172.21.15.161:0/1830269668 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd208035020 con 0x7fd21814be70 2024-03-28T23:34:33.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.916+0000 7fd21559a640 1 -- 172.21.15.161:0/1830269668 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd208031830 con 0x7fd21814be70 2024-03-28T23:34:33.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.916+0000 7fd21559a640 1 -- 172.21.15.161:0/1830269668 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd208031c00 con 0x7fd21814be70 2024-03-28T23:34:33.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.916+0000 7fd21c93e640 1 -- 172.21.15.161:0/1830269668 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd21814be70 msgr2=0x7fd2180ab1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:33.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.916+0000 7fd21c93e640 1 --2- 172.21.15.161:0/1830269668 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd21814be70 0x7fd2180ab1a0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fd2080097a0 tx=0x7fd20802b8d0 comp rx=0 tx=0).stop 2024-03-28T23:34:33.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.916+0000 7fd21c93e640 1 -- 172.21.15.161:0/1830269668 shutdown_connections 2024-03-28T23:34:33.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.916+0000 7fd21c93e640 1 --2- 172.21.15.161:0/1830269668 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd21814be70 0x7fd2180ab1a0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:33.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.916+0000 7fd21c93e640 1 -- 172.21.15.161:0/1830269668 >> 172.21.15.161:0/1830269668 conn(0x7fd2180c6670 msgr2=0x7fd2180c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:33.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.916+0000 7fd21c93e640 1 -- 172.21.15.161:0/1830269668 shutdown_connections 2024-03-28T23:34:33.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.916+0000 7fd21c93e640 1 -- 172.21.15.161:0/1830269668 wait complete. 2024-03-28T23:34:33.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.918+0000 7fd21c93e640 1 Processor -- start 2024-03-28T23:34:33.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.918+0000 7fd21c93e640 1 -- start start 2024-03-28T23:34:33.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.918+0000 7fd21c93e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2180aacd0 0x7fd218148450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:33.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.918+0000 7fd21c93e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2180ab6e0 0x7fd2181df830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:33.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.918+0000 7fd21c93e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd21814be70 0x7fd2181eb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:33.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.918+0000 7fd21c93e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd21806b2d0 con 0x7fd2180aacd0 2024-03-28T23:34:33.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.918+0000 7fd21c93e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd21806b150 con 0x7fd21814be70 2024-03-28T23:34:33.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.918+0000 7fd21c93e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd2180c1520 con 0x7fd2180ab6e0 2024-03-28T23:34:33.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.918+0000 7fd21659c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2180aacd0 0x7fd218148450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:33.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.918+0000 7fd21659c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2180aacd0 0x7fd218148450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34780/0 (socket says 172.21.15.161:34780) 2024-03-28T23:34:33.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.918+0000 7fd21659c640 1 -- 172.21.15.161:0/891012199 learned_addr learned my addr 172.21.15.161:0/891012199 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:33.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.919+0000 7fd21659c640 1 -- 172.21.15.161:0/891012199 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2180ab6e0 msgr2=0x7fd2181df830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:33.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.919+0000 7fd21659c640 1 --2- 172.21.15.161:0/891012199 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2180ab6e0 0x7fd2181df830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:33.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.919+0000 7fd21659c640 1 -- 172.21.15.161:0/891012199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd21814be70 msgr2=0x7fd2181eb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:33.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.919+0000 7fd21659c640 1 --2- 172.21.15.161:0/891012199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd21814be70 0x7fd2181eb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:33.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.919+0000 7fd21659c640 1 -- 172.21.15.161:0/891012199 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd21814de10 con 0x7fd2180aacd0 2024-03-28T23:34:33.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.919+0000 7fd21659c640 1 --2- 172.21.15.161:0/891012199 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2180aacd0 0x7fd218148450 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fd2080098c0 tx=0x7fd2080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:33.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.919+0000 7fd206ffd640 1 -- 172.21.15.161:0/891012199 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd208041070 con 0x7fd2180aacd0 2024-03-28T23:34:33.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.919+0000 7fd206ffd640 1 -- 172.21.15.161:0/891012199 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd20803aa30 con 0x7fd2180aacd0 2024-03-28T23:34:33.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.919+0000 7fd206ffd640 1 -- 172.21.15.161:0/891012199 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd208035040 con 0x7fd2180aacd0 2024-03-28T23:34:33.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.919+0000 7fd21c93e640 1 -- 172.21.15.161:0/891012199 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2181eadd0 con 0x7fd2180aacd0 2024-03-28T23:34:33.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.919+0000 7fd21c93e640 1 -- 172.21.15.161:0/891012199 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd21814f4c0 con 0x7fd2180aacd0 2024-03-28T23:34:33.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.920+0000 7fd21c93e640 1 -- 172.21.15.161:0/891012199 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd21804f4b0 con 0x7fd2180aacd0 2024-03-28T23:34:33.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.921+0000 7fd206ffd640 1 -- 172.21.15.161:0/891012199 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd20804c060 con 0x7fd2180aacd0 2024-03-28T23:34:33.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.921+0000 7fd206ffd640 1 --2- 172.21.15.161:0/891012199 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd1f806b4d0 0x7fd1f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:33.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.921+0000 7fd206ffd640 1 -- 172.21.15.161:0/891012199 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5249+0+0 (secure 0 0 0) 0x7fd2080abb60 con 0x7fd2180aacd0 2024-03-28T23:34:33.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.921+0000 7fd215d9b640 1 --2- 172.21.15.161:0/891012199 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd1f806b4d0 0x7fd1f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:33.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.922+0000 7fd215d9b640 1 --2- 172.21.15.161:0/891012199 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd1f806b4d0 0x7fd1f806d990 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd20c000f60 tx=0x7fd20c0079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:33.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:33.925+0000 7fd206ffd640 1 -- 172.21.15.161:0/891012199 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd20807aba0 con 0x7fd2180aacd0 2024-03-28T23:34:34.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.100+0000 7fd21c93e640 1 -- 172.21.15.161:0/891012199 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd2180c3d70 con 0x7fd2180aacd0 2024-03-28T23:34:34.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.101+0000 7fd206ffd640 1 -- 172.21.15.161:0/891012199 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) ==== 74+0+13210 (secure 0 0 0) 0x7fd20807a4a0 con 0x7fd2180aacd0 2024-03-28T23:34:34.101 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:34.102 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":118,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:34:33.422846+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:34:34.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.103+0000 7fd21c93e640 1 -- 172.21.15.161:0/891012199 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd1f806b4d0 msgr2=0x7fd1f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:34.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.103+0000 7fd21c93e640 1 --2- 172.21.15.161:0/891012199 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd1f806b4d0 0x7fd1f806d990 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd20c000f60 tx=0x7fd20c0079b0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.103+0000 7fd21c93e640 1 -- 172.21.15.161:0/891012199 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2180aacd0 msgr2=0x7fd218148450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:34.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.103+0000 7fd21c93e640 1 --2- 172.21.15.161:0/891012199 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2180aacd0 0x7fd218148450 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fd2080098c0 tx=0x7fd2080046c0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.103+0000 7fd21c93e640 1 -- 172.21.15.161:0/891012199 shutdown_connections 2024-03-28T23:34:34.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.103+0000 7fd21c93e640 1 --2- 172.21.15.161:0/891012199 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd1f806b4d0 0x7fd1f806d990 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.103+0000 7fd21c93e640 1 --2- 172.21.15.161:0/891012199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd21814be70 0x7fd2181eb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.103+0000 7fd21c93e640 1 --2- 172.21.15.161:0/891012199 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2180ab6e0 0x7fd2181df830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.103+0000 7fd21c93e640 1 --2- 172.21.15.161:0/891012199 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2180aacd0 0x7fd218148450 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.103+0000 7fd21c93e640 1 -- 172.21.15.161:0/891012199 >> 172.21.15.161:0/891012199 conn(0x7fd2180c6670 msgr2=0x7fd2180c1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:34.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.103+0000 7fd21c93e640 1 -- 172.21.15.161:0/891012199 shutdown_connections 2024-03-28T23:34:34.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.103+0000 7fd21c93e640 1 -- 172.21.15.161:0/891012199 wait complete. 2024-03-28T23:34:34.115 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:34:34.115 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:34:34.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.218+0000 7f424b5bc640 1 Processor -- start 2024-03-28T23:34:34.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.218+0000 7f424b5bc640 1 -- start start 2024-03-28T23:34:34.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.219+0000 7f424b5bc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f42440727d0 0x7f4244156480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:34.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.219+0000 7f424b5bc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4244152eb0 0x7f42440c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:34.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.219+0000 7f424b5bc640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4244153830 con 0x7f42440c7920 2024-03-28T23:34:34.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.219+0000 7f424b5bc640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f4244153b10 con 0x7f4244152eb0 2024-03-28T23:34:34.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.219+0000 7f424b5bc640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f4244153df0 con 0x7f42440727d0 2024-03-28T23:34:34.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.219+0000 7f4249b59640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4244152eb0 0x7f42440c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:34.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.219+0000 7f4248b57640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f42440c7920 0x7f4244070100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:51006/0 (socket says 172.21.15.161:51006) 2024-03-28T23:34:34.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.219+0000 7f4248b57640 1 -- 172.21.15.161:0/1677772296 learned_addr learned my addr 172.21.15.161:0/1677772296 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:34.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.220+0000 7f4249358640 1 --2- 172.21.15.161:0/1677772296 >> v2:172.21.15.170:3300/0 conn(0x7f42440727d0 0x7f4244156480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:34.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.220+0000 7f4249b59640 1 -- 172.21.15.161:0/1677772296 >> v2:172.21.15.170:3300/0 conn(0x7f42440727d0 msgr2=0x7f4244156480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:34.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.220+0000 7f4249b59640 1 --2- 172.21.15.161:0/1677772296 >> v2:172.21.15.170:3300/0 conn(0x7f42440727d0 0x7f4244156480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.220+0000 7f4249b59640 1 -- 172.21.15.161:0/1677772296 >> v1:172.21.15.161:6789/0 conn(0x7f42440c7920 legacy=0x7f4244070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:34.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.220+0000 7f4249b59640 1 -- 172.21.15.161:0/1677772296 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f424406f900 con 0x7f4244152eb0 2024-03-28T23:34:34.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.220+0000 7f423bfff640 1 -- 172.21.15.161:0/1677772296 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1227182985 0 0) 0x7f4244153830 con 0x7f42440c7920 2024-03-28T23:34:34.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.220+0000 7f4249b59640 1 --2- 172.21.15.161:0/1677772296 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4244152eb0 0x7f42440c45f0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f4244073570 tx=0x7f423c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18692b26df7bc8f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:34.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.221+0000 7f423bfff640 1 -- 172.21.15.161:0/1677772296 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423c0315a0 con 0x7f4244152eb0 2024-03-28T23:34:34.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.221+0000 7f423bfff640 1 -- 172.21.15.161:0/1677772296 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f423c037070 con 0x7f4244152eb0 2024-03-28T23:34:34.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.221+0000 7f423bfff640 1 -- 172.21.15.161:0/1677772296 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423c031960 con 0x7f4244152eb0 2024-03-28T23:34:34.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.221+0000 7f424b5bc640 1 -- 172.21.15.161:0/1677772296 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4244152eb0 msgr2=0x7f42440c45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:34.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.221+0000 7f424b5bc640 1 --2- 172.21.15.161:0/1677772296 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4244152eb0 0x7f42440c45f0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f4244073570 tx=0x7f423c02b840 comp rx=0 tx=0).stop 2024-03-28T23:34:34.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.221+0000 7f424b5bc640 1 -- 172.21.15.161:0/1677772296 shutdown_connections 2024-03-28T23:34:34.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.221+0000 7f424b5bc640 1 --2- 172.21.15.161:0/1677772296 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4244152eb0 0x7f42440c45f0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.221+0000 7f424b5bc640 1 --2- 172.21.15.161:0/1677772296 >> v2:172.21.15.170:3300/0 conn(0x7f42440727d0 0x7f4244156480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.221+0000 7f424b5bc640 1 -- 172.21.15.161:0/1677772296 >> 172.21.15.161:0/1677772296 conn(0x7f42440aa660 msgr2=0x7f42440aaa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:34.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.221+0000 7f424b5bc640 1 -- 172.21.15.161:0/1677772296 shutdown_connections 2024-03-28T23:34:34.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.221+0000 7f424b5bc640 1 -- 172.21.15.161:0/1677772296 wait complete. 2024-03-28T23:34:34.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.222+0000 7f424b5bc640 1 Processor -- start 2024-03-28T23:34:34.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.222+0000 7f424b5bc640 1 -- start start 2024-03-28T23:34:34.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.223+0000 7f424b5bc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42440727d0 0x7f42440c5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:34.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.223+0000 7f424b5bc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42440c7920 0x7f424414b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:34.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.223+0000 7f424b5bc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4244152eb0 0x7f424414a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:34.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.223+0000 7f424b5bc640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4244072d30 con 0x7f42440727d0 2024-03-28T23:34:34.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.223+0000 7f424b5bc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4244072bb0 con 0x7f4244152eb0 2024-03-28T23:34:34.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.223+0000 7f424b5bc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4244153340 con 0x7f42440c7920 2024-03-28T23:34:34.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.223+0000 7f4249358640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42440727d0 0x7f42440c5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:34.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.223+0000 7f4249358640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42440727d0 0x7f42440c5c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34792/0 (socket says 172.21.15.161:34792) 2024-03-28T23:34:34.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.223+0000 7f4249358640 1 -- 172.21.15.161:0/1525800332 learned_addr learned my addr 172.21.15.161:0/1525800332 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:34.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.223+0000 7f4249b59640 1 --2- 172.21.15.161:0/1525800332 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4244152eb0 0x7f424414a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:34.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.223+0000 7f4248b57640 1 --2- 172.21.15.161:0/1525800332 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42440c7920 0x7f424414b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:34.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.223+0000 7f4249358640 1 -- 172.21.15.161:0/1525800332 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42440c7920 msgr2=0x7f424414b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:34.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.223+0000 7f4249358640 1 --2- 172.21.15.161:0/1525800332 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42440c7920 0x7f424414b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.223+0000 7f4249358640 1 -- 172.21.15.161:0/1525800332 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4244152eb0 msgr2=0x7f424414a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:34.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.223+0000 7f4249358640 1 --2- 172.21.15.161:0/1525800332 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4244152eb0 0x7f424414a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.223+0000 7f4249358640 1 -- 172.21.15.161:0/1525800332 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42441e80c0 con 0x7f42440727d0 2024-03-28T23:34:34.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.224+0000 7f4249358640 1 --2- 172.21.15.161:0/1525800332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42440727d0 0x7f42440c5c50 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f4240007ef0 tx=0x7f424000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:34.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.224+0000 7f4239ffb640 1 -- 172.21.15.161:0/1525800332 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4240014620 con 0x7f42440727d0 2024-03-28T23:34:34.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.224+0000 7f4239ffb640 1 -- 172.21.15.161:0/1525800332 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4240019070 con 0x7f42440727d0 2024-03-28T23:34:34.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.224+0000 7f4239ffb640 1 -- 172.21.15.161:0/1525800332 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4240014a00 con 0x7f42440727d0 2024-03-28T23:34:34.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.224+0000 7f424b5bc640 1 -- 172.21.15.161:0/1525800332 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42440acb50 con 0x7f42440727d0 2024-03-28T23:34:34.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.224+0000 7f424b5bc640 1 -- 172.21.15.161:0/1525800332 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42440c6190 con 0x7f42440727d0 2024-03-28T23:34:34.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.225+0000 7f424b5bc640 1 -- 172.21.15.161:0/1525800332 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f424404f4b0 con 0x7f42440727d0 2024-03-28T23:34:34.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.225+0000 7f4239ffb640 1 -- 172.21.15.161:0/1525800332 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4240014ba0 con 0x7f42440727d0 2024-03-28T23:34:34.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.226+0000 7f4239ffb640 1 --2- 172.21.15.161:0/1525800332 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f422806b4d0 0x7f422806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:34.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.226+0000 7f4239ffb640 1 -- 172.21.15.161:0/1525800332 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5249+0+0 (secure 0 0 0) 0x7f424008a850 con 0x7f42440727d0 2024-03-28T23:34:34.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.226+0000 7f4248b57640 1 --2- 172.21.15.161:0/1525800332 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f422806b4d0 0x7f422806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:34.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.226+0000 7f4248b57640 1 --2- 172.21.15.161:0/1525800332 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f422806b4d0 0x7f422806d990 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f42441dff10 tx=0x7f4234007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:34.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.230+0000 7f4239ffb640 1 -- 172.21.15.161:0/1525800332 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4240059250 con 0x7f42440727d0 2024-03-28T23:34:34.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.403+0000 7f424b5bc640 1 -- 172.21.15.161:0/1525800332 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f42440ae770 con 0x7f42440727d0 2024-03-28T23:34:34.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.404+0000 7f4239ffb640 1 -- 172.21.15.161:0/1525800332 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v118) ==== 93+0+10 (secure 0 0 0) 0x7f424005d2f0 con 0x7f42440727d0 2024-03-28T23:34:34.404 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:34:34.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.405+0000 7f424b5bc640 1 -- 172.21.15.161:0/1525800332 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f422806b4d0 msgr2=0x7f422806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:34.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.405+0000 7f424b5bc640 1 --2- 172.21.15.161:0/1525800332 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f422806b4d0 0x7f422806d990 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f42441dff10 tx=0x7f4234007d50 comp rx=0 tx=0).stop 2024-03-28T23:34:34.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.406+0000 7f424b5bc640 1 -- 172.21.15.161:0/1525800332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42440727d0 msgr2=0x7f42440c5c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:34.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.406+0000 7f424b5bc640 1 --2- 172.21.15.161:0/1525800332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42440727d0 0x7f42440c5c50 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f4240007ef0 tx=0x7f424000e4b0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.406+0000 7f424b5bc640 1 -- 172.21.15.161:0/1525800332 shutdown_connections 2024-03-28T23:34:34.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.406+0000 7f424b5bc640 1 --2- 172.21.15.161:0/1525800332 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f422806b4d0 0x7f422806d990 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.406+0000 7f424b5bc640 1 --2- 172.21.15.161:0/1525800332 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4244152eb0 0x7f424414a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.406+0000 7f424b5bc640 1 --2- 172.21.15.161:0/1525800332 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42440c7920 0x7f424414b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.406+0000 7f424b5bc640 1 --2- 172.21.15.161:0/1525800332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42440727d0 0x7f42440c5c50 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.406+0000 7f424b5bc640 1 -- 172.21.15.161:0/1525800332 >> 172.21.15.161:0/1525800332 conn(0x7f42440aa660 msgr2=0x7f42440c4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:34.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.406+0000 7f424b5bc640 1 -- 172.21.15.161:0/1525800332 shutdown_connections 2024-03-28T23:34:34.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.406+0000 7f424b5bc640 1 -- 172.21.15.161:0/1525800332 wait complete. 2024-03-28T23:34:34.417 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T23:34:34.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.517+0000 7f56b760b640 1 Processor -- start 2024-03-28T23:34:34.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.517+0000 7f56b760b640 1 -- start start 2024-03-28T23:34:34.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.518+0000 7f56b760b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56b014f730 0x7f56b00aad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:34.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.518+0000 7f56b760b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f56b0151f80 0x7f56b00c21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:34.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.518+0000 7f56b760b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56b00ab250 con 0x7f56b00ac310 2024-03-28T23:34:34.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.518+0000 7f56b760b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f56b0152f00 con 0x7f56b0151f80 2024-03-28T23:34:34.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.518+0000 7f56b760b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f56b0155990 con 0x7f56b014f730 2024-03-28T23:34:34.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.518+0000 7f56b5ba8640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f56b0151f80 0x7f56b00c21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:34.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.518+0000 7f56b53a7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56b014f730 0x7f56b00aad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:34.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.518+0000 7f56b4ba6640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f56b00ac310 0x7f56b0155e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:43080/0 (socket says 172.21.15.161:43080) 2024-03-28T23:34:34.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.518+0000 7f56b4ba6640 1 -- 172.21.15.161:0/2140469289 learned_addr learned my addr 172.21.15.161:0/2140469289 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:34.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.518+0000 7f56a7fff640 1 -- 172.21.15.161:0/2140469289 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 382333149 0 0) 0x7f56b00ab250 con 0x7f56b00ac310 2024-03-28T23:34:34.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.518+0000 7f56a7fff640 1 -- 172.21.15.161:0/2140469289 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f569c003520 con 0x7f56b00ac310 2024-03-28T23:34:34.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.518+0000 7f56b53a7640 1 -- 172.21.15.161:0/2140469289 >> v2:172.21.15.170:3300/0 conn(0x7f56b0151f80 msgr2=0x7f56b00c21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:34.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.518+0000 7f56b53a7640 1 --2- 172.21.15.161:0/2140469289 >> v2:172.21.15.170:3300/0 conn(0x7f56b0151f80 0x7f56b00c21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.518+0000 7f56b53a7640 1 -- 172.21.15.161:0/2140469289 >> v1:172.21.15.170:6789/0 conn(0x7f56b00ac310 legacy=0x7f56b0155e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:34.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.519+0000 7f56b53a7640 1 -- 172.21.15.161:0/2140469289 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b0152940 con 0x7f56b014f730 2024-03-28T23:34:34.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.519+0000 7f56b5ba8640 1 --2- 172.21.15.161:0/2140469289 >> v2:172.21.15.170:3300/0 conn(0x7f56b0151f80 0x7f56b00c21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:34:34.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.519+0000 7f56b53a7640 1 --2- 172.21.15.161:0/2140469289 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56b014f730 0x7f56b00aad10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0097a0 tx=0x7f56ac02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf4d9629804775ea server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:34.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.519+0000 7f56a7fff640 1 -- 172.21.15.161:0/2140469289 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ac0315a0 con 0x7f56b014f730 2024-03-28T23:34:34.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.520+0000 7f56a7fff640 1 -- 172.21.15.161:0/2140469289 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56ac037070 con 0x7f56b014f730 2024-03-28T23:34:34.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.520+0000 7f56a7fff640 1 -- 172.21.15.161:0/2140469289 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ac031960 con 0x7f56b014f730 2024-03-28T23:34:34.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.520+0000 7f56b760b640 1 -- 172.21.15.161:0/2140469289 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56b014f730 msgr2=0x7f56b00aad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:34.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.520+0000 7f56b760b640 1 --2- 172.21.15.161:0/2140469289 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56b014f730 0x7f56b00aad10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0097a0 tx=0x7f56ac02b840 comp rx=0 tx=0).stop 2024-03-28T23:34:34.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.520+0000 7f56b760b640 1 -- 172.21.15.161:0/2140469289 shutdown_connections 2024-03-28T23:34:34.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.520+0000 7f56b760b640 1 --2- 172.21.15.161:0/2140469289 >> v2:172.21.15.170:3300/0 conn(0x7f56b0151f80 0x7f56b00c21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.520+0000 7f56b760b640 1 --2- 172.21.15.161:0/2140469289 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56b014f730 0x7f56b00aad10 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.520+0000 7f56b760b640 1 -- 172.21.15.161:0/2140469289 >> 172.21.15.161:0/2140469289 conn(0x7f56b00c66d0 msgr2=0x7f56b00c6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:34.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.520+0000 7f56b760b640 1 -- 172.21.15.161:0/2140469289 shutdown_connections 2024-03-28T23:34:34.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.520+0000 7f56b760b640 1 -- 172.21.15.161:0/2140469289 wait complete. 2024-03-28T23:34:34.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.521+0000 7f56b760b640 1 Processor -- start 2024-03-28T23:34:34.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.521+0000 7f56b760b640 1 -- start start 2024-03-28T23:34:34.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.522+0000 7f56b760b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56b00ac310 0x7f56b01ee770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:34.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.522+0000 7f56b760b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56b014f730 0x7f56b0072c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:34.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.522+0000 7f56b760b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56b0151f80 0x7f56b01ea680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:34.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.522+0000 7f56b760b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f56b014fb10 con 0x7f56b00ac310 2024-03-28T23:34:34.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.522+0000 7f56b760b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f56b0155620 con 0x7f56b0151f80 2024-03-28T23:34:34.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.522+0000 7f56b760b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f56b0152410 con 0x7f56b014f730 2024-03-28T23:34:34.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.522+0000 7f56b53a7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56b00ac310 0x7f56b01ee770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:34.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.522+0000 7f56b53a7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56b00ac310 0x7f56b01ee770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34804/0 (socket says 172.21.15.161:34804) 2024-03-28T23:34:34.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.522+0000 7f56b53a7640 1 -- 172.21.15.161:0/2592797611 learned_addr learned my addr 172.21.15.161:0/2592797611 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:34.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.522+0000 7f56b4ba6640 1 --2- 172.21.15.161:0/2592797611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56b014f730 0x7f56b0072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:34.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.522+0000 7f56b5ba8640 1 --2- 172.21.15.161:0/2592797611 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56b0151f80 0x7f56b01ea680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:34.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.522+0000 7f56b53a7640 1 -- 172.21.15.161:0/2592797611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56b014f730 msgr2=0x7f56b0072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:34.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.522+0000 7f56b53a7640 1 --2- 172.21.15.161:0/2592797611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56b014f730 0x7f56b0072c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.522+0000 7f56b53a7640 1 -- 172.21.15.161:0/2592797611 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56b0151f80 msgr2=0x7f56b01ea680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:34.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.522+0000 7f56b53a7640 1 --2- 172.21.15.161:0/2592797611 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56b0151f80 0x7f56b01ea680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.522+0000 7f56b53a7640 1 -- 172.21.15.161:0/2592797611 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b00c6240 con 0x7f56b00ac310 2024-03-28T23:34:34.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.523+0000 7f56b53a7640 1 --2- 172.21.15.161:0/2592797611 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56b00ac310 0x7f56b01ee770 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f56ac034040 tx=0x7f56ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:34.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.523+0000 7f56a5ffb640 1 -- 172.21.15.161:0/2592797611 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ac032630 con 0x7f56b00ac310 2024-03-28T23:34:34.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.523+0000 7f56a5ffb640 1 -- 172.21.15.161:0/2592797611 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56ac0327f0 con 0x7f56b00ac310 2024-03-28T23:34:34.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.523+0000 7f56a5ffb640 1 -- 172.21.15.161:0/2592797611 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ac032be0 con 0x7f56b00ac310 2024-03-28T23:34:34.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.523+0000 7f56b760b640 1 -- 172.21.15.161:0/2592797611 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56b00c5f60 con 0x7f56b00ac310 2024-03-28T23:34:34.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.523+0000 7f56b760b640 1 -- 172.21.15.161:0/2592797611 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56b01edf70 con 0x7f56b00ac310 2024-03-28T23:34:34.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.524+0000 7f56b760b640 1 -- 172.21.15.161:0/2592797611 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56b004f550 con 0x7f56b00ac310 2024-03-28T23:34:34.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.524+0000 7f56a5ffb640 1 -- 172.21.15.161:0/2592797611 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f56ac04e430 con 0x7f56b00ac310 2024-03-28T23:34:34.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.525+0000 7f56a5ffb640 1 --2- 172.21.15.161:0/2592797611 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f569406b4d0 0x7f569406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:34.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.525+0000 7f56a5ffb640 1 -- 172.21.15.161:0/2592797611 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5249+0+0 (secure 0 0 0) 0x7f56ac02f030 con 0x7f56b00ac310 2024-03-28T23:34:34.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.525+0000 7f56b4ba6640 1 --2- 172.21.15.161:0/2592797611 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f569406b4d0 0x7f569406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:34.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.525+0000 7f56b4ba6640 1 --2- 172.21.15.161:0/2592797611 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f569406b4d0 0x7f569406d990 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f56b0069760 tx=0x7f56a0007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:34.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.529+0000 7f56a5ffb640 1 -- 172.21.15.161:0/2592797611 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f56ac040070 con 0x7f56b00ac310 2024-03-28T23:34:34.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.704+0000 7f56b760b640 1 -- 172.21.15.161:0/2592797611 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f56b00c1ef0 con 0x7f56b00ac310 2024-03-28T23:34:34.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.704+0000 7f56a5ffb640 1 -- 172.21.15.161:0/2592797611 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v118) ==== 84+0+65 (secure 0 0 0) 0x7f56ac07ad30 con 0x7f56b00ac310 2024-03-28T23:34:34.705 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:34.705 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2024-03-28T23:34:34.705 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:34:34.705 INFO:teuthology.orchestra.run.smithi161.stdout: "poolnum": 1, 2024-03-28T23:34:34.705 INFO:teuthology.orchestra.run.smithi161.stdout: "poolname": ".mgr" 2024-03-28T23:34:34.705 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:34:34.705 INFO:teuthology.orchestra.run.smithi161.stdout:] 2024-03-28T23:34:34.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.706+0000 7f56b760b640 1 -- 172.21.15.161:0/2592797611 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f569406b4d0 msgr2=0x7f569406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:34.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.706+0000 7f56b760b640 1 --2- 172.21.15.161:0/2592797611 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f569406b4d0 0x7f569406d990 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f56b0069760 tx=0x7f56a0007d30 comp rx=0 tx=0).stop 2024-03-28T23:34:34.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.706+0000 7f56b760b640 1 -- 172.21.15.161:0/2592797611 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56b00ac310 msgr2=0x7f56b01ee770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:34.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.706+0000 7f56b760b640 1 --2- 172.21.15.161:0/2592797611 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56b00ac310 0x7f56b01ee770 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f56ac034040 tx=0x7f56ac0046c0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.706+0000 7f56b760b640 1 -- 172.21.15.161:0/2592797611 shutdown_connections 2024-03-28T23:34:34.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.706+0000 7f56b760b640 1 --2- 172.21.15.161:0/2592797611 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f569406b4d0 0x7f569406d990 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.706+0000 7f56b760b640 1 --2- 172.21.15.161:0/2592797611 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56b0151f80 0x7f56b01ea680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.706+0000 7f56b760b640 1 --2- 172.21.15.161:0/2592797611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56b014f730 0x7f56b0072c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.706+0000 7f56b760b640 1 --2- 172.21.15.161:0/2592797611 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56b00ac310 0x7f56b01ee770 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.706+0000 7f56b760b640 1 -- 172.21.15.161:0/2592797611 >> 172.21.15.161:0/2592797611 conn(0x7f56b00c66d0 msgr2=0x7f56b0150810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:34.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.706+0000 7f56b760b640 1 -- 172.21.15.161:0/2592797611 shutdown_connections 2024-03-28T23:34:34.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.706+0000 7f56b760b640 1 -- 172.21.15.161:0/2592797611 wait complete. 2024-03-28T23:34:34.718 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-28T23:34:34.718 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-28T23:34:34.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.823+0000 7f27297a0640 1 Processor -- start 2024-03-28T23:34:34.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.823+0000 7f27297a0640 1 -- start start 2024-03-28T23:34:34.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.824+0000 7f27297a0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f27241526b0 0x7f27240c7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:34.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.824+0000 7f27297a0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f27240c8e40 0x7f27240c76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:34.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.824+0000 7f27297a0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27240be160 con 0x7f27240c8720 2024-03-28T23:34:34.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.824+0000 7f27297a0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2724158600 con 0x7f27240c8e40 2024-03-28T23:34:34.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.824+0000 7f27297a0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f2724158850 con 0x7f27241526b0 2024-03-28T23:34:34.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.824+0000 7f2728f9f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f27240c8e40 0x7f27240c76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:34.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.824+0000 7f27237fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f27240c8720 0x7f272406bfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:51018/0 (socket says 172.21.15.161:51018) 2024-03-28T23:34:34.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.824+0000 7f27237fe640 1 -- 172.21.15.161:0/2178116411 learned_addr learned my addr 172.21.15.161:0/2178116411 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:34.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.824+0000 7f2728f9f640 1 -- 172.21.15.161:0/2178116411 >> v2:172.21.15.170:3300/0 conn(0x7f27241526b0 msgr2=0x7f27240c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:34.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.824+0000 7f2728f9f640 1 --2- 172.21.15.161:0/2178116411 >> v2:172.21.15.170:3300/0 conn(0x7f27241526b0 0x7f27240c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.824+0000 7f2728f9f640 1 -- 172.21.15.161:0/2178116411 >> v1:172.21.15.161:6789/0 conn(0x7f27240c8720 legacy=0x7f272406bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:34.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.824+0000 7f2728f9f640 1 -- 172.21.15.161:0/2178116411 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2724158b30 con 0x7f27240c8e40 2024-03-28T23:34:34.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.825+0000 7f2728f9f640 1 --2- 172.21.15.161:0/2178116411 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27240c8e40 0x7f27240c76a0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f27240cb880 tx=0x7f271c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40e886bcc02ab12a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:34.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.825+0000 7f2722ffd640 1 -- 172.21.15.161:0/2178116411 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271c039070 con 0x7f27240c8e40 2024-03-28T23:34:34.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.825+0000 7f2722ffd640 1 -- 172.21.15.161:0/2178116411 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f271c02cc80 con 0x7f27240c8e40 2024-03-28T23:34:34.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.825+0000 7f2722ffd640 1 -- 172.21.15.161:0/2178116411 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271c034660 con 0x7f27240c8e40 2024-03-28T23:34:34.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.825+0000 7f27297a0640 1 -- 172.21.15.161:0/2178116411 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27240c8e40 msgr2=0x7f27240c76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:34.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.825+0000 7f27297a0640 1 --2- 172.21.15.161:0/2178116411 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27240c8e40 0x7f27240c76a0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f27240cb880 tx=0x7f271c02bed0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.825+0000 7f27297a0640 1 -- 172.21.15.161:0/2178116411 shutdown_connections 2024-03-28T23:34:34.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.825+0000 7f27297a0640 1 --2- 172.21.15.161:0/2178116411 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27240c8e40 0x7f27240c76a0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.825+0000 7f27297a0640 1 --2- 172.21.15.161:0/2178116411 >> v2:172.21.15.170:3300/0 conn(0x7f27241526b0 0x7f27240c7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.825+0000 7f27297a0640 1 -- 172.21.15.161:0/2178116411 >> 172.21.15.161:0/2178116411 conn(0x7f27240ac410 msgr2=0x7f27240ac7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:34.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.825+0000 7f27297a0640 1 -- 172.21.15.161:0/2178116411 shutdown_connections 2024-03-28T23:34:34.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.825+0000 7f27297a0640 1 -- 172.21.15.161:0/2178116411 wait complete. 2024-03-28T23:34:34.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.826+0000 7f27297a0640 1 Processor -- start 2024-03-28T23:34:34.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.827+0000 7f27297a0640 1 -- start start 2024-03-28T23:34:34.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.827+0000 7f27297a0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27240c8720 0x7f2724156cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:34.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.827+0000 7f27297a0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f27240c8e40 0x7f27241eb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:34.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.827+0000 7f27297a0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f27241526b0 0x7f27241eb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:34.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.827+0000 7f27297a0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2724152a90 con 0x7f27240c8720 2024-03-28T23:34:34.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.827+0000 7f27297a0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f27241590f0 con 0x7f27241526b0 2024-03-28T23:34:34.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.827+0000 7f27297a0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f272406c6f0 con 0x7f27240c8e40 2024-03-28T23:34:34.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.827+0000 7f2723fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27240c8720 0x7f2724156cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:34.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.827+0000 7f2723fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27240c8720 0x7f2724156cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34826/0 (socket says 172.21.15.161:34826) 2024-03-28T23:34:34.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.828+0000 7f2723fff640 1 -- 172.21.15.161:0/478259989 learned_addr learned my addr 172.21.15.161:0/478259989 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:34.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.828+0000 7f27237fe640 1 --2- 172.21.15.161:0/478259989 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f27240c8e40 0x7f27241eb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:34.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.828+0000 7f2728f9f640 1 --2- 172.21.15.161:0/478259989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f27241526b0 0x7f27241eb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:34.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.828+0000 7f2723fff640 1 -- 172.21.15.161:0/478259989 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f27240c8e40 msgr2=0x7f27241eb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:34.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.828+0000 7f2723fff640 1 --2- 172.21.15.161:0/478259989 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f27240c8e40 0x7f27241eb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.828+0000 7f2723fff640 1 -- 172.21.15.161:0/478259989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f27241526b0 msgr2=0x7f27241eb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:34.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.828+0000 7f2723fff640 1 --2- 172.21.15.161:0/478259989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f27241526b0 0x7f27241eb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:34.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.828+0000 7f2723fff640 1 -- 172.21.15.161:0/478259989 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27240aa790 con 0x7f27240c8720 2024-03-28T23:34:34.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.828+0000 7f27237fe640 1 --2- 172.21.15.161:0/478259989 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f27240c8e40 0x7f27241eb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:34.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.828+0000 7f2728f9f640 1 --2- 172.21.15.161:0/478259989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f27241526b0 0x7f27241eb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:34.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.828+0000 7f2723fff640 1 --2- 172.21.15.161:0/478259989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27240c8720 0x7f2724156cc0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f271400b410 tx=0x7f271400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:34.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.828+0000 7f2720ff9640 1 -- 172.21.15.161:0/478259989 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2714015020 con 0x7f27240c8720 2024-03-28T23:34:34.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.828+0000 7f2720ff9640 1 -- 172.21.15.161:0/478259989 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2714011850 con 0x7f27240c8720 2024-03-28T23:34:34.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.828+0000 7f27297a0640 1 -- 172.21.15.161:0/478259989 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27240aadd0 con 0x7f27240c8720 2024-03-28T23:34:34.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.828+0000 7f2720ff9640 1 -- 172.21.15.161:0/478259989 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2714011c40 con 0x7f27240c8720 2024-03-28T23:34:34.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.828+0000 7f27297a0640 1 -- 172.21.15.161:0/478259989 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2724157200 con 0x7f27240c8720 2024-03-28T23:34:34.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.829+0000 7f27297a0640 1 -- 172.21.15.161:0/478259989 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27240c7c00 con 0x7f27240c8720 2024-03-28T23:34:34.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.830+0000 7f2720ff9640 1 -- 172.21.15.161:0/478259989 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2714018020 con 0x7f27240c8720 2024-03-28T23:34:34.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.830+0000 7f2720ff9640 1 --2- 172.21.15.161:0/478259989 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f271006b500 0x7f271006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:34.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.830+0000 7f27237fe640 1 --2- 172.21.15.161:0/478259989 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f271006b500 0x7f271006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:34.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.830+0000 7f2720ff9640 1 -- 172.21.15.161:0/478259989 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5249+0+0 (secure 0 0 0) 0x7f2714089ee0 con 0x7f27240c8720 2024-03-28T23:34:34.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.830+0000 7f27237fe640 1 --2- 172.21.15.161:0/478259989 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f271006b500 0x7f271006d9c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f2724151840 tx=0x7f2718009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:34.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:34.834+0000 7f2720ff9640 1 -- 172.21.15.161:0/478259989 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f27240c7c00 con 0x7f27240c8720 2024-03-28T23:34:35.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.005+0000 7f27297a0640 1 -- 172.21.15.161:0/478259989 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f272414da30 con 0x7f27240c8720 2024-03-28T23:34:35.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.435+0000 7f2720ff9640 1 -- 172.21.15.161:0/478259989 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v119) ==== 138+0+0 (secure 0 0 0) 0x7f272414da30 con 0x7f27240c8720 2024-03-28T23:34:35.435 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_metadata' created 2024-03-28T23:34:35.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.437+0000 7f27297a0640 1 -- 172.21.15.161:0/478259989 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f271006b500 msgr2=0x7f271006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:35.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.437+0000 7f27297a0640 1 --2- 172.21.15.161:0/478259989 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f271006b500 0x7f271006d9c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f2724151840 tx=0x7f2718009040 comp rx=0 tx=0).stop 2024-03-28T23:34:35.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.437+0000 7f27297a0640 1 -- 172.21.15.161:0/478259989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27240c8720 msgr2=0x7f2724156cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:35.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.437+0000 7f27297a0640 1 --2- 172.21.15.161:0/478259989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27240c8720 0x7f2724156cc0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f271400b410 tx=0x7f271400b8d0 comp rx=0 tx=0).stop 2024-03-28T23:34:35.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.437+0000 7f27297a0640 1 -- 172.21.15.161:0/478259989 shutdown_connections 2024-03-28T23:34:35.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.437+0000 7f27297a0640 1 --2- 172.21.15.161:0/478259989 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f271006b500 0x7f271006d9c0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:35.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.437+0000 7f27297a0640 1 --2- 172.21.15.161:0/478259989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f27241526b0 0x7f27241eb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:35.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.437+0000 7f27297a0640 1 --2- 172.21.15.161:0/478259989 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f27240c8e40 0x7f27241eb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:35.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.437+0000 7f27297a0640 1 --2- 172.21.15.161:0/478259989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27240c8720 0x7f2724156cc0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:35.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.437+0000 7f27297a0640 1 -- 172.21.15.161:0/478259989 >> 172.21.15.161:0/478259989 conn(0x7f27240ac410 msgr2=0x7f27240cac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:35.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.437+0000 7f27297a0640 1 -- 172.21.15.161:0/478259989 shutdown_connections 2024-03-28T23:34:35.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.438+0000 7f27297a0640 1 -- 172.21.15.161:0/478259989 wait complete. 2024-03-28T23:34:35.452 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T23:34:35.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.561+0000 7f1a5aa09640 1 Processor -- start 2024-03-28T23:34:35.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.561+0000 7f1a5aa09640 1 -- start start 2024-03-28T23:34:35.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.561+0000 7f1a5aa09640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1a540a96f0 0x7f1a5415e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:35.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.561+0000 7f1a5aa09640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a540c2f50 con 0x7f1a54150520 2024-03-28T23:34:35.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.561+0000 7f1a5aa09640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a541530d0 con 0x7f1a540c1040 2024-03-28T23:34:35.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.561+0000 7f1a5aa09640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f1a540a9c20 con 0x7f1a540a96f0 2024-03-28T23:34:35.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.562+0000 7f1a53fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f1a54150520 0x7f1a540ab280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:51022/0 (socket says 172.21.15.161:51022) 2024-03-28T23:34:35.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.562+0000 7f1a53fff640 1 -- 172.21.15.161:0/3446255670 learned_addr learned my addr 172.21.15.161:0/3446255670 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:35.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.562+0000 7f1a58fa6640 1 --2- 172.21.15.161:0/3446255670 >> v2:172.21.15.170:3300/0 conn(0x7f1a540a96f0 0x7f1a5415e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:35.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.562+0000 7f1a52ffd640 1 -- 172.21.15.161:0/3446255670 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1548841181 0 0) 0x7f1a540c2f50 con 0x7f1a54150520 2024-03-28T23:34:35.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.562+0000 7f1a52ffd640 1 -- 172.21.15.161:0/3446255670 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a3c003520 con 0x7f1a54150520 2024-03-28T23:34:35.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.562+0000 7f1a52ffd640 1 -- 172.21.15.161:0/3446255670 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1426446504 0 0) 0x7f1a3c003520 con 0x7f1a54150520 2024-03-28T23:34:35.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.562+0000 7f1a58fa6640 1 -- 172.21.15.161:0/3446255670 >> v1:172.21.15.170:6789/0 conn(0x7f1a540c1040 legacy=0x7f1a541535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:35.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.562+0000 7f1a58fa6640 1 -- 172.21.15.161:0/3446255670 >> v1:172.21.15.161:6789/0 conn(0x7f1a54150520 legacy=0x7f1a540ab280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:35.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.562+0000 7f1a58fa6640 1 -- 172.21.15.161:0/3446255670 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a540c2990 con 0x7f1a540a96f0 2024-03-28T23:34:35.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.563+0000 7f1a58fa6640 1 --2- 172.21.15.161:0/3446255670 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a540a96f0 0x7f1a5415e8c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c009d90 tx=0x7f1a4c02bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3495309e0b01a0e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:35.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.563+0000 7f1a52ffd640 1 -- 172.21.15.161:0/3446255670 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c039070 con 0x7f1a540a96f0 2024-03-28T23:34:35.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.563+0000 7f1a52ffd640 1 -- 172.21.15.161:0/3446255670 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a4c02c8b0 con 0x7f1a540a96f0 2024-03-28T23:34:35.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.563+0000 7f1a52ffd640 1 -- 172.21.15.161:0/3446255670 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a44001670 con 0x7f1a540a96f0 2024-03-28T23:34:35.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.564+0000 7f1a5aa09640 1 -- 172.21.15.161:0/3446255670 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a540a96f0 msgr2=0x7f1a5415e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:35.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.564+0000 7f1a5aa09640 1 --2- 172.21.15.161:0/3446255670 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a540a96f0 0x7f1a5415e8c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c009d90 tx=0x7f1a4c02bd80 comp rx=0 tx=0).stop 2024-03-28T23:34:35.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.564+0000 7f1a5aa09640 1 -- 172.21.15.161:0/3446255670 shutdown_connections 2024-03-28T23:34:35.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.564+0000 7f1a5aa09640 1 --2- 172.21.15.161:0/3446255670 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a540a96f0 0x7f1a5415e8c0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:35.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.564+0000 7f1a5aa09640 1 -- 172.21.15.161:0/3446255670 >> 172.21.15.161:0/3446255670 conn(0x7f1a540a83f0 msgr2=0x7f1a540a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:35.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.564+0000 7f1a5aa09640 1 -- 172.21.15.161:0/3446255670 shutdown_connections 2024-03-28T23:34:35.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.564+0000 7f1a5aa09640 1 -- 172.21.15.161:0/3446255670 wait complete. 2024-03-28T23:34:35.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.565+0000 7f1a5aa09640 1 Processor -- start 2024-03-28T23:34:35.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.565+0000 7f1a5aa09640 1 -- start start 2024-03-28T23:34:35.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.565+0000 7f1a5aa09640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a540a96f0 0x7f1a540729d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:35.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.566+0000 7f1a5aa09640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a540c1040 0x7f1a54156cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:35.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.566+0000 7f1a5aa09640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a54150520 0x7f1a5414ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:35.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.566+0000 7f1a5aa09640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1a540ab990 con 0x7f1a540a96f0 2024-03-28T23:34:35.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.566+0000 7f1a5aa09640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1a54150900 con 0x7f1a540c1040 2024-03-28T23:34:35.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.566+0000 7f1a5aa09640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1a540abb10 con 0x7f1a54150520 2024-03-28T23:34:35.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.566+0000 7f1a53fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a540a96f0 0x7f1a540729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:35.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.566+0000 7f1a53fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a540a96f0 0x7f1a540729d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34830/0 (socket says 172.21.15.161:34830) 2024-03-28T23:34:35.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.566+0000 7f1a53fff640 1 -- 172.21.15.161:0/1781436987 learned_addr learned my addr 172.21.15.161:0/1781436987 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:35.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.566+0000 7f1a537fe640 1 --2- 172.21.15.161:0/1781436987 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a540c1040 0x7f1a54156cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:35.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.566+0000 7f1a58fa6640 1 --2- 172.21.15.161:0/1781436987 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a54150520 0x7f1a5414ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:35.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.566+0000 7f1a53fff640 1 -- 172.21.15.161:0/1781436987 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a54150520 msgr2=0x7f1a5414ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:35.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.566+0000 7f1a53fff640 1 --2- 172.21.15.161:0/1781436987 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a54150520 0x7f1a5414ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:35.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.566+0000 7f1a53fff640 1 -- 172.21.15.161:0/1781436987 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a540c1040 msgr2=0x7f1a54156cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:35.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.566+0000 7f1a53fff640 1 --2- 172.21.15.161:0/1781436987 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a540c1040 0x7f1a54156cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:35.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.566+0000 7f1a53fff640 1 -- 172.21.15.161:0/1781436987 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a541544a0 con 0x7f1a540a96f0 2024-03-28T23:34:35.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.566+0000 7f1a58fa6640 1 --2- 172.21.15.161:0/1781436987 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a54150520 0x7f1a5414ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:35.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.566+0000 7f1a53fff640 1 --2- 172.21.15.161:0/1781436987 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a540a96f0 0x7f1a540729d0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f1a4400c730 tx=0x7f1a4400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:35.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.566+0000 7f1a50ff9640 1 -- 172.21.15.161:0/1781436987 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a44017070 con 0x7f1a540a96f0 2024-03-28T23:34:35.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.566+0000 7f1a50ff9640 1 -- 172.21.15.161:0/1781436987 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a44005af0 con 0x7f1a540a96f0 2024-03-28T23:34:35.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.566+0000 7f1a50ff9640 1 -- 172.21.15.161:0/1781436987 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a44007070 con 0x7f1a540a96f0 2024-03-28T23:34:35.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.566+0000 7f1a5aa09640 1 -- 172.21.15.161:0/1781436987 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a54154780 con 0x7f1a540a96f0 2024-03-28T23:34:35.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.566+0000 7f1a5aa09640 1 -- 172.21.15.161:0/1781436987 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a5414a900 con 0x7f1a540a96f0 2024-03-28T23:34:35.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.567+0000 7f1a5aa09640 1 -- 172.21.15.161:0/1781436987 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a54154ca0 con 0x7f1a540a96f0 2024-03-28T23:34:35.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.568+0000 7f1a50ff9640 1 -- 172.21.15.161:0/1781436987 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1a4401a070 con 0x7f1a540a96f0 2024-03-28T23:34:35.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.568+0000 7f1a50ff9640 1 --2- 172.21.15.161:0/1781436987 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1a4006b500 0x7f1a4006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:35.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.568+0000 7f1a50ff9640 1 -- 172.21.15.161:0/1781436987 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5619+0+0 (secure 0 0 0) 0x7f1a440895c0 con 0x7f1a540a96f0 2024-03-28T23:34:35.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.568+0000 7f1a537fe640 1 --2- 172.21.15.161:0/1781436987 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1a4006b500 0x7f1a4006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:35.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.569+0000 7f1a537fe640 1 --2- 172.21.15.161:0/1781436987 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1a4006b500 0x7f1a4006d9c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f1a540ab650 tx=0x7f1a48009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:35.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.573+0000 7f1a50ff9640 1 -- 172.21.15.161:0/1781436987 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1a44057ed0 con 0x7f1a540a96f0 2024-03-28T23:34:35.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:35.746+0000 7f1a5aa09640 1 -- 172.21.15.161:0/1781436987 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f1a5404f4b0 con 0x7f1a540a96f0 2024-03-28T23:34:36.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.437+0000 7f1a50ff9640 1 -- 172.21.15.161:0/1781436987 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v120) ==== 170+0+0 (secure 0 0 0) 0x7f1a4405bf70 con 0x7f1a540a96f0 2024-03-28T23:34:36.438 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data' created 2024-03-28T23:34:36.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.440+0000 7f1a5aa09640 1 -- 172.21.15.161:0/1781436987 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1a4006b500 msgr2=0x7f1a4006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:36.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.440+0000 7f1a5aa09640 1 --2- 172.21.15.161:0/1781436987 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1a4006b500 0x7f1a4006d9c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f1a540ab650 tx=0x7f1a48009040 comp rx=0 tx=0).stop 2024-03-28T23:34:36.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.440+0000 7f1a5aa09640 1 -- 172.21.15.161:0/1781436987 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a540a96f0 msgr2=0x7f1a540729d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:36.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.440+0000 7f1a5aa09640 1 --2- 172.21.15.161:0/1781436987 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a540a96f0 0x7f1a540729d0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f1a4400c730 tx=0x7f1a4400cbf0 comp rx=0 tx=0).stop 2024-03-28T23:34:36.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.441+0000 7f1a5aa09640 1 -- 172.21.15.161:0/1781436987 shutdown_connections 2024-03-28T23:34:36.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.441+0000 7f1a5aa09640 1 --2- 172.21.15.161:0/1781436987 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1a4006b500 0x7f1a4006d9c0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:36.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.441+0000 7f1a5aa09640 1 --2- 172.21.15.161:0/1781436987 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a54150520 0x7f1a5414ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:36.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.441+0000 7f1a5aa09640 1 --2- 172.21.15.161:0/1781436987 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a540c1040 0x7f1a54156cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:36.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.441+0000 7f1a5aa09640 1 --2- 172.21.15.161:0/1781436987 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a540a96f0 0x7f1a540729d0 secure :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f1a4400c730 tx=0x7f1a4400cbf0 comp rx=0 tx=0).stop 2024-03-28T23:34:36.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.441+0000 7f1a5aa09640 1 -- 172.21.15.161:0/1781436987 >> 172.21.15.161:0/1781436987 conn(0x7f1a540a83f0 msgr2=0x7f1a54150da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:36.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.442+0000 7f1a5aa09640 1 -- 172.21.15.161:0/1781436987 shutdown_connections 2024-03-28T23:34:36.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.442+0000 7f1a5aa09640 1 -- 172.21.15.161:0/1781436987 wait complete. 2024-03-28T23:34:36.469 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-28T23:34:36.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.574+0000 7fe37109d640 1 Processor -- start 2024-03-28T23:34:36.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.574+0000 7fe37109d640 1 -- start start 2024-03-28T23:34:36.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.575+0000 7fe37109d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe36c0a96f0 0x7fe36c15e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:36.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.575+0000 7fe37109d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe36c0c2f50 con 0x7fe36c0c1040 2024-03-28T23:34:36.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.575+0000 7fe37109d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe36c1530d0 con 0x7fe36c150520 2024-03-28T23:34:36.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.575+0000 7fe37109d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe36c0a9c20 con 0x7fe36c0a96f0 2024-03-28T23:34:36.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.575+0000 7fe36b59e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe36c0a96f0 0x7fe36c15e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:36.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.575+0000 7fe36b59e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe36c0a96f0 0x7fe36c15e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34840/0 (socket says 172.21.15.161:34840) 2024-03-28T23:34:36.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.575+0000 7fe36b59e640 1 -- 172.21.15.161:0/3975374962 learned_addr learned my addr 172.21.15.161:0/3975374962 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:36.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.575+0000 7fe36a59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe36c0c1040 0x7fe36c1535b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:51026/0 (socket says 172.21.15.161:51026) 2024-03-28T23:34:36.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.576+0000 7fe36b59e640 1 -- 172.21.15.161:0/3975374962 >> v1:172.21.15.170:6789/0 conn(0x7fe36c150520 legacy=0x7fe36c0ab280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:36.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.576+0000 7fe36b59e640 1 -- 172.21.15.161:0/3975374962 >> v1:172.21.15.161:6789/0 conn(0x7fe36c0c1040 legacy=0x7fe36c1535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:36.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.576+0000 7fe369d9b640 1 -- 172.21.15.161:0/3975374962 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 92242989 0 0) 0x7fe36c0c2f50 con 0x7fe36c0c1040 2024-03-28T23:34:36.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.576+0000 7fe36b59e640 1 -- 172.21.15.161:0/3975374962 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe36c0c2990 con 0x7fe36c0a96f0 2024-03-28T23:34:36.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.576+0000 7fe36b59e640 1 --2- 172.21.15.161:0/3975374962 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe36c0a96f0 0x7fe36c15e8c0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fe364009e50 tx=0x7fe36402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b69065e629e53096 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:36.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.576+0000 7fe369d9b640 1 -- 172.21.15.161:0/3975374962 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36402ca80 con 0x7fe36c0a96f0 2024-03-28T23:34:36.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.577+0000 7fe369d9b640 1 -- 172.21.15.161:0/3975374962 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe36402cc20 con 0x7fe36c0a96f0 2024-03-28T23:34:36.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.577+0000 7fe369d9b640 1 -- 172.21.15.161:0/3975374962 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3640343f0 con 0x7fe36c0a96f0 2024-03-28T23:34:36.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.577+0000 7fe37109d640 1 -- 172.21.15.161:0/3975374962 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe36c0a96f0 msgr2=0x7fe36c15e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:36.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.577+0000 7fe37109d640 1 --2- 172.21.15.161:0/3975374962 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe36c0a96f0 0x7fe36c15e8c0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fe364009e50 tx=0x7fe36402bee0 comp rx=0 tx=0).stop 2024-03-28T23:34:36.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.577+0000 7fe37109d640 1 -- 172.21.15.161:0/3975374962 shutdown_connections 2024-03-28T23:34:36.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.577+0000 7fe37109d640 1 --2- 172.21.15.161:0/3975374962 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe36c0a96f0 0x7fe36c15e8c0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:36.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.577+0000 7fe37109d640 1 -- 172.21.15.161:0/3975374962 >> 172.21.15.161:0/3975374962 conn(0x7fe36c0a83f0 msgr2=0x7fe36c0a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:36.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.577+0000 7fe37109d640 1 -- 172.21.15.161:0/3975374962 shutdown_connections 2024-03-28T23:34:36.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.577+0000 7fe37109d640 1 -- 172.21.15.161:0/3975374962 wait complete. 2024-03-28T23:34:36.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.578+0000 7fe37109d640 1 Processor -- start 2024-03-28T23:34:36.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.578+0000 7fe37109d640 1 -- start start 2024-03-28T23:34:36.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.579+0000 7fe37109d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe36c0a96f0 0x7fe36c1e87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:36.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.579+0000 7fe37109d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe36c0c1040 0x7fe36c14a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:36.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.579+0000 7fe37109d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe36c150520 0x7fe36c0729d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:36.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.579+0000 7fe37109d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe36c0ab990 con 0x7fe36c150520 2024-03-28T23:34:36.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.579+0000 7fe37109d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe36c150900 con 0x7fe36c0c1040 2024-03-28T23:34:36.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.579+0000 7fe37109d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe36c0abb10 con 0x7fe36c0a96f0 2024-03-28T23:34:36.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.579+0000 7fe36b59e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe36c150520 0x7fe36c0729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:36.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.579+0000 7fe36b59e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe36c150520 0x7fe36c0729d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34846/0 (socket says 172.21.15.161:34846) 2024-03-28T23:34:36.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.579+0000 7fe36b59e640 1 -- 172.21.15.161:0/3691182343 learned_addr learned my addr 172.21.15.161:0/3691182343 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:36.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.579+0000 7fe36a59c640 1 --2- 172.21.15.161:0/3691182343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe36c0c1040 0x7fe36c14a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:36.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.579+0000 7fe36ad9d640 1 --2- 172.21.15.161:0/3691182343 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe36c0a96f0 0x7fe36c1e87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:36.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.579+0000 7fe36b59e640 1 -- 172.21.15.161:0/3691182343 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe36c0a96f0 msgr2=0x7fe36c1e87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:36.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.580+0000 7fe36b59e640 1 --2- 172.21.15.161:0/3691182343 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe36c0a96f0 0x7fe36c1e87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:36.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.580+0000 7fe36b59e640 1 -- 172.21.15.161:0/3691182343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe36c0c1040 msgr2=0x7fe36c14a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:36.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.580+0000 7fe36b59e640 1 --2- 172.21.15.161:0/3691182343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe36c0c1040 0x7fe36c14a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:36.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.580+0000 7fe36b59e640 1 -- 172.21.15.161:0/3691182343 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe36c14d420 con 0x7fe36c150520 2024-03-28T23:34:36.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.580+0000 7fe36a59c640 1 --2- 172.21.15.161:0/3691182343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe36c0c1040 0x7fe36c14a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:36.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.580+0000 7fe36ad9d640 1 --2- 172.21.15.161:0/3691182343 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe36c0a96f0 0x7fe36c1e87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:36.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.580+0000 7fe36b59e640 1 --2- 172.21.15.161:0/3691182343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe36c150520 0x7fe36c0729d0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fe364036040 tx=0x7fe364004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:36.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.580+0000 7fe3537fe640 1 -- 172.21.15.161:0/3691182343 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe364043020 con 0x7fe36c150520 2024-03-28T23:34:36.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.580+0000 7fe3537fe640 1 -- 172.21.15.161:0/3691182343 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe36403e450 con 0x7fe36c150520 2024-03-28T23:34:36.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.580+0000 7fe37109d640 1 -- 172.21.15.161:0/3691182343 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe36c14d700 con 0x7fe36c150520 2024-03-28T23:34:36.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.580+0000 7fe3537fe640 1 -- 172.21.15.161:0/3691182343 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe364009510 con 0x7fe36c150520 2024-03-28T23:34:36.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.580+0000 7fe37109d640 1 -- 172.21.15.161:0/3691182343 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe36c1ec010 con 0x7fe36c150520 2024-03-28T23:34:36.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.581+0000 7fe37109d640 1 -- 172.21.15.161:0/3691182343 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe36c04f4b0 con 0x7fe36c150520 2024-03-28T23:34:36.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.581+0000 7fe3537fe640 1 -- 172.21.15.161:0/3691182343 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe364046020 con 0x7fe36c150520 2024-03-28T23:34:36.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.582+0000 7fe3537fe640 1 --2- 172.21.15.161:0/3691182343 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe35806b500 0x7fe35806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:36.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.582+0000 7fe3537fe640 1 -- 172.21.15.161:0/3691182343 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 6001+0+0 (secure 0 0 0) 0x7fe3640aca00 con 0x7fe36c150520 2024-03-28T23:34:36.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.582+0000 7fe36ad9d640 1 --2- 172.21.15.161:0/3691182343 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe35806b500 0x7fe35806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:36.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.582+0000 7fe36ad9d640 1 --2- 172.21.15.161:0/3691182343 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe35806b500 0x7fe35806d9c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fe36c151d20 tx=0x7fe360009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:36.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.587+0000 7fe3537fe640 1 -- 172.21.15.161:0/3691182343 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe36404d070 con 0x7fe36c150520 2024-03-28T23:34:36.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:36.795+0000 7fe37109d640 1 -- 172.21.15.161:0/3691182343 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fe36c1ec1e0 con 0x7fe36c150520 2024-03-28T23:34:37.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.445+0000 7fe3537fe640 1 -- 172.21.15.161:0/3691182343 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 19 and data pool 20 v105) ==== 174+0+0 (secure 0 0 0) 0x7fe36407b790 con 0x7fe36c150520 2024-03-28T23:34:37.451 INFO:teuthology.orchestra.run.smithi161.stderr:new fs with metadata pool 19 and data pool 20 2024-03-28T23:34:37.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.457+0000 7fe37109d640 1 -- 172.21.15.161:0/3691182343 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe35806b500 msgr2=0x7fe35806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:37.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.457+0000 7fe37109d640 1 --2- 172.21.15.161:0/3691182343 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe35806b500 0x7fe35806d9c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fe36c151d20 tx=0x7fe360009040 comp rx=0 tx=0).stop 2024-03-28T23:34:37.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.457+0000 7fe37109d640 1 -- 172.21.15.161:0/3691182343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe36c150520 msgr2=0x7fe36c0729d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:37.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.457+0000 7fe37109d640 1 --2- 172.21.15.161:0/3691182343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe36c150520 0x7fe36c0729d0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fe364036040 tx=0x7fe364004a40 comp rx=0 tx=0).stop 2024-03-28T23:34:37.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.458+0000 7fe37109d640 1 -- 172.21.15.161:0/3691182343 shutdown_connections 2024-03-28T23:34:37.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.459+0000 7fe37109d640 1 --2- 172.21.15.161:0/3691182343 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe35806b500 0x7fe35806d9c0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:37.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.459+0000 7fe37109d640 1 --2- 172.21.15.161:0/3691182343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe36c150520 0x7fe36c0729d0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:37.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.459+0000 7fe37109d640 1 --2- 172.21.15.161:0/3691182343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe36c0c1040 0x7fe36c14a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:37.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.459+0000 7fe37109d640 1 --2- 172.21.15.161:0/3691182343 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe36c0a96f0 0x7fe36c1e87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:37.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.459+0000 7fe37109d640 1 -- 172.21.15.161:0/3691182343 >> 172.21.15.161:0/3691182343 conn(0x7fe36c0a83f0 msgr2=0x7fe36c151d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:37.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.460+0000 7fe37109d640 1 -- 172.21.15.161:0/3691182343 shutdown_connections 2024-03-28T23:34:37.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.460+0000 7fe37109d640 1 -- 172.21.15.161:0/3691182343 wait complete. 2024-03-28T23:34:37.485 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:34:37.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.589+0000 7f5709c64640 1 Processor -- start 2024-03-28T23:34:37.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.589+0000 7f5709c64640 1 -- start start 2024-03-28T23:34:37.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.590+0000 7f5709c64640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f570414eca0 0x7f57040c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:37.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.590+0000 7f5709c64640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f57040c1dc0 0x7f570406ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:37.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.590+0000 7f5709c64640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5704151850 con 0x7f570406cf60 2024-03-28T23:34:37.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.590+0000 7f5709c64640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f5704152460 con 0x7f57040c1dc0 2024-03-28T23:34:37.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.590+0000 7f5709c64640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5704151b30 con 0x7f570414eca0 2024-03-28T23:34:37.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.590+0000 7f5703fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f570406cf60 0x7f57040a98c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:51036/0 (socket says 172.21.15.161:51036) 2024-03-28T23:34:37.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.590+0000 7f5703fff640 1 -- 172.21.15.161:0/2569188955 learned_addr learned my addr 172.21.15.161:0/2569188955 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:37.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.590+0000 7f57037fe640 1 --2- 172.21.15.161:0/2569188955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f570414eca0 0x7f57040c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:37.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.590+0000 7f5702ffd640 1 --2- 172.21.15.161:0/2569188955 >> v2:172.21.15.170:3300/0 conn(0x7f57040c1dc0 0x7f570406ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:37.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.591+0000 7f57027fc640 1 -- 172.21.15.161:0/2569188955 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3779799888 0 0) 0x7f5704151850 con 0x7f570406cf60 2024-03-28T23:34:37.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.591+0000 7f57027fc640 1 -- 172.21.15.161:0/2569188955 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56ec005730 con 0x7f570406cf60 2024-03-28T23:34:37.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.591+0000 7f5702ffd640 1 -- 172.21.15.161:0/2569188955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f570414eca0 msgr2=0x7f57040c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:37.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.591+0000 7f5702ffd640 1 --2- 172.21.15.161:0/2569188955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f570414eca0 0x7f57040c1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:37.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.591+0000 7f5702ffd640 1 -- 172.21.15.161:0/2569188955 >> v1:172.21.15.161:6789/0 conn(0x7f570406cf60 legacy=0x7f57040a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:37.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.591+0000 7f57027fc640 1 -- 172.21.15.161:0/2569188955 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2342710197 0 0) 0x7f56ec005730 con 0x7f570406cf60 2024-03-28T23:34:37.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.591+0000 7f5702ffd640 1 -- 172.21.15.161:0/2569188955 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5704151e10 con 0x7f57040c1dc0 2024-03-28T23:34:37.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.592+0000 7f5702ffd640 1 --2- 172.21.15.161:0/2569188955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57040c1dc0 0x7f570406ca20 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f56f80097a0 tx=0x7f56f802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f61eb6735c41a67b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:37.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.592+0000 7f57027fc640 1 -- 172.21.15.161:0/2569188955 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f8037070 con 0x7f57040c1dc0 2024-03-28T23:34:37.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.592+0000 7f57027fc640 1 -- 172.21.15.161:0/2569188955 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56f8030510 con 0x7f57040c1dc0 2024-03-28T23:34:37.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.592+0000 7f57027fc640 1 -- 172.21.15.161:0/2569188955 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56fc0019f0 con 0x7f57040c1dc0 2024-03-28T23:34:37.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.592+0000 7f5709c64640 1 -- 172.21.15.161:0/2569188955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57040c1dc0 msgr2=0x7f570406ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:37.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.592+0000 7f5709c64640 1 --2- 172.21.15.161:0/2569188955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57040c1dc0 0x7f570406ca20 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f56f80097a0 tx=0x7f56f802b710 comp rx=0 tx=0).stop 2024-03-28T23:34:37.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.592+0000 7f5709c64640 1 -- 172.21.15.161:0/2569188955 shutdown_connections 2024-03-28T23:34:37.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.592+0000 7f5709c64640 1 --2- 172.21.15.161:0/2569188955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57040c1dc0 0x7f570406ca20 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:37.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.592+0000 7f5709c64640 1 --2- 172.21.15.161:0/2569188955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f570414eca0 0x7f57040c1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:37.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.592+0000 7f5709c64640 1 -- 172.21.15.161:0/2569188955 >> 172.21.15.161:0/2569188955 conn(0x7f57040c6670 msgr2=0x7f57040c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:37.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.592+0000 7f5709c64640 1 -- 172.21.15.161:0/2569188955 shutdown_connections 2024-03-28T23:34:37.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.592+0000 7f5709c64640 1 -- 172.21.15.161:0/2569188955 wait complete. 2024-03-28T23:34:37.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.593+0000 7f5709c64640 1 Processor -- start 2024-03-28T23:34:37.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.594+0000 7f5709c64640 1 -- start start 2024-03-28T23:34:37.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.594+0000 7f5709c64640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f570406cf60 0x7f570406b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:37.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.594+0000 7f5709c64640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57040c1dc0 0x7f570414b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:37.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.594+0000 7f5709c64640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f570414eca0 0x7f570414b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:37.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.594+0000 7f5709c64640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f57041514b0 con 0x7f57040c1dc0 2024-03-28T23:34:37.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.594+0000 7f5709c64640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f570414f080 con 0x7f570414eca0 2024-03-28T23:34:37.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.594+0000 7f5709c64640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5704151630 con 0x7f570406cf60 2024-03-28T23:34:37.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.594+0000 7f5702ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57040c1dc0 0x7f570414b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:37.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.594+0000 7f5702ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57040c1dc0 0x7f570414b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34852/0 (socket says 172.21.15.161:34852) 2024-03-28T23:34:37.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.594+0000 7f5702ffd640 1 -- 172.21.15.161:0/3192162773 learned_addr learned my addr 172.21.15.161:0/3192162773 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:37.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.595+0000 7f5703fff640 1 --2- 172.21.15.161:0/3192162773 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f570414eca0 0x7f570414b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:37.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.595+0000 7f57037fe640 1 --2- 172.21.15.161:0/3192162773 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f570406cf60 0x7f570406b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:37.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.595+0000 7f5702ffd640 1 -- 172.21.15.161:0/3192162773 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f570406cf60 msgr2=0x7f570406b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:37.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.595+0000 7f5702ffd640 1 --2- 172.21.15.161:0/3192162773 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f570406cf60 0x7f570406b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:37.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.595+0000 7f5702ffd640 1 -- 172.21.15.161:0/3192162773 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f570414eca0 msgr2=0x7f570414b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:37.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.595+0000 7f5702ffd640 1 --2- 172.21.15.161:0/3192162773 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f570414eca0 0x7f570414b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:37.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.595+0000 7f5702ffd640 1 -- 172.21.15.161:0/3192162773 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f570414c1e0 con 0x7f57040c1dc0 2024-03-28T23:34:37.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.595+0000 7f5703fff640 1 --2- 172.21.15.161:0/3192162773 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f570414eca0 0x7f570414b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:37.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.595+0000 7f5702ffd640 1 --2- 172.21.15.161:0/3192162773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57040c1dc0 0x7f570414b360 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f56f802b750 tx=0x7f56f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:37.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.595+0000 7f56ebfff640 1 -- 172.21.15.161:0/3192162773 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f8041020 con 0x7f57040c1dc0 2024-03-28T23:34:37.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.595+0000 7f56ebfff640 1 -- 172.21.15.161:0/3192162773 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56f803c870 con 0x7f57040c1dc0 2024-03-28T23:34:37.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.595+0000 7f5709c64640 1 -- 172.21.15.161:0/3192162773 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f570414c4c0 con 0x7f57040c1dc0 2024-03-28T23:34:37.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.595+0000 7f56ebfff640 1 -- 172.21.15.161:0/3192162773 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f8037040 con 0x7f57040c1dc0 2024-03-28T23:34:37.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.595+0000 7f5709c64640 1 -- 172.21.15.161:0/3192162773 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f570414ab60 con 0x7f57040c1dc0 2024-03-28T23:34:37.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.596+0000 7f5709c64640 1 -- 172.21.15.161:0/3192162773 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f570404f4b0 con 0x7f57040c1dc0 2024-03-28T23:34:37.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.597+0000 7f56ebfff640 1 -- 172.21.15.161:0/3192162773 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f56f8044020 con 0x7f57040c1dc0 2024-03-28T23:34:37.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.597+0000 7f56ebfff640 1 --2- 172.21.15.161:0/3192162773 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f56f006b4d0 0x7f56f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:37.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.597+0000 7f56ebfff640 1 -- 172.21.15.161:0/3192162773 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7f56f80abe80 con 0x7f57040c1dc0 2024-03-28T23:34:37.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.597+0000 7f57037fe640 1 --2- 172.21.15.161:0/3192162773 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f56f006b4d0 0x7f56f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:37.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.597+0000 7f57037fe640 1 --2- 172.21.15.161:0/3192162773 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f56f006b4d0 0x7f56f006d990 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f56f4009970 tx=0x7f56f4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:37.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.601+0000 7f56ebfff640 1 -- 172.21.15.161:0/3192162773 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f56f804b030 con 0x7f57040c1dc0 2024-03-28T23:34:37.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.776+0000 7f5709c64640 1 -- 172.21.15.161:0/3192162773 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f570406bc00 con 0x7f57040c1dc0 2024-03-28T23:34:37.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.777+0000 7f56ebfff640 1 -- 172.21.15.161:0/3192162773 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) ==== 74+0+16949 (secure 0 0 0) 0x7f56f807aaf0 con 0x7f57040c1dc0 2024-03-28T23:34:37.777 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:37.778 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":121,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:34:36.796301+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:34:35.025049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-28T23:34:35.765110+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:34:37.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.779+0000 7f5709c64640 1 -- 172.21.15.161:0/3192162773 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f56f006b4d0 msgr2=0x7f56f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:37.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.779+0000 7f5709c64640 1 --2- 172.21.15.161:0/3192162773 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f56f006b4d0 0x7f56f006d990 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f56f4009970 tx=0x7f56f4008040 comp rx=0 tx=0).stop 2024-03-28T23:34:37.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.779+0000 7f5709c64640 1 -- 172.21.15.161:0/3192162773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57040c1dc0 msgr2=0x7f570414b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:37.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.779+0000 7f5709c64640 1 --2- 172.21.15.161:0/3192162773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57040c1dc0 0x7f570414b360 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f56f802b750 tx=0x7f56f80046c0 comp rx=0 tx=0).stop 2024-03-28T23:34:37.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.779+0000 7f5709c64640 1 -- 172.21.15.161:0/3192162773 shutdown_connections 2024-03-28T23:34:37.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.779+0000 7f5709c64640 1 --2- 172.21.15.161:0/3192162773 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f56f006b4d0 0x7f56f006d990 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:37.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.779+0000 7f5709c64640 1 --2- 172.21.15.161:0/3192162773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57040c1dc0 0x7f570414b360 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:37.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.779+0000 7f5709c64640 1 --2- 172.21.15.161:0/3192162773 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f570414eca0 0x7f570414b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:37.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.779+0000 7f5709c64640 1 --2- 172.21.15.161:0/3192162773 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f570406cf60 0x7f570406b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:37.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.779+0000 7f5709c64640 1 -- 172.21.15.161:0/3192162773 >> 172.21.15.161:0/3192162773 conn(0x7f57040c6670 msgr2=0x7f57041495c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:37.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.779+0000 7f5709c64640 1 -- 172.21.15.161:0/3192162773 shutdown_connections 2024-03-28T23:34:37.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.779+0000 7f5709c64640 1 -- 172.21.15.161:0/3192162773 wait complete. 2024-03-28T23:34:37.793 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:34:37.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.932+0000 7fbdc8fd0640 1 Processor -- start 2024-03-28T23:34:37.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.932+0000 7fbdc8fd0640 1 -- start start 2024-03-28T23:34:37.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.933+0000 7fbdc8fd0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fbdc40c5410 0x7fbdc40c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:37.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.933+0000 7fbdc8fd0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbdc41553a0 0x7fbdc4155850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:37.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.933+0000 7fbdc8fd0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdc4158a30 con 0x7fbdc4152890 2024-03-28T23:34:37.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.933+0000 7fbdc8fd0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fbdc40c8480 con 0x7fbdc41553a0 2024-03-28T23:34:37.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.933+0000 7fbdc8fd0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fbdc40c5d90 con 0x7fbdc40c5410 2024-03-28T23:34:37.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.933+0000 7fbdc2d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbdc41553a0 0x7fbdc4155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:37.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.933+0000 7fbdc2d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbdc41553a0 0x7fbdc4155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34858/0 (socket says 172.21.15.161:34858) 2024-03-28T23:34:37.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.933+0000 7fbdc2d9d640 1 -- 172.21.15.161:0/1703145517 learned_addr learned my addr 172.21.15.161:0/1703145517 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:37.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.933+0000 7fbdc1d9b640 1 --2- 172.21.15.161:0/1703145517 >> v2:172.21.15.170:3300/0 conn(0x7fbdc40c5410 0x7fbdc40c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:37.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.934+0000 7fbdc2d9d640 1 -- 172.21.15.161:0/1703145517 >> v2:172.21.15.170:3300/0 conn(0x7fbdc40c5410 msgr2=0x7fbdc40c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:37.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.934+0000 7fbdc2d9d640 1 --2- 172.21.15.161:0/1703145517 >> v2:172.21.15.170:3300/0 conn(0x7fbdc40c5410 0x7fbdc40c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:37.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.934+0000 7fbdc2d9d640 1 -- 172.21.15.161:0/1703145517 >> v1:172.21.15.170:6789/0 conn(0x7fbdc4152890 legacy=0x7fbdc40c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:37.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.934+0000 7fbdc2d9d640 1 -- 172.21.15.161:0/1703145517 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdc4158d10 con 0x7fbdc41553a0 2024-03-28T23:34:37.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.934+0000 7fbdc2d9d640 1 --2- 172.21.15.161:0/1703145517 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbdc41553a0 0x7fbdc4155850 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fbdc40ad6e0 tx=0x7fbdbc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db23fd1842ee23db server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:37.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.934+0000 7fbdc159a640 1 -- 172.21.15.161:0/1703145517 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdbc035020 con 0x7fbdc41553a0 2024-03-28T23:34:37.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.935+0000 7fbdc159a640 1 -- 172.21.15.161:0/1703145517 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdbc031870 con 0x7fbdc41553a0 2024-03-28T23:34:37.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.935+0000 7fbdc159a640 1 -- 172.21.15.161:0/1703145517 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdbc031c40 con 0x7fbdc41553a0 2024-03-28T23:34:37.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.935+0000 7fbdc8fd0640 1 -- 172.21.15.161:0/1703145517 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbdc41553a0 msgr2=0x7fbdc4155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:37.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.935+0000 7fbdc8fd0640 1 --2- 172.21.15.161:0/1703145517 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbdc41553a0 0x7fbdc4155850 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fbdc40ad6e0 tx=0x7fbdbc02b910 comp rx=0 tx=0).stop 2024-03-28T23:34:37.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.935+0000 7fbdc8fd0640 1 -- 172.21.15.161:0/1703145517 shutdown_connections 2024-03-28T23:34:37.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.935+0000 7fbdc8fd0640 1 --2- 172.21.15.161:0/1703145517 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbdc41553a0 0x7fbdc4155850 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:37.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.935+0000 7fbdc8fd0640 1 --2- 172.21.15.161:0/1703145517 >> v2:172.21.15.170:3300/0 conn(0x7fbdc40c5410 0x7fbdc40c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:37.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.935+0000 7fbdc8fd0640 1 -- 172.21.15.161:0/1703145517 >> 172.21.15.161:0/1703145517 conn(0x7fbdc40ac620 msgr2=0x7fbdc40aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:37.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.935+0000 7fbdc8fd0640 1 -- 172.21.15.161:0/1703145517 shutdown_connections 2024-03-28T23:34:37.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.935+0000 7fbdc8fd0640 1 -- 172.21.15.161:0/1703145517 wait complete. 2024-03-28T23:34:37.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.936+0000 7fbdc8fd0640 1 Processor -- start 2024-03-28T23:34:37.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.936+0000 7fbdc8fd0640 1 -- start start 2024-03-28T23:34:37.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.937+0000 7fbdc8fd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbdc40c5410 0x7fbdc414a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:37.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.937+0000 7fbdc8fd0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbdc4152890 0x7fbdc4147e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:37.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.937+0000 7fbdc8fd0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbdc41553a0 0x7fbdc4148350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:37.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.937+0000 7fbdc8fd0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc41592d0 con 0x7fbdc4152890 2024-03-28T23:34:37.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.937+0000 7fbdc8fd0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc4159640 con 0x7fbdc41553a0 2024-03-28T23:34:37.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.937+0000 7fbdc8fd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbdc40c90a0 con 0x7fbdc40c5410 2024-03-28T23:34:37.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.937+0000 7fbdc1d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbdc4152890 0x7fbdc4147e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:37.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.937+0000 7fbdc1d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbdc4152890 0x7fbdc4147e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34872/0 (socket says 172.21.15.161:34872) 2024-03-28T23:34:37.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.937+0000 7fbdc1d9b640 1 -- 172.21.15.161:0/1254515954 learned_addr learned my addr 172.21.15.161:0/1254515954 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:37.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.937+0000 7fbdc259c640 1 --2- 172.21.15.161:0/1254515954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbdc40c5410 0x7fbdc414a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:37.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.937+0000 7fbdc2d9d640 1 --2- 172.21.15.161:0/1254515954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbdc41553a0 0x7fbdc4148350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:37.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.937+0000 7fbdc1d9b640 1 -- 172.21.15.161:0/1254515954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbdc40c5410 msgr2=0x7fbdc414a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:37.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.937+0000 7fbdc1d9b640 1 --2- 172.21.15.161:0/1254515954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbdc40c5410 0x7fbdc414a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:37.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.937+0000 7fbdc1d9b640 1 -- 172.21.15.161:0/1254515954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbdc41553a0 msgr2=0x7fbdc4148350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:37.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.937+0000 7fbdc1d9b640 1 --2- 172.21.15.161:0/1254515954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbdc41553a0 0x7fbdc4148350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:37.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.937+0000 7fbdc1d9b640 1 -- 172.21.15.161:0/1254515954 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdc40be900 con 0x7fbdc4152890 2024-03-28T23:34:37.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.938+0000 7fbdc259c640 1 --2- 172.21.15.161:0/1254515954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbdc40c5410 0x7fbdc414a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:37.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.938+0000 7fbdc2d9d640 1 --2- 172.21.15.161:0/1254515954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbdc41553a0 0x7fbdc4148350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:37.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.938+0000 7fbdc1d9b640 1 --2- 172.21.15.161:0/1254515954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbdc4152890 0x7fbdc4147e10 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fbdb800e870 tx=0x7fbdb800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:37.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.938+0000 7fbdb2ffd640 1 -- 172.21.15.161:0/1254515954 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb8017070 con 0x7fbdc4152890 2024-03-28T23:34:37.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.938+0000 7fbdb2ffd640 1 -- 172.21.15.161:0/1254515954 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdb800c940 con 0x7fbdc4152890 2024-03-28T23:34:37.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.938+0000 7fbdb2ffd640 1 -- 172.21.15.161:0/1254515954 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb800cd10 con 0x7fbdc4152890 2024-03-28T23:34:37.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.938+0000 7fbdc8fd0640 1 -- 172.21.15.161:0/1254515954 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdc40bebe0 con 0x7fbdc4152890 2024-03-28T23:34:37.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.938+0000 7fbdc8fd0640 1 -- 172.21.15.161:0/1254515954 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdc414a640 con 0x7fbdc4152890 2024-03-28T23:34:37.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.939+0000 7fbdc8fd0640 1 -- 172.21.15.161:0/1254515954 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdc40bedb0 con 0x7fbdc4152890 2024-03-28T23:34:37.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.939+0000 7fbdb2ffd640 1 -- 172.21.15.161:0/1254515954 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbdb801a070 con 0x7fbdc4152890 2024-03-28T23:34:37.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.940+0000 7fbdb2ffd640 1 --2- 172.21.15.161:0/1254515954 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbda406b4d0 0x7fbda406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:37.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.940+0000 7fbdb2ffd640 1 -- 172.21.15.161:0/1254515954 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7fbdb808a630 con 0x7fbdc4152890 2024-03-28T23:34:37.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.940+0000 7fbdc259c640 1 --2- 172.21.15.161:0/1254515954 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbda406b4d0 0x7fbda406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:37.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.940+0000 7fbdc259c640 1 --2- 172.21.15.161:0/1254515954 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbda406b4d0 0x7fbda406d990 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fbdc40c4eb0 tx=0x7fbdb4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:37.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:37.944+0000 7fbdb2ffd640 1 -- 172.21.15.161:0/1254515954 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbdc40bedb0 con 0x7fbdc4152890 2024-03-28T23:34:38.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.117+0000 7fbdc8fd0640 1 -- 172.21.15.161:0/1254515954 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbdc4151a50 con 0x7fbdc4152890 2024-03-28T23:34:38.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.118+0000 7fbdb2ffd640 1 -- 172.21.15.161:0/1254515954 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) ==== 74+0+16949 (secure 0 0 0) 0x7fbdb8059470 con 0x7fbdc4152890 2024-03-28T23:34:38.119 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:38.119 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":121,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:34:36.796301+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:34:35.025049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-28T23:34:35.765110+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:34:38.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.120+0000 7fbdc8fd0640 1 -- 172.21.15.161:0/1254515954 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbda406b4d0 msgr2=0x7fbda406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:38.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.120+0000 7fbdc8fd0640 1 --2- 172.21.15.161:0/1254515954 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbda406b4d0 0x7fbda406d990 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fbdc40c4eb0 tx=0x7fbdb4009040 comp rx=0 tx=0).stop 2024-03-28T23:34:38.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.120+0000 7fbdc8fd0640 1 -- 172.21.15.161:0/1254515954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbdc4152890 msgr2=0x7fbdc4147e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:38.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.120+0000 7fbdc8fd0640 1 --2- 172.21.15.161:0/1254515954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbdc4152890 0x7fbdc4147e10 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fbdb800e870 tx=0x7fbdb800ed30 comp rx=0 tx=0).stop 2024-03-28T23:34:38.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.120+0000 7fbdc8fd0640 1 -- 172.21.15.161:0/1254515954 shutdown_connections 2024-03-28T23:34:38.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.120+0000 7fbdc8fd0640 1 --2- 172.21.15.161:0/1254515954 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbda406b4d0 0x7fbda406d990 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:38.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.120+0000 7fbdc8fd0640 1 --2- 172.21.15.161:0/1254515954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbdc4152890 0x7fbdc4147e10 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:38.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.120+0000 7fbdc8fd0640 1 --2- 172.21.15.161:0/1254515954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbdc41553a0 0x7fbdc4148350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:38.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.120+0000 7fbdc8fd0640 1 --2- 172.21.15.161:0/1254515954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbdc40c5410 0x7fbdc414a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:38.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.120+0000 7fbdc8fd0640 1 -- 172.21.15.161:0/1254515954 >> 172.21.15.161:0/1254515954 conn(0x7fbdc40ac620 msgr2=0x7fbdc406bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:38.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.120+0000 7fbdc8fd0640 1 -- 172.21.15.161:0/1254515954 shutdown_connections 2024-03-28T23:34:38.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.120+0000 7fbdc8fd0640 1 -- 172.21.15.161:0/1254515954 wait complete. 2024-03-28T23:34:38.132 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-28T23:34:38.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.235+0000 7fb58bd3e640 1 Processor -- start 2024-03-28T23:34:38.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.236+0000 7fb58bd3e640 1 -- start start 2024-03-28T23:34:38.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.236+0000 7fb58bd3e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb584152bc0 0x7fb5840c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:38.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.236+0000 7fb58bd3e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb5840c8a80 0x7fb5840c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:38.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.236+0000 7fb58bd3e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5840c6f60 con 0x7fb5840c9420 2024-03-28T23:34:38.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.236+0000 7fb58bd3e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb5840c6120 con 0x7fb5840c8a80 2024-03-28T23:34:38.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.236+0000 7fb58bd3e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb5840c6400 con 0x7fb584152bc0 2024-03-28T23:34:38.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.236+0000 7fb5892d9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb5840c8a80 0x7fb5840c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:38.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.236+0000 7fb5892d9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb5840c8a80 0x7fb5840c8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34886/0 (socket says 172.21.15.161:34886) 2024-03-28T23:34:38.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.236+0000 7fb5892d9640 1 -- 172.21.15.161:0/1012737368 learned_addr learned my addr 172.21.15.161:0/1012737368 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:38.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.237+0000 7fb5892d9640 1 -- 172.21.15.161:0/1012737368 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb584152bc0 msgr2=0x7fb5840c0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:38.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.237+0000 7fb589ada640 1 --2- 172.21.15.161:0/1012737368 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb584152bc0 0x7fb5840c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:38.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.237+0000 7fb5892d9640 1 --2- 172.21.15.161:0/1012737368 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb584152bc0 0x7fb5840c0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:38.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.237+0000 7fb5892d9640 1 -- 172.21.15.161:0/1012737368 >> v1:172.21.15.170:6789/0 conn(0x7fb5840c9420 legacy=0x7fb5841589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:38.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.237+0000 7fb5892d9640 1 -- 172.21.15.161:0/1012737368 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5840c7240 con 0x7fb5840c8a80 2024-03-28T23:34:38.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.237+0000 7fb5892d9640 1 --2- 172.21.15.161:0/1012737368 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5840c8a80 0x7fb5840c8ee0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fb578009870 tx=0x7fb57802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd9d6c0cddd8b2c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:38.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.238+0000 7fb588ad8640 1 -- 172.21.15.161:0/1012737368 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb578035020 con 0x7fb5840c8a80 2024-03-28T23:34:38.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.238+0000 7fb588ad8640 1 -- 172.21.15.161:0/1012737368 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb578031640 con 0x7fb5840c8a80 2024-03-28T23:34:38.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.238+0000 7fb588ad8640 1 -- 172.21.15.161:0/1012737368 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57802d040 con 0x7fb5840c8a80 2024-03-28T23:34:38.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.238+0000 7fb58bd3e640 1 -- 172.21.15.161:0/1012737368 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5840c8a80 msgr2=0x7fb5840c8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:38.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.238+0000 7fb58bd3e640 1 --2- 172.21.15.161:0/1012737368 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5840c8a80 0x7fb5840c8ee0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fb578009870 tx=0x7fb57802b8f0 comp rx=0 tx=0).stop 2024-03-28T23:34:38.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.238+0000 7fb58bd3e640 1 -- 172.21.15.161:0/1012737368 shutdown_connections 2024-03-28T23:34:38.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.238+0000 7fb58bd3e640 1 --2- 172.21.15.161:0/1012737368 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5840c8a80 0x7fb5840c8ee0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:38.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.238+0000 7fb58bd3e640 1 --2- 172.21.15.161:0/1012737368 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb584152bc0 0x7fb5840c0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:38.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.238+0000 7fb58bd3e640 1 -- 172.21.15.161:0/1012737368 >> 172.21.15.161:0/1012737368 conn(0x7fb5840caa80 msgr2=0x7fb5840cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:38.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.238+0000 7fb58bd3e640 1 -- 172.21.15.161:0/1012737368 shutdown_connections 2024-03-28T23:34:38.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.238+0000 7fb58bd3e640 1 -- 172.21.15.161:0/1012737368 wait complete. 2024-03-28T23:34:38.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.239+0000 7fb58bd3e640 1 Processor -- start 2024-03-28T23:34:38.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.239+0000 7fb58bd3e640 1 -- start start 2024-03-28T23:34:38.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.240+0000 7fb58bd3e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5840c8a80 0x7fb5840ade30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:38.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.240+0000 7fb58bd3e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5840c9420 0x7fb5840ae370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:38.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.240+0000 7fb58bd3e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb584152bc0 0x7fb5841485b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:38.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.240+0000 7fb58bd3e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb584152fa0 con 0x7fb584152bc0 2024-03-28T23:34:38.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.240+0000 7fb58bd3e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb5841590c0 con 0x7fb5840c9420 2024-03-28T23:34:38.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.240+0000 7fb58bd3e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb5841494a0 con 0x7fb5840c8a80 2024-03-28T23:34:38.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.240+0000 7fb58a2db640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb584152bc0 0x7fb5841485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:38.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.240+0000 7fb58a2db640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb584152bc0 0x7fb5841485b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34890/0 (socket says 172.21.15.161:34890) 2024-03-28T23:34:38.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.240+0000 7fb58a2db640 1 -- 172.21.15.161:0/3201865081 learned_addr learned my addr 172.21.15.161:0/3201865081 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:38.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.240+0000 7fb589ada640 1 --2- 172.21.15.161:0/3201865081 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5840c8a80 0x7fb5840ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:38.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.240+0000 7fb5892d9640 1 --2- 172.21.15.161:0/3201865081 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5840c9420 0x7fb5840ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:38.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.240+0000 7fb58a2db640 1 -- 172.21.15.161:0/3201865081 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5840c8a80 msgr2=0x7fb5840ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:38.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.240+0000 7fb58a2db640 1 --2- 172.21.15.161:0/3201865081 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5840c8a80 0x7fb5840ade30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:38.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.240+0000 7fb58a2db640 1 -- 172.21.15.161:0/3201865081 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5840c9420 msgr2=0x7fb5840ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:38.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.240+0000 7fb58a2db640 1 --2- 172.21.15.161:0/3201865081 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5840c9420 0x7fb5840ae370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:38.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.240+0000 7fb58a2db640 1 -- 172.21.15.161:0/3201865081 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb584158ac0 con 0x7fb584152bc0 2024-03-28T23:34:38.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.241+0000 7fb58a2db640 1 --2- 172.21.15.161:0/3201865081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb584152bc0 0x7fb5841485b0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fb57c00c7b0 tx=0x7fb57c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:38.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.241+0000 7fb5767fc640 1 -- 172.21.15.161:0/3201865081 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57c004b60 con 0x7fb584152bc0 2024-03-28T23:34:38.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.241+0000 7fb5767fc640 1 -- 172.21.15.161:0/3201865081 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb57c004d00 con 0x7fb584152bc0 2024-03-28T23:34:38.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.241+0000 7fb58bd3e640 1 -- 172.21.15.161:0/3201865081 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb584158da0 con 0x7fb584152bc0 2024-03-28T23:34:38.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.241+0000 7fb5767fc640 1 -- 172.21.15.161:0/3201865081 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57c00aa90 con 0x7fb584152bc0 2024-03-28T23:34:38.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.241+0000 7fb58bd3e640 1 -- 172.21.15.161:0/3201865081 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb584149780 con 0x7fb584152bc0 2024-03-28T23:34:38.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.242+0000 7fb58bd3e640 1 -- 172.21.15.161:0/3201865081 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5841588b0 con 0x7fb584152bc0 2024-03-28T23:34:38.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.242+0000 7fb5767fc640 1 -- 172.21.15.161:0/3201865081 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb57c00ac30 con 0x7fb584152bc0 2024-03-28T23:34:38.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.243+0000 7fb5767fc640 1 --2- 172.21.15.161:0/3201865081 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb56806b4d0 0x7fb56806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:38.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.243+0000 7fb5767fc640 1 -- 172.21.15.161:0/3201865081 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6101+0+0 (secure 0 0 0) 0x7fb57c08a4c0 con 0x7fb584152bc0 2024-03-28T23:34:38.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.243+0000 7fb589ada640 1 --2- 172.21.15.161:0/3201865081 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb56806b4d0 0x7fb56806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:38.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.243+0000 7fb589ada640 1 --2- 172.21.15.161:0/3201865081 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb56806b4d0 0x7fb56806d990 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb5840abeb0 tx=0x7fb580008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:38.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.247+0000 7fb5767fc640 1 -- 172.21.15.161:0/3201865081 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb57c020070 con 0x7fb584152bc0 2024-03-28T23:34:38.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.455+0000 7fb58bd3e640 1 -- 172.21.15.161:0/3201865081 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fb58404f4b0 con 0x7fb584152bc0 2024-03-28T23:34:38.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.459+0000 7fb5767fc640 1 -- 172.21.15.161:0/3201865081 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v106) ==== 118+0+0 (secure 0 0 0) 0x7fb57c058bf0 con 0x7fb584152bc0 2024-03-28T23:34:38.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.461+0000 7fb58bd3e640 1 -- 172.21.15.161:0/3201865081 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb56806b4d0 msgr2=0x7fb56806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:38.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.461+0000 7fb58bd3e640 1 --2- 172.21.15.161:0/3201865081 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb56806b4d0 0x7fb56806d990 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb5840abeb0 tx=0x7fb580008040 comp rx=0 tx=0).stop 2024-03-28T23:34:38.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.461+0000 7fb58bd3e640 1 -- 172.21.15.161:0/3201865081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb584152bc0 msgr2=0x7fb5841485b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:38.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.461+0000 7fb58bd3e640 1 --2- 172.21.15.161:0/3201865081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb584152bc0 0x7fb5841485b0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fb57c00c7b0 tx=0x7fb57c00cc70 comp rx=0 tx=0).stop 2024-03-28T23:34:38.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.461+0000 7fb58bd3e640 1 -- 172.21.15.161:0/3201865081 shutdown_connections 2024-03-28T23:34:38.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.461+0000 7fb58bd3e640 1 --2- 172.21.15.161:0/3201865081 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb56806b4d0 0x7fb56806d990 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:38.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.461+0000 7fb58bd3e640 1 --2- 172.21.15.161:0/3201865081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb584152bc0 0x7fb5841485b0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:38.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.461+0000 7fb58bd3e640 1 --2- 172.21.15.161:0/3201865081 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5840c9420 0x7fb5840ae370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:38.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.461+0000 7fb58bd3e640 1 --2- 172.21.15.161:0/3201865081 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5840c8a80 0x7fb5840ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:38.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.461+0000 7fb58bd3e640 1 -- 172.21.15.161:0/3201865081 >> 172.21.15.161:0/3201865081 conn(0x7fb5840caa80 msgr2=0x7fb58414e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:38.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.461+0000 7fb58bd3e640 1 -- 172.21.15.161:0/3201865081 shutdown_connections 2024-03-28T23:34:38.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.462+0000 7fb58bd3e640 1 -- 172.21.15.161:0/3201865081 wait complete. 2024-03-28T23:34:38.476 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-28T23:34:38.476 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-28T23:34:38.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.582+0000 7f750cbdd640 1 Processor -- start 2024-03-28T23:34:38.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.582+0000 7f750cbdd640 1 -- start start 2024-03-28T23:34:38.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.583+0000 7f750cbdd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f75080c0330 0x7f75080c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:38.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.583+0000 7f750cbdd640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75080c6f60 con 0x7f7508152bc0 2024-03-28T23:34:38.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.583+0000 7f750cbdd640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75080c6120 con 0x7f75081589b0 2024-03-28T23:34:38.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.583+0000 7f750cbdd640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f75080c6400 con 0x7f75080c0330 2024-03-28T23:34:38.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.583+0000 7f74fffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f75080c0330 0x7f75080c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:38.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.583+0000 7f750659c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f7508152bc0 0x7f75080c8a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:51046/0 (socket says 172.21.15.161:51046) 2024-03-28T23:34:38.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.583+0000 7f750659c640 1 -- 172.21.15.161:0/934961062 learned_addr learned my addr 172.21.15.161:0/934961062 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:38.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.584+0000 7f74fffff640 1 -- 172.21.15.161:0/934961062 >> v1:172.21.15.170:6789/0 conn(0x7f75081589b0 legacy=0x7f7508158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:38.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.584+0000 7f74fffff640 1 -- 172.21.15.161:0/934961062 >> v1:172.21.15.161:6789/0 conn(0x7f7508152bc0 legacy=0x7f75080c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:38.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.584+0000 7f74fffff640 1 -- 172.21.15.161:0/934961062 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75080c96d0 con 0x7f75080c0330 2024-03-28T23:34:38.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.584+0000 7f7505d9b640 1 -- 172.21.15.161:0/934961062 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3360757005 0 0) 0x7f75080c6f60 con 0x7f7508152bc0 2024-03-28T23:34:38.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.584+0000 7f74fffff640 1 --2- 172.21.15.161:0/934961062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75080c0330 0x7f75080c9190 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f74f00097a0 tx=0x7f74f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4e3315e8bf36e25 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:38.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.584+0000 7f7505d9b640 1 -- 172.21.15.161:0/934961062 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f0035020 con 0x7f75080c0330 2024-03-28T23:34:38.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.584+0000 7f7505d9b640 1 -- 172.21.15.161:0/934961062 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74f0031830 con 0x7f75080c0330 2024-03-28T23:34:38.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.584+0000 7f7505d9b640 1 -- 172.21.15.161:0/934961062 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f0031c00 con 0x7f75080c0330 2024-03-28T23:34:38.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.585+0000 7f750cbdd640 1 -- 172.21.15.161:0/934961062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75080c0330 msgr2=0x7f75080c9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:38.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.585+0000 7f750cbdd640 1 --2- 172.21.15.161:0/934961062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75080c0330 0x7f75080c9190 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f74f00097a0 tx=0x7f74f002b8d0 comp rx=0 tx=0).stop 2024-03-28T23:34:38.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.585+0000 7f750cbdd640 1 -- 172.21.15.161:0/934961062 shutdown_connections 2024-03-28T23:34:38.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.585+0000 7f750cbdd640 1 --2- 172.21.15.161:0/934961062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75080c0330 0x7f75080c9190 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:38.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.585+0000 7f750cbdd640 1 -- 172.21.15.161:0/934961062 >> 172.21.15.161:0/934961062 conn(0x7f75080caa80 msgr2=0x7f75080cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:38.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.585+0000 7f750cbdd640 1 -- 172.21.15.161:0/934961062 shutdown_connections 2024-03-28T23:34:38.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.585+0000 7f750cbdd640 1 -- 172.21.15.161:0/934961062 wait complete. 2024-03-28T23:34:38.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.586+0000 7f750cbdd640 1 Processor -- start 2024-03-28T23:34:38.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.586+0000 7f750cbdd640 1 -- start start 2024-03-28T23:34:38.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.587+0000 7f750cbdd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75080c0330 0x7f750814a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:38.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.587+0000 7f750cbdd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7508152bc0 0x7f750806c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:38.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.587+0000 7f750cbdd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75081589b0 0x7f7508155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:38.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.587+0000 7f750cbdd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7508152fa0 con 0x7f75081589b0 2024-03-28T23:34:38.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.587+0000 7f750cbdd640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f75080c6bf0 con 0x7f7508152bc0 2024-03-28T23:34:38.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.587+0000 7f750cbdd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f750814a9d0 con 0x7f75080c0330 2024-03-28T23:34:38.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.587+0000 7f7506d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75081589b0 0x7f7508155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:38.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.587+0000 7f7506d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75081589b0 0x7f7508155830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34906/0 (socket says 172.21.15.161:34906) 2024-03-28T23:34:38.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.587+0000 7f7506d9d640 1 -- 172.21.15.161:0/4158102185 learned_addr learned my addr 172.21.15.161:0/4158102185 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:38.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.587+0000 7f74fffff640 1 --2- 172.21.15.161:0/4158102185 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7508152bc0 0x7f750806c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:38.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.587+0000 7f750659c640 1 --2- 172.21.15.161:0/4158102185 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75080c0330 0x7f750814a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:38.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.587+0000 7f7506d9d640 1 -- 172.21.15.161:0/4158102185 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75080c0330 msgr2=0x7f750814a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:38.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.587+0000 7f7506d9d640 1 --2- 172.21.15.161:0/4158102185 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75080c0330 0x7f750814a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:38.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.587+0000 7f7506d9d640 1 -- 172.21.15.161:0/4158102185 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7508152bc0 msgr2=0x7f750806c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:38.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.587+0000 7f7506d9d640 1 --2- 172.21.15.161:0/4158102185 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7508152bc0 0x7f750806c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:38.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.588+0000 7f7506d9d640 1 -- 172.21.15.161:0/4158102185 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f750814a1f0 con 0x7f75081589b0 2024-03-28T23:34:38.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.588+0000 7f750659c640 1 --2- 172.21.15.161:0/4158102185 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75080c0330 0x7f750814a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:38.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.588+0000 7f7506d9d640 1 --2- 172.21.15.161:0/4158102185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75081589b0 0x7f7508155830 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f750000b970 tx=0x7f750000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:38.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.588+0000 7f74feffd640 1 -- 172.21.15.161:0/4158102185 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7500019070 con 0x7f75081589b0 2024-03-28T23:34:38.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.588+0000 7f74feffd640 1 -- 172.21.15.161:0/4158102185 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f750000cae0 con 0x7f75081589b0 2024-03-28T23:34:38.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.588+0000 7f750cbdd640 1 -- 172.21.15.161:0/4158102185 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75081eb630 con 0x7f75081589b0 2024-03-28T23:34:38.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.588+0000 7f74feffd640 1 -- 172.21.15.161:0/4158102185 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7500014440 con 0x7f75081589b0 2024-03-28T23:34:38.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.588+0000 7f750cbdd640 1 -- 172.21.15.161:0/4158102185 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7508156600 con 0x7f75081589b0 2024-03-28T23:34:38.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.589+0000 7f750cbdd640 1 -- 172.21.15.161:0/4158102185 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7508158e30 con 0x7f75081589b0 2024-03-28T23:34:38.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.589+0000 7f74feffd640 1 -- 172.21.15.161:0/4158102185 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f750001c070 con 0x7f75081589b0 2024-03-28T23:34:38.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.590+0000 7f74feffd640 1 --2- 172.21.15.161:0/4158102185 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f74e806b4d0 0x7f74e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:38.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.590+0000 7f74feffd640 1 -- 172.21.15.161:0/4158102185 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6101+0+0 (secure 0 0 0) 0x7f7500089de0 con 0x7f75081589b0 2024-03-28T23:34:38.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.590+0000 7f750659c640 1 --2- 172.21.15.161:0/4158102185 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f74e806b4d0 0x7f74e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:38.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.590+0000 7f750659c640 1 --2- 172.21.15.161:0/4158102185 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f74e806b4d0 0x7f74e806d990 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f74f8000f60 tx=0x7f74f8007c30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:38.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.594+0000 7f74feffd640 1 -- 172.21.15.161:0/4158102185 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7508158e30 con 0x7f75081589b0 2024-03-28T23:34:38.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:38.812+0000 7f750cbdd640 1 -- 172.21.15.161:0/4158102185 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f750804f4b0 con 0x7f75081589b0 2024-03-28T23:34:39.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.466+0000 7f74feffd640 1 -- 172.21.15.161:0/4158102185 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v107) ==== 122+0+0 (secure 0 0 0) 0x7f7500058510 con 0x7f75081589b0 2024-03-28T23:34:39.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.469+0000 7f750cbdd640 1 -- 172.21.15.161:0/4158102185 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f74e806b4d0 msgr2=0x7f74e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:39.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.469+0000 7f750cbdd640 1 --2- 172.21.15.161:0/4158102185 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f74e806b4d0 0x7f74e806d990 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f74f8000f60 tx=0x7f74f8007c30 comp rx=0 tx=0).stop 2024-03-28T23:34:39.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.469+0000 7f750cbdd640 1 -- 172.21.15.161:0/4158102185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75081589b0 msgr2=0x7f7508155830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:39.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.469+0000 7f750cbdd640 1 --2- 172.21.15.161:0/4158102185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75081589b0 0x7f7508155830 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f750000b970 tx=0x7f750000be30 comp rx=0 tx=0).stop 2024-03-28T23:34:39.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.469+0000 7f750cbdd640 1 -- 172.21.15.161:0/4158102185 shutdown_connections 2024-03-28T23:34:39.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.469+0000 7f750cbdd640 1 --2- 172.21.15.161:0/4158102185 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f74e806b4d0 0x7f74e806d990 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:39.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.469+0000 7f750cbdd640 1 --2- 172.21.15.161:0/4158102185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75081589b0 0x7f7508155830 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:39.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.469+0000 7f750cbdd640 1 --2- 172.21.15.161:0/4158102185 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7508152bc0 0x7f750806c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:39.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.469+0000 7f750cbdd640 1 --2- 172.21.15.161:0/4158102185 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75080c0330 0x7f750814a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:39.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.469+0000 7f750cbdd640 1 -- 172.21.15.161:0/4158102185 >> 172.21.15.161:0/4158102185 conn(0x7f75080caa80 msgr2=0x7f750814d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:39.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.469+0000 7f750cbdd640 1 -- 172.21.15.161:0/4158102185 shutdown_connections 2024-03-28T23:34:39.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.469+0000 7f750cbdd640 1 -- 172.21.15.161:0/4158102185 wait complete. 2024-03-28T23:34:39.484 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-28T23:34:39.484 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:34:39.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.588+0000 7f76c152c640 1 Processor -- start 2024-03-28T23:34:39.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.588+0000 7f76c152c640 1 -- start start 2024-03-28T23:34:39.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.589+0000 7f76c152c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76bc06ca20 0x7f76bc06ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:39.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.589+0000 7f76c152c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f76bc0a98c0 0x7f76bc0a9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:39.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.589+0000 7f76c152c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76bc151850 con 0x7f76bc14eca0 2024-03-28T23:34:39.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.589+0000 7f76c152c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f76bc152460 con 0x7f76bc0a98c0 2024-03-28T23:34:39.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.589+0000 7f76c152c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f76bc151b30 con 0x7f76bc06ca20 2024-03-28T23:34:39.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.589+0000 7f76bb7fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f76bc0a98c0 0x7f76bc0a9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:39.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.589+0000 7f76ba7fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76bc06ca20 0x7f76bc06ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:39.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.589+0000 7f76baffd640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f76bc14eca0 0x7f76bc0c1880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:43130/0 (socket says 172.21.15.161:43130) 2024-03-28T23:34:39.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.589+0000 7f76baffd640 1 -- 172.21.15.161:0/3737631084 learned_addr learned my addr 172.21.15.161:0/3737631084 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:39.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.590+0000 7f76b9ffb640 1 -- 172.21.15.161:0/3737631084 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 884075539 0 0) 0x7f76bc151850 con 0x7f76bc14eca0 2024-03-28T23:34:39.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.590+0000 7f76b9ffb640 1 -- 172.21.15.161:0/3737631084 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76a8003520 con 0x7f76bc14eca0 2024-03-28T23:34:39.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.590+0000 7f76bb7fe640 1 -- 172.21.15.161:0/3737631084 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76bc06ca20 msgr2=0x7f76bc06ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:39.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.590+0000 7f76bb7fe640 1 --2- 172.21.15.161:0/3737631084 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76bc06ca20 0x7f76bc06ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:39.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.590+0000 7f76bb7fe640 1 -- 172.21.15.161:0/3737631084 >> v1:172.21.15.170:6789/0 conn(0x7f76bc14eca0 legacy=0x7f76bc0c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:39.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.590+0000 7f76bb7fe640 1 -- 172.21.15.161:0/3737631084 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76bc151e10 con 0x7f76bc0a98c0 2024-03-28T23:34:39.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.590+0000 7f76ba7fc640 1 --2- 172.21.15.161:0/3737631084 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76bc06ca20 0x7f76bc06ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:34:39.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.591+0000 7f76bb7fe640 1 --2- 172.21.15.161:0/3737631084 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76bc0a98c0 0x7f76bc0a9d20 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f76b4009d90 tx=0x7f76b402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=322bdf9ff41e3b03 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:39.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.591+0000 7f76b9ffb640 1 -- 172.21.15.161:0/3737631084 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b4039070 con 0x7f76bc0a98c0 2024-03-28T23:34:39.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.591+0000 7f76b9ffb640 1 -- 172.21.15.161:0/3737631084 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76b402cbc0 con 0x7f76bc0a98c0 2024-03-28T23:34:39.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.591+0000 7f76b9ffb640 1 -- 172.21.15.161:0/3737631084 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b4034440 con 0x7f76bc0a98c0 2024-03-28T23:34:39.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.591+0000 7f76c152c640 1 -- 172.21.15.161:0/3737631084 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76bc0a98c0 msgr2=0x7f76bc0a9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:39.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.591+0000 7f76c152c640 1 --2- 172.21.15.161:0/3737631084 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76bc0a98c0 0x7f76bc0a9d20 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f76b4009d90 tx=0x7f76b402be10 comp rx=0 tx=0).stop 2024-03-28T23:34:39.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.591+0000 7f76c152c640 1 -- 172.21.15.161:0/3737631084 shutdown_connections 2024-03-28T23:34:39.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.591+0000 7f76c152c640 1 --2- 172.21.15.161:0/3737631084 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76bc0a98c0 0x7f76bc0a9d20 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:39.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.591+0000 7f76c152c640 1 --2- 172.21.15.161:0/3737631084 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76bc06ca20 0x7f76bc06ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:39.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.591+0000 7f76c152c640 1 -- 172.21.15.161:0/3737631084 >> 172.21.15.161:0/3737631084 conn(0x7f76bc0c6670 msgr2=0x7f76bc0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:39.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.591+0000 7f76c152c640 1 -- 172.21.15.161:0/3737631084 shutdown_connections 2024-03-28T23:34:39.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.591+0000 7f76c152c640 1 -- 172.21.15.161:0/3737631084 wait complete. 2024-03-28T23:34:39.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.593+0000 7f76c152c640 1 Processor -- start 2024-03-28T23:34:39.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.593+0000 7f76c152c640 1 -- start start 2024-03-28T23:34:39.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.593+0000 7f76c152c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76bc06ca20 0x7f76bc143ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:39.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.593+0000 7f76c152c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76bc0a98c0 0x7f76bc13f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:39.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.593+0000 7f76c152c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76bc14eca0 0x7f76bc13fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:39.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.593+0000 7f76c152c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f76bc1514b0 con 0x7f76bc0a98c0 2024-03-28T23:34:39.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.593+0000 7f76c152c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f76bc14f080 con 0x7f76bc06ca20 2024-03-28T23:34:39.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.593+0000 7f76c152c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f76bc151630 con 0x7f76bc14eca0 2024-03-28T23:34:39.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.593+0000 7f76ba7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76bc0a98c0 0x7f76bc13f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:39.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.593+0000 7f76ba7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76bc0a98c0 0x7f76bc13f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34914/0 (socket says 172.21.15.161:34914) 2024-03-28T23:34:39.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.593+0000 7f76ba7fc640 1 -- 172.21.15.161:0/2588094336 learned_addr learned my addr 172.21.15.161:0/2588094336 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:39.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.593+0000 7f76baffd640 1 --2- 172.21.15.161:0/2588094336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76bc06ca20 0x7f76bc143ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:39.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.593+0000 7f76bb7fe640 1 --2- 172.21.15.161:0/2588094336 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76bc14eca0 0x7f76bc13fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:39.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.594+0000 7f76ba7fc640 1 -- 172.21.15.161:0/2588094336 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76bc14eca0 msgr2=0x7f76bc13fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:39.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.594+0000 7f76ba7fc640 1 --2- 172.21.15.161:0/2588094336 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76bc14eca0 0x7f76bc13fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:39.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.594+0000 7f76ba7fc640 1 -- 172.21.15.161:0/2588094336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76bc06ca20 msgr2=0x7f76bc143ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:39.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.594+0000 7f76ba7fc640 1 --2- 172.21.15.161:0/2588094336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76bc06ca20 0x7f76bc143ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:39.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.594+0000 7f76ba7fc640 1 -- 172.21.15.161:0/2588094336 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76bc155430 con 0x7f76bc0a98c0 2024-03-28T23:34:39.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.594+0000 7f76ba7fc640 1 --2- 172.21.15.161:0/2588094336 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76bc0a98c0 0x7f76bc13f830 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f76ac00ec10 tx=0x7f76ac00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:39.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.594+0000 7f76b37fe640 1 -- 172.21.15.161:0/2588094336 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ac017070 con 0x7f76bc0a98c0 2024-03-28T23:34:39.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.594+0000 7f76b37fe640 1 -- 172.21.15.161:0/2588094336 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76ac00cc30 con 0x7f76bc0a98c0 2024-03-28T23:34:39.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.594+0000 7f76c152c640 1 -- 172.21.15.161:0/2588094336 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76bc154eb0 con 0x7f76bc0a98c0 2024-03-28T23:34:39.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.594+0000 7f76b37fe640 1 -- 172.21.15.161:0/2588094336 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ac013470 con 0x7f76bc0a98c0 2024-03-28T23:34:39.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.594+0000 7f76c152c640 1 -- 172.21.15.161:0/2588094336 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76bc1455f0 con 0x7f76bc0a98c0 2024-03-28T23:34:39.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.595+0000 7f76c152c640 1 -- 172.21.15.161:0/2588094336 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76bc04f4b0 con 0x7f76bc0a98c0 2024-03-28T23:34:39.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.596+0000 7f76b37fe640 1 -- 172.21.15.161:0/2588094336 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f76ac01a070 con 0x7f76bc0a98c0 2024-03-28T23:34:39.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.596+0000 7f76b37fe640 1 --2- 172.21.15.161:0/2588094336 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f769c06b500 0x7f769c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:39.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.596+0000 7f76b37fe640 1 -- 172.21.15.161:0/2588094336 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6101+0+0 (secure 0 0 0) 0x7f76ac08a1f0 con 0x7f76bc0a98c0 2024-03-28T23:34:39.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.596+0000 7f76baffd640 1 --2- 172.21.15.161:0/2588094336 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f769c06b500 0x7f769c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:39.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.597+0000 7f76baffd640 1 --2- 172.21.15.161:0/2588094336 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f769c06b500 0x7f769c06d9c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f76a4005880 tx=0x7f76a4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:39.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.605+0000 7f76b37fe640 1 -- 172.21.15.161:0/2588094336 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f76ac0588a0 con 0x7f76bc0a98c0 2024-03-28T23:34:39.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.820+0000 7f76c152c640 1 -- 172.21.15.161:0/2588094336 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f76bc0c62b0 con 0x7f76bc0a98c0 2024-03-28T23:34:39.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.821+0000 7f76b37fe640 1 -- 172.21.15.161:0/2588094336 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+5198 (secure 0 0 0) 0x7f76ac05c940 con 0x7f76bc0a98c0 2024-03-28T23:34:39.821 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:39.821 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":107,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7318,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3734750343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3734750343},{"type":"v1","addr":"172.21.15.170:6835","nonce":3734750343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/4140438862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":4140438862},{"type":"v1","addr":"172.21.15.170:6834","nonce":4140438862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/878057589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":878057589},{"type":"v1","addr":"172.21.15.161:6835","nonce":878057589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:34:36.796608+0000","modified":"2024-03-28T23:34:38.813377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7974},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7974":{"gid":7974,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6837/3432367324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":3432367324},{"type":"v1","addr":"172.21.15.161:6837","nonce":3432367324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7974,"qdb_cluster":[7974]},"id":9}]} 2024-03-28T23:34:39.822 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 107 2024-03-28T23:34:39.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.822+0000 7f76c152c640 1 -- 172.21.15.161:0/2588094336 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f769c06b500 msgr2=0x7f769c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:39.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.822+0000 7f76c152c640 1 --2- 172.21.15.161:0/2588094336 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f769c06b500 0x7f769c06d9c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f76a4005880 tx=0x7f76a4009040 comp rx=0 tx=0).stop 2024-03-28T23:34:39.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.822+0000 7f76c152c640 1 -- 172.21.15.161:0/2588094336 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76bc0a98c0 msgr2=0x7f76bc13f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:39.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.822+0000 7f76c152c640 1 --2- 172.21.15.161:0/2588094336 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76bc0a98c0 0x7f76bc13f830 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f76ac00ec10 tx=0x7f76ac00c520 comp rx=0 tx=0).stop 2024-03-28T23:34:39.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.823+0000 7f76c152c640 1 -- 172.21.15.161:0/2588094336 shutdown_connections 2024-03-28T23:34:39.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.823+0000 7f76c152c640 1 --2- 172.21.15.161:0/2588094336 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f769c06b500 0x7f769c06d9c0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:39.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.823+0000 7f76c152c640 1 --2- 172.21.15.161:0/2588094336 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76bc0a98c0 0x7f76bc13f830 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:39.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.823+0000 7f76c152c640 1 --2- 172.21.15.161:0/2588094336 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76bc14eca0 0x7f76bc13fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:39.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.823+0000 7f76c152c640 1 --2- 172.21.15.161:0/2588094336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76bc06ca20 0x7f76bc143ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:39.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.823+0000 7f76c152c640 1 -- 172.21.15.161:0/2588094336 >> 172.21.15.161:0/2588094336 conn(0x7f76bc0c6670 msgr2=0x7f76bc150170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:39.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.823+0000 7f76c152c640 1 -- 172.21.15.161:0/2588094336 shutdown_connections 2024-03-28T23:34:39.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.823+0000 7f76c152c640 1 -- 172.21.15.161:0/2588094336 wait complete. 2024-03-28T23:34:39.838 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:34:39.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.940+0000 7fdc14b81640 1 Processor -- start 2024-03-28T23:34:39.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.940+0000 7fdc14b81640 1 -- start start 2024-03-28T23:34:39.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.940+0000 7fdc14b81640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fdc100c3390 0x7fdc100c37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:39.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.940+0000 7fdc14b81640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdc101589d0 0x7fdc10158e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:39.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.940+0000 7fdc14b81640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc10155480 con 0x7fdc1006dd60 2024-03-28T23:34:39.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.941+0000 7fdc14b81640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fdc10155760 con 0x7fdc100c3390 2024-03-28T23:34:39.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.941+0000 7fdc14b81640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdc10155a40 con 0x7fdc101589d0 2024-03-28T23:34:39.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.941+0000 7fdc0e59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fdc1006dd60 0x7fdc100a7d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:51056/0 (socket says 172.21.15.161:51056) 2024-03-28T23:34:39.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.941+0000 7fdc0e59c640 1 -- 172.21.15.161:0/3223879011 learned_addr learned my addr 172.21.15.161:0/3223879011 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:39.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.941+0000 7fdc0d59a640 1 -- 172.21.15.161:0/3223879011 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3836846493 0 0) 0x7fdc10155480 con 0x7fdc1006dd60 2024-03-28T23:34:39.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.941+0000 7fdc0ed9d640 1 --2- 172.21.15.161:0/3223879011 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdc101589d0 0x7fdc10158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:39.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.941+0000 7fdc0d59a640 1 -- 172.21.15.161:0/3223879011 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbf8003520 con 0x7fdc1006dd60 2024-03-28T23:34:39.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.941+0000 7fdc0dd9b640 1 --2- 172.21.15.161:0/3223879011 >> v2:172.21.15.170:3300/0 conn(0x7fdc100c3390 0x7fdc100c37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:39.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.941+0000 7fdc0d59a640 1 -- 172.21.15.161:0/3223879011 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 259568045 0 0) 0x7fdbf8003520 con 0x7fdc1006dd60 2024-03-28T23:34:39.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.941+0000 7fdc0d59a640 1 -- 172.21.15.161:0/3223879011 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdc101589d0 msgr2=0x7fdc10158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:39.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.941+0000 7fdc0d59a640 1 --2- 172.21.15.161:0/3223879011 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdc101589d0 0x7fdc10158e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:39.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.941+0000 7fdc0d59a640 1 -- 172.21.15.161:0/3223879011 >> v2:172.21.15.170:3300/0 conn(0x7fdc100c3390 msgr2=0x7fdc100c37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:39.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.941+0000 7fdc0d59a640 1 --2- 172.21.15.161:0/3223879011 >> v2:172.21.15.170:3300/0 conn(0x7fdc100c3390 0x7fdc100c37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:39.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.941+0000 7fdc0d59a640 1 -- 172.21.15.161:0/3223879011 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc10155d20 con 0x7fdc1006dd60 2024-03-28T23:34:39.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.941+0000 7fdc0d59a640 1 -- 172.21.15.161:0/3223879011 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fdc00001670 con 0x7fdc1006dd60 2024-03-28T23:34:39.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.941+0000 7fdc0d59a640 1 -- 172.21.15.161:0/3223879011 >> v1:172.21.15.161:6789/0 conn(0x7fdc1006dd60 legacy=0x7fdc100a7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:39.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.941+0000 7fdc0d59a640 1 --2- 172.21.15.161:0/3223879011 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdc101589d0 0x7fdbf8003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:39.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.941+0000 7fdc0d59a640 1 --2- 172.21.15.161:0/3223879011 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdbf8004010 0x7fdbf8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:39.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.941+0000 7fdc0d59a640 1 --2- 172.21.15.161:0/3223879011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdbf8004d40 0x7fdbf80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:39.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.941+0000 7fdc0d59a640 1 -- 172.21.15.161:0/3223879011 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdc10155760 con 0x7fdbf8004010 2024-03-28T23:34:39.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.941+0000 7fdc0d59a640 1 -- 172.21.15.161:0/3223879011 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdc10155a40 con 0x7fdbf8004d40 2024-03-28T23:34:39.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.941+0000 7fdc0d59a640 1 -- 172.21.15.161:0/3223879011 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdbf800bb80 con 0x7fdc101589d0 2024-03-28T23:34:39.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.941+0000 7fdc0dd9b640 1 --2- 172.21.15.161:0/3223879011 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdbf8004010 0x7fdbf8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:39.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.942+0000 7fdc0e59c640 1 --2- 172.21.15.161:0/3223879011 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdc101589d0 0x7fdbf8003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:39.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.942+0000 7fdc0ed9d640 1 --2- 172.21.15.161:0/3223879011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdbf8004d40 0x7fdbf80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:39.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.942+0000 7fdc0dd9b640 1 -- 172.21.15.161:0/3223879011 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdc101589d0 msgr2=0x7fdbf8003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:39.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.942+0000 7fdc0dd9b640 1 --2- 172.21.15.161:0/3223879011 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdc101589d0 0x7fdbf8003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:39.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.942+0000 7fdc0dd9b640 1 -- 172.21.15.161:0/3223879011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdbf8004d40 msgr2=0x7fdbf80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:39.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.942+0000 7fdc0dd9b640 1 --2- 172.21.15.161:0/3223879011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdbf8004d40 0x7fdbf80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:39.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.942+0000 7fdc0dd9b640 1 -- 172.21.15.161:0/3223879011 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbf8003520 con 0x7fdbf8004010 2024-03-28T23:34:39.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.943+0000 7fdc0dd9b640 1 --2- 172.21.15.161:0/3223879011 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdbf8004010 0x7fdbf8004800 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fdc04009ac0 tx=0x7fdbf8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26408808144414d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:39.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.943+0000 7fdc0d59a640 1 -- 172.21.15.161:0/3223879011 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc04039070 con 0x7fdbf8004010 2024-03-28T23:34:39.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.943+0000 7fdc0d59a640 1 -- 172.21.15.161:0/3223879011 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc0402c710 con 0x7fdbf8004010 2024-03-28T23:34:39.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.943+0000 7fdc0d59a640 1 -- 172.21.15.161:0/3223879011 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc04009690 con 0x7fdbf8004010 2024-03-28T23:34:39.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.943+0000 7fdc14b81640 1 -- 172.21.15.161:0/3223879011 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdbf8004010 msgr2=0x7fdbf8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:39.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.943+0000 7fdc14b81640 1 --2- 172.21.15.161:0/3223879011 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdbf8004010 0x7fdbf8004800 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fdc04009ac0 tx=0x7fdbf8005830 comp rx=0 tx=0).stop 2024-03-28T23:34:39.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.943+0000 7fdc0ed9d640 1 -- 172.21.15.161:0/3223879011 reap_dead start 2024-03-28T23:34:39.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.943+0000 7fdc14b81640 1 -- 172.21.15.161:0/3223879011 shutdown_connections 2024-03-28T23:34:39.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.943+0000 7fdc14b81640 1 -- 172.21.15.161:0/3223879011 >> 172.21.15.161:0/3223879011 conn(0x7fdc100aa740 msgr2=0x7fdc100aab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:39.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.943+0000 7fdc14b81640 1 -- 172.21.15.161:0/3223879011 shutdown_connections 2024-03-28T23:34:39.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.943+0000 7fdc14b81640 1 -- 172.21.15.161:0/3223879011 wait complete. 2024-03-28T23:34:39.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.945+0000 7fdc14b81640 1 Processor -- start 2024-03-28T23:34:39.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.945+0000 7fdc14b81640 1 -- start start 2024-03-28T23:34:39.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.945+0000 7fdc14b81640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdbf8004010 0x7fdc100ae250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:39.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.945+0000 7fdc14b81640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc101e2480 0x7fdc101df660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:39.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.945+0000 7fdc14b81640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc101e1ac0 0x7fdc101dfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:39.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.945+0000 7fdc14b81640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdc100c6ab0 con 0x7fdc101e2480 2024-03-28T23:34:39.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.945+0000 7fdc14b81640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdc100a8410 con 0x7fdc101e1ac0 2024-03-28T23:34:39.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.945+0000 7fdc14b81640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdc10158e10 con 0x7fdbf8004010 2024-03-28T23:34:39.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.945+0000 7fdc0dd9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc101e2480 0x7fdc101df660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:39.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.945+0000 7fdc0dd9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc101e2480 0x7fdc101df660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34932/0 (socket says 172.21.15.161:34932) 2024-03-28T23:34:39.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.945+0000 7fdc0dd9b640 1 -- 172.21.15.161:0/1053083544 learned_addr learned my addr 172.21.15.161:0/1053083544 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:39.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.946+0000 7fdc0e59c640 1 --2- 172.21.15.161:0/1053083544 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdbf8004010 0x7fdc100ae250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:39.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.946+0000 7fdc0ed9d640 1 --2- 172.21.15.161:0/1053083544 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc101e1ac0 0x7fdc101dfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:39.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.946+0000 7fdc0dd9b640 1 -- 172.21.15.161:0/1053083544 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdbf8004010 msgr2=0x7fdc100ae250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:39.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.946+0000 7fdc0dd9b640 1 --2- 172.21.15.161:0/1053083544 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdbf8004010 0x7fdc100ae250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:39.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.946+0000 7fdc0dd9b640 1 -- 172.21.15.161:0/1053083544 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc101e1ac0 msgr2=0x7fdc101dfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:39.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.946+0000 7fdc0dd9b640 1 --2- 172.21.15.161:0/1053083544 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc101e1ac0 0x7fdc101dfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:39.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.946+0000 7fdc0dd9b640 1 -- 172.21.15.161:0/1053083544 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc1014e8f0 con 0x7fdc101e2480 2024-03-28T23:34:39.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.946+0000 7fdc0dd9b640 1 --2- 172.21.15.161:0/1053083544 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc101e2480 0x7fdc101df660 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fdc04009ac0 tx=0x7fdc04004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:39.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.946+0000 7fdbfeffd640 1 -- 172.21.15.161:0/1053083544 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc04039050 con 0x7fdc101e2480 2024-03-28T23:34:39.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.946+0000 7fdbfeffd640 1 -- 172.21.15.161:0/1053083544 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc04045070 con 0x7fdc101e2480 2024-03-28T23:34:39.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.946+0000 7fdc14b81640 1 -- 172.21.15.161:0/1053083544 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc101dee60 con 0x7fdc101e2480 2024-03-28T23:34:39.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.946+0000 7fdbfeffd640 1 -- 172.21.15.161:0/1053083544 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc04003a50 con 0x7fdc101e2480 2024-03-28T23:34:39.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.946+0000 7fdc14b81640 1 -- 172.21.15.161:0/1053083544 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc100ad3c0 con 0x7fdc101e2480 2024-03-28T23:34:39.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.947+0000 7fdc14b81640 1 -- 172.21.15.161:0/1053083544 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc1004f4b0 con 0x7fdc101e2480 2024-03-28T23:34:39.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.948+0000 7fdbfeffd640 1 -- 172.21.15.161:0/1053083544 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdc04003bf0 con 0x7fdc101e2480 2024-03-28T23:34:39.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.948+0000 7fdbfeffd640 1 --2- 172.21.15.161:0/1053083544 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdbf006b4d0 0x7fdbf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:39.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.948+0000 7fdbfeffd640 1 -- 172.21.15.161:0/1053083544 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6101+0+0 (secure 0 0 0) 0x7fdc040abf40 con 0x7fdc101e2480 2024-03-28T23:34:39.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.948+0000 7fdc0e59c640 1 --2- 172.21.15.161:0/1053083544 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdbf006b4d0 0x7fdbf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:39.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.949+0000 7fdc0e59c640 1 --2- 172.21.15.161:0/1053083544 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdbf006b4d0 0x7fdbf006d990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fdc00000fb0 tx=0x7fdc00009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:39.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:39.952+0000 7fdbfeffd640 1 -- 172.21.15.161:0/1053083544 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdc0404e020 con 0x7fdc101e2480 2024-03-28T23:34:40.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.131+0000 7fdc14b81640 1 -- 172.21.15.161:0/1053083544 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdc1006e250 con 0x7fdc101e2480 2024-03-28T23:34:40.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.133+0000 7fdbfeffd640 1 -- 172.21.15.161:0/1053083544 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) ==== 74+0+16936 (secure 0 0 0) 0x7fdc0407aea0 con 0x7fdc101e2480 2024-03-28T23:34:40.133 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:40.134 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":122,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:34:38.459426+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:34:35.025049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-28T23:34:35.765110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"122","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:34:40.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.135+0000 7fdc14b81640 1 -- 172.21.15.161:0/1053083544 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdbf006b4d0 msgr2=0x7fdbf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:40.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.135+0000 7fdc14b81640 1 --2- 172.21.15.161:0/1053083544 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdbf006b4d0 0x7fdbf006d990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fdc00000fb0 tx=0x7fdc00009040 comp rx=0 tx=0).stop 2024-03-28T23:34:40.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.135+0000 7fdc14b81640 1 -- 172.21.15.161:0/1053083544 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc101e2480 msgr2=0x7fdc101df660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:40.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.135+0000 7fdc14b81640 1 --2- 172.21.15.161:0/1053083544 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc101e2480 0x7fdc101df660 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fdc04009ac0 tx=0x7fdc04004730 comp rx=0 tx=0).stop 2024-03-28T23:34:40.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.135+0000 7fdc14b81640 1 -- 172.21.15.161:0/1053083544 shutdown_connections 2024-03-28T23:34:40.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.135+0000 7fdc14b81640 1 --2- 172.21.15.161:0/1053083544 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdbf006b4d0 0x7fdbf006d990 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.135+0000 7fdc14b81640 1 --2- 172.21.15.161:0/1053083544 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc101e2480 0x7fdc101df660 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.135+0000 7fdc14b81640 1 --2- 172.21.15.161:0/1053083544 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc101e1ac0 0x7fdc101dfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.135+0000 7fdc14b81640 1 --2- 172.21.15.161:0/1053083544 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdbf8004010 0x7fdc100ae250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.135+0000 7fdc14b81640 1 -- 172.21.15.161:0/1053083544 >> 172.21.15.161:0/1053083544 conn(0x7fdc100aa740 msgr2=0x7fdc100c1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:40.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.135+0000 7fdc14b81640 1 -- 172.21.15.161:0/1053083544 shutdown_connections 2024-03-28T23:34:40.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.135+0000 7fdc14b81640 1 -- 172.21.15.161:0/1053083544 wait complete. 2024-03-28T23:34:40.147 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T23:34:40.147 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:34:40.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.249+0000 7f00ded19640 1 Processor -- start 2024-03-28T23:34:40.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.249+0000 7f00ded19640 1 -- start start 2024-03-28T23:34:40.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.250+0000 7f00ded19640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f00d8152890 0x7f00d80c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:40.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.250+0000 7f00ded19640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f00d80c8ed0 0x7f00d80c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:40.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.250+0000 7f00ded19640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00d80c5950 0x7f00d81553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:40.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.250+0000 7f00ded19640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f00d80c8480 con 0x7f00d80c8ed0 2024-03-28T23:34:40.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.250+0000 7f00ded19640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f00d8159640 con 0x7f00d8152890 2024-03-28T23:34:40.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.250+0000 7f00ded19640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f00d8158a30 con 0x7f00d80c5950 2024-03-28T23:34:40.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.250+0000 7f00d7fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f00d80c8ed0 0x7f00d80c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:40.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.250+0000 7f00d7fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f00d80c8ed0 0x7f00d80c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34934/0 (socket says 172.21.15.161:34934) 2024-03-28T23:34:40.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.250+0000 7f00d7fff640 1 -- 172.21.15.161:0/2723831249 learned_addr learned my addr 172.21.15.161:0/2723831249 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:40.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.251+0000 7f00dd2b6640 1 --2- 172.21.15.161:0/2723831249 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00d80c5950 0x7f00d81553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:40.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.251+0000 7f00dcab5640 1 --2- 172.21.15.161:0/2723831249 >> v2:172.21.15.170:3300/0 conn(0x7f00d8152890 0x7f00d80c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:40.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.251+0000 7f00d7fff640 1 -- 172.21.15.161:0/2723831249 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00d80c5950 msgr2=0x7f00d81553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:40.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.251+0000 7f00d7fff640 1 --2- 172.21.15.161:0/2723831249 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00d80c5950 0x7f00d81553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.251+0000 7f00d7fff640 1 -- 172.21.15.161:0/2723831249 >> v2:172.21.15.170:3300/0 conn(0x7f00d8152890 msgr2=0x7f00d80c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:40.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.251+0000 7f00d7fff640 1 --2- 172.21.15.161:0/2723831249 >> v2:172.21.15.170:3300/0 conn(0x7f00d8152890 0x7f00d80c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.251+0000 7f00d7fff640 1 -- 172.21.15.161:0/2723831249 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d8158d10 con 0x7f00d80c8ed0 2024-03-28T23:34:40.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.251+0000 7f00d7fff640 1 --2- 172.21.15.161:0/2723831249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00d80c8ed0 0x7f00d80c5410 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f00d808cc80 tx=0x7f00c402b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0e0e76d577302d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:40.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.252+0000 7f00d77fe640 1 -- 172.21.15.161:0/2723831249 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c4038070 con 0x7f00d80c8ed0 2024-03-28T23:34:40.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.252+0000 7f00d77fe640 1 -- 172.21.15.161:0/2723831249 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00c4004030 con 0x7f00d80c8ed0 2024-03-28T23:34:40.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.252+0000 7f00d77fe640 1 -- 172.21.15.161:0/2723831249 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c4004460 con 0x7f00d80c8ed0 2024-03-28T23:34:40.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.252+0000 7f00ded19640 1 -- 172.21.15.161:0/2723831249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00d80c8ed0 msgr2=0x7f00d80c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:40.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.252+0000 7f00ded19640 1 --2- 172.21.15.161:0/2723831249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00d80c8ed0 0x7f00d80c5410 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f00d808cc80 tx=0x7f00c402b7c0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.252+0000 7f00ded19640 1 -- 172.21.15.161:0/2723831249 shutdown_connections 2024-03-28T23:34:40.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.252+0000 7f00ded19640 1 --2- 172.21.15.161:0/2723831249 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00d80c5950 0x7f00d81553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.252+0000 7f00ded19640 1 --2- 172.21.15.161:0/2723831249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00d80c8ed0 0x7f00d80c5410 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.252+0000 7f00ded19640 1 --2- 172.21.15.161:0/2723831249 >> v2:172.21.15.170:3300/0 conn(0x7f00d8152890 0x7f00d80c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.252+0000 7f00ded19640 1 -- 172.21.15.161:0/2723831249 >> 172.21.15.161:0/2723831249 conn(0x7f00d80ac620 msgr2=0x7f00d80aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:40.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.252+0000 7f00ded19640 1 -- 172.21.15.161:0/2723831249 shutdown_connections 2024-03-28T23:34:40.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.252+0000 7f00ded19640 1 -- 172.21.15.161:0/2723831249 wait complete. 2024-03-28T23:34:40.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.253+0000 7f00ded19640 1 Processor -- start 2024-03-28T23:34:40.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.253+0000 7f00ded19640 1 -- start start 2024-03-28T23:34:40.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.254+0000 7f00ded19640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00d80c5950 0x7f00d8155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:40.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.254+0000 7f00ded19640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00d80c8ed0 0x7f00d814eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:40.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.254+0000 7f00ded19640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00d8152890 0x7f00d81472e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:40.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.254+0000 7f00ded19640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f00d8152c70 con 0x7f00d8152890 2024-03-28T23:34:40.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.254+0000 7f00ded19640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f00d81592d0 con 0x7f00d80c5950 2024-03-28T23:34:40.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.254+0000 7f00ded19640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f00d8152df0 con 0x7f00d80c8ed0 2024-03-28T23:34:40.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.254+0000 7f00dd2b6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00d8152890 0x7f00d81472e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:40.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.254+0000 7f00dd2b6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00d8152890 0x7f00d81472e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34944/0 (socket says 172.21.15.161:34944) 2024-03-28T23:34:40.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.254+0000 7f00dd2b6640 1 -- 172.21.15.161:0/2612000526 learned_addr learned my addr 172.21.15.161:0/2612000526 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:40.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.255+0000 7f00d7fff640 1 --2- 172.21.15.161:0/2612000526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00d80c8ed0 0x7f00d814eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:40.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.255+0000 7f00dcab5640 1 --2- 172.21.15.161:0/2612000526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00d80c5950 0x7f00d8155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:40.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.255+0000 7f00dd2b6640 1 -- 172.21.15.161:0/2612000526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00d80c8ed0 msgr2=0x7f00d814eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:40.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.255+0000 7f00dd2b6640 1 --2- 172.21.15.161:0/2612000526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00d80c8ed0 0x7f00d814eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.255+0000 7f00dd2b6640 1 -- 172.21.15.161:0/2612000526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00d80c5950 msgr2=0x7f00d8155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:40.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.255+0000 7f00dd2b6640 1 --2- 172.21.15.161:0/2612000526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00d80c5950 0x7f00d8155aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.255+0000 7f00dd2b6640 1 -- 172.21.15.161:0/2612000526 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d814f750 con 0x7f00d8152890 2024-03-28T23:34:40.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.255+0000 7f00dcab5640 1 --2- 172.21.15.161:0/2612000526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00d80c5950 0x7f00d8155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:40.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.255+0000 7f00dd2b6640 1 --2- 172.21.15.161:0/2612000526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00d8152890 0x7f00d81472e0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f00d000eed0 tx=0x7f00d000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:40.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.255+0000 7f00d57fa640 1 -- 172.21.15.161:0/2612000526 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d0018020 con 0x7f00d8152890 2024-03-28T23:34:40.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.255+0000 7f00d57fa640 1 -- 172.21.15.161:0/2612000526 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00d000bd20 con 0x7f00d8152890 2024-03-28T23:34:40.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.255+0000 7f00ded19640 1 -- 172.21.15.161:0/2612000526 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00d814fa30 con 0x7f00d8152890 2024-03-28T23:34:40.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.255+0000 7f00d57fa640 1 -- 172.21.15.161:0/2612000526 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d0014640 con 0x7f00d8152890 2024-03-28T23:34:40.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.255+0000 7f00ded19640 1 -- 172.21.15.161:0/2612000526 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00d814cfb0 con 0x7f00d8152890 2024-03-28T23:34:40.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.256+0000 7f00ded19640 1 -- 172.21.15.161:0/2612000526 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00d804f4b0 con 0x7f00d8152890 2024-03-28T23:34:40.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.256+0000 7f00d57fa640 1 -- 172.21.15.161:0/2612000526 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f00d0005d10 con 0x7f00d8152890 2024-03-28T23:34:40.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.257+0000 7f00d57fa640 1 --2- 172.21.15.161:0/2612000526 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f00bc06b4d0 0x7f00bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:40.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.257+0000 7f00d57fa640 1 -- 172.21.15.161:0/2612000526 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6101+0+0 (secure 0 0 0) 0x7f00d008a980 con 0x7f00d8152890 2024-03-28T23:34:40.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.257+0000 7f00dcab5640 1 --2- 172.21.15.161:0/2612000526 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f00bc06b4d0 0x7f00bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:40.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.257+0000 7f00dcab5640 1 --2- 172.21.15.161:0/2612000526 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f00bc06b4d0 0x7f00bc06d990 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f00cc009ac0 tx=0x7f00cc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:40.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.261+0000 7f00d57fa640 1 -- 172.21.15.161:0/2612000526 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f00d0022070 con 0x7f00d8152890 2024-03-28T23:34:40.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.436+0000 7f00ded19640 1 -- 172.21.15.161:0/2612000526 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f00d80c8140 con 0x7f00bc06b4d0 2024-03-28T23:34:40.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.442+0000 7f00d57fa640 1 -- 172.21.15.161:0/2612000526 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252555 (secure 0 0 0) 0x7f00d80c8140 con 0x7f00bc06b4d0 2024-03-28T23:34:40.445 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:40.447 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:34:40.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.445+0000 7f00ded19640 1 -- 172.21.15.161:0/2612000526 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f00bc06b4d0 msgr2=0x7f00bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:40.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.445+0000 7f00ded19640 1 --2- 172.21.15.161:0/2612000526 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f00bc06b4d0 0x7f00bc06d990 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f00cc009ac0 tx=0x7f00cc008040 comp rx=0 tx=0).stop 2024-03-28T23:34:40.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.445+0000 7f00ded19640 1 -- 172.21.15.161:0/2612000526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00d8152890 msgr2=0x7f00d81472e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:40.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.445+0000 7f00ded19640 1 --2- 172.21.15.161:0/2612000526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00d8152890 0x7f00d81472e0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f00d000eed0 tx=0x7f00d000b480 comp rx=0 tx=0).stop 2024-03-28T23:34:40.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.445+0000 7f00ded19640 1 -- 172.21.15.161:0/2612000526 shutdown_connections 2024-03-28T23:34:40.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.445+0000 7f00ded19640 1 --2- 172.21.15.161:0/2612000526 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f00bc06b4d0 0x7f00bc06d990 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.445+0000 7f00ded19640 1 --2- 172.21.15.161:0/2612000526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00d8152890 0x7f00d81472e0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.445+0000 7f00ded19640 1 --2- 172.21.15.161:0/2612000526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00d80c8ed0 0x7f00d814eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.445+0000 7f00ded19640 1 --2- 172.21.15.161:0/2612000526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00d80c5950 0x7f00d8155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.445+0000 7f00ded19640 1 -- 172.21.15.161:0/2612000526 >> 172.21.15.161:0/2612000526 conn(0x7f00d80ac620 msgr2=0x7f00d8150b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:40.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.445+0000 7f00ded19640 1 -- 172.21.15.161:0/2612000526 shutdown_connections 2024-03-28T23:34:40.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.445+0000 7f00ded19640 1 -- 172.21.15.161:0/2612000526 wait complete. 2024-03-28T23:34:40.458 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2024-03-28T23:34:39.367840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":261552,"kb_used_data":12092,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713168,"statfs":{"total":773094113280,"available":772826284032,"internally_reserved":0,"allocated":12382208,"data_stored":7059648,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000743"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782365+0000","last_change":"2024-03-28T23:34:37.467867+0000","last_active":"2024-03-28T23:34:38.782365+0000","last_peered":"2024-03-28T23:34:38.782365+0000","last_clean":"2024-03-28T23:34:38.782365+0000","last_became_active":"2024-03-28T23:34:37.467387+0000","last_became_peered":"2024-03-28T23:34:37.467387+0000","last_unstale":"2024-03-28T23:34:38.782365+0000","last_undegraded":"2024-03-28T23:34:38.782365+0000","last_fullsized":"2024-03-28T23:34:38.782365+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:18.855042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466585+0000","last_change":"2024-03-28T23:34:37.461910+0000","last_active":"2024-03-28T23:34:38.466585+0000","last_peered":"2024-03-28T23:34:38.466585+0000","last_clean":"2024-03-28T23:34:38.466585+0000","last_became_active":"2024-03-28T23:34:37.461637+0000","last_became_peered":"2024-03-28T23:34:37.461637+0000","last_unstale":"2024-03-28T23:34:38.466585+0000","last_undegraded":"2024-03-28T23:34:38.466585+0000","last_fullsized":"2024-03-28T23:34:38.466585+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:21:21.872575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.467916+0000","last_change":"2024-03-28T23:34:37.470292+0000","last_active":"2024-03-28T23:34:38.467916+0000","last_peered":"2024-03-28T23:34:38.467916+0000","last_clean":"2024-03-28T23:34:38.467916+0000","last_became_active":"2024-03-28T23:34:37.470023+0000","last_became_peered":"2024-03-28T23:34:37.470023+0000","last_unstale":"2024-03-28T23:34:38.467916+0000","last_undegraded":"2024-03-28T23:34:38.467916+0000","last_fullsized":"2024-03-28T23:34:38.467916+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:30:08.395305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466877+0000","last_change":"2024-03-28T23:34:37.457378+0000","last_active":"2024-03-28T23:34:38.466877+0000","last_peered":"2024-03-28T23:34:38.466877+0000","last_clean":"2024-03-28T23:34:38.466877+0000","last_became_active":"2024-03-28T23:34:37.457104+0000","last_became_peered":"2024-03-28T23:34:37.457104+0000","last_unstale":"2024-03-28T23:34:38.466877+0000","last_undegraded":"2024-03-28T23:34:38.466877+0000","last_fullsized":"2024-03-28T23:34:38.466877+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:13.997477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781529+0000","last_change":"2024-03-28T23:34:37.471264+0000","last_active":"2024-03-28T23:34:38.781529+0000","last_peered":"2024-03-28T23:34:38.781529+0000","last_clean":"2024-03-28T23:34:38.781529+0000","last_became_active":"2024-03-28T23:34:37.470794+0000","last_became_peered":"2024-03-28T23:34:37.470794+0000","last_unstale":"2024-03-28T23:34:38.781529+0000","last_undegraded":"2024-03-28T23:34:38.781529+0000","last_fullsized":"2024-03-28T23:34:38.781529+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:30:00.138855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781492+0000","last_change":"2024-03-28T23:34:37.472995+0000","last_active":"2024-03-28T23:34:38.781492+0000","last_peered":"2024-03-28T23:34:38.781492+0000","last_clean":"2024-03-28T23:34:38.781492+0000","last_became_active":"2024-03-28T23:34:37.472728+0000","last_became_peered":"2024-03-28T23:34:37.472728+0000","last_unstale":"2024-03-28T23:34:38.781492+0000","last_undegraded":"2024-03-28T23:34:38.781492+0000","last_fullsized":"2024-03-28T23:34:38.781492+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:35:53.095955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781256+0000","last_change":"2024-03-28T23:34:37.470521+0000","last_active":"2024-03-28T23:34:38.781256+0000","last_peered":"2024-03-28T23:34:38.781256+0000","last_clean":"2024-03-28T23:34:38.781256+0000","last_became_active":"2024-03-28T23:34:37.470246+0000","last_became_peered":"2024-03-28T23:34:37.470246+0000","last_unstale":"2024-03-28T23:34:38.781256+0000","last_undegraded":"2024-03-28T23:34:38.781256+0000","last_fullsized":"2024-03-28T23:34:38.781256+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:47:47.623344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466349+0000","last_change":"2024-03-28T23:34:37.460567+0000","last_active":"2024-03-28T23:34:38.466349+0000","last_peered":"2024-03-28T23:34:38.466349+0000","last_clean":"2024-03-28T23:34:38.466349+0000","last_became_active":"2024-03-28T23:34:37.460293+0000","last_became_peered":"2024-03-28T23:34:37.460293+0000","last_unstale":"2024-03-28T23:34:38.466349+0000","last_undegraded":"2024-03-28T23:34:38.466349+0000","last_fullsized":"2024-03-28T23:34:38.466349+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:30:06.874314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781214+0000","last_change":"2024-03-28T23:34:37.467651+0000","last_active":"2024-03-28T23:34:38.781214+0000","last_peered":"2024-03-28T23:34:38.781214+0000","last_clean":"2024-03-28T23:34:38.781214+0000","last_became_active":"2024-03-28T23:34:37.465619+0000","last_became_peered":"2024-03-28T23:34:37.465619+0000","last_unstale":"2024-03-28T23:34:38.781214+0000","last_undegraded":"2024-03-28T23:34:38.781214+0000","last_fullsized":"2024-03-28T23:34:38.781214+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:13.997477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781576+0000","last_change":"2024-03-28T23:34:37.463391+0000","last_active":"2024-03-28T23:34:38.781576+0000","last_peered":"2024-03-28T23:34:38.781576+0000","last_clean":"2024-03-28T23:34:38.781576+0000","last_became_active":"2024-03-28T23:34:37.463119+0000","last_became_peered":"2024-03-28T23:34:37.463119+0000","last_unstale":"2024-03-28T23:34:38.781576+0000","last_undegraded":"2024-03-28T23:34:38.781576+0000","last_fullsized":"2024-03-28T23:34:38.781576+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:58:07.787997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.465928+0000","last_change":"2024-03-28T23:34:37.473059+0000","last_active":"2024-03-28T23:34:38.465928+0000","last_peered":"2024-03-28T23:34:38.465928+0000","last_clean":"2024-03-28T23:34:38.465928+0000","last_became_active":"2024-03-28T23:34:37.472762+0000","last_became_peered":"2024-03-28T23:34:37.472762+0000","last_unstale":"2024-03-28T23:34:38.465928+0000","last_undegraded":"2024-03-28T23:34:38.465928+0000","last_fullsized":"2024-03-28T23:34:38.465928+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:42:48.983466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781962+0000","last_change":"2024-03-28T23:34:37.461291+0000","last_active":"2024-03-28T23:34:38.781962+0000","last_peered":"2024-03-28T23:34:38.781962+0000","last_clean":"2024-03-28T23:34:38.781962+0000","last_became_active":"2024-03-28T23:34:37.460978+0000","last_became_peered":"2024-03-28T23:34:37.460978+0000","last_unstale":"2024-03-28T23:34:38.781962+0000","last_undegraded":"2024-03-28T23:34:38.781962+0000","last_fullsized":"2024-03-28T23:34:38.781962+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:27:04.373061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782562+0000","last_change":"2024-03-28T23:34:37.473983+0000","last_active":"2024-03-28T23:34:38.782562+0000","last_peered":"2024-03-28T23:34:38.782562+0000","last_clean":"2024-03-28T23:34:38.782562+0000","last_became_active":"2024-03-28T23:34:37.473395+0000","last_became_peered":"2024-03-28T23:34:37.473395+0000","last_unstale":"2024-03-28T23:34:38.782562+0000","last_undegraded":"2024-03-28T23:34:38.782562+0000","last_fullsized":"2024-03-28T23:34:38.782562+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:03:48.824778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781724+0000","last_change":"2024-03-28T23:34:37.469706+0000","last_active":"2024-03-28T23:34:38.781724+0000","last_peered":"2024-03-28T23:34:38.781724+0000","last_clean":"2024-03-28T23:34:38.781724+0000","last_became_active":"2024-03-28T23:34:37.469428+0000","last_became_peered":"2024-03-28T23:34:37.469428+0000","last_unstale":"2024-03-28T23:34:38.781724+0000","last_undegraded":"2024-03-28T23:34:38.781724+0000","last_fullsized":"2024-03-28T23:34:38.781724+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:25:16.277513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781365+0000","last_change":"2024-03-28T23:34:37.471525+0000","last_active":"2024-03-28T23:34:38.781365+0000","last_peered":"2024-03-28T23:34:38.781365+0000","last_clean":"2024-03-28T23:34:38.781365+0000","last_became_active":"2024-03-28T23:34:37.471278+0000","last_became_peered":"2024-03-28T23:34:37.471278+0000","last_unstale":"2024-03-28T23:34:38.781365+0000","last_undegraded":"2024-03-28T23:34:38.781365+0000","last_fullsized":"2024-03-28T23:34:38.781365+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:03:35.705086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781057+0000","last_change":"2024-03-28T23:34:37.467233+0000","last_active":"2024-03-28T23:34:38.781057+0000","last_peered":"2024-03-28T23:34:38.781057+0000","last_clean":"2024-03-28T23:34:38.781057+0000","last_became_active":"2024-03-28T23:34:37.466763+0000","last_became_peered":"2024-03-28T23:34:37.466763+0000","last_unstale":"2024-03-28T23:34:38.781057+0000","last_undegraded":"2024-03-28T23:34:38.781057+0000","last_fullsized":"2024-03-28T23:34:38.781057+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:47:28.781650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781341+0000","last_change":"2024-03-28T23:34:37.466118+0000","last_active":"2024-03-28T23:34:38.781341+0000","last_peered":"2024-03-28T23:34:38.781341+0000","last_clean":"2024-03-28T23:34:38.781341+0000","last_became_active":"2024-03-28T23:34:37.465845+0000","last_became_peered":"2024-03-28T23:34:37.465845+0000","last_unstale":"2024-03-28T23:34:38.781341+0000","last_undegraded":"2024-03-28T23:34:38.781341+0000","last_fullsized":"2024-03-28T23:34:38.781341+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:31:35.930069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782627+0000","last_change":"2024-03-28T23:34:37.474500+0000","last_active":"2024-03-28T23:34:38.782627+0000","last_peered":"2024-03-28T23:34:38.782627+0000","last_clean":"2024-03-28T23:34:38.782627+0000","last_became_active":"2024-03-28T23:34:37.474199+0000","last_became_peered":"2024-03-28T23:34:37.474199+0000","last_unstale":"2024-03-28T23:34:38.782627+0000","last_undegraded":"2024-03-28T23:34:38.782627+0000","last_fullsized":"2024-03-28T23:34:38.782627+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:06:07.825526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781611+0000","last_change":"2024-03-28T23:34:37.467214+0000","last_active":"2024-03-28T23:34:38.781611+0000","last_peered":"2024-03-28T23:34:38.781611+0000","last_clean":"2024-03-28T23:34:38.781611+0000","last_became_active":"2024-03-28T23:34:37.466797+0000","last_became_peered":"2024-03-28T23:34:37.466797+0000","last_unstale":"2024-03-28T23:34:38.781611+0000","last_undegraded":"2024-03-28T23:34:38.781611+0000","last_fullsized":"2024-03-28T23:34:38.781611+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:32.284640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782589+0000","last_change":"2024-03-28T23:34:37.461075+0000","last_active":"2024-03-28T23:34:38.782589+0000","last_peered":"2024-03-28T23:34:38.782589+0000","last_clean":"2024-03-28T23:34:38.782589+0000","last_became_active":"2024-03-28T23:34:37.460798+0000","last_became_peered":"2024-03-28T23:34:37.460798+0000","last_unstale":"2024-03-28T23:34:38.782589+0000","last_undegraded":"2024-03-28T23:34:38.782589+0000","last_fullsized":"2024-03-28T23:34:38.782589+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:08:12.545863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.465991+0000","last_change":"2024-03-28T23:34:37.472505+0000","last_active":"2024-03-28T23:34:38.465991+0000","last_peered":"2024-03-28T23:34:38.465991+0000","last_clean":"2024-03-28T23:34:38.465991+0000","last_became_active":"2024-03-28T23:34:37.472063+0000","last_became_peered":"2024-03-28T23:34:37.472063+0000","last_unstale":"2024-03-28T23:34:38.465991+0000","last_undegraded":"2024-03-28T23:34:38.465991+0000","last_fullsized":"2024-03-28T23:34:38.465991+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:26.729239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781583+0000","last_change":"2024-03-28T23:34:37.469192+0000","last_active":"2024-03-28T23:34:38.781583+0000","last_peered":"2024-03-28T23:34:38.781583+0000","last_clean":"2024-03-28T23:34:38.781583+0000","last_became_active":"2024-03-28T23:34:37.468171+0000","last_became_peered":"2024-03-28T23:34:37.468171+0000","last_unstale":"2024-03-28T23:34:38.781583+0000","last_undegraded":"2024-03-28T23:34:38.781583+0000","last_fullsized":"2024-03-28T23:34:38.781583+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:51.400972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466565+0000","last_change":"2024-03-28T23:34:37.461758+0000","last_active":"2024-03-28T23:34:38.466565+0000","last_peered":"2024-03-28T23:34:38.466565+0000","last_clean":"2024-03-28T23:34:38.466565+0000","last_became_active":"2024-03-28T23:34:37.461240+0000","last_became_peered":"2024-03-28T23:34:37.461240+0000","last_unstale":"2024-03-28T23:34:38.466565+0000","last_undegraded":"2024-03-28T23:34:38.466565+0000","last_fullsized":"2024-03-28T23:34:38.466565+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:01.408086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781188+0000","last_change":"2024-03-28T23:34:37.467029+0000","last_active":"2024-03-28T23:34:38.781188+0000","last_peered":"2024-03-28T23:34:38.781188+0000","last_clean":"2024-03-28T23:34:38.781188+0000","last_became_active":"2024-03-28T23:34:37.466414+0000","last_became_peered":"2024-03-28T23:34:37.466414+0000","last_unstale":"2024-03-28T23:34:38.781188+0000","last_undegraded":"2024-03-28T23:34:38.781188+0000","last_fullsized":"2024-03-28T23:34:38.781188+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:21:21.872575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.468190+0000","last_change":"2024-03-28T23:34:37.472238+0000","last_active":"2024-03-28T23:34:38.468190+0000","last_peered":"2024-03-28T23:34:38.468190+0000","last_clean":"2024-03-28T23:34:38.468190+0000","last_became_active":"2024-03-28T23:34:37.471771+0000","last_became_peered":"2024-03-28T23:34:37.471771+0000","last_unstale":"2024-03-28T23:34:38.468190+0000","last_undegraded":"2024-03-28T23:34:38.468190+0000","last_fullsized":"2024-03-28T23:34:38.468190+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:43:36.888242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.467812+0000","last_change":"2024-03-28T23:34:37.469369+0000","last_active":"2024-03-28T23:34:38.467812+0000","last_peered":"2024-03-28T23:34:38.467812+0000","last_clean":"2024-03-28T23:34:38.467812+0000","last_became_active":"2024-03-28T23:34:37.469026+0000","last_became_peered":"2024-03-28T23:34:37.469026+0000","last_unstale":"2024-03-28T23:34:38.467812+0000","last_undegraded":"2024-03-28T23:34:38.467812+0000","last_fullsized":"2024-03-28T23:34:38.467812+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:52:21.097148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781821+0000","last_change":"2024-03-28T23:34:37.475553+0000","last_active":"2024-03-28T23:34:38.781821+0000","last_peered":"2024-03-28T23:34:38.781821+0000","last_clean":"2024-03-28T23:34:38.781821+0000","last_became_active":"2024-03-28T23:34:37.475280+0000","last_became_peered":"2024-03-28T23:34:37.475280+0000","last_unstale":"2024-03-28T23:34:38.781821+0000","last_undegraded":"2024-03-28T23:34:38.781821+0000","last_fullsized":"2024-03-28T23:34:38.781821+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:34:54.305199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782819+0000","last_change":"2024-03-28T23:34:37.462192+0000","last_active":"2024-03-28T23:34:38.782819+0000","last_peered":"2024-03-28T23:34:38.782819+0000","last_clean":"2024-03-28T23:34:38.782819+0000","last_became_active":"2024-03-28T23:34:37.461921+0000","last_became_peered":"2024-03-28T23:34:37.461921+0000","last_unstale":"2024-03-28T23:34:38.782819+0000","last_undegraded":"2024-03-28T23:34:38.782819+0000","last_fullsized":"2024-03-28T23:34:38.782819+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:41.535721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.467819+0000","last_change":"2024-03-28T23:34:37.477937+0000","last_active":"2024-03-28T23:34:38.467819+0000","last_peered":"2024-03-28T23:34:38.467819+0000","last_clean":"2024-03-28T23:34:38.467819+0000","last_became_active":"2024-03-28T23:34:37.476449+0000","last_became_peered":"2024-03-28T23:34:37.476449+0000","last_unstale":"2024-03-28T23:34:38.467819+0000","last_undegraded":"2024-03-28T23:34:38.467819+0000","last_fullsized":"2024-03-28T23:34:38.467819+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:57.794609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.467858+0000","last_change":"2024-03-28T23:34:37.471383+0000","last_active":"2024-03-28T23:34:38.467858+0000","last_peered":"2024-03-28T23:34:38.467858+0000","last_clean":"2024-03-28T23:34:38.467858+0000","last_became_active":"2024-03-28T23:34:37.471096+0000","last_became_peered":"2024-03-28T23:34:37.471096+0000","last_unstale":"2024-03-28T23:34:38.467858+0000","last_undegraded":"2024-03-28T23:34:38.467858+0000","last_fullsized":"2024-03-28T23:34:38.467858+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:35:11.976716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466308+0000","last_change":"2024-03-28T23:34:37.454074+0000","last_active":"2024-03-28T23:34:38.466308+0000","last_peered":"2024-03-28T23:34:38.466308+0000","last_clean":"2024-03-28T23:34:38.466308+0000","last_became_active":"2024-03-28T23:34:37.453758+0000","last_became_peered":"2024-03-28T23:34:37.453758+0000","last_unstale":"2024-03-28T23:34:38.466308+0000","last_undegraded":"2024-03-28T23:34:38.466308+0000","last_fullsized":"2024-03-28T23:34:38.466308+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:02:47.244704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781168+0000","last_change":"2024-03-28T23:34:37.457986+0000","last_active":"2024-03-28T23:34:38.781168+0000","last_peered":"2024-03-28T23:34:38.781168+0000","last_clean":"2024-03-28T23:34:38.781168+0000","last_became_active":"2024-03-28T23:34:37.457694+0000","last_became_peered":"2024-03-28T23:34:37.457694+0000","last_unstale":"2024-03-28T23:34:38.781168+0000","last_undegraded":"2024-03-28T23:34:38.781168+0000","last_fullsized":"2024-03-28T23:34:38.781168+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:56:27.639060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466322+0000","last_change":"2024-03-28T23:34:37.462490+0000","last_active":"2024-03-28T23:34:38.466322+0000","last_peered":"2024-03-28T23:34:38.466322+0000","last_clean":"2024-03-28T23:34:38.466322+0000","last_became_active":"2024-03-28T23:34:37.462185+0000","last_became_peered":"2024-03-28T23:34:37.462185+0000","last_unstale":"2024-03-28T23:34:38.466322+0000","last_undegraded":"2024-03-28T23:34:38.466322+0000","last_fullsized":"2024-03-28T23:34:38.466322+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:11:25.844380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"121'4","reported_seq":18,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.468140+0000","last_change":"2024-03-28T23:34:36.457009+0000","last_active":"2024-03-28T23:34:38.468140+0000","last_peered":"2024-03-28T23:34:38.468140+0000","last_clean":"2024-03-28T23:34:38.468140+0000","last_became_active":"2024-03-28T23:34:36.456735+0000","last_became_peered":"2024-03-28T23:34:36.456735+0000","last_unstale":"2024-03-28T23:34:38.468140+0000","last_undegraded":"2024-03-28T23:34:38.468140+0000","last_fullsized":"2024-03-28T23:34:38.468140+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:58.569300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781535+0000","last_change":"2024-03-28T23:34:37.468232+0000","last_active":"2024-03-28T23:34:38.781535+0000","last_peered":"2024-03-28T23:34:38.781535+0000","last_clean":"2024-03-28T23:34:38.781535+0000","last_became_active":"2024-03-28T23:34:37.467912+0000","last_became_peered":"2024-03-28T23:34:37.467912+0000","last_unstale":"2024-03-28T23:34:38.781535+0000","last_undegraded":"2024-03-28T23:34:38.781535+0000","last_fullsized":"2024-03-28T23:34:38.781535+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:36:52.766599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"121'1","reported_seq":15,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782869+0000","last_change":"2024-03-28T23:34:36.448047+0000","last_active":"2024-03-28T23:34:38.782869+0000","last_peered":"2024-03-28T23:34:38.782869+0000","last_clean":"2024-03-28T23:34:38.782869+0000","last_became_active":"2024-03-28T23:34:36.447047+0000","last_became_peered":"2024-03-28T23:34:36.447047+0000","last_unstale":"2024-03-28T23:34:38.782869+0000","last_undegraded":"2024-03-28T23:34:38.782869+0000","last_fullsized":"2024-03-28T23:34:38.782869+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:00.402470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466384+0000","last_change":"2024-03-28T23:34:37.459132+0000","last_active":"2024-03-28T23:34:38.466384+0000","last_peered":"2024-03-28T23:34:38.466384+0000","last_clean":"2024-03-28T23:34:38.466384+0000","last_became_active":"2024-03-28T23:34:37.458861+0000","last_became_peered":"2024-03-28T23:34:37.458861+0000","last_unstale":"2024-03-28T23:34:38.466384+0000","last_undegraded":"2024-03-28T23:34:38.466384+0000","last_fullsized":"2024-03-28T23:34:38.466384+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:47:28.781650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"121'2","reported_seq":16,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781693+0000","last_change":"2024-03-28T23:34:36.455626+0000","last_active":"2024-03-28T23:34:38.781693+0000","last_peered":"2024-03-28T23:34:38.781693+0000","last_clean":"2024-03-28T23:34:38.781693+0000","last_became_active":"2024-03-28T23:34:36.455356+0000","last_became_peered":"2024-03-28T23:34:36.455356+0000","last_unstale":"2024-03-28T23:34:38.781693+0000","last_undegraded":"2024-03-28T23:34:38.781693+0000","last_fullsized":"2024-03-28T23:34:38.781693+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:42:47.977850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781968+0000","last_change":"2024-03-28T23:34:37.470435+0000","last_active":"2024-03-28T23:34:38.781968+0000","last_peered":"2024-03-28T23:34:38.781968+0000","last_clean":"2024-03-28T23:34:38.781968+0000","last_became_active":"2024-03-28T23:34:37.470185+0000","last_became_peered":"2024-03-28T23:34:37.470185+0000","last_unstale":"2024-03-28T23:34:38.781968+0000","last_undegraded":"2024-03-28T23:34:38.781968+0000","last_fullsized":"2024-03-28T23:34:38.781968+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:20:48.203536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"121'2","reported_seq":16,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781393+0000","last_change":"2024-03-28T23:34:36.456195+0000","last_active":"2024-03-28T23:34:38.781393+0000","last_peered":"2024-03-28T23:34:38.781393+0000","last_clean":"2024-03-28T23:34:38.781393+0000","last_became_active":"2024-03-28T23:34:36.455573+0000","last_became_peered":"2024-03-28T23:34:36.455573+0000","last_unstale":"2024-03-28T23:34:38.781393+0000","last_undegraded":"2024-03-28T23:34:38.781393+0000","last_fullsized":"2024-03-28T23:34:38.781393+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:05:34.122514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466353+0000","last_change":"2024-03-28T23:34:37.456376+0000","last_active":"2024-03-28T23:34:38.466353+0000","last_peered":"2024-03-28T23:34:38.466353+0000","last_clean":"2024-03-28T23:34:38.466353+0000","last_became_active":"2024-03-28T23:34:37.456003+0000","last_became_peered":"2024-03-28T23:34:37.456003+0000","last_unstale":"2024-03-28T23:34:38.466353+0000","last_undegraded":"2024-03-28T23:34:38.466353+0000","last_fullsized":"2024-03-28T23:34:38.466353+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:28:49.929508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"121'1","reported_seq":15,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466216+0000","last_change":"2024-03-28T23:34:36.457412+0000","last_active":"2024-03-28T23:34:38.466216+0000","last_peered":"2024-03-28T23:34:38.466216+0000","last_clean":"2024-03-28T23:34:38.466216+0000","last_became_active":"2024-03-28T23:34:36.457131+0000","last_became_peered":"2024-03-28T23:34:36.457131+0000","last_unstale":"2024-03-28T23:34:38.466216+0000","last_undegraded":"2024-03-28T23:34:38.466216+0000","last_fullsized":"2024-03-28T23:34:38.466216+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:35:10.971100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466382+0000","last_change":"2024-03-28T23:34:37.459773+0000","last_active":"2024-03-28T23:34:38.466382+0000","last_peered":"2024-03-28T23:34:38.466382+0000","last_clean":"2024-03-28T23:34:38.466382+0000","last_became_active":"2024-03-28T23:34:37.459471+0000","last_became_peered":"2024-03-28T23:34:37.459471+0000","last_unstale":"2024-03-28T23:34:38.466382+0000","last_undegraded":"2024-03-28T23:34:38.466382+0000","last_fullsized":"2024-03-28T23:34:38.466382+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:42:48.983466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"121'1","reported_seq":15,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782575+0000","last_change":"2024-03-28T23:34:36.453536+0000","last_active":"2024-03-28T23:34:38.782575+0000","last_peered":"2024-03-28T23:34:38.782575+0000","last_clean":"2024-03-28T23:34:38.782575+0000","last_became_active":"2024-03-28T23:34:36.453009+0000","last_became_peered":"2024-03-28T23:34:36.453009+0000","last_unstale":"2024-03-28T23:34:38.782575+0000","last_undegraded":"2024-03-28T23:34:38.782575+0000","last_fullsized":"2024-03-28T23:34:38.782575+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:25.723622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782766+0000","last_change":"2024-03-28T23:34:37.460940+0000","last_active":"2024-03-28T23:34:38.782766+0000","last_peered":"2024-03-28T23:34:38.782766+0000","last_clean":"2024-03-28T23:34:38.782766+0000","last_became_active":"2024-03-28T23:34:37.460615+0000","last_became_peered":"2024-03-28T23:34:37.460615+0000","last_unstale":"2024-03-28T23:34:38.782766+0000","last_undegraded":"2024-03-28T23:34:38.782766+0000","last_fullsized":"2024-03-28T23:34:38.782766+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:03:44.505317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781982+0000","last_change":"2024-03-28T23:34:36.461740+0000","last_active":"2024-03-28T23:34:38.781982+0000","last_peered":"2024-03-28T23:34:38.781982+0000","last_clean":"2024-03-28T23:34:38.781982+0000","last_became_active":"2024-03-28T23:34:36.461440+0000","last_became_peered":"2024-03-28T23:34:36.461440+0000","last_unstale":"2024-03-28T23:34:38.781982+0000","last_undegraded":"2024-03-28T23:34:38.781982+0000","last_fullsized":"2024-03-28T23:34:38.781982+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:35:10.971100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781706+0000","last_change":"2024-03-28T23:34:37.475659+0000","last_active":"2024-03-28T23:34:38.781706+0000","last_peered":"2024-03-28T23:34:38.781706+0000","last_clean":"2024-03-28T23:34:38.781706+0000","last_became_active":"2024-03-28T23:34:37.475385+0000","last_became_peered":"2024-03-28T23:34:37.475385+0000","last_unstale":"2024-03-28T23:34:38.781706+0000","last_undegraded":"2024-03-28T23:34:38.781706+0000","last_fullsized":"2024-03-28T23:34:38.781706+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:28:10.183556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.467484+0000","last_change":"2024-03-28T23:34:36.458280+0000","last_active":"2024-03-28T23:34:38.467484+0000","last_peered":"2024-03-28T23:34:38.467484+0000","last_clean":"2024-03-28T23:34:38.467484+0000","last_became_active":"2024-03-28T23:34:36.457939+0000","last_became_peered":"2024-03-28T23:34:36.457939+0000","last_unstale":"2024-03-28T23:34:38.467484+0000","last_undegraded":"2024-03-28T23:34:38.467484+0000","last_fullsized":"2024-03-28T23:34:38.467484+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:00:38.115038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781212+0000","last_change":"2024-03-28T23:34:37.473431+0000","last_active":"2024-03-28T23:34:38.781212+0000","last_peered":"2024-03-28T23:34:38.781212+0000","last_clean":"2024-03-28T23:34:38.781212+0000","last_became_active":"2024-03-28T23:34:37.473042+0000","last_became_peered":"2024-03-28T23:34:37.473042+0000","last_unstale":"2024-03-28T23:34:38.781212+0000","last_undegraded":"2024-03-28T23:34:38.781212+0000","last_fullsized":"2024-03-28T23:34:38.781212+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:51:55.988297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466255+0000","last_change":"2024-03-28T23:34:36.449244+0000","last_active":"2024-03-28T23:34:38.466255+0000","last_peered":"2024-03-28T23:34:38.466255+0000","last_clean":"2024-03-28T23:34:38.466255+0000","last_became_active":"2024-03-28T23:34:36.448890+0000","last_became_peered":"2024-03-28T23:34:36.448890+0000","last_unstale":"2024-03-28T23:34:38.466255+0000","last_undegraded":"2024-03-28T23:34:38.466255+0000","last_fullsized":"2024-03-28T23:34:38.466255+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:22.354809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782114+0000","last_change":"2024-03-28T23:34:37.462559+0000","last_active":"2024-03-28T23:34:38.782114+0000","last_peered":"2024-03-28T23:34:38.782114+0000","last_clean":"2024-03-28T23:34:38.782114+0000","last_became_active":"2024-03-28T23:34:37.460383+0000","last_became_peered":"2024-03-28T23:34:37.460383+0000","last_unstale":"2024-03-28T23:34:38.782114+0000","last_undegraded":"2024-03-28T23:34:38.782114+0000","last_fullsized":"2024-03-28T23:34:38.782114+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:28:23.658834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782065+0000","last_change":"2024-03-28T23:34:36.456062+0000","last_active":"2024-03-28T23:34:38.782065+0000","last_peered":"2024-03-28T23:34:38.782065+0000","last_clean":"2024-03-28T23:34:38.782065+0000","last_became_active":"2024-03-28T23:34:36.455793+0000","last_became_peered":"2024-03-28T23:34:36.455793+0000","last_unstale":"2024-03-28T23:34:38.782065+0000","last_undegraded":"2024-03-28T23:34:38.782065+0000","last_fullsized":"2024-03-28T23:34:38.782065+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:11:40.847837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.467876+0000","last_change":"2024-03-28T23:34:37.471932+0000","last_active":"2024-03-28T23:34:38.467876+0000","last_peered":"2024-03-28T23:34:38.467876+0000","last_clean":"2024-03-28T23:34:38.467876+0000","last_became_active":"2024-03-28T23:34:37.470143+0000","last_became_peered":"2024-03-28T23:34:37.470143+0000","last_unstale":"2024-03-28T23:34:38.467876+0000","last_undegraded":"2024-03-28T23:34:38.467876+0000","last_fullsized":"2024-03-28T23:34:38.467876+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:20.557474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"121'3","reported_seq":17,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781794+0000","last_change":"2024-03-28T23:34:36.457100+0000","last_active":"2024-03-28T23:34:38.781794+0000","last_peered":"2024-03-28T23:34:38.781794+0000","last_clean":"2024-03-28T23:34:38.781794+0000","last_became_active":"2024-03-28T23:34:36.456724+0000","last_became_peered":"2024-03-28T23:34:36.456724+0000","last_unstale":"2024-03-28T23:34:38.781794+0000","last_undegraded":"2024-03-28T23:34:38.781794+0000","last_fullsized":"2024-03-28T23:34:38.781794+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:30:05.868698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.467837+0000","last_change":"2024-03-28T23:34:37.477251+0000","last_active":"2024-03-28T23:34:38.467837+0000","last_peered":"2024-03-28T23:34:38.467837+0000","last_clean":"2024-03-28T23:34:38.467837+0000","last_became_active":"2024-03-28T23:34:37.476655+0000","last_became_peered":"2024-03-28T23:34:37.476655+0000","last_unstale":"2024-03-28T23:34:38.467837+0000","last_undegraded":"2024-03-28T23:34:38.467837+0000","last_fullsized":"2024-03-28T23:34:38.467837+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:48.842819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"121'6","reported_seq":20,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781519+0000","last_change":"2024-03-28T23:34:36.456363+0000","last_active":"2024-03-28T23:34:38.781519+0000","last_peered":"2024-03-28T23:34:38.781519+0000","last_clean":"2024-03-28T23:34:38.781519+0000","last_became_active":"2024-03-28T23:34:36.455864+0000","last_became_peered":"2024-03-28T23:34:36.455864+0000","last_unstale":"2024-03-28T23:34:38.781519+0000","last_undegraded":"2024-03-28T23:34:38.781519+0000","last_fullsized":"2024-03-28T23:34:38.781519+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:47:37.317672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782959+0000","last_change":"2024-03-28T23:34:37.469727+0000","last_active":"2024-03-28T23:34:38.782959+0000","last_peered":"2024-03-28T23:34:38.782959+0000","last_clean":"2024-03-28T23:34:38.782959+0000","last_became_active":"2024-03-28T23:34:37.469464+0000","last_became_peered":"2024-03-28T23:34:37.469464+0000","last_unstale":"2024-03-28T23:34:38.782959+0000","last_undegraded":"2024-03-28T23:34:38.782959+0000","last_fullsized":"2024-03-28T23:34:38.782959+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:42:31.409331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"121'2","reported_seq":16,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782122+0000","last_change":"2024-03-28T23:34:36.457197+0000","last_active":"2024-03-28T23:34:38.782122+0000","last_peered":"2024-03-28T23:34:38.782122+0000","last_clean":"2024-03-28T23:34:38.782122+0000","last_became_active":"2024-03-28T23:34:36.456770+0000","last_became_peered":"2024-03-28T23:34:36.456770+0000","last_unstale":"2024-03-28T23:34:38.782122+0000","last_undegraded":"2024-03-28T23:34:38.782122+0000","last_fullsized":"2024-03-28T23:34:38.782122+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:49:48.121807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781812+0000","last_change":"2024-03-28T23:34:37.462948+0000","last_active":"2024-03-28T23:34:38.781812+0000","last_peered":"2024-03-28T23:34:38.781812+0000","last_clean":"2024-03-28T23:34:38.781812+0000","last_became_active":"2024-03-28T23:34:37.462688+0000","last_became_peered":"2024-03-28T23:34:37.462688+0000","last_unstale":"2024-03-28T23:34:38.781812+0000","last_undegraded":"2024-03-28T23:34:38.781812+0000","last_fullsized":"2024-03-28T23:34:38.781812+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:06:07.825526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.467679+0000","last_change":"2024-03-28T23:34:36.459082+0000","last_active":"2024-03-28T23:34:38.467679+0000","last_peered":"2024-03-28T23:34:38.467679+0000","last_clean":"2024-03-28T23:34:38.467679+0000","last_became_active":"2024-03-28T23:34:36.458802+0000","last_became_peered":"2024-03-28T23:34:36.458802+0000","last_unstale":"2024-03-28T23:34:38.467679+0000","last_undegraded":"2024-03-28T23:34:38.467679+0000","last_fullsized":"2024-03-28T23:34:38.467679+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:17:03.981240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782448+0000","last_change":"2024-03-28T23:34:37.471748+0000","last_active":"2024-03-28T23:34:38.782448+0000","last_peered":"2024-03-28T23:34:38.782448+0000","last_clean":"2024-03-28T23:34:38.782448+0000","last_became_active":"2024-03-28T23:34:37.471486+0000","last_became_peered":"2024-03-28T23:34:37.471486+0000","last_unstale":"2024-03-28T23:34:38.782448+0000","last_undegraded":"2024-03-28T23:34:38.782448+0000","last_fullsized":"2024-03-28T23:34:38.782448+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:28:10.183556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"121'8","reported_seq":22,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781280+0000","last_change":"2024-03-28T23:34:36.458302+0000","last_active":"2024-03-28T23:34:38.781280+0000","last_peered":"2024-03-28T23:34:38.781280+0000","last_clean":"2024-03-28T23:34:38.781280+0000","last_became_active":"2024-03-28T23:34:36.457992+0000","last_became_peered":"2024-03-28T23:34:36.457992+0000","last_unstale":"2024-03-28T23:34:38.781280+0000","last_undegraded":"2024-03-28T23:34:38.781280+0000","last_fullsized":"2024-03-28T23:34:38.781280+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:19.551857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.467334+0000","last_change":"2024-03-28T23:34:37.467808+0000","last_active":"2024-03-28T23:34:38.467334+0000","last_peered":"2024-03-28T23:34:38.467334+0000","last_clean":"2024-03-28T23:34:38.467334+0000","last_became_active":"2024-03-28T23:34:37.467493+0000","last_became_peered":"2024-03-28T23:34:37.467493+0000","last_unstale":"2024-03-28T23:34:38.467334+0000","last_undegraded":"2024-03-28T23:34:38.467334+0000","last_fullsized":"2024-03-28T23:34:38.467334+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:46:55.411569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"121'1","reported_seq":15,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466349+0000","last_change":"2024-03-28T23:34:36.457449+0000","last_active":"2024-03-28T23:34:38.466349+0000","last_peered":"2024-03-28T23:34:38.466349+0000","last_clean":"2024-03-28T23:34:38.466349+0000","last_became_active":"2024-03-28T23:34:36.457041+0000","last_became_peered":"2024-03-28T23:34:36.457041+0000","last_unstale":"2024-03-28T23:34:38.466349+0000","last_undegraded":"2024-03-28T23:34:38.466349+0000","last_fullsized":"2024-03-28T23:34:38.466349+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:54:50.082017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781365+0000","last_change":"2024-03-28T23:34:37.465288+0000","last_active":"2024-03-28T23:34:38.781365+0000","last_peered":"2024-03-28T23:34:38.781365+0000","last_clean":"2024-03-28T23:34:38.781365+0000","last_became_active":"2024-03-28T23:34:37.465019+0000","last_became_peered":"2024-03-28T23:34:37.465019+0000","last_unstale":"2024-03-28T23:34:38.781365+0000","last_undegraded":"2024-03-28T23:34:38.781365+0000","last_fullsized":"2024-03-28T23:34:38.781365+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:03:48.824778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"121'2","reported_seq":16,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.468055+0000","last_change":"2024-03-28T23:34:36.459332+0000","last_active":"2024-03-28T23:34:38.468055+0000","last_peered":"2024-03-28T23:34:38.468055+0000","last_clean":"2024-03-28T23:34:38.468055+0000","last_became_active":"2024-03-28T23:34:36.458972+0000","last_became_peered":"2024-03-28T23:34:36.458972+0000","last_unstale":"2024-03-28T23:34:38.468055+0000","last_undegraded":"2024-03-28T23:34:38.468055+0000","last_fullsized":"2024-03-28T23:34:38.468055+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:55.285139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781518+0000","last_change":"2024-03-28T23:34:37.476449+0000","last_active":"2024-03-28T23:34:38.781518+0000","last_peered":"2024-03-28T23:34:38.781518+0000","last_clean":"2024-03-28T23:34:38.781518+0000","last_became_active":"2024-03-28T23:34:37.475904+0000","last_became_peered":"2024-03-28T23:34:37.475904+0000","last_unstale":"2024-03-28T23:34:38.781518+0000","last_undegraded":"2024-03-28T23:34:38.781518+0000","last_fullsized":"2024-03-28T23:34:38.781518+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:30:00.138855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"121'1","reported_seq":15,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781891+0000","last_change":"2024-03-28T23:34:36.457693+0000","last_active":"2024-03-28T23:34:38.781891+0000","last_peered":"2024-03-28T23:34:38.781891+0000","last_clean":"2024-03-28T23:34:38.781891+0000","last_became_active":"2024-03-28T23:34:36.457199+0000","last_became_peered":"2024-03-28T23:34:36.457199+0000","last_unstale":"2024-03-28T23:34:38.781891+0000","last_undegraded":"2024-03-28T23:34:38.781891+0000","last_fullsized":"2024-03-28T23:34:38.781891+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:22.354809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781609+0000","last_change":"2024-03-28T23:34:37.475892+0000","last_active":"2024-03-28T23:34:38.781609+0000","last_peered":"2024-03-28T23:34:38.781609+0000","last_clean":"2024-03-28T23:34:38.781609+0000","last_became_active":"2024-03-28T23:34:37.475597+0000","last_became_peered":"2024-03-28T23:34:37.475597+0000","last_unstale":"2024-03-28T23:34:38.781609+0000","last_undegraded":"2024-03-28T23:34:38.781609+0000","last_fullsized":"2024-03-28T23:34:38.781609+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:08:55.299299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782692+0000","last_change":"2024-03-28T23:34:36.451020+0000","last_active":"2024-03-28T23:34:38.782692+0000","last_peered":"2024-03-28T23:34:38.782692+0000","last_clean":"2024-03-28T23:34:38.782692+0000","last_became_active":"2024-03-28T23:34:36.450691+0000","last_became_peered":"2024-03-28T23:34:36.450691+0000","last_unstale":"2024-03-28T23:34:38.782692+0000","last_undegraded":"2024-03-28T23:34:38.782692+0000","last_fullsized":"2024-03-28T23:34:38.782692+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:47:27.776034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782703+0000","last_change":"2024-03-28T23:34:37.463237+0000","last_active":"2024-03-28T23:34:38.782703+0000","last_peered":"2024-03-28T23:34:38.782703+0000","last_clean":"2024-03-28T23:34:38.782703+0000","last_became_active":"2024-03-28T23:34:37.462997+0000","last_became_peered":"2024-03-28T23:34:37.462997+0000","last_unstale":"2024-03-28T23:34:38.782703+0000","last_undegraded":"2024-03-28T23:34:38.782703+0000","last_fullsized":"2024-03-28T23:34:38.782703+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:34:54.305199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"121'2","reported_seq":16,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466339+0000","last_change":"2024-03-28T23:34:36.459222+0000","last_active":"2024-03-28T23:34:38.466339+0000","last_peered":"2024-03-28T23:34:38.466339+0000","last_clean":"2024-03-28T23:34:38.466339+0000","last_became_active":"2024-03-28T23:34:36.458928+0000","last_became_peered":"2024-03-28T23:34:36.458928+0000","last_unstale":"2024-03-28T23:34:38.466339+0000","last_undegraded":"2024-03-28T23:34:38.466339+0000","last_fullsized":"2024-03-28T23:34:38.466339+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:19.551857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782280+0000","last_change":"2024-03-28T23:34:37.469032+0000","last_active":"2024-03-28T23:34:38.782280+0000","last_peered":"2024-03-28T23:34:38.782280+0000","last_clean":"2024-03-28T23:34:38.782280+0000","last_became_active":"2024-03-28T23:34:37.468770+0000","last_became_peered":"2024-03-28T23:34:37.468770+0000","last_unstale":"2024-03-28T23:34:38.782280+0000","last_undegraded":"2024-03-28T23:34:38.782280+0000","last_fullsized":"2024-03-28T23:34:38.782280+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:27:23.671506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781813+0000","last_change":"2024-03-28T23:34:36.454376+0000","last_active":"2024-03-28T23:34:38.781813+0000","last_peered":"2024-03-28T23:34:38.781813+0000","last_clean":"2024-03-28T23:34:38.781813+0000","last_became_active":"2024-03-28T23:34:36.453491+0000","last_became_peered":"2024-03-28T23:34:36.453491+0000","last_unstale":"2024-03-28T23:34:38.781813+0000","last_undegraded":"2024-03-28T23:34:38.781813+0000","last_fullsized":"2024-03-28T23:34:38.781813+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:27:35.273330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782852+0000","last_change":"2024-03-28T23:34:37.474290+0000","last_active":"2024-03-28T23:34:38.782852+0000","last_peered":"2024-03-28T23:34:38.782852+0000","last_clean":"2024-03-28T23:34:38.782852+0000","last_became_active":"2024-03-28T23:34:37.473919+0000","last_became_peered":"2024-03-28T23:34:37.473919+0000","last_unstale":"2024-03-28T23:34:38.782852+0000","last_undegraded":"2024-03-28T23:34:38.782852+0000","last_fullsized":"2024-03-28T23:34:38.782852+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:35:42.635666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781383+0000","last_change":"2024-03-28T23:34:36.454063+0000","last_active":"2024-03-28T23:34:38.781383+0000","last_peered":"2024-03-28T23:34:38.781383+0000","last_clean":"2024-03-28T23:34:38.781383+0000","last_became_active":"2024-03-28T23:34:36.453766+0000","last_became_peered":"2024-03-28T23:34:36.453766+0000","last_unstale":"2024-03-28T23:34:38.781383+0000","last_undegraded":"2024-03-28T23:34:38.781383+0000","last_fullsized":"2024-03-28T23:34:38.781383+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:02:46.239087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466162+0000","last_change":"2024-03-28T23:34:37.473020+0000","last_active":"2024-03-28T23:34:38.466162+0000","last_peered":"2024-03-28T23:34:38.466162+0000","last_clean":"2024-03-28T23:34:38.466162+0000","last_became_active":"2024-03-28T23:34:37.472738+0000","last_became_peered":"2024-03-28T23:34:37.472738+0000","last_unstale":"2024-03-28T23:34:38.466162+0000","last_undegraded":"2024-03-28T23:34:38.466162+0000","last_fullsized":"2024-03-28T23:34:38.466162+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:22:03.340749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781366+0000","last_change":"2024-03-28T23:34:36.455295+0000","last_active":"2024-03-28T23:34:38.781366+0000","last_peered":"2024-03-28T23:34:38.781366+0000","last_clean":"2024-03-28T23:34:38.781366+0000","last_became_active":"2024-03-28T23:34:36.454979+0000","last_became_peered":"2024-03-28T23:34:36.454979+0000","last_unstale":"2024-03-28T23:34:38.781366+0000","last_undegraded":"2024-03-28T23:34:38.781366+0000","last_fullsized":"2024-03-28T23:34:38.781366+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:30:07.389688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782089+0000","last_change":"2024-03-28T23:34:37.459502+0000","last_active":"2024-03-28T23:34:38.782089+0000","last_peered":"2024-03-28T23:34:38.782089+0000","last_clean":"2024-03-28T23:34:38.782089+0000","last_became_active":"2024-03-28T23:34:37.459215+0000","last_became_peered":"2024-03-28T23:34:37.459215+0000","last_unstale":"2024-03-28T23:34:38.782089+0000","last_undegraded":"2024-03-28T23:34:38.782089+0000","last_fullsized":"2024-03-28T23:34:38.782089+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:32.284640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"121'2","reported_seq":16,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782334+0000","last_change":"2024-03-28T23:34:36.455380+0000","last_active":"2024-03-28T23:34:38.782334+0000","last_peered":"2024-03-28T23:34:38.782334+0000","last_clean":"2024-03-28T23:34:38.782334+0000","last_became_active":"2024-03-28T23:34:36.455138+0000","last_became_peered":"2024-03-28T23:34:36.455138+0000","last_unstale":"2024-03-28T23:34:38.782334+0000","last_undegraded":"2024-03-28T23:34:38.782334+0000","last_fullsized":"2024-03-28T23:34:38.782334+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:40:40.550780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.467567+0000","last_change":"2024-03-28T23:34:37.478064+0000","last_active":"2024-03-28T23:34:38.467567+0000","last_peered":"2024-03-28T23:34:38.467567+0000","last_clean":"2024-03-28T23:34:38.467567+0000","last_became_active":"2024-03-28T23:34:37.476145+0000","last_became_peered":"2024-03-28T23:34:37.476145+0000","last_unstale":"2024-03-28T23:34:38.467567+0000","last_undegraded":"2024-03-28T23:34:38.467567+0000","last_fullsized":"2024-03-28T23:34:38.467567+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:36:12.915872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466750+0000","last_change":"2024-03-28T23:34:36.456217+0000","last_active":"2024-03-28T23:34:38.466750+0000","last_peered":"2024-03-28T23:34:38.466750+0000","last_clean":"2024-03-28T23:34:38.466750+0000","last_became_active":"2024-03-28T23:34:36.455926+0000","last_became_peered":"2024-03-28T23:34:36.455926+0000","last_unstale":"2024-03-28T23:34:38.466750+0000","last_undegraded":"2024-03-28T23:34:38.466750+0000","last_fullsized":"2024-03-28T23:34:38.466750+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:04:10.240355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781221+0000","last_change":"2024-03-28T23:34:37.469842+0000","last_active":"2024-03-28T23:34:38.781221+0000","last_peered":"2024-03-28T23:34:38.781221+0000","last_clean":"2024-03-28T23:34:38.781221+0000","last_became_active":"2024-03-28T23:34:37.469558+0000","last_became_peered":"2024-03-28T23:34:37.469558+0000","last_unstale":"2024-03-28T23:34:38.781221+0000","last_undegraded":"2024-03-28T23:34:38.781221+0000","last_fullsized":"2024-03-28T23:34:38.781221+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:28:23.658834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466219+0000","last_change":"2024-03-28T23:34:36.459124+0000","last_active":"2024-03-28T23:34:38.466219+0000","last_peered":"2024-03-28T23:34:38.466219+0000","last_clean":"2024-03-28T23:34:38.466219+0000","last_became_active":"2024-03-28T23:34:36.458755+0000","last_became_peered":"2024-03-28T23:34:36.458755+0000","last_unstale":"2024-03-28T23:34:38.466219+0000","last_undegraded":"2024-03-28T23:34:38.466219+0000","last_fullsized":"2024-03-28T23:34:38.466219+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:30:07.389688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781756+0000","last_change":"2024-03-28T23:34:37.462480+0000","last_active":"2024-03-28T23:34:38.781756+0000","last_peered":"2024-03-28T23:34:38.781756+0000","last_clean":"2024-03-28T23:34:38.781756+0000","last_became_active":"2024-03-28T23:34:37.462174+0000","last_became_peered":"2024-03-28T23:34:37.462174+0000","last_unstale":"2024-03-28T23:34:38.781756+0000","last_undegraded":"2024-03-28T23:34:38.781756+0000","last_fullsized":"2024-03-28T23:34:38.781756+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:23.487869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781271+0000","last_change":"2024-03-28T23:34:36.458354+0000","last_active":"2024-03-28T23:34:38.781271+0000","last_peered":"2024-03-28T23:34:38.781271+0000","last_clean":"2024-03-28T23:34:38.781271+0000","last_became_active":"2024-03-28T23:34:36.458062+0000","last_became_peered":"2024-03-28T23:34:36.458062+0000","last_unstale":"2024-03-28T23:34:38.781271+0000","last_undegraded":"2024-03-28T23:34:38.781271+0000","last_fullsized":"2024-03-28T23:34:38.781271+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:04:10.240355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781174+0000","last_change":"2024-03-28T23:34:37.463095+0000","last_active":"2024-03-28T23:34:38.781174+0000","last_peered":"2024-03-28T23:34:38.781174+0000","last_clean":"2024-03-28T23:34:38.781174+0000","last_became_active":"2024-03-28T23:34:37.462783+0000","last_became_peered":"2024-03-28T23:34:37.462783+0000","last_unstale":"2024-03-28T23:34:38.781174+0000","last_undegraded":"2024-03-28T23:34:38.781174+0000","last_fullsized":"2024-03-28T23:34:38.781174+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:01.408086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466097+0000","last_change":"2024-03-28T23:34:36.450794+0000","last_active":"2024-03-28T23:34:38.466097+0000","last_peered":"2024-03-28T23:34:38.466097+0000","last_clean":"2024-03-28T23:34:38.466097+0000","last_became_active":"2024-03-28T23:34:36.450523+0000","last_became_peered":"2024-03-28T23:34:36.450523+0000","last_unstale":"2024-03-28T23:34:38.466097+0000","last_undegraded":"2024-03-28T23:34:38.466097+0000","last_fullsized":"2024-03-28T23:34:38.466097+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:43:35.882626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781776+0000","last_change":"2024-03-28T23:34:37.463134+0000","last_active":"2024-03-28T23:34:38.781776+0000","last_peered":"2024-03-28T23:34:38.781776+0000","last_clean":"2024-03-28T23:34:38.781776+0000","last_became_active":"2024-03-28T23:34:37.462602+0000","last_became_peered":"2024-03-28T23:34:37.462602+0000","last_unstale":"2024-03-28T23:34:38.781776+0000","last_undegraded":"2024-03-28T23:34:38.781776+0000","last_fullsized":"2024-03-28T23:34:38.781776+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:51:29.849040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781462+0000","last_change":"2024-03-28T23:34:36.454661+0000","last_active":"2024-03-28T23:34:38.781462+0000","last_peered":"2024-03-28T23:34:38.781462+0000","last_clean":"2024-03-28T23:34:38.781462+0000","last_became_active":"2024-03-28T23:34:36.454375+0000","last_became_peered":"2024-03-28T23:34:36.454375+0000","last_unstale":"2024-03-28T23:34:38.781462+0000","last_undegraded":"2024-03-28T23:34:38.781462+0000","last_fullsized":"2024-03-28T23:34:38.781462+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:28:48.923891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782346+0000","last_change":"2024-03-28T23:34:37.460449+0000","last_active":"2024-03-28T23:34:38.782346+0000","last_peered":"2024-03-28T23:34:38.782346+0000","last_clean":"2024-03-28T23:34:38.782346+0000","last_became_active":"2024-03-28T23:34:37.460160+0000","last_became_peered":"2024-03-28T23:34:37.460160+0000","last_unstale":"2024-03-28T23:34:38.782346+0000","last_undegraded":"2024-03-28T23:34:38.782346+0000","last_fullsized":"2024-03-28T23:34:38.782346+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:47:47.623344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782194+0000","last_change":"2024-03-28T23:34:36.455971+0000","last_active":"2024-03-28T23:34:38.782194+0000","last_peered":"2024-03-28T23:34:38.782194+0000","last_clean":"2024-03-28T23:34:38.782194+0000","last_became_active":"2024-03-28T23:34:36.455683+0000","last_became_peered":"2024-03-28T23:34:36.455683+0000","last_unstale":"2024-03-28T23:34:38.782194+0000","last_undegraded":"2024-03-28T23:34:38.782194+0000","last_fullsized":"2024-03-28T23:34:38.782194+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:43:35.882626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781747+0000","last_change":"2024-03-28T23:34:37.471183+0000","last_active":"2024-03-28T23:34:38.781747+0000","last_peered":"2024-03-28T23:34:38.781747+0000","last_clean":"2024-03-28T23:34:38.781747+0000","last_became_active":"2024-03-28T23:34:37.470909+0000","last_became_peered":"2024-03-28T23:34:37.470909+0000","last_unstale":"2024-03-28T23:34:38.781747+0000","last_undegraded":"2024-03-28T23:34:38.781747+0000","last_fullsized":"2024-03-28T23:34:38.781747+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:08:12.545863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782175+0000","last_change":"2024-03-28T23:34:36.455827+0000","last_active":"2024-03-28T23:34:38.782175+0000","last_peered":"2024-03-28T23:34:38.782175+0000","last_clean":"2024-03-28T23:34:38.782175+0000","last_became_active":"2024-03-28T23:34:36.455548+0000","last_became_peered":"2024-03-28T23:34:36.455548+0000","last_unstale":"2024-03-28T23:34:38.782175+0000","last_undegraded":"2024-03-28T23:34:38.782175+0000","last_fullsized":"2024-03-28T23:34:38.782175+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:22:02.335132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.467946+0000","last_change":"2024-03-28T23:34:37.469642+0000","last_active":"2024-03-28T23:34:38.467946+0000","last_peered":"2024-03-28T23:34:38.467946+0000","last_clean":"2024-03-28T23:34:38.467946+0000","last_became_active":"2024-03-28T23:34:37.469376+0000","last_became_peered":"2024-03-28T23:34:37.469376+0000","last_unstale":"2024-03-28T23:34:38.467946+0000","last_undegraded":"2024-03-28T23:34:38.467946+0000","last_fullsized":"2024-03-28T23:34:38.467946+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:04:11.245971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781363+0000","last_change":"2024-03-28T23:34:36.460170+0000","last_active":"2024-03-28T23:34:38.781363+0000","last_peered":"2024-03-28T23:34:38.781363+0000","last_clean":"2024-03-28T23:34:38.781363+0000","last_became_active":"2024-03-28T23:34:36.459872+0000","last_became_peered":"2024-03-28T23:34:36.459872+0000","last_unstale":"2024-03-28T23:34:38.781363+0000","last_undegraded":"2024-03-28T23:34:38.781363+0000","last_fullsized":"2024-03-28T23:34:38.781363+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:11:24.838764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":363,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781069+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:34:38.781069+0000","last_peered":"2024-03-28T23:34:38.781069+0000","last_clean":"2024-03-28T23:34:38.781069+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:34:38.781069+0000","last_undegraded":"2024-03-28T23:34:38.781069+0000","last_fullsized":"2024-03-28T23:34:38.781069+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32740,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339100,"statfs":{"total":96636764160,"available":96603238400,"internally_reserved":0,"allocated":1597440,"data_stored":1322149,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574911,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32680,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339160,"statfs":{"total":96636764160,"available":96603299840,"internally_reserved":0,"allocated":1536000,"data_stored":737443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574911,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32676,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339164,"statfs":{"total":96636764160,"available":96603303936,"internally_reserved":0,"allocated":1531904,"data_stored":736793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574910,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32672,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":1527808,"data_stored":736771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32684,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339156,"statfs":{"total":96636764160,"available":96603295744,"internally_reserved":0,"allocated":1531904,"data_stored":731761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574910,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32676,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339164,"statfs":{"total":96636764160,"available":96603303936,"internally_reserved":0,"allocated":1523712,"data_stored":730695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32676,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339164,"statfs":{"total":96636764160,"available":96603303936,"internally_reserved":0,"allocated":1531904,"data_stored":736841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32748,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339092,"statfs":{"total":96636764160,"available":96603230208,"internally_reserved":0,"allocated":1601536,"data_stored":1327195,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:34:40.462 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:34:40.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.600+0000 7f9803264640 1 Processor -- start 2024-03-28T23:34:40.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.600+0000 7f9803264640 1 -- start start 2024-03-28T23:34:40.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.601+0000 7f9803264640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97fc152890 0x7f97fc0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:40.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.601+0000 7f9803264640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f97fc0c8ed0 0x7f97fc0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:40.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.601+0000 7f9803264640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f97fc0c5950 0x7f97fc1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:40.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.601+0000 7f9803264640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f97fc0c5db0 con 0x7f97fc0c8ed0 2024-03-28T23:34:40.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.601+0000 7f9803264640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f97fc0c8480 con 0x7f97fc0c5950 2024-03-28T23:34:40.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.601+0000 7f9803264640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f97fc159640 con 0x7f97fc152890 2024-03-28T23:34:40.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.601+0000 7f97fbfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f97fc0c8ed0 0x7f97fc0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:40.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.601+0000 7f97fbfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f97fc0c8ed0 0x7f97fc0c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54550/0 (socket says 172.21.15.161:54550) 2024-03-28T23:34:40.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.601+0000 7f97fbfff640 1 -- 172.21.15.161:0/1784536333 learned_addr learned my addr 172.21.15.161:0/1784536333 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:40.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.601+0000 7f9801000640 1 --2- 172.21.15.161:0/1784536333 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97fc152890 0x7f97fc0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:40.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.601+0000 7f9801801640 1 --2- 172.21.15.161:0/1784536333 >> v2:172.21.15.170:3300/0 conn(0x7f97fc0c5950 0x7f97fc1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:40.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.601+0000 7f97fbfff640 1 -- 172.21.15.161:0/1784536333 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97fc152890 msgr2=0x7f97fc0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:40.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.601+0000 7f97fbfff640 1 --2- 172.21.15.161:0/1784536333 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97fc152890 0x7f97fc0c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.601+0000 7f97fbfff640 1 -- 172.21.15.161:0/1784536333 >> v2:172.21.15.170:3300/0 conn(0x7f97fc0c5950 msgr2=0x7f97fc1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:40.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.601+0000 7f97fbfff640 1 --2- 172.21.15.161:0/1784536333 >> v2:172.21.15.170:3300/0 conn(0x7f97fc0c5950 0x7f97fc1553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.601+0000 7f97fbfff640 1 -- 172.21.15.161:0/1784536333 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97fc158a30 con 0x7f97fc0c8ed0 2024-03-28T23:34:40.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.601+0000 7f9801801640 1 --2- 172.21.15.161:0/1784536333 >> v2:172.21.15.170:3300/0 conn(0x7f97fc0c5950 0x7f97fc1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:40.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.602+0000 7f97fbfff640 1 --2- 172.21.15.161:0/1784536333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97fc0c8ed0 0x7f97fc0c5410 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0c4e90 tx=0x7f97e802b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=332446cd58b857a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:40.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.602+0000 7f97fb7fe640 1 -- 172.21.15.161:0/1784536333 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e8038070 con 0x7f97fc0c8ed0 2024-03-28T23:34:40.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.602+0000 7f97fb7fe640 1 -- 172.21.15.161:0/1784536333 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97e8004030 con 0x7f97fc0c8ed0 2024-03-28T23:34:40.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.602+0000 7f97fb7fe640 1 -- 172.21.15.161:0/1784536333 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e8004460 con 0x7f97fc0c8ed0 2024-03-28T23:34:40.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.603+0000 7f9803264640 1 -- 172.21.15.161:0/1784536333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97fc0c8ed0 msgr2=0x7f97fc0c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:40.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.603+0000 7f9803264640 1 --2- 172.21.15.161:0/1784536333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97fc0c8ed0 0x7f97fc0c5410 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0c4e90 tx=0x7f97e802b7c0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.603+0000 7f9803264640 1 -- 172.21.15.161:0/1784536333 shutdown_connections 2024-03-28T23:34:40.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.603+0000 7f9803264640 1 --2- 172.21.15.161:0/1784536333 >> v2:172.21.15.170:3300/0 conn(0x7f97fc0c5950 0x7f97fc1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.603+0000 7f9803264640 1 --2- 172.21.15.161:0/1784536333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97fc0c8ed0 0x7f97fc0c5410 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.603+0000 7f9803264640 1 --2- 172.21.15.161:0/1784536333 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97fc152890 0x7f97fc0c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.603+0000 7f9803264640 1 -- 172.21.15.161:0/1784536333 >> 172.21.15.161:0/1784536333 conn(0x7f97fc0ac620 msgr2=0x7f97fc0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:40.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.603+0000 7f9803264640 1 -- 172.21.15.161:0/1784536333 shutdown_connections 2024-03-28T23:34:40.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.603+0000 7f9803264640 1 -- 172.21.15.161:0/1784536333 wait complete. 2024-03-28T23:34:40.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.604+0000 7f9803264640 1 Processor -- start 2024-03-28T23:34:40.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.604+0000 7f9803264640 1 -- start start 2024-03-28T23:34:40.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.605+0000 7f9803264640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97fc0c5950 0x7f97fc14e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:40.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.605+0000 7f9803264640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97fc0c8ed0 0x7f97fc1e7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:40.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.605+0000 7f9803264640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97fc152890 0x7f97fc1e8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:40.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.605+0000 7f9803264640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f97fc1592d0 con 0x7f97fc0c5950 2024-03-28T23:34:40.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.605+0000 7f9803264640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f97fc158ff0 con 0x7f97fc0c8ed0 2024-03-28T23:34:40.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.605+0000 7f9803264640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f97fc152c70 con 0x7f97fc152890 2024-03-28T23:34:40.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.605+0000 7f9801000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97fc0c5950 0x7f97fc14e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:40.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.605+0000 7f9801000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97fc0c5950 0x7f97fc14e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54566/0 (socket says 172.21.15.161:54566) 2024-03-28T23:34:40.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.605+0000 7f9801000640 1 -- 172.21.15.161:0/1279365880 learned_addr learned my addr 172.21.15.161:0/1279365880 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:40.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.605+0000 7f97fbfff640 1 --2- 172.21.15.161:0/1279365880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97fc0c8ed0 0x7f97fc1e7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:40.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.605+0000 7f9801801640 1 --2- 172.21.15.161:0/1279365880 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97fc152890 0x7f97fc1e8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:40.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.605+0000 7f9801000640 1 -- 172.21.15.161:0/1279365880 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97fc152890 msgr2=0x7f97fc1e8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:40.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.605+0000 7f9801000640 1 --2- 172.21.15.161:0/1279365880 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97fc152890 0x7f97fc1e8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.605+0000 7f9801000640 1 -- 172.21.15.161:0/1279365880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97fc0c8ed0 msgr2=0x7f97fc1e7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:40.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.605+0000 7f9801000640 1 --2- 172.21.15.161:0/1279365880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97fc0c8ed0 0x7f97fc1e7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.605+0000 7f9801000640 1 -- 172.21.15.161:0/1279365880 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97fc1e8860 con 0x7f97fc0c5950 2024-03-28T23:34:40.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.605+0000 7f9801000640 1 --2- 172.21.15.161:0/1279365880 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97fc0c5950 0x7f97fc14e080 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f97f000e9b0 tx=0x7f97f000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:40.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.606+0000 7f97f97fa640 1 -- 172.21.15.161:0/1279365880 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f0017070 con 0x7f97fc0c5950 2024-03-28T23:34:40.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.606+0000 7f97f97fa640 1 -- 172.21.15.161:0/1279365880 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97f000c9a0 con 0x7f97fc0c5950 2024-03-28T23:34:40.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.606+0000 7f97f97fa640 1 -- 172.21.15.161:0/1279365880 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f000cd90 con 0x7f97fc0c5950 2024-03-28T23:34:40.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.606+0000 7f9803264640 1 -- 172.21.15.161:0/1279365880 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97fc1ee2f0 con 0x7f97fc0c5950 2024-03-28T23:34:40.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.606+0000 7f9803264640 1 -- 172.21.15.161:0/1279365880 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97fc1ee870 con 0x7f97fc0c5950 2024-03-28T23:34:40.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.607+0000 7f9803264640 1 -- 172.21.15.161:0/1279365880 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97fc04f4b0 con 0x7f97fc0c5950 2024-03-28T23:34:40.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.607+0000 7f97f97fa640 1 -- 172.21.15.161:0/1279365880 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f97f001a070 con 0x7f97fc0c5950 2024-03-28T23:34:40.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.608+0000 7f97f97fa640 1 --2- 172.21.15.161:0/1279365880 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f97e006b4d0 0x7f97e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:40.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.608+0000 7f97fbfff640 1 --2- 172.21.15.161:0/1279365880 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f97e006b4d0 0x7f97e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:40.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.608+0000 7f97f97fa640 1 -- 172.21.15.161:0/1279365880 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6101+0+0 (secure 0 0 0) 0x7f97f008a290 con 0x7f97fc0c5950 2024-03-28T23:34:40.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.608+0000 7f97fbfff640 1 --2- 172.21.15.161:0/1279365880 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f97e006b4d0 0x7f97e006d990 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f97e8035000 tx=0x7f97e8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:40.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.612+0000 7f97f97fa640 1 -- 172.21.15.161:0/1279365880 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f97fc1e8860 con 0x7f97fc0c5950 2024-03-28T23:34:40.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.782+0000 7f9803264640 1 -- 172.21.15.161:0/1279365880 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f97fc0ac400 con 0x7f97e006b4d0 2024-03-28T23:34:40.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.787+0000 7f97f97fa640 1 -- 172.21.15.161:0/1279365880 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252555 (secure 0 0 0) 0x7f97fc0ac400 con 0x7f97e006b4d0 2024-03-28T23:34:40.790 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:40.792 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:34:40.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.791+0000 7f9803264640 1 -- 172.21.15.161:0/1279365880 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f97e006b4d0 msgr2=0x7f97e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:40.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.791+0000 7f9803264640 1 --2- 172.21.15.161:0/1279365880 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f97e006b4d0 0x7f97e006d990 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f97e8035000 tx=0x7f97e8030000 comp rx=0 tx=0).stop 2024-03-28T23:34:40.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.791+0000 7f9803264640 1 -- 172.21.15.161:0/1279365880 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97fc0c5950 msgr2=0x7f97fc14e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:40.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.791+0000 7f9803264640 1 --2- 172.21.15.161:0/1279365880 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97fc0c5950 0x7f97fc14e080 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f97f000e9b0 tx=0x7f97f000ee70 comp rx=0 tx=0).stop 2024-03-28T23:34:40.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.791+0000 7f9803264640 1 -- 172.21.15.161:0/1279365880 shutdown_connections 2024-03-28T23:34:40.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.791+0000 7f9803264640 1 --2- 172.21.15.161:0/1279365880 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f97e006b4d0 0x7f97e006d990 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.791+0000 7f9803264640 1 --2- 172.21.15.161:0/1279365880 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97fc152890 0x7f97fc1e8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.791+0000 7f9803264640 1 --2- 172.21.15.161:0/1279365880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97fc0c8ed0 0x7f97fc1e7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.791+0000 7f9803264640 1 --2- 172.21.15.161:0/1279365880 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97fc0c5950 0x7f97fc14e080 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.791+0000 7f9803264640 1 -- 172.21.15.161:0/1279365880 >> 172.21.15.161:0/1279365880 conn(0x7f97fc0ac620 msgr2=0x7f97fc06c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:40.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.791+0000 7f9803264640 1 -- 172.21.15.161:0/1279365880 shutdown_connections 2024-03-28T23:34:40.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.791+0000 7f9803264640 1 -- 172.21.15.161:0/1279365880 wait complete. 2024-03-28T23:34:40.802 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2024-03-28T23:34:39.367840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":261552,"kb_used_data":12092,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713168,"statfs":{"total":773094113280,"available":772826284032,"internally_reserved":0,"allocated":12382208,"data_stored":7059648,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000743"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782365+0000","last_change":"2024-03-28T23:34:37.467867+0000","last_active":"2024-03-28T23:34:38.782365+0000","last_peered":"2024-03-28T23:34:38.782365+0000","last_clean":"2024-03-28T23:34:38.782365+0000","last_became_active":"2024-03-28T23:34:37.467387+0000","last_became_peered":"2024-03-28T23:34:37.467387+0000","last_unstale":"2024-03-28T23:34:38.782365+0000","last_undegraded":"2024-03-28T23:34:38.782365+0000","last_fullsized":"2024-03-28T23:34:38.782365+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:18.855042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466585+0000","last_change":"2024-03-28T23:34:37.461910+0000","last_active":"2024-03-28T23:34:38.466585+0000","last_peered":"2024-03-28T23:34:38.466585+0000","last_clean":"2024-03-28T23:34:38.466585+0000","last_became_active":"2024-03-28T23:34:37.461637+0000","last_became_peered":"2024-03-28T23:34:37.461637+0000","last_unstale":"2024-03-28T23:34:38.466585+0000","last_undegraded":"2024-03-28T23:34:38.466585+0000","last_fullsized":"2024-03-28T23:34:38.466585+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:21:21.872575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.467916+0000","last_change":"2024-03-28T23:34:37.470292+0000","last_active":"2024-03-28T23:34:38.467916+0000","last_peered":"2024-03-28T23:34:38.467916+0000","last_clean":"2024-03-28T23:34:38.467916+0000","last_became_active":"2024-03-28T23:34:37.470023+0000","last_became_peered":"2024-03-28T23:34:37.470023+0000","last_unstale":"2024-03-28T23:34:38.467916+0000","last_undegraded":"2024-03-28T23:34:38.467916+0000","last_fullsized":"2024-03-28T23:34:38.467916+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:30:08.395305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466877+0000","last_change":"2024-03-28T23:34:37.457378+0000","last_active":"2024-03-28T23:34:38.466877+0000","last_peered":"2024-03-28T23:34:38.466877+0000","last_clean":"2024-03-28T23:34:38.466877+0000","last_became_active":"2024-03-28T23:34:37.457104+0000","last_became_peered":"2024-03-28T23:34:37.457104+0000","last_unstale":"2024-03-28T23:34:38.466877+0000","last_undegraded":"2024-03-28T23:34:38.466877+0000","last_fullsized":"2024-03-28T23:34:38.466877+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:13.997477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781529+0000","last_change":"2024-03-28T23:34:37.471264+0000","last_active":"2024-03-28T23:34:38.781529+0000","last_peered":"2024-03-28T23:34:38.781529+0000","last_clean":"2024-03-28T23:34:38.781529+0000","last_became_active":"2024-03-28T23:34:37.470794+0000","last_became_peered":"2024-03-28T23:34:37.470794+0000","last_unstale":"2024-03-28T23:34:38.781529+0000","last_undegraded":"2024-03-28T23:34:38.781529+0000","last_fullsized":"2024-03-28T23:34:38.781529+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:30:00.138855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781492+0000","last_change":"2024-03-28T23:34:37.472995+0000","last_active":"2024-03-28T23:34:38.781492+0000","last_peered":"2024-03-28T23:34:38.781492+0000","last_clean":"2024-03-28T23:34:38.781492+0000","last_became_active":"2024-03-28T23:34:37.472728+0000","last_became_peered":"2024-03-28T23:34:37.472728+0000","last_unstale":"2024-03-28T23:34:38.781492+0000","last_undegraded":"2024-03-28T23:34:38.781492+0000","last_fullsized":"2024-03-28T23:34:38.781492+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:35:53.095955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781256+0000","last_change":"2024-03-28T23:34:37.470521+0000","last_active":"2024-03-28T23:34:38.781256+0000","last_peered":"2024-03-28T23:34:38.781256+0000","last_clean":"2024-03-28T23:34:38.781256+0000","last_became_active":"2024-03-28T23:34:37.470246+0000","last_became_peered":"2024-03-28T23:34:37.470246+0000","last_unstale":"2024-03-28T23:34:38.781256+0000","last_undegraded":"2024-03-28T23:34:38.781256+0000","last_fullsized":"2024-03-28T23:34:38.781256+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:47:47.623344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466349+0000","last_change":"2024-03-28T23:34:37.460567+0000","last_active":"2024-03-28T23:34:38.466349+0000","last_peered":"2024-03-28T23:34:38.466349+0000","last_clean":"2024-03-28T23:34:38.466349+0000","last_became_active":"2024-03-28T23:34:37.460293+0000","last_became_peered":"2024-03-28T23:34:37.460293+0000","last_unstale":"2024-03-28T23:34:38.466349+0000","last_undegraded":"2024-03-28T23:34:38.466349+0000","last_fullsized":"2024-03-28T23:34:38.466349+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:30:06.874314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781214+0000","last_change":"2024-03-28T23:34:37.467651+0000","last_active":"2024-03-28T23:34:38.781214+0000","last_peered":"2024-03-28T23:34:38.781214+0000","last_clean":"2024-03-28T23:34:38.781214+0000","last_became_active":"2024-03-28T23:34:37.465619+0000","last_became_peered":"2024-03-28T23:34:37.465619+0000","last_unstale":"2024-03-28T23:34:38.781214+0000","last_undegraded":"2024-03-28T23:34:38.781214+0000","last_fullsized":"2024-03-28T23:34:38.781214+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:13.997477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781576+0000","last_change":"2024-03-28T23:34:37.463391+0000","last_active":"2024-03-28T23:34:38.781576+0000","last_peered":"2024-03-28T23:34:38.781576+0000","last_clean":"2024-03-28T23:34:38.781576+0000","last_became_active":"2024-03-28T23:34:37.463119+0000","last_became_peered":"2024-03-28T23:34:37.463119+0000","last_unstale":"2024-03-28T23:34:38.781576+0000","last_undegraded":"2024-03-28T23:34:38.781576+0000","last_fullsized":"2024-03-28T23:34:38.781576+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:58:07.787997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.465928+0000","last_change":"2024-03-28T23:34:37.473059+0000","last_active":"2024-03-28T23:34:38.465928+0000","last_peered":"2024-03-28T23:34:38.465928+0000","last_clean":"2024-03-28T23:34:38.465928+0000","last_became_active":"2024-03-28T23:34:37.472762+0000","last_became_peered":"2024-03-28T23:34:37.472762+0000","last_unstale":"2024-03-28T23:34:38.465928+0000","last_undegraded":"2024-03-28T23:34:38.465928+0000","last_fullsized":"2024-03-28T23:34:38.465928+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:42:48.983466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781962+0000","last_change":"2024-03-28T23:34:37.461291+0000","last_active":"2024-03-28T23:34:38.781962+0000","last_peered":"2024-03-28T23:34:38.781962+0000","last_clean":"2024-03-28T23:34:38.781962+0000","last_became_active":"2024-03-28T23:34:37.460978+0000","last_became_peered":"2024-03-28T23:34:37.460978+0000","last_unstale":"2024-03-28T23:34:38.781962+0000","last_undegraded":"2024-03-28T23:34:38.781962+0000","last_fullsized":"2024-03-28T23:34:38.781962+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:27:04.373061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782562+0000","last_change":"2024-03-28T23:34:37.473983+0000","last_active":"2024-03-28T23:34:38.782562+0000","last_peered":"2024-03-28T23:34:38.782562+0000","last_clean":"2024-03-28T23:34:38.782562+0000","last_became_active":"2024-03-28T23:34:37.473395+0000","last_became_peered":"2024-03-28T23:34:37.473395+0000","last_unstale":"2024-03-28T23:34:38.782562+0000","last_undegraded":"2024-03-28T23:34:38.782562+0000","last_fullsized":"2024-03-28T23:34:38.782562+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:03:48.824778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781724+0000","last_change":"2024-03-28T23:34:37.469706+0000","last_active":"2024-03-28T23:34:38.781724+0000","last_peered":"2024-03-28T23:34:38.781724+0000","last_clean":"2024-03-28T23:34:38.781724+0000","last_became_active":"2024-03-28T23:34:37.469428+0000","last_became_peered":"2024-03-28T23:34:37.469428+0000","last_unstale":"2024-03-28T23:34:38.781724+0000","last_undegraded":"2024-03-28T23:34:38.781724+0000","last_fullsized":"2024-03-28T23:34:38.781724+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:25:16.277513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781365+0000","last_change":"2024-03-28T23:34:37.471525+0000","last_active":"2024-03-28T23:34:38.781365+0000","last_peered":"2024-03-28T23:34:38.781365+0000","last_clean":"2024-03-28T23:34:38.781365+0000","last_became_active":"2024-03-28T23:34:37.471278+0000","last_became_peered":"2024-03-28T23:34:37.471278+0000","last_unstale":"2024-03-28T23:34:38.781365+0000","last_undegraded":"2024-03-28T23:34:38.781365+0000","last_fullsized":"2024-03-28T23:34:38.781365+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:03:35.705086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781057+0000","last_change":"2024-03-28T23:34:37.467233+0000","last_active":"2024-03-28T23:34:38.781057+0000","last_peered":"2024-03-28T23:34:38.781057+0000","last_clean":"2024-03-28T23:34:38.781057+0000","last_became_active":"2024-03-28T23:34:37.466763+0000","last_became_peered":"2024-03-28T23:34:37.466763+0000","last_unstale":"2024-03-28T23:34:38.781057+0000","last_undegraded":"2024-03-28T23:34:38.781057+0000","last_fullsized":"2024-03-28T23:34:38.781057+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:47:28.781650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781341+0000","last_change":"2024-03-28T23:34:37.466118+0000","last_active":"2024-03-28T23:34:38.781341+0000","last_peered":"2024-03-28T23:34:38.781341+0000","last_clean":"2024-03-28T23:34:38.781341+0000","last_became_active":"2024-03-28T23:34:37.465845+0000","last_became_peered":"2024-03-28T23:34:37.465845+0000","last_unstale":"2024-03-28T23:34:38.781341+0000","last_undegraded":"2024-03-28T23:34:38.781341+0000","last_fullsized":"2024-03-28T23:34:38.781341+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:31:35.930069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782627+0000","last_change":"2024-03-28T23:34:37.474500+0000","last_active":"2024-03-28T23:34:38.782627+0000","last_peered":"2024-03-28T23:34:38.782627+0000","last_clean":"2024-03-28T23:34:38.782627+0000","last_became_active":"2024-03-28T23:34:37.474199+0000","last_became_peered":"2024-03-28T23:34:37.474199+0000","last_unstale":"2024-03-28T23:34:38.782627+0000","last_undegraded":"2024-03-28T23:34:38.782627+0000","last_fullsized":"2024-03-28T23:34:38.782627+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:06:07.825526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781611+0000","last_change":"2024-03-28T23:34:37.467214+0000","last_active":"2024-03-28T23:34:38.781611+0000","last_peered":"2024-03-28T23:34:38.781611+0000","last_clean":"2024-03-28T23:34:38.781611+0000","last_became_active":"2024-03-28T23:34:37.466797+0000","last_became_peered":"2024-03-28T23:34:37.466797+0000","last_unstale":"2024-03-28T23:34:38.781611+0000","last_undegraded":"2024-03-28T23:34:38.781611+0000","last_fullsized":"2024-03-28T23:34:38.781611+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:32.284640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782589+0000","last_change":"2024-03-28T23:34:37.461075+0000","last_active":"2024-03-28T23:34:38.782589+0000","last_peered":"2024-03-28T23:34:38.782589+0000","last_clean":"2024-03-28T23:34:38.782589+0000","last_became_active":"2024-03-28T23:34:37.460798+0000","last_became_peered":"2024-03-28T23:34:37.460798+0000","last_unstale":"2024-03-28T23:34:38.782589+0000","last_undegraded":"2024-03-28T23:34:38.782589+0000","last_fullsized":"2024-03-28T23:34:38.782589+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:08:12.545863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.465991+0000","last_change":"2024-03-28T23:34:37.472505+0000","last_active":"2024-03-28T23:34:38.465991+0000","last_peered":"2024-03-28T23:34:38.465991+0000","last_clean":"2024-03-28T23:34:38.465991+0000","last_became_active":"2024-03-28T23:34:37.472063+0000","last_became_peered":"2024-03-28T23:34:37.472063+0000","last_unstale":"2024-03-28T23:34:38.465991+0000","last_undegraded":"2024-03-28T23:34:38.465991+0000","last_fullsized":"2024-03-28T23:34:38.465991+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:26.729239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781583+0000","last_change":"2024-03-28T23:34:37.469192+0000","last_active":"2024-03-28T23:34:38.781583+0000","last_peered":"2024-03-28T23:34:38.781583+0000","last_clean":"2024-03-28T23:34:38.781583+0000","last_became_active":"2024-03-28T23:34:37.468171+0000","last_became_peered":"2024-03-28T23:34:37.468171+0000","last_unstale":"2024-03-28T23:34:38.781583+0000","last_undegraded":"2024-03-28T23:34:38.781583+0000","last_fullsized":"2024-03-28T23:34:38.781583+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:51.400972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466565+0000","last_change":"2024-03-28T23:34:37.461758+0000","last_active":"2024-03-28T23:34:38.466565+0000","last_peered":"2024-03-28T23:34:38.466565+0000","last_clean":"2024-03-28T23:34:38.466565+0000","last_became_active":"2024-03-28T23:34:37.461240+0000","last_became_peered":"2024-03-28T23:34:37.461240+0000","last_unstale":"2024-03-28T23:34:38.466565+0000","last_undegraded":"2024-03-28T23:34:38.466565+0000","last_fullsized":"2024-03-28T23:34:38.466565+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:01.408086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781188+0000","last_change":"2024-03-28T23:34:37.467029+0000","last_active":"2024-03-28T23:34:38.781188+0000","last_peered":"2024-03-28T23:34:38.781188+0000","last_clean":"2024-03-28T23:34:38.781188+0000","last_became_active":"2024-03-28T23:34:37.466414+0000","last_became_peered":"2024-03-28T23:34:37.466414+0000","last_unstale":"2024-03-28T23:34:38.781188+0000","last_undegraded":"2024-03-28T23:34:38.781188+0000","last_fullsized":"2024-03-28T23:34:38.781188+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:21:21.872575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.468190+0000","last_change":"2024-03-28T23:34:37.472238+0000","last_active":"2024-03-28T23:34:38.468190+0000","last_peered":"2024-03-28T23:34:38.468190+0000","last_clean":"2024-03-28T23:34:38.468190+0000","last_became_active":"2024-03-28T23:34:37.471771+0000","last_became_peered":"2024-03-28T23:34:37.471771+0000","last_unstale":"2024-03-28T23:34:38.468190+0000","last_undegraded":"2024-03-28T23:34:38.468190+0000","last_fullsized":"2024-03-28T23:34:38.468190+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:43:36.888242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.467812+0000","last_change":"2024-03-28T23:34:37.469369+0000","last_active":"2024-03-28T23:34:38.467812+0000","last_peered":"2024-03-28T23:34:38.467812+0000","last_clean":"2024-03-28T23:34:38.467812+0000","last_became_active":"2024-03-28T23:34:37.469026+0000","last_became_peered":"2024-03-28T23:34:37.469026+0000","last_unstale":"2024-03-28T23:34:38.467812+0000","last_undegraded":"2024-03-28T23:34:38.467812+0000","last_fullsized":"2024-03-28T23:34:38.467812+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:52:21.097148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781821+0000","last_change":"2024-03-28T23:34:37.475553+0000","last_active":"2024-03-28T23:34:38.781821+0000","last_peered":"2024-03-28T23:34:38.781821+0000","last_clean":"2024-03-28T23:34:38.781821+0000","last_became_active":"2024-03-28T23:34:37.475280+0000","last_became_peered":"2024-03-28T23:34:37.475280+0000","last_unstale":"2024-03-28T23:34:38.781821+0000","last_undegraded":"2024-03-28T23:34:38.781821+0000","last_fullsized":"2024-03-28T23:34:38.781821+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:34:54.305199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782819+0000","last_change":"2024-03-28T23:34:37.462192+0000","last_active":"2024-03-28T23:34:38.782819+0000","last_peered":"2024-03-28T23:34:38.782819+0000","last_clean":"2024-03-28T23:34:38.782819+0000","last_became_active":"2024-03-28T23:34:37.461921+0000","last_became_peered":"2024-03-28T23:34:37.461921+0000","last_unstale":"2024-03-28T23:34:38.782819+0000","last_undegraded":"2024-03-28T23:34:38.782819+0000","last_fullsized":"2024-03-28T23:34:38.782819+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:41.535721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.467819+0000","last_change":"2024-03-28T23:34:37.477937+0000","last_active":"2024-03-28T23:34:38.467819+0000","last_peered":"2024-03-28T23:34:38.467819+0000","last_clean":"2024-03-28T23:34:38.467819+0000","last_became_active":"2024-03-28T23:34:37.476449+0000","last_became_peered":"2024-03-28T23:34:37.476449+0000","last_unstale":"2024-03-28T23:34:38.467819+0000","last_undegraded":"2024-03-28T23:34:38.467819+0000","last_fullsized":"2024-03-28T23:34:38.467819+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:57.794609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.467858+0000","last_change":"2024-03-28T23:34:37.471383+0000","last_active":"2024-03-28T23:34:38.467858+0000","last_peered":"2024-03-28T23:34:38.467858+0000","last_clean":"2024-03-28T23:34:38.467858+0000","last_became_active":"2024-03-28T23:34:37.471096+0000","last_became_peered":"2024-03-28T23:34:37.471096+0000","last_unstale":"2024-03-28T23:34:38.467858+0000","last_undegraded":"2024-03-28T23:34:38.467858+0000","last_fullsized":"2024-03-28T23:34:38.467858+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:35:11.976716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466308+0000","last_change":"2024-03-28T23:34:37.454074+0000","last_active":"2024-03-28T23:34:38.466308+0000","last_peered":"2024-03-28T23:34:38.466308+0000","last_clean":"2024-03-28T23:34:38.466308+0000","last_became_active":"2024-03-28T23:34:37.453758+0000","last_became_peered":"2024-03-28T23:34:37.453758+0000","last_unstale":"2024-03-28T23:34:38.466308+0000","last_undegraded":"2024-03-28T23:34:38.466308+0000","last_fullsized":"2024-03-28T23:34:38.466308+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:02:47.244704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781168+0000","last_change":"2024-03-28T23:34:37.457986+0000","last_active":"2024-03-28T23:34:38.781168+0000","last_peered":"2024-03-28T23:34:38.781168+0000","last_clean":"2024-03-28T23:34:38.781168+0000","last_became_active":"2024-03-28T23:34:37.457694+0000","last_became_peered":"2024-03-28T23:34:37.457694+0000","last_unstale":"2024-03-28T23:34:38.781168+0000","last_undegraded":"2024-03-28T23:34:38.781168+0000","last_fullsized":"2024-03-28T23:34:38.781168+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:56:27.639060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466322+0000","last_change":"2024-03-28T23:34:37.462490+0000","last_active":"2024-03-28T23:34:38.466322+0000","last_peered":"2024-03-28T23:34:38.466322+0000","last_clean":"2024-03-28T23:34:38.466322+0000","last_became_active":"2024-03-28T23:34:37.462185+0000","last_became_peered":"2024-03-28T23:34:37.462185+0000","last_unstale":"2024-03-28T23:34:38.466322+0000","last_undegraded":"2024-03-28T23:34:38.466322+0000","last_fullsized":"2024-03-28T23:34:38.466322+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:11:25.844380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"121'4","reported_seq":18,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.468140+0000","last_change":"2024-03-28T23:34:36.457009+0000","last_active":"2024-03-28T23:34:38.468140+0000","last_peered":"2024-03-28T23:34:38.468140+0000","last_clean":"2024-03-28T23:34:38.468140+0000","last_became_active":"2024-03-28T23:34:36.456735+0000","last_became_peered":"2024-03-28T23:34:36.456735+0000","last_unstale":"2024-03-28T23:34:38.468140+0000","last_undegraded":"2024-03-28T23:34:38.468140+0000","last_fullsized":"2024-03-28T23:34:38.468140+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:58.569300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781535+0000","last_change":"2024-03-28T23:34:37.468232+0000","last_active":"2024-03-28T23:34:38.781535+0000","last_peered":"2024-03-28T23:34:38.781535+0000","last_clean":"2024-03-28T23:34:38.781535+0000","last_became_active":"2024-03-28T23:34:37.467912+0000","last_became_peered":"2024-03-28T23:34:37.467912+0000","last_unstale":"2024-03-28T23:34:38.781535+0000","last_undegraded":"2024-03-28T23:34:38.781535+0000","last_fullsized":"2024-03-28T23:34:38.781535+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:36:52.766599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"121'1","reported_seq":15,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782869+0000","last_change":"2024-03-28T23:34:36.448047+0000","last_active":"2024-03-28T23:34:38.782869+0000","last_peered":"2024-03-28T23:34:38.782869+0000","last_clean":"2024-03-28T23:34:38.782869+0000","last_became_active":"2024-03-28T23:34:36.447047+0000","last_became_peered":"2024-03-28T23:34:36.447047+0000","last_unstale":"2024-03-28T23:34:38.782869+0000","last_undegraded":"2024-03-28T23:34:38.782869+0000","last_fullsized":"2024-03-28T23:34:38.782869+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:00.402470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466384+0000","last_change":"2024-03-28T23:34:37.459132+0000","last_active":"2024-03-28T23:34:38.466384+0000","last_peered":"2024-03-28T23:34:38.466384+0000","last_clean":"2024-03-28T23:34:38.466384+0000","last_became_active":"2024-03-28T23:34:37.458861+0000","last_became_peered":"2024-03-28T23:34:37.458861+0000","last_unstale":"2024-03-28T23:34:38.466384+0000","last_undegraded":"2024-03-28T23:34:38.466384+0000","last_fullsized":"2024-03-28T23:34:38.466384+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:47:28.781650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"121'2","reported_seq":16,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781693+0000","last_change":"2024-03-28T23:34:36.455626+0000","last_active":"2024-03-28T23:34:38.781693+0000","last_peered":"2024-03-28T23:34:38.781693+0000","last_clean":"2024-03-28T23:34:38.781693+0000","last_became_active":"2024-03-28T23:34:36.455356+0000","last_became_peered":"2024-03-28T23:34:36.455356+0000","last_unstale":"2024-03-28T23:34:38.781693+0000","last_undegraded":"2024-03-28T23:34:38.781693+0000","last_fullsized":"2024-03-28T23:34:38.781693+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:42:47.977850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781968+0000","last_change":"2024-03-28T23:34:37.470435+0000","last_active":"2024-03-28T23:34:38.781968+0000","last_peered":"2024-03-28T23:34:38.781968+0000","last_clean":"2024-03-28T23:34:38.781968+0000","last_became_active":"2024-03-28T23:34:37.470185+0000","last_became_peered":"2024-03-28T23:34:37.470185+0000","last_unstale":"2024-03-28T23:34:38.781968+0000","last_undegraded":"2024-03-28T23:34:38.781968+0000","last_fullsized":"2024-03-28T23:34:38.781968+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:20:48.203536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"121'2","reported_seq":16,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781393+0000","last_change":"2024-03-28T23:34:36.456195+0000","last_active":"2024-03-28T23:34:38.781393+0000","last_peered":"2024-03-28T23:34:38.781393+0000","last_clean":"2024-03-28T23:34:38.781393+0000","last_became_active":"2024-03-28T23:34:36.455573+0000","last_became_peered":"2024-03-28T23:34:36.455573+0000","last_unstale":"2024-03-28T23:34:38.781393+0000","last_undegraded":"2024-03-28T23:34:38.781393+0000","last_fullsized":"2024-03-28T23:34:38.781393+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:05:34.122514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466353+0000","last_change":"2024-03-28T23:34:37.456376+0000","last_active":"2024-03-28T23:34:38.466353+0000","last_peered":"2024-03-28T23:34:38.466353+0000","last_clean":"2024-03-28T23:34:38.466353+0000","last_became_active":"2024-03-28T23:34:37.456003+0000","last_became_peered":"2024-03-28T23:34:37.456003+0000","last_unstale":"2024-03-28T23:34:38.466353+0000","last_undegraded":"2024-03-28T23:34:38.466353+0000","last_fullsized":"2024-03-28T23:34:38.466353+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:28:49.929508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"121'1","reported_seq":15,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466216+0000","last_change":"2024-03-28T23:34:36.457412+0000","last_active":"2024-03-28T23:34:38.466216+0000","last_peered":"2024-03-28T23:34:38.466216+0000","last_clean":"2024-03-28T23:34:38.466216+0000","last_became_active":"2024-03-28T23:34:36.457131+0000","last_became_peered":"2024-03-28T23:34:36.457131+0000","last_unstale":"2024-03-28T23:34:38.466216+0000","last_undegraded":"2024-03-28T23:34:38.466216+0000","last_fullsized":"2024-03-28T23:34:38.466216+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:35:10.971100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466382+0000","last_change":"2024-03-28T23:34:37.459773+0000","last_active":"2024-03-28T23:34:38.466382+0000","last_peered":"2024-03-28T23:34:38.466382+0000","last_clean":"2024-03-28T23:34:38.466382+0000","last_became_active":"2024-03-28T23:34:37.459471+0000","last_became_peered":"2024-03-28T23:34:37.459471+0000","last_unstale":"2024-03-28T23:34:38.466382+0000","last_undegraded":"2024-03-28T23:34:38.466382+0000","last_fullsized":"2024-03-28T23:34:38.466382+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:42:48.983466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"121'1","reported_seq":15,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782575+0000","last_change":"2024-03-28T23:34:36.453536+0000","last_active":"2024-03-28T23:34:38.782575+0000","last_peered":"2024-03-28T23:34:38.782575+0000","last_clean":"2024-03-28T23:34:38.782575+0000","last_became_active":"2024-03-28T23:34:36.453009+0000","last_became_peered":"2024-03-28T23:34:36.453009+0000","last_unstale":"2024-03-28T23:34:38.782575+0000","last_undegraded":"2024-03-28T23:34:38.782575+0000","last_fullsized":"2024-03-28T23:34:38.782575+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:05:25.723622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782766+0000","last_change":"2024-03-28T23:34:37.460940+0000","last_active":"2024-03-28T23:34:38.782766+0000","last_peered":"2024-03-28T23:34:38.782766+0000","last_clean":"2024-03-28T23:34:38.782766+0000","last_became_active":"2024-03-28T23:34:37.460615+0000","last_became_peered":"2024-03-28T23:34:37.460615+0000","last_unstale":"2024-03-28T23:34:38.782766+0000","last_undegraded":"2024-03-28T23:34:38.782766+0000","last_fullsized":"2024-03-28T23:34:38.782766+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:03:44.505317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781982+0000","last_change":"2024-03-28T23:34:36.461740+0000","last_active":"2024-03-28T23:34:38.781982+0000","last_peered":"2024-03-28T23:34:38.781982+0000","last_clean":"2024-03-28T23:34:38.781982+0000","last_became_active":"2024-03-28T23:34:36.461440+0000","last_became_peered":"2024-03-28T23:34:36.461440+0000","last_unstale":"2024-03-28T23:34:38.781982+0000","last_undegraded":"2024-03-28T23:34:38.781982+0000","last_fullsized":"2024-03-28T23:34:38.781982+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:35:10.971100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781706+0000","last_change":"2024-03-28T23:34:37.475659+0000","last_active":"2024-03-28T23:34:38.781706+0000","last_peered":"2024-03-28T23:34:38.781706+0000","last_clean":"2024-03-28T23:34:38.781706+0000","last_became_active":"2024-03-28T23:34:37.475385+0000","last_became_peered":"2024-03-28T23:34:37.475385+0000","last_unstale":"2024-03-28T23:34:38.781706+0000","last_undegraded":"2024-03-28T23:34:38.781706+0000","last_fullsized":"2024-03-28T23:34:38.781706+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:28:10.183556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.467484+0000","last_change":"2024-03-28T23:34:36.458280+0000","last_active":"2024-03-28T23:34:38.467484+0000","last_peered":"2024-03-28T23:34:38.467484+0000","last_clean":"2024-03-28T23:34:38.467484+0000","last_became_active":"2024-03-28T23:34:36.457939+0000","last_became_peered":"2024-03-28T23:34:36.457939+0000","last_unstale":"2024-03-28T23:34:38.467484+0000","last_undegraded":"2024-03-28T23:34:38.467484+0000","last_fullsized":"2024-03-28T23:34:38.467484+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:00:38.115038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781212+0000","last_change":"2024-03-28T23:34:37.473431+0000","last_active":"2024-03-28T23:34:38.781212+0000","last_peered":"2024-03-28T23:34:38.781212+0000","last_clean":"2024-03-28T23:34:38.781212+0000","last_became_active":"2024-03-28T23:34:37.473042+0000","last_became_peered":"2024-03-28T23:34:37.473042+0000","last_unstale":"2024-03-28T23:34:38.781212+0000","last_undegraded":"2024-03-28T23:34:38.781212+0000","last_fullsized":"2024-03-28T23:34:38.781212+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:51:55.988297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466255+0000","last_change":"2024-03-28T23:34:36.449244+0000","last_active":"2024-03-28T23:34:38.466255+0000","last_peered":"2024-03-28T23:34:38.466255+0000","last_clean":"2024-03-28T23:34:38.466255+0000","last_became_active":"2024-03-28T23:34:36.448890+0000","last_became_peered":"2024-03-28T23:34:36.448890+0000","last_unstale":"2024-03-28T23:34:38.466255+0000","last_undegraded":"2024-03-28T23:34:38.466255+0000","last_fullsized":"2024-03-28T23:34:38.466255+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:22.354809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782114+0000","last_change":"2024-03-28T23:34:37.462559+0000","last_active":"2024-03-28T23:34:38.782114+0000","last_peered":"2024-03-28T23:34:38.782114+0000","last_clean":"2024-03-28T23:34:38.782114+0000","last_became_active":"2024-03-28T23:34:37.460383+0000","last_became_peered":"2024-03-28T23:34:37.460383+0000","last_unstale":"2024-03-28T23:34:38.782114+0000","last_undegraded":"2024-03-28T23:34:38.782114+0000","last_fullsized":"2024-03-28T23:34:38.782114+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:28:23.658834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782065+0000","last_change":"2024-03-28T23:34:36.456062+0000","last_active":"2024-03-28T23:34:38.782065+0000","last_peered":"2024-03-28T23:34:38.782065+0000","last_clean":"2024-03-28T23:34:38.782065+0000","last_became_active":"2024-03-28T23:34:36.455793+0000","last_became_peered":"2024-03-28T23:34:36.455793+0000","last_unstale":"2024-03-28T23:34:38.782065+0000","last_undegraded":"2024-03-28T23:34:38.782065+0000","last_fullsized":"2024-03-28T23:34:38.782065+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:11:40.847837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.467876+0000","last_change":"2024-03-28T23:34:37.471932+0000","last_active":"2024-03-28T23:34:38.467876+0000","last_peered":"2024-03-28T23:34:38.467876+0000","last_clean":"2024-03-28T23:34:38.467876+0000","last_became_active":"2024-03-28T23:34:37.470143+0000","last_became_peered":"2024-03-28T23:34:37.470143+0000","last_unstale":"2024-03-28T23:34:38.467876+0000","last_undegraded":"2024-03-28T23:34:38.467876+0000","last_fullsized":"2024-03-28T23:34:38.467876+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:20.557474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"121'3","reported_seq":17,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781794+0000","last_change":"2024-03-28T23:34:36.457100+0000","last_active":"2024-03-28T23:34:38.781794+0000","last_peered":"2024-03-28T23:34:38.781794+0000","last_clean":"2024-03-28T23:34:38.781794+0000","last_became_active":"2024-03-28T23:34:36.456724+0000","last_became_peered":"2024-03-28T23:34:36.456724+0000","last_unstale":"2024-03-28T23:34:38.781794+0000","last_undegraded":"2024-03-28T23:34:38.781794+0000","last_fullsized":"2024-03-28T23:34:38.781794+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:30:05.868698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.467837+0000","last_change":"2024-03-28T23:34:37.477251+0000","last_active":"2024-03-28T23:34:38.467837+0000","last_peered":"2024-03-28T23:34:38.467837+0000","last_clean":"2024-03-28T23:34:38.467837+0000","last_became_active":"2024-03-28T23:34:37.476655+0000","last_became_peered":"2024-03-28T23:34:37.476655+0000","last_unstale":"2024-03-28T23:34:38.467837+0000","last_undegraded":"2024-03-28T23:34:38.467837+0000","last_fullsized":"2024-03-28T23:34:38.467837+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:48.842819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"121'6","reported_seq":20,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781519+0000","last_change":"2024-03-28T23:34:36.456363+0000","last_active":"2024-03-28T23:34:38.781519+0000","last_peered":"2024-03-28T23:34:38.781519+0000","last_clean":"2024-03-28T23:34:38.781519+0000","last_became_active":"2024-03-28T23:34:36.455864+0000","last_became_peered":"2024-03-28T23:34:36.455864+0000","last_unstale":"2024-03-28T23:34:38.781519+0000","last_undegraded":"2024-03-28T23:34:38.781519+0000","last_fullsized":"2024-03-28T23:34:38.781519+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:47:37.317672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782959+0000","last_change":"2024-03-28T23:34:37.469727+0000","last_active":"2024-03-28T23:34:38.782959+0000","last_peered":"2024-03-28T23:34:38.782959+0000","last_clean":"2024-03-28T23:34:38.782959+0000","last_became_active":"2024-03-28T23:34:37.469464+0000","last_became_peered":"2024-03-28T23:34:37.469464+0000","last_unstale":"2024-03-28T23:34:38.782959+0000","last_undegraded":"2024-03-28T23:34:38.782959+0000","last_fullsized":"2024-03-28T23:34:38.782959+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:42:31.409331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"121'2","reported_seq":16,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782122+0000","last_change":"2024-03-28T23:34:36.457197+0000","last_active":"2024-03-28T23:34:38.782122+0000","last_peered":"2024-03-28T23:34:38.782122+0000","last_clean":"2024-03-28T23:34:38.782122+0000","last_became_active":"2024-03-28T23:34:36.456770+0000","last_became_peered":"2024-03-28T23:34:36.456770+0000","last_unstale":"2024-03-28T23:34:38.782122+0000","last_undegraded":"2024-03-28T23:34:38.782122+0000","last_fullsized":"2024-03-28T23:34:38.782122+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:49:48.121807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781812+0000","last_change":"2024-03-28T23:34:37.462948+0000","last_active":"2024-03-28T23:34:38.781812+0000","last_peered":"2024-03-28T23:34:38.781812+0000","last_clean":"2024-03-28T23:34:38.781812+0000","last_became_active":"2024-03-28T23:34:37.462688+0000","last_became_peered":"2024-03-28T23:34:37.462688+0000","last_unstale":"2024-03-28T23:34:38.781812+0000","last_undegraded":"2024-03-28T23:34:38.781812+0000","last_fullsized":"2024-03-28T23:34:38.781812+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:06:07.825526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.467679+0000","last_change":"2024-03-28T23:34:36.459082+0000","last_active":"2024-03-28T23:34:38.467679+0000","last_peered":"2024-03-28T23:34:38.467679+0000","last_clean":"2024-03-28T23:34:38.467679+0000","last_became_active":"2024-03-28T23:34:36.458802+0000","last_became_peered":"2024-03-28T23:34:36.458802+0000","last_unstale":"2024-03-28T23:34:38.467679+0000","last_undegraded":"2024-03-28T23:34:38.467679+0000","last_fullsized":"2024-03-28T23:34:38.467679+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:17:03.981240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782448+0000","last_change":"2024-03-28T23:34:37.471748+0000","last_active":"2024-03-28T23:34:38.782448+0000","last_peered":"2024-03-28T23:34:38.782448+0000","last_clean":"2024-03-28T23:34:38.782448+0000","last_became_active":"2024-03-28T23:34:37.471486+0000","last_became_peered":"2024-03-28T23:34:37.471486+0000","last_unstale":"2024-03-28T23:34:38.782448+0000","last_undegraded":"2024-03-28T23:34:38.782448+0000","last_fullsized":"2024-03-28T23:34:38.782448+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:28:10.183556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"121'8","reported_seq":22,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781280+0000","last_change":"2024-03-28T23:34:36.458302+0000","last_active":"2024-03-28T23:34:38.781280+0000","last_peered":"2024-03-28T23:34:38.781280+0000","last_clean":"2024-03-28T23:34:38.781280+0000","last_became_active":"2024-03-28T23:34:36.457992+0000","last_became_peered":"2024-03-28T23:34:36.457992+0000","last_unstale":"2024-03-28T23:34:38.781280+0000","last_undegraded":"2024-03-28T23:34:38.781280+0000","last_fullsized":"2024-03-28T23:34:38.781280+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:19.551857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.467334+0000","last_change":"2024-03-28T23:34:37.467808+0000","last_active":"2024-03-28T23:34:38.467334+0000","last_peered":"2024-03-28T23:34:38.467334+0000","last_clean":"2024-03-28T23:34:38.467334+0000","last_became_active":"2024-03-28T23:34:37.467493+0000","last_became_peered":"2024-03-28T23:34:37.467493+0000","last_unstale":"2024-03-28T23:34:38.467334+0000","last_undegraded":"2024-03-28T23:34:38.467334+0000","last_fullsized":"2024-03-28T23:34:38.467334+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:46:55.411569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"121'1","reported_seq":15,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466349+0000","last_change":"2024-03-28T23:34:36.457449+0000","last_active":"2024-03-28T23:34:38.466349+0000","last_peered":"2024-03-28T23:34:38.466349+0000","last_clean":"2024-03-28T23:34:38.466349+0000","last_became_active":"2024-03-28T23:34:36.457041+0000","last_became_peered":"2024-03-28T23:34:36.457041+0000","last_unstale":"2024-03-28T23:34:38.466349+0000","last_undegraded":"2024-03-28T23:34:38.466349+0000","last_fullsized":"2024-03-28T23:34:38.466349+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:54:50.082017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781365+0000","last_change":"2024-03-28T23:34:37.465288+0000","last_active":"2024-03-28T23:34:38.781365+0000","last_peered":"2024-03-28T23:34:38.781365+0000","last_clean":"2024-03-28T23:34:38.781365+0000","last_became_active":"2024-03-28T23:34:37.465019+0000","last_became_peered":"2024-03-28T23:34:37.465019+0000","last_unstale":"2024-03-28T23:34:38.781365+0000","last_undegraded":"2024-03-28T23:34:38.781365+0000","last_fullsized":"2024-03-28T23:34:38.781365+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:03:48.824778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"121'2","reported_seq":16,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.468055+0000","last_change":"2024-03-28T23:34:36.459332+0000","last_active":"2024-03-28T23:34:38.468055+0000","last_peered":"2024-03-28T23:34:38.468055+0000","last_clean":"2024-03-28T23:34:38.468055+0000","last_became_active":"2024-03-28T23:34:36.458972+0000","last_became_peered":"2024-03-28T23:34:36.458972+0000","last_unstale":"2024-03-28T23:34:38.468055+0000","last_undegraded":"2024-03-28T23:34:38.468055+0000","last_fullsized":"2024-03-28T23:34:38.468055+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:55.285139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781518+0000","last_change":"2024-03-28T23:34:37.476449+0000","last_active":"2024-03-28T23:34:38.781518+0000","last_peered":"2024-03-28T23:34:38.781518+0000","last_clean":"2024-03-28T23:34:38.781518+0000","last_became_active":"2024-03-28T23:34:37.475904+0000","last_became_peered":"2024-03-28T23:34:37.475904+0000","last_unstale":"2024-03-28T23:34:38.781518+0000","last_undegraded":"2024-03-28T23:34:38.781518+0000","last_fullsized":"2024-03-28T23:34:38.781518+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:30:00.138855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"121'1","reported_seq":15,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781891+0000","last_change":"2024-03-28T23:34:36.457693+0000","last_active":"2024-03-28T23:34:38.781891+0000","last_peered":"2024-03-28T23:34:38.781891+0000","last_clean":"2024-03-28T23:34:38.781891+0000","last_became_active":"2024-03-28T23:34:36.457199+0000","last_became_peered":"2024-03-28T23:34:36.457199+0000","last_unstale":"2024-03-28T23:34:38.781891+0000","last_undegraded":"2024-03-28T23:34:38.781891+0000","last_fullsized":"2024-03-28T23:34:38.781891+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:22.354809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781609+0000","last_change":"2024-03-28T23:34:37.475892+0000","last_active":"2024-03-28T23:34:38.781609+0000","last_peered":"2024-03-28T23:34:38.781609+0000","last_clean":"2024-03-28T23:34:38.781609+0000","last_became_active":"2024-03-28T23:34:37.475597+0000","last_became_peered":"2024-03-28T23:34:37.475597+0000","last_unstale":"2024-03-28T23:34:38.781609+0000","last_undegraded":"2024-03-28T23:34:38.781609+0000","last_fullsized":"2024-03-28T23:34:38.781609+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:08:55.299299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782692+0000","last_change":"2024-03-28T23:34:36.451020+0000","last_active":"2024-03-28T23:34:38.782692+0000","last_peered":"2024-03-28T23:34:38.782692+0000","last_clean":"2024-03-28T23:34:38.782692+0000","last_became_active":"2024-03-28T23:34:36.450691+0000","last_became_peered":"2024-03-28T23:34:36.450691+0000","last_unstale":"2024-03-28T23:34:38.782692+0000","last_undegraded":"2024-03-28T23:34:38.782692+0000","last_fullsized":"2024-03-28T23:34:38.782692+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:47:27.776034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782703+0000","last_change":"2024-03-28T23:34:37.463237+0000","last_active":"2024-03-28T23:34:38.782703+0000","last_peered":"2024-03-28T23:34:38.782703+0000","last_clean":"2024-03-28T23:34:38.782703+0000","last_became_active":"2024-03-28T23:34:37.462997+0000","last_became_peered":"2024-03-28T23:34:37.462997+0000","last_unstale":"2024-03-28T23:34:38.782703+0000","last_undegraded":"2024-03-28T23:34:38.782703+0000","last_fullsized":"2024-03-28T23:34:38.782703+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:34:54.305199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"121'2","reported_seq":16,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466339+0000","last_change":"2024-03-28T23:34:36.459222+0000","last_active":"2024-03-28T23:34:38.466339+0000","last_peered":"2024-03-28T23:34:38.466339+0000","last_clean":"2024-03-28T23:34:38.466339+0000","last_became_active":"2024-03-28T23:34:36.458928+0000","last_became_peered":"2024-03-28T23:34:36.458928+0000","last_unstale":"2024-03-28T23:34:38.466339+0000","last_undegraded":"2024-03-28T23:34:38.466339+0000","last_fullsized":"2024-03-28T23:34:38.466339+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:19.551857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782280+0000","last_change":"2024-03-28T23:34:37.469032+0000","last_active":"2024-03-28T23:34:38.782280+0000","last_peered":"2024-03-28T23:34:38.782280+0000","last_clean":"2024-03-28T23:34:38.782280+0000","last_became_active":"2024-03-28T23:34:37.468770+0000","last_became_peered":"2024-03-28T23:34:37.468770+0000","last_unstale":"2024-03-28T23:34:38.782280+0000","last_undegraded":"2024-03-28T23:34:38.782280+0000","last_fullsized":"2024-03-28T23:34:38.782280+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:27:23.671506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781813+0000","last_change":"2024-03-28T23:34:36.454376+0000","last_active":"2024-03-28T23:34:38.781813+0000","last_peered":"2024-03-28T23:34:38.781813+0000","last_clean":"2024-03-28T23:34:38.781813+0000","last_became_active":"2024-03-28T23:34:36.453491+0000","last_became_peered":"2024-03-28T23:34:36.453491+0000","last_unstale":"2024-03-28T23:34:38.781813+0000","last_undegraded":"2024-03-28T23:34:38.781813+0000","last_fullsized":"2024-03-28T23:34:38.781813+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:27:35.273330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782852+0000","last_change":"2024-03-28T23:34:37.474290+0000","last_active":"2024-03-28T23:34:38.782852+0000","last_peered":"2024-03-28T23:34:38.782852+0000","last_clean":"2024-03-28T23:34:38.782852+0000","last_became_active":"2024-03-28T23:34:37.473919+0000","last_became_peered":"2024-03-28T23:34:37.473919+0000","last_unstale":"2024-03-28T23:34:38.782852+0000","last_undegraded":"2024-03-28T23:34:38.782852+0000","last_fullsized":"2024-03-28T23:34:38.782852+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:35:42.635666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781383+0000","last_change":"2024-03-28T23:34:36.454063+0000","last_active":"2024-03-28T23:34:38.781383+0000","last_peered":"2024-03-28T23:34:38.781383+0000","last_clean":"2024-03-28T23:34:38.781383+0000","last_became_active":"2024-03-28T23:34:36.453766+0000","last_became_peered":"2024-03-28T23:34:36.453766+0000","last_unstale":"2024-03-28T23:34:38.781383+0000","last_undegraded":"2024-03-28T23:34:38.781383+0000","last_fullsized":"2024-03-28T23:34:38.781383+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:02:46.239087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466162+0000","last_change":"2024-03-28T23:34:37.473020+0000","last_active":"2024-03-28T23:34:38.466162+0000","last_peered":"2024-03-28T23:34:38.466162+0000","last_clean":"2024-03-28T23:34:38.466162+0000","last_became_active":"2024-03-28T23:34:37.472738+0000","last_became_peered":"2024-03-28T23:34:37.472738+0000","last_unstale":"2024-03-28T23:34:38.466162+0000","last_undegraded":"2024-03-28T23:34:38.466162+0000","last_fullsized":"2024-03-28T23:34:38.466162+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:22:03.340749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781366+0000","last_change":"2024-03-28T23:34:36.455295+0000","last_active":"2024-03-28T23:34:38.781366+0000","last_peered":"2024-03-28T23:34:38.781366+0000","last_clean":"2024-03-28T23:34:38.781366+0000","last_became_active":"2024-03-28T23:34:36.454979+0000","last_became_peered":"2024-03-28T23:34:36.454979+0000","last_unstale":"2024-03-28T23:34:38.781366+0000","last_undegraded":"2024-03-28T23:34:38.781366+0000","last_fullsized":"2024-03-28T23:34:38.781366+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:30:07.389688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782089+0000","last_change":"2024-03-28T23:34:37.459502+0000","last_active":"2024-03-28T23:34:38.782089+0000","last_peered":"2024-03-28T23:34:38.782089+0000","last_clean":"2024-03-28T23:34:38.782089+0000","last_became_active":"2024-03-28T23:34:37.459215+0000","last_became_peered":"2024-03-28T23:34:37.459215+0000","last_unstale":"2024-03-28T23:34:38.782089+0000","last_undegraded":"2024-03-28T23:34:38.782089+0000","last_fullsized":"2024-03-28T23:34:38.782089+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:32.284640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"121'2","reported_seq":16,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782334+0000","last_change":"2024-03-28T23:34:36.455380+0000","last_active":"2024-03-28T23:34:38.782334+0000","last_peered":"2024-03-28T23:34:38.782334+0000","last_clean":"2024-03-28T23:34:38.782334+0000","last_became_active":"2024-03-28T23:34:36.455138+0000","last_became_peered":"2024-03-28T23:34:36.455138+0000","last_unstale":"2024-03-28T23:34:38.782334+0000","last_undegraded":"2024-03-28T23:34:38.782334+0000","last_fullsized":"2024-03-28T23:34:38.782334+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:40:40.550780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.467567+0000","last_change":"2024-03-28T23:34:37.478064+0000","last_active":"2024-03-28T23:34:38.467567+0000","last_peered":"2024-03-28T23:34:38.467567+0000","last_clean":"2024-03-28T23:34:38.467567+0000","last_became_active":"2024-03-28T23:34:37.476145+0000","last_became_peered":"2024-03-28T23:34:37.476145+0000","last_unstale":"2024-03-28T23:34:38.467567+0000","last_undegraded":"2024-03-28T23:34:38.467567+0000","last_fullsized":"2024-03-28T23:34:38.467567+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:36:12.915872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466750+0000","last_change":"2024-03-28T23:34:36.456217+0000","last_active":"2024-03-28T23:34:38.466750+0000","last_peered":"2024-03-28T23:34:38.466750+0000","last_clean":"2024-03-28T23:34:38.466750+0000","last_became_active":"2024-03-28T23:34:36.455926+0000","last_became_peered":"2024-03-28T23:34:36.455926+0000","last_unstale":"2024-03-28T23:34:38.466750+0000","last_undegraded":"2024-03-28T23:34:38.466750+0000","last_fullsized":"2024-03-28T23:34:38.466750+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:04:10.240355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781221+0000","last_change":"2024-03-28T23:34:37.469842+0000","last_active":"2024-03-28T23:34:38.781221+0000","last_peered":"2024-03-28T23:34:38.781221+0000","last_clean":"2024-03-28T23:34:38.781221+0000","last_became_active":"2024-03-28T23:34:37.469558+0000","last_became_peered":"2024-03-28T23:34:37.469558+0000","last_unstale":"2024-03-28T23:34:38.781221+0000","last_undegraded":"2024-03-28T23:34:38.781221+0000","last_fullsized":"2024-03-28T23:34:38.781221+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:28:23.658834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466219+0000","last_change":"2024-03-28T23:34:36.459124+0000","last_active":"2024-03-28T23:34:38.466219+0000","last_peered":"2024-03-28T23:34:38.466219+0000","last_clean":"2024-03-28T23:34:38.466219+0000","last_became_active":"2024-03-28T23:34:36.458755+0000","last_became_peered":"2024-03-28T23:34:36.458755+0000","last_unstale":"2024-03-28T23:34:38.466219+0000","last_undegraded":"2024-03-28T23:34:38.466219+0000","last_fullsized":"2024-03-28T23:34:38.466219+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:30:07.389688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781756+0000","last_change":"2024-03-28T23:34:37.462480+0000","last_active":"2024-03-28T23:34:38.781756+0000","last_peered":"2024-03-28T23:34:38.781756+0000","last_clean":"2024-03-28T23:34:38.781756+0000","last_became_active":"2024-03-28T23:34:37.462174+0000","last_became_peered":"2024-03-28T23:34:37.462174+0000","last_unstale":"2024-03-28T23:34:38.781756+0000","last_undegraded":"2024-03-28T23:34:38.781756+0000","last_fullsized":"2024-03-28T23:34:38.781756+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:23.487869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781271+0000","last_change":"2024-03-28T23:34:36.458354+0000","last_active":"2024-03-28T23:34:38.781271+0000","last_peered":"2024-03-28T23:34:38.781271+0000","last_clean":"2024-03-28T23:34:38.781271+0000","last_became_active":"2024-03-28T23:34:36.458062+0000","last_became_peered":"2024-03-28T23:34:36.458062+0000","last_unstale":"2024-03-28T23:34:38.781271+0000","last_undegraded":"2024-03-28T23:34:38.781271+0000","last_fullsized":"2024-03-28T23:34:38.781271+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:04:10.240355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781174+0000","last_change":"2024-03-28T23:34:37.463095+0000","last_active":"2024-03-28T23:34:38.781174+0000","last_peered":"2024-03-28T23:34:38.781174+0000","last_clean":"2024-03-28T23:34:38.781174+0000","last_became_active":"2024-03-28T23:34:37.462783+0000","last_became_peered":"2024-03-28T23:34:37.462783+0000","last_unstale":"2024-03-28T23:34:38.781174+0000","last_undegraded":"2024-03-28T23:34:38.781174+0000","last_fullsized":"2024-03-28T23:34:38.781174+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:01.408086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.466097+0000","last_change":"2024-03-28T23:34:36.450794+0000","last_active":"2024-03-28T23:34:38.466097+0000","last_peered":"2024-03-28T23:34:38.466097+0000","last_clean":"2024-03-28T23:34:38.466097+0000","last_became_active":"2024-03-28T23:34:36.450523+0000","last_became_peered":"2024-03-28T23:34:36.450523+0000","last_unstale":"2024-03-28T23:34:38.466097+0000","last_undegraded":"2024-03-28T23:34:38.466097+0000","last_fullsized":"2024-03-28T23:34:38.466097+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:43:35.882626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781776+0000","last_change":"2024-03-28T23:34:37.463134+0000","last_active":"2024-03-28T23:34:38.781776+0000","last_peered":"2024-03-28T23:34:38.781776+0000","last_clean":"2024-03-28T23:34:38.781776+0000","last_became_active":"2024-03-28T23:34:37.462602+0000","last_became_peered":"2024-03-28T23:34:37.462602+0000","last_unstale":"2024-03-28T23:34:38.781776+0000","last_undegraded":"2024-03-28T23:34:38.781776+0000","last_fullsized":"2024-03-28T23:34:38.781776+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:51:29.849040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781462+0000","last_change":"2024-03-28T23:34:36.454661+0000","last_active":"2024-03-28T23:34:38.781462+0000","last_peered":"2024-03-28T23:34:38.781462+0000","last_clean":"2024-03-28T23:34:38.781462+0000","last_became_active":"2024-03-28T23:34:36.454375+0000","last_became_peered":"2024-03-28T23:34:36.454375+0000","last_unstale":"2024-03-28T23:34:38.781462+0000","last_undegraded":"2024-03-28T23:34:38.781462+0000","last_fullsized":"2024-03-28T23:34:38.781462+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:28:48.923891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782346+0000","last_change":"2024-03-28T23:34:37.460449+0000","last_active":"2024-03-28T23:34:38.782346+0000","last_peered":"2024-03-28T23:34:38.782346+0000","last_clean":"2024-03-28T23:34:38.782346+0000","last_became_active":"2024-03-28T23:34:37.460160+0000","last_became_peered":"2024-03-28T23:34:37.460160+0000","last_unstale":"2024-03-28T23:34:38.782346+0000","last_undegraded":"2024-03-28T23:34:38.782346+0000","last_fullsized":"2024-03-28T23:34:38.782346+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:47:47.623344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782194+0000","last_change":"2024-03-28T23:34:36.455971+0000","last_active":"2024-03-28T23:34:38.782194+0000","last_peered":"2024-03-28T23:34:38.782194+0000","last_clean":"2024-03-28T23:34:38.782194+0000","last_became_active":"2024-03-28T23:34:36.455683+0000","last_became_peered":"2024-03-28T23:34:36.455683+0000","last_unstale":"2024-03-28T23:34:38.782194+0000","last_undegraded":"2024-03-28T23:34:38.782194+0000","last_fullsized":"2024-03-28T23:34:38.782194+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:43:35.882626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781747+0000","last_change":"2024-03-28T23:34:37.471183+0000","last_active":"2024-03-28T23:34:38.781747+0000","last_peered":"2024-03-28T23:34:38.781747+0000","last_clean":"2024-03-28T23:34:38.781747+0000","last_became_active":"2024-03-28T23:34:37.470909+0000","last_became_peered":"2024-03-28T23:34:37.470909+0000","last_unstale":"2024-03-28T23:34:38.781747+0000","last_undegraded":"2024-03-28T23:34:38.781747+0000","last_fullsized":"2024-03-28T23:34:38.781747+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:08:12.545863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.782175+0000","last_change":"2024-03-28T23:34:36.455827+0000","last_active":"2024-03-28T23:34:38.782175+0000","last_peered":"2024-03-28T23:34:38.782175+0000","last_clean":"2024-03-28T23:34:38.782175+0000","last_became_active":"2024-03-28T23:34:36.455548+0000","last_became_peered":"2024-03-28T23:34:36.455548+0000","last_unstale":"2024-03-28T23:34:38.782175+0000","last_undegraded":"2024-03-28T23:34:38.782175+0000","last_fullsized":"2024-03-28T23:34:38.782175+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:22:02.335132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.467946+0000","last_change":"2024-03-28T23:34:37.469642+0000","last_active":"2024-03-28T23:34:38.467946+0000","last_peered":"2024-03-28T23:34:38.467946+0000","last_clean":"2024-03-28T23:34:38.467946+0000","last_became_active":"2024-03-28T23:34:37.469376+0000","last_became_peered":"2024-03-28T23:34:37.469376+0000","last_unstale":"2024-03-28T23:34:38.467946+0000","last_undegraded":"2024-03-28T23:34:38.467946+0000","last_fullsized":"2024-03-28T23:34:38.467946+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:36.435831+0000","last_clean_scrub_stamp":"2024-03-28T23:34:36.435831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:04:11.245971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781363+0000","last_change":"2024-03-28T23:34:36.460170+0000","last_active":"2024-03-28T23:34:38.781363+0000","last_peered":"2024-03-28T23:34:38.781363+0000","last_clean":"2024-03-28T23:34:38.781363+0000","last_became_active":"2024-03-28T23:34:36.459872+0000","last_became_peered":"2024-03-28T23:34:36.459872+0000","last_unstale":"2024-03-28T23:34:38.781363+0000","last_undegraded":"2024-03-28T23:34:38.781363+0000","last_fullsized":"2024-03-28T23:34:38.781363+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:34:35.430214+0000","last_clean_scrub_stamp":"2024-03-28T23:34:35.430214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:11:24.838764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":363,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-28T23:34:38.781069+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:34:38.781069+0000","last_peered":"2024-03-28T23:34:38.781069+0000","last_clean":"2024-03-28T23:34:38.781069+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:34:38.781069+0000","last_undegraded":"2024-03-28T23:34:38.781069+0000","last_fullsized":"2024-03-28T23:34:38.781069+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32740,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339100,"statfs":{"total":96636764160,"available":96603238400,"internally_reserved":0,"allocated":1597440,"data_stored":1322149,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574911,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32680,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339160,"statfs":{"total":96636764160,"available":96603299840,"internally_reserved":0,"allocated":1536000,"data_stored":737443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574911,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32676,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339164,"statfs":{"total":96636764160,"available":96603303936,"internally_reserved":0,"allocated":1531904,"data_stored":736793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574910,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32672,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":1527808,"data_stored":736771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32684,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339156,"statfs":{"total":96636764160,"available":96603295744,"internally_reserved":0,"allocated":1531904,"data_stored":731761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574910,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32676,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339164,"statfs":{"total":96636764160,"available":96603303936,"internally_reserved":0,"allocated":1523712,"data_stored":730695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32676,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339164,"statfs":{"total":96636764160,"available":96603303936,"internally_reserved":0,"allocated":1531904,"data_stored":736841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32748,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339092,"statfs":{"total":96636764160,"available":96603230208,"internally_reserved":0,"allocated":1601536,"data_stored":1327195,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:34:40.806 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T23:34:40.806 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-28T23:34:40.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.951+0000 7fb79f4c0640 1 Processor -- start 2024-03-28T23:34:40.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.952+0000 7fb79f4c0640 1 -- start start 2024-03-28T23:34:40.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.952+0000 7fb79f4c0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7981529d0 0x7fb79806bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:40.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.952+0000 7fb79f4c0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb7980c6750 0x7fb7980c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:40.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.952+0000 7fb79f4c0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb798152db0 con 0x7fb798155c30 2024-03-28T23:34:40.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.952+0000 7fb79f4c0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fb7980c7210 con 0x7fb7980c6750 2024-03-28T23:34:40.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.952+0000 7fb79f4c0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb7980c6c50 con 0x7fb7981529d0 2024-03-28T23:34:40.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.953+0000 7fb79da5d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb798155c30 0x7fb7980c54f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50328/0 (socket says 172.21.15.161:50328) 2024-03-28T23:34:40.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.953+0000 7fb79da5d640 1 -- 172.21.15.161:0/1400672378 learned_addr learned my addr 172.21.15.161:0/1400672378 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:40.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.953+0000 7fb79d25c640 1 --2- 172.21.15.161:0/1400672378 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7981529d0 0x7fb79806bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:40.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.953+0000 7fb79ca5b640 1 --2- 172.21.15.161:0/1400672378 >> v2:172.21.15.170:3300/0 conn(0x7fb7980c6750 0x7fb7980c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:40.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.953+0000 7fb78ffff640 1 -- 172.21.15.161:0/1400672378 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 817044389 0 0) 0x7fb798152db0 con 0x7fb798155c30 2024-03-28T23:34:40.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.953+0000 7fb78ffff640 1 -- 172.21.15.161:0/1400672378 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb784003520 con 0x7fb798155c30 2024-03-28T23:34:40.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.953+0000 7fb79d25c640 1 -- 172.21.15.161:0/1400672378 >> v2:172.21.15.170:3300/0 conn(0x7fb7980c6750 msgr2=0x7fb7980c88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:40.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.953+0000 7fb79d25c640 1 --2- 172.21.15.161:0/1400672378 >> v2:172.21.15.170:3300/0 conn(0x7fb7980c6750 0x7fb7980c88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.953+0000 7fb79d25c640 1 -- 172.21.15.161:0/1400672378 >> v1:172.21.15.161:6789/0 conn(0x7fb798155c30 legacy=0x7fb7980c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:40.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.953+0000 7fb79d25c640 1 -- 172.21.15.161:0/1400672378 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7980c6f30 con 0x7fb7981529d0 2024-03-28T23:34:40.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.953+0000 7fb79ca5b640 1 --2- 172.21.15.161:0/1400672378 >> v2:172.21.15.170:3300/0 conn(0x7fb7980c6750 0x7fb7980c88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:34:40.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.954+0000 7fb79d25c640 1 --2- 172.21.15.161:0/1400672378 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7981529d0 0x7fb79806bb60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb7940097a0 tx=0x7fb79402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a643e126cbfd2c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:40.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.954+0000 7fb78ffff640 1 -- 172.21.15.161:0/1400672378 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb794035020 con 0x7fb7981529d0 2024-03-28T23:34:40.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.954+0000 7fb78ffff640 1 -- 172.21.15.161:0/1400672378 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb794031830 con 0x7fb7981529d0 2024-03-28T23:34:40.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.954+0000 7fb78ffff640 1 -- 172.21.15.161:0/1400672378 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb794031be0 con 0x7fb7981529d0 2024-03-28T23:34:40.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.955+0000 7fb79f4c0640 1 -- 172.21.15.161:0/1400672378 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7981529d0 msgr2=0x7fb79806bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:40.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.955+0000 7fb79f4c0640 1 --2- 172.21.15.161:0/1400672378 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7981529d0 0x7fb79806bb60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb7940097a0 tx=0x7fb79402b8d0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.955+0000 7fb79f4c0640 1 -- 172.21.15.161:0/1400672378 shutdown_connections 2024-03-28T23:34:40.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.955+0000 7fb79f4c0640 1 --2- 172.21.15.161:0/1400672378 >> v2:172.21.15.170:3300/0 conn(0x7fb7980c6750 0x7fb7980c88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.955+0000 7fb79f4c0640 1 --2- 172.21.15.161:0/1400672378 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7981529d0 0x7fb79806bb60 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.955+0000 7fb79f4c0640 1 -- 172.21.15.161:0/1400672378 >> 172.21.15.161:0/1400672378 conn(0x7fb7980c7430 msgr2=0x7fb7980c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:40.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.955+0000 7fb79f4c0640 1 -- 172.21.15.161:0/1400672378 shutdown_connections 2024-03-28T23:34:40.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.955+0000 7fb79f4c0640 1 -- 172.21.15.161:0/1400672378 wait complete. 2024-03-28T23:34:40.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.956+0000 7fb79f4c0640 1 Processor -- start 2024-03-28T23:34:40.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.956+0000 7fb79f4c0640 1 -- start start 2024-03-28T23:34:40.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.956+0000 7fb79f4c0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7980c6750 0x7fb7981534f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:40.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.956+0000 7fb79f4c0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7981529d0 0x7fb798154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:40.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.956+0000 7fb79f4c0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb798155c30 0x7fb798154e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:40.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.956+0000 7fb79f4c0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb7980aa160 con 0x7fb7980c6750 2024-03-28T23:34:40.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.956+0000 7fb79f4c0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb798155710 con 0x7fb798155c30 2024-03-28T23:34:40.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.956+0000 7fb79f4c0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb7980c5c00 con 0x7fb7981529d0 2024-03-28T23:34:40.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.956+0000 7fb79d25c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7980c6750 0x7fb7981534f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:40.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.957+0000 7fb79d25c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7980c6750 0x7fb7981534f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54568/0 (socket says 172.21.15.161:54568) 2024-03-28T23:34:40.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.957+0000 7fb79d25c640 1 -- 172.21.15.161:0/1434359027 learned_addr learned my addr 172.21.15.161:0/1434359027 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:40.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.957+0000 7fb79ca5b640 1 --2- 172.21.15.161:0/1434359027 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7981529d0 0x7fb798154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:40.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.957+0000 7fb79da5d640 1 --2- 172.21.15.161:0/1434359027 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb798155c30 0x7fb798154e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:40.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.957+0000 7fb79d25c640 1 -- 172.21.15.161:0/1434359027 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7981529d0 msgr2=0x7fb798154920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:40.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.957+0000 7fb79d25c640 1 --2- 172.21.15.161:0/1434359027 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7981529d0 0x7fb798154920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.957+0000 7fb79d25c640 1 -- 172.21.15.161:0/1434359027 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb798155c30 msgr2=0x7fb798154e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:40.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.957+0000 7fb79d25c640 1 --2- 172.21.15.161:0/1434359027 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb798155c30 0x7fb798154e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:40.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.957+0000 7fb79d25c640 1 -- 172.21.15.161:0/1434359027 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb798149ee0 con 0x7fb7980c6750 2024-03-28T23:34:40.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.957+0000 7fb79d25c640 1 --2- 172.21.15.161:0/1434359027 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7980c6750 0x7fb7981534f0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fb79402b910 tx=0x7fb7940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:40.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.957+0000 7fb78dffb640 1 -- 172.21.15.161:0/1434359027 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb794040070 con 0x7fb7980c6750 2024-03-28T23:34:40.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.957+0000 7fb78dffb640 1 -- 172.21.15.161:0/1434359027 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb79403a910 con 0x7fb7980c6750 2024-03-28T23:34:40.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.957+0000 7fb79f4c0640 1 -- 172.21.15.161:0/1434359027 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7981df250 con 0x7fb7980c6750 2024-03-28T23:34:40.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.957+0000 7fb78dffb640 1 -- 172.21.15.161:0/1434359027 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb794035040 con 0x7fb7980c6750 2024-03-28T23:34:40.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.957+0000 7fb79f4c0640 1 -- 172.21.15.161:0/1434359027 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb798153a30 con 0x7fb7980c6750 2024-03-28T23:34:40.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.958+0000 7fb79f4c0640 1 -- 172.21.15.161:0/1434359027 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb79804f4b0 con 0x7fb7980c6750 2024-03-28T23:34:40.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.959+0000 7fb78dffb640 1 -- 172.21.15.161:0/1434359027 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb79404b060 con 0x7fb7980c6750 2024-03-28T23:34:40.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.959+0000 7fb78dffb640 1 --2- 172.21.15.161:0/1434359027 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb77c06b500 0x7fb77c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:40.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.959+0000 7fb78dffb640 1 -- 172.21.15.161:0/1434359027 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6101+0+0 (secure 0 0 0) 0x7fb7940ab8b0 con 0x7fb7980c6750 2024-03-28T23:34:40.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.959+0000 7fb79ca5b640 1 --2- 172.21.15.161:0/1434359027 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb77c06b500 0x7fb77c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:40.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.960+0000 7fb79ca5b640 1 --2- 172.21.15.161:0/1434359027 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb77c06b500 0x7fb77c06d9c0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fb788009560 tx=0x7fb788009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:40.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:40.964+0000 7fb78dffb640 1 -- 172.21.15.161:0/1434359027 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb794079fe0 con 0x7fb7980c6750 2024-03-28T23:34:41.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.359+0000 7fb79f4c0640 1 -- 172.21.15.161:0/1434359027 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fb7980ca650 con 0x7fb7980c6750 2024-03-28T23:34:41.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.361+0000 7fb78dffb640 1 -- 172.21.15.161:0/1434359027 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) ==== 190+0+147 (secure 0 0 0) 0x7fb79407e080 con 0x7fb7980c6750 2024-03-28T23:34:41.361 INFO:teuthology.orchestra.run.smithi161.stdout:[client.0] 2024-03-28T23:34:41.361 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQA6/QVmc1krAxAAEKsn1Tuuv9OlFZpB4PFhDg== 2024-03-28T23:34:41.361 INFO:teuthology.orchestra.run.smithi161.stdout: caps mds = "allow" 2024-03-28T23:34:41.361 INFO:teuthology.orchestra.run.smithi161.stdout: caps mon = "allow r" 2024-03-28T23:34:41.362 INFO:teuthology.orchestra.run.smithi161.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-28T23:34:41.362 INFO:teuthology.orchestra.run.smithi161.stderr:updated caps for client.0 2024-03-28T23:34:41.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.363+0000 7fb79f4c0640 1 -- 172.21.15.161:0/1434359027 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb77c06b500 msgr2=0x7fb77c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:41.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.363+0000 7fb79f4c0640 1 --2- 172.21.15.161:0/1434359027 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb77c06b500 0x7fb77c06d9c0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fb788009560 tx=0x7fb788009190 comp rx=0 tx=0).stop 2024-03-28T23:34:41.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.363+0000 7fb79f4c0640 1 -- 172.21.15.161:0/1434359027 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7980c6750 msgr2=0x7fb7981534f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:41.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.363+0000 7fb79f4c0640 1 --2- 172.21.15.161:0/1434359027 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7980c6750 0x7fb7981534f0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fb79402b910 tx=0x7fb7940046c0 comp rx=0 tx=0).stop 2024-03-28T23:34:41.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.363+0000 7fb79f4c0640 1 -- 172.21.15.161:0/1434359027 shutdown_connections 2024-03-28T23:34:41.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.363+0000 7fb79f4c0640 1 --2- 172.21.15.161:0/1434359027 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb77c06b500 0x7fb77c06d9c0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:41.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.363+0000 7fb79f4c0640 1 --2- 172.21.15.161:0/1434359027 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb798155c30 0x7fb798154e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:41.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.363+0000 7fb79f4c0640 1 --2- 172.21.15.161:0/1434359027 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7981529d0 0x7fb798154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:41.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.363+0000 7fb79f4c0640 1 --2- 172.21.15.161:0/1434359027 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7980c6750 0x7fb7981534f0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:41.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.363+0000 7fb79f4c0640 1 -- 172.21.15.161:0/1434359027 >> 172.21.15.161:0/1434359027 conn(0x7fb7980c7430 msgr2=0x7fb7981e1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:41.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.363+0000 7fb79f4c0640 1 -- 172.21.15.161:0/1434359027 shutdown_connections 2024-03-28T23:34:41.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.363+0000 7fb79f4c0640 1 -- 172.21.15.161:0/1434359027 wait complete. 2024-03-28T23:34:41.375 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:34:41.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.479+0000 7f3451409640 1 Processor -- start 2024-03-28T23:34:41.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.480+0000 7f3451409640 1 -- start start 2024-03-28T23:34:41.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.480+0000 7f3451409640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f344c0c3770 0x7f344c0c1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:41.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.480+0000 7f3451409640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f344c0c2300 0x7f344c15fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:41.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.480+0000 7f3451409640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f344c0c3bd0 con 0x7f344c0c3390 2024-03-28T23:34:41.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.480+0000 7f3451409640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f344c155e80 con 0x7f344c0c2300 2024-03-28T23:34:41.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.480+0000 7f3451409640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f344c0a8410 con 0x7f344c0c3770 2024-03-28T23:34:41.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.480+0000 7f344b7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f344c0c2300 0x7f344c15fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:41.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.480+0000 7f344affd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f344c0c3390 0x7f344c0a7d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50330/0 (socket says 172.21.15.161:50330) 2024-03-28T23:34:41.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.480+0000 7f344affd640 1 -- 172.21.15.161:0/1402962281 learned_addr learned my addr 172.21.15.161:0/1402962281 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:41.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.481+0000 7f344a7fc640 1 --2- 172.21.15.161:0/1402962281 >> v2:172.21.15.170:3300/0 conn(0x7f344c0c3770 0x7f344c0c1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:41.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.481+0000 7f344b7fe640 1 -- 172.21.15.161:0/1402962281 >> v2:172.21.15.170:3300/0 conn(0x7f344c0c3770 msgr2=0x7f344c0c1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:41.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.481+0000 7f344b7fe640 1 --2- 172.21.15.161:0/1402962281 >> v2:172.21.15.170:3300/0 conn(0x7f344c0c3770 0x7f344c0c1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:41.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.481+0000 7f344b7fe640 1 -- 172.21.15.161:0/1402962281 >> v1:172.21.15.161:6789/0 conn(0x7f344c0c3390 legacy=0x7f344c0a7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:41.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.481+0000 7f344b7fe640 1 -- 172.21.15.161:0/1402962281 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f344c155270 con 0x7f344c0c2300 2024-03-28T23:34:41.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.481+0000 7f344b7fe640 1 --2- 172.21.15.161:0/1402962281 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f344c0c2300 0x7f344c15fae0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f344c0c3080 tx=0x7f344402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61d691699470b199 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:41.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.482+0000 7f3449ffb640 1 -- 172.21.15.161:0/1402962281 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3444039070 con 0x7f344c0c2300 2024-03-28T23:34:41.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.482+0000 7f3449ffb640 1 -- 172.21.15.161:0/1402962281 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f344402cbf0 con 0x7f344c0c2300 2024-03-28T23:34:41.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.482+0000 7f3449ffb640 1 -- 172.21.15.161:0/1402962281 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3444034440 con 0x7f344c0c2300 2024-03-28T23:34:41.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.482+0000 7f3451409640 1 -- 172.21.15.161:0/1402962281 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f344c0c2300 msgr2=0x7f344c15fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:41.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.482+0000 7f3451409640 1 --2- 172.21.15.161:0/1402962281 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f344c0c2300 0x7f344c15fae0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f344c0c3080 tx=0x7f344402be40 comp rx=0 tx=0).stop 2024-03-28T23:34:41.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.482+0000 7f3451409640 1 -- 172.21.15.161:0/1402962281 shutdown_connections 2024-03-28T23:34:41.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.482+0000 7f3451409640 1 --2- 172.21.15.161:0/1402962281 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f344c0c2300 0x7f344c15fae0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:41.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.482+0000 7f3451409640 1 --2- 172.21.15.161:0/1402962281 >> v2:172.21.15.170:3300/0 conn(0x7f344c0c3770 0x7f344c0c1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:41.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.482+0000 7f3451409640 1 -- 172.21.15.161:0/1402962281 >> 172.21.15.161:0/1402962281 conn(0x7f344c0aa740 msgr2=0x7f344c0aab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:41.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.482+0000 7f3451409640 1 -- 172.21.15.161:0/1402962281 shutdown_connections 2024-03-28T23:34:41.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.482+0000 7f3451409640 1 -- 172.21.15.161:0/1402962281 wait complete. 2024-03-28T23:34:41.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.484+0000 7f3451409640 1 Processor -- start 2024-03-28T23:34:41.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.484+0000 7f3451409640 1 -- start start 2024-03-28T23:34:41.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.484+0000 7f3451409640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f344c0c2300 0x7f344c0ae390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:41.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.484+0000 7f3451409640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f344c0c3390 0x7f344c156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:41.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.484+0000 7f3451409640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f344c0c3770 0x7f344c06ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:41.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.484+0000 7f3451409640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f344c155b10 con 0x7f344c0c3390 2024-03-28T23:34:41.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.484+0000 7f3451409640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f344c155830 con 0x7f344c0c2300 2024-03-28T23:34:41.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.484+0000 7f3451409640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f344c152a80 con 0x7f344c0c3770 2024-03-28T23:34:41.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.484+0000 7f344a7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f344c0c3390 0x7f344c156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:41.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.484+0000 7f344a7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f344c0c3390 0x7f344c156460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54584/0 (socket says 172.21.15.161:54584) 2024-03-28T23:34:41.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.484+0000 7f344a7fc640 1 -- 172.21.15.161:0/1828095065 learned_addr learned my addr 172.21.15.161:0/1828095065 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:41.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.485+0000 7f344affd640 1 --2- 172.21.15.161:0/1828095065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f344c0c2300 0x7f344c0ae390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:41.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.485+0000 7f344b7fe640 1 --2- 172.21.15.161:0/1828095065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f344c0c3770 0x7f344c06ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:41.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.485+0000 7f344a7fc640 1 -- 172.21.15.161:0/1828095065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f344c0c3770 msgr2=0x7f344c06ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:41.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.485+0000 7f344a7fc640 1 --2- 172.21.15.161:0/1828095065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f344c0c3770 0x7f344c06ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:41.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.485+0000 7f344a7fc640 1 -- 172.21.15.161:0/1828095065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f344c0c2300 msgr2=0x7f344c0ae390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:41.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.485+0000 7f344a7fc640 1 --2- 172.21.15.161:0/1828095065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f344c0c2300 0x7f344c0ae390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:41.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.485+0000 7f344a7fc640 1 -- 172.21.15.161:0/1828095065 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f344c0ab8d0 con 0x7f344c0c3390 2024-03-28T23:34:41.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.485+0000 7f344affd640 1 --2- 172.21.15.161:0/1828095065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f344c0c2300 0x7f344c0ae390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:41.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.485+0000 7f344a7fc640 1 --2- 172.21.15.161:0/1828095065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f344c0c3390 0x7f344c156460 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f344000d5f0 tx=0x7f344000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:41.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.485+0000 7f34337fe640 1 -- 172.21.15.161:0/1828095065 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34400125f0 con 0x7f344c0c3390 2024-03-28T23:34:41.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.485+0000 7f34337fe640 1 -- 172.21.15.161:0/1828095065 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3440019070 con 0x7f344c0c3390 2024-03-28T23:34:41.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.485+0000 7f34337fe640 1 -- 172.21.15.161:0/1828095065 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34400129d0 con 0x7f344c0c3390 2024-03-28T23:34:41.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.485+0000 7f3451409640 1 -- 172.21.15.161:0/1828095065 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f344c1ebfe0 con 0x7f344c0c3390 2024-03-28T23:34:41.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.485+0000 7f3451409640 1 -- 172.21.15.161:0/1828095065 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f344c1ec2c0 con 0x7f344c0c3390 2024-03-28T23:34:41.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.486+0000 7f3451409640 1 -- 172.21.15.161:0/1828095065 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f344c04f4b0 con 0x7f344c0c3390 2024-03-28T23:34:41.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.487+0000 7f34337fe640 1 -- 172.21.15.161:0/1828095065 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3440012b70 con 0x7f344c0c3390 2024-03-28T23:34:41.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.487+0000 7f34337fe640 1 --2- 172.21.15.161:0/1828095065 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f343806b4d0 0x7f343806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:41.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.487+0000 7f34337fe640 1 -- 172.21.15.161:0/1828095065 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6101+0+0 (secure 0 0 0) 0x7f344008a710 con 0x7f344c0c3390 2024-03-28T23:34:41.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.487+0000 7f344affd640 1 --2- 172.21.15.161:0/1828095065 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f343806b4d0 0x7f343806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:41.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.488+0000 7f344affd640 1 --2- 172.21.15.161:0/1828095065 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f343806b4d0 0x7f343806d990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f344c072e70 tx=0x7f343c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:41.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.492+0000 7f34337fe640 1 -- 172.21.15.161:0/1828095065 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3440059490 con 0x7f344c0c3390 2024-03-28T23:34:41.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.708+0000 7f3451409640 1 -- 172.21.15.161:0/1828095065 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f344c0ace90 con 0x7f344c0c3390 2024-03-28T23:34:41.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.708+0000 7f34337fe640 1 -- 172.21.15.161:0/1828095065 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+5198 (secure 0 0 0) 0x7f3440058d90 con 0x7f344c0c3390 2024-03-28T23:34:41.709 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:41.709 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":107,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7318,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3734750343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3734750343},{"type":"v1","addr":"172.21.15.170:6835","nonce":3734750343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/4140438862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":4140438862},{"type":"v1","addr":"172.21.15.170:6834","nonce":4140438862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/878057589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":878057589},{"type":"v1","addr":"172.21.15.161:6835","nonce":878057589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:34:36.796608+0000","modified":"2024-03-28T23:34:38.813377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7974},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7974":{"gid":7974,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6837/3432367324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":3432367324},{"type":"v1","addr":"172.21.15.161:6837","nonce":3432367324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7974,"qdb_cluster":[7974]},"id":9}]} 2024-03-28T23:34:41.709 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 107 2024-03-28T23:34:41.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.710+0000 7f3451409640 1 -- 172.21.15.161:0/1828095065 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f343806b4d0 msgr2=0x7f343806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:41.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.710+0000 7f3451409640 1 --2- 172.21.15.161:0/1828095065 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f343806b4d0 0x7f343806d990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f344c072e70 tx=0x7f343c009040 comp rx=0 tx=0).stop 2024-03-28T23:34:41.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.710+0000 7f3451409640 1 -- 172.21.15.161:0/1828095065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f344c0c3390 msgr2=0x7f344c156460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:41.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.710+0000 7f3451409640 1 --2- 172.21.15.161:0/1828095065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f344c0c3390 0x7f344c156460 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f344000d5f0 tx=0x7f344000dab0 comp rx=0 tx=0).stop 2024-03-28T23:34:41.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.710+0000 7f3451409640 1 -- 172.21.15.161:0/1828095065 shutdown_connections 2024-03-28T23:34:41.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.710+0000 7f3451409640 1 --2- 172.21.15.161:0/1828095065 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f343806b4d0 0x7f343806d990 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:41.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.710+0000 7f3451409640 1 --2- 172.21.15.161:0/1828095065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f344c0c3390 0x7f344c156460 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:41.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.710+0000 7f3451409640 1 --2- 172.21.15.161:0/1828095065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f344c0c3770 0x7f344c06ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:41.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.710+0000 7f3451409640 1 --2- 172.21.15.161:0/1828095065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f344c0c2300 0x7f344c0ae390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:41.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.710+0000 7f3451409640 1 -- 172.21.15.161:0/1828095065 >> 172.21.15.161:0/1828095065 conn(0x7f344c0aa740 msgr2=0x7f344c152fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:41.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.710+0000 7f3451409640 1 -- 172.21.15.161:0/1828095065 shutdown_connections 2024-03-28T23:34:41.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:41.711+0000 7f3451409640 1 -- 172.21.15.161:0/1828095065 wait complete. 2024-03-28T23:34:42.127 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:34:36.796608+0000', 'modified': '2024-03-28T23:34:38.813377+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7974}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7974': {'gid': 7974, 'name': 'a', 'rank': 0, 'incarnation': 105, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.161:6837/3432367324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 3432367324}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 3432367324}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7974, 'qdb_cluster': [7974]} 2024-03-28T23:34:42.128 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:34:42.128 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-28T23:34:42.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.232+0000 7fbda0809640 1 Processor -- start 2024-03-28T23:34:42.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.232+0000 7fbda0809640 1 -- start start 2024-03-28T23:34:42.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.233+0000 7fbda0809640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd9c152a80 0x7fbd9c155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.233+0000 7fbda0809640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd9c06c3c0 con 0x7fbd9c06cb80 2024-03-28T23:34:42.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.233+0000 7fbda0809640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd9c0c5b90 con 0x7fbd9c06d9f0 2024-03-28T23:34:42.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.233+0000 7fbda0809640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbd9c156250 con 0x7fbd9c152a80 2024-03-28T23:34:42.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.233+0000 7fbd9ad9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fbd9c06cb80 0x7fbd9c0c74c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50342/0 (socket says 172.21.15.161:50342) 2024-03-28T23:34:42.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.233+0000 7fbd9ad9d640 1 -- 172.21.15.161:0/2085325490 learned_addr learned my addr 172.21.15.161:0/2085325490 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:42.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.233+0000 7fbd9959a640 1 -- 172.21.15.161:0/2085325490 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3257679859 0 0) 0x7fbd9c06c3c0 con 0x7fbd9c06cb80 2024-03-28T23:34:42.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.233+0000 7fbd9a59c640 1 --2- 172.21.15.161:0/2085325490 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd9c152a80 0x7fbd9c155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.233+0000 7fbd9959a640 1 -- 172.21.15.161:0/2085325490 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd88003520 con 0x7fbd9c06cb80 2024-03-28T23:34:42.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.233+0000 7fbd9959a640 1 -- 172.21.15.161:0/2085325490 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1965651242 0 0) 0x7fbd88003520 con 0x7fbd9c06cb80 2024-03-28T23:34:42.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.233+0000 7fbd9959a640 1 -- 172.21.15.161:0/2085325490 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd9c152a80 msgr2=0x7fbd9c155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:42.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.233+0000 7fbd9959a640 1 --2- 172.21.15.161:0/2085325490 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd9c152a80 0x7fbd9c155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.233+0000 7fbd9959a640 1 -- 172.21.15.161:0/2085325490 >> v1:172.21.15.170:6789/0 conn(0x7fbd9c06d9f0 legacy=0x7fbd9c0c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:42.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.233+0000 7fbd9959a640 1 -- 172.21.15.161:0/2085325490 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd9c0c55d0 con 0x7fbd9c06cb80 2024-03-28T23:34:42.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.233+0000 7fbd9959a640 1 -- 172.21.15.161:0/2085325490 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fbd94001670 con 0x7fbd9c06cb80 2024-03-28T23:34:42.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.233+0000 7fbd9959a640 1 -- 172.21.15.161:0/2085325490 >> v1:172.21.15.161:6789/0 conn(0x7fbd9c06cb80 legacy=0x7fbd9c0c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:42.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.233+0000 7fbd9959a640 1 --2- 172.21.15.161:0/2085325490 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd9c152a80 0x7fbd88003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.233+0000 7fbd9959a640 1 --2- 172.21.15.161:0/2085325490 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd88004040 0x7fbd88004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.233+0000 7fbd9959a640 1 --2- 172.21.15.161:0/2085325490 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd88004db0 0x7fbd88005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.233+0000 7fbd9959a640 1 -- 172.21.15.161:0/2085325490 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbd9c156250 con 0x7fbd88004040 2024-03-28T23:34:42.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.233+0000 7fbd9959a640 1 -- 172.21.15.161:0/2085325490 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8800bcc0 con 0x7fbd88004db0 2024-03-28T23:34:42.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.233+0000 7fbd9959a640 1 -- 172.21.15.161:0/2085325490 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbd8800bec0 con 0x7fbd9c152a80 2024-03-28T23:34:42.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.234+0000 7fbd99d9b640 1 --2- 172.21.15.161:0/2085325490 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd88004040 0x7fbd88004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.234+0000 7fbd9a59c640 1 --2- 172.21.15.161:0/2085325490 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd9c152a80 0x7fbd88003ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.234+0000 7fbd9ad9d640 1 --2- 172.21.15.161:0/2085325490 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd88004db0 0x7fbd88005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.234+0000 7fbd99d9b640 1 -- 172.21.15.161:0/2085325490 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd9c152a80 msgr2=0x7fbd88003ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:42.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.234+0000 7fbd99d9b640 1 --2- 172.21.15.161:0/2085325490 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd9c152a80 0x7fbd88003ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.234+0000 7fbd99d9b640 1 -- 172.21.15.161:0/2085325490 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd88004db0 msgr2=0x7fbd88005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:42.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.234+0000 7fbd99d9b640 1 --2- 172.21.15.161:0/2085325490 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd88004db0 0x7fbd88005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.234+0000 7fbd99d9b640 1 -- 172.21.15.161:0/2085325490 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd88003520 con 0x7fbd88004040 2024-03-28T23:34:42.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.235+0000 7fbd99d9b640 1 --2- 172.21.15.161:0/2085325490 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd88004040 0x7fbd88004800 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fbd84001750 tx=0x7fbd88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f11d15341445867d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:42.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.235+0000 7fbd9959a640 1 -- 172.21.15.161:0/2085325490 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd84001370 con 0x7fbd88004040 2024-03-28T23:34:42.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.235+0000 7fbd9959a640 1 -- 172.21.15.161:0/2085325490 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd840052d0 con 0x7fbd88004040 2024-03-28T23:34:42.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.235+0000 7fbd9959a640 1 -- 172.21.15.161:0/2085325490 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd84003830 con 0x7fbd88004040 2024-03-28T23:34:42.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.235+0000 7fbda0809640 1 -- 172.21.15.161:0/2085325490 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd88004040 msgr2=0x7fbd88004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:42.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.235+0000 7fbda0809640 1 --2- 172.21.15.161:0/2085325490 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd88004040 0x7fbd88004800 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fbd84001750 tx=0x7fbd88005830 comp rx=0 tx=0).stop 2024-03-28T23:34:42.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.235+0000 7fbd9ad9d640 1 -- 172.21.15.161:0/2085325490 reap_dead start 2024-03-28T23:34:42.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.235+0000 7fbda0809640 1 -- 172.21.15.161:0/2085325490 shutdown_connections 2024-03-28T23:34:42.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.235+0000 7fbda0809640 1 -- 172.21.15.161:0/2085325490 >> 172.21.15.161:0/2085325490 conn(0x7fbd9c0ac720 msgr2=0x7fbd9c0acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:42.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.235+0000 7fbda0809640 1 -- 172.21.15.161:0/2085325490 shutdown_connections 2024-03-28T23:34:42.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.235+0000 7fbda0809640 1 -- 172.21.15.161:0/2085325490 wait complete. 2024-03-28T23:34:42.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.237+0000 7fbda0809640 1 Processor -- start 2024-03-28T23:34:42.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.237+0000 7fbda0809640 1 -- start start 2024-03-28T23:34:42.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.237+0000 7fbda0809640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd88004040 0x7fbd9c151b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.237+0000 7fbda0809640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd9c156600 0x7fbd9c14b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.237+0000 7fbda0809640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd9c14b5e0 0x7fbd9c0af570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.237+0000 7fbda0809640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbd9c0c7bd0 con 0x7fbd9c14b5e0 2024-03-28T23:34:42.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.237+0000 7fbda0809640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbd9c06c6f0 con 0x7fbd9c156600 2024-03-28T23:34:42.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.237+0000 7fbda0809640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbd9c0c7d50 con 0x7fbd88004040 2024-03-28T23:34:42.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.237+0000 7fbd9ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd9c14b5e0 0x7fbd9c0af570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.237+0000 7fbd9ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd9c14b5e0 0x7fbd9c0af570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54596/0 (socket says 172.21.15.161:54596) 2024-03-28T23:34:42.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.238+0000 7fbd9ad9d640 1 -- 172.21.15.161:0/471015347 learned_addr learned my addr 172.21.15.161:0/471015347 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:42.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.238+0000 7fbd9a59c640 1 --2- 172.21.15.161:0/471015347 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd88004040 0x7fbd9c151b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.238+0000 7fbd99d9b640 1 --2- 172.21.15.161:0/471015347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd9c156600 0x7fbd9c14b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.238+0000 7fbd9ad9d640 1 -- 172.21.15.161:0/471015347 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd88004040 msgr2=0x7fbd9c151b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:42.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.238+0000 7fbd9ad9d640 1 --2- 172.21.15.161:0/471015347 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd88004040 0x7fbd9c151b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.238+0000 7fbd9ad9d640 1 -- 172.21.15.161:0/471015347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd9c156600 msgr2=0x7fbd9c14b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:42.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.238+0000 7fbd9ad9d640 1 --2- 172.21.15.161:0/471015347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd9c156600 0x7fbd9c14b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.238+0000 7fbd9ad9d640 1 -- 172.21.15.161:0/471015347 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd9c0ae7d0 con 0x7fbd9c14b5e0 2024-03-28T23:34:42.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.238+0000 7fbd9ad9d640 1 --2- 172.21.15.161:0/471015347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd9c14b5e0 0x7fbd9c0af570 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c074cc0 tx=0x7fbd94007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:42.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.238+0000 7fbd92ffd640 1 -- 172.21.15.161:0/471015347 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd9401a070 con 0x7fbd9c14b5e0 2024-03-28T23:34:42.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.238+0000 7fbd92ffd640 1 -- 172.21.15.161:0/471015347 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd9400f6a0 con 0x7fbd9c14b5e0 2024-03-28T23:34:42.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.238+0000 7fbda0809640 1 -- 172.21.15.161:0/471015347 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd9c1eedb0 con 0x7fbd9c14b5e0 2024-03-28T23:34:42.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.238+0000 7fbd92ffd640 1 -- 172.21.15.161:0/471015347 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd9400fa90 con 0x7fbd9c14b5e0 2024-03-28T23:34:42.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.238+0000 7fbda0809640 1 -- 172.21.15.161:0/471015347 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd9c0aed70 con 0x7fbd9c14b5e0 2024-03-28T23:34:42.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.240+0000 7fbd92ffd640 1 -- 172.21.15.161:0/471015347 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbd9401d070 con 0x7fbd9c14b5e0 2024-03-28T23:34:42.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.240+0000 7fbd92ffd640 1 --2- 172.21.15.161:0/471015347 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbd7c06b500 0x7fbd7c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.240+0000 7fbd92ffd640 1 -- 172.21.15.161:0/471015347 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6101+0+0 (secure 0 0 0) 0x7fbd940897d0 con 0x7fbd9c14b5e0 2024-03-28T23:34:42.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.240+0000 7fbd9a59c640 1 --2- 172.21.15.161:0/471015347 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbd7c06b500 0x7fbd7c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.241+0000 7fbd9a59c640 1 --2- 172.21.15.161:0/471015347 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbd7c06b500 0x7fbd7c06d9c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c014510 tx=0x7fbd8c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:42.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.241+0000 7fbda33e1740 1 Processor -- start 2024-03-28T23:34:42.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.242+0000 7fbda33e1740 1 -- start start 2024-03-28T23:34:42.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.243+0000 7fbda33e1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cbf49e9db0 0x55cbf49e7890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.243+0000 7fbda33e1740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf49ea190 0x55cbf49e7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.243+0000 7fbda33e1740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cbf49e8310 0x55cbf49e44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.243+0000 7fbda33e1740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55cbf494daf0 con 0x55cbf49ea190 2024-03-28T23:34:42.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.243+0000 7fbda33e1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55cbf48f69d0 con 0x55cbf49e9db0 2024-03-28T23:34:42.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.243+0000 7fbda33e1740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55cbf4949b90 con 0x55cbf49e8310 2024-03-28T23:34:42.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.243+0000 7fbd99d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf49ea190 0x55cbf49e7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.243+0000 7fbd99d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf49ea190 0x55cbf49e7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54598/0 (socket says 172.21.15.161:54598) 2024-03-28T23:34:42.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.243+0000 7fbd99d9b640 1 -- 172.21.15.161:0/2789396295 learned_addr learned my addr 172.21.15.161:0/2789396295 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:42.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.243+0000 7fbd9a59c640 1 --2- 172.21.15.161:0/2789396295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cbf49e9db0 0x55cbf49e7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.243+0000 7fbd99d9b640 1 -- 172.21.15.161:0/2789396295 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cbf49e8310 msgr2=0x55cbf49e44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:42.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.243+0000 7fbd99d9b640 1 --2- 172.21.15.161:0/2789396295 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cbf49e8310 0x55cbf49e44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.243+0000 7fbd99d9b640 1 -- 172.21.15.161:0/2789396295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cbf49e9db0 msgr2=0x55cbf49e7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:42.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.243+0000 7fbd99d9b640 1 --2- 172.21.15.161:0/2789396295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cbf49e9db0 0x55cbf49e7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.243+0000 7fbd99d9b640 1 -- 172.21.15.161:0/2789396295 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbf46622d0 con 0x55cbf49ea190 2024-03-28T23:34:42.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.243+0000 7fbd99d9b640 1 --2- 172.21.15.161:0/2789396295 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf49ea190 0x55cbf49e7dd0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fbd84005040 tx=0x7fbd8402f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbe03d072e0f9aa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:42.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.244+0000 7fbd7bfff640 1 -- 172.21.15.161:0/2789396295 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd84042070 con 0x55cbf49ea190 2024-03-28T23:34:42.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.244+0000 7fbd7bfff640 1 -- 172.21.15.161:0/2789396295 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd84005150 con 0x55cbf49ea190 2024-03-28T23:34:42.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.244+0000 7fbd7bfff640 1 -- 172.21.15.161:0/2789396295 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd84007040 con 0x55cbf49ea190 2024-03-28T23:34:42.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.244+0000 7fbda33e1740 1 -- 172.21.15.161:0/2789396295 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf49ea190 msgr2=0x55cbf49e7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:42.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.244+0000 7fbda33e1740 1 --2- 172.21.15.161:0/2789396295 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf49ea190 0x55cbf49e7dd0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fbd84005040 tx=0x7fbd8402f000 comp rx=0 tx=0).stop 2024-03-28T23:34:42.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.244+0000 7fbda33e1740 1 -- 172.21.15.161:0/2789396295 shutdown_connections 2024-03-28T23:34:42.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.244+0000 7fbda33e1740 1 --2- 172.21.15.161:0/2789396295 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf49ea190 0x55cbf49e7dd0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.244+0000 7fbda33e1740 1 --2- 172.21.15.161:0/2789396295 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cbf49e8310 0x55cbf49e44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.244+0000 7fbda33e1740 1 --2- 172.21.15.161:0/2789396295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cbf49e9db0 0x55cbf49e7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.245+0000 7fbda33e1740 1 -- 172.21.15.161:0/2789396295 >> 172.21.15.161:0/2789396295 conn(0x55cbf49e6630 msgr2=0x55cbf4a0b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:42.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.245+0000 7fbda33e1740 1 -- 172.21.15.161:0/2789396295 shutdown_connections 2024-03-28T23:34:42.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.245+0000 7fbda33e1740 1 -- 172.21.15.161:0/2789396295 wait complete. 2024-03-28T23:34:42.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.245+0000 7fbda33e1740 1 -- start start 2024-03-28T23:34:42.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.245+0000 7fbda33e1740 1 Processor -- start 2024-03-28T23:34:42.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.246+0000 7fbda33e1740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cbf49e9db0 0x55cbf4a0a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.247+0000 7fbda33e1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cbf49ea190 0x55cbf4a1d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.247+0000 7fbda33e1740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf4a1d760 0x55cbf4a07cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.247+0000 7fbda33e1740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55cbf4a0c090 con 0x55cbf4a1d760 2024-03-28T23:34:42.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.247+0000 7fbda33e1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55cbf4909f70 con 0x55cbf49ea190 2024-03-28T23:34:42.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.247+0000 7fbda33e1740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55cbf4a0bad0 con 0x55cbf49e9db0 2024-03-28T23:34:42.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.247+0000 7fbd9ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf4a1d760 0x55cbf4a07cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.247+0000 7fbd9ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf4a1d760 0x55cbf4a07cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54610/0 (socket says 172.21.15.161:54610) 2024-03-28T23:34:42.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.247+0000 7fbd9ad9d640 1 -- 172.21.15.161:0/1411012154 learned_addr learned my addr 172.21.15.161:0/1411012154 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:42.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.247+0000 7fbda33e1740 10 client.0 _ll_register_callbacks cb 0x55cbf4661ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:34:42.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.247+0000 7fbd9a59c640 1 --2- 172.21.15.161:0/1411012154 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cbf49e9db0 0x55cbf4a0a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.247+0000 7fbd99d9b640 1 --2- 172.21.15.161:0/1411012154 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cbf49ea190 0x55cbf4a1d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.247+0000 7fbd9ad9d640 1 -- 172.21.15.161:0/1411012154 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cbf49e9db0 msgr2=0x55cbf4a0a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:42.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.247+0000 7fbd9ad9d640 1 --2- 172.21.15.161:0/1411012154 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cbf49e9db0 0x55cbf4a0a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.247+0000 7fbd9ad9d640 1 -- 172.21.15.161:0/1411012154 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cbf49ea190 msgr2=0x55cbf4a1d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:42.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.247+0000 7fbd9ad9d640 1 --2- 172.21.15.161:0/1411012154 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cbf49ea190 0x55cbf4a1d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.247+0000 7fbd9ad9d640 1 -- 172.21.15.161:0/1411012154 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbf472c6b0 con 0x55cbf4a1d760 2024-03-28T23:34:42.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.247+0000 7fbd9ad9d640 1 --2- 172.21.15.161:0/1411012154 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf4a1d760 0x55cbf4a07cc0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fbd94054240 tx=0x7fbd94057ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc34ed61fcc9992a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:42.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.247+0000 7fbd797fa640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:34:42.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.247+0000 7fbd797fa640 1 -- 172.21.15.161:0/1411012154 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd94060240 con 0x55cbf4a1d760 2024-03-28T23:34:42.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.247+0000 7fbd797fa640 1 -- 172.21.15.161:0/1411012154 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 6101+0+0 (secure 0 0 0) 0x7fbd94057230 con 0x55cbf4a1d760 2024-03-28T23:34:42.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.247+0000 7fbda33e1740 1 -- 172.21.15.161:0/1411012154 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd84005150 con 0x55cbf4a1d760 2024-03-28T23:34:42.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.247+0000 7fbd797fa640 1 -- 172.21.15.161:0/1411012154 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7fbd94057430 con 0x55cbf4a1d760 2024-03-28T23:34:42.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.247+0000 7fbda33e1740 10 client.8109 fetch_fsmap learned FSMap version 107 2024-03-28T23:34:42.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.247+0000 7fbda33e1740 1 -- 172.21.15.161:0/1411012154 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=2+}) -- 0x55cbf4a0e860 con 0x55cbf4a1d760 2024-03-28T23:34:42.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.248+0000 7fbd797fa640 1 -- 172.21.15.161:0/1411012154 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd84005150 con 0x55cbf4a1d760 2024-03-28T23:34:42.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.248+0000 7fbd797fa640 1 -- 172.21.15.161:0/1411012154 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 107) ==== 3494+0+0 (secure 0 0 0) 0x7fbd9406fb50 con 0x55cbf4a1d760 2024-03-28T23:34:42.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.248+0000 7fbda33e1740 10 client.8109 fetch_fsmap finished waiting for FSMap version 107 2024-03-28T23:34:42.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.248+0000 7fbda33e1740 10 client.8109 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T23:34:42.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.248+0000 7fbda33e1740 20 client.8109 populate_metadata read hostname 'smithi161' 2024-03-28T23:34:42.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.248+0000 7fbda33e1740 1 --2- 172.21.15.161:0/1411012154 >> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] conn(0x55cbf4a3a270 0x55cbf4a44f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.248+0000 7fbda33e1740 4 client.8109 mds_command: new command op to 7974 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:34:42.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.248+0000 7fbda33e1740 1 -- 172.21.15.161:0/1411012154 --> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbf495b030 con 0x55cbf4a3a270 2024-03-28T23:34:42.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.248+0000 7fbd9a59c640 1 --2- 172.21.15.161:0/1411012154 >> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] conn(0x55cbf4a3a270 0x55cbf4a44f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.248+0000 7fbd797fa640 10 client.8109 ms_handle_connect on v2:172.21.15.161:6836/3432367324 2024-03-28T23:34:42.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.248+0000 7fbd9a59c640 1 --2- 172.21.15.161:0/1411012154 >> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] conn(0x55cbf4a3a270 0x55cbf4a44f40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=194ea87c90f4c519 server_cookie=ec4517aba718db5 in_seq=0 out_seq=0 2024-03-28T23:34:42.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.249+0000 7fbd797fa640 1 -- 172.21.15.161:0/1411012154 <== mds.0 v2:172.21.15.161:6836/3432367324 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cbf495b030 con 0x55cbf4a3a270 2024-03-28T23:34:42.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.249+0000 7fbd797fa640 10 client.8109 handle_command_reply: tid=0 2024-03-28T23:34:42.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.249+0000 7fbda33e1740 1 client.8109 shutdown 2024-03-28T23:34:42.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.249+0000 7fbda33e1740 1 -- 172.21.15.161:0/1411012154 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf4a1d760 msgr2=0x55cbf4a07cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:42.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.249+0000 7fbda33e1740 1 --2- 172.21.15.161:0/1411012154 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf4a1d760 0x55cbf4a07cc0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fbd94054240 tx=0x7fbd94057ea0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.249+0000 7fbda33e1740 1 -- 172.21.15.161:0/1411012154 shutdown_connections 2024-03-28T23:34:42.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.249+0000 7fbda33e1740 1 --2- 172.21.15.161:0/1411012154 >> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] conn(0x55cbf4a3a270 0x55cbf4a44f40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.249+0000 7fbda33e1740 1 --2- 172.21.15.161:0/1411012154 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf4a1d760 0x55cbf4a07cc0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.249+0000 7fbda33e1740 1 --2- 172.21.15.161:0/1411012154 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cbf49ea190 0x55cbf4a1d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.249+0000 7fbd797fa640 0 client.8109 ms_handle_reset on v2:172.21.15.161:6836/3432367324 2024-03-28T23:34:42.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.249+0000 7fbda33e1740 1 --2- 172.21.15.161:0/1411012154 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cbf49e9db0 0x55cbf4a0a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.249+0000 7fbda33e1740 1 -- 172.21.15.161:0/1411012154 >> 172.21.15.161:0/1411012154 conn(0x55cbf49e8310 msgr2=0x55cbf49e8740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:42.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.249+0000 7fbda33e1740 1 -- 172.21.15.161:0/1411012154 shutdown_connections 2024-03-28T23:34:42.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.249+0000 7fbda33e1740 1 -- 172.21.15.161:0/1411012154 wait complete. 2024-03-28T23:34:42.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.249+0000 7fbda33e1740 20 client.8109 trim_cache size 0 max 16384 2024-03-28T23:34:42.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.284+0000 7fbda33e1740 1 Processor -- start 2024-03-28T23:34:42.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.284+0000 7fbda33e1740 1 -- start start 2024-03-28T23:34:42.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.285+0000 7fbda33e1740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf49e9db0 0x55cbf49e4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.285+0000 7fbda33e1740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cbf49ea190 0x55cbf4a07800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.285+0000 7fbda33e1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cbf49eb060 0x55cbf49ea570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.285+0000 7fbda33e1740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbd9c1f7670 con 0x55cbf49e9db0 2024-03-28T23:34:42.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.285+0000 7fbda33e1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbd9c14e070 con 0x55cbf49eb060 2024-03-28T23:34:42.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.285+0000 7fbda33e1740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbd9c1f7020 con 0x55cbf49ea190 2024-03-28T23:34:42.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.285+0000 7fbd9a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf49e9db0 0x55cbf49e4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.285+0000 7fbd9a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf49e9db0 0x55cbf49e4eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54624/0 (socket says 172.21.15.161:54624) 2024-03-28T23:34:42.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.285+0000 7fbd9a59c640 1 -- 172.21.15.161:0/1188065601 learned_addr learned my addr 172.21.15.161:0/1188065601 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:42.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.285+0000 7fbd99d9b640 1 --2- 172.21.15.161:0/1188065601 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cbf49ea190 0x55cbf4a07800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.285+0000 7fbd9ad9d640 1 --2- 172.21.15.161:0/1188065601 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cbf49eb060 0x55cbf49ea570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.285+0000 7fbd9a59c640 1 -- 172.21.15.161:0/1188065601 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cbf49ea190 msgr2=0x55cbf4a07800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:42.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.285+0000 7fbd9a59c640 1 --2- 172.21.15.161:0/1188065601 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cbf49ea190 0x55cbf4a07800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.285+0000 7fbd9a59c640 1 -- 172.21.15.161:0/1188065601 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cbf49eb060 msgr2=0x55cbf49ea570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:42.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.285+0000 7fbd9a59c640 1 --2- 172.21.15.161:0/1188065601 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cbf49eb060 0x55cbf49ea570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.285+0000 7fbd9a59c640 1 -- 172.21.15.161:0/1188065601 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbf4a0f4a0 con 0x55cbf49e9db0 2024-03-28T23:34:42.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.285+0000 7fbd9ad9d640 1 --2- 172.21.15.161:0/1188065601 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cbf49eb060 0x55cbf49ea570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:42.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.285+0000 7fbd9a59c640 1 --2- 172.21.15.161:0/1188065601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf49e9db0 0x55cbf49e4eb0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x55cbf4a3a1d0 tx=0x7fbd8c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6566a371d5500390 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:42.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.286+0000 7fbd7b7fe640 1 -- 172.21.15.161:0/1188065601 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c02c550 con 0x55cbf49e9db0 2024-03-28T23:34:42.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.286+0000 7fbd7b7fe640 1 -- 172.21.15.161:0/1188065601 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd8c032070 con 0x55cbf49e9db0 2024-03-28T23:34:42.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.286+0000 7fbd7b7fe640 1 -- 172.21.15.161:0/1188065601 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c023430 con 0x55cbf49e9db0 2024-03-28T23:34:42.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.286+0000 7fbda33e1740 1 -- 172.21.15.161:0/1188065601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf49e9db0 msgr2=0x55cbf49e4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:42.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.286+0000 7fbda33e1740 1 --2- 172.21.15.161:0/1188065601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf49e9db0 0x55cbf49e4eb0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x55cbf4a3a1d0 tx=0x7fbd8c017000 comp rx=0 tx=0).stop 2024-03-28T23:34:42.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.286+0000 7fbda33e1740 1 -- 172.21.15.161:0/1188065601 shutdown_connections 2024-03-28T23:34:42.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.286+0000 7fbda33e1740 1 --2- 172.21.15.161:0/1188065601 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cbf49eb060 0x55cbf49ea570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.286+0000 7fbda33e1740 1 --2- 172.21.15.161:0/1188065601 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cbf49ea190 0x55cbf4a07800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.286+0000 7fbda33e1740 1 --2- 172.21.15.161:0/1188065601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf49e9db0 0x55cbf49e4eb0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.286+0000 7fbda33e1740 1 -- 172.21.15.161:0/1188065601 >> 172.21.15.161:0/1188065601 conn(0x55cbf49e8310 msgr2=0x55cbf49df760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:42.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.286+0000 7fbda33e1740 1 -- 172.21.15.161:0/1188065601 shutdown_connections 2024-03-28T23:34:42.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.286+0000 7fbda33e1740 1 -- 172.21.15.161:0/1188065601 wait complete. 2024-03-28T23:34:42.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.287+0000 7fbda33e1740 1 -- start start 2024-03-28T23:34:42.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.287+0000 7fbda33e1740 1 Processor -- start 2024-03-28T23:34:42.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.288+0000 7fbda33e1740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf49e9db0 0x55cbf49e7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.288+0000 7fbda33e1740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cbf49ea190 0x55cbf4a1b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.288+0000 7fbda33e1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cbf49eb060 0x55cbf49e86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.288+0000 7fbda33e1740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55cbf4a1dc40 con 0x55cbf49e9db0 2024-03-28T23:34:42.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.288+0000 7fbda33e1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55cbf4a086d0 con 0x55cbf49eb060 2024-03-28T23:34:42.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.288+0000 7fbda33e1740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55cbf4a1cf10 con 0x55cbf49ea190 2024-03-28T23:34:42.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.288+0000 7fbda33e1740 10 client.0 _ll_register_callbacks cb 0x7fbd8c009b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:34:42.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.288+0000 7fbd9a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf49e9db0 0x55cbf49e7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.288+0000 7fbd9a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf49e9db0 0x55cbf49e7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54640/0 (socket says 172.21.15.161:54640) 2024-03-28T23:34:42.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.288+0000 7fbd9a59c640 1 -- 172.21.15.161:0/879548330 learned_addr learned my addr 172.21.15.161:0/879548330 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:42.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.288+0000 7fbd99d9b640 1 --2- 172.21.15.161:0/879548330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cbf49ea190 0x55cbf4a1b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.288+0000 7fbd9ad9d640 1 --2- 172.21.15.161:0/879548330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cbf49eb060 0x55cbf49e86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.288+0000 7fbd9a59c640 1 -- 172.21.15.161:0/879548330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cbf49ea190 msgr2=0x55cbf4a1b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:42.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.288+0000 7fbd9a59c640 1 --2- 172.21.15.161:0/879548330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cbf49ea190 0x55cbf4a1b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.288+0000 7fbd9a59c640 1 -- 172.21.15.161:0/879548330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cbf49eb060 msgr2=0x55cbf49e86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:42.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.288+0000 7fbd9a59c640 1 --2- 172.21.15.161:0/879548330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cbf49eb060 0x55cbf49e86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.288+0000 7fbd9a59c640 1 -- 172.21.15.161:0/879548330 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbf4a0ce90 con 0x55cbf49e9db0 2024-03-28T23:34:42.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.288+0000 7fbd9a59c640 1 --2- 172.21.15.161:0/879548330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf49e9db0 0x55cbf49e7800 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c014da0 tx=0x7fbd8c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa4c5c7aed697f6f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:42.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.288+0000 7fbd797fa640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:34:42.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.288+0000 7fbd797fa640 1 -- 172.21.15.161:0/879548330 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c038460 con 0x55cbf49e9db0 2024-03-28T23:34:42.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.288+0000 7fbd797fa640 1 -- 172.21.15.161:0/879548330 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 6101+0+0 (secure 0 0 0) 0x7fbd8c038600 con 0x55cbf49e9db0 2024-03-28T23:34:42.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.288+0000 7fbda33e1740 1 -- 172.21.15.161:0/879548330 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd8c032070 con 0x55cbf49e9db0 2024-03-28T23:34:42.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.289+0000 7fbd797fa640 1 -- 172.21.15.161:0/879548330 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7fbd8c045050 con 0x55cbf49e9db0 2024-03-28T23:34:42.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.289+0000 7fbda33e1740 10 client.8115 fetch_fsmap learned FSMap version 107 2024-03-28T23:34:42.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.289+0000 7fbda33e1740 1 -- 172.21.15.161:0/879548330 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=2+}) -- 0x55cbf4a0adc0 con 0x55cbf49e9db0 2024-03-28T23:34:42.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.289+0000 7fbd797fa640 1 -- 172.21.15.161:0/879548330 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd8c032070 con 0x55cbf49e9db0 2024-03-28T23:34:42.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.289+0000 7fbd797fa640 1 -- 172.21.15.161:0/879548330 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 107) ==== 3494+0+0 (secure 0 0 0) 0x7fbd8c048020 con 0x55cbf49e9db0 2024-03-28T23:34:42.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.289+0000 7fbda33e1740 10 client.8115 fetch_fsmap finished waiting for FSMap version 107 2024-03-28T23:34:42.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.289+0000 7fbda33e1740 10 client.8115 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T23:34:42.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.289+0000 7fbda33e1740 20 client.8115 populate_metadata read hostname 'smithi161' 2024-03-28T23:34:42.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.289+0000 7fbda33e1740 1 --2- 172.21.15.161:0/879548330 >> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] conn(0x55cbf49e9280 0x55cbf4aa0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.289+0000 7fbda33e1740 4 client.8115 mds_command: new command op to 7974 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:34:42.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.289+0000 7fbda33e1740 1 -- 172.21.15.161:0/879548330 --> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] -- command(tid 0: {"prefix": "status"}) -- 0x55cbf4955380 con 0x55cbf49e9280 2024-03-28T23:34:42.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.289+0000 7fbd99d9b640 1 --2- 172.21.15.161:0/879548330 >> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] conn(0x55cbf49e9280 0x55cbf4aa0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.289+0000 7fbd797fa640 10 client.8115 ms_handle_connect on v2:172.21.15.161:6836/3432367324 2024-03-28T23:34:42.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.289+0000 7fbd99d9b640 1 --2- 172.21.15.161:0/879548330 >> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] conn(0x55cbf49e9280 0x55cbf4aa0af0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bc72e21542fce83 server_cookie=d53d22f5ec936854 in_seq=0 out_seq=0 2024-03-28T23:34:42.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.289+0000 7fbd797fa640 1 -- 172.21.15.161:0/879548330 <== mds.0 v2:172.21.15.161:6836/3432367324 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x55cbf4955380 con 0x55cbf49e9280 2024-03-28T23:34:42.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.290+0000 7fbd797fa640 10 client.8115 handle_command_reply: tid=0 2024-03-28T23:34:42.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.290+0000 7fbda33e1740 1 client.8115 shutdown 2024-03-28T23:34:42.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.290+0000 7fbda33e1740 1 -- 172.21.15.161:0/879548330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf49e9db0 msgr2=0x55cbf49e7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:42.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.290+0000 7fbda33e1740 1 --2- 172.21.15.161:0/879548330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf49e9db0 0x55cbf49e7800 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c014da0 tx=0x7fbd8c004ab0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.290+0000 7fbda33e1740 1 -- 172.21.15.161:0/879548330 shutdown_connections 2024-03-28T23:34:42.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.290+0000 7fbda33e1740 1 --2- 172.21.15.161:0/879548330 >> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] conn(0x55cbf49e9280 0x55cbf4aa0af0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.290+0000 7fbda33e1740 1 --2- 172.21.15.161:0/879548330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cbf49eb060 0x55cbf49e86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.290+0000 7fbda33e1740 1 --2- 172.21.15.161:0/879548330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cbf49ea190 0x55cbf4a1b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.290+0000 7fbd797fa640 0 client.8115 ms_handle_reset on v2:172.21.15.161:6836/3432367324 2024-03-28T23:34:42.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.290+0000 7fbda33e1740 1 --2- 172.21.15.161:0/879548330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbf49e9db0 0x55cbf49e7800 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.290+0000 7fbda33e1740 1 -- 172.21.15.161:0/879548330 >> 172.21.15.161:0/879548330 conn(0x55cbf49e8310 msgr2=0x55cbf4a07800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:42.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.290+0000 7fbda33e1740 1 -- 172.21.15.161:0/879548330 shutdown_connections 2024-03-28T23:34:42.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.290+0000 7fbda33e1740 1 -- 172.21.15.161:0/879548330 wait complete. 2024-03-28T23:34:42.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.290+0000 7fbda33e1740 20 client.8115 trim_cache size 0 max 16384 2024-03-28T23:34:42.297 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:34:42.297 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:34:42.297 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:34:42.297 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 7974, 2024-03-28T23:34:42.298 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:34:42.298 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:34:42.298 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:34:42.298 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 4.8440425210000004, 2024-03-28T23:34:42.298 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 107, 2024-03-28T23:34:42.298 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 122, 2024-03-28T23:34:42.298 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 122, 2024-03-28T23:34:42.298 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 10.132298456999999 2024-03-28T23:34:42.298 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:34:42.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.290+0000 7fbda0809640 1 -- 172.21.15.161:0/471015347 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbd7c06b500 msgr2=0x7fbd7c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:42.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.290+0000 7fbda0809640 1 --2- 172.21.15.161:0/471015347 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbd7c06b500 0x7fbd7c06d9c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c014510 tx=0x7fbd8c00b040 comp rx=0 tx=0).stop 2024-03-28T23:34:42.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.290+0000 7fbda0809640 1 -- 172.21.15.161:0/471015347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd9c14b5e0 msgr2=0x7fbd9c0af570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:42.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.290+0000 7fbda0809640 1 --2- 172.21.15.161:0/471015347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd9c14b5e0 0x7fbd9c0af570 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c074cc0 tx=0x7fbd94007430 comp rx=0 tx=0).stop 2024-03-28T23:34:42.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.291+0000 7fbda0809640 1 -- 172.21.15.161:0/471015347 shutdown_connections 2024-03-28T23:34:42.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.291+0000 7fbda0809640 1 --2- 172.21.15.161:0/471015347 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbd7c06b500 0x7fbd7c06d9c0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.291+0000 7fbda0809640 1 --2- 172.21.15.161:0/471015347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd9c14b5e0 0x7fbd9c0af570 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.291+0000 7fbda0809640 1 --2- 172.21.15.161:0/471015347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd9c156600 0x7fbd9c14b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.291+0000 7fbda0809640 1 --2- 172.21.15.161:0/471015347 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbd88004040 0x7fbd9c151b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.291+0000 7fbda0809640 1 -- 172.21.15.161:0/471015347 >> 172.21.15.161:0/471015347 conn(0x7fbd9c0ac720 msgr2=0x7fbd9c0c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:42.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.291+0000 7fbda0809640 1 -- 172.21.15.161:0/471015347 shutdown_connections 2024-03-28T23:34:42.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.291+0000 7fbda0809640 1 -- 172.21.15.161:0/471015347 wait complete. 2024-03-28T23:34:42.304 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:34:42.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.405+0000 7f0f916c0640 1 Processor -- start 2024-03-28T23:34:42.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.405+0000 7f0f916c0640 1 -- start start 2024-03-28T23:34:42.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.405+0000 7f0f916c0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0f8c1532e0 0x7f0f8c0c14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.405+0000 7f0f916c0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0f8c153b60 0x7f0f8c1502a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.405+0000 7f0f916c0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f8c0736b0 con 0x7f0f8c1507e0 2024-03-28T23:34:42.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.405+0000 7f0f916c0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0f8c150c50 con 0x7f0f8c153b60 2024-03-28T23:34:42.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.405+0000 7f0f916c0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f0f8c0730f0 con 0x7f0f8c1532e0 2024-03-28T23:34:42.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.406+0000 7f0f83fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0f8c153b60 0x7f0f8c1502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.406+0000 7f0f83fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0f8c153b60 0x7f0f8c1502a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54644/0 (socket says 172.21.15.161:54644) 2024-03-28T23:34:42.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.406+0000 7f0f83fff640 1 -- 172.21.15.161:0/3065928247 learned_addr learned my addr 172.21.15.161:0/3065928247 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:42.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.406+0000 7f0f83fff640 1 -- 172.21.15.161:0/3065928247 >> v2:172.21.15.170:3300/0 conn(0x7f0f8c1532e0 msgr2=0x7f0f8c0c14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:42.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.406+0000 7f0f8affd640 1 --2- 172.21.15.161:0/3065928247 >> v2:172.21.15.170:3300/0 conn(0x7f0f8c1532e0 0x7f0f8c0c14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.406+0000 7f0f83fff640 1 --2- 172.21.15.161:0/3065928247 >> v2:172.21.15.170:3300/0 conn(0x7f0f8c1532e0 0x7f0f8c0c14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.406+0000 7f0f83fff640 1 -- 172.21.15.161:0/3065928247 >> v1:172.21.15.170:6789/0 conn(0x7f0f8c1507e0 legacy=0x7f0f8c0c2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:42.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.406+0000 7f0f83fff640 1 -- 172.21.15.161:0/3065928247 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f8c0733d0 con 0x7f0f8c153b60 2024-03-28T23:34:42.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.406+0000 7f0f8affd640 1 --2- 172.21.15.161:0/3065928247 >> v2:172.21.15.170:3300/0 conn(0x7f0f8c1532e0 0x7f0f8c0c14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:42.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.407+0000 7f0f83fff640 1 --2- 172.21.15.161:0/3065928247 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f8c153b60 0x7f0f8c1502a0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c04fe50 tx=0x7f0f7402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4562cb029c7243fe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:42.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.407+0000 7f0f8a7fc640 1 -- 172.21.15.161:0/3065928247 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7402e070 con 0x7f0f8c153b60 2024-03-28T23:34:42.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.407+0000 7f0f8a7fc640 1 -- 172.21.15.161:0/3065928247 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f74037070 con 0x7f0f8c153b60 2024-03-28T23:34:42.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.407+0000 7f0f8a7fc640 1 -- 172.21.15.161:0/3065928247 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f74031a00 con 0x7f0f8c153b60 2024-03-28T23:34:42.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.407+0000 7f0f916c0640 1 -- 172.21.15.161:0/3065928247 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f8c153b60 msgr2=0x7f0f8c1502a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:42.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.407+0000 7f0f916c0640 1 --2- 172.21.15.161:0/3065928247 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f8c153b60 0x7f0f8c1502a0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c04fe50 tx=0x7f0f7402b880 comp rx=0 tx=0).stop 2024-03-28T23:34:42.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.407+0000 7f0f916c0640 1 -- 172.21.15.161:0/3065928247 shutdown_connections 2024-03-28T23:34:42.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.407+0000 7f0f916c0640 1 --2- 172.21.15.161:0/3065928247 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f8c153b60 0x7f0f8c1502a0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.407+0000 7f0f916c0640 1 --2- 172.21.15.161:0/3065928247 >> v2:172.21.15.170:3300/0 conn(0x7f0f8c1532e0 0x7f0f8c0c14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.407+0000 7f0f916c0640 1 -- 172.21.15.161:0/3065928247 >> 172.21.15.161:0/3065928247 conn(0x7f0f8c0c6b40 msgr2=0x7f0f8c0c6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:42.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.407+0000 7f0f916c0640 1 -- 172.21.15.161:0/3065928247 shutdown_connections 2024-03-28T23:34:42.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.407+0000 7f0f916c0640 1 -- 172.21.15.161:0/3065928247 wait complete. 2024-03-28T23:34:42.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.408+0000 7f0f916c0640 1 Processor -- start 2024-03-28T23:34:42.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.409+0000 7f0f916c0640 1 -- start start 2024-03-28T23:34:42.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.409+0000 7f0f916c0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f8c1507e0 0x7f0f8c156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.409+0000 7f0f916c0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f8c1532e0 0x7f0f8c1570f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.409+0000 7f0f916c0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f8c153b60 0x7f0f8c0c2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.409+0000 7f0f916c0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0f8c0c1a00 con 0x7f0f8c1507e0 2024-03-28T23:34:42.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.409+0000 7f0f916c0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0f8c070d40 con 0x7f0f8c1532e0 2024-03-28T23:34:42.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.409+0000 7f0f916c0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0f8c0c1b80 con 0x7f0f8c153b60 2024-03-28T23:34:42.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.409+0000 7f0f8affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f8c1507e0 0x7f0f8c156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.409+0000 7f0f8affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f8c1507e0 0x7f0f8c156bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54658/0 (socket says 172.21.15.161:54658) 2024-03-28T23:34:42.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.409+0000 7f0f8affd640 1 -- 172.21.15.161:0/1314369862 learned_addr learned my addr 172.21.15.161:0/1314369862 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:42.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.409+0000 7f0f83fff640 1 --2- 172.21.15.161:0/1314369862 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f8c1532e0 0x7f0f8c1570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.410+0000 7f0f8b7fe640 1 --2- 172.21.15.161:0/1314369862 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f8c153b60 0x7f0f8c0c2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.410+0000 7f0f8affd640 1 -- 172.21.15.161:0/1314369862 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f8c153b60 msgr2=0x7f0f8c0c2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:42.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.410+0000 7f0f8affd640 1 --2- 172.21.15.161:0/1314369862 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f8c153b60 0x7f0f8c0c2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.410+0000 7f0f8affd640 1 -- 172.21.15.161:0/1314369862 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f8c1532e0 msgr2=0x7f0f8c1570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:42.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.410+0000 7f0f8affd640 1 --2- 172.21.15.161:0/1314369862 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f8c1532e0 0x7f0f8c1570f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.410+0000 7f0f8affd640 1 -- 172.21.15.161:0/1314369862 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f8c06c980 con 0x7f0f8c1507e0 2024-03-28T23:34:42.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.410+0000 7f0f83fff640 1 --2- 172.21.15.161:0/1314369862 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f8c1532e0 0x7f0f8c1570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:42.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.410+0000 7f0f8b7fe640 1 --2- 172.21.15.161:0/1314369862 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f8c153b60 0x7f0f8c0c2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:42.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.410+0000 7f0f8affd640 1 --2- 172.21.15.161:0/1314369862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f8c1507e0 0x7f0f8c156bb0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c00c820 tx=0x7f0f7c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:42.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.410+0000 7f0f837fe640 1 -- 172.21.15.161:0/1314369862 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c015070 con 0x7f0f8c1507e0 2024-03-28T23:34:42.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.410+0000 7f0f837fe640 1 -- 172.21.15.161:0/1314369862 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f7c007a30 con 0x7f0f8c1507e0 2024-03-28T23:34:42.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.410+0000 7f0f837fe640 1 -- 172.21.15.161:0/1314369862 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c007e00 con 0x7f0f8c1507e0 2024-03-28T23:34:42.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.410+0000 7f0f916c0640 1 -- 172.21.15.161:0/1314369862 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f8c06cc60 con 0x7f0f8c1507e0 2024-03-28T23:34:42.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.410+0000 7f0f916c0640 1 -- 172.21.15.161:0/1314369862 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f8c0c3a10 con 0x7f0f8c1507e0 2024-03-28T23:34:42.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.411+0000 7f0f916c0640 1 -- 172.21.15.161:0/1314369862 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f8c0c2140 con 0x7f0f8c1507e0 2024-03-28T23:34:42.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.412+0000 7f0f837fe640 1 -- 172.21.15.161:0/1314369862 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0f7c018070 con 0x7f0f8c1507e0 2024-03-28T23:34:42.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.412+0000 7f0f837fe640 1 --2- 172.21.15.161:0/1314369862 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0f6c06b500 0x7f0f6c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.412+0000 7f0f837fe640 1 -- 172.21.15.161:0/1314369862 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6101+0+0 (secure 0 0 0) 0x7f0f7c08a290 con 0x7f0f8c1507e0 2024-03-28T23:34:42.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.412+0000 7f0f83fff640 1 --2- 172.21.15.161:0/1314369862 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0f6c06b500 0x7f0f6c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.412+0000 7f0f83fff640 1 --2- 172.21.15.161:0/1314369862 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0f6c06b500 0x7f0f6c06d9c0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f0f74036000 tx=0x7f0f7403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:42.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.420+0000 7f0f837fe640 1 -- 172.21.15.161:0/1314369862 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0f7c059010 con 0x7f0f8c1507e0 2024-03-28T23:34:42.430 INFO:tasks.ceph.mds.a.smithi161.stderr:2024-03-28T23:34:42.430+0000 7fd53c781640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:34:42.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.636+0000 7f0f916c0640 1 -- 172.21.15.161:0/1314369862 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f8c04f4b0 con 0x7f0f8c1507e0 2024-03-28T23:34:42.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.637+0000 7f0f837fe640 1 -- 172.21.15.161:0/1314369862 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+5198 (secure 0 0 0) 0x7f0f7c058910 con 0x7f0f8c1507e0 2024-03-28T23:34:42.638 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:42.638 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":107,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7318,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3734750343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3734750343},{"type":"v1","addr":"172.21.15.170:6835","nonce":3734750343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/4140438862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":4140438862},{"type":"v1","addr":"172.21.15.170:6834","nonce":4140438862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/878057589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":878057589},{"type":"v1","addr":"172.21.15.161:6835","nonce":878057589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:34:36.796608+0000","modified":"2024-03-28T23:34:38.813377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7974},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7974":{"gid":7974,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6837/3432367324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":3432367324},{"type":"v1","addr":"172.21.15.161:6837","nonce":3432367324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7974,"qdb_cluster":[7974]},"id":9}]} 2024-03-28T23:34:42.638 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 107 2024-03-28T23:34:42.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.639+0000 7f0f916c0640 1 -- 172.21.15.161:0/1314369862 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0f6c06b500 msgr2=0x7f0f6c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:42.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.639+0000 7f0f916c0640 1 --2- 172.21.15.161:0/1314369862 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0f6c06b500 0x7f0f6c06d9c0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f0f74036000 tx=0x7f0f7403d040 comp rx=0 tx=0).stop 2024-03-28T23:34:42.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.639+0000 7f0f916c0640 1 -- 172.21.15.161:0/1314369862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f8c1507e0 msgr2=0x7f0f8c156bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:42.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.639+0000 7f0f916c0640 1 --2- 172.21.15.161:0/1314369862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f8c1507e0 0x7f0f8c156bb0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c00c820 tx=0x7f0f7c00cc20 comp rx=0 tx=0).stop 2024-03-28T23:34:42.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.639+0000 7f0f916c0640 1 -- 172.21.15.161:0/1314369862 shutdown_connections 2024-03-28T23:34:42.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.639+0000 7f0f916c0640 1 --2- 172.21.15.161:0/1314369862 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0f6c06b500 0x7f0f6c06d9c0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.639+0000 7f0f916c0640 1 --2- 172.21.15.161:0/1314369862 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f8c153b60 0x7f0f8c0c2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.639+0000 7f0f916c0640 1 --2- 172.21.15.161:0/1314369862 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f8c1532e0 0x7f0f8c1570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.639+0000 7f0f916c0640 1 --2- 172.21.15.161:0/1314369862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f8c1507e0 0x7f0f8c156bb0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.639+0000 7f0f916c0640 1 -- 172.21.15.161:0/1314369862 >> 172.21.15.161:0/1314369862 conn(0x7f0f8c0c6b40 msgr2=0x7f0f8c150e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:42.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.639+0000 7f0f916c0640 1 -- 172.21.15.161:0/1314369862 shutdown_connections 2024-03-28T23:34:42.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.639+0000 7f0f916c0640 1 -- 172.21.15.161:0/1314369862 wait complete. 2024-03-28T23:34:42.651 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 107, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:34:36.796608+0000", "damaged": [], "data_pools": [ 20 ], "enabled": true, "epoch": 107, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7974": { "addr": "172.21.15.161:6837/3432367324", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6836", "nonce": 3432367324, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 3432367324, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7974, "incarnation": 105, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 19, "modified": "2024-03-28T23:34:38.813377+0000", "qdb_cluster": [ 7974 ], "qdb_leader": 7974, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7974 } } } ], "standbys": [ { "addr": "172.21.15.170:6835/3734750343", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 3734750343, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 3734750343, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 103, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7318, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.170:6834/4140438862", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 4140438862, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 4140438862, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 103, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7321, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6835/878057589", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 878057589, "type": "v2" }, { "addr": "172.21.15.161:6835", "nonce": 878057589, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 103, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7965, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:34:42.651 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-28T23:34:42.651 INFO:tasks.cephfs.test_failover:current = [{'gid': 7974, 'name': 'a', 'rank': 0, 'incarnation': 105, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.161:6837/3432367324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 3432367324}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 3432367324}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 7318, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/3734750343', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 3734750343}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 3734750343}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 103}, {'gid': 7965, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6835/878057589', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 878057589}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 878057589}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 103}, {'gid': 7321, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6834/4140438862', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 4140438862}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 4140438862}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 103}] 2024-03-28T23:34:42.652 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.a mds_join_fs cephfs 2024-03-28T23:34:42.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.756+0000 7fcacaba0640 1 Processor -- start 2024-03-28T23:34:42.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.756+0000 7fcacaba0640 1 -- start start 2024-03-28T23:34:42.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.757+0000 7fcacaba0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcac4152840 0x7fcac4155b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.757+0000 7fcacaba0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fcac406c5d0 0x7fcac406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.757+0000 7fcacaba0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcac40c7560 con 0x7fcac40c88b0 2024-03-28T23:34:42.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.757+0000 7fcacaba0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fcac40c7e00 con 0x7fcac4152840 2024-03-28T23:34:42.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.757+0000 7fcacaba0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fcac40c7840 con 0x7fcac406c5d0 2024-03-28T23:34:42.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.757+0000 7fcac9b9e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcac4152840 0x7fcac4155b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.757+0000 7fcac939d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fcac40c88b0 0x7fcac40c5330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50356/0 (socket says 172.21.15.161:50356) 2024-03-28T23:34:42.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.757+0000 7fcac939d640 1 -- 172.21.15.161:0/1867774984 learned_addr learned my addr 172.21.15.161:0/1867774984 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:42.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.757+0000 7fcaca39f640 1 --2- 172.21.15.161:0/1867774984 >> v2:172.21.15.170:3300/0 conn(0x7fcac406c5d0 0x7fcac406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.757+0000 7fcac9b9e640 1 -- 172.21.15.161:0/1867774984 >> v2:172.21.15.170:3300/0 conn(0x7fcac406c5d0 msgr2=0x7fcac406ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:42.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.757+0000 7fcac9b9e640 1 --2- 172.21.15.161:0/1867774984 >> v2:172.21.15.170:3300/0 conn(0x7fcac406c5d0 0x7fcac406ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.757+0000 7fcac9b9e640 1 -- 172.21.15.161:0/1867774984 >> v1:172.21.15.161:6789/0 conn(0x7fcac40c88b0 legacy=0x7fcac40c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:42.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.757+0000 7fcac9b9e640 1 -- 172.21.15.161:0/1867774984 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcac40c7b20 con 0x7fcac4152840 2024-03-28T23:34:42.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.758+0000 7fcac9b9e640 1 --2- 172.21.15.161:0/1867774984 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcac4152840 0x7fcac4155b30 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fcac406d540 tx=0x7fcac002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac9c544064501a64 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:42.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.759+0000 7fcac8b9c640 1 -- 172.21.15.161:0/1867774984 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac00315a0 con 0x7fcac4152840 2024-03-28T23:34:42.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.759+0000 7fcac8b9c640 1 -- 172.21.15.161:0/1867774984 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcac0037070 con 0x7fcac4152840 2024-03-28T23:34:42.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.759+0000 7fcac8b9c640 1 -- 172.21.15.161:0/1867774984 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac0031960 con 0x7fcac4152840 2024-03-28T23:34:42.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.759+0000 7fcacaba0640 1 -- 172.21.15.161:0/1867774984 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcac4152840 msgr2=0x7fcac4155b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:42.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.759+0000 7fcacaba0640 1 --2- 172.21.15.161:0/1867774984 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcac4152840 0x7fcac4155b30 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fcac406d540 tx=0x7fcac002b840 comp rx=0 tx=0).stop 2024-03-28T23:34:42.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.759+0000 7fcacaba0640 1 -- 172.21.15.161:0/1867774984 shutdown_connections 2024-03-28T23:34:42.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.759+0000 7fcacaba0640 1 --2- 172.21.15.161:0/1867774984 >> v2:172.21.15.170:3300/0 conn(0x7fcac406c5d0 0x7fcac406ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.759+0000 7fcacaba0640 1 --2- 172.21.15.161:0/1867774984 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcac4152840 0x7fcac4155b30 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.759+0000 7fcacaba0640 1 -- 172.21.15.161:0/1867774984 >> 172.21.15.161:0/1867774984 conn(0x7fcac4073160 msgr2=0x7fcac40ac5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:42.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.759+0000 7fcacaba0640 1 -- 172.21.15.161:0/1867774984 shutdown_connections 2024-03-28T23:34:42.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.759+0000 7fcacaba0640 1 -- 172.21.15.161:0/1867774984 wait complete. 2024-03-28T23:34:42.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.760+0000 7fcacaba0640 1 Processor -- start 2024-03-28T23:34:42.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.760+0000 7fcacaba0640 1 -- start start 2024-03-28T23:34:42.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.761+0000 7fcacaba0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcac406c5d0 0x7fcac41ee130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.761+0000 7fcacaba0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcac40c88b0 0x7fcac414b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.761+0000 7fcacaba0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcac4152840 0x7fcac414bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.761+0000 7fcacaba0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcac40be340 con 0x7fcac406c5d0 2024-03-28T23:34:42.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.761+0000 7fcacaba0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fcac40be1c0 con 0x7fcac40c88b0 2024-03-28T23:34:42.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.761+0000 7fcacaba0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fcac406cfc0 con 0x7fcac4152840 2024-03-28T23:34:42.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.761+0000 7fcac9b9e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcac406c5d0 0x7fcac41ee130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.761+0000 7fcac9b9e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcac406c5d0 0x7fcac41ee130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54682/0 (socket says 172.21.15.161:54682) 2024-03-28T23:34:42.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.761+0000 7fcac9b9e640 1 -- 172.21.15.161:0/561533669 learned_addr learned my addr 172.21.15.161:0/561533669 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:42.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.761+0000 7fcac939d640 1 --2- 172.21.15.161:0/561533669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcac40c88b0 0x7fcac414b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.761+0000 7fcaca39f640 1 --2- 172.21.15.161:0/561533669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcac4152840 0x7fcac414bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.761+0000 7fcac9b9e640 1 -- 172.21.15.161:0/561533669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcac4152840 msgr2=0x7fcac414bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:42.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.761+0000 7fcac9b9e640 1 --2- 172.21.15.161:0/561533669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcac4152840 0x7fcac414bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.761+0000 7fcac9b9e640 1 -- 172.21.15.161:0/561533669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcac40c88b0 msgr2=0x7fcac414b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:42.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.761+0000 7fcac9b9e640 1 --2- 172.21.15.161:0/561533669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcac40c88b0 0x7fcac414b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:42.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.761+0000 7fcac9b9e640 1 -- 172.21.15.161:0/561533669 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcac40c3a20 con 0x7fcac406c5d0 2024-03-28T23:34:42.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.761+0000 7fcaca39f640 1 --2- 172.21.15.161:0/561533669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcac4152840 0x7fcac414bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:42.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.762+0000 7fcac9b9e640 1 --2- 172.21.15.161:0/561533669 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcac406c5d0 0x7fcac41ee130 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009890 tx=0x7fcac00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:42.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.762+0000 7fcaba7fc640 1 -- 172.21.15.161:0/561533669 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac0032630 con 0x7fcac406c5d0 2024-03-28T23:34:42.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.762+0000 7fcaba7fc640 1 -- 172.21.15.161:0/561533669 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcac00327d0 con 0x7fcac406c5d0 2024-03-28T23:34:42.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.762+0000 7fcaba7fc640 1 -- 172.21.15.161:0/561533669 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac0032ba0 con 0x7fcac406c5d0 2024-03-28T23:34:42.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.762+0000 7fcacaba0640 1 -- 172.21.15.161:0/561533669 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcac41eebb0 con 0x7fcac406c5d0 2024-03-28T23:34:42.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.762+0000 7fcacaba0640 1 -- 172.21.15.161:0/561533669 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcac4151570 con 0x7fcac406c5d0 2024-03-28T23:34:42.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.763+0000 7fcacaba0640 1 -- 172.21.15.161:0/561533669 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcac404f4b0 con 0x7fcac406c5d0 2024-03-28T23:34:42.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.763+0000 7fcaba7fc640 1 -- 172.21.15.161:0/561533669 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcac003b3f0 con 0x7fcac406c5d0 2024-03-28T23:34:42.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.764+0000 7fcaba7fc640 1 --2- 172.21.15.161:0/561533669 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcaa806b500 0x7fcaa806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:42.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.764+0000 7fcac939d640 1 --2- 172.21.15.161:0/561533669 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcaa806b500 0x7fcaa806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:42.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.764+0000 7fcaba7fc640 1 -- 172.21.15.161:0/561533669 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6101+0+0 (secure 0 0 0) 0x7fcac002f030 con 0x7fcac406c5d0 2024-03-28T23:34:42.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.764+0000 7fcac939d640 1 --2- 172.21.15.161:0/561533669 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcaa806b500 0x7fcaa806d9c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fcac4151e20 tx=0x7fcab4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:42.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:42.768+0000 7fcaba7fc640 1 -- 172.21.15.161:0/561533669 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcac0040070 con 0x7fcac406c5d0 2024-03-28T23:34:43.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.015+0000 7fcacaba0640 1 -- 172.21.15.161:0/561533669 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) -- 0x7fcac406d810 con 0x7fcac406c5d0 2024-03-28T23:34:43.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.018+0000 7fcaba7fc640 1 -- 172.21.15.161:0/561533669 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v42)=0 v42) ==== 116+0+0 (secure 0 0 0) 0x7fcac007aef0 con 0x7fcac406c5d0 2024-03-28T23:34:43.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.020+0000 7fcacaba0640 1 -- 172.21.15.161:0/561533669 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcaa806b500 msgr2=0x7fcaa806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:43.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.020+0000 7fcacaba0640 1 --2- 172.21.15.161:0/561533669 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcaa806b500 0x7fcaa806d9c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fcac4151e20 tx=0x7fcab4009040 comp rx=0 tx=0).stop 2024-03-28T23:34:43.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.020+0000 7fcacaba0640 1 -- 172.21.15.161:0/561533669 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcac406c5d0 msgr2=0x7fcac41ee130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:43.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.020+0000 7fcacaba0640 1 --2- 172.21.15.161:0/561533669 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcac406c5d0 0x7fcac41ee130 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009890 tx=0x7fcac00046c0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.020+0000 7fcacaba0640 1 -- 172.21.15.161:0/561533669 shutdown_connections 2024-03-28T23:34:43.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.020+0000 7fcacaba0640 1 --2- 172.21.15.161:0/561533669 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fcaa806b500 0x7fcaa806d9c0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.020+0000 7fcacaba0640 1 --2- 172.21.15.161:0/561533669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcac4152840 0x7fcac414bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.020+0000 7fcacaba0640 1 --2- 172.21.15.161:0/561533669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcac40c88b0 0x7fcac414b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.020+0000 7fcacaba0640 1 --2- 172.21.15.161:0/561533669 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcac406c5d0 0x7fcac41ee130 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.020+0000 7fcacaba0640 1 -- 172.21.15.161:0/561533669 >> 172.21.15.161:0/561533669 conn(0x7fcac4073160 msgr2=0x7fcac40c52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:43.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.020+0000 7fcacaba0640 1 -- 172.21.15.161:0/561533669 shutdown_connections 2024-03-28T23:34:43.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.020+0000 7fcacaba0640 1 -- 172.21.15.161:0/561533669 wait complete. 2024-03-28T23:34:43.034 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:34:43.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.139+0000 7f9270e55640 1 Processor -- start 2024-03-28T23:34:43.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.139+0000 7f9270e55640 1 -- start start 2024-03-28T23:34:43.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.140+0000 7f9270e55640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f926c155d10 0x7f926c0c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.140+0000 7f9270e55640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f926c06c3c0 con 0x7f926c152a80 2024-03-28T23:34:43.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.140+0000 7f9270e55640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f926c0c5b90 con 0x7f926c06d9f0 2024-03-28T23:34:43.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.140+0000 7f9270e55640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f926c06c6f0 con 0x7f926c155d10 2024-03-28T23:34:43.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.140+0000 7f926a59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f926c152a80 0x7f926c0c93a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50362/0 (socket says 172.21.15.161:50362) 2024-03-28T23:34:43.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.140+0000 7f926a59c640 1 -- 172.21.15.161:0/4207579549 learned_addr learned my addr 172.21.15.161:0/4207579549 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:43.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.140+0000 7f9269d9b640 1 -- 172.21.15.161:0/4207579549 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3864876457 0 0) 0x7f926c06c3c0 con 0x7f926c152a80 2024-03-28T23:34:43.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.140+0000 7f9269d9b640 1 -- 172.21.15.161:0/4207579549 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9258003520 con 0x7f926c152a80 2024-03-28T23:34:43.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.140+0000 7f926ad9d640 1 --2- 172.21.15.161:0/4207579549 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f926c155d10 0x7f926c0c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.140+0000 7f9269d9b640 1 -- 172.21.15.161:0/4207579549 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3006511671 0 0) 0x7f9258003520 con 0x7f926c152a80 2024-03-28T23:34:43.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.140+0000 7f9269d9b640 1 -- 172.21.15.161:0/4207579549 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f926c155d10 msgr2=0x7f926c0c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:43.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.140+0000 7f9269d9b640 1 --2- 172.21.15.161:0/4207579549 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f926c155d10 0x7f926c0c74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.140+0000 7f9269d9b640 1 -- 172.21.15.161:0/4207579549 >> v1:172.21.15.170:6789/0 conn(0x7f926c06d9f0 legacy=0x7f926c06cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:43.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.140+0000 7f9269d9b640 1 -- 172.21.15.161:0/4207579549 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f926c0c55d0 con 0x7f926c152a80 2024-03-28T23:34:43.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.140+0000 7f9269d9b640 1 -- 172.21.15.161:0/4207579549 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f925c001670 con 0x7f926c152a80 2024-03-28T23:34:43.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.140+0000 7f9269d9b640 1 -- 172.21.15.161:0/4207579549 >> v1:172.21.15.161:6789/0 conn(0x7f926c152a80 legacy=0x7f926c0c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:43.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.141+0000 7f9269d9b640 1 --2- 172.21.15.161:0/4207579549 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f926c155d10 0x7f9258003ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.141+0000 7f9269d9b640 1 --2- 172.21.15.161:0/4207579549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9258004060 0x7f9258004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.141+0000 7f9269d9b640 1 --2- 172.21.15.161:0/4207579549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9258004e60 0x7f925800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.141+0000 7f9269d9b640 1 -- 172.21.15.161:0/4207579549 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f926c06c6f0 con 0x7f9258004060 2024-03-28T23:34:43.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.141+0000 7f9269d9b640 1 -- 172.21.15.161:0/4207579549 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9258005680 con 0x7f9258004e60 2024-03-28T23:34:43.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.141+0000 7f9269d9b640 1 -- 172.21.15.161:0/4207579549 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f925800bf00 con 0x7f926c155d10 2024-03-28T23:34:43.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.141+0000 7f9261d9b640 1 --2- 172.21.15.161:0/4207579549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9258004060 0x7f9258004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.141+0000 7f926a59c640 1 --2- 172.21.15.161:0/4207579549 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f926c155d10 0x7f9258003ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.141+0000 7f926ad9d640 1 --2- 172.21.15.161:0/4207579549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9258004e60 0x7f925800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.141+0000 7f9261d9b640 1 -- 172.21.15.161:0/4207579549 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f926c155d10 msgr2=0x7f9258003ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:43.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.141+0000 7f9261d9b640 1 --2- 172.21.15.161:0/4207579549 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f926c155d10 0x7f9258003ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.141+0000 7f9261d9b640 1 -- 172.21.15.161:0/4207579549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9258004e60 msgr2=0x7f925800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:43.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.141+0000 7f9261d9b640 1 --2- 172.21.15.161:0/4207579549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9258004e60 0x7f925800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.141+0000 7f9261d9b640 1 -- 172.21.15.161:0/4207579549 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9258003520 con 0x7f9258004060 2024-03-28T23:34:43.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.142+0000 7f9261d9b640 1 --2- 172.21.15.161:0/4207579549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9258004060 0x7f9258004800 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f9254001780 tx=0x7f9258005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91e6d9344de307e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:43.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.142+0000 7f9269d9b640 1 -- 172.21.15.161:0/4207579549 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9254001370 con 0x7f9258004060 2024-03-28T23:34:43.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.142+0000 7f9269d9b640 1 -- 172.21.15.161:0/4207579549 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92540052d0 con 0x7f9258004060 2024-03-28T23:34:43.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.142+0000 7f9269d9b640 1 -- 172.21.15.161:0/4207579549 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9254003830 con 0x7f9258004060 2024-03-28T23:34:43.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.142+0000 7f9270e55640 1 -- 172.21.15.161:0/4207579549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9258004060 msgr2=0x7f9258004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:43.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.142+0000 7f9270e55640 1 --2- 172.21.15.161:0/4207579549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9258004060 0x7f9258004800 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f9254001780 tx=0x7f9258005830 comp rx=0 tx=0).stop 2024-03-28T23:34:43.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.142+0000 7f926ad9d640 1 -- 172.21.15.161:0/4207579549 reap_dead start 2024-03-28T23:34:43.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.142+0000 7f9270e55640 1 -- 172.21.15.161:0/4207579549 shutdown_connections 2024-03-28T23:34:43.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.142+0000 7f9270e55640 1 -- 172.21.15.161:0/4207579549 >> 172.21.15.161:0/4207579549 conn(0x7f926c0ac720 msgr2=0x7f926c0acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:43.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.142+0000 7f9270e55640 1 -- 172.21.15.161:0/4207579549 shutdown_connections 2024-03-28T23:34:43.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.142+0000 7f9270e55640 1 -- 172.21.15.161:0/4207579549 wait complete. 2024-03-28T23:34:43.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.144+0000 7f9270e55640 1 Processor -- start 2024-03-28T23:34:43.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.144+0000 7f9270e55640 1 -- start start 2024-03-28T23:34:43.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.144+0000 7f9270e55640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9258004060 0x7f926c1eb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.145+0000 7f9270e55640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f926c0af650 0x7f926c1ee820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.145+0000 7f9270e55640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f926c149510 0x7f926c1eed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.145+0000 7f9270e55640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f926c0c74c0 con 0x7f926c149510 2024-03-28T23:34:43.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.145+0000 7f9270e55640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f926c1561a0 con 0x7f9258004060 2024-03-28T23:34:43.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.145+0000 7f9270e55640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f926c0c7640 con 0x7f926c0af650 2024-03-28T23:34:43.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.145+0000 7f926ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f926c149510 0x7f926c1eed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.145+0000 7f926ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f926c149510 0x7f926c1eed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54692/0 (socket says 172.21.15.161:54692) 2024-03-28T23:34:43.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.145+0000 7f926ad9d640 1 -- 172.21.15.161:0/1585991885 learned_addr learned my addr 172.21.15.161:0/1585991885 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:43.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.145+0000 7f926a59c640 1 --2- 172.21.15.161:0/1585991885 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9258004060 0x7f926c1eb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.145+0000 7f9261d9b640 1 --2- 172.21.15.161:0/1585991885 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f926c0af650 0x7f926c1ee820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.145+0000 7f926ad9d640 1 -- 172.21.15.161:0/1585991885 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f926c0af650 msgr2=0x7f926c1ee820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:43.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.145+0000 7f926ad9d640 1 --2- 172.21.15.161:0/1585991885 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f926c0af650 0x7f926c1ee820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.145+0000 7f926ad9d640 1 -- 172.21.15.161:0/1585991885 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9258004060 msgr2=0x7f926c1eb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:43.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.145+0000 7f926ad9d640 1 --2- 172.21.15.161:0/1585991885 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9258004060 0x7f926c1eb790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.145+0000 7f926ad9d640 1 -- 172.21.15.161:0/1585991885 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f926c0ad9f0 con 0x7f926c149510 2024-03-28T23:34:43.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.145+0000 7f926ad9d640 1 --2- 172.21.15.161:0/1585991885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f926c149510 0x7f926c1eed60 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f926c074cc0 tx=0x7f926400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:43.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.145+0000 7f92637fe640 1 -- 172.21.15.161:0/1585991885 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926400fde0 con 0x7f926c149510 2024-03-28T23:34:43.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.145+0000 7f92637fe640 1 -- 172.21.15.161:0/1585991885 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9264015430 con 0x7f926c149510 2024-03-28T23:34:43.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.145+0000 7f9270e55640 1 -- 172.21.15.161:0/1585991885 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f926c1eaf40 con 0x7f926c149510 2024-03-28T23:34:43.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.145+0000 7f92637fe640 1 -- 172.21.15.161:0/1585991885 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9264015820 con 0x7f926c149510 2024-03-28T23:34:43.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.145+0000 7f9270e55640 1 -- 172.21.15.161:0/1585991885 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f926c0ae580 con 0x7f926c149510 2024-03-28T23:34:43.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.146+0000 7f9270e55640 1 -- 172.21.15.161:0/1585991885 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f926c04f4b0 con 0x7f926c149510 2024-03-28T23:34:43.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.147+0000 7f92637fe640 1 -- 172.21.15.161:0/1585991885 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f92640159c0 con 0x7f926c149510 2024-03-28T23:34:43.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.147+0000 7f92637fe640 1 --2- 172.21.15.161:0/1585991885 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f924c06b4d0 0x7f924c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.147+0000 7f92637fe640 1 -- 172.21.15.161:0/1585991885 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6101+0+0 (secure 0 0 0) 0x7f926408b150 con 0x7f926c149510 2024-03-28T23:34:43.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.147+0000 7f926a59c640 1 --2- 172.21.15.161:0/1585991885 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f924c06b4d0 0x7f924c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.148+0000 7f926a59c640 1 --2- 172.21.15.161:0/1585991885 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f924c06b4d0 0x7f924c06d990 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f926c1598c0 tx=0x7f925c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:43.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.156+0000 7f92637fe640 1 -- 172.21.15.161:0/1585991885 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f926c0ad9f0 con 0x7f926c149510 2024-03-28T23:34:43.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.385+0000 7f9270e55640 1 -- 172.21.15.161:0/1585991885 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f926c06d7d0 con 0x7f926c149510 2024-03-28T23:34:43.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.386+0000 7f92637fe640 1 -- 172.21.15.161:0/1585991885 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+5198 (secure 0 0 0) 0x7f9264019030 con 0x7f926c149510 2024-03-28T23:34:43.386 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:43.386 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":107,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7318,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3734750343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3734750343},{"type":"v1","addr":"172.21.15.170:6835","nonce":3734750343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/4140438862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":4140438862},{"type":"v1","addr":"172.21.15.170:6834","nonce":4140438862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/878057589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":878057589},{"type":"v1","addr":"172.21.15.161:6835","nonce":878057589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:34:36.796608+0000","modified":"2024-03-28T23:34:38.813377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7974},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7974":{"gid":7974,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6837/3432367324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":3432367324},{"type":"v1","addr":"172.21.15.161:6837","nonce":3432367324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7974,"qdb_cluster":[7974]},"id":9}]} 2024-03-28T23:34:43.387 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 107 2024-03-28T23:34:43.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.388+0000 7f9270e55640 1 -- 172.21.15.161:0/1585991885 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f924c06b4d0 msgr2=0x7f924c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:43.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.388+0000 7f9270e55640 1 --2- 172.21.15.161:0/1585991885 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f924c06b4d0 0x7f924c06d990 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f926c1598c0 tx=0x7f925c009040 comp rx=0 tx=0).stop 2024-03-28T23:34:43.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.388+0000 7f9270e55640 1 -- 172.21.15.161:0/1585991885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f926c149510 msgr2=0x7f926c1eed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:43.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.388+0000 7f9270e55640 1 --2- 172.21.15.161:0/1585991885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f926c149510 0x7f926c1eed60 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f926c074cc0 tx=0x7f926400f580 comp rx=0 tx=0).stop 2024-03-28T23:34:43.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.388+0000 7f9270e55640 1 -- 172.21.15.161:0/1585991885 shutdown_connections 2024-03-28T23:34:43.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.388+0000 7f9270e55640 1 --2- 172.21.15.161:0/1585991885 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f924c06b4d0 0x7f924c06d990 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.388+0000 7f9270e55640 1 --2- 172.21.15.161:0/1585991885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f926c149510 0x7f926c1eed60 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.388+0000 7f9270e55640 1 --2- 172.21.15.161:0/1585991885 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f926c0af650 0x7f926c1ee820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.388+0000 7f9270e55640 1 --2- 172.21.15.161:0/1585991885 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9258004060 0x7f926c1eb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.388+0000 7f9270e55640 1 -- 172.21.15.161:0/1585991885 >> 172.21.15.161:0/1585991885 conn(0x7f926c0ac720 msgr2=0x7f926c153100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:43.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.388+0000 7f9270e55640 1 -- 172.21.15.161:0/1585991885 shutdown_connections 2024-03-28T23:34:43.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.388+0000 7f9270e55640 1 -- 172.21.15.161:0/1585991885 wait complete. 2024-03-28T23:34:43.805 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:34:36.796608+0000', 'modified': '2024-03-28T23:34:38.813377+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7974}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7974': {'gid': 7974, 'name': 'a', 'rank': 0, 'incarnation': 105, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.161:6837/3432367324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 3432367324}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 3432367324}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7974, 'qdb_cluster': [7974]} 2024-03-28T23:34:43.806 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:34:43.806 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-28T23:34:43.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.909+0000 7f310a8ae640 1 Processor -- start 2024-03-28T23:34:43.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.909+0000 7f310a8ae640 1 -- start start 2024-03-28T23:34:43.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.909+0000 7f310a8ae640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3104152890 0x7f31040c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.909+0000 7f310a8ae640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f31040c8ed0 0x7f31040c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.909+0000 7f310a8ae640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3104158a30 con 0x7f31040c5950 2024-03-28T23:34:43.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.909+0000 7f310a8ae640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f31040c8480 con 0x7f31040c8ed0 2024-03-28T23:34:43.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.909+0000 7f310a8ae640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f3104159640 con 0x7f3104152890 2024-03-28T23:34:43.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.909+0000 7f31037fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f31040c8ed0 0x7f31040c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.910+0000 7f3108e4b640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f31040c5950 0x7f31041553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50372/0 (socket says 172.21.15.161:50372) 2024-03-28T23:34:43.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.910+0000 7f3108e4b640 1 -- 172.21.15.161:0/1730637574 learned_addr learned my addr 172.21.15.161:0/1730637574 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:43.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.910+0000 7f31037fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f31040c8ed0 0x7f31040c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54702/0 (socket says 172.21.15.161:54702) 2024-03-28T23:34:43.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.910+0000 7f31037fe640 1 -- 172.21.15.161:0/1730637574 >> v2:172.21.15.170:3300/0 conn(0x7f3104152890 msgr2=0x7f31040c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:43.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.910+0000 7f3103fff640 1 --2- 172.21.15.161:0/1730637574 >> v2:172.21.15.170:3300/0 conn(0x7f3104152890 0x7f31040c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.910+0000 7f31037fe640 1 --2- 172.21.15.161:0/1730637574 >> v2:172.21.15.170:3300/0 conn(0x7f3104152890 0x7f31040c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.910+0000 7f31037fe640 1 -- 172.21.15.161:0/1730637574 >> v1:172.21.15.161:6789/0 conn(0x7f31040c5950 legacy=0x7f31041553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:43.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.910+0000 7f31037fe640 1 -- 172.21.15.161:0/1730637574 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3104158d10 con 0x7f31040c8ed0 2024-03-28T23:34:43.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.910+0000 7f3102ffd640 1 -- 172.21.15.161:0/1730637574 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1651083045 0 0) 0x7f3104158a30 con 0x7f31040c5950 2024-03-28T23:34:43.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.910+0000 7f3103fff640 1 --2- 172.21.15.161:0/1730637574 >> v2:172.21.15.170:3300/0 conn(0x7f3104152890 0x7f31040c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:43.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.911+0000 7f31037fe640 1 --2- 172.21.15.161:0/1730637574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31040c8ed0 0x7f31040c5410 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f310408cc80 tx=0x7f30f802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c07841502c32270a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:43.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.911+0000 7f3102ffd640 1 -- 172.21.15.161:0/1730637574 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f8038070 con 0x7f31040c8ed0 2024-03-28T23:34:43.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.911+0000 7f3102ffd640 1 -- 172.21.15.161:0/1730637574 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30f8031700 con 0x7f31040c8ed0 2024-03-28T23:34:43.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.911+0000 7f3102ffd640 1 -- 172.21.15.161:0/1730637574 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f8031ad0 con 0x7f31040c8ed0 2024-03-28T23:34:43.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.912+0000 7f310a8ae640 1 -- 172.21.15.161:0/1730637574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31040c8ed0 msgr2=0x7f31040c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:43.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.912+0000 7f310a8ae640 1 --2- 172.21.15.161:0/1730637574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31040c8ed0 0x7f31040c5410 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f310408cc80 tx=0x7f30f802b7a0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.912+0000 7f310a8ae640 1 -- 172.21.15.161:0/1730637574 shutdown_connections 2024-03-28T23:34:43.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.912+0000 7f310a8ae640 1 --2- 172.21.15.161:0/1730637574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31040c8ed0 0x7f31040c5410 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.912+0000 7f310a8ae640 1 --2- 172.21.15.161:0/1730637574 >> v2:172.21.15.170:3300/0 conn(0x7f3104152890 0x7f31040c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.912+0000 7f310a8ae640 1 -- 172.21.15.161:0/1730637574 >> 172.21.15.161:0/1730637574 conn(0x7f31040ac620 msgr2=0x7f31040aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:43.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.912+0000 7f310a8ae640 1 -- 172.21.15.161:0/1730637574 shutdown_connections 2024-03-28T23:34:43.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.912+0000 7f310a8ae640 1 -- 172.21.15.161:0/1730637574 wait complete. 2024-03-28T23:34:43.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.913+0000 7f310a8ae640 1 Processor -- start 2024-03-28T23:34:43.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.914+0000 7f310a8ae640 1 -- start start 2024-03-28T23:34:43.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.915+0000 7f310a8ae640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31040c5950 0x7f31040be750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.915+0000 7f310a8ae640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31040c8ed0 0x7f3104156540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.915+0000 7f310a8ae640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3104152890 0x7f3104156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.915+0000 7f310a8ae640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3104152c70 con 0x7f31040c8ed0 2024-03-28T23:34:43.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.915+0000 7f310a8ae640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f31041592d0 con 0x7f3104152890 2024-03-28T23:34:43.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.915+0000 7f310a8ae640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3104152df0 con 0x7f31040c5950 2024-03-28T23:34:43.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.915+0000 7f31037fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31040c8ed0 0x7f3104156540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.915+0000 7f31037fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31040c8ed0 0x7f3104156540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54716/0 (socket says 172.21.15.161:54716) 2024-03-28T23:34:43.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.915+0000 7f31037fe640 1 -- 172.21.15.161:0/844910517 learned_addr learned my addr 172.21.15.161:0/844910517 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:43.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.915+0000 7f3108e4b640 1 --2- 172.21.15.161:0/844910517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3104152890 0x7f3104156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.915+0000 7f3103fff640 1 --2- 172.21.15.161:0/844910517 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31040c5950 0x7f31040be750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.915+0000 7f31037fe640 1 -- 172.21.15.161:0/844910517 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31040c5950 msgr2=0x7f31040be750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:43.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.915+0000 7f31037fe640 1 --2- 172.21.15.161:0/844910517 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31040c5950 0x7f31040be750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.915+0000 7f31037fe640 1 -- 172.21.15.161:0/844910517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3104152890 msgr2=0x7f3104156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:43.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.915+0000 7f31037fe640 1 --2- 172.21.15.161:0/844910517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3104152890 0x7f3104156a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.915+0000 7f31037fe640 1 -- 172.21.15.161:0/844910517 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31040bede0 con 0x7f31040c8ed0 2024-03-28T23:34:43.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.915+0000 7f3103fff640 1 --2- 172.21.15.161:0/844910517 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31040c5950 0x7f31040be750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:43.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.915+0000 7f31037fe640 1 --2- 172.21.15.161:0/844910517 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31040c8ed0 0x7f3104156540 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f30f8000c00 tx=0x7f30f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:43.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.915+0000 7f3100ff9640 1 -- 172.21.15.161:0/844910517 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f8043070 con 0x7f31040c8ed0 2024-03-28T23:34:43.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.915+0000 7f3100ff9640 1 -- 172.21.15.161:0/844910517 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30f8004220 con 0x7f31040c8ed0 2024-03-28T23:34:43.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.915+0000 7f310a8ae640 1 -- 172.21.15.161:0/844910517 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31041551c0 con 0x7f31040c8ed0 2024-03-28T23:34:43.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.915+0000 7f3100ff9640 1 -- 172.21.15.161:0/844910517 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f8038040 con 0x7f31040c8ed0 2024-03-28T23:34:43.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.916+0000 7f310a8ae640 1 -- 172.21.15.161:0/844910517 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3104155780 con 0x7f31040c8ed0 2024-03-28T23:34:43.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.917+0000 7f3100ff9640 1 -- 172.21.15.161:0/844910517 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f30f8005990 con 0x7f31040c8ed0 2024-03-28T23:34:43.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.918+0000 7f3100ff9640 1 --2- 172.21.15.161:0/844910517 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f30f006b4d0 0x7f30f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.918+0000 7f3100ff9640 1 -- 172.21.15.161:0/844910517 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6101+0+0 (secure 0 0 0) 0x7f30f80ac0e0 con 0x7f31040c8ed0 2024-03-28T23:34:43.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.918+0000 7f3103fff640 1 --2- 172.21.15.161:0/844910517 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f30f006b4d0 0x7f30f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.918+0000 7f3103fff640 1 --2- 172.21.15.161:0/844910517 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f30f006b4d0 0x7f30f006d990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f30f400aa30 tx=0x7f30f4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:43.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.918+0000 7f310d56d740 1 Processor -- start 2024-03-28T23:34:43.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.918+0000 7f310d56d740 1 -- start start 2024-03-28T23:34:43.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.919+0000 7f310d56d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556268917db0 0x556268915890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.919+0000 7f310d56d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556268918190 0x556268915dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.919+0000 7f310d56d740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556268916310 0x5562689124e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.919+0000 7f310d56d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55626887baf0 con 0x556268917db0 2024-03-28T23:34:43.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.919+0000 7f310d56d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5562688249d0 con 0x556268918190 2024-03-28T23:34:43.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.919+0000 7f310d56d740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x556268877b90 con 0x556268916310 2024-03-28T23:34:43.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.919+0000 7f3103fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556268917db0 0x556268915890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.919+0000 7f3103fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556268917db0 0x556268915890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54724/0 (socket says 172.21.15.161:54724) 2024-03-28T23:34:43.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.919+0000 7f3103fff640 1 -- 172.21.15.161:0/2398638127 learned_addr learned my addr 172.21.15.161:0/2398638127 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:43.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.919+0000 7f3103fff640 1 -- 172.21.15.161:0/2398638127 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556268916310 msgr2=0x5562689124e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:43.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.919+0000 7f3103fff640 1 --2- 172.21.15.161:0/2398638127 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556268916310 0x5562689124e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.919+0000 7f3103fff640 1 -- 172.21.15.161:0/2398638127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556268918190 msgr2=0x556268915dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:43.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.919+0000 7f31037fe640 1 --2- 172.21.15.161:0/2398638127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556268918190 0x556268915dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.919+0000 7f3103fff640 1 --2- 172.21.15.161:0/2398638127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556268918190 0x556268915dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.920+0000 7f3103fff640 1 -- 172.21.15.161:0/2398638127 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562685902d0 con 0x556268917db0 2024-03-28T23:34:43.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.920+0000 7f3103fff640 1 --2- 172.21.15.161:0/2398638127 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556268917db0 0x556268915890 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f30f400fb00 tx=0x7f30f4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3157e9023fa65823 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:43.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.920+0000 7f30e9ffb640 1 -- 172.21.15.161:0/2398638127 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f401b050 con 0x556268917db0 2024-03-28T23:34:43.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.920+0000 7f30e9ffb640 1 -- 172.21.15.161:0/2398638127 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30f4025070 con 0x556268917db0 2024-03-28T23:34:43.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.920+0000 7f30e9ffb640 1 -- 172.21.15.161:0/2398638127 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f4014e20 con 0x556268917db0 2024-03-28T23:34:43.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.920+0000 7f310d56d740 1 -- 172.21.15.161:0/2398638127 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556268917db0 msgr2=0x556268915890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:43.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.920+0000 7f310d56d740 1 --2- 172.21.15.161:0/2398638127 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556268917db0 0x556268915890 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f30f400fb00 tx=0x7f30f4012000 comp rx=0 tx=0).stop 2024-03-28T23:34:43.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.920+0000 7f310d56d740 1 -- 172.21.15.161:0/2398638127 shutdown_connections 2024-03-28T23:34:43.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.920+0000 7f310d56d740 1 --2- 172.21.15.161:0/2398638127 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556268916310 0x5562689124e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.920+0000 7f310d56d740 1 --2- 172.21.15.161:0/2398638127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556268918190 0x556268915dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.920+0000 7f310d56d740 1 --2- 172.21.15.161:0/2398638127 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556268917db0 0x556268915890 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.920+0000 7f310d56d740 1 -- 172.21.15.161:0/2398638127 >> 172.21.15.161:0/2398638127 conn(0x556268914630 msgr2=0x556268939220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:43.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.921+0000 7f310d56d740 1 -- 172.21.15.161:0/2398638127 shutdown_connections 2024-03-28T23:34:43.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.921+0000 7f310d56d740 1 -- 172.21.15.161:0/2398638127 wait complete. 2024-03-28T23:34:43.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.921+0000 7f310d56d740 1 -- start start 2024-03-28T23:34:43.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.921+0000 7f310d56d740 1 Processor -- start 2024-03-28T23:34:43.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.922+0000 7f310d56d740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556268917db0 0x556268953df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.922+0000 7f310d56d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556268918190 0x5562689654b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.922+0000 7f310d56d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55626893f460 0x556268934a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.922+0000 7f310d56d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55626893a090 con 0x55626893f460 2024-03-28T23:34:43.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.922+0000 7f310d56d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x556268837f70 con 0x556268918190 2024-03-28T23:34:43.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.922+0000 7f310d56d740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x556268939ad0 con 0x556268917db0 2024-03-28T23:34:43.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.922+0000 7f310d56d740 10 client.0 _ll_register_callbacks cb 0x55626858fac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:34:43.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.922+0000 7f3108e4b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55626893f460 0x556268934a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.922+0000 7f3108e4b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55626893f460 0x556268934a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54738/0 (socket says 172.21.15.161:54738) 2024-03-28T23:34:43.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.922+0000 7f3108e4b640 1 -- 172.21.15.161:0/579638876 learned_addr learned my addr 172.21.15.161:0/579638876 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:43.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.922+0000 7f31037fe640 1 --2- 172.21.15.161:0/579638876 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556268918190 0x5562689654b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.922+0000 7f3103fff640 1 --2- 172.21.15.161:0/579638876 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556268917db0 0x556268953df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.922+0000 7f3108e4b640 1 -- 172.21.15.161:0/579638876 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556268917db0 msgr2=0x556268953df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:43.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.922+0000 7f3108e4b640 1 --2- 172.21.15.161:0/579638876 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556268917db0 0x556268953df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.922+0000 7f3108e4b640 1 -- 172.21.15.161:0/579638876 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556268918190 msgr2=0x5562689654b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:43.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.922+0000 7f3108e4b640 1 --2- 172.21.15.161:0/579638876 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556268918190 0x5562689654b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.922+0000 7f3108e4b640 1 -- 172.21.15.161:0/579638876 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55626865a6b0 con 0x55626893f460 2024-03-28T23:34:43.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.923+0000 7f3108e4b640 1 --2- 172.21.15.161:0/579638876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55626893f460 0x556268934a70 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f30fc007340 tx=0x7f30fc007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99269fef0903f0b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:43.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.923+0000 7f30e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:34:43.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.923+0000 7f30e2ffd640 1 -- 172.21.15.161:0/579638876 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30fc01a060 con 0x55626893f460 2024-03-28T23:34:43.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.923+0000 7f30e2ffd640 1 -- 172.21.15.161:0/579638876 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 6101+0+0 (secure 0 0 0) 0x7f30fc015c90 con 0x55626893f460 2024-03-28T23:34:43.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.923+0000 7f310d56d740 1 -- 172.21.15.161:0/579638876 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30f4025070 con 0x55626893f460 2024-03-28T23:34:43.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.923+0000 7f30e2ffd640 1 -- 172.21.15.161:0/579638876 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7f30fc0079f0 con 0x55626893f460 2024-03-28T23:34:43.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.923+0000 7f310d56d740 10 client.8151 fetch_fsmap learned FSMap version 107 2024-03-28T23:34:43.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.923+0000 7f310d56d740 1 -- 172.21.15.161:0/579638876 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=2+}) -- 0x556268964c40 con 0x55626893f460 2024-03-28T23:34:43.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.923+0000 7f30e2ffd640 1 -- 172.21.15.161:0/579638876 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30f4025070 con 0x55626893f460 2024-03-28T23:34:43.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.923+0000 7f30e2ffd640 1 -- 172.21.15.161:0/579638876 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 107) ==== 3494+0+0 (secure 0 0 0) 0x7f30fc00f940 con 0x55626893f460 2024-03-28T23:34:43.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.923+0000 7f310d56d740 10 client.8151 fetch_fsmap finished waiting for FSMap version 107 2024-03-28T23:34:43.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.923+0000 7f310d56d740 10 client.8151 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T23:34:43.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.923+0000 7f310d56d740 20 client.8151 populate_metadata read hostname 'smithi161' 2024-03-28T23:34:43.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.923+0000 7f310d56d740 1 --2- 172.21.15.161:0/579638876 >> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] conn(0x556268967f50 0x556268972c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.923+0000 7f310d56d740 4 client.8151 mds_command: new command op to 7974 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:34:43.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.923+0000 7f310d56d740 1 -- 172.21.15.161:0/579638876 --> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556268889030 con 0x556268967f50 2024-03-28T23:34:43.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.923+0000 7f3103fff640 1 --2- 172.21.15.161:0/579638876 >> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] conn(0x556268967f50 0x556268972c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.924+0000 7f30e2ffd640 10 client.8151 ms_handle_connect on v2:172.21.15.161:6836/3432367324 2024-03-28T23:34:43.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.924+0000 7f3103fff640 1 --2- 172.21.15.161:0/579638876 >> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] conn(0x556268967f50 0x556268972c10 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfde529e3a90a18 server_cookie=96e022574e5f1a19 in_seq=0 out_seq=0 2024-03-28T23:34:43.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.924+0000 7f30e2ffd640 1 -- 172.21.15.161:0/579638876 <== mds.0 v2:172.21.15.161:6836/3432367324 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556268889030 con 0x556268967f50 2024-03-28T23:34:43.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.924+0000 7f30e2ffd640 10 client.8151 handle_command_reply: tid=0 2024-03-28T23:34:43.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.924+0000 7f310d56d740 1 client.8151 shutdown 2024-03-28T23:34:43.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.924+0000 7f310d56d740 1 -- 172.21.15.161:0/579638876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55626893f460 msgr2=0x556268934a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:43.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.924+0000 7f310d56d740 1 --2- 172.21.15.161:0/579638876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55626893f460 0x556268934a70 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f30fc007340 tx=0x7f30fc007800 comp rx=0 tx=0).stop 2024-03-28T23:34:43.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.924+0000 7f310d56d740 1 -- 172.21.15.161:0/579638876 shutdown_connections 2024-03-28T23:34:43.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.924+0000 7f310d56d740 1 --2- 172.21.15.161:0/579638876 >> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] conn(0x556268967f50 0x556268972c10 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.924+0000 7f310d56d740 1 --2- 172.21.15.161:0/579638876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55626893f460 0x556268934a70 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.924+0000 7f310d56d740 1 --2- 172.21.15.161:0/579638876 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556268918190 0x5562689654b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.924+0000 7f30e2ffd640 0 client.8151 ms_handle_reset on v2:172.21.15.161:6836/3432367324 2024-03-28T23:34:43.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.924+0000 7f310d56d740 1 --2- 172.21.15.161:0/579638876 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556268917db0 0x556268953df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.924+0000 7f310d56d740 1 -- 172.21.15.161:0/579638876 >> 172.21.15.161:0/579638876 conn(0x556268916310 msgr2=0x5562689112b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:43.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.924+0000 7f310d56d740 1 -- 172.21.15.161:0/579638876 shutdown_connections 2024-03-28T23:34:43.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.925+0000 7f310d56d740 1 -- 172.21.15.161:0/579638876 wait complete. 2024-03-28T23:34:43.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.925+0000 7f310d56d740 20 client.8151 trim_cache size 0 max 16384 2024-03-28T23:34:43.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.942+0000 7f310d56d740 1 Processor -- start 2024-03-28T23:34:43.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.942+0000 7f310d56d740 1 -- start start 2024-03-28T23:34:43.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.943+0000 7f310d56d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556268917db0 0x556268911a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.943+0000 7f310d56d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556268918190 0x556268957460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.943+0000 7f310d56d740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55626893f460 0x5562689166f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.943+0000 7f310d56d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f31041f6d10 con 0x556268917db0 2024-03-28T23:34:43.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.943+0000 7f310d56d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f31041eb440 con 0x556268918190 2024-03-28T23:34:43.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.943+0000 7f310d56d740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f31041f6920 con 0x55626893f460 2024-03-28T23:34:43.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.943+0000 7f3103fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556268917db0 0x556268911a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.943+0000 7f3103fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556268917db0 0x556268911a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54748/0 (socket says 172.21.15.161:54748) 2024-03-28T23:34:43.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.943+0000 7f3103fff640 1 -- 172.21.15.161:0/2460020413 learned_addr learned my addr 172.21.15.161:0/2460020413 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:43.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.943+0000 7f3103fff640 1 -- 172.21.15.161:0/2460020413 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55626893f460 msgr2=0x5562689166f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:43.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.943+0000 7f3103fff640 1 --2- 172.21.15.161:0/2460020413 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55626893f460 0x5562689166f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.943+0000 7f31037fe640 1 --2- 172.21.15.161:0/2460020413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556268918190 0x556268957460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.943+0000 7f3103fff640 1 -- 172.21.15.161:0/2460020413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556268918190 msgr2=0x556268957460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:43.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.943+0000 7f3103fff640 1 --2- 172.21.15.161:0/2460020413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556268918190 0x556268957460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.943+0000 7f3103fff640 1 -- 172.21.15.161:0/2460020413 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556268873050 con 0x556268917db0 2024-03-28T23:34:43.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.943+0000 7f31037fe640 1 --2- 172.21.15.161:0/2460020413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556268918190 0x556268957460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:43.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.944+0000 7f3103fff640 1 --2- 172.21.15.161:0/2460020413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556268917db0 0x556268911a00 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x556268957430 tx=0x7f30f4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b2068ba6cf612e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:43.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.944+0000 7f30e97fa640 1 -- 172.21.15.161:0/2460020413 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f403b070 con 0x556268917db0 2024-03-28T23:34:43.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.944+0000 7f30e97fa640 1 -- 172.21.15.161:0/2460020413 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30f4025040 con 0x556268917db0 2024-03-28T23:34:43.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.944+0000 7f30e97fa640 1 -- 172.21.15.161:0/2460020413 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f401b040 con 0x556268917db0 2024-03-28T23:34:43.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.944+0000 7f310d56d740 1 -- 172.21.15.161:0/2460020413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556268917db0 msgr2=0x556268911a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:43.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.944+0000 7f310d56d740 1 --2- 172.21.15.161:0/2460020413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556268917db0 0x556268911a00 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x556268957430 tx=0x7f30f4014000 comp rx=0 tx=0).stop 2024-03-28T23:34:43.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.944+0000 7f310d56d740 1 -- 172.21.15.161:0/2460020413 shutdown_connections 2024-03-28T23:34:43.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.944+0000 7f310d56d740 1 --2- 172.21.15.161:0/2460020413 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55626893f460 0x5562689166f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.944+0000 7f310d56d740 1 --2- 172.21.15.161:0/2460020413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556268918190 0x556268957460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.944+0000 7f310d56d740 1 --2- 172.21.15.161:0/2460020413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556268917db0 0x556268911a00 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.944+0000 7f310d56d740 1 -- 172.21.15.161:0/2460020413 >> 172.21.15.161:0/2460020413 conn(0x556268916310 msgr2=0x5562689544b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:43.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.944+0000 7f310d56d740 1 -- 172.21.15.161:0/2460020413 shutdown_connections 2024-03-28T23:34:43.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.944+0000 7f310d56d740 1 -- 172.21.15.161:0/2460020413 wait complete. 2024-03-28T23:34:43.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.945+0000 7f310d56d740 1 -- start start 2024-03-28T23:34:43.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.945+0000 7f310d56d740 1 Processor -- start 2024-03-28T23:34:43.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.946+0000 7f310d56d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556268917db0 0x556268931a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.946+0000 7f310d56d740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556268918190 0x556268915570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.946+0000 7f310d56d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55626893f460 0x556268915ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.947+0000 7f310d56d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55626893e2a0 con 0x55626893f460 2024-03-28T23:34:43.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.947+0000 7f310d56d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x556268965f90 con 0x556268917db0 2024-03-28T23:34:43.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.947+0000 7f310d56d740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x556268965be0 con 0x556268918190 2024-03-28T23:34:43.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.947+0000 7f310d56d740 10 client.0 _ll_register_callbacks cb 0x7f30fc004bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:34:43.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.947+0000 7f3108e4b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55626893f460 0x556268915ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.947+0000 7f3108e4b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55626893f460 0x556268915ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54762/0 (socket says 172.21.15.161:54762) 2024-03-28T23:34:43.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.947+0000 7f3108e4b640 1 -- 172.21.15.161:0/2609558746 learned_addr learned my addr 172.21.15.161:0/2609558746 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:43.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.947+0000 7f3103fff640 1 --2- 172.21.15.161:0/2609558746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556268917db0 0x556268931a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.947+0000 7f31037fe640 1 --2- 172.21.15.161:0/2609558746 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556268918190 0x556268915570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.947+0000 7f3108e4b640 1 -- 172.21.15.161:0/2609558746 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556268918190 msgr2=0x556268915570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:43.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.947+0000 7f3108e4b640 1 --2- 172.21.15.161:0/2609558746 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556268918190 0x556268915570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.947+0000 7f3108e4b640 1 -- 172.21.15.161:0/2609558746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556268917db0 msgr2=0x556268931a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:43.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.947+0000 7f3108e4b640 1 --2- 172.21.15.161:0/2609558746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556268917db0 0x556268931a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.947+0000 7f3108e4b640 1 -- 172.21.15.161:0/2609558746 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556268951f30 con 0x55626893f460 2024-03-28T23:34:43.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.947+0000 7f3108e4b640 1 --2- 172.21.15.161:0/2609558746 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55626893f460 0x556268915ab0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f30fc007840 tx=0x7f30fc0015c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab1e0d41d77b57cb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:43.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.947+0000 7f30e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:34:43.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.947+0000 7f30e2ffd640 1 -- 172.21.15.161:0/2609558746 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30fc0011c0 con 0x55626893f460 2024-03-28T23:34:43.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.947+0000 7f30e2ffd640 1 -- 172.21.15.161:0/2609558746 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 6101+0+0 (secure 0 0 0) 0x7f30fc02ac60 con 0x55626893f460 2024-03-28T23:34:43.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.947+0000 7f310d56d740 1 -- 172.21.15.161:0/2609558746 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30f4025040 con 0x55626893f460 2024-03-28T23:34:43.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.947+0000 7f30e2ffd640 1 -- 172.21.15.161:0/2609558746 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7f30fc02ae60 con 0x55626893f460 2024-03-28T23:34:43.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.947+0000 7f310d56d740 10 client.8157 fetch_fsmap learned FSMap version 107 2024-03-28T23:34:43.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.947+0000 7f310d56d740 1 -- 172.21.15.161:0/2609558746 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=2+}) -- 0x556268935480 con 0x55626893f460 2024-03-28T23:34:43.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.948+0000 7f30e2ffd640 1 -- 172.21.15.161:0/2609558746 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30f4025040 con 0x55626893f460 2024-03-28T23:34:43.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.948+0000 7f30e2ffd640 1 -- 172.21.15.161:0/2609558746 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 107) ==== 3494+0+0 (secure 0 0 0) 0x7f30fc031050 con 0x55626893f460 2024-03-28T23:34:43.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.948+0000 7f310d56d740 10 client.8157 fetch_fsmap finished waiting for FSMap version 107 2024-03-28T23:34:43.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.948+0000 7f310d56d740 10 client.8157 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T23:34:43.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.948+0000 7f310d56d740 20 client.8157 populate_metadata read hostname 'smithi161' 2024-03-28T23:34:43.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.948+0000 7f310d56d740 1 --2- 172.21.15.161:0/2609558746 >> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] conn(0x5562689cbe20 0x5562689cc200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:43.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.948+0000 7f310d56d740 4 client.8157 mds_command: new command op to 7974 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:34:43.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.948+0000 7f310d56d740 1 -- 172.21.15.161:0/2609558746 --> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] -- command(tid 0: {"prefix": "status"}) -- 0x556268883380 con 0x5562689cbe20 2024-03-28T23:34:43.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.948+0000 7f3103fff640 1 --2- 172.21.15.161:0/2609558746 >> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] conn(0x5562689cbe20 0x5562689cc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:43.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.948+0000 7f3103fff640 1 --2- 172.21.15.161:0/2609558746 >> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] conn(0x5562689cbe20 0x5562689cc200 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9702a60201b96905 server_cookie=1c1fdd156f584737 in_seq=0 out_seq=0 2024-03-28T23:34:43.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.948+0000 7f30e2ffd640 10 client.8157 ms_handle_connect on v2:172.21.15.161:6836/3432367324 2024-03-28T23:34:43.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.948+0000 7f30e2ffd640 1 -- 172.21.15.161:0/2609558746 <== mds.0 v2:172.21.15.161:6836/3432367324 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x556268883380 con 0x5562689cbe20 2024-03-28T23:34:43.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.948+0000 7f30e2ffd640 10 client.8157 handle_command_reply: tid=0 2024-03-28T23:34:43.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.948+0000 7f310d56d740 1 client.8157 shutdown 2024-03-28T23:34:43.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.949+0000 7f310d56d740 1 -- 172.21.15.161:0/2609558746 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55626893f460 msgr2=0x556268915ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:43.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.949+0000 7f310d56d740 1 --2- 172.21.15.161:0/2609558746 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55626893f460 0x556268915ab0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f30fc007840 tx=0x7f30fc0015c0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.949+0000 7f310d56d740 1 -- 172.21.15.161:0/2609558746 shutdown_connections 2024-03-28T23:34:43.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.949+0000 7f310d56d740 1 --2- 172.21.15.161:0/2609558746 >> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] conn(0x5562689cbe20 0x5562689cc200 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.949+0000 7f310d56d740 1 --2- 172.21.15.161:0/2609558746 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55626893f460 0x556268915ab0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.949+0000 7f310d56d740 1 --2- 172.21.15.161:0/2609558746 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556268918190 0x556268915570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.949+0000 7f30e2ffd640 0 client.8157 ms_handle_reset on v2:172.21.15.161:6836/3432367324 2024-03-28T23:34:43.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.949+0000 7f310d56d740 1 --2- 172.21.15.161:0/2609558746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556268917db0 0x556268931a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.949+0000 7f310d56d740 1 -- 172.21.15.161:0/2609558746 >> 172.21.15.161:0/2609558746 conn(0x556268916310 msgr2=0x556268957430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:43.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.949+0000 7f310d56d740 1 -- 172.21.15.161:0/2609558746 shutdown_connections 2024-03-28T23:34:43.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.949+0000 7f310d56d740 1 -- 172.21.15.161:0/2609558746 wait complete. 2024-03-28T23:34:43.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.949+0000 7f310d56d740 20 client.8157 trim_cache size 0 max 16384 2024-03-28T23:34:43.956 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:34:43.956 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:34:43.956 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:34:43.956 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 7974, 2024-03-28T23:34:43.956 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:34:43.956 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:34:43.956 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:34:43.956 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 6.5028838310000001, 2024-03-28T23:34:43.956 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 107, 2024-03-28T23:34:43.956 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 122, 2024-03-28T23:34:43.956 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 122, 2024-03-28T23:34:43.956 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 11.791137859000001 2024-03-28T23:34:43.956 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:34:43.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.949+0000 7f310a8ae640 1 -- 172.21.15.161:0/844910517 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f30f006b4d0 msgr2=0x7f30f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:43.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.949+0000 7f310a8ae640 1 --2- 172.21.15.161:0/844910517 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f30f006b4d0 0x7f30f006d990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f30f400aa30 tx=0x7f30f4008040 comp rx=0 tx=0).stop 2024-03-28T23:34:43.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.949+0000 7f310a8ae640 1 -- 172.21.15.161:0/844910517 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31040c8ed0 msgr2=0x7f3104156540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:43.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.949+0000 7f310a8ae640 1 --2- 172.21.15.161:0/844910517 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31040c8ed0 0x7f3104156540 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f30f8000c00 tx=0x7f30f80046c0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.949+0000 7f310a8ae640 1 -- 172.21.15.161:0/844910517 shutdown_connections 2024-03-28T23:34:43.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.949+0000 7f310a8ae640 1 --2- 172.21.15.161:0/844910517 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f30f006b4d0 0x7f30f006d990 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.949+0000 7f310a8ae640 1 --2- 172.21.15.161:0/844910517 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31040c8ed0 0x7f3104156540 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.949+0000 7f310a8ae640 1 --2- 172.21.15.161:0/844910517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3104152890 0x7f3104156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.949+0000 7f310a8ae640 1 --2- 172.21.15.161:0/844910517 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31040c5950 0x7f31040be750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:43.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.949+0000 7f310a8ae640 1 -- 172.21.15.161:0/844910517 >> 172.21.15.161:0/844910517 conn(0x7f31040ac620 msgr2=0x7f31040c9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:43.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.949+0000 7f310a8ae640 1 -- 172.21.15.161:0/844910517 shutdown_connections 2024-03-28T23:34:43.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:43.949+0000 7f310a8ae640 1 -- 172.21.15.161:0/844910517 wait complete. 2024-03-28T23:34:43.960 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 107, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:34:36.796608+0000", "damaged": [], "data_pools": [ 20 ], "enabled": true, "epoch": 107, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7974": { "addr": "172.21.15.161:6837/3432367324", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6836", "nonce": 3432367324, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 3432367324, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7974, "incarnation": 105, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 19, "modified": "2024-03-28T23:34:38.813377+0000", "qdb_cluster": [ 7974 ], "qdb_leader": 7974, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7974 } } } ], "standbys": [ { "addr": "172.21.15.170:6835/3734750343", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 3734750343, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 3734750343, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 103, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7318, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.170:6834/4140438862", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 4140438862, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 4140438862, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 103, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7321, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6835/878057589", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 878057589, "type": "v2" }, { "addr": "172.21.15.161:6835", "nonce": 878057589, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 103, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7965, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:34:43.961 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': 9, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-28T23:34:43.961 INFO:tasks.cephfs.test_failover:current = [{'gid': 7974, 'name': 'a', 'rank': 0, 'incarnation': 105, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.161:6837/3432367324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 3432367324}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 3432367324}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 7318, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/3734750343', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 3734750343}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 3734750343}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 103}, {'gid': 7965, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6835/878057589', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 878057589}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 878057589}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 103}, {'gid': 7321, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6834/4140438862', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 4140438862}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 4140438862}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 103}] 2024-03-28T23:34:43.962 DEBUG:tasks.cephfs.test_failover:9 != -1 2024-03-28T23:34:43.962 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-28T23:34:48.963 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:34:49.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.081+0000 7f2dacf81640 1 Processor -- start 2024-03-28T23:34:49.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.081+0000 7f2dacf81640 1 -- start start 2024-03-28T23:34:49.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.082+0000 7f2dacf81640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2da8152890 0x7f2da80c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.082+0000 7f2dacf81640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2da80c5b20 0x7f2da81553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.082+0000 7f2dacf81640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2da8158a30 con 0x7f2da80c8ed0 2024-03-28T23:34:49.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.082+0000 7f2dacf81640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2da80c8480 con 0x7f2da8152890 2024-03-28T23:34:49.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.082+0000 7f2dacf81640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2da8159640 con 0x7f2da80c5b20 2024-03-28T23:34:49.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.082+0000 7f2da659c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2da8152890 0x7f2da80c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.082+0000 7f2da5d9b640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f2da80c8ed0 0x7f2da80c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50388/0 (socket says 172.21.15.161:50388) 2024-03-28T23:34:49.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.082+0000 7f2da5d9b640 1 -- 172.21.15.161:0/2883249679 learned_addr learned my addr 172.21.15.161:0/2883249679 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:49.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.083+0000 7f2da6d9d640 1 --2- 172.21.15.161:0/2883249679 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2da80c5b20 0x7f2da81553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.083+0000 7f2da659c640 1 -- 172.21.15.161:0/2883249679 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2da80c5b20 msgr2=0x7f2da81553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:49.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.083+0000 7f2da659c640 1 --2- 172.21.15.161:0/2883249679 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2da80c5b20 0x7f2da81553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.083+0000 7f2da659c640 1 -- 172.21.15.161:0/2883249679 >> v1:172.21.15.161:6789/0 conn(0x7f2da80c8ed0 legacy=0x7f2da80c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:49.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.083+0000 7f2da659c640 1 -- 172.21.15.161:0/2883249679 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da8158d10 con 0x7f2da8152890 2024-03-28T23:34:49.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.083+0000 7f2da559a640 1 -- 172.21.15.161:0/2883249679 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1500499331 0 0) 0x7f2da8158a30 con 0x7f2da80c8ed0 2024-03-28T23:34:49.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.084+0000 7f2da659c640 1 --2- 172.21.15.161:0/2883249679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2da8152890 0x7f2da80c8990 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f2da81597c0 tx=0x7f2da002bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b87e35a169efef9e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:49.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.084+0000 7f2da559a640 1 -- 172.21.15.161:0/2883249679 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da0039070 con 0x7f2da8152890 2024-03-28T23:34:49.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.084+0000 7f2da559a640 1 -- 172.21.15.161:0/2883249679 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2da002cb20 con 0x7f2da8152890 2024-03-28T23:34:49.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.084+0000 7f2da559a640 1 -- 172.21.15.161:0/2883249679 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da0034440 con 0x7f2da8152890 2024-03-28T23:34:49.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.084+0000 7f2dacf81640 1 -- 172.21.15.161:0/2883249679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2da8152890 msgr2=0x7f2da80c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:49.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.084+0000 7f2dacf81640 1 --2- 172.21.15.161:0/2883249679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2da8152890 0x7f2da80c8990 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f2da81597c0 tx=0x7f2da002bd70 comp rx=0 tx=0).stop 2024-03-28T23:34:49.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.084+0000 7f2dacf81640 1 -- 172.21.15.161:0/2883249679 shutdown_connections 2024-03-28T23:34:49.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.084+0000 7f2dacf81640 1 --2- 172.21.15.161:0/2883249679 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2da80c5b20 0x7f2da81553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.084+0000 7f2dacf81640 1 --2- 172.21.15.161:0/2883249679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2da8152890 0x7f2da80c8990 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.084+0000 7f2dacf81640 1 -- 172.21.15.161:0/2883249679 >> 172.21.15.161:0/2883249679 conn(0x7f2da80ac620 msgr2=0x7f2da80aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:49.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.084+0000 7f2dacf81640 1 -- 172.21.15.161:0/2883249679 shutdown_connections 2024-03-28T23:34:49.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.084+0000 7f2dacf81640 1 -- 172.21.15.161:0/2883249679 wait complete. 2024-03-28T23:34:49.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.086+0000 7f2dacf81640 1 Processor -- start 2024-03-28T23:34:49.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.086+0000 7f2dacf81640 1 -- start start 2024-03-28T23:34:49.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.086+0000 7f2dacf81640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2da80c5b20 0x7f2da80c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.086+0000 7f2dacf81640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2da80c8ed0 0x7f2da8149d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.086+0000 7f2dacf81640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2da8152890 0x7f2da81eb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.086+0000 7f2dacf81640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2da8152c70 con 0x7f2da80c5b20 2024-03-28T23:34:49.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.086+0000 7f2dacf81640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2da81592d0 con 0x7f2da80c8ed0 2024-03-28T23:34:49.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.086+0000 7f2dacf81640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2da8152df0 con 0x7f2da8152890 2024-03-28T23:34:49.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.086+0000 7f2da659c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2da80c5b20 0x7f2da80c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.086+0000 7f2da659c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2da80c5b20 0x7f2da80c1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54788/0 (socket says 172.21.15.161:54788) 2024-03-28T23:34:49.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.086+0000 7f2da659c640 1 -- 172.21.15.161:0/816653402 learned_addr learned my addr 172.21.15.161:0/816653402 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:49.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.087+0000 7f2da659c640 1 -- 172.21.15.161:0/816653402 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2da8152890 msgr2=0x7f2da81eb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:49.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.087+0000 7f2da659c640 1 --2- 172.21.15.161:0/816653402 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2da8152890 0x7f2da81eb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.087+0000 7f2da659c640 1 -- 172.21.15.161:0/816653402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2da80c8ed0 msgr2=0x7f2da8149d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:49.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.087+0000 7f2da659c640 1 --2- 172.21.15.161:0/816653402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2da80c8ed0 0x7f2da8149d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.087+0000 7f2da659c640 1 -- 172.21.15.161:0/816653402 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da80bff70 con 0x7f2da80c5b20 2024-03-28T23:34:49.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.087+0000 7f2da659c640 1 --2- 172.21.15.161:0/816653402 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2da80c5b20 0x7f2da80c1b60 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f2da0009e80 tx=0x7f2da0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:49.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.087+0000 7f2d9effd640 1 -- 172.21.15.161:0/816653402 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da0039050 con 0x7f2da80c5b20 2024-03-28T23:34:49.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.087+0000 7f2d9effd640 1 -- 172.21.15.161:0/816653402 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2da0045070 con 0x7f2da80c5b20 2024-03-28T23:34:49.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.087+0000 7f2dacf81640 1 -- 172.21.15.161:0/816653402 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2da80beea0 con 0x7f2da80c5b20 2024-03-28T23:34:49.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.087+0000 7f2d9effd640 1 -- 172.21.15.161:0/816653402 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da003ea50 con 0x7f2da80c5b20 2024-03-28T23:34:49.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.087+0000 7f2dacf81640 1 -- 172.21.15.161:0/816653402 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2da80bf460 con 0x7f2da80c5b20 2024-03-28T23:34:49.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.088+0000 7f2dacf81640 1 -- 172.21.15.161:0/816653402 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2da80c58c0 con 0x7f2da80c5b20 2024-03-28T23:34:49.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.089+0000 7f2d9effd640 1 -- 172.21.15.161:0/816653402 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2da002c9e0 con 0x7f2da80c5b20 2024-03-28T23:34:49.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.089+0000 7f2d9effd640 1 --2- 172.21.15.161:0/816653402 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2d8806b500 0x7f2d8806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.089+0000 7f2d9effd640 1 -- 172.21.15.161:0/816653402 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6101+0+0 (secure 0 0 0) 0x7f2da00acb00 con 0x7f2da80c5b20 2024-03-28T23:34:49.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.089+0000 7f2da5d9b640 1 --2- 172.21.15.161:0/816653402 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2d8806b500 0x7f2d8806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.090+0000 7f2da5d9b640 1 --2- 172.21.15.161:0/816653402 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2d8806b500 0x7f2d8806d9c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f2d900010e0 tx=0x7f2d900077c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:49.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.093+0000 7f2d9effd640 1 -- 172.21.15.161:0/816653402 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2da80c58c0 con 0x7f2da80c5b20 2024-03-28T23:34:49.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.308+0000 7f2dacf81640 1 -- 172.21.15.161:0/816653402 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2da804f4b0 con 0x7f2da80c5b20 2024-03-28T23:34:49.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.309+0000 7f2d9effd640 1 -- 172.21.15.161:0/816653402 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) ==== 95+0+5197 (secure 0 0 0) 0x7f2da004e020 con 0x7f2da80c5b20 2024-03-28T23:34:49.309 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:49.309 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":108,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7318,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3734750343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3734750343},{"type":"v1","addr":"172.21.15.170:6835","nonce":3734750343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/4140438862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":4140438862},{"type":"v1","addr":"172.21.15.170:6834","nonce":4140438862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/878057589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":878057589},{"type":"v1","addr":"172.21.15.161:6835","nonce":878057589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:34:36.796608+0000","modified":"2024-03-28T23:34:45.482535+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7974},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7974":{"gid":7974,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":5,"addr":"172.21.15.161:6837/3432367324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":3432367324},{"type":"v1","addr":"172.21.15.161:6837","nonce":3432367324}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7974,"qdb_cluster":[7974]},"id":9}]} 2024-03-28T23:34:49.309 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 108 2024-03-28T23:34:49.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.311+0000 7f2dacf81640 1 -- 172.21.15.161:0/816653402 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2d8806b500 msgr2=0x7f2d8806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:49.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.311+0000 7f2dacf81640 1 --2- 172.21.15.161:0/816653402 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2d8806b500 0x7f2d8806d9c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f2d900010e0 tx=0x7f2d900077c0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.311+0000 7f2dacf81640 1 -- 172.21.15.161:0/816653402 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2da80c5b20 msgr2=0x7f2da80c1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:49.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.311+0000 7f2dacf81640 1 --2- 172.21.15.161:0/816653402 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2da80c5b20 0x7f2da80c1b60 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f2da0009e80 tx=0x7f2da0004a40 comp rx=0 tx=0).stop 2024-03-28T23:34:49.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.311+0000 7f2dacf81640 1 -- 172.21.15.161:0/816653402 shutdown_connections 2024-03-28T23:34:49.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.311+0000 7f2dacf81640 1 --2- 172.21.15.161:0/816653402 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2d8806b500 0x7f2d8806d9c0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.311+0000 7f2dacf81640 1 --2- 172.21.15.161:0/816653402 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2da8152890 0x7f2da81eb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.311+0000 7f2dacf81640 1 --2- 172.21.15.161:0/816653402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2da80c8ed0 0x7f2da8149d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.311+0000 7f2dacf81640 1 --2- 172.21.15.161:0/816653402 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2da80c5b20 0x7f2da80c1b60 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.311+0000 7f2dacf81640 1 -- 172.21.15.161:0/816653402 >> 172.21.15.161:0/816653402 conn(0x7f2da80ac620 msgr2=0x7f2da81512e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:49.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.311+0000 7f2dacf81640 1 -- 172.21.15.161:0/816653402 shutdown_connections 2024-03-28T23:34:49.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.311+0000 7f2dacf81640 1 -- 172.21.15.161:0/816653402 wait complete. 2024-03-28T23:34:49.724 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:34:36.796608+0000', 'modified': '2024-03-28T23:34:45.482535+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7974}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7974': {'gid': 7974, 'name': 'a', 'rank': 0, 'incarnation': 105, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.161:6837/3432367324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 3432367324}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 3432367324}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7974, 'qdb_cluster': [7974]} 2024-03-28T23:34:49.724 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:34:49.725 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-28T23:34:49.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.828+0000 7fe27dda0640 1 Processor -- start 2024-03-28T23:34:49.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.828+0000 7fe27dda0640 1 -- start start 2024-03-28T23:34:49.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.829+0000 7fe27dda0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe278159370 0x7fe2780cad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.829+0000 7fe27dda0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe278155170 0x7fe278155600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.829+0000 7fe27dda0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2781588d0 con 0x7fe278152880 2024-03-28T23:34:49.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.829+0000 7fe27dda0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe2780cb6c0 con 0x7fe278159370 2024-03-28T23:34:49.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.829+0000 7fe27dda0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe278158ee0 con 0x7fe278155170 2024-03-28T23:34:49.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.829+0000 7fe277fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe278159370 0x7fe2780cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.829+0000 7fe277fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe278159370 0x7fe2780cad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54796/0 (socket says 172.21.15.161:54796) 2024-03-28T23:34:49.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.829+0000 7fe277fff640 1 -- 172.21.15.161:0/2734544764 learned_addr learned my addr 172.21.15.161:0/2734544764 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:49.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.829+0000 7fe27cd9e640 1 --1- 172.21.15.161:0/2734544764 >> v1:172.21.15.161:6789/0 conn(0x7fe278152880 0x7fe2780c5370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50390/0 (socket says 172.21.15.161:50390) 2024-03-28T23:34:49.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.829+0000 7fe27d59f640 1 --2- 172.21.15.161:0/2734544764 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe278155170 0x7fe278155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.829+0000 7fe277fff640 1 -- 172.21.15.161:0/2734544764 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe278155170 msgr2=0x7fe278155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:49.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.829+0000 7fe277fff640 1 --2- 172.21.15.161:0/2734544764 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe278155170 0x7fe278155600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.829+0000 7fe277fff640 1 -- 172.21.15.161:0/2734544764 >> v1:172.21.15.161:6789/0 conn(0x7fe278152880 legacy=0x7fe2780c5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:49.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.829+0000 7fe277fff640 1 -- 172.21.15.161:0/2734544764 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2780cb3e0 con 0x7fe278159370 2024-03-28T23:34:49.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.829+0000 7fe2777fe640 1 -- 172.21.15.161:0/2734544764 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 761183014 0 0) 0x7fe2781588d0 con 0x7fe278152880 2024-03-28T23:34:49.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.829+0000 7fe27d59f640 1 --2- 172.21.15.161:0/2734544764 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe278155170 0x7fe278155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:49.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.830+0000 7fe277fff640 1 --2- 172.21.15.161:0/2734544764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe278159370 0x7fe2780cad40 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fe264009870 tx=0x7fe26402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=854dfc15708c32ff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:49.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.830+0000 7fe2777fe640 1 -- 172.21.15.161:0/2734544764 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe264035020 con 0x7fe278159370 2024-03-28T23:34:49.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.830+0000 7fe2777fe640 1 -- 172.21.15.161:0/2734544764 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe264031640 con 0x7fe278159370 2024-03-28T23:34:49.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.830+0000 7fe2777fe640 1 -- 172.21.15.161:0/2734544764 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26402d040 con 0x7fe278159370 2024-03-28T23:34:49.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.830+0000 7fe27dda0640 1 -- 172.21.15.161:0/2734544764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe278159370 msgr2=0x7fe2780cad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:49.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.830+0000 7fe27dda0640 1 --2- 172.21.15.161:0/2734544764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe278159370 0x7fe2780cad40 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fe264009870 tx=0x7fe26402b8f0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.830+0000 7fe27dda0640 1 -- 172.21.15.161:0/2734544764 shutdown_connections 2024-03-28T23:34:49.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.830+0000 7fe27dda0640 1 --2- 172.21.15.161:0/2734544764 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe278155170 0x7fe278155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.830+0000 7fe27dda0640 1 --2- 172.21.15.161:0/2734544764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe278159370 0x7fe2780cad40 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.830+0000 7fe27dda0640 1 -- 172.21.15.161:0/2734544764 >> 172.21.15.161:0/2734544764 conn(0x7fe278072a60 msgr2=0x7fe278072e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:49.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.830+0000 7fe27dda0640 1 -- 172.21.15.161:0/2734544764 shutdown_connections 2024-03-28T23:34:49.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.830+0000 7fe27dda0640 1 -- 172.21.15.161:0/2734544764 wait complete. 2024-03-28T23:34:49.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.832+0000 7fe27dda0640 1 Processor -- start 2024-03-28T23:34:49.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.832+0000 7fe27dda0640 1 -- start start 2024-03-28T23:34:49.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.833+0000 7fe27dda0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe278152880 0x7fe2780c4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.833+0000 7fe27dda0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe278155170 0x7fe2780c7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.833+0000 7fe27dda0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe278159370 0x7fe2781edfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.833+0000 7fe27dda0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe2780c9300 con 0x7fe278155170 2024-03-28T23:34:49.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.833+0000 7fe27dda0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe278152c60 con 0x7fe278152880 2024-03-28T23:34:49.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.833+0000 7fe27dda0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe2780c5a80 con 0x7fe278159370 2024-03-28T23:34:49.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.834+0000 7fe277fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe278155170 0x7fe2780c7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.834+0000 7fe277fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe278155170 0x7fe2780c7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54806/0 (socket says 172.21.15.161:54806) 2024-03-28T23:34:49.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.834+0000 7fe277fff640 1 -- 172.21.15.161:0/1938979914 learned_addr learned my addr 172.21.15.161:0/1938979914 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:49.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.834+0000 7fe27cd9e640 1 --2- 172.21.15.161:0/1938979914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe278152880 0x7fe2780c4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.834+0000 7fe27d59f640 1 --2- 172.21.15.161:0/1938979914 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe278159370 0x7fe2781edfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.834+0000 7fe277fff640 1 -- 172.21.15.161:0/1938979914 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe278159370 msgr2=0x7fe2781edfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:49.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.834+0000 7fe277fff640 1 --2- 172.21.15.161:0/1938979914 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe278159370 0x7fe2781edfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.834+0000 7fe277fff640 1 -- 172.21.15.161:0/1938979914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe278152880 msgr2=0x7fe2780c4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:49.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.834+0000 7fe277fff640 1 --2- 172.21.15.161:0/1938979914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe278152880 0x7fe2780c4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.834+0000 7fe277fff640 1 -- 172.21.15.161:0/1938979914 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2781edae0 con 0x7fe278155170 2024-03-28T23:34:49.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.834+0000 7fe27cd9e640 1 --2- 172.21.15.161:0/1938979914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe278152880 0x7fe2780c4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:49.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.834+0000 7fe27d59f640 1 --2- 172.21.15.161:0/1938979914 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe278159370 0x7fe2781edfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:49.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.834+0000 7fe277fff640 1 --2- 172.21.15.161:0/1938979914 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe278155170 0x7fe2780c7670 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fe264000c00 tx=0x7fe2640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:49.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.834+0000 7fe2757fa640 1 -- 172.21.15.161:0/1938979914 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe264040070 con 0x7fe278155170 2024-03-28T23:34:49.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.834+0000 7fe2757fa640 1 -- 172.21.15.161:0/1938979914 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe264039c10 con 0x7fe278155170 2024-03-28T23:34:49.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.834+0000 7fe2757fa640 1 -- 172.21.15.161:0/1938979914 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe264035040 con 0x7fe278155170 2024-03-28T23:34:49.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.834+0000 7fe27dda0640 1 -- 172.21.15.161:0/1938979914 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2781eaa40 con 0x7fe278155170 2024-03-28T23:34:49.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.834+0000 7fe27dda0640 1 -- 172.21.15.161:0/1938979914 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2781eb000 con 0x7fe278155170 2024-03-28T23:34:49.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.836+0000 7fe2757fa640 1 -- 172.21.15.161:0/1938979914 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe26404b060 con 0x7fe278155170 2024-03-28T23:34:49.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.836+0000 7fe2757fa640 1 --2- 172.21.15.161:0/1938979914 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe25c06b4d0 0x7fe25c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.836+0000 7fe2757fa640 1 -- 172.21.15.161:0/1938979914 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6101+0+0 (secure 0 0 0) 0x7fe2640abe30 con 0x7fe278155170 2024-03-28T23:34:49.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.836+0000 7fe27cd9e640 1 --2- 172.21.15.161:0/1938979914 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe25c06b4d0 0x7fe25c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.837+0000 7fe27cd9e640 1 --2- 172.21.15.161:0/1938979914 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe25c06b4d0 0x7fe25c06d990 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe26c000f60 tx=0x7fe26c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:49.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.837+0000 7fe281be8740 1 Processor -- start 2024-03-28T23:34:49.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.837+0000 7fe281be8740 1 -- start start 2024-03-28T23:34:49.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.838+0000 7fe281be8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ca8787bf90 0x55ca87879a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.838+0000 7fe281be8740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ca8787c370 0x55ca87879fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.838+0000 7fe281be8740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787a4f0 0x55ca878766c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.838+0000 7fe281be8740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55ca877dfaf0 con 0x55ca8787a4f0 2024-03-28T23:34:49.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.838+0000 7fe281be8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55ca877889d0 con 0x55ca8787bf90 2024-03-28T23:34:49.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.838+0000 7fe281be8740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55ca877dbb90 con 0x55ca8787c370 2024-03-28T23:34:49.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.838+0000 7fe27d59f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787a4f0 0x55ca878766c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.838+0000 7fe27d59f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787a4f0 0x55ca878766c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54808/0 (socket says 172.21.15.161:54808) 2024-03-28T23:34:49.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.838+0000 7fe27d59f640 1 -- 172.21.15.161:0/3260751980 learned_addr learned my addr 172.21.15.161:0/3260751980 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:49.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.838+0000 7fe27cd9e640 1 --2- 172.21.15.161:0/3260751980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ca8787bf90 0x55ca87879a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.838+0000 7fe277fff640 1 --2- 172.21.15.161:0/3260751980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ca8787c370 0x55ca87879fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.838+0000 7fe27d59f640 1 -- 172.21.15.161:0/3260751980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ca8787c370 msgr2=0x55ca87879fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:49.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.838+0000 7fe27d59f640 1 --2- 172.21.15.161:0/3260751980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ca8787c370 0x55ca87879fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.838+0000 7fe27d59f640 1 -- 172.21.15.161:0/3260751980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ca8787bf90 msgr2=0x55ca87879a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:49.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.838+0000 7fe27d59f640 1 --2- 172.21.15.161:0/3260751980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ca8787bf90 0x55ca87879a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.838+0000 7fe27d59f640 1 -- 172.21.15.161:0/3260751980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca874f42d0 con 0x55ca8787a4f0 2024-03-28T23:34:49.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.838+0000 7fe277fff640 1 --2- 172.21.15.161:0/3260751980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ca8787c370 0x55ca87879fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:49.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.838+0000 7fe27cd9e640 1 --2- 172.21.15.161:0/3260751980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ca8787bf90 0x55ca87879a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:49.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.838+0000 7fe27d59f640 1 --2- 172.21.15.161:0/3260751980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787a4f0 0x55ca878766c0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fe270007f80 tx=0x7fe27000d800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc06a71d8173d935 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:49.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.839+0000 7fe2627fc640 1 -- 172.21.15.161:0/3260751980 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27000f920 con 0x55ca8787a4f0 2024-03-28T23:34:49.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.839+0000 7fe2627fc640 1 -- 172.21.15.161:0/3260751980 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe27000fac0 con 0x55ca8787a4f0 2024-03-28T23:34:49.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.839+0000 7fe2627fc640 1 -- 172.21.15.161:0/3260751980 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe270018480 con 0x55ca8787a4f0 2024-03-28T23:34:49.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.839+0000 7fe281be8740 1 -- 172.21.15.161:0/3260751980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787a4f0 msgr2=0x55ca878766c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:49.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.839+0000 7fe281be8740 1 --2- 172.21.15.161:0/3260751980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787a4f0 0x55ca878766c0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fe270007f80 tx=0x7fe27000d800 comp rx=0 tx=0).stop 2024-03-28T23:34:49.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.839+0000 7fe281be8740 1 -- 172.21.15.161:0/3260751980 shutdown_connections 2024-03-28T23:34:49.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.839+0000 7fe281be8740 1 --2- 172.21.15.161:0/3260751980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787a4f0 0x55ca878766c0 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.839+0000 7fe281be8740 1 --2- 172.21.15.161:0/3260751980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ca8787c370 0x55ca87879fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.839+0000 7fe281be8740 1 --2- 172.21.15.161:0/3260751980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ca8787bf90 0x55ca87879a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.839+0000 7fe281be8740 1 -- 172.21.15.161:0/3260751980 >> 172.21.15.161:0/3260751980 conn(0x55ca87878810 msgr2=0x55ca8789d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:49.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.839+0000 7fe281be8740 1 -- 172.21.15.161:0/3260751980 shutdown_connections 2024-03-28T23:34:49.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.839+0000 7fe281be8740 1 -- 172.21.15.161:0/3260751980 wait complete. 2024-03-28T23:34:49.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.840+0000 7fe281be8740 1 -- start start 2024-03-28T23:34:49.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.840+0000 7fe281be8740 1 Processor -- start 2024-03-28T23:34:49.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.841+0000 7fe281be8740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787bf90 0x55ca878bb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.841+0000 7fe281be8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ca8787c370 0x55ca878cba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.841+0000 7fe281be8740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ca878994c0 0x55ca8789e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.841+0000 7fe281be8740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55ca8789e270 con 0x55ca8787bf90 2024-03-28T23:34:49.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.841+0000 7fe281be8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55ca8779bf70 con 0x55ca8787c370 2024-03-28T23:34:49.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.841+0000 7fe281be8740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55ca8789dcb0 con 0x55ca878994c0 2024-03-28T23:34:49.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.841+0000 7fe281be8740 10 client.0 _ll_register_callbacks cb 0x55ca874f3ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:34:49.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.841+0000 7fe27cd9e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787bf90 0x55ca878bb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.841+0000 7fe27cd9e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787bf90 0x55ca878bb6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54818/0 (socket says 172.21.15.161:54818) 2024-03-28T23:34:49.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.841+0000 7fe27cd9e640 1 -- 172.21.15.161:0/240330721 learned_addr learned my addr 172.21.15.161:0/240330721 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:49.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.841+0000 7fe277fff640 1 --2- 172.21.15.161:0/240330721 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ca8787c370 0x55ca878cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.841+0000 7fe27d59f640 1 --2- 172.21.15.161:0/240330721 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ca878994c0 0x55ca8789e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.841+0000 7fe27cd9e640 1 -- 172.21.15.161:0/240330721 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ca878994c0 msgr2=0x55ca8789e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:49.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.841+0000 7fe27cd9e640 1 --2- 172.21.15.161:0/240330721 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ca878994c0 0x55ca8789e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.841+0000 7fe27cd9e640 1 -- 172.21.15.161:0/240330721 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ca8787c370 msgr2=0x55ca878cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:49.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.841+0000 7fe27cd9e640 1 --2- 172.21.15.161:0/240330721 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ca8787c370 0x55ca878cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.841+0000 7fe27cd9e640 1 -- 172.21.15.161:0/240330721 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca875be6b0 con 0x55ca8787bf90 2024-03-28T23:34:49.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.841+0000 7fe277fff640 1 --2- 172.21.15.161:0/240330721 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ca8787c370 0x55ca878cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:49.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.841+0000 7fe27cd9e640 1 --2- 172.21.15.161:0/240330721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787bf90 0x55ca878bb6b0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fe26c005b50 tx=0x7fe26c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=908ffee47910fb61 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:49.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.841+0000 7fe2577fe640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:34:49.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.841+0000 7fe2577fe640 1 -- 172.21.15.161:0/240330721 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26c026060 con 0x55ca8787bf90 2024-03-28T23:34:49.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.841+0000 7fe2577fe640 1 -- 172.21.15.161:0/240330721 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 6101+0+0 (secure 0 0 0) 0x7fe26c021c20 con 0x55ca8787bf90 2024-03-28T23:34:49.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.841+0000 7fe281be8740 1 -- 172.21.15.161:0/240330721 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe27000fac0 con 0x55ca8787bf90 2024-03-28T23:34:49.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.842+0000 7fe2577fe640 1 -- 172.21.15.161:0/240330721 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7fe26c021e20 con 0x55ca8787bf90 2024-03-28T23:34:49.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.842+0000 7fe281be8740 10 client.8181 fetch_fsmap learned FSMap version 108 2024-03-28T23:34:49.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.842+0000 7fe281be8740 1 -- 172.21.15.161:0/240330721 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=2+}) -- 0x55ca878a2870 con 0x55ca8787bf90 2024-03-28T23:34:49.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.842+0000 7fe2577fe640 1 -- 172.21.15.161:0/240330721 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe27000fac0 con 0x55ca8787bf90 2024-03-28T23:34:49.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.842+0000 7fe2577fe640 1 -- 172.21.15.161:0/240330721 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 108) ==== 3494+0+0 (secure 0 0 0) 0x7fe26c012410 con 0x55ca8787bf90 2024-03-28T23:34:49.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.842+0000 7fe281be8740 10 client.8181 fetch_fsmap finished waiting for FSMap version 108 2024-03-28T23:34:49.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.842+0000 7fe281be8740 10 client.8181 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T23:34:49.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.842+0000 7fe281be8740 20 client.8181 populate_metadata read hostname 'smithi161' 2024-03-28T23:34:49.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.842+0000 7fe281be8740 1 --2- 172.21.15.161:0/240330721 >> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] conn(0x55ca8789f200 0x55ca878d6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.842+0000 7fe281be8740 4 client.8181 mds_command: new command op to 7974 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:34:49.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.842+0000 7fe281be8740 1 -- 172.21.15.161:0/240330721 --> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca877ed030 con 0x55ca8789f200 2024-03-28T23:34:49.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.842+0000 7fe277fff640 1 --2- 172.21.15.161:0/240330721 >> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] conn(0x55ca8789f200 0x55ca878d6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.842+0000 7fe2577fe640 10 client.8181 ms_handle_connect on v2:172.21.15.161:6836/3432367324 2024-03-28T23:34:49.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.842+0000 7fe277fff640 1 --2- 172.21.15.161:0/240330721 >> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] conn(0x55ca8789f200 0x55ca878d6ef0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f607029cb4848c1 server_cookie=9f8770fa6bf94717 in_seq=0 out_seq=0 2024-03-28T23:34:49.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.843+0000 7fe2577fe640 1 -- 172.21.15.161:0/240330721 <== mds.0 v2:172.21.15.161:6836/3432367324 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ca877ed030 con 0x55ca8789f200 2024-03-28T23:34:49.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.843+0000 7fe2577fe640 10 client.8181 handle_command_reply: tid=0 2024-03-28T23:34:49.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.843+0000 7fe281be8740 1 client.8181 shutdown 2024-03-28T23:34:49.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.843+0000 7fe281be8740 1 -- 172.21.15.161:0/240330721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787bf90 msgr2=0x55ca878bb6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:49.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.843+0000 7fe281be8740 1 --2- 172.21.15.161:0/240330721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787bf90 0x55ca878bb6b0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fe26c005b50 tx=0x7fe26c000ed0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.843+0000 7fe281be8740 1 -- 172.21.15.161:0/240330721 shutdown_connections 2024-03-28T23:34:49.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.843+0000 7fe281be8740 1 --2- 172.21.15.161:0/240330721 >> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] conn(0x55ca8789f200 0x55ca878d6ef0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.843+0000 7fe281be8740 1 --2- 172.21.15.161:0/240330721 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ca878994c0 0x55ca8789e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.843+0000 7fe281be8740 1 --2- 172.21.15.161:0/240330721 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ca8787c370 0x55ca878cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.843+0000 7fe2577fe640 0 client.8181 ms_handle_reset on v2:172.21.15.161:6836/3432367324 2024-03-28T23:34:49.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.843+0000 7fe281be8740 1 --2- 172.21.15.161:0/240330721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787bf90 0x55ca878bb6b0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.843+0000 7fe281be8740 1 -- 172.21.15.161:0/240330721 >> 172.21.15.161:0/240330721 conn(0x55ca8787a4f0 msgr2=0x55ca87875550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:49.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.843+0000 7fe281be8740 1 -- 172.21.15.161:0/240330721 shutdown_connections 2024-03-28T23:34:49.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.843+0000 7fe281be8740 1 -- 172.21.15.161:0/240330721 wait complete. 2024-03-28T23:34:49.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.843+0000 7fe281be8740 20 client.8181 trim_cache size 0 max 16384 2024-03-28T23:34:49.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.861+0000 7fe281be8740 1 Processor -- start 2024-03-28T23:34:49.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.861+0000 7fe281be8740 1 -- start start 2024-03-28T23:34:49.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.862+0000 7fe281be8740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ca8787bf90 0x55ca878dc890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.862+0000 7fe281be8740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787c370 0x55ca878989e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.862+0000 7fe281be8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ca878994c0 0x55ca878799e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.862+0000 7fe281be8740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe2781f7c80 con 0x55ca8787c370 2024-03-28T23:34:49.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.862+0000 7fe281be8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe2781f6ca0 con 0x55ca878994c0 2024-03-28T23:34:49.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.862+0000 7fe281be8740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe2781f6080 con 0x55ca8787bf90 2024-03-28T23:34:49.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.862+0000 7fe277fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787c370 0x55ca878989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.862+0000 7fe277fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787c370 0x55ca878989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54832/0 (socket says 172.21.15.161:54832) 2024-03-28T23:34:49.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.862+0000 7fe277fff640 1 -- 172.21.15.161:0/159175223 learned_addr learned my addr 172.21.15.161:0/159175223 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:49.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.862+0000 7fe277fff640 1 -- 172.21.15.161:0/159175223 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ca8787bf90 msgr2=0x55ca878dc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:49.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.862+0000 7fe277fff640 1 --2- 172.21.15.161:0/159175223 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ca8787bf90 0x55ca878dc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.862+0000 7fe277fff640 1 -- 172.21.15.161:0/159175223 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ca878994c0 msgr2=0x55ca878799e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:49.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.862+0000 7fe277fff640 1 --2- 172.21.15.161:0/159175223 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ca878994c0 0x55ca878799e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.862+0000 7fe277fff640 1 -- 172.21.15.161:0/159175223 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca878992d0 con 0x55ca8787c370 2024-03-28T23:34:49.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.862+0000 7fe277fff640 1 --2- 172.21.15.161:0/159175223 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787c370 0x55ca878989e0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fe264009300 tx=0x7fe26407d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d5428a68b23a233 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:49.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.862+0000 7fe261ffb640 1 -- 172.21.15.161:0/159175223 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2640a7070 con 0x55ca8787c370 2024-03-28T23:34:49.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.862+0000 7fe261ffb640 1 -- 172.21.15.161:0/159175223 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2640787b0 con 0x55ca8787c370 2024-03-28T23:34:49.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.862+0000 7fe261ffb640 1 -- 172.21.15.161:0/159175223 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe264082730 con 0x55ca8787c370 2024-03-28T23:34:49.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.863+0000 7fe281be8740 1 -- 172.21.15.161:0/159175223 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787c370 msgr2=0x55ca878989e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:49.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.863+0000 7fe281be8740 1 --2- 172.21.15.161:0/159175223 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787c370 0x55ca878989e0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fe264009300 tx=0x7fe26407d040 comp rx=0 tx=0).stop 2024-03-28T23:34:49.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.863+0000 7fe281be8740 1 -- 172.21.15.161:0/159175223 shutdown_connections 2024-03-28T23:34:49.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.863+0000 7fe281be8740 1 --2- 172.21.15.161:0/159175223 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787c370 0x55ca878989e0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.863+0000 7fe281be8740 1 --2- 172.21.15.161:0/159175223 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ca878994c0 0x55ca878799e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.863+0000 7fe281be8740 1 --2- 172.21.15.161:0/159175223 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ca8787bf90 0x55ca878dc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.863+0000 7fe281be8740 1 -- 172.21.15.161:0/159175223 >> 172.21.15.161:0/159175223 conn(0x55ca8787a4f0 msgr2=0x55ca878c8390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:49.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.863+0000 7fe281be8740 1 -- 172.21.15.161:0/159175223 shutdown_connections 2024-03-28T23:34:49.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.863+0000 7fe281be8740 1 -- 172.21.15.161:0/159175223 wait complete. 2024-03-28T23:34:49.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.863+0000 7fe281be8740 1 -- start start 2024-03-28T23:34:49.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.864+0000 7fe281be8740 1 Processor -- start 2024-03-28T23:34:49.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.864+0000 7fe281be8740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787bf90 0x55ca878db310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.864+0000 7fe281be8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ca8787c370 0x55ca878d1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.864+0000 7fe281be8740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ca878994c0 0x55ca8792f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.864+0000 7fe281be8740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55ca878bc210 con 0x55ca8787bf90 2024-03-28T23:34:49.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.864+0000 7fe281be8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe2781f6650 con 0x55ca8787c370 2024-03-28T23:34:49.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.864+0000 7fe281be8740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55ca8789eb80 con 0x55ca878994c0 2024-03-28T23:34:49.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.864+0000 7fe281be8740 10 client.0 _ll_register_callbacks cb 0x7fe26c0013d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:34:49.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.864+0000 7fe27cd9e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787bf90 0x55ca878db310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.864+0000 7fe27cd9e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787bf90 0x55ca878db310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54846/0 (socket says 172.21.15.161:54846) 2024-03-28T23:34:49.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.864+0000 7fe27cd9e640 1 -- 172.21.15.161:0/3772358914 learned_addr learned my addr 172.21.15.161:0/3772358914 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:49.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.865+0000 7fe27d59f640 1 --2- 172.21.15.161:0/3772358914 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ca878994c0 0x55ca8792f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.865+0000 7fe277fff640 1 --2- 172.21.15.161:0/3772358914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ca8787c370 0x55ca878d1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.865+0000 7fe27cd9e640 1 -- 172.21.15.161:0/3772358914 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ca878994c0 msgr2=0x55ca8792f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:49.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.865+0000 7fe27cd9e640 1 --2- 172.21.15.161:0/3772358914 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ca878994c0 0x55ca8792f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.865+0000 7fe27cd9e640 1 -- 172.21.15.161:0/3772358914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ca8787c370 msgr2=0x55ca878d1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:49.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.865+0000 7fe27cd9e640 1 --2- 172.21.15.161:0/3772358914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ca8787c370 0x55ca878d1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.865+0000 7fe27cd9e640 1 -- 172.21.15.161:0/3772358914 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca87935b80 con 0x55ca8787bf90 2024-03-28T23:34:49.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.865+0000 7fe277fff640 1 --2- 172.21.15.161:0/3772358914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ca8787c370 0x55ca878d1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:49.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.865+0000 7fe27cd9e640 1 --2- 172.21.15.161:0/3772358914 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787bf90 0x55ca878db310 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fe26c005fd0 tx=0x7fe26c001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48a7f2d771d1c9eb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:49.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.865+0000 7fe2577fe640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:34:49.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.865+0000 7fe2577fe640 1 -- 172.21.15.161:0/3772358914 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26c00dba0 con 0x55ca8787bf90 2024-03-28T23:34:49.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.865+0000 7fe2577fe640 1 -- 172.21.15.161:0/3772358914 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 6101+0+0 (secure 0 0 0) 0x7fe26c01d070 con 0x55ca8787bf90 2024-03-28T23:34:49.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.865+0000 7fe281be8740 1 -- 172.21.15.161:0/3772358914 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2640787b0 con 0x55ca8787bf90 2024-03-28T23:34:49.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.865+0000 7fe2577fe640 1 -- 172.21.15.161:0/3772358914 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7fe26c03d050 con 0x55ca8787bf90 2024-03-28T23:34:49.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.865+0000 7fe281be8740 10 client.8187 fetch_fsmap learned FSMap version 108 2024-03-28T23:34:49.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.865+0000 7fe281be8740 1 -- 172.21.15.161:0/3772358914 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=2+}) -- 0x55ca878dc930 con 0x55ca8787bf90 2024-03-28T23:34:49.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.865+0000 7fe2577fe640 1 -- 172.21.15.161:0/3772358914 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2640787b0 con 0x55ca8787bf90 2024-03-28T23:34:49.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.865+0000 7fe2577fe640 1 -- 172.21.15.161:0/3772358914 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 108) ==== 3494+0+0 (secure 0 0 0) 0x7fe26c040020 con 0x55ca8787bf90 2024-03-28T23:34:49.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.865+0000 7fe281be8740 10 client.8187 fetch_fsmap finished waiting for FSMap version 108 2024-03-28T23:34:49.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.866+0000 7fe281be8740 10 client.8187 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T23:34:49.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.866+0000 7fe281be8740 20 client.8187 populate_metadata read hostname 'smithi161' 2024-03-28T23:34:49.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.866+0000 7fe281be8740 1 --2- 172.21.15.161:0/3772358914 >> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] conn(0x55ca878c4820 0x55ca878c4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.866+0000 7fe281be8740 4 client.8187 mds_command: new command op to 7974 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:34:49.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.866+0000 7fe281be8740 1 -- 172.21.15.161:0/3772358914 --> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] -- command(tid 0: {"prefix": "status"}) -- 0x55ca878a3aa0 con 0x55ca878c4820 2024-03-28T23:34:49.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.866+0000 7fe277fff640 1 --2- 172.21.15.161:0/3772358914 >> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] conn(0x55ca878c4820 0x55ca878c4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.866+0000 7fe2577fe640 10 client.8187 ms_handle_connect on v2:172.21.15.161:6836/3432367324 2024-03-28T23:34:49.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.866+0000 7fe277fff640 1 --2- 172.21.15.161:0/3772358914 >> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] conn(0x55ca878c4820 0x55ca878c4c00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e06b67c6552a0991 server_cookie=37d5ef1c1d332eaa in_seq=0 out_seq=0 2024-03-28T23:34:49.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.866+0000 7fe2577fe640 1 -- 172.21.15.161:0/3772358914 <== mds.0 v2:172.21.15.161:6836/3432367324 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55ca878a3aa0 con 0x55ca878c4820 2024-03-28T23:34:49.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.866+0000 7fe2577fe640 10 client.8187 handle_command_reply: tid=0 2024-03-28T23:34:49.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.866+0000 7fe281be8740 1 client.8187 shutdown 2024-03-28T23:34:49.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.866+0000 7fe281be8740 1 -- 172.21.15.161:0/3772358914 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787bf90 msgr2=0x55ca878db310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:49.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.866+0000 7fe281be8740 1 --2- 172.21.15.161:0/3772358914 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787bf90 0x55ca878db310 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fe26c005fd0 tx=0x7fe26c001080 comp rx=0 tx=0).stop 2024-03-28T23:34:49.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.866+0000 7fe281be8740 1 -- 172.21.15.161:0/3772358914 shutdown_connections 2024-03-28T23:34:49.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.866+0000 7fe281be8740 1 --2- 172.21.15.161:0/3772358914 >> [v2:172.21.15.161:6836/3432367324,v1:172.21.15.161:6837/3432367324] conn(0x55ca878c4820 0x55ca878c4c00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.866+0000 7fe281be8740 1 --2- 172.21.15.161:0/3772358914 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ca878994c0 0x55ca8792f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.866+0000 7fe281be8740 1 --2- 172.21.15.161:0/3772358914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ca8787c370 0x55ca878d1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.866+0000 7fe2577fe640 0 client.8187 ms_handle_reset on v2:172.21.15.161:6836/3432367324 2024-03-28T23:34:49.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.866+0000 7fe281be8740 1 --2- 172.21.15.161:0/3772358914 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ca8787bf90 0x55ca878db310 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.866+0000 7fe281be8740 1 -- 172.21.15.161:0/3772358914 >> 172.21.15.161:0/3772358914 conn(0x55ca8787a4f0 msgr2=0x55ca878ba470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:49.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.866+0000 7fe281be8740 1 -- 172.21.15.161:0/3772358914 shutdown_connections 2024-03-28T23:34:49.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.866+0000 7fe281be8740 1 -- 172.21.15.161:0/3772358914 wait complete. 2024-03-28T23:34:49.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.867+0000 7fe281be8740 20 client.8187 trim_cache size 0 max 16384 2024-03-28T23:34:49.873 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:34:49.873 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:34:49.873 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:34:49.873 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 7974, 2024-03-28T23:34:49.873 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:34:49.873 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:34:49.873 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:34:49.873 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 12.420659504, 2024-03-28T23:34:49.874 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 108, 2024-03-28T23:34:49.874 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 122, 2024-03-28T23:34:49.874 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 122, 2024-03-28T23:34:49.874 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 17.708914599 2024-03-28T23:34:49.874 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:34:49.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.867+0000 7fe27dda0640 1 -- 172.21.15.161:0/1938979914 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe25c06b4d0 msgr2=0x7fe25c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:49.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.867+0000 7fe27dda0640 1 --2- 172.21.15.161:0/1938979914 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe25c06b4d0 0x7fe25c06d990 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe26c000f60 tx=0x7fe26c009040 comp rx=0 tx=0).stop 2024-03-28T23:34:49.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.867+0000 7fe27dda0640 1 -- 172.21.15.161:0/1938979914 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe278155170 msgr2=0x7fe2780c7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:49.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.867+0000 7fe27dda0640 1 --2- 172.21.15.161:0/1938979914 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe278155170 0x7fe2780c7670 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fe264000c00 tx=0x7fe2640046c0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.867+0000 7fe27dda0640 1 -- 172.21.15.161:0/1938979914 shutdown_connections 2024-03-28T23:34:49.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.867+0000 7fe27dda0640 1 --2- 172.21.15.161:0/1938979914 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe25c06b4d0 0x7fe25c06d990 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.867+0000 7fe27dda0640 1 --2- 172.21.15.161:0/1938979914 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe278155170 0x7fe2780c7670 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.867+0000 7fe27dda0640 1 --2- 172.21.15.161:0/1938979914 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe278159370 0x7fe2781edfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.867+0000 7fe27dda0640 1 --2- 172.21.15.161:0/1938979914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe278152880 0x7fe2780c4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.867+0000 7fe27dda0640 1 -- 172.21.15.161:0/1938979914 >> 172.21.15.161:0/1938979914 conn(0x7fe278072a60 msgr2=0x7fe278151af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:49.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.867+0000 7fe27dda0640 1 -- 172.21.15.161:0/1938979914 shutdown_connections 2024-03-28T23:34:49.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.867+0000 7fe27dda0640 1 -- 172.21.15.161:0/1938979914 wait complete. 2024-03-28T23:34:49.879 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:34:36.796608+0000", "damaged": [], "data_pools": [ 20 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7974": { "addr": "172.21.15.161:6837/3432367324", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6836", "nonce": 3432367324, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 3432367324, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7974, "incarnation": 105, "join_fscid": 9, "name": "a", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 19, "modified": "2024-03-28T23:34:45.482535+0000", "qdb_cluster": [ 7974 ], "qdb_leader": 7974, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7974 } } } ], "standbys": [ { "addr": "172.21.15.170:6835/3734750343", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 3734750343, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 3734750343, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 103, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7318, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.170:6834/4140438862", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 4140438862, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 4140438862, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 103, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7321, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6835/878057589", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 878057589, "type": "v2" }, { "addr": "172.21.15.161:6835", "nonce": 878057589, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 103, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7965, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:34:49.880 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': 9, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-28T23:34:49.880 INFO:tasks.cephfs.test_failover:current = [{'gid': 7974, 'name': 'a', 'rank': 0, 'incarnation': 105, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.161:6837/3432367324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 3432367324}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 3432367324}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 7318, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/3734750343', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 3734750343}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 3734750343}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 103}, {'gid': 7965, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6835/878057589', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 878057589}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 878057589}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 103}, {'gid': 7321, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6834/4140438862', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 4140438862}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 4140438862}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 103}] 2024-03-28T23:34:49.880 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-28T23:34:49.880 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail 9:0 2024-03-28T23:34:49.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.984+0000 7f7c81cc0640 1 Processor -- start 2024-03-28T23:34:49.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.984+0000 7f7c81cc0640 1 -- start start 2024-03-28T23:34:49.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c81cc0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7c7c0c5410 0x7f7c7c0c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c81cc0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c1553a0 0x7f7c7c155850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c81cc0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c7c158a30 con 0x7f7c7c152890 2024-03-28T23:34:49.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c81cc0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f7c7c0c8480 con 0x7f7c7c0c5410 2024-03-28T23:34:49.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c81cc0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7c7c0c5d90 con 0x7f7c7c1553a0 2024-03-28T23:34:49.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c7b7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f7c7c152890 0x7f7c7c0c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50404/0 (socket says 172.21.15.161:50404) 2024-03-28T23:34:49.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c7b7fe640 1 -- 172.21.15.161:0/783545856 learned_addr learned my addr 172.21.15.161:0/783545856 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:49.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c7bfff640 1 --2- 172.21.15.161:0/783545856 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c1553a0 0x7f7c7c155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c7affd640 1 --2- 172.21.15.161:0/783545856 >> v2:172.21.15.170:3300/0 conn(0x7f7c7c0c5410 0x7f7c7c0c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c7a7fc640 1 -- 172.21.15.161:0/783545856 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2941817211 0 0) 0x7f7c7c158a30 con 0x7f7c7c152890 2024-03-28T23:34:49.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c7a7fc640 1 -- 172.21.15.161:0/783545856 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c64003520 con 0x7f7c7c152890 2024-03-28T23:34:49.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c7a7fc640 1 -- 172.21.15.161:0/783545856 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2397709560 0 0) 0x7f7c64003520 con 0x7f7c7c152890 2024-03-28T23:34:49.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c7a7fc640 1 -- 172.21.15.161:0/783545856 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c1553a0 msgr2=0x7f7c7c155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:49.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c7a7fc640 1 --2- 172.21.15.161:0/783545856 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c1553a0 0x7f7c7c155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c7a7fc640 1 -- 172.21.15.161:0/783545856 >> v2:172.21.15.170:3300/0 conn(0x7f7c7c0c5410 msgr2=0x7f7c7c0c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:49.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c7a7fc640 1 --2- 172.21.15.161:0/783545856 >> v2:172.21.15.170:3300/0 conn(0x7f7c7c0c5410 0x7f7c7c0c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c7a7fc640 1 -- 172.21.15.161:0/783545856 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c7c158d10 con 0x7f7c7c152890 2024-03-28T23:34:49.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c7a7fc640 1 -- 172.21.15.161:0/783545856 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f7c70001670 con 0x7f7c7c152890 2024-03-28T23:34:49.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c7bfff640 1 --2- 172.21.15.161:0/783545856 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c1553a0 0x7f7c7c155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:34:49.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c7a7fc640 1 -- 172.21.15.161:0/783545856 >> v1:172.21.15.161:6789/0 conn(0x7f7c7c152890 legacy=0x7f7c7c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:49.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c7a7fc640 1 --2- 172.21.15.161:0/783545856 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c64003a90 0x7f7c64003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c7a7fc640 1 --2- 172.21.15.161:0/783545856 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c1553a0 0x7f7c64004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c7a7fc640 1 --2- 172.21.15.161:0/783545856 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c64004d40 0x7f7c640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c7a7fc640 1 -- 172.21.15.161:0/783545856 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7c0c5d90 con 0x7f7c64003a90 2024-03-28T23:34:49.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c7a7fc640 1 -- 172.21.15.161:0/783545856 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6400b9f0 con 0x7f7c64004d40 2024-03-28T23:34:49.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.985+0000 7f7c7a7fc640 1 -- 172.21.15.161:0/783545856 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7c6400bbf0 con 0x7f7c7c1553a0 2024-03-28T23:34:49.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.986+0000 7f7c7b7fe640 1 --2- 172.21.15.161:0/783545856 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c64003a90 0x7f7c64003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.986+0000 7f7c7affd640 1 --2- 172.21.15.161:0/783545856 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c1553a0 0x7f7c64004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.986+0000 7f7c7bfff640 1 --2- 172.21.15.161:0/783545856 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c64004d40 0x7f7c640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.986+0000 7f7c7b7fe640 1 -- 172.21.15.161:0/783545856 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c1553a0 msgr2=0x7f7c64004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:49.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.986+0000 7f7c7b7fe640 1 --2- 172.21.15.161:0/783545856 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c1553a0 0x7f7c64004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.986+0000 7f7c7b7fe640 1 -- 172.21.15.161:0/783545856 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c64004d40 msgr2=0x7f7c640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:49.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.986+0000 7f7c7b7fe640 1 --2- 172.21.15.161:0/783545856 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c64004d40 0x7f7c640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.986+0000 7f7c7b7fe640 1 -- 172.21.15.161:0/783545856 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c64003520 con 0x7f7c64003a90 2024-03-28T23:34:49.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.987+0000 7f7c7b7fe640 1 --2- 172.21.15.161:0/783545856 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c64003a90 0x7f7c64003eb0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f7c7000c210 tx=0x7f7c64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32eb112181b7cfb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:49.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.987+0000 7f7c7a7fc640 1 -- 172.21.15.161:0/783545856 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c70007d50 con 0x7f7c64003a90 2024-03-28T23:34:49.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.987+0000 7f7c7a7fc640 1 -- 172.21.15.161:0/783545856 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c70030070 con 0x7f7c64003a90 2024-03-28T23:34:49.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.987+0000 7f7c7a7fc640 1 -- 172.21.15.161:0/783545856 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c7002cd80 con 0x7f7c64003a90 2024-03-28T23:34:49.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.988+0000 7f7c81cc0640 1 -- 172.21.15.161:0/783545856 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c64003a90 msgr2=0x7f7c64003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:49.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.988+0000 7f7c81cc0640 1 --2- 172.21.15.161:0/783545856 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c64003a90 0x7f7c64003eb0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f7c7000c210 tx=0x7f7c64005830 comp rx=0 tx=0).stop 2024-03-28T23:34:49.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.988+0000 7f7c7bfff640 1 -- 172.21.15.161:0/783545856 reap_dead start 2024-03-28T23:34:49.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.988+0000 7f7c81cc0640 1 -- 172.21.15.161:0/783545856 shutdown_connections 2024-03-28T23:34:49.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.988+0000 7f7c81cc0640 1 -- 172.21.15.161:0/783545856 >> 172.21.15.161:0/783545856 conn(0x7f7c7c0ac620 msgr2=0x7f7c7c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:49.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.988+0000 7f7c81cc0640 1 -- 172.21.15.161:0/783545856 shutdown_connections 2024-03-28T23:34:49.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.988+0000 7f7c81cc0640 1 -- 172.21.15.161:0/783545856 wait complete. 2024-03-28T23:34:49.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.989+0000 7f7c81cc0640 1 Processor -- start 2024-03-28T23:34:49.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.989+0000 7f7c81cc0640 1 -- start start 2024-03-28T23:34:49.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.990+0000 7f7c81cc0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c64003a90 0x7f7c7c157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.990+0000 7f7c81cc0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c1510f0 0x7f7c7c1edf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.990+0000 7f7c81cc0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c7c1eb610 0x7f7c7c148b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.990+0000 7f7c81cc0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7c1592d0 con 0x7f7c64003a90 2024-03-28T23:34:49.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.990+0000 7f7c81cc0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7c159640 con 0x7f7c7c1eb610 2024-03-28T23:34:49.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.990+0000 7f7c81cc0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7c7c0c90a0 con 0x7f7c7c1510f0 2024-03-28T23:34:49.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.990+0000 7f7c7b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c64003a90 0x7f7c7c157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.990+0000 7f7c7b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c64003a90 0x7f7c7c157e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54864/0 (socket says 172.21.15.161:54864) 2024-03-28T23:34:49.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.990+0000 7f7c7b7fe640 1 -- 172.21.15.161:0/856821491 learned_addr learned my addr 172.21.15.161:0/856821491 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:49.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.990+0000 7f7c7affd640 1 --2- 172.21.15.161:0/856821491 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c1510f0 0x7f7c7c1edf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.990+0000 7f7c7bfff640 1 --2- 172.21.15.161:0/856821491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c7c1eb610 0x7f7c7c148b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.990+0000 7f7c7b7fe640 1 -- 172.21.15.161:0/856821491 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c1510f0 msgr2=0x7f7c7c1edf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:49.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.990+0000 7f7c7b7fe640 1 --2- 172.21.15.161:0/856821491 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c1510f0 0x7f7c7c1edf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.990+0000 7f7c7b7fe640 1 -- 172.21.15.161:0/856821491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c7c1eb610 msgr2=0x7f7c7c148b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:49.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.990+0000 7f7c7b7fe640 1 --2- 172.21.15.161:0/856821491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c7c1eb610 0x7f7c7c148b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:49.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.990+0000 7f7c7b7fe640 1 -- 172.21.15.161:0/856821491 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c7c14f960 con 0x7f7c64003a90 2024-03-28T23:34:49.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.990+0000 7f7c7b7fe640 1 --2- 172.21.15.161:0/856821491 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c64003a90 0x7f7c7c157e20 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f7c7000c210 tx=0x7f7c70001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:49.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.990+0000 7f7c63fff640 1 -- 172.21.15.161:0/856821491 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c70045070 con 0x7f7c64003a90 2024-03-28T23:34:49.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.990+0000 7f7c63fff640 1 -- 172.21.15.161:0/856821491 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c70004bf0 con 0x7f7c64003a90 2024-03-28T23:34:49.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.990+0000 7f7c63fff640 1 -- 172.21.15.161:0/856821491 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c70001c90 con 0x7f7c64003a90 2024-03-28T23:34:49.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.990+0000 7f7c81cc0640 1 -- 172.21.15.161:0/856821491 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c7c1515f0 con 0x7f7c64003a90 2024-03-28T23:34:49.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.990+0000 7f7c81cc0640 1 -- 172.21.15.161:0/856821491 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c7c1eae10 con 0x7f7c64003a90 2024-03-28T23:34:49.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.991+0000 7f7c81cc0640 1 -- 172.21.15.161:0/856821491 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c7c04f4b0 con 0x7f7c64003a90 2024-03-28T23:34:49.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.992+0000 7f7c63fff640 1 -- 172.21.15.161:0/856821491 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7c70007070 con 0x7f7c64003a90 2024-03-28T23:34:49.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.992+0000 7f7c63fff640 1 --2- 172.21.15.161:0/856821491 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7c6806b500 0x7f7c6806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:49.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.992+0000 7f7c63fff640 1 -- 172.21.15.161:0/856821491 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6101+0+0 (secure 0 0 0) 0x7f7c700ab2e0 con 0x7f7c64003a90 2024-03-28T23:34:49.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.992+0000 7f7c7affd640 1 --2- 172.21.15.161:0/856821491 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7c6806b500 0x7f7c6806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:49.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.993+0000 7f7c7affd640 1 --2- 172.21.15.161:0/856821491 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7c6806b500 0x7f7c6806d9c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c151040 tx=0x7f7c6c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:49.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:49.997+0000 7f7c63fff640 1 -- 172.21.15.161:0/856821491 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7c7c14f960 con 0x7f7c64003a90 2024-03-28T23:34:50.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.244+0000 7f7c81cc0640 1 -- 172.21.15.161:0/856821491 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "9:0"} v 0) -- 0x7f7c7c0cb3a0 con 0x7f7c64003a90 2024-03-28T23:34:50.245 INFO:tasks.ceph.mon.a.smithi161.stderr:2024-03-28T23:34:50.244+0000 7fdc7b85b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T23:34:50.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.246+0000 7f7c63fff640 1 -- 172.21.15.161:0/856821491 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "9:0"}]=0 failed mds gid 7974 v109) ==== 97+0+0 (secure 0 0 0) 0x7f7c700ab0e0 con 0x7f7c64003a90 2024-03-28T23:34:50.246 INFO:teuthology.orchestra.run.smithi161.stderr:failed mds gid 7974 2024-03-28T23:34:50.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.249+0000 7f7c81cc0640 1 -- 172.21.15.161:0/856821491 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7c6806b500 msgr2=0x7f7c6806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:50.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.249+0000 7f7c81cc0640 1 --2- 172.21.15.161:0/856821491 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7c6806b500 0x7f7c6806d9c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c151040 tx=0x7f7c6c008040 comp rx=0 tx=0).stop 2024-03-28T23:34:50.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.249+0000 7f7c81cc0640 1 -- 172.21.15.161:0/856821491 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c64003a90 msgr2=0x7f7c7c157e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:50.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.249+0000 7f7c81cc0640 1 --2- 172.21.15.161:0/856821491 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c64003a90 0x7f7c7c157e20 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f7c7000c210 tx=0x7f7c70001a40 comp rx=0 tx=0).stop 2024-03-28T23:34:50.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.249+0000 7f7c81cc0640 1 -- 172.21.15.161:0/856821491 shutdown_connections 2024-03-28T23:34:50.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.249+0000 7f7c81cc0640 1 --2- 172.21.15.161:0/856821491 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7c6806b500 0x7f7c6806d9c0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:50.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.249+0000 7f7c81cc0640 1 --2- 172.21.15.161:0/856821491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c7c1eb610 0x7f7c7c148b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:50.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.249+0000 7f7c81cc0640 1 --2- 172.21.15.161:0/856821491 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c1510f0 0x7f7c7c1edf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:50.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.249+0000 7f7c81cc0640 1 --2- 172.21.15.161:0/856821491 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c64003a90 0x7f7c7c157e20 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:50.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.249+0000 7f7c81cc0640 1 -- 172.21.15.161:0/856821491 >> 172.21.15.161:0/856821491 conn(0x7f7c7c0ac620 msgr2=0x7f7c7c0ca250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:50.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.249+0000 7f7c81cc0640 1 -- 172.21.15.161:0/856821491 shutdown_connections 2024-03-28T23:34:50.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.249+0000 7f7c81cc0640 1 -- 172.21.15.161:0/856821491 wait complete. 2024-03-28T23:34:50.251 INFO:tasks.ceph.mds.a.smithi161.stderr: -349> 2024-03-28T23:34:42.430+0000 7fd53c781640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:34:50.264 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:34:50.278 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-28T23:34:50.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.364+0000 7fdb719a0640 1 Processor -- start 2024-03-28T23:34:50.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.364+0000 7fdb719a0640 1 -- start start 2024-03-28T23:34:50.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.365+0000 7fdb719a0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdb6c06c670 0x7fdb6c0aa830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:50.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.365+0000 7fdb719a0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb6c151760 con 0x7fdb6c0a8eb0 2024-03-28T23:34:50.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.365+0000 7fdb719a0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb6c06cde0 con 0x7fdb6c14ed30 2024-03-28T23:34:50.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.365+0000 7fdb719a0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fdb6c06d3a0 con 0x7fdb6c06c670 2024-03-28T23:34:50.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.365+0000 7fdb7119f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdb6c06c670 0x7fdb6c0aa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:50.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.365+0000 7fdb6bfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fdb6c0a8eb0 0x7fdb6c151f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50430/0 (socket says 172.21.15.161:50430) 2024-03-28T23:34:50.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.365+0000 7fdb6bfff640 1 -- 172.21.15.161:0/2626224211 learned_addr learned my addr 172.21.15.161:0/2626224211 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:50.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.365+0000 7fdb7119f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdb6c06c670 0x7fdb6c0aa830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54896/0 (socket says 172.21.15.161:54896) 2024-03-28T23:34:50.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.365+0000 7fdb7119f640 1 -- 172.21.15.161:0/2626224211 >> v1:172.21.15.170:6789/0 conn(0x7fdb6c14ed30 legacy=0x7fdb6c0a9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:50.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.365+0000 7fdb7119f640 1 -- 172.21.15.161:0/2626224211 >> v1:172.21.15.161:6789/0 conn(0x7fdb6c0a8eb0 legacy=0x7fdb6c151f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:50.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.365+0000 7fdb7119f640 1 -- 172.21.15.161:0/2626224211 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb6c06d0c0 con 0x7fdb6c06c670 2024-03-28T23:34:50.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.365+0000 7fdb6b7fe640 1 -- 172.21.15.161:0/2626224211 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1711253783 0 0) 0x7fdb6c151760 con 0x7fdb6c0a8eb0 2024-03-28T23:34:50.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.366+0000 7fdb7119f640 1 --2- 172.21.15.161:0/2626224211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb6c06c670 0x7fdb6c0aa830 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c00a940 tx=0x7fdb5c02cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1bc71f6aeccc7d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:50.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.366+0000 7fdb6b7fe640 1 -- 172.21.15.161:0/2626224211 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c03a070 con 0x7fdb6c06c670 2024-03-28T23:34:50.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.366+0000 7fdb6b7fe640 1 -- 172.21.15.161:0/2626224211 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb5c02db40 con 0x7fdb6c06c670 2024-03-28T23:34:50.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.366+0000 7fdb6b7fe640 1 -- 172.21.15.161:0/2626224211 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c00b040 con 0x7fdb6c06c670 2024-03-28T23:34:50.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.367+0000 7fdb719a0640 1 -- 172.21.15.161:0/2626224211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb6c06c670 msgr2=0x7fdb6c0aa830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:50.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.367+0000 7fdb719a0640 1 --2- 172.21.15.161:0/2626224211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb6c06c670 0x7fdb6c0aa830 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c00a940 tx=0x7fdb5c02cd90 comp rx=0 tx=0).stop 2024-03-28T23:34:50.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.367+0000 7fdb719a0640 1 -- 172.21.15.161:0/2626224211 shutdown_connections 2024-03-28T23:34:50.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.367+0000 7fdb719a0640 1 --2- 172.21.15.161:0/2626224211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb6c06c670 0x7fdb6c0aa830 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:50.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.367+0000 7fdb719a0640 1 -- 172.21.15.161:0/2626224211 >> 172.21.15.161:0/2626224211 conn(0x7fdb6c0c66d0 msgr2=0x7fdb6c0c6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:50.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.367+0000 7fdb719a0640 1 -- 172.21.15.161:0/2626224211 shutdown_connections 2024-03-28T23:34:50.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.367+0000 7fdb719a0640 1 -- 172.21.15.161:0/2626224211 wait complete. 2024-03-28T23:34:50.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.368+0000 7fdb719a0640 1 Processor -- start 2024-03-28T23:34:50.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.368+0000 7fdb719a0640 1 -- start start 2024-03-28T23:34:50.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.369+0000 7fdb719a0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdb6c06c670 0x7fdb6c1dd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:50.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.369+0000 7fdb719a0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb6c0a8eb0 0x7fdb6c14c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:50.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.369+0000 7fdb719a0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb6c14ed30 0x7fdb6c1e8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:50.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.369+0000 7fdb719a0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6c0aad70 con 0x7fdb6c14ed30 2024-03-28T23:34:50.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.369+0000 7fdb719a0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6c06cae0 con 0x7fdb6c0a8eb0 2024-03-28T23:34:50.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.369+0000 7fdb719a0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdb6c0aaef0 con 0x7fdb6c06c670 2024-03-28T23:34:50.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.369+0000 7fdb7119f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb6c14ed30 0x7fdb6c1e8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:50.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.369+0000 7fdb7119f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb6c14ed30 0x7fdb6c1e8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54902/0 (socket says 172.21.15.161:54902) 2024-03-28T23:34:50.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.369+0000 7fdb7119f640 1 -- 172.21.15.161:0/1263482419 learned_addr learned my addr 172.21.15.161:0/1263482419 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:50.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.369+0000 7fdb6bfff640 1 --2- 172.21.15.161:0/1263482419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb6c0a8eb0 0x7fdb6c14c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:50.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.369+0000 7fdb7099e640 1 --2- 172.21.15.161:0/1263482419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdb6c06c670 0x7fdb6c1dd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:50.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.369+0000 7fdb7119f640 1 -- 172.21.15.161:0/1263482419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdb6c06c670 msgr2=0x7fdb6c1dd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:50.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.369+0000 7fdb7119f640 1 --2- 172.21.15.161:0/1263482419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdb6c06c670 0x7fdb6c1dd3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:50.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.369+0000 7fdb7119f640 1 -- 172.21.15.161:0/1263482419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb6c0a8eb0 msgr2=0x7fdb6c14c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:50.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.369+0000 7fdb7119f640 1 --2- 172.21.15.161:0/1263482419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb6c0a8eb0 0x7fdb6c14c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:50.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.369+0000 7fdb7119f640 1 -- 172.21.15.161:0/1263482419 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb6c1ec790 con 0x7fdb6c14ed30 2024-03-28T23:34:50.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.369+0000 7fdb7119f640 1 --2- 172.21.15.161:0/1263482419 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb6c14ed30 0x7fdb6c1e8ff0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c038040 tx=0x7fdb5c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:50.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.369+0000 7fdb697fa640 1 -- 172.21.15.161:0/1263482419 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c045070 con 0x7fdb6c14ed30 2024-03-28T23:34:50.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.369+0000 7fdb697fa640 1 -- 172.21.15.161:0/1263482419 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb5c004030 con 0x7fdb6c14ed30 2024-03-28T23:34:50.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.369+0000 7fdb719a0640 1 -- 172.21.15.161:0/1263482419 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb6c1e87f0 con 0x7fdb6c14ed30 2024-03-28T23:34:50.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.369+0000 7fdb697fa640 1 -- 172.21.15.161:0/1263482419 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c03a040 con 0x7fdb6c14ed30 2024-03-28T23:34:50.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.369+0000 7fdb719a0640 1 -- 172.21.15.161:0/1263482419 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb6c1eca70 con 0x7fdb6c14ed30 2024-03-28T23:34:50.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.371+0000 7fdb719a0640 1 -- 172.21.15.161:0/1263482419 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb6c04f4b0 con 0x7fdb6c14ed30 2024-03-28T23:34:50.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.371+0000 7fdb697fa640 1 -- 172.21.15.161:0/1263482419 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdb5c048070 con 0x7fdb6c14ed30 2024-03-28T23:34:50.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.371+0000 7fdb697fa640 1 --2- 172.21.15.161:0/1263482419 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdb5806b4d0 0x7fdb5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:50.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.371+0000 7fdb697fa640 1 -- 172.21.15.161:0/1263482419 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6187+0+0 (secure 0 0 0) 0x7fdb5c0abe10 con 0x7fdb6c14ed30 2024-03-28T23:34:50.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.371+0000 7fdb7099e640 1 --2- 172.21.15.161:0/1263482419 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdb5806b4d0 0x7fdb5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:50.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.372+0000 7fdb7099e640 1 --2- 172.21.15.161:0/1263482419 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdb5806b4d0 0x7fdb5806d990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fdb64007fd0 tx=0x7fdb64009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:50.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.376+0000 7fdb697fa640 1 -- 172.21.15.161:0/1263482419 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdb5c07ab40 con 0x7fdb6c14ed30 2024-03-28T23:34:50.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.582+0000 7fdb719a0640 1 -- 172.21.15.161:0/1263482419 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb6c0c2ea0 con 0x7fdb6c14ed30 2024-03-28T23:34:50.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.583+0000 7fdb697fa640 1 -- 172.21.15.161:0/1263482419 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+4364 (secure 0 0 0) 0x7fdb5c07a440 con 0x7fdb6c14ed30 2024-03-28T23:34:50.584 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:50.584 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":110,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7318,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3734750343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3734750343},{"type":"v1","addr":"172.21.15.170:6835","nonce":3734750343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/4140438862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":4140438862},{"type":"v1","addr":"172.21.15.170:6834","nonce":4140438862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:34:36.796608+0000","modified":"2024-03-28T23:34:50.250272+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7965},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7965":{"gid":7965,"name":"c","rank":0,"incarnation":110,"state":"up:replay","state_seq":1,"addr":"172.21.15.161:6835/878057589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":878057589},{"type":"v1","addr":"172.21.15.161:6835","nonce":878057589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":9}]} 2024-03-28T23:34:50.584 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 110 2024-03-28T23:34:50.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.585+0000 7fdb719a0640 1 -- 172.21.15.161:0/1263482419 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdb5806b4d0 msgr2=0x7fdb5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:50.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.585+0000 7fdb719a0640 1 --2- 172.21.15.161:0/1263482419 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdb5806b4d0 0x7fdb5806d990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fdb64007fd0 tx=0x7fdb64009040 comp rx=0 tx=0).stop 2024-03-28T23:34:50.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.585+0000 7fdb719a0640 1 -- 172.21.15.161:0/1263482419 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb6c14ed30 msgr2=0x7fdb6c1e8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:50.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.585+0000 7fdb719a0640 1 --2- 172.21.15.161:0/1263482419 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb6c14ed30 0x7fdb6c1e8ff0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c038040 tx=0x7fdb5c004730 comp rx=0 tx=0).stop 2024-03-28T23:34:50.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.585+0000 7fdb719a0640 1 -- 172.21.15.161:0/1263482419 shutdown_connections 2024-03-28T23:34:50.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.585+0000 7fdb719a0640 1 --2- 172.21.15.161:0/1263482419 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdb5806b4d0 0x7fdb5806d990 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:50.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.585+0000 7fdb719a0640 1 --2- 172.21.15.161:0/1263482419 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb6c14ed30 0x7fdb6c1e8ff0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:50.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.585+0000 7fdb719a0640 1 --2- 172.21.15.161:0/1263482419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb6c0a8eb0 0x7fdb6c14c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:50.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.585+0000 7fdb719a0640 1 --2- 172.21.15.161:0/1263482419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdb6c06c670 0x7fdb6c1dd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:50.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.585+0000 7fdb719a0640 1 -- 172.21.15.161:0/1263482419 >> 172.21.15.161:0/1263482419 conn(0x7fdb6c0c66d0 msgr2=0x7fdb6c0c0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:50.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.585+0000 7fdb719a0640 1 -- 172.21.15.161:0/1263482419 shutdown_connections 2024-03-28T23:34:50.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:50.585+0000 7fdb719a0640 1 -- 172.21.15.161:0/1263482419 wait complete. 2024-03-28T23:34:50.999 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 110, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:34:36.796608+0000', 'modified': '2024-03-28T23:34:50.250272+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 123, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7965}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7965': {'gid': 7965, 'name': 'c', 'rank': 0, 'incarnation': 110, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.161:6835/878057589', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 878057589}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 878057589}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-28T23:34:50.999 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7965:up:replay 2024-03-28T23:34:52.000 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:34:52.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.112+0000 7f8af5f2c640 1 Processor -- start 2024-03-28T23:34:52.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.112+0000 7f8af5f2c640 1 -- start start 2024-03-28T23:34:52.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.113+0000 7f8af5f2c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8af00aacd0 0x7f8af015d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:52.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.113+0000 7f8af5f2c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8af014c250 con 0x7f8af006ce80 2024-03-28T23:34:52.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.113+0000 7f8af5f2c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8af01529c0 con 0x7f8af014be70 2024-03-28T23:34:52.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.113+0000 7f8af5f2c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f8af00a9dc0 con 0x7f8af00aacd0 2024-03-28T23:34:52.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.113+0000 7f8aeeffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f8af006ce80 0x7f8af00aa5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44872/0 (socket says 172.21.15.161:44872) 2024-03-28T23:34:52.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.113+0000 7f8aeeffd640 1 -- 172.21.15.161:0/1875221152 learned_addr learned my addr 172.21.15.161:0/1875221152 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:52.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.113+0000 7f8aeffff640 1 --2- 172.21.15.161:0/1875221152 >> v2:172.21.15.170:3300/0 conn(0x7f8af00aacd0 0x7f8af015d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:52.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.113+0000 7f8aee7fc640 1 -- 172.21.15.161:0/1875221152 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3318084818 0 0) 0x7f8af014c250 con 0x7f8af006ce80 2024-03-28T23:34:52.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.113+0000 7f8aee7fc640 1 -- 172.21.15.161:0/1875221152 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ad8005730 con 0x7f8af006ce80 2024-03-28T23:34:52.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.113+0000 7f8aee7fc640 1 -- 172.21.15.161:0/1875221152 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3180921509 0 0) 0x7f8ad8005730 con 0x7f8af006ce80 2024-03-28T23:34:52.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.113+0000 7f8aee7fc640 1 -- 172.21.15.161:0/1875221152 >> v2:172.21.15.170:3300/0 conn(0x7f8af00aacd0 msgr2=0x7f8af015d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:52.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.113+0000 7f8aee7fc640 1 --2- 172.21.15.161:0/1875221152 >> v2:172.21.15.170:3300/0 conn(0x7f8af00aacd0 0x7f8af015d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:52.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.113+0000 7f8aee7fc640 1 -- 172.21.15.161:0/1875221152 >> v1:172.21.15.170:6789/0 conn(0x7f8af014be70 legacy=0x7f8af00ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:52.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.113+0000 7f8aee7fc640 1 -- 172.21.15.161:0/1875221152 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8af00aa0a0 con 0x7f8af006ce80 2024-03-28T23:34:52.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.113+0000 7f8aee7fc640 1 -- 172.21.15.161:0/1875221152 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f8ae4001670 con 0x7f8af006ce80 2024-03-28T23:34:52.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.114+0000 7f8aee7fc640 1 -- 172.21.15.161:0/1875221152 >> v1:172.21.15.161:6789/0 conn(0x7f8af006ce80 legacy=0x7f8af00aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:52.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.114+0000 7f8aee7fc640 1 --2- 172.21.15.161:0/1875221152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ad8005ca0 0x7f8ad80060c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:52.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.114+0000 7f8aee7fc640 1 --2- 172.21.15.161:0/1875221152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ad8006a10 0x7f8ad8006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:52.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.114+0000 7f8aee7fc640 1 --2- 172.21.15.161:0/1875221152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ad80074a0 0x7f8ad800dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:52.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.114+0000 7f8aee7fc640 1 -- 172.21.15.161:0/1875221152 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8af00a9dc0 con 0x7f8ad80074a0 2024-03-28T23:34:52.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.114+0000 7f8aee7fc640 1 -- 172.21.15.161:0/1875221152 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad800e3f0 con 0x7f8ad8006a10 2024-03-28T23:34:52.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.114+0000 7f8aee7fc640 1 -- 172.21.15.161:0/1875221152 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8ad800e5f0 con 0x7f8ad8005ca0 2024-03-28T23:34:52.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.114+0000 7f8aeffff640 1 --2- 172.21.15.161:0/1875221152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ad80074a0 0x7f8ad800dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:52.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.114+0000 7f8aeeffd640 1 --2- 172.21.15.161:0/1875221152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ad8006a10 0x7f8ad8006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:52.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.114+0000 7f8aef7fe640 1 --2- 172.21.15.161:0/1875221152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ad8005ca0 0x7f8ad80060c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:52.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.114+0000 7f8aeffff640 1 -- 172.21.15.161:0/1875221152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ad8005ca0 msgr2=0x7f8ad80060c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:52.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.114+0000 7f8aeffff640 1 --2- 172.21.15.161:0/1875221152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ad8005ca0 0x7f8ad80060c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:52.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.114+0000 7f8aeffff640 1 -- 172.21.15.161:0/1875221152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ad8006a10 msgr2=0x7f8ad8006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:52.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.114+0000 7f8aeffff640 1 --2- 172.21.15.161:0/1875221152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ad8006a10 0x7f8ad8006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:52.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.114+0000 7f8aeffff640 1 -- 172.21.15.161:0/1875221152 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad8005730 con 0x7f8ad80074a0 2024-03-28T23:34:52.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.115+0000 7f8aeffff640 1 -- 172.21.15.161:0/1875221152 reap_dead start 2024-03-28T23:34:52.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.115+0000 7f8aeffff640 1 --2- 172.21.15.161:0/1875221152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ad80074a0 0x7f8ad800dbd0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8009fc0 tx=0x7f8ad8007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e613250e4f2b8a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:52.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.115+0000 7f8aee7fc640 1 -- 172.21.15.161:0/1875221152 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ae8009c10 con 0x7f8ad80074a0 2024-03-28T23:34:52.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.115+0000 7f8aee7fc640 1 -- 172.21.15.161:0/1875221152 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ae802cca0 con 0x7f8ad80074a0 2024-03-28T23:34:52.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.115+0000 7f8aee7fc640 1 -- 172.21.15.161:0/1875221152 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ae8003bd0 con 0x7f8ad80074a0 2024-03-28T23:34:52.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.115+0000 7f8af5f2c640 1 -- 172.21.15.161:0/1875221152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ad80074a0 msgr2=0x7f8ad800dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:52.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.115+0000 7f8af5f2c640 1 --2- 172.21.15.161:0/1875221152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ad80074a0 0x7f8ad800dbd0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8009fc0 tx=0x7f8ad8007a40 comp rx=0 tx=0).stop 2024-03-28T23:34:52.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.115+0000 7f8af5f2c640 1 -- 172.21.15.161:0/1875221152 shutdown_connections 2024-03-28T23:34:52.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.115+0000 7f8af5f2c640 1 --2- 172.21.15.161:0/1875221152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ad80074a0 0x7f8ad800dbd0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:52.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.115+0000 7f8af5f2c640 1 -- 172.21.15.161:0/1875221152 >> 172.21.15.161:0/1875221152 conn(0x7f8af00c6670 msgr2=0x7f8af00c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:52.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.116+0000 7f8af5f2c640 1 -- 172.21.15.161:0/1875221152 shutdown_connections 2024-03-28T23:34:52.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.116+0000 7f8af5f2c640 1 -- 172.21.15.161:0/1875221152 wait complete. 2024-03-28T23:34:52.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.117+0000 7f8af5f2c640 1 Processor -- start 2024-03-28T23:34:52.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.117+0000 7f8af5f2c640 1 -- start start 2024-03-28T23:34:52.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.117+0000 7f8af5f2c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ad80074a0 0x7f8af014f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:52.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.117+0000 7f8af5f2c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8af01e66a0 0x7f8af00c5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:52.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.117+0000 7f8af5f2c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8af014ce90 0x7f8af014eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:52.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.117+0000 7f8af5f2c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8af006b2d0 con 0x7f8af01e66a0 2024-03-28T23:34:52.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.117+0000 7f8af5f2c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8af006b150 con 0x7f8ad80074a0 2024-03-28T23:34:52.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.117+0000 7f8af5f2c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8af00c1520 con 0x7f8af014ce90 2024-03-28T23:34:52.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.118+0000 7f8aeeffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8af01e66a0 0x7f8af00c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:52.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.118+0000 7f8aeeffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8af01e66a0 0x7f8af00c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59686/0 (socket says 172.21.15.161:59686) 2024-03-28T23:34:52.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.118+0000 7f8aeeffd640 1 -- 172.21.15.161:0/928234424 learned_addr learned my addr 172.21.15.161:0/928234424 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:52.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.118+0000 7f8aeffff640 1 --2- 172.21.15.161:0/928234424 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8af014ce90 0x7f8af014eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:52.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.118+0000 7f8aef7fe640 1 --2- 172.21.15.161:0/928234424 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ad80074a0 0x7f8af014f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:52.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.118+0000 7f8aeeffd640 1 -- 172.21.15.161:0/928234424 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8af014ce90 msgr2=0x7f8af014eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:52.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.118+0000 7f8aeeffd640 1 --2- 172.21.15.161:0/928234424 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8af014ce90 0x7f8af014eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:52.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.118+0000 7f8aeeffd640 1 -- 172.21.15.161:0/928234424 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ad80074a0 msgr2=0x7f8af014f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:52.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.118+0000 7f8aeeffd640 1 --2- 172.21.15.161:0/928234424 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ad80074a0 0x7f8af014f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:52.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.118+0000 7f8aeeffd640 1 -- 172.21.15.161:0/928234424 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8af014e1a0 con 0x7f8af01e66a0 2024-03-28T23:34:52.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.118+0000 7f8aeeffd640 1 --2- 172.21.15.161:0/928234424 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8af01e66a0 0x7f8af00c5410 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f8ae4007290 tx=0x7f8ae4007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:52.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.118+0000 7f8ad7fff640 1 -- 172.21.15.161:0/928234424 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ae400f8e0 con 0x7f8af01e66a0 2024-03-28T23:34:52.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.118+0000 7f8ad7fff640 1 -- 172.21.15.161:0/928234424 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ae4018070 con 0x7f8af01e66a0 2024-03-28T23:34:52.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.118+0000 7f8ad7fff640 1 -- 172.21.15.161:0/928234424 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ae400fce0 con 0x7f8af01e66a0 2024-03-28T23:34:52.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.118+0000 7f8af5f2c640 1 -- 172.21.15.161:0/928234424 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8af014fea0 con 0x7f8af01e66a0 2024-03-28T23:34:52.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.118+0000 7f8af5f2c640 1 -- 172.21.15.161:0/928234424 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8af0155e00 con 0x7f8af01e66a0 2024-03-28T23:34:52.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.119+0000 7f8af5f2c640 1 -- 172.21.15.161:0/928234424 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8af004f4b0 con 0x7f8af01e66a0 2024-03-28T23:34:52.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.120+0000 7f8ad7fff640 1 -- 172.21.15.161:0/928234424 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8ae401d460 con 0x7f8af01e66a0 2024-03-28T23:34:52.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.120+0000 7f8ad7fff640 1 --2- 172.21.15.161:0/928234424 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8adc06b500 0x7f8adc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:52.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.120+0000 7f8aef7fe640 1 --2- 172.21.15.161:0/928234424 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8adc06b500 0x7f8adc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:52.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.120+0000 7f8ad7fff640 1 -- 172.21.15.161:0/928234424 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6187+0+0 (secure 0 0 0) 0x7f8ae408a260 con 0x7f8af01e66a0 2024-03-28T23:34:52.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.121+0000 7f8aef7fe640 1 --2- 172.21.15.161:0/928234424 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8adc06b500 0x7f8adc06d9c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0000c00 tx=0x7f8ae0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:52.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.125+0000 7f8ad7fff640 1 -- 172.21.15.161:0/928234424 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8af014fea0 con 0x7f8af01e66a0 2024-03-28T23:34:52.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.333+0000 7f8af5f2c640 1 -- 172.21.15.161:0/928234424 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8af0149380 con 0x7f8af01e66a0 2024-03-28T23:34:52.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.333+0000 7f8ad7fff640 1 -- 172.21.15.161:0/928234424 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+5192 (secure 0 0 0) 0x7f8ae4058940 con 0x7f8af01e66a0 2024-03-28T23:34:52.334 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:52.334 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":112,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7318,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3734750343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3734750343},{"type":"v1","addr":"172.21.15.170:6835","nonce":3734750343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/4140438862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":4140438862},{"type":"v1","addr":"172.21.15.170:6834","nonce":4140438862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":8199,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/3929102513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":3929102513},{"type":"v1","addr":"172.21.15.161:6837","nonce":3929102513}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:34:36.796608+0000","modified":"2024-03-28T23:34:51.254907+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7965},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7965":{"gid":7965,"name":"c","rank":0,"incarnation":110,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.161:6835/878057589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":878057589},{"type":"v1","addr":"172.21.15.161:6835","nonce":878057589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":9}]} 2024-03-28T23:34:52.334 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 112 2024-03-28T23:34:52.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.335+0000 7f8af5f2c640 1 -- 172.21.15.161:0/928234424 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8adc06b500 msgr2=0x7f8adc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:52.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.335+0000 7f8af5f2c640 1 --2- 172.21.15.161:0/928234424 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8adc06b500 0x7f8adc06d9c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0000c00 tx=0x7f8ae0009040 comp rx=0 tx=0).stop 2024-03-28T23:34:52.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.335+0000 7f8af5f2c640 1 -- 172.21.15.161:0/928234424 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8af01e66a0 msgr2=0x7f8af00c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:52.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.335+0000 7f8af5f2c640 1 --2- 172.21.15.161:0/928234424 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8af01e66a0 0x7f8af00c5410 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f8ae4007290 tx=0x7f8ae4007750 comp rx=0 tx=0).stop 2024-03-28T23:34:52.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.335+0000 7f8af5f2c640 1 -- 172.21.15.161:0/928234424 shutdown_connections 2024-03-28T23:34:52.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.335+0000 7f8af5f2c640 1 --2- 172.21.15.161:0/928234424 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8adc06b500 0x7f8adc06d9c0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:52.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.335+0000 7f8af5f2c640 1 --2- 172.21.15.161:0/928234424 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8af01e66a0 0x7f8af00c5410 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:52.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.335+0000 7f8af5f2c640 1 --2- 172.21.15.161:0/928234424 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8af014ce90 0x7f8af014eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:52.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.335+0000 7f8af5f2c640 1 --2- 172.21.15.161:0/928234424 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ad80074a0 0x7f8af014f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:52.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.335+0000 7f8af5f2c640 1 -- 172.21.15.161:0/928234424 >> 172.21.15.161:0/928234424 conn(0x7f8af00c6670 msgr2=0x7f8af01515f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:52.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.335+0000 7f8af5f2c640 1 -- 172.21.15.161:0/928234424 shutdown_connections 2024-03-28T23:34:52.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:52.335+0000 7f8af5f2c640 1 -- 172.21.15.161:0/928234424 wait complete. 2024-03-28T23:34:52.749 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 112, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:34:36.796608+0000', 'modified': '2024-03-28T23:34:51.254907+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 123, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7965}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7965': {'gid': 7965, 'name': 'c', 'rank': 0, 'incarnation': 110, 'state': 'up:rejoin', 'state_seq': 9, 'addr': '172.21.15.161:6835/878057589', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 878057589}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 878057589}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-28T23:34:52.749 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7965:up:rejoin 2024-03-28T23:34:53.299 INFO:tasks.ceph.mds.c.smithi161.stdout:starting mds.c at 2024-03-28T23:34:53.750 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:34:53.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.862+0000 7f7b8c579640 1 Processor -- start 2024-03-28T23:34:53.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.862+0000 7f7b8c579640 1 -- start start 2024-03-28T23:34:53.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.863+0000 7f7b8c579640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b8414be70 0x7f7b840ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:53.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.863+0000 7f7b8c579640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7b840ab6e0 0x7f7b840aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:53.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.863+0000 7f7b8c579640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b8414c250 con 0x7f7b8406ce80 2024-03-28T23:34:53.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.863+0000 7f7b8c579640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f7b841529c0 con 0x7f7b840ab6e0 2024-03-28T23:34:53.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.863+0000 7f7b8c579640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7b840a9dc0 con 0x7f7b8414be70 2024-03-28T23:34:53.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.863+0000 7f7b89b14640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7b840ab6e0 0x7f7b840aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:53.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.863+0000 7f7b89b14640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7b840ab6e0 0x7f7b840aa5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59714/0 (socket says 172.21.15.161:59714) 2024-03-28T23:34:53.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.863+0000 7f7b89b14640 1 -- 172.21.15.161:0/1681483450 learned_addr learned my addr 172.21.15.161:0/1681483450 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:53.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.863+0000 7f7b8a315640 1 --2- 172.21.15.161:0/1681483450 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b8414be70 0x7f7b840ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:53.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.864+0000 7f7b89b14640 1 -- 172.21.15.161:0/1681483450 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b8414be70 msgr2=0x7f7b840ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:53.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.864+0000 7f7b89b14640 1 --2- 172.21.15.161:0/1681483450 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b8414be70 0x7f7b840ab1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:53.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.864+0000 7f7b89b14640 1 -- 172.21.15.161:0/1681483450 >> v1:172.21.15.170:6789/0 conn(0x7f7b8406ce80 legacy=0x7f7b841553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:53.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.864+0000 7f7b89b14640 1 -- 172.21.15.161:0/1681483450 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b840aa0a0 con 0x7f7b840ab6e0 2024-03-28T23:34:53.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.864+0000 7f7b89b14640 1 --2- 172.21.15.161:0/1681483450 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b840ab6e0 0x7f7b840aa5c0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009870 tx=0x7f7b7402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e0506893fcda749 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:53.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.864+0000 7f7b89313640 1 -- 172.21.15.161:0/1681483450 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b74035020 con 0x7f7b840ab6e0 2024-03-28T23:34:53.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.865+0000 7f7b89313640 1 -- 172.21.15.161:0/1681483450 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b74031640 con 0x7f7b840ab6e0 2024-03-28T23:34:53.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.865+0000 7f7b89313640 1 -- 172.21.15.161:0/1681483450 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7402d040 con 0x7f7b840ab6e0 2024-03-28T23:34:53.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.865+0000 7f7b8c579640 1 -- 172.21.15.161:0/1681483450 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b840ab6e0 msgr2=0x7f7b840aa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:53.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.865+0000 7f7b8c579640 1 --2- 172.21.15.161:0/1681483450 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b840ab6e0 0x7f7b840aa5c0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009870 tx=0x7f7b7402b8f0 comp rx=0 tx=0).stop 2024-03-28T23:34:53.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.865+0000 7f7b8c579640 1 -- 172.21.15.161:0/1681483450 shutdown_connections 2024-03-28T23:34:53.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.865+0000 7f7b8c579640 1 --2- 172.21.15.161:0/1681483450 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b840ab6e0 0x7f7b840aa5c0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:53.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.865+0000 7f7b8c579640 1 --2- 172.21.15.161:0/1681483450 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b8414be70 0x7f7b840ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:53.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.865+0000 7f7b8c579640 1 -- 172.21.15.161:0/1681483450 >> 172.21.15.161:0/1681483450 conn(0x7f7b840c6670 msgr2=0x7f7b840c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:53.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.865+0000 7f7b8c579640 1 -- 172.21.15.161:0/1681483450 shutdown_connections 2024-03-28T23:34:53.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.865+0000 7f7b8c579640 1 -- 172.21.15.161:0/1681483450 wait complete. 2024-03-28T23:34:53.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.866+0000 7f7b8c579640 1 Processor -- start 2024-03-28T23:34:53.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.866+0000 7f7b8c579640 1 -- start start 2024-03-28T23:34:53.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.867+0000 7f7b8c579640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b8406ce80 0x7f7b841531b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:53.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.867+0000 7f7b8c579640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b840ab6e0 0x7f7b840c1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:53.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.867+0000 7f7b8c579640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b8414be70 0x7f7b84142c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:53.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.867+0000 7f7b8c579640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8406b2d0 con 0x7f7b8406ce80 2024-03-28T23:34:53.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.867+0000 7f7b8c579640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8406b150 con 0x7f7b840ab6e0 2024-03-28T23:34:53.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.867+0000 7f7b8c579640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7b840c1520 con 0x7f7b8414be70 2024-03-28T23:34:53.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.867+0000 7f7b8a315640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b8406ce80 0x7f7b841531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:53.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.867+0000 7f7b8a315640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b8406ce80 0x7f7b841531b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59718/0 (socket says 172.21.15.161:59718) 2024-03-28T23:34:53.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.867+0000 7f7b8a315640 1 -- 172.21.15.161:0/1181854235 learned_addr learned my addr 172.21.15.161:0/1181854235 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.867+0000 7f7b89b14640 1 --2- 172.21.15.161:0/1181854235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b840ab6e0 0x7f7b840c1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.867+0000 7f7b8ab16640 1 --2- 172.21.15.161:0/1181854235 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b8414be70 0x7f7b84142c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.867+0000 7f7b8a315640 1 -- 172.21.15.161:0/1181854235 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b8414be70 msgr2=0x7f7b84142c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.867+0000 7f7b8a315640 1 --2- 172.21.15.161:0/1181854235 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b8414be70 0x7f7b84142c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.867+0000 7f7b8a315640 1 -- 172.21.15.161:0/1181854235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b840ab6e0 msgr2=0x7f7b840c1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.867+0000 7f7b8a315640 1 --2- 172.21.15.161:0/1181854235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b840ab6e0 0x7f7b840c1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.867+0000 7f7b8a315640 1 -- 172.21.15.161:0/1181854235 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b84146a40 con 0x7f7b8406ce80 2024-03-28T23:34:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.867+0000 7f7b8a315640 1 --2- 172.21.15.161:0/1181854235 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b8406ce80 0x7f7b841531b0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f7b8000d800 tx=0x7f7b8000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.868+0000 7f7b7affd640 1 -- 172.21.15.161:0/1181854235 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8000b6c0 con 0x7f7b8406ce80 2024-03-28T23:34:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.868+0000 7f7b7affd640 1 -- 172.21.15.161:0/1181854235 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b80019070 con 0x7f7b8406ce80 2024-03-28T23:34:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.868+0000 7f7b7affd640 1 -- 172.21.15.161:0/1181854235 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8000ba80 con 0x7f7b8406ce80 2024-03-28T23:34:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.868+0000 7f7b8c579640 1 -- 172.21.15.161:0/1181854235 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b841e8840 con 0x7f7b8406ce80 2024-03-28T23:34:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.868+0000 7f7b8c579640 1 -- 172.21.15.161:0/1181854235 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b841e8e00 con 0x7f7b8406ce80 2024-03-28T23:34:53.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.869+0000 7f7b8c579640 1 -- 172.21.15.161:0/1181854235 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b8404f4b0 con 0x7f7b8406ce80 2024-03-28T23:34:53.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.869+0000 7f7b7affd640 1 -- 172.21.15.161:0/1181854235 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7b8000bc20 con 0x7f7b8406ce80 2024-03-28T23:34:53.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.869+0000 7f7b7affd640 1 --2- 172.21.15.161:0/1181854235 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7b6806b4d0 0x7f7b6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:53.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.870+0000 7f7b7affd640 1 -- 172.21.15.161:0/1181854235 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 6273+0+0 (secure 0 0 0) 0x7f7b8008a860 con 0x7f7b8406ce80 2024-03-28T23:34:53.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.870+0000 7f7b89b14640 1 --2- 172.21.15.161:0/1181854235 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7b6806b4d0 0x7f7b6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:53.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.870+0000 7f7b89b14640 1 --2- 172.21.15.161:0/1181854235 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7b6806b4d0 0x7f7b6806d990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7b74036000 tx=0x7f7b7403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:53.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:53.874+0000 7f7b7affd640 1 -- 172.21.15.161:0/1181854235 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7b80059530 con 0x7f7b8406ce80 2024-03-28T23:34:54.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:54.079+0000 7f7b8c579640 1 -- 172.21.15.161:0/1181854235 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b840c0ff0 con 0x7f7b8406ce80 2024-03-28T23:34:54.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:54.080+0000 7f7b7affd640 1 -- 172.21.15.161:0/1181854235 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) ==== 95+0+4366 (secure 0 0 0) 0x7f7b80058e30 con 0x7f7b8406ce80 2024-03-28T23:34:54.080 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:54.080 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":114,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7318,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3734750343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3734750343},{"type":"v1","addr":"172.21.15.170:6835","nonce":3734750343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/4140438862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":4140438862},{"type":"v1","addr":"172.21.15.170:6834","nonce":4140438862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:34:36.796608+0000","modified":"2024-03-28T23:34:53.266730+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"a","rank":0,"incarnation":114,"state":"up:replay","state_seq":1,"addr":"172.21.15.161:6837/3929102513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":3929102513},{"type":"v1","addr":"172.21.15.161:6837","nonce":3929102513}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":9}]} 2024-03-28T23:34:54.081 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 114 2024-03-28T23:34:54.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:54.082+0000 7f7b8c579640 1 -- 172.21.15.161:0/1181854235 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7b6806b4d0 msgr2=0x7f7b6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:54.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:54.082+0000 7f7b8c579640 1 --2- 172.21.15.161:0/1181854235 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7b6806b4d0 0x7f7b6806d990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7b74036000 tx=0x7f7b7403c040 comp rx=0 tx=0).stop 2024-03-28T23:34:54.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:54.082+0000 7f7b8c579640 1 -- 172.21.15.161:0/1181854235 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b8406ce80 msgr2=0x7f7b841531b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:54.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:54.082+0000 7f7b8c579640 1 --2- 172.21.15.161:0/1181854235 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b8406ce80 0x7f7b841531b0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f7b8000d800 tx=0x7f7b8000dcc0 comp rx=0 tx=0).stop 2024-03-28T23:34:54.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:54.082+0000 7f7b8c579640 1 -- 172.21.15.161:0/1181854235 shutdown_connections 2024-03-28T23:34:54.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:54.082+0000 7f7b8c579640 1 --2- 172.21.15.161:0/1181854235 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7b6806b4d0 0x7f7b6806d990 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:54.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:54.082+0000 7f7b8c579640 1 --2- 172.21.15.161:0/1181854235 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b8414be70 0x7f7b84142c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:54.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:54.082+0000 7f7b8c579640 1 --2- 172.21.15.161:0/1181854235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b840ab6e0 0x7f7b840c1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:54.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:54.082+0000 7f7b8c579640 1 --2- 172.21.15.161:0/1181854235 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b8406ce80 0x7f7b841531b0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:54.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:54.082+0000 7f7b8c579640 1 -- 172.21.15.161:0/1181854235 >> 172.21.15.161:0/1181854235 conn(0x7f7b840c6670 msgr2=0x7f7b840c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:54.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:54.082+0000 7f7b8c579640 1 -- 172.21.15.161:0/1181854235 shutdown_connections 2024-03-28T23:34:54.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:54.082+0000 7f7b8c579640 1 -- 172.21.15.161:0/1181854235 wait complete. 2024-03-28T23:34:54.496 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 114, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:34:36.796608+0000', 'modified': '2024-03-28T23:34:53.266730+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 124, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8199}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8199': {'gid': 8199, 'name': 'a', 'rank': 0, 'incarnation': 114, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.161:6837/3929102513', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 3929102513}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 3929102513}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-28T23:34:54.497 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8199:up:replay 2024-03-28T23:34:55.497 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:34:55.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.610+0000 7f1be0ba1640 1 Processor -- start 2024-03-28T23:34:55.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.610+0000 7f1be0ba1640 1 -- start start 2024-03-28T23:34:55.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.611+0000 7f1be0ba1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1bdc0c3390 0x7f1bdc0c3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:55.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.611+0000 7f1be0ba1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1bdc0a7d00 0x7f1bdc0c1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:55.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.611+0000 7f1be0ba1640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1bdc0a8160 0x7f1bdc0c2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:55.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.611+0000 7f1be0ba1640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1bdc155e80 con 0x7f1bdc0c3390 2024-03-28T23:34:55.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.611+0000 7f1be0ba1640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f1bdc155270 con 0x7f1bdc0a8160 2024-03-28T23:34:55.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.611+0000 7f1be0ba1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1bdc155550 con 0x7f1bdc0a7d00 2024-03-28T23:34:55.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.611+0000 7f1bda59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1bdc0c3390 0x7f1bdc0c3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:55.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.612+0000 7f1bda59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1bdc0c3390 0x7f1bdc0c3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59720/0 (socket says 172.21.15.161:59720) 2024-03-28T23:34:55.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.612+0000 7f1bda59c640 1 -- 172.21.15.161:0/31011049 learned_addr learned my addr 172.21.15.161:0/31011049 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:55.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.612+0000 7f1bda59c640 1 -- 172.21.15.161:0/31011049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1bdc0a7d00 msgr2=0x7f1bdc0c1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:55.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.612+0000 7f1bda59c640 1 --2- 172.21.15.161:0/31011049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1bdc0a7d00 0x7f1bdc0c1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:55.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.612+0000 7f1bda59c640 1 -- 172.21.15.161:0/31011049 >> v2:172.21.15.170:3300/0 conn(0x7f1bdc0a8160 msgr2=0x7f1bdc0c2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:55.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.612+0000 7f1bdad9d640 1 --2- 172.21.15.161:0/31011049 >> v2:172.21.15.170:3300/0 conn(0x7f1bdc0a8160 0x7f1bdc0c2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:55.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.612+0000 7f1bda59c640 1 --2- 172.21.15.161:0/31011049 >> v2:172.21.15.170:3300/0 conn(0x7f1bdc0a8160 0x7f1bdc0c2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:55.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.612+0000 7f1bda59c640 1 -- 172.21.15.161:0/31011049 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bdc155830 con 0x7f1bdc0c3390 2024-03-28T23:34:55.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.612+0000 7f1bdad9d640 1 --2- 172.21.15.161:0/31011049 >> v2:172.21.15.170:3300/0 conn(0x7f1bdc0a8160 0x7f1bdc0c2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:55.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.613+0000 7f1bda59c640 1 --2- 172.21.15.161:0/31011049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bdc0c3390 0x7f1bdc0c3770 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc072000 tx=0x7f1bc402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35753eb772fcc1ad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:55.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.613+0000 7f1bd959a640 1 -- 172.21.15.161:0/31011049 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc402e070 con 0x7f1bdc0c3390 2024-03-28T23:34:55.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.613+0000 7f1bd959a640 1 -- 172.21.15.161:0/31011049 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bc4037070 con 0x7f1bdc0c3390 2024-03-28T23:34:55.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.613+0000 7f1bd959a640 1 -- 172.21.15.161:0/31011049 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc4031a00 con 0x7f1bdc0c3390 2024-03-28T23:34:55.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.613+0000 7f1be0ba1640 1 -- 172.21.15.161:0/31011049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bdc0c3390 msgr2=0x7f1bdc0c3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:55.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.613+0000 7f1be0ba1640 1 --2- 172.21.15.161:0/31011049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bdc0c3390 0x7f1bdc0c3770 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc072000 tx=0x7f1bc402b880 comp rx=0 tx=0).stop 2024-03-28T23:34:55.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.613+0000 7f1be0ba1640 1 -- 172.21.15.161:0/31011049 shutdown_connections 2024-03-28T23:34:55.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.613+0000 7f1be0ba1640 1 --2- 172.21.15.161:0/31011049 >> v2:172.21.15.170:3300/0 conn(0x7f1bdc0a8160 0x7f1bdc0c2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:55.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.613+0000 7f1be0ba1640 1 --2- 172.21.15.161:0/31011049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1bdc0a7d00 0x7f1bdc0c1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:55.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.613+0000 7f1be0ba1640 1 --2- 172.21.15.161:0/31011049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bdc0c3390 0x7f1bdc0c3770 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:55.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.613+0000 7f1be0ba1640 1 -- 172.21.15.161:0/31011049 >> 172.21.15.161:0/31011049 conn(0x7f1bdc0aa740 msgr2=0x7f1bdc0aab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:55.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.613+0000 7f1be0ba1640 1 -- 172.21.15.161:0/31011049 shutdown_connections 2024-03-28T23:34:55.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.614+0000 7f1be0ba1640 1 -- 172.21.15.161:0/31011049 wait complete. 2024-03-28T23:34:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.615+0000 7f1be0ba1640 1 Processor -- start 2024-03-28T23:34:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.615+0000 7f1be0ba1640 1 -- start start 2024-03-28T23:34:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.615+0000 7f1be0ba1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bdc0a7d00 0x7f1bdc1ee030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.616+0000 7f1be0ba1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1bdc0a8160 0x7f1bdc1ee570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.616+0000 7f1be0ba1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bdc0c3390 0x7f1bdc158390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.616+0000 7f1be0ba1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1bdc152c00 con 0x7f1bdc0c3390 2024-03-28T23:34:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.616+0000 7f1be0ba1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1bdc152a80 con 0x7f1bdc0a7d00 2024-03-28T23:34:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.616+0000 7f1be0ba1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1bdc0c6970 con 0x7f1bdc0a8160 2024-03-28T23:34:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.616+0000 7f1bdad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bdc0c3390 0x7f1bdc158390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.616+0000 7f1bdad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bdc0c3390 0x7f1bdc158390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59736/0 (socket says 172.21.15.161:59736) 2024-03-28T23:34:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.616+0000 7f1bdad9d640 1 -- 172.21.15.161:0/1918184704 learned_addr learned my addr 172.21.15.161:0/1918184704 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.616+0000 7f1bda59c640 1 --2- 172.21.15.161:0/1918184704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bdc0a7d00 0x7f1bdc1ee030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.616+0000 7f1bd9d9b640 1 --2- 172.21.15.161:0/1918184704 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1bdc0a8160 0x7f1bdc1ee570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.616+0000 7f1bdad9d640 1 -- 172.21.15.161:0/1918184704 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1bdc0a8160 msgr2=0x7f1bdc1ee570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.616+0000 7f1bdad9d640 1 --2- 172.21.15.161:0/1918184704 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1bdc0a8160 0x7f1bdc1ee570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.616+0000 7f1bdad9d640 1 -- 172.21.15.161:0/1918184704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bdc0a7d00 msgr2=0x7f1bdc1ee030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.616+0000 7f1bdad9d640 1 --2- 172.21.15.161:0/1918184704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bdc0a7d00 0x7f1bdc1ee030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.616+0000 7f1bdad9d640 1 -- 172.21.15.161:0/1918184704 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bdc146e00 con 0x7f1bdc0c3390 2024-03-28T23:34:55.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.616+0000 7f1bdad9d640 1 --2- 172.21.15.161:0/1918184704 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bdc0c3390 0x7f1bdc158390 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f1bd400cea0 tx=0x7f1bd400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:55.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.616+0000 7f1bd2ffd640 1 -- 172.21.15.161:0/1918184704 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd4003b90 con 0x7f1bdc0c3390 2024-03-28T23:34:55.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.616+0000 7f1bd2ffd640 1 -- 172.21.15.161:0/1918184704 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bd4003d50 con 0x7f1bdc0c3390 2024-03-28T23:34:55.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.616+0000 7f1be0ba1640 1 -- 172.21.15.161:0/1918184704 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bdc0abf00 con 0x7f1bdc0c3390 2024-03-28T23:34:55.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.616+0000 7f1bd2ffd640 1 -- 172.21.15.161:0/1918184704 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd4004260 con 0x7f1bdc0c3390 2024-03-28T23:34:55.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.616+0000 7f1be0ba1640 1 -- 172.21.15.161:0/1918184704 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bdc0ac1a0 con 0x7f1bdc0c3390 2024-03-28T23:34:55.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.617+0000 7f1be0ba1640 1 -- 172.21.15.161:0/1918184704 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bdc158070 con 0x7f1bdc0c3390 2024-03-28T23:34:55.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.618+0000 7f1bd2ffd640 1 -- 172.21.15.161:0/1918184704 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1bd4004400 con 0x7f1bdc0c3390 2024-03-28T23:34:55.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.618+0000 7f1bd2ffd640 1 --2- 172.21.15.161:0/1918184704 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1bbc06b4d0 0x7f1bbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:55.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.618+0000 7f1bd2ffd640 1 -- 172.21.15.161:0/1918184704 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 6273+0+0 (secure 0 0 0) 0x7f1bd408ae50 con 0x7f1bdc0c3390 2024-03-28T23:34:55.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.618+0000 7f1bda59c640 1 --2- 172.21.15.161:0/1918184704 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1bbc06b4d0 0x7f1bbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:55.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.619+0000 7f1bda59c640 1 --2- 172.21.15.161:0/1918184704 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1bbc06b4d0 0x7f1bbc06d990 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4036000 tx=0x7f1bc403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:55.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.623+0000 7f1bd2ffd640 1 -- 172.21.15.161:0/1918184704 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1bdc158070 con 0x7f1bdc0c3390 2024-03-28T23:34:55.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.828+0000 7f1be0ba1640 1 -- 172.21.15.161:0/1918184704 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1bdc04f4b0 con 0x7f1bdc0c3390 2024-03-28T23:34:55.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.829+0000 7f1bd2ffd640 1 -- 172.21.15.161:0/1918184704 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) ==== 95+0+5192 (secure 0 0 0) 0x7f1bd4059a90 con 0x7f1bdc0c3390 2024-03-28T23:34:55.830 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:55.830 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":116,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7318,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3734750343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3734750343},{"type":"v1","addr":"172.21.15.170:6835","nonce":3734750343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/4140438862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":4140438862},{"type":"v1","addr":"172.21.15.170:6834","nonce":4140438862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":8220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/896089063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":896089063},{"type":"v1","addr":"172.21.15.161:6835","nonce":896089063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:34:36.796608+0000","modified":"2024-03-28T23:34:54.273158+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"a","rank":0,"incarnation":114,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.161:6837/3929102513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":3929102513},{"type":"v1","addr":"172.21.15.161:6837","nonce":3929102513}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":9}]} 2024-03-28T23:34:55.830 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 116 2024-03-28T23:34:55.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.831+0000 7f1be0ba1640 1 -- 172.21.15.161:0/1918184704 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1bbc06b4d0 msgr2=0x7f1bbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:55.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.831+0000 7f1be0ba1640 1 --2- 172.21.15.161:0/1918184704 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1bbc06b4d0 0x7f1bbc06d990 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4036000 tx=0x7f1bc403d040 comp rx=0 tx=0).stop 2024-03-28T23:34:55.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.831+0000 7f1be0ba1640 1 -- 172.21.15.161:0/1918184704 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bdc0c3390 msgr2=0x7f1bdc158390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:55.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.831+0000 7f1be0ba1640 1 --2- 172.21.15.161:0/1918184704 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bdc0c3390 0x7f1bdc158390 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f1bd400cea0 tx=0x7f1bd400cf90 comp rx=0 tx=0).stop 2024-03-28T23:34:55.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.831+0000 7f1be0ba1640 1 -- 172.21.15.161:0/1918184704 shutdown_connections 2024-03-28T23:34:55.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.831+0000 7f1be0ba1640 1 --2- 172.21.15.161:0/1918184704 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1bbc06b4d0 0x7f1bbc06d990 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:55.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.831+0000 7f1be0ba1640 1 --2- 172.21.15.161:0/1918184704 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bdc0c3390 0x7f1bdc158390 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:55.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.831+0000 7f1be0ba1640 1 --2- 172.21.15.161:0/1918184704 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1bdc0a8160 0x7f1bdc1ee570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:55.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.831+0000 7f1be0ba1640 1 --2- 172.21.15.161:0/1918184704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bdc0a7d00 0x7f1bdc1ee030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:55.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.831+0000 7f1be0ba1640 1 -- 172.21.15.161:0/1918184704 >> 172.21.15.161:0/1918184704 conn(0x7f1bdc0aa740 msgr2=0x7f1bdc152fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:55.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.831+0000 7f1be0ba1640 1 -- 172.21.15.161:0/1918184704 shutdown_connections 2024-03-28T23:34:55.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:55.831+0000 7f1be0ba1640 1 -- 172.21.15.161:0/1918184704 wait complete. 2024-03-28T23:34:56.014 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:34:56.016 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:34:56.245 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:34:36.796608+0000', 'modified': '2024-03-28T23:34:54.273158+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 124, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8199}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8199': {'gid': 8199, 'name': 'a', 'rank': 0, 'incarnation': 114, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.161:6837/3929102513', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 3929102513}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 3929102513}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-28T23:34:56.245 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8199:up:rejoin 2024-03-28T23:34:57.246 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:34:57.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.349+0000 7f09f6247640 1 Processor -- start 2024-03-28T23:34:57.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.349+0000 7f09f6247640 1 -- start start 2024-03-28T23:34:57.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.350+0000 7f09f6247640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f09f0152890 0x7f09f00c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:57.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.350+0000 7f09f6247640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09f00c8ed0 0x7f09f00c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:57.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.350+0000 7f09f6247640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f09f00c5950 0x7f09f01553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:57.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.350+0000 7f09f6247640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f09f00c8480 con 0x7f09f00c5950 2024-03-28T23:34:57.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.350+0000 7f09f6247640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f09f0159640 con 0x7f09f0152890 2024-03-28T23:34:57.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.350+0000 7f09f6247640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f09f0158a30 con 0x7f09f00c8ed0 2024-03-28T23:34:57.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.350+0000 7f09effff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f09f00c5950 0x7f09f01553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:57.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.350+0000 7f09effff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f09f00c5950 0x7f09f01553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59746/0 (socket says 172.21.15.161:59746) 2024-03-28T23:34:57.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.350+0000 7f09effff640 1 -- 172.21.15.161:0/2791094492 learned_addr learned my addr 172.21.15.161:0/2791094492 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:57.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.350+0000 7f09ef7fe640 1 --2- 172.21.15.161:0/2791094492 >> v2:172.21.15.170:3300/0 conn(0x7f09f0152890 0x7f09f00c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:57.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.350+0000 7f09eeffd640 1 --2- 172.21.15.161:0/2791094492 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09f00c8ed0 0x7f09f00c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:57.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.350+0000 7f09effff640 1 -- 172.21.15.161:0/2791094492 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09f00c8ed0 msgr2=0x7f09f00c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:57.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.350+0000 7f09effff640 1 --2- 172.21.15.161:0/2791094492 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09f00c8ed0 0x7f09f00c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:57.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.350+0000 7f09effff640 1 -- 172.21.15.161:0/2791094492 >> v2:172.21.15.170:3300/0 conn(0x7f09f0152890 msgr2=0x7f09f00c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:57.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.350+0000 7f09effff640 1 --2- 172.21.15.161:0/2791094492 >> v2:172.21.15.170:3300/0 conn(0x7f09f0152890 0x7f09f00c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:57.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.350+0000 7f09effff640 1 -- 172.21.15.161:0/2791094492 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f0158d10 con 0x7f09f00c5950 2024-03-28T23:34:57.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.351+0000 7f09effff640 1 --2- 172.21.15.161:0/2791094492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09f00c5950 0x7f09f01553a0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f09f008cc80 tx=0x7f09e802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3de5fbb6944a2ab1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:57.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.352+0000 7f09ee7fc640 1 -- 172.21.15.161:0/2791094492 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e802c950 con 0x7f09f00c5950 2024-03-28T23:34:57.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.352+0000 7f09ee7fc640 1 -- 172.21.15.161:0/2791094492 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09e802caf0 con 0x7f09f00c5950 2024-03-28T23:34:57.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.352+0000 7f09ee7fc640 1 -- 172.21.15.161:0/2791094492 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e80343f0 con 0x7f09f00c5950 2024-03-28T23:34:57.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.352+0000 7f09f6247640 1 -- 172.21.15.161:0/2791094492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09f00c5950 msgr2=0x7f09f01553a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:57.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.352+0000 7f09f6247640 1 --2- 172.21.15.161:0/2791094492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09f00c5950 0x7f09f01553a0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f09f008cc80 tx=0x7f09e802bdb0 comp rx=0 tx=0).stop 2024-03-28T23:34:57.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.352+0000 7f09f6247640 1 -- 172.21.15.161:0/2791094492 shutdown_connections 2024-03-28T23:34:57.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.352+0000 7f09f6247640 1 --2- 172.21.15.161:0/2791094492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09f00c5950 0x7f09f01553a0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:57.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.352+0000 7f09f6247640 1 --2- 172.21.15.161:0/2791094492 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09f00c8ed0 0x7f09f00c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:57.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.352+0000 7f09f6247640 1 --2- 172.21.15.161:0/2791094492 >> v2:172.21.15.170:3300/0 conn(0x7f09f0152890 0x7f09f00c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:57.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.352+0000 7f09f6247640 1 -- 172.21.15.161:0/2791094492 >> 172.21.15.161:0/2791094492 conn(0x7f09f00ac620 msgr2=0x7f09f00aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:57.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.352+0000 7f09f6247640 1 -- 172.21.15.161:0/2791094492 shutdown_connections 2024-03-28T23:34:57.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.352+0000 7f09f6247640 1 -- 172.21.15.161:0/2791094492 wait complete. 2024-03-28T23:34:57.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.353+0000 7f09f6247640 1 Processor -- start 2024-03-28T23:34:57.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.354+0000 7f09f6247640 1 -- start start 2024-03-28T23:34:57.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.354+0000 7f09f6247640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09f00c5950 0x7f09f0157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:57.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.354+0000 7f09f6247640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09f00c8ed0 0x7f09f00afd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:57.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.354+0000 7f09f6247640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09f0152890 0x7f09f00ae290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:57.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.354+0000 7f09f6247640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f09f0152c70 con 0x7f09f0152890 2024-03-28T23:34:57.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.354+0000 7f09f6247640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f09f01592d0 con 0x7f09f00c8ed0 2024-03-28T23:34:57.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.354+0000 7f09f6247640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f09f0152df0 con 0x7f09f00c5950 2024-03-28T23:34:57.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.354+0000 7f09effff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09f0152890 0x7f09f00ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:57.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.354+0000 7f09effff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09f0152890 0x7f09f00ae290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59748/0 (socket says 172.21.15.161:59748) 2024-03-28T23:34:57.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.354+0000 7f09effff640 1 -- 172.21.15.161:0/844459183 learned_addr learned my addr 172.21.15.161:0/844459183 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:57.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.355+0000 7f09ef7fe640 1 --2- 172.21.15.161:0/844459183 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09f00c5950 0x7f09f0157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:57.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.355+0000 7f09eeffd640 1 --2- 172.21.15.161:0/844459183 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09f00c8ed0 0x7f09f00afd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:57.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.355+0000 7f09effff640 1 -- 172.21.15.161:0/844459183 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09f00c5950 msgr2=0x7f09f0157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:57.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.355+0000 7f09effff640 1 --2- 172.21.15.161:0/844459183 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09f00c5950 0x7f09f0157ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:57.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.355+0000 7f09effff640 1 -- 172.21.15.161:0/844459183 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09f00c8ed0 msgr2=0x7f09f00afd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:57.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.355+0000 7f09effff640 1 --2- 172.21.15.161:0/844459183 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09f00c8ed0 0x7f09f00afd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:57.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.355+0000 7f09effff640 1 -- 172.21.15.161:0/844459183 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f00c1ff0 con 0x7f09f0152890 2024-03-28T23:34:57.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.355+0000 7f09ef7fe640 1 --2- 172.21.15.161:0/844459183 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09f00c5950 0x7f09f0157ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:57.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.355+0000 7f09effff640 1 --2- 172.21.15.161:0/844459183 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09f0152890 0x7f09f00ae290 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f09e8036040 tx=0x7f09e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:57.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.355+0000 7f09d7fff640 1 -- 172.21.15.161:0/844459183 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e8043020 con 0x7f09f0152890 2024-03-28T23:34:57.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.355+0000 7f09d7fff640 1 -- 172.21.15.161:0/844459183 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09e803e450 con 0x7f09f0152890 2024-03-28T23:34:57.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.355+0000 7f09d7fff640 1 -- 172.21.15.161:0/844459183 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e8009510 con 0x7f09f0152890 2024-03-28T23:34:57.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.355+0000 7f09f6247640 1 -- 172.21.15.161:0/844459183 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09f01eb520 con 0x7f09f0152890 2024-03-28T23:34:57.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.355+0000 7f09f6247640 1 -- 172.21.15.161:0/844459183 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09f01ebae0 con 0x7f09f0152890 2024-03-28T23:34:57.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.356+0000 7f09f6247640 1 -- 172.21.15.161:0/844459183 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09f01ebce0 con 0x7f09f0152890 2024-03-28T23:34:57.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.356+0000 7f09d7fff640 1 -- 172.21.15.161:0/844459183 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f09e8046020 con 0x7f09f0152890 2024-03-28T23:34:57.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.357+0000 7f09d7fff640 1 --2- 172.21.15.161:0/844459183 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f09dc06b4d0 0x7f09dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:57.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.357+0000 7f09d7fff640 1 -- 172.21.15.161:0/844459183 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 6273+0+0 (secure 0 0 0) 0x7f09e80aca30 con 0x7f09f0152890 2024-03-28T23:34:57.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.357+0000 7f09ef7fe640 1 --2- 172.21.15.161:0/844459183 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f09dc06b4d0 0x7f09dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:57.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.357+0000 7f09ef7fe640 1 --2- 172.21.15.161:0/844459183 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f09dc06b4d0 0x7f09dc06d990 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f09e4009ac0 tx=0x7f09e4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:57.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.361+0000 7f09d7fff640 1 -- 172.21.15.161:0/844459183 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f09f01ebce0 con 0x7f09f0152890 2024-03-28T23:34:57.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.721+0000 7f09f6247640 1 -- 172.21.15.161:0/844459183 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09f01eb310 con 0x7f09f0152890 2024-03-28T23:34:57.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.722+0000 7f09d7fff640 1 -- 172.21.15.161:0/844459183 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+5199 (secure 0 0 0) 0x7f09e804d030 con 0x7f09f0152890 2024-03-28T23:34:57.723 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:57.723 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":117,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7318,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3734750343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3734750343},{"type":"v1","addr":"172.21.15.170:6835","nonce":3734750343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/4140438862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":4140438862},{"type":"v1","addr":"172.21.15.170:6834","nonce":4140438862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":8220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/896089063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":896089063},{"type":"v1","addr":"172.21.15.161:6835","nonce":896089063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:34:36.796608+0000","modified":"2024-03-28T23:34:56.277306+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"a","rank":0,"incarnation":114,"state":"up:active","state_seq":4,"addr":"172.21.15.161:6837/3929102513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":3929102513},{"type":"v1","addr":"172.21.15.161:6837","nonce":3929102513}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8199,"qdb_cluster":[8199]},"id":9}]} 2024-03-28T23:34:57.723 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 117 2024-03-28T23:34:57.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.724+0000 7f09f6247640 1 -- 172.21.15.161:0/844459183 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f09dc06b4d0 msgr2=0x7f09dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:57.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.724+0000 7f09f6247640 1 --2- 172.21.15.161:0/844459183 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f09dc06b4d0 0x7f09dc06d990 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f09e4009ac0 tx=0x7f09e4008040 comp rx=0 tx=0).stop 2024-03-28T23:34:57.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.724+0000 7f09f6247640 1 -- 172.21.15.161:0/844459183 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09f0152890 msgr2=0x7f09f00ae290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:57.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.724+0000 7f09f6247640 1 --2- 172.21.15.161:0/844459183 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09f0152890 0x7f09f00ae290 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f09e8036040 tx=0x7f09e8004a40 comp rx=0 tx=0).stop 2024-03-28T23:34:57.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.724+0000 7f09f6247640 1 -- 172.21.15.161:0/844459183 shutdown_connections 2024-03-28T23:34:57.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.724+0000 7f09f6247640 1 --2- 172.21.15.161:0/844459183 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f09dc06b4d0 0x7f09dc06d990 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:57.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.724+0000 7f09f6247640 1 --2- 172.21.15.161:0/844459183 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09f0152890 0x7f09f00ae290 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:57.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.724+0000 7f09f6247640 1 --2- 172.21.15.161:0/844459183 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09f00c8ed0 0x7f09f00afd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:57.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.724+0000 7f09f6247640 1 --2- 172.21.15.161:0/844459183 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09f00c5950 0x7f09f0157ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:57.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.724+0000 7f09f6247640 1 -- 172.21.15.161:0/844459183 >> 172.21.15.161:0/844459183 conn(0x7f09f00ac620 msgr2=0x7f09f014a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:57.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.724+0000 7f09f6247640 1 -- 172.21.15.161:0/844459183 shutdown_connections 2024-03-28T23:34:57.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:57.724+0000 7f09f6247640 1 -- 172.21.15.161:0/844459183 wait complete. 2024-03-28T23:34:58.142 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:34:36.796608+0000', 'modified': '2024-03-28T23:34:56.277306+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 124, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8199}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8199': {'gid': 8199, 'name': 'a', 'rank': 0, 'incarnation': 114, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.161:6837/3929102513', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 3929102513}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 3929102513}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8199, 'qdb_cluster': [8199]} 2024-03-28T23:34:58.142 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:34:58.142 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-28T23:34:58.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.244+0000 7f01e7ad6640 1 Processor -- start 2024-03-28T23:34:58.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.244+0000 7f01e7ad6640 1 -- start start 2024-03-28T23:34:58.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.245+0000 7f01e7ad6640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f01e0152890 0x7f01e00c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.245+0000 7f01e7ad6640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f01e00c5b20 0x7f01e01553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.245+0000 7f01e7ad6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e0158a30 con 0x7f01e00c8ed0 2024-03-28T23:34:58.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.245+0000 7f01e7ad6640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f01e00c8480 con 0x7f01e00c5b20 2024-03-28T23:34:58.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.245+0000 7f01e7ad6640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f01e0159640 con 0x7f01e0152890 2024-03-28T23:34:58.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.245+0000 7f01e6073640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f01e00c5b20 0x7f01e01553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.245+0000 7f01e5071640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f01e00c8ed0 0x7f01e00c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44882/0 (socket says 172.21.15.161:44882) 2024-03-28T23:34:58.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.245+0000 7f01e5071640 1 -- 172.21.15.161:0/1250361291 learned_addr learned my addr 172.21.15.161:0/1250361291 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:58.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.245+0000 7f01e5872640 1 --2- 172.21.15.161:0/1250361291 >> v2:172.21.15.170:3300/0 conn(0x7f01e0152890 0x7f01e00c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.245+0000 7f01e6073640 1 -- 172.21.15.161:0/1250361291 >> v2:172.21.15.170:3300/0 conn(0x7f01e0152890 msgr2=0x7f01e00c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:58.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.245+0000 7f01e6073640 1 --2- 172.21.15.161:0/1250361291 >> v2:172.21.15.170:3300/0 conn(0x7f01e0152890 0x7f01e00c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.245+0000 7f01e6073640 1 -- 172.21.15.161:0/1250361291 >> v1:172.21.15.161:6789/0 conn(0x7f01e00c8ed0 legacy=0x7f01e00c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:58.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.245+0000 7f01e6073640 1 -- 172.21.15.161:0/1250361291 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e0158d10 con 0x7f01e00c5b20 2024-03-28T23:34:58.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.245+0000 7f01e4870640 1 -- 172.21.15.161:0/1250361291 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 477512265 0 0) 0x7f01e0158a30 con 0x7f01e00c8ed0 2024-03-28T23:34:58.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.247+0000 7f01e6073640 1 --2- 172.21.15.161:0/1250361291 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01e00c5b20 0x7f01e01553a0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f01e008cc80 tx=0x7f01d802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=221126cc1fd04136 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:58.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.247+0000 7f01e4870640 1 -- 172.21.15.161:0/1250361291 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d8039070 con 0x7f01e00c5b20 2024-03-28T23:34:58.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.247+0000 7f01e4870640 1 -- 172.21.15.161:0/1250361291 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01d802cb60 con 0x7f01e00c5b20 2024-03-28T23:34:58.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.247+0000 7f01e4870640 1 -- 172.21.15.161:0/1250361291 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d8034440 con 0x7f01e00c5b20 2024-03-28T23:34:58.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.247+0000 7f01e7ad6640 1 -- 172.21.15.161:0/1250361291 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01e00c5b20 msgr2=0x7f01e01553a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:58.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.247+0000 7f01e7ad6640 1 --2- 172.21.15.161:0/1250361291 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01e00c5b20 0x7f01e01553a0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f01e008cc80 tx=0x7f01d802bdb0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.247+0000 7f01e7ad6640 1 -- 172.21.15.161:0/1250361291 shutdown_connections 2024-03-28T23:34:58.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.247+0000 7f01e7ad6640 1 --2- 172.21.15.161:0/1250361291 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01e00c5b20 0x7f01e01553a0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.247+0000 7f01e7ad6640 1 --2- 172.21.15.161:0/1250361291 >> v2:172.21.15.170:3300/0 conn(0x7f01e0152890 0x7f01e00c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.247+0000 7f01e7ad6640 1 -- 172.21.15.161:0/1250361291 >> 172.21.15.161:0/1250361291 conn(0x7f01e00ac620 msgr2=0x7f01e00aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:58.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.248+0000 7f01e7ad6640 1 -- 172.21.15.161:0/1250361291 shutdown_connections 2024-03-28T23:34:58.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.248+0000 7f01e7ad6640 1 -- 172.21.15.161:0/1250361291 wait complete. 2024-03-28T23:34:58.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.249+0000 7f01e7ad6640 1 Processor -- start 2024-03-28T23:34:58.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.249+0000 7f01e7ad6640 1 -- start start 2024-03-28T23:34:58.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.249+0000 7f01e7ad6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01e00c5b20 0x7f01e00c0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.249+0000 7f01e7ad6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01e00c8ed0 0x7f01e0153c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.249+0000 7f01e7ad6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f01e0152890 0x7f01e0157640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.249+0000 7f01e7ad6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f01e0152c70 con 0x7f01e00c5b20 2024-03-28T23:34:58.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.249+0000 7f01e7ad6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f01e01592d0 con 0x7f01e00c8ed0 2024-03-28T23:34:58.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.249+0000 7f01e7ad6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f01e0152df0 con 0x7f01e0152890 2024-03-28T23:34:58.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.249+0000 7f01e5872640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01e00c5b20 0x7f01e00c0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.249+0000 7f01e5872640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01e00c5b20 0x7f01e00c0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59772/0 (socket says 172.21.15.161:59772) 2024-03-28T23:34:58.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.249+0000 7f01e5872640 1 -- 172.21.15.161:0/1840426658 learned_addr learned my addr 172.21.15.161:0/1840426658 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:58.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.250+0000 7f01e6073640 1 --2- 172.21.15.161:0/1840426658 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f01e0152890 0x7f01e0157640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.250+0000 7f01e5071640 1 --2- 172.21.15.161:0/1840426658 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01e00c8ed0 0x7f01e0153c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.250+0000 7f01e5872640 1 -- 172.21.15.161:0/1840426658 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f01e0152890 msgr2=0x7f01e0157640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:58.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.250+0000 7f01e5872640 1 --2- 172.21.15.161:0/1840426658 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f01e0152890 0x7f01e0157640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.250+0000 7f01e5872640 1 -- 172.21.15.161:0/1840426658 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01e00c8ed0 msgr2=0x7f01e0153c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:58.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.250+0000 7f01e5872640 1 --2- 172.21.15.161:0/1840426658 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01e00c8ed0 0x7f01e0153c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.250+0000 7f01e5872640 1 -- 172.21.15.161:0/1840426658 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e00c0ab0 con 0x7f01e00c5b20 2024-03-28T23:34:58.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.250+0000 7f01e6073640 1 --2- 172.21.15.161:0/1840426658 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f01e0152890 0x7f01e0157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:58.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.250+0000 7f01e5071640 1 --2- 172.21.15.161:0/1840426658 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01e00c8ed0 0x7f01e0153c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:34:58.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.250+0000 7f01e5872640 1 --2- 172.21.15.161:0/1840426658 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01e00c5b20 0x7f01e00c0570 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f01d000d620 tx=0x7f01d000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:58.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.250+0000 7f01de7fc640 1 -- 172.21.15.161:0/1840426658 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d000b620 con 0x7f01e00c5b20 2024-03-28T23:34:58.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.250+0000 7f01de7fc640 1 -- 172.21.15.161:0/1840426658 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01d0019070 con 0x7f01e00c5b20 2024-03-28T23:34:58.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.250+0000 7f01e7ad6640 1 -- 172.21.15.161:0/1840426658 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01e0155ae0 con 0x7f01e00c5b20 2024-03-28T23:34:58.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.250+0000 7f01de7fc640 1 -- 172.21.15.161:0/1840426658 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d000ba00 con 0x7f01e00c5b20 2024-03-28T23:34:58.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.250+0000 7f01e7ad6640 1 -- 172.21.15.161:0/1840426658 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01e01560a0 con 0x7f01e00c5b20 2024-03-28T23:34:58.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.252+0000 7f01de7fc640 1 -- 172.21.15.161:0/1840426658 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f01d000bba0 con 0x7f01e00c5b20 2024-03-28T23:34:58.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.252+0000 7f01de7fc640 1 --2- 172.21.15.161:0/1840426658 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f01cc06b4d0 0x7f01cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.252+0000 7f01de7fc640 1 -- 172.21.15.161:0/1840426658 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 6273+0+0 (secure 0 0 0) 0x7f01d008a7f0 con 0x7f01e00c5b20 2024-03-28T23:34:58.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.252+0000 7f01e5071640 1 --2- 172.21.15.161:0/1840426658 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f01cc06b4d0 0x7f01cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.253+0000 7f01e5071640 1 --2- 172.21.15.161:0/1840426658 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f01cc06b4d0 0x7f01cc06d990 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f01e014f440 tx=0x7f01d4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:58.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.253+0000 7f01ea78a740 1 Processor -- start 2024-03-28T23:34:58.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.253+0000 7f01ea78a740 1 -- start start 2024-03-28T23:34:58.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.254+0000 7f01ea78a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556c20d71db0 0x556c20d6f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.254+0000 7f01ea78a740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20d72190 0x556c20d6fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.254+0000 7f01ea78a740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556c20d70310 0x556c20d6c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.254+0000 7f01ea78a740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x556c20cd5af0 con 0x556c20d72190 2024-03-28T23:34:58.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.254+0000 7f01ea78a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x556c20c7e9d0 con 0x556c20d71db0 2024-03-28T23:34:58.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.254+0000 7f01ea78a740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x556c20cd1b90 con 0x556c20d70310 2024-03-28T23:34:58.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.254+0000 7f01e5071640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20d72190 0x556c20d6fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.254+0000 7f01e5071640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20d72190 0x556c20d6fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59778/0 (socket says 172.21.15.161:59778) 2024-03-28T23:34:58.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.254+0000 7f01e5071640 1 -- 172.21.15.161:0/1773245331 learned_addr learned my addr 172.21.15.161:0/1773245331 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:58.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.254+0000 7f01e5872640 1 --2- 172.21.15.161:0/1773245331 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556c20d71db0 0x556c20d6f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.254+0000 7f01e6073640 1 --2- 172.21.15.161:0/1773245331 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556c20d70310 0x556c20d6c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.254+0000 7f01e5071640 1 -- 172.21.15.161:0/1773245331 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556c20d70310 msgr2=0x556c20d6c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:58.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.254+0000 7f01e5071640 1 --2- 172.21.15.161:0/1773245331 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556c20d70310 0x556c20d6c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.254+0000 7f01e5071640 1 -- 172.21.15.161:0/1773245331 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556c20d71db0 msgr2=0x556c20d6f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:58.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.254+0000 7f01e5071640 1 --2- 172.21.15.161:0/1773245331 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556c20d71db0 0x556c20d6f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.254+0000 7f01e5071640 1 -- 172.21.15.161:0/1773245331 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c209ea2d0 con 0x556c20d72190 2024-03-28T23:34:58.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.254+0000 7f01e5872640 1 --2- 172.21.15.161:0/1773245331 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556c20d71db0 0x556c20d6f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:58.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.254+0000 7f01e5071640 1 --2- 172.21.15.161:0/1773245331 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20d72190 0x556c20d6fdd0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f01d4007360 tx=0x7f01d4010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58dd138ed0eb3616 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:58.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.255+0000 7f01c77fe640 1 -- 172.21.15.161:0/1773245331 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d400d9a0 con 0x556c20d72190 2024-03-28T23:34:58.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.255+0000 7f01c77fe640 1 -- 172.21.15.161:0/1773245331 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01d400db40 con 0x556c20d72190 2024-03-28T23:34:58.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.255+0000 7f01c77fe640 1 -- 172.21.15.161:0/1773245331 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d4020450 con 0x556c20d72190 2024-03-28T23:34:58.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.255+0000 7f01ea78a740 1 -- 172.21.15.161:0/1773245331 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20d72190 msgr2=0x556c20d6fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:58.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.255+0000 7f01ea78a740 1 --2- 172.21.15.161:0/1773245331 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20d72190 0x556c20d6fdd0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f01d4007360 tx=0x7f01d4010000 comp rx=0 tx=0).stop 2024-03-28T23:34:58.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.255+0000 7f01ea78a740 1 -- 172.21.15.161:0/1773245331 shutdown_connections 2024-03-28T23:34:58.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.255+0000 7f01ea78a740 1 --2- 172.21.15.161:0/1773245331 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20d72190 0x556c20d6fdd0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.255+0000 7f01ea78a740 1 --2- 172.21.15.161:0/1773245331 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556c20d70310 0x556c20d6c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.255+0000 7f01ea78a740 1 --2- 172.21.15.161:0/1773245331 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556c20d71db0 0x556c20d6f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.255+0000 7f01ea78a740 1 -- 172.21.15.161:0/1773245331 >> 172.21.15.161:0/1773245331 conn(0x556c20d6e630 msgr2=0x556c20d93220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:58.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.255+0000 7f01ea78a740 1 -- 172.21.15.161:0/1773245331 shutdown_connections 2024-03-28T23:34:58.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.255+0000 7f01ea78a740 1 -- 172.21.15.161:0/1773245331 wait complete. 2024-03-28T23:34:58.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.256+0000 7f01ea78a740 1 -- start start 2024-03-28T23:34:58.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.256+0000 7f01ea78a740 1 Processor -- start 2024-03-28T23:34:58.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.257+0000 7f01ea78a740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20d71db0 0x556c20d6c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.257+0000 7f01ea78a740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556c20d72190 0x556c20d95c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.257+0000 7f01ea78a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556c20da5c30 0x556c20d8fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.257+0000 7f01ea78a740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x556c20d94090 con 0x556c20d71db0 2024-03-28T23:34:58.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.257+0000 7f01ea78a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x556c20c91f70 con 0x556c20da5c30 2024-03-28T23:34:58.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.257+0000 7f01ea78a740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x556c20d93ad0 con 0x556c20d72190 2024-03-28T23:34:58.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.257+0000 7f01ea78a740 10 client.0 _ll_register_callbacks cb 0x556c209e9ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:34:58.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.257+0000 7f01e5872640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20d71db0 0x556c20d6c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.257+0000 7f01e5872640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20d71db0 0x556c20d6c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59788/0 (socket says 172.21.15.161:59788) 2024-03-28T23:34:58.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.257+0000 7f01e5872640 1 -- 172.21.15.161:0/273721295 learned_addr learned my addr 172.21.15.161:0/273721295 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:58.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.257+0000 7f01e5071640 1 --2- 172.21.15.161:0/273721295 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556c20d72190 0x556c20d95c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.257+0000 7f01e6073640 1 --2- 172.21.15.161:0/273721295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556c20da5c30 0x556c20d8fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.257+0000 7f01e5872640 1 -- 172.21.15.161:0/273721295 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556c20d72190 msgr2=0x556c20d95c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:58.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.257+0000 7f01e5872640 1 --2- 172.21.15.161:0/273721295 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556c20d72190 0x556c20d95c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.257+0000 7f01e5872640 1 -- 172.21.15.161:0/273721295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556c20da5c30 msgr2=0x556c20d8fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:58.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.257+0000 7f01e5872640 1 --2- 172.21.15.161:0/273721295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556c20da5c30 0x556c20d8fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.257+0000 7f01e5872640 1 -- 172.21.15.161:0/273721295 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c20ab46b0 con 0x556c20d71db0 2024-03-28T23:34:58.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.257+0000 7f01e5872640 1 --2- 172.21.15.161:0/273721295 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20d71db0 0x556c20d6c730 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f01d0058ac0 tx=0x7f01d0059600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdc6ccf5365c0162 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:58.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.257+0000 7f01c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:34:58.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.257+0000 7f01c4ff9640 1 -- 172.21.15.161:0/273721295 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d0087060 con 0x556c20d71db0 2024-03-28T23:34:58.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.258+0000 7f01c4ff9640 1 -- 172.21.15.161:0/273721295 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 6273+0+0 (secure 0 0 0) 0x7f01d0055430 con 0x556c20d71db0 2024-03-28T23:34:58.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.258+0000 7f01ea78a740 1 -- 172.21.15.161:0/273721295 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01d400db40 con 0x556c20d71db0 2024-03-28T23:34:58.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.258+0000 7f01c4ff9640 1 -- 172.21.15.161:0/273721295 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=117) ==== 24+0+0 (secure 0 0 0) 0x7f01d0070070 con 0x556c20d71db0 2024-03-28T23:34:58.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.258+0000 7f01ea78a740 10 client.8253 fetch_fsmap learned FSMap version 117 2024-03-28T23:34:58.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.258+0000 7f01ea78a740 1 -- 172.21.15.161:0/273721295 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=117,monmap=2+}) -- 0x556c20d961d0 con 0x556c20d71db0 2024-03-28T23:34:58.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.258+0000 7f01c4ff9640 1 -- 172.21.15.161:0/273721295 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01d400db40 con 0x556c20d71db0 2024-03-28T23:34:58.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.258+0000 7f01c4ff9640 1 -- 172.21.15.161:0/273721295 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 117) ==== 3494+0+0 (secure 0 0 0) 0x7f01d0013070 con 0x556c20d71db0 2024-03-28T23:34:58.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.258+0000 7f01ea78a740 10 client.8253 fetch_fsmap finished waiting for FSMap version 117 2024-03-28T23:34:58.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.258+0000 7f01ea78a740 10 client.8253 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T23:34:58.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.258+0000 7f01ea78a740 20 client.8253 populate_metadata read hostname 'smithi161' 2024-03-28T23:34:58.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.258+0000 7f01ea78a740 1 --2- 172.21.15.161:0/273721295 >> [v2:172.21.15.161:6836/3929102513,v1:172.21.15.161:6837/3929102513] conn(0x556c20dc2220 0x556c20dcd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.258+0000 7f01ea78a740 4 client.8253 mds_command: new command op to 8199 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:34:58.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.258+0000 7f01ea78a740 1 -- 172.21.15.161:0/273721295 --> [v2:172.21.15.161:6836/3929102513,v1:172.21.15.161:6837/3929102513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c20ce3030 con 0x556c20dc2220 2024-03-28T23:34:58.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.258+0000 7f01e5071640 1 --2- 172.21.15.161:0/273721295 >> [v2:172.21.15.161:6836/3929102513,v1:172.21.15.161:6837/3929102513] conn(0x556c20dc2220 0x556c20dcd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.258+0000 7f01c4ff9640 10 client.8253 ms_handle_connect on v2:172.21.15.161:6836/3929102513 2024-03-28T23:34:58.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.259+0000 7f01e5071640 1 --2- 172.21.15.161:0/273721295 >> [v2:172.21.15.161:6836/3929102513,v1:172.21.15.161:6837/3929102513] conn(0x556c20dc2220 0x556c20dcd000 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b598d0e3b118f66 server_cookie=f9008474bdf1fa5f in_seq=0 out_seq=0 2024-03-28T23:34:58.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.259+0000 7f01c4ff9640 1 -- 172.21.15.161:0/273721295 <== mds.0 v2:172.21.15.161:6836/3929102513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556c20ce3030 con 0x556c20dc2220 2024-03-28T23:34:58.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.259+0000 7f01c4ff9640 10 client.8253 handle_command_reply: tid=0 2024-03-28T23:34:58.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.259+0000 7f01ea78a740 1 client.8253 shutdown 2024-03-28T23:34:58.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.259+0000 7f01ea78a740 1 -- 172.21.15.161:0/273721295 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20d71db0 msgr2=0x556c20d6c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:58.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.259+0000 7f01ea78a740 1 --2- 172.21.15.161:0/273721295 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20d71db0 0x556c20d6c730 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f01d0058ac0 tx=0x7f01d0059600 comp rx=0 tx=0).stop 2024-03-28T23:34:58.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.259+0000 7f01ea78a740 1 -- 172.21.15.161:0/273721295 shutdown_connections 2024-03-28T23:34:58.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.259+0000 7f01ea78a740 1 --2- 172.21.15.161:0/273721295 >> [v2:172.21.15.161:6836/3929102513,v1:172.21.15.161:6837/3929102513] conn(0x556c20dc2220 0x556c20dcd000 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.259+0000 7f01ea78a740 1 --2- 172.21.15.161:0/273721295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556c20da5c30 0x556c20d8fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.259+0000 7f01c4ff9640 0 client.8253 ms_handle_reset on v2:172.21.15.161:6836/3929102513 2024-03-28T23:34:58.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.259+0000 7f01ea78a740 1 --2- 172.21.15.161:0/273721295 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556c20d72190 0x556c20d95c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.259+0000 7f01ea78a740 1 --2- 172.21.15.161:0/273721295 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20d71db0 0x556c20d6c730 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.259+0000 7f01ea78a740 1 -- 172.21.15.161:0/273721295 >> 172.21.15.161:0/273721295 conn(0x556c20d70310 msgr2=0x556c20dab620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:58.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.259+0000 7f01ea78a740 1 -- 172.21.15.161:0/273721295 shutdown_connections 2024-03-28T23:34:58.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.260+0000 7f01ea78a740 1 -- 172.21.15.161:0/273721295 wait complete. 2024-03-28T23:34:58.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.260+0000 7f01ea78a740 20 client.8253 trim_cache size 0 max 16384 2024-03-28T23:34:58.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.278+0000 7f01ea78a740 1 Processor -- start 2024-03-28T23:34:58.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.278+0000 7f01ea78a740 1 -- start start 2024-03-28T23:34:58.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.278+0000 7f01ea78a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556c20d71db0 0x556c20da5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.278+0000 7f01ea78a740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556c20d72190 0x556c20daaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.278+0000 7f01ea78a740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20da5c30 0x556c20dab430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.278+0000 7f01ea78a740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f01e01f6f60 con 0x556c20da5c30 2024-03-28T23:34:58.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.278+0000 7f01ea78a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f01e01eed30 con 0x556c20d71db0 2024-03-28T23:34:58.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.278+0000 7f01ea78a740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f01e01f6910 con 0x556c20d72190 2024-03-28T23:34:58.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.279+0000 7f01e6073640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20da5c30 0x556c20dab430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.279+0000 7f01e6073640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20da5c30 0x556c20dab430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59804/0 (socket says 172.21.15.161:59804) 2024-03-28T23:34:58.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.279+0000 7f01e6073640 1 -- 172.21.15.161:0/1121180010 learned_addr learned my addr 172.21.15.161:0/1121180010 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:58.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.279+0000 7f01e5872640 1 --2- 172.21.15.161:0/1121180010 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556c20d71db0 0x556c20da5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.279+0000 7f01e5071640 1 --2- 172.21.15.161:0/1121180010 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556c20d72190 0x556c20daaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.279+0000 7f01e6073640 1 -- 172.21.15.161:0/1121180010 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556c20d72190 msgr2=0x556c20daaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:58.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.279+0000 7f01e6073640 1 --2- 172.21.15.161:0/1121180010 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556c20d72190 0x556c20daaef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.279+0000 7f01e6073640 1 -- 172.21.15.161:0/1121180010 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556c20d71db0 msgr2=0x556c20da5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:58.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.279+0000 7f01e6073640 1 --2- 172.21.15.161:0/1121180010 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556c20d71db0 0x556c20da5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.279+0000 7f01e6073640 1 -- 172.21.15.161:0/1121180010 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c20dabf30 con 0x556c20da5c30 2024-03-28T23:34:58.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.279+0000 7f01e6073640 1 --2- 172.21.15.161:0/1121180010 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20da5c30 0x556c20dab430 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f01d8037040 tx=0x7f01d802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51e32fcd6d23547f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:58.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.281+0000 7f01c6ffd640 1 -- 172.21.15.161:0/1121180010 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d8039040 con 0x556c20da5c30 2024-03-28T23:34:58.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.281+0000 7f01c6ffd640 1 -- 172.21.15.161:0/1121180010 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01d8034440 con 0x556c20da5c30 2024-03-28T23:34:58.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.281+0000 7f01c6ffd640 1 -- 172.21.15.161:0/1121180010 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d8034830 con 0x556c20da5c30 2024-03-28T23:34:58.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.282+0000 7f01ea78a740 1 -- 172.21.15.161:0/1121180010 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20da5c30 msgr2=0x556c20dab430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:58.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.282+0000 7f01ea78a740 1 --2- 172.21.15.161:0/1121180010 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20da5c30 0x556c20dab430 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f01d8037040 tx=0x7f01d802e000 comp rx=0 tx=0).stop 2024-03-28T23:34:58.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.282+0000 7f01ea78a740 1 -- 172.21.15.161:0/1121180010 shutdown_connections 2024-03-28T23:34:58.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.282+0000 7f01ea78a740 1 --2- 172.21.15.161:0/1121180010 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20da5c30 0x556c20dab430 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.282+0000 7f01ea78a740 1 --2- 172.21.15.161:0/1121180010 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556c20d72190 0x556c20daaef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.282+0000 7f01ea78a740 1 --2- 172.21.15.161:0/1121180010 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556c20d71db0 0x556c20da5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.282+0000 7f01ea78a740 1 -- 172.21.15.161:0/1121180010 >> 172.21.15.161:0/1121180010 conn(0x556c20d70310 msgr2=0x556c20d67760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:58.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.282+0000 7f01ea78a740 1 -- 172.21.15.161:0/1121180010 shutdown_connections 2024-03-28T23:34:58.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.282+0000 7f01ea78a740 1 -- 172.21.15.161:0/1121180010 wait complete. 2024-03-28T23:34:58.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.282+0000 7f01ea78a740 1 -- start start 2024-03-28T23:34:58.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.282+0000 7f01ea78a740 1 Processor -- start 2024-03-28T23:34:58.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.284+0000 7f01ea78a740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20d71db0 0x556c20dd18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.284+0000 7f01ea78a740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556c20d72190 0x556c20e29570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.284+0000 7f01ea78a740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556c20da5c30 0x556c20daa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.284+0000 7f01ea78a740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x556c20da4ec0 con 0x556c20d71db0 2024-03-28T23:34:58.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.284+0000 7f01ea78a740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x556c20da6030 con 0x556c20da5c30 2024-03-28T23:34:58.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.284+0000 7f01ea78a740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x556c20dc1c90 con 0x556c20d72190 2024-03-28T23:34:58.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.284+0000 7f01ea78a740 10 client.0 _ll_register_callbacks cb 0x7f01d0058720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:34:58.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.284+0000 7f01e5872640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20d71db0 0x556c20dd18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.284+0000 7f01e5872640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20d71db0 0x556c20dd18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59816/0 (socket says 172.21.15.161:59816) 2024-03-28T23:34:58.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.284+0000 7f01e5872640 1 -- 172.21.15.161:0/4164132413 learned_addr learned my addr 172.21.15.161:0/4164132413 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:58.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.284+0000 7f01e5071640 1 --2- 172.21.15.161:0/4164132413 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556c20d72190 0x556c20e29570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.284+0000 7f01e6073640 1 --2- 172.21.15.161:0/4164132413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556c20da5c30 0x556c20daa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.284+0000 7f01e5872640 1 -- 172.21.15.161:0/4164132413 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556c20d72190 msgr2=0x556c20e29570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:58.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.284+0000 7f01e5872640 1 --2- 172.21.15.161:0/4164132413 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556c20d72190 0x556c20e29570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.284+0000 7f01e5872640 1 -- 172.21.15.161:0/4164132413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556c20da5c30 msgr2=0x556c20daa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:58.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.284+0000 7f01e5872640 1 --2- 172.21.15.161:0/4164132413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556c20da5c30 0x556c20daa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.284+0000 7f01e5872640 1 -- 172.21.15.161:0/4164132413 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c20dbc110 con 0x556c20d71db0 2024-03-28T23:34:58.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.284+0000 7f01e5071640 1 --2- 172.21.15.161:0/4164132413 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556c20d72190 0x556c20e29570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:58.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.284+0000 7f01e5872640 1 --2- 172.21.15.161:0/4164132413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20d71db0 0x556c20dd18c0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f01d0088bc0 tx=0x7f01d0088a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33c010893dfbdca1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:58.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.284+0000 7f01c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:34:58.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.284+0000 7f01c4ff9640 1 -- 172.21.15.161:0/4164132413 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d0087040 con 0x556c20d71db0 2024-03-28T23:34:58.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.284+0000 7f01c4ff9640 1 -- 172.21.15.161:0/4164132413 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 6273+0+0 (secure 0 0 0) 0x7f01d0070d50 con 0x556c20d71db0 2024-03-28T23:34:58.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.285+0000 7f01ea78a740 1 -- 172.21.15.161:0/4164132413 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01d8034440 con 0x556c20d71db0 2024-03-28T23:34:58.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.285+0000 7f01c4ff9640 1 -- 172.21.15.161:0/4164132413 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=117) ==== 24+0+0 (secure 0 0 0) 0x7f01d00612a0 con 0x556c20d71db0 2024-03-28T23:34:58.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.285+0000 7f01ea78a740 10 client.8259 fetch_fsmap learned FSMap version 117 2024-03-28T23:34:58.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.285+0000 7f01ea78a740 1 -- 172.21.15.161:0/4164132413 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=117,monmap=2+}) -- 0x556c20d6d540 con 0x556c20d71db0 2024-03-28T23:34:58.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.285+0000 7f01c4ff9640 1 -- 172.21.15.161:0/4164132413 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01d8034440 con 0x556c20d71db0 2024-03-28T23:34:58.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.285+0000 7f01c4ff9640 1 -- 172.21.15.161:0/4164132413 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 117) ==== 3494+0+0 (secure 0 0 0) 0x7f01d0082080 con 0x556c20d71db0 2024-03-28T23:34:58.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.285+0000 7f01ea78a740 10 client.8259 fetch_fsmap finished waiting for FSMap version 117 2024-03-28T23:34:58.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.285+0000 7f01ea78a740 10 client.8259 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T23:34:58.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.285+0000 7f01ea78a740 20 client.8259 populate_metadata read hostname 'smithi161' 2024-03-28T23:34:58.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.285+0000 7f01ea78a740 1 --2- 172.21.15.161:0/4164132413 >> [v2:172.21.15.161:6836/3929102513,v1:172.21.15.161:6837/3929102513] conn(0x556c20e26190 0x556c20e2c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.285+0000 7f01ea78a740 4 client.8259 mds_command: new command op to 8199 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:34:58.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.285+0000 7f01ea78a740 1 -- 172.21.15.161:0/4164132413 --> [v2:172.21.15.161:6836/3929102513,v1:172.21.15.161:6837/3929102513] -- command(tid 0: {"prefix": "status"}) -- 0x556c20cdd380 con 0x556c20e26190 2024-03-28T23:34:58.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.285+0000 7f01e5071640 1 --2- 172.21.15.161:0/4164132413 >> [v2:172.21.15.161:6836/3929102513,v1:172.21.15.161:6837/3929102513] conn(0x556c20e26190 0x556c20e2c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.285+0000 7f01c4ff9640 10 client.8259 ms_handle_connect on v2:172.21.15.161:6836/3929102513 2024-03-28T23:34:58.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.286+0000 7f01e5071640 1 --2- 172.21.15.161:0/4164132413 >> [v2:172.21.15.161:6836/3929102513,v1:172.21.15.161:6837/3929102513] conn(0x556c20e26190 0x556c20e2c060 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d86c077e5713548d server_cookie=4a29cc2dd391a052 in_seq=0 out_seq=0 2024-03-28T23:34:58.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.286+0000 7f01c4ff9640 1 -- 172.21.15.161:0/4164132413 <== mds.0 v2:172.21.15.161:6836/3929102513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x556c20cdd380 con 0x556c20e26190 2024-03-28T23:34:58.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.286+0000 7f01c4ff9640 10 client.8259 handle_command_reply: tid=0 2024-03-28T23:34:58.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.286+0000 7f01ea78a740 1 client.8259 shutdown 2024-03-28T23:34:58.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.286+0000 7f01ea78a740 1 -- 172.21.15.161:0/4164132413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20d71db0 msgr2=0x556c20dd18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:58.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.286+0000 7f01ea78a740 1 --2- 172.21.15.161:0/4164132413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20d71db0 0x556c20dd18c0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f01d0088bc0 tx=0x7f01d0088a40 comp rx=0 tx=0).stop 2024-03-28T23:34:58.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.286+0000 7f01ea78a740 1 -- 172.21.15.161:0/4164132413 shutdown_connections 2024-03-28T23:34:58.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.286+0000 7f01ea78a740 1 --2- 172.21.15.161:0/4164132413 >> [v2:172.21.15.161:6836/3929102513,v1:172.21.15.161:6837/3929102513] conn(0x556c20e26190 0x556c20e2c060 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.286+0000 7f01ea78a740 1 --2- 172.21.15.161:0/4164132413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556c20da5c30 0x556c20daa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.286+0000 7f01ea78a740 1 --2- 172.21.15.161:0/4164132413 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x556c20d72190 0x556c20e29570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.286+0000 7f01ea78a740 1 --2- 172.21.15.161:0/4164132413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x556c20d71db0 0x556c20dd18c0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.286+0000 7f01ea78a740 1 -- 172.21.15.161:0/4164132413 >> 172.21.15.161:0/4164132413 conn(0x556c20d70310 msgr2=0x556c20da5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:58.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.286+0000 7f01c4ff9640 0 client.8259 ms_handle_reset on v2:172.21.15.161:6836/3929102513 2024-03-28T23:34:58.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.286+0000 7f01ea78a740 1 -- 172.21.15.161:0/4164132413 shutdown_connections 2024-03-28T23:34:58.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.286+0000 7f01ea78a740 1 -- 172.21.15.161:0/4164132413 wait complete. 2024-03-28T23:34:58.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.286+0000 7f01ea78a740 20 client.8259 trim_cache size 0 max 16384 2024-03-28T23:34:58.292 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:34:58.292 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:34:58.292 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:34:58.292 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 8199, 2024-03-28T23:34:58.292 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:34:58.292 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:34:58.292 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:34:58.292 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 5.0183149309999999, 2024-03-28T23:34:58.292 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 117, 2024-03-28T23:34:58.292 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 124, 2024-03-28T23:34:58.292 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 124, 2024-03-28T23:34:58.293 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 8.0070824199999997 2024-03-28T23:34:58.293 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:34:58.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.287+0000 7f01e7ad6640 1 -- 172.21.15.161:0/1840426658 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f01cc06b4d0 msgr2=0x7f01cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:58.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.287+0000 7f01e7ad6640 1 --2- 172.21.15.161:0/1840426658 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f01cc06b4d0 0x7f01cc06d990 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f01e014f440 tx=0x7f01d4009040 comp rx=0 tx=0).stop 2024-03-28T23:34:58.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.287+0000 7f01e7ad6640 1 -- 172.21.15.161:0/1840426658 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01e00c5b20 msgr2=0x7f01e00c0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:58.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.287+0000 7f01e7ad6640 1 --2- 172.21.15.161:0/1840426658 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01e00c5b20 0x7f01e00c0570 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f01d000d620 tx=0x7f01d000dae0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.287+0000 7f01e7ad6640 1 -- 172.21.15.161:0/1840426658 shutdown_connections 2024-03-28T23:34:58.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.287+0000 7f01e7ad6640 1 --2- 172.21.15.161:0/1840426658 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f01cc06b4d0 0x7f01cc06d990 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.287+0000 7f01e7ad6640 1 --2- 172.21.15.161:0/1840426658 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f01e0152890 0x7f01e0157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.287+0000 7f01e7ad6640 1 --2- 172.21.15.161:0/1840426658 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01e00c8ed0 0x7f01e0153c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.287+0000 7f01e7ad6640 1 --2- 172.21.15.161:0/1840426658 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01e00c5b20 0x7f01e00c0570 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.287+0000 7f01e7ad6640 1 -- 172.21.15.161:0/1840426658 >> 172.21.15.161:0/1840426658 conn(0x7f01e00ac620 msgr2=0x7f01e00c7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:58.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.287+0000 7f01e7ad6640 1 -- 172.21.15.161:0/1840426658 shutdown_connections 2024-03-28T23:34:58.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.287+0000 7f01e7ad6640 1 -- 172.21.15.161:0/1840426658 wait complete. 2024-03-28T23:34:58.298 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 117, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:34:36.796608+0000", "damaged": [], "data_pools": [ 20 ], "enabled": true, "epoch": 117, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8199": { "addr": "172.21.15.161:6837/3929102513", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6836", "nonce": 3929102513, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 3929102513, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8199, "incarnation": 114, "join_fscid": 9, "name": "a", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 124, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 19, "modified": "2024-03-28T23:34:56.277306+0000", "qdb_cluster": [ 8199 ], "qdb_leader": 8199, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8199 } } } ], "standbys": [ { "addr": "172.21.15.170:6835/3734750343", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 3734750343, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 3734750343, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 103, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7318, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.170:6834/4140438862", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 4140438862, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 4140438862, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 103, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7321, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6835/896089063", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 896089063, "type": "v2" }, { "addr": "172.21.15.161:6835", "nonce": 896089063, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 115, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8220, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:34:58.299 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': 9, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-28T23:34:58.299 INFO:tasks.cephfs.test_failover:current = [{'gid': 8199, 'name': 'a', 'rank': 0, 'incarnation': 114, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.161:6837/3929102513', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 3929102513}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 3929102513}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 7318, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/3734750343', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 3734750343}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 3734750343}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 103}, {'gid': 8220, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6835/896089063', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 896089063}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 896089063}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 115}, {'gid': 7321, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6834/4140438862', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 4140438862}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 4140438862}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 103}] 2024-03-28T23:34:58.300 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-28T23:34:58.300 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-28T23:34:58.300 DEBUG:teuthology.orchestra.run.smithi161:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:34:58.338 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:34:58.381 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-28T23:34:58.381 INFO:tasks.cephfs.mount:Terminating background process 2024-03-28T23:34:58.381 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:34:58.381 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:34:58.396 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20793 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-28T23:34:58.396 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi161.front.sepia.ceph.com 2024-03-28T23:34:58.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:34:58.396 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-28T23:34:58.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:34:58.451 INFO:teuthology.orchestra.run.smithi161.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-28T23:34:58.451 INFO:tasks.cephfs.mount:Terminating background process 2024-03-28T23:34:58.451 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:34:58.452 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:34:58.470 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056374,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19690 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-28T23:34:58.470 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi170.front.sepia.ceph.com 2024-03-28T23:34:58.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:34:58.471 DEBUG:teuthology.orchestra.run.smithi170:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-28T23:34:58.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:34:58.528 INFO:teuthology.orchestra.run.smithi170.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-28T23:34:58.529 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:34:58.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.630+0000 7fb2fbf8c640 1 Processor -- start 2024-03-28T23:34:58.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.630+0000 7fb2fbf8c640 1 -- start start 2024-03-28T23:34:58.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.631+0000 7fb2fbf8c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2f4155d10 0x7fb2f40c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.631+0000 7fb2fbf8c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2f406c3c0 con 0x7fb2f4152a80 2024-03-28T23:34:58.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.631+0000 7fb2fbf8c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2f40c5b90 con 0x7fb2f406d9f0 2024-03-28T23:34:58.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.631+0000 7fb2fbf8c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb2f406c6f0 con 0x7fb2f4155d10 2024-03-28T23:34:58.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.631+0000 7fb2f9d28640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb2f4152a80 0x7fb2f40c93a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44890/0 (socket says 172.21.15.161:44890) 2024-03-28T23:34:58.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.631+0000 7fb2f9d28640 1 -- 172.21.15.161:0/99130572 learned_addr learned my addr 172.21.15.161:0/99130572 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:58.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2f8d26640 1 -- 172.21.15.161:0/99130572 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 153205657 0 0) 0x7fb2f406c3c0 con 0x7fb2f4152a80 2024-03-28T23:34:58.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2fa529640 1 --2- 172.21.15.161:0/99130572 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2f4155d10 0x7fb2f40c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2f8d26640 1 -- 172.21.15.161:0/99130572 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2e0003520 con 0x7fb2f4152a80 2024-03-28T23:34:58.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2f8d26640 1 -- 172.21.15.161:0/99130572 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3120104220 0 0) 0x7fb2e0003520 con 0x7fb2f4152a80 2024-03-28T23:34:58.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2f8d26640 1 -- 172.21.15.161:0/99130572 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2f4155d10 msgr2=0x7fb2f40c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:58.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2f8d26640 1 --2- 172.21.15.161:0/99130572 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2f4155d10 0x7fb2f40c74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2f8d26640 1 -- 172.21.15.161:0/99130572 >> v1:172.21.15.170:6789/0 conn(0x7fb2f406d9f0 legacy=0x7fb2f406cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:58.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2f8d26640 1 -- 172.21.15.161:0/99130572 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2f40c55d0 con 0x7fb2f4152a80 2024-03-28T23:34:58.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2f8d26640 1 -- 172.21.15.161:0/99130572 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fb2e4001670 con 0x7fb2f4152a80 2024-03-28T23:34:58.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2f8d26640 1 -- 172.21.15.161:0/99130572 >> v1:172.21.15.161:6789/0 conn(0x7fb2f4152a80 legacy=0x7fb2f40c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:58.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2f8d26640 1 --2- 172.21.15.161:0/99130572 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2e0003ac0 0x7fb2e0003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2f8d26640 1 --2- 172.21.15.161:0/99130572 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2f4155d10 0x7fb2e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2f8d26640 1 --2- 172.21.15.161:0/99130572 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2e0004f40 0x7fb2e000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2f8d26640 1 -- 172.21.15.161:0/99130572 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f406c6f0 con 0x7fb2e0003ac0 2024-03-28T23:34:58.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2f8d26640 1 -- 172.21.15.161:0/99130572 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e0005670 con 0x7fb2e0004f40 2024-03-28T23:34:58.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2f8d26640 1 -- 172.21.15.161:0/99130572 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb2e000bf80 con 0x7fb2f4155d10 2024-03-28T23:34:58.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2f9d28640 1 --2- 172.21.15.161:0/99130572 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2e0003ac0 0x7fb2e0003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2f9527640 1 --2- 172.21.15.161:0/99130572 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2f4155d10 0x7fb2e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2fa529640 1 --2- 172.21.15.161:0/99130572 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2e0004f40 0x7fb2e000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2f9d28640 1 -- 172.21.15.161:0/99130572 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2f4155d10 msgr2=0x7fb2e0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:58.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2f9d28640 1 --2- 172.21.15.161:0/99130572 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2f4155d10 0x7fb2e0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2f9d28640 1 -- 172.21.15.161:0/99130572 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2e0004f40 msgr2=0x7fb2e000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:58.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2f9d28640 1 --2- 172.21.15.161:0/99130572 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2e0004f40 0x7fb2e000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2f9d28640 1 -- 172.21.15.161:0/99130572 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2e0003520 con 0x7fb2e0003ac0 2024-03-28T23:34:58.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.632+0000 7fb2fa529640 1 --2- 172.21.15.161:0/99130572 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2e0004f40 0x7fb2e000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:58.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.633+0000 7fb2f9d28640 1 --2- 172.21.15.161:0/99130572 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2e0003ac0 0x7fb2e0003f00 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fb2e400c210 tx=0x7fb2e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d25fc0f5cf937b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:58.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.633+0000 7fb2f8d26640 1 -- 172.21.15.161:0/99130572 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e4007d10 con 0x7fb2e0003ac0 2024-03-28T23:34:58.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.633+0000 7fb2f8d26640 1 -- 172.21.15.161:0/99130572 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2e4030070 con 0x7fb2e0003ac0 2024-03-28T23:34:58.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.633+0000 7fb2f8d26640 1 -- 172.21.15.161:0/99130572 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e4005d30 con 0x7fb2e0003ac0 2024-03-28T23:34:58.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.634+0000 7fb2fbf8c640 1 -- 172.21.15.161:0/99130572 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2e0003ac0 msgr2=0x7fb2e0003f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:58.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.634+0000 7fb2fbf8c640 1 --2- 172.21.15.161:0/99130572 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2e0003ac0 0x7fb2e0003f00 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fb2e400c210 tx=0x7fb2e0005830 comp rx=0 tx=0).stop 2024-03-28T23:34:58.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.634+0000 7fb2fa529640 1 -- 172.21.15.161:0/99130572 reap_dead start 2024-03-28T23:34:58.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.634+0000 7fb2fbf8c640 1 -- 172.21.15.161:0/99130572 shutdown_connections 2024-03-28T23:34:58.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.634+0000 7fb2fbf8c640 1 -- 172.21.15.161:0/99130572 >> 172.21.15.161:0/99130572 conn(0x7fb2f40ac720 msgr2=0x7fb2f40acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:58.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.634+0000 7fb2fbf8c640 1 -- 172.21.15.161:0/99130572 shutdown_connections 2024-03-28T23:34:58.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.634+0000 7fb2fbf8c640 1 -- 172.21.15.161:0/99130572 wait complete. 2024-03-28T23:34:58.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.635+0000 7fb2fbf8c640 1 Processor -- start 2024-03-28T23:34:58.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.635+0000 7fb2fbf8c640 1 -- start start 2024-03-28T23:34:58.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.636+0000 7fb2fbf8c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2e0003ac0 0x7fb2f40c09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.636+0000 7fb2fbf8c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2f40c0f20 0x7fb2f414c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.636+0000 7fb2fbf8c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2f4156600 0x7fb2f414c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.636+0000 7fb2fbf8c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f40c74c0 con 0x7fb2e0003ac0 2024-03-28T23:34:58.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.636+0000 7fb2fbf8c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f41561a0 con 0x7fb2f4156600 2024-03-28T23:34:58.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.636+0000 7fb2fbf8c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb2f40c7640 con 0x7fb2f40c0f20 2024-03-28T23:34:58.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.636+0000 7fb2f9d28640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2e0003ac0 0x7fb2f40c09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.636+0000 7fb2f9d28640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2e0003ac0 0x7fb2f40c09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59838/0 (socket says 172.21.15.161:59838) 2024-03-28T23:34:58.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.636+0000 7fb2f9d28640 1 -- 172.21.15.161:0/2136724841 learned_addr learned my addr 172.21.15.161:0/2136724841 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:58.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.636+0000 7fb2f9527640 1 --2- 172.21.15.161:0/2136724841 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2f40c0f20 0x7fb2f414c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.636+0000 7fb2fa529640 1 --2- 172.21.15.161:0/2136724841 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2f4156600 0x7fb2f414c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.636+0000 7fb2f9d28640 1 -- 172.21.15.161:0/2136724841 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2f40c0f20 msgr2=0x7fb2f414c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:58.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.636+0000 7fb2f9d28640 1 --2- 172.21.15.161:0/2136724841 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2f40c0f20 0x7fb2f414c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.636+0000 7fb2f9d28640 1 -- 172.21.15.161:0/2136724841 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2f4156600 msgr2=0x7fb2f414c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:58.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.636+0000 7fb2f9d28640 1 --2- 172.21.15.161:0/2136724841 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2f4156600 0x7fb2f414c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.636+0000 7fb2f9d28640 1 -- 172.21.15.161:0/2136724841 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2f41eaed0 con 0x7fb2e0003ac0 2024-03-28T23:34:58.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.636+0000 7fb2fa529640 1 --2- 172.21.15.161:0/2136724841 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2f4156600 0x7fb2f414c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:58.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.636+0000 7fb2f9527640 1 --2- 172.21.15.161:0/2136724841 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2f40c0f20 0x7fb2f414c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:58.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.636+0000 7fb2f9d28640 1 --2- 172.21.15.161:0/2136724841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2e0003ac0 0x7fb2f40c09e0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fb2e402c540 tx=0x7fb2f4155290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:58.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.636+0000 7fb2ea7fc640 1 -- 172.21.15.161:0/2136724841 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e4045070 con 0x7fb2e0003ac0 2024-03-28T23:34:58.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.636+0000 7fb2ea7fc640 1 -- 172.21.15.161:0/2136724841 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2e4001080 con 0x7fb2e0003ac0 2024-03-28T23:34:58.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.636+0000 7fb2fbf8c640 1 -- 172.21.15.161:0/2136724841 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2f40ac3a0 con 0x7fb2e0003ac0 2024-03-28T23:34:58.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.636+0000 7fb2ea7fc640 1 -- 172.21.15.161:0/2136724841 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e4001c70 con 0x7fb2e0003ac0 2024-03-28T23:34:58.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.636+0000 7fb2fbf8c640 1 -- 172.21.15.161:0/2136724841 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2f41eb470 con 0x7fb2e0003ac0 2024-03-28T23:34:58.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.638+0000 7fb2fbf8c640 1 -- 172.21.15.161:0/2136724841 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2f404f4b0 con 0x7fb2e0003ac0 2024-03-28T23:34:58.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.638+0000 7fb2ea7fc640 1 -- 172.21.15.161:0/2136724841 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb2e4007070 con 0x7fb2e0003ac0 2024-03-28T23:34:58.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.638+0000 7fb2ea7fc640 1 --2- 172.21.15.161:0/2136724841 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb2d806b4d0 0x7fb2d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.638+0000 7fb2ea7fc640 1 -- 172.21.15.161:0/2136724841 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 6273+0+0 (secure 0 0 0) 0x7fb2e40ab110 con 0x7fb2e0003ac0 2024-03-28T23:34:58.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.638+0000 7fb2f9527640 1 --2- 172.21.15.161:0/2136724841 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb2d806b4d0 0x7fb2d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.639+0000 7fb2f9527640 1 --2- 172.21.15.161:0/2136724841 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb2d806b4d0 0x7fb2d806d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fb2f414e450 tx=0x7fb2f0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:58.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.646+0000 7fb2ea7fc640 1 -- 172.21.15.161:0/2136724841 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb2e4079870 con 0x7fb2e0003ac0 2024-03-28T23:34:58.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.861+0000 7fb2fbf8c640 1 -- 172.21.15.161:0/2136724841 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2f40b0a00 con 0x7fb2e0003ac0 2024-03-28T23:34:58.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.862+0000 7fb2ea7fc640 1 -- 172.21.15.161:0/2136724841 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+5199 (secure 0 0 0) 0x7fb2e407d910 con 0x7fb2e0003ac0 2024-03-28T23:34:58.863 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:58.863 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":117,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7318,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3734750343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3734750343},{"type":"v1","addr":"172.21.15.170:6835","nonce":3734750343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/4140438862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":4140438862},{"type":"v1","addr":"172.21.15.170:6834","nonce":4140438862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":8220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/896089063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":896089063},{"type":"v1","addr":"172.21.15.161:6835","nonce":896089063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:34:36.796608+0000","modified":"2024-03-28T23:34:56.277306+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"a","rank":0,"incarnation":114,"state":"up:active","state_seq":4,"addr":"172.21.15.161:6837/3929102513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":3929102513},{"type":"v1","addr":"172.21.15.161:6837","nonce":3929102513}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8199,"qdb_cluster":[8199]},"id":9}]} 2024-03-28T23:34:58.863 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 117 2024-03-28T23:34:58.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.864+0000 7fb2fbf8c640 1 -- 172.21.15.161:0/2136724841 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb2d806b4d0 msgr2=0x7fb2d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:58.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.864+0000 7fb2fbf8c640 1 --2- 172.21.15.161:0/2136724841 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb2d806b4d0 0x7fb2d806d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fb2f414e450 tx=0x7fb2f0009040 comp rx=0 tx=0).stop 2024-03-28T23:34:58.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.864+0000 7fb2fbf8c640 1 -- 172.21.15.161:0/2136724841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2e0003ac0 msgr2=0x7fb2f40c09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:58.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.864+0000 7fb2fbf8c640 1 --2- 172.21.15.161:0/2136724841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2e0003ac0 0x7fb2f40c09e0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fb2e402c540 tx=0x7fb2f4155290 comp rx=0 tx=0).stop 2024-03-28T23:34:58.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.864+0000 7fb2fbf8c640 1 -- 172.21.15.161:0/2136724841 shutdown_connections 2024-03-28T23:34:58.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.864+0000 7fb2fbf8c640 1 --2- 172.21.15.161:0/2136724841 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb2d806b4d0 0x7fb2d806d990 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.864+0000 7fb2fbf8c640 1 --2- 172.21.15.161:0/2136724841 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2f4156600 0x7fb2f414c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.864+0000 7fb2fbf8c640 1 --2- 172.21.15.161:0/2136724841 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2f40c0f20 0x7fb2f414c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.864+0000 7fb2fbf8c640 1 --2- 172.21.15.161:0/2136724841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2e0003ac0 0x7fb2f40c09e0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.864+0000 7fb2fbf8c640 1 -- 172.21.15.161:0/2136724841 >> 172.21.15.161:0/2136724841 conn(0x7fb2f40ac720 msgr2=0x7fb2f414ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:58.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.864+0000 7fb2fbf8c640 1 -- 172.21.15.161:0/2136724841 shutdown_connections 2024-03-28T23:34:58.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.864+0000 7fb2fbf8c640 1 -- 172.21.15.161:0/2136724841 wait complete. 2024-03-28T23:34:58.876 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:34:58.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.977+0000 7fc8c5b65640 1 Processor -- start 2024-03-28T23:34:58.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.977+0000 7fc8c5b65640 1 -- start start 2024-03-28T23:34:58.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.978+0000 7fc8c5b65640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8c00c5850 0x7fc8c0155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.978+0000 7fc8c5b65640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8c00c5d00 con 0x7fc8c00c5410 2024-03-28T23:34:58.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.978+0000 7fc8c5b65640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8c0158a30 con 0x7fc8c0152890 2024-03-28T23:34:58.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.978+0000 7fc8c5b65640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc8c00c8480 con 0x7fc8c00c5850 2024-03-28T23:34:58.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.978+0000 7fc8b6ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc8c00c5410 0x7fc8c01553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44904/0 (socket says 172.21.15.161:44904) 2024-03-28T23:34:58.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.978+0000 7fc8b6ffd640 1 -- 172.21.15.161:0/956289842 learned_addr learned my addr 172.21.15.161:0/956289842 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:58.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.979+0000 7fc8bffff640 1 --2- 172.21.15.161:0/956289842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8c00c5850 0x7fc8c0155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.979+0000 7fc8beffd640 1 -- 172.21.15.161:0/956289842 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1348734058 0 0) 0x7fc8c00c5d00 con 0x7fc8c00c5410 2024-03-28T23:34:58.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.979+0000 7fc8beffd640 1 -- 172.21.15.161:0/956289842 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8ac003520 con 0x7fc8c00c5410 2024-03-28T23:34:58.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.979+0000 7fc8beffd640 1 -- 172.21.15.161:0/956289842 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3501835395 0 0) 0x7fc8c0158a30 con 0x7fc8c0152890 2024-03-28T23:34:58.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.979+0000 7fc8beffd640 1 -- 172.21.15.161:0/956289842 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8c00c5d00 con 0x7fc8c0152890 2024-03-28T23:34:58.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.979+0000 7fc8bffff640 1 -- 172.21.15.161:0/956289842 >> v1:172.21.15.170:6789/0 conn(0x7fc8c0152890 legacy=0x7fc8c00c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:58.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.979+0000 7fc8beffd640 1 -- 172.21.15.161:0/956289842 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3762501239 0 0) 0x7fc8ac003520 con 0x7fc8c00c5410 2024-03-28T23:34:58.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.979+0000 7fc8bffff640 1 -- 172.21.15.161:0/956289842 >> v1:172.21.15.161:6789/0 conn(0x7fc8c00c5410 legacy=0x7fc8c01553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:58.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.979+0000 7fc8bffff640 1 -- 172.21.15.161:0/956289842 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8c0158d10 con 0x7fc8c00c5850 2024-03-28T23:34:58.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.980+0000 7fc8bffff640 1 --2- 172.21.15.161:0/956289842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8c00c5850 0x7fc8c0155ab0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8009dc0 tx=0x7fc8b802bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a95f3e9932931e26 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:58.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.980+0000 7fc8beffd640 1 -- 172.21.15.161:0/956289842 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b8039070 con 0x7fc8c00c5850 2024-03-28T23:34:58.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.980+0000 7fc8beffd640 1 -- 172.21.15.161:0/956289842 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8b802c880 con 0x7fc8c00c5850 2024-03-28T23:34:58.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.980+0000 7fc8beffd640 1 -- 172.21.15.161:0/956289842 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a80017f0 con 0x7fc8c00c5850 2024-03-28T23:34:58.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.980+0000 7fc8c5b65640 1 -- 172.21.15.161:0/956289842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8c00c5850 msgr2=0x7fc8c0155ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:58.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.980+0000 7fc8c5b65640 1 --2- 172.21.15.161:0/956289842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8c00c5850 0x7fc8c0155ab0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8009dc0 tx=0x7fc8b802bd50 comp rx=0 tx=0).stop 2024-03-28T23:34:58.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.981+0000 7fc8c5b65640 1 -- 172.21.15.161:0/956289842 shutdown_connections 2024-03-28T23:34:58.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.981+0000 7fc8c5b65640 1 --2- 172.21.15.161:0/956289842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8c00c5850 0x7fc8c0155ab0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.981+0000 7fc8c5b65640 1 -- 172.21.15.161:0/956289842 >> 172.21.15.161:0/956289842 conn(0x7fc8c00ac620 msgr2=0x7fc8c00aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:58.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.981+0000 7fc8c5b65640 1 -- 172.21.15.161:0/956289842 shutdown_connections 2024-03-28T23:34:58.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.981+0000 7fc8c5b65640 1 -- 172.21.15.161:0/956289842 wait complete. 2024-03-28T23:34:58.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.982+0000 7fc8c5b65640 1 Processor -- start 2024-03-28T23:34:58.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.982+0000 7fc8c5b65640 1 -- start start 2024-03-28T23:34:58.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.983+0000 7fc8c5b65640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8c00c5410 0x7fc8c01eeca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.983+0000 7fc8c5b65640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8c00c5850 0x7fc8c01eb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.983+0000 7fc8c5b65640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8c0152890 0x7fc8c0157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.983+0000 7fc8c5b65640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc8c01592d0 con 0x7fc8c0152890 2024-03-28T23:34:58.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.983+0000 7fc8c5b65640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc8c0159640 con 0x7fc8c00c5850 2024-03-28T23:34:58.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.983+0000 7fc8c5b65640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc8c0155ff0 con 0x7fc8c00c5410 2024-03-28T23:34:58.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.983+0000 7fc8bffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8c0152890 0x7fc8c0157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.983+0000 7fc8bffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8c0152890 0x7fc8c0157a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59848/0 (socket says 172.21.15.161:59848) 2024-03-28T23:34:58.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.983+0000 7fc8bffff640 1 -- 172.21.15.161:0/195542075 learned_addr learned my addr 172.21.15.161:0/195542075 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:58.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.983+0000 7fc8b6ffd640 1 --2- 172.21.15.161:0/195542075 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8c00c5850 0x7fc8c01eb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.983+0000 7fc8bf7fe640 1 --2- 172.21.15.161:0/195542075 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8c00c5410 0x7fc8c01eeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.984+0000 7fc8bffff640 1 -- 172.21.15.161:0/195542075 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8c00c5410 msgr2=0x7fc8c01eeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:58.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.984+0000 7fc8bffff640 1 --2- 172.21.15.161:0/195542075 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8c00c5410 0x7fc8c01eeca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.984+0000 7fc8bffff640 1 -- 172.21.15.161:0/195542075 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8c00c5850 msgr2=0x7fc8c01eb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:58.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.984+0000 7fc8bffff640 1 --2- 172.21.15.161:0/195542075 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8c00c5850 0x7fc8c01eb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:58.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.984+0000 7fc8bffff640 1 -- 172.21.15.161:0/195542075 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8c0154720 con 0x7fc8c0152890 2024-03-28T23:34:58.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.984+0000 7fc8bffff640 1 --2- 172.21.15.161:0/195542075 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8c0152890 0x7fc8c0157a50 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8009ee0 tx=0x7fc8b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:58.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.984+0000 7fc8bcff9640 1 -- 172.21.15.161:0/195542075 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b8039050 con 0x7fc8c0152890 2024-03-28T23:34:58.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.984+0000 7fc8bcff9640 1 -- 172.21.15.161:0/195542075 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8b8045070 con 0x7fc8c0152890 2024-03-28T23:34:58.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.984+0000 7fc8c5b65640 1 -- 172.21.15.161:0/195542075 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8c0154a00 con 0x7fc8c0152890 2024-03-28T23:34:58.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.984+0000 7fc8bcff9640 1 -- 172.21.15.161:0/195542075 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b8009450 con 0x7fc8c0152890 2024-03-28T23:34:58.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.984+0000 7fc8c5b65640 1 -- 172.21.15.161:0/195542075 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8c0157f90 con 0x7fc8c0152890 2024-03-28T23:34:58.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.985+0000 7fc8c5b65640 1 -- 172.21.15.161:0/195542075 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8c004f4b0 con 0x7fc8c0152890 2024-03-28T23:34:58.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.986+0000 7fc8bcff9640 1 -- 172.21.15.161:0/195542075 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc8b8040690 con 0x7fc8c0152890 2024-03-28T23:34:58.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.986+0000 7fc8bcff9640 1 --2- 172.21.15.161:0/195542075 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc8a006b4d0 0x7fc8a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:58.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.986+0000 7fc8bcff9640 1 -- 172.21.15.161:0/195542075 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 6273+0+0 (secure 0 0 0) 0x7fc8b80aca10 con 0x7fc8c0152890 2024-03-28T23:34:58.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.986+0000 7fc8bf7fe640 1 --2- 172.21.15.161:0/195542075 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc8a006b4d0 0x7fc8a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:58.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.987+0000 7fc8bf7fe640 1 --2- 172.21.15.161:0/195542075 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc8a006b4d0 0x7fc8a006d990 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc8b00011e0 tx=0x7fc8b0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:58.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:58.991+0000 7fc8bcff9640 1 -- 172.21.15.161:0/195542075 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc8b807b1b0 con 0x7fc8c0152890 2024-03-28T23:34:59.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.169+0000 7fc8c5b65640 1 -- 172.21.15.161:0/195542075 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc8c00ac400 con 0x7fc8c0152890 2024-03-28T23:34:59.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.170+0000 7fc8bcff9640 1 -- 172.21.15.161:0/195542075 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) ==== 74+0+17197 (secure 0 0 0) 0x7fc8b807f250 con 0x7fc8c0152890 2024-03-28T23:34:59.171 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:34:59.171 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":124,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:34:53.266559+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:34:35.025049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-28T23:34:35.765110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"122","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6835/878057589":"2024-03-29T23:34:53.266532+0000","172.21.15.161:6834/878057589":"2024-03-29T23:34:53.266532+0000","172.21.15.161:6837/3432367324":"2024-03-29T23:34:50.245540+0000","172.21.15.161:6836/3432367324":"2024-03-29T23:34:50.245540+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:34:59.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.172+0000 7fc8c5b65640 1 -- 172.21.15.161:0/195542075 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc8a006b4d0 msgr2=0x7fc8a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:59.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.172+0000 7fc8c5b65640 1 --2- 172.21.15.161:0/195542075 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc8a006b4d0 0x7fc8a006d990 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc8b00011e0 tx=0x7fc8b0009040 comp rx=0 tx=0).stop 2024-03-28T23:34:59.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.172+0000 7fc8c5b65640 1 -- 172.21.15.161:0/195542075 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8c0152890 msgr2=0x7fc8c0157a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:59.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.172+0000 7fc8c5b65640 1 --2- 172.21.15.161:0/195542075 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8c0152890 0x7fc8c0157a50 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8009ee0 tx=0x7fc8b8004a40 comp rx=0 tx=0).stop 2024-03-28T23:34:59.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.173+0000 7fc8c5b65640 1 -- 172.21.15.161:0/195542075 shutdown_connections 2024-03-28T23:34:59.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.173+0000 7fc8c5b65640 1 --2- 172.21.15.161:0/195542075 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc8a006b4d0 0x7fc8a006d990 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.173+0000 7fc8c5b65640 1 --2- 172.21.15.161:0/195542075 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8c0152890 0x7fc8c0157a50 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.173+0000 7fc8c5b65640 1 --2- 172.21.15.161:0/195542075 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8c00c5850 0x7fc8c01eb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.173+0000 7fc8c5b65640 1 --2- 172.21.15.161:0/195542075 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8c00c5410 0x7fc8c01eeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.173+0000 7fc8c5b65640 1 -- 172.21.15.161:0/195542075 >> 172.21.15.161:0/195542075 conn(0x7fc8c00ac620 msgr2=0x7fc8c014cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:59.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.173+0000 7fc8c5b65640 1 -- 172.21.15.161:0/195542075 shutdown_connections 2024-03-28T23:34:59.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.173+0000 7fc8c5b65640 1 -- 172.21.15.161:0/195542075 wait complete. 2024-03-28T23:34:59.184 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:34:35.025049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '121', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:34:35.765110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '122', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:34:59.184 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:34:59.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.288+0000 7f56932d4640 1 Processor -- start 2024-03-28T23:34:59.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.288+0000 7f56932d4640 1 -- start start 2024-03-28T23:34:59.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.289+0000 7f56932d4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f568c1529d0 0x7f568c06bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:59.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.289+0000 7f56932d4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f568c0c6750 0x7f568c0c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:59.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.289+0000 7f56932d4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f568c155c30 0x7f568c0c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:59.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.289+0000 7f56932d4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f568c152db0 con 0x7f568c0c6750 2024-03-28T23:34:59.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.289+0000 7f56932d4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f568c0c7210 con 0x7f568c1529d0 2024-03-28T23:34:59.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.289+0000 7f56932d4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f568c0c6c50 con 0x7f568c155c30 2024-03-28T23:34:59.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.289+0000 7f569086f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f568c0c6750 0x7f568c0c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:59.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.289+0000 7f569086f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f568c0c6750 0x7f568c0c88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59858/0 (socket says 172.21.15.161:59858) 2024-03-28T23:34:59.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.289+0000 7f569086f640 1 -- 172.21.15.161:0/682269973 learned_addr learned my addr 172.21.15.161:0/682269973 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:59.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.289+0000 7f5691871640 1 --2- 172.21.15.161:0/682269973 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f568c155c30 0x7f568c0c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:59.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.289+0000 7f5691070640 1 --2- 172.21.15.161:0/682269973 >> v2:172.21.15.170:3300/0 conn(0x7f568c1529d0 0x7f568c06bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:59.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.289+0000 7f569086f640 1 -- 172.21.15.161:0/682269973 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f568c155c30 msgr2=0x7f568c0c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:59.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.289+0000 7f569086f640 1 --2- 172.21.15.161:0/682269973 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f568c155c30 0x7f568c0c54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.289+0000 7f569086f640 1 -- 172.21.15.161:0/682269973 >> v2:172.21.15.170:3300/0 conn(0x7f568c1529d0 msgr2=0x7f568c06bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:59.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.289+0000 7f569086f640 1 --2- 172.21.15.161:0/682269973 >> v2:172.21.15.170:3300/0 conn(0x7f568c1529d0 0x7f568c06bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.289+0000 7f569086f640 1 -- 172.21.15.161:0/682269973 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f568c0c6f30 con 0x7f568c0c6750 2024-03-28T23:34:59.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.289+0000 7f5691070640 1 --2- 172.21.15.161:0/682269973 >> v2:172.21.15.170:3300/0 conn(0x7f568c1529d0 0x7f568c06bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:59.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.289+0000 7f5691871640 1 --2- 172.21.15.161:0/682269973 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f568c155c30 0x7f568c0c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:59.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.290+0000 7f569086f640 1 --2- 172.21.15.161:0/682269973 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f568c0c6750 0x7f568c0c88b0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f568c0689e0 tx=0x7f568802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4f8167ec33c4488 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:59.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.290+0000 7f5683fff640 1 -- 172.21.15.161:0/682269973 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5688035020 con 0x7f568c0c6750 2024-03-28T23:34:59.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.290+0000 7f5683fff640 1 -- 172.21.15.161:0/682269973 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5688031870 con 0x7f568c0c6750 2024-03-28T23:34:59.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.290+0000 7f5683fff640 1 -- 172.21.15.161:0/682269973 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5688031c40 con 0x7f568c0c6750 2024-03-28T23:34:59.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.291+0000 7f56932d4640 1 -- 172.21.15.161:0/682269973 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f568c0c6750 msgr2=0x7f568c0c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:59.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.291+0000 7f56932d4640 1 --2- 172.21.15.161:0/682269973 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f568c0c6750 0x7f568c0c88b0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f568c0689e0 tx=0x7f568802b910 comp rx=0 tx=0).stop 2024-03-28T23:34:59.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.291+0000 7f56932d4640 1 -- 172.21.15.161:0/682269973 shutdown_connections 2024-03-28T23:34:59.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.291+0000 7f56932d4640 1 --2- 172.21.15.161:0/682269973 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f568c155c30 0x7f568c0c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.291+0000 7f56932d4640 1 --2- 172.21.15.161:0/682269973 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f568c0c6750 0x7f568c0c88b0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.291+0000 7f56932d4640 1 --2- 172.21.15.161:0/682269973 >> v2:172.21.15.170:3300/0 conn(0x7f568c1529d0 0x7f568c06bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.291+0000 7f56932d4640 1 -- 172.21.15.161:0/682269973 >> 172.21.15.161:0/682269973 conn(0x7f568c0c7430 msgr2=0x7f568c0c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:59.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.291+0000 7f56932d4640 1 -- 172.21.15.161:0/682269973 shutdown_connections 2024-03-28T23:34:59.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.291+0000 7f56932d4640 1 -- 172.21.15.161:0/682269973 wait complete. 2024-03-28T23:34:59.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.293+0000 7f56932d4640 1 Processor -- start 2024-03-28T23:34:59.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.293+0000 7f56932d4640 1 -- start start 2024-03-28T23:34:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.294+0000 7f56932d4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f568c0c6750 0x7f568c0c4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.294+0000 7f56932d4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f568c1529d0 0x7f568c1472a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.294+0000 7f56932d4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f568c155c30 0x7f568c153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.294+0000 7f56932d4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f568c0aa160 con 0x7f568c155c30 2024-03-28T23:34:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.294+0000 7f56932d4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f568c155710 con 0x7f568c0c6750 2024-03-28T23:34:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.294+0000 7f56932d4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f568c1560e0 con 0x7f568c1529d0 2024-03-28T23:34:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.294+0000 7f5691871640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f568c155c30 0x7f568c153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.294+0000 7f5691871640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f568c155c30 0x7f568c153570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59866/0 (socket says 172.21.15.161:59866) 2024-03-28T23:34:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.294+0000 7f5691871640 1 -- 172.21.15.161:0/4235296726 learned_addr learned my addr 172.21.15.161:0/4235296726 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.294+0000 7f5691070640 1 --2- 172.21.15.161:0/4235296726 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f568c0c6750 0x7f568c0c4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:59.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.294+0000 7f569086f640 1 --2- 172.21.15.161:0/4235296726 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f568c1529d0 0x7f568c1472a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:59.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.294+0000 7f5691871640 1 -- 172.21.15.161:0/4235296726 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f568c1529d0 msgr2=0x7f568c1472a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:59.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.294+0000 7f5691871640 1 --2- 172.21.15.161:0/4235296726 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f568c1529d0 0x7f568c1472a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.294+0000 7f5691871640 1 -- 172.21.15.161:0/4235296726 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f568c0c6750 msgr2=0x7f568c0c4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:59.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.294+0000 7f5691871640 1 --2- 172.21.15.161:0/4235296726 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f568c0c6750 0x7f568c0c4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.294+0000 7f5691871640 1 -- 172.21.15.161:0/4235296726 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f568c1eb500 con 0x7f568c155c30 2024-03-28T23:34:59.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.294+0000 7f5691070640 1 --2- 172.21.15.161:0/4235296726 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f568c0c6750 0x7f568c0c4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:59.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.294+0000 7f569086f640 1 --2- 172.21.15.161:0/4235296726 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f568c1529d0 0x7f568c1472a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:34:59.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.295+0000 7f5691871640 1 --2- 172.21.15.161:0/4235296726 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f568c155c30 0x7f568c153570 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f5684007df0 tx=0x7f568400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:59.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.295+0000 7f5681ffb640 1 -- 172.21.15.161:0/4235296726 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568401a070 con 0x7f568c155c30 2024-03-28T23:34:59.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.295+0000 7f5681ffb640 1 -- 172.21.15.161:0/4235296726 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5684003990 con 0x7f568c155c30 2024-03-28T23:34:59.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.295+0000 7f56932d4640 1 -- 172.21.15.161:0/4235296726 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f568c1eb7e0 con 0x7f568c155c30 2024-03-28T23:34:59.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.295+0000 7f5681ffb640 1 -- 172.21.15.161:0/4235296726 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5684003d80 con 0x7f568c155c30 2024-03-28T23:34:59.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.295+0000 7f56932d4640 1 -- 172.21.15.161:0/4235296726 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f568c1ebda0 con 0x7f568c155c30 2024-03-28T23:34:59.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.296+0000 7f5681ffb640 1 -- 172.21.15.161:0/4235296726 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f568401d070 con 0x7f568c155c30 2024-03-28T23:34:59.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.296+0000 7f56932d4640 1 -- 172.21.15.161:0/4235296726 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f568c04f4b0 con 0x7f568c155c30 2024-03-28T23:34:59.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.297+0000 7f5681ffb640 1 --2- 172.21.15.161:0/4235296726 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f567006b4d0 0x7f567006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:59.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.297+0000 7f5681ffb640 1 -- 172.21.15.161:0/4235296726 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 6273+0+0 (secure 0 0 0) 0x7f568408a630 con 0x7f568c155c30 2024-03-28T23:34:59.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.297+0000 7f5691070640 1 --2- 172.21.15.161:0/4235296726 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f567006b4d0 0x7f567006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:59.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.297+0000 7f5691070640 1 --2- 172.21.15.161:0/4235296726 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f567006b4d0 0x7f567006d990 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f567c00c940 tx=0x7f567c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:59.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.301+0000 7f5681ffb640 1 -- 172.21.15.161:0/4235296726 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5684010070 con 0x7f568c155c30 2024-03-28T23:34:59.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.595+0000 7f56932d4640 1 -- 172.21.15.161:0/4235296726 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f568c150910 con 0x7f568c155c30 2024-03-28T23:34:59.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.596+0000 7f5681ffb640 1 -- 172.21.15.161:0/4235296726 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v124) ==== 93+0+10 (secure 0 0 0) 0x7f5684059300 con 0x7f568c155c30 2024-03-28T23:34:59.597 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:34:59.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.598+0000 7f56932d4640 1 -- 172.21.15.161:0/4235296726 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f567006b4d0 msgr2=0x7f567006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:59.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.598+0000 7f56932d4640 1 --2- 172.21.15.161:0/4235296726 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f567006b4d0 0x7f567006d990 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f567c00c940 tx=0x7f567c008040 comp rx=0 tx=0).stop 2024-03-28T23:34:59.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.598+0000 7f56932d4640 1 -- 172.21.15.161:0/4235296726 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f568c155c30 msgr2=0x7f568c153570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:59.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.598+0000 7f56932d4640 1 --2- 172.21.15.161:0/4235296726 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f568c155c30 0x7f568c153570 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f5684007df0 tx=0x7f568400d4a0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.598+0000 7f56932d4640 1 -- 172.21.15.161:0/4235296726 shutdown_connections 2024-03-28T23:34:59.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.598+0000 7f56932d4640 1 --2- 172.21.15.161:0/4235296726 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f567006b4d0 0x7f567006d990 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.598+0000 7f56932d4640 1 --2- 172.21.15.161:0/4235296726 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f568c155c30 0x7f568c153570 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.598+0000 7f56932d4640 1 --2- 172.21.15.161:0/4235296726 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f568c1529d0 0x7f568c1472a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.598+0000 7f56932d4640 1 --2- 172.21.15.161:0/4235296726 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f568c0c6750 0x7f568c0c4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.598+0000 7f56932d4640 1 -- 172.21.15.161:0/4235296726 >> 172.21.15.161:0/4235296726 conn(0x7f568c0c7430 msgr2=0x7f568c14ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:59.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.599+0000 7f56932d4640 1 -- 172.21.15.161:0/4235296726 shutdown_connections 2024-03-28T23:34:59.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.599+0000 7f56932d4640 1 -- 172.21.15.161:0/4235296726 wait complete. 2024-03-28T23:34:59.611 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:34:59.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.717+0000 7f6b5ae55640 1 Processor -- start 2024-03-28T23:34:59.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.717+0000 7f6b5ae55640 1 -- start start 2024-03-28T23:34:59.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.717+0000 7f6b5ae55640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6b540c0330 0x7f6b540c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:59.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.717+0000 7f6b5ae55640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b540c6f60 con 0x7f6b541589b0 2024-03-28T23:34:59.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.717+0000 7f6b5ae55640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b540c6120 con 0x7f6b54152bc0 2024-03-28T23:34:59.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.717+0000 7f6b5ae55640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f6b540c6400 con 0x7f6b540c0330 2024-03-28T23:34:59.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.717+0000 7f6b593f2640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6b541589b0 0x7f6b54158e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44906/0 (socket says 172.21.15.161:44906) 2024-03-28T23:34:59.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.717+0000 7f6b593f2640 1 -- 172.21.15.161:0/2328295108 learned_addr learned my addr 172.21.15.161:0/2328295108 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:34:59.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.718+0000 7f6b537fe640 1 -- 172.21.15.161:0/2328295108 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1898538315 0 0) 0x7f6b540c6f60 con 0x7f6b541589b0 2024-03-28T23:34:59.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.718+0000 7f6b537fe640 1 -- 172.21.15.161:0/2328295108 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b44003520 con 0x7f6b541589b0 2024-03-28T23:34:59.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.718+0000 7f6b53fff640 1 --2- 172.21.15.161:0/2328295108 >> v2:172.21.15.170:3300/0 conn(0x7f6b540c0330 0x7f6b540c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:59.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.718+0000 7f6b537fe640 1 -- 172.21.15.161:0/2328295108 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1344735105 0 0) 0x7f6b44003520 con 0x7f6b541589b0 2024-03-28T23:34:59.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.718+0000 7f6b537fe640 1 -- 172.21.15.161:0/2328295108 >> v2:172.21.15.170:3300/0 conn(0x7f6b540c0330 msgr2=0x7f6b540c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:59.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.718+0000 7f6b537fe640 1 --2- 172.21.15.161:0/2328295108 >> v2:172.21.15.170:3300/0 conn(0x7f6b540c0330 0x7f6b540c9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.718+0000 7f6b537fe640 1 -- 172.21.15.161:0/2328295108 >> v1:172.21.15.170:6789/0 conn(0x7f6b54152bc0 legacy=0x7f6b540c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:59.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.718+0000 7f6b537fe640 1 -- 172.21.15.161:0/2328295108 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b540c96d0 con 0x7f6b541589b0 2024-03-28T23:34:59.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.718+0000 7f6b537fe640 1 -- 172.21.15.161:0/2328295108 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f6b4c0019f0 con 0x7f6b541589b0 2024-03-28T23:34:59.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.718+0000 7f6b537fe640 1 -- 172.21.15.161:0/2328295108 >> v1:172.21.15.161:6789/0 conn(0x7f6b541589b0 legacy=0x7f6b54158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:59.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.718+0000 7f6b537fe640 1 --2- 172.21.15.161:0/2328295108 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b44003a90 0x7f6b44003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:59.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.718+0000 7f6b537fe640 1 --2- 172.21.15.161:0/2328295108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6b44004800 0x7f6b44004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:59.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.718+0000 7f6b537fe640 1 --2- 172.21.15.161:0/2328295108 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b44005210 0x7f6b4400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:59.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.718+0000 7f6b537fe640 1 -- 172.21.15.161:0/2328295108 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6b540c6400 con 0x7f6b44004800 2024-03-28T23:34:59.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.718+0000 7f6b537fe640 1 -- 172.21.15.161:0/2328295108 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4400c140 con 0x7f6b44003a90 2024-03-28T23:34:59.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.718+0000 7f6b537fe640 1 -- 172.21.15.161:0/2328295108 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6b4400c340 con 0x7f6b44005210 2024-03-28T23:34:59.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.718+0000 7f6b53fff640 1 --2- 172.21.15.161:0/2328295108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6b44004800 0x7f6b44004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:59.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.718+0000 7f6b58bf1640 1 --2- 172.21.15.161:0/2328295108 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b44003a90 0x7f6b44003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:59.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.718+0000 7f6b593f2640 1 --2- 172.21.15.161:0/2328295108 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b44005210 0x7f6b4400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:59.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.719+0000 7f6b53fff640 1 -- 172.21.15.161:0/2328295108 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b44005210 msgr2=0x7f6b4400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:59.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.719+0000 7f6b53fff640 1 --2- 172.21.15.161:0/2328295108 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b44005210 0x7f6b4400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.719+0000 7f6b53fff640 1 -- 172.21.15.161:0/2328295108 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b44003a90 msgr2=0x7f6b44003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:34:59.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.719+0000 7f6b53fff640 1 --2- 172.21.15.161:0/2328295108 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b44003a90 0x7f6b44003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.719+0000 7f6b53fff640 1 -- 172.21.15.161:0/2328295108 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b44003520 con 0x7f6b44004800 2024-03-28T23:34:59.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.719+0000 7f6b593f2640 1 -- 172.21.15.161:0/2328295108 reap_dead start 2024-03-28T23:34:59.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.719+0000 7f6b53fff640 1 --2- 172.21.15.161:0/2328295108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6b44004800 0x7f6b44004c80 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f6b40009a90 tx=0x7f6b44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8355a93ecc9ab1b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:59.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.719+0000 7f6b537fe640 1 -- 172.21.15.161:0/2328295108 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b40039070 con 0x7f6b44004800 2024-03-28T23:34:59.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.719+0000 7f6b537fe640 1 -- 172.21.15.161:0/2328295108 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b4002c680 con 0x7f6b44004800 2024-03-28T23:34:59.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.719+0000 7f6b537fe640 1 -- 172.21.15.161:0/2328295108 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b40009690 con 0x7f6b44004800 2024-03-28T23:34:59.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.720+0000 7f6b5ae55640 1 -- 172.21.15.161:0/2328295108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6b44004800 msgr2=0x7f6b44004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:59.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.720+0000 7f6b5ae55640 1 --2- 172.21.15.161:0/2328295108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6b44004800 0x7f6b44004c80 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f6b40009a90 tx=0x7f6b44005830 comp rx=0 tx=0).stop 2024-03-28T23:34:59.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.720+0000 7f6b5ae55640 1 -- 172.21.15.161:0/2328295108 shutdown_connections 2024-03-28T23:34:59.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.720+0000 7f6b5ae55640 1 --2- 172.21.15.161:0/2328295108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6b44004800 0x7f6b44004c80 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.720+0000 7f6b5ae55640 1 -- 172.21.15.161:0/2328295108 >> 172.21.15.161:0/2328295108 conn(0x7f6b540caa80 msgr2=0x7f6b540cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:59.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.720+0000 7f6b5ae55640 1 -- 172.21.15.161:0/2328295108 shutdown_connections 2024-03-28T23:34:59.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.720+0000 7f6b5ae55640 1 -- 172.21.15.161:0/2328295108 wait complete. 2024-03-28T23:34:59.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.721+0000 7f6b5ae55640 1 Processor -- start 2024-03-28T23:34:59.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.721+0000 7f6b5ae55640 1 -- start start 2024-03-28T23:34:59.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.722+0000 7f6b5ae55640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b44004800 0x7f6b541484d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:59.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.722+0000 7f6b5ae55640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6b54158550 0x7f6b541516a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:59.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.722+0000 7f6b5ae55640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b541540b0 0x7f6b541567b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:59.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.722+0000 7f6b5ae55640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6b54152fa0 con 0x7f6b54158550 2024-03-28T23:34:59.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.722+0000 7f6b5ae55640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6b540c6bf0 con 0x7f6b44004800 2024-03-28T23:34:59.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.722+0000 7f6b5ae55640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6b541eb2a0 con 0x7f6b541540b0 2024-03-28T23:34:59.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.722+0000 7f6b53fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6b54158550 0x7f6b541516a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:59.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.722+0000 7f6b53fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6b54158550 0x7f6b541516a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59878/0 (socket says 172.21.15.161:59878) 2024-03-28T23:34:59.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.722+0000 7f6b53fff640 1 -- 172.21.15.161:0/1957173334 learned_addr learned my addr 172.21.15.161:0/1957173334 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:34:59.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.722+0000 7f6b53fff640 1 -- 172.21.15.161:0/1957173334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b541540b0 msgr2=0x7f6b541567b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:59.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.722+0000 7f6b53fff640 1 --2- 172.21.15.161:0/1957173334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b541540b0 0x7f6b541567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.722+0000 7f6b53fff640 1 -- 172.21.15.161:0/1957173334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b44004800 msgr2=0x7f6b541484d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:59.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.722+0000 7f6b53fff640 1 --2- 172.21.15.161:0/1957173334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b44004800 0x7f6b541484d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.722+0000 7f6b53fff640 1 -- 172.21.15.161:0/1957173334 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b541481f0 con 0x7f6b54158550 2024-03-28T23:34:59.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.723+0000 7f6b53fff640 1 --2- 172.21.15.161:0/1957173334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6b54158550 0x7f6b541516a0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f6b40000c00 tx=0x7f6b40004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:59.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.723+0000 7f6b517fa640 1 -- 172.21.15.161:0/1957173334 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b40042020 con 0x7f6b54158550 2024-03-28T23:34:59.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.723+0000 7f6b517fa640 1 -- 172.21.15.161:0/1957173334 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b4003a040 con 0x7f6b54158550 2024-03-28T23:34:59.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.723+0000 7f6b5ae55640 1 -- 172.21.15.161:0/1957173334 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b541eac50 con 0x7f6b54158550 2024-03-28T23:34:59.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.723+0000 7f6b517fa640 1 -- 172.21.15.161:0/1957173334 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b40039070 con 0x7f6b54158550 2024-03-28T23:34:59.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.723+0000 7f6b5ae55640 1 -- 172.21.15.161:0/1957173334 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b54151be0 con 0x7f6b54158550 2024-03-28T23:34:59.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.724+0000 7f6b5ae55640 1 -- 172.21.15.161:0/1957173334 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b540c66e0 con 0x7f6b54158550 2024-03-28T23:34:59.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.725+0000 7f6b517fa640 1 -- 172.21.15.161:0/1957173334 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6b40045020 con 0x7f6b54158550 2024-03-28T23:34:59.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.725+0000 7f6b517fa640 1 --2- 172.21.15.161:0/1957173334 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6b3806b500 0x7f6b3806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:34:59.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.725+0000 7f6b517fa640 1 -- 172.21.15.161:0/1957173334 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 6273+0+0 (secure 0 0 0) 0x7f6b400ab6f0 con 0x7f6b54158550 2024-03-28T23:34:59.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.725+0000 7f6b58bf1640 1 --2- 172.21.15.161:0/1957173334 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6b3806b500 0x7f6b3806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:34:59.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.726+0000 7f6b58bf1640 1 --2- 172.21.15.161:0/1957173334 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6b3806b500 0x7f6b3806d9c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f6b48000c00 tx=0x7f6b48009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:34:59.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.730+0000 7f6b517fa640 1 -- 172.21.15.161:0/1957173334 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6b540c66e0 con 0x7f6b54158550 2024-03-28T23:34:59.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.905+0000 7f6b5ae55640 1 -- 172.21.15.161:0/1957173334 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6b541eaa60 con 0x7f6b54158550 2024-03-28T23:34:59.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.905+0000 7f6b517fa640 1 -- 172.21.15.161:0/1957173334 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v124) ==== 104+0+11 (secure 0 0 0) 0x7f6b4004c070 con 0x7f6b54158550 2024-03-28T23:34:59.907 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:34:59.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.907+0000 7f6b5ae55640 1 -- 172.21.15.161:0/1957173334 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6b3806b500 msgr2=0x7f6b3806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:59.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.907+0000 7f6b5ae55640 1 --2- 172.21.15.161:0/1957173334 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6b3806b500 0x7f6b3806d9c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f6b48000c00 tx=0x7f6b48009040 comp rx=0 tx=0).stop 2024-03-28T23:34:59.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.907+0000 7f6b5ae55640 1 -- 172.21.15.161:0/1957173334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6b54158550 msgr2=0x7f6b541516a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:34:59.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.907+0000 7f6b5ae55640 1 --2- 172.21.15.161:0/1957173334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6b54158550 0x7f6b541516a0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f6b40000c00 tx=0x7f6b40004730 comp rx=0 tx=0).stop 2024-03-28T23:34:59.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.907+0000 7f6b5ae55640 1 -- 172.21.15.161:0/1957173334 shutdown_connections 2024-03-28T23:34:59.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.907+0000 7f6b5ae55640 1 --2- 172.21.15.161:0/1957173334 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6b3806b500 0x7f6b3806d9c0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.907+0000 7f6b5ae55640 1 --2- 172.21.15.161:0/1957173334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6b54158550 0x7f6b541516a0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.907+0000 7f6b5ae55640 1 --2- 172.21.15.161:0/1957173334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b541540b0 0x7f6b541567b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.907+0000 7f6b5ae55640 1 --2- 172.21.15.161:0/1957173334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b44004800 0x7f6b541484d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:34:59.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.907+0000 7f6b5ae55640 1 -- 172.21.15.161:0/1957173334 >> 172.21.15.161:0/1957173334 conn(0x7f6b540caa80 msgr2=0x7f6b5414e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:34:59.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.907+0000 7f6b5ae55640 1 -- 172.21.15.161:0/1957173334 shutdown_connections 2024-03-28T23:34:59.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:34:59.907+0000 7f6b5ae55640 1 -- 172.21.15.161:0/1957173334 wait complete. 2024-03-28T23:34:59.919 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:35:00.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.023+0000 7f6f346f4640 1 Processor -- start 2024-03-28T23:35:00.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.023+0000 7f6f346f4640 1 -- start start 2024-03-28T23:35:00.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.024+0000 7f6f346f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f2c0c6750 0x7f6f2c06bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:00.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.024+0000 7f6f346f4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6f2c0c88b0 0x7f6f2c0c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:00.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.024+0000 7f6f346f4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f2c152db0 con 0x7f6f2c1529d0 2024-03-28T23:35:00.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.024+0000 7f6f346f4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6f2c0c8d60 con 0x7f6f2c0c88b0 2024-03-28T23:35:00.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.024+0000 7f6f346f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6f2c0c7210 con 0x7f6f2c0c6750 2024-03-28T23:35:00.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.024+0000 7f6f32c91640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6f2c0c88b0 0x7f6f2c0c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:00.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.024+0000 7f6f32c91640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6f2c0c88b0 0x7f6f2c0c54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59886/0 (socket says 172.21.15.161:59886) 2024-03-28T23:35:00.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.024+0000 7f6f32490640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6f2c1529d0 0x7f6f2c155c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44916/0 (socket says 172.21.15.161:44916) 2024-03-28T23:35:00.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.024+0000 7f6f32c91640 1 -- 172.21.15.161:0/1392018468 learned_addr learned my addr 172.21.15.161:0/1392018468 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:00.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.024+0000 7f6f31c8f640 1 --2- 172.21.15.161:0/1392018468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f2c0c6750 0x7f6f2c06bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:00.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.024+0000 7f6f32c91640 1 -- 172.21.15.161:0/1392018468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f2c0c6750 msgr2=0x7f6f2c06bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:00.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.024+0000 7f6f32c91640 1 --2- 172.21.15.161:0/1392018468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f2c0c6750 0x7f6f2c06bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.025+0000 7f6f32c91640 1 -- 172.21.15.161:0/1392018468 >> v1:172.21.15.161:6789/0 conn(0x7f6f2c1529d0 legacy=0x7f6f2c155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:00.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.025+0000 7f6f3148e640 1 -- 172.21.15.161:0/1392018468 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 948895718 0 0) 0x7f6f2c152db0 con 0x7f6f2c1529d0 2024-03-28T23:35:00.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.025+0000 7f6f32c91640 1 -- 172.21.15.161:0/1392018468 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f2c0c6c50 con 0x7f6f2c0c88b0 2024-03-28T23:35:00.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.026+0000 7f6f32c91640 1 --2- 172.21.15.161:0/1392018468 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f2c0c88b0 0x7f6f2c0c54f0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f6f24009e50 tx=0x7f6f2402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1790e22cf39ec1d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:00.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.026+0000 7f6f3148e640 1 -- 172.21.15.161:0/1392018468 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2402cb00 con 0x7f6f2c0c88b0 2024-03-28T23:35:00.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.026+0000 7f6f3148e640 1 -- 172.21.15.161:0/1392018468 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f2402cca0 con 0x7f6f2c0c88b0 2024-03-28T23:35:00.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.026+0000 7f6f3148e640 1 -- 172.21.15.161:0/1392018468 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f24034610 con 0x7f6f2c0c88b0 2024-03-28T23:35:00.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.026+0000 7f6f346f4640 1 -- 172.21.15.161:0/1392018468 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f2c0c88b0 msgr2=0x7f6f2c0c54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:00.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.026+0000 7f6f346f4640 1 --2- 172.21.15.161:0/1392018468 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f2c0c88b0 0x7f6f2c0c54f0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f6f24009e50 tx=0x7f6f2402bf60 comp rx=0 tx=0).stop 2024-03-28T23:35:00.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.027+0000 7f6f346f4640 1 -- 172.21.15.161:0/1392018468 shutdown_connections 2024-03-28T23:35:00.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.027+0000 7f6f346f4640 1 --2- 172.21.15.161:0/1392018468 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f2c0c88b0 0x7f6f2c0c54f0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.027+0000 7f6f346f4640 1 --2- 172.21.15.161:0/1392018468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f2c0c6750 0x7f6f2c06bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.027+0000 7f6f346f4640 1 -- 172.21.15.161:0/1392018468 >> 172.21.15.161:0/1392018468 conn(0x7f6f2c0c7430 msgr2=0x7f6f2c0c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:00.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.027+0000 7f6f346f4640 1 -- 172.21.15.161:0/1392018468 shutdown_connections 2024-03-28T23:35:00.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.027+0000 7f6f346f4640 1 -- 172.21.15.161:0/1392018468 wait complete. 2024-03-28T23:35:00.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.028+0000 7f6f346f4640 1 Processor -- start 2024-03-28T23:35:00.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.028+0000 7f6f346f4640 1 -- start start 2024-03-28T23:35:00.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.029+0000 7f6f346f4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f2c0c6750 0x7f6f2c1eeb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:00.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.029+0000 7f6f346f4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f2c0c88b0 0x7f6f2c0abc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:00.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.029+0000 7f6f346f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f2c1529d0 0x7f6f2c0afec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:00.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.029+0000 7f6f346f4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6f2c155710 con 0x7f6f2c0c88b0 2024-03-28T23:35:00.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.029+0000 7f6f346f4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6f2c155430 con 0x7f6f2c0c6750 2024-03-28T23:35:00.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.029+0000 7f6f346f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6f2c0aa160 con 0x7f6f2c1529d0 2024-03-28T23:35:00.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.030+0000 7f6f31c8f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f2c0c88b0 0x7f6f2c0abc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:00.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.030+0000 7f6f31c8f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f2c0c88b0 0x7f6f2c0abc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59900/0 (socket says 172.21.15.161:59900) 2024-03-28T23:35:00.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.030+0000 7f6f31c8f640 1 -- 172.21.15.161:0/3132155003 learned_addr learned my addr 172.21.15.161:0/3132155003 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:00.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.030+0000 7f6f32490640 1 --2- 172.21.15.161:0/3132155003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f2c0c6750 0x7f6f2c1eeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:00.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.030+0000 7f6f32c91640 1 --2- 172.21.15.161:0/3132155003 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f2c1529d0 0x7f6f2c0afec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:00.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.030+0000 7f6f31c8f640 1 -- 172.21.15.161:0/3132155003 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f2c1529d0 msgr2=0x7f6f2c0afec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:00.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.030+0000 7f6f31c8f640 1 --2- 172.21.15.161:0/3132155003 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f2c1529d0 0x7f6f2c0afec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.030+0000 7f6f31c8f640 1 -- 172.21.15.161:0/3132155003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f2c0c6750 msgr2=0x7f6f2c1eeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:00.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.030+0000 7f6f31c8f640 1 --2- 172.21.15.161:0/3132155003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f2c0c6750 0x7f6f2c1eeb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.030+0000 7f6f31c8f640 1 -- 172.21.15.161:0/3132155003 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f2c14eb70 con 0x7f6f2c0c88b0 2024-03-28T23:35:00.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.030+0000 7f6f31c8f640 1 --2- 172.21.15.161:0/3132155003 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f2c0c88b0 0x7f6f2c0abc10 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f6f2000d620 tx=0x7f6f2000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:00.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.030+0000 7f6f1effd640 1 -- 172.21.15.161:0/3132155003 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2000b620 con 0x7f6f2c0c88b0 2024-03-28T23:35:00.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.030+0000 7f6f1effd640 1 -- 172.21.15.161:0/3132155003 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f20019070 con 0x7f6f2c0c88b0 2024-03-28T23:35:00.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.030+0000 7f6f346f4640 1 -- 172.21.15.161:0/3132155003 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f2c1ef040 con 0x7f6f2c0c88b0 2024-03-28T23:35:00.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.030+0000 7f6f1effd640 1 -- 172.21.15.161:0/3132155003 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2000ba00 con 0x7f6f2c0c88b0 2024-03-28T23:35:00.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.030+0000 7f6f346f4640 1 -- 172.21.15.161:0/3132155003 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f2c14f0f0 con 0x7f6f2c0c88b0 2024-03-28T23:35:00.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.032+0000 7f6f346f4640 1 -- 172.21.15.161:0/3132155003 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f2c04f4b0 con 0x7f6f2c0c88b0 2024-03-28T23:35:00.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.032+0000 7f6f1effd640 1 -- 172.21.15.161:0/3132155003 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6f2000bba0 con 0x7f6f2c0c88b0 2024-03-28T23:35:00.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.032+0000 7f6f1effd640 1 --2- 172.21.15.161:0/3132155003 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6f1006b500 0x7f6f1006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:00.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.033+0000 7f6f32490640 1 --2- 172.21.15.161:0/3132155003 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6f1006b500 0x7f6f1006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:00.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.033+0000 7f6f1effd640 1 -- 172.21.15.161:0/3132155003 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 6273+0+0 (secure 0 0 0) 0x7f6f2008a6e0 con 0x7f6f2c0c88b0 2024-03-28T23:35:00.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.033+0000 7f6f32490640 1 --2- 172.21.15.161:0/3132155003 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6f1006b500 0x7f6f1006d9c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f6f28007fd0 tx=0x7f6f28007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:00.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.037+0000 7f6f1effd640 1 -- 172.21.15.161:0/3132155003 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6f20058cd0 con 0x7f6f2c0c88b0 2024-03-28T23:35:00.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.288+0000 7f6f346f4640 1 -- 172.21.15.161:0/3132155003 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6f2c06d800 con 0x7f6f2c0c88b0 2024-03-28T23:35:00.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.289+0000 7f6f1effd640 1 -- 172.21.15.161:0/3132155003 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v124) ==== 100+0+11 (secure 0 0 0) 0x7f6f2005cd70 con 0x7f6f2c0c88b0 2024-03-28T23:35:00.290 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:35:00.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.291+0000 7f6f346f4640 1 -- 172.21.15.161:0/3132155003 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6f1006b500 msgr2=0x7f6f1006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:00.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.291+0000 7f6f346f4640 1 --2- 172.21.15.161:0/3132155003 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6f1006b500 0x7f6f1006d9c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f6f28007fd0 tx=0x7f6f28007de0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.291+0000 7f6f346f4640 1 -- 172.21.15.161:0/3132155003 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f2c0c88b0 msgr2=0x7f6f2c0abc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:00.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.291+0000 7f6f346f4640 1 --2- 172.21.15.161:0/3132155003 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f2c0c88b0 0x7f6f2c0abc10 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f6f2000d620 tx=0x7f6f2000dae0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.291+0000 7f6f346f4640 1 -- 172.21.15.161:0/3132155003 shutdown_connections 2024-03-28T23:35:00.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.291+0000 7f6f346f4640 1 --2- 172.21.15.161:0/3132155003 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6f1006b500 0x7f6f1006d9c0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.291+0000 7f6f346f4640 1 --2- 172.21.15.161:0/3132155003 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f2c0c88b0 0x7f6f2c0abc10 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.291+0000 7f6f346f4640 1 --2- 172.21.15.161:0/3132155003 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f2c1529d0 0x7f6f2c0afec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.291+0000 7f6f346f4640 1 --2- 172.21.15.161:0/3132155003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f2c0c6750 0x7f6f2c1eeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.291+0000 7f6f346f4640 1 -- 172.21.15.161:0/3132155003 >> 172.21.15.161:0/3132155003 conn(0x7f6f2c0c7430 msgr2=0x7f6f2c0c9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:00.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.291+0000 7f6f346f4640 1 -- 172.21.15.161:0/3132155003 shutdown_connections 2024-03-28T23:35:00.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.291+0000 7f6f346f4640 1 -- 172.21.15.161:0/3132155003 wait complete. 2024-03-28T23:35:00.304 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:35:00.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.404+0000 7fdba7455640 1 Processor -- start 2024-03-28T23:35:00.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.404+0000 7fdba7455640 1 -- start start 2024-03-28T23:35:00.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.405+0000 7fdba7455640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdba0152890 0x7fdba00c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:00.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.405+0000 7fdba7455640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdba00c8ed0 0x7fdba00c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:00.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.405+0000 7fdba7455640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba0158a30 con 0x7fdba00c5950 2024-03-28T23:35:00.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.405+0000 7fdba7455640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fdba00c8480 con 0x7fdba0152890 2024-03-28T23:35:00.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.405+0000 7fdba7455640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdba0159640 con 0x7fdba00c8ed0 2024-03-28T23:35:00.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.405+0000 7fdba51f1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdba0152890 0x7fdba00c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:00.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.405+0000 7fdba51f1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdba0152890 0x7fdba00c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59910/0 (socket says 172.21.15.161:59910) 2024-03-28T23:35:00.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.405+0000 7fdba51f1640 1 -- 172.21.15.161:0/797085721 learned_addr learned my addr 172.21.15.161:0/797085721 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:00.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.405+0000 7fdba51f1640 1 -- 172.21.15.161:0/797085721 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdba00c8ed0 msgr2=0x7fdba00c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:00.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.405+0000 7fdba49f0640 1 --2- 172.21.15.161:0/797085721 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdba00c8ed0 0x7fdba00c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:00.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.405+0000 7fdba51f1640 1 --2- 172.21.15.161:0/797085721 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdba00c8ed0 0x7fdba00c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.405+0000 7fdba51f1640 1 -- 172.21.15.161:0/797085721 >> v1:172.21.15.170:6789/0 conn(0x7fdba00c5950 legacy=0x7fdba01553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:00.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.405+0000 7fdba51f1640 1 -- 172.21.15.161:0/797085721 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba0158d10 con 0x7fdba0152890 2024-03-28T23:35:00.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.406+0000 7fdba51f1640 1 --2- 172.21.15.161:0/797085721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdba0152890 0x7fdba00c8990 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009870 tx=0x7fdb9c02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c35ca84ae30c3a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:00.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.406+0000 7fdb93fff640 1 -- 172.21.15.161:0/797085721 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c038070 con 0x7fdba0152890 2024-03-28T23:35:00.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.406+0000 7fdb93fff640 1 -- 172.21.15.161:0/797085721 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb9c031740 con 0x7fdba0152890 2024-03-28T23:35:00.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.406+0000 7fdb93fff640 1 -- 172.21.15.161:0/797085721 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c02f050 con 0x7fdba0152890 2024-03-28T23:35:00.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.407+0000 7fdba7455640 1 -- 172.21.15.161:0/797085721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdba0152890 msgr2=0x7fdba00c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:00.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.407+0000 7fdba7455640 1 --2- 172.21.15.161:0/797085721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdba0152890 0x7fdba00c8990 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009870 tx=0x7fdb9c02b7e0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.407+0000 7fdba7455640 1 -- 172.21.15.161:0/797085721 shutdown_connections 2024-03-28T23:35:00.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.407+0000 7fdba7455640 1 --2- 172.21.15.161:0/797085721 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdba00c8ed0 0x7fdba00c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.407+0000 7fdba7455640 1 --2- 172.21.15.161:0/797085721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdba0152890 0x7fdba00c8990 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.407+0000 7fdba7455640 1 -- 172.21.15.161:0/797085721 >> 172.21.15.161:0/797085721 conn(0x7fdba00ac620 msgr2=0x7fdba00aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:00.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.407+0000 7fdba7455640 1 -- 172.21.15.161:0/797085721 shutdown_connections 2024-03-28T23:35:00.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.407+0000 7fdba7455640 1 -- 172.21.15.161:0/797085721 wait complete. 2024-03-28T23:35:00.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.408+0000 7fdba7455640 1 Processor -- start 2024-03-28T23:35:00.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.408+0000 7fdba7455640 1 -- start start 2024-03-28T23:35:00.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.409+0000 7fdba7455640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdba00c5950 0x7fdba00ada90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:00.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.409+0000 7fdba7455640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdba00c8ed0 0x7fdba01eee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:00.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.409+0000 7fdba7455640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdba0152890 0x7fdba0158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:00.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.409+0000 7fdba7455640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdba0152c70 con 0x7fdba0152890 2024-03-28T23:35:00.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.409+0000 7fdba7455640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdba01592d0 con 0x7fdba00c5950 2024-03-28T23:35:00.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.409+0000 7fdba7455640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdba0152df0 con 0x7fdba00c8ed0 2024-03-28T23:35:00.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.409+0000 7fdba59f2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdba0152890 0x7fdba0158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:00.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.409+0000 7fdba59f2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdba0152890 0x7fdba0158020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59922/0 (socket says 172.21.15.161:59922) 2024-03-28T23:35:00.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.409+0000 7fdba59f2640 1 -- 172.21.15.161:0/3268587623 learned_addr learned my addr 172.21.15.161:0/3268587623 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:00.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.409+0000 7fdba51f1640 1 --2- 172.21.15.161:0/3268587623 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdba00c5950 0x7fdba00ada90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:00.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.409+0000 7fdba49f0640 1 --2- 172.21.15.161:0/3268587623 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdba00c8ed0 0x7fdba01eee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:00.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.409+0000 7fdba59f2640 1 -- 172.21.15.161:0/3268587623 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdba00c8ed0 msgr2=0x7fdba01eee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:00.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.409+0000 7fdba59f2640 1 --2- 172.21.15.161:0/3268587623 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdba00c8ed0 0x7fdba01eee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.409+0000 7fdba59f2640 1 -- 172.21.15.161:0/3268587623 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdba00c5950 msgr2=0x7fdba00ada90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:00.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.409+0000 7fdba59f2640 1 --2- 172.21.15.161:0/3268587623 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdba00c5950 0x7fdba00ada90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.409+0000 7fdba59f2640 1 -- 172.21.15.161:0/3268587623 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba01555e0 con 0x7fdba0152890 2024-03-28T23:35:00.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.409+0000 7fdba59f2640 1 --2- 172.21.15.161:0/3268587623 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdba0152890 0x7fdba0158020 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fdb9800b970 tx=0x7fdb9800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:00.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.409+0000 7fdb91ffb640 1 -- 172.21.15.161:0/3268587623 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb98019070 con 0x7fdba0152890 2024-03-28T23:35:00.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.409+0000 7fdb91ffb640 1 -- 172.21.15.161:0/3268587623 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb9800cb40 con 0x7fdba0152890 2024-03-28T23:35:00.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.409+0000 7fdb91ffb640 1 -- 172.21.15.161:0/3268587623 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb98014440 con 0x7fdba0152890 2024-03-28T23:35:00.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.409+0000 7fdba7455640 1 -- 172.21.15.161:0/3268587623 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdba01558c0 con 0x7fdba0152890 2024-03-28T23:35:00.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.410+0000 7fdba7455640 1 -- 172.21.15.161:0/3268587623 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdba0158560 con 0x7fdba0152890 2024-03-28T23:35:00.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.411+0000 7fdb91ffb640 1 -- 172.21.15.161:0/3268587623 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdb9801c070 con 0x7fdba0152890 2024-03-28T23:35:00.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.411+0000 7fdba7455640 1 -- 172.21.15.161:0/3268587623 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdba01eae10 con 0x7fdba0152890 2024-03-28T23:35:00.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.411+0000 7fdb91ffb640 1 --2- 172.21.15.161:0/3268587623 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdb8406b4d0 0x7fdb8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:00.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.411+0000 7fdb91ffb640 1 -- 172.21.15.161:0/3268587623 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 6273+0+0 (secure 0 0 0) 0x7fdb98089fe0 con 0x7fdba0152890 2024-03-28T23:35:00.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.411+0000 7fdba51f1640 1 --2- 172.21.15.161:0/3268587623 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdb8406b4d0 0x7fdb8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:00.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.412+0000 7fdba51f1640 1 --2- 172.21.15.161:0/3268587623 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdb8406b4d0 0x7fdb8406d990 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c035000 tx=0x7fdb9c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:00.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.416+0000 7fdb91ffb640 1 -- 172.21.15.161:0/3268587623 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdba01eae10 con 0x7fdba0152890 2024-03-28T23:35:00.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.645+0000 7fdba7455640 1 -- 172.21.15.161:0/3268587623 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdba004f4b0 con 0x7fdba0152890 2024-03-28T23:35:00.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.646+0000 7fdb91ffb640 1 -- 172.21.15.161:0/3268587623 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+5199 (secure 0 0 0) 0x7fdb980585e0 con 0x7fdba0152890 2024-03-28T23:35:00.647 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:00.647 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":117,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7318,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3734750343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3734750343},{"type":"v1","addr":"172.21.15.170:6835","nonce":3734750343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/4140438862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":4140438862},{"type":"v1","addr":"172.21.15.170:6834","nonce":4140438862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":8220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/896089063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":896089063},{"type":"v1","addr":"172.21.15.161:6835","nonce":896089063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:34:36.796608+0000","modified":"2024-03-28T23:34:56.277306+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"a","rank":0,"incarnation":114,"state":"up:active","state_seq":4,"addr":"172.21.15.161:6837/3929102513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":3929102513},{"type":"v1","addr":"172.21.15.161:6837","nonce":3929102513}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8199,"qdb_cluster":[8199]},"id":9}]} 2024-03-28T23:35:00.648 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 117 2024-03-28T23:35:00.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.648+0000 7fdba7455640 1 -- 172.21.15.161:0/3268587623 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdb8406b4d0 msgr2=0x7fdb8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:00.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.648+0000 7fdba7455640 1 --2- 172.21.15.161:0/3268587623 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdb8406b4d0 0x7fdb8406d990 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c035000 tx=0x7fdb9c008030 comp rx=0 tx=0).stop 2024-03-28T23:35:00.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.648+0000 7fdba7455640 1 -- 172.21.15.161:0/3268587623 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdba0152890 msgr2=0x7fdba0158020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:00.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.648+0000 7fdba7455640 1 --2- 172.21.15.161:0/3268587623 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdba0152890 0x7fdba0158020 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fdb9800b970 tx=0x7fdb9800be30 comp rx=0 tx=0).stop 2024-03-28T23:35:00.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.648+0000 7fdba7455640 1 -- 172.21.15.161:0/3268587623 shutdown_connections 2024-03-28T23:35:00.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.648+0000 7fdba7455640 1 --2- 172.21.15.161:0/3268587623 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdb8406b4d0 0x7fdb8406d990 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.648+0000 7fdba7455640 1 --2- 172.21.15.161:0/3268587623 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdba0152890 0x7fdba0158020 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.648+0000 7fdba7455640 1 --2- 172.21.15.161:0/3268587623 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdba00c8ed0 0x7fdba01eee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.648+0000 7fdba7455640 1 --2- 172.21.15.161:0/3268587623 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdba00c5950 0x7fdba00ada90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.648+0000 7fdba7455640 1 -- 172.21.15.161:0/3268587623 >> 172.21.15.161:0/3268587623 conn(0x7fdba00ac620 msgr2=0x7fdba0150de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:00.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.648+0000 7fdba7455640 1 -- 172.21.15.161:0/3268587623 shutdown_connections 2024-03-28T23:35:00.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.648+0000 7fdba7455640 1 -- 172.21.15.161:0/3268587623 wait complete. 2024-03-28T23:35:00.660 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:35:00.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.762+0000 7f8b17599640 1 Processor -- start 2024-03-28T23:35:00.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.762+0000 7f8b17599640 1 -- start start 2024-03-28T23:35:00.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.763+0000 7f8b17599640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b10152890 0x7f8b100c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:00.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.763+0000 7f8b17599640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b10158a30 con 0x7f8b100c8ed0 2024-03-28T23:35:00.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.763+0000 7f8b17599640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b10158d10 con 0x7f8b100c5b20 2024-03-28T23:35:00.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.763+0000 7f8b17599640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8b100c8480 con 0x7f8b10152890 2024-03-28T23:35:00.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.763+0000 7f8b14b34640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f8b100c8ed0 0x7f8b100c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45256/0 (socket says 172.21.15.161:45256) 2024-03-28T23:35:00.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.763+0000 7f8b14b34640 1 -- 172.21.15.161:0/3171868514 learned_addr learned my addr 172.21.15.161:0/3171868514 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:00.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.763+0000 7f8b07fff640 1 -- 172.21.15.161:0/3171868514 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 563083457 0 0) 0x7f8b10158a30 con 0x7f8b100c8ed0 2024-03-28T23:35:00.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.763+0000 7f8b07fff640 1 -- 172.21.15.161:0/3171868514 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8afc003520 con 0x7f8b100c8ed0 2024-03-28T23:35:00.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.763+0000 7f8b15335640 1 --2- 172.21.15.161:0/3171868514 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b10152890 0x7f8b100c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:00.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.763+0000 7f8b07fff640 1 -- 172.21.15.161:0/3171868514 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 355363263 0 0) 0x7f8afc003520 con 0x7f8b100c8ed0 2024-03-28T23:35:00.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.763+0000 7f8b07fff640 1 -- 172.21.15.161:0/3171868514 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b10152890 msgr2=0x7f8b100c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:00.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.763+0000 7f8b07fff640 1 --2- 172.21.15.161:0/3171868514 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b10152890 0x7f8b100c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.763+0000 7f8b07fff640 1 -- 172.21.15.161:0/3171868514 >> v1:172.21.15.170:6789/0 conn(0x7f8b100c5b20 legacy=0x7f8b101553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:00.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.763+0000 7f8b07fff640 1 -- 172.21.15.161:0/3171868514 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b10158ff0 con 0x7f8b100c8ed0 2024-03-28T23:35:00.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.763+0000 7f8b07fff640 1 -- 172.21.15.161:0/3171868514 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f8b000017f0 con 0x7f8b100c8ed0 2024-03-28T23:35:00.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.763+0000 7f8b07fff640 1 -- 172.21.15.161:0/3171868514 >> v1:172.21.15.161:6789/0 conn(0x7f8b100c8ed0 legacy=0x7f8b100c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:00.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.763+0000 7f8b07fff640 1 --2- 172.21.15.161:0/3171868514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8afc003ac0 0x7f8afc003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:00.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.763+0000 7f8b07fff640 1 --2- 172.21.15.161:0/3171868514 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b10152890 0x7f8afc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:00.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.763+0000 7f8b07fff640 1 --2- 172.21.15.161:0/3171868514 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8afc004db0 0x7f8afc005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:00.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.764+0000 7f8b07fff640 1 -- 172.21.15.161:0/3171868514 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc00bc30 con 0x7f8afc003ac0 2024-03-28T23:35:00.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.764+0000 7f8b07fff640 1 -- 172.21.15.161:0/3171868514 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc00bde0 con 0x7f8afc004db0 2024-03-28T23:35:00.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.764+0000 7f8b07fff640 1 -- 172.21.15.161:0/3171868514 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8afc00bfe0 con 0x7f8b10152890 2024-03-28T23:35:00.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.764+0000 7f8b15335640 1 --2- 172.21.15.161:0/3171868514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8afc003ac0 0x7f8afc003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:00.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.764+0000 7f8b15b36640 1 --2- 172.21.15.161:0/3171868514 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8afc004db0 0x7f8afc005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:00.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.764+0000 7f8b14b34640 1 --2- 172.21.15.161:0/3171868514 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b10152890 0x7f8afc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:00.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.764+0000 7f8b15335640 1 -- 172.21.15.161:0/3171868514 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b10152890 msgr2=0x7f8afc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:00.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.764+0000 7f8b15335640 1 --2- 172.21.15.161:0/3171868514 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b10152890 0x7f8afc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.764+0000 7f8b15335640 1 -- 172.21.15.161:0/3171868514 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8afc004db0 msgr2=0x7f8afc005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:00.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.764+0000 7f8b15335640 1 --2- 172.21.15.161:0/3171868514 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8afc004db0 0x7f8afc005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.764+0000 7f8b15335640 1 -- 172.21.15.161:0/3171868514 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8afc003520 con 0x7f8afc003ac0 2024-03-28T23:35:00.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.765+0000 7f8b15335640 1 --2- 172.21.15.161:0/3171868514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8afc003ac0 0x7f8afc003ee0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c009ac0 tx=0x7f8afc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=916bb0fd24b500d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:00.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.765+0000 7f8b07fff640 1 -- 172.21.15.161:0/3171868514 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c039070 con 0x7f8afc003ac0 2024-03-28T23:35:00.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.765+0000 7f8b07fff640 1 -- 172.21.15.161:0/3171868514 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b0c02c6b0 con 0x7f8afc003ac0 2024-03-28T23:35:00.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.765+0000 7f8b07fff640 1 -- 172.21.15.161:0/3171868514 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c009690 con 0x7f8afc003ac0 2024-03-28T23:35:00.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.765+0000 7f8b17599640 1 -- 172.21.15.161:0/3171868514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8afc003ac0 msgr2=0x7f8afc003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:00.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.765+0000 7f8b17599640 1 --2- 172.21.15.161:0/3171868514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8afc003ac0 0x7f8afc003ee0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c009ac0 tx=0x7f8afc005830 comp rx=0 tx=0).stop 2024-03-28T23:35:00.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.765+0000 7f8b15b36640 1 -- 172.21.15.161:0/3171868514 reap_dead start 2024-03-28T23:35:00.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.765+0000 7f8b17599640 1 -- 172.21.15.161:0/3171868514 shutdown_connections 2024-03-28T23:35:00.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.765+0000 7f8b17599640 1 -- 172.21.15.161:0/3171868514 >> 172.21.15.161:0/3171868514 conn(0x7f8b100ac620 msgr2=0x7f8b100aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:00.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.766+0000 7f8b17599640 1 -- 172.21.15.161:0/3171868514 shutdown_connections 2024-03-28T23:35:00.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.766+0000 7f8b17599640 1 -- 172.21.15.161:0/3171868514 wait complete. 2024-03-28T23:35:00.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.767+0000 7f8b17599640 1 Processor -- start 2024-03-28T23:35:00.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.767+0000 7f8b17599640 1 -- start start 2024-03-28T23:35:00.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.767+0000 7f8b17599640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8afc003ac0 0x7f8b1014cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:00.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.767+0000 7f8b17599640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b100ab060 0x7f8b10149d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:00.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.767+0000 7f8b17599640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b100ac0b0 0x7f8b101eb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:00.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.767+0000 7f8b17599640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8b10152c70 con 0x7f8b100ac0b0 2024-03-28T23:35:00.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.767+0000 7f8b17599640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8b10159640 con 0x7f8b100ab060 2024-03-28T23:35:00.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.767+0000 7f8b17599640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8b10152df0 con 0x7f8afc003ac0 2024-03-28T23:35:00.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.767+0000 7f8b15b36640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b100ac0b0 0x7f8b101eb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:00.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.767+0000 7f8b15b36640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b100ac0b0 0x7f8b101eb250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57374/0 (socket says 172.21.15.161:57374) 2024-03-28T23:35:00.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.767+0000 7f8b15b36640 1 -- 172.21.15.161:0/1230267438 learned_addr learned my addr 172.21.15.161:0/1230267438 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:00.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.768+0000 7f8b15335640 1 --2- 172.21.15.161:0/1230267438 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8afc003ac0 0x7f8b1014cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:00.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.768+0000 7f8b14b34640 1 --2- 172.21.15.161:0/1230267438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b100ab060 0x7f8b10149d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:00.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.768+0000 7f8b15b36640 1 -- 172.21.15.161:0/1230267438 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8afc003ac0 msgr2=0x7f8b1014cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:00.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.768+0000 7f8b15b36640 1 --2- 172.21.15.161:0/1230267438 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8afc003ac0 0x7f8b1014cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.768+0000 7f8b15b36640 1 -- 172.21.15.161:0/1230267438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b100ab060 msgr2=0x7f8b10149d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:00.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.768+0000 7f8b15b36640 1 --2- 172.21.15.161:0/1230267438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b100ab060 0x7f8b10149d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.768+0000 7f8b15b36640 1 -- 172.21.15.161:0/1230267438 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b101ee9d0 con 0x7f8b100ac0b0 2024-03-28T23:35:00.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.768+0000 7f8b14b34640 1 --2- 172.21.15.161:0/1230267438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b100ab060 0x7f8b10149d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:00.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.768+0000 7f8b15b36640 1 --2- 172.21.15.161:0/1230267438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b100ac0b0 0x7f8b101eb250 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f8b100aa2e0 tx=0x7f8b0800fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:00.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.768+0000 7f8b05ffb640 1 -- 172.21.15.161:0/1230267438 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b0801a070 con 0x7f8b100ac0b0 2024-03-28T23:35:00.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.768+0000 7f8b05ffb640 1 -- 172.21.15.161:0/1230267438 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b08006d60 con 0x7f8b100ac0b0 2024-03-28T23:35:00.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.768+0000 7f8b05ffb640 1 -- 172.21.15.161:0/1230267438 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b08003b90 con 0x7f8b100ac0b0 2024-03-28T23:35:00.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.768+0000 7f8b17599640 1 -- 172.21.15.161:0/1230267438 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b101eaa50 con 0x7f8b100ac0b0 2024-03-28T23:35:00.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.768+0000 7f8b17599640 1 -- 172.21.15.161:0/1230267438 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b101eb790 con 0x7f8b100ac0b0 2024-03-28T23:35:00.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.769+0000 7f8b17599640 1 -- 172.21.15.161:0/1230267438 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b1004f4b0 con 0x7f8b100ac0b0 2024-03-28T23:35:00.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.769+0000 7f8b05ffb640 1 -- 172.21.15.161:0/1230267438 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8b0801d070 con 0x7f8b100ac0b0 2024-03-28T23:35:00.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.770+0000 7f8b05ffb640 1 --2- 172.21.15.161:0/1230267438 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8af406b500 0x7f8af406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:00.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.770+0000 7f8b15335640 1 --2- 172.21.15.161:0/1230267438 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8af406b500 0x7f8af406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:00.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.770+0000 7f8b05ffb640 1 -- 172.21.15.161:0/1230267438 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 6273+0+0 (secure 0 0 0) 0x7f8b08089a50 con 0x7f8b100ac0b0 2024-03-28T23:35:00.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.771+0000 7f8b15335640 1 --2- 172.21.15.161:0/1230267438 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8af406b500 0x7f8af406d9c0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c036000 tx=0x7f8b0c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:00.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.774+0000 7f8b05ffb640 1 -- 172.21.15.161:0/1230267438 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8b08058670 con 0x7f8b100ac0b0 2024-03-28T23:35:00.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.978+0000 7f8b17599640 1 -- 172.21.15.161:0/1230267438 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8b1014c3f0 con 0x7f8b100ac0b0 2024-03-28T23:35:00.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.979+0000 7f8b05ffb640 1 -- 172.21.15.161:0/1230267438 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) ==== 74+0+17197 (secure 0 0 0) 0x7f8b08057f70 con 0x7f8b100ac0b0 2024-03-28T23:35:00.980 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:00.981 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":124,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:34:53.266559+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:34:35.025049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-28T23:34:35.765110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"122","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6835/878057589":"2024-03-29T23:34:53.266532+0000","172.21.15.161:6834/878057589":"2024-03-29T23:34:53.266532+0000","172.21.15.161:6837/3432367324":"2024-03-29T23:34:50.245540+0000","172.21.15.161:6836/3432367324":"2024-03-29T23:34:50.245540+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:35:00.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.981+0000 7f8b17599640 1 -- 172.21.15.161:0/1230267438 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8af406b500 msgr2=0x7f8af406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:00.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.981+0000 7f8b17599640 1 --2- 172.21.15.161:0/1230267438 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8af406b500 0x7f8af406d9c0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c036000 tx=0x7f8b0c030000 comp rx=0 tx=0).stop 2024-03-28T23:35:00.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.981+0000 7f8b17599640 1 -- 172.21.15.161:0/1230267438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b100ac0b0 msgr2=0x7f8b101eb250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:00.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.981+0000 7f8b17599640 1 --2- 172.21.15.161:0/1230267438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b100ac0b0 0x7f8b101eb250 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f8b100aa2e0 tx=0x7f8b0800fd40 comp rx=0 tx=0).stop 2024-03-28T23:35:00.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.981+0000 7f8b17599640 1 -- 172.21.15.161:0/1230267438 shutdown_connections 2024-03-28T23:35:00.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.981+0000 7f8b17599640 1 --2- 172.21.15.161:0/1230267438 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8af406b500 0x7f8af406d9c0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.981+0000 7f8b17599640 1 --2- 172.21.15.161:0/1230267438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b100ac0b0 0x7f8b101eb250 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.981+0000 7f8b17599640 1 --2- 172.21.15.161:0/1230267438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b100ab060 0x7f8b10149d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.981+0000 7f8b17599640 1 --2- 172.21.15.161:0/1230267438 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8afc003ac0 0x7f8b1014cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:00.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.981+0000 7f8b17599640 1 -- 172.21.15.161:0/1230267438 >> 172.21.15.161:0/1230267438 conn(0x7f8b100ac620 msgr2=0x7f8b100c9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:00.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.981+0000 7f8b17599640 1 -- 172.21.15.161:0/1230267438 shutdown_connections 2024-03-28T23:35:00.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:00.981+0000 7f8b17599640 1 -- 172.21.15.161:0/1230267438 wait complete. 2024-03-28T23:35:00.994 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-28T23:35:00.994 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:35:01.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.094+0000 7f1e5463b640 1 Processor -- start 2024-03-28T23:35:01.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.094+0000 7f1e5463b640 1 -- start start 2024-03-28T23:35:01.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.095+0000 7f1e5463b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e4c0c5850 0x7f1e4c155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:01.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.095+0000 7f1e5463b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e4c0c5d00 con 0x7f1e4c152890 2024-03-28T23:35:01.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.095+0000 7f1e5463b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e4c158a30 con 0x7f1e4c0c5410 2024-03-28T23:35:01.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.095+0000 7f1e5463b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1e4c0c8480 con 0x7f1e4c0c5850 2024-03-28T23:35:01.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.095+0000 7f1e523d7640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f1e4c152890 0x7f1e4c0c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45268/0 (socket says 172.21.15.161:45268) 2024-03-28T23:35:01.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.095+0000 7f1e523d7640 1 -- 172.21.15.161:0/2184183093 learned_addr learned my addr 172.21.15.161:0/2184183093 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:01.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.095+0000 7f1e52bd8640 1 --2- 172.21.15.161:0/2184183093 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e4c0c5850 0x7f1e4c155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:01.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.095+0000 7f1e513d5640 1 -- 172.21.15.161:0/2184183093 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2572759282 0 0) 0x7f1e4c0c5d00 con 0x7f1e4c152890 2024-03-28T23:35:01.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.095+0000 7f1e513d5640 1 -- 172.21.15.161:0/2184183093 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e34003520 con 0x7f1e4c152890 2024-03-28T23:35:01.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.096+0000 7f1e513d5640 1 -- 172.21.15.161:0/2184183093 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3330012204 0 0) 0x7f1e4c158a30 con 0x7f1e4c0c5410 2024-03-28T23:35:01.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.096+0000 7f1e52bd8640 1 -- 172.21.15.161:0/2184183093 >> v1:172.21.15.170:6789/0 conn(0x7f1e4c0c5410 legacy=0x7f1e4c1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:01.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.096+0000 7f1e52bd8640 1 -- 172.21.15.161:0/2184183093 >> v1:172.21.15.161:6789/0 conn(0x7f1e4c152890 legacy=0x7f1e4c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:01.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.096+0000 7f1e52bd8640 1 -- 172.21.15.161:0/2184183093 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e4c158d10 con 0x7f1e4c0c5850 2024-03-28T23:35:01.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.096+0000 7f1e52bd8640 1 --2- 172.21.15.161:0/2184183093 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e4c0c5850 0x7f1e4c155ab0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c0097d0 tx=0x7f1e3c02b690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb14e06ba2e3a4d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:01.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.097+0000 7f1e513d5640 1 -- 172.21.15.161:0/2184183093 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c037070 con 0x7f1e4c0c5850 2024-03-28T23:35:01.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.097+0000 7f1e513d5640 1 -- 172.21.15.161:0/2184183093 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e3c030510 con 0x7f1e4c0c5850 2024-03-28T23:35:01.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.097+0000 7f1e513d5640 1 -- 172.21.15.161:0/2184183093 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e440019f0 con 0x7f1e4c0c5850 2024-03-28T23:35:01.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.097+0000 7f1e5463b640 1 -- 172.21.15.161:0/2184183093 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e4c0c5850 msgr2=0x7f1e4c155ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:01.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.097+0000 7f1e5463b640 1 --2- 172.21.15.161:0/2184183093 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e4c0c5850 0x7f1e4c155ab0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c0097d0 tx=0x7f1e3c02b690 comp rx=0 tx=0).stop 2024-03-28T23:35:01.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.097+0000 7f1e5463b640 1 -- 172.21.15.161:0/2184183093 shutdown_connections 2024-03-28T23:35:01.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.097+0000 7f1e5463b640 1 --2- 172.21.15.161:0/2184183093 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e4c0c5850 0x7f1e4c155ab0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:01.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.097+0000 7f1e5463b640 1 -- 172.21.15.161:0/2184183093 >> 172.21.15.161:0/2184183093 conn(0x7f1e4c0ac620 msgr2=0x7f1e4c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:01.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.098+0000 7f1e5463b640 1 -- 172.21.15.161:0/2184183093 shutdown_connections 2024-03-28T23:35:01.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.098+0000 7f1e5463b640 1 -- 172.21.15.161:0/2184183093 wait complete. 2024-03-28T23:35:01.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.099+0000 7f1e5463b640 1 Processor -- start 2024-03-28T23:35:01.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.099+0000 7f1e5463b640 1 -- start start 2024-03-28T23:35:01.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.100+0000 7f1e5463b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e4c0c5410 0x7f1e4c0c0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:01.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.100+0000 7f1e5463b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e4c0c5850 0x7f1e4c0c1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:01.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.100+0000 7f1e5463b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e4c152890 0x7f1e4c153e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.100+0000 7f1e5463b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4c1592d0 con 0x7f1e4c152890 2024-03-28T23:35:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.100+0000 7f1e5463b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4c159640 con 0x7f1e4c0c5850 2024-03-28T23:35:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.100+0000 7f1e5463b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1e4c155ff0 con 0x7f1e4c0c5410 2024-03-28T23:35:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.100+0000 7f1e52bd8640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e4c152890 0x7f1e4c153e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.101+0000 7f1e52bd8640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e4c152890 0x7f1e4c153e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57386/0 (socket says 172.21.15.161:57386) 2024-03-28T23:35:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.101+0000 7f1e52bd8640 1 -- 172.21.15.161:0/2449709156 learned_addr learned my addr 172.21.15.161:0/2449709156 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.101+0000 7f1e523d7640 1 --2- 172.21.15.161:0/2449709156 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e4c0c5410 0x7f1e4c0c0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.101+0000 7f1e51bd6640 1 --2- 172.21.15.161:0/2449709156 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e4c0c5850 0x7f1e4c0c1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.101+0000 7f1e52bd8640 1 -- 172.21.15.161:0/2449709156 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e4c0c5410 msgr2=0x7f1e4c0c0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.101+0000 7f1e52bd8640 1 --2- 172.21.15.161:0/2449709156 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e4c0c5410 0x7f1e4c0c0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.101+0000 7f1e52bd8640 1 -- 172.21.15.161:0/2449709156 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e4c0c5850 msgr2=0x7f1e4c0c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.101+0000 7f1e52bd8640 1 --2- 172.21.15.161:0/2449709156 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e4c0c5850 0x7f1e4c0c1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.101+0000 7f1e52bd8640 1 -- 172.21.15.161:0/2449709156 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e4c153280 con 0x7f1e4c152890 2024-03-28T23:35:01.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.101+0000 7f1e51bd6640 1 --2- 172.21.15.161:0/2449709156 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e4c0c5850 0x7f1e4c0c1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:01.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.101+0000 7f1e52bd8640 1 --2- 172.21.15.161:0/2449709156 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e4c152890 0x7f1e4c153e10 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c009720 tx=0x7f1e3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:01.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.101+0000 7f1e4affd640 1 -- 172.21.15.161:0/2449709156 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c037040 con 0x7f1e4c152890 2024-03-28T23:35:01.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.101+0000 7f1e4affd640 1 -- 172.21.15.161:0/2449709156 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e3c004230 con 0x7f1e4c152890 2024-03-28T23:35:01.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.101+0000 7f1e4affd640 1 -- 172.21.15.161:0/2449709156 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c03e670 con 0x7f1e4c152890 2024-03-28T23:35:01.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.101+0000 7f1e5463b640 1 -- 172.21.15.161:0/2449709156 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e4c1ef0e0 con 0x7f1e4c152890 2024-03-28T23:35:01.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.101+0000 7f1e5463b640 1 -- 172.21.15.161:0/2449709156 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e4c1ee620 con 0x7f1e4c152890 2024-03-28T23:35:01.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.102+0000 7f1e5463b640 1 -- 172.21.15.161:0/2449709156 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e4c04f4b0 con 0x7f1e4c152890 2024-03-28T23:35:01.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.103+0000 7f1e4affd640 1 -- 172.21.15.161:0/2449709156 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1e3c03a040 con 0x7f1e4c152890 2024-03-28T23:35:01.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.103+0000 7f1e4affd640 1 --2- 172.21.15.161:0/2449709156 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1e3806b4d0 0x7f1e3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:01.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.103+0000 7f1e523d7640 1 --2- 172.21.15.161:0/2449709156 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1e3806b4d0 0x7f1e3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:01.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.103+0000 7f1e4affd640 1 -- 172.21.15.161:0/2449709156 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 6273+0+0 (secure 0 0 0) 0x7f1e3c0abdf0 con 0x7f1e4c152890 2024-03-28T23:35:01.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.104+0000 7f1e523d7640 1 --2- 172.21.15.161:0/2449709156 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1e3806b4d0 0x7f1e3806d990 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f1e44004fd0 tx=0x7f1e44004d60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:01.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.108+0000 7f1e4affd640 1 -- 172.21.15.161:0/2449709156 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1e3c042020 con 0x7f1e4c152890 2024-03-28T23:35:01.258 INFO:tasks.ceph.mds.a.smithi161.stderr:2024-03-28T23:35:01.256+0000 7fcfc472a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:35:01.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.321+0000 7f1e5463b640 1 -- 172.21.15.161:0/2449709156 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e4c0ac400 con 0x7f1e4c152890 2024-03-28T23:35:01.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.322+0000 7f1e4affd640 1 -- 172.21.15.161:0/2449709156 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+5199 (secure 0 0 0) 0x7f1e3c07a510 con 0x7f1e4c152890 2024-03-28T23:35:01.323 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:01.323 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":117,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7318,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3734750343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3734750343},{"type":"v1","addr":"172.21.15.170:6835","nonce":3734750343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/4140438862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":4140438862},{"type":"v1","addr":"172.21.15.170:6834","nonce":4140438862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":8220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/896089063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":896089063},{"type":"v1","addr":"172.21.15.161:6835","nonce":896089063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:34:36.796608+0000","modified":"2024-03-28T23:34:56.277306+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"a","rank":0,"incarnation":114,"state":"up:active","state_seq":4,"addr":"172.21.15.161:6837/3929102513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":3929102513},{"type":"v1","addr":"172.21.15.161:6837","nonce":3929102513}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8199,"qdb_cluster":[8199]},"id":9}]} 2024-03-28T23:35:01.324 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 117 2024-03-28T23:35:01.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.324+0000 7f1e5463b640 1 -- 172.21.15.161:0/2449709156 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1e3806b4d0 msgr2=0x7f1e3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:01.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.324+0000 7f1e5463b640 1 --2- 172.21.15.161:0/2449709156 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1e3806b4d0 0x7f1e3806d990 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f1e44004fd0 tx=0x7f1e44004d60 comp rx=0 tx=0).stop 2024-03-28T23:35:01.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.324+0000 7f1e5463b640 1 -- 172.21.15.161:0/2449709156 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e4c152890 msgr2=0x7f1e4c153e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:01.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.324+0000 7f1e5463b640 1 --2- 172.21.15.161:0/2449709156 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e4c152890 0x7f1e4c153e10 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c009720 tx=0x7f1e3c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:35:01.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.324+0000 7f1e5463b640 1 -- 172.21.15.161:0/2449709156 shutdown_connections 2024-03-28T23:35:01.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.324+0000 7f1e5463b640 1 --2- 172.21.15.161:0/2449709156 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1e3806b4d0 0x7f1e3806d990 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:01.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.324+0000 7f1e5463b640 1 --2- 172.21.15.161:0/2449709156 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e4c152890 0x7f1e4c153e10 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:01.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.324+0000 7f1e5463b640 1 --2- 172.21.15.161:0/2449709156 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e4c0c5850 0x7f1e4c0c1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:01.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.324+0000 7f1e5463b640 1 --2- 172.21.15.161:0/2449709156 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e4c0c5410 0x7f1e4c0c0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:01.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.324+0000 7f1e5463b640 1 -- 172.21.15.161:0/2449709156 >> 172.21.15.161:0/2449709156 conn(0x7f1e4c0ac620 msgr2=0x7f1e4c06bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:01.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.324+0000 7f1e5463b640 1 -- 172.21.15.161:0/2449709156 shutdown_connections 2024-03-28T23:35:01.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.324+0000 7f1e5463b640 1 -- 172.21.15.161:0/2449709156 wait complete. 2024-03-28T23:35:01.336 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:35:01.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.437+0000 7f86f6823640 1 Processor -- start 2024-03-28T23:35:01.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.437+0000 7f86f6823640 1 -- start start 2024-03-28T23:35:01.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.438+0000 7f86f6823640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86f006fc60 0x7f86f00700e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:01.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.438+0000 7f86f6823640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86f0153050 con 0x7f86f0072810 2024-03-28T23:35:01.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.438+0000 7f86f6823640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86f00c3f10 con 0x7f86f00c3390 2024-03-28T23:35:01.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.438+0000 7f86f6823640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f86f00a8520 con 0x7f86f006fc60 2024-03-28T23:35:01.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.438+0000 7f86effff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f86f0072810 0x7f86f00c4840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45272/0 (socket says 172.21.15.161:45272) 2024-03-28T23:35:01.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.438+0000 7f86effff640 1 -- 172.21.15.161:0/4090768321 learned_addr learned my addr 172.21.15.161:0/4090768321 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:01.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.438+0000 7f86ef7fe640 1 --2- 172.21.15.161:0/4090768321 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86f006fc60 0x7f86f00700e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:01.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.438+0000 7f86eeffd640 1 -- 172.21.15.161:0/4090768321 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2417362223 0 0) 0x7f86f0153050 con 0x7f86f0072810 2024-03-28T23:35:01.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.438+0000 7f86eeffd640 1 -- 172.21.15.161:0/4090768321 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86d8003520 con 0x7f86f0072810 2024-03-28T23:35:01.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.438+0000 7f86eeffd640 1 -- 172.21.15.161:0/4090768321 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3435997967 0 0) 0x7f86f00c3f10 con 0x7f86f00c3390 2024-03-28T23:35:01.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.439+0000 7f86eeffd640 1 -- 172.21.15.161:0/4090768321 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86f0153050 con 0x7f86f00c3390 2024-03-28T23:35:01.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.439+0000 7f86eeffd640 1 -- 172.21.15.161:0/4090768321 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 240786210 0 0) 0x7f86d8003520 con 0x7f86f0072810 2024-03-28T23:35:01.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.439+0000 7f86eeffd640 1 -- 172.21.15.161:0/4090768321 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86f006fc60 msgr2=0x7f86f00700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:01.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.439+0000 7f86eeffd640 1 --2- 172.21.15.161:0/4090768321 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86f006fc60 0x7f86f00700e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:01.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.439+0000 7f86eeffd640 1 -- 172.21.15.161:0/4090768321 >> v1:172.21.15.170:6789/0 conn(0x7f86f00c3390 legacy=0x7f86f00c3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:01.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.439+0000 7f86eeffd640 1 -- 172.21.15.161:0/4090768321 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f0153310 con 0x7f86f0072810 2024-03-28T23:35:01.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.439+0000 7f86eeffd640 1 -- 172.21.15.161:0/4090768321 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f86e4001670 con 0x7f86f0072810 2024-03-28T23:35:01.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.439+0000 7f86ef7fe640 1 --2- 172.21.15.161:0/4090768321 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86f006fc60 0x7f86f00700e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:01.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.439+0000 7f86eeffd640 1 -- 172.21.15.161:0/4090768321 >> v1:172.21.15.161:6789/0 conn(0x7f86f0072810 legacy=0x7f86f00c4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:01.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.439+0000 7f86eeffd640 1 --2- 172.21.15.161:0/4090768321 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86f006fc60 0x7f86d8003c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:01.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.439+0000 7f86eeffd640 1 --2- 172.21.15.161:0/4090768321 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86d80041e0 0x7f86d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:01.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.439+0000 7f86eeffd640 1 --2- 172.21.15.161:0/4090768321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86d8004d40 0x7f86d80051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:01.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.439+0000 7f86eeffd640 1 -- 172.21.15.161:0/4090768321 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f86d800bc20 con 0x7f86d8004d40 2024-03-28T23:35:01.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.439+0000 7f86eeffd640 1 -- 172.21.15.161:0/4090768321 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f86d800bdd0 con 0x7f86d80041e0 2024-03-28T23:35:01.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.439+0000 7f86eeffd640 1 -- 172.21.15.161:0/4090768321 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f86d800bfd0 con 0x7f86f006fc60 2024-03-28T23:35:01.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.439+0000 7f86f4dc0640 1 --2- 172.21.15.161:0/4090768321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86d8004d40 0x7f86d80051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:01.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.439+0000 7f86effff640 1 --2- 172.21.15.161:0/4090768321 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86f006fc60 0x7f86d8003c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:01.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.439+0000 7f86ef7fe640 1 --2- 172.21.15.161:0/4090768321 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86d80041e0 0x7f86d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:01.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.439+0000 7f86f4dc0640 1 -- 172.21.15.161:0/4090768321 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86f006fc60 msgr2=0x7f86d8003c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:01.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.439+0000 7f86f4dc0640 1 --2- 172.21.15.161:0/4090768321 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86f006fc60 0x7f86d8003c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:01.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.439+0000 7f86f4dc0640 1 -- 172.21.15.161:0/4090768321 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86d80041e0 msgr2=0x7f86d8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:01.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.439+0000 7f86f4dc0640 1 --2- 172.21.15.161:0/4090768321 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86d80041e0 0x7f86d8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:01.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.439+0000 7f86f4dc0640 1 -- 172.21.15.161:0/4090768321 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86d8003520 con 0x7f86d8004d40 2024-03-28T23:35:01.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.439+0000 7f86ef7fe640 1 --2- 172.21.15.161:0/4090768321 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86d80041e0 0x7f86d8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:01.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.440+0000 7f86f4dc0640 1 --2- 172.21.15.161:0/4090768321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86d8004d40 0x7f86d80051a0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f86e8001fd0 tx=0x7f86d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a6b3d1a2345aceb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:01.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.440+0000 7f86eeffd640 1 -- 172.21.15.161:0/4090768321 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e8001bf0 con 0x7f86d8004d40 2024-03-28T23:35:01.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.440+0000 7f86eeffd640 1 -- 172.21.15.161:0/4090768321 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86e8003b70 con 0x7f86d8004d40 2024-03-28T23:35:01.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.440+0000 7f86eeffd640 1 -- 172.21.15.161:0/4090768321 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e80050d0 con 0x7f86d8004d40 2024-03-28T23:35:01.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.440+0000 7f86f6823640 1 -- 172.21.15.161:0/4090768321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86d8004d40 msgr2=0x7f86d80051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:01.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.440+0000 7f86f6823640 1 --2- 172.21.15.161:0/4090768321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86d8004d40 0x7f86d80051a0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f86e8001fd0 tx=0x7f86d8005830 comp rx=0 tx=0).stop 2024-03-28T23:35:01.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.440+0000 7f86f4dc0640 1 -- 172.21.15.161:0/4090768321 reap_dead start 2024-03-28T23:35:01.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.441+0000 7f86f6823640 1 -- 172.21.15.161:0/4090768321 shutdown_connections 2024-03-28T23:35:01.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.441+0000 7f86f6823640 1 -- 172.21.15.161:0/4090768321 >> 172.21.15.161:0/4090768321 conn(0x7f86f00aa740 msgr2=0x7f86f00aab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:01.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.441+0000 7f86f6823640 1 -- 172.21.15.161:0/4090768321 shutdown_connections 2024-03-28T23:35:01.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.441+0000 7f86f6823640 1 -- 172.21.15.161:0/4090768321 wait complete. 2024-03-28T23:35:01.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.442+0000 7f86f6823640 1 Processor -- start 2024-03-28T23:35:01.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.442+0000 7f86f6823640 1 -- start start 2024-03-28T23:35:01.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.442+0000 7f86f6823640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86d8004d40 0x7f86f0156250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:01.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.442+0000 7f86f6823640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86f0155850 0x7f86f0156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:01.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.442+0000 7f86f6823640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86f0148390 0x7f86f0148810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:01.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.443+0000 7f86f6823640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f86f0153bb0 con 0x7f86d8004d40 2024-03-28T23:35:01.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.443+0000 7f86f6823640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f86f01538d0 con 0x7f86f0155850 2024-03-28T23:35:01.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.443+0000 7f86f6823640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f86f00c4f50 con 0x7f86f0148390 2024-03-28T23:35:01.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.443+0000 7f86effff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86d8004d40 0x7f86f0156250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:01.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.443+0000 7f86effff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86d8004d40 0x7f86f0156250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57408/0 (socket says 172.21.15.161:57408) 2024-03-28T23:35:01.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.443+0000 7f86effff640 1 -- 172.21.15.161:0/1938708655 learned_addr learned my addr 172.21.15.161:0/1938708655 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:01.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.443+0000 7f86effff640 1 -- 172.21.15.161:0/1938708655 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86f0148390 msgr2=0x7f86f0148810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:01.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.443+0000 7f86effff640 1 --2- 172.21.15.161:0/1938708655 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86f0148390 0x7f86f0148810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:01.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.443+0000 7f86effff640 1 -- 172.21.15.161:0/1938708655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86f0155850 msgr2=0x7f86f0156790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:01.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.443+0000 7f86ef7fe640 1 --2- 172.21.15.161:0/1938708655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86f0155850 0x7f86f0156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:01.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.443+0000 7f86effff640 1 --2- 172.21.15.161:0/1938708655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86f0155850 0x7f86f0156790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:01.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.443+0000 7f86effff640 1 -- 172.21.15.161:0/1938708655 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f01490a0 con 0x7f86d8004d40 2024-03-28T23:35:01.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.443+0000 7f86effff640 1 --2- 172.21.15.161:0/1938708655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86d8004d40 0x7f86f0156250 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f86e40017e0 tx=0x7f86e4007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:01.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.443+0000 7f86ecff9640 1 -- 172.21.15.161:0/1938708655 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e400f7b0 con 0x7f86d8004d40 2024-03-28T23:35:01.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.443+0000 7f86ecff9640 1 -- 172.21.15.161:0/1938708655 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86e4018070 con 0x7f86d8004d40 2024-03-28T23:35:01.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.443+0000 7f86f6823640 1 -- 172.21.15.161:0/1938708655 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86f01543f0 con 0x7f86d8004d40 2024-03-28T23:35:01.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.443+0000 7f86ecff9640 1 -- 172.21.15.161:0/1938708655 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e400fb70 con 0x7f86d8004d40 2024-03-28T23:35:01.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.443+0000 7f86f6823640 1 -- 172.21.15.161:0/1938708655 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86f0148d50 con 0x7f86d8004d40 2024-03-28T23:35:01.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.445+0000 7f86f6823640 1 -- 172.21.15.161:0/1938708655 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86f004f4b0 con 0x7f86d8004d40 2024-03-28T23:35:01.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.445+0000 7f86ecff9640 1 -- 172.21.15.161:0/1938708655 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f86e400fd10 con 0x7f86d8004d40 2024-03-28T23:35:01.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.445+0000 7f86ecff9640 1 --2- 172.21.15.161:0/1938708655 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f86dc06b500 0x7f86dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:01.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.445+0000 7f86ecff9640 1 -- 172.21.15.161:0/1938708655 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 6273+0+0 (secure 0 0 0) 0x7f86e408a180 con 0x7f86d8004d40 2024-03-28T23:35:01.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.445+0000 7f86ef7fe640 1 --2- 172.21.15.161:0/1938708655 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f86dc06b500 0x7f86dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:01.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.445+0000 7f86ef7fe640 1 --2- 172.21.15.161:0/1938708655 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f86dc06b500 0x7f86dc06d9c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f86f014e8d0 tx=0x7f86e0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:01.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.452+0000 7f86ecff9640 1 -- 172.21.15.161:0/1938708655 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f86e4058e50 con 0x7f86d8004d40 2024-03-28T23:35:01.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.665+0000 7f86f6823640 1 -- 172.21.15.161:0/1938708655 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86f00c78f0 con 0x7f86d8004d40 2024-03-28T23:35:01.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.665+0000 7f86ecff9640 1 -- 172.21.15.161:0/1938708655 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+5199 (secure 0 0 0) 0x7f86e4058750 con 0x7f86d8004d40 2024-03-28T23:35:01.667 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:01.667 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":117,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7318,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3734750343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3734750343},{"type":"v1","addr":"172.21.15.170:6835","nonce":3734750343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/4140438862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":4140438862},{"type":"v1","addr":"172.21.15.170:6834","nonce":4140438862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":8220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/896089063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":896089063},{"type":"v1","addr":"172.21.15.161:6835","nonce":896089063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:34:36.796608+0000","modified":"2024-03-28T23:34:56.277306+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"a","rank":0,"incarnation":114,"state":"up:active","state_seq":4,"addr":"172.21.15.161:6837/3929102513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":3929102513},{"type":"v1","addr":"172.21.15.161:6837","nonce":3929102513}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8199,"qdb_cluster":[8199]},"id":9}]} 2024-03-28T23:35:01.667 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 117 2024-03-28T23:35:01.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.667+0000 7f86f6823640 1 -- 172.21.15.161:0/1938708655 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f86dc06b500 msgr2=0x7f86dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:01.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.667+0000 7f86f6823640 1 --2- 172.21.15.161:0/1938708655 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f86dc06b500 0x7f86dc06d9c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f86f014e8d0 tx=0x7f86e0008040 comp rx=0 tx=0).stop 2024-03-28T23:35:01.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.667+0000 7f86f6823640 1 -- 172.21.15.161:0/1938708655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86d8004d40 msgr2=0x7f86f0156250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:01.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.667+0000 7f86f6823640 1 --2- 172.21.15.161:0/1938708655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86d8004d40 0x7f86f0156250 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f86e40017e0 tx=0x7f86e4007430 comp rx=0 tx=0).stop 2024-03-28T23:35:01.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.667+0000 7f86f6823640 1 -- 172.21.15.161:0/1938708655 shutdown_connections 2024-03-28T23:35:01.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.667+0000 7f86f6823640 1 --2- 172.21.15.161:0/1938708655 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f86dc06b500 0x7f86dc06d9c0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:01.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.667+0000 7f86f6823640 1 --2- 172.21.15.161:0/1938708655 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86f0148390 0x7f86f0148810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:01.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.667+0000 7f86f6823640 1 --2- 172.21.15.161:0/1938708655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86f0155850 0x7f86f0156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:01.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.667+0000 7f86f6823640 1 --2- 172.21.15.161:0/1938708655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86d8004d40 0x7f86f0156250 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:01.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.667+0000 7f86f6823640 1 -- 172.21.15.161:0/1938708655 >> 172.21.15.161:0/1938708655 conn(0x7f86f00aa740 msgr2=0x7f86f00aab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:01.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.667+0000 7f86f6823640 1 -- 172.21.15.161:0/1938708655 shutdown_connections 2024-03-28T23:35:01.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.667+0000 7f86f6823640 1 -- 172.21.15.161:0/1938708655 wait complete. 2024-03-28T23:35:01.681 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:35:02.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.781+0000 7f154f8a4640 1 Processor -- start 2024-03-28T23:35:02.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.782+0000 7f154f8a4640 1 -- start start 2024-03-28T23:35:02.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.782+0000 7f154f8a4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1548166000 0x7f15480c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:02.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.782+0000 7f154f8a4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f15480a9680 0x7f15480a9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:02.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.782+0000 7f154f8a4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15480aa010 con 0x7f15480c37a0 2024-03-28T23:35:02.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.782+0000 7f154f8a4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f15480c1820 con 0x7f15480a9680 2024-03-28T23:35:02.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.782+0000 7f154f8a4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f15480c1b00 con 0x7f1548166000 2024-03-28T23:35:02.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.782+0000 7f154de41640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f15480a9680 0x7f15480a9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:02.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.782+0000 7f154d640640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f15480c37a0 0x7f15481629d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45282/0 (socket says 172.21.15.161:45282) 2024-03-28T23:35:02.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.782+0000 7f154d640640 1 -- 172.21.15.161:0/3697150947 learned_addr learned my addr 172.21.15.161:0/3697150947 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:02.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.783+0000 7f1547fff640 1 -- 172.21.15.161:0/3697150947 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1078913500 0 0) 0x7f15480aa010 con 0x7f15480c37a0 2024-03-28T23:35:02.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:01.783+0000 7f154ce3f640 1 --2- 172.21.15.161:0/3697150947 >> v2:172.21.15.170:3300/0 conn(0x7f1548166000 0x7f15480c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:02.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.037+0000 7f1547fff640 1 -- 172.21.15.161:0/3697150947 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1530003500 con 0x7f15480c37a0 2024-03-28T23:35:02.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.037+0000 7f154de41640 1 -- 172.21.15.161:0/3697150947 >> v2:172.21.15.170:3300/0 conn(0x7f1548166000 msgr2=0x7f15480c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:02.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.037+0000 7f154de41640 1 --2- 172.21.15.161:0/3697150947 >> v2:172.21.15.170:3300/0 conn(0x7f1548166000 0x7f15480c2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.037+0000 7f154de41640 1 -- 172.21.15.161:0/3697150947 >> v1:172.21.15.161:6789/0 conn(0x7f15480c37a0 legacy=0x7f15481629d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:02.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.037+0000 7f1547fff640 1 -- 172.21.15.161:0/3697150947 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2025451117 0 0) 0x7f1530003500 con 0x7f15480c37a0 2024-03-28T23:35:02.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.037+0000 7f154de41640 1 -- 172.21.15.161:0/3697150947 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15480c1de0 con 0x7f15480a9680 2024-03-28T23:35:02.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.038+0000 7f154de41640 1 --2- 172.21.15.161:0/3697150947 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15480a9680 0x7f15480a9ad0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f15480c10e0 tx=0x7f153802bcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6826a34be0bccb0c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:02.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.038+0000 7f1547fff640 1 -- 172.21.15.161:0/3697150947 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1538039070 con 0x7f15480a9680 2024-03-28T23:35:02.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.038+0000 7f1547fff640 1 -- 172.21.15.161:0/3697150947 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f153802c7f0 con 0x7f15480a9680 2024-03-28T23:35:02.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.038+0000 7f1547fff640 1 -- 172.21.15.161:0/3697150947 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15400019f0 con 0x7f15480a9680 2024-03-28T23:35:02.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.039+0000 7f154f8a4640 1 -- 172.21.15.161:0/3697150947 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15480a9680 msgr2=0x7f15480a9ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:02.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.039+0000 7f154f8a4640 1 --2- 172.21.15.161:0/3697150947 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15480a9680 0x7f15480a9ad0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f15480c10e0 tx=0x7f153802bcc0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.039+0000 7f154f8a4640 1 -- 172.21.15.161:0/3697150947 shutdown_connections 2024-03-28T23:35:02.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.039+0000 7f154f8a4640 1 --2- 172.21.15.161:0/3697150947 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15480a9680 0x7f15480a9ad0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.039+0000 7f154f8a4640 1 --2- 172.21.15.161:0/3697150947 >> v2:172.21.15.170:3300/0 conn(0x7f1548166000 0x7f15480c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.039+0000 7f154f8a4640 1 -- 172.21.15.161:0/3697150947 >> 172.21.15.161:0/3697150947 conn(0x7f154806d8e0 msgr2=0x7f154806dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:02.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.039+0000 7f154f8a4640 1 -- 172.21.15.161:0/3697150947 shutdown_connections 2024-03-28T23:35:02.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.039+0000 7f154f8a4640 1 -- 172.21.15.161:0/3697150947 wait complete. 2024-03-28T23:35:02.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.040+0000 7f154f8a4640 1 Processor -- start 2024-03-28T23:35:02.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.040+0000 7f154f8a4640 1 -- start start 2024-03-28T23:35:02.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.041+0000 7f154f8a4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15480a9680 0x7f15481fb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:02.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.041+0000 7f154f8a4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15480c37a0 0x7f15481f8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:02.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.041+0000 7f154f8a4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1548166000 0x7f1548162a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:02.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.041+0000 7f154f8a4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f15480c25e0 con 0x7f1548166000 2024-03-28T23:35:02.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.041+0000 7f154f8a4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f154808b490 con 0x7f15480c37a0 2024-03-28T23:35:02.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.041+0000 7f154f8a4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f15481fb3b0 con 0x7f15480a9680 2024-03-28T23:35:02.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.041+0000 7f154de41640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1548166000 0x7f1548162a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:02.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.041+0000 7f154de41640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1548166000 0x7f1548162a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57430/0 (socket says 172.21.15.161:57430) 2024-03-28T23:35:02.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.041+0000 7f154de41640 1 -- 172.21.15.161:0/2431750013 learned_addr learned my addr 172.21.15.161:0/2431750013 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:02.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.041+0000 7f154ce3f640 1 --2- 172.21.15.161:0/2431750013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15480c37a0 0x7f15481f8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:02.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.041+0000 7f154d640640 1 --2- 172.21.15.161:0/2431750013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15480a9680 0x7f15481fb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:02.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.041+0000 7f154de41640 1 -- 172.21.15.161:0/2431750013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15480a9680 msgr2=0x7f15481fb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:02.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.041+0000 7f154de41640 1 --2- 172.21.15.161:0/2431750013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15480a9680 0x7f15481fb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.041+0000 7f154de41640 1 -- 172.21.15.161:0/2431750013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15480c37a0 msgr2=0x7f15481f8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:02.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.041+0000 7f154de41640 1 --2- 172.21.15.161:0/2431750013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15480c37a0 0x7f15481f8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.041+0000 7f154de41640 1 -- 172.21.15.161:0/2431750013 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15481f8690 con 0x7f1548166000 2024-03-28T23:35:02.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.041+0000 7f154ce3f640 1 --2- 172.21.15.161:0/2431750013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15480c37a0 0x7f15481f8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:02.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.041+0000 7f154d640640 1 --2- 172.21.15.161:0/2431750013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15480a9680 0x7f15481fb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:02.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.041+0000 7f154de41640 1 --2- 172.21.15.161:0/2431750013 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1548166000 0x7f1548162a10 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f153800b940 tx=0x7f15380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:02.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.041+0000 7f1545ffb640 1 -- 172.21.15.161:0/2431750013 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1538042020 con 0x7f1548166000 2024-03-28T23:35:02.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.041+0000 7f1545ffb640 1 -- 172.21.15.161:0/2431750013 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f153803a040 con 0x7f1548166000 2024-03-28T23:35:02.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.042+0000 7f154f8a4640 1 -- 172.21.15.161:0/2431750013 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15480bd130 con 0x7f1548166000 2024-03-28T23:35:02.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.042+0000 7f1545ffb640 1 -- 172.21.15.161:0/2431750013 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1538039070 con 0x7f1548166000 2024-03-28T23:35:02.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.042+0000 7f154f8a4640 1 -- 172.21.15.161:0/2431750013 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15480bd6c0 con 0x7f1548166000 2024-03-28T23:35:02.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.043+0000 7f154f8a4640 1 -- 172.21.15.161:0/2431750013 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f154804f4b0 con 0x7f1548166000 2024-03-28T23:35:02.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.043+0000 7f1545ffb640 1 -- 172.21.15.161:0/2431750013 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1538045020 con 0x7f1548166000 2024-03-28T23:35:02.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.043+0000 7f1545ffb640 1 --2- 172.21.15.161:0/2431750013 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f153406b4d0 0x7f153406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:02.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.044+0000 7f1545ffb640 1 -- 172.21.15.161:0/2431750013 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 6273+0+0 (secure 0 0 0) 0x7f15380ac180 con 0x7f1548166000 2024-03-28T23:35:02.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.044+0000 7f154d640640 1 --2- 172.21.15.161:0/2431750013 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f153406b4d0 0x7f153406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:02.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.044+0000 7f154d640640 1 --2- 172.21.15.161:0/2431750013 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f153406b4d0 0x7f153406d990 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f1540002230 tx=0x7f1540009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:02.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.048+0000 7f1545ffb640 1 -- 172.21.15.161:0/2431750013 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f153804c030 con 0x7f1548166000 2024-03-28T23:35:02.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.224+0000 7f154f8a4640 1 -- 172.21.15.161:0/2431750013 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f15480c1460 con 0x7f1548166000 2024-03-28T23:35:02.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.225+0000 7f1545ffb640 1 -- 172.21.15.161:0/2431750013 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) ==== 74+0+17197 (secure 0 0 0) 0x7f15481f8690 con 0x7f1548166000 2024-03-28T23:35:02.227 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:02.227 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":124,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:34:53.266559+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:34:35.025049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-28T23:34:35.765110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"122","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6835/878057589":"2024-03-29T23:34:53.266532+0000","172.21.15.161:6834/878057589":"2024-03-29T23:34:53.266532+0000","172.21.15.161:6837/3432367324":"2024-03-29T23:34:50.245540+0000","172.21.15.161:6836/3432367324":"2024-03-29T23:34:50.245540+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:35:02.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.227+0000 7f154f8a4640 1 -- 172.21.15.161:0/2431750013 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f153406b4d0 msgr2=0x7f153406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:02.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.227+0000 7f154f8a4640 1 --2- 172.21.15.161:0/2431750013 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f153406b4d0 0x7f153406d990 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f1540002230 tx=0x7f1540009040 comp rx=0 tx=0).stop 2024-03-28T23:35:02.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.227+0000 7f154f8a4640 1 -- 172.21.15.161:0/2431750013 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1548166000 msgr2=0x7f1548162a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:02.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.227+0000 7f154f8a4640 1 --2- 172.21.15.161:0/2431750013 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1548166000 0x7f1548162a10 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f153800b940 tx=0x7f15380046c0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.228+0000 7f154f8a4640 1 -- 172.21.15.161:0/2431750013 shutdown_connections 2024-03-28T23:35:02.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.228+0000 7f154f8a4640 1 --2- 172.21.15.161:0/2431750013 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f153406b4d0 0x7f153406d990 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.228+0000 7f154f8a4640 1 --2- 172.21.15.161:0/2431750013 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1548166000 0x7f1548162a10 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.228+0000 7f154f8a4640 1 --2- 172.21.15.161:0/2431750013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15480c37a0 0x7f15481f8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.228+0000 7f154f8a4640 1 --2- 172.21.15.161:0/2431750013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15480a9680 0x7f15481fb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.228+0000 7f154f8a4640 1 -- 172.21.15.161:0/2431750013 >> 172.21.15.161:0/2431750013 conn(0x7f154806d8e0 msgr2=0x7f154806dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:02.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.228+0000 7f154f8a4640 1 -- 172.21.15.161:0/2431750013 shutdown_connections 2024-03-28T23:35:02.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.228+0000 7f154f8a4640 1 -- 172.21.15.161:0/2431750013 wait complete. 2024-03-28T23:35:02.243 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-28T23:35:02.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.345+0000 7f6582bb5640 1 Processor -- start 2024-03-28T23:35:02.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.345+0000 7f6582bb5640 1 -- start start 2024-03-28T23:35:02.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.346+0000 7f6582bb5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f657c150600 0x7f657c0a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:02.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.346+0000 7f6582bb5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f657c0a8fc0 0x7f657c153270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:02.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.346+0000 7f6582bb5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f657c1567c0 con 0x7f657c0aacb0 2024-03-28T23:35:02.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.346+0000 7f6582bb5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f657c156570 con 0x7f657c0a8fc0 2024-03-28T23:35:02.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.346+0000 7f6582bb5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f657c156aa0 con 0x7f657c150600 2024-03-28T23:35:02.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.346+0000 7f657bfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f657c0a8fc0 0x7f657c153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:02.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.346+0000 7f6581152640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f657c0aacb0 0x7f657c1537b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:40364/0 (socket says 172.21.15.161:40364) 2024-03-28T23:35:02.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.346+0000 7f6581152640 1 -- 172.21.15.161:0/1231289180 learned_addr learned my addr 172.21.15.161:0/1231289180 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:02.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.346+0000 7f6580951640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f657c150600 0x7f657c0a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:02.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.346+0000 7f657bfff640 1 -- 172.21.15.161:0/1231289180 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f657c150600 msgr2=0x7f657c0a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:02.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.346+0000 7f657bfff640 1 --2- 172.21.15.161:0/1231289180 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f657c150600 0x7f657c0a9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.346+0000 7f657bfff640 1 -- 172.21.15.161:0/1231289180 >> v1:172.21.15.170:6789/0 conn(0x7f657c0aacb0 legacy=0x7f657c1537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:02.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.346+0000 7f657bfff640 1 -- 172.21.15.161:0/1231289180 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f657c156d80 con 0x7f657c0a8fc0 2024-03-28T23:35:02.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.346+0000 7f657b7fe640 1 -- 172.21.15.161:0/1231289180 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 348344318 0 0) 0x7f657c1567c0 con 0x7f657c0aacb0 2024-03-28T23:35:02.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.346+0000 7f6580951640 1 --2- 172.21.15.161:0/1231289180 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f657c150600 0x7f657c0a9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:02.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.347+0000 7f657bfff640 1 --2- 172.21.15.161:0/1231289180 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f657c0a8fc0 0x7f657c153270 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f65680097a0 tx=0x7f656802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34c62bf8a9c998c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:02.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.347+0000 7f657b7fe640 1 -- 172.21.15.161:0/1231289180 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65680315a0 con 0x7f657c0a8fc0 2024-03-28T23:35:02.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.347+0000 7f657b7fe640 1 -- 172.21.15.161:0/1231289180 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6568037070 con 0x7f657c0a8fc0 2024-03-28T23:35:02.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.347+0000 7f657b7fe640 1 -- 172.21.15.161:0/1231289180 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6568031960 con 0x7f657c0a8fc0 2024-03-28T23:35:02.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.348+0000 7f6582bb5640 1 -- 172.21.15.161:0/1231289180 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f657c0a8fc0 msgr2=0x7f657c153270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:02.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.348+0000 7f6582bb5640 1 --2- 172.21.15.161:0/1231289180 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f657c0a8fc0 0x7f657c153270 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f65680097a0 tx=0x7f656802b840 comp rx=0 tx=0).stop 2024-03-28T23:35:02.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.348+0000 7f6582bb5640 1 -- 172.21.15.161:0/1231289180 shutdown_connections 2024-03-28T23:35:02.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.348+0000 7f6582bb5640 1 --2- 172.21.15.161:0/1231289180 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f657c0a8fc0 0x7f657c153270 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.348+0000 7f6582bb5640 1 --2- 172.21.15.161:0/1231289180 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f657c150600 0x7f657c0a9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.348+0000 7f6582bb5640 1 -- 172.21.15.161:0/1231289180 >> 172.21.15.161:0/1231289180 conn(0x7f657c0c6670 msgr2=0x7f657c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:02.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.348+0000 7f6582bb5640 1 -- 172.21.15.161:0/1231289180 shutdown_connections 2024-03-28T23:35:02.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.348+0000 7f6582bb5640 1 -- 172.21.15.161:0/1231289180 wait complete. 2024-03-28T23:35:02.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.349+0000 7f6582bb5640 1 Processor -- start 2024-03-28T23:35:02.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.349+0000 7f6582bb5640 1 -- start start 2024-03-28T23:35:02.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.350+0000 7f6582bb5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f657c0a8fc0 0x7f657c1522d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:02.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.350+0000 7f6582bb5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f657c0aacb0 0x7f657c1e59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:02.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.350+0000 7f6582bb5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f657c150600 0x7f657c1e5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:02.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.350+0000 7f6582bb5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f657c153ec0 con 0x7f657c0aacb0 2024-03-28T23:35:02.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.350+0000 7f6582bb5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f657c157340 con 0x7f657c0a8fc0 2024-03-28T23:35:02.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.350+0000 7f6582bb5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f657c1509e0 con 0x7f657c150600 2024-03-28T23:35:02.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.350+0000 7f657bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f657c0aacb0 0x7f657c1e59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:02.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.350+0000 7f657bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f657c0aacb0 0x7f657c1e59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57442/0 (socket says 172.21.15.161:57442) 2024-03-28T23:35:02.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.350+0000 7f657bfff640 1 -- 172.21.15.161:0/2339017778 learned_addr learned my addr 172.21.15.161:0/2339017778 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:02.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.350+0000 7f6580951640 1 --2- 172.21.15.161:0/2339017778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f657c0a8fc0 0x7f657c1522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:02.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.350+0000 7f6581152640 1 --2- 172.21.15.161:0/2339017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f657c150600 0x7f657c1e5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:02.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.350+0000 7f657bfff640 1 -- 172.21.15.161:0/2339017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f657c150600 msgr2=0x7f657c1e5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:02.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.350+0000 7f657bfff640 1 --2- 172.21.15.161:0/2339017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f657c150600 0x7f657c1e5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.350+0000 7f657bfff640 1 -- 172.21.15.161:0/2339017778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f657c0a8fc0 msgr2=0x7f657c1522d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:02.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.350+0000 7f657bfff640 1 --2- 172.21.15.161:0/2339017778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f657c0a8fc0 0x7f657c1522d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.350+0000 7f657bfff640 1 -- 172.21.15.161:0/2339017778 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f657c155030 con 0x7f657c0aacb0 2024-03-28T23:35:02.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.350+0000 7f6580951640 1 --2- 172.21.15.161:0/2339017778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f657c0a8fc0 0x7f657c1522d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:02.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.351+0000 7f657bfff640 1 --2- 172.21.15.161:0/2339017778 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f657c0aacb0 0x7f657c1e59d0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f6568034040 tx=0x7f65680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:02.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.351+0000 7f65797fa640 1 -- 172.21.15.161:0/2339017778 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6568004230 con 0x7f657c0aacb0 2024-03-28T23:35:02.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.351+0000 7f65797fa640 1 -- 172.21.15.161:0/2339017778 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65680043d0 con 0x7f657c0aacb0 2024-03-28T23:35:02.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.351+0000 7f6582bb5640 1 -- 172.21.15.161:0/2339017778 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f657c156100 con 0x7f657c0aacb0 2024-03-28T23:35:02.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.351+0000 7f65797fa640 1 -- 172.21.15.161:0/2339017778 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6568003850 con 0x7f657c0aacb0 2024-03-28T23:35:02.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.351+0000 7f6582bb5640 1 -- 172.21.15.161:0/2339017778 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f657c152810 con 0x7f657c0aacb0 2024-03-28T23:35:02.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.352+0000 7f6582bb5640 1 -- 172.21.15.161:0/2339017778 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f657c0abbc0 con 0x7f657c0aacb0 2024-03-28T23:35:02.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.352+0000 7f65797fa640 1 -- 172.21.15.161:0/2339017778 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f65680039f0 con 0x7f657c0aacb0 2024-03-28T23:35:02.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.353+0000 7f65797fa640 1 --2- 172.21.15.161:0/2339017778 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f656006b4d0 0x7f656006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:02.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.353+0000 7f65797fa640 1 -- 172.21.15.161:0/2339017778 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 6273+0+0 (secure 0 0 0) 0x7f656802f030 con 0x7f657c0aacb0 2024-03-28T23:35:02.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.353+0000 7f6580951640 1 --2- 172.21.15.161:0/2339017778 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f656006b4d0 0x7f656006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:02.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.353+0000 7f6580951640 1 --2- 172.21.15.161:0/2339017778 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f656006b4d0 0x7f656006d990 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f6570009560 tx=0x7f6570009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:02.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.357+0000 7f65797fa640 1 -- 172.21.15.161:0/2339017778 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f657c0abbc0 con 0x7f657c0aacb0 2024-03-28T23:35:02.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.572+0000 7f6582bb5640 1 -- 172.21.15.161:0/2339017778 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f657c04f4b0 con 0x7f657c0aacb0 2024-03-28T23:35:02.574 INFO:tasks.ceph.mon.a.smithi161.stderr:2024-03-28T23:35:02.572+0000 7fdc7b85b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T23:35:02.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.574+0000 7f65797fa640 1 -- 172.21.15.161:0/2339017778 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v118) ==== 161+0+0 (secure 0 0 0) 0x7f6568041070 con 0x7f657c0aacb0 2024-03-28T23:35:02.575 INFO:teuthology.orchestra.run.smithi161.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-28T23:35:02.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.577+0000 7f6582bb5640 1 -- 172.21.15.161:0/2339017778 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f656006b4d0 msgr2=0x7f656006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:02.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.577+0000 7f6582bb5640 1 --2- 172.21.15.161:0/2339017778 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f656006b4d0 0x7f656006d990 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f6570009560 tx=0x7f6570009190 comp rx=0 tx=0).stop 2024-03-28T23:35:02.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.577+0000 7f6582bb5640 1 -- 172.21.15.161:0/2339017778 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f657c0aacb0 msgr2=0x7f657c1e59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:02.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.577+0000 7f6582bb5640 1 --2- 172.21.15.161:0/2339017778 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f657c0aacb0 0x7f657c1e59d0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f6568034040 tx=0x7f65680046c0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.577+0000 7f6582bb5640 1 -- 172.21.15.161:0/2339017778 shutdown_connections 2024-03-28T23:35:02.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.577+0000 7f6582bb5640 1 --2- 172.21.15.161:0/2339017778 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f656006b4d0 0x7f656006d990 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.577+0000 7f6582bb5640 1 --2- 172.21.15.161:0/2339017778 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f657c0aacb0 0x7f657c1e59d0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.577+0000 7f6582bb5640 1 --2- 172.21.15.161:0/2339017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f657c150600 0x7f657c1e5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.577+0000 7f6582bb5640 1 --2- 172.21.15.161:0/2339017778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f657c0a8fc0 0x7f657c1522d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.578+0000 7f6582bb5640 1 -- 172.21.15.161:0/2339017778 >> 172.21.15.161:0/2339017778 conn(0x7f657c0c6670 msgr2=0x7f657c149ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:02.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.578+0000 7f6582bb5640 1 -- 172.21.15.161:0/2339017778 shutdown_connections 2024-03-28T23:35:02.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.578+0000 7f6582bb5640 1 -- 172.21.15.161:0/2339017778 wait complete. 2024-03-28T23:35:02.580 INFO:tasks.ceph.mds.a.smithi161.stderr: -67> 2024-03-28T23:35:01.256+0000 7fcfc472a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:35:02.590 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-28T23:35:02.604 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-28T23:35:02.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.693+0000 7f8368a8a640 1 Processor -- start 2024-03-28T23:35:02.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.693+0000 7f8368a8a640 1 -- start start 2024-03-28T23:35:02.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.693+0000 7f8368a8a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f836406cdd0 0x7f83641524c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:02.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.693+0000 7f8368a8a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f836406ba50 0x7f8364152a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:02.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.693+0000 7f8368a8a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83640c2f10 con 0x7f836414f5c0 2024-03-28T23:35:02.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.693+0000 7f8368a8a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f83640c74d0 con 0x7f836406ba50 2024-03-28T23:35:02.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.693+0000 7f8368a8a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f83640c6fa0 con 0x7f836406cdd0 2024-03-28T23:35:02.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.694+0000 7f8362d9d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f836406ba50 0x7f8364152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:02.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.694+0000 7f836259c640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f836414f5c0 0x7f83640c2150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:40368/0 (socket says 172.21.15.161:40368) 2024-03-28T23:35:02.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.694+0000 7f836259c640 1 -- 172.21.15.161:0/1671453776 learned_addr learned my addr 172.21.15.161:0/1671453776 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:02.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.694+0000 7f8359d9b640 1 --2- 172.21.15.161:0/1671453776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f836406cdd0 0x7f83641524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:02.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.694+0000 7f8359d9b640 1 -- 172.21.15.161:0/1671453776 >> v2:172.21.15.170:3300/0 conn(0x7f836406ba50 msgr2=0x7f8364152a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:02.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.694+0000 7f8359d9b640 1 --2- 172.21.15.161:0/1671453776 >> v2:172.21.15.170:3300/0 conn(0x7f836406ba50 0x7f8364152a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.694+0000 7f8359d9b640 1 -- 172.21.15.161:0/1671453776 >> v1:172.21.15.170:6789/0 conn(0x7f836414f5c0 legacy=0x7f83640c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:02.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.694+0000 7f8359d9b640 1 -- 172.21.15.161:0/1671453776 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f836414f9a0 con 0x7f836406cdd0 2024-03-28T23:35:02.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.694+0000 7f8362d9d640 1 --2- 172.21.15.161:0/1671453776 >> v2:172.21.15.170:3300/0 conn(0x7f836406ba50 0x7f8364152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:02.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.695+0000 7f8359d9b640 1 --2- 172.21.15.161:0/1671453776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f836406cdd0 0x7f83641524c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f834c0097a0 tx=0x7f834c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cb9c86dff44eaf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:02.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.695+0000 7f8361d9b640 1 -- 172.21.15.161:0/1671453776 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834c035020 con 0x7f836406cdd0 2024-03-28T23:35:02.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.695+0000 7f8361d9b640 1 -- 172.21.15.161:0/1671453776 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f834c031810 con 0x7f836406cdd0 2024-03-28T23:35:02.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.695+0000 7f8361d9b640 1 -- 172.21.15.161:0/1671453776 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834c031be0 con 0x7f836406cdd0 2024-03-28T23:35:02.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.695+0000 7f8368a8a640 1 -- 172.21.15.161:0/1671453776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f836406cdd0 msgr2=0x7f83641524c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:02.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.695+0000 7f8368a8a640 1 --2- 172.21.15.161:0/1671453776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f836406cdd0 0x7f83641524c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f834c0097a0 tx=0x7f834c02b8b0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.696+0000 7f8368a8a640 1 -- 172.21.15.161:0/1671453776 shutdown_connections 2024-03-28T23:35:02.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.696+0000 7f8368a8a640 1 --2- 172.21.15.161:0/1671453776 >> v2:172.21.15.170:3300/0 conn(0x7f836406ba50 0x7f8364152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.696+0000 7f8368a8a640 1 --2- 172.21.15.161:0/1671453776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f836406cdd0 0x7f83641524c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.696+0000 7f8368a8a640 1 -- 172.21.15.161:0/1671453776 >> 172.21.15.161:0/1671453776 conn(0x7f83640a83f0 msgr2=0x7f83640a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:02.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.696+0000 7f8368a8a640 1 -- 172.21.15.161:0/1671453776 shutdown_connections 2024-03-28T23:35:02.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.696+0000 7f8368a8a640 1 -- 172.21.15.161:0/1671453776 wait complete. 2024-03-28T23:35:02.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.697+0000 7f8368a8a640 1 Processor -- start 2024-03-28T23:35:02.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.697+0000 7f8368a8a640 1 -- start start 2024-03-28T23:35:02.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.698+0000 7f8368a8a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f836406ba50 0x7f836414e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:02.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.698+0000 7f8368a8a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f836406cdd0 0x7f83641467c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:02.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.698+0000 7f8368a8a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f836414f5c0 0x7f8364146d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:02.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.698+0000 7f8368a8a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f836406c140 con 0x7f836406ba50 2024-03-28T23:35:02.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.698+0000 7f8368a8a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8364152030 con 0x7f836414f5c0 2024-03-28T23:35:02.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.698+0000 7f8368a8a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f836406c3e0 con 0x7f836406cdd0 2024-03-28T23:35:02.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.698+0000 7f836259c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f836406ba50 0x7f836414e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:02.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.698+0000 7f836259c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f836406ba50 0x7f836414e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57476/0 (socket says 172.21.15.161:57476) 2024-03-28T23:35:02.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.698+0000 7f836259c640 1 -- 172.21.15.161:0/2942289661 learned_addr learned my addr 172.21.15.161:0/2942289661 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:02.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.698+0000 7f8359d9b640 1 --2- 172.21.15.161:0/2942289661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f836406cdd0 0x7f83641467c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:02.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.698+0000 7f8362d9d640 1 --2- 172.21.15.161:0/2942289661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f836414f5c0 0x7f8364146d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:02.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.698+0000 7f836259c640 1 -- 172.21.15.161:0/2942289661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f836406cdd0 msgr2=0x7f83641467c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:02.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.698+0000 7f836259c640 1 --2- 172.21.15.161:0/2942289661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f836406cdd0 0x7f83641467c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.698+0000 7f836259c640 1 -- 172.21.15.161:0/2942289661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f836414f5c0 msgr2=0x7f8364146d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:02.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.698+0000 7f836259c640 1 --2- 172.21.15.161:0/2942289661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f836414f5c0 0x7f8364146d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:02.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.698+0000 7f836259c640 1 -- 172.21.15.161:0/2942289661 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83640730e0 con 0x7f836406ba50 2024-03-28T23:35:02.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.698+0000 7f8362d9d640 1 --2- 172.21.15.161:0/2942289661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f836414f5c0 0x7f8364146d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:02.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.698+0000 7f836259c640 1 --2- 172.21.15.161:0/2942289661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f836406ba50 0x7f836414e6a0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f835400c430 tx=0x7f835400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:02.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.698+0000 7f835b7fe640 1 -- 172.21.15.161:0/2942289661 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8354017070 con 0x7f836406ba50 2024-03-28T23:35:02.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.698+0000 7f835b7fe640 1 -- 172.21.15.161:0/2942289661 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8354005ce0 con 0x7f836406ba50 2024-03-28T23:35:02.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.698+0000 7f8368a8a640 1 -- 172.21.15.161:0/2942289661 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8364070840 con 0x7f836406ba50 2024-03-28T23:35:02.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.699+0000 7f835b7fe640 1 -- 172.21.15.161:0/2942289661 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83540072f0 con 0x7f836406ba50 2024-03-28T23:35:02.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.699+0000 7f8368a8a640 1 -- 172.21.15.161:0/2942289661 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f836414ebe0 con 0x7f836406ba50 2024-03-28T23:35:02.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.699+0000 7f8368a8a640 1 -- 172.21.15.161:0/2942289661 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f836404f4b0 con 0x7f836406ba50 2024-03-28T23:35:02.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.700+0000 7f835b7fe640 1 -- 172.21.15.161:0/2942289661 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f835401a070 con 0x7f836406ba50 2024-03-28T23:35:02.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.700+0000 7f835b7fe640 1 --2- 172.21.15.161:0/2942289661 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f834406b4d0 0x7f834406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:02.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.700+0000 7f835b7fe640 1 -- 172.21.15.161:0/2942289661 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 6359+0+0 (secure 0 0 0) 0x7f83540897a0 con 0x7f836406ba50 2024-03-28T23:35:02.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.700+0000 7f8359d9b640 1 --2- 172.21.15.161:0/2942289661 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f834406b4d0 0x7f834406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:02.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.701+0000 7f8359d9b640 1 --2- 172.21.15.161:0/2942289661 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f834406b4d0 0x7f834406d990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f834c036000 tx=0x7f834c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:02.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.705+0000 7f835b7fe640 1 -- 172.21.15.161:0/2942289661 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f83640c2f10 con 0x7f836406ba50 2024-03-28T23:35:02.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:02.910+0000 7f8368a8a640 1 -- 172.21.15.161:0/2942289661 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f83640c2cf0 con 0x7f836406ba50 2024-03-28T23:35:03.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.576+0000 7f835b7fe640 1 -- 172.21.15.161:0/2942289661 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v119) ==== 104+0+0 (secure 0 0 0) 0x7f8354058420 con 0x7f836406ba50 2024-03-28T23:35:03.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.579+0000 7f8368a8a640 1 -- 172.21.15.161:0/2942289661 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f834406b4d0 msgr2=0x7f834406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:03.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.579+0000 7f8368a8a640 1 --2- 172.21.15.161:0/2942289661 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f834406b4d0 0x7f834406d990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f834c036000 tx=0x7f834c008030 comp rx=0 tx=0).stop 2024-03-28T23:35:03.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.579+0000 7f8368a8a640 1 -- 172.21.15.161:0/2942289661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f836406ba50 msgr2=0x7f836414e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:03.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.579+0000 7f8368a8a640 1 --2- 172.21.15.161:0/2942289661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f836406ba50 0x7f836414e6a0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f835400c430 tx=0x7f835400c8f0 comp rx=0 tx=0).stop 2024-03-28T23:35:03.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.579+0000 7f8368a8a640 1 -- 172.21.15.161:0/2942289661 shutdown_connections 2024-03-28T23:35:03.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.579+0000 7f8368a8a640 1 --2- 172.21.15.161:0/2942289661 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f834406b4d0 0x7f834406d990 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:03.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.579+0000 7f8368a8a640 1 --2- 172.21.15.161:0/2942289661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f836414f5c0 0x7f8364146d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:03.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.579+0000 7f8368a8a640 1 --2- 172.21.15.161:0/2942289661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f836406cdd0 0x7f83641467c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:03.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.579+0000 7f8368a8a640 1 --2- 172.21.15.161:0/2942289661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f836406ba50 0x7f836414e6a0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:03.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.579+0000 7f8368a8a640 1 -- 172.21.15.161:0/2942289661 >> 172.21.15.161:0/2942289661 conn(0x7f83640a83f0 msgr2=0x7f836414c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:03.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.579+0000 7f8368a8a640 1 -- 172.21.15.161:0/2942289661 shutdown_connections 2024-03-28T23:35:03.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.579+0000 7f8368a8a640 1 -- 172.21.15.161:0/2942289661 wait complete. 2024-03-28T23:35:03.594 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-28T23:35:03.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.742+0000 7fc0822c0640 1 Processor -- start 2024-03-28T23:35:03.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.742+0000 7fc0822c0640 1 -- start start 2024-03-28T23:35:03.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.743+0000 7fc0822c0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc07c152170 0x7fc07c06cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:03.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.743+0000 7fc0822c0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc07c0a5cf0 0x7fc07c0c1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:03.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.743+0000 7fc0822c0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc07c06d8e0 con 0x7fc07c0c1f70 2024-03-28T23:35:03.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.743+0000 7fc0822c0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc07c0c4900 con 0x7fc07c152170 2024-03-28T23:35:03.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.743+0000 7fc0822c0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fc07c06dbc0 con 0x7fc07c0a5cf0 2024-03-28T23:35:03.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.743+0000 7fc07bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc07c152170 0x7fc07c06cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:03.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.743+0000 7fc07bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc07c152170 0x7fc07c06cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57486/0 (socket says 172.21.15.161:57486) 2024-03-28T23:35:03.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.743+0000 7fc07bfff640 1 -- 172.21.15.161:0/1628086380 learned_addr learned my addr 172.21.15.161:0/1628086380 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:03.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.743+0000 7fc07b7fe640 1 --2- 172.21.15.161:0/1628086380 >> v2:172.21.15.170:3300/0 conn(0x7fc07c0a5cf0 0x7fc07c0c1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:03.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.743+0000 7fc07bfff640 1 -- 172.21.15.161:0/1628086380 >> v2:172.21.15.170:3300/0 conn(0x7fc07c0a5cf0 msgr2=0x7fc07c0c1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:03.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.743+0000 7fc07bfff640 1 --2- 172.21.15.161:0/1628086380 >> v2:172.21.15.170:3300/0 conn(0x7fc07c0a5cf0 0x7fc07c0c1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:03.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.743+0000 7fc07bfff640 1 -- 172.21.15.161:0/1628086380 >> v1:172.21.15.170:6789/0 conn(0x7fc07c0c1f70 legacy=0x7fc07c0c5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:03.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.743+0000 7fc07bfff640 1 -- 172.21.15.161:0/1628086380 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc07c0c4b50 con 0x7fc07c152170 2024-03-28T23:35:03.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.744+0000 7fc07b7fe640 1 --2- 172.21.15.161:0/1628086380 >> v2:172.21.15.170:3300/0 conn(0x7fc07c0a5cf0 0x7fc07c0c1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:03.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.744+0000 7fc07bfff640 1 --2- 172.21.15.161:0/1628086380 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc07c152170 0x7fc07c06cf70 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fc07c06c4d0 tx=0x7fc06c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ba5050b4f4a70f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:03.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.744+0000 7fc07affd640 1 -- 172.21.15.161:0/1628086380 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc06c038070 con 0x7fc07c152170 2024-03-28T23:35:03.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.744+0000 7fc07affd640 1 -- 172.21.15.161:0/1628086380 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc06c02f050 con 0x7fc07c152170 2024-03-28T23:35:03.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.744+0000 7fc07affd640 1 -- 172.21.15.161:0/1628086380 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc06c031970 con 0x7fc07c152170 2024-03-28T23:35:03.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.745+0000 7fc0822c0640 1 -- 172.21.15.161:0/1628086380 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc07c152170 msgr2=0x7fc07c06cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:03.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.745+0000 7fc0822c0640 1 --2- 172.21.15.161:0/1628086380 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc07c152170 0x7fc07c06cf70 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fc07c06c4d0 tx=0x7fc06c02b7f0 comp rx=0 tx=0).stop 2024-03-28T23:35:03.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.745+0000 7fc0822c0640 1 -- 172.21.15.161:0/1628086380 shutdown_connections 2024-03-28T23:35:03.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.745+0000 7fc0822c0640 1 --2- 172.21.15.161:0/1628086380 >> v2:172.21.15.170:3300/0 conn(0x7fc07c0a5cf0 0x7fc07c0c1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:03.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.745+0000 7fc0822c0640 1 --2- 172.21.15.161:0/1628086380 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc07c152170 0x7fc07c06cf70 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:03.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.745+0000 7fc0822c0640 1 -- 172.21.15.161:0/1628086380 >> 172.21.15.161:0/1628086380 conn(0x7fc07c0c6e20 msgr2=0x7fc07c0c7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:03.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.745+0000 7fc0822c0640 1 -- 172.21.15.161:0/1628086380 shutdown_connections 2024-03-28T23:35:03.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.745+0000 7fc0822c0640 1 -- 172.21.15.161:0/1628086380 wait complete. 2024-03-28T23:35:03.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.746+0000 7fc0822c0640 1 Processor -- start 2024-03-28T23:35:03.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.746+0000 7fc0822c0640 1 -- start start 2024-03-28T23:35:03.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.747+0000 7fc0822c0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc07c0a5cf0 0x7fc07c0ab5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:03.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.747+0000 7fc0822c0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc07c0c1f70 0x7fc07c14ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:03.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.747+0000 7fc0822c0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc07c152170 0x7fc07c1540b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:03.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.747+0000 7fc0822c0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc07c06d630 con 0x7fc07c0c1f70 2024-03-28T23:35:03.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.747+0000 7fc0822c0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc07c06d4b0 con 0x7fc07c152170 2024-03-28T23:35:03.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.747+0000 7fc0822c0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc07c152550 con 0x7fc07c0a5cf0 2024-03-28T23:35:03.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.747+0000 7fc07b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc07c0c1f70 0x7fc07c14ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:03.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.747+0000 7fc07b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc07c0c1f70 0x7fc07c14ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57494/0 (socket says 172.21.15.161:57494) 2024-03-28T23:35:03.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.747+0000 7fc07b7fe640 1 -- 172.21.15.161:0/2461175225 learned_addr learned my addr 172.21.15.161:0/2461175225 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:03.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.747+0000 7fc07bfff640 1 --2- 172.21.15.161:0/2461175225 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc07c0a5cf0 0x7fc07c0ab5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:03.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.747+0000 7fc08085d640 1 --2- 172.21.15.161:0/2461175225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc07c152170 0x7fc07c1540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:03.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.747+0000 7fc07b7fe640 1 -- 172.21.15.161:0/2461175225 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc07c0a5cf0 msgr2=0x7fc07c0ab5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:03.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.747+0000 7fc07b7fe640 1 --2- 172.21.15.161:0/2461175225 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc07c0a5cf0 0x7fc07c0ab5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:03.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.747+0000 7fc07b7fe640 1 -- 172.21.15.161:0/2461175225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc07c152170 msgr2=0x7fc07c1540b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:03.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.747+0000 7fc07b7fe640 1 --2- 172.21.15.161:0/2461175225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc07c152170 0x7fc07c1540b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:03.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.747+0000 7fc07b7fe640 1 -- 172.21.15.161:0/2461175225 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc07c153aa0 con 0x7fc07c0c1f70 2024-03-28T23:35:03.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.748+0000 7fc07b7fe640 1 --2- 172.21.15.161:0/2461175225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc07c0c1f70 0x7fc07c14ae20 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fc07000c7f0 tx=0x7fc07000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:03.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.748+0000 7fc078ff9640 1 -- 172.21.15.161:0/2461175225 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc070015070 con 0x7fc07c0c1f70 2024-03-28T23:35:03.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.748+0000 7fc078ff9640 1 -- 172.21.15.161:0/2461175225 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc070007a30 con 0x7fc07c0c1f70 2024-03-28T23:35:03.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.748+0000 7fc078ff9640 1 -- 172.21.15.161:0/2461175225 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc070007e00 con 0x7fc07c0c1f70 2024-03-28T23:35:03.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.748+0000 7fc0822c0640 1 -- 172.21.15.161:0/2461175225 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc07c14a230 con 0x7fc07c0c1f70 2024-03-28T23:35:03.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.748+0000 7fc0822c0640 1 -- 172.21.15.161:0/2461175225 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc07c153730 con 0x7fc07c0c1f70 2024-03-28T23:35:03.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.749+0000 7fc0822c0640 1 -- 172.21.15.161:0/2461175225 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc07c068f90 con 0x7fc07c0c1f70 2024-03-28T23:35:03.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.749+0000 7fc078ff9640 1 -- 172.21.15.161:0/2461175225 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc070018070 con 0x7fc07c0c1f70 2024-03-28T23:35:03.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.749+0000 7fc078ff9640 1 --2- 172.21.15.161:0/2461175225 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc06806b4d0 0x7fc06806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:03.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.750+0000 7fc078ff9640 1 -- 172.21.15.161:0/2461175225 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 6359+0+0 (secure 0 0 0) 0x7fc07008a350 con 0x7fc07c0c1f70 2024-03-28T23:35:03.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.750+0000 7fc07bfff640 1 --2- 172.21.15.161:0/2461175225 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc06806b4d0 0x7fc06806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:03.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.750+0000 7fc07bfff640 1 --2- 172.21.15.161:0/2461175225 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc06806b4d0 0x7fc06806d990 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fc06c035000 tx=0x7fc06c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:03.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.754+0000 7fc078ff9640 1 -- 172.21.15.161:0/2461175225 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc070058fd0 con 0x7fc07c0c1f70 2024-03-28T23:35:03.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:03.924+0000 7fc0822c0640 1 -- 172.21.15.161:0/2461175225 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc07c153e10 con 0x7fc07c0c1f70 2024-03-28T23:35:04.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.577+0000 7fc078ff9640 1 -- 172.21.15.161:0/2461175225 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v126) ==== 181+0+0 (secure 0 0 0) 0x7fc0700588d0 con 0x7fc07c0c1f70 2024-03-28T23:35:04.578 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_metadata' removed 2024-03-28T23:35:04.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.580+0000 7fc0822c0640 1 -- 172.21.15.161:0/2461175225 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc06806b4d0 msgr2=0x7fc06806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:04.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.580+0000 7fc0822c0640 1 --2- 172.21.15.161:0/2461175225 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc06806b4d0 0x7fc06806d990 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fc06c035000 tx=0x7fc06c008030 comp rx=0 tx=0).stop 2024-03-28T23:35:04.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.580+0000 7fc0822c0640 1 -- 172.21.15.161:0/2461175225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc07c0c1f70 msgr2=0x7fc07c14ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:04.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.580+0000 7fc0822c0640 1 --2- 172.21.15.161:0/2461175225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc07c0c1f70 0x7fc07c14ae20 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fc07000c7f0 tx=0x7fc07000cbf0 comp rx=0 tx=0).stop 2024-03-28T23:35:04.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.580+0000 7fc0822c0640 1 -- 172.21.15.161:0/2461175225 shutdown_connections 2024-03-28T23:35:04.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.580+0000 7fc0822c0640 1 --2- 172.21.15.161:0/2461175225 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc06806b4d0 0x7fc06806d990 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:04.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.580+0000 7fc0822c0640 1 --2- 172.21.15.161:0/2461175225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc07c0c1f70 0x7fc07c14ae20 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:04.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.580+0000 7fc0822c0640 1 --2- 172.21.15.161:0/2461175225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc07c152170 0x7fc07c1540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:04.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.580+0000 7fc0822c0640 1 --2- 172.21.15.161:0/2461175225 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc07c0a5cf0 0x7fc07c0ab5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:04.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.580+0000 7fc0822c0640 1 -- 172.21.15.161:0/2461175225 >> 172.21.15.161:0/2461175225 conn(0x7fc07c0c6e20 msgr2=0x7fc07c06c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:04.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.580+0000 7fc0822c0640 1 -- 172.21.15.161:0/2461175225 shutdown_connections 2024-03-28T23:35:04.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.581+0000 7fc0822c0640 1 -- 172.21.15.161:0/2461175225 wait complete. 2024-03-28T23:35:04.595 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-28T23:35:04.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.705+0000 7f641e19a640 1 Processor -- start 2024-03-28T23:35:04.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.705+0000 7f641e19a640 1 -- start start 2024-03-28T23:35:04.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.706+0000 7f641e19a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6418152740 0x7f641806b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:04.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.706+0000 7f641e19a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64181561d0 con 0x7f64180c53e0 2024-03-28T23:35:04.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.706+0000 7f641e19a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f641806bb70 con 0x7f6418155d70 2024-03-28T23:35:04.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.706+0000 7f641e19a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f641806be50 con 0x7f6418152740 2024-03-28T23:35:04.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.706+0000 7f6417fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f64180c53e0 0x7f64180c7470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45300/0 (socket says 172.21.15.161:45300) 2024-03-28T23:35:04.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.706+0000 7f6417fff640 1 -- 172.21.15.161:0/306274265 learned_addr learned my addr 172.21.15.161:0/306274265 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:04.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.706+0000 7f64167fc640 1 -- 172.21.15.161:0/306274265 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1466772358 0 0) 0x7f64181561d0 con 0x7f64180c53e0 2024-03-28T23:35:04.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.706+0000 7f64167fc640 1 -- 172.21.15.161:0/306274265 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6404003520 con 0x7f64180c53e0 2024-03-28T23:35:04.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.706+0000 7f64177fe640 1 --2- 172.21.15.161:0/306274265 >> v2:172.21.15.170:3300/0 conn(0x7f6418152740 0x7f641806b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:04.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.706+0000 7f64167fc640 1 -- 172.21.15.161:0/306274265 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3667331531 0 0) 0x7f6404003520 con 0x7f64180c53e0 2024-03-28T23:35:04.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.706+0000 7f64167fc640 1 -- 172.21.15.161:0/306274265 >> v2:172.21.15.170:3300/0 conn(0x7f6418152740 msgr2=0x7f641806b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:04.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.706+0000 7f64167fc640 1 --2- 172.21.15.161:0/306274265 >> v2:172.21.15.170:3300/0 conn(0x7f6418152740 0x7f641806b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:04.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.706+0000 7f64167fc640 1 -- 172.21.15.161:0/306274265 >> v1:172.21.15.170:6789/0 conn(0x7f6418155d70 legacy=0x7f64180c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:04.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.706+0000 7f64167fc640 1 -- 172.21.15.161:0/306274265 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64180c8960 con 0x7f64180c53e0 2024-03-28T23:35:04.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.706+0000 7f64167fc640 1 -- 172.21.15.161:0/306274265 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f6410001670 con 0x7f64180c53e0 2024-03-28T23:35:04.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.706+0000 7f64177fe640 1 --2- 172.21.15.161:0/306274265 >> v2:172.21.15.170:3300/0 conn(0x7f6418152740 0x7f641806b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:04.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.706+0000 7f64167fc640 1 -- 172.21.15.161:0/306274265 >> v1:172.21.15.161:6789/0 conn(0x7f64180c53e0 legacy=0x7f64180c7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:04.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.706+0000 7f64167fc640 1 --2- 172.21.15.161:0/306274265 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64040039b0 0x7f6404003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:04.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.707+0000 7f64167fc640 1 --2- 172.21.15.161:0/306274265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6404004800 0x7f6404004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:04.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.707+0000 7f64167fc640 1 --2- 172.21.15.161:0/306274265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64040052e0 0x7f640400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:04.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.707+0000 7f64167fc640 1 -- 172.21.15.161:0/306274265 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f641806be50 con 0x7f6404004800 2024-03-28T23:35:04.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.707+0000 7f64167fc640 1 -- 172.21.15.161:0/306274265 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f640400c1b0 con 0x7f64040052e0 2024-03-28T23:35:04.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.707+0000 7f64167fc640 1 -- 172.21.15.161:0/306274265 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f640400c3b0 con 0x7f64040039b0 2024-03-28T23:35:04.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.707+0000 7f6416ffd640 1 --2- 172.21.15.161:0/306274265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6404004800 0x7f6404004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:04.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.707+0000 7f6417fff640 1 --2- 172.21.15.161:0/306274265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64040052e0 0x7f640400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:04.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.707+0000 7f64177fe640 1 --2- 172.21.15.161:0/306274265 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64040039b0 0x7f6404003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:04.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.707+0000 7f6416ffd640 1 -- 172.21.15.161:0/306274265 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64040039b0 msgr2=0x7f6404003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:04.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.707+0000 7f6416ffd640 1 --2- 172.21.15.161:0/306274265 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64040039b0 0x7f6404003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:04.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.707+0000 7f6416ffd640 1 -- 172.21.15.161:0/306274265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64040052e0 msgr2=0x7f640400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:04.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.707+0000 7f6416ffd640 1 --2- 172.21.15.161:0/306274265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64040052e0 0x7f640400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:04.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.707+0000 7f6416ffd640 1 -- 172.21.15.161:0/306274265 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6404003520 con 0x7f6404004800 2024-03-28T23:35:04.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.707+0000 7f6417fff640 1 -- 172.21.15.161:0/306274265 reap_dead start 2024-03-28T23:35:04.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.708+0000 7f6416ffd640 1 --2- 172.21.15.161:0/306274265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6404004800 0x7f6404004c80 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f6400007fd0 tx=0x7f6404005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdca32dad47b942c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:04.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.708+0000 7f64167fc640 1 -- 172.21.15.161:0/306274265 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64000013c0 con 0x7f6404004800 2024-03-28T23:35:04.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.708+0000 7f64167fc640 1 -- 172.21.15.161:0/306274265 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64000052d0 con 0x7f6404004800 2024-03-28T23:35:04.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.708+0000 7f64167fc640 1 -- 172.21.15.161:0/306274265 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6400003640 con 0x7f6404004800 2024-03-28T23:35:04.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.708+0000 7f641e19a640 1 -- 172.21.15.161:0/306274265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6404004800 msgr2=0x7f6404004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:04.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.708+0000 7f641e19a640 1 --2- 172.21.15.161:0/306274265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6404004800 0x7f6404004c80 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f6400007fd0 tx=0x7f6404005830 comp rx=0 tx=0).stop 2024-03-28T23:35:04.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.708+0000 7f641e19a640 1 -- 172.21.15.161:0/306274265 shutdown_connections 2024-03-28T23:35:04.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.708+0000 7f641e19a640 1 --2- 172.21.15.161:0/306274265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6404004800 0x7f6404004c80 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:04.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.708+0000 7f641e19a640 1 -- 172.21.15.161:0/306274265 >> 172.21.15.161:0/306274265 conn(0x7f64180cac40 msgr2=0x7f64180cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:04.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.708+0000 7f641e19a640 1 -- 172.21.15.161:0/306274265 shutdown_connections 2024-03-28T23:35:04.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.709+0000 7f641e19a640 1 -- 172.21.15.161:0/306274265 wait complete. 2024-03-28T23:35:04.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.710+0000 7f641e19a640 1 Processor -- start 2024-03-28T23:35:04.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.710+0000 7f641e19a640 1 -- start start 2024-03-28T23:35:04.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.710+0000 7f641e19a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6404004800 0x7f64180c4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:04.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.710+0000 7f641e19a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64181498a0 0x7f6418150970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:04.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.710+0000 7f641e19a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6418150eb0 0x7f64181ee300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:04.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.710+0000 7f641e19a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f64180c5890 con 0x7f6418150eb0 2024-03-28T23:35:04.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.710+0000 7f641e19a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f64180c8f20 con 0x7f64181498a0 2024-03-28T23:35:04.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.710+0000 7f641e19a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6418152b20 con 0x7f6404004800 2024-03-28T23:35:04.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.711+0000 7f6417fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6418150eb0 0x7f64181ee300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:04.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.711+0000 7f6417fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6418150eb0 0x7f64181ee300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57516/0 (socket says 172.21.15.161:57516) 2024-03-28T23:35:04.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.711+0000 7f6417fff640 1 -- 172.21.15.161:0/2425193593 learned_addr learned my addr 172.21.15.161:0/2425193593 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:04.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.711+0000 7f64177fe640 1 --2- 172.21.15.161:0/2425193593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6404004800 0x7f64180c4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:04.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.711+0000 7f6416ffd640 1 --2- 172.21.15.161:0/2425193593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64181498a0 0x7f6418150970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:04.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.711+0000 7f6417fff640 1 -- 172.21.15.161:0/2425193593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6404004800 msgr2=0x7f64180c4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:04.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.711+0000 7f6417fff640 1 --2- 172.21.15.161:0/2425193593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6404004800 0x7f64180c4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:04.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.711+0000 7f6417fff640 1 -- 172.21.15.161:0/2425193593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64181498a0 msgr2=0x7f6418150970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:04.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.711+0000 7f6417fff640 1 --2- 172.21.15.161:0/2425193593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64181498a0 0x7f6418150970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:04.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.711+0000 7f6417fff640 1 -- 172.21.15.161:0/2425193593 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64180c1d80 con 0x7f6418150eb0 2024-03-28T23:35:04.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.711+0000 7f6417fff640 1 --2- 172.21.15.161:0/2425193593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6418150eb0 0x7f64181ee300 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f6418155260 tx=0x7f6410007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:04.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.711+0000 7f640ffff640 1 -- 172.21.15.161:0/2425193593 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f641001a070 con 0x7f6418150eb0 2024-03-28T23:35:04.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.711+0000 7f640ffff640 1 -- 172.21.15.161:0/2425193593 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f641000f620 con 0x7f6418150eb0 2024-03-28T23:35:04.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.711+0000 7f641e19a640 1 -- 172.21.15.161:0/2425193593 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64181eed80 con 0x7f6418150eb0 2024-03-28T23:35:04.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.711+0000 7f640ffff640 1 -- 172.21.15.161:0/2425193593 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f641000fa10 con 0x7f6418150eb0 2024-03-28T23:35:04.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.711+0000 7f641e19a640 1 -- 172.21.15.161:0/2425193593 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64181ee840 con 0x7f6418150eb0 2024-03-28T23:35:04.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.712+0000 7f641e19a640 1 -- 172.21.15.161:0/2425193593 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6418068f90 con 0x7f6418150eb0 2024-03-28T23:35:04.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.713+0000 7f640ffff640 1 -- 172.21.15.161:0/2425193593 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f641001d070 con 0x7f6418150eb0 2024-03-28T23:35:04.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.713+0000 7f640ffff640 1 --2- 172.21.15.161:0/2425193593 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f63f806b4d0 0x7f63f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:04.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.713+0000 7f640ffff640 1 -- 172.21.15.161:0/2425193593 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5921+0+0 (secure 0 0 0) 0x7f6410089980 con 0x7f6418150eb0 2024-03-28T23:35:04.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.713+0000 7f64177fe640 1 --2- 172.21.15.161:0/2425193593 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f63f806b4d0 0x7f63f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:04.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.714+0000 7f64177fe640 1 --2- 172.21.15.161:0/2425193593 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f63f806b4d0 0x7f63f806d990 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f64180c9de0 tx=0x7f6408003bf0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:04.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.717+0000 7f640ffff640 1 -- 172.21.15.161:0/2425193593 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f64100580d0 con 0x7f6418150eb0 2024-03-28T23:35:04.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:04.890+0000 7f641e19a640 1 -- 172.21.15.161:0/2425193593 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f64180ca4f0 con 0x7f6418150eb0 2024-03-28T23:35:05.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.578+0000 7f640ffff640 1 -- 172.21.15.161:0/2425193593 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v127) ==== 169+0+0 (secure 0 0 0) 0x7f641005c170 con 0x7f6418150eb0 2024-03-28T23:35:05.580 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data' removed 2024-03-28T23:35:05.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.580+0000 7f641e19a640 1 -- 172.21.15.161:0/2425193593 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f63f806b4d0 msgr2=0x7f63f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:05.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.580+0000 7f641e19a640 1 --2- 172.21.15.161:0/2425193593 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f63f806b4d0 0x7f63f806d990 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f64180c9de0 tx=0x7f6408003bf0 comp rx=0 tx=0).stop 2024-03-28T23:35:05.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.580+0000 7f641e19a640 1 -- 172.21.15.161:0/2425193593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6418150eb0 msgr2=0x7f64181ee300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:05.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.580+0000 7f641e19a640 1 --2- 172.21.15.161:0/2425193593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6418150eb0 0x7f64181ee300 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f6418155260 tx=0x7f6410007430 comp rx=0 tx=0).stop 2024-03-28T23:35:05.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.580+0000 7f641e19a640 1 -- 172.21.15.161:0/2425193593 shutdown_connections 2024-03-28T23:35:05.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.580+0000 7f641e19a640 1 --2- 172.21.15.161:0/2425193593 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f63f806b4d0 0x7f63f806d990 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:05.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.580+0000 7f641e19a640 1 --2- 172.21.15.161:0/2425193593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6418150eb0 0x7f64181ee300 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:05.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.580+0000 7f641e19a640 1 --2- 172.21.15.161:0/2425193593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64181498a0 0x7f6418150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:05.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.580+0000 7f641e19a640 1 --2- 172.21.15.161:0/2425193593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6404004800 0x7f64180c4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:05.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.580+0000 7f641e19a640 1 -- 172.21.15.161:0/2425193593 >> 172.21.15.161:0/2425193593 conn(0x7f64180cac40 msgr2=0x7f64180cb020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:05.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.580+0000 7f641e19a640 1 -- 172.21.15.161:0/2425193593 shutdown_connections 2024-03-28T23:35:05.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.580+0000 7f641e19a640 1 -- 172.21.15.161:0/2425193593 wait complete. 2024-03-28T23:35:05.597 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-28T23:35:05.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.701+0000 7f3ca6eeb640 1 Processor -- start 2024-03-28T23:35:05.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.701+0000 7f3ca6eeb640 1 -- start start 2024-03-28T23:35:05.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.702+0000 7f3ca6eeb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ca00c1040 0x7f3ca01535b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:05.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.702+0000 7f3ca6eeb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ca00c2f50 con 0x7f3ca00a96f0 2024-03-28T23:35:05.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.702+0000 7f3ca6eeb640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ca01530d0 con 0x7f3ca0150520 2024-03-28T23:35:05.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.702+0000 7f3ca6eeb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3ca00a9c20 con 0x7f3ca00c1040 2024-03-28T23:35:05.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.702+0000 7f3ca5488640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f3ca00a96f0 0x7f3ca015d8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45304/0 (socket says 172.21.15.161:45304) 2024-03-28T23:35:05.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.702+0000 7f3ca5488640 1 -- 172.21.15.161:0/3596266198 learned_addr learned my addr 172.21.15.161:0/3596266198 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:05.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.702+0000 7f3c9ffff640 1 --2- 172.21.15.161:0/3596266198 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ca00c1040 0x7f3ca01535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:05.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.702+0000 7f3c9f7fe640 1 -- 172.21.15.161:0/3596266198 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1740717308 0 0) 0x7f3ca00c2f50 con 0x7f3ca00a96f0 2024-03-28T23:35:05.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.702+0000 7f3c9f7fe640 1 -- 172.21.15.161:0/3596266198 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c88003520 con 0x7f3ca00a96f0 2024-03-28T23:35:05.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.702+0000 7f3c9f7fe640 1 -- 172.21.15.161:0/3596266198 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2001299852 0 0) 0x7f3c88003520 con 0x7f3ca00a96f0 2024-03-28T23:35:05.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.702+0000 7f3c9f7fe640 1 -- 172.21.15.161:0/3596266198 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ca00c1040 msgr2=0x7f3ca01535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:05.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.702+0000 7f3c9f7fe640 1 --2- 172.21.15.161:0/3596266198 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ca00c1040 0x7f3ca01535b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:05.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.702+0000 7f3c9f7fe640 1 -- 172.21.15.161:0/3596266198 >> v1:172.21.15.170:6789/0 conn(0x7f3ca0150520 legacy=0x7f3ca00ab280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:05.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.702+0000 7f3c9f7fe640 1 -- 172.21.15.161:0/3596266198 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ca00c2990 con 0x7f3ca00a96f0 2024-03-28T23:35:05.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.702+0000 7f3c9f7fe640 1 -- 172.21.15.161:0/3596266198 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f3c980019f0 con 0x7f3ca00a96f0 2024-03-28T23:35:05.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.703+0000 7f3c9f7fe640 1 -- 172.21.15.161:0/3596266198 >> v1:172.21.15.161:6789/0 conn(0x7f3ca00a96f0 legacy=0x7f3ca015d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:05.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.703+0000 7f3c9f7fe640 1 --2- 172.21.15.161:0/3596266198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c88003ac0 0x7f3c88003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:05.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.703+0000 7f3c9f7fe640 1 --2- 172.21.15.161:0/3596266198 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c88004800 0x7f3c88004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:05.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.703+0000 7f3c9f7fe640 1 --2- 172.21.15.161:0/3596266198 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ca00c1040 0x7f3c8800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:05.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.703+0000 7f3c9f7fe640 1 -- 172.21.15.161:0/3596266198 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca00a9c20 con 0x7f3c88004800 2024-03-28T23:35:05.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.703+0000 7f3c9f7fe640 1 -- 172.21.15.161:0/3596266198 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3c880054f0 con 0x7f3c88003ac0 2024-03-28T23:35:05.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.703+0000 7f3c9f7fe640 1 -- 172.21.15.161:0/3596266198 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3c8800bf00 con 0x7f3ca00c1040 2024-03-28T23:35:05.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.703+0000 7f3c9ffff640 1 --2- 172.21.15.161:0/3596266198 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c88004800 0x7f3c88004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:05.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.703+0000 7f3ca4c87640 1 --2- 172.21.15.161:0/3596266198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c88003ac0 0x7f3c88003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:05.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.703+0000 7f3ca5488640 1 --2- 172.21.15.161:0/3596266198 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ca00c1040 0x7f3c8800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:05.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.703+0000 7f3c9ffff640 1 -- 172.21.15.161:0/3596266198 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ca00c1040 msgr2=0x7f3c8800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:05.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.703+0000 7f3c9ffff640 1 --2- 172.21.15.161:0/3596266198 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ca00c1040 0x7f3c8800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:05.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.703+0000 7f3c9ffff640 1 -- 172.21.15.161:0/3596266198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c88003ac0 msgr2=0x7f3c88003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:05.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.703+0000 7f3c9ffff640 1 --2- 172.21.15.161:0/3596266198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c88003ac0 0x7f3c88003f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:05.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.703+0000 7f3c9ffff640 1 -- 172.21.15.161:0/3596266198 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c88003520 con 0x7f3c88004800 2024-03-28T23:35:05.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.704+0000 7f3c9ffff640 1 --2- 172.21.15.161:0/3596266198 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c88004800 0x7f3c88004c80 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f3c94009a90 tx=0x7f3c88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1230897e9863ce60 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:05.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.704+0000 7f3c9f7fe640 1 -- 172.21.15.161:0/3596266198 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c94039070 con 0x7f3c88004800 2024-03-28T23:35:05.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.704+0000 7f3c9f7fe640 1 -- 172.21.15.161:0/3596266198 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c9402c680 con 0x7f3c88004800 2024-03-28T23:35:05.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.704+0000 7f3c9f7fe640 1 -- 172.21.15.161:0/3596266198 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c94009690 con 0x7f3c88004800 2024-03-28T23:35:05.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.704+0000 7f3ca6eeb640 1 -- 172.21.15.161:0/3596266198 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c88004800 msgr2=0x7f3c88004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:05.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.704+0000 7f3ca6eeb640 1 --2- 172.21.15.161:0/3596266198 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c88004800 0x7f3c88004c80 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f3c94009a90 tx=0x7f3c88005830 comp rx=0 tx=0).stop 2024-03-28T23:35:05.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.704+0000 7f3ca5488640 1 -- 172.21.15.161:0/3596266198 reap_dead start 2024-03-28T23:35:05.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.704+0000 7f3ca6eeb640 1 -- 172.21.15.161:0/3596266198 shutdown_connections 2024-03-28T23:35:05.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.704+0000 7f3ca6eeb640 1 -- 172.21.15.161:0/3596266198 >> 172.21.15.161:0/3596266198 conn(0x7f3ca00a83f0 msgr2=0x7f3ca00a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:05.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.705+0000 7f3ca6eeb640 1 -- 172.21.15.161:0/3596266198 shutdown_connections 2024-03-28T23:35:05.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.705+0000 7f3ca6eeb640 1 -- 172.21.15.161:0/3596266198 wait complete. 2024-03-28T23:35:05.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.706+0000 7f3ca6eeb640 1 Processor -- start 2024-03-28T23:35:05.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.706+0000 7f3ca6eeb640 1 -- start start 2024-03-28T23:35:05.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.706+0000 7f3ca6eeb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c88004800 0x7f3ca0145790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:05.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.706+0000 7f3ca6eeb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ca014c5f0 0x7f3ca0156ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:05.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.706+0000 7f3ca6eeb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ca00729b0 0x7f3ca0072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:05.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.706+0000 7f3ca6eeb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca00ab990 con 0x7f3ca014c5f0 2024-03-28T23:35:05.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.706+0000 7f3ca6eeb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca0150900 con 0x7f3c88004800 2024-03-28T23:35:05.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.706+0000 7f3ca6eeb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3ca00abb10 con 0x7f3ca00729b0 2024-03-28T23:35:05.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.707+0000 7f3c9ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ca014c5f0 0x7f3ca0156ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:05.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.707+0000 7f3c9ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ca014c5f0 0x7f3ca0156ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57538/0 (socket says 172.21.15.161:57538) 2024-03-28T23:35:05.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.707+0000 7f3c9ffff640 1 -- 172.21.15.161:0/378292594 learned_addr learned my addr 172.21.15.161:0/378292594 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:05.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.707+0000 7f3ca5488640 1 --2- 172.21.15.161:0/378292594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ca00729b0 0x7f3ca0072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:05.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.707+0000 7f3ca4c87640 1 --2- 172.21.15.161:0/378292594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c88004800 0x7f3ca0145790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:05.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.707+0000 7f3c9ffff640 1 -- 172.21.15.161:0/378292594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ca00729b0 msgr2=0x7f3ca0072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:05.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.707+0000 7f3c9ffff640 1 --2- 172.21.15.161:0/378292594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ca00729b0 0x7f3ca0072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:05.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.707+0000 7f3c9ffff640 1 -- 172.21.15.161:0/378292594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c88004800 msgr2=0x7f3ca0145790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:05.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.707+0000 7f3c9ffff640 1 --2- 172.21.15.161:0/378292594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c88004800 0x7f3ca0145790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:05.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.707+0000 7f3c9ffff640 1 -- 172.21.15.161:0/378292594 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ca014b9c0 con 0x7f3ca014c5f0 2024-03-28T23:35:05.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.707+0000 7f3ca4c87640 1 --2- 172.21.15.161:0/378292594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c88004800 0x7f3ca0145790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:05.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.707+0000 7f3c9ffff640 1 --2- 172.21.15.161:0/378292594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ca014c5f0 0x7f3ca0156ee0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f3c9402c2f0 tx=0x7f3c94004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:05.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.707+0000 7f3c9d7fa640 1 -- 172.21.15.161:0/378292594 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c94042020 con 0x7f3ca014c5f0 2024-03-28T23:35:05.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.707+0000 7f3c9d7fa640 1 -- 172.21.15.161:0/378292594 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c9403a040 con 0x7f3ca014c5f0 2024-03-28T23:35:05.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.707+0000 7f3ca6eeb640 1 -- 172.21.15.161:0/378292594 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ca014db20 con 0x7f3ca014c5f0 2024-03-28T23:35:05.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.707+0000 7f3c9d7fa640 1 -- 172.21.15.161:0/378292594 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c94039070 con 0x7f3ca014c5f0 2024-03-28T23:35:05.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.707+0000 7f3ca6eeb640 1 -- 172.21.15.161:0/378292594 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ca014b370 con 0x7f3ca014c5f0 2024-03-28T23:35:05.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.708+0000 7f3ca6eeb640 1 -- 172.21.15.161:0/378292594 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ca004f4b0 con 0x7f3ca014c5f0 2024-03-28T23:35:05.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.709+0000 7f3c9d7fa640 1 -- 172.21.15.161:0/378292594 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3c94045020 con 0x7f3ca014c5f0 2024-03-28T23:35:05.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.709+0000 7f3c9d7fa640 1 --2- 172.21.15.161:0/378292594 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3c8c06b500 0x7f3c8c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:05.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.709+0000 7f3c9d7fa640 1 -- 172.21.15.161:0/378292594 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5507+0+0 (secure 0 0 0) 0x7f3c940ab810 con 0x7f3ca014c5f0 2024-03-28T23:35:05.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.709+0000 7f3ca4c87640 1 --2- 172.21.15.161:0/378292594 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3c8c06b500 0x7f3c8c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:05.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.710+0000 7f3ca4c87640 1 --2- 172.21.15.161:0/378292594 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3c8c06b500 0x7f3c8c06d9c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0151120 tx=0x7f3c90009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:05.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.714+0000 7f3c9d7fa640 1 -- 172.21.15.161:0/378292594 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3c9404c030 con 0x7f3ca014c5f0 2024-03-28T23:35:05.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.886+0000 7f3ca6eeb640 1 -- 172.21.15.161:0/378292594 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f3ca014df90 con 0x7f3ca014c5f0 2024-03-28T23:35:05.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.891+0000 7f3c9d7fa640 1 -- 172.21.15.161:0/378292594 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v43) ==== 104+0+0 (secure 0 0 0) 0x7f3c9407a900 con 0x7f3ca014c5f0 2024-03-28T23:35:05.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.893+0000 7f3ca6eeb640 1 -- 172.21.15.161:0/378292594 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3c8c06b500 msgr2=0x7f3c8c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:05.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.893+0000 7f3ca6eeb640 1 --2- 172.21.15.161:0/378292594 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3c8c06b500 0x7f3c8c06d9c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0151120 tx=0x7f3c90009040 comp rx=0 tx=0).stop 2024-03-28T23:35:05.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.893+0000 7f3ca6eeb640 1 -- 172.21.15.161:0/378292594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ca014c5f0 msgr2=0x7f3ca0156ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:05.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.893+0000 7f3ca6eeb640 1 --2- 172.21.15.161:0/378292594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ca014c5f0 0x7f3ca0156ee0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f3c9402c2f0 tx=0x7f3c94004730 comp rx=0 tx=0).stop 2024-03-28T23:35:05.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.893+0000 7f3ca6eeb640 1 -- 172.21.15.161:0/378292594 shutdown_connections 2024-03-28T23:35:05.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.893+0000 7f3ca6eeb640 1 --2- 172.21.15.161:0/378292594 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3c8c06b500 0x7f3c8c06d9c0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:05.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.893+0000 7f3ca6eeb640 1 --2- 172.21.15.161:0/378292594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ca014c5f0 0x7f3ca0156ee0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:05.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.893+0000 7f3ca6eeb640 1 --2- 172.21.15.161:0/378292594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ca00729b0 0x7f3ca0072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:05.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.893+0000 7f3ca6eeb640 1 --2- 172.21.15.161:0/378292594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c88004800 0x7f3ca0145790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:05.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.894+0000 7f3ca6eeb640 1 -- 172.21.15.161:0/378292594 >> 172.21.15.161:0/378292594 conn(0x7f3ca00a83f0 msgr2=0x7f3ca00ab280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:05.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.894+0000 7f3ca6eeb640 1 -- 172.21.15.161:0/378292594 shutdown_connections 2024-03-28T23:35:05.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:05.894+0000 7f3ca6eeb640 1 -- 172.21.15.161:0/378292594 wait complete. 2024-03-28T23:35:05.906 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.a mds_join_fs 2024-03-28T23:35:06.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.012+0000 7f725c0ed640 1 Processor -- start 2024-03-28T23:35:06.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.012+0000 7f725c0ed640 1 -- start start 2024-03-28T23:35:06.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.013+0000 7f725c0ed640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7254152940 0x7f72540c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:06.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.013+0000 7f725c0ed640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f72540ca580 0x7f72540c8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:06.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.013+0000 7f725c0ed640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72540c6df0 con 0x7f72540c9480 2024-03-28T23:35:06.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.013+0000 7f725c0ed640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f72540c70d0 con 0x7f7254152940 2024-03-28T23:35:06.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.013+0000 7f725c0ed640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f72540c5830 con 0x7f72540ca580 2024-03-28T23:35:06.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.013+0000 7f7259e89640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7254152940 0x7f72540c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:06.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.013+0000 7f7259e89640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7254152940 0x7f72540c8a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57540/0 (socket says 172.21.15.161:57540) 2024-03-28T23:35:06.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.013+0000 7f7259e89640 1 -- 172.21.15.161:0/4265824068 learned_addr learned my addr 172.21.15.161:0/4265824068 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:06.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.013+0000 7f7259688640 1 --2- 172.21.15.161:0/4265824068 >> v2:172.21.15.170:3300/0 conn(0x7f72540ca580 0x7f72540c8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:06.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.014+0000 7f7259e89640 1 -- 172.21.15.161:0/4265824068 >> v2:172.21.15.170:3300/0 conn(0x7f72540ca580 msgr2=0x7f72540c8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:06.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.014+0000 7f7259e89640 1 --2- 172.21.15.161:0/4265824068 >> v2:172.21.15.170:3300/0 conn(0x7f72540ca580 0x7f72540c8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.014+0000 7f7259e89640 1 -- 172.21.15.161:0/4265824068 >> v1:172.21.15.170:6789/0 conn(0x7f72540c9480 legacy=0x7f7254155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:06.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.014+0000 7f7259e89640 1 -- 172.21.15.161:0/4265824068 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72540c5b10 con 0x7f7254152940 2024-03-28T23:35:06.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.014+0000 7f7259e89640 1 --2- 172.21.15.161:0/4265824068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7254152940 0x7f72540c8a00 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f72540b0b70 tx=0x7f725002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aec575a8904bfcab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:06.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.014+0000 7f7258e87640 1 -- 172.21.15.161:0/4265824068 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725002c9f0 con 0x7f7254152940 2024-03-28T23:35:06.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.014+0000 7f7258e87640 1 -- 172.21.15.161:0/4265824068 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f725002cb90 con 0x7f7254152940 2024-03-28T23:35:06.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.015+0000 7f7258e87640 1 -- 172.21.15.161:0/4265824068 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72500343f0 con 0x7f7254152940 2024-03-28T23:35:06.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.015+0000 7f725c0ed640 1 -- 172.21.15.161:0/4265824068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7254152940 msgr2=0x7f72540c8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:06.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.015+0000 7f725c0ed640 1 --2- 172.21.15.161:0/4265824068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7254152940 0x7f72540c8a00 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f72540b0b70 tx=0x7f725002be50 comp rx=0 tx=0).stop 2024-03-28T23:35:06.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.015+0000 7f725c0ed640 1 -- 172.21.15.161:0/4265824068 shutdown_connections 2024-03-28T23:35:06.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.015+0000 7f725c0ed640 1 --2- 172.21.15.161:0/4265824068 >> v2:172.21.15.170:3300/0 conn(0x7f72540ca580 0x7f72540c8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.015+0000 7f725c0ed640 1 --2- 172.21.15.161:0/4265824068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7254152940 0x7f72540c8a00 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.015+0000 7f725c0ed640 1 -- 172.21.15.161:0/4265824068 >> 172.21.15.161:0/4265824068 conn(0x7f72540caa00 msgr2=0x7f72540cae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:06.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.015+0000 7f725c0ed640 1 -- 172.21.15.161:0/4265824068 shutdown_connections 2024-03-28T23:35:06.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.015+0000 7f725c0ed640 1 -- 172.21.15.161:0/4265824068 wait complete. 2024-03-28T23:35:06.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.016+0000 7f725c0ed640 1 Processor -- start 2024-03-28T23:35:06.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.016+0000 7f725c0ed640 1 -- start start 2024-03-28T23:35:06.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.017+0000 7f725c0ed640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72540c9480 0x7f725414be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:06.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.017+0000 7f725c0ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72540ca580 0x7f72540c42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:06.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.017+0000 7f725c0ed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7254152940 0x7f72541540c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:06.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.017+0000 7f725c0ed640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f72540c61c0 con 0x7f72540c9480 2024-03-28T23:35:06.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.017+0000 7f725c0ed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f72540c6040 con 0x7f7254152940 2024-03-28T23:35:06.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.017+0000 7f725c0ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7254155930 con 0x7f72540ca580 2024-03-28T23:35:06.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.017+0000 7f7259e89640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72540c9480 0x7f725414be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:06.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.017+0000 7f7259e89640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72540c9480 0x7f725414be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57542/0 (socket says 172.21.15.161:57542) 2024-03-28T23:35:06.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.017+0000 7f7259e89640 1 -- 172.21.15.161:0/288325213 learned_addr learned my addr 172.21.15.161:0/288325213 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:06.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.017+0000 7f7259688640 1 --2- 172.21.15.161:0/288325213 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72540ca580 0x7f72540c42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:06.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.017+0000 7f7259e89640 1 -- 172.21.15.161:0/288325213 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72540ca580 msgr2=0x7f72540c42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:06.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.017+0000 7f725a68a640 1 --2- 172.21.15.161:0/288325213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7254152940 0x7f72541540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:06.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.017+0000 7f7259e89640 1 --2- 172.21.15.161:0/288325213 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72540ca580 0x7f72540c42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.017+0000 7f7259e89640 1 -- 172.21.15.161:0/288325213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7254152940 msgr2=0x7f72541540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:06.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.017+0000 7f7259e89640 1 --2- 172.21.15.161:0/288325213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7254152940 0x7f72541540c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.017+0000 7f7259e89640 1 -- 172.21.15.161:0/288325213 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72540ae0c0 con 0x7f72540c9480 2024-03-28T23:35:06.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.017+0000 7f7259e89640 1 --2- 172.21.15.161:0/288325213 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72540c9480 0x7f725414be60 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f7250000f80 tx=0x7f7250004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:06.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.018+0000 7f724a7fc640 1 -- 172.21.15.161:0/288325213 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7250044020 con 0x7f72540c9480 2024-03-28T23:35:06.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.018+0000 7f724a7fc640 1 -- 172.21.15.161:0/288325213 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7250040470 con 0x7f72540c9480 2024-03-28T23:35:06.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.018+0000 7f725c0ed640 1 -- 172.21.15.161:0/288325213 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72541554d0 con 0x7f72540c9480 2024-03-28T23:35:06.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.018+0000 7f724a7fc640 1 -- 172.21.15.161:0/288325213 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7250009510 con 0x7f72540c9480 2024-03-28T23:35:06.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.018+0000 7f725c0ed640 1 -- 172.21.15.161:0/288325213 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72541ebb10 con 0x7f72540c9480 2024-03-28T23:35:06.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.019+0000 7f725c0ed640 1 -- 172.21.15.161:0/288325213 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72540ae2c0 con 0x7f72540c9480 2024-03-28T23:35:06.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.019+0000 7f724a7fc640 1 -- 172.21.15.161:0/288325213 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7250047020 con 0x7f72540c9480 2024-03-28T23:35:06.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.020+0000 7f724a7fc640 1 --2- 172.21.15.161:0/288325213 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f723806b4d0 0x7f723806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:06.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.020+0000 7f7259688640 1 --2- 172.21.15.161:0/288325213 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f723806b4d0 0x7f723806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:06.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.020+0000 7f724a7fc640 1 -- 172.21.15.161:0/288325213 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5507+0+0 (secure 0 0 0) 0x7f72500aca20 con 0x7f72540c9480 2024-03-28T23:35:06.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.020+0000 7f7259688640 1 --2- 172.21.15.161:0/288325213 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f723806b4d0 0x7f723806d990 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7254154c60 tx=0x7f7244008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:06.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.024+0000 7f724a7fc640 1 -- 172.21.15.161:0/288325213 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f72540ae2c0 con 0x7f72540c9480 2024-03-28T23:35:06.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.197+0000 7f725c0ed640 1 -- 172.21.15.161:0/288325213 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.a", "name": "mds_join_fs"} v 0) -- 0x7f725404f4b0 con 0x7f72540c9480 2024-03-28T23:35:06.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.200+0000 7f724a7fc640 1 -- 172.21.15.161:0/288325213 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.a", "name": "mds_join_fs"}]=0 v44) ==== 96+0+0 (secure 0 0 0) 0x7f725004e030 con 0x7f72540c9480 2024-03-28T23:35:06.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.202+0000 7f725c0ed640 1 -- 172.21.15.161:0/288325213 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f723806b4d0 msgr2=0x7f723806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:06.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.202+0000 7f725c0ed640 1 --2- 172.21.15.161:0/288325213 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f723806b4d0 0x7f723806d990 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7254154c60 tx=0x7f7244008040 comp rx=0 tx=0).stop 2024-03-28T23:35:06.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.202+0000 7f725c0ed640 1 -- 172.21.15.161:0/288325213 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72540c9480 msgr2=0x7f725414be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:06.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.202+0000 7f725c0ed640 1 --2- 172.21.15.161:0/288325213 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72540c9480 0x7f725414be60 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f7250000f80 tx=0x7f7250004a40 comp rx=0 tx=0).stop 2024-03-28T23:35:06.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.202+0000 7f725c0ed640 1 -- 172.21.15.161:0/288325213 shutdown_connections 2024-03-28T23:35:06.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.202+0000 7f725c0ed640 1 --2- 172.21.15.161:0/288325213 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f723806b4d0 0x7f723806d990 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.202+0000 7f725c0ed640 1 --2- 172.21.15.161:0/288325213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7254152940 0x7f72541540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.202+0000 7f725c0ed640 1 --2- 172.21.15.161:0/288325213 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72540ca580 0x7f72540c42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.202+0000 7f725c0ed640 1 --2- 172.21.15.161:0/288325213 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72540c9480 0x7f725414be60 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.202+0000 7f725c0ed640 1 -- 172.21.15.161:0/288325213 >> 172.21.15.161:0/288325213 conn(0x7f72540caa00 msgr2=0x7f72540cade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:06.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.202+0000 7f725c0ed640 1 -- 172.21.15.161:0/288325213 shutdown_connections 2024-03-28T23:35:06.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.202+0000 7f725c0ed640 1 -- 172.21.15.161:0/288325213 wait complete. 2024-03-28T23:35:06.214 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady' 2024-03-28T23:35:06.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.314+0000 7f42a528f640 1 Processor -- start 2024-03-28T23:35:06.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.314+0000 7f42a528f640 1 -- start start 2024-03-28T23:35:06.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.315+0000 7f42a528f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f42a014be70 0x7f42a00ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:06.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.315+0000 7f42a528f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f42a00ab6e0 0x7f42a00aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:06.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.315+0000 7f42a528f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42a006ce80 0x7f42a00aab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.315+0000 7f42a528f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f42a014c250 con 0x7f42a00ab6e0 2024-03-28T23:35:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.315+0000 7f42a528f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f42a01529c0 con 0x7f42a014be70 2024-03-28T23:35:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.315+0000 7f42a528f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f42a00a9dc0 con 0x7f42a006ce80 2024-03-28T23:35:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.315+0000 7f429e7fc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f42a00ab6e0 0x7f42a00aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.315+0000 7f429e7fc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f42a00ab6e0 0x7f42a00aa5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57550/0 (socket says 172.21.15.161:57550) 2024-03-28T23:35:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.315+0000 7f429e7fc640 1 -- 172.21.15.161:0/27725088 learned_addr learned my addr 172.21.15.161:0/27725088 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.315+0000 7f429e7fc640 1 -- 172.21.15.161:0/27725088 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42a006ce80 msgr2=0x7f42a00aab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.315+0000 7f429e7fc640 1 --2- 172.21.15.161:0/27725088 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42a006ce80 0x7f42a00aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.315+0000 7f429e7fc640 1 -- 172.21.15.161:0/27725088 >> v2:172.21.15.170:3300/0 conn(0x7f42a014be70 msgr2=0x7f42a00ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.315+0000 7f429e7fc640 1 --2- 172.21.15.161:0/27725088 >> v2:172.21.15.170:3300/0 conn(0x7f42a014be70 0x7f42a00ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.315+0000 7f429e7fc640 1 -- 172.21.15.161:0/27725088 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42a00aa0a0 con 0x7f42a00ab6e0 2024-03-28T23:35:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.316+0000 7f429e7fc640 1 --2- 172.21.15.161:0/27725088 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42a00ab6e0 0x7f42a00aa5c0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f42a00c7520 tx=0x7f429402dfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f69f2dfabeaf1c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.316+0000 7f429dffb640 1 -- 172.21.15.161:0/27725088 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429403b070 con 0x7f42a00ab6e0 2024-03-28T23:35:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.316+0000 7f429dffb640 1 -- 172.21.15.161:0/27725088 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f429402ed70 con 0x7f42a00ab6e0 2024-03-28T23:35:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.316+0000 7f429dffb640 1 -- 172.21.15.161:0/27725088 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4294036660 con 0x7f42a00ab6e0 2024-03-28T23:35:06.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.317+0000 7f42a528f640 1 -- 172.21.15.161:0/27725088 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42a00ab6e0 msgr2=0x7f42a00aa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:06.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.317+0000 7f42a528f640 1 --2- 172.21.15.161:0/27725088 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42a00ab6e0 0x7f42a00aa5c0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f42a00c7520 tx=0x7f429402dfc0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.317+0000 7f42a528f640 1 -- 172.21.15.161:0/27725088 shutdown_connections 2024-03-28T23:35:06.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.317+0000 7f42a528f640 1 --2- 172.21.15.161:0/27725088 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42a006ce80 0x7f42a00aab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.317+0000 7f42a528f640 1 --2- 172.21.15.161:0/27725088 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42a00ab6e0 0x7f42a00aa5c0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.317+0000 7f42a528f640 1 --2- 172.21.15.161:0/27725088 >> v2:172.21.15.170:3300/0 conn(0x7f42a014be70 0x7f42a00ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.317+0000 7f42a528f640 1 -- 172.21.15.161:0/27725088 >> 172.21.15.161:0/27725088 conn(0x7f42a00c6670 msgr2=0x7f42a00c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:06.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.317+0000 7f42a528f640 1 -- 172.21.15.161:0/27725088 shutdown_connections 2024-03-28T23:35:06.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.317+0000 7f42a528f640 1 -- 172.21.15.161:0/27725088 wait complete. 2024-03-28T23:35:06.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.318+0000 7f42a528f640 1 Processor -- start 2024-03-28T23:35:06.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.318+0000 7f42a528f640 1 -- start start 2024-03-28T23:35:06.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.319+0000 7f42a528f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42a006ce80 0x7f42a00c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:06.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.319+0000 7f42a528f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42a00ab6e0 0x7f42a01e6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:06.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.319+0000 7f42a528f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42a014be70 0x7f42a01e7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:06.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.319+0000 7f42a528f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f42a006b2d0 con 0x7f42a00ab6e0 2024-03-28T23:35:06.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.319+0000 7f42a528f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f42a006b150 con 0x7f42a006ce80 2024-03-28T23:35:06.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.319+0000 7f42a528f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f42a00c1520 con 0x7f42a014be70 2024-03-28T23:35:06.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.319+0000 7f429e7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42a00ab6e0 0x7f42a01e6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:06.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.319+0000 7f429e7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42a00ab6e0 0x7f42a01e6d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57564/0 (socket says 172.21.15.161:57564) 2024-03-28T23:35:06.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.319+0000 7f429e7fc640 1 -- 172.21.15.161:0/2685810029 learned_addr learned my addr 172.21.15.161:0/2685810029 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:06.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.319+0000 7f429effd640 1 --2- 172.21.15.161:0/2685810029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42a006ce80 0x7f42a00c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:06.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.319+0000 7f429f7fe640 1 --2- 172.21.15.161:0/2685810029 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42a014be70 0x7f42a01e7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:06.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.319+0000 7f429e7fc640 1 -- 172.21.15.161:0/2685810029 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42a014be70 msgr2=0x7f42a01e7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:06.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.319+0000 7f429e7fc640 1 --2- 172.21.15.161:0/2685810029 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42a014be70 0x7f42a01e7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.319+0000 7f429e7fc640 1 -- 172.21.15.161:0/2685810029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42a006ce80 msgr2=0x7f42a00c59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:06.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.319+0000 7f429e7fc640 1 --2- 172.21.15.161:0/2685810029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42a006ce80 0x7f42a00c59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.319+0000 7f429e7fc640 1 -- 172.21.15.161:0/2685810029 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42a01ea230 con 0x7f42a00ab6e0 2024-03-28T23:35:06.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.319+0000 7f429effd640 1 --2- 172.21.15.161:0/2685810029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42a006ce80 0x7f42a00c59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:06.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.319+0000 7f429f7fe640 1 --2- 172.21.15.161:0/2685810029 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42a014be70 0x7f42a01e7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:06.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.319+0000 7f429e7fc640 1 --2- 172.21.15.161:0/2685810029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42a00ab6e0 0x7f42a01e6d20 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f429402e000 tx=0x7f42940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:06.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.319+0000 7f42877fe640 1 -- 172.21.15.161:0/2685810029 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4294045070 con 0x7f42a00ab6e0 2024-03-28T23:35:06.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.319+0000 7f42877fe640 1 -- 172.21.15.161:0/2685810029 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4294036e20 con 0x7f42a00ab6e0 2024-03-28T23:35:06.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.319+0000 7f42a528f640 1 -- 172.21.15.161:0/2685810029 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42a01ea510 con 0x7f42a00ab6e0 2024-03-28T23:35:06.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.319+0000 7f42877fe640 1 -- 172.21.15.161:0/2685810029 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429403b040 con 0x7f42a00ab6e0 2024-03-28T23:35:06.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.319+0000 7f42a528f640 1 -- 172.21.15.161:0/2685810029 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42a01eaad0 con 0x7f42a00ab6e0 2024-03-28T23:35:06.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.320+0000 7f42a528f640 1 -- 172.21.15.161:0/2685810029 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42a004f4b0 con 0x7f42a00ab6e0 2024-03-28T23:35:06.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.321+0000 7f42877fe640 1 -- 172.21.15.161:0/2685810029 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4294048070 con 0x7f42a00ab6e0 2024-03-28T23:35:06.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.321+0000 7f42877fe640 1 --2- 172.21.15.161:0/2685810029 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f428c06b4d0 0x7f428c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:06.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.321+0000 7f429effd640 1 --2- 172.21.15.161:0/2685810029 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f428c06b4d0 0x7f428c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:06.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.321+0000 7f42877fe640 1 -- 172.21.15.161:0/2685810029 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5507+0+0 (secure 0 0 0) 0x7f42940ae960 con 0x7f42a00ab6e0 2024-03-28T23:35:06.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.322+0000 7f429effd640 1 --2- 172.21.15.161:0/2685810029 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f428c06b4d0 0x7f428c06d990 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4290009890 tx=0x7f4290008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:06.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.326+0000 7f42877fe640 1 -- 172.21.15.161:0/2685810029 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f429407d930 con 0x7f42a00ab6e0 2024-03-28T23:35:06.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.571+0000 7f42a528f640 1 -- 172.21.15.161:0/2685810029 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady"]} v 0) -- 0x7f42a00c0ff0 con 0x7f42a00ab6e0 2024-03-28T23:35:06.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.589+0000 7f42877fe640 1 -- 172.21.15.161:0/2685810029 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady"]}]=0 v310) ==== 145+0+0 (secure 0 0 0) 0x7f429407d230 con 0x7f42a00ab6e0 2024-03-28T23:35:06.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.591+0000 7f42a528f640 1 -- 172.21.15.161:0/2685810029 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f428c06b4d0 msgr2=0x7f428c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:06.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.591+0000 7f42a528f640 1 --2- 172.21.15.161:0/2685810029 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f428c06b4d0 0x7f428c06d990 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4290009890 tx=0x7f4290008040 comp rx=0 tx=0).stop 2024-03-28T23:35:06.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.591+0000 7f42a528f640 1 -- 172.21.15.161:0/2685810029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42a00ab6e0 msgr2=0x7f42a01e6d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:06.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.591+0000 7f42a528f640 1 --2- 172.21.15.161:0/2685810029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42a00ab6e0 0x7f42a01e6d20 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f429402e000 tx=0x7f42940046c0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.591+0000 7f42a528f640 1 -- 172.21.15.161:0/2685810029 shutdown_connections 2024-03-28T23:35:06.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.591+0000 7f42a528f640 1 --2- 172.21.15.161:0/2685810029 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f428c06b4d0 0x7f428c06d990 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.591+0000 7f42a528f640 1 --2- 172.21.15.161:0/2685810029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42a00ab6e0 0x7f42a01e6d20 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.591+0000 7f42a528f640 1 --2- 172.21.15.161:0/2685810029 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42a014be70 0x7f42a01e7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.591+0000 7f42a528f640 1 --2- 172.21.15.161:0/2685810029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42a006ce80 0x7f42a00c59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.591+0000 7f42a528f640 1 -- 172.21.15.161:0/2685810029 >> 172.21.15.161:0/2685810029 conn(0x7f42a00c6670 msgr2=0x7f42a014a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:06.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.591+0000 7f42a528f640 1 -- 172.21.15.161:0/2685810029 shutdown_connections 2024-03-28T23:35:06.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.591+0000 7f42a528f640 1 -- 172.21.15.161:0/2685810029 wait complete. 2024-03-28T23:35:06.603 INFO:tasks.cephfs_test_runner:That a sole MDS with mds_join_fs set will come back as active eventually even after failover. ... ok 2024-03-28T23:35:06.603 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_unset (tasks.cephfs.test_failover.TestClusterAffinity) That unsetting mds_join_fs will cause failover if another high-affinity standby exists. 2024-03-28T23:35:06.603 INFO:tasks.cephfs_test_runner:test_join_fs_unset (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-28T23:35:06.604 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:35:06.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.704+0000 7f3f71bc2640 1 Processor -- start 2024-03-28T23:35:06.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.704+0000 7f3f71bc2640 1 -- start start 2024-03-28T23:35:06.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.705+0000 7f3f71bc2640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3f6c06ce80 0x7f3f6c0aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:06.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.705+0000 7f3f71bc2640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f6c14c250 con 0x7f3f6c14be70 2024-03-28T23:35:06.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.705+0000 7f3f71bc2640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f6c1529c0 con 0x7f3f6c0a9680 2024-03-28T23:35:06.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.705+0000 7f3f71bc2640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f3f6c0a9dc0 con 0x7f3f6c06ce80 2024-03-28T23:35:06.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.705+0000 7f3f6b7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f3f6c14be70 0x7f3f6c0ab1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45312/0 (socket says 172.21.15.161:45312) 2024-03-28T23:35:06.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.705+0000 7f3f6b7fe640 1 -- 172.21.15.161:0/3375114103 learned_addr learned my addr 172.21.15.161:0/3375114103 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:06.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.705+0000 7f3f6affd640 1 --2- 172.21.15.161:0/3375114103 >> v2:172.21.15.170:3300/0 conn(0x7f3f6c06ce80 0x7f3f6c0aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:06.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.705+0000 7f3f6a7fc640 1 -- 172.21.15.161:0/3375114103 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2057846055 0 0) 0x7f3f6c14c250 con 0x7f3f6c14be70 2024-03-28T23:35:06.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.705+0000 7f3f6a7fc640 1 -- 172.21.15.161:0/3375114103 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f54003520 con 0x7f3f6c14be70 2024-03-28T23:35:06.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.705+0000 7f3f6a7fc640 1 -- 172.21.15.161:0/3375114103 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2214928940 0 0) 0x7f3f54003520 con 0x7f3f6c14be70 2024-03-28T23:35:06.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.705+0000 7f3f6a7fc640 1 -- 172.21.15.161:0/3375114103 >> v2:172.21.15.170:3300/0 conn(0x7f3f6c06ce80 msgr2=0x7f3f6c0aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:06.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.705+0000 7f3f6a7fc640 1 --2- 172.21.15.161:0/3375114103 >> v2:172.21.15.170:3300/0 conn(0x7f3f6c06ce80 0x7f3f6c0aa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.705+0000 7f3f6a7fc640 1 -- 172.21.15.161:0/3375114103 >> v1:172.21.15.170:6789/0 conn(0x7f3f6c0a9680 legacy=0x7f3f6c15c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:06.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.705+0000 7f3f6a7fc640 1 -- 172.21.15.161:0/3375114103 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f6c0aa0a0 con 0x7f3f6c14be70 2024-03-28T23:35:06.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.705+0000 7f3f6a7fc640 1 -- 172.21.15.161:0/3375114103 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f3f5c001670 con 0x7f3f6c14be70 2024-03-28T23:35:06.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.705+0000 7f3f6a7fc640 1 -- 172.21.15.161:0/3375114103 >> v1:172.21.15.161:6789/0 conn(0x7f3f6c14be70 legacy=0x7f3f6c0ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:06.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.705+0000 7f3f6a7fc640 1 --2- 172.21.15.161:0/3375114103 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f54003a90 0x7f3f54003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:06.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.705+0000 7f3f6a7fc640 1 --2- 172.21.15.161:0/3375114103 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f54004800 0x7f3f54004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:06.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.705+0000 7f3f6a7fc640 1 --2- 172.21.15.161:0/3375114103 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f54005290 0x7f3f5400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:06.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.705+0000 7f3f6a7fc640 1 -- 172.21.15.161:0/3375114103 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3f6c0a9dc0 con 0x7f3f54003a90 2024-03-28T23:35:06.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.705+0000 7f3f6a7fc640 1 -- 172.21.15.161:0/3375114103 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5400c1e0 con 0x7f3f54004800 2024-03-28T23:35:06.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.705+0000 7f3f6a7fc640 1 -- 172.21.15.161:0/3375114103 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3f5400c3e0 con 0x7f3f54005290 2024-03-28T23:35:06.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.706+0000 7f3f6b7fe640 1 --2- 172.21.15.161:0/3375114103 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f54003a90 0x7f3f54003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:06.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.706+0000 7f3f6affd640 1 --2- 172.21.15.161:0/3375114103 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f54004800 0x7f3f54004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:06.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.706+0000 7f3f6bfff640 1 --2- 172.21.15.161:0/3375114103 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f54005290 0x7f3f5400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:06.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.706+0000 7f3f6b7fe640 1 -- 172.21.15.161:0/3375114103 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f54005290 msgr2=0x7f3f5400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:06.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.706+0000 7f3f6b7fe640 1 --2- 172.21.15.161:0/3375114103 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f54005290 0x7f3f5400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.706+0000 7f3f6b7fe640 1 -- 172.21.15.161:0/3375114103 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f54004800 msgr2=0x7f3f54004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:06.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.706+0000 7f3f6b7fe640 1 --2- 172.21.15.161:0/3375114103 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f54004800 0x7f3f54004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.706+0000 7f3f6b7fe640 1 -- 172.21.15.161:0/3375114103 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f54003520 con 0x7f3f54003a90 2024-03-28T23:35:06.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.706+0000 7f3f6bfff640 1 -- 172.21.15.161:0/3375114103 reap_dead start 2024-03-28T23:35:06.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.707+0000 7f3f6b7fe640 1 --2- 172.21.15.161:0/3375114103 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f54003a90 0x7f3f54003eb0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c00c210 tx=0x7f3f54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff5377f704882c15 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:06.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.707+0000 7f3f6a7fc640 1 -- 172.21.15.161:0/3375114103 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c007d10 con 0x7f3f54003a90 2024-03-28T23:35:06.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.707+0000 7f3f6a7fc640 1 -- 172.21.15.161:0/3375114103 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f5c030070 con 0x7f3f54003a90 2024-03-28T23:35:06.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.707+0000 7f3f6a7fc640 1 -- 172.21.15.161:0/3375114103 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c005d30 con 0x7f3f54003a90 2024-03-28T23:35:06.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.707+0000 7f3f71bc2640 1 -- 172.21.15.161:0/3375114103 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f54003a90 msgr2=0x7f3f54003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:06.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.707+0000 7f3f71bc2640 1 --2- 172.21.15.161:0/3375114103 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f54003a90 0x7f3f54003eb0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c00c210 tx=0x7f3f54005830 comp rx=0 tx=0).stop 2024-03-28T23:35:06.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.707+0000 7f3f71bc2640 1 -- 172.21.15.161:0/3375114103 shutdown_connections 2024-03-28T23:35:06.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.707+0000 7f3f71bc2640 1 --2- 172.21.15.161:0/3375114103 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f54003a90 0x7f3f54003eb0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.707+0000 7f3f71bc2640 1 -- 172.21.15.161:0/3375114103 >> 172.21.15.161:0/3375114103 conn(0x7f3f6c0c6670 msgr2=0x7f3f6c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:06.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.707+0000 7f3f71bc2640 1 -- 172.21.15.161:0/3375114103 shutdown_connections 2024-03-28T23:35:06.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.707+0000 7f3f71bc2640 1 -- 172.21.15.161:0/3375114103 wait complete. 2024-03-28T23:35:06.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.708+0000 7f3f71bc2640 1 Processor -- start 2024-03-28T23:35:06.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.709+0000 7f3f71bc2640 1 -- start start 2024-03-28T23:35:06.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.709+0000 7f3f71bc2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f54003a90 0x7f3f6c1ee490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:06.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.709+0000 7f3f71bc2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f6c155f10 0x7f3f6c1dfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:06.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.709+0000 7f3f71bc2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f6c1e7920 0x7f3f6c1e7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:06.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.709+0000 7f3f71bc2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3f6c06b2d0 con 0x7f3f6c155f10 2024-03-28T23:35:06.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.709+0000 7f3f71bc2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3f6c06b150 con 0x7f3f6c1e7920 2024-03-28T23:35:06.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.709+0000 7f3f71bc2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3f6c0c1520 con 0x7f3f54003a90 2024-03-28T23:35:06.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.709+0000 7f3f6affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f6c155f10 0x7f3f6c1dfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:06.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.709+0000 7f3f6affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f6c155f10 0x7f3f6c1dfaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57576/0 (socket says 172.21.15.161:57576) 2024-03-28T23:35:06.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.709+0000 7f3f6affd640 1 -- 172.21.15.161:0/375920667 learned_addr learned my addr 172.21.15.161:0/375920667 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:06.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.710+0000 7f3f6b7fe640 1 --2- 172.21.15.161:0/375920667 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f54003a90 0x7f3f6c1ee490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:06.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.710+0000 7f3f6bfff640 1 --2- 172.21.15.161:0/375920667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f6c1e7920 0x7f3f6c1e7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:06.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.710+0000 7f3f6affd640 1 -- 172.21.15.161:0/375920667 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f54003a90 msgr2=0x7f3f6c1ee490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:06.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.710+0000 7f3f6affd640 1 --2- 172.21.15.161:0/375920667 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f54003a90 0x7f3f6c1ee490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.710+0000 7f3f6affd640 1 -- 172.21.15.161:0/375920667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f6c1e7920 msgr2=0x7f3f6c1e7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:06.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.710+0000 7f3f6affd640 1 --2- 172.21.15.161:0/375920667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f6c1e7920 0x7f3f6c1e7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.710+0000 7f3f6affd640 1 -- 172.21.15.161:0/375920667 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f6c155af0 con 0x7f3f6c155f10 2024-03-28T23:35:06.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.710+0000 7f3f6bfff640 1 --2- 172.21.15.161:0/375920667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f6c1e7920 0x7f3f6c1e7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:06.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.710+0000 7f3f6affd640 1 --2- 172.21.15.161:0/375920667 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f6c155f10 0x7f3f6c1dfaa0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f3f60011a70 tx=0x7f3f60011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:06.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.710+0000 7f3f53fff640 1 -- 172.21.15.161:0/375920667 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6000fb50 con 0x7f3f6c155f10 2024-03-28T23:35:06.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.710+0000 7f3f53fff640 1 -- 172.21.15.161:0/375920667 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f6000fcf0 con 0x7f3f6c155f10 2024-03-28T23:35:06.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.710+0000 7f3f71bc2640 1 -- 172.21.15.161:0/375920667 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f6c0c5ef0 con 0x7f3f6c155f10 2024-03-28T23:35:06.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.710+0000 7f3f53fff640 1 -- 172.21.15.161:0/375920667 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6000fea0 con 0x7f3f6c155f10 2024-03-28T23:35:06.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.710+0000 7f3f71bc2640 1 -- 172.21.15.161:0/375920667 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f6c0c50c0 con 0x7f3f6c155f10 2024-03-28T23:35:06.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.711+0000 7f3f71bc2640 1 -- 172.21.15.161:0/375920667 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f6c04f4b0 con 0x7f3f6c155f10 2024-03-28T23:35:06.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.711+0000 7f3f53fff640 1 -- 172.21.15.161:0/375920667 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3f60012040 con 0x7f3f6c155f10 2024-03-28T23:35:06.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.712+0000 7f3f53fff640 1 --2- 172.21.15.161:0/375920667 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3f5806b500 0x7f3f5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:06.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.712+0000 7f3f53fff640 1 -- 172.21.15.161:0/375920667 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5507+0+0 (secure 0 0 0) 0x7f3f6008a6e0 con 0x7f3f6c155f10 2024-03-28T23:35:06.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.712+0000 7f3f6b7fe640 1 --2- 172.21.15.161:0/375920667 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3f5806b500 0x7f3f5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:06.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.712+0000 7f3f6b7fe640 1 --2- 172.21.15.161:0/375920667 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3f5806b500 0x7f3f5806d9c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c034000 tx=0x7f3f5c039040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:06.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.716+0000 7f3f53fff640 1 -- 172.21.15.161:0/375920667 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3f60019030 con 0x7f3f6c155f10 2024-03-28T23:35:06.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.889+0000 7f3f71bc2640 1 -- 172.21.15.161:0/375920667 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3f6c0c0ff0 con 0x7f3f6c155f10 2024-03-28T23:35:06.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.890+0000 7f3f53fff640 1 -- 172.21.15.161:0/375920667 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) ==== 74+0+13603 (secure 0 0 0) 0x7f3f60059690 con 0x7f3f6c155f10 2024-03-28T23:35:06.890 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:06.890 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":127,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:35:05.578788+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6837/3929102513":"2024-03-29T23:35:02.574541+0000","172.21.15.161:6835/878057589":"2024-03-29T23:34:53.266532+0000","172.21.15.161:6836/3929102513":"2024-03-29T23:35:02.574541+0000","172.21.15.161:6834/878057589":"2024-03-29T23:34:53.266532+0000","172.21.15.161:6837/3432367324":"2024-03-29T23:34:50.245540+0000","172.21.15.161:6836/3432367324":"2024-03-29T23:34:50.245540+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:35:06.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.892+0000 7f3f71bc2640 1 -- 172.21.15.161:0/375920667 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3f5806b500 msgr2=0x7f3f5806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:06.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.892+0000 7f3f71bc2640 1 --2- 172.21.15.161:0/375920667 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3f5806b500 0x7f3f5806d9c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c034000 tx=0x7f3f5c039040 comp rx=0 tx=0).stop 2024-03-28T23:35:06.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.892+0000 7f3f71bc2640 1 -- 172.21.15.161:0/375920667 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f6c155f10 msgr2=0x7f3f6c1dfaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:06.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.892+0000 7f3f71bc2640 1 --2- 172.21.15.161:0/375920667 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f6c155f10 0x7f3f6c1dfaa0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f3f60011a70 tx=0x7f3f60011f30 comp rx=0 tx=0).stop 2024-03-28T23:35:06.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.892+0000 7f3f71bc2640 1 -- 172.21.15.161:0/375920667 shutdown_connections 2024-03-28T23:35:06.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.892+0000 7f3f71bc2640 1 --2- 172.21.15.161:0/375920667 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3f5806b500 0x7f3f5806d9c0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.892+0000 7f3f71bc2640 1 --2- 172.21.15.161:0/375920667 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f6c155f10 0x7f3f6c1dfaa0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.892+0000 7f3f71bc2640 1 --2- 172.21.15.161:0/375920667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f6c1e7920 0x7f3f6c1e7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.892+0000 7f3f71bc2640 1 --2- 172.21.15.161:0/375920667 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f54003a90 0x7f3f6c1ee490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:06.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.892+0000 7f3f71bc2640 1 -- 172.21.15.161:0/375920667 >> 172.21.15.161:0/375920667 conn(0x7f3f6c0c6670 msgr2=0x7f3f6c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:06.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.892+0000 7f3f71bc2640 1 -- 172.21.15.161:0/375920667 shutdown_connections 2024-03-28T23:35:06.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:06.892+0000 7f3f71bc2640 1 -- 172.21.15.161:0/375920667 wait complete. 2024-03-28T23:35:06.903 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:35:06.904 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:35:07.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.007+0000 7fb020f02640 1 Processor -- start 2024-03-28T23:35:07.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.007+0000 7fb020f02640 1 -- start start 2024-03-28T23:35:07.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb020f02640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb01c151e70 0x7fb01c152250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:07.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb020f02640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb01c06cdd0 0x7fb01c15c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:07.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb020f02640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb01c0c1e40 con 0x7fb01c152790 2024-03-28T23:35:07.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb020f02640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fb01c0c2120 con 0x7fb01c151e70 2024-03-28T23:35:07.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb020f02640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb01c0c2400 con 0x7fb01c06cdd0 2024-03-28T23:35:07.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb013fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb01c152790 0x7fb01c06c6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45318/0 (socket says 172.21.15.161:45318) 2024-03-28T23:35:07.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb013fff640 1 -- 172.21.15.161:0/3493413680 learned_addr learned my addr 172.21.15.161:0/3493413680 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:07.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb01ad9d640 1 --2- 172.21.15.161:0/3493413680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb01c06cdd0 0x7fb01c15c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:07.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb01a59c640 1 --2- 172.21.15.161:0/3493413680 >> v2:172.21.15.170:3300/0 conn(0x7fb01c151e70 0x7fb01c152250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:07.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb019d9b640 1 -- 172.21.15.161:0/3493413680 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1608373411 0 0) 0x7fb01c0c1e40 con 0x7fb01c152790 2024-03-28T23:35:07.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb019d9b640 1 -- 172.21.15.161:0/3493413680 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb008005730 con 0x7fb01c152790 2024-03-28T23:35:07.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb019d9b640 1 -- 172.21.15.161:0/3493413680 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3546349415 0 0) 0x7fb008005730 con 0x7fb01c152790 2024-03-28T23:35:07.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb019d9b640 1 -- 172.21.15.161:0/3493413680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb01c06cdd0 msgr2=0x7fb01c15c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:07.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb019d9b640 1 --2- 172.21.15.161:0/3493413680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb01c06cdd0 0x7fb01c15c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb019d9b640 1 -- 172.21.15.161:0/3493413680 >> v2:172.21.15.170:3300/0 conn(0x7fb01c151e70 msgr2=0x7fb01c152250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:07.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb019d9b640 1 --2- 172.21.15.161:0/3493413680 >> v2:172.21.15.170:3300/0 conn(0x7fb01c151e70 0x7fb01c152250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb019d9b640 1 -- 172.21.15.161:0/3493413680 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb01c0c26e0 con 0x7fb01c152790 2024-03-28T23:35:07.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb019d9b640 1 -- 172.21.15.161:0/3493413680 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fb004001670 con 0x7fb01c152790 2024-03-28T23:35:07.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb01a59c640 1 --2- 172.21.15.161:0/3493413680 >> v2:172.21.15.170:3300/0 conn(0x7fb01c151e70 0x7fb01c152250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:07.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb019d9b640 1 -- 172.21.15.161:0/3493413680 >> v1:172.21.15.161:6789/0 conn(0x7fb01c152790 legacy=0x7fb01c06c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:07.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb019d9b640 1 --2- 172.21.15.161:0/3493413680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb008005bc0 0x7fb008005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:07.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb019d9b640 1 --2- 172.21.15.161:0/3493413680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb01c06cdd0 0x7fb008006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:07.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb019d9b640 1 --2- 172.21.15.161:0/3493413680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb008006f50 0x7fb0080073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:07.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb019d9b640 1 -- 172.21.15.161:0/3493413680 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb01c0c2120 con 0x7fb008005bc0 2024-03-28T23:35:07.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb019d9b640 1 -- 172.21.15.161:0/3493413680 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb01c0c2400 con 0x7fb008006f50 2024-03-28T23:35:07.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.008+0000 7fb019d9b640 1 -- 172.21.15.161:0/3493413680 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb00800dc00 con 0x7fb01c06cdd0 2024-03-28T23:35:07.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.009+0000 7fb01a59c640 1 --2- 172.21.15.161:0/3493413680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb008005bc0 0x7fb008005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:07.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.009+0000 7fb013fff640 1 --2- 172.21.15.161:0/3493413680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb01c06cdd0 0x7fb008006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:07.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.009+0000 7fb01ad9d640 1 --2- 172.21.15.161:0/3493413680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb008006f50 0x7fb0080073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:07.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.009+0000 7fb01a59c640 1 -- 172.21.15.161:0/3493413680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb01c06cdd0 msgr2=0x7fb008006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:07.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.009+0000 7fb01a59c640 1 --2- 172.21.15.161:0/3493413680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb01c06cdd0 0x7fb008006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.009+0000 7fb01a59c640 1 -- 172.21.15.161:0/3493413680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb008006f50 msgr2=0x7fb0080073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:07.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.009+0000 7fb01a59c640 1 --2- 172.21.15.161:0/3493413680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb008006f50 0x7fb0080073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.009+0000 7fb01a59c640 1 -- 172.21.15.161:0/3493413680 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb008005730 con 0x7fb008005bc0 2024-03-28T23:35:07.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.009+0000 7fb01a59c640 1 --2- 172.21.15.161:0/3493413680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb008005bc0 0x7fb008005fe0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009c90 tx=0x7fb008007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd8a9583f49537e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:07.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.010+0000 7fb019d9b640 1 -- 172.21.15.161:0/3493413680 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c039070 con 0x7fb008005bc0 2024-03-28T23:35:07.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.010+0000 7fb019d9b640 1 -- 172.21.15.161:0/3493413680 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb00c02c980 con 0x7fb008005bc0 2024-03-28T23:35:07.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.010+0000 7fb019d9b640 1 -- 172.21.15.161:0/3493413680 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c009860 con 0x7fb008005bc0 2024-03-28T23:35:07.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.010+0000 7fb020f02640 1 -- 172.21.15.161:0/3493413680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb008005bc0 msgr2=0x7fb008005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:07.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.010+0000 7fb020f02640 1 --2- 172.21.15.161:0/3493413680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb008005bc0 0x7fb008005fe0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009c90 tx=0x7fb008007a40 comp rx=0 tx=0).stop 2024-03-28T23:35:07.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.010+0000 7fb01ad9d640 1 -- 172.21.15.161:0/3493413680 reap_dead start 2024-03-28T23:35:07.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.010+0000 7fb020f02640 1 -- 172.21.15.161:0/3493413680 shutdown_connections 2024-03-28T23:35:07.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.010+0000 7fb020f02640 1 -- 172.21.15.161:0/3493413680 >> 172.21.15.161:0/3493413680 conn(0x7fb01c0c3390 msgr2=0x7fb01c0c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:07.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.010+0000 7fb020f02640 1 -- 172.21.15.161:0/3493413680 shutdown_connections 2024-03-28T23:35:07.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.010+0000 7fb020f02640 1 -- 172.21.15.161:0/3493413680 wait complete. 2024-03-28T23:35:07.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.011+0000 7fb020f02640 1 Processor -- start 2024-03-28T23:35:07.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.011+0000 7fb020f02640 1 -- start start 2024-03-28T23:35:07.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.012+0000 7fb020f02640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb008005bc0 0x7fb01c1db7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:07.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.012+0000 7fb020f02640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb01c0ab6a0 0x7fb01c1e6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:07.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.012+0000 7fb020f02640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb01c0abb20 0x7fb01c1e71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:07.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.012+0000 7fb020f02640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb01c0c2d80 con 0x7fb01c0abb20 2024-03-28T23:35:07.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.012+0000 7fb020f02640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb01c0c2c00 con 0x7fb008005bc0 2024-03-28T23:35:07.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.012+0000 7fb020f02640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb01c0c6350 con 0x7fb01c0ab6a0 2024-03-28T23:35:07.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.012+0000 7fb01ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb01c0abb20 0x7fb01c1e71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:07.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.012+0000 7fb01ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb01c0abb20 0x7fb01c1e71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57594/0 (socket says 172.21.15.161:57594) 2024-03-28T23:35:07.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.012+0000 7fb01ad9d640 1 -- 172.21.15.161:0/412519008 learned_addr learned my addr 172.21.15.161:0/412519008 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:07.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.012+0000 7fb013fff640 1 --2- 172.21.15.161:0/412519008 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb01c0ab6a0 0x7fb01c1e6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:07.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.012+0000 7fb01a59c640 1 --2- 172.21.15.161:0/412519008 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb008005bc0 0x7fb01c1db7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:07.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.012+0000 7fb01ad9d640 1 -- 172.21.15.161:0/412519008 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb01c0ab6a0 msgr2=0x7fb01c1e6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:07.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.012+0000 7fb01ad9d640 1 --2- 172.21.15.161:0/412519008 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb01c0ab6a0 0x7fb01c1e6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.012+0000 7fb01ad9d640 1 -- 172.21.15.161:0/412519008 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb008005bc0 msgr2=0x7fb01c1db7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:07.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.012+0000 7fb01ad9d640 1 --2- 172.21.15.161:0/412519008 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb008005bc0 0x7fb01c1db7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.013+0000 7fb01ad9d640 1 -- 172.21.15.161:0/412519008 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb01c14ca30 con 0x7fb01c0abb20 2024-03-28T23:35:07.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.013+0000 7fb013fff640 1 --2- 172.21.15.161:0/412519008 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb01c0ab6a0 0x7fb01c1e6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:07.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.013+0000 7fb01ad9d640 1 --2- 172.21.15.161:0/412519008 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb01c0abb20 0x7fb01c1e71f0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fb014007d10 tx=0x7fb0140115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:07.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.013+0000 7fb012ffd640 1 -- 172.21.15.161:0/412519008 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb014011e40 con 0x7fb01c0abb20 2024-03-28T23:35:07.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.013+0000 7fb012ffd640 1 -- 172.21.15.161:0/412519008 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb01401b070 con 0x7fb01c0abb20 2024-03-28T23:35:07.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.013+0000 7fb020f02640 1 -- 172.21.15.161:0/412519008 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb01c1dde80 con 0x7fb01c0abb20 2024-03-28T23:35:07.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.013+0000 7fb012ffd640 1 -- 172.21.15.161:0/412519008 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb014003990 con 0x7fb01c0abb20 2024-03-28T23:35:07.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.013+0000 7fb020f02640 1 -- 172.21.15.161:0/412519008 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb01c1e7980 con 0x7fb01c0abb20 2024-03-28T23:35:07.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.014+0000 7fb020f02640 1 -- 172.21.15.161:0/412519008 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb01c04f4b0 con 0x7fb01c0abb20 2024-03-28T23:35:07.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.014+0000 7fb012ffd640 1 -- 172.21.15.161:0/412519008 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb014005d10 con 0x7fb01c0abb20 2024-03-28T23:35:07.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.015+0000 7fb012ffd640 1 --2- 172.21.15.161:0/412519008 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faffc06b4d0 0x7faffc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:07.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.015+0000 7fb012ffd640 1 -- 172.21.15.161:0/412519008 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5507+0+0 (secure 0 0 0) 0x7fb01408a660 con 0x7fb01c0abb20 2024-03-28T23:35:07.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.015+0000 7fb01a59c640 1 --2- 172.21.15.161:0/412519008 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faffc06b4d0 0x7faffc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:07.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.015+0000 7fb01a59c640 1 --2- 172.21.15.161:0/412519008 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faffc06b4d0 0x7faffc06d990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fb00c036000 tx=0x7fb00c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:07.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.019+0000 7fb012ffd640 1 -- 172.21.15.161:0/412519008 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb014025020 con 0x7fb01c0abb20 2024-03-28T23:35:07.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.188+0000 7fb020f02640 1 -- 172.21.15.161:0/412519008 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb01c0c2f80 con 0x7fb01c0abb20 2024-03-28T23:35:07.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.189+0000 7fb012ffd640 1 -- 172.21.15.161:0/412519008 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v127) ==== 93+0+10 (secure 0 0 0) 0x7fb014014350 con 0x7fb01c0abb20 2024-03-28T23:35:07.189 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:35:07.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.190+0000 7fb020f02640 1 -- 172.21.15.161:0/412519008 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faffc06b4d0 msgr2=0x7faffc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:07.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.190+0000 7fb020f02640 1 --2- 172.21.15.161:0/412519008 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faffc06b4d0 0x7faffc06d990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fb00c036000 tx=0x7fb00c030000 comp rx=0 tx=0).stop 2024-03-28T23:35:07.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.190+0000 7fb020f02640 1 -- 172.21.15.161:0/412519008 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb01c0abb20 msgr2=0x7fb01c1e71f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:07.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.190+0000 7fb020f02640 1 --2- 172.21.15.161:0/412519008 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb01c0abb20 0x7fb01c1e71f0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fb014007d10 tx=0x7fb0140115a0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.191+0000 7fb020f02640 1 -- 172.21.15.161:0/412519008 shutdown_connections 2024-03-28T23:35:07.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.191+0000 7fb020f02640 1 --2- 172.21.15.161:0/412519008 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faffc06b4d0 0x7faffc06d990 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.191+0000 7fb020f02640 1 --2- 172.21.15.161:0/412519008 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb01c0abb20 0x7fb01c1e71f0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.191+0000 7fb020f02640 1 --2- 172.21.15.161:0/412519008 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb01c0ab6a0 0x7fb01c1e6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.191+0000 7fb020f02640 1 --2- 172.21.15.161:0/412519008 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb008005bc0 0x7fb01c1db7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.191+0000 7fb020f02640 1 -- 172.21.15.161:0/412519008 >> 172.21.15.161:0/412519008 conn(0x7fb01c0c3390 msgr2=0x7fb01c0c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:07.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.191+0000 7fb020f02640 1 -- 172.21.15.161:0/412519008 shutdown_connections 2024-03-28T23:35:07.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.191+0000 7fb020f02640 1 -- 172.21.15.161:0/412519008 wait complete. 2024-03-28T23:35:07.202 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset' 2024-03-28T23:35:07.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.303+0000 7f905ad48640 1 Processor -- start 2024-03-28T23:35:07.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.303+0000 7f905ad48640 1 -- start start 2024-03-28T23:35:07.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.304+0000 7f905ad48640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f905414eca0 0x7f90540c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:07.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.304+0000 7f905ad48640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f90540c1dc0 0x7f905406ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:07.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.304+0000 7f905ad48640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9054151850 con 0x7f905406cf60 2024-03-28T23:35:07.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.304+0000 7f905ad48640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f9054152460 con 0x7f905414eca0 2024-03-28T23:35:07.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.304+0000 7f905ad48640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f9054151b30 con 0x7f90540c1dc0 2024-03-28T23:35:07.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.304+0000 7f9058ae4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f905414eca0 0x7f90540c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:07.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.304+0000 7f9058ae4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f905414eca0 0x7f90540c1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57598/0 (socket says 172.21.15.161:57598) 2024-03-28T23:35:07.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.304+0000 7f9058ae4640 1 -- 172.21.15.161:0/499577722 learned_addr learned my addr 172.21.15.161:0/499577722 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:07.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.304+0000 7f9058ae4640 1 -- 172.21.15.161:0/499577722 >> v2:172.21.15.170:3300/0 conn(0x7f90540c1dc0 msgr2=0x7f905406ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:07.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.304+0000 7f9058ae4640 1 --2- 172.21.15.161:0/499577722 >> v2:172.21.15.170:3300/0 conn(0x7f90540c1dc0 0x7f905406ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.304+0000 7f9058ae4640 1 -- 172.21.15.161:0/499577722 >> v1:172.21.15.170:6789/0 conn(0x7f905406cf60 legacy=0x7f90540a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:07.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.304+0000 7f9058ae4640 1 -- 172.21.15.161:0/499577722 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9054151e10 con 0x7f905414eca0 2024-03-28T23:35:07.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.305+0000 7f9058ae4640 1 --2- 172.21.15.161:0/499577722 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f905414eca0 0x7f90540c1880 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f90540c3070 tx=0x7f904402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5895e2ca4e628f80 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:07.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.305+0000 7f90537fe640 1 -- 172.21.15.161:0/499577722 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904402e070 con 0x7f905414eca0 2024-03-28T23:35:07.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.305+0000 7f90537fe640 1 -- 172.21.15.161:0/499577722 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9044037070 con 0x7f905414eca0 2024-03-28T23:35:07.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.305+0000 7f90537fe640 1 -- 172.21.15.161:0/499577722 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9044031a00 con 0x7f905414eca0 2024-03-28T23:35:07.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.306+0000 7f905ad48640 1 -- 172.21.15.161:0/499577722 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f905414eca0 msgr2=0x7f90540c1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:07.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.306+0000 7f905ad48640 1 --2- 172.21.15.161:0/499577722 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f905414eca0 0x7f90540c1880 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f90540c3070 tx=0x7f904402b880 comp rx=0 tx=0).stop 2024-03-28T23:35:07.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.306+0000 7f905ad48640 1 -- 172.21.15.161:0/499577722 shutdown_connections 2024-03-28T23:35:07.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.306+0000 7f905ad48640 1 --2- 172.21.15.161:0/499577722 >> v2:172.21.15.170:3300/0 conn(0x7f90540c1dc0 0x7f905406ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.306+0000 7f905ad48640 1 --2- 172.21.15.161:0/499577722 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f905414eca0 0x7f90540c1880 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.306+0000 7f905ad48640 1 -- 172.21.15.161:0/499577722 >> 172.21.15.161:0/499577722 conn(0x7f90540c6670 msgr2=0x7f90540c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:07.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.306+0000 7f905ad48640 1 -- 172.21.15.161:0/499577722 shutdown_connections 2024-03-28T23:35:07.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.306+0000 7f905ad48640 1 -- 172.21.15.161:0/499577722 wait complete. 2024-03-28T23:35:07.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.307+0000 7f905ad48640 1 Processor -- start 2024-03-28T23:35:07.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.307+0000 7f905ad48640 1 -- start start 2024-03-28T23:35:07.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.308+0000 7f905ad48640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f905406cf60 0x7f90540c1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:07.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.308+0000 7f905ad48640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90540c1dc0 0x7f90540c0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:07.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.308+0000 7f905ad48640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f905414eca0 0x7f90540c0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:07.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.308+0000 7f905ad48640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f90541514b0 con 0x7f905406cf60 2024-03-28T23:35:07.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.308+0000 7f905ad48640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f905414f080 con 0x7f90540c1dc0 2024-03-28T23:35:07.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.308+0000 7f905ad48640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9054151630 con 0x7f905414eca0 2024-03-28T23:35:07.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.308+0000 7f9058ae4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f905406cf60 0x7f90540c1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:07.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.308+0000 7f9058ae4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f905406cf60 0x7f90540c1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57604/0 (socket says 172.21.15.161:57604) 2024-03-28T23:35:07.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.308+0000 7f9058ae4640 1 -- 172.21.15.161:0/3801908394 learned_addr learned my addr 172.21.15.161:0/3801908394 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:07.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.308+0000 7f9053fff640 1 --2- 172.21.15.161:0/3801908394 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90540c1dc0 0x7f90540c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:07.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.308+0000 7f9058ae4640 1 -- 172.21.15.161:0/3801908394 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f905414eca0 msgr2=0x7f90540c0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:07.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.308+0000 7f90592e5640 1 --2- 172.21.15.161:0/3801908394 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f905414eca0 0x7f90540c0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:07.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.308+0000 7f9058ae4640 1 --2- 172.21.15.161:0/3801908394 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f905414eca0 0x7f90540c0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.308+0000 7f9058ae4640 1 -- 172.21.15.161:0/3801908394 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90540c1dc0 msgr2=0x7f90540c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:07.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.308+0000 7f9058ae4640 1 --2- 172.21.15.161:0/3801908394 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90540c1dc0 0x7f90540c0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.308+0000 7f9058ae4640 1 -- 172.21.15.161:0/3801908394 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90541ebdb0 con 0x7f905406cf60 2024-03-28T23:35:07.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.308+0000 7f9053fff640 1 --2- 172.21.15.161:0/3801908394 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90540c1dc0 0x7f90540c0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:07.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.308+0000 7f9058ae4640 1 --2- 172.21.15.161:0/3801908394 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f905406cf60 0x7f90540c1160 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f9044000c00 tx=0x7f90440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:07.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.309+0000 7f90517fa640 1 -- 172.21.15.161:0/3801908394 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9044032880 con 0x7f905406cf60 2024-03-28T23:35:07.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.309+0000 7f90517fa640 1 -- 172.21.15.161:0/3801908394 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9044032a20 con 0x7f905406cf60 2024-03-28T23:35:07.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.309+0000 7f905ad48640 1 -- 172.21.15.161:0/3801908394 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90541ec090 con 0x7f905406cf60 2024-03-28T23:35:07.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.309+0000 7f90517fa640 1 -- 172.21.15.161:0/3801908394 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9044032e10 con 0x7f905406cf60 2024-03-28T23:35:07.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.309+0000 7f905ad48640 1 -- 172.21.15.161:0/3801908394 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90541de050 con 0x7f905406cf60 2024-03-28T23:35:07.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.310+0000 7f905ad48640 1 -- 172.21.15.161:0/3801908394 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f905404f4b0 con 0x7f905406cf60 2024-03-28T23:35:07.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.310+0000 7f90517fa640 1 -- 172.21.15.161:0/3801908394 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f90440453f0 con 0x7f905406cf60 2024-03-28T23:35:07.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.310+0000 7f90517fa640 1 --2- 172.21.15.161:0/3801908394 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f904006b500 0x7f904006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:07.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.310+0000 7f90517fa640 1 -- 172.21.15.161:0/3801908394 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5507+0+0 (secure 0 0 0) 0x7f90440ac110 con 0x7f905406cf60 2024-03-28T23:35:07.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.310+0000 7f9053fff640 1 --2- 172.21.15.161:0/3801908394 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f904006b500 0x7f904006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:07.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.311+0000 7f9053fff640 1 --2- 172.21.15.161:0/3801908394 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f904006b500 0x7f904006d9c0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f9048006fd0 tx=0x7f9048008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:07.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.315+0000 7f90517fa640 1 -- 172.21.15.161:0/3801908394 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9044040070 con 0x7f905406cf60 2024-03-28T23:35:07.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.554+0000 7f905ad48640 1 -- 172.21.15.161:0/3801908394 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset"]} v 0) -- 0x7f90541de790 con 0x7f905406cf60 2024-03-28T23:35:07.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.593+0000 7f90517fa640 1 -- 172.21.15.161:0/3801908394 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset"]}]=0 v311) ==== 147+0+0 (secure 0 0 0) 0x7f904407b0e0 con 0x7f905406cf60 2024-03-28T23:35:07.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.595+0000 7f905ad48640 1 -- 172.21.15.161:0/3801908394 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f904006b500 msgr2=0x7f904006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:07.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.595+0000 7f905ad48640 1 --2- 172.21.15.161:0/3801908394 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f904006b500 0x7f904006d9c0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f9048006fd0 tx=0x7f9048008040 comp rx=0 tx=0).stop 2024-03-28T23:35:07.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.596+0000 7f905ad48640 1 -- 172.21.15.161:0/3801908394 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f905406cf60 msgr2=0x7f90540c1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:07.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.596+0000 7f905ad48640 1 --2- 172.21.15.161:0/3801908394 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f905406cf60 0x7f90540c1160 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f9044000c00 tx=0x7f90440046c0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.596+0000 7f905ad48640 1 -- 172.21.15.161:0/3801908394 shutdown_connections 2024-03-28T23:35:07.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.596+0000 7f905ad48640 1 --2- 172.21.15.161:0/3801908394 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f904006b500 0x7f904006d9c0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.596+0000 7f905ad48640 1 --2- 172.21.15.161:0/3801908394 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f905414eca0 0x7f90540c0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.596+0000 7f905ad48640 1 --2- 172.21.15.161:0/3801908394 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90540c1dc0 0x7f90540c0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.596+0000 7f905ad48640 1 --2- 172.21.15.161:0/3801908394 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f905406cf60 0x7f90540c1160 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.596+0000 7f905ad48640 1 -- 172.21.15.161:0/3801908394 >> 172.21.15.161:0/3801908394 conn(0x7f90540c6670 msgr2=0x7f90540aa0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:07.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.596+0000 7f905ad48640 1 -- 172.21.15.161:0/3801908394 shutdown_connections 2024-03-28T23:35:07.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.596+0000 7f905ad48640 1 -- 172.21.15.161:0/3801908394 wait complete. 2024-03-28T23:35:07.607 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-28T23:35:07.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.710+0000 7facbbe84640 1 Processor -- start 2024-03-28T23:35:07.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.710+0000 7facbbe84640 1 -- start start 2024-03-28T23:35:07.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.711+0000 7facbbe84640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facb40c1040 0x7facb41535b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:07.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.711+0000 7facbbe84640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7facb40a96f0 0x7facb415d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:07.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.711+0000 7facbbe84640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facb40c2f50 con 0x7facb4150520 2024-03-28T23:35:07.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.711+0000 7facbbe84640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7facb40a9c20 con 0x7facb40a96f0 2024-03-28T23:35:07.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.711+0000 7facbbe84640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7facb4150900 con 0x7facb40c1040 2024-03-28T23:35:07.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.711+0000 7facba421640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7facb40a96f0 0x7facb415d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:07.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.711+0000 7facba421640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7facb40a96f0 0x7facb415d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57610/0 (socket says 172.21.15.161:57610) 2024-03-28T23:35:07.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.711+0000 7facba421640 1 -- 172.21.15.161:0/3124100814 learned_addr learned my addr 172.21.15.161:0/3124100814 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:07.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.712+0000 7facba421640 1 -- 172.21.15.161:0/3124100814 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facb40c1040 msgr2=0x7facb41535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:07.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.712+0000 7facba421640 1 --2- 172.21.15.161:0/3124100814 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facb40c1040 0x7facb41535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.712+0000 7facba421640 1 -- 172.21.15.161:0/3124100814 >> v1:172.21.15.170:6789/0 conn(0x7facb4150520 legacy=0x7facb40ab280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:07.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.712+0000 7facba421640 1 -- 172.21.15.161:0/3124100814 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facb41530d0 con 0x7facb40a96f0 2024-03-28T23:35:07.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.712+0000 7facba421640 1 --2- 172.21.15.161:0/3124100814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7facb40a96f0 0x7facb415d8b0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7facac009870 tx=0x7facac02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b68957410b745f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:07.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.712+0000 7facb8c1e640 1 -- 172.21.15.161:0/3124100814 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facac0317c0 con 0x7facb40a96f0 2024-03-28T23:35:07.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.712+0000 7facb8c1e640 1 -- 172.21.15.161:0/3124100814 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facac037070 con 0x7facb40a96f0 2024-03-28T23:35:07.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.712+0000 7facb8c1e640 1 -- 172.21.15.161:0/3124100814 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facac031b80 con 0x7facb40a96f0 2024-03-28T23:35:07.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.713+0000 7facbbe84640 1 -- 172.21.15.161:0/3124100814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7facb40a96f0 msgr2=0x7facb415d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:07.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.713+0000 7facbbe84640 1 --2- 172.21.15.161:0/3124100814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7facb40a96f0 0x7facb415d8b0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7facac009870 tx=0x7facac02b860 comp rx=0 tx=0).stop 2024-03-28T23:35:07.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.713+0000 7facbbe84640 1 -- 172.21.15.161:0/3124100814 shutdown_connections 2024-03-28T23:35:07.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.713+0000 7facbbe84640 1 --2- 172.21.15.161:0/3124100814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7facb40a96f0 0x7facb415d8b0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.713+0000 7facbbe84640 1 --2- 172.21.15.161:0/3124100814 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facb40c1040 0x7facb41535b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.713+0000 7facbbe84640 1 -- 172.21.15.161:0/3124100814 >> 172.21.15.161:0/3124100814 conn(0x7facb40a83f0 msgr2=0x7facb40a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:07.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.713+0000 7facbbe84640 1 -- 172.21.15.161:0/3124100814 shutdown_connections 2024-03-28T23:35:07.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.713+0000 7facbbe84640 1 -- 172.21.15.161:0/3124100814 wait complete. 2024-03-28T23:35:07.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.714+0000 7facbbe84640 1 Processor -- start 2024-03-28T23:35:07.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.714+0000 7facbbe84640 1 -- start start 2024-03-28T23:35:07.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.715+0000 7facbbe84640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facb40a96f0 0x7facb41eba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:07.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.715+0000 7facbbe84640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facb40c1040 0x7facb40bfce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:07.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.715+0000 7facbbe84640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7facb4150520 0x7facb41ec2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:07.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.715+0000 7facbbe84640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7facb40ab990 con 0x7facb4150520 2024-03-28T23:35:07.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.715+0000 7facbbe84640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7facb40c2c70 con 0x7facb40a96f0 2024-03-28T23:35:07.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.715+0000 7facbbe84640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7facb40abb10 con 0x7facb40c1040 2024-03-28T23:35:07.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.715+0000 7facba421640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7facb4150520 0x7facb41ec2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:07.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.715+0000 7facba421640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7facb4150520 0x7facb41ec2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57620/0 (socket says 172.21.15.161:57620) 2024-03-28T23:35:07.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.715+0000 7facba421640 1 -- 172.21.15.161:0/3481124527 learned_addr learned my addr 172.21.15.161:0/3481124527 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:07.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.715+0000 7facb9c20640 1 --2- 172.21.15.161:0/3481124527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facb40a96f0 0x7facb41eba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:07.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.715+0000 7facb941f640 1 --2- 172.21.15.161:0/3481124527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facb40c1040 0x7facb40bfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:07.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.715+0000 7facba421640 1 -- 172.21.15.161:0/3481124527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facb40c1040 msgr2=0x7facb40bfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:07.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.715+0000 7facba421640 1 --2- 172.21.15.161:0/3481124527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facb40c1040 0x7facb40bfce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.715+0000 7facba421640 1 -- 172.21.15.161:0/3481124527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facb40a96f0 msgr2=0x7facb41eba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:07.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.715+0000 7facba421640 1 --2- 172.21.15.161:0/3481124527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facb40a96f0 0x7facb41eba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.715+0000 7facba421640 1 -- 172.21.15.161:0/3481124527 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facb41eef00 con 0x7facb4150520 2024-03-28T23:35:07.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.716+0000 7facba421640 1 --2- 172.21.15.161:0/3481124527 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7facb4150520 0x7facb41ec2c0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7facac009870 tx=0x7facac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:07.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.716+0000 7facaa7fc640 1 -- 172.21.15.161:0/3481124527 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facac032650 con 0x7facb4150520 2024-03-28T23:35:07.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.716+0000 7facaa7fc640 1 -- 172.21.15.161:0/3481124527 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facac032810 con 0x7facb4150520 2024-03-28T23:35:07.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.716+0000 7facaa7fc640 1 -- 172.21.15.161:0/3481124527 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facac032c00 con 0x7facb4150520 2024-03-28T23:35:07.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.716+0000 7facbbe84640 1 -- 172.21.15.161:0/3481124527 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facb41eaf60 con 0x7facb4150520 2024-03-28T23:35:07.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.716+0000 7facbbe84640 1 -- 172.21.15.161:0/3481124527 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facb40c0220 con 0x7facb4150520 2024-03-28T23:35:07.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.717+0000 7facbbe84640 1 -- 172.21.15.161:0/3481124527 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facb404f4b0 con 0x7facb4150520 2024-03-28T23:35:07.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.717+0000 7facaa7fc640 1 -- 172.21.15.161:0/3481124527 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7facac0453f0 con 0x7facb4150520 2024-03-28T23:35:07.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.717+0000 7facaa7fc640 1 --2- 172.21.15.161:0/3481124527 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fac9806b4d0 0x7fac9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:07.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.718+0000 7facaa7fc640 1 -- 172.21.15.161:0/3481124527 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5507+0+0 (secure 0 0 0) 0x7facac0ac080 con 0x7facb4150520 2024-03-28T23:35:07.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.718+0000 7facb9c20640 1 --2- 172.21.15.161:0/3481124527 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fac9806b4d0 0x7fac9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:07.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.718+0000 7facb9c20640 1 --2- 172.21.15.161:0/3481124527 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fac9806b4d0 0x7fac9806d990 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7facb000b450 tx=0x7facb000d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:07.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.722+0000 7facaa7fc640 1 -- 172.21.15.161:0/3481124527 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7facac040070 con 0x7facb4150520 2024-03-28T23:35:07.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.895+0000 7facbbe84640 1 -- 172.21.15.161:0/3481124527 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7facb40c2640 con 0x7facb4150520 2024-03-28T23:35:07.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.898+0000 7facaa7fc640 1 -- 172.21.15.161:0/3481124527 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v45)=0 v45) ==== 122+0+0 (secure 0 0 0) 0x7facac07b050 con 0x7facb4150520 2024-03-28T23:35:07.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.900+0000 7facbbe84640 1 -- 172.21.15.161:0/3481124527 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fac9806b4d0 msgr2=0x7fac9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:07.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.900+0000 7facbbe84640 1 --2- 172.21.15.161:0/3481124527 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fac9806b4d0 0x7fac9806d990 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7facb000b450 tx=0x7facb000d040 comp rx=0 tx=0).stop 2024-03-28T23:35:07.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.900+0000 7facbbe84640 1 -- 172.21.15.161:0/3481124527 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7facb4150520 msgr2=0x7facb41ec2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:07.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.900+0000 7facbbe84640 1 --2- 172.21.15.161:0/3481124527 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7facb4150520 0x7facb41ec2c0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7facac009870 tx=0x7facac0046c0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.900+0000 7facbbe84640 1 -- 172.21.15.161:0/3481124527 shutdown_connections 2024-03-28T23:35:07.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.900+0000 7facbbe84640 1 --2- 172.21.15.161:0/3481124527 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fac9806b4d0 0x7fac9806d990 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.900+0000 7facbbe84640 1 --2- 172.21.15.161:0/3481124527 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7facb4150520 0x7facb41ec2c0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.900+0000 7facbbe84640 1 --2- 172.21.15.161:0/3481124527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facb40c1040 0x7facb40bfce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.900+0000 7facbbe84640 1 --2- 172.21.15.161:0/3481124527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facb40a96f0 0x7facb41eba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:07.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.900+0000 7facbbe84640 1 -- 172.21.15.161:0/3481124527 >> 172.21.15.161:0/3481124527 conn(0x7facb40a83f0 msgr2=0x7facb40c1490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:07.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.900+0000 7facbbe84640 1 -- 172.21.15.161:0/3481124527 shutdown_connections 2024-03-28T23:35:07.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:07.900+0000 7facbbe84640 1 -- 172.21.15.161:0/3481124527 wait complete. 2024-03-28T23:35:07.911 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-28T23:35:07.912 DEBUG:teuthology.orchestra.run.smithi161:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:35:07.966 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:35:08.021 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-28T23:35:08.021 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:35:08.021 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:35:08.036 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20793 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-28T23:35:08.036 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:35:08.036 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:35:08.054 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056374,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19690 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-28T23:35:08.054 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-28T23:35:08.054 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:35:08.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.176+0000 7f5fc2664640 1 Processor -- start 2024-03-28T23:35:08.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.176+0000 7f5fc2664640 1 -- start start 2024-03-28T23:35:08.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.177+0000 7f5fc2664640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5fbc150600 0x7f5fbc0a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:08.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.177+0000 7f5fc2664640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5fbc0a8fc0 0x7f5fbc153270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:08.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.177+0000 7f5fc2664640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5fbc0aacb0 0x7f5fbc1537b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:08.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.177+0000 7f5fc2664640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5fbc156570 con 0x7f5fbc0aacb0 2024-03-28T23:35:08.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.177+0000 7f5fc2664640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f5fbc1567c0 con 0x7f5fbc0a8fc0 2024-03-28T23:35:08.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.177+0000 7f5fc2664640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5fbc156aa0 con 0x7f5fbc150600 2024-03-28T23:35:08.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.177+0000 7f5fbb7fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5fbc0a8fc0 0x7f5fbc153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:08.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.177+0000 7f5fbbfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5fbc150600 0x7f5fbc0a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:08.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.177+0000 7f5fbb7fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5fbc0a8fc0 0x7f5fbc153270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:51916/0 (socket says 172.21.15.161:51916) 2024-03-28T23:35:08.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.177+0000 7f5fbb7fe640 1 -- 172.21.15.161:0/1201428751 learned_addr learned my addr 172.21.15.161:0/1201428751 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:08.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.177+0000 7f5fc0c01640 1 --2- 172.21.15.161:0/1201428751 >> v2:172.21.15.161:3300/0 conn(0x7f5fbc0aacb0 0x7f5fbc1537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:08.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.177+0000 7f5fc0c01640 1 -- 172.21.15.161:0/1201428751 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5fbc150600 msgr2=0x7f5fbc0a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:08.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.177+0000 7f5fc0c01640 1 --2- 172.21.15.161:0/1201428751 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5fbc150600 0x7f5fbc0a9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:08.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.178+0000 7f5fc0c01640 1 -- 172.21.15.161:0/1201428751 >> v2:172.21.15.170:3300/0 conn(0x7f5fbc0a8fc0 msgr2=0x7f5fbc153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:08.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.178+0000 7f5fc0c01640 1 --2- 172.21.15.161:0/1201428751 >> v2:172.21.15.170:3300/0 conn(0x7f5fbc0a8fc0 0x7f5fbc153270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:08.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.178+0000 7f5fc0c01640 1 -- 172.21.15.161:0/1201428751 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fbc156d80 con 0x7f5fbc0aacb0 2024-03-28T23:35:08.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.178+0000 7f5fc0c01640 1 --2- 172.21.15.161:0/1201428751 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5fbc0aacb0 0x7f5fbc1537b0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc156cb0 tx=0x7f5fb402bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fddc6c42ad216ae7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:08.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.178+0000 7f5fbaffd640 1 -- 172.21.15.161:0/1201428751 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb4039070 con 0x7f5fbc0aacb0 2024-03-28T23:35:08.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.178+0000 7f5fbaffd640 1 -- 172.21.15.161:0/1201428751 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fb402cb30 con 0x7f5fbc0aacb0 2024-03-28T23:35:08.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.178+0000 7f5fbaffd640 1 -- 172.21.15.161:0/1201428751 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb4034440 con 0x7f5fbc0aacb0 2024-03-28T23:35:08.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.179+0000 7f5fc2664640 1 -- 172.21.15.161:0/1201428751 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5fbc0aacb0 msgr2=0x7f5fbc1537b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:08.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.179+0000 7f5fc2664640 1 --2- 172.21.15.161:0/1201428751 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5fbc0aacb0 0x7f5fbc1537b0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc156cb0 tx=0x7f5fb402bd80 comp rx=0 tx=0).stop 2024-03-28T23:35:08.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.179+0000 7f5fc2664640 1 -- 172.21.15.161:0/1201428751 shutdown_connections 2024-03-28T23:35:08.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.179+0000 7f5fc2664640 1 --2- 172.21.15.161:0/1201428751 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5fbc0aacb0 0x7f5fbc1537b0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:08.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.179+0000 7f5fc2664640 1 --2- 172.21.15.161:0/1201428751 >> v2:172.21.15.170:3300/0 conn(0x7f5fbc0a8fc0 0x7f5fbc153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:08.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.179+0000 7f5fc2664640 1 --2- 172.21.15.161:0/1201428751 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5fbc150600 0x7f5fbc0a9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:08.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.179+0000 7f5fc2664640 1 -- 172.21.15.161:0/1201428751 >> 172.21.15.161:0/1201428751 conn(0x7f5fbc0c6670 msgr2=0x7f5fbc0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:08.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.179+0000 7f5fc2664640 1 -- 172.21.15.161:0/1201428751 shutdown_connections 2024-03-28T23:35:08.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.179+0000 7f5fc2664640 1 -- 172.21.15.161:0/1201428751 wait complete. 2024-03-28T23:35:08.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.180+0000 7f5fc2664640 1 Processor -- start 2024-03-28T23:35:08.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.181+0000 7f5fc2664640 1 -- start start 2024-03-28T23:35:08.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.181+0000 7f5fc2664640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5fbc0a8fc0 0x7f5fbc1e9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:08.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.181+0000 7f5fc2664640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5fbc0aacb0 0x7f5fbc0c38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:08.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.181+0000 7f5fc2664640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5fbc150600 0x7f5fbc1de9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:08.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.181+0000 7f5fc2664640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5fbc1509e0 con 0x7f5fbc0aacb0 2024-03-28T23:35:08.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.181+0000 7f5fc2664640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5fbc157340 con 0x7f5fbc150600 2024-03-28T23:35:08.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.181+0000 7f5fc2664640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5fbc0aa9e0 con 0x7f5fbc0a8fc0 2024-03-28T23:35:08.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.181+0000 7f5fbb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5fbc0aacb0 0x7f5fbc0c38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:08.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.181+0000 7f5fbb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5fbc0aacb0 0x7f5fbc0c38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57644/0 (socket says 172.21.15.161:57644) 2024-03-28T23:35:08.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.181+0000 7f5fbb7fe640 1 -- 172.21.15.161:0/1440665529 learned_addr learned my addr 172.21.15.161:0/1440665529 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:08.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.181+0000 7f5fbbfff640 1 --2- 172.21.15.161:0/1440665529 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5fbc0a8fc0 0x7f5fbc1e9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:08.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.181+0000 7f5fc0c01640 1 --2- 172.21.15.161:0/1440665529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5fbc150600 0x7f5fbc1de9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:08.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.182+0000 7f5fbb7fe640 1 -- 172.21.15.161:0/1440665529 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5fbc0a8fc0 msgr2=0x7f5fbc1e9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:08.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.182+0000 7f5fbb7fe640 1 --2- 172.21.15.161:0/1440665529 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5fbc0a8fc0 0x7f5fbc1e9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:08.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.182+0000 7f5fbb7fe640 1 -- 172.21.15.161:0/1440665529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5fbc150600 msgr2=0x7f5fbc1de9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:08.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.182+0000 7f5fbb7fe640 1 --2- 172.21.15.161:0/1440665529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5fbc150600 0x7f5fbc1de9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:08.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.182+0000 7f5fbb7fe640 1 -- 172.21.15.161:0/1440665529 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fbc1e8c40 con 0x7f5fbc0aacb0 2024-03-28T23:35:08.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.182+0000 7f5fbb7fe640 1 --2- 172.21.15.161:0/1440665529 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5fbc0aacb0 0x7f5fbc0c38d0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f5fb000e900 tx=0x7f5fb000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:08.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.182+0000 7f5fb8ff9640 1 -- 172.21.15.161:0/1440665529 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb0017070 con 0x7f5fbc0aacb0 2024-03-28T23:35:08.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.182+0000 7f5fb8ff9640 1 -- 172.21.15.161:0/1440665529 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fb000ca00 con 0x7f5fbc0aacb0 2024-03-28T23:35:08.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.182+0000 7f5fb8ff9640 1 -- 172.21.15.161:0/1440665529 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb000cdd0 con 0x7f5fbc0aacb0 2024-03-28T23:35:08.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.182+0000 7f5fc2664640 1 -- 172.21.15.161:0/1440665529 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fbc1dcfc0 con 0x7f5fbc0aacb0 2024-03-28T23:35:08.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.182+0000 7f5fc2664640 1 -- 172.21.15.161:0/1440665529 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fbc1e8f00 con 0x7f5fbc0aacb0 2024-03-28T23:35:08.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.183+0000 7f5fc2664640 1 -- 172.21.15.161:0/1440665529 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fbc04f4b0 con 0x7f5fbc0aacb0 2024-03-28T23:35:08.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.183+0000 7f5fb8ff9640 1 -- 172.21.15.161:0/1440665529 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5fb001a070 con 0x7f5fbc0aacb0 2024-03-28T23:35:08.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.184+0000 7f5fb8ff9640 1 --2- 172.21.15.161:0/1440665529 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5fa806b4d0 0x7f5fa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:08.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.184+0000 7f5fbbfff640 1 --2- 172.21.15.161:0/1440665529 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5fa806b4d0 0x7f5fa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:08.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.184+0000 7f5fb8ff9640 1 -- 172.21.15.161:0/1440665529 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5507+0+0 (secure 0 0 0) 0x7f5fb008a0b0 con 0x7f5fbc0aacb0 2024-03-28T23:35:08.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.184+0000 7f5fbbfff640 1 --2- 172.21.15.161:0/1440665529 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5fa806b4d0 0x7f5fa806d990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f5fac009a20 tx=0x7f5fac009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:08.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.188+0000 7f5fb8ff9640 1 -- 172.21.15.161:0/1440665529 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5fb00589b0 con 0x7f5fbc0aacb0 2024-03-28T23:35:08.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.394+0000 7f5fc2664640 1 -- 172.21.15.161:0/1440665529 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5fbc153dc0 con 0x7f5fbc0aacb0 2024-03-28T23:35:08.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.395+0000 7f5fb8ff9640 1 -- 172.21.15.161:0/1440665529 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+3883 (secure 0 0 0) 0x7f5fb005ca50 con 0x7f5fbc0aacb0 2024-03-28T23:35:08.395 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:08.396 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":119,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7318,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3734750343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3734750343},{"type":"v1","addr":"172.21.15.170:6835","nonce":3734750343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/4140438862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":4140438862},{"type":"v1","addr":"172.21.15.170:6834","nonce":4140438862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103},{"gid":8220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/896089063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":896089063},{"type":"v1","addr":"172.21.15.161:6835","nonce":896089063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":115},{"gid":8337,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/2735584888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2735584888},{"type":"v1","addr":"172.21.15.161:6837","nonce":2735584888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119}],"filesystems":[]} 2024-03-28T23:35:08.396 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 119 2024-03-28T23:35:08.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.397+0000 7f5fc2664640 1 -- 172.21.15.161:0/1440665529 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5fa806b4d0 msgr2=0x7f5fa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:08.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.397+0000 7f5fc2664640 1 --2- 172.21.15.161:0/1440665529 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5fa806b4d0 0x7f5fa806d990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f5fac009a20 tx=0x7f5fac009190 comp rx=0 tx=0).stop 2024-03-28T23:35:08.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.397+0000 7f5fc2664640 1 -- 172.21.15.161:0/1440665529 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5fbc0aacb0 msgr2=0x7f5fbc0c38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:08.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.397+0000 7f5fc2664640 1 --2- 172.21.15.161:0/1440665529 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5fbc0aacb0 0x7f5fbc0c38d0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f5fb000e900 tx=0x7f5fb000edc0 comp rx=0 tx=0).stop 2024-03-28T23:35:08.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.397+0000 7f5fc2664640 1 -- 172.21.15.161:0/1440665529 shutdown_connections 2024-03-28T23:35:08.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.397+0000 7f5fc2664640 1 --2- 172.21.15.161:0/1440665529 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5fa806b4d0 0x7f5fa806d990 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:08.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.397+0000 7f5fc2664640 1 --2- 172.21.15.161:0/1440665529 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5fbc0aacb0 0x7f5fbc0c38d0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:08.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.397+0000 7f5fc2664640 1 --2- 172.21.15.161:0/1440665529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5fbc150600 0x7f5fbc1de9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:08.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.397+0000 7f5fc2664640 1 --2- 172.21.15.161:0/1440665529 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5fbc0a8fc0 0x7f5fbc1e9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:08.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.397+0000 7f5fc2664640 1 -- 172.21.15.161:0/1440665529 >> 172.21.15.161:0/1440665529 conn(0x7f5fbc0c6670 msgr2=0x7f5fbc0728a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:08.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.397+0000 7f5fc2664640 1 -- 172.21.15.161:0/1440665529 shutdown_connections 2024-03-28T23:35:08.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.397+0000 7f5fc2664640 1 -- 172.21.15.161:0/1440665529 wait complete. 2024-03-28T23:35:08.409 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T23:35:08.409 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-28T23:35:08.409 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T23:35:08.409 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:35:08.409 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T23:35:08.409 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-28T23:35:08.409 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T23:35:08.410 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:35:08.410 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-28T23:35:08.410 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-28T23:35:08.410 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-28T23:35:08.410 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:35:08.410 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-28T23:35:08.410 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-28T23:35:08.410 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-28T23:35:08.411 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:35:08.418 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T23:35:08.418 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T23:35:08.419 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T23:35:08.419 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T23:35:08.421 INFO:tasks.ceph.mds.b:Stopped 2024-03-28T23:35:08.422 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-28T23:35:08.422 INFO:tasks.ceph.mds.d:Stopped 2024-03-28T23:35:08.422 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-28T23:35:08.461 INFO:tasks.ceph.mds.a:Started 2024-03-28T23:35:08.461 DEBUG:teuthology.parallel:result is None 2024-03-28T23:35:08.461 INFO:tasks.ceph.mds.c:Started 2024-03-28T23:35:08.462 DEBUG:teuthology.parallel:result is None 2024-03-28T23:35:08.465 INFO:tasks.ceph.mds.b:Started 2024-03-28T23:35:08.465 DEBUG:teuthology.parallel:result is None 2024-03-28T23:35:08.465 INFO:tasks.ceph.mds.d:Started 2024-03-28T23:35:08.466 DEBUG:teuthology.parallel:result is None 2024-03-28T23:35:08.466 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-28T23:35:08.537 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-28T23:35:08.538 INFO:tasks.ceph.mds.c.smithi161.stdout:starting mds.c at 2024-03-28T23:35:08.549 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2024-03-28T23:35:08.549 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2024-03-28T23:35:08.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.571+0000 7f65af9ae640 1 Processor -- start 2024-03-28T23:35:08.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.571+0000 7f65af9ae640 1 -- start start 2024-03-28T23:35:08.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.572+0000 7f65af9ae640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f65a80c5410 0x7f65a80c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:08.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.572+0000 7f65af9ae640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f65a81553a0 0x7f65a8155850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:08.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.572+0000 7f65af9ae640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65a8158a30 con 0x7f65a8152890 2024-03-28T23:35:08.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.572+0000 7f65af9ae640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f65a80c8480 con 0x7f65a81553a0 2024-03-28T23:35:08.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.572+0000 7f65af9ae640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f65a80c5d90 con 0x7f65a80c5410 2024-03-28T23:35:08.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.572+0000 7f65adf4b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f65a81553a0 0x7f65a8155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:08.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.572+0000 7f65ad74a640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f65a8152890 0x7f65a80c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45344/0 (socket says 172.21.15.161:45344) 2024-03-28T23:35:08.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.572+0000 7f65ad74a640 1 -- 172.21.15.161:0/3099587793 learned_addr learned my addr 172.21.15.161:0/3099587793 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:08.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.572+0000 7f65acf49640 1 --2- 172.21.15.161:0/3099587793 >> v2:172.21.15.170:3300/0 conn(0x7f65a80c5410 0x7f65a80c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:08.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.572+0000 7f65adf4b640 1 -- 172.21.15.161:0/3099587793 >> v2:172.21.15.170:3300/0 conn(0x7f65a80c5410 msgr2=0x7f65a80c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:08.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.572+0000 7f65adf4b640 1 --2- 172.21.15.161:0/3099587793 >> v2:172.21.15.170:3300/0 conn(0x7f65a80c5410 0x7f65a80c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:08.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.573+0000 7f65adf4b640 1 -- 172.21.15.161:0/3099587793 >> v1:172.21.15.161:6789/0 conn(0x7f65a8152890 legacy=0x7f65a80c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:08.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.573+0000 7f659ffff640 1 -- 172.21.15.161:0/3099587793 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2727425007 0 0) 0x7f65a8158a30 con 0x7f65a8152890 2024-03-28T23:35:08.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.573+0000 7f65adf4b640 1 -- 172.21.15.161:0/3099587793 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a8158d10 con 0x7f65a81553a0 2024-03-28T23:35:08.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.573+0000 7f65adf4b640 1 --2- 172.21.15.161:0/3099587793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65a81553a0 0x7f65a8155850 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f65a80ad6e0 tx=0x7f65a002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54cc587d145b6c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:08.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.573+0000 7f659ffff640 1 -- 172.21.15.161:0/3099587793 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a0039070 con 0x7f65a81553a0 2024-03-28T23:35:08.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.573+0000 7f659ffff640 1 -- 172.21.15.161:0/3099587793 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65a002cbf0 con 0x7f65a81553a0 2024-03-28T23:35:08.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.573+0000 7f659ffff640 1 -- 172.21.15.161:0/3099587793 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a0034440 con 0x7f65a81553a0 2024-03-28T23:35:08.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.574+0000 7f65af9ae640 1 -- 172.21.15.161:0/3099587793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65a81553a0 msgr2=0x7f65a8155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:08.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.574+0000 7f65af9ae640 1 --2- 172.21.15.161:0/3099587793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65a81553a0 0x7f65a8155850 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f65a80ad6e0 tx=0x7f65a002be40 comp rx=0 tx=0).stop 2024-03-28T23:35:08.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.574+0000 7f65af9ae640 1 -- 172.21.15.161:0/3099587793 shutdown_connections 2024-03-28T23:35:08.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.574+0000 7f65af9ae640 1 --2- 172.21.15.161:0/3099587793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65a81553a0 0x7f65a8155850 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:08.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.574+0000 7f65af9ae640 1 --2- 172.21.15.161:0/3099587793 >> v2:172.21.15.170:3300/0 conn(0x7f65a80c5410 0x7f65a80c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:08.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.574+0000 7f65af9ae640 1 -- 172.21.15.161:0/3099587793 >> 172.21.15.161:0/3099587793 conn(0x7f65a80ac620 msgr2=0x7f65a80aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:08.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.574+0000 7f65af9ae640 1 -- 172.21.15.161:0/3099587793 shutdown_connections 2024-03-28T23:35:08.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.574+0000 7f65af9ae640 1 -- 172.21.15.161:0/3099587793 wait complete. 2024-03-28T23:35:08.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.575+0000 7f65af9ae640 1 Processor -- start 2024-03-28T23:35:08.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.575+0000 7f65af9ae640 1 -- start start 2024-03-28T23:35:08.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.576+0000 7f65af9ae640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65a80c5410 0x7f65a814e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:08.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.576+0000 7f65af9ae640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65a8152890 0x7f65a814f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:08.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.576+0000 7f65af9ae640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65a81553a0 0x7f65a814f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:08.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.576+0000 7f65af9ae640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f65a81592d0 con 0x7f65a8152890 2024-03-28T23:35:08.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.576+0000 7f65af9ae640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f65a8159640 con 0x7f65a81553a0 2024-03-28T23:35:08.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.576+0000 7f65af9ae640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f65a80c90a0 con 0x7f65a80c5410 2024-03-28T23:35:08.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.576+0000 7f65acf49640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65a8152890 0x7f65a814f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:08.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.576+0000 7f65acf49640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65a8152890 0x7f65a814f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57678/0 (socket says 172.21.15.161:57678) 2024-03-28T23:35:08.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.576+0000 7f65acf49640 1 -- 172.21.15.161:0/2102677845 learned_addr learned my addr 172.21.15.161:0/2102677845 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:08.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.576+0000 7f65adf4b640 1 --2- 172.21.15.161:0/2102677845 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65a81553a0 0x7f65a814f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:08.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.576+0000 7f65ad74a640 1 --2- 172.21.15.161:0/2102677845 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65a80c5410 0x7f65a814e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:08.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.576+0000 7f65acf49640 1 -- 172.21.15.161:0/2102677845 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65a80c5410 msgr2=0x7f65a814e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:08.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.576+0000 7f65acf49640 1 --2- 172.21.15.161:0/2102677845 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65a80c5410 0x7f65a814e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:08.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.576+0000 7f65acf49640 1 -- 172.21.15.161:0/2102677845 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65a81553a0 msgr2=0x7f65a814f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:08.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.576+0000 7f65acf49640 1 --2- 172.21.15.161:0/2102677845 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65a81553a0 0x7f65a814f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:08.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.576+0000 7f65acf49640 1 -- 172.21.15.161:0/2102677845 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a814d800 con 0x7f65a8152890 2024-03-28T23:35:08.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.576+0000 7f65acf49640 1 --2- 172.21.15.161:0/2102677845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65a8152890 0x7f65a814f2d0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f659800e840 tx=0x7f659800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:08.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.577+0000 7f659dffb640 1 -- 172.21.15.161:0/2102677845 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6598017070 con 0x7f65a8152890 2024-03-28T23:35:08.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.577+0000 7f659dffb640 1 -- 172.21.15.161:0/2102677845 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f659800c940 con 0x7f65a8152890 2024-03-28T23:35:08.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.577+0000 7f65af9ae640 1 -- 172.21.15.161:0/2102677845 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65a814d520 con 0x7f65a8152890 2024-03-28T23:35:08.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.577+0000 7f659dffb640 1 -- 172.21.15.161:0/2102677845 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659800cd10 con 0x7f65a8152890 2024-03-28T23:35:08.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.577+0000 7f65af9ae640 1 -- 172.21.15.161:0/2102677845 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65a81eb560 con 0x7f65a8152890 2024-03-28T23:35:08.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.578+0000 7f65af9ae640 1 -- 172.21.15.161:0/2102677845 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65a804f4b0 con 0x7f65a8152890 2024-03-28T23:35:08.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.578+0000 7f659dffb640 1 -- 172.21.15.161:0/2102677845 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f659801a070 con 0x7f65a8152890 2024-03-28T23:35:08.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.579+0000 7f659dffb640 1 --2- 172.21.15.161:0/2102677845 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f658c06b4d0 0x7f658c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:08.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.579+0000 7f659dffb640 1 -- 172.21.15.161:0/2102677845 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5507+0+0 (secure 0 0 0) 0x7f659808a380 con 0x7f65a8152890 2024-03-28T23:35:08.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.579+0000 7f65ad74a640 1 --2- 172.21.15.161:0/2102677845 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f658c06b4d0 0x7f658c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:08.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.579+0000 7f65ad74a640 1 --2- 172.21.15.161:0/2102677845 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f658c06b4d0 0x7f658c06d990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f65a4000f60 tx=0x7f65a4007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:08.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.583+0000 7f659dffb640 1 -- 172.21.15.161:0/2102677845 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6598058c70 con 0x7f65a8152890 2024-03-28T23:35:08.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:08.770+0000 7f65af9ae640 1 -- 172.21.15.161:0/2102677845 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f65a806bff0 con 0x7f65a8152890 2024-03-28T23:35:09.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.615+0000 7f659dffb640 1 -- 172.21.15.161:0/2102677845 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v132) ==== 97+0+0 (secure 0 0 0) 0x7f659805cd10 con 0x7f65a8152890 2024-03-28T23:35:09.615 INFO:teuthology.orchestra.run.smithi161.stderr: removed all blocklist entries 2024-03-28T23:35:09.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.617+0000 7f65af9ae640 1 -- 172.21.15.161:0/2102677845 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f658c06b4d0 msgr2=0x7f658c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:09.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.617+0000 7f65af9ae640 1 --2- 172.21.15.161:0/2102677845 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f658c06b4d0 0x7f658c06d990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f65a4000f60 tx=0x7f65a4007d50 comp rx=0 tx=0).stop 2024-03-28T23:35:09.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.617+0000 7f65af9ae640 1 -- 172.21.15.161:0/2102677845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65a8152890 msgr2=0x7f65a814f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:09.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.617+0000 7f65af9ae640 1 --2- 172.21.15.161:0/2102677845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65a8152890 0x7f65a814f2d0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f659800e840 tx=0x7f659800ed00 comp rx=0 tx=0).stop 2024-03-28T23:35:09.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.617+0000 7f65af9ae640 1 -- 172.21.15.161:0/2102677845 shutdown_connections 2024-03-28T23:35:09.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.617+0000 7f65af9ae640 1 --2- 172.21.15.161:0/2102677845 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f658c06b4d0 0x7f658c06d990 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:09.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.617+0000 7f65af9ae640 1 --2- 172.21.15.161:0/2102677845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65a8152890 0x7f65a814f2d0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:09.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.617+0000 7f65af9ae640 1 --2- 172.21.15.161:0/2102677845 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65a81553a0 0x7f65a814f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:09.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.617+0000 7f65af9ae640 1 --2- 172.21.15.161:0/2102677845 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65a80c5410 0x7f65a814e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:09.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.617+0000 7f65af9ae640 1 -- 172.21.15.161:0/2102677845 >> 172.21.15.161:0/2102677845 conn(0x7f65a80ac620 msgr2=0x7f65a806c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:09.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.618+0000 7f65af9ae640 1 -- 172.21.15.161:0/2102677845 shutdown_connections 2024-03-28T23:35:09.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.618+0000 7f65af9ae640 1 -- 172.21.15.161:0/2102677845 wait complete. 2024-03-28T23:35:09.631 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-28T23:35:09.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.738+0000 7f49ad7f4640 1 Processor -- start 2024-03-28T23:35:09.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.738+0000 7f49ad7f4640 1 -- start start 2024-03-28T23:35:09.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.738+0000 7f49ad7f4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f49a80c5850 0x7f49a8155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:09.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.738+0000 7f49ad7f4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49a80c5d00 con 0x7f49a8152890 2024-03-28T23:35:09.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.738+0000 7f49ad7f4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49a8158a30 con 0x7f49a80c5410 2024-03-28T23:35:09.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.738+0000 7f49ad7f4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f49a80c8480 con 0x7f49a80c5850 2024-03-28T23:35:09.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.738+0000 7f49a77fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f49a80c5850 0x7f49a8155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:09.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.738+0000 7f49a6ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f49a8152890 0x7f49a80c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45346/0 (socket says 172.21.15.161:45346) 2024-03-28T23:35:09.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.738+0000 7f49a6ffd640 1 -- 172.21.15.161:0/3799603130 learned_addr learned my addr 172.21.15.161:0/3799603130 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:09.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.739+0000 7f49a77fe640 1 -- 172.21.15.161:0/3799603130 >> v1:172.21.15.170:6789/0 conn(0x7f49a80c5410 legacy=0x7f49a81553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:09.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.739+0000 7f49a77fe640 1 -- 172.21.15.161:0/3799603130 >> v1:172.21.15.161:6789/0 conn(0x7f49a8152890 legacy=0x7f49a80c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:09.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.739+0000 7f49a77fe640 1 -- 172.21.15.161:0/3799603130 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a8158d10 con 0x7f49a80c5850 2024-03-28T23:35:09.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.739+0000 7f49a5ffb640 1 -- 172.21.15.161:0/3799603130 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1709883115 0 0) 0x7f49a80c5d00 con 0x7f49a8152890 2024-03-28T23:35:09.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.739+0000 7f49a77fe640 1 --2- 172.21.15.161:0/3799603130 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49a80c5850 0x7f49a8155ab0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f49a0009d90 tx=0x7f49a002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58e0000750c53a6e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:09.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.740+0000 7f49a5ffb640 1 -- 172.21.15.161:0/3799603130 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a0039070 con 0x7f49a80c5850 2024-03-28T23:35:09.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.740+0000 7f49a5ffb640 1 -- 172.21.15.161:0/3799603130 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49a002cbd0 con 0x7f49a80c5850 2024-03-28T23:35:09.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.740+0000 7f49a5ffb640 1 -- 172.21.15.161:0/3799603130 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a0034440 con 0x7f49a80c5850 2024-03-28T23:35:09.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.740+0000 7f49ad7f4640 1 -- 172.21.15.161:0/3799603130 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49a80c5850 msgr2=0x7f49a8155ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:09.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.740+0000 7f49ad7f4640 1 --2- 172.21.15.161:0/3799603130 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49a80c5850 0x7f49a8155ab0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f49a0009d90 tx=0x7f49a002be20 comp rx=0 tx=0).stop 2024-03-28T23:35:09.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.740+0000 7f49ad7f4640 1 -- 172.21.15.161:0/3799603130 shutdown_connections 2024-03-28T23:35:09.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.740+0000 7f49ad7f4640 1 --2- 172.21.15.161:0/3799603130 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49a80c5850 0x7f49a8155ab0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:09.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.740+0000 7f49ad7f4640 1 -- 172.21.15.161:0/3799603130 >> 172.21.15.161:0/3799603130 conn(0x7f49a80ac620 msgr2=0x7f49a80aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:09.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.740+0000 7f49ad7f4640 1 -- 172.21.15.161:0/3799603130 shutdown_connections 2024-03-28T23:35:09.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.740+0000 7f49ad7f4640 1 -- 172.21.15.161:0/3799603130 wait complete. 2024-03-28T23:35:09.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.741+0000 7f49ad7f4640 1 Processor -- start 2024-03-28T23:35:09.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.741+0000 7f49ad7f4640 1 -- start start 2024-03-28T23:35:09.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.742+0000 7f49ad7f4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49a80c5410 0x7f49a81e8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:09.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.742+0000 7f49ad7f4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49a80c5850 0x7f49a80beed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:09.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.742+0000 7f49ad7f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49a8152890 0x7f49a80c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:09.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.742+0000 7f49ad7f4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f49a81592d0 con 0x7f49a80c5410 2024-03-28T23:35:09.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.742+0000 7f49ad7f4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f49a8159640 con 0x7f49a80c5850 2024-03-28T23:35:09.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.742+0000 7f49ad7f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f49a8155ff0 con 0x7f49a8152890 2024-03-28T23:35:09.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.742+0000 7f49a6ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49a80c5410 0x7f49a81e8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:09.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.742+0000 7f49a6ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49a80c5410 0x7f49a81e8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57702/0 (socket says 172.21.15.161:57702) 2024-03-28T23:35:09.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.742+0000 7f49a6ffd640 1 -- 172.21.15.161:0/1981755664 learned_addr learned my addr 172.21.15.161:0/1981755664 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:09.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.742+0000 7f49a6ffd640 1 -- 172.21.15.161:0/1981755664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49a8152890 msgr2=0x7f49a80c9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:09.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.742+0000 7f49a6ffd640 1 --2- 172.21.15.161:0/1981755664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49a8152890 0x7f49a80c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:09.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.742+0000 7f49a6ffd640 1 -- 172.21.15.161:0/1981755664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49a80c5850 msgr2=0x7f49a80beed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:09.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.742+0000 7f49a6ffd640 1 --2- 172.21.15.161:0/1981755664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49a80c5850 0x7f49a80beed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:09.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.743+0000 7f49a6ffd640 1 -- 172.21.15.161:0/1981755664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a8155230 con 0x7f49a80c5410 2024-03-28T23:35:09.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.743+0000 7f49a6ffd640 1 --2- 172.21.15.161:0/1981755664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49a80c5410 0x7f49a81e8380 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f499c00c500 tx=0x7f499c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:09.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.743+0000 7f498f7fe640 1 -- 172.21.15.161:0/1981755664 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f499c018070 con 0x7f49a80c5410 2024-03-28T23:35:09.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.743+0000 7f498f7fe640 1 -- 172.21.15.161:0/1981755664 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f499c007580 con 0x7f49a80c5410 2024-03-28T23:35:09.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.743+0000 7f49ad7f4640 1 -- 172.21.15.161:0/1981755664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49a8154300 con 0x7f49a80c5410 2024-03-28T23:35:09.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.743+0000 7f498f7fe640 1 -- 172.21.15.161:0/1981755664 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f499c004a20 con 0x7f49a80c5410 2024-03-28T23:35:09.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.743+0000 7f49ad7f4640 1 -- 172.21.15.161:0/1981755664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49a8153830 con 0x7f49a80c5410 2024-03-28T23:35:09.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.744+0000 7f49ad7f4640 1 -- 172.21.15.161:0/1981755664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49a8153a30 con 0x7f49a80c5410 2024-03-28T23:35:09.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.744+0000 7f498f7fe640 1 -- 172.21.15.161:0/1981755664 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f499c01b070 con 0x7f49a80c5410 2024-03-28T23:35:09.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.745+0000 7f498f7fe640 1 --2- 172.21.15.161:0/1981755664 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f499406b4d0 0x7f499406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:09.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.745+0000 7f498f7fe640 1 -- 172.21.15.161:0/1981755664 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5249+0+0 (secure 0 0 0) 0x7f499c0892e0 con 0x7f49a80c5410 2024-03-28T23:35:09.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.745+0000 7f49a67fc640 1 --2- 172.21.15.161:0/1981755664 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f499406b4d0 0x7f499406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:09.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.745+0000 7f49a67fc640 1 --2- 172.21.15.161:0/1981755664 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f499406b4d0 0x7f499406d990 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f49a80ca610 tx=0x7f4998006b90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:09.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.749+0000 7f498f7fe640 1 -- 172.21.15.161:0/1981755664 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f499c0583b0 con 0x7f49a80c5410 2024-03-28T23:35:09.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.987+0000 7f49ad7f4640 1 -- 172.21.15.161:0/1981755664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f49a804f4b0 con 0x7f49a80c5410 2024-03-28T23:35:09.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.987+0000 7f498f7fe640 1 -- 172.21.15.161:0/1981755664 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) ==== 80+0+4724 (secure 0 0 0) 0x7f499c057cb0 con 0x7f49a80c5410 2024-03-28T23:35:09.988 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:09.988 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:35:09.988 INFO:teuthology.orchestra.run.smithi161.stdout: "auth_dump": [ 2024-03-28T23:35:09.988 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:09.988 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.a", 2024-03-28T23:35:09.988 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmggtoJhAASLXbUICLBhjRr19aUAuZHw==", 2024-03-28T23:35:09.988 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:09.988 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:35:09.988 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:35:09.988 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:35:09.988 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:09.988 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:09.989 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:09.989 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:09.989 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.b", 2024-03-28T23:35:09.989 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmQbEOMxAA74al6+dMzcN/ngwNZK+aHA==", 2024-03-28T23:35:09.989 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:09.989 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:35:09.989 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:35:09.989 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:35:09.989 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:09.989 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:09.989 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:09.989 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:09.989 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.c", 2024-03-28T23:35:09.989 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmjf/qLRAAosLbpT8aCWvrKAQnj8Uh3g==", 2024-03-28T23:35:09.989 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:09.990 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:35:09.990 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:35:09.990 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:35:09.990 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:09.990 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:09.990 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:09.990 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:09.990 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.d", 2024-03-28T23:35:09.990 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmpiZ1OhAAjYrFAaDosDS2Y+BvU4c6JQ==", 2024-03-28T23:35:09.990 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:09.990 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:35:09.990 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:35:09.990 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:35:09.991 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:09.991 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:09.991 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:09.991 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:09.991 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.0", 2024-03-28T23:35:09.991 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA9/QVmAFPmGBAAS0SkRRwsupm6jm1sqFXJ6g==", 2024-03-28T23:35:09.991 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:09.991 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:35:09.991 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:35:09.991 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:09.991 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:09.991 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:09.991 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:09.991 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.1", 2024-03-28T23:35:09.992 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBA/QVmNBTVFxAA1htU9ilznfLUDqKmJaQwzg==", 2024-03-28T23:35:09.992 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:09.992 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:35:09.992 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:35:09.992 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:09.992 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:09.992 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:09.992 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:09.992 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.2", 2024-03-28T23:35:09.992 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBD/QVmg+4YFhAA9TvfFVKT3nQT7/QUOf3Spw==", 2024-03-28T23:35:09.992 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:09.992 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:35:09.993 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:35:09.993 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:09.993 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:09.993 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:09.993 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:09.993 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.3", 2024-03-28T23:35:09.993 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBG/QVmgiHDFBAAQPtbZT6a1ZGmkisBX0NMgA==", 2024-03-28T23:35:09.994 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:09.994 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:35:09.994 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:35:09.994 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:09.994 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:09.994 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:09.994 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:09.994 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.4", 2024-03-28T23:35:09.994 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBK/QVmahhZIBAAJfotg/dusbB3usreZlRuPA==", 2024-03-28T23:35:09.994 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:09.994 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:35:09.994 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:35:09.994 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:09.995 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:09.995 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:09.995 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:09.995 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.5", 2024-03-28T23:35:09.995 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBN/QVmzjlDHhAAli7pLyIo+MVdwSBNJYVg5g==", 2024-03-28T23:35:09.995 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:09.995 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:35:09.995 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:35:09.995 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:09.995 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:09.995 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:09.995 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:09.995 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.6", 2024-03-28T23:35:09.995 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBQ/QVmtCFSHBAASVfV/5mY/3JNiI0YMT7Vww==", 2024-03-28T23:35:09.996 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:09.996 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:35:09.996 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:35:09.996 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:09.996 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:09.996 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:09.996 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:09.996 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.7", 2024-03-28T23:35:09.996 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBT/QVm2J7eGhAAaIeZxmLwQBBE1glCxjKbhA==", 2024-03-28T23:35:09.996 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:09.996 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:35:09.996 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:35:09.996 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:09.996 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:09.997 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:09.997 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:09.997 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.0", 2024-03-28T23:35:09.997 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA6/QVmc1krAxAAEKsn1Tuuv9OlFZpB4PFhDg==", 2024-03-28T23:35:09.997 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:09.997 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:35:09.997 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow r", 2024-03-28T23:35:09.997 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-28T23:35:09.997 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:09.997 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:09.997 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:09.997 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.1", 2024-03-28T23:35:09.997 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA6/QVm1B2cBhAAmHDeeFwpFlV/y3NIGWUXHQ==", 2024-03-28T23:35:09.997 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:09.997 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:35:09.997 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow r", 2024-03-28T23:35:09.998 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow rw", 2024-03-28T23:35:09.998 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow rwx" 2024-03-28T23:35:09.998 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:09.998 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:09.998 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:09.998 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.admin", 2024-03-28T23:35:09.998 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmzzJ7CRAAYqfhnIkfNJJ3O+VWC2LP+Q==", 2024-03-28T23:35:09.998 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:09.998 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-28T23:35:09.998 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:35:09.998 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:35:09.998 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:09.999 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:09.999 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:09.999 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:09.999 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mgr.x", 2024-03-28T23:35:09.999 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVm7TD3IhAACByGRGpvJWDpF22NuovHWw==", 2024-03-28T23:35:09.999 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:09.999 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-28T23:35:09.999 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile mgr", 2024-03-28T23:35:09.999 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:09.999 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:09.999 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:09.999 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:09.999 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mgr.y", 2024-03-28T23:35:09.999 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVm0hLlHhAAh8cewPtZ/TbKrU488kxrBg==", 2024-03-28T23:35:09.999 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:10.000 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-28T23:35:10.000 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile mgr", 2024-03-28T23:35:10.000 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:10.000 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:10.000 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:10.000 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-03-28T23:35:10.000 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:35:10.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.989+0000 7f49ad7f4640 1 -- 172.21.15.161:0/1981755664 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f499406b4d0 msgr2=0x7f499406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:10.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.989+0000 7f49ad7f4640 1 --2- 172.21.15.161:0/1981755664 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f499406b4d0 0x7f499406d990 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f49a80ca610 tx=0x7f4998006b90 comp rx=0 tx=0).stop 2024-03-28T23:35:10.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.989+0000 7f49ad7f4640 1 -- 172.21.15.161:0/1981755664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49a80c5410 msgr2=0x7f49a81e8380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:10.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.989+0000 7f49ad7f4640 1 --2- 172.21.15.161:0/1981755664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49a80c5410 0x7f49a81e8380 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f499c00c500 tx=0x7f499c00c9c0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.989+0000 7f49ad7f4640 1 -- 172.21.15.161:0/1981755664 shutdown_connections 2024-03-28T23:35:10.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.989+0000 7f49ad7f4640 1 --2- 172.21.15.161:0/1981755664 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f499406b4d0 0x7f499406d990 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.989+0000 7f49ad7f4640 1 --2- 172.21.15.161:0/1981755664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49a8152890 0x7f49a80c9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.989+0000 7f49ad7f4640 1 --2- 172.21.15.161:0/1981755664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49a80c5850 0x7f49a80beed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.989+0000 7f49ad7f4640 1 --2- 172.21.15.161:0/1981755664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49a80c5410 0x7f49a81e8380 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.989+0000 7f49ad7f4640 1 -- 172.21.15.161:0/1981755664 >> 172.21.15.161:0/1981755664 conn(0x7f49a80ac620 msgr2=0x7f49a814ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:10.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.989+0000 7f49ad7f4640 1 -- 172.21.15.161:0/1981755664 shutdown_connections 2024-03-28T23:35:10.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:09.989+0000 7f49ad7f4640 1 -- 172.21.15.161:0/1981755664 wait complete. 2024-03-28T23:35:10.003 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:35:10.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.103+0000 7f951721c640 1 Processor -- start 2024-03-28T23:35:10.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.103+0000 7f951721c640 1 -- start start 2024-03-28T23:35:10.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.104+0000 7f951721c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f951014be70 0x7f95100ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:10.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.104+0000 7f951721c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95100ab6e0 0x7f95100aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:10.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.104+0000 7f951721c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f951014c250 con 0x7f951006ce80 2024-03-28T23:35:10.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.104+0000 7f951721c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f95101529c0 con 0x7f951014be70 2024-03-28T23:35:10.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.104+0000 7f951721c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f95100a9dc0 con 0x7f95100ab6e0 2024-03-28T23:35:10.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.104+0000 7f9514fb8640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f951014be70 0x7f95100ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:10.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.104+0000 7f950ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95100ab6e0 0x7f95100aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:10.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.104+0000 7f950ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95100ab6e0 0x7f95100aa5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:57014/0 (socket says 172.21.15.161:57014) 2024-03-28T23:35:10.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.104+0000 7f9514fb8640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f951014be70 0x7f95100ab1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:52004/0 (socket says 172.21.15.161:52004) 2024-03-28T23:35:10.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.104+0000 7f950ffff640 1 -- 172.21.15.161:0/4180629838 learned_addr learned my addr 172.21.15.161:0/4180629838 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:10.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.104+0000 7f95157b9640 1 --1- 172.21.15.161:0/4180629838 >> v1:172.21.15.170:6789/0 conn(0x7f951006ce80 0x7f95101553b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:40480/0 (socket says 172.21.15.161:40480) 2024-03-28T23:35:10.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.104+0000 7f950ffff640 1 -- 172.21.15.161:0/4180629838 >> v2:172.21.15.170:3300/0 conn(0x7f951014be70 msgr2=0x7f95100ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:10.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.104+0000 7f950ffff640 1 --2- 172.21.15.161:0/4180629838 >> v2:172.21.15.170:3300/0 conn(0x7f951014be70 0x7f95100ab1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.104+0000 7f950ffff640 1 -- 172.21.15.161:0/4180629838 >> v1:172.21.15.170:6789/0 conn(0x7f951006ce80 legacy=0x7f95101553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:10.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.104+0000 7f950ffff640 1 -- 172.21.15.161:0/4180629838 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95100aa0a0 con 0x7f95100ab6e0 2024-03-28T23:35:10.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.104+0000 7f9514fb8640 1 --2- 172.21.15.161:0/4180629838 >> v2:172.21.15.170:3300/0 conn(0x7f951014be70 0x7f95100ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:10.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.105+0000 7f950ffff640 1 --2- 172.21.15.161:0/4180629838 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95100ab6e0 0x7f95100aa5c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f94fc009870 tx=0x7f94fc02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd71d341820464f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:10.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.105+0000 7f950f7fe640 1 -- 172.21.15.161:0/4180629838 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94fc035020 con 0x7f95100ab6e0 2024-03-28T23:35:10.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.105+0000 7f950f7fe640 1 -- 172.21.15.161:0/4180629838 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94fc031640 con 0x7f95100ab6e0 2024-03-28T23:35:10.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.105+0000 7f950f7fe640 1 -- 172.21.15.161:0/4180629838 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94fc02d040 con 0x7f95100ab6e0 2024-03-28T23:35:10.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.106+0000 7f951721c640 1 -- 172.21.15.161:0/4180629838 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95100ab6e0 msgr2=0x7f95100aa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:10.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.106+0000 7f951721c640 1 --2- 172.21.15.161:0/4180629838 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95100ab6e0 0x7f95100aa5c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f94fc009870 tx=0x7f94fc02b8f0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.106+0000 7f951721c640 1 -- 172.21.15.161:0/4180629838 shutdown_connections 2024-03-28T23:35:10.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.106+0000 7f951721c640 1 --2- 172.21.15.161:0/4180629838 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f95100ab6e0 0x7f95100aa5c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.106+0000 7f951721c640 1 --2- 172.21.15.161:0/4180629838 >> v2:172.21.15.170:3300/0 conn(0x7f951014be70 0x7f95100ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.106+0000 7f951721c640 1 -- 172.21.15.161:0/4180629838 >> 172.21.15.161:0/4180629838 conn(0x7f95100c6670 msgr2=0x7f95100c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:10.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.106+0000 7f951721c640 1 -- 172.21.15.161:0/4180629838 shutdown_connections 2024-03-28T23:35:10.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.106+0000 7f951721c640 1 -- 172.21.15.161:0/4180629838 wait complete. 2024-03-28T23:35:10.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.107+0000 7f951721c640 1 Processor -- start 2024-03-28T23:35:10.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.107+0000 7f951721c640 1 -- start start 2024-03-28T23:35:10.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.108+0000 7f951721c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f951006ce80 0x7f951014c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:10.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.108+0000 7f951721c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95100ab6e0 0x7f95101ebce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:10.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.108+0000 7f951721c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f951014be70 0x7f95101ec220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:10.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.108+0000 7f951721c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f951006b2d0 con 0x7f95100ab6e0 2024-03-28T23:35:10.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.108+0000 7f951721c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f951006b150 con 0x7f951014be70 2024-03-28T23:35:10.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.108+0000 7f951721c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f95100c1520 con 0x7f951006ce80 2024-03-28T23:35:10.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.108+0000 7f950ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95100ab6e0 0x7f95101ebce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:10.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.108+0000 7f950ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95100ab6e0 0x7f95101ebce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57712/0 (socket says 172.21.15.161:57712) 2024-03-28T23:35:10.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.108+0000 7f950ffff640 1 -- 172.21.15.161:0/2976583565 learned_addr learned my addr 172.21.15.161:0/2976583565 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:10.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.108+0000 7f9514fb8640 1 --2- 172.21.15.161:0/2976583565 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f951006ce80 0x7f951014c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:10.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.108+0000 7f95157b9640 1 --2- 172.21.15.161:0/2976583565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f951014be70 0x7f95101ec220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:10.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.108+0000 7f950ffff640 1 -- 172.21.15.161:0/2976583565 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f951006ce80 msgr2=0x7f951014c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:10.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.108+0000 7f950ffff640 1 --2- 172.21.15.161:0/2976583565 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f951006ce80 0x7f951014c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.108+0000 7f950ffff640 1 -- 172.21.15.161:0/2976583565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f951014be70 msgr2=0x7f95101ec220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:10.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.108+0000 7f950ffff640 1 --2- 172.21.15.161:0/2976583565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f951014be70 0x7f95101ec220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.108+0000 7f950ffff640 1 -- 172.21.15.161:0/2976583565 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f951014ec10 con 0x7f95100ab6e0 2024-03-28T23:35:10.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.108+0000 7f95157b9640 1 --2- 172.21.15.161:0/2976583565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f951014be70 0x7f95101ec220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:10.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.109+0000 7f950ffff640 1 --2- 172.21.15.161:0/2976583565 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95100ab6e0 0x7f95101ebce0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f94fc000c00 tx=0x7f94fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:10.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.109+0000 7f950d7fa640 1 -- 172.21.15.161:0/2976583565 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94fc040070 con 0x7f95100ab6e0 2024-03-28T23:35:10.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.109+0000 7f950d7fa640 1 -- 172.21.15.161:0/2976583565 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94fc0399f0 con 0x7f95100ab6e0 2024-03-28T23:35:10.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.109+0000 7f950d7fa640 1 -- 172.21.15.161:0/2976583565 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94fc035040 con 0x7f95100ab6e0 2024-03-28T23:35:10.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.109+0000 7f951721c640 1 -- 172.21.15.161:0/2976583565 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95100c09d0 con 0x7f95100ab6e0 2024-03-28T23:35:10.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.109+0000 7f951721c640 1 -- 172.21.15.161:0/2976583565 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95101ecb00 con 0x7f95100ab6e0 2024-03-28T23:35:10.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.110+0000 7f951721c640 1 -- 172.21.15.161:0/2976583565 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f951004f4b0 con 0x7f95100ab6e0 2024-03-28T23:35:10.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.110+0000 7f950d7fa640 1 -- 172.21.15.161:0/2976583565 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f94fc04b060 con 0x7f95100ab6e0 2024-03-28T23:35:10.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.110+0000 7f950d7fa640 1 --2- 172.21.15.161:0/2976583565 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f94f406b4d0 0x7f94f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:10.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.111+0000 7f950d7fa640 1 -- 172.21.15.161:0/2976583565 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5249+0+0 (secure 0 0 0) 0x7f94fc0ab7c0 con 0x7f95100ab6e0 2024-03-28T23:35:10.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.111+0000 7f9514fb8640 1 --2- 172.21.15.161:0/2976583565 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f94f406b4d0 0x7f94f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:10.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.111+0000 7f9514fb8640 1 --2- 172.21.15.161:0/2976583565 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f94f406b4d0 0x7f94f406d990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f9504009560 tx=0x7f9504008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:10.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.115+0000 7f950d7fa640 1 -- 172.21.15.161:0/2976583565 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f94fc07a890 con 0x7f95100ab6e0 2024-03-28T23:35:10.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.288+0000 7f951721c640 1 -- 172.21.15.161:0/2976583565 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f95100c0ff0 con 0x7f95100ab6e0 2024-03-28T23:35:10.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.289+0000 7f950d7fa640 1 -- 172.21.15.161:0/2976583565 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) ==== 74+0+13210 (secure 0 0 0) 0x7f94fc07a190 con 0x7f95100ab6e0 2024-03-28T23:35:10.289 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:10.289 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":132,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:35:09.612459+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:35:10.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.291+0000 7f951721c640 1 -- 172.21.15.161:0/2976583565 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f94f406b4d0 msgr2=0x7f94f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:10.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.291+0000 7f951721c640 1 --2- 172.21.15.161:0/2976583565 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f94f406b4d0 0x7f94f406d990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f9504009560 tx=0x7f9504008040 comp rx=0 tx=0).stop 2024-03-28T23:35:10.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.291+0000 7f951721c640 1 -- 172.21.15.161:0/2976583565 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95100ab6e0 msgr2=0x7f95101ebce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:10.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.291+0000 7f951721c640 1 --2- 172.21.15.161:0/2976583565 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95100ab6e0 0x7f95101ebce0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f94fc000c00 tx=0x7f94fc0046c0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.291+0000 7f951721c640 1 -- 172.21.15.161:0/2976583565 shutdown_connections 2024-03-28T23:35:10.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.291+0000 7f951721c640 1 --2- 172.21.15.161:0/2976583565 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f94f406b4d0 0x7f94f406d990 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.291+0000 7f951721c640 1 --2- 172.21.15.161:0/2976583565 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95100ab6e0 0x7f95101ebce0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.291+0000 7f951721c640 1 --2- 172.21.15.161:0/2976583565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f951014be70 0x7f95101ec220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.291+0000 7f951721c640 1 --2- 172.21.15.161:0/2976583565 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f951006ce80 0x7f951014c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.291+0000 7f951721c640 1 -- 172.21.15.161:0/2976583565 >> 172.21.15.161:0/2976583565 conn(0x7f95100c6670 msgr2=0x7f95100c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:10.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.291+0000 7f951721c640 1 -- 172.21.15.161:0/2976583565 shutdown_connections 2024-03-28T23:35:10.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.291+0000 7f951721c640 1 -- 172.21.15.161:0/2976583565 wait complete. 2024-03-28T23:35:10.302 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:35:10.303 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:35:10.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.406+0000 7f71ffc40640 1 Processor -- start 2024-03-28T23:35:10.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.406+0000 7f71ffc40640 1 -- start start 2024-03-28T23:35:10.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.406+0000 7f71ffc40640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f71f806cdd0 0x7f71f815d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:10.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.406+0000 7f71ffc40640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71f8152960 con 0x7f71f80c6350 2024-03-28T23:35:10.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.406+0000 7f71ffc40640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71f80c1e40 con 0x7f71f8151e70 2024-03-28T23:35:10.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.406+0000 7f71ffc40640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f71f80c2120 con 0x7f71f806cdd0 2024-03-28T23:35:10.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.407+0000 7f71fd1db640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f71f80c6350 0x7f71f806c6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45362/0 (socket says 172.21.15.161:45362) 2024-03-28T23:35:10.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.407+0000 7f71fd1db640 1 -- 172.21.15.161:0/1064399661 learned_addr learned my addr 172.21.15.161:0/1064399661 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:10.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.407+0000 7f71fc9da640 1 -- 172.21.15.161:0/1064399661 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1198058614 0 0) 0x7f71f8152960 con 0x7f71f80c6350 2024-03-28T23:35:10.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.407+0000 7f71fc9da640 1 -- 172.21.15.161:0/1064399661 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71e4003520 con 0x7f71f80c6350 2024-03-28T23:35:10.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.407+0000 7f71fe1dd640 1 --2- 172.21.15.161:0/1064399661 >> v2:172.21.15.170:3300/0 conn(0x7f71f806cdd0 0x7f71f815d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:10.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.407+0000 7f71fc9da640 1 -- 172.21.15.161:0/1064399661 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3350911657 0 0) 0x7f71e4003520 con 0x7f71f80c6350 2024-03-28T23:35:10.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.407+0000 7f71fc9da640 1 -- 172.21.15.161:0/1064399661 >> v2:172.21.15.170:3300/0 conn(0x7f71f806cdd0 msgr2=0x7f71f815d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:10.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.407+0000 7f71fc9da640 1 --2- 172.21.15.161:0/1064399661 >> v2:172.21.15.170:3300/0 conn(0x7f71f806cdd0 0x7f71f815d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.407+0000 7f71fc9da640 1 -- 172.21.15.161:0/1064399661 >> v1:172.21.15.170:6789/0 conn(0x7f71f8151e70 legacy=0x7f71f8152250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:10.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.407+0000 7f71fc9da640 1 -- 172.21.15.161:0/1064399661 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71f80c2400 con 0x7f71f80c6350 2024-03-28T23:35:10.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.407+0000 7f71fc9da640 1 -- 172.21.15.161:0/1064399661 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f71e8001670 con 0x7f71f80c6350 2024-03-28T23:35:10.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.407+0000 7f71fc9da640 1 -- 172.21.15.161:0/1064399661 >> v1:172.21.15.161:6789/0 conn(0x7f71f80c6350 legacy=0x7f71f806c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:10.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.407+0000 7f71fc9da640 1 --2- 172.21.15.161:0/1064399661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f71e40039b0 0x7f71e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:10.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.407+0000 7f71fc9da640 1 --2- 172.21.15.161:0/1064399661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f71e4004800 0x7f71e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:10.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.407+0000 7f71fc9da640 1 --2- 172.21.15.161:0/1064399661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71e40051c0 0x7f71e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:10.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.407+0000 7f71fc9da640 1 -- 172.21.15.161:0/1064399661 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f71f80c2120 con 0x7f71e40051c0 2024-03-28T23:35:10.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.407+0000 7f71fc9da640 1 -- 172.21.15.161:0/1064399661 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f71e400c160 con 0x7f71e40039b0 2024-03-28T23:35:10.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.407+0000 7f71fc9da640 1 -- 172.21.15.161:0/1064399661 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f71e400c360 con 0x7f71e4004800 2024-03-28T23:35:10.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.407+0000 7f71fe1dd640 1 --2- 172.21.15.161:0/1064399661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71e40051c0 0x7f71e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:10.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.407+0000 7f71fd1db640 1 --2- 172.21.15.161:0/1064399661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f71e4004800 0x7f71e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:10.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.407+0000 7f71fd9dc640 1 --2- 172.21.15.161:0/1064399661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f71e40039b0 0x7f71e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:10.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.408+0000 7f71fe1dd640 1 -- 172.21.15.161:0/1064399661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f71e4004800 msgr2=0x7f71e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:10.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.408+0000 7f71fe1dd640 1 --2- 172.21.15.161:0/1064399661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f71e4004800 0x7f71e4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.408+0000 7f71fe1dd640 1 -- 172.21.15.161:0/1064399661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f71e40039b0 msgr2=0x7f71e4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:10.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.408+0000 7f71fe1dd640 1 --2- 172.21.15.161:0/1064399661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f71e40039b0 0x7f71e4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.408+0000 7f71fe1dd640 1 -- 172.21.15.161:0/1064399661 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71e4003520 con 0x7f71e40051c0 2024-03-28T23:35:10.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.408+0000 7f71fe1dd640 1 -- 172.21.15.161:0/1064399661 reap_dead start 2024-03-28T23:35:10.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.408+0000 7f71fe1dd640 1 --2- 172.21.15.161:0/1064399661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71e40051c0 0x7f71e400b9c0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f71f4009fc0 tx=0x7f71e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f023f0c1ceb072b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:10.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.409+0000 7f71fc9da640 1 -- 172.21.15.161:0/1064399661 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71f4009c10 con 0x7f71e40051c0 2024-03-28T23:35:10.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.409+0000 7f71fc9da640 1 -- 172.21.15.161:0/1064399661 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71f402cca0 con 0x7f71e40051c0 2024-03-28T23:35:10.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.409+0000 7f71fc9da640 1 -- 172.21.15.161:0/1064399661 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71f4003bd0 con 0x7f71e40051c0 2024-03-28T23:35:10.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.409+0000 7f71ffc40640 1 -- 172.21.15.161:0/1064399661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71e40051c0 msgr2=0x7f71e400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:10.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.409+0000 7f71ffc40640 1 --2- 172.21.15.161:0/1064399661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71e40051c0 0x7f71e400b9c0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f71f4009fc0 tx=0x7f71e4005830 comp rx=0 tx=0).stop 2024-03-28T23:35:10.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.409+0000 7f71ffc40640 1 -- 172.21.15.161:0/1064399661 shutdown_connections 2024-03-28T23:35:10.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.409+0000 7f71ffc40640 1 --2- 172.21.15.161:0/1064399661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71e40051c0 0x7f71e400b9c0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.409+0000 7f71ffc40640 1 -- 172.21.15.161:0/1064399661 >> 172.21.15.161:0/1064399661 conn(0x7f71f80c3390 msgr2=0x7f71f80c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:10.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.409+0000 7f71ffc40640 1 -- 172.21.15.161:0/1064399661 shutdown_connections 2024-03-28T23:35:10.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.409+0000 7f71ffc40640 1 -- 172.21.15.161:0/1064399661 wait complete. 2024-03-28T23:35:10.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.410+0000 7f71ffc40640 1 Processor -- start 2024-03-28T23:35:10.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.410+0000 7f71ffc40640 1 -- start start 2024-03-28T23:35:10.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.411+0000 7f71ffc40640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f71e40051c0 0x7f71f81eca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:10.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.411+0000 7f71ffc40640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f71f80ac2f0 0x7f71f8072a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:10.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.411+0000 7f71ffc40640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71f8144840 0x7f71f8144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:10.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.411+0000 7f71ffc40640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f71f80c2c00 con 0x7f71f8144840 2024-03-28T23:35:10.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.411+0000 7f71ffc40640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f71f814f810 con 0x7f71f80ac2f0 2024-03-28T23:35:10.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.411+0000 7f71ffc40640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f71f80c2d80 con 0x7f71e40051c0 2024-03-28T23:35:10.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.411+0000 7f71fe1dd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71f8144840 0x7f71f8144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:10.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.411+0000 7f71fe1dd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71f8144840 0x7f71f8144cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57736/0 (socket says 172.21.15.161:57736) 2024-03-28T23:35:10.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.411+0000 7f71fe1dd640 1 -- 172.21.15.161:0/1036226576 learned_addr learned my addr 172.21.15.161:0/1036226576 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:10.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.411+0000 7f71fd1db640 1 --2- 172.21.15.161:0/1036226576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f71f80ac2f0 0x7f71f8072a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:10.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.411+0000 7f71fd9dc640 1 --2- 172.21.15.161:0/1036226576 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f71e40051c0 0x7f71f81eca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:10.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.411+0000 7f71fe1dd640 1 -- 172.21.15.161:0/1036226576 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f71e40051c0 msgr2=0x7f71f81eca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:10.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.411+0000 7f71fe1dd640 1 --2- 172.21.15.161:0/1036226576 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f71e40051c0 0x7f71f81eca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.411+0000 7f71fe1dd640 1 -- 172.21.15.161:0/1036226576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f71f80ac2f0 msgr2=0x7f71f8072a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:10.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.411+0000 7f71fe1dd640 1 --2- 172.21.15.161:0/1036226576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f71f80ac2f0 0x7f71f8072a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.411+0000 7f71fe1dd640 1 -- 172.21.15.161:0/1036226576 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71f81ed0e0 con 0x7f71f8144840 2024-03-28T23:35:10.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.411+0000 7f71fd9dc640 1 --2- 172.21.15.161:0/1036226576 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f71e40051c0 0x7f71f81eca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:10.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.412+0000 7f71fe1dd640 1 --2- 172.21.15.161:0/1036226576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71f8144840 0x7f71f8144cc0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f71f4035d50 tx=0x7f71f4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:10.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.412+0000 7f71ee7fc640 1 -- 172.21.15.161:0/1036226576 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71f4003990 con 0x7f71f8144840 2024-03-28T23:35:10.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.412+0000 7f71ee7fc640 1 -- 172.21.15.161:0/1036226576 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71f4044070 con 0x7f71f8144840 2024-03-28T23:35:10.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.412+0000 7f71ee7fc640 1 -- 172.21.15.161:0/1036226576 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71f4003d90 con 0x7f71f8144840 2024-03-28T23:35:10.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.412+0000 7f71ffc40640 1 -- 172.21.15.161:0/1036226576 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71f81e9bc0 con 0x7f71f8144840 2024-03-28T23:35:10.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.412+0000 7f71ffc40640 1 -- 172.21.15.161:0/1036226576 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71f81e90e0 con 0x7f71f8144840 2024-03-28T23:35:10.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.413+0000 7f71ffc40640 1 -- 172.21.15.161:0/1036226576 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71f80694e0 con 0x7f71f8144840 2024-03-28T23:35:10.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.413+0000 7f71ee7fc640 1 -- 172.21.15.161:0/1036226576 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f71f4005d10 con 0x7f71f8144840 2024-03-28T23:35:10.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.414+0000 7f71ee7fc640 1 --2- 172.21.15.161:0/1036226576 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f71dc06b4d0 0x7f71dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:10.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.414+0000 7f71ee7fc640 1 -- 172.21.15.161:0/1036226576 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5249+0+0 (secure 0 0 0) 0x7f71f40ac5e0 con 0x7f71f8144840 2024-03-28T23:35:10.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.414+0000 7f71fd9dc640 1 --2- 172.21.15.161:0/1036226576 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f71dc06b4d0 0x7f71dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:10.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.414+0000 7f71fd9dc640 1 --2- 172.21.15.161:0/1036226576 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f71dc06b4d0 0x7f71dc06d990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f71f0000c00 tx=0x7f71f0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:10.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.418+0000 7f71ee7fc640 1 -- 172.21.15.161:0/1036226576 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f71f80694e0 con 0x7f71f8144840 2024-03-28T23:35:10.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.586+0000 7f71ffc40640 1 -- 172.21.15.161:0/1036226576 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f71f804f4b0 con 0x7f71f8144840 2024-03-28T23:35:10.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.587+0000 7f71ee7fc640 1 -- 172.21.15.161:0/1036226576 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v132) ==== 93+0+10 (secure 0 0 0) 0x7f71f81ed0e0 con 0x7f71f8144840 2024-03-28T23:35:10.588 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:35:10.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.589+0000 7f71ffc40640 1 -- 172.21.15.161:0/1036226576 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f71dc06b4d0 msgr2=0x7f71dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:10.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.589+0000 7f71ffc40640 1 --2- 172.21.15.161:0/1036226576 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f71dc06b4d0 0x7f71dc06d990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f71f0000c00 tx=0x7f71f0009040 comp rx=0 tx=0).stop 2024-03-28T23:35:10.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.589+0000 7f71ffc40640 1 -- 172.21.15.161:0/1036226576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71f8144840 msgr2=0x7f71f8144cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:10.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.589+0000 7f71ffc40640 1 --2- 172.21.15.161:0/1036226576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71f8144840 0x7f71f8144cc0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f71f4035d50 tx=0x7f71f4009750 comp rx=0 tx=0).stop 2024-03-28T23:35:10.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.589+0000 7f71ffc40640 1 -- 172.21.15.161:0/1036226576 shutdown_connections 2024-03-28T23:35:10.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.589+0000 7f71ffc40640 1 --2- 172.21.15.161:0/1036226576 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f71dc06b4d0 0x7f71dc06d990 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.589+0000 7f71ffc40640 1 --2- 172.21.15.161:0/1036226576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71f8144840 0x7f71f8144cc0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.589+0000 7f71ffc40640 1 --2- 172.21.15.161:0/1036226576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f71f80ac2f0 0x7f71f8072a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.589+0000 7f71ffc40640 1 --2- 172.21.15.161:0/1036226576 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f71e40051c0 0x7f71f81eca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.589+0000 7f71ffc40640 1 -- 172.21.15.161:0/1036226576 >> 172.21.15.161:0/1036226576 conn(0x7f71f80c3390 msgr2=0x7f71f806c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:10.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.589+0000 7f71ffc40640 1 -- 172.21.15.161:0/1036226576 shutdown_connections 2024-03-28T23:35:10.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.589+0000 7f71ffc40640 1 -- 172.21.15.161:0/1036226576 wait complete. 2024-03-28T23:35:10.601 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T23:35:10.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.701+0000 7f0bf2e0e640 1 Processor -- start 2024-03-28T23:35:10.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.701+0000 7f0bf2e0e640 1 -- start start 2024-03-28T23:35:10.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.702+0000 7f0bf2e0e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0bec1592e0 0x7f0bec0c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:10.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.702+0000 7f0bf2e0e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bec159730 con 0x7f0bec14f260 2024-03-28T23:35:10.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.702+0000 7f0bf2e0e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bec0cb4e0 con 0x7f0bec0caf60 2024-03-28T23:35:10.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.702+0000 7f0bf2e0e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0bec0c9080 con 0x7f0bec1592e0 2024-03-28T23:35:10.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.702+0000 7f0bf13ab640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0bec1592e0 0x7f0bec0c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:10.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.702+0000 7f0bf0baa640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f0bec14f260 0x7f0bec155c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45214/0 (socket says 172.21.15.161:45214) 2024-03-28T23:35:10.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.702+0000 7f0bf0baa640 1 -- 172.21.15.161:0/2378151309 learned_addr learned my addr 172.21.15.161:0/2378151309 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:10.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.703+0000 7f0bf13ab640 1 -- 172.21.15.161:0/2378151309 >> v1:172.21.15.170:6789/0 conn(0x7f0bec0caf60 legacy=0x7f0bec0c8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:10.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.703+0000 7f0beb7fe640 1 -- 172.21.15.161:0/2378151309 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4153679799 0 0) 0x7f0bec159730 con 0x7f0bec14f260 2024-03-28T23:35:10.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.703+0000 7f0bf13ab640 1 -- 172.21.15.161:0/2378151309 >> v1:172.21.15.161:6789/0 conn(0x7f0bec14f260 legacy=0x7f0bec155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:10.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.703+0000 7f0bf13ab640 1 -- 172.21.15.161:0/2378151309 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bec0cb7c0 con 0x7f0bec1592e0 2024-03-28T23:35:10.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.703+0000 7f0bf13ab640 1 --2- 172.21.15.161:0/2378151309 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0bec1592e0 0x7f0bec0c53f0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f0be4009d90 tx=0x7f0be402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0e6fb66756d33d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:10.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.704+0000 7f0beb7fe640 1 -- 172.21.15.161:0/2378151309 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be4039070 con 0x7f0bec1592e0 2024-03-28T23:35:10.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.704+0000 7f0beb7fe640 1 -- 172.21.15.161:0/2378151309 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0be402cbd0 con 0x7f0bec1592e0 2024-03-28T23:35:10.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.704+0000 7f0beb7fe640 1 -- 172.21.15.161:0/2378151309 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be4034440 con 0x7f0bec1592e0 2024-03-28T23:35:10.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.704+0000 7f0bf2e0e640 1 -- 172.21.15.161:0/2378151309 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0bec1592e0 msgr2=0x7f0bec0c53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:10.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.704+0000 7f0bf2e0e640 1 --2- 172.21.15.161:0/2378151309 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0bec1592e0 0x7f0bec0c53f0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f0be4009d90 tx=0x7f0be402be20 comp rx=0 tx=0).stop 2024-03-28T23:35:10.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.704+0000 7f0bf2e0e640 1 -- 172.21.15.161:0/2378151309 shutdown_connections 2024-03-28T23:35:10.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.704+0000 7f0bf2e0e640 1 --2- 172.21.15.161:0/2378151309 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0bec1592e0 0x7f0bec0c53f0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.704+0000 7f0bf2e0e640 1 -- 172.21.15.161:0/2378151309 >> 172.21.15.161:0/2378151309 conn(0x7f0bec072ca0 msgr2=0x7f0bec0730b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:10.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.704+0000 7f0bf2e0e640 1 -- 172.21.15.161:0/2378151309 shutdown_connections 2024-03-28T23:35:10.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.704+0000 7f0bf2e0e640 1 -- 172.21.15.161:0/2378151309 wait complete. 2024-03-28T23:35:10.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.705+0000 7f0bf2e0e640 1 Processor -- start 2024-03-28T23:35:10.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.705+0000 7f0bf2e0e640 1 -- start start 2024-03-28T23:35:10.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.706+0000 7f0bf2e0e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0bec0caf60 0x7f0bec0ad960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:10.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.706+0000 7f0bf2e0e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0bec14f260 0x7f0bec149be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:10.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.706+0000 7f0bf2e0e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0bec1592e0 0x7f0bec14a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:10.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.706+0000 7f0bf2e0e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0bec158ae0 con 0x7f0bec1592e0 2024-03-28T23:35:10.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.706+0000 7f0bf2e0e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0bec158e50 con 0x7f0bec0caf60 2024-03-28T23:35:10.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.706+0000 7f0bf2e0e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0bec14f640 con 0x7f0bec14f260 2024-03-28T23:35:10.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.706+0000 7f0bf13ab640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0bec1592e0 0x7f0bec14a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:10.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.706+0000 7f0bf13ab640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0bec1592e0 0x7f0bec14a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47410/0 (socket says 172.21.15.161:47410) 2024-03-28T23:35:10.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.706+0000 7f0bf13ab640 1 -- 172.21.15.161:0/1618830038 learned_addr learned my addr 172.21.15.161:0/1618830038 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:10.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.706+0000 7f0bf0baa640 1 --2- 172.21.15.161:0/1618830038 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0bec0caf60 0x7f0bec0ad960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:10.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.706+0000 7f0bebfff640 1 --2- 172.21.15.161:0/1618830038 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0bec14f260 0x7f0bec149be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:10.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.706+0000 7f0bf13ab640 1 -- 172.21.15.161:0/1618830038 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0bec14f260 msgr2=0x7f0bec149be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:10.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.706+0000 7f0bf13ab640 1 --2- 172.21.15.161:0/1618830038 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0bec14f260 0x7f0bec149be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.706+0000 7f0bf13ab640 1 -- 172.21.15.161:0/1618830038 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0bec0caf60 msgr2=0x7f0bec0ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:10.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.706+0000 7f0bf13ab640 1 --2- 172.21.15.161:0/1618830038 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0bec0caf60 0x7f0bec0ad960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.706+0000 7f0bf13ab640 1 -- 172.21.15.161:0/1618830038 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bec14a660 con 0x7f0bec1592e0 2024-03-28T23:35:10.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.706+0000 7f0bebfff640 1 --2- 172.21.15.161:0/1618830038 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0bec14f260 0x7f0bec149be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:10.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.707+0000 7f0bf13ab640 1 --2- 172.21.15.161:0/1618830038 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0bec1592e0 0x7f0bec14a120 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f0be4009d90 tx=0x7f0be4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:10.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.707+0000 7f0be97fa640 1 -- 172.21.15.161:0/1618830038 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be4044020 con 0x7f0bec1592e0 2024-03-28T23:35:10.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.707+0000 7f0be97fa640 1 -- 172.21.15.161:0/1618830038 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0be403f470 con 0x7f0bec1592e0 2024-03-28T23:35:10.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.707+0000 7f0bf2e0e640 1 -- 172.21.15.161:0/1618830038 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bec0c07c0 con 0x7f0bec1592e0 2024-03-28T23:35:10.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.707+0000 7f0be97fa640 1 -- 172.21.15.161:0/1618830038 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be4039040 con 0x7f0bec1592e0 2024-03-28T23:35:10.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.707+0000 7f0bf2e0e640 1 -- 172.21.15.161:0/1618830038 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bec0c0c10 con 0x7f0bec1592e0 2024-03-28T23:35:10.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.708+0000 7f0bf2e0e640 1 -- 172.21.15.161:0/1618830038 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bec04f550 con 0x7f0bec1592e0 2024-03-28T23:35:10.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.708+0000 7f0be97fa640 1 -- 172.21.15.161:0/1618830038 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0be4047020 con 0x7f0bec1592e0 2024-03-28T23:35:10.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.709+0000 7f0be97fa640 1 --2- 172.21.15.161:0/1618830038 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0bd006b4d0 0x7f0bd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:10.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.709+0000 7f0be97fa640 1 -- 172.21.15.161:0/1618830038 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5249+0+0 (secure 0 0 0) 0x7f0be40acad0 con 0x7f0bec1592e0 2024-03-28T23:35:10.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.709+0000 7f0bf0baa640 1 --2- 172.21.15.161:0/1618830038 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0bd006b4d0 0x7f0bd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:10.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.709+0000 7f0bf0baa640 1 --2- 172.21.15.161:0/1618830038 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0bd006b4d0 0x7f0bd006d990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f0be0000f60 tx=0x7f0be0007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:10.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.713+0000 7f0be97fa640 1 -- 172.21.15.161:0/1618830038 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0be404e030 con 0x7f0bec1592e0 2024-03-28T23:35:10.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.889+0000 7f0bf2e0e640 1 -- 172.21.15.161:0/1618830038 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f0bec06cfe0 con 0x7f0bec1592e0 2024-03-28T23:35:10.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.889+0000 7f0be97fa640 1 -- 172.21.15.161:0/1618830038 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v132) ==== 84+0+65 (secure 0 0 0) 0x7f0be407bba0 con 0x7f0bec1592e0 2024-03-28T23:35:10.889 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:10.890 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2024-03-28T23:35:10.890 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:10.890 INFO:teuthology.orchestra.run.smithi161.stdout: "poolnum": 1, 2024-03-28T23:35:10.890 INFO:teuthology.orchestra.run.smithi161.stdout: "poolname": ".mgr" 2024-03-28T23:35:10.890 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:10.890 INFO:teuthology.orchestra.run.smithi161.stdout:] 2024-03-28T23:35:10.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.891+0000 7f0bf2e0e640 1 -- 172.21.15.161:0/1618830038 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0bd006b4d0 msgr2=0x7f0bd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:10.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.891+0000 7f0bf2e0e640 1 --2- 172.21.15.161:0/1618830038 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0bd006b4d0 0x7f0bd006d990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f0be0000f60 tx=0x7f0be0007d50 comp rx=0 tx=0).stop 2024-03-28T23:35:10.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.891+0000 7f0bf2e0e640 1 -- 172.21.15.161:0/1618830038 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0bec1592e0 msgr2=0x7f0bec14a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:10.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.891+0000 7f0bf2e0e640 1 --2- 172.21.15.161:0/1618830038 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0bec1592e0 0x7f0bec14a120 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f0be4009d90 tx=0x7f0be4004a40 comp rx=0 tx=0).stop 2024-03-28T23:35:10.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.891+0000 7f0bf2e0e640 1 -- 172.21.15.161:0/1618830038 shutdown_connections 2024-03-28T23:35:10.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.891+0000 7f0bf2e0e640 1 --2- 172.21.15.161:0/1618830038 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0bd006b4d0 0x7f0bd006d990 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.891+0000 7f0bf2e0e640 1 --2- 172.21.15.161:0/1618830038 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0bec1592e0 0x7f0bec14a120 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.891+0000 7f0bf2e0e640 1 --2- 172.21.15.161:0/1618830038 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0bec14f260 0x7f0bec149be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.891+0000 7f0bf2e0e640 1 --2- 172.21.15.161:0/1618830038 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0bec0caf60 0x7f0bec0ad960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:10.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.891+0000 7f0bf2e0e640 1 -- 172.21.15.161:0/1618830038 >> 172.21.15.161:0/1618830038 conn(0x7f0bec072ca0 msgr2=0x7f0bec0c52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:10.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.891+0000 7f0bf2e0e640 1 -- 172.21.15.161:0/1618830038 shutdown_connections 2024-03-28T23:35:10.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:10.891+0000 7f0bf2e0e640 1 -- 172.21.15.161:0/1618830038 wait complete. 2024-03-28T23:35:10.902 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-28T23:35:10.903 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-28T23:35:11.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.007+0000 7ff2e253e640 1 Processor -- start 2024-03-28T23:35:11.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.007+0000 7ff2e253e640 1 -- start start 2024-03-28T23:35:11.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.008+0000 7ff2e253e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff2dc0ac430 0x7ff2dc152880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:11.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.008+0000 7ff2e253e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff2dc0c30c0 0x7ff2dc0c24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:11.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.008+0000 7ff2e253e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2dc0c2a10 0x7ff2dc0c19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:11.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.008+0000 7ff2e253e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff2dc152dc0 con 0x7ff2dc0c30c0 2024-03-28T23:35:11.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.008+0000 7ff2e253e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7ff2dc0c3540 con 0x7ff2dc0ac430 2024-03-28T23:35:11.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.008+0000 7ff2e253e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff2dc0c3820 con 0x7ff2dc0c2a10 2024-03-28T23:35:11.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.008+0000 7ff2db7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff2dc0c30c0 0x7ff2dc0c24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:11.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.008+0000 7ff2db7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff2dc0c30c0 0x7ff2dc0c24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47414/0 (socket says 172.21.15.161:47414) 2024-03-28T23:35:11.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.008+0000 7ff2db7fe640 1 -- 172.21.15.161:0/799868239 learned_addr learned my addr 172.21.15.161:0/799868239 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:11.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.008+0000 7ff2dbfff640 1 --2- 172.21.15.161:0/799868239 >> v2:172.21.15.170:3300/0 conn(0x7ff2dc0ac430 0x7ff2dc152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:11.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.008+0000 7ff2e0adb640 1 --2- 172.21.15.161:0/799868239 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2dc0c2a10 0x7ff2dc0c19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:11.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.008+0000 7ff2db7fe640 1 -- 172.21.15.161:0/799868239 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2dc0c2a10 msgr2=0x7ff2dc0c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:11.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.008+0000 7ff2db7fe640 1 --2- 172.21.15.161:0/799868239 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2dc0c2a10 0x7ff2dc0c19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:11.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.008+0000 7ff2db7fe640 1 -- 172.21.15.161:0/799868239 >> v2:172.21.15.170:3300/0 conn(0x7ff2dc0ac430 msgr2=0x7ff2dc152880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:11.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.008+0000 7ff2db7fe640 1 --2- 172.21.15.161:0/799868239 >> v2:172.21.15.170:3300/0 conn(0x7ff2dc0ac430 0x7ff2dc152880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:11.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.008+0000 7ff2db7fe640 1 -- 172.21.15.161:0/799868239 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2dc0c3b00 con 0x7ff2dc0c30c0 2024-03-28T23:35:11.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.008+0000 7ff2dbfff640 1 --2- 172.21.15.161:0/799868239 >> v2:172.21.15.170:3300/0 conn(0x7ff2dc0ac430 0x7ff2dc152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:11.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.009+0000 7ff2db7fe640 1 --2- 172.21.15.161:0/799868239 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff2dc0c30c0 0x7ff2dc0c24d0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc0a95f0 tx=0x7ff2d002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6f56af158acf51c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:11.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.009+0000 7ff2daffd640 1 -- 172.21.15.161:0/799868239 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d002e070 con 0x7ff2dc0c30c0 2024-03-28T23:35:11.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.009+0000 7ff2daffd640 1 -- 172.21.15.161:0/799868239 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2d0037070 con 0x7ff2dc0c30c0 2024-03-28T23:35:11.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.009+0000 7ff2daffd640 1 -- 172.21.15.161:0/799868239 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d0031a00 con 0x7ff2dc0c30c0 2024-03-28T23:35:11.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.010+0000 7ff2e253e640 1 -- 172.21.15.161:0/799868239 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff2dc0c30c0 msgr2=0x7ff2dc0c24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:11.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.010+0000 7ff2e253e640 1 --2- 172.21.15.161:0/799868239 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff2dc0c30c0 0x7ff2dc0c24d0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc0a95f0 tx=0x7ff2d002b880 comp rx=0 tx=0).stop 2024-03-28T23:35:11.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.010+0000 7ff2e253e640 1 -- 172.21.15.161:0/799868239 shutdown_connections 2024-03-28T23:35:11.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.010+0000 7ff2e253e640 1 --2- 172.21.15.161:0/799868239 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2dc0c2a10 0x7ff2dc0c19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:11.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.010+0000 7ff2e253e640 1 --2- 172.21.15.161:0/799868239 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff2dc0c30c0 0x7ff2dc0c24d0 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:11.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.010+0000 7ff2e253e640 1 --2- 172.21.15.161:0/799868239 >> v2:172.21.15.170:3300/0 conn(0x7ff2dc0ac430 0x7ff2dc152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:11.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.010+0000 7ff2e253e640 1 -- 172.21.15.161:0/799868239 >> 172.21.15.161:0/799868239 conn(0x7ff2dc06b150 msgr2=0x7ff2dc06b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:11.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.010+0000 7ff2e253e640 1 -- 172.21.15.161:0/799868239 shutdown_connections 2024-03-28T23:35:11.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.010+0000 7ff2e253e640 1 -- 172.21.15.161:0/799868239 wait complete. 2024-03-28T23:35:11.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.011+0000 7ff2e253e640 1 Processor -- start 2024-03-28T23:35:11.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.011+0000 7ff2e253e640 1 -- start start 2024-03-28T23:35:11.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.012+0000 7ff2e253e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff2dc0ac430 0x7ff2dc0bfe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:11.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.012+0000 7ff2e253e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2dc0c2a10 0x7ff2dc154240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:11.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.012+0000 7ff2e253e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2dc0c30c0 0x7ff2dc154780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:11.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.012+0000 7ff2e253e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff2dc0c6440 con 0x7ff2dc0ac430 2024-03-28T23:35:11.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.012+0000 7ff2e253e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff2dc0c1490 con 0x7ff2dc0c30c0 2024-03-28T23:35:11.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.012+0000 7ff2e253e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff2dc0ac810 con 0x7ff2dc0c2a10 2024-03-28T23:35:11.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.012+0000 7ff2dbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff2dc0ac430 0x7ff2dc0bfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:11.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.012+0000 7ff2dbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff2dc0ac430 0x7ff2dc0bfe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47416/0 (socket says 172.21.15.161:47416) 2024-03-28T23:35:11.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.012+0000 7ff2dbfff640 1 -- 172.21.15.161:0/330565574 learned_addr learned my addr 172.21.15.161:0/330565574 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:11.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.012+0000 7ff2e0adb640 1 --2- 172.21.15.161:0/330565574 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2dc0c30c0 0x7ff2dc154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:11.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.012+0000 7ff2db7fe640 1 --2- 172.21.15.161:0/330565574 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2dc0c2a10 0x7ff2dc154240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:11.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.012+0000 7ff2dbfff640 1 -- 172.21.15.161:0/330565574 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2dc0c2a10 msgr2=0x7ff2dc154240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:11.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.012+0000 7ff2dbfff640 1 --2- 172.21.15.161:0/330565574 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2dc0c2a10 0x7ff2dc154240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:11.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.012+0000 7ff2dbfff640 1 -- 172.21.15.161:0/330565574 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2dc0c30c0 msgr2=0x7ff2dc154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:11.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.012+0000 7ff2dbfff640 1 --2- 172.21.15.161:0/330565574 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2dc0c30c0 0x7ff2dc154780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:11.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.012+0000 7ff2dbfff640 1 -- 172.21.15.161:0/330565574 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2dc1eb820 con 0x7ff2dc0ac430 2024-03-28T23:35:11.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.013+0000 7ff2dbfff640 1 --2- 172.21.15.161:0/330565574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff2dc0ac430 0x7ff2dc0bfe50 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc00d800 tx=0x7ff2cc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:11.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.013+0000 7ff2d8ff9640 1 -- 172.21.15.161:0/330565574 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc018070 con 0x7ff2dc0ac430 2024-03-28T23:35:11.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.013+0000 7ff2d8ff9640 1 -- 172.21.15.161:0/330565574 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2cc004030 con 0x7ff2dc0ac430 2024-03-28T23:35:11.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.013+0000 7ff2e253e640 1 -- 172.21.15.161:0/330565574 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2dc0c0900 con 0x7ff2dc0ac430 2024-03-28T23:35:11.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.013+0000 7ff2d8ff9640 1 -- 172.21.15.161:0/330565574 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc004400 con 0x7ff2dc0ac430 2024-03-28T23:35:11.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.013+0000 7ff2e253e640 1 -- 172.21.15.161:0/330565574 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2dc0c0390 con 0x7ff2dc0ac430 2024-03-28T23:35:11.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.014+0000 7ff2e253e640 1 -- 172.21.15.161:0/330565574 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2dc068f90 con 0x7ff2dc0ac430 2024-03-28T23:35:11.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.014+0000 7ff2d8ff9640 1 -- 172.21.15.161:0/330565574 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff2cc0096a0 con 0x7ff2dc0ac430 2024-03-28T23:35:11.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.014+0000 7ff2d8ff9640 1 --2- 172.21.15.161:0/330565574 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff2c806b4d0 0x7ff2c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:11.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.015+0000 7ff2d8ff9640 1 -- 172.21.15.161:0/330565574 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5249+0+0 (secure 0 0 0) 0x7ff2cc08a380 con 0x7ff2dc0ac430 2024-03-28T23:35:11.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.015+0000 7ff2db7fe640 1 --2- 172.21.15.161:0/330565574 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff2c806b4d0 0x7ff2c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:11.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.015+0000 7ff2db7fe640 1 --2- 172.21.15.161:0/330565574 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff2c806b4d0 0x7ff2c806d990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7ff2d0036000 tx=0x7ff2d003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:11.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.019+0000 7ff2d8ff9640 1 -- 172.21.15.161:0/330565574 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff2cc059450 con 0x7ff2dc0ac430 2024-03-28T23:35:11.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.189+0000 7ff2e253e640 1 -- 172.21.15.161:0/330565574 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7ff2dc0c6220 con 0x7ff2dc0ac430 2024-03-28T23:35:11.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.623+0000 7ff2d8ff9640 1 -- 172.21.15.161:0/330565574 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v133) ==== 138+0+0 (secure 0 0 0) 0x7ff2cc058d50 con 0x7ff2dc0ac430 2024-03-28T23:35:11.624 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_metadata' created 2024-03-28T23:35:11.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.625+0000 7ff2e253e640 1 -- 172.21.15.161:0/330565574 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff2c806b4d0 msgr2=0x7ff2c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:11.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.625+0000 7ff2e253e640 1 --2- 172.21.15.161:0/330565574 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff2c806b4d0 0x7ff2c806d990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7ff2d0036000 tx=0x7ff2d003d040 comp rx=0 tx=0).stop 2024-03-28T23:35:11.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.625+0000 7ff2e253e640 1 -- 172.21.15.161:0/330565574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff2dc0ac430 msgr2=0x7ff2dc0bfe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:11.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.625+0000 7ff2e253e640 1 --2- 172.21.15.161:0/330565574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff2dc0ac430 0x7ff2dc0bfe50 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc00d800 tx=0x7ff2cc00dcc0 comp rx=0 tx=0).stop 2024-03-28T23:35:11.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.626+0000 7ff2e253e640 1 -- 172.21.15.161:0/330565574 shutdown_connections 2024-03-28T23:35:11.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.626+0000 7ff2e253e640 1 --2- 172.21.15.161:0/330565574 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff2c806b4d0 0x7ff2c806d990 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:11.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.626+0000 7ff2e253e640 1 --2- 172.21.15.161:0/330565574 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2dc0c30c0 0x7ff2dc154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:11.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.626+0000 7ff2e253e640 1 --2- 172.21.15.161:0/330565574 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2dc0c2a10 0x7ff2dc154240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:11.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.626+0000 7ff2e253e640 1 --2- 172.21.15.161:0/330565574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff2dc0ac430 0x7ff2dc0bfe50 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:11.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.626+0000 7ff2e253e640 1 -- 172.21.15.161:0/330565574 >> 172.21.15.161:0/330565574 conn(0x7ff2dc06b150 msgr2=0x7ff2dc06b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:11.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.627+0000 7ff2e253e640 1 -- 172.21.15.161:0/330565574 shutdown_connections 2024-03-28T23:35:11.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.627+0000 7ff2e253e640 1 -- 172.21.15.161:0/330565574 wait complete. 2024-03-28T23:35:11.641 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T23:35:11.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.749+0000 7f3b21be2640 1 Processor -- start 2024-03-28T23:35:11.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.749+0000 7f3b21be2640 1 -- start start 2024-03-28T23:35:11.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.749+0000 7f3b21be2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3b1c0c1040 0x7f3b1c1535b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:11.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.749+0000 7f3b21be2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b1c0a96f0 0x7f3b1c15d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:11.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.749+0000 7f3b21be2640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b1c0c2f50 con 0x7f3b1c150520 2024-03-28T23:35:11.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.750+0000 7f3b21be2640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f3b1c0a9c20 con 0x7f3b1c0c1040 2024-03-28T23:35:11.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.750+0000 7f3b21be2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3b1c150900 con 0x7f3b1c0a96f0 2024-03-28T23:35:11.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.750+0000 7f3b1affd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3b1c0c1040 0x7f3b1c1535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:11.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.750+0000 7f3b1affd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3b1c0c1040 0x7f3b1c1535b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47422/0 (socket says 172.21.15.161:47422) 2024-03-28T23:35:11.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.750+0000 7f3b1affd640 1 -- 172.21.15.161:0/3577950705 learned_addr learned my addr 172.21.15.161:0/3577950705 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:11.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.750+0000 7f3b1bfff640 1 --2- 172.21.15.161:0/3577950705 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b1c0a96f0 0x7f3b1c15d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:11.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.750+0000 7f3b1affd640 1 -- 172.21.15.161:0/3577950705 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b1c0a96f0 msgr2=0x7f3b1c15d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:11.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.750+0000 7f3b1affd640 1 --2- 172.21.15.161:0/3577950705 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b1c0a96f0 0x7f3b1c15d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:11.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.750+0000 7f3b1affd640 1 -- 172.21.15.161:0/3577950705 >> v1:172.21.15.170:6789/0 conn(0x7f3b1c150520 legacy=0x7f3b1c0ab280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:11.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.750+0000 7f3b1affd640 1 -- 172.21.15.161:0/3577950705 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b1c1530d0 con 0x7f3b1c0c1040 2024-03-28T23:35:11.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.750+0000 7f3b1bfff640 1 --2- 172.21.15.161:0/3577950705 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b1c0a96f0 0x7f3b1c15d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:11.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.751+0000 7f3b1affd640 1 --2- 172.21.15.161:0/3577950705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b1c0c1040 0x7f3b1c1535b0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f3b10009870 tx=0x7f3b1002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a207e4d260624e18 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:11.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.751+0000 7f3b1a7fc640 1 -- 172.21.15.161:0/3577950705 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b100317c0 con 0x7f3b1c0c1040 2024-03-28T23:35:11.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.751+0000 7f3b1a7fc640 1 -- 172.21.15.161:0/3577950705 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b10037070 con 0x7f3b1c0c1040 2024-03-28T23:35:11.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.751+0000 7f3b1a7fc640 1 -- 172.21.15.161:0/3577950705 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b10031b80 con 0x7f3b1c0c1040 2024-03-28T23:35:11.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.751+0000 7f3b21be2640 1 -- 172.21.15.161:0/3577950705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b1c0c1040 msgr2=0x7f3b1c1535b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:11.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.751+0000 7f3b21be2640 1 --2- 172.21.15.161:0/3577950705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b1c0c1040 0x7f3b1c1535b0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f3b10009870 tx=0x7f3b1002b860 comp rx=0 tx=0).stop 2024-03-28T23:35:11.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.751+0000 7f3b21be2640 1 -- 172.21.15.161:0/3577950705 shutdown_connections 2024-03-28T23:35:11.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.751+0000 7f3b21be2640 1 --2- 172.21.15.161:0/3577950705 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b1c0a96f0 0x7f3b1c15d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:11.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.751+0000 7f3b21be2640 1 --2- 172.21.15.161:0/3577950705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b1c0c1040 0x7f3b1c1535b0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:11.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.751+0000 7f3b21be2640 1 -- 172.21.15.161:0/3577950705 >> 172.21.15.161:0/3577950705 conn(0x7f3b1c0a83f0 msgr2=0x7f3b1c0a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:11.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.751+0000 7f3b21be2640 1 -- 172.21.15.161:0/3577950705 shutdown_connections 2024-03-28T23:35:11.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.751+0000 7f3b21be2640 1 -- 172.21.15.161:0/3577950705 wait complete. 2024-03-28T23:35:11.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.752+0000 7f3b21be2640 1 Processor -- start 2024-03-28T23:35:11.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.753+0000 7f3b21be2640 1 -- start start 2024-03-28T23:35:11.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.753+0000 7f3b21be2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b1c0a96f0 0x7f3b1c1ea1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:11.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.753+0000 7f3b21be2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b1c0c1040 0x7f3b1c1558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:11.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.753+0000 7f3b21be2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b1c150520 0x7f3b1c1ec2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:11.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.753+0000 7f3b21be2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3b1c0ab990 con 0x7f3b1c0a96f0 2024-03-28T23:35:11.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.753+0000 7f3b21be2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3b1c0c2c70 con 0x7f3b1c150520 2024-03-28T23:35:11.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.753+0000 7f3b21be2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3b1c0abb10 con 0x7f3b1c0c1040 2024-03-28T23:35:11.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.753+0000 7f3b1b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b1c0a96f0 0x7f3b1c1ea1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:11.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.753+0000 7f3b1b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b1c0a96f0 0x7f3b1c1ea1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47428/0 (socket says 172.21.15.161:47428) 2024-03-28T23:35:11.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.753+0000 7f3b1b7fe640 1 -- 172.21.15.161:0/1221283161 learned_addr learned my addr 172.21.15.161:0/1221283161 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:11.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.754+0000 7f3b1affd640 1 --2- 172.21.15.161:0/1221283161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b1c0c1040 0x7f3b1c1558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:11.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.754+0000 7f3b1bfff640 1 --2- 172.21.15.161:0/1221283161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b1c150520 0x7f3b1c1ec2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:11.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.754+0000 7f3b1b7fe640 1 -- 172.21.15.161:0/1221283161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b1c0c1040 msgr2=0x7f3b1c1558e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:11.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.754+0000 7f3b1b7fe640 1 --2- 172.21.15.161:0/1221283161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b1c0c1040 0x7f3b1c1558e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:11.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.754+0000 7f3b1b7fe640 1 -- 172.21.15.161:0/1221283161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b1c150520 msgr2=0x7f3b1c1ec2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:11.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.754+0000 7f3b1b7fe640 1 --2- 172.21.15.161:0/1221283161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b1c150520 0x7f3b1c1ec2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:11.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.754+0000 7f3b1b7fe640 1 -- 172.21.15.161:0/1221283161 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b1c1ea6e0 con 0x7f3b1c0a96f0 2024-03-28T23:35:11.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.754+0000 7f3b1b7fe640 1 --2- 172.21.15.161:0/1221283161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b1c0a96f0 0x7f3b1c1ea1a0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c00c410 tx=0x7f3b0c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:11.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.754+0000 7f3b03fff640 1 -- 172.21.15.161:0/1221283161 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c017070 con 0x7f3b1c0a96f0 2024-03-28T23:35:11.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.754+0000 7f3b03fff640 1 -- 172.21.15.161:0/1221283161 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b0c0035f0 con 0x7f3b1c0a96f0 2024-03-28T23:35:11.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.754+0000 7f3b03fff640 1 -- 172.21.15.161:0/1221283161 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c0039e0 con 0x7f3b1c0a96f0 2024-03-28T23:35:11.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.754+0000 7f3b21be2640 1 -- 172.21.15.161:0/1221283161 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b1c1ecf00 con 0x7f3b1c0a96f0 2024-03-28T23:35:11.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.754+0000 7f3b21be2640 1 -- 172.21.15.161:0/1221283161 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b1c1553c0 con 0x7f3b1c0a96f0 2024-03-28T23:35:11.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.755+0000 7f3b21be2640 1 -- 172.21.15.161:0/1221283161 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b1c04f4b0 con 0x7f3b1c0a96f0 2024-03-28T23:35:11.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.756+0000 7f3b03fff640 1 -- 172.21.15.161:0/1221283161 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3b0c003b80 con 0x7f3b1c0a96f0 2024-03-28T23:35:11.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.756+0000 7f3b03fff640 1 --2- 172.21.15.161:0/1221283161 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3b0806b4d0 0x7f3b0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:11.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.757+0000 7f3b03fff640 1 -- 172.21.15.161:0/1221283161 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5619+0+0 (secure 0 0 0) 0x7f3b0c0894e0 con 0x7f3b1c0a96f0 2024-03-28T23:35:11.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.757+0000 7f3b1affd640 1 --2- 172.21.15.161:0/1221283161 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3b0806b4d0 0x7f3b0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:11.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.757+0000 7f3b1affd640 1 --2- 172.21.15.161:0/1221283161 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3b0806b4d0 0x7f3b0806d990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3b10036000 tx=0x7f3b1003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:11.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.761+0000 7f3b03fff640 1 -- 172.21.15.161:0/1221283161 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3b0c058440 con 0x7f3b1c0a96f0 2024-03-28T23:35:11.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:11.936+0000 7f3b21be2640 1 -- 172.21.15.161:0/1221283161 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f3b1c0c2640 con 0x7f3b1c0a96f0 2024-03-28T23:35:12.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.625+0000 7f3b03fff640 1 -- 172.21.15.161:0/1221283161 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v134) ==== 170+0+0 (secure 0 0 0) 0x7f3b0c0010c0 con 0x7f3b1c0a96f0 2024-03-28T23:35:12.626 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data' created 2024-03-28T23:35:12.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.629+0000 7f3b21be2640 1 -- 172.21.15.161:0/1221283161 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3b0806b4d0 msgr2=0x7f3b0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:12.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.629+0000 7f3b21be2640 1 --2- 172.21.15.161:0/1221283161 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3b0806b4d0 0x7f3b0806d990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3b10036000 tx=0x7f3b1003d040 comp rx=0 tx=0).stop 2024-03-28T23:35:12.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.629+0000 7f3b21be2640 1 -- 172.21.15.161:0/1221283161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b1c0a96f0 msgr2=0x7f3b1c1ea1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:12.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.629+0000 7f3b21be2640 1 --2- 172.21.15.161:0/1221283161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b1c0a96f0 0x7f3b1c1ea1a0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c00c410 tx=0x7f3b0c00c8d0 comp rx=0 tx=0).stop 2024-03-28T23:35:12.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.629+0000 7f3b21be2640 1 -- 172.21.15.161:0/1221283161 shutdown_connections 2024-03-28T23:35:12.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.629+0000 7f3b21be2640 1 --2- 172.21.15.161:0/1221283161 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3b0806b4d0 0x7f3b0806d990 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:12.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.629+0000 7f3b21be2640 1 --2- 172.21.15.161:0/1221283161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b1c150520 0x7f3b1c1ec2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:12.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.629+0000 7f3b21be2640 1 --2- 172.21.15.161:0/1221283161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b1c0c1040 0x7f3b1c1558e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:12.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.629+0000 7f3b21be2640 1 --2- 172.21.15.161:0/1221283161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b1c0a96f0 0x7f3b1c1ea1a0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:12.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.629+0000 7f3b21be2640 1 -- 172.21.15.161:0/1221283161 >> 172.21.15.161:0/1221283161 conn(0x7f3b1c0a83f0 msgr2=0x7f3b1c14e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:12.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.632+0000 7f3b21be2640 1 -- 172.21.15.161:0/1221283161 shutdown_connections 2024-03-28T23:35:12.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.632+0000 7f3b21be2640 1 -- 172.21.15.161:0/1221283161 wait complete. 2024-03-28T23:35:12.655 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-28T23:35:12.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.760+0000 7f4caba8a640 1 Processor -- start 2024-03-28T23:35:12.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.760+0000 7f4caba8a640 1 -- start start 2024-03-28T23:35:12.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.761+0000 7f4caba8a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4ca40c7330 0x7f4ca414e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:12.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.761+0000 7f4caba8a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ca40abc00 con 0x7f4ca40a9960 2024-03-28T23:35:12.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.761+0000 7f4caba8a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ca406bf70 con 0x7f4ca40733e0 2024-03-28T23:35:12.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.761+0000 7f4caba8a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f4ca40aa3a0 con 0x7f4ca40c7330 2024-03-28T23:35:12.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.761+0000 7f4ca9826640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f4ca40a9960 0x7f4ca4072970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45224/0 (socket says 172.21.15.161:45224) 2024-03-28T23:35:12.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.761+0000 7f4ca9826640 1 -- 172.21.15.161:0/357824520 learned_addr learned my addr 172.21.15.161:0/357824520 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:12.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.761+0000 7f4ca9025640 1 --2- 172.21.15.161:0/357824520 >> v2:172.21.15.170:3300/0 conn(0x7f4ca40c7330 0x7f4ca414e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:12.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.761+0000 7f4ca8824640 1 -- 172.21.15.161:0/357824520 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 310706535 0 0) 0x7f4ca40abc00 con 0x7f4ca40a9960 2024-03-28T23:35:12.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.761+0000 7f4ca8824640 1 -- 172.21.15.161:0/357824520 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c90004630 con 0x7f4ca40a9960 2024-03-28T23:35:12.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.761+0000 7f4ca8824640 1 -- 172.21.15.161:0/357824520 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4118707741 0 0) 0x7f4c90004630 con 0x7f4ca40a9960 2024-03-28T23:35:12.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.761+0000 7f4ca8824640 1 -- 172.21.15.161:0/357824520 >> v2:172.21.15.170:3300/0 conn(0x7f4ca40c7330 msgr2=0x7f4ca414e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:12.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.761+0000 7f4ca8824640 1 --2- 172.21.15.161:0/357824520 >> v2:172.21.15.170:3300/0 conn(0x7f4ca40c7330 0x7f4ca414e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:12.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.761+0000 7f4ca8824640 1 -- 172.21.15.161:0/357824520 >> v1:172.21.15.170:6789/0 conn(0x7f4ca40733e0 legacy=0x7f4ca414e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:12.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.761+0000 7f4ca8824640 1 -- 172.21.15.161:0/357824520 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca406bcd0 con 0x7f4ca40a9960 2024-03-28T23:35:12.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.761+0000 7f4ca8824640 1 -- 172.21.15.161:0/357824520 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f4ca0001670 con 0x7f4ca40a9960 2024-03-28T23:35:12.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.761+0000 7f4ca8824640 1 -- 172.21.15.161:0/357824520 >> v1:172.21.15.161:6789/0 conn(0x7f4ca40a9960 legacy=0x7f4ca4072970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:12.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.761+0000 7f4ca8824640 1 --2- 172.21.15.161:0/357824520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c90004ba0 0x7f4c90004fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:12.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.761+0000 7f4ca8824640 1 --2- 172.21.15.161:0/357824520 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c90005910 0x7f4c90005d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:12.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.761+0000 7f4ca8824640 1 --2- 172.21.15.161:0/357824520 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c900064d0 0x7f4c9000cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:12.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.761+0000 7f4ca8824640 1 -- 172.21.15.161:0/357824520 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9000d360 con 0x7f4c90005910 2024-03-28T23:35:12.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.762+0000 7f4ca8824640 1 -- 172.21.15.161:0/357824520 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9000d510 con 0x7f4c90004ba0 2024-03-28T23:35:12.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.762+0000 7f4ca8824640 1 -- 172.21.15.161:0/357824520 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4c9000d710 con 0x7f4c900064d0 2024-03-28T23:35:12.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.762+0000 7f4ca9025640 1 --2- 172.21.15.161:0/357824520 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c90005910 0x7f4c90005d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:12.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.762+0000 7f4ca9826640 1 --2- 172.21.15.161:0/357824520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c90004ba0 0x7f4c90004fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:12.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.762+0000 7f4caa027640 1 --2- 172.21.15.161:0/357824520 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c900064d0 0x7f4c9000cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:12.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.762+0000 7f4ca9025640 1 -- 172.21.15.161:0/357824520 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c900064d0 msgr2=0x7f4c9000cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:12.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.762+0000 7f4ca9025640 1 --2- 172.21.15.161:0/357824520 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c900064d0 0x7f4c9000cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:12.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.762+0000 7f4ca9025640 1 -- 172.21.15.161:0/357824520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c90004ba0 msgr2=0x7f4c90004fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:12.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.762+0000 7f4ca9025640 1 --2- 172.21.15.161:0/357824520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c90004ba0 0x7f4c90004fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:12.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.762+0000 7f4caa027640 1 -- 172.21.15.161:0/357824520 reap_dead start 2024-03-28T23:35:12.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.762+0000 7f4ca9025640 1 -- 172.21.15.161:0/357824520 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c90004630 con 0x7f4c90005910 2024-03-28T23:35:12.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.762+0000 7f4ca9826640 1 --2- 172.21.15.161:0/357824520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c90004ba0 0x7f4c90004fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:12.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.763+0000 7f4ca9025640 1 --2- 172.21.15.161:0/357824520 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c90005910 0x7f4c90005d90 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f4c94009a90 tx=0x7f4c90006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e80f0becc78bbbf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:12.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.763+0000 7f4ca8824640 1 -- 172.21.15.161:0/357824520 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c94039070 con 0x7f4c90005910 2024-03-28T23:35:12.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.763+0000 7f4ca8824640 1 -- 172.21.15.161:0/357824520 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c9402c6b0 con 0x7f4c90005910 2024-03-28T23:35:12.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.763+0000 7f4ca8824640 1 -- 172.21.15.161:0/357824520 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c94009690 con 0x7f4c90005910 2024-03-28T23:35:12.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.763+0000 7f4caba8a640 1 -- 172.21.15.161:0/357824520 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c90005910 msgr2=0x7f4c90005d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:12.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.763+0000 7f4caba8a640 1 --2- 172.21.15.161:0/357824520 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c90005910 0x7f4c90005d90 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f4c94009a90 tx=0x7f4c90006940 comp rx=0 tx=0).stop 2024-03-28T23:35:12.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.763+0000 7f4caba8a640 1 -- 172.21.15.161:0/357824520 shutdown_connections 2024-03-28T23:35:12.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.763+0000 7f4caba8a640 1 --2- 172.21.15.161:0/357824520 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c90005910 0x7f4c90005d90 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:12.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.763+0000 7f4caba8a640 1 -- 172.21.15.161:0/357824520 >> 172.21.15.161:0/357824520 conn(0x7f4ca40a83f0 msgr2=0x7f4ca40a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:12.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.764+0000 7f4caba8a640 1 -- 172.21.15.161:0/357824520 shutdown_connections 2024-03-28T23:35:12.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.764+0000 7f4caba8a640 1 -- 172.21.15.161:0/357824520 wait complete. 2024-03-28T23:35:12.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.765+0000 7f4caba8a640 1 Processor -- start 2024-03-28T23:35:12.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.765+0000 7f4caba8a640 1 -- start start 2024-03-28T23:35:12.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.765+0000 7f4caba8a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c90005910 0x7f4ca4150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:12.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.765+0000 7f4caba8a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ca414aca0 0x7f4ca4147760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:12.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.765+0000 7f4caba8a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ca41e9d50 0x7f4ca41ea1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:12.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.765+0000 7f4caba8a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca40a9d40 con 0x7f4c90005910 2024-03-28T23:35:12.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.765+0000 7f4caba8a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca4152250 con 0x7f4ca41e9d50 2024-03-28T23:35:12.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.765+0000 7f4caba8a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4ca40a9ec0 con 0x7f4ca414aca0 2024-03-28T23:35:12.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.765+0000 7f4ca9826640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c90005910 0x7f4ca4150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:12.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.765+0000 7f4ca9826640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c90005910 0x7f4ca4150f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47458/0 (socket says 172.21.15.161:47458) 2024-03-28T23:35:12.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.765+0000 7f4ca9826640 1 -- 172.21.15.161:0/4226844368 learned_addr learned my addr 172.21.15.161:0/4226844368 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:12.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.766+0000 7f4caa027640 1 --2- 172.21.15.161:0/4226844368 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ca41e9d50 0x7f4ca41ea1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:12.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.766+0000 7f4ca9025640 1 --2- 172.21.15.161:0/4226844368 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ca414aca0 0x7f4ca4147760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:12.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.766+0000 7f4ca9826640 1 -- 172.21.15.161:0/4226844368 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ca414aca0 msgr2=0x7f4ca4147760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:12.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.766+0000 7f4ca9826640 1 --2- 172.21.15.161:0/4226844368 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ca414aca0 0x7f4ca4147760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:12.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.766+0000 7f4ca9826640 1 -- 172.21.15.161:0/4226844368 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ca41e9d50 msgr2=0x7f4ca41ea1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:12.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.766+0000 7f4ca9826640 1 --2- 172.21.15.161:0/4226844368 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ca41e9d50 0x7f4ca41ea1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:12.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.766+0000 7f4ca9826640 1 -- 172.21.15.161:0/4226844368 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca41dc4d0 con 0x7f4c90005910 2024-03-28T23:35:12.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.766+0000 7f4ca9025640 1 --2- 172.21.15.161:0/4226844368 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ca414aca0 0x7f4ca4147760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:12.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.766+0000 7f4ca9826640 1 --2- 172.21.15.161:0/4226844368 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c90005910 0x7f4ca4150f90 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f4ca00072f0 tx=0x7f4ca00077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:12.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.766+0000 7f4c9a7fc640 1 -- 172.21.15.161:0/4226844368 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca001a070 con 0x7f4c90005910 2024-03-28T23:35:12.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.766+0000 7f4c9a7fc640 1 -- 172.21.15.161:0/4226844368 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ca000f960 con 0x7f4c90005910 2024-03-28T23:35:12.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.766+0000 7f4caba8a640 1 -- 172.21.15.161:0/4226844368 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ca41dc7b0 con 0x7f4c90005910 2024-03-28T23:35:12.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.766+0000 7f4c9a7fc640 1 -- 172.21.15.161:0/4226844368 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca0007cf0 con 0x7f4c90005910 2024-03-28T23:35:12.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.766+0000 7f4caba8a640 1 -- 172.21.15.161:0/4226844368 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ca4147ca0 con 0x7f4c90005910 2024-03-28T23:35:12.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.767+0000 7f4caba8a640 1 -- 172.21.15.161:0/4226844368 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ca414f510 con 0x7f4c90005910 2024-03-28T23:35:12.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.767+0000 7f4c9a7fc640 1 -- 172.21.15.161:0/4226844368 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4ca001d070 con 0x7f4c90005910 2024-03-28T23:35:12.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.768+0000 7f4c9a7fc640 1 --2- 172.21.15.161:0/4226844368 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4c8806b4d0 0x7f4c8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:12.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.768+0000 7f4c9a7fc640 1 -- 172.21.15.161:0/4226844368 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 6001+0+0 (secure 0 0 0) 0x7f4ca0089bd0 con 0x7f4c90005910 2024-03-28T23:35:12.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.768+0000 7f4ca9025640 1 --2- 172.21.15.161:0/4226844368 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4c8806b4d0 0x7f4c8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:12.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.768+0000 7f4ca9025640 1 --2- 172.21.15.161:0/4226844368 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4c8806b4d0 0x7f4c8806d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f4c94036000 tx=0x7f4c94030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:12.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.773+0000 7f4c9a7fc640 1 -- 172.21.15.161:0/4226844368 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4ca00589b0 con 0x7f4c90005910 2024-03-28T23:35:12.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:12.981+0000 7f4caba8a640 1 -- 172.21.15.161:0/4226844368 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f4ca404f4b0 con 0x7f4c90005910 2024-03-28T23:35:13.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.645+0000 7f4c9a7fc640 1 -- 172.21.15.161:0/4226844368 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 21 and data pool 22 v121) ==== 174+0+0 (secure 0 0 0) 0x7f4ca00582b0 con 0x7f4c90005910 2024-03-28T23:35:13.646 INFO:teuthology.orchestra.run.smithi161.stderr:new fs with metadata pool 21 and data pool 22 2024-03-28T23:35:13.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.651+0000 7f4caba8a640 1 -- 172.21.15.161:0/4226844368 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4c8806b4d0 msgr2=0x7f4c8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:13.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.651+0000 7f4caba8a640 1 --2- 172.21.15.161:0/4226844368 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4c8806b4d0 0x7f4c8806d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f4c94036000 tx=0x7f4c94030000 comp rx=0 tx=0).stop 2024-03-28T23:35:13.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.651+0000 7f4caba8a640 1 -- 172.21.15.161:0/4226844368 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c90005910 msgr2=0x7f4ca4150f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:13.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.651+0000 7f4caba8a640 1 --2- 172.21.15.161:0/4226844368 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c90005910 0x7f4ca4150f90 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f4ca00072f0 tx=0x7f4ca00077b0 comp rx=0 tx=0).stop 2024-03-28T23:35:13.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.651+0000 7f4caba8a640 1 -- 172.21.15.161:0/4226844368 shutdown_connections 2024-03-28T23:35:13.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.651+0000 7f4caba8a640 1 --2- 172.21.15.161:0/4226844368 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4c8806b4d0 0x7f4c8806d990 secure :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f4c94036000 tx=0x7f4c94030000 comp rx=0 tx=0).stop 2024-03-28T23:35:13.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.651+0000 7f4caba8a640 1 --2- 172.21.15.161:0/4226844368 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ca41e9d50 0x7f4ca41ea1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:13.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.651+0000 7f4caba8a640 1 --2- 172.21.15.161:0/4226844368 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ca414aca0 0x7f4ca4147760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:13.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.651+0000 7f4caba8a640 1 --2- 172.21.15.161:0/4226844368 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c90005910 0x7f4ca4150f90 secure :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f4ca00072f0 tx=0x7f4ca00077b0 comp rx=0 tx=0).stop 2024-03-28T23:35:13.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.651+0000 7f4caba8a640 1 -- 172.21.15.161:0/4226844368 >> 172.21.15.161:0/4226844368 conn(0x7f4ca40a83f0 msgr2=0x7f4ca4070470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:13.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.655+0000 7f4caba8a640 1 -- 172.21.15.161:0/4226844368 shutdown_connections 2024-03-28T23:35:13.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.656+0000 7f4caba8a640 1 -- 172.21.15.161:0/4226844368 wait complete. 2024-03-28T23:35:13.675 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:35:13.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.777+0000 7f3abbc9d640 1 Processor -- start 2024-03-28T23:35:13.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.777+0000 7f3abbc9d640 1 -- start start 2024-03-28T23:35:13.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.778+0000 7f3abbc9d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3ab414eca0 0x7f3ab40c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:13.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.778+0000 7f3abbc9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ab40c1dc0 0x7f3ab406ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:13.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.778+0000 7f3abbc9d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ab4151850 con 0x7f3ab406cf60 2024-03-28T23:35:13.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.778+0000 7f3abbc9d640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f3ab4152460 con 0x7f3ab414eca0 2024-03-28T23:35:13.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.778+0000 7f3abbc9d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3ab4151b30 con 0x7f3ab40c1dc0 2024-03-28T23:35:13.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.778+0000 7f3aba23a640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f3ab406cf60 0x7f3ab40a98c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45238/0 (socket says 172.21.15.161:45238) 2024-03-28T23:35:13.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.778+0000 7f3aba23a640 1 -- 172.21.15.161:0/181252470 learned_addr learned my addr 172.21.15.161:0/181252470 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:13.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.778+0000 7f3ab9a39640 1 --2- 172.21.15.161:0/181252470 >> v2:172.21.15.170:3300/0 conn(0x7f3ab414eca0 0x7f3ab40c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:13.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.778+0000 7f3ab9238640 1 --2- 172.21.15.161:0/181252470 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ab40c1dc0 0x7f3ab406ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:13.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.778+0000 7f3ab8a37640 1 -- 172.21.15.161:0/181252470 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2957844261 0 0) 0x7f3ab4151850 con 0x7f3ab406cf60 2024-03-28T23:35:13.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.778+0000 7f3ab8a37640 1 -- 172.21.15.161:0/181252470 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3aa0004630 con 0x7f3ab406cf60 2024-03-28T23:35:13.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.778+0000 7f3ab9a39640 1 -- 172.21.15.161:0/181252470 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ab40c1dc0 msgr2=0x7f3ab406ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:13.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.778+0000 7f3ab9a39640 1 --2- 172.21.15.161:0/181252470 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ab40c1dc0 0x7f3ab406ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:13.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.778+0000 7f3ab9a39640 1 -- 172.21.15.161:0/181252470 >> v1:172.21.15.161:6789/0 conn(0x7f3ab406cf60 legacy=0x7f3ab40a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:13.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.778+0000 7f3ab8a37640 1 -- 172.21.15.161:0/181252470 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1267126086 0 0) 0x7f3aa0004630 con 0x7f3ab406cf60 2024-03-28T23:35:13.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.779+0000 7f3ab9a39640 1 -- 172.21.15.161:0/181252470 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ab4151e10 con 0x7f3ab414eca0 2024-03-28T23:35:13.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.779+0000 7f3ab9a39640 1 --2- 172.21.15.161:0/181252470 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ab414eca0 0x7f3ab40c1880 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3aa40097a0 tx=0x7f3aa402b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f0e364f3fe8d02d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:13.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.780+0000 7f3ab8a37640 1 -- 172.21.15.161:0/181252470 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa4037070 con 0x7f3ab414eca0 2024-03-28T23:35:13.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.780+0000 7f3ab8a37640 1 -- 172.21.15.161:0/181252470 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3aa4030510 con 0x7f3ab414eca0 2024-03-28T23:35:13.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.780+0000 7f3ab8a37640 1 -- 172.21.15.161:0/181252470 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aac0019f0 con 0x7f3ab414eca0 2024-03-28T23:35:13.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.780+0000 7f3abbc9d640 1 -- 172.21.15.161:0/181252470 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ab414eca0 msgr2=0x7f3ab40c1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:13.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.780+0000 7f3abbc9d640 1 --2- 172.21.15.161:0/181252470 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ab414eca0 0x7f3ab40c1880 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3aa40097a0 tx=0x7f3aa402b710 comp rx=0 tx=0).stop 2024-03-28T23:35:13.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.780+0000 7f3abbc9d640 1 -- 172.21.15.161:0/181252470 shutdown_connections 2024-03-28T23:35:13.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.780+0000 7f3abbc9d640 1 --2- 172.21.15.161:0/181252470 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ab40c1dc0 0x7f3ab406ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:13.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.780+0000 7f3abbc9d640 1 --2- 172.21.15.161:0/181252470 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ab414eca0 0x7f3ab40c1880 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:13.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.780+0000 7f3abbc9d640 1 -- 172.21.15.161:0/181252470 >> 172.21.15.161:0/181252470 conn(0x7f3ab40c6670 msgr2=0x7f3ab40c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:13.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.780+0000 7f3abbc9d640 1 -- 172.21.15.161:0/181252470 shutdown_connections 2024-03-28T23:35:13.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.780+0000 7f3abbc9d640 1 -- 172.21.15.161:0/181252470 wait complete. 2024-03-28T23:35:13.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.781+0000 7f3abbc9d640 1 Processor -- start 2024-03-28T23:35:13.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.781+0000 7f3abbc9d640 1 -- start start 2024-03-28T23:35:13.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.782+0000 7f3abbc9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ab406cf60 0x7f3ab41dbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:13.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.782+0000 7f3abbc9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ab40c1dc0 0x7f3ab41ec410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:13.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.782+0000 7f3abbc9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ab414eca0 0x7f3ab4148680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:13.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.782+0000 7f3abbc9d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab41514b0 con 0x7f3ab40c1dc0 2024-03-28T23:35:13.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.782+0000 7f3abbc9d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab414f080 con 0x7f3ab414eca0 2024-03-28T23:35:13.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.782+0000 7f3abbc9d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3ab4151630 con 0x7f3ab406cf60 2024-03-28T23:35:13.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.782+0000 7f3ab9238640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ab40c1dc0 0x7f3ab41ec410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:13.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.782+0000 7f3ab9238640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ab40c1dc0 0x7f3ab41ec410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47470/0 (socket says 172.21.15.161:47470) 2024-03-28T23:35:13.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.782+0000 7f3ab9238640 1 -- 172.21.15.161:0/890177262 learned_addr learned my addr 172.21.15.161:0/890177262 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:13.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.782+0000 7f3ab9a39640 1 --2- 172.21.15.161:0/890177262 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ab406cf60 0x7f3ab41dbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:13.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.782+0000 7f3aba23a640 1 --2- 172.21.15.161:0/890177262 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ab414eca0 0x7f3ab4148680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:13.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.782+0000 7f3ab9238640 1 -- 172.21.15.161:0/890177262 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ab406cf60 msgr2=0x7f3ab41dbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:13.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.782+0000 7f3ab9238640 1 --2- 172.21.15.161:0/890177262 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ab406cf60 0x7f3ab41dbbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:13.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.782+0000 7f3ab9238640 1 -- 172.21.15.161:0/890177262 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ab414eca0 msgr2=0x7f3ab4148680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:13.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.782+0000 7f3ab9238640 1 --2- 172.21.15.161:0/890177262 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ab414eca0 0x7f3ab4148680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:13.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.782+0000 7f3ab9238640 1 -- 172.21.15.161:0/890177262 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ab41551d0 con 0x7f3ab40c1dc0 2024-03-28T23:35:13.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.782+0000 7f3aba23a640 1 --2- 172.21.15.161:0/890177262 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ab414eca0 0x7f3ab4148680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:13.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.782+0000 7f3ab9a39640 1 --2- 172.21.15.161:0/890177262 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ab406cf60 0x7f3ab41dbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:13.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.782+0000 7f3ab9238640 1 --2- 172.21.15.161:0/890177262 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ab40c1dc0 0x7f3ab41ec410 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f3ab000e9e0 tx=0x7f3ab000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:13.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.783+0000 7f3aaa7fc640 1 -- 172.21.15.161:0/890177262 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0017070 con 0x7f3ab40c1dc0 2024-03-28T23:35:13.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.783+0000 7f3aaa7fc640 1 -- 172.21.15.161:0/890177262 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ab000c9d0 con 0x7f3ab40c1dc0 2024-03-28T23:35:13.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.783+0000 7f3aaa7fc640 1 -- 172.21.15.161:0/890177262 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab000cdc0 con 0x7f3ab40c1dc0 2024-03-28T23:35:13.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.783+0000 7f3abbc9d640 1 -- 172.21.15.161:0/890177262 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ab41554b0 con 0x7f3ab40c1dc0 2024-03-28T23:35:13.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.783+0000 7f3abbc9d640 1 -- 172.21.15.161:0/890177262 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ab41ebef0 con 0x7f3ab40c1dc0 2024-03-28T23:35:13.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.784+0000 7f3abbc9d640 1 -- 172.21.15.161:0/890177262 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ab404f4b0 con 0x7f3ab40c1dc0 2024-03-28T23:35:13.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.784+0000 7f3aaa7fc640 1 -- 172.21.15.161:0/890177262 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3ab001a070 con 0x7f3ab40c1dc0 2024-03-28T23:35:13.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.784+0000 7f3aaa7fc640 1 --2- 172.21.15.161:0/890177262 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3a9806b500 0x7f3a9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:13.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.784+0000 7f3aaa7fc640 1 -- 172.21.15.161:0/890177262 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 6101+0+0 (secure 0 0 0) 0x7f3ab008a2f0 con 0x7f3ab40c1dc0 2024-03-28T23:35:13.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.784+0000 7f3ab9a39640 1 --2- 172.21.15.161:0/890177262 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3a9806b500 0x7f3a9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:13.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.785+0000 7f3ab9a39640 1 --2- 172.21.15.161:0/890177262 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3a9806b500 0x7f3a9806d9c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4034000 tx=0x7f3aa4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:13.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.789+0000 7f3aaa7fc640 1 -- 172.21.15.161:0/890177262 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3ab0059070 con 0x7f3ab40c1dc0 2024-03-28T23:35:13.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.961+0000 7f3abbc9d640 1 -- 172.21.15.161:0/890177262 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3ab406c0c0 con 0x7f3ab40c1dc0 2024-03-28T23:35:13.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.962+0000 7f3aaa7fc640 1 -- 172.21.15.161:0/890177262 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) ==== 74+0+16941 (secure 0 0 0) 0x7f3ab0058970 con 0x7f3ab40c1dc0 2024-03-28T23:35:13.963 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:13.963 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":135,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:35:12.982251+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:35:11.208674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-28T23:35:11.954908+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:35:13.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.964+0000 7f3abbc9d640 1 -- 172.21.15.161:0/890177262 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3a9806b500 msgr2=0x7f3a9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:13.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.964+0000 7f3abbc9d640 1 --2- 172.21.15.161:0/890177262 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3a9806b500 0x7f3a9806d9c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4034000 tx=0x7f3aa4008030 comp rx=0 tx=0).stop 2024-03-28T23:35:13.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.964+0000 7f3abbc9d640 1 -- 172.21.15.161:0/890177262 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ab40c1dc0 msgr2=0x7f3ab41ec410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:13.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.964+0000 7f3abbc9d640 1 --2- 172.21.15.161:0/890177262 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ab40c1dc0 0x7f3ab41ec410 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f3ab000e9e0 tx=0x7f3ab000eea0 comp rx=0 tx=0).stop 2024-03-28T23:35:13.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.964+0000 7f3abbc9d640 1 -- 172.21.15.161:0/890177262 shutdown_connections 2024-03-28T23:35:13.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.964+0000 7f3abbc9d640 1 --2- 172.21.15.161:0/890177262 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3a9806b500 0x7f3a9806d9c0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:13.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.964+0000 7f3abbc9d640 1 --2- 172.21.15.161:0/890177262 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ab40c1dc0 0x7f3ab41ec410 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:13.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.964+0000 7f3abbc9d640 1 --2- 172.21.15.161:0/890177262 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ab414eca0 0x7f3ab4148680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:13.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.964+0000 7f3abbc9d640 1 --2- 172.21.15.161:0/890177262 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ab406cf60 0x7f3ab41dbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:13.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.964+0000 7f3abbc9d640 1 -- 172.21.15.161:0/890177262 >> 172.21.15.161:0/890177262 conn(0x7f3ab40c6670 msgr2=0x7f3ab40c0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:13.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.964+0000 7f3abbc9d640 1 -- 172.21.15.161:0/890177262 shutdown_connections 2024-03-28T23:35:13.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:13.964+0000 7f3abbc9d640 1 -- 172.21.15.161:0/890177262 wait complete. 2024-03-28T23:35:13.976 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:35:14.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.077+0000 7f017b89d640 1 Processor -- start 2024-03-28T23:35:14.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.077+0000 7f017b89d640 1 -- start start 2024-03-28T23:35:14.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.078+0000 7f017b89d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0174152890 0x7f01740c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:14.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.078+0000 7f017b89d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f01740c5b20 0x7f01741553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:14.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.078+0000 7f017b89d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0174158a30 con 0x7f01740c8ed0 2024-03-28T23:35:14.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.078+0000 7f017b89d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f01740c8480 con 0x7f01740c5b20 2024-03-28T23:35:14.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.078+0000 7f017b89d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0174159640 con 0x7f0174152890 2024-03-28T23:35:14.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.078+0000 7f0179e3a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f01740c5b20 0x7f01741553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:14.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.078+0000 7f0179e3a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f01740c5b20 0x7f01741553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47476/0 (socket says 172.21.15.161:47476) 2024-03-28T23:35:14.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.078+0000 7f0179e3a640 1 -- 172.21.15.161:0/3012685276 learned_addr learned my addr 172.21.15.161:0/3012685276 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:14.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.079+0000 7f0179e3a640 1 -- 172.21.15.161:0/3012685276 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0174152890 msgr2=0x7f01740c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:14.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.079+0000 7f0179e3a640 1 --2- 172.21.15.161:0/3012685276 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0174152890 0x7f01740c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:14.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.079+0000 7f0179e3a640 1 -- 172.21.15.161:0/3012685276 >> v1:172.21.15.170:6789/0 conn(0x7f01740c8ed0 legacy=0x7f01740c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:14.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.079+0000 7f0179e3a640 1 -- 172.21.15.161:0/3012685276 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0174158d10 con 0x7f01740c5b20 2024-03-28T23:35:14.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.079+0000 7f0179e3a640 1 --2- 172.21.15.161:0/3012685276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01740c5b20 0x7f01741553a0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f01741597c0 tx=0x7f016c02bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90c71d7c3da79b8a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:14.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.080+0000 7f016bfff640 1 -- 172.21.15.161:0/3012685276 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c02c930 con 0x7f01740c5b20 2024-03-28T23:35:14.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.080+0000 7f016bfff640 1 -- 172.21.15.161:0/3012685276 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f016c02cad0 con 0x7f01740c5b20 2024-03-28T23:35:14.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.080+0000 7f016bfff640 1 -- 172.21.15.161:0/3012685276 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c0343f0 con 0x7f01740c5b20 2024-03-28T23:35:14.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.080+0000 7f017b89d640 1 -- 172.21.15.161:0/3012685276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01740c5b20 msgr2=0x7f01741553a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:14.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.080+0000 7f017b89d640 1 --2- 172.21.15.161:0/3012685276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01740c5b20 0x7f01741553a0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f01741597c0 tx=0x7f016c02bd90 comp rx=0 tx=0).stop 2024-03-28T23:35:14.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.080+0000 7f017b89d640 1 -- 172.21.15.161:0/3012685276 shutdown_connections 2024-03-28T23:35:14.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.080+0000 7f017b89d640 1 --2- 172.21.15.161:0/3012685276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01740c5b20 0x7f01741553a0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:14.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.080+0000 7f017b89d640 1 --2- 172.21.15.161:0/3012685276 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0174152890 0x7f01740c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:14.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.080+0000 7f017b89d640 1 -- 172.21.15.161:0/3012685276 >> 172.21.15.161:0/3012685276 conn(0x7f01740ac620 msgr2=0x7f01740aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:14.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.081+0000 7f017b89d640 1 -- 172.21.15.161:0/3012685276 shutdown_connections 2024-03-28T23:35:14.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.081+0000 7f017b89d640 1 -- 172.21.15.161:0/3012685276 wait complete. 2024-03-28T23:35:14.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.082+0000 7f017b89d640 1 Processor -- start 2024-03-28T23:35:14.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.082+0000 7f017b89d640 1 -- start start 2024-03-28T23:35:14.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.082+0000 7f017b89d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01740c5b20 0x7f01741ee910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:14.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.082+0000 7f017b89d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f01740c8ed0 0x7f0174154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:14.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.082+0000 7f017b89d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0174152890 0x7f01741498e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:14.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.082+0000 7f017b89d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0174152c70 con 0x7f0174152890 2024-03-28T23:35:14.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.082+0000 7f017b89d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f01741592d0 con 0x7f01740c5b20 2024-03-28T23:35:14.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.082+0000 7f017b89d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0174152df0 con 0x7f01740c8ed0 2024-03-28T23:35:14.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.082+0000 7f0179e3a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0174152890 0x7f01741498e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:14.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.082+0000 7f0179e3a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0174152890 0x7f01741498e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47478/0 (socket says 172.21.15.161:47478) 2024-03-28T23:35:14.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.082+0000 7f0179e3a640 1 -- 172.21.15.161:0/2291284375 learned_addr learned my addr 172.21.15.161:0/2291284375 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:14.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.083+0000 7f0179639640 1 --2- 172.21.15.161:0/2291284375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01740c5b20 0x7f01741ee910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:14.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.083+0000 7f0179e3a640 1 -- 172.21.15.161:0/2291284375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f01740c8ed0 msgr2=0x7f0174154300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:14.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.083+0000 7f0178e38640 1 --2- 172.21.15.161:0/2291284375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f01740c8ed0 0x7f0174154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:14.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.083+0000 7f0179e3a640 1 --2- 172.21.15.161:0/2291284375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f01740c8ed0 0x7f0174154300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:14.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.083+0000 7f0179e3a640 1 -- 172.21.15.161:0/2291284375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01740c5b20 msgr2=0x7f01741ee910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:14.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.083+0000 7f0179e3a640 1 --2- 172.21.15.161:0/2291284375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01740c5b20 0x7f01741ee910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:14.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.083+0000 7f0179e3a640 1 -- 172.21.15.161:0/2291284375 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0174156310 con 0x7f0174152890 2024-03-28T23:35:14.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.083+0000 7f0179639640 1 --2- 172.21.15.161:0/2291284375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01740c5b20 0x7f01741ee910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:14.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.083+0000 7f0179e3a640 1 --2- 172.21.15.161:0/2291284375 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0174152890 0x7f01741498e0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f016c02c3a0 tx=0x7f016c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:14.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.083+0000 7f0169ffb640 1 -- 172.21.15.161:0/2291284375 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c043020 con 0x7f0174152890 2024-03-28T23:35:14.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.083+0000 7f0169ffb640 1 -- 172.21.15.161:0/2291284375 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f016c03e450 con 0x7f0174152890 2024-03-28T23:35:14.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.083+0000 7f017b89d640 1 -- 172.21.15.161:0/2291284375 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0174155b70 con 0x7f0174152890 2024-03-28T23:35:14.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.083+0000 7f0169ffb640 1 -- 172.21.15.161:0/2291284375 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c009510 con 0x7f0174152890 2024-03-28T23:35:14.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.083+0000 7f017b89d640 1 -- 172.21.15.161:0/2291284375 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01740c24e0 con 0x7f0174152890 2024-03-28T23:35:14.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.084+0000 7f017b89d640 1 -- 172.21.15.161:0/2291284375 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f017404f4b0 con 0x7f0174152890 2024-03-28T23:35:14.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.085+0000 7f0169ffb640 1 -- 172.21.15.161:0/2291284375 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f016c046020 con 0x7f0174152890 2024-03-28T23:35:14.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.085+0000 7f0169ffb640 1 --2- 172.21.15.161:0/2291284375 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f015806b4d0 0x7f015806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:14.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.085+0000 7f0179639640 1 --2- 172.21.15.161:0/2291284375 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f015806b4d0 0x7f015806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:14.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.085+0000 7f0169ffb640 1 -- 172.21.15.161:0/2291284375 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 6101+0+0 (secure 0 0 0) 0x7f016c0acf90 con 0x7f0174152890 2024-03-28T23:35:14.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.086+0000 7f0179639640 1 --2- 172.21.15.161:0/2291284375 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f015806b4d0 0x7f015806d990 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f0170006fd0 tx=0x7f0170008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:14.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.090+0000 7f0169ffb640 1 -- 172.21.15.161:0/2291284375 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f016c04d020 con 0x7f0174152890 2024-03-28T23:35:14.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.268+0000 7f017b89d640 1 -- 172.21.15.161:0/2291284375 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f01740c8140 con 0x7f0174152890 2024-03-28T23:35:14.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.269+0000 7f0169ffb640 1 -- 172.21.15.161:0/2291284375 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) ==== 74+0+16941 (secure 0 0 0) 0x7f0174155b70 con 0x7f0174152890 2024-03-28T23:35:14.270 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:14.270 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":135,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:35:12.982251+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:35:11.208674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-28T23:35:11.954908+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:35:14.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.271+0000 7f017b89d640 1 -- 172.21.15.161:0/2291284375 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f015806b4d0 msgr2=0x7f015806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:14.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.271+0000 7f017b89d640 1 --2- 172.21.15.161:0/2291284375 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f015806b4d0 0x7f015806d990 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f0170006fd0 tx=0x7f0170008040 comp rx=0 tx=0).stop 2024-03-28T23:35:14.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.271+0000 7f017b89d640 1 -- 172.21.15.161:0/2291284375 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0174152890 msgr2=0x7f01741498e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:14.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.271+0000 7f017b89d640 1 --2- 172.21.15.161:0/2291284375 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0174152890 0x7f01741498e0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f016c02c3a0 tx=0x7f016c004a40 comp rx=0 tx=0).stop 2024-03-28T23:35:14.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.271+0000 7f017b89d640 1 -- 172.21.15.161:0/2291284375 shutdown_connections 2024-03-28T23:35:14.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.271+0000 7f017b89d640 1 --2- 172.21.15.161:0/2291284375 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f015806b4d0 0x7f015806d990 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:14.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.271+0000 7f017b89d640 1 --2- 172.21.15.161:0/2291284375 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0174152890 0x7f01741498e0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:14.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.271+0000 7f017b89d640 1 --2- 172.21.15.161:0/2291284375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f01740c8ed0 0x7f0174154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:14.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.271+0000 7f017b89d640 1 --2- 172.21.15.161:0/2291284375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f01740c5b20 0x7f01741ee910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:14.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.271+0000 7f017b89d640 1 -- 172.21.15.161:0/2291284375 >> 172.21.15.161:0/2291284375 conn(0x7f01740ac620 msgr2=0x7f01740c5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:14.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.271+0000 7f017b89d640 1 -- 172.21.15.161:0/2291284375 shutdown_connections 2024-03-28T23:35:14.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.271+0000 7f017b89d640 1 -- 172.21.15.161:0/2291284375 wait complete. 2024-03-28T23:35:14.283 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-28T23:35:14.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.396+0000 7fdade76c640 1 Processor -- start 2024-03-28T23:35:14.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.396+0000 7fdade76c640 1 -- start start 2024-03-28T23:35:14.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.396+0000 7fdade76c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdad80c7f10 0x7fdad8156600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:14.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.396+0000 7fdade76c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fdad8072660 0x7fdad80c4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:14.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.396+0000 7fdade76c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdad80a8390 con 0x7fdad81538e0 2024-03-28T23:35:14.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.396+0000 7fdade76c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fdad80c48a0 con 0x7fdad80c7f10 2024-03-28T23:35:14.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.396+0000 7fdade76c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fdad8153120 con 0x7fdad8072660 2024-03-28T23:35:14.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.397+0000 7fdad7fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdad80c7f10 0x7fdad8156600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:14.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.397+0000 7fdadcd09640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fdad81538e0 0x7fdad80c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45248/0 (socket says 172.21.15.161:45248) 2024-03-28T23:35:14.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.397+0000 7fdadcd09640 1 -- 172.21.15.161:0/1589804225 learned_addr learned my addr 172.21.15.161:0/1589804225 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:14.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.397+0000 7fdad77fe640 1 --2- 172.21.15.161:0/1589804225 >> v2:172.21.15.170:3300/0 conn(0x7fdad8072660 0x7fdad80c4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:14.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.397+0000 7fdad7fff640 1 -- 172.21.15.161:0/1589804225 >> v2:172.21.15.170:3300/0 conn(0x7fdad8072660 msgr2=0x7fdad80c4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:14.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.397+0000 7fdad7fff640 1 --2- 172.21.15.161:0/1589804225 >> v2:172.21.15.170:3300/0 conn(0x7fdad8072660 0x7fdad80c4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:14.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.397+0000 7fdad7fff640 1 -- 172.21.15.161:0/1589804225 >> v1:172.21.15.161:6789/0 conn(0x7fdad81538e0 legacy=0x7fdad80c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:14.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.397+0000 7fdad7fff640 1 -- 172.21.15.161:0/1589804225 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdad80c4ad0 con 0x7fdad80c7f10 2024-03-28T23:35:14.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.397+0000 7fdad6ffd640 1 -- 172.21.15.161:0/1589804225 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3718690495 0 0) 0x7fdad0001490 con 0x7fdad81538e0 2024-03-28T23:35:14.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.397+0000 7fdad77fe640 1 --2- 172.21.15.161:0/1589804225 >> v2:172.21.15.170:3300/0 conn(0x7fdad8072660 0x7fdad80c4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:14.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.398+0000 7fdad7fff640 1 --2- 172.21.15.161:0/1589804225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdad80c7f10 0x7fdad8156600 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fdad80711f0 tx=0x7fdac802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5714c440cc71ee4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:14.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.398+0000 7fdad6ffd640 1 -- 172.21.15.161:0/1589804225 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac8035020 con 0x7fdad80c7f10 2024-03-28T23:35:14.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.398+0000 7fdad6ffd640 1 -- 172.21.15.161:0/1589804225 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdac8031830 con 0x7fdad80c7f10 2024-03-28T23:35:14.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.398+0000 7fdad6ffd640 1 -- 172.21.15.161:0/1589804225 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac8031c00 con 0x7fdad80c7f10 2024-03-28T23:35:14.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.398+0000 7fdade76c640 1 -- 172.21.15.161:0/1589804225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdad80c7f10 msgr2=0x7fdad8156600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:14.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.398+0000 7fdade76c640 1 --2- 172.21.15.161:0/1589804225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdad80c7f10 0x7fdad8156600 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fdad80711f0 tx=0x7fdac802b8d0 comp rx=0 tx=0).stop 2024-03-28T23:35:14.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.398+0000 7fdade76c640 1 -- 172.21.15.161:0/1589804225 shutdown_connections 2024-03-28T23:35:14.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.398+0000 7fdade76c640 1 --2- 172.21.15.161:0/1589804225 >> v2:172.21.15.170:3300/0 conn(0x7fdad8072660 0x7fdad80c4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:14.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.398+0000 7fdade76c640 1 --2- 172.21.15.161:0/1589804225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdad80c7f10 0x7fdad8156600 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:14.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.398+0000 7fdade76c640 1 -- 172.21.15.161:0/1589804225 >> 172.21.15.161:0/1589804225 conn(0x7fdad80c88b0 msgr2=0x7fdad80c8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:14.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.398+0000 7fdade76c640 1 -- 172.21.15.161:0/1589804225 shutdown_connections 2024-03-28T23:35:14.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.398+0000 7fdade76c640 1 -- 172.21.15.161:0/1589804225 wait complete. 2024-03-28T23:35:14.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.399+0000 7fdade76c640 1 Processor -- start 2024-03-28T23:35:14.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.400+0000 7fdade76c640 1 -- start start 2024-03-28T23:35:14.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.400+0000 7fdade76c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdad8072660 0x7fdad81543b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:14.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.400+0000 7fdade76c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdad80c7f10 0x7fdad81ec410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:14.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.400+0000 7fdade76c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdad81538e0 0x7fdad81ec950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:14.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.400+0000 7fdade76c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdad80c5090 con 0x7fdad81538e0 2024-03-28T23:35:14.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.400+0000 7fdade76c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdad81533c0 con 0x7fdad8072660 2024-03-28T23:35:14.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.400+0000 7fdade76c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdad80c82f0 con 0x7fdad80c7f10 2024-03-28T23:35:14.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.400+0000 7fdadcd09640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdad81538e0 0x7fdad81ec950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:14.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.400+0000 7fdadcd09640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdad81538e0 0x7fdad81ec950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47504/0 (socket says 172.21.15.161:47504) 2024-03-28T23:35:14.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.401+0000 7fdadcd09640 1 -- 172.21.15.161:0/1434020617 learned_addr learned my addr 172.21.15.161:0/1434020617 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:14.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.401+0000 7fdad7fff640 1 --2- 172.21.15.161:0/1434020617 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdad8072660 0x7fdad81543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:14.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.401+0000 7fdad77fe640 1 --2- 172.21.15.161:0/1434020617 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdad80c7f10 0x7fdad81ec410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:14.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.401+0000 7fdadcd09640 1 -- 172.21.15.161:0/1434020617 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdad80c7f10 msgr2=0x7fdad81ec410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:14.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.401+0000 7fdadcd09640 1 --2- 172.21.15.161:0/1434020617 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdad80c7f10 0x7fdad81ec410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:14.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.401+0000 7fdadcd09640 1 -- 172.21.15.161:0/1434020617 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdad8072660 msgr2=0x7fdad81543b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:14.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.401+0000 7fdadcd09640 1 --2- 172.21.15.161:0/1434020617 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdad8072660 0x7fdad81543b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:14.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.401+0000 7fdadcd09640 1 -- 172.21.15.161:0/1434020617 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdad81de680 con 0x7fdad81538e0 2024-03-28T23:35:14.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.401+0000 7fdad7fff640 1 --2- 172.21.15.161:0/1434020617 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdad8072660 0x7fdad81543b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:14.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.401+0000 7fdadcd09640 1 --2- 172.21.15.161:0/1434020617 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdad81538e0 0x7fdad81ec950 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fdad000c820 tx=0x7fdad000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:14.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.401+0000 7fdad4ff9640 1 -- 172.21.15.161:0/1434020617 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad0018070 con 0x7fdad81538e0 2024-03-28T23:35:14.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.401+0000 7fdad4ff9640 1 -- 172.21.15.161:0/1434020617 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdad000a5e0 con 0x7fdad81538e0 2024-03-28T23:35:14.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.401+0000 7fdade76c640 1 -- 172.21.15.161:0/1434020617 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdad81de960 con 0x7fdad81538e0 2024-03-28T23:35:14.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.401+0000 7fdad4ff9640 1 -- 172.21.15.161:0/1434020617 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad000a9d0 con 0x7fdad81538e0 2024-03-28T23:35:14.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.401+0000 7fdade76c640 1 -- 172.21.15.161:0/1434020617 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdad80c2da0 con 0x7fdad81538e0 2024-03-28T23:35:14.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.402+0000 7fdade76c640 1 -- 172.21.15.161:0/1434020617 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdad804f4b0 con 0x7fdad81538e0 2024-03-28T23:35:14.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.403+0000 7fdad4ff9640 1 -- 172.21.15.161:0/1434020617 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdad001b070 con 0x7fdad81538e0 2024-03-28T23:35:14.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.403+0000 7fdad4ff9640 1 --2- 172.21.15.161:0/1434020617 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdac406b500 0x7fdac406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:14.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.403+0000 7fdad4ff9640 1 -- 172.21.15.161:0/1434020617 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 6101+0+0 (secure 0 0 0) 0x7fdad0089920 con 0x7fdad81538e0 2024-03-28T23:35:14.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.403+0000 7fdad7fff640 1 --2- 172.21.15.161:0/1434020617 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdac406b500 0x7fdac406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:14.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.403+0000 7fdad7fff640 1 --2- 172.21.15.161:0/1434020617 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdac406b500 0x7fdac406d9c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fdac8036000 tx=0x7fdac8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:14.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.407+0000 7fdad4ff9640 1 -- 172.21.15.161:0/1434020617 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdad80a8390 con 0x7fdad81538e0 2024-03-28T23:35:14.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.628+0000 7fdade76c640 1 -- 172.21.15.161:0/1434020617 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fdad80c9280 con 0x7fdad81538e0 2024-03-28T23:35:14.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.959+0000 7fdad4ff9640 1 -- 172.21.15.161:0/1434020617 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v123) ==== 118+0+0 (secure 0 0 0) 0x7fdad0058610 con 0x7fdad81538e0 2024-03-28T23:35:14.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.962+0000 7fdade76c640 1 -- 172.21.15.161:0/1434020617 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdac406b500 msgr2=0x7fdac406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:14.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.962+0000 7fdade76c640 1 --2- 172.21.15.161:0/1434020617 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdac406b500 0x7fdac406d9c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fdac8036000 tx=0x7fdac8008030 comp rx=0 tx=0).stop 2024-03-28T23:35:14.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.962+0000 7fdade76c640 1 -- 172.21.15.161:0/1434020617 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdad81538e0 msgr2=0x7fdad81ec950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:14.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.962+0000 7fdade76c640 1 --2- 172.21.15.161:0/1434020617 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdad81538e0 0x7fdad81ec950 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fdad000c820 tx=0x7fdad000cce0 comp rx=0 tx=0).stop 2024-03-28T23:35:14.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.962+0000 7fdade76c640 1 -- 172.21.15.161:0/1434020617 shutdown_connections 2024-03-28T23:35:14.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.962+0000 7fdade76c640 1 --2- 172.21.15.161:0/1434020617 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdac406b500 0x7fdac406d9c0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:14.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.962+0000 7fdade76c640 1 --2- 172.21.15.161:0/1434020617 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdad81538e0 0x7fdad81ec950 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:14.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.962+0000 7fdade76c640 1 --2- 172.21.15.161:0/1434020617 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdad80c7f10 0x7fdad81ec410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:14.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.962+0000 7fdade76c640 1 --2- 172.21.15.161:0/1434020617 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdad8072660 0x7fdad81543b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:14.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.962+0000 7fdade76c640 1 -- 172.21.15.161:0/1434020617 >> 172.21.15.161:0/1434020617 conn(0x7fdad80c88b0 msgr2=0x7fdad8150e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:14.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.962+0000 7fdade76c640 1 -- 172.21.15.161:0/1434020617 shutdown_connections 2024-03-28T23:35:14.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:14.962+0000 7fdade76c640 1 -- 172.21.15.161:0/1434020617 wait complete. 2024-03-28T23:35:14.973 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-28T23:35:14.974 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-28T23:35:15.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.106+0000 7f97cb9e2640 1 Processor -- start 2024-03-28T23:35:15.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.106+0000 7f97cb9e2640 1 -- start start 2024-03-28T23:35:15.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.107+0000 7f97cb9e2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f97c4152750 0x7f97c40c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:15.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.107+0000 7f97cb9e2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97c4155110 0x7f97c4155560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:15.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.107+0000 7f97cb9e2640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c406eaf0 con 0x7f97c406c490 2024-03-28T23:35:15.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.107+0000 7f97cb9e2640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f97c4158700 con 0x7f97c4152750 2024-03-28T23:35:15.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.107+0000 7f97cb9e2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f97c4152b30 con 0x7f97c4155110 2024-03-28T23:35:15.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.107+0000 7f97c977e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f97c4152750 0x7f97c40c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:15.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.107+0000 7f97c977e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f97c4152750 0x7f97c40c60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47520/0 (socket says 172.21.15.161:47520) 2024-03-28T23:35:15.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.107+0000 7f97c977e640 1 -- 172.21.15.161:0/4209221281 learned_addr learned my addr 172.21.15.161:0/4209221281 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:15.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.107+0000 7f97c8f7d640 1 --1- 172.21.15.161:0/4209221281 >> v1:172.21.15.161:6789/0 conn(0x7f97c406c490 0x7f97c40bfea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45250/0 (socket says 172.21.15.161:45250) 2024-03-28T23:35:15.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.107+0000 7f97c977e640 1 -- 172.21.15.161:0/4209221281 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97c4155110 msgr2=0x7f97c4155560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:15.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.107+0000 7f97c977e640 1 --2- 172.21.15.161:0/4209221281 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97c4155110 0x7f97c4155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:15.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.107+0000 7f97c977e640 1 -- 172.21.15.161:0/4209221281 >> v1:172.21.15.161:6789/0 conn(0x7f97c406c490 legacy=0x7f97c40bfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:15.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.107+0000 7f97bbfff640 1 -- 172.21.15.161:0/4209221281 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2076819728 0 0) 0x7f97c406eaf0 con 0x7f97c406c490 2024-03-28T23:35:15.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.107+0000 7f97c977e640 1 -- 172.21.15.161:0/4209221281 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c4158950 con 0x7f97c4152750 2024-03-28T23:35:15.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.108+0000 7f97c977e640 1 --2- 172.21.15.161:0/4209221281 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97c4152750 0x7f97c40c60e0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f97c0009870 tx=0x7f97c002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61d0d3b222b2f408 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:15.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.108+0000 7f97bbfff640 1 -- 172.21.15.161:0/4209221281 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c00317c0 con 0x7f97c4152750 2024-03-28T23:35:15.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.108+0000 7f97bbfff640 1 -- 172.21.15.161:0/4209221281 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97c0037070 con 0x7f97c4152750 2024-03-28T23:35:15.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.108+0000 7f97bbfff640 1 -- 172.21.15.161:0/4209221281 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c0031b80 con 0x7f97c4152750 2024-03-28T23:35:15.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.109+0000 7f97cb9e2640 1 -- 172.21.15.161:0/4209221281 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97c4152750 msgr2=0x7f97c40c60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:15.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.109+0000 7f97cb9e2640 1 --2- 172.21.15.161:0/4209221281 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97c4152750 0x7f97c40c60e0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f97c0009870 tx=0x7f97c002b860 comp rx=0 tx=0).stop 2024-03-28T23:35:15.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.109+0000 7f97cb9e2640 1 -- 172.21.15.161:0/4209221281 shutdown_connections 2024-03-28T23:35:15.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.109+0000 7f97cb9e2640 1 --2- 172.21.15.161:0/4209221281 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97c4155110 0x7f97c4155560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:15.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.109+0000 7f97cb9e2640 1 --2- 172.21.15.161:0/4209221281 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97c4152750 0x7f97c40c60e0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:15.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.109+0000 7f97cb9e2640 1 -- 172.21.15.161:0/4209221281 >> 172.21.15.161:0/4209221281 conn(0x7f97c40c6b30 msgr2=0x7f97c40c6f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:15.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.109+0000 7f97cb9e2640 1 -- 172.21.15.161:0/4209221281 shutdown_connections 2024-03-28T23:35:15.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.109+0000 7f97cb9e2640 1 -- 172.21.15.161:0/4209221281 wait complete. 2024-03-28T23:35:15.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.110+0000 7f97cb9e2640 1 Processor -- start 2024-03-28T23:35:15.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.110+0000 7f97cb9e2640 1 -- start start 2024-03-28T23:35:15.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.111+0000 7f97cb9e2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97c406c490 0x7f97c4070de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:15.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.111+0000 7f97cb9e2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97c4152750 0x7f97c414d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:15.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.111+0000 7f97cb9e2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97c4155110 0x7f97c41eb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:15.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.111+0000 7f97cb9e2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f97c41591f0 con 0x7f97c406c490 2024-03-28T23:35:15.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.111+0000 7f97cb9e2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f97c4158f10 con 0x7f97c4152750 2024-03-28T23:35:15.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.111+0000 7f97cb9e2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f97c40c05b0 con 0x7f97c4155110 2024-03-28T23:35:15.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.111+0000 7f97c977e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97c406c490 0x7f97c4070de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:15.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.111+0000 7f97c977e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97c406c490 0x7f97c4070de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47536/0 (socket says 172.21.15.161:47536) 2024-03-28T23:35:15.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.111+0000 7f97c977e640 1 -- 172.21.15.161:0/1448354122 learned_addr learned my addr 172.21.15.161:0/1448354122 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:15.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.111+0000 7f97c9f7f640 1 --2- 172.21.15.161:0/1448354122 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97c4155110 0x7f97c41eb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:15.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.111+0000 7f97c8f7d640 1 --2- 172.21.15.161:0/1448354122 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97c4152750 0x7f97c414d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:15.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.111+0000 7f97c977e640 1 -- 172.21.15.161:0/1448354122 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97c4155110 msgr2=0x7f97c41eb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:15.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.111+0000 7f97c977e640 1 --2- 172.21.15.161:0/1448354122 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97c4155110 0x7f97c41eb460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:15.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.111+0000 7f97c977e640 1 -- 172.21.15.161:0/1448354122 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97c4152750 msgr2=0x7f97c414d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:15.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.111+0000 7f97c977e640 1 --2- 172.21.15.161:0/1448354122 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97c4152750 0x7f97c414d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:15.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.111+0000 7f97c977e640 1 -- 172.21.15.161:0/1448354122 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c40707a0 con 0x7f97c406c490 2024-03-28T23:35:15.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.111+0000 7f97c8f7d640 1 --2- 172.21.15.161:0/1448354122 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97c4152750 0x7f97c414d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:15.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.111+0000 7f97c977e640 1 --2- 172.21.15.161:0/1448354122 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97c406c490 0x7f97c4070de0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f97c0009870 tx=0x7f97c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:15.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.111+0000 7f97b9ffb640 1 -- 172.21.15.161:0/1448354122 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c0032860 con 0x7f97c406c490 2024-03-28T23:35:15.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.111+0000 7f97b9ffb640 1 -- 172.21.15.161:0/1448354122 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97c0032a20 con 0x7f97c406c490 2024-03-28T23:35:15.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.111+0000 7f97cb9e2640 1 -- 172.21.15.161:0/1448354122 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97c4070a80 con 0x7f97c406c490 2024-03-28T23:35:15.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.111+0000 7f97b9ffb640 1 -- 172.21.15.161:0/1448354122 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c0032e10 con 0x7f97c406c490 2024-03-28T23:35:15.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.112+0000 7f97cb9e2640 1 -- 172.21.15.161:0/1448354122 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97c41eaf40 con 0x7f97c406c490 2024-03-28T23:35:15.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.113+0000 7f97cb9e2640 1 -- 172.21.15.161:0/1448354122 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97c41ead50 con 0x7f97c406c490 2024-03-28T23:35:15.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.113+0000 7f97b9ffb640 1 -- 172.21.15.161:0/1448354122 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f97c004d3f0 con 0x7f97c406c490 2024-03-28T23:35:15.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.113+0000 7f97b9ffb640 1 --2- 172.21.15.161:0/1448354122 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f97a806b4d0 0x7f97a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:15.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.114+0000 7f97c8f7d640 1 --2- 172.21.15.161:0/1448354122 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f97a806b4d0 0x7f97a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:15.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.114+0000 7f97b9ffb640 1 -- 172.21.15.161:0/1448354122 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 6101+0+0 (secure 0 0 0) 0x7f97c00ac080 con 0x7f97c406c490 2024-03-28T23:35:15.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.114+0000 7f97c8f7d640 1 --2- 172.21.15.161:0/1448354122 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f97a806b4d0 0x7f97a806d990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f97c4157f40 tx=0x7f97b4007cc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:15.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.118+0000 7f97b9ffb640 1 -- 172.21.15.161:0/1448354122 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f97c0040070 con 0x7f97c406c490 2024-03-28T23:35:15.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.336+0000 7f97cb9e2640 1 -- 172.21.15.161:0/1448354122 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f97c404f4b0 con 0x7f97c406c490 2024-03-28T23:35:15.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.968+0000 7f97b9ffb640 1 -- 172.21.15.161:0/1448354122 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v124) ==== 122+0+0 (secure 0 0 0) 0x7f97c007ad70 con 0x7f97c406c490 2024-03-28T23:35:15.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.971+0000 7f97cb9e2640 1 -- 172.21.15.161:0/1448354122 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f97a806b4d0 msgr2=0x7f97a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:15.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.971+0000 7f97cb9e2640 1 --2- 172.21.15.161:0/1448354122 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f97a806b4d0 0x7f97a806d990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f97c4157f40 tx=0x7f97b4007cc0 comp rx=0 tx=0).stop 2024-03-28T23:35:15.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.971+0000 7f97cb9e2640 1 -- 172.21.15.161:0/1448354122 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97c406c490 msgr2=0x7f97c4070de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:15.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.971+0000 7f97cb9e2640 1 --2- 172.21.15.161:0/1448354122 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97c406c490 0x7f97c4070de0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f97c0009870 tx=0x7f97c00046c0 comp rx=0 tx=0).stop 2024-03-28T23:35:15.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.971+0000 7f97cb9e2640 1 -- 172.21.15.161:0/1448354122 shutdown_connections 2024-03-28T23:35:15.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.971+0000 7f97cb9e2640 1 --2- 172.21.15.161:0/1448354122 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f97a806b4d0 0x7f97a806d990 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:15.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.971+0000 7f97cb9e2640 1 --2- 172.21.15.161:0/1448354122 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97c4155110 0x7f97c41eb460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:15.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.971+0000 7f97cb9e2640 1 --2- 172.21.15.161:0/1448354122 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97c4152750 0x7f97c414d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:15.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.971+0000 7f97cb9e2640 1 --2- 172.21.15.161:0/1448354122 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97c406c490 0x7f97c4070de0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:15.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.971+0000 7f97cb9e2640 1 -- 172.21.15.161:0/1448354122 >> 172.21.15.161:0/1448354122 conn(0x7f97c40c6b30 msgr2=0x7f97c40c6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:15.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.971+0000 7f97cb9e2640 1 -- 172.21.15.161:0/1448354122 shutdown_connections 2024-03-28T23:35:15.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:15.971+0000 7f97cb9e2640 1 -- 172.21.15.161:0/1448354122 wait complete. 2024-03-28T23:35:15.983 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-28T23:35:15.983 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:35:16.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.089+0000 7fd3eb0d4640 1 Processor -- start 2024-03-28T23:35:16.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.089+0000 7fd3eb0d4640 1 -- start start 2024-03-28T23:35:16.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.090+0000 7fd3eb0d4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd3e40c3410 0x7fd3e41591e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:16.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.090+0000 7fd3eb0d4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3e4155d30 con 0x7fd3e40c3ae0 2024-03-28T23:35:16.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.090+0000 7fd3eb0d4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3e40c1e10 con 0x7fd3e414f240 2024-03-28T23:35:16.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.090+0000 7fd3eb0d4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd3e4156d10 con 0x7fd3e40c3410 2024-03-28T23:35:16.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.090+0000 7fd3e3fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd3e40c3410 0x7fd3e41591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:16.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.090+0000 7fd3e9671640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd3e40c3ae0 0x7fd3e40c66a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45260/0 (socket says 172.21.15.161:45260) 2024-03-28T23:35:16.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.090+0000 7fd3e9671640 1 -- 172.21.15.161:0/3613362312 learned_addr learned my addr 172.21.15.161:0/3613362312 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:16.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.090+0000 7fd3e3fff640 1 -- 172.21.15.161:0/3613362312 >> v1:172.21.15.170:6789/0 conn(0x7fd3e414f240 legacy=0x7fd3e4156600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:16.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.090+0000 7fd3e3fff640 1 -- 172.21.15.161:0/3613362312 >> v1:172.21.15.161:6789/0 conn(0x7fd3e40c3ae0 legacy=0x7fd3e40c66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:16.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.090+0000 7fd3e3fff640 1 -- 172.21.15.161:0/3613362312 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3e4155fd0 con 0x7fd3e40c3410 2024-03-28T23:35:16.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.091+0000 7fd3e3fff640 1 --2- 172.21.15.161:0/3613362312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3e40c3410 0x7fd3e41591e0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fd3d00097a0 tx=0x7fd3d002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=560e23569c2cd95f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:16.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.091+0000 7fd3e37fe640 1 -- 172.21.15.161:0/3613362312 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d00315a0 con 0x7fd3e40c3410 2024-03-28T23:35:16.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.091+0000 7fd3e37fe640 1 -- 172.21.15.161:0/3613362312 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3d0037070 con 0x7fd3e40c3410 2024-03-28T23:35:16.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.091+0000 7fd3e37fe640 1 -- 172.21.15.161:0/3613362312 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0031960 con 0x7fd3e40c3410 2024-03-28T23:35:16.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.091+0000 7fd3eb0d4640 1 -- 172.21.15.161:0/3613362312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3e40c3410 msgr2=0x7fd3e41591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:16.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.091+0000 7fd3eb0d4640 1 --2- 172.21.15.161:0/3613362312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3e40c3410 0x7fd3e41591e0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fd3d00097a0 tx=0x7fd3d002b840 comp rx=0 tx=0).stop 2024-03-28T23:35:16.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.091+0000 7fd3eb0d4640 1 -- 172.21.15.161:0/3613362312 shutdown_connections 2024-03-28T23:35:16.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.091+0000 7fd3eb0d4640 1 --2- 172.21.15.161:0/3613362312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3e40c3410 0x7fd3e41591e0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.091+0000 7fd3eb0d4640 1 -- 172.21.15.161:0/3613362312 >> 172.21.15.161:0/3613362312 conn(0x7fd3e40c7790 msgr2=0x7fd3e40c7ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:16.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.092+0000 7fd3eb0d4640 1 -- 172.21.15.161:0/3613362312 shutdown_connections 2024-03-28T23:35:16.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.092+0000 7fd3eb0d4640 1 -- 172.21.15.161:0/3613362312 wait complete. 2024-03-28T23:35:16.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.093+0000 7fd3eb0d4640 1 Processor -- start 2024-03-28T23:35:16.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.093+0000 7fd3eb0d4640 1 -- start start 2024-03-28T23:35:16.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.094+0000 7fd3eb0d4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3e40c3410 0x7fd3e4071920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:16.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.094+0000 7fd3eb0d4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3e40c3ae0 0x7fd3e414d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:16.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.094+0000 7fd3eb0d4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3e414f240 0x7fd3e414e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:16.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.095+0000 7fd3eb0d4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e41589e0 con 0x7fd3e414f240 2024-03-28T23:35:16.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.095+0000 7fd3eb0d4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e414f620 con 0x7fd3e40c3ae0 2024-03-28T23:35:16.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.095+0000 7fd3eb0d4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd3e4158cc0 con 0x7fd3e40c3410 2024-03-28T23:35:16.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.095+0000 7fd3e9671640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3e414f240 0x7fd3e414e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:16.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.095+0000 7fd3e9671640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3e414f240 0x7fd3e414e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47564/0 (socket says 172.21.15.161:47564) 2024-03-28T23:35:16.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.095+0000 7fd3e9671640 1 -- 172.21.15.161:0/1874858582 learned_addr learned my addr 172.21.15.161:0/1874858582 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:16.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.095+0000 7fd3e3fff640 1 --2- 172.21.15.161:0/1874858582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3e40c3ae0 0x7fd3e414d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:16.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.095+0000 7fd3e9671640 1 -- 172.21.15.161:0/1874858582 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3e40c3410 msgr2=0x7fd3e4071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:16.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.095+0000 7fd3e9671640 1 --2- 172.21.15.161:0/1874858582 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3e40c3410 0x7fd3e4071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.095+0000 7fd3e9671640 1 -- 172.21.15.161:0/1874858582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3e40c3ae0 msgr2=0x7fd3e414d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:16.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.095+0000 7fd3e9671640 1 --2- 172.21.15.161:0/1874858582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3e40c3ae0 0x7fd3e414d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.095+0000 7fd3e9671640 1 -- 172.21.15.161:0/1874858582 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3e4072f80 con 0x7fd3e414f240 2024-03-28T23:35:16.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.095+0000 7fd3e9671640 1 --2- 172.21.15.161:0/1874858582 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3e414f240 0x7fd3e414e390 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc00cbc0 tx=0x7fd3dc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:16.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.095+0000 7fd3e17fa640 1 -- 172.21.15.161:0/1874858582 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc017070 con 0x7fd3e414f240 2024-03-28T23:35:16.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.095+0000 7fd3e17fa640 1 -- 172.21.15.161:0/1874858582 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3dc00ad40 con 0x7fd3e414f240 2024-03-28T23:35:16.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.095+0000 7fd3eb0d4640 1 -- 172.21.15.161:0/1874858582 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3e4073260 con 0x7fd3e414f240 2024-03-28T23:35:16.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.095+0000 7fd3e17fa640 1 -- 172.21.15.161:0/1874858582 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc012600 con 0x7fd3e414f240 2024-03-28T23:35:16.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.095+0000 7fd3eb0d4640 1 -- 172.21.15.161:0/1874858582 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3e414e8d0 con 0x7fd3e414f240 2024-03-28T23:35:16.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.096+0000 7fd3eb0d4640 1 -- 172.21.15.161:0/1874858582 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3e40734a0 con 0x7fd3e414f240 2024-03-28T23:35:16.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.097+0000 7fd3e17fa640 1 -- 172.21.15.161:0/1874858582 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd3dc01a070 con 0x7fd3e414f240 2024-03-28T23:35:16.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.097+0000 7fd3e17fa640 1 --2- 172.21.15.161:0/1874858582 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd3c806b4d0 0x7fd3c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:16.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.097+0000 7fd3e17fa640 1 -- 172.21.15.161:0/1874858582 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 6101+0+0 (secure 0 0 0) 0x7fd3dc089f60 con 0x7fd3e414f240 2024-03-28T23:35:16.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.097+0000 7fd3e8e70640 1 --2- 172.21.15.161:0/1874858582 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd3c806b4d0 0x7fd3c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:16.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.098+0000 7fd3e8e70640 1 --2- 172.21.15.161:0/1874858582 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd3c806b4d0 0x7fd3c806d990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8009650 tx=0x7fd3d8006ba0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:16.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.101+0000 7fd3e17fa640 1 -- 172.21.15.161:0/1874858582 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd3e40734a0 con 0x7fd3e414f240 2024-03-28T23:35:16.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.316+0000 7fd3eb0d4640 1 -- 172.21.15.161:0/1874858582 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3e404f4b0 con 0x7fd3e414f240 2024-03-28T23:35:16.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.317+0000 7fd3e17fa640 1 -- 172.21.15.161:0/1874858582 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) ==== 95+0+5203 (secure 0 0 0) 0x7fd3dc058690 con 0x7fd3e414f240 2024-03-28T23:35:16.317 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:16.318 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":124,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6740,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3586875977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3586875977},{"type":"v1","addr":"172.21.15.170:6835","nonce":3586875977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120},{"gid":7639,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/3182285172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3182285172},{"type":"v1","addr":"172.21.15.170:6834","nonce":3182285172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120},{"gid":8409,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/3037890643","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3037890643},{"type":"v1","addr":"172.21.15.161:6837","nonce":3037890643}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:12.982566+0000","modified":"2024-03-28T23:35:15.337512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8412},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8412":{"gid":8412,"name":"a","rank":0,"incarnation":121,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6836/2527502405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":2527502405},{"type":"v1","addr":"172.21.15.161:6836","nonce":2527502405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8412,"qdb_cluster":[8412]},"id":10}]} 2024-03-28T23:35:16.318 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 124 2024-03-28T23:35:16.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.319+0000 7fd3eb0d4640 1 -- 172.21.15.161:0/1874858582 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd3c806b4d0 msgr2=0x7fd3c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:16.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.319+0000 7fd3eb0d4640 1 --2- 172.21.15.161:0/1874858582 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd3c806b4d0 0x7fd3c806d990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8009650 tx=0x7fd3d8006ba0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.319+0000 7fd3eb0d4640 1 -- 172.21.15.161:0/1874858582 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3e414f240 msgr2=0x7fd3e414e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:16.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.319+0000 7fd3eb0d4640 1 --2- 172.21.15.161:0/1874858582 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3e414f240 0x7fd3e414e390 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc00cbc0 tx=0x7fd3dc00a430 comp rx=0 tx=0).stop 2024-03-28T23:35:16.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.319+0000 7fd3eb0d4640 1 -- 172.21.15.161:0/1874858582 shutdown_connections 2024-03-28T23:35:16.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.319+0000 7fd3eb0d4640 1 --2- 172.21.15.161:0/1874858582 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd3c806b4d0 0x7fd3c806d990 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.319+0000 7fd3eb0d4640 1 --2- 172.21.15.161:0/1874858582 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3e414f240 0x7fd3e414e390 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.319+0000 7fd3eb0d4640 1 --2- 172.21.15.161:0/1874858582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3e40c3ae0 0x7fd3e414d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.319+0000 7fd3eb0d4640 1 --2- 172.21.15.161:0/1874858582 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3e40c3410 0x7fd3e4071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.319+0000 7fd3eb0d4640 1 -- 172.21.15.161:0/1874858582 >> 172.21.15.161:0/1874858582 conn(0x7fd3e40c7790 msgr2=0x7fd3e40c6340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:16.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.319+0000 7fd3eb0d4640 1 -- 172.21.15.161:0/1874858582 shutdown_connections 2024-03-28T23:35:16.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.319+0000 7fd3eb0d4640 1 -- 172.21.15.161:0/1874858582 wait complete. 2024-03-28T23:35:16.331 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:35:16.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.433+0000 7f7bbe215640 1 Processor -- start 2024-03-28T23:35:16.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.433+0000 7f7bbe215640 1 -- start start 2024-03-28T23:35:16.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.434+0000 7f7bbe215640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7bb8152890 0x7f7bb80c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:16.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.434+0000 7f7bbe215640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bb80c5b20 0x7f7bb81553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:16.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.434+0000 7f7bbe215640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bb8158a30 con 0x7f7bb80c8ed0 2024-03-28T23:35:16.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.434+0000 7f7bbe215640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f7bb80c8480 con 0x7f7bb8152890 2024-03-28T23:35:16.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.434+0000 7f7bbe215640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7bb8159640 con 0x7f7bb80c5b20 2024-03-28T23:35:16.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.434+0000 7f7bb77fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7bb8152890 0x7f7bb80c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:16.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.434+0000 7f7bb6ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f7bb80c8ed0 0x7f7bb80c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45266/0 (socket says 172.21.15.161:45266) 2024-03-28T23:35:16.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.434+0000 7f7bb6ffd640 1 -- 172.21.15.161:0/1549818438 learned_addr learned my addr 172.21.15.161:0/1549818438 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:16.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.434+0000 7f7bb7fff640 1 --2- 172.21.15.161:0/1549818438 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bb80c5b20 0x7f7bb81553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:16.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.434+0000 7f7bb77fe640 1 -- 172.21.15.161:0/1549818438 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bb80c5b20 msgr2=0x7f7bb81553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:16.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.434+0000 7f7bb77fe640 1 --2- 172.21.15.161:0/1549818438 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bb80c5b20 0x7f7bb81553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.434+0000 7f7bb77fe640 1 -- 172.21.15.161:0/1549818438 >> v1:172.21.15.161:6789/0 conn(0x7f7bb80c8ed0 legacy=0x7f7bb80c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:16.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.434+0000 7f7bb77fe640 1 -- 172.21.15.161:0/1549818438 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bb8158d10 con 0x7f7bb8152890 2024-03-28T23:35:16.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.434+0000 7f7bb67fc640 1 -- 172.21.15.161:0/1549818438 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2154112565 0 0) 0x7f7bb8158a30 con 0x7f7bb80c8ed0 2024-03-28T23:35:16.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.434+0000 7f7bb7fff640 1 --2- 172.21.15.161:0/1549818438 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bb80c5b20 0x7f7bb81553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:16.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.435+0000 7f7bb77fe640 1 --2- 172.21.15.161:0/1549818438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bb8152890 0x7f7bb80c8990 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f7bb81597c0 tx=0x7f7ba802b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5918551f34d4dbb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:16.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.435+0000 7f7bb67fc640 1 -- 172.21.15.161:0/1549818438 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba8039070 con 0x7f7bb8152890 2024-03-28T23:35:16.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.435+0000 7f7bb67fc640 1 -- 172.21.15.161:0/1549818438 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ba80316f0 con 0x7f7bb8152890 2024-03-28T23:35:16.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.435+0000 7f7bb67fc640 1 -- 172.21.15.161:0/1549818438 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba8031ac0 con 0x7f7bb8152890 2024-03-28T23:35:16.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.435+0000 7f7bbe215640 1 -- 172.21.15.161:0/1549818438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bb8152890 msgr2=0x7f7bb80c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:16.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.435+0000 7f7bbe215640 1 --2- 172.21.15.161:0/1549818438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bb8152890 0x7f7bb80c8990 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f7bb81597c0 tx=0x7f7ba802b790 comp rx=0 tx=0).stop 2024-03-28T23:35:16.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.436+0000 7f7bbe215640 1 -- 172.21.15.161:0/1549818438 shutdown_connections 2024-03-28T23:35:16.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.436+0000 7f7bbe215640 1 --2- 172.21.15.161:0/1549818438 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bb80c5b20 0x7f7bb81553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.436+0000 7f7bbe215640 1 --2- 172.21.15.161:0/1549818438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bb8152890 0x7f7bb80c8990 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.436+0000 7f7bbe215640 1 -- 172.21.15.161:0/1549818438 >> 172.21.15.161:0/1549818438 conn(0x7f7bb80ac620 msgr2=0x7f7bb80aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:16.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.436+0000 7f7bbe215640 1 -- 172.21.15.161:0/1549818438 shutdown_connections 2024-03-28T23:35:16.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.436+0000 7f7bbe215640 1 -- 172.21.15.161:0/1549818438 wait complete. 2024-03-28T23:35:16.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.437+0000 7f7bbe215640 1 Processor -- start 2024-03-28T23:35:16.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.437+0000 7f7bbe215640 1 -- start start 2024-03-28T23:35:16.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.437+0000 7f7bbe215640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7bb80c5b20 0x7f7bb81ee3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:16.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.437+0000 7f7bbe215640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bb80c8ed0 0x7f7bb81ec580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:16.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.438+0000 7f7bbe215640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bb8152890 0x7f7bb81ecac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:16.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.438+0000 7f7bbe215640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb8152c70 con 0x7f7bb80c8ed0 2024-03-28T23:35:16.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.438+0000 7f7bbe215640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb81592d0 con 0x7f7bb80c5b20 2024-03-28T23:35:16.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.438+0000 7f7bbe215640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7bb8152df0 con 0x7f7bb8152890 2024-03-28T23:35:16.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.438+0000 7f7bb6ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bb80c8ed0 0x7f7bb81ec580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:16.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.438+0000 7f7bb6ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bb80c8ed0 0x7f7bb81ec580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47578/0 (socket says 172.21.15.161:47578) 2024-03-28T23:35:16.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.438+0000 7f7bb6ffd640 1 -- 172.21.15.161:0/1529426479 learned_addr learned my addr 172.21.15.161:0/1529426479 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:16.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.438+0000 7f7bb77fe640 1 --2- 172.21.15.161:0/1529426479 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7bb80c5b20 0x7f7bb81ee3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:16.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.438+0000 7f7bb7fff640 1 --2- 172.21.15.161:0/1529426479 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bb8152890 0x7f7bb81ecac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:16.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.438+0000 7f7bb6ffd640 1 -- 172.21.15.161:0/1529426479 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bb8152890 msgr2=0x7f7bb81ecac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:16.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.438+0000 7f7bb6ffd640 1 --2- 172.21.15.161:0/1529426479 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bb8152890 0x7f7bb81ecac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.438+0000 7f7bb6ffd640 1 -- 172.21.15.161:0/1529426479 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7bb80c5b20 msgr2=0x7f7bb81ee3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:16.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.438+0000 7f7bb6ffd640 1 --2- 172.21.15.161:0/1529426479 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7bb80c5b20 0x7f7bb81ee3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.438+0000 7f7bb6ffd640 1 -- 172.21.15.161:0/1529426479 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bb81eedc0 con 0x7f7bb80c8ed0 2024-03-28T23:35:16.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.438+0000 7f7bb6ffd640 1 --2- 172.21.15.161:0/1529426479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bb80c8ed0 0x7f7bb81ec580 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f7bac00c430 tx=0x7f7bac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:16.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.438+0000 7f7b9ffff640 1 -- 172.21.15.161:0/1529426479 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bac017070 con 0x7f7bb80c8ed0 2024-03-28T23:35:16.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.438+0000 7f7b9ffff640 1 -- 172.21.15.161:0/1529426479 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bac0074e0 con 0x7f7bb80c8ed0 2024-03-28T23:35:16.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.438+0000 7f7bbe215640 1 -- 172.21.15.161:0/1529426479 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bb8153300 con 0x7f7bb80c8ed0 2024-03-28T23:35:16.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.438+0000 7f7b9ffff640 1 -- 172.21.15.161:0/1529426479 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bac0059e0 con 0x7f7bb80c8ed0 2024-03-28T23:35:16.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.438+0000 7f7bbe215640 1 -- 172.21.15.161:0/1529426479 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bb814e5f0 con 0x7f7bb80c8ed0 2024-03-28T23:35:16.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.439+0000 7f7bbe215640 1 -- 172.21.15.161:0/1529426479 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bb81ec0b0 con 0x7f7bb80c8ed0 2024-03-28T23:35:16.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.440+0000 7f7b9ffff640 1 -- 172.21.15.161:0/1529426479 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7bac01a070 con 0x7f7bb80c8ed0 2024-03-28T23:35:16.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.440+0000 7f7b9ffff640 1 --2- 172.21.15.161:0/1529426479 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7ba406b4d0 0x7f7ba406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:16.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.440+0000 7f7b9ffff640 1 -- 172.21.15.161:0/1529426479 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 6101+0+0 (secure 0 0 0) 0x7f7bac0896a0 con 0x7f7bb80c8ed0 2024-03-28T23:35:16.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.440+0000 7f7bb77fe640 1 --2- 172.21.15.161:0/1529426479 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7ba406b4d0 0x7f7ba406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:16.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.441+0000 7f7bb77fe640 1 --2- 172.21.15.161:0/1529426479 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7ba406b4d0 0x7f7ba406d990 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8036000 tx=0x7f7ba8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:16.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.445+0000 7f7b9ffff640 1 -- 172.21.15.161:0/1529426479 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7bac058420 con 0x7f7bb80c8ed0 2024-03-28T23:35:16.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.624+0000 7f7bbe215640 1 -- 172.21.15.161:0/1529426479 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7bb8152fa0 con 0x7f7bb80c8ed0 2024-03-28T23:35:16.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.625+0000 7f7b9ffff640 1 -- 172.21.15.161:0/1529426479 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v136) ==== 74+0+16928 (secure 0 0 0) 0x7f7bb8152fa0 con 0x7f7bb80c8ed0 2024-03-28T23:35:16.626 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:16.626 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":136,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:35:14.642185+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:35:11.208674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-28T23:35:11.954908+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"136","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:35:16.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.628+0000 7f7bbe215640 1 -- 172.21.15.161:0/1529426479 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7ba406b4d0 msgr2=0x7f7ba406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:16.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.628+0000 7f7bbe215640 1 --2- 172.21.15.161:0/1529426479 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7ba406b4d0 0x7f7ba406d990 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8036000 tx=0x7f7ba8008030 comp rx=0 tx=0).stop 2024-03-28T23:35:16.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.628+0000 7f7bbe215640 1 -- 172.21.15.161:0/1529426479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bb80c8ed0 msgr2=0x7f7bb81ec580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:16.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.628+0000 7f7bbe215640 1 --2- 172.21.15.161:0/1529426479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bb80c8ed0 0x7f7bb81ec580 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f7bac00c430 tx=0x7f7bac00c8f0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.628+0000 7f7bbe215640 1 -- 172.21.15.161:0/1529426479 shutdown_connections 2024-03-28T23:35:16.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.628+0000 7f7bbe215640 1 --2- 172.21.15.161:0/1529426479 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7ba406b4d0 0x7f7ba406d990 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.628+0000 7f7bbe215640 1 --2- 172.21.15.161:0/1529426479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bb80c8ed0 0x7f7bb81ec580 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.628+0000 7f7bbe215640 1 --2- 172.21.15.161:0/1529426479 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bb8152890 0x7f7bb81ecac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.628+0000 7f7bbe215640 1 --2- 172.21.15.161:0/1529426479 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7bb80c5b20 0x7f7bb81ee3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.628+0000 7f7bbe215640 1 -- 172.21.15.161:0/1529426479 >> 172.21.15.161:0/1529426479 conn(0x7f7bb80ac620 msgr2=0x7f7bb80c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:16.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.628+0000 7f7bbe215640 1 -- 172.21.15.161:0/1529426479 shutdown_connections 2024-03-28T23:35:16.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.628+0000 7f7bbe215640 1 -- 172.21.15.161:0/1529426479 wait complete. 2024-03-28T23:35:16.640 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T23:35:16.640 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:35:16.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.740+0000 7fef34fa0640 1 Processor -- start 2024-03-28T23:35:16.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.740+0000 7fef34fa0640 1 -- start start 2024-03-28T23:35:16.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.741+0000 7fef34fa0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fef30152850 0x7fef30159240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:16.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.741+0000 7fef34fa0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fef3006c090 0x7fef3006c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:16.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.741+0000 7fef34fa0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef30159780 con 0x7fef300c8ec0 2024-03-28T23:35:16.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.741+0000 7fef34fa0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fef30152c30 con 0x7fef30152850 2024-03-28T23:35:16.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.741+0000 7fef34fa0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fef301553e0 con 0x7fef3006c090 2024-03-28T23:35:16.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.741+0000 7fef2f7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fef30152850 0x7fef30159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:16.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.741+0000 7fef2effd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fef300c8ec0 0x7fef300c52b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45280/0 (socket says 172.21.15.161:45280) 2024-03-28T23:35:16.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.741+0000 7fef2f7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fef30152850 0x7fef30159240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47580/0 (socket says 172.21.15.161:47580) 2024-03-28T23:35:16.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.741+0000 7fef2effd640 1 -- 172.21.15.161:0/1544640703 learned_addr learned my addr 172.21.15.161:0/1544640703 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:16.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.742+0000 7fef2ffff640 1 --2- 172.21.15.161:0/1544640703 >> v2:172.21.15.170:3300/0 conn(0x7fef3006c090 0x7fef3006c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:16.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.742+0000 7fef2f7fe640 1 -- 172.21.15.161:0/1544640703 >> v2:172.21.15.170:3300/0 conn(0x7fef3006c090 msgr2=0x7fef3006c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:16.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.742+0000 7fef2f7fe640 1 --2- 172.21.15.161:0/1544640703 >> v2:172.21.15.170:3300/0 conn(0x7fef3006c090 0x7fef3006c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.742+0000 7fef2f7fe640 1 -- 172.21.15.161:0/1544640703 >> v1:172.21.15.161:6789/0 conn(0x7fef300c8ec0 legacy=0x7fef300c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:16.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.742+0000 7fef2f7fe640 1 -- 172.21.15.161:0/1544640703 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef301556c0 con 0x7fef30152850 2024-03-28T23:35:16.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.742+0000 7fef2ffff640 1 --2- 172.21.15.161:0/1544640703 >> v2:172.21.15.170:3300/0 conn(0x7fef3006c090 0x7fef3006c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:16.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.742+0000 7fef2f7fe640 1 --2- 172.21.15.161:0/1544640703 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef30152850 0x7fef30159240 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fef300c4df0 tx=0x7fef1802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56979a081ba5fdbb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:16.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.742+0000 7fef2e7fc640 1 -- 172.21.15.161:0/1544640703 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef180315a0 con 0x7fef30152850 2024-03-28T23:35:16.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.743+0000 7fef2e7fc640 1 -- 172.21.15.161:0/1544640703 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef18037070 con 0x7fef30152850 2024-03-28T23:35:16.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.743+0000 7fef2e7fc640 1 -- 172.21.15.161:0/1544640703 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef18031960 con 0x7fef30152850 2024-03-28T23:35:16.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.743+0000 7fef34fa0640 1 -- 172.21.15.161:0/1544640703 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef30152850 msgr2=0x7fef30159240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:16.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.743+0000 7fef34fa0640 1 --2- 172.21.15.161:0/1544640703 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef30152850 0x7fef30159240 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fef300c4df0 tx=0x7fef1802b840 comp rx=0 tx=0).stop 2024-03-28T23:35:16.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.743+0000 7fef34fa0640 1 -- 172.21.15.161:0/1544640703 shutdown_connections 2024-03-28T23:35:16.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.743+0000 7fef34fa0640 1 --2- 172.21.15.161:0/1544640703 >> v2:172.21.15.170:3300/0 conn(0x7fef3006c090 0x7fef3006c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.743+0000 7fef34fa0640 1 --2- 172.21.15.161:0/1544640703 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef30152850 0x7fef30159240 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.743+0000 7fef34fa0640 1 -- 172.21.15.161:0/1544640703 >> 172.21.15.161:0/1544640703 conn(0x7fef300ac4c0 msgr2=0x7fef300ac8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:16.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.743+0000 7fef34fa0640 1 -- 172.21.15.161:0/1544640703 shutdown_connections 2024-03-28T23:35:16.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.743+0000 7fef34fa0640 1 -- 172.21.15.161:0/1544640703 wait complete. 2024-03-28T23:35:16.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.745+0000 7fef34fa0640 1 Processor -- start 2024-03-28T23:35:16.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.745+0000 7fef34fa0640 1 -- start start 2024-03-28T23:35:16.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.746+0000 7fef34fa0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fef3006c090 0x7fef301ee970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:16.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.746+0000 7fef34fa0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef300c8ec0 0x7fef300ab720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:16.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.746+0000 7fef34fa0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fef30152850 0x7fef30148cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:16.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.746+0000 7fef34fa0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fef30155f60 con 0x7fef300c8ec0 2024-03-28T23:35:16.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.746+0000 7fef34fa0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fef30155c80 con 0x7fef30152850 2024-03-28T23:35:16.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.746+0000 7fef34fa0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fef300c59c0 con 0x7fef3006c090 2024-03-28T23:35:16.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.746+0000 7fef2effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef300c8ec0 0x7fef300ab720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:16.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.746+0000 7fef2effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef300c8ec0 0x7fef300ab720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47596/0 (socket says 172.21.15.161:47596) 2024-03-28T23:35:16.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.746+0000 7fef2effd640 1 -- 172.21.15.161:0/1715971859 learned_addr learned my addr 172.21.15.161:0/1715971859 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:16.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.746+0000 7fef2ffff640 1 --2- 172.21.15.161:0/1715971859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fef30152850 0x7fef30148cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:16.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.747+0000 7fef2f7fe640 1 --2- 172.21.15.161:0/1715971859 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fef3006c090 0x7fef301ee970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:16.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.747+0000 7fef2effd640 1 -- 172.21.15.161:0/1715971859 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fef3006c090 msgr2=0x7fef301ee970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:16.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.747+0000 7fef2effd640 1 --2- 172.21.15.161:0/1715971859 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fef3006c090 0x7fef301ee970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.747+0000 7fef2effd640 1 -- 172.21.15.161:0/1715971859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fef30152850 msgr2=0x7fef30148cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:16.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.747+0000 7fef2effd640 1 --2- 172.21.15.161:0/1715971859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fef30152850 0x7fef30148cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.747+0000 7fef2effd640 1 -- 172.21.15.161:0/1715971859 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef301566f0 con 0x7fef300c8ec0 2024-03-28T23:35:16.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.747+0000 7fef2ffff640 1 --2- 172.21.15.161:0/1715971859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fef30152850 0x7fef30148cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:16.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.747+0000 7fef2f7fe640 1 --2- 172.21.15.161:0/1715971859 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fef3006c090 0x7fef301ee970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:16.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.747+0000 7fef2effd640 1 --2- 172.21.15.161:0/1715971859 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef300c8ec0 0x7fef300ab720 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fef2000c400 tx=0x7fef2000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:16.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.747+0000 7fef27fff640 1 -- 172.21.15.161:0/1715971859 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef20017070 con 0x7fef300c8ec0 2024-03-28T23:35:16.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.747+0000 7fef27fff640 1 -- 172.21.15.161:0/1715971859 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef20005ce0 con 0x7fef300c8ec0 2024-03-28T23:35:16.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.747+0000 7fef27fff640 1 -- 172.21.15.161:0/1715971859 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef200072f0 con 0x7fef300c8ec0 2024-03-28T23:35:16.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.747+0000 7fef34fa0640 1 -- 172.21.15.161:0/1715971859 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef301569d0 con 0x7fef300c8ec0 2024-03-28T23:35:16.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.747+0000 7fef34fa0640 1 -- 172.21.15.161:0/1715971859 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef300abc60 con 0x7fef300c8ec0 2024-03-28T23:35:16.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.748+0000 7fef34fa0640 1 -- 172.21.15.161:0/1715971859 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef30067970 con 0x7fef300c8ec0 2024-03-28T23:35:16.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.749+0000 7fef27fff640 1 -- 172.21.15.161:0/1715971859 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fef2001a070 con 0x7fef300c8ec0 2024-03-28T23:35:16.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.749+0000 7fef27fff640 1 --2- 172.21.15.161:0/1715971859 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fef1006b4d0 0x7fef1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:16.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.749+0000 7fef27fff640 1 -- 172.21.15.161:0/1715971859 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 6101+0+0 (secure 0 0 0) 0x7fef200896b0 con 0x7fef300c8ec0 2024-03-28T23:35:16.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.749+0000 7fef2f7fe640 1 --2- 172.21.15.161:0/1715971859 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fef1006b4d0 0x7fef1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:16.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.749+0000 7fef2f7fe640 1 --2- 172.21.15.161:0/1715971859 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fef1006b4d0 0x7fef1006d990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fef18036000 tx=0x7fef18008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:16.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.757+0000 7fef27fff640 1 -- 172.21.15.161:0/1715971859 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fef30067970 con 0x7fef300c8ec0 2024-03-28T23:35:16.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.932+0000 7fef34fa0640 1 -- 172.21.15.161:0/1715971859 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fef301564e0 con 0x7fef1006b4d0 2024-03-28T23:35:16.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.938+0000 7fef27fff640 1 -- 172.21.15.161:0/1715971859 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252494 (secure 0 0 0) 0x7fef301564e0 con 0x7fef1006b4d0 2024-03-28T23:35:16.941 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:16.943 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:35:16.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.941+0000 7fef34fa0640 1 -- 172.21.15.161:0/1715971859 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fef1006b4d0 msgr2=0x7fef1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:16.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.941+0000 7fef34fa0640 1 --2- 172.21.15.161:0/1715971859 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fef1006b4d0 0x7fef1006d990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fef18036000 tx=0x7fef18008030 comp rx=0 tx=0).stop 2024-03-28T23:35:16.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.941+0000 7fef34fa0640 1 -- 172.21.15.161:0/1715971859 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef300c8ec0 msgr2=0x7fef300ab720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:16.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.941+0000 7fef34fa0640 1 --2- 172.21.15.161:0/1715971859 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef300c8ec0 0x7fef300ab720 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fef2000c400 tx=0x7fef2000c8c0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.941+0000 7fef34fa0640 1 -- 172.21.15.161:0/1715971859 shutdown_connections 2024-03-28T23:35:16.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.941+0000 7fef34fa0640 1 --2- 172.21.15.161:0/1715971859 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fef1006b4d0 0x7fef1006d990 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.941+0000 7fef34fa0640 1 --2- 172.21.15.161:0/1715971859 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef300c8ec0 0x7fef300ab720 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.941+0000 7fef34fa0640 1 --2- 172.21.15.161:0/1715971859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fef30152850 0x7fef30148cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.941+0000 7fef34fa0640 1 --2- 172.21.15.161:0/1715971859 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fef3006c090 0x7fef301ee970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:16.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.941+0000 7fef34fa0640 1 -- 172.21.15.161:0/1715971859 >> 172.21.15.161:0/1715971859 conn(0x7fef300ac4c0 msgr2=0x7fef300c51b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:16.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.941+0000 7fef34fa0640 1 -- 172.21.15.161:0/1715971859 shutdown_connections 2024-03-28T23:35:16.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:16.941+0000 7fef34fa0640 1 -- 172.21.15.161:0/1715971859 wait complete. 2024-03-28T23:35:16.953 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2024-03-28T23:35:15.374824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262900,"kb_used_data":13404,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754711820,"statfs":{"total":773094113280,"available":772824903680,"internally_reserved":0,"allocated":13725696,"data_stored":7717463,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000623"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.668039+0000","last_change":"2024-03-28T23:35:13.667376+0000","last_active":"2024-03-28T23:35:13.668039+0000","last_peered":"2024-03-28T23:35:13.668039+0000","last_clean":"2024-03-28T23:35:13.668039+0000","last_became_active":"2024-03-28T23:35:13.667026+0000","last_became_peered":"2024-03-28T23:35:13.667026+0000","last_unstale":"2024-03-28T23:35:13.668039+0000","last_undegraded":"2024-03-28T23:35:13.668039+0000","last_fullsized":"2024-03-28T23:35:13.668039+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:42:35.979766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.647675+0000","last_change":"2024-03-28T23:35:13.645679+0000","last_active":"2024-03-28T23:35:13.647675+0000","last_peered":"2024-03-28T23:35:13.647675+0000","last_clean":"2024-03-28T23:35:13.647675+0000","last_became_active":"2024-03-28T23:35:13.645424+0000","last_became_peered":"2024-03-28T23:35:13.645424+0000","last_unstale":"2024-03-28T23:35:13.647675+0000","last_undegraded":"2024-03-28T23:35:13.647675+0000","last_fullsized":"2024-03-28T23:35:13.647675+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:23:21.067376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.647705+0000","last_change":"2024-03-28T23:35:13.645175+0000","last_active":"2024-03-28T23:35:13.647705+0000","last_peered":"2024-03-28T23:35:13.647705+0000","last_clean":"2024-03-28T23:35:13.647705+0000","last_became_active":"2024-03-28T23:35:13.644892+0000","last_became_peered":"2024-03-28T23:35:13.644892+0000","last_unstale":"2024-03-28T23:35:13.647705+0000","last_undegraded":"2024-03-28T23:35:13.647705+0000","last_fullsized":"2024-03-28T23:35:13.647705+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:18:00.612518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.652351+0000","last_change":"2024-03-28T23:35:13.650727+0000","last_active":"2024-03-28T23:35:13.652351+0000","last_peered":"2024-03-28T23:35:13.652351+0000","last_clean":"2024-03-28T23:35:13.652351+0000","last_became_active":"2024-03-28T23:35:13.650440+0000","last_became_peered":"2024-03-28T23:35:13.650440+0000","last_unstale":"2024-03-28T23:35:13.652351+0000","last_undegraded":"2024-03-28T23:35:13.652351+0000","last_fullsized":"2024-03-28T23:35:13.652351+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:29.284277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.663136+0000","last_change":"2024-03-28T23:35:13.658789+0000","last_active":"2024-03-28T23:35:13.663136+0000","last_peered":"2024-03-28T23:35:13.663136+0000","last_clean":"2024-03-28T23:35:13.663136+0000","last_became_active":"2024-03-28T23:35:13.658102+0000","last_became_peered":"2024-03-28T23:35:13.658102+0000","last_unstale":"2024-03-28T23:35:13.663136+0000","last_undegraded":"2024-03-28T23:35:13.663136+0000","last_fullsized":"2024-03-28T23:35:13.663136+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:21:58.117340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.662162+0000","last_change":"2024-03-28T23:35:13.657489+0000","last_active":"2024-03-28T23:35:13.662162+0000","last_peered":"2024-03-28T23:35:13.662162+0000","last_clean":"2024-03-28T23:35:13.662162+0000","last_became_active":"2024-03-28T23:35:13.657241+0000","last_became_peered":"2024-03-28T23:35:13.657241+0000","last_unstale":"2024-03-28T23:35:13.662162+0000","last_undegraded":"2024-03-28T23:35:13.662162+0000","last_fullsized":"2024-03-28T23:35:13.662162+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:00:37.265722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.661451+0000","last_change":"2024-03-28T23:35:13.658858+0000","last_active":"2024-03-28T23:35:13.661451+0000","last_peered":"2024-03-28T23:35:13.661451+0000","last_clean":"2024-03-28T23:35:13.661451+0000","last_became_active":"2024-03-28T23:35:13.658594+0000","last_became_peered":"2024-03-28T23:35:13.658594+0000","last_unstale":"2024-03-28T23:35:13.661451+0000","last_undegraded":"2024-03-28T23:35:13.661451+0000","last_fullsized":"2024-03-28T23:35:13.661451+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:10.144427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.653436+0000","last_change":"2024-03-28T23:35:13.650219+0000","last_active":"2024-03-28T23:35:13.653436+0000","last_peered":"2024-03-28T23:35:13.653436+0000","last_clean":"2024-03-28T23:35:13.653436+0000","last_became_active":"2024-03-28T23:35:13.649864+0000","last_became_peered":"2024-03-28T23:35:13.649864+0000","last_unstale":"2024-03-28T23:35:13.653436+0000","last_undegraded":"2024-03-28T23:35:13.653436+0000","last_fullsized":"2024-03-28T23:35:13.653436+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:56:33.475772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.660216+0000","last_change":"2024-03-28T23:35:13.654353+0000","last_active":"2024-03-28T23:35:13.660216+0000","last_peered":"2024-03-28T23:35:13.660216+0000","last_clean":"2024-03-28T23:35:13.660216+0000","last_became_active":"2024-03-28T23:35:13.654090+0000","last_became_peered":"2024-03-28T23:35:13.654090+0000","last_unstale":"2024-03-28T23:35:13.660216+0000","last_undegraded":"2024-03-28T23:35:13.660216+0000","last_fullsized":"2024-03-28T23:35:13.660216+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:52:35.593391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.669840+0000","last_change":"2024-03-28T23:35:13.663286+0000","last_active":"2024-03-28T23:35:13.669840+0000","last_peered":"2024-03-28T23:35:13.669840+0000","last_clean":"2024-03-28T23:35:13.669840+0000","last_became_active":"2024-03-28T23:35:13.662622+0000","last_became_peered":"2024-03-28T23:35:13.662622+0000","last_unstale":"2024-03-28T23:35:13.669840+0000","last_undegraded":"2024-03-28T23:35:13.669840+0000","last_fullsized":"2024-03-28T23:35:13.669840+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:11:04.347579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.648301+0000","last_change":"2024-03-28T23:35:13.642520+0000","last_active":"2024-03-28T23:35:13.648301+0000","last_peered":"2024-03-28T23:35:13.648301+0000","last_clean":"2024-03-28T23:35:13.648301+0000","last_became_active":"2024-03-28T23:35:13.642244+0000","last_became_peered":"2024-03-28T23:35:13.642244+0000","last_unstale":"2024-03-28T23:35:13.648301+0000","last_undegraded":"2024-03-28T23:35:13.648301+0000","last_fullsized":"2024-03-28T23:35:13.648301+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:59:33.982931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.665326+0000","last_change":"2024-03-28T23:35:13.661843+0000","last_active":"2024-03-28T23:35:13.665326+0000","last_peered":"2024-03-28T23:35:13.665326+0000","last_clean":"2024-03-28T23:35:13.665326+0000","last_became_active":"2024-03-28T23:35:13.661276+0000","last_became_peered":"2024-03-28T23:35:13.661276+0000","last_unstale":"2024-03-28T23:35:13.665326+0000","last_undegraded":"2024-03-28T23:35:13.665326+0000","last_fullsized":"2024-03-28T23:35:13.665326+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:06:08.472963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.669867+0000","last_change":"2024-03-28T23:35:13.664375+0000","last_active":"2024-03-28T23:35:13.669867+0000","last_peered":"2024-03-28T23:35:13.669867+0000","last_clean":"2024-03-28T23:35:13.669867+0000","last_became_active":"2024-03-28T23:35:13.663245+0000","last_became_peered":"2024-03-28T23:35:13.663245+0000","last_unstale":"2024-03-28T23:35:13.669867+0000","last_undegraded":"2024-03-28T23:35:13.669867+0000","last_fullsized":"2024-03-28T23:35:13.669867+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:56:20.856416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.653459+0000","last_change":"2024-03-28T23:35:13.651904+0000","last_active":"2024-03-28T23:35:13.653459+0000","last_peered":"2024-03-28T23:35:13.653459+0000","last_clean":"2024-03-28T23:35:13.653459+0000","last_became_active":"2024-03-28T23:35:13.651636+0000","last_became_peered":"2024-03-28T23:35:13.651636+0000","last_unstale":"2024-03-28T23:35:13.653459+0000","last_undegraded":"2024-03-28T23:35:13.653459+0000","last_fullsized":"2024-03-28T23:35:13.653459+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:52:06.037363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.658856+0000","last_change":"2024-03-28T23:35:13.651843+0000","last_active":"2024-03-28T23:35:13.658856+0000","last_peered":"2024-03-28T23:35:13.658856+0000","last_clean":"2024-03-28T23:35:13.658856+0000","last_became_active":"2024-03-28T23:35:13.651568+0000","last_became_peered":"2024-03-28T23:35:13.651568+0000","last_unstale":"2024-03-28T23:35:13.658856+0000","last_undegraded":"2024-03-28T23:35:13.658856+0000","last_fullsized":"2024-03-28T23:35:13.658856+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:20:20.561594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.666842+0000","last_change":"2024-03-28T23:35:13.660451+0000","last_active":"2024-03-28T23:35:13.666842+0000","last_peered":"2024-03-28T23:35:13.666842+0000","last_clean":"2024-03-28T23:35:13.666842+0000","last_became_active":"2024-03-28T23:35:13.660166+0000","last_became_peered":"2024-03-28T23:35:13.660166+0000","last_unstale":"2024-03-28T23:35:13.666842+0000","last_undegraded":"2024-03-28T23:35:13.666842+0000","last_fullsized":"2024-03-28T23:35:13.666842+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:12:40.626571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.659006+0000","last_change":"2024-03-28T23:35:13.656742+0000","last_active":"2024-03-28T23:35:13.659006+0000","last_peered":"2024-03-28T23:35:13.659006+0000","last_clean":"2024-03-28T23:35:13.659006+0000","last_became_active":"2024-03-28T23:35:13.655664+0000","last_became_peered":"2024-03-28T23:35:13.655664+0000","last_unstale":"2024-03-28T23:35:13.659006+0000","last_undegraded":"2024-03-28T23:35:13.659006+0000","last_fullsized":"2024-03-28T23:35:13.659006+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:48:23.811667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.669881+0000","last_change":"2024-03-28T23:35:13.664804+0000","last_active":"2024-03-28T23:35:13.669881+0000","last_peered":"2024-03-28T23:35:13.669881+0000","last_clean":"2024-03-28T23:35:13.669881+0000","last_became_active":"2024-03-28T23:35:13.664532+0000","last_became_peered":"2024-03-28T23:35:13.664532+0000","last_unstale":"2024-03-28T23:35:13.669881+0000","last_undegraded":"2024-03-28T23:35:13.669881+0000","last_fullsized":"2024-03-28T23:35:13.669881+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:44:13.076565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.651799+0000","last_change":"2024-03-28T23:35:13.648461+0000","last_active":"2024-03-28T23:35:13.651799+0000","last_peered":"2024-03-28T23:35:13.651799+0000","last_clean":"2024-03-28T23:35:13.651799+0000","last_became_active":"2024-03-28T23:35:13.648183+0000","last_became_peered":"2024-03-28T23:35:13.648183+0000","last_unstale":"2024-03-28T23:35:13.651799+0000","last_undegraded":"2024-03-28T23:35:13.651799+0000","last_fullsized":"2024-03-28T23:35:13.651799+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:17.724044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.662399+0000","last_change":"2024-03-28T23:35:13.651360+0000","last_active":"2024-03-28T23:35:13.662399+0000","last_peered":"2024-03-28T23:35:13.662399+0000","last_clean":"2024-03-28T23:35:13.662399+0000","last_became_active":"2024-03-28T23:35:13.651068+0000","last_became_peered":"2024-03-28T23:35:13.651068+0000","last_unstale":"2024-03-28T23:35:13.662399+0000","last_undegraded":"2024-03-28T23:35:13.662399+0000","last_fullsized":"2024-03-28T23:35:13.662399+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:05.948124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.656699+0000","last_change":"2024-03-28T23:35:13.650915+0000","last_active":"2024-03-28T23:35:13.656699+0000","last_peered":"2024-03-28T23:35:13.656699+0000","last_clean":"2024-03-28T23:35:13.656699+0000","last_became_active":"2024-03-28T23:35:13.650635+0000","last_became_peered":"2024-03-28T23:35:13.650635+0000","last_unstale":"2024-03-28T23:35:13.656699+0000","last_undegraded":"2024-03-28T23:35:13.656699+0000","last_fullsized":"2024-03-28T23:35:13.656699+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:43:07.597654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.661340+0000","last_change":"2024-03-28T23:35:13.659128+0000","last_active":"2024-03-28T23:35:13.661340+0000","last_peered":"2024-03-28T23:35:13.661340+0000","last_clean":"2024-03-28T23:35:13.661340+0000","last_became_active":"2024-03-28T23:35:13.658777+0000","last_became_peered":"2024-03-28T23:35:13.658777+0000","last_unstale":"2024-03-28T23:35:13.661340+0000","last_undegraded":"2024-03-28T23:35:13.661340+0000","last_fullsized":"2024-03-28T23:35:13.661340+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:05.801713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.666486+0000","last_change":"2024-03-28T23:35:13.664261+0000","last_active":"2024-03-28T23:35:13.666486+0000","last_peered":"2024-03-28T23:35:13.666486+0000","last_clean":"2024-03-28T23:35:13.666486+0000","last_became_active":"2024-03-28T23:35:13.663991+0000","last_became_peered":"2024-03-28T23:35:13.663991+0000","last_unstale":"2024-03-28T23:35:13.666486+0000","last_undegraded":"2024-03-28T23:35:13.666486+0000","last_fullsized":"2024-03-28T23:35:13.666486+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:05.801713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.664471+0000","last_change":"2024-03-28T23:35:13.660718+0000","last_active":"2024-03-28T23:35:13.664471+0000","last_peered":"2024-03-28T23:35:13.664471+0000","last_clean":"2024-03-28T23:35:13.664471+0000","last_became_active":"2024-03-28T23:35:13.660433+0000","last_became_peered":"2024-03-28T23:35:13.660433+0000","last_unstale":"2024-03-28T23:35:13.664471+0000","last_undegraded":"2024-03-28T23:35:13.664471+0000","last_fullsized":"2024-03-28T23:35:13.664471+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:30:44.583627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.667969+0000","last_change":"2024-03-28T23:35:13.666123+0000","last_active":"2024-03-28T23:35:13.667969+0000","last_peered":"2024-03-28T23:35:13.667969+0000","last_clean":"2024-03-28T23:35:13.667969+0000","last_became_active":"2024-03-28T23:35:13.665852+0000","last_became_peered":"2024-03-28T23:35:13.665852+0000","last_unstale":"2024-03-28T23:35:13.667969+0000","last_undegraded":"2024-03-28T23:35:13.667969+0000","last_fullsized":"2024-03-28T23:35:13.667969+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:23:21.067376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.666743+0000","last_change":"2024-03-28T23:35:13.663983+0000","last_active":"2024-03-28T23:35:13.666743+0000","last_peered":"2024-03-28T23:35:13.666743+0000","last_clean":"2024-03-28T23:35:13.666743+0000","last_became_active":"2024-03-28T23:35:13.663503+0000","last_became_peered":"2024-03-28T23:35:13.663503+0000","last_unstale":"2024-03-28T23:35:13.666743+0000","last_undegraded":"2024-03-28T23:35:13.666743+0000","last_fullsized":"2024-03-28T23:35:13.666743+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:18:00.612518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.660166+0000","last_change":"2024-03-28T23:35:13.653802+0000","last_active":"2024-03-28T23:35:13.660166+0000","last_peered":"2024-03-28T23:35:13.660166+0000","last_clean":"2024-03-28T23:35:13.660166+0000","last_became_active":"2024-03-28T23:35:13.653491+0000","last_became_peered":"2024-03-28T23:35:13.653491+0000","last_unstale":"2024-03-28T23:35:13.660166+0000","last_undegraded":"2024-03-28T23:35:13.660166+0000","last_fullsized":"2024-03-28T23:35:13.660166+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:59:31.065907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.656594+0000","last_change":"2024-03-28T23:35:13.655570+0000","last_active":"2024-03-28T23:35:13.656594+0000","last_peered":"2024-03-28T23:35:13.656594+0000","last_clean":"2024-03-28T23:35:13.656594+0000","last_became_active":"2024-03-28T23:35:13.655310+0000","last_became_peered":"2024-03-28T23:35:13.655310+0000","last_unstale":"2024-03-28T23:35:13.656594+0000","last_undegraded":"2024-03-28T23:35:13.656594+0000","last_fullsized":"2024-03-28T23:35:13.656594+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:56.745797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.663117+0000","last_change":"2024-03-28T23:35:13.660800+0000","last_active":"2024-03-28T23:35:13.663117+0000","last_peered":"2024-03-28T23:35:13.663117+0000","last_clean":"2024-03-28T23:35:13.663117+0000","last_became_active":"2024-03-28T23:35:13.660312+0000","last_became_peered":"2024-03-28T23:35:13.660312+0000","last_unstale":"2024-03-28T23:35:13.663117+0000","last_undegraded":"2024-03-28T23:35:13.663117+0000","last_fullsized":"2024-03-28T23:35:13.663117+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:56:55.719063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.653094+0000","last_change":"2024-03-28T23:35:13.648911+0000","last_active":"2024-03-28T23:35:13.653094+0000","last_peered":"2024-03-28T23:35:13.653094+0000","last_clean":"2024-03-28T23:35:13.653094+0000","last_became_active":"2024-03-28T23:35:13.648655+0000","last_became_peered":"2024-03-28T23:35:13.648655+0000","last_unstale":"2024-03-28T23:35:13.653094+0000","last_undegraded":"2024-03-28T23:35:13.653094+0000","last_fullsized":"2024-03-28T23:35:13.653094+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:35:48.165039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.662417+0000","last_change":"2024-03-28T23:35:13.658385+0000","last_active":"2024-03-28T23:35:13.662417+0000","last_peered":"2024-03-28T23:35:13.662417+0000","last_clean":"2024-03-28T23:35:13.662417+0000","last_became_active":"2024-03-28T23:35:13.658094+0000","last_became_peered":"2024-03-28T23:35:13.658094+0000","last_unstale":"2024-03-28T23:35:13.662417+0000","last_undegraded":"2024-03-28T23:35:13.662417+0000","last_fullsized":"2024-03-28T23:35:13.662417+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:26:10.661199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.653347+0000","last_change":"2024-03-28T23:35:13.649502+0000","last_active":"2024-03-28T23:35:13.653347+0000","last_peered":"2024-03-28T23:35:13.653347+0000","last_clean":"2024-03-28T23:35:13.653347+0000","last_became_active":"2024-03-28T23:35:13.649255+0000","last_became_peered":"2024-03-28T23:35:13.649255+0000","last_unstale":"2024-03-28T23:35:13.653347+0000","last_undegraded":"2024-03-28T23:35:13.653347+0000","last_fullsized":"2024-03-28T23:35:13.653347+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:49:07.618234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.647576+0000","last_change":"2024-03-28T23:35:13.643224+0000","last_active":"2024-03-28T23:35:13.647576+0000","last_peered":"2024-03-28T23:35:13.647576+0000","last_clean":"2024-03-28T23:35:13.647576+0000","last_became_active":"2024-03-28T23:35:13.642959+0000","last_became_peered":"2024-03-28T23:35:13.642959+0000","last_unstale":"2024-03-28T23:35:13.647576+0000","last_undegraded":"2024-03-28T23:35:13.647576+0000","last_fullsized":"2024-03-28T23:35:13.647576+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:08.266698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"135'4","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.663708+0000","last_change":"2024-03-28T23:35:12.639567+0000","last_active":"2024-03-28T23:35:13.663708+0000","last_peered":"2024-03-28T23:35:13.663708+0000","last_clean":"2024-03-28T23:35:13.663708+0000","last_became_active":"2024-03-28T23:35:12.639270+0000","last_became_peered":"2024-03-28T23:35:12.639270+0000","last_unstale":"2024-03-28T23:35:13.663708+0000","last_undegraded":"2024-03-28T23:35:13.663708+0000","last_fullsized":"2024-03-28T23:35:13.663708+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:09:30.480929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.658488+0000","last_change":"2024-03-28T23:35:13.651426+0000","last_active":"2024-03-28T23:35:13.658488+0000","last_peered":"2024-03-28T23:35:13.658488+0000","last_clean":"2024-03-28T23:35:13.658488+0000","last_became_active":"2024-03-28T23:35:13.651155+0000","last_became_peered":"2024-03-28T23:35:13.651155+0000","last_unstale":"2024-03-28T23:35:13.658488+0000","last_undegraded":"2024-03-28T23:35:13.658488+0000","last_fullsized":"2024-03-28T23:35:13.658488+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:38.174144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"135'1","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.665288+0000","last_change":"2024-03-28T23:35:12.642892+0000","last_active":"2024-03-28T23:35:13.665288+0000","last_peered":"2024-03-28T23:35:13.665288+0000","last_clean":"2024-03-28T23:35:13.665288+0000","last_became_active":"2024-03-28T23:35:12.642271+0000","last_became_peered":"2024-03-28T23:35:12.642271+0000","last_unstale":"2024-03-28T23:35:13.665288+0000","last_undegraded":"2024-03-28T23:35:13.665288+0000","last_fullsized":"2024-03-28T23:35:13.665288+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:06.777884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.649003+0000","last_change":"2024-03-28T23:35:13.648277+0000","last_active":"2024-03-28T23:35:13.649003+0000","last_peered":"2024-03-28T23:35:13.649003+0000","last_clean":"2024-03-28T23:35:13.649003+0000","last_became_active":"2024-03-28T23:35:13.647949+0000","last_became_peered":"2024-03-28T23:35:13.647949+0000","last_unstale":"2024-03-28T23:35:13.649003+0000","last_undegraded":"2024-03-28T23:35:13.649003+0000","last_fullsized":"2024-03-28T23:35:13.649003+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:44:30.945933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"135'2","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.664918+0000","last_change":"2024-03-28T23:35:12.640335+0000","last_active":"2024-03-28T23:35:13.664918+0000","last_peered":"2024-03-28T23:35:13.664918+0000","last_clean":"2024-03-28T23:35:13.664918+0000","last_became_active":"2024-03-28T23:35:12.639942+0000","last_became_peered":"2024-03-28T23:35:12.639942+0000","last_unstale":"2024-03-28T23:35:13.664918+0000","last_undegraded":"2024-03-28T23:35:13.664918+0000","last_fullsized":"2024-03-28T23:35:13.664918+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:29:30.580679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.666598+0000","last_change":"2024-03-28T23:35:13.664926+0000","last_active":"2024-03-28T23:35:13.666598+0000","last_peered":"2024-03-28T23:35:13.666598+0000","last_clean":"2024-03-28T23:35:13.666598+0000","last_became_active":"2024-03-28T23:35:13.663695+0000","last_became_peered":"2024-03-28T23:35:13.663695+0000","last_unstale":"2024-03-28T23:35:13.666598+0000","last_undegraded":"2024-03-28T23:35:13.666598+0000","last_fullsized":"2024-03-28T23:35:13.666598+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:52:35.593391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"135'2","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.663945+0000","last_change":"2024-03-28T23:35:12.644928+0000","last_active":"2024-03-28T23:35:13.663945+0000","last_peered":"2024-03-28T23:35:13.663945+0000","last_clean":"2024-03-28T23:35:13.663945+0000","last_became_active":"2024-03-28T23:35:12.644436+0000","last_became_peered":"2024-03-28T23:35:12.644436+0000","last_unstale":"2024-03-28T23:35:13.663945+0000","last_undegraded":"2024-03-28T23:35:13.663945+0000","last_fullsized":"2024-03-28T23:35:13.663945+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:56:54.712369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.660459+0000","last_change":"2024-03-28T23:35:13.649833+0000","last_active":"2024-03-28T23:35:13.660459+0000","last_peered":"2024-03-28T23:35:13.660459+0000","last_clean":"2024-03-28T23:35:13.660459+0000","last_became_active":"2024-03-28T23:35:13.649547+0000","last_became_peered":"2024-03-28T23:35:13.649547+0000","last_unstale":"2024-03-28T23:35:13.660459+0000","last_undegraded":"2024-03-28T23:35:13.660459+0000","last_fullsized":"2024-03-28T23:35:13.660459+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:45:31.073396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"135'1","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.667150+0000","last_change":"2024-03-28T23:35:12.648684+0000","last_active":"2024-03-28T23:35:13.667150+0000","last_peered":"2024-03-28T23:35:13.667150+0000","last_clean":"2024-03-28T23:35:13.667150+0000","last_became_active":"2024-03-28T23:35:12.648422+0000","last_became_peered":"2024-03-28T23:35:12.648422+0000","last_unstale":"2024-03-28T23:35:13.667150+0000","last_undegraded":"2024-03-28T23:35:13.667150+0000","last_fullsized":"2024-03-28T23:35:13.667150+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:36:17.817296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.654283+0000","last_change":"2024-03-28T23:35:13.653558+0000","last_active":"2024-03-28T23:35:13.654283+0000","last_peered":"2024-03-28T23:35:13.654283+0000","last_clean":"2024-03-28T23:35:13.654283+0000","last_became_active":"2024-03-28T23:35:13.653292+0000","last_became_peered":"2024-03-28T23:35:13.653292+0000","last_unstale":"2024-03-28T23:35:13.654283+0000","last_undegraded":"2024-03-28T23:35:13.654283+0000","last_fullsized":"2024-03-28T23:35:13.654283+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:28:59.847157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"135'1","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.663988+0000","last_change":"2024-03-28T23:35:12.643414+0000","last_active":"2024-03-28T23:35:13.663988+0000","last_peered":"2024-03-28T23:35:13.663988+0000","last_clean":"2024-03-28T23:35:13.663988+0000","last_became_active":"2024-03-28T23:35:12.643155+0000","last_became_peered":"2024-03-28T23:35:12.643155+0000","last_unstale":"2024-03-28T23:35:13.663988+0000","last_undegraded":"2024-03-28T23:35:13.663988+0000","last_fullsized":"2024-03-28T23:35:13.663988+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:06.777884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.658457+0000","last_change":"2024-03-28T23:35:13.655141+0000","last_active":"2024-03-28T23:35:13.658457+0000","last_peered":"2024-03-28T23:35:13.658457+0000","last_clean":"2024-03-28T23:35:13.658457+0000","last_became_active":"2024-03-28T23:35:13.651870+0000","last_became_peered":"2024-03-28T23:35:13.651870+0000","last_unstale":"2024-03-28T23:35:13.658457+0000","last_undegraded":"2024-03-28T23:35:13.658457+0000","last_fullsized":"2024-03-28T23:35:13.658457+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:43:52.639889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.658232+0000","last_change":"2024-03-28T23:35:12.646919+0000","last_active":"2024-03-28T23:35:13.658232+0000","last_peered":"2024-03-28T23:35:13.658232+0000","last_clean":"2024-03-28T23:35:13.658232+0000","last_became_active":"2024-03-28T23:35:12.646465+0000","last_became_peered":"2024-03-28T23:35:12.646465+0000","last_unstale":"2024-03-28T23:35:13.658232+0000","last_undegraded":"2024-03-28T23:35:13.658232+0000","last_fullsized":"2024-03-28T23:35:13.658232+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:20:05.303840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.661875+0000","last_change":"2024-03-28T23:35:13.660220+0000","last_active":"2024-03-28T23:35:13.661875+0000","last_peered":"2024-03-28T23:35:13.661875+0000","last_clean":"2024-03-28T23:35:13.661875+0000","last_became_active":"2024-03-28T23:35:13.659947+0000","last_became_peered":"2024-03-28T23:35:13.659947+0000","last_unstale":"2024-03-28T23:35:13.661875+0000","last_undegraded":"2024-03-28T23:35:13.661875+0000","last_fullsized":"2024-03-28T23:35:13.661875+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:08.266698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.648958+0000","last_change":"2024-03-28T23:35:12.642121+0000","last_active":"2024-03-28T23:35:13.648958+0000","last_peered":"2024-03-28T23:35:13.648958+0000","last_clean":"2024-03-28T23:35:13.648958+0000","last_became_active":"2024-03-28T23:35:12.641797+0000","last_became_peered":"2024-03-28T23:35:12.641797+0000","last_unstale":"2024-03-28T23:35:13.648958+0000","last_undegraded":"2024-03-28T23:35:13.648958+0000","last_fullsized":"2024-03-28T23:35:13.648958+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:58.669498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.653469+0000","last_change":"2024-03-28T23:35:13.651744+0000","last_active":"2024-03-28T23:35:13.653469+0000","last_peered":"2024-03-28T23:35:13.653469+0000","last_clean":"2024-03-28T23:35:13.653469+0000","last_became_active":"2024-03-28T23:35:13.651412+0000","last_became_peered":"2024-03-28T23:35:13.651412+0000","last_unstale":"2024-03-28T23:35:13.653469+0000","last_undegraded":"2024-03-28T23:35:13.653469+0000","last_fullsized":"2024-03-28T23:35:13.653469+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:45:59.335574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.639452+0000","last_change":"2024-03-28T23:35:12.640263+0000","last_active":"2024-03-28T23:35:13.639452+0000","last_peered":"2024-03-28T23:35:13.639452+0000","last_clean":"2024-03-28T23:35:13.639452+0000","last_became_active":"2024-03-28T23:35:12.639948+0000","last_became_peered":"2024-03-28T23:35:12.639948+0000","last_unstale":"2024-03-28T23:35:13.639452+0000","last_undegraded":"2024-03-28T23:35:13.639452+0000","last_fullsized":"2024-03-28T23:35:13.639452+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:58:42.969626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.648325+0000","last_change":"2024-03-28T23:35:13.644780+0000","last_active":"2024-03-28T23:35:13.648325+0000","last_peered":"2024-03-28T23:35:13.648325+0000","last_clean":"2024-03-28T23:35:13.648325+0000","last_became_active":"2024-03-28T23:35:13.644502+0000","last_became_peered":"2024-03-28T23:35:13.644502+0000","last_unstale":"2024-03-28T23:35:13.648325+0000","last_undegraded":"2024-03-28T23:35:13.648325+0000","last_fullsized":"2024-03-28T23:35:13.648325+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:13:25.031142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.659108+0000","last_change":"2024-03-28T23:35:12.638492+0000","last_active":"2024-03-28T23:35:13.659108+0000","last_peered":"2024-03-28T23:35:13.659108+0000","last_clean":"2024-03-28T23:35:13.659108+0000","last_became_active":"2024-03-28T23:35:12.638024+0000","last_became_peered":"2024-03-28T23:35:12.638024+0000","last_unstale":"2024-03-28T23:35:13.659108+0000","last_undegraded":"2024-03-28T23:35:13.659108+0000","last_fullsized":"2024-03-28T23:35:13.659108+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:56:54.712369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.666873+0000","last_change":"2024-03-28T23:35:13.663102+0000","last_active":"2024-03-28T23:35:13.666873+0000","last_peered":"2024-03-28T23:35:13.666873+0000","last_clean":"2024-03-28T23:35:13.666873+0000","last_became_active":"2024-03-28T23:35:13.662837+0000","last_became_peered":"2024-03-28T23:35:13.662837+0000","last_unstale":"2024-03-28T23:35:13.666873+0000","last_undegraded":"2024-03-28T23:35:13.666873+0000","last_fullsized":"2024-03-28T23:35:13.666873+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:04:47.434294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"135'3","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.662466+0000","last_change":"2024-03-28T23:35:12.644495+0000","last_active":"2024-03-28T23:35:13.662466+0000","last_peered":"2024-03-28T23:35:13.662466+0000","last_clean":"2024-03-28T23:35:13.662466+0000","last_became_active":"2024-03-28T23:35:12.644192+0000","last_became_peered":"2024-03-28T23:35:12.644192+0000","last_unstale":"2024-03-28T23:35:13.662466+0000","last_undegraded":"2024-03-28T23:35:13.662466+0000","last_fullsized":"2024-03-28T23:35:13.662466+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:52:56.278777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.647547+0000","last_change":"2024-03-28T23:35:13.646606+0000","last_active":"2024-03-28T23:35:13.647547+0000","last_peered":"2024-03-28T23:35:13.647547+0000","last_clean":"2024-03-28T23:35:13.647547+0000","last_became_active":"2024-03-28T23:35:13.645294+0000","last_became_peered":"2024-03-28T23:35:13.645294+0000","last_unstale":"2024-03-28T23:35:13.647547+0000","last_undegraded":"2024-03-28T23:35:13.647547+0000","last_fullsized":"2024-03-28T23:35:13.647547+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:04:20.693639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"135'2","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.663622+0000","last_change":"2024-03-28T23:35:12.639522+0000","last_active":"2024-03-28T23:35:13.663622+0000","last_peered":"2024-03-28T23:35:13.663622+0000","last_clean":"2024-03-28T23:35:13.663622+0000","last_became_active":"2024-03-28T23:35:12.639228+0000","last_became_peered":"2024-03-28T23:35:12.639228+0000","last_unstale":"2024-03-28T23:35:13.663622+0000","last_undegraded":"2024-03-28T23:35:13.663622+0000","last_fullsized":"2024-03-28T23:35:13.663622+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:58:42.969626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.647483+0000","last_change":"2024-03-28T23:35:13.644995+0000","last_active":"2024-03-28T23:35:13.647483+0000","last_peered":"2024-03-28T23:35:13.647483+0000","last_clean":"2024-03-28T23:35:13.647483+0000","last_became_active":"2024-03-28T23:35:13.644704+0000","last_became_peered":"2024-03-28T23:35:13.644704+0000","last_unstale":"2024-03-28T23:35:13.647483+0000","last_undegraded":"2024-03-28T23:35:13.647483+0000","last_fullsized":"2024-03-28T23:35:13.647483+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:55:50.543042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.641410+0000","last_change":"2024-03-28T23:35:12.649404+0000","last_active":"2024-03-28T23:35:13.641410+0000","last_peered":"2024-03-28T23:35:13.641410+0000","last_clean":"2024-03-28T23:35:13.641410+0000","last_became_active":"2024-03-28T23:35:12.649124+0000","last_became_peered":"2024-03-28T23:35:12.649124+0000","last_unstale":"2024-03-28T23:35:13.641410+0000","last_undegraded":"2024-03-28T23:35:13.641410+0000","last_fullsized":"2024-03-28T23:35:13.641410+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:49.179107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.660122+0000","last_change":"2024-03-28T23:35:13.654390+0000","last_active":"2024-03-28T23:35:13.660122+0000","last_peered":"2024-03-28T23:35:13.660122+0000","last_clean":"2024-03-28T23:35:13.660122+0000","last_became_active":"2024-03-28T23:35:13.654133+0000","last_became_peered":"2024-03-28T23:35:13.654133+0000","last_unstale":"2024-03-28T23:35:13.660122+0000","last_undegraded":"2024-03-28T23:35:13.660122+0000","last_fullsized":"2024-03-28T23:35:13.660122+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:12:15.667139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.641837+0000","last_change":"2024-03-28T23:35:12.642344+0000","last_active":"2024-03-28T23:35:13.641837+0000","last_peered":"2024-03-28T23:35:13.641837+0000","last_clean":"2024-03-28T23:35:13.641837+0000","last_became_active":"2024-03-28T23:35:12.642083+0000","last_became_peered":"2024-03-28T23:35:12.642083+0000","last_unstale":"2024-03-28T23:35:13.641837+0000","last_undegraded":"2024-03-28T23:35:13.641837+0000","last_fullsized":"2024-03-28T23:35:13.641837+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:32:11.111699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.658515+0000","last_change":"2024-03-28T23:35:13.656685+0000","last_active":"2024-03-28T23:35:13.658515+0000","last_peered":"2024-03-28T23:35:13.658515+0000","last_clean":"2024-03-28T23:35:13.658515+0000","last_became_active":"2024-03-28T23:35:13.656341+0000","last_became_peered":"2024-03-28T23:35:13.656341+0000","last_unstale":"2024-03-28T23:35:13.658515+0000","last_undegraded":"2024-03-28T23:35:13.658515+0000","last_fullsized":"2024-03-28T23:35:13.658515+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:29:31.587373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"135'8","reported_seq":20,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.667837+0000","last_change":"2024-03-28T23:35:12.642224+0000","last_active":"2024-03-28T23:35:13.667837+0000","last_peered":"2024-03-28T23:35:13.667837+0000","last_clean":"2024-03-28T23:35:13.667837+0000","last_became_active":"2024-03-28T23:35:12.641904+0000","last_became_peered":"2024-03-28T23:35:12.641904+0000","last_unstale":"2024-03-28T23:35:13.667837+0000","last_undegraded":"2024-03-28T23:35:13.667837+0000","last_fullsized":"2024-03-28T23:35:13.667837+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:45:58.328881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.655866+0000","last_change":"2024-03-28T23:35:13.650951+0000","last_active":"2024-03-28T23:35:13.655866+0000","last_peered":"2024-03-28T23:35:13.655866+0000","last_clean":"2024-03-28T23:35:13.655866+0000","last_became_active":"2024-03-28T23:35:13.650675+0000","last_became_peered":"2024-03-28T23:35:13.650675+0000","last_unstale":"2024-03-28T23:35:13.655866+0000","last_undegraded":"2024-03-28T23:35:13.655866+0000","last_fullsized":"2024-03-28T23:35:13.655866+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:59:31.065907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"135'1","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.660044+0000","last_change":"2024-03-28T23:35:12.641012+0000","last_active":"2024-03-28T23:35:13.660044+0000","last_peered":"2024-03-28T23:35:13.660044+0000","last_clean":"2024-03-28T23:35:13.660044+0000","last_became_active":"2024-03-28T23:35:12.640730+0000","last_became_peered":"2024-03-28T23:35:12.640730+0000","last_unstale":"2024-03-28T23:35:13.660044+0000","last_undegraded":"2024-03-28T23:35:13.660044+0000","last_fullsized":"2024-03-28T23:35:13.660044+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:36:48.097501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.651821+0000","last_change":"2024-03-28T23:35:13.649333+0000","last_active":"2024-03-28T23:35:13.651821+0000","last_peered":"2024-03-28T23:35:13.651821+0000","last_clean":"2024-03-28T23:35:13.651821+0000","last_became_active":"2024-03-28T23:35:13.649043+0000","last_became_peered":"2024-03-28T23:35:13.649043+0000","last_unstale":"2024-03-28T23:35:13.651821+0000","last_undegraded":"2024-03-28T23:35:13.651821+0000","last_fullsized":"2024-03-28T23:35:13.651821+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:43:07.597654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"135'2","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.661758+0000","last_change":"2024-03-28T23:35:12.643321+0000","last_active":"2024-03-28T23:35:13.661758+0000","last_peered":"2024-03-28T23:35:13.661758+0000","last_clean":"2024-03-28T23:35:13.661758+0000","last_became_active":"2024-03-28T23:35:12.642620+0000","last_became_peered":"2024-03-28T23:35:12.642620+0000","last_unstale":"2024-03-28T23:35:13.661758+0000","last_undegraded":"2024-03-28T23:35:13.661758+0000","last_fullsized":"2024-03-28T23:35:13.661758+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:02.820689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.670751+0000","last_change":"2024-03-28T23:35:13.670471+0000","last_active":"2024-03-28T23:35:13.670751+0000","last_peered":"2024-03-28T23:35:13.670751+0000","last_clean":"2024-03-28T23:35:13.670751+0000","last_became_active":"2024-03-28T23:35:13.670250+0000","last_became_peered":"2024-03-28T23:35:13.670250+0000","last_unstale":"2024-03-28T23:35:13.670751+0000","last_undegraded":"2024-03-28T23:35:13.670751+0000","last_fullsized":"2024-03-28T23:35:13.670751+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:38.174144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"135'1","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.666751+0000","last_change":"2024-03-28T23:35:12.647880+0000","last_active":"2024-03-28T23:35:13.666751+0000","last_peered":"2024-03-28T23:35:13.666751+0000","last_clean":"2024-03-28T23:35:13.666751+0000","last_became_active":"2024-03-28T23:35:12.647569+0000","last_became_peered":"2024-03-28T23:35:12.647569+0000","last_unstale":"2024-03-28T23:35:13.666751+0000","last_undegraded":"2024-03-28T23:35:13.666751+0000","last_fullsized":"2024-03-28T23:35:13.666751+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:04.795020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.664878+0000","last_change":"2024-03-28T23:35:13.663606+0000","last_active":"2024-03-28T23:35:13.664878+0000","last_peered":"2024-03-28T23:35:13.664878+0000","last_clean":"2024-03-28T23:35:13.664878+0000","last_became_active":"2024-03-28T23:35:13.662932+0000","last_became_peered":"2024-03-28T23:35:13.662932+0000","last_unstale":"2024-03-28T23:35:13.664878+0000","last_undegraded":"2024-03-28T23:35:13.664878+0000","last_fullsized":"2024-03-28T23:35:13.664878+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:55:50.543042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.640312+0000","last_change":"2024-03-28T23:35:12.649384+0000","last_active":"2024-03-28T23:35:13.640312+0000","last_peered":"2024-03-28T23:35:13.640312+0000","last_clean":"2024-03-28T23:35:13.640312+0000","last_became_active":"2024-03-28T23:35:12.649091+0000","last_became_peered":"2024-03-28T23:35:12.649091+0000","last_unstale":"2024-03-28T23:35:13.640312+0000","last_undegraded":"2024-03-28T23:35:13.640312+0000","last_fullsized":"2024-03-28T23:35:13.640312+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:36.589716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.648787+0000","last_change":"2024-03-28T23:35:13.646580+0000","last_active":"2024-03-28T23:35:13.648787+0000","last_peered":"2024-03-28T23:35:13.648787+0000","last_clean":"2024-03-28T23:35:13.648787+0000","last_became_active":"2024-03-28T23:35:13.646302+0000","last_became_peered":"2024-03-28T23:35:13.646302+0000","last_unstale":"2024-03-28T23:35:13.648787+0000","last_undegraded":"2024-03-28T23:35:13.648787+0000","last_fullsized":"2024-03-28T23:35:13.648787+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:59.676192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"135'2","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.661930+0000","last_change":"2024-03-28T23:35:12.647523+0000","last_active":"2024-03-28T23:35:13.661930+0000","last_peered":"2024-03-28T23:35:13.661930+0000","last_clean":"2024-03-28T23:35:13.661930+0000","last_became_active":"2024-03-28T23:35:12.647236+0000","last_became_peered":"2024-03-28T23:35:12.647236+0000","last_unstale":"2024-03-28T23:35:13.661930+0000","last_undegraded":"2024-03-28T23:35:13.661930+0000","last_fullsized":"2024-03-28T23:35:13.661930+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:02:42.078319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.665066+0000","last_change":"2024-03-28T23:35:13.660735+0000","last_active":"2024-03-28T23:35:13.665066+0000","last_peered":"2024-03-28T23:35:13.665066+0000","last_clean":"2024-03-28T23:35:13.665066+0000","last_became_active":"2024-03-28T23:35:13.659615+0000","last_became_peered":"2024-03-28T23:35:13.659615+0000","last_unstale":"2024-03-28T23:35:13.665066+0000","last_undegraded":"2024-03-28T23:35:13.665066+0000","last_fullsized":"2024-03-28T23:35:13.665066+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:07.784578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.645511+0000","last_change":"2024-03-28T23:35:12.645582+0000","last_active":"2024-03-28T23:35:13.645511+0000","last_peered":"2024-03-28T23:35:13.645511+0000","last_clean":"2024-03-28T23:35:13.645511+0000","last_became_active":"2024-03-28T23:35:12.644240+0000","last_became_peered":"2024-03-28T23:35:12.644240+0000","last_unstale":"2024-03-28T23:35:13.645511+0000","last_undegraded":"2024-03-28T23:35:13.645511+0000","last_fullsized":"2024-03-28T23:35:13.645511+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:06:43.007155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.644159+0000","last_change":"2024-03-28T23:35:13.642628+0000","last_active":"2024-03-28T23:35:13.644159+0000","last_peered":"2024-03-28T23:35:13.644159+0000","last_clean":"2024-03-28T23:35:13.644159+0000","last_became_active":"2024-03-28T23:35:13.642073+0000","last_became_peered":"2024-03-28T23:35:13.642073+0000","last_unstale":"2024-03-28T23:35:13.644159+0000","last_undegraded":"2024-03-28T23:35:13.644159+0000","last_fullsized":"2024-03-28T23:35:13.644159+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:09:31.487622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.642168+0000","last_change":"2024-03-28T23:35:12.647839+0000","last_active":"2024-03-28T23:35:13.642168+0000","last_peered":"2024-03-28T23:35:13.642168+0000","last_clean":"2024-03-28T23:35:13.642168+0000","last_became_active":"2024-03-28T23:35:12.647555+0000","last_became_peered":"2024-03-28T23:35:12.647555+0000","last_unstale":"2024-03-28T23:35:13.642168+0000","last_undegraded":"2024-03-28T23:35:13.642168+0000","last_fullsized":"2024-03-28T23:35:13.642168+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:56.498229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.647426+0000","last_change":"2024-03-28T23:35:13.644343+0000","last_active":"2024-03-28T23:35:13.647426+0000","last_peered":"2024-03-28T23:35:13.647426+0000","last_clean":"2024-03-28T23:35:13.647426+0000","last_became_active":"2024-03-28T23:35:13.644092+0000","last_became_peered":"2024-03-28T23:35:13.644092+0000","last_unstale":"2024-03-28T23:35:13.647426+0000","last_undegraded":"2024-03-28T23:35:13.647426+0000","last_fullsized":"2024-03-28T23:35:13.647426+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:35:56.095101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.645652+0000","last_change":"2024-03-28T23:35:12.645726+0000","last_active":"2024-03-28T23:35:13.645652+0000","last_peered":"2024-03-28T23:35:13.645652+0000","last_clean":"2024-03-28T23:35:13.645652+0000","last_became_active":"2024-03-28T23:35:12.645445+0000","last_became_peered":"2024-03-28T23:35:12.645445+0000","last_unstale":"2024-03-28T23:35:13.645652+0000","last_undegraded":"2024-03-28T23:35:13.645652+0000","last_fullsized":"2024-03-28T23:35:13.645652+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:29.486828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.662285+0000","last_change":"2024-03-28T23:35:13.658564+0000","last_active":"2024-03-28T23:35:13.662285+0000","last_peered":"2024-03-28T23:35:13.662285+0000","last_clean":"2024-03-28T23:35:13.662285+0000","last_became_active":"2024-03-28T23:35:13.658307+0000","last_became_peered":"2024-03-28T23:35:13.658307+0000","last_unstale":"2024-03-28T23:35:13.662285+0000","last_undegraded":"2024-03-28T23:35:13.662285+0000","last_fullsized":"2024-03-28T23:35:13.662285+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:01:53.915171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"135'2","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.667050+0000","last_change":"2024-03-28T23:35:12.647906+0000","last_active":"2024-03-28T23:35:13.667050+0000","last_peered":"2024-03-28T23:35:13.667050+0000","last_clean":"2024-03-28T23:35:13.667050+0000","last_became_active":"2024-03-28T23:35:12.647628+0000","last_became_peered":"2024-03-28T23:35:12.647628+0000","last_unstale":"2024-03-28T23:35:13.667050+0000","last_undegraded":"2024-03-28T23:35:13.667050+0000","last_fullsized":"2024-03-28T23:35:13.667050+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:21:57.054205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.659365+0000","last_change":"2024-03-28T23:35:13.657630+0000","last_active":"2024-03-28T23:35:13.659365+0000","last_peered":"2024-03-28T23:35:13.659365+0000","last_clean":"2024-03-28T23:35:13.659365+0000","last_became_active":"2024-03-28T23:35:13.656840+0000","last_became_peered":"2024-03-28T23:35:13.656840+0000","last_unstale":"2024-03-28T23:35:13.659365+0000","last_undegraded":"2024-03-28T23:35:13.659365+0000","last_fullsized":"2024-03-28T23:35:13.659365+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:04:25.013101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.659923+0000","last_change":"2024-03-28T23:35:12.641279+0000","last_active":"2024-03-28T23:35:13.659923+0000","last_peered":"2024-03-28T23:35:13.659923+0000","last_clean":"2024-03-28T23:35:13.659923+0000","last_became_active":"2024-03-28T23:35:12.640940+0000","last_became_peered":"2024-03-28T23:35:12.640940+0000","last_unstale":"2024-03-28T23:35:13.659923+0000","last_undegraded":"2024-03-28T23:35:13.659923+0000","last_fullsized":"2024-03-28T23:35:13.659923+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:08:22.631411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.664911+0000","last_change":"2024-03-28T23:35:13.663088+0000","last_active":"2024-03-28T23:35:13.664911+0000","last_peered":"2024-03-28T23:35:13.664911+0000","last_clean":"2024-03-28T23:35:13.664911+0000","last_became_active":"2024-03-28T23:35:13.662832+0000","last_became_peered":"2024-03-28T23:35:13.662832+0000","last_unstale":"2024-03-28T23:35:13.664911+0000","last_undegraded":"2024-03-28T23:35:13.664911+0000","last_fullsized":"2024-03-28T23:35:13.664911+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:35:56.095101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.638094+0000","last_change":"2024-03-28T23:35:12.647070+0000","last_active":"2024-03-28T23:35:13.638094+0000","last_peered":"2024-03-28T23:35:13.638094+0000","last_clean":"2024-03-28T23:35:13.638094+0000","last_became_active":"2024-03-28T23:35:12.646725+0000","last_became_peered":"2024-03-28T23:35:12.646725+0000","last_unstale":"2024-03-28T23:35:13.638094+0000","last_undegraded":"2024-03-28T23:35:13.638094+0000","last_fullsized":"2024-03-28T23:35:13.638094+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:08:22.631411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.656716+0000","last_change":"2024-03-28T23:35:13.650244+0000","last_active":"2024-03-28T23:35:13.656716+0000","last_peered":"2024-03-28T23:35:13.656716+0000","last_clean":"2024-03-28T23:35:13.656716+0000","last_became_active":"2024-03-28T23:35:13.649988+0000","last_became_peered":"2024-03-28T23:35:13.649988+0000","last_unstale":"2024-03-28T23:35:13.656716+0000","last_undegraded":"2024-03-28T23:35:13.656716+0000","last_fullsized":"2024-03-28T23:35:13.656716+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:17.724044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.644457+0000","last_change":"2024-03-28T23:35:12.639799+0000","last_active":"2024-03-28T23:35:13.644457+0000","last_peered":"2024-03-28T23:35:13.644457+0000","last_clean":"2024-03-28T23:35:13.644457+0000","last_became_active":"2024-03-28T23:35:12.639284+0000","last_became_peered":"2024-03-28T23:35:12.639284+0000","last_unstale":"2024-03-28T23:35:13.644457+0000","last_undegraded":"2024-03-28T23:35:13.644457+0000","last_fullsized":"2024-03-28T23:35:13.644457+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:02.820689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.661000+0000","last_change":"2024-03-28T23:35:13.656089+0000","last_active":"2024-03-28T23:35:13.661000+0000","last_peered":"2024-03-28T23:35:13.661000+0000","last_clean":"2024-03-28T23:35:13.661000+0000","last_became_active":"2024-03-28T23:35:13.655818+0000","last_became_peered":"2024-03-28T23:35:13.655818+0000","last_unstale":"2024-03-28T23:35:13.661000+0000","last_undegraded":"2024-03-28T23:35:13.661000+0000","last_fullsized":"2024-03-28T23:35:13.661000+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:35.363250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.652401+0000","last_change":"2024-03-28T23:35:12.647657+0000","last_active":"2024-03-28T23:35:13.652401+0000","last_peered":"2024-03-28T23:35:13.652401+0000","last_clean":"2024-03-28T23:35:13.652401+0000","last_became_active":"2024-03-28T23:35:12.647393+0000","last_became_peered":"2024-03-28T23:35:12.647393+0000","last_unstale":"2024-03-28T23:35:13.652401+0000","last_undegraded":"2024-03-28T23:35:13.652401+0000","last_fullsized":"2024-03-28T23:35:13.652401+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:56:32.469078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.664435+0000","last_change":"2024-03-28T23:35:13.660791+0000","last_active":"2024-03-28T23:35:13.664435+0000","last_peered":"2024-03-28T23:35:13.664435+0000","last_clean":"2024-03-28T23:35:13.664435+0000","last_became_active":"2024-03-28T23:35:13.660461+0000","last_became_peered":"2024-03-28T23:35:13.660461+0000","last_unstale":"2024-03-28T23:35:13.664435+0000","last_undegraded":"2024-03-28T23:35:13.664435+0000","last_fullsized":"2024-03-28T23:35:13.664435+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:30:36.327178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.640915+0000","last_change":"2024-03-28T23:35:12.644816+0000","last_active":"2024-03-28T23:35:13.640915+0000","last_peered":"2024-03-28T23:35:13.640915+0000","last_clean":"2024-03-28T23:35:13.640915+0000","last_became_active":"2024-03-28T23:35:12.644561+0000","last_became_peered":"2024-03-28T23:35:12.644561+0000","last_unstale":"2024-03-28T23:35:13.640915+0000","last_undegraded":"2024-03-28T23:35:13.640915+0000","last_fullsized":"2024-03-28T23:35:13.640915+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:47:30.593198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.658281+0000","last_change":"2024-03-28T23:35:13.650966+0000","last_active":"2024-03-28T23:35:13.658281+0000","last_peered":"2024-03-28T23:35:13.658281+0000","last_clean":"2024-03-28T23:35:13.658281+0000","last_became_active":"2024-03-28T23:35:13.650706+0000","last_became_peered":"2024-03-28T23:35:13.650706+0000","last_unstale":"2024-03-28T23:35:13.658281+0000","last_undegraded":"2024-03-28T23:35:13.658281+0000","last_fullsized":"2024-03-28T23:35:13.658281+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:42:35.979766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.658369+0000","last_change":"2024-03-28T23:35:12.642111+0000","last_active":"2024-03-28T23:35:13.658369+0000","last_peered":"2024-03-28T23:35:13.658369+0000","last_clean":"2024-03-28T23:35:13.658369+0000","last_became_active":"2024-03-28T23:35:12.641157+0000","last_became_peered":"2024-03-28T23:35:12.641157+0000","last_unstale":"2024-03-28T23:35:13.658369+0000","last_undegraded":"2024-03-28T23:35:13.658369+0000","last_fullsized":"2024-03-28T23:35:13.658369+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:34.356556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.662961+0000","last_change":"2024-03-28T23:35:13.660571+0000","last_active":"2024-03-28T23:35:13.662961+0000","last_peered":"2024-03-28T23:35:13.662961+0000","last_clean":"2024-03-28T23:35:13.662961+0000","last_became_active":"2024-03-28T23:35:13.659952+0000","last_became_peered":"2024-03-28T23:35:13.659952+0000","last_unstale":"2024-03-28T23:35:13.662961+0000","last_undegraded":"2024-03-28T23:35:13.662961+0000","last_fullsized":"2024-03-28T23:35:13.662961+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:10.144427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.638589+0000","last_change":"2024-03-28T23:35:12.635657+0000","last_active":"2024-03-28T23:35:13.638589+0000","last_peered":"2024-03-28T23:35:13.638589+0000","last_clean":"2024-03-28T23:35:13.638589+0000","last_became_active":"2024-03-28T23:35:12.635355+0000","last_became_peered":"2024-03-28T23:35:12.635355+0000","last_unstale":"2024-03-28T23:35:13.638589+0000","last_undegraded":"2024-03-28T23:35:13.638589+0000","last_fullsized":"2024-03-28T23:35:13.638589+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:43:51.633196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.658099+0000","last_change":"2024-03-28T23:35:13.655220+0000","last_active":"2024-03-28T23:35:13.658099+0000","last_peered":"2024-03-28T23:35:13.658099+0000","last_clean":"2024-03-28T23:35:13.658099+0000","last_became_active":"2024-03-28T23:35:13.654937+0000","last_became_peered":"2024-03-28T23:35:13.654937+0000","last_unstale":"2024-03-28T23:35:13.658099+0000","last_undegraded":"2024-03-28T23:35:13.658099+0000","last_fullsized":"2024-03-28T23:35:13.658099+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:20:06.310533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"135'6","reported_seq":18,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.672609+0000","last_change":"2024-03-28T23:35:12.648536+0000","last_active":"2024-03-28T23:35:13.672609+0000","last_peered":"2024-03-28T23:35:13.672609+0000","last_clean":"2024-03-28T23:35:13.672609+0000","last_became_active":"2024-03-28T23:35:12.648243+0000","last_became_peered":"2024-03-28T23:35:12.648243+0000","last_unstale":"2024-03-28T23:35:13.672609+0000","last_undegraded":"2024-03-28T23:35:13.672609+0000","last_fullsized":"2024-03-28T23:35:13.672609+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:45:26.072989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":385,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.638646+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:35:13.638646+0000","last_peered":"2024-03-28T23:35:13.638646+0000","last_clean":"2024-03-28T23:35:13.638646+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:35:13.638646+0000","last_undegraded":"2024-03-28T23:35:13.638646+0000","last_fullsized":"2024-03-28T23:35:13.638646+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574917,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32904,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338936,"statfs":{"total":96636764160,"available":96603070464,"internally_reserved":0,"allocated":1765376,"data_stored":1408064,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574918,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1699840,"data_stored":817744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574918,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32844,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1695744,"data_stored":817674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574917,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32856,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338984,"statfs":{"total":96636764160,"available":96603119616,"internally_reserved":0,"allocated":1708032,"data_stored":819286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32852,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338988,"statfs":{"total":96636764160,"available":96603123712,"internally_reserved":0,"allocated":1703936,"data_stored":818290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574917,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32852,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338988,"statfs":{"total":96636764160,"available":96603123712,"internally_reserved":0,"allocated":1703936,"data_stored":817786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1699840,"data_stored":817744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574917,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32896,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338944,"statfs":{"total":96636764160,"available":96603078656,"internally_reserved":0,"allocated":1748992,"data_stored":1400875,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:35:16.957 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:35:17.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.095+0000 7f0165e7c640 1 Processor -- start 2024-03-28T23:35:17.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.095+0000 7f0165e7c640 1 -- start start 2024-03-28T23:35:17.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.096+0000 7f0165e7c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0160152890 0x7f01600c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:17.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.096+0000 7f0165e7c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f01600c5b20 0x7f01601553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:17.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.096+0000 7f0165e7c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0160158a30 con 0x7f01600c8ed0 2024-03-28T23:35:17.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.096+0000 7f0165e7c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f01600c8480 con 0x7f01600c5b20 2024-03-28T23:35:17.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.096+0000 7f0165e7c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0160159640 con 0x7f0160152890 2024-03-28T23:35:17.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.096+0000 7f015ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f01600c5b20 0x7f01601553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:17.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.096+0000 7f015effd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f01600c8ed0 0x7f01600c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45284/0 (socket says 172.21.15.161:45284) 2024-03-28T23:35:17.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.096+0000 7f015effd640 1 -- 172.21.15.161:0/556004665 learned_addr learned my addr 172.21.15.161:0/556004665 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:17.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.097+0000 7f015ffff640 1 -- 172.21.15.161:0/556004665 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0160152890 msgr2=0x7f01600c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:17.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.097+0000 7f015ffff640 1 --2- 172.21.15.161:0/556004665 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0160152890 0x7f01600c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.097+0000 7f015ffff640 1 -- 172.21.15.161:0/556004665 >> v1:172.21.15.161:6789/0 conn(0x7f01600c8ed0 legacy=0x7f01600c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:17.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.097+0000 7f015ffff640 1 -- 172.21.15.161:0/556004665 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0160158d10 con 0x7f01600c5b20 2024-03-28T23:35:17.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.097+0000 7f015e7fc640 1 -- 172.21.15.161:0/556004665 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3306842109 0 0) 0x7f0160158a30 con 0x7f01600c8ed0 2024-03-28T23:35:17.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.097+0000 7f015ffff640 1 --2- 172.21.15.161:0/556004665 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01600c5b20 0x7f01601553a0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f01601597c0 tx=0x7f015802bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=746b1a1575504a60 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:17.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.097+0000 7f015e7fc640 1 -- 172.21.15.161:0/556004665 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0158039070 con 0x7f01600c5b20 2024-03-28T23:35:17.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.097+0000 7f015e7fc640 1 -- 172.21.15.161:0/556004665 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f015802cb20 con 0x7f01600c5b20 2024-03-28T23:35:17.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.098+0000 7f015e7fc640 1 -- 172.21.15.161:0/556004665 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0158034440 con 0x7f01600c5b20 2024-03-28T23:35:17.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.098+0000 7f0165e7c640 1 -- 172.21.15.161:0/556004665 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01600c5b20 msgr2=0x7f01601553a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:17.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.098+0000 7f0165e7c640 1 --2- 172.21.15.161:0/556004665 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01600c5b20 0x7f01601553a0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f01601597c0 tx=0x7f015802bd70 comp rx=0 tx=0).stop 2024-03-28T23:35:17.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.098+0000 7f0165e7c640 1 -- 172.21.15.161:0/556004665 shutdown_connections 2024-03-28T23:35:17.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.098+0000 7f0165e7c640 1 --2- 172.21.15.161:0/556004665 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01600c5b20 0x7f01601553a0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.098+0000 7f0165e7c640 1 --2- 172.21.15.161:0/556004665 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0160152890 0x7f01600c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.098+0000 7f0165e7c640 1 -- 172.21.15.161:0/556004665 >> 172.21.15.161:0/556004665 conn(0x7f01600ac620 msgr2=0x7f01600aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:17.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.098+0000 7f0165e7c640 1 -- 172.21.15.161:0/556004665 shutdown_connections 2024-03-28T23:35:17.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.098+0000 7f0165e7c640 1 -- 172.21.15.161:0/556004665 wait complete. 2024-03-28T23:35:17.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.100+0000 7f0165e7c640 1 Processor -- start 2024-03-28T23:35:17.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.100+0000 7f0165e7c640 1 -- start start 2024-03-28T23:35:17.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.101+0000 7f0165e7c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01600c5b20 0x7f01600c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:17.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.101+0000 7f0165e7c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f01600c8ed0 0x7f0160147190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:17.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.101+0000 7f0165e7c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0160152890 0x7f016014a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:17.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.101+0000 7f0165e7c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0160152c70 con 0x7f01600c5b20 2024-03-28T23:35:17.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.101+0000 7f0165e7c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f01601592d0 con 0x7f0160152890 2024-03-28T23:35:17.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.101+0000 7f0165e7c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0160152df0 con 0x7f01600c8ed0 2024-03-28T23:35:17.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.101+0000 7f015f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01600c5b20 0x7f01600c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:17.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.101+0000 7f015f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01600c5b20 0x7f01600c1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47616/0 (socket says 172.21.15.161:47616) 2024-03-28T23:35:17.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.101+0000 7f015f7fe640 1 -- 172.21.15.161:0/1037692653 learned_addr learned my addr 172.21.15.161:0/1037692653 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:17.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.101+0000 7f015effd640 1 --2- 172.21.15.161:0/1037692653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f01600c8ed0 0x7f0160147190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:17.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.101+0000 7f015ffff640 1 --2- 172.21.15.161:0/1037692653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0160152890 0x7f016014a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:17.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.101+0000 7f015f7fe640 1 -- 172.21.15.161:0/1037692653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f01600c8ed0 msgr2=0x7f0160147190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:17.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.101+0000 7f015f7fe640 1 --2- 172.21.15.161:0/1037692653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f01600c8ed0 0x7f0160147190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.101+0000 7f015f7fe640 1 -- 172.21.15.161:0/1037692653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0160152890 msgr2=0x7f016014a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:17.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.101+0000 7f015f7fe640 1 --2- 172.21.15.161:0/1037692653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0160152890 0x7f016014a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.101+0000 7f015f7fe640 1 -- 172.21.15.161:0/1037692653 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01601568d0 con 0x7f01600c5b20 2024-03-28T23:35:17.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.101+0000 7f015f7fe640 1 --2- 172.21.15.161:0/1037692653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01600c5b20 0x7f01600c1b60 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f015000b4d0 tx=0x7f015000b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:17.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.101+0000 7f0157fff640 1 -- 172.21.15.161:0/1037692653 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0150014070 con 0x7f01600c5b20 2024-03-28T23:35:17.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.101+0000 7f0157fff640 1 -- 172.21.15.161:0/1037692653 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0150010870 con 0x7f01600c5b20 2024-03-28T23:35:17.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.101+0000 7f0165e7c640 1 -- 172.21.15.161:0/1037692653 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0160156bb0 con 0x7f01600c5b20 2024-03-28T23:35:17.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.101+0000 7f0157fff640 1 -- 172.21.15.161:0/1037692653 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0150010c60 con 0x7f01600c5b20 2024-03-28T23:35:17.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.101+0000 7f0165e7c640 1 -- 172.21.15.161:0/1037692653 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01601497b0 con 0x7f01600c5b20 2024-03-28T23:35:17.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.103+0000 7f0165e7c640 1 -- 172.21.15.161:0/1037692653 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f016004f4b0 con 0x7f01600c5b20 2024-03-28T23:35:17.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.103+0000 7f0157fff640 1 -- 172.21.15.161:0/1037692653 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0150017070 con 0x7f01600c5b20 2024-03-28T23:35:17.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.103+0000 7f0157fff640 1 --2- 172.21.15.161:0/1037692653 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f014006b4d0 0x7f014006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:17.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.103+0000 7f015effd640 1 --2- 172.21.15.161:0/1037692653 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f014006b4d0 0x7f014006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:17.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.103+0000 7f0157fff640 1 -- 172.21.15.161:0/1037692653 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 6101+0+0 (secure 0 0 0) 0x7f015008a060 con 0x7f01600c5b20 2024-03-28T23:35:17.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.104+0000 7f015effd640 1 --2- 172.21.15.161:0/1037692653 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f014006b4d0 0x7f014006d990 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f01601e8510 tx=0x7f0148009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:17.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.108+0000 7f0157fff640 1 -- 172.21.15.161:0/1037692653 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0150058790 con 0x7f01600c5b20 2024-03-28T23:35:17.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.281+0000 7f0165e7c640 1 -- 172.21.15.161:0/1037692653 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f01601476d0 con 0x7f014006b4d0 2024-03-28T23:35:17.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.287+0000 7f0157fff640 1 -- 172.21.15.161:0/1037692653 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252494 (secure 0 0 0) 0x7f014800f7a0 con 0x7f014006b4d0 2024-03-28T23:35:17.290 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:17.292 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:35:17.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.290+0000 7f0165e7c640 1 -- 172.21.15.161:0/1037692653 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f014006b4d0 msgr2=0x7f014006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:17.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.290+0000 7f0165e7c640 1 --2- 172.21.15.161:0/1037692653 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f014006b4d0 0x7f014006d990 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f01601e8510 tx=0x7f0148009040 comp rx=0 tx=0).stop 2024-03-28T23:35:17.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.290+0000 7f0165e7c640 1 -- 172.21.15.161:0/1037692653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01600c5b20 msgr2=0x7f01600c1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:17.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.290+0000 7f0165e7c640 1 --2- 172.21.15.161:0/1037692653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01600c5b20 0x7f01600c1b60 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f015000b4d0 tx=0x7f015000b990 comp rx=0 tx=0).stop 2024-03-28T23:35:17.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.290+0000 7f0165e7c640 1 -- 172.21.15.161:0/1037692653 shutdown_connections 2024-03-28T23:35:17.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.290+0000 7f0165e7c640 1 --2- 172.21.15.161:0/1037692653 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f014006b4d0 0x7f014006d990 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.290+0000 7f0165e7c640 1 --2- 172.21.15.161:0/1037692653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0160152890 0x7f016014a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.290+0000 7f0165e7c640 1 --2- 172.21.15.161:0/1037692653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f01600c8ed0 0x7f0160147190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.290+0000 7f0165e7c640 1 --2- 172.21.15.161:0/1037692653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01600c5b20 0x7f01600c1b60 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.290+0000 7f0165e7c640 1 -- 172.21.15.161:0/1037692653 >> 172.21.15.161:0/1037692653 conn(0x7f01600ac620 msgr2=0x7f0160150cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:17.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.290+0000 7f0165e7c640 1 -- 172.21.15.161:0/1037692653 shutdown_connections 2024-03-28T23:35:17.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.290+0000 7f0165e7c640 1 -- 172.21.15.161:0/1037692653 wait complete. 2024-03-28T23:35:17.302 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2024-03-28T23:35:15.374824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262900,"kb_used_data":13404,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754711820,"statfs":{"total":773094113280,"available":772824903680,"internally_reserved":0,"allocated":13725696,"data_stored":7717463,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000623"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.668039+0000","last_change":"2024-03-28T23:35:13.667376+0000","last_active":"2024-03-28T23:35:13.668039+0000","last_peered":"2024-03-28T23:35:13.668039+0000","last_clean":"2024-03-28T23:35:13.668039+0000","last_became_active":"2024-03-28T23:35:13.667026+0000","last_became_peered":"2024-03-28T23:35:13.667026+0000","last_unstale":"2024-03-28T23:35:13.668039+0000","last_undegraded":"2024-03-28T23:35:13.668039+0000","last_fullsized":"2024-03-28T23:35:13.668039+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:42:35.979766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.647675+0000","last_change":"2024-03-28T23:35:13.645679+0000","last_active":"2024-03-28T23:35:13.647675+0000","last_peered":"2024-03-28T23:35:13.647675+0000","last_clean":"2024-03-28T23:35:13.647675+0000","last_became_active":"2024-03-28T23:35:13.645424+0000","last_became_peered":"2024-03-28T23:35:13.645424+0000","last_unstale":"2024-03-28T23:35:13.647675+0000","last_undegraded":"2024-03-28T23:35:13.647675+0000","last_fullsized":"2024-03-28T23:35:13.647675+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:23:21.067376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.647705+0000","last_change":"2024-03-28T23:35:13.645175+0000","last_active":"2024-03-28T23:35:13.647705+0000","last_peered":"2024-03-28T23:35:13.647705+0000","last_clean":"2024-03-28T23:35:13.647705+0000","last_became_active":"2024-03-28T23:35:13.644892+0000","last_became_peered":"2024-03-28T23:35:13.644892+0000","last_unstale":"2024-03-28T23:35:13.647705+0000","last_undegraded":"2024-03-28T23:35:13.647705+0000","last_fullsized":"2024-03-28T23:35:13.647705+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:18:00.612518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.652351+0000","last_change":"2024-03-28T23:35:13.650727+0000","last_active":"2024-03-28T23:35:13.652351+0000","last_peered":"2024-03-28T23:35:13.652351+0000","last_clean":"2024-03-28T23:35:13.652351+0000","last_became_active":"2024-03-28T23:35:13.650440+0000","last_became_peered":"2024-03-28T23:35:13.650440+0000","last_unstale":"2024-03-28T23:35:13.652351+0000","last_undegraded":"2024-03-28T23:35:13.652351+0000","last_fullsized":"2024-03-28T23:35:13.652351+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:29.284277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.663136+0000","last_change":"2024-03-28T23:35:13.658789+0000","last_active":"2024-03-28T23:35:13.663136+0000","last_peered":"2024-03-28T23:35:13.663136+0000","last_clean":"2024-03-28T23:35:13.663136+0000","last_became_active":"2024-03-28T23:35:13.658102+0000","last_became_peered":"2024-03-28T23:35:13.658102+0000","last_unstale":"2024-03-28T23:35:13.663136+0000","last_undegraded":"2024-03-28T23:35:13.663136+0000","last_fullsized":"2024-03-28T23:35:13.663136+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:21:58.117340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.662162+0000","last_change":"2024-03-28T23:35:13.657489+0000","last_active":"2024-03-28T23:35:13.662162+0000","last_peered":"2024-03-28T23:35:13.662162+0000","last_clean":"2024-03-28T23:35:13.662162+0000","last_became_active":"2024-03-28T23:35:13.657241+0000","last_became_peered":"2024-03-28T23:35:13.657241+0000","last_unstale":"2024-03-28T23:35:13.662162+0000","last_undegraded":"2024-03-28T23:35:13.662162+0000","last_fullsized":"2024-03-28T23:35:13.662162+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:00:37.265722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.661451+0000","last_change":"2024-03-28T23:35:13.658858+0000","last_active":"2024-03-28T23:35:13.661451+0000","last_peered":"2024-03-28T23:35:13.661451+0000","last_clean":"2024-03-28T23:35:13.661451+0000","last_became_active":"2024-03-28T23:35:13.658594+0000","last_became_peered":"2024-03-28T23:35:13.658594+0000","last_unstale":"2024-03-28T23:35:13.661451+0000","last_undegraded":"2024-03-28T23:35:13.661451+0000","last_fullsized":"2024-03-28T23:35:13.661451+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:10.144427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.653436+0000","last_change":"2024-03-28T23:35:13.650219+0000","last_active":"2024-03-28T23:35:13.653436+0000","last_peered":"2024-03-28T23:35:13.653436+0000","last_clean":"2024-03-28T23:35:13.653436+0000","last_became_active":"2024-03-28T23:35:13.649864+0000","last_became_peered":"2024-03-28T23:35:13.649864+0000","last_unstale":"2024-03-28T23:35:13.653436+0000","last_undegraded":"2024-03-28T23:35:13.653436+0000","last_fullsized":"2024-03-28T23:35:13.653436+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:56:33.475772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.660216+0000","last_change":"2024-03-28T23:35:13.654353+0000","last_active":"2024-03-28T23:35:13.660216+0000","last_peered":"2024-03-28T23:35:13.660216+0000","last_clean":"2024-03-28T23:35:13.660216+0000","last_became_active":"2024-03-28T23:35:13.654090+0000","last_became_peered":"2024-03-28T23:35:13.654090+0000","last_unstale":"2024-03-28T23:35:13.660216+0000","last_undegraded":"2024-03-28T23:35:13.660216+0000","last_fullsized":"2024-03-28T23:35:13.660216+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:52:35.593391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.669840+0000","last_change":"2024-03-28T23:35:13.663286+0000","last_active":"2024-03-28T23:35:13.669840+0000","last_peered":"2024-03-28T23:35:13.669840+0000","last_clean":"2024-03-28T23:35:13.669840+0000","last_became_active":"2024-03-28T23:35:13.662622+0000","last_became_peered":"2024-03-28T23:35:13.662622+0000","last_unstale":"2024-03-28T23:35:13.669840+0000","last_undegraded":"2024-03-28T23:35:13.669840+0000","last_fullsized":"2024-03-28T23:35:13.669840+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:11:04.347579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.648301+0000","last_change":"2024-03-28T23:35:13.642520+0000","last_active":"2024-03-28T23:35:13.648301+0000","last_peered":"2024-03-28T23:35:13.648301+0000","last_clean":"2024-03-28T23:35:13.648301+0000","last_became_active":"2024-03-28T23:35:13.642244+0000","last_became_peered":"2024-03-28T23:35:13.642244+0000","last_unstale":"2024-03-28T23:35:13.648301+0000","last_undegraded":"2024-03-28T23:35:13.648301+0000","last_fullsized":"2024-03-28T23:35:13.648301+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:59:33.982931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.665326+0000","last_change":"2024-03-28T23:35:13.661843+0000","last_active":"2024-03-28T23:35:13.665326+0000","last_peered":"2024-03-28T23:35:13.665326+0000","last_clean":"2024-03-28T23:35:13.665326+0000","last_became_active":"2024-03-28T23:35:13.661276+0000","last_became_peered":"2024-03-28T23:35:13.661276+0000","last_unstale":"2024-03-28T23:35:13.665326+0000","last_undegraded":"2024-03-28T23:35:13.665326+0000","last_fullsized":"2024-03-28T23:35:13.665326+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:06:08.472963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.669867+0000","last_change":"2024-03-28T23:35:13.664375+0000","last_active":"2024-03-28T23:35:13.669867+0000","last_peered":"2024-03-28T23:35:13.669867+0000","last_clean":"2024-03-28T23:35:13.669867+0000","last_became_active":"2024-03-28T23:35:13.663245+0000","last_became_peered":"2024-03-28T23:35:13.663245+0000","last_unstale":"2024-03-28T23:35:13.669867+0000","last_undegraded":"2024-03-28T23:35:13.669867+0000","last_fullsized":"2024-03-28T23:35:13.669867+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:56:20.856416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.653459+0000","last_change":"2024-03-28T23:35:13.651904+0000","last_active":"2024-03-28T23:35:13.653459+0000","last_peered":"2024-03-28T23:35:13.653459+0000","last_clean":"2024-03-28T23:35:13.653459+0000","last_became_active":"2024-03-28T23:35:13.651636+0000","last_became_peered":"2024-03-28T23:35:13.651636+0000","last_unstale":"2024-03-28T23:35:13.653459+0000","last_undegraded":"2024-03-28T23:35:13.653459+0000","last_fullsized":"2024-03-28T23:35:13.653459+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:52:06.037363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.658856+0000","last_change":"2024-03-28T23:35:13.651843+0000","last_active":"2024-03-28T23:35:13.658856+0000","last_peered":"2024-03-28T23:35:13.658856+0000","last_clean":"2024-03-28T23:35:13.658856+0000","last_became_active":"2024-03-28T23:35:13.651568+0000","last_became_peered":"2024-03-28T23:35:13.651568+0000","last_unstale":"2024-03-28T23:35:13.658856+0000","last_undegraded":"2024-03-28T23:35:13.658856+0000","last_fullsized":"2024-03-28T23:35:13.658856+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:20:20.561594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.666842+0000","last_change":"2024-03-28T23:35:13.660451+0000","last_active":"2024-03-28T23:35:13.666842+0000","last_peered":"2024-03-28T23:35:13.666842+0000","last_clean":"2024-03-28T23:35:13.666842+0000","last_became_active":"2024-03-28T23:35:13.660166+0000","last_became_peered":"2024-03-28T23:35:13.660166+0000","last_unstale":"2024-03-28T23:35:13.666842+0000","last_undegraded":"2024-03-28T23:35:13.666842+0000","last_fullsized":"2024-03-28T23:35:13.666842+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:12:40.626571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.659006+0000","last_change":"2024-03-28T23:35:13.656742+0000","last_active":"2024-03-28T23:35:13.659006+0000","last_peered":"2024-03-28T23:35:13.659006+0000","last_clean":"2024-03-28T23:35:13.659006+0000","last_became_active":"2024-03-28T23:35:13.655664+0000","last_became_peered":"2024-03-28T23:35:13.655664+0000","last_unstale":"2024-03-28T23:35:13.659006+0000","last_undegraded":"2024-03-28T23:35:13.659006+0000","last_fullsized":"2024-03-28T23:35:13.659006+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:48:23.811667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.669881+0000","last_change":"2024-03-28T23:35:13.664804+0000","last_active":"2024-03-28T23:35:13.669881+0000","last_peered":"2024-03-28T23:35:13.669881+0000","last_clean":"2024-03-28T23:35:13.669881+0000","last_became_active":"2024-03-28T23:35:13.664532+0000","last_became_peered":"2024-03-28T23:35:13.664532+0000","last_unstale":"2024-03-28T23:35:13.669881+0000","last_undegraded":"2024-03-28T23:35:13.669881+0000","last_fullsized":"2024-03-28T23:35:13.669881+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:44:13.076565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.651799+0000","last_change":"2024-03-28T23:35:13.648461+0000","last_active":"2024-03-28T23:35:13.651799+0000","last_peered":"2024-03-28T23:35:13.651799+0000","last_clean":"2024-03-28T23:35:13.651799+0000","last_became_active":"2024-03-28T23:35:13.648183+0000","last_became_peered":"2024-03-28T23:35:13.648183+0000","last_unstale":"2024-03-28T23:35:13.651799+0000","last_undegraded":"2024-03-28T23:35:13.651799+0000","last_fullsized":"2024-03-28T23:35:13.651799+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:17.724044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.662399+0000","last_change":"2024-03-28T23:35:13.651360+0000","last_active":"2024-03-28T23:35:13.662399+0000","last_peered":"2024-03-28T23:35:13.662399+0000","last_clean":"2024-03-28T23:35:13.662399+0000","last_became_active":"2024-03-28T23:35:13.651068+0000","last_became_peered":"2024-03-28T23:35:13.651068+0000","last_unstale":"2024-03-28T23:35:13.662399+0000","last_undegraded":"2024-03-28T23:35:13.662399+0000","last_fullsized":"2024-03-28T23:35:13.662399+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:05.948124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.656699+0000","last_change":"2024-03-28T23:35:13.650915+0000","last_active":"2024-03-28T23:35:13.656699+0000","last_peered":"2024-03-28T23:35:13.656699+0000","last_clean":"2024-03-28T23:35:13.656699+0000","last_became_active":"2024-03-28T23:35:13.650635+0000","last_became_peered":"2024-03-28T23:35:13.650635+0000","last_unstale":"2024-03-28T23:35:13.656699+0000","last_undegraded":"2024-03-28T23:35:13.656699+0000","last_fullsized":"2024-03-28T23:35:13.656699+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:43:07.597654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.661340+0000","last_change":"2024-03-28T23:35:13.659128+0000","last_active":"2024-03-28T23:35:13.661340+0000","last_peered":"2024-03-28T23:35:13.661340+0000","last_clean":"2024-03-28T23:35:13.661340+0000","last_became_active":"2024-03-28T23:35:13.658777+0000","last_became_peered":"2024-03-28T23:35:13.658777+0000","last_unstale":"2024-03-28T23:35:13.661340+0000","last_undegraded":"2024-03-28T23:35:13.661340+0000","last_fullsized":"2024-03-28T23:35:13.661340+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:05.801713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.666486+0000","last_change":"2024-03-28T23:35:13.664261+0000","last_active":"2024-03-28T23:35:13.666486+0000","last_peered":"2024-03-28T23:35:13.666486+0000","last_clean":"2024-03-28T23:35:13.666486+0000","last_became_active":"2024-03-28T23:35:13.663991+0000","last_became_peered":"2024-03-28T23:35:13.663991+0000","last_unstale":"2024-03-28T23:35:13.666486+0000","last_undegraded":"2024-03-28T23:35:13.666486+0000","last_fullsized":"2024-03-28T23:35:13.666486+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:05.801713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.664471+0000","last_change":"2024-03-28T23:35:13.660718+0000","last_active":"2024-03-28T23:35:13.664471+0000","last_peered":"2024-03-28T23:35:13.664471+0000","last_clean":"2024-03-28T23:35:13.664471+0000","last_became_active":"2024-03-28T23:35:13.660433+0000","last_became_peered":"2024-03-28T23:35:13.660433+0000","last_unstale":"2024-03-28T23:35:13.664471+0000","last_undegraded":"2024-03-28T23:35:13.664471+0000","last_fullsized":"2024-03-28T23:35:13.664471+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:30:44.583627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.667969+0000","last_change":"2024-03-28T23:35:13.666123+0000","last_active":"2024-03-28T23:35:13.667969+0000","last_peered":"2024-03-28T23:35:13.667969+0000","last_clean":"2024-03-28T23:35:13.667969+0000","last_became_active":"2024-03-28T23:35:13.665852+0000","last_became_peered":"2024-03-28T23:35:13.665852+0000","last_unstale":"2024-03-28T23:35:13.667969+0000","last_undegraded":"2024-03-28T23:35:13.667969+0000","last_fullsized":"2024-03-28T23:35:13.667969+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:23:21.067376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.666743+0000","last_change":"2024-03-28T23:35:13.663983+0000","last_active":"2024-03-28T23:35:13.666743+0000","last_peered":"2024-03-28T23:35:13.666743+0000","last_clean":"2024-03-28T23:35:13.666743+0000","last_became_active":"2024-03-28T23:35:13.663503+0000","last_became_peered":"2024-03-28T23:35:13.663503+0000","last_unstale":"2024-03-28T23:35:13.666743+0000","last_undegraded":"2024-03-28T23:35:13.666743+0000","last_fullsized":"2024-03-28T23:35:13.666743+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:18:00.612518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.660166+0000","last_change":"2024-03-28T23:35:13.653802+0000","last_active":"2024-03-28T23:35:13.660166+0000","last_peered":"2024-03-28T23:35:13.660166+0000","last_clean":"2024-03-28T23:35:13.660166+0000","last_became_active":"2024-03-28T23:35:13.653491+0000","last_became_peered":"2024-03-28T23:35:13.653491+0000","last_unstale":"2024-03-28T23:35:13.660166+0000","last_undegraded":"2024-03-28T23:35:13.660166+0000","last_fullsized":"2024-03-28T23:35:13.660166+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:59:31.065907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.656594+0000","last_change":"2024-03-28T23:35:13.655570+0000","last_active":"2024-03-28T23:35:13.656594+0000","last_peered":"2024-03-28T23:35:13.656594+0000","last_clean":"2024-03-28T23:35:13.656594+0000","last_became_active":"2024-03-28T23:35:13.655310+0000","last_became_peered":"2024-03-28T23:35:13.655310+0000","last_unstale":"2024-03-28T23:35:13.656594+0000","last_undegraded":"2024-03-28T23:35:13.656594+0000","last_fullsized":"2024-03-28T23:35:13.656594+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:56.745797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.663117+0000","last_change":"2024-03-28T23:35:13.660800+0000","last_active":"2024-03-28T23:35:13.663117+0000","last_peered":"2024-03-28T23:35:13.663117+0000","last_clean":"2024-03-28T23:35:13.663117+0000","last_became_active":"2024-03-28T23:35:13.660312+0000","last_became_peered":"2024-03-28T23:35:13.660312+0000","last_unstale":"2024-03-28T23:35:13.663117+0000","last_undegraded":"2024-03-28T23:35:13.663117+0000","last_fullsized":"2024-03-28T23:35:13.663117+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:56:55.719063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.653094+0000","last_change":"2024-03-28T23:35:13.648911+0000","last_active":"2024-03-28T23:35:13.653094+0000","last_peered":"2024-03-28T23:35:13.653094+0000","last_clean":"2024-03-28T23:35:13.653094+0000","last_became_active":"2024-03-28T23:35:13.648655+0000","last_became_peered":"2024-03-28T23:35:13.648655+0000","last_unstale":"2024-03-28T23:35:13.653094+0000","last_undegraded":"2024-03-28T23:35:13.653094+0000","last_fullsized":"2024-03-28T23:35:13.653094+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:35:48.165039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.662417+0000","last_change":"2024-03-28T23:35:13.658385+0000","last_active":"2024-03-28T23:35:13.662417+0000","last_peered":"2024-03-28T23:35:13.662417+0000","last_clean":"2024-03-28T23:35:13.662417+0000","last_became_active":"2024-03-28T23:35:13.658094+0000","last_became_peered":"2024-03-28T23:35:13.658094+0000","last_unstale":"2024-03-28T23:35:13.662417+0000","last_undegraded":"2024-03-28T23:35:13.662417+0000","last_fullsized":"2024-03-28T23:35:13.662417+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:26:10.661199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.653347+0000","last_change":"2024-03-28T23:35:13.649502+0000","last_active":"2024-03-28T23:35:13.653347+0000","last_peered":"2024-03-28T23:35:13.653347+0000","last_clean":"2024-03-28T23:35:13.653347+0000","last_became_active":"2024-03-28T23:35:13.649255+0000","last_became_peered":"2024-03-28T23:35:13.649255+0000","last_unstale":"2024-03-28T23:35:13.653347+0000","last_undegraded":"2024-03-28T23:35:13.653347+0000","last_fullsized":"2024-03-28T23:35:13.653347+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:49:07.618234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.647576+0000","last_change":"2024-03-28T23:35:13.643224+0000","last_active":"2024-03-28T23:35:13.647576+0000","last_peered":"2024-03-28T23:35:13.647576+0000","last_clean":"2024-03-28T23:35:13.647576+0000","last_became_active":"2024-03-28T23:35:13.642959+0000","last_became_peered":"2024-03-28T23:35:13.642959+0000","last_unstale":"2024-03-28T23:35:13.647576+0000","last_undegraded":"2024-03-28T23:35:13.647576+0000","last_fullsized":"2024-03-28T23:35:13.647576+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:08.266698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"135'4","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.663708+0000","last_change":"2024-03-28T23:35:12.639567+0000","last_active":"2024-03-28T23:35:13.663708+0000","last_peered":"2024-03-28T23:35:13.663708+0000","last_clean":"2024-03-28T23:35:13.663708+0000","last_became_active":"2024-03-28T23:35:12.639270+0000","last_became_peered":"2024-03-28T23:35:12.639270+0000","last_unstale":"2024-03-28T23:35:13.663708+0000","last_undegraded":"2024-03-28T23:35:13.663708+0000","last_fullsized":"2024-03-28T23:35:13.663708+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:09:30.480929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.658488+0000","last_change":"2024-03-28T23:35:13.651426+0000","last_active":"2024-03-28T23:35:13.658488+0000","last_peered":"2024-03-28T23:35:13.658488+0000","last_clean":"2024-03-28T23:35:13.658488+0000","last_became_active":"2024-03-28T23:35:13.651155+0000","last_became_peered":"2024-03-28T23:35:13.651155+0000","last_unstale":"2024-03-28T23:35:13.658488+0000","last_undegraded":"2024-03-28T23:35:13.658488+0000","last_fullsized":"2024-03-28T23:35:13.658488+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:38.174144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"135'1","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.665288+0000","last_change":"2024-03-28T23:35:12.642892+0000","last_active":"2024-03-28T23:35:13.665288+0000","last_peered":"2024-03-28T23:35:13.665288+0000","last_clean":"2024-03-28T23:35:13.665288+0000","last_became_active":"2024-03-28T23:35:12.642271+0000","last_became_peered":"2024-03-28T23:35:12.642271+0000","last_unstale":"2024-03-28T23:35:13.665288+0000","last_undegraded":"2024-03-28T23:35:13.665288+0000","last_fullsized":"2024-03-28T23:35:13.665288+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:06.777884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.649003+0000","last_change":"2024-03-28T23:35:13.648277+0000","last_active":"2024-03-28T23:35:13.649003+0000","last_peered":"2024-03-28T23:35:13.649003+0000","last_clean":"2024-03-28T23:35:13.649003+0000","last_became_active":"2024-03-28T23:35:13.647949+0000","last_became_peered":"2024-03-28T23:35:13.647949+0000","last_unstale":"2024-03-28T23:35:13.649003+0000","last_undegraded":"2024-03-28T23:35:13.649003+0000","last_fullsized":"2024-03-28T23:35:13.649003+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:44:30.945933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"135'2","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.664918+0000","last_change":"2024-03-28T23:35:12.640335+0000","last_active":"2024-03-28T23:35:13.664918+0000","last_peered":"2024-03-28T23:35:13.664918+0000","last_clean":"2024-03-28T23:35:13.664918+0000","last_became_active":"2024-03-28T23:35:12.639942+0000","last_became_peered":"2024-03-28T23:35:12.639942+0000","last_unstale":"2024-03-28T23:35:13.664918+0000","last_undegraded":"2024-03-28T23:35:13.664918+0000","last_fullsized":"2024-03-28T23:35:13.664918+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:29:30.580679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.666598+0000","last_change":"2024-03-28T23:35:13.664926+0000","last_active":"2024-03-28T23:35:13.666598+0000","last_peered":"2024-03-28T23:35:13.666598+0000","last_clean":"2024-03-28T23:35:13.666598+0000","last_became_active":"2024-03-28T23:35:13.663695+0000","last_became_peered":"2024-03-28T23:35:13.663695+0000","last_unstale":"2024-03-28T23:35:13.666598+0000","last_undegraded":"2024-03-28T23:35:13.666598+0000","last_fullsized":"2024-03-28T23:35:13.666598+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:52:35.593391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"135'2","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.663945+0000","last_change":"2024-03-28T23:35:12.644928+0000","last_active":"2024-03-28T23:35:13.663945+0000","last_peered":"2024-03-28T23:35:13.663945+0000","last_clean":"2024-03-28T23:35:13.663945+0000","last_became_active":"2024-03-28T23:35:12.644436+0000","last_became_peered":"2024-03-28T23:35:12.644436+0000","last_unstale":"2024-03-28T23:35:13.663945+0000","last_undegraded":"2024-03-28T23:35:13.663945+0000","last_fullsized":"2024-03-28T23:35:13.663945+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:56:54.712369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.660459+0000","last_change":"2024-03-28T23:35:13.649833+0000","last_active":"2024-03-28T23:35:13.660459+0000","last_peered":"2024-03-28T23:35:13.660459+0000","last_clean":"2024-03-28T23:35:13.660459+0000","last_became_active":"2024-03-28T23:35:13.649547+0000","last_became_peered":"2024-03-28T23:35:13.649547+0000","last_unstale":"2024-03-28T23:35:13.660459+0000","last_undegraded":"2024-03-28T23:35:13.660459+0000","last_fullsized":"2024-03-28T23:35:13.660459+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:45:31.073396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"135'1","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.667150+0000","last_change":"2024-03-28T23:35:12.648684+0000","last_active":"2024-03-28T23:35:13.667150+0000","last_peered":"2024-03-28T23:35:13.667150+0000","last_clean":"2024-03-28T23:35:13.667150+0000","last_became_active":"2024-03-28T23:35:12.648422+0000","last_became_peered":"2024-03-28T23:35:12.648422+0000","last_unstale":"2024-03-28T23:35:13.667150+0000","last_undegraded":"2024-03-28T23:35:13.667150+0000","last_fullsized":"2024-03-28T23:35:13.667150+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:36:17.817296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.654283+0000","last_change":"2024-03-28T23:35:13.653558+0000","last_active":"2024-03-28T23:35:13.654283+0000","last_peered":"2024-03-28T23:35:13.654283+0000","last_clean":"2024-03-28T23:35:13.654283+0000","last_became_active":"2024-03-28T23:35:13.653292+0000","last_became_peered":"2024-03-28T23:35:13.653292+0000","last_unstale":"2024-03-28T23:35:13.654283+0000","last_undegraded":"2024-03-28T23:35:13.654283+0000","last_fullsized":"2024-03-28T23:35:13.654283+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:28:59.847157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"135'1","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.663988+0000","last_change":"2024-03-28T23:35:12.643414+0000","last_active":"2024-03-28T23:35:13.663988+0000","last_peered":"2024-03-28T23:35:13.663988+0000","last_clean":"2024-03-28T23:35:13.663988+0000","last_became_active":"2024-03-28T23:35:12.643155+0000","last_became_peered":"2024-03-28T23:35:12.643155+0000","last_unstale":"2024-03-28T23:35:13.663988+0000","last_undegraded":"2024-03-28T23:35:13.663988+0000","last_fullsized":"2024-03-28T23:35:13.663988+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:06.777884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.658457+0000","last_change":"2024-03-28T23:35:13.655141+0000","last_active":"2024-03-28T23:35:13.658457+0000","last_peered":"2024-03-28T23:35:13.658457+0000","last_clean":"2024-03-28T23:35:13.658457+0000","last_became_active":"2024-03-28T23:35:13.651870+0000","last_became_peered":"2024-03-28T23:35:13.651870+0000","last_unstale":"2024-03-28T23:35:13.658457+0000","last_undegraded":"2024-03-28T23:35:13.658457+0000","last_fullsized":"2024-03-28T23:35:13.658457+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:43:52.639889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.658232+0000","last_change":"2024-03-28T23:35:12.646919+0000","last_active":"2024-03-28T23:35:13.658232+0000","last_peered":"2024-03-28T23:35:13.658232+0000","last_clean":"2024-03-28T23:35:13.658232+0000","last_became_active":"2024-03-28T23:35:12.646465+0000","last_became_peered":"2024-03-28T23:35:12.646465+0000","last_unstale":"2024-03-28T23:35:13.658232+0000","last_undegraded":"2024-03-28T23:35:13.658232+0000","last_fullsized":"2024-03-28T23:35:13.658232+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:20:05.303840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.661875+0000","last_change":"2024-03-28T23:35:13.660220+0000","last_active":"2024-03-28T23:35:13.661875+0000","last_peered":"2024-03-28T23:35:13.661875+0000","last_clean":"2024-03-28T23:35:13.661875+0000","last_became_active":"2024-03-28T23:35:13.659947+0000","last_became_peered":"2024-03-28T23:35:13.659947+0000","last_unstale":"2024-03-28T23:35:13.661875+0000","last_undegraded":"2024-03-28T23:35:13.661875+0000","last_fullsized":"2024-03-28T23:35:13.661875+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:08.266698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.648958+0000","last_change":"2024-03-28T23:35:12.642121+0000","last_active":"2024-03-28T23:35:13.648958+0000","last_peered":"2024-03-28T23:35:13.648958+0000","last_clean":"2024-03-28T23:35:13.648958+0000","last_became_active":"2024-03-28T23:35:12.641797+0000","last_became_peered":"2024-03-28T23:35:12.641797+0000","last_unstale":"2024-03-28T23:35:13.648958+0000","last_undegraded":"2024-03-28T23:35:13.648958+0000","last_fullsized":"2024-03-28T23:35:13.648958+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:58.669498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.653469+0000","last_change":"2024-03-28T23:35:13.651744+0000","last_active":"2024-03-28T23:35:13.653469+0000","last_peered":"2024-03-28T23:35:13.653469+0000","last_clean":"2024-03-28T23:35:13.653469+0000","last_became_active":"2024-03-28T23:35:13.651412+0000","last_became_peered":"2024-03-28T23:35:13.651412+0000","last_unstale":"2024-03-28T23:35:13.653469+0000","last_undegraded":"2024-03-28T23:35:13.653469+0000","last_fullsized":"2024-03-28T23:35:13.653469+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:45:59.335574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.639452+0000","last_change":"2024-03-28T23:35:12.640263+0000","last_active":"2024-03-28T23:35:13.639452+0000","last_peered":"2024-03-28T23:35:13.639452+0000","last_clean":"2024-03-28T23:35:13.639452+0000","last_became_active":"2024-03-28T23:35:12.639948+0000","last_became_peered":"2024-03-28T23:35:12.639948+0000","last_unstale":"2024-03-28T23:35:13.639452+0000","last_undegraded":"2024-03-28T23:35:13.639452+0000","last_fullsized":"2024-03-28T23:35:13.639452+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:58:42.969626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.648325+0000","last_change":"2024-03-28T23:35:13.644780+0000","last_active":"2024-03-28T23:35:13.648325+0000","last_peered":"2024-03-28T23:35:13.648325+0000","last_clean":"2024-03-28T23:35:13.648325+0000","last_became_active":"2024-03-28T23:35:13.644502+0000","last_became_peered":"2024-03-28T23:35:13.644502+0000","last_unstale":"2024-03-28T23:35:13.648325+0000","last_undegraded":"2024-03-28T23:35:13.648325+0000","last_fullsized":"2024-03-28T23:35:13.648325+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:13:25.031142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.659108+0000","last_change":"2024-03-28T23:35:12.638492+0000","last_active":"2024-03-28T23:35:13.659108+0000","last_peered":"2024-03-28T23:35:13.659108+0000","last_clean":"2024-03-28T23:35:13.659108+0000","last_became_active":"2024-03-28T23:35:12.638024+0000","last_became_peered":"2024-03-28T23:35:12.638024+0000","last_unstale":"2024-03-28T23:35:13.659108+0000","last_undegraded":"2024-03-28T23:35:13.659108+0000","last_fullsized":"2024-03-28T23:35:13.659108+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:56:54.712369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.666873+0000","last_change":"2024-03-28T23:35:13.663102+0000","last_active":"2024-03-28T23:35:13.666873+0000","last_peered":"2024-03-28T23:35:13.666873+0000","last_clean":"2024-03-28T23:35:13.666873+0000","last_became_active":"2024-03-28T23:35:13.662837+0000","last_became_peered":"2024-03-28T23:35:13.662837+0000","last_unstale":"2024-03-28T23:35:13.666873+0000","last_undegraded":"2024-03-28T23:35:13.666873+0000","last_fullsized":"2024-03-28T23:35:13.666873+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:04:47.434294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"135'3","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.662466+0000","last_change":"2024-03-28T23:35:12.644495+0000","last_active":"2024-03-28T23:35:13.662466+0000","last_peered":"2024-03-28T23:35:13.662466+0000","last_clean":"2024-03-28T23:35:13.662466+0000","last_became_active":"2024-03-28T23:35:12.644192+0000","last_became_peered":"2024-03-28T23:35:12.644192+0000","last_unstale":"2024-03-28T23:35:13.662466+0000","last_undegraded":"2024-03-28T23:35:13.662466+0000","last_fullsized":"2024-03-28T23:35:13.662466+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:52:56.278777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.647547+0000","last_change":"2024-03-28T23:35:13.646606+0000","last_active":"2024-03-28T23:35:13.647547+0000","last_peered":"2024-03-28T23:35:13.647547+0000","last_clean":"2024-03-28T23:35:13.647547+0000","last_became_active":"2024-03-28T23:35:13.645294+0000","last_became_peered":"2024-03-28T23:35:13.645294+0000","last_unstale":"2024-03-28T23:35:13.647547+0000","last_undegraded":"2024-03-28T23:35:13.647547+0000","last_fullsized":"2024-03-28T23:35:13.647547+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:04:20.693639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"135'2","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.663622+0000","last_change":"2024-03-28T23:35:12.639522+0000","last_active":"2024-03-28T23:35:13.663622+0000","last_peered":"2024-03-28T23:35:13.663622+0000","last_clean":"2024-03-28T23:35:13.663622+0000","last_became_active":"2024-03-28T23:35:12.639228+0000","last_became_peered":"2024-03-28T23:35:12.639228+0000","last_unstale":"2024-03-28T23:35:13.663622+0000","last_undegraded":"2024-03-28T23:35:13.663622+0000","last_fullsized":"2024-03-28T23:35:13.663622+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:58:42.969626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.647483+0000","last_change":"2024-03-28T23:35:13.644995+0000","last_active":"2024-03-28T23:35:13.647483+0000","last_peered":"2024-03-28T23:35:13.647483+0000","last_clean":"2024-03-28T23:35:13.647483+0000","last_became_active":"2024-03-28T23:35:13.644704+0000","last_became_peered":"2024-03-28T23:35:13.644704+0000","last_unstale":"2024-03-28T23:35:13.647483+0000","last_undegraded":"2024-03-28T23:35:13.647483+0000","last_fullsized":"2024-03-28T23:35:13.647483+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:55:50.543042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.641410+0000","last_change":"2024-03-28T23:35:12.649404+0000","last_active":"2024-03-28T23:35:13.641410+0000","last_peered":"2024-03-28T23:35:13.641410+0000","last_clean":"2024-03-28T23:35:13.641410+0000","last_became_active":"2024-03-28T23:35:12.649124+0000","last_became_peered":"2024-03-28T23:35:12.649124+0000","last_unstale":"2024-03-28T23:35:13.641410+0000","last_undegraded":"2024-03-28T23:35:13.641410+0000","last_fullsized":"2024-03-28T23:35:13.641410+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:49.179107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.660122+0000","last_change":"2024-03-28T23:35:13.654390+0000","last_active":"2024-03-28T23:35:13.660122+0000","last_peered":"2024-03-28T23:35:13.660122+0000","last_clean":"2024-03-28T23:35:13.660122+0000","last_became_active":"2024-03-28T23:35:13.654133+0000","last_became_peered":"2024-03-28T23:35:13.654133+0000","last_unstale":"2024-03-28T23:35:13.660122+0000","last_undegraded":"2024-03-28T23:35:13.660122+0000","last_fullsized":"2024-03-28T23:35:13.660122+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:12:15.667139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.641837+0000","last_change":"2024-03-28T23:35:12.642344+0000","last_active":"2024-03-28T23:35:13.641837+0000","last_peered":"2024-03-28T23:35:13.641837+0000","last_clean":"2024-03-28T23:35:13.641837+0000","last_became_active":"2024-03-28T23:35:12.642083+0000","last_became_peered":"2024-03-28T23:35:12.642083+0000","last_unstale":"2024-03-28T23:35:13.641837+0000","last_undegraded":"2024-03-28T23:35:13.641837+0000","last_fullsized":"2024-03-28T23:35:13.641837+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:32:11.111699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.658515+0000","last_change":"2024-03-28T23:35:13.656685+0000","last_active":"2024-03-28T23:35:13.658515+0000","last_peered":"2024-03-28T23:35:13.658515+0000","last_clean":"2024-03-28T23:35:13.658515+0000","last_became_active":"2024-03-28T23:35:13.656341+0000","last_became_peered":"2024-03-28T23:35:13.656341+0000","last_unstale":"2024-03-28T23:35:13.658515+0000","last_undegraded":"2024-03-28T23:35:13.658515+0000","last_fullsized":"2024-03-28T23:35:13.658515+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:29:31.587373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"135'8","reported_seq":20,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.667837+0000","last_change":"2024-03-28T23:35:12.642224+0000","last_active":"2024-03-28T23:35:13.667837+0000","last_peered":"2024-03-28T23:35:13.667837+0000","last_clean":"2024-03-28T23:35:13.667837+0000","last_became_active":"2024-03-28T23:35:12.641904+0000","last_became_peered":"2024-03-28T23:35:12.641904+0000","last_unstale":"2024-03-28T23:35:13.667837+0000","last_undegraded":"2024-03-28T23:35:13.667837+0000","last_fullsized":"2024-03-28T23:35:13.667837+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:45:58.328881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.655866+0000","last_change":"2024-03-28T23:35:13.650951+0000","last_active":"2024-03-28T23:35:13.655866+0000","last_peered":"2024-03-28T23:35:13.655866+0000","last_clean":"2024-03-28T23:35:13.655866+0000","last_became_active":"2024-03-28T23:35:13.650675+0000","last_became_peered":"2024-03-28T23:35:13.650675+0000","last_unstale":"2024-03-28T23:35:13.655866+0000","last_undegraded":"2024-03-28T23:35:13.655866+0000","last_fullsized":"2024-03-28T23:35:13.655866+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:59:31.065907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"135'1","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.660044+0000","last_change":"2024-03-28T23:35:12.641012+0000","last_active":"2024-03-28T23:35:13.660044+0000","last_peered":"2024-03-28T23:35:13.660044+0000","last_clean":"2024-03-28T23:35:13.660044+0000","last_became_active":"2024-03-28T23:35:12.640730+0000","last_became_peered":"2024-03-28T23:35:12.640730+0000","last_unstale":"2024-03-28T23:35:13.660044+0000","last_undegraded":"2024-03-28T23:35:13.660044+0000","last_fullsized":"2024-03-28T23:35:13.660044+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:36:48.097501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.651821+0000","last_change":"2024-03-28T23:35:13.649333+0000","last_active":"2024-03-28T23:35:13.651821+0000","last_peered":"2024-03-28T23:35:13.651821+0000","last_clean":"2024-03-28T23:35:13.651821+0000","last_became_active":"2024-03-28T23:35:13.649043+0000","last_became_peered":"2024-03-28T23:35:13.649043+0000","last_unstale":"2024-03-28T23:35:13.651821+0000","last_undegraded":"2024-03-28T23:35:13.651821+0000","last_fullsized":"2024-03-28T23:35:13.651821+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:43:07.597654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"135'2","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.661758+0000","last_change":"2024-03-28T23:35:12.643321+0000","last_active":"2024-03-28T23:35:13.661758+0000","last_peered":"2024-03-28T23:35:13.661758+0000","last_clean":"2024-03-28T23:35:13.661758+0000","last_became_active":"2024-03-28T23:35:12.642620+0000","last_became_peered":"2024-03-28T23:35:12.642620+0000","last_unstale":"2024-03-28T23:35:13.661758+0000","last_undegraded":"2024-03-28T23:35:13.661758+0000","last_fullsized":"2024-03-28T23:35:13.661758+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:02.820689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.670751+0000","last_change":"2024-03-28T23:35:13.670471+0000","last_active":"2024-03-28T23:35:13.670751+0000","last_peered":"2024-03-28T23:35:13.670751+0000","last_clean":"2024-03-28T23:35:13.670751+0000","last_became_active":"2024-03-28T23:35:13.670250+0000","last_became_peered":"2024-03-28T23:35:13.670250+0000","last_unstale":"2024-03-28T23:35:13.670751+0000","last_undegraded":"2024-03-28T23:35:13.670751+0000","last_fullsized":"2024-03-28T23:35:13.670751+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:38.174144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"135'1","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.666751+0000","last_change":"2024-03-28T23:35:12.647880+0000","last_active":"2024-03-28T23:35:13.666751+0000","last_peered":"2024-03-28T23:35:13.666751+0000","last_clean":"2024-03-28T23:35:13.666751+0000","last_became_active":"2024-03-28T23:35:12.647569+0000","last_became_peered":"2024-03-28T23:35:12.647569+0000","last_unstale":"2024-03-28T23:35:13.666751+0000","last_undegraded":"2024-03-28T23:35:13.666751+0000","last_fullsized":"2024-03-28T23:35:13.666751+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:04.795020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.664878+0000","last_change":"2024-03-28T23:35:13.663606+0000","last_active":"2024-03-28T23:35:13.664878+0000","last_peered":"2024-03-28T23:35:13.664878+0000","last_clean":"2024-03-28T23:35:13.664878+0000","last_became_active":"2024-03-28T23:35:13.662932+0000","last_became_peered":"2024-03-28T23:35:13.662932+0000","last_unstale":"2024-03-28T23:35:13.664878+0000","last_undegraded":"2024-03-28T23:35:13.664878+0000","last_fullsized":"2024-03-28T23:35:13.664878+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:55:50.543042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.640312+0000","last_change":"2024-03-28T23:35:12.649384+0000","last_active":"2024-03-28T23:35:13.640312+0000","last_peered":"2024-03-28T23:35:13.640312+0000","last_clean":"2024-03-28T23:35:13.640312+0000","last_became_active":"2024-03-28T23:35:12.649091+0000","last_became_peered":"2024-03-28T23:35:12.649091+0000","last_unstale":"2024-03-28T23:35:13.640312+0000","last_undegraded":"2024-03-28T23:35:13.640312+0000","last_fullsized":"2024-03-28T23:35:13.640312+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:36.589716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.648787+0000","last_change":"2024-03-28T23:35:13.646580+0000","last_active":"2024-03-28T23:35:13.648787+0000","last_peered":"2024-03-28T23:35:13.648787+0000","last_clean":"2024-03-28T23:35:13.648787+0000","last_became_active":"2024-03-28T23:35:13.646302+0000","last_became_peered":"2024-03-28T23:35:13.646302+0000","last_unstale":"2024-03-28T23:35:13.648787+0000","last_undegraded":"2024-03-28T23:35:13.648787+0000","last_fullsized":"2024-03-28T23:35:13.648787+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:59.676192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"135'2","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.661930+0000","last_change":"2024-03-28T23:35:12.647523+0000","last_active":"2024-03-28T23:35:13.661930+0000","last_peered":"2024-03-28T23:35:13.661930+0000","last_clean":"2024-03-28T23:35:13.661930+0000","last_became_active":"2024-03-28T23:35:12.647236+0000","last_became_peered":"2024-03-28T23:35:12.647236+0000","last_unstale":"2024-03-28T23:35:13.661930+0000","last_undegraded":"2024-03-28T23:35:13.661930+0000","last_fullsized":"2024-03-28T23:35:13.661930+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:02:42.078319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.665066+0000","last_change":"2024-03-28T23:35:13.660735+0000","last_active":"2024-03-28T23:35:13.665066+0000","last_peered":"2024-03-28T23:35:13.665066+0000","last_clean":"2024-03-28T23:35:13.665066+0000","last_became_active":"2024-03-28T23:35:13.659615+0000","last_became_peered":"2024-03-28T23:35:13.659615+0000","last_unstale":"2024-03-28T23:35:13.665066+0000","last_undegraded":"2024-03-28T23:35:13.665066+0000","last_fullsized":"2024-03-28T23:35:13.665066+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:07.784578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.645511+0000","last_change":"2024-03-28T23:35:12.645582+0000","last_active":"2024-03-28T23:35:13.645511+0000","last_peered":"2024-03-28T23:35:13.645511+0000","last_clean":"2024-03-28T23:35:13.645511+0000","last_became_active":"2024-03-28T23:35:12.644240+0000","last_became_peered":"2024-03-28T23:35:12.644240+0000","last_unstale":"2024-03-28T23:35:13.645511+0000","last_undegraded":"2024-03-28T23:35:13.645511+0000","last_fullsized":"2024-03-28T23:35:13.645511+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:06:43.007155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.644159+0000","last_change":"2024-03-28T23:35:13.642628+0000","last_active":"2024-03-28T23:35:13.644159+0000","last_peered":"2024-03-28T23:35:13.644159+0000","last_clean":"2024-03-28T23:35:13.644159+0000","last_became_active":"2024-03-28T23:35:13.642073+0000","last_became_peered":"2024-03-28T23:35:13.642073+0000","last_unstale":"2024-03-28T23:35:13.644159+0000","last_undegraded":"2024-03-28T23:35:13.644159+0000","last_fullsized":"2024-03-28T23:35:13.644159+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:09:31.487622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.642168+0000","last_change":"2024-03-28T23:35:12.647839+0000","last_active":"2024-03-28T23:35:13.642168+0000","last_peered":"2024-03-28T23:35:13.642168+0000","last_clean":"2024-03-28T23:35:13.642168+0000","last_became_active":"2024-03-28T23:35:12.647555+0000","last_became_peered":"2024-03-28T23:35:12.647555+0000","last_unstale":"2024-03-28T23:35:13.642168+0000","last_undegraded":"2024-03-28T23:35:13.642168+0000","last_fullsized":"2024-03-28T23:35:13.642168+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:56.498229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.647426+0000","last_change":"2024-03-28T23:35:13.644343+0000","last_active":"2024-03-28T23:35:13.647426+0000","last_peered":"2024-03-28T23:35:13.647426+0000","last_clean":"2024-03-28T23:35:13.647426+0000","last_became_active":"2024-03-28T23:35:13.644092+0000","last_became_peered":"2024-03-28T23:35:13.644092+0000","last_unstale":"2024-03-28T23:35:13.647426+0000","last_undegraded":"2024-03-28T23:35:13.647426+0000","last_fullsized":"2024-03-28T23:35:13.647426+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:35:56.095101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.645652+0000","last_change":"2024-03-28T23:35:12.645726+0000","last_active":"2024-03-28T23:35:13.645652+0000","last_peered":"2024-03-28T23:35:13.645652+0000","last_clean":"2024-03-28T23:35:13.645652+0000","last_became_active":"2024-03-28T23:35:12.645445+0000","last_became_peered":"2024-03-28T23:35:12.645445+0000","last_unstale":"2024-03-28T23:35:13.645652+0000","last_undegraded":"2024-03-28T23:35:13.645652+0000","last_fullsized":"2024-03-28T23:35:13.645652+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:29.486828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.662285+0000","last_change":"2024-03-28T23:35:13.658564+0000","last_active":"2024-03-28T23:35:13.662285+0000","last_peered":"2024-03-28T23:35:13.662285+0000","last_clean":"2024-03-28T23:35:13.662285+0000","last_became_active":"2024-03-28T23:35:13.658307+0000","last_became_peered":"2024-03-28T23:35:13.658307+0000","last_unstale":"2024-03-28T23:35:13.662285+0000","last_undegraded":"2024-03-28T23:35:13.662285+0000","last_fullsized":"2024-03-28T23:35:13.662285+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:01:53.915171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"135'2","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.667050+0000","last_change":"2024-03-28T23:35:12.647906+0000","last_active":"2024-03-28T23:35:13.667050+0000","last_peered":"2024-03-28T23:35:13.667050+0000","last_clean":"2024-03-28T23:35:13.667050+0000","last_became_active":"2024-03-28T23:35:12.647628+0000","last_became_peered":"2024-03-28T23:35:12.647628+0000","last_unstale":"2024-03-28T23:35:13.667050+0000","last_undegraded":"2024-03-28T23:35:13.667050+0000","last_fullsized":"2024-03-28T23:35:13.667050+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:21:57.054205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.659365+0000","last_change":"2024-03-28T23:35:13.657630+0000","last_active":"2024-03-28T23:35:13.659365+0000","last_peered":"2024-03-28T23:35:13.659365+0000","last_clean":"2024-03-28T23:35:13.659365+0000","last_became_active":"2024-03-28T23:35:13.656840+0000","last_became_peered":"2024-03-28T23:35:13.656840+0000","last_unstale":"2024-03-28T23:35:13.659365+0000","last_undegraded":"2024-03-28T23:35:13.659365+0000","last_fullsized":"2024-03-28T23:35:13.659365+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:04:25.013101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.659923+0000","last_change":"2024-03-28T23:35:12.641279+0000","last_active":"2024-03-28T23:35:13.659923+0000","last_peered":"2024-03-28T23:35:13.659923+0000","last_clean":"2024-03-28T23:35:13.659923+0000","last_became_active":"2024-03-28T23:35:12.640940+0000","last_became_peered":"2024-03-28T23:35:12.640940+0000","last_unstale":"2024-03-28T23:35:13.659923+0000","last_undegraded":"2024-03-28T23:35:13.659923+0000","last_fullsized":"2024-03-28T23:35:13.659923+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:08:22.631411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.664911+0000","last_change":"2024-03-28T23:35:13.663088+0000","last_active":"2024-03-28T23:35:13.664911+0000","last_peered":"2024-03-28T23:35:13.664911+0000","last_clean":"2024-03-28T23:35:13.664911+0000","last_became_active":"2024-03-28T23:35:13.662832+0000","last_became_peered":"2024-03-28T23:35:13.662832+0000","last_unstale":"2024-03-28T23:35:13.664911+0000","last_undegraded":"2024-03-28T23:35:13.664911+0000","last_fullsized":"2024-03-28T23:35:13.664911+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:35:56.095101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.638094+0000","last_change":"2024-03-28T23:35:12.647070+0000","last_active":"2024-03-28T23:35:13.638094+0000","last_peered":"2024-03-28T23:35:13.638094+0000","last_clean":"2024-03-28T23:35:13.638094+0000","last_became_active":"2024-03-28T23:35:12.646725+0000","last_became_peered":"2024-03-28T23:35:12.646725+0000","last_unstale":"2024-03-28T23:35:13.638094+0000","last_undegraded":"2024-03-28T23:35:13.638094+0000","last_fullsized":"2024-03-28T23:35:13.638094+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:08:22.631411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.656716+0000","last_change":"2024-03-28T23:35:13.650244+0000","last_active":"2024-03-28T23:35:13.656716+0000","last_peered":"2024-03-28T23:35:13.656716+0000","last_clean":"2024-03-28T23:35:13.656716+0000","last_became_active":"2024-03-28T23:35:13.649988+0000","last_became_peered":"2024-03-28T23:35:13.649988+0000","last_unstale":"2024-03-28T23:35:13.656716+0000","last_undegraded":"2024-03-28T23:35:13.656716+0000","last_fullsized":"2024-03-28T23:35:13.656716+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:17.724044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.644457+0000","last_change":"2024-03-28T23:35:12.639799+0000","last_active":"2024-03-28T23:35:13.644457+0000","last_peered":"2024-03-28T23:35:13.644457+0000","last_clean":"2024-03-28T23:35:13.644457+0000","last_became_active":"2024-03-28T23:35:12.639284+0000","last_became_peered":"2024-03-28T23:35:12.639284+0000","last_unstale":"2024-03-28T23:35:13.644457+0000","last_undegraded":"2024-03-28T23:35:13.644457+0000","last_fullsized":"2024-03-28T23:35:13.644457+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:02.820689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.661000+0000","last_change":"2024-03-28T23:35:13.656089+0000","last_active":"2024-03-28T23:35:13.661000+0000","last_peered":"2024-03-28T23:35:13.661000+0000","last_clean":"2024-03-28T23:35:13.661000+0000","last_became_active":"2024-03-28T23:35:13.655818+0000","last_became_peered":"2024-03-28T23:35:13.655818+0000","last_unstale":"2024-03-28T23:35:13.661000+0000","last_undegraded":"2024-03-28T23:35:13.661000+0000","last_fullsized":"2024-03-28T23:35:13.661000+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:35.363250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.652401+0000","last_change":"2024-03-28T23:35:12.647657+0000","last_active":"2024-03-28T23:35:13.652401+0000","last_peered":"2024-03-28T23:35:13.652401+0000","last_clean":"2024-03-28T23:35:13.652401+0000","last_became_active":"2024-03-28T23:35:12.647393+0000","last_became_peered":"2024-03-28T23:35:12.647393+0000","last_unstale":"2024-03-28T23:35:13.652401+0000","last_undegraded":"2024-03-28T23:35:13.652401+0000","last_fullsized":"2024-03-28T23:35:13.652401+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:56:32.469078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.664435+0000","last_change":"2024-03-28T23:35:13.660791+0000","last_active":"2024-03-28T23:35:13.664435+0000","last_peered":"2024-03-28T23:35:13.664435+0000","last_clean":"2024-03-28T23:35:13.664435+0000","last_became_active":"2024-03-28T23:35:13.660461+0000","last_became_peered":"2024-03-28T23:35:13.660461+0000","last_unstale":"2024-03-28T23:35:13.664435+0000","last_undegraded":"2024-03-28T23:35:13.664435+0000","last_fullsized":"2024-03-28T23:35:13.664435+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:30:36.327178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.640915+0000","last_change":"2024-03-28T23:35:12.644816+0000","last_active":"2024-03-28T23:35:13.640915+0000","last_peered":"2024-03-28T23:35:13.640915+0000","last_clean":"2024-03-28T23:35:13.640915+0000","last_became_active":"2024-03-28T23:35:12.644561+0000","last_became_peered":"2024-03-28T23:35:12.644561+0000","last_unstale":"2024-03-28T23:35:13.640915+0000","last_undegraded":"2024-03-28T23:35:13.640915+0000","last_fullsized":"2024-03-28T23:35:13.640915+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:47:30.593198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.658281+0000","last_change":"2024-03-28T23:35:13.650966+0000","last_active":"2024-03-28T23:35:13.658281+0000","last_peered":"2024-03-28T23:35:13.658281+0000","last_clean":"2024-03-28T23:35:13.658281+0000","last_became_active":"2024-03-28T23:35:13.650706+0000","last_became_peered":"2024-03-28T23:35:13.650706+0000","last_unstale":"2024-03-28T23:35:13.658281+0000","last_undegraded":"2024-03-28T23:35:13.658281+0000","last_fullsized":"2024-03-28T23:35:13.658281+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:42:35.979766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.658369+0000","last_change":"2024-03-28T23:35:12.642111+0000","last_active":"2024-03-28T23:35:13.658369+0000","last_peered":"2024-03-28T23:35:13.658369+0000","last_clean":"2024-03-28T23:35:13.658369+0000","last_became_active":"2024-03-28T23:35:12.641157+0000","last_became_peered":"2024-03-28T23:35:12.641157+0000","last_unstale":"2024-03-28T23:35:13.658369+0000","last_undegraded":"2024-03-28T23:35:13.658369+0000","last_fullsized":"2024-03-28T23:35:13.658369+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:09:34.356556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.662961+0000","last_change":"2024-03-28T23:35:13.660571+0000","last_active":"2024-03-28T23:35:13.662961+0000","last_peered":"2024-03-28T23:35:13.662961+0000","last_clean":"2024-03-28T23:35:13.662961+0000","last_became_active":"2024-03-28T23:35:13.659952+0000","last_became_peered":"2024-03-28T23:35:13.659952+0000","last_unstale":"2024-03-28T23:35:13.662961+0000","last_undegraded":"2024-03-28T23:35:13.662961+0000","last_fullsized":"2024-03-28T23:35:13.662961+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:10.144427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.638589+0000","last_change":"2024-03-28T23:35:12.635657+0000","last_active":"2024-03-28T23:35:13.638589+0000","last_peered":"2024-03-28T23:35:13.638589+0000","last_clean":"2024-03-28T23:35:13.638589+0000","last_became_active":"2024-03-28T23:35:12.635355+0000","last_became_peered":"2024-03-28T23:35:12.635355+0000","last_unstale":"2024-03-28T23:35:13.638589+0000","last_undegraded":"2024-03-28T23:35:13.638589+0000","last_fullsized":"2024-03-28T23:35:13.638589+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:43:51.633196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":9,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.658099+0000","last_change":"2024-03-28T23:35:13.655220+0000","last_active":"2024-03-28T23:35:13.658099+0000","last_peered":"2024-03-28T23:35:13.658099+0000","last_clean":"2024-03-28T23:35:13.658099+0000","last_became_active":"2024-03-28T23:35:13.654937+0000","last_became_peered":"2024-03-28T23:35:13.654937+0000","last_unstale":"2024-03-28T23:35:13.658099+0000","last_undegraded":"2024-03-28T23:35:13.658099+0000","last_fullsized":"2024-03-28T23:35:13.658099+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:12.624154+0000","last_clean_scrub_stamp":"2024-03-28T23:35:12.624154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:20:06.310533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"135'6","reported_seq":18,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.672609+0000","last_change":"2024-03-28T23:35:12.648536+0000","last_active":"2024-03-28T23:35:13.672609+0000","last_peered":"2024-03-28T23:35:13.672609+0000","last_clean":"2024-03-28T23:35:13.672609+0000","last_became_active":"2024-03-28T23:35:12.648243+0000","last_became_peered":"2024-03-28T23:35:12.648243+0000","last_unstale":"2024-03-28T23:35:13.672609+0000","last_undegraded":"2024-03-28T23:35:13.672609+0000","last_fullsized":"2024-03-28T23:35:13.672609+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:11.617460+0000","last_clean_scrub_stamp":"2024-03-28T23:35:11.617460+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:45:26.072989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":385,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-28T23:35:13.638646+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:35:13.638646+0000","last_peered":"2024-03-28T23:35:13.638646+0000","last_clean":"2024-03-28T23:35:13.638646+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:35:13.638646+0000","last_undegraded":"2024-03-28T23:35:13.638646+0000","last_fullsized":"2024-03-28T23:35:13.638646+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574917,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32904,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338936,"statfs":{"total":96636764160,"available":96603070464,"internally_reserved":0,"allocated":1765376,"data_stored":1408064,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574918,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1699840,"data_stored":817744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574918,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32844,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1695744,"data_stored":817674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574917,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32856,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338984,"statfs":{"total":96636764160,"available":96603119616,"internally_reserved":0,"allocated":1708032,"data_stored":819286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32852,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338988,"statfs":{"total":96636764160,"available":96603123712,"internally_reserved":0,"allocated":1703936,"data_stored":818290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574917,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32852,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338988,"statfs":{"total":96636764160,"available":96603123712,"internally_reserved":0,"allocated":1703936,"data_stored":817786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1699840,"data_stored":817744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574917,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32896,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338944,"statfs":{"total":96636764160,"available":96603078656,"internally_reserved":0,"allocated":1748992,"data_stored":1400875,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:35:17.305 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T23:35:17.306 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-28T23:35:17.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.451+0000 7efffdafb640 1 Processor -- start 2024-03-28T23:35:17.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.451+0000 7efffdafb640 1 -- start start 2024-03-28T23:35:17.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.451+0000 7efffdafb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efff813c430 0x7efff80c2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:17.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.451+0000 7efffdafb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efff8072b50 0x7efff8072fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:17.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.451+0000 7efffdafb640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efff8153500 con 0x7efff8152fb0 2024-03-28T23:35:17.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.451+0000 7efffdafb640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7efff813c810 con 0x7efff8072b50 2024-03-28T23:35:17.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.451+0000 7efffdafb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7efff81537e0 con 0x7efff813c430 2024-03-28T23:35:17.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.452+0000 7efff7fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efff8072b50 0x7efff8072fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:17.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.452+0000 7efff7fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efff8072b50 0x7efff8072fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47618/0 (socket says 172.21.15.161:47618) 2024-03-28T23:35:17.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.452+0000 7efff7fff640 1 -- 172.21.15.161:0/173869390 learned_addr learned my addr 172.21.15.161:0/173869390 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:17.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.452+0000 7efff77fe640 1 --2- 172.21.15.161:0/173869390 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efff813c430 0x7efff80c2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:17.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.452+0000 7efff7fff640 1 -- 172.21.15.161:0/173869390 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efff813c430 msgr2=0x7efff80c2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:17.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.452+0000 7efff7fff640 1 --2- 172.21.15.161:0/173869390 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efff813c430 0x7efff80c2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.452+0000 7efff7fff640 1 -- 172.21.15.161:0/173869390 >> v1:172.21.15.170:6789/0 conn(0x7efff8152fb0 legacy=0x7efff806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:17.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.452+0000 7efff7fff640 1 -- 172.21.15.161:0/173869390 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efff8153ac0 con 0x7efff8072b50 2024-03-28T23:35:17.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.452+0000 7efff77fe640 1 --2- 172.21.15.161:0/173869390 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efff813c430 0x7efff80c2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:17.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.453+0000 7efff7fff640 1 --2- 172.21.15.161:0/173869390 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efff8072b50 0x7efff8072fe0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7efff0009e50 tx=0x7efff002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6adf4a5527d3782 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:17.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.453+0000 7efff67fc640 1 -- 172.21.15.161:0/173869390 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff002c9e0 con 0x7efff8072b50 2024-03-28T23:35:17.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.453+0000 7efff67fc640 1 -- 172.21.15.161:0/173869390 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efff002cb80 con 0x7efff8072b50 2024-03-28T23:35:17.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.453+0000 7efff67fc640 1 -- 172.21.15.161:0/173869390 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff00343f0 con 0x7efff8072b50 2024-03-28T23:35:17.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.453+0000 7efffdafb640 1 -- 172.21.15.161:0/173869390 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efff8072b50 msgr2=0x7efff8072fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:17.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.453+0000 7efffdafb640 1 --2- 172.21.15.161:0/173869390 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efff8072b50 0x7efff8072fe0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7efff0009e50 tx=0x7efff002be40 comp rx=0 tx=0).stop 2024-03-28T23:35:17.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.453+0000 7efffdafb640 1 -- 172.21.15.161:0/173869390 shutdown_connections 2024-03-28T23:35:17.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.453+0000 7efffdafb640 1 --2- 172.21.15.161:0/173869390 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efff8072b50 0x7efff8072fe0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.453+0000 7efffdafb640 1 --2- 172.21.15.161:0/173869390 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efff813c430 0x7efff80c2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.453+0000 7efffdafb640 1 -- 172.21.15.161:0/173869390 >> 172.21.15.161:0/173869390 conn(0x7efff80c7760 msgr2=0x7efff80c7b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:17.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.453+0000 7efffdafb640 1 -- 172.21.15.161:0/173869390 shutdown_connections 2024-03-28T23:35:17.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.453+0000 7efffdafb640 1 -- 172.21.15.161:0/173869390 wait complete. 2024-03-28T23:35:17.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.454+0000 7efffdafb640 1 Processor -- start 2024-03-28T23:35:17.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.455+0000 7efffdafb640 1 -- start start 2024-03-28T23:35:17.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.455+0000 7efffdafb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efff8072b50 0x7efff81eec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:17.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.455+0000 7efffdafb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efff813c430 0x7efff8134bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:17.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.455+0000 7efffdafb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efff8152fb0 0x7efff8155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:17.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.455+0000 7efffdafb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efff806eb30 con 0x7efff8072b50 2024-03-28T23:35:17.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.455+0000 7efffdafb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7efff80c3650 con 0x7efff8152fb0 2024-03-28T23:35:17.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.455+0000 7efffdafb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7efff806ecb0 con 0x7efff813c430 2024-03-28T23:35:17.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.455+0000 7efff77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efff8072b50 0x7efff81eec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:17.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.455+0000 7efff77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efff8072b50 0x7efff81eec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47626/0 (socket says 172.21.15.161:47626) 2024-03-28T23:35:17.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.455+0000 7efff77fe640 1 -- 172.21.15.161:0/4117095437 learned_addr learned my addr 172.21.15.161:0/4117095437 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:17.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.456+0000 7efff77fe640 1 -- 172.21.15.161:0/4117095437 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efff813c430 msgr2=0x7efff8134bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:17.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.456+0000 7efff77fe640 1 --2- 172.21.15.161:0/4117095437 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efff813c430 0x7efff8134bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.456+0000 7efff77fe640 1 -- 172.21.15.161:0/4117095437 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efff8152fb0 msgr2=0x7efff8155630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:17.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.456+0000 7efff77fe640 1 --2- 172.21.15.161:0/4117095437 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efff8152fb0 0x7efff8155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.456+0000 7efff77fe640 1 -- 172.21.15.161:0/4117095437 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efff81ef1d0 con 0x7efff8072b50 2024-03-28T23:35:17.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.456+0000 7efff77fe640 1 --2- 172.21.15.161:0/4117095437 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efff8072b50 0x7efff81eec90 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7effe800c7f0 tx=0x7effe800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:17.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.456+0000 7effdffff640 1 -- 172.21.15.161:0/4117095437 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe8015070 con 0x7efff8072b50 2024-03-28T23:35:17.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.456+0000 7effdffff640 1 -- 172.21.15.161:0/4117095437 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effe8007820 con 0x7efff8072b50 2024-03-28T23:35:17.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.456+0000 7efffdafb640 1 -- 172.21.15.161:0/4117095437 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efff80be5e0 con 0x7efff8072b50 2024-03-28T23:35:17.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.456+0000 7effdffff640 1 -- 172.21.15.161:0/4117095437 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe8007bd0 con 0x7efff8072b50 2024-03-28T23:35:17.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.456+0000 7efffdafb640 1 -- 172.21.15.161:0/4117095437 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efff8139f30 con 0x7efff8072b50 2024-03-28T23:35:17.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.457+0000 7efffdafb640 1 -- 172.21.15.161:0/4117095437 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efff81eb730 con 0x7efff8072b50 2024-03-28T23:35:17.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.458+0000 7effdffff640 1 -- 172.21.15.161:0/4117095437 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7effe8018070 con 0x7efff8072b50 2024-03-28T23:35:17.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.458+0000 7effdffff640 1 --2- 172.21.15.161:0/4117095437 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7effe406b4d0 0x7effe406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:17.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.458+0000 7effdffff640 1 -- 172.21.15.161:0/4117095437 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 6101+0+0 (secure 0 0 0) 0x7effe808a4c0 con 0x7efff8072b50 2024-03-28T23:35:17.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.458+0000 7efff6ffd640 1 --2- 172.21.15.161:0/4117095437 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7effe406b4d0 0x7effe406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:17.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.459+0000 7efff6ffd640 1 --2- 172.21.15.161:0/4117095437 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7effe406b4d0 0x7effe406d990 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7effec006fd0 tx=0x7effec006bc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:17.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.462+0000 7effdffff640 1 -- 172.21.15.161:0/4117095437 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7effe80591b0 con 0x7efff8072b50 2024-03-28T23:35:17.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.709+0000 7efffdafb640 1 -- 172.21.15.161:0/4117095437 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7efff804f4b0 con 0x7efff8072b50 2024-03-28T23:35:17.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.711+0000 7effdffff640 1 -- 172.21.15.161:0/4117095437 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) ==== 190+0+147 (secure 0 0 0) 0x7effe8058ab0 con 0x7efff8072b50 2024-03-28T23:35:17.711 INFO:teuthology.orchestra.run.smithi161.stdout:[client.0] 2024-03-28T23:35:17.711 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQA6/QVmc1krAxAAEKsn1Tuuv9OlFZpB4PFhDg== 2024-03-28T23:35:17.711 INFO:teuthology.orchestra.run.smithi161.stdout: caps mds = "allow" 2024-03-28T23:35:17.712 INFO:teuthology.orchestra.run.smithi161.stdout: caps mon = "allow r" 2024-03-28T23:35:17.712 INFO:teuthology.orchestra.run.smithi161.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-28T23:35:17.712 INFO:teuthology.orchestra.run.smithi161.stderr:updated caps for client.0 2024-03-28T23:35:17.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.713+0000 7efffdafb640 1 -- 172.21.15.161:0/4117095437 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7effe406b4d0 msgr2=0x7effe406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:17.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.713+0000 7efffdafb640 1 --2- 172.21.15.161:0/4117095437 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7effe406b4d0 0x7effe406d990 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7effec006fd0 tx=0x7effec006bc0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.713+0000 7efffdafb640 1 -- 172.21.15.161:0/4117095437 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efff8072b50 msgr2=0x7efff81eec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:17.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.713+0000 7efffdafb640 1 --2- 172.21.15.161:0/4117095437 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efff8072b50 0x7efff81eec90 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7effe800c7f0 tx=0x7effe800ccb0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.713+0000 7efffdafb640 1 -- 172.21.15.161:0/4117095437 shutdown_connections 2024-03-28T23:35:17.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.713+0000 7efffdafb640 1 --2- 172.21.15.161:0/4117095437 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7effe406b4d0 0x7effe406d990 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.713+0000 7efffdafb640 1 --2- 172.21.15.161:0/4117095437 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efff8152fb0 0x7efff8155630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.713+0000 7efffdafb640 1 --2- 172.21.15.161:0/4117095437 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efff813c430 0x7efff8134bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.713+0000 7efffdafb640 1 --2- 172.21.15.161:0/4117095437 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efff8072b50 0x7efff81eec90 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.713+0000 7efffdafb640 1 -- 172.21.15.161:0/4117095437 >> 172.21.15.161:0/4117095437 conn(0x7efff80c7760 msgr2=0x7efff8152870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:17.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.713+0000 7efffdafb640 1 -- 172.21.15.161:0/4117095437 shutdown_connections 2024-03-28T23:35:17.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.713+0000 7efffdafb640 1 -- 172.21.15.161:0/4117095437 wait complete. 2024-03-28T23:35:17.724 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:35:17.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.825+0000 7f697088a640 1 Processor -- start 2024-03-28T23:35:17.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.826+0000 7f697088a640 1 -- start start 2024-03-28T23:35:17.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.826+0000 7f697088a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f696c0c5850 0x7f696c155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:17.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.826+0000 7f697088a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f696c0c5d00 con 0x7f696c152890 2024-03-28T23:35:17.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.826+0000 7f697088a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f696c158a30 con 0x7f696c0c5410 2024-03-28T23:35:17.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.826+0000 7f697088a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f696c0c8480 con 0x7f696c0c5850 2024-03-28T23:35:17.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.827+0000 7f696a59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f696c152890 0x7f696c0c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45298/0 (socket says 172.21.15.161:45298) 2024-03-28T23:35:17.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.827+0000 7f696a59c640 1 -- 172.21.15.161:0/1730169968 learned_addr learned my addr 172.21.15.161:0/1730169968 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:17.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.827+0000 7f696959a640 1 -- 172.21.15.161:0/1730169968 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3714891040 0 0) 0x7f696c0c5d00 con 0x7f696c152890 2024-03-28T23:35:17.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.827+0000 7f696959a640 1 -- 172.21.15.161:0/1730169968 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6954003520 con 0x7f696c152890 2024-03-28T23:35:17.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.827+0000 7f696ad9d640 1 --2- 172.21.15.161:0/1730169968 >> v2:172.21.15.170:3300/0 conn(0x7f696c0c5850 0x7f696c155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:17.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.827+0000 7f696959a640 1 -- 172.21.15.161:0/1730169968 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2112919941 0 0) 0x7f6954003520 con 0x7f696c152890 2024-03-28T23:35:17.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.827+0000 7f696959a640 1 -- 172.21.15.161:0/1730169968 >> v2:172.21.15.170:3300/0 conn(0x7f696c0c5850 msgr2=0x7f696c155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:17.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.827+0000 7f696959a640 1 --2- 172.21.15.161:0/1730169968 >> v2:172.21.15.170:3300/0 conn(0x7f696c0c5850 0x7f696c155ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.827+0000 7f696959a640 1 -- 172.21.15.161:0/1730169968 >> v1:172.21.15.170:6789/0 conn(0x7f696c0c5410 legacy=0x7f696c1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:17.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.827+0000 7f696959a640 1 -- 172.21.15.161:0/1730169968 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f696c158d10 con 0x7f696c152890 2024-03-28T23:35:17.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.827+0000 7f696959a640 1 -- 172.21.15.161:0/1730169968 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f695c001670 con 0x7f696c152890 2024-03-28T23:35:17.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.827+0000 7f696959a640 1 -- 172.21.15.161:0/1730169968 >> v1:172.21.15.161:6789/0 conn(0x7f696c152890 legacy=0x7f696c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:17.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.827+0000 7f696959a640 1 --2- 172.21.15.161:0/1730169968 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6954003a90 0x7f6954003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:17.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.827+0000 7f696959a640 1 --2- 172.21.15.161:0/1730169968 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6954004800 0x7f6954004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:17.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.827+0000 7f696959a640 1 --2- 172.21.15.161:0/1730169968 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6954005290 0x7f695400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:17.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.827+0000 7f696959a640 1 -- 172.21.15.161:0/1730169968 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f695400c1b0 con 0x7f6954003a90 2024-03-28T23:35:17.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.827+0000 7f696959a640 1 -- 172.21.15.161:0/1730169968 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f695400c360 con 0x7f6954005290 2024-03-28T23:35:17.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.827+0000 7f696959a640 1 -- 172.21.15.161:0/1730169968 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f695400c560 con 0x7f6954004800 2024-03-28T23:35:17.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.827+0000 7f696a59c640 1 --2- 172.21.15.161:0/1730169968 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6954003a90 0x7f6954003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:17.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.827+0000 7f696ad9d640 1 --2- 172.21.15.161:0/1730169968 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6954005290 0x7f695400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:17.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.827+0000 7f6969d9b640 1 --2- 172.21.15.161:0/1730169968 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6954004800 0x7f6954004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:17.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.828+0000 7f696a59c640 1 -- 172.21.15.161:0/1730169968 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6954004800 msgr2=0x7f6954004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:17.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.828+0000 7f696a59c640 1 --2- 172.21.15.161:0/1730169968 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6954004800 0x7f6954004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.828+0000 7f696a59c640 1 -- 172.21.15.161:0/1730169968 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6954005290 msgr2=0x7f695400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:17.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.828+0000 7f696a59c640 1 --2- 172.21.15.161:0/1730169968 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6954005290 0x7f695400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.828+0000 7f696a59c640 1 -- 172.21.15.161:0/1730169968 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6954003520 con 0x7f6954003a90 2024-03-28T23:35:17.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.828+0000 7f696ad9d640 1 -- 172.21.15.161:0/1730169968 reap_dead start 2024-03-28T23:35:17.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.828+0000 7f6969d9b640 1 --2- 172.21.15.161:0/1730169968 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6954004800 0x7f6954004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:17.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.828+0000 7f696a59c640 1 --2- 172.21.15.161:0/1730169968 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6954003a90 0x7f6954003eb0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f695c00c290 tx=0x7f6954005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=435af96f05f132c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:17.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.828+0000 7f696959a640 1 -- 172.21.15.161:0/1730169968 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695c007d30 con 0x7f6954003a90 2024-03-28T23:35:17.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.829+0000 7f696959a640 1 -- 172.21.15.161:0/1730169968 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f695c030070 con 0x7f6954003a90 2024-03-28T23:35:17.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.829+0000 7f696959a640 1 -- 172.21.15.161:0/1730169968 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695c005d30 con 0x7f6954003a90 2024-03-28T23:35:17.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.829+0000 7f697088a640 1 -- 172.21.15.161:0/1730169968 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6954003a90 msgr2=0x7f6954003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:17.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.829+0000 7f697088a640 1 --2- 172.21.15.161:0/1730169968 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6954003a90 0x7f6954003eb0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f695c00c290 tx=0x7f6954005830 comp rx=0 tx=0).stop 2024-03-28T23:35:17.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.829+0000 7f697088a640 1 -- 172.21.15.161:0/1730169968 shutdown_connections 2024-03-28T23:35:17.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.829+0000 7f697088a640 1 --2- 172.21.15.161:0/1730169968 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6954003a90 0x7f6954003eb0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.829+0000 7f697088a640 1 -- 172.21.15.161:0/1730169968 >> 172.21.15.161:0/1730169968 conn(0x7f696c0ac620 msgr2=0x7f696c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:17.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.829+0000 7f697088a640 1 -- 172.21.15.161:0/1730169968 shutdown_connections 2024-03-28T23:35:17.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.829+0000 7f697088a640 1 -- 172.21.15.161:0/1730169968 wait complete. 2024-03-28T23:35:17.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.830+0000 7f697088a640 1 Processor -- start 2024-03-28T23:35:17.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.830+0000 7f697088a640 1 -- start start 2024-03-28T23:35:17.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.831+0000 7f697088a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6954003a90 0x7f696c0c2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:17.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.831+0000 7f697088a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f696c1ee310 0x7f696c1ebfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:17.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.831+0000 7f697088a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f696c1e7ee0 0x7f696c1e8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:17.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.831+0000 7f697088a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f696c1592d0 con 0x7f6954003a90 2024-03-28T23:35:17.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.831+0000 7f697088a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f696c159640 con 0x7f696c1e7ee0 2024-03-28T23:35:17.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.831+0000 7f697088a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f696c152c70 con 0x7f696c1ee310 2024-03-28T23:35:17.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.831+0000 7f696a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6954003a90 0x7f696c0c2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:17.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.831+0000 7f696a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6954003a90 0x7f696c0c2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47636/0 (socket says 172.21.15.161:47636) 2024-03-28T23:35:17.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.831+0000 7f696a59c640 1 -- 172.21.15.161:0/3937568197 learned_addr learned my addr 172.21.15.161:0/3937568197 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:17.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.831+0000 7f696ad9d640 1 --2- 172.21.15.161:0/3937568197 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f696c1e7ee0 0x7f696c1e8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:17.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.831+0000 7f6969d9b640 1 --2- 172.21.15.161:0/3937568197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f696c1ee310 0x7f696c1ebfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:17.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.831+0000 7f696a59c640 1 -- 172.21.15.161:0/3937568197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f696c1ee310 msgr2=0x7f696c1ebfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:17.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.831+0000 7f696a59c640 1 --2- 172.21.15.161:0/3937568197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f696c1ee310 0x7f696c1ebfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.831+0000 7f696a59c640 1 -- 172.21.15.161:0/3937568197 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f696c1e7ee0 msgr2=0x7f696c1e8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:17.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.831+0000 7f696a59c640 1 --2- 172.21.15.161:0/3937568197 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f696c1e7ee0 0x7f696c1e8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:17.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.831+0000 7f696a59c640 1 -- 172.21.15.161:0/3937568197 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f696c1eeb50 con 0x7f6954003a90 2024-03-28T23:35:17.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.832+0000 7f696ad9d640 1 --2- 172.21.15.161:0/3937568197 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f696c1e7ee0 0x7f696c1e8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:17.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.832+0000 7f6969d9b640 1 --2- 172.21.15.161:0/3937568197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f696c1ee310 0x7f696c1ebfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:17.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.832+0000 7f696a59c640 1 --2- 172.21.15.161:0/3937568197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6954003a90 0x7f696c0c2e80 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f695c00c3b0 tx=0x7f695c001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:17.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.832+0000 7f695affd640 1 -- 172.21.15.161:0/3937568197 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695c001c80 con 0x7f6954003a90 2024-03-28T23:35:17.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.832+0000 7f695affd640 1 -- 172.21.15.161:0/3937568197 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f695c004bf0 con 0x7f6954003a90 2024-03-28T23:35:17.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.832+0000 7f697088a640 1 -- 172.21.15.161:0/3937568197 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f696c0c2680 con 0x7f6954003a90 2024-03-28T23:35:17.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.832+0000 7f695affd640 1 -- 172.21.15.161:0/3937568197 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695c007d30 con 0x7f6954003a90 2024-03-28T23:35:17.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.832+0000 7f697088a640 1 -- 172.21.15.161:0/3937568197 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f696c158740 con 0x7f6954003a90 2024-03-28T23:35:17.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.833+0000 7f697088a640 1 -- 172.21.15.161:0/3937568197 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f696c04f4b0 con 0x7f6954003a90 2024-03-28T23:35:17.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.834+0000 7f695affd640 1 -- 172.21.15.161:0/3937568197 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f695c048020 con 0x7f6954003a90 2024-03-28T23:35:17.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.834+0000 7f695affd640 1 --2- 172.21.15.161:0/3937568197 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f694c06b500 0x7f694c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:17.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.834+0000 7f695affd640 1 -- 172.21.15.161:0/3937568197 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 6101+0+0 (secure 0 0 0) 0x7f695c0ab520 con 0x7f6954003a90 2024-03-28T23:35:17.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.834+0000 7f6969d9b640 1 --2- 172.21.15.161:0/3937568197 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f694c06b500 0x7f694c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:17.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.835+0000 7f6969d9b640 1 --2- 172.21.15.161:0/3937568197 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f694c06b500 0x7f694c06d9c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f6960000c00 tx=0x7f6960009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:17.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:17.839+0000 7f695affd640 1 -- 172.21.15.161:0/3937568197 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f695c07a2a0 con 0x7f6954003a90 2024-03-28T23:35:18.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.054+0000 7f697088a640 1 -- 172.21.15.161:0/3937568197 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f696c0c8140 con 0x7f6954003a90 2024-03-28T23:35:18.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.054+0000 7f695affd640 1 -- 172.21.15.161:0/3937568197 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) ==== 95+0+5203 (secure 0 0 0) 0x7f695c079ba0 con 0x7f6954003a90 2024-03-28T23:35:18.055 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:18.055 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":124,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6740,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3586875977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3586875977},{"type":"v1","addr":"172.21.15.170:6835","nonce":3586875977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120},{"gid":7639,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/3182285172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3182285172},{"type":"v1","addr":"172.21.15.170:6834","nonce":3182285172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120},{"gid":8409,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/3037890643","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3037890643},{"type":"v1","addr":"172.21.15.161:6837","nonce":3037890643}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:12.982566+0000","modified":"2024-03-28T23:35:15.337512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8412},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8412":{"gid":8412,"name":"a","rank":0,"incarnation":121,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6836/2527502405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":2527502405},{"type":"v1","addr":"172.21.15.161:6836","nonce":2527502405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8412,"qdb_cluster":[8412]},"id":10}]} 2024-03-28T23:35:18.055 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 124 2024-03-28T23:35:18.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.056+0000 7f697088a640 1 -- 172.21.15.161:0/3937568197 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f694c06b500 msgr2=0x7f694c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:18.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.056+0000 7f697088a640 1 --2- 172.21.15.161:0/3937568197 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f694c06b500 0x7f694c06d9c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f6960000c00 tx=0x7f6960009040 comp rx=0 tx=0).stop 2024-03-28T23:35:18.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.056+0000 7f697088a640 1 -- 172.21.15.161:0/3937568197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6954003a90 msgr2=0x7f696c0c2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:18.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.056+0000 7f697088a640 1 --2- 172.21.15.161:0/3937568197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6954003a90 0x7f696c0c2e80 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f695c00c3b0 tx=0x7f695c001a40 comp rx=0 tx=0).stop 2024-03-28T23:35:18.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.057+0000 7f697088a640 1 -- 172.21.15.161:0/3937568197 shutdown_connections 2024-03-28T23:35:18.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.057+0000 7f697088a640 1 --2- 172.21.15.161:0/3937568197 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f694c06b500 0x7f694c06d9c0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.057+0000 7f697088a640 1 --2- 172.21.15.161:0/3937568197 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f696c1e7ee0 0x7f696c1e8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.057+0000 7f697088a640 1 --2- 172.21.15.161:0/3937568197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f696c1ee310 0x7f696c1ebfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.057+0000 7f697088a640 1 --2- 172.21.15.161:0/3937568197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6954003a90 0x7f696c0c2e80 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.057+0000 7f697088a640 1 -- 172.21.15.161:0/3937568197 >> 172.21.15.161:0/3937568197 conn(0x7f696c0ac620 msgr2=0x7f696c14cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:18.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.057+0000 7f697088a640 1 -- 172.21.15.161:0/3937568197 shutdown_connections 2024-03-28T23:35:18.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.057+0000 7f697088a640 1 -- 172.21.15.161:0/3937568197 wait complete. 2024-03-28T23:35:18.470 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 124, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:35:12.982566+0000', 'modified': '2024-03-28T23:35:15.337512+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8412}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8412': {'gid': 8412, 'name': 'a', 'rank': 0, 'incarnation': 121, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.161:6836/2527502405', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 2527502405}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 2527502405}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8412, 'qdb_cluster': [8412]} 2024-03-28T23:35:18.470 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:35:18.471 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-28T23:35:18.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.574+0000 7f59e927c640 1 Processor -- start 2024-03-28T23:35:18.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.574+0000 7f59e927c640 1 -- start start 2024-03-28T23:35:18.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.575+0000 7f59e927c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59e4150690 0x7f59e4072fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.575+0000 7f59e927c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f59e4072850 0x7f59e41530c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.575+0000 7f59e927c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59e40c5120 con 0x7f59e4150a70 2024-03-28T23:35:18.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.575+0000 7f59e927c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f59e4156630 con 0x7f59e4072850 2024-03-28T23:35:18.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.575+0000 7f59e927c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f59e40736e0 con 0x7f59e4150690 2024-03-28T23:35:18.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.575+0000 7f59e37fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f59e4072850 0x7f59e41530c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.575+0000 7f59e2ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59e4150690 0x7f59e4072fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.575+0000 7f59e37fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f59e4072850 0x7f59e41530c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:47410/0 (socket says 172.21.15.161:47410) 2024-03-28T23:35:18.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.575+0000 7f59e27fc640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f59e4150a70 0x7f59e40c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:34182/0 (socket says 172.21.15.161:34182) 2024-03-28T23:35:18.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.575+0000 7f59e37fe640 1 -- 172.21.15.161:0/2306905207 learned_addr learned my addr 172.21.15.161:0/2306905207 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:18.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.575+0000 7f59e2ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59e4150690 0x7f59e4072fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:36078/0 (socket says 172.21.15.161:36078) 2024-03-28T23:35:18.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.575+0000 7f59e37fe640 1 -- 172.21.15.161:0/2306905207 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59e4150690 msgr2=0x7f59e4072fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:18.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.575+0000 7f59e37fe640 1 --2- 172.21.15.161:0/2306905207 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59e4150690 0x7f59e4072fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.575+0000 7f59e37fe640 1 -- 172.21.15.161:0/2306905207 >> v1:172.21.15.170:6789/0 conn(0x7f59e4150a70 legacy=0x7f59e40c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:18.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.575+0000 7f59e1ffb640 1 -- 172.21.15.161:0/2306905207 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2092811903 0 0) 0x7f59e40c5120 con 0x7f59e4150a70 2024-03-28T23:35:18.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.575+0000 7f59e37fe640 1 -- 172.21.15.161:0/2306905207 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e41568f0 con 0x7f59e4072850 2024-03-28T23:35:18.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.576+0000 7f59e37fe640 1 --2- 172.21.15.161:0/2306905207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59e4072850 0x7f59e41530c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f59dc009e50 tx=0x7f59dc02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26769386cbc52321 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.576+0000 7f59e1ffb640 1 -- 172.21.15.161:0/2306905207 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc02cb00 con 0x7f59e4072850 2024-03-28T23:35:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.576+0000 7f59e1ffb640 1 -- 172.21.15.161:0/2306905207 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59dc02cca0 con 0x7f59e4072850 2024-03-28T23:35:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.576+0000 7f59e1ffb640 1 -- 172.21.15.161:0/2306905207 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc034610 con 0x7f59e4072850 2024-03-28T23:35:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.577+0000 7f59e927c640 1 -- 172.21.15.161:0/2306905207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59e4072850 msgr2=0x7f59e41530c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.577+0000 7f59e927c640 1 --2- 172.21.15.161:0/2306905207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59e4072850 0x7f59e41530c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f59dc009e50 tx=0x7f59dc02bf60 comp rx=0 tx=0).stop 2024-03-28T23:35:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.577+0000 7f59e927c640 1 -- 172.21.15.161:0/2306905207 shutdown_connections 2024-03-28T23:35:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.577+0000 7f59e927c640 1 --2- 172.21.15.161:0/2306905207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59e4072850 0x7f59e41530c0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.577+0000 7f59e927c640 1 --2- 172.21.15.161:0/2306905207 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59e4150690 0x7f59e4072fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.577+0000 7f59e927c640 1 -- 172.21.15.161:0/2306905207 >> 172.21.15.161:0/2306905207 conn(0x7f59e40c88b0 msgr2=0x7f59e40c8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.577+0000 7f59e927c640 1 -- 172.21.15.161:0/2306905207 shutdown_connections 2024-03-28T23:35:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.577+0000 7f59e927c640 1 -- 172.21.15.161:0/2306905207 wait complete. 2024-03-28T23:35:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.578+0000 7f59e927c640 1 Processor -- start 2024-03-28T23:35:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.578+0000 7f59e927c640 1 -- start start 2024-03-28T23:35:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.579+0000 7f59e927c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59e4072850 0x7f59e40c1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.579+0000 7f59e927c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59e4150690 0x7f59e406da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.579+0000 7f59e927c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59e4150a70 0x7f59e4144d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.579+0000 7f59e927c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f59e40c5a50 con 0x7f59e4150690 2024-03-28T23:35:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.579+0000 7f59e927c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f59e41573d0 con 0x7f59e4150a70 2024-03-28T23:35:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.579+0000 7f59e927c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f59e40c5bd0 con 0x7f59e4072850 2024-03-28T23:35:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.579+0000 7f59e27fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59e4150690 0x7f59e406da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.579+0000 7f59e27fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59e4150690 0x7f59e406da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47648/0 (socket says 172.21.15.161:47648) 2024-03-28T23:35:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.579+0000 7f59e27fc640 1 -- 172.21.15.161:0/3548891971 learned_addr learned my addr 172.21.15.161:0/3548891971 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.579+0000 7f59e2ffd640 1 --2- 172.21.15.161:0/3548891971 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59e4072850 0x7f59e40c1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.579+0000 7f59e37fe640 1 --2- 172.21.15.161:0/3548891971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59e4150a70 0x7f59e4144d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.579+0000 7f59e27fc640 1 -- 172.21.15.161:0/3548891971 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59e4072850 msgr2=0x7f59e40c1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:18.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.579+0000 7f59e27fc640 1 --2- 172.21.15.161:0/3548891971 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59e4072850 0x7f59e40c1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.579+0000 7f59e27fc640 1 -- 172.21.15.161:0/3548891971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59e4150a70 msgr2=0x7f59e4144d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:18.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.579+0000 7f59e27fc640 1 --2- 172.21.15.161:0/3548891971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59e4150a70 0x7f59e4144d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.579+0000 7f59e27fc640 1 -- 172.21.15.161:0/3548891971 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e41ede40 con 0x7f59e4150690 2024-03-28T23:35:18.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.579+0000 7f59e27fc640 1 --2- 172.21.15.161:0/3548891971 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59e4150690 0x7f59e406da20 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f59d800c430 tx=0x7f59d800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:18.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.580+0000 7f59cb7fe640 1 -- 172.21.15.161:0/3548891971 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d80074e0 con 0x7f59e4150690 2024-03-28T23:35:18.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.580+0000 7f59cb7fe640 1 -- 172.21.15.161:0/3548891971 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59d80047e0 con 0x7f59e4150690 2024-03-28T23:35:18.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.580+0000 7f59e927c640 1 -- 172.21.15.161:0/3548891971 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59e41ee120 con 0x7f59e4150690 2024-03-28T23:35:18.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.580+0000 7f59cb7fe640 1 -- 172.21.15.161:0/3548891971 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d8004bd0 con 0x7f59e4150690 2024-03-28T23:35:18.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.580+0000 7f59e927c640 1 -- 172.21.15.161:0/3548891971 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59e41ee6e0 con 0x7f59e4150690 2024-03-28T23:35:18.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.581+0000 7f59cb7fe640 1 -- 172.21.15.161:0/3548891971 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f59d8011070 con 0x7f59e4150690 2024-03-28T23:35:18.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.582+0000 7f59cb7fe640 1 --2- 172.21.15.161:0/3548891971 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f59d006b4d0 0x7f59d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.582+0000 7f59e2ffd640 1 --2- 172.21.15.161:0/3548891971 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f59d006b4d0 0x7f59d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.582+0000 7f59cb7fe640 1 -- 172.21.15.161:0/3548891971 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 6101+0+0 (secure 0 0 0) 0x7f59d8089fc0 con 0x7f59e4150690 2024-03-28T23:35:18.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.582+0000 7f59e2ffd640 1 --2- 172.21.15.161:0/3548891971 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f59d006b4d0 0x7f59d006d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f59e414be60 tx=0x7f59d4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:18.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.582+0000 7f59ebe59740 1 Processor -- start 2024-03-28T23:35:18.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.583+0000 7f59ebe59740 1 -- start start 2024-03-28T23:35:18.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.583+0000 7f59ebe59740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c3bdb0 0x55aba3c39890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.583+0000 7f59ebe59740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aba3c3c190 0x55aba3c39dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.583+0000 7f59ebe59740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aba3c3a310 0x55aba3c364e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.583+0000 7f59ebe59740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55aba3b9faf0 con 0x55aba3c3bdb0 2024-03-28T23:35:18.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.583+0000 7f59ebe59740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55aba3b489d0 con 0x55aba3c3c190 2024-03-28T23:35:18.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.583+0000 7f59ebe59740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55aba3b9bb90 con 0x55aba3c3a310 2024-03-28T23:35:18.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.583+0000 7f59e2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c3bdb0 0x55aba3c39890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.583+0000 7f59e2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c3bdb0 0x55aba3c39890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47656/0 (socket says 172.21.15.161:47656) 2024-03-28T23:35:18.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.583+0000 7f59e2ffd640 1 -- 172.21.15.161:0/3664060681 learned_addr learned my addr 172.21.15.161:0/3664060681 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:18.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.584+0000 7f59e37fe640 1 --2- 172.21.15.161:0/3664060681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aba3c3a310 0x55aba3c364e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.584+0000 7f59e27fc640 1 --2- 172.21.15.161:0/3664060681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aba3c3c190 0x55aba3c39dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.584+0000 7f59e2ffd640 1 -- 172.21.15.161:0/3664060681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aba3c3a310 msgr2=0x55aba3c364e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:18.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.584+0000 7f59e2ffd640 1 --2- 172.21.15.161:0/3664060681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aba3c3a310 0x55aba3c364e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.584+0000 7f59e2ffd640 1 -- 172.21.15.161:0/3664060681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aba3c3c190 msgr2=0x55aba3c39dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:18.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.584+0000 7f59e2ffd640 1 --2- 172.21.15.161:0/3664060681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aba3c3c190 0x55aba3c39dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.584+0000 7f59e2ffd640 1 -- 172.21.15.161:0/3664060681 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aba38b42d0 con 0x55aba3c3bdb0 2024-03-28T23:35:18.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.584+0000 7f59e37fe640 1 --2- 172.21.15.161:0/3664060681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aba3c3a310 0x55aba3c364e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:18.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.584+0000 7f59e2ffd640 1 --2- 172.21.15.161:0/3664060681 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c3bdb0 0x55aba3c39890 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f59d4011cc0 tx=0x7f59d4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f7479b451f33fa7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:18.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.584+0000 7f59c8ff9640 1 -- 172.21.15.161:0/3664060681 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4011cf0 con 0x55aba3c3bdb0 2024-03-28T23:35:18.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.584+0000 7f59c8ff9640 1 -- 172.21.15.161:0/3664060681 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59d4026070 con 0x55aba3c3bdb0 2024-03-28T23:35:18.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.584+0000 7f59c8ff9640 1 -- 172.21.15.161:0/3664060681 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4021640 con 0x55aba3c3bdb0 2024-03-28T23:35:18.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.585+0000 7f59ebe59740 1 -- 172.21.15.161:0/3664060681 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c3bdb0 msgr2=0x55aba3c39890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:18.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.585+0000 7f59ebe59740 1 --2- 172.21.15.161:0/3664060681 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c3bdb0 0x55aba3c39890 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f59d4011cc0 tx=0x7f59d4014000 comp rx=0 tx=0).stop 2024-03-28T23:35:18.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.585+0000 7f59ebe59740 1 -- 172.21.15.161:0/3664060681 shutdown_connections 2024-03-28T23:35:18.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.585+0000 7f59ebe59740 1 --2- 172.21.15.161:0/3664060681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aba3c3a310 0x55aba3c364e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.585+0000 7f59ebe59740 1 --2- 172.21.15.161:0/3664060681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aba3c3c190 0x55aba3c39dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.585+0000 7f59ebe59740 1 --2- 172.21.15.161:0/3664060681 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c3bdb0 0x55aba3c39890 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.585+0000 7f59ebe59740 1 -- 172.21.15.161:0/3664060681 >> 172.21.15.161:0/3664060681 conn(0x55aba3c38630 msgr2=0x55aba3c5d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:18.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.585+0000 7f59ebe59740 1 -- 172.21.15.161:0/3664060681 shutdown_connections 2024-03-28T23:35:18.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.585+0000 7f59ebe59740 1 -- 172.21.15.161:0/3664060681 wait complete. 2024-03-28T23:35:18.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.585+0000 7f59ebe59740 1 -- start start 2024-03-28T23:35:18.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.585+0000 7f59ebe59740 1 Processor -- start 2024-03-28T23:35:18.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.586+0000 7f59ebe59740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aba3c3bdb0 0x55aba3c89440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.586+0000 7f59ebe59740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aba3c3c190 0x55aba3c62150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.586+0000 7f59ebe59740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c63510 0x55aba3c889e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.586+0000 7f59ebe59740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55aba3c5e090 con 0x55aba3c63510 2024-03-28T23:35:18.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.586+0000 7f59ebe59740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55aba3b5bf70 con 0x55aba3c3c190 2024-03-28T23:35:18.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.586+0000 7f59ebe59740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55aba3c5dad0 con 0x55aba3c3bdb0 2024-03-28T23:35:18.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.586+0000 7f59ebe59740 10 client.0 _ll_register_callbacks cb 0x55aba38b3ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:35:18.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.586+0000 7f59e37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c63510 0x55aba3c889e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.586+0000 7f59e37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c63510 0x55aba3c889e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47668/0 (socket says 172.21.15.161:47668) 2024-03-28T23:35:18.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.586+0000 7f59e37fe640 1 -- 172.21.15.161:0/2142223976 learned_addr learned my addr 172.21.15.161:0/2142223976 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:18.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.586+0000 7f59e27fc640 1 --2- 172.21.15.161:0/2142223976 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aba3c3c190 0x55aba3c62150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.587+0000 7f59e37fe640 1 -- 172.21.15.161:0/2142223976 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aba3c3bdb0 msgr2=0x55aba3c89440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:18.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.587+0000 7f59e2ffd640 1 --2- 172.21.15.161:0/2142223976 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aba3c3bdb0 0x55aba3c89440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.587+0000 7f59e37fe640 1 --2- 172.21.15.161:0/2142223976 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aba3c3bdb0 0x55aba3c89440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.587+0000 7f59e37fe640 1 -- 172.21.15.161:0/2142223976 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aba3c3c190 msgr2=0x55aba3c62150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:18.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.587+0000 7f59e37fe640 1 --2- 172.21.15.161:0/2142223976 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aba3c3c190 0x55aba3c62150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.587+0000 7f59e37fe640 1 -- 172.21.15.161:0/2142223976 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aba397e6b0 con 0x55aba3c63510 2024-03-28T23:35:18.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.587+0000 7f59e27fc640 1 --2- 172.21.15.161:0/2142223976 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aba3c3c190 0x55aba3c62150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:18.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.587+0000 7f59e2ffd640 1 --2- 172.21.15.161:0/2142223976 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aba3c3bdb0 0x55aba3c89440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:18.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.587+0000 7f59e37fe640 1 --2- 172.21.15.161:0/2142223976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c63510 0x55aba3c889e0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f59dc009f70 tx=0x7f59dc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95dda469bd3629e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:18.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.587+0000 7f59c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:35:18.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.587+0000 7f59c1ffb640 1 -- 172.21.15.161:0/2142223976 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc009510 con 0x55aba3c63510 2024-03-28T23:35:18.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.587+0000 7f59ebe59740 1 -- 172.21.15.161:0/2142223976 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59d4026070 con 0x55aba3c63510 2024-03-28T23:35:18.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.587+0000 7f59c1ffb640 1 -- 172.21.15.161:0/2142223976 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 6101+0+0 (secure 0 0 0) 0x7f59dc034c30 con 0x55aba3c63510 2024-03-28T23:35:18.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.587+0000 7f59c1ffb640 1 -- 172.21.15.161:0/2142223976 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=124) ==== 24+0+0 (secure 0 0 0) 0x7f59dc050050 con 0x55aba3c63510 2024-03-28T23:35:18.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.587+0000 7f59ebe59740 10 client.8565 fetch_fsmap learned FSMap version 124 2024-03-28T23:35:18.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.587+0000 7f59ebe59740 1 -- 172.21.15.161:0/2142223976 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=124,monmap=2+}) -- 0x55aba3c884c0 con 0x55aba3c63510 2024-03-28T23:35:18.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.587+0000 7f59c1ffb640 1 -- 172.21.15.161:0/2142223976 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59d4026070 con 0x55aba3c63510 2024-03-28T23:35:18.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.587+0000 7f59c1ffb640 1 -- 172.21.15.161:0/2142223976 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 124) ==== 3494+0+0 (secure 0 0 0) 0x7f59dc053020 con 0x55aba3c63510 2024-03-28T23:35:18.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.587+0000 7f59ebe59740 10 client.8565 fetch_fsmap finished waiting for FSMap version 124 2024-03-28T23:35:18.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.587+0000 7f59ebe59740 10 client.8565 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T23:35:18.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.588+0000 7f59ebe59740 20 client.8565 populate_metadata read hostname 'smithi161' 2024-03-28T23:35:18.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.588+0000 7f59ebe59740 1 --2- 172.21.15.161:0/2142223976 >> [v2:172.21.15.161:6834/2527502405,v1:172.21.15.161:6836/2527502405] conn(0x55aba3c8bb90 0x55aba3c58720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.588+0000 7f59ebe59740 4 client.8565 mds_command: new command op to 8412 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:35:18.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.588+0000 7f59ebe59740 1 -- 172.21.15.161:0/2142223976 --> [v2:172.21.15.161:6834/2527502405,v1:172.21.15.161:6836/2527502405] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aba3bad030 con 0x55aba3c8bb90 2024-03-28T23:35:18.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.588+0000 7f59e2ffd640 1 --2- 172.21.15.161:0/2142223976 >> [v2:172.21.15.161:6834/2527502405,v1:172.21.15.161:6836/2527502405] conn(0x55aba3c8bb90 0x55aba3c58720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.588+0000 7f59c1ffb640 10 client.8565 ms_handle_connect on v2:172.21.15.161:6834/2527502405 2024-03-28T23:35:18.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.588+0000 7f59e2ffd640 1 --2- 172.21.15.161:0/2142223976 >> [v2:172.21.15.161:6834/2527502405,v1:172.21.15.161:6836/2527502405] conn(0x55aba3c8bb90 0x55aba3c58720 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f01b10b541637d64 server_cookie=64b6a5a89392cbc2 in_seq=0 out_seq=0 2024-03-28T23:35:18.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.589+0000 7f59c1ffb640 1 -- 172.21.15.161:0/2142223976 <== mds.0 v2:172.21.15.161:6834/2527502405 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55aba3bad030 con 0x55aba3c8bb90 2024-03-28T23:35:18.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.589+0000 7f59c1ffb640 10 client.8565 handle_command_reply: tid=0 2024-03-28T23:35:18.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.589+0000 7f59ebe59740 1 client.8565 shutdown 2024-03-28T23:35:18.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.589+0000 7f59ebe59740 1 -- 172.21.15.161:0/2142223976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c63510 msgr2=0x55aba3c889e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:18.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.589+0000 7f59ebe59740 1 --2- 172.21.15.161:0/2142223976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c63510 0x55aba3c889e0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f59dc009f70 tx=0x7f59dc004ab0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.589+0000 7f59ebe59740 1 -- 172.21.15.161:0/2142223976 shutdown_connections 2024-03-28T23:35:18.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.589+0000 7f59ebe59740 1 --2- 172.21.15.161:0/2142223976 >> [v2:172.21.15.161:6834/2527502405,v1:172.21.15.161:6836/2527502405] conn(0x55aba3c8bb90 0x55aba3c58720 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.589+0000 7f59ebe59740 1 --2- 172.21.15.161:0/2142223976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c63510 0x55aba3c889e0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.589+0000 7f59ebe59740 1 --2- 172.21.15.161:0/2142223976 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aba3c3c190 0x55aba3c62150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.589+0000 7f59c1ffb640 0 client.8565 ms_handle_reset on v2:172.21.15.161:6834/2527502405 2024-03-28T23:35:18.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.589+0000 7f59ebe59740 1 --2- 172.21.15.161:0/2142223976 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aba3c3bdb0 0x55aba3c89440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.589+0000 7f59ebe59740 1 -- 172.21.15.161:0/2142223976 >> 172.21.15.161:0/2142223976 conn(0x55aba3c3a310 msgr2=0x55aba3c352b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:18.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.589+0000 7f59ebe59740 1 -- 172.21.15.161:0/2142223976 shutdown_connections 2024-03-28T23:35:18.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.589+0000 7f59ebe59740 1 -- 172.21.15.161:0/2142223976 wait complete. 2024-03-28T23:35:18.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.589+0000 7f59ebe59740 20 client.8565 trim_cache size 0 max 16384 2024-03-28T23:35:18.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.608+0000 7f59ebe59740 1 Processor -- start 2024-03-28T23:35:18.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.608+0000 7f59ebe59740 1 -- start start 2024-03-28T23:35:18.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.608+0000 7f59ebe59740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aba3c3bdb0 0x55aba3c588e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.608+0000 7f59ebe59740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c3c190 0x55aba3c39400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.608+0000 7f59ebe59740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aba3c63510 0x55aba3c39940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.608+0000 7f59ebe59740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f59e41f7e30 con 0x55aba3c3c190 2024-03-28T23:35:18.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.608+0000 7f59ebe59740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f59e41f6e60 con 0x55aba3c63510 2024-03-28T23:35:18.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.608+0000 7f59ebe59740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f59e41eb6d0 con 0x55aba3c3bdb0 2024-03-28T23:35:18.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.608+0000 7f59e27fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c3c190 0x55aba3c39400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.608+0000 7f59e27fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c3c190 0x55aba3c39400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47682/0 (socket says 172.21.15.161:47682) 2024-03-28T23:35:18.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.609+0000 7f59e27fc640 1 -- 172.21.15.161:0/2240581157 learned_addr learned my addr 172.21.15.161:0/2240581157 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:18.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.609+0000 7f59e2ffd640 1 --2- 172.21.15.161:0/2240581157 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aba3c3bdb0 0x55aba3c588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.609+0000 7f59e37fe640 1 --2- 172.21.15.161:0/2240581157 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aba3c63510 0x55aba3c39940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.609+0000 7f59e27fc640 1 -- 172.21.15.161:0/2240581157 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aba3c3bdb0 msgr2=0x55aba3c588e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:18.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.609+0000 7f59e27fc640 1 --2- 172.21.15.161:0/2240581157 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aba3c3bdb0 0x55aba3c588e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.609+0000 7f59e27fc640 1 -- 172.21.15.161:0/2240581157 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aba3c63510 msgr2=0x55aba3c39940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:18.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.609+0000 7f59e27fc640 1 --2- 172.21.15.161:0/2240581157 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aba3c63510 0x55aba3c39940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.609+0000 7f59e27fc640 1 -- 172.21.15.161:0/2240581157 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aba3c897b0 con 0x55aba3c3c190 2024-03-28T23:35:18.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.609+0000 7f59e27fc640 1 --2- 172.21.15.161:0/2240581157 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c3c190 0x55aba3c39400 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f59d8054aa0 tx=0x7f59d8058280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9475e877403647f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:18.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.609+0000 7f59c3fff640 1 -- 172.21.15.161:0/2240581157 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d805c790 con 0x55aba3c3c190 2024-03-28T23:35:18.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.609+0000 7f59c3fff640 1 -- 172.21.15.161:0/2240581157 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59d8060d50 con 0x55aba3c3c190 2024-03-28T23:35:18.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.609+0000 7f59c3fff640 1 -- 172.21.15.161:0/2240581157 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d8004d70 con 0x55aba3c3c190 2024-03-28T23:35:18.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.610+0000 7f59ebe59740 1 -- 172.21.15.161:0/2240581157 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c3c190 msgr2=0x55aba3c39400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:18.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.610+0000 7f59ebe59740 1 --2- 172.21.15.161:0/2240581157 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c3c190 0x55aba3c39400 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f59d8054aa0 tx=0x7f59d8058280 comp rx=0 tx=0).stop 2024-03-28T23:35:18.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.610+0000 7f59ebe59740 1 -- 172.21.15.161:0/2240581157 shutdown_connections 2024-03-28T23:35:18.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.610+0000 7f59ebe59740 1 --2- 172.21.15.161:0/2240581157 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c3c190 0x55aba3c39400 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.610+0000 7f59ebe59740 1 --2- 172.21.15.161:0/2240581157 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aba3c63510 0x55aba3c39940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.610+0000 7f59ebe59740 1 --2- 172.21.15.161:0/2240581157 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aba3c3bdb0 0x55aba3c588e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.610+0000 7f59ebe59740 1 -- 172.21.15.161:0/2240581157 >> 172.21.15.161:0/2240581157 conn(0x55aba3c3a310 msgr2=0x55aba3c61f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:18.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.610+0000 7f59ebe59740 1 -- 172.21.15.161:0/2240581157 shutdown_connections 2024-03-28T23:35:18.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.610+0000 7f59ebe59740 1 -- 172.21.15.161:0/2240581157 wait complete. 2024-03-28T23:35:18.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.610+0000 7f59ebe59740 1 -- start start 2024-03-28T23:35:18.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.611+0000 7f59ebe59740 1 Processor -- start 2024-03-28T23:35:18.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.612+0000 7f59ebe59740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c3bdb0 0x55aba3c39800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.612+0000 7f59ebe59740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aba3c3c190 0x55aba3cf54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.612+0000 7f59ebe59740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aba3c63510 0x55aba3cef230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.612+0000 7f59ebe59740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55aba3c8ba10 con 0x55aba3c3bdb0 2024-03-28T23:35:18.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.612+0000 7f59ebe59740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f59e41f6810 con 0x55aba3c3c190 2024-03-28T23:35:18.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.612+0000 7f59ebe59740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55aba3c8c060 con 0x55aba3c63510 2024-03-28T23:35:18.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.612+0000 7f59ebe59740 10 client.0 _ll_register_callbacks cb 0x7f59d4009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:35:18.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.612+0000 7f59e2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c3bdb0 0x55aba3c39800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.612+0000 7f59e2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c3bdb0 0x55aba3c39800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47698/0 (socket says 172.21.15.161:47698) 2024-03-28T23:35:18.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.612+0000 7f59e2ffd640 1 -- 172.21.15.161:0/1133645202 learned_addr learned my addr 172.21.15.161:0/1133645202 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.612+0000 7f59e27fc640 1 --2- 172.21.15.161:0/1133645202 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aba3c3c190 0x55aba3cf54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.612+0000 7f59e37fe640 1 --2- 172.21.15.161:0/1133645202 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aba3c63510 0x55aba3cef230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.612+0000 7f59e2ffd640 1 -- 172.21.15.161:0/1133645202 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aba3c63510 msgr2=0x55aba3cef230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.612+0000 7f59e2ffd640 1 --2- 172.21.15.161:0/1133645202 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aba3c63510 0x55aba3cef230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.612+0000 7f59e2ffd640 1 -- 172.21.15.161:0/1133645202 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aba3c3c190 msgr2=0x55aba3cf54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.612+0000 7f59e2ffd640 1 --2- 172.21.15.161:0/1133645202 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aba3c3c190 0x55aba3cf54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.612+0000 7f59e2ffd640 1 -- 172.21.15.161:0/1133645202 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aba3c62500 con 0x55aba3c3bdb0 2024-03-28T23:35:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.612+0000 7f59e37fe640 1 --2- 172.21.15.161:0/1133645202 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aba3c63510 0x55aba3cef230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.612+0000 7f59e2ffd640 1 --2- 172.21.15.161:0/1133645202 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c3bdb0 0x55aba3c39800 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f59d4008010 tx=0x7f59d4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ab7feee60a626b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.612+0000 7f59c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:35:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.612+0000 7f59c1ffb640 1 -- 172.21.15.161:0/1133645202 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4011780 con 0x55aba3c3bdb0 2024-03-28T23:35:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.612+0000 7f59c1ffb640 1 -- 172.21.15.161:0/1133645202 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 6101+0+0 (secure 0 0 0) 0x7f59d4034aa0 con 0x55aba3c3bdb0 2024-03-28T23:35:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.612+0000 7f59ebe59740 1 -- 172.21.15.161:0/1133645202 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59d8060d50 con 0x55aba3c3bdb0 2024-03-28T23:35:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.613+0000 7f59c1ffb640 1 -- 172.21.15.161:0/1133645202 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=124) ==== 24+0+0 (secure 0 0 0) 0x7f59d4042050 con 0x55aba3c3bdb0 2024-03-28T23:35:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.613+0000 7f59ebe59740 10 client.8571 fetch_fsmap learned FSMap version 124 2024-03-28T23:35:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.613+0000 7f59ebe59740 1 -- 172.21.15.161:0/1133645202 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=124,monmap=2+}) -- 0x55aba3cf5a00 con 0x55aba3c3bdb0 2024-03-28T23:35:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.613+0000 7f59c1ffb640 1 -- 172.21.15.161:0/1133645202 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59d8060d50 con 0x55aba3c3bdb0 2024-03-28T23:35:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.613+0000 7f59c1ffb640 1 -- 172.21.15.161:0/1133645202 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 124) ==== 3494+0+0 (secure 0 0 0) 0x7f59d4045020 con 0x55aba3c3bdb0 2024-03-28T23:35:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.613+0000 7f59ebe59740 10 client.8571 fetch_fsmap finished waiting for FSMap version 124 2024-03-28T23:35:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.613+0000 7f59ebe59740 10 client.8571 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T23:35:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.613+0000 7f59ebe59740 20 client.8571 populate_metadata read hostname 'smithi161' 2024-03-28T23:35:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.613+0000 7f59ebe59740 1 --2- 172.21.15.161:0/1133645202 >> [v2:172.21.15.161:6834/2527502405,v1:172.21.15.161:6836/2527502405] conn(0x55aba3c8c5d0 0x55aba3c8c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.613+0000 7f59ebe59740 4 client.8571 mds_command: new command op to 8412 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:35:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.613+0000 7f59ebe59740 1 -- 172.21.15.161:0/1133645202 --> [v2:172.21.15.161:6834/2527502405,v1:172.21.15.161:6836/2527502405] -- command(tid 0: {"prefix": "status"}) -- 0x55aba3ba7380 con 0x55aba3c8c5d0 2024-03-28T23:35:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.613+0000 7f59e27fc640 1 --2- 172.21.15.161:0/1133645202 >> [v2:172.21.15.161:6834/2527502405,v1:172.21.15.161:6836/2527502405] conn(0x55aba3c8c5d0 0x55aba3c8c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.613+0000 7f59e27fc640 1 --2- 172.21.15.161:0/1133645202 >> [v2:172.21.15.161:6834/2527502405,v1:172.21.15.161:6836/2527502405] conn(0x55aba3c8c5d0 0x55aba3c8c9b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59d71053058693be server_cookie=ef77d34d89c784ac in_seq=0 out_seq=0 2024-03-28T23:35:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.613+0000 7f59c1ffb640 10 client.8571 ms_handle_connect on v2:172.21.15.161:6834/2527502405 2024-03-28T23:35:18.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.613+0000 7f59c1ffb640 1 -- 172.21.15.161:0/1133645202 <== mds.0 v2:172.21.15.161:6834/2527502405 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x55aba3ba7380 con 0x55aba3c8c5d0 2024-03-28T23:35:18.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.613+0000 7f59c1ffb640 10 client.8571 handle_command_reply: tid=0 2024-03-28T23:35:18.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.613+0000 7f59ebe59740 1 client.8571 shutdown 2024-03-28T23:35:18.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.614+0000 7f59ebe59740 1 -- 172.21.15.161:0/1133645202 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c3bdb0 msgr2=0x55aba3c39800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:18.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.614+0000 7f59ebe59740 1 --2- 172.21.15.161:0/1133645202 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c3bdb0 0x55aba3c39800 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f59d4008010 tx=0x7f59d4004730 comp rx=0 tx=0).stop 2024-03-28T23:35:18.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.614+0000 7f59ebe59740 1 -- 172.21.15.161:0/1133645202 shutdown_connections 2024-03-28T23:35:18.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.614+0000 7f59ebe59740 1 --2- 172.21.15.161:0/1133645202 >> [v2:172.21.15.161:6834/2527502405,v1:172.21.15.161:6836/2527502405] conn(0x55aba3c8c5d0 0x55aba3c8c9b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.614+0000 7f59ebe59740 1 --2- 172.21.15.161:0/1133645202 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aba3c63510 0x55aba3cef230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.614+0000 7f59ebe59740 1 --2- 172.21.15.161:0/1133645202 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aba3c3c190 0x55aba3cf54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.614+0000 7f59c1ffb640 0 client.8571 ms_handle_reset on v2:172.21.15.161:6834/2527502405 2024-03-28T23:35:18.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.614+0000 7f59ebe59740 1 --2- 172.21.15.161:0/1133645202 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aba3c3bdb0 0x55aba3c39800 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.614+0000 7f59ebe59740 1 -- 172.21.15.161:0/1133645202 >> 172.21.15.161:0/1133645202 conn(0x55aba3c3a310 msgr2=0x55aba3c79710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:18.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.614+0000 7f59ebe59740 1 -- 172.21.15.161:0/1133645202 shutdown_connections 2024-03-28T23:35:18.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.614+0000 7f59ebe59740 1 -- 172.21.15.161:0/1133645202 wait complete. 2024-03-28T23:35:18.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.614+0000 7f59ebe59740 20 client.8571 trim_cache size 0 max 16384 2024-03-28T23:35:18.621 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:35:18.621 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:35:18.621 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:35:18.621 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 8412, 2024-03-28T23:35:18.621 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:35:18.621 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:35:18.621 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:35:18.622 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 4.9711354500000002, 2024-03-28T23:35:18.622 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 124, 2024-03-28T23:35:18.622 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 136, 2024-03-28T23:35:18.622 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 136, 2024-03-28T23:35:18.622 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 10.075164613 2024-03-28T23:35:18.622 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:35:18.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.614+0000 7f59e927c640 1 -- 172.21.15.161:0/3548891971 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f59d006b4d0 msgr2=0x7f59d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:18.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.614+0000 7f59e927c640 1 --2- 172.21.15.161:0/3548891971 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f59d006b4d0 0x7f59d006d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f59e414be60 tx=0x7f59d4008040 comp rx=0 tx=0).stop 2024-03-28T23:35:18.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.614+0000 7f59e927c640 1 -- 172.21.15.161:0/3548891971 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59e4150690 msgr2=0x7f59e406da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:18.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.614+0000 7f59e927c640 1 --2- 172.21.15.161:0/3548891971 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59e4150690 0x7f59e406da20 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f59d800c430 tx=0x7f59d800c8f0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.614+0000 7f59e927c640 1 -- 172.21.15.161:0/3548891971 shutdown_connections 2024-03-28T23:35:18.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.614+0000 7f59e927c640 1 --2- 172.21.15.161:0/3548891971 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f59d006b4d0 0x7f59d006d990 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.614+0000 7f59e927c640 1 --2- 172.21.15.161:0/3548891971 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59e4150690 0x7f59e406da20 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.614+0000 7f59e927c640 1 --2- 172.21.15.161:0/3548891971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59e4150a70 0x7f59e4144d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.614+0000 7f59e927c640 1 --2- 172.21.15.161:0/3548891971 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59e4072850 0x7f59e40c1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.614+0000 7f59e927c640 1 -- 172.21.15.161:0/3548891971 >> 172.21.15.161:0/3548891971 conn(0x7f59e40c88b0 msgr2=0x7f59e414e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:18.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.614+0000 7f59e927c640 1 -- 172.21.15.161:0/3548891971 shutdown_connections 2024-03-28T23:35:18.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.614+0000 7f59e927c640 1 -- 172.21.15.161:0/3548891971 wait complete. 2024-03-28T23:35:18.626 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:35:18.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.729+0000 7f0436c4e640 1 Processor -- start 2024-03-28T23:35:18.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.729+0000 7f0436c4e640 1 -- start start 2024-03-28T23:35:18.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.729+0000 7f0436c4e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f04300c5410 0x7f04300c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.729+0000 7f0436c4e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0430158a30 con 0x7f0430152890 2024-03-28T23:35:18.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.729+0000 7f0436c4e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0430158d10 con 0x7f04301553a0 2024-03-28T23:35:18.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.729+0000 7f0436c4e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f04300c8480 con 0x7f04300c5410 2024-03-28T23:35:18.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.729+0000 7f04349ea640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f0430152890 0x7f04300c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45306/0 (socket says 172.21.15.161:45306) 2024-03-28T23:35:18.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.729+0000 7f04349ea640 1 -- 172.21.15.161:0/846997962 learned_addr learned my addr 172.21.15.161:0/846997962 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:18.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.730+0000 7f042f7fe640 1 -- 172.21.15.161:0/846997962 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 397217334 0 0) 0x7f0430158a30 con 0x7f0430152890 2024-03-28T23:35:18.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.730+0000 7f042ffff640 1 --2- 172.21.15.161:0/846997962 >> v2:172.21.15.170:3300/0 conn(0x7f04300c5410 0x7f04300c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.730+0000 7f042f7fe640 1 -- 172.21.15.161:0/846997962 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0420003520 con 0x7f0430152890 2024-03-28T23:35:18.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.730+0000 7f042f7fe640 1 -- 172.21.15.161:0/846997962 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3024763201 0 0) 0x7f0420003520 con 0x7f0430152890 2024-03-28T23:35:18.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.730+0000 7f042f7fe640 1 -- 172.21.15.161:0/846997962 >> v2:172.21.15.170:3300/0 conn(0x7f04300c5410 msgr2=0x7f04300c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:18.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.730+0000 7f042f7fe640 1 --2- 172.21.15.161:0/846997962 >> v2:172.21.15.170:3300/0 conn(0x7f04300c5410 0x7f04300c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.730+0000 7f042f7fe640 1 -- 172.21.15.161:0/846997962 >> v1:172.21.15.170:6789/0 conn(0x7f04301553a0 legacy=0x7f0430155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:18.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.730+0000 7f042f7fe640 1 -- 172.21.15.161:0/846997962 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0430158ff0 con 0x7f0430152890 2024-03-28T23:35:18.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.730+0000 7f042f7fe640 1 -- 172.21.15.161:0/846997962 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f0424001670 con 0x7f0430152890 2024-03-28T23:35:18.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.730+0000 7f042f7fe640 1 -- 172.21.15.161:0/846997962 >> v1:172.21.15.161:6789/0 conn(0x7f0430152890 legacy=0x7f04300c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:18.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.730+0000 7f042f7fe640 1 --2- 172.21.15.161:0/846997962 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0420003a90 0x7f0420003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.730+0000 7f042f7fe640 1 --2- 172.21.15.161:0/846997962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0420004800 0x7f0420004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.730+0000 7f042f7fe640 1 --2- 172.21.15.161:0/846997962 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0420005290 0x7f042000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.730+0000 7f042f7fe640 1 -- 172.21.15.161:0/846997962 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f042000c1b0 con 0x7f0420005290 2024-03-28T23:35:18.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.730+0000 7f042f7fe640 1 -- 172.21.15.161:0/846997962 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f042000c360 con 0x7f0420004800 2024-03-28T23:35:18.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.730+0000 7f042f7fe640 1 -- 172.21.15.161:0/846997962 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f042000c560 con 0x7f0420003a90 2024-03-28T23:35:18.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.730+0000 7f04351eb640 1 --2- 172.21.15.161:0/846997962 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0420005290 0x7f042000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.730+0000 7f042ffff640 1 --2- 172.21.15.161:0/846997962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0420004800 0x7f0420004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.730+0000 7f04349ea640 1 --2- 172.21.15.161:0/846997962 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0420003a90 0x7f0420003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.731+0000 7f04351eb640 1 -- 172.21.15.161:0/846997962 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0420003a90 msgr2=0x7f0420003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:18.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.731+0000 7f04351eb640 1 --2- 172.21.15.161:0/846997962 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0420003a90 0x7f0420003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.731+0000 7f04351eb640 1 -- 172.21.15.161:0/846997962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0420004800 msgr2=0x7f0420004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:18.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.731+0000 7f04351eb640 1 --2- 172.21.15.161:0/846997962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0420004800 0x7f0420004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.731+0000 7f04351eb640 1 -- 172.21.15.161:0/846997962 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0420003520 con 0x7f0420005290 2024-03-28T23:35:18.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.731+0000 7f042ffff640 1 --2- 172.21.15.161:0/846997962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0420004800 0x7f0420004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:18.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.731+0000 7f04351eb640 1 -- 172.21.15.161:0/846997962 reap_dead start 2024-03-28T23:35:18.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.731+0000 7f04351eb640 1 --2- 172.21.15.161:0/846997962 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0420005290 0x7f042000b9c0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f0428001e50 tx=0x7f0420005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f233dc3a0cb32157 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:18.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.731+0000 7f042f7fe640 1 -- 172.21.15.161:0/846997962 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c009680 con 0x7f0420005290 2024-03-28T23:35:18.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.731+0000 7f042f7fe640 1 -- 172.21.15.161:0/846997962 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0428003b90 con 0x7f0420005290 2024-03-28T23:35:18.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.731+0000 7f042f7fe640 1 -- 172.21.15.161:0/846997962 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0428001a70 con 0x7f0420005290 2024-03-28T23:35:18.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.732+0000 7f0436c4e640 1 -- 172.21.15.161:0/846997962 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0420005290 msgr2=0x7f042000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:18.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.732+0000 7f0436c4e640 1 --2- 172.21.15.161:0/846997962 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0420005290 0x7f042000b9c0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f0428001e50 tx=0x7f0420005830 comp rx=0 tx=0).stop 2024-03-28T23:35:18.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.732+0000 7f0436c4e640 1 -- 172.21.15.161:0/846997962 shutdown_connections 2024-03-28T23:35:18.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.732+0000 7f0436c4e640 1 --2- 172.21.15.161:0/846997962 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0420005290 0x7f042000b9c0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.732+0000 7f0436c4e640 1 -- 172.21.15.161:0/846997962 >> 172.21.15.161:0/846997962 conn(0x7f04300ac620 msgr2=0x7f04300aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:18.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.732+0000 7f0436c4e640 1 -- 172.21.15.161:0/846997962 shutdown_connections 2024-03-28T23:35:18.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.732+0000 7f0436c4e640 1 -- 172.21.15.161:0/846997962 wait complete. 2024-03-28T23:35:18.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.733+0000 7f0436c4e640 1 Processor -- start 2024-03-28T23:35:18.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.733+0000 7f0436c4e640 1 -- start start 2024-03-28T23:35:18.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.734+0000 7f0436c4e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0420005290 0x7f04300c92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.734+0000 7f0436c4e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04301ee890 0x7f0430157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.734+0000 7f0436c4e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f043014cfe0 0x7f04300c4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.734+0000 7f0436c4e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0430152c70 con 0x7f04301ee890 2024-03-28T23:35:18.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.734+0000 7f0436c4e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0430159640 con 0x7f0420005290 2024-03-28T23:35:18.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.734+0000 7f0436c4e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0430152df0 con 0x7f043014cfe0 2024-03-28T23:35:18.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.734+0000 7f042ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04301ee890 0x7f0430157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.734+0000 7f042ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04301ee890 0x7f0430157290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47716/0 (socket says 172.21.15.161:47716) 2024-03-28T23:35:18.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.734+0000 7f042ffff640 1 -- 172.21.15.161:0/2619803761 learned_addr learned my addr 172.21.15.161:0/2619803761 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:18.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.735+0000 7f04349ea640 1 --2- 172.21.15.161:0/2619803761 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0420005290 0x7f04300c92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.735+0000 7f042ffff640 1 -- 172.21.15.161:0/2619803761 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f043014cfe0 msgr2=0x7f04300c4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:18.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.735+0000 7f04351eb640 1 --2- 172.21.15.161:0/2619803761 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f043014cfe0 0x7f04300c4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.735+0000 7f042ffff640 1 --2- 172.21.15.161:0/2619803761 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f043014cfe0 0x7f04300c4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.735+0000 7f042ffff640 1 -- 172.21.15.161:0/2619803761 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0420005290 msgr2=0x7f04300c92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:18.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.735+0000 7f042ffff640 1 --2- 172.21.15.161:0/2619803761 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0420005290 0x7f04300c92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.735+0000 7f042ffff640 1 -- 172.21.15.161:0/2619803761 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04300c9e50 con 0x7f04301ee890 2024-03-28T23:35:18.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.735+0000 7f04351eb640 1 --2- 172.21.15.161:0/2619803761 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f043014cfe0 0x7f04300c4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:18.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.735+0000 7f04349ea640 1 --2- 172.21.15.161:0/2619803761 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0420005290 0x7f04300c92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:18.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.735+0000 7f042ffff640 1 --2- 172.21.15.161:0/2619803761 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04301ee890 0x7f0430157290 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f041c011990 tx=0x7f041c011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:18.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.735+0000 7f042d7fa640 1 -- 172.21.15.161:0/2619803761 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c00fa10 con 0x7f04301ee890 2024-03-28T23:35:18.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.735+0000 7f042d7fa640 1 -- 172.21.15.161:0/2619803761 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f041c00fbb0 con 0x7f04301ee890 2024-03-28T23:35:18.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.735+0000 7f0436c4e640 1 -- 172.21.15.161:0/2619803761 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04300c8800 con 0x7f04301ee890 2024-03-28T23:35:18.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.735+0000 7f042d7fa640 1 -- 172.21.15.161:0/2619803761 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c014440 con 0x7f04301ee890 2024-03-28T23:35:18.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.735+0000 7f0436c4e640 1 -- 172.21.15.161:0/2619803761 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04300c8dc0 con 0x7f04301ee890 2024-03-28T23:35:18.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.737+0000 7f042d7fa640 1 -- 172.21.15.161:0/2619803761 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f041c012040 con 0x7f04301ee890 2024-03-28T23:35:18.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.737+0000 7f0436c4e640 1 -- 172.21.15.161:0/2619803761 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0430067a50 con 0x7f04301ee890 2024-03-28T23:35:18.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.737+0000 7f042d7fa640 1 --2- 172.21.15.161:0/2619803761 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f041406b500 0x7f041406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:18.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.737+0000 7f042d7fa640 1 -- 172.21.15.161:0/2619803761 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 6101+0+0 (secure 0 0 0) 0x7f041c08a8f0 con 0x7f04301ee890 2024-03-28T23:35:18.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.737+0000 7f04349ea640 1 --2- 172.21.15.161:0/2619803761 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f041406b500 0x7f041406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:18.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.737+0000 7f04349ea640 1 --2- 172.21.15.161:0/2619803761 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f041406b500 0x7f041406d9c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0424001050 tx=0x7f0424009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:18.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.741+0000 7f042d7fa640 1 -- 172.21.15.161:0/2619803761 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0430067a50 con 0x7f04301ee890 2024-03-28T23:35:18.904 INFO:tasks.ceph.mds.a.smithi161.stderr:2024-03-28T23:35:18.904+0000 7f84c43af640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:35:18.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.957+0000 7f0436c4e640 1 -- 172.21.15.161:0/2619803761 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f043004f4b0 con 0x7f04301ee890 2024-03-28T23:35:18.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.958+0000 7f042d7fa640 1 -- 172.21.15.161:0/2619803761 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) ==== 95+0+5203 (secure 0 0 0) 0x7f041c019030 con 0x7f04301ee890 2024-03-28T23:35:18.959 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:18.959 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":124,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6740,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3586875977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3586875977},{"type":"v1","addr":"172.21.15.170:6835","nonce":3586875977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120},{"gid":7639,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/3182285172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3182285172},{"type":"v1","addr":"172.21.15.170:6834","nonce":3182285172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120},{"gid":8409,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/3037890643","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3037890643},{"type":"v1","addr":"172.21.15.161:6837","nonce":3037890643}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:12.982566+0000","modified":"2024-03-28T23:35:15.337512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8412},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8412":{"gid":8412,"name":"a","rank":0,"incarnation":121,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6836/2527502405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":2527502405},{"type":"v1","addr":"172.21.15.161:6836","nonce":2527502405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8412,"qdb_cluster":[8412]},"id":10}]} 2024-03-28T23:35:18.959 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 124 2024-03-28T23:35:18.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.960+0000 7f0436c4e640 1 -- 172.21.15.161:0/2619803761 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f041406b500 msgr2=0x7f041406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:18.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.960+0000 7f0436c4e640 1 --2- 172.21.15.161:0/2619803761 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f041406b500 0x7f041406d9c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0424001050 tx=0x7f0424009040 comp rx=0 tx=0).stop 2024-03-28T23:35:18.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.960+0000 7f0436c4e640 1 -- 172.21.15.161:0/2619803761 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04301ee890 msgr2=0x7f0430157290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:18.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.960+0000 7f0436c4e640 1 --2- 172.21.15.161:0/2619803761 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04301ee890 0x7f0430157290 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f041c011990 tx=0x7f041c011e50 comp rx=0 tx=0).stop 2024-03-28T23:35:18.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.960+0000 7f0436c4e640 1 -- 172.21.15.161:0/2619803761 shutdown_connections 2024-03-28T23:35:18.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.960+0000 7f0436c4e640 1 --2- 172.21.15.161:0/2619803761 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f041406b500 0x7f041406d9c0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.960+0000 7f0436c4e640 1 --2- 172.21.15.161:0/2619803761 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04301ee890 0x7f0430157290 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.960+0000 7f0436c4e640 1 --2- 172.21.15.161:0/2619803761 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f043014cfe0 0x7f04300c4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.960+0000 7f0436c4e640 1 --2- 172.21.15.161:0/2619803761 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0420005290 0x7f04300c92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:18.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.960+0000 7f0436c4e640 1 -- 172.21.15.161:0/2619803761 >> 172.21.15.161:0/2619803761 conn(0x7f04300ac620 msgr2=0x7f043006c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:18.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.960+0000 7f0436c4e640 1 -- 172.21.15.161:0/2619803761 shutdown_connections 2024-03-28T23:35:18.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:18.960+0000 7f0436c4e640 1 -- 172.21.15.161:0/2619803761 wait complete. 2024-03-28T23:35:18.976 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 124, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:35:12.982566+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 124, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8412": { "addr": "172.21.15.161:6836/2527502405", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 2527502405, "type": "v2" }, { "addr": "172.21.15.161:6836", "nonce": 2527502405, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8412, "incarnation": 121, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-03-28T23:35:15.337512+0000", "qdb_cluster": [ 8412 ], "qdb_leader": 8412, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8412 } } } ], "standbys": [ { "addr": "172.21.15.170:6835/3586875977", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 3586875977, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 3586875977, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 120, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6740, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.170:6834/3182285172", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 3182285172, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 3182285172, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 120, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7639, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6837/3037890643", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6835", "nonce": 3037890643, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 3037890643, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 120, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8409, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:35:18.976 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-28T23:35:18.977 INFO:tasks.cephfs.test_failover:current = [{'gid': 8412, 'name': 'a', 'rank': 0, 'incarnation': 121, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.161:6836/2527502405', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 2527502405}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 2527502405}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 6740, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/3586875977', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 3586875977}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 3586875977}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 120}, {'gid': 8409, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6837/3037890643', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 3037890643}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 3037890643}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 120}, {'gid': 7639, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6834/3182285172', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 3182285172}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 3182285172}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 120}] 2024-03-28T23:35:18.977 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.b mds_join_fs cephfs 2024-03-28T23:35:19.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.086+0000 7fa8cbdb6640 1 Processor -- start 2024-03-28T23:35:19.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.086+0000 7fa8cbdb6640 1 -- start start 2024-03-28T23:35:19.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.087+0000 7fa8cbdb6640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa8c40c0330 0x7fa8c40c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:19.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.087+0000 7fa8cbdb6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8c40c6f60 con 0x7fa8c4152bc0 2024-03-28T23:35:19.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.087+0000 7fa8cbdb6640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8c40c6120 con 0x7fa8c41589b0 2024-03-28T23:35:19.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.087+0000 7fa8cbdb6640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fa8c40c6400 con 0x7fa8c40c0330 2024-03-28T23:35:19.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.087+0000 7fa8c9b52640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa8c4152bc0 0x7fa8c40c8a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45310/0 (socket says 172.21.15.161:45310) 2024-03-28T23:35:19.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.087+0000 7fa8c9b52640 1 -- 172.21.15.161:0/1740219822 learned_addr learned my addr 172.21.15.161:0/1740219822 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:19.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.087+0000 7fa8c9351640 1 --2- 172.21.15.161:0/1740219822 >> v2:172.21.15.170:3300/0 conn(0x7fa8c40c0330 0x7fa8c40c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:19.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.087+0000 7fa8c8b50640 1 -- 172.21.15.161:0/1740219822 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3206749924 0 0) 0x7fa8c40c6f60 con 0x7fa8c4152bc0 2024-03-28T23:35:19.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.087+0000 7fa8c8b50640 1 -- 172.21.15.161:0/1740219822 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8b0003520 con 0x7fa8c4152bc0 2024-03-28T23:35:19.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.087+0000 7fa8c8b50640 1 -- 172.21.15.161:0/1740219822 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2601814434 0 0) 0x7fa8c40c6120 con 0x7fa8c41589b0 2024-03-28T23:35:19.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.087+0000 7fa8c8b50640 1 -- 172.21.15.161:0/1740219822 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8c40c6f60 con 0x7fa8c41589b0 2024-03-28T23:35:19.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.087+0000 7fa8c8b50640 1 -- 172.21.15.161:0/1740219822 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 396238318 0 0) 0x7fa8b0003520 con 0x7fa8c4152bc0 2024-03-28T23:35:19.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.087+0000 7fa8c8b50640 1 -- 172.21.15.161:0/1740219822 >> v2:172.21.15.170:3300/0 conn(0x7fa8c40c0330 msgr2=0x7fa8c40c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:19.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.087+0000 7fa8c8b50640 1 --2- 172.21.15.161:0/1740219822 >> v2:172.21.15.170:3300/0 conn(0x7fa8c40c0330 0x7fa8c40c9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.087+0000 7fa8c8b50640 1 -- 172.21.15.161:0/1740219822 >> v1:172.21.15.170:6789/0 conn(0x7fa8c41589b0 legacy=0x7fa8c4158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:19.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.087+0000 7fa8c8b50640 1 -- 172.21.15.161:0/1740219822 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c40c96d0 con 0x7fa8c4152bc0 2024-03-28T23:35:19.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.087+0000 7fa8c8b50640 1 -- 172.21.15.161:0/1740219822 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fa8c00019f0 con 0x7fa8c4152bc0 2024-03-28T23:35:19.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.087+0000 7fa8c8b50640 1 -- 172.21.15.161:0/1740219822 >> v1:172.21.15.161:6789/0 conn(0x7fa8c4152bc0 legacy=0x7fa8c40c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:19.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.087+0000 7fa8c8b50640 1 --2- 172.21.15.161:0/1740219822 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8b0003c20 0x7fa8b0004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:19.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.087+0000 7fa8c8b50640 1 --2- 172.21.15.161:0/1740219822 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8b0004800 0x7fa8b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:19.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.087+0000 7fa8c8b50640 1 --2- 172.21.15.161:0/1740219822 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8b0005210 0x7fa8b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:19.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.087+0000 7fa8c8b50640 1 -- 172.21.15.161:0/1740219822 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c40c6400 con 0x7fa8b0003c20 2024-03-28T23:35:19.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.087+0000 7fa8c8b50640 1 -- 172.21.15.161:0/1740219822 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b000c140 con 0x7fa8b0004800 2024-03-28T23:35:19.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.087+0000 7fa8c8b50640 1 -- 172.21.15.161:0/1740219822 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa8b000c340 con 0x7fa8b0005210 2024-03-28T23:35:19.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.088+0000 7fa8c9b52640 1 --2- 172.21.15.161:0/1740219822 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8b0003c20 0x7fa8b0004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:19.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.088+0000 7fa8c9351640 1 --2- 172.21.15.161:0/1740219822 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8b0004800 0x7fa8b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:19.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.088+0000 7fa8ca353640 1 --2- 172.21.15.161:0/1740219822 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8b0005210 0x7fa8b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:19.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.088+0000 7fa8c9b52640 1 -- 172.21.15.161:0/1740219822 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8b0005210 msgr2=0x7fa8b000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:19.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.088+0000 7fa8c9b52640 1 --2- 172.21.15.161:0/1740219822 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8b0005210 0x7fa8b000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.088+0000 7fa8c9b52640 1 -- 172.21.15.161:0/1740219822 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8b0004800 msgr2=0x7fa8b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:19.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.088+0000 7fa8c9b52640 1 --2- 172.21.15.161:0/1740219822 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8b0004800 0x7fa8b0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.088+0000 7fa8c9b52640 1 -- 172.21.15.161:0/1740219822 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8b0003520 con 0x7fa8b0003c20 2024-03-28T23:35:19.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.088+0000 7fa8ca353640 1 -- 172.21.15.161:0/1740219822 reap_dead start 2024-03-28T23:35:19.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.089+0000 7fa8c9b52640 1 --2- 172.21.15.161:0/1740219822 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8b0003c20 0x7fa8b0004020 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fa8c000a800 tx=0x7fa8b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef04a37d77044216 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:19.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.089+0000 7fa8c8b50640 1 -- 172.21.15.161:0/1740219822 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c000a430 con 0x7fa8b0003c20 2024-03-28T23:35:19.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.089+0000 7fa8c8b50640 1 -- 172.21.15.161:0/1740219822 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8c0034650 con 0x7fa8b0003c20 2024-03-28T23:35:19.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.089+0000 7fa8c8b50640 1 -- 172.21.15.161:0/1740219822 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c0034a80 con 0x7fa8b0003c20 2024-03-28T23:35:19.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.089+0000 7fa8cbdb6640 1 -- 172.21.15.161:0/1740219822 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8b0003c20 msgr2=0x7fa8b0004020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:19.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.089+0000 7fa8cbdb6640 1 --2- 172.21.15.161:0/1740219822 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8b0003c20 0x7fa8b0004020 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fa8c000a800 tx=0x7fa8b0005830 comp rx=0 tx=0).stop 2024-03-28T23:35:19.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.089+0000 7fa8cbdb6640 1 -- 172.21.15.161:0/1740219822 shutdown_connections 2024-03-28T23:35:19.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.089+0000 7fa8cbdb6640 1 --2- 172.21.15.161:0/1740219822 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8b0003c20 0x7fa8b0004020 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.089+0000 7fa8cbdb6640 1 -- 172.21.15.161:0/1740219822 >> 172.21.15.161:0/1740219822 conn(0x7fa8c40caa80 msgr2=0x7fa8c40cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:19.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.089+0000 7fa8cbdb6640 1 -- 172.21.15.161:0/1740219822 shutdown_connections 2024-03-28T23:35:19.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.089+0000 7fa8cbdb6640 1 -- 172.21.15.161:0/1740219822 wait complete. 2024-03-28T23:35:19.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.090+0000 7fa8cbdb6640 1 Processor -- start 2024-03-28T23:35:19.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.091+0000 7fa8cbdb6640 1 -- start start 2024-03-28T23:35:19.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.091+0000 7fa8cbdb6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8b0003c20 0x7fa8c40c2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:19.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.091+0000 7fa8cbdb6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8c414a0e0 0x7fa8c4149530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:19.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.091+0000 7fa8cbdb6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8c414a610 0x7fa8c4149a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:19.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.091+0000 7fa8cbdb6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c4152fa0 con 0x7fa8c414a610 2024-03-28T23:35:19.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.091+0000 7fa8cbdb6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c40c6bf0 con 0x7fa8c414a0e0 2024-03-28T23:35:19.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.091+0000 7fa8cbdb6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa8c41ee9e0 con 0x7fa8b0003c20 2024-03-28T23:35:19.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.091+0000 7fa8ca353640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8c414a610 0x7fa8c4149a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:19.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.091+0000 7fa8ca353640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8c414a610 0x7fa8c4149a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47736/0 (socket says 172.21.15.161:47736) 2024-03-28T23:35:19.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.091+0000 7fa8ca353640 1 -- 172.21.15.161:0/3680827486 learned_addr learned my addr 172.21.15.161:0/3680827486 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:19.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.091+0000 7fa8c9b52640 1 --2- 172.21.15.161:0/3680827486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8b0003c20 0x7fa8c40c2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:19.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.091+0000 7fa8c9351640 1 --2- 172.21.15.161:0/3680827486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8c414a0e0 0x7fa8c4149530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:19.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.092+0000 7fa8ca353640 1 -- 172.21.15.161:0/3680827486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8b0003c20 msgr2=0x7fa8c40c2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:19.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.092+0000 7fa8ca353640 1 --2- 172.21.15.161:0/3680827486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8b0003c20 0x7fa8c40c2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.092+0000 7fa8ca353640 1 -- 172.21.15.161:0/3680827486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8c414a0e0 msgr2=0x7fa8c4149530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:19.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.092+0000 7fa8ca353640 1 --2- 172.21.15.161:0/3680827486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8c414a0e0 0x7fa8c4149530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.092+0000 7fa8ca353640 1 -- 172.21.15.161:0/3680827486 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c41eec30 con 0x7fa8c414a610 2024-03-28T23:35:19.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.092+0000 7fa8ca353640 1 --2- 172.21.15.161:0/3680827486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8c414a610 0x7fa8c4149a70 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fa8c40c0fc0 tx=0x7fa8bc00fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:19.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.092+0000 7fa8ba7fc640 1 -- 172.21.15.161:0/3680827486 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc019070 con 0x7fa8c414a610 2024-03-28T23:35:19.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.092+0000 7fa8ba7fc640 1 -- 172.21.15.161:0/3680827486 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8bc005030 con 0x7fa8c414a610 2024-03-28T23:35:19.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.092+0000 7fa8ba7fc640 1 -- 172.21.15.161:0/3680827486 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc005400 con 0x7fa8c414a610 2024-03-28T23:35:19.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.092+0000 7fa8cbdb6640 1 -- 172.21.15.161:0/3680827486 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8c40b0220 con 0x7fa8c414a610 2024-03-28T23:35:19.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.092+0000 7fa8cbdb6640 1 -- 172.21.15.161:0/3680827486 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8c40c2980 con 0x7fa8c414a610 2024-03-28T23:35:19.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.093+0000 7fa8cbdb6640 1 -- 172.21.15.161:0/3680827486 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8c414da90 con 0x7fa8c414a610 2024-03-28T23:35:19.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.093+0000 7fa8ba7fc640 1 -- 172.21.15.161:0/3680827486 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa8bc01c070 con 0x7fa8c414a610 2024-03-28T23:35:19.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.094+0000 7fa8ba7fc640 1 --2- 172.21.15.161:0/3680827486 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa8a806b4d0 0x7fa8a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:19.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.094+0000 7fa8ba7fc640 1 -- 172.21.15.161:0/3680827486 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 6101+0+0 (secure 0 0 0) 0x7fa8bc0890e0 con 0x7fa8c414a610 2024-03-28T23:35:19.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.094+0000 7fa8c9b52640 1 --2- 172.21.15.161:0/3680827486 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa8a806b4d0 0x7fa8a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:19.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.094+0000 7fa8c9b52640 1 --2- 172.21.15.161:0/3680827486 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa8a806b4d0 0x7fa8a806d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0034000 tx=0x7fa8c0009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:19.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.102+0000 7fa8ba7fc640 1 -- 172.21.15.161:0/3680827486 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa8c414da90 con 0x7fa8c414a610 2024-03-28T23:35:19.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.278+0000 7fa8cbdb6640 1 -- 172.21.15.161:0/3680827486 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) -- 0x7fa8c404f4b0 con 0x7fa8c414a610 2024-03-28T23:35:19.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.281+0000 7fa8ba7fc640 1 -- 172.21.15.161:0/3680827486 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v46)=0 v46) ==== 116+0+0 (secure 0 0 0) 0x7fa8bc057e60 con 0x7fa8c414a610 2024-03-28T23:35:19.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.283+0000 7fa8cbdb6640 1 -- 172.21.15.161:0/3680827486 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa8a806b4d0 msgr2=0x7fa8a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:19.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.283+0000 7fa8cbdb6640 1 --2- 172.21.15.161:0/3680827486 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa8a806b4d0 0x7fa8a806d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0034000 tx=0x7fa8c0009030 comp rx=0 tx=0).stop 2024-03-28T23:35:19.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.283+0000 7fa8cbdb6640 1 -- 172.21.15.161:0/3680827486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8c414a610 msgr2=0x7fa8c4149a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:19.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.283+0000 7fa8cbdb6640 1 --2- 172.21.15.161:0/3680827486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8c414a610 0x7fa8c4149a70 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fa8c40c0fc0 tx=0x7fa8bc00fc00 comp rx=0 tx=0).stop 2024-03-28T23:35:19.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.283+0000 7fa8cbdb6640 1 -- 172.21.15.161:0/3680827486 shutdown_connections 2024-03-28T23:35:19.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.283+0000 7fa8cbdb6640 1 --2- 172.21.15.161:0/3680827486 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa8a806b4d0 0x7fa8a806d990 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.283+0000 7fa8cbdb6640 1 --2- 172.21.15.161:0/3680827486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8c414a610 0x7fa8c4149a70 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.283+0000 7fa8cbdb6640 1 --2- 172.21.15.161:0/3680827486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8c414a0e0 0x7fa8c4149530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.283+0000 7fa8cbdb6640 1 --2- 172.21.15.161:0/3680827486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8b0003c20 0x7fa8c40c2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.283+0000 7fa8cbdb6640 1 -- 172.21.15.161:0/3680827486 >> 172.21.15.161:0/3680827486 conn(0x7fa8c40caa80 msgr2=0x7fa8c40c58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:19.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.284+0000 7fa8cbdb6640 1 -- 172.21.15.161:0/3680827486 shutdown_connections 2024-03-28T23:35:19.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.284+0000 7fa8cbdb6640 1 -- 172.21.15.161:0/3680827486 wait complete. 2024-03-28T23:35:19.295 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs 2024-03-28T23:35:19.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.400+0000 7feed636b640 1 Processor -- start 2024-03-28T23:35:19.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.400+0000 7feed636b640 1 -- start start 2024-03-28T23:35:19.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.401+0000 7feed636b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feed00ca580 0x7feed00c9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:19.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.401+0000 7feed636b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7feed0155220 0x7feed01556b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:19.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.401+0000 7feed636b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feed00c6df0 con 0x7feed0152940 2024-03-28T23:35:19.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.401+0000 7feed636b640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7feed00c70d0 con 0x7feed0155220 2024-03-28T23:35:19.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.401+0000 7feed636b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7feed00c5830 con 0x7feed00ca580 2024-03-28T23:35:19.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.401+0000 7feed4908640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7feed0155220 0x7feed01556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:19.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.401+0000 7feed4908640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7feed0155220 0x7feed01556b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47752/0 (socket says 172.21.15.161:47752) 2024-03-28T23:35:19.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.401+0000 7feed4908640 1 -- 172.21.15.161:0/1603405548 learned_addr learned my addr 172.21.15.161:0/1603405548 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:19.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.401+0000 7feed4908640 1 -- 172.21.15.161:0/1603405548 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feed00ca580 msgr2=0x7feed00c9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:19.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.401+0000 7feecf7fe640 1 --2- 172.21.15.161:0/1603405548 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feed00ca580 0x7feed00c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:19.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.401+0000 7feed4908640 1 --2- 172.21.15.161:0/1603405548 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feed00ca580 0x7feed00c9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.401+0000 7feed4908640 1 -- 172.21.15.161:0/1603405548 >> v1:172.21.15.170:6789/0 conn(0x7feed0152940 legacy=0x7feed00c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:19.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.401+0000 7feed4908640 1 -- 172.21.15.161:0/1603405548 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feed00c5b10 con 0x7feed0155220 2024-03-28T23:35:19.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.402+0000 7feed4908640 1 --2- 172.21.15.161:0/1603405548 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feed0155220 0x7feed01556b0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7feec8009e50 tx=0x7feec802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48ff21f52821814b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:19.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.402+0000 7feeceffd640 1 -- 172.21.15.161:0/1603405548 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec802c950 con 0x7feed0155220 2024-03-28T23:35:19.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.402+0000 7feeceffd640 1 -- 172.21.15.161:0/1603405548 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feec802caf0 con 0x7feed0155220 2024-03-28T23:35:19.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.402+0000 7feeceffd640 1 -- 172.21.15.161:0/1603405548 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec80343f0 con 0x7feed0155220 2024-03-28T23:35:19.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.403+0000 7feed636b640 1 -- 172.21.15.161:0/1603405548 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feed0155220 msgr2=0x7feed01556b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:19.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.403+0000 7feed636b640 1 --2- 172.21.15.161:0/1603405548 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feed0155220 0x7feed01556b0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7feec8009e50 tx=0x7feec802bdb0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.403+0000 7feed636b640 1 -- 172.21.15.161:0/1603405548 shutdown_connections 2024-03-28T23:35:19.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.403+0000 7feed636b640 1 --2- 172.21.15.161:0/1603405548 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feed0155220 0x7feed01556b0 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.403+0000 7feed636b640 1 --2- 172.21.15.161:0/1603405548 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feed00ca580 0x7feed00c9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.403+0000 7feed636b640 1 -- 172.21.15.161:0/1603405548 >> 172.21.15.161:0/1603405548 conn(0x7feed00caa00 msgr2=0x7feed00cae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:19.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.403+0000 7feed636b640 1 -- 172.21.15.161:0/1603405548 shutdown_connections 2024-03-28T23:35:19.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.403+0000 7feed636b640 1 -- 172.21.15.161:0/1603405548 wait complete. 2024-03-28T23:35:19.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.404+0000 7feed636b640 1 Processor -- start 2024-03-28T23:35:19.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.405+0000 7feed636b640 1 -- start start 2024-03-28T23:35:19.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.406+0000 7feed636b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feed00ca580 0x7feed00ae3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:19.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.406+0000 7feed636b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feed0152940 0x7feed00ab730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:19.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.406+0000 7feed636b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feed0155220 0x7feed006bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:19.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.406+0000 7feed636b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7feed0152ea0 con 0x7feed0155220 2024-03-28T23:35:19.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.406+0000 7feed636b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7feed0152d20 con 0x7feed0152940 2024-03-28T23:35:19.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.406+0000 7feed636b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7feed00c6040 con 0x7feed00ca580 2024-03-28T23:35:19.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.406+0000 7feed4908640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feed0155220 0x7feed006bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:19.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.406+0000 7feed4908640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feed0155220 0x7feed006bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47768/0 (socket says 172.21.15.161:47768) 2024-03-28T23:35:19.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.406+0000 7feed4908640 1 -- 172.21.15.161:0/321672857 learned_addr learned my addr 172.21.15.161:0/321672857 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:19.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.406+0000 7feecf7fe640 1 --2- 172.21.15.161:0/321672857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feed0152940 0x7feed00ab730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:19.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.406+0000 7feecffff640 1 --2- 172.21.15.161:0/321672857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feed00ca580 0x7feed00ae3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:19.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.406+0000 7feed4908640 1 -- 172.21.15.161:0/321672857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feed00ca580 msgr2=0x7feed00ae3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:19.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.406+0000 7feed4908640 1 --2- 172.21.15.161:0/321672857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feed00ca580 0x7feed00ae3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.406+0000 7feed4908640 1 -- 172.21.15.161:0/321672857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feed0152940 msgr2=0x7feed00ab730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:19.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.406+0000 7feed4908640 1 --2- 172.21.15.161:0/321672857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feed0152940 0x7feed00ab730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.406+0000 7feed4908640 1 -- 172.21.15.161:0/321672857 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feed006cc90 con 0x7feed0155220 2024-03-28T23:35:19.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.406+0000 7feed4908640 1 --2- 172.21.15.161:0/321672857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feed0155220 0x7feed006bf30 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7feec8009f70 tx=0x7feec8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:19.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.406+0000 7feeccff9640 1 -- 172.21.15.161:0/321672857 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec8043020 con 0x7feed0155220 2024-03-28T23:35:19.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.406+0000 7feeccff9640 1 -- 172.21.15.161:0/321672857 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feec803e450 con 0x7feed0155220 2024-03-28T23:35:19.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.407+0000 7feed636b640 1 -- 172.21.15.161:0/321672857 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feed00c4ac0 con 0x7feed0155220 2024-03-28T23:35:19.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.407+0000 7feeccff9640 1 -- 172.21.15.161:0/321672857 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec803e820 con 0x7feed0155220 2024-03-28T23:35:19.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.407+0000 7feed636b640 1 -- 172.21.15.161:0/321672857 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feed006d1e0 con 0x7feed0155220 2024-03-28T23:35:19.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.408+0000 7feed636b640 1 -- 172.21.15.161:0/321672857 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feed0068f90 con 0x7feed0155220 2024-03-28T23:35:19.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.408+0000 7feeccff9640 1 -- 172.21.15.161:0/321672857 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7feec8046020 con 0x7feed0155220 2024-03-28T23:35:19.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.408+0000 7feeccff9640 1 --2- 172.21.15.161:0/321672857 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7feeb006b4d0 0x7feeb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:19.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.409+0000 7feecffff640 1 --2- 172.21.15.161:0/321672857 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7feeb006b4d0 0x7feeb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:19.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.409+0000 7feeccff9640 1 -- 172.21.15.161:0/321672857 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 6101+0+0 (secure 0 0 0) 0x7feec80accc0 con 0x7feed0155220 2024-03-28T23:35:19.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.409+0000 7feecffff640 1 --2- 172.21.15.161:0/321672857 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7feeb006b4d0 0x7feeb006d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7feec0006fd0 tx=0x7feec0009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:19.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.413+0000 7feeccff9640 1 -- 172.21.15.161:0/321672857 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7feed0068f90 con 0x7feed0155220 2024-03-28T23:35:19.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.591+0000 7feed636b640 1 -- 172.21.15.161:0/321672857 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) -- 0x7feed004f4b0 con 0x7feed0155220 2024-03-28T23:35:19.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.594+0000 7feeccff9640 1 -- 172.21.15.161:0/321672857 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v47)=0 v47) ==== 116+0+0 (secure 0 0 0) 0x7feec804d030 con 0x7feed0155220 2024-03-28T23:35:19.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.597+0000 7feed636b640 1 -- 172.21.15.161:0/321672857 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7feeb006b4d0 msgr2=0x7feeb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:19.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.597+0000 7feed636b640 1 --2- 172.21.15.161:0/321672857 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7feeb006b4d0 0x7feeb006d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7feec0006fd0 tx=0x7feec0009060 comp rx=0 tx=0).stop 2024-03-28T23:35:19.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.597+0000 7feed636b640 1 -- 172.21.15.161:0/321672857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feed0155220 msgr2=0x7feed006bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:19.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.597+0000 7feed636b640 1 --2- 172.21.15.161:0/321672857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feed0155220 0x7feed006bf30 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7feec8009f70 tx=0x7feec8004a40 comp rx=0 tx=0).stop 2024-03-28T23:35:19.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.597+0000 7feed636b640 1 -- 172.21.15.161:0/321672857 shutdown_connections 2024-03-28T23:35:19.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.597+0000 7feed636b640 1 --2- 172.21.15.161:0/321672857 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7feeb006b4d0 0x7feeb006d990 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.597+0000 7feed636b640 1 --2- 172.21.15.161:0/321672857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feed0155220 0x7feed006bf30 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.597+0000 7feed636b640 1 --2- 172.21.15.161:0/321672857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feed0152940 0x7feed00ab730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.597+0000 7feed636b640 1 --2- 172.21.15.161:0/321672857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feed00ca580 0x7feed00ae3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.597+0000 7feed636b640 1 -- 172.21.15.161:0/321672857 >> 172.21.15.161:0/321672857 conn(0x7feed00caa00 msgr2=0x7feed0154a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:19.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.597+0000 7feed636b640 1 -- 172.21.15.161:0/321672857 shutdown_connections 2024-03-28T23:35:19.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.597+0000 7feed636b640 1 -- 172.21.15.161:0/321672857 wait complete. 2024-03-28T23:35:19.612 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:35:19.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.717+0000 7fd733581640 1 Processor -- start 2024-03-28T23:35:19.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.717+0000 7fd733581640 1 -- start start 2024-03-28T23:35:19.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.718+0000 7fd733581640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd72c152890 0x7fd72c0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:19.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.718+0000 7fd733581640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd72c0c8ed0 0x7fd72c0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:19.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.718+0000 7fd733581640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd72c0c5950 0x7fd72c1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:19.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.718+0000 7fd733581640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd72c0c8480 con 0x7fd72c152890 2024-03-28T23:35:19.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.718+0000 7fd733581640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd72c159640 con 0x7fd72c0c5950 2024-03-28T23:35:19.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.718+0000 7fd733581640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd72c158a30 con 0x7fd72c0c8ed0 2024-03-28T23:35:19.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.718+0000 7fd73131d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd72c152890 0x7fd72c0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:19.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.718+0000 7fd73131d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd72c152890 0x7fd72c0c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47774/0 (socket says 172.21.15.161:47774) 2024-03-28T23:35:19.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.718+0000 7fd73131d640 1 -- 172.21.15.161:0/948414846 learned_addr learned my addr 172.21.15.161:0/948414846 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:19.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.719+0000 7fd73131d640 1 -- 172.21.15.161:0/948414846 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd72c0c8ed0 msgr2=0x7fd72c0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:19.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.719+0000 7fd730b1c640 1 --2- 172.21.15.161:0/948414846 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd72c0c8ed0 0x7fd72c0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:19.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.719+0000 7fd731b1e640 1 --2- 172.21.15.161:0/948414846 >> v2:172.21.15.170:3300/0 conn(0x7fd72c0c5950 0x7fd72c1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:19.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.719+0000 7fd73131d640 1 --2- 172.21.15.161:0/948414846 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd72c0c8ed0 0x7fd72c0c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.719+0000 7fd73131d640 1 -- 172.21.15.161:0/948414846 >> v2:172.21.15.170:3300/0 conn(0x7fd72c0c5950 msgr2=0x7fd72c1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:19.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.719+0000 7fd73131d640 1 --2- 172.21.15.161:0/948414846 >> v2:172.21.15.170:3300/0 conn(0x7fd72c0c5950 0x7fd72c1553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.719+0000 7fd73131d640 1 -- 172.21.15.161:0/948414846 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72c158d10 con 0x7fd72c152890 2024-03-28T23:35:19.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.720+0000 7fd73131d640 1 --2- 172.21.15.161:0/948414846 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd72c152890 0x7fd72c0c8990 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fd72c0b0c00 tx=0x7fd72802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc38f8604f949fc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:19.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.720+0000 7fd723fff640 1 -- 172.21.15.161:0/948414846 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd728038070 con 0x7fd72c152890 2024-03-28T23:35:19.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.720+0000 7fd723fff640 1 -- 172.21.15.161:0/948414846 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd72802f050 con 0x7fd72c152890 2024-03-28T23:35:19.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.720+0000 7fd723fff640 1 -- 172.21.15.161:0/948414846 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd728031970 con 0x7fd72c152890 2024-03-28T23:35:19.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.721+0000 7fd733581640 1 -- 172.21.15.161:0/948414846 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd72c152890 msgr2=0x7fd72c0c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:19.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.721+0000 7fd733581640 1 --2- 172.21.15.161:0/948414846 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd72c152890 0x7fd72c0c8990 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fd72c0b0c00 tx=0x7fd72802b7f0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.721+0000 7fd733581640 1 -- 172.21.15.161:0/948414846 shutdown_connections 2024-03-28T23:35:19.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.721+0000 7fd733581640 1 --2- 172.21.15.161:0/948414846 >> v2:172.21.15.170:3300/0 conn(0x7fd72c0c5950 0x7fd72c1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.721+0000 7fd733581640 1 --2- 172.21.15.161:0/948414846 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd72c0c8ed0 0x7fd72c0c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.721+0000 7fd733581640 1 --2- 172.21.15.161:0/948414846 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd72c152890 0x7fd72c0c8990 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.721+0000 7fd733581640 1 -- 172.21.15.161:0/948414846 >> 172.21.15.161:0/948414846 conn(0x7fd72c0ac620 msgr2=0x7fd72c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:19.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.721+0000 7fd733581640 1 -- 172.21.15.161:0/948414846 shutdown_connections 2024-03-28T23:35:19.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.721+0000 7fd733581640 1 -- 172.21.15.161:0/948414846 wait complete. 2024-03-28T23:35:19.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.722+0000 7fd733581640 1 Processor -- start 2024-03-28T23:35:19.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.722+0000 7fd733581640 1 -- start start 2024-03-28T23:35:19.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.723+0000 7fd733581640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd72c0c5950 0x7fd72c1475d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:19.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.723+0000 7fd733581640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd72c0c8ed0 0x7fd72c154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:19.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.723+0000 7fd733581640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd72c152890 0x7fd72c155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:19.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.723+0000 7fd733581640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c152c70 con 0x7fd72c0c5950 2024-03-28T23:35:19.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.723+0000 7fd733581640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c1592d0 con 0x7fd72c152890 2024-03-28T23:35:19.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.723+0000 7fd733581640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd72c152df0 con 0x7fd72c0c8ed0 2024-03-28T23:35:19.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.723+0000 7fd73131d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd72c0c5950 0x7fd72c1475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:19.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.723+0000 7fd73131d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd72c0c5950 0x7fd72c1475d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47776/0 (socket says 172.21.15.161:47776) 2024-03-28T23:35:19.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.723+0000 7fd73131d640 1 -- 172.21.15.161:0/3142871602 learned_addr learned my addr 172.21.15.161:0/3142871602 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:19.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.723+0000 7fd731b1e640 1 --2- 172.21.15.161:0/3142871602 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd72c152890 0x7fd72c155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:19.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.723+0000 7fd730b1c640 1 --2- 172.21.15.161:0/3142871602 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd72c0c8ed0 0x7fd72c154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:19.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.723+0000 7fd73131d640 1 -- 172.21.15.161:0/3142871602 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd72c0c8ed0 msgr2=0x7fd72c154ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:19.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.723+0000 7fd73131d640 1 --2- 172.21.15.161:0/3142871602 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd72c0c8ed0 0x7fd72c154ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.723+0000 7fd73131d640 1 -- 172.21.15.161:0/3142871602 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd72c152890 msgr2=0x7fd72c155000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:19.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.723+0000 7fd73131d640 1 --2- 172.21.15.161:0/3142871602 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd72c152890 0x7fd72c155000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.723+0000 7fd73131d640 1 -- 172.21.15.161:0/3142871602 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72c1571a0 con 0x7fd72c0c5950 2024-03-28T23:35:19.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.723+0000 7fd730b1c640 1 --2- 172.21.15.161:0/3142871602 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd72c0c8ed0 0x7fd72c154ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:19.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.723+0000 7fd731b1e640 1 --2- 172.21.15.161:0/3142871602 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd72c152890 0x7fd72c155000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:19.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.723+0000 7fd73131d640 1 --2- 172.21.15.161:0/3142871602 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd72c0c5950 0x7fd72c1475d0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fd72802b830 tx=0x7fd7280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:19.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.723+0000 7fd721ffb640 1 -- 172.21.15.161:0/3142871602 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd728042070 con 0x7fd72c0c5950 2024-03-28T23:35:19.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.723+0000 7fd721ffb640 1 -- 172.21.15.161:0/3142871602 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd72803d870 con 0x7fd72c0c5950 2024-03-28T23:35:19.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.723+0000 7fd733581640 1 -- 172.21.15.161:0/3142871602 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd72c157480 con 0x7fd72c0c5950 2024-03-28T23:35:19.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.723+0000 7fd721ffb640 1 -- 172.21.15.161:0/3142871602 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd728038040 con 0x7fd72c0c5950 2024-03-28T23:35:19.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.724+0000 7fd733581640 1 -- 172.21.15.161:0/3142871602 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd72c153fe0 con 0x7fd72c0c5950 2024-03-28T23:35:19.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.724+0000 7fd733581640 1 -- 172.21.15.161:0/3142871602 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd72c04f4b0 con 0x7fd72c0c5950 2024-03-28T23:35:19.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.725+0000 7fd721ffb640 1 -- 172.21.15.161:0/3142871602 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd728045070 con 0x7fd72c0c5950 2024-03-28T23:35:19.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.725+0000 7fd721ffb640 1 --2- 172.21.15.161:0/3142871602 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd71006b4d0 0x7fd71006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:19.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.725+0000 7fd721ffb640 1 -- 172.21.15.161:0/3142871602 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 6101+0+0 (secure 0 0 0) 0x7fd7280ac340 con 0x7fd72c0c5950 2024-03-28T23:35:19.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.725+0000 7fd730b1c640 1 --2- 172.21.15.161:0/3142871602 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd71006b4d0 0x7fd71006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:19.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.726+0000 7fd730b1c640 1 --2- 172.21.15.161:0/3142871602 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd71006b4d0 0x7fd71006d990 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fd71c006fd0 tx=0x7fd71c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:19.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.730+0000 7fd721ffb640 1 -- 172.21.15.161:0/3142871602 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd72807b1e0 con 0x7fd72c0c5950 2024-03-28T23:35:19.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.944+0000 7fd733581640 1 -- 172.21.15.161:0/3142871602 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd72c156570 con 0x7fd72c0c5950 2024-03-28T23:35:19.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.945+0000 7fd721ffb640 1 -- 172.21.15.161:0/3142871602 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) ==== 95+0+5203 (secure 0 0 0) 0x7fd72807aae0 con 0x7fd72c0c5950 2024-03-28T23:35:19.945 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:19.946 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":124,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6740,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3586875977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3586875977},{"type":"v1","addr":"172.21.15.170:6835","nonce":3586875977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120},{"gid":7639,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/3182285172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3182285172},{"type":"v1","addr":"172.21.15.170:6834","nonce":3182285172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120},{"gid":8409,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/3037890643","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3037890643},{"type":"v1","addr":"172.21.15.161:6837","nonce":3037890643}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:12.982566+0000","modified":"2024-03-28T23:35:15.337512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8412},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8412":{"gid":8412,"name":"a","rank":0,"incarnation":121,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6836/2527502405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":2527502405},{"type":"v1","addr":"172.21.15.161:6836","nonce":2527502405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8412,"qdb_cluster":[8412]},"id":10}]} 2024-03-28T23:35:19.946 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 124 2024-03-28T23:35:19.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.947+0000 7fd733581640 1 -- 172.21.15.161:0/3142871602 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd71006b4d0 msgr2=0x7fd71006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:19.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.947+0000 7fd733581640 1 --2- 172.21.15.161:0/3142871602 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd71006b4d0 0x7fd71006d990 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fd71c006fd0 tx=0x7fd71c008040 comp rx=0 tx=0).stop 2024-03-28T23:35:19.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.947+0000 7fd733581640 1 -- 172.21.15.161:0/3142871602 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd72c0c5950 msgr2=0x7fd72c1475d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:19.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.947+0000 7fd733581640 1 --2- 172.21.15.161:0/3142871602 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd72c0c5950 0x7fd72c1475d0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fd72802b830 tx=0x7fd7280046c0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.947+0000 7fd733581640 1 -- 172.21.15.161:0/3142871602 shutdown_connections 2024-03-28T23:35:19.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.947+0000 7fd733581640 1 --2- 172.21.15.161:0/3142871602 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd71006b4d0 0x7fd71006d990 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.947+0000 7fd733581640 1 --2- 172.21.15.161:0/3142871602 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd72c152890 0x7fd72c155000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.947+0000 7fd733581640 1 --2- 172.21.15.161:0/3142871602 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd72c0c8ed0 0x7fd72c154ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.947+0000 7fd733581640 1 --2- 172.21.15.161:0/3142871602 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd72c0c5950 0x7fd72c1475d0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:19.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.947+0000 7fd733581640 1 -- 172.21.15.161:0/3142871602 >> 172.21.15.161:0/3142871602 conn(0x7fd72c0ac620 msgr2=0x7fd72c0c8660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:19.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.947+0000 7fd733581640 1 -- 172.21.15.161:0/3142871602 shutdown_connections 2024-03-28T23:35:19.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:19.947+0000 7fd733581640 1 -- 172.21.15.161:0/3142871602 wait complete. 2024-03-28T23:35:20.362 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 124, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:35:12.982566+0000', 'modified': '2024-03-28T23:35:15.337512+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8412}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8412': {'gid': 8412, 'name': 'a', 'rank': 0, 'incarnation': 121, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.161:6836/2527502405', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 2527502405}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 2527502405}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8412, 'qdb_cluster': [8412]} 2024-03-28T23:35:20.362 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:35:20.362 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-28T23:35:20.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.466+0000 7f9df9ffd640 1 Processor -- start 2024-03-28T23:35:20.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.467+0000 7f9df9ffd640 1 -- start start 2024-03-28T23:35:20.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.467+0000 7f9df9ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9df4152890 0x7f9df40c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:20.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.467+0000 7f9df9ffd640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9df4158a30 con 0x7f9df40c5b20 2024-03-28T23:35:20.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.467+0000 7f9df9ffd640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9df4158d10 con 0x7f9df40c8ed0 2024-03-28T23:35:20.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.467+0000 7f9df9ffd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9df40c8480 con 0x7f9df4152890 2024-03-28T23:35:20.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.467+0000 7f9df3fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f9df40c5b20 0x7f9df41553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45314/0 (socket says 172.21.15.161:45314) 2024-03-28T23:35:20.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.467+0000 7f9df3fff640 1 -- 172.21.15.161:0/847792107 learned_addr learned my addr 172.21.15.161:0/847792107 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:20.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.468+0000 7f9df27fc640 1 -- 172.21.15.161:0/847792107 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3538219768 0 0) 0x7f9df4158a30 con 0x7f9df40c5b20 2024-03-28T23:35:20.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.468+0000 7f9df37fe640 1 --2- 172.21.15.161:0/847792107 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9df4152890 0x7f9df40c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.468+0000 7f9df27fc640 1 -- 172.21.15.161:0/847792107 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ddc003520 con 0x7f9df40c5b20 2024-03-28T23:35:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.468+0000 7f9df27fc640 1 -- 172.21.15.161:0/847792107 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 435050859 0 0) 0x7f9ddc003520 con 0x7f9df40c5b20 2024-03-28T23:35:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.468+0000 7f9df27fc640 1 -- 172.21.15.161:0/847792107 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9df4152890 msgr2=0x7f9df40c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.468+0000 7f9df27fc640 1 --2- 172.21.15.161:0/847792107 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9df4152890 0x7f9df40c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.468+0000 7f9df27fc640 1 -- 172.21.15.161:0/847792107 >> v1:172.21.15.170:6789/0 conn(0x7f9df40c8ed0 legacy=0x7f9df40c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.468+0000 7f9df27fc640 1 -- 172.21.15.161:0/847792107 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df4158ff0 con 0x7f9df40c5b20 2024-03-28T23:35:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.468+0000 7f9df27fc640 1 -- 172.21.15.161:0/847792107 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f9dec0019f0 con 0x7f9df40c5b20 2024-03-28T23:35:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.468+0000 7f9df27fc640 1 -- 172.21.15.161:0/847792107 >> v1:172.21.15.161:6789/0 conn(0x7f9df40c5b20 legacy=0x7f9df41553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.468+0000 7f9df27fc640 1 --2- 172.21.15.161:0/847792107 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9df4152890 0x7f9ddc003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.468+0000 7f9df27fc640 1 --2- 172.21.15.161:0/847792107 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ddc004090 0x7f9ddc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.468+0000 7f9df27fc640 1 --2- 172.21.15.161:0/847792107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ddc004e60 0x7f9ddc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.468+0000 7f9df27fc640 1 -- 172.21.15.161:0/847792107 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9ddc005650 con 0x7f9ddc004090 2024-03-28T23:35:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.468+0000 7f9df27fc640 1 -- 172.21.15.161:0/847792107 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9ddc00bf00 con 0x7f9ddc004e60 2024-03-28T23:35:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.468+0000 7f9df27fc640 1 -- 172.21.15.161:0/847792107 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9ddc00c0b0 con 0x7f9df4152890 2024-03-28T23:35:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.468+0000 7f9df2ffd640 1 --2- 172.21.15.161:0/847792107 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ddc004090 0x7f9ddc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.468+0000 7f9df37fe640 1 --2- 172.21.15.161:0/847792107 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9df4152890 0x7f9ddc003ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.468+0000 7f9df3fff640 1 --2- 172.21.15.161:0/847792107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ddc004e60 0x7f9ddc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.468+0000 7f9df2ffd640 1 -- 172.21.15.161:0/847792107 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9df4152890 msgr2=0x7f9ddc003ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.468+0000 7f9df2ffd640 1 --2- 172.21.15.161:0/847792107 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9df4152890 0x7f9ddc003ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.468+0000 7f9df2ffd640 1 -- 172.21.15.161:0/847792107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ddc004e60 msgr2=0x7f9ddc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.468+0000 7f9df2ffd640 1 --2- 172.21.15.161:0/847792107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ddc004e60 0x7f9ddc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.468+0000 7f9df2ffd640 1 -- 172.21.15.161:0/847792107 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ddc003520 con 0x7f9ddc004090 2024-03-28T23:35:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.469+0000 7f9df2ffd640 1 --2- 172.21.15.161:0/847792107 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ddc004090 0x7f9ddc004800 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f9de4001770 tx=0x7f9ddc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be4e767d2953492d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.469+0000 7f9df27fc640 1 -- 172.21.15.161:0/847792107 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de40013c0 con 0x7f9ddc004090 2024-03-28T23:35:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.469+0000 7f9df27fc640 1 -- 172.21.15.161:0/847792107 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9de40052d0 con 0x7f9ddc004090 2024-03-28T23:35:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.469+0000 7f9df27fc640 1 -- 172.21.15.161:0/847792107 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de4003640 con 0x7f9ddc004090 2024-03-28T23:35:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.470+0000 7f9df9ffd640 1 -- 172.21.15.161:0/847792107 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ddc004090 msgr2=0x7f9ddc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.470+0000 7f9df9ffd640 1 --2- 172.21.15.161:0/847792107 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ddc004090 0x7f9ddc004800 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f9de4001770 tx=0x7f9ddc005830 comp rx=0 tx=0).stop 2024-03-28T23:35:20.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.470+0000 7f9df3fff640 1 -- 172.21.15.161:0/847792107 reap_dead start 2024-03-28T23:35:20.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.470+0000 7f9df9ffd640 1 -- 172.21.15.161:0/847792107 shutdown_connections 2024-03-28T23:35:20.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.470+0000 7f9df9ffd640 1 -- 172.21.15.161:0/847792107 >> 172.21.15.161:0/847792107 conn(0x7f9df40ac620 msgr2=0x7f9df40aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:20.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.470+0000 7f9df9ffd640 1 -- 172.21.15.161:0/847792107 shutdown_connections 2024-03-28T23:35:20.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.470+0000 7f9df9ffd640 1 -- 172.21.15.161:0/847792107 wait complete. 2024-03-28T23:35:20.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.471+0000 7f9df9ffd640 1 Processor -- start 2024-03-28T23:35:20.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.471+0000 7f9df9ffd640 1 -- start start 2024-03-28T23:35:20.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.472+0000 7f9df9ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ddc004090 0x7f9df414d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:20.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.472+0000 7f9df9ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9df414cde0 0x7f9df40bde90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:20.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.472+0000 7f9df9ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9df40be3d0 0x7f9df40aa430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:20.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.472+0000 7f9df9ffd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9df4152c70 con 0x7f9ddc004090 2024-03-28T23:35:20.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.472+0000 7f9df9ffd640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9df4159640 con 0x7f9df414cde0 2024-03-28T23:35:20.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.472+0000 7f9df9ffd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9df4152df0 con 0x7f9df40be3d0 2024-03-28T23:35:20.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.472+0000 7f9df37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ddc004090 0x7f9df414d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:20.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.472+0000 7f9df37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ddc004090 0x7f9df414d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47788/0 (socket says 172.21.15.161:47788) 2024-03-28T23:35:20.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.472+0000 7f9df37fe640 1 -- 172.21.15.161:0/3405193049 learned_addr learned my addr 172.21.15.161:0/3405193049 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:20.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.472+0000 7f9df3fff640 1 --2- 172.21.15.161:0/3405193049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9df40be3d0 0x7f9df40aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:20.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.472+0000 7f9df2ffd640 1 --2- 172.21.15.161:0/3405193049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9df414cde0 0x7f9df40bde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:20.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.472+0000 7f9df37fe640 1 -- 172.21.15.161:0/3405193049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9df40be3d0 msgr2=0x7f9df40aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:20.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.472+0000 7f9df37fe640 1 --2- 172.21.15.161:0/3405193049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9df40be3d0 0x7f9df40aa430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.472+0000 7f9df37fe640 1 -- 172.21.15.161:0/3405193049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9df414cde0 msgr2=0x7f9df40bde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:20.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.472+0000 7f9df37fe640 1 --2- 172.21.15.161:0/3405193049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9df414cde0 0x7f9df40bde90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.472+0000 7f9df37fe640 1 -- 172.21.15.161:0/3405193049 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df40be850 con 0x7f9ddc004090 2024-03-28T23:35:20.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.473+0000 7f9df37fe640 1 --2- 172.21.15.161:0/3405193049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ddc004090 0x7f9df414d9d0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f9de80119a0 tx=0x7f9de8011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:20.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.473+0000 7f9ddbfff640 1 -- 172.21.15.161:0/3405193049 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de800faf0 con 0x7f9ddc004090 2024-03-28T23:35:20.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.473+0000 7f9ddbfff640 1 -- 172.21.15.161:0/3405193049 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9de800fc90 con 0x7f9ddc004090 2024-03-28T23:35:20.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.473+0000 7f9ddbfff640 1 -- 172.21.15.161:0/3405193049 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de8014440 con 0x7f9ddc004090 2024-03-28T23:35:20.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.473+0000 7f9df9ffd640 1 -- 172.21.15.161:0/3405193049 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9df414fae0 con 0x7f9ddc004090 2024-03-28T23:35:20.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.473+0000 7f9df9ffd640 1 -- 172.21.15.161:0/3405193049 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9df414f800 con 0x7f9ddc004090 2024-03-28T23:35:20.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.474+0000 7f9ddbfff640 1 -- 172.21.15.161:0/3405193049 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9de8012040 con 0x7f9ddc004090 2024-03-28T23:35:20.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.475+0000 7f9ddbfff640 1 --2- 172.21.15.161:0/3405193049 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9de006b500 0x7f9de006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:20.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.475+0000 7f9ddbfff640 1 -- 172.21.15.161:0/3405193049 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 6101+0+0 (secure 0 0 0) 0x7f9de808a9e0 con 0x7f9ddc004090 2024-03-28T23:35:20.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.475+0000 7f9df2ffd640 1 --2- 172.21.15.161:0/3405193049 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9de006b500 0x7f9de006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:20.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.475+0000 7f9df2ffd640 1 --2- 172.21.15.161:0/3405193049 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9de006b500 0x7f9de006d9c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f9de4001770 tx=0x7f9de4009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:20.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.476+0000 7f9dfcbda740 1 Processor -- start 2024-03-28T23:35:20.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.476+0000 7f9dfcbda740 1 -- start start 2024-03-28T23:35:20.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.477+0000 7f9dfcbda740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557c05345db0 0x557c05343890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:20.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.477+0000 7f9dfcbda740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557c05346190 0x557c05343dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:20.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.477+0000 7f9dfcbda740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05344310 0x557c053404e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:20.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.477+0000 7f9dfcbda740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x557c052a9af0 con 0x557c05344310 2024-03-28T23:35:20.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.477+0000 7f9dfcbda740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x557c052529d0 con 0x557c05345db0 2024-03-28T23:35:20.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.477+0000 7f9dfcbda740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x557c052a5b90 con 0x557c05346190 2024-03-28T23:35:20.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.477+0000 7f9df3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05344310 0x557c053404e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:20.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.477+0000 7f9df3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05344310 0x557c053404e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47798/0 (socket says 172.21.15.161:47798) 2024-03-28T23:35:20.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.477+0000 7f9df3fff640 1 -- 172.21.15.161:0/3685048915 learned_addr learned my addr 172.21.15.161:0/3685048915 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:20.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.477+0000 7f9df37fe640 1 --2- 172.21.15.161:0/3685048915 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557c05345db0 0x557c05343890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:20.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.477+0000 7f9df2ffd640 1 --2- 172.21.15.161:0/3685048915 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557c05346190 0x557c05343dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:20.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.477+0000 7f9df3fff640 1 -- 172.21.15.161:0/3685048915 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557c05346190 msgr2=0x557c05343dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:20.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.477+0000 7f9df3fff640 1 --2- 172.21.15.161:0/3685048915 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557c05346190 0x557c05343dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.477+0000 7f9df3fff640 1 -- 172.21.15.161:0/3685048915 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557c05345db0 msgr2=0x557c05343890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:20.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.477+0000 7f9df3fff640 1 --2- 172.21.15.161:0/3685048915 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557c05345db0 0x557c05343890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.477+0000 7f9df3fff640 1 -- 172.21.15.161:0/3685048915 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c04fbe2d0 con 0x557c05344310 2024-03-28T23:35:20.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.478+0000 7f9df3fff640 1 --2- 172.21.15.161:0/3685048915 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05344310 0x557c053404e0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f9df40c5f00 tx=0x7f9dec004f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=567f4f4d2662f43a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:20.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.478+0000 7f9dd97fa640 1 -- 172.21.15.161:0/3685048915 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec007590 con 0x557c05344310 2024-03-28T23:35:20.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.478+0000 7f9dd97fa640 1 -- 172.21.15.161:0/3685048915 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dec019070 con 0x557c05344310 2024-03-28T23:35:20.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.478+0000 7f9dd97fa640 1 -- 172.21.15.161:0/3685048915 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec007970 con 0x557c05344310 2024-03-28T23:35:20.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.479+0000 7f9dfcbda740 1 -- 172.21.15.161:0/3685048915 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05344310 msgr2=0x557c053404e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:20.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.479+0000 7f9dfcbda740 1 --2- 172.21.15.161:0/3685048915 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05344310 0x557c053404e0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f9df40c5f00 tx=0x7f9dec004f30 comp rx=0 tx=0).stop 2024-03-28T23:35:20.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.479+0000 7f9dfcbda740 1 -- 172.21.15.161:0/3685048915 shutdown_connections 2024-03-28T23:35:20.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.479+0000 7f9dfcbda740 1 --2- 172.21.15.161:0/3685048915 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05344310 0x557c053404e0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.479+0000 7f9dfcbda740 1 --2- 172.21.15.161:0/3685048915 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557c05346190 0x557c05343dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.479+0000 7f9dfcbda740 1 --2- 172.21.15.161:0/3685048915 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557c05345db0 0x557c05343890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.479+0000 7f9dfcbda740 1 -- 172.21.15.161:0/3685048915 >> 172.21.15.161:0/3685048915 conn(0x557c05342630 msgr2=0x557c05367220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.479+0000 7f9dfcbda740 1 -- 172.21.15.161:0/3685048915 shutdown_connections 2024-03-28T23:35:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.479+0000 7f9dfcbda740 1 -- 172.21.15.161:0/3685048915 wait complete. 2024-03-28T23:35:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.479+0000 7f9dfcbda740 1 -- start start 2024-03-28T23:35:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.479+0000 7f9dfcbda740 1 Processor -- start 2024-03-28T23:35:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.480+0000 7f9dfcbda740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05345db0 0x557c0533f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.480+0000 7f9dfcbda740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557c05346190 0x557c05392630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.480+0000 7f9dfcbda740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557c0536c2a0 0x557c05362d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.480+0000 7f9dfcbda740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x557c05368090 con 0x557c05345db0 2024-03-28T23:35:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.480+0000 7f9dfcbda740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x557c05265f70 con 0x557c05346190 2024-03-28T23:35:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.480+0000 7f9dfcbda740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x557c05367ad0 con 0x557c0536c2a0 2024-03-28T23:35:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.480+0000 7f9dfcbda740 10 client.0 _ll_register_callbacks cb 0x557c04fbdac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:35:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.480+0000 7f9df37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05345db0 0x557c0533f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.480+0000 7f9df37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05345db0 0x557c0533f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:47806/0 (socket says 172.21.15.161:47806) 2024-03-28T23:35:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.480+0000 7f9df37fe640 1 -- 172.21.15.161:0/454884214 learned_addr learned my addr 172.21.15.161:0/454884214 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.480+0000 7f9df3fff640 1 --2- 172.21.15.161:0/454884214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557c0536c2a0 0x557c05362d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.480+0000 7f9df2ffd640 1 --2- 172.21.15.161:0/454884214 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557c05346190 0x557c05392630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.480+0000 7f9df37fe640 1 -- 172.21.15.161:0/454884214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557c0536c2a0 msgr2=0x557c05362d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.480+0000 7f9df37fe640 1 --2- 172.21.15.161:0/454884214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557c0536c2a0 0x557c05362d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.480+0000 7f9df37fe640 1 -- 172.21.15.161:0/454884214 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557c05346190 msgr2=0x557c05392630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.480+0000 7f9df37fe640 1 --2- 172.21.15.161:0/454884214 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557c05346190 0x557c05392630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.480+0000 7f9df37fe640 1 -- 172.21.15.161:0/454884214 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c050886b0 con 0x557c05345db0 2024-03-28T23:35:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.480+0000 7f9df3fff640 1 --2- 172.21.15.161:0/454884214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557c0536c2a0 0x557c05362d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.480+0000 7f9df2ffd640 1 --2- 172.21.15.161:0/454884214 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557c05346190 0x557c05392630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.481+0000 7f9df37fe640 1 --2- 172.21.15.161:0/454884214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05345db0 0x557c0533f6d0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f9de800f610 tx=0x7f9de805d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8044d6f24de64711 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.481+0000 7f9dd27fc640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:35:20.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.481+0000 7f9dd27fc640 1 -- 172.21.15.161:0/454884214 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de8086030 con 0x557c05345db0 2024-03-28T23:35:20.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.481+0000 7f9dd27fc640 1 -- 172.21.15.161:0/454884214 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 6101+0+0 (secure 0 0 0) 0x7f9de80573e0 con 0x557c05345db0 2024-03-28T23:35:20.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.481+0000 7f9dfcbda740 1 -- 172.21.15.161:0/454884214 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9dec019070 con 0x557c05345db0 2024-03-28T23:35:20.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.481+0000 7f9dd27fc640 1 -- 172.21.15.161:0/454884214 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=124) ==== 24+0+0 (secure 0 0 0) 0x7f9de801c570 con 0x557c05345db0 2024-03-28T23:35:20.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.481+0000 7f9dfcbda740 10 client.8607 fetch_fsmap learned FSMap version 124 2024-03-28T23:35:20.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.481+0000 7f9dfcbda740 1 -- 172.21.15.161:0/454884214 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=124,monmap=2+}) -- 0x557c05392150 con 0x557c05345db0 2024-03-28T23:35:20.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.481+0000 7f9dd27fc640 1 -- 172.21.15.161:0/454884214 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dec019070 con 0x557c05345db0 2024-03-28T23:35:20.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.481+0000 7f9dd27fc640 1 -- 172.21.15.161:0/454884214 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 124) ==== 3494+0+0 (secure 0 0 0) 0x7f9de8025030 con 0x557c05345db0 2024-03-28T23:35:20.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.481+0000 7f9dfcbda740 10 client.8607 fetch_fsmap finished waiting for FSMap version 124 2024-03-28T23:35:20.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.481+0000 7f9dfcbda740 10 client.8607 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T23:35:20.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.481+0000 7f9dfcbda740 20 client.8607 populate_metadata read hostname 'smithi161' 2024-03-28T23:35:20.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.481+0000 7f9dfcbda740 1 --2- 172.21.15.161:0/454884214 >> [v2:172.21.15.161:6834/2527502405,v1:172.21.15.161:6836/2527502405] conn(0x557c05385e80 0x557c053a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:20.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.481+0000 7f9dfcbda740 4 client.8607 mds_command: new command op to 8412 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:35:20.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.481+0000 7f9dfcbda740 1 -- 172.21.15.161:0/454884214 --> [v2:172.21.15.161:6834/2527502405,v1:172.21.15.161:6836/2527502405] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c052b7030 con 0x557c05385e80 2024-03-28T23:35:20.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.481+0000 7f9df2ffd640 1 --2- 172.21.15.161:0/454884214 >> [v2:172.21.15.161:6834/2527502405,v1:172.21.15.161:6836/2527502405] conn(0x557c05385e80 0x557c053a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:20.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.482+0000 7f9dd27fc640 10 client.8607 ms_handle_connect on v2:172.21.15.161:6834/2527502405 2024-03-28T23:35:20.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.482+0000 7f9df2ffd640 1 --2- 172.21.15.161:0/454884214 >> [v2:172.21.15.161:6834/2527502405,v1:172.21.15.161:6836/2527502405] conn(0x557c05385e80 0x557c053a0bf0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4066ddcec13e11d9 server_cookie=23d464888ad79102 in_seq=0 out_seq=0 2024-03-28T23:35:20.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.482+0000 7f9dd27fc640 1 -- 172.21.15.161:0/454884214 <== mds.0 v2:172.21.15.161:6834/2527502405 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557c052b7030 con 0x557c05385e80 2024-03-28T23:35:20.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.482+0000 7f9dd27fc640 10 client.8607 handle_command_reply: tid=0 2024-03-28T23:35:20.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.482+0000 7f9dfcbda740 1 client.8607 shutdown 2024-03-28T23:35:20.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.482+0000 7f9dfcbda740 1 -- 172.21.15.161:0/454884214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05345db0 msgr2=0x557c0533f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:20.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.482+0000 7f9dfcbda740 1 --2- 172.21.15.161:0/454884214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05345db0 0x557c0533f6d0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f9de800f610 tx=0x7f9de805d2e0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.482+0000 7f9dfcbda740 1 -- 172.21.15.161:0/454884214 shutdown_connections 2024-03-28T23:35:20.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.482+0000 7f9dfcbda740 1 --2- 172.21.15.161:0/454884214 >> [v2:172.21.15.161:6834/2527502405,v1:172.21.15.161:6836/2527502405] conn(0x557c05385e80 0x557c053a0bf0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.482+0000 7f9dfcbda740 1 --2- 172.21.15.161:0/454884214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557c0536c2a0 0x557c05362d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.482+0000 7f9dfcbda740 1 --2- 172.21.15.161:0/454884214 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557c05346190 0x557c05392630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.482+0000 7f9dd27fc640 0 client.8607 ms_handle_reset on v2:172.21.15.161:6834/2527502405 2024-03-28T23:35:20.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.482+0000 7f9dfcbda740 1 --2- 172.21.15.161:0/454884214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05345db0 0x557c0533f6d0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.482+0000 7f9dfcbda740 1 -- 172.21.15.161:0/454884214 >> 172.21.15.161:0/454884214 conn(0x557c05344310 msgr2=0x557c05340d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:20.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.483+0000 7f9dfcbda740 1 -- 172.21.15.161:0/454884214 shutdown_connections 2024-03-28T23:35:20.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.483+0000 7f9dfcbda740 1 -- 172.21.15.161:0/454884214 wait complete. 2024-03-28T23:35:20.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.483+0000 7f9dfcbda740 20 client.8607 trim_cache size 0 max 16384 2024-03-28T23:35:20.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.501+0000 7f9dfcbda740 1 Processor -- start 2024-03-28T23:35:20.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.501+0000 7f9dfcbda740 1 -- start start 2024-03-28T23:35:20.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.501+0000 7f9dfcbda740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557c05345db0 0x557c053629c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:20.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.501+0000 7f9dfcbda740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05346190 0x557c05362f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:20.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.501+0000 7f9dfcbda740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557c0536c2a0 0x557c0533fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:20.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.502+0000 7f9dfcbda740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9df41f7560 con 0x557c05346190 2024-03-28T23:35:20.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.502+0000 7f9dfcbda740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9df4157710 con 0x557c0536c2a0 2024-03-28T23:35:20.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.502+0000 7f9dfcbda740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f9df41f6f10 con 0x557c05345db0 2024-03-28T23:35:20.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.502+0000 7f9df2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05346190 0x557c05362f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:20.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.502+0000 7f9df2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05346190 0x557c05362f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58898/0 (socket says 172.21.15.161:58898) 2024-03-28T23:35:20.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.502+0000 7f9df2ffd640 1 -- 172.21.15.161:0/1223408133 learned_addr learned my addr 172.21.15.161:0/1223408133 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:20.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.502+0000 7f9df37fe640 1 --2- 172.21.15.161:0/1223408133 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557c05345db0 0x557c053629c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:20.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.502+0000 7f9df3fff640 1 --2- 172.21.15.161:0/1223408133 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557c0536c2a0 0x557c0533fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:20.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.502+0000 7f9df2ffd640 1 -- 172.21.15.161:0/1223408133 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557c05345db0 msgr2=0x557c053629c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:20.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.502+0000 7f9df2ffd640 1 --2- 172.21.15.161:0/1223408133 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557c05345db0 0x557c053629c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.502+0000 7f9df2ffd640 1 -- 172.21.15.161:0/1223408133 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557c0536c2a0 msgr2=0x557c0533fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:20.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.502+0000 7f9df2ffd640 1 --2- 172.21.15.161:0/1223408133 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557c0536c2a0 0x557c0533fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.502+0000 7f9df2ffd640 1 -- 172.21.15.161:0/1223408133 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c05384290 con 0x557c05346190 2024-03-28T23:35:20.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.502+0000 7f9df2ffd640 1 --2- 172.21.15.161:0/1223408133 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05346190 0x557c05362f00 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f9de402bd30 tx=0x7f9de4030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6174371ea3f8c83 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:20.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.502+0000 7f9dd8ff9640 1 -- 172.21.15.161:0/1223408133 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de4057070 con 0x557c05346190 2024-03-28T23:35:20.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.502+0000 7f9dd8ff9640 1 -- 172.21.15.161:0/1223408133 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9de4052c30 con 0x557c05346190 2024-03-28T23:35:20.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.502+0000 7f9dd8ff9640 1 -- 172.21.15.161:0/1223408133 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de4042040 con 0x557c05346190 2024-03-28T23:35:20.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.503+0000 7f9dfcbda740 1 -- 172.21.15.161:0/1223408133 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05346190 msgr2=0x557c05362f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:20.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.503+0000 7f9dfcbda740 1 --2- 172.21.15.161:0/1223408133 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05346190 0x557c05362f00 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f9de402bd30 tx=0x7f9de4030000 comp rx=0 tx=0).stop 2024-03-28T23:35:20.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.503+0000 7f9dfcbda740 1 -- 172.21.15.161:0/1223408133 shutdown_connections 2024-03-28T23:35:20.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.503+0000 7f9dfcbda740 1 --2- 172.21.15.161:0/1223408133 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05346190 0x557c05362f00 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.503+0000 7f9dfcbda740 1 --2- 172.21.15.161:0/1223408133 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557c0536c2a0 0x557c0533fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.503+0000 7f9dfcbda740 1 --2- 172.21.15.161:0/1223408133 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557c05345db0 0x557c053629c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.503+0000 7f9dfcbda740 1 -- 172.21.15.161:0/1223408133 >> 172.21.15.161:0/1223408133 conn(0x557c05344310 msgr2=0x557c05384c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:20.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.503+0000 7f9dfcbda740 1 -- 172.21.15.161:0/1223408133 shutdown_connections 2024-03-28T23:35:20.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.503+0000 7f9dfcbda740 1 -- 172.21.15.161:0/1223408133 wait complete. 2024-03-28T23:35:20.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.504+0000 7f9dfcbda740 1 -- start start 2024-03-28T23:35:20.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.504+0000 7f9dfcbda740 1 Processor -- start 2024-03-28T23:35:20.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.504+0000 7f9dfcbda740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05345db0 0x557c05362dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:20.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.504+0000 7f9dfcbda740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557c05346190 0x557c05343400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:20.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.504+0000 7f9dfcbda740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557c0536c2a0 0x557c0537dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:20.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.504+0000 7f9dfcbda740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x557c0536bf50 con 0x557c05345db0 2024-03-28T23:35:20.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.504+0000 7f9dfcbda740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x557c05396440 con 0x557c05346190 2024-03-28T23:35:20.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.504+0000 7f9dfcbda740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x557c053635a0 con 0x557c0536c2a0 2024-03-28T23:35:20.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.504+0000 7f9dfcbda740 10 client.0 _ll_register_callbacks cb 0x7f9de8059060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:35:20.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.504+0000 7f9df37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05345db0 0x557c05362dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:20.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.504+0000 7f9df37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05345db0 0x557c05362dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58902/0 (socket says 172.21.15.161:58902) 2024-03-28T23:35:20.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.504+0000 7f9df37fe640 1 -- 172.21.15.161:0/2003633503 learned_addr learned my addr 172.21.15.161:0/2003633503 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:20.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.505+0000 7f9df2ffd640 1 --2- 172.21.15.161:0/2003633503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557c05346190 0x557c05343400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:20.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.505+0000 7f9df3fff640 1 --2- 172.21.15.161:0/2003633503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557c0536c2a0 0x557c0537dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:20.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.505+0000 7f9df37fe640 1 -- 172.21.15.161:0/2003633503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557c0536c2a0 msgr2=0x557c0537dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:20.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.505+0000 7f9df37fe640 1 --2- 172.21.15.161:0/2003633503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557c0536c2a0 0x557c0537dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.505+0000 7f9df37fe640 1 -- 172.21.15.161:0/2003633503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557c05346190 msgr2=0x557c05343400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:20.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.505+0000 7f9df37fe640 1 --2- 172.21.15.161:0/2003633503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557c05346190 0x557c05343400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.505+0000 7f9df37fe640 1 -- 172.21.15.161:0/2003633503 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c0539bee0 con 0x557c05345db0 2024-03-28T23:35:20.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.505+0000 7f9df2ffd640 1 --2- 172.21.15.161:0/2003633503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557c05346190 0x557c05343400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:20.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.505+0000 7f9df37fe640 1 --2- 172.21.15.161:0/2003633503 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05345db0 0x557c05362dc0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f9de808ad00 tx=0x7f9de8088ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=898b2ea9fb1ed3c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:20.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.505+0000 7f9dd27fc640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:35:20.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.505+0000 7f9dd27fc640 1 -- 172.21.15.161:0/2003633503 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de8086070 con 0x557c05345db0 2024-03-28T23:35:20.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.505+0000 7f9dd27fc640 1 -- 172.21.15.161:0/2003633503 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 6101+0+0 (secure 0 0 0) 0x7f9de80584b0 con 0x557c05345db0 2024-03-28T23:35:20.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.505+0000 7f9dfcbda740 1 -- 172.21.15.161:0/2003633503 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9de4052c30 con 0x557c05345db0 2024-03-28T23:35:20.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.505+0000 7f9dd27fc640 1 -- 172.21.15.161:0/2003633503 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=124) ==== 24+0+0 (secure 0 0 0) 0x7f9de80589c0 con 0x557c05345db0 2024-03-28T23:35:20.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.505+0000 7f9dfcbda740 10 client.8613 fetch_fsmap learned FSMap version 124 2024-03-28T23:35:20.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.505+0000 7f9dfcbda740 1 -- 172.21.15.161:0/2003633503 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=124,monmap=2+}) -- 0x557c05343e60 con 0x557c05345db0 2024-03-28T23:35:20.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.505+0000 7f9dd27fc640 1 -- 172.21.15.161:0/2003633503 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9de4052c30 con 0x557c05345db0 2024-03-28T23:35:20.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.505+0000 7f9dd27fc640 1 -- 172.21.15.161:0/2003633503 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 124) ==== 3494+0+0 (secure 0 0 0) 0x7f9de8025040 con 0x557c05345db0 2024-03-28T23:35:20.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.505+0000 7f9dfcbda740 10 client.8613 fetch_fsmap finished waiting for FSMap version 124 2024-03-28T23:35:20.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.505+0000 7f9dfcbda740 10 client.8613 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T23:35:20.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.505+0000 7f9dfcbda740 20 client.8613 populate_metadata read hostname 'smithi161' 2024-03-28T23:35:20.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.505+0000 7f9dfcbda740 1 --2- 172.21.15.161:0/2003633503 >> [v2:172.21.15.161:6834/2527502405,v1:172.21.15.161:6836/2527502405] conn(0x557c0537e8a0 0x557c053f9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:20.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.505+0000 7f9dfcbda740 4 client.8613 mds_command: new command op to 8412 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:35:20.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.505+0000 7f9dfcbda740 1 -- 172.21.15.161:0/2003633503 --> [v2:172.21.15.161:6834/2527502405,v1:172.21.15.161:6836/2527502405] -- command(tid 0: {"prefix": "status"}) -- 0x557c052b1380 con 0x557c0537e8a0 2024-03-28T23:35:20.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.506+0000 7f9df2ffd640 1 --2- 172.21.15.161:0/2003633503 >> [v2:172.21.15.161:6834/2527502405,v1:172.21.15.161:6836/2527502405] conn(0x557c0537e8a0 0x557c053f9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:20.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.506+0000 7f9dd27fc640 10 client.8613 ms_handle_connect on v2:172.21.15.161:6834/2527502405 2024-03-28T23:35:20.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.506+0000 7f9df2ffd640 1 --2- 172.21.15.161:0/2003633503 >> [v2:172.21.15.161:6834/2527502405,v1:172.21.15.161:6836/2527502405] conn(0x557c0537e8a0 0x557c053f9f80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=531c7ac699106cba server_cookie=1bf2cffdd49f77be in_seq=0 out_seq=0 2024-03-28T23:35:20.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.506+0000 7f9dd27fc640 1 -- 172.21.15.161:0/2003633503 <== mds.0 v2:172.21.15.161:6834/2527502405 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x557c052b1380 con 0x557c0537e8a0 2024-03-28T23:35:20.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.506+0000 7f9dd27fc640 10 client.8613 handle_command_reply: tid=0 2024-03-28T23:35:20.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.506+0000 7f9dfcbda740 1 client.8613 shutdown 2024-03-28T23:35:20.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.506+0000 7f9dfcbda740 1 -- 172.21.15.161:0/2003633503 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05345db0 msgr2=0x557c05362dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:20.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.506+0000 7f9dfcbda740 1 --2- 172.21.15.161:0/2003633503 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05345db0 0x557c05362dc0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f9de808ad00 tx=0x7f9de8088ce0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.506+0000 7f9dfcbda740 1 -- 172.21.15.161:0/2003633503 shutdown_connections 2024-03-28T23:35:20.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.506+0000 7f9dfcbda740 1 --2- 172.21.15.161:0/2003633503 >> [v2:172.21.15.161:6834/2527502405,v1:172.21.15.161:6836/2527502405] conn(0x557c0537e8a0 0x557c053f9f80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.506+0000 7f9dd27fc640 0 client.8613 ms_handle_reset on v2:172.21.15.161:6834/2527502405 2024-03-28T23:35:20.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.506+0000 7f9dfcbda740 1 --2- 172.21.15.161:0/2003633503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x557c0536c2a0 0x557c0537dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.506+0000 7f9dfcbda740 1 --2- 172.21.15.161:0/2003633503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x557c05346190 0x557c05343400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.506+0000 7f9dfcbda740 1 --2- 172.21.15.161:0/2003633503 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c05345db0 0x557c05362dc0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.506+0000 7f9dfcbda740 1 -- 172.21.15.161:0/2003633503 >> 172.21.15.161:0/2003633503 conn(0x557c05344310 msgr2=0x557c053447b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:20.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.506+0000 7f9dfcbda740 1 -- 172.21.15.161:0/2003633503 shutdown_connections 2024-03-28T23:35:20.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.506+0000 7f9dfcbda740 1 -- 172.21.15.161:0/2003633503 wait complete. 2024-03-28T23:35:20.514 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.507+0000 7f9dfcbda740 20 client.8613 trim_cache size 0 max 16384 2024-03-28T23:35:20.515 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:35:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:35:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:35:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 8412, 2024-03-28T23:35:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:35:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:35:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:35:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 6.8637200380000003, 2024-03-28T23:35:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 124, 2024-03-28T23:35:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 136, 2024-03-28T23:35:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 136, 2024-03-28T23:35:20.515 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 11.96774952 2024-03-28T23:35:20.515 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:35:20.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.507+0000 7f9df9ffd640 1 -- 172.21.15.161:0/3405193049 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9de006b500 msgr2=0x7f9de006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:20.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.507+0000 7f9df9ffd640 1 --2- 172.21.15.161:0/3405193049 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9de006b500 0x7f9de006d9c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f9de4001770 tx=0x7f9de4009030 comp rx=0 tx=0).stop 2024-03-28T23:35:20.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.507+0000 7f9df9ffd640 1 -- 172.21.15.161:0/3405193049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ddc004090 msgr2=0x7f9df414d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:20.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.507+0000 7f9df9ffd640 1 --2- 172.21.15.161:0/3405193049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ddc004090 0x7f9df414d9d0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f9de80119a0 tx=0x7f9de8011e60 comp rx=0 tx=0).stop 2024-03-28T23:35:20.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.507+0000 7f9df9ffd640 1 -- 172.21.15.161:0/3405193049 shutdown_connections 2024-03-28T23:35:20.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.507+0000 7f9df9ffd640 1 --2- 172.21.15.161:0/3405193049 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f9de006b500 0x7f9de006d9c0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.507+0000 7f9df9ffd640 1 --2- 172.21.15.161:0/3405193049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9df40be3d0 0x7f9df40aa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.507+0000 7f9df9ffd640 1 --2- 172.21.15.161:0/3405193049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9df414cde0 0x7f9df40bde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.507+0000 7f9df9ffd640 1 --2- 172.21.15.161:0/3405193049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ddc004090 0x7f9df414d9d0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:20.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.507+0000 7f9df9ffd640 1 -- 172.21.15.161:0/3405193049 >> 172.21.15.161:0/3405193049 conn(0x7f9df40ac620 msgr2=0x7f9df4150b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:20.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.507+0000 7f9df9ffd640 1 -- 172.21.15.161:0/3405193049 shutdown_connections 2024-03-28T23:35:20.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:20.507+0000 7f9df9ffd640 1 -- 172.21.15.161:0/3405193049 wait complete. 2024-03-28T23:35:20.519 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 124, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:35:12.982566+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 124, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8412": { "addr": "172.21.15.161:6836/2527502405", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 2527502405, "type": "v2" }, { "addr": "172.21.15.161:6836", "nonce": 2527502405, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8412, "incarnation": 121, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-03-28T23:35:15.337512+0000", "qdb_cluster": [ 8412 ], "qdb_leader": 8412, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8412 } } } ], "standbys": [ { "addr": "172.21.15.170:6835/3586875977", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 3586875977, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 3586875977, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 120, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6740, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.170:6834/3182285172", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 3182285172, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 3182285172, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 120, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7639, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6837/3037890643", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6835", "nonce": 3037890643, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 3037890643, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 120, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8409, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:35:20.520 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 10, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 10, 'name': 'd'}] 2024-03-28T23:35:20.520 INFO:tasks.cephfs.test_failover:current = [{'gid': 8412, 'name': 'a', 'rank': 0, 'incarnation': 121, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.161:6836/2527502405', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 2527502405}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 2527502405}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 6740, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/3586875977', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 3586875977}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 3586875977}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 120}, {'gid': 8409, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6837/3037890643', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 3037890643}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 3037890643}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 120}, {'gid': 7639, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6834/3182285172', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 3182285172}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 3182285172}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 120}] 2024-03-28T23:35:20.520 DEBUG:tasks.cephfs.test_failover:10 != -1 2024-03-28T23:35:20.520 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-28T23:35:21.288 INFO:tasks.ceph.mds.a.smithi161.stderr: -104> 2024-03-28T23:35:18.904+0000 7f84c43af640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:35:21.313 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-28T23:35:25.522 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:35:25.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.634+0000 7f2559da7640 1 Processor -- start 2024-03-28T23:35:25.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.634+0000 7f2559da7640 1 -- start start 2024-03-28T23:35:25.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.634+0000 7f2559da7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2554150600 0x7f25540a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:25.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.634+0000 7f2559da7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25540aacb0 0x7f2554153980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:25.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.634+0000 7f2559da7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25541567c0 con 0x7f25540a8fc0 2024-03-28T23:35:25.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.634+0000 7f2559da7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f2554156570 con 0x7f2554150600 2024-03-28T23:35:25.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.634+0000 7f2559da7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2554156aa0 con 0x7f25540aacb0 2024-03-28T23:35:25.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.635+0000 7f25537fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2554150600 0x7f25540a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:25.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.635+0000 7f2553fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25540aacb0 0x7f2554153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:25.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.635+0000 7f2552ffd640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f25540a8fc0 0x7f2554153270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:42092/0 (socket says 172.21.15.161:42092) 2024-03-28T23:35:25.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.635+0000 7f25537fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2554150600 0x7f25540a9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:50278/0 (socket says 172.21.15.161:50278) 2024-03-28T23:35:25.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.635+0000 7f2553fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25540aacb0 0x7f2554153980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:45738/0 (socket says 172.21.15.161:45738) 2024-03-28T23:35:25.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.635+0000 7f2552ffd640 1 -- 172.21.15.161:0/3212231237 learned_addr learned my addr 172.21.15.161:0/3212231237 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:25.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.635+0000 7f25537fe640 1 -- 172.21.15.161:0/3212231237 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25540aacb0 msgr2=0x7f2554153980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:25.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.635+0000 7f25537fe640 1 --2- 172.21.15.161:0/3212231237 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25540aacb0 0x7f2554153980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:25.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.635+0000 7f25537fe640 1 -- 172.21.15.161:0/3212231237 >> v1:172.21.15.170:6789/0 conn(0x7f25540a8fc0 legacy=0x7f2554153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:25.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.635+0000 7f25537fe640 1 -- 172.21.15.161:0/3212231237 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2554156d80 con 0x7f2554150600 2024-03-28T23:35:25.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.635+0000 7f25527fc640 1 -- 172.21.15.161:0/3212231237 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3261130476 0 0) 0x7f25541567c0 con 0x7f25540a8fc0 2024-03-28T23:35:25.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.635+0000 7f2553fff640 1 --2- 172.21.15.161:0/3212231237 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25540aacb0 0x7f2554153980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:25.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.636+0000 7f25537fe640 1 --2- 172.21.15.161:0/3212231237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2554150600 0x7f25540a9f40 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f25440097a0 tx=0x7f254402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c961e38d4dc04cd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:25.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.636+0000 7f25527fc640 1 -- 172.21.15.161:0/3212231237 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25440315a0 con 0x7f2554150600 2024-03-28T23:35:25.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.636+0000 7f25527fc640 1 -- 172.21.15.161:0/3212231237 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2544037070 con 0x7f2554150600 2024-03-28T23:35:25.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.636+0000 7f25527fc640 1 -- 172.21.15.161:0/3212231237 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2544031960 con 0x7f2554150600 2024-03-28T23:35:25.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.637+0000 7f2559da7640 1 -- 172.21.15.161:0/3212231237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2554150600 msgr2=0x7f25540a9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:25.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.637+0000 7f2559da7640 1 --2- 172.21.15.161:0/3212231237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2554150600 0x7f25540a9f40 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f25440097a0 tx=0x7f254402b840 comp rx=0 tx=0).stop 2024-03-28T23:35:25.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.637+0000 7f2559da7640 1 -- 172.21.15.161:0/3212231237 shutdown_connections 2024-03-28T23:35:25.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.637+0000 7f2559da7640 1 --2- 172.21.15.161:0/3212231237 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25540aacb0 0x7f2554153980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:25.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.637+0000 7f2559da7640 1 --2- 172.21.15.161:0/3212231237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2554150600 0x7f25540a9f40 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:25.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.637+0000 7f2559da7640 1 -- 172.21.15.161:0/3212231237 >> 172.21.15.161:0/3212231237 conn(0x7f25540c6670 msgr2=0x7f25540c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:25.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.637+0000 7f2559da7640 1 -- 172.21.15.161:0/3212231237 shutdown_connections 2024-03-28T23:35:25.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.637+0000 7f2559da7640 1 -- 172.21.15.161:0/3212231237 wait complete. 2024-03-28T23:35:25.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.638+0000 7f2559da7640 1 Processor -- start 2024-03-28T23:35:25.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.638+0000 7f2559da7640 1 -- start start 2024-03-28T23:35:25.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.639+0000 7f2559da7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25540a8fc0 0x7f2554151820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:25.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.639+0000 7f2559da7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25540aacb0 0x7f25541ec0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:25.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.639+0000 7f2559da7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2554150600 0x7f25541ea990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:25.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.639+0000 7f2559da7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2554153ec0 con 0x7f25540aacb0 2024-03-28T23:35:25.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.639+0000 7f2559da7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2554157340 con 0x7f25540a8fc0 2024-03-28T23:35:25.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.639+0000 7f2559da7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f25541509e0 con 0x7f2554150600 2024-03-28T23:35:25.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.639+0000 7f25537fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25540a8fc0 0x7f2554151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:25.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.639+0000 7f2552ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25540aacb0 0x7f25541ec0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:25.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.639+0000 7f25537fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25540a8fc0 0x7f2554151820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:50290/0 (socket says 172.21.15.161:50290) 2024-03-28T23:35:25.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.639+0000 7f2553fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2554150600 0x7f25541ea990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:25.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.639+0000 7f25537fe640 1 -- 172.21.15.161:0/3144464339 learned_addr learned my addr 172.21.15.161:0/3144464339 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:25.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.639+0000 7f25537fe640 1 -- 172.21.15.161:0/3144464339 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2554150600 msgr2=0x7f25541ea990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:25.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.639+0000 7f25537fe640 1 --2- 172.21.15.161:0/3144464339 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2554150600 0x7f25541ea990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:25.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.639+0000 7f25537fe640 1 -- 172.21.15.161:0/3144464339 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25540aacb0 msgr2=0x7f25541ec0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:25.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.639+0000 7f25537fe640 1 --2- 172.21.15.161:0/3144464339 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25540aacb0 0x7f25541ec0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:25.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.639+0000 7f25537fe640 1 -- 172.21.15.161:0/3144464339 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25541ecb20 con 0x7f25540a8fc0 2024-03-28T23:35:25.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.639+0000 7f2553fff640 1 --2- 172.21.15.161:0/3144464339 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2554150600 0x7f25541ea990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:25.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.639+0000 7f25537fe640 1 --2- 172.21.15.161:0/3144464339 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25540a8fc0 0x7f2554151820 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f25440097a0 tx=0x7f25440046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:25.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.639+0000 7f253bfff640 1 -- 172.21.15.161:0/3144464339 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2544004030 con 0x7f25540a8fc0 2024-03-28T23:35:25.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.640+0000 7f253bfff640 1 -- 172.21.15.161:0/3144464339 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25440041d0 con 0x7f25540a8fc0 2024-03-28T23:35:25.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.640+0000 7f253bfff640 1 -- 172.21.15.161:0/3144464339 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2544003610 con 0x7f25540a8fc0 2024-03-28T23:35:25.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.640+0000 7f2559da7640 1 -- 172.21.15.161:0/3144464339 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25541e9eb0 con 0x7f25540a8fc0 2024-03-28T23:35:25.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.640+0000 7f2559da7640 1 -- 172.21.15.161:0/3144464339 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25540c5fe0 con 0x7f25540a8fc0 2024-03-28T23:35:25.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.641+0000 7f2559da7640 1 -- 172.21.15.161:0/3144464339 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f255404f4b0 con 0x7f25540a8fc0 2024-03-28T23:35:25.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.641+0000 7f253bfff640 1 -- 172.21.15.161:0/3144464339 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f25440037b0 con 0x7f25540a8fc0 2024-03-28T23:35:25.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.641+0000 7f253bfff640 1 --2- 172.21.15.161:0/3144464339 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f254006b4d0 0x7f254006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:25.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.642+0000 7f253bfff640 1 -- 172.21.15.161:0/3144464339 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 6187+0+0 (secure 0 0 0) 0x7f254402f030 con 0x7f25540a8fc0 2024-03-28T23:35:25.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.642+0000 7f2552ffd640 1 --2- 172.21.15.161:0/3144464339 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f254006b4d0 0x7f254006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:25.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.642+0000 7f2552ffd640 1 --2- 172.21.15.161:0/3144464339 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f254006b4d0 0x7f254006d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f25541ebd90 tx=0x7f2548007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:25.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.646+0000 7f253bfff640 1 -- 172.21.15.161:0/3144464339 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2544041070 con 0x7f25540a8fc0 2024-03-28T23:35:25.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.852+0000 7f2559da7640 1 -- 172.21.15.161:0/3144464339 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f255414cdf0 con 0x7f25540a8fc0 2024-03-28T23:35:25.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.853+0000 7f253bfff640 1 -- 172.21.15.161:0/3144464339 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) ==== 95+0+5202 (secure 0 0 0) 0x7f254407ae30 con 0x7f25540a8fc0 2024-03-28T23:35:25.853 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:25.854 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":129,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7639,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.170:6834/3182285172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3182285172},{"type":"v1","addr":"172.21.15.170:6834","nonce":3182285172}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":125},{"gid":8409,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/3037890643","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3037890643},{"type":"v1","addr":"172.21.15.161:6837","nonce":3037890643}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120},{"gid":8622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/819329318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":819329318},{"type":"v1","addr":"172.21.15.161:6836","nonce":819329318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:12.982566+0000","modified":"2024-03-28T23:35:24.297578+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":137,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":126,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6835/3586875977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3586875977},{"type":"v1","addr":"172.21.15.170:6835","nonce":3586875977}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":10}]} 2024-03-28T23:35:25.854 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 129 2024-03-28T23:35:25.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.855+0000 7f2559da7640 1 -- 172.21.15.161:0/3144464339 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f254006b4d0 msgr2=0x7f254006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:25.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.855+0000 7f2559da7640 1 --2- 172.21.15.161:0/3144464339 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f254006b4d0 0x7f254006d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f25541ebd90 tx=0x7f2548007de0 comp rx=0 tx=0).stop 2024-03-28T23:35:25.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.855+0000 7f2559da7640 1 -- 172.21.15.161:0/3144464339 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25540a8fc0 msgr2=0x7f2554151820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:25.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.855+0000 7f2559da7640 1 --2- 172.21.15.161:0/3144464339 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25540a8fc0 0x7f2554151820 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f25440097a0 tx=0x7f25440046c0 comp rx=0 tx=0).stop 2024-03-28T23:35:25.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.855+0000 7f2559da7640 1 -- 172.21.15.161:0/3144464339 shutdown_connections 2024-03-28T23:35:25.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.855+0000 7f2559da7640 1 --2- 172.21.15.161:0/3144464339 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f254006b4d0 0x7f254006d990 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:25.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.855+0000 7f2559da7640 1 --2- 172.21.15.161:0/3144464339 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25540aacb0 0x7f25541ec0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:25.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.855+0000 7f2559da7640 1 --2- 172.21.15.161:0/3144464339 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2554150600 0x7f25541ea990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:25.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.855+0000 7f2559da7640 1 --2- 172.21.15.161:0/3144464339 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25540a8fc0 0x7f2554151820 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:25.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.855+0000 7f2559da7640 1 -- 172.21.15.161:0/3144464339 >> 172.21.15.161:0/3144464339 conn(0x7f25540c6670 msgr2=0x7f2554153100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:25.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.855+0000 7f2559da7640 1 -- 172.21.15.161:0/3144464339 shutdown_connections 2024-03-28T23:35:25.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:25.855+0000 7f2559da7640 1 -- 172.21.15.161:0/3144464339 wait complete. 2024-03-28T23:35:26.144 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:35:26.146 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:35:26.269 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 129, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:35:12.982566+0000', 'modified': '2024-03-28T23:35:24.297578+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 137, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6740}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6740': {'gid': 6740, 'name': 'b', 'rank': 0, 'incarnation': 126, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.170:6835/3586875977', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 3586875977}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 3586875977}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6740, 'qdb_cluster': [6740]} 2024-03-28T23:35:26.269 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:35:26.269 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T23:35:26.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.371+0000 7fe8e2840640 1 Processor -- start 2024-03-28T23:35:26.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.371+0000 7fe8e2840640 1 -- start start 2024-03-28T23:35:26.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.372+0000 7fe8e2840640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe8dc152890 0x7fe8dc0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:26.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.372+0000 7fe8e2840640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe8dc0c8ed0 0x7fe8dc0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:26.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.372+0000 7fe8e2840640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8dc158a30 con 0x7fe8dc0c5950 2024-03-28T23:35:26.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.372+0000 7fe8e2840640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe8dc0c8480 con 0x7fe8dc0c8ed0 2024-03-28T23:35:26.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.372+0000 7fe8e2840640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fe8dc159640 con 0x7fe8dc152890 2024-03-28T23:35:26.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.372+0000 7fe8db7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe8dc0c8ed0 0x7fe8dc0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:26.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.372+0000 7fe8db7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe8dc0c8ed0 0x7fe8dc0c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58946/0 (socket says 172.21.15.161:58946) 2024-03-28T23:35:26.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.372+0000 7fe8db7fe640 1 -- 172.21.15.161:0/1317187644 learned_addr learned my addr 172.21.15.161:0/1317187644 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:26.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.372+0000 7fe8e0ddd640 1 --1- 172.21.15.161:0/1317187644 >> v1:172.21.15.161:6789/0 conn(0x7fe8dc0c5950 0x7fe8dc1553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:56860/0 (socket says 172.21.15.161:56860) 2024-03-28T23:35:26.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.372+0000 7fe8dbfff640 1 --2- 172.21.15.161:0/1317187644 >> v2:172.21.15.170:3300/0 conn(0x7fe8dc152890 0x7fe8dc0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:26.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.373+0000 7fe8db7fe640 1 -- 172.21.15.161:0/1317187644 >> v2:172.21.15.170:3300/0 conn(0x7fe8dc152890 msgr2=0x7fe8dc0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:26.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.373+0000 7fe8db7fe640 1 --2- 172.21.15.161:0/1317187644 >> v2:172.21.15.170:3300/0 conn(0x7fe8dc152890 0x7fe8dc0c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.373+0000 7fe8db7fe640 1 -- 172.21.15.161:0/1317187644 >> v1:172.21.15.161:6789/0 conn(0x7fe8dc0c5950 legacy=0x7fe8dc1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:26.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.373+0000 7fe8daffd640 1 -- 172.21.15.161:0/1317187644 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 416842750 0 0) 0x7fe8dc158a30 con 0x7fe8dc0c5950 2024-03-28T23:35:26.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.373+0000 7fe8db7fe640 1 -- 172.21.15.161:0/1317187644 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8dc158d10 con 0x7fe8dc0c8ed0 2024-03-28T23:35:26.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.373+0000 7fe8db7fe640 1 --2- 172.21.15.161:0/1317187644 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8dc0c8ed0 0x7fe8dc0c5410 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc08cc80 tx=0x7fe8d002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83f55a8832421636 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:26.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.373+0000 7fe8daffd640 1 -- 172.21.15.161:0/1317187644 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0037070 con 0x7fe8dc0c8ed0 2024-03-28T23:35:26.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.374+0000 7fe8daffd640 1 -- 172.21.15.161:0/1317187644 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8d0004230 con 0x7fe8dc0c8ed0 2024-03-28T23:35:26.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.374+0000 7fe8daffd640 1 -- 172.21.15.161:0/1317187644 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0003610 con 0x7fe8dc0c8ed0 2024-03-28T23:35:26.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.374+0000 7fe8e2840640 1 -- 172.21.15.161:0/1317187644 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8dc0c8ed0 msgr2=0x7fe8dc0c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:26.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.374+0000 7fe8e2840640 1 --2- 172.21.15.161:0/1317187644 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8dc0c8ed0 0x7fe8dc0c5410 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc08cc80 tx=0x7fe8d002b860 comp rx=0 tx=0).stop 2024-03-28T23:35:26.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.374+0000 7fe8e2840640 1 -- 172.21.15.161:0/1317187644 shutdown_connections 2024-03-28T23:35:26.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.374+0000 7fe8e2840640 1 --2- 172.21.15.161:0/1317187644 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8dc0c8ed0 0x7fe8dc0c5410 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.374+0000 7fe8e2840640 1 --2- 172.21.15.161:0/1317187644 >> v2:172.21.15.170:3300/0 conn(0x7fe8dc152890 0x7fe8dc0c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.374+0000 7fe8e2840640 1 -- 172.21.15.161:0/1317187644 >> 172.21.15.161:0/1317187644 conn(0x7fe8dc0ac620 msgr2=0x7fe8dc0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:26.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.374+0000 7fe8e2840640 1 -- 172.21.15.161:0/1317187644 shutdown_connections 2024-03-28T23:35:26.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.374+0000 7fe8e2840640 1 -- 172.21.15.161:0/1317187644 wait complete. 2024-03-28T23:35:26.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.375+0000 7fe8e2840640 1 Processor -- start 2024-03-28T23:35:26.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.375+0000 7fe8e2840640 1 -- start start 2024-03-28T23:35:26.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.376+0000 7fe8e2840640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8dc0c5950 0x7fe8dc0c1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:26.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.376+0000 7fe8e2840640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8dc0c8ed0 0x7fe8dc0bf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:26.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.376+0000 7fe8e2840640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8dc152890 0x7fe8dc0bf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:26.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.376+0000 7fe8e2840640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe8dc152c70 con 0x7fe8dc0c8ed0 2024-03-28T23:35:26.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.376+0000 7fe8e2840640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe8dc1592d0 con 0x7fe8dc152890 2024-03-28T23:35:26.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.376+0000 7fe8e2840640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe8dc152df0 con 0x7fe8dc0c5950 2024-03-28T23:35:26.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.376+0000 7fe8db7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8dc0c8ed0 0x7fe8dc0bf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:26.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.376+0000 7fe8db7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8dc0c8ed0 0x7fe8dc0bf4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58952/0 (socket says 172.21.15.161:58952) 2024-03-28T23:35:26.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.376+0000 7fe8db7fe640 1 -- 172.21.15.161:0/3917607564 learned_addr learned my addr 172.21.15.161:0/3917607564 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:26.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.376+0000 7fe8dbfff640 1 --2- 172.21.15.161:0/3917607564 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8dc0c5950 0x7fe8dc0c1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:26.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.376+0000 7fe8e0ddd640 1 --2- 172.21.15.161:0/3917607564 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8dc152890 0x7fe8dc0bf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:26.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.377+0000 7fe8db7fe640 1 -- 172.21.15.161:0/3917607564 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8dc0c5950 msgr2=0x7fe8dc0c1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:26.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.377+0000 7fe8db7fe640 1 --2- 172.21.15.161:0/3917607564 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8dc0c5950 0x7fe8dc0c1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.377+0000 7fe8db7fe640 1 -- 172.21.15.161:0/3917607564 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8dc152890 msgr2=0x7fe8dc0bf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:26.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.377+0000 7fe8db7fe640 1 --2- 172.21.15.161:0/3917607564 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8dc152890 0x7fe8dc0bf9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.377+0000 7fe8db7fe640 1 -- 172.21.15.161:0/3917607564 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8dc1e86c0 con 0x7fe8dc0c8ed0 2024-03-28T23:35:26.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.377+0000 7fe8db7fe640 1 --2- 172.21.15.161:0/3917607564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8dc0c8ed0 0x7fe8dc0bf4b0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fe8d002b8a0 tx=0x7fe8d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:26.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.377+0000 7fe8d8ff9640 1 -- 172.21.15.161:0/3917607564 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0037050 con 0x7fe8dc0c8ed0 2024-03-28T23:35:26.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.377+0000 7fe8d8ff9640 1 -- 172.21.15.161:0/3917607564 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8d0043070 con 0x7fe8dc0c8ed0 2024-03-28T23:35:26.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.377+0000 7fe8d8ff9640 1 -- 172.21.15.161:0/3917607564 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0003d80 con 0x7fe8dc0c8ed0 2024-03-28T23:35:26.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.377+0000 7fe8e2840640 1 -- 172.21.15.161:0/3917607564 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8dc0be6f0 con 0x7fe8dc0c8ed0 2024-03-28T23:35:26.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.377+0000 7fe8e2840640 1 -- 172.21.15.161:0/3917607564 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8dc0becb0 con 0x7fe8dc0c8ed0 2024-03-28T23:35:26.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.378+0000 7fe8d8ff9640 1 -- 172.21.15.161:0/3917607564 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe8d0006040 con 0x7fe8dc0c8ed0 2024-03-28T23:35:26.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.379+0000 7fe8d8ff9640 1 --2- 172.21.15.161:0/3917607564 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe8c806b4d0 0x7fe8c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:26.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.379+0000 7fe8dbfff640 1 --2- 172.21.15.161:0/3917607564 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe8c806b4d0 0x7fe8c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:26.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.379+0000 7fe8d8ff9640 1 -- 172.21.15.161:0/3917607564 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 6187+0+0 (secure 0 0 0) 0x7fe8d00ac6c0 con 0x7fe8dc0c8ed0 2024-03-28T23:35:26.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.379+0000 7fe8dbfff640 1 --2- 172.21.15.161:0/3917607564 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe8c806b4d0 0x7fe8c806d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc151980 tx=0x7fe8cc009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:26.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.380+0000 7fe8e5428740 1 Processor -- start 2024-03-28T23:35:26.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.380+0000 7fe8e5428740 1 -- start start 2024-03-28T23:35:26.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.380+0000 7fe8e5428740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2cddb0 0x5582be2cb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:26.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.380+0000 7fe8e5428740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5582be2ce190 0x5582be2cbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:26.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.380+0000 7fe8e5428740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5582be2cc310 0x5582be2c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:26.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.380+0000 7fe8e5428740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5582be231af0 con 0x5582be2cddb0 2024-03-28T23:35:26.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.380+0000 7fe8e5428740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5582be1da9d0 con 0x5582be2cc310 2024-03-28T23:35:26.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.380+0000 7fe8e5428740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x5582be22db90 con 0x5582be2ce190 2024-03-28T23:35:26.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.381+0000 7fe8dbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2cddb0 0x5582be2cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:26.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.381+0000 7fe8dbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2cddb0 0x5582be2cb890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58958/0 (socket says 172.21.15.161:58958) 2024-03-28T23:35:26.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.381+0000 7fe8dbfff640 1 -- 172.21.15.161:0/1517815261 learned_addr learned my addr 172.21.15.161:0/1517815261 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:26.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.381+0000 7fe8db7fe640 1 --2- 172.21.15.161:0/1517815261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5582be2ce190 0x5582be2cbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:26.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.381+0000 7fe8e0ddd640 1 --2- 172.21.15.161:0/1517815261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5582be2cc310 0x5582be2c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:26.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.381+0000 7fe8dbfff640 1 -- 172.21.15.161:0/1517815261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5582be2ce190 msgr2=0x5582be2cbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:26.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.381+0000 7fe8dbfff640 1 --2- 172.21.15.161:0/1517815261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5582be2ce190 0x5582be2cbdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.381+0000 7fe8dbfff640 1 -- 172.21.15.161:0/1517815261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5582be2cc310 msgr2=0x5582be2c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:26.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.381+0000 7fe8dbfff640 1 --2- 172.21.15.161:0/1517815261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5582be2cc310 0x5582be2c84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.381+0000 7fe8dbfff640 1 -- 172.21.15.161:0/1517815261 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582bdf462d0 con 0x5582be2cddb0 2024-03-28T23:35:26.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.381+0000 7fe8dbfff640 1 --2- 172.21.15.161:0/1517815261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2cddb0 0x5582be2cb890 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc006a90 tx=0x7fe8cc014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3b3e295bef5ab01 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:26.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.381+0000 7fe8c1ffb640 1 -- 172.21.15.161:0/1517815261 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8cc01b070 con 0x5582be2cddb0 2024-03-28T23:35:26.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.381+0000 7fe8c1ffb640 1 -- 172.21.15.161:0/1517815261 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8cc027070 con 0x5582be2cddb0 2024-03-28T23:35:26.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.381+0000 7fe8c1ffb640 1 -- 172.21.15.161:0/1517815261 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8cc003a20 con 0x5582be2cddb0 2024-03-28T23:35:26.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.382+0000 7fe8e5428740 1 -- 172.21.15.161:0/1517815261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2cddb0 msgr2=0x5582be2cb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:26.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.382+0000 7fe8e5428740 1 --2- 172.21.15.161:0/1517815261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2cddb0 0x5582be2cb890 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc006a90 tx=0x7fe8cc014000 comp rx=0 tx=0).stop 2024-03-28T23:35:26.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.382+0000 7fe8e5428740 1 -- 172.21.15.161:0/1517815261 shutdown_connections 2024-03-28T23:35:26.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.382+0000 7fe8e5428740 1 --2- 172.21.15.161:0/1517815261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5582be2cc310 0x5582be2c84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.382+0000 7fe8e5428740 1 --2- 172.21.15.161:0/1517815261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5582be2ce190 0x5582be2cbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.382+0000 7fe8e5428740 1 --2- 172.21.15.161:0/1517815261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2cddb0 0x5582be2cb890 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.382+0000 7fe8e5428740 1 -- 172.21.15.161:0/1517815261 >> 172.21.15.161:0/1517815261 conn(0x5582be2ca630 msgr2=0x5582be2ef220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:26.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.382+0000 7fe8e5428740 1 -- 172.21.15.161:0/1517815261 shutdown_connections 2024-03-28T23:35:26.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.382+0000 7fe8e5428740 1 -- 172.21.15.161:0/1517815261 wait complete. 2024-03-28T23:35:26.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.383+0000 7fe8e5428740 1 -- start start 2024-03-28T23:35:26.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.383+0000 7fe8e5428740 1 Processor -- start 2024-03-28T23:35:26.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.383+0000 7fe8e5428740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5582be2cddb0 0x5582be30a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:26.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.383+0000 7fe8e5428740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2ce190 0x5582be2f1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:26.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.384+0000 7fe8e5428740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5582be2f2ff0 0x5582be2eadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:26.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.384+0000 7fe8e5428740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5582be2f0090 con 0x5582be2ce190 2024-03-28T23:35:26.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.384+0000 7fe8e5428740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5582be1edf70 con 0x5582be2f2ff0 2024-03-28T23:35:26.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.384+0000 7fe8e5428740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x5582be2efad0 con 0x5582be2cddb0 2024-03-28T23:35:26.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.384+0000 7fe8e5428740 10 client.0 _ll_register_callbacks cb 0x5582bdf45ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:35:26.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.384+0000 7fe8db7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2ce190 0x5582be2f1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:26.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.384+0000 7fe8db7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2ce190 0x5582be2f1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58974/0 (socket says 172.21.15.161:58974) 2024-03-28T23:35:26.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.384+0000 7fe8db7fe640 1 -- 172.21.15.161:0/3740849815 learned_addr learned my addr 172.21.15.161:0/3740849815 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:26.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.384+0000 7fe8e0ddd640 1 --2- 172.21.15.161:0/3740849815 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5582be2f2ff0 0x5582be2eadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:26.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.384+0000 7fe8dbfff640 1 --2- 172.21.15.161:0/3740849815 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5582be2cddb0 0x5582be30a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:26.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.384+0000 7fe8db7fe640 1 -- 172.21.15.161:0/3740849815 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5582be2cddb0 msgr2=0x5582be30a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:26.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.384+0000 7fe8db7fe640 1 --2- 172.21.15.161:0/3740849815 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5582be2cddb0 0x5582be30a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.384+0000 7fe8db7fe640 1 -- 172.21.15.161:0/3740849815 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5582be2f2ff0 msgr2=0x5582be2eadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:26.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.384+0000 7fe8db7fe640 1 --2- 172.21.15.161:0/3740849815 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5582be2f2ff0 0x5582be2eadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.384+0000 7fe8db7fe640 1 -- 172.21.15.161:0/3740849815 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582be0106b0 con 0x5582be2ce190 2024-03-28T23:35:26.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.384+0000 7fe8db7fe640 1 --2- 172.21.15.161:0/3740849815 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2ce190 0x5582be2f1620 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00095c0 tx=0x7fe8d007a860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=199b928502f2c55d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:26.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.384+0000 7fe8baffd640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:35:26.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.384+0000 7fe8baffd640 1 -- 172.21.15.161:0/3740849815 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0096020 con 0x5582be2ce190 2024-03-28T23:35:26.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.384+0000 7fe8baffd640 1 -- 172.21.15.161:0/3740849815 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 6187+0+0 (secure 0 0 0) 0x7fe8d00982c0 con 0x5582be2ce190 2024-03-28T23:35:26.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.384+0000 7fe8e5428740 1 -- 172.21.15.161:0/3740849815 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8cc027070 con 0x5582be2ce190 2024-03-28T23:35:26.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.384+0000 7fe8baffd640 1 -- 172.21.15.161:0/3740849815 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=129) ==== 24+0+0 (secure 0 0 0) 0x7fe8d00810b0 con 0x5582be2ce190 2024-03-28T23:35:26.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.384+0000 7fe8e5428740 10 client.8640 fetch_fsmap learned FSMap version 129 2024-03-28T23:35:26.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.384+0000 7fe8e5428740 1 -- 172.21.15.161:0/3740849815 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=129,monmap=2+}) -- 0x5582be2f3b70 con 0x5582be2ce190 2024-03-28T23:35:26.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.384+0000 7fe8baffd640 1 -- 172.21.15.161:0/3740849815 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8cc027070 con 0x5582be2ce190 2024-03-28T23:35:26.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.385+0000 7fe8baffd640 1 -- 172.21.15.161:0/3740849815 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 129) ==== 3494+0+0 (secure 0 0 0) 0x7fe8d007a290 con 0x5582be2ce190 2024-03-28T23:35:26.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.385+0000 7fe8e5428740 10 client.8640 fetch_fsmap finished waiting for FSMap version 129 2024-03-28T23:35:26.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.385+0000 7fe8e5428740 10 client.8640 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:35:26.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.385+0000 7fe8e5428740 20 client.8640 populate_metadata read hostname 'smithi161' 2024-03-28T23:35:26.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.385+0000 7fe8e5428740 1 --2- 172.21.15.161:0/3740849815 >> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] conn(0x5582be2f0630 0x5582be2f0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:26.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.385+0000 7fe8e5428740 4 client.8640 mds_command: new command op to 6740 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:35:26.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.385+0000 7fe8e5428740 1 -- 172.21.15.161:0/3740849815 --> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582be23f030 con 0x5582be2f0630 2024-03-28T23:35:26.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.385+0000 7fe8dbfff640 1 --2- 172.21.15.161:0/3740849815 >> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] conn(0x5582be2f0630 0x5582be2f0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:26.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.385+0000 7fe8baffd640 10 client.8640 ms_handle_connect on v2:172.21.15.170:6833/3586875977 2024-03-28T23:35:26.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.385+0000 7fe8dbfff640 1 --2- 172.21.15.161:0/3740849815 >> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] conn(0x5582be2f0630 0x5582be2f0a30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=457db6ef6af4bd9c server_cookie=6e6bdcd91d34ea2 in_seq=0 out_seq=0 2024-03-28T23:35:26.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.386+0000 7fe8baffd640 1 -- 172.21.15.161:0/3740849815 <== mds.0 v2:172.21.15.170:6833/3586875977 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5582be23f030 con 0x5582be2f0630 2024-03-28T23:35:26.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.386+0000 7fe8baffd640 10 client.8640 handle_command_reply: tid=0 2024-03-28T23:35:26.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.386+0000 7fe8e5428740 1 client.8640 shutdown 2024-03-28T23:35:26.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.386+0000 7fe8e5428740 1 -- 172.21.15.161:0/3740849815 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2ce190 msgr2=0x5582be2f1620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:26.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.386+0000 7fe8e5428740 1 --2- 172.21.15.161:0/3740849815 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2ce190 0x5582be2f1620 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00095c0 tx=0x7fe8d007a860 comp rx=0 tx=0).stop 2024-03-28T23:35:26.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.386+0000 7fe8e5428740 1 -- 172.21.15.161:0/3740849815 shutdown_connections 2024-03-28T23:35:26.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.386+0000 7fe8e5428740 1 --2- 172.21.15.161:0/3740849815 >> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] conn(0x5582be2f0630 0x5582be2f0a30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.386+0000 7fe8e5428740 1 --2- 172.21.15.161:0/3740849815 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2ce190 0x5582be2f1620 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.386+0000 7fe8e5428740 1 --2- 172.21.15.161:0/3740849815 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5582be2f2ff0 0x5582be2eadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.386+0000 7fe8e5428740 1 --2- 172.21.15.161:0/3740849815 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5582be2cddb0 0x5582be30a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.386+0000 7fe8baffd640 0 client.8640 ms_handle_reset on v2:172.21.15.170:6833/3586875977 2024-03-28T23:35:26.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.386+0000 7fe8e5428740 1 -- 172.21.15.161:0/3740849815 >> 172.21.15.161:0/3740849815 conn(0x5582be2cc310 msgr2=0x5582be2ebb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:26.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.386+0000 7fe8e5428740 1 -- 172.21.15.161:0/3740849815 shutdown_connections 2024-03-28T23:35:26.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.386+0000 7fe8e5428740 1 -- 172.21.15.161:0/3740849815 wait complete. 2024-03-28T23:35:26.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.386+0000 7fe8e5428740 20 client.8640 trim_cache size 0 max 16384 2024-03-28T23:35:26.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.404+0000 7fe8e5428740 1 Processor -- start 2024-03-28T23:35:26.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.404+0000 7fe8e5428740 1 -- start start 2024-03-28T23:35:26.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.405+0000 7fe8e5428740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5582be2cddb0 0x5582be2c9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:26.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.405+0000 7fe8e5428740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2ce190 0x5582be2f0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:26.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.405+0000 7fe8e5428740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5582be2f2ff0 0x5582be2f0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:26.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.405+0000 7fe8e5428740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe8dc1f7d00 con 0x5582be2ce190 2024-03-28T23:35:26.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.405+0000 7fe8e5428740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe8dc1f6d70 con 0x5582be2cddb0 2024-03-28T23:35:26.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.405+0000 7fe8e5428740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe8dc1eeb30 con 0x5582be2f2ff0 2024-03-28T23:35:26.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.405+0000 7fe8db7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2ce190 0x5582be2f0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:26.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.405+0000 7fe8db7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2ce190 0x5582be2f0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58978/0 (socket says 172.21.15.161:58978) 2024-03-28T23:35:26.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.405+0000 7fe8db7fe640 1 -- 172.21.15.161:0/3177179469 learned_addr learned my addr 172.21.15.161:0/3177179469 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:26.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.405+0000 7fe8dbfff640 1 --2- 172.21.15.161:0/3177179469 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5582be2cddb0 0x5582be2c9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:26.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.405+0000 7fe8e0ddd640 1 --2- 172.21.15.161:0/3177179469 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5582be2f2ff0 0x5582be2f0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:26.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.405+0000 7fe8db7fe640 1 -- 172.21.15.161:0/3177179469 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5582be2f2ff0 msgr2=0x5582be2f0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:26.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.405+0000 7fe8db7fe640 1 --2- 172.21.15.161:0/3177179469 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5582be2f2ff0 0x5582be2f0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.405+0000 7fe8db7fe640 1 -- 172.21.15.161:0/3177179469 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5582be2cddb0 msgr2=0x5582be2c9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:26.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.405+0000 7fe8db7fe640 1 --2- 172.21.15.161:0/3177179469 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5582be2cddb0 0x5582be2c9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.405+0000 7fe8db7fe640 1 -- 172.21.15.161:0/3177179469 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582be2c7040 con 0x5582be2ce190 2024-03-28T23:35:26.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.405+0000 7fe8db7fe640 1 --2- 172.21.15.161:0/3177179469 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2ce190 0x5582be2f0a10 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00ac9e0 tx=0x7fe8d008f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0c8e689e80f0b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:26.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.406+0000 7fe8c17fa640 1 -- 172.21.15.161:0/3177179469 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0096040 con 0x5582be2ce190 2024-03-28T23:35:26.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.406+0000 7fe8c17fa640 1 -- 172.21.15.161:0/3177179469 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8d0038040 con 0x5582be2ce190 2024-03-28T23:35:26.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.406+0000 7fe8c17fa640 1 -- 172.21.15.161:0/3177179469 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d007a290 con 0x5582be2ce190 2024-03-28T23:35:26.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.406+0000 7fe8e5428740 1 -- 172.21.15.161:0/3177179469 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2ce190 msgr2=0x5582be2f0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:26.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.406+0000 7fe8e5428740 1 --2- 172.21.15.161:0/3177179469 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2ce190 0x5582be2f0a10 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00ac9e0 tx=0x7fe8d008f000 comp rx=0 tx=0).stop 2024-03-28T23:35:26.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.406+0000 7fe8e5428740 1 -- 172.21.15.161:0/3177179469 shutdown_connections 2024-03-28T23:35:26.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.406+0000 7fe8e5428740 1 --2- 172.21.15.161:0/3177179469 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2ce190 0x5582be2f0a10 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.406+0000 7fe8e5428740 1 --2- 172.21.15.161:0/3177179469 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5582be2f2ff0 0x5582be2f0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.406+0000 7fe8e5428740 1 --2- 172.21.15.161:0/3177179469 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5582be2cddb0 0x5582be2c9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.406+0000 7fe8e5428740 1 -- 172.21.15.161:0/3177179469 >> 172.21.15.161:0/3177179469 conn(0x5582be2cc310 msgr2=0x5582be309e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:26.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.406+0000 7fe8e5428740 1 -- 172.21.15.161:0/3177179469 shutdown_connections 2024-03-28T23:35:26.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.406+0000 7fe8e5428740 1 -- 172.21.15.161:0/3177179469 wait complete. 2024-03-28T23:35:26.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.407+0000 7fe8e5428740 1 -- start start 2024-03-28T23:35:26.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.407+0000 7fe8e5428740 1 Processor -- start 2024-03-28T23:35:26.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.408+0000 7fe8e5428740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2cddb0 0x5582be2cbd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:26.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.408+0000 7fe8e5428740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5582be2ce190 0x5582be2ef120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:26.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.408+0000 7fe8e5428740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5582be2f2ff0 0x5582be3811c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:26.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.408+0000 7fe8e5428740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5582be2f04b0 con 0x5582be2cddb0 2024-03-28T23:35:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.408+0000 7fe8e5428740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe8dc1f6720 con 0x5582be2ce190 2024-03-28T23:35:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.408+0000 7fe8e5428740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x5582be2f4550 con 0x5582be2f2ff0 2024-03-28T23:35:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.408+0000 7fe8e5428740 10 client.0 _ll_register_callbacks cb 0x7fe8d00771b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:35:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.408+0000 7fe8dbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2cddb0 0x5582be2cbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.408+0000 7fe8dbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2cddb0 0x5582be2cbd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58980/0 (socket says 172.21.15.161:58980) 2024-03-28T23:35:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.408+0000 7fe8dbfff640 1 -- 172.21.15.161:0/23858016 learned_addr learned my addr 172.21.15.161:0/23858016 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.408+0000 7fe8e0ddd640 1 --2- 172.21.15.161:0/23858016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5582be2f2ff0 0x5582be3811c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.408+0000 7fe8db7fe640 1 --2- 172.21.15.161:0/23858016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5582be2ce190 0x5582be2ef120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.408+0000 7fe8dbfff640 1 -- 172.21.15.161:0/23858016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5582be2f2ff0 msgr2=0x5582be3811c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.408+0000 7fe8dbfff640 1 --2- 172.21.15.161:0/23858016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5582be2f2ff0 0x5582be3811c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.408+0000 7fe8dbfff640 1 -- 172.21.15.161:0/23858016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5582be2ce190 msgr2=0x5582be2ef120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.408+0000 7fe8dbfff640 1 --2- 172.21.15.161:0/23858016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5582be2ce190 0x5582be2ef120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.408+0000 7fe8dbfff640 1 -- 172.21.15.161:0/23858016 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582be2cb7c0 con 0x5582be2cddb0 2024-03-28T23:35:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.408+0000 7fe8dbfff640 1 --2- 172.21.15.161:0/23858016 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2cddb0 0x5582be2cbd70 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc011e80 tx=0x7fe8cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a506c3842a03c69f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.408+0000 7fe8baffd640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:35:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.409+0000 7fe8baffd640 1 -- 172.21.15.161:0/23858016 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8cc01b040 con 0x5582be2cddb0 2024-03-28T23:35:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.409+0000 7fe8baffd640 1 -- 172.21.15.161:0/23858016 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 6187+0+0 (secure 0 0 0) 0x7fe8cc042cc0 con 0x5582be2cddb0 2024-03-28T23:35:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.409+0000 7fe8e5428740 1 -- 172.21.15.161:0/23858016 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8d0038040 con 0x5582be2cddb0 2024-03-28T23:35:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.409+0000 7fe8baffd640 1 -- 172.21.15.161:0/23858016 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=129) ==== 24+0+0 (secure 0 0 0) 0x7fe8cc049050 con 0x5582be2cddb0 2024-03-28T23:35:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.409+0000 7fe8e5428740 10 client.8646 fetch_fsmap learned FSMap version 129 2024-03-28T23:35:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.409+0000 7fe8e5428740 1 -- 172.21.15.161:0/23858016 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=129,monmap=2+}) -- 0x5582be32d9f0 con 0x5582be2cddb0 2024-03-28T23:35:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.409+0000 7fe8baffd640 1 -- 172.21.15.161:0/23858016 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8d0038040 con 0x5582be2cddb0 2024-03-28T23:35:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.409+0000 7fe8baffd640 1 -- 172.21.15.161:0/23858016 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 129) ==== 3494+0+0 (secure 0 0 0) 0x7fe8cc04c020 con 0x5582be2cddb0 2024-03-28T23:35:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.409+0000 7fe8e5428740 10 client.8646 fetch_fsmap finished waiting for FSMap version 129 2024-03-28T23:35:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.409+0000 7fe8e5428740 10 client.8646 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:35:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.409+0000 7fe8e5428740 20 client.8646 populate_metadata read hostname 'smithi161' 2024-03-28T23:35:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.409+0000 7fe8e5428740 1 --2- 172.21.15.161:0/23858016 >> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] conn(0x5582be2c7220 0x5582be2c7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:26.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.409+0000 7fe8e5428740 4 client.8646 mds_command: new command op to 6740 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:35:26.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.409+0000 7fe8e5428740 1 -- 172.21.15.161:0/23858016 --> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] -- command(tid 0: {"prefix": "status"}) -- 0x5582be239380 con 0x5582be2c7220 2024-03-28T23:35:26.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.409+0000 7fe8db7fe640 1 --2- 172.21.15.161:0/23858016 >> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] conn(0x5582be2c7220 0x5582be2c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:26.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.410+0000 7fe8baffd640 10 client.8646 ms_handle_connect on v2:172.21.15.170:6833/3586875977 2024-03-28T23:35:26.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.410+0000 7fe8db7fe640 1 --2- 172.21.15.161:0/23858016 >> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] conn(0x5582be2c7220 0x5582be2c7600 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27f602a6f078e590 server_cookie=5496412ad6c934cb in_seq=0 out_seq=0 2024-03-28T23:35:26.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.410+0000 7fe8baffd640 1 -- 172.21.15.161:0/23858016 <== mds.0 v2:172.21.15.170:6833/3586875977 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x5582be239380 con 0x5582be2c7220 2024-03-28T23:35:26.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.410+0000 7fe8baffd640 10 client.8646 handle_command_reply: tid=0 2024-03-28T23:35:26.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.410+0000 7fe8e5428740 1 client.8646 shutdown 2024-03-28T23:35:26.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.410+0000 7fe8e5428740 1 -- 172.21.15.161:0/23858016 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2cddb0 msgr2=0x5582be2cbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:26.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.410+0000 7fe8e5428740 1 --2- 172.21.15.161:0/23858016 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2cddb0 0x5582be2cbd70 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc011e80 tx=0x7fe8cc004730 comp rx=0 tx=0).stop 2024-03-28T23:35:26.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.410+0000 7fe8e5428740 1 -- 172.21.15.161:0/23858016 shutdown_connections 2024-03-28T23:35:26.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.410+0000 7fe8e5428740 1 --2- 172.21.15.161:0/23858016 >> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] conn(0x5582be2c7220 0x5582be2c7600 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.410+0000 7fe8e5428740 1 --2- 172.21.15.161:0/23858016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5582be2f2ff0 0x5582be3811c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.410+0000 7fe8e5428740 1 --2- 172.21.15.161:0/23858016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5582be2ce190 0x5582be2ef120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.410+0000 7fe8baffd640 0 client.8646 ms_handle_reset on v2:172.21.15.170:6833/3586875977 2024-03-28T23:35:26.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.410+0000 7fe8e5428740 1 --2- 172.21.15.161:0/23858016 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5582be2cddb0 0x5582be2cbd70 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.410+0000 7fe8e5428740 1 -- 172.21.15.161:0/23858016 >> 172.21.15.161:0/23858016 conn(0x5582be2cc310 msgr2=0x5582be309e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:26.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.410+0000 7fe8e5428740 1 -- 172.21.15.161:0/23858016 shutdown_connections 2024-03-28T23:35:26.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.410+0000 7fe8e5428740 1 -- 172.21.15.161:0/23858016 wait complete. 2024-03-28T23:35:26.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.410+0000 7fe8e5428740 20 client.8646 trim_cache size 0 max 16384 2024-03-28T23:35:26.417 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:35:26.417 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:35:26.417 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:35:26.417 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 6740, 2024-03-28T23:35:26.417 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:35:26.418 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:35:26.418 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:35:26.418 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 5.1242226469999999, 2024-03-28T23:35:26.418 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 129, 2024-03-28T23:35:26.418 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 137, 2024-03-28T23:35:26.418 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 137, 2024-03-28T23:35:26.418 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 17.859586959000001 2024-03-28T23:35:26.418 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:35:26.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.411+0000 7fe8e2840640 1 -- 172.21.15.161:0/3917607564 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe8c806b4d0 msgr2=0x7fe8c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:26.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.411+0000 7fe8e2840640 1 --2- 172.21.15.161:0/3917607564 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe8c806b4d0 0x7fe8c806d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc151980 tx=0x7fe8cc009190 comp rx=0 tx=0).stop 2024-03-28T23:35:26.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.411+0000 7fe8e2840640 1 -- 172.21.15.161:0/3917607564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8dc0c8ed0 msgr2=0x7fe8dc0bf4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:26.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.411+0000 7fe8e2840640 1 --2- 172.21.15.161:0/3917607564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8dc0c8ed0 0x7fe8dc0bf4b0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fe8d002b8a0 tx=0x7fe8d00046c0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.411+0000 7fe8e2840640 1 -- 172.21.15.161:0/3917607564 shutdown_connections 2024-03-28T23:35:26.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.411+0000 7fe8e2840640 1 --2- 172.21.15.161:0/3917607564 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe8c806b4d0 0x7fe8c806d990 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.411+0000 7fe8e2840640 1 --2- 172.21.15.161:0/3917607564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8dc0c8ed0 0x7fe8dc0bf4b0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.411+0000 7fe8e2840640 1 --2- 172.21.15.161:0/3917607564 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8dc152890 0x7fe8dc0bf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.411+0000 7fe8e2840640 1 --2- 172.21.15.161:0/3917607564 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8dc0c5950 0x7fe8dc0c1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:26.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.411+0000 7fe8e2840640 1 -- 172.21.15.161:0/3917607564 >> 172.21.15.161:0/3917607564 conn(0x7fe8dc0ac620 msgr2=0x7fe8dc0c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:26.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.411+0000 7fe8e2840640 1 -- 172.21.15.161:0/3917607564 shutdown_connections 2024-03-28T23:35:26.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:26.411+0000 7fe8e2840640 1 -- 172.21.15.161:0/3917607564 wait complete. 2024-03-28T23:35:26.422 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 129, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:35:12.982566+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 129, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6740": { "addr": "172.21.15.170:6835/3586875977", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 3586875977, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 3586875977, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6740, "incarnation": 126, "join_fscid": 10, "name": "b", "rank": 0, "state": "up:active", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 137, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-03-28T23:35:24.297578+0000", "qdb_cluster": [ 6740 ], "qdb_leader": 6740, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6740 } } } ], "standbys": [ { "addr": "172.21.15.170:6834/3182285172", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 3182285172, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 3182285172, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 125, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7639, "incarnation": 0, "join_fscid": 10, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 4 }, { "addr": "172.21.15.161:6837/3037890643", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6835", "nonce": 3037890643, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 3037890643, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 120, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8409, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6836/819329318", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 819329318, "type": "v2" }, { "addr": "172.21.15.161:6836", "nonce": 819329318, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 127, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8622, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:35:26.423 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 10, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 10, 'name': 'd'}] 2024-03-28T23:35:26.423 INFO:tasks.cephfs.test_failover:current = [{'gid': 8622, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6836/819329318', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 819329318}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 819329318}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 127}, {'gid': 6740, 'name': 'b', 'rank': 0, 'incarnation': 126, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.170:6835/3586875977', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 3586875977}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 3586875977}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 8409, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6837/3037890643', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 3037890643}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 3037890643}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 120}, {'gid': 7639, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 4, 'addr': '172.21.15.170:6834/3182285172', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 3182285172}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 3182285172}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 125}] 2024-03-28T23:35:26.423 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-28T23:35:28.905 INFO:tasks.ceph.mds.b.smithi170.stderr:2024-03-28T23:35:28.904+0000 7ff45e6b8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:35:31.424 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:35:31.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.537+0000 7fbf6bf79640 1 Processor -- start 2024-03-28T23:35:31.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.538+0000 7fbf6bf79640 1 -- start start 2024-03-28T23:35:31.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.538+0000 7fbf6bf79640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbf64152890 0x7fbf640c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:31.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.538+0000 7fbf6bf79640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fbf640c8ed0 0x7fbf640c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:31.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.538+0000 7fbf6bf79640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf64158a30 con 0x7fbf640c5950 2024-03-28T23:35:31.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.539+0000 7fbf6bf79640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fbf640c8480 con 0x7fbf64152890 2024-03-28T23:35:31.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.539+0000 7fbf6bf79640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fbf64159640 con 0x7fbf640c8ed0 2024-03-28T23:35:31.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.539+0000 7fbf69d15640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbf64152890 0x7fbf640c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:31.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.539+0000 7fbf69d15640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbf64152890 0x7fbf640c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42526/0 (socket says 172.21.15.161:42526) 2024-03-28T23:35:31.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.539+0000 7fbf69d15640 1 -- 172.21.15.161:0/2647896284 learned_addr learned my addr 172.21.15.161:0/2647896284 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:31.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.539+0000 7fbf69514640 1 --2- 172.21.15.161:0/2647896284 >> v2:172.21.15.170:3300/0 conn(0x7fbf640c8ed0 0x7fbf640c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:31.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.539+0000 7fbf69d15640 1 -- 172.21.15.161:0/2647896284 >> v2:172.21.15.170:3300/0 conn(0x7fbf640c8ed0 msgr2=0x7fbf640c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:31.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.539+0000 7fbf69d15640 1 --2- 172.21.15.161:0/2647896284 >> v2:172.21.15.170:3300/0 conn(0x7fbf640c8ed0 0x7fbf640c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:31.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.539+0000 7fbf69d15640 1 -- 172.21.15.161:0/2647896284 >> v1:172.21.15.170:6789/0 conn(0x7fbf640c5950 legacy=0x7fbf641553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:31.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.539+0000 7fbf69d15640 1 -- 172.21.15.161:0/2647896284 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf64158d10 con 0x7fbf64152890 2024-03-28T23:35:31.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.540+0000 7fbf69d15640 1 --2- 172.21.15.161:0/2647896284 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf64152890 0x7fbf640c8990 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fbf640ad6e0 tx=0x7fbf5c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87ce8f2bf4fb3d91 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:31.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.540+0000 7fbf68d13640 1 -- 172.21.15.161:0/2647896284 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c02e070 con 0x7fbf64152890 2024-03-28T23:35:31.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.540+0000 7fbf68d13640 1 -- 172.21.15.161:0/2647896284 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf5c037070 con 0x7fbf64152890 2024-03-28T23:35:31.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.540+0000 7fbf68d13640 1 -- 172.21.15.161:0/2647896284 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c031a00 con 0x7fbf64152890 2024-03-28T23:35:31.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.540+0000 7fbf6bf79640 1 -- 172.21.15.161:0/2647896284 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf64152890 msgr2=0x7fbf640c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:31.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.540+0000 7fbf6bf79640 1 --2- 172.21.15.161:0/2647896284 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf64152890 0x7fbf640c8990 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fbf640ad6e0 tx=0x7fbf5c02b880 comp rx=0 tx=0).stop 2024-03-28T23:35:31.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.540+0000 7fbf6bf79640 1 -- 172.21.15.161:0/2647896284 shutdown_connections 2024-03-28T23:35:31.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.540+0000 7fbf6bf79640 1 --2- 172.21.15.161:0/2647896284 >> v2:172.21.15.170:3300/0 conn(0x7fbf640c8ed0 0x7fbf640c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:31.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.540+0000 7fbf6bf79640 1 --2- 172.21.15.161:0/2647896284 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf64152890 0x7fbf640c8990 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:31.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.540+0000 7fbf6bf79640 1 -- 172.21.15.161:0/2647896284 >> 172.21.15.161:0/2647896284 conn(0x7fbf640ac620 msgr2=0x7fbf640aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:31.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.541+0000 7fbf6bf79640 1 -- 172.21.15.161:0/2647896284 shutdown_connections 2024-03-28T23:35:31.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.541+0000 7fbf6bf79640 1 -- 172.21.15.161:0/2647896284 wait complete. 2024-03-28T23:35:31.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.542+0000 7fbf6bf79640 1 Processor -- start 2024-03-28T23:35:31.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.542+0000 7fbf6bf79640 1 -- start start 2024-03-28T23:35:31.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.542+0000 7fbf6bf79640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf640c5950 0x7fbf640c3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:31.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.542+0000 7fbf6bf79640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf640c8ed0 0x7fbf640ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:31.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.542+0000 7fbf6bf79640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf64152890 0x7fbf641491d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:31.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.542+0000 7fbf6bf79640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbf64152c70 con 0x7fbf640c8ed0 2024-03-28T23:35:31.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.542+0000 7fbf6bf79640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbf641592d0 con 0x7fbf64152890 2024-03-28T23:35:31.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.542+0000 7fbf6bf79640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbf64152df0 con 0x7fbf640c5950 2024-03-28T23:35:31.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.542+0000 7fbf69514640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf640c8ed0 0x7fbf640ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:31.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.543+0000 7fbf69514640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf640c8ed0 0x7fbf640ab6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42536/0 (socket says 172.21.15.161:42536) 2024-03-28T23:35:31.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.543+0000 7fbf69514640 1 -- 172.21.15.161:0/3696175962 learned_addr learned my addr 172.21.15.161:0/3696175962 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:31.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.543+0000 7fbf69d15640 1 --2- 172.21.15.161:0/3696175962 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf640c5950 0x7fbf640c3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:31.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.543+0000 7fbf6a516640 1 --2- 172.21.15.161:0/3696175962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf64152890 0x7fbf641491d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:31.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.543+0000 7fbf69514640 1 -- 172.21.15.161:0/3696175962 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf640c5950 msgr2=0x7fbf640c3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:31.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.543+0000 7fbf69514640 1 --2- 172.21.15.161:0/3696175962 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf640c5950 0x7fbf640c3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:31.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.543+0000 7fbf69514640 1 -- 172.21.15.161:0/3696175962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf64152890 msgr2=0x7fbf641491d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:31.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.543+0000 7fbf69514640 1 --2- 172.21.15.161:0/3696175962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf64152890 0x7fbf641491d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:31.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.543+0000 7fbf69514640 1 -- 172.21.15.161:0/3696175962 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf640c0540 con 0x7fbf640c8ed0 2024-03-28T23:35:31.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.543+0000 7fbf69514640 1 --2- 172.21.15.161:0/3696175962 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf640c8ed0 0x7fbf640ab6f0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fbf5400e8d0 tx=0x7fbf5400ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:31.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.543+0000 7fbf5a7fc640 1 -- 172.21.15.161:0/3696175962 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf54017070 con 0x7fbf640c8ed0 2024-03-28T23:35:31.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.543+0000 7fbf5a7fc640 1 -- 172.21.15.161:0/3696175962 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf5400c940 con 0x7fbf640c8ed0 2024-03-28T23:35:31.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.543+0000 7fbf5a7fc640 1 -- 172.21.15.161:0/3696175962 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf5400cd10 con 0x7fbf640c8ed0 2024-03-28T23:35:31.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.543+0000 7fbf6bf79640 1 -- 172.21.15.161:0/3696175962 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf640c4580 con 0x7fbf640c8ed0 2024-03-28T23:35:31.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.543+0000 7fbf6bf79640 1 -- 172.21.15.161:0/3696175962 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf640c4b40 con 0x7fbf640c8ed0 2024-03-28T23:35:31.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.544+0000 7fbf6bf79640 1 -- 172.21.15.161:0/3696175962 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf6404f4b0 con 0x7fbf640c8ed0 2024-03-28T23:35:31.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.545+0000 7fbf5a7fc640 1 -- 172.21.15.161:0/3696175962 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbf5401a070 con 0x7fbf640c8ed0 2024-03-28T23:35:31.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.545+0000 7fbf5a7fc640 1 --2- 172.21.15.161:0/3696175962 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf4806b4d0 0x7fbf4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:31.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.545+0000 7fbf5a7fc640 1 -- 172.21.15.161:0/3696175962 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 6187+0+0 (secure 0 0 0) 0x7fbf5408a630 con 0x7fbf640c8ed0 2024-03-28T23:35:31.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.545+0000 7fbf69d15640 1 --2- 172.21.15.161:0/3696175962 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf4806b4d0 0x7fbf4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:31.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.546+0000 7fbf69d15640 1 --2- 172.21.15.161:0/3696175962 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf4806b4d0 0x7fbf4806d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c036000 tx=0x7fbf5c03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:31.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.550+0000 7fbf5a7fc640 1 -- 172.21.15.161:0/3696175962 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbf54058d10 con 0x7fbf640c8ed0 2024-03-28T23:35:31.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.756+0000 7fbf6bf79640 1 -- 172.21.15.161:0/3696175962 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf6414eac0 con 0x7fbf640c8ed0 2024-03-28T23:35:31.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.757+0000 7fbf5a7fc640 1 -- 172.21.15.161:0/3696175962 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) ==== 95+0+5202 (secure 0 0 0) 0x7fbf5405cdb0 con 0x7fbf640c8ed0 2024-03-28T23:35:31.758 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:31.758 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":129,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7639,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.170:6834/3182285172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3182285172},{"type":"v1","addr":"172.21.15.170:6834","nonce":3182285172}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":125},{"gid":8409,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/3037890643","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3037890643},{"type":"v1","addr":"172.21.15.161:6837","nonce":3037890643}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120},{"gid":8622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/819329318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":819329318},{"type":"v1","addr":"172.21.15.161:6836","nonce":819329318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:12.982566+0000","modified":"2024-03-28T23:35:24.297578+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":137,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":126,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6835/3586875977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3586875977},{"type":"v1","addr":"172.21.15.170:6835","nonce":3586875977}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":10}]} 2024-03-28T23:35:31.758 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 129 2024-03-28T23:35:31.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.759+0000 7fbf6bf79640 1 -- 172.21.15.161:0/3696175962 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf4806b4d0 msgr2=0x7fbf4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:31.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.759+0000 7fbf6bf79640 1 --2- 172.21.15.161:0/3696175962 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf4806b4d0 0x7fbf4806d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c036000 tx=0x7fbf5c03d040 comp rx=0 tx=0).stop 2024-03-28T23:35:31.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.759+0000 7fbf6bf79640 1 -- 172.21.15.161:0/3696175962 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf640c8ed0 msgr2=0x7fbf640ab6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:31.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.759+0000 7fbf6bf79640 1 --2- 172.21.15.161:0/3696175962 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf640c8ed0 0x7fbf640ab6f0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fbf5400e8d0 tx=0x7fbf5400ed90 comp rx=0 tx=0).stop 2024-03-28T23:35:31.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.759+0000 7fbf6bf79640 1 -- 172.21.15.161:0/3696175962 shutdown_connections 2024-03-28T23:35:31.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.759+0000 7fbf6bf79640 1 --2- 172.21.15.161:0/3696175962 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf4806b4d0 0x7fbf4806d990 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:31.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.759+0000 7fbf6bf79640 1 --2- 172.21.15.161:0/3696175962 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf640c8ed0 0x7fbf640ab6f0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:31.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.759+0000 7fbf6bf79640 1 --2- 172.21.15.161:0/3696175962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf64152890 0x7fbf641491d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:31.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.759+0000 7fbf6bf79640 1 --2- 172.21.15.161:0/3696175962 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf640c5950 0x7fbf640c3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:31.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.759+0000 7fbf6bf79640 1 -- 172.21.15.161:0/3696175962 >> 172.21.15.161:0/3696175962 conn(0x7fbf640ac620 msgr2=0x7fbf6414ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:31.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.759+0000 7fbf6bf79640 1 -- 172.21.15.161:0/3696175962 shutdown_connections 2024-03-28T23:35:31.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:31.759+0000 7fbf6bf79640 1 -- 172.21.15.161:0/3696175962 wait complete. 2024-03-28T23:35:32.173 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 129, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:35:12.982566+0000', 'modified': '2024-03-28T23:35:24.297578+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 137, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6740}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6740': {'gid': 6740, 'name': 'b', 'rank': 0, 'incarnation': 126, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.170:6835/3586875977', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 3586875977}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 3586875977}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6740, 'qdb_cluster': [6740]} 2024-03-28T23:35:32.173 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:35:32.173 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T23:35:32.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.278+0000 7f310e647640 1 Processor -- start 2024-03-28T23:35:32.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.278+0000 7f310e647640 1 -- start start 2024-03-28T23:35:32.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.278+0000 7f310e647640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3108150600 0x7f31080a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.278+0000 7f310e647640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f31080a8fc0 0x7f3108153270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.279+0000 7f310e647640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31081567c0 con 0x7f31080aacb0 2024-03-28T23:35:32.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.279+0000 7f310e647640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f3108156570 con 0x7f31080a8fc0 2024-03-28T23:35:32.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.279+0000 7f310e647640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3108156aa0 con 0x7f3108150600 2024-03-28T23:35:32.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.279+0000 7f31077fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f31080a8fc0 0x7f3108153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.279+0000 7f3107fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3108150600 0x7f31080a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.279+0000 7f310cbe4640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f31080aacb0 0x7f31081537b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:47032/0 (socket says 172.21.15.161:47032) 2024-03-28T23:35:32.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.279+0000 7f310cbe4640 1 -- 172.21.15.161:0/2014576026 learned_addr learned my addr 172.21.15.161:0/2014576026 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:32.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.279+0000 7f31077fe640 1 -- 172.21.15.161:0/2014576026 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3108150600 msgr2=0x7f31080a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:32.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.279+0000 7f31077fe640 1 --2- 172.21.15.161:0/2014576026 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3108150600 0x7f31080a9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.279+0000 7f31077fe640 1 -- 172.21.15.161:0/2014576026 >> v1:172.21.15.170:6789/0 conn(0x7f31080aacb0 legacy=0x7f31081537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:32.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.279+0000 7f31077fe640 1 -- 172.21.15.161:0/2014576026 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3108156d80 con 0x7f31080a8fc0 2024-03-28T23:35:32.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.279+0000 7f3106ffd640 1 -- 172.21.15.161:0/2014576026 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3798844233 0 0) 0x7f31081567c0 con 0x7f31080aacb0 2024-03-28T23:35:32.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.279+0000 7f3107fff640 1 --2- 172.21.15.161:0/2014576026 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3108150600 0x7f31080a9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:32.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.280+0000 7f31077fe640 1 --2- 172.21.15.161:0/2014576026 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31080a8fc0 0x7f3108153270 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f30fc0097a0 tx=0x7f30fc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cd67fddf970e596 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:32.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.280+0000 7f3106ffd640 1 -- 172.21.15.161:0/2014576026 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30fc0315a0 con 0x7f31080a8fc0 2024-03-28T23:35:32.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.280+0000 7f3106ffd640 1 -- 172.21.15.161:0/2014576026 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30fc037070 con 0x7f31080a8fc0 2024-03-28T23:35:32.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.280+0000 7f3106ffd640 1 -- 172.21.15.161:0/2014576026 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30fc031960 con 0x7f31080a8fc0 2024-03-28T23:35:32.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.281+0000 7f310e647640 1 -- 172.21.15.161:0/2014576026 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31080a8fc0 msgr2=0x7f3108153270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:32.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.281+0000 7f310e647640 1 --2- 172.21.15.161:0/2014576026 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31080a8fc0 0x7f3108153270 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f30fc0097a0 tx=0x7f30fc02b840 comp rx=0 tx=0).stop 2024-03-28T23:35:32.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.281+0000 7f310e647640 1 -- 172.21.15.161:0/2014576026 shutdown_connections 2024-03-28T23:35:32.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.281+0000 7f310e647640 1 --2- 172.21.15.161:0/2014576026 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31080a8fc0 0x7f3108153270 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.281+0000 7f310e647640 1 --2- 172.21.15.161:0/2014576026 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3108150600 0x7f31080a9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.281+0000 7f310e647640 1 -- 172.21.15.161:0/2014576026 >> 172.21.15.161:0/2014576026 conn(0x7f31080c6670 msgr2=0x7f31080c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:32.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.281+0000 7f310e647640 1 -- 172.21.15.161:0/2014576026 shutdown_connections 2024-03-28T23:35:32.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.281+0000 7f310e647640 1 -- 172.21.15.161:0/2014576026 wait complete. 2024-03-28T23:35:32.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.282+0000 7f310e647640 1 Processor -- start 2024-03-28T23:35:32.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.282+0000 7f310e647640 1 -- start start 2024-03-28T23:35:32.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.283+0000 7f310e647640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31080a8fc0 0x7f31081e6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.283+0000 7f310e647640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31080aacb0 0x7f3108151f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.283+0000 7f310e647640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3108150600 0x7f31081e6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.283+0000 7f310e647640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3108153ec0 con 0x7f31080aacb0 2024-03-28T23:35:32.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.283+0000 7f310e647640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3108157340 con 0x7f31080a8fc0 2024-03-28T23:35:32.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.283+0000 7f310e647640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f31081509e0 con 0x7f3108150600 2024-03-28T23:35:32.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.283+0000 7f31077fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31080aacb0 0x7f3108151f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.283+0000 7f3107fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31080a8fc0 0x7f31081e6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.283+0000 7f31077fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31080aacb0 0x7f3108151f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42540/0 (socket says 172.21.15.161:42540) 2024-03-28T23:35:32.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.283+0000 7f31077fe640 1 -- 172.21.15.161:0/1457041249 learned_addr learned my addr 172.21.15.161:0/1457041249 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:32.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.283+0000 7f310cbe4640 1 --2- 172.21.15.161:0/1457041249 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3108150600 0x7f31081e6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.283+0000 7f3107fff640 1 -- 172.21.15.161:0/1457041249 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3108150600 msgr2=0x7f31081e6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:32.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.283+0000 7f3107fff640 1 --2- 172.21.15.161:0/1457041249 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3108150600 0x7f31081e6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.283+0000 7f3107fff640 1 -- 172.21.15.161:0/1457041249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31080aacb0 msgr2=0x7f3108151f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:32.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.283+0000 7f3107fff640 1 --2- 172.21.15.161:0/1457041249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31080aacb0 0x7f3108151f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.283+0000 7f3107fff640 1 -- 172.21.15.161:0/1457041249 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31081519f0 con 0x7f31080a8fc0 2024-03-28T23:35:32.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.283+0000 7f31077fe640 1 --2- 172.21.15.161:0/1457041249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31080aacb0 0x7f3108151f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:32.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.283+0000 7f3107fff640 1 --2- 172.21.15.161:0/1457041249 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31080a8fc0 0x7f31081e6720 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f30f8012fc0 tx=0x7f30f8010490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:32.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.283+0000 7f3104ff9640 1 -- 172.21.15.161:0/1457041249 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f8010da0 con 0x7f31080a8fc0 2024-03-28T23:35:32.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.283+0000 7f3104ff9640 1 -- 172.21.15.161:0/1457041249 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30f801d070 con 0x7f31080a8fc0 2024-03-28T23:35:32.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.283+0000 7f310e647640 1 -- 172.21.15.161:0/1457041249 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31080c42e0 con 0x7f31080a8fc0 2024-03-28T23:35:32.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.283+0000 7f3104ff9640 1 -- 172.21.15.161:0/1457041249 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f80186c0 con 0x7f31080a8fc0 2024-03-28T23:35:32.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.284+0000 7f310e647640 1 -- 172.21.15.161:0/1457041249 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31080c48a0 con 0x7f31080a8fc0 2024-03-28T23:35:32.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.285+0000 7f3104ff9640 1 -- 172.21.15.161:0/1457041249 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f30f8016050 con 0x7f31080a8fc0 2024-03-28T23:35:32.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.286+0000 7f3104ff9640 1 --2- 172.21.15.161:0/1457041249 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f30f406b500 0x7f30f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.286+0000 7f3104ff9640 1 -- 172.21.15.161:0/1457041249 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 6187+0+0 (secure 0 0 0) 0x7f30f808dfa0 con 0x7f31080a8fc0 2024-03-28T23:35:32.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.286+0000 7f31077fe640 1 --2- 172.21.15.161:0/1457041249 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f30f406b500 0x7f30f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.286+0000 7f31077fe640 1 --2- 172.21.15.161:0/1457041249 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f30f406b500 0x7f30f406d9c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f30fc02b880 tx=0x7f30fc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:32.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.286+0000 7f311122e740 1 Processor -- start 2024-03-28T23:35:32.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.286+0000 7f311122e740 1 -- start start 2024-03-28T23:35:32.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.287+0000 7f311122e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb1160db0 0x55ddb115e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.287+0000 7f311122e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ddb1161190 0x55ddb115edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.287+0000 7f311122e740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ddb115f310 0x55ddb115b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.287+0000 7f311122e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55ddb10c4af0 con 0x55ddb1160db0 2024-03-28T23:35:32.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.287+0000 7f311122e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55ddb106d9d0 con 0x55ddb1161190 2024-03-28T23:35:32.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.287+0000 7f311122e740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55ddb10c0b90 con 0x55ddb115f310 2024-03-28T23:35:32.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.287+0000 7f3107fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb1160db0 0x55ddb115e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.287+0000 7f3107fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb1160db0 0x55ddb115e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42546/0 (socket says 172.21.15.161:42546) 2024-03-28T23:35:32.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.287+0000 7f3107fff640 1 -- 172.21.15.161:0/3580668674 learned_addr learned my addr 172.21.15.161:0/3580668674 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:32.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.287+0000 7f310cbe4640 1 --2- 172.21.15.161:0/3580668674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ddb115f310 0x55ddb115b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.287+0000 7f31077fe640 1 --2- 172.21.15.161:0/3580668674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ddb1161190 0x55ddb115edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.287+0000 7f3107fff640 1 -- 172.21.15.161:0/3580668674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ddb115f310 msgr2=0x55ddb115b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:32.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.287+0000 7f3107fff640 1 --2- 172.21.15.161:0/3580668674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ddb115f310 0x55ddb115b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.287+0000 7f3107fff640 1 -- 172.21.15.161:0/3580668674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ddb1161190 msgr2=0x55ddb115edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:32.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.287+0000 7f3107fff640 1 --2- 172.21.15.161:0/3580668674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ddb1161190 0x55ddb115edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.287+0000 7f3107fff640 1 -- 172.21.15.161:0/3580668674 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddb0dd92d0 con 0x55ddb1160db0 2024-03-28T23:35:32.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.287+0000 7f31077fe640 1 --2- 172.21.15.161:0/3580668674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ddb1161190 0x55ddb115edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:32.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.288+0000 7f3107fff640 1 --2- 172.21.15.161:0/3580668674 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb1160db0 0x55ddb115e890 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f30f8058ae0 tx=0x7f30f800df00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ca47130ed752a32 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:32.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.288+0000 7f30edffb640 1 -- 172.21.15.161:0/3580668674 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f8074070 con 0x55ddb1160db0 2024-03-28T23:35:32.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.288+0000 7f30edffb640 1 -- 172.21.15.161:0/3580668674 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30f8064ce0 con 0x55ddb1160db0 2024-03-28T23:35:32.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.288+0000 7f30edffb640 1 -- 172.21.15.161:0/3580668674 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f800d720 con 0x55ddb1160db0 2024-03-28T23:35:32.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.288+0000 7f311122e740 1 -- 172.21.15.161:0/3580668674 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb1160db0 msgr2=0x55ddb115e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:32.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.288+0000 7f311122e740 1 --2- 172.21.15.161:0/3580668674 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb1160db0 0x55ddb115e890 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f30f8058ae0 tx=0x7f30f800df00 comp rx=0 tx=0).stop 2024-03-28T23:35:32.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.288+0000 7f311122e740 1 -- 172.21.15.161:0/3580668674 shutdown_connections 2024-03-28T23:35:32.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.288+0000 7f311122e740 1 --2- 172.21.15.161:0/3580668674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ddb115f310 0x55ddb115b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.288+0000 7f311122e740 1 --2- 172.21.15.161:0/3580668674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ddb1161190 0x55ddb115edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.288+0000 7f311122e740 1 --2- 172.21.15.161:0/3580668674 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb1160db0 0x55ddb115e890 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.288+0000 7f311122e740 1 -- 172.21.15.161:0/3580668674 >> 172.21.15.161:0/3580668674 conn(0x55ddb115d630 msgr2=0x55ddb1182220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:32.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.288+0000 7f311122e740 1 -- 172.21.15.161:0/3580668674 shutdown_connections 2024-03-28T23:35:32.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.288+0000 7f311122e740 1 -- 172.21.15.161:0/3580668674 wait complete. 2024-03-28T23:35:32.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.289+0000 7f311122e740 1 -- start start 2024-03-28T23:35:32.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.289+0000 7f311122e740 1 Processor -- start 2024-03-28T23:35:32.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.290+0000 7f311122e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ddb1160db0 0x55ddb119d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.290+0000 7f311122e740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ddb1161190 0x55ddb11ad730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.290+0000 7f311122e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb11adc70 0x55ddb117dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.290+0000 7f311122e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55ddb1183090 con 0x55ddb11adc70 2024-03-28T23:35:32.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.290+0000 7f311122e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55ddb1080f70 con 0x55ddb1160db0 2024-03-28T23:35:32.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.290+0000 7f311122e740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55ddb1182ad0 con 0x55ddb1161190 2024-03-28T23:35:32.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.290+0000 7f311122e740 10 client.0 _ll_register_callbacks cb 0x55ddb0dd8ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:35:32.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.290+0000 7f3107fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ddb1160db0 0x55ddb119d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.290+0000 7f310cbe4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb11adc70 0x55ddb117dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.290+0000 7f31077fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ddb1161190 0x55ddb11ad730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.290+0000 7f3107fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ddb1160db0 0x55ddb119d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:35418/0 (socket says 172.21.15.161:35418) 2024-03-28T23:35:32.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.290+0000 7f3107fff640 1 -- 172.21.15.161:0/1674551604 learned_addr learned my addr 172.21.15.161:0/1674551604 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:32.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.290+0000 7f310cbe4640 1 -- 172.21.15.161:0/1674551604 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ddb1161190 msgr2=0x55ddb11ad730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:32.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.290+0000 7f310cbe4640 1 --2- 172.21.15.161:0/1674551604 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ddb1161190 0x55ddb11ad730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.290+0000 7f310cbe4640 1 -- 172.21.15.161:0/1674551604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ddb1160db0 msgr2=0x55ddb119d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:32.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.290+0000 7f310cbe4640 1 --2- 172.21.15.161:0/1674551604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ddb1160db0 0x55ddb119d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.290+0000 7f310cbe4640 1 -- 172.21.15.161:0/1674551604 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddb0ea36b0 con 0x55ddb11adc70 2024-03-28T23:35:32.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.290+0000 7f3107fff640 1 --2- 172.21.15.161:0/1674551604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ddb1160db0 0x55ddb119d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:32.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.290+0000 7f310cbe4640 1 --2- 172.21.15.161:0/1674551604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb11adc70 0x55ddb117dec0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f3100007fd0 tx=0x7f31000077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f93f2ca3601de05 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:32.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.290+0000 7f30e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:35:32.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.291+0000 7f30e6ffd640 1 -- 172.21.15.161:0/1674551604 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31000127f0 con 0x55ddb11adc70 2024-03-28T23:35:32.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.291+0000 7f30e6ffd640 1 -- 172.21.15.161:0/1674551604 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 6187+0+0 (secure 0 0 0) 0x7f3100012990 con 0x55ddb11adc70 2024-03-28T23:35:32.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.291+0000 7f311122e740 1 -- 172.21.15.161:0/1674551604 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30f8064ce0 con 0x55ddb11adc70 2024-03-28T23:35:32.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.291+0000 7f30e6ffd640 1 -- 172.21.15.161:0/1674551604 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=129) ==== 24+0+0 (secure 0 0 0) 0x7f3100022050 con 0x55ddb11adc70 2024-03-28T23:35:32.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.291+0000 7f311122e740 10 client.8661 fetch_fsmap learned FSMap version 129 2024-03-28T23:35:32.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.291+0000 7f311122e740 1 -- 172.21.15.161:0/1674551604 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=129,monmap=2+}) -- 0x55ddb1183460 con 0x55ddb11adc70 2024-03-28T23:35:32.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.291+0000 7f30e6ffd640 1 -- 172.21.15.161:0/1674551604 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30f8064ce0 con 0x55ddb11adc70 2024-03-28T23:35:32.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.291+0000 7f30e6ffd640 1 -- 172.21.15.161:0/1674551604 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 129) ==== 3494+0+0 (secure 0 0 0) 0x7f3100025020 con 0x55ddb11adc70 2024-03-28T23:35:32.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.291+0000 7f311122e740 10 client.8661 fetch_fsmap finished waiting for FSMap version 129 2024-03-28T23:35:32.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.291+0000 7f311122e740 10 client.8661 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:35:32.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.291+0000 7f311122e740 20 client.8661 populate_metadata read hostname 'smithi161' 2024-03-28T23:35:32.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.291+0000 7f311122e740 1 --2- 172.21.15.161:0/1674551604 >> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] conn(0x55ddb11b0be0 0x55ddb11b0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.291+0000 7f311122e740 4 client.8661 mds_command: new command op to 6740 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:35:32.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.291+0000 7f311122e740 1 -- 172.21.15.161:0/1674551604 --> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ddb10d2030 con 0x55ddb11b0be0 2024-03-28T23:35:32.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.291+0000 7f3107fff640 1 --2- 172.21.15.161:0/1674551604 >> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] conn(0x55ddb11b0be0 0x55ddb11b0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.292+0000 7f30e6ffd640 10 client.8661 ms_handle_connect on v2:172.21.15.170:6833/3586875977 2024-03-28T23:35:32.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.292+0000 7f3107fff640 1 --2- 172.21.15.161:0/1674551604 >> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] conn(0x55ddb11b0be0 0x55ddb11b0fc0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=503666c486519b2d server_cookie=1e3c8d10b9b0da95 in_seq=0 out_seq=0 2024-03-28T23:35:32.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.292+0000 7f30e6ffd640 1 -- 172.21.15.161:0/1674551604 <== mds.0 v2:172.21.15.170:6833/3586875977 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ddb10d2030 con 0x55ddb11b0be0 2024-03-28T23:35:32.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.292+0000 7f30e6ffd640 10 client.8661 handle_command_reply: tid=0 2024-03-28T23:35:32.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.292+0000 7f311122e740 1 client.8661 shutdown 2024-03-28T23:35:32.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.292+0000 7f311122e740 1 -- 172.21.15.161:0/1674551604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb11adc70 msgr2=0x55ddb117dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:32.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.292+0000 7f311122e740 1 --2- 172.21.15.161:0/1674551604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb11adc70 0x55ddb117dec0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f3100007fd0 tx=0x7f31000077d0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.292+0000 7f311122e740 1 -- 172.21.15.161:0/1674551604 shutdown_connections 2024-03-28T23:35:32.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.292+0000 7f311122e740 1 --2- 172.21.15.161:0/1674551604 >> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] conn(0x55ddb11b0be0 0x55ddb11b0fc0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.292+0000 7f311122e740 1 --2- 172.21.15.161:0/1674551604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb11adc70 0x55ddb117dec0 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.292+0000 7f311122e740 1 --2- 172.21.15.161:0/1674551604 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ddb1161190 0x55ddb11ad730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.292+0000 7f311122e740 1 --2- 172.21.15.161:0/1674551604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ddb1160db0 0x55ddb119d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.292+0000 7f30e6ffd640 0 client.8661 ms_handle_reset on v2:172.21.15.170:6833/3586875977 2024-03-28T23:35:32.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.292+0000 7f311122e740 1 -- 172.21.15.161:0/1674551604 >> 172.21.15.161:0/1674551604 conn(0x55ddb115f310 msgr2=0x55ddb117ec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:32.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.292+0000 7f311122e740 1 -- 172.21.15.161:0/1674551604 shutdown_connections 2024-03-28T23:35:32.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.292+0000 7f311122e740 1 -- 172.21.15.161:0/1674551604 wait complete. 2024-03-28T23:35:32.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.293+0000 7f311122e740 20 client.8661 trim_cache size 0 max 16384 2024-03-28T23:35:32.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.310+0000 7f311122e740 1 Processor -- start 2024-03-28T23:35:32.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.310+0000 7f311122e740 1 -- start start 2024-03-28T23:35:32.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.311+0000 7f311122e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ddb1160db0 0x55ddb11b7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.311+0000 7f311122e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb1161190 0x55ddb11aa5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.311+0000 7f311122e740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ddb11adc70 0x55ddb11aab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.311+0000 7f311122e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f31081f2a60 con 0x55ddb1161190 2024-03-28T23:35:32.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.311+0000 7f311122e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f31081ea880 con 0x55ddb1160db0 2024-03-28T23:35:32.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.311+0000 7f311122e740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f31081f2410 con 0x55ddb11adc70 2024-03-28T23:35:32.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.311+0000 7f31077fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb1161190 0x55ddb11aa5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.311+0000 7f31077fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb1161190 0x55ddb11aa5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42564/0 (socket says 172.21.15.161:42564) 2024-03-28T23:35:32.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.311+0000 7f31077fe640 1 -- 172.21.15.161:0/4211153483 learned_addr learned my addr 172.21.15.161:0/4211153483 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:32.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.311+0000 7f310cbe4640 1 --2- 172.21.15.161:0/4211153483 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ddb11adc70 0x55ddb11aab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.311+0000 7f3107fff640 1 --2- 172.21.15.161:0/4211153483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ddb1160db0 0x55ddb11b7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.311+0000 7f31077fe640 1 -- 172.21.15.161:0/4211153483 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ddb11adc70 msgr2=0x55ddb11aab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:32.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.311+0000 7f31077fe640 1 --2- 172.21.15.161:0/4211153483 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ddb11adc70 0x55ddb11aab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.311+0000 7f31077fe640 1 -- 172.21.15.161:0/4211153483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ddb1160db0 msgr2=0x55ddb11b7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:32.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.311+0000 7f31077fe640 1 --2- 172.21.15.161:0/4211153483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ddb1160db0 0x55ddb11b7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.311+0000 7f31077fe640 1 -- 172.21.15.161:0/4211153483 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddb115fd60 con 0x55ddb1161190 2024-03-28T23:35:32.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.311+0000 7f31077fe640 1 --2- 172.21.15.161:0/4211153483 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb1161190 0x55ddb11aa5f0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f30fc003610 tx=0x7f30fc031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49e990b6e292cc50 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:32.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.312+0000 7f30ed7fa640 1 -- 172.21.15.161:0/4211153483 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30fc0317a0 con 0x55ddb1161190 2024-03-28T23:35:32.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.312+0000 7f30ed7fa640 1 -- 172.21.15.161:0/4211153483 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30fc037040 con 0x55ddb1161190 2024-03-28T23:35:32.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.312+0000 7f30ed7fa640 1 -- 172.21.15.161:0/4211153483 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30fc031b40 con 0x55ddb1161190 2024-03-28T23:35:32.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.312+0000 7f311122e740 1 -- 172.21.15.161:0/4211153483 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb1161190 msgr2=0x55ddb11aa5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:32.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.312+0000 7f311122e740 1 --2- 172.21.15.161:0/4211153483 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb1161190 0x55ddb11aa5f0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f30fc003610 tx=0x7f30fc031000 comp rx=0 tx=0).stop 2024-03-28T23:35:32.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.312+0000 7f311122e740 1 -- 172.21.15.161:0/4211153483 shutdown_connections 2024-03-28T23:35:32.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.312+0000 7f311122e740 1 --2- 172.21.15.161:0/4211153483 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb1161190 0x55ddb11aa5f0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.312+0000 7f311122e740 1 --2- 172.21.15.161:0/4211153483 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ddb11adc70 0x55ddb11aab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.312+0000 7f311122e740 1 --2- 172.21.15.161:0/4211153483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ddb1160db0 0x55ddb11b7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.312+0000 7f311122e740 1 -- 172.21.15.161:0/4211153483 >> 172.21.15.161:0/4211153483 conn(0x55ddb115f310 msgr2=0x55ddb119f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:32.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.312+0000 7f311122e740 1 -- 172.21.15.161:0/4211153483 shutdown_connections 2024-03-28T23:35:32.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.312+0000 7f311122e740 1 -- 172.21.15.161:0/4211153483 wait complete. 2024-03-28T23:35:32.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.313+0000 7f311122e740 1 -- start start 2024-03-28T23:35:32.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.313+0000 7f311122e740 1 Processor -- start 2024-03-28T23:35:32.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.313+0000 7f311122e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb1160db0 0x55ddb11a99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f311122e740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ddb1161190 0x55ddb115e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f311122e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ddb11adc70 0x55ddb1213b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f311122e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55ddb1188820 con 0x55ddb1160db0 2024-03-28T23:35:32.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f311122e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55ddb11b0a60 con 0x55ddb11adc70 2024-03-28T23:35:32.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f311122e740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55ddb11860d0 con 0x55ddb1161190 2024-03-28T23:35:32.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f311122e740 10 client.0 _ll_register_callbacks cb 0x7f3100002030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:35:32.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f3107fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb1160db0 0x55ddb11a99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f3107fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb1160db0 0x55ddb11a99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42566/0 (socket says 172.21.15.161:42566) 2024-03-28T23:35:32.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f3107fff640 1 -- 172.21.15.161:0/2722338220 learned_addr learned my addr 172.21.15.161:0/2722338220 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:32.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f31077fe640 1 --2- 172.21.15.161:0/2722338220 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ddb1161190 0x55ddb115e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f310cbe4640 1 --2- 172.21.15.161:0/2722338220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ddb11adc70 0x55ddb1213b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f3107fff640 1 -- 172.21.15.161:0/2722338220 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ddb1161190 msgr2=0x55ddb115e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:32.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f3107fff640 1 --2- 172.21.15.161:0/2722338220 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ddb1161190 0x55ddb115e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f3107fff640 1 -- 172.21.15.161:0/2722338220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ddb11adc70 msgr2=0x55ddb1213b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:32.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f3107fff640 1 --2- 172.21.15.161:0/2722338220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ddb11adc70 0x55ddb1213b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f3107fff640 1 -- 172.21.15.161:0/2722338220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddb11ad320 con 0x55ddb1160db0 2024-03-28T23:35:32.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f3107fff640 1 --2- 172.21.15.161:0/2722338220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb1160db0 0x55ddb11a99e0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f30f8058ae0 tx=0x7f30f8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f03c963072799bc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:32.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f30e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:35:32.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f30e6ffd640 1 -- 172.21.15.161:0/2722338220 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f8074040 con 0x55ddb1160db0 2024-03-28T23:35:32.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f30e6ffd640 1 -- 172.21.15.161:0/2722338220 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 6187+0+0 (secure 0 0 0) 0x7f30f8077cf0 con 0x55ddb1160db0 2024-03-28T23:35:32.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f311122e740 1 -- 172.21.15.161:0/2722338220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30fc037040 con 0x55ddb1160db0 2024-03-28T23:35:32.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f30e6ffd640 1 -- 172.21.15.161:0/2722338220 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=129) ==== 24+0+0 (secure 0 0 0) 0x7f30f8064ce0 con 0x55ddb1160db0 2024-03-28T23:35:32.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f311122e740 10 client.8667 fetch_fsmap learned FSMap version 129 2024-03-28T23:35:32.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f311122e740 1 -- 172.21.15.161:0/2722338220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=129,monmap=2+}) -- 0x55ddb1162180 con 0x55ddb1160db0 2024-03-28T23:35:32.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.314+0000 7f30e6ffd640 1 -- 172.21.15.161:0/2722338220 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30fc037040 con 0x55ddb1160db0 2024-03-28T23:35:32.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.315+0000 7f30e6ffd640 1 -- 172.21.15.161:0/2722338220 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 129) ==== 3494+0+0 (secure 0 0 0) 0x7f30f8085ce0 con 0x55ddb1160db0 2024-03-28T23:35:32.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.315+0000 7f311122e740 10 client.8667 fetch_fsmap finished waiting for FSMap version 129 2024-03-28T23:35:32.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.315+0000 7f311122e740 10 client.8667 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:35:32.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.315+0000 7f311122e740 20 client.8667 populate_metadata read hostname 'smithi161' 2024-03-28T23:35:32.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.315+0000 7f311122e740 1 --2- 172.21.15.161:0/2722338220 >> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] conn(0x55ddb11838a0 0x55ddb1183c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.315+0000 7f311122e740 4 client.8667 mds_command: new command op to 6740 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:35:32.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.315+0000 7f311122e740 1 -- 172.21.15.161:0/2722338220 --> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] -- command(tid 0: {"prefix": "status"}) -- 0x55ddb10cc380 con 0x55ddb11838a0 2024-03-28T23:35:32.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.315+0000 7f31077fe640 1 --2- 172.21.15.161:0/2722338220 >> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] conn(0x55ddb11838a0 0x55ddb1183c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.315+0000 7f30e6ffd640 10 client.8667 ms_handle_connect on v2:172.21.15.170:6833/3586875977 2024-03-28T23:35:32.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.315+0000 7f31077fe640 1 --2- 172.21.15.161:0/2722338220 >> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] conn(0x55ddb11838a0 0x55ddb1183c80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=597e4769fd0d4509 server_cookie=5a0c8ed8f4bbb8b1 in_seq=0 out_seq=0 2024-03-28T23:35:32.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.315+0000 7f30e6ffd640 1 -- 172.21.15.161:0/2722338220 <== mds.0 v2:172.21.15.170:6833/3586875977 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x55ddb10cc380 con 0x55ddb11838a0 2024-03-28T23:35:32.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.315+0000 7f30e6ffd640 10 client.8667 handle_command_reply: tid=0 2024-03-28T23:35:32.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.315+0000 7f311122e740 1 client.8667 shutdown 2024-03-28T23:35:32.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.315+0000 7f311122e740 1 -- 172.21.15.161:0/2722338220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb1160db0 msgr2=0x55ddb11a99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:32.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.315+0000 7f311122e740 1 --2- 172.21.15.161:0/2722338220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb1160db0 0x55ddb11a99e0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f30f8058ae0 tx=0x7f30f8000bc0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.315+0000 7f311122e740 1 -- 172.21.15.161:0/2722338220 shutdown_connections 2024-03-28T23:35:32.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.315+0000 7f311122e740 1 --2- 172.21.15.161:0/2722338220 >> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] conn(0x55ddb11838a0 0x55ddb1183c80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.315+0000 7f311122e740 1 --2- 172.21.15.161:0/2722338220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ddb11adc70 0x55ddb1213b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.315+0000 7f311122e740 1 --2- 172.21.15.161:0/2722338220 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ddb1161190 0x55ddb115e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.315+0000 7f30e6ffd640 0 client.8667 ms_handle_reset on v2:172.21.15.170:6833/3586875977 2024-03-28T23:35:32.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.315+0000 7f311122e740 1 --2- 172.21.15.161:0/2722338220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ddb1160db0 0x55ddb11a99e0 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.316+0000 7f311122e740 1 -- 172.21.15.161:0/2722338220 >> 172.21.15.161:0/2722338220 conn(0x55ddb115f310 msgr2=0x55ddb115c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:32.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.316+0000 7f311122e740 1 -- 172.21.15.161:0/2722338220 shutdown_connections 2024-03-28T23:35:32.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.316+0000 7f311122e740 1 -- 172.21.15.161:0/2722338220 wait complete. 2024-03-28T23:35:32.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.316+0000 7f311122e740 20 client.8667 trim_cache size 0 max 16384 2024-03-28T23:35:32.324 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:35:32.325 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:35:32.325 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:35:32.325 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 6740, 2024-03-28T23:35:32.325 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:35:32.325 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:35:32.325 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:35:32.325 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 11.029765611, 2024-03-28T23:35:32.325 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 129, 2024-03-28T23:35:32.325 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 137, 2024-03-28T23:35:32.325 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 137, 2024-03-28T23:35:32.325 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 23.765128440000002 2024-03-28T23:35:32.325 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:35:32.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.316+0000 7f310e647640 1 -- 172.21.15.161:0/1457041249 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f30f406b500 msgr2=0x7f30f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:32.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.316+0000 7f310e647640 1 --2- 172.21.15.161:0/1457041249 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f30f406b500 0x7f30f406d9c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f30fc02b880 tx=0x7f30fc008030 comp rx=0 tx=0).stop 2024-03-28T23:35:32.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.316+0000 7f310e647640 1 -- 172.21.15.161:0/1457041249 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31080a8fc0 msgr2=0x7f31081e6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:32.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.316+0000 7f310e647640 1 --2- 172.21.15.161:0/1457041249 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31080a8fc0 0x7f31081e6720 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f30f8012fc0 tx=0x7f30f8010490 comp rx=0 tx=0).stop 2024-03-28T23:35:32.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.316+0000 7f310e647640 1 -- 172.21.15.161:0/1457041249 shutdown_connections 2024-03-28T23:35:32.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.316+0000 7f310e647640 1 --2- 172.21.15.161:0/1457041249 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f30f406b500 0x7f30f406d9c0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.316+0000 7f310e647640 1 --2- 172.21.15.161:0/1457041249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31080aacb0 0x7f3108151f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.316+0000 7f310e647640 1 --2- 172.21.15.161:0/1457041249 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3108150600 0x7f31081e6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.316+0000 7f310e647640 1 --2- 172.21.15.161:0/1457041249 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31080a8fc0 0x7f31081e6720 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.316+0000 7f310e647640 1 -- 172.21.15.161:0/1457041249 >> 172.21.15.161:0/1457041249 conn(0x7f31080c6670 msgr2=0x7f31080c34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:32.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.316+0000 7f310e647640 1 -- 172.21.15.161:0/1457041249 shutdown_connections 2024-03-28T23:35:32.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.316+0000 7f310e647640 1 -- 172.21.15.161:0/1457041249 wait complete. 2024-03-28T23:35:32.329 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.b mds_join_fs 2024-03-28T23:35:32.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.431+0000 7f600d215640 1 Processor -- start 2024-03-28T23:35:32.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.431+0000 7f600d215640 1 -- start start 2024-03-28T23:35:32.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.432+0000 7f600d215640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f600806cdd0 0x7f60081524c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.432+0000 7f600d215640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f600806ba50 0x7f6008152a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.432+0000 7f600d215640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60080c2f10 con 0x7f600814f5c0 2024-03-28T23:35:32.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.432+0000 7f600d215640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f60080c74d0 con 0x7f600806ba50 2024-03-28T23:35:32.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.432+0000 7f600d215640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f60080c6fa0 con 0x7f600806cdd0 2024-03-28T23:35:32.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.432+0000 7f600659c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f600806cdd0 0x7f60081524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.432+0000 7f600759e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f600806ba50 0x7f6008152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.432+0000 7f600659c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f600806cdd0 0x7f60081524c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:58572/0 (socket says 172.21.15.161:58572) 2024-03-28T23:35:32.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.432+0000 7f600759e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f600806ba50 0x7f6008152a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:35430/0 (socket says 172.21.15.161:35430) 2024-03-28T23:35:32.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.432+0000 7f600659c640 1 -- 172.21.15.161:0/2928000308 learned_addr learned my addr 172.21.15.161:0/2928000308 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:32.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.432+0000 7f600659c640 1 -- 172.21.15.161:0/2928000308 >> v2:172.21.15.170:3300/0 conn(0x7f600806ba50 msgr2=0x7f6008152a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:32.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.432+0000 7f600659c640 1 --2- 172.21.15.161:0/2928000308 >> v2:172.21.15.170:3300/0 conn(0x7f600806ba50 0x7f6008152a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.432+0000 7f600659c640 1 -- 172.21.15.161:0/2928000308 >> v1:172.21.15.170:6789/0 conn(0x7f600814f5c0 legacy=0x7f60080c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:32.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.432+0000 7f600659c640 1 -- 172.21.15.161:0/2928000308 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f600814f9a0 con 0x7f600806cdd0 2024-03-28T23:35:32.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.432+0000 7f600759e640 1 --2- 172.21.15.161:0/2928000308 >> v2:172.21.15.170:3300/0 conn(0x7f600806ba50 0x7f6008152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:32.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.433+0000 7f600659c640 1 --2- 172.21.15.161:0/2928000308 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f600806cdd0 0x7f60081524c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc009870 tx=0x7f5ffc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20e5ad2609ff2407 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:32.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.433+0000 7f6005d9b640 1 -- 172.21.15.161:0/2928000308 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc034070 con 0x7f600806cdd0 2024-03-28T23:35:32.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.434+0000 7f6005d9b640 1 -- 172.21.15.161:0/2928000308 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ffc030840 con 0x7f600806cdd0 2024-03-28T23:35:32.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.434+0000 7f6005d9b640 1 -- 172.21.15.161:0/2928000308 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc030c10 con 0x7f600806cdd0 2024-03-28T23:35:32.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.434+0000 7f600d215640 1 -- 172.21.15.161:0/2928000308 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f600806cdd0 msgr2=0x7f60081524c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:32.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.434+0000 7f600d215640 1 --2- 172.21.15.161:0/2928000308 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f600806cdd0 0x7f60081524c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc009870 tx=0x7f5ffc02b980 comp rx=0 tx=0).stop 2024-03-28T23:35:32.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.434+0000 7f600d215640 1 -- 172.21.15.161:0/2928000308 shutdown_connections 2024-03-28T23:35:32.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.434+0000 7f600d215640 1 --2- 172.21.15.161:0/2928000308 >> v2:172.21.15.170:3300/0 conn(0x7f600806ba50 0x7f6008152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.434+0000 7f600d215640 1 --2- 172.21.15.161:0/2928000308 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f600806cdd0 0x7f60081524c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.434+0000 7f600d215640 1 -- 172.21.15.161:0/2928000308 >> 172.21.15.161:0/2928000308 conn(0x7f60080a83f0 msgr2=0x7f60080a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:32.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.434+0000 7f600d215640 1 -- 172.21.15.161:0/2928000308 shutdown_connections 2024-03-28T23:35:32.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.434+0000 7f600d215640 1 -- 172.21.15.161:0/2928000308 wait complete. 2024-03-28T23:35:32.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.436+0000 7f600d215640 1 Processor -- start 2024-03-28T23:35:32.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.436+0000 7f600d215640 1 -- start start 2024-03-28T23:35:32.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.436+0000 7f600d215640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f600806ba50 0x7f600814b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.436+0000 7f600d215640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f600806cdd0 0x7f6008149ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.436+0000 7f600d215640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f600814f5c0 0x7f6008153e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.436+0000 7f600d215640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f600806c140 con 0x7f600814f5c0 2024-03-28T23:35:32.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.436+0000 7f600d215640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6008152030 con 0x7f600806cdd0 2024-03-28T23:35:32.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.436+0000 7f600d215640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f600806c3e0 con 0x7f600806ba50 2024-03-28T23:35:32.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.437+0000 7f600759e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f600814f5c0 0x7f6008153e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.437+0000 7f6006d9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f600806ba50 0x7f600814b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.437+0000 7f600659c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f600806cdd0 0x7f6008149ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.437+0000 7f600659c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f600806cdd0 0x7f6008149ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:35434/0 (socket says 172.21.15.161:35434) 2024-03-28T23:35:32.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.437+0000 7f600759e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f600814f5c0 0x7f6008153e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42570/0 (socket says 172.21.15.161:42570) 2024-03-28T23:35:32.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.437+0000 7f6006d9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f600806ba50 0x7f600814b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:58578/0 (socket says 172.21.15.161:58578) 2024-03-28T23:35:32.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.437+0000 7f600659c640 1 -- 172.21.15.161:0/1818232668 learned_addr learned my addr 172.21.15.161:0/1818232668 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:32.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.437+0000 7f600659c640 1 -- 172.21.15.161:0/1818232668 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f600806ba50 msgr2=0x7f600814b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:32.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.437+0000 7f600659c640 1 --2- 172.21.15.161:0/1818232668 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f600806ba50 0x7f600814b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.437+0000 7f600659c640 1 -- 172.21.15.161:0/1818232668 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f600814f5c0 msgr2=0x7f6008153e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:32.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.437+0000 7f600659c640 1 --2- 172.21.15.161:0/1818232668 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f600814f5c0 0x7f6008153e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.437+0000 7f600659c640 1 -- 172.21.15.161:0/1818232668 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f600814a7b0 con 0x7f600806cdd0 2024-03-28T23:35:32.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.437+0000 7f600759e640 1 --2- 172.21.15.161:0/1818232668 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f600814f5c0 0x7f6008153e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:32.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.437+0000 7f600659c640 1 --2- 172.21.15.161:0/1818232668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f600806cdd0 0x7f6008149ab0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc02b9c0 tx=0x7f5ffc004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:32.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.437+0000 7f5fef7fe640 1 -- 172.21.15.161:0/1818232668 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc048070 con 0x7f600806cdd0 2024-03-28T23:35:32.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.437+0000 7f5fef7fe640 1 -- 172.21.15.161:0/1818232668 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ffc030640 con 0x7f600806cdd0 2024-03-28T23:35:32.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.437+0000 7f600d215640 1 -- 172.21.15.161:0/1818232668 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6008148ba0 con 0x7f600806cdd0 2024-03-28T23:35:32.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.437+0000 7f5fef7fe640 1 -- 172.21.15.161:0/1818232668 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc034040 con 0x7f600806cdd0 2024-03-28T23:35:32.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.437+0000 7f600d215640 1 -- 172.21.15.161:0/1818232668 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6008149ff0 con 0x7f600806cdd0 2024-03-28T23:35:32.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.438+0000 7f600d215640 1 -- 172.21.15.161:0/1818232668 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60080ac9e0 con 0x7f600806cdd0 2024-03-28T23:35:32.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.439+0000 7f5fef7fe640 1 -- 172.21.15.161:0/1818232668 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5ffc04b070 con 0x7f600806cdd0 2024-03-28T23:35:32.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.439+0000 7f5fef7fe640 1 --2- 172.21.15.161:0/1818232668 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5ff406b4d0 0x7f5ff406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.439+0000 7f5fef7fe640 1 -- 172.21.15.161:0/1818232668 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 6187+0+0 (secure 0 0 0) 0x7f5ffc0ac170 con 0x7f600806cdd0 2024-03-28T23:35:32.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.439+0000 7f6006d9d640 1 --2- 172.21.15.161:0/1818232668 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5ff406b4d0 0x7f5ff406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.440+0000 7f6006d9d640 1 --2- 172.21.15.161:0/1818232668 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5ff406b4d0 0x7f5ff406d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8001e70 tx=0x7f5ff8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:32.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.446+0000 7f5fef7fe640 1 -- 172.21.15.161:0/1818232668 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f60080ac9e0 con 0x7f600806cdd0 2024-03-28T23:35:32.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.618+0000 7f600d215640 1 -- 172.21.15.161:0/1818232668 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"} v 0) -- 0x7f6008068f90 con 0x7f600806cdd0 2024-03-28T23:35:32.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.622+0000 7f5fef7fe640 1 -- 172.21.15.161:0/1818232668 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"}]=0 v48) ==== 96+0+0 (secure 0 0 0) 0x7f6008068f90 con 0x7f600806cdd0 2024-03-28T23:35:32.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.624+0000 7f600d215640 1 -- 172.21.15.161:0/1818232668 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5ff406b4d0 msgr2=0x7f5ff406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:32.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.624+0000 7f600d215640 1 --2- 172.21.15.161:0/1818232668 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5ff406b4d0 0x7f5ff406d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8001e70 tx=0x7f5ff8009040 comp rx=0 tx=0).stop 2024-03-28T23:35:32.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.624+0000 7f600d215640 1 -- 172.21.15.161:0/1818232668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f600806cdd0 msgr2=0x7f6008149ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:32.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.624+0000 7f600d215640 1 --2- 172.21.15.161:0/1818232668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f600806cdd0 0x7f6008149ab0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc02b9c0 tx=0x7f5ffc004730 comp rx=0 tx=0).stop 2024-03-28T23:35:32.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.624+0000 7f600d215640 1 -- 172.21.15.161:0/1818232668 shutdown_connections 2024-03-28T23:35:32.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.624+0000 7f600d215640 1 --2- 172.21.15.161:0/1818232668 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5ff406b4d0 0x7f5ff406d990 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.624+0000 7f600d215640 1 --2- 172.21.15.161:0/1818232668 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f600814f5c0 0x7f6008153e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.624+0000 7f600d215640 1 --2- 172.21.15.161:0/1818232668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f600806cdd0 0x7f6008149ab0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.624+0000 7f600d215640 1 --2- 172.21.15.161:0/1818232668 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f600806ba50 0x7f600814b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.624+0000 7f600d215640 1 -- 172.21.15.161:0/1818232668 >> 172.21.15.161:0/1818232668 conn(0x7f60080a83f0 msgr2=0x7f60080c6220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:32.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.624+0000 7f600d215640 1 -- 172.21.15.161:0/1818232668 shutdown_connections 2024-03-28T23:35:32.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.624+0000 7f600d215640 1 -- 172.21.15.161:0/1818232668 wait complete. 2024-03-28T23:35:32.635 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:35:32.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.735+0000 7f41d1e6b640 1 Processor -- start 2024-03-28T23:35:32.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.735+0000 7f41d1e6b640 1 -- start start 2024-03-28T23:35:32.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.736+0000 7f41d1e6b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f41cc14be70 0x7f41cc0ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.736+0000 7f41d1e6b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41cc0ab6e0 0x7f41cc0aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.736+0000 7f41d1e6b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41cc14c250 con 0x7f41cc06ce80 2024-03-28T23:35:32.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.736+0000 7f41d1e6b640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f41cc1529c0 con 0x7f41cc14be70 2024-03-28T23:35:32.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.736+0000 7f41d1e6b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f41cc0a9dc0 con 0x7f41cc0ab6e0 2024-03-28T23:35:32.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.736+0000 7f41cb7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f41cc14be70 0x7f41cc0ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.736+0000 7f41cb7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f41cc14be70 0x7f41cc0ab1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42574/0 (socket says 172.21.15.161:42574) 2024-03-28T23:35:32.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.736+0000 7f41cb7fe640 1 -- 172.21.15.161:0/971940190 learned_addr learned my addr 172.21.15.161:0/971940190 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:32.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.736+0000 7f41c2ffd640 1 --2- 172.21.15.161:0/971940190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41cc0ab6e0 0x7f41cc0aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.737+0000 7f41cb7fe640 1 -- 172.21.15.161:0/971940190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41cc0ab6e0 msgr2=0x7f41cc0aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:32.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.737+0000 7f41cb7fe640 1 --2- 172.21.15.161:0/971940190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41cc0ab6e0 0x7f41cc0aa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.737+0000 7f41cb7fe640 1 -- 172.21.15.161:0/971940190 >> v1:172.21.15.170:6789/0 conn(0x7f41cc06ce80 legacy=0x7f41cc1553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:32.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.737+0000 7f41cb7fe640 1 -- 172.21.15.161:0/971940190 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41cc0aa0a0 con 0x7f41cc14be70 2024-03-28T23:35:32.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.737+0000 7f41cb7fe640 1 --2- 172.21.15.161:0/971940190 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41cc14be70 0x7f41cc0ab1a0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f41bc009870 tx=0x7f41bc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be3f7235f5cb75f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:32.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.738+0000 7f41caffd640 1 -- 172.21.15.161:0/971940190 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc035020 con 0x7f41cc14be70 2024-03-28T23:35:32.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.738+0000 7f41caffd640 1 -- 172.21.15.161:0/971940190 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41bc031640 con 0x7f41cc14be70 2024-03-28T23:35:32.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.738+0000 7f41caffd640 1 -- 172.21.15.161:0/971940190 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc02d040 con 0x7f41cc14be70 2024-03-28T23:35:32.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.738+0000 7f41d1e6b640 1 -- 172.21.15.161:0/971940190 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41cc14be70 msgr2=0x7f41cc0ab1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:32.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.738+0000 7f41d1e6b640 1 --2- 172.21.15.161:0/971940190 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41cc14be70 0x7f41cc0ab1a0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f41bc009870 tx=0x7f41bc02b8f0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.738+0000 7f41d1e6b640 1 -- 172.21.15.161:0/971940190 shutdown_connections 2024-03-28T23:35:32.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.738+0000 7f41d1e6b640 1 --2- 172.21.15.161:0/971940190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41cc0ab6e0 0x7f41cc0aa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.738+0000 7f41d1e6b640 1 --2- 172.21.15.161:0/971940190 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41cc14be70 0x7f41cc0ab1a0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.738+0000 7f41d1e6b640 1 -- 172.21.15.161:0/971940190 >> 172.21.15.161:0/971940190 conn(0x7f41cc0c6670 msgr2=0x7f41cc0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:32.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.738+0000 7f41d1e6b640 1 -- 172.21.15.161:0/971940190 shutdown_connections 2024-03-28T23:35:32.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.738+0000 7f41d1e6b640 1 -- 172.21.15.161:0/971940190 wait complete. 2024-03-28T23:35:32.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.739+0000 7f41d1e6b640 1 Processor -- start 2024-03-28T23:35:32.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.740+0000 7f41d1e6b640 1 -- start start 2024-03-28T23:35:32.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.740+0000 7f41d1e6b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41cc06ce80 0x7f41cc155d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.740+0000 7f41d1e6b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41cc0ab6e0 0x7f41cc154a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.740+0000 7f41d1e6b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41cc14be70 0x7f41cc0c5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.740+0000 7f41d1e6b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc06b2d0 con 0x7f41cc14be70 2024-03-28T23:35:32.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.740+0000 7f41d1e6b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc06b150 con 0x7f41cc0ab6e0 2024-03-28T23:35:32.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.740+0000 7f41d1e6b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f41cc0c1520 con 0x7f41cc06ce80 2024-03-28T23:35:32.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.740+0000 7f41cbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41cc14be70 0x7f41cc0c5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.740+0000 7f41cbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41cc14be70 0x7f41cc0c5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42584/0 (socket says 172.21.15.161:42584) 2024-03-28T23:35:32.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.740+0000 7f41cbfff640 1 -- 172.21.15.161:0/3572653646 learned_addr learned my addr 172.21.15.161:0/3572653646 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:32.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.741+0000 7f41c2ffd640 1 --2- 172.21.15.161:0/3572653646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41cc0ab6e0 0x7f41cc154a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.741+0000 7f41cb7fe640 1 --2- 172.21.15.161:0/3572653646 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41cc06ce80 0x7f41cc155d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.741+0000 7f41cbfff640 1 -- 172.21.15.161:0/3572653646 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41cc06ce80 msgr2=0x7f41cc155d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:32.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.741+0000 7f41cbfff640 1 --2- 172.21.15.161:0/3572653646 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41cc06ce80 0x7f41cc155d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.741+0000 7f41cbfff640 1 -- 172.21.15.161:0/3572653646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41cc0ab6e0 msgr2=0x7f41cc154a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:32.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.741+0000 7f41cbfff640 1 --2- 172.21.15.161:0/3572653646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41cc0ab6e0 0x7f41cc154a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.741+0000 7f41cbfff640 1 -- 172.21.15.161:0/3572653646 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41cc1562a0 con 0x7f41cc14be70 2024-03-28T23:35:32.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.741+0000 7f41cbfff640 1 --2- 172.21.15.161:0/3572653646 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41cc14be70 0x7f41cc0c5820 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f41c400c6b0 tx=0x7f41c400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:32.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.741+0000 7f41c8ff9640 1 -- 172.21.15.161:0/3572653646 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c4003b70 con 0x7f41cc14be70 2024-03-28T23:35:32.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.741+0000 7f41c8ff9640 1 -- 172.21.15.161:0/3572653646 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41c4003d30 con 0x7f41cc14be70 2024-03-28T23:35:32.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.741+0000 7f41c8ff9640 1 -- 172.21.15.161:0/3572653646 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c4005310 con 0x7f41cc14be70 2024-03-28T23:35:32.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.741+0000 7f41d1e6b640 1 -- 172.21.15.161:0/3572653646 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41cc1e9b50 con 0x7f41cc14be70 2024-03-28T23:35:32.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.741+0000 7f41d1e6b640 1 -- 172.21.15.161:0/3572653646 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41cc1ea110 con 0x7f41cc14be70 2024-03-28T23:35:32.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.742+0000 7f41d1e6b640 1 -- 172.21.15.161:0/3572653646 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41cc04f4b0 con 0x7f41cc14be70 2024-03-28T23:35:32.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.742+0000 7f41c8ff9640 1 -- 172.21.15.161:0/3572653646 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f41c40054b0 con 0x7f41cc14be70 2024-03-28T23:35:32.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.743+0000 7f41c8ff9640 1 --2- 172.21.15.161:0/3572653646 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f41b006b500 0x7f41b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:32.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.743+0000 7f41c8ff9640 1 -- 172.21.15.161:0/3572653646 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 6187+0+0 (secure 0 0 0) 0x7f41c408a010 con 0x7f41cc14be70 2024-03-28T23:35:32.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.743+0000 7f41cb7fe640 1 --2- 172.21.15.161:0/3572653646 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f41b006b500 0x7f41b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:32.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.743+0000 7f41cb7fe640 1 --2- 172.21.15.161:0/3572653646 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f41b006b500 0x7f41b006d9c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f41bc036000 tx=0x7f41bc03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:32.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.747+0000 7f41c8ff9640 1 -- 172.21.15.161:0/3572653646 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f41c40586f0 con 0x7f41cc14be70 2024-03-28T23:35:32.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.954+0000 7f41d1e6b640 1 -- 172.21.15.161:0/3572653646 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41cc0c0ff0 con 0x7f41cc14be70 2024-03-28T23:35:32.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.955+0000 7f41c8ff9640 1 -- 172.21.15.161:0/3572653646 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) ==== 95+0+5202 (secure 0 0 0) 0x7f41c405c790 con 0x7f41cc14be70 2024-03-28T23:35:32.955 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:32.955 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":129,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7639,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.170:6834/3182285172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3182285172},{"type":"v1","addr":"172.21.15.170:6834","nonce":3182285172}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":125},{"gid":8409,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/3037890643","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3037890643},{"type":"v1","addr":"172.21.15.161:6837","nonce":3037890643}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120},{"gid":8622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/819329318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":819329318},{"type":"v1","addr":"172.21.15.161:6836","nonce":819329318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:12.982566+0000","modified":"2024-03-28T23:35:24.297578+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":137,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":126,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6835/3586875977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3586875977},{"type":"v1","addr":"172.21.15.170:6835","nonce":3586875977}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":10}]} 2024-03-28T23:35:32.956 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 129 2024-03-28T23:35:32.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.956+0000 7f41d1e6b640 1 -- 172.21.15.161:0/3572653646 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f41b006b500 msgr2=0x7f41b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:32.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.956+0000 7f41d1e6b640 1 --2- 172.21.15.161:0/3572653646 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f41b006b500 0x7f41b006d9c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f41bc036000 tx=0x7f41bc03c040 comp rx=0 tx=0).stop 2024-03-28T23:35:32.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.956+0000 7f41d1e6b640 1 -- 172.21.15.161:0/3572653646 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41cc14be70 msgr2=0x7f41cc0c5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:32.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.956+0000 7f41d1e6b640 1 --2- 172.21.15.161:0/3572653646 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41cc14be70 0x7f41cc0c5820 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f41c400c6b0 tx=0x7f41c400cb70 comp rx=0 tx=0).stop 2024-03-28T23:35:32.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.957+0000 7f41d1e6b640 1 -- 172.21.15.161:0/3572653646 shutdown_connections 2024-03-28T23:35:32.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.957+0000 7f41d1e6b640 1 --2- 172.21.15.161:0/3572653646 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f41b006b500 0x7f41b006d9c0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.957+0000 7f41d1e6b640 1 --2- 172.21.15.161:0/3572653646 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41cc14be70 0x7f41cc0c5820 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.957+0000 7f41d1e6b640 1 --2- 172.21.15.161:0/3572653646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41cc0ab6e0 0x7f41cc154a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.957+0000 7f41d1e6b640 1 --2- 172.21.15.161:0/3572653646 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41cc06ce80 0x7f41cc155d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:32.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.957+0000 7f41d1e6b640 1 -- 172.21.15.161:0/3572653646 >> 172.21.15.161:0/3572653646 conn(0x7f41cc0c6670 msgr2=0x7f41cc06b480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:32.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.957+0000 7f41d1e6b640 1 -- 172.21.15.161:0/3572653646 shutdown_connections 2024-03-28T23:35:32.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:32.957+0000 7f41d1e6b640 1 -- 172.21.15.161:0/3572653646 wait complete. 2024-03-28T23:35:33.371 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 129, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:35:12.982566+0000', 'modified': '2024-03-28T23:35:24.297578+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 137, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6740}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6740': {'gid': 6740, 'name': 'b', 'rank': 0, 'incarnation': 126, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.170:6835/3586875977', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 3586875977}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 3586875977}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6740, 'qdb_cluster': [6740]} 2024-03-28T23:35:33.371 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:35:33.372 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T23:35:33.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.480+0000 7f67c1ba1640 1 Processor -- start 2024-03-28T23:35:33.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.480+0000 7f67c1ba1640 1 -- start start 2024-03-28T23:35:33.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.481+0000 7f67c1ba1640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f67bc0c5410 0x7f67bc0c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:33.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.481+0000 7f67c1ba1640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67bc158a30 con 0x7f67bc152890 2024-03-28T23:35:33.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.481+0000 7f67c1ba1640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67bc158d10 con 0x7f67bc1553a0 2024-03-28T23:35:33.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.481+0000 7f67c1ba1640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f67bc0c8480 con 0x7f67bc0c5410 2024-03-28T23:35:33.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.481+0000 7f67bb7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f67bc152890 0x7f67bc0c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:54476/0 (socket says 172.21.15.161:54476) 2024-03-28T23:35:33.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.481+0000 7f67bb7fe640 1 -- 172.21.15.161:0/45670842 learned_addr learned my addr 172.21.15.161:0/45670842 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:33.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.481+0000 7f67b2ffd640 1 --2- 172.21.15.161:0/45670842 >> v2:172.21.15.170:3300/0 conn(0x7f67bc0c5410 0x7f67bc0c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:33.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.482+0000 7f67baffd640 1 -- 172.21.15.161:0/45670842 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2418059872 0 0) 0x7f67bc158a30 con 0x7f67bc152890 2024-03-28T23:35:33.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.482+0000 7f67baffd640 1 -- 172.21.15.161:0/45670842 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67a8003520 con 0x7f67bc152890 2024-03-28T23:35:33.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.482+0000 7f67b2ffd640 1 -- 172.21.15.161:0/45670842 >> v1:172.21.15.170:6789/0 conn(0x7f67bc1553a0 legacy=0x7f67bc155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:33.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.482+0000 7f67b2ffd640 1 -- 172.21.15.161:0/45670842 >> v1:172.21.15.161:6789/0 conn(0x7f67bc152890 legacy=0x7f67bc0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:33.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.482+0000 7f67b2ffd640 1 -- 172.21.15.161:0/45670842 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67bc158ff0 con 0x7f67bc0c5410 2024-03-28T23:35:33.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.483+0000 7f67b2ffd640 1 --2- 172.21.15.161:0/45670842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67bc0c5410 0x7f67bc0c5850 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f67a4009790 tx=0x7f67a402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62b8efda2c339086 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:33.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.483+0000 7f67baffd640 1 -- 172.21.15.161:0/45670842 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a4035020 con 0x7f67bc0c5410 2024-03-28T23:35:33.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.483+0000 7f67baffd640 1 -- 172.21.15.161:0/45670842 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67a4031830 con 0x7f67bc0c5410 2024-03-28T23:35:33.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.483+0000 7f67baffd640 1 -- 172.21.15.161:0/45670842 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a4031be0 con 0x7f67bc0c5410 2024-03-28T23:35:33.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.483+0000 7f67c1ba1640 1 -- 172.21.15.161:0/45670842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67bc0c5410 msgr2=0x7f67bc0c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:33.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.483+0000 7f67c1ba1640 1 --2- 172.21.15.161:0/45670842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67bc0c5410 0x7f67bc0c5850 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f67a4009790 tx=0x7f67a402b8d0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.483+0000 7f67c1ba1640 1 -- 172.21.15.161:0/45670842 shutdown_connections 2024-03-28T23:35:33.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.483+0000 7f67c1ba1640 1 --2- 172.21.15.161:0/45670842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67bc0c5410 0x7f67bc0c5850 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.483+0000 7f67c1ba1640 1 -- 172.21.15.161:0/45670842 >> 172.21.15.161:0/45670842 conn(0x7f67bc0ac620 msgr2=0x7f67bc0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:33.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.483+0000 7f67c1ba1640 1 -- 172.21.15.161:0/45670842 shutdown_connections 2024-03-28T23:35:33.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.483+0000 7f67c1ba1640 1 -- 172.21.15.161:0/45670842 wait complete. 2024-03-28T23:35:33.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.484+0000 7f67c1ba1640 1 Processor -- start 2024-03-28T23:35:33.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.485+0000 7f67c1ba1640 1 -- start start 2024-03-28T23:35:33.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.485+0000 7f67c1ba1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67bc0c5410 0x7f67bc1500b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:33.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.485+0000 7f67c1ba1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67bc152890 0x7f67bc14e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:33.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.485+0000 7f67c1ba1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67bc1553a0 0x7f67bc145800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:33.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.485+0000 7f67c1ba1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc159640 con 0x7f67bc1553a0 2024-03-28T23:35:33.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.485+0000 7f67c1ba1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc0c5d90 con 0x7f67bc152890 2024-03-28T23:35:33.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.485+0000 7f67c1ba1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f67bc0c90a0 con 0x7f67bc0c5410 2024-03-28T23:35:33.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.485+0000 7f67bbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67bc1553a0 0x7f67bc145800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:33.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.485+0000 7f67bbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67bc1553a0 0x7f67bc145800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42592/0 (socket says 172.21.15.161:42592) 2024-03-28T23:35:33.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.485+0000 7f67bbfff640 1 -- 172.21.15.161:0/4292517789 learned_addr learned my addr 172.21.15.161:0/4292517789 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:33.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.485+0000 7f67b2ffd640 1 --2- 172.21.15.161:0/4292517789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67bc152890 0x7f67bc14e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:33.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.485+0000 7f67bb7fe640 1 --2- 172.21.15.161:0/4292517789 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67bc0c5410 0x7f67bc1500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:33.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.486+0000 7f67b2ffd640 1 -- 172.21.15.161:0/4292517789 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67bc0c5410 msgr2=0x7f67bc1500b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:33.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.486+0000 7f67b2ffd640 1 --2- 172.21.15.161:0/4292517789 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67bc0c5410 0x7f67bc1500b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.486+0000 7f67b2ffd640 1 -- 172.21.15.161:0/4292517789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67bc1553a0 msgr2=0x7f67bc145800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:33.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.486+0000 7f67b2ffd640 1 --2- 172.21.15.161:0/4292517789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67bc1553a0 0x7f67bc145800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.486+0000 7f67b2ffd640 1 -- 172.21.15.161:0/4292517789 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67bc1517b0 con 0x7f67bc152890 2024-03-28T23:35:33.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.486+0000 7f67bbfff640 1 --2- 172.21.15.161:0/4292517789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67bc1553a0 0x7f67bc145800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:33.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.486+0000 7f67b2ffd640 1 --2- 172.21.15.161:0/4292517789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67bc152890 0x7f67bc14e090 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f67a4033000 tx=0x7f67a40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:33.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.486+0000 7f67b8ff9640 1 -- 172.21.15.161:0/4292517789 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a4035040 con 0x7f67bc152890 2024-03-28T23:35:33.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.486+0000 7f67b8ff9640 1 -- 172.21.15.161:0/4292517789 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67a4004230 con 0x7f67bc152890 2024-03-28T23:35:33.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.486+0000 7f67c1ba1640 1 -- 172.21.15.161:0/4292517789 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67bc1514d0 con 0x7f67bc152890 2024-03-28T23:35:33.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.486+0000 7f67b8ff9640 1 -- 172.21.15.161:0/4292517789 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a4005990 con 0x7f67bc152890 2024-03-28T23:35:33.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.486+0000 7f67c1ba1640 1 -- 172.21.15.161:0/4292517789 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67bc1eaff0 con 0x7f67bc152890 2024-03-28T23:35:33.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.487+0000 7f67b8ff9640 1 -- 172.21.15.161:0/4292517789 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f67a4038020 con 0x7f67bc152890 2024-03-28T23:35:33.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.488+0000 7f67b8ff9640 1 --2- 172.21.15.161:0/4292517789 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f679c06b500 0x7f679c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:33.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.488+0000 7f67b8ff9640 1 -- 172.21.15.161:0/4292517789 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 6187+0+0 (secure 0 0 0) 0x7f67a40abce0 con 0x7f67bc152890 2024-03-28T23:35:33.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.488+0000 7f67bb7fe640 1 --2- 172.21.15.161:0/4292517789 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f679c06b500 0x7f679c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:33.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.488+0000 7f67bb7fe640 1 --2- 172.21.15.161:0/4292517789 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f679c06b500 0x7f679c06d9c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f67ac0011e0 tx=0x7f67ac007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:33.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.489+0000 7f67c4786740 1 Processor -- start 2024-03-28T23:35:33.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.489+0000 7f67c4786740 1 -- start start 2024-03-28T23:35:33.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.490+0000 7f67c4786740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e836737db0 0x55e836735890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:33.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.490+0000 7f67c4786740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e836738190 0x55e836735dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:33.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.490+0000 7f67c4786740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55e836736310 0x55e8367324e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:33.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.490+0000 7f67c4786740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55e83669baf0 con 0x55e836738190 2024-03-28T23:35:33.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.490+0000 7f67c4786740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55e8366449d0 con 0x55e836737db0 2024-03-28T23:35:33.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.490+0000 7f67c4786740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55e836697b90 con 0x55e836736310 2024-03-28T23:35:33.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.490+0000 7f67b2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e836738190 0x55e836735dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:33.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.490+0000 7f67b2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e836738190 0x55e836735dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42594/0 (socket says 172.21.15.161:42594) 2024-03-28T23:35:33.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.490+0000 7f67b2ffd640 1 -- 172.21.15.161:0/2487276457 learned_addr learned my addr 172.21.15.161:0/2487276457 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:33.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.490+0000 7f67bb7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e836737db0 0x55e836735890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:33.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.490+0000 7f67bbfff640 1 --2- 172.21.15.161:0/2487276457 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55e836736310 0x55e8367324e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:33.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.490+0000 7f67b2ffd640 1 -- 172.21.15.161:0/2487276457 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55e836736310 msgr2=0x55e8367324e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:33.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.490+0000 7f67b2ffd640 1 --2- 172.21.15.161:0/2487276457 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55e836736310 0x55e8367324e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.490+0000 7f67b2ffd640 1 -- 172.21.15.161:0/2487276457 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e836737db0 msgr2=0x55e836735890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:33.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.490+0000 7f67b2ffd640 1 --2- 172.21.15.161:0/2487276457 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e836737db0 0x55e836735890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.490+0000 7f67b2ffd640 1 -- 172.21.15.161:0/2487276457 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8363b02d0 con 0x55e836738190 2024-03-28T23:35:33.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.490+0000 7f67b2ffd640 1 --2- 172.21.15.161:0/2487276457 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e836738190 0x55e836735dd0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f67a4082470 tx=0x7f67a4009110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ecd92e47999c193 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:33.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.490+0000 7f67b17fa640 1 -- 172.21.15.161:0/2487276457 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a404a070 con 0x55e836738190 2024-03-28T23:35:33.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.490+0000 7f67b17fa640 1 -- 172.21.15.161:0/2487276457 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67a407e3e0 con 0x55e836738190 2024-03-28T23:35:33.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.490+0000 7f67b17fa640 1 -- 172.21.15.161:0/2487276457 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a40829a0 con 0x55e836738190 2024-03-28T23:35:33.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.491+0000 7f67c4786740 1 -- 172.21.15.161:0/2487276457 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e836738190 msgr2=0x55e836735dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:33.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.491+0000 7f67c4786740 1 --2- 172.21.15.161:0/2487276457 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e836738190 0x55e836735dd0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f67a4082470 tx=0x7f67a4009110 comp rx=0 tx=0).stop 2024-03-28T23:35:33.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.491+0000 7f67c4786740 1 -- 172.21.15.161:0/2487276457 shutdown_connections 2024-03-28T23:35:33.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.491+0000 7f67c4786740 1 --2- 172.21.15.161:0/2487276457 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e836738190 0x55e836735dd0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.491+0000 7f67c4786740 1 --2- 172.21.15.161:0/2487276457 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55e836736310 0x55e8367324e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.491+0000 7f67c4786740 1 --2- 172.21.15.161:0/2487276457 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e836737db0 0x55e836735890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.491+0000 7f67c4786740 1 -- 172.21.15.161:0/2487276457 >> 172.21.15.161:0/2487276457 conn(0x55e836734630 msgr2=0x55e836759220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:33.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.491+0000 7f67c4786740 1 -- 172.21.15.161:0/2487276457 shutdown_connections 2024-03-28T23:35:33.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.491+0000 7f67c4786740 1 -- 172.21.15.161:0/2487276457 wait complete. 2024-03-28T23:35:33.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.492+0000 7f67c4786740 1 -- start start 2024-03-28T23:35:33.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.492+0000 7f67c4786740 1 Processor -- start 2024-03-28T23:35:33.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.492+0000 7f67c4786740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e836737db0 0x55e836784ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:33.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.492+0000 7f67c4786740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e836738190 0x55e8367582f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:33.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.492+0000 7f67c4786740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55e83675cfc0 0x55e83675bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:33.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.492+0000 7f67c4786740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55e83675a090 con 0x55e836737db0 2024-03-28T23:35:33.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.492+0000 7f67c4786740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55e836657f70 con 0x55e836738190 2024-03-28T23:35:33.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.492+0000 7f67c4786740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55e836759ad0 con 0x55e83675cfc0 2024-03-28T23:35:33.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.492+0000 7f67c4786740 10 client.0 _ll_register_callbacks cb 0x55e8363afac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:35:33.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.492+0000 7f67bb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e836737db0 0x55e836784ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:33.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.492+0000 7f67bb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e836737db0 0x55e836784ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42606/0 (socket says 172.21.15.161:42606) 2024-03-28T23:35:33.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.492+0000 7f67bb7fe640 1 -- 172.21.15.161:0/3033942883 learned_addr learned my addr 172.21.15.161:0/3033942883 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:33.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.493+0000 7f67bbfff640 1 --2- 172.21.15.161:0/3033942883 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55e83675cfc0 0x55e83675bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:33.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.493+0000 7f67b2ffd640 1 --2- 172.21.15.161:0/3033942883 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e836738190 0x55e8367582f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:33.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.493+0000 7f67bb7fe640 1 -- 172.21.15.161:0/3033942883 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55e83675cfc0 msgr2=0x55e83675bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:33.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.493+0000 7f67bb7fe640 1 --2- 172.21.15.161:0/3033942883 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55e83675cfc0 0x55e83675bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.493+0000 7f67bb7fe640 1 -- 172.21.15.161:0/3033942883 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e836738190 msgr2=0x55e8367582f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:33.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.493+0000 7f67bb7fe640 1 --2- 172.21.15.161:0/3033942883 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e836738190 0x55e8367582f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.493+0000 7f67bb7fe640 1 -- 172.21.15.161:0/3033942883 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e83647a6b0 con 0x55e836737db0 2024-03-28T23:35:33.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.493+0000 7f67bb7fe640 1 --2- 172.21.15.161:0/3033942883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e836737db0 0x55e836784ed0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f67ac006610 tx=0x7f67ac001150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a0b91e33318f36 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:33.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.493+0000 7f679a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:35:33.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.493+0000 7f679a7fc640 1 -- 172.21.15.161:0/3033942883 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac021410 con 0x55e836737db0 2024-03-28T23:35:33.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.493+0000 7f679a7fc640 1 -- 172.21.15.161:0/3033942883 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 6187+0+0 (secure 0 0 0) 0x7f67ac024960 con 0x55e836737db0 2024-03-28T23:35:33.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.493+0000 7f67c4786740 1 -- 172.21.15.161:0/3033942883 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67a407e3e0 con 0x55e836737db0 2024-03-28T23:35:33.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.493+0000 7f679a7fc640 1 -- 172.21.15.161:0/3033942883 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=129) ==== 24+0+0 (secure 0 0 0) 0x7f67ac024b60 con 0x55e836737db0 2024-03-28T23:35:33.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.493+0000 7f67c4786740 10 client.8688 fetch_fsmap learned FSMap version 129 2024-03-28T23:35:33.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.493+0000 7f67c4786740 1 -- 172.21.15.161:0/3033942883 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=129,monmap=2+}) -- 0x55e836758f30 con 0x55e836737db0 2024-03-28T23:35:33.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.493+0000 7f679a7fc640 1 -- 172.21.15.161:0/3033942883 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67a407e3e0 con 0x55e836737db0 2024-03-28T23:35:33.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.493+0000 7f679a7fc640 1 -- 172.21.15.161:0/3033942883 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 129) ==== 3494+0+0 (secure 0 0 0) 0x7f67ac0215b0 con 0x55e836737db0 2024-03-28T23:35:33.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.494+0000 7f67c4786740 10 client.8688 fetch_fsmap finished waiting for FSMap version 129 2024-03-28T23:35:33.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.494+0000 7f67c4786740 10 client.8688 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:35:33.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.494+0000 7f67c4786740 20 client.8688 populate_metadata read hostname 'smithi161' 2024-03-28T23:35:33.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.494+0000 7f67c4786740 1 --2- 172.21.15.161:0/3033942883 >> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] conn(0x55e83675ac20 0x55e836792f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:33.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.494+0000 7f67c4786740 4 client.8688 mds_command: new command op to 6740 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:35:33.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.494+0000 7f67c4786740 1 -- 172.21.15.161:0/3033942883 --> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e8366a9030 con 0x55e83675ac20 2024-03-28T23:35:33.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.494+0000 7f67b2ffd640 1 --2- 172.21.15.161:0/3033942883 >> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] conn(0x55e83675ac20 0x55e836792f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:33.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.494+0000 7f679a7fc640 10 client.8688 ms_handle_connect on v2:172.21.15.170:6833/3586875977 2024-03-28T23:35:33.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.494+0000 7f67b2ffd640 1 --2- 172.21.15.161:0/3033942883 >> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] conn(0x55e83675ac20 0x55e836792f70 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1068f31a72bd27a5 server_cookie=737de552cec0ca0 in_seq=0 out_seq=0 2024-03-28T23:35:33.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.495+0000 7f679a7fc640 1 -- 172.21.15.161:0/3033942883 <== mds.0 v2:172.21.15.170:6833/3586875977 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e8366a9030 con 0x55e83675ac20 2024-03-28T23:35:33.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.495+0000 7f679a7fc640 10 client.8688 handle_command_reply: tid=0 2024-03-28T23:35:33.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.495+0000 7f67c4786740 1 client.8688 shutdown 2024-03-28T23:35:33.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.495+0000 7f67c4786740 1 -- 172.21.15.161:0/3033942883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e836737db0 msgr2=0x55e836784ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:33.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.495+0000 7f67c4786740 1 --2- 172.21.15.161:0/3033942883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e836737db0 0x55e836784ed0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f67ac006610 tx=0x7f67ac001150 comp rx=0 tx=0).stop 2024-03-28T23:35:33.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.495+0000 7f67c4786740 1 -- 172.21.15.161:0/3033942883 shutdown_connections 2024-03-28T23:35:33.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.495+0000 7f67c4786740 1 --2- 172.21.15.161:0/3033942883 >> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] conn(0x55e83675ac20 0x55e836792f70 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.495+0000 7f67c4786740 1 --2- 172.21.15.161:0/3033942883 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55e83675cfc0 0x55e83675bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.495+0000 7f67c4786740 1 --2- 172.21.15.161:0/3033942883 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e836738190 0x55e8367582f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.495+0000 7f679a7fc640 0 client.8688 ms_handle_reset on v2:172.21.15.170:6833/3586875977 2024-03-28T23:35:33.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.495+0000 7f67c4786740 1 --2- 172.21.15.161:0/3033942883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e836737db0 0x55e836784ed0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.495+0000 7f67c4786740 1 -- 172.21.15.161:0/3033942883 >> 172.21.15.161:0/3033942883 conn(0x55e836736310 msgr2=0x55e8367312b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:33.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.495+0000 7f67c4786740 1 -- 172.21.15.161:0/3033942883 shutdown_connections 2024-03-28T23:35:33.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.495+0000 7f67c4786740 1 -- 172.21.15.161:0/3033942883 wait complete. 2024-03-28T23:35:33.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.495+0000 7f67c4786740 20 client.8688 trim_cache size 0 max 16384 2024-03-28T23:35:33.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.513+0000 7f67c4786740 1 Processor -- start 2024-03-28T23:35:33.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.513+0000 7f67c4786740 1 -- start start 2024-03-28T23:35:33.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.514+0000 7f67c4786740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e836737db0 0x55e8367711e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:33.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.514+0000 7f67c4786740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55e836738190 0x55e836780770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:33.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.514+0000 7f67c4786740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e83675cfc0 0x55e836780cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:33.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.514+0000 7f67c4786740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc1f6e20 con 0x55e836737db0 2024-03-28T23:35:33.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.514+0000 7f67c4786740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc1eebe0 con 0x55e83675cfc0 2024-03-28T23:35:33.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.514+0000 7f67c4786740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f67bc1f67d0 con 0x55e836738190 2024-03-28T23:35:33.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.514+0000 7f67bb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e836737db0 0x55e8367711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:33.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.514+0000 7f67bb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e836737db0 0x55e8367711e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42618/0 (socket says 172.21.15.161:42618) 2024-03-28T23:35:33.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.514+0000 7f67bb7fe640 1 -- 172.21.15.161:0/3689261692 learned_addr learned my addr 172.21.15.161:0/3689261692 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:33.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.514+0000 7f67bbfff640 1 --2- 172.21.15.161:0/3689261692 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e83675cfc0 0x55e836780cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:33.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.514+0000 7f67b2ffd640 1 --2- 172.21.15.161:0/3689261692 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55e836738190 0x55e836780770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:33.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.514+0000 7f67bb7fe640 1 -- 172.21.15.161:0/3689261692 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55e836738190 msgr2=0x55e836780770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:33.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.514+0000 7f67bb7fe640 1 --2- 172.21.15.161:0/3689261692 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55e836738190 0x55e836780770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.514+0000 7f67bb7fe640 1 -- 172.21.15.161:0/3689261692 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e83675cfc0 msgr2=0x55e836780cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:33.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.514+0000 7f67bb7fe640 1 --2- 172.21.15.161:0/3689261692 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e83675cfc0 0x55e836780cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.514+0000 7f67bb7fe640 1 -- 172.21.15.161:0/3689261692 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e836787d40 con 0x55e836737db0 2024-03-28T23:35:33.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.514+0000 7f67bb7fe640 1 --2- 172.21.15.161:0/3689261692 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e836737db0 0x55e8367711e0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f67ac0062f0 tx=0x7f67ac01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaa038b3cf577df server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:33.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.514+0000 7f67b0ff9640 1 -- 172.21.15.161:0/3689261692 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac024d00 con 0x55e836737db0 2024-03-28T23:35:33.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.514+0000 7f67b0ff9640 1 -- 172.21.15.161:0/3689261692 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67ac001bb0 con 0x55e836737db0 2024-03-28T23:35:33.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.514+0000 7f67b0ff9640 1 -- 172.21.15.161:0/3689261692 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac021610 con 0x55e836737db0 2024-03-28T23:35:33.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.515+0000 7f67c4786740 1 -- 172.21.15.161:0/3689261692 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e836737db0 msgr2=0x55e8367711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:33.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.515+0000 7f67c4786740 1 --2- 172.21.15.161:0/3689261692 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e836737db0 0x55e8367711e0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f67ac0062f0 tx=0x7f67ac01b000 comp rx=0 tx=0).stop 2024-03-28T23:35:33.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.515+0000 7f67c4786740 1 -- 172.21.15.161:0/3689261692 shutdown_connections 2024-03-28T23:35:33.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.515+0000 7f67c4786740 1 --2- 172.21.15.161:0/3689261692 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e83675cfc0 0x55e836780cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.515+0000 7f67c4786740 1 --2- 172.21.15.161:0/3689261692 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55e836738190 0x55e836780770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.515+0000 7f67c4786740 1 --2- 172.21.15.161:0/3689261692 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e836737db0 0x55e8367711e0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.515+0000 7f67c4786740 1 -- 172.21.15.161:0/3689261692 >> 172.21.15.161:0/3689261692 conn(0x55e836736310 msgr2=0x55e83672d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:33.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.515+0000 7f67c4786740 1 -- 172.21.15.161:0/3689261692 shutdown_connections 2024-03-28T23:35:33.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.515+0000 7f67c4786740 1 -- 172.21.15.161:0/3689261692 wait complete. 2024-03-28T23:35:33.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.516+0000 7f67c4786740 1 -- start start 2024-03-28T23:35:33.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.516+0000 7f67c4786740 1 Processor -- start 2024-03-28T23:35:33.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.516+0000 7f67c4786740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e836737db0 0x55e836759b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:33.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.516+0000 7f67c4786740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55e836738190 0x55e8367778b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:33.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.516+0000 7f67c4786740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e83675cfc0 0x55e83676b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:33.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.516+0000 7f67c4786740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55e836788850 con 0x55e83675cfc0 2024-03-28T23:35:33.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.516+0000 7f67c4786740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55e83675aaa0 con 0x55e836737db0 2024-03-28T23:35:33.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.516+0000 7f67c4786740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55e8367707a0 con 0x55e836738190 2024-03-28T23:35:33.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.516+0000 7f67c4786740 10 client.0 _ll_register_callbacks cb 0x7f67ac005930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:35:33.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.516+0000 7f67bbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e83675cfc0 0x55e83676b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:33.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.516+0000 7f67b2ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55e836738190 0x55e8367778b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:33.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.516+0000 7f67bb7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e836737db0 0x55e836759b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:33.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.516+0000 7f67b2ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55e836738190 0x55e8367778b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:58656/0 (socket says 172.21.15.161:58656) 2024-03-28T23:35:33.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.516+0000 7f67bbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e83675cfc0 0x55e83676b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42624/0 (socket says 172.21.15.161:42624) 2024-03-28T23:35:33.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.517+0000 7f67b2ffd640 1 -- 172.21.15.161:0/860305284 learned_addr learned my addr 172.21.15.161:0/860305284 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:33.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.517+0000 7f67b2ffd640 1 -- 172.21.15.161:0/860305284 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e836737db0 msgr2=0x55e836759b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:33.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.517+0000 7f67b2ffd640 1 --2- 172.21.15.161:0/860305284 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e836737db0 0x55e836759b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.517+0000 7f67b2ffd640 1 -- 172.21.15.161:0/860305284 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e83675cfc0 msgr2=0x55e83676b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:33.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.517+0000 7f67b2ffd640 1 --2- 172.21.15.161:0/860305284 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e83675cfc0 0x55e83676b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.517+0000 7f67b2ffd640 1 -- 172.21.15.161:0/860305284 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) -- 0x55e8367e3880 con 0x55e836738190 2024-03-28T23:35:33.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.517+0000 7f67bbfff640 1 --2- 172.21.15.161:0/860305284 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e83675cfc0 0x55e83676b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:33.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.517+0000 7f67b2ffd640 1 --2- 172.21.15.161:0/860305284 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55e836738190 0x55e8367778b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f67a4009790 tx=0x7f67a40a9f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fe045b09a7f3185 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:33.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.517+0000 7f679a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.170:3301/0 2024-03-28T23:35:33.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.517+0000 7f679a7fc640 1 -- 172.21.15.161:0/860305284 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a404a040 con 0x55e836738190 2024-03-28T23:35:33.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.517+0000 7f679a7fc640 1 -- 172.21.15.161:0/860305284 <== mon.2 v2:172.21.15.170:3301/0 2 ==== osd_map(137..137 src has 1..137) ==== 6187+0+0 (secure 0 0 0) 0x7f67a4078640 con 0x55e836738190 2024-03-28T23:35:33.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.517+0000 7f67c4786740 1 -- 172.21.15.161:0/860305284 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67ac001bb0 con 0x55e836738190 2024-03-28T23:35:33.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.517+0000 7f679a7fc640 1 -- 172.21.15.161:0/860305284 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_get_version_reply(handle=1 version=129) ==== 24+0+0 (secure 0 0 0) 0x7f67a40765e0 con 0x55e836738190 2024-03-28T23:35:33.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.517+0000 7f67c4786740 10 client.6896 fetch_fsmap learned FSMap version 129 2024-03-28T23:35:33.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.517+0000 7f67c4786740 1 -- 172.21.15.161:0/860305284 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,fsmap=129,monmap=2+}) -- 0x55e836736a60 con 0x55e836738190 2024-03-28T23:35:33.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.517+0000 7f679a7fc640 1 -- 172.21.15.161:0/860305284 <== mon.2 v2:172.21.15.170:3301/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67ac001bb0 con 0x55e836738190 2024-03-28T23:35:33.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.517+0000 7f679a7fc640 1 -- 172.21.15.161:0/860305284 <== mon.2 v2:172.21.15.170:3301/0 5 ==== fsmap(e 129) ==== 3494+0+0 (secure 0 0 0) 0x7f67a40769b0 con 0x55e836738190 2024-03-28T23:35:33.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.517+0000 7f67c4786740 10 client.6896 fetch_fsmap finished waiting for FSMap version 129 2024-03-28T23:35:33.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.517+0000 7f67c4786740 10 client.6896 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:35:33.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.518+0000 7f67c4786740 20 client.6896 populate_metadata read hostname 'smithi161' 2024-03-28T23:35:33.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.518+0000 7f67c4786740 1 --2- 172.21.15.161:0/860305284 >> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] conn(0x55e83676bf00 0x55e83676c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:33.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.518+0000 7f67c4786740 4 client.6896 mds_command: new command op to 6740 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:35:33.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.518+0000 7f67c4786740 1 -- 172.21.15.161:0/860305284 --> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] -- command(tid 0: {"prefix": "status"}) -- 0x55e8366a3380 con 0x55e83676bf00 2024-03-28T23:35:33.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.518+0000 7f67bb7fe640 1 --2- 172.21.15.161:0/860305284 >> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] conn(0x55e83676bf00 0x55e83676c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:33.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.518+0000 7f67bb7fe640 1 --2- 172.21.15.161:0/860305284 >> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] conn(0x55e83676bf00 0x55e83676c2e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=198d63993cd6b2bc server_cookie=ec4faad193745997 in_seq=0 out_seq=0 2024-03-28T23:35:33.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.518+0000 7f679a7fc640 10 client.6896 ms_handle_connect on v2:172.21.15.170:6833/3586875977 2024-03-28T23:35:33.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.518+0000 7f679a7fc640 1 -- 172.21.15.161:0/860305284 <== mds.0 v2:172.21.15.170:6833/3586875977 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x55e8366a3380 con 0x55e83676bf00 2024-03-28T23:35:33.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.518+0000 7f679a7fc640 10 client.6896 handle_command_reply: tid=0 2024-03-28T23:35:33.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.518+0000 7f67c4786740 1 client.6896 shutdown 2024-03-28T23:35:33.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.518+0000 7f67c4786740 1 -- 172.21.15.161:0/860305284 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55e836738190 msgr2=0x55e8367778b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:33.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.518+0000 7f67c4786740 1 --2- 172.21.15.161:0/860305284 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55e836738190 0x55e8367778b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f67a4009790 tx=0x7f67a40a9f90 comp rx=0 tx=0).stop 2024-03-28T23:35:33.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.518+0000 7f67c4786740 1 -- 172.21.15.161:0/860305284 shutdown_connections 2024-03-28T23:35:33.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.518+0000 7f67c4786740 1 --2- 172.21.15.161:0/860305284 >> [v2:172.21.15.170:6833/3586875977,v1:172.21.15.170:6835/3586875977] conn(0x55e83676bf00 0x55e83676c2e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.518+0000 7f67c4786740 1 --2- 172.21.15.161:0/860305284 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55e83675cfc0 0x55e83676b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.518+0000 7f67c4786740 1 --2- 172.21.15.161:0/860305284 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55e836738190 0x55e8367778b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.518+0000 7f67c4786740 1 --2- 172.21.15.161:0/860305284 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e836737db0 0x55e836759b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.518+0000 7f679a7fc640 0 client.6896 ms_handle_reset on v2:172.21.15.170:6833/3586875977 2024-03-28T23:35:33.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.518+0000 7f67c4786740 1 -- 172.21.15.161:0/860305284 >> 172.21.15.161:0/860305284 conn(0x55e836736310 msgr2=0x55e836735400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:33.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.518+0000 7f67c4786740 1 -- 172.21.15.161:0/860305284 shutdown_connections 2024-03-28T23:35:33.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.518+0000 7f67c4786740 1 -- 172.21.15.161:0/860305284 wait complete. 2024-03-28T23:35:33.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.519+0000 7f67c4786740 20 client.6896 trim_cache size 0 max 16384 2024-03-28T23:35:33.526 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:35:33.527 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:35:33.527 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:35:33.527 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 6740, 2024-03-28T23:35:33.527 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:35:33.527 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:35:33.527 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:35:33.527 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 12.232671916999999, 2024-03-28T23:35:33.527 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 129, 2024-03-28T23:35:33.527 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 137, 2024-03-28T23:35:33.527 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 137, 2024-03-28T23:35:33.527 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 24.968037877 2024-03-28T23:35:33.527 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:35:33.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.519+0000 7f67c1ba1640 1 -- 172.21.15.161:0/4292517789 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f679c06b500 msgr2=0x7f679c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:33.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.519+0000 7f67c1ba1640 1 --2- 172.21.15.161:0/4292517789 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f679c06b500 0x7f679c06d9c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f67ac0011e0 tx=0x7f67ac007d30 comp rx=0 tx=0).stop 2024-03-28T23:35:33.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.519+0000 7f67c1ba1640 1 -- 172.21.15.161:0/4292517789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67bc152890 msgr2=0x7f67bc14e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:33.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.519+0000 7f67c1ba1640 1 --2- 172.21.15.161:0/4292517789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67bc152890 0x7f67bc14e090 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f67a4033000 tx=0x7f67a40046c0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.519+0000 7f67c1ba1640 1 -- 172.21.15.161:0/4292517789 shutdown_connections 2024-03-28T23:35:33.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.519+0000 7f67c1ba1640 1 --2- 172.21.15.161:0/4292517789 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f679c06b500 0x7f679c06d9c0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.519+0000 7f67c1ba1640 1 --2- 172.21.15.161:0/4292517789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67bc1553a0 0x7f67bc145800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.519+0000 7f67c1ba1640 1 --2- 172.21.15.161:0/4292517789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67bc152890 0x7f67bc14e090 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.519+0000 7f67c1ba1640 1 --2- 172.21.15.161:0/4292517789 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67bc0c5410 0x7f67bc1500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:33.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.519+0000 7f67c1ba1640 1 -- 172.21.15.161:0/4292517789 >> 172.21.15.161:0/4292517789 conn(0x7f67bc0ac620 msgr2=0x7f67bc0bddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:33.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.519+0000 7f67c1ba1640 1 -- 172.21.15.161:0/4292517789 shutdown_connections 2024-03-28T23:35:33.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:33.519+0000 7f67c1ba1640 1 -- 172.21.15.161:0/4292517789 wait complete. 2024-03-28T23:35:33.530 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 129, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:35:12.982566+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 129, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6740": { "addr": "172.21.15.170:6835/3586875977", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 3586875977, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 3586875977, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6740, "incarnation": 126, "join_fscid": 10, "name": "b", "rank": 0, "state": "up:active", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 137, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-03-28T23:35:24.297578+0000", "qdb_cluster": [ 6740 ], "qdb_leader": 6740, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6740 } } } ], "standbys": [ { "addr": "172.21.15.170:6834/3182285172", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 3182285172, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 3182285172, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 125, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7639, "incarnation": 0, "join_fscid": 10, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 4 }, { "addr": "172.21.15.161:6837/3037890643", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6835", "nonce": 3037890643, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 3037890643, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 120, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8409, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6836/819329318", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 819329318, "type": "v2" }, { "addr": "172.21.15.161:6836", "nonce": 819329318, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 127, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8622, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:35:33.531 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 10, 'name': 'd', 'state': 'up:active'}] 2024-03-28T23:35:33.532 INFO:tasks.cephfs.test_failover:current = [{'gid': 8622, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6836/819329318', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 819329318}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 819329318}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 127}, {'gid': 6740, 'name': 'b', 'rank': 0, 'incarnation': 126, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.170:6835/3586875977', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 3586875977}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 3586875977}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 8409, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6837/3037890643', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 3037890643}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 3037890643}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 120}, {'gid': 7639, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 4, 'addr': '172.21.15.170:6834/3182285172', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 3182285172}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 3182285172}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 125}] 2024-03-28T23:35:33.532 DEBUG:tasks.cephfs.test_failover:-1 != 10 2024-03-28T23:35:33.532 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-28T23:35:38.533 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:35:38.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.645+0000 7fd17909a640 1 Processor -- start 2024-03-28T23:35:38.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.645+0000 7fd17909a640 1 -- start start 2024-03-28T23:35:38.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.645+0000 7fd17909a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd174155b90 0x7fd174158750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:38.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.645+0000 7fd17909a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd1740ac150 0x7fd174158c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:38.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.645+0000 7fd17909a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd174066fe0 con 0x7fd17406c3d0 2024-03-28T23:35:38.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.645+0000 7fd17909a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd1741591d0 con 0x7fd1740ac150 2024-03-28T23:35:38.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.645+0000 7fd17909a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd174156070 con 0x7fd174155b90 2024-03-28T23:35:38.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.646+0000 7fd17359e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd1740ac150 0x7fd174158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:38.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.646+0000 7fd17259c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd174155b90 0x7fd174158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:38.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.646+0000 7fd17359e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd1740ac150 0x7fd174158c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:35500/0 (socket says 172.21.15.161:35500) 2024-03-28T23:35:38.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.646+0000 7fd17259c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd174155b90 0x7fd174158750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:58666/0 (socket says 172.21.15.161:58666) 2024-03-28T23:35:38.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.646+0000 7fd172d9d640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fd17406c3d0 0x7fd174152580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:47072/0 (socket says 172.21.15.161:47072) 2024-03-28T23:35:38.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.646+0000 7fd17359e640 1 -- 172.21.15.161:0/2110338426 learned_addr learned my addr 172.21.15.161:0/2110338426 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:38.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.646+0000 7fd17359e640 1 -- 172.21.15.161:0/2110338426 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd174155b90 msgr2=0x7fd174158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:38.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.646+0000 7fd17359e640 1 --2- 172.21.15.161:0/2110338426 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd174155b90 0x7fd174158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:38.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.646+0000 7fd17359e640 1 -- 172.21.15.161:0/2110338426 >> v1:172.21.15.170:6789/0 conn(0x7fd17406c3d0 legacy=0x7fd174152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:38.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.646+0000 7fd17359e640 1 -- 172.21.15.161:0/2110338426 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1740672c0 con 0x7fd1740ac150 2024-03-28T23:35:38.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.647+0000 7fd17359e640 1 --2- 172.21.15.161:0/2110338426 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1740ac150 0x7fd174158c90 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd164009870 tx=0x7fd16402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5dc4958ab35ac1f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:38.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.647+0000 7fd171d9b640 1 -- 172.21.15.161:0/2110338426 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164034070 con 0x7fd1740ac150 2024-03-28T23:35:38.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.647+0000 7fd171d9b640 1 -- 172.21.15.161:0/2110338426 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd164030840 con 0x7fd1740ac150 2024-03-28T23:35:38.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.647+0000 7fd171d9b640 1 -- 172.21.15.161:0/2110338426 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164030c10 con 0x7fd1740ac150 2024-03-28T23:35:38.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.647+0000 7fd17909a640 1 -- 172.21.15.161:0/2110338426 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1740ac150 msgr2=0x7fd174158c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:38.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.647+0000 7fd17909a640 1 --2- 172.21.15.161:0/2110338426 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1740ac150 0x7fd174158c90 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd164009870 tx=0x7fd16402b980 comp rx=0 tx=0).stop 2024-03-28T23:35:38.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.647+0000 7fd17909a640 1 -- 172.21.15.161:0/2110338426 shutdown_connections 2024-03-28T23:35:38.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.647+0000 7fd17909a640 1 --2- 172.21.15.161:0/2110338426 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1740ac150 0x7fd174158c90 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:38.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.647+0000 7fd17909a640 1 --2- 172.21.15.161:0/2110338426 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd174155b90 0x7fd174158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:38.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.647+0000 7fd17909a640 1 -- 172.21.15.161:0/2110338426 >> 172.21.15.161:0/2110338426 conn(0x7fd1740c3be0 msgr2=0x7fd1740c3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:38.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.647+0000 7fd17909a640 1 -- 172.21.15.161:0/2110338426 shutdown_connections 2024-03-28T23:35:38.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.647+0000 7fd17909a640 1 -- 172.21.15.161:0/2110338426 wait complete. 2024-03-28T23:35:38.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.649+0000 7fd17909a640 1 Processor -- start 2024-03-28T23:35:38.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.649+0000 7fd17909a640 1 -- start start 2024-03-28T23:35:38.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.649+0000 7fd17909a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd17406c3d0 0x7fd1741df640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:38.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.649+0000 7fd17909a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1740ac150 0x7fd1741edec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:38.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.649+0000 7fd17909a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd174155b90 0x7fd1741de570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:38.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.649+0000 7fd17909a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd174152e10 con 0x7fd1740ac150 2024-03-28T23:35:38.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.650+0000 7fd17909a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd174152c90 con 0x7fd174155b90 2024-03-28T23:35:38.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.650+0000 7fd17909a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd1740c21a0 con 0x7fd17406c3d0 2024-03-28T23:35:38.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.650+0000 7fd17259c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1740ac150 0x7fd1741edec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:38.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.650+0000 7fd17259c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1740ac150 0x7fd1741edec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42634/0 (socket says 172.21.15.161:42634) 2024-03-28T23:35:38.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.650+0000 7fd172d9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd17406c3d0 0x7fd1741df640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:38.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.650+0000 7fd17259c640 1 -- 172.21.15.161:0/4261057367 learned_addr learned my addr 172.21.15.161:0/4261057367 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:38.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.650+0000 7fd17359e640 1 --2- 172.21.15.161:0/4261057367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd174155b90 0x7fd1741de570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:38.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.650+0000 7fd172d9d640 1 -- 172.21.15.161:0/4261057367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd174155b90 msgr2=0x7fd1741de570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:38.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.650+0000 7fd172d9d640 1 --2- 172.21.15.161:0/4261057367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd174155b90 0x7fd1741de570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:38.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.650+0000 7fd172d9d640 1 -- 172.21.15.161:0/4261057367 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1740ac150 msgr2=0x7fd1741edec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:38.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.650+0000 7fd172d9d640 1 --2- 172.21.15.161:0/4261057367 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1740ac150 0x7fd1741edec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:38.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.650+0000 7fd172d9d640 1 -- 172.21.15.161:0/4261057367 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1741dfb80 con 0x7fd17406c3d0 2024-03-28T23:35:38.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.650+0000 7fd17259c640 1 --2- 172.21.15.161:0/4261057367 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1740ac150 0x7fd1741edec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:38.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.650+0000 7fd17359e640 1 --2- 172.21.15.161:0/4261057367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd174155b90 0x7fd1741de570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:38.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.650+0000 7fd172d9d640 1 --2- 172.21.15.161:0/4261057367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd17406c3d0 0x7fd1741df640 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd16c0012e0 tx=0x7fd16c00cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:38.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.650+0000 7fd16b7fe640 1 -- 172.21.15.161:0/4261057367 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16c018070 con 0x7fd17406c3d0 2024-03-28T23:35:38.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.650+0000 7fd16b7fe640 1 -- 172.21.15.161:0/4261057367 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd16c004d60 con 0x7fd17406c3d0 2024-03-28T23:35:38.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.650+0000 7fd17909a640 1 -- 172.21.15.161:0/4261057367 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd174156dc0 con 0x7fd17406c3d0 2024-03-28T23:35:38.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.650+0000 7fd16b7fe640 1 -- 172.21.15.161:0/4261057367 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16c00a880 con 0x7fd17406c3d0 2024-03-28T23:35:38.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.651+0000 7fd17909a640 1 -- 172.21.15.161:0/4261057367 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1741ee730 con 0x7fd17406c3d0 2024-03-28T23:35:38.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.651+0000 7fd17909a640 1 -- 172.21.15.161:0/4261057367 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd174068f90 con 0x7fd17406c3d0 2024-03-28T23:35:38.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.652+0000 7fd16b7fe640 1 -- 172.21.15.161:0/4261057367 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd16c00ac30 con 0x7fd17406c3d0 2024-03-28T23:35:38.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.652+0000 7fd16b7fe640 1 --2- 172.21.15.161:0/4261057367 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd15406b4d0 0x7fd15406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:38.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.652+0000 7fd16b7fe640 1 -- 172.21.15.161:0/4261057367 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(139..139 src has 1..139) ==== 6245+0+0 (secure 0 0 0) 0x7fd16c089b00 con 0x7fd17406c3d0 2024-03-28T23:35:38.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.652+0000 7fd17259c640 1 --2- 172.21.15.161:0/4261057367 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd15406b4d0 0x7fd15406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:38.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.653+0000 7fd17259c640 1 --2- 172.21.15.161:0/4261057367 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd15406b4d0 0x7fd15406d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fd1740c10d0 tx=0x7fd15c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:38.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.657+0000 7fd16b7fe640 1 -- 172.21.15.161:0/4261057367 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd174068f90 con 0x7fd17406c3d0 2024-03-28T23:35:38.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.864+0000 7fd17909a640 1 -- 172.21.15.161:0/4261057367 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd17404f4b0 con 0x7fd17406c3d0 2024-03-28T23:35:38.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.864+0000 7fd16b7fe640 1 -- 172.21.15.161:0/4261057367 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) ==== 95+0+5203 (secure 0 0 0) 0x7fd16c058120 con 0x7fd17406c3d0 2024-03-28T23:35:38.865 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:38.865 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":130,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7639,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.170:6834/3182285172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3182285172},{"type":"v1","addr":"172.21.15.170:6834","nonce":3182285172}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":125},{"gid":8409,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/3037890643","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3037890643},{"type":"v1","addr":"172.21.15.161:6837","nonce":3037890643}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120},{"gid":8622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/819329318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":819329318},{"type":"v1","addr":"172.21.15.161:6836","nonce":819329318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:12.982566+0000","modified":"2024-03-28T23:35:35.299347+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":137,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":126,"state":"up:active","state_seq":10,"addr":"172.21.15.170:6835/3586875977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":3586875977},{"type":"v1","addr":"172.21.15.170:6835","nonce":3586875977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":10}]} 2024-03-28T23:35:38.866 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 130 2024-03-28T23:35:38.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.866+0000 7fd17909a640 1 -- 172.21.15.161:0/4261057367 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd15406b4d0 msgr2=0x7fd15406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:38.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.866+0000 7fd17909a640 1 --2- 172.21.15.161:0/4261057367 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd15406b4d0 0x7fd15406d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fd1740c10d0 tx=0x7fd15c009190 comp rx=0 tx=0).stop 2024-03-28T23:35:38.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.866+0000 7fd17909a640 1 -- 172.21.15.161:0/4261057367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd17406c3d0 msgr2=0x7fd1741df640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:38.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.866+0000 7fd17909a640 1 --2- 172.21.15.161:0/4261057367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd17406c3d0 0x7fd1741df640 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd16c0012e0 tx=0x7fd16c00cb90 comp rx=0 tx=0).stop 2024-03-28T23:35:38.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.866+0000 7fd17909a640 1 -- 172.21.15.161:0/4261057367 shutdown_connections 2024-03-28T23:35:38.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.866+0000 7fd17909a640 1 --2- 172.21.15.161:0/4261057367 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd15406b4d0 0x7fd15406d990 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:38.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.866+0000 7fd17909a640 1 --2- 172.21.15.161:0/4261057367 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1740ac150 0x7fd1741edec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:38.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.866+0000 7fd17909a640 1 --2- 172.21.15.161:0/4261057367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd174155b90 0x7fd1741de570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:38.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.866+0000 7fd17909a640 1 --2- 172.21.15.161:0/4261057367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd17406c3d0 0x7fd1741df640 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:38.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.866+0000 7fd17909a640 1 -- 172.21.15.161:0/4261057367 >> 172.21.15.161:0/4261057367 conn(0x7fd1740c3be0 msgr2=0x7fd17406e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:38.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.866+0000 7fd17909a640 1 -- 172.21.15.161:0/4261057367 shutdown_connections 2024-03-28T23:35:38.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:38.866+0000 7fd17909a640 1 -- 172.21.15.161:0/4261057367 wait complete. 2024-03-28T23:35:38.969 INFO:tasks.ceph.mds.b.smithi170.stderr: -420> 2024-03-28T23:35:28.904+0000 7ff45e6b8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:35:38.994 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2024-03-28T23:35:39.280 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 130, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:35:12.982566+0000', 'modified': '2024-03-28T23:35:35.299347+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 137, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6740}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6740': {'gid': 6740, 'name': 'b', 'rank': 0, 'incarnation': 126, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.170:6835/3586875977', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 3586875977}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 3586875977}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6740, 'qdb_cluster': [6740]} 2024-03-28T23:35:39.280 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:35:39.281 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T23:35:39.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.385+0000 7f4f6e18c640 1 Processor -- start 2024-03-28T23:35:39.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.385+0000 7f4f6e18c640 1 -- start start 2024-03-28T23:35:39.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.385+0000 7f4f6e18c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4f680c5410 0x7f4f680c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:39.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.385+0000 7f4f6e18c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f681553a0 0x7f4f68155850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:39.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.385+0000 7f4f6e18c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f68158a30 con 0x7f4f68152890 2024-03-28T23:35:39.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.385+0000 7f4f6e18c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f4f680c8480 con 0x7f4f680c5410 2024-03-28T23:35:39.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.385+0000 7f4f6e18c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4f680c5d90 con 0x7f4f681553a0 2024-03-28T23:35:39.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.386+0000 7f4f66ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4f680c5410 0x7f4f680c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:39.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.386+0000 7f4f677fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f4f68152890 0x7f4f680c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:54478/0 (socket says 172.21.15.161:54478) 2024-03-28T23:35:39.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.386+0000 7f4f677fe640 1 -- 172.21.15.161:0/1317261160 learned_addr learned my addr 172.21.15.161:0/1317261160 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:39.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.386+0000 7f4f67fff640 1 --2- 172.21.15.161:0/1317261160 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f681553a0 0x7f4f68155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:39.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.386+0000 7f4f66ffd640 1 -- 172.21.15.161:0/1317261160 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f681553a0 msgr2=0x7f4f68155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:39.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.386+0000 7f4f66ffd640 1 --2- 172.21.15.161:0/1317261160 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f681553a0 0x7f4f68155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:39.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.386+0000 7f4f66ffd640 1 -- 172.21.15.161:0/1317261160 >> v1:172.21.15.161:6789/0 conn(0x7f4f68152890 legacy=0x7f4f680c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:39.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.386+0000 7f4f667fc640 1 -- 172.21.15.161:0/1317261160 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 670637027 0 0) 0x7f4f68158a30 con 0x7f4f68152890 2024-03-28T23:35:39.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.386+0000 7f4f66ffd640 1 -- 172.21.15.161:0/1317261160 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f68158d10 con 0x7f4f680c5410 2024-03-28T23:35:39.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.387+0000 7f4f66ffd640 1 --2- 172.21.15.161:0/1317261160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f680c5410 0x7f4f680c5850 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c009790 tx=0x7f4f5c02b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c92fcd009dbbe280 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:39.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.387+0000 7f4f667fc640 1 -- 172.21.15.161:0/1317261160 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c031770 con 0x7f4f680c5410 2024-03-28T23:35:39.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.387+0000 7f4f667fc640 1 -- 172.21.15.161:0/1317261160 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f5c037070 con 0x7f4f680c5410 2024-03-28T23:35:39.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.387+0000 7f4f667fc640 1 -- 172.21.15.161:0/1317261160 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c031b30 con 0x7f4f680c5410 2024-03-28T23:35:39.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.387+0000 7f4f6e18c640 1 -- 172.21.15.161:0/1317261160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f680c5410 msgr2=0x7f4f680c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:39.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.387+0000 7f4f6e18c640 1 --2- 172.21.15.161:0/1317261160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f680c5410 0x7f4f680c5850 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c009790 tx=0x7f4f5c02b810 comp rx=0 tx=0).stop 2024-03-28T23:35:39.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.387+0000 7f4f6e18c640 1 -- 172.21.15.161:0/1317261160 shutdown_connections 2024-03-28T23:35:39.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.387+0000 7f4f6e18c640 1 --2- 172.21.15.161:0/1317261160 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f681553a0 0x7f4f68155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:39.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.387+0000 7f4f6e18c640 1 --2- 172.21.15.161:0/1317261160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f680c5410 0x7f4f680c5850 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:39.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.387+0000 7f4f6e18c640 1 -- 172.21.15.161:0/1317261160 >> 172.21.15.161:0/1317261160 conn(0x7f4f680ac620 msgr2=0x7f4f680aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:39.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.387+0000 7f4f6e18c640 1 -- 172.21.15.161:0/1317261160 shutdown_connections 2024-03-28T23:35:39.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.387+0000 7f4f6e18c640 1 -- 172.21.15.161:0/1317261160 wait complete. 2024-03-28T23:35:39.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.388+0000 7f4f6e18c640 1 Processor -- start 2024-03-28T23:35:39.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.389+0000 7f4f6e18c640 1 -- start start 2024-03-28T23:35:39.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.389+0000 7f4f6e18c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f680c5410 0x7f4f68148030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:39.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.389+0000 7f4f6e18c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f68152890 0x7f4f68148570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:39.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.389+0000 7f4f6e18c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f681553a0 0x7f4f68156c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:39.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.389+0000 7f4f6e18c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4f681592d0 con 0x7f4f68152890 2024-03-28T23:35:39.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.389+0000 7f4f6e18c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4f68159640 con 0x7f4f681553a0 2024-03-28T23:35:39.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.389+0000 7f4f6e18c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4f680c90a0 con 0x7f4f680c5410 2024-03-28T23:35:39.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.389+0000 7f4f66ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f68152890 0x7f4f68148570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:39.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.389+0000 7f4f66ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f68152890 0x7f4f68148570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42642/0 (socket says 172.21.15.161:42642) 2024-03-28T23:35:39.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.389+0000 7f4f66ffd640 1 -- 172.21.15.161:0/3614582810 learned_addr learned my addr 172.21.15.161:0/3614582810 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:39.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.390+0000 7f4f67fff640 1 --2- 172.21.15.161:0/3614582810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f681553a0 0x7f4f68156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:39.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.390+0000 7f4f677fe640 1 --2- 172.21.15.161:0/3614582810 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f680c5410 0x7f4f68148030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:39.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.390+0000 7f4f66ffd640 1 -- 172.21.15.161:0/3614582810 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f680c5410 msgr2=0x7f4f68148030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:39.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.390+0000 7f4f66ffd640 1 --2- 172.21.15.161:0/3614582810 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f680c5410 0x7f4f68148030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:39.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.390+0000 7f4f66ffd640 1 -- 172.21.15.161:0/3614582810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f681553a0 msgr2=0x7f4f68156c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:39.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.390+0000 7f4f66ffd640 1 --2- 172.21.15.161:0/3614582810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f681553a0 0x7f4f68156c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:39.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.390+0000 7f4f66ffd640 1 -- 172.21.15.161:0/3614582810 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f681eedc0 con 0x7f4f68152890 2024-03-28T23:35:39.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.390+0000 7f4f67fff640 1 --2- 172.21.15.161:0/3614582810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f681553a0 0x7f4f68156c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:39.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.390+0000 7f4f66ffd640 1 --2- 172.21.15.161:0/3614582810 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f68152890 0x7f4f68148570 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c000c00 tx=0x7f4f5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:39.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.390+0000 7f4f4ffff640 1 -- 172.21.15.161:0/3614582810 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c032600 con 0x7f4f68152890 2024-03-28T23:35:39.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.390+0000 7f4f4ffff640 1 -- 172.21.15.161:0/3614582810 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f5c0327a0 con 0x7f4f68152890 2024-03-28T23:35:39.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.390+0000 7f4f6e18c640 1 -- 172.21.15.161:0/3614582810 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f681ef0a0 con 0x7f4f68152890 2024-03-28T23:35:39.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.390+0000 7f4f4ffff640 1 -- 172.21.15.161:0/3614582810 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c032b70 con 0x7f4f68152890 2024-03-28T23:35:39.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.390+0000 7f4f6e18c640 1 -- 172.21.15.161:0/3614582810 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f6814f320 con 0x7f4f68152890 2024-03-28T23:35:39.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.391+0000 7f4f4ffff640 1 -- 172.21.15.161:0/3614582810 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4f5c04c060 con 0x7f4f68152890 2024-03-28T23:35:39.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.392+0000 7f4f4ffff640 1 --2- 172.21.15.161:0/3614582810 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4f5406b4d0 0x7f4f5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:39.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.392+0000 7f4f677fe640 1 --2- 172.21.15.161:0/3614582810 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4f5406b4d0 0x7f4f5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:39.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.392+0000 7f4f4ffff640 1 -- 172.21.15.161:0/3614582810 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 6331+0+0 (secure 0 0 0) 0x7f4f5c0abd80 con 0x7f4f68152890 2024-03-28T23:35:39.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.392+0000 7f4f677fe640 1 --2- 172.21.15.161:0/3614582810 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4f5406b4d0 0x7f4f5406d990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4f58000f60 tx=0x7f4f58007b70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:39.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.393+0000 7f4f70d73740 1 Processor -- start 2024-03-28T23:35:39.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.393+0000 7f4f70d73740 1 -- start start 2024-03-28T23:35:39.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.394+0000 7f4f70d73740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55891d211db0 0x55891d20f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:39.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.394+0000 7f4f70d73740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55891d212190 0x55891d20fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:39.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.394+0000 7f4f70d73740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55891d210310 0x55891d20c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:39.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.394+0000 7f4f70d73740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55891d175af0 con 0x55891d211db0 2024-03-28T23:35:39.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.394+0000 7f4f70d73740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55891d11e9d0 con 0x55891d210310 2024-03-28T23:35:39.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.394+0000 7f4f70d73740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55891d171b90 con 0x55891d212190 2024-03-28T23:35:39.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.394+0000 7f4f677fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55891d211db0 0x55891d20f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:39.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.394+0000 7f4f677fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55891d211db0 0x55891d20f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42650/0 (socket says 172.21.15.161:42650) 2024-03-28T23:35:39.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.394+0000 7f4f677fe640 1 -- 172.21.15.161:0/546340351 learned_addr learned my addr 172.21.15.161:0/546340351 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:39.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.394+0000 7f4f66ffd640 1 --2- 172.21.15.161:0/546340351 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55891d212190 0x55891d20fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:39.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.394+0000 7f4f67fff640 1 --2- 172.21.15.161:0/546340351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55891d210310 0x55891d20c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:39.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.394+0000 7f4f677fe640 1 -- 172.21.15.161:0/546340351 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55891d212190 msgr2=0x55891d20fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:39.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.394+0000 7f4f677fe640 1 --2- 172.21.15.161:0/546340351 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55891d212190 0x55891d20fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:39.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.394+0000 7f4f677fe640 1 -- 172.21.15.161:0/546340351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55891d210310 msgr2=0x55891d20c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:39.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.394+0000 7f4f677fe640 1 --2- 172.21.15.161:0/546340351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55891d210310 0x55891d20c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:39.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.394+0000 7f4f677fe640 1 -- 172.21.15.161:0/546340351 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55891ce8a2d0 con 0x55891d211db0 2024-03-28T23:35:39.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.394+0000 7f4f677fe640 1 --2- 172.21.15.161:0/546340351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55891d211db0 0x55891d20f890 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f4f58005d80 tx=0x7f4f58012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45259e6ebcf3a4f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:39.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.394+0000 7f4f4d7fa640 1 -- 172.21.15.161:0/546340351 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f58015110 con 0x55891d211db0 2024-03-28T23:35:39.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.394+0000 7f4f4d7fa640 1 -- 172.21.15.161:0/546340351 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f580159c0 con 0x55891d211db0 2024-03-28T23:35:39.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.394+0000 7f4f4d7fa640 1 -- 172.21.15.161:0/546340351 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f58015d70 con 0x55891d211db0 2024-03-28T23:35:39.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.395+0000 7f4f70d73740 1 -- 172.21.15.161:0/546340351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55891d211db0 msgr2=0x55891d20f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:39.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.395+0000 7f4f70d73740 1 --2- 172.21.15.161:0/546340351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55891d211db0 0x55891d20f890 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f4f58005d80 tx=0x7f4f58012000 comp rx=0 tx=0).stop 2024-03-28T23:35:39.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.395+0000 7f4f70d73740 1 -- 172.21.15.161:0/546340351 shutdown_connections 2024-03-28T23:35:39.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.395+0000 7f4f70d73740 1 --2- 172.21.15.161:0/546340351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55891d210310 0x55891d20c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:39.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.395+0000 7f4f70d73740 1 --2- 172.21.15.161:0/546340351 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55891d212190 0x55891d20fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:39.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.395+0000 7f4f70d73740 1 --2- 172.21.15.161:0/546340351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55891d211db0 0x55891d20f890 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:39.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.395+0000 7f4f70d73740 1 -- 172.21.15.161:0/546340351 >> 172.21.15.161:0/546340351 conn(0x55891d20e630 msgr2=0x55891d233220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:39.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.395+0000 7f4f70d73740 1 -- 172.21.15.161:0/546340351 shutdown_connections 2024-03-28T23:35:39.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.395+0000 7f4f70d73740 1 -- 172.21.15.161:0/546340351 wait complete. 2024-03-28T23:35:39.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.396+0000 7f4f70d73740 1 -- start start 2024-03-28T23:35:39.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.396+0000 7f4f70d73740 1 Processor -- start 2024-03-28T23:35:39.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.396+0000 7f4f70d73740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55891d211db0 0x55891d22eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:39.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.396+0000 7f4f70d73740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55891d212190 0x55891d236fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:39.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.397+0000 7f4f70d73740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55891d25e040 0x55891d231f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:39.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.397+0000 7f4f70d73740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55891d234090 con 0x55891d25e040 2024-03-28T23:35:39.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.397+0000 7f4f70d73740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55891d131f70 con 0x55891d211db0 2024-03-28T23:35:39.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.397+0000 7f4f70d73740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55891d233ad0 con 0x55891d212190 2024-03-28T23:35:39.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.397+0000 7f4f70d73740 10 client.0 _ll_register_callbacks cb 0x55891ce89ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:35:39.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.397+0000 7f4f67fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55891d25e040 0x55891d231f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:39.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.397+0000 7f4f67fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55891d25e040 0x55891d231f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42662/0 (socket says 172.21.15.161:42662) 2024-03-28T23:35:39.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.397+0000 7f4f67fff640 1 -- 172.21.15.161:0/359651828 learned_addr learned my addr 172.21.15.161:0/359651828 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:39.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.397+0000 7f4f677fe640 1 --2- 172.21.15.161:0/359651828 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55891d211db0 0x55891d22eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:39.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.397+0000 7f4f66ffd640 1 --2- 172.21.15.161:0/359651828 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55891d212190 0x55891d236fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:39.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.397+0000 7f4f67fff640 1 -- 172.21.15.161:0/359651828 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55891d212190 msgr2=0x55891d236fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:39.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.397+0000 7f4f67fff640 1 --2- 172.21.15.161:0/359651828 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55891d212190 0x55891d236fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:39.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.397+0000 7f4f67fff640 1 -- 172.21.15.161:0/359651828 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55891d211db0 msgr2=0x55891d22eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:39.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.397+0000 7f4f67fff640 1 --2- 172.21.15.161:0/359651828 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55891d211db0 0x55891d22eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:39.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.397+0000 7f4f67fff640 1 -- 172.21.15.161:0/359651828 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55891cf546b0 con 0x55891d25e040 2024-03-28T23:35:39.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.397+0000 7f4f67fff640 1 --2- 172.21.15.161:0/359651828 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55891d25e040 0x55891d231f90 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f4f60007de0 tx=0x7f4f60012610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc987fad0d6e443e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:39.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.397+0000 7f4f467fc640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:35:39.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.397+0000 7f4f467fc640 1 -- 172.21.15.161:0/359651828 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f6001c020 con 0x55891d25e040 2024-03-28T23:35:39.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.397+0000 7f4f467fc640 1 -- 172.21.15.161:0/359651828 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(140..140 src has 1..140) ==== 6331+0+0 (secure 0 0 0) 0x7f4f60018670 con 0x55891d25e040 2024-03-28T23:35:39.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.397+0000 7f4f70d73740 1 -- 172.21.15.161:0/359651828 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f580159c0 con 0x55891d25e040 2024-03-28T23:35:39.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.398+0000 7f4f467fc640 1 -- 172.21.15.161:0/359651828 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=131) ==== 24+0+0 (secure 0 0 0) 0x7f4f60026050 con 0x55891d25e040 2024-03-28T23:35:39.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.398+0000 7f4f70d73740 10 client.8715 fetch_fsmap learned FSMap version 131 2024-03-28T23:35:39.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.398+0000 7f4f70d73740 1 -- 172.21.15.161:0/359651828 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=131,monmap=2+}) -- 0x55891d237e60 con 0x55891d25e040 2024-03-28T23:35:39.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.398+0000 7f4f467fc640 1 -- 172.21.15.161:0/359651828 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f580159c0 con 0x55891d25e040 2024-03-28T23:35:39.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.398+0000 7f4f467fc640 1 -- 172.21.15.161:0/359651828 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 131) ==== 2881+0+0 (secure 0 0 0) 0x7f4f60029020 con 0x55891d25e040 2024-03-28T23:35:39.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.398+0000 7f4f70d73740 10 client.8715 fetch_fsmap finished waiting for FSMap version 131 2024-03-28T23:35:39.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.398+0000 7f4f70d73740 -1 client.8715 resolve_mds: no MDS daemons found by name `b' 2024-03-28T23:35:39.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.398+0000 7f4f70d73740 -1 client.8715 FSMap: e131 2024-03-28T23:35:39.406 INFO:teuthology.orchestra.run.smithi161.stderr:enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T23:35:39.406 INFO:teuthology.orchestra.run.smithi161.stderr:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-28T23:35:39.406 INFO:teuthology.orchestra.run.smithi161.stderr:legacy client fscid: 10 2024-03-28T23:35:39.406 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:35:39.406 INFO:teuthology.orchestra.run.smithi161.stderr:Filesystem 'cephfs' (10) 2024-03-28T23:35:39.406 INFO:teuthology.orchestra.run.smithi161.stderr:fs_name cephfs 2024-03-28T23:35:39.406 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 131 2024-03-28T23:35:39.406 INFO:teuthology.orchestra.run.smithi161.stderr:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-28T23:35:39.406 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-28T23:35:12.982566+0000 2024-03-28T23:35:39.406 INFO:teuthology.orchestra.run.smithi161.stderr:modified 2024-03-28T23:35:38.963320+0000 2024-03-28T23:35:39.406 INFO:teuthology.orchestra.run.smithi161.stderr:tableserver 0 2024-03-28T23:35:39.407 INFO:teuthology.orchestra.run.smithi161.stderr:root 0 2024-03-28T23:35:39.407 INFO:teuthology.orchestra.run.smithi161.stderr:session_timeout 60 2024-03-28T23:35:39.407 INFO:teuthology.orchestra.run.smithi161.stderr:session_autoclose 300 2024-03-28T23:35:39.407 INFO:teuthology.orchestra.run.smithi161.stderr:max_file_size 1099511627776 2024-03-28T23:35:39.407 INFO:teuthology.orchestra.run.smithi161.stderr:max_xattr_size 65536 2024-03-28T23:35:39.407 INFO:teuthology.orchestra.run.smithi161.stderr:required_client_features {} 2024-03-28T23:35:39.407 INFO:teuthology.orchestra.run.smithi161.stderr:last_failure 0 2024-03-28T23:35:39.407 INFO:teuthology.orchestra.run.smithi161.stderr:last_failure_osd_epoch 140 2024-03-28T23:35:39.407 INFO:teuthology.orchestra.run.smithi161.stderr:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-28T23:35:39.407 INFO:teuthology.orchestra.run.smithi161.stderr:max_mds 1 2024-03-28T23:35:39.407 INFO:teuthology.orchestra.run.smithi161.stderr:in 0 2024-03-28T23:35:39.407 INFO:teuthology.orchestra.run.smithi161.stderr:up {0=7639} 2024-03-28T23:35:39.407 INFO:teuthology.orchestra.run.smithi161.stderr:failed 2024-03-28T23:35:39.408 INFO:teuthology.orchestra.run.smithi161.stderr:damaged 2024-03-28T23:35:39.408 INFO:teuthology.orchestra.run.smithi161.stderr:stopped 2024-03-28T23:35:39.408 INFO:teuthology.orchestra.run.smithi161.stderr:data_pools [22] 2024-03-28T23:35:39.408 INFO:teuthology.orchestra.run.smithi161.stderr:metadata_pool 21 2024-03-28T23:35:39.408 INFO:teuthology.orchestra.run.smithi161.stderr:inline_data disabled 2024-03-28T23:35:39.408 INFO:teuthology.orchestra.run.smithi161.stderr:balancer 2024-03-28T23:35:39.408 INFO:teuthology.orchestra.run.smithi161.stderr:bal_rank_mask -1 2024-03-28T23:35:39.408 INFO:teuthology.orchestra.run.smithi161.stderr:standby_count_wanted 0 2024-03-28T23:35:39.408 INFO:teuthology.orchestra.run.smithi161.stderr:qdb_cluster leader: 0 members: 2024-03-28T23:35:39.408 INFO:teuthology.orchestra.run.smithi161.stderr:[mds.d{0:7639} state up:replay seq 4 join_fscid=10 addr [v2:172.21.15.170:6832/3182285172,v1:172.21.15.170:6834/3182285172] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T23:35:39.408 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:35:39.408 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:35:39.408 INFO:teuthology.orchestra.run.smithi161.stderr:Standby daemons: 2024-03-28T23:35:39.408 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:35:39.408 INFO:teuthology.orchestra.run.smithi161.stderr:[mds.c{-1:8409} state up:standby seq 1 addr [v2:172.21.15.161:6835/3037890643,v1:172.21.15.161:6837/3037890643] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T23:35:39.409 INFO:teuthology.orchestra.run.smithi161.stderr:[mds.a{-1:8622} state up:standby seq 1 addr [v2:172.21.15.161:6834/819329318,v1:172.21.15.161:6836/819329318] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T23:35:39.409 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:35:39.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.398+0000 7f4f70d73740 1 client.8715 shutdown 2024-03-28T23:35:39.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.398+0000 7f4f70d73740 1 -- 172.21.15.161:0/359651828 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55891d25e040 msgr2=0x55891d231f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:39.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.398+0000 7f4f70d73740 1 --2- 172.21.15.161:0/359651828 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55891d25e040 0x55891d231f90 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f4f60007de0 tx=0x7f4f60012610 comp rx=0 tx=0).stop 2024-03-28T23:35:39.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.398+0000 7f4f70d73740 1 -- 172.21.15.161:0/359651828 shutdown_connections 2024-03-28T23:35:39.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.398+0000 7f4f70d73740 1 --2- 172.21.15.161:0/359651828 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55891d25e040 0x55891d231f90 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:39.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.398+0000 7f4f70d73740 1 --2- 172.21.15.161:0/359651828 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55891d212190 0x55891d236fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:39.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.398+0000 7f4f70d73740 1 --2- 172.21.15.161:0/359651828 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55891d211db0 0x55891d22eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:39.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.398+0000 7f4f70d73740 1 -- 172.21.15.161:0/359651828 >> 172.21.15.161:0/359651828 conn(0x55891d210310 msgr2=0x55891d24e9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:39.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.398+0000 7f4f70d73740 1 -- 172.21.15.161:0/359651828 shutdown_connections 2024-03-28T23:35:39.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.398+0000 7f4f70d73740 1 -- 172.21.15.161:0/359651828 wait complete. 2024-03-28T23:35:39.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:39.399+0000 7f4f70d73740 20 client.8715 trim_cache size 0 max 16384 2024-03-28T23:35:39.409 INFO:teuthology.orchestra.run.smithi161.stderr:Error ENOENT: problem getting command descriptions from mds.b 2024-03-28T23:35:39.410 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-28T23:35:40.412 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:35:40.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.523+0000 7f6ac02ed640 1 Processor -- start 2024-03-28T23:35:40.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.523+0000 7f6ac02ed640 1 -- start start 2024-03-28T23:35:40.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.524+0000 7f6ac02ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ab8150590 0x7f6ab80a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:40.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.524+0000 7f6ac02ed640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6ab80aacb0 0x7f6ab8153860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:40.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.524+0000 7f6ac02ed640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ab81566f0 con 0x7f6ab80a8fc0 2024-03-28T23:35:40.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.524+0000 7f6ac02ed640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6ab81569d0 con 0x7f6ab80aacb0 2024-03-28T23:35:40.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.524+0000 7f6ac02ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6ab8156cb0 con 0x7f6ab8150590 2024-03-28T23:35:40.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.524+0000 7f6abe88a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6ab80aacb0 0x7f6ab8153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:40.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.524+0000 7f6abe88a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6ab80aacb0 0x7f6ab8153860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58614/0 (socket says 172.21.15.161:58614) 2024-03-28T23:35:40.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.524+0000 7f6abe88a640 1 -- 172.21.15.161:0/3501614624 learned_addr learned my addr 172.21.15.161:0/3501614624 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:40.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.524+0000 7f6abe089640 1 --2- 172.21.15.161:0/3501614624 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ab8150590 0x7f6ab80a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:40.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.524+0000 7f6abe88a640 1 -- 172.21.15.161:0/3501614624 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ab8150590 msgr2=0x7f6ab80a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:40.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.524+0000 7f6abe88a640 1 --2- 172.21.15.161:0/3501614624 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ab8150590 0x7f6ab80a9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:40.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.524+0000 7f6abe88a640 1 -- 172.21.15.161:0/3501614624 >> v1:172.21.15.170:6789/0 conn(0x7f6ab80a8fc0 legacy=0x7f6ab8153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:40.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.524+0000 7f6abe88a640 1 -- 172.21.15.161:0/3501614624 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ab8156f90 con 0x7f6ab80aacb0 2024-03-28T23:35:40.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.525+0000 7f6abe88a640 1 --2- 172.21.15.161:0/3501614624 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ab80aacb0 0x7f6ab8153860 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0009e50 tx=0x7f6ab002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e82f43db4d76968 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:40.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.525+0000 7f6abd087640 1 -- 172.21.15.161:0/3501614624 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab002c9e0 con 0x7f6ab80aacb0 2024-03-28T23:35:40.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.525+0000 7f6abd087640 1 -- 172.21.15.161:0/3501614624 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ab002cb80 con 0x7f6ab80aacb0 2024-03-28T23:35:40.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.525+0000 7f6abd087640 1 -- 172.21.15.161:0/3501614624 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab00343f0 con 0x7f6ab80aacb0 2024-03-28T23:35:40.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.526+0000 7f6ac02ed640 1 -- 172.21.15.161:0/3501614624 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ab80aacb0 msgr2=0x7f6ab8153860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:40.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.526+0000 7f6ac02ed640 1 --2- 172.21.15.161:0/3501614624 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ab80aacb0 0x7f6ab8153860 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0009e50 tx=0x7f6ab002be40 comp rx=0 tx=0).stop 2024-03-28T23:35:40.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.526+0000 7f6ac02ed640 1 -- 172.21.15.161:0/3501614624 shutdown_connections 2024-03-28T23:35:40.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.526+0000 7f6ac02ed640 1 --2- 172.21.15.161:0/3501614624 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ab80aacb0 0x7f6ab8153860 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:40.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.526+0000 7f6ac02ed640 1 --2- 172.21.15.161:0/3501614624 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ab8150590 0x7f6ab80a9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:40.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.526+0000 7f6ac02ed640 1 -- 172.21.15.161:0/3501614624 >> 172.21.15.161:0/3501614624 conn(0x7f6ab80c6670 msgr2=0x7f6ab80c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:40.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.526+0000 7f6ac02ed640 1 -- 172.21.15.161:0/3501614624 shutdown_connections 2024-03-28T23:35:40.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.526+0000 7f6ac02ed640 1 -- 172.21.15.161:0/3501614624 wait complete. 2024-03-28T23:35:40.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.527+0000 7f6ac02ed640 1 Processor -- start 2024-03-28T23:35:40.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.527+0000 7f6ac02ed640 1 -- start start 2024-03-28T23:35:40.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.527+0000 7f6ac02ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ab80a8fc0 0x7f6ab81ecd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:40.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.527+0000 7f6ac02ed640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ab80aacb0 0x7f6ab81478e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:40.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.527+0000 7f6ac02ed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ab8150590 0x7f6ab8145420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:40.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.527+0000 7f6ac02ed640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab8150970 con 0x7f6ab80aacb0 2024-03-28T23:35:40.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.527+0000 7f6ac02ed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab80aa9c0 con 0x7f6ab8150590 2024-03-28T23:35:40.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.528+0000 7f6ac02ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6ab80aa480 con 0x7f6ab80a8fc0 2024-03-28T23:35:40.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.528+0000 7f6abd888640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ab80aacb0 0x7f6ab81478e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:40.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.528+0000 7f6abd888640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ab80aacb0 0x7f6ab81478e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58622/0 (socket says 172.21.15.161:58622) 2024-03-28T23:35:40.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.528+0000 7f6abd888640 1 -- 172.21.15.161:0/3943381922 learned_addr learned my addr 172.21.15.161:0/3943381922 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:40.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.528+0000 7f6abe089640 1 --2- 172.21.15.161:0/3943381922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ab80a8fc0 0x7f6ab81ecd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:40.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.528+0000 7f6abe88a640 1 --2- 172.21.15.161:0/3943381922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ab8150590 0x7f6ab8145420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:40.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.528+0000 7f6abd888640 1 -- 172.21.15.161:0/3943381922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ab80a8fc0 msgr2=0x7f6ab81ecd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:40.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.528+0000 7f6abd888640 1 --2- 172.21.15.161:0/3943381922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ab80a8fc0 0x7f6ab81ecd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:40.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.528+0000 7f6abd888640 1 -- 172.21.15.161:0/3943381922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ab8150590 msgr2=0x7f6ab8145420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:40.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.528+0000 7f6abd888640 1 --2- 172.21.15.161:0/3943381922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ab8150590 0x7f6ab8145420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:40.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.528+0000 7f6abd888640 1 -- 172.21.15.161:0/3943381922 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ab81e9030 con 0x7f6ab80aacb0 2024-03-28T23:35:40.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.528+0000 7f6abe88a640 1 --2- 172.21.15.161:0/3943381922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ab8150590 0x7f6ab8145420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:40.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.528+0000 7f6abd888640 1 --2- 172.21.15.161:0/3943381922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ab80aacb0 0x7f6ab81478e0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f6aa800c3f0 tx=0x7f6aa800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:40.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.528+0000 7f6aaeffd640 1 -- 172.21.15.161:0/3943381922 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa8017070 con 0x7f6ab80aacb0 2024-03-28T23:35:40.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.528+0000 7f6aaeffd640 1 -- 172.21.15.161:0/3943381922 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6aa80052f0 con 0x7f6ab80aacb0 2024-03-28T23:35:40.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.528+0000 7f6ac02ed640 1 -- 172.21.15.161:0/3943381922 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ab81e9310 con 0x7f6ab80aacb0 2024-03-28T23:35:40.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.528+0000 7f6aaeffd640 1 -- 172.21.15.161:0/3943381922 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa8003850 con 0x7f6ab80aacb0 2024-03-28T23:35:40.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.528+0000 7f6ac02ed640 1 -- 172.21.15.161:0/3943381922 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ab81e98d0 con 0x7f6ab80aacb0 2024-03-28T23:35:40.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.529+0000 7f6ac02ed640 1 -- 172.21.15.161:0/3943381922 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ab804f4b0 con 0x7f6ab80aacb0 2024-03-28T23:35:40.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.530+0000 7f6aaeffd640 1 -- 172.21.15.161:0/3943381922 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6aa801a070 con 0x7f6ab80aacb0 2024-03-28T23:35:40.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.530+0000 7f6aaeffd640 1 --2- 172.21.15.161:0/3943381922 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6a9c06b500 0x7f6a9c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:40.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.530+0000 7f6abe089640 1 --2- 172.21.15.161:0/3943381922 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6a9c06b500 0x7f6a9c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:40.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.530+0000 7f6aaeffd640 1 -- 172.21.15.161:0/3943381922 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 6331+0+0 (secure 0 0 0) 0x7f6aa8089630 con 0x7f6ab80aacb0 2024-03-28T23:35:40.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.531+0000 7f6abe089640 1 --2- 172.21.15.161:0/3943381922 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6a9c06b500 0x7f6a9c06d9c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f6ab40079a0 tx=0x7f6ab4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:40.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.535+0000 7f6aaeffd640 1 -- 172.21.15.161:0/3943381922 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6aa8057c80 con 0x7f6ab80aacb0 2024-03-28T23:35:40.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.742+0000 7f6ac02ed640 1 -- 172.21.15.161:0/3943381922 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ab80a9d20 con 0x7f6ab80aacb0 2024-03-28T23:35:40.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.742+0000 7f6aaeffd640 1 -- 172.21.15.161:0/3943381922 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) ==== 95+0+5198 (secure 0 0 0) 0x7f6aa805bd20 con 0x7f6ab80aacb0 2024-03-28T23:35:40.743 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:40.743 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":132,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7870,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2828648906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":2828648906},{"type":"v1","addr":"172.21.15.170:6835","nonce":2828648906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":132},{"gid":8409,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/3037890643","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3037890643},{"type":"v1","addr":"172.21.15.161:6837","nonce":3037890643}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120},{"gid":8622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/819329318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":819329318},{"type":"v1","addr":"172.21.15.161:6836","nonce":819329318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:12.982566+0000","modified":"2024-03-28T23:35:38.974405+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":140,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7639},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7639":{"gid":7639,"name":"d","rank":0,"incarnation":131,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.170:6834/3182285172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3182285172},{"type":"v1","addr":"172.21.15.170:6834","nonce":3182285172}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":10}]} 2024-03-28T23:35:40.743 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 132 2024-03-28T23:35:40.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.744+0000 7f6ac02ed640 1 -- 172.21.15.161:0/3943381922 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6a9c06b500 msgr2=0x7f6a9c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:40.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.744+0000 7f6ac02ed640 1 --2- 172.21.15.161:0/3943381922 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6a9c06b500 0x7f6a9c06d9c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f6ab40079a0 tx=0x7f6ab4008040 comp rx=0 tx=0).stop 2024-03-28T23:35:40.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.744+0000 7f6ac02ed640 1 -- 172.21.15.161:0/3943381922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ab80aacb0 msgr2=0x7f6ab81478e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:40.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.744+0000 7f6ac02ed640 1 --2- 172.21.15.161:0/3943381922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ab80aacb0 0x7f6ab81478e0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f6aa800c3f0 tx=0x7f6aa800c8b0 comp rx=0 tx=0).stop 2024-03-28T23:35:40.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.744+0000 7f6ac02ed640 1 -- 172.21.15.161:0/3943381922 shutdown_connections 2024-03-28T23:35:40.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.744+0000 7f6ac02ed640 1 --2- 172.21.15.161:0/3943381922 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6a9c06b500 0x7f6a9c06d9c0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:40.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.744+0000 7f6ac02ed640 1 --2- 172.21.15.161:0/3943381922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ab80aacb0 0x7f6ab81478e0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:40.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.744+0000 7f6ac02ed640 1 --2- 172.21.15.161:0/3943381922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ab8150590 0x7f6ab8145420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:40.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.744+0000 7f6ac02ed640 1 --2- 172.21.15.161:0/3943381922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ab80a8fc0 0x7f6ab81ecd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:40.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.744+0000 7f6ac02ed640 1 -- 172.21.15.161:0/3943381922 >> 172.21.15.161:0/3943381922 conn(0x7f6ab80c6670 msgr2=0x7f6ab81487f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:40.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.744+0000 7f6ac02ed640 1 -- 172.21.15.161:0/3943381922 shutdown_connections 2024-03-28T23:35:40.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:40.744+0000 7f6ac02ed640 1 -- 172.21.15.161:0/3943381922 wait complete. 2024-03-28T23:35:41.158 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 132, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:35:12.982566+0000', 'modified': '2024-03-28T23:35:38.974405+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 140, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7639}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7639': {'gid': 7639, 'name': 'd', 'rank': 0, 'incarnation': 131, 'state': 'up:reconnect', 'state_seq': 9, 'addr': '172.21.15.170:6834/3182285172', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 3182285172}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 3182285172}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-28T23:35:41.158 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7639:up:reconnect 2024-03-28T23:35:42.159 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:35:42.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.272+0000 7fbf819e5640 1 Processor -- start 2024-03-28T23:35:42.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.272+0000 7fbf819e5640 1 -- start start 2024-03-28T23:35:42.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.273+0000 7fbf819e5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c06d9f0 0x7fbf7c155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:42.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.273+0000 7fbf819e5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf7c06c3c0 con 0x7fbf7c06cb80 2024-03-28T23:35:42.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.273+0000 7fbf819e5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf7c0c5b90 con 0x7fbf7c152a80 2024-03-28T23:35:42.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.273+0000 7fbf819e5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbf7c156250 con 0x7fbf7c06d9f0 2024-03-28T23:35:42.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.273+0000 7fbf7b7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fbf7c06cb80 0x7fbf7c0c74c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38152/0 (socket says 172.21.15.161:38152) 2024-03-28T23:35:42.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.273+0000 7fbf7b7fe640 1 -- 172.21.15.161:0/3493939318 learned_addr learned my addr 172.21.15.161:0/3493939318 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:42.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.273+0000 7fbf73fff640 1 --2- 172.21.15.161:0/3493939318 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c06d9f0 0x7fbf7c155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:42.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.273+0000 7fbf7a7fc640 1 -- 172.21.15.161:0/3493939318 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2071213144 0 0) 0x7fbf7c06c3c0 con 0x7fbf7c06cb80 2024-03-28T23:35:42.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.273+0000 7fbf7a7fc640 1 -- 172.21.15.161:0/3493939318 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf68003520 con 0x7fbf7c06cb80 2024-03-28T23:35:42.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.273+0000 7fbf7a7fc640 1 -- 172.21.15.161:0/3493939318 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2541258488 0 0) 0x7fbf7c0c5b90 con 0x7fbf7c152a80 2024-03-28T23:35:42.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.273+0000 7fbf7a7fc640 1 -- 172.21.15.161:0/3493939318 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf7c06c3c0 con 0x7fbf7c152a80 2024-03-28T23:35:42.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.273+0000 7fbf73fff640 1 -- 172.21.15.161:0/3493939318 >> v1:172.21.15.170:6789/0 conn(0x7fbf7c152a80 legacy=0x7fbf7c0c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:42.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.273+0000 7fbf73fff640 1 -- 172.21.15.161:0/3493939318 >> v1:172.21.15.161:6789/0 conn(0x7fbf7c06cb80 legacy=0x7fbf7c0c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:42.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.273+0000 7fbf73fff640 1 -- 172.21.15.161:0/3493939318 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf7c0c55d0 con 0x7fbf7c06d9f0 2024-03-28T23:35:42.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.274+0000 7fbf73fff640 1 --2- 172.21.15.161:0/3493939318 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c06d9f0 0x7fbf7c155d10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbf640097d0 tx=0x7fbf6402b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22018ff0a11fa8b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:42.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.274+0000 7fbf7a7fc640 1 -- 172.21.15.161:0/3493939318 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf64031770 con 0x7fbf7c06d9f0 2024-03-28T23:35:42.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.274+0000 7fbf7a7fc640 1 -- 172.21.15.161:0/3493939318 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf64037070 con 0x7fbf7c06d9f0 2024-03-28T23:35:42.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.274+0000 7fbf7a7fc640 1 -- 172.21.15.161:0/3493939318 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf64031b10 con 0x7fbf7c06d9f0 2024-03-28T23:35:42.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.275+0000 7fbf819e5640 1 -- 172.21.15.161:0/3493939318 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c06d9f0 msgr2=0x7fbf7c155d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:42.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.275+0000 7fbf819e5640 1 --2- 172.21.15.161:0/3493939318 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c06d9f0 0x7fbf7c155d10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbf640097d0 tx=0x7fbf6402b810 comp rx=0 tx=0).stop 2024-03-28T23:35:42.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.275+0000 7fbf819e5640 1 -- 172.21.15.161:0/3493939318 shutdown_connections 2024-03-28T23:35:42.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.275+0000 7fbf819e5640 1 --2- 172.21.15.161:0/3493939318 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c06d9f0 0x7fbf7c155d10 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:42.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.275+0000 7fbf819e5640 1 -- 172.21.15.161:0/3493939318 >> 172.21.15.161:0/3493939318 conn(0x7fbf7c0ac720 msgr2=0x7fbf7c0acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:42.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.275+0000 7fbf819e5640 1 -- 172.21.15.161:0/3493939318 shutdown_connections 2024-03-28T23:35:42.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.275+0000 7fbf819e5640 1 -- 172.21.15.161:0/3493939318 wait complete. 2024-03-28T23:35:42.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.276+0000 7fbf819e5640 1 Processor -- start 2024-03-28T23:35:42.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.276+0000 7fbf819e5640 1 -- start start 2024-03-28T23:35:42.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.277+0000 7fbf819e5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf7c06cb80 0x7fbf7c154730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:42.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.277+0000 7fbf819e5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c06d9f0 0x7fbf7c14af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:42.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.277+0000 7fbf819e5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c152a80 0x7fbf7c14b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:42.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.277+0000 7fbf819e5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbf7c0c7bd0 con 0x7fbf7c06d9f0 2024-03-28T23:35:42.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.277+0000 7fbf819e5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbf7c06c6f0 con 0x7fbf7c06cb80 2024-03-28T23:35:42.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.277+0000 7fbf819e5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbf7c0c7d50 con 0x7fbf7c152a80 2024-03-28T23:35:42.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.277+0000 7fbf73fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c06d9f0 0x7fbf7c14af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:42.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.277+0000 7fbf73fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c06d9f0 0x7fbf7c14af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58636/0 (socket says 172.21.15.161:58636) 2024-03-28T23:35:42.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.277+0000 7fbf73fff640 1 -- 172.21.15.161:0/1966357304 learned_addr learned my addr 172.21.15.161:0/1966357304 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:42.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.277+0000 7fbf7b7fe640 1 --2- 172.21.15.161:0/1966357304 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c152a80 0x7fbf7c14b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:42.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.277+0000 7fbf7affd640 1 --2- 172.21.15.161:0/1966357304 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf7c06cb80 0x7fbf7c154730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:42.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.277+0000 7fbf73fff640 1 -- 172.21.15.161:0/1966357304 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c152a80 msgr2=0x7fbf7c14b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:42.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.277+0000 7fbf73fff640 1 --2- 172.21.15.161:0/1966357304 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c152a80 0x7fbf7c14b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:42.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.277+0000 7fbf73fff640 1 -- 172.21.15.161:0/1966357304 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf7c06cb80 msgr2=0x7fbf7c154730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:42.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.277+0000 7fbf73fff640 1 --2- 172.21.15.161:0/1966357304 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf7c06cb80 0x7fbf7c154730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:42.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.277+0000 7fbf73fff640 1 -- 172.21.15.161:0/1966357304 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf7c1eb020 con 0x7fbf7c06d9f0 2024-03-28T23:35:42.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.278+0000 7fbf73fff640 1 --2- 172.21.15.161:0/1966357304 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c06d9f0 0x7fbf7c14af00 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fbf64000c00 tx=0x7fbf640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:42.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.278+0000 7fbf78ff9640 1 -- 172.21.15.161:0/1966357304 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf64031ed0 con 0x7fbf7c06d9f0 2024-03-28T23:35:42.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.278+0000 7fbf78ff9640 1 -- 172.21.15.161:0/1966357304 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf64045600 con 0x7fbf7c06d9f0 2024-03-28T23:35:42.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.278+0000 7fbf819e5640 1 -- 172.21.15.161:0/1966357304 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf7c1eb300 con 0x7fbf7c06d9f0 2024-03-28T23:35:42.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.278+0000 7fbf78ff9640 1 -- 172.21.15.161:0/1966357304 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf640459b0 con 0x7fbf7c06d9f0 2024-03-28T23:35:42.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.278+0000 7fbf819e5640 1 -- 172.21.15.161:0/1966357304 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf7c1eb8c0 con 0x7fbf7c06d9f0 2024-03-28T23:35:42.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.279+0000 7fbf819e5640 1 -- 172.21.15.161:0/1966357304 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf7c04f4b0 con 0x7fbf7c06d9f0 2024-03-28T23:35:42.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.279+0000 7fbf78ff9640 1 -- 172.21.15.161:0/1966357304 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbf64031070 con 0x7fbf7c06d9f0 2024-03-28T23:35:42.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.280+0000 7fbf78ff9640 1 --2- 172.21.15.161:0/1966357304 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf5c06b4d0 0x7fbf5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:42.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.280+0000 7fbf7affd640 1 --2- 172.21.15.161:0/1966357304 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf5c06b4d0 0x7fbf5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:42.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.280+0000 7fbf78ff9640 1 -- 172.21.15.161:0/1966357304 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 6331+0+0 (secure 0 0 0) 0x7fbf640abbc0 con 0x7fbf7c06d9f0 2024-03-28T23:35:42.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.280+0000 7fbf7affd640 1 --2- 172.21.15.161:0/1966357304 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf5c06b4d0 0x7fbf5c06d990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c007fd0 tx=0x7fbf6c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:42.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.284+0000 7fbf78ff9640 1 -- 172.21.15.161:0/1966357304 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbf64040070 con 0x7fbf7c06d9f0 2024-03-28T23:35:42.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.492+0000 7fbf819e5640 1 -- 172.21.15.161:0/1966357304 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf7c14bcf0 con 0x7fbf7c06d9f0 2024-03-28T23:35:42.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.492+0000 7fbf78ff9640 1 -- 172.21.15.161:0/1966357304 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) ==== 95+0+5203 (secure 0 0 0) 0x7fbf6407a2b0 con 0x7fbf7c06d9f0 2024-03-28T23:35:42.493 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:42.493 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":134,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7870,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2828648906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":2828648906},{"type":"v1","addr":"172.21.15.170:6835","nonce":2828648906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":132},{"gid":8409,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/3037890643","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3037890643},{"type":"v1","addr":"172.21.15.161:6837","nonce":3037890643}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120},{"gid":8622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/819329318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":819329318},{"type":"v1","addr":"172.21.15.161:6836","nonce":819329318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:12.982566+0000","modified":"2024-03-28T23:35:41.970209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":140,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7639},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7639":{"gid":7639,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":11,"addr":"172.21.15.170:6834/3182285172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3182285172},{"type":"v1","addr":"172.21.15.170:6834","nonce":3182285172}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7639,"qdb_cluster":[7639]},"id":10}]} 2024-03-28T23:35:42.493 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 134 2024-03-28T23:35:42.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.495+0000 7fbf819e5640 1 -- 172.21.15.161:0/1966357304 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf5c06b4d0 msgr2=0x7fbf5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:42.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.495+0000 7fbf819e5640 1 --2- 172.21.15.161:0/1966357304 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf5c06b4d0 0x7fbf5c06d990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c007fd0 tx=0x7fbf6c009040 comp rx=0 tx=0).stop 2024-03-28T23:35:42.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.495+0000 7fbf819e5640 1 -- 172.21.15.161:0/1966357304 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c06d9f0 msgr2=0x7fbf7c14af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:42.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.495+0000 7fbf819e5640 1 --2- 172.21.15.161:0/1966357304 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c06d9f0 0x7fbf7c14af00 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fbf64000c00 tx=0x7fbf640046c0 comp rx=0 tx=0).stop 2024-03-28T23:35:42.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.495+0000 7fbf819e5640 1 -- 172.21.15.161:0/1966357304 shutdown_connections 2024-03-28T23:35:42.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.495+0000 7fbf819e5640 1 --2- 172.21.15.161:0/1966357304 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf5c06b4d0 0x7fbf5c06d990 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:42.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.495+0000 7fbf819e5640 1 --2- 172.21.15.161:0/1966357304 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf7c06d9f0 0x7fbf7c14af00 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:42.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.495+0000 7fbf819e5640 1 --2- 172.21.15.161:0/1966357304 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7c152a80 0x7fbf7c14b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:42.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.495+0000 7fbf819e5640 1 --2- 172.21.15.161:0/1966357304 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf7c06cb80 0x7fbf7c154730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:42.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.495+0000 7fbf819e5640 1 -- 172.21.15.161:0/1966357304 >> 172.21.15.161:0/1966357304 conn(0x7fbf7c0ac720 msgr2=0x7fbf7c0acb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:42.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.495+0000 7fbf819e5640 1 -- 172.21.15.161:0/1966357304 shutdown_connections 2024-03-28T23:35:42.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:42.495+0000 7fbf819e5640 1 -- 172.21.15.161:0/1966357304 wait complete. 2024-03-28T23:35:42.909 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 134, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:35:12.982566+0000', 'modified': '2024-03-28T23:35:41.970209+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 140, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7639}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7639': {'gid': 7639, 'name': 'd', 'rank': 0, 'incarnation': 131, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.170:6834/3182285172', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 3182285172}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 3182285172}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7639, 'qdb_cluster': [7639]} 2024-03-28T23:35:42.909 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:35:42.909 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-28T23:35:43.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.013+0000 7f0ca237a640 1 Processor -- start 2024-03-28T23:35:43.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.014+0000 7f0ca237a640 1 -- start start 2024-03-28T23:35:43.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.014+0000 7f0ca237a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0c9c0c5410 0x7f0c9c0c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.014+0000 7f0ca237a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0c9c1553a0 0x7f0c9c155850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.014+0000 7f0ca237a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c9c158a30 con 0x7f0c9c152890 2024-03-28T23:35:43.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.014+0000 7f0ca237a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0c9c0c8480 con 0x7f0c9c1553a0 2024-03-28T23:35:43.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.014+0000 7f0ca237a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f0c9c0c5d90 con 0x7f0c9c0c5410 2024-03-28T23:35:43.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.014+0000 7f0ca0917640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0c9c1553a0 0x7f0c9c155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.014+0000 7f0ca0917640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0c9c1553a0 0x7f0c9c155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58652/0 (socket says 172.21.15.161:58652) 2024-03-28T23:35:43.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.014+0000 7f0ca0917640 1 -- 172.21.15.161:0/1271878800 learned_addr learned my addr 172.21.15.161:0/1271878800 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:43.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.014+0000 7f0c9b7fe640 1 --2- 172.21.15.161:0/1271878800 >> v2:172.21.15.170:3300/0 conn(0x7f0c9c0c5410 0x7f0c9c0c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.015+0000 7f0ca0917640 1 -- 172.21.15.161:0/1271878800 >> v2:172.21.15.170:3300/0 conn(0x7f0c9c0c5410 msgr2=0x7f0c9c0c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:43.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.015+0000 7f0ca0917640 1 --2- 172.21.15.161:0/1271878800 >> v2:172.21.15.170:3300/0 conn(0x7f0c9c0c5410 0x7f0c9c0c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.015+0000 7f0ca0917640 1 -- 172.21.15.161:0/1271878800 >> v1:172.21.15.170:6789/0 conn(0x7f0c9c152890 legacy=0x7f0c9c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:43.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.015+0000 7f0ca0917640 1 -- 172.21.15.161:0/1271878800 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c9c158d10 con 0x7f0c9c1553a0 2024-03-28T23:35:43.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.015+0000 7f0ca0917640 1 --2- 172.21.15.161:0/1271878800 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c9c1553a0 0x7f0c9c155850 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c0ad6e0 tx=0x7f0c9402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bd86bbc8bb15aac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:43.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.016+0000 7f0c9affd640 1 -- 172.21.15.161:0/1271878800 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9402ca80 con 0x7f0c9c1553a0 2024-03-28T23:35:43.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.016+0000 7f0c9affd640 1 -- 172.21.15.161:0/1271878800 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c9402cc20 con 0x7f0c9c1553a0 2024-03-28T23:35:43.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.016+0000 7f0c9affd640 1 -- 172.21.15.161:0/1271878800 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c940343f0 con 0x7f0c9c1553a0 2024-03-28T23:35:43.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.016+0000 7f0ca237a640 1 -- 172.21.15.161:0/1271878800 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c9c1553a0 msgr2=0x7f0c9c155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:43.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.016+0000 7f0ca237a640 1 --2- 172.21.15.161:0/1271878800 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c9c1553a0 0x7f0c9c155850 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c0ad6e0 tx=0x7f0c9402bee0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.016+0000 7f0ca237a640 1 -- 172.21.15.161:0/1271878800 shutdown_connections 2024-03-28T23:35:43.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.016+0000 7f0ca237a640 1 --2- 172.21.15.161:0/1271878800 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c9c1553a0 0x7f0c9c155850 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.016+0000 7f0ca237a640 1 --2- 172.21.15.161:0/1271878800 >> v2:172.21.15.170:3300/0 conn(0x7f0c9c0c5410 0x7f0c9c0c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.016+0000 7f0ca237a640 1 -- 172.21.15.161:0/1271878800 >> 172.21.15.161:0/1271878800 conn(0x7f0c9c0ac620 msgr2=0x7f0c9c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:43.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.016+0000 7f0ca237a640 1 -- 172.21.15.161:0/1271878800 shutdown_connections 2024-03-28T23:35:43.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.016+0000 7f0ca237a640 1 -- 172.21.15.161:0/1271878800 wait complete. 2024-03-28T23:35:43.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.017+0000 7f0ca237a640 1 Processor -- start 2024-03-28T23:35:43.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.017+0000 7f0ca237a640 1 -- start start 2024-03-28T23:35:43.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.018+0000 7f0ca237a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c9c0c5410 0x7f0c9c14a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.018+0000 7f0ca237a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c9c152890 0x7f0c9c147e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.018+0000 7f0ca237a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c9c1553a0 0x7f0c9c148350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.018+0000 7f0ca237a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9c1592d0 con 0x7f0c9c1553a0 2024-03-28T23:35:43.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.018+0000 7f0ca237a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9c159640 con 0x7f0c9c152890 2024-03-28T23:35:43.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.018+0000 7f0ca237a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0c9c0c90a0 con 0x7f0c9c0c5410 2024-03-28T23:35:43.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.018+0000 7f0ca0917640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c9c1553a0 0x7f0c9c148350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.018+0000 7f0ca0917640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c9c1553a0 0x7f0c9c148350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58656/0 (socket says 172.21.15.161:58656) 2024-03-28T23:35:43.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.018+0000 7f0ca0917640 1 -- 172.21.15.161:0/381811800 learned_addr learned my addr 172.21.15.161:0/381811800 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:43.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.018+0000 7f0c9b7fe640 1 --2- 172.21.15.161:0/381811800 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c9c152890 0x7f0c9c147e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.018+0000 7f0c9bfff640 1 --2- 172.21.15.161:0/381811800 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c9c0c5410 0x7f0c9c14a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.018+0000 7f0ca0917640 1 -- 172.21.15.161:0/381811800 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c9c0c5410 msgr2=0x7f0c9c14a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:43.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.018+0000 7f0ca0917640 1 --2- 172.21.15.161:0/381811800 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c9c0c5410 0x7f0c9c14a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.018+0000 7f0ca0917640 1 -- 172.21.15.161:0/381811800 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c9c152890 msgr2=0x7f0c9c147e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:43.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.018+0000 7f0ca0917640 1 --2- 172.21.15.161:0/381811800 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c9c152890 0x7f0c9c147e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.018+0000 7f0ca0917640 1 -- 172.21.15.161:0/381811800 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c9c0be900 con 0x7f0c9c1553a0 2024-03-28T23:35:43.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.019+0000 7f0ca0917640 1 --2- 172.21.15.161:0/381811800 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c9c1553a0 0x7f0c9c148350 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f0c94009f40 tx=0x7f0c94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:43.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.019+0000 7f0c98ff9640 1 -- 172.21.15.161:0/381811800 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c94043020 con 0x7f0c9c1553a0 2024-03-28T23:35:43.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.019+0000 7f0c98ff9640 1 -- 172.21.15.161:0/381811800 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c9403e450 con 0x7f0c9c1553a0 2024-03-28T23:35:43.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.019+0000 7f0c98ff9640 1 -- 172.21.15.161:0/381811800 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c94009510 con 0x7f0c9c1553a0 2024-03-28T23:35:43.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.019+0000 7f0ca237a640 1 -- 172.21.15.161:0/381811800 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c9c0bebe0 con 0x7f0c9c1553a0 2024-03-28T23:35:43.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.019+0000 7f0ca237a640 1 -- 172.21.15.161:0/381811800 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c9c14a640 con 0x7f0c9c1553a0 2024-03-28T23:35:43.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.021+0000 7f0c98ff9640 1 -- 172.21.15.161:0/381811800 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0c94046020 con 0x7f0c9c1553a0 2024-03-28T23:35:43.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.021+0000 7f0c98ff9640 1 --2- 172.21.15.161:0/381811800 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0c8806b4d0 0x7f0c8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.021+0000 7f0c98ff9640 1 -- 172.21.15.161:0/381811800 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 6331+0+0 (secure 0 0 0) 0x7f0c940acab0 con 0x7f0c9c1553a0 2024-03-28T23:35:43.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.021+0000 7f0c9bfff640 1 --2- 172.21.15.161:0/381811800 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0c8806b4d0 0x7f0c8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.021+0000 7f0c9bfff640 1 --2- 172.21.15.161:0/381811800 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0c8806b4d0 0x7f0c8806d990 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c0c4eb0 tx=0x7f0c8c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:43.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.022+0000 7f0ca4bfd740 1 Processor -- start 2024-03-28T23:35:43.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.022+0000 7f0ca4bfd740 1 -- start start 2024-03-28T23:35:43.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.022+0000 7f0ca4bfd740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5573987fbdb0 0x5573987f9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.022+0000 7f0ca4bfd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573987fc190 0x5573987f9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.022+0000 7f0ca4bfd740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573987fa310 0x5573987f64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.022+0000 7f0ca4bfd740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55739875faf0 con 0x5573987fa310 2024-03-28T23:35:43.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.023+0000 7f0ca4bfd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5573987089d0 con 0x5573987fc190 2024-03-28T23:35:43.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.023+0000 7f0ca4bfd740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55739875bb90 con 0x5573987fbdb0 2024-03-28T23:35:43.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.023+0000 7f0ca0917640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573987fa310 0x5573987f64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.023+0000 7f0ca0917640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573987fa310 0x5573987f64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58664/0 (socket says 172.21.15.161:58664) 2024-03-28T23:35:43.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.023+0000 7f0ca0917640 1 -- 172.21.15.161:0/970797184 learned_addr learned my addr 172.21.15.161:0/970797184 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:43.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.023+0000 7f0c9b7fe640 1 --2- 172.21.15.161:0/970797184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573987fc190 0x5573987f9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.023+0000 7f0c9bfff640 1 --2- 172.21.15.161:0/970797184 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5573987fbdb0 0x5573987f9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.023+0000 7f0ca0917640 1 -- 172.21.15.161:0/970797184 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5573987fbdb0 msgr2=0x5573987f9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:43.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.023+0000 7f0ca0917640 1 --2- 172.21.15.161:0/970797184 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5573987fbdb0 0x5573987f9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.023+0000 7f0ca0917640 1 -- 172.21.15.161:0/970797184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573987fc190 msgr2=0x5573987f9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:43.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.023+0000 7f0ca0917640 1 --2- 172.21.15.161:0/970797184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573987fc190 0x5573987f9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.023+0000 7f0ca0917640 1 -- 172.21.15.161:0/970797184 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573984742d0 con 0x5573987fa310 2024-03-28T23:35:43.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.023+0000 7f0c9bfff640 1 --2- 172.21.15.161:0/970797184 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5573987fbdb0 0x5573987f9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:43.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.023+0000 7f0ca0917640 1 --2- 172.21.15.161:0/970797184 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573987fa310 0x5573987f64e0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f0c94077680 tx=0x7f0c9407b0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbf39869fb6c2137 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:43.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.023+0000 7f0c81ffb640 1 -- 172.21.15.161:0/970797184 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9407f2c0 con 0x5573987fa310 2024-03-28T23:35:43.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.023+0000 7f0c81ffb640 1 -- 172.21.15.161:0/970797184 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c94083880 con 0x5573987fa310 2024-03-28T23:35:43.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.023+0000 7f0c81ffb640 1 -- 172.21.15.161:0/970797184 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c940792c0 con 0x5573987fa310 2024-03-28T23:35:43.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.024+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/970797184 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573987fa310 msgr2=0x5573987f64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:43.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.024+0000 7f0ca4bfd740 1 --2- 172.21.15.161:0/970797184 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573987fa310 0x5573987f64e0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f0c94077680 tx=0x7f0c9407b0c0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.024+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/970797184 shutdown_connections 2024-03-28T23:35:43.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.024+0000 7f0ca4bfd740 1 --2- 172.21.15.161:0/970797184 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573987fa310 0x5573987f64e0 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.024+0000 7f0ca4bfd740 1 --2- 172.21.15.161:0/970797184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573987fc190 0x5573987f9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.024+0000 7f0ca4bfd740 1 --2- 172.21.15.161:0/970797184 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5573987fbdb0 0x5573987f9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.024+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/970797184 >> 172.21.15.161:0/970797184 conn(0x5573987f8630 msgr2=0x55739881d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:43.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.024+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/970797184 shutdown_connections 2024-03-28T23:35:43.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.024+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/970797184 wait complete. 2024-03-28T23:35:43.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.024+0000 7f0ca4bfd740 1 -- start start 2024-03-28T23:35:43.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.025+0000 7f0ca4bfd740 1 Processor -- start 2024-03-28T23:35:43.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.025+0000 7f0ca4bfd740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5573987fbdb0 0x557398819090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.025+0000 7f0ca4bfd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573987fc190 0x55739884b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.025+0000 7f0ca4bfd740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557398820f40 0x55739881c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.025+0000 7f0ca4bfd740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55739881e090 con 0x557398820f40 2024-03-28T23:35:43.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.025+0000 7f0ca4bfd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55739871bf70 con 0x5573987fc190 2024-03-28T23:35:43.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.025+0000 7f0ca4bfd740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55739881dad0 con 0x5573987fbdb0 2024-03-28T23:35:43.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.025+0000 7f0ca4bfd740 10 client.0 _ll_register_callbacks cb 0x557398473ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:35:43.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.025+0000 7f0ca0917640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557398820f40 0x55739881c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.025+0000 7f0ca0917640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557398820f40 0x55739881c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58680/0 (socket says 172.21.15.161:58680) 2024-03-28T23:35:43.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.025+0000 7f0ca0917640 1 -- 172.21.15.161:0/2532917810 learned_addr learned my addr 172.21.15.161:0/2532917810 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:43.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.025+0000 7f0c9bfff640 1 --2- 172.21.15.161:0/2532917810 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5573987fbdb0 0x557398819090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.025+0000 7f0c9b7fe640 1 --2- 172.21.15.161:0/2532917810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573987fc190 0x55739884b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.026+0000 7f0ca0917640 1 -- 172.21.15.161:0/2532917810 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5573987fbdb0 msgr2=0x557398819090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:43.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.026+0000 7f0ca0917640 1 --2- 172.21.15.161:0/2532917810 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5573987fbdb0 0x557398819090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.026+0000 7f0ca0917640 1 -- 172.21.15.161:0/2532917810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573987fc190 msgr2=0x55739884b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:43.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.026+0000 7f0ca0917640 1 --2- 172.21.15.161:0/2532917810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573987fc190 0x55739884b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.026+0000 7f0ca0917640 1 -- 172.21.15.161:0/2532917810 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55739853e6b0 con 0x557398820f40 2024-03-28T23:35:43.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.026+0000 7f0ca0917640 1 --2- 172.21.15.161:0/2532917810 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557398820f40 0x55739881c210 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f0c94009f10 tx=0x7f0c94009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81e5d18026a55fc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:43.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.026+0000 7f0c7affd640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:35:43.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.026+0000 7f0c7affd640 1 -- 172.21.15.161:0/2532917810 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c94087070 con 0x557398820f40 2024-03-28T23:35:43.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.026+0000 7f0c7affd640 1 -- 172.21.15.161:0/2532917810 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(140..140 src has 1..140) ==== 6331+0+0 (secure 0 0 0) 0x7f0c94091040 con 0x557398820f40 2024-03-28T23:35:43.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.026+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/2532917810 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c94083880 con 0x557398820f40 2024-03-28T23:35:43.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.026+0000 7f0c7affd640 1 -- 172.21.15.161:0/2532917810 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=134) ==== 24+0+0 (secure 0 0 0) 0x7f0c9407f2c0 con 0x557398820f40 2024-03-28T23:35:43.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.026+0000 7f0ca4bfd740 10 client.8739 fetch_fsmap learned FSMap version 134 2024-03-28T23:35:43.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.026+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/2532917810 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=134,monmap=2+}) -- 0x55739884bb80 con 0x557398820f40 2024-03-28T23:35:43.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.026+0000 7f0c7affd640 1 -- 172.21.15.161:0/2532917810 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c94083880 con 0x557398820f40 2024-03-28T23:35:43.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.026+0000 7f0c7affd640 1 -- 172.21.15.161:0/2532917810 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 134) ==== 3494+0+0 (secure 0 0 0) 0x7f0c9402ca90 con 0x557398820f40 2024-03-28T23:35:43.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.026+0000 7f0ca4bfd740 10 client.8739 fetch_fsmap finished waiting for FSMap version 134 2024-03-28T23:35:43.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.026+0000 7f0ca4bfd740 10 client.8739 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-28T23:35:43.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.026+0000 7f0ca4bfd740 20 client.8739 populate_metadata read hostname 'smithi161' 2024-03-28T23:35:43.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.026+0000 7f0ca4bfd740 1 --2- 172.21.15.161:0/2532917810 >> [v2:172.21.15.170:6832/3182285172,v1:172.21.15.170:6834/3182285172] conn(0x557398848250 0x557398848630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.026+0000 7f0ca4bfd740 4 client.8739 mds_command: new command op to 7639 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:35:43.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.026+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/2532917810 --> [v2:172.21.15.170:6832/3182285172,v1:172.21.15.170:6834/3182285172] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55739876d030 con 0x557398848250 2024-03-28T23:35:43.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.027+0000 7f0c9bfff640 1 --2- 172.21.15.161:0/2532917810 >> [v2:172.21.15.170:6832/3182285172,v1:172.21.15.170:6834/3182285172] conn(0x557398848250 0x557398848630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.027+0000 7f0c7affd640 10 client.8739 ms_handle_connect on v2:172.21.15.170:6832/3182285172 2024-03-28T23:35:43.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.027+0000 7f0c9bfff640 1 --2- 172.21.15.161:0/2532917810 >> [v2:172.21.15.170:6832/3182285172,v1:172.21.15.170:6834/3182285172] conn(0x557398848250 0x557398848630 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17e3b5706dab5e2d server_cookie=65d768882ac38005 in_seq=0 out_seq=0 2024-03-28T23:35:43.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.027+0000 7f0c7affd640 1 -- 172.21.15.161:0/2532917810 <== mds.0 v2:172.21.15.170:6832/3182285172 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55739876d030 con 0x557398848250 2024-03-28T23:35:43.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.027+0000 7f0c7affd640 10 client.8739 handle_command_reply: tid=0 2024-03-28T23:35:43.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.028+0000 7f0ca4bfd740 1 client.8739 shutdown 2024-03-28T23:35:43.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.028+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/2532917810 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557398820f40 msgr2=0x55739881c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:43.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.028+0000 7f0ca4bfd740 1 --2- 172.21.15.161:0/2532917810 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557398820f40 0x55739881c210 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f0c94009f10 tx=0x7f0c94009810 comp rx=0 tx=0).stop 2024-03-28T23:35:43.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.028+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/2532917810 shutdown_connections 2024-03-28T23:35:43.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.028+0000 7f0ca4bfd740 1 --2- 172.21.15.161:0/2532917810 >> [v2:172.21.15.170:6832/3182285172,v1:172.21.15.170:6834/3182285172] conn(0x557398848250 0x557398848630 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.028+0000 7f0ca4bfd740 1 --2- 172.21.15.161:0/2532917810 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557398820f40 0x55739881c210 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.028+0000 7f0ca4bfd740 1 --2- 172.21.15.161:0/2532917810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573987fc190 0x55739884b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.028+0000 7f0ca4bfd740 1 --2- 172.21.15.161:0/2532917810 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5573987fbdb0 0x557398819090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.028+0000 7f0c7affd640 0 client.8739 ms_handle_reset on v2:172.21.15.170:6832/3182285172 2024-03-28T23:35:43.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.028+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/2532917810 >> 172.21.15.161:0/2532917810 conn(0x5573987fa310 msgr2=0x5573987fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:43.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.028+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/2532917810 shutdown_connections 2024-03-28T23:35:43.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.028+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/2532917810 wait complete. 2024-03-28T23:35:43.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.028+0000 7f0ca4bfd740 20 client.8739 trim_cache size 0 max 16384 2024-03-28T23:35:43.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.046+0000 7f0ca4bfd740 1 Processor -- start 2024-03-28T23:35:43.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.046+0000 7f0ca4bfd740 1 -- start start 2024-03-28T23:35:43.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.046+0000 7f0ca4bfd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573987fbdb0 0x5573987f56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.046+0000 7f0ca4bfd740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5573987fc190 0x557398848af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.046+0000 7f0ca4bfd740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557398820f40 0x5573987f91d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.046+0000 7f0ca4bfd740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9c1f7080 con 0x557398820f40 2024-03-28T23:35:43.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.046+0000 7f0ca4bfd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9c1f6530 con 0x5573987fbdb0 2024-03-28T23:35:43.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.046+0000 7f0ca4bfd740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0c9c1f6a60 con 0x5573987fc190 2024-03-28T23:35:43.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.046+0000 7f0ca0917640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557398820f40 0x5573987f91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.046+0000 7f0ca0917640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557398820f40 0x5573987f91d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58694/0 (socket says 172.21.15.161:58694) 2024-03-28T23:35:43.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.046+0000 7f0ca0917640 1 -- 172.21.15.161:0/3547153343 learned_addr learned my addr 172.21.15.161:0/3547153343 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:43.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.046+0000 7f0c9b7fe640 1 --2- 172.21.15.161:0/3547153343 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5573987fc190 0x557398848af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.046+0000 7f0c9bfff640 1 --2- 172.21.15.161:0/3547153343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573987fbdb0 0x5573987f56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.047+0000 7f0ca0917640 1 -- 172.21.15.161:0/3547153343 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5573987fc190 msgr2=0x557398848af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:43.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.047+0000 7f0ca0917640 1 --2- 172.21.15.161:0/3547153343 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5573987fc190 0x557398848af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.047+0000 7f0ca0917640 1 -- 172.21.15.161:0/3547153343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573987fbdb0 msgr2=0x5573987f56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:43.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.047+0000 7f0ca0917640 1 --2- 172.21.15.161:0/3547153343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573987fbdb0 0x5573987f56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.047+0000 7f0ca0917640 1 -- 172.21.15.161:0/3547153343 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573987f5040 con 0x557398820f40 2024-03-28T23:35:43.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.047+0000 7f0c9bfff640 1 --2- 172.21.15.161:0/3547153343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573987fbdb0 0x5573987f56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:43.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.047+0000 7f0ca0917640 1 --2- 172.21.15.161:0/3547153343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557398820f40 0x5573987f91d0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f0c94009c90 tx=0x7f0c940a9000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=577be45a2fabb1ff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:43.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.047+0000 7f0c817fa640 1 -- 172.21.15.161:0/3547153343 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c94087040 con 0x557398820f40 2024-03-28T23:35:43.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.047+0000 7f0c817fa640 1 -- 172.21.15.161:0/3547153343 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c94083880 con 0x557398820f40 2024-03-28T23:35:43.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.047+0000 7f0c817fa640 1 -- 172.21.15.161:0/3547153343 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9402ca90 con 0x557398820f40 2024-03-28T23:35:43.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.047+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/3547153343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557398820f40 msgr2=0x5573987f91d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:43.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.047+0000 7f0ca4bfd740 1 --2- 172.21.15.161:0/3547153343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557398820f40 0x5573987f91d0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f0c94009c90 tx=0x7f0c940a9000 comp rx=0 tx=0).stop 2024-03-28T23:35:43.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.047+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/3547153343 shutdown_connections 2024-03-28T23:35:43.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.047+0000 7f0ca4bfd740 1 --2- 172.21.15.161:0/3547153343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557398820f40 0x5573987f91d0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.047+0000 7f0ca4bfd740 1 --2- 172.21.15.161:0/3547153343 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5573987fc190 0x557398848af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.047+0000 7f0ca4bfd740 1 --2- 172.21.15.161:0/3547153343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573987fbdb0 0x5573987f56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.047+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/3547153343 >> 172.21.15.161:0/3547153343 conn(0x5573987fa310 msgr2=0x5573987f1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:43.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.047+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/3547153343 shutdown_connections 2024-03-28T23:35:43.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.047+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/3547153343 wait complete. 2024-03-28T23:35:43.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.048+0000 7f0ca4bfd740 1 -- start start 2024-03-28T23:35:43.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.048+0000 7f0ca4bfd740 1 Processor -- start 2024-03-28T23:35:43.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.049+0000 7f0ca4bfd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573987fbdb0 0x5573988b5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.049+0000 7f0ca4bfd740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5573987fc190 0x55739885b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.049+0000 7f0ca4bfd740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557398820f40 0x5573988ac1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.049+0000 7f0ca4bfd740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55739881dad0 con 0x557398820f40 2024-03-28T23:35:43.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.049+0000 7f0ca4bfd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5573988480d0 con 0x5573987fbdb0 2024-03-28T23:35:43.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.049+0000 7f0ca4bfd740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55739881e090 con 0x5573987fc190 2024-03-28T23:35:43.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.049+0000 7f0ca4bfd740 10 client.0 _ll_register_callbacks cb 0x7f0c94077570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:35:43.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.049+0000 7f0ca0917640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557398820f40 0x5573988ac1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.049+0000 7f0c9bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573987fbdb0 0x5573988b5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.049+0000 7f0ca0917640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557398820f40 0x5573988ac1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58708/0 (socket says 172.21.15.161:58708) 2024-03-28T23:35:43.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.049+0000 7f0ca0917640 1 -- 172.21.15.161:0/2587546653 learned_addr learned my addr 172.21.15.161:0/2587546653 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:43.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.049+0000 7f0c9b7fe640 1 --2- 172.21.15.161:0/2587546653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5573987fc190 0x55739885b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.049+0000 7f0ca0917640 1 -- 172.21.15.161:0/2587546653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5573987fc190 msgr2=0x55739885b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:43.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.049+0000 7f0ca0917640 1 --2- 172.21.15.161:0/2587546653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5573987fc190 0x55739885b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.049+0000 7f0ca0917640 1 -- 172.21.15.161:0/2587546653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573987fbdb0 msgr2=0x5573988b5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:43.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.049+0000 7f0ca0917640 1 --2- 172.21.15.161:0/2587546653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573987fbdb0 0x5573988b5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.049+0000 7f0ca0917640 1 -- 172.21.15.161:0/2587546653 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55739881b980 con 0x557398820f40 2024-03-28T23:35:43.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.049+0000 7f0ca0917640 1 --2- 172.21.15.161:0/2587546653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557398820f40 0x5573988ac1a0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f0c94009360 tx=0x7f0c94009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd161c28f8d0f497 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:43.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.049+0000 7f0c7affd640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:35:43.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.049+0000 7f0c7affd640 1 -- 172.21.15.161:0/2587546653 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c94087070 con 0x557398820f40 2024-03-28T23:35:43.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.050+0000 7f0c7affd640 1 -- 172.21.15.161:0/2587546653 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(140..140 src has 1..140) ==== 6331+0+0 (secure 0 0 0) 0x7f0c94049d20 con 0x557398820f40 2024-03-28T23:35:43.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.050+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/2587546653 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c94083880 con 0x557398820f40 2024-03-28T23:35:43.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.050+0000 7f0c7affd640 1 -- 172.21.15.161:0/2587546653 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=134) ==== 24+0+0 (secure 0 0 0) 0x7f0c9409e050 con 0x557398820f40 2024-03-28T23:35:43.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.050+0000 7f0ca4bfd740 10 client.8745 fetch_fsmap learned FSMap version 134 2024-03-28T23:35:43.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.050+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/2587546653 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=134,monmap=2+}) -- 0x55739881b980 con 0x557398820f40 2024-03-28T23:35:43.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.050+0000 7f0c7affd640 1 -- 172.21.15.161:0/2587546653 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c94083880 con 0x557398820f40 2024-03-28T23:35:43.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.050+0000 7f0c7affd640 1 -- 172.21.15.161:0/2587546653 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 134) ==== 3494+0+0 (secure 0 0 0) 0x7f0c940a1020 con 0x557398820f40 2024-03-28T23:35:43.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.050+0000 7f0ca4bfd740 10 client.8745 fetch_fsmap finished waiting for FSMap version 134 2024-03-28T23:35:43.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.050+0000 7f0ca4bfd740 10 client.8745 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-28T23:35:43.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.050+0000 7f0ca4bfd740 20 client.8745 populate_metadata read hostname 'smithi161' 2024-03-28T23:35:43.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.050+0000 7f0ca4bfd740 1 --2- 172.21.15.161:0/2587546653 >> [v2:172.21.15.170:6832/3182285172,v1:172.21.15.170:6834/3182285172] conn(0x5573988ac750 0x5573988b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.050+0000 7f0ca4bfd740 4 client.8745 mds_command: new command op to 7639 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:35:43.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.050+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/2587546653 --> [v2:172.21.15.170:6832/3182285172,v1:172.21.15.170:6834/3182285172] -- command(tid 0: {"prefix": "status"}) -- 0x557398820150 con 0x5573988ac750 2024-03-28T23:35:43.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.050+0000 7f0c9bfff640 1 --2- 172.21.15.161:0/2587546653 >> [v2:172.21.15.170:6832/3182285172,v1:172.21.15.170:6834/3182285172] conn(0x5573988ac750 0x5573988b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.050+0000 7f0c7affd640 10 client.8745 ms_handle_connect on v2:172.21.15.170:6832/3182285172 2024-03-28T23:35:43.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.050+0000 7f0c9bfff640 1 --2- 172.21.15.161:0/2587546653 >> [v2:172.21.15.170:6832/3182285172,v1:172.21.15.170:6834/3182285172] conn(0x5573988ac750 0x5573988b2480 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c99bbc3f785c4825 server_cookie=5c9dd4bf898238f in_seq=0 out_seq=0 2024-03-28T23:35:43.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.051+0000 7f0c7affd640 1 -- 172.21.15.161:0/2587546653 <== mds.0 v2:172.21.15.170:6832/3182285172 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x557398820150 con 0x5573988ac750 2024-03-28T23:35:43.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.051+0000 7f0c7affd640 10 client.8745 handle_command_reply: tid=0 2024-03-28T23:35:43.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.051+0000 7f0ca4bfd740 1 client.8745 shutdown 2024-03-28T23:35:43.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.051+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/2587546653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557398820f40 msgr2=0x5573988ac1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:43.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.051+0000 7f0ca4bfd740 1 --2- 172.21.15.161:0/2587546653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557398820f40 0x5573988ac1a0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f0c94009360 tx=0x7f0c94009040 comp rx=0 tx=0).stop 2024-03-28T23:35:43.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.051+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/2587546653 shutdown_connections 2024-03-28T23:35:43.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.051+0000 7f0ca4bfd740 1 --2- 172.21.15.161:0/2587546653 >> [v2:172.21.15.170:6832/3182285172,v1:172.21.15.170:6834/3182285172] conn(0x5573988ac750 0x5573988b2480 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.051+0000 7f0ca4bfd740 1 --2- 172.21.15.161:0/2587546653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557398820f40 0x5573988ac1a0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.051+0000 7f0c7affd640 0 client.8745 ms_handle_reset on v2:172.21.15.170:6832/3182285172 2024-03-28T23:35:43.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.051+0000 7f0ca4bfd740 1 --2- 172.21.15.161:0/2587546653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5573987fc190 0x55739885b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.051+0000 7f0ca4bfd740 1 --2- 172.21.15.161:0/2587546653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5573987fbdb0 0x5573988b5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.051+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/2587546653 >> 172.21.15.161:0/2587546653 conn(0x5573987fa310 msgr2=0x5573987fa6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:43.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.051+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/2587546653 shutdown_connections 2024-03-28T23:35:43.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.051+0000 7f0ca4bfd740 1 -- 172.21.15.161:0/2587546653 wait complete. 2024-03-28T23:35:43.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.051+0000 7f0ca4bfd740 20 client.8745 trim_cache size 0 max 16384 2024-03-28T23:35:43.060 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:35:43.060 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:35:43.060 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:35:43.060 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 7639, 2024-03-28T23:35:43.060 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:35:43.060 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:35:43.060 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:35:43.060 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 4.0865857649999997, 2024-03-28T23:35:43.060 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 134, 2024-03-28T23:35:43.060 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 140, 2024-03-28T23:35:43.061 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 140, 2024-03-28T23:35:43.061 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 34.500496677999998 2024-03-28T23:35:43.061 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:35:43.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.051+0000 7f0ca237a640 1 -- 172.21.15.161:0/381811800 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0c8806b4d0 msgr2=0x7f0c8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:43.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.051+0000 7f0ca237a640 1 --2- 172.21.15.161:0/381811800 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0c8806b4d0 0x7f0c8806d990 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c0c4eb0 tx=0x7f0c8c009040 comp rx=0 tx=0).stop 2024-03-28T23:35:43.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.051+0000 7f0ca237a640 1 -- 172.21.15.161:0/381811800 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c9c1553a0 msgr2=0x7f0c9c148350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:43.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.051+0000 7f0ca237a640 1 --2- 172.21.15.161:0/381811800 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c9c1553a0 0x7f0c9c148350 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f0c94009f40 tx=0x7f0c94004a40 comp rx=0 tx=0).stop 2024-03-28T23:35:43.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.052+0000 7f0ca237a640 1 -- 172.21.15.161:0/381811800 shutdown_connections 2024-03-28T23:35:43.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.052+0000 7f0ca237a640 1 --2- 172.21.15.161:0/381811800 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0c8806b4d0 0x7f0c8806d990 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.052+0000 7f0ca237a640 1 --2- 172.21.15.161:0/381811800 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c9c1553a0 0x7f0c9c148350 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.052+0000 7f0ca237a640 1 --2- 172.21.15.161:0/381811800 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c9c152890 0x7f0c9c147e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.052+0000 7f0ca237a640 1 --2- 172.21.15.161:0/381811800 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c9c0c5410 0x7f0c9c14a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.052+0000 7f0ca237a640 1 -- 172.21.15.161:0/381811800 >> 172.21.15.161:0/381811800 conn(0x7f0c9c0ac620 msgr2=0x7f0c9c06bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:43.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.052+0000 7f0ca237a640 1 -- 172.21.15.161:0/381811800 shutdown_connections 2024-03-28T23:35:43.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.052+0000 7f0ca237a640 1 -- 172.21.15.161:0/381811800 wait complete. 2024-03-28T23:35:43.063 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 134, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:35:12.982566+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 134, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7639": { "addr": "172.21.15.170:6834/3182285172", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 3182285172, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 3182285172, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7639, "incarnation": 131, "join_fscid": 10, "name": "d", "rank": 0, "state": "up:active", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 140, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-03-28T23:35:41.970209+0000", "qdb_cluster": [ 7639 ], "qdb_leader": 7639, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7639 } } } ], "standbys": [ { "addr": "172.21.15.170:6835/2828648906", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 2828648906, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 2828648906, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 132, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7870, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6837/3037890643", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6835", "nonce": 3037890643, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 3037890643, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 120, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8409, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6836/819329318", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 819329318, "type": "v2" }, { "addr": "172.21.15.161:6836", "nonce": 819329318, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 127, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8622, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:35:43.064 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 10, 'name': 'd', 'state': 'up:active'}] 2024-03-28T23:35:43.064 INFO:tasks.cephfs.test_failover:current = [{'gid': 8622, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6836/819329318', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 819329318}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 819329318}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 127}, {'gid': 7870, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/2828648906', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 2828648906}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 2828648906}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 132}, {'gid': 8409, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6837/3037890643', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 3037890643}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 3037890643}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 120}, {'gid': 7639, 'name': 'd', 'rank': 0, 'incarnation': 131, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.170:6834/3182285172', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 3182285172}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 3182285172}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}] 2024-03-28T23:35:43.065 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-28T23:35:43.065 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-28T23:35:43.065 DEBUG:teuthology.orchestra.run.smithi161:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:35:43.108 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:35:43.154 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-28T23:35:43.155 INFO:tasks.cephfs.mount:Terminating background process 2024-03-28T23:35:43.155 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:35:43.155 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:35:43.170 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20793 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-28T23:35:43.170 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi161.front.sepia.ceph.com 2024-03-28T23:35:43.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:35:43.170 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-28T23:35:43.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:35:43.224 INFO:teuthology.orchestra.run.smithi161.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-28T23:35:43.225 INFO:tasks.cephfs.mount:Terminating background process 2024-03-28T23:35:43.225 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:35:43.225 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:35:43.240 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056374,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19690 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-28T23:35:43.240 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi170.front.sepia.ceph.com 2024-03-28T23:35:43.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:35:43.240 DEBUG:teuthology.orchestra.run.smithi170:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-28T23:35:43.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:35:43.295 INFO:teuthology.orchestra.run.smithi170.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-28T23:35:43.295 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:35:43.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.396+0000 7fa366781640 1 Processor -- start 2024-03-28T23:35:43.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.396+0000 7fa366781640 1 -- start start 2024-03-28T23:35:43.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.397+0000 7fa366781640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa360150600 0x7fa3600a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.397+0000 7fa366781640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa3600a8fc0 0x7fa360153270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.397+0000 7fa366781640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3601567c0 con 0x7fa3600aacb0 2024-03-28T23:35:43.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.397+0000 7fa366781640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa360156570 con 0x7fa360150600 2024-03-28T23:35:43.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.397+0000 7fa366781640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fa360156aa0 con 0x7fa3600a8fc0 2024-03-28T23:35:43.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.397+0000 7fa35ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa360150600 0x7fa3600a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.397+0000 7fa35ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa360150600 0x7fa3600a9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58720/0 (socket says 172.21.15.161:58720) 2024-03-28T23:35:43.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.397+0000 7fa35ffff640 1 -- 172.21.15.161:0/2991502173 learned_addr learned my addr 172.21.15.161:0/2991502173 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:43.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.397+0000 7fa35f7fe640 1 --2- 172.21.15.161:0/2991502173 >> v2:172.21.15.170:3300/0 conn(0x7fa3600a8fc0 0x7fa360153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.397+0000 7fa35ffff640 1 -- 172.21.15.161:0/2991502173 >> v2:172.21.15.170:3300/0 conn(0x7fa3600a8fc0 msgr2=0x7fa360153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:43.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.397+0000 7fa35ffff640 1 --2- 172.21.15.161:0/2991502173 >> v2:172.21.15.170:3300/0 conn(0x7fa3600a8fc0 0x7fa360153270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.397+0000 7fa35ffff640 1 -- 172.21.15.161:0/2991502173 >> v1:172.21.15.170:6789/0 conn(0x7fa3600aacb0 legacy=0x7fa3601537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:43.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.397+0000 7fa35ffff640 1 -- 172.21.15.161:0/2991502173 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa360156d80 con 0x7fa360150600 2024-03-28T23:35:43.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.398+0000 7fa35ffff640 1 --2- 172.21.15.161:0/2991502173 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa360150600 0x7fa3600a9f40 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fa36006f570 tx=0x7fa35002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb40fc7861fe7831 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:43.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.398+0000 7fa35effd640 1 -- 172.21.15.161:0/2991502173 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa350035020 con 0x7fa360150600 2024-03-28T23:35:43.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.398+0000 7fa35effd640 1 -- 172.21.15.161:0/2991502173 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa350031870 con 0x7fa360150600 2024-03-28T23:35:43.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.398+0000 7fa35effd640 1 -- 172.21.15.161:0/2991502173 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa350031c40 con 0x7fa360150600 2024-03-28T23:35:43.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.399+0000 7fa366781640 1 -- 172.21.15.161:0/2991502173 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa360150600 msgr2=0x7fa3600a9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:43.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.399+0000 7fa366781640 1 --2- 172.21.15.161:0/2991502173 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa360150600 0x7fa3600a9f40 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fa36006f570 tx=0x7fa35002b910 comp rx=0 tx=0).stop 2024-03-28T23:35:43.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.399+0000 7fa366781640 1 -- 172.21.15.161:0/2991502173 shutdown_connections 2024-03-28T23:35:43.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.399+0000 7fa366781640 1 --2- 172.21.15.161:0/2991502173 >> v2:172.21.15.170:3300/0 conn(0x7fa3600a8fc0 0x7fa360153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.399+0000 7fa366781640 1 --2- 172.21.15.161:0/2991502173 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa360150600 0x7fa3600a9f40 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.399+0000 7fa366781640 1 -- 172.21.15.161:0/2991502173 >> 172.21.15.161:0/2991502173 conn(0x7fa3600c6670 msgr2=0x7fa3600c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:43.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.399+0000 7fa366781640 1 -- 172.21.15.161:0/2991502173 shutdown_connections 2024-03-28T23:35:43.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.399+0000 7fa366781640 1 -- 172.21.15.161:0/2991502173 wait complete. 2024-03-28T23:35:43.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.400+0000 7fa366781640 1 Processor -- start 2024-03-28T23:35:43.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.400+0000 7fa366781640 1 -- start start 2024-03-28T23:35:43.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.401+0000 7fa366781640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3600a8fc0 0x7fa3600c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.401+0000 7fa366781640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3600aacb0 0x7fa36014e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.401+0000 7fa366781640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa360150600 0x7fa36014ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.401+0000 7fa366781640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa360153ec0 con 0x7fa3600a8fc0 2024-03-28T23:35:43.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.401+0000 7fa366781640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa360157340 con 0x7fa3600aacb0 2024-03-28T23:35:43.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.401+0000 7fa366781640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa3601509e0 con 0x7fa360150600 2024-03-28T23:35:43.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.401+0000 7fa35ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3600a8fc0 0x7fa3600c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.401+0000 7fa35ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3600a8fc0 0x7fa3600c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58732/0 (socket says 172.21.15.161:58732) 2024-03-28T23:35:43.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.401+0000 7fa35ffff640 1 -- 172.21.15.161:0/3791589375 learned_addr learned my addr 172.21.15.161:0/3791589375 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:43.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.401+0000 7fa35f7fe640 1 --2- 172.21.15.161:0/3791589375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3600aacb0 0x7fa36014e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.401+0000 7fa364d1e640 1 --2- 172.21.15.161:0/3791589375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa360150600 0x7fa36014ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.401+0000 7fa35ffff640 1 -- 172.21.15.161:0/3791589375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa360150600 msgr2=0x7fa36014ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:43.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.401+0000 7fa35ffff640 1 --2- 172.21.15.161:0/3791589375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa360150600 0x7fa36014ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.401+0000 7fa35ffff640 1 -- 172.21.15.161:0/3791589375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3600aacb0 msgr2=0x7fa36014e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:43.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.401+0000 7fa35ffff640 1 --2- 172.21.15.161:0/3791589375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3600aacb0 0x7fa36014e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.401+0000 7fa35ffff640 1 -- 172.21.15.161:0/3791589375 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3600c3ec0 con 0x7fa3600a8fc0 2024-03-28T23:35:43.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.401+0000 7fa35ffff640 1 --2- 172.21.15.161:0/3791589375 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3600a8fc0 0x7fa3600c49a0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fa35002b950 tx=0x7fa3500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:43.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.401+0000 7fa35cff9640 1 -- 172.21.15.161:0/3791589375 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa350040070 con 0x7fa3600a8fc0 2024-03-28T23:35:43.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.401+0000 7fa35cff9640 1 -- 172.21.15.161:0/3791589375 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa350030d00 con 0x7fa3600a8fc0 2024-03-28T23:35:43.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.401+0000 7fa366781640 1 -- 172.21.15.161:0/3791589375 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3600c41a0 con 0x7fa3600a8fc0 2024-03-28T23:35:43.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.401+0000 7fa35cff9640 1 -- 172.21.15.161:0/3791589375 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa350035040 con 0x7fa3600a8fc0 2024-03-28T23:35:43.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.401+0000 7fa366781640 1 -- 172.21.15.161:0/3791589375 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa36014e420 con 0x7fa3600a8fc0 2024-03-28T23:35:43.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.402+0000 7fa366781640 1 -- 172.21.15.161:0/3791589375 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa36004f4b0 con 0x7fa3600a8fc0 2024-03-28T23:35:43.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.403+0000 7fa35cff9640 1 -- 172.21.15.161:0/3791589375 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa35004b060 con 0x7fa3600a8fc0 2024-03-28T23:35:43.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.403+0000 7fa35cff9640 1 --2- 172.21.15.161:0/3791589375 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa34c06b500 0x7fa34c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.403+0000 7fa35f7fe640 1 --2- 172.21.15.161:0/3791589375 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa34c06b500 0x7fa34c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.403+0000 7fa35cff9640 1 -- 172.21.15.161:0/3791589375 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 6331+0+0 (secure 0 0 0) 0x7fa3500abf80 con 0x7fa3600a8fc0 2024-03-28T23:35:43.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.404+0000 7fa35f7fe640 1 --2- 172.21.15.161:0/3791589375 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa34c06b500 0x7fa34c06d9c0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fa36014dce0 tx=0x7fa354009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:43.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.408+0000 7fa35cff9640 1 -- 172.21.15.161:0/3791589375 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa35007a5d0 con 0x7fa3600a8fc0 2024-03-28T23:35:43.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.613+0000 7fa366781640 1 -- 172.21.15.161:0/3791589375 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3600c4510 con 0x7fa3600a8fc0 2024-03-28T23:35:43.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.613+0000 7fa35cff9640 1 -- 172.21.15.161:0/3791589375 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) ==== 95+0+5203 (secure 0 0 0) 0x7fa35007e670 con 0x7fa3600a8fc0 2024-03-28T23:35:43.614 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:43.614 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":134,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7870,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2828648906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":2828648906},{"type":"v1","addr":"172.21.15.170:6835","nonce":2828648906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":132},{"gid":8409,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/3037890643","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3037890643},{"type":"v1","addr":"172.21.15.161:6837","nonce":3037890643}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120},{"gid":8622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/819329318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":819329318},{"type":"v1","addr":"172.21.15.161:6836","nonce":819329318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:12.982566+0000","modified":"2024-03-28T23:35:41.970209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":140,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7639},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7639":{"gid":7639,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":11,"addr":"172.21.15.170:6834/3182285172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3182285172},{"type":"v1","addr":"172.21.15.170:6834","nonce":3182285172}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7639,"qdb_cluster":[7639]},"id":10}]} 2024-03-28T23:35:43.614 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 134 2024-03-28T23:35:43.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.615+0000 7fa366781640 1 -- 172.21.15.161:0/3791589375 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa34c06b500 msgr2=0x7fa34c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:43.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.615+0000 7fa366781640 1 --2- 172.21.15.161:0/3791589375 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa34c06b500 0x7fa34c06d9c0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fa36014dce0 tx=0x7fa354009190 comp rx=0 tx=0).stop 2024-03-28T23:35:43.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.615+0000 7fa366781640 1 -- 172.21.15.161:0/3791589375 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3600a8fc0 msgr2=0x7fa3600c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:43.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.615+0000 7fa366781640 1 --2- 172.21.15.161:0/3791589375 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3600a8fc0 0x7fa3600c49a0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fa35002b950 tx=0x7fa3500046c0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.615+0000 7fa366781640 1 -- 172.21.15.161:0/3791589375 shutdown_connections 2024-03-28T23:35:43.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.615+0000 7fa366781640 1 --2- 172.21.15.161:0/3791589375 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa34c06b500 0x7fa34c06d9c0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.615+0000 7fa366781640 1 --2- 172.21.15.161:0/3791589375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa360150600 0x7fa36014ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.615+0000 7fa366781640 1 --2- 172.21.15.161:0/3791589375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3600aacb0 0x7fa36014e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.615+0000 7fa366781640 1 --2- 172.21.15.161:0/3791589375 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3600a8fc0 0x7fa3600c49a0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.615+0000 7fa366781640 1 -- 172.21.15.161:0/3791589375 >> 172.21.15.161:0/3791589375 conn(0x7fa3600c6670 msgr2=0x7fa360148cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:43.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.615+0000 7fa366781640 1 -- 172.21.15.161:0/3791589375 shutdown_connections 2024-03-28T23:35:43.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.615+0000 7fa366781640 1 -- 172.21.15.161:0/3791589375 wait complete. 2024-03-28T23:35:43.627 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:35:43.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.726+0000 7f37d7d5c640 1 Processor -- start 2024-03-28T23:35:43.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.727+0000 7f37d7d5c640 1 -- start start 2024-03-28T23:35:43.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.727+0000 7f37d7d5c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f37d006ce80 0x7f37d00aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.727+0000 7f37d7d5c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f37d00a9680 0x7f37d00aab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.727+0000 7f37d7d5c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37d014c250 con 0x7f37d014be70 2024-03-28T23:35:43.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.727+0000 7f37d7d5c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f37d01529c0 con 0x7f37d006ce80 2024-03-28T23:35:43.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.727+0000 7f37d7d5c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f37d00a9dc0 con 0x7f37d00a9680 2024-03-28T23:35:43.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.728+0000 7f37d52f7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f37d006ce80 0x7f37d00aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.728+0000 7f37d52f7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f37d006ce80 0x7f37d00aa5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58744/0 (socket says 172.21.15.161:58744) 2024-03-28T23:35:43.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.728+0000 7f37d52f7640 1 -- 172.21.15.161:0/1650576041 learned_addr learned my addr 172.21.15.161:0/1650576041 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:43.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.728+0000 7f37d62f9640 1 --2- 172.21.15.161:0/1650576041 >> v2:172.21.15.170:3300/0 conn(0x7f37d00a9680 0x7f37d00aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.728+0000 7f37d52f7640 1 -- 172.21.15.161:0/1650576041 >> v2:172.21.15.170:3300/0 conn(0x7f37d00a9680 msgr2=0x7f37d00aab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:43.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.728+0000 7f37d52f7640 1 --2- 172.21.15.161:0/1650576041 >> v2:172.21.15.170:3300/0 conn(0x7f37d00a9680 0x7f37d00aab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.728+0000 7f37d52f7640 1 -- 172.21.15.161:0/1650576041 >> v1:172.21.15.170:6789/0 conn(0x7f37d014be70 legacy=0x7f37d00ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:43.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.728+0000 7f37d52f7640 1 -- 172.21.15.161:0/1650576041 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37d00aa0a0 con 0x7f37d006ce80 2024-03-28T23:35:43.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.729+0000 7f37d52f7640 1 --2- 172.21.15.161:0/1650576041 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37d006ce80 0x7f37d00aa5c0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f37d0073960 tx=0x7f37cc02e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=228b4478c04c1b10 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:43.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.729+0000 7f37d4af6640 1 -- 172.21.15.161:0/1650576041 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37cc03b070 con 0x7f37d006ce80 2024-03-28T23:35:43.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.729+0000 7f37d4af6640 1 -- 172.21.15.161:0/1650576041 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37cc02ee00 con 0x7f37d006ce80 2024-03-28T23:35:43.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.729+0000 7f37d4af6640 1 -- 172.21.15.161:0/1650576041 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37cc036660 con 0x7f37d006ce80 2024-03-28T23:35:43.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.729+0000 7f37d7d5c640 1 -- 172.21.15.161:0/1650576041 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37d006ce80 msgr2=0x7f37d00aa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:43.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.729+0000 7f37d7d5c640 1 --2- 172.21.15.161:0/1650576041 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37d006ce80 0x7f37d00aa5c0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f37d0073960 tx=0x7f37cc02e050 comp rx=0 tx=0).stop 2024-03-28T23:35:43.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.729+0000 7f37d7d5c640 1 -- 172.21.15.161:0/1650576041 shutdown_connections 2024-03-28T23:35:43.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.729+0000 7f37d7d5c640 1 --2- 172.21.15.161:0/1650576041 >> v2:172.21.15.170:3300/0 conn(0x7f37d00a9680 0x7f37d00aab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.729+0000 7f37d7d5c640 1 --2- 172.21.15.161:0/1650576041 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37d006ce80 0x7f37d00aa5c0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.729+0000 7f37d7d5c640 1 -- 172.21.15.161:0/1650576041 >> 172.21.15.161:0/1650576041 conn(0x7f37d00c6670 msgr2=0x7f37d00c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:43.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.729+0000 7f37d7d5c640 1 -- 172.21.15.161:0/1650576041 shutdown_connections 2024-03-28T23:35:43.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.729+0000 7f37d7d5c640 1 -- 172.21.15.161:0/1650576041 wait complete. 2024-03-28T23:35:43.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.730+0000 7f37d7d5c640 1 Processor -- start 2024-03-28T23:35:43.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.730+0000 7f37d7d5c640 1 -- start start 2024-03-28T23:35:43.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.731+0000 7f37d7d5c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37d006ce80 0x7f37d0151920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.731+0000 7f37d7d5c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37d00a9680 0x7f37d00c5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.731+0000 7f37d7d5c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37d014be70 0x7f37d00c5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.731+0000 7f37d7d5c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f37d00aba30 con 0x7f37d014be70 2024-03-28T23:35:43.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.731+0000 7f37d7d5c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f37d00ab8b0 con 0x7f37d00a9680 2024-03-28T23:35:43.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.731+0000 7f37d7d5c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f37d006b150 con 0x7f37d006ce80 2024-03-28T23:35:43.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.731+0000 7f37d62f9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37d014be70 0x7f37d00c5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.731+0000 7f37d62f9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37d014be70 0x7f37d00c5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58752/0 (socket says 172.21.15.161:58752) 2024-03-28T23:35:43.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.731+0000 7f37d62f9640 1 -- 172.21.15.161:0/3626526215 learned_addr learned my addr 172.21.15.161:0/3626526215 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:43.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.731+0000 7f37d52f7640 1 --2- 172.21.15.161:0/3626526215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37d00a9680 0x7f37d00c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.731+0000 7f37d5af8640 1 --2- 172.21.15.161:0/3626526215 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37d006ce80 0x7f37d0151920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.732+0000 7f37d62f9640 1 -- 172.21.15.161:0/3626526215 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37d006ce80 msgr2=0x7f37d0151920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:43.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.732+0000 7f37d62f9640 1 --2- 172.21.15.161:0/3626526215 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37d006ce80 0x7f37d0151920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.732+0000 7f37d62f9640 1 -- 172.21.15.161:0/3626526215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37d00a9680 msgr2=0x7f37d00c5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:43.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.732+0000 7f37d62f9640 1 --2- 172.21.15.161:0/3626526215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37d00a9680 0x7f37d00c5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.732+0000 7f37d62f9640 1 -- 172.21.15.161:0/3626526215 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37d0148920 con 0x7f37d014be70 2024-03-28T23:35:43.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.732+0000 7f37d52f7640 1 --2- 172.21.15.161:0/3626526215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37d00a9680 0x7f37d00c5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:43.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.732+0000 7f37d62f9640 1 --2- 172.21.15.161:0/3626526215 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37d014be70 0x7f37d00c5a40 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f37c800efc0 tx=0x7f37c800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:43.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.732+0000 7f37c27fc640 1 -- 172.21.15.161:0/3626526215 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c8018020 con 0x7f37d014be70 2024-03-28T23:35:43.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.732+0000 7f37c27fc640 1 -- 172.21.15.161:0/3626526215 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37c8014480 con 0x7f37d014be70 2024-03-28T23:35:43.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.732+0000 7f37d7d5c640 1 -- 172.21.15.161:0/3626526215 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37d0148460 con 0x7f37d014be70 2024-03-28T23:35:43.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.732+0000 7f37c27fc640 1 -- 172.21.15.161:0/3626526215 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c8014870 con 0x7f37d014be70 2024-03-28T23:35:43.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.732+0000 7f37d7d5c640 1 -- 172.21.15.161:0/3626526215 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37d0151e60 con 0x7f37d014be70 2024-03-28T23:35:43.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.733+0000 7f37d7d5c640 1 -- 172.21.15.161:0/3626526215 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37d004f4b0 con 0x7f37d014be70 2024-03-28T23:35:43.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.733+0000 7f37c27fc640 1 -- 172.21.15.161:0/3626526215 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f37c8005d10 con 0x7f37d014be70 2024-03-28T23:35:43.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.734+0000 7f37c27fc640 1 --2- 172.21.15.161:0/3626526215 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f37b406b500 0x7f37b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:43.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.734+0000 7f37c27fc640 1 -- 172.21.15.161:0/3626526215 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 6331+0+0 (secure 0 0 0) 0x7f37c808ad00 con 0x7f37d014be70 2024-03-28T23:35:43.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.734+0000 7f37d5af8640 1 --2- 172.21.15.161:0/3626526215 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f37b406b500 0x7f37b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:43.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.734+0000 7f37d5af8640 1 --2- 172.21.15.161:0/3626526215 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f37b406b500 0x7f37b406d9c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f37d00ab430 tx=0x7f37c4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:43.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.738+0000 7f37c27fc640 1 -- 172.21.15.161:0/3626526215 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f37c8022070 con 0x7f37d014be70 2024-03-28T23:35:43.905 INFO:tasks.ceph.mds.d.smithi170.stderr:2024-03-28T23:35:43.904+0000 7fd0ac614640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:35:43.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.914+0000 7f37d7d5c640 1 -- 172.21.15.161:0/3626526215 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f37d00c0ff0 con 0x7f37d014be70 2024-03-28T23:35:43.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.915+0000 7f37c27fc640 1 -- 172.21.15.161:0/3626526215 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) ==== 74+0+17349 (secure 0 0 0) 0x7f37c8010040 con 0x7f37d014be70 2024-03-28T23:35:43.916 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:43.916 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":140,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:35:38.963165+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:35:11.208674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-28T23:35:11.954908+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"136","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"22.21","mappings":[{"from":3,"to":0}]},{"pgid":"22.37","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6835/3586875977":"2024-03-29T23:35:38.963130+0000","172.21.15.170:6833/3586875977":"2024-03-29T23:35:38.963130+0000","172.21.15.161:6836/2527502405":"2024-03-29T23:35:21.284385+0000","172.21.15.161:6834/2527502405":"2024-03-29T23:35:21.284385+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:35:43.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.917+0000 7f37d7d5c640 1 -- 172.21.15.161:0/3626526215 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f37b406b500 msgr2=0x7f37b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:43.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.917+0000 7f37d7d5c640 1 --2- 172.21.15.161:0/3626526215 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f37b406b500 0x7f37b406d9c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f37d00ab430 tx=0x7f37c4009040 comp rx=0 tx=0).stop 2024-03-28T23:35:43.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.917+0000 7f37d7d5c640 1 -- 172.21.15.161:0/3626526215 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37d014be70 msgr2=0x7f37d00c5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:43.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.917+0000 7f37d7d5c640 1 --2- 172.21.15.161:0/3626526215 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37d014be70 0x7f37d00c5a40 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f37c800efc0 tx=0x7f37c800c490 comp rx=0 tx=0).stop 2024-03-28T23:35:43.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.917+0000 7f37d7d5c640 1 -- 172.21.15.161:0/3626526215 shutdown_connections 2024-03-28T23:35:43.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.917+0000 7f37d7d5c640 1 --2- 172.21.15.161:0/3626526215 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f37b406b500 0x7f37b406d9c0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.917+0000 7f37d7d5c640 1 --2- 172.21.15.161:0/3626526215 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37d014be70 0x7f37d00c5a40 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.917+0000 7f37d7d5c640 1 --2- 172.21.15.161:0/3626526215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37d00a9680 0x7f37d00c5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.917+0000 7f37d7d5c640 1 --2- 172.21.15.161:0/3626526215 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37d006ce80 0x7f37d0151920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:43.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.917+0000 7f37d7d5c640 1 -- 172.21.15.161:0/3626526215 >> 172.21.15.161:0/3626526215 conn(0x7f37d00c6670 msgr2=0x7f37d00c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:43.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.918+0000 7f37d7d5c640 1 -- 172.21.15.161:0/3626526215 shutdown_connections 2024-03-28T23:35:43.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:43.918+0000 7f37d7d5c640 1 -- 172.21.15.161:0/3626526215 wait complete. 2024-03-28T23:35:43.929 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:35:11.208674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '135', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:35:11.954908+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '136', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:35:43.929 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:35:44.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.033+0000 7f09e0c8f640 1 Processor -- start 2024-03-28T23:35:44.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.033+0000 7f09e0c8f640 1 -- start start 2024-03-28T23:35:44.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.034+0000 7f09e0c8f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09dc1529d0 0x7f09dc06bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.034+0000 7f09e0c8f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f09dc0c6750 0x7f09dc0c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.034+0000 7f09e0c8f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f09dc155c30 0x7f09dc0c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.034+0000 7f09e0c8f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f09dc152db0 con 0x7f09dc155c30 2024-03-28T23:35:44.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.034+0000 7f09e0c8f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f09dc0c7210 con 0x7f09dc0c6750 2024-03-28T23:35:44.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.034+0000 7f09e0c8f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f09dc0c6c50 con 0x7f09dc1529d0 2024-03-28T23:35:44.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.034+0000 7f09dad9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f09dc155c30 0x7f09dc0c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.034+0000 7f09dad9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f09dc155c30 0x7f09dc0c54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58766/0 (socket says 172.21.15.161:58766) 2024-03-28T23:35:44.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.034+0000 7f09dad9d640 1 -- 172.21.15.161:0/1577547714 learned_addr learned my addr 172.21.15.161:0/1577547714 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:44.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.034+0000 7f09da59c640 1 --2- 172.21.15.161:0/1577547714 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09dc1529d0 0x7f09dc06bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.034+0000 7f09d9d9b640 1 --2- 172.21.15.161:0/1577547714 >> v2:172.21.15.170:3300/0 conn(0x7f09dc0c6750 0x7f09dc0c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.035+0000 7f09dad9d640 1 -- 172.21.15.161:0/1577547714 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09dc1529d0 msgr2=0x7f09dc06bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:44.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.035+0000 7f09dad9d640 1 --2- 172.21.15.161:0/1577547714 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09dc1529d0 0x7f09dc06bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.035+0000 7f09dad9d640 1 -- 172.21.15.161:0/1577547714 >> v2:172.21.15.170:3300/0 conn(0x7f09dc0c6750 msgr2=0x7f09dc0c88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:44.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.035+0000 7f09dad9d640 1 --2- 172.21.15.161:0/1577547714 >> v2:172.21.15.170:3300/0 conn(0x7f09dc0c6750 0x7f09dc0c88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.035+0000 7f09dad9d640 1 -- 172.21.15.161:0/1577547714 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09dc0c6f30 con 0x7f09dc155c30 2024-03-28T23:35:44.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.035+0000 7f09d9d9b640 1 --2- 172.21.15.161:0/1577547714 >> v2:172.21.15.170:3300/0 conn(0x7f09dc0c6750 0x7f09dc0c88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:44.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.035+0000 7f09dad9d640 1 --2- 172.21.15.161:0/1577547714 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09dc155c30 0x7f09dc0c54f0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f09dc068a40 tx=0x7f09d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b3604bd22225cf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:44.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.036+0000 7f09d959a640 1 -- 172.21.15.161:0/1577547714 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d4035020 con 0x7f09dc155c30 2024-03-28T23:35:44.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.036+0000 7f09d959a640 1 -- 172.21.15.161:0/1577547714 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09d4031870 con 0x7f09dc155c30 2024-03-28T23:35:44.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.036+0000 7f09d959a640 1 -- 172.21.15.161:0/1577547714 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d4031c40 con 0x7f09dc155c30 2024-03-28T23:35:44.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.036+0000 7f09e0c8f640 1 -- 172.21.15.161:0/1577547714 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09dc155c30 msgr2=0x7f09dc0c54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:44.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.036+0000 7f09e0c8f640 1 --2- 172.21.15.161:0/1577547714 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09dc155c30 0x7f09dc0c54f0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f09dc068a40 tx=0x7f09d402b910 comp rx=0 tx=0).stop 2024-03-28T23:35:44.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.036+0000 7f09e0c8f640 1 -- 172.21.15.161:0/1577547714 shutdown_connections 2024-03-28T23:35:44.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.036+0000 7f09e0c8f640 1 --2- 172.21.15.161:0/1577547714 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09dc155c30 0x7f09dc0c54f0 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.036+0000 7f09e0c8f640 1 --2- 172.21.15.161:0/1577547714 >> v2:172.21.15.170:3300/0 conn(0x7f09dc0c6750 0x7f09dc0c88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.036+0000 7f09e0c8f640 1 --2- 172.21.15.161:0/1577547714 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09dc1529d0 0x7f09dc06bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.036+0000 7f09e0c8f640 1 -- 172.21.15.161:0/1577547714 >> 172.21.15.161:0/1577547714 conn(0x7f09dc0c7430 msgr2=0x7f09dc0c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:44.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.036+0000 7f09e0c8f640 1 -- 172.21.15.161:0/1577547714 shutdown_connections 2024-03-28T23:35:44.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.036+0000 7f09e0c8f640 1 -- 172.21.15.161:0/1577547714 wait complete. 2024-03-28T23:35:44.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.037+0000 7f09e0c8f640 1 Processor -- start 2024-03-28T23:35:44.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.038+0000 7f09e0c8f640 1 -- start start 2024-03-28T23:35:44.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.038+0000 7f09e0c8f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09dc0c6750 0x7f09dc1545c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.038+0000 7f09e0c8f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09dc1529d0 0x7f09dc0c3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.038+0000 7f09e0c8f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09dc155c30 0x7f09dc0ab480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.038+0000 7f09e0c8f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc0aa160 con 0x7f09dc1529d0 2024-03-28T23:35:44.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.038+0000 7f09e0c8f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc155710 con 0x7f09dc155c30 2024-03-28T23:35:44.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.038+0000 7f09e0c8f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f09dc1560e0 con 0x7f09dc0c6750 2024-03-28T23:35:44.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.038+0000 7f09d9d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09dc1529d0 0x7f09dc0c3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.038+0000 7f09d9d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09dc1529d0 0x7f09dc0c3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58772/0 (socket says 172.21.15.161:58772) 2024-03-28T23:35:44.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.038+0000 7f09d9d9b640 1 -- 172.21.15.161:0/866186641 learned_addr learned my addr 172.21.15.161:0/866186641 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:44.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.038+0000 7f09da59c640 1 --2- 172.21.15.161:0/866186641 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09dc0c6750 0x7f09dc1545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.038+0000 7f09dad9d640 1 --2- 172.21.15.161:0/866186641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09dc155c30 0x7f09dc0ab480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.039+0000 7f09da59c640 1 -- 172.21.15.161:0/866186641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09dc155c30 msgr2=0x7f09dc0ab480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:44.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.039+0000 7f09da59c640 1 --2- 172.21.15.161:0/866186641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09dc155c30 0x7f09dc0ab480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.039+0000 7f09da59c640 1 -- 172.21.15.161:0/866186641 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09dc1529d0 msgr2=0x7f09dc0c3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:44.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.039+0000 7f09da59c640 1 --2- 172.21.15.161:0/866186641 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09dc1529d0 0x7f09dc0c3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.039+0000 7f09da59c640 1 -- 172.21.15.161:0/866186641 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09dc0c25b0 con 0x7f09dc0c6750 2024-03-28T23:35:44.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.039+0000 7f09d9d9b640 1 --2- 172.21.15.161:0/866186641 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09dc1529d0 0x7f09dc0c3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:44.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.039+0000 7f09da59c640 1 --2- 172.21.15.161:0/866186641 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09dc0c6750 0x7f09dc1545c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00ebf0 tx=0x7f09cc00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:44.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.039+0000 7f09caffd640 1 -- 172.21.15.161:0/866186641 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc018020 con 0x7f09dc0c6750 2024-03-28T23:35:44.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.039+0000 7f09caffd640 1 -- 172.21.15.161:0/866186641 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09cc00cda0 con 0x7f09dc0c6750 2024-03-28T23:35:44.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.039+0000 7f09e0c8f640 1 -- 172.21.15.161:0/866186641 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09dc154cd0 con 0x7f09dc0c6750 2024-03-28T23:35:44.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.039+0000 7f09caffd640 1 -- 172.21.15.161:0/866186641 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc0146c0 con 0x7f09dc0c6750 2024-03-28T23:35:44.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.039+0000 7f09e0c8f640 1 -- 172.21.15.161:0/866186641 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f09dc0c2870 con 0x7f09dc0c6750 2024-03-28T23:35:44.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.040+0000 7f09e0c8f640 1 -- 172.21.15.161:0/866186641 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09dc04f4b0 con 0x7f09dc0c6750 2024-03-28T23:35:44.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.040+0000 7f09caffd640 1 -- 172.21.15.161:0/866186641 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f09cc005d10 con 0x7f09dc0c6750 2024-03-28T23:35:44.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.041+0000 7f09caffd640 1 --2- 172.21.15.161:0/866186641 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f09bc06b4d0 0x7f09bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.041+0000 7f09caffd640 1 -- 172.21.15.161:0/866186641 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(140..140 src has 1..140) ==== 6331+0+0 (secure 0 0 0) 0x7f09cc08afe0 con 0x7f09dc0c6750 2024-03-28T23:35:44.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.041+0000 7f09d9d9b640 1 --2- 172.21.15.161:0/866186641 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f09bc06b4d0 0x7f09bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.041+0000 7f09d9d9b640 1 --2- 172.21.15.161:0/866186641 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f09bc06b4d0 0x7f09bc06d990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f09d0009560 tx=0x7f09d0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:44.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.045+0000 7f09caffd640 1 -- 172.21.15.161:0/866186641 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f09cc022030 con 0x7f09dc0c6750 2024-03-28T23:35:44.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.217+0000 7f09e0c8f640 1 -- 172.21.15.161:0/866186641 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f09dc0ca650 con 0x7f09dc0c6750 2024-03-28T23:35:44.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.218+0000 7f09caffd640 1 -- 172.21.15.161:0/866186641 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v140) ==== 93+0+10 (secure 0 0 0) 0x7f09cc010040 con 0x7f09dc0c6750 2024-03-28T23:35:44.225 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:35:44.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.220+0000 7f09e0c8f640 1 -- 172.21.15.161:0/866186641 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f09bc06b4d0 msgr2=0x7f09bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:44.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.220+0000 7f09e0c8f640 1 --2- 172.21.15.161:0/866186641 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f09bc06b4d0 0x7f09bc06d990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f09d0009560 tx=0x7f09d0009190 comp rx=0 tx=0).stop 2024-03-28T23:35:44.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.220+0000 7f09e0c8f640 1 -- 172.21.15.161:0/866186641 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09dc0c6750 msgr2=0x7f09dc1545c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:44.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.220+0000 7f09e0c8f640 1 --2- 172.21.15.161:0/866186641 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09dc0c6750 0x7f09dc1545c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00ebf0 tx=0x7f09cc00c490 comp rx=0 tx=0).stop 2024-03-28T23:35:44.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.220+0000 7f09e0c8f640 1 -- 172.21.15.161:0/866186641 shutdown_connections 2024-03-28T23:35:44.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.220+0000 7f09e0c8f640 1 --2- 172.21.15.161:0/866186641 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f09bc06b4d0 0x7f09bc06d990 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.220+0000 7f09e0c8f640 1 --2- 172.21.15.161:0/866186641 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09dc1529d0 0x7f09dc0c3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.220+0000 7f09e0c8f640 1 --2- 172.21.15.161:0/866186641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09dc155c30 0x7f09dc0ab480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.220+0000 7f09e0c8f640 1 --2- 172.21.15.161:0/866186641 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09dc0c6750 0x7f09dc1545c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.220+0000 7f09e0c8f640 1 -- 172.21.15.161:0/866186641 >> 172.21.15.161:0/866186641 conn(0x7f09dc0c7430 msgr2=0x7f09dc150e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:44.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.220+0000 7f09e0c8f640 1 -- 172.21.15.161:0/866186641 shutdown_connections 2024-03-28T23:35:44.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.220+0000 7f09e0c8f640 1 -- 172.21.15.161:0/866186641 wait complete. 2024-03-28T23:35:44.231 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:35:44.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.335+0000 7f8b0ba8f640 1 Processor -- start 2024-03-28T23:35:44.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.335+0000 7f8b0ba8f640 1 -- start start 2024-03-28T23:35:44.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.336+0000 7f8b0ba8f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8b0406cdd0 0x7f8b041524c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.336+0000 7f8b0ba8f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b0406ba50 0x7f8b04152a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.336+0000 7f8b0ba8f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b040c2f10 con 0x7f8b0414f5c0 2024-03-28T23:35:44.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.336+0000 7f8b0ba8f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f8b040c74d0 con 0x7f8b0406cdd0 2024-03-28T23:35:44.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.336+0000 7f8b0ba8f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8b040c6fa0 con 0x7f8b0406ba50 2024-03-28T23:35:44.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.336+0000 7f8b0902a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8b0406cdd0 0x7f8b041524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.336+0000 7f8b0982b640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f8b0414f5c0 0x7f8b040c2150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:54806/0 (socket says 172.21.15.161:54806) 2024-03-28T23:35:44.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.336+0000 7f8b0982b640 1 -- 172.21.15.161:0/3598841936 learned_addr learned my addr 172.21.15.161:0/3598841936 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:44.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.336+0000 7f8b0a02c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b0406ba50 0x7f8b04152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.336+0000 7f8b08829640 1 -- 172.21.15.161:0/3598841936 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3716559948 0 0) 0x7f8b00001490 con 0x7f8b0414f5c0 2024-03-28T23:35:44.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.337+0000 7f8b0a02c640 1 -- 172.21.15.161:0/3598841936 >> v2:172.21.15.170:3300/0 conn(0x7f8b0406cdd0 msgr2=0x7f8b041524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:44.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.337+0000 7f8b0a02c640 1 --2- 172.21.15.161:0/3598841936 >> v2:172.21.15.170:3300/0 conn(0x7f8b0406cdd0 0x7f8b041524c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.337+0000 7f8b0a02c640 1 -- 172.21.15.161:0/3598841936 >> v1:172.21.15.170:6789/0 conn(0x7f8b0414f5c0 legacy=0x7f8b040c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:44.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.337+0000 7f8b0a02c640 1 -- 172.21.15.161:0/3598841936 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b0414f9a0 con 0x7f8b0406ba50 2024-03-28T23:35:44.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.337+0000 7f8b0902a640 1 --2- 172.21.15.161:0/3598841936 >> v2:172.21.15.170:3300/0 conn(0x7f8b0406cdd0 0x7f8b041524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:44.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.337+0000 7f8b0a02c640 1 --2- 172.21.15.161:0/3598841936 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b0406ba50 0x7f8b04152a00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8afc0097a0 tx=0x7f8afc02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bf6ba38f69eb677 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:44.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.337+0000 7f8b08829640 1 -- 172.21.15.161:0/3598841936 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8afc035020 con 0x7f8b0406ba50 2024-03-28T23:35:44.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.337+0000 7f8b08829640 1 -- 172.21.15.161:0/3598841936 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8afc031810 con 0x7f8b0406ba50 2024-03-28T23:35:44.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.337+0000 7f8b08829640 1 -- 172.21.15.161:0/3598841936 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8afc031be0 con 0x7f8b0406ba50 2024-03-28T23:35:44.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.338+0000 7f8b0ba8f640 1 -- 172.21.15.161:0/3598841936 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b0406ba50 msgr2=0x7f8b04152a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:44.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.338+0000 7f8b0ba8f640 1 --2- 172.21.15.161:0/3598841936 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b0406ba50 0x7f8b04152a00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8afc0097a0 tx=0x7f8afc02b8b0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.338+0000 7f8b0ba8f640 1 -- 172.21.15.161:0/3598841936 shutdown_connections 2024-03-28T23:35:44.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.338+0000 7f8b0ba8f640 1 --2- 172.21.15.161:0/3598841936 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b0406ba50 0x7f8b04152a00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.338+0000 7f8b0ba8f640 1 --2- 172.21.15.161:0/3598841936 >> v2:172.21.15.170:3300/0 conn(0x7f8b0406cdd0 0x7f8b041524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.338+0000 7f8b0ba8f640 1 -- 172.21.15.161:0/3598841936 >> 172.21.15.161:0/3598841936 conn(0x7f8b040a83f0 msgr2=0x7f8b040a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:44.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.338+0000 7f8b0ba8f640 1 -- 172.21.15.161:0/3598841936 shutdown_connections 2024-03-28T23:35:44.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.338+0000 7f8b0ba8f640 1 -- 172.21.15.161:0/3598841936 wait complete. 2024-03-28T23:35:44.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.339+0000 7f8b0ba8f640 1 Processor -- start 2024-03-28T23:35:44.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.339+0000 7f8b0ba8f640 1 -- start start 2024-03-28T23:35:44.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.340+0000 7f8b0ba8f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b0406ba50 0x7f8b041eb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.340+0000 7f8b0ba8f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b0406cdd0 0x7f8b0415f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.340+0000 7f8b0ba8f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b0414f5c0 0x7f8b041ef7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.340+0000 7f8b0ba8f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8b0406c140 con 0x7f8b0406cdd0 2024-03-28T23:35:44.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.340+0000 7f8b0ba8f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8b04152030 con 0x7f8b0406ba50 2024-03-28T23:35:44.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.340+0000 7f8b0ba8f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8b0406c3e0 con 0x7f8b0414f5c0 2024-03-28T23:35:44.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.340+0000 7f8b0902a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b0406cdd0 0x7f8b0415f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.340+0000 7f8b0902a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b0406cdd0 0x7f8b0415f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58780/0 (socket says 172.21.15.161:58780) 2024-03-28T23:35:44.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.340+0000 7f8b0902a640 1 -- 172.21.15.161:0/1101687640 learned_addr learned my addr 172.21.15.161:0/1101687640 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:44.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.340+0000 7f8b0a02c640 1 --2- 172.21.15.161:0/1101687640 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b0414f5c0 0x7f8b041ef7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.340+0000 7f8b0982b640 1 --2- 172.21.15.161:0/1101687640 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b0406ba50 0x7f8b041eb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.340+0000 7f8b0902a640 1 -- 172.21.15.161:0/1101687640 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b0414f5c0 msgr2=0x7f8b041ef7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:44.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.340+0000 7f8b0902a640 1 --2- 172.21.15.161:0/1101687640 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b0414f5c0 0x7f8b041ef7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.340+0000 7f8b0902a640 1 -- 172.21.15.161:0/1101687640 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b0406ba50 msgr2=0x7f8b041eb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:44.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.340+0000 7f8b0902a640 1 --2- 172.21.15.161:0/1101687640 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b0406ba50 0x7f8b041eb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.340+0000 7f8b0902a640 1 -- 172.21.15.161:0/1101687640 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b04155880 con 0x7f8b0406cdd0 2024-03-28T23:35:44.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.341+0000 7f8b0902a640 1 --2- 172.21.15.161:0/1101687640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b0406cdd0 0x7f8b0415f080 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f8af400ebe0 tx=0x7f8af400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:44.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.341+0000 7f8afa7fc640 1 -- 172.21.15.161:0/1101687640 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af4017070 con 0x7f8b0406cdd0 2024-03-28T23:35:44.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.341+0000 7f8afa7fc640 1 -- 172.21.15.161:0/1101687640 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8af400cc30 con 0x7f8b0406cdd0 2024-03-28T23:35:44.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.341+0000 7f8b0ba8f640 1 -- 172.21.15.161:0/1101687640 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b04155b60 con 0x7f8b0406cdd0 2024-03-28T23:35:44.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.341+0000 7f8afa7fc640 1 -- 172.21.15.161:0/1101687640 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af4013470 con 0x7f8b0406cdd0 2024-03-28T23:35:44.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.341+0000 7f8b0ba8f640 1 -- 172.21.15.161:0/1101687640 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b041eeff0 con 0x7f8b0406cdd0 2024-03-28T23:35:44.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.342+0000 7f8b0ba8f640 1 -- 172.21.15.161:0/1101687640 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b040c2d30 con 0x7f8b0406cdd0 2024-03-28T23:35:44.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.342+0000 7f8afa7fc640 1 -- 172.21.15.161:0/1101687640 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8af401a070 con 0x7f8b0406cdd0 2024-03-28T23:35:44.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.343+0000 7f8afa7fc640 1 --2- 172.21.15.161:0/1101687640 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8ae806b4d0 0x7f8ae806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.343+0000 7f8b0982b640 1 --2- 172.21.15.161:0/1101687640 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8ae806b4d0 0x7f8ae806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.343+0000 7f8afa7fc640 1 -- 172.21.15.161:0/1101687640 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 6331+0+0 (secure 0 0 0) 0x7f8af408a6e0 con 0x7f8b0406cdd0 2024-03-28T23:35:44.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.343+0000 7f8b0982b640 1 --2- 172.21.15.161:0/1101687640 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8ae806b4d0 0x7f8ae806d990 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f8b000012e0 tx=0x7f8b00001630 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:44.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.347+0000 7f8afa7fc640 1 -- 172.21.15.161:0/1101687640 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8b040c2d30 con 0x7f8b0406cdd0 2024-03-28T23:35:44.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.519+0000 7f8b0ba8f640 1 -- 172.21.15.161:0/1101687640 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8b0404f4b0 con 0x7f8b0406cdd0 2024-03-28T23:35:44.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.520+0000 7f8afa7fc640 1 -- 172.21.15.161:0/1101687640 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v140) ==== 104+0+11 (secure 0 0 0) 0x7f8af4058ca0 con 0x7f8b0406cdd0 2024-03-28T23:35:44.520 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:35:44.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.521+0000 7f8b0ba8f640 1 -- 172.21.15.161:0/1101687640 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8ae806b4d0 msgr2=0x7f8ae806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:44.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.521+0000 7f8b0ba8f640 1 --2- 172.21.15.161:0/1101687640 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8ae806b4d0 0x7f8ae806d990 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f8b000012e0 tx=0x7f8b00001630 comp rx=0 tx=0).stop 2024-03-28T23:35:44.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.521+0000 7f8b0ba8f640 1 -- 172.21.15.161:0/1101687640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b0406cdd0 msgr2=0x7f8b0415f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:44.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.521+0000 7f8b0ba8f640 1 --2- 172.21.15.161:0/1101687640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b0406cdd0 0x7f8b0415f080 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f8af400ebe0 tx=0x7f8af400c520 comp rx=0 tx=0).stop 2024-03-28T23:35:44.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.521+0000 7f8b0ba8f640 1 -- 172.21.15.161:0/1101687640 shutdown_connections 2024-03-28T23:35:44.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.521+0000 7f8b0ba8f640 1 --2- 172.21.15.161:0/1101687640 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8ae806b4d0 0x7f8ae806d990 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.521+0000 7f8b0ba8f640 1 --2- 172.21.15.161:0/1101687640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b0406cdd0 0x7f8b0415f080 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.521+0000 7f8b0ba8f640 1 --2- 172.21.15.161:0/1101687640 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b0414f5c0 0x7f8b041ef7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.521+0000 7f8b0ba8f640 1 --2- 172.21.15.161:0/1101687640 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b0406ba50 0x7f8b041eb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.521+0000 7f8b0ba8f640 1 -- 172.21.15.161:0/1101687640 >> 172.21.15.161:0/1101687640 conn(0x7f8b040a83f0 msgr2=0x7f8b040c5f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:44.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.522+0000 7f8b0ba8f640 1 -- 172.21.15.161:0/1101687640 shutdown_connections 2024-03-28T23:35:44.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.522+0000 7f8b0ba8f640 1 -- 172.21.15.161:0/1101687640 wait complete. 2024-03-28T23:35:44.533 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:35:44.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.636+0000 7f15ce240640 1 Processor -- start 2024-03-28T23:35:44.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.636+0000 7f15ce240640 1 -- start start 2024-03-28T23:35:44.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.637+0000 7f15ce240640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f15c80c6750 0x7f15c806bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.637+0000 7f15ce240640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f15c80c88b0 0x7f15c80c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.637+0000 7f15ce240640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15c8152db0 con 0x7f15c81529d0 2024-03-28T23:35:44.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.637+0000 7f15ce240640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f15c80c7210 con 0x7f15c80c6750 2024-03-28T23:35:44.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.637+0000 7f15ce240640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f15c80c6c50 con 0x7f15c80c88b0 2024-03-28T23:35:44.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.637+0000 7f15c6ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f15c80c6750 0x7f15c806bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.637+0000 7f15c77fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f15c81529d0 0x7f15c8155c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38166/0 (socket says 172.21.15.161:38166) 2024-03-28T23:35:44.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.637+0000 7f15c77fe640 1 -- 172.21.15.161:0/2136917314 learned_addr learned my addr 172.21.15.161:0/2136917314 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:44.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.637+0000 7f15c7fff640 1 --2- 172.21.15.161:0/2136917314 >> v2:172.21.15.170:3300/0 conn(0x7f15c80c88b0 0x7f15c80c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.637+0000 7f15c6ffd640 1 -- 172.21.15.161:0/2136917314 >> v2:172.21.15.170:3300/0 conn(0x7f15c80c88b0 msgr2=0x7f15c80c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:44.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.637+0000 7f15c6ffd640 1 --2- 172.21.15.161:0/2136917314 >> v2:172.21.15.170:3300/0 conn(0x7f15c80c88b0 0x7f15c80c54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.637+0000 7f15c6ffd640 1 -- 172.21.15.161:0/2136917314 >> v1:172.21.15.161:6789/0 conn(0x7f15c81529d0 legacy=0x7f15c8155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:44.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.637+0000 7f15c6ffd640 1 -- 172.21.15.161:0/2136917314 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15c80c8d40 con 0x7f15c80c6750 2024-03-28T23:35:44.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.637+0000 7f15c67fc640 1 -- 172.21.15.161:0/2136917314 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 985274135 0 0) 0x7f15c8152db0 con 0x7f15c81529d0 2024-03-28T23:35:44.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.638+0000 7f15c6ffd640 1 --2- 172.21.15.161:0/2136917314 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15c80c6750 0x7f15c806bb60 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f15c8074ba0 tx=0x7f15bc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed07fa2799544b30 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:44.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.638+0000 7f15c67fc640 1 -- 172.21.15.161:0/2136917314 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15bc035020 con 0x7f15c80c6750 2024-03-28T23:35:44.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.638+0000 7f15c67fc640 1 -- 172.21.15.161:0/2136917314 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15bc031830 con 0x7f15c80c6750 2024-03-28T23:35:44.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.638+0000 7f15c67fc640 1 -- 172.21.15.161:0/2136917314 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15bc031c00 con 0x7f15c80c6750 2024-03-28T23:35:44.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.639+0000 7f15ce240640 1 -- 172.21.15.161:0/2136917314 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15c80c6750 msgr2=0x7f15c806bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:44.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.639+0000 7f15ce240640 1 --2- 172.21.15.161:0/2136917314 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15c80c6750 0x7f15c806bb60 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f15c8074ba0 tx=0x7f15bc02b8d0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.639+0000 7f15ce240640 1 -- 172.21.15.161:0/2136917314 shutdown_connections 2024-03-28T23:35:44.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.639+0000 7f15ce240640 1 --2- 172.21.15.161:0/2136917314 >> v2:172.21.15.170:3300/0 conn(0x7f15c80c88b0 0x7f15c80c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.639+0000 7f15ce240640 1 --2- 172.21.15.161:0/2136917314 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15c80c6750 0x7f15c806bb60 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.639+0000 7f15ce240640 1 -- 172.21.15.161:0/2136917314 >> 172.21.15.161:0/2136917314 conn(0x7f15c80c7430 msgr2=0x7f15c80c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:44.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.639+0000 7f15ce240640 1 -- 172.21.15.161:0/2136917314 shutdown_connections 2024-03-28T23:35:44.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.639+0000 7f15ce240640 1 -- 172.21.15.161:0/2136917314 wait complete. 2024-03-28T23:35:44.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.640+0000 7f15ce240640 1 Processor -- start 2024-03-28T23:35:44.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.640+0000 7f15ce240640 1 -- start start 2024-03-28T23:35:44.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.641+0000 7f15ce240640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15c80c6750 0x7f15c80afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.641+0000 7f15ce240640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15c80c88b0 0x7f15c814ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.641+0000 7f15ce240640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15c81529d0 0x7f15c814cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.641+0000 7f15ce240640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f15c8155710 con 0x7f15c80c6750 2024-03-28T23:35:44.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.641+0000 7f15ce240640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f15c8155430 con 0x7f15c80c88b0 2024-03-28T23:35:44.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.641+0000 7f15ce240640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f15c80aa160 con 0x7f15c81529d0 2024-03-28T23:35:44.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.641+0000 7f15c77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15c80c6750 0x7f15c80afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.641+0000 7f15c77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15c80c6750 0x7f15c80afe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58800/0 (socket says 172.21.15.161:58800) 2024-03-28T23:35:44.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.641+0000 7f15c77fe640 1 -- 172.21.15.161:0/33249609 learned_addr learned my addr 172.21.15.161:0/33249609 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:44.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.641+0000 7f15c6ffd640 1 --2- 172.21.15.161:0/33249609 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15c80c88b0 0x7f15c814ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.641+0000 7f15c7fff640 1 --2- 172.21.15.161:0/33249609 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15c81529d0 0x7f15c814cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.641+0000 7f15c77fe640 1 -- 172.21.15.161:0/33249609 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15c81529d0 msgr2=0x7f15c814cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:44.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.641+0000 7f15c77fe640 1 --2- 172.21.15.161:0/33249609 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15c81529d0 0x7f15c814cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.641+0000 7f15c77fe640 1 -- 172.21.15.161:0/33249609 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15c80c88b0 msgr2=0x7f15c814ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:44.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.641+0000 7f15c77fe640 1 --2- 172.21.15.161:0/33249609 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15c80c88b0 0x7f15c814ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.641+0000 7f15c77fe640 1 -- 172.21.15.161:0/33249609 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15c80b0370 con 0x7f15c80c6750 2024-03-28T23:35:44.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.642+0000 7f15c77fe640 1 --2- 172.21.15.161:0/33249609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15c80c6750 0x7f15c80afe30 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f15b800c4d0 tx=0x7f15b800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:44.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.642+0000 7f15affff640 1 -- 172.21.15.161:0/33249609 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b8018070 con 0x7f15c80c6750 2024-03-28T23:35:44.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.642+0000 7f15affff640 1 -- 172.21.15.161:0/33249609 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15b8007500 con 0x7f15c80c6750 2024-03-28T23:35:44.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.642+0000 7f15ce240640 1 -- 172.21.15.161:0/33249609 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15c81ebba0 con 0x7f15c80c6750 2024-03-28T23:35:44.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.642+0000 7f15affff640 1 -- 172.21.15.161:0/33249609 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b8004a20 con 0x7f15c80c6750 2024-03-28T23:35:44.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.642+0000 7f15ce240640 1 -- 172.21.15.161:0/33249609 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15c814dd70 con 0x7f15c80c6750 2024-03-28T23:35:44.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.643+0000 7f15ce240640 1 -- 172.21.15.161:0/33249609 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15c804f4b0 con 0x7f15c80c6750 2024-03-28T23:35:44.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.643+0000 7f15affff640 1 -- 172.21.15.161:0/33249609 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f15b8004bc0 con 0x7f15c80c6750 2024-03-28T23:35:44.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.643+0000 7f15affff640 1 --2- 172.21.15.161:0/33249609 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f15b406b4d0 0x7f15b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.644+0000 7f15affff640 1 -- 172.21.15.161:0/33249609 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 6331+0+0 (secure 0 0 0) 0x7f15b80896f0 con 0x7f15c80c6750 2024-03-28T23:35:44.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.644+0000 7f15c6ffd640 1 --2- 172.21.15.161:0/33249609 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f15b406b4d0 0x7f15b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.644+0000 7f15c6ffd640 1 --2- 172.21.15.161:0/33249609 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f15b406b4d0 0x7f15b406d990 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f15bc036000 tx=0x7f15bc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:44.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.648+0000 7f15affff640 1 -- 172.21.15.161:0/33249609 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f15b8057c30 con 0x7f15c80c6750 2024-03-28T23:35:44.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.820+0000 7f15ce240640 1 -- 172.21.15.161:0/33249609 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f15c80ca650 con 0x7f15c80c6750 2024-03-28T23:35:44.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.820+0000 7f15affff640 1 -- 172.21.15.161:0/33249609 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v140) ==== 100+0+11 (secure 0 0 0) 0x7f15b805bcd0 con 0x7f15c80c6750 2024-03-28T23:35:44.821 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:35:44.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.822+0000 7f15ce240640 1 -- 172.21.15.161:0/33249609 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f15b406b4d0 msgr2=0x7f15b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:44.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.822+0000 7f15ce240640 1 --2- 172.21.15.161:0/33249609 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f15b406b4d0 0x7f15b406d990 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f15bc036000 tx=0x7f15bc008030 comp rx=0 tx=0).stop 2024-03-28T23:35:44.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.822+0000 7f15ce240640 1 -- 172.21.15.161:0/33249609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15c80c6750 msgr2=0x7f15c80afe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:44.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.822+0000 7f15ce240640 1 --2- 172.21.15.161:0/33249609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15c80c6750 0x7f15c80afe30 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f15b800c4d0 tx=0x7f15b800c990 comp rx=0 tx=0).stop 2024-03-28T23:35:44.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.822+0000 7f15ce240640 1 -- 172.21.15.161:0/33249609 shutdown_connections 2024-03-28T23:35:44.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.822+0000 7f15ce240640 1 --2- 172.21.15.161:0/33249609 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f15b406b4d0 0x7f15b406d990 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.822+0000 7f15ce240640 1 --2- 172.21.15.161:0/33249609 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15c81529d0 0x7f15c814cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.822+0000 7f15ce240640 1 --2- 172.21.15.161:0/33249609 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15c80c88b0 0x7f15c814ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.822+0000 7f15ce240640 1 --2- 172.21.15.161:0/33249609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15c80c6750 0x7f15c80afe30 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.822+0000 7f15ce240640 1 -- 172.21.15.161:0/33249609 >> 172.21.15.161:0/33249609 conn(0x7f15c80c7430 msgr2=0x7f15c8074020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:44.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.822+0000 7f15ce240640 1 -- 172.21.15.161:0/33249609 shutdown_connections 2024-03-28T23:35:44.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.822+0000 7f15ce240640 1 -- 172.21.15.161:0/33249609 wait complete. 2024-03-28T23:35:44.834 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:35:44.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.934+0000 7facb55a7640 1 Processor -- start 2024-03-28T23:35:44.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.934+0000 7facb55a7640 1 -- start start 2024-03-28T23:35:44.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.935+0000 7facb55a7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facb014be70 0x7facb00ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.935+0000 7facb55a7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7facb00ab6e0 0x7facb00aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.935+0000 7facb55a7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facb014c250 con 0x7facb006ce80 2024-03-28T23:35:44.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.935+0000 7facb55a7640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7facb01529c0 con 0x7facb00ab6e0 2024-03-28T23:35:44.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.935+0000 7facb55a7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7facb00a9dc0 con 0x7facb014be70 2024-03-28T23:35:44.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.935+0000 7facae7fc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7facb00ab6e0 0x7facb00aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.935+0000 7facae7fc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7facb00ab6e0 0x7facb00aa5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58802/0 (socket says 172.21.15.161:58802) 2024-03-28T23:35:44.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.935+0000 7facae7fc640 1 -- 172.21.15.161:0/2042790978 learned_addr learned my addr 172.21.15.161:0/2042790978 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:44.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.935+0000 7facaeffd640 1 --2- 172.21.15.161:0/2042790978 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facb014be70 0x7facb00ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.935+0000 7facae7fc640 1 -- 172.21.15.161:0/2042790978 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facb014be70 msgr2=0x7facb00ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:44.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.935+0000 7facae7fc640 1 --2- 172.21.15.161:0/2042790978 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facb014be70 0x7facb00ab1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.935+0000 7facae7fc640 1 -- 172.21.15.161:0/2042790978 >> v1:172.21.15.170:6789/0 conn(0x7facb006ce80 legacy=0x7facb01553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:44.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.935+0000 7facae7fc640 1 -- 172.21.15.161:0/2042790978 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facb00aa0a0 con 0x7facb00ab6e0 2024-03-28T23:35:44.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.935+0000 7facadffb640 1 -- 172.21.15.161:0/2042790978 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 453326075 0 0) 0x7facb014c250 con 0x7facb006ce80 2024-03-28T23:35:44.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.936+0000 7facae7fc640 1 --2- 172.21.15.161:0/2042790978 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7facb00ab6e0 0x7facb00aa5c0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7faca400bb10 tx=0x7faca402e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37d8c7319298a564 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:44.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.936+0000 7facadffb640 1 -- 172.21.15.161:0/2042790978 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca403b070 con 0x7facb00ab6e0 2024-03-28T23:35:44.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.936+0000 7facadffb640 1 -- 172.21.15.161:0/2042790978 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faca4036440 con 0x7facb00ab6e0 2024-03-28T23:35:44.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.936+0000 7facadffb640 1 -- 172.21.15.161:0/2042790978 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca4036810 con 0x7facb00ab6e0 2024-03-28T23:35:44.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.937+0000 7facb55a7640 1 -- 172.21.15.161:0/2042790978 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7facb00ab6e0 msgr2=0x7facb00aa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:44.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.937+0000 7facb55a7640 1 --2- 172.21.15.161:0/2042790978 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7facb00ab6e0 0x7facb00aa5c0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7faca400bb10 tx=0x7faca402e0c0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.937+0000 7facb55a7640 1 -- 172.21.15.161:0/2042790978 shutdown_connections 2024-03-28T23:35:44.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.937+0000 7facb55a7640 1 --2- 172.21.15.161:0/2042790978 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7facb00ab6e0 0x7facb00aa5c0 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.937+0000 7facb55a7640 1 --2- 172.21.15.161:0/2042790978 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facb014be70 0x7facb00ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.937+0000 7facb55a7640 1 -- 172.21.15.161:0/2042790978 >> 172.21.15.161:0/2042790978 conn(0x7facb00c6670 msgr2=0x7facb00c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:44.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.937+0000 7facb55a7640 1 -- 172.21.15.161:0/2042790978 shutdown_connections 2024-03-28T23:35:44.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.937+0000 7facb55a7640 1 -- 172.21.15.161:0/2042790978 wait complete. 2024-03-28T23:35:44.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.938+0000 7facb55a7640 1 Processor -- start 2024-03-28T23:35:44.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.938+0000 7facb55a7640 1 -- start start 2024-03-28T23:35:44.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.939+0000 7facb55a7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7facb006ce80 0x7facb01543f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.939+0000 7facb55a7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facb00ab6e0 0x7facb0154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.939+0000 7facb55a7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facb014be70 0x7facb01e9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.939+0000 7facb55a7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7facb006b2d0 con 0x7facb006ce80 2024-03-28T23:35:44.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.939+0000 7facb55a7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7facb006b150 con 0x7facb00ab6e0 2024-03-28T23:35:44.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.939+0000 7facb55a7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7facb00c1520 con 0x7facb014be70 2024-03-28T23:35:44.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.939+0000 7facaeffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7facb006ce80 0x7facb01543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.939+0000 7facaeffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7facb006ce80 0x7facb01543f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58814/0 (socket says 172.21.15.161:58814) 2024-03-28T23:35:44.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.939+0000 7facaeffd640 1 -- 172.21.15.161:0/1235157648 learned_addr learned my addr 172.21.15.161:0/1235157648 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:44.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.939+0000 7facae7fc640 1 --2- 172.21.15.161:0/1235157648 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facb00ab6e0 0x7facb0154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.939+0000 7facaf7fe640 1 --2- 172.21.15.161:0/1235157648 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facb014be70 0x7facb01e9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.939+0000 7facaeffd640 1 -- 172.21.15.161:0/1235157648 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facb014be70 msgr2=0x7facb01e9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:44.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.939+0000 7facaeffd640 1 --2- 172.21.15.161:0/1235157648 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facb014be70 0x7facb01e9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.939+0000 7facaeffd640 1 -- 172.21.15.161:0/1235157648 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facb00ab6e0 msgr2=0x7facb0154930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:44.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.939+0000 7facaeffd640 1 --2- 172.21.15.161:0/1235157648 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facb00ab6e0 0x7facb0154930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:44.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.939+0000 7facaeffd640 1 -- 172.21.15.161:0/1235157648 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facb0145200 con 0x7facb006ce80 2024-03-28T23:35:44.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.939+0000 7facaf7fe640 1 --2- 172.21.15.161:0/1235157648 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facb014be70 0x7facb01e9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:44.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.939+0000 7facaeffd640 1 --2- 172.21.15.161:0/1235157648 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7facb006ce80 0x7facb01543f0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7faca000e8a0 tx=0x7faca000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:44.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.940+0000 7fac977fe640 1 -- 172.21.15.161:0/1235157648 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca0017070 con 0x7facb006ce80 2024-03-28T23:35:44.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.940+0000 7fac977fe640 1 -- 172.21.15.161:0/1235157648 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faca000c8e0 con 0x7facb006ce80 2024-03-28T23:35:44.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.940+0000 7fac977fe640 1 -- 172.21.15.161:0/1235157648 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca000ccd0 con 0x7facb006ce80 2024-03-28T23:35:44.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.940+0000 7facb55a7640 1 -- 172.21.15.161:0/1235157648 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facb01454e0 con 0x7facb006ce80 2024-03-28T23:35:44.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.940+0000 7facb55a7640 1 -- 172.21.15.161:0/1235157648 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facb00c0a50 con 0x7facb006ce80 2024-03-28T23:35:44.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.941+0000 7facb55a7640 1 -- 172.21.15.161:0/1235157648 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facb004f4b0 con 0x7facb006ce80 2024-03-28T23:35:44.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.941+0000 7fac977fe640 1 -- 172.21.15.161:0/1235157648 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7faca001a070 con 0x7facb006ce80 2024-03-28T23:35:44.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.941+0000 7fac977fe640 1 --2- 172.21.15.161:0/1235157648 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fac9c06b4d0 0x7fac9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:44.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.941+0000 7fac977fe640 1 -- 172.21.15.161:0/1235157648 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 6331+0+0 (secure 0 0 0) 0x7faca008a510 con 0x7facb006ce80 2024-03-28T23:35:44.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.941+0000 7facae7fc640 1 --2- 172.21.15.161:0/1235157648 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fac9c06b4d0 0x7fac9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:44.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.942+0000 7facae7fc640 1 --2- 172.21.15.161:0/1235157648 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fac9c06b4d0 0x7fac9c06d990 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7faca4038000 tx=0x7faca4032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:44.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:44.946+0000 7fac977fe640 1 -- 172.21.15.161:0/1235157648 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7faca0058b60 con 0x7facb006ce80 2024-03-28T23:35:45.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.154+0000 7facb55a7640 1 -- 172.21.15.161:0/1235157648 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7facb00c0ff0 con 0x7facb006ce80 2024-03-28T23:35:45.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.155+0000 7fac977fe640 1 -- 172.21.15.161:0/1235157648 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) ==== 95+0+5203 (secure 0 0 0) 0x7faca005cc00 con 0x7facb006ce80 2024-03-28T23:35:45.155 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:45.155 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":134,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7870,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2828648906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":2828648906},{"type":"v1","addr":"172.21.15.170:6835","nonce":2828648906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":132},{"gid":8409,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/3037890643","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3037890643},{"type":"v1","addr":"172.21.15.161:6837","nonce":3037890643}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120},{"gid":8622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/819329318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":819329318},{"type":"v1","addr":"172.21.15.161:6836","nonce":819329318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:12.982566+0000","modified":"2024-03-28T23:35:41.970209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":140,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7639},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7639":{"gid":7639,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":11,"addr":"172.21.15.170:6834/3182285172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3182285172},{"type":"v1","addr":"172.21.15.170:6834","nonce":3182285172}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7639,"qdb_cluster":[7639]},"id":10}]} 2024-03-28T23:35:45.156 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 134 2024-03-28T23:35:45.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.157+0000 7facb55a7640 1 -- 172.21.15.161:0/1235157648 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fac9c06b4d0 msgr2=0x7fac9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:45.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.157+0000 7facb55a7640 1 --2- 172.21.15.161:0/1235157648 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fac9c06b4d0 0x7fac9c06d990 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7faca4038000 tx=0x7faca4032000 comp rx=0 tx=0).stop 2024-03-28T23:35:45.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.157+0000 7facb55a7640 1 -- 172.21.15.161:0/1235157648 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7facb006ce80 msgr2=0x7facb01543f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:45.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.157+0000 7facb55a7640 1 --2- 172.21.15.161:0/1235157648 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7facb006ce80 0x7facb01543f0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7faca000e8a0 tx=0x7faca000ed60 comp rx=0 tx=0).stop 2024-03-28T23:35:45.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.157+0000 7facb55a7640 1 -- 172.21.15.161:0/1235157648 shutdown_connections 2024-03-28T23:35:45.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.157+0000 7facb55a7640 1 --2- 172.21.15.161:0/1235157648 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fac9c06b4d0 0x7fac9c06d990 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.157+0000 7facb55a7640 1 --2- 172.21.15.161:0/1235157648 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facb014be70 0x7facb01e9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.157+0000 7facb55a7640 1 --2- 172.21.15.161:0/1235157648 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facb00ab6e0 0x7facb0154930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.157+0000 7facb55a7640 1 --2- 172.21.15.161:0/1235157648 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7facb006ce80 0x7facb01543f0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.157+0000 7facb55a7640 1 -- 172.21.15.161:0/1235157648 >> 172.21.15.161:0/1235157648 conn(0x7facb00c6670 msgr2=0x7facb00c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:45.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.157+0000 7facb55a7640 1 -- 172.21.15.161:0/1235157648 shutdown_connections 2024-03-28T23:35:45.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.157+0000 7facb55a7640 1 -- 172.21.15.161:0/1235157648 wait complete. 2024-03-28T23:35:45.168 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:35:45.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.269+0000 7f584544d640 1 Processor -- start 2024-03-28T23:35:45.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.269+0000 7f584544d640 1 -- start start 2024-03-28T23:35:45.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.270+0000 7f584544d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58400c3770 0x7f58400c1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:45.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.270+0000 7f584544d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f58400c2300 0x7f584015fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:45.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.270+0000 7f584544d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58400c3bd0 con 0x7f58400c3390 2024-03-28T23:35:45.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.270+0000 7f584544d640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f5840155e80 con 0x7f58400c2300 2024-03-28T23:35:45.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.270+0000 7f584544d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f58400a8410 con 0x7f58400c3770 2024-03-28T23:35:45.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.270+0000 7f583e7fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58400c3770 0x7f58400c1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:45.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.270+0000 7f583f7fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f58400c2300 0x7f584015fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:45.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.270+0000 7f583effd640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f58400c3390 0x7f58400a7d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:54828/0 (socket says 172.21.15.161:54828) 2024-03-28T23:35:45.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.270+0000 7f583effd640 1 -- 172.21.15.161:0/3358203353 learned_addr learned my addr 172.21.15.161:0/3358203353 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:45.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.270+0000 7f583dffb640 1 -- 172.21.15.161:0/3358203353 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 446213620 0 0) 0x7f58400c3bd0 con 0x7f58400c3390 2024-03-28T23:35:45.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.270+0000 7f583e7fc640 1 -- 172.21.15.161:0/3358203353 >> v2:172.21.15.170:3300/0 conn(0x7f58400c2300 msgr2=0x7f584015fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:45.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.270+0000 7f583e7fc640 1 --2- 172.21.15.161:0/3358203353 >> v2:172.21.15.170:3300/0 conn(0x7f58400c2300 0x7f584015fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.270+0000 7f583e7fc640 1 -- 172.21.15.161:0/3358203353 >> v1:172.21.15.170:6789/0 conn(0x7f58400c3390 legacy=0x7f58400a7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:45.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.270+0000 7f583e7fc640 1 -- 172.21.15.161:0/3358203353 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5840155270 con 0x7f58400c3770 2024-03-28T23:35:45.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.271+0000 7f583e7fc640 1 --2- 172.21.15.161:0/3358203353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58400c3770 0x7f58400c1dc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f58340097a0 tx=0x7f583402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b3576fa1958ac9a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:45.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.271+0000 7f583dffb640 1 -- 172.21.15.161:0/3358203353 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58340315a0 con 0x7f58400c3770 2024-03-28T23:35:45.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.271+0000 7f583dffb640 1 -- 172.21.15.161:0/3358203353 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5834037070 con 0x7f58400c3770 2024-03-28T23:35:45.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.271+0000 7f583dffb640 1 -- 172.21.15.161:0/3358203353 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5834031960 con 0x7f58400c3770 2024-03-28T23:35:45.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.272+0000 7f584544d640 1 -- 172.21.15.161:0/3358203353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58400c3770 msgr2=0x7f58400c1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:45.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.272+0000 7f584544d640 1 --2- 172.21.15.161:0/3358203353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58400c3770 0x7f58400c1dc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f58340097a0 tx=0x7f583402b840 comp rx=0 tx=0).stop 2024-03-28T23:35:45.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.272+0000 7f584544d640 1 -- 172.21.15.161:0/3358203353 shutdown_connections 2024-03-28T23:35:45.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.272+0000 7f584544d640 1 --2- 172.21.15.161:0/3358203353 >> v2:172.21.15.170:3300/0 conn(0x7f58400c2300 0x7f584015fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.272+0000 7f584544d640 1 --2- 172.21.15.161:0/3358203353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58400c3770 0x7f58400c1dc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.272+0000 7f584544d640 1 -- 172.21.15.161:0/3358203353 >> 172.21.15.161:0/3358203353 conn(0x7f58400aa740 msgr2=0x7f58400aab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:45.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.272+0000 7f584544d640 1 -- 172.21.15.161:0/3358203353 shutdown_connections 2024-03-28T23:35:45.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.272+0000 7f584544d640 1 -- 172.21.15.161:0/3358203353 wait complete. 2024-03-28T23:35:45.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.273+0000 7f584544d640 1 Processor -- start 2024-03-28T23:35:45.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.273+0000 7f584544d640 1 -- start start 2024-03-28T23:35:45.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.274+0000 7f584544d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58400c2300 0x7f584014be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:45.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.274+0000 7f584544d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58400c3390 0x7f58400ae220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:45.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.274+0000 7f584544d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58400c3770 0x7f58400aba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:45.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.274+0000 7f584544d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5840155b10 con 0x7f58400c3390 2024-03-28T23:35:45.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.274+0000 7f584544d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5840155830 con 0x7f58400c3770 2024-03-28T23:35:45.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.274+0000 7f584544d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5840152a80 con 0x7f58400c2300 2024-03-28T23:35:45.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.274+0000 7f583e7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58400c3390 0x7f58400ae220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:45.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.274+0000 7f583effd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58400c2300 0x7f584014be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:45.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.274+0000 7f583f7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58400c3770 0x7f58400aba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:45.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.274+0000 7f583effd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58400c2300 0x7f584014be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:41484/0 (socket says 172.21.15.161:41484) 2024-03-28T23:35:45.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.274+0000 7f583e7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58400c3390 0x7f58400ae220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58816/0 (socket says 172.21.15.161:58816) 2024-03-28T23:35:45.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.274+0000 7f583effd640 1 -- 172.21.15.161:0/3114540105 learned_addr learned my addr 172.21.15.161:0/3114540105 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:45.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.274+0000 7f583effd640 1 -- 172.21.15.161:0/3114540105 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58400c3770 msgr2=0x7f58400aba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:45.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.274+0000 7f583effd640 1 --2- 172.21.15.161:0/3114540105 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58400c3770 0x7f58400aba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.274+0000 7f583effd640 1 -- 172.21.15.161:0/3114540105 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58400c3390 msgr2=0x7f58400ae220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:45.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.274+0000 7f583effd640 1 --2- 172.21.15.161:0/3114540105 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58400c3390 0x7f58400ae220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.274+0000 7f583effd640 1 -- 172.21.15.161:0/3114540105 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58401eadf0 con 0x7f58400c2300 2024-03-28T23:35:45.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.274+0000 7f583effd640 1 --2- 172.21.15.161:0/3114540105 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58400c2300 0x7f584014be30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f583000c4d0 tx=0x7f583000c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:45.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.275+0000 7f58277fe640 1 -- 172.21.15.161:0/3114540105 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5830018070 con 0x7f58400c2300 2024-03-28T23:35:45.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.275+0000 7f58277fe640 1 -- 172.21.15.161:0/3114540105 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5830007500 con 0x7f58400c2300 2024-03-28T23:35:45.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.275+0000 7f584544d640 1 -- 172.21.15.161:0/3114540105 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f584014bb50 con 0x7f58400c2300 2024-03-28T23:35:45.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.275+0000 7f58277fe640 1 -- 172.21.15.161:0/3114540105 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5830004a20 con 0x7f58400c2300 2024-03-28T23:35:45.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.275+0000 7f584544d640 1 -- 172.21.15.161:0/3114540105 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f58401eb390 con 0x7f58400c2300 2024-03-28T23:35:45.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.275+0000 7f584544d640 1 -- 172.21.15.161:0/3114540105 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58400c43d0 con 0x7f58400c2300 2024-03-28T23:35:45.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.276+0000 7f58277fe640 1 -- 172.21.15.161:0/3114540105 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f583001b070 con 0x7f58400c2300 2024-03-28T23:35:45.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.276+0000 7f58277fe640 1 --2- 172.21.15.161:0/3114540105 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f582c06b4d0 0x7f582c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:45.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.276+0000 7f58277fe640 1 -- 172.21.15.161:0/3114540105 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(140..140 src has 1..140) ==== 6331+0+0 (secure 0 0 0) 0x7f5830089640 con 0x7f58400c2300 2024-03-28T23:35:45.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.276+0000 7f583e7fc640 1 --2- 172.21.15.161:0/3114540105 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f582c06b4d0 0x7f582c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:45.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.277+0000 7f583e7fc640 1 --2- 172.21.15.161:0/3114540105 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f582c06b4d0 0x7f582c06d990 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5834036000 tx=0x7f583403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:45.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.281+0000 7f58277fe640 1 -- 172.21.15.161:0/3114540105 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f58400c43d0 con 0x7f58400c2300 2024-03-28T23:35:45.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.455+0000 7f584544d640 1 -- 172.21.15.161:0/3114540105 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f584004f4b0 con 0x7f58400c2300 2024-03-28T23:35:45.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.457+0000 7f58277fe640 1 -- 172.21.15.161:0/3114540105 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) ==== 74+0+17349 (secure 0 0 0) 0x7f5830057c90 con 0x7f58400c2300 2024-03-28T23:35:45.457 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:45.457 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":140,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:35:38.963165+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:35:11.208674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-28T23:35:11.954908+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"136","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"22.21","mappings":[{"from":3,"to":0}]},{"pgid":"22.37","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6835/3586875977":"2024-03-29T23:35:38.963130+0000","172.21.15.170:6833/3586875977":"2024-03-29T23:35:38.963130+0000","172.21.15.161:6836/2527502405":"2024-03-29T23:35:21.284385+0000","172.21.15.161:6834/2527502405":"2024-03-29T23:35:21.284385+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:35:45.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.458+0000 7f584544d640 1 -- 172.21.15.161:0/3114540105 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f582c06b4d0 msgr2=0x7f582c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:45.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.458+0000 7f584544d640 1 --2- 172.21.15.161:0/3114540105 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f582c06b4d0 0x7f582c06d990 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5834036000 tx=0x7f583403d040 comp rx=0 tx=0).stop 2024-03-28T23:35:45.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.458+0000 7f584544d640 1 -- 172.21.15.161:0/3114540105 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58400c2300 msgr2=0x7f584014be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:45.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.458+0000 7f584544d640 1 --2- 172.21.15.161:0/3114540105 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58400c2300 0x7f584014be30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f583000c4d0 tx=0x7f583000c990 comp rx=0 tx=0).stop 2024-03-28T23:35:45.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.458+0000 7f584544d640 1 -- 172.21.15.161:0/3114540105 shutdown_connections 2024-03-28T23:35:45.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.458+0000 7f584544d640 1 --2- 172.21.15.161:0/3114540105 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f582c06b4d0 0x7f582c06d990 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.458+0000 7f584544d640 1 --2- 172.21.15.161:0/3114540105 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58400c3390 0x7f58400ae220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.458+0000 7f584544d640 1 --2- 172.21.15.161:0/3114540105 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58400c3770 0x7f58400aba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.458+0000 7f584544d640 1 --2- 172.21.15.161:0/3114540105 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58400c2300 0x7f584014be30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.458+0000 7f584544d640 1 -- 172.21.15.161:0/3114540105 >> 172.21.15.161:0/3114540105 conn(0x7f58400aa740 msgr2=0x7f5840153ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:45.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.459+0000 7f584544d640 1 -- 172.21.15.161:0/3114540105 shutdown_connections 2024-03-28T23:35:45.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.459+0000 7f584544d640 1 -- 172.21.15.161:0/3114540105 wait complete. 2024-03-28T23:35:45.470 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-28T23:35:45.470 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:35:45.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.571+0000 7f5606e40640 1 Processor -- start 2024-03-28T23:35:45.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.571+0000 7f5606e40640 1 -- start start 2024-03-28T23:35:45.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.572+0000 7f5606e40640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f560014be70 0x7f56000ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:45.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.572+0000 7f5606e40640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f560014c250 con 0x7f56000aacd0 2024-03-28T23:35:45.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.572+0000 7f5606e40640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56001529c0 con 0x7f56000ab6e0 2024-03-28T23:35:45.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.572+0000 7f5606e40640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f56000a9dc0 con 0x7f560014be70 2024-03-28T23:35:45.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.572+0000 7f5604bdc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f560014be70 0x7f56000ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:45.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.572+0000 7f56053dd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f56000aacd0 0x7f560015c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38170/0 (socket says 172.21.15.161:38170) 2024-03-28T23:35:45.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.572+0000 7f56053dd640 1 -- 172.21.15.161:0/655181863 learned_addr learned my addr 172.21.15.161:0/655181863 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:45.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.572+0000 7f55f77fe640 1 -- 172.21.15.161:0/655181863 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2773697589 0 0) 0x7f56001529c0 con 0x7f56000ab6e0 2024-03-28T23:35:45.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.572+0000 7f5604bdc640 1 -- 172.21.15.161:0/655181863 >> v1:172.21.15.170:6789/0 conn(0x7f56000ab6e0 legacy=0x7f56000aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:45.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.572+0000 7f5604bdc640 1 -- 172.21.15.161:0/655181863 >> v1:172.21.15.161:6789/0 conn(0x7f56000aacd0 legacy=0x7f560015c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:45.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.572+0000 7f5604bdc640 1 -- 172.21.15.161:0/655181863 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56000aa0a0 con 0x7f560014be70 2024-03-28T23:35:45.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.573+0000 7f5604bdc640 1 --2- 172.21.15.161:0/655181863 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f560014be70 0x7f56000ab1a0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f55fc0097a0 tx=0x7f55fc02b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4515f74935e52390 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:45.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.573+0000 7f55f77fe640 1 -- 172.21.15.161:0/655181863 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55fc037070 con 0x7f560014be70 2024-03-28T23:35:45.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.573+0000 7f55f77fe640 1 -- 172.21.15.161:0/655181863 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55fc030710 con 0x7f560014be70 2024-03-28T23:35:45.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.573+0000 7f55f77fe640 1 -- 172.21.15.161:0/655181863 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55fc030ae0 con 0x7f560014be70 2024-03-28T23:35:45.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.573+0000 7f5606e40640 1 -- 172.21.15.161:0/655181863 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f560014be70 msgr2=0x7f56000ab1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:45.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.573+0000 7f5606e40640 1 --2- 172.21.15.161:0/655181863 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f560014be70 0x7f56000ab1a0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f55fc0097a0 tx=0x7f55fc02b710 comp rx=0 tx=0).stop 2024-03-28T23:35:45.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.574+0000 7f5606e40640 1 -- 172.21.15.161:0/655181863 shutdown_connections 2024-03-28T23:35:45.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.574+0000 7f5606e40640 1 --2- 172.21.15.161:0/655181863 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f560014be70 0x7f56000ab1a0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.574+0000 7f5606e40640 1 -- 172.21.15.161:0/655181863 >> 172.21.15.161:0/655181863 conn(0x7f56000c6670 msgr2=0x7f56000c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:45.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.574+0000 7f5606e40640 1 -- 172.21.15.161:0/655181863 shutdown_connections 2024-03-28T23:35:45.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.574+0000 7f5606e40640 1 -- 172.21.15.161:0/655181863 wait complete. 2024-03-28T23:35:45.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.575+0000 7f5606e40640 1 Processor -- start 2024-03-28T23:35:45.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.575+0000 7f5606e40640 1 -- start start 2024-03-28T23:35:45.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.575+0000 7f5606e40640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56000aacd0 0x7f56001e8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:45.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.575+0000 7f5606e40640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56000ab6e0 0x7f56001db910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:45.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.575+0000 7f5606e40640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f560014be70 0x7f56001ec800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:45.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.575+0000 7f5606e40640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f560006b2d0 con 0x7f560014be70 2024-03-28T23:35:45.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.575+0000 7f5606e40640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f560006b150 con 0x7f56000aacd0 2024-03-28T23:35:45.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.575+0000 7f5606e40640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f56000c1520 con 0x7f56000ab6e0 2024-03-28T23:35:45.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.576+0000 7f56053dd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f560014be70 0x7f56001ec800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:45.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.576+0000 7f56053dd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f560014be70 0x7f56001ec800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58838/0 (socket says 172.21.15.161:58838) 2024-03-28T23:35:45.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.576+0000 7f56053dd640 1 -- 172.21.15.161:0/4102618282 learned_addr learned my addr 172.21.15.161:0/4102618282 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:45.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.576+0000 7f5604bdc640 1 --2- 172.21.15.161:0/4102618282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56000aacd0 0x7f56001e8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:45.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.576+0000 7f55f7fff640 1 --2- 172.21.15.161:0/4102618282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56000ab6e0 0x7f56001db910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:45.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.576+0000 7f56053dd640 1 -- 172.21.15.161:0/4102618282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56000ab6e0 msgr2=0x7f56001db910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:45.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.576+0000 7f56053dd640 1 --2- 172.21.15.161:0/4102618282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56000ab6e0 0x7f56001db910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.576+0000 7f56053dd640 1 -- 172.21.15.161:0/4102618282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56000aacd0 msgr2=0x7f56001e8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:45.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.576+0000 7f56053dd640 1 --2- 172.21.15.161:0/4102618282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56000aacd0 0x7f56001e8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.576+0000 7f56053dd640 1 -- 172.21.15.161:0/4102618282 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56001dd060 con 0x7f560014be70 2024-03-28T23:35:45.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.576+0000 7f5604bdc640 1 --2- 172.21.15.161:0/4102618282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56000aacd0 0x7f56001e8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:45.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.576+0000 7f56053dd640 1 --2- 172.21.15.161:0/4102618282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f560014be70 0x7f56001ec800 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f55f800c910 tx=0x7f55f800cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:45.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.576+0000 7f55f57fa640 1 -- 172.21.15.161:0/4102618282 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f8018070 con 0x7f560014be70 2024-03-28T23:35:45.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.576+0000 7f55f57fa640 1 -- 172.21.15.161:0/4102618282 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55f800a6a0 con 0x7f560014be70 2024-03-28T23:35:45.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.576+0000 7f5606e40640 1 -- 172.21.15.161:0/4102618282 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f560014f310 con 0x7f560014be70 2024-03-28T23:35:45.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.576+0000 7f55f57fa640 1 -- 172.21.15.161:0/4102618282 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f800aad0 con 0x7f560014be70 2024-03-28T23:35:45.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.576+0000 7f5606e40640 1 -- 172.21.15.161:0/4102618282 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f560014f890 con 0x7f560014be70 2024-03-28T23:35:45.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.577+0000 7f5606e40640 1 -- 172.21.15.161:0/4102618282 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f560004f4b0 con 0x7f560014be70 2024-03-28T23:35:45.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.578+0000 7f55f57fa640 1 -- 172.21.15.161:0/4102618282 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f55f801b070 con 0x7f560014be70 2024-03-28T23:35:45.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.578+0000 7f55f57fa640 1 --2- 172.21.15.161:0/4102618282 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f55ec06b4d0 0x7f55ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:45.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.578+0000 7f5604bdc640 1 --2- 172.21.15.161:0/4102618282 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f55ec06b4d0 0x7f55ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:45.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.578+0000 7f55f57fa640 1 -- 172.21.15.161:0/4102618282 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 6331+0+0 (secure 0 0 0) 0x7f55f80899c0 con 0x7f560014be70 2024-03-28T23:35:45.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.579+0000 7f5604bdc640 1 --2- 172.21.15.161:0/4102618282 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f55ec06b4d0 0x7f55ec06d990 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f55fc034000 tx=0x7f55fc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:45.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.583+0000 7f55f57fa640 1 -- 172.21.15.161:0/4102618282 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f55f808be20 con 0x7f560014be70 2024-03-28T23:35:45.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.794+0000 7f5606e40640 1 -- 172.21.15.161:0/4102618282 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56000c0ff0 con 0x7f560014be70 2024-03-28T23:35:45.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.795+0000 7f55f57fa640 1 -- 172.21.15.161:0/4102618282 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) ==== 95+0+5203 (secure 0 0 0) 0x7f55f800ac70 con 0x7f560014be70 2024-03-28T23:35:45.795 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:45.795 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":134,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7870,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2828648906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":2828648906},{"type":"v1","addr":"172.21.15.170:6835","nonce":2828648906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":132},{"gid":8409,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/3037890643","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3037890643},{"type":"v1","addr":"172.21.15.161:6837","nonce":3037890643}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120},{"gid":8622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/819329318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":819329318},{"type":"v1","addr":"172.21.15.161:6836","nonce":819329318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:12.982566+0000","modified":"2024-03-28T23:35:41.970209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":140,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7639},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7639":{"gid":7639,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":11,"addr":"172.21.15.170:6834/3182285172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3182285172},{"type":"v1","addr":"172.21.15.170:6834","nonce":3182285172}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7639,"qdb_cluster":[7639]},"id":10}]} 2024-03-28T23:35:45.795 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 134 2024-03-28T23:35:45.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.797+0000 7f5606e40640 1 -- 172.21.15.161:0/4102618282 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f55ec06b4d0 msgr2=0x7f55ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:45.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.797+0000 7f5606e40640 1 --2- 172.21.15.161:0/4102618282 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f55ec06b4d0 0x7f55ec06d990 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f55fc034000 tx=0x7f55fc008030 comp rx=0 tx=0).stop 2024-03-28T23:35:45.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.797+0000 7f5606e40640 1 -- 172.21.15.161:0/4102618282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f560014be70 msgr2=0x7f56001ec800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:45.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.797+0000 7f5606e40640 1 --2- 172.21.15.161:0/4102618282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f560014be70 0x7f56001ec800 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f55f800c910 tx=0x7f55f800cdd0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.797+0000 7f5606e40640 1 -- 172.21.15.161:0/4102618282 shutdown_connections 2024-03-28T23:35:45.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.797+0000 7f5606e40640 1 --2- 172.21.15.161:0/4102618282 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f55ec06b4d0 0x7f55ec06d990 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.797+0000 7f5606e40640 1 --2- 172.21.15.161:0/4102618282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f560014be70 0x7f56001ec800 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.797+0000 7f5606e40640 1 --2- 172.21.15.161:0/4102618282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56000ab6e0 0x7f56001db910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.797+0000 7f5606e40640 1 --2- 172.21.15.161:0/4102618282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56000aacd0 0x7f56001e8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.797+0000 7f5606e40640 1 -- 172.21.15.161:0/4102618282 >> 172.21.15.161:0/4102618282 conn(0x7f56000c6670 msgr2=0x7f56001515a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:45.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.797+0000 7f5606e40640 1 -- 172.21.15.161:0/4102618282 shutdown_connections 2024-03-28T23:35:45.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.797+0000 7f5606e40640 1 -- 172.21.15.161:0/4102618282 wait complete. 2024-03-28T23:35:45.808 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:35:45.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.908+0000 7fba234f4640 1 Processor -- start 2024-03-28T23:35:45.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.908+0000 7fba234f4640 1 -- start start 2024-03-28T23:35:45.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.909+0000 7fba234f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba1c0c5410 0x7fba1c0c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:45.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.909+0000 7fba234f4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba1c158a30 con 0x7fba1c152890 2024-03-28T23:35:45.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.909+0000 7fba234f4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba1c158d10 con 0x7fba1c1553a0 2024-03-28T23:35:45.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.909+0000 7fba234f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fba1c0c8480 con 0x7fba1c0c5410 2024-03-28T23:35:45.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.909+0000 7fba21290640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fba1c152890 0x7fba1c0c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38174/0 (socket says 172.21.15.161:38174) 2024-03-28T23:35:45.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.909+0000 7fba21290640 1 -- 172.21.15.161:0/990531640 learned_addr learned my addr 172.21.15.161:0/990531640 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:45.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.909+0000 7fba20a8f640 1 --2- 172.21.15.161:0/990531640 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba1c0c5410 0x7fba1c0c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:45.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.910+0000 7fba13fff640 1 -- 172.21.15.161:0/990531640 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 538157458 0 0) 0x7fba1c158a30 con 0x7fba1c152890 2024-03-28T23:35:45.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.910+0000 7fba13fff640 1 -- 172.21.15.161:0/990531640 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba08003520 con 0x7fba1c152890 2024-03-28T23:35:45.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.910+0000 7fba13fff640 1 -- 172.21.15.161:0/990531640 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2836197732 0 0) 0x7fba1c158d10 con 0x7fba1c1553a0 2024-03-28T23:35:45.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.910+0000 7fba13fff640 1 -- 172.21.15.161:0/990531640 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba1c158a30 con 0x7fba1c1553a0 2024-03-28T23:35:45.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.910+0000 7fba20a8f640 1 -- 172.21.15.161:0/990531640 >> v1:172.21.15.170:6789/0 conn(0x7fba1c1553a0 legacy=0x7fba1c155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:45.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.910+0000 7fba20a8f640 1 -- 172.21.15.161:0/990531640 >> v1:172.21.15.161:6789/0 conn(0x7fba1c152890 legacy=0x7fba1c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:45.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.910+0000 7fba20a8f640 1 -- 172.21.15.161:0/990531640 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba1c158ff0 con 0x7fba1c0c5410 2024-03-28T23:35:45.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.910+0000 7fba20a8f640 1 --2- 172.21.15.161:0/990531640 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba1c0c5410 0x7fba1c0c5850 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fba0c0097c0 tx=0x7fba0c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3f8cbbff6317f02 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:45.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.911+0000 7fba13fff640 1 -- 172.21.15.161:0/990531640 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0c035020 con 0x7fba1c0c5410 2024-03-28T23:35:45.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.911+0000 7fba13fff640 1 -- 172.21.15.161:0/990531640 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba0c031800 con 0x7fba1c0c5410 2024-03-28T23:35:45.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.911+0000 7fba13fff640 1 -- 172.21.15.161:0/990531640 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0c031bd0 con 0x7fba1c0c5410 2024-03-28T23:35:45.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.911+0000 7fba234f4640 1 -- 172.21.15.161:0/990531640 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba1c0c5410 msgr2=0x7fba1c0c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:45.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.911+0000 7fba234f4640 1 --2- 172.21.15.161:0/990531640 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba1c0c5410 0x7fba1c0c5850 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fba0c0097c0 tx=0x7fba0c02b8a0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.911+0000 7fba234f4640 1 -- 172.21.15.161:0/990531640 shutdown_connections 2024-03-28T23:35:45.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.911+0000 7fba234f4640 1 --2- 172.21.15.161:0/990531640 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba1c0c5410 0x7fba1c0c5850 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.911+0000 7fba234f4640 1 -- 172.21.15.161:0/990531640 >> 172.21.15.161:0/990531640 conn(0x7fba1c0ac620 msgr2=0x7fba1c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:45.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.911+0000 7fba234f4640 1 -- 172.21.15.161:0/990531640 shutdown_connections 2024-03-28T23:35:45.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.911+0000 7fba234f4640 1 -- 172.21.15.161:0/990531640 wait complete. 2024-03-28T23:35:45.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.912+0000 7fba234f4640 1 Processor -- start 2024-03-28T23:35:45.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.912+0000 7fba234f4640 1 -- start start 2024-03-28T23:35:45.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.913+0000 7fba234f4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba1c0c5410 0x7fba1c157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:45.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.913+0000 7fba234f4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba1c152890 0x7fba1c146f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:45.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.913+0000 7fba234f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba1c1553a0 0x7fba1c147460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:45.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.913+0000 7fba234f4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fba1c159640 con 0x7fba1c0c5410 2024-03-28T23:35:45.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.913+0000 7fba234f4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fba1c0c5d90 con 0x7fba1c152890 2024-03-28T23:35:45.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.913+0000 7fba234f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fba1c0c90a0 con 0x7fba1c1553a0 2024-03-28T23:35:45.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.913+0000 7fba21290640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba1c0c5410 0x7fba1c157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:45.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.913+0000 7fba21290640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba1c0c5410 0x7fba1c157a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58846/0 (socket says 172.21.15.161:58846) 2024-03-28T23:35:45.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.913+0000 7fba21290640 1 -- 172.21.15.161:0/1020836466 learned_addr learned my addr 172.21.15.161:0/1020836466 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:45.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.913+0000 7fba20a8f640 1 --2- 172.21.15.161:0/1020836466 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba1c152890 0x7fba1c146f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:45.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.913+0000 7fba21a91640 1 --2- 172.21.15.161:0/1020836466 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba1c1553a0 0x7fba1c147460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:45.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.913+0000 7fba21290640 1 -- 172.21.15.161:0/1020836466 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba1c1553a0 msgr2=0x7fba1c147460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:45.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.913+0000 7fba21290640 1 --2- 172.21.15.161:0/1020836466 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba1c1553a0 0x7fba1c147460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.913+0000 7fba21290640 1 -- 172.21.15.161:0/1020836466 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba1c152890 msgr2=0x7fba1c146f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:45.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.913+0000 7fba21290640 1 --2- 172.21.15.161:0/1020836466 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba1c152890 0x7fba1c146f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:45.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.913+0000 7fba21290640 1 -- 172.21.15.161:0/1020836466 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba1c14cd80 con 0x7fba1c0c5410 2024-03-28T23:35:45.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.914+0000 7fba21290640 1 --2- 172.21.15.161:0/1020836466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba1c0c5410 0x7fba1c157a30 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fba1800cbf0 tx=0x7fba1800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:45.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.914+0000 7fba11ffb640 1 -- 172.21.15.161:0/1020836466 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba18017070 con 0x7fba1c0c5410 2024-03-28T23:35:45.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.914+0000 7fba11ffb640 1 -- 172.21.15.161:0/1020836466 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba1800ad50 con 0x7fba1c0c5410 2024-03-28T23:35:45.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.914+0000 7fba234f4640 1 -- 172.21.15.161:0/1020836466 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba1c14d060 con 0x7fba1c0c5410 2024-03-28T23:35:45.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.914+0000 7fba11ffb640 1 -- 172.21.15.161:0/1020836466 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba18012620 con 0x7fba1c0c5410 2024-03-28T23:35:45.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.914+0000 7fba234f4640 1 -- 172.21.15.161:0/1020836466 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba1c1ef390 con 0x7fba1c0c5410 2024-03-28T23:35:45.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.915+0000 7fba234f4640 1 -- 172.21.15.161:0/1020836466 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba1c04f4b0 con 0x7fba1c0c5410 2024-03-28T23:35:45.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.915+0000 7fba11ffb640 1 -- 172.21.15.161:0/1020836466 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fba1801a070 con 0x7fba1c0c5410 2024-03-28T23:35:45.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.916+0000 7fba11ffb640 1 --2- 172.21.15.161:0/1020836466 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fba0006b4d0 0x7fba0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:45.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.916+0000 7fba11ffb640 1 -- 172.21.15.161:0/1020836466 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 6331+0+0 (secure 0 0 0) 0x7fba1808a2b0 con 0x7fba1c0c5410 2024-03-28T23:35:45.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.916+0000 7fba20a8f640 1 --2- 172.21.15.161:0/1020836466 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fba0006b4d0 0x7fba0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:45.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.916+0000 7fba20a8f640 1 --2- 172.21.15.161:0/1020836466 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fba0006b4d0 0x7fba0006d990 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fba0c036000 tx=0x7fba0c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:45.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:45.920+0000 7fba11ffb640 1 -- 172.21.15.161:0/1020836466 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fba18058880 con 0x7fba1c0c5410 2024-03-28T23:35:46.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.129+0000 7fba234f4640 1 -- 172.21.15.161:0/1020836466 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba1c0ac400 con 0x7fba1c0c5410 2024-03-28T23:35:46.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.129+0000 7fba11ffb640 1 -- 172.21.15.161:0/1020836466 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) ==== 95+0+5203 (secure 0 0 0) 0x7fba1805c920 con 0x7fba1c0c5410 2024-03-28T23:35:46.130 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:46.130 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":134,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7870,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2828648906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":2828648906},{"type":"v1","addr":"172.21.15.170:6835","nonce":2828648906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":132},{"gid":8409,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/3037890643","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3037890643},{"type":"v1","addr":"172.21.15.161:6837","nonce":3037890643}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120},{"gid":8622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/819329318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":819329318},{"type":"v1","addr":"172.21.15.161:6836","nonce":819329318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:12.982566+0000","modified":"2024-03-28T23:35:41.970209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":140,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7639},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7639":{"gid":7639,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":11,"addr":"172.21.15.170:6834/3182285172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3182285172},{"type":"v1","addr":"172.21.15.170:6834","nonce":3182285172}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7639,"qdb_cluster":[7639]},"id":10}]} 2024-03-28T23:35:46.130 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 134 2024-03-28T23:35:46.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.131+0000 7fba234f4640 1 -- 172.21.15.161:0/1020836466 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fba0006b4d0 msgr2=0x7fba0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:46.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.131+0000 7fba234f4640 1 --2- 172.21.15.161:0/1020836466 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fba0006b4d0 0x7fba0006d990 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fba0c036000 tx=0x7fba0c008030 comp rx=0 tx=0).stop 2024-03-28T23:35:46.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.131+0000 7fba234f4640 1 -- 172.21.15.161:0/1020836466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba1c0c5410 msgr2=0x7fba1c157a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:46.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.131+0000 7fba234f4640 1 --2- 172.21.15.161:0/1020836466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba1c0c5410 0x7fba1c157a30 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fba1800cbf0 tx=0x7fba1800a430 comp rx=0 tx=0).stop 2024-03-28T23:35:46.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.131+0000 7fba234f4640 1 -- 172.21.15.161:0/1020836466 shutdown_connections 2024-03-28T23:35:46.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.131+0000 7fba234f4640 1 --2- 172.21.15.161:0/1020836466 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fba0006b4d0 0x7fba0006d990 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.131+0000 7fba234f4640 1 --2- 172.21.15.161:0/1020836466 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba1c1553a0 0x7fba1c147460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.131+0000 7fba234f4640 1 --2- 172.21.15.161:0/1020836466 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba1c152890 0x7fba1c146f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.131+0000 7fba234f4640 1 --2- 172.21.15.161:0/1020836466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba1c0c5410 0x7fba1c157a30 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.131+0000 7fba234f4640 1 -- 172.21.15.161:0/1020836466 >> 172.21.15.161:0/1020836466 conn(0x7fba1c0ac620 msgr2=0x7fba1c0caa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:46.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.131+0000 7fba234f4640 1 -- 172.21.15.161:0/1020836466 shutdown_connections 2024-03-28T23:35:46.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.131+0000 7fba234f4640 1 -- 172.21.15.161:0/1020836466 wait complete. 2024-03-28T23:35:46.143 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:35:46.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.243+0000 7f72d4c9d640 1 Processor -- start 2024-03-28T23:35:46.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.243+0000 7f72d4c9d640 1 -- start start 2024-03-28T23:35:46.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.244+0000 7f72d4c9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f72d0152890 0x7f72d00c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:46.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.244+0000 7f72d4c9d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f72d00c8ed0 0x7f72d00c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:46.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.244+0000 7f72d4c9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72d00c5950 0x7f72d01553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:46.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.244+0000 7f72d4c9d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f72d00c8480 con 0x7f72d0152890 2024-03-28T23:35:46.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.244+0000 7f72d4c9d640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f72d0159640 con 0x7f72d00c8ed0 2024-03-28T23:35:46.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.244+0000 7f72d4c9d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f72d0158a30 con 0x7f72d00c5950 2024-03-28T23:35:46.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.244+0000 7f72ce59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f72d0152890 0x7f72d00c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:46.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.244+0000 7f72ce59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f72d0152890 0x7f72d00c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58854/0 (socket says 172.21.15.161:58854) 2024-03-28T23:35:46.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.244+0000 7f72ce59c640 1 -- 172.21.15.161:0/3645603404 learned_addr learned my addr 172.21.15.161:0/3645603404 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:46.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.244+0000 7f72ced9d640 1 --2- 172.21.15.161:0/3645603404 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72d00c5950 0x7f72d01553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:46.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.244+0000 7f72c7fff640 1 --2- 172.21.15.161:0/3645603404 >> v2:172.21.15.170:3300/0 conn(0x7f72d00c8ed0 0x7f72d00c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:46.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.245+0000 7f72ce59c640 1 -- 172.21.15.161:0/3645603404 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72d00c5950 msgr2=0x7f72d01553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:46.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.245+0000 7f72ce59c640 1 --2- 172.21.15.161:0/3645603404 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72d00c5950 0x7f72d01553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.245+0000 7f72ce59c640 1 -- 172.21.15.161:0/3645603404 >> v2:172.21.15.170:3300/0 conn(0x7f72d00c8ed0 msgr2=0x7f72d00c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:46.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.245+0000 7f72ce59c640 1 --2- 172.21.15.161:0/3645603404 >> v2:172.21.15.170:3300/0 conn(0x7f72d00c8ed0 0x7f72d00c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.245+0000 7f72ce59c640 1 -- 172.21.15.161:0/3645603404 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d0158d10 con 0x7f72d0152890 2024-03-28T23:35:46.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.245+0000 7f72ce59c640 1 --2- 172.21.15.161:0/3645603404 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72d0152890 0x7f72d00c8990 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f72d00ad6e0 tx=0x7f72c002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25691e701af6c60c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:46.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.245+0000 7f72cdd9b640 1 -- 172.21.15.161:0/3645603404 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c0038070 con 0x7f72d0152890 2024-03-28T23:35:46.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.246+0000 7f72cdd9b640 1 -- 172.21.15.161:0/3645603404 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72c0031740 con 0x7f72d0152890 2024-03-28T23:35:46.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.246+0000 7f72cdd9b640 1 -- 172.21.15.161:0/3645603404 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c002f050 con 0x7f72d0152890 2024-03-28T23:35:46.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.246+0000 7f72d4c9d640 1 -- 172.21.15.161:0/3645603404 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72d0152890 msgr2=0x7f72d00c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:46.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.246+0000 7f72d4c9d640 1 --2- 172.21.15.161:0/3645603404 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72d0152890 0x7f72d00c8990 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f72d00ad6e0 tx=0x7f72c002b7e0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.246+0000 7f72d4c9d640 1 -- 172.21.15.161:0/3645603404 shutdown_connections 2024-03-28T23:35:46.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.246+0000 7f72d4c9d640 1 --2- 172.21.15.161:0/3645603404 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72d00c5950 0x7f72d01553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.246+0000 7f72d4c9d640 1 --2- 172.21.15.161:0/3645603404 >> v2:172.21.15.170:3300/0 conn(0x7f72d00c8ed0 0x7f72d00c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.246+0000 7f72d4c9d640 1 --2- 172.21.15.161:0/3645603404 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72d0152890 0x7f72d00c8990 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.246+0000 7f72d4c9d640 1 -- 172.21.15.161:0/3645603404 >> 172.21.15.161:0/3645603404 conn(0x7f72d00ac620 msgr2=0x7f72d00aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.246+0000 7f72d4c9d640 1 -- 172.21.15.161:0/3645603404 shutdown_connections 2024-03-28T23:35:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.246+0000 7f72d4c9d640 1 -- 172.21.15.161:0/3645603404 wait complete. 2024-03-28T23:35:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.247+0000 7f72d4c9d640 1 Processor -- start 2024-03-28T23:35:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.247+0000 7f72d4c9d640 1 -- start start 2024-03-28T23:35:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.248+0000 7f72d4c9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72d00c5950 0x7f72d0148f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.248+0000 7f72d4c9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72d00c8ed0 0x7f72d0154b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.248+0000 7f72d4c9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72d0152890 0x7f72d01550d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.248+0000 7f72d4c9d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f72d0152c70 con 0x7f72d00c5950 2024-03-28T23:35:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.248+0000 7f72d4c9d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f72d01592d0 con 0x7f72d0152890 2024-03-28T23:35:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.248+0000 7f72d4c9d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f72d0152df0 con 0x7f72d00c8ed0 2024-03-28T23:35:46.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.248+0000 7f72ce59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72d00c5950 0x7f72d0148f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:46.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.248+0000 7f72ce59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72d00c5950 0x7f72d0148f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58864/0 (socket says 172.21.15.161:58864) 2024-03-28T23:35:46.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.248+0000 7f72ce59c640 1 -- 172.21.15.161:0/835458445 learned_addr learned my addr 172.21.15.161:0/835458445 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:46.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.248+0000 7f72c7fff640 1 --2- 172.21.15.161:0/835458445 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72d00c8ed0 0x7f72d0154b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:46.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.248+0000 7f72ced9d640 1 --2- 172.21.15.161:0/835458445 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72d0152890 0x7f72d01550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:46.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.248+0000 7f72ce59c640 1 -- 172.21.15.161:0/835458445 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72d00c8ed0 msgr2=0x7f72d0154b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:46.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.248+0000 7f72ce59c640 1 --2- 172.21.15.161:0/835458445 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72d00c8ed0 0x7f72d0154b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.248+0000 7f72ce59c640 1 -- 172.21.15.161:0/835458445 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72d0152890 msgr2=0x7f72d01550d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:46.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.248+0000 7f72ce59c640 1 --2- 172.21.15.161:0/835458445 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72d0152890 0x7f72d01550d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.248+0000 7f72ce59c640 1 -- 172.21.15.161:0/835458445 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d014eea0 con 0x7f72d00c5950 2024-03-28T23:35:46.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.249+0000 7f72ce59c640 1 --2- 172.21.15.161:0/835458445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72d00c5950 0x7f72d0148f40 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f72c002b820 tx=0x7f72c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:46.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.249+0000 7f72c6ffd640 1 -- 172.21.15.161:0/835458445 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c0042070 con 0x7f72d00c5950 2024-03-28T23:35:46.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.249+0000 7f72c6ffd640 1 -- 172.21.15.161:0/835458445 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72c003d890 con 0x7f72d00c5950 2024-03-28T23:35:46.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.249+0000 7f72c6ffd640 1 -- 172.21.15.161:0/835458445 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c0038040 con 0x7f72d00c5950 2024-03-28T23:35:46.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.249+0000 7f72d4c9d640 1 -- 172.21.15.161:0/835458445 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72d014f180 con 0x7f72d00c5950 2024-03-28T23:35:46.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.249+0000 7f72d4c9d640 1 -- 172.21.15.161:0/835458445 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72d014f4c0 con 0x7f72d00c5950 2024-03-28T23:35:46.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.250+0000 7f72d4c9d640 1 -- 172.21.15.161:0/835458445 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72d004f4b0 con 0x7f72d00c5950 2024-03-28T23:35:46.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.250+0000 7f72c6ffd640 1 -- 172.21.15.161:0/835458445 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f72c0045070 con 0x7f72d00c5950 2024-03-28T23:35:46.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.251+0000 7f72c6ffd640 1 --2- 172.21.15.161:0/835458445 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f72b006b500 0x7f72b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:46.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.251+0000 7f72c7fff640 1 --2- 172.21.15.161:0/835458445 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f72b006b500 0x7f72b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:46.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.251+0000 7f72c6ffd640 1 -- 172.21.15.161:0/835458445 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 6331+0+0 (secure 0 0 0) 0x7f72c00abe00 con 0x7f72d00c5950 2024-03-28T23:35:46.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.251+0000 7f72c7fff640 1 --2- 172.21.15.161:0/835458445 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f72b006b500 0x7f72b006d9c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f72d00c1a70 tx=0x7f72b8009110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:46.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.255+0000 7f72c6ffd640 1 -- 172.21.15.161:0/835458445 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f72c004c030 con 0x7f72d00c5950 2024-03-28T23:35:46.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.427+0000 7f72d4c9d640 1 -- 172.21.15.161:0/835458445 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f72d00c4ec0 con 0x7f72d00c5950 2024-03-28T23:35:46.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.428+0000 7f72c6ffd640 1 -- 172.21.15.161:0/835458445 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) ==== 74+0+17349 (secure 0 0 0) 0x7f72c007a450 con 0x7f72d00c5950 2024-03-28T23:35:46.429 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:46.429 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":140,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:35:38.963165+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:35:11.208674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-28T23:35:11.954908+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"136","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"22.21","mappings":[{"from":3,"to":0}]},{"pgid":"22.37","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6835/3586875977":"2024-03-29T23:35:38.963130+0000","172.21.15.170:6833/3586875977":"2024-03-29T23:35:38.963130+0000","172.21.15.161:6836/2527502405":"2024-03-29T23:35:21.284385+0000","172.21.15.161:6834/2527502405":"2024-03-29T23:35:21.284385+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:35:46.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.430+0000 7f72d4c9d640 1 -- 172.21.15.161:0/835458445 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f72b006b500 msgr2=0x7f72b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:46.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.430+0000 7f72d4c9d640 1 --2- 172.21.15.161:0/835458445 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f72b006b500 0x7f72b006d9c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f72d00c1a70 tx=0x7f72b8009110 comp rx=0 tx=0).stop 2024-03-28T23:35:46.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.430+0000 7f72d4c9d640 1 -- 172.21.15.161:0/835458445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72d00c5950 msgr2=0x7f72d0148f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:46.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.430+0000 7f72d4c9d640 1 --2- 172.21.15.161:0/835458445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72d00c5950 0x7f72d0148f40 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f72c002b820 tx=0x7f72c00046c0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.430+0000 7f72d4c9d640 1 -- 172.21.15.161:0/835458445 shutdown_connections 2024-03-28T23:35:46.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.430+0000 7f72d4c9d640 1 --2- 172.21.15.161:0/835458445 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f72b006b500 0x7f72b006d9c0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.430+0000 7f72d4c9d640 1 --2- 172.21.15.161:0/835458445 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72d0152890 0x7f72d01550d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.430+0000 7f72d4c9d640 1 --2- 172.21.15.161:0/835458445 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72d00c8ed0 0x7f72d0154b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.430+0000 7f72d4c9d640 1 --2- 172.21.15.161:0/835458445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72d00c5950 0x7f72d0148f40 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.430+0000 7f72d4c9d640 1 -- 172.21.15.161:0/835458445 >> 172.21.15.161:0/835458445 conn(0x7f72d00ac620 msgr2=0x7f72d00c8660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:46.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.430+0000 7f72d4c9d640 1 -- 172.21.15.161:0/835458445 shutdown_connections 2024-03-28T23:35:46.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.430+0000 7f72d4c9d640 1 -- 172.21.15.161:0/835458445 wait complete. 2024-03-28T23:35:46.442 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-28T23:35:46.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.543+0000 7f8e3e80f640 1 Processor -- start 2024-03-28T23:35:46.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.543+0000 7f8e3e80f640 1 -- start start 2024-03-28T23:35:46.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.544+0000 7f8e3e80f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8e3814eca0 0x7f8e380c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:46.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.544+0000 7f8e3e80f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8e380a98c0 0x7f8e380a9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:46.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.544+0000 7f8e3e80f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e38151850 con 0x7f8e380c1dc0 2024-03-28T23:35:46.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.544+0000 7f8e3e80f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8e38152460 con 0x7f8e3814eca0 2024-03-28T23:35:46.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.544+0000 7f8e3e80f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f8e38151b30 con 0x7f8e380a98c0 2024-03-28T23:35:46.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.544+0000 7f8e37fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8e3814eca0 0x7f8e380c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:46.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.544+0000 7f8e37fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8e3814eca0 0x7f8e380c1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58870/0 (socket says 172.21.15.161:58870) 2024-03-28T23:35:46.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.544+0000 7f8e37fff640 1 -- 172.21.15.161:0/1798316064 learned_addr learned my addr 172.21.15.161:0/1798316064 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:46.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.544+0000 7f8e3cdac640 1 --2- 172.21.15.161:0/1798316064 >> v2:172.21.15.170:3300/0 conn(0x7f8e380a98c0 0x7f8e380a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:46.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.544+0000 7f8e37fff640 1 -- 172.21.15.161:0/1798316064 >> v2:172.21.15.170:3300/0 conn(0x7f8e380a98c0 msgr2=0x7f8e380a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:46.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.544+0000 7f8e37fff640 1 --2- 172.21.15.161:0/1798316064 >> v2:172.21.15.170:3300/0 conn(0x7f8e380a98c0 0x7f8e380a9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.544+0000 7f8e37fff640 1 -- 172.21.15.161:0/1798316064 >> v1:172.21.15.170:6789/0 conn(0x7f8e380c1dc0 legacy=0x7f8e3806ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:46.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.544+0000 7f8e37fff640 1 -- 172.21.15.161:0/1798316064 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e38151e10 con 0x7f8e3814eca0 2024-03-28T23:35:46.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.545+0000 7f8e37fff640 1 --2- 172.21.15.161:0/1798316064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8e3814eca0 0x7f8e380c1880 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f8e380c2f90 tx=0x7f8e2802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57616ed71de36f60 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:46.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.545+0000 7f8e36ffd640 1 -- 172.21.15.161:0/1798316064 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2803b070 con 0x7f8e3814eca0 2024-03-28T23:35:46.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.545+0000 7f8e36ffd640 1 -- 172.21.15.161:0/1798316064 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e28036440 con 0x7f8e3814eca0 2024-03-28T23:35:46.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.545+0000 7f8e36ffd640 1 -- 172.21.15.161:0/1798316064 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e28036810 con 0x7f8e3814eca0 2024-03-28T23:35:46.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.546+0000 7f8e3e80f640 1 -- 172.21.15.161:0/1798316064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8e3814eca0 msgr2=0x7f8e380c1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:46.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.546+0000 7f8e3e80f640 1 --2- 172.21.15.161:0/1798316064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8e3814eca0 0x7f8e380c1880 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f8e380c2f90 tx=0x7f8e2802e0e0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.546+0000 7f8e3e80f640 1 -- 172.21.15.161:0/1798316064 shutdown_connections 2024-03-28T23:35:46.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.546+0000 7f8e3e80f640 1 --2- 172.21.15.161:0/1798316064 >> v2:172.21.15.170:3300/0 conn(0x7f8e380a98c0 0x7f8e380a9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.546+0000 7f8e3e80f640 1 --2- 172.21.15.161:0/1798316064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8e3814eca0 0x7f8e380c1880 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.546+0000 7f8e3e80f640 1 -- 172.21.15.161:0/1798316064 >> 172.21.15.161:0/1798316064 conn(0x7f8e380c6670 msgr2=0x7f8e380c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:46.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.546+0000 7f8e3e80f640 1 -- 172.21.15.161:0/1798316064 shutdown_connections 2024-03-28T23:35:46.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.546+0000 7f8e3e80f640 1 -- 172.21.15.161:0/1798316064 wait complete. 2024-03-28T23:35:46.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.547+0000 7f8e3e80f640 1 Processor -- start 2024-03-28T23:35:46.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.547+0000 7f8e3e80f640 1 -- start start 2024-03-28T23:35:46.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.548+0000 7f8e3e80f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8e380a98c0 0x7f8e38152eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:46.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.548+0000 7f8e3e80f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8e380c1dc0 0x7f8e380c5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:46.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.548+0000 7f8e3e80f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e3814eca0 0x7f8e3814a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:46.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.548+0000 7f8e3e80f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8e381514b0 con 0x7f8e380c1dc0 2024-03-28T23:35:46.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.548+0000 7f8e3e80f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3814f080 con 0x7f8e3814eca0 2024-03-28T23:35:46.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.548+0000 7f8e3e80f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8e38151630 con 0x7f8e380a98c0 2024-03-28T23:35:46.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.548+0000 7f8e377fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8e380c1dc0 0x7f8e380c5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:46.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.548+0000 7f8e377fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8e380c1dc0 0x7f8e380c5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58886/0 (socket says 172.21.15.161:58886) 2024-03-28T23:35:46.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.548+0000 7f8e377fe640 1 -- 172.21.15.161:0/2939367134 learned_addr learned my addr 172.21.15.161:0/2939367134 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:46.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.548+0000 7f8e37fff640 1 --2- 172.21.15.161:0/2939367134 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8e380a98c0 0x7f8e38152eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:46.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.548+0000 7f8e3cdac640 1 --2- 172.21.15.161:0/2939367134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e3814eca0 0x7f8e3814a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:46.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.548+0000 7f8e377fe640 1 -- 172.21.15.161:0/2939367134 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8e380a98c0 msgr2=0x7f8e38152eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:46.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.548+0000 7f8e377fe640 1 --2- 172.21.15.161:0/2939367134 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8e380a98c0 0x7f8e38152eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.548+0000 7f8e377fe640 1 -- 172.21.15.161:0/2939367134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e3814eca0 msgr2=0x7f8e3814a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:46.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.548+0000 7f8e377fe640 1 --2- 172.21.15.161:0/2939367134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e3814eca0 0x7f8e3814a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.548+0000 7f8e377fe640 1 -- 172.21.15.161:0/2939367134 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e38072d30 con 0x7f8e380c1dc0 2024-03-28T23:35:46.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.548+0000 7f8e3cdac640 1 --2- 172.21.15.161:0/2939367134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e3814eca0 0x7f8e3814a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:46.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.548+0000 7f8e377fe640 1 --2- 172.21.15.161:0/2939367134 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8e380c1dc0 0x7f8e380c5320 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c00c400 tx=0x7f8e2c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:46.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.548+0000 7f8e34ff9640 1 -- 172.21.15.161:0/2939367134 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c0052d0 con 0x7f8e380c1dc0 2024-03-28T23:35:46.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.548+0000 7f8e34ff9640 1 -- 172.21.15.161:0/2939367134 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e2c005490 con 0x7f8e380c1dc0 2024-03-28T23:35:46.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.548+0000 7f8e3e80f640 1 -- 172.21.15.161:0/2939367134 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e3814df20 con 0x7f8e380c1dc0 2024-03-28T23:35:46.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.549+0000 7f8e34ff9640 1 -- 172.21.15.161:0/2939367134 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c007260 con 0x7f8e380c1dc0 2024-03-28T23:35:46.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.549+0000 7f8e3e80f640 1 -- 172.21.15.161:0/2939367134 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e38149ec0 con 0x7f8e380c1dc0 2024-03-28T23:35:46.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.549+0000 7f8e3e80f640 1 -- 172.21.15.161:0/2939367134 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e3804f4b0 con 0x7f8e380c1dc0 2024-03-28T23:35:46.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.550+0000 7f8e34ff9640 1 -- 172.21.15.161:0/2939367134 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8e2c007400 con 0x7f8e380c1dc0 2024-03-28T23:35:46.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.550+0000 7f8e34ff9640 1 --2- 172.21.15.161:0/2939367134 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8e2406b4d0 0x7f8e2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:46.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.550+0000 7f8e34ff9640 1 -- 172.21.15.161:0/2939367134 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 6331+0+0 (secure 0 0 0) 0x7f8e2c089880 con 0x7f8e380c1dc0 2024-03-28T23:35:46.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.550+0000 7f8e37fff640 1 --2- 172.21.15.161:0/2939367134 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8e2406b4d0 0x7f8e2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:46.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.551+0000 7f8e37fff640 1 --2- 172.21.15.161:0/2939367134 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8e2406b4d0 0x7f8e2406d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f8e28038000 tx=0x7f8e28032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:46.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.555+0000 7f8e34ff9640 1 -- 172.21.15.161:0/2939367134 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8e2c057ed0 con 0x7f8e380c1dc0 2024-03-28T23:35:46.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.762+0000 7f8e3e80f640 1 -- 172.21.15.161:0/2939367134 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f8e380c2d50 con 0x7f8e380c1dc0 2024-03-28T23:35:46.763 INFO:tasks.ceph.mon.a.smithi161.stderr:2024-03-28T23:35:46.763+0000 7fdc7b85b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T23:35:46.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.764+0000 7f8e34ff9640 1 -- 172.21.15.161:0/2939367134 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v135) ==== 161+0+0 (secure 0 0 0) 0x7f8e2c05bf70 con 0x7f8e380c1dc0 2024-03-28T23:35:46.764 INFO:teuthology.orchestra.run.smithi161.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-28T23:35:46.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.766+0000 7f8e3e80f640 1 -- 172.21.15.161:0/2939367134 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8e2406b4d0 msgr2=0x7f8e2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:46.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.766+0000 7f8e3e80f640 1 --2- 172.21.15.161:0/2939367134 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8e2406b4d0 0x7f8e2406d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f8e28038000 tx=0x7f8e28032000 comp rx=0 tx=0).stop 2024-03-28T23:35:46.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.766+0000 7f8e3e80f640 1 -- 172.21.15.161:0/2939367134 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8e380c1dc0 msgr2=0x7f8e380c5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:46.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.766+0000 7f8e3e80f640 1 --2- 172.21.15.161:0/2939367134 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8e380c1dc0 0x7f8e380c5320 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c00c400 tx=0x7f8e2c00c8c0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.766+0000 7f8e3e80f640 1 -- 172.21.15.161:0/2939367134 shutdown_connections 2024-03-28T23:35:46.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.766+0000 7f8e3e80f640 1 --2- 172.21.15.161:0/2939367134 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8e2406b4d0 0x7f8e2406d990 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.766+0000 7f8e3e80f640 1 --2- 172.21.15.161:0/2939367134 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8e380c1dc0 0x7f8e380c5320 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.766+0000 7f8e3e80f640 1 --2- 172.21.15.161:0/2939367134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e3814eca0 0x7f8e3814a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.766+0000 7f8e3e80f640 1 --2- 172.21.15.161:0/2939367134 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8e380a98c0 0x7f8e38152eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.766+0000 7f8e3e80f640 1 -- 172.21.15.161:0/2939367134 >> 172.21.15.161:0/2939367134 conn(0x7f8e380c6670 msgr2=0x7f8e380c1570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:46.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.766+0000 7f8e3e80f640 1 -- 172.21.15.161:0/2939367134 shutdown_connections 2024-03-28T23:35:46.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.766+0000 7f8e3e80f640 1 -- 172.21.15.161:0/2939367134 wait complete. 2024-03-28T23:35:46.768 INFO:tasks.ceph.mds.d.smithi170.stderr: -92> 2024-03-28T23:35:43.904+0000 7fd0ac614640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:35:46.778 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-28T23:35:46.791 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2024-03-28T23:35:46.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.880+0000 7f5303c14640 1 Processor -- start 2024-03-28T23:35:46.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.880+0000 7f5303c14640 1 -- start start 2024-03-28T23:35:46.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.881+0000 7f5303c14640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52fc0c0330 0x7f52fc0c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:46.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.881+0000 7f5303c14640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52fc0c6f60 con 0x7f52fc152bc0 2024-03-28T23:35:46.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.881+0000 7f5303c14640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52fc0c6120 con 0x7f52fc1589b0 2024-03-28T23:35:46.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.881+0000 7f5303c14640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f52fc0c6400 con 0x7f52fc0c0330 2024-03-28T23:35:46.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.881+0000 7f53019b0640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f52fc152bc0 0x7f52fc0c8a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38176/0 (socket says 172.21.15.161:38176) 2024-03-28T23:35:46.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.881+0000 7f53019b0640 1 -- 172.21.15.161:0/2423861076 learned_addr learned my addr 172.21.15.161:0/2423861076 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:46.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53011af640 1 --2- 172.21.15.161:0/2423861076 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52fc0c0330 0x7f52fc0c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:46.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53009ae640 1 -- 172.21.15.161:0/2423861076 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 114767514 0 0) 0x7f52fc0c6f60 con 0x7f52fc152bc0 2024-03-28T23:35:46.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53009ae640 1 -- 172.21.15.161:0/2423861076 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52e8003520 con 0x7f52fc152bc0 2024-03-28T23:35:46.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53009ae640 1 -- 172.21.15.161:0/2423861076 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1982482844 0 0) 0x7f52fc0c6120 con 0x7f52fc1589b0 2024-03-28T23:35:46.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53009ae640 1 -- 172.21.15.161:0/2423861076 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52fc0c6f60 con 0x7f52fc1589b0 2024-03-28T23:35:46.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53009ae640 1 -- 172.21.15.161:0/2423861076 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3080688797 0 0) 0x7f52e8003520 con 0x7f52fc152bc0 2024-03-28T23:35:46.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53009ae640 1 -- 172.21.15.161:0/2423861076 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52fc0c0330 msgr2=0x7f52fc0c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:46.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53009ae640 1 --2- 172.21.15.161:0/2423861076 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52fc0c0330 0x7f52fc0c9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53009ae640 1 -- 172.21.15.161:0/2423861076 >> v1:172.21.15.170:6789/0 conn(0x7f52fc1589b0 legacy=0x7f52fc158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:46.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53009ae640 1 -- 172.21.15.161:0/2423861076 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52fc0c96d0 con 0x7f52fc152bc0 2024-03-28T23:35:46.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53009ae640 1 -- 172.21.15.161:0/2423861076 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f52f80019f0 con 0x7f52fc152bc0 2024-03-28T23:35:46.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53011af640 1 --2- 172.21.15.161:0/2423861076 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52fc0c0330 0x7f52fc0c9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:46.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53009ae640 1 -- 172.21.15.161:0/2423861076 >> v1:172.21.15.161:6789/0 conn(0x7f52fc152bc0 legacy=0x7f52fc0c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:46.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53009ae640 1 --2- 172.21.15.161:0/2423861076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52e8003c50 0x7f52e8004050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:46.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53009ae640 1 --2- 172.21.15.161:0/2423861076 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52fc0c0330 0x7f52e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:46.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53009ae640 1 --2- 172.21.15.161:0/2423861076 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52e8004d40 0x7f52e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:46.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53009ae640 1 -- 172.21.15.161:0/2423861076 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f52fc0c6400 con 0x7f52e8003c50 2024-03-28T23:35:46.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53009ae640 1 -- 172.21.15.161:0/2423861076 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f52e800bad0 con 0x7f52e8004d40 2024-03-28T23:35:46.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53009ae640 1 -- 172.21.15.161:0/2423861076 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f52e800bcd0 con 0x7f52fc0c0330 2024-03-28T23:35:46.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53019b0640 1 --2- 172.21.15.161:0/2423861076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52e8003c50 0x7f52e8004050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:46.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53011af640 1 --2- 172.21.15.161:0/2423861076 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52fc0c0330 0x7f52e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:46.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53021b1640 1 --2- 172.21.15.161:0/2423861076 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52e8004d40 0x7f52e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:46.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53019b0640 1 -- 172.21.15.161:0/2423861076 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52fc0c0330 msgr2=0x7f52e8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:46.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53019b0640 1 --2- 172.21.15.161:0/2423861076 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52fc0c0330 0x7f52e8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53019b0640 1 -- 172.21.15.161:0/2423861076 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52e8004d40 msgr2=0x7f52e80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:46.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53019b0640 1 --2- 172.21.15.161:0/2423861076 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52e8004d40 0x7f52e80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.882+0000 7f53019b0640 1 -- 172.21.15.161:0/2423861076 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e8003520 con 0x7f52e8003c50 2024-03-28T23:35:46.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.883+0000 7f53019b0640 1 --2- 172.21.15.161:0/2423861076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52e8003c50 0x7f52e8004050 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f52f800a820 tx=0x7f52e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b78a3d7ecb708d2b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:46.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.883+0000 7f53009ae640 1 -- 172.21.15.161:0/2423861076 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f800a450 con 0x7f52e8003c50 2024-03-28T23:35:46.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.883+0000 7f53009ae640 1 -- 172.21.15.161:0/2423861076 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52f80072d0 con 0x7f52e8003c50 2024-03-28T23:35:46.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.883+0000 7f53009ae640 1 -- 172.21.15.161:0/2423861076 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f8007700 con 0x7f52e8003c50 2024-03-28T23:35:46.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.884+0000 7f5303c14640 1 -- 172.21.15.161:0/2423861076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52e8003c50 msgr2=0x7f52e8004050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:46.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.884+0000 7f5303c14640 1 --2- 172.21.15.161:0/2423861076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52e8003c50 0x7f52e8004050 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f52f800a820 tx=0x7f52e8005830 comp rx=0 tx=0).stop 2024-03-28T23:35:46.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.884+0000 7f53021b1640 1 -- 172.21.15.161:0/2423861076 reap_dead start 2024-03-28T23:35:46.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.884+0000 7f5303c14640 1 -- 172.21.15.161:0/2423861076 shutdown_connections 2024-03-28T23:35:46.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.884+0000 7f5303c14640 1 -- 172.21.15.161:0/2423861076 >> 172.21.15.161:0/2423861076 conn(0x7f52fc0caa80 msgr2=0x7f52fc0cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:46.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.884+0000 7f5303c14640 1 -- 172.21.15.161:0/2423861076 shutdown_connections 2024-03-28T23:35:46.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.884+0000 7f5303c14640 1 -- 172.21.15.161:0/2423861076 wait complete. 2024-03-28T23:35:46.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.885+0000 7f5303c14640 1 Processor -- start 2024-03-28T23:35:46.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.885+0000 7f5303c14640 1 -- start start 2024-03-28T23:35:46.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.885+0000 7f5303c14640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52e8003c50 0x7f52fc1dfef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:46.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.885+0000 7f5303c14640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52fc1503e0 0x7f52fc156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:46.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.885+0000 7f5303c14640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52fc1deb60 0x7f52fc149e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:46.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.886+0000 7f5303c14640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f52fc152fa0 con 0x7f52e8003c50 2024-03-28T23:35:46.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.886+0000 7f5303c14640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f52fc0c6bf0 con 0x7f52fc1deb60 2024-03-28T23:35:46.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.886+0000 7f5303c14640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f52fc1e1560 con 0x7f52fc1503e0 2024-03-28T23:35:46.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.886+0000 7f53019b0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52e8003c50 0x7f52fc1dfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:46.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.886+0000 7f53019b0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52e8003c50 0x7f52fc1dfef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58916/0 (socket says 172.21.15.161:58916) 2024-03-28T23:35:46.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.886+0000 7f53019b0640 1 -- 172.21.15.161:0/2358549160 learned_addr learned my addr 172.21.15.161:0/2358549160 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:46.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.886+0000 7f53021b1640 1 --2- 172.21.15.161:0/2358549160 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52fc1deb60 0x7f52fc149e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:46.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.886+0000 7f53011af640 1 --2- 172.21.15.161:0/2358549160 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52fc1503e0 0x7f52fc156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:46.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.886+0000 7f53019b0640 1 -- 172.21.15.161:0/2358549160 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52fc1503e0 msgr2=0x7f52fc156d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:46.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.886+0000 7f53019b0640 1 --2- 172.21.15.161:0/2358549160 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52fc1503e0 0x7f52fc156d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.886+0000 7f53019b0640 1 -- 172.21.15.161:0/2358549160 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52fc1deb60 msgr2=0x7f52fc149e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:46.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.886+0000 7f53019b0640 1 --2- 172.21.15.161:0/2358549160 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52fc1deb60 0x7f52fc149e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:46.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.886+0000 7f53019b0640 1 -- 172.21.15.161:0/2358549160 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52fc1e12c0 con 0x7f52e8003c50 2024-03-28T23:35:46.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.886+0000 7f53019b0640 1 --2- 172.21.15.161:0/2358549160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52e8003c50 0x7f52fc1dfef0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f52f8000f80 tx=0x7f52f8001d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:46.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.886+0000 7f52f27fc640 1 -- 172.21.15.161:0/2358549160 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f8001f90 con 0x7f52e8003c50 2024-03-28T23:35:46.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.886+0000 7f52f27fc640 1 -- 172.21.15.161:0/2358549160 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52f803a040 con 0x7f52e8003c50 2024-03-28T23:35:46.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.886+0000 7f52f27fc640 1 -- 172.21.15.161:0/2358549160 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f80078b0 con 0x7f52e8003c50 2024-03-28T23:35:46.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.886+0000 7f5303c14640 1 -- 172.21.15.161:0/2358549160 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52fc14cdb0 con 0x7f52e8003c50 2024-03-28T23:35:46.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.886+0000 7f5303c14640 1 -- 172.21.15.161:0/2358549160 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52fc1de6b0 con 0x7f52e8003c50 2024-03-28T23:35:46.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.887+0000 7f5303c14640 1 -- 172.21.15.161:0/2358549160 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52fc04f4b0 con 0x7f52e8003c50 2024-03-28T23:35:46.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.888+0000 7f52f27fc640 1 -- 172.21.15.161:0/2358549160 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f52f8048450 con 0x7f52e8003c50 2024-03-28T23:35:46.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.888+0000 7f52f27fc640 1 --2- 172.21.15.161:0/2358549160 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f52e006b500 0x7f52e006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:46.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.888+0000 7f52f27fc640 1 -- 172.21.15.161:0/2358549160 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 6417+0+0 (secure 0 0 0) 0x7f52f80ac690 con 0x7f52e8003c50 2024-03-28T23:35:46.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.888+0000 7f53011af640 1 --2- 172.21.15.161:0/2358549160 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f52e006b500 0x7f52e006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:46.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.889+0000 7f53011af640 1 --2- 172.21.15.161:0/2358549160 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f52e006b500 0x7f52e006d9c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f52fc0ac4e0 tx=0x7f52ec009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:46.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:46.893+0000 7f52f27fc640 1 -- 172.21.15.161:0/2358549160 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f52f804d020 con 0x7f52e8003c50 2024-03-28T23:35:47.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.100+0000 7f5303c14640 1 -- 172.21.15.161:0/2358549160 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f52fc158470 con 0x7f52e8003c50 2024-03-28T23:35:47.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.764+0000 7f52f27fc640 1 -- 172.21.15.161:0/2358549160 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v136) ==== 104+0+0 (secure 0 0 0) 0x7f52f807ac80 con 0x7f52e8003c50 2024-03-28T23:35:47.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.766+0000 7f5303c14640 1 -- 172.21.15.161:0/2358549160 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f52e006b500 msgr2=0x7f52e006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:47.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.767+0000 7f5303c14640 1 --2- 172.21.15.161:0/2358549160 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f52e006b500 0x7f52e006d9c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f52fc0ac4e0 tx=0x7f52ec009190 comp rx=0 tx=0).stop 2024-03-28T23:35:47.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.767+0000 7f5303c14640 1 -- 172.21.15.161:0/2358549160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52e8003c50 msgr2=0x7f52fc1dfef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:47.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.767+0000 7f5303c14640 1 --2- 172.21.15.161:0/2358549160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52e8003c50 0x7f52fc1dfef0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f52f8000f80 tx=0x7f52f8001d40 comp rx=0 tx=0).stop 2024-03-28T23:35:47.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.767+0000 7f5303c14640 1 -- 172.21.15.161:0/2358549160 shutdown_connections 2024-03-28T23:35:47.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.767+0000 7f5303c14640 1 --2- 172.21.15.161:0/2358549160 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f52e006b500 0x7f52e006d9c0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:47.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.767+0000 7f5303c14640 1 --2- 172.21.15.161:0/2358549160 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52fc1deb60 0x7f52fc149e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:47.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.767+0000 7f5303c14640 1 --2- 172.21.15.161:0/2358549160 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52fc1503e0 0x7f52fc156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:47.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.767+0000 7f5303c14640 1 --2- 172.21.15.161:0/2358549160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52e8003c50 0x7f52fc1dfef0 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:47.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.767+0000 7f5303c14640 1 -- 172.21.15.161:0/2358549160 >> 172.21.15.161:0/2358549160 conn(0x7f52fc0caa80 msgr2=0x7f52fc0c5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:47.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.767+0000 7f5303c14640 1 -- 172.21.15.161:0/2358549160 shutdown_connections 2024-03-28T23:35:47.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.767+0000 7f5303c14640 1 -- 172.21.15.161:0/2358549160 wait complete. 2024-03-28T23:35:47.780 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-28T23:35:47.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.887+0000 7f24c838a640 1 Processor -- start 2024-03-28T23:35:47.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.887+0000 7f24c838a640 1 -- start start 2024-03-28T23:35:47.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c838a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24c0155430 0x7f24c00c3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:47.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c838a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24c0152010 con 0x7f24c00c3920 2024-03-28T23:35:47.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c838a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24c01522f0 con 0x7f24c00c3080 2024-03-28T23:35:47.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c838a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f24c01525d0 con 0x7f24c0155430 2024-03-28T23:35:47.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c6126640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f24c00c3920 0x7f24c006cb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38188/0 (socket says 172.21.15.161:38188) 2024-03-28T23:35:47.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c6126640 1 -- 172.21.15.161:0/104367864 learned_addr learned my addr 172.21.15.161:0/104367864 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:47.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c6927640 1 --2- 172.21.15.161:0/104367864 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24c0155430 0x7f24c00c3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:47.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c5124640 1 -- 172.21.15.161:0/104367864 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3476400827 0 0) 0x7f24c0152010 con 0x7f24c00c3920 2024-03-28T23:35:47.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c5124640 1 -- 172.21.15.161:0/104367864 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24ac003520 con 0x7f24c00c3920 2024-03-28T23:35:47.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c5124640 1 -- 172.21.15.161:0/104367864 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1549169474 0 0) 0x7f24c01522f0 con 0x7f24c00c3080 2024-03-28T23:35:47.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c5124640 1 -- 172.21.15.161:0/104367864 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24c0152010 con 0x7f24c00c3080 2024-03-28T23:35:47.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c5124640 1 -- 172.21.15.161:0/104367864 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2082926002 0 0) 0x7f24ac003520 con 0x7f24c00c3920 2024-03-28T23:35:47.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c5124640 1 -- 172.21.15.161:0/104367864 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24c0155430 msgr2=0x7f24c00c3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:47.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c5124640 1 --2- 172.21.15.161:0/104367864 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24c0155430 0x7f24c00c3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:47.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c5124640 1 -- 172.21.15.161:0/104367864 >> v1:172.21.15.170:6789/0 conn(0x7f24c00c3080 legacy=0x7f24c006d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:47.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c5124640 1 -- 172.21.15.161:0/104367864 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24c01528b0 con 0x7f24c00c3920 2024-03-28T23:35:47.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c5124640 1 -- 172.21.15.161:0/104367864 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f24bc001690 con 0x7f24c00c3920 2024-03-28T23:35:47.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c5124640 1 -- 172.21.15.161:0/104367864 >> v1:172.21.15.161:6789/0 conn(0x7f24c00c3920 legacy=0x7f24c006cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:47.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c5124640 1 --2- 172.21.15.161:0/104367864 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24c0155430 0x7f24ac003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:47.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c5124640 1 --2- 172.21.15.161:0/104367864 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24ac0040d0 0x7f24ac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:47.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c5124640 1 --2- 172.21.15.161:0/104367864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24ac004d40 0x7f24ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:47.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c5124640 1 -- 172.21.15.161:0/104367864 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f24c01525d0 con 0x7f24ac0040d0 2024-03-28T23:35:47.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c5124640 1 -- 172.21.15.161:0/104367864 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac00bbd0 con 0x7f24ac004d40 2024-03-28T23:35:47.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c5124640 1 -- 172.21.15.161:0/104367864 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f24ac00bdd0 con 0x7f24c0155430 2024-03-28T23:35:47.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.888+0000 7f24c5925640 1 --2- 172.21.15.161:0/104367864 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24ac0040d0 0x7f24ac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:47.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.889+0000 7f24c6126640 1 --2- 172.21.15.161:0/104367864 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24c0155430 0x7f24ac003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:47.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.889+0000 7f24c6927640 1 --2- 172.21.15.161:0/104367864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24ac004d40 0x7f24ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:47.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.889+0000 7f24c5925640 1 -- 172.21.15.161:0/104367864 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24c0155430 msgr2=0x7f24ac003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:47.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.889+0000 7f24c5925640 1 --2- 172.21.15.161:0/104367864 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24c0155430 0x7f24ac003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:47.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.889+0000 7f24c5925640 1 -- 172.21.15.161:0/104367864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24ac004d40 msgr2=0x7f24ac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:47.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.889+0000 7f24c5925640 1 --2- 172.21.15.161:0/104367864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24ac004d40 0x7f24ac0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:47.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.889+0000 7f24c5925640 1 -- 172.21.15.161:0/104367864 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24ac003520 con 0x7f24ac0040d0 2024-03-28T23:35:47.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.889+0000 7f24c6927640 1 --2- 172.21.15.161:0/104367864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24ac004d40 0x7f24ac0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:47.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.889+0000 7f24c5925640 1 --2- 172.21.15.161:0/104367864 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24ac0040d0 0x7f24ac004800 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f24b00018a0 tx=0x7f24ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8e00a2b75faf415 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:47.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.890+0000 7f24c5124640 1 -- 172.21.15.161:0/104367864 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b0037070 con 0x7f24ac0040d0 2024-03-28T23:35:47.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.890+0000 7f24c5124640 1 -- 172.21.15.161:0/104367864 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24b00052d0 con 0x7f24ac0040d0 2024-03-28T23:35:47.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.890+0000 7f24c5124640 1 -- 172.21.15.161:0/104367864 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b00014f0 con 0x7f24ac0040d0 2024-03-28T23:35:47.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.890+0000 7f24c838a640 1 -- 172.21.15.161:0/104367864 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24ac0040d0 msgr2=0x7f24ac004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:47.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.890+0000 7f24c838a640 1 --2- 172.21.15.161:0/104367864 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24ac0040d0 0x7f24ac004800 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f24b00018a0 tx=0x7f24ac005830 comp rx=0 tx=0).stop 2024-03-28T23:35:47.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.890+0000 7f24c6927640 1 -- 172.21.15.161:0/104367864 reap_dead start 2024-03-28T23:35:47.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.890+0000 7f24c838a640 1 -- 172.21.15.161:0/104367864 shutdown_connections 2024-03-28T23:35:47.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.890+0000 7f24c838a640 1 -- 172.21.15.161:0/104367864 >> 172.21.15.161:0/104367864 conn(0x7f24c00c6e20 msgr2=0x7f24c00c7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:47.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.890+0000 7f24c838a640 1 -- 172.21.15.161:0/104367864 shutdown_connections 2024-03-28T23:35:47.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.890+0000 7f24c838a640 1 -- 172.21.15.161:0/104367864 wait complete. 2024-03-28T23:35:47.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.891+0000 7f24c838a640 1 Processor -- start 2024-03-28T23:35:47.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.891+0000 7f24c838a640 1 -- start start 2024-03-28T23:35:47.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.892+0000 7f24c838a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24ac0040d0 0x7f24c00bd900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:47.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.892+0000 7f24c838a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24c014f470 0x7f24c0150a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:47.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.892+0000 7f24c838a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24c0150f50 0x7f24c0145560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:47.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.892+0000 7f24c838a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f24c00c3d00 con 0x7f24c0150f50 2024-03-28T23:35:47.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.892+0000 7f24c838a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f24c00c2630 con 0x7f24ac0040d0 2024-03-28T23:35:47.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.892+0000 7f24c838a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f24c00c3e80 con 0x7f24c014f470 2024-03-28T23:35:47.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.892+0000 7f24c6927640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24c0150f50 0x7f24c0145560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:47.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.892+0000 7f24c6927640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24c0150f50 0x7f24c0145560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58942/0 (socket says 172.21.15.161:58942) 2024-03-28T23:35:47.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.892+0000 7f24c6927640 1 -- 172.21.15.161:0/1511999236 learned_addr learned my addr 172.21.15.161:0/1511999236 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:47.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.892+0000 7f24c6927640 1 -- 172.21.15.161:0/1511999236 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24c014f470 msgr2=0x7f24c0150a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:47.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.892+0000 7f24c5925640 1 --2- 172.21.15.161:0/1511999236 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24c014f470 0x7f24c0150a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:47.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.892+0000 7f24c6126640 1 --2- 172.21.15.161:0/1511999236 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24ac0040d0 0x7f24c00bd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:47.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.892+0000 7f24c6927640 1 --2- 172.21.15.161:0/1511999236 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24c014f470 0x7f24c0150a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:47.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.892+0000 7f24c6927640 1 -- 172.21.15.161:0/1511999236 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24ac0040d0 msgr2=0x7f24c00bd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:47.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.892+0000 7f24c6927640 1 --2- 172.21.15.161:0/1511999236 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24ac0040d0 0x7f24c00bd900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:47.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.892+0000 7f24c6927640 1 -- 172.21.15.161:0/1511999236 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24c015f130 con 0x7f24c0150f50 2024-03-28T23:35:47.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.893+0000 7f24c6927640 1 --2- 172.21.15.161:0/1511999236 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24c0150f50 0x7f24c0145560 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f24c006c4f0 tx=0x7f24b8010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:47.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.893+0000 7f24b6ffd640 1 -- 172.21.15.161:0/1511999236 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b8010e00 con 0x7f24c0150f50 2024-03-28T23:35:47.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.893+0000 7f24b6ffd640 1 -- 172.21.15.161:0/1511999236 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24b8003990 con 0x7f24c0150f50 2024-03-28T23:35:47.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.893+0000 7f24c838a640 1 -- 172.21.15.161:0/1511999236 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24c015f410 con 0x7f24c0150f50 2024-03-28T23:35:47.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.893+0000 7f24b6ffd640 1 -- 172.21.15.161:0/1511999236 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b8003dc0 con 0x7f24c0150f50 2024-03-28T23:35:47.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.893+0000 7f24c838a640 1 -- 172.21.15.161:0/1511999236 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24c0151be0 con 0x7f24c0150f50 2024-03-28T23:35:47.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.894+0000 7f24c838a640 1 -- 172.21.15.161:0/1511999236 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24c0068f90 con 0x7f24c0150f50 2024-03-28T23:35:47.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.894+0000 7f24b6ffd640 1 -- 172.21.15.161:0/1511999236 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f24b8005d10 con 0x7f24c0150f50 2024-03-28T23:35:47.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.895+0000 7f24b6ffd640 1 --2- 172.21.15.161:0/1511999236 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f24a406b500 0x7f24a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:47.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.895+0000 7f24b6ffd640 1 -- 172.21.15.161:0/1511999236 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 6417+0+0 (secure 0 0 0) 0x7f24b808aba0 con 0x7f24c0150f50 2024-03-28T23:35:47.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.895+0000 7f24c6126640 1 --2- 172.21.15.161:0/1511999236 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f24a406b500 0x7f24a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:47.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.895+0000 7f24c6126640 1 --2- 172.21.15.161:0/1511999236 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f24a406b500 0x7f24a406d9c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f24c0156580 tx=0x7f24bc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:47.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:47.899+0000 7f24b6ffd640 1 -- 172.21.15.161:0/1511999236 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f24b8025020 con 0x7f24c0150f50 2024-03-28T23:35:48.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:48.073+0000 7f24c838a640 1 -- 172.21.15.161:0/1511999236 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f24c00c65b0 con 0x7f24c0150f50 2024-03-28T23:35:49.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.009+0000 7f24b6ffd640 1 -- 172.21.15.161:0/1511999236 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v142) ==== 181+0+0 (secure 0 0 0) 0x7f24b8059750 con 0x7f24c0150f50 2024-03-28T23:35:49.011 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_metadata' removed 2024-03-28T23:35:49.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.012+0000 7f24c838a640 1 -- 172.21.15.161:0/1511999236 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f24a406b500 msgr2=0x7f24a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:49.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.012+0000 7f24c838a640 1 --2- 172.21.15.161:0/1511999236 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f24a406b500 0x7f24a406d9c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f24c0156580 tx=0x7f24bc009040 comp rx=0 tx=0).stop 2024-03-28T23:35:49.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.012+0000 7f24c838a640 1 -- 172.21.15.161:0/1511999236 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24c0150f50 msgr2=0x7f24c0145560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:49.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.012+0000 7f24c838a640 1 --2- 172.21.15.161:0/1511999236 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24c0150f50 0x7f24c0145560 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f24c006c4f0 tx=0x7f24b8010590 comp rx=0 tx=0).stop 2024-03-28T23:35:49.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.012+0000 7f24c838a640 1 -- 172.21.15.161:0/1511999236 shutdown_connections 2024-03-28T23:35:49.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.012+0000 7f24c838a640 1 --2- 172.21.15.161:0/1511999236 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f24a406b500 0x7f24a406d9c0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:49.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.012+0000 7f24c838a640 1 --2- 172.21.15.161:0/1511999236 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24c0150f50 0x7f24c0145560 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:49.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.012+0000 7f24c838a640 1 --2- 172.21.15.161:0/1511999236 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24c014f470 0x7f24c0150a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:49.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.012+0000 7f24c838a640 1 --2- 172.21.15.161:0/1511999236 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24ac0040d0 0x7f24c00bd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:49.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.012+0000 7f24c838a640 1 -- 172.21.15.161:0/1511999236 >> 172.21.15.161:0/1511999236 conn(0x7f24c00c6e20 msgr2=0x7f24c0072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:49.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.013+0000 7f24c838a640 1 -- 172.21.15.161:0/1511999236 shutdown_connections 2024-03-28T23:35:49.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.013+0000 7f24c838a640 1 -- 172.21.15.161:0/1511999236 wait complete. 2024-03-28T23:35:49.027 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-28T23:35:49.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.136+0000 7febeee88640 1 Processor -- start 2024-03-28T23:35:49.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.136+0000 7febeee88640 1 -- start start 2024-03-28T23:35:49.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.137+0000 7febeee88640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7febe80c30c0 0x7febe8152880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:49.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.137+0000 7febeee88640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7febe80c19b0 0x7febe80c1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:49.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.137+0000 7febeee88640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febe80c3540 con 0x7febe80ac430 2024-03-28T23:35:49.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.137+0000 7febeee88640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7febe8152dc0 con 0x7febe80c30c0 2024-03-28T23:35:49.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.137+0000 7febeee88640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7febe80c2300 con 0x7febe80c19b0 2024-03-28T23:35:49.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.137+0000 7febdffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7febe80c30c0 0x7febe8152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:49.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.137+0000 7febdffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7febe80c30c0 0x7febe8152880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58960/0 (socket says 172.21.15.161:58960) 2024-03-28T23:35:49.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.137+0000 7febdffff640 1 -- 172.21.15.161:0/4200196064 learned_addr learned my addr 172.21.15.161:0/4200196064 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:49.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.137+0000 7febed425640 1 --2- 172.21.15.161:0/4200196064 >> v2:172.21.15.170:3300/0 conn(0x7febe80c19b0 0x7febe80c1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:49.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.137+0000 7febdffff640 1 -- 172.21.15.161:0/4200196064 >> v2:172.21.15.170:3300/0 conn(0x7febe80c19b0 msgr2=0x7febe80c1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:49.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.137+0000 7febdffff640 1 --2- 172.21.15.161:0/4200196064 >> v2:172.21.15.170:3300/0 conn(0x7febe80c19b0 0x7febe80c1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:49.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.137+0000 7febdffff640 1 -- 172.21.15.161:0/4200196064 >> v1:172.21.15.170:6789/0 conn(0x7febe80ac430 legacy=0x7febe80c24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:49.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.137+0000 7febdffff640 1 -- 172.21.15.161:0/4200196064 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe80c3820 con 0x7febe80c30c0 2024-03-28T23:35:49.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.137+0000 7febed425640 1 --2- 172.21.15.161:0/4200196064 >> v2:172.21.15.170:3300/0 conn(0x7febe80c19b0 0x7febe80c1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:49.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.138+0000 7febdffff640 1 --2- 172.21.15.161:0/4200196064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febe80c30c0 0x7febe8152880 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7febe80c61b0 tx=0x7febd802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24da203bc58c6ccc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:49.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.138+0000 7febdf7fe640 1 -- 172.21.15.161:0/4200196064 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd802e070 con 0x7febe80c30c0 2024-03-28T23:35:49.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.138+0000 7febdf7fe640 1 -- 172.21.15.161:0/4200196064 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febd8037070 con 0x7febe80c30c0 2024-03-28T23:35:49.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.138+0000 7febdf7fe640 1 -- 172.21.15.161:0/4200196064 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd8031a00 con 0x7febe80c30c0 2024-03-28T23:35:49.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.138+0000 7febeee88640 1 -- 172.21.15.161:0/4200196064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febe80c30c0 msgr2=0x7febe8152880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:49.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.138+0000 7febeee88640 1 --2- 172.21.15.161:0/4200196064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febe80c30c0 0x7febe8152880 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7febe80c61b0 tx=0x7febd802b880 comp rx=0 tx=0).stop 2024-03-28T23:35:49.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.139+0000 7febeee88640 1 -- 172.21.15.161:0/4200196064 shutdown_connections 2024-03-28T23:35:49.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.139+0000 7febeee88640 1 --2- 172.21.15.161:0/4200196064 >> v2:172.21.15.170:3300/0 conn(0x7febe80c19b0 0x7febe80c1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:49.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.139+0000 7febeee88640 1 --2- 172.21.15.161:0/4200196064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febe80c30c0 0x7febe8152880 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:49.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.139+0000 7febeee88640 1 -- 172.21.15.161:0/4200196064 >> 172.21.15.161:0/4200196064 conn(0x7febe806b150 msgr2=0x7febe806b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:49.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.139+0000 7febeee88640 1 -- 172.21.15.161:0/4200196064 shutdown_connections 2024-03-28T23:35:49.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.139+0000 7febeee88640 1 -- 172.21.15.161:0/4200196064 wait complete. 2024-03-28T23:35:49.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.140+0000 7febeee88640 1 Processor -- start 2024-03-28T23:35:49.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.140+0000 7febeee88640 1 -- start start 2024-03-28T23:35:49.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.140+0000 7febeee88640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febe80ac430 0x7febe8155910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:49.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.140+0000 7febeee88640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7febe80c19b0 0x7febe814a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:49.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.140+0000 7febeee88640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7febe80c30c0 0x7febe8149ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:49.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.140+0000 7febeee88640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7febe80c1490 con 0x7febe80ac430 2024-03-28T23:35:49.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.140+0000 7febeee88640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7febe80c11b0 con 0x7febe80c19b0 2024-03-28T23:35:49.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.140+0000 7febeee88640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7febe80c6440 con 0x7febe80c30c0 2024-03-28T23:35:49.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.141+0000 7febecc24640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febe80ac430 0x7febe8155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:49.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.141+0000 7febecc24640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febe80ac430 0x7febe8155910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58972/0 (socket says 172.21.15.161:58972) 2024-03-28T23:35:49.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.141+0000 7febecc24640 1 -- 172.21.15.161:0/3468697517 learned_addr learned my addr 172.21.15.161:0/3468697517 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:49.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.141+0000 7febdffff640 1 --2- 172.21.15.161:0/3468697517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7febe80c19b0 0x7febe814a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:49.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.141+0000 7febed425640 1 --2- 172.21.15.161:0/3468697517 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7febe80c30c0 0x7febe8149ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:49.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.141+0000 7febecc24640 1 -- 172.21.15.161:0/3468697517 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7febe80c30c0 msgr2=0x7febe8149ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:49.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.141+0000 7febecc24640 1 --2- 172.21.15.161:0/3468697517 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7febe80c30c0 0x7febe8149ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:49.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.141+0000 7febecc24640 1 -- 172.21.15.161:0/3468697517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7febe80c19b0 msgr2=0x7febe814a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:49.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.141+0000 7febecc24640 1 --2- 172.21.15.161:0/3468697517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7febe80c19b0 0x7febe814a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:49.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.141+0000 7febecc24640 1 -- 172.21.15.161:0/3468697517 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe8156260 con 0x7febe80ac430 2024-03-28T23:35:49.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.141+0000 7febed425640 1 --2- 172.21.15.161:0/3468697517 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7febe80c30c0 0x7febe8149ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:49.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.141+0000 7febecc24640 1 --2- 172.21.15.161:0/3468697517 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febe80ac430 0x7febe8155910 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7febe400b3f0 tx=0x7febe400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:49.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.141+0000 7febdd7fa640 1 -- 172.21.15.161:0/3468697517 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe4015020 con 0x7febe80ac430 2024-03-28T23:35:49.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.141+0000 7febdd7fa640 1 -- 172.21.15.161:0/3468697517 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febe4011830 con 0x7febe80ac430 2024-03-28T23:35:49.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.141+0000 7febdd7fa640 1 -- 172.21.15.161:0/3468697517 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe4011c20 con 0x7febe80ac430 2024-03-28T23:35:49.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.141+0000 7febeee88640 1 -- 172.21.15.161:0/3468697517 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febe8153290 con 0x7febe80ac430 2024-03-28T23:35:49.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.141+0000 7febeee88640 1 -- 172.21.15.161:0/3468697517 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febe8153530 con 0x7febe80ac430 2024-03-28T23:35:49.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.142+0000 7febeee88640 1 -- 172.21.15.161:0/3468697517 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febe8068f90 con 0x7febe80ac430 2024-03-28T23:35:49.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.143+0000 7febdd7fa640 1 -- 172.21.15.161:0/3468697517 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7febe4018020 con 0x7febe80ac430 2024-03-28T23:35:49.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.143+0000 7febdd7fa640 1 --2- 172.21.15.161:0/3468697517 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7febd406b4d0 0x7febd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:49.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.143+0000 7febdffff640 1 --2- 172.21.15.161:0/3468697517 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7febd406b4d0 0x7febd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:49.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.143+0000 7febdd7fa640 1 -- 172.21.15.161:0/3468697517 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5979+0+0 (secure 0 0 0) 0x7febe408a120 con 0x7febe80ac430 2024-03-28T23:35:49.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.144+0000 7febdffff640 1 --2- 172.21.15.161:0/3468697517 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7febd406b4d0 0x7febd406d990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7febd8036000 tx=0x7febd803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:49.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.148+0000 7febdd7fa640 1 -- 172.21.15.161:0/3468697517 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7febe401f020 con 0x7febe80ac430 2024-03-28T23:35:49.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:49.327+0000 7febeee88640 1 -- 172.21.15.161:0/3468697517 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7febe80c6220 con 0x7febe80ac430 2024-03-28T23:35:50.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.016+0000 7febdd7fa640 1 -- 172.21.15.161:0/3468697517 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v143) ==== 169+0+0 (secure 0 0 0) 0x7febe4058e90 con 0x7febe80ac430 2024-03-28T23:35:50.018 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data' removed 2024-03-28T23:35:50.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.019+0000 7febeee88640 1 -- 172.21.15.161:0/3468697517 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7febd406b4d0 msgr2=0x7febd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:50.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.019+0000 7febeee88640 1 --2- 172.21.15.161:0/3468697517 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7febd406b4d0 0x7febd406d990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7febd8036000 tx=0x7febd803d040 comp rx=0 tx=0).stop 2024-03-28T23:35:50.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.019+0000 7febeee88640 1 -- 172.21.15.161:0/3468697517 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febe80ac430 msgr2=0x7febe8155910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:50.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.019+0000 7febeee88640 1 --2- 172.21.15.161:0/3468697517 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febe80ac430 0x7febe8155910 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7febe400b3f0 tx=0x7febe400b8b0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.020+0000 7febeee88640 1 -- 172.21.15.161:0/3468697517 shutdown_connections 2024-03-28T23:35:50.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.020+0000 7febeee88640 1 --2- 172.21.15.161:0/3468697517 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7febd406b4d0 0x7febd406d990 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.020+0000 7febeee88640 1 --2- 172.21.15.161:0/3468697517 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7febe80c30c0 0x7febe8149ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.020+0000 7febeee88640 1 --2- 172.21.15.161:0/3468697517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7febe80c19b0 0x7febe814a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.020+0000 7febeee88640 1 --2- 172.21.15.161:0/3468697517 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febe80ac430 0x7febe8155910 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.020+0000 7febeee88640 1 -- 172.21.15.161:0/3468697517 >> 172.21.15.161:0/3468697517 conn(0x7febe806b150 msgr2=0x7febe806b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:50.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.020+0000 7febeee88640 1 -- 172.21.15.161:0/3468697517 shutdown_connections 2024-03-28T23:35:50.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.020+0000 7febeee88640 1 -- 172.21.15.161:0/3468697517 wait complete. 2024-03-28T23:35:50.036 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-28T23:35:50.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.142+0000 7f64b05d0640 1 Processor -- start 2024-03-28T23:35:50.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.142+0000 7f64b05d0640 1 -- start start 2024-03-28T23:35:50.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.143+0000 7f64b05d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64a80a7d00 0x7f64a80c1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:50.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.143+0000 7f64b05d0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64a80c28e0 con 0x7f64a81506b0 2024-03-28T23:35:50.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.143+0000 7f64b05d0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64a80c2bc0 con 0x7f64a8153840 2024-03-28T23:35:50.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.143+0000 7f64b05d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f64a80c0d80 con 0x7f64a80a7d00 2024-03-28T23:35:50.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.143+0000 7f64ae36c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f64a81506b0 0x7f64a80a96f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38196/0 (socket says 172.21.15.161:38196) 2024-03-28T23:35:50.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.143+0000 7f64ae36c640 1 -- 172.21.15.161:0/1266774327 learned_addr learned my addr 172.21.15.161:0/1266774327 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:50.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.143+0000 7f64adb6b640 1 --2- 172.21.15.161:0/1266774327 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64a80a7d00 0x7f64a80c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:50.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.143+0000 7f64ad36a640 1 -- 172.21.15.161:0/1266774327 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1976690701 0 0) 0x7f64a80c28e0 con 0x7f64a81506b0 2024-03-28T23:35:50.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.143+0000 7f64ad36a640 1 -- 172.21.15.161:0/1266774327 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6494003520 con 0x7f64a81506b0 2024-03-28T23:35:50.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.143+0000 7f64ad36a640 1 -- 172.21.15.161:0/1266774327 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1785248002 0 0) 0x7f6494003520 con 0x7f64a81506b0 2024-03-28T23:35:50.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.143+0000 7f64ad36a640 1 -- 172.21.15.161:0/1266774327 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64a80a7d00 msgr2=0x7f64a80c1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:50.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.143+0000 7f64ad36a640 1 --2- 172.21.15.161:0/1266774327 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64a80a7d00 0x7f64a80c1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.143+0000 7f64ad36a640 1 -- 172.21.15.161:0/1266774327 >> v1:172.21.15.170:6789/0 conn(0x7f64a8153840 legacy=0x7f64a80ab260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:50.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.143+0000 7f64ad36a640 1 -- 172.21.15.161:0/1266774327 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64a8153040 con 0x7f64a81506b0 2024-03-28T23:35:50.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.143+0000 7f64ad36a640 1 -- 172.21.15.161:0/1266774327 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f64a4001670 con 0x7f64a81506b0 2024-03-28T23:35:50.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.143+0000 7f64ad36a640 1 -- 172.21.15.161:0/1266774327 >> v1:172.21.15.161:6789/0 conn(0x7f64a81506b0 legacy=0x7f64a80a96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:50.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.143+0000 7f64ad36a640 1 --2- 172.21.15.161:0/1266774327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6494003ac0 0x7f6494003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:50.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.143+0000 7f64ad36a640 1 --2- 172.21.15.161:0/1266774327 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64a80a7d00 0x7f6494004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:50.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.143+0000 7f64ad36a640 1 --2- 172.21.15.161:0/1266774327 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6494004d40 0x7f64940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:50.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.143+0000 7f64ad36a640 1 -- 172.21.15.161:0/1266774327 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f64a80c0d80 con 0x7f6494003ac0 2024-03-28T23:35:50.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.143+0000 7f64ad36a640 1 -- 172.21.15.161:0/1266774327 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f649400bbd0 con 0x7f6494004d40 2024-03-28T23:35:50.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.143+0000 7f64ad36a640 1 -- 172.21.15.161:0/1266774327 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f649400bdd0 con 0x7f64a80a7d00 2024-03-28T23:35:50.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.143+0000 7f64ae36c640 1 --2- 172.21.15.161:0/1266774327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6494003ac0 0x7f6494003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:50.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.144+0000 7f64adb6b640 1 --2- 172.21.15.161:0/1266774327 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64a80a7d00 0x7f6494004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:50.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.144+0000 7f64aeb6d640 1 --2- 172.21.15.161:0/1266774327 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6494004d40 0x7f64940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:50.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.144+0000 7f64ae36c640 1 -- 172.21.15.161:0/1266774327 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64a80a7d00 msgr2=0x7f6494004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:50.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.144+0000 7f64ae36c640 1 --2- 172.21.15.161:0/1266774327 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64a80a7d00 0x7f6494004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.144+0000 7f64ae36c640 1 -- 172.21.15.161:0/1266774327 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6494004d40 msgr2=0x7f64940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:50.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.144+0000 7f64ae36c640 1 --2- 172.21.15.161:0/1266774327 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6494004d40 0x7f64940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.144+0000 7f64ae36c640 1 -- 172.21.15.161:0/1266774327 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6494003520 con 0x7f6494003ac0 2024-03-28T23:35:50.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.145+0000 7f64ae36c640 1 --2- 172.21.15.161:0/1266774327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6494003ac0 0x7f6494003ee0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f64a4001eb0 tx=0x7f6494005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3ca0c9fbcdff9cd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:50.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.145+0000 7f64ad36a640 1 -- 172.21.15.161:0/1266774327 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a4007d30 con 0x7f6494003ac0 2024-03-28T23:35:50.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.145+0000 7f64ad36a640 1 -- 172.21.15.161:0/1266774327 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64a4030070 con 0x7f6494003ac0 2024-03-28T23:35:50.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.145+0000 7f64ad36a640 1 -- 172.21.15.161:0/1266774327 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a4005d30 con 0x7f6494003ac0 2024-03-28T23:35:50.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.145+0000 7f64b05d0640 1 -- 172.21.15.161:0/1266774327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6494003ac0 msgr2=0x7f6494003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:50.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.145+0000 7f64b05d0640 1 --2- 172.21.15.161:0/1266774327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6494003ac0 0x7f6494003ee0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f64a4001eb0 tx=0x7f6494005830 comp rx=0 tx=0).stop 2024-03-28T23:35:50.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.145+0000 7f64aeb6d640 1 -- 172.21.15.161:0/1266774327 reap_dead start 2024-03-28T23:35:50.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.145+0000 7f64b05d0640 1 -- 172.21.15.161:0/1266774327 shutdown_connections 2024-03-28T23:35:50.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.145+0000 7f64b05d0640 1 -- 172.21.15.161:0/1266774327 >> 172.21.15.161:0/1266774327 conn(0x7f64a80a83f0 msgr2=0x7f64a80a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:50.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.145+0000 7f64b05d0640 1 -- 172.21.15.161:0/1266774327 shutdown_connections 2024-03-28T23:35:50.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.145+0000 7f64b05d0640 1 -- 172.21.15.161:0/1266774327 wait complete. 2024-03-28T23:35:50.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.146+0000 7f64b05d0640 1 Processor -- start 2024-03-28T23:35:50.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.146+0000 7f64b05d0640 1 -- start start 2024-03-28T23:35:50.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.147+0000 7f64b05d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6494003ac0 0x7f64a814d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:50.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.147+0000 7f64b05d0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64a8156f70 0x7f64a80731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:50.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.147+0000 7f64b05d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64a81e9740 0x7f64a8151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:50.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.147+0000 7f64b05d0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f64a80ab970 con 0x7f64a8156f70 2024-03-28T23:35:50.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.147+0000 7f64b05d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f64a80c2ea0 con 0x7f64a81e9740 2024-03-28T23:35:50.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.147+0000 7f64b05d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f64a80abaf0 con 0x7f6494003ac0 2024-03-28T23:35:50.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.147+0000 7f64adb6b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64a8156f70 0x7f64a80731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:50.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.147+0000 7f64adb6b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64a8156f70 0x7f64a80731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58988/0 (socket says 172.21.15.161:58988) 2024-03-28T23:35:50.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.147+0000 7f64adb6b640 1 -- 172.21.15.161:0/2077903994 learned_addr learned my addr 172.21.15.161:0/2077903994 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:50.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.147+0000 7f64aeb6d640 1 --2- 172.21.15.161:0/2077903994 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64a81e9740 0x7f64a8151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:50.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.147+0000 7f64ae36c640 1 --2- 172.21.15.161:0/2077903994 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6494003ac0 0x7f64a814d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:50.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.147+0000 7f64adb6b640 1 -- 172.21.15.161:0/2077903994 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6494003ac0 msgr2=0x7f64a814d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:50.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.147+0000 7f64adb6b640 1 --2- 172.21.15.161:0/2077903994 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6494003ac0 0x7f64a814d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.147+0000 7f64adb6b640 1 -- 172.21.15.161:0/2077903994 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64a81e9740 msgr2=0x7f64a8151df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:50.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.147+0000 7f64adb6b640 1 --2- 172.21.15.161:0/2077903994 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64a81e9740 0x7f64a8151df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.147+0000 7f64adb6b640 1 -- 172.21.15.161:0/2077903994 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64a81ecf70 con 0x7f64a8156f70 2024-03-28T23:35:50.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.147+0000 7f64aeb6d640 1 --2- 172.21.15.161:0/2077903994 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64a81e9740 0x7f64a8151df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:50.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.148+0000 7f64adb6b640 1 --2- 172.21.15.161:0/2077903994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64a8156f70 0x7f64a80731e0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f64980119a0 tx=0x7f6498011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:50.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.148+0000 7f649effd640 1 -- 172.21.15.161:0/2077903994 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649800fa90 con 0x7f64a8156f70 2024-03-28T23:35:50.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.148+0000 7f649effd640 1 -- 172.21.15.161:0/2077903994 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f649800fc30 con 0x7f64a8156f70 2024-03-28T23:35:50.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.148+0000 7f64b05d0640 1 -- 172.21.15.161:0/2077903994 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64a81e8f40 con 0x7f64a8156f70 2024-03-28T23:35:50.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.148+0000 7f649effd640 1 -- 172.21.15.161:0/2077903994 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6498014440 con 0x7f64a8156f70 2024-03-28T23:35:50.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.148+0000 7f64b05d0640 1 -- 172.21.15.161:0/2077903994 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64a8073720 con 0x7f64a8156f70 2024-03-28T23:35:50.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.149+0000 7f64b05d0640 1 -- 172.21.15.161:0/2077903994 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64a804f4b0 con 0x7f64a8156f70 2024-03-28T23:35:50.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.149+0000 7f649effd640 1 -- 172.21.15.161:0/2077903994 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6498012040 con 0x7f64a8156f70 2024-03-28T23:35:50.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.150+0000 7f649effd640 1 --2- 172.21.15.161:0/2077903994 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f648c06b500 0x7f648c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:50.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.150+0000 7f649effd640 1 -- 172.21.15.161:0/2077903994 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5507+0+0 (secure 0 0 0) 0x7f649808a5d0 con 0x7f64a8156f70 2024-03-28T23:35:50.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.150+0000 7f64ae36c640 1 --2- 172.21.15.161:0/2077903994 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f648c06b500 0x7f648c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:50.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.150+0000 7f64ae36c640 1 --2- 172.21.15.161:0/2077903994 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f648c06b500 0x7f648c06d9c0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f64a4034000 tx=0x7f64a4009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:50.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.154+0000 7f649effd640 1 -- 172.21.15.161:0/2077903994 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6498019030 con 0x7f64a8156f70 2024-03-28T23:35:50.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.331+0000 7f64b05d0640 1 -- 172.21.15.161:0/2077903994 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f64a80aab50 con 0x7f64a8156f70 2024-03-28T23:35:50.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.335+0000 7f649effd640 1 -- 172.21.15.161:0/2077903994 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v49) ==== 104+0+0 (secure 0 0 0) 0x7f6498058ec0 con 0x7f64a8156f70 2024-03-28T23:35:50.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.337+0000 7f64b05d0640 1 -- 172.21.15.161:0/2077903994 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f648c06b500 msgr2=0x7f648c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:50.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.337+0000 7f64b05d0640 1 --2- 172.21.15.161:0/2077903994 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f648c06b500 0x7f648c06d9c0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f64a4034000 tx=0x7f64a4009030 comp rx=0 tx=0).stop 2024-03-28T23:35:50.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.337+0000 7f64b05d0640 1 -- 172.21.15.161:0/2077903994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64a8156f70 msgr2=0x7f64a80731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:50.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.337+0000 7f64b05d0640 1 --2- 172.21.15.161:0/2077903994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64a8156f70 0x7f64a80731e0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f64980119a0 tx=0x7f6498011e60 comp rx=0 tx=0).stop 2024-03-28T23:35:50.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.337+0000 7f64b05d0640 1 -- 172.21.15.161:0/2077903994 shutdown_connections 2024-03-28T23:35:50.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.337+0000 7f64b05d0640 1 --2- 172.21.15.161:0/2077903994 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f648c06b500 0x7f648c06d9c0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.337+0000 7f64b05d0640 1 --2- 172.21.15.161:0/2077903994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64a8156f70 0x7f64a80731e0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.337+0000 7f64b05d0640 1 --2- 172.21.15.161:0/2077903994 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64a81e9740 0x7f64a8151df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.337+0000 7f64b05d0640 1 --2- 172.21.15.161:0/2077903994 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6494003ac0 0x7f64a814d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.337+0000 7f64b05d0640 1 -- 172.21.15.161:0/2077903994 >> 172.21.15.161:0/2077903994 conn(0x7f64a80a83f0 msgr2=0x7f64a80a9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:50.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.337+0000 7f64b05d0640 1 -- 172.21.15.161:0/2077903994 shutdown_connections 2024-03-28T23:35:50.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.337+0000 7f64b05d0640 1 -- 172.21.15.161:0/2077903994 wait complete. 2024-03-28T23:35:50.349 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-03-28T23:35:50.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.451+0000 7fec66713640 1 Processor -- start 2024-03-28T23:35:50.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.451+0000 7fec66713640 1 -- start start 2024-03-28T23:35:50.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.451+0000 7fec66713640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fec60150520 0x7fec600ab280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:50.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.451+0000 7fec66713640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fec600a96f0 0x7fec6015d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:50.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.451+0000 7fec66713640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec600c2f50 con 0x7fec600c1040 2024-03-28T23:35:50.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.451+0000 7fec66713640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fec600a9c20 con 0x7fec60150520 2024-03-28T23:35:50.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.451+0000 7fec66713640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fec60150900 con 0x7fec600a96f0 2024-03-28T23:35:50.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.451+0000 7fec5ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fec60150520 0x7fec600ab280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:50.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.451+0000 7fec5ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fec60150520 0x7fec600ab280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58994/0 (socket says 172.21.15.161:58994) 2024-03-28T23:35:50.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.451+0000 7fec5ffff640 1 -- 172.21.15.161:0/2824236655 learned_addr learned my addr 172.21.15.161:0/2824236655 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:50.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.452+0000 7fec64cb0640 1 --2- 172.21.15.161:0/2824236655 >> v2:172.21.15.170:3300/0 conn(0x7fec600a96f0 0x7fec6015d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:50.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.452+0000 7fec5ffff640 1 -- 172.21.15.161:0/2824236655 >> v2:172.21.15.170:3300/0 conn(0x7fec600a96f0 msgr2=0x7fec6015d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:50.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.452+0000 7fec5ffff640 1 --2- 172.21.15.161:0/2824236655 >> v2:172.21.15.170:3300/0 conn(0x7fec600a96f0 0x7fec6015d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.452+0000 7fec5ffff640 1 -- 172.21.15.161:0/2824236655 >> v1:172.21.15.170:6789/0 conn(0x7fec600c1040 legacy=0x7fec601535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:50.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.452+0000 7fec5ffff640 1 -- 172.21.15.161:0/2824236655 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec601530d0 con 0x7fec60150520 2024-03-28T23:35:50.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.453+0000 7fec5ffff640 1 --2- 172.21.15.161:0/2824236655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec60150520 0x7fec600ab280 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fec600a9460 tx=0x7fec5002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fda46552716257a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:50.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.453+0000 7fec5effd640 1 -- 172.21.15.161:0/2824236655 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec50035020 con 0x7fec60150520 2024-03-28T23:35:50.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.453+0000 7fec5effd640 1 -- 172.21.15.161:0/2824236655 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec50031870 con 0x7fec60150520 2024-03-28T23:35:50.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.453+0000 7fec5effd640 1 -- 172.21.15.161:0/2824236655 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec50031c40 con 0x7fec60150520 2024-03-28T23:35:50.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.453+0000 7fec66713640 1 -- 172.21.15.161:0/2824236655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec60150520 msgr2=0x7fec600ab280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:50.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.453+0000 7fec66713640 1 --2- 172.21.15.161:0/2824236655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec60150520 0x7fec600ab280 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fec600a9460 tx=0x7fec5002b910 comp rx=0 tx=0).stop 2024-03-28T23:35:50.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.453+0000 7fec66713640 1 -- 172.21.15.161:0/2824236655 shutdown_connections 2024-03-28T23:35:50.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.453+0000 7fec66713640 1 --2- 172.21.15.161:0/2824236655 >> v2:172.21.15.170:3300/0 conn(0x7fec600a96f0 0x7fec6015d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.453+0000 7fec66713640 1 --2- 172.21.15.161:0/2824236655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec60150520 0x7fec600ab280 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.453+0000 7fec66713640 1 -- 172.21.15.161:0/2824236655 >> 172.21.15.161:0/2824236655 conn(0x7fec600a83f0 msgr2=0x7fec600a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:50.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.453+0000 7fec66713640 1 -- 172.21.15.161:0/2824236655 shutdown_connections 2024-03-28T23:35:50.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.453+0000 7fec66713640 1 -- 172.21.15.161:0/2824236655 wait complete. 2024-03-28T23:35:50.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.454+0000 7fec66713640 1 Processor -- start 2024-03-28T23:35:50.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.455+0000 7fec66713640 1 -- start start 2024-03-28T23:35:50.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.455+0000 7fec66713640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec600a96f0 0x7fec601567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:50.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.455+0000 7fec66713640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec600c1040 0x7fec601ec2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:50.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.455+0000 7fec66713640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec60150520 0x7fec601ec800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:50.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.455+0000 7fec66713640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fec600c0c00 con 0x7fec60150520 2024-03-28T23:35:50.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.455+0000 7fec66713640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fec600c2c70 con 0x7fec600a96f0 2024-03-28T23:35:50.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.455+0000 7fec66713640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fec600c0d80 con 0x7fec600c1040 2024-03-28T23:35:50.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.455+0000 7fec64cb0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec60150520 0x7fec601ec800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:50.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.455+0000 7fec64cb0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec60150520 0x7fec601ec800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59002/0 (socket says 172.21.15.161:59002) 2024-03-28T23:35:50.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.455+0000 7fec64cb0640 1 -- 172.21.15.161:0/2887591042 learned_addr learned my addr 172.21.15.161:0/2887591042 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:50.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.455+0000 7fec5f7fe640 1 --2- 172.21.15.161:0/2887591042 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec600c1040 0x7fec601ec2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:50.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.455+0000 7fec5ffff640 1 --2- 172.21.15.161:0/2887591042 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec600a96f0 0x7fec601567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:50.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.456+0000 7fec64cb0640 1 -- 172.21.15.161:0/2887591042 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec600c1040 msgr2=0x7fec601ec2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:50.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.456+0000 7fec64cb0640 1 --2- 172.21.15.161:0/2887591042 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec600c1040 0x7fec601ec2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.456+0000 7fec64cb0640 1 -- 172.21.15.161:0/2887591042 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec600a96f0 msgr2=0x7fec601567f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:50.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.456+0000 7fec64cb0640 1 --2- 172.21.15.161:0/2887591042 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec600a96f0 0x7fec601567f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.456+0000 7fec64cb0640 1 -- 172.21.15.161:0/2887591042 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec60148dc0 con 0x7fec60150520 2024-03-28T23:35:50.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.456+0000 7fec5ffff640 1 --2- 172.21.15.161:0/2887591042 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec600a96f0 0x7fec601567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:50.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.456+0000 7fec64cb0640 1 --2- 172.21.15.161:0/2887591042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec60150520 0x7fec601ec800 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fec58007d80 tx=0x7fec5800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:50.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.456+0000 7fec5cff9640 1 -- 172.21.15.161:0/2887591042 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec58014420 con 0x7fec60150520 2024-03-28T23:35:50.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.456+0000 7fec5cff9640 1 -- 172.21.15.161:0/2887591042 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec58019070 con 0x7fec60150520 2024-03-28T23:35:50.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.456+0000 7fec66713640 1 -- 172.21.15.161:0/2887591042 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec601490a0 con 0x7fec60150520 2024-03-28T23:35:50.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.456+0000 7fec5cff9640 1 -- 172.21.15.161:0/2887591042 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec58014800 con 0x7fec60150520 2024-03-28T23:35:50.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.456+0000 7fec66713640 1 -- 172.21.15.161:0/2887591042 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec60154e80 con 0x7fec60150520 2024-03-28T23:35:50.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.457+0000 7fec66713640 1 -- 172.21.15.161:0/2887591042 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec6004f4b0 con 0x7fec60150520 2024-03-28T23:35:50.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.457+0000 7fec5cff9640 1 -- 172.21.15.161:0/2887591042 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fec580149a0 con 0x7fec60150520 2024-03-28T23:35:50.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.458+0000 7fec5cff9640 1 --2- 172.21.15.161:0/2887591042 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fec4c06b4d0 0x7fec4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:50.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.458+0000 7fec5cff9640 1 -- 172.21.15.161:0/2887591042 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5507+0+0 (secure 0 0 0) 0x7fec5808a910 con 0x7fec60150520 2024-03-28T23:35:50.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.458+0000 7fec5ffff640 1 --2- 172.21.15.161:0/2887591042 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fec4c06b4d0 0x7fec4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:50.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.458+0000 7fec5ffff640 1 --2- 172.21.15.161:0/2887591042 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fec4c06b4d0 0x7fec4c06d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fec50036000 tx=0x7fec5003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:50.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.462+0000 7fec5cff9640 1 -- 172.21.15.161:0/2887591042 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fec580598e0 con 0x7fec60150520 2024-03-28T23:35:50.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.634+0000 7fec66713640 1 -- 172.21.15.161:0/2887591042 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) -- 0x7fec600c2640 con 0x7fec60150520 2024-03-28T23:35:50.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.637+0000 7fec5cff9640 1 -- 172.21.15.161:0/2887591042 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v50) ==== 96+0+0 (secure 0 0 0) 0x7fec580591e0 con 0x7fec60150520 2024-03-28T23:35:50.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.639+0000 7fec66713640 1 -- 172.21.15.161:0/2887591042 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fec4c06b4d0 msgr2=0x7fec4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:50.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.639+0000 7fec66713640 1 --2- 172.21.15.161:0/2887591042 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fec4c06b4d0 0x7fec4c06d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fec50036000 tx=0x7fec5003c040 comp rx=0 tx=0).stop 2024-03-28T23:35:50.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.639+0000 7fec66713640 1 -- 172.21.15.161:0/2887591042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec60150520 msgr2=0x7fec601ec800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:50.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.639+0000 7fec66713640 1 --2- 172.21.15.161:0/2887591042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec60150520 0x7fec601ec800 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fec58007d80 tx=0x7fec5800e4b0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.639+0000 7fec66713640 1 -- 172.21.15.161:0/2887591042 shutdown_connections 2024-03-28T23:35:50.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.639+0000 7fec66713640 1 --2- 172.21.15.161:0/2887591042 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fec4c06b4d0 0x7fec4c06d990 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.639+0000 7fec66713640 1 --2- 172.21.15.161:0/2887591042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec60150520 0x7fec601ec800 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.639+0000 7fec66713640 1 --2- 172.21.15.161:0/2887591042 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec600c1040 0x7fec601ec2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.639+0000 7fec66713640 1 --2- 172.21.15.161:0/2887591042 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec600a96f0 0x7fec601567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.639+0000 7fec66713640 1 -- 172.21.15.161:0/2887591042 >> 172.21.15.161:0/2887591042 conn(0x7fec600a83f0 msgr2=0x7fec6014e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:50.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.640+0000 7fec66713640 1 -- 172.21.15.161:0/2887591042 shutdown_connections 2024-03-28T23:35:50.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.640+0000 7fec66713640 1 -- 172.21.15.161:0/2887591042 wait complete. 2024-03-28T23:35:50.652 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.b mds_join_fs 2024-03-28T23:35:50.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.753+0000 7fb06c313640 1 Processor -- start 2024-03-28T23:35:50.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.753+0000 7fb06c313640 1 -- start start 2024-03-28T23:35:50.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.754+0000 7fb06c313640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb0640a9960 0x7fb064072970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:50.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.754+0000 7fb06c313640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0640c7330 0x7fb06414e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:50.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.754+0000 7fb06c313640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb0640733e0 0x7fb06414e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:50.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.754+0000 7fb06c313640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb0640aa3a0 con 0x7fb0640a9960 2024-03-28T23:35:50.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.754+0000 7fb06c313640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fb0640abc00 con 0x7fb0640733e0 2024-03-28T23:35:50.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.754+0000 7fb06c313640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb06406bf70 con 0x7fb0640c7330 2024-03-28T23:35:50.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.754+0000 7fb06a0af640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb0640a9960 0x7fb064072970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:50.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.754+0000 7fb06a0af640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb0640a9960 0x7fb064072970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57376/0 (socket says 172.21.15.161:57376) 2024-03-28T23:35:50.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.754+0000 7fb06a0af640 1 -- 172.21.15.161:0/551408459 learned_addr learned my addr 172.21.15.161:0/551408459 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:50.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.755+0000 7fb06a8b0640 1 --2- 172.21.15.161:0/551408459 >> v2:172.21.15.170:3300/0 conn(0x7fb0640733e0 0x7fb06414e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:50.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.755+0000 7fb0698ae640 1 --2- 172.21.15.161:0/551408459 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0640c7330 0x7fb06414e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:50.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.755+0000 7fb06a0af640 1 -- 172.21.15.161:0/551408459 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0640c7330 msgr2=0x7fb06414e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:50.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.755+0000 7fb06a0af640 1 --2- 172.21.15.161:0/551408459 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0640c7330 0x7fb06414e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.755+0000 7fb06a0af640 1 -- 172.21.15.161:0/551408459 >> v2:172.21.15.170:3300/0 conn(0x7fb0640733e0 msgr2=0x7fb06414e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:50.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.755+0000 7fb06a0af640 1 --2- 172.21.15.161:0/551408459 >> v2:172.21.15.170:3300/0 conn(0x7fb0640733e0 0x7fb06414e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.755+0000 7fb06a0af640 1 -- 172.21.15.161:0/551408459 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb06406bcd0 con 0x7fb0640a9960 2024-03-28T23:35:50.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.755+0000 7fb06a8b0640 1 --2- 172.21.15.161:0/551408459 >> v2:172.21.15.170:3300/0 conn(0x7fb0640733e0 0x7fb06414e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:50.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.755+0000 7fb06a0af640 1 --2- 172.21.15.161:0/551408459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0640a9960 0x7fb064072970 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fb06406bea0 tx=0x7fb05c02c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1ab2ec2546c83b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:50.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.756+0000 7fb0690ad640 1 -- 172.21.15.161:0/551408459 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c036020 con 0x7fb0640a9960 2024-03-28T23:35:50.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.756+0000 7fb0690ad640 1 -- 172.21.15.161:0/551408459 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb05c032640 con 0x7fb0640a9960 2024-03-28T23:35:50.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.756+0000 7fb0690ad640 1 -- 172.21.15.161:0/551408459 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c02e040 con 0x7fb0640a9960 2024-03-28T23:35:50.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.756+0000 7fb06c313640 1 -- 172.21.15.161:0/551408459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0640a9960 msgr2=0x7fb064072970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:50.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.756+0000 7fb06c313640 1 --2- 172.21.15.161:0/551408459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0640a9960 0x7fb064072970 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fb06406bea0 tx=0x7fb05c02c8f0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.756+0000 7fb06c313640 1 -- 172.21.15.161:0/551408459 shutdown_connections 2024-03-28T23:35:50.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.756+0000 7fb06c313640 1 --2- 172.21.15.161:0/551408459 >> v2:172.21.15.170:3300/0 conn(0x7fb0640733e0 0x7fb06414e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.756+0000 7fb06c313640 1 --2- 172.21.15.161:0/551408459 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0640c7330 0x7fb06414e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.756+0000 7fb06c313640 1 --2- 172.21.15.161:0/551408459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0640a9960 0x7fb064072970 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.756+0000 7fb06c313640 1 -- 172.21.15.161:0/551408459 >> 172.21.15.161:0/551408459 conn(0x7fb0640a83f0 msgr2=0x7fb0640a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:50.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.756+0000 7fb06c313640 1 -- 172.21.15.161:0/551408459 shutdown_connections 2024-03-28T23:35:50.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.756+0000 7fb06c313640 1 -- 172.21.15.161:0/551408459 wait complete. 2024-03-28T23:35:50.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.757+0000 7fb06c313640 1 Processor -- start 2024-03-28T23:35:50.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.757+0000 7fb06c313640 1 -- start start 2024-03-28T23:35:50.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.758+0000 7fb06c313640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0640733e0 0x7fb0641eeff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:50.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.758+0000 7fb06c313640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0640a9960 0x7fb0640bfce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:50.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.758+0000 7fb06c313640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0640c7330 0x7fb064154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:50.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.758+0000 7fb06c313640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb0640a9d40 con 0x7fb0640a9960 2024-03-28T23:35:50.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.758+0000 7fb06c313640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb064152250 con 0x7fb0640c7330 2024-03-28T23:35:50.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.758+0000 7fb06c313640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb0640a9ec0 con 0x7fb0640733e0 2024-03-28T23:35:50.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.758+0000 7fb0698ae640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0640a9960 0x7fb0640bfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:50.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.758+0000 7fb0698ae640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0640a9960 0x7fb0640bfce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57388/0 (socket says 172.21.15.161:57388) 2024-03-28T23:35:50.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.758+0000 7fb0698ae640 1 -- 172.21.15.161:0/1171957333 learned_addr learned my addr 172.21.15.161:0/1171957333 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:50.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.758+0000 7fb06a0af640 1 --2- 172.21.15.161:0/1171957333 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0640733e0 0x7fb0641eeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:50.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.758+0000 7fb06a8b0640 1 --2- 172.21.15.161:0/1171957333 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0640c7330 0x7fb064154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:50.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.758+0000 7fb0698ae640 1 -- 172.21.15.161:0/1171957333 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0640733e0 msgr2=0x7fb0641eeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:50.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.758+0000 7fb0698ae640 1 --2- 172.21.15.161:0/1171957333 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0640733e0 0x7fb0641eeff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.758+0000 7fb0698ae640 1 -- 172.21.15.161:0/1171957333 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0640c7330 msgr2=0x7fb064154ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:50.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.758+0000 7fb0698ae640 1 --2- 172.21.15.161:0/1171957333 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0640c7330 0x7fb064154ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.758+0000 7fb0698ae640 1 -- 172.21.15.161:0/1171957333 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0640c6a90 con 0x7fb0640a9960 2024-03-28T23:35:50.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.759+0000 7fb0698ae640 1 --2- 172.21.15.161:0/1171957333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0640a9960 0x7fb0640bfce0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fb05400d620 tx=0x7fb05400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:50.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.759+0000 7fb05affd640 1 -- 172.21.15.161:0/1171957333 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05400b620 con 0x7fb0640a9960 2024-03-28T23:35:50.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.759+0000 7fb05affd640 1 -- 172.21.15.161:0/1171957333 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb054019070 con 0x7fb0640a9960 2024-03-28T23:35:50.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.759+0000 7fb06c313640 1 -- 172.21.15.161:0/1171957333 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0641511e0 con 0x7fb0640a9960 2024-03-28T23:35:50.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.759+0000 7fb05affd640 1 -- 172.21.15.161:0/1171957333 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05400ba00 con 0x7fb0640a9960 2024-03-28T23:35:50.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.759+0000 7fb06c313640 1 -- 172.21.15.161:0/1171957333 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0641ee510 con 0x7fb0640a9960 2024-03-28T23:35:50.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.760+0000 7fb06c313640 1 -- 172.21.15.161:0/1171957333 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0640c2bd0 con 0x7fb0640a9960 2024-03-28T23:35:50.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.760+0000 7fb05affd640 1 -- 172.21.15.161:0/1171957333 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb05400bba0 con 0x7fb0640a9960 2024-03-28T23:35:50.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.761+0000 7fb05affd640 1 --2- 172.21.15.161:0/1171957333 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb04806b500 0x7fb04806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:50.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.761+0000 7fb06a0af640 1 --2- 172.21.15.161:0/1171957333 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb04806b500 0x7fb04806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:50.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.761+0000 7fb05affd640 1 -- 172.21.15.161:0/1171957333 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5507+0+0 (secure 0 0 0) 0x7fb05408a350 con 0x7fb0640a9960 2024-03-28T23:35:50.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.761+0000 7fb06a0af640 1 --2- 172.21.15.161:0/1171957333 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb04806b500 0x7fb04806d9c0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fb05c037000 tx=0x7fb05c03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:50.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.765+0000 7fb05affd640 1 -- 172.21.15.161:0/1171957333 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb054058cd0 con 0x7fb0640a9960 2024-03-28T23:35:50.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.940+0000 7fb06c313640 1 -- 172.21.15.161:0/1171957333 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"} v 0) -- 0x7fb06406b1b0 con 0x7fb0640a9960 2024-03-28T23:35:50.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.941+0000 7fb05affd640 1 -- 172.21.15.161:0/1171957333 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"}]=0 v50) ==== 96+0+0 (secure 0 0 0) 0x7fb05405cd70 con 0x7fb0640a9960 2024-03-28T23:35:50.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.942+0000 7fb06c313640 1 -- 172.21.15.161:0/1171957333 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb04806b500 msgr2=0x7fb04806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:50.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.942+0000 7fb06c313640 1 --2- 172.21.15.161:0/1171957333 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb04806b500 0x7fb04806d9c0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fb05c037000 tx=0x7fb05c03d040 comp rx=0 tx=0).stop 2024-03-28T23:35:50.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.942+0000 7fb06c313640 1 -- 172.21.15.161:0/1171957333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0640a9960 msgr2=0x7fb0640bfce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:50.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.942+0000 7fb06c313640 1 --2- 172.21.15.161:0/1171957333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0640a9960 0x7fb0640bfce0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fb05400d620 tx=0x7fb05400dae0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.942+0000 7fb06c313640 1 -- 172.21.15.161:0/1171957333 shutdown_connections 2024-03-28T23:35:50.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.942+0000 7fb06c313640 1 --2- 172.21.15.161:0/1171957333 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb04806b500 0x7fb04806d9c0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.942+0000 7fb06c313640 1 --2- 172.21.15.161:0/1171957333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0640a9960 0x7fb0640bfce0 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.942+0000 7fb06c313640 1 --2- 172.21.15.161:0/1171957333 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0640c7330 0x7fb064154ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.942+0000 7fb06c313640 1 --2- 172.21.15.161:0/1171957333 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0640733e0 0x7fb0641eeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:50.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.942+0000 7fb06c313640 1 -- 172.21.15.161:0/1171957333 >> 172.21.15.161:0/1171957333 conn(0x7fb0640a83f0 msgr2=0x7fb0640a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:50.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.943+0000 7fb06c313640 1 -- 172.21.15.161:0/1171957333 shutdown_connections 2024-03-28T23:35:50.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:50.943+0000 7fb06c313640 1 -- 172.21.15.161:0/1171957333 wait complete. 2024-03-28T23:35:50.954 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset' 2024-03-28T23:35:51.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.055+0000 7fa13548f640 1 Processor -- start 2024-03-28T23:35:51.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.055+0000 7fa13548f640 1 -- start start 2024-03-28T23:35:51.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.056+0000 7fa13548f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa130152a80 0x7fa130155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:51.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.056+0000 7fa13548f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa13006cb80 0x7fa1300c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:51.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.056+0000 7fa13548f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa13006c3c0 con 0x7fa13006d9f0 2024-03-28T23:35:51.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.056+0000 7fa13548f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa130156250 con 0x7fa130152a80 2024-03-28T23:35:51.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.056+0000 7fa13548f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa13006c6f0 con 0x7fa13006cb80 2024-03-28T23:35:51.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.056+0000 7fa12effd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa130152a80 0x7fa130155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:51.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.056+0000 7fa12effd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa130152a80 0x7fa130155d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57390/0 (socket says 172.21.15.161:57390) 2024-03-28T23:35:51.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.056+0000 7fa12effd640 1 -- 172.21.15.161:0/1751001904 learned_addr learned my addr 172.21.15.161:0/1751001904 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:51.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.056+0000 7fa12f7fe640 1 --2- 172.21.15.161:0/1751001904 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa13006cb80 0x7fa1300c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:51.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.056+0000 7fa12effd640 1 -- 172.21.15.161:0/1751001904 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa13006cb80 msgr2=0x7fa1300c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:51.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.056+0000 7fa12effd640 1 --2- 172.21.15.161:0/1751001904 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa13006cb80 0x7fa1300c74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:51.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.056+0000 7fa12effd640 1 -- 172.21.15.161:0/1751001904 >> v1:172.21.15.170:6789/0 conn(0x7fa13006d9f0 legacy=0x7fa1300c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:51.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.056+0000 7fa12effd640 1 -- 172.21.15.161:0/1751001904 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1300c5b90 con 0x7fa130152a80 2024-03-28T23:35:51.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.057+0000 7fa12effd640 1 --2- 172.21.15.161:0/1751001904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa130152a80 0x7fa130155d10 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7fa120009e50 tx=0x7fa12002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f21a81df8c3bd52 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:51.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.057+0000 7fa12dffb640 1 -- 172.21.15.161:0/1751001904 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12002c9e0 con 0x7fa130152a80 2024-03-28T23:35:51.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.057+0000 7fa12dffb640 1 -- 172.21.15.161:0/1751001904 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa12002cb80 con 0x7fa130152a80 2024-03-28T23:35:51.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.057+0000 7fa12dffb640 1 -- 172.21.15.161:0/1751001904 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1200343f0 con 0x7fa130152a80 2024-03-28T23:35:51.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.058+0000 7fa13548f640 1 -- 172.21.15.161:0/1751001904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa130152a80 msgr2=0x7fa130155d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:51.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.058+0000 7fa13548f640 1 --2- 172.21.15.161:0/1751001904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa130152a80 0x7fa130155d10 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7fa120009e50 tx=0x7fa12002be40 comp rx=0 tx=0).stop 2024-03-28T23:35:51.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.058+0000 7fa13548f640 1 -- 172.21.15.161:0/1751001904 shutdown_connections 2024-03-28T23:35:51.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.058+0000 7fa13548f640 1 --2- 172.21.15.161:0/1751001904 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa13006cb80 0x7fa1300c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:51.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.058+0000 7fa13548f640 1 --2- 172.21.15.161:0/1751001904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa130152a80 0x7fa130155d10 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:51.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.058+0000 7fa13548f640 1 -- 172.21.15.161:0/1751001904 >> 172.21.15.161:0/1751001904 conn(0x7fa1300ac720 msgr2=0x7fa1300acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:51.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.058+0000 7fa13548f640 1 -- 172.21.15.161:0/1751001904 shutdown_connections 2024-03-28T23:35:51.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.058+0000 7fa13548f640 1 -- 172.21.15.161:0/1751001904 wait complete. 2024-03-28T23:35:51.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.059+0000 7fa13548f640 1 Processor -- start 2024-03-28T23:35:51.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.059+0000 7fa13548f640 1 -- start start 2024-03-28T23:35:51.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.060+0000 7fa13548f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa13006cb80 0x7fa1301ee3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:51.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.060+0000 7fa13548f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa13006d9f0 0x7fa13014a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:51.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.060+0000 7fa13548f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa130152a80 0x7fa13014abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:51.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.060+0000 7fa13548f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa13006d010 con 0x7fa13006d9f0 2024-03-28T23:35:51.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.060+0000 7fa13548f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa1300c58b0 con 0x7fa13006cb80 2024-03-28T23:35:51.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.060+0000 7fa13548f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa13006d190 con 0x7fa130152a80 2024-03-28T23:35:51.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.060+0000 7fa12e7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa13006d9f0 0x7fa13014a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:51.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.060+0000 7fa12e7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa13006d9f0 0x7fa13014a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57402/0 (socket says 172.21.15.161:57402) 2024-03-28T23:35:51.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.060+0000 7fa12e7fc640 1 -- 172.21.15.161:0/2656627817 learned_addr learned my addr 172.21.15.161:0/2656627817 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:51.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.060+0000 7fa12f7fe640 1 --2- 172.21.15.161:0/2656627817 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa130152a80 0x7fa13014abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:51.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.060+0000 7fa12effd640 1 --2- 172.21.15.161:0/2656627817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa13006cb80 0x7fa1301ee3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:51.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.060+0000 7fa12e7fc640 1 -- 172.21.15.161:0/2656627817 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa130152a80 msgr2=0x7fa13014abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:51.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.060+0000 7fa12e7fc640 1 --2- 172.21.15.161:0/2656627817 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa130152a80 0x7fa13014abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:51.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.060+0000 7fa12e7fc640 1 -- 172.21.15.161:0/2656627817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa13006cb80 msgr2=0x7fa1301ee3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:51.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.060+0000 7fa12e7fc640 1 --2- 172.21.15.161:0/2656627817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa13006cb80 0x7fa1301ee3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:51.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.060+0000 7fa12e7fc640 1 -- 172.21.15.161:0/2656627817 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa13014b0f0 con 0x7fa13006d9f0 2024-03-28T23:35:51.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.060+0000 7fa12e7fc640 1 --2- 172.21.15.161:0/2656627817 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa13006d9f0 0x7fa13014a670 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fa12400c400 tx=0x7fa12400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:51.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.060+0000 7fa1177fe640 1 -- 172.21.15.161:0/2656627817 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1240052d0 con 0x7fa13006d9f0 2024-03-28T23:35:51.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.060+0000 7fa1177fe640 1 -- 172.21.15.161:0/2656627817 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa124005490 con 0x7fa13006d9f0 2024-03-28T23:35:51.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.060+0000 7fa13548f640 1 -- 172.21.15.161:0/2656627817 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa13014b9f0 con 0x7fa13006d9f0 2024-03-28T23:35:51.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.061+0000 7fa1177fe640 1 -- 172.21.15.161:0/2656627817 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa124007260 con 0x7fa13006d9f0 2024-03-28T23:35:51.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.061+0000 7fa13548f640 1 -- 172.21.15.161:0/2656627817 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa130155210 con 0x7fa13006d9f0 2024-03-28T23:35:51.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.062+0000 7fa13548f640 1 -- 172.21.15.161:0/2656627817 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa13014b750 con 0x7fa13006d9f0 2024-03-28T23:35:51.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.062+0000 7fa1177fe640 1 -- 172.21.15.161:0/2656627817 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa124007400 con 0x7fa13006d9f0 2024-03-28T23:35:51.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.062+0000 7fa1177fe640 1 --2- 172.21.15.161:0/2656627817 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa11c06b4d0 0x7fa11c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:51.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.062+0000 7fa1177fe640 1 -- 172.21.15.161:0/2656627817 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5507+0+0 (secure 0 0 0) 0x7fa124089530 con 0x7fa13006d9f0 2024-03-28T23:35:51.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.062+0000 7fa12effd640 1 --2- 172.21.15.161:0/2656627817 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa11c06b4d0 0x7fa11c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:51.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.063+0000 7fa12effd640 1 --2- 172.21.15.161:0/2656627817 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa11c06b4d0 0x7fa11c06d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fa120035000 tx=0x7fa120030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:51.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.067+0000 7fa1177fe640 1 -- 172.21.15.161:0/2656627817 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa124057eb0 con 0x7fa13006d9f0 2024-03-28T23:35:51.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:51.305+0000 7fa13548f640 1 -- 172.21.15.161:0/2656627817 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset"]} v 0) -- 0x7fa13014b300 con 0x7fa13006d9f0 2024-03-28T23:35:52.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.022+0000 7fa1177fe640 1 -- 172.21.15.161:0/2656627817 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset"]}]=0 v350) ==== 144+0+0 (secure 0 0 0) 0x7fa12405bf50 con 0x7fa13006d9f0 2024-03-28T23:35:52.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.024+0000 7fa13548f640 1 -- 172.21.15.161:0/2656627817 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa11c06b4d0 msgr2=0x7fa11c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:52.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.024+0000 7fa13548f640 1 --2- 172.21.15.161:0/2656627817 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa11c06b4d0 0x7fa11c06d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fa120035000 tx=0x7fa120030000 comp rx=0 tx=0).stop 2024-03-28T23:35:52.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.024+0000 7fa13548f640 1 -- 172.21.15.161:0/2656627817 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa13006d9f0 msgr2=0x7fa13014a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:52.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.024+0000 7fa13548f640 1 --2- 172.21.15.161:0/2656627817 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa13006d9f0 0x7fa13014a670 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fa12400c400 tx=0x7fa12400c8c0 comp rx=0 tx=0).stop 2024-03-28T23:35:52.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.024+0000 7fa13548f640 1 -- 172.21.15.161:0/2656627817 shutdown_connections 2024-03-28T23:35:52.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.024+0000 7fa13548f640 1 --2- 172.21.15.161:0/2656627817 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa11c06b4d0 0x7fa11c06d990 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:52.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.024+0000 7fa13548f640 1 --2- 172.21.15.161:0/2656627817 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa13006d9f0 0x7fa13014a670 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:52.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.024+0000 7fa13548f640 1 --2- 172.21.15.161:0/2656627817 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa130152a80 0x7fa13014abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:52.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.024+0000 7fa13548f640 1 --2- 172.21.15.161:0/2656627817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa13006cb80 0x7fa1301ee3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:52.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.024+0000 7fa13548f640 1 -- 172.21.15.161:0/2656627817 >> 172.21.15.161:0/2656627817 conn(0x7fa1300ac720 msgr2=0x7fa1300ac000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:52.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.025+0000 7fa13548f640 1 -- 172.21.15.161:0/2656627817 shutdown_connections 2024-03-28T23:35:52.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.025+0000 7fa13548f640 1 -- 172.21.15.161:0/2656627817 wait complete. 2024-03-28T23:35:52.038 INFO:tasks.cephfs_test_runner:That unsetting mds_join_fs will cause failover if another high-affinity standby exists. ... ok 2024-03-28T23:35:52.038 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_vanilla (tasks.cephfs.test_failover.TestClusterAffinity) That a vanilla standby is preferred over others with mds_join_fs set to another fs. 2024-03-28T23:35:52.038 INFO:tasks.cephfs_test_runner:test_join_fs_vanilla (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-28T23:35:52.039 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:35:52.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.141+0000 7f34d87e9640 1 Processor -- start 2024-03-28T23:35:52.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.141+0000 7f34d87e9640 1 -- start start 2024-03-28T23:35:52.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.142+0000 7f34d87e9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f34d00c5850 0x7f34d0155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:52.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.142+0000 7f34d87e9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34d00c5d00 con 0x7f34d00c5410 2024-03-28T23:35:52.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.142+0000 7f34d87e9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34d0158a30 con 0x7f34d0152890 2024-03-28T23:35:52.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.142+0000 7f34d87e9640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f34d00c8480 con 0x7f34d00c5850 2024-03-28T23:35:52.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.142+0000 7f34d6d86640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f34d00c5850 0x7f34d0155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:52.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.142+0000 7f34d5d84640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f34d00c5410 0x7f34d01553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:54270/0 (socket says 172.21.15.161:54270) 2024-03-28T23:35:52.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.142+0000 7f34d5d84640 1 -- 172.21.15.161:0/3561334802 learned_addr learned my addr 172.21.15.161:0/3561334802 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:52.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.142+0000 7f34d6d86640 1 --2- 172.21.15.161:0/3561334802 >> v2:172.21.15.161:3300/0 conn(0x7f34d00c5850 0x7f34d0155ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57406/0 (socket says 172.21.15.161:57406) 2024-03-28T23:35:52.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.143+0000 7f34d6d86640 1 -- 172.21.15.161:0/3561334802 >> v1:172.21.15.170:6789/0 conn(0x7f34d0152890 legacy=0x7f34d00c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:52.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.143+0000 7f34d6d86640 1 -- 172.21.15.161:0/3561334802 >> v1:172.21.15.161:6789/0 conn(0x7f34d00c5410 legacy=0x7f34d01553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:52.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.143+0000 7f34d6d86640 1 -- 172.21.15.161:0/3561334802 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34d0158d10 con 0x7f34d00c5850 2024-03-28T23:35:52.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.143+0000 7f34d6d86640 1 --2- 172.21.15.161:0/3561334802 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34d00c5850 0x7f34d0155ab0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f34c8009d90 tx=0x7f34c802bc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c37636ee022234 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:52.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.144+0000 7f34d5583640 1 -- 172.21.15.161:0/3561334802 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c8039070 con 0x7f34d00c5850 2024-03-28T23:35:52.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.144+0000 7f34d5583640 1 -- 172.21.15.161:0/3561334802 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34c802c990 con 0x7f34d00c5850 2024-03-28T23:35:52.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.144+0000 7f34d5583640 1 -- 172.21.15.161:0/3561334802 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c802cd60 con 0x7f34d00c5850 2024-03-28T23:35:52.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.144+0000 7f34d87e9640 1 -- 172.21.15.161:0/3561334802 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34d00c5850 msgr2=0x7f34d0155ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:52.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.144+0000 7f34d87e9640 1 --2- 172.21.15.161:0/3561334802 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34d00c5850 0x7f34d0155ab0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f34c8009d90 tx=0x7f34c802bc60 comp rx=0 tx=0).stop 2024-03-28T23:35:52.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.144+0000 7f34d87e9640 1 -- 172.21.15.161:0/3561334802 shutdown_connections 2024-03-28T23:35:52.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.144+0000 7f34d87e9640 1 --2- 172.21.15.161:0/3561334802 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34d00c5850 0x7f34d0155ab0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:52.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.144+0000 7f34d87e9640 1 -- 172.21.15.161:0/3561334802 >> 172.21.15.161:0/3561334802 conn(0x7f34d00ac620 msgr2=0x7f34d00aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:52.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.144+0000 7f34d87e9640 1 -- 172.21.15.161:0/3561334802 shutdown_connections 2024-03-28T23:35:52.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.144+0000 7f34d87e9640 1 -- 172.21.15.161:0/3561334802 wait complete. 2024-03-28T23:35:52.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.145+0000 7f34d87e9640 1 Processor -- start 2024-03-28T23:35:52.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.145+0000 7f34d87e9640 1 -- start start 2024-03-28T23:35:52.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.146+0000 7f34d87e9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f34d00c5410 0x7f34d0157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:52.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.146+0000 7f34d87e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34d00c5850 0x7f34d0154d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:52.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.146+0000 7f34d87e9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34d0152890 0x7f34d014bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:52.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.146+0000 7f34d87e9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f34d01592d0 con 0x7f34d0152890 2024-03-28T23:35:52.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.146+0000 7f34d87e9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f34d0159640 con 0x7f34d00c5410 2024-03-28T23:35:52.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.146+0000 7f34d87e9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f34d0155ff0 con 0x7f34d00c5850 2024-03-28T23:35:52.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.146+0000 7f34d6d86640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34d0152890 0x7f34d014bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:52.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.146+0000 7f34d6d86640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34d0152890 0x7f34d014bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57422/0 (socket says 172.21.15.161:57422) 2024-03-28T23:35:52.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.146+0000 7f34d6d86640 1 -- 172.21.15.161:0/376915668 learned_addr learned my addr 172.21.15.161:0/376915668 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:52.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.146+0000 7f34d6585640 1 --2- 172.21.15.161:0/376915668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f34d00c5410 0x7f34d0157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:52.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.146+0000 7f34d6d86640 1 -- 172.21.15.161:0/376915668 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34d00c5850 msgr2=0x7f34d0154d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:52.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.146+0000 7f34d6d86640 1 --2- 172.21.15.161:0/376915668 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34d00c5850 0x7f34d0154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:52.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.146+0000 7f34d6d86640 1 -- 172.21.15.161:0/376915668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f34d00c5410 msgr2=0x7f34d0157540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:52.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.146+0000 7f34d6d86640 1 --2- 172.21.15.161:0/376915668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f34d00c5410 0x7f34d0157540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:52.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.146+0000 7f34d6d86640 1 -- 172.21.15.161:0/376915668 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34d0153d60 con 0x7f34d0152890 2024-03-28T23:35:52.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.146+0000 7f34d6585640 1 --2- 172.21.15.161:0/376915668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f34d00c5410 0x7f34d0157540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:52.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.147+0000 7f34d6d86640 1 --2- 172.21.15.161:0/376915668 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34d0152890 0x7f34d014bc40 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f34c8009d90 tx=0x7f34c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:52.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.147+0000 7f34c2ffd640 1 -- 172.21.15.161:0/376915668 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c8044020 con 0x7f34d0152890 2024-03-28T23:35:52.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.147+0000 7f34c2ffd640 1 -- 172.21.15.161:0/376915668 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34c8034cf0 con 0x7f34d0152890 2024-03-28T23:35:52.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.147+0000 7f34d87e9640 1 -- 172.21.15.161:0/376915668 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34d01eb850 con 0x7f34d0152890 2024-03-28T23:35:52.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.147+0000 7f34c2ffd640 1 -- 172.21.15.161:0/376915668 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c8039040 con 0x7f34d0152890 2024-03-28T23:35:52.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.147+0000 7f34d87e9640 1 -- 172.21.15.161:0/376915668 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34d0157a80 con 0x7f34d0152890 2024-03-28T23:35:52.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.148+0000 7f34d87e9640 1 -- 172.21.15.161:0/376915668 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34d004f4b0 con 0x7f34d0152890 2024-03-28T23:35:52.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.148+0000 7f34c2ffd640 1 -- 172.21.15.161:0/376915668 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f34c8047020 con 0x7f34d0152890 2024-03-28T23:35:52.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.148+0000 7f34c2ffd640 1 --2- 172.21.15.161:0/376915668 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f34b406b500 0x7f34b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:52.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.149+0000 7f34c2ffd640 1 -- 172.21.15.161:0/376915668 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5507+0+0 (secure 0 0 0) 0x7f34c80ac5b0 con 0x7f34d0152890 2024-03-28T23:35:52.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.149+0000 7f34d6585640 1 --2- 172.21.15.161:0/376915668 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f34b406b500 0x7f34b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:52.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.149+0000 7f34d6585640 1 --2- 172.21.15.161:0/376915668 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f34b406b500 0x7f34b406d9c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f34d0157f00 tx=0x7f34c4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:52.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.153+0000 7f34c2ffd640 1 -- 172.21.15.161:0/376915668 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f34c804e070 con 0x7f34d0152890 2024-03-28T23:35:52.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.325+0000 7f34d87e9640 1 -- 172.21.15.161:0/376915668 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f34d00c8140 con 0x7f34d0152890 2024-03-28T23:35:52.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.326+0000 7f34c2ffd640 1 -- 172.21.15.161:0/376915668 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) ==== 74+0+13605 (secure 0 0 0) 0x7f34c807b760 con 0x7f34d0152890 2024-03-28T23:35:52.328 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:52.328 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":143,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:35:50.011884+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6834/3182285172":"2024-03-29T23:35:46.763641+0000","172.21.15.170:6832/3182285172":"2024-03-29T23:35:46.763641+0000","172.21.15.170:6835/3586875977":"2024-03-29T23:35:38.963130+0000","172.21.15.170:6833/3586875977":"2024-03-29T23:35:38.963130+0000","172.21.15.161:6836/2527502405":"2024-03-29T23:35:21.284385+0000","172.21.15.161:6834/2527502405":"2024-03-29T23:35:21.284385+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:35:52.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.328+0000 7f34d87e9640 1 -- 172.21.15.161:0/376915668 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f34b406b500 msgr2=0x7f34b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:52.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.328+0000 7f34d87e9640 1 --2- 172.21.15.161:0/376915668 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f34b406b500 0x7f34b406d9c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f34d0157f00 tx=0x7f34c4009040 comp rx=0 tx=0).stop 2024-03-28T23:35:52.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.328+0000 7f34d87e9640 1 -- 172.21.15.161:0/376915668 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34d0152890 msgr2=0x7f34d014bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:52.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.328+0000 7f34d87e9640 1 --2- 172.21.15.161:0/376915668 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34d0152890 0x7f34d014bc40 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f34c8009d90 tx=0x7f34c8004a40 comp rx=0 tx=0).stop 2024-03-28T23:35:52.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.328+0000 7f34d87e9640 1 -- 172.21.15.161:0/376915668 shutdown_connections 2024-03-28T23:35:52.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.328+0000 7f34d87e9640 1 --2- 172.21.15.161:0/376915668 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f34b406b500 0x7f34b406d9c0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:52.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.328+0000 7f34d87e9640 1 --2- 172.21.15.161:0/376915668 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34d0152890 0x7f34d014bc40 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:52.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.328+0000 7f34d87e9640 1 --2- 172.21.15.161:0/376915668 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34d00c5850 0x7f34d0154d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:52.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.328+0000 7f34d87e9640 1 --2- 172.21.15.161:0/376915668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f34d00c5410 0x7f34d0157540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:52.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.328+0000 7f34d87e9640 1 -- 172.21.15.161:0/376915668 >> 172.21.15.161:0/376915668 conn(0x7f34d00ac620 msgr2=0x7f34d014d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:52.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.328+0000 7f34d87e9640 1 -- 172.21.15.161:0/376915668 shutdown_connections 2024-03-28T23:35:52.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.328+0000 7f34d87e9640 1 -- 172.21.15.161:0/376915668 wait complete. 2024-03-28T23:35:52.341 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:35:52.341 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:35:52.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.444+0000 7f777e1e9640 1 Processor -- start 2024-03-28T23:35:52.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.444+0000 7f777e1e9640 1 -- start start 2024-03-28T23:35:52.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.445+0000 7f777e1e9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7778151e70 0x7f7778152250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:52.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.445+0000 7f777e1e9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77780c1e40 con 0x7f777806cdd0 2024-03-28T23:35:52.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.445+0000 7f777e1e9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77780c2120 con 0x7f7778152790 2024-03-28T23:35:52.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.445+0000 7f777e1e9640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f77780c2400 con 0x7f7778151e70 2024-03-28T23:35:52.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.445+0000 7f77777fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7778151e70 0x7f7778152250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:52.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.445+0000 7f7777fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f777806cdd0 0x7f777815c760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:54280/0 (socket says 172.21.15.161:54280) 2024-03-28T23:35:52.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.445+0000 7f7777fff640 1 -- 172.21.15.161:0/2066791735 learned_addr learned my addr 172.21.15.161:0/2066791735 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:52.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.445+0000 7f77777fe640 1 -- 172.21.15.161:0/2066791735 >> v1:172.21.15.170:6789/0 conn(0x7f7778152790 legacy=0x7f777806c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:52.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.445+0000 7f77777fe640 1 -- 172.21.15.161:0/2066791735 >> v1:172.21.15.161:6789/0 conn(0x7f777806cdd0 legacy=0x7f777815c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:52.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.446+0000 7f77777fe640 1 -- 172.21.15.161:0/2066791735 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77780c26e0 con 0x7f7778151e70 2024-03-28T23:35:52.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.445+0000 7f77767fc640 1 -- 172.21.15.161:0/2066791735 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3965626811 0 0) 0x7f77780c1e40 con 0x7f777806cdd0 2024-03-28T23:35:52.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.446+0000 7f77777fe640 1 --2- 172.21.15.161:0/2066791735 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7778151e70 0x7f7778152250 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f77680097a0 tx=0x7f776802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f008e003ed3af19 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:52.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.446+0000 7f77767fc640 1 -- 172.21.15.161:0/2066791735 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77680315a0 con 0x7f7778151e70 2024-03-28T23:35:52.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.446+0000 7f77767fc640 1 -- 172.21.15.161:0/2066791735 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7768037070 con 0x7f7778151e70 2024-03-28T23:35:52.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.446+0000 7f77767fc640 1 -- 172.21.15.161:0/2066791735 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7768031960 con 0x7f7778151e70 2024-03-28T23:35:52.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.447+0000 7f777e1e9640 1 -- 172.21.15.161:0/2066791735 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7778151e70 msgr2=0x7f7778152250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:52.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.447+0000 7f777e1e9640 1 --2- 172.21.15.161:0/2066791735 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7778151e70 0x7f7778152250 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f77680097a0 tx=0x7f776802b840 comp rx=0 tx=0).stop 2024-03-28T23:35:52.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.447+0000 7f777e1e9640 1 -- 172.21.15.161:0/2066791735 shutdown_connections 2024-03-28T23:35:52.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.447+0000 7f777e1e9640 1 --2- 172.21.15.161:0/2066791735 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7778151e70 0x7f7778152250 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:52.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.447+0000 7f777e1e9640 1 -- 172.21.15.161:0/2066791735 >> 172.21.15.161:0/2066791735 conn(0x7f77780c3390 msgr2=0x7f77780c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:52.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.447+0000 7f777e1e9640 1 -- 172.21.15.161:0/2066791735 shutdown_connections 2024-03-28T23:35:52.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.447+0000 7f777e1e9640 1 -- 172.21.15.161:0/2066791735 wait complete. 2024-03-28T23:35:52.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.448+0000 7f777e1e9640 1 Processor -- start 2024-03-28T23:35:52.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.448+0000 7f777e1e9640 1 -- start start 2024-03-28T23:35:52.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.449+0000 7f777e1e9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f777806cdd0 0x7f77780a7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:52.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.449+0000 7f777e1e9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7778151e70 0x7f77781ee1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:52.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.449+0000 7f777e1e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77780c05c0 0x7f77781ee730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:52.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.449+0000 7f777e1e9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f77780c2d80 con 0x7f7778151e70 2024-03-28T23:35:52.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.449+0000 7f777e1e9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f77780c2c00 con 0x7f777806cdd0 2024-03-28T23:35:52.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.449+0000 7f777e1e9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f77780c6350 con 0x7f77780c05c0 2024-03-28T23:35:52.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.449+0000 7f7776ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7778151e70 0x7f77781ee1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:52.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.449+0000 7f7776ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7778151e70 0x7f77781ee1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57428/0 (socket says 172.21.15.161:57428) 2024-03-28T23:35:52.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.449+0000 7f7776ffd640 1 -- 172.21.15.161:0/1075088086 learned_addr learned my addr 172.21.15.161:0/1075088086 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:52.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.449+0000 7f77777fe640 1 --2- 172.21.15.161:0/1075088086 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f777806cdd0 0x7f77780a7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:52.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.449+0000 7f7777fff640 1 --2- 172.21.15.161:0/1075088086 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77780c05c0 0x7f77781ee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:52.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.449+0000 7f7776ffd640 1 -- 172.21.15.161:0/1075088086 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77780c05c0 msgr2=0x7f77781ee730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:52.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.449+0000 7f7776ffd640 1 --2- 172.21.15.161:0/1075088086 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77780c05c0 0x7f77781ee730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:52.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.449+0000 7f7776ffd640 1 -- 172.21.15.161:0/1075088086 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f777806cdd0 msgr2=0x7f77780a7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:52.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.449+0000 7f7776ffd640 1 --2- 172.21.15.161:0/1075088086 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f777806cdd0 0x7f77780a7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:52.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.449+0000 7f7776ffd640 1 -- 172.21.15.161:0/1075088086 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77781518e0 con 0x7f7778151e70 2024-03-28T23:35:52.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.449+0000 7f7777fff640 1 --2- 172.21.15.161:0/1075088086 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77780c05c0 0x7f77781ee730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:52.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.449+0000 7f7776ffd640 1 --2- 172.21.15.161:0/1075088086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7778151e70 0x7f77781ee1f0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f776c009fd0 tx=0x7f776c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:52.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.449+0000 7f775ffff640 1 -- 172.21.15.161:0/1075088086 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c015020 con 0x7f7778151e70 2024-03-28T23:35:52.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.450+0000 7f775ffff640 1 -- 172.21.15.161:0/1075088086 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f776c011830 con 0x7f7778151e70 2024-03-28T23:35:52.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.450+0000 7f775ffff640 1 -- 172.21.15.161:0/1075088086 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c011c00 con 0x7f7778151e70 2024-03-28T23:35:52.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.450+0000 7f777e1e9640 1 -- 172.21.15.161:0/1075088086 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f777814ebe0 con 0x7f7778151e70 2024-03-28T23:35:52.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.450+0000 7f777e1e9640 1 -- 172.21.15.161:0/1075088086 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77780aab40 con 0x7f7778151e70 2024-03-28T23:35:52.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.451+0000 7f777e1e9640 1 -- 172.21.15.161:0/1075088086 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f777804f4b0 con 0x7f7778151e70 2024-03-28T23:35:52.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.451+0000 7f775ffff640 1 -- 172.21.15.161:0/1075088086 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f776c018020 con 0x7f7778151e70 2024-03-28T23:35:52.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.451+0000 7f775ffff640 1 --2- 172.21.15.161:0/1075088086 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f776406b500 0x7f776406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:52.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.451+0000 7f775ffff640 1 -- 172.21.15.161:0/1075088086 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5507+0+0 (secure 0 0 0) 0x7f776c089eb0 con 0x7f7778151e70 2024-03-28T23:35:52.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.452+0000 7f77777fe640 1 --2- 172.21.15.161:0/1075088086 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f776406b500 0x7f776406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:52.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.452+0000 7f77777fe640 1 --2- 172.21.15.161:0/1075088086 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f776406b500 0x7f776406d9c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f7768036000 tx=0x7f776803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:52.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.456+0000 7f775ffff640 1 -- 172.21.15.161:0/1075088086 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f776c01f020 con 0x7f7778151e70 2024-03-28T23:35:52.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.626+0000 7f777e1e9640 1 -- 172.21.15.161:0/1075088086 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f77780c1af0 con 0x7f7778151e70 2024-03-28T23:35:52.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.626+0000 7f775ffff640 1 -- 172.21.15.161:0/1075088086 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) ==== 93+0+10 (secure 0 0 0) 0x7f776c058e80 con 0x7f7778151e70 2024-03-28T23:35:52.628 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:35:52.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.628+0000 7f777e1e9640 1 -- 172.21.15.161:0/1075088086 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f776406b500 msgr2=0x7f776406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:52.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.628+0000 7f777e1e9640 1 --2- 172.21.15.161:0/1075088086 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f776406b500 0x7f776406d9c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f7768036000 tx=0x7f776803d040 comp rx=0 tx=0).stop 2024-03-28T23:35:52.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.628+0000 7f777e1e9640 1 -- 172.21.15.161:0/1075088086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7778151e70 msgr2=0x7f77781ee1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:52.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.628+0000 7f777e1e9640 1 --2- 172.21.15.161:0/1075088086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7778151e70 0x7f77781ee1f0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f776c009fd0 tx=0x7f776c00b870 comp rx=0 tx=0).stop 2024-03-28T23:35:52.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.628+0000 7f777e1e9640 1 -- 172.21.15.161:0/1075088086 shutdown_connections 2024-03-28T23:35:52.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.628+0000 7f777e1e9640 1 --2- 172.21.15.161:0/1075088086 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f776406b500 0x7f776406d9c0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:52.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.628+0000 7f777e1e9640 1 --2- 172.21.15.161:0/1075088086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7778151e70 0x7f77781ee1f0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:52.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.628+0000 7f777e1e9640 1 --2- 172.21.15.161:0/1075088086 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77780c05c0 0x7f77781ee730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:52.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.628+0000 7f777e1e9640 1 --2- 172.21.15.161:0/1075088086 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f777806cdd0 0x7f77780a7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:52.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.628+0000 7f777e1e9640 1 -- 172.21.15.161:0/1075088086 >> 172.21.15.161:0/1075088086 conn(0x7f77780c3390 msgr2=0x7f7778150ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:52.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.629+0000 7f777e1e9640 1 -- 172.21.15.161:0/1075088086 shutdown_connections 2024-03-28T23:35:52.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.629+0000 7f777e1e9640 1 -- 172.21.15.161:0/1075088086 wait complete. 2024-03-28T23:35:52.641 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla' 2024-03-28T23:35:52.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.740+0000 7fd0df48f640 1 Processor -- start 2024-03-28T23:35:52.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.740+0000 7fd0df48f640 1 -- start start 2024-03-28T23:35:52.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.741+0000 7fd0df48f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd0d814be70 0x7fd0d80ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:52.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.741+0000 7fd0df48f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0d80aacd0 0x7fd0d815c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:52.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.741+0000 7fd0df48f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0d814c250 con 0x7fd0d80ab6e0 2024-03-28T23:35:52.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.741+0000 7fd0df48f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd0d81529c0 con 0x7fd0d814be70 2024-03-28T23:35:52.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.741+0000 7fd0df48f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd0d80a9dc0 con 0x7fd0d80aacd0 2024-03-28T23:35:52.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.741+0000 7fd0dd22b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd0d814be70 0x7fd0d80ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:52.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.741+0000 7fd0dca2a640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd0d80ab6e0 0x7fd0d80aa5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:54292/0 (socket says 172.21.15.161:54292) 2024-03-28T23:35:52.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.741+0000 7fd0dd22b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd0d814be70 0x7fd0d80ab1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57444/0 (socket says 172.21.15.161:57444) 2024-03-28T23:35:52.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.741+0000 7fd0dca2a640 1 -- 172.21.15.161:0/2256775676 learned_addr learned my addr 172.21.15.161:0/2256775676 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:52.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.741+0000 7fd0dda2c640 1 --2- 172.21.15.161:0/2256775676 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0d80aacd0 0x7fd0d815c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:52.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.741+0000 7fd0dd22b640 1 -- 172.21.15.161:0/2256775676 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0d80aacd0 msgr2=0x7fd0d815c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:52.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.741+0000 7fd0dd22b640 1 --2- 172.21.15.161:0/2256775676 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0d80aacd0 0x7fd0d815c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:52.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.741+0000 7fd0dd22b640 1 -- 172.21.15.161:0/2256775676 >> v1:172.21.15.161:6789/0 conn(0x7fd0d80ab6e0 legacy=0x7fd0d80aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:52.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.741+0000 7fd0cffff640 1 -- 172.21.15.161:0/2256775676 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2339662937 0 0) 0x7fd0d814c250 con 0x7fd0d80ab6e0 2024-03-28T23:35:52.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.741+0000 7fd0dd22b640 1 -- 172.21.15.161:0/2256775676 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0d80aa0a0 con 0x7fd0d814be70 2024-03-28T23:35:52.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.742+0000 7fd0dd22b640 1 --2- 172.21.15.161:0/2256775676 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0d814be70 0x7fd0d80ab1a0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fd0d40097a0 tx=0x7fd0d402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20ea7c5c816bed4d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:52.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.742+0000 7fd0cffff640 1 -- 172.21.15.161:0/2256775676 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d40315a0 con 0x7fd0d814be70 2024-03-28T23:35:52.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.742+0000 7fd0cffff640 1 -- 172.21.15.161:0/2256775676 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0d4037070 con 0x7fd0d814be70 2024-03-28T23:35:52.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.742+0000 7fd0cffff640 1 -- 172.21.15.161:0/2256775676 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d4031960 con 0x7fd0d814be70 2024-03-28T23:35:52.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.743+0000 7fd0df48f640 1 -- 172.21.15.161:0/2256775676 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0d814be70 msgr2=0x7fd0d80ab1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:52.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.743+0000 7fd0df48f640 1 --2- 172.21.15.161:0/2256775676 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0d814be70 0x7fd0d80ab1a0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fd0d40097a0 tx=0x7fd0d402b840 comp rx=0 tx=0).stop 2024-03-28T23:35:52.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.743+0000 7fd0df48f640 1 -- 172.21.15.161:0/2256775676 shutdown_connections 2024-03-28T23:35:52.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.743+0000 7fd0df48f640 1 --2- 172.21.15.161:0/2256775676 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0d80aacd0 0x7fd0d815c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:52.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.743+0000 7fd0df48f640 1 --2- 172.21.15.161:0/2256775676 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0d814be70 0x7fd0d80ab1a0 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:52.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.743+0000 7fd0df48f640 1 -- 172.21.15.161:0/2256775676 >> 172.21.15.161:0/2256775676 conn(0x7fd0d80c6670 msgr2=0x7fd0d80c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:52.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.743+0000 7fd0df48f640 1 -- 172.21.15.161:0/2256775676 shutdown_connections 2024-03-28T23:35:52.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.743+0000 7fd0df48f640 1 -- 172.21.15.161:0/2256775676 wait complete. 2024-03-28T23:35:52.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.744+0000 7fd0df48f640 1 Processor -- start 2024-03-28T23:35:52.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.744+0000 7fd0df48f640 1 -- start start 2024-03-28T23:35:52.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.745+0000 7fd0df48f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0d80aacd0 0x7fd0d814d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:52.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.745+0000 7fd0df48f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0d80ab6e0 0x7fd0d80c5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:52.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.745+0000 7fd0df48f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0d814be70 0x7fd0d81dfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:52.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.745+0000 7fd0df48f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d806b2d0 con 0x7fd0d80ab6e0 2024-03-28T23:35:52.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.745+0000 7fd0df48f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d806b150 con 0x7fd0d80aacd0 2024-03-28T23:35:52.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.745+0000 7fd0df48f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd0d80c1520 con 0x7fd0d814be70 2024-03-28T23:35:52.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.745+0000 7fd0dca2a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0d80ab6e0 0x7fd0d80c5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:52.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.745+0000 7fd0dca2a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0d80ab6e0 0x7fd0d80c5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57450/0 (socket says 172.21.15.161:57450) 2024-03-28T23:35:52.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.745+0000 7fd0dca2a640 1 -- 172.21.15.161:0/1114711413 learned_addr learned my addr 172.21.15.161:0/1114711413 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:52.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.745+0000 7fd0dda2c640 1 --2- 172.21.15.161:0/1114711413 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0d814be70 0x7fd0d81dfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:52.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.745+0000 7fd0dd22b640 1 --2- 172.21.15.161:0/1114711413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0d80aacd0 0x7fd0d814d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:52.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.745+0000 7fd0dca2a640 1 -- 172.21.15.161:0/1114711413 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0d814be70 msgr2=0x7fd0d81dfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:52.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.745+0000 7fd0dca2a640 1 --2- 172.21.15.161:0/1114711413 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0d814be70 0x7fd0d81dfaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:52.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.745+0000 7fd0dca2a640 1 -- 172.21.15.161:0/1114711413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0d80aacd0 msgr2=0x7fd0d814d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:52.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.745+0000 7fd0dca2a640 1 --2- 172.21.15.161:0/1114711413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0d80aacd0 0x7fd0d814d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:52.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.745+0000 7fd0dca2a640 1 -- 172.21.15.161:0/1114711413 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0d814d000 con 0x7fd0d80ab6e0 2024-03-28T23:35:52.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.745+0000 7fd0dd22b640 1 --2- 172.21.15.161:0/1114711413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0d80aacd0 0x7fd0d814d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:52.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.745+0000 7fd0dca2a640 1 --2- 172.21.15.161:0/1114711413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0d80ab6e0 0x7fd0d80c5cd0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fd0c800c430 tx=0x7fd0c800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:52.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.745+0000 7fd0cdffb640 1 -- 172.21.15.161:0/1114711413 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c80074e0 con 0x7fd0d80ab6e0 2024-03-28T23:35:52.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.745+0000 7fd0cdffb640 1 -- 172.21.15.161:0/1114711413 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0c80047e0 con 0x7fd0d80ab6e0 2024-03-28T23:35:52.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.745+0000 7fd0df48f640 1 -- 172.21.15.161:0/1114711413 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0d80c6210 con 0x7fd0d80ab6e0 2024-03-28T23:35:52.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.745+0000 7fd0cdffb640 1 -- 172.21.15.161:0/1114711413 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c8004bd0 con 0x7fd0d80ab6e0 2024-03-28T23:35:52.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.746+0000 7fd0df48f640 1 -- 172.21.15.161:0/1114711413 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0d80c4f90 con 0x7fd0d80ab6e0 2024-03-28T23:35:52.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.747+0000 7fd0df48f640 1 -- 172.21.15.161:0/1114711413 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0d804f4b0 con 0x7fd0d80ab6e0 2024-03-28T23:35:52.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.747+0000 7fd0cdffb640 1 -- 172.21.15.161:0/1114711413 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd0c8011070 con 0x7fd0d80ab6e0 2024-03-28T23:35:52.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.747+0000 7fd0cdffb640 1 --2- 172.21.15.161:0/1114711413 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd0bc06b4d0 0x7fd0bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:52.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.748+0000 7fd0cdffb640 1 -- 172.21.15.161:0/1114711413 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5507+0+0 (secure 0 0 0) 0x7fd0c8089d60 con 0x7fd0d80ab6e0 2024-03-28T23:35:52.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.748+0000 7fd0dd22b640 1 --2- 172.21.15.161:0/1114711413 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd0bc06b4d0 0x7fd0bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:52.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.748+0000 7fd0dd22b640 1 --2- 172.21.15.161:0/1114711413 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd0bc06b4d0 0x7fd0bc06d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4036000 tx=0x7fd0d4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:52.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.752+0000 7fd0cdffb640 1 -- 172.21.15.161:0/1114711413 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd0c8058cb0 con 0x7fd0d80ab6e0 2024-03-28T23:35:52.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:52.989+0000 7fd0df48f640 1 -- 172.21.15.161:0/1114711413 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla"]} v 0) -- 0x7fd0d80c0ff0 con 0x7fd0d80ab6e0 2024-03-28T23:35:53.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.024+0000 7fd0cdffb640 1 -- 172.21.15.161:0/1114711413 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla"]}]=0 v351) ==== 149+0+0 (secure 0 0 0) 0x7fd0c80585b0 con 0x7fd0d80ab6e0 2024-03-28T23:35:53.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.026+0000 7fd0df48f640 1 -- 172.21.15.161:0/1114711413 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd0bc06b4d0 msgr2=0x7fd0bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:53.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.026+0000 7fd0df48f640 1 --2- 172.21.15.161:0/1114711413 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd0bc06b4d0 0x7fd0bc06d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4036000 tx=0x7fd0d4008030 comp rx=0 tx=0).stop 2024-03-28T23:35:53.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.026+0000 7fd0df48f640 1 -- 172.21.15.161:0/1114711413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0d80ab6e0 msgr2=0x7fd0d80c5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:53.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.026+0000 7fd0df48f640 1 --2- 172.21.15.161:0/1114711413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0d80ab6e0 0x7fd0d80c5cd0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fd0c800c430 tx=0x7fd0c800c8f0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.026+0000 7fd0df48f640 1 -- 172.21.15.161:0/1114711413 shutdown_connections 2024-03-28T23:35:53.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.026+0000 7fd0df48f640 1 --2- 172.21.15.161:0/1114711413 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd0bc06b4d0 0x7fd0bc06d990 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.026+0000 7fd0df48f640 1 --2- 172.21.15.161:0/1114711413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0d80ab6e0 0x7fd0d80c5cd0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.026+0000 7fd0df48f640 1 --2- 172.21.15.161:0/1114711413 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0d814be70 0x7fd0d81dfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.026+0000 7fd0df48f640 1 --2- 172.21.15.161:0/1114711413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0d80aacd0 0x7fd0d814d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.026+0000 7fd0df48f640 1 -- 172.21.15.161:0/1114711413 >> 172.21.15.161:0/1114711413 conn(0x7fd0d80c6670 msgr2=0x7fd0d8072780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:53.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.026+0000 7fd0df48f640 1 -- 172.21.15.161:0/1114711413 shutdown_connections 2024-03-28T23:35:53.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.026+0000 7fd0df48f640 1 -- 172.21.15.161:0/1114711413 wait complete. 2024-03-28T23:35:53.038 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-28T23:35:53.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.141+0000 7f6243215640 1 Processor -- start 2024-03-28T23:35:53.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.141+0000 7f6243215640 1 -- start start 2024-03-28T23:35:53.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.142+0000 7f6243215640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f623c14f5c0 0x7f623c0c2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:53.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.142+0000 7f6243215640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f623c06cdd0 0x7f623c1524c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:53.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.142+0000 7f6243215640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f623c0c2690 0x7f623c152a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:53.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.142+0000 7f6243215640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f623c0c74d0 con 0x7f623c14f5c0 2024-03-28T23:35:53.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.142+0000 7f6243215640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f623c0c6fa0 con 0x7f623c06cdd0 2024-03-28T23:35:53.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.142+0000 7f6243215640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f623c152030 con 0x7f623c0c2690 2024-03-28T23:35:53.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.142+0000 7f6240fb1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f623c14f5c0 0x7f623c0c2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:53.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.142+0000 7f6240fb1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f623c14f5c0 0x7f623c0c2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57460/0 (socket says 172.21.15.161:57460) 2024-03-28T23:35:53.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.142+0000 7f6240fb1640 1 -- 172.21.15.161:0/2842811598 learned_addr learned my addr 172.21.15.161:0/2842811598 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:53.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.142+0000 7f623bfff640 1 --2- 172.21.15.161:0/2842811598 >> v2:172.21.15.170:3300/0 conn(0x7f623c06cdd0 0x7f623c1524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:53.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.142+0000 7f62417b2640 1 --2- 172.21.15.161:0/2842811598 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f623c0c2690 0x7f623c152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:53.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.142+0000 7f6240fb1640 1 -- 172.21.15.161:0/2842811598 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f623c0c2690 msgr2=0x7f623c152a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:53.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.142+0000 7f6240fb1640 1 --2- 172.21.15.161:0/2842811598 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f623c0c2690 0x7f623c152a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.142+0000 7f6240fb1640 1 -- 172.21.15.161:0/2842811598 >> v2:172.21.15.170:3300/0 conn(0x7f623c06cdd0 msgr2=0x7f623c1524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:53.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.142+0000 7f6240fb1640 1 --2- 172.21.15.161:0/2842811598 >> v2:172.21.15.170:3300/0 conn(0x7f623c06cdd0 0x7f623c1524c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.142+0000 7f6240fb1640 1 -- 172.21.15.161:0/2842811598 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f623c0c2f10 con 0x7f623c14f5c0 2024-03-28T23:35:53.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.143+0000 7f6240fb1640 1 --2- 172.21.15.161:0/2842811598 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f623c14f5c0 0x7f623c0c2150 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f623c06b800 tx=0x7f622c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb0ebb7efc37f2f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:53.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.143+0000 7f623b7fe640 1 -- 172.21.15.161:0/2842811598 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622c035020 con 0x7f623c14f5c0 2024-03-28T23:35:53.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.143+0000 7f623b7fe640 1 -- 172.21.15.161:0/2842811598 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f622c031870 con 0x7f623c14f5c0 2024-03-28T23:35:53.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.143+0000 7f623b7fe640 1 -- 172.21.15.161:0/2842811598 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622c031c40 con 0x7f623c14f5c0 2024-03-28T23:35:53.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.144+0000 7f6243215640 1 -- 172.21.15.161:0/2842811598 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f623c14f5c0 msgr2=0x7f623c0c2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:53.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.144+0000 7f6243215640 1 --2- 172.21.15.161:0/2842811598 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f623c14f5c0 0x7f623c0c2150 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f623c06b800 tx=0x7f622c02b910 comp rx=0 tx=0).stop 2024-03-28T23:35:53.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.144+0000 7f6243215640 1 -- 172.21.15.161:0/2842811598 shutdown_connections 2024-03-28T23:35:53.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.144+0000 7f6243215640 1 --2- 172.21.15.161:0/2842811598 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f623c0c2690 0x7f623c152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.144+0000 7f6243215640 1 --2- 172.21.15.161:0/2842811598 >> v2:172.21.15.170:3300/0 conn(0x7f623c06cdd0 0x7f623c1524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.144+0000 7f6243215640 1 --2- 172.21.15.161:0/2842811598 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f623c14f5c0 0x7f623c0c2150 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.144+0000 7f6243215640 1 -- 172.21.15.161:0/2842811598 >> 172.21.15.161:0/2842811598 conn(0x7f623c0a83f0 msgr2=0x7f623c0a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:53.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.144+0000 7f6243215640 1 -- 172.21.15.161:0/2842811598 shutdown_connections 2024-03-28T23:35:53.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.144+0000 7f6243215640 1 -- 172.21.15.161:0/2842811598 wait complete. 2024-03-28T23:35:53.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.145+0000 7f6243215640 1 Processor -- start 2024-03-28T23:35:53.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.145+0000 7f6243215640 1 -- start start 2024-03-28T23:35:53.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.146+0000 7f6243215640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f623c06cdd0 0x7f623c1e6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:53.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.146+0000 7f6243215640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f623c0c2690 0x7f623c1e9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:53.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.146+0000 7f6243215640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f623c14f5c0 0x7f623c0722f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:53.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.146+0000 7f6243215640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f623c06c140 con 0x7f623c0c2690 2024-03-28T23:35:53.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.146+0000 7f6243215640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f623c14f9a0 con 0x7f623c14f5c0 2024-03-28T23:35:53.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.146+0000 7f6243215640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f623c06c3e0 con 0x7f623c06cdd0 2024-03-28T23:35:53.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.146+0000 7f623bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f623c0c2690 0x7f623c1e9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:53.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.146+0000 7f623bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f623c0c2690 0x7f623c1e9580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57478/0 (socket says 172.21.15.161:57478) 2024-03-28T23:35:53.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.146+0000 7f623bfff640 1 -- 172.21.15.161:0/4011698680 learned_addr learned my addr 172.21.15.161:0/4011698680 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:53.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.146+0000 7f6240fb1640 1 --2- 172.21.15.161:0/4011698680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f623c06cdd0 0x7f623c1e6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:53.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.146+0000 7f62417b2640 1 --2- 172.21.15.161:0/4011698680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f623c14f5c0 0x7f623c0722f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:53.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.146+0000 7f623bfff640 1 -- 172.21.15.161:0/4011698680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f623c06cdd0 msgr2=0x7f623c1e6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:53.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.146+0000 7f623bfff640 1 --2- 172.21.15.161:0/4011698680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f623c06cdd0 0x7f623c1e6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.146+0000 7f623bfff640 1 -- 172.21.15.161:0/4011698680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f623c14f5c0 msgr2=0x7f623c0722f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:53.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.146+0000 7f623bfff640 1 --2- 172.21.15.161:0/4011698680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f623c14f5c0 0x7f623c0722f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.146+0000 7f623bfff640 1 -- 172.21.15.161:0/4011698680 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f623c14b090 con 0x7f623c0c2690 2024-03-28T23:35:53.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.146+0000 7f623bfff640 1 --2- 172.21.15.161:0/4011698680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f623c0c2690 0x7f623c1e9580 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f623000d7d0 tx=0x7f623000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:53.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.146+0000 7f62397fa640 1 -- 172.21.15.161:0/4011698680 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6230018030 con 0x7f623c0c2690 2024-03-28T23:35:53.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.146+0000 7f62397fa640 1 -- 172.21.15.161:0/4011698680 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6230004020 con 0x7f623c0c2690 2024-03-28T23:35:53.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.146+0000 7f62397fa640 1 -- 172.21.15.161:0/4011698680 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6230004230 con 0x7f623c0c2690 2024-03-28T23:35:53.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.146+0000 7f6243215640 1 -- 172.21.15.161:0/4011698680 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f623c14b370 con 0x7f623c0c2690 2024-03-28T23:35:53.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.147+0000 7f6243215640 1 -- 172.21.15.161:0/4011698680 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f623c149900 con 0x7f623c0c2690 2024-03-28T23:35:53.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.148+0000 7f6243215640 1 -- 172.21.15.161:0/4011698680 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f623c04f4b0 con 0x7f623c0c2690 2024-03-28T23:35:53.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.148+0000 7f62397fa640 1 -- 172.21.15.161:0/4011698680 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f623001b070 con 0x7f623c0c2690 2024-03-28T23:35:53.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.148+0000 7f62397fa640 1 --2- 172.21.15.161:0/4011698680 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f622806b4d0 0x7f622806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:53.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.148+0000 7f62397fa640 1 -- 172.21.15.161:0/4011698680 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5507+0+0 (secure 0 0 0) 0x7f623008a120 con 0x7f623c0c2690 2024-03-28T23:35:53.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.148+0000 7f6240fb1640 1 --2- 172.21.15.161:0/4011698680 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f622806b4d0 0x7f622806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:53.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.149+0000 7f6240fb1640 1 --2- 172.21.15.161:0/4011698680 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f622806b4d0 0x7f622806d990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f622c036000 tx=0x7f622c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:53.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.153+0000 7f62397fa640 1 -- 172.21.15.161:0/4011698680 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6230058aa0 con 0x7f623c0c2690 2024-03-28T23:35:53.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.325+0000 7f6243215640 1 -- 172.21.15.161:0/4011698680 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f623c06b830 con 0x7f623c0c2690 2024-03-28T23:35:53.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.328+0000 7f62397fa640 1 -- 172.21.15.161:0/4011698680 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v51)=0 v51) ==== 122+0+0 (secure 0 0 0) 0x7f623005cb40 con 0x7f623c0c2690 2024-03-28T23:35:53.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.330+0000 7f6243215640 1 -- 172.21.15.161:0/4011698680 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f622806b4d0 msgr2=0x7f622806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:53.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.330+0000 7f6243215640 1 --2- 172.21.15.161:0/4011698680 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f622806b4d0 0x7f622806d990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f622c036000 tx=0x7f622c03c040 comp rx=0 tx=0).stop 2024-03-28T23:35:53.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.330+0000 7f6243215640 1 -- 172.21.15.161:0/4011698680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f623c0c2690 msgr2=0x7f623c1e9580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:53.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.330+0000 7f6243215640 1 --2- 172.21.15.161:0/4011698680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f623c0c2690 0x7f623c1e9580 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f623000d7d0 tx=0x7f623000dc90 comp rx=0 tx=0).stop 2024-03-28T23:35:53.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.330+0000 7f6243215640 1 -- 172.21.15.161:0/4011698680 shutdown_connections 2024-03-28T23:35:53.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.330+0000 7f6243215640 1 --2- 172.21.15.161:0/4011698680 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f622806b4d0 0x7f622806d990 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.330+0000 7f6243215640 1 --2- 172.21.15.161:0/4011698680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f623c0c2690 0x7f623c1e9580 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.330+0000 7f6243215640 1 --2- 172.21.15.161:0/4011698680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f623c14f5c0 0x7f623c0722f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.330+0000 7f6243215640 1 --2- 172.21.15.161:0/4011698680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f623c06cdd0 0x7f623c1e6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.330+0000 7f6243215640 1 -- 172.21.15.161:0/4011698680 >> 172.21.15.161:0/4011698680 conn(0x7f623c0a83f0 msgr2=0x7f623c14e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:53.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.330+0000 7f6243215640 1 -- 172.21.15.161:0/4011698680 shutdown_connections 2024-03-28T23:35:53.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.330+0000 7f6243215640 1 -- 172.21.15.161:0/4011698680 wait complete. 2024-03-28T23:35:53.343 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-28T23:35:53.343 DEBUG:teuthology.orchestra.run.smithi161:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:35:53.384 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:35:53.429 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-28T23:35:53.429 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:35:53.429 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:35:53.444 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20793 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-28T23:35:53.444 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:35:53.444 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:35:53.462 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056374,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19690 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-28T23:35:53.462 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-28T23:35:53.462 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:35:53.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.583+0000 7f32964e1640 1 Processor -- start 2024-03-28T23:35:53.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.583+0000 7f32964e1640 1 -- start start 2024-03-28T23:35:53.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.584+0000 7f32964e1640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f32900a8fc0 0x7f3290153270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:53.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.584+0000 7f32964e1640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32901567c0 con 0x7f32900aacb0 2024-03-28T23:35:53.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.584+0000 7f32964e1640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3290156aa0 con 0x7f3290150600 2024-03-28T23:35:53.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.584+0000 7f32964e1640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f3290156570 con 0x7f32900a8fc0 2024-03-28T23:35:53.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.584+0000 7f3294a7e640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f32900aacb0 0x7f32901537b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:54302/0 (socket says 172.21.15.161:54302) 2024-03-28T23:35:53.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.584+0000 7f3294a7e640 1 -- 172.21.15.161:0/1645769824 learned_addr learned my addr 172.21.15.161:0/1645769824 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:53.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.584+0000 7f328effd640 1 -- 172.21.15.161:0/1645769824 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3547862580 0 0) 0x7f32901567c0 con 0x7f32900aacb0 2024-03-28T23:35:53.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.584+0000 7f328f7fe640 1 --2- 172.21.15.161:0/1645769824 >> v2:172.21.15.170:3300/0 conn(0x7f32900a8fc0 0x7f3290153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:53.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.584+0000 7f328effd640 1 -- 172.21.15.161:0/1645769824 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3278003520 con 0x7f32900aacb0 2024-03-28T23:35:53.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.584+0000 7f328effd640 1 -- 172.21.15.161:0/1645769824 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2020202227 0 0) 0x7f3278003520 con 0x7f32900aacb0 2024-03-28T23:35:53.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.584+0000 7f328effd640 1 -- 172.21.15.161:0/1645769824 >> v2:172.21.15.170:3300/0 conn(0x7f32900a8fc0 msgr2=0x7f3290153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:53.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.584+0000 7f328effd640 1 --2- 172.21.15.161:0/1645769824 >> v2:172.21.15.170:3300/0 conn(0x7f32900a8fc0 0x7f3290153270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.584+0000 7f328effd640 1 -- 172.21.15.161:0/1645769824 >> v1:172.21.15.170:6789/0 conn(0x7f3290150600 legacy=0x7f32900a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:53.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.584+0000 7f328effd640 1 -- 172.21.15.161:0/1645769824 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3290156d80 con 0x7f32900aacb0 2024-03-28T23:35:53.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.584+0000 7f328effd640 1 -- 172.21.15.161:0/1645769824 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f32880019f0 con 0x7f32900aacb0 2024-03-28T23:35:53.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.584+0000 7f328effd640 1 -- 172.21.15.161:0/1645769824 >> v1:172.21.15.161:6789/0 conn(0x7f32900aacb0 legacy=0x7f32901537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:53.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.584+0000 7f328effd640 1 --2- 172.21.15.161:0/1645769824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3278003a90 0x7f3278003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:53.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.584+0000 7f328effd640 1 --2- 172.21.15.161:0/1645769824 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3278004800 0x7f3278004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:53.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.585+0000 7f328effd640 1 --2- 172.21.15.161:0/1645769824 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3278005290 0x7f327800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:53.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.585+0000 7f328effd640 1 -- 172.21.15.161:0/1645769824 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3290156570 con 0x7f3278005290 2024-03-28T23:35:53.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.585+0000 7f328effd640 1 -- 172.21.15.161:0/1645769824 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f327800c1e0 con 0x7f3278003a90 2024-03-28T23:35:53.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.585+0000 7f328effd640 1 -- 172.21.15.161:0/1645769824 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f327800c3e0 con 0x7f3278004800 2024-03-28T23:35:53.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.585+0000 7f3294a7e640 1 --2- 172.21.15.161:0/1645769824 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3278005290 0x7f327800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:53.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.585+0000 7f328ffff640 1 --2- 172.21.15.161:0/1645769824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3278003a90 0x7f3278003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:53.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.585+0000 7f328f7fe640 1 --2- 172.21.15.161:0/1645769824 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3278004800 0x7f3278004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:53.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.585+0000 7f3294a7e640 1 -- 172.21.15.161:0/1645769824 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3278004800 msgr2=0x7f3278004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:53.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.585+0000 7f3294a7e640 1 --2- 172.21.15.161:0/1645769824 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3278004800 0x7f3278004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.585+0000 7f3294a7e640 1 -- 172.21.15.161:0/1645769824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3278003a90 msgr2=0x7f3278003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:53.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.585+0000 7f3294a7e640 1 --2- 172.21.15.161:0/1645769824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3278003a90 0x7f3278003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.585+0000 7f3294a7e640 1 -- 172.21.15.161:0/1645769824 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3278003520 con 0x7f3278005290 2024-03-28T23:35:53.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.586+0000 7f3294a7e640 1 -- 172.21.15.161:0/1645769824 reap_dead start 2024-03-28T23:35:53.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.586+0000 7f3294a7e640 1 --2- 172.21.15.161:0/1645769824 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3278005290 0x7f327800b9c0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f328800a840 tx=0x7f3278005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f9217ecc27b9052 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:53.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.586+0000 7f328effd640 1 -- 172.21.15.161:0/1645769824 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328800a460 con 0x7f3278005290 2024-03-28T23:35:53.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.586+0000 7f328effd640 1 -- 172.21.15.161:0/1645769824 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3288034650 con 0x7f3278005290 2024-03-28T23:35:53.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.586+0000 7f328effd640 1 -- 172.21.15.161:0/1645769824 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3288034a40 con 0x7f3278005290 2024-03-28T23:35:53.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.586+0000 7f32964e1640 1 -- 172.21.15.161:0/1645769824 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3278005290 msgr2=0x7f327800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:53.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.586+0000 7f32964e1640 1 --2- 172.21.15.161:0/1645769824 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3278005290 0x7f327800b9c0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f328800a840 tx=0x7f3278005830 comp rx=0 tx=0).stop 2024-03-28T23:35:53.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.586+0000 7f32964e1640 1 -- 172.21.15.161:0/1645769824 shutdown_connections 2024-03-28T23:35:53.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.586+0000 7f32964e1640 1 --2- 172.21.15.161:0/1645769824 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3278005290 0x7f327800b9c0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.586+0000 7f32964e1640 1 -- 172.21.15.161:0/1645769824 >> 172.21.15.161:0/1645769824 conn(0x7f32900c6670 msgr2=0x7f32900c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:53.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.586+0000 7f32964e1640 1 -- 172.21.15.161:0/1645769824 shutdown_connections 2024-03-28T23:35:53.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.586+0000 7f32964e1640 1 -- 172.21.15.161:0/1645769824 wait complete. 2024-03-28T23:35:53.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.587+0000 7f32964e1640 1 Processor -- start 2024-03-28T23:35:53.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.588+0000 7f32964e1640 1 -- start start 2024-03-28T23:35:53.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.588+0000 7f32964e1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3278005290 0x7f329014aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:53.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.588+0000 7f32964e1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32900c06b0 0x7f3290151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:53.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.588+0000 7f32964e1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32900c0160 0x7f32901e94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:53.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.588+0000 7f32964e1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f32901509e0 con 0x7f32900c0160 2024-03-28T23:35:53.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.588+0000 7f32964e1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3290157340 con 0x7f3278005290 2024-03-28T23:35:53.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.588+0000 7f32964e1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f32900aa9e0 con 0x7f32900c06b0 2024-03-28T23:35:53.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.588+0000 7f3294a7e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32900c0160 0x7f32901e94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:53.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.588+0000 7f3294a7e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32900c0160 0x7f32901e94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57496/0 (socket says 172.21.15.161:57496) 2024-03-28T23:35:53.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.588+0000 7f3294a7e640 1 -- 172.21.15.161:0/1837316498 learned_addr learned my addr 172.21.15.161:0/1837316498 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:53.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.588+0000 7f328f7fe640 1 --2- 172.21.15.161:0/1837316498 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32900c06b0 0x7f3290151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:53.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.588+0000 7f328ffff640 1 --2- 172.21.15.161:0/1837316498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3278005290 0x7f329014aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:53.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.589+0000 7f3294a7e640 1 -- 172.21.15.161:0/1837316498 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32900c06b0 msgr2=0x7f3290151f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:53.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.589+0000 7f3294a7e640 1 --2- 172.21.15.161:0/1837316498 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32900c06b0 0x7f3290151f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.589+0000 7f3294a7e640 1 -- 172.21.15.161:0/1837316498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3278005290 msgr2=0x7f329014aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:53.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.589+0000 7f3294a7e640 1 --2- 172.21.15.161:0/1837316498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3278005290 0x7f329014aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.589+0000 7f3294a7e640 1 -- 172.21.15.161:0/1837316498 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32901e8ce0 con 0x7f32900c0160 2024-03-28T23:35:53.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.589+0000 7f3294a7e640 1 --2- 172.21.15.161:0/1837316498 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32900c0160 0x7f32901e94a0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f328800a960 tx=0x7f3290156c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:53.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.589+0000 7f328cff9640 1 -- 172.21.15.161:0/1837316498 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3288039040 con 0x7f32900c0160 2024-03-28T23:35:53.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.589+0000 7f328cff9640 1 -- 172.21.15.161:0/1837316498 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3288043070 con 0x7f32900c0160 2024-03-28T23:35:53.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.589+0000 7f32964e1640 1 -- 172.21.15.161:0/1837316498 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32901df6e0 con 0x7f32900c0160 2024-03-28T23:35:53.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.589+0000 7f328cff9640 1 -- 172.21.15.161:0/1837316498 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3288004d60 con 0x7f32900c0160 2024-03-28T23:35:53.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.589+0000 7f32964e1640 1 -- 172.21.15.161:0/1837316498 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32901e0080 con 0x7f32900c0160 2024-03-28T23:35:53.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.590+0000 7f32964e1640 1 -- 172.21.15.161:0/1837316498 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f329004f4b0 con 0x7f32900c0160 2024-03-28T23:35:53.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.590+0000 7f328cff9640 1 -- 172.21.15.161:0/1837316498 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3288034650 con 0x7f32900c0160 2024-03-28T23:35:53.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.591+0000 7f328cff9640 1 --2- 172.21.15.161:0/1837316498 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f327c06b4d0 0x7f327c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:53.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.591+0000 7f328ffff640 1 --2- 172.21.15.161:0/1837316498 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f327c06b4d0 0x7f327c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:53.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.591+0000 7f328cff9640 1 -- 172.21.15.161:0/1837316498 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5507+0+0 (secure 0 0 0) 0x7f32880abda0 con 0x7f32900c0160 2024-03-28T23:35:53.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.591+0000 7f328ffff640 1 --2- 172.21.15.161:0/1837316498 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f327c06b4d0 0x7f327c06d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f3280000c00 tx=0x7f3280009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:53.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.595+0000 7f328cff9640 1 -- 172.21.15.161:0/1837316498 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f328807ad70 con 0x7f32900c0160 2024-03-28T23:35:53.805 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.804+0000 7f32964e1640 1 -- 172.21.15.161:0/1837316498 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f329014ccd0 con 0x7f32900c0160 2024-03-28T23:35:53.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.804+0000 7f328cff9640 1 -- 172.21.15.161:0/1837316498 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) ==== 95+0+3883 (secure 0 0 0) 0x7f328807a670 con 0x7f32900c0160 2024-03-28T23:35:53.806 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:53.806 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":136,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7870,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2828648906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":2828648906},{"type":"v1","addr":"172.21.15.170:6835","nonce":2828648906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":132},{"gid":7999,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/4271599793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":4271599793},{"type":"v1","addr":"172.21.15.170:6834","nonce":4271599793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":136},{"gid":8409,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/3037890643","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3037890643},{"type":"v1","addr":"172.21.15.161:6837","nonce":3037890643}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120},{"gid":8622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/819329318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":819329318},{"type":"v1","addr":"172.21.15.161:6836","nonce":819329318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[]} 2024-03-28T23:35:53.807 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 136 2024-03-28T23:35:53.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.806+0000 7f32964e1640 1 -- 172.21.15.161:0/1837316498 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f327c06b4d0 msgr2=0x7f327c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:53.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.807+0000 7f32964e1640 1 --2- 172.21.15.161:0/1837316498 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f327c06b4d0 0x7f327c06d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f3280000c00 tx=0x7f3280009040 comp rx=0 tx=0).stop 2024-03-28T23:35:53.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.807+0000 7f32964e1640 1 -- 172.21.15.161:0/1837316498 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32900c0160 msgr2=0x7f32901e94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:53.809 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.807+0000 7f32964e1640 1 --2- 172.21.15.161:0/1837316498 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32900c0160 0x7f32901e94a0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f328800a960 tx=0x7f3290156c70 comp rx=0 tx=0).stop 2024-03-28T23:35:53.809 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.807+0000 7f32964e1640 1 -- 172.21.15.161:0/1837316498 shutdown_connections 2024-03-28T23:35:53.809 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.807+0000 7f32964e1640 1 --2- 172.21.15.161:0/1837316498 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f327c06b4d0 0x7f327c06d990 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.809 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.807+0000 7f32964e1640 1 --2- 172.21.15.161:0/1837316498 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32900c0160 0x7f32901e94a0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.809 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.807+0000 7f32964e1640 1 --2- 172.21.15.161:0/1837316498 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32900c06b0 0x7f3290151f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.809 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.807+0000 7f32964e1640 1 --2- 172.21.15.161:0/1837316498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3278005290 0x7f329014aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.809 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.807+0000 7f32964e1640 1 -- 172.21.15.161:0/1837316498 >> 172.21.15.161:0/1837316498 conn(0x7f32900c6670 msgr2=0x7f329014dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:53.809 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.807+0000 7f32964e1640 1 -- 172.21.15.161:0/1837316498 shutdown_connections 2024-03-28T23:35:53.809 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.807+0000 7f32964e1640 1 -- 172.21.15.161:0/1837316498 wait complete. 2024-03-28T23:35:53.820 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T23:35:53.820 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-28T23:35:53.820 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T23:35:53.820 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:35:53.821 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T23:35:53.821 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-28T23:35:53.821 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T23:35:53.821 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:35:53.821 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-28T23:35:53.821 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-28T23:35:53.821 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-28T23:35:53.821 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:35:53.822 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-28T23:35:53.822 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-28T23:35:53.822 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-28T23:35:53.822 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:35:53.830 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T23:35:53.830 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T23:35:53.831 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T23:35:53.831 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T23:35:53.833 INFO:tasks.ceph.mds.b:Stopped 2024-03-28T23:35:53.833 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-28T23:35:53.834 INFO:tasks.ceph.mds.d:Stopped 2024-03-28T23:35:53.834 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-28T23:35:53.873 INFO:tasks.ceph.mds.a:Started 2024-03-28T23:35:53.873 DEBUG:teuthology.parallel:result is None 2024-03-28T23:35:53.873 INFO:tasks.ceph.mds.c:Started 2024-03-28T23:35:53.874 DEBUG:teuthology.parallel:result is None 2024-03-28T23:35:53.876 INFO:tasks.ceph.mds.b:Started 2024-03-28T23:35:53.876 DEBUG:teuthology.parallel:result is None 2024-03-28T23:35:53.877 INFO:tasks.ceph.mds.d:Started 2024-03-28T23:35:53.877 DEBUG:teuthology.parallel:result is None 2024-03-28T23:35:53.877 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-28T23:35:53.949 INFO:tasks.ceph.mds.c.smithi161.stdout:starting mds.c at 2024-03-28T23:35:53.949 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-28T23:35:53.958 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2024-03-28T23:35:53.958 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2024-03-28T23:35:53.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.982+0000 7fe38e43b640 1 Processor -- start 2024-03-28T23:35:53.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.982+0000 7fe38e43b640 1 -- start start 2024-03-28T23:35:53.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.983+0000 7fe38e43b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe388152a80 0x7fe388155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:53.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.983+0000 7fe38e43b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe38806d9f0 0x7fe3880c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:53.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.983+0000 7fe38e43b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3880c98e0 con 0x7fe38806cb80 2024-03-28T23:35:53.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.983+0000 7fe38e43b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fe388156250 con 0x7fe388152a80 2024-03-28T23:35:53.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.983+0000 7fe38e43b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe38806c6f0 con 0x7fe38806d9f0 2024-03-28T23:35:53.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.984+0000 7fe387fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe388152a80 0x7fe388155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:53.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.984+0000 7fe3877fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe38806d9f0 0x7fe3880c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:53.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.984+0000 7fe3877fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe38806d9f0 0x7fe3880c93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:53826/0 (socket says 172.21.15.161:53826) 2024-03-28T23:35:53.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.984+0000 7fe3877fe640 1 -- 172.21.15.161:0/1452247826 learned_addr learned my addr 172.21.15.161:0/1452247826 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:53.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.984+0000 7fe3877fe640 1 -- 172.21.15.161:0/1452247826 >> v2:172.21.15.170:3300/0 conn(0x7fe388152a80 msgr2=0x7fe388155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:53.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.984+0000 7fe3877fe640 1 --2- 172.21.15.161:0/1452247826 >> v2:172.21.15.170:3300/0 conn(0x7fe388152a80 0x7fe388155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.984+0000 7fe3877fe640 1 -- 172.21.15.161:0/1452247826 >> v1:172.21.15.170:6789/0 conn(0x7fe38806cb80 legacy=0x7fe3880c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:53.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.984+0000 7fe3877fe640 1 -- 172.21.15.161:0/1452247826 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe38806c3c0 con 0x7fe38806d9f0 2024-03-28T23:35:53.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.985+0000 7fe3877fe640 1 --2- 172.21.15.161:0/1452247826 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe38806d9f0 0x7fe3880c93a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe37c009870 tx=0x7fe37c02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24917bc6846503b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:53.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.985+0000 7fe386ffd640 1 -- 172.21.15.161:0/1452247826 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37c0317c0 con 0x7fe38806d9f0 2024-03-28T23:35:53.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.985+0000 7fe386ffd640 1 -- 172.21.15.161:0/1452247826 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe37c037070 con 0x7fe38806d9f0 2024-03-28T23:35:53.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.985+0000 7fe386ffd640 1 -- 172.21.15.161:0/1452247826 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37c031b80 con 0x7fe38806d9f0 2024-03-28T23:35:53.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.985+0000 7fe38e43b640 1 -- 172.21.15.161:0/1452247826 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe38806d9f0 msgr2=0x7fe3880c93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:53.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.985+0000 7fe38e43b640 1 --2- 172.21.15.161:0/1452247826 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe38806d9f0 0x7fe3880c93a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe37c009870 tx=0x7fe37c02b860 comp rx=0 tx=0).stop 2024-03-28T23:35:53.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.985+0000 7fe38e43b640 1 -- 172.21.15.161:0/1452247826 shutdown_connections 2024-03-28T23:35:53.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.985+0000 7fe38e43b640 1 --2- 172.21.15.161:0/1452247826 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe38806d9f0 0x7fe3880c93a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.985+0000 7fe38e43b640 1 --2- 172.21.15.161:0/1452247826 >> v2:172.21.15.170:3300/0 conn(0x7fe388152a80 0x7fe388155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.985+0000 7fe38e43b640 1 -- 172.21.15.161:0/1452247826 >> 172.21.15.161:0/1452247826 conn(0x7fe3880ac720 msgr2=0x7fe3880acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:53.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.985+0000 7fe38e43b640 1 -- 172.21.15.161:0/1452247826 shutdown_connections 2024-03-28T23:35:53.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.985+0000 7fe38e43b640 1 -- 172.21.15.161:0/1452247826 wait complete. 2024-03-28T23:35:53.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.986+0000 7fe38e43b640 1 Processor -- start 2024-03-28T23:35:53.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.987+0000 7fe38e43b640 1 -- start start 2024-03-28T23:35:53.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.987+0000 7fe38e43b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe38806cb80 0x7fe388156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:53.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.987+0000 7fe38e43b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe38806d9f0 0x7fe388157c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:53.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.987+0000 7fe38e43b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe388152a80 0x7fe3881ebee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:53.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.987+0000 7fe38e43b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe3880c58b0 con 0x7fe38806d9f0 2024-03-28T23:35:53.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.987+0000 7fe38e43b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe3880c55d0 con 0x7fe388152a80 2024-03-28T23:35:53.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.987+0000 7fe38e43b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe3880c7bd0 con 0x7fe38806cb80 2024-03-28T23:35:53.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.987+0000 7fe3877fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe38806d9f0 0x7fe388157c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:53.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.987+0000 7fe3877fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe38806d9f0 0x7fe388157c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57546/0 (socket says 172.21.15.161:57546) 2024-03-28T23:35:53.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.987+0000 7fe3877fe640 1 -- 172.21.15.161:0/4162112415 learned_addr learned my addr 172.21.15.161:0/4162112415 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:53.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.987+0000 7fe387fff640 1 --2- 172.21.15.161:0/4162112415 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe38806cb80 0x7fe388156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:53.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.987+0000 7fe38c9d8640 1 --2- 172.21.15.161:0/4162112415 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe388152a80 0x7fe3881ebee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:53.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.988+0000 7fe3877fe640 1 -- 172.21.15.161:0/4162112415 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe38806cb80 msgr2=0x7fe388156d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:53.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.988+0000 7fe3877fe640 1 --2- 172.21.15.161:0/4162112415 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe38806cb80 0x7fe388156d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.988+0000 7fe3877fe640 1 -- 172.21.15.161:0/4162112415 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe388152a80 msgr2=0x7fe3881ebee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:53.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.988+0000 7fe3877fe640 1 --2- 172.21.15.161:0/4162112415 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe388152a80 0x7fe3881ebee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:53.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.988+0000 7fe3877fe640 1 -- 172.21.15.161:0/4162112415 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3880c4890 con 0x7fe38806d9f0 2024-03-28T23:35:53.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.988+0000 7fe3877fe640 1 --2- 172.21.15.161:0/4162112415 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe38806d9f0 0x7fe388157c20 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fe37c000c00 tx=0x7fe37c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:53.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.988+0000 7fe384ff9640 1 -- 172.21.15.161:0/4162112415 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37c032650 con 0x7fe38806d9f0 2024-03-28T23:35:53.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.988+0000 7fe384ff9640 1 -- 172.21.15.161:0/4162112415 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe37c0327f0 con 0x7fe38806d9f0 2024-03-28T23:35:53.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.988+0000 7fe38e43b640 1 -- 172.21.15.161:0/4162112415 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe388158b10 con 0x7fe38806d9f0 2024-03-28T23:35:53.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.988+0000 7fe384ff9640 1 -- 172.21.15.161:0/4162112415 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37c032be0 con 0x7fe38806d9f0 2024-03-28T23:35:53.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.988+0000 7fe38e43b640 1 -- 172.21.15.161:0/4162112415 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3881590d0 con 0x7fe38806d9f0 2024-03-28T23:35:53.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.989+0000 7fe38e43b640 1 -- 172.21.15.161:0/4162112415 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe388158940 con 0x7fe38806d9f0 2024-03-28T23:35:53.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.990+0000 7fe384ff9640 1 -- 172.21.15.161:0/4162112415 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe37c04c060 con 0x7fe38806d9f0 2024-03-28T23:35:53.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.990+0000 7fe384ff9640 1 --2- 172.21.15.161:0/4162112415 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe37406b4d0 0x7fe37406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:53.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.990+0000 7fe387fff640 1 --2- 172.21.15.161:0/4162112415 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe37406b4d0 0x7fe37406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:53.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.990+0000 7fe384ff9640 1 -- 172.21.15.161:0/4162112415 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5507+0+0 (secure 0 0 0) 0x7fe37c0abb90 con 0x7fe38806d9f0 2024-03-28T23:35:53.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.991+0000 7fe387fff640 1 --2- 172.21.15.161:0/4162112415 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe37406b4d0 0x7fe37406d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fe378009aa0 tx=0x7fe378009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:53.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:53.995+0000 7fe384ff9640 1 -- 172.21.15.161:0/4162112415 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe37c040070 con 0x7fe38806d9f0 2024-03-28T23:35:54.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:54.180+0000 7fe38e43b640 1 -- 172.21.15.161:0/4162112415 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fe38804f4b0 con 0x7fe38806d9f0 2024-03-28T23:35:55.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.036+0000 7fe384ff9640 1 -- 172.21.15.161:0/4162112415 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v148) ==== 97+0+0 (secure 0 0 0) 0x7fe37c07a480 con 0x7fe38806d9f0 2024-03-28T23:35:55.037 INFO:teuthology.orchestra.run.smithi161.stderr: removed all blocklist entries 2024-03-28T23:35:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.039+0000 7fe38e43b640 1 -- 172.21.15.161:0/4162112415 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe37406b4d0 msgr2=0x7fe37406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.039+0000 7fe38e43b640 1 --2- 172.21.15.161:0/4162112415 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe37406b4d0 0x7fe37406d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fe378009aa0 tx=0x7fe378009190 comp rx=0 tx=0).stop 2024-03-28T23:35:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.039+0000 7fe38e43b640 1 -- 172.21.15.161:0/4162112415 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe38806d9f0 msgr2=0x7fe388157c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.039+0000 7fe38e43b640 1 --2- 172.21.15.161:0/4162112415 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe38806d9f0 0x7fe388157c20 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fe37c000c00 tx=0x7fe37c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.039+0000 7fe38e43b640 1 -- 172.21.15.161:0/4162112415 shutdown_connections 2024-03-28T23:35:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.039+0000 7fe38e43b640 1 --2- 172.21.15.161:0/4162112415 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe37406b4d0 0x7fe37406d990 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.039+0000 7fe38e43b640 1 --2- 172.21.15.161:0/4162112415 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe38806d9f0 0x7fe388157c20 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.039+0000 7fe38e43b640 1 --2- 172.21.15.161:0/4162112415 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe388152a80 0x7fe3881ebee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.039+0000 7fe38e43b640 1 --2- 172.21.15.161:0/4162112415 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe38806cb80 0x7fe388156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.039+0000 7fe38e43b640 1 -- 172.21.15.161:0/4162112415 >> 172.21.15.161:0/4162112415 conn(0x7fe3880ac720 msgr2=0x7fe3880abab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:55.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.039+0000 7fe38e43b640 1 -- 172.21.15.161:0/4162112415 shutdown_connections 2024-03-28T23:35:55.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.039+0000 7fe38e43b640 1 -- 172.21.15.161:0/4162112415 wait complete. 2024-03-28T23:35:55.052 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-28T23:35:55.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.157+0000 7f7a6b6c0640 1 Processor -- start 2024-03-28T23:35:55.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.157+0000 7f7a6b6c0640 1 -- start start 2024-03-28T23:35:55.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a6b6c0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a64150600 0x7f7a640a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:55.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a6b6c0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a641567c0 con 0x7f7a640aacb0 2024-03-28T23:35:55.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a6b6c0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a64156aa0 con 0x7f7a640a8fc0 2024-03-28T23:35:55.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a6b6c0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7a64156570 con 0x7f7a64150600 2024-03-28T23:35:55.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a69c5d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f7a640aacb0 0x7f7a64153980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:54328/0 (socket says 172.21.15.161:54328) 2024-03-28T23:35:55.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a69c5d640 1 -- 172.21.15.161:0/3060083744 learned_addr learned my addr 172.21.15.161:0/3060083744 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:55.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a6945c640 1 --2- 172.21.15.161:0/3060083744 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a64150600 0x7f7a640a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:55.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a5bfff640 1 -- 172.21.15.161:0/3060083744 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3611441010 0 0) 0x7f7a641567c0 con 0x7f7a640aacb0 2024-03-28T23:35:55.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a5bfff640 1 -- 172.21.15.161:0/3060083744 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a50003520 con 0x7f7a640aacb0 2024-03-28T23:35:55.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a5bfff640 1 -- 172.21.15.161:0/3060083744 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4216123648 0 0) 0x7f7a64156aa0 con 0x7f7a640a8fc0 2024-03-28T23:35:55.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a5bfff640 1 -- 172.21.15.161:0/3060083744 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a641567c0 con 0x7f7a640a8fc0 2024-03-28T23:35:55.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a5bfff640 1 -- 172.21.15.161:0/3060083744 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3278702212 0 0) 0x7f7a50003520 con 0x7f7a640aacb0 2024-03-28T23:35:55.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a5bfff640 1 -- 172.21.15.161:0/3060083744 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a64150600 msgr2=0x7f7a640a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:55.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a5bfff640 1 --2- 172.21.15.161:0/3060083744 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a64150600 0x7f7a640a9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a5bfff640 1 -- 172.21.15.161:0/3060083744 >> v1:172.21.15.170:6789/0 conn(0x7f7a640a8fc0 legacy=0x7f7a64153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:55.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a5bfff640 1 -- 172.21.15.161:0/3060083744 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a64156d80 con 0x7f7a640aacb0 2024-03-28T23:35:55.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a5bfff640 1 -- 172.21.15.161:0/3060083744 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f7a5c0019f0 con 0x7f7a640aacb0 2024-03-28T23:35:55.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a5bfff640 1 -- 172.21.15.161:0/3060083744 >> v1:172.21.15.161:6789/0 conn(0x7f7a640aacb0 legacy=0x7f7a64153980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:55.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a5bfff640 1 --2- 172.21.15.161:0/3060083744 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a50003c50 0x7f7a50004070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:55.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a5bfff640 1 --2- 172.21.15.161:0/3060083744 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a50004800 0x7f7a50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:55.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a5bfff640 1 --2- 172.21.15.161:0/3060083744 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a64150600 0x7f7a5000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:55.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a5bfff640 1 -- 172.21.15.161:0/3060083744 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7a64156570 con 0x7f7a50003c50 2024-03-28T23:35:55.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a5bfff640 1 -- 172.21.15.161:0/3060083744 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7a500055d0 con 0x7f7a50004800 2024-03-28T23:35:55.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a5bfff640 1 -- 172.21.15.161:0/3060083744 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7a5000bf00 con 0x7f7a64150600 2024-03-28T23:35:55.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.158+0000 7f7a6945c640 1 --2- 172.21.15.161:0/3060083744 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a50003c50 0x7f7a50004070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:55.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.159+0000 7f7a68c5b640 1 --2- 172.21.15.161:0/3060083744 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a50004800 0x7f7a50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:55.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.159+0000 7f7a69c5d640 1 --2- 172.21.15.161:0/3060083744 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a64150600 0x7f7a5000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:55.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.159+0000 7f7a6945c640 1 -- 172.21.15.161:0/3060083744 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a64150600 msgr2=0x7f7a5000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:55.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.159+0000 7f7a6945c640 1 --2- 172.21.15.161:0/3060083744 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a64150600 0x7f7a5000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.159+0000 7f7a6945c640 1 -- 172.21.15.161:0/3060083744 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a50004800 msgr2=0x7f7a50004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:55.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.159+0000 7f7a6945c640 1 --2- 172.21.15.161:0/3060083744 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a50004800 0x7f7a50004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.159+0000 7f7a6945c640 1 -- 172.21.15.161:0/3060083744 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a50003520 con 0x7f7a50003c50 2024-03-28T23:35:55.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.159+0000 7f7a69c5d640 1 --2- 172.21.15.161:0/3060083744 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a64150600 0x7f7a5000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:55.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.159+0000 7f7a6945c640 1 --2- 172.21.15.161:0/3060083744 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a50003c50 0x7f7a50004070 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f7a60009af0 tx=0x7f7a50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=365de43442eb8fbe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:55.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.160+0000 7f7a5bfff640 1 -- 172.21.15.161:0/3060083744 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a60039070 con 0x7f7a50003c50 2024-03-28T23:35:55.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.160+0000 7f7a5bfff640 1 -- 172.21.15.161:0/3060083744 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a6002c6b0 con 0x7f7a50003c50 2024-03-28T23:35:55.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.160+0000 7f7a5bfff640 1 -- 172.21.15.161:0/3060083744 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a60009690 con 0x7f7a50003c50 2024-03-28T23:35:55.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.160+0000 7f7a6b6c0640 1 -- 172.21.15.161:0/3060083744 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a50003c50 msgr2=0x7f7a50004070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:55.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.160+0000 7f7a6b6c0640 1 --2- 172.21.15.161:0/3060083744 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a50003c50 0x7f7a50004070 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f7a60009af0 tx=0x7f7a50005830 comp rx=0 tx=0).stop 2024-03-28T23:35:55.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.160+0000 7f7a69c5d640 1 -- 172.21.15.161:0/3060083744 reap_dead start 2024-03-28T23:35:55.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.160+0000 7f7a6b6c0640 1 -- 172.21.15.161:0/3060083744 shutdown_connections 2024-03-28T23:35:55.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.160+0000 7f7a6b6c0640 1 -- 172.21.15.161:0/3060083744 >> 172.21.15.161:0/3060083744 conn(0x7f7a640c6670 msgr2=0x7f7a640c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:55.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.160+0000 7f7a6b6c0640 1 -- 172.21.15.161:0/3060083744 shutdown_connections 2024-03-28T23:35:55.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.160+0000 7f7a6b6c0640 1 -- 172.21.15.161:0/3060083744 wait complete. 2024-03-28T23:35:55.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.161+0000 7f7a6b6c0640 1 Processor -- start 2024-03-28T23:35:55.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.161+0000 7f7a6b6c0640 1 -- start start 2024-03-28T23:35:55.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.162+0000 7f7a6b6c0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a50003c50 0x7f7a640bf860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:55.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.162+0000 7f7a6b6c0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a640bfda0 0x7f7a64148a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:55.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.162+0000 7f7a6b6c0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a641ed2d0 0x7f7a641e9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:55.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.162+0000 7f7a6b6c0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7a641509e0 con 0x7f7a50003c50 2024-03-28T23:35:55.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.162+0000 7f7a6b6c0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7a64157340 con 0x7f7a641ed2d0 2024-03-28T23:35:55.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.162+0000 7f7a6b6c0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7a640aa9e0 con 0x7f7a640bfda0 2024-03-28T23:35:55.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.162+0000 7f7a6945c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a50003c50 0x7f7a640bf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:55.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.162+0000 7f7a6945c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a50003c50 0x7f7a640bf860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57562/0 (socket says 172.21.15.161:57562) 2024-03-28T23:35:55.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.162+0000 7f7a6945c640 1 -- 172.21.15.161:0/561867001 learned_addr learned my addr 172.21.15.161:0/561867001 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:55.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.162+0000 7f7a68c5b640 1 --2- 172.21.15.161:0/561867001 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a640bfda0 0x7f7a64148a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:55.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.162+0000 7f7a69c5d640 1 --2- 172.21.15.161:0/561867001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a641ed2d0 0x7f7a641e9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:55.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.162+0000 7f7a6945c640 1 -- 172.21.15.161:0/561867001 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a640bfda0 msgr2=0x7f7a64148a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:55.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.162+0000 7f7a6945c640 1 --2- 172.21.15.161:0/561867001 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a640bfda0 0x7f7a64148a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.162+0000 7f7a6945c640 1 -- 172.21.15.161:0/561867001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a641ed2d0 msgr2=0x7f7a641e9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:55.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.162+0000 7f7a6945c640 1 --2- 172.21.15.161:0/561867001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a641ed2d0 0x7f7a641e9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.162+0000 7f7a6945c640 1 -- 172.21.15.161:0/561867001 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a640c0a20 con 0x7f7a50003c50 2024-03-28T23:35:55.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.162+0000 7f7a69c5d640 1 --2- 172.21.15.161:0/561867001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a641ed2d0 0x7f7a641e9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:55.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.163+0000 7f7a6945c640 1 --2- 172.21.15.161:0/561867001 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a50003c50 0x7f7a640bf860 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f7a60037040 tx=0x7f7a60004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:55.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.163+0000 7f7a59ffb640 1 -- 172.21.15.161:0/561867001 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a60042020 con 0x7f7a50003c50 2024-03-28T23:35:55.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.163+0000 7f7a59ffb640 1 -- 172.21.15.161:0/561867001 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a6003a040 con 0x7f7a50003c50 2024-03-28T23:35:55.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.163+0000 7f7a6b6c0640 1 -- 172.21.15.161:0/561867001 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a640c0740 con 0x7f7a50003c50 2024-03-28T23:35:55.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.163+0000 7f7a59ffb640 1 -- 172.21.15.161:0/561867001 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a60039070 con 0x7f7a50003c50 2024-03-28T23:35:55.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.163+0000 7f7a6b6c0640 1 -- 172.21.15.161:0/561867001 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a641ecad0 con 0x7f7a50003c50 2024-03-28T23:35:55.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.164+0000 7f7a6b6c0640 1 -- 172.21.15.161:0/561867001 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a6404f4b0 con 0x7f7a50003c50 2024-03-28T23:35:55.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.164+0000 7f7a59ffb640 1 -- 172.21.15.161:0/561867001 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7a60045020 con 0x7f7a50003c50 2024-03-28T23:35:55.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.165+0000 7f7a59ffb640 1 --2- 172.21.15.161:0/561867001 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7a4806b4d0 0x7f7a4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:55.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.165+0000 7f7a59ffb640 1 -- 172.21.15.161:0/561867001 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 5249+0+0 (secure 0 0 0) 0x7f7a600ab490 con 0x7f7a50003c50 2024-03-28T23:35:55.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.165+0000 7f7a68c5b640 1 --2- 172.21.15.161:0/561867001 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7a4806b4d0 0x7f7a4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:55.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.165+0000 7f7a68c5b640 1 --2- 172.21.15.161:0/561867001 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7a4806b4d0 0x7f7a4806d990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f7a540010e0 tx=0x7f7a54009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:55.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.169+0000 7f7a59ffb640 1 -- 172.21.15.161:0/561867001 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7a6004c070 con 0x7f7a50003c50 2024-03-28T23:35:55.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.409+0000 7f7a6b6c0640 1 -- 172.21.15.161:0/561867001 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f7a6414cbc0 con 0x7f7a50003c50 2024-03-28T23:35:55.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.409+0000 7f7a59ffb640 1 -- 172.21.15.161:0/561867001 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) ==== 80+0+4724 (secure 0 0 0) 0x7f7a6007a560 con 0x7f7a50003c50 2024-03-28T23:35:55.410 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:55.410 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:35:55.410 INFO:teuthology.orchestra.run.smithi161.stdout: "auth_dump": [ 2024-03-28T23:35:55.410 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:55.410 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.a", 2024-03-28T23:35:55.410 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmggtoJhAASLXbUICLBhjRr19aUAuZHw==", 2024-03-28T23:35:55.410 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:55.410 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:35:55.410 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:35:55.410 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:35:55.411 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:55.411 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:55.411 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:55.411 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:55.411 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.b", 2024-03-28T23:35:55.411 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmQbEOMxAA74al6+dMzcN/ngwNZK+aHA==", 2024-03-28T23:35:55.411 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:55.411 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:35:55.411 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:35:55.411 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:35:55.411 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:55.411 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:55.411 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:55.411 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:55.412 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.c", 2024-03-28T23:35:55.412 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmjf/qLRAAosLbpT8aCWvrKAQnj8Uh3g==", 2024-03-28T23:35:55.412 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:55.412 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:35:55.412 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:35:55.412 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:35:55.412 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:55.412 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:55.412 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:55.412 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:55.412 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.d", 2024-03-28T23:35:55.412 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmpiZ1OhAAjYrFAaDosDS2Y+BvU4c6JQ==", 2024-03-28T23:35:55.412 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:55.412 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:35:55.412 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:35:55.413 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:35:55.413 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:55.413 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:55.413 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:55.413 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:55.413 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.0", 2024-03-28T23:35:55.413 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA9/QVmAFPmGBAAS0SkRRwsupm6jm1sqFXJ6g==", 2024-03-28T23:35:55.413 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:55.413 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:35:55.413 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:35:55.413 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:55.413 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:55.413 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:55.413 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:55.414 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.1", 2024-03-28T23:35:55.414 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBA/QVmNBTVFxAA1htU9ilznfLUDqKmJaQwzg==", 2024-03-28T23:35:55.414 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:55.414 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:35:55.414 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:35:55.414 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:55.414 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:55.414 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:55.414 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:55.414 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.2", 2024-03-28T23:35:55.414 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBD/QVmg+4YFhAA9TvfFVKT3nQT7/QUOf3Spw==", 2024-03-28T23:35:55.414 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:55.414 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:35:55.414 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:35:55.414 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:55.415 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:55.415 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:55.415 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:55.415 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.3", 2024-03-28T23:35:55.415 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBG/QVmgiHDFBAAQPtbZT6a1ZGmkisBX0NMgA==", 2024-03-28T23:35:55.415 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:55.415 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:35:55.415 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:35:55.415 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:55.415 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:55.415 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:55.415 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:55.415 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.4", 2024-03-28T23:35:55.415 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBK/QVmahhZIBAAJfotg/dusbB3usreZlRuPA==", 2024-03-28T23:35:55.416 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:55.416 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:35:55.416 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:35:55.416 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:55.416 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:55.416 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:55.416 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:55.416 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.5", 2024-03-28T23:35:55.416 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBN/QVmzjlDHhAAli7pLyIo+MVdwSBNJYVg5g==", 2024-03-28T23:35:55.416 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:55.416 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:35:55.416 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:35:55.416 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:55.416 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:55.416 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:55.416 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:55.417 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.6", 2024-03-28T23:35:55.417 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBQ/QVmtCFSHBAASVfV/5mY/3JNiI0YMT7Vww==", 2024-03-28T23:35:55.417 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:55.417 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:35:55.417 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:35:55.417 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:55.417 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:55.417 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:55.417 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:55.417 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.7", 2024-03-28T23:35:55.417 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBT/QVm2J7eGhAAaIeZxmLwQBBE1glCxjKbhA==", 2024-03-28T23:35:55.417 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:55.417 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-28T23:35:55.417 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-28T23:35:55.417 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:55.417 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:55.418 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:55.418 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:55.418 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.0", 2024-03-28T23:35:55.418 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA6/QVmc1krAxAAEKsn1Tuuv9OlFZpB4PFhDg==", 2024-03-28T23:35:55.418 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:55.418 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:35:55.418 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow r", 2024-03-28T23:35:55.418 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-28T23:35:55.418 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:55.418 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:55.418 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:55.418 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.1", 2024-03-28T23:35:55.418 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA6/QVm1B2cBhAAmHDeeFwpFlV/y3NIGWUXHQ==", 2024-03-28T23:35:55.418 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:55.419 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-28T23:35:55.419 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow r", 2024-03-28T23:35:55.419 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow rw", 2024-03-28T23:35:55.419 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow rwx" 2024-03-28T23:35:55.419 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:55.419 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:55.419 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:55.419 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.admin", 2024-03-28T23:35:55.419 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVmzzJ7CRAAYqfhnIkfNJJ3O+VWC2LP+Q==", 2024-03-28T23:35:55.419 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:55.419 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-28T23:35:55.419 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-28T23:35:55.419 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-28T23:35:55.419 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:55.419 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:55.420 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:55.420 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:55.420 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mgr.x", 2024-03-28T23:35:55.420 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVm7TD3IhAACByGRGpvJWDpF22NuovHWw==", 2024-03-28T23:35:55.420 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:55.420 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-28T23:35:55.420 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile mgr", 2024-03-28T23:35:55.420 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:55.420 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:55.420 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:35:55.420 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:55.420 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mgr.y", 2024-03-28T23:35:55.420 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQA5/QVm0hLlHhAAh8cewPtZ/TbKrU488kxrBg==", 2024-03-28T23:35:55.420 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-28T23:35:55.420 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-28T23:35:55.420 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile mgr", 2024-03-28T23:35:55.421 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-28T23:35:55.421 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:55.421 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:55.421 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-03-28T23:35:55.421 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:35:55.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.411+0000 7f7a6b6c0640 1 -- 172.21.15.161:0/561867001 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7a4806b4d0 msgr2=0x7f7a4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:55.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.411+0000 7f7a6b6c0640 1 --2- 172.21.15.161:0/561867001 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7a4806b4d0 0x7f7a4806d990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f7a540010e0 tx=0x7f7a54009040 comp rx=0 tx=0).stop 2024-03-28T23:35:55.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.411+0000 7f7a6b6c0640 1 -- 172.21.15.161:0/561867001 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a50003c50 msgr2=0x7f7a640bf860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:55.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.411+0000 7f7a6b6c0640 1 --2- 172.21.15.161:0/561867001 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a50003c50 0x7f7a640bf860 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f7a60037040 tx=0x7f7a60004730 comp rx=0 tx=0).stop 2024-03-28T23:35:55.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.412+0000 7f7a6b6c0640 1 -- 172.21.15.161:0/561867001 shutdown_connections 2024-03-28T23:35:55.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.412+0000 7f7a6b6c0640 1 --2- 172.21.15.161:0/561867001 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7a4806b4d0 0x7f7a4806d990 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.412+0000 7f7a6b6c0640 1 --2- 172.21.15.161:0/561867001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a641ed2d0 0x7f7a641e9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.412+0000 7f7a6b6c0640 1 --2- 172.21.15.161:0/561867001 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a640bfda0 0x7f7a64148a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.412+0000 7f7a6b6c0640 1 --2- 172.21.15.161:0/561867001 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a50003c50 0x7f7a640bf860 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.412+0000 7f7a6b6c0640 1 -- 172.21.15.161:0/561867001 >> 172.21.15.161:0/561867001 conn(0x7f7a640c6670 msgr2=0x7f7a6414f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:55.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.412+0000 7f7a6b6c0640 1 -- 172.21.15.161:0/561867001 shutdown_connections 2024-03-28T23:35:55.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.412+0000 7f7a6b6c0640 1 -- 172.21.15.161:0/561867001 wait complete. 2024-03-28T23:35:55.424 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:35:55.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.524+0000 7f91708c9640 1 Processor -- start 2024-03-28T23:35:55.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.524+0000 7f91708c9640 1 -- start start 2024-03-28T23:35:55.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.525+0000 7f91708c9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f916c0c5410 0x7f916c0c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:55.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.525+0000 7f91708c9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f916c1553a0 0x7f916c155820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:55.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.525+0000 7f91708c9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f916c158a30 con 0x7f916c152890 2024-03-28T23:35:55.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.525+0000 7f91708c9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f916c0c8480 con 0x7f916c1553a0 2024-03-28T23:35:55.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.525+0000 7f91708c9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f916c0c5d90 con 0x7f916c0c5410 2024-03-28T23:35:55.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.525+0000 7f916a59c640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f916c152890 0x7f916c0c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:35214/0 (socket says 172.21.15.161:35214) 2024-03-28T23:35:55.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.525+0000 7f9169d9b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f916c0c5410 0x7f916c0c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:55.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.526+0000 7f916a59c640 1 -- 172.21.15.161:0/1482234575 learned_addr learned my addr 172.21.15.161:0/1482234575 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:55.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.525+0000 7f916ad9d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f916c1553a0 0x7f916c155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:55.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.526+0000 7f916ad9d640 1 -- 172.21.15.161:0/1482234575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f916c0c5410 msgr2=0x7f916c0c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:55.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.526+0000 7f916ad9d640 1 --2- 172.21.15.161:0/1482234575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f916c0c5410 0x7f916c0c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.526+0000 7f916ad9d640 1 -- 172.21.15.161:0/1482234575 >> v1:172.21.15.170:6789/0 conn(0x7f916c152890 legacy=0x7f916c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:55.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.526+0000 7f916959a640 1 -- 172.21.15.161:0/1482234575 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3631281225 0 0) 0x7f916c158a30 con 0x7f916c152890 2024-03-28T23:35:55.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.526+0000 7f916ad9d640 1 -- 172.21.15.161:0/1482234575 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f916c158d10 con 0x7f916c1553a0 2024-03-28T23:35:55.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.526+0000 7f9169d9b640 1 --2- 172.21.15.161:0/1482234575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f916c0c5410 0x7f916c0c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:35:55.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.527+0000 7f916ad9d640 1 --2- 172.21.15.161:0/1482234575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f916c1553a0 0x7f916c155820 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f91640097a0 tx=0x7f916402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8571e450dac38531 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:55.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.527+0000 7f916959a640 1 -- 172.21.15.161:0/1482234575 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91640315a0 con 0x7f916c1553a0 2024-03-28T23:35:55.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.527+0000 7f916959a640 1 -- 172.21.15.161:0/1482234575 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9164037070 con 0x7f916c1553a0 2024-03-28T23:35:55.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.527+0000 7f916959a640 1 -- 172.21.15.161:0/1482234575 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9164031960 con 0x7f916c1553a0 2024-03-28T23:35:55.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.527+0000 7f91708c9640 1 -- 172.21.15.161:0/1482234575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f916c1553a0 msgr2=0x7f916c155820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:55.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.527+0000 7f91708c9640 1 --2- 172.21.15.161:0/1482234575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f916c1553a0 0x7f916c155820 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f91640097a0 tx=0x7f916402b840 comp rx=0 tx=0).stop 2024-03-28T23:35:55.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.528+0000 7f91708c9640 1 -- 172.21.15.161:0/1482234575 shutdown_connections 2024-03-28T23:35:55.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.528+0000 7f91708c9640 1 --2- 172.21.15.161:0/1482234575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f916c1553a0 0x7f916c155820 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.528+0000 7f91708c9640 1 --2- 172.21.15.161:0/1482234575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f916c0c5410 0x7f916c0c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.528+0000 7f91708c9640 1 -- 172.21.15.161:0/1482234575 >> 172.21.15.161:0/1482234575 conn(0x7f916c0ac620 msgr2=0x7f916c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:55.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.528+0000 7f91708c9640 1 -- 172.21.15.161:0/1482234575 shutdown_connections 2024-03-28T23:35:55.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.528+0000 7f91708c9640 1 -- 172.21.15.161:0/1482234575 wait complete. 2024-03-28T23:35:55.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.529+0000 7f91708c9640 1 Processor -- start 2024-03-28T23:35:55.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.529+0000 7f91708c9640 1 -- start start 2024-03-28T23:35:55.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.529+0000 7f91708c9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f916c0c5410 0x7f916c1ea840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:55.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.529+0000 7f91708c9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f916c152890 0x7f916c06ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:55.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.529+0000 7f91708c9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f916c1553a0 0x7f916c1ee3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:55.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.529+0000 7f91708c9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f916c1592d0 con 0x7f916c152890 2024-03-28T23:35:55.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.529+0000 7f91708c9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f916c159640 con 0x7f916c0c5410 2024-03-28T23:35:55.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.529+0000 7f91708c9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f916c0c90a0 con 0x7f916c1553a0 2024-03-28T23:35:55.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.530+0000 7f9169d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f916c152890 0x7f916c06ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:55.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.530+0000 7f9169d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f916c152890 0x7f916c06ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57574/0 (socket says 172.21.15.161:57574) 2024-03-28T23:35:55.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.530+0000 7f9169d9b640 1 -- 172.21.15.161:0/560083967 learned_addr learned my addr 172.21.15.161:0/560083967 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:55.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.530+0000 7f916ad9d640 1 --2- 172.21.15.161:0/560083967 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f916c1553a0 0x7f916c1ee3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:55.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.530+0000 7f916a59c640 1 --2- 172.21.15.161:0/560083967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f916c0c5410 0x7f916c1ea840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:55.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.530+0000 7f9169d9b640 1 -- 172.21.15.161:0/560083967 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f916c1553a0 msgr2=0x7f916c1ee3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:55.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.530+0000 7f9169d9b640 1 --2- 172.21.15.161:0/560083967 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f916c1553a0 0x7f916c1ee3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.530+0000 7f9169d9b640 1 -- 172.21.15.161:0/560083967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f916c0c5410 msgr2=0x7f916c1ea840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:55.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.530+0000 7f9169d9b640 1 --2- 172.21.15.161:0/560083967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f916c0c5410 0x7f916c1ea840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.530+0000 7f9169d9b640 1 -- 172.21.15.161:0/560083967 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f916c1eae90 con 0x7f916c152890 2024-03-28T23:35:55.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.530+0000 7f916ad9d640 1 --2- 172.21.15.161:0/560083967 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f916c1553a0 0x7f916c1ee3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:55.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.530+0000 7f9169d9b640 1 --2- 172.21.15.161:0/560083967 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f916c152890 0x7f916c06ca00 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f916000e900 tx=0x7f916000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:55.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.530+0000 7f915affd640 1 -- 172.21.15.161:0/560083967 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916000ca00 con 0x7f916c152890 2024-03-28T23:35:55.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.530+0000 7f915affd640 1 -- 172.21.15.161:0/560083967 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f916000cba0 con 0x7f916c152890 2024-03-28T23:35:55.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.530+0000 7f915affd640 1 -- 172.21.15.161:0/560083967 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9160005990 con 0x7f916c152890 2024-03-28T23:35:55.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.530+0000 7f91708c9640 1 -- 172.21.15.161:0/560083967 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f916c1eb170 con 0x7f916c152890 2024-03-28T23:35:55.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.530+0000 7f91708c9640 1 -- 172.21.15.161:0/560083967 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f916c1eb730 con 0x7f916c152890 2024-03-28T23:35:55.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.531+0000 7f91708c9640 1 -- 172.21.15.161:0/560083967 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f916c04f4b0 con 0x7f916c152890 2024-03-28T23:35:55.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.532+0000 7f915affd640 1 -- 172.21.15.161:0/560083967 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9160005b30 con 0x7f916c152890 2024-03-28T23:35:55.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.532+0000 7f915affd640 1 --2- 172.21.15.161:0/560083967 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f914c06b500 0x7f914c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:55.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.532+0000 7f915affd640 1 -- 172.21.15.161:0/560083967 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 5249+0+0 (secure 0 0 0) 0x7f916008a280 con 0x7f916c152890 2024-03-28T23:35:55.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.532+0000 7f916a59c640 1 --2- 172.21.15.161:0/560083967 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f914c06b500 0x7f914c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:55.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.532+0000 7f916a59c640 1 --2- 172.21.15.161:0/560083967 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f914c06b500 0x7f914c06d9c0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f915c0012e0 tx=0x7f915c001700 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:55.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.536+0000 7f915affd640 1 -- 172.21.15.161:0/560083967 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9160017020 con 0x7f916c152890 2024-03-28T23:35:55.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.708+0000 7f91708c9640 1 -- 172.21.15.161:0/560083967 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f916c0c8140 con 0x7f916c152890 2024-03-28T23:35:55.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.709+0000 7f915affd640 1 -- 172.21.15.161:0/560083967 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) ==== 74+0+13210 (secure 0 0 0) 0x7f9160058d00 con 0x7f916c152890 2024-03-28T23:35:55.709 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:55.710 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":148,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:35:55.035912+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:35:55.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.711+0000 7f91708c9640 1 -- 172.21.15.161:0/560083967 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f914c06b500 msgr2=0x7f914c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:55.711 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.711+0000 7f91708c9640 1 --2- 172.21.15.161:0/560083967 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f914c06b500 0x7f914c06d9c0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f915c0012e0 tx=0x7f915c001700 comp rx=0 tx=0).stop 2024-03-28T23:35:55.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.711+0000 7f91708c9640 1 -- 172.21.15.161:0/560083967 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f916c152890 msgr2=0x7f916c06ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:55.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.711+0000 7f91708c9640 1 --2- 172.21.15.161:0/560083967 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f916c152890 0x7f916c06ca00 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f916000e900 tx=0x7f916000edc0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.711+0000 7f91708c9640 1 -- 172.21.15.161:0/560083967 shutdown_connections 2024-03-28T23:35:55.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.711+0000 7f91708c9640 1 --2- 172.21.15.161:0/560083967 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f914c06b500 0x7f914c06d9c0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.711+0000 7f91708c9640 1 --2- 172.21.15.161:0/560083967 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f916c152890 0x7f916c06ca00 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.711+0000 7f91708c9640 1 --2- 172.21.15.161:0/560083967 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f916c1553a0 0x7f916c1ee3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.711+0000 7f91708c9640 1 --2- 172.21.15.161:0/560083967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f916c0c5410 0x7f916c1ea840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.711+0000 7f91708c9640 1 -- 172.21.15.161:0/560083967 >> 172.21.15.161:0/560083967 conn(0x7f916c0ac620 msgr2=0x7f916c152fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:55.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.711+0000 7f91708c9640 1 -- 172.21.15.161:0/560083967 shutdown_connections 2024-03-28T23:35:55.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.711+0000 7f91708c9640 1 -- 172.21.15.161:0/560083967 wait complete. 2024-03-28T23:35:55.723 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:35:55.723 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:35:55.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.828+0000 7fdaaffe9640 1 Processor -- start 2024-03-28T23:35:55.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.828+0000 7fdaaffe9640 1 -- start start 2024-03-28T23:35:55.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.829+0000 7fdaaffe9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdaa8151e70 0x7fdaa8152250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:55.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.829+0000 7fdaaffe9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdaa8152790 0x7fdaa806c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:55.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.829+0000 7fdaaffe9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdaa80c1e40 con 0x7fdaa80c6350 2024-03-28T23:35:55.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.829+0000 7fdaaffe9640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fdaa80c2120 con 0x7fdaa8151e70 2024-03-28T23:35:55.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.829+0000 7fdaaffe9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdaa80c2400 con 0x7fdaa8152790 2024-03-28T23:35:55.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.829+0000 7fdaadd85640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdaa8151e70 0x7fdaa8152250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:55.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.829+0000 7fdaae586640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fdaa80c6350 0x7fdaa81553d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:54332/0 (socket says 172.21.15.161:54332) 2024-03-28T23:35:55.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.829+0000 7fdaadd85640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdaa8151e70 0x7fdaa8152250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57580/0 (socket says 172.21.15.161:57580) 2024-03-28T23:35:55.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.829+0000 7fdaae586640 1 -- 172.21.15.161:0/3481800179 learned_addr learned my addr 172.21.15.161:0/3481800179 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:55.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.829+0000 7fdaad584640 1 --2- 172.21.15.161:0/3481800179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdaa8152790 0x7fdaa806c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:55.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.830+0000 7fdaadd85640 1 -- 172.21.15.161:0/3481800179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdaa8152790 msgr2=0x7fdaa806c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:55.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.830+0000 7fdaadd85640 1 --2- 172.21.15.161:0/3481800179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdaa8152790 0x7fdaa806c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.830+0000 7fdaadd85640 1 -- 172.21.15.161:0/3481800179 >> v1:172.21.15.161:6789/0 conn(0x7fdaa80c6350 legacy=0x7fdaa81553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:55.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.830+0000 7fdaadd85640 1 -- 172.21.15.161:0/3481800179 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa80c26e0 con 0x7fdaa8151e70 2024-03-28T23:35:55.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.830+0000 7fdaacd83640 1 -- 172.21.15.161:0/3481800179 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1082221735 0 0) 0x7fdaa80c1e40 con 0x7fdaa80c6350 2024-03-28T23:35:55.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.830+0000 7fdaad584640 1 --2- 172.21.15.161:0/3481800179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdaa8152790 0x7fdaa806c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:55.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.830+0000 7fdaadd85640 1 --2- 172.21.15.161:0/3481800179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaa8151e70 0x7fdaa8152250 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fdaa40097a0 tx=0x7fdaa402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b80ad091bbddca2d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:55.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.830+0000 7fdaacd83640 1 -- 172.21.15.161:0/3481800179 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa40315a0 con 0x7fdaa8151e70 2024-03-28T23:35:55.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.830+0000 7fdaacd83640 1 -- 172.21.15.161:0/3481800179 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaa4037070 con 0x7fdaa8151e70 2024-03-28T23:35:55.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.830+0000 7fdaacd83640 1 -- 172.21.15.161:0/3481800179 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4031960 con 0x7fdaa8151e70 2024-03-28T23:35:55.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.831+0000 7fdaaffe9640 1 -- 172.21.15.161:0/3481800179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaa8151e70 msgr2=0x7fdaa8152250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:55.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.831+0000 7fdaaffe9640 1 --2- 172.21.15.161:0/3481800179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaa8151e70 0x7fdaa8152250 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fdaa40097a0 tx=0x7fdaa402b840 comp rx=0 tx=0).stop 2024-03-28T23:35:55.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.831+0000 7fdaaffe9640 1 -- 172.21.15.161:0/3481800179 shutdown_connections 2024-03-28T23:35:55.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.831+0000 7fdaaffe9640 1 --2- 172.21.15.161:0/3481800179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdaa8152790 0x7fdaa806c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.831+0000 7fdaaffe9640 1 --2- 172.21.15.161:0/3481800179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaa8151e70 0x7fdaa8152250 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.831+0000 7fdaaffe9640 1 -- 172.21.15.161:0/3481800179 >> 172.21.15.161:0/3481800179 conn(0x7fdaa80c3390 msgr2=0x7fdaa80c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:55.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.831+0000 7fdaaffe9640 1 -- 172.21.15.161:0/3481800179 shutdown_connections 2024-03-28T23:35:55.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.831+0000 7fdaaffe9640 1 -- 172.21.15.161:0/3481800179 wait complete. 2024-03-28T23:35:55.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.832+0000 7fdaaffe9640 1 Processor -- start 2024-03-28T23:35:55.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.832+0000 7fdaaffe9640 1 -- start start 2024-03-28T23:35:55.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.833+0000 7fdaaffe9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaa80c6350 0x7fdaa8072370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:55.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.833+0000 7fdaaffe9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdaa8151e70 0x7fdaa8070f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:55.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.833+0000 7fdaaffe9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaa8155f80 0x7fdaa80aacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:55.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.833+0000 7fdaaffe9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa80c2d80 con 0x7fdaa80c6350 2024-03-28T23:35:55.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.833+0000 7fdaaffe9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa80c2c00 con 0x7fdaa8155f80 2024-03-28T23:35:55.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.833+0000 7fdaaffe9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdaa80a5fb0 con 0x7fdaa8151e70 2024-03-28T23:35:55.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.833+0000 7fdaadd85640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaa80c6350 0x7fdaa8072370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:55.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.833+0000 7fdaadd85640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaa80c6350 0x7fdaa8072370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57590/0 (socket says 172.21.15.161:57590) 2024-03-28T23:35:55.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.833+0000 7fdaadd85640 1 -- 172.21.15.161:0/2157696002 learned_addr learned my addr 172.21.15.161:0/2157696002 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:55.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.833+0000 7fdaad584640 1 --2- 172.21.15.161:0/2157696002 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdaa8151e70 0x7fdaa8070f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:55.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.833+0000 7fdaae586640 1 --2- 172.21.15.161:0/2157696002 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaa8155f80 0x7fdaa80aacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:55.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.833+0000 7fdaadd85640 1 -- 172.21.15.161:0/2157696002 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdaa8151e70 msgr2=0x7fdaa8070f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:55.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.833+0000 7fdaadd85640 1 --2- 172.21.15.161:0/2157696002 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdaa8151e70 0x7fdaa8070f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.833+0000 7fdaadd85640 1 -- 172.21.15.161:0/2157696002 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaa8155f80 msgr2=0x7fdaa80aacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:55.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.833+0000 7fdaadd85640 1 --2- 172.21.15.161:0/2157696002 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaa8155f80 0x7fdaa80aacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:55.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.833+0000 7fdaadd85640 1 -- 172.21.15.161:0/2157696002 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa80ac3a0 con 0x7fdaa80c6350 2024-03-28T23:35:55.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.833+0000 7fdaae586640 1 --2- 172.21.15.161:0/2157696002 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaa8155f80 0x7fdaa80aacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:55.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.833+0000 7fdaadd85640 1 --2- 172.21.15.161:0/2157696002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaa80c6350 0x7fdaa8072370 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4034040 tx=0x7fdaa40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:55.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.833+0000 7fda9e7fc640 1 -- 172.21.15.161:0/2157696002 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4032840 con 0x7fdaa80c6350 2024-03-28T23:35:55.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.833+0000 7fda9e7fc640 1 -- 172.21.15.161:0/2157696002 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaa4032a00 con 0x7fdaa80c6350 2024-03-28T23:35:55.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.833+0000 7fdaaffe9640 1 -- 172.21.15.161:0/2157696002 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaa80ac660 con 0x7fdaa80c6350 2024-03-28T23:35:55.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.833+0000 7fda9e7fc640 1 -- 172.21.15.161:0/2157696002 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4032df0 con 0x7fdaa80c6350 2024-03-28T23:35:55.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.834+0000 7fdaaffe9640 1 -- 172.21.15.161:0/2157696002 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaa80ab200 con 0x7fdaa80c6350 2024-03-28T23:35:55.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.835+0000 7fdaaffe9640 1 -- 172.21.15.161:0/2157696002 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaa804f4b0 con 0x7fdaa80c6350 2024-03-28T23:35:55.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.835+0000 7fda9e7fc640 1 -- 172.21.15.161:0/2157696002 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdaa404e430 con 0x7fdaa80c6350 2024-03-28T23:35:55.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.835+0000 7fda9e7fc640 1 --2- 172.21.15.161:0/2157696002 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fda8c06b4d0 0x7fda8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:55.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.835+0000 7fda9e7fc640 1 -- 172.21.15.161:0/2157696002 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 5249+0+0 (secure 0 0 0) 0x7fdaa402f030 con 0x7fdaa80c6350 2024-03-28T23:35:55.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.835+0000 7fdaad584640 1 --2- 172.21.15.161:0/2157696002 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fda8c06b4d0 0x7fda8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:55.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.836+0000 7fdaad584640 1 --2- 172.21.15.161:0/2157696002 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fda8c06b4d0 0x7fda8c06d990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fdaa80a80e0 tx=0x7fda98008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:55.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:55.840+0000 7fda9e7fc640 1 -- 172.21.15.161:0/2157696002 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdaa4040070 con 0x7fdaa80c6350 2024-03-28T23:35:56.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.011+0000 7fdaaffe9640 1 -- 172.21.15.161:0/2157696002 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdaa80c2f80 con 0x7fdaa80c6350 2024-03-28T23:35:56.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.012+0000 7fda9e7fc640 1 -- 172.21.15.161:0/2157696002 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v148) ==== 93+0+10 (secure 0 0 0) 0x7fdaa407afd0 con 0x7fdaa80c6350 2024-03-28T23:35:56.012 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:35:56.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.014+0000 7fdaaffe9640 1 -- 172.21.15.161:0/2157696002 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fda8c06b4d0 msgr2=0x7fda8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:56.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.014+0000 7fdaaffe9640 1 --2- 172.21.15.161:0/2157696002 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fda8c06b4d0 0x7fda8c06d990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fdaa80a80e0 tx=0x7fda98008040 comp rx=0 tx=0).stop 2024-03-28T23:35:56.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.014+0000 7fdaaffe9640 1 -- 172.21.15.161:0/2157696002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaa80c6350 msgr2=0x7fdaa8072370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:56.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.014+0000 7fdaaffe9640 1 --2- 172.21.15.161:0/2157696002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaa80c6350 0x7fdaa8072370 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4034040 tx=0x7fdaa40046c0 comp rx=0 tx=0).stop 2024-03-28T23:35:56.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.014+0000 7fdaaffe9640 1 -- 172.21.15.161:0/2157696002 shutdown_connections 2024-03-28T23:35:56.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.014+0000 7fdaaffe9640 1 --2- 172.21.15.161:0/2157696002 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fda8c06b4d0 0x7fda8c06d990 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:56.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.014+0000 7fdaaffe9640 1 --2- 172.21.15.161:0/2157696002 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaa8155f80 0x7fdaa80aacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:56.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.014+0000 7fdaaffe9640 1 --2- 172.21.15.161:0/2157696002 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdaa8151e70 0x7fdaa8070f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:56.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.014+0000 7fdaaffe9640 1 --2- 172.21.15.161:0/2157696002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaa80c6350 0x7fdaa8072370 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:56.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.014+0000 7fdaaffe9640 1 -- 172.21.15.161:0/2157696002 >> 172.21.15.161:0/2157696002 conn(0x7fdaa80c3390 msgr2=0x7fdaa80c5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:56.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.014+0000 7fdaaffe9640 1 -- 172.21.15.161:0/2157696002 shutdown_connections 2024-03-28T23:35:56.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.014+0000 7fdaaffe9640 1 -- 172.21.15.161:0/2157696002 wait complete. 2024-03-28T23:35:56.025 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T23:35:56.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.126+0000 7fc7b474f640 1 Processor -- start 2024-03-28T23:35:56.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.126+0000 7fc7b474f640 1 -- start start 2024-03-28T23:35:56.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.126+0000 7fc7b474f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7ac14f730 0x7fc7ac0aad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:56.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.126+0000 7fc7b474f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc7ac151f80 0x7fc7ac0c21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:56.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.126+0000 7fc7b474f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7ac0ab250 con 0x7fc7ac0ac310 2024-03-28T23:35:56.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.126+0000 7fc7b474f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fc7ac152f00 con 0x7fc7ac151f80 2024-03-28T23:35:56.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.126+0000 7fc7b474f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc7ac155990 con 0x7fc7ac14f730 2024-03-28T23:35:56.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.127+0000 7fc7b1cea640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc7ac0ac310 0x7fc7ac155e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:54348/0 (socket says 172.21.15.161:54348) 2024-03-28T23:35:56.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.127+0000 7fc7b1cea640 1 -- 172.21.15.161:0/1838413450 learned_addr learned my addr 172.21.15.161:0/1838413450 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:56.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.127+0000 7fc7b24eb640 1 --2- 172.21.15.161:0/1838413450 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7ac14f730 0x7fc7ac0aad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:56.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.127+0000 7fc7b2cec640 1 --2- 172.21.15.161:0/1838413450 >> v2:172.21.15.170:3300/0 conn(0x7fc7ac151f80 0x7fc7ac0c21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:56.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.127+0000 7fc7b14e9640 1 -- 172.21.15.161:0/1838413450 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2748571784 0 0) 0x7fc7ac0ab250 con 0x7fc7ac0ac310 2024-03-28T23:35:56.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.127+0000 7fc7b14e9640 1 -- 172.21.15.161:0/1838413450 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc798005730 con 0x7fc7ac0ac310 2024-03-28T23:35:56.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.127+0000 7fc7b24eb640 1 -- 172.21.15.161:0/1838413450 >> v2:172.21.15.170:3300/0 conn(0x7fc7ac151f80 msgr2=0x7fc7ac0c21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:56.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.127+0000 7fc7b24eb640 1 --2- 172.21.15.161:0/1838413450 >> v2:172.21.15.170:3300/0 conn(0x7fc7ac151f80 0x7fc7ac0c21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:56.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.127+0000 7fc7b24eb640 1 -- 172.21.15.161:0/1838413450 >> v1:172.21.15.161:6789/0 conn(0x7fc7ac0ac310 legacy=0x7fc7ac155e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:56.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.127+0000 7fc7b24eb640 1 -- 172.21.15.161:0/1838413450 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7ac152940 con 0x7fc7ac14f730 2024-03-28T23:35:56.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.128+0000 7fc7b24eb640 1 --2- 172.21.15.161:0/1838413450 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7ac14f730 0x7fc7ac0aad10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc79c0097a0 tx=0x7fc79c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de415dcfb1ea0298 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:56.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.128+0000 7fc7b14e9640 1 -- 172.21.15.161:0/1838413450 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79c0315a0 con 0x7fc7ac14f730 2024-03-28T23:35:56.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.128+0000 7fc7b14e9640 1 -- 172.21.15.161:0/1838413450 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc79c037070 con 0x7fc7ac14f730 2024-03-28T23:35:56.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.128+0000 7fc7b14e9640 1 -- 172.21.15.161:0/1838413450 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79c031940 con 0x7fc7ac14f730 2024-03-28T23:35:56.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.129+0000 7fc7b474f640 1 -- 172.21.15.161:0/1838413450 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7ac14f730 msgr2=0x7fc7ac0aad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:56.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.129+0000 7fc7b474f640 1 --2- 172.21.15.161:0/1838413450 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7ac14f730 0x7fc7ac0aad10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc79c0097a0 tx=0x7fc79c02b840 comp rx=0 tx=0).stop 2024-03-28T23:35:56.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.129+0000 7fc7b474f640 1 -- 172.21.15.161:0/1838413450 shutdown_connections 2024-03-28T23:35:56.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.129+0000 7fc7b474f640 1 --2- 172.21.15.161:0/1838413450 >> v2:172.21.15.170:3300/0 conn(0x7fc7ac151f80 0x7fc7ac0c21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:56.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.129+0000 7fc7b474f640 1 --2- 172.21.15.161:0/1838413450 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7ac14f730 0x7fc7ac0aad10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:56.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.129+0000 7fc7b474f640 1 -- 172.21.15.161:0/1838413450 >> 172.21.15.161:0/1838413450 conn(0x7fc7ac0c66d0 msgr2=0x7fc7ac0c6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:56.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.129+0000 7fc7b474f640 1 -- 172.21.15.161:0/1838413450 shutdown_connections 2024-03-28T23:35:56.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.129+0000 7fc7b474f640 1 -- 172.21.15.161:0/1838413450 wait complete. 2024-03-28T23:35:56.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.130+0000 7fc7b474f640 1 Processor -- start 2024-03-28T23:35:56.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.130+0000 7fc7b474f640 1 -- start start 2024-03-28T23:35:56.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.131+0000 7fc7b474f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7ac0ac310 0x7fc7ac0c4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:56.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.131+0000 7fc7b474f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7ac14f730 0x7fc7ac14c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:56.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.131+0000 7fc7b474f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7ac151f80 0x7fc7ac14cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:56.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.131+0000 7fc7b474f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ac14fb10 con 0x7fc7ac14f730 2024-03-28T23:35:56.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.131+0000 7fc7b474f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ac155620 con 0x7fc7ac151f80 2024-03-28T23:35:56.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.131+0000 7fc7b474f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc7ac152410 con 0x7fc7ac0ac310 2024-03-28T23:35:56.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.131+0000 7fc7b1cea640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7ac14f730 0x7fc7ac14c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:56.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.131+0000 7fc7b1cea640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7ac14f730 0x7fc7ac14c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57596/0 (socket says 172.21.15.161:57596) 2024-03-28T23:35:56.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.131+0000 7fc7b1cea640 1 -- 172.21.15.161:0/4039100592 learned_addr learned my addr 172.21.15.161:0/4039100592 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:56.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.131+0000 7fc7b2cec640 1 --2- 172.21.15.161:0/4039100592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7ac151f80 0x7fc7ac14cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:56.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.131+0000 7fc7b24eb640 1 --2- 172.21.15.161:0/4039100592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7ac0ac310 0x7fc7ac0c4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:56.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.131+0000 7fc7b1cea640 1 -- 172.21.15.161:0/4039100592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7ac0ac310 msgr2=0x7fc7ac0c4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:56.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.131+0000 7fc7b1cea640 1 --2- 172.21.15.161:0/4039100592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7ac0ac310 0x7fc7ac0c4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:56.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.131+0000 7fc7b1cea640 1 -- 172.21.15.161:0/4039100592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7ac151f80 msgr2=0x7fc7ac14cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:56.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.131+0000 7fc7b1cea640 1 --2- 172.21.15.161:0/4039100592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7ac151f80 0x7fc7ac14cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:56.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.131+0000 7fc7b1cea640 1 -- 172.21.15.161:0/4039100592 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7ac0c1bb0 con 0x7fc7ac14f730 2024-03-28T23:35:56.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.131+0000 7fc7b1cea640 1 --2- 172.21.15.161:0/4039100592 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7ac14f730 0x7fc7ac14c9e0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fc7a800c400 tx=0x7fc7a800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:56.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.131+0000 7fc7a2ffd640 1 -- 172.21.15.161:0/4039100592 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a8017070 con 0x7fc7ac14f730 2024-03-28T23:35:56.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.131+0000 7fc7a2ffd640 1 -- 172.21.15.161:0/4039100592 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7a8005d00 con 0x7fc7ac14f730 2024-03-28T23:35:56.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.131+0000 7fc7a2ffd640 1 -- 172.21.15.161:0/4039100592 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a8007310 con 0x7fc7ac14f730 2024-03-28T23:35:56.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.131+0000 7fc7b474f640 1 -- 172.21.15.161:0/4039100592 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7ac0c1030 con 0x7fc7ac14f730 2024-03-28T23:35:56.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.131+0000 7fc7b474f640 1 -- 172.21.15.161:0/4039100592 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7ac0c15f0 con 0x7fc7ac14f730 2024-03-28T23:35:56.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.132+0000 7fc7b474f640 1 -- 172.21.15.161:0/4039100592 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7ac06c160 con 0x7fc7ac14f730 2024-03-28T23:35:56.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.133+0000 7fc7a2ffd640 1 -- 172.21.15.161:0/4039100592 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc7a801a070 con 0x7fc7ac14f730 2024-03-28T23:35:56.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.133+0000 7fc7a2ffd640 1 --2- 172.21.15.161:0/4039100592 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc79006b4d0 0x7fc79006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:56.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.133+0000 7fc7a2ffd640 1 -- 172.21.15.161:0/4039100592 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 5249+0+0 (secure 0 0 0) 0x7fc7a8089320 con 0x7fc7ac14f730 2024-03-28T23:35:56.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.133+0000 7fc7b24eb640 1 --2- 172.21.15.161:0/4039100592 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc79006b4d0 0x7fc79006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:56.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.134+0000 7fc7b24eb640 1 --2- 172.21.15.161:0/4039100592 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc79006b4d0 0x7fc79006d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fc79c036000 tx=0x7fc79c03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:56.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.138+0000 7fc7a2ffd640 1 -- 172.21.15.161:0/4039100592 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc7ac06c160 con 0x7fc7ac14f730 2024-03-28T23:35:56.291 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:35:56.293 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:35:56.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.313+0000 7fc7b474f640 1 -- 172.21.15.161:0/4039100592 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fc7ac0abee0 con 0x7fc7ac14f730 2024-03-28T23:35:56.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.313+0000 7fc7a2ffd640 1 -- 172.21.15.161:0/4039100592 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v148) ==== 84+0+65 (secure 0 0 0) 0x7fc7a8057da0 con 0x7fc7ac14f730 2024-03-28T23:35:56.313 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:56.313 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2024-03-28T23:35:56.314 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:35:56.314 INFO:teuthology.orchestra.run.smithi161.stdout: "poolnum": 1, 2024-03-28T23:35:56.314 INFO:teuthology.orchestra.run.smithi161.stdout: "poolname": ".mgr" 2024-03-28T23:35:56.314 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:35:56.314 INFO:teuthology.orchestra.run.smithi161.stdout:] 2024-03-28T23:35:56.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.315+0000 7fc7b474f640 1 -- 172.21.15.161:0/4039100592 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc79006b4d0 msgr2=0x7fc79006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:56.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.315+0000 7fc7b474f640 1 --2- 172.21.15.161:0/4039100592 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc79006b4d0 0x7fc79006d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fc79c036000 tx=0x7fc79c03d040 comp rx=0 tx=0).stop 2024-03-28T23:35:56.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.315+0000 7fc7b474f640 1 -- 172.21.15.161:0/4039100592 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7ac14f730 msgr2=0x7fc7ac14c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:56.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.315+0000 7fc7b474f640 1 --2- 172.21.15.161:0/4039100592 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7ac14f730 0x7fc7ac14c9e0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fc7a800c400 tx=0x7fc7a800c8c0 comp rx=0 tx=0).stop 2024-03-28T23:35:56.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.315+0000 7fc7b474f640 1 -- 172.21.15.161:0/4039100592 shutdown_connections 2024-03-28T23:35:56.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.315+0000 7fc7b474f640 1 --2- 172.21.15.161:0/4039100592 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc79006b4d0 0x7fc79006d990 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:56.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.315+0000 7fc7b474f640 1 --2- 172.21.15.161:0/4039100592 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7ac14f730 0x7fc7ac14c9e0 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:56.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.315+0000 7fc7b474f640 1 --2- 172.21.15.161:0/4039100592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7ac151f80 0x7fc7ac14cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:56.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.315+0000 7fc7b474f640 1 --2- 172.21.15.161:0/4039100592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7ac0ac310 0x7fc7ac0c4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:56.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.315+0000 7fc7b474f640 1 -- 172.21.15.161:0/4039100592 >> 172.21.15.161:0/4039100592 conn(0x7fc7ac0c66d0 msgr2=0x7fc7ac14ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:56.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.315+0000 7fc7b474f640 1 -- 172.21.15.161:0/4039100592 shutdown_connections 2024-03-28T23:35:56.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.315+0000 7fc7b474f640 1 -- 172.21.15.161:0/4039100592 wait complete. 2024-03-28T23:35:56.327 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-28T23:35:56.327 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-28T23:35:56.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.486+0000 7f211b95e640 1 Processor -- start 2024-03-28T23:35:56.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.486+0000 7f211b95e640 1 -- start start 2024-03-28T23:35:56.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.487+0000 7f211b95e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f21140ac430 0x7f2114152880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:56.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.487+0000 7f211b95e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21140c3540 con 0x7f21140c30c0 2024-03-28T23:35:56.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.487+0000 7f211b95e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21140c3820 con 0x7f21140c19b0 2024-03-28T23:35:56.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.487+0000 7f211b95e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f2114152dc0 con 0x7f21140ac430 2024-03-28T23:35:56.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.487+0000 7f2118ef9640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f21140c30c0 0x7f21140c24d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:54350/0 (socket says 172.21.15.161:54350) 2024-03-28T23:35:56.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.487+0000 7f2118ef9640 1 -- 172.21.15.161:0/1635318790 learned_addr learned my addr 172.21.15.161:0/1635318790 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:56.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.487+0000 7f210bfff640 1 -- 172.21.15.161:0/1635318790 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2100388755 0 0) 0x7f21140c3540 con 0x7f21140c30c0 2024-03-28T23:35:56.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.487+0000 7f210bfff640 1 -- 172.21.15.161:0/1635318790 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2100003520 con 0x7f21140c30c0 2024-03-28T23:35:56.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.487+0000 7f21196fa640 1 --2- 172.21.15.161:0/1635318790 >> v2:172.21.15.170:3300/0 conn(0x7f21140ac430 0x7f2114152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:56.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.488+0000 7f210bfff640 1 -- 172.21.15.161:0/1635318790 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4201377965 0 0) 0x7f2100003520 con 0x7f21140c30c0 2024-03-28T23:35:56.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.488+0000 7f210bfff640 1 -- 172.21.15.161:0/1635318790 >> v2:172.21.15.170:3300/0 conn(0x7f21140ac430 msgr2=0x7f2114152880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:56.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.488+0000 7f210bfff640 1 --2- 172.21.15.161:0/1635318790 >> v2:172.21.15.170:3300/0 conn(0x7f21140ac430 0x7f2114152880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:56.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.488+0000 7f210bfff640 1 -- 172.21.15.161:0/1635318790 >> v1:172.21.15.170:6789/0 conn(0x7f21140c19b0 legacy=0x7f211415c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:56.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.488+0000 7f210bfff640 1 -- 172.21.15.161:0/1635318790 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21140c3b00 con 0x7f21140c30c0 2024-03-28T23:35:56.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.488+0000 7f210bfff640 1 -- 172.21.15.161:0/1635318790 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f2104001670 con 0x7f21140c30c0 2024-03-28T23:35:56.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.488+0000 7f210bfff640 1 -- 172.21.15.161:0/1635318790 >> v1:172.21.15.161:6789/0 conn(0x7f21140c30c0 legacy=0x7f21140c24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:56.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.488+0000 7f210bfff640 1 --2- 172.21.15.161:0/1635318790 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21000039b0 0x7f2100003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:56.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.488+0000 7f210bfff640 1 --2- 172.21.15.161:0/1635318790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2100004800 0x7f2100004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:56.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.488+0000 7f210bfff640 1 --2- 172.21.15.161:0/1635318790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21000051c0 0x7f210000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:56.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.488+0000 7f210bfff640 1 -- 172.21.15.161:0/1635318790 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2114152dc0 con 0x7f21000039b0 2024-03-28T23:35:56.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.488+0000 7f210bfff640 1 -- 172.21.15.161:0/1635318790 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f210000c160 con 0x7f21000051c0 2024-03-28T23:35:56.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.488+0000 7f210bfff640 1 -- 172.21.15.161:0/1635318790 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f210000c360 con 0x7f2100004800 2024-03-28T23:35:56.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.488+0000 7f21196fa640 1 --2- 172.21.15.161:0/1635318790 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21000039b0 0x7f2100003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:56.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.488+0000 7f2118ef9640 1 --2- 172.21.15.161:0/1635318790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2100004800 0x7f2100004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:56.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.488+0000 7f2119efb640 1 --2- 172.21.15.161:0/1635318790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21000051c0 0x7f210000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:56.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.488+0000 7f21196fa640 1 -- 172.21.15.161:0/1635318790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2100004800 msgr2=0x7f2100004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:56.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.488+0000 7f21196fa640 1 --2- 172.21.15.161:0/1635318790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2100004800 0x7f2100004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:56.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.488+0000 7f21196fa640 1 -- 172.21.15.161:0/1635318790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21000051c0 msgr2=0x7f210000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:56.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.488+0000 7f21196fa640 1 --2- 172.21.15.161:0/1635318790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21000051c0 0x7f210000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:56.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.488+0000 7f21196fa640 1 -- 172.21.15.161:0/1635318790 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2100003520 con 0x7f21000039b0 2024-03-28T23:35:56.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.488+0000 7f2119efb640 1 -- 172.21.15.161:0/1635318790 reap_dead start 2024-03-28T23:35:56.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.489+0000 7f21196fa640 1 --2- 172.21.15.161:0/1635318790 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21000039b0 0x7f2100003dd0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f2110009ac0 tx=0x7f2100005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ff2aa4e694ccf35 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:56.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.489+0000 7f210bfff640 1 -- 172.21.15.161:0/1635318790 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2110039070 con 0x7f21000039b0 2024-03-28T23:35:56.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.489+0000 7f210bfff640 1 -- 172.21.15.161:0/1635318790 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f211002c6b0 con 0x7f21000039b0 2024-03-28T23:35:56.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.489+0000 7f210bfff640 1 -- 172.21.15.161:0/1635318790 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2110009690 con 0x7f21000039b0 2024-03-28T23:35:56.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.489+0000 7f211b95e640 1 -- 172.21.15.161:0/1635318790 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21000039b0 msgr2=0x7f2100003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:56.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.489+0000 7f211b95e640 1 --2- 172.21.15.161:0/1635318790 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21000039b0 0x7f2100003dd0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f2110009ac0 tx=0x7f2100005830 comp rx=0 tx=0).stop 2024-03-28T23:35:56.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.489+0000 7f211b95e640 1 -- 172.21.15.161:0/1635318790 shutdown_connections 2024-03-28T23:35:56.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.489+0000 7f211b95e640 1 --2- 172.21.15.161:0/1635318790 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21000039b0 0x7f2100003dd0 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:56.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.489+0000 7f211b95e640 1 -- 172.21.15.161:0/1635318790 >> 172.21.15.161:0/1635318790 conn(0x7f211406b150 msgr2=0x7f211406b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:56.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.490+0000 7f211b95e640 1 -- 172.21.15.161:0/1635318790 shutdown_connections 2024-03-28T23:35:56.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.490+0000 7f211b95e640 1 -- 172.21.15.161:0/1635318790 wait complete. 2024-03-28T23:35:56.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.491+0000 7f211b95e640 1 Processor -- start 2024-03-28T23:35:56.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.491+0000 7f211b95e640 1 -- start start 2024-03-28T23:35:56.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.491+0000 7f211b95e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21000039b0 0x7f21141dde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:56.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.491+0000 7f211b95e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2114153440 0x7f21141538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:56.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.491+0000 7f211b95e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21140a64d0 0x7f21140a6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:56.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.491+0000 7f211b95e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f21140c6440 con 0x7f21000039b0 2024-03-28T23:35:56.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.491+0000 7f211b95e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f21140c1490 con 0x7f2114153440 2024-03-28T23:35:56.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.491+0000 7f211b95e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f21140ac810 con 0x7f21140a64d0 2024-03-28T23:35:56.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.491+0000 7f21196fa640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21000039b0 0x7f21141dde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:56.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.491+0000 7f21196fa640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21000039b0 0x7f21141dde70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57602/0 (socket says 172.21.15.161:57602) 2024-03-28T23:35:56.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.491+0000 7f21196fa640 1 -- 172.21.15.161:0/1976339145 learned_addr learned my addr 172.21.15.161:0/1976339145 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:56.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.492+0000 7f21196fa640 1 -- 172.21.15.161:0/1976339145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21140a64d0 msgr2=0x7f21140a6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:56.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.492+0000 7f21196fa640 1 --2- 172.21.15.161:0/1976339145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21140a64d0 0x7f21140a6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:56.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.492+0000 7f21196fa640 1 -- 172.21.15.161:0/1976339145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2114153440 msgr2=0x7f21141538c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:56.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.492+0000 7f21196fa640 1 --2- 172.21.15.161:0/1976339145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2114153440 0x7f21141538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:56.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.492+0000 7f21196fa640 1 -- 172.21.15.161:0/1976339145 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2114156170 con 0x7f21000039b0 2024-03-28T23:35:56.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.492+0000 7f21196fa640 1 --2- 172.21.15.161:0/1976339145 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21000039b0 0x7f21141dde70 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f2110000c00 tx=0x7f2110004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:56.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.492+0000 7f2109ffb640 1 -- 172.21.15.161:0/1976339145 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2110042020 con 0x7f21000039b0 2024-03-28T23:35:56.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.492+0000 7f2109ffb640 1 -- 172.21.15.161:0/1976339145 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f211003a040 con 0x7f21000039b0 2024-03-28T23:35:56.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.492+0000 7f211b95e640 1 -- 172.21.15.161:0/1976339145 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21140a8420 con 0x7f21000039b0 2024-03-28T23:35:56.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.492+0000 7f2109ffb640 1 -- 172.21.15.161:0/1976339145 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2110039070 con 0x7f21000039b0 2024-03-28T23:35:56.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.492+0000 7f211b95e640 1 -- 172.21.15.161:0/1976339145 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21140a89e0 con 0x7f21000039b0 2024-03-28T23:35:56.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.493+0000 7f211b95e640 1 -- 172.21.15.161:0/1976339145 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2114068f90 con 0x7f21000039b0 2024-03-28T23:35:56.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.493+0000 7f2109ffb640 1 -- 172.21.15.161:0/1976339145 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2110045020 con 0x7f21000039b0 2024-03-28T23:35:56.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.494+0000 7f2109ffb640 1 --2- 172.21.15.161:0/1976339145 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f20f806b500 0x7f20f806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:56.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.494+0000 7f2109ffb640 1 -- 172.21.15.161:0/1976339145 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 5249+0+0 (secure 0 0 0) 0x7f21100ab310 con 0x7f21000039b0 2024-03-28T23:35:56.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.494+0000 7f2118ef9640 1 --2- 172.21.15.161:0/1976339145 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f20f806b500 0x7f20f806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:56.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.494+0000 7f2118ef9640 1 --2- 172.21.15.161:0/1976339145 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f20f806b500 0x7f20f806d9c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f2104001eb0 tx=0x7f2104000ed0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:56.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.498+0000 7f2109ffb640 1 -- 172.21.15.161:0/1976339145 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f211004c070 con 0x7f21000039b0 2024-03-28T23:35:56.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:56.668+0000 7f211b95e640 1 -- 172.21.15.161:0/1976339145 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f21140c0c50 con 0x7f21000039b0 2024-03-28T23:35:57.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.046+0000 7f2109ffb640 1 -- 172.21.15.161:0/1976339145 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v149) ==== 138+0+0 (secure 0 0 0) 0x7f21100a9e20 con 0x7f21000039b0 2024-03-28T23:35:57.046 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_metadata' created 2024-03-28T23:35:57.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.050+0000 7f211b95e640 1 -- 172.21.15.161:0/1976339145 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f20f806b500 msgr2=0x7f20f806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:57.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.050+0000 7f211b95e640 1 --2- 172.21.15.161:0/1976339145 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f20f806b500 0x7f20f806d9c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f2104001eb0 tx=0x7f2104000ed0 comp rx=0 tx=0).stop 2024-03-28T23:35:57.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.050+0000 7f211b95e640 1 -- 172.21.15.161:0/1976339145 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21000039b0 msgr2=0x7f21141dde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:57.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.050+0000 7f211b95e640 1 --2- 172.21.15.161:0/1976339145 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21000039b0 0x7f21141dde70 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f2110000c00 tx=0x7f2110004730 comp rx=0 tx=0).stop 2024-03-28T23:35:57.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.051+0000 7f211b95e640 1 -- 172.21.15.161:0/1976339145 shutdown_connections 2024-03-28T23:35:57.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.051+0000 7f211b95e640 1 --2- 172.21.15.161:0/1976339145 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f20f806b500 0x7f20f806d9c0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:57.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.051+0000 7f211b95e640 1 --2- 172.21.15.161:0/1976339145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21140a64d0 0x7f21140a6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:57.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.051+0000 7f211b95e640 1 --2- 172.21.15.161:0/1976339145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2114153440 0x7f21141538c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:57.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.051+0000 7f211b95e640 1 --2- 172.21.15.161:0/1976339145 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21000039b0 0x7f21141dde70 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:57.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.051+0000 7f211b95e640 1 -- 172.21.15.161:0/1976339145 >> 172.21.15.161:0/1976339145 conn(0x7f211406b150 msgr2=0x7f2114150d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:57.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.052+0000 7f211b95e640 1 -- 172.21.15.161:0/1976339145 shutdown_connections 2024-03-28T23:35:57.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.052+0000 7f211b95e640 1 -- 172.21.15.161:0/1976339145 wait complete. 2024-03-28T23:35:57.065 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T23:35:57.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.174+0000 7f894b8cd640 1 Processor -- start 2024-03-28T23:35:57.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.174+0000 7f894b8cd640 1 -- start start 2024-03-28T23:35:57.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.175+0000 7f894b8cd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8944152bd0 0x7f894415d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:57.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.175+0000 7f894b8cd640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89440c2f10 con 0x7f894406cdd0 2024-03-28T23:35:57.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.175+0000 7f894b8cd640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f894414f9a0 con 0x7f894414f5c0 2024-03-28T23:35:57.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.175+0000 7f894b8cd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f89440c74d0 con 0x7f8944152bd0 2024-03-28T23:35:57.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.175+0000 7f8948e68640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f894406cdd0 0x7f89441524c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:54360/0 (socket says 172.21.15.161:54360) 2024-03-28T23:35:57.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.175+0000 7f8948e68640 1 -- 172.21.15.161:0/2165842676 learned_addr learned my addr 172.21.15.161:0/2165842676 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:57.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.175+0000 7f893bfff640 1 -- 172.21.15.161:0/2165842676 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2659509736 0 0) 0x7f8934001110 con 0x7f894406cdd0 2024-03-28T23:35:57.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.175+0000 7f893bfff640 1 -- 172.21.15.161:0/2165842676 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8930003520 con 0x7f894406cdd0 2024-03-28T23:35:57.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.175+0000 7f8949e6a640 1 --2- 172.21.15.161:0/2165842676 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8944152bd0 0x7f894415d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:57.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.175+0000 7f893bfff640 1 -- 172.21.15.161:0/2165842676 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2976705716 0 0) 0x7f8930003520 con 0x7f894406cdd0 2024-03-28T23:35:57.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.175+0000 7f893bfff640 1 -- 172.21.15.161:0/2165842676 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8944152bd0 msgr2=0x7f894415d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:57.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.175+0000 7f893bfff640 1 --2- 172.21.15.161:0/2165842676 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8944152bd0 0x7f894415d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:57.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.175+0000 7f893bfff640 1 -- 172.21.15.161:0/2165842676 >> v1:172.21.15.170:6789/0 conn(0x7f894414f5c0 legacy=0x7f89440c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:57.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.175+0000 7f893bfff640 1 -- 172.21.15.161:0/2165842676 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f894406c140 con 0x7f894406cdd0 2024-03-28T23:35:57.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.175+0000 7f893bfff640 1 -- 172.21.15.161:0/2165842676 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f8934001840 con 0x7f894406cdd0 2024-03-28T23:35:57.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.176+0000 7f893bfff640 1 -- 172.21.15.161:0/2165842676 >> v1:172.21.15.161:6789/0 conn(0x7f894406cdd0 legacy=0x7f89441524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:57.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.176+0000 7f893bfff640 1 --2- 172.21.15.161:0/2165842676 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8930003ac0 0x7f8930003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:57.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.176+0000 7f893bfff640 1 --2- 172.21.15.161:0/2165842676 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8944152bd0 0x7f8930004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:57.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.176+0000 7f893bfff640 1 --2- 172.21.15.161:0/2165842676 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8930004d40 0x7f89300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:57.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.176+0000 7f893bfff640 1 -- 172.21.15.161:0/2165842676 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f89440c74d0 con 0x7f8930003ac0 2024-03-28T23:35:57.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.176+0000 7f893bfff640 1 -- 172.21.15.161:0/2165842676 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f893000bb50 con 0x7f8930004d40 2024-03-28T23:35:57.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.176+0000 7f893bfff640 1 -- 172.21.15.161:0/2165842676 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f893000df60 con 0x7f8944152bd0 2024-03-28T23:35:57.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.176+0000 7f8949669640 1 --2- 172.21.15.161:0/2165842676 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8930003ac0 0x7f8930003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:57.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.176+0000 7f8948e68640 1 --2- 172.21.15.161:0/2165842676 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8944152bd0 0x7f8930004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:57.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.176+0000 7f8949e6a640 1 --2- 172.21.15.161:0/2165842676 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8930004d40 0x7f89300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:57.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.176+0000 7f8949669640 1 -- 172.21.15.161:0/2165842676 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8944152bd0 msgr2=0x7f8930004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:57.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.176+0000 7f8949669640 1 --2- 172.21.15.161:0/2165842676 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8944152bd0 0x7f8930004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:57.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.176+0000 7f8949669640 1 -- 172.21.15.161:0/2165842676 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8930004d40 msgr2=0x7f89300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:57.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.176+0000 7f8949669640 1 --2- 172.21.15.161:0/2165842676 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8930004d40 0x7f89300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:57.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.176+0000 7f8949669640 1 -- 172.21.15.161:0/2165842676 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8930003520 con 0x7f8930003ac0 2024-03-28T23:35:57.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.176+0000 7f8949e6a640 1 --2- 172.21.15.161:0/2165842676 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8930004d40 0x7f89300051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:57.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.177+0000 7f8949669640 1 --2- 172.21.15.161:0/2165842676 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8930003ac0 0x7f8930003ee0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f8940001750 tx=0x7f8930005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=726d2e40723c1dc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:57.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.177+0000 7f893bfff640 1 -- 172.21.15.161:0/2165842676 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8940001370 con 0x7f8930003ac0 2024-03-28T23:35:57.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.177+0000 7f893bfff640 1 -- 172.21.15.161:0/2165842676 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89400052d0 con 0x7f8930003ac0 2024-03-28T23:35:57.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.177+0000 7f893bfff640 1 -- 172.21.15.161:0/2165842676 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8940003830 con 0x7f8930003ac0 2024-03-28T23:35:57.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.177+0000 7f894b8cd640 1 -- 172.21.15.161:0/2165842676 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8930003ac0 msgr2=0x7f8930003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:57.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.177+0000 7f894b8cd640 1 --2- 172.21.15.161:0/2165842676 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8930003ac0 0x7f8930003ee0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f8940001750 tx=0x7f8930005830 comp rx=0 tx=0).stop 2024-03-28T23:35:57.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.177+0000 7f8949e6a640 1 -- 172.21.15.161:0/2165842676 reap_dead start 2024-03-28T23:35:57.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.177+0000 7f894b8cd640 1 -- 172.21.15.161:0/2165842676 shutdown_connections 2024-03-28T23:35:57.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.177+0000 7f894b8cd640 1 -- 172.21.15.161:0/2165842676 >> 172.21.15.161:0/2165842676 conn(0x7f89440a83f0 msgr2=0x7f89440a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:57.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.177+0000 7f894b8cd640 1 -- 172.21.15.161:0/2165842676 shutdown_connections 2024-03-28T23:35:57.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.177+0000 7f894b8cd640 1 -- 172.21.15.161:0/2165842676 wait complete. 2024-03-28T23:35:57.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.179+0000 7f894b8cd640 1 Processor -- start 2024-03-28T23:35:57.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.179+0000 7f894b8cd640 1 -- start start 2024-03-28T23:35:57.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.179+0000 7f894b8cd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8930003ac0 0x7f8944073220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:57.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.179+0000 7f894b8cd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8944155d80 0x7f89440c4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:57.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.179+0000 7f894b8cd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89441ea3e0 0x7f8944155230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:57.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.179+0000 7f894b8cd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8944152030 con 0x7f89441ea3e0 2024-03-28T23:35:57.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.179+0000 7f894b8cd640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f89440c6fa0 con 0x7f8944155d80 2024-03-28T23:35:57.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.179+0000 7f894b8cd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f894406c3e0 con 0x7f8930003ac0 2024-03-28T23:35:57.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.179+0000 7f8949e6a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89441ea3e0 0x7f8944155230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:57.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.179+0000 7f8949e6a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89441ea3e0 0x7f8944155230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57626/0 (socket says 172.21.15.161:57626) 2024-03-28T23:35:57.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.179+0000 7f8949e6a640 1 -- 172.21.15.161:0/954037762 learned_addr learned my addr 172.21.15.161:0/954037762 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:57.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.180+0000 7f8949669640 1 --2- 172.21.15.161:0/954037762 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8930003ac0 0x7f8944073220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:57.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.180+0000 7f8948e68640 1 --2- 172.21.15.161:0/954037762 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8944155d80 0x7f89440c4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:57.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.180+0000 7f8949e6a640 1 -- 172.21.15.161:0/954037762 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8930003ac0 msgr2=0x7f8944073220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:57.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.180+0000 7f8949e6a640 1 --2- 172.21.15.161:0/954037762 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8930003ac0 0x7f8944073220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:57.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.180+0000 7f8949e6a640 1 -- 172.21.15.161:0/954037762 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8944155d80 msgr2=0x7f89440c4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:57.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.180+0000 7f8949e6a640 1 --2- 172.21.15.161:0/954037762 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8944155d80 0x7f89440c4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:57.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.180+0000 7f8949e6a640 1 -- 172.21.15.161:0/954037762 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8944071360 con 0x7f89441ea3e0 2024-03-28T23:35:57.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.180+0000 7f8948e68640 1 --2- 172.21.15.161:0/954037762 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8944155d80 0x7f89440c4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:35:57.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.180+0000 7f8949e6a640 1 --2- 172.21.15.161:0/954037762 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89441ea3e0 0x7f8944155230 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f893c007cf0 tx=0x7f893c010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:57.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.180+0000 7f8939ffb640 1 -- 172.21.15.161:0/954037762 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f893c01a030 con 0x7f89441ea3e0 2024-03-28T23:35:57.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.180+0000 7f8939ffb640 1 -- 172.21.15.161:0/954037762 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f893c010dd0 con 0x7f89441ea3e0 2024-03-28T23:35:57.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.180+0000 7f8939ffb640 1 -- 172.21.15.161:0/954037762 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f893c003b90 con 0x7f89441ea3e0 2024-03-28T23:35:57.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.180+0000 7f894b8cd640 1 -- 172.21.15.161:0/954037762 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89441ead00 con 0x7f89441ea3e0 2024-03-28T23:35:57.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.180+0000 7f894b8cd640 1 -- 172.21.15.161:0/954037762 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8944071970 con 0x7f89441ea3e0 2024-03-28T23:35:57.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.181+0000 7f894b8cd640 1 -- 172.21.15.161:0/954037762 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f894404f4b0 con 0x7f89441ea3e0 2024-03-28T23:35:57.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.182+0000 7f8939ffb640 1 -- 172.21.15.161:0/954037762 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f893c005d10 con 0x7f89441ea3e0 2024-03-28T23:35:57.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.182+0000 7f8939ffb640 1 --2- 172.21.15.161:0/954037762 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f892806b4d0 0x7f892806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:57.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.182+0000 7f8939ffb640 1 -- 172.21.15.161:0/954037762 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 5619+0+0 (secure 0 0 0) 0x7f893c08aa90 con 0x7f89441ea3e0 2024-03-28T23:35:57.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.182+0000 7f8949669640 1 --2- 172.21.15.161:0/954037762 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f892806b4d0 0x7f892806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:57.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.183+0000 7f8949669640 1 --2- 172.21.15.161:0/954037762 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f892806b4d0 0x7f892806d990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f8940033000 tx=0x7f8940009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:57.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.187+0000 7f8939ffb640 1 -- 172.21.15.161:0/954037762 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f893c024070 con 0x7f89441ea3e0 2024-03-28T23:35:57.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:57.359+0000 7f894b8cd640 1 -- 172.21.15.161:0/954037762 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f894414bca0 con 0x7f89441ea3e0 2024-03-28T23:35:58.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.050+0000 7f8939ffb640 1 -- 172.21.15.161:0/954037762 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v150) ==== 170+0+0 (secure 0 0 0) 0x7f893c0599f0 con 0x7f89441ea3e0 2024-03-28T23:35:58.051 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data' created 2024-03-28T23:35:58.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.053+0000 7f894b8cd640 1 -- 172.21.15.161:0/954037762 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f892806b4d0 msgr2=0x7f892806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:58.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.053+0000 7f894b8cd640 1 --2- 172.21.15.161:0/954037762 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f892806b4d0 0x7f892806d990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f8940033000 tx=0x7f8940009030 comp rx=0 tx=0).stop 2024-03-28T23:35:58.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.053+0000 7f894b8cd640 1 -- 172.21.15.161:0/954037762 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89441ea3e0 msgr2=0x7f8944155230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:58.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.053+0000 7f894b8cd640 1 --2- 172.21.15.161:0/954037762 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89441ea3e0 0x7f8944155230 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f893c007cf0 tx=0x7f893c010590 comp rx=0 tx=0).stop 2024-03-28T23:35:58.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.053+0000 7f894b8cd640 1 -- 172.21.15.161:0/954037762 shutdown_connections 2024-03-28T23:35:58.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.053+0000 7f894b8cd640 1 --2- 172.21.15.161:0/954037762 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f892806b4d0 0x7f892806d990 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:58.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.053+0000 7f894b8cd640 1 --2- 172.21.15.161:0/954037762 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89441ea3e0 0x7f8944155230 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:58.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.053+0000 7f894b8cd640 1 --2- 172.21.15.161:0/954037762 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8944155d80 0x7f89440c4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:58.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.053+0000 7f894b8cd640 1 --2- 172.21.15.161:0/954037762 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8930003ac0 0x7f8944073220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:58.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.053+0000 7f894b8cd640 1 -- 172.21.15.161:0/954037762 >> 172.21.15.161:0/954037762 conn(0x7f89440a83f0 msgr2=0x7f89440aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:58.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.053+0000 7f894b8cd640 1 -- 172.21.15.161:0/954037762 shutdown_connections 2024-03-28T23:35:58.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.053+0000 7f894b8cd640 1 -- 172.21.15.161:0/954037762 wait complete. 2024-03-28T23:35:58.082 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-28T23:35:58.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.186+0000 7f31dd8da640 1 Processor -- start 2024-03-28T23:35:58.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.186+0000 7f31dd8da640 1 -- start start 2024-03-28T23:35:58.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.187+0000 7f31dd8da640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f31d8152940 0x7f31d80c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:58.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.187+0000 7f31dd8da640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f31d80ca580 0x7f31d80c8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:58.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.187+0000 7f31dd8da640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31d80c6df0 con 0x7f31d80c9480 2024-03-28T23:35:58.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.187+0000 7f31dd8da640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f31d80c70d0 con 0x7f31d80ca580 2024-03-28T23:35:58.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.187+0000 7f31dd8da640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f31d80c5830 con 0x7f31d8152940 2024-03-28T23:35:58.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.187+0000 7f31d67fc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f31d80ca580 0x7f31d80c8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:58.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.187+0000 7f31d77fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f31d80c9480 0x7f31d8155220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:54372/0 (socket says 172.21.15.161:54372) 2024-03-28T23:35:58.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.187+0000 7f31d67fc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f31d80ca580 0x7f31d80c8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57638/0 (socket says 172.21.15.161:57638) 2024-03-28T23:35:58.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.187+0000 7f31d77fe640 1 -- 172.21.15.161:0/3962519104 learned_addr learned my addr 172.21.15.161:0/3962519104 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:58.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.187+0000 7f31d6ffd640 1 --2- 172.21.15.161:0/3962519104 >> v2:172.21.15.170:3300/0 conn(0x7f31d8152940 0x7f31d80c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:58.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.187+0000 7f31d67fc640 1 -- 172.21.15.161:0/3962519104 >> v2:172.21.15.170:3300/0 conn(0x7f31d8152940 msgr2=0x7f31d80c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:58.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.187+0000 7f31d67fc640 1 --2- 172.21.15.161:0/3962519104 >> v2:172.21.15.170:3300/0 conn(0x7f31d8152940 0x7f31d80c8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:58.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.187+0000 7f31d67fc640 1 -- 172.21.15.161:0/3962519104 >> v1:172.21.15.161:6789/0 conn(0x7f31d80c9480 legacy=0x7f31d8155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:58.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.187+0000 7f31d67fc640 1 -- 172.21.15.161:0/3962519104 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d80c5b10 con 0x7f31d80ca580 2024-03-28T23:35:58.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.187+0000 7f31d5ffb640 1 -- 172.21.15.161:0/3962519104 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2330504500 0 0) 0x7f31d80c6df0 con 0x7f31d80c9480 2024-03-28T23:35:58.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.188+0000 7f31d67fc640 1 --2- 172.21.15.161:0/3962519104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31d80ca580 0x7f31d80c8f40 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f31d80aa440 tx=0x7f31cc02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b919f9654c23d2e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:58.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.188+0000 7f31d5ffb640 1 -- 172.21.15.161:0/3962519104 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc038070 con 0x7f31d80ca580 2024-03-28T23:35:58.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.188+0000 7f31d5ffb640 1 -- 172.21.15.161:0/3962519104 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31cc031710 con 0x7f31d80ca580 2024-03-28T23:35:58.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.188+0000 7f31d5ffb640 1 -- 172.21.15.161:0/3962519104 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc031ae0 con 0x7f31d80ca580 2024-03-28T23:35:58.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.189+0000 7f31dd8da640 1 -- 172.21.15.161:0/3962519104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31d80ca580 msgr2=0x7f31d80c8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:58.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.189+0000 7f31dd8da640 1 --2- 172.21.15.161:0/3962519104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31d80ca580 0x7f31d80c8f40 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f31d80aa440 tx=0x7f31cc02b7b0 comp rx=0 tx=0).stop 2024-03-28T23:35:58.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.189+0000 7f31dd8da640 1 -- 172.21.15.161:0/3962519104 shutdown_connections 2024-03-28T23:35:58.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.189+0000 7f31dd8da640 1 --2- 172.21.15.161:0/3962519104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31d80ca580 0x7f31d80c8f40 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:58.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.189+0000 7f31dd8da640 1 --2- 172.21.15.161:0/3962519104 >> v2:172.21.15.170:3300/0 conn(0x7f31d8152940 0x7f31d80c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:58.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.189+0000 7f31dd8da640 1 -- 172.21.15.161:0/3962519104 >> 172.21.15.161:0/3962519104 conn(0x7f31d80caa00 msgr2=0x7f31d80cae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:58.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.189+0000 7f31dd8da640 1 -- 172.21.15.161:0/3962519104 shutdown_connections 2024-03-28T23:35:58.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.189+0000 7f31dd8da640 1 -- 172.21.15.161:0/3962519104 wait complete. 2024-03-28T23:35:58.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.190+0000 7f31dd8da640 1 Processor -- start 2024-03-28T23:35:58.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.190+0000 7f31dd8da640 1 -- start start 2024-03-28T23:35:58.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.190+0000 7f31dd8da640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31d80c9480 0x7f31d814dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:58.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.190+0000 7f31dd8da640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31d80ca580 0x7f31d814dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:58.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.190+0000 7f31dd8da640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31d8152940 0x7f31d81ee630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:58.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.190+0000 7f31dd8da640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f31d80c61c0 con 0x7f31d80c9480 2024-03-28T23:35:58.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.190+0000 7f31dd8da640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f31d80c6040 con 0x7f31d80ca580 2024-03-28T23:35:58.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.190+0000 7f31dd8da640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f31d8155930 con 0x7f31d8152940 2024-03-28T23:35:58.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.191+0000 7f31d6ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31d80c9480 0x7f31d814dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:58.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.191+0000 7f31d6ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31d80c9480 0x7f31d814dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57650/0 (socket says 172.21.15.161:57650) 2024-03-28T23:35:58.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.191+0000 7f31d6ffd640 1 -- 172.21.15.161:0/2560249822 learned_addr learned my addr 172.21.15.161:0/2560249822 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:58.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.191+0000 7f31d6ffd640 1 -- 172.21.15.161:0/2560249822 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31d8152940 msgr2=0x7f31d81ee630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:58.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.191+0000 7f31d6ffd640 1 --2- 172.21.15.161:0/2560249822 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31d8152940 0x7f31d81ee630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:58.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.191+0000 7f31d6ffd640 1 -- 172.21.15.161:0/2560249822 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31d80ca580 msgr2=0x7f31d814dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:58.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.191+0000 7f31d6ffd640 1 --2- 172.21.15.161:0/2560249822 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31d80ca580 0x7f31d814dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:58.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.191+0000 7f31d6ffd640 1 -- 172.21.15.161:0/2560249822 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d8067af0 con 0x7f31d80c9480 2024-03-28T23:35:58.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.191+0000 7f31d6ffd640 1 --2- 172.21.15.161:0/2560249822 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31d80c9480 0x7f31d814dab0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f31c8009d10 tx=0x7f31c800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:58.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.191+0000 7f31bffff640 1 -- 172.21.15.161:0/2560249822 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c8018020 con 0x7f31d80c9480 2024-03-28T23:35:58.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.191+0000 7f31bffff640 1 -- 172.21.15.161:0/2560249822 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31c8014480 con 0x7f31d80c9480 2024-03-28T23:35:58.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.191+0000 7f31dd8da640 1 -- 172.21.15.161:0/2560249822 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31d80c4510 con 0x7f31d80c9480 2024-03-28T23:35:58.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.191+0000 7f31bffff640 1 -- 172.21.15.161:0/2560249822 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c8014830 con 0x7f31d80c9480 2024-03-28T23:35:58.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.191+0000 7f31dd8da640 1 -- 172.21.15.161:0/2560249822 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31d81ec510 con 0x7f31d80c9480 2024-03-28T23:35:58.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.192+0000 7f31dd8da640 1 -- 172.21.15.161:0/2560249822 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31d8151170 con 0x7f31d80c9480 2024-03-28T23:35:58.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.193+0000 7f31bffff640 1 -- 172.21.15.161:0/2560249822 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f31c8005d10 con 0x7f31d80c9480 2024-03-28T23:35:58.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.193+0000 7f31bffff640 1 --2- 172.21.15.161:0/2560249822 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f31c406b4d0 0x7f31c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:58.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.193+0000 7f31bffff640 1 -- 172.21.15.161:0/2560249822 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(150..150 src has 1..150) ==== 6001+0+0 (secure 0 0 0) 0x7f31c808ae10 con 0x7f31d80c9480 2024-03-28T23:35:58.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.193+0000 7f31d67fc640 1 --2- 172.21.15.161:0/2560249822 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f31c406b4d0 0x7f31c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:58.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.193+0000 7f31d67fc640 1 --2- 172.21.15.161:0/2560249822 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f31c406b4d0 0x7f31c406d990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f31cc035000 tx=0x7f31cc030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:58.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.197+0000 7f31bffff640 1 -- 172.21.15.161:0/2560249822 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f31d8151170 con 0x7f31d80c9480 2024-03-28T23:35:58.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:58.404+0000 7f31dd8da640 1 -- 172.21.15.161:0/2560249822 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f31d804f4b0 con 0x7f31d80c9480 2024-03-28T23:35:59.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.052+0000 7f31bffff640 1 -- 172.21.15.161:0/2560249822 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 23 and data pool 24 v138) ==== 174+0+0 (secure 0 0 0) 0x7f31c8022070 con 0x7f31d80c9480 2024-03-28T23:35:59.053 INFO:teuthology.orchestra.run.smithi161.stderr:new fs with metadata pool 23 and data pool 24 2024-03-28T23:35:59.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.054+0000 7f31dd8da640 1 -- 172.21.15.161:0/2560249822 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f31c406b4d0 msgr2=0x7f31c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:59.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.054+0000 7f31dd8da640 1 --2- 172.21.15.161:0/2560249822 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f31c406b4d0 0x7f31c406d990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f31cc035000 tx=0x7f31cc030000 comp rx=0 tx=0).stop 2024-03-28T23:35:59.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.054+0000 7f31dd8da640 1 -- 172.21.15.161:0/2560249822 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31d80c9480 msgr2=0x7f31d814dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:59.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.054+0000 7f31dd8da640 1 --2- 172.21.15.161:0/2560249822 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31d80c9480 0x7f31d814dab0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f31c8009d10 tx=0x7f31c800c490 comp rx=0 tx=0).stop 2024-03-28T23:35:59.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.055+0000 7f31dd8da640 1 -- 172.21.15.161:0/2560249822 shutdown_connections 2024-03-28T23:35:59.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.055+0000 7f31dd8da640 1 --2- 172.21.15.161:0/2560249822 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f31c406b4d0 0x7f31c406d990 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.055+0000 7f31dd8da640 1 --2- 172.21.15.161:0/2560249822 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31d8152940 0x7f31d81ee630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.055+0000 7f31dd8da640 1 --2- 172.21.15.161:0/2560249822 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31d80ca580 0x7f31d814dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.055+0000 7f31dd8da640 1 --2- 172.21.15.161:0/2560249822 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31d80c9480 0x7f31d814dab0 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.055+0000 7f31dd8da640 1 -- 172.21.15.161:0/2560249822 >> 172.21.15.161:0/2560249822 conn(0x7f31d80caa00 msgr2=0x7f31d80c8900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:59.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.055+0000 7f31dd8da640 1 -- 172.21.15.161:0/2560249822 shutdown_connections 2024-03-28T23:35:59.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.055+0000 7f31dd8da640 1 -- 172.21.15.161:0/2560249822 wait complete. 2024-03-28T23:35:59.088 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:35:59.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.191+0000 7f323166b640 1 Processor -- start 2024-03-28T23:35:59.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.191+0000 7f323166b640 1 -- start start 2024-03-28T23:35:59.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.191+0000 7f323166b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f322c152a80 0x7f322c155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:59.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.191+0000 7f323166b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f322c06d9f0 0x7f322c0c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:59.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.191+0000 7f323166b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f322c06cb80 0x7f322c0c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:59.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.191+0000 7f323166b640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f322c156250 con 0x7f322c06cb80 2024-03-28T23:35:59.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.191+0000 7f323166b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f322c06c6f0 con 0x7f322c152a80 2024-03-28T23:35:59.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.191+0000 7f323166b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f322c0c98e0 con 0x7f322c06d9f0 2024-03-28T23:35:59.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.191+0000 7f322b7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f322c06cb80 0x7f322c0c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:59.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.191+0000 7f322b7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f322c06cb80 0x7f322c0c74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57656/0 (socket says 172.21.15.161:57656) 2024-03-28T23:35:59.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.191+0000 7f322b7fe640 1 -- 172.21.15.161:0/1278359816 learned_addr learned my addr 172.21.15.161:0/1278359816 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:59.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.192+0000 7f322a7fc640 1 --2- 172.21.15.161:0/1278359816 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f322c06d9f0 0x7f322c0c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:59.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.192+0000 7f322affd640 1 --2- 172.21.15.161:0/1278359816 >> v2:172.21.15.170:3300/0 conn(0x7f322c152a80 0x7f322c155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:59.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.192+0000 7f322b7fe640 1 -- 172.21.15.161:0/1278359816 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f322c06d9f0 msgr2=0x7f322c0c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:59.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.192+0000 7f322b7fe640 1 --2- 172.21.15.161:0/1278359816 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f322c06d9f0 0x7f322c0c93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.192+0000 7f322b7fe640 1 -- 172.21.15.161:0/1278359816 >> v2:172.21.15.170:3300/0 conn(0x7f322c152a80 msgr2=0x7f322c155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:59.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.192+0000 7f322b7fe640 1 --2- 172.21.15.161:0/1278359816 >> v2:172.21.15.170:3300/0 conn(0x7f322c152a80 0x7f322c155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.192+0000 7f322b7fe640 1 -- 172.21.15.161:0/1278359816 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f322c06c3c0 con 0x7f322c06cb80 2024-03-28T23:35:59.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.192+0000 7f322a7fc640 1 --2- 172.21.15.161:0/1278359816 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f322c06d9f0 0x7f322c0c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:35:59.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.192+0000 7f322b7fe640 1 --2- 172.21.15.161:0/1278359816 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f322c06cb80 0x7f322c0c74c0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f322c0c7ff0 tx=0x7f322402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9645c7eb25b2931 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:59.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.193+0000 7f3229ffb640 1 -- 172.21.15.161:0/1278359816 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322402c950 con 0x7f322c06cb80 2024-03-28T23:35:59.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.193+0000 7f3229ffb640 1 -- 172.21.15.161:0/1278359816 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f322402caf0 con 0x7f322c06cb80 2024-03-28T23:35:59.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.193+0000 7f3229ffb640 1 -- 172.21.15.161:0/1278359816 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32240343f0 con 0x7f322c06cb80 2024-03-28T23:35:59.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.193+0000 7f323166b640 1 -- 172.21.15.161:0/1278359816 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f322c06cb80 msgr2=0x7f322c0c74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:59.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.193+0000 7f323166b640 1 --2- 172.21.15.161:0/1278359816 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f322c06cb80 0x7f322c0c74c0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f322c0c7ff0 tx=0x7f322402bdb0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.193+0000 7f323166b640 1 -- 172.21.15.161:0/1278359816 shutdown_connections 2024-03-28T23:35:59.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.193+0000 7f323166b640 1 --2- 172.21.15.161:0/1278359816 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f322c06cb80 0x7f322c0c74c0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.193+0000 7f323166b640 1 --2- 172.21.15.161:0/1278359816 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f322c06d9f0 0x7f322c0c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.193+0000 7f323166b640 1 --2- 172.21.15.161:0/1278359816 >> v2:172.21.15.170:3300/0 conn(0x7f322c152a80 0x7f322c155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.193+0000 7f323166b640 1 -- 172.21.15.161:0/1278359816 >> 172.21.15.161:0/1278359816 conn(0x7f322c0ac720 msgr2=0x7f322c0acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:59.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.193+0000 7f323166b640 1 -- 172.21.15.161:0/1278359816 shutdown_connections 2024-03-28T23:35:59.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.193+0000 7f323166b640 1 -- 172.21.15.161:0/1278359816 wait complete. 2024-03-28T23:35:59.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.194+0000 7f323166b640 1 Processor -- start 2024-03-28T23:35:59.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.194+0000 7f323166b640 1 -- start start 2024-03-28T23:35:59.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.195+0000 7f323166b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f322c06cb80 0x7f322c149150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:59.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.195+0000 7f323166b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f322c06d9f0 0x7f322c0c3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:59.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.195+0000 7f323166b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f322c152a80 0x7f322c0c3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:59.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.195+0000 7f323166b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f322c0c58b0 con 0x7f322c06cb80 2024-03-28T23:35:59.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.195+0000 7f323166b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f322c0c55d0 con 0x7f322c06d9f0 2024-03-28T23:35:59.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.195+0000 7f323166b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f322c06d010 con 0x7f322c152a80 2024-03-28T23:35:59.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.195+0000 7f322affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f322c06cb80 0x7f322c149150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:59.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.195+0000 7f322affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f322c06cb80 0x7f322c149150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57672/0 (socket says 172.21.15.161:57672) 2024-03-28T23:35:59.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.195+0000 7f322affd640 1 -- 172.21.15.161:0/2708785599 learned_addr learned my addr 172.21.15.161:0/2708785599 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:59.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.195+0000 7f322affd640 1 -- 172.21.15.161:0/2708785599 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f322c152a80 msgr2=0x7f322c0c3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:59.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.195+0000 7f322affd640 1 --2- 172.21.15.161:0/2708785599 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f322c152a80 0x7f322c0c3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.195+0000 7f322affd640 1 -- 172.21.15.161:0/2708785599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f322c06d9f0 msgr2=0x7f322c0c3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:59.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.195+0000 7f322affd640 1 --2- 172.21.15.161:0/2708785599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f322c06d9f0 0x7f322c0c3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.195+0000 7f322affd640 1 -- 172.21.15.161:0/2708785599 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f322c14f150 con 0x7f322c06cb80 2024-03-28T23:35:59.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.196+0000 7f322affd640 1 --2- 172.21.15.161:0/2708785599 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f322c06cb80 0x7f322c149150 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f322000d5f0 tx=0x7f322000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:59.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.196+0000 7f32137fe640 1 -- 172.21.15.161:0/2708785599 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32200125f0 con 0x7f322c06cb80 2024-03-28T23:35:59.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.196+0000 7f32137fe640 1 -- 172.21.15.161:0/2708785599 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3220019070 con 0x7f322c06cb80 2024-03-28T23:35:59.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.196+0000 7f323166b640 1 -- 172.21.15.161:0/2708785599 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f322c14ee70 con 0x7f322c06cb80 2024-03-28T23:35:59.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.196+0000 7f32137fe640 1 -- 172.21.15.161:0/2708785599 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32200129b0 con 0x7f322c06cb80 2024-03-28T23:35:59.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.196+0000 7f323166b640 1 -- 172.21.15.161:0/2708785599 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f322c14e680 con 0x7f322c06cb80 2024-03-28T23:35:59.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.197+0000 7f323166b640 1 -- 172.21.15.161:0/2708785599 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f322c04f4b0 con 0x7f322c06cb80 2024-03-28T23:35:59.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.197+0000 7f32137fe640 1 -- 172.21.15.161:0/2708785599 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3220012b50 con 0x7f322c06cb80 2024-03-28T23:35:59.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.198+0000 7f32137fe640 1 --2- 172.21.15.161:0/2708785599 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f321806b4d0 0x7f321806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:59.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.198+0000 7f32137fe640 1 -- 172.21.15.161:0/2708785599 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 6101+0+0 (secure 0 0 0) 0x7f322008a740 con 0x7f322c06cb80 2024-03-28T23:35:59.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.198+0000 7f322a7fc640 1 --2- 172.21.15.161:0/2708785599 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f321806b4d0 0x7f321806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:59.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.198+0000 7f322a7fc640 1 --2- 172.21.15.161:0/2708785599 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f321806b4d0 0x7f321806d990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f322c14e2d0 tx=0x7f321c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:59.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.202+0000 7f32137fe640 1 -- 172.21.15.161:0/2708785599 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f32200594c0 con 0x7f322c06cb80 2024-03-28T23:35:59.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.376+0000 7f323166b640 1 -- 172.21.15.161:0/2708785599 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f322c0c4a90 con 0x7f322c06cb80 2024-03-28T23:35:59.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.378+0000 7f32137fe640 1 -- 172.21.15.161:0/2708785599 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) ==== 74+0+16933 (secure 0 0 0) 0x7f3220058dc0 con 0x7f322c06cb80 2024-03-28T23:35:59.378 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:59.378 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":151,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:35:58.406106+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:35:56.687416+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-28T23:35:57.379120+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:35:59.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.380+0000 7f323166b640 1 -- 172.21.15.161:0/2708785599 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f321806b4d0 msgr2=0x7f321806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:59.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.380+0000 7f323166b640 1 --2- 172.21.15.161:0/2708785599 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f321806b4d0 0x7f321806d990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f322c14e2d0 tx=0x7f321c008040 comp rx=0 tx=0).stop 2024-03-28T23:35:59.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.380+0000 7f323166b640 1 -- 172.21.15.161:0/2708785599 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f322c06cb80 msgr2=0x7f322c149150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:59.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.380+0000 7f323166b640 1 --2- 172.21.15.161:0/2708785599 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f322c06cb80 0x7f322c149150 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f322000d5f0 tx=0x7f322000dab0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.380+0000 7f323166b640 1 -- 172.21.15.161:0/2708785599 shutdown_connections 2024-03-28T23:35:59.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.380+0000 7f323166b640 1 --2- 172.21.15.161:0/2708785599 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f321806b4d0 0x7f321806d990 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.380+0000 7f323166b640 1 --2- 172.21.15.161:0/2708785599 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f322c152a80 0x7f322c0c3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.380+0000 7f323166b640 1 --2- 172.21.15.161:0/2708785599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f322c06d9f0 0x7f322c0c3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.380+0000 7f323166b640 1 --2- 172.21.15.161:0/2708785599 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f322c06cb80 0x7f322c149150 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.380+0000 7f323166b640 1 -- 172.21.15.161:0/2708785599 >> 172.21.15.161:0/2708785599 conn(0x7f322c0ac720 msgr2=0x7f322c0acb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:59.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.380+0000 7f323166b640 1 -- 172.21.15.161:0/2708785599 shutdown_connections 2024-03-28T23:35:59.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.380+0000 7f323166b640 1 -- 172.21.15.161:0/2708785599 wait complete. 2024-03-28T23:35:59.391 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:35:59.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.492+0000 7f57920fb640 1 Processor -- start 2024-03-28T23:35:59.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.492+0000 7f57920fb640 1 -- start start 2024-03-28T23:35:59.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.493+0000 7f57920fb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f578c150590 0x7f578c0a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:59.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.493+0000 7f57920fb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f578c0aacb0 0x7f578c153860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:59.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.493+0000 7f57920fb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f578c1566f0 con 0x7f578c0a8fc0 2024-03-28T23:35:59.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.493+0000 7f57920fb640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f578c1569d0 con 0x7f578c150590 2024-03-28T23:35:59.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.493+0000 7f57920fb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f578c156cb0 con 0x7f578c0aacb0 2024-03-28T23:35:59.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.493+0000 7f578b7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f578c150590 0x7f578c0a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:59.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.493+0000 7f578affd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f578c0a8fc0 0x7f578c153150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:54384/0 (socket says 172.21.15.161:54384) 2024-03-28T23:35:59.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.493+0000 7f578affd640 1 -- 172.21.15.161:0/3008001760 learned_addr learned my addr 172.21.15.161:0/3008001760 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:59.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.494+0000 7f578bfff640 1 --2- 172.21.15.161:0/3008001760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f578c0aacb0 0x7f578c153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:59.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.494+0000 7f578b7fe640 1 -- 172.21.15.161:0/3008001760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f578c0aacb0 msgr2=0x7f578c153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:59.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.494+0000 7f578b7fe640 1 --2- 172.21.15.161:0/3008001760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f578c0aacb0 0x7f578c153860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.494+0000 7f578b7fe640 1 -- 172.21.15.161:0/3008001760 >> v1:172.21.15.161:6789/0 conn(0x7f578c0a8fc0 legacy=0x7f578c153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:59.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.494+0000 7f578b7fe640 1 -- 172.21.15.161:0/3008001760 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f578c156f90 con 0x7f578c150590 2024-03-28T23:35:59.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.494+0000 7f578a7fc640 1 -- 172.21.15.161:0/3008001760 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3771827790 0 0) 0x7f578c1566f0 con 0x7f578c0a8fc0 2024-03-28T23:35:59.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.494+0000 7f578b7fe640 1 --2- 172.21.15.161:0/3008001760 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f578c150590 0x7f578c0a9f40 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f577c0097a0 tx=0x7f577c02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db5d9139ca601a45 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:59.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.494+0000 7f578a7fc640 1 -- 172.21.15.161:0/3008001760 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c038070 con 0x7f578c150590 2024-03-28T23:35:59.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.495+0000 7f578a7fc640 1 -- 172.21.15.161:0/3008001760 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f577c031700 con 0x7f578c150590 2024-03-28T23:35:59.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.495+0000 7f578a7fc640 1 -- 172.21.15.161:0/3008001760 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c031ad0 con 0x7f578c150590 2024-03-28T23:35:59.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.495+0000 7f57920fb640 1 -- 172.21.15.161:0/3008001760 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f578c150590 msgr2=0x7f578c0a9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:59.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.495+0000 7f57920fb640 1 --2- 172.21.15.161:0/3008001760 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f578c150590 0x7f578c0a9f40 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f577c0097a0 tx=0x7f577c02b7a0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.495+0000 7f57920fb640 1 -- 172.21.15.161:0/3008001760 shutdown_connections 2024-03-28T23:35:59.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.495+0000 7f57920fb640 1 --2- 172.21.15.161:0/3008001760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f578c0aacb0 0x7f578c153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.495+0000 7f57920fb640 1 --2- 172.21.15.161:0/3008001760 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f578c150590 0x7f578c0a9f40 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.495+0000 7f57920fb640 1 -- 172.21.15.161:0/3008001760 >> 172.21.15.161:0/3008001760 conn(0x7f578c0c6670 msgr2=0x7f578c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:59.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.495+0000 7f57920fb640 1 -- 172.21.15.161:0/3008001760 shutdown_connections 2024-03-28T23:35:59.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.495+0000 7f57920fb640 1 -- 172.21.15.161:0/3008001760 wait complete. 2024-03-28T23:35:59.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.496+0000 7f57920fb640 1 Processor -- start 2024-03-28T23:35:59.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.496+0000 7f57920fb640 1 -- start start 2024-03-28T23:35:59.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.497+0000 7f57920fb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f578c0a8fc0 0x7f578c0bfc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:59.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.497+0000 7f57920fb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f578c0aacb0 0x7f578c1e7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:59.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.497+0000 7f57920fb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f578c150590 0x7f578c1e7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:59.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.497+0000 7f57920fb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f578c150970 con 0x7f578c0a8fc0 2024-03-28T23:35:59.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.497+0000 7f57920fb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f578c0aa9c0 con 0x7f578c150590 2024-03-28T23:35:59.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.497+0000 7f57920fb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f578c0aa480 con 0x7f578c0aacb0 2024-03-28T23:35:59.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.497+0000 7f578b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f578c0a8fc0 0x7f578c0bfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:59.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.497+0000 7f578b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f578c0a8fc0 0x7f578c0bfc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57692/0 (socket says 172.21.15.161:57692) 2024-03-28T23:35:59.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.497+0000 7f578b7fe640 1 -- 172.21.15.161:0/2063223152 learned_addr learned my addr 172.21.15.161:0/2063223152 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:59.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.497+0000 7f578affd640 1 --2- 172.21.15.161:0/2063223152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f578c0aacb0 0x7f578c1e7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:59.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.497+0000 7f578bfff640 1 --2- 172.21.15.161:0/2063223152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f578c150590 0x7f578c1e7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:59.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.497+0000 7f578b7fe640 1 -- 172.21.15.161:0/2063223152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f578c0aacb0 msgr2=0x7f578c1e7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:59.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.497+0000 7f578b7fe640 1 --2- 172.21.15.161:0/2063223152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f578c0aacb0 0x7f578c1e7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.497+0000 7f578b7fe640 1 -- 172.21.15.161:0/2063223152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f578c150590 msgr2=0x7f578c1e7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:59.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.497+0000 7f578b7fe640 1 --2- 172.21.15.161:0/2063223152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f578c150590 0x7f578c1e7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.497+0000 7f578b7fe640 1 -- 172.21.15.161:0/2063223152 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f578c0c1550 con 0x7f578c0a8fc0 2024-03-28T23:35:59.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.498+0000 7f578b7fe640 1 --2- 172.21.15.161:0/2063223152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f578c0a8fc0 0x7f578c0bfc70 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f577c000c00 tx=0x7f577c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:59.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.498+0000 7f5773fff640 1 -- 172.21.15.161:0/2063223152 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c042070 con 0x7f578c0a8fc0 2024-03-28T23:35:59.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.498+0000 7f5773fff640 1 -- 172.21.15.161:0/2063223152 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f577c033a30 con 0x7f578c0a8fc0 2024-03-28T23:35:59.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.498+0000 7f57920fb640 1 -- 172.21.15.161:0/2063223152 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f578c06c920 con 0x7f578c0a8fc0 2024-03-28T23:35:59.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.498+0000 7f5773fff640 1 -- 172.21.15.161:0/2063223152 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c038040 con 0x7f578c0a8fc0 2024-03-28T23:35:59.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.498+0000 7f57920fb640 1 -- 172.21.15.161:0/2063223152 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f578c06cbc0 con 0x7f578c0a8fc0 2024-03-28T23:35:59.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.499+0000 7f57920fb640 1 -- 172.21.15.161:0/2063223152 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f578c04f4b0 con 0x7f578c0a8fc0 2024-03-28T23:35:59.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.499+0000 7f5773fff640 1 -- 172.21.15.161:0/2063223152 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f577c045070 con 0x7f578c0a8fc0 2024-03-28T23:35:59.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.500+0000 7f5773fff640 1 --2- 172.21.15.161:0/2063223152 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f577806b500 0x7f577806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:59.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.500+0000 7f5773fff640 1 -- 172.21.15.161:0/2063223152 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 6101+0+0 (secure 0 0 0) 0x7f577c0abf30 con 0x7f578c0a8fc0 2024-03-28T23:35:59.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.500+0000 7f578affd640 1 --2- 172.21.15.161:0/2063223152 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f577806b500 0x7f577806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:59.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.500+0000 7f578affd640 1 --2- 172.21.15.161:0/2063223152 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f577806b500 0x7f577806d9c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f578c0c1310 tx=0x7f5780009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:59.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.504+0000 7f5773fff640 1 -- 172.21.15.161:0/2063223152 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f578c0c1550 con 0x7f578c0a8fc0 2024-03-28T23:35:59.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.677+0000 7f57920fb640 1 -- 172.21.15.161:0/2063223152 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f578c0a9d20 con 0x7f578c0a8fc0 2024-03-28T23:35:59.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.679+0000 7f5773fff640 1 -- 172.21.15.161:0/2063223152 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) ==== 74+0+16933 (secure 0 0 0) 0x7f577c030070 con 0x7f578c0a8fc0 2024-03-28T23:35:59.679 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:35:59.679 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":151,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:35:58.406106+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:35:56.687416+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-28T23:35:57.379120+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:35:59.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.681+0000 7f57920fb640 1 -- 172.21.15.161:0/2063223152 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f577806b500 msgr2=0x7f577806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:59.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.681+0000 7f57920fb640 1 --2- 172.21.15.161:0/2063223152 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f577806b500 0x7f577806d9c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f578c0c1310 tx=0x7f5780009040 comp rx=0 tx=0).stop 2024-03-28T23:35:59.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.681+0000 7f57920fb640 1 -- 172.21.15.161:0/2063223152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f578c0a8fc0 msgr2=0x7f578c0bfc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:59.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.681+0000 7f57920fb640 1 --2- 172.21.15.161:0/2063223152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f578c0a8fc0 0x7f578c0bfc70 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f577c000c00 tx=0x7f577c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.681+0000 7f57920fb640 1 -- 172.21.15.161:0/2063223152 shutdown_connections 2024-03-28T23:35:59.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.681+0000 7f57920fb640 1 --2- 172.21.15.161:0/2063223152 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f577806b500 0x7f577806d9c0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.681+0000 7f57920fb640 1 --2- 172.21.15.161:0/2063223152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f578c150590 0x7f578c1e7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.681+0000 7f57920fb640 1 --2- 172.21.15.161:0/2063223152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f578c0aacb0 0x7f578c1e7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.681+0000 7f57920fb640 1 --2- 172.21.15.161:0/2063223152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f578c0a8fc0 0x7f578c0bfc70 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.681+0000 7f57920fb640 1 -- 172.21.15.161:0/2063223152 >> 172.21.15.161:0/2063223152 conn(0x7f578c0c6670 msgr2=0x7f578c070470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:59.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.681+0000 7f57920fb640 1 -- 172.21.15.161:0/2063223152 shutdown_connections 2024-03-28T23:35:59.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.681+0000 7f57920fb640 1 -- 172.21.15.161:0/2063223152 wait complete. 2024-03-28T23:35:59.693 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-28T23:35:59.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.797+0000 7f2ec38da640 1 Processor -- start 2024-03-28T23:35:59.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.797+0000 7f2ec38da640 1 -- start start 2024-03-28T23:35:59.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ec38da640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ebc152bd0 0x7f2ebc15d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:59.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ec38da640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ebc0c2f10 con 0x7f2ebc14f5c0 2024-03-28T23:35:59.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ec38da640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ebc14f9a0 con 0x7f2ebc06cdd0 2024-03-28T23:35:59.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ec38da640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2ebc0c74d0 con 0x7f2ebc152bd0 2024-03-28T23:35:59.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ec1676640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f2ebc14f5c0 0x7f2ebc0c2150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:54390/0 (socket says 172.21.15.161:54390) 2024-03-28T23:35:59.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ec1676640 1 -- 172.21.15.161:0/3106547674 learned_addr learned my addr 172.21.15.161:0/3106547674 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:35:59.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ec1e77640 1 --2- 172.21.15.161:0/3106547674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ebc152bd0 0x7f2ebc15d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:59.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ebbfff640 1 -- 172.21.15.161:0/3106547674 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2487362730 0 0) 0x7f2eac001490 con 0x7f2ebc14f5c0 2024-03-28T23:35:59.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ebbfff640 1 -- 172.21.15.161:0/3106547674 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ea4003520 con 0x7f2ebc14f5c0 2024-03-28T23:35:59.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ebbfff640 1 -- 172.21.15.161:0/3106547674 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 684210753 0 0) 0x7f2ea4003520 con 0x7f2ebc14f5c0 2024-03-28T23:35:59.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ebbfff640 1 -- 172.21.15.161:0/3106547674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ebc152bd0 msgr2=0x7f2ebc15d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:59.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ebbfff640 1 --2- 172.21.15.161:0/3106547674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ebc152bd0 0x7f2ebc15d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ebbfff640 1 -- 172.21.15.161:0/3106547674 >> v1:172.21.15.170:6789/0 conn(0x7f2ebc06cdd0 legacy=0x7f2ebc1524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:59.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ebbfff640 1 -- 172.21.15.161:0/3106547674 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ebc06c140 con 0x7f2ebc14f5c0 2024-03-28T23:35:59.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ebbfff640 1 -- 172.21.15.161:0/3106547674 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f2eac001bc0 con 0x7f2ebc14f5c0 2024-03-28T23:35:59.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ebbfff640 1 -- 172.21.15.161:0/3106547674 >> v1:172.21.15.161:6789/0 conn(0x7f2ebc14f5c0 legacy=0x7f2ebc0c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:59.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ebbfff640 1 --2- 172.21.15.161:0/3106547674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ea4003ac0 0x7f2ea4003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:59.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ebbfff640 1 --2- 172.21.15.161:0/3106547674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ebc152bd0 0x7f2ea4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:59.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ebbfff640 1 --2- 172.21.15.161:0/3106547674 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ea4004d40 0x7f2ea40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:59.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ebbfff640 1 -- 172.21.15.161:0/3106547674 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2ebc0c74d0 con 0x7f2ea4004d40 2024-03-28T23:35:59.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ebbfff640 1 -- 172.21.15.161:0/3106547674 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea400bb50 con 0x7f2ea4003ac0 2024-03-28T23:35:59.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ebbfff640 1 -- 172.21.15.161:0/3106547674 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2ea400bd50 con 0x7f2ebc152bd0 2024-03-28T23:35:59.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ec1e77640 1 --2- 172.21.15.161:0/3106547674 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ea4004d40 0x7f2ea40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:59.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ec1676640 1 --2- 172.21.15.161:0/3106547674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ea4003ac0 0x7f2ea4003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:59.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.798+0000 7f2ec0e75640 1 --2- 172.21.15.161:0/3106547674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ebc152bd0 0x7f2ea4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:59.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.799+0000 7f2ec1e77640 1 -- 172.21.15.161:0/3106547674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ebc152bd0 msgr2=0x7f2ea4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:59.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.799+0000 7f2ec1e77640 1 --2- 172.21.15.161:0/3106547674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ebc152bd0 0x7f2ea4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.799+0000 7f2ec1e77640 1 -- 172.21.15.161:0/3106547674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ea4003ac0 msgr2=0x7f2ea4003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:35:59.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.799+0000 7f2ec1e77640 1 --2- 172.21.15.161:0/3106547674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ea4003ac0 0x7f2ea4003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.799+0000 7f2ec1e77640 1 -- 172.21.15.161:0/3106547674 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea4003520 con 0x7f2ea4004d40 2024-03-28T23:35:59.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.799+0000 7f2ec1e77640 1 --2- 172.21.15.161:0/3106547674 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ea4004d40 0x7f2ea40051c0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4009ac0 tx=0x7f2ea4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c1fe2dbd25082df server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:59.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.799+0000 7f2ebbfff640 1 -- 172.21.15.161:0/3106547674 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4039070 con 0x7f2ea4004d40 2024-03-28T23:35:59.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.800+0000 7f2ebbfff640 1 -- 172.21.15.161:0/3106547674 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2eb402c6b0 con 0x7f2ea4004d40 2024-03-28T23:35:59.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.800+0000 7f2ebbfff640 1 -- 172.21.15.161:0/3106547674 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4009690 con 0x7f2ea4004d40 2024-03-28T23:35:59.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.800+0000 7f2ec38da640 1 -- 172.21.15.161:0/3106547674 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ea4004d40 msgr2=0x7f2ea40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:59.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.800+0000 7f2ec38da640 1 --2- 172.21.15.161:0/3106547674 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ea4004d40 0x7f2ea40051c0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4009ac0 tx=0x7f2ea4005830 comp rx=0 tx=0).stop 2024-03-28T23:35:59.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.800+0000 7f2ec1e77640 1 -- 172.21.15.161:0/3106547674 reap_dead start 2024-03-28T23:35:59.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.800+0000 7f2ec38da640 1 -- 172.21.15.161:0/3106547674 shutdown_connections 2024-03-28T23:35:59.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.800+0000 7f2ec38da640 1 -- 172.21.15.161:0/3106547674 >> 172.21.15.161:0/3106547674 conn(0x7f2ebc0a83f0 msgr2=0x7f2ebc0a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:35:59.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.800+0000 7f2ec38da640 1 -- 172.21.15.161:0/3106547674 shutdown_connections 2024-03-28T23:35:59.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.800+0000 7f2ec38da640 1 -- 172.21.15.161:0/3106547674 wait complete. 2024-03-28T23:35:59.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.801+0000 7f2ec38da640 1 Processor -- start 2024-03-28T23:35:59.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.801+0000 7f2ec38da640 1 -- start start 2024-03-28T23:35:59.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.802+0000 7f2ec38da640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ebc1ebb90 0x7f2ebc1ebf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:59.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.802+0000 7f2ec38da640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ebc1eb3e0 0x7f2ebc0711b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:59.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.802+0000 7f2ec38da640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ebc0716f0 0x7f2ebc14a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:59.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.802+0000 7f2ec38da640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2ebc152030 con 0x7f2ebc0716f0 2024-03-28T23:35:59.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.802+0000 7f2ec38da640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2ebc0c6fa0 con 0x7f2ebc1eb3e0 2024-03-28T23:35:59.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.802+0000 7f2ec38da640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2ebc06c3e0 con 0x7f2ebc1ebb90 2024-03-28T23:35:59.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.802+0000 7f2ec1e77640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ebc0716f0 0x7f2ebc14a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:59.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.802+0000 7f2ec1e77640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ebc0716f0 0x7f2ebc14a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57706/0 (socket says 172.21.15.161:57706) 2024-03-28T23:35:59.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.802+0000 7f2ec1e77640 1 -- 172.21.15.161:0/938041074 learned_addr learned my addr 172.21.15.161:0/938041074 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:35:59.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.802+0000 7f2ec1676640 1 --2- 172.21.15.161:0/938041074 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ebc1ebb90 0x7f2ebc1ebf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:59.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.802+0000 7f2ec0e75640 1 --2- 172.21.15.161:0/938041074 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ebc1eb3e0 0x7f2ebc0711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:59.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.802+0000 7f2ec1e77640 1 -- 172.21.15.161:0/938041074 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ebc1ebb90 msgr2=0x7f2ebc1ebf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:59.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.802+0000 7f2ec1e77640 1 --2- 172.21.15.161:0/938041074 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ebc1ebb90 0x7f2ebc1ebf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.802+0000 7f2ec1e77640 1 -- 172.21.15.161:0/938041074 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ebc1eb3e0 msgr2=0x7f2ebc0711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:35:59.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.802+0000 7f2ec1e77640 1 --2- 172.21.15.161:0/938041074 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ebc1eb3e0 0x7f2ebc0711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:35:59.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.802+0000 7f2ec1e77640 1 -- 172.21.15.161:0/938041074 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ebc071e30 con 0x7f2ebc0716f0 2024-03-28T23:35:59.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.802+0000 7f2ec1e77640 1 --2- 172.21.15.161:0/938041074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ebc0716f0 0x7f2ebc14a920 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4009be0 tx=0x7f2eb4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:59.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.803+0000 7f2eb9ffb640 1 -- 172.21.15.161:0/938041074 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4039070 con 0x7f2ebc0716f0 2024-03-28T23:35:59.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.803+0000 7f2eb9ffb640 1 -- 172.21.15.161:0/938041074 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2eb4043070 con 0x7f2ebc0716f0 2024-03-28T23:35:59.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.803+0000 7f2ec38da640 1 -- 172.21.15.161:0/938041074 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ebc072110 con 0x7f2ebc0716f0 2024-03-28T23:35:59.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.803+0000 7f2eb9ffb640 1 -- 172.21.15.161:0/938041074 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4009690 con 0x7f2ebc0716f0 2024-03-28T23:35:59.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.803+0000 7f2ec38da640 1 -- 172.21.15.161:0/938041074 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ebc1eea80 con 0x7f2ebc0716f0 2024-03-28T23:35:59.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.804+0000 7f2ec38da640 1 -- 172.21.15.161:0/938041074 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ebc04f4b0 con 0x7f2ebc0716f0 2024-03-28T23:35:59.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.804+0000 7f2eb9ffb640 1 -- 172.21.15.161:0/938041074 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2eb4003850 con 0x7f2ebc0716f0 2024-03-28T23:35:59.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.805+0000 7f2eb9ffb640 1 --2- 172.21.15.161:0/938041074 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2ea806b4d0 0x7f2ea806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:35:59.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.805+0000 7f2ec1676640 1 --2- 172.21.15.161:0/938041074 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2ea806b4d0 0x7f2ea806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:35:59.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.805+0000 7f2eb9ffb640 1 -- 172.21.15.161:0/938041074 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 6101+0+0 (secure 0 0 0) 0x7f2eb40ab110 con 0x7f2ebc0716f0 2024-03-28T23:35:59.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.805+0000 7f2ec1676640 1 --2- 172.21.15.161:0/938041074 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2ea806b4d0 0x7f2ea806d990 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f2eac000f80 tx=0x7f2eac006040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:35:59.809 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:35:59.809+0000 7f2eb9ffb640 1 -- 172.21.15.161:0/938041074 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2eb404c020 con 0x7f2ebc0716f0 2024-03-28T23:36:00.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.017+0000 7f2ec38da640 1 -- 172.21.15.161:0/938041074 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f2ebc1eec50 con 0x7f2ebc0716f0 2024-03-28T23:36:00.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.066+0000 7f2eb9ffb640 1 -- 172.21.15.161:0/938041074 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v139) ==== 118+0+0 (secure 0 0 0) 0x7f2eb4079900 con 0x7f2ebc0716f0 2024-03-28T23:36:00.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.068+0000 7f2ec38da640 1 -- 172.21.15.161:0/938041074 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2ea806b4d0 msgr2=0x7f2ea806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:00.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.068+0000 7f2ec38da640 1 --2- 172.21.15.161:0/938041074 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2ea806b4d0 0x7f2ea806d990 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f2eac000f80 tx=0x7f2eac006040 comp rx=0 tx=0).stop 2024-03-28T23:36:00.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.068+0000 7f2ec38da640 1 -- 172.21.15.161:0/938041074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ebc0716f0 msgr2=0x7f2ebc14a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:00.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.068+0000 7f2ec38da640 1 --2- 172.21.15.161:0/938041074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ebc0716f0 0x7f2ebc14a920 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4009be0 tx=0x7f2eb4004730 comp rx=0 tx=0).stop 2024-03-28T23:36:00.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.068+0000 7f2ec38da640 1 -- 172.21.15.161:0/938041074 shutdown_connections 2024-03-28T23:36:00.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.068+0000 7f2ec38da640 1 --2- 172.21.15.161:0/938041074 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2ea806b4d0 0x7f2ea806d990 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:00.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.068+0000 7f2ec38da640 1 --2- 172.21.15.161:0/938041074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ebc0716f0 0x7f2ebc14a920 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:00.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.068+0000 7f2ec38da640 1 --2- 172.21.15.161:0/938041074 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ebc1eb3e0 0x7f2ebc0711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:00.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.069+0000 7f2ec38da640 1 --2- 172.21.15.161:0/938041074 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ebc1ebb90 0x7f2ebc1ebf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:00.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.069+0000 7f2ec38da640 1 -- 172.21.15.161:0/938041074 >> 172.21.15.161:0/938041074 conn(0x7f2ebc0a83f0 msgr2=0x7f2ebc0728f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:00.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.069+0000 7f2ec38da640 1 -- 172.21.15.161:0/938041074 shutdown_connections 2024-03-28T23:36:00.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.069+0000 7f2ec38da640 1 -- 172.21.15.161:0/938041074 wait complete. 2024-03-28T23:36:00.080 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-28T23:36:00.080 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-28T23:36:00.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.184+0000 7f74526ed640 1 Processor -- start 2024-03-28T23:36:00.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.184+0000 7f74526ed640 1 -- start start 2024-03-28T23:36:00.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.185+0000 7f74526ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f744c14f5c0 0x7f744c0c2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:00.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.185+0000 7f74526ed640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f744c0c2f10 con 0x7f744c152bd0 2024-03-28T23:36:00.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.185+0000 7f74526ed640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f744c14f9a0 con 0x7f744c06cdd0 2024-03-28T23:36:00.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.185+0000 7f74526ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f744c0c74d0 con 0x7f744c14f5c0 2024-03-28T23:36:00.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.185+0000 7f7450c8a640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f744c152bd0 0x7f744c15c830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:54406/0 (socket says 172.21.15.161:54406) 2024-03-28T23:36:00.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.185+0000 7f7450c8a640 1 -- 172.21.15.161:0/1785729266 learned_addr learned my addr 172.21.15.161:0/1785729266 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.185+0000 7f744bfff640 1 --2- 172.21.15.161:0/1785729266 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f744c14f5c0 0x7f744c0c2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.185+0000 7f744affd640 1 -- 172.21.15.161:0/1785729266 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1526578466 0 0) 0x7f7444001490 con 0x7f744c152bd0 2024-03-28T23:36:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.185+0000 7f744affd640 1 -- 172.21.15.161:0/1785729266 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7434003520 con 0x7f744c152bd0 2024-03-28T23:36:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.185+0000 7f744affd640 1 -- 172.21.15.161:0/1785729266 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 16226123 0 0) 0x7f744c14f9a0 con 0x7f744c06cdd0 2024-03-28T23:36:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.185+0000 7f744affd640 1 -- 172.21.15.161:0/1785729266 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7444001490 con 0x7f744c06cdd0 2024-03-28T23:36:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.185+0000 7f744bfff640 1 -- 172.21.15.161:0/1785729266 >> v1:172.21.15.170:6789/0 conn(0x7f744c06cdd0 legacy=0x7f744c1524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.185+0000 7f744bfff640 1 -- 172.21.15.161:0/1785729266 >> v1:172.21.15.161:6789/0 conn(0x7f744c152bd0 legacy=0x7f744c15c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.185+0000 7f744bfff640 1 -- 172.21.15.161:0/1785729266 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f744c06c140 con 0x7f744c14f5c0 2024-03-28T23:36:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.186+0000 7f744bfff640 1 --2- 172.21.15.161:0/1785729266 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f744c14f5c0 0x7f744c0c2150 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f743c0097d0 tx=0x7f743c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53c88d6fd85da2c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.186+0000 7f744affd640 1 -- 172.21.15.161:0/1785729266 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c035020 con 0x7f744c14f5c0 2024-03-28T23:36:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.186+0000 7f744affd640 1 -- 172.21.15.161:0/1785729266 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f743c031890 con 0x7f744c14f5c0 2024-03-28T23:36:00.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.186+0000 7f744affd640 1 -- 172.21.15.161:0/1785729266 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c031c60 con 0x7f744c14f5c0 2024-03-28T23:36:00.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.187+0000 7f74526ed640 1 -- 172.21.15.161:0/1785729266 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f744c14f5c0 msgr2=0x7f744c0c2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:00.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.187+0000 7f74526ed640 1 --2- 172.21.15.161:0/1785729266 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f744c14f5c0 0x7f744c0c2150 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f743c0097d0 tx=0x7f743c02b930 comp rx=0 tx=0).stop 2024-03-28T23:36:00.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.187+0000 7f74526ed640 1 -- 172.21.15.161:0/1785729266 shutdown_connections 2024-03-28T23:36:00.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.187+0000 7f74526ed640 1 --2- 172.21.15.161:0/1785729266 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f744c14f5c0 0x7f744c0c2150 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:00.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.187+0000 7f74526ed640 1 -- 172.21.15.161:0/1785729266 >> 172.21.15.161:0/1785729266 conn(0x7f744c0a83f0 msgr2=0x7f744c0a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:00.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.187+0000 7f74526ed640 1 -- 172.21.15.161:0/1785729266 shutdown_connections 2024-03-28T23:36:00.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.187+0000 7f74526ed640 1 -- 172.21.15.161:0/1785729266 wait complete. 2024-03-28T23:36:00.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.188+0000 7f74526ed640 1 Processor -- start 2024-03-28T23:36:00.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.188+0000 7f74526ed640 1 -- start start 2024-03-28T23:36:00.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.189+0000 7f74526ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f744c06cdd0 0x7f744c1e7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:00.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.189+0000 7f74526ed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f744c14f5c0 0x7f744c0c54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:00.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.189+0000 7f74526ed640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f744c152bd0 0x7f744c148850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:00.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.189+0000 7f74526ed640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f744c152030 con 0x7f744c152bd0 2024-03-28T23:36:00.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.189+0000 7f74526ed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f744c0c6fa0 con 0x7f744c14f5c0 2024-03-28T23:36:00.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.189+0000 7f74526ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f744c06c3e0 con 0x7f744c06cdd0 2024-03-28T23:36:00.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.189+0000 7f7450c8a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f744c152bd0 0x7f744c148850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:00.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.189+0000 7f7450c8a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f744c152bd0 0x7f744c148850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57710/0 (socket says 172.21.15.161:57710) 2024-03-28T23:36:00.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.189+0000 7f7450c8a640 1 -- 172.21.15.161:0/1836953783 learned_addr learned my addr 172.21.15.161:0/1836953783 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:00.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.189+0000 7f744bfff640 1 --2- 172.21.15.161:0/1836953783 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f744c06cdd0 0x7f744c1e7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:00.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.189+0000 7f744b7fe640 1 --2- 172.21.15.161:0/1836953783 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f744c14f5c0 0x7f744c0c54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:00.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.189+0000 7f7450c8a640 1 -- 172.21.15.161:0/1836953783 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f744c06cdd0 msgr2=0x7f744c1e7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:00.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.189+0000 7f7450c8a640 1 --2- 172.21.15.161:0/1836953783 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f744c06cdd0 0x7f744c1e7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:00.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.189+0000 7f7450c8a640 1 -- 172.21.15.161:0/1836953783 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f744c14f5c0 msgr2=0x7f744c0c54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:00.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.189+0000 7f7450c8a640 1 --2- 172.21.15.161:0/1836953783 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f744c14f5c0 0x7f744c0c54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:00.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.189+0000 7f7450c8a640 1 -- 172.21.15.161:0/1836953783 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f744c0c6080 con 0x7f744c152bd0 2024-03-28T23:36:00.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.189+0000 7f744bfff640 1 --2- 172.21.15.161:0/1836953783 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f744c06cdd0 0x7f744c1e7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:00.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.189+0000 7f7450c8a640 1 --2- 172.21.15.161:0/1836953783 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f744c152bd0 0x7f744c148850 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f744400cbf0 tx=0x7f744400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:00.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.189+0000 7f7448ff9640 1 -- 172.21.15.161:0/1836953783 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7444017070 con 0x7f744c152bd0 2024-03-28T23:36:00.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.189+0000 7f7448ff9640 1 -- 172.21.15.161:0/1836953783 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f744400acf0 con 0x7f744c152bd0 2024-03-28T23:36:00.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.189+0000 7f74526ed640 1 -- 172.21.15.161:0/1836953783 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f744c155ed0 con 0x7f744c152bd0 2024-03-28T23:36:00.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.189+0000 7f7448ff9640 1 -- 172.21.15.161:0/1836953783 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7444012600 con 0x7f744c152bd0 2024-03-28T23:36:00.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.189+0000 7f74526ed640 1 -- 172.21.15.161:0/1836953783 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f744c071f00 con 0x7f744c152bd0 2024-03-28T23:36:00.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.190+0000 7f74526ed640 1 -- 172.21.15.161:0/1836953783 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f744c04f4b0 con 0x7f744c152bd0 2024-03-28T23:36:00.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.191+0000 7f7448ff9640 1 -- 172.21.15.161:0/1836953783 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f744401a070 con 0x7f744c152bd0 2024-03-28T23:36:00.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.191+0000 7f7448ff9640 1 --2- 172.21.15.161:0/1836953783 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f743806b4d0 0x7f743806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:00.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.191+0000 7f7448ff9640 1 -- 172.21.15.161:0/1836953783 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 6101+0+0 (secure 0 0 0) 0x7f744408a130 con 0x7f744c152bd0 2024-03-28T23:36:00.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.191+0000 7f744bfff640 1 --2- 172.21.15.161:0/1836953783 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f743806b4d0 0x7f743806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:00.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.192+0000 7f744bfff640 1 --2- 172.21.15.161:0/1836953783 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f743806b4d0 0x7f743806d990 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f743c036000 tx=0x7f743c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:00.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.196+0000 7f7448ff9640 1 -- 172.21.15.161:0/1836953783 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f744c0c6080 con 0x7f744c152bd0 2024-03-28T23:36:00.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:00.405+0000 7f74526ed640 1 -- 172.21.15.161:0/1836953783 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f744c0720d0 con 0x7f744c152bd0 2024-03-28T23:36:01.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.071+0000 7f7448ff9640 1 -- 172.21.15.161:0/1836953783 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v140) ==== 122+0+0 (secure 0 0 0) 0x7f744c0c2f10 con 0x7f744c152bd0 2024-03-28T23:36:01.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.073+0000 7f74526ed640 1 -- 172.21.15.161:0/1836953783 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f743806b4d0 msgr2=0x7f743806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:01.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.073+0000 7f74526ed640 1 --2- 172.21.15.161:0/1836953783 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f743806b4d0 0x7f743806d990 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f743c036000 tx=0x7f743c008030 comp rx=0 tx=0).stop 2024-03-28T23:36:01.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.073+0000 7f74526ed640 1 -- 172.21.15.161:0/1836953783 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f744c152bd0 msgr2=0x7f744c148850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:01.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.073+0000 7f74526ed640 1 --2- 172.21.15.161:0/1836953783 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f744c152bd0 0x7f744c148850 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f744400cbf0 tx=0x7f744400a430 comp rx=0 tx=0).stop 2024-03-28T23:36:01.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.073+0000 7f74526ed640 1 -- 172.21.15.161:0/1836953783 shutdown_connections 2024-03-28T23:36:01.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.073+0000 7f74526ed640 1 --2- 172.21.15.161:0/1836953783 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f743806b4d0 0x7f743806d990 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.073+0000 7f74526ed640 1 --2- 172.21.15.161:0/1836953783 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f744c152bd0 0x7f744c148850 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.073+0000 7f74526ed640 1 --2- 172.21.15.161:0/1836953783 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f744c14f5c0 0x7f744c0c54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.073+0000 7f74526ed640 1 --2- 172.21.15.161:0/1836953783 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f744c06cdd0 0x7f744c1e7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.073+0000 7f74526ed640 1 -- 172.21.15.161:0/1836953783 >> 172.21.15.161:0/1836953783 conn(0x7f744c0a83f0 msgr2=0x7f744c14bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:01.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.074+0000 7f74526ed640 1 -- 172.21.15.161:0/1836953783 shutdown_connections 2024-03-28T23:36:01.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.074+0000 7f74526ed640 1 -- 172.21.15.161:0/1836953783 wait complete. 2024-03-28T23:36:01.086 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-28T23:36:01.087 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:36:01.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.193+0000 7fc7728a4640 1 Processor -- start 2024-03-28T23:36:01.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.193+0000 7fc7728a4640 1 -- start start 2024-03-28T23:36:01.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.193+0000 7fc7728a4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc76c152890 0x7fc76c0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:01.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.193+0000 7fc7728a4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc76c0c8ed0 0x7fc76c0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:01.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.193+0000 7fc7728a4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc76c0c5950 0x7fc76c1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:01.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.193+0000 7fc7728a4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc76c0c8480 con 0x7fc76c0c8ed0 2024-03-28T23:36:01.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.193+0000 7fc7728a4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fc76c159640 con 0x7fc76c152890 2024-03-28T23:36:01.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.193+0000 7fc7728a4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc76c158a30 con 0x7fc76c0c5950 2024-03-28T23:36:01.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.194+0000 7fc76b7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc76c0c8ed0 0x7fc76c0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:01.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.194+0000 7fc76b7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc76c0c8ed0 0x7fc76c0c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53692/0 (socket says 172.21.15.161:53692) 2024-03-28T23:36:01.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.194+0000 7fc76b7fe640 1 -- 172.21.15.161:0/3322140400 learned_addr learned my addr 172.21.15.161:0/3322140400 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:01.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.194+0000 7fc770e41640 1 --2- 172.21.15.161:0/3322140400 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc76c0c5950 0x7fc76c1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:01.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.194+0000 7fc76bfff640 1 --2- 172.21.15.161:0/3322140400 >> v2:172.21.15.170:3300/0 conn(0x7fc76c152890 0x7fc76c0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:01.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.194+0000 7fc76b7fe640 1 -- 172.21.15.161:0/3322140400 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc76c0c5950 msgr2=0x7fc76c1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:01.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.194+0000 7fc76b7fe640 1 --2- 172.21.15.161:0/3322140400 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc76c0c5950 0x7fc76c1553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.194+0000 7fc76b7fe640 1 -- 172.21.15.161:0/3322140400 >> v2:172.21.15.170:3300/0 conn(0x7fc76c152890 msgr2=0x7fc76c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:01.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.194+0000 7fc76b7fe640 1 --2- 172.21.15.161:0/3322140400 >> v2:172.21.15.170:3300/0 conn(0x7fc76c152890 0x7fc76c0c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.194+0000 7fc76b7fe640 1 -- 172.21.15.161:0/3322140400 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc76c158d10 con 0x7fc76c0c8ed0 2024-03-28T23:36:01.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.195+0000 7fc76b7fe640 1 --2- 172.21.15.161:0/3322140400 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc76c0c8ed0 0x7fc76c0c5410 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fc76c08cc80 tx=0x7fc76002b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65e4c30003eb0ec3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:01.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.195+0000 7fc76affd640 1 -- 172.21.15.161:0/3322140400 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc760038070 con 0x7fc76c0c8ed0 2024-03-28T23:36:01.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.195+0000 7fc76affd640 1 -- 172.21.15.161:0/3322140400 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc760004030 con 0x7fc76c0c8ed0 2024-03-28T23:36:01.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.195+0000 7fc76affd640 1 -- 172.21.15.161:0/3322140400 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc760004460 con 0x7fc76c0c8ed0 2024-03-28T23:36:01.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.195+0000 7fc7728a4640 1 -- 172.21.15.161:0/3322140400 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc76c0c8ed0 msgr2=0x7fc76c0c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:01.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.195+0000 7fc7728a4640 1 --2- 172.21.15.161:0/3322140400 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc76c0c8ed0 0x7fc76c0c5410 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fc76c08cc80 tx=0x7fc76002b7c0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.195+0000 7fc7728a4640 1 -- 172.21.15.161:0/3322140400 shutdown_connections 2024-03-28T23:36:01.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.195+0000 7fc7728a4640 1 --2- 172.21.15.161:0/3322140400 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc76c0c5950 0x7fc76c1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.195+0000 7fc7728a4640 1 --2- 172.21.15.161:0/3322140400 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc76c0c8ed0 0x7fc76c0c5410 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.195+0000 7fc7728a4640 1 --2- 172.21.15.161:0/3322140400 >> v2:172.21.15.170:3300/0 conn(0x7fc76c152890 0x7fc76c0c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.195+0000 7fc7728a4640 1 -- 172.21.15.161:0/3322140400 >> 172.21.15.161:0/3322140400 conn(0x7fc76c0ac620 msgr2=0x7fc76c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:01.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.195+0000 7fc7728a4640 1 -- 172.21.15.161:0/3322140400 shutdown_connections 2024-03-28T23:36:01.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.195+0000 7fc7728a4640 1 -- 172.21.15.161:0/3322140400 wait complete. 2024-03-28T23:36:01.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.196+0000 7fc7728a4640 1 Processor -- start 2024-03-28T23:36:01.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.197+0000 7fc7728a4640 1 -- start start 2024-03-28T23:36:01.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.197+0000 7fc7728a4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc76c0c5950 0x7fc76c155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:01.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.197+0000 7fc7728a4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc76c0c8ed0 0x7fc76c14eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:01.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.197+0000 7fc7728a4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc76c152890 0x7fc76c1472e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:01.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.197+0000 7fc7728a4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c152c70 con 0x7fc76c152890 2024-03-28T23:36:01.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.197+0000 7fc7728a4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c1592d0 con 0x7fc76c0c8ed0 2024-03-28T23:36:01.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.197+0000 7fc7728a4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc76c152df0 con 0x7fc76c0c5950 2024-03-28T23:36:01.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.197+0000 7fc770e41640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc76c152890 0x7fc76c1472e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:01.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.197+0000 7fc770e41640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc76c152890 0x7fc76c1472e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53694/0 (socket says 172.21.15.161:53694) 2024-03-28T23:36:01.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.197+0000 7fc770e41640 1 -- 172.21.15.161:0/112357631 learned_addr learned my addr 172.21.15.161:0/112357631 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:01.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.197+0000 7fc76bfff640 1 --2- 172.21.15.161:0/112357631 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc76c0c5950 0x7fc76c155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:01.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.197+0000 7fc76b7fe640 1 --2- 172.21.15.161:0/112357631 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc76c0c8ed0 0x7fc76c14eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:01.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.198+0000 7fc770e41640 1 -- 172.21.15.161:0/112357631 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc76c0c5950 msgr2=0x7fc76c155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:01.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.198+0000 7fc770e41640 1 --2- 172.21.15.161:0/112357631 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc76c0c5950 0x7fc76c155aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.198+0000 7fc770e41640 1 -- 172.21.15.161:0/112357631 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc76c0c8ed0 msgr2=0x7fc76c14eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:01.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.198+0000 7fc770e41640 1 --2- 172.21.15.161:0/112357631 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc76c0c8ed0 0x7fc76c14eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.198+0000 7fc770e41640 1 -- 172.21.15.161:0/112357631 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc76c14f750 con 0x7fc76c152890 2024-03-28T23:36:01.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.198+0000 7fc770e41640 1 --2- 172.21.15.161:0/112357631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc76c152890 0x7fc76c1472e0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fc76400eed0 tx=0x7fc76400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:01.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.198+0000 7fc768ff9640 1 -- 172.21.15.161:0/112357631 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc764018020 con 0x7fc76c152890 2024-03-28T23:36:01.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.198+0000 7fc768ff9640 1 -- 172.21.15.161:0/112357631 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc76400bd20 con 0x7fc76c152890 2024-03-28T23:36:01.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.198+0000 7fc7728a4640 1 -- 172.21.15.161:0/112357631 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc76c14fa30 con 0x7fc76c152890 2024-03-28T23:36:01.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.198+0000 7fc768ff9640 1 -- 172.21.15.161:0/112357631 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc764014640 con 0x7fc76c152890 2024-03-28T23:36:01.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.198+0000 7fc7728a4640 1 -- 172.21.15.161:0/112357631 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc76c14cfb0 con 0x7fc76c152890 2024-03-28T23:36:01.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.199+0000 7fc7728a4640 1 -- 172.21.15.161:0/112357631 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc76c04f4b0 con 0x7fc76c152890 2024-03-28T23:36:01.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.200+0000 7fc768ff9640 1 -- 172.21.15.161:0/112357631 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc764005d10 con 0x7fc76c152890 2024-03-28T23:36:01.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.200+0000 7fc768ff9640 1 --2- 172.21.15.161:0/112357631 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc75806b4d0 0x7fc75806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:01.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.200+0000 7fc768ff9640 1 -- 172.21.15.161:0/112357631 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 6101+0+0 (secure 0 0 0) 0x7fc76408a7f0 con 0x7fc76c152890 2024-03-28T23:36:01.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.200+0000 7fc76bfff640 1 --2- 172.21.15.161:0/112357631 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc75806b4d0 0x7fc75806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:01.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.201+0000 7fc76bfff640 1 --2- 172.21.15.161:0/112357631 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc75806b4d0 0x7fc75806d990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fc75c009b40 tx=0x7fc75c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:01.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.204+0000 7fc768ff9640 1 -- 172.21.15.161:0/112357631 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc764022070 con 0x7fc76c152890 2024-03-28T23:36:01.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.411+0000 7fc7728a4640 1 -- 172.21.15.161:0/112357631 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc76c0c8140 con 0x7fc76c152890 2024-03-28T23:36:01.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.411+0000 7fc768ff9640 1 -- 172.21.15.161:0/112357631 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) ==== 95+0+5200 (secure 0 0 0) 0x7fc764058f20 con 0x7fc76c152890 2024-03-28T23:36:01.412 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:01.412 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":140,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7061,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4026332703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4026332703},{"type":"v1","addr":"172.21.15.161:6836","nonce":4026332703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":7070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/557522063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":557522063},{"type":"v1","addr":"172.21.15.170:6835","nonce":557522063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":8056,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/2391019771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":2391019771},{"type":"v1","addr":"172.21.15.161:6837","nonce":2391019771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:58.406403+0000","modified":"2024-03-28T23:36:00.406209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8068},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8068":{"gid":8068,"name":"b","rank":0,"incarnation":138,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6834/1393137291","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1393137291},{"type":"v1","addr":"172.21.15.170:6834","nonce":1393137291}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8068,"qdb_cluster":[8068]},"id":11}]} 2024-03-28T23:36:01.413 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 140 2024-03-28T23:36:01.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.413+0000 7fc7728a4640 1 -- 172.21.15.161:0/112357631 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc75806b4d0 msgr2=0x7fc75806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:01.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.413+0000 7fc7728a4640 1 --2- 172.21.15.161:0/112357631 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc75806b4d0 0x7fc75806d990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fc75c009b40 tx=0x7fc75c009190 comp rx=0 tx=0).stop 2024-03-28T23:36:01.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.413+0000 7fc7728a4640 1 -- 172.21.15.161:0/112357631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc76c152890 msgr2=0x7fc76c1472e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:01.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.413+0000 7fc7728a4640 1 --2- 172.21.15.161:0/112357631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc76c152890 0x7fc76c1472e0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fc76400eed0 tx=0x7fc76400b480 comp rx=0 tx=0).stop 2024-03-28T23:36:01.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.413+0000 7fc7728a4640 1 -- 172.21.15.161:0/112357631 shutdown_connections 2024-03-28T23:36:01.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.413+0000 7fc7728a4640 1 --2- 172.21.15.161:0/112357631 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc75806b4d0 0x7fc75806d990 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.413+0000 7fc7728a4640 1 --2- 172.21.15.161:0/112357631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc76c152890 0x7fc76c1472e0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.413+0000 7fc7728a4640 1 --2- 172.21.15.161:0/112357631 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc76c0c8ed0 0x7fc76c14eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.414+0000 7fc7728a4640 1 --2- 172.21.15.161:0/112357631 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc76c0c5950 0x7fc76c155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.414+0000 7fc7728a4640 1 -- 172.21.15.161:0/112357631 >> 172.21.15.161:0/112357631 conn(0x7fc76c0ac620 msgr2=0x7fc76c150b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:01.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.414+0000 7fc7728a4640 1 -- 172.21.15.161:0/112357631 shutdown_connections 2024-03-28T23:36:01.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.414+0000 7fc7728a4640 1 -- 172.21.15.161:0/112357631 wait complete. 2024-03-28T23:36:01.425 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:36:01.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.526+0000 7fa26b864640 1 Processor -- start 2024-03-28T23:36:01.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.526+0000 7fa26b864640 1 -- start start 2024-03-28T23:36:01.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.527+0000 7fa26b864640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa26406c160 0x7fa26406e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:01.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.527+0000 7fa26b864640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa26406c9a0 con 0x7fa26406c540 2024-03-28T23:36:01.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.527+0000 7fa26b864640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa26406f650 con 0x7fa2640c1cf0 2024-03-28T23:36:01.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.527+0000 7fa26b864640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fa26406e290 con 0x7fa26406c160 2024-03-28T23:36:01.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.527+0000 7fa268dff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa26406c540 0x7fa264150560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50066/0 (socket says 172.21.15.161:50066) 2024-03-28T23:36:01.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.527+0000 7fa268dff640 1 -- 172.21.15.161:0/323397503 learned_addr learned my addr 172.21.15.161:0/323397503 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:01.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.527+0000 7fa269600640 1 --2- 172.21.15.161:0/323397503 >> v2:172.21.15.170:3300/0 conn(0x7fa26406c160 0x7fa26406e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:01.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.527+0000 7fa25bfff640 1 -- 172.21.15.161:0/323397503 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4275619848 0 0) 0x7fa26406c9a0 con 0x7fa26406c540 2024-03-28T23:36:01.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.527+0000 7fa25bfff640 1 -- 172.21.15.161:0/323397503 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa250003520 con 0x7fa26406c540 2024-03-28T23:36:01.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.527+0000 7fa269600640 1 -- 172.21.15.161:0/323397503 >> v1:172.21.15.170:6789/0 conn(0x7fa2640c1cf0 legacy=0x7fa26415d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:01.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.527+0000 7fa269600640 1 -- 172.21.15.161:0/323397503 >> v1:172.21.15.161:6789/0 conn(0x7fa26406c540 legacy=0x7fa264150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:01.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.527+0000 7fa269600640 1 -- 172.21.15.161:0/323397503 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa26406f370 con 0x7fa26406c160 2024-03-28T23:36:01.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.528+0000 7fa269600640 1 --2- 172.21.15.161:0/323397503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa26406c160 0x7fa26406e720 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fa2600097a0 tx=0x7fa26002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c085d37621e7d814 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:01.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.528+0000 7fa25bfff640 1 -- 172.21.15.161:0/323397503 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa260035020 con 0x7fa26406c160 2024-03-28T23:36:01.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.528+0000 7fa25bfff640 1 -- 172.21.15.161:0/323397503 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa260031830 con 0x7fa26406c160 2024-03-28T23:36:01.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.528+0000 7fa25bfff640 1 -- 172.21.15.161:0/323397503 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa260031be0 con 0x7fa26406c160 2024-03-28T23:36:01.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.529+0000 7fa26b864640 1 -- 172.21.15.161:0/323397503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa26406c160 msgr2=0x7fa26406e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:01.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.529+0000 7fa26b864640 1 --2- 172.21.15.161:0/323397503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa26406c160 0x7fa26406e720 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fa2600097a0 tx=0x7fa26002b8d0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.529+0000 7fa26b864640 1 -- 172.21.15.161:0/323397503 shutdown_connections 2024-03-28T23:36:01.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.529+0000 7fa26b864640 1 --2- 172.21.15.161:0/323397503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa26406c160 0x7fa26406e720 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.529+0000 7fa26b864640 1 -- 172.21.15.161:0/323397503 >> 172.21.15.161:0/323397503 conn(0x7fa2640c3be0 msgr2=0x7fa2640c3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:01.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.529+0000 7fa26b864640 1 -- 172.21.15.161:0/323397503 shutdown_connections 2024-03-28T23:36:01.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.529+0000 7fa26b864640 1 -- 172.21.15.161:0/323397503 wait complete. 2024-03-28T23:36:01.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.530+0000 7fa26b864640 1 Processor -- start 2024-03-28T23:36:01.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.530+0000 7fa26b864640 1 -- start start 2024-03-28T23:36:01.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.531+0000 7fa26b864640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa26406c160 0x7fa2641dd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:01.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.531+0000 7fa26b864640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa26406c540 0x7fa264152760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:01.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.531+0000 7fa26b864640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2640c1cf0 0x7fa2641d92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:01.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.531+0000 7fa26b864640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa26406df20 con 0x7fa26406c540 2024-03-28T23:36:01.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.531+0000 7fa26b864640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa2640c2180 con 0x7fa26406c160 2024-03-28T23:36:01.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.531+0000 7fa26b864640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa2640c2390 con 0x7fa2640c1cf0 2024-03-28T23:36:01.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.531+0000 7fa268dff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa26406c540 0x7fa264152760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:01.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.531+0000 7fa268dff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa26406c540 0x7fa264152760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53702/0 (socket says 172.21.15.161:53702) 2024-03-28T23:36:01.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.531+0000 7fa268dff640 1 -- 172.21.15.161:0/3497209413 learned_addr learned my addr 172.21.15.161:0/3497209413 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:01.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.531+0000 7fa269600640 1 --2- 172.21.15.161:0/3497209413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa26406c160 0x7fa2641dd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:01.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.531+0000 7fa269e01640 1 --2- 172.21.15.161:0/3497209413 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2640c1cf0 0x7fa2641d92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:01.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.531+0000 7fa268dff640 1 -- 172.21.15.161:0/3497209413 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2640c1cf0 msgr2=0x7fa2641d92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:01.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.531+0000 7fa268dff640 1 --2- 172.21.15.161:0/3497209413 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2640c1cf0 0x7fa2641d92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.531+0000 7fa268dff640 1 -- 172.21.15.161:0/3497209413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa26406c160 msgr2=0x7fa2641dd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:01.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.531+0000 7fa268dff640 1 --2- 172.21.15.161:0/3497209413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa26406c160 0x7fa2641dd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.531+0000 7fa268dff640 1 -- 172.21.15.161:0/3497209413 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa264153080 con 0x7fa26406c540 2024-03-28T23:36:01.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.531+0000 7fa268dff640 1 --2- 172.21.15.161:0/3497209413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa26406c540 0x7fa264152760 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fa25400c500 tx=0x7fa25400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:01.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.532+0000 7fa259ffb640 1 -- 172.21.15.161:0/3497209413 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa254017070 con 0x7fa26406c540 2024-03-28T23:36:01.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.532+0000 7fa259ffb640 1 -- 172.21.15.161:0/3497209413 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa254005ce0 con 0x7fa26406c540 2024-03-28T23:36:01.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.532+0000 7fa26b864640 1 -- 172.21.15.161:0/3497209413 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2641ec6c0 con 0x7fa26406c540 2024-03-28T23:36:01.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.532+0000 7fa259ffb640 1 -- 172.21.15.161:0/3497209413 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa254007330 con 0x7fa26406c540 2024-03-28T23:36:01.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.532+0000 7fa26b864640 1 -- 172.21.15.161:0/3497209413 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa264152ca0 con 0x7fa26406c540 2024-03-28T23:36:01.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.533+0000 7fa26b864640 1 -- 172.21.15.161:0/3497209413 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa26404f4b0 con 0x7fa26406c540 2024-03-28T23:36:01.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.533+0000 7fa259ffb640 1 -- 172.21.15.161:0/3497209413 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa25401a070 con 0x7fa26406c540 2024-03-28T23:36:01.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.533+0000 7fa259ffb640 1 --2- 172.21.15.161:0/3497209413 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa24806b500 0x7fa24806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:01.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.534+0000 7fa259ffb640 1 -- 172.21.15.161:0/3497209413 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 6101+0+0 (secure 0 0 0) 0x7fa2540896a0 con 0x7fa26406c540 2024-03-28T23:36:01.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.534+0000 7fa269600640 1 --2- 172.21.15.161:0/3497209413 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa24806b500 0x7fa24806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:01.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.534+0000 7fa269600640 1 --2- 172.21.15.161:0/3497209413 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa24806b500 0x7fa24806d9c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fa260036000 tx=0x7fa260008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:01.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.538+0000 7fa259ffb640 1 -- 172.21.15.161:0/3497209413 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa254057e90 con 0x7fa26406c540 2024-03-28T23:36:01.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.712+0000 7fa26b864640 1 -- 172.21.15.161:0/3497209413 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa2640c6940 con 0x7fa26406c540 2024-03-28T23:36:01.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.713+0000 7fa259ffb640 1 -- 172.21.15.161:0/3497209413 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) ==== 74+0+16920 (secure 0 0 0) 0x7fa25405bf30 con 0x7fa26406c540 2024-03-28T23:36:01.714 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:01.714 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":152,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:36:00.062710+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:35:56.687416+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-28T23:35:57.379120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:36:01.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.715+0000 7fa26b864640 1 -- 172.21.15.161:0/3497209413 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa24806b500 msgr2=0x7fa24806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:01.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.715+0000 7fa26b864640 1 --2- 172.21.15.161:0/3497209413 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa24806b500 0x7fa24806d9c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fa260036000 tx=0x7fa260008030 comp rx=0 tx=0).stop 2024-03-28T23:36:01.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.715+0000 7fa26b864640 1 -- 172.21.15.161:0/3497209413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa26406c540 msgr2=0x7fa264152760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:01.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.715+0000 7fa26b864640 1 --2- 172.21.15.161:0/3497209413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa26406c540 0x7fa264152760 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fa25400c500 tx=0x7fa25400c9c0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.716+0000 7fa26b864640 1 -- 172.21.15.161:0/3497209413 shutdown_connections 2024-03-28T23:36:01.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.716+0000 7fa26b864640 1 --2- 172.21.15.161:0/3497209413 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa24806b500 0x7fa24806d9c0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.716+0000 7fa26b864640 1 --2- 172.21.15.161:0/3497209413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa26406c540 0x7fa264152760 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.716+0000 7fa26b864640 1 --2- 172.21.15.161:0/3497209413 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2640c1cf0 0x7fa2641d92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.716+0000 7fa26b864640 1 --2- 172.21.15.161:0/3497209413 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa26406c160 0x7fa2641dd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.716+0000 7fa26b864640 1 -- 172.21.15.161:0/3497209413 >> 172.21.15.161:0/3497209413 conn(0x7fa2640c3be0 msgr2=0x7fa26414bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:01.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.716+0000 7fa26b864640 1 -- 172.21.15.161:0/3497209413 shutdown_connections 2024-03-28T23:36:01.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.716+0000 7fa26b864640 1 -- 172.21.15.161:0/3497209413 wait complete. 2024-03-28T23:36:01.727 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T23:36:01.728 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:36:01.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.829+0000 7f186a365640 1 Processor -- start 2024-03-28T23:36:01.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.829+0000 7f186a365640 1 -- start start 2024-03-28T23:36:01.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.829+0000 7f186a365640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f186414be70 0x7f18640ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:01.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.829+0000 7f186a365640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f18640aacd0 0x7f186415c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:01.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.829+0000 7f186a365640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f186414c250 con 0x7f18640ab6e0 2024-03-28T23:36:01.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.829+0000 7f186a365640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f18641529c0 con 0x7f18640aacd0 2024-03-28T23:36:01.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.829+0000 7f186a365640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f18640a9dc0 con 0x7f186414be70 2024-03-28T23:36:01.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.830+0000 7f1868902640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f18640aacd0 0x7f186415c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:01.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.830+0000 7f1868902640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f18640aacd0 0x7f186415c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53710/0 (socket says 172.21.15.161:53710) 2024-03-28T23:36:01.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.830+0000 7f1868902640 1 -- 172.21.15.161:0/2587429324 learned_addr learned my addr 172.21.15.161:0/2587429324 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:01.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.830+0000 7f18637fe640 1 --1- 172.21.15.161:0/2587429324 >> v1:172.21.15.161:6789/0 conn(0x7f18640ab6e0 0x7f18640aa5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50074/0 (socket says 172.21.15.161:50074) 2024-03-28T23:36:01.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.830+0000 7f1863fff640 1 --2- 172.21.15.161:0/2587429324 >> v2:172.21.15.170:3300/0 conn(0x7f186414be70 0x7f18640ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:01.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.830+0000 7f1868902640 1 -- 172.21.15.161:0/2587429324 >> v2:172.21.15.170:3300/0 conn(0x7f186414be70 msgr2=0x7f18640ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:01.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.830+0000 7f1868902640 1 --2- 172.21.15.161:0/2587429324 >> v2:172.21.15.170:3300/0 conn(0x7f186414be70 0x7f18640ab1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.830+0000 7f1868902640 1 -- 172.21.15.161:0/2587429324 >> v1:172.21.15.161:6789/0 conn(0x7f18640ab6e0 legacy=0x7f18640aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:01.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.830+0000 7f1868902640 1 -- 172.21.15.161:0/2587429324 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18640aa0a0 con 0x7f18640aacd0 2024-03-28T23:36:01.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.830+0000 7f1862ffd640 1 -- 172.21.15.161:0/2587429324 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2861056053 0 0) 0x7f186414c250 con 0x7f18640ab6e0 2024-03-28T23:36:01.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.831+0000 7f1868902640 1 --2- 172.21.15.161:0/2587429324 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18640aacd0 0x7f186415c740 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f18640c7520 tx=0x7f185c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1af678646ada97dc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:01.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.831+0000 7f1862ffd640 1 -- 172.21.15.161:0/2587429324 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185c02ca80 con 0x7f18640aacd0 2024-03-28T23:36:01.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.831+0000 7f1862ffd640 1 -- 172.21.15.161:0/2587429324 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f185c02cc20 con 0x7f18640aacd0 2024-03-28T23:36:01.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.831+0000 7f1862ffd640 1 -- 172.21.15.161:0/2587429324 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185c0343f0 con 0x7f18640aacd0 2024-03-28T23:36:01.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.831+0000 7f186a365640 1 -- 172.21.15.161:0/2587429324 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18640aacd0 msgr2=0x7f186415c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:01.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.831+0000 7f186a365640 1 --2- 172.21.15.161:0/2587429324 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18640aacd0 0x7f186415c740 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f18640c7520 tx=0x7f185c02bee0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.831+0000 7f186a365640 1 -- 172.21.15.161:0/2587429324 shutdown_connections 2024-03-28T23:36:01.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.831+0000 7f186a365640 1 --2- 172.21.15.161:0/2587429324 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18640aacd0 0x7f186415c740 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.831+0000 7f186a365640 1 --2- 172.21.15.161:0/2587429324 >> v2:172.21.15.170:3300/0 conn(0x7f186414be70 0x7f18640ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.831+0000 7f186a365640 1 -- 172.21.15.161:0/2587429324 >> 172.21.15.161:0/2587429324 conn(0x7f18640c6670 msgr2=0x7f18640c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:01.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.831+0000 7f186a365640 1 -- 172.21.15.161:0/2587429324 shutdown_connections 2024-03-28T23:36:01.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.831+0000 7f186a365640 1 -- 172.21.15.161:0/2587429324 wait complete. 2024-03-28T23:36:01.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.832+0000 7f186a365640 1 Processor -- start 2024-03-28T23:36:01.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.833+0000 7f186a365640 1 -- start start 2024-03-28T23:36:01.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.833+0000 7f186a365640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18640aacd0 0x7f1864153640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:01.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.833+0000 7f186a365640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18640ab6e0 0x7f186414ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:01.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.833+0000 7f186a365640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f186414be70 0x7f186414d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:01.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.833+0000 7f186a365640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f186406b2d0 con 0x7f18640ab6e0 2024-03-28T23:36:01.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.833+0000 7f186a365640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f186406b150 con 0x7f18640aacd0 2024-03-28T23:36:01.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.833+0000 7f186a365640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f18640c1520 con 0x7f186414be70 2024-03-28T23:36:01.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.833+0000 7f18637fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18640ab6e0 0x7f186414ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:01.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.833+0000 7f18637fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18640ab6e0 0x7f186414ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53720/0 (socket says 172.21.15.161:53720) 2024-03-28T23:36:01.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.833+0000 7f18637fe640 1 -- 172.21.15.161:0/4198777124 learned_addr learned my addr 172.21.15.161:0/4198777124 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:01.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.833+0000 7f1863fff640 1 --2- 172.21.15.161:0/4198777124 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18640aacd0 0x7f1864153640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:01.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.833+0000 7f1868902640 1 --2- 172.21.15.161:0/4198777124 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f186414be70 0x7f186414d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:01.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.834+0000 7f18637fe640 1 -- 172.21.15.161:0/4198777124 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f186414be70 msgr2=0x7f186414d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:01.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.834+0000 7f18637fe640 1 --2- 172.21.15.161:0/4198777124 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f186414be70 0x7f186414d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.834+0000 7f18637fe640 1 -- 172.21.15.161:0/4198777124 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18640aacd0 msgr2=0x7f1864153640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:01.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.834+0000 7f18637fe640 1 --2- 172.21.15.161:0/4198777124 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18640aacd0 0x7f1864153640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:01.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.834+0000 7f18637fe640 1 -- 172.21.15.161:0/4198777124 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f186414c6d0 con 0x7f18640ab6e0 2024-03-28T23:36:01.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.834+0000 7f18637fe640 1 --2- 172.21.15.161:0/4198777124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18640ab6e0 0x7f186414ed20 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f185800c430 tx=0x7f185800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:01.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.834+0000 7f1860ff9640 1 -- 172.21.15.161:0/4198777124 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18580074e0 con 0x7f18640ab6e0 2024-03-28T23:36:01.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.834+0000 7f1860ff9640 1 -- 172.21.15.161:0/4198777124 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18580047e0 con 0x7f18640ab6e0 2024-03-28T23:36:01.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.834+0000 7f186a365640 1 -- 172.21.15.161:0/4198777124 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f186414c9b0 con 0x7f18640ab6e0 2024-03-28T23:36:01.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.834+0000 7f1860ff9640 1 -- 172.21.15.161:0/4198777124 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1858004bd0 con 0x7f18640ab6e0 2024-03-28T23:36:01.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.834+0000 7f186a365640 1 -- 172.21.15.161:0/4198777124 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f186414cf70 con 0x7f18640ab6e0 2024-03-28T23:36:01.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.835+0000 7f186a365640 1 -- 172.21.15.161:0/4198777124 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f186404f4b0 con 0x7f18640ab6e0 2024-03-28T23:36:01.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.836+0000 7f1860ff9640 1 -- 172.21.15.161:0/4198777124 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1858011070 con 0x7f18640ab6e0 2024-03-28T23:36:01.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.836+0000 7f1860ff9640 1 --2- 172.21.15.161:0/4198777124 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f185006b4d0 0x7f185006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:01.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.836+0000 7f1860ff9640 1 -- 172.21.15.161:0/4198777124 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 6101+0+0 (secure 0 0 0) 0x7f1858089da0 con 0x7f18640ab6e0 2024-03-28T23:36:01.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.836+0000 7f1863fff640 1 --2- 172.21.15.161:0/4198777124 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f185006b4d0 0x7f185006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:01.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.836+0000 7f1863fff640 1 --2- 172.21.15.161:0/4198777124 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f185006b4d0 0x7f185006d990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f1854009b40 tx=0x7f1854009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:01.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:01.840+0000 7f1860ff9640 1 -- 172.21.15.161:0/4198777124 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f18580584d0 con 0x7f18640ab6e0 2024-03-28T23:36:02.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.009+0000 7f186a365640 1 -- 172.21.15.161:0/4198777124 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f186414c480 con 0x7f185006b4d0 2024-03-28T23:36:02.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.014+0000 7f1860ff9640 1 -- 172.21.15.161:0/4198777124 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252483 (secure 0 0 0) 0x7f186414c480 con 0x7f185006b4d0 2024-03-28T23:36:02.017 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:02.019 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:36:02.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.018+0000 7f186a365640 1 -- 172.21.15.161:0/4198777124 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f185006b4d0 msgr2=0x7f185006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:02.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.018+0000 7f186a365640 1 --2- 172.21.15.161:0/4198777124 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f185006b4d0 0x7f185006d990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f1854009b40 tx=0x7f1854009190 comp rx=0 tx=0).stop 2024-03-28T23:36:02.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.018+0000 7f186a365640 1 -- 172.21.15.161:0/4198777124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18640ab6e0 msgr2=0x7f186414ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:02.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.018+0000 7f186a365640 1 --2- 172.21.15.161:0/4198777124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18640ab6e0 0x7f186414ed20 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f185800c430 tx=0x7f185800c8f0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.018+0000 7f186a365640 1 -- 172.21.15.161:0/4198777124 shutdown_connections 2024-03-28T23:36:02.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.018+0000 7f186a365640 1 --2- 172.21.15.161:0/4198777124 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f185006b4d0 0x7f185006d990 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.018+0000 7f186a365640 1 --2- 172.21.15.161:0/4198777124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18640ab6e0 0x7f186414ed20 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.018+0000 7f186a365640 1 --2- 172.21.15.161:0/4198777124 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f186414be70 0x7f186414d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.018+0000 7f186a365640 1 --2- 172.21.15.161:0/4198777124 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18640aacd0 0x7f1864153640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.018+0000 7f186a365640 1 -- 172.21.15.161:0/4198777124 >> 172.21.15.161:0/4198777124 conn(0x7f18640c6670 msgr2=0x7f18640c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:02.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.018+0000 7f186a365640 1 -- 172.21.15.161:0/4198777124 shutdown_connections 2024-03-28T23:36:02.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.018+0000 7f186a365640 1 -- 172.21.15.161:0/4198777124 wait complete. 2024-03-28T23:36:02.029 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-03-28T23:36:01.384431+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":45,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264424,"kb_used_data":14944,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754710296,"statfs":{"total":773094113280,"available":772823343104,"internally_reserved":0,"allocated":15302656,"data_stored":8495189,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.337135"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.083764+0000","last_change":"2024-03-28T23:35:59.077622+0000","last_active":"2024-03-28T23:35:59.083764+0000","last_peered":"2024-03-28T23:35:59.083764+0000","last_clean":"2024-03-28T23:35:59.083764+0000","last_became_active":"2024-03-28T23:35:59.077350+0000","last_became_peered":"2024-03-28T23:35:59.077350+0000","last_unstale":"2024-03-28T23:35:59.083764+0000","last_undegraded":"2024-03-28T23:35:59.083764+0000","last_fullsized":"2024-03-28T23:35:59.083764+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:25:34.583929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.075033+0000","last_change":"2024-03-28T23:35:59.069924+0000","last_active":"2024-03-28T23:35:59.075033+0000","last_peered":"2024-03-28T23:35:59.075033+0000","last_clean":"2024-03-28T23:35:59.075033+0000","last_became_active":"2024-03-28T23:35:59.069650+0000","last_became_peered":"2024-03-28T23:35:59.069650+0000","last_unstale":"2024-03-28T23:35:59.075033+0000","last_undegraded":"2024-03-28T23:35:59.075033+0000","last_fullsized":"2024-03-28T23:35:59.075033+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:16:54.129411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.084431+0000","last_change":"2024-03-28T23:35:59.082632+0000","last_active":"2024-03-28T23:35:59.084431+0000","last_peered":"2024-03-28T23:35:59.084431+0000","last_clean":"2024-03-28T23:35:59.084431+0000","last_became_active":"2024-03-28T23:35:59.082360+0000","last_became_peered":"2024-03-28T23:35:59.082360+0000","last_unstale":"2024-03-28T23:35:59.084431+0000","last_undegraded":"2024-03-28T23:35:59.084431+0000","last_fullsized":"2024-03-28T23:35:59.084431+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:46:16.496511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.085195+0000","last_change":"2024-03-28T23:35:59.084222+0000","last_active":"2024-03-28T23:35:59.085195+0000","last_peered":"2024-03-28T23:35:59.085195+0000","last_clean":"2024-03-28T23:35:59.085195+0000","last_became_active":"2024-03-28T23:35:59.083603+0000","last_became_peered":"2024-03-28T23:35:59.083603+0000","last_unstale":"2024-03-28T23:35:59.085195+0000","last_undegraded":"2024-03-28T23:35:59.085195+0000","last_fullsized":"2024-03-28T23:35:59.085195+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:51.371239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.078962+0000","last_change":"2024-03-28T23:35:59.074398+0000","last_active":"2024-03-28T23:35:59.078962+0000","last_peered":"2024-03-28T23:35:59.078962+0000","last_clean":"2024-03-28T23:35:59.078962+0000","last_became_active":"2024-03-28T23:35:59.074143+0000","last_became_peered":"2024-03-28T23:35:59.074143+0000","last_unstale":"2024-03-28T23:35:59.078962+0000","last_undegraded":"2024-03-28T23:35:59.078962+0000","last_fullsized":"2024-03-28T23:35:59.078962+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:20:21.877305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.073065+0000","last_change":"2024-03-28T23:35:59.066035+0000","last_active":"2024-03-28T23:35:59.073065+0000","last_peered":"2024-03-28T23:35:59.073065+0000","last_clean":"2024-03-28T23:35:59.073065+0000","last_became_active":"2024-03-28T23:35:59.065595+0000","last_became_peered":"2024-03-28T23:35:59.065595+0000","last_unstale":"2024-03-28T23:35:59.073065+0000","last_undegraded":"2024-03-28T23:35:59.073065+0000","last_fullsized":"2024-03-28T23:35:59.073065+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:13.332551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.083779+0000","last_change":"2024-03-28T23:35:59.076253+0000","last_active":"2024-03-28T23:35:59.083779+0000","last_peered":"2024-03-28T23:35:59.083779+0000","last_clean":"2024-03-28T23:35:59.083779+0000","last_became_active":"2024-03-28T23:35:59.075998+0000","last_became_peered":"2024-03-28T23:35:59.075998+0000","last_unstale":"2024-03-28T23:35:59.083779+0000","last_undegraded":"2024-03-28T23:35:59.083779+0000","last_fullsized":"2024-03-28T23:35:59.083779+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:50:09.386042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.084031+0000","last_change":"2024-03-28T23:35:59.077597+0000","last_active":"2024-03-28T23:35:59.084031+0000","last_peered":"2024-03-28T23:35:59.084031+0000","last_clean":"2024-03-28T23:35:59.084031+0000","last_became_active":"2024-03-28T23:35:59.077343+0000","last_became_peered":"2024-03-28T23:35:59.077343+0000","last_unstale":"2024-03-28T23:35:59.084031+0000","last_undegraded":"2024-03-28T23:35:59.084031+0000","last_fullsized":"2024-03-28T23:35:59.084031+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:53:22.705919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.078370+0000","last_change":"2024-03-28T23:35:59.073690+0000","last_active":"2024-03-28T23:35:59.078370+0000","last_peered":"2024-03-28T23:35:59.078370+0000","last_clean":"2024-03-28T23:35:59.078370+0000","last_became_active":"2024-03-28T23:35:59.073405+0000","last_became_peered":"2024-03-28T23:35:59.073405+0000","last_unstale":"2024-03-28T23:35:59.078370+0000","last_undegraded":"2024-03-28T23:35:59.078370+0000","last_fullsized":"2024-03-28T23:35:59.078370+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:06:37.212712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.085104+0000","last_change":"2024-03-28T23:35:59.077619+0000","last_active":"2024-03-28T23:35:59.085104+0000","last_peered":"2024-03-28T23:35:59.085104+0000","last_clean":"2024-03-28T23:35:59.085104+0000","last_became_active":"2024-03-28T23:35:59.077386+0000","last_became_peered":"2024-03-28T23:35:59.077386+0000","last_unstale":"2024-03-28T23:35:59.085104+0000","last_undegraded":"2024-03-28T23:35:59.085104+0000","last_fullsized":"2024-03-28T23:35:59.085104+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:26:01.109468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.076035+0000","last_change":"2024-03-28T23:35:59.074672+0000","last_active":"2024-03-28T23:35:59.076035+0000","last_peered":"2024-03-28T23:35:59.076035+0000","last_clean":"2024-03-28T23:35:59.076035+0000","last_became_active":"2024-03-28T23:35:59.074385+0000","last_became_peered":"2024-03-28T23:35:59.074385+0000","last_unstale":"2024-03-28T23:35:59.076035+0000","last_undegraded":"2024-03-28T23:35:59.076035+0000","last_fullsized":"2024-03-28T23:35:59.076035+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:39:54.469103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.075149+0000","last_change":"2024-03-28T23:35:59.072384+0000","last_active":"2024-03-28T23:35:59.075149+0000","last_peered":"2024-03-28T23:35:59.075149+0000","last_clean":"2024-03-28T23:35:59.075149+0000","last_became_active":"2024-03-28T23:35:59.071941+0000","last_became_peered":"2024-03-28T23:35:59.071941+0000","last_unstale":"2024-03-28T23:35:59.075149+0000","last_undegraded":"2024-03-28T23:35:59.075149+0000","last_fullsized":"2024-03-28T23:35:59.075149+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:58:37.461735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.082119+0000","last_change":"2024-03-28T23:35:59.076869+0000","last_active":"2024-03-28T23:35:59.082119+0000","last_peered":"2024-03-28T23:35:59.082119+0000","last_clean":"2024-03-28T23:35:59.082119+0000","last_became_active":"2024-03-28T23:35:59.076407+0000","last_became_peered":"2024-03-28T23:35:59.076407+0000","last_unstale":"2024-03-28T23:35:59.082119+0000","last_undegraded":"2024-03-28T23:35:59.082119+0000","last_fullsized":"2024-03-28T23:35:59.082119+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:06:08.235927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.088062+0000","last_change":"2024-03-28T23:35:59.082855+0000","last_active":"2024-03-28T23:35:59.088062+0000","last_peered":"2024-03-28T23:35:59.088062+0000","last_clean":"2024-03-28T23:35:59.088062+0000","last_became_active":"2024-03-28T23:35:59.082581+0000","last_became_peered":"2024-03-28T23:35:59.082581+0000","last_unstale":"2024-03-28T23:35:59.088062+0000","last_undegraded":"2024-03-28T23:35:59.088062+0000","last_fullsized":"2024-03-28T23:35:59.088062+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:44:23.521230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.081073+0000","last_change":"2024-03-28T23:35:59.080077+0000","last_active":"2024-03-28T23:35:59.081073+0000","last_peered":"2024-03-28T23:35:59.081073+0000","last_clean":"2024-03-28T23:35:59.081073+0000","last_became_active":"2024-03-28T23:35:59.079742+0000","last_became_peered":"2024-03-28T23:35:59.079742+0000","last_unstale":"2024-03-28T23:35:59.081073+0000","last_undegraded":"2024-03-28T23:35:59.081073+0000","last_fullsized":"2024-03-28T23:35:59.081073+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:16:54.129411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.084371+0000","last_change":"2024-03-28T23:35:59.081618+0000","last_active":"2024-03-28T23:35:59.084371+0000","last_peered":"2024-03-28T23:35:59.084371+0000","last_clean":"2024-03-28T23:35:59.084371+0000","last_became_active":"2024-03-28T23:35:59.080909+0000","last_became_peered":"2024-03-28T23:35:59.080909+0000","last_unstale":"2024-03-28T23:35:59.084371+0000","last_undegraded":"2024-03-28T23:35:59.084371+0000","last_fullsized":"2024-03-28T23:35:59.084371+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:46:12.502797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.078351+0000","last_change":"2024-03-28T23:35:59.072123+0000","last_active":"2024-03-28T23:35:59.078351+0000","last_peered":"2024-03-28T23:35:59.078351+0000","last_clean":"2024-03-28T23:35:59.078351+0000","last_became_active":"2024-03-28T23:35:59.071827+0000","last_became_peered":"2024-03-28T23:35:59.071827+0000","last_unstale":"2024-03-28T23:35:59.078351+0000","last_undegraded":"2024-03-28T23:35:59.078351+0000","last_fullsized":"2024-03-28T23:35:59.078351+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:16:00.430973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.072067+0000","last_change":"2024-03-28T23:35:59.070208+0000","last_active":"2024-03-28T23:35:59.072067+0000","last_peered":"2024-03-28T23:35:59.072067+0000","last_clean":"2024-03-28T23:35:59.072067+0000","last_became_active":"2024-03-28T23:35:59.069928+0000","last_became_peered":"2024-03-28T23:35:59.069928+0000","last_unstale":"2024-03-28T23:35:59.072067+0000","last_undegraded":"2024-03-28T23:35:59.072067+0000","last_fullsized":"2024-03-28T23:35:59.072067+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:21:05.984709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.075476+0000","last_change":"2024-03-28T23:35:59.073450+0000","last_active":"2024-03-28T23:35:59.075476+0000","last_peered":"2024-03-28T23:35:59.075476+0000","last_clean":"2024-03-28T23:35:59.075476+0000","last_became_active":"2024-03-28T23:35:59.073178+0000","last_became_peered":"2024-03-28T23:35:59.073178+0000","last_unstale":"2024-03-28T23:35:59.075476+0000","last_undegraded":"2024-03-28T23:35:59.075476+0000","last_fullsized":"2024-03-28T23:35:59.075476+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:18:01.009699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.086118+0000","last_change":"2024-03-28T23:35:59.080957+0000","last_active":"2024-03-28T23:35:59.086118+0000","last_peered":"2024-03-28T23:35:59.086118+0000","last_clean":"2024-03-28T23:35:59.086118+0000","last_became_active":"2024-03-28T23:35:59.080693+0000","last_became_peered":"2024-03-28T23:35:59.080693+0000","last_unstale":"2024-03-28T23:35:59.086118+0000","last_undegraded":"2024-03-28T23:35:59.086118+0000","last_fullsized":"2024-03-28T23:35:59.086118+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:55.567541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.088426+0000","last_change":"2024-03-28T23:35:59.080315+0000","last_active":"2024-03-28T23:35:59.088426+0000","last_peered":"2024-03-28T23:35:59.088426+0000","last_clean":"2024-03-28T23:35:59.088426+0000","last_became_active":"2024-03-28T23:35:59.080039+0000","last_became_peered":"2024-03-28T23:35:59.080039+0000","last_unstale":"2024-03-28T23:35:59.088426+0000","last_undegraded":"2024-03-28T23:35:59.088426+0000","last_fullsized":"2024-03-28T23:35:59.088426+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:36:46.114123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.075611+0000","last_change":"2024-03-28T23:35:59.074798+0000","last_active":"2024-03-28T23:35:59.075611+0000","last_peered":"2024-03-28T23:35:59.075611+0000","last_clean":"2024-03-28T23:35:59.075611+0000","last_became_active":"2024-03-28T23:35:59.074539+0000","last_became_peered":"2024-03-28T23:35:59.074539+0000","last_unstale":"2024-03-28T23:35:59.075611+0000","last_undegraded":"2024-03-28T23:35:59.075611+0000","last_fullsized":"2024-03-28T23:35:59.075611+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:16:00.430973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.087868+0000","last_change":"2024-03-28T23:35:59.079811+0000","last_active":"2024-03-28T23:35:59.087868+0000","last_peered":"2024-03-28T23:35:59.087868+0000","last_clean":"2024-03-28T23:35:59.087868+0000","last_became_active":"2024-03-28T23:35:59.079538+0000","last_became_peered":"2024-03-28T23:35:59.079538+0000","last_unstale":"2024-03-28T23:35:59.087868+0000","last_undegraded":"2024-03-28T23:35:59.087868+0000","last_fullsized":"2024-03-28T23:35:59.087868+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:06:08.235927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.084348+0000","last_change":"2024-03-28T23:35:59.080958+0000","last_active":"2024-03-28T23:35:59.084348+0000","last_peered":"2024-03-28T23:35:59.084348+0000","last_clean":"2024-03-28T23:35:59.084348+0000","last_became_active":"2024-03-28T23:35:59.080586+0000","last_became_peered":"2024-03-28T23:35:59.080586+0000","last_unstale":"2024-03-28T23:35:59.084348+0000","last_undegraded":"2024-03-28T23:35:59.084348+0000","last_fullsized":"2024-03-28T23:35:59.084348+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:27:42.928036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.080019+0000","last_change":"2024-03-28T23:35:59.076376+0000","last_active":"2024-03-28T23:35:59.080019+0000","last_peered":"2024-03-28T23:35:59.080019+0000","last_clean":"2024-03-28T23:35:59.080019+0000","last_became_active":"2024-03-28T23:35:59.076009+0000","last_became_peered":"2024-03-28T23:35:59.076009+0000","last_unstale":"2024-03-28T23:35:59.080019+0000","last_undegraded":"2024-03-28T23:35:59.080019+0000","last_fullsized":"2024-03-28T23:35:59.080019+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:31:11.799526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.075918+0000","last_change":"2024-03-28T23:35:59.069546+0000","last_active":"2024-03-28T23:35:59.075918+0000","last_peered":"2024-03-28T23:35:59.075918+0000","last_clean":"2024-03-28T23:35:59.075918+0000","last_became_active":"2024-03-28T23:35:59.069283+0000","last_became_peered":"2024-03-28T23:35:59.069283+0000","last_unstale":"2024-03-28T23:35:59.075918+0000","last_undegraded":"2024-03-28T23:35:59.075918+0000","last_fullsized":"2024-03-28T23:35:59.075918+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:57:18.898886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.075788+0000","last_change":"2024-03-28T23:35:59.073880+0000","last_active":"2024-03-28T23:35:59.075788+0000","last_peered":"2024-03-28T23:35:59.075788+0000","last_clean":"2024-03-28T23:35:59.075788+0000","last_became_active":"2024-03-28T23:35:59.073579+0000","last_became_peered":"2024-03-28T23:35:59.073579+0000","last_unstale":"2024-03-28T23:35:59.075788+0000","last_undegraded":"2024-03-28T23:35:59.075788+0000","last_fullsized":"2024-03-28T23:35:59.075788+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:41:12.571144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.088498+0000","last_change":"2024-03-28T23:35:59.082034+0000","last_active":"2024-03-28T23:35:59.088498+0000","last_peered":"2024-03-28T23:35:59.088498+0000","last_clean":"2024-03-28T23:35:59.088498+0000","last_became_active":"2024-03-28T23:35:59.081763+0000","last_became_peered":"2024-03-28T23:35:59.081763+0000","last_unstale":"2024-03-28T23:35:59.088498+0000","last_undegraded":"2024-03-28T23:35:59.088498+0000","last_fullsized":"2024-03-28T23:35:59.088498+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:25.528570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.075280+0000","last_change":"2024-03-28T23:35:59.073561+0000","last_active":"2024-03-28T23:35:59.075280+0000","last_peered":"2024-03-28T23:35:59.075280+0000","last_clean":"2024-03-28T23:35:59.075280+0000","last_became_active":"2024-03-28T23:35:59.073301+0000","last_became_peered":"2024-03-28T23:35:59.073301+0000","last_unstale":"2024-03-28T23:35:59.075280+0000","last_undegraded":"2024-03-28T23:35:59.075280+0000","last_fullsized":"2024-03-28T23:35:59.075280+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:22:08.306031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.076818+0000","last_change":"2024-03-28T23:35:59.074502+0000","last_active":"2024-03-28T23:35:59.076818+0000","last_peered":"2024-03-28T23:35:59.076818+0000","last_clean":"2024-03-28T23:35:59.076818+0000","last_became_active":"2024-03-28T23:35:59.074214+0000","last_became_peered":"2024-03-28T23:35:59.074214+0000","last_unstale":"2024-03-28T23:35:59.076818+0000","last_undegraded":"2024-03-28T23:35:59.076818+0000","last_fullsized":"2024-03-28T23:35:59.076818+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:44:38.063004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.082391+0000","last_change":"2024-03-28T23:35:59.080452+0000","last_active":"2024-03-28T23:35:59.082391+0000","last_peered":"2024-03-28T23:35:59.082391+0000","last_clean":"2024-03-28T23:35:59.082391+0000","last_became_active":"2024-03-28T23:35:59.080141+0000","last_became_peered":"2024-03-28T23:35:59.080141+0000","last_unstale":"2024-03-28T23:35:59.082391+0000","last_undegraded":"2024-03-28T23:35:59.082391+0000","last_fullsized":"2024-03-28T23:35:59.082391+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:58:37.461735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.072611+0000","last_change":"2024-03-28T23:35:59.068642+0000","last_active":"2024-03-28T23:35:59.072611+0000","last_peered":"2024-03-28T23:35:59.072611+0000","last_clean":"2024-03-28T23:35:59.072611+0000","last_became_active":"2024-03-28T23:35:59.068349+0000","last_became_peered":"2024-03-28T23:35:59.068349+0000","last_unstale":"2024-03-28T23:35:59.072611+0000","last_undegraded":"2024-03-28T23:35:59.072611+0000","last_fullsized":"2024-03-28T23:35:59.072611+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:13:23.597258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.084852+0000","last_change":"2024-03-28T23:35:59.083382+0000","last_active":"2024-03-28T23:35:59.084852+0000","last_peered":"2024-03-28T23:35:59.084852+0000","last_clean":"2024-03-28T23:35:59.084852+0000","last_became_active":"2024-03-28T23:35:59.083111+0000","last_became_peered":"2024-03-28T23:35:59.083111+0000","last_unstale":"2024-03-28T23:35:59.084852+0000","last_undegraded":"2024-03-28T23:35:59.084852+0000","last_fullsized":"2024-03-28T23:35:59.084852+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:53:22.705919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"151'4","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.081431+0000","last_change":"2024-03-28T23:35:58.071726+0000","last_active":"2024-03-28T23:35:59.081431+0000","last_peered":"2024-03-28T23:35:59.081431+0000","last_clean":"2024-03-28T23:35:59.081431+0000","last_became_active":"2024-03-28T23:35:58.071384+0000","last_became_peered":"2024-03-28T23:35:58.071384+0000","last_unstale":"2024-03-28T23:35:59.081431+0000","last_undegraded":"2024-03-28T23:35:59.081431+0000","last_fullsized":"2024-03-28T23:35:59.081431+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:31.860347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.088517+0000","last_change":"2024-03-28T23:35:59.082507+0000","last_active":"2024-03-28T23:35:59.088517+0000","last_peered":"2024-03-28T23:35:59.088517+0000","last_clean":"2024-03-28T23:35:59.088517+0000","last_became_active":"2024-03-28T23:35:59.082004+0000","last_became_peered":"2024-03-28T23:35:59.082004+0000","last_unstale":"2024-03-28T23:35:59.088517+0000","last_undegraded":"2024-03-28T23:35:59.088517+0000","last_fullsized":"2024-03-28T23:35:59.088517+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:30:33.779993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"151'1","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.085338+0000","last_change":"2024-03-28T23:35:58.070559+0000","last_active":"2024-03-28T23:35:59.085338+0000","last_peered":"2024-03-28T23:35:59.085338+0000","last_clean":"2024-03-28T23:35:59.085338+0000","last_became_active":"2024-03-28T23:35:58.070259+0000","last_became_peered":"2024-03-28T23:35:58.070259+0000","last_unstale":"2024-03-28T23:35:59.085338+0000","last_undegraded":"2024-03-28T23:35:59.085338+0000","last_fullsized":"2024-03-28T23:35:59.085338+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:23.832628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.088041+0000","last_change":"2024-03-28T23:35:59.082185+0000","last_active":"2024-03-28T23:35:59.088041+0000","last_peered":"2024-03-28T23:35:59.088041+0000","last_clean":"2024-03-28T23:35:59.088041+0000","last_became_active":"2024-03-28T23:35:59.081892+0000","last_became_peered":"2024-03-28T23:35:59.081892+0000","last_unstale":"2024-03-28T23:35:59.088041+0000","last_undegraded":"2024-03-28T23:35:59.088041+0000","last_fullsized":"2024-03-28T23:35:59.088041+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:36:46.114123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"151'2","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.082991+0000","last_change":"2024-03-28T23:35:58.069940+0000","last_active":"2024-03-28T23:35:59.082991+0000","last_peered":"2024-03-28T23:35:59.082991+0000","last_clean":"2024-03-28T23:35:59.082991+0000","last_became_active":"2024-03-28T23:35:58.069533+0000","last_became_peered":"2024-03-28T23:35:58.069533+0000","last_unstale":"2024-03-28T23:35:59.082991+0000","last_undegraded":"2024-03-28T23:35:59.082991+0000","last_fullsized":"2024-03-28T23:35:59.082991+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:14.868910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.083889+0000","last_change":"2024-03-28T23:35:59.080903+0000","last_active":"2024-03-28T23:35:59.083889+0000","last_peered":"2024-03-28T23:35:59.083889+0000","last_clean":"2024-03-28T23:35:59.083889+0000","last_became_active":"2024-03-28T23:35:59.080632+0000","last_became_peered":"2024-03-28T23:35:59.080632+0000","last_unstale":"2024-03-28T23:35:59.083889+0000","last_undegraded":"2024-03-28T23:35:59.083889+0000","last_fullsized":"2024-03-28T23:35:59.083889+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:42:41.721990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"151'2","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.079035+0000","last_change":"2024-03-28T23:35:58.064254+0000","last_active":"2024-03-28T23:35:59.079035+0000","last_peered":"2024-03-28T23:35:59.079035+0000","last_clean":"2024-03-28T23:35:59.079035+0000","last_became_active":"2024-03-28T23:35:58.063972+0000","last_became_peered":"2024-03-28T23:35:58.063972+0000","last_unstale":"2024-03-28T23:35:59.079035+0000","last_undegraded":"2024-03-28T23:35:59.079035+0000","last_fullsized":"2024-03-28T23:35:59.079035+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:36:14.911401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.083989+0000","last_change":"2024-03-28T23:35:59.077647+0000","last_active":"2024-03-28T23:35:59.083989+0000","last_peered":"2024-03-28T23:35:59.083989+0000","last_clean":"2024-03-28T23:35:59.083989+0000","last_became_active":"2024-03-28T23:35:59.077374+0000","last_became_peered":"2024-03-28T23:35:59.077374+0000","last_unstale":"2024-03-28T23:35:59.083989+0000","last_undegraded":"2024-03-28T23:35:59.083989+0000","last_fullsized":"2024-03-28T23:35:59.083989+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:20:09.724407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"151'1","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.088173+0000","last_change":"2024-03-28T23:35:58.074449+0000","last_active":"2024-03-28T23:35:59.088173+0000","last_peered":"2024-03-28T23:35:59.088173+0000","last_clean":"2024-03-28T23:35:59.088173+0000","last_became_active":"2024-03-28T23:35:58.074171+0000","last_became_peered":"2024-03-28T23:35:58.074171+0000","last_unstale":"2024-03-28T23:35:59.088173+0000","last_undegraded":"2024-03-28T23:35:59.088173+0000","last_fullsized":"2024-03-28T23:35:59.088173+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:41:31.356630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.087567+0000","last_change":"2024-03-28T23:35:59.086653+0000","last_active":"2024-03-28T23:35:59.087567+0000","last_peered":"2024-03-28T23:35:59.087567+0000","last_clean":"2024-03-28T23:35:59.087567+0000","last_became_active":"2024-03-28T23:35:59.086398+0000","last_became_peered":"2024-03-28T23:35:59.086398+0000","last_unstale":"2024-03-28T23:35:59.087567+0000","last_undegraded":"2024-03-28T23:35:59.087567+0000","last_fullsized":"2024-03-28T23:35:59.087567+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:57:41.142177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"151'1","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.081283+0000","last_change":"2024-03-28T23:35:58.073421+0000","last_active":"2024-03-28T23:35:59.081283+0000","last_peered":"2024-03-28T23:35:59.081283+0000","last_clean":"2024-03-28T23:35:59.081283+0000","last_became_active":"2024-03-28T23:35:58.072004+0000","last_became_peered":"2024-03-28T23:35:58.072004+0000","last_unstale":"2024-03-28T23:35:59.081283+0000","last_undegraded":"2024-03-28T23:35:59.081283+0000","last_fullsized":"2024-03-28T23:35:59.081283+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:10.808877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.072847+0000","last_change":"2024-03-28T23:35:59.063776+0000","last_active":"2024-03-28T23:35:59.072847+0000","last_peered":"2024-03-28T23:35:59.072847+0000","last_clean":"2024-03-28T23:35:59.072847+0000","last_became_active":"2024-03-28T23:35:59.063214+0000","last_became_peered":"2024-03-28T23:35:59.063214+0000","last_unstale":"2024-03-28T23:35:59.072847+0000","last_undegraded":"2024-03-28T23:35:59.072847+0000","last_fullsized":"2024-03-28T23:35:59.072847+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:49:53.041348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.057618+0000","last_change":"2024-03-28T23:35:58.072230+0000","last_active":"2024-03-28T23:35:59.057618+0000","last_peered":"2024-03-28T23:35:59.057618+0000","last_clean":"2024-03-28T23:35:59.057618+0000","last_became_active":"2024-03-28T23:35:58.071967+0000","last_became_peered":"2024-03-28T23:35:58.071967+0000","last_unstale":"2024-03-28T23:35:59.057618+0000","last_undegraded":"2024-03-28T23:35:59.057618+0000","last_fullsized":"2024-03-28T23:35:59.057618+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:07:28.431728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.076748+0000","last_change":"2024-03-28T23:35:59.070728+0000","last_active":"2024-03-28T23:35:59.076748+0000","last_peered":"2024-03-28T23:35:59.076748+0000","last_clean":"2024-03-28T23:35:59.076748+0000","last_became_active":"2024-03-28T23:35:59.070489+0000","last_became_peered":"2024-03-28T23:35:59.070489+0000","last_unstale":"2024-03-28T23:35:59.076748+0000","last_undegraded":"2024-03-28T23:35:59.076748+0000","last_fullsized":"2024-03-28T23:35:59.076748+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:26:59.457089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.061652+0000","last_change":"2024-03-28T23:35:58.068807+0000","last_active":"2024-03-28T23:35:59.061652+0000","last_peered":"2024-03-28T23:35:59.061652+0000","last_clean":"2024-03-28T23:35:59.061652+0000","last_became_active":"2024-03-28T23:35:58.068530+0000","last_became_peered":"2024-03-28T23:35:58.068530+0000","last_unstale":"2024-03-28T23:35:59.061652+0000","last_undegraded":"2024-03-28T23:35:59.061652+0000","last_fullsized":"2024-03-28T23:35:59.061652+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:12.327316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.083907+0000","last_change":"2024-03-28T23:35:59.082657+0000","last_active":"2024-03-28T23:35:59.083907+0000","last_peered":"2024-03-28T23:35:59.083907+0000","last_clean":"2024-03-28T23:35:59.083907+0000","last_became_active":"2024-03-28T23:35:59.082390+0000","last_became_peered":"2024-03-28T23:35:59.082390+0000","last_unstale":"2024-03-28T23:35:59.083907+0000","last_undegraded":"2024-03-28T23:35:59.083907+0000","last_fullsized":"2024-03-28T23:35:59.083907+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:50:09.386042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.065738+0000","last_change":"2024-03-28T23:35:58.075215+0000","last_active":"2024-03-28T23:35:59.065738+0000","last_peered":"2024-03-28T23:35:59.065738+0000","last_clean":"2024-03-28T23:35:59.065738+0000","last_became_active":"2024-03-28T23:35:58.074891+0000","last_became_peered":"2024-03-28T23:35:58.074891+0000","last_unstale":"2024-03-28T23:35:59.065738+0000","last_undegraded":"2024-03-28T23:35:59.065738+0000","last_fullsized":"2024-03-28T23:35:59.065738+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:16.223515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.088005+0000","last_change":"2024-03-28T23:35:59.079653+0000","last_active":"2024-03-28T23:35:59.088005+0000","last_peered":"2024-03-28T23:35:59.088005+0000","last_clean":"2024-03-28T23:35:59.088005+0000","last_became_active":"2024-03-28T23:35:59.079390+0000","last_became_peered":"2024-03-28T23:35:59.079390+0000","last_unstale":"2024-03-28T23:35:59.088005+0000","last_undegraded":"2024-03-28T23:35:59.088005+0000","last_fullsized":"2024-03-28T23:35:59.088005+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:08:30.907410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"151'3","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.076316+0000","last_change":"2024-03-28T23:35:58.059191+0000","last_active":"2024-03-28T23:35:59.076316+0000","last_peered":"2024-03-28T23:35:59.076316+0000","last_clean":"2024-03-28T23:35:59.076316+0000","last_became_active":"2024-03-28T23:35:58.058860+0000","last_became_peered":"2024-03-28T23:35:58.058860+0000","last_unstale":"2024-03-28T23:35:59.076316+0000","last_undegraded":"2024-03-28T23:35:59.076316+0000","last_fullsized":"2024-03-28T23:35:59.076316+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:22.564724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.080775+0000","last_change":"2024-03-28T23:35:59.074941+0000","last_active":"2024-03-28T23:35:59.080775+0000","last_peered":"2024-03-28T23:35:59.080775+0000","last_clean":"2024-03-28T23:35:59.080775+0000","last_became_active":"2024-03-28T23:35:59.074636+0000","last_became_peered":"2024-03-28T23:35:59.074636+0000","last_unstale":"2024-03-28T23:35:59.080775+0000","last_undegraded":"2024-03-28T23:35:59.080775+0000","last_fullsized":"2024-03-28T23:35:59.080775+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:20:51.733648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"151'6","reported_seq":18,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.089573+0000","last_change":"2024-03-28T23:35:58.075069+0000","last_active":"2024-03-28T23:35:59.089573+0000","last_peered":"2024-03-28T23:35:59.089573+0000","last_clean":"2024-03-28T23:35:59.089573+0000","last_became_active":"2024-03-28T23:35:58.074795+0000","last_became_peered":"2024-03-28T23:35:58.074795+0000","last_unstale":"2024-03-28T23:35:59.089573+0000","last_undegraded":"2024-03-28T23:35:59.089573+0000","last_fullsized":"2024-03-28T23:35:59.089573+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:49:07.831860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.080125+0000","last_change":"2024-03-28T23:35:59.076575+0000","last_active":"2024-03-28T23:35:59.080125+0000","last_peered":"2024-03-28T23:35:59.080125+0000","last_clean":"2024-03-28T23:35:59.080125+0000","last_became_active":"2024-03-28T23:35:59.076319+0000","last_became_peered":"2024-03-28T23:35:59.076319+0000","last_unstale":"2024-03-28T23:35:59.080125+0000","last_undegraded":"2024-03-28T23:35:59.080125+0000","last_fullsized":"2024-03-28T23:35:59.080125+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:53.207692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"151'2","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.086105+0000","last_change":"2024-03-28T23:35:58.073277+0000","last_active":"2024-03-28T23:35:59.086105+0000","last_peered":"2024-03-28T23:35:59.086105+0000","last_clean":"2024-03-28T23:35:59.086105+0000","last_became_active":"2024-03-28T23:35:58.071396+0000","last_became_peered":"2024-03-28T23:35:58.071396+0000","last_unstale":"2024-03-28T23:35:59.086105+0000","last_undegraded":"2024-03-28T23:35:59.086105+0000","last_fullsized":"2024-03-28T23:35:59.086105+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:36:25.234053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.078275+0000","last_change":"2024-03-28T23:35:59.072656+0000","last_active":"2024-03-28T23:35:59.078275+0000","last_peered":"2024-03-28T23:35:59.078275+0000","last_clean":"2024-03-28T23:35:59.078275+0000","last_became_active":"2024-03-28T23:35:59.072400+0000","last_became_peered":"2024-03-28T23:35:59.072400+0000","last_unstale":"2024-03-28T23:35:59.078275+0000","last_undegraded":"2024-03-28T23:35:59.078275+0000","last_fullsized":"2024-03-28T23:35:59.078275+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:11.349017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.069463+0000","last_change":"2024-03-28T23:35:58.071626+0000","last_active":"2024-03-28T23:35:59.069463+0000","last_peered":"2024-03-28T23:35:59.069463+0000","last_clean":"2024-03-28T23:35:59.069463+0000","last_became_active":"2024-03-28T23:35:58.071339+0000","last_became_peered":"2024-03-28T23:35:58.071339+0000","last_unstale":"2024-03-28T23:35:59.069463+0000","last_undegraded":"2024-03-28T23:35:59.069463+0000","last_fullsized":"2024-03-28T23:35:59.069463+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:10.808877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.076765+0000","last_change":"2024-03-28T23:35:59.076190+0000","last_active":"2024-03-28T23:35:59.076765+0000","last_peered":"2024-03-28T23:35:59.076765+0000","last_clean":"2024-03-28T23:35:59.076765+0000","last_became_active":"2024-03-28T23:35:59.075881+0000","last_became_peered":"2024-03-28T23:35:59.075881+0000","last_unstale":"2024-03-28T23:35:59.076765+0000","last_undegraded":"2024-03-28T23:35:59.076765+0000","last_fullsized":"2024-03-28T23:35:59.076765+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:41:32.361865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.068514+0000","last_change":"2024-03-28T23:35:58.062542+0000","last_active":"2024-03-28T23:35:59.068514+0000","last_peered":"2024-03-28T23:35:59.068514+0000","last_clean":"2024-03-28T23:35:59.068514+0000","last_became_active":"2024-03-28T23:35:58.062074+0000","last_became_peered":"2024-03-28T23:35:58.062074+0000","last_unstale":"2024-03-28T23:35:59.068514+0000","last_undegraded":"2024-03-28T23:35:59.068514+0000","last_fullsized":"2024-03-28T23:35:59.068514+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:23:41.041246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.074170+0000","last_change":"2024-03-28T23:35:59.069831+0000","last_active":"2024-03-28T23:35:59.074170+0000","last_peered":"2024-03-28T23:35:59.074170+0000","last_clean":"2024-03-28T23:35:59.074170+0000","last_became_active":"2024-03-28T23:35:59.069272+0000","last_became_peered":"2024-03-28T23:35:59.069272+0000","last_unstale":"2024-03-28T23:35:59.074170+0000","last_undegraded":"2024-03-28T23:35:59.074170+0000","last_fullsized":"2024-03-28T23:35:59.074170+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:46:44.758689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"151'8","reported_seq":20,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.087517+0000","last_change":"2024-03-28T23:35:58.071172+0000","last_active":"2024-03-28T23:35:59.087517+0000","last_peered":"2024-03-28T23:35:59.087517+0000","last_clean":"2024-03-28T23:35:59.087517+0000","last_became_active":"2024-03-28T23:35:58.070901+0000","last_became_peered":"2024-03-28T23:35:58.070901+0000","last_unstale":"2024-03-28T23:35:59.087517+0000","last_undegraded":"2024-03-28T23:35:59.087517+0000","last_fullsized":"2024-03-28T23:35:59.087517+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:16:53.124176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.072109+0000","last_change":"2024-03-28T23:35:59.067559+0000","last_active":"2024-03-28T23:35:59.072109+0000","last_peered":"2024-03-28T23:35:59.072109+0000","last_clean":"2024-03-28T23:35:59.072109+0000","last_became_active":"2024-03-28T23:35:59.067265+0000","last_became_peered":"2024-03-28T23:35:59.067265+0000","last_unstale":"2024-03-28T23:35:59.072109+0000","last_undegraded":"2024-03-28T23:35:59.072109+0000","last_fullsized":"2024-03-28T23:35:59.072109+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:41:12.571144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"151'1","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.085113+0000","last_change":"2024-03-28T23:35:58.064795+0000","last_active":"2024-03-28T23:35:59.085113+0000","last_peered":"2024-03-28T23:35:59.085113+0000","last_clean":"2024-03-28T23:35:59.085113+0000","last_became_active":"2024-03-28T23:35:58.064506+0000","last_became_peered":"2024-03-28T23:35:58.064506+0000","last_unstale":"2024-03-28T23:35:59.085113+0000","last_undegraded":"2024-03-28T23:35:59.085113+0000","last_fullsized":"2024-03-28T23:35:59.085113+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:36:25.234053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.077291+0000","last_change":"2024-03-28T23:35:59.075241+0000","last_active":"2024-03-28T23:35:59.077291+0000","last_peered":"2024-03-28T23:35:59.077291+0000","last_clean":"2024-03-28T23:35:59.077291+0000","last_became_active":"2024-03-28T23:35:59.074170+0000","last_became_peered":"2024-03-28T23:35:59.074170+0000","last_unstale":"2024-03-28T23:35:59.077291+0000","last_undegraded":"2024-03-28T23:35:59.077291+0000","last_fullsized":"2024-03-28T23:35:59.077291+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:17.010487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"151'2","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.087771+0000","last_change":"2024-03-28T23:35:58.070169+0000","last_active":"2024-03-28T23:35:59.087771+0000","last_peered":"2024-03-28T23:35:59.087771+0000","last_clean":"2024-03-28T23:35:59.087771+0000","last_became_active":"2024-03-28T23:35:58.069913+0000","last_became_peered":"2024-03-28T23:35:58.069913+0000","last_unstale":"2024-03-28T23:35:59.087771+0000","last_undegraded":"2024-03-28T23:35:59.087771+0000","last_fullsized":"2024-03-28T23:35:59.087771+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:22:07.300796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.083433+0000","last_change":"2024-03-28T23:35:59.081763+0000","last_active":"2024-03-28T23:35:59.083433+0000","last_peered":"2024-03-28T23:35:59.083433+0000","last_clean":"2024-03-28T23:35:59.083433+0000","last_became_active":"2024-03-28T23:35:59.081465+0000","last_became_peered":"2024-03-28T23:35:59.081465+0000","last_unstale":"2024-03-28T23:35:59.083433+0000","last_undegraded":"2024-03-28T23:35:59.083433+0000","last_fullsized":"2024-03-28T23:35:59.083433+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:22:08.306031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"151'1","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.085508+0000","last_change":"2024-03-28T23:35:58.070308+0000","last_active":"2024-03-28T23:35:59.085508+0000","last_peered":"2024-03-28T23:35:59.085508+0000","last_clean":"2024-03-28T23:35:59.085508+0000","last_became_active":"2024-03-28T23:35:58.070024+0000","last_became_peered":"2024-03-28T23:35:58.070024+0000","last_unstale":"2024-03-28T23:35:59.085508+0000","last_undegraded":"2024-03-28T23:35:59.085508+0000","last_fullsized":"2024-03-28T23:35:59.085508+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:49:07.831860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.076088+0000","last_change":"2024-03-28T23:35:59.075326+0000","last_active":"2024-03-28T23:35:59.076088+0000","last_peered":"2024-03-28T23:35:59.076088+0000","last_clean":"2024-03-28T23:35:59.076088+0000","last_became_active":"2024-03-28T23:35:59.075063+0000","last_became_peered":"2024-03-28T23:35:59.075063+0000","last_unstale":"2024-03-28T23:35:59.076088+0000","last_undegraded":"2024-03-28T23:35:59.076088+0000","last_fullsized":"2024-03-28T23:35:59.076088+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:24:07.077373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.063380+0000","last_change":"2024-03-28T23:35:58.073249+0000","last_active":"2024-03-28T23:35:59.063380+0000","last_peered":"2024-03-28T23:35:59.063380+0000","last_clean":"2024-03-28T23:35:59.063380+0000","last_became_active":"2024-03-28T23:35:58.072992+0000","last_became_peered":"2024-03-28T23:35:58.072992+0000","last_unstale":"2024-03-28T23:35:59.063380+0000","last_undegraded":"2024-03-28T23:35:59.063380+0000","last_fullsized":"2024-03-28T23:35:59.063380+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:58:36.456500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.088483+0000","last_change":"2024-03-28T23:35:59.082001+0000","last_active":"2024-03-28T23:35:59.088483+0000","last_peered":"2024-03-28T23:35:59.088483+0000","last_clean":"2024-03-28T23:35:59.088483+0000","last_became_active":"2024-03-28T23:35:59.081706+0000","last_became_peered":"2024-03-28T23:35:59.081706+0000","last_unstale":"2024-03-28T23:35:59.088483+0000","last_undegraded":"2024-03-28T23:35:59.088483+0000","last_fullsized":"2024-03-28T23:35:59.088483+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:49:26.989533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"151'2","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.087777+0000","last_change":"2024-03-28T23:35:58.072448+0000","last_active":"2024-03-28T23:35:59.087777+0000","last_peered":"2024-03-28T23:35:59.087777+0000","last_clean":"2024-03-28T23:35:59.087777+0000","last_became_active":"2024-03-28T23:35:58.071494+0000","last_became_peered":"2024-03-28T23:35:58.071494+0000","last_unstale":"2024-03-28T23:35:59.087777+0000","last_undegraded":"2024-03-28T23:35:59.087777+0000","last_fullsized":"2024-03-28T23:35:59.087777+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:54:03.045194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.087880+0000","last_change":"2024-03-28T23:35:59.080948+0000","last_active":"2024-03-28T23:35:59.087880+0000","last_peered":"2024-03-28T23:35:59.087880+0000","last_clean":"2024-03-28T23:35:59.087880+0000","last_became_active":"2024-03-28T23:35:59.080672+0000","last_became_peered":"2024-03-28T23:35:59.080672+0000","last_unstale":"2024-03-28T23:35:59.087880+0000","last_undegraded":"2024-03-28T23:35:59.087880+0000","last_fullsized":"2024-03-28T23:35:59.087880+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:08:30.907410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.065989+0000","last_change":"2024-03-28T23:35:58.071966+0000","last_active":"2024-03-28T23:35:59.065989+0000","last_peered":"2024-03-28T23:35:59.065989+0000","last_clean":"2024-03-28T23:35:59.065989+0000","last_became_active":"2024-03-28T23:35:58.071656+0000","last_became_peered":"2024-03-28T23:35:58.071656+0000","last_unstale":"2024-03-28T23:35:59.065989+0000","last_undegraded":"2024-03-28T23:35:59.065989+0000","last_fullsized":"2024-03-28T23:35:59.065989+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:53:34.853174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.076260+0000","last_change":"2024-03-28T23:35:59.074451+0000","last_active":"2024-03-28T23:35:59.076260+0000","last_peered":"2024-03-28T23:35:59.076260+0000","last_clean":"2024-03-28T23:35:59.076260+0000","last_became_active":"2024-03-28T23:35:59.074181+0000","last_became_peered":"2024-03-28T23:35:59.074181+0000","last_unstale":"2024-03-28T23:35:59.076260+0000","last_undegraded":"2024-03-28T23:35:59.076260+0000","last_fullsized":"2024-03-28T23:35:59.076260+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:31:11.799526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.058590+0000","last_change":"2024-03-28T23:35:58.067234+0000","last_active":"2024-03-28T23:35:59.058590+0000","last_peered":"2024-03-28T23:35:59.058590+0000","last_clean":"2024-03-28T23:35:59.058590+0000","last_became_active":"2024-03-28T23:35:58.066942+0000","last_became_peered":"2024-03-28T23:35:58.066942+0000","last_unstale":"2024-03-28T23:35:59.058590+0000","last_undegraded":"2024-03-28T23:35:59.058590+0000","last_fullsized":"2024-03-28T23:35:59.058590+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:37:03.241869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.085605+0000","last_change":"2024-03-28T23:35:59.084751+0000","last_active":"2024-03-28T23:35:59.085605+0000","last_peered":"2024-03-28T23:35:59.085605+0000","last_clean":"2024-03-28T23:35:59.085605+0000","last_became_active":"2024-03-28T23:35:59.084472+0000","last_became_peered":"2024-03-28T23:35:59.084472+0000","last_unstale":"2024-03-28T23:35:59.085605+0000","last_undegraded":"2024-03-28T23:35:59.085605+0000","last_fullsized":"2024-03-28T23:35:59.085605+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:20:09.724407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.062392+0000","last_change":"2024-03-28T23:35:58.067542+0000","last_active":"2024-03-28T23:35:59.062392+0000","last_peered":"2024-03-28T23:35:59.062392+0000","last_clean":"2024-03-28T23:35:59.062392+0000","last_became_active":"2024-03-28T23:35:58.067013+0000","last_became_peered":"2024-03-28T23:35:58.067013+0000","last_unstale":"2024-03-28T23:35:59.062392+0000","last_undegraded":"2024-03-28T23:35:59.062392+0000","last_fullsized":"2024-03-28T23:35:59.062392+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:41:31.356630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.078682+0000","last_change":"2024-03-28T23:35:59.077874+0000","last_active":"2024-03-28T23:35:59.078682+0000","last_peered":"2024-03-28T23:35:59.078682+0000","last_clean":"2024-03-28T23:35:59.078682+0000","last_became_active":"2024-03-28T23:35:59.077613+0000","last_became_peered":"2024-03-28T23:35:59.077613+0000","last_unstale":"2024-03-28T23:35:59.078682+0000","last_undegraded":"2024-03-28T23:35:59.078682+0000","last_fullsized":"2024-03-28T23:35:59.078682+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:13.332551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"151'2","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.081724+0000","last_change":"2024-03-28T23:35:58.066436+0000","last_active":"2024-03-28T23:35:59.081724+0000","last_peered":"2024-03-28T23:35:59.081724+0000","last_clean":"2024-03-28T23:35:59.081724+0000","last_became_active":"2024-03-28T23:35:58.063834+0000","last_became_peered":"2024-03-28T23:35:58.063834+0000","last_unstale":"2024-03-28T23:35:59.081724+0000","last_undegraded":"2024-03-28T23:35:59.081724+0000","last_fullsized":"2024-03-28T23:35:59.081724+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:37:09.547486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.078369+0000","last_change":"2024-03-28T23:35:59.074166+0000","last_active":"2024-03-28T23:35:59.078369+0000","last_peered":"2024-03-28T23:35:59.078369+0000","last_clean":"2024-03-28T23:35:59.078369+0000","last_became_active":"2024-03-28T23:35:59.073916+0000","last_became_peered":"2024-03-28T23:35:59.073916+0000","last_unstale":"2024-03-28T23:35:59.078369+0000","last_undegraded":"2024-03-28T23:35:59.078369+0000","last_fullsized":"2024-03-28T23:35:59.078369+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:42:41.721990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.067718+0000","last_change":"2024-03-28T23:35:58.078010+0000","last_active":"2024-03-28T23:35:59.067718+0000","last_peered":"2024-03-28T23:35:59.067718+0000","last_clean":"2024-03-28T23:35:59.067718+0000","last_became_active":"2024-03-28T23:35:58.077736+0000","last_became_peered":"2024-03-28T23:35:58.077736+0000","last_unstale":"2024-03-28T23:35:59.067718+0000","last_undegraded":"2024-03-28T23:35:59.067718+0000","last_fullsized":"2024-03-28T23:35:59.067718+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:54:03.045194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.074265+0000","last_change":"2024-03-28T23:35:59.062460+0000","last_active":"2024-03-28T23:35:59.074265+0000","last_peered":"2024-03-28T23:35:59.074265+0000","last_clean":"2024-03-28T23:35:59.074265+0000","last_became_active":"2024-03-28T23:35:59.062185+0000","last_became_peered":"2024-03-28T23:35:59.062185+0000","last_unstale":"2024-03-28T23:35:59.074265+0000","last_undegraded":"2024-03-28T23:35:59.074265+0000","last_fullsized":"2024-03-28T23:35:59.074265+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:45:16.369047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.066288+0000","last_change":"2024-03-28T23:35:58.072356+0000","last_active":"2024-03-28T23:35:59.066288+0000","last_peered":"2024-03-28T23:35:59.066288+0000","last_clean":"2024-03-28T23:35:59.066288+0000","last_became_active":"2024-03-28T23:35:58.072111+0000","last_became_peered":"2024-03-28T23:35:58.072111+0000","last_unstale":"2024-03-28T23:35:59.066288+0000","last_undegraded":"2024-03-28T23:35:59.066288+0000","last_fullsized":"2024-03-28T23:35:59.066288+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:23.832628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.086800+0000","last_change":"2024-03-28T23:35:59.079950+0000","last_active":"2024-03-28T23:35:59.086800+0000","last_peered":"2024-03-28T23:35:59.086800+0000","last_clean":"2024-03-28T23:35:59.086800+0000","last_became_active":"2024-03-28T23:35:59.079675+0000","last_became_peered":"2024-03-28T23:35:59.079675+0000","last_unstale":"2024-03-28T23:35:59.086800+0000","last_undegraded":"2024-03-28T23:35:59.086800+0000","last_fullsized":"2024-03-28T23:35:59.086800+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:14:30.938161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.058232+0000","last_change":"2024-03-28T23:35:58.068737+0000","last_active":"2024-03-28T23:35:59.058232+0000","last_peered":"2024-03-28T23:35:59.058232+0000","last_clean":"2024-03-28T23:35:59.058232+0000","last_became_active":"2024-03-28T23:35:58.068481+0000","last_became_peered":"2024-03-28T23:35:58.068481+0000","last_unstale":"2024-03-28T23:35:59.058232+0000","last_undegraded":"2024-03-28T23:35:59.058232+0000","last_fullsized":"2024-03-28T23:35:59.058232+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:05:05.111519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.083843+0000","last_change":"2024-03-28T23:35:59.077779+0000","last_active":"2024-03-28T23:35:59.083843+0000","last_peered":"2024-03-28T23:35:59.083843+0000","last_clean":"2024-03-28T23:35:59.083843+0000","last_became_active":"2024-03-28T23:35:59.077529+0000","last_became_peered":"2024-03-28T23:35:59.077529+0000","last_unstale":"2024-03-28T23:35:59.083843+0000","last_undegraded":"2024-03-28T23:35:59.083843+0000","last_fullsized":"2024-03-28T23:35:59.083843+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:03:28.508127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.057364+0000","last_change":"2024-03-28T23:35:58.072655+0000","last_active":"2024-03-28T23:35:59.057364+0000","last_peered":"2024-03-28T23:35:59.057364+0000","last_clean":"2024-03-28T23:35:59.057364+0000","last_became_active":"2024-03-28T23:35:58.072379+0000","last_became_peered":"2024-03-28T23:35:58.072379+0000","last_unstale":"2024-03-28T23:35:59.057364+0000","last_undegraded":"2024-03-28T23:35:59.057364+0000","last_fullsized":"2024-03-28T23:35:59.057364+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:57:05.274295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.082831+0000","last_change":"2024-03-28T23:35:59.080115+0000","last_active":"2024-03-28T23:35:59.082831+0000","last_peered":"2024-03-28T23:35:59.082831+0000","last_clean":"2024-03-28T23:35:59.082831+0000","last_became_active":"2024-03-28T23:35:59.079809+0000","last_became_peered":"2024-03-28T23:35:59.079809+0000","last_unstale":"2024-03-28T23:35:59.082831+0000","last_undegraded":"2024-03-28T23:35:59.082831+0000","last_fullsized":"2024-03-28T23:35:59.082831+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:06:37.212712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.069602+0000","last_change":"2024-03-28T23:35:58.071453+0000","last_active":"2024-03-28T23:35:59.069602+0000","last_peered":"2024-03-28T23:35:59.069602+0000","last_clean":"2024-03-28T23:35:59.069602+0000","last_became_active":"2024-03-28T23:35:58.071072+0000","last_became_peered":"2024-03-28T23:35:58.071072+0000","last_unstale":"2024-03-28T23:35:59.069602+0000","last_undegraded":"2024-03-28T23:35:59.069602+0000","last_fullsized":"2024-03-28T23:35:59.069602+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:20:20.872070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.088472+0000","last_change":"2024-03-28T23:35:59.081608+0000","last_active":"2024-03-28T23:35:59.088472+0000","last_peered":"2024-03-28T23:35:59.088472+0000","last_clean":"2024-03-28T23:35:59.088472+0000","last_became_active":"2024-03-28T23:35:59.080839+0000","last_became_peered":"2024-03-28T23:35:59.080839+0000","last_unstale":"2024-03-28T23:35:59.088472+0000","last_undegraded":"2024-03-28T23:35:59.088472+0000","last_fullsized":"2024-03-28T23:35:59.088472+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:44:23.521230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.068681+0000","last_change":"2024-03-28T23:35:58.067740+0000","last_active":"2024-03-28T23:35:59.068681+0000","last_peered":"2024-03-28T23:35:59.068681+0000","last_clean":"2024-03-28T23:35:59.068681+0000","last_became_active":"2024-03-28T23:35:58.067418+0000","last_became_peered":"2024-03-28T23:35:58.067418+0000","last_unstale":"2024-03-28T23:35:59.068681+0000","last_undegraded":"2024-03-28T23:35:59.068681+0000","last_fullsized":"2024-03-28T23:35:59.068681+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:00:58.303444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.084161+0000","last_change":"2024-03-28T23:35:59.079990+0000","last_active":"2024-03-28T23:35:59.084161+0000","last_peered":"2024-03-28T23:35:59.084161+0000","last_clean":"2024-03-28T23:35:59.084161+0000","last_became_active":"2024-03-28T23:35:59.079717+0000","last_became_peered":"2024-03-28T23:35:59.079717+0000","last_unstale":"2024-03-28T23:35:59.084161+0000","last_undegraded":"2024-03-28T23:35:59.084161+0000","last_fullsized":"2024-03-28T23:35:59.084161+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:10:20.786364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.070181+0000","last_change":"2024-03-28T23:35:58.071103+0000","last_active":"2024-03-28T23:35:59.070181+0000","last_peered":"2024-03-28T23:35:59.070181+0000","last_clean":"2024-03-28T23:35:59.070181+0000","last_became_active":"2024-03-28T23:35:58.070587+0000","last_became_peered":"2024-03-28T23:35:58.070587+0000","last_unstale":"2024-03-28T23:35:59.070181+0000","last_undegraded":"2024-03-28T23:35:59.070181+0000","last_fullsized":"2024-03-28T23:35:59.070181+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:16.223515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.079241+0000","last_change":"2024-03-28T23:35:59.075847+0000","last_active":"2024-03-28T23:35:59.079241+0000","last_peered":"2024-03-28T23:35:59.079241+0000","last_clean":"2024-03-28T23:35:59.079241+0000","last_became_active":"2024-03-28T23:35:59.072889+0000","last_became_peered":"2024-03-28T23:35:59.072889+0000","last_unstale":"2024-03-28T23:35:59.079241+0000","last_undegraded":"2024-03-28T23:35:59.079241+0000","last_fullsized":"2024-03-28T23:35:59.079241+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:11.349017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.067569+0000","last_change":"2024-03-28T23:35:58.078046+0000","last_active":"2024-03-28T23:35:59.067569+0000","last_peered":"2024-03-28T23:35:59.067569+0000","last_clean":"2024-03-28T23:35:59.067569+0000","last_became_active":"2024-03-28T23:35:58.077780+0000","last_became_peered":"2024-03-28T23:35:58.077780+0000","last_unstale":"2024-03-28T23:35:59.067569+0000","last_undegraded":"2024-03-28T23:35:59.067569+0000","last_fullsized":"2024-03-28T23:35:59.067569+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:37:09.547486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":415,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.062474+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:35:59.062474+0000","last_peered":"2024-03-28T23:35:59.062474+0000","last_clean":"2024-03-28T23:35:59.062474+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:35:59.062474+0000","last_undegraded":"2024-03-28T23:35:59.062474+0000","last_fullsized":"2024-03-28T23:35:59.062474+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1966080,"data_stored":1508793,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1888256,"data_stored":911258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574927,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1888256,"data_stored":911258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574926,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1888256,"data_stored":911258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574926,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33060,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338780,"statfs":{"total":96636764160,"available":96602910720,"internally_reserved":0,"allocated":1916928,"data_stored":919701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574926,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1888256,"data_stored":911258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33056,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338784,"statfs":{"total":96636764160,"available":96602914816,"internally_reserved":0,"allocated":1912832,"data_stored":920037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574926,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33096,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338744,"statfs":{"total":96636764160,"available":96602873856,"internally_reserved":0,"allocated":1953792,"data_stored":1501626,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:36:02.033 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:36:02.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.172+0000 7f62250c9640 1 Processor -- start 2024-03-28T23:36:02.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.172+0000 7f62250c9640 1 -- start start 2024-03-28T23:36:02.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.172+0000 7f62250c9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f622014eca0 0x7f62200c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:02.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.172+0000 7f62250c9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6220151850 con 0x7f62200c1dc0 2024-03-28T23:36:02.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.172+0000 7f62250c9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6220151b30 con 0x7f62200a98c0 2024-03-28T23:36:02.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.172+0000 7f62250c9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f6220152460 con 0x7f622014eca0 2024-03-28T23:36:02.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.173+0000 7f621e59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f62200c1dc0 0x7f622006ca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50084/0 (socket says 172.21.15.161:50084) 2024-03-28T23:36:02.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.173+0000 7f621e59c640 1 -- 172.21.15.161:0/2309024216 learned_addr learned my addr 172.21.15.161:0/2309024216 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:02.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.173+0000 7f621ed9d640 1 --2- 172.21.15.161:0/2309024216 >> v2:172.21.15.170:3300/0 conn(0x7f622014eca0 0x7f62200c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:02.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.173+0000 7f621dd9b640 1 -- 172.21.15.161:0/2309024216 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3560076124 0 0) 0x7f6220151850 con 0x7f62200c1dc0 2024-03-28T23:36:02.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.173+0000 7f621dd9b640 1 -- 172.21.15.161:0/2309024216 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6208003520 con 0x7f62200c1dc0 2024-03-28T23:36:02.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.173+0000 7f621ed9d640 1 -- 172.21.15.161:0/2309024216 >> v1:172.21.15.170:6789/0 conn(0x7f62200a98c0 legacy=0x7f62200a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:02.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.173+0000 7f621ed9d640 1 -- 172.21.15.161:0/2309024216 >> v1:172.21.15.161:6789/0 conn(0x7f62200c1dc0 legacy=0x7f622006ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:02.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.173+0000 7f621ed9d640 1 -- 172.21.15.161:0/2309024216 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6220151e10 con 0x7f622014eca0 2024-03-28T23:36:02.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.174+0000 7f621ed9d640 1 --2- 172.21.15.161:0/2309024216 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f622014eca0 0x7f62200c1880 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f62140097a0 tx=0x7f621402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ed39e4b932dbbc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:02.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.174+0000 7f621dd9b640 1 -- 172.21.15.161:0/2309024216 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6214035020 con 0x7f622014eca0 2024-03-28T23:36:02.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.174+0000 7f621dd9b640 1 -- 172.21.15.161:0/2309024216 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6214031830 con 0x7f622014eca0 2024-03-28T23:36:02.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.174+0000 7f621dd9b640 1 -- 172.21.15.161:0/2309024216 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6214031c00 con 0x7f622014eca0 2024-03-28T23:36:02.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.174+0000 7f62250c9640 1 -- 172.21.15.161:0/2309024216 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f622014eca0 msgr2=0x7f62200c1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:02.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.174+0000 7f62250c9640 1 --2- 172.21.15.161:0/2309024216 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f622014eca0 0x7f62200c1880 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f62140097a0 tx=0x7f621402b8d0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.174+0000 7f62250c9640 1 -- 172.21.15.161:0/2309024216 shutdown_connections 2024-03-28T23:36:02.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.174+0000 7f62250c9640 1 --2- 172.21.15.161:0/2309024216 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f622014eca0 0x7f62200c1880 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.174+0000 7f62250c9640 1 -- 172.21.15.161:0/2309024216 >> 172.21.15.161:0/2309024216 conn(0x7f62200c6670 msgr2=0x7f62200c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:02.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.174+0000 7f62250c9640 1 -- 172.21.15.161:0/2309024216 shutdown_connections 2024-03-28T23:36:02.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.174+0000 7f62250c9640 1 -- 172.21.15.161:0/2309024216 wait complete. 2024-03-28T23:36:02.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.176+0000 7f62250c9640 1 Processor -- start 2024-03-28T23:36:02.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.176+0000 7f62250c9640 1 -- start start 2024-03-28T23:36:02.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.176+0000 7f62250c9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62200a98c0 0x7f6220072850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:02.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.177+0000 7f62250c9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62200c1dc0 0x7f62200bb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:02.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.177+0000 7f62250c9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f622014eca0 0x7f62200bbb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:02.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.177+0000 7f62250c9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f62201514b0 con 0x7f62200a98c0 2024-03-28T23:36:02.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.177+0000 7f62250c9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f622014f080 con 0x7f62200c1dc0 2024-03-28T23:36:02.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.177+0000 7f62250c9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6220151630 con 0x7f622014eca0 2024-03-28T23:36:02.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.177+0000 7f621ed9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62200a98c0 0x7f6220072850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:02.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.177+0000 7f621ed9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62200a98c0 0x7f6220072850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53724/0 (socket says 172.21.15.161:53724) 2024-03-28T23:36:02.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.177+0000 7f621ed9d640 1 -- 172.21.15.161:0/4180612176 learned_addr learned my addr 172.21.15.161:0/4180612176 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:02.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.177+0000 7f621e59c640 1 --2- 172.21.15.161:0/4180612176 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62200c1dc0 0x7f62200bb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:02.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.177+0000 7f621f59e640 1 --2- 172.21.15.161:0/4180612176 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f622014eca0 0x7f62200bbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:02.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.177+0000 7f621ed9d640 1 -- 172.21.15.161:0/4180612176 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f622014eca0 msgr2=0x7f62200bbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:02.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.177+0000 7f621ed9d640 1 --2- 172.21.15.161:0/4180612176 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f622014eca0 0x7f62200bbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.177+0000 7f621ed9d640 1 -- 172.21.15.161:0/4180612176 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62200c1dc0 msgr2=0x7f62200bb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:02.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.177+0000 7f621ed9d640 1 --2- 172.21.15.161:0/4180612176 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62200c1dc0 0x7f62200bb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.177+0000 7f621ed9d640 1 -- 172.21.15.161:0/4180612176 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6220146c70 con 0x7f62200a98c0 2024-03-28T23:36:02.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.177+0000 7f621ed9d640 1 --2- 172.21.15.161:0/4180612176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62200a98c0 0x7f6220072850 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f6214034040 tx=0x7f62140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:02.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.177+0000 7f62077fe640 1 -- 172.21.15.161:0/4180612176 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6214040070 con 0x7f62200a98c0 2024-03-28T23:36:02.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.177+0000 7f62077fe640 1 -- 172.21.15.161:0/4180612176 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f621403a910 con 0x7f62200a98c0 2024-03-28T23:36:02.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.177+0000 7f62250c9640 1 -- 172.21.15.161:0/4180612176 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6220147210 con 0x7f62200a98c0 2024-03-28T23:36:02.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.177+0000 7f62077fe640 1 -- 172.21.15.161:0/4180612176 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6214035040 con 0x7f62200a98c0 2024-03-28T23:36:02.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.177+0000 7f62250c9640 1 -- 172.21.15.161:0/4180612176 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6220146f30 con 0x7f62200a98c0 2024-03-28T23:36:02.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.178+0000 7f62250c9640 1 -- 172.21.15.161:0/4180612176 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f622004f4b0 con 0x7f62200a98c0 2024-03-28T23:36:02.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.179+0000 7f62077fe640 1 -- 172.21.15.161:0/4180612176 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6214038020 con 0x7f62200a98c0 2024-03-28T23:36:02.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.179+0000 7f62077fe640 1 --2- 172.21.15.161:0/4180612176 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f620c06b4d0 0x7f620c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:02.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.179+0000 7f62077fe640 1 -- 172.21.15.161:0/4180612176 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 6101+0+0 (secure 0 0 0) 0x7f62140abb90 con 0x7f62200a98c0 2024-03-28T23:36:02.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.179+0000 7f621e59c640 1 --2- 172.21.15.161:0/4180612176 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f620c06b4d0 0x7f620c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:02.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.180+0000 7f621e59c640 1 --2- 172.21.15.161:0/4180612176 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f620c06b4d0 0x7f620c06d990 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f62200a8c90 tx=0x7f6210007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:02.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.183+0000 7f62077fe640 1 -- 172.21.15.161:0/4180612176 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6214031800 con 0x7f62200a98c0 2024-03-28T23:36:02.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.352+0000 7f62250c9640 1 -- 172.21.15.161:0/4180612176 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f62200c6af0 con 0x7f620c06b4d0 2024-03-28T23:36:02.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.358+0000 7f62077fe640 1 -- 172.21.15.161:0/4180612176 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252483 (secure 0 0 0) 0x7f62200c6af0 con 0x7f620c06b4d0 2024-03-28T23:36:02.361 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:02.362 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:36:02.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.361+0000 7f62250c9640 1 -- 172.21.15.161:0/4180612176 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f620c06b4d0 msgr2=0x7f620c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:02.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.361+0000 7f62250c9640 1 --2- 172.21.15.161:0/4180612176 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f620c06b4d0 0x7f620c06d990 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f62200a8c90 tx=0x7f6210007d30 comp rx=0 tx=0).stop 2024-03-28T23:36:02.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.361+0000 7f62250c9640 1 -- 172.21.15.161:0/4180612176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62200a98c0 msgr2=0x7f6220072850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:02.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.361+0000 7f62250c9640 1 --2- 172.21.15.161:0/4180612176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62200a98c0 0x7f6220072850 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f6214034040 tx=0x7f62140046c0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.361+0000 7f62250c9640 1 -- 172.21.15.161:0/4180612176 shutdown_connections 2024-03-28T23:36:02.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.361+0000 7f62250c9640 1 --2- 172.21.15.161:0/4180612176 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f620c06b4d0 0x7f620c06d990 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.361+0000 7f62250c9640 1 --2- 172.21.15.161:0/4180612176 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f622014eca0 0x7f62200bbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.361+0000 7f62250c9640 1 --2- 172.21.15.161:0/4180612176 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62200c1dc0 0x7f62200bb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.361+0000 7f62250c9640 1 --2- 172.21.15.161:0/4180612176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62200a98c0 0x7f6220072850 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.361+0000 7f62250c9640 1 -- 172.21.15.161:0/4180612176 >> 172.21.15.161:0/4180612176 conn(0x7f62200c6670 msgr2=0x7f6220140770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:02.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.361+0000 7f62250c9640 1 -- 172.21.15.161:0/4180612176 shutdown_connections 2024-03-28T23:36:02.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.361+0000 7f62250c9640 1 -- 172.21.15.161:0/4180612176 wait complete. 2024-03-28T23:36:02.372 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-03-28T23:36:01.384431+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":45,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264424,"kb_used_data":14944,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754710296,"statfs":{"total":773094113280,"available":772823343104,"internally_reserved":0,"allocated":15302656,"data_stored":8495189,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.337135"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.083764+0000","last_change":"2024-03-28T23:35:59.077622+0000","last_active":"2024-03-28T23:35:59.083764+0000","last_peered":"2024-03-28T23:35:59.083764+0000","last_clean":"2024-03-28T23:35:59.083764+0000","last_became_active":"2024-03-28T23:35:59.077350+0000","last_became_peered":"2024-03-28T23:35:59.077350+0000","last_unstale":"2024-03-28T23:35:59.083764+0000","last_undegraded":"2024-03-28T23:35:59.083764+0000","last_fullsized":"2024-03-28T23:35:59.083764+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:25:34.583929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.075033+0000","last_change":"2024-03-28T23:35:59.069924+0000","last_active":"2024-03-28T23:35:59.075033+0000","last_peered":"2024-03-28T23:35:59.075033+0000","last_clean":"2024-03-28T23:35:59.075033+0000","last_became_active":"2024-03-28T23:35:59.069650+0000","last_became_peered":"2024-03-28T23:35:59.069650+0000","last_unstale":"2024-03-28T23:35:59.075033+0000","last_undegraded":"2024-03-28T23:35:59.075033+0000","last_fullsized":"2024-03-28T23:35:59.075033+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:16:54.129411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.084431+0000","last_change":"2024-03-28T23:35:59.082632+0000","last_active":"2024-03-28T23:35:59.084431+0000","last_peered":"2024-03-28T23:35:59.084431+0000","last_clean":"2024-03-28T23:35:59.084431+0000","last_became_active":"2024-03-28T23:35:59.082360+0000","last_became_peered":"2024-03-28T23:35:59.082360+0000","last_unstale":"2024-03-28T23:35:59.084431+0000","last_undegraded":"2024-03-28T23:35:59.084431+0000","last_fullsized":"2024-03-28T23:35:59.084431+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:46:16.496511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.085195+0000","last_change":"2024-03-28T23:35:59.084222+0000","last_active":"2024-03-28T23:35:59.085195+0000","last_peered":"2024-03-28T23:35:59.085195+0000","last_clean":"2024-03-28T23:35:59.085195+0000","last_became_active":"2024-03-28T23:35:59.083603+0000","last_became_peered":"2024-03-28T23:35:59.083603+0000","last_unstale":"2024-03-28T23:35:59.085195+0000","last_undegraded":"2024-03-28T23:35:59.085195+0000","last_fullsized":"2024-03-28T23:35:59.085195+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:51.371239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.078962+0000","last_change":"2024-03-28T23:35:59.074398+0000","last_active":"2024-03-28T23:35:59.078962+0000","last_peered":"2024-03-28T23:35:59.078962+0000","last_clean":"2024-03-28T23:35:59.078962+0000","last_became_active":"2024-03-28T23:35:59.074143+0000","last_became_peered":"2024-03-28T23:35:59.074143+0000","last_unstale":"2024-03-28T23:35:59.078962+0000","last_undegraded":"2024-03-28T23:35:59.078962+0000","last_fullsized":"2024-03-28T23:35:59.078962+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:20:21.877305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.073065+0000","last_change":"2024-03-28T23:35:59.066035+0000","last_active":"2024-03-28T23:35:59.073065+0000","last_peered":"2024-03-28T23:35:59.073065+0000","last_clean":"2024-03-28T23:35:59.073065+0000","last_became_active":"2024-03-28T23:35:59.065595+0000","last_became_peered":"2024-03-28T23:35:59.065595+0000","last_unstale":"2024-03-28T23:35:59.073065+0000","last_undegraded":"2024-03-28T23:35:59.073065+0000","last_fullsized":"2024-03-28T23:35:59.073065+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:13.332551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.083779+0000","last_change":"2024-03-28T23:35:59.076253+0000","last_active":"2024-03-28T23:35:59.083779+0000","last_peered":"2024-03-28T23:35:59.083779+0000","last_clean":"2024-03-28T23:35:59.083779+0000","last_became_active":"2024-03-28T23:35:59.075998+0000","last_became_peered":"2024-03-28T23:35:59.075998+0000","last_unstale":"2024-03-28T23:35:59.083779+0000","last_undegraded":"2024-03-28T23:35:59.083779+0000","last_fullsized":"2024-03-28T23:35:59.083779+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:50:09.386042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.084031+0000","last_change":"2024-03-28T23:35:59.077597+0000","last_active":"2024-03-28T23:35:59.084031+0000","last_peered":"2024-03-28T23:35:59.084031+0000","last_clean":"2024-03-28T23:35:59.084031+0000","last_became_active":"2024-03-28T23:35:59.077343+0000","last_became_peered":"2024-03-28T23:35:59.077343+0000","last_unstale":"2024-03-28T23:35:59.084031+0000","last_undegraded":"2024-03-28T23:35:59.084031+0000","last_fullsized":"2024-03-28T23:35:59.084031+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:53:22.705919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.078370+0000","last_change":"2024-03-28T23:35:59.073690+0000","last_active":"2024-03-28T23:35:59.078370+0000","last_peered":"2024-03-28T23:35:59.078370+0000","last_clean":"2024-03-28T23:35:59.078370+0000","last_became_active":"2024-03-28T23:35:59.073405+0000","last_became_peered":"2024-03-28T23:35:59.073405+0000","last_unstale":"2024-03-28T23:35:59.078370+0000","last_undegraded":"2024-03-28T23:35:59.078370+0000","last_fullsized":"2024-03-28T23:35:59.078370+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:06:37.212712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.085104+0000","last_change":"2024-03-28T23:35:59.077619+0000","last_active":"2024-03-28T23:35:59.085104+0000","last_peered":"2024-03-28T23:35:59.085104+0000","last_clean":"2024-03-28T23:35:59.085104+0000","last_became_active":"2024-03-28T23:35:59.077386+0000","last_became_peered":"2024-03-28T23:35:59.077386+0000","last_unstale":"2024-03-28T23:35:59.085104+0000","last_undegraded":"2024-03-28T23:35:59.085104+0000","last_fullsized":"2024-03-28T23:35:59.085104+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:26:01.109468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.076035+0000","last_change":"2024-03-28T23:35:59.074672+0000","last_active":"2024-03-28T23:35:59.076035+0000","last_peered":"2024-03-28T23:35:59.076035+0000","last_clean":"2024-03-28T23:35:59.076035+0000","last_became_active":"2024-03-28T23:35:59.074385+0000","last_became_peered":"2024-03-28T23:35:59.074385+0000","last_unstale":"2024-03-28T23:35:59.076035+0000","last_undegraded":"2024-03-28T23:35:59.076035+0000","last_fullsized":"2024-03-28T23:35:59.076035+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:39:54.469103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.075149+0000","last_change":"2024-03-28T23:35:59.072384+0000","last_active":"2024-03-28T23:35:59.075149+0000","last_peered":"2024-03-28T23:35:59.075149+0000","last_clean":"2024-03-28T23:35:59.075149+0000","last_became_active":"2024-03-28T23:35:59.071941+0000","last_became_peered":"2024-03-28T23:35:59.071941+0000","last_unstale":"2024-03-28T23:35:59.075149+0000","last_undegraded":"2024-03-28T23:35:59.075149+0000","last_fullsized":"2024-03-28T23:35:59.075149+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:58:37.461735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.082119+0000","last_change":"2024-03-28T23:35:59.076869+0000","last_active":"2024-03-28T23:35:59.082119+0000","last_peered":"2024-03-28T23:35:59.082119+0000","last_clean":"2024-03-28T23:35:59.082119+0000","last_became_active":"2024-03-28T23:35:59.076407+0000","last_became_peered":"2024-03-28T23:35:59.076407+0000","last_unstale":"2024-03-28T23:35:59.082119+0000","last_undegraded":"2024-03-28T23:35:59.082119+0000","last_fullsized":"2024-03-28T23:35:59.082119+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:06:08.235927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.088062+0000","last_change":"2024-03-28T23:35:59.082855+0000","last_active":"2024-03-28T23:35:59.088062+0000","last_peered":"2024-03-28T23:35:59.088062+0000","last_clean":"2024-03-28T23:35:59.088062+0000","last_became_active":"2024-03-28T23:35:59.082581+0000","last_became_peered":"2024-03-28T23:35:59.082581+0000","last_unstale":"2024-03-28T23:35:59.088062+0000","last_undegraded":"2024-03-28T23:35:59.088062+0000","last_fullsized":"2024-03-28T23:35:59.088062+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:44:23.521230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.081073+0000","last_change":"2024-03-28T23:35:59.080077+0000","last_active":"2024-03-28T23:35:59.081073+0000","last_peered":"2024-03-28T23:35:59.081073+0000","last_clean":"2024-03-28T23:35:59.081073+0000","last_became_active":"2024-03-28T23:35:59.079742+0000","last_became_peered":"2024-03-28T23:35:59.079742+0000","last_unstale":"2024-03-28T23:35:59.081073+0000","last_undegraded":"2024-03-28T23:35:59.081073+0000","last_fullsized":"2024-03-28T23:35:59.081073+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:16:54.129411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.084371+0000","last_change":"2024-03-28T23:35:59.081618+0000","last_active":"2024-03-28T23:35:59.084371+0000","last_peered":"2024-03-28T23:35:59.084371+0000","last_clean":"2024-03-28T23:35:59.084371+0000","last_became_active":"2024-03-28T23:35:59.080909+0000","last_became_peered":"2024-03-28T23:35:59.080909+0000","last_unstale":"2024-03-28T23:35:59.084371+0000","last_undegraded":"2024-03-28T23:35:59.084371+0000","last_fullsized":"2024-03-28T23:35:59.084371+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:46:12.502797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.078351+0000","last_change":"2024-03-28T23:35:59.072123+0000","last_active":"2024-03-28T23:35:59.078351+0000","last_peered":"2024-03-28T23:35:59.078351+0000","last_clean":"2024-03-28T23:35:59.078351+0000","last_became_active":"2024-03-28T23:35:59.071827+0000","last_became_peered":"2024-03-28T23:35:59.071827+0000","last_unstale":"2024-03-28T23:35:59.078351+0000","last_undegraded":"2024-03-28T23:35:59.078351+0000","last_fullsized":"2024-03-28T23:35:59.078351+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:16:00.430973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.072067+0000","last_change":"2024-03-28T23:35:59.070208+0000","last_active":"2024-03-28T23:35:59.072067+0000","last_peered":"2024-03-28T23:35:59.072067+0000","last_clean":"2024-03-28T23:35:59.072067+0000","last_became_active":"2024-03-28T23:35:59.069928+0000","last_became_peered":"2024-03-28T23:35:59.069928+0000","last_unstale":"2024-03-28T23:35:59.072067+0000","last_undegraded":"2024-03-28T23:35:59.072067+0000","last_fullsized":"2024-03-28T23:35:59.072067+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:21:05.984709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.075476+0000","last_change":"2024-03-28T23:35:59.073450+0000","last_active":"2024-03-28T23:35:59.075476+0000","last_peered":"2024-03-28T23:35:59.075476+0000","last_clean":"2024-03-28T23:35:59.075476+0000","last_became_active":"2024-03-28T23:35:59.073178+0000","last_became_peered":"2024-03-28T23:35:59.073178+0000","last_unstale":"2024-03-28T23:35:59.075476+0000","last_undegraded":"2024-03-28T23:35:59.075476+0000","last_fullsized":"2024-03-28T23:35:59.075476+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:18:01.009699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.086118+0000","last_change":"2024-03-28T23:35:59.080957+0000","last_active":"2024-03-28T23:35:59.086118+0000","last_peered":"2024-03-28T23:35:59.086118+0000","last_clean":"2024-03-28T23:35:59.086118+0000","last_became_active":"2024-03-28T23:35:59.080693+0000","last_became_peered":"2024-03-28T23:35:59.080693+0000","last_unstale":"2024-03-28T23:35:59.086118+0000","last_undegraded":"2024-03-28T23:35:59.086118+0000","last_fullsized":"2024-03-28T23:35:59.086118+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:55.567541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.088426+0000","last_change":"2024-03-28T23:35:59.080315+0000","last_active":"2024-03-28T23:35:59.088426+0000","last_peered":"2024-03-28T23:35:59.088426+0000","last_clean":"2024-03-28T23:35:59.088426+0000","last_became_active":"2024-03-28T23:35:59.080039+0000","last_became_peered":"2024-03-28T23:35:59.080039+0000","last_unstale":"2024-03-28T23:35:59.088426+0000","last_undegraded":"2024-03-28T23:35:59.088426+0000","last_fullsized":"2024-03-28T23:35:59.088426+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:36:46.114123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.075611+0000","last_change":"2024-03-28T23:35:59.074798+0000","last_active":"2024-03-28T23:35:59.075611+0000","last_peered":"2024-03-28T23:35:59.075611+0000","last_clean":"2024-03-28T23:35:59.075611+0000","last_became_active":"2024-03-28T23:35:59.074539+0000","last_became_peered":"2024-03-28T23:35:59.074539+0000","last_unstale":"2024-03-28T23:35:59.075611+0000","last_undegraded":"2024-03-28T23:35:59.075611+0000","last_fullsized":"2024-03-28T23:35:59.075611+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:16:00.430973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.087868+0000","last_change":"2024-03-28T23:35:59.079811+0000","last_active":"2024-03-28T23:35:59.087868+0000","last_peered":"2024-03-28T23:35:59.087868+0000","last_clean":"2024-03-28T23:35:59.087868+0000","last_became_active":"2024-03-28T23:35:59.079538+0000","last_became_peered":"2024-03-28T23:35:59.079538+0000","last_unstale":"2024-03-28T23:35:59.087868+0000","last_undegraded":"2024-03-28T23:35:59.087868+0000","last_fullsized":"2024-03-28T23:35:59.087868+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:06:08.235927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.084348+0000","last_change":"2024-03-28T23:35:59.080958+0000","last_active":"2024-03-28T23:35:59.084348+0000","last_peered":"2024-03-28T23:35:59.084348+0000","last_clean":"2024-03-28T23:35:59.084348+0000","last_became_active":"2024-03-28T23:35:59.080586+0000","last_became_peered":"2024-03-28T23:35:59.080586+0000","last_unstale":"2024-03-28T23:35:59.084348+0000","last_undegraded":"2024-03-28T23:35:59.084348+0000","last_fullsized":"2024-03-28T23:35:59.084348+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:27:42.928036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.080019+0000","last_change":"2024-03-28T23:35:59.076376+0000","last_active":"2024-03-28T23:35:59.080019+0000","last_peered":"2024-03-28T23:35:59.080019+0000","last_clean":"2024-03-28T23:35:59.080019+0000","last_became_active":"2024-03-28T23:35:59.076009+0000","last_became_peered":"2024-03-28T23:35:59.076009+0000","last_unstale":"2024-03-28T23:35:59.080019+0000","last_undegraded":"2024-03-28T23:35:59.080019+0000","last_fullsized":"2024-03-28T23:35:59.080019+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:31:11.799526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.075918+0000","last_change":"2024-03-28T23:35:59.069546+0000","last_active":"2024-03-28T23:35:59.075918+0000","last_peered":"2024-03-28T23:35:59.075918+0000","last_clean":"2024-03-28T23:35:59.075918+0000","last_became_active":"2024-03-28T23:35:59.069283+0000","last_became_peered":"2024-03-28T23:35:59.069283+0000","last_unstale":"2024-03-28T23:35:59.075918+0000","last_undegraded":"2024-03-28T23:35:59.075918+0000","last_fullsized":"2024-03-28T23:35:59.075918+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:57:18.898886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.075788+0000","last_change":"2024-03-28T23:35:59.073880+0000","last_active":"2024-03-28T23:35:59.075788+0000","last_peered":"2024-03-28T23:35:59.075788+0000","last_clean":"2024-03-28T23:35:59.075788+0000","last_became_active":"2024-03-28T23:35:59.073579+0000","last_became_peered":"2024-03-28T23:35:59.073579+0000","last_unstale":"2024-03-28T23:35:59.075788+0000","last_undegraded":"2024-03-28T23:35:59.075788+0000","last_fullsized":"2024-03-28T23:35:59.075788+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:41:12.571144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.088498+0000","last_change":"2024-03-28T23:35:59.082034+0000","last_active":"2024-03-28T23:35:59.088498+0000","last_peered":"2024-03-28T23:35:59.088498+0000","last_clean":"2024-03-28T23:35:59.088498+0000","last_became_active":"2024-03-28T23:35:59.081763+0000","last_became_peered":"2024-03-28T23:35:59.081763+0000","last_unstale":"2024-03-28T23:35:59.088498+0000","last_undegraded":"2024-03-28T23:35:59.088498+0000","last_fullsized":"2024-03-28T23:35:59.088498+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:25.528570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.075280+0000","last_change":"2024-03-28T23:35:59.073561+0000","last_active":"2024-03-28T23:35:59.075280+0000","last_peered":"2024-03-28T23:35:59.075280+0000","last_clean":"2024-03-28T23:35:59.075280+0000","last_became_active":"2024-03-28T23:35:59.073301+0000","last_became_peered":"2024-03-28T23:35:59.073301+0000","last_unstale":"2024-03-28T23:35:59.075280+0000","last_undegraded":"2024-03-28T23:35:59.075280+0000","last_fullsized":"2024-03-28T23:35:59.075280+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:22:08.306031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.076818+0000","last_change":"2024-03-28T23:35:59.074502+0000","last_active":"2024-03-28T23:35:59.076818+0000","last_peered":"2024-03-28T23:35:59.076818+0000","last_clean":"2024-03-28T23:35:59.076818+0000","last_became_active":"2024-03-28T23:35:59.074214+0000","last_became_peered":"2024-03-28T23:35:59.074214+0000","last_unstale":"2024-03-28T23:35:59.076818+0000","last_undegraded":"2024-03-28T23:35:59.076818+0000","last_fullsized":"2024-03-28T23:35:59.076818+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:44:38.063004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.082391+0000","last_change":"2024-03-28T23:35:59.080452+0000","last_active":"2024-03-28T23:35:59.082391+0000","last_peered":"2024-03-28T23:35:59.082391+0000","last_clean":"2024-03-28T23:35:59.082391+0000","last_became_active":"2024-03-28T23:35:59.080141+0000","last_became_peered":"2024-03-28T23:35:59.080141+0000","last_unstale":"2024-03-28T23:35:59.082391+0000","last_undegraded":"2024-03-28T23:35:59.082391+0000","last_fullsized":"2024-03-28T23:35:59.082391+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:58:37.461735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.072611+0000","last_change":"2024-03-28T23:35:59.068642+0000","last_active":"2024-03-28T23:35:59.072611+0000","last_peered":"2024-03-28T23:35:59.072611+0000","last_clean":"2024-03-28T23:35:59.072611+0000","last_became_active":"2024-03-28T23:35:59.068349+0000","last_became_peered":"2024-03-28T23:35:59.068349+0000","last_unstale":"2024-03-28T23:35:59.072611+0000","last_undegraded":"2024-03-28T23:35:59.072611+0000","last_fullsized":"2024-03-28T23:35:59.072611+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:13:23.597258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.084852+0000","last_change":"2024-03-28T23:35:59.083382+0000","last_active":"2024-03-28T23:35:59.084852+0000","last_peered":"2024-03-28T23:35:59.084852+0000","last_clean":"2024-03-28T23:35:59.084852+0000","last_became_active":"2024-03-28T23:35:59.083111+0000","last_became_peered":"2024-03-28T23:35:59.083111+0000","last_unstale":"2024-03-28T23:35:59.084852+0000","last_undegraded":"2024-03-28T23:35:59.084852+0000","last_fullsized":"2024-03-28T23:35:59.084852+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:53:22.705919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"151'4","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.081431+0000","last_change":"2024-03-28T23:35:58.071726+0000","last_active":"2024-03-28T23:35:59.081431+0000","last_peered":"2024-03-28T23:35:59.081431+0000","last_clean":"2024-03-28T23:35:59.081431+0000","last_became_active":"2024-03-28T23:35:58.071384+0000","last_became_peered":"2024-03-28T23:35:58.071384+0000","last_unstale":"2024-03-28T23:35:59.081431+0000","last_undegraded":"2024-03-28T23:35:59.081431+0000","last_fullsized":"2024-03-28T23:35:59.081431+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:31.860347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.088517+0000","last_change":"2024-03-28T23:35:59.082507+0000","last_active":"2024-03-28T23:35:59.088517+0000","last_peered":"2024-03-28T23:35:59.088517+0000","last_clean":"2024-03-28T23:35:59.088517+0000","last_became_active":"2024-03-28T23:35:59.082004+0000","last_became_peered":"2024-03-28T23:35:59.082004+0000","last_unstale":"2024-03-28T23:35:59.088517+0000","last_undegraded":"2024-03-28T23:35:59.088517+0000","last_fullsized":"2024-03-28T23:35:59.088517+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:30:33.779993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"151'1","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.085338+0000","last_change":"2024-03-28T23:35:58.070559+0000","last_active":"2024-03-28T23:35:59.085338+0000","last_peered":"2024-03-28T23:35:59.085338+0000","last_clean":"2024-03-28T23:35:59.085338+0000","last_became_active":"2024-03-28T23:35:58.070259+0000","last_became_peered":"2024-03-28T23:35:58.070259+0000","last_unstale":"2024-03-28T23:35:59.085338+0000","last_undegraded":"2024-03-28T23:35:59.085338+0000","last_fullsized":"2024-03-28T23:35:59.085338+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:23.832628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.088041+0000","last_change":"2024-03-28T23:35:59.082185+0000","last_active":"2024-03-28T23:35:59.088041+0000","last_peered":"2024-03-28T23:35:59.088041+0000","last_clean":"2024-03-28T23:35:59.088041+0000","last_became_active":"2024-03-28T23:35:59.081892+0000","last_became_peered":"2024-03-28T23:35:59.081892+0000","last_unstale":"2024-03-28T23:35:59.088041+0000","last_undegraded":"2024-03-28T23:35:59.088041+0000","last_fullsized":"2024-03-28T23:35:59.088041+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:36:46.114123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"151'2","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.082991+0000","last_change":"2024-03-28T23:35:58.069940+0000","last_active":"2024-03-28T23:35:59.082991+0000","last_peered":"2024-03-28T23:35:59.082991+0000","last_clean":"2024-03-28T23:35:59.082991+0000","last_became_active":"2024-03-28T23:35:58.069533+0000","last_became_peered":"2024-03-28T23:35:58.069533+0000","last_unstale":"2024-03-28T23:35:59.082991+0000","last_undegraded":"2024-03-28T23:35:59.082991+0000","last_fullsized":"2024-03-28T23:35:59.082991+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:14.868910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.083889+0000","last_change":"2024-03-28T23:35:59.080903+0000","last_active":"2024-03-28T23:35:59.083889+0000","last_peered":"2024-03-28T23:35:59.083889+0000","last_clean":"2024-03-28T23:35:59.083889+0000","last_became_active":"2024-03-28T23:35:59.080632+0000","last_became_peered":"2024-03-28T23:35:59.080632+0000","last_unstale":"2024-03-28T23:35:59.083889+0000","last_undegraded":"2024-03-28T23:35:59.083889+0000","last_fullsized":"2024-03-28T23:35:59.083889+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:42:41.721990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"151'2","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.079035+0000","last_change":"2024-03-28T23:35:58.064254+0000","last_active":"2024-03-28T23:35:59.079035+0000","last_peered":"2024-03-28T23:35:59.079035+0000","last_clean":"2024-03-28T23:35:59.079035+0000","last_became_active":"2024-03-28T23:35:58.063972+0000","last_became_peered":"2024-03-28T23:35:58.063972+0000","last_unstale":"2024-03-28T23:35:59.079035+0000","last_undegraded":"2024-03-28T23:35:59.079035+0000","last_fullsized":"2024-03-28T23:35:59.079035+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:36:14.911401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.083989+0000","last_change":"2024-03-28T23:35:59.077647+0000","last_active":"2024-03-28T23:35:59.083989+0000","last_peered":"2024-03-28T23:35:59.083989+0000","last_clean":"2024-03-28T23:35:59.083989+0000","last_became_active":"2024-03-28T23:35:59.077374+0000","last_became_peered":"2024-03-28T23:35:59.077374+0000","last_unstale":"2024-03-28T23:35:59.083989+0000","last_undegraded":"2024-03-28T23:35:59.083989+0000","last_fullsized":"2024-03-28T23:35:59.083989+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:20:09.724407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"151'1","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.088173+0000","last_change":"2024-03-28T23:35:58.074449+0000","last_active":"2024-03-28T23:35:59.088173+0000","last_peered":"2024-03-28T23:35:59.088173+0000","last_clean":"2024-03-28T23:35:59.088173+0000","last_became_active":"2024-03-28T23:35:58.074171+0000","last_became_peered":"2024-03-28T23:35:58.074171+0000","last_unstale":"2024-03-28T23:35:59.088173+0000","last_undegraded":"2024-03-28T23:35:59.088173+0000","last_fullsized":"2024-03-28T23:35:59.088173+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:41:31.356630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.087567+0000","last_change":"2024-03-28T23:35:59.086653+0000","last_active":"2024-03-28T23:35:59.087567+0000","last_peered":"2024-03-28T23:35:59.087567+0000","last_clean":"2024-03-28T23:35:59.087567+0000","last_became_active":"2024-03-28T23:35:59.086398+0000","last_became_peered":"2024-03-28T23:35:59.086398+0000","last_unstale":"2024-03-28T23:35:59.087567+0000","last_undegraded":"2024-03-28T23:35:59.087567+0000","last_fullsized":"2024-03-28T23:35:59.087567+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:57:41.142177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"151'1","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.081283+0000","last_change":"2024-03-28T23:35:58.073421+0000","last_active":"2024-03-28T23:35:59.081283+0000","last_peered":"2024-03-28T23:35:59.081283+0000","last_clean":"2024-03-28T23:35:59.081283+0000","last_became_active":"2024-03-28T23:35:58.072004+0000","last_became_peered":"2024-03-28T23:35:58.072004+0000","last_unstale":"2024-03-28T23:35:59.081283+0000","last_undegraded":"2024-03-28T23:35:59.081283+0000","last_fullsized":"2024-03-28T23:35:59.081283+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:10.808877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.072847+0000","last_change":"2024-03-28T23:35:59.063776+0000","last_active":"2024-03-28T23:35:59.072847+0000","last_peered":"2024-03-28T23:35:59.072847+0000","last_clean":"2024-03-28T23:35:59.072847+0000","last_became_active":"2024-03-28T23:35:59.063214+0000","last_became_peered":"2024-03-28T23:35:59.063214+0000","last_unstale":"2024-03-28T23:35:59.072847+0000","last_undegraded":"2024-03-28T23:35:59.072847+0000","last_fullsized":"2024-03-28T23:35:59.072847+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:49:53.041348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.057618+0000","last_change":"2024-03-28T23:35:58.072230+0000","last_active":"2024-03-28T23:35:59.057618+0000","last_peered":"2024-03-28T23:35:59.057618+0000","last_clean":"2024-03-28T23:35:59.057618+0000","last_became_active":"2024-03-28T23:35:58.071967+0000","last_became_peered":"2024-03-28T23:35:58.071967+0000","last_unstale":"2024-03-28T23:35:59.057618+0000","last_undegraded":"2024-03-28T23:35:59.057618+0000","last_fullsized":"2024-03-28T23:35:59.057618+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:07:28.431728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.076748+0000","last_change":"2024-03-28T23:35:59.070728+0000","last_active":"2024-03-28T23:35:59.076748+0000","last_peered":"2024-03-28T23:35:59.076748+0000","last_clean":"2024-03-28T23:35:59.076748+0000","last_became_active":"2024-03-28T23:35:59.070489+0000","last_became_peered":"2024-03-28T23:35:59.070489+0000","last_unstale":"2024-03-28T23:35:59.076748+0000","last_undegraded":"2024-03-28T23:35:59.076748+0000","last_fullsized":"2024-03-28T23:35:59.076748+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:26:59.457089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.061652+0000","last_change":"2024-03-28T23:35:58.068807+0000","last_active":"2024-03-28T23:35:59.061652+0000","last_peered":"2024-03-28T23:35:59.061652+0000","last_clean":"2024-03-28T23:35:59.061652+0000","last_became_active":"2024-03-28T23:35:58.068530+0000","last_became_peered":"2024-03-28T23:35:58.068530+0000","last_unstale":"2024-03-28T23:35:59.061652+0000","last_undegraded":"2024-03-28T23:35:59.061652+0000","last_fullsized":"2024-03-28T23:35:59.061652+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:12.327316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.083907+0000","last_change":"2024-03-28T23:35:59.082657+0000","last_active":"2024-03-28T23:35:59.083907+0000","last_peered":"2024-03-28T23:35:59.083907+0000","last_clean":"2024-03-28T23:35:59.083907+0000","last_became_active":"2024-03-28T23:35:59.082390+0000","last_became_peered":"2024-03-28T23:35:59.082390+0000","last_unstale":"2024-03-28T23:35:59.083907+0000","last_undegraded":"2024-03-28T23:35:59.083907+0000","last_fullsized":"2024-03-28T23:35:59.083907+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:50:09.386042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.065738+0000","last_change":"2024-03-28T23:35:58.075215+0000","last_active":"2024-03-28T23:35:59.065738+0000","last_peered":"2024-03-28T23:35:59.065738+0000","last_clean":"2024-03-28T23:35:59.065738+0000","last_became_active":"2024-03-28T23:35:58.074891+0000","last_became_peered":"2024-03-28T23:35:58.074891+0000","last_unstale":"2024-03-28T23:35:59.065738+0000","last_undegraded":"2024-03-28T23:35:59.065738+0000","last_fullsized":"2024-03-28T23:35:59.065738+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:16.223515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.088005+0000","last_change":"2024-03-28T23:35:59.079653+0000","last_active":"2024-03-28T23:35:59.088005+0000","last_peered":"2024-03-28T23:35:59.088005+0000","last_clean":"2024-03-28T23:35:59.088005+0000","last_became_active":"2024-03-28T23:35:59.079390+0000","last_became_peered":"2024-03-28T23:35:59.079390+0000","last_unstale":"2024-03-28T23:35:59.088005+0000","last_undegraded":"2024-03-28T23:35:59.088005+0000","last_fullsized":"2024-03-28T23:35:59.088005+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:08:30.907410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"151'3","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.076316+0000","last_change":"2024-03-28T23:35:58.059191+0000","last_active":"2024-03-28T23:35:59.076316+0000","last_peered":"2024-03-28T23:35:59.076316+0000","last_clean":"2024-03-28T23:35:59.076316+0000","last_became_active":"2024-03-28T23:35:58.058860+0000","last_became_peered":"2024-03-28T23:35:58.058860+0000","last_unstale":"2024-03-28T23:35:59.076316+0000","last_undegraded":"2024-03-28T23:35:59.076316+0000","last_fullsized":"2024-03-28T23:35:59.076316+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:22.564724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.080775+0000","last_change":"2024-03-28T23:35:59.074941+0000","last_active":"2024-03-28T23:35:59.080775+0000","last_peered":"2024-03-28T23:35:59.080775+0000","last_clean":"2024-03-28T23:35:59.080775+0000","last_became_active":"2024-03-28T23:35:59.074636+0000","last_became_peered":"2024-03-28T23:35:59.074636+0000","last_unstale":"2024-03-28T23:35:59.080775+0000","last_undegraded":"2024-03-28T23:35:59.080775+0000","last_fullsized":"2024-03-28T23:35:59.080775+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:20:51.733648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"151'6","reported_seq":18,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.089573+0000","last_change":"2024-03-28T23:35:58.075069+0000","last_active":"2024-03-28T23:35:59.089573+0000","last_peered":"2024-03-28T23:35:59.089573+0000","last_clean":"2024-03-28T23:35:59.089573+0000","last_became_active":"2024-03-28T23:35:58.074795+0000","last_became_peered":"2024-03-28T23:35:58.074795+0000","last_unstale":"2024-03-28T23:35:59.089573+0000","last_undegraded":"2024-03-28T23:35:59.089573+0000","last_fullsized":"2024-03-28T23:35:59.089573+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:49:07.831860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.080125+0000","last_change":"2024-03-28T23:35:59.076575+0000","last_active":"2024-03-28T23:35:59.080125+0000","last_peered":"2024-03-28T23:35:59.080125+0000","last_clean":"2024-03-28T23:35:59.080125+0000","last_became_active":"2024-03-28T23:35:59.076319+0000","last_became_peered":"2024-03-28T23:35:59.076319+0000","last_unstale":"2024-03-28T23:35:59.080125+0000","last_undegraded":"2024-03-28T23:35:59.080125+0000","last_fullsized":"2024-03-28T23:35:59.080125+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:53.207692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"151'2","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.086105+0000","last_change":"2024-03-28T23:35:58.073277+0000","last_active":"2024-03-28T23:35:59.086105+0000","last_peered":"2024-03-28T23:35:59.086105+0000","last_clean":"2024-03-28T23:35:59.086105+0000","last_became_active":"2024-03-28T23:35:58.071396+0000","last_became_peered":"2024-03-28T23:35:58.071396+0000","last_unstale":"2024-03-28T23:35:59.086105+0000","last_undegraded":"2024-03-28T23:35:59.086105+0000","last_fullsized":"2024-03-28T23:35:59.086105+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:36:25.234053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.078275+0000","last_change":"2024-03-28T23:35:59.072656+0000","last_active":"2024-03-28T23:35:59.078275+0000","last_peered":"2024-03-28T23:35:59.078275+0000","last_clean":"2024-03-28T23:35:59.078275+0000","last_became_active":"2024-03-28T23:35:59.072400+0000","last_became_peered":"2024-03-28T23:35:59.072400+0000","last_unstale":"2024-03-28T23:35:59.078275+0000","last_undegraded":"2024-03-28T23:35:59.078275+0000","last_fullsized":"2024-03-28T23:35:59.078275+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:11.349017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.069463+0000","last_change":"2024-03-28T23:35:58.071626+0000","last_active":"2024-03-28T23:35:59.069463+0000","last_peered":"2024-03-28T23:35:59.069463+0000","last_clean":"2024-03-28T23:35:59.069463+0000","last_became_active":"2024-03-28T23:35:58.071339+0000","last_became_peered":"2024-03-28T23:35:58.071339+0000","last_unstale":"2024-03-28T23:35:59.069463+0000","last_undegraded":"2024-03-28T23:35:59.069463+0000","last_fullsized":"2024-03-28T23:35:59.069463+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:10.808877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.076765+0000","last_change":"2024-03-28T23:35:59.076190+0000","last_active":"2024-03-28T23:35:59.076765+0000","last_peered":"2024-03-28T23:35:59.076765+0000","last_clean":"2024-03-28T23:35:59.076765+0000","last_became_active":"2024-03-28T23:35:59.075881+0000","last_became_peered":"2024-03-28T23:35:59.075881+0000","last_unstale":"2024-03-28T23:35:59.076765+0000","last_undegraded":"2024-03-28T23:35:59.076765+0000","last_fullsized":"2024-03-28T23:35:59.076765+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:41:32.361865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.068514+0000","last_change":"2024-03-28T23:35:58.062542+0000","last_active":"2024-03-28T23:35:59.068514+0000","last_peered":"2024-03-28T23:35:59.068514+0000","last_clean":"2024-03-28T23:35:59.068514+0000","last_became_active":"2024-03-28T23:35:58.062074+0000","last_became_peered":"2024-03-28T23:35:58.062074+0000","last_unstale":"2024-03-28T23:35:59.068514+0000","last_undegraded":"2024-03-28T23:35:59.068514+0000","last_fullsized":"2024-03-28T23:35:59.068514+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:23:41.041246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.074170+0000","last_change":"2024-03-28T23:35:59.069831+0000","last_active":"2024-03-28T23:35:59.074170+0000","last_peered":"2024-03-28T23:35:59.074170+0000","last_clean":"2024-03-28T23:35:59.074170+0000","last_became_active":"2024-03-28T23:35:59.069272+0000","last_became_peered":"2024-03-28T23:35:59.069272+0000","last_unstale":"2024-03-28T23:35:59.074170+0000","last_undegraded":"2024-03-28T23:35:59.074170+0000","last_fullsized":"2024-03-28T23:35:59.074170+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:46:44.758689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"151'8","reported_seq":20,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.087517+0000","last_change":"2024-03-28T23:35:58.071172+0000","last_active":"2024-03-28T23:35:59.087517+0000","last_peered":"2024-03-28T23:35:59.087517+0000","last_clean":"2024-03-28T23:35:59.087517+0000","last_became_active":"2024-03-28T23:35:58.070901+0000","last_became_peered":"2024-03-28T23:35:58.070901+0000","last_unstale":"2024-03-28T23:35:59.087517+0000","last_undegraded":"2024-03-28T23:35:59.087517+0000","last_fullsized":"2024-03-28T23:35:59.087517+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:16:53.124176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.072109+0000","last_change":"2024-03-28T23:35:59.067559+0000","last_active":"2024-03-28T23:35:59.072109+0000","last_peered":"2024-03-28T23:35:59.072109+0000","last_clean":"2024-03-28T23:35:59.072109+0000","last_became_active":"2024-03-28T23:35:59.067265+0000","last_became_peered":"2024-03-28T23:35:59.067265+0000","last_unstale":"2024-03-28T23:35:59.072109+0000","last_undegraded":"2024-03-28T23:35:59.072109+0000","last_fullsized":"2024-03-28T23:35:59.072109+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:41:12.571144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"151'1","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.085113+0000","last_change":"2024-03-28T23:35:58.064795+0000","last_active":"2024-03-28T23:35:59.085113+0000","last_peered":"2024-03-28T23:35:59.085113+0000","last_clean":"2024-03-28T23:35:59.085113+0000","last_became_active":"2024-03-28T23:35:58.064506+0000","last_became_peered":"2024-03-28T23:35:58.064506+0000","last_unstale":"2024-03-28T23:35:59.085113+0000","last_undegraded":"2024-03-28T23:35:59.085113+0000","last_fullsized":"2024-03-28T23:35:59.085113+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:36:25.234053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.077291+0000","last_change":"2024-03-28T23:35:59.075241+0000","last_active":"2024-03-28T23:35:59.077291+0000","last_peered":"2024-03-28T23:35:59.077291+0000","last_clean":"2024-03-28T23:35:59.077291+0000","last_became_active":"2024-03-28T23:35:59.074170+0000","last_became_peered":"2024-03-28T23:35:59.074170+0000","last_unstale":"2024-03-28T23:35:59.077291+0000","last_undegraded":"2024-03-28T23:35:59.077291+0000","last_fullsized":"2024-03-28T23:35:59.077291+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:30:17.010487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"151'2","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.087771+0000","last_change":"2024-03-28T23:35:58.070169+0000","last_active":"2024-03-28T23:35:59.087771+0000","last_peered":"2024-03-28T23:35:59.087771+0000","last_clean":"2024-03-28T23:35:59.087771+0000","last_became_active":"2024-03-28T23:35:58.069913+0000","last_became_peered":"2024-03-28T23:35:58.069913+0000","last_unstale":"2024-03-28T23:35:59.087771+0000","last_undegraded":"2024-03-28T23:35:59.087771+0000","last_fullsized":"2024-03-28T23:35:59.087771+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:22:07.300796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.083433+0000","last_change":"2024-03-28T23:35:59.081763+0000","last_active":"2024-03-28T23:35:59.083433+0000","last_peered":"2024-03-28T23:35:59.083433+0000","last_clean":"2024-03-28T23:35:59.083433+0000","last_became_active":"2024-03-28T23:35:59.081465+0000","last_became_peered":"2024-03-28T23:35:59.081465+0000","last_unstale":"2024-03-28T23:35:59.083433+0000","last_undegraded":"2024-03-28T23:35:59.083433+0000","last_fullsized":"2024-03-28T23:35:59.083433+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:22:08.306031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"151'1","reported_seq":13,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.085508+0000","last_change":"2024-03-28T23:35:58.070308+0000","last_active":"2024-03-28T23:35:59.085508+0000","last_peered":"2024-03-28T23:35:59.085508+0000","last_clean":"2024-03-28T23:35:59.085508+0000","last_became_active":"2024-03-28T23:35:58.070024+0000","last_became_peered":"2024-03-28T23:35:58.070024+0000","last_unstale":"2024-03-28T23:35:59.085508+0000","last_undegraded":"2024-03-28T23:35:59.085508+0000","last_fullsized":"2024-03-28T23:35:59.085508+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:49:07.831860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.076088+0000","last_change":"2024-03-28T23:35:59.075326+0000","last_active":"2024-03-28T23:35:59.076088+0000","last_peered":"2024-03-28T23:35:59.076088+0000","last_clean":"2024-03-28T23:35:59.076088+0000","last_became_active":"2024-03-28T23:35:59.075063+0000","last_became_peered":"2024-03-28T23:35:59.075063+0000","last_unstale":"2024-03-28T23:35:59.076088+0000","last_undegraded":"2024-03-28T23:35:59.076088+0000","last_fullsized":"2024-03-28T23:35:59.076088+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:24:07.077373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.063380+0000","last_change":"2024-03-28T23:35:58.073249+0000","last_active":"2024-03-28T23:35:59.063380+0000","last_peered":"2024-03-28T23:35:59.063380+0000","last_clean":"2024-03-28T23:35:59.063380+0000","last_became_active":"2024-03-28T23:35:58.072992+0000","last_became_peered":"2024-03-28T23:35:58.072992+0000","last_unstale":"2024-03-28T23:35:59.063380+0000","last_undegraded":"2024-03-28T23:35:59.063380+0000","last_fullsized":"2024-03-28T23:35:59.063380+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:58:36.456500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.088483+0000","last_change":"2024-03-28T23:35:59.082001+0000","last_active":"2024-03-28T23:35:59.088483+0000","last_peered":"2024-03-28T23:35:59.088483+0000","last_clean":"2024-03-28T23:35:59.088483+0000","last_became_active":"2024-03-28T23:35:59.081706+0000","last_became_peered":"2024-03-28T23:35:59.081706+0000","last_unstale":"2024-03-28T23:35:59.088483+0000","last_undegraded":"2024-03-28T23:35:59.088483+0000","last_fullsized":"2024-03-28T23:35:59.088483+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:49:26.989533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"151'2","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.087777+0000","last_change":"2024-03-28T23:35:58.072448+0000","last_active":"2024-03-28T23:35:59.087777+0000","last_peered":"2024-03-28T23:35:59.087777+0000","last_clean":"2024-03-28T23:35:59.087777+0000","last_became_active":"2024-03-28T23:35:58.071494+0000","last_became_peered":"2024-03-28T23:35:58.071494+0000","last_unstale":"2024-03-28T23:35:59.087777+0000","last_undegraded":"2024-03-28T23:35:59.087777+0000","last_fullsized":"2024-03-28T23:35:59.087777+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:54:03.045194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.087880+0000","last_change":"2024-03-28T23:35:59.080948+0000","last_active":"2024-03-28T23:35:59.087880+0000","last_peered":"2024-03-28T23:35:59.087880+0000","last_clean":"2024-03-28T23:35:59.087880+0000","last_became_active":"2024-03-28T23:35:59.080672+0000","last_became_peered":"2024-03-28T23:35:59.080672+0000","last_unstale":"2024-03-28T23:35:59.087880+0000","last_undegraded":"2024-03-28T23:35:59.087880+0000","last_fullsized":"2024-03-28T23:35:59.087880+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:08:30.907410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.065989+0000","last_change":"2024-03-28T23:35:58.071966+0000","last_active":"2024-03-28T23:35:59.065989+0000","last_peered":"2024-03-28T23:35:59.065989+0000","last_clean":"2024-03-28T23:35:59.065989+0000","last_became_active":"2024-03-28T23:35:58.071656+0000","last_became_peered":"2024-03-28T23:35:58.071656+0000","last_unstale":"2024-03-28T23:35:59.065989+0000","last_undegraded":"2024-03-28T23:35:59.065989+0000","last_fullsized":"2024-03-28T23:35:59.065989+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:53:34.853174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.076260+0000","last_change":"2024-03-28T23:35:59.074451+0000","last_active":"2024-03-28T23:35:59.076260+0000","last_peered":"2024-03-28T23:35:59.076260+0000","last_clean":"2024-03-28T23:35:59.076260+0000","last_became_active":"2024-03-28T23:35:59.074181+0000","last_became_peered":"2024-03-28T23:35:59.074181+0000","last_unstale":"2024-03-28T23:35:59.076260+0000","last_undegraded":"2024-03-28T23:35:59.076260+0000","last_fullsized":"2024-03-28T23:35:59.076260+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:31:11.799526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.058590+0000","last_change":"2024-03-28T23:35:58.067234+0000","last_active":"2024-03-28T23:35:59.058590+0000","last_peered":"2024-03-28T23:35:59.058590+0000","last_clean":"2024-03-28T23:35:59.058590+0000","last_became_active":"2024-03-28T23:35:58.066942+0000","last_became_peered":"2024-03-28T23:35:58.066942+0000","last_unstale":"2024-03-28T23:35:59.058590+0000","last_undegraded":"2024-03-28T23:35:59.058590+0000","last_fullsized":"2024-03-28T23:35:59.058590+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:37:03.241869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.085605+0000","last_change":"2024-03-28T23:35:59.084751+0000","last_active":"2024-03-28T23:35:59.085605+0000","last_peered":"2024-03-28T23:35:59.085605+0000","last_clean":"2024-03-28T23:35:59.085605+0000","last_became_active":"2024-03-28T23:35:59.084472+0000","last_became_peered":"2024-03-28T23:35:59.084472+0000","last_unstale":"2024-03-28T23:35:59.085605+0000","last_undegraded":"2024-03-28T23:35:59.085605+0000","last_fullsized":"2024-03-28T23:35:59.085605+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:20:09.724407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.062392+0000","last_change":"2024-03-28T23:35:58.067542+0000","last_active":"2024-03-28T23:35:59.062392+0000","last_peered":"2024-03-28T23:35:59.062392+0000","last_clean":"2024-03-28T23:35:59.062392+0000","last_became_active":"2024-03-28T23:35:58.067013+0000","last_became_peered":"2024-03-28T23:35:58.067013+0000","last_unstale":"2024-03-28T23:35:59.062392+0000","last_undegraded":"2024-03-28T23:35:59.062392+0000","last_fullsized":"2024-03-28T23:35:59.062392+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:41:31.356630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.078682+0000","last_change":"2024-03-28T23:35:59.077874+0000","last_active":"2024-03-28T23:35:59.078682+0000","last_peered":"2024-03-28T23:35:59.078682+0000","last_clean":"2024-03-28T23:35:59.078682+0000","last_became_active":"2024-03-28T23:35:59.077613+0000","last_became_peered":"2024-03-28T23:35:59.077613+0000","last_unstale":"2024-03-28T23:35:59.078682+0000","last_undegraded":"2024-03-28T23:35:59.078682+0000","last_fullsized":"2024-03-28T23:35:59.078682+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:13.332551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"151'2","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.081724+0000","last_change":"2024-03-28T23:35:58.066436+0000","last_active":"2024-03-28T23:35:59.081724+0000","last_peered":"2024-03-28T23:35:59.081724+0000","last_clean":"2024-03-28T23:35:59.081724+0000","last_became_active":"2024-03-28T23:35:58.063834+0000","last_became_peered":"2024-03-28T23:35:58.063834+0000","last_unstale":"2024-03-28T23:35:59.081724+0000","last_undegraded":"2024-03-28T23:35:59.081724+0000","last_fullsized":"2024-03-28T23:35:59.081724+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:37:09.547486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.078369+0000","last_change":"2024-03-28T23:35:59.074166+0000","last_active":"2024-03-28T23:35:59.078369+0000","last_peered":"2024-03-28T23:35:59.078369+0000","last_clean":"2024-03-28T23:35:59.078369+0000","last_became_active":"2024-03-28T23:35:59.073916+0000","last_became_peered":"2024-03-28T23:35:59.073916+0000","last_unstale":"2024-03-28T23:35:59.078369+0000","last_undegraded":"2024-03-28T23:35:59.078369+0000","last_fullsized":"2024-03-28T23:35:59.078369+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:42:41.721990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.067718+0000","last_change":"2024-03-28T23:35:58.078010+0000","last_active":"2024-03-28T23:35:59.067718+0000","last_peered":"2024-03-28T23:35:59.067718+0000","last_clean":"2024-03-28T23:35:59.067718+0000","last_became_active":"2024-03-28T23:35:58.077736+0000","last_became_peered":"2024-03-28T23:35:58.077736+0000","last_unstale":"2024-03-28T23:35:59.067718+0000","last_undegraded":"2024-03-28T23:35:59.067718+0000","last_fullsized":"2024-03-28T23:35:59.067718+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:54:03.045194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.074265+0000","last_change":"2024-03-28T23:35:59.062460+0000","last_active":"2024-03-28T23:35:59.074265+0000","last_peered":"2024-03-28T23:35:59.074265+0000","last_clean":"2024-03-28T23:35:59.074265+0000","last_became_active":"2024-03-28T23:35:59.062185+0000","last_became_peered":"2024-03-28T23:35:59.062185+0000","last_unstale":"2024-03-28T23:35:59.074265+0000","last_undegraded":"2024-03-28T23:35:59.074265+0000","last_fullsized":"2024-03-28T23:35:59.074265+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:45:16.369047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.066288+0000","last_change":"2024-03-28T23:35:58.072356+0000","last_active":"2024-03-28T23:35:59.066288+0000","last_peered":"2024-03-28T23:35:59.066288+0000","last_clean":"2024-03-28T23:35:59.066288+0000","last_became_active":"2024-03-28T23:35:58.072111+0000","last_became_peered":"2024-03-28T23:35:58.072111+0000","last_unstale":"2024-03-28T23:35:59.066288+0000","last_undegraded":"2024-03-28T23:35:59.066288+0000","last_fullsized":"2024-03-28T23:35:59.066288+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:23.832628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.086800+0000","last_change":"2024-03-28T23:35:59.079950+0000","last_active":"2024-03-28T23:35:59.086800+0000","last_peered":"2024-03-28T23:35:59.086800+0000","last_clean":"2024-03-28T23:35:59.086800+0000","last_became_active":"2024-03-28T23:35:59.079675+0000","last_became_peered":"2024-03-28T23:35:59.079675+0000","last_unstale":"2024-03-28T23:35:59.086800+0000","last_undegraded":"2024-03-28T23:35:59.086800+0000","last_fullsized":"2024-03-28T23:35:59.086800+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:14:30.938161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.058232+0000","last_change":"2024-03-28T23:35:58.068737+0000","last_active":"2024-03-28T23:35:59.058232+0000","last_peered":"2024-03-28T23:35:59.058232+0000","last_clean":"2024-03-28T23:35:59.058232+0000","last_became_active":"2024-03-28T23:35:58.068481+0000","last_became_peered":"2024-03-28T23:35:58.068481+0000","last_unstale":"2024-03-28T23:35:59.058232+0000","last_undegraded":"2024-03-28T23:35:59.058232+0000","last_fullsized":"2024-03-28T23:35:59.058232+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:05:05.111519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.083843+0000","last_change":"2024-03-28T23:35:59.077779+0000","last_active":"2024-03-28T23:35:59.083843+0000","last_peered":"2024-03-28T23:35:59.083843+0000","last_clean":"2024-03-28T23:35:59.083843+0000","last_became_active":"2024-03-28T23:35:59.077529+0000","last_became_peered":"2024-03-28T23:35:59.077529+0000","last_unstale":"2024-03-28T23:35:59.083843+0000","last_undegraded":"2024-03-28T23:35:59.083843+0000","last_fullsized":"2024-03-28T23:35:59.083843+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:03:28.508127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.057364+0000","last_change":"2024-03-28T23:35:58.072655+0000","last_active":"2024-03-28T23:35:59.057364+0000","last_peered":"2024-03-28T23:35:59.057364+0000","last_clean":"2024-03-28T23:35:59.057364+0000","last_became_active":"2024-03-28T23:35:58.072379+0000","last_became_peered":"2024-03-28T23:35:58.072379+0000","last_unstale":"2024-03-28T23:35:59.057364+0000","last_undegraded":"2024-03-28T23:35:59.057364+0000","last_fullsized":"2024-03-28T23:35:59.057364+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:57:05.274295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.082831+0000","last_change":"2024-03-28T23:35:59.080115+0000","last_active":"2024-03-28T23:35:59.082831+0000","last_peered":"2024-03-28T23:35:59.082831+0000","last_clean":"2024-03-28T23:35:59.082831+0000","last_became_active":"2024-03-28T23:35:59.079809+0000","last_became_peered":"2024-03-28T23:35:59.079809+0000","last_unstale":"2024-03-28T23:35:59.082831+0000","last_undegraded":"2024-03-28T23:35:59.082831+0000","last_fullsized":"2024-03-28T23:35:59.082831+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:06:37.212712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.069602+0000","last_change":"2024-03-28T23:35:58.071453+0000","last_active":"2024-03-28T23:35:59.069602+0000","last_peered":"2024-03-28T23:35:59.069602+0000","last_clean":"2024-03-28T23:35:59.069602+0000","last_became_active":"2024-03-28T23:35:58.071072+0000","last_became_peered":"2024-03-28T23:35:58.071072+0000","last_unstale":"2024-03-28T23:35:59.069602+0000","last_undegraded":"2024-03-28T23:35:59.069602+0000","last_fullsized":"2024-03-28T23:35:59.069602+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:20:20.872070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.088472+0000","last_change":"2024-03-28T23:35:59.081608+0000","last_active":"2024-03-28T23:35:59.088472+0000","last_peered":"2024-03-28T23:35:59.088472+0000","last_clean":"2024-03-28T23:35:59.088472+0000","last_became_active":"2024-03-28T23:35:59.080839+0000","last_became_peered":"2024-03-28T23:35:59.080839+0000","last_unstale":"2024-03-28T23:35:59.088472+0000","last_undegraded":"2024-03-28T23:35:59.088472+0000","last_fullsized":"2024-03-28T23:35:59.088472+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:44:23.521230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.068681+0000","last_change":"2024-03-28T23:35:58.067740+0000","last_active":"2024-03-28T23:35:59.068681+0000","last_peered":"2024-03-28T23:35:59.068681+0000","last_clean":"2024-03-28T23:35:59.068681+0000","last_became_active":"2024-03-28T23:35:58.067418+0000","last_became_peered":"2024-03-28T23:35:58.067418+0000","last_unstale":"2024-03-28T23:35:59.068681+0000","last_undegraded":"2024-03-28T23:35:59.068681+0000","last_fullsized":"2024-03-28T23:35:59.068681+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:00:58.303444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.084161+0000","last_change":"2024-03-28T23:35:59.079990+0000","last_active":"2024-03-28T23:35:59.084161+0000","last_peered":"2024-03-28T23:35:59.084161+0000","last_clean":"2024-03-28T23:35:59.084161+0000","last_became_active":"2024-03-28T23:35:59.079717+0000","last_became_peered":"2024-03-28T23:35:59.079717+0000","last_unstale":"2024-03-28T23:35:59.084161+0000","last_undegraded":"2024-03-28T23:35:59.084161+0000","last_fullsized":"2024-03-28T23:35:59.084161+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:10:20.786364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.070181+0000","last_change":"2024-03-28T23:35:58.071103+0000","last_active":"2024-03-28T23:35:59.070181+0000","last_peered":"2024-03-28T23:35:59.070181+0000","last_clean":"2024-03-28T23:35:59.070181+0000","last_became_active":"2024-03-28T23:35:58.070587+0000","last_became_peered":"2024-03-28T23:35:58.070587+0000","last_unstale":"2024-03-28T23:35:59.070181+0000","last_undegraded":"2024-03-28T23:35:59.070181+0000","last_fullsized":"2024-03-28T23:35:59.070181+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:16.223515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":9,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.079241+0000","last_change":"2024-03-28T23:35:59.075847+0000","last_active":"2024-03-28T23:35:59.079241+0000","last_peered":"2024-03-28T23:35:59.079241+0000","last_clean":"2024-03-28T23:35:59.079241+0000","last_became_active":"2024-03-28T23:35:59.072889+0000","last_became_peered":"2024-03-28T23:35:59.072889+0000","last_unstale":"2024-03-28T23:35:59.079241+0000","last_undegraded":"2024-03-28T23:35:59.079241+0000","last_fullsized":"2024-03-28T23:35:59.079241+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:11.349017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.067569+0000","last_change":"2024-03-28T23:35:58.078046+0000","last_active":"2024-03-28T23:35:59.067569+0000","last_peered":"2024-03-28T23:35:59.067569+0000","last_clean":"2024-03-28T23:35:59.067569+0000","last_became_active":"2024-03-28T23:35:58.077780+0000","last_became_peered":"2024-03-28T23:35:58.077780+0000","last_unstale":"2024-03-28T23:35:59.067569+0000","last_undegraded":"2024-03-28T23:35:59.067569+0000","last_fullsized":"2024-03-28T23:35:59.067569+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:37:09.547486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":415,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-28T23:35:59.062474+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:35:59.062474+0000","last_peered":"2024-03-28T23:35:59.062474+0000","last_clean":"2024-03-28T23:35:59.062474+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:35:59.062474+0000","last_undegraded":"2024-03-28T23:35:59.062474+0000","last_fullsized":"2024-03-28T23:35:59.062474+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1966080,"data_stored":1508793,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1888256,"data_stored":911258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574927,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1888256,"data_stored":911258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574926,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1888256,"data_stored":911258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574926,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33060,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338780,"statfs":{"total":96636764160,"available":96602910720,"internally_reserved":0,"allocated":1916928,"data_stored":919701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574926,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1888256,"data_stored":911258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33056,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338784,"statfs":{"total":96636764160,"available":96602914816,"internally_reserved":0,"allocated":1912832,"data_stored":920037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574926,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33096,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338744,"statfs":{"total":96636764160,"available":96602873856,"internally_reserved":0,"allocated":1953792,"data_stored":1501626,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:36:02.376 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T23:36:02.376 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-28T23:36:02.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.521+0000 7fd679547640 1 Processor -- start 2024-03-28T23:36:02.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.521+0000 7fd679547640 1 -- start start 2024-03-28T23:36:02.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.522+0000 7fd679547640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd6741554c0 0x7fd6741558a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:02.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.522+0000 7fd679547640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6740c6010 con 0x7fd6740ca540 2024-03-28T23:36:02.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.522+0000 7fd679547640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6740c62f0 con 0x7fd6740bfc70 2024-03-28T23:36:02.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.522+0000 7fd679547640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd6740c6eb0 con 0x7fd6741554c0 2024-03-28T23:36:02.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.522+0000 7fd672ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd6741554c0 0x7fd6741558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:02.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.522+0000 7fd6727fc640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd6740ca540 0x7fd6740c55e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50094/0 (socket says 172.21.15.161:50094) 2024-03-28T23:36:02.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.522+0000 7fd6727fc640 1 -- 172.21.15.161:0/1252874600 learned_addr learned my addr 172.21.15.161:0/1252874600 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:02.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.522+0000 7fd672ffd640 1 -- 172.21.15.161:0/1252874600 >> v1:172.21.15.170:6789/0 conn(0x7fd6740bfc70 legacy=0x7fd674152620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:02.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.522+0000 7fd672ffd640 1 -- 172.21.15.161:0/1252874600 >> v1:172.21.15.161:6789/0 conn(0x7fd6740ca540 legacy=0x7fd6740c55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:02.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.522+0000 7fd672ffd640 1 -- 172.21.15.161:0/1252874600 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6740c5cf0 con 0x7fd6741554c0 2024-03-28T23:36:02.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.523+0000 7fd672ffd640 1 --2- 172.21.15.161:0/1252874600 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6741554c0 0x7fd6741558a0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fd6640097d0 tx=0x7fd66402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac6978e2cf5bd8a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:02.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.523+0000 7fd671ffb640 1 -- 172.21.15.161:0/1252874600 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd664035020 con 0x7fd6741554c0 2024-03-28T23:36:02.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.523+0000 7fd671ffb640 1 -- 172.21.15.161:0/1252874600 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd664031830 con 0x7fd6741554c0 2024-03-28T23:36:02.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.523+0000 7fd671ffb640 1 -- 172.21.15.161:0/1252874600 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd664031c00 con 0x7fd6741554c0 2024-03-28T23:36:02.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.523+0000 7fd679547640 1 -- 172.21.15.161:0/1252874600 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6741554c0 msgr2=0x7fd6741558a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:02.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.523+0000 7fd679547640 1 --2- 172.21.15.161:0/1252874600 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6741554c0 0x7fd6741558a0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fd6640097d0 tx=0x7fd66402b8d0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.524+0000 7fd679547640 1 -- 172.21.15.161:0/1252874600 shutdown_connections 2024-03-28T23:36:02.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.524+0000 7fd679547640 1 --2- 172.21.15.161:0/1252874600 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6741554c0 0x7fd6741558a0 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.524+0000 7fd679547640 1 -- 172.21.15.161:0/1252874600 >> 172.21.15.161:0/1252874600 conn(0x7fd6740ca9c0 msgr2=0x7fd6740cadd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:02.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.524+0000 7fd679547640 1 -- 172.21.15.161:0/1252874600 shutdown_connections 2024-03-28T23:36:02.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.524+0000 7fd679547640 1 -- 172.21.15.161:0/1252874600 wait complete. 2024-03-28T23:36:02.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.525+0000 7fd679547640 1 Processor -- start 2024-03-28T23:36:02.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.525+0000 7fd679547640 1 -- start start 2024-03-28T23:36:02.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.525+0000 7fd679547640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6740bfc70 0x7fd674149720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:02.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.525+0000 7fd679547640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6740ca540 0x7fd6741546b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:02.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.525+0000 7fd679547640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6741554c0 0x7fd6740afd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:02.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.525+0000 7fd679547640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd674152d30 con 0x7fd6740bfc70 2024-03-28T23:36:02.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.525+0000 7fd679547640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd674155eb0 con 0x7fd6741554c0 2024-03-28T23:36:02.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.525+0000 7fd679547640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd674152eb0 con 0x7fd6740ca540 2024-03-28T23:36:02.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.526+0000 7fd672ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6740bfc70 0x7fd674149720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:02.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.526+0000 7fd672ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6740bfc70 0x7fd674149720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53740/0 (socket says 172.21.15.161:53740) 2024-03-28T23:36:02.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.526+0000 7fd672ffd640 1 -- 172.21.15.161:0/626555710 learned_addr learned my addr 172.21.15.161:0/626555710 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:02.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.526+0000 7fd6727fc640 1 --2- 172.21.15.161:0/626555710 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6740ca540 0x7fd6741546b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:02.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.526+0000 7fd6737fe640 1 --2- 172.21.15.161:0/626555710 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6741554c0 0x7fd6740afd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:02.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.526+0000 7fd672ffd640 1 -- 172.21.15.161:0/626555710 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6740ca540 msgr2=0x7fd6741546b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:02.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.526+0000 7fd672ffd640 1 --2- 172.21.15.161:0/626555710 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6740ca540 0x7fd6741546b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.526+0000 7fd672ffd640 1 -- 172.21.15.161:0/626555710 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6741554c0 msgr2=0x7fd6740afd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:02.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.526+0000 7fd672ffd640 1 --2- 172.21.15.161:0/626555710 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6741554c0 0x7fd6740afd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.526+0000 7fd672ffd640 1 -- 172.21.15.161:0/626555710 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6741ebef0 con 0x7fd6740bfc70 2024-03-28T23:36:02.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.526+0000 7fd672ffd640 1 --2- 172.21.15.161:0/626555710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6740bfc70 0x7fd674149720 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fd6640098f0 tx=0x7fd6640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:02.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.526+0000 7fd65b7fe640 1 -- 172.21.15.161:0/626555710 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd664041070 con 0x7fd6740bfc70 2024-03-28T23:36:02.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.526+0000 7fd65b7fe640 1 -- 172.21.15.161:0/626555710 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd66403a990 con 0x7fd6740bfc70 2024-03-28T23:36:02.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.526+0000 7fd679547640 1 -- 172.21.15.161:0/626555710 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd674149440 con 0x7fd6740bfc70 2024-03-28T23:36:02.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.526+0000 7fd65b7fe640 1 -- 172.21.15.161:0/626555710 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd664035040 con 0x7fd6740bfc70 2024-03-28T23:36:02.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.526+0000 7fd679547640 1 -- 172.21.15.161:0/626555710 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd674154bf0 con 0x7fd6740bfc70 2024-03-28T23:36:02.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.527+0000 7fd679547640 1 -- 172.21.15.161:0/626555710 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6741eba00 con 0x7fd6740bfc70 2024-03-28T23:36:02.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.528+0000 7fd65b7fe640 1 -- 172.21.15.161:0/626555710 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd664038020 con 0x7fd6740bfc70 2024-03-28T23:36:02.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.528+0000 7fd65b7fe640 1 --2- 172.21.15.161:0/626555710 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd66006b4d0 0x7fd66006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:02.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.528+0000 7fd65b7fe640 1 -- 172.21.15.161:0/626555710 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 6101+0+0 (secure 0 0 0) 0x7fd6640abd30 con 0x7fd6740bfc70 2024-03-28T23:36:02.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.528+0000 7fd6727fc640 1 --2- 172.21.15.161:0/626555710 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd66006b4d0 0x7fd66006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:02.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.529+0000 7fd6727fc640 1 --2- 172.21.15.161:0/626555710 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd66006b4d0 0x7fd66006d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fd67406cf00 tx=0x7fd668007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:02.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.532+0000 7fd65b7fe640 1 -- 172.21.15.161:0/626555710 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd66407a460 con 0x7fd6740bfc70 2024-03-28T23:36:02.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.769+0000 7fd679547640 1 -- 172.21.15.161:0/626555710 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fd67404f4b0 con 0x7fd6740bfc70 2024-03-28T23:36:02.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.770+0000 7fd65b7fe640 1 -- 172.21.15.161:0/626555710 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) ==== 190+0+147 (secure 0 0 0) 0x7fd66407e500 con 0x7fd6740bfc70 2024-03-28T23:36:02.771 INFO:teuthology.orchestra.run.smithi161.stdout:[client.0] 2024-03-28T23:36:02.771 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQA6/QVmc1krAxAAEKsn1Tuuv9OlFZpB4PFhDg== 2024-03-28T23:36:02.771 INFO:teuthology.orchestra.run.smithi161.stdout: caps mds = "allow" 2024-03-28T23:36:02.771 INFO:teuthology.orchestra.run.smithi161.stdout: caps mon = "allow r" 2024-03-28T23:36:02.771 INFO:teuthology.orchestra.run.smithi161.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-28T23:36:02.771 INFO:teuthology.orchestra.run.smithi161.stderr:updated caps for client.0 2024-03-28T23:36:02.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.772+0000 7fd679547640 1 -- 172.21.15.161:0/626555710 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd66006b4d0 msgr2=0x7fd66006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:02.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.772+0000 7fd679547640 1 --2- 172.21.15.161:0/626555710 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd66006b4d0 0x7fd66006d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fd67406cf00 tx=0x7fd668007e80 comp rx=0 tx=0).stop 2024-03-28T23:36:02.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.772+0000 7fd679547640 1 -- 172.21.15.161:0/626555710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6740bfc70 msgr2=0x7fd674149720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:02.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.772+0000 7fd679547640 1 --2- 172.21.15.161:0/626555710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6740bfc70 0x7fd674149720 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fd6640098f0 tx=0x7fd6640046c0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.773+0000 7fd679547640 1 -- 172.21.15.161:0/626555710 shutdown_connections 2024-03-28T23:36:02.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.773+0000 7fd679547640 1 --2- 172.21.15.161:0/626555710 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd66006b4d0 0x7fd66006d990 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.773+0000 7fd679547640 1 --2- 172.21.15.161:0/626555710 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6741554c0 0x7fd6740afd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.773+0000 7fd679547640 1 --2- 172.21.15.161:0/626555710 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6740ca540 0x7fd6741546b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.773+0000 7fd679547640 1 --2- 172.21.15.161:0/626555710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6740bfc70 0x7fd674149720 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.773+0000 7fd679547640 1 -- 172.21.15.161:0/626555710 >> 172.21.15.161:0/626555710 conn(0x7fd6740ca9c0 msgr2=0x7fd6740c92d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:02.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.773+0000 7fd679547640 1 -- 172.21.15.161:0/626555710 shutdown_connections 2024-03-28T23:36:02.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.773+0000 7fd679547640 1 -- 172.21.15.161:0/626555710 wait complete. 2024-03-28T23:36:02.784 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:36:02.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.885+0000 7f029c51e640 1 Processor -- start 2024-03-28T23:36:02.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.885+0000 7f029c51e640 1 -- start start 2024-03-28T23:36:02.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.886+0000 7f029c51e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f02940aacd0 0x7f029415d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:02.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.886+0000 7f029c51e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f029414c250 con 0x7f029414be70 2024-03-28T23:36:02.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.886+0000 7f029c51e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02941529c0 con 0x7f029406ce80 2024-03-28T23:36:02.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.886+0000 7f029c51e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f02940a9dc0 con 0x7f02940aacd0 2024-03-28T23:36:02.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.886+0000 7f029a2ba640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f029414be70 0x7f02940ab1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50100/0 (socket says 172.21.15.161:50100) 2024-03-28T23:36:02.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.886+0000 7f029a2ba640 1 -- 172.21.15.161:0/4291819947 learned_addr learned my addr 172.21.15.161:0/4291819947 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:02.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.886+0000 7f029aabb640 1 --2- 172.21.15.161:0/4291819947 >> v2:172.21.15.170:3300/0 conn(0x7f02940aacd0 0x7f029415d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:02.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.886+0000 7f02992b8640 1 -- 172.21.15.161:0/4291819947 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3161827462 0 0) 0x7f029414c250 con 0x7f029414be70 2024-03-28T23:36:02.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.886+0000 7f02992b8640 1 -- 172.21.15.161:0/4291819947 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0280003520 con 0x7f029414be70 2024-03-28T23:36:02.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.886+0000 7f029aabb640 1 -- 172.21.15.161:0/4291819947 >> v1:172.21.15.170:6789/0 conn(0x7f029406ce80 legacy=0x7f02940aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:02.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.886+0000 7f029aabb640 1 -- 172.21.15.161:0/4291819947 >> v1:172.21.15.161:6789/0 conn(0x7f029414be70 legacy=0x7f02940ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:02.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.886+0000 7f029aabb640 1 -- 172.21.15.161:0/4291819947 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02940aa0a0 con 0x7f02940aacd0 2024-03-28T23:36:02.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.887+0000 7f029aabb640 1 --2- 172.21.15.161:0/4291819947 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f02940aacd0 0x7f029415d750 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f028c009d90 tx=0x7f028c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c84f12345d54adb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:02.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.887+0000 7f02992b8640 1 -- 172.21.15.161:0/4291819947 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028c039070 con 0x7f02940aacd0 2024-03-28T23:36:02.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.887+0000 7f02992b8640 1 -- 172.21.15.161:0/4291819947 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f028c02cc60 con 0x7f02940aacd0 2024-03-28T23:36:02.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.887+0000 7f02992b8640 1 -- 172.21.15.161:0/4291819947 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028c034660 con 0x7f02940aacd0 2024-03-28T23:36:02.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.888+0000 7f029c51e640 1 -- 172.21.15.161:0/4291819947 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f02940aacd0 msgr2=0x7f029415d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:02.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.888+0000 7f029c51e640 1 --2- 172.21.15.161:0/4291819947 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f02940aacd0 0x7f029415d750 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f028c009d90 tx=0x7f028c02beb0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.888+0000 7f029c51e640 1 -- 172.21.15.161:0/4291819947 shutdown_connections 2024-03-28T23:36:02.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.888+0000 7f029c51e640 1 --2- 172.21.15.161:0/4291819947 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f02940aacd0 0x7f029415d750 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.888+0000 7f029c51e640 1 -- 172.21.15.161:0/4291819947 >> 172.21.15.161:0/4291819947 conn(0x7f02940c6670 msgr2=0x7f02940c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:02.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.888+0000 7f029c51e640 1 -- 172.21.15.161:0/4291819947 shutdown_connections 2024-03-28T23:36:02.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.888+0000 7f029c51e640 1 -- 172.21.15.161:0/4291819947 wait complete. 2024-03-28T23:36:02.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.889+0000 7f029c51e640 1 Processor -- start 2024-03-28T23:36:02.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.889+0000 7f029c51e640 1 -- start start 2024-03-28T23:36:02.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.889+0000 7f029c51e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f029406ce80 0x7f029414ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:02.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.889+0000 7f029c51e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02940aacd0 0x7f029414e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:02.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.889+0000 7f029c51e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f029414be70 0x7f029414e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:02.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.889+0000 7f029c51e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f02940aba30 con 0x7f029406ce80 2024-03-28T23:36:02.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.889+0000 7f029c51e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f02940ab8b0 con 0x7f029414be70 2024-03-28T23:36:02.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.889+0000 7f029c51e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f029406b150 con 0x7f02940aacd0 2024-03-28T23:36:02.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.889+0000 7f029a2ba640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f029406ce80 0x7f029414ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:02.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.890+0000 7f029a2ba640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f029406ce80 0x7f029414ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53750/0 (socket says 172.21.15.161:53750) 2024-03-28T23:36:02.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.890+0000 7f029a2ba640 1 -- 172.21.15.161:0/1645660259 learned_addr learned my addr 172.21.15.161:0/1645660259 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:02.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.890+0000 7f029aabb640 1 --2- 172.21.15.161:0/1645660259 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f029414be70 0x7f029414e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:02.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.890+0000 7f0299ab9640 1 --2- 172.21.15.161:0/1645660259 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02940aacd0 0x7f029414e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:02.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.890+0000 7f029a2ba640 1 -- 172.21.15.161:0/1645660259 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02940aacd0 msgr2=0x7f029414e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:02.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.890+0000 7f029a2ba640 1 --2- 172.21.15.161:0/1645660259 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02940aacd0 0x7f029414e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.890+0000 7f029a2ba640 1 -- 172.21.15.161:0/1645660259 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f029414be70 msgr2=0x7f029414e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:02.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.890+0000 7f029a2ba640 1 --2- 172.21.15.161:0/1645660259 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f029414be70 0x7f029414e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:02.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.890+0000 7f029a2ba640 1 -- 172.21.15.161:0/1645660259 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f029414d9a0 con 0x7f029406ce80 2024-03-28T23:36:02.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.890+0000 7f029a2ba640 1 --2- 172.21.15.161:0/1645660259 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f029406ce80 0x7f029414ca00 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f029000c410 tx=0x7f029000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:02.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.890+0000 7f028affd640 1 -- 172.21.15.161:0/1645660259 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0290017070 con 0x7f029406ce80 2024-03-28T23:36:02.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.890+0000 7f028affd640 1 -- 172.21.15.161:0/1645660259 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0290005ce0 con 0x7f029406ce80 2024-03-28T23:36:02.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.890+0000 7f028affd640 1 -- 172.21.15.161:0/1645660259 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02900072f0 con 0x7f029406ce80 2024-03-28T23:36:02.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.890+0000 7f029c51e640 1 -- 172.21.15.161:0/1645660259 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02941555a0 con 0x7f029406ce80 2024-03-28T23:36:02.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.890+0000 7f029c51e640 1 -- 172.21.15.161:0/1645660259 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f029414cf40 con 0x7f029406ce80 2024-03-28T23:36:02.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.891+0000 7f029c51e640 1 -- 172.21.15.161:0/1645660259 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f029404f4b0 con 0x7f029406ce80 2024-03-28T23:36:02.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.892+0000 7f028affd640 1 -- 172.21.15.161:0/1645660259 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f029001a070 con 0x7f029406ce80 2024-03-28T23:36:02.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.892+0000 7f028affd640 1 --2- 172.21.15.161:0/1645660259 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f027806b4d0 0x7f027806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:02.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.892+0000 7f028affd640 1 -- 172.21.15.161:0/1645660259 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 6101+0+0 (secure 0 0 0) 0x7f0290089710 con 0x7f029406ce80 2024-03-28T23:36:02.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.892+0000 7f0299ab9640 1 --2- 172.21.15.161:0/1645660259 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f027806b4d0 0x7f027806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:02.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.892+0000 7f0299ab9640 1 --2- 172.21.15.161:0/1645660259 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f027806b4d0 0x7f027806d990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f02941ed0d0 tx=0x7f0284009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:02.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:02.896+0000 7f028affd640 1 -- 172.21.15.161:0/1645660259 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0290057f00 con 0x7f029406ce80 2024-03-28T23:36:03.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.104+0000 7f029c51e640 1 -- 172.21.15.161:0/1645660259 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02940c0ff0 con 0x7f029406ce80 2024-03-28T23:36:03.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.105+0000 7f028affd640 1 -- 172.21.15.161:0/1645660259 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) ==== 95+0+5200 (secure 0 0 0) 0x7f029005bfa0 con 0x7f029406ce80 2024-03-28T23:36:03.106 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:03.106 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":140,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7061,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4026332703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4026332703},{"type":"v1","addr":"172.21.15.161:6836","nonce":4026332703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":7070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/557522063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":557522063},{"type":"v1","addr":"172.21.15.170:6835","nonce":557522063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":8056,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/2391019771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":2391019771},{"type":"v1","addr":"172.21.15.161:6837","nonce":2391019771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:58.406403+0000","modified":"2024-03-28T23:36:00.406209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8068},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8068":{"gid":8068,"name":"b","rank":0,"incarnation":138,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6834/1393137291","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1393137291},{"type":"v1","addr":"172.21.15.170:6834","nonce":1393137291}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8068,"qdb_cluster":[8068]},"id":11}]} 2024-03-28T23:36:03.106 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 140 2024-03-28T23:36:03.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.107+0000 7f029c51e640 1 -- 172.21.15.161:0/1645660259 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f027806b4d0 msgr2=0x7f027806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:03.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.107+0000 7f029c51e640 1 --2- 172.21.15.161:0/1645660259 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f027806b4d0 0x7f027806d990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f02941ed0d0 tx=0x7f0284009040 comp rx=0 tx=0).stop 2024-03-28T23:36:03.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.107+0000 7f029c51e640 1 -- 172.21.15.161:0/1645660259 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f029406ce80 msgr2=0x7f029414ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:03.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.107+0000 7f029c51e640 1 --2- 172.21.15.161:0/1645660259 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f029406ce80 0x7f029414ca00 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f029000c410 tx=0x7f029000c8d0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.107+0000 7f029c51e640 1 -- 172.21.15.161:0/1645660259 shutdown_connections 2024-03-28T23:36:03.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.107+0000 7f029c51e640 1 --2- 172.21.15.161:0/1645660259 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f027806b4d0 0x7f027806d990 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.107+0000 7f029c51e640 1 --2- 172.21.15.161:0/1645660259 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f029414be70 0x7f029414e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.107+0000 7f029c51e640 1 --2- 172.21.15.161:0/1645660259 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02940aacd0 0x7f029414e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.107+0000 7f029c51e640 1 --2- 172.21.15.161:0/1645660259 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f029406ce80 0x7f029414ca00 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.107+0000 7f029c51e640 1 -- 172.21.15.161:0/1645660259 >> 172.21.15.161:0/1645660259 conn(0x7f02940c6670 msgr2=0x7f02940c3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:03.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.107+0000 7f029c51e640 1 -- 172.21.15.161:0/1645660259 shutdown_connections 2024-03-28T23:36:03.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.107+0000 7f029c51e640 1 -- 172.21.15.161:0/1645660259 wait complete. 2024-03-28T23:36:03.520 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 140, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:35:58.406403+0000', 'modified': '2024-03-28T23:36:00.406209+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8068}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8068': {'gid': 8068, 'name': 'b', 'rank': 0, 'incarnation': 138, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.170:6834/1393137291', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 1393137291}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 1393137291}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8068, 'qdb_cluster': [8068]} 2024-03-28T23:36:03.520 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:36:03.521 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T23:36:03.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.624+0000 7f5717781640 1 Processor -- start 2024-03-28T23:36:03.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.625+0000 7f5717781640 1 -- start start 2024-03-28T23:36:03.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.625+0000 7f5717781640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f571006ce80 0x7f57100aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.625+0000 7f5717781640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f571014c250 con 0x7f571014be70 2024-03-28T23:36:03.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.625+0000 7f5717781640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57101529c0 con 0x7f57100a9680 2024-03-28T23:36:03.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.625+0000 7f5717781640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f57100a9dc0 con 0x7f571006ce80 2024-03-28T23:36:03.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.626+0000 7f571551d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f571014be70 0x7f57100ab1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50108/0 (socket says 172.21.15.161:50108) 2024-03-28T23:36:03.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.626+0000 7f571551d640 1 -- 172.21.15.161:0/3568750337 learned_addr learned my addr 172.21.15.161:0/3568750337 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:03.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.626+0000 7f5714d1c640 1 --2- 172.21.15.161:0/3568750337 >> v2:172.21.15.170:3300/0 conn(0x7f571006ce80 0x7f57100aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.626+0000 7f5707fff640 1 -- 172.21.15.161:0/3568750337 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3313436893 0 0) 0x7f571014c250 con 0x7f571014be70 2024-03-28T23:36:03.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.626+0000 7f5707fff640 1 -- 172.21.15.161:0/3568750337 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56fc003520 con 0x7f571014be70 2024-03-28T23:36:03.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.626+0000 7f5714d1c640 1 -- 172.21.15.161:0/3568750337 >> v1:172.21.15.170:6789/0 conn(0x7f57100a9680 legacy=0x7f571015c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:03.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.626+0000 7f5707fff640 1 -- 172.21.15.161:0/3568750337 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3530820641 0 0) 0x7f57101529c0 con 0x7f57100a9680 2024-03-28T23:36:03.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.626+0000 7f5714d1c640 1 -- 172.21.15.161:0/3568750337 >> v1:172.21.15.161:6789/0 conn(0x7f571014be70 legacy=0x7f57100ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:03.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.626+0000 7f5714d1c640 1 -- 172.21.15.161:0/3568750337 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57100aa0a0 con 0x7f571006ce80 2024-03-28T23:36:03.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.627+0000 7f5714d1c640 1 --2- 172.21.15.161:0/3568750337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f571006ce80 0x7f57100aa5c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f570c0097a0 tx=0x7f570c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=457334ff876d5e31 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:03.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.627+0000 7f5707fff640 1 -- 172.21.15.161:0/3568750337 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c035020 con 0x7f571006ce80 2024-03-28T23:36:03.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.627+0000 7f5707fff640 1 -- 172.21.15.161:0/3568750337 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f570c031830 con 0x7f571006ce80 2024-03-28T23:36:03.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.627+0000 7f5707fff640 1 -- 172.21.15.161:0/3568750337 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c031c00 con 0x7f571006ce80 2024-03-28T23:36:03.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.627+0000 7f5717781640 1 -- 172.21.15.161:0/3568750337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f571006ce80 msgr2=0x7f57100aa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:03.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.627+0000 7f5717781640 1 --2- 172.21.15.161:0/3568750337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f571006ce80 0x7f57100aa5c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f570c0097a0 tx=0x7f570c02b8d0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.627+0000 7f5717781640 1 -- 172.21.15.161:0/3568750337 shutdown_connections 2024-03-28T23:36:03.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.627+0000 7f5717781640 1 --2- 172.21.15.161:0/3568750337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f571006ce80 0x7f57100aa5c0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.627+0000 7f5717781640 1 -- 172.21.15.161:0/3568750337 >> 172.21.15.161:0/3568750337 conn(0x7f57100c6670 msgr2=0x7f57100c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:03.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.628+0000 7f5717781640 1 -- 172.21.15.161:0/3568750337 shutdown_connections 2024-03-28T23:36:03.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.628+0000 7f5717781640 1 -- 172.21.15.161:0/3568750337 wait complete. 2024-03-28T23:36:03.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.629+0000 7f5717781640 1 Processor -- start 2024-03-28T23:36:03.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.629+0000 7f5717781640 1 -- start start 2024-03-28T23:36:03.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.629+0000 7f5717781640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f571006ce80 0x7f57101ead80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.629+0000 7f5717781640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57100a9680 0x7f57101456e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.629+0000 7f5717781640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f571014be70 0x7f5710145c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.629+0000 7f5717781640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f57100aba30 con 0x7f57100a9680 2024-03-28T23:36:03.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.629+0000 7f5717781640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f57100ab8b0 con 0x7f571006ce80 2024-03-28T23:36:03.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.629+0000 7f5717781640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f571006b150 con 0x7f571014be70 2024-03-28T23:36:03.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.629+0000 7f5714d1c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57100a9680 0x7f57101456e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.629+0000 7f5714d1c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57100a9680 0x7f57101456e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53754/0 (socket says 172.21.15.161:53754) 2024-03-28T23:36:03.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.630+0000 7f5714d1c640 1 -- 172.21.15.161:0/3292011080 learned_addr learned my addr 172.21.15.161:0/3292011080 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:03.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.630+0000 7f571551d640 1 --2- 172.21.15.161:0/3292011080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f571006ce80 0x7f57101ead80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.630+0000 7f5715d1e640 1 --2- 172.21.15.161:0/3292011080 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f571014be70 0x7f5710145c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.630+0000 7f571551d640 1 -- 172.21.15.161:0/3292011080 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f571014be70 msgr2=0x7f5710145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:03.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.630+0000 7f571551d640 1 --2- 172.21.15.161:0/3292011080 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f571014be70 0x7f5710145c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.630+0000 7f571551d640 1 -- 172.21.15.161:0/3292011080 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57100a9680 msgr2=0x7f57101456e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:03.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.630+0000 7f571551d640 1 --2- 172.21.15.161:0/3292011080 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57100a9680 0x7f57101456e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.630+0000 7f571551d640 1 -- 172.21.15.161:0/3292011080 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5710155c80 con 0x7f571006ce80 2024-03-28T23:36:03.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.630+0000 7f5714d1c640 1 --2- 172.21.15.161:0/3292011080 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57100a9680 0x7f57101456e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:03.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.630+0000 7f571551d640 1 --2- 172.21.15.161:0/3292011080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f571006ce80 0x7f57101ead80 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f570000c390 tx=0x7f5700001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:03.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.630+0000 7f5705ffb640 1 -- 172.21.15.161:0/3292011080 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5700017070 con 0x7f571006ce80 2024-03-28T23:36:03.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.630+0000 7f5705ffb640 1 -- 172.21.15.161:0/3292011080 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5700005ce0 con 0x7f571006ce80 2024-03-28T23:36:03.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.630+0000 7f5705ffb640 1 -- 172.21.15.161:0/3292011080 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57000072f0 con 0x7f571006ce80 2024-03-28T23:36:03.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.630+0000 7f5717781640 1 -- 172.21.15.161:0/3292011080 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57101db5c0 con 0x7f571006ce80 2024-03-28T23:36:03.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.630+0000 7f5717781640 1 -- 172.21.15.161:0/3292011080 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57101eb5c0 con 0x7f571006ce80 2024-03-28T23:36:03.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.632+0000 7f5705ffb640 1 -- 172.21.15.161:0/3292011080 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f570001a070 con 0x7f571006ce80 2024-03-28T23:36:03.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.632+0000 7f5705ffb640 1 --2- 172.21.15.161:0/3292011080 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f56f406b4d0 0x7f56f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.632+0000 7f5705ffb640 1 -- 172.21.15.161:0/3292011080 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 6101+0+0 (secure 0 0 0) 0x7f57000895b0 con 0x7f571006ce80 2024-03-28T23:36:03.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.632+0000 7f5714d1c640 1 --2- 172.21.15.161:0/3292011080 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f56f406b4d0 0x7f56f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.633+0000 7f5714d1c640 1 --2- 172.21.15.161:0/3292011080 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f56f406b4d0 0x7f56f406d990 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f570c02b910 tx=0x7f570c03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:03.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.633+0000 7f571a363740 1 Processor -- start 2024-03-28T23:36:03.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.633+0000 7f571a363740 1 -- start start 2024-03-28T23:36:03.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.634+0000 7f571a363740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562b2bccddb0 0x562b2bccb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.634+0000 7f571a363740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bcce190 0x562b2bccbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.634+0000 7f571a363740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b2bccc310 0x562b2bcc84e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.634+0000 7f571a363740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x562b2bc31af0 con 0x562b2bcce190 2024-03-28T23:36:03.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.634+0000 7f571a363740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x562b2bbda9d0 con 0x562b2bccc310 2024-03-28T23:36:03.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.634+0000 7f571a363740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x562b2bc2db90 con 0x562b2bccddb0 2024-03-28T23:36:03.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.634+0000 7f5714d1c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bcce190 0x562b2bccbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.634+0000 7f571551d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562b2bccddb0 0x562b2bccb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.634+0000 7f5714d1c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bcce190 0x562b2bccbdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53758/0 (socket says 172.21.15.161:53758) 2024-03-28T23:36:03.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.634+0000 7f5714d1c640 1 -- 172.21.15.161:0/1593028964 learned_addr learned my addr 172.21.15.161:0/1593028964 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:03.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.634+0000 7f5715d1e640 1 --2- 172.21.15.161:0/1593028964 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b2bccc310 0x562b2bcc84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.634+0000 7f5714d1c640 1 -- 172.21.15.161:0/1593028964 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562b2bccddb0 msgr2=0x562b2bccb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:03.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.634+0000 7f5714d1c640 1 --2- 172.21.15.161:0/1593028964 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562b2bccddb0 0x562b2bccb890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.634+0000 7f5714d1c640 1 -- 172.21.15.161:0/1593028964 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b2bccc310 msgr2=0x562b2bcc84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:03.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.634+0000 7f5714d1c640 1 --2- 172.21.15.161:0/1593028964 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b2bccc310 0x562b2bcc84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.634+0000 7f5714d1c640 1 -- 172.21.15.161:0/1593028964 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b2b9462d0 con 0x562b2bcce190 2024-03-28T23:36:03.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.634+0000 7f571551d640 1 --2- 172.21.15.161:0/1593028964 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562b2bccddb0 0x562b2bccb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:03.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.634+0000 7f5715d1e640 1 --2- 172.21.15.161:0/1593028964 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b2bccc310 0x562b2bcc84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:03.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.634+0000 7f5714d1c640 1 --2- 172.21.15.161:0/1593028964 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bcce190 0x562b2bccbdd0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f570c009890 tx=0x7f570c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=547bffda5fc621eb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:03.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.635+0000 7f56f2ffd640 1 -- 172.21.15.161:0/1593028964 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c054070 con 0x562b2bcce190 2024-03-28T23:36:03.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.635+0000 7f56f2ffd640 1 -- 172.21.15.161:0/1593028964 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f570c0036d0 con 0x562b2bcce190 2024-03-28T23:36:03.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.635+0000 7f56f2ffd640 1 -- 172.21.15.161:0/1593028964 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c035040 con 0x562b2bcce190 2024-03-28T23:36:03.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.635+0000 7f571a363740 1 -- 172.21.15.161:0/1593028964 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bcce190 msgr2=0x562b2bccbdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:03.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.635+0000 7f571a363740 1 --2- 172.21.15.161:0/1593028964 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bcce190 0x562b2bccbdd0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f570c009890 tx=0x7f570c030000 comp rx=0 tx=0).stop 2024-03-28T23:36:03.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.635+0000 7f571a363740 1 -- 172.21.15.161:0/1593028964 shutdown_connections 2024-03-28T23:36:03.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.635+0000 7f571a363740 1 --2- 172.21.15.161:0/1593028964 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bcce190 0x562b2bccbdd0 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.635+0000 7f571a363740 1 --2- 172.21.15.161:0/1593028964 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b2bccc310 0x562b2bcc84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.635+0000 7f571a363740 1 --2- 172.21.15.161:0/1593028964 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562b2bccddb0 0x562b2bccb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.635+0000 7f571a363740 1 -- 172.21.15.161:0/1593028964 >> 172.21.15.161:0/1593028964 conn(0x562b2bcca630 msgr2=0x562b2bcef220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:03.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.635+0000 7f571a363740 1 -- 172.21.15.161:0/1593028964 shutdown_connections 2024-03-28T23:36:03.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.635+0000 7f571a363740 1 -- 172.21.15.161:0/1593028964 wait complete. 2024-03-28T23:36:03.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.636+0000 7f571a363740 1 -- start start 2024-03-28T23:36:03.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.636+0000 7f571a363740 1 Processor -- start 2024-03-28T23:36:03.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.637+0000 7f571a363740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b2bccddb0 0x562b2bd02410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.637+0000 7f571a363740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bcce190 0x562b2bd1a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.637+0000 7f571a363740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562b2bcec580 0x562b2bd099f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.637+0000 7f571a363740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x562b2bcf0090 con 0x562b2bcce190 2024-03-28T23:36:03.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.637+0000 7f571a363740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x562b2bbedf70 con 0x562b2bccddb0 2024-03-28T23:36:03.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.637+0000 7f571a363740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x562b2bcefad0 con 0x562b2bcec580 2024-03-28T23:36:03.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.637+0000 7f571a363740 10 client.0 _ll_register_callbacks cb 0x562b2b945ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:36:03.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.637+0000 7f571551d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b2bccddb0 0x562b2bd02410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.637+0000 7f5714d1c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bcce190 0x562b2bd1a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.637+0000 7f5714d1c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bcce190 0x562b2bd1a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53772/0 (socket says 172.21.15.161:53772) 2024-03-28T23:36:03.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.637+0000 7f5715d1e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562b2bcec580 0x562b2bd099f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.637+0000 7f5714d1c640 1 -- 172.21.15.161:0/3932500788 learned_addr learned my addr 172.21.15.161:0/3932500788 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:03.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.637+0000 7f571551d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b2bccddb0 0x562b2bd02410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:58120/0 (socket says 172.21.15.161:58120) 2024-03-28T23:36:03.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.637+0000 7f5714d1c640 1 -- 172.21.15.161:0/3932500788 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562b2bcec580 msgr2=0x562b2bd099f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:03.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.637+0000 7f5714d1c640 1 --2- 172.21.15.161:0/3932500788 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562b2bcec580 0x562b2bd099f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.637+0000 7f5714d1c640 1 -- 172.21.15.161:0/3932500788 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b2bccddb0 msgr2=0x562b2bd02410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:03.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.637+0000 7f5714d1c640 1 --2- 172.21.15.161:0/3932500788 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b2bccddb0 0x562b2bd02410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.637+0000 7f5714d1c640 1 -- 172.21.15.161:0/3932500788 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b2ba106b0 con 0x562b2bcce190 2024-03-28T23:36:03.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.637+0000 7f571551d640 1 --2- 172.21.15.161:0/3932500788 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b2bccddb0 0x562b2bd02410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:03.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.637+0000 7f5714d1c640 1 --2- 172.21.15.161:0/3932500788 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bcce190 0x562b2bd1a3c0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f570c000c00 tx=0x7f570c02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86c4f735bf71a88 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:03.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.637+0000 7f56ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:36:03.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.637+0000 7f56ebfff640 1 -- 172.21.15.161:0/3932500788 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c05d060 con 0x562b2bcce190 2024-03-28T23:36:03.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.637+0000 7f56ebfff640 1 -- 172.21.15.161:0/3932500788 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(152..152 src has 1..152) ==== 6101+0+0 (secure 0 0 0) 0x7f570c03b630 con 0x562b2bcce190 2024-03-28T23:36:03.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.637+0000 7f571a363740 1 -- 172.21.15.161:0/3932500788 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f570c0036d0 con 0x562b2bcce190 2024-03-28T23:36:03.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.638+0000 7f56ebfff640 1 -- 172.21.15.161:0/3932500788 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=140) ==== 24+0+0 (secure 0 0 0) 0x7f570c063050 con 0x562b2bcce190 2024-03-28T23:36:03.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.638+0000 7f571a363740 10 client.9054 fetch_fsmap learned FSMap version 140 2024-03-28T23:36:03.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.638+0000 7f571a363740 1 -- 172.21.15.161:0/3932500788 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=140,monmap=2+}) -- 0x562b2bd00e00 con 0x562b2bcce190 2024-03-28T23:36:03.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.638+0000 7f56ebfff640 1 -- 172.21.15.161:0/3932500788 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f570c0036d0 con 0x562b2bcce190 2024-03-28T23:36:03.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.638+0000 7f56ebfff640 1 -- 172.21.15.161:0/3932500788 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 140) ==== 3494+0+0 (secure 0 0 0) 0x7f570c03b830 con 0x562b2bcce190 2024-03-28T23:36:03.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.638+0000 7f571a363740 10 client.9054 fetch_fsmap finished waiting for FSMap version 140 2024-03-28T23:36:03.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.638+0000 7f571a363740 10 client.9054 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:36:03.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.638+0000 7f571a363740 20 client.9054 populate_metadata read hostname 'smithi161' 2024-03-28T23:36:03.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.638+0000 7f571a363740 1 --2- 172.21.15.161:0/3932500788 >> [v2:172.21.15.170:6832/1393137291,v1:172.21.15.170:6834/1393137291] conn(0x562b2bcf0e10 0x562b2bd28f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.638+0000 7f571a363740 4 client.9054 mds_command: new command op to 8068 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:36:03.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.638+0000 7f571a363740 1 -- 172.21.15.161:0/3932500788 --> [v2:172.21.15.170:6832/1393137291,v1:172.21.15.170:6834/1393137291] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b2bc3f030 con 0x562b2bcf0e10 2024-03-28T23:36:03.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.638+0000 7f571551d640 1 --2- 172.21.15.161:0/3932500788 >> [v2:172.21.15.170:6832/1393137291,v1:172.21.15.170:6834/1393137291] conn(0x562b2bcf0e10 0x562b2bd28f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.638+0000 7f56ebfff640 10 client.9054 ms_handle_connect on v2:172.21.15.170:6832/1393137291 2024-03-28T23:36:03.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.638+0000 7f571551d640 1 --2- 172.21.15.161:0/3932500788 >> [v2:172.21.15.170:6832/1393137291,v1:172.21.15.170:6834/1393137291] conn(0x562b2bcf0e10 0x562b2bd28f70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f96b62ea8cac6943 server_cookie=d61380eedca0fe2a in_seq=0 out_seq=0 2024-03-28T23:36:03.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.639+0000 7f56ebfff640 1 -- 172.21.15.161:0/3932500788 <== mds.0 v2:172.21.15.170:6832/1393137291 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562b2bc3f030 con 0x562b2bcf0e10 2024-03-28T23:36:03.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.639+0000 7f56ebfff640 10 client.9054 handle_command_reply: tid=0 2024-03-28T23:36:03.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.639+0000 7f571a363740 1 client.9054 shutdown 2024-03-28T23:36:03.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.639+0000 7f571a363740 1 -- 172.21.15.161:0/3932500788 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bcce190 msgr2=0x562b2bd1a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:03.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.639+0000 7f571a363740 1 --2- 172.21.15.161:0/3932500788 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bcce190 0x562b2bd1a3c0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f570c000c00 tx=0x7f570c02e000 comp rx=0 tx=0).stop 2024-03-28T23:36:03.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.639+0000 7f571a363740 1 -- 172.21.15.161:0/3932500788 shutdown_connections 2024-03-28T23:36:03.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.639+0000 7f571a363740 1 --2- 172.21.15.161:0/3932500788 >> [v2:172.21.15.170:6832/1393137291,v1:172.21.15.170:6834/1393137291] conn(0x562b2bcf0e10 0x562b2bd28f70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.639+0000 7f571a363740 1 --2- 172.21.15.161:0/3932500788 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bcce190 0x562b2bd1a3c0 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.639+0000 7f56ebfff640 0 client.9054 ms_handle_reset on v2:172.21.15.170:6832/1393137291 2024-03-28T23:36:03.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.639+0000 7f571a363740 1 --2- 172.21.15.161:0/3932500788 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562b2bcec580 0x562b2bd099f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.639+0000 7f571a363740 1 --2- 172.21.15.161:0/3932500788 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b2bccddb0 0x562b2bd02410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.639+0000 7f571a363740 1 -- 172.21.15.161:0/3932500788 >> 172.21.15.161:0/3932500788 conn(0x562b2bccc310 msgr2=0x562b2bccce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:03.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.639+0000 7f571a363740 1 -- 172.21.15.161:0/3932500788 shutdown_connections 2024-03-28T23:36:03.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.639+0000 7f571a363740 1 -- 172.21.15.161:0/3932500788 wait complete. 2024-03-28T23:36:03.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.639+0000 7f571a363740 20 client.9054 trim_cache size 0 max 16384 2024-03-28T23:36:03.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.657+0000 7f571a363740 1 Processor -- start 2024-03-28T23:36:03.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.657+0000 7f571a363740 1 -- start start 2024-03-28T23:36:03.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.658+0000 7f571a363740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b2bccddb0 0x562b2bd02310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.658+0000 7f571a363740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bcce190 0x562b2bd2dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.658+0000 7f571a363740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562b2bcec580 0x562b2bd2e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.658+0000 7f571a363740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f57101539f0 con 0x562b2bcce190 2024-03-28T23:36:03.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.658+0000 7f571a363740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f57100c0250 con 0x562b2bccddb0 2024-03-28T23:36:03.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.658+0000 7f571a363740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f57101533a0 con 0x562b2bcec580 2024-03-28T23:36:03.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.658+0000 7f5714d1c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bcce190 0x562b2bd2dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.658+0000 7f5714d1c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bcce190 0x562b2bd2dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53778/0 (socket says 172.21.15.161:53778) 2024-03-28T23:36:03.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.658+0000 7f5715d1e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562b2bcec580 0x562b2bd2e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.658+0000 7f5714d1c640 1 -- 172.21.15.161:0/3564231879 learned_addr learned my addr 172.21.15.161:0/3564231879 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:03.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.658+0000 7f571551d640 1 --2- 172.21.15.161:0/3564231879 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b2bccddb0 0x562b2bd02310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.658+0000 7f5714d1c640 1 -- 172.21.15.161:0/3564231879 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562b2bcec580 msgr2=0x562b2bd2e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:03.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.658+0000 7f5714d1c640 1 --2- 172.21.15.161:0/3564231879 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562b2bcec580 0x562b2bd2e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.658+0000 7f5714d1c640 1 -- 172.21.15.161:0/3564231879 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b2bccddb0 msgr2=0x562b2bd02310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:03.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.658+0000 7f5714d1c640 1 --2- 172.21.15.161:0/3564231879 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b2bccddb0 0x562b2bd02310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.658+0000 7f5714d1c640 1 -- 172.21.15.161:0/3564231879 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b2bd0adf0 con 0x562b2bcce190 2024-03-28T23:36:03.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.658+0000 7f571551d640 1 --2- 172.21.15.161:0/3564231879 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b2bccddb0 0x562b2bd02310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:03.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.658+0000 7f5715d1e640 1 --2- 172.21.15.161:0/3564231879 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562b2bcec580 0x562b2bd2e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:03.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.658+0000 7f5714d1c640 1 --2- 172.21.15.161:0/3564231879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bcce190 0x562b2bd2dfa0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f570c0096a0 tx=0x7f570c048070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90d3921220e9cf4d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:03.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.659+0000 7f56f27fc640 1 -- 172.21.15.161:0/3564231879 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c054070 con 0x562b2bcce190 2024-03-28T23:36:03.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.659+0000 7f56f27fc640 1 -- 172.21.15.161:0/3564231879 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f570c03be50 con 0x562b2bcce190 2024-03-28T23:36:03.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.659+0000 7f56f27fc640 1 -- 172.21.15.161:0/3564231879 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c035040 con 0x562b2bcce190 2024-03-28T23:36:03.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.659+0000 7f571a363740 1 -- 172.21.15.161:0/3564231879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bcce190 msgr2=0x562b2bd2dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:03.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.659+0000 7f571a363740 1 --2- 172.21.15.161:0/3564231879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bcce190 0x562b2bd2dfa0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f570c0096a0 tx=0x7f570c048070 comp rx=0 tx=0).stop 2024-03-28T23:36:03.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.659+0000 7f571a363740 1 -- 172.21.15.161:0/3564231879 shutdown_connections 2024-03-28T23:36:03.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.659+0000 7f571a363740 1 --2- 172.21.15.161:0/3564231879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bcce190 0x562b2bd2dfa0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.659+0000 7f571a363740 1 --2- 172.21.15.161:0/3564231879 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562b2bcec580 0x562b2bd2e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.659+0000 7f571a363740 1 --2- 172.21.15.161:0/3564231879 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b2bccddb0 0x562b2bd02310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.659+0000 7f571a363740 1 -- 172.21.15.161:0/3564231879 >> 172.21.15.161:0/3564231879 conn(0x562b2bccc310 msgr2=0x562b2bd0a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:03.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.659+0000 7f571a363740 1 -- 172.21.15.161:0/3564231879 shutdown_connections 2024-03-28T23:36:03.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.659+0000 7f571a363740 1 -- 172.21.15.161:0/3564231879 wait complete. 2024-03-28T23:36:03.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.660+0000 7f571a363740 1 -- start start 2024-03-28T23:36:03.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.660+0000 7f571a363740 1 Processor -- start 2024-03-28T23:36:03.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f571a363740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bccddb0 0x562b2bd0d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f571a363740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562b2bcce190 0x562b2bd87050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f571a363740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b2bcec580 0x562b2bd7ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f571a363740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x562b2bd1ac60 con 0x562b2bccddb0 2024-03-28T23:36:03.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f571a363740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x562b2bcf0090 con 0x562b2bcec580 2024-03-28T23:36:03.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f571a363740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x562b2bd1a900 con 0x562b2bcce190 2024-03-28T23:36:03.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f571a363740 10 client.0 _ll_register_callbacks cb 0x7f570c052070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:36:03.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f571551d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bccddb0 0x562b2bd0d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f571551d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bccddb0 0x562b2bd0d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53790/0 (socket says 172.21.15.161:53790) 2024-03-28T23:36:03.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f571551d640 1 -- 172.21.15.161:0/2667904452 learned_addr learned my addr 172.21.15.161:0/2667904452 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:03.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f5715d1e640 1 --2- 172.21.15.161:0/2667904452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b2bcec580 0x562b2bd7ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f5714d1c640 1 --2- 172.21.15.161:0/2667904452 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562b2bcce190 0x562b2bd87050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f571551d640 1 -- 172.21.15.161:0/2667904452 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562b2bcce190 msgr2=0x562b2bd87050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:03.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f571551d640 1 --2- 172.21.15.161:0/2667904452 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562b2bcce190 0x562b2bd87050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f571551d640 1 -- 172.21.15.161:0/2667904452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b2bcec580 msgr2=0x562b2bd7ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:03.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f571551d640 1 --2- 172.21.15.161:0/2667904452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b2bcec580 0x562b2bd7ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f571551d640 1 -- 172.21.15.161:0/2667904452 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b2bd1e450 con 0x562b2bccddb0 2024-03-28T23:36:03.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f571551d640 1 --2- 172.21.15.161:0/2667904452 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bccddb0 0x562b2bd0d5a0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f57000540b0 tx=0x7f5700087040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14214559581b2eac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:03.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f56ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:36:03.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f56ebfff640 1 -- 172.21.15.161:0/2667904452 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570006bb60 con 0x562b2bccddb0 2024-03-28T23:36:03.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f56ebfff640 1 -- 172.21.15.161:0/2667904452 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(152..152 src has 1..152) ==== 6101+0+0 (secure 0 0 0) 0x7f570006bd00 con 0x562b2bccddb0 2024-03-28T23:36:03.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f571a363740 1 -- 172.21.15.161:0/2667904452 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f570c03be50 con 0x562b2bccddb0 2024-03-28T23:36:03.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f56ebfff640 1 -- 172.21.15.161:0/2667904452 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=140) ==== 24+0+0 (secure 0 0 0) 0x7f57000570a0 con 0x562b2bccddb0 2024-03-28T23:36:03.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f571a363740 10 client.9060 fetch_fsmap learned FSMap version 140 2024-03-28T23:36:03.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f571a363740 1 -- 172.21.15.161:0/2667904452 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=140,monmap=2+}) -- 0x562b2bd02ad0 con 0x562b2bccddb0 2024-03-28T23:36:03.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.661+0000 7f56ebfff640 1 -- 172.21.15.161:0/2667904452 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f570c03be50 con 0x562b2bccddb0 2024-03-28T23:36:03.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.662+0000 7f56ebfff640 1 -- 172.21.15.161:0/2667904452 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 140) ==== 3494+0+0 (secure 0 0 0) 0x7f5700057240 con 0x562b2bccddb0 2024-03-28T23:36:03.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.662+0000 7f571a363740 10 client.9060 fetch_fsmap finished waiting for FSMap version 140 2024-03-28T23:36:03.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.662+0000 7f571a363740 10 client.9060 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:36:03.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.662+0000 7f571a363740 20 client.9060 populate_metadata read hostname 'smithi161' 2024-03-28T23:36:03.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.662+0000 7f571a363740 1 --2- 172.21.15.161:0/2667904452 >> [v2:172.21.15.170:6832/1393137291,v1:172.21.15.170:6834/1393137291] conn(0x562b2bccc950 0x562b2bcccd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.662+0000 7f571a363740 4 client.9060 mds_command: new command op to 8068 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:36:03.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.662+0000 7f571a363740 1 -- 172.21.15.161:0/2667904452 --> [v2:172.21.15.170:6832/1393137291,v1:172.21.15.170:6834/1393137291] -- command(tid 0: {"prefix": "status"}) -- 0x562b2bc39380 con 0x562b2bccc950 2024-03-28T23:36:03.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.662+0000 7f5714d1c640 1 --2- 172.21.15.161:0/2667904452 >> [v2:172.21.15.170:6832/1393137291,v1:172.21.15.170:6834/1393137291] conn(0x562b2bccc950 0x562b2bcccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.662+0000 7f56ebfff640 10 client.9060 ms_handle_connect on v2:172.21.15.170:6832/1393137291 2024-03-28T23:36:03.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.662+0000 7f5714d1c640 1 --2- 172.21.15.161:0/2667904452 >> [v2:172.21.15.170:6832/1393137291,v1:172.21.15.170:6834/1393137291] conn(0x562b2bccc950 0x562b2bcccd30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35e7793507efe2e9 server_cookie=a15a107eba7c32c3 in_seq=0 out_seq=0 2024-03-28T23:36:03.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.662+0000 7f56ebfff640 1 -- 172.21.15.161:0/2667904452 <== mds.0 v2:172.21.15.170:6832/1393137291 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x562b2bc39380 con 0x562b2bccc950 2024-03-28T23:36:03.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.662+0000 7f56ebfff640 10 client.9060 handle_command_reply: tid=0 2024-03-28T23:36:03.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.662+0000 7f571a363740 1 client.9060 shutdown 2024-03-28T23:36:03.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.662+0000 7f571a363740 1 -- 172.21.15.161:0/2667904452 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bccddb0 msgr2=0x562b2bd0d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:03.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.662+0000 7f571a363740 1 --2- 172.21.15.161:0/2667904452 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bccddb0 0x562b2bd0d5a0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f57000540b0 tx=0x7f5700087040 comp rx=0 tx=0).stop 2024-03-28T23:36:03.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.662+0000 7f571a363740 1 -- 172.21.15.161:0/2667904452 shutdown_connections 2024-03-28T23:36:03.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.662+0000 7f571a363740 1 --2- 172.21.15.161:0/2667904452 >> [v2:172.21.15.170:6832/1393137291,v1:172.21.15.170:6834/1393137291] conn(0x562b2bccc950 0x562b2bcccd30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.662+0000 7f571a363740 1 --2- 172.21.15.161:0/2667904452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562b2bcec580 0x562b2bd7ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.662+0000 7f571a363740 1 --2- 172.21.15.161:0/2667904452 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562b2bcce190 0x562b2bd87050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.662+0000 7f571a363740 1 --2- 172.21.15.161:0/2667904452 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562b2bccddb0 0x562b2bd0d5a0 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.662+0000 7f56ebfff640 0 client.9060 ms_handle_reset on v2:172.21.15.170:6832/1393137291 2024-03-28T23:36:03.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.662+0000 7f571a363740 1 -- 172.21.15.161:0/2667904452 >> 172.21.15.161:0/2667904452 conn(0x562b2bccc310 msgr2=0x562b2bccb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:03.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.663+0000 7f571a363740 1 -- 172.21.15.161:0/2667904452 shutdown_connections 2024-03-28T23:36:03.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.663+0000 7f571a363740 1 -- 172.21.15.161:0/2667904452 wait complete. 2024-03-28T23:36:03.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.663+0000 7f571a363740 20 client.9060 trim_cache size 0 max 16384 2024-03-28T23:36:03.671 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:36:03.671 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:36:03.671 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:36:03.671 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 8068, 2024-03-28T23:36:03.671 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:36:03.671 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:36:03.671 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:36:03.671 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 4.6093295190000001, 2024-03-28T23:36:03.671 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 140, 2024-03-28T23:36:03.672 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 152, 2024-03-28T23:36:03.672 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 152, 2024-03-28T23:36:03.672 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 9.7030093669999999 2024-03-28T23:36:03.672 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:36:03.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.663+0000 7f5717781640 1 -- 172.21.15.161:0/3292011080 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f56f406b4d0 msgr2=0x7f56f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:03.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.663+0000 7f5717781640 1 --2- 172.21.15.161:0/3292011080 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f56f406b4d0 0x7f56f406d990 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f570c02b910 tx=0x7f570c03d040 comp rx=0 tx=0).stop 2024-03-28T23:36:03.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.663+0000 7f5717781640 1 -- 172.21.15.161:0/3292011080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f571006ce80 msgr2=0x7f57101ead80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:03.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.663+0000 7f5717781640 1 --2- 172.21.15.161:0/3292011080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f571006ce80 0x7f57101ead80 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f570000c390 tx=0x7f5700001e70 comp rx=0 tx=0).stop 2024-03-28T23:36:03.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.663+0000 7f5717781640 1 -- 172.21.15.161:0/3292011080 shutdown_connections 2024-03-28T23:36:03.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.663+0000 7f5717781640 1 --2- 172.21.15.161:0/3292011080 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f56f406b4d0 0x7f56f406d990 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.663+0000 7f5717781640 1 --2- 172.21.15.161:0/3292011080 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57100a9680 0x7f57101456e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.663+0000 7f5717781640 1 --2- 172.21.15.161:0/3292011080 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f571014be70 0x7f5710145c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.663+0000 7f5717781640 1 --2- 172.21.15.161:0/3292011080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f571006ce80 0x7f57101ead80 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.663+0000 7f5717781640 1 -- 172.21.15.161:0/3292011080 >> 172.21.15.161:0/3292011080 conn(0x7f57100c6670 msgr2=0x7f571014ad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:03.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.663+0000 7f5717781640 1 -- 172.21.15.161:0/3292011080 shutdown_connections 2024-03-28T23:36:03.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.663+0000 7f5717781640 1 -- 172.21.15.161:0/3292011080 wait complete. 2024-03-28T23:36:03.675 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:36:03.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.775+0000 7f0647f65640 1 Processor -- start 2024-03-28T23:36:03.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.775+0000 7f0647f65640 1 -- start start 2024-03-28T23:36:03.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.776+0000 7f0647f65640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0640150590 0x7f06400a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.776+0000 7f0647f65640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06400a8fc0 0x7f0640153150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.776+0000 7f0647f65640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06401566f0 con 0x7f06400aacb0 2024-03-28T23:36:03.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.776+0000 7f0647f65640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f06401569d0 con 0x7f0640150590 2024-03-28T23:36:03.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.776+0000 7f0647f65640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0640156cb0 con 0x7f06400a8fc0 2024-03-28T23:36:03.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.776+0000 7f0646502640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f06400aacb0 0x7f0640153690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50120/0 (socket says 172.21.15.161:50120) 2024-03-28T23:36:03.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.776+0000 7f0646502640 1 -- 172.21.15.161:0/4276847577 learned_addr learned my addr 172.21.15.161:0/4276847577 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:03.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.776+0000 7f0645500640 1 --2- 172.21.15.161:0/4276847577 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06400a8fc0 0x7f0640153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.776+0000 7f0645d01640 1 --2- 172.21.15.161:0/4276847577 >> v2:172.21.15.170:3300/0 conn(0x7f0640150590 0x7f06400a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.776+0000 7f0644cff640 1 -- 172.21.15.161:0/4276847577 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3707429257 0 0) 0x7f06401566f0 con 0x7f06400aacb0 2024-03-28T23:36:03.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.776+0000 7f0645500640 1 -- 172.21.15.161:0/4276847577 >> v2:172.21.15.170:3300/0 conn(0x7f0640150590 msgr2=0x7f06400a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:03.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.776+0000 7f0645500640 1 --2- 172.21.15.161:0/4276847577 >> v2:172.21.15.170:3300/0 conn(0x7f0640150590 0x7f06400a9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.776+0000 7f0645500640 1 -- 172.21.15.161:0/4276847577 >> v1:172.21.15.161:6789/0 conn(0x7f06400aacb0 legacy=0x7f0640153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:03.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.776+0000 7f0645500640 1 -- 172.21.15.161:0/4276847577 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0640156f90 con 0x7f06400a8fc0 2024-03-28T23:36:03.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.776+0000 7f0645d01640 1 --2- 172.21.15.161:0/4276847577 >> v2:172.21.15.170:3300/0 conn(0x7f0640150590 0x7f06400a9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:03.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.777+0000 7f0645500640 1 --2- 172.21.15.161:0/4276847577 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06400a8fc0 0x7f0640153150 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f06300097a0 tx=0x7f063002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10f8f78acea4023e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:03.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.777+0000 7f0644cff640 1 -- 172.21.15.161:0/4276847577 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0630038070 con 0x7f06400a8fc0 2024-03-28T23:36:03.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.777+0000 7f0644cff640 1 -- 172.21.15.161:0/4276847577 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0630031700 con 0x7f06400a8fc0 2024-03-28T23:36:03.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.777+0000 7f0644cff640 1 -- 172.21.15.161:0/4276847577 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0630031ad0 con 0x7f06400a8fc0 2024-03-28T23:36:03.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.778+0000 7f0647f65640 1 -- 172.21.15.161:0/4276847577 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06400a8fc0 msgr2=0x7f0640153150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:03.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.778+0000 7f0647f65640 1 --2- 172.21.15.161:0/4276847577 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06400a8fc0 0x7f0640153150 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f06300097a0 tx=0x7f063002b7a0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.778+0000 7f0647f65640 1 -- 172.21.15.161:0/4276847577 shutdown_connections 2024-03-28T23:36:03.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.778+0000 7f0647f65640 1 --2- 172.21.15.161:0/4276847577 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06400a8fc0 0x7f0640153150 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.778+0000 7f0647f65640 1 --2- 172.21.15.161:0/4276847577 >> v2:172.21.15.170:3300/0 conn(0x7f0640150590 0x7f06400a9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.778+0000 7f0647f65640 1 -- 172.21.15.161:0/4276847577 >> 172.21.15.161:0/4276847577 conn(0x7f06400c6670 msgr2=0x7f06400c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:03.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.778+0000 7f0647f65640 1 -- 172.21.15.161:0/4276847577 shutdown_connections 2024-03-28T23:36:03.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.778+0000 7f0647f65640 1 -- 172.21.15.161:0/4276847577 wait complete. 2024-03-28T23:36:03.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.779+0000 7f0647f65640 1 Processor -- start 2024-03-28T23:36:03.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.779+0000 7f0647f65640 1 -- start start 2024-03-28T23:36:03.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.780+0000 7f0647f65640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06400a8fc0 0x7f0640154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.780+0000 7f0647f65640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06400aacb0 0x7f0640141e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.780+0000 7f0647f65640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0640150590 0x7f06401423c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.780+0000 7f0647f65640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0640150970 con 0x7f0640150590 2024-03-28T23:36:03.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.780+0000 7f0647f65640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f06400aa9c0 con 0x7f06400aacb0 2024-03-28T23:36:03.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.780+0000 7f0647f65640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f06400aa480 con 0x7f06400a8fc0 2024-03-28T23:36:03.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.780+0000 7f0646502640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0640150590 0x7f06401423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.780+0000 7f0646502640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0640150590 0x7f06401423c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53802/0 (socket says 172.21.15.161:53802) 2024-03-28T23:36:03.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.780+0000 7f0646502640 1 -- 172.21.15.161:0/3936161380 learned_addr learned my addr 172.21.15.161:0/3936161380 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:03.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.780+0000 7f0645d01640 1 --2- 172.21.15.161:0/3936161380 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06400a8fc0 0x7f0640154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.780+0000 7f0645500640 1 --2- 172.21.15.161:0/3936161380 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06400aacb0 0x7f0640141e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.780+0000 7f0646502640 1 -- 172.21.15.161:0/3936161380 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06400a8fc0 msgr2=0x7f0640154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:03.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.780+0000 7f0646502640 1 --2- 172.21.15.161:0/3936161380 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06400a8fc0 0x7f0640154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.780+0000 7f0646502640 1 -- 172.21.15.161:0/3936161380 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06400aacb0 msgr2=0x7f0640141e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:03.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.780+0000 7f0646502640 1 --2- 172.21.15.161:0/3936161380 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06400aacb0 0x7f0640141e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.780+0000 7f0646502640 1 -- 172.21.15.161:0/3936161380 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06400c4220 con 0x7f0640150590 2024-03-28T23:36:03.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.780+0000 7f0645500640 1 --2- 172.21.15.161:0/3936161380 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06400aacb0 0x7f0640141e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:03.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.780+0000 7f0646502640 1 --2- 172.21.15.161:0/3936161380 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0640150590 0x7f06401423c0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f063800cb20 tx=0x7f063800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:03.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.780+0000 7f06367fc640 1 -- 172.21.15.161:0/3936161380 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063800aca0 con 0x7f0640150590 2024-03-28T23:36:03.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.780+0000 7f06367fc640 1 -- 172.21.15.161:0/3936161380 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0638012400 con 0x7f0640150590 2024-03-28T23:36:03.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.780+0000 7f0647f65640 1 -- 172.21.15.161:0/3936161380 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0640142c20 con 0x7f0640150590 2024-03-28T23:36:03.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.781+0000 7f06367fc640 1 -- 172.21.15.161:0/3936161380 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063800ae60 con 0x7f0640150590 2024-03-28T23:36:03.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.781+0000 7f0647f65640 1 -- 172.21.15.161:0/3936161380 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f064014ee70 con 0x7f0640150590 2024-03-28T23:36:03.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.782+0000 7f0647f65640 1 -- 172.21.15.161:0/3936161380 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f064004f4b0 con 0x7f0640150590 2024-03-28T23:36:03.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.782+0000 7f06367fc640 1 -- 172.21.15.161:0/3936161380 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f06380127f0 con 0x7f0640150590 2024-03-28T23:36:03.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.782+0000 7f06367fc640 1 --2- 172.21.15.161:0/3936161380 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f062406b4d0 0x7f062406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:03.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.783+0000 7f0645d01640 1 --2- 172.21.15.161:0/3936161380 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f062406b4d0 0x7f062406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:03.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.783+0000 7f06367fc640 1 -- 172.21.15.161:0/3936161380 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 6101+0+0 (secure 0 0 0) 0x7f063808a940 con 0x7f0640150590 2024-03-28T23:36:03.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.783+0000 7f0645d01640 1 --2- 172.21.15.161:0/3936161380 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f062406b4d0 0x7f062406d990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f063c009560 tx=0x7f063c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:03.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.787+0000 7f06367fc640 1 -- 172.21.15.161:0/3936161380 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0638058f60 con 0x7f0640150590 2024-03-28T23:36:03.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.958+0000 7f0647f65640 1 -- 172.21.15.161:0/3936161380 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f06400a9d20 con 0x7f0640150590 2024-03-28T23:36:03.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.959+0000 7f06367fc640 1 -- 172.21.15.161:0/3936161380 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) ==== 74+0+16920 (secure 0 0 0) 0x7f063805d000 con 0x7f0640150590 2024-03-28T23:36:03.960 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:03.960 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":152,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:36:00.062710+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:35:56.687416+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-28T23:35:57.379120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:36:03.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.961+0000 7f0647f65640 1 -- 172.21.15.161:0/3936161380 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f062406b4d0 msgr2=0x7f062406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:03.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.961+0000 7f0647f65640 1 --2- 172.21.15.161:0/3936161380 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f062406b4d0 0x7f062406d990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f063c009560 tx=0x7f063c009190 comp rx=0 tx=0).stop 2024-03-28T23:36:03.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.961+0000 7f0647f65640 1 -- 172.21.15.161:0/3936161380 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0640150590 msgr2=0x7f06401423c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:03.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.961+0000 7f0647f65640 1 --2- 172.21.15.161:0/3936161380 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0640150590 0x7f06401423c0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f063800cb20 tx=0x7f063800a060 comp rx=0 tx=0).stop 2024-03-28T23:36:03.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.961+0000 7f0647f65640 1 -- 172.21.15.161:0/3936161380 shutdown_connections 2024-03-28T23:36:03.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.961+0000 7f0647f65640 1 --2- 172.21.15.161:0/3936161380 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f062406b4d0 0x7f062406d990 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.961+0000 7f0647f65640 1 --2- 172.21.15.161:0/3936161380 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0640150590 0x7f06401423c0 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.961+0000 7f0647f65640 1 --2- 172.21.15.161:0/3936161380 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06400aacb0 0x7f0640141e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.961+0000 7f0647f65640 1 --2- 172.21.15.161:0/3936161380 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06400a8fc0 0x7f0640154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:03.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.961+0000 7f0647f65640 1 -- 172.21.15.161:0/3936161380 >> 172.21.15.161:0/3936161380 conn(0x7f06400c6670 msgr2=0x7f06400c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:03.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.961+0000 7f0647f65640 1 -- 172.21.15.161:0/3936161380 shutdown_connections 2024-03-28T23:36:03.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:03.961+0000 7f0647f65640 1 -- 172.21.15.161:0/3936161380 wait complete. 2024-03-28T23:36:03.973 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:35:56.687416+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '151', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:35:57.379120+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '152', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:36:03.973 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:36:04.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.076+0000 7f2700fbc640 1 Processor -- start 2024-03-28T23:36:04.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.076+0000 7f2700fbc640 1 -- start start 2024-03-28T23:36:04.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.077+0000 7f2700fbc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f26fc1529d0 0x7f26fc06bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:04.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.077+0000 7f2700fbc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26fc0c6750 0x7f26fc0c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:04.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.077+0000 7f2700fbc640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26fc152db0 con 0x7f26fc155c30 2024-03-28T23:36:04.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.077+0000 7f2700fbc640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f26fc0c7210 con 0x7f26fc1529d0 2024-03-28T23:36:04.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.077+0000 7f2700fbc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f26fc0c6c50 con 0x7f26fc0c6750 2024-03-28T23:36:04.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.077+0000 7f26fad9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f26fc155c30 0x7f26fc0c54f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50122/0 (socket says 172.21.15.161:50122) 2024-03-28T23:36:04.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.077+0000 7f26fad9d640 1 -- 172.21.15.161:0/3330419007 learned_addr learned my addr 172.21.15.161:0/3330419007 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:04.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.077+0000 7f26fa59c640 1 --2- 172.21.15.161:0/3330419007 >> v2:172.21.15.170:3300/0 conn(0x7f26fc1529d0 0x7f26fc06bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:04.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.077+0000 7f26f9d9b640 1 --2- 172.21.15.161:0/3330419007 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26fc0c6750 0x7f26fc0c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:04.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.077+0000 7f26fa59c640 1 -- 172.21.15.161:0/3330419007 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26fc0c6750 msgr2=0x7f26fc0c88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:04.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.077+0000 7f26fa59c640 1 --2- 172.21.15.161:0/3330419007 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26fc0c6750 0x7f26fc0c88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.077+0000 7f26fa59c640 1 -- 172.21.15.161:0/3330419007 >> v1:172.21.15.161:6789/0 conn(0x7f26fc155c30 legacy=0x7f26fc0c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:04.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.078+0000 7f26fa59c640 1 -- 172.21.15.161:0/3330419007 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26fc0c6f30 con 0x7f26fc1529d0 2024-03-28T23:36:04.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.078+0000 7f26fa59c640 1 --2- 172.21.15.161:0/3330419007 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26fc1529d0 0x7f26fc06bb60 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0097a0 tx=0x7f26ec02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c365f15da600c8a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:04.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.079+0000 7f26f959a640 1 -- 172.21.15.161:0/3330419007 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec035020 con 0x7f26fc1529d0 2024-03-28T23:36:04.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.079+0000 7f26f959a640 1 -- 172.21.15.161:0/3330419007 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26ec031830 con 0x7f26fc1529d0 2024-03-28T23:36:04.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.079+0000 7f26f959a640 1 -- 172.21.15.161:0/3330419007 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec031c00 con 0x7f26fc1529d0 2024-03-28T23:36:04.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.080+0000 7f2700fbc640 1 -- 172.21.15.161:0/3330419007 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26fc1529d0 msgr2=0x7f26fc06bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:04.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.080+0000 7f2700fbc640 1 --2- 172.21.15.161:0/3330419007 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26fc1529d0 0x7f26fc06bb60 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0097a0 tx=0x7f26ec02b8d0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.080+0000 7f2700fbc640 1 -- 172.21.15.161:0/3330419007 shutdown_connections 2024-03-28T23:36:04.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.080+0000 7f2700fbc640 1 --2- 172.21.15.161:0/3330419007 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26fc0c6750 0x7f26fc0c88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.080+0000 7f2700fbc640 1 --2- 172.21.15.161:0/3330419007 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26fc1529d0 0x7f26fc06bb60 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.080+0000 7f2700fbc640 1 -- 172.21.15.161:0/3330419007 >> 172.21.15.161:0/3330419007 conn(0x7f26fc0c7430 msgr2=0x7f26fc0c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:04.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.080+0000 7f2700fbc640 1 -- 172.21.15.161:0/3330419007 shutdown_connections 2024-03-28T23:36:04.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.080+0000 7f2700fbc640 1 -- 172.21.15.161:0/3330419007 wait complete. 2024-03-28T23:36:04.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.081+0000 7f2700fbc640 1 Processor -- start 2024-03-28T23:36:04.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.081+0000 7f2700fbc640 1 -- start start 2024-03-28T23:36:04.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.082+0000 7f2700fbc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26fc0c6750 0x7f26fc0c4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:04.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.082+0000 7f2700fbc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26fc1529d0 0x7f26fc1dfed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:04.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.082+0000 7f2700fbc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26fc155c30 0x7f26fc1defb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:04.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.082+0000 7f2700fbc640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc0aa160 con 0x7f26fc0c6750 2024-03-28T23:36:04.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.082+0000 7f2700fbc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc155710 con 0x7f26fc1529d0 2024-03-28T23:36:04.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.082+0000 7f2700fbc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f26fc0c5c00 con 0x7f26fc155c30 2024-03-28T23:36:04.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.082+0000 7f26fa59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26fc0c6750 0x7f26fc0c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:04.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.082+0000 7f26fa59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26fc0c6750 0x7f26fc0c4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53814/0 (socket says 172.21.15.161:53814) 2024-03-28T23:36:04.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.082+0000 7f26fa59c640 1 -- 172.21.15.161:0/1062183614 learned_addr learned my addr 172.21.15.161:0/1062183614 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:04.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.082+0000 7f26f9d9b640 1 --2- 172.21.15.161:0/1062183614 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26fc1529d0 0x7f26fc1dfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:04.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.082+0000 7f26fad9d640 1 --2- 172.21.15.161:0/1062183614 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26fc155c30 0x7f26fc1defb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:04.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.082+0000 7f26fa59c640 1 -- 172.21.15.161:0/1062183614 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26fc155c30 msgr2=0x7f26fc1defb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:04.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.082+0000 7f26fa59c640 1 --2- 172.21.15.161:0/1062183614 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26fc155c30 0x7f26fc1defb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.082+0000 7f26fa59c640 1 -- 172.21.15.161:0/1062183614 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26fc1529d0 msgr2=0x7f26fc1dfed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:04.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.082+0000 7f26fa59c640 1 --2- 172.21.15.161:0/1062183614 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26fc1529d0 0x7f26fc1dfed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.082+0000 7f26fa59c640 1 -- 172.21.15.161:0/1062183614 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26fc154300 con 0x7f26fc0c6750 2024-03-28T23:36:04.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.083+0000 7f26fa59c640 1 --2- 172.21.15.161:0/1062183614 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26fc0c6750 0x7f26fc0c4410 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f26ec02bee0 tx=0x7f26ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:04.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.083+0000 7f26eaffd640 1 -- 172.21.15.161:0/1062183614 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec035040 con 0x7f26fc0c6750 2024-03-28T23:36:04.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.083+0000 7f26eaffd640 1 -- 172.21.15.161:0/1062183614 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26ec03b920 con 0x7f26fc0c6750 2024-03-28T23:36:04.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.083+0000 7f26eaffd640 1 -- 172.21.15.161:0/1062183614 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec03bd10 con 0x7f26fc0c6750 2024-03-28T23:36:04.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.083+0000 7f2700fbc640 1 -- 172.21.15.161:0/1062183614 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26fc149d90 con 0x7f26fc0c6750 2024-03-28T23:36:04.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.083+0000 7f2700fbc640 1 -- 172.21.15.161:0/1062183614 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26fc1545c0 con 0x7f26fc0c6750 2024-03-28T23:36:04.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.084+0000 7f2700fbc640 1 -- 172.21.15.161:0/1062183614 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26fc06d720 con 0x7f26fc0c6750 2024-03-28T23:36:04.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.084+0000 7f26eaffd640 1 -- 172.21.15.161:0/1062183614 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f26ec038020 con 0x7f26fc0c6750 2024-03-28T23:36:04.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.085+0000 7f26eaffd640 1 --2- 172.21.15.161:0/1062183614 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f26dc06b4d0 0x7f26dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:04.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.085+0000 7f26eaffd640 1 -- 172.21.15.161:0/1062183614 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 6101+0+0 (secure 0 0 0) 0x7f26ec0ac3c0 con 0x7f26fc0c6750 2024-03-28T23:36:04.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.085+0000 7f26f9d9b640 1 --2- 172.21.15.161:0/1062183614 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f26dc06b4d0 0x7f26dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:04.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.085+0000 7f26f9d9b640 1 --2- 172.21.15.161:0/1062183614 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f26dc06b4d0 0x7f26dc06d990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f26fc1531e0 tx=0x7f26f0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:04.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.089+0000 7f26eaffd640 1 -- 172.21.15.161:0/1062183614 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f26fc06d720 con 0x7f26fc0c6750 2024-03-28T23:36:04.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.263+0000 7f2700fbc640 1 -- 172.21.15.161:0/1062183614 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f26fc04f4b0 con 0x7f26fc0c6750 2024-03-28T23:36:04.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.263+0000 7f26eaffd640 1 -- 172.21.15.161:0/1062183614 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v152) ==== 93+0+10 (secure 0 0 0) 0x7f26ec031800 con 0x7f26fc0c6750 2024-03-28T23:36:04.264 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:36:04.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.265+0000 7f2700fbc640 1 -- 172.21.15.161:0/1062183614 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f26dc06b4d0 msgr2=0x7f26dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:04.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.265+0000 7f2700fbc640 1 --2- 172.21.15.161:0/1062183614 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f26dc06b4d0 0x7f26dc06d990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f26fc1531e0 tx=0x7f26f0009190 comp rx=0 tx=0).stop 2024-03-28T23:36:04.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.265+0000 7f2700fbc640 1 -- 172.21.15.161:0/1062183614 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26fc0c6750 msgr2=0x7f26fc0c4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:04.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.265+0000 7f2700fbc640 1 --2- 172.21.15.161:0/1062183614 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26fc0c6750 0x7f26fc0c4410 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f26ec02bee0 tx=0x7f26ec0046c0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.265+0000 7f2700fbc640 1 -- 172.21.15.161:0/1062183614 shutdown_connections 2024-03-28T23:36:04.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.265+0000 7f2700fbc640 1 --2- 172.21.15.161:0/1062183614 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f26dc06b4d0 0x7f26dc06d990 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.265+0000 7f2700fbc640 1 --2- 172.21.15.161:0/1062183614 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26fc155c30 0x7f26fc1defb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.265+0000 7f2700fbc640 1 --2- 172.21.15.161:0/1062183614 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26fc1529d0 0x7f26fc1dfed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.265+0000 7f2700fbc640 1 --2- 172.21.15.161:0/1062183614 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26fc0c6750 0x7f26fc0c4410 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.265+0000 7f2700fbc640 1 -- 172.21.15.161:0/1062183614 >> 172.21.15.161:0/1062183614 conn(0x7f26fc0c7430 msgr2=0x7f26fc1e19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:04.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.265+0000 7f2700fbc640 1 -- 172.21.15.161:0/1062183614 shutdown_connections 2024-03-28T23:36:04.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.265+0000 7f2700fbc640 1 -- 172.21.15.161:0/1062183614 wait complete. 2024-03-28T23:36:04.276 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:36:04.337 INFO:tasks.ceph.mds.b.smithi170.stderr:2024-03-28T23:36:04.337+0000 7f8d22ab5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:36:04.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.380+0000 7f34b95c2640 1 Processor -- start 2024-03-28T23:36:04.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.380+0000 7f34b95c2640 1 -- start start 2024-03-28T23:36:04.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.381+0000 7f34b95c2640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f34b40c4e40 0x7f34b40c8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:04.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.381+0000 7f34b95c2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34b4072fe0 0x7f34b40c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:04.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.381+0000 7f34b95c2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f34b40c5880 0x7f34b4158860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:04.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.381+0000 7f34b95c2640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f34b4156060 con 0x7f34b40c5880 2024-03-28T23:36:04.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.381+0000 7f34b95c2640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f34b4072870 con 0x7f34b40c4e40 2024-03-28T23:36:04.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.381+0000 7f34b95c2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f34b4072b50 con 0x7f34b4072fe0 2024-03-28T23:36:04.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.381+0000 7f34b2ffd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f34b40c4e40 0x7f34b40c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:04.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.381+0000 7f34b37fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f34b40c5880 0x7f34b4158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:04.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.381+0000 7f34b2ffd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f34b40c4e40 0x7f34b40c8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:58184/0 (socket says 172.21.15.161:58184) 2024-03-28T23:36:04.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.381+0000 7f34b2ffd640 1 -- 172.21.15.161:0/2921045381 learned_addr learned my addr 172.21.15.161:0/2921045381 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:04.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.381+0000 7f34b27fc640 1 --2- 172.21.15.161:0/2921045381 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34b4072fe0 0x7f34b40c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:04.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.381+0000 7f34b37fe640 1 -- 172.21.15.161:0/2921045381 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34b4072fe0 msgr2=0x7f34b40c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:04.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.381+0000 7f34b37fe640 1 --2- 172.21.15.161:0/2921045381 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34b4072fe0 0x7f34b40c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.381+0000 7f34b37fe640 1 -- 172.21.15.161:0/2921045381 >> v2:172.21.15.170:3300/0 conn(0x7f34b40c4e40 msgr2=0x7f34b40c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:04.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.381+0000 7f34b37fe640 1 --2- 172.21.15.161:0/2921045381 >> v2:172.21.15.170:3300/0 conn(0x7f34b40c4e40 0x7f34b40c8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.381+0000 7f34b37fe640 1 -- 172.21.15.161:0/2921045381 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b4155cf0 con 0x7f34b40c5880 2024-03-28T23:36:04.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.381+0000 7f34b2ffd640 1 --2- 172.21.15.161:0/2921045381 >> v2:172.21.15.170:3300/0 conn(0x7f34b40c4e40 0x7f34b40c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:04.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.382+0000 7f34b37fe640 1 --2- 172.21.15.161:0/2921045381 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34b40c5880 0x7f34b4158860 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f34b40c98f0 tx=0x7f34ac02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a35b00ea49c8b664 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:04.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.382+0000 7f34b1ffb640 1 -- 172.21.15.161:0/2921045381 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac039070 con 0x7f34b40c5880 2024-03-28T23:36:04.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.382+0000 7f34b1ffb640 1 -- 172.21.15.161:0/2921045381 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34ac02cbd0 con 0x7f34b40c5880 2024-03-28T23:36:04.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.382+0000 7f34b1ffb640 1 -- 172.21.15.161:0/2921045381 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac034440 con 0x7f34b40c5880 2024-03-28T23:36:04.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.383+0000 7f34b95c2640 1 -- 172.21.15.161:0/2921045381 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34b40c5880 msgr2=0x7f34b4158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:04.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.383+0000 7f34b95c2640 1 --2- 172.21.15.161:0/2921045381 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34b40c5880 0x7f34b4158860 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f34b40c98f0 tx=0x7f34ac02be20 comp rx=0 tx=0).stop 2024-03-28T23:36:04.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.383+0000 7f34b95c2640 1 -- 172.21.15.161:0/2921045381 shutdown_connections 2024-03-28T23:36:04.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.383+0000 7f34b95c2640 1 --2- 172.21.15.161:0/2921045381 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34b40c5880 0x7f34b4158860 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.383+0000 7f34b95c2640 1 --2- 172.21.15.161:0/2921045381 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34b4072fe0 0x7f34b40c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.383+0000 7f34b95c2640 1 --2- 172.21.15.161:0/2921045381 >> v2:172.21.15.170:3300/0 conn(0x7f34b40c4e40 0x7f34b40c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.383+0000 7f34b95c2640 1 -- 172.21.15.161:0/2921045381 >> 172.21.15.161:0/2921045381 conn(0x7f34b40c88b0 msgr2=0x7f34b40c8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:04.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.383+0000 7f34b95c2640 1 -- 172.21.15.161:0/2921045381 shutdown_connections 2024-03-28T23:36:04.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.383+0000 7f34b95c2640 1 -- 172.21.15.161:0/2921045381 wait complete. 2024-03-28T23:36:04.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.384+0000 7f34b95c2640 1 Processor -- start 2024-03-28T23:36:04.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.384+0000 7f34b95c2640 1 -- start start 2024-03-28T23:36:04.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.385+0000 7f34b95c2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f34b4072fe0 0x7f34b40ad950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:04.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.385+0000 7f34b95c2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34b40c4e40 0x7f34b41eb410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:04.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.385+0000 7f34b95c2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34b40c5880 0x7f34b40ae4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:04.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.385+0000 7f34b95c2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f34b4158f20 con 0x7f34b40c4e40 2024-03-28T23:36:04.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.385+0000 7f34b95c2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f34b4158da0 con 0x7f34b4072fe0 2024-03-28T23:36:04.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.385+0000 7f34b95c2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f34b41590a0 con 0x7f34b40c5880 2024-03-28T23:36:04.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.385+0000 7f34b27fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34b40c4e40 0x7f34b41eb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:04.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.385+0000 7f34b27fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34b40c4e40 0x7f34b41eb410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53830/0 (socket says 172.21.15.161:53830) 2024-03-28T23:36:04.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.385+0000 7f34b27fc640 1 -- 172.21.15.161:0/407065678 learned_addr learned my addr 172.21.15.161:0/407065678 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:04.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.385+0000 7f34b2ffd640 1 --2- 172.21.15.161:0/407065678 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f34b4072fe0 0x7f34b40ad950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:04.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.385+0000 7f34b37fe640 1 --2- 172.21.15.161:0/407065678 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34b40c5880 0x7f34b40ae4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:04.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.385+0000 7f34b27fc640 1 -- 172.21.15.161:0/407065678 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34b40c5880 msgr2=0x7f34b40ae4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:04.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.385+0000 7f34b27fc640 1 --2- 172.21.15.161:0/407065678 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34b40c5880 0x7f34b40ae4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.385+0000 7f34b27fc640 1 -- 172.21.15.161:0/407065678 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f34b4072fe0 msgr2=0x7f34b40ad950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:04.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.385+0000 7f34b27fc640 1 --2- 172.21.15.161:0/407065678 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f34b4072fe0 0x7f34b40ad950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.385+0000 7f34b27fc640 1 -- 172.21.15.161:0/407065678 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b406d3a0 con 0x7f34b40c4e40 2024-03-28T23:36:04.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.385+0000 7f34b27fc640 1 --2- 172.21.15.161:0/407065678 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34b40c4e40 0x7f34b41eb410 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f34a800e890 tx=0x7f34a800ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:04.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.386+0000 7f349b7fe640 1 -- 172.21.15.161:0/407065678 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a8017070 con 0x7f34b40c4e40 2024-03-28T23:36:04.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.386+0000 7f349b7fe640 1 -- 172.21.15.161:0/407065678 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34a800c940 con 0x7f34b40c4e40 2024-03-28T23:36:04.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.386+0000 7f34b95c2640 1 -- 172.21.15.161:0/407065678 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34b406d680 con 0x7f34b40c4e40 2024-03-28T23:36:04.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.386+0000 7f349b7fe640 1 -- 172.21.15.161:0/407065678 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a800cd10 con 0x7f34b40c4e40 2024-03-28T23:36:04.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.386+0000 7f34b95c2640 1 -- 172.21.15.161:0/407065678 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34b406dc40 con 0x7f34b40c4e40 2024-03-28T23:36:04.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.387+0000 7f34b95c2640 1 -- 172.21.15.161:0/407065678 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34b404f4b0 con 0x7f34b40c4e40 2024-03-28T23:36:04.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.387+0000 7f349b7fe640 1 -- 172.21.15.161:0/407065678 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f34a801a070 con 0x7f34b40c4e40 2024-03-28T23:36:04.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.387+0000 7f349b7fe640 1 --2- 172.21.15.161:0/407065678 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f34a006b4d0 0x7f34a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:04.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.387+0000 7f349b7fe640 1 -- 172.21.15.161:0/407065678 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 6101+0+0 (secure 0 0 0) 0x7f34a808a210 con 0x7f34b40c4e40 2024-03-28T23:36:04.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.388+0000 7f34b2ffd640 1 --2- 172.21.15.161:0/407065678 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f34a006b4d0 0x7f34a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:04.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.388+0000 7f34b2ffd640 1 --2- 172.21.15.161:0/407065678 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f34a006b4d0 0x7f34a006d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f34a4009630 tx=0x7f34a4009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:04.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.392+0000 7f349b7fe640 1 -- 172.21.15.161:0/407065678 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f34a8058940 con 0x7f34b40c4e40 2024-03-28T23:36:04.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.562+0000 7f34b95c2640 1 -- 172.21.15.161:0/407065678 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f34b406e050 con 0x7f34b40c4e40 2024-03-28T23:36:04.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.563+0000 7f349b7fe640 1 -- 172.21.15.161:0/407065678 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v152) ==== 104+0+11 (secure 0 0 0) 0x7f34a805c9e0 con 0x7f34b40c4e40 2024-03-28T23:36:04.563 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:36:04.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.565+0000 7f34b95c2640 1 -- 172.21.15.161:0/407065678 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f34a006b4d0 msgr2=0x7f34a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:04.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.565+0000 7f34b95c2640 1 --2- 172.21.15.161:0/407065678 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f34a006b4d0 0x7f34a006d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f34a4009630 tx=0x7f34a4009240 comp rx=0 tx=0).stop 2024-03-28T23:36:04.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.565+0000 7f34b95c2640 1 -- 172.21.15.161:0/407065678 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34b40c4e40 msgr2=0x7f34b41eb410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:04.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.565+0000 7f34b95c2640 1 --2- 172.21.15.161:0/407065678 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34b40c4e40 0x7f34b41eb410 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f34a800e890 tx=0x7f34a800ed50 comp rx=0 tx=0).stop 2024-03-28T23:36:04.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.565+0000 7f34b95c2640 1 -- 172.21.15.161:0/407065678 shutdown_connections 2024-03-28T23:36:04.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.565+0000 7f34b95c2640 1 --2- 172.21.15.161:0/407065678 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f34a006b4d0 0x7f34a006d990 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.565+0000 7f34b95c2640 1 --2- 172.21.15.161:0/407065678 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34b40c4e40 0x7f34b41eb410 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.565+0000 7f34b95c2640 1 --2- 172.21.15.161:0/407065678 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34b40c5880 0x7f34b40ae4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.565+0000 7f34b95c2640 1 --2- 172.21.15.161:0/407065678 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f34b4072fe0 0x7f34b40ad950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.565+0000 7f34b95c2640 1 -- 172.21.15.161:0/407065678 >> 172.21.15.161:0/407065678 conn(0x7f34b40c88b0 msgr2=0x7f34b40c26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:04.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.565+0000 7f34b95c2640 1 -- 172.21.15.161:0/407065678 shutdown_connections 2024-03-28T23:36:04.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.565+0000 7f34b95c2640 1 -- 172.21.15.161:0/407065678 wait complete. 2024-03-28T23:36:04.576 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:36:04.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.681+0000 7fd350ba7640 1 Processor -- start 2024-03-28T23:36:04.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.681+0000 7fd350ba7640 1 -- start start 2024-03-28T23:36:04.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.681+0000 7fd350ba7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd34c06cdd0 0x7fd34c15d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:04.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.681+0000 7fd350ba7640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd34c152960 con 0x7fd34c151e70 2024-03-28T23:36:04.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.681+0000 7fd350ba7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd34c0c1e40 con 0x7fd34c0c6350 2024-03-28T23:36:04.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.681+0000 7fd350ba7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd34c0c2120 con 0x7fd34c06cdd0 2024-03-28T23:36:04.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.682+0000 7fd34a59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd34c151e70 0x7fd34c152250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50138/0 (socket says 172.21.15.161:50138) 2024-03-28T23:36:04.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.682+0000 7fd34a59c640 1 -- 172.21.15.161:0/2606509578 learned_addr learned my addr 172.21.15.161:0/2606509578 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:04.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.682+0000 7fd34ad9d640 1 --2- 172.21.15.161:0/2606509578 >> v2:172.21.15.170:3300/0 conn(0x7fd34c06cdd0 0x7fd34c15d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:04.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.682+0000 7fd34959a640 1 -- 172.21.15.161:0/2606509578 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2168480912 0 0) 0x7fd34c152960 con 0x7fd34c151e70 2024-03-28T23:36:04.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.682+0000 7fd34ad9d640 1 -- 172.21.15.161:0/2606509578 >> v1:172.21.15.170:6789/0 conn(0x7fd34c0c6350 legacy=0x7fd34c06c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:04.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.682+0000 7fd34ad9d640 1 -- 172.21.15.161:0/2606509578 >> v1:172.21.15.161:6789/0 conn(0x7fd34c151e70 legacy=0x7fd34c152250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:04.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.682+0000 7fd34ad9d640 1 -- 172.21.15.161:0/2606509578 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd34c0c2400 con 0x7fd34c06cdd0 2024-03-28T23:36:04.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.683+0000 7fd34ad9d640 1 --2- 172.21.15.161:0/2606509578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd34c06cdd0 0x7fd34c15d770 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fd344009d90 tx=0x7fd34402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6ce90af8bc269ef server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:04.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.683+0000 7fd34959a640 1 -- 172.21.15.161:0/2606509578 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd344039070 con 0x7fd34c06cdd0 2024-03-28T23:36:04.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.683+0000 7fd34959a640 1 -- 172.21.15.161:0/2606509578 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd34402cc60 con 0x7fd34c06cdd0 2024-03-28T23:36:04.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.683+0000 7fd34959a640 1 -- 172.21.15.161:0/2606509578 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd344034660 con 0x7fd34c06cdd0 2024-03-28T23:36:04.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.683+0000 7fd350ba7640 1 -- 172.21.15.161:0/2606509578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd34c06cdd0 msgr2=0x7fd34c15d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:04.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.683+0000 7fd350ba7640 1 --2- 172.21.15.161:0/2606509578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd34c06cdd0 0x7fd34c15d770 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fd344009d90 tx=0x7fd34402beb0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.683+0000 7fd350ba7640 1 -- 172.21.15.161:0/2606509578 shutdown_connections 2024-03-28T23:36:04.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.683+0000 7fd350ba7640 1 --2- 172.21.15.161:0/2606509578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd34c06cdd0 0x7fd34c15d770 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.683+0000 7fd350ba7640 1 -- 172.21.15.161:0/2606509578 >> 172.21.15.161:0/2606509578 conn(0x7fd34c0c3390 msgr2=0x7fd34c0c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:04.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.684+0000 7fd350ba7640 1 -- 172.21.15.161:0/2606509578 shutdown_connections 2024-03-28T23:36:04.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.684+0000 7fd350ba7640 1 -- 172.21.15.161:0/2606509578 wait complete. 2024-03-28T23:36:04.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.685+0000 7fd350ba7640 1 Processor -- start 2024-03-28T23:36:04.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.685+0000 7fd350ba7640 1 -- start start 2024-03-28T23:36:04.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.685+0000 7fd350ba7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd34c06cdd0 0x7fd34c1de820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:04.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.685+0000 7fd350ba7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd34c0c6350 0x7fd34c142810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:04.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.685+0000 7fd350ba7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd34c151e70 0x7fd34c1ebee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:04.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.685+0000 7fd350ba7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd34c0c2c00 con 0x7fd34c0c6350 2024-03-28T23:36:04.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.685+0000 7fd350ba7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd34c14f810 con 0x7fd34c06cdd0 2024-03-28T23:36:04.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.685+0000 7fd350ba7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd34c0c2d80 con 0x7fd34c151e70 2024-03-28T23:36:04.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.685+0000 7fd349d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd34c0c6350 0x7fd34c142810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:04.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.686+0000 7fd349d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd34c0c6350 0x7fd34c142810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53846/0 (socket says 172.21.15.161:53846) 2024-03-28T23:36:04.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.686+0000 7fd349d9b640 1 -- 172.21.15.161:0/4151250980 learned_addr learned my addr 172.21.15.161:0/4151250980 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:04.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.686+0000 7fd34a59c640 1 --2- 172.21.15.161:0/4151250980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd34c06cdd0 0x7fd34c1de820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:04.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.686+0000 7fd34ad9d640 1 --2- 172.21.15.161:0/4151250980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd34c151e70 0x7fd34c1ebee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:04.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.686+0000 7fd349d9b640 1 -- 172.21.15.161:0/4151250980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd34c151e70 msgr2=0x7fd34c1ebee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:04.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.686+0000 7fd349d9b640 1 --2- 172.21.15.161:0/4151250980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd34c151e70 0x7fd34c1ebee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.686+0000 7fd349d9b640 1 -- 172.21.15.161:0/4151250980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd34c06cdd0 msgr2=0x7fd34c1de820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:04.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.686+0000 7fd349d9b640 1 --2- 172.21.15.161:0/4151250980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd34c06cdd0 0x7fd34c1de820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.686+0000 7fd349d9b640 1 -- 172.21.15.161:0/4151250980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd34c14f040 con 0x7fd34c0c6350 2024-03-28T23:36:04.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.686+0000 7fd349d9b640 1 --2- 172.21.15.161:0/4151250980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd34c0c6350 0x7fd34c142810 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fd33c00c450 tx=0x7fd33c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:04.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.686+0000 7fd33affd640 1 -- 172.21.15.161:0/4151250980 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd33c0052d0 con 0x7fd34c0c6350 2024-03-28T23:36:04.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.686+0000 7fd350ba7640 1 -- 172.21.15.161:0/4151250980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd34c14ed60 con 0x7fd34c0c6350 2024-03-28T23:36:04.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.686+0000 7fd33affd640 1 -- 172.21.15.161:0/4151250980 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd33c005470 con 0x7fd34c0c6350 2024-03-28T23:36:04.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.686+0000 7fd33affd640 1 -- 172.21.15.161:0/4151250980 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd33c012870 con 0x7fd34c0c6350 2024-03-28T23:36:04.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.686+0000 7fd350ba7640 1 -- 172.21.15.161:0/4151250980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd34c148b30 con 0x7fd34c0c6350 2024-03-28T23:36:04.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.687+0000 7fd350ba7640 1 -- 172.21.15.161:0/4151250980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd34c04f4b0 con 0x7fd34c0c6350 2024-03-28T23:36:04.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.688+0000 7fd33affd640 1 -- 172.21.15.161:0/4151250980 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd33c012a10 con 0x7fd34c0c6350 2024-03-28T23:36:04.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.688+0000 7fd33affd640 1 --2- 172.21.15.161:0/4151250980 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd32c06b4d0 0x7fd32c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:04.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.688+0000 7fd33affd640 1 -- 172.21.15.161:0/4151250980 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 6101+0+0 (secure 0 0 0) 0x7fd33c089e20 con 0x7fd34c0c6350 2024-03-28T23:36:04.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.688+0000 7fd34a59c640 1 --2- 172.21.15.161:0/4151250980 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd32c06b4d0 0x7fd32c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:04.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.688+0000 7fd34a59c640 1 --2- 172.21.15.161:0/4151250980 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd32c06b4d0 0x7fd32c06d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd340007f70 tx=0x7fd340007f00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:04.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.692+0000 7fd33affd640 1 -- 172.21.15.161:0/4151250980 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd33c0584d0 con 0x7fd34c0c6350 2024-03-28T23:36:04.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.863+0000 7fd350ba7640 1 -- 172.21.15.161:0/4151250980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd34c0c2f80 con 0x7fd34c0c6350 2024-03-28T23:36:04.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.863+0000 7fd33affd640 1 -- 172.21.15.161:0/4151250980 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v152) ==== 100+0+11 (secure 0 0 0) 0x7fd33c05c570 con 0x7fd34c0c6350 2024-03-28T23:36:04.864 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:36:04.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.865+0000 7fd350ba7640 1 -- 172.21.15.161:0/4151250980 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd32c06b4d0 msgr2=0x7fd32c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:04.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.865+0000 7fd350ba7640 1 --2- 172.21.15.161:0/4151250980 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd32c06b4d0 0x7fd32c06d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd340007f70 tx=0x7fd340007f00 comp rx=0 tx=0).stop 2024-03-28T23:36:04.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.865+0000 7fd350ba7640 1 -- 172.21.15.161:0/4151250980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd34c0c6350 msgr2=0x7fd34c142810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:04.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.865+0000 7fd350ba7640 1 --2- 172.21.15.161:0/4151250980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd34c0c6350 0x7fd34c142810 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fd33c00c450 tx=0x7fd33c00c910 comp rx=0 tx=0).stop 2024-03-28T23:36:04.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.865+0000 7fd350ba7640 1 -- 172.21.15.161:0/4151250980 shutdown_connections 2024-03-28T23:36:04.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.865+0000 7fd350ba7640 1 --2- 172.21.15.161:0/4151250980 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd32c06b4d0 0x7fd32c06d990 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.865+0000 7fd350ba7640 1 --2- 172.21.15.161:0/4151250980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd34c0c6350 0x7fd34c142810 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.865+0000 7fd350ba7640 1 --2- 172.21.15.161:0/4151250980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd34c151e70 0x7fd34c1ebee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.865+0000 7fd350ba7640 1 --2- 172.21.15.161:0/4151250980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd34c06cdd0 0x7fd34c1de820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.865+0000 7fd350ba7640 1 -- 172.21.15.161:0/4151250980 >> 172.21.15.161:0/4151250980 conn(0x7fd34c0c3390 msgr2=0x7fd34c06d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:04.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.865+0000 7fd350ba7640 1 -- 172.21.15.161:0/4151250980 shutdown_connections 2024-03-28T23:36:04.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.865+0000 7fd350ba7640 1 -- 172.21.15.161:0/4151250980 wait complete. 2024-03-28T23:36:04.877 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T23:36:04.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.977+0000 7f4074380640 1 Processor -- start 2024-03-28T23:36:04.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.977+0000 7f4074380640 1 -- start start 2024-03-28T23:36:04.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.978+0000 7f4074380640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f406c14f730 0x7f406c0aad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:04.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.978+0000 7f4074380640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f406c151f80 0x7f406c0c21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:04.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.978+0000 7f4074380640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f406c0ab250 con 0x7f406c0ac310 2024-03-28T23:36:04.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.978+0000 7f4074380640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f406c152f00 con 0x7f406c151f80 2024-03-28T23:36:04.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.978+0000 7f4074380640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f406c155990 con 0x7f406c14f730 2024-03-28T23:36:04.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.979+0000 7f407291d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f406c151f80 0x7f406c0c21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:04.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.979+0000 7f407211c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f406c14f730 0x7f406c0aad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:04.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.979+0000 7f407291d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f406c151f80 0x7f406c0c21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53862/0 (socket says 172.21.15.161:53862) 2024-03-28T23:36:04.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.979+0000 7f407291d640 1 -- 172.21.15.161:0/2172567139 learned_addr learned my addr 172.21.15.161:0/2172567139 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:04.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.979+0000 7f407291d640 1 -- 172.21.15.161:0/2172567139 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f406c14f730 msgr2=0x7f406c0aad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:04.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.979+0000 7f407291d640 1 --2- 172.21.15.161:0/2172567139 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f406c14f730 0x7f406c0aad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.979+0000 7f407291d640 1 -- 172.21.15.161:0/2172567139 >> v1:172.21.15.170:6789/0 conn(0x7f406c0ac310 legacy=0x7f406c155e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:04.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.979+0000 7f407291d640 1 -- 172.21.15.161:0/2172567139 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f406c152940 con 0x7f406c151f80 2024-03-28T23:36:04.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.979+0000 7f407111a640 1 -- 172.21.15.161:0/2172567139 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3176928057 0 0) 0x7f406c0ab250 con 0x7f406c0ac310 2024-03-28T23:36:04.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.979+0000 7f407211c640 1 --2- 172.21.15.161:0/2172567139 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f406c14f730 0x7f406c0aad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:04.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.980+0000 7f407291d640 1 --2- 172.21.15.161:0/2172567139 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f406c151f80 0x7f406c0c21e0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f406400d150 tx=0x7f406402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5134216da62e0c0b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:04.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.980+0000 7f407111a640 1 -- 172.21.15.161:0/2172567139 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406402ea80 con 0x7f406c151f80 2024-03-28T23:36:04.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.980+0000 7f407111a640 1 -- 172.21.15.161:0/2172567139 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f406402ec20 con 0x7f406c151f80 2024-03-28T23:36:04.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.980+0000 7f407111a640 1 -- 172.21.15.161:0/2172567139 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4064009600 con 0x7f406c151f80 2024-03-28T23:36:04.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.980+0000 7f4074380640 1 -- 172.21.15.161:0/2172567139 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f406c151f80 msgr2=0x7f406c0c21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:04.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.980+0000 7f4074380640 1 --2- 172.21.15.161:0/2172567139 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f406c151f80 0x7f406c0c21e0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f406400d150 tx=0x7f406402e0e0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.980+0000 7f4074380640 1 -- 172.21.15.161:0/2172567139 shutdown_connections 2024-03-28T23:36:04.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.980+0000 7f4074380640 1 --2- 172.21.15.161:0/2172567139 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f406c151f80 0x7f406c0c21e0 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.980+0000 7f4074380640 1 --2- 172.21.15.161:0/2172567139 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f406c14f730 0x7f406c0aad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.980+0000 7f4074380640 1 -- 172.21.15.161:0/2172567139 >> 172.21.15.161:0/2172567139 conn(0x7f406c0c66d0 msgr2=0x7f406c0c6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:04.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.980+0000 7f4074380640 1 -- 172.21.15.161:0/2172567139 shutdown_connections 2024-03-28T23:36:04.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.980+0000 7f4074380640 1 -- 172.21.15.161:0/2172567139 wait complete. 2024-03-28T23:36:04.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.981+0000 7f4074380640 1 Processor -- start 2024-03-28T23:36:04.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.981+0000 7f4074380640 1 -- start start 2024-03-28T23:36:04.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.982+0000 7f4074380640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f406c0ac310 0x7f406c072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:04.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.982+0000 7f4074380640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f406c14f730 0x7f406c1edc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:04.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.982+0000 7f4074380640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f406c151f80 0x7f406c1eb090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:04.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.982+0000 7f4074380640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f406c14fb10 con 0x7f406c14f730 2024-03-28T23:36:04.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.982+0000 7f4074380640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f406c155620 con 0x7f406c151f80 2024-03-28T23:36:04.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.982+0000 7f4074380640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f406c152410 con 0x7f406c0ac310 2024-03-28T23:36:04.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.982+0000 7f407191b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f406c14f730 0x7f406c1edc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:04.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.982+0000 7f407191b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f406c14f730 0x7f406c1edc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53874/0 (socket says 172.21.15.161:53874) 2024-03-28T23:36:04.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.982+0000 7f407191b640 1 -- 172.21.15.161:0/1595103607 learned_addr learned my addr 172.21.15.161:0/1595103607 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:04.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.982+0000 7f407291d640 1 --2- 172.21.15.161:0/1595103607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f406c151f80 0x7f406c1eb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:04.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.982+0000 7f407211c640 1 --2- 172.21.15.161:0/1595103607 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f406c0ac310 0x7f406c072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:04.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.982+0000 7f407191b640 1 -- 172.21.15.161:0/1595103607 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f406c0ac310 msgr2=0x7f406c072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:04.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.982+0000 7f407191b640 1 --2- 172.21.15.161:0/1595103607 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f406c0ac310 0x7f406c072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.982+0000 7f407191b640 1 -- 172.21.15.161:0/1595103607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f406c151f80 msgr2=0x7f406c1eb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:04.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.982+0000 7f407191b640 1 --2- 172.21.15.161:0/1595103607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f406c151f80 0x7f406c1eb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:04.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.983+0000 7f407191b640 1 -- 172.21.15.161:0/1595103607 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f406c1ea8d0 con 0x7f406c14f730 2024-03-28T23:36:04.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.983+0000 7f407191b640 1 --2- 172.21.15.161:0/1595103607 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f406c14f730 0x7f406c1edc50 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f406800c550 tx=0x7f406800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:04.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.983+0000 7f4062ffd640 1 -- 172.21.15.161:0/1595103607 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4068017070 con 0x7f406c14f730 2024-03-28T23:36:04.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.983+0000 7f4062ffd640 1 -- 172.21.15.161:0/1595103607 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40680035f0 con 0x7f406c14f730 2024-03-28T23:36:04.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.983+0000 7f4074380640 1 -- 172.21.15.161:0/1595103607 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f406c0c5db0 con 0x7f406c14f730 2024-03-28T23:36:04.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.983+0000 7f4062ffd640 1 -- 172.21.15.161:0/1595103607 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40680039e0 con 0x7f406c14f730 2024-03-28T23:36:04.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.983+0000 7f4074380640 1 -- 172.21.15.161:0/1595103607 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f406c1ee190 con 0x7f406c14f730 2024-03-28T23:36:04.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.984+0000 7f4074380640 1 -- 172.21.15.161:0/1595103607 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f406c04f550 con 0x7f406c14f730 2024-03-28T23:36:04.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.984+0000 7f4062ffd640 1 -- 172.21.15.161:0/1595103607 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f406801a070 con 0x7f406c14f730 2024-03-28T23:36:04.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.985+0000 7f4062ffd640 1 --2- 172.21.15.161:0/1595103607 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f405006b4d0 0x7f405006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:04.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.985+0000 7f4062ffd640 1 -- 172.21.15.161:0/1595103607 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 6101+0+0 (secure 0 0 0) 0x7f4068089670 con 0x7f406c14f730 2024-03-28T23:36:04.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.985+0000 7f407211c640 1 --2- 172.21.15.161:0/1595103607 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f405006b4d0 0x7f405006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:04.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.985+0000 7f407211c640 1 --2- 172.21.15.161:0/1595103607 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f405006b4d0 0x7f405006d990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f406c06e4f0 tx=0x7f405c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:04.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:04.989+0000 7f4062ffd640 1 -- 172.21.15.161:0/1595103607 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4068057da0 con 0x7f406c14f730 2024-03-28T23:36:05.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.164+0000 7f4074380640 1 -- 172.21.15.161:0/1595103607 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f406c069440 con 0x7f406c14f730 2024-03-28T23:36:05.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.164+0000 7f4062ffd640 1 -- 172.21.15.161:0/1595103607 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v152) ==== 84+0+209 (secure 0 0 0) 0x7f406805be40 con 0x7f406c14f730 2024-03-28T23:36:05.165 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:05.165 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2024-03-28T23:36:05.165 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:36:05.165 INFO:teuthology.orchestra.run.smithi161.stdout: "poolnum": 1, 2024-03-28T23:36:05.165 INFO:teuthology.orchestra.run.smithi161.stdout: "poolname": ".mgr" 2024-03-28T23:36:05.165 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:36:05.165 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:36:05.165 INFO:teuthology.orchestra.run.smithi161.stdout: "poolnum": 23, 2024-03-28T23:36:05.165 INFO:teuthology.orchestra.run.smithi161.stdout: "poolname": "cephfs_metadata" 2024-03-28T23:36:05.165 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-28T23:36:05.165 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-28T23:36:05.165 INFO:teuthology.orchestra.run.smithi161.stdout: "poolnum": 24, 2024-03-28T23:36:05.165 INFO:teuthology.orchestra.run.smithi161.stdout: "poolname": "cephfs_data" 2024-03-28T23:36:05.166 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-28T23:36:05.166 INFO:teuthology.orchestra.run.smithi161.stdout:] 2024-03-28T23:36:05.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.166+0000 7f4074380640 1 -- 172.21.15.161:0/1595103607 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f405006b4d0 msgr2=0x7f405006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:05.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.166+0000 7f4074380640 1 --2- 172.21.15.161:0/1595103607 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f405006b4d0 0x7f405006d990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f406c06e4f0 tx=0x7f405c009190 comp rx=0 tx=0).stop 2024-03-28T23:36:05.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.166+0000 7f4074380640 1 -- 172.21.15.161:0/1595103607 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f406c14f730 msgr2=0x7f406c1edc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:05.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.166+0000 7f4074380640 1 --2- 172.21.15.161:0/1595103607 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f406c14f730 0x7f406c1edc50 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f406800c550 tx=0x7f406800ca10 comp rx=0 tx=0).stop 2024-03-28T23:36:05.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.166+0000 7f4074380640 1 -- 172.21.15.161:0/1595103607 shutdown_connections 2024-03-28T23:36:05.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.166+0000 7f4074380640 1 --2- 172.21.15.161:0/1595103607 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f405006b4d0 0x7f405006d990 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:05.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.166+0000 7f4074380640 1 --2- 172.21.15.161:0/1595103607 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f406c14f730 0x7f406c1edc50 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:05.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.166+0000 7f4074380640 1 --2- 172.21.15.161:0/1595103607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f406c151f80 0x7f406c1eb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:05.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.166+0000 7f4074380640 1 --2- 172.21.15.161:0/1595103607 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f406c0ac310 0x7f406c072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:05.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.166+0000 7f4074380640 1 -- 172.21.15.161:0/1595103607 >> 172.21.15.161:0/1595103607 conn(0x7f406c0c66d0 msgr2=0x7f406c0c19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:05.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.166+0000 7f4074380640 1 -- 172.21.15.161:0/1595103607 shutdown_connections 2024-03-28T23:36:05.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.166+0000 7f4074380640 1 -- 172.21.15.161:0/1595103607 wait complete. 2024-03-28T23:36:05.178 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs2' 2024-03-28T23:36:05.178 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_metadata --pg_num_min 64 2024-03-28T23:36:05.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.282+0000 7f8cb8fa1640 1 Processor -- start 2024-03-28T23:36:05.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.282+0000 7f8cb8fa1640 1 -- start start 2024-03-28T23:36:05.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.283+0000 7f8cb8fa1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cb40abb70 0x7f8cb40abf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:05.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.283+0000 7f8cb8fa1640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8cb40c46b0 0x7f8cb40c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:05.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.283+0000 7f8cb8fa1640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cb4149ec0 con 0x7f8cb406d390 2024-03-28T23:36:05.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.283+0000 7f8cb8fa1640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f8cb4153ab0 con 0x7f8cb40c46b0 2024-03-28T23:36:05.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.283+0000 7f8cb8fa1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8cb406e400 con 0x7f8cb40abb70 2024-03-28T23:36:05.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.283+0000 7f8cb259c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cb40abb70 0x7f8cb40abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:05.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.283+0000 7f8cb2d9d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8cb40c46b0 0x7f8cb40c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:05.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.283+0000 7f8cb259c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cb40abb70 0x7f8cb40abf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:42838/0 (socket says 172.21.15.161:42838) 2024-03-28T23:36:05.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.283+0000 7f8cb259c640 1 -- 172.21.15.161:0/3618162120 learned_addr learned my addr 172.21.15.161:0/3618162120 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:05.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.283+0000 7f8cabfff640 1 --1- 172.21.15.161:0/3618162120 >> v1:172.21.15.170:6789/0 conn(0x7f8cb406d390 0x7f8cb406dcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:38794/0 (socket says 172.21.15.161:38794) 2024-03-28T23:36:05.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.283+0000 7f8cb259c640 1 -- 172.21.15.161:0/3618162120 >> v2:172.21.15.170:3300/0 conn(0x7f8cb40c46b0 msgr2=0x7f8cb40c4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:05.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.283+0000 7f8cb259c640 1 --2- 172.21.15.161:0/3618162120 >> v2:172.21.15.170:3300/0 conn(0x7f8cb40c46b0 0x7f8cb40c4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:05.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.283+0000 7f8cb259c640 1 -- 172.21.15.161:0/3618162120 >> v1:172.21.15.170:6789/0 conn(0x7f8cb406d390 legacy=0x7f8cb406dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:05.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.283+0000 7f8cb259c640 1 -- 172.21.15.161:0/3618162120 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cb4153180 con 0x7f8cb40abb70 2024-03-28T23:36:05.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.283+0000 7f8cb2d9d640 1 --2- 172.21.15.161:0/3618162120 >> v2:172.21.15.170:3300/0 conn(0x7f8cb40c46b0 0x7f8cb40c4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:05.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.284+0000 7f8cb259c640 1 --2- 172.21.15.161:0/3618162120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cb40abb70 0x7f8cb40abf50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4009870 tx=0x7f8ca402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27125d61b11b604 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:05.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.284+0000 7f8cb1d9b640 1 -- 172.21.15.161:0/3618162120 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca4034070 con 0x7f8cb40abb70 2024-03-28T23:36:05.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.284+0000 7f8cb1d9b640 1 -- 172.21.15.161:0/3618162120 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ca4030840 con 0x7f8cb40abb70 2024-03-28T23:36:05.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.284+0000 7f8cb1d9b640 1 -- 172.21.15.161:0/3618162120 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca4030c10 con 0x7f8cb40abb70 2024-03-28T23:36:05.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.285+0000 7f8cb8fa1640 1 -- 172.21.15.161:0/3618162120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cb40abb70 msgr2=0x7f8cb40abf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:05.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.285+0000 7f8cb8fa1640 1 --2- 172.21.15.161:0/3618162120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cb40abb70 0x7f8cb40abf50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4009870 tx=0x7f8ca402b980 comp rx=0 tx=0).stop 2024-03-28T23:36:05.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.285+0000 7f8cb8fa1640 1 -- 172.21.15.161:0/3618162120 shutdown_connections 2024-03-28T23:36:05.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.285+0000 7f8cb8fa1640 1 --2- 172.21.15.161:0/3618162120 >> v2:172.21.15.170:3300/0 conn(0x7f8cb40c46b0 0x7f8cb40c4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:05.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.285+0000 7f8cb8fa1640 1 --2- 172.21.15.161:0/3618162120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cb40abb70 0x7f8cb40abf50 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:05.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.285+0000 7f8cb8fa1640 1 -- 172.21.15.161:0/3618162120 >> 172.21.15.161:0/3618162120 conn(0x7f8cb406b150 msgr2=0x7f8cb406b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:05.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.285+0000 7f8cb8fa1640 1 -- 172.21.15.161:0/3618162120 shutdown_connections 2024-03-28T23:36:05.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.285+0000 7f8cb8fa1640 1 -- 172.21.15.161:0/3618162120 wait complete. 2024-03-28T23:36:05.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.286+0000 7f8cb8fa1640 1 Processor -- start 2024-03-28T23:36:05.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.286+0000 7f8cb8fa1640 1 -- start start 2024-03-28T23:36:05.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.287+0000 7f8cb8fa1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cb406d390 0x7f8cb41ed420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:05.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.287+0000 7f8cb8fa1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cb40abb70 0x7f8cb41e9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:05.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.287+0000 7f8cb8fa1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cb40c46b0 0x7f8cb40a6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:05.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.287+0000 7f8cb8fa1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb40c50a0 con 0x7f8cb40c46b0 2024-03-28T23:36:05.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.287+0000 7f8cb8fa1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb4153740 con 0x7f8cb40abb70 2024-03-28T23:36:05.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.287+0000 7f8cb8fa1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8cb40c5220 con 0x7f8cb406d390 2024-03-28T23:36:05.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.287+0000 7f8cb2d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cb40c46b0 0x7f8cb40a6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:05.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.287+0000 7f8cb2d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cb40c46b0 0x7f8cb40a6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53880/0 (socket says 172.21.15.161:53880) 2024-03-28T23:36:05.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.287+0000 7f8cb2d9d640 1 -- 172.21.15.161:0/4134451132 learned_addr learned my addr 172.21.15.161:0/4134451132 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:05.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.287+0000 7f8cb259c640 1 --2- 172.21.15.161:0/4134451132 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cb406d390 0x7f8cb41ed420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:05.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.287+0000 7f8cabfff640 1 --2- 172.21.15.161:0/4134451132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cb40abb70 0x7f8cb41e9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:05.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.287+0000 7f8cb2d9d640 1 -- 172.21.15.161:0/4134451132 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cb406d390 msgr2=0x7f8cb41ed420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:05.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.287+0000 7f8cb2d9d640 1 --2- 172.21.15.161:0/4134451132 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cb406d390 0x7f8cb41ed420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:05.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.287+0000 7f8cb2d9d640 1 -- 172.21.15.161:0/4134451132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cb40abb70 msgr2=0x7f8cb41e9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:05.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.287+0000 7f8cb2d9d640 1 --2- 172.21.15.161:0/4134451132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cb40abb70 0x7f8cb41e9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:05.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.287+0000 7f8cb2d9d640 1 -- 172.21.15.161:0/4134451132 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cb40bc3c0 con 0x7f8cb40c46b0 2024-03-28T23:36:05.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.287+0000 7f8cb259c640 1 --2- 172.21.15.161:0/4134451132 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cb406d390 0x7f8cb41ed420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:05.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.287+0000 7f8cb2d9d640 1 --2- 172.21.15.161:0/4134451132 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cb40c46b0 0x7f8cb40a6ee0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f8cac00efc0 tx=0x7f8cac00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:05.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.287+0000 7f8caaffd640 1 -- 172.21.15.161:0/4134451132 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cac018030 con 0x7f8cb40c46b0 2024-03-28T23:36:05.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.287+0000 7f8caaffd640 1 -- 172.21.15.161:0/4134451132 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cac00cdf0 con 0x7f8cb40c46b0 2024-03-28T23:36:05.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.287+0000 7f8cb8fa1640 1 -- 172.21.15.161:0/4134451132 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cb41ea3a0 con 0x7f8cb40c46b0 2024-03-28T23:36:05.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.287+0000 7f8caaffd640 1 -- 172.21.15.161:0/4134451132 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cac003bb0 con 0x7f8cb40c46b0 2024-03-28T23:36:05.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.287+0000 7f8cb8fa1640 1 -- 172.21.15.161:0/4134451132 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cb4070a50 con 0x7f8cb40c46b0 2024-03-28T23:36:05.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.288+0000 7f8cb8fa1640 1 -- 172.21.15.161:0/4134451132 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cb4068f90 con 0x7f8cb40c46b0 2024-03-28T23:36:05.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.289+0000 7f8caaffd640 1 -- 172.21.15.161:0/4134451132 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8cac005d10 con 0x7f8cb40c46b0 2024-03-28T23:36:05.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.289+0000 7f8caaffd640 1 --2- 172.21.15.161:0/4134451132 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8c9406b4d0 0x7f8c9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:05.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.289+0000 7f8caaffd640 1 -- 172.21.15.161:0/4134451132 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 6101+0+0 (secure 0 0 0) 0x7f8cac08ab80 con 0x7f8cb40c46b0 2024-03-28T23:36:05.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.290+0000 7f8cb259c640 1 --2- 172.21.15.161:0/4134451132 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8c9406b4d0 0x7f8c9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:05.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.290+0000 7f8cb259c640 1 --2- 172.21.15.161:0/4134451132 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8c9406b4d0 0x7f8c9406d990 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4036000 tx=0x7f8ca4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:05.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.294+0000 7f8caaffd640 1 -- 172.21.15.161:0/4134451132 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8cac022070 con 0x7f8cb40c46b0 2024-03-28T23:36:05.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:05.465+0000 7f8cb8fa1640 1 -- 172.21.15.161:0/4134451132 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64} v 0) -- 0x7f8cb406d7c0 con 0x7f8cb40c46b0 2024-03-28T23:36:06.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.085+0000 7f8caaffd640 1 -- 172.21.15.161:0/4134451132 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64}]=0 pool 'cephfs2_metadata' created v153) ==== 140+0+0 (secure 0 0 0) 0x7f8cb41ea3a0 con 0x7f8cb40c46b0 2024-03-28T23:36:06.085 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs2_metadata' created 2024-03-28T23:36:06.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.090+0000 7f8cb8fa1640 1 -- 172.21.15.161:0/4134451132 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8c9406b4d0 msgr2=0x7f8c9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:06.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.090+0000 7f8cb8fa1640 1 --2- 172.21.15.161:0/4134451132 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8c9406b4d0 0x7f8c9406d990 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4036000 tx=0x7f8ca4008030 comp rx=0 tx=0).stop 2024-03-28T23:36:06.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.090+0000 7f8cb8fa1640 1 -- 172.21.15.161:0/4134451132 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cb40c46b0 msgr2=0x7f8cb40a6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:06.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.090+0000 7f8cb8fa1640 1 --2- 172.21.15.161:0/4134451132 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cb40c46b0 0x7f8cb40a6ee0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f8cac00efc0 tx=0x7f8cac00c490 comp rx=0 tx=0).stop 2024-03-28T23:36:06.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.090+0000 7f8cb8fa1640 1 -- 172.21.15.161:0/4134451132 shutdown_connections 2024-03-28T23:36:06.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.090+0000 7f8cb8fa1640 1 --2- 172.21.15.161:0/4134451132 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8c9406b4d0 0x7f8c9406d990 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:06.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.090+0000 7f8cb8fa1640 1 --2- 172.21.15.161:0/4134451132 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cb40c46b0 0x7f8cb40a6ee0 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:06.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.090+0000 7f8cb8fa1640 1 --2- 172.21.15.161:0/4134451132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cb40abb70 0x7f8cb41e9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:06.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.090+0000 7f8cb8fa1640 1 --2- 172.21.15.161:0/4134451132 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cb406d390 0x7f8cb41ed420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:06.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.090+0000 7f8cb8fa1640 1 -- 172.21.15.161:0/4134451132 >> 172.21.15.161:0/4134451132 conn(0x7f8cb406b150 msgr2=0x7f8cb40c2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:06.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.090+0000 7f8cb8fa1640 1 -- 172.21.15.161:0/4134451132 shutdown_connections 2024-03-28T23:36:06.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.090+0000 7f8cb8fa1640 1 -- 172.21.15.161:0/4134451132 wait complete. 2024-03-28T23:36:06.107 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T23:36:06.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.220+0000 7ff79f779640 1 Processor -- start 2024-03-28T23:36:06.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.220+0000 7ff79f779640 1 -- start start 2024-03-28T23:36:06.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.220+0000 7ff79f779640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7980a9960 0x7ff798072970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:06.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.221+0000 7ff79f779640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff7980733e0 0x7ff79814eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:06.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.221+0000 7ff79f779640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7980abc00 con 0x7ff7980c7330 2024-03-28T23:36:06.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.221+0000 7ff79f779640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff7980aa3a0 con 0x7ff7980733e0 2024-03-28T23:36:06.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.221+0000 7ff79f779640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff79806bf70 con 0x7ff7980a9960 2024-03-28T23:36:06.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.221+0000 7ff79dd16640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff7980733e0 0x7ff79814eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:06.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.221+0000 7ff79dd16640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff7980733e0 0x7ff79814eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53882/0 (socket says 172.21.15.161:53882) 2024-03-28T23:36:06.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.221+0000 7ff79dd16640 1 -- 172.21.15.161:0/2910264273 learned_addr learned my addr 172.21.15.161:0/2910264273 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:06.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.221+0000 7ff79d515640 1 --2- 172.21.15.161:0/2910264273 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7980a9960 0x7ff798072970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:06.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.221+0000 7ff79dd16640 1 -- 172.21.15.161:0/2910264273 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7980a9960 msgr2=0x7ff798072970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:06.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.221+0000 7ff79dd16640 1 --2- 172.21.15.161:0/2910264273 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7980a9960 0x7ff798072970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:06.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.221+0000 7ff78ffff640 1 -- 172.21.15.161:0/2910264273 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2744635328 0 0) 0x7ff7980abc00 con 0x7ff7980c7330 2024-03-28T23:36:06.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.221+0000 7ff79dd16640 1 -- 172.21.15.161:0/2910264273 >> v1:172.21.15.170:6789/0 conn(0x7ff7980c7330 legacy=0x7ff79814e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:06.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.221+0000 7ff79dd16640 1 -- 172.21.15.161:0/2910264273 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff79806bcd0 con 0x7ff7980733e0 2024-03-28T23:36:06.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.222+0000 7ff79dd16640 1 --2- 172.21.15.161:0/2910264273 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7980733e0 0x7ff79814eb70 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7ff79000d150 tx=0x7ff79002e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bb833552584a9d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:06.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.222+0000 7ff78ffff640 1 -- 172.21.15.161:0/2910264273 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79002e9e0 con 0x7ff7980733e0 2024-03-28T23:36:06.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.222+0000 7ff78ffff640 1 -- 172.21.15.161:0/2910264273 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff79002eb80 con 0x7ff7980733e0 2024-03-28T23:36:06.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.222+0000 7ff78ffff640 1 -- 172.21.15.161:0/2910264273 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff790009600 con 0x7ff7980733e0 2024-03-28T23:36:06.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.222+0000 7ff79f779640 1 -- 172.21.15.161:0/2910264273 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7980733e0 msgr2=0x7ff79814eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:06.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.222+0000 7ff79f779640 1 --2- 172.21.15.161:0/2910264273 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7980733e0 0x7ff79814eb70 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7ff79000d150 tx=0x7ff79002e040 comp rx=0 tx=0).stop 2024-03-28T23:36:06.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.222+0000 7ff79f779640 1 -- 172.21.15.161:0/2910264273 shutdown_connections 2024-03-28T23:36:06.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.222+0000 7ff79f779640 1 --2- 172.21.15.161:0/2910264273 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7980733e0 0x7ff79814eb70 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:06.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.222+0000 7ff79f779640 1 --2- 172.21.15.161:0/2910264273 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7980a9960 0x7ff798072970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:06.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.222+0000 7ff79f779640 1 -- 172.21.15.161:0/2910264273 >> 172.21.15.161:0/2910264273 conn(0x7ff7980a83f0 msgr2=0x7ff7980a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:06.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.223+0000 7ff79f779640 1 -- 172.21.15.161:0/2910264273 shutdown_connections 2024-03-28T23:36:06.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.223+0000 7ff79f779640 1 -- 172.21.15.161:0/2910264273 wait complete. 2024-03-28T23:36:06.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.224+0000 7ff79f779640 1 Processor -- start 2024-03-28T23:36:06.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.224+0000 7ff79f779640 1 -- start start 2024-03-28T23:36:06.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.224+0000 7ff79f779640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7980733e0 0x7ff7981e6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:06.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.224+0000 7ff79f779640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff7980a9960 0x7ff7981e7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:06.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.224+0000 7ff79f779640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7980c7330 0x7ff79814c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:06.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.224+0000 7ff79f779640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff7980ab820 con 0x7ff7980733e0 2024-03-28T23:36:06.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.224+0000 7ff79f779640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff798152250 con 0x7ff7980a9960 2024-03-28T23:36:06.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.224+0000 7ff79f779640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff7980ab9a0 con 0x7ff7980c7330 2024-03-28T23:36:06.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.224+0000 7ff79d515640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7980733e0 0x7ff7981e6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:06.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.224+0000 7ff79d515640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7980733e0 0x7ff7981e6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53886/0 (socket says 172.21.15.161:53886) 2024-03-28T23:36:06.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.224+0000 7ff79d515640 1 -- 172.21.15.161:0/873604372 learned_addr learned my addr 172.21.15.161:0/873604372 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:06.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.225+0000 7ff79dd16640 1 --2- 172.21.15.161:0/873604372 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7980c7330 0x7ff79814c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:06.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.225+0000 7ff79cd14640 1 --2- 172.21.15.161:0/873604372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff7980a9960 0x7ff7981e7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:06.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.225+0000 7ff79d515640 1 -- 172.21.15.161:0/873604372 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7980c7330 msgr2=0x7ff79814c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:06.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.225+0000 7ff79d515640 1 --2- 172.21.15.161:0/873604372 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7980c7330 0x7ff79814c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:06.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.225+0000 7ff79d515640 1 -- 172.21.15.161:0/873604372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff7980a9960 msgr2=0x7ff7981e7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:06.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.225+0000 7ff79d515640 1 --2- 172.21.15.161:0/873604372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff7980a9960 0x7ff7981e7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:06.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.225+0000 7ff79d515640 1 -- 172.21.15.161:0/873604372 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff79814aea0 con 0x7ff7980733e0 2024-03-28T23:36:06.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.225+0000 7ff79d515640 1 --2- 172.21.15.161:0/873604372 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7980733e0 0x7ff7981e6890 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7ff79400e9b0 tx=0x7ff79400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:06.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.225+0000 7ff78dffb640 1 -- 172.21.15.161:0/873604372 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff794017070 con 0x7ff7980733e0 2024-03-28T23:36:06.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.225+0000 7ff78dffb640 1 -- 172.21.15.161:0/873604372 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff79400c9a0 con 0x7ff7980733e0 2024-03-28T23:36:06.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.225+0000 7ff79f779640 1 -- 172.21.15.161:0/873604372 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff79814b180 con 0x7ff7980733e0 2024-03-28T23:36:06.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.225+0000 7ff78dffb640 1 -- 172.21.15.161:0/873604372 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79400cd90 con 0x7ff7980733e0 2024-03-28T23:36:06.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.225+0000 7ff79f779640 1 -- 172.21.15.161:0/873604372 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff79814b740 con 0x7ff7980733e0 2024-03-28T23:36:06.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.226+0000 7ff79f779640 1 -- 172.21.15.161:0/873604372 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff79804f4b0 con 0x7ff7980733e0 2024-03-28T23:36:06.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.227+0000 7ff78dffb640 1 -- 172.21.15.161:0/873604372 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff79401a070 con 0x7ff7980733e0 2024-03-28T23:36:06.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.227+0000 7ff78dffb640 1 --2- 172.21.15.161:0/873604372 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff77c06b4d0 0x7ff77c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:06.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.227+0000 7ff79cd14640 1 --2- 172.21.15.161:0/873604372 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff77c06b4d0 0x7ff77c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:06.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.227+0000 7ff78dffb640 1 -- 172.21.15.161:0/873604372 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 6472+0+0 (secure 0 0 0) 0x7ff79408a7a0 con 0x7ff7980733e0 2024-03-28T23:36:06.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.228+0000 7ff79cd14640 1 --2- 172.21.15.161:0/873604372 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff77c06b4d0 0x7ff77c06d990 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff7980a8e70 tx=0x7ff788009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:06.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.232+0000 7ff78dffb640 1 -- 172.21.15.161:0/873604372 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff794058d60 con 0x7ff7980733e0 2024-03-28T23:36:06.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:06.404+0000 7ff79f779640 1 -- 172.21.15.161:0/873604372 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7ff7980737c0 con 0x7ff7980733e0 2024-03-28T23:36:07.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.087+0000 7ff78dffb640 1 -- 172.21.15.161:0/873604372 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs2_data' created v154) ==== 172+0+0 (secure 0 0 0) 0x7ff79405ce00 con 0x7ff7980733e0 2024-03-28T23:36:07.088 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs2_data' created 2024-03-28T23:36:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.091+0000 7ff79f779640 1 -- 172.21.15.161:0/873604372 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff77c06b4d0 msgr2=0x7ff77c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.091+0000 7ff79f779640 1 --2- 172.21.15.161:0/873604372 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff77c06b4d0 0x7ff77c06d990 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff7980a8e70 tx=0x7ff788009040 comp rx=0 tx=0).stop 2024-03-28T23:36:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.091+0000 7ff79f779640 1 -- 172.21.15.161:0/873604372 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7980733e0 msgr2=0x7ff7981e6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.091+0000 7ff79f779640 1 --2- 172.21.15.161:0/873604372 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7980733e0 0x7ff7981e6890 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7ff79400e9b0 tx=0x7ff79400ee70 comp rx=0 tx=0).stop 2024-03-28T23:36:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.091+0000 7ff79f779640 1 -- 172.21.15.161:0/873604372 shutdown_connections 2024-03-28T23:36:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.091+0000 7ff79f779640 1 --2- 172.21.15.161:0/873604372 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff77c06b4d0 0x7ff77c06d990 secure :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff7980a8e70 tx=0x7ff788009040 comp rx=0 tx=0).stop 2024-03-28T23:36:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.091+0000 7ff79f779640 1 --2- 172.21.15.161:0/873604372 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7980c7330 0x7ff79814c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.091+0000 7ff79f779640 1 --2- 172.21.15.161:0/873604372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff7980a9960 0x7ff7981e7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.091+0000 7ff79f779640 1 --2- 172.21.15.161:0/873604372 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7980733e0 0x7ff7981e6890 secure :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7ff79400e9b0 tx=0x7ff79400ee70 comp rx=0 tx=0).stop 2024-03-28T23:36:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.091+0000 7ff79f779640 1 -- 172.21.15.161:0/873604372 >> 172.21.15.161:0/873604372 conn(0x7ff7980a83f0 msgr2=0x7ff7980c1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:07.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.093+0000 7ff79f779640 1 -- 172.21.15.161:0/873604372 shutdown_connections 2024-03-28T23:36:07.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.093+0000 7ff79f779640 1 -- 172.21.15.161:0/873604372 wait complete. 2024-03-28T23:36:07.119 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs2 cephfs2_metadata cephfs2_data 2024-03-28T23:36:07.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.225+0000 7f76a5fc9640 1 Processor -- start 2024-03-28T23:36:07.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.225+0000 7f76a5fc9640 1 -- start start 2024-03-28T23:36:07.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.226+0000 7f76a5fc9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f76a0152bc0 0x7f76a00c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:07.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.226+0000 7f76a5fc9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76a00c8a80 0x7f76a00c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:07.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.226+0000 7f76a5fc9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f76a00c9420 0x7f76a01589b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:07.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.226+0000 7f76a5fc9640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f76a00c6f60 con 0x7f76a00c9420 2024-03-28T23:36:07.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.226+0000 7f76a5fc9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f76a00c6120 con 0x7f76a0152bc0 2024-03-28T23:36:07.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.226+0000 7f76a5fc9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f76a00c6400 con 0x7f76a00c8a80 2024-03-28T23:36:07.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.226+0000 7f769ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f76a00c9420 0x7f76a01589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:07.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.226+0000 7f769ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f76a00c9420 0x7f76a01589b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53900/0 (socket says 172.21.15.161:53900) 2024-03-28T23:36:07.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.226+0000 7f769ffff640 1 -- 172.21.15.161:0/1275418147 learned_addr learned my addr 172.21.15.161:0/1275418147 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:07.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.226+0000 7f769f7fe640 1 --2- 172.21.15.161:0/1275418147 >> v2:172.21.15.170:3300/0 conn(0x7f76a0152bc0 0x7f76a00c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:07.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.226+0000 7f769effd640 1 --2- 172.21.15.161:0/1275418147 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76a00c8a80 0x7f76a00c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:07.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.227+0000 7f769ffff640 1 -- 172.21.15.161:0/1275418147 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76a00c8a80 msgr2=0x7f76a00c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:07.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.227+0000 7f769ffff640 1 --2- 172.21.15.161:0/1275418147 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76a00c8a80 0x7f76a00c8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:07.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.227+0000 7f769ffff640 1 -- 172.21.15.161:0/1275418147 >> v2:172.21.15.170:3300/0 conn(0x7f76a0152bc0 msgr2=0x7f76a00c0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:07.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.227+0000 7f769ffff640 1 --2- 172.21.15.161:0/1275418147 >> v2:172.21.15.170:3300/0 conn(0x7f76a0152bc0 0x7f76a00c0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:07.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.227+0000 7f769ffff640 1 -- 172.21.15.161:0/1275418147 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76a00c7240 con 0x7f76a00c9420 2024-03-28T23:36:07.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.227+0000 7f769ffff640 1 --2- 172.21.15.161:0/1275418147 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76a00c9420 0x7f76a01589b0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f76a00aa410 tx=0x7f769802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc76fe375e80fff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:07.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.228+0000 7f769e7fc640 1 -- 172.21.15.161:0/1275418147 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f769802ca70 con 0x7f76a00c9420 2024-03-28T23:36:07.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.228+0000 7f769e7fc640 1 -- 172.21.15.161:0/1275418147 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f769802cc10 con 0x7f76a00c9420 2024-03-28T23:36:07.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.228+0000 7f769e7fc640 1 -- 172.21.15.161:0/1275418147 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76980343f0 con 0x7f76a00c9420 2024-03-28T23:36:07.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.228+0000 7f76a5fc9640 1 -- 172.21.15.161:0/1275418147 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76a00c9420 msgr2=0x7f76a01589b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:07.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.228+0000 7f76a5fc9640 1 --2- 172.21.15.161:0/1275418147 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76a00c9420 0x7f76a01589b0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f76a00aa410 tx=0x7f769802bed0 comp rx=0 tx=0).stop 2024-03-28T23:36:07.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.228+0000 7f76a5fc9640 1 -- 172.21.15.161:0/1275418147 shutdown_connections 2024-03-28T23:36:07.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.228+0000 7f76a5fc9640 1 --2- 172.21.15.161:0/1275418147 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76a00c9420 0x7f76a01589b0 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:07.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.228+0000 7f76a5fc9640 1 --2- 172.21.15.161:0/1275418147 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76a00c8a80 0x7f76a00c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:07.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.228+0000 7f76a5fc9640 1 --2- 172.21.15.161:0/1275418147 >> v2:172.21.15.170:3300/0 conn(0x7f76a0152bc0 0x7f76a00c0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:07.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.228+0000 7f76a5fc9640 1 -- 172.21.15.161:0/1275418147 >> 172.21.15.161:0/1275418147 conn(0x7f76a00caa80 msgr2=0x7f76a00cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:07.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.228+0000 7f76a5fc9640 1 -- 172.21.15.161:0/1275418147 shutdown_connections 2024-03-28T23:36:07.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.228+0000 7f76a5fc9640 1 -- 172.21.15.161:0/1275418147 wait complete. 2024-03-28T23:36:07.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.229+0000 7f76a5fc9640 1 Processor -- start 2024-03-28T23:36:07.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.229+0000 7f76a5fc9640 1 -- start start 2024-03-28T23:36:07.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.230+0000 7f76a5fc9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76a00c8a80 0x7f76a01503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:07.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.230+0000 7f76a5fc9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76a00c9420 0x7f76a006c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:07.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.230+0000 7f76a5fc9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76a0152bc0 0x7f76a0154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:07.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.230+0000 7f76a5fc9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f76a0152fa0 con 0x7f76a00c8a80 2024-03-28T23:36:07.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.230+0000 7f76a5fc9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f76a0158ef0 con 0x7f76a00c9420 2024-03-28T23:36:07.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.230+0000 7f76a5fc9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f76a0154a50 con 0x7f76a0152bc0 2024-03-28T23:36:07.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.230+0000 7f769f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76a00c8a80 0x7f76a01503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:07.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.230+0000 7f769f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76a00c8a80 0x7f76a01503e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53912/0 (socket says 172.21.15.161:53912) 2024-03-28T23:36:07.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.230+0000 7f769f7fe640 1 -- 172.21.15.161:0/1851342083 learned_addr learned my addr 172.21.15.161:0/1851342083 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:07.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.230+0000 7f769ffff640 1 --2- 172.21.15.161:0/1851342083 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76a0152bc0 0x7f76a0154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:07.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.230+0000 7f769effd640 1 --2- 172.21.15.161:0/1851342083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76a00c9420 0x7f76a006c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:07.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.230+0000 7f769f7fe640 1 -- 172.21.15.161:0/1851342083 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76a0152bc0 msgr2=0x7f76a0154ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:07.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.230+0000 7f769f7fe640 1 --2- 172.21.15.161:0/1851342083 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76a0152bc0 0x7f76a0154ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:07.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.231+0000 7f769f7fe640 1 -- 172.21.15.161:0/1851342083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76a00c9420 msgr2=0x7f76a006c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:07.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.231+0000 7f769f7fe640 1 --2- 172.21.15.161:0/1851342083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76a00c9420 0x7f76a006c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:07.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.231+0000 7f769f7fe640 1 -- 172.21.15.161:0/1851342083 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76a006ceb0 con 0x7f76a00c8a80 2024-03-28T23:36:07.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.231+0000 7f769f7fe640 1 --2- 172.21.15.161:0/1851342083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76a00c8a80 0x7f76a01503e0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f769400e870 tx=0x7f769400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:07.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.231+0000 7f7687fff640 1 -- 172.21.15.161:0/1851342083 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7694017070 con 0x7f76a00c8a80 2024-03-28T23:36:07.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.231+0000 7f7687fff640 1 -- 172.21.15.161:0/1851342083 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f769400c940 con 0x7f76a00c8a80 2024-03-28T23:36:07.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.231+0000 7f7687fff640 1 -- 172.21.15.161:0/1851342083 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f769400cd30 con 0x7f76a00c8a80 2024-03-28T23:36:07.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.231+0000 7f76a5fc9640 1 -- 172.21.15.161:0/1851342083 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76a0150a70 con 0x7f76a00c8a80 2024-03-28T23:36:07.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.231+0000 7f76a5fc9640 1 -- 172.21.15.161:0/1851342083 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76a0151030 con 0x7f76a00c8a80 2024-03-28T23:36:07.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.232+0000 7f76a5fc9640 1 -- 172.21.15.161:0/1851342083 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76a0156570 con 0x7f76a00c8a80 2024-03-28T23:36:07.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.232+0000 7f7687fff640 1 -- 172.21.15.161:0/1851342083 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f769401a070 con 0x7f76a00c8a80 2024-03-28T23:36:07.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.233+0000 7f7687fff640 1 --2- 172.21.15.161:0/1851342083 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f768c06b4d0 0x7f768c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:07.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.233+0000 7f7687fff640 1 -- 172.21.15.161:0/1851342083 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(154..154 src has 1..154) ==== 6855+0+0 (secure 0 0 0) 0x7f769408a880 con 0x7f76a00c8a80 2024-03-28T23:36:07.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.233+0000 7f769effd640 1 --2- 172.21.15.161:0/1851342083 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f768c06b4d0 0x7f768c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:07.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.233+0000 7f769effd640 1 --2- 172.21.15.161:0/1851342083 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f768c06b4d0 0x7f768c06d990 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f76a0152560 tx=0x7f7690009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:07.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.237+0000 7f7687fff640 1 -- 172.21.15.161:0/1851342083 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f76a0156570 con 0x7f76a00c8a80 2024-03-28T23:36:07.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:07.447+0000 7f76a5fc9640 1 -- 172.21.15.161:0/1851342083 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"} v 0) -- 0x7f76a00ab7a0 con 0x7f76a00c8a80 2024-03-28T23:36:08.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.105+0000 7f7687fff640 1 -- 172.21.15.161:0/1851342083 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"}]=0 new fs with metadata pool 25 and data pool 26 v141) ==== 177+0+0 (secure 0 0 0) 0x7f76a00ab7a0 con 0x7f76a00c8a80 2024-03-28T23:36:08.111 INFO:teuthology.orchestra.run.smithi161.stderr:new fs with metadata pool 25 and data pool 26 2024-03-28T23:36:08.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.118+0000 7f76a5fc9640 1 -- 172.21.15.161:0/1851342083 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f768c06b4d0 msgr2=0x7f768c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:08.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.118+0000 7f76a5fc9640 1 --2- 172.21.15.161:0/1851342083 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f768c06b4d0 0x7f768c06d990 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f76a0152560 tx=0x7f7690009190 comp rx=0 tx=0).stop 2024-03-28T23:36:08.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.118+0000 7f76a5fc9640 1 -- 172.21.15.161:0/1851342083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76a00c8a80 msgr2=0x7f76a01503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:08.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.118+0000 7f76a5fc9640 1 --2- 172.21.15.161:0/1851342083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76a00c8a80 0x7f76a01503e0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f769400e870 tx=0x7f769400ed30 comp rx=0 tx=0).stop 2024-03-28T23:36:08.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.119+0000 7f76a5fc9640 1 -- 172.21.15.161:0/1851342083 shutdown_connections 2024-03-28T23:36:08.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.119+0000 7f76a5fc9640 1 --2- 172.21.15.161:0/1851342083 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f768c06b4d0 0x7f768c06d990 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.119+0000 7f76a5fc9640 1 --2- 172.21.15.161:0/1851342083 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76a0152bc0 0x7f76a0154ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.119+0000 7f76a5fc9640 1 --2- 172.21.15.161:0/1851342083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76a00c9420 0x7f76a006c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.119+0000 7f76a5fc9640 1 --2- 172.21.15.161:0/1851342083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f76a00c8a80 0x7f76a01503e0 secure :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f769400e870 tx=0x7f769400ed30 comp rx=0 tx=0).stop 2024-03-28T23:36:08.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.119+0000 7f76a5fc9640 1 -- 172.21.15.161:0/1851342083 >> 172.21.15.161:0/1851342083 conn(0x7f76a00caa80 msgr2=0x7f76a00c9800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:08.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.121+0000 7f76a5fc9640 1 -- 172.21.15.161:0/1851342083 shutdown_connections 2024-03-28T23:36:08.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.121+0000 7f76a5fc9640 1 -- 172.21.15.161:0/1851342083 wait complete. 2024-03-28T23:36:08.138 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:36:08.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.242+0000 7fa37b63b640 1 Processor -- start 2024-03-28T23:36:08.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.242+0000 7fa37b63b640 1 -- start start 2024-03-28T23:36:08.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.243+0000 7fa37b63b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa374150600 0x7fa3740a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:08.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.243+0000 7fa37b63b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa3740a8fc0 0x7fa374153270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:08.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.243+0000 7fa37b63b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa3740aacb0 0x7fa3741537b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:08.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.243+0000 7fa37b63b640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa374156570 con 0x7fa3740aacb0 2024-03-28T23:36:08.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.243+0000 7fa37b63b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fa3741567c0 con 0x7fa374150600 2024-03-28T23:36:08.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.243+0000 7fa37b63b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa374156aa0 con 0x7fa3740a8fc0 2024-03-28T23:36:08.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.243+0000 7fa379bd8640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa3740aacb0 0x7fa3741537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:08.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.243+0000 7fa379bd8640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa3740aacb0 0x7fa3741537b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53918/0 (socket says 172.21.15.161:53918) 2024-03-28T23:36:08.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.243+0000 7fa379bd8640 1 -- 172.21.15.161:0/1715466173 learned_addr learned my addr 172.21.15.161:0/1715466173 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:08.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.243+0000 7fa378bd6640 1 --2- 172.21.15.161:0/1715466173 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa3740a8fc0 0x7fa374153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:08.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.243+0000 7fa3793d7640 1 --2- 172.21.15.161:0/1715466173 >> v2:172.21.15.170:3300/0 conn(0x7fa374150600 0x7fa3740a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:08.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.243+0000 7fa379bd8640 1 -- 172.21.15.161:0/1715466173 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa3740a8fc0 msgr2=0x7fa374153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:08.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.243+0000 7fa379bd8640 1 --2- 172.21.15.161:0/1715466173 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa3740a8fc0 0x7fa374153270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.243+0000 7fa379bd8640 1 -- 172.21.15.161:0/1715466173 >> v2:172.21.15.170:3300/0 conn(0x7fa374150600 msgr2=0x7fa3740a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:08.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.243+0000 7fa379bd8640 1 --2- 172.21.15.161:0/1715466173 >> v2:172.21.15.170:3300/0 conn(0x7fa374150600 0x7fa3740a9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.243+0000 7fa379bd8640 1 -- 172.21.15.161:0/1715466173 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa374156d80 con 0x7fa3740aacb0 2024-03-28T23:36:08.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.244+0000 7fa379bd8640 1 --2- 172.21.15.161:0/1715466173 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3740aacb0 0x7fa3741537b0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fa374156cb0 tx=0x7fa36c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57dd2bf5936eb630 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:08.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.244+0000 7fa36bfff640 1 -- 172.21.15.161:0/1715466173 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36c02c9e0 con 0x7fa3740aacb0 2024-03-28T23:36:08.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.244+0000 7fa36bfff640 1 -- 172.21.15.161:0/1715466173 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa36c02cb80 con 0x7fa3740aacb0 2024-03-28T23:36:08.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.244+0000 7fa36bfff640 1 -- 172.21.15.161:0/1715466173 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36c0343f0 con 0x7fa3740aacb0 2024-03-28T23:36:08.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.244+0000 7fa37b63b640 1 -- 172.21.15.161:0/1715466173 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3740aacb0 msgr2=0x7fa3741537b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:08.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.244+0000 7fa37b63b640 1 --2- 172.21.15.161:0/1715466173 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3740aacb0 0x7fa3741537b0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fa374156cb0 tx=0x7fa36c02be40 comp rx=0 tx=0).stop 2024-03-28T23:36:08.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.245+0000 7fa37b63b640 1 -- 172.21.15.161:0/1715466173 shutdown_connections 2024-03-28T23:36:08.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.245+0000 7fa37b63b640 1 --2- 172.21.15.161:0/1715466173 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3740aacb0 0x7fa3741537b0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.245+0000 7fa37b63b640 1 --2- 172.21.15.161:0/1715466173 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa3740a8fc0 0x7fa374153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.245+0000 7fa37b63b640 1 --2- 172.21.15.161:0/1715466173 >> v2:172.21.15.170:3300/0 conn(0x7fa374150600 0x7fa3740a9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.245+0000 7fa37b63b640 1 -- 172.21.15.161:0/1715466173 >> 172.21.15.161:0/1715466173 conn(0x7fa3740c6670 msgr2=0x7fa3740c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:08.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.245+0000 7fa37b63b640 1 -- 172.21.15.161:0/1715466173 shutdown_connections 2024-03-28T23:36:08.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.245+0000 7fa37b63b640 1 -- 172.21.15.161:0/1715466173 wait complete. 2024-03-28T23:36:08.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.246+0000 7fa37b63b640 1 Processor -- start 2024-03-28T23:36:08.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.246+0000 7fa37b63b640 1 -- start start 2024-03-28T23:36:08.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.246+0000 7fa37b63b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3740a8fc0 0x7fa37414ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:08.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.246+0000 7fa37b63b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa3740aacb0 0x7fa374145800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:08.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.246+0000 7fa37b63b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa374150600 0x7fa374145d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:08.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.246+0000 7fa37b63b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa3741509e0 con 0x7fa3740a8fc0 2024-03-28T23:36:08.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.246+0000 7fa37b63b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa374157340 con 0x7fa374150600 2024-03-28T23:36:08.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.246+0000 7fa37b63b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa3740aa9e0 con 0x7fa3740aacb0 2024-03-28T23:36:08.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.246+0000 7fa3793d7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3740a8fc0 0x7fa37414ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:08.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.247+0000 7fa3793d7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3740a8fc0 0x7fa37414ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53930/0 (socket says 172.21.15.161:53930) 2024-03-28T23:36:08.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.247+0000 7fa3793d7640 1 -- 172.21.15.161:0/3016130117 learned_addr learned my addr 172.21.15.161:0/3016130117 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:08.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.247+0000 7fa378bd6640 1 --2- 172.21.15.161:0/3016130117 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa3740aacb0 0x7fa374145800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:08.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.247+0000 7fa379bd8640 1 --2- 172.21.15.161:0/3016130117 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa374150600 0x7fa374145d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:08.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.247+0000 7fa3793d7640 1 -- 172.21.15.161:0/3016130117 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa3740aacb0 msgr2=0x7fa374145800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:08.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.247+0000 7fa3793d7640 1 --2- 172.21.15.161:0/3016130117 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa3740aacb0 0x7fa374145800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.247+0000 7fa3793d7640 1 -- 172.21.15.161:0/3016130117 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa374150600 msgr2=0x7fa374145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:08.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.247+0000 7fa3793d7640 1 --2- 172.21.15.161:0/3016130117 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa374150600 0x7fa374145d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.247+0000 7fa3793d7640 1 -- 172.21.15.161:0/3016130117 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa374146280 con 0x7fa3740a8fc0 2024-03-28T23:36:08.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.247+0000 7fa3793d7640 1 --2- 172.21.15.161:0/3016130117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3740a8fc0 0x7fa37414ed10 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fa37000e8a0 tx=0x7fa37000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:08.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.247+0000 7fa369ffb640 1 -- 172.21.15.161:0/3016130117 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa370017070 con 0x7fa3740a8fc0 2024-03-28T23:36:08.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.247+0000 7fa369ffb640 1 -- 172.21.15.161:0/3016130117 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa37000c940 con 0x7fa3740a8fc0 2024-03-28T23:36:08.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.247+0000 7fa37b63b640 1 -- 172.21.15.161:0/3016130117 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3741eabc0 con 0x7fa3740a8fc0 2024-03-28T23:36:08.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.247+0000 7fa369ffb640 1 -- 172.21.15.161:0/3016130117 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa37000cd30 con 0x7fa3740a8fc0 2024-03-28T23:36:08.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.247+0000 7fa37b63b640 1 -- 172.21.15.161:0/3016130117 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3741eb180 con 0x7fa3740a8fc0 2024-03-28T23:36:08.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.248+0000 7fa37b63b640 1 -- 172.21.15.161:0/3016130117 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa37404f4b0 con 0x7fa3740a8fc0 2024-03-28T23:36:08.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.249+0000 7fa369ffb640 1 -- 172.21.15.161:0/3016130117 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa37001a070 con 0x7fa3740a8fc0 2024-03-28T23:36:08.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.249+0000 7fa369ffb640 1 --2- 172.21.15.161:0/3016130117 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa35806b500 0x7fa35806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:08.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.249+0000 7fa369ffb640 1 -- 172.21.15.161:0/3016130117 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 6957+0+0 (secure 0 0 0) 0x7fa37008a740 con 0x7fa3740a8fc0 2024-03-28T23:36:08.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.249+0000 7fa378bd6640 1 --2- 172.21.15.161:0/3016130117 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa35806b500 0x7fa35806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:08.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.249+0000 7fa378bd6640 1 --2- 172.21.15.161:0/3016130117 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa35806b500 0x7fa35806d9c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa3740a93c0 tx=0x7fa364009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:08.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.254+0000 7fa369ffb640 1 -- 172.21.15.161:0/3016130117 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa370058b20 con 0x7fa3740a8fc0 2024-03-28T23:36:08.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.427+0000 7fa37b63b640 1 -- 172.21.15.161:0/3016130117 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa3740a9d20 con 0x7fa3740a8fc0 2024-03-28T23:36:08.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.429+0000 7fa369ffb640 1 -- 172.21.15.161:0/3016130117 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) ==== 74+0+20651 (secure 0 0 0) 0x7fa37005cbc0 con 0x7fa3740a8fc0 2024-03-28T23:36:08.430 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:08.430 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":155,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:36:07.448631+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:35:56.687416+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-28T23:35:57.379120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:36:05.484451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:36:06.423073+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:36:08.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.431+0000 7fa37b63b640 1 -- 172.21.15.161:0/3016130117 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa35806b500 msgr2=0x7fa35806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:08.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.431+0000 7fa37b63b640 1 --2- 172.21.15.161:0/3016130117 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa35806b500 0x7fa35806d9c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa3740a93c0 tx=0x7fa364009190 comp rx=0 tx=0).stop 2024-03-28T23:36:08.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.431+0000 7fa37b63b640 1 -- 172.21.15.161:0/3016130117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3740a8fc0 msgr2=0x7fa37414ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:08.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.431+0000 7fa37b63b640 1 --2- 172.21.15.161:0/3016130117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3740a8fc0 0x7fa37414ed10 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fa37000e8a0 tx=0x7fa37000ed60 comp rx=0 tx=0).stop 2024-03-28T23:36:08.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.431+0000 7fa37b63b640 1 -- 172.21.15.161:0/3016130117 shutdown_connections 2024-03-28T23:36:08.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.431+0000 7fa37b63b640 1 --2- 172.21.15.161:0/3016130117 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa35806b500 0x7fa35806d9c0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.431+0000 7fa37b63b640 1 --2- 172.21.15.161:0/3016130117 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa374150600 0x7fa374145d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.431+0000 7fa37b63b640 1 --2- 172.21.15.161:0/3016130117 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa3740aacb0 0x7fa374145800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.431+0000 7fa37b63b640 1 --2- 172.21.15.161:0/3016130117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3740a8fc0 0x7fa37414ed10 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.431+0000 7fa37b63b640 1 -- 172.21.15.161:0/3016130117 >> 172.21.15.161:0/3016130117 conn(0x7fa3740c6670 msgr2=0x7fa37414e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:08.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.431+0000 7fa37b63b640 1 -- 172.21.15.161:0/3016130117 shutdown_connections 2024-03-28T23:36:08.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.431+0000 7fa37b63b640 1 -- 172.21.15.161:0/3016130117 wait complete. 2024-03-28T23:36:08.443 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:36:08.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.543+0000 7ffb80ccd640 1 Processor -- start 2024-03-28T23:36:08.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.543+0000 7ffb80ccd640 1 -- start start 2024-03-28T23:36:08.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.543+0000 7ffb80ccd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ffb7c14be70 0x7ffb7c0ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:08.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.543+0000 7ffb80ccd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ffb7c0ab6e0 0x7ffb7c0aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:08.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.543+0000 7ffb80ccd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb7c06ce80 0x7ffb7c0aab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:08.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.543+0000 7ffb80ccd640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ffb7c14c250 con 0x7ffb7c14be70 2024-03-28T23:36:08.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.543+0000 7ffb80ccd640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7ffb7c1529c0 con 0x7ffb7c0ab6e0 2024-03-28T23:36:08.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.543+0000 7ffb80ccd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ffb7c0a9dc0 con 0x7ffb7c06ce80 2024-03-28T23:36:08.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.544+0000 7ffb7a59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ffb7c14be70 0x7ffb7c0ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:08.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.544+0000 7ffb7a59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ffb7c14be70 0x7ffb7c0ab1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53936/0 (socket says 172.21.15.161:53936) 2024-03-28T23:36:08.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.544+0000 7ffb7a59c640 1 -- 172.21.15.161:0/3072648429 learned_addr learned my addr 172.21.15.161:0/3072648429 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:08.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.544+0000 7ffb7ad9d640 1 --2- 172.21.15.161:0/3072648429 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb7c06ce80 0x7ffb7c0aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:08.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.544+0000 7ffb71d9b640 1 --2- 172.21.15.161:0/3072648429 >> v2:172.21.15.170:3300/0 conn(0x7ffb7c0ab6e0 0x7ffb7c0aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:08.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.544+0000 7ffb7ad9d640 1 -- 172.21.15.161:0/3072648429 >> v2:172.21.15.170:3300/0 conn(0x7ffb7c0ab6e0 msgr2=0x7ffb7c0aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:08.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.544+0000 7ffb7ad9d640 1 --2- 172.21.15.161:0/3072648429 >> v2:172.21.15.170:3300/0 conn(0x7ffb7c0ab6e0 0x7ffb7c0aa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.544+0000 7ffb7ad9d640 1 -- 172.21.15.161:0/3072648429 >> v2:172.21.15.161:3300/0 conn(0x7ffb7c14be70 msgr2=0x7ffb7c0ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:08.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.544+0000 7ffb7ad9d640 1 --2- 172.21.15.161:0/3072648429 >> v2:172.21.15.161:3300/0 conn(0x7ffb7c14be70 0x7ffb7c0ab1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.544+0000 7ffb7ad9d640 1 -- 172.21.15.161:0/3072648429 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb7c0aa0a0 con 0x7ffb7c06ce80 2024-03-28T23:36:08.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.544+0000 7ffb7a59c640 1 --2- 172.21.15.161:0/3072648429 >> v2:172.21.15.161:3300/0 conn(0x7ffb7c14be70 0x7ffb7c0ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:08.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.545+0000 7ffb7ad9d640 1 --2- 172.21.15.161:0/3072648429 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb7c06ce80 0x7ffb7c0aab00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c06f540 tx=0x7ffb7402bd00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8aa0caf7da59ed6e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:08.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.545+0000 7ffb79d9b640 1 -- 172.21.15.161:0/3072648429 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb74039070 con 0x7ffb7c06ce80 2024-03-28T23:36:08.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.545+0000 7ffb79d9b640 1 -- 172.21.15.161:0/3072648429 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb7402cab0 con 0x7ffb7c06ce80 2024-03-28T23:36:08.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.545+0000 7ffb79d9b640 1 -- 172.21.15.161:0/3072648429 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb74034440 con 0x7ffb7c06ce80 2024-03-28T23:36:08.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.545+0000 7ffb80ccd640 1 -- 172.21.15.161:0/3072648429 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb7c06ce80 msgr2=0x7ffb7c0aab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:08.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.545+0000 7ffb80ccd640 1 --2- 172.21.15.161:0/3072648429 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb7c06ce80 0x7ffb7c0aab00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c06f540 tx=0x7ffb7402bd00 comp rx=0 tx=0).stop 2024-03-28T23:36:08.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.545+0000 7ffb80ccd640 1 -- 172.21.15.161:0/3072648429 shutdown_connections 2024-03-28T23:36:08.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.545+0000 7ffb80ccd640 1 --2- 172.21.15.161:0/3072648429 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb7c06ce80 0x7ffb7c0aab00 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.545+0000 7ffb80ccd640 1 --2- 172.21.15.161:0/3072648429 >> v2:172.21.15.170:3300/0 conn(0x7ffb7c0ab6e0 0x7ffb7c0aa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.545+0000 7ffb80ccd640 1 --2- 172.21.15.161:0/3072648429 >> v2:172.21.15.161:3300/0 conn(0x7ffb7c14be70 0x7ffb7c0ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.545+0000 7ffb80ccd640 1 -- 172.21.15.161:0/3072648429 >> 172.21.15.161:0/3072648429 conn(0x7ffb7c0c6670 msgr2=0x7ffb7c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:08.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.546+0000 7ffb80ccd640 1 -- 172.21.15.161:0/3072648429 shutdown_connections 2024-03-28T23:36:08.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.546+0000 7ffb80ccd640 1 -- 172.21.15.161:0/3072648429 wait complete. 2024-03-28T23:36:08.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.547+0000 7ffb80ccd640 1 Processor -- start 2024-03-28T23:36:08.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.547+0000 7ffb80ccd640 1 -- start start 2024-03-28T23:36:08.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.548+0000 7ffb80ccd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb7c06ce80 0x7ffb7c1dc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:08.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.548+0000 7ffb80ccd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb7c0ab6e0 0x7ffb7c06c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:08.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.548+0000 7ffb80ccd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb7c14be70 0x7ffb7c1e8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:08.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.548+0000 7ffb80ccd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ffb7c06b2d0 con 0x7ffb7c14be70 2024-03-28T23:36:08.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.548+0000 7ffb80ccd640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ffb7c06b150 con 0x7ffb7c06ce80 2024-03-28T23:36:08.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.548+0000 7ffb80ccd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ffb7c0c1520 con 0x7ffb7c0ab6e0 2024-03-28T23:36:08.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.548+0000 7ffb7ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb7c14be70 0x7ffb7c1e8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:08.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.548+0000 7ffb7ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb7c14be70 0x7ffb7c1e8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53950/0 (socket says 172.21.15.161:53950) 2024-03-28T23:36:08.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.548+0000 7ffb7ad9d640 1 -- 172.21.15.161:0/1878938966 learned_addr learned my addr 172.21.15.161:0/1878938966 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:08.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.548+0000 7ffb7a59c640 1 --2- 172.21.15.161:0/1878938966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb7c06ce80 0x7ffb7c1dc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:08.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.548+0000 7ffb71d9b640 1 --2- 172.21.15.161:0/1878938966 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb7c0ab6e0 0x7ffb7c06c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:08.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.548+0000 7ffb7a59c640 1 -- 172.21.15.161:0/1878938966 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb7c0ab6e0 msgr2=0x7ffb7c06c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:08.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.548+0000 7ffb7a59c640 1 --2- 172.21.15.161:0/1878938966 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb7c0ab6e0 0x7ffb7c06c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.548+0000 7ffb7a59c640 1 -- 172.21.15.161:0/1878938966 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb7c14be70 msgr2=0x7ffb7c1e8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:08.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.548+0000 7ffb7a59c640 1 --2- 172.21.15.161:0/1878938966 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb7c14be70 0x7ffb7c1e8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.548+0000 7ffb7a59c640 1 -- 172.21.15.161:0/1878938966 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb7c1dce40 con 0x7ffb7c06ce80 2024-03-28T23:36:08.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.548+0000 7ffb71d9b640 1 --2- 172.21.15.161:0/1878938966 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb7c0ab6e0 0x7ffb7c06c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:08.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.548+0000 7ffb7a59c640 1 --2- 172.21.15.161:0/1878938966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb7c06ce80 0x7ffb7c1dc900 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c00e860 tx=0x7ffb6c00ed20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:08.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.548+0000 7ffb737fe640 1 -- 172.21.15.161:0/1878938966 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb6c004030 con 0x7ffb7c06ce80 2024-03-28T23:36:08.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.548+0000 7ffb737fe640 1 -- 172.21.15.161:0/1878938966 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb6c0041d0 con 0x7ffb7c06ce80 2024-03-28T23:36:08.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.548+0000 7ffb80ccd640 1 -- 172.21.15.161:0/1878938966 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb7c14d670 con 0x7ffb7c06ce80 2024-03-28T23:36:08.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.548+0000 7ffb737fe640 1 -- 172.21.15.161:0/1878938966 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb6c00c940 con 0x7ffb7c06ce80 2024-03-28T23:36:08.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.549+0000 7ffb80ccd640 1 -- 172.21.15.161:0/1878938966 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb7c1ecb70 con 0x7ffb7c06ce80 2024-03-28T23:36:08.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.549+0000 7ffb80ccd640 1 -- 172.21.15.161:0/1878938966 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb7c04f4b0 con 0x7ffb7c06ce80 2024-03-28T23:36:08.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.550+0000 7ffb737fe640 1 -- 172.21.15.161:0/1878938966 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ffb6c00cae0 con 0x7ffb7c06ce80 2024-03-28T23:36:08.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.550+0000 7ffb737fe640 1 --2- 172.21.15.161:0/1878938966 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffb5c06b4d0 0x7ffb5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:08.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.550+0000 7ffb737fe640 1 -- 172.21.15.161:0/1878938966 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 6957+0+0 (secure 0 0 0) 0x7ffb6c08a5a0 con 0x7ffb7c06ce80 2024-03-28T23:36:08.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.550+0000 7ffb71d9b640 1 --2- 172.21.15.161:0/1878938966 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffb5c06b4d0 0x7ffb5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:08.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.551+0000 7ffb71d9b640 1 --2- 172.21.15.161:0/1878938966 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffb5c06b4d0 0x7ffb5c06d990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c1e98a0 tx=0x7ffb64012040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:08.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.555+0000 7ffb737fe640 1 -- 172.21.15.161:0/1878938966 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ffb6c017020 con 0x7ffb7c06ce80 2024-03-28T23:36:08.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.729+0000 7ffb80ccd640 1 -- 172.21.15.161:0/1878938966 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffb7c1ec6c0 con 0x7ffb7c06ce80 2024-03-28T23:36:08.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.730+0000 7ffb737fe640 1 -- 172.21.15.161:0/1878938966 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) ==== 74+0+20651 (secure 0 0 0) 0x7ffb6c058980 con 0x7ffb7c06ce80 2024-03-28T23:36:08.731 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:08.731 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":155,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:36:07.448631+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:35:56.687416+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-28T23:35:57.379120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:36:05.484451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:36:06.423073+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:36:08.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.732+0000 7ffb80ccd640 1 -- 172.21.15.161:0/1878938966 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffb5c06b4d0 msgr2=0x7ffb5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:08.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.732+0000 7ffb80ccd640 1 --2- 172.21.15.161:0/1878938966 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffb5c06b4d0 0x7ffb5c06d990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c1e98a0 tx=0x7ffb64012040 comp rx=0 tx=0).stop 2024-03-28T23:36:08.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.732+0000 7ffb80ccd640 1 -- 172.21.15.161:0/1878938966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb7c06ce80 msgr2=0x7ffb7c1dc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:08.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.732+0000 7ffb80ccd640 1 --2- 172.21.15.161:0/1878938966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb7c06ce80 0x7ffb7c1dc900 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c00e860 tx=0x7ffb6c00ed20 comp rx=0 tx=0).stop 2024-03-28T23:36:08.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.732+0000 7ffb80ccd640 1 -- 172.21.15.161:0/1878938966 shutdown_connections 2024-03-28T23:36:08.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.732+0000 7ffb80ccd640 1 --2- 172.21.15.161:0/1878938966 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ffb5c06b4d0 0x7ffb5c06d990 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.732+0000 7ffb80ccd640 1 --2- 172.21.15.161:0/1878938966 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb7c14be70 0x7ffb7c1e8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.732+0000 7ffb80ccd640 1 --2- 172.21.15.161:0/1878938966 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb7c0ab6e0 0x7ffb7c06c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.732+0000 7ffb80ccd640 1 --2- 172.21.15.161:0/1878938966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb7c06ce80 0x7ffb7c1dc900 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.732+0000 7ffb80ccd640 1 -- 172.21.15.161:0/1878938966 >> 172.21.15.161:0/1878938966 conn(0x7ffb7c0c6670 msgr2=0x7ffb7c06b480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:08.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.732+0000 7ffb80ccd640 1 -- 172.21.15.161:0/1878938966 shutdown_connections 2024-03-28T23:36:08.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.732+0000 7ffb80ccd640 1 -- 172.21.15.161:0/1878938966 wait complete. 2024-03-28T23:36:08.744 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 standby_count_wanted 0 2024-03-28T23:36:08.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.848+0000 7f90430e1640 1 Processor -- start 2024-03-28T23:36:08.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.848+0000 7f90430e1640 1 -- start start 2024-03-28T23:36:08.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.849+0000 7f90430e1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f903c0c7330 0x7f903c14e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:08.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.849+0000 7f90430e1640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f903c0733e0 0x7f903c14e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:08.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.849+0000 7f90430e1640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f903c0abc00 con 0x7f903c0a9960 2024-03-28T23:36:08.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.849+0000 7f90430e1640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f903c0aa3a0 con 0x7f903c0733e0 2024-03-28T23:36:08.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.849+0000 7f90430e1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f903c06bf70 con 0x7f903c0c7330 2024-03-28T23:36:08.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.849+0000 7f903bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f903c0c7330 0x7f903c14e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:08.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.849+0000 7f9040e7d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f903c0a9960 0x7f903c072970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50148/0 (socket says 172.21.15.161:50148) 2024-03-28T23:36:08.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.849+0000 7f904167e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f903c0733e0 0x7f903c14e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:08.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.849+0000 7f9040e7d640 1 -- 172.21.15.161:0/873746011 learned_addr learned my addr 172.21.15.161:0/873746011 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:08.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.849+0000 7f904167e640 1 -- 172.21.15.161:0/873746011 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f903c0c7330 msgr2=0x7f903c14e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:08.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.849+0000 7f904167e640 1 --2- 172.21.15.161:0/873746011 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f903c0c7330 0x7f903c14e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.849+0000 7f904167e640 1 -- 172.21.15.161:0/873746011 >> v1:172.21.15.161:6789/0 conn(0x7f903c0a9960 legacy=0x7f903c072970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:08.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.849+0000 7f903b7fe640 1 -- 172.21.15.161:0/873746011 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2930414012 0 0) 0x7f903c0abc00 con 0x7f903c0a9960 2024-03-28T23:36:08.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.849+0000 7f904167e640 1 -- 172.21.15.161:0/873746011 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f903c06bcd0 con 0x7f903c0733e0 2024-03-28T23:36:08.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.849+0000 7f903bfff640 1 --2- 172.21.15.161:0/873746011 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f903c0c7330 0x7f903c14e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:08.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.850+0000 7f904167e640 1 --2- 172.21.15.161:0/873746011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f903c0733e0 0x7f903c14e9a0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f9034009d90 tx=0x7f903402bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a4873182bd831ca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:08.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.850+0000 7f903b7fe640 1 -- 172.21.15.161:0/873746011 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9034039070 con 0x7f903c0733e0 2024-03-28T23:36:08.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.850+0000 7f903b7fe640 1 -- 172.21.15.161:0/873746011 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f903402cb30 con 0x7f903c0733e0 2024-03-28T23:36:08.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.850+0000 7f903b7fe640 1 -- 172.21.15.161:0/873746011 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9034034440 con 0x7f903c0733e0 2024-03-28T23:36:08.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.851+0000 7f90430e1640 1 -- 172.21.15.161:0/873746011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f903c0733e0 msgr2=0x7f903c14e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:08.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.851+0000 7f90430e1640 1 --2- 172.21.15.161:0/873746011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f903c0733e0 0x7f903c14e9a0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f9034009d90 tx=0x7f903402bd80 comp rx=0 tx=0).stop 2024-03-28T23:36:08.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.851+0000 7f90430e1640 1 -- 172.21.15.161:0/873746011 shutdown_connections 2024-03-28T23:36:08.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.851+0000 7f90430e1640 1 --2- 172.21.15.161:0/873746011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f903c0733e0 0x7f903c14e9a0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.851+0000 7f90430e1640 1 --2- 172.21.15.161:0/873746011 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f903c0c7330 0x7f903c14e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.851+0000 7f90430e1640 1 -- 172.21.15.161:0/873746011 >> 172.21.15.161:0/873746011 conn(0x7f903c0a83f0 msgr2=0x7f903c0a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:08.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.851+0000 7f90430e1640 1 -- 172.21.15.161:0/873746011 shutdown_connections 2024-03-28T23:36:08.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.851+0000 7f90430e1640 1 -- 172.21.15.161:0/873746011 wait complete. 2024-03-28T23:36:08.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.852+0000 7f90430e1640 1 Processor -- start 2024-03-28T23:36:08.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.852+0000 7f90430e1640 1 -- start start 2024-03-28T23:36:08.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.853+0000 7f90430e1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f903c0733e0 0x7f903c0c61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:08.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.853+0000 7f90430e1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f903c0a9960 0x7f903c14c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:08.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.853+0000 7f90430e1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f903c0c7330 0x7f903c152910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:08.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.853+0000 7f90430e1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f903c073080 con 0x7f903c0c7330 2024-03-28T23:36:08.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.853+0000 7f90430e1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f903c152250 con 0x7f903c0733e0 2024-03-28T23:36:08.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.853+0000 7f90430e1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f903c073200 con 0x7f903c0a9960 2024-03-28T23:36:08.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.853+0000 7f904167e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f903c0c7330 0x7f903c152910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:08.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.853+0000 7f903bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f903c0a9960 0x7f903c14c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:08.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.853+0000 7f904167e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f903c0c7330 0x7f903c152910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53964/0 (socket says 172.21.15.161:53964) 2024-03-28T23:36:08.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.853+0000 7f904167e640 1 -- 172.21.15.161:0/3619514293 learned_addr learned my addr 172.21.15.161:0/3619514293 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:08.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.853+0000 7f9040e7d640 1 --2- 172.21.15.161:0/3619514293 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f903c0733e0 0x7f903c0c61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:08.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.853+0000 7f903bfff640 1 -- 172.21.15.161:0/3619514293 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f903c0733e0 msgr2=0x7f903c0c61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:08.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.853+0000 7f903bfff640 1 --2- 172.21.15.161:0/3619514293 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f903c0733e0 0x7f903c0c61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.853+0000 7f903bfff640 1 -- 172.21.15.161:0/3619514293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f903c0c7330 msgr2=0x7f903c152910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:08.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.853+0000 7f903bfff640 1 --2- 172.21.15.161:0/3619514293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f903c0c7330 0x7f903c152910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:08.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.853+0000 7f903bfff640 1 -- 172.21.15.161:0/3619514293 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f903c14dac0 con 0x7f903c0a9960 2024-03-28T23:36:08.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.853+0000 7f904167e640 1 --2- 172.21.15.161:0/3619514293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f903c0c7330 0x7f903c152910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:08.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.853+0000 7f903bfff640 1 --2- 172.21.15.161:0/3619514293 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f903c0a9960 0x7f903c14c3c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f902800e870 tx=0x7f902800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:08.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.853+0000 7f90397fa640 1 -- 172.21.15.161:0/3619514293 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9028017070 con 0x7f903c0a9960 2024-03-28T23:36:08.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.853+0000 7f90397fa640 1 -- 172.21.15.161:0/3619514293 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f902800c940 con 0x7f903c0a9960 2024-03-28T23:36:08.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.853+0000 7f90430e1640 1 -- 172.21.15.161:0/3619514293 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f903c14dda0 con 0x7f903c0a9960 2024-03-28T23:36:08.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.853+0000 7f90397fa640 1 -- 172.21.15.161:0/3619514293 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902800cd10 con 0x7f903c0a9960 2024-03-28T23:36:08.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.854+0000 7f90430e1640 1 -- 172.21.15.161:0/3619514293 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f903c14e360 con 0x7f903c0a9960 2024-03-28T23:36:08.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.854+0000 7f90430e1640 1 -- 172.21.15.161:0/3619514293 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f903c04f4b0 con 0x7f903c0a9960 2024-03-28T23:36:08.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.855+0000 7f90397fa640 1 -- 172.21.15.161:0/3619514293 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f902801a070 con 0x7f903c0a9960 2024-03-28T23:36:08.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.855+0000 7f90397fa640 1 --2- 172.21.15.161:0/3619514293 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f902006b500 0x7f902006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:08.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.855+0000 7f90397fa640 1 -- 172.21.15.161:0/3619514293 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(155..155 src has 1..155) ==== 6957+0+0 (secure 0 0 0) 0x7f902808a6e0 con 0x7f903c0a9960 2024-03-28T23:36:08.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.855+0000 7f9040e7d640 1 --2- 172.21.15.161:0/3619514293 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f902006b500 0x7f902006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:08.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.856+0000 7f9040e7d640 1 --2- 172.21.15.161:0/3619514293 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f902006b500 0x7f902006d9c0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f9030000f60 tx=0x7f9030007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:08.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:08.860+0000 7f90397fa640 1 -- 172.21.15.161:0/3619514293 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9028058ac0 con 0x7f903c0a9960 2024-03-28T23:36:09.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:09.075+0000 7f90430e1640 1 -- 172.21.15.161:0/3619514293 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f903c0a8870 con 0x7f903c0a9960 2024-03-28T23:36:09.337 INFO:tasks.ceph.mds.a.smithi161.stderr:2024-03-28T23:36:09.337+0000 7f4944c2e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:36:09.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:09.974+0000 7f90397fa640 1 -- 172.21.15.161:0/3619514293 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"}]=0 v143) ==== 119+0+0 (secure 0 0 0) 0x7f902805cb60 con 0x7f903c0a9960 2024-03-28T23:36:09.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:09.976+0000 7f90430e1640 1 -- 172.21.15.161:0/3619514293 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f902006b500 msgr2=0x7f902006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:09.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:09.976+0000 7f90430e1640 1 --2- 172.21.15.161:0/3619514293 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f902006b500 0x7f902006d9c0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f9030000f60 tx=0x7f9030007d50 comp rx=0 tx=0).stop 2024-03-28T23:36:09.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:09.976+0000 7f90430e1640 1 -- 172.21.15.161:0/3619514293 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f903c0a9960 msgr2=0x7f903c14c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:09.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:09.976+0000 7f90430e1640 1 --2- 172.21.15.161:0/3619514293 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f903c0a9960 0x7f903c14c3c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f902800e870 tx=0x7f902800ed30 comp rx=0 tx=0).stop 2024-03-28T23:36:09.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:09.977+0000 7f90430e1640 1 -- 172.21.15.161:0/3619514293 shutdown_connections 2024-03-28T23:36:09.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:09.977+0000 7f90430e1640 1 --2- 172.21.15.161:0/3619514293 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f902006b500 0x7f902006d9c0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:09.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:09.977+0000 7f90430e1640 1 --2- 172.21.15.161:0/3619514293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f903c0c7330 0x7f903c152910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:09.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:09.977+0000 7f90430e1640 1 --2- 172.21.15.161:0/3619514293 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f903c0a9960 0x7f903c14c3c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:09.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:09.977+0000 7f90430e1640 1 --2- 172.21.15.161:0/3619514293 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f903c0733e0 0x7f903c0c61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:09.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:09.977+0000 7f90430e1640 1 -- 172.21.15.161:0/3619514293 >> 172.21.15.161:0/3619514293 conn(0x7f903c0a83f0 msgr2=0x7f903c151990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:09.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:09.977+0000 7f90430e1640 1 -- 172.21.15.161:0/3619514293 shutdown_connections 2024-03-28T23:36:09.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:09.977+0000 7f90430e1640 1 -- 172.21.15.161:0/3619514293 wait complete. 2024-03-28T23:36:09.988 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-28T23:36:09.988 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 allow_standby_replay false 2024-03-28T23:36:10.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.091+0000 7fe42a840640 1 Processor -- start 2024-03-28T23:36:10.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.091+0000 7fe42a840640 1 -- start start 2024-03-28T23:36:10.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.092+0000 7fe42a840640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe424152bc0 0x7fe4240c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:10.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.092+0000 7fe42a840640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe4240c8a80 0x7fe4240c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:10.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.092+0000 7fe42a840640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4240c6f60 con 0x7fe4240c9420 2024-03-28T23:36:10.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.092+0000 7fe42a840640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fe4240c6120 con 0x7fe4240c8a80 2024-03-28T23:36:10.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.092+0000 7fe42a840640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe4240c6400 con 0x7fe424152bc0 2024-03-28T23:36:10.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.092+0000 7fe428ddd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe4240c9420 0x7fe4241589b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50150/0 (socket says 172.21.15.161:50150) 2024-03-28T23:36:10.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.092+0000 7fe428ddd640 1 -- 172.21.15.161:0/4275699853 learned_addr learned my addr 172.21.15.161:0/4275699853 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:10.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.092+0000 7fe423fff640 1 --2- 172.21.15.161:0/4275699853 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe424152bc0 0x7fe4240c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:10.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.092+0000 7fe4237fe640 1 --2- 172.21.15.161:0/4275699853 >> v2:172.21.15.170:3300/0 conn(0x7fe4240c8a80 0x7fe4240c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:10.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.093+0000 7fe422ffd640 1 -- 172.21.15.161:0/4275699853 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3421875828 0 0) 0x7fe4240c6f60 con 0x7fe4240c9420 2024-03-28T23:36:10.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.093+0000 7fe422ffd640 1 -- 172.21.15.161:0/4275699853 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe40c003520 con 0x7fe4240c9420 2024-03-28T23:36:10.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.093+0000 7fe423fff640 1 -- 172.21.15.161:0/4275699853 >> v2:172.21.15.170:3300/0 conn(0x7fe4240c8a80 msgr2=0x7fe4240c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:10.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.093+0000 7fe423fff640 1 --2- 172.21.15.161:0/4275699853 >> v2:172.21.15.170:3300/0 conn(0x7fe4240c8a80 0x7fe4240c8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:10.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.093+0000 7fe423fff640 1 -- 172.21.15.161:0/4275699853 >> v1:172.21.15.161:6789/0 conn(0x7fe4240c9420 legacy=0x7fe4241589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:10.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.093+0000 7fe423fff640 1 -- 172.21.15.161:0/4275699853 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4240c7240 con 0x7fe424152bc0 2024-03-28T23:36:10.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.093+0000 7fe4237fe640 1 --2- 172.21.15.161:0/4275699853 >> v2:172.21.15.170:3300/0 conn(0x7fe4240c8a80 0x7fe4240c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:10.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.093+0000 7fe423fff640 1 --2- 172.21.15.161:0/4275699853 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe424152bc0 0x7fe4240c0330 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe4180097a0 tx=0x7fe41802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d86f5d1e0cdab32b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:10.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.094+0000 7fe422ffd640 1 -- 172.21.15.161:0/4275699853 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe418035020 con 0x7fe424152bc0 2024-03-28T23:36:10.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.094+0000 7fe422ffd640 1 -- 172.21.15.161:0/4275699853 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe418031830 con 0x7fe424152bc0 2024-03-28T23:36:10.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.094+0000 7fe422ffd640 1 -- 172.21.15.161:0/4275699853 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe418031c00 con 0x7fe424152bc0 2024-03-28T23:36:10.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.094+0000 7fe42a840640 1 -- 172.21.15.161:0/4275699853 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe424152bc0 msgr2=0x7fe4240c0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:10.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.094+0000 7fe42a840640 1 --2- 172.21.15.161:0/4275699853 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe424152bc0 0x7fe4240c0330 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe4180097a0 tx=0x7fe41802b8d0 comp rx=0 tx=0).stop 2024-03-28T23:36:10.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.094+0000 7fe42a840640 1 -- 172.21.15.161:0/4275699853 shutdown_connections 2024-03-28T23:36:10.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.094+0000 7fe42a840640 1 --2- 172.21.15.161:0/4275699853 >> v2:172.21.15.170:3300/0 conn(0x7fe4240c8a80 0x7fe4240c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:10.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.094+0000 7fe42a840640 1 --2- 172.21.15.161:0/4275699853 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe424152bc0 0x7fe4240c0330 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:10.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.094+0000 7fe42a840640 1 -- 172.21.15.161:0/4275699853 >> 172.21.15.161:0/4275699853 conn(0x7fe4240caa80 msgr2=0x7fe4240cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:10.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.094+0000 7fe42a840640 1 -- 172.21.15.161:0/4275699853 shutdown_connections 2024-03-28T23:36:10.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.094+0000 7fe42a840640 1 -- 172.21.15.161:0/4275699853 wait complete. 2024-03-28T23:36:10.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.095+0000 7fe42a840640 1 Processor -- start 2024-03-28T23:36:10.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.095+0000 7fe42a840640 1 -- start start 2024-03-28T23:36:10.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.096+0000 7fe42a840640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4240c8a80 0x7fe424157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:10.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.096+0000 7fe42a840640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4240c9420 0x7fe4241571f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:10.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.096+0000 7fe42a840640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe424152bc0 0x7fe4241e7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:10.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.096+0000 7fe42a840640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe424152fa0 con 0x7fe424152bc0 2024-03-28T23:36:10.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.096+0000 7fe42a840640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241590c0 con 0x7fe4240c8a80 2024-03-28T23:36:10.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.096+0000 7fe42a840640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe424158e50 con 0x7fe4240c9420 2024-03-28T23:36:10.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.096+0000 7fe428ddd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe424152bc0 0x7fe4241e7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:10.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.096+0000 7fe428ddd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe424152bc0 0x7fe4241e7b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53968/0 (socket says 172.21.15.161:53968) 2024-03-28T23:36:10.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.096+0000 7fe428ddd640 1 -- 172.21.15.161:0/3701537393 learned_addr learned my addr 172.21.15.161:0/3701537393 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:10.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.096+0000 7fe4237fe640 1 --2- 172.21.15.161:0/3701537393 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4240c9420 0x7fe4241571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:10.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.096+0000 7fe423fff640 1 --2- 172.21.15.161:0/3701537393 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4240c8a80 0x7fe424157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:10.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.097+0000 7fe4237fe640 1 -- 172.21.15.161:0/3701537393 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4240c8a80 msgr2=0x7fe424157d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:10.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.097+0000 7fe4237fe640 1 --2- 172.21.15.161:0/3701537393 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4240c8a80 0x7fe424157d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:10.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.097+0000 7fe4237fe640 1 -- 172.21.15.161:0/3701537393 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe424152bc0 msgr2=0x7fe4241e7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:10.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.097+0000 7fe4237fe640 1 --2- 172.21.15.161:0/3701537393 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe424152bc0 0x7fe4241e7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:10.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.097+0000 7fe4237fe640 1 -- 172.21.15.161:0/3701537393 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4240c2590 con 0x7fe4240c9420 2024-03-28T23:36:10.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.097+0000 7fe428ddd640 1 --2- 172.21.15.161:0/3701537393 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe424152bc0 0x7fe4241e7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:10.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.097+0000 7fe4237fe640 1 --2- 172.21.15.161:0/3701537393 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4240c9420 0x7fe4241571f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe41400e930 tx=0x7fe41400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:10.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.097+0000 7fe420ff9640 1 -- 172.21.15.161:0/3701537393 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe414017070 con 0x7fe4240c9420 2024-03-28T23:36:10.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.097+0000 7fe420ff9640 1 -- 172.21.15.161:0/3701537393 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe41400ca00 con 0x7fe4240c9420 2024-03-28T23:36:10.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.097+0000 7fe420ff9640 1 -- 172.21.15.161:0/3701537393 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41400cdd0 con 0x7fe4240c9420 2024-03-28T23:36:10.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.097+0000 7fe42a840640 1 -- 172.21.15.161:0/3701537393 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe424158370 con 0x7fe4240c9420 2024-03-28T23:36:10.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.097+0000 7fe42a840640 1 -- 172.21.15.161:0/3701537393 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe424158930 con 0x7fe4240c9420 2024-03-28T23:36:10.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.098+0000 7fe42a840640 1 -- 172.21.15.161:0/3701537393 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe42404f4b0 con 0x7fe4240c9420 2024-03-28T23:36:10.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.099+0000 7fe420ff9640 1 -- 172.21.15.161:0/3701537393 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe41401a070 con 0x7fe4240c9420 2024-03-28T23:36:10.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.099+0000 7fe420ff9640 1 --2- 172.21.15.161:0/3701537393 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe41006b4d0 0x7fe41006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:10.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.099+0000 7fe420ff9640 1 -- 172.21.15.161:0/3701537393 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(156..156 src has 1..156) ==== 6957+0+0 (secure 0 0 0) 0x7fe41408a800 con 0x7fe4240c9420 2024-03-28T23:36:10.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.099+0000 7fe423fff640 1 --2- 172.21.15.161:0/3701537393 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe41006b4d0 0x7fe41006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:10.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.099+0000 7fe423fff640 1 --2- 172.21.15.161:0/3701537393 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe41006b4d0 0x7fe41006d990 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fe418036000 tx=0x7fe41803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:10.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.103+0000 7fe420ff9640 1 -- 172.21.15.161:0/3701537393 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe414058be0 con 0x7fe4240c9420 2024-03-28T23:36:10.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:10.477+0000 7fe42a840640 1 -- 172.21.15.161:0/3701537393 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fe42414f2a0 con 0x7fe4240c9420 2024-03-28T23:36:11.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.122+0000 7fe420ff9640 1 -- 172.21.15.161:0/3701537393 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"}]=0 v144) ==== 123+0+0 (secure 0 0 0) 0x7fe41405cc80 con 0x7fe4240c9420 2024-03-28T23:36:11.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.124+0000 7fe42a840640 1 -- 172.21.15.161:0/3701537393 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe41006b4d0 msgr2=0x7fe41006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:11.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.124+0000 7fe42a840640 1 --2- 172.21.15.161:0/3701537393 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe41006b4d0 0x7fe41006d990 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fe418036000 tx=0x7fe41803c040 comp rx=0 tx=0).stop 2024-03-28T23:36:11.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.124+0000 7fe42a840640 1 -- 172.21.15.161:0/3701537393 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4240c9420 msgr2=0x7fe4241571f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:11.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.124+0000 7fe42a840640 1 --2- 172.21.15.161:0/3701537393 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4240c9420 0x7fe4241571f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe41400e930 tx=0x7fe41400edf0 comp rx=0 tx=0).stop 2024-03-28T23:36:11.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.124+0000 7fe42a840640 1 -- 172.21.15.161:0/3701537393 shutdown_connections 2024-03-28T23:36:11.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.124+0000 7fe42a840640 1 --2- 172.21.15.161:0/3701537393 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe41006b4d0 0x7fe41006d990 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:11.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.124+0000 7fe42a840640 1 --2- 172.21.15.161:0/3701537393 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe424152bc0 0x7fe4241e7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:11.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.124+0000 7fe42a840640 1 --2- 172.21.15.161:0/3701537393 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4240c9420 0x7fe4241571f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:11.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.124+0000 7fe42a840640 1 --2- 172.21.15.161:0/3701537393 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4240c8a80 0x7fe424157d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:11.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.124+0000 7fe42a840640 1 -- 172.21.15.161:0/3701537393 >> 172.21.15.161:0/3701537393 conn(0x7fe4240caa80 msgr2=0x7fe42414e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:11.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.124+0000 7fe42a840640 1 -- 172.21.15.161:0/3701537393 shutdown_connections 2024-03-28T23:36:11.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.124+0000 7fe42a840640 1 -- 172.21.15.161:0/3701537393 wait complete. 2024-03-28T23:36:11.139 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs2' 2024-03-28T23:36:11.139 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:36:11.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.241+0000 7f66dca15640 1 Processor -- start 2024-03-28T23:36:11.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.241+0000 7f66dca15640 1 -- start start 2024-03-28T23:36:11.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.242+0000 7f66dca15640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f66d8152890 0x7f66d80c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:11.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.242+0000 7f66dca15640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66d8158a30 con 0x7f66d80c8ed0 2024-03-28T23:36:11.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.242+0000 7f66dca15640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66d8158d10 con 0x7f66d80c5b20 2024-03-28T23:36:11.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.242+0000 7f66dca15640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f66d80c8480 con 0x7f66d8152890 2024-03-28T23:36:11.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.242+0000 7f66d659c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f66d8152890 0x7f66d80c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:11.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.242+0000 7f66d659c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f66d8152890 0x7f66d80c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46594/0 (socket says 172.21.15.161:46594) 2024-03-28T23:36:11.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.242+0000 7f66cffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f66d80c8ed0 0x7f66d80c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41706/0 (socket says 172.21.15.161:41706) 2024-03-28T23:36:11.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.242+0000 7f66d659c640 1 -- 172.21.15.161:0/3702208815 learned_addr learned my addr 172.21.15.161:0/3702208815 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:11.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.242+0000 7f66d659c640 1 -- 172.21.15.161:0/3702208815 >> v1:172.21.15.170:6789/0 conn(0x7f66d80c5b20 legacy=0x7f66d81553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:11.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.242+0000 7f66d659c640 1 -- 172.21.15.161:0/3702208815 >> v1:172.21.15.161:6789/0 conn(0x7f66d80c8ed0 legacy=0x7f66d80c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:11.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.242+0000 7f66d659c640 1 -- 172.21.15.161:0/3702208815 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66d8158ff0 con 0x7f66d8152890 2024-03-28T23:36:11.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.243+0000 7f66d659c640 1 --2- 172.21.15.161:0/3702208815 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66d8152890 0x7f66d80c8990 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f66c8009870 tx=0x7f66c802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc18d07291d9e860 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:11.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.243+0000 7f66d5d9b640 1 -- 172.21.15.161:0/3702208815 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c8035020 con 0x7f66d8152890 2024-03-28T23:36:11.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.243+0000 7f66d5d9b640 1 -- 172.21.15.161:0/3702208815 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66c8031870 con 0x7f66d8152890 2024-03-28T23:36:11.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.243+0000 7f66d5d9b640 1 -- 172.21.15.161:0/3702208815 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c8031c40 con 0x7f66d8152890 2024-03-28T23:36:11.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.244+0000 7f66dca15640 1 -- 172.21.15.161:0/3702208815 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66d8152890 msgr2=0x7f66d80c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:11.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.244+0000 7f66dca15640 1 --2- 172.21.15.161:0/3702208815 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66d8152890 0x7f66d80c8990 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f66c8009870 tx=0x7f66c802b910 comp rx=0 tx=0).stop 2024-03-28T23:36:11.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.244+0000 7f66dca15640 1 -- 172.21.15.161:0/3702208815 shutdown_connections 2024-03-28T23:36:11.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.244+0000 7f66dca15640 1 --2- 172.21.15.161:0/3702208815 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66d8152890 0x7f66d80c8990 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:11.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.244+0000 7f66dca15640 1 -- 172.21.15.161:0/3702208815 >> 172.21.15.161:0/3702208815 conn(0x7f66d80ac620 msgr2=0x7f66d80aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:11.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.244+0000 7f66dca15640 1 -- 172.21.15.161:0/3702208815 shutdown_connections 2024-03-28T23:36:11.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.244+0000 7f66dca15640 1 -- 172.21.15.161:0/3702208815 wait complete. 2024-03-28T23:36:11.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.245+0000 7f66dca15640 1 Processor -- start 2024-03-28T23:36:11.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.245+0000 7f66dca15640 1 -- start start 2024-03-28T23:36:11.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.246+0000 7f66dca15640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66d80c5b20 0x7f66d814f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:11.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.246+0000 7f66dca15640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66d80c8ed0 0x7f66d81eb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:11.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.246+0000 7f66dca15640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66d8152890 0x7f66d8148830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:11.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.246+0000 7f66dca15640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f66d8152c70 con 0x7f66d80c8ed0 2024-03-28T23:36:11.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.246+0000 7f66dca15640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f66d8159640 con 0x7f66d8152890 2024-03-28T23:36:11.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.246+0000 7f66dca15640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f66d8152df0 con 0x7f66d80c5b20 2024-03-28T23:36:11.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.246+0000 7f66cffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66d80c8ed0 0x7f66d81eb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:11.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.246+0000 7f66cffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66d80c8ed0 0x7f66d81eb610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46602/0 (socket says 172.21.15.161:46602) 2024-03-28T23:36:11.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.246+0000 7f66cffff640 1 -- 172.21.15.161:0/3106387903 learned_addr learned my addr 172.21.15.161:0/3106387903 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:11.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.246+0000 7f66d659c640 1 --2- 172.21.15.161:0/3106387903 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66d80c5b20 0x7f66d814f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:11.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.246+0000 7f66d6d9d640 1 --2- 172.21.15.161:0/3106387903 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66d8152890 0x7f66d8148830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:11.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.246+0000 7f66cffff640 1 -- 172.21.15.161:0/3106387903 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66d80c5b20 msgr2=0x7f66d814f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:11.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.246+0000 7f66cffff640 1 --2- 172.21.15.161:0/3106387903 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66d80c5b20 0x7f66d814f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:11.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.246+0000 7f66cffff640 1 -- 172.21.15.161:0/3106387903 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66d8152890 msgr2=0x7f66d8148830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:11.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.246+0000 7f66cffff640 1 --2- 172.21.15.161:0/3106387903 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66d8152890 0x7f66d8148830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:11.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.246+0000 7f66cffff640 1 -- 172.21.15.161:0/3106387903 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66d81eae80 con 0x7f66d80c8ed0 2024-03-28T23:36:11.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.246+0000 7f66d6d9d640 1 --2- 172.21.15.161:0/3106387903 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66d8152890 0x7f66d8148830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:36:11.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.246+0000 7f66cffff640 1 --2- 172.21.15.161:0/3106387903 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66d80c8ed0 0x7f66d81eb610 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f66c000c4d0 tx=0x7f66c000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:11.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.247+0000 7f66ceffd640 1 -- 172.21.15.161:0/3106387903 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c0017070 con 0x7f66d80c8ed0 2024-03-28T23:36:11.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.247+0000 7f66ceffd640 1 -- 172.21.15.161:0/3106387903 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66c0007480 con 0x7f66d80c8ed0 2024-03-28T23:36:11.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.247+0000 7f66dca15640 1 -- 172.21.15.161:0/3106387903 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66d81eee60 con 0x7f66d80c8ed0 2024-03-28T23:36:11.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.247+0000 7f66ceffd640 1 -- 172.21.15.161:0/3106387903 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c00059a0 con 0x7f66d80c8ed0 2024-03-28T23:36:11.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.247+0000 7f66dca15640 1 -- 172.21.15.161:0/3106387903 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66d81ef100 con 0x7f66d80c8ed0 2024-03-28T23:36:11.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.248+0000 7f66dca15640 1 -- 172.21.15.161:0/3106387903 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66d8068f90 con 0x7f66d80c8ed0 2024-03-28T23:36:11.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.248+0000 7f66ceffd640 1 -- 172.21.15.161:0/3106387903 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f66c001a070 con 0x7f66d80c8ed0 2024-03-28T23:36:11.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.249+0000 7f66ceffd640 1 --2- 172.21.15.161:0/3106387903 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f66b806b4d0 0x7f66b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:11.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.249+0000 7f66ceffd640 1 -- 172.21.15.161:0/3106387903 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 6957+0+0 (secure 0 0 0) 0x7f66c0089860 con 0x7f66d80c8ed0 2024-03-28T23:36:11.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.249+0000 7f66d659c640 1 --2- 172.21.15.161:0/3106387903 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f66b806b4d0 0x7f66b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:11.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.249+0000 7f66d659c640 1 --2- 172.21.15.161:0/3106387903 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f66b806b4d0 0x7f66b806d990 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f66c8036000 tx=0x7f66c803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:11.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.256+0000 7f66ceffd640 1 -- 172.21.15.161:0/3106387903 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f66d8068f90 con 0x7f66d80c8ed0 2024-03-28T23:36:11.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.578+0000 7f66dca15640 1 -- 172.21.15.161:0/3106387903 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66d804f4b0 con 0x7f66d80c8ed0 2024-03-28T23:36:11.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.579+0000 7f66ceffd640 1 -- 172.21.15.161:0/3106387903 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) ==== 95+0+6519 (secure 0 0 0) 0x7f66c0057c40 con 0x7f66d80c8ed0 2024-03-28T23:36:11.579 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:11.580 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":144,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7061,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4026332703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4026332703},{"type":"v1","addr":"172.21.15.161:6836","nonce":4026332703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":7070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/557522063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":557522063},{"type":"v1","addr":"172.21.15.170:6835","nonce":557522063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:58.406403+0000","modified":"2024-03-28T23:36:00.406209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8068},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8068":{"gid":8068,"name":"b","rank":0,"incarnation":138,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6834/1393137291","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1393137291},{"type":"v1","addr":"172.21.15.170:6834","nonce":1393137291}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8068,"qdb_cluster":[8068]},"id":11},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:36:07.448976+0000","modified":"2024-03-28T23:36:10.478828+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8056":{"gid":8056,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":5,"addr":"172.21.15.161:6837/2391019771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":2391019771},{"type":"v1","addr":"172.21.15.161:6837","nonce":2391019771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8056,"qdb_cluster":[8056]},"id":12}]} 2024-03-28T23:36:11.580 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 144 2024-03-28T23:36:11.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.581+0000 7f66dca15640 1 -- 172.21.15.161:0/3106387903 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f66b806b4d0 msgr2=0x7f66b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:11.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.581+0000 7f66dca15640 1 --2- 172.21.15.161:0/3106387903 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f66b806b4d0 0x7f66b806d990 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f66c8036000 tx=0x7f66c803c040 comp rx=0 tx=0).stop 2024-03-28T23:36:11.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.581+0000 7f66dca15640 1 -- 172.21.15.161:0/3106387903 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66d80c8ed0 msgr2=0x7f66d81eb610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:11.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.581+0000 7f66dca15640 1 --2- 172.21.15.161:0/3106387903 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66d80c8ed0 0x7f66d81eb610 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f66c000c4d0 tx=0x7f66c000c990 comp rx=0 tx=0).stop 2024-03-28T23:36:11.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.581+0000 7f66dca15640 1 -- 172.21.15.161:0/3106387903 shutdown_connections 2024-03-28T23:36:11.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.581+0000 7f66dca15640 1 --2- 172.21.15.161:0/3106387903 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f66b806b4d0 0x7f66b806d990 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:11.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.581+0000 7f66dca15640 1 --2- 172.21.15.161:0/3106387903 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66d80c8ed0 0x7f66d81eb610 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:11.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.581+0000 7f66dca15640 1 --2- 172.21.15.161:0/3106387903 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66d8152890 0x7f66d8148830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:11.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.581+0000 7f66dca15640 1 --2- 172.21.15.161:0/3106387903 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66d80c5b20 0x7f66d814f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:11.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.581+0000 7f66dca15640 1 -- 172.21.15.161:0/3106387903 >> 172.21.15.161:0/3106387903 conn(0x7f66d80ac620 msgr2=0x7f66d80c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:11.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.581+0000 7f66dca15640 1 -- 172.21.15.161:0/3106387903 shutdown_connections 2024-03-28T23:36:11.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.581+0000 7f66dca15640 1 -- 172.21.15.161:0/3106387903 wait complete. 2024-03-28T23:36:11.592 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:36:11.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.695+0000 7fa54e31e640 1 Processor -- start 2024-03-28T23:36:11.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.695+0000 7fa54e31e640 1 -- start start 2024-03-28T23:36:11.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.695+0000 7fa54e31e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa548152890 0x7fa5480c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:11.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.696+0000 7fa54e31e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa5480c5b20 0x7fa5481553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:11.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.696+0000 7fa54e31e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa548158a30 con 0x7fa5480c8ed0 2024-03-28T23:36:11.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.696+0000 7fa54e31e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fa5480c8480 con 0x7fa5480c5b20 2024-03-28T23:36:11.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.696+0000 7fa54e31e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa548159640 con 0x7fa548152890 2024-03-28T23:36:11.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.696+0000 7fa547fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa548152890 0x7fa5480c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:11.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.696+0000 7fa54c8bb640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa5480c5b20 0x7fa5481553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:11.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.696+0000 7fa5477fe640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fa5480c8ed0 0x7fa5480c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:59498/0 (socket says 172.21.15.161:59498) 2024-03-28T23:36:11.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.696+0000 7fa5477fe640 1 -- 172.21.15.161:0/2418725969 learned_addr learned my addr 172.21.15.161:0/2418725969 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:11.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.696+0000 7fa54c8bb640 1 --2- 172.21.15.161:0/2418725969 >> v2:172.21.15.170:3300/0 conn(0x7fa5480c5b20 0x7fa5481553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:53530/0 (socket says 172.21.15.161:53530) 2024-03-28T23:36:11.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.696+0000 7fa547fff640 1 -- 172.21.15.161:0/2418725969 >> v2:172.21.15.170:3300/0 conn(0x7fa5480c5b20 msgr2=0x7fa5481553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:11.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.696+0000 7fa547fff640 1 --2- 172.21.15.161:0/2418725969 >> v2:172.21.15.170:3300/0 conn(0x7fa5480c5b20 0x7fa5481553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:11.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.696+0000 7fa547fff640 1 -- 172.21.15.161:0/2418725969 >> v1:172.21.15.170:6789/0 conn(0x7fa5480c8ed0 legacy=0x7fa5480c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:11.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.696+0000 7fa547fff640 1 -- 172.21.15.161:0/2418725969 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa548158d10 con 0x7fa548152890 2024-03-28T23:36:11.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.696+0000 7fa54c8bb640 1 --2- 172.21.15.161:0/2418725969 >> v2:172.21.15.170:3300/0 conn(0x7fa5480c5b20 0x7fa5481553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:11.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.697+0000 7fa547fff640 1 --2- 172.21.15.161:0/2418725969 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa548152890 0x7fa5480c8990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa5481597c0 tx=0x7fa53802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d105cd3363b21248 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:11.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.697+0000 7fa546ffd640 1 -- 172.21.15.161:0/2418725969 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa538039070 con 0x7fa548152890 2024-03-28T23:36:11.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.697+0000 7fa546ffd640 1 -- 172.21.15.161:0/2418725969 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5380316f0 con 0x7fa548152890 2024-03-28T23:36:11.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.697+0000 7fa546ffd640 1 -- 172.21.15.161:0/2418725969 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa538031ac0 con 0x7fa548152890 2024-03-28T23:36:11.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.698+0000 7fa54e31e640 1 -- 172.21.15.161:0/2418725969 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa548152890 msgr2=0x7fa5480c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:11.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.698+0000 7fa54e31e640 1 --2- 172.21.15.161:0/2418725969 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa548152890 0x7fa5480c8990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa5481597c0 tx=0x7fa53802b790 comp rx=0 tx=0).stop 2024-03-28T23:36:11.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.698+0000 7fa54e31e640 1 -- 172.21.15.161:0/2418725969 shutdown_connections 2024-03-28T23:36:11.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.698+0000 7fa54e31e640 1 --2- 172.21.15.161:0/2418725969 >> v2:172.21.15.170:3300/0 conn(0x7fa5480c5b20 0x7fa5481553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:11.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.698+0000 7fa54e31e640 1 --2- 172.21.15.161:0/2418725969 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa548152890 0x7fa5480c8990 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:11.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.698+0000 7fa54e31e640 1 -- 172.21.15.161:0/2418725969 >> 172.21.15.161:0/2418725969 conn(0x7fa5480ac620 msgr2=0x7fa5480aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:11.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.698+0000 7fa54e31e640 1 -- 172.21.15.161:0/2418725969 shutdown_connections 2024-03-28T23:36:11.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.698+0000 7fa54e31e640 1 -- 172.21.15.161:0/2418725969 wait complete. 2024-03-28T23:36:11.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.699+0000 7fa54e31e640 1 Processor -- start 2024-03-28T23:36:11.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.699+0000 7fa54e31e640 1 -- start start 2024-03-28T23:36:11.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.700+0000 7fa54e31e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa5480c5b20 0x7fa5480ab900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:11.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.700+0000 7fa54e31e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5480c8ed0 0x7fa548155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:11.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.700+0000 7fa54e31e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa548152890 0x7fa548155870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:11.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.700+0000 7fa54e31e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa548152c70 con 0x7fa5480c8ed0 2024-03-28T23:36:11.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.700+0000 7fa54e31e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa5481592d0 con 0x7fa548152890 2024-03-28T23:36:11.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.700+0000 7fa54e31e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa548152df0 con 0x7fa5480c5b20 2024-03-28T23:36:11.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.700+0000 7fa5477fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5480c8ed0 0x7fa548155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:11.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.700+0000 7fa5477fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5480c8ed0 0x7fa548155330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46616/0 (socket says 172.21.15.161:46616) 2024-03-28T23:36:11.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.700+0000 7fa547fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa5480c5b20 0x7fa5480ab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:11.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.700+0000 7fa54c8bb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa548152890 0x7fa548155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:11.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.700+0000 7fa5477fe640 1 -- 172.21.15.161:0/1232885431 learned_addr learned my addr 172.21.15.161:0/1232885431 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:11.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.700+0000 7fa547fff640 1 -- 172.21.15.161:0/1232885431 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa548152890 msgr2=0x7fa548155870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:11.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.700+0000 7fa547fff640 1 --2- 172.21.15.161:0/1232885431 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa548152890 0x7fa548155870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:11.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.700+0000 7fa547fff640 1 -- 172.21.15.161:0/1232885431 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5480c8ed0 msgr2=0x7fa548155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:11.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.700+0000 7fa547fff640 1 --2- 172.21.15.161:0/1232885431 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5480c8ed0 0x7fa548155330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:11.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.700+0000 7fa547fff640 1 -- 172.21.15.161:0/1232885431 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5481ebef0 con 0x7fa5480c5b20 2024-03-28T23:36:11.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.700+0000 7fa5477fe640 1 --2- 172.21.15.161:0/1232885431 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5480c8ed0 0x7fa548155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:11.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.700+0000 7fa547fff640 1 --2- 172.21.15.161:0/1232885431 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa5480c5b20 0x7fa5480ab900 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa538033000 tx=0x7fa5380046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:11.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.700+0000 7fa544ff9640 1 -- 172.21.15.161:0/1232885431 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa538043070 con 0x7fa5480c5b20 2024-03-28T23:36:11.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.700+0000 7fa544ff9640 1 -- 172.21.15.161:0/1232885431 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5380346a0 con 0x7fa5480c5b20 2024-03-28T23:36:11.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.700+0000 7fa544ff9640 1 -- 172.21.15.161:0/1232885431 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa538039040 con 0x7fa5480c5b20 2024-03-28T23:36:11.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.700+0000 7fa54e31e640 1 -- 172.21.15.161:0/1232885431 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5481eeb00 con 0x7fa5480c5b20 2024-03-28T23:36:11.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.700+0000 7fa54e31e640 1 -- 172.21.15.161:0/1232885431 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5481eede0 con 0x7fa5480c5b20 2024-03-28T23:36:11.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.701+0000 7fa54e31e640 1 -- 172.21.15.161:0/1232885431 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5480abea0 con 0x7fa5480c5b20 2024-03-28T23:36:11.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.702+0000 7fa544ff9640 1 -- 172.21.15.161:0/1232885431 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa53803c070 con 0x7fa5480c5b20 2024-03-28T23:36:11.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.702+0000 7fa544ff9640 1 --2- 172.21.15.161:0/1232885431 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa53406b500 0x7fa53406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:11.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.702+0000 7fa5477fe640 1 --2- 172.21.15.161:0/1232885431 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa53406b500 0x7fa53406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:11.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.703+0000 7fa544ff9640 1 -- 172.21.15.161:0/1232885431 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(156..156 src has 1..156) ==== 6957+0+0 (secure 0 0 0) 0x7fa5380abd90 con 0x7fa5480c5b20 2024-03-28T23:36:11.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.703+0000 7fa5477fe640 1 --2- 172.21.15.161:0/1232885431 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa53406b500 0x7fa53406d9c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fa5480c4d50 tx=0x7fa53c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:11.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.707+0000 7fa544ff9640 1 -- 172.21.15.161:0/1232885431 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa53807a290 con 0x7fa5480c5b20 2024-03-28T23:36:11.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.885+0000 7fa54e31e640 1 -- 172.21.15.161:0/1232885431 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa54804f4b0 con 0x7fa5480c5b20 2024-03-28T23:36:11.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.887+0000 7fa544ff9640 1 -- 172.21.15.161:0/1232885431 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v156) ==== 74+0+20638 (secure 0 0 0) 0x7fa53807e330 con 0x7fa5480c5b20 2024-03-28T23:36:11.888 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:11.888 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":156,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:36:09.105369+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:35:56.687416+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-28T23:35:57.379120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:36:05.484451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:36:06.423073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:36:11.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.889+0000 7fa54e31e640 1 -- 172.21.15.161:0/1232885431 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa53406b500 msgr2=0x7fa53406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:11.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.889+0000 7fa54e31e640 1 --2- 172.21.15.161:0/1232885431 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa53406b500 0x7fa53406d9c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fa5480c4d50 tx=0x7fa53c009040 comp rx=0 tx=0).stop 2024-03-28T23:36:11.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.889+0000 7fa54e31e640 1 -- 172.21.15.161:0/1232885431 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa5480c5b20 msgr2=0x7fa5480ab900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:11.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.889+0000 7fa54e31e640 1 --2- 172.21.15.161:0/1232885431 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa5480c5b20 0x7fa5480ab900 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa538033000 tx=0x7fa5380046c0 comp rx=0 tx=0).stop 2024-03-28T23:36:11.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.889+0000 7fa54e31e640 1 -- 172.21.15.161:0/1232885431 shutdown_connections 2024-03-28T23:36:11.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.889+0000 7fa54e31e640 1 --2- 172.21.15.161:0/1232885431 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa53406b500 0x7fa53406d9c0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:11.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.889+0000 7fa54e31e640 1 --2- 172.21.15.161:0/1232885431 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5480c8ed0 0x7fa548155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:11.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.889+0000 7fa54e31e640 1 --2- 172.21.15.161:0/1232885431 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa548152890 0x7fa548155870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:11.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.889+0000 7fa54e31e640 1 --2- 172.21.15.161:0/1232885431 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa5480c5b20 0x7fa5480ab900 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:11.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.889+0000 7fa54e31e640 1 -- 172.21.15.161:0/1232885431 >> 172.21.15.161:0/1232885431 conn(0x7fa5480ac620 msgr2=0x7fa54806c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:11.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.889+0000 7fa54e31e640 1 -- 172.21.15.161:0/1232885431 shutdown_connections 2024-03-28T23:36:11.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:11.889+0000 7fa54e31e640 1 -- 172.21.15.161:0/1232885431 wait complete. 2024-03-28T23:36:11.901 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T23:36:11.901 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:36:12.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.002+0000 7f29b8e15640 1 Processor -- start 2024-03-28T23:36:12.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.003+0000 7f29b8e15640 1 -- start start 2024-03-28T23:36:12.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.003+0000 7f29b8e15640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f29b4155d10 0x7f29b40c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:12.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.003+0000 7f29b8e15640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29b406c3c0 con 0x7f29b406d9f0 2024-03-28T23:36:12.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.003+0000 7f29b8e15640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29b40c5b90 con 0x7f29b4152a80 2024-03-28T23:36:12.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.003+0000 7f29b8e15640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f29b406c6f0 con 0x7f29b4155d10 2024-03-28T23:36:12.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.004+0000 7f29b1d9b640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f29b406d9f0 0x7f29b406cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41712/0 (socket says 172.21.15.161:41712) 2024-03-28T23:36:12.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.004+0000 7f29b1d9b640 1 -- 172.21.15.161:0/170309757 learned_addr learned my addr 172.21.15.161:0/170309757 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:12.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.004+0000 7f29b2d9d640 1 --2- 172.21.15.161:0/170309757 >> v2:172.21.15.170:3300/0 conn(0x7f29b4155d10 0x7f29b40c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:12.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.004+0000 7f29b159a640 1 -- 172.21.15.161:0/170309757 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1135839004 0 0) 0x7f29b406c3c0 con 0x7f29b406d9f0 2024-03-28T23:36:12.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.004+0000 7f29b159a640 1 -- 172.21.15.161:0/170309757 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29a0003520 con 0x7f29b406d9f0 2024-03-28T23:36:12.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.004+0000 7f29b2d9d640 1 -- 172.21.15.161:0/170309757 >> v1:172.21.15.170:6789/0 conn(0x7f29b4152a80 legacy=0x7f29b40c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:12.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.004+0000 7f29b2d9d640 1 -- 172.21.15.161:0/170309757 >> v1:172.21.15.161:6789/0 conn(0x7f29b406d9f0 legacy=0x7f29b406cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:12.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.004+0000 7f29b159a640 1 -- 172.21.15.161:0/170309757 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1979232136 0 0) 0x7f29b40c5b90 con 0x7f29b4152a80 2024-03-28T23:36:12.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.004+0000 7f29b2d9d640 1 -- 172.21.15.161:0/170309757 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b40c55d0 con 0x7f29b4155d10 2024-03-28T23:36:12.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.005+0000 7f29b2d9d640 1 --2- 172.21.15.161:0/170309757 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29b4155d10 0x7f29b40c74c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f29ac0097a0 tx=0x7f29ac02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee3b05cb910a95d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:12.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.005+0000 7f29b159a640 1 -- 172.21.15.161:0/170309757 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ac0315a0 con 0x7f29b4155d10 2024-03-28T23:36:12.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.005+0000 7f29b159a640 1 -- 172.21.15.161:0/170309757 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29ac037070 con 0x7f29b4155d10 2024-03-28T23:36:12.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.005+0000 7f29b159a640 1 -- 172.21.15.161:0/170309757 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ac031960 con 0x7f29b4155d10 2024-03-28T23:36:12.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.005+0000 7f29b8e15640 1 -- 172.21.15.161:0/170309757 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29b4155d10 msgr2=0x7f29b40c74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:12.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.005+0000 7f29b8e15640 1 --2- 172.21.15.161:0/170309757 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29b4155d10 0x7f29b40c74c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f29ac0097a0 tx=0x7f29ac02b840 comp rx=0 tx=0).stop 2024-03-28T23:36:12.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.006+0000 7f29b8e15640 1 -- 172.21.15.161:0/170309757 shutdown_connections 2024-03-28T23:36:12.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.006+0000 7f29b8e15640 1 --2- 172.21.15.161:0/170309757 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29b4155d10 0x7f29b40c74c0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.006+0000 7f29b8e15640 1 -- 172.21.15.161:0/170309757 >> 172.21.15.161:0/170309757 conn(0x7f29b40ac720 msgr2=0x7f29b40acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:12.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.006+0000 7f29b8e15640 1 -- 172.21.15.161:0/170309757 shutdown_connections 2024-03-28T23:36:12.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.006+0000 7f29b8e15640 1 -- 172.21.15.161:0/170309757 wait complete. 2024-03-28T23:36:12.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.007+0000 7f29b8e15640 1 Processor -- start 2024-03-28T23:36:12.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.007+0000 7f29b8e15640 1 -- start start 2024-03-28T23:36:12.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.008+0000 7f29b8e15640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29b406d9f0 0x7f29b40c3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:12.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.008+0000 7f29b8e15640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29b4152a80 0x7f29b40c4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:12.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.008+0000 7f29b8e15640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29b4155d10 0x7f29b40aeca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:12.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.008+0000 7f29b8e15640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f29b40c7a00 con 0x7f29b4155d10 2024-03-28T23:36:12.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.008+0000 7f29b8e15640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f29b41561a0 con 0x7f29b406d9f0 2024-03-28T23:36:12.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.008+0000 7f29b8e15640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f29b40c7b80 con 0x7f29b4152a80 2024-03-28T23:36:12.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.008+0000 7f29b2d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29b4155d10 0x7f29b40aeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:12.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.008+0000 7f29b2d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29b4155d10 0x7f29b40aeca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46620/0 (socket says 172.21.15.161:46620) 2024-03-28T23:36:12.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.008+0000 7f29b2d9d640 1 -- 172.21.15.161:0/4269355606 learned_addr learned my addr 172.21.15.161:0/4269355606 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:12.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.008+0000 7f29b259c640 1 --2- 172.21.15.161:0/4269355606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29b406d9f0 0x7f29b40c3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:12.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.008+0000 7f29b1d9b640 1 --2- 172.21.15.161:0/4269355606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29b4152a80 0x7f29b40c4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:12.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.008+0000 7f29b2d9d640 1 -- 172.21.15.161:0/4269355606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29b4152a80 msgr2=0x7f29b40c4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:12.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.008+0000 7f29b2d9d640 1 --2- 172.21.15.161:0/4269355606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29b4152a80 0x7f29b40c4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.008+0000 7f29b2d9d640 1 -- 172.21.15.161:0/4269355606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29b406d9f0 msgr2=0x7f29b40c3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:12.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.008+0000 7f29b2d9d640 1 --2- 172.21.15.161:0/4269355606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29b406d9f0 0x7f29b40c3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.008+0000 7f29b2d9d640 1 -- 172.21.15.161:0/4269355606 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b40c2050 con 0x7f29b4155d10 2024-03-28T23:36:12.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.008+0000 7f29b1d9b640 1 --2- 172.21.15.161:0/4269355606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29b4152a80 0x7f29b40c4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:12.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.008+0000 7f29b259c640 1 --2- 172.21.15.161:0/4269355606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29b406d9f0 0x7f29b40c3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:12.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.008+0000 7f29b2d9d640 1 --2- 172.21.15.161:0/4269355606 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29b4155d10 0x7f29b40aeca0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f29ac0097a0 tx=0x7f29ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:12.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.008+0000 7f29aaffd640 1 -- 172.21.15.161:0/4269355606 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ac04a070 con 0x7f29b4155d10 2024-03-28T23:36:12.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.008+0000 7f29aaffd640 1 -- 172.21.15.161:0/4269355606 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29ac032430 con 0x7f29b4155d10 2024-03-28T23:36:12.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.008+0000 7f29b8e15640 1 -- 172.21.15.161:0/4269355606 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29b40c2330 con 0x7f29b4155d10 2024-03-28T23:36:12.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.008+0000 7f29aaffd640 1 -- 172.21.15.161:0/4269355606 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ac0327e0 con 0x7f29b4155d10 2024-03-28T23:36:12.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.009+0000 7f29b8e15640 1 -- 172.21.15.161:0/4269355606 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29b40b06a0 con 0x7f29b4155d10 2024-03-28T23:36:12.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.009+0000 7f29b8e15640 1 -- 172.21.15.161:0/4269355606 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29b40c2550 con 0x7f29b4155d10 2024-03-28T23:36:12.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.010+0000 7f29aaffd640 1 -- 172.21.15.161:0/4269355606 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f29ac032980 con 0x7f29b4155d10 2024-03-28T23:36:12.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.010+0000 7f29aaffd640 1 --2- 172.21.15.161:0/4269355606 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f299406b4d0 0x7f299406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:12.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.010+0000 7f29aaffd640 1 -- 172.21.15.161:0/4269355606 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 6957+0+0 (secure 0 0 0) 0x7f29ac02f020 con 0x7f29b4155d10 2024-03-28T23:36:12.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.010+0000 7f29b259c640 1 --2- 172.21.15.161:0/4269355606 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f299406b4d0 0x7f299406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:12.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.011+0000 7f29b259c640 1 --2- 172.21.15.161:0/4269355606 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f299406b4d0 0x7f299406d990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f299c000f60 tx=0x7f299c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:12.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.015+0000 7f29aaffd640 1 -- 172.21.15.161:0/4269355606 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f29ac040070 con 0x7f29b4155d10 2024-03-28T23:36:12.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.192+0000 7f29b8e15640 1 -- 172.21.15.161:0/4269355606 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f29b40c4a70 con 0x7f299406b4d0 2024-03-28T23:36:12.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.207+0000 7f29aaffd640 1 -- 172.21.15.161:0/4269355606 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+491517 (secure 0 0 0) 0x7f29b40c4a70 con 0x7f299406b4d0 2024-03-28T23:36:12.213 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:12.216 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:36:12.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.211+0000 7f29b8e15640 1 -- 172.21.15.161:0/4269355606 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f299406b4d0 msgr2=0x7f299406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:12.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.211+0000 7f29b8e15640 1 --2- 172.21.15.161:0/4269355606 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f299406b4d0 0x7f299406d990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f299c000f60 tx=0x7f299c009040 comp rx=0 tx=0).stop 2024-03-28T23:36:12.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.211+0000 7f29b8e15640 1 -- 172.21.15.161:0/4269355606 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29b4155d10 msgr2=0x7f29b40aeca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:12.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.211+0000 7f29b8e15640 1 --2- 172.21.15.161:0/4269355606 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29b4155d10 0x7f29b40aeca0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f29ac0097a0 tx=0x7f29ac0046c0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.211+0000 7f29b8e15640 1 -- 172.21.15.161:0/4269355606 shutdown_connections 2024-03-28T23:36:12.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.211+0000 7f29b8e15640 1 --2- 172.21.15.161:0/4269355606 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f299406b4d0 0x7f299406d990 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.211+0000 7f29b8e15640 1 --2- 172.21.15.161:0/4269355606 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29b4155d10 0x7f29b40aeca0 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.211+0000 7f29b8e15640 1 --2- 172.21.15.161:0/4269355606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29b4152a80 0x7f29b40c4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.211+0000 7f29b8e15640 1 --2- 172.21.15.161:0/4269355606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29b406d9f0 0x7f29b40c3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.211+0000 7f29b8e15640 1 -- 172.21.15.161:0/4269355606 >> 172.21.15.161:0/4269355606 conn(0x7f29b40ac720 msgr2=0x7f29b40ca020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:12.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.212+0000 7f29b8e15640 1 -- 172.21.15.161:0/4269355606 shutdown_connections 2024-03-28T23:36:12.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.212+0000 7f29b8e15640 1 -- 172.21.15.161:0/4269355606 wait complete. 2024-03-28T23:36:12.223 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-03-28T23:36:11.386445+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":290,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264980,"kb_used_data":15500,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754709740,"statfs":{"total":773094113280,"available":772822773760,"internally_reserved":0,"allocated":15872000,"data_stored":8779741,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.299846"},"pg_stats":[{"pgid":"26.3d","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.191136+0000","last_change":"2024-03-28T23:36:08.189106+0000","last_active":"2024-03-28T23:36:08.191136+0000","last_peered":"2024-03-28T23:36:08.191136+0000","last_clean":"2024-03-28T23:36:08.191136+0000","last_became_active":"2024-03-28T23:36:08.188825+0000","last_became_peered":"2024-03-28T23:36:08.188825+0000","last_unstale":"2024-03-28T23:36:08.191136+0000","last_undegraded":"2024-03-28T23:36:08.191136+0000","last_fullsized":"2024-03-28T23:36:08.191136+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:11:58.809402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.3f","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.105776+0000","last_change":"2024-03-28T23:35:59.077622+0000","last_active":"2024-03-28T23:36:08.105776+0000","last_peered":"2024-03-28T23:36:08.105776+0000","last_clean":"2024-03-28T23:36:08.105776+0000","last_became_active":"2024-03-28T23:35:59.077350+0000","last_became_peered":"2024-03-28T23:35:59.077350+0000","last_unstale":"2024-03-28T23:36:08.105776+0000","last_undegraded":"2024-03-28T23:36:08.105776+0000","last_fullsized":"2024-03-28T23:36:08.105776+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:39:54.469103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.120041+0000","last_change":"2024-03-28T23:36:08.115888+0000","last_active":"2024-03-28T23:36:08.120041+0000","last_peered":"2024-03-28T23:36:08.120041+0000","last_clean":"2024-03-28T23:36:08.120041+0000","last_became_active":"2024-03-28T23:36:08.115551+0000","last_became_peered":"2024-03-28T23:36:08.115551+0000","last_unstale":"2024-03-28T23:36:08.120041+0000","last_undegraded":"2024-03-28T23:36:08.120041+0000","last_fullsized":"2024-03-28T23:36:08.120041+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:34:48.075949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109844+0000","last_change":"2024-03-28T23:35:59.069924+0000","last_active":"2024-03-28T23:36:09.109844+0000","last_peered":"2024-03-28T23:36:09.109844+0000","last_clean":"2024-03-28T23:36:09.109844+0000","last_became_active":"2024-03-28T23:35:59.069650+0000","last_became_peered":"2024-03-28T23:35:59.069650+0000","last_unstale":"2024-03-28T23:36:09.109844+0000","last_undegraded":"2024-03-28T23:36:09.109844+0000","last_fullsized":"2024-03-28T23:36:09.109844+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:32.347982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120230+0000","last_change":"2024-03-28T23:36:08.124687+0000","last_active":"2024-03-28T23:36:09.120230+0000","last_peered":"2024-03-28T23:36:09.120230+0000","last_clean":"2024-03-28T23:36:09.120230+0000","last_became_active":"2024-03-28T23:36:08.123746+0000","last_became_peered":"2024-03-28T23:36:08.123746+0000","last_unstale":"2024-03-28T23:36:09.120230+0000","last_undegraded":"2024-03-28T23:36:09.120230+0000","last_fullsized":"2024-03-28T23:36:09.120230+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:34:48.075949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.105213+0000","last_change":"2024-03-28T23:35:59.082632+0000","last_active":"2024-03-28T23:36:08.105213+0000","last_peered":"2024-03-28T23:36:08.105213+0000","last_clean":"2024-03-28T23:36:08.105213+0000","last_became_active":"2024-03-28T23:35:59.082360+0000","last_became_peered":"2024-03-28T23:35:59.082360+0000","last_unstale":"2024-03-28T23:36:08.105213+0000","last_undegraded":"2024-03-28T23:36:08.105213+0000","last_fullsized":"2024-03-28T23:36:08.105213+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:00:59.308679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120174+0000","last_change":"2024-03-28T23:36:08.113999+0000","last_active":"2024-03-28T23:36:09.120174+0000","last_peered":"2024-03-28T23:36:09.120174+0000","last_clean":"2024-03-28T23:36:09.120174+0000","last_became_active":"2024-03-28T23:36:08.113514+0000","last_became_peered":"2024-03-28T23:36:08.113514+0000","last_unstale":"2024-03-28T23:36:09.120174+0000","last_undegraded":"2024-03-28T23:36:09.120174+0000","last_fullsized":"2024-03-28T23:36:09.120174+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:24:21.264344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.105918+0000","last_change":"2024-03-28T23:35:59.084222+0000","last_active":"2024-03-28T23:36:08.105918+0000","last_peered":"2024-03-28T23:36:08.105918+0000","last_clean":"2024-03-28T23:36:08.105918+0000","last_became_active":"2024-03-28T23:35:59.083603+0000","last_became_peered":"2024-03-28T23:35:59.083603+0000","last_unstale":"2024-03-28T23:36:08.105918+0000","last_undegraded":"2024-03-28T23:36:08.105918+0000","last_fullsized":"2024-03-28T23:36:08.105918+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:18:01.009699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117621+0000","last_change":"2024-03-28T23:36:08.115492+0000","last_active":"2024-03-28T23:36:09.117621+0000","last_peered":"2024-03-28T23:36:09.117621+0000","last_clean":"2024-03-28T23:36:09.117621+0000","last_became_active":"2024-03-28T23:36:08.112264+0000","last_became_peered":"2024-03-28T23:36:08.112264+0000","last_unstale":"2024-03-28T23:36:09.117621+0000","last_undegraded":"2024-03-28T23:36:09.117621+0000","last_fullsized":"2024-03-28T23:36:09.117621+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:41.904291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.108709+0000","last_change":"2024-03-28T23:35:59.074398+0000","last_active":"2024-03-28T23:36:08.108709+0000","last_peered":"2024-03-28T23:36:08.108709+0000","last_clean":"2024-03-28T23:36:08.108709+0000","last_became_active":"2024-03-28T23:35:59.074143+0000","last_became_peered":"2024-03-28T23:35:59.074143+0000","last_unstale":"2024-03-28T23:36:08.108709+0000","last_undegraded":"2024-03-28T23:36:08.108709+0000","last_fullsized":"2024-03-28T23:36:08.108709+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:47.081742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109102+0000","last_change":"2024-03-28T23:36:08.106097+0000","last_active":"2024-03-28T23:36:09.109102+0000","last_peered":"2024-03-28T23:36:09.109102+0000","last_clean":"2024-03-28T23:36:09.109102+0000","last_became_active":"2024-03-28T23:36:08.105118+0000","last_became_peered":"2024-03-28T23:36:08.105118+0000","last_unstale":"2024-03-28T23:36:09.109102+0000","last_undegraded":"2024-03-28T23:36:09.109102+0000","last_fullsized":"2024-03-28T23:36:09.109102+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:46:20.625726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108697+0000","last_change":"2024-03-28T23:35:59.066035+0000","last_active":"2024-03-28T23:36:09.108697+0000","last_peered":"2024-03-28T23:36:09.108697+0000","last_clean":"2024-03-28T23:36:09.108697+0000","last_became_active":"2024-03-28T23:35:59.065595+0000","last_became_peered":"2024-03-28T23:35:59.065595+0000","last_unstale":"2024-03-28T23:36:09.108697+0000","last_undegraded":"2024-03-28T23:36:09.108697+0000","last_fullsized":"2024-03-28T23:36:09.108697+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:26:11.783759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109402+0000","last_change":"2024-03-28T23:36:08.120716+0000","last_active":"2024-03-28T23:36:09.109402+0000","last_peered":"2024-03-28T23:36:09.109402+0000","last_clean":"2024-03-28T23:36:09.109402+0000","last_became_active":"2024-03-28T23:36:08.120057+0000","last_became_peered":"2024-03-28T23:36:08.120057+0000","last_unstale":"2024-03-28T23:36:09.109402+0000","last_undegraded":"2024-03-28T23:36:09.109402+0000","last_fullsized":"2024-03-28T23:36:09.109402+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:47.824582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119603+0000","last_change":"2024-03-28T23:35:59.076253+0000","last_active":"2024-03-28T23:36:09.119603+0000","last_peered":"2024-03-28T23:36:09.119603+0000","last_clean":"2024-03-28T23:36:09.119603+0000","last_became_active":"2024-03-28T23:35:59.075998+0000","last_became_peered":"2024-03-28T23:35:59.075998+0000","last_unstale":"2024-03-28T23:36:09.119603+0000","last_undegraded":"2024-03-28T23:36:09.119603+0000","last_fullsized":"2024-03-28T23:36:09.119603+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:51:06.388640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109031+0000","last_change":"2024-03-28T23:36:08.127064+0000","last_active":"2024-03-28T23:36:09.109031+0000","last_peered":"2024-03-28T23:36:09.109031+0000","last_clean":"2024-03-28T23:36:09.109031+0000","last_became_active":"2024-03-28T23:36:08.126601+0000","last_became_peered":"2024-03-28T23:36:08.126601+0000","last_unstale":"2024-03-28T23:36:09.109031+0000","last_undegraded":"2024-03-28T23:36:09.109031+0000","last_fullsized":"2024-03-28T23:36:09.109031+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:12:47.028863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119392+0000","last_change":"2024-03-28T23:35:59.077597+0000","last_active":"2024-03-28T23:36:09.119392+0000","last_peered":"2024-03-28T23:36:09.119392+0000","last_clean":"2024-03-28T23:36:09.119392+0000","last_became_active":"2024-03-28T23:35:59.077343+0000","last_became_peered":"2024-03-28T23:35:59.077343+0000","last_unstale":"2024-03-28T23:36:09.119392+0000","last_undegraded":"2024-03-28T23:36:09.119392+0000","last_fullsized":"2024-03-28T23:36:09.119392+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:26:11.783759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109119+0000","last_change":"2024-03-28T23:36:08.191205+0000","last_active":"2024-03-28T23:36:09.109119+0000","last_peered":"2024-03-28T23:36:09.109119+0000","last_clean":"2024-03-28T23:36:09.109119+0000","last_became_active":"2024-03-28T23:36:08.190885+0000","last_became_peered":"2024-03-28T23:36:08.190885+0000","last_unstale":"2024-03-28T23:36:09.109119+0000","last_undegraded":"2024-03-28T23:36:09.109119+0000","last_fullsized":"2024-03-28T23:36:09.109119+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:49:36.028241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120477+0000","last_change":"2024-03-28T23:35:59.073690+0000","last_active":"2024-03-28T23:36:09.120477+0000","last_peered":"2024-03-28T23:36:09.120477+0000","last_clean":"2024-03-28T23:36:09.120477+0000","last_became_active":"2024-03-28T23:35:59.073405+0000","last_became_peered":"2024-03-28T23:35:59.073405+0000","last_unstale":"2024-03-28T23:36:09.120477+0000","last_undegraded":"2024-03-28T23:36:09.120477+0000","last_fullsized":"2024-03-28T23:36:09.120477+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:17.228750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.120020+0000","last_change":"2024-03-28T23:36:08.115324+0000","last_active":"2024-03-28T23:36:08.120020+0000","last_peered":"2024-03-28T23:36:08.120020+0000","last_clean":"2024-03-28T23:36:08.120020+0000","last_became_active":"2024-03-28T23:36:08.115043+0000","last_became_peered":"2024-03-28T23:36:08.115043+0000","last_unstale":"2024-03-28T23:36:08.120020+0000","last_undegraded":"2024-03-28T23:36:08.120020+0000","last_fullsized":"2024-03-28T23:36:08.120020+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:39:47.021555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120287+0000","last_change":"2024-03-28T23:35:59.077619+0000","last_active":"2024-03-28T23:36:09.120287+0000","last_peered":"2024-03-28T23:36:09.120287+0000","last_clean":"2024-03-28T23:36:09.120287+0000","last_became_active":"2024-03-28T23:35:59.077386+0000","last_became_peered":"2024-03-28T23:35:59.077386+0000","last_unstale":"2024-03-28T23:36:09.120287+0000","last_undegraded":"2024-03-28T23:36:09.120287+0000","last_fullsized":"2024-03-28T23:36:09.120287+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:49:08.837095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120375+0000","last_change":"2024-03-28T23:36:08.191811+0000","last_active":"2024-03-28T23:36:09.120375+0000","last_peered":"2024-03-28T23:36:09.120375+0000","last_clean":"2024-03-28T23:36:09.120375+0000","last_became_active":"2024-03-28T23:36:08.191518+0000","last_became_peered":"2024-03-28T23:36:08.191518+0000","last_unstale":"2024-03-28T23:36:09.120375+0000","last_undegraded":"2024-03-28T23:36:09.120375+0000","last_fullsized":"2024-03-28T23:36:09.120375+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:09:57.210214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109258+0000","last_change":"2024-03-28T23:35:59.074672+0000","last_active":"2024-03-28T23:36:09.109258+0000","last_peered":"2024-03-28T23:36:09.109258+0000","last_clean":"2024-03-28T23:36:09.109258+0000","last_became_active":"2024-03-28T23:35:59.074385+0000","last_became_peered":"2024-03-28T23:35:59.074385+0000","last_unstale":"2024-03-28T23:36:09.109258+0000","last_undegraded":"2024-03-28T23:36:09.109258+0000","last_fullsized":"2024-03-28T23:36:09.109258+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:53:22.705919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109941+0000","last_change":"2024-03-28T23:36:08.116471+0000","last_active":"2024-03-28T23:36:09.109941+0000","last_peered":"2024-03-28T23:36:09.109941+0000","last_clean":"2024-03-28T23:36:09.109941+0000","last_became_active":"2024-03-28T23:36:08.116123+0000","last_became_peered":"2024-03-28T23:36:08.116123+0000","last_unstale":"2024-03-28T23:36:09.109941+0000","last_undegraded":"2024-03-28T23:36:09.109941+0000","last_fullsized":"2024-03-28T23:36:09.109941+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:34:32.575972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.110121+0000","last_change":"2024-03-28T23:35:59.072384+0000","last_active":"2024-03-28T23:36:09.110121+0000","last_peered":"2024-03-28T23:36:09.110121+0000","last_clean":"2024-03-28T23:36:09.110121+0000","last_became_active":"2024-03-28T23:35:59.071941+0000","last_became_peered":"2024-03-28T23:35:59.071941+0000","last_unstale":"2024-03-28T23:36:09.110121+0000","last_undegraded":"2024-03-28T23:36:09.110121+0000","last_fullsized":"2024-03-28T23:36:09.110121+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:36:26.239288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119748+0000","last_change":"2024-03-28T23:36:08.125010+0000","last_active":"2024-03-28T23:36:09.119748+0000","last_peered":"2024-03-28T23:36:09.119748+0000","last_clean":"2024-03-28T23:36:09.119748+0000","last_became_active":"2024-03-28T23:36:08.124704+0000","last_became_peered":"2024-03-28T23:36:08.124704+0000","last_unstale":"2024-03-28T23:36:09.119748+0000","last_undegraded":"2024-03-28T23:36:09.119748+0000","last_fullsized":"2024-03-28T23:36:09.119748+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:26:23.533905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109517+0000","last_change":"2024-03-28T23:35:59.076869+0000","last_active":"2024-03-28T23:36:09.109517+0000","last_peered":"2024-03-28T23:36:09.109517+0000","last_clean":"2024-03-28T23:36:09.109517+0000","last_became_active":"2024-03-28T23:35:59.076407+0000","last_became_peered":"2024-03-28T23:35:59.076407+0000","last_unstale":"2024-03-28T23:36:09.109517+0000","last_undegraded":"2024-03-28T23:36:09.109517+0000","last_fullsized":"2024-03-28T23:36:09.109517+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:47.081742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108907+0000","last_change":"2024-03-28T23:36:08.118544+0000","last_active":"2024-03-28T23:36:09.108907+0000","last_peered":"2024-03-28T23:36:09.108907+0000","last_clean":"2024-03-28T23:36:09.108907+0000","last_became_active":"2024-03-28T23:36:08.116769+0000","last_became_peered":"2024-03-28T23:36:08.116769+0000","last_unstale":"2024-03-28T23:36:09.108907+0000","last_undegraded":"2024-03-28T23:36:09.108907+0000","last_fullsized":"2024-03-28T23:36:09.108907+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:30:47.451354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.110495+0000","last_change":"2024-03-28T23:35:59.082855+0000","last_active":"2024-03-28T23:36:08.110495+0000","last_peered":"2024-03-28T23:36:08.110495+0000","last_clean":"2024-03-28T23:36:08.110495+0000","last_became_active":"2024-03-28T23:35:59.082581+0000","last_became_peered":"2024-03-28T23:35:59.082581+0000","last_unstale":"2024-03-28T23:36:08.110495+0000","last_undegraded":"2024-03-28T23:36:08.110495+0000","last_fullsized":"2024-03-28T23:36:08.110495+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:25.528570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.192047+0000","last_change":"2024-03-28T23:36:08.191168+0000","last_active":"2024-03-28T23:36:08.192047+0000","last_peered":"2024-03-28T23:36:08.192047+0000","last_clean":"2024-03-28T23:36:08.192047+0000","last_became_active":"2024-03-28T23:36:08.190750+0000","last_became_peered":"2024-03-28T23:36:08.190750+0000","last_unstale":"2024-03-28T23:36:08.192047+0000","last_undegraded":"2024-03-28T23:36:08.192047+0000","last_fullsized":"2024-03-28T23:36:08.192047+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:26.267458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108242+0000","last_change":"2024-03-28T23:35:59.080077+0000","last_active":"2024-03-28T23:36:09.108242+0000","last_peered":"2024-03-28T23:36:09.108242+0000","last_clean":"2024-03-28T23:36:09.108242+0000","last_became_active":"2024-03-28T23:35:59.079742+0000","last_became_peered":"2024-03-28T23:35:59.079742+0000","last_unstale":"2024-03-28T23:36:09.108242+0000","last_undegraded":"2024-03-28T23:36:09.108242+0000","last_fullsized":"2024-03-28T23:36:09.108242+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:11.349017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117502+0000","last_change":"2024-03-28T23:36:08.111135+0000","last_active":"2024-03-28T23:36:09.117502+0000","last_peered":"2024-03-28T23:36:09.117502+0000","last_clean":"2024-03-28T23:36:09.117502+0000","last_became_active":"2024-03-28T23:36:08.110815+0000","last_became_peered":"2024-03-28T23:36:08.110815+0000","last_unstale":"2024-03-28T23:36:09.117502+0000","last_undegraded":"2024-03-28T23:36:09.117502+0000","last_fullsized":"2024-03-28T23:36:09.117502+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:57:15.318239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.104103+0000","last_change":"2024-03-28T23:35:59.081618+0000","last_active":"2024-03-28T23:36:08.104103+0000","last_peered":"2024-03-28T23:36:08.104103+0000","last_clean":"2024-03-28T23:36:08.104103+0000","last_became_active":"2024-03-28T23:35:59.080909+0000","last_became_peered":"2024-03-28T23:35:59.080909+0000","last_unstale":"2024-03-28T23:36:08.104103+0000","last_undegraded":"2024-03-28T23:36:08.104103+0000","last_fullsized":"2024-03-28T23:36:08.104103+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:15.874144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.110770+0000","last_change":"2024-03-28T23:36:08.109599+0000","last_active":"2024-03-28T23:36:08.110770+0000","last_peered":"2024-03-28T23:36:08.110770+0000","last_clean":"2024-03-28T23:36:08.110770+0000","last_became_active":"2024-03-28T23:36:08.109324+0000","last_became_peered":"2024-03-28T23:36:08.109324+0000","last_unstale":"2024-03-28T23:36:08.110770+0000","last_undegraded":"2024-03-28T23:36:08.110770+0000","last_fullsized":"2024-03-28T23:36:08.110770+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:14:39.976869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120523+0000","last_change":"2024-03-28T23:35:59.072123+0000","last_active":"2024-03-28T23:36:09.120523+0000","last_peered":"2024-03-28T23:36:09.120523+0000","last_clean":"2024-03-28T23:36:09.120523+0000","last_became_active":"2024-03-28T23:35:59.071827+0000","last_became_peered":"2024-03-28T23:35:59.071827+0000","last_unstale":"2024-03-28T23:36:09.120523+0000","last_undegraded":"2024-03-28T23:36:09.120523+0000","last_fullsized":"2024-03-28T23:36:09.120523+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:11.814112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109514+0000","last_change":"2024-03-28T23:36:08.118037+0000","last_active":"2024-03-28T23:36:09.109514+0000","last_peered":"2024-03-28T23:36:09.109514+0000","last_clean":"2024-03-28T23:36:09.109514+0000","last_became_active":"2024-03-28T23:36:08.117767+0000","last_became_peered":"2024-03-28T23:36:08.117767+0000","last_unstale":"2024-03-28T23:36:09.109514+0000","last_undegraded":"2024-03-28T23:36:09.109514+0000","last_fullsized":"2024-03-28T23:36:09.109514+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:07:12.245731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.099214+0000","last_change":"2024-03-28T23:35:59.070208+0000","last_active":"2024-03-28T23:36:08.099214+0000","last_peered":"2024-03-28T23:36:08.099214+0000","last_clean":"2024-03-28T23:36:08.099214+0000","last_became_active":"2024-03-28T23:35:59.069928+0000","last_became_peered":"2024-03-28T23:35:59.069928+0000","last_unstale":"2024-03-28T23:36:08.099214+0000","last_undegraded":"2024-03-28T23:36:08.099214+0000","last_fullsized":"2024-03-28T23:36:08.099214+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:43:53.020768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109999+0000","last_change":"2024-03-28T23:36:08.117259+0000","last_active":"2024-03-28T23:36:09.109999+0000","last_peered":"2024-03-28T23:36:09.109999+0000","last_clean":"2024-03-28T23:36:09.109999+0000","last_became_active":"2024-03-28T23:36:08.116626+0000","last_became_peered":"2024-03-28T23:36:08.116626+0000","last_unstale":"2024-03-28T23:36:09.109999+0000","last_undegraded":"2024-03-28T23:36:09.109999+0000","last_fullsized":"2024-03-28T23:36:09.109999+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:25:31.117661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109336+0000","last_change":"2024-03-28T23:35:59.073450+0000","last_active":"2024-03-28T23:36:09.109336+0000","last_peered":"2024-03-28T23:36:09.109336+0000","last_clean":"2024-03-28T23:36:09.109336+0000","last_became_active":"2024-03-28T23:35:59.073178+0000","last_became_peered":"2024-03-28T23:35:59.073178+0000","last_unstale":"2024-03-28T23:36:09.109336+0000","last_undegraded":"2024-03-28T23:36:09.109336+0000","last_fullsized":"2024-03-28T23:36:09.109336+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:20:09.724407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.110645+0000","last_change":"2024-03-28T23:36:08.109669+0000","last_active":"2024-03-28T23:36:08.110645+0000","last_peered":"2024-03-28T23:36:08.110645+0000","last_clean":"2024-03-28T23:36:08.110645+0000","last_became_active":"2024-03-28T23:36:08.109427+0000","last_became_peered":"2024-03-28T23:36:08.109427+0000","last_unstale":"2024-03-28T23:36:08.110645+0000","last_undegraded":"2024-03-28T23:36:08.110645+0000","last_fullsized":"2024-03-28T23:36:08.110645+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:36:44.285827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117711+0000","last_change":"2024-03-28T23:35:59.080957+0000","last_active":"2024-03-28T23:36:09.117711+0000","last_peered":"2024-03-28T23:36:09.117711+0000","last_clean":"2024-03-28T23:36:09.117711+0000","last_became_active":"2024-03-28T23:35:59.080693+0000","last_became_peered":"2024-03-28T23:35:59.080693+0000","last_unstale":"2024-03-28T23:36:09.117711+0000","last_undegraded":"2024-03-28T23:36:09.117711+0000","last_fullsized":"2024-03-28T23:36:09.117711+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:53.689813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109053+0000","last_change":"2024-03-28T23:36:08.125499+0000","last_active":"2024-03-28T23:36:09.109053+0000","last_peered":"2024-03-28T23:36:09.109053+0000","last_clean":"2024-03-28T23:36:09.109053+0000","last_became_active":"2024-03-28T23:36:08.125246+0000","last_became_peered":"2024-03-28T23:36:08.125246+0000","last_unstale":"2024-03-28T23:36:09.109053+0000","last_undegraded":"2024-03-28T23:36:09.109053+0000","last_fullsized":"2024-03-28T23:36:09.109053+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:25:29.717562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109620+0000","last_change":"2024-03-28T23:35:59.080315+0000","last_active":"2024-03-28T23:36:09.109620+0000","last_peered":"2024-03-28T23:36:09.109620+0000","last_clean":"2024-03-28T23:36:09.109620+0000","last_became_active":"2024-03-28T23:35:59.080039+0000","last_became_peered":"2024-03-28T23:35:59.080039+0000","last_unstale":"2024-03-28T23:36:09.109620+0000","last_undegraded":"2024-03-28T23:36:09.109620+0000","last_fullsized":"2024-03-28T23:36:09.109620+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:25:20.678853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119582+0000","last_change":"2024-03-28T23:36:08.124154+0000","last_active":"2024-03-28T23:36:09.119582+0000","last_peered":"2024-03-28T23:36:09.119582+0000","last_clean":"2024-03-28T23:36:09.119582+0000","last_became_active":"2024-03-28T23:36:08.123927+0000","last_became_peered":"2024-03-28T23:36:08.123927+0000","last_unstale":"2024-03-28T23:36:09.119582+0000","last_undegraded":"2024-03-28T23:36:09.119582+0000","last_fullsized":"2024-03-28T23:36:09.119582+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:12:20.749280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108784+0000","last_change":"2024-03-28T23:35:59.074798+0000","last_active":"2024-03-28T23:36:09.108784+0000","last_peered":"2024-03-28T23:36:09.108784+0000","last_clean":"2024-03-28T23:36:09.108784+0000","last_became_active":"2024-03-28T23:35:59.074539+0000","last_became_peered":"2024-03-28T23:35:59.074539+0000","last_unstale":"2024-03-28T23:36:09.108784+0000","last_undegraded":"2024-03-28T23:36:09.108784+0000","last_fullsized":"2024-03-28T23:36:09.108784+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:51:06.388640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119806+0000","last_change":"2024-03-28T23:36:08.191662+0000","last_active":"2024-03-28T23:36:09.119806+0000","last_peered":"2024-03-28T23:36:09.119806+0000","last_clean":"2024-03-28T23:36:09.119806+0000","last_became_active":"2024-03-28T23:36:08.191396+0000","last_became_peered":"2024-03-28T23:36:08.191396+0000","last_unstale":"2024-03-28T23:36:09.119806+0000","last_undegraded":"2024-03-28T23:36:09.119806+0000","last_fullsized":"2024-03-28T23:36:09.119806+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:13:33.254712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.106708+0000","last_change":"2024-03-28T23:35:59.079811+0000","last_active":"2024-03-28T23:36:08.106708+0000","last_peered":"2024-03-28T23:36:08.106708+0000","last_clean":"2024-03-28T23:36:08.106708+0000","last_became_active":"2024-03-28T23:35:59.079538+0000","last_became_peered":"2024-03-28T23:35:59.079538+0000","last_unstale":"2024-03-28T23:36:08.106708+0000","last_undegraded":"2024-03-28T23:36:08.106708+0000","last_fullsized":"2024-03-28T23:36:08.106708+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:49:26.989533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119743+0000","last_change":"2024-03-28T23:36:08.191742+0000","last_active":"2024-03-28T23:36:09.119743+0000","last_peered":"2024-03-28T23:36:09.119743+0000","last_clean":"2024-03-28T23:36:09.119743+0000","last_became_active":"2024-03-28T23:36:08.191455+0000","last_became_peered":"2024-03-28T23:36:08.191455+0000","last_unstale":"2024-03-28T23:36:09.119743+0000","last_undegraded":"2024-03-28T23:36:09.119743+0000","last_fullsized":"2024-03-28T23:36:09.119743+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:21:51.431166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.104132+0000","last_change":"2024-03-28T23:35:59.080958+0000","last_active":"2024-03-28T23:36:08.104132+0000","last_peered":"2024-03-28T23:36:08.104132+0000","last_clean":"2024-03-28T23:36:08.104132+0000","last_became_active":"2024-03-28T23:35:59.080586+0000","last_became_peered":"2024-03-28T23:35:59.080586+0000","last_unstale":"2024-03-28T23:36:08.104132+0000","last_undegraded":"2024-03-28T23:36:08.104132+0000","last_fullsized":"2024-03-28T23:36:08.104132+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:53:35.858409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109688+0000","last_change":"2024-03-28T23:36:08.112625+0000","last_active":"2024-03-28T23:36:09.109688+0000","last_peered":"2024-03-28T23:36:09.109688+0000","last_clean":"2024-03-28T23:36:09.109688+0000","last_became_active":"2024-03-28T23:36:08.112430+0000","last_became_peered":"2024-03-28T23:36:08.112430+0000","last_unstale":"2024-03-28T23:36:09.109688+0000","last_undegraded":"2024-03-28T23:36:09.109688+0000","last_fullsized":"2024-03-28T23:36:09.109688+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:08:07.459225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.110219+0000","last_change":"2024-03-28T23:35:59.076376+0000","last_active":"2024-03-28T23:36:09.110219+0000","last_peered":"2024-03-28T23:36:09.110219+0000","last_clean":"2024-03-28T23:36:09.110219+0000","last_became_active":"2024-03-28T23:35:59.076009+0000","last_became_peered":"2024-03-28T23:35:59.076009+0000","last_unstale":"2024-03-28T23:36:09.110219+0000","last_undegraded":"2024-03-28T23:36:09.110219+0000","last_fullsized":"2024-03-28T23:36:09.110219+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:30:23.264136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109420+0000","last_change":"2024-03-28T23:36:08.116900+0000","last_active":"2024-03-28T23:36:09.109420+0000","last_peered":"2024-03-28T23:36:09.109420+0000","last_clean":"2024-03-28T23:36:09.109420+0000","last_became_active":"2024-03-28T23:36:08.116268+0000","last_became_peered":"2024-03-28T23:36:08.116268+0000","last_unstale":"2024-03-28T23:36:09.109420+0000","last_undegraded":"2024-03-28T23:36:09.109420+0000","last_fullsized":"2024-03-28T23:36:09.109420+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:25:16.454692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117915+0000","last_change":"2024-03-28T23:35:59.069546+0000","last_active":"2024-03-28T23:36:09.117915+0000","last_peered":"2024-03-28T23:36:09.117915+0000","last_clean":"2024-03-28T23:36:09.117915+0000","last_became_active":"2024-03-28T23:35:59.069283+0000","last_became_peered":"2024-03-28T23:35:59.069283+0000","last_unstale":"2024-03-28T23:36:09.117915+0000","last_undegraded":"2024-03-28T23:36:09.117915+0000","last_fullsized":"2024-03-28T23:36:09.117915+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:36:41.518215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.118029+0000","last_change":"2024-03-28T23:36:08.117498+0000","last_active":"2024-03-28T23:36:09.118029+0000","last_peered":"2024-03-28T23:36:09.118029+0000","last_clean":"2024-03-28T23:36:09.118029+0000","last_became_active":"2024-03-28T23:36:08.117232+0000","last_became_peered":"2024-03-28T23:36:08.117232+0000","last_unstale":"2024-03-28T23:36:09.118029+0000","last_undegraded":"2024-03-28T23:36:09.118029+0000","last_fullsized":"2024-03-28T23:36:09.118029+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:27:51.966745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109675+0000","last_change":"2024-03-28T23:35:59.073880+0000","last_active":"2024-03-28T23:36:09.109675+0000","last_peered":"2024-03-28T23:36:09.109675+0000","last_clean":"2024-03-28T23:36:09.109675+0000","last_became_active":"2024-03-28T23:35:59.073579+0000","last_became_peered":"2024-03-28T23:35:59.073579+0000","last_unstale":"2024-03-28T23:36:09.109675+0000","last_undegraded":"2024-03-28T23:36:09.109675+0000","last_fullsized":"2024-03-28T23:36:09.109675+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:54:04.050429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109021+0000","last_change":"2024-03-28T23:36:08.118174+0000","last_active":"2024-03-28T23:36:09.109021+0000","last_peered":"2024-03-28T23:36:09.109021+0000","last_clean":"2024-03-28T23:36:09.109021+0000","last_became_active":"2024-03-28T23:36:08.117488+0000","last_became_peered":"2024-03-28T23:36:08.117488+0000","last_unstale":"2024-03-28T23:36:09.109021+0000","last_undegraded":"2024-03-28T23:36:09.109021+0000","last_fullsized":"2024-03-28T23:36:09.109021+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:44:32.559938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109361+0000","last_change":"2024-03-28T23:35:59.082034+0000","last_active":"2024-03-28T23:36:09.109361+0000","last_peered":"2024-03-28T23:36:09.109361+0000","last_clean":"2024-03-28T23:36:09.109361+0000","last_became_active":"2024-03-28T23:35:59.081763+0000","last_became_peered":"2024-03-28T23:35:59.081763+0000","last_unstale":"2024-03-28T23:36:09.109361+0000","last_undegraded":"2024-03-28T23:36:09.109361+0000","last_fullsized":"2024-03-28T23:36:09.109361+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:07:58.420516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117476+0000","last_change":"2024-03-28T23:36:08.128000+0000","last_active":"2024-03-28T23:36:09.117476+0000","last_peered":"2024-03-28T23:36:09.117476+0000","last_clean":"2024-03-28T23:36:09.117476+0000","last_became_active":"2024-03-28T23:36:08.127716+0000","last_became_peered":"2024-03-28T23:36:08.127716+0000","last_unstale":"2024-03-28T23:36:09.117476+0000","last_undegraded":"2024-03-28T23:36:09.117476+0000","last_fullsized":"2024-03-28T23:36:09.117476+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:11:58.809402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108982+0000","last_change":"2024-03-28T23:35:59.073561+0000","last_active":"2024-03-28T23:36:09.108982+0000","last_peered":"2024-03-28T23:36:09.108982+0000","last_clean":"2024-03-28T23:36:09.108982+0000","last_became_active":"2024-03-28T23:35:59.073301+0000","last_became_peered":"2024-03-28T23:35:59.073301+0000","last_unstale":"2024-03-28T23:36:09.108982+0000","last_undegraded":"2024-03-28T23:36:09.108982+0000","last_fullsized":"2024-03-28T23:36:09.108982+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:24.837863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108984+0000","last_change":"2024-03-28T23:36:08.116426+0000","last_active":"2024-03-28T23:36:09.108984+0000","last_peered":"2024-03-28T23:36:09.108984+0000","last_clean":"2024-03-28T23:36:09.108984+0000","last_became_active":"2024-03-28T23:36:08.116113+0000","last_became_peered":"2024-03-28T23:36:08.116113+0000","last_unstale":"2024-03-28T23:36:09.108984+0000","last_undegraded":"2024-03-28T23:36:09.108984+0000","last_fullsized":"2024-03-28T23:36:09.108984+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:18:29.164476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.099368+0000","last_change":"2024-03-28T23:35:59.074502+0000","last_active":"2024-03-28T23:36:08.099368+0000","last_peered":"2024-03-28T23:36:08.099368+0000","last_clean":"2024-03-28T23:36:08.099368+0000","last_became_active":"2024-03-28T23:35:59.074214+0000","last_became_peered":"2024-03-28T23:35:59.074214+0000","last_unstale":"2024-03-28T23:36:08.099368+0000","last_undegraded":"2024-03-28T23:36:08.099368+0000","last_fullsized":"2024-03-28T23:36:08.099368+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:03.147158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108645+0000","last_change":"2024-03-28T23:36:08.120168+0000","last_active":"2024-03-28T23:36:09.108645+0000","last_peered":"2024-03-28T23:36:09.108645+0000","last_clean":"2024-03-28T23:36:09.108645+0000","last_became_active":"2024-03-28T23:36:08.119896+0000","last_became_peered":"2024-03-28T23:36:08.119896+0000","last_unstale":"2024-03-28T23:36:09.108645+0000","last_undegraded":"2024-03-28T23:36:09.108645+0000","last_fullsized":"2024-03-28T23:36:09.108645+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:08:39.946119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108354+0000","last_change":"2024-03-28T23:35:59.080452+0000","last_active":"2024-03-28T23:36:09.108354+0000","last_peered":"2024-03-28T23:36:09.108354+0000","last_clean":"2024-03-28T23:36:09.108354+0000","last_became_active":"2024-03-28T23:35:59.080141+0000","last_became_peered":"2024-03-28T23:35:59.080141+0000","last_unstale":"2024-03-28T23:36:09.108354+0000","last_undegraded":"2024-03-28T23:36:09.108354+0000","last_fullsized":"2024-03-28T23:36:09.108354+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:16:00.430973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108717+0000","last_change":"2024-03-28T23:36:08.125030+0000","last_active":"2024-03-28T23:36:09.108717+0000","last_peered":"2024-03-28T23:36:09.108717+0000","last_clean":"2024-03-28T23:36:09.108717+0000","last_became_active":"2024-03-28T23:36:08.124772+0000","last_became_peered":"2024-03-28T23:36:08.124772+0000","last_unstale":"2024-03-28T23:36:09.108717+0000","last_undegraded":"2024-03-28T23:36:09.108717+0000","last_fullsized":"2024-03-28T23:36:09.108717+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:36:55.152832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.188603+0000","last_change":"2024-03-28T23:35:59.068642+0000","last_active":"2024-03-28T23:36:08.188603+0000","last_peered":"2024-03-28T23:36:08.188603+0000","last_clean":"2024-03-28T23:36:08.188603+0000","last_became_active":"2024-03-28T23:35:59.068349+0000","last_became_peered":"2024-03-28T23:35:59.068349+0000","last_unstale":"2024-03-28T23:36:08.188603+0000","last_undegraded":"2024-03-28T23:36:08.188603+0000","last_fullsized":"2024-03-28T23:36:08.188603+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:29.399434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.116944+0000","last_change":"2024-03-28T23:36:08.116485+0000","last_active":"2024-03-28T23:36:08.116944+0000","last_peered":"2024-03-28T23:36:08.116944+0000","last_clean":"2024-03-28T23:36:08.116944+0000","last_became_active":"2024-03-28T23:36:08.116068+0000","last_became_peered":"2024-03-28T23:36:08.116068+0000","last_unstale":"2024-03-28T23:36:08.116944+0000","last_undegraded":"2024-03-28T23:36:08.116944+0000","last_fullsized":"2024-03-28T23:36:08.116944+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:44.966490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.106595+0000","last_change":"2024-03-28T23:36:07.106269+0000","last_active":"2024-03-28T23:36:08.106595+0000","last_peered":"2024-03-28T23:36:08.106595+0000","last_clean":"2024-03-28T23:36:08.106595+0000","last_became_active":"2024-03-28T23:36:07.106026+0000","last_became_peered":"2024-03-28T23:36:07.106026+0000","last_unstale":"2024-03-28T23:36:08.106595+0000","last_undegraded":"2024-03-28T23:36:08.106595+0000","last_fullsized":"2024-03-28T23:36:08.106595+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:55.739448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109156+0000","last_change":"2024-03-28T23:35:59.083382+0000","last_active":"2024-03-28T23:36:09.109156+0000","last_peered":"2024-03-28T23:36:09.109156+0000","last_clean":"2024-03-28T23:36:09.109156+0000","last_became_active":"2024-03-28T23:35:59.083111+0000","last_became_peered":"2024-03-28T23:35:59.083111+0000","last_unstale":"2024-03-28T23:36:09.109156+0000","last_undegraded":"2024-03-28T23:36:09.109156+0000","last_fullsized":"2024-03-28T23:36:09.109156+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:36:35.247118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"151'4","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.105955+0000","last_change":"2024-03-28T23:35:58.071726+0000","last_active":"2024-03-28T23:36:08.105955+0000","last_peered":"2024-03-28T23:36:08.105955+0000","last_clean":"2024-03-28T23:36:08.105955+0000","last_became_active":"2024-03-28T23:35:58.071384+0000","last_became_peered":"2024-03-28T23:35:58.071384+0000","last_unstale":"2024-03-28T23:36:08.105955+0000","last_undegraded":"2024-03-28T23:36:08.105955+0000","last_fullsized":"2024-03-28T23:36:08.105955+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:31.860347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.130226+0000","last_change":"2024-03-28T23:36:08.129860+0000","last_active":"2024-03-28T23:36:08.130226+0000","last_peered":"2024-03-28T23:36:08.130226+0000","last_clean":"2024-03-28T23:36:08.130226+0000","last_became_active":"2024-03-28T23:36:08.129659+0000","last_became_peered":"2024-03-28T23:36:08.129659+0000","last_unstale":"2024-03-28T23:36:08.130226+0000","last_undegraded":"2024-03-28T23:36:08.130226+0000","last_fullsized":"2024-03-28T23:36:08.130226+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:26:20.822468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"155'8","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108307+0000","last_change":"2024-03-28T23:36:07.108715+0000","last_active":"2024-03-28T23:36:09.108307+0000","last_peered":"2024-03-28T23:36:09.108307+0000","last_clean":"2024-03-28T23:36:09.108307+0000","last_became_active":"2024-03-28T23:36:07.108286+0000","last_became_peered":"2024-03-28T23:36:07.108286+0000","last_unstale":"2024-03-28T23:36:09.108307+0000","last_undegraded":"2024-03-28T23:36:09.108307+0000","last_fullsized":"2024-03-28T23:36:09.108307+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:25.263616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108953+0000","last_change":"2024-03-28T23:35:59.082507+0000","last_active":"2024-03-28T23:36:09.108953+0000","last_peered":"2024-03-28T23:36:09.108953+0000","last_clean":"2024-03-28T23:36:09.108953+0000","last_became_active":"2024-03-28T23:35:59.082004+0000","last_became_peered":"2024-03-28T23:35:59.082004+0000","last_unstale":"2024-03-28T23:36:09.108953+0000","last_undegraded":"2024-03-28T23:36:09.108953+0000","last_fullsized":"2024-03-28T23:36:09.108953+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:54:41.513429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"151'1","reported_seq":21,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.109522+0000","last_change":"2024-03-28T23:35:58.070559+0000","last_active":"2024-03-28T23:36:08.109522+0000","last_peered":"2024-03-28T23:36:08.109522+0000","last_clean":"2024-03-28T23:36:08.109522+0000","last_became_active":"2024-03-28T23:35:58.070259+0000","last_became_peered":"2024-03-28T23:35:58.070259+0000","last_unstale":"2024-03-28T23:36:08.109522+0000","last_undegraded":"2024-03-28T23:36:08.109522+0000","last_fullsized":"2024-03-28T23:36:08.109522+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:23.832628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117349+0000","last_change":"2024-03-28T23:36:08.112610+0000","last_active":"2024-03-28T23:36:09.117349+0000","last_peered":"2024-03-28T23:36:09.117349+0000","last_clean":"2024-03-28T23:36:09.117349+0000","last_became_active":"2024-03-28T23:36:08.111833+0000","last_became_peered":"2024-03-28T23:36:08.111833+0000","last_unstale":"2024-03-28T23:36:09.117349+0000","last_undegraded":"2024-03-28T23:36:09.117349+0000","last_fullsized":"2024-03-28T23:36:09.117349+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:13:35.088394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.124175+0000","last_change":"2024-03-28T23:36:07.101366+0000","last_active":"2024-03-28T23:36:08.124175+0000","last_peered":"2024-03-28T23:36:08.124175+0000","last_clean":"2024-03-28T23:36:08.124175+0000","last_became_active":"2024-03-28T23:36:07.101016+0000","last_became_peered":"2024-03-28T23:36:07.101016+0000","last_unstale":"2024-03-28T23:36:08.124175+0000","last_undegraded":"2024-03-28T23:36:08.124175+0000","last_fullsized":"2024-03-28T23:36:08.124175+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:42:49.756856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.113087+0000","last_change":"2024-03-28T23:35:59.082185+0000","last_active":"2024-03-28T23:36:08.113087+0000","last_peered":"2024-03-28T23:36:08.113087+0000","last_clean":"2024-03-28T23:36:08.113087+0000","last_became_active":"2024-03-28T23:35:59.081892+0000","last_became_peered":"2024-03-28T23:35:59.081892+0000","last_unstale":"2024-03-28T23:36:08.113087+0000","last_undegraded":"2024-03-28T23:36:08.113087+0000","last_fullsized":"2024-03-28T23:36:08.113087+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:30:33.779993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"151'2","reported_seq":24,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109940+0000","last_change":"2024-03-28T23:35:58.069940+0000","last_active":"2024-03-28T23:36:09.109940+0000","last_peered":"2024-03-28T23:36:09.109940+0000","last_clean":"2024-03-28T23:36:09.109940+0000","last_became_active":"2024-03-28T23:35:58.069533+0000","last_became_peered":"2024-03-28T23:35:58.069533+0000","last_unstale":"2024-03-28T23:36:09.109940+0000","last_undegraded":"2024-03-28T23:36:09.109940+0000","last_fullsized":"2024-03-28T23:36:09.109940+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:14.868910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108903+0000","last_change":"2024-03-28T23:36:08.192478+0000","last_active":"2024-03-28T23:36:09.108903+0000","last_peered":"2024-03-28T23:36:09.108903+0000","last_clean":"2024-03-28T23:36:09.108903+0000","last_became_active":"2024-03-28T23:36:08.192173+0000","last_became_peered":"2024-03-28T23:36:08.192173+0000","last_unstale":"2024-03-28T23:36:09.108903+0000","last_undegraded":"2024-03-28T23:36:09.108903+0000","last_fullsized":"2024-03-28T23:36:09.108903+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:33.489134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120148+0000","last_change":"2024-03-28T23:36:07.111796+0000","last_active":"2024-03-28T23:36:09.120148+0000","last_peered":"2024-03-28T23:36:09.120148+0000","last_clean":"2024-03-28T23:36:09.120148+0000","last_became_active":"2024-03-28T23:36:07.111521+0000","last_became_peered":"2024-03-28T23:36:07.111521+0000","last_unstale":"2024-03-28T23:36:09.120148+0000","last_undegraded":"2024-03-28T23:36:09.120148+0000","last_fullsized":"2024-03-28T23:36:09.120148+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:49:35.024399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109457+0000","last_change":"2024-03-28T23:35:59.080903+0000","last_active":"2024-03-28T23:36:09.109457+0000","last_peered":"2024-03-28T23:36:09.109457+0000","last_clean":"2024-03-28T23:36:09.109457+0000","last_became_active":"2024-03-28T23:35:59.080632+0000","last_became_peered":"2024-03-28T23:35:59.080632+0000","last_unstale":"2024-03-28T23:36:09.109457+0000","last_undegraded":"2024-03-28T23:36:09.109457+0000","last_fullsized":"2024-03-28T23:36:09.109457+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:37:10.552720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"151'2","reported_seq":24,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117800+0000","last_change":"2024-03-28T23:35:58.064254+0000","last_active":"2024-03-28T23:36:09.117800+0000","last_peered":"2024-03-28T23:36:09.117800+0000","last_clean":"2024-03-28T23:36:09.117800+0000","last_became_active":"2024-03-28T23:35:58.063972+0000","last_became_peered":"2024-03-28T23:35:58.063972+0000","last_unstale":"2024-03-28T23:36:09.117800+0000","last_undegraded":"2024-03-28T23:36:09.117800+0000","last_fullsized":"2024-03-28T23:36:09.117800+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:36:14.911401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.110627+0000","last_change":"2024-03-28T23:36:08.109190+0000","last_active":"2024-03-28T23:36:08.110627+0000","last_peered":"2024-03-28T23:36:08.110627+0000","last_clean":"2024-03-28T23:36:08.110627+0000","last_became_active":"2024-03-28T23:36:08.108910+0000","last_became_peered":"2024-03-28T23:36:08.108910+0000","last_unstale":"2024-03-28T23:36:08.110627+0000","last_undegraded":"2024-03-28T23:36:08.110627+0000","last_fullsized":"2024-03-28T23:36:08.110627+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:26:10.148176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"155'3","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.129846+0000","last_change":"2024-03-28T23:36:07.115429+0000","last_active":"2024-03-28T23:36:08.129846+0000","last_peered":"2024-03-28T23:36:08.129846+0000","last_clean":"2024-03-28T23:36:08.129846+0000","last_became_active":"2024-03-28T23:36:07.115174+0000","last_became_peered":"2024-03-28T23:36:07.115174+0000","last_unstale":"2024-03-28T23:36:08.129846+0000","last_undegraded":"2024-03-28T23:36:08.129846+0000","last_fullsized":"2024-03-28T23:36:08.129846+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:21:22.235323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120195+0000","last_change":"2024-03-28T23:35:59.077647+0000","last_active":"2024-03-28T23:36:09.120195+0000","last_peered":"2024-03-28T23:36:09.120195+0000","last_clean":"2024-03-28T23:36:09.120195+0000","last_became_active":"2024-03-28T23:35:59.077374+0000","last_became_peered":"2024-03-28T23:35:59.077374+0000","last_unstale":"2024-03-28T23:36:09.120195+0000","last_undegraded":"2024-03-28T23:36:09.120195+0000","last_fullsized":"2024-03-28T23:36:09.120195+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:37:10.552720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"151'1","reported_seq":23,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119886+0000","last_change":"2024-03-28T23:35:58.074449+0000","last_active":"2024-03-28T23:36:09.119886+0000","last_peered":"2024-03-28T23:36:09.119886+0000","last_clean":"2024-03-28T23:36:09.119886+0000","last_became_active":"2024-03-28T23:35:58.074171+0000","last_became_peered":"2024-03-28T23:35:58.074171+0000","last_unstale":"2024-03-28T23:36:09.119886+0000","last_undegraded":"2024-03-28T23:36:09.119886+0000","last_fullsized":"2024-03-28T23:36:09.119886+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:41:31.356630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120382+0000","last_change":"2024-03-28T23:36:08.115737+0000","last_active":"2024-03-28T23:36:09.120382+0000","last_peered":"2024-03-28T23:36:09.120382+0000","last_clean":"2024-03-28T23:36:09.120382+0000","last_became_active":"2024-03-28T23:36:08.115468+0000","last_became_peered":"2024-03-28T23:36:08.115468+0000","last_unstale":"2024-03-28T23:36:09.120382+0000","last_undegraded":"2024-03-28T23:36:09.120382+0000","last_fullsized":"2024-03-28T23:36:09.120382+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:54.124466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"155'6","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.125866+0000","last_change":"2024-03-28T23:36:07.098719+0000","last_active":"2024-03-28T23:36:08.125866+0000","last_peered":"2024-03-28T23:36:08.125866+0000","last_clean":"2024-03-28T23:36:08.125866+0000","last_became_active":"2024-03-28T23:36:07.098395+0000","last_became_peered":"2024-03-28T23:36:07.098395+0000","last_unstale":"2024-03-28T23:36:08.125866+0000","last_undegraded":"2024-03-28T23:36:08.125866+0000","last_fullsized":"2024-03-28T23:36:08.125866+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:53:30.740785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117996+0000","last_change":"2024-03-28T23:35:59.086653+0000","last_active":"2024-03-28T23:36:09.117996+0000","last_peered":"2024-03-28T23:36:09.117996+0000","last_clean":"2024-03-28T23:36:09.117996+0000","last_became_active":"2024-03-28T23:35:59.086398+0000","last_became_peered":"2024-03-28T23:35:59.086398+0000","last_unstale":"2024-03-28T23:36:09.117996+0000","last_undegraded":"2024-03-28T23:36:09.117996+0000","last_fullsized":"2024-03-28T23:36:09.117996+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:18:46.035632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"151'1","reported_seq":21,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.112936+0000","last_change":"2024-03-28T23:35:58.073421+0000","last_active":"2024-03-28T23:36:08.112936+0000","last_peered":"2024-03-28T23:36:08.112936+0000","last_clean":"2024-03-28T23:36:08.112936+0000","last_became_active":"2024-03-28T23:35:58.072004+0000","last_became_peered":"2024-03-28T23:35:58.072004+0000","last_unstale":"2024-03-28T23:36:08.112936+0000","last_undegraded":"2024-03-28T23:36:08.112936+0000","last_fullsized":"2024-03-28T23:36:08.112936+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:10.808877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.110002+0000","last_change":"2024-03-28T23:36:08.110229+0000","last_active":"2024-03-28T23:36:09.110002+0000","last_peered":"2024-03-28T23:36:09.110002+0000","last_clean":"2024-03-28T23:36:09.110002+0000","last_became_active":"2024-03-28T23:36:08.109984+0000","last_became_peered":"2024-03-28T23:36:08.109984+0000","last_unstale":"2024-03-28T23:36:09.110002+0000","last_undegraded":"2024-03-28T23:36:09.110002+0000","last_fullsized":"2024-03-28T23:36:09.110002+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:58:10.521424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.110114+0000","last_change":"2024-03-28T23:36:07.111385+0000","last_active":"2024-03-28T23:36:09.110114+0000","last_peered":"2024-03-28T23:36:09.110114+0000","last_clean":"2024-03-28T23:36:09.110114+0000","last_became_active":"2024-03-28T23:36:07.111075+0000","last_became_peered":"2024-03-28T23:36:07.111075+0000","last_unstale":"2024-03-28T23:36:09.110114+0000","last_undegraded":"2024-03-28T23:36:09.110114+0000","last_fullsized":"2024-03-28T23:36:09.110114+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:13:32.250870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117969+0000","last_change":"2024-03-28T23:35:59.063776+0000","last_active":"2024-03-28T23:36:09.117969+0000","last_peered":"2024-03-28T23:36:09.117969+0000","last_clean":"2024-03-28T23:36:09.117969+0000","last_became_active":"2024-03-28T23:35:59.063214+0000","last_became_peered":"2024-03-28T23:35:59.063214+0000","last_unstale":"2024-03-28T23:36:09.117969+0000","last_undegraded":"2024-03-28T23:36:09.117969+0000","last_fullsized":"2024-03-28T23:36:09.117969+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:24:06.490491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117611+0000","last_change":"2024-03-28T23:35:58.072230+0000","last_active":"2024-03-28T23:36:09.117611+0000","last_peered":"2024-03-28T23:36:09.117611+0000","last_clean":"2024-03-28T23:36:09.117611+0000","last_became_active":"2024-03-28T23:35:58.071967+0000","last_became_peered":"2024-03-28T23:35:58.071967+0000","last_unstale":"2024-03-28T23:36:09.117611+0000","last_undegraded":"2024-03-28T23:36:09.117611+0000","last_fullsized":"2024-03-28T23:36:09.117611+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:07:28.431728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109881+0000","last_change":"2024-03-28T23:36:08.118205+0000","last_active":"2024-03-28T23:36:09.109881+0000","last_peered":"2024-03-28T23:36:09.109881+0000","last_clean":"2024-03-28T23:36:09.109881+0000","last_became_active":"2024-03-28T23:36:08.117900+0000","last_became_peered":"2024-03-28T23:36:08.117900+0000","last_unstale":"2024-03-28T23:36:09.109881+0000","last_undegraded":"2024-03-28T23:36:09.109881+0000","last_fullsized":"2024-03-28T23:36:09.109881+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:14.662449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120310+0000","last_change":"2024-03-28T23:36:07.115533+0000","last_active":"2024-03-28T23:36:09.120310+0000","last_peered":"2024-03-28T23:36:09.120310+0000","last_clean":"2024-03-28T23:36:09.120310+0000","last_became_active":"2024-03-28T23:36:07.115238+0000","last_became_peered":"2024-03-28T23:36:07.115238+0000","last_unstale":"2024-03-28T23:36:09.120310+0000","last_undegraded":"2024-03-28T23:36:09.120310+0000","last_fullsized":"2024-03-28T23:36:09.120310+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:44:31.556096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.105868+0000","last_change":"2024-03-28T23:35:59.070728+0000","last_active":"2024-03-28T23:36:08.105868+0000","last_peered":"2024-03-28T23:36:08.105868+0000","last_clean":"2024-03-28T23:36:08.105868+0000","last_became_active":"2024-03-28T23:35:59.070489+0000","last_became_peered":"2024-03-28T23:35:59.070489+0000","last_unstale":"2024-03-28T23:36:08.105868+0000","last_undegraded":"2024-03-28T23:36:08.105868+0000","last_fullsized":"2024-03-28T23:36:08.105868+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:24:07.077373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119635+0000","last_change":"2024-03-28T23:35:58.068807+0000","last_active":"2024-03-28T23:36:09.119635+0000","last_peered":"2024-03-28T23:36:09.119635+0000","last_clean":"2024-03-28T23:36:09.119635+0000","last_became_active":"2024-03-28T23:35:58.068530+0000","last_became_peered":"2024-03-28T23:35:58.068530+0000","last_unstale":"2024-03-28T23:36:09.119635+0000","last_undegraded":"2024-03-28T23:36:09.119635+0000","last_fullsized":"2024-03-28T23:36:09.119635+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:12.327316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119898+0000","last_change":"2024-03-28T23:36:08.113949+0000","last_active":"2024-03-28T23:36:09.119898+0000","last_peered":"2024-03-28T23:36:09.119898+0000","last_clean":"2024-03-28T23:36:09.119898+0000","last_became_active":"2024-03-28T23:36:08.113679+0000","last_became_peered":"2024-03-28T23:36:08.113679+0000","last_unstale":"2024-03-28T23:36:09.119898+0000","last_undegraded":"2024-03-28T23:36:09.119898+0000","last_fullsized":"2024-03-28T23:36:09.119898+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:39:57.434327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.104220+0000","last_change":"2024-03-28T23:36:07.101098+0000","last_active":"2024-03-28T23:36:08.104220+0000","last_peered":"2024-03-28T23:36:08.104220+0000","last_clean":"2024-03-28T23:36:08.104220+0000","last_became_active":"2024-03-28T23:36:07.100726+0000","last_became_peered":"2024-03-28T23:36:07.100726+0000","last_unstale":"2024-03-28T23:36:08.104220+0000","last_undegraded":"2024-03-28T23:36:08.104220+0000","last_fullsized":"2024-03-28T23:36:08.104220+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:06:45.247578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108910+0000","last_change":"2024-03-28T23:35:59.082657+0000","last_active":"2024-03-28T23:36:09.108910+0000","last_peered":"2024-03-28T23:36:09.108910+0000","last_clean":"2024-03-28T23:36:09.108910+0000","last_became_active":"2024-03-28T23:35:59.082390+0000","last_became_peered":"2024-03-28T23:35:59.082390+0000","last_unstale":"2024-03-28T23:36:09.108910+0000","last_undegraded":"2024-03-28T23:36:09.108910+0000","last_fullsized":"2024-03-28T23:36:09.108910+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:14:30.938161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.110165+0000","last_change":"2024-03-28T23:35:58.075215+0000","last_active":"2024-03-28T23:36:09.110165+0000","last_peered":"2024-03-28T23:36:09.110165+0000","last_clean":"2024-03-28T23:36:09.110165+0000","last_became_active":"2024-03-28T23:35:58.074891+0000","last_became_peered":"2024-03-28T23:35:58.074891+0000","last_unstale":"2024-03-28T23:36:09.110165+0000","last_undegraded":"2024-03-28T23:36:09.110165+0000","last_fullsized":"2024-03-28T23:36:09.110165+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:16.223515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.120777+0000","last_change":"2024-03-28T23:36:08.119235+0000","last_active":"2024-03-28T23:36:08.120777+0000","last_peered":"2024-03-28T23:36:08.120777+0000","last_clean":"2024-03-28T23:36:08.120777+0000","last_became_active":"2024-03-28T23:36:08.118875+0000","last_became_peered":"2024-03-28T23:36:08.118875+0000","last_unstale":"2024-03-28T23:36:08.120777+0000","last_undegraded":"2024-03-28T23:36:08.120777+0000","last_fullsized":"2024-03-28T23:36:08.120777+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:41.386690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109451+0000","last_change":"2024-03-28T23:36:07.110312+0000","last_active":"2024-03-28T23:36:09.109451+0000","last_peered":"2024-03-28T23:36:09.109451+0000","last_clean":"2024-03-28T23:36:09.109451+0000","last_became_active":"2024-03-28T23:36:07.109966+0000","last_became_peered":"2024-03-28T23:36:07.109966+0000","last_unstale":"2024-03-28T23:36:09.109451+0000","last_undegraded":"2024-03-28T23:36:09.109451+0000","last_fullsized":"2024-03-28T23:36:09.109451+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:12:19.745437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109057+0000","last_change":"2024-03-28T23:35:59.079653+0000","last_active":"2024-03-28T23:36:09.109057+0000","last_peered":"2024-03-28T23:36:09.109057+0000","last_clean":"2024-03-28T23:36:09.109057+0000","last_became_active":"2024-03-28T23:35:59.079390+0000","last_became_peered":"2024-03-28T23:35:59.079390+0000","last_unstale":"2024-03-28T23:36:09.109057+0000","last_undegraded":"2024-03-28T23:36:09.109057+0000","last_fullsized":"2024-03-28T23:36:09.109057+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:39:27.912295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"151'3","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.105309+0000","last_change":"2024-03-28T23:35:58.059191+0000","last_active":"2024-03-28T23:36:08.105309+0000","last_peered":"2024-03-28T23:36:08.105309+0000","last_clean":"2024-03-28T23:36:08.105309+0000","last_became_active":"2024-03-28T23:35:58.058860+0000","last_became_peered":"2024-03-28T23:35:58.058860+0000","last_unstale":"2024-03-28T23:36:08.105309+0000","last_undegraded":"2024-03-28T23:36:08.105309+0000","last_fullsized":"2024-03-28T23:36:08.105309+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:22.564724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119515+0000","last_change":"2024-03-28T23:36:08.121789+0000","last_active":"2024-03-28T23:36:09.119515+0000","last_peered":"2024-03-28T23:36:09.119515+0000","last_clean":"2024-03-28T23:36:09.119515+0000","last_became_active":"2024-03-28T23:36:08.121527+0000","last_became_peered":"2024-03-28T23:36:08.121527+0000","last_unstale":"2024-03-28T23:36:09.119515+0000","last_undegraded":"2024-03-28T23:36:09.119515+0000","last_fullsized":"2024-03-28T23:36:09.119515+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:56.743290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"155'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.129755+0000","last_change":"2024-03-28T23:36:07.111476+0000","last_active":"2024-03-28T23:36:08.129755+0000","last_peered":"2024-03-28T23:36:08.129755+0000","last_clean":"2024-03-28T23:36:08.129755+0000","last_became_active":"2024-03-28T23:36:07.111180+0000","last_became_peered":"2024-03-28T23:36:07.111180+0000","last_unstale":"2024-03-28T23:36:08.129755+0000","last_undegraded":"2024-03-28T23:36:08.129755+0000","last_fullsized":"2024-03-28T23:36:08.129755+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:33.319869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117521+0000","last_change":"2024-03-28T23:35:59.074941+0000","last_active":"2024-03-28T23:36:09.117521+0000","last_peered":"2024-03-28T23:36:09.117521+0000","last_clean":"2024-03-28T23:36:09.117521+0000","last_became_active":"2024-03-28T23:35:59.074636+0000","last_became_peered":"2024-03-28T23:35:59.074636+0000","last_unstale":"2024-03-28T23:36:09.117521+0000","last_undegraded":"2024-03-28T23:36:09.117521+0000","last_fullsized":"2024-03-28T23:36:09.117521+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:09.061218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"151'6","reported_seq":28,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109577+0000","last_change":"2024-03-28T23:35:58.075069+0000","last_active":"2024-03-28T23:36:09.109577+0000","last_peered":"2024-03-28T23:36:09.109577+0000","last_clean":"2024-03-28T23:36:09.109577+0000","last_became_active":"2024-03-28T23:35:58.074795+0000","last_became_peered":"2024-03-28T23:35:58.074795+0000","last_unstale":"2024-03-28T23:36:09.109577+0000","last_undegraded":"2024-03-28T23:36:09.109577+0000","last_fullsized":"2024-03-28T23:36:09.109577+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:49:07.831860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109150+0000","last_change":"2024-03-28T23:36:08.108153+0000","last_active":"2024-03-28T23:36:09.109150+0000","last_peered":"2024-03-28T23:36:09.109150+0000","last_clean":"2024-03-28T23:36:09.109150+0000","last_became_active":"2024-03-28T23:36:08.107857+0000","last_became_peered":"2024-03-28T23:36:08.107857+0000","last_unstale":"2024-03-28T23:36:09.109150+0000","last_undegraded":"2024-03-28T23:36:09.109150+0000","last_fullsized":"2024-03-28T23:36:09.109150+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:34.567279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109214+0000","last_change":"2024-03-28T23:36:07.099721+0000","last_active":"2024-03-28T23:36:09.109214+0000","last_peered":"2024-03-28T23:36:09.109214+0000","last_clean":"2024-03-28T23:36:09.109214+0000","last_became_active":"2024-03-28T23:36:07.099471+0000","last_became_peered":"2024-03-28T23:36:07.099471+0000","last_unstale":"2024-03-28T23:36:09.109214+0000","last_undegraded":"2024-03-28T23:36:09.109214+0000","last_fullsized":"2024-03-28T23:36:09.109214+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:37:18.587587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.099051+0000","last_change":"2024-03-28T23:35:59.076575+0000","last_active":"2024-03-28T23:36:08.099051+0000","last_peered":"2024-03-28T23:36:08.099051+0000","last_clean":"2024-03-28T23:36:08.099051+0000","last_became_active":"2024-03-28T23:35:59.076319+0000","last_became_peered":"2024-03-28T23:35:59.076319+0000","last_unstale":"2024-03-28T23:36:08.099051+0000","last_undegraded":"2024-03-28T23:36:08.099051+0000","last_fullsized":"2024-03-28T23:36:08.099051+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:28:45.099306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"151'2","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.108854+0000","last_change":"2024-03-28T23:35:58.073277+0000","last_active":"2024-03-28T23:36:08.108854+0000","last_peered":"2024-03-28T23:36:08.108854+0000","last_clean":"2024-03-28T23:36:08.108854+0000","last_became_active":"2024-03-28T23:35:58.071396+0000","last_became_peered":"2024-03-28T23:35:58.071396+0000","last_unstale":"2024-03-28T23:36:08.108854+0000","last_undegraded":"2024-03-28T23:36:08.108854+0000","last_fullsized":"2024-03-28T23:36:08.108854+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:36:25.234053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.190384+0000","last_change":"2024-03-28T23:36:08.188439+0000","last_active":"2024-03-28T23:36:08.190384+0000","last_peered":"2024-03-28T23:36:08.190384+0000","last_clean":"2024-03-28T23:36:08.190384+0000","last_became_active":"2024-03-28T23:36:08.187657+0000","last_became_peered":"2024-03-28T23:36:08.187657+0000","last_unstale":"2024-03-28T23:36:08.190384+0000","last_undegraded":"2024-03-28T23:36:08.190384+0000","last_fullsized":"2024-03-28T23:36:08.190384+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:22:52.579163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.123985+0000","last_change":"2024-03-28T23:36:07.097646+0000","last_active":"2024-03-28T23:36:08.123985+0000","last_peered":"2024-03-28T23:36:08.123985+0000","last_clean":"2024-03-28T23:36:08.123985+0000","last_became_active":"2024-03-28T23:36:07.097397+0000","last_became_peered":"2024-03-28T23:36:07.097397+0000","last_unstale":"2024-03-28T23:36:08.123985+0000","last_undegraded":"2024-03-28T23:36:08.123985+0000","last_fullsized":"2024-03-28T23:36:08.123985+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:20:17.759273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120104+0000","last_change":"2024-03-28T23:35:59.072656+0000","last_active":"2024-03-28T23:36:09.120104+0000","last_peered":"2024-03-28T23:36:09.120104+0000","last_clean":"2024-03-28T23:36:09.120104+0000","last_became_active":"2024-03-28T23:35:59.072400+0000","last_became_peered":"2024-03-28T23:35:59.072400+0000","last_unstale":"2024-03-28T23:36:09.120104+0000","last_undegraded":"2024-03-28T23:36:09.120104+0000","last_fullsized":"2024-03-28T23:36:09.120104+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:54:04.050429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109212+0000","last_change":"2024-03-28T23:35:58.071626+0000","last_active":"2024-03-28T23:36:09.109212+0000","last_peered":"2024-03-28T23:36:09.109212+0000","last_clean":"2024-03-28T23:36:09.109212+0000","last_became_active":"2024-03-28T23:35:58.071339+0000","last_became_peered":"2024-03-28T23:35:58.071339+0000","last_unstale":"2024-03-28T23:36:09.109212+0000","last_undegraded":"2024-03-28T23:36:09.109212+0000","last_fullsized":"2024-03-28T23:36:09.109212+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:10.808877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109855+0000","last_change":"2024-03-28T23:36:08.119583+0000","last_active":"2024-03-28T23:36:09.109855+0000","last_peered":"2024-03-28T23:36:09.109855+0000","last_clean":"2024-03-28T23:36:09.109855+0000","last_became_active":"2024-03-28T23:36:08.119334+0000","last_became_peered":"2024-03-28T23:36:08.119334+0000","last_unstale":"2024-03-28T23:36:09.109855+0000","last_undegraded":"2024-03-28T23:36:09.109855+0000","last_fullsized":"2024-03-28T23:36:09.109855+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:30:32.302845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"155'4","reported_seq":16,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.130565+0000","last_change":"2024-03-28T23:36:07.101042+0000","last_active":"2024-03-28T23:36:08.130565+0000","last_peered":"2024-03-28T23:36:08.130565+0000","last_clean":"2024-03-28T23:36:08.130565+0000","last_became_active":"2024-03-28T23:36:07.100702+0000","last_became_peered":"2024-03-28T23:36:07.100702+0000","last_unstale":"2024-03-28T23:36:08.130565+0000","last_undegraded":"2024-03-28T23:36:08.130565+0000","last_fullsized":"2024-03-28T23:36:08.130565+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:26:22.530062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109482+0000","last_change":"2024-03-28T23:35:59.076190+0000","last_active":"2024-03-28T23:36:09.109482+0000","last_peered":"2024-03-28T23:36:09.109482+0000","last_clean":"2024-03-28T23:36:09.109482+0000","last_became_active":"2024-03-28T23:35:59.075881+0000","last_became_peered":"2024-03-28T23:35:59.075881+0000","last_unstale":"2024-03-28T23:36:09.109482+0000","last_undegraded":"2024-03-28T23:36:09.109482+0000","last_fullsized":"2024-03-28T23:36:09.109482+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:50:09.386042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108479+0000","last_change":"2024-03-28T23:35:58.062542+0000","last_active":"2024-03-28T23:36:09.108479+0000","last_peered":"2024-03-28T23:36:09.108479+0000","last_clean":"2024-03-28T23:36:09.108479+0000","last_became_active":"2024-03-28T23:35:58.062074+0000","last_became_peered":"2024-03-28T23:35:58.062074+0000","last_unstale":"2024-03-28T23:36:09.108479+0000","last_undegraded":"2024-03-28T23:36:09.108479+0000","last_fullsized":"2024-03-28T23:36:09.108479+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:23:41.041246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117942+0000","last_change":"2024-03-28T23:36:08.111324+0000","last_active":"2024-03-28T23:36:09.117942+0000","last_peered":"2024-03-28T23:36:09.117942+0000","last_clean":"2024-03-28T23:36:09.117942+0000","last_became_active":"2024-03-28T23:36:08.111028+0000","last_became_peered":"2024-03-28T23:36:08.111028+0000","last_unstale":"2024-03-28T23:36:09.117942+0000","last_undegraded":"2024-03-28T23:36:09.117942+0000","last_fullsized":"2024-03-28T23:36:09.117942+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:25:43.622638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109968+0000","last_change":"2024-03-28T23:36:07.108343+0000","last_active":"2024-03-28T23:36:09.109968+0000","last_peered":"2024-03-28T23:36:09.109968+0000","last_clean":"2024-03-28T23:36:09.109968+0000","last_became_active":"2024-03-28T23:36:07.108078+0000","last_became_peered":"2024-03-28T23:36:07.108078+0000","last_unstale":"2024-03-28T23:36:09.109968+0000","last_undegraded":"2024-03-28T23:36:09.109968+0000","last_fullsized":"2024-03-28T23:36:09.109968+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:26:22.530062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.118076+0000","last_change":"2024-03-28T23:35:59.069831+0000","last_active":"2024-03-28T23:36:09.118076+0000","last_peered":"2024-03-28T23:36:09.118076+0000","last_clean":"2024-03-28T23:36:09.118076+0000","last_became_active":"2024-03-28T23:35:59.069272+0000","last_became_peered":"2024-03-28T23:35:59.069272+0000","last_unstale":"2024-03-28T23:36:09.118076+0000","last_undegraded":"2024-03-28T23:36:09.118076+0000","last_fullsized":"2024-03-28T23:36:09.118076+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:56:35.966156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"151'8","reported_seq":30,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119657+0000","last_change":"2024-03-28T23:35:58.071172+0000","last_active":"2024-03-28T23:36:09.119657+0000","last_peered":"2024-03-28T23:36:09.119657+0000","last_clean":"2024-03-28T23:36:09.119657+0000","last_became_active":"2024-03-28T23:35:58.070901+0000","last_became_peered":"2024-03-28T23:35:58.070901+0000","last_unstale":"2024-03-28T23:36:09.119657+0000","last_undegraded":"2024-03-28T23:36:09.119657+0000","last_fullsized":"2024-03-28T23:36:09.119657+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:16:53.124176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120049+0000","last_change":"2024-03-28T23:36:08.119458+0000","last_active":"2024-03-28T23:36:09.120049+0000","last_peered":"2024-03-28T23:36:09.120049+0000","last_clean":"2024-03-28T23:36:09.120049+0000","last_became_active":"2024-03-28T23:36:08.119179+0000","last_became_peered":"2024-03-28T23:36:08.119179+0000","last_unstale":"2024-03-28T23:36:09.120049+0000","last_undegraded":"2024-03-28T23:36:09.120049+0000","last_fullsized":"2024-03-28T23:36:09.120049+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:34.323711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117398+0000","last_change":"2024-03-28T23:36:07.098526+0000","last_active":"2024-03-28T23:36:09.117398+0000","last_peered":"2024-03-28T23:36:09.117398+0000","last_clean":"2024-03-28T23:36:09.117398+0000","last_became_active":"2024-03-28T23:36:07.098254+0000","last_became_peered":"2024-03-28T23:36:07.098254+0000","last_unstale":"2024-03-28T23:36:09.117398+0000","last_undegraded":"2024-03-28T23:36:09.117398+0000","last_fullsized":"2024-03-28T23:36:09.117398+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:28:53.134172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119537+0000","last_change":"2024-03-28T23:35:59.067559+0000","last_active":"2024-03-28T23:36:09.119537+0000","last_peered":"2024-03-28T23:36:09.119537+0000","last_clean":"2024-03-28T23:36:09.119537+0000","last_became_active":"2024-03-28T23:35:59.067265+0000","last_became_peered":"2024-03-28T23:35:59.067265+0000","last_unstale":"2024-03-28T23:36:09.119537+0000","last_undegraded":"2024-03-28T23:36:09.119537+0000","last_fullsized":"2024-03-28T23:36:09.119537+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:32.347982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"151'1","reported_seq":23,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.110247+0000","last_change":"2024-03-28T23:35:58.064795+0000","last_active":"2024-03-28T23:36:09.110247+0000","last_peered":"2024-03-28T23:36:09.110247+0000","last_clean":"2024-03-28T23:36:09.110247+0000","last_became_active":"2024-03-28T23:35:58.064506+0000","last_became_peered":"2024-03-28T23:35:58.064506+0000","last_unstale":"2024-03-28T23:36:09.110247+0000","last_undegraded":"2024-03-28T23:36:09.110247+0000","last_fullsized":"2024-03-28T23:36:09.110247+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:36:25.234053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108780+0000","last_change":"2024-03-28T23:36:08.114918+0000","last_active":"2024-03-28T23:36:09.108780+0000","last_peered":"2024-03-28T23:36:09.108780+0000","last_clean":"2024-03-28T23:36:09.108780+0000","last_became_active":"2024-03-28T23:36:08.114663+0000","last_became_peered":"2024-03-28T23:36:08.114663+0000","last_unstale":"2024-03-28T23:36:09.108780+0000","last_undegraded":"2024-03-28T23:36:09.108780+0000","last_fullsized":"2024-03-28T23:36:09.108780+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:56.120451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.187957+0000","last_change":"2024-03-28T23:36:07.113899+0000","last_active":"2024-03-28T23:36:08.187957+0000","last_peered":"2024-03-28T23:36:08.187957+0000","last_clean":"2024-03-28T23:36:08.187957+0000","last_became_active":"2024-03-28T23:36:07.113678+0000","last_became_peered":"2024-03-28T23:36:07.113678+0000","last_unstale":"2024-03-28T23:36:08.187957+0000","last_undegraded":"2024-03-28T23:36:08.187957+0000","last_fullsized":"2024-03-28T23:36:08.187957+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:27:04.119180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.187228+0000","last_change":"2024-03-28T23:35:59.075241+0000","last_active":"2024-03-28T23:36:08.187228+0000","last_peered":"2024-03-28T23:36:08.187228+0000","last_clean":"2024-03-28T23:36:08.187228+0000","last_became_active":"2024-03-28T23:35:59.074170+0000","last_became_peered":"2024-03-28T23:35:59.074170+0000","last_unstale":"2024-03-28T23:36:08.187228+0000","last_undegraded":"2024-03-28T23:36:08.187228+0000","last_fullsized":"2024-03-28T23:36:08.187228+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:16.910736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"151'2","reported_seq":24,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120382+0000","last_change":"2024-03-28T23:35:58.070169+0000","last_active":"2024-03-28T23:36:09.120382+0000","last_peered":"2024-03-28T23:36:09.120382+0000","last_clean":"2024-03-28T23:36:09.120382+0000","last_became_active":"2024-03-28T23:35:58.069913+0000","last_became_peered":"2024-03-28T23:35:58.069913+0000","last_unstale":"2024-03-28T23:36:09.120382+0000","last_undegraded":"2024-03-28T23:36:09.120382+0000","last_fullsized":"2024-03-28T23:36:09.120382+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:22:07.300796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.192034+0000","last_change":"2024-03-28T23:36:08.190757+0000","last_active":"2024-03-28T23:36:08.192034+0000","last_peered":"2024-03-28T23:36:08.192034+0000","last_clean":"2024-03-28T23:36:08.192034+0000","last_became_active":"2024-03-28T23:36:08.190491+0000","last_became_peered":"2024-03-28T23:36:08.190491+0000","last_unstale":"2024-03-28T23:36:08.192034+0000","last_undegraded":"2024-03-28T23:36:08.192034+0000","last_fullsized":"2024-03-28T23:36:08.192034+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:32.608667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108894+0000","last_change":"2024-03-28T23:36:07.103900+0000","last_active":"2024-03-28T23:36:09.108894+0000","last_peered":"2024-03-28T23:36:09.108894+0000","last_clean":"2024-03-28T23:36:09.108894+0000","last_became_active":"2024-03-28T23:36:07.103589+0000","last_became_peered":"2024-03-28T23:36:07.103589+0000","last_unstale":"2024-03-28T23:36:09.108894+0000","last_undegraded":"2024-03-28T23:36:09.108894+0000","last_fullsized":"2024-03-28T23:36:09.108894+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:08:38.942276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108337+0000","last_change":"2024-03-28T23:35:59.081763+0000","last_active":"2024-03-28T23:36:09.108337+0000","last_peered":"2024-03-28T23:36:09.108337+0000","last_clean":"2024-03-28T23:36:09.108337+0000","last_became_active":"2024-03-28T23:35:59.081465+0000","last_became_peered":"2024-03-28T23:35:59.081465+0000","last_unstale":"2024-03-28T23:36:09.108337+0000","last_undegraded":"2024-03-28T23:36:09.108337+0000","last_fullsized":"2024-03-28T23:36:09.108337+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:06:37.212712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"151'1","reported_seq":21,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.109458+0000","last_change":"2024-03-28T23:35:58.070308+0000","last_active":"2024-03-28T23:36:08.109458+0000","last_peered":"2024-03-28T23:36:08.109458+0000","last_clean":"2024-03-28T23:36:08.109458+0000","last_became_active":"2024-03-28T23:35:58.070024+0000","last_became_peered":"2024-03-28T23:35:58.070024+0000","last_unstale":"2024-03-28T23:36:08.109458+0000","last_undegraded":"2024-03-28T23:36:08.109458+0000","last_fullsized":"2024-03-28T23:36:08.109458+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:49:07.831860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.191661+0000","last_change":"2024-03-28T23:36:08.190948+0000","last_active":"2024-03-28T23:36:08.191661+0000","last_peered":"2024-03-28T23:36:08.191661+0000","last_clean":"2024-03-28T23:36:08.191661+0000","last_became_active":"2024-03-28T23:36:08.190659+0000","last_became_peered":"2024-03-28T23:36:08.190659+0000","last_unstale":"2024-03-28T23:36:08.191661+0000","last_undegraded":"2024-03-28T23:36:08.191661+0000","last_fullsized":"2024-03-28T23:36:08.191661+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:51:15.427349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108653+0000","last_change":"2024-03-28T23:36:07.110907+0000","last_active":"2024-03-28T23:36:09.108653+0000","last_peered":"2024-03-28T23:36:09.108653+0000","last_clean":"2024-03-28T23:36:09.108653+0000","last_became_active":"2024-03-28T23:36:07.110651+0000","last_became_peered":"2024-03-28T23:36:07.110651+0000","last_unstale":"2024-03-28T23:36:09.108653+0000","last_undegraded":"2024-03-28T23:36:09.108653+0000","last_fullsized":"2024-03-28T23:36:09.108653+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:06:16.270793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108755+0000","last_change":"2024-03-28T23:35:59.075326+0000","last_active":"2024-03-28T23:36:09.108755+0000","last_peered":"2024-03-28T23:36:09.108755+0000","last_clean":"2024-03-28T23:36:09.108755+0000","last_became_active":"2024-03-28T23:35:59.075063+0000","last_became_peered":"2024-03-28T23:35:59.075063+0000","last_unstale":"2024-03-28T23:36:09.108755+0000","last_undegraded":"2024-03-28T23:36:09.108755+0000","last_fullsized":"2024-03-28T23:36:09.108755+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:42:41.721990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120144+0000","last_change":"2024-03-28T23:35:58.073249+0000","last_active":"2024-03-28T23:36:09.120144+0000","last_peered":"2024-03-28T23:36:09.120144+0000","last_clean":"2024-03-28T23:36:09.120144+0000","last_became_active":"2024-03-28T23:35:58.072992+0000","last_became_peered":"2024-03-28T23:35:58.072992+0000","last_unstale":"2024-03-28T23:36:09.120144+0000","last_undegraded":"2024-03-28T23:36:09.120144+0000","last_fullsized":"2024-03-28T23:36:09.120144+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:58:36.456500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109797+0000","last_change":"2024-03-28T23:36:08.190992+0000","last_active":"2024-03-28T23:36:09.109797+0000","last_peered":"2024-03-28T23:36:09.109797+0000","last_clean":"2024-03-28T23:36:09.109797+0000","last_became_active":"2024-03-28T23:36:08.190725+0000","last_became_peered":"2024-03-28T23:36:08.190725+0000","last_unstale":"2024-03-28T23:36:09.109797+0000","last_undegraded":"2024-03-28T23:36:09.109797+0000","last_fullsized":"2024-03-28T23:36:09.109797+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:42:54.376192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109850+0000","last_change":"2024-03-28T23:36:07.110460+0000","last_active":"2024-03-28T23:36:09.109850+0000","last_peered":"2024-03-28T23:36:09.109850+0000","last_clean":"2024-03-28T23:36:09.109850+0000","last_became_active":"2024-03-28T23:36:07.110195+0000","last_became_peered":"2024-03-28T23:36:07.110195+0000","last_unstale":"2024-03-28T23:36:09.109850+0000","last_undegraded":"2024-03-28T23:36:09.109850+0000","last_fullsized":"2024-03-28T23:36:09.109850+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:39:56.430484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109471+0000","last_change":"2024-03-28T23:35:59.082001+0000","last_active":"2024-03-28T23:36:09.109471+0000","last_peered":"2024-03-28T23:36:09.109471+0000","last_clean":"2024-03-28T23:36:09.109471+0000","last_became_active":"2024-03-28T23:35:59.081706+0000","last_became_peered":"2024-03-28T23:35:59.081706+0000","last_unstale":"2024-03-28T23:36:09.109471+0000","last_undegraded":"2024-03-28T23:36:09.109471+0000","last_fullsized":"2024-03-28T23:36:09.109471+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:42:45.337484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"151'2","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.108957+0000","last_change":"2024-03-28T23:35:58.072448+0000","last_active":"2024-03-28T23:36:08.108957+0000","last_peered":"2024-03-28T23:36:08.108957+0000","last_clean":"2024-03-28T23:36:08.108957+0000","last_became_active":"2024-03-28T23:35:58.071494+0000","last_became_peered":"2024-03-28T23:35:58.071494+0000","last_unstale":"2024-03-28T23:36:08.108957+0000","last_undegraded":"2024-03-28T23:36:08.108957+0000","last_fullsized":"2024-03-28T23:36:08.108957+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:54:03.045194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.118018+0000","last_change":"2024-03-28T23:36:08.108038+0000","last_active":"2024-03-28T23:36:09.118018+0000","last_peered":"2024-03-28T23:36:09.118018+0000","last_clean":"2024-03-28T23:36:09.118018+0000","last_became_active":"2024-03-28T23:36:08.107752+0000","last_became_peered":"2024-03-28T23:36:08.107752+0000","last_unstale":"2024-03-28T23:36:09.118018+0000","last_undegraded":"2024-03-28T23:36:09.118018+0000","last_fullsized":"2024-03-28T23:36:09.118018+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:03:37.546835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109569+0000","last_change":"2024-03-28T23:36:07.114878+0000","last_active":"2024-03-28T23:36:09.109569+0000","last_peered":"2024-03-28T23:36:09.109569+0000","last_clean":"2024-03-28T23:36:09.109569+0000","last_became_active":"2024-03-28T23:36:07.114586+0000","last_became_peered":"2024-03-28T23:36:07.114586+0000","last_unstale":"2024-03-28T23:36:09.109569+0000","last_undegraded":"2024-03-28T23:36:09.109569+0000","last_fullsized":"2024-03-28T23:36:09.109569+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:21:50.427323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.109362+0000","last_change":"2024-03-28T23:35:59.080948+0000","last_active":"2024-03-28T23:36:08.109362+0000","last_peered":"2024-03-28T23:36:08.109362+0000","last_clean":"2024-03-28T23:36:08.109362+0000","last_became_active":"2024-03-28T23:35:59.080672+0000","last_became_peered":"2024-03-28T23:35:59.080672+0000","last_unstale":"2024-03-28T23:36:08.109362+0000","last_undegraded":"2024-03-28T23:36:08.109362+0000","last_fullsized":"2024-03-28T23:36:08.109362+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:39:27.912295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108487+0000","last_change":"2024-03-28T23:35:58.071966+0000","last_active":"2024-03-28T23:36:09.108487+0000","last_peered":"2024-03-28T23:36:09.108487+0000","last_clean":"2024-03-28T23:36:09.108487+0000","last_became_active":"2024-03-28T23:35:58.071656+0000","last_became_peered":"2024-03-28T23:35:58.071656+0000","last_unstale":"2024-03-28T23:36:09.108487+0000","last_undegraded":"2024-03-28T23:36:09.108487+0000","last_fullsized":"2024-03-28T23:36:09.108487+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:53:34.853174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117583+0000","last_change":"2024-03-28T23:36:08.108836+0000","last_active":"2024-03-28T23:36:09.117583+0000","last_peered":"2024-03-28T23:36:09.117583+0000","last_clean":"2024-03-28T23:36:09.117583+0000","last_became_active":"2024-03-28T23:36:08.108577+0000","last_became_peered":"2024-03-28T23:36:08.108577+0000","last_unstale":"2024-03-28T23:36:09.117583+0000","last_undegraded":"2024-03-28T23:36:09.117583+0000","last_fullsized":"2024-03-28T23:36:09.117583+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:32.608667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.127528+0000","last_change":"2024-03-28T23:36:07.103362+0000","last_active":"2024-03-28T23:36:08.127528+0000","last_peered":"2024-03-28T23:36:08.127528+0000","last_clean":"2024-03-28T23:36:08.127528+0000","last_became_active":"2024-03-28T23:36:07.103086+0000","last_became_peered":"2024-03-28T23:36:07.103086+0000","last_unstale":"2024-03-28T23:36:08.127528+0000","last_undegraded":"2024-03-28T23:36:08.127528+0000","last_fullsized":"2024-03-28T23:36:08.127528+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:13:32.250870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.109402+0000","last_change":"2024-03-28T23:35:59.074451+0000","last_active":"2024-03-28T23:36:08.109402+0000","last_peered":"2024-03-28T23:36:08.109402+0000","last_clean":"2024-03-28T23:36:08.109402+0000","last_became_active":"2024-03-28T23:35:59.074181+0000","last_became_peered":"2024-03-28T23:35:59.074181+0000","last_unstale":"2024-03-28T23:36:08.109402+0000","last_undegraded":"2024-03-28T23:36:08.109402+0000","last_fullsized":"2024-03-28T23:36:08.109402+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:54:41.513429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117696+0000","last_change":"2024-03-28T23:35:58.067234+0000","last_active":"2024-03-28T23:36:09.117696+0000","last_peered":"2024-03-28T23:36:09.117696+0000","last_clean":"2024-03-28T23:36:09.117696+0000","last_became_active":"2024-03-28T23:35:58.066942+0000","last_became_peered":"2024-03-28T23:35:58.066942+0000","last_unstale":"2024-03-28T23:36:09.117696+0000","last_undegraded":"2024-03-28T23:36:09.117696+0000","last_fullsized":"2024-03-28T23:36:09.117696+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:37:03.241869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120158+0000","last_change":"2024-03-28T23:36:08.121754+0000","last_active":"2024-03-28T23:36:09.120158+0000","last_peered":"2024-03-28T23:36:09.120158+0000","last_clean":"2024-03-28T23:36:09.120158+0000","last_became_active":"2024-03-28T23:36:08.121455+0000","last_became_peered":"2024-03-28T23:36:08.121455+0000","last_unstale":"2024-03-28T23:36:09.120158+0000","last_undegraded":"2024-03-28T23:36:09.120158+0000","last_fullsized":"2024-03-28T23:36:09.120158+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:39:47.021555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"156'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119016+0000","last_change":"2024-03-28T23:36:07.112395+0000","last_active":"2024-03-28T23:36:09.119016+0000","last_peered":"2024-03-28T23:36:09.119016+0000","last_clean":"2024-03-28T23:36:09.119016+0000","last_became_active":"2024-03-28T23:36:07.112088+0000","last_became_peered":"2024-03-28T23:36:07.112088+0000","last_unstale":"2024-03-28T23:36:09.119016+0000","last_undegraded":"2024-03-28T23:36:09.119016+0000","last_fullsized":"2024-03-28T23:36:09.119016+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:37.434300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109946+0000","last_change":"2024-03-28T23:35:59.084751+0000","last_active":"2024-03-28T23:36:09.109946+0000","last_peered":"2024-03-28T23:36:09.109946+0000","last_clean":"2024-03-28T23:36:09.109946+0000","last_became_active":"2024-03-28T23:35:59.084472+0000","last_became_peered":"2024-03-28T23:35:59.084472+0000","last_unstale":"2024-03-28T23:36:09.109946+0000","last_undegraded":"2024-03-28T23:36:09.109946+0000","last_fullsized":"2024-03-28T23:36:09.109946+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:17.228750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109252+0000","last_change":"2024-03-28T23:35:58.067542+0000","last_active":"2024-03-28T23:36:09.109252+0000","last_peered":"2024-03-28T23:36:09.109252+0000","last_clean":"2024-03-28T23:36:09.109252+0000","last_became_active":"2024-03-28T23:35:58.067013+0000","last_became_peered":"2024-03-28T23:35:58.067013+0000","last_unstale":"2024-03-28T23:36:09.109252+0000","last_undegraded":"2024-03-28T23:36:09.109252+0000","last_fullsized":"2024-03-28T23:36:09.109252+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:41:31.356630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.119756+0000","last_change":"2024-03-28T23:36:08.116214+0000","last_active":"2024-03-28T23:36:08.119756+0000","last_peered":"2024-03-28T23:36:08.119756+0000","last_clean":"2024-03-28T23:36:08.119756+0000","last_became_active":"2024-03-28T23:36:08.115064+0000","last_became_peered":"2024-03-28T23:36:08.115064+0000","last_unstale":"2024-03-28T23:36:08.119756+0000","last_undegraded":"2024-03-28T23:36:08.119756+0000","last_fullsized":"2024-03-28T23:36:08.119756+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:09:57.210214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117327+0000","last_change":"2024-03-28T23:36:07.109162+0000","last_active":"2024-03-28T23:36:09.117327+0000","last_peered":"2024-03-28T23:36:09.117327+0000","last_clean":"2024-03-28T23:36:09.117327+0000","last_became_active":"2024-03-28T23:36:07.108846+0000","last_became_peered":"2024-03-28T23:36:07.108846+0000","last_unstale":"2024-03-28T23:36:09.117327+0000","last_undegraded":"2024-03-28T23:36:09.117327+0000","last_fullsized":"2024-03-28T23:36:09.117327+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:24.945602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108490+0000","last_change":"2024-03-28T23:35:59.077874+0000","last_active":"2024-03-28T23:36:09.108490+0000","last_peered":"2024-03-28T23:36:09.108490+0000","last_clean":"2024-03-28T23:36:09.108490+0000","last_became_active":"2024-03-28T23:35:59.077613+0000","last_became_peered":"2024-03-28T23:35:59.077613+0000","last_unstale":"2024-03-28T23:36:09.108490+0000","last_undegraded":"2024-03-28T23:36:09.108490+0000","last_fullsized":"2024-03-28T23:36:09.108490+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:41:12.571144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"151'2","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.110233+0000","last_change":"2024-03-28T23:35:58.066436+0000","last_active":"2024-03-28T23:36:08.110233+0000","last_peered":"2024-03-28T23:36:08.110233+0000","last_clean":"2024-03-28T23:36:08.110233+0000","last_became_active":"2024-03-28T23:35:58.063834+0000","last_became_peered":"2024-03-28T23:35:58.063834+0000","last_unstale":"2024-03-28T23:36:08.110233+0000","last_undegraded":"2024-03-28T23:36:08.110233+0000","last_fullsized":"2024-03-28T23:36:08.110233+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:37:09.547486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109600+0000","last_change":"2024-03-28T23:36:08.122302+0000","last_active":"2024-03-28T23:36:09.109600+0000","last_peered":"2024-03-28T23:36:09.109600+0000","last_clean":"2024-03-28T23:36:09.109600+0000","last_became_active":"2024-03-28T23:36:08.122059+0000","last_became_peered":"2024-03-28T23:36:08.122059+0000","last_unstale":"2024-03-28T23:36:09.109600+0000","last_undegraded":"2024-03-28T23:36:09.109600+0000","last_fullsized":"2024-03-28T23:36:09.109600+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:24:51.260030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.188657+0000","last_change":"2024-03-28T23:36:07.099612+0000","last_active":"2024-03-28T23:36:08.188657+0000","last_peered":"2024-03-28T23:36:08.188657+0000","last_clean":"2024-03-28T23:36:08.188657+0000","last_became_active":"2024-03-28T23:36:07.099349+0000","last_became_peered":"2024-03-28T23:36:07.099349+0000","last_unstale":"2024-03-28T23:36:08.188657+0000","last_undegraded":"2024-03-28T23:36:08.188657+0000","last_fullsized":"2024-03-28T23:36:08.188657+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:13:34.084552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119706+0000","last_change":"2024-03-28T23:35:59.074166+0000","last_active":"2024-03-28T23:36:09.119706+0000","last_peered":"2024-03-28T23:36:09.119706+0000","last_clean":"2024-03-28T23:36:09.119706+0000","last_became_active":"2024-03-28T23:35:59.073916+0000","last_became_peered":"2024-03-28T23:35:59.073916+0000","last_unstale":"2024-03-28T23:36:09.119706+0000","last_undegraded":"2024-03-28T23:36:09.119706+0000","last_fullsized":"2024-03-28T23:36:09.119706+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:24.837863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"152'1","reported_seq":23,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.110078+0000","last_change":"2024-03-28T23:35:58.078010+0000","last_active":"2024-03-28T23:36:09.110078+0000","last_peered":"2024-03-28T23:36:09.110078+0000","last_clean":"2024-03-28T23:36:09.110078+0000","last_became_active":"2024-03-28T23:35:58.077736+0000","last_became_peered":"2024-03-28T23:35:58.077736+0000","last_unstale":"2024-03-28T23:36:09.110078+0000","last_undegraded":"2024-03-28T23:36:09.110078+0000","last_fullsized":"2024-03-28T23:36:09.110078+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:54:03.045194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119474+0000","last_change":"2024-03-28T23:36:08.120046+0000","last_active":"2024-03-28T23:36:09.119474+0000","last_peered":"2024-03-28T23:36:09.119474+0000","last_clean":"2024-03-28T23:36:09.119474+0000","last_became_active":"2024-03-28T23:36:08.119656+0000","last_became_peered":"2024-03-28T23:36:08.119656+0000","last_unstale":"2024-03-28T23:36:09.119474+0000","last_undegraded":"2024-03-28T23:36:09.119474+0000","last_fullsized":"2024-03-28T23:36:09.119474+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:21:23.239166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.126994+0000","last_change":"2024-03-28T23:36:07.111369+0000","last_active":"2024-03-28T23:36:08.126994+0000","last_peered":"2024-03-28T23:36:08.126994+0000","last_clean":"2024-03-28T23:36:08.126994+0000","last_became_active":"2024-03-28T23:36:07.111103+0000","last_became_peered":"2024-03-28T23:36:07.111103+0000","last_unstale":"2024-03-28T23:36:08.126994+0000","last_undegraded":"2024-03-28T23:36:08.126994+0000","last_fullsized":"2024-03-28T23:36:08.126994+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:50:17.420909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117513+0000","last_change":"2024-03-28T23:35:59.062460+0000","last_active":"2024-03-28T23:36:09.117513+0000","last_peered":"2024-03-28T23:36:09.117513+0000","last_clean":"2024-03-28T23:36:09.117513+0000","last_became_active":"2024-03-28T23:35:59.062185+0000","last_became_peered":"2024-03-28T23:35:59.062185+0000","last_unstale":"2024-03-28T23:36:09.117513+0000","last_undegraded":"2024-03-28T23:36:09.117513+0000","last_fullsized":"2024-03-28T23:36:09.117513+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:51.224827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109998+0000","last_change":"2024-03-28T23:35:58.072356+0000","last_active":"2024-03-28T23:36:09.109998+0000","last_peered":"2024-03-28T23:36:09.109998+0000","last_clean":"2024-03-28T23:36:09.109998+0000","last_became_active":"2024-03-28T23:35:58.072111+0000","last_became_peered":"2024-03-28T23:35:58.072111+0000","last_unstale":"2024-03-28T23:36:09.109998+0000","last_undegraded":"2024-03-28T23:36:09.109998+0000","last_fullsized":"2024-03-28T23:36:09.109998+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:23.832628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.119081+0000","last_change":"2024-03-28T23:36:08.115504+0000","last_active":"2024-03-28T23:36:08.119081+0000","last_peered":"2024-03-28T23:36:08.119081+0000","last_clean":"2024-03-28T23:36:08.119081+0000","last_became_active":"2024-03-28T23:36:08.115249+0000","last_became_peered":"2024-03-28T23:36:08.115249+0000","last_unstale":"2024-03-28T23:36:08.119081+0000","last_undegraded":"2024-03-28T23:36:08.119081+0000","last_fullsized":"2024-03-28T23:36:08.119081+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:24:21.264344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.110229+0000","last_change":"2024-03-28T23:36:07.105646+0000","last_active":"2024-03-28T23:36:08.110229+0000","last_peered":"2024-03-28T23:36:08.110229+0000","last_clean":"2024-03-28T23:36:08.110229+0000","last_became_active":"2024-03-28T23:36:07.105391+0000","last_became_peered":"2024-03-28T23:36:07.105391+0000","last_unstale":"2024-03-28T23:36:08.110229+0000","last_undegraded":"2024-03-28T23:36:08.110229+0000","last_fullsized":"2024-03-28T23:36:08.110229+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:39:56.430484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119805+0000","last_change":"2024-03-28T23:35:59.079950+0000","last_active":"2024-03-28T23:36:09.119805+0000","last_peered":"2024-03-28T23:36:09.119805+0000","last_clean":"2024-03-28T23:36:09.119805+0000","last_became_active":"2024-03-28T23:35:59.079675+0000","last_became_peered":"2024-03-28T23:35:59.079675+0000","last_unstale":"2024-03-28T23:36:09.119805+0000","last_undegraded":"2024-03-28T23:36:09.119805+0000","last_fullsized":"2024-03-28T23:36:09.119805+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:36:26.239288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117843+0000","last_change":"2024-03-28T23:35:58.068737+0000","last_active":"2024-03-28T23:36:09.117843+0000","last_peered":"2024-03-28T23:36:09.117843+0000","last_clean":"2024-03-28T23:36:09.117843+0000","last_became_active":"2024-03-28T23:35:58.068481+0000","last_became_peered":"2024-03-28T23:35:58.068481+0000","last_unstale":"2024-03-28T23:36:09.117843+0000","last_undegraded":"2024-03-28T23:36:09.117843+0000","last_fullsized":"2024-03-28T23:36:09.117843+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:05:05.111519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109331+0000","last_change":"2024-03-28T23:36:08.117859+0000","last_active":"2024-03-28T23:36:09.109331+0000","last_peered":"2024-03-28T23:36:09.109331+0000","last_clean":"2024-03-28T23:36:09.109331+0000","last_became_active":"2024-03-28T23:36:08.117178+0000","last_became_peered":"2024-03-28T23:36:08.117178+0000","last_unstale":"2024-03-28T23:36:09.109331+0000","last_undegraded":"2024-03-28T23:36:09.109331+0000","last_fullsized":"2024-03-28T23:36:09.109331+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:30:42.818702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"155'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.191075+0000","last_change":"2024-03-28T23:36:07.115327+0000","last_active":"2024-03-28T23:36:08.191075+0000","last_peered":"2024-03-28T23:36:08.191075+0000","last_clean":"2024-03-28T23:36:08.191075+0000","last_became_active":"2024-03-28T23:36:07.115138+0000","last_became_peered":"2024-03-28T23:36:07.115138+0000","last_unstale":"2024-03-28T23:36:08.191075+0000","last_undegraded":"2024-03-28T23:36:08.191075+0000","last_fullsized":"2024-03-28T23:36:08.191075+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:02:47.373151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.106013+0000","last_change":"2024-03-28T23:35:59.077779+0000","last_active":"2024-03-28T23:36:08.106013+0000","last_peered":"2024-03-28T23:36:08.106013+0000","last_clean":"2024-03-28T23:36:08.106013+0000","last_became_active":"2024-03-28T23:35:59.077529+0000","last_became_peered":"2024-03-28T23:35:59.077529+0000","last_unstale":"2024-03-28T23:36:08.106013+0000","last_undegraded":"2024-03-28T23:36:08.106013+0000","last_fullsized":"2024-03-28T23:36:08.106013+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:23:42.046480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.104848+0000","last_change":"2024-03-28T23:35:58.072655+0000","last_active":"2024-03-28T23:36:08.104848+0000","last_peered":"2024-03-28T23:36:08.104848+0000","last_clean":"2024-03-28T23:36:08.104848+0000","last_became_active":"2024-03-28T23:35:58.072379+0000","last_became_peered":"2024-03-28T23:35:58.072379+0000","last_unstale":"2024-03-28T23:36:08.104848+0000","last_undegraded":"2024-03-28T23:36:08.104848+0000","last_fullsized":"2024-03-28T23:36:08.104848+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:57:05.274295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120498+0000","last_change":"2024-03-28T23:36:08.125170+0000","last_active":"2024-03-28T23:36:09.120498+0000","last_peered":"2024-03-28T23:36:09.120498+0000","last_clean":"2024-03-28T23:36:09.120498+0000","last_became_active":"2024-03-28T23:36:08.124892+0000","last_became_peered":"2024-03-28T23:36:08.124892+0000","last_unstale":"2024-03-28T23:36:09.120498+0000","last_undegraded":"2024-03-28T23:36:09.120498+0000","last_fullsized":"2024-03-28T23:36:09.120498+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:36.142805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109742+0000","last_change":"2024-03-28T23:36:07.102970+0000","last_active":"2024-03-28T23:36:09.109742+0000","last_peered":"2024-03-28T23:36:09.109742+0000","last_clean":"2024-03-28T23:36:09.109742+0000","last_became_active":"2024-03-28T23:36:07.102708+0000","last_became_peered":"2024-03-28T23:36:07.102708+0000","last_unstale":"2024-03-28T23:36:09.109742+0000","last_undegraded":"2024-03-28T23:36:09.109742+0000","last_fullsized":"2024-03-28T23:36:09.109742+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:36:34.274154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.110032+0000","last_change":"2024-03-28T23:35:59.080115+0000","last_active":"2024-03-28T23:36:09.110032+0000","last_peered":"2024-03-28T23:36:09.110032+0000","last_clean":"2024-03-28T23:36:09.110032+0000","last_became_active":"2024-03-28T23:35:59.079809+0000","last_became_peered":"2024-03-28T23:35:59.079809+0000","last_unstale":"2024-03-28T23:36:09.110032+0000","last_undegraded":"2024-03-28T23:36:09.110032+0000","last_fullsized":"2024-03-28T23:36:09.110032+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:49:08.837095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109260+0000","last_change":"2024-03-28T23:35:58.071453+0000","last_active":"2024-03-28T23:36:09.109260+0000","last_peered":"2024-03-28T23:36:09.109260+0000","last_clean":"2024-03-28T23:36:09.109260+0000","last_became_active":"2024-03-28T23:35:58.071072+0000","last_became_peered":"2024-03-28T23:35:58.071072+0000","last_unstale":"2024-03-28T23:36:09.109260+0000","last_undegraded":"2024-03-28T23:36:09.109260+0000","last_fullsized":"2024-03-28T23:36:09.109260+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:20:20.872070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117786+0000","last_change":"2024-03-28T23:36:08.127731+0000","last_active":"2024-03-28T23:36:09.117786+0000","last_peered":"2024-03-28T23:36:09.117786+0000","last_clean":"2024-03-28T23:36:09.117786+0000","last_became_active":"2024-03-28T23:36:08.127458+0000","last_became_peered":"2024-03-28T23:36:08.127458+0000","last_unstale":"2024-03-28T23:36:09.117786+0000","last_undegraded":"2024-03-28T23:36:09.117786+0000","last_fullsized":"2024-03-28T23:36:09.117786+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:27:08.495798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109055+0000","last_change":"2024-03-28T23:36:07.108521+0000","last_active":"2024-03-28T23:36:09.109055+0000","last_peered":"2024-03-28T23:36:09.109055+0000","last_clean":"2024-03-28T23:36:09.109055+0000","last_became_active":"2024-03-28T23:36:07.108225+0000","last_became_peered":"2024-03-28T23:36:07.108225+0000","last_unstale":"2024-03-28T23:36:09.109055+0000","last_undegraded":"2024-03-28T23:36:09.109055+0000","last_fullsized":"2024-03-28T23:36:09.109055+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:32.872729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.110002+0000","last_change":"2024-03-28T23:35:59.081608+0000","last_active":"2024-03-28T23:36:09.110002+0000","last_peered":"2024-03-28T23:36:09.110002+0000","last_clean":"2024-03-28T23:36:09.110002+0000","last_became_active":"2024-03-28T23:35:59.080839+0000","last_became_peered":"2024-03-28T23:35:59.080839+0000","last_unstale":"2024-03-28T23:36:09.110002+0000","last_undegraded":"2024-03-28T23:36:09.110002+0000","last_fullsized":"2024-03-28T23:36:09.110002+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:46:11.587017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108567+0000","last_change":"2024-03-28T23:35:58.067740+0000","last_active":"2024-03-28T23:36:09.108567+0000","last_peered":"2024-03-28T23:36:09.108567+0000","last_clean":"2024-03-28T23:36:09.108567+0000","last_became_active":"2024-03-28T23:35:58.067418+0000","last_became_peered":"2024-03-28T23:35:58.067418+0000","last_unstale":"2024-03-28T23:36:09.108567+0000","last_undegraded":"2024-03-28T23:36:09.108567+0000","last_fullsized":"2024-03-28T23:36:09.108567+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:00:58.303444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109755+0000","last_change":"2024-03-28T23:36:08.117608+0000","last_active":"2024-03-28T23:36:09.109755+0000","last_peered":"2024-03-28T23:36:09.109755+0000","last_clean":"2024-03-28T23:36:09.109755+0000","last_became_active":"2024-03-28T23:36:08.116979+0000","last_became_peered":"2024-03-28T23:36:08.116979+0000","last_unstale":"2024-03-28T23:36:09.109755+0000","last_undegraded":"2024-03-28T23:36:09.109755+0000","last_fullsized":"2024-03-28T23:36:09.109755+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:36.635820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.128691+0000","last_change":"2024-03-28T23:36:07.107556+0000","last_active":"2024-03-28T23:36:08.128691+0000","last_peered":"2024-03-28T23:36:08.128691+0000","last_clean":"2024-03-28T23:36:08.128691+0000","last_became_active":"2024-03-28T23:36:07.107318+0000","last_became_peered":"2024-03-28T23:36:07.107318+0000","last_unstale":"2024-03-28T23:36:08.128691+0000","last_undegraded":"2024-03-28T23:36:08.128691+0000","last_fullsized":"2024-03-28T23:36:08.128691+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:21:50.427323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.118086+0000","last_change":"2024-03-28T23:35:59.079990+0000","last_active":"2024-03-28T23:36:09.118086+0000","last_peered":"2024-03-28T23:36:09.118086+0000","last_clean":"2024-03-28T23:36:09.118086+0000","last_became_active":"2024-03-28T23:35:59.079717+0000","last_became_peered":"2024-03-28T23:35:59.079717+0000","last_unstale":"2024-03-28T23:36:09.118086+0000","last_undegraded":"2024-03-28T23:36:09.118086+0000","last_fullsized":"2024-03-28T23:36:09.118086+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:21.402880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.106949+0000","last_change":"2024-03-28T23:35:58.071103+0000","last_active":"2024-03-28T23:36:08.106949+0000","last_peered":"2024-03-28T23:36:08.106949+0000","last_clean":"2024-03-28T23:36:08.106949+0000","last_became_active":"2024-03-28T23:35:58.070587+0000","last_became_peered":"2024-03-28T23:35:58.070587+0000","last_unstale":"2024-03-28T23:36:08.106949+0000","last_undegraded":"2024-03-28T23:36:08.106949+0000","last_fullsized":"2024-03-28T23:36:08.106949+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:16.223515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.190953+0000","last_change":"2024-03-28T23:36:08.190060+0000","last_active":"2024-03-28T23:36:08.190953+0000","last_peered":"2024-03-28T23:36:08.190953+0000","last_clean":"2024-03-28T23:36:08.190953+0000","last_became_active":"2024-03-28T23:36:08.189108+0000","last_became_peered":"2024-03-28T23:36:08.189108+0000","last_unstale":"2024-03-28T23:36:08.190953+0000","last_undegraded":"2024-03-28T23:36:08.190953+0000","last_fullsized":"2024-03-28T23:36:08.190953+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:57:15.318239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.107131+0000","last_change":"2024-03-28T23:36:07.099839+0000","last_active":"2024-03-28T23:36:08.107131+0000","last_peered":"2024-03-28T23:36:08.107131+0000","last_clean":"2024-03-28T23:36:08.107131+0000","last_became_active":"2024-03-28T23:36:07.099575+0000","last_became_peered":"2024-03-28T23:36:07.099575+0000","last_unstale":"2024-03-28T23:36:08.107131+0000","last_undegraded":"2024-03-28T23:36:08.107131+0000","last_fullsized":"2024-03-28T23:36:08.107131+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:12:19.745437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109162+0000","last_change":"2024-03-28T23:35:59.075847+0000","last_active":"2024-03-28T23:36:09.109162+0000","last_peered":"2024-03-28T23:36:09.109162+0000","last_clean":"2024-03-28T23:36:09.109162+0000","last_became_active":"2024-03-28T23:35:59.072889+0000","last_became_peered":"2024-03-28T23:35:59.072889+0000","last_unstale":"2024-03-28T23:36:09.109162+0000","last_undegraded":"2024-03-28T23:36:09.109162+0000","last_fullsized":"2024-03-28T23:36:09.109162+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:35.927782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.110001+0000","last_change":"2024-03-28T23:35:58.078046+0000","last_active":"2024-03-28T23:36:09.110001+0000","last_peered":"2024-03-28T23:36:09.110001+0000","last_clean":"2024-03-28T23:36:09.110001+0000","last_became_active":"2024-03-28T23:35:58.077780+0000","last_became_peered":"2024-03-28T23:35:58.077780+0000","last_unstale":"2024-03-28T23:36:09.110001+0000","last_undegraded":"2024-03-28T23:36:09.110001+0000","last_fullsized":"2024-03-28T23:36:09.110001+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:37:09.547486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":425,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109339+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:36:09.109339+0000","last_peered":"2024-03-28T23:36:09.109339+0000","last_clean":"2024-03-28T23:36:09.109339+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:36:09.109339+0000","last_undegraded":"2024-03-28T23:36:09.109339+0000","last_fullsized":"2024-03-28T23:36:09.109339+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574928,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33148,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338692,"statfs":{"total":96636764160,"available":96602820608,"internally_reserved":0,"allocated":2015232,"data_stored":1538095,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574929,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33096,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338744,"statfs":{"total":96636764160,"available":96602873856,"internally_reserved":0,"allocated":1953792,"data_stored":947797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574929,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33116,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338724,"statfs":{"total":96636764160,"available":96602853376,"internally_reserved":0,"allocated":1974272,"data_stored":949485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574928,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33124,"kb_used_data":1936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338716,"statfs":{"total":96636764160,"available":96602845184,"internally_reserved":0,"allocated":1982464,"data_stored":950137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574928,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33124,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338716,"statfs":{"total":96636764160,"available":96602845184,"internally_reserved":0,"allocated":1990656,"data_stored":957652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574928,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33096,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338744,"statfs":{"total":96636764160,"available":96602873856,"internally_reserved":0,"allocated":1953792,"data_stored":948309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33104,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338736,"statfs":{"total":96636764160,"available":96602865664,"internally_reserved":0,"allocated":1961984,"data_stored":949339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574928,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33172,"kb_used_data":1992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338668,"statfs":{"total":96636764160,"available":96602796032,"internally_reserved":0,"allocated":2039808,"data_stored":1538927,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:36:12.231 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:36:12.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.364+0000 7f693e674640 1 Processor -- start 2024-03-28T23:36:12.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.365+0000 7f693e674640 1 -- start start 2024-03-28T23:36:12.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.365+0000 7f693e674640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6938152890 0x7f69380c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:12.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.365+0000 7f693e674640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69380c8ed0 0x7f69380c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:12.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.365+0000 7f693e674640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f69380c5950 0x7f69381553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:12.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.365+0000 7f693e674640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f69380c8480 con 0x7f69380c5950 2024-03-28T23:36:12.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.365+0000 7f693e674640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f6938159640 con 0x7f6938152890 2024-03-28T23:36:12.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.365+0000 7f693e674640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6938158a30 con 0x7f69380c8ed0 2024-03-28T23:36:12.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.365+0000 7f693cc11640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f69380c5950 0x7f69381553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:12.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.365+0000 7f693cc11640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f69380c5950 0x7f69381553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46624/0 (socket says 172.21.15.161:46624) 2024-03-28T23:36:12.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.365+0000 7f693cc11640 1 -- 172.21.15.161:0/1984251967 learned_addr learned my addr 172.21.15.161:0/1984251967 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:12.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.365+0000 7f6937fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6938152890 0x7f69380c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:12.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.365+0000 7f69377fe640 1 --2- 172.21.15.161:0/1984251967 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69380c8ed0 0x7f69380c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:12.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.366+0000 7f693cc11640 1 -- 172.21.15.161:0/1984251967 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69380c8ed0 msgr2=0x7f69380c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:12.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.366+0000 7f693cc11640 1 --2- 172.21.15.161:0/1984251967 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69380c8ed0 0x7f69380c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.366+0000 7f693cc11640 1 -- 172.21.15.161:0/1984251967 >> v2:172.21.15.170:3300/0 conn(0x7f6938152890 msgr2=0x7f69380c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:12.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.366+0000 7f693cc11640 1 --2- 172.21.15.161:0/1984251967 >> v2:172.21.15.170:3300/0 conn(0x7f6938152890 0x7f69380c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.366+0000 7f693cc11640 1 -- 172.21.15.161:0/1984251967 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6938158d10 con 0x7f69380c5950 2024-03-28T23:36:12.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.366+0000 7f6937fff640 1 --2- 172.21.15.161:0/1984251967 >> v2:172.21.15.170:3300/0 conn(0x7f6938152890 0x7f69380c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:12.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.366+0000 7f693cc11640 1 --2- 172.21.15.161:0/1984251967 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69380c5950 0x7f69381553a0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f693808cc80 tx=0x7f693002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7997ca8d5288ba5f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:12.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.367+0000 7f6936ffd640 1 -- 172.21.15.161:0/1984251967 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693002c950 con 0x7f69380c5950 2024-03-28T23:36:12.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.367+0000 7f6936ffd640 1 -- 172.21.15.161:0/1984251967 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f693002caf0 con 0x7f69380c5950 2024-03-28T23:36:12.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.367+0000 7f6936ffd640 1 -- 172.21.15.161:0/1984251967 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69300343f0 con 0x7f69380c5950 2024-03-28T23:36:12.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.367+0000 7f693e674640 1 -- 172.21.15.161:0/1984251967 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69380c5950 msgr2=0x7f69381553a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:12.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.367+0000 7f693e674640 1 --2- 172.21.15.161:0/1984251967 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69380c5950 0x7f69381553a0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f693808cc80 tx=0x7f693002bdb0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.367+0000 7f693e674640 1 -- 172.21.15.161:0/1984251967 shutdown_connections 2024-03-28T23:36:12.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.367+0000 7f693e674640 1 --2- 172.21.15.161:0/1984251967 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69380c5950 0x7f69381553a0 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.367+0000 7f693e674640 1 --2- 172.21.15.161:0/1984251967 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69380c8ed0 0x7f69380c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.367+0000 7f693e674640 1 --2- 172.21.15.161:0/1984251967 >> v2:172.21.15.170:3300/0 conn(0x7f6938152890 0x7f69380c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.367+0000 7f693e674640 1 -- 172.21.15.161:0/1984251967 >> 172.21.15.161:0/1984251967 conn(0x7f69380ac620 msgr2=0x7f69380aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:12.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.367+0000 7f693e674640 1 -- 172.21.15.161:0/1984251967 shutdown_connections 2024-03-28T23:36:12.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.367+0000 7f693e674640 1 -- 172.21.15.161:0/1984251967 wait complete. 2024-03-28T23:36:12.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.368+0000 7f693e674640 1 Processor -- start 2024-03-28T23:36:12.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.369+0000 7f693e674640 1 -- start start 2024-03-28T23:36:12.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.369+0000 7f693e674640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69380c5950 0x7f6938157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:12.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.369+0000 7f693e674640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69380c8ed0 0x7f69380afd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:12.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.369+0000 7f693e674640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6938152890 0x7f69380ae290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:12.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.369+0000 7f693e674640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6938152c70 con 0x7f6938152890 2024-03-28T23:36:12.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.369+0000 7f693e674640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f69381592d0 con 0x7f69380c8ed0 2024-03-28T23:36:12.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.369+0000 7f693e674640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6938152df0 con 0x7f69380c5950 2024-03-28T23:36:12.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.369+0000 7f693cc11640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6938152890 0x7f69380ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:12.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.369+0000 7f693cc11640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6938152890 0x7f69380ae290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46630/0 (socket says 172.21.15.161:46630) 2024-03-28T23:36:12.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.369+0000 7f693cc11640 1 -- 172.21.15.161:0/1595287030 learned_addr learned my addr 172.21.15.161:0/1595287030 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:12.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.369+0000 7f69377fe640 1 --2- 172.21.15.161:0/1595287030 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69380c8ed0 0x7f69380afd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:12.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.369+0000 7f6937fff640 1 --2- 172.21.15.161:0/1595287030 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69380c5950 0x7f6938157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:12.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.370+0000 7f693cc11640 1 -- 172.21.15.161:0/1595287030 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69380c5950 msgr2=0x7f6938157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:12.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.370+0000 7f693cc11640 1 --2- 172.21.15.161:0/1595287030 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69380c5950 0x7f6938157ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.370+0000 7f693cc11640 1 -- 172.21.15.161:0/1595287030 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69380c8ed0 msgr2=0x7f69380afd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:12.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.370+0000 7f693cc11640 1 --2- 172.21.15.161:0/1595287030 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69380c8ed0 0x7f69380afd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.370+0000 7f693cc11640 1 -- 172.21.15.161:0/1595287030 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69380c1ff0 con 0x7f6938152890 2024-03-28T23:36:12.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.370+0000 7f693cc11640 1 --2- 172.21.15.161:0/1595287030 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6938152890 0x7f69380ae290 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f6930036040 tx=0x7f6930004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:12.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.370+0000 7f6934ff9640 1 -- 172.21.15.161:0/1595287030 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6930043020 con 0x7f6938152890 2024-03-28T23:36:12.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.370+0000 7f6934ff9640 1 -- 172.21.15.161:0/1595287030 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f693003e450 con 0x7f6938152890 2024-03-28T23:36:12.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.370+0000 7f6934ff9640 1 -- 172.21.15.161:0/1595287030 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6930009510 con 0x7f6938152890 2024-03-28T23:36:12.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.370+0000 7f693e674640 1 -- 172.21.15.161:0/1595287030 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69381eb520 con 0x7f6938152890 2024-03-28T23:36:12.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.370+0000 7f693e674640 1 -- 172.21.15.161:0/1595287030 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69381ebae0 con 0x7f6938152890 2024-03-28T23:36:12.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.371+0000 7f693e674640 1 -- 172.21.15.161:0/1595287030 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69381ebce0 con 0x7f6938152890 2024-03-28T23:36:12.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.372+0000 7f6934ff9640 1 -- 172.21.15.161:0/1595287030 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6930046020 con 0x7f6938152890 2024-03-28T23:36:12.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.372+0000 7f6934ff9640 1 --2- 172.21.15.161:0/1595287030 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f692406b4d0 0x7f692406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:12.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.372+0000 7f6934ff9640 1 -- 172.21.15.161:0/1595287030 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 6957+0+0 (secure 0 0 0) 0x7f69300acb70 con 0x7f6938152890 2024-03-28T23:36:12.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.372+0000 7f6937fff640 1 --2- 172.21.15.161:0/1595287030 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f692406b4d0 0x7f692406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:12.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.372+0000 7f6937fff640 1 --2- 172.21.15.161:0/1595287030 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f692406b4d0 0x7f692406d990 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6928009560 tx=0x7f6928009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:12.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.380+0000 7f6934ff9640 1 -- 172.21.15.161:0/1595287030 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f69381ebce0 con 0x7f6938152890 2024-03-28T23:36:12.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.555+0000 7f693e674640 1 -- 172.21.15.161:0/1595287030 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f69381eb310 con 0x7f692406b4d0 2024-03-28T23:36:12.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.566+0000 7f6934ff9640 1 -- 172.21.15.161:0/1595287030 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+491517 (secure 0 0 0) 0x7f69381eb310 con 0x7f692406b4d0 2024-03-28T23:36:12.571 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:12.575 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:36:12.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.570+0000 7f693e674640 1 -- 172.21.15.161:0/1595287030 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f692406b4d0 msgr2=0x7f692406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:12.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.570+0000 7f693e674640 1 --2- 172.21.15.161:0/1595287030 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f692406b4d0 0x7f692406d990 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6928009560 tx=0x7f6928009190 comp rx=0 tx=0).stop 2024-03-28T23:36:12.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.570+0000 7f693e674640 1 -- 172.21.15.161:0/1595287030 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6938152890 msgr2=0x7f69380ae290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:12.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.570+0000 7f693e674640 1 --2- 172.21.15.161:0/1595287030 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6938152890 0x7f69380ae290 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f6930036040 tx=0x7f6930004a40 comp rx=0 tx=0).stop 2024-03-28T23:36:12.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.570+0000 7f693e674640 1 -- 172.21.15.161:0/1595287030 shutdown_connections 2024-03-28T23:36:12.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.570+0000 7f693e674640 1 --2- 172.21.15.161:0/1595287030 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f692406b4d0 0x7f692406d990 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.570+0000 7f693e674640 1 --2- 172.21.15.161:0/1595287030 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6938152890 0x7f69380ae290 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.570+0000 7f693e674640 1 --2- 172.21.15.161:0/1595287030 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69380c8ed0 0x7f69380afd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.570+0000 7f693e674640 1 --2- 172.21.15.161:0/1595287030 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69380c5950 0x7f6938157ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.570+0000 7f693e674640 1 -- 172.21.15.161:0/1595287030 >> 172.21.15.161:0/1595287030 conn(0x7f69380ac620 msgr2=0x7f693814a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:12.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.570+0000 7f693e674640 1 -- 172.21.15.161:0/1595287030 shutdown_connections 2024-03-28T23:36:12.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.570+0000 7f693e674640 1 -- 172.21.15.161:0/1595287030 wait complete. 2024-03-28T23:36:12.582 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-03-28T23:36:11.386445+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":290,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264980,"kb_used_data":15500,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754709740,"statfs":{"total":773094113280,"available":772822773760,"internally_reserved":0,"allocated":15872000,"data_stored":8779741,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.299846"},"pg_stats":[{"pgid":"26.3d","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.191136+0000","last_change":"2024-03-28T23:36:08.189106+0000","last_active":"2024-03-28T23:36:08.191136+0000","last_peered":"2024-03-28T23:36:08.191136+0000","last_clean":"2024-03-28T23:36:08.191136+0000","last_became_active":"2024-03-28T23:36:08.188825+0000","last_became_peered":"2024-03-28T23:36:08.188825+0000","last_unstale":"2024-03-28T23:36:08.191136+0000","last_undegraded":"2024-03-28T23:36:08.191136+0000","last_fullsized":"2024-03-28T23:36:08.191136+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:11:58.809402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.3f","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.105776+0000","last_change":"2024-03-28T23:35:59.077622+0000","last_active":"2024-03-28T23:36:08.105776+0000","last_peered":"2024-03-28T23:36:08.105776+0000","last_clean":"2024-03-28T23:36:08.105776+0000","last_became_active":"2024-03-28T23:35:59.077350+0000","last_became_peered":"2024-03-28T23:35:59.077350+0000","last_unstale":"2024-03-28T23:36:08.105776+0000","last_undegraded":"2024-03-28T23:36:08.105776+0000","last_fullsized":"2024-03-28T23:36:08.105776+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:39:54.469103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.120041+0000","last_change":"2024-03-28T23:36:08.115888+0000","last_active":"2024-03-28T23:36:08.120041+0000","last_peered":"2024-03-28T23:36:08.120041+0000","last_clean":"2024-03-28T23:36:08.120041+0000","last_became_active":"2024-03-28T23:36:08.115551+0000","last_became_peered":"2024-03-28T23:36:08.115551+0000","last_unstale":"2024-03-28T23:36:08.120041+0000","last_undegraded":"2024-03-28T23:36:08.120041+0000","last_fullsized":"2024-03-28T23:36:08.120041+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:34:48.075949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109844+0000","last_change":"2024-03-28T23:35:59.069924+0000","last_active":"2024-03-28T23:36:09.109844+0000","last_peered":"2024-03-28T23:36:09.109844+0000","last_clean":"2024-03-28T23:36:09.109844+0000","last_became_active":"2024-03-28T23:35:59.069650+0000","last_became_peered":"2024-03-28T23:35:59.069650+0000","last_unstale":"2024-03-28T23:36:09.109844+0000","last_undegraded":"2024-03-28T23:36:09.109844+0000","last_fullsized":"2024-03-28T23:36:09.109844+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:32.347982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120230+0000","last_change":"2024-03-28T23:36:08.124687+0000","last_active":"2024-03-28T23:36:09.120230+0000","last_peered":"2024-03-28T23:36:09.120230+0000","last_clean":"2024-03-28T23:36:09.120230+0000","last_became_active":"2024-03-28T23:36:08.123746+0000","last_became_peered":"2024-03-28T23:36:08.123746+0000","last_unstale":"2024-03-28T23:36:09.120230+0000","last_undegraded":"2024-03-28T23:36:09.120230+0000","last_fullsized":"2024-03-28T23:36:09.120230+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:34:48.075949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.105213+0000","last_change":"2024-03-28T23:35:59.082632+0000","last_active":"2024-03-28T23:36:08.105213+0000","last_peered":"2024-03-28T23:36:08.105213+0000","last_clean":"2024-03-28T23:36:08.105213+0000","last_became_active":"2024-03-28T23:35:59.082360+0000","last_became_peered":"2024-03-28T23:35:59.082360+0000","last_unstale":"2024-03-28T23:36:08.105213+0000","last_undegraded":"2024-03-28T23:36:08.105213+0000","last_fullsized":"2024-03-28T23:36:08.105213+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:00:59.308679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120174+0000","last_change":"2024-03-28T23:36:08.113999+0000","last_active":"2024-03-28T23:36:09.120174+0000","last_peered":"2024-03-28T23:36:09.120174+0000","last_clean":"2024-03-28T23:36:09.120174+0000","last_became_active":"2024-03-28T23:36:08.113514+0000","last_became_peered":"2024-03-28T23:36:08.113514+0000","last_unstale":"2024-03-28T23:36:09.120174+0000","last_undegraded":"2024-03-28T23:36:09.120174+0000","last_fullsized":"2024-03-28T23:36:09.120174+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:24:21.264344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.105918+0000","last_change":"2024-03-28T23:35:59.084222+0000","last_active":"2024-03-28T23:36:08.105918+0000","last_peered":"2024-03-28T23:36:08.105918+0000","last_clean":"2024-03-28T23:36:08.105918+0000","last_became_active":"2024-03-28T23:35:59.083603+0000","last_became_peered":"2024-03-28T23:35:59.083603+0000","last_unstale":"2024-03-28T23:36:08.105918+0000","last_undegraded":"2024-03-28T23:36:08.105918+0000","last_fullsized":"2024-03-28T23:36:08.105918+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:18:01.009699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117621+0000","last_change":"2024-03-28T23:36:08.115492+0000","last_active":"2024-03-28T23:36:09.117621+0000","last_peered":"2024-03-28T23:36:09.117621+0000","last_clean":"2024-03-28T23:36:09.117621+0000","last_became_active":"2024-03-28T23:36:08.112264+0000","last_became_peered":"2024-03-28T23:36:08.112264+0000","last_unstale":"2024-03-28T23:36:09.117621+0000","last_undegraded":"2024-03-28T23:36:09.117621+0000","last_fullsized":"2024-03-28T23:36:09.117621+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:41.904291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.108709+0000","last_change":"2024-03-28T23:35:59.074398+0000","last_active":"2024-03-28T23:36:08.108709+0000","last_peered":"2024-03-28T23:36:08.108709+0000","last_clean":"2024-03-28T23:36:08.108709+0000","last_became_active":"2024-03-28T23:35:59.074143+0000","last_became_peered":"2024-03-28T23:35:59.074143+0000","last_unstale":"2024-03-28T23:36:08.108709+0000","last_undegraded":"2024-03-28T23:36:08.108709+0000","last_fullsized":"2024-03-28T23:36:08.108709+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:47.081742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109102+0000","last_change":"2024-03-28T23:36:08.106097+0000","last_active":"2024-03-28T23:36:09.109102+0000","last_peered":"2024-03-28T23:36:09.109102+0000","last_clean":"2024-03-28T23:36:09.109102+0000","last_became_active":"2024-03-28T23:36:08.105118+0000","last_became_peered":"2024-03-28T23:36:08.105118+0000","last_unstale":"2024-03-28T23:36:09.109102+0000","last_undegraded":"2024-03-28T23:36:09.109102+0000","last_fullsized":"2024-03-28T23:36:09.109102+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:46:20.625726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108697+0000","last_change":"2024-03-28T23:35:59.066035+0000","last_active":"2024-03-28T23:36:09.108697+0000","last_peered":"2024-03-28T23:36:09.108697+0000","last_clean":"2024-03-28T23:36:09.108697+0000","last_became_active":"2024-03-28T23:35:59.065595+0000","last_became_peered":"2024-03-28T23:35:59.065595+0000","last_unstale":"2024-03-28T23:36:09.108697+0000","last_undegraded":"2024-03-28T23:36:09.108697+0000","last_fullsized":"2024-03-28T23:36:09.108697+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:26:11.783759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109402+0000","last_change":"2024-03-28T23:36:08.120716+0000","last_active":"2024-03-28T23:36:09.109402+0000","last_peered":"2024-03-28T23:36:09.109402+0000","last_clean":"2024-03-28T23:36:09.109402+0000","last_became_active":"2024-03-28T23:36:08.120057+0000","last_became_peered":"2024-03-28T23:36:08.120057+0000","last_unstale":"2024-03-28T23:36:09.109402+0000","last_undegraded":"2024-03-28T23:36:09.109402+0000","last_fullsized":"2024-03-28T23:36:09.109402+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:47.824582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119603+0000","last_change":"2024-03-28T23:35:59.076253+0000","last_active":"2024-03-28T23:36:09.119603+0000","last_peered":"2024-03-28T23:36:09.119603+0000","last_clean":"2024-03-28T23:36:09.119603+0000","last_became_active":"2024-03-28T23:35:59.075998+0000","last_became_peered":"2024-03-28T23:35:59.075998+0000","last_unstale":"2024-03-28T23:36:09.119603+0000","last_undegraded":"2024-03-28T23:36:09.119603+0000","last_fullsized":"2024-03-28T23:36:09.119603+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:51:06.388640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109031+0000","last_change":"2024-03-28T23:36:08.127064+0000","last_active":"2024-03-28T23:36:09.109031+0000","last_peered":"2024-03-28T23:36:09.109031+0000","last_clean":"2024-03-28T23:36:09.109031+0000","last_became_active":"2024-03-28T23:36:08.126601+0000","last_became_peered":"2024-03-28T23:36:08.126601+0000","last_unstale":"2024-03-28T23:36:09.109031+0000","last_undegraded":"2024-03-28T23:36:09.109031+0000","last_fullsized":"2024-03-28T23:36:09.109031+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:12:47.028863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119392+0000","last_change":"2024-03-28T23:35:59.077597+0000","last_active":"2024-03-28T23:36:09.119392+0000","last_peered":"2024-03-28T23:36:09.119392+0000","last_clean":"2024-03-28T23:36:09.119392+0000","last_became_active":"2024-03-28T23:35:59.077343+0000","last_became_peered":"2024-03-28T23:35:59.077343+0000","last_unstale":"2024-03-28T23:36:09.119392+0000","last_undegraded":"2024-03-28T23:36:09.119392+0000","last_fullsized":"2024-03-28T23:36:09.119392+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:26:11.783759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109119+0000","last_change":"2024-03-28T23:36:08.191205+0000","last_active":"2024-03-28T23:36:09.109119+0000","last_peered":"2024-03-28T23:36:09.109119+0000","last_clean":"2024-03-28T23:36:09.109119+0000","last_became_active":"2024-03-28T23:36:08.190885+0000","last_became_peered":"2024-03-28T23:36:08.190885+0000","last_unstale":"2024-03-28T23:36:09.109119+0000","last_undegraded":"2024-03-28T23:36:09.109119+0000","last_fullsized":"2024-03-28T23:36:09.109119+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:49:36.028241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120477+0000","last_change":"2024-03-28T23:35:59.073690+0000","last_active":"2024-03-28T23:36:09.120477+0000","last_peered":"2024-03-28T23:36:09.120477+0000","last_clean":"2024-03-28T23:36:09.120477+0000","last_became_active":"2024-03-28T23:35:59.073405+0000","last_became_peered":"2024-03-28T23:35:59.073405+0000","last_unstale":"2024-03-28T23:36:09.120477+0000","last_undegraded":"2024-03-28T23:36:09.120477+0000","last_fullsized":"2024-03-28T23:36:09.120477+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:17.228750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.120020+0000","last_change":"2024-03-28T23:36:08.115324+0000","last_active":"2024-03-28T23:36:08.120020+0000","last_peered":"2024-03-28T23:36:08.120020+0000","last_clean":"2024-03-28T23:36:08.120020+0000","last_became_active":"2024-03-28T23:36:08.115043+0000","last_became_peered":"2024-03-28T23:36:08.115043+0000","last_unstale":"2024-03-28T23:36:08.120020+0000","last_undegraded":"2024-03-28T23:36:08.120020+0000","last_fullsized":"2024-03-28T23:36:08.120020+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:39:47.021555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120287+0000","last_change":"2024-03-28T23:35:59.077619+0000","last_active":"2024-03-28T23:36:09.120287+0000","last_peered":"2024-03-28T23:36:09.120287+0000","last_clean":"2024-03-28T23:36:09.120287+0000","last_became_active":"2024-03-28T23:35:59.077386+0000","last_became_peered":"2024-03-28T23:35:59.077386+0000","last_unstale":"2024-03-28T23:36:09.120287+0000","last_undegraded":"2024-03-28T23:36:09.120287+0000","last_fullsized":"2024-03-28T23:36:09.120287+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:49:08.837095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120375+0000","last_change":"2024-03-28T23:36:08.191811+0000","last_active":"2024-03-28T23:36:09.120375+0000","last_peered":"2024-03-28T23:36:09.120375+0000","last_clean":"2024-03-28T23:36:09.120375+0000","last_became_active":"2024-03-28T23:36:08.191518+0000","last_became_peered":"2024-03-28T23:36:08.191518+0000","last_unstale":"2024-03-28T23:36:09.120375+0000","last_undegraded":"2024-03-28T23:36:09.120375+0000","last_fullsized":"2024-03-28T23:36:09.120375+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:09:57.210214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109258+0000","last_change":"2024-03-28T23:35:59.074672+0000","last_active":"2024-03-28T23:36:09.109258+0000","last_peered":"2024-03-28T23:36:09.109258+0000","last_clean":"2024-03-28T23:36:09.109258+0000","last_became_active":"2024-03-28T23:35:59.074385+0000","last_became_peered":"2024-03-28T23:35:59.074385+0000","last_unstale":"2024-03-28T23:36:09.109258+0000","last_undegraded":"2024-03-28T23:36:09.109258+0000","last_fullsized":"2024-03-28T23:36:09.109258+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:53:22.705919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109941+0000","last_change":"2024-03-28T23:36:08.116471+0000","last_active":"2024-03-28T23:36:09.109941+0000","last_peered":"2024-03-28T23:36:09.109941+0000","last_clean":"2024-03-28T23:36:09.109941+0000","last_became_active":"2024-03-28T23:36:08.116123+0000","last_became_peered":"2024-03-28T23:36:08.116123+0000","last_unstale":"2024-03-28T23:36:09.109941+0000","last_undegraded":"2024-03-28T23:36:09.109941+0000","last_fullsized":"2024-03-28T23:36:09.109941+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:34:32.575972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.110121+0000","last_change":"2024-03-28T23:35:59.072384+0000","last_active":"2024-03-28T23:36:09.110121+0000","last_peered":"2024-03-28T23:36:09.110121+0000","last_clean":"2024-03-28T23:36:09.110121+0000","last_became_active":"2024-03-28T23:35:59.071941+0000","last_became_peered":"2024-03-28T23:35:59.071941+0000","last_unstale":"2024-03-28T23:36:09.110121+0000","last_undegraded":"2024-03-28T23:36:09.110121+0000","last_fullsized":"2024-03-28T23:36:09.110121+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:36:26.239288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119748+0000","last_change":"2024-03-28T23:36:08.125010+0000","last_active":"2024-03-28T23:36:09.119748+0000","last_peered":"2024-03-28T23:36:09.119748+0000","last_clean":"2024-03-28T23:36:09.119748+0000","last_became_active":"2024-03-28T23:36:08.124704+0000","last_became_peered":"2024-03-28T23:36:08.124704+0000","last_unstale":"2024-03-28T23:36:09.119748+0000","last_undegraded":"2024-03-28T23:36:09.119748+0000","last_fullsized":"2024-03-28T23:36:09.119748+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:26:23.533905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109517+0000","last_change":"2024-03-28T23:35:59.076869+0000","last_active":"2024-03-28T23:36:09.109517+0000","last_peered":"2024-03-28T23:36:09.109517+0000","last_clean":"2024-03-28T23:36:09.109517+0000","last_became_active":"2024-03-28T23:35:59.076407+0000","last_became_peered":"2024-03-28T23:35:59.076407+0000","last_unstale":"2024-03-28T23:36:09.109517+0000","last_undegraded":"2024-03-28T23:36:09.109517+0000","last_fullsized":"2024-03-28T23:36:09.109517+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:47.081742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108907+0000","last_change":"2024-03-28T23:36:08.118544+0000","last_active":"2024-03-28T23:36:09.108907+0000","last_peered":"2024-03-28T23:36:09.108907+0000","last_clean":"2024-03-28T23:36:09.108907+0000","last_became_active":"2024-03-28T23:36:08.116769+0000","last_became_peered":"2024-03-28T23:36:08.116769+0000","last_unstale":"2024-03-28T23:36:09.108907+0000","last_undegraded":"2024-03-28T23:36:09.108907+0000","last_fullsized":"2024-03-28T23:36:09.108907+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:30:47.451354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.110495+0000","last_change":"2024-03-28T23:35:59.082855+0000","last_active":"2024-03-28T23:36:08.110495+0000","last_peered":"2024-03-28T23:36:08.110495+0000","last_clean":"2024-03-28T23:36:08.110495+0000","last_became_active":"2024-03-28T23:35:59.082581+0000","last_became_peered":"2024-03-28T23:35:59.082581+0000","last_unstale":"2024-03-28T23:36:08.110495+0000","last_undegraded":"2024-03-28T23:36:08.110495+0000","last_fullsized":"2024-03-28T23:36:08.110495+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:25.528570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.192047+0000","last_change":"2024-03-28T23:36:08.191168+0000","last_active":"2024-03-28T23:36:08.192047+0000","last_peered":"2024-03-28T23:36:08.192047+0000","last_clean":"2024-03-28T23:36:08.192047+0000","last_became_active":"2024-03-28T23:36:08.190750+0000","last_became_peered":"2024-03-28T23:36:08.190750+0000","last_unstale":"2024-03-28T23:36:08.192047+0000","last_undegraded":"2024-03-28T23:36:08.192047+0000","last_fullsized":"2024-03-28T23:36:08.192047+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:26.267458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108242+0000","last_change":"2024-03-28T23:35:59.080077+0000","last_active":"2024-03-28T23:36:09.108242+0000","last_peered":"2024-03-28T23:36:09.108242+0000","last_clean":"2024-03-28T23:36:09.108242+0000","last_became_active":"2024-03-28T23:35:59.079742+0000","last_became_peered":"2024-03-28T23:35:59.079742+0000","last_unstale":"2024-03-28T23:36:09.108242+0000","last_undegraded":"2024-03-28T23:36:09.108242+0000","last_fullsized":"2024-03-28T23:36:09.108242+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:59:11.349017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117502+0000","last_change":"2024-03-28T23:36:08.111135+0000","last_active":"2024-03-28T23:36:09.117502+0000","last_peered":"2024-03-28T23:36:09.117502+0000","last_clean":"2024-03-28T23:36:09.117502+0000","last_became_active":"2024-03-28T23:36:08.110815+0000","last_became_peered":"2024-03-28T23:36:08.110815+0000","last_unstale":"2024-03-28T23:36:09.117502+0000","last_undegraded":"2024-03-28T23:36:09.117502+0000","last_fullsized":"2024-03-28T23:36:09.117502+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:57:15.318239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.104103+0000","last_change":"2024-03-28T23:35:59.081618+0000","last_active":"2024-03-28T23:36:08.104103+0000","last_peered":"2024-03-28T23:36:08.104103+0000","last_clean":"2024-03-28T23:36:08.104103+0000","last_became_active":"2024-03-28T23:35:59.080909+0000","last_became_peered":"2024-03-28T23:35:59.080909+0000","last_unstale":"2024-03-28T23:36:08.104103+0000","last_undegraded":"2024-03-28T23:36:08.104103+0000","last_fullsized":"2024-03-28T23:36:08.104103+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:15.874144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.110770+0000","last_change":"2024-03-28T23:36:08.109599+0000","last_active":"2024-03-28T23:36:08.110770+0000","last_peered":"2024-03-28T23:36:08.110770+0000","last_clean":"2024-03-28T23:36:08.110770+0000","last_became_active":"2024-03-28T23:36:08.109324+0000","last_became_peered":"2024-03-28T23:36:08.109324+0000","last_unstale":"2024-03-28T23:36:08.110770+0000","last_undegraded":"2024-03-28T23:36:08.110770+0000","last_fullsized":"2024-03-28T23:36:08.110770+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:14:39.976869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120523+0000","last_change":"2024-03-28T23:35:59.072123+0000","last_active":"2024-03-28T23:36:09.120523+0000","last_peered":"2024-03-28T23:36:09.120523+0000","last_clean":"2024-03-28T23:36:09.120523+0000","last_became_active":"2024-03-28T23:35:59.071827+0000","last_became_peered":"2024-03-28T23:35:59.071827+0000","last_unstale":"2024-03-28T23:36:09.120523+0000","last_undegraded":"2024-03-28T23:36:09.120523+0000","last_fullsized":"2024-03-28T23:36:09.120523+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:11.814112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109514+0000","last_change":"2024-03-28T23:36:08.118037+0000","last_active":"2024-03-28T23:36:09.109514+0000","last_peered":"2024-03-28T23:36:09.109514+0000","last_clean":"2024-03-28T23:36:09.109514+0000","last_became_active":"2024-03-28T23:36:08.117767+0000","last_became_peered":"2024-03-28T23:36:08.117767+0000","last_unstale":"2024-03-28T23:36:09.109514+0000","last_undegraded":"2024-03-28T23:36:09.109514+0000","last_fullsized":"2024-03-28T23:36:09.109514+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:07:12.245731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.099214+0000","last_change":"2024-03-28T23:35:59.070208+0000","last_active":"2024-03-28T23:36:08.099214+0000","last_peered":"2024-03-28T23:36:08.099214+0000","last_clean":"2024-03-28T23:36:08.099214+0000","last_became_active":"2024-03-28T23:35:59.069928+0000","last_became_peered":"2024-03-28T23:35:59.069928+0000","last_unstale":"2024-03-28T23:36:08.099214+0000","last_undegraded":"2024-03-28T23:36:08.099214+0000","last_fullsized":"2024-03-28T23:36:08.099214+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:43:53.020768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109999+0000","last_change":"2024-03-28T23:36:08.117259+0000","last_active":"2024-03-28T23:36:09.109999+0000","last_peered":"2024-03-28T23:36:09.109999+0000","last_clean":"2024-03-28T23:36:09.109999+0000","last_became_active":"2024-03-28T23:36:08.116626+0000","last_became_peered":"2024-03-28T23:36:08.116626+0000","last_unstale":"2024-03-28T23:36:09.109999+0000","last_undegraded":"2024-03-28T23:36:09.109999+0000","last_fullsized":"2024-03-28T23:36:09.109999+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:25:31.117661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109336+0000","last_change":"2024-03-28T23:35:59.073450+0000","last_active":"2024-03-28T23:36:09.109336+0000","last_peered":"2024-03-28T23:36:09.109336+0000","last_clean":"2024-03-28T23:36:09.109336+0000","last_became_active":"2024-03-28T23:35:59.073178+0000","last_became_peered":"2024-03-28T23:35:59.073178+0000","last_unstale":"2024-03-28T23:36:09.109336+0000","last_undegraded":"2024-03-28T23:36:09.109336+0000","last_fullsized":"2024-03-28T23:36:09.109336+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:20:09.724407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.110645+0000","last_change":"2024-03-28T23:36:08.109669+0000","last_active":"2024-03-28T23:36:08.110645+0000","last_peered":"2024-03-28T23:36:08.110645+0000","last_clean":"2024-03-28T23:36:08.110645+0000","last_became_active":"2024-03-28T23:36:08.109427+0000","last_became_peered":"2024-03-28T23:36:08.109427+0000","last_unstale":"2024-03-28T23:36:08.110645+0000","last_undegraded":"2024-03-28T23:36:08.110645+0000","last_fullsized":"2024-03-28T23:36:08.110645+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:36:44.285827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117711+0000","last_change":"2024-03-28T23:35:59.080957+0000","last_active":"2024-03-28T23:36:09.117711+0000","last_peered":"2024-03-28T23:36:09.117711+0000","last_clean":"2024-03-28T23:36:09.117711+0000","last_became_active":"2024-03-28T23:35:59.080693+0000","last_became_peered":"2024-03-28T23:35:59.080693+0000","last_unstale":"2024-03-28T23:36:09.117711+0000","last_undegraded":"2024-03-28T23:36:09.117711+0000","last_fullsized":"2024-03-28T23:36:09.117711+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:05:53.689813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109053+0000","last_change":"2024-03-28T23:36:08.125499+0000","last_active":"2024-03-28T23:36:09.109053+0000","last_peered":"2024-03-28T23:36:09.109053+0000","last_clean":"2024-03-28T23:36:09.109053+0000","last_became_active":"2024-03-28T23:36:08.125246+0000","last_became_peered":"2024-03-28T23:36:08.125246+0000","last_unstale":"2024-03-28T23:36:09.109053+0000","last_undegraded":"2024-03-28T23:36:09.109053+0000","last_fullsized":"2024-03-28T23:36:09.109053+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:25:29.717562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109620+0000","last_change":"2024-03-28T23:35:59.080315+0000","last_active":"2024-03-28T23:36:09.109620+0000","last_peered":"2024-03-28T23:36:09.109620+0000","last_clean":"2024-03-28T23:36:09.109620+0000","last_became_active":"2024-03-28T23:35:59.080039+0000","last_became_peered":"2024-03-28T23:35:59.080039+0000","last_unstale":"2024-03-28T23:36:09.109620+0000","last_undegraded":"2024-03-28T23:36:09.109620+0000","last_fullsized":"2024-03-28T23:36:09.109620+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:25:20.678853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119582+0000","last_change":"2024-03-28T23:36:08.124154+0000","last_active":"2024-03-28T23:36:09.119582+0000","last_peered":"2024-03-28T23:36:09.119582+0000","last_clean":"2024-03-28T23:36:09.119582+0000","last_became_active":"2024-03-28T23:36:08.123927+0000","last_became_peered":"2024-03-28T23:36:08.123927+0000","last_unstale":"2024-03-28T23:36:09.119582+0000","last_undegraded":"2024-03-28T23:36:09.119582+0000","last_fullsized":"2024-03-28T23:36:09.119582+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:12:20.749280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108784+0000","last_change":"2024-03-28T23:35:59.074798+0000","last_active":"2024-03-28T23:36:09.108784+0000","last_peered":"2024-03-28T23:36:09.108784+0000","last_clean":"2024-03-28T23:36:09.108784+0000","last_became_active":"2024-03-28T23:35:59.074539+0000","last_became_peered":"2024-03-28T23:35:59.074539+0000","last_unstale":"2024-03-28T23:36:09.108784+0000","last_undegraded":"2024-03-28T23:36:09.108784+0000","last_fullsized":"2024-03-28T23:36:09.108784+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:51:06.388640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119806+0000","last_change":"2024-03-28T23:36:08.191662+0000","last_active":"2024-03-28T23:36:09.119806+0000","last_peered":"2024-03-28T23:36:09.119806+0000","last_clean":"2024-03-28T23:36:09.119806+0000","last_became_active":"2024-03-28T23:36:08.191396+0000","last_became_peered":"2024-03-28T23:36:08.191396+0000","last_unstale":"2024-03-28T23:36:09.119806+0000","last_undegraded":"2024-03-28T23:36:09.119806+0000","last_fullsized":"2024-03-28T23:36:09.119806+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:13:33.254712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.106708+0000","last_change":"2024-03-28T23:35:59.079811+0000","last_active":"2024-03-28T23:36:08.106708+0000","last_peered":"2024-03-28T23:36:08.106708+0000","last_clean":"2024-03-28T23:36:08.106708+0000","last_became_active":"2024-03-28T23:35:59.079538+0000","last_became_peered":"2024-03-28T23:35:59.079538+0000","last_unstale":"2024-03-28T23:36:08.106708+0000","last_undegraded":"2024-03-28T23:36:08.106708+0000","last_fullsized":"2024-03-28T23:36:08.106708+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:49:26.989533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119743+0000","last_change":"2024-03-28T23:36:08.191742+0000","last_active":"2024-03-28T23:36:09.119743+0000","last_peered":"2024-03-28T23:36:09.119743+0000","last_clean":"2024-03-28T23:36:09.119743+0000","last_became_active":"2024-03-28T23:36:08.191455+0000","last_became_peered":"2024-03-28T23:36:08.191455+0000","last_unstale":"2024-03-28T23:36:09.119743+0000","last_undegraded":"2024-03-28T23:36:09.119743+0000","last_fullsized":"2024-03-28T23:36:09.119743+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:21:51.431166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.104132+0000","last_change":"2024-03-28T23:35:59.080958+0000","last_active":"2024-03-28T23:36:08.104132+0000","last_peered":"2024-03-28T23:36:08.104132+0000","last_clean":"2024-03-28T23:36:08.104132+0000","last_became_active":"2024-03-28T23:35:59.080586+0000","last_became_peered":"2024-03-28T23:35:59.080586+0000","last_unstale":"2024-03-28T23:36:08.104132+0000","last_undegraded":"2024-03-28T23:36:08.104132+0000","last_fullsized":"2024-03-28T23:36:08.104132+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:53:35.858409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109688+0000","last_change":"2024-03-28T23:36:08.112625+0000","last_active":"2024-03-28T23:36:09.109688+0000","last_peered":"2024-03-28T23:36:09.109688+0000","last_clean":"2024-03-28T23:36:09.109688+0000","last_became_active":"2024-03-28T23:36:08.112430+0000","last_became_peered":"2024-03-28T23:36:08.112430+0000","last_unstale":"2024-03-28T23:36:09.109688+0000","last_undegraded":"2024-03-28T23:36:09.109688+0000","last_fullsized":"2024-03-28T23:36:09.109688+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:08:07.459225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.110219+0000","last_change":"2024-03-28T23:35:59.076376+0000","last_active":"2024-03-28T23:36:09.110219+0000","last_peered":"2024-03-28T23:36:09.110219+0000","last_clean":"2024-03-28T23:36:09.110219+0000","last_became_active":"2024-03-28T23:35:59.076009+0000","last_became_peered":"2024-03-28T23:35:59.076009+0000","last_unstale":"2024-03-28T23:36:09.110219+0000","last_undegraded":"2024-03-28T23:36:09.110219+0000","last_fullsized":"2024-03-28T23:36:09.110219+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:30:23.264136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109420+0000","last_change":"2024-03-28T23:36:08.116900+0000","last_active":"2024-03-28T23:36:09.109420+0000","last_peered":"2024-03-28T23:36:09.109420+0000","last_clean":"2024-03-28T23:36:09.109420+0000","last_became_active":"2024-03-28T23:36:08.116268+0000","last_became_peered":"2024-03-28T23:36:08.116268+0000","last_unstale":"2024-03-28T23:36:09.109420+0000","last_undegraded":"2024-03-28T23:36:09.109420+0000","last_fullsized":"2024-03-28T23:36:09.109420+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:25:16.454692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117915+0000","last_change":"2024-03-28T23:35:59.069546+0000","last_active":"2024-03-28T23:36:09.117915+0000","last_peered":"2024-03-28T23:36:09.117915+0000","last_clean":"2024-03-28T23:36:09.117915+0000","last_became_active":"2024-03-28T23:35:59.069283+0000","last_became_peered":"2024-03-28T23:35:59.069283+0000","last_unstale":"2024-03-28T23:36:09.117915+0000","last_undegraded":"2024-03-28T23:36:09.117915+0000","last_fullsized":"2024-03-28T23:36:09.117915+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:36:41.518215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.118029+0000","last_change":"2024-03-28T23:36:08.117498+0000","last_active":"2024-03-28T23:36:09.118029+0000","last_peered":"2024-03-28T23:36:09.118029+0000","last_clean":"2024-03-28T23:36:09.118029+0000","last_became_active":"2024-03-28T23:36:08.117232+0000","last_became_peered":"2024-03-28T23:36:08.117232+0000","last_unstale":"2024-03-28T23:36:09.118029+0000","last_undegraded":"2024-03-28T23:36:09.118029+0000","last_fullsized":"2024-03-28T23:36:09.118029+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:27:51.966745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109675+0000","last_change":"2024-03-28T23:35:59.073880+0000","last_active":"2024-03-28T23:36:09.109675+0000","last_peered":"2024-03-28T23:36:09.109675+0000","last_clean":"2024-03-28T23:36:09.109675+0000","last_became_active":"2024-03-28T23:35:59.073579+0000","last_became_peered":"2024-03-28T23:35:59.073579+0000","last_unstale":"2024-03-28T23:36:09.109675+0000","last_undegraded":"2024-03-28T23:36:09.109675+0000","last_fullsized":"2024-03-28T23:36:09.109675+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:54:04.050429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109021+0000","last_change":"2024-03-28T23:36:08.118174+0000","last_active":"2024-03-28T23:36:09.109021+0000","last_peered":"2024-03-28T23:36:09.109021+0000","last_clean":"2024-03-28T23:36:09.109021+0000","last_became_active":"2024-03-28T23:36:08.117488+0000","last_became_peered":"2024-03-28T23:36:08.117488+0000","last_unstale":"2024-03-28T23:36:09.109021+0000","last_undegraded":"2024-03-28T23:36:09.109021+0000","last_fullsized":"2024-03-28T23:36:09.109021+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:44:32.559938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109361+0000","last_change":"2024-03-28T23:35:59.082034+0000","last_active":"2024-03-28T23:36:09.109361+0000","last_peered":"2024-03-28T23:36:09.109361+0000","last_clean":"2024-03-28T23:36:09.109361+0000","last_became_active":"2024-03-28T23:35:59.081763+0000","last_became_peered":"2024-03-28T23:35:59.081763+0000","last_unstale":"2024-03-28T23:36:09.109361+0000","last_undegraded":"2024-03-28T23:36:09.109361+0000","last_fullsized":"2024-03-28T23:36:09.109361+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:07:58.420516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117476+0000","last_change":"2024-03-28T23:36:08.128000+0000","last_active":"2024-03-28T23:36:09.117476+0000","last_peered":"2024-03-28T23:36:09.117476+0000","last_clean":"2024-03-28T23:36:09.117476+0000","last_became_active":"2024-03-28T23:36:08.127716+0000","last_became_peered":"2024-03-28T23:36:08.127716+0000","last_unstale":"2024-03-28T23:36:09.117476+0000","last_undegraded":"2024-03-28T23:36:09.117476+0000","last_fullsized":"2024-03-28T23:36:09.117476+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:11:58.809402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108982+0000","last_change":"2024-03-28T23:35:59.073561+0000","last_active":"2024-03-28T23:36:09.108982+0000","last_peered":"2024-03-28T23:36:09.108982+0000","last_clean":"2024-03-28T23:36:09.108982+0000","last_became_active":"2024-03-28T23:35:59.073301+0000","last_became_peered":"2024-03-28T23:35:59.073301+0000","last_unstale":"2024-03-28T23:36:09.108982+0000","last_undegraded":"2024-03-28T23:36:09.108982+0000","last_fullsized":"2024-03-28T23:36:09.108982+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:24.837863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108984+0000","last_change":"2024-03-28T23:36:08.116426+0000","last_active":"2024-03-28T23:36:09.108984+0000","last_peered":"2024-03-28T23:36:09.108984+0000","last_clean":"2024-03-28T23:36:09.108984+0000","last_became_active":"2024-03-28T23:36:08.116113+0000","last_became_peered":"2024-03-28T23:36:08.116113+0000","last_unstale":"2024-03-28T23:36:09.108984+0000","last_undegraded":"2024-03-28T23:36:09.108984+0000","last_fullsized":"2024-03-28T23:36:09.108984+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:18:29.164476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.099368+0000","last_change":"2024-03-28T23:35:59.074502+0000","last_active":"2024-03-28T23:36:08.099368+0000","last_peered":"2024-03-28T23:36:08.099368+0000","last_clean":"2024-03-28T23:36:08.099368+0000","last_became_active":"2024-03-28T23:35:59.074214+0000","last_became_peered":"2024-03-28T23:35:59.074214+0000","last_unstale":"2024-03-28T23:36:08.099368+0000","last_undegraded":"2024-03-28T23:36:08.099368+0000","last_fullsized":"2024-03-28T23:36:08.099368+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:03.147158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108645+0000","last_change":"2024-03-28T23:36:08.120168+0000","last_active":"2024-03-28T23:36:09.108645+0000","last_peered":"2024-03-28T23:36:09.108645+0000","last_clean":"2024-03-28T23:36:09.108645+0000","last_became_active":"2024-03-28T23:36:08.119896+0000","last_became_peered":"2024-03-28T23:36:08.119896+0000","last_unstale":"2024-03-28T23:36:09.108645+0000","last_undegraded":"2024-03-28T23:36:09.108645+0000","last_fullsized":"2024-03-28T23:36:09.108645+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:08:39.946119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108354+0000","last_change":"2024-03-28T23:35:59.080452+0000","last_active":"2024-03-28T23:36:09.108354+0000","last_peered":"2024-03-28T23:36:09.108354+0000","last_clean":"2024-03-28T23:36:09.108354+0000","last_became_active":"2024-03-28T23:35:59.080141+0000","last_became_peered":"2024-03-28T23:35:59.080141+0000","last_unstale":"2024-03-28T23:36:09.108354+0000","last_undegraded":"2024-03-28T23:36:09.108354+0000","last_fullsized":"2024-03-28T23:36:09.108354+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:16:00.430973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108717+0000","last_change":"2024-03-28T23:36:08.125030+0000","last_active":"2024-03-28T23:36:09.108717+0000","last_peered":"2024-03-28T23:36:09.108717+0000","last_clean":"2024-03-28T23:36:09.108717+0000","last_became_active":"2024-03-28T23:36:08.124772+0000","last_became_peered":"2024-03-28T23:36:08.124772+0000","last_unstale":"2024-03-28T23:36:09.108717+0000","last_undegraded":"2024-03-28T23:36:09.108717+0000","last_fullsized":"2024-03-28T23:36:09.108717+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:36:55.152832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.188603+0000","last_change":"2024-03-28T23:35:59.068642+0000","last_active":"2024-03-28T23:36:08.188603+0000","last_peered":"2024-03-28T23:36:08.188603+0000","last_clean":"2024-03-28T23:36:08.188603+0000","last_became_active":"2024-03-28T23:35:59.068349+0000","last_became_peered":"2024-03-28T23:35:59.068349+0000","last_unstale":"2024-03-28T23:36:08.188603+0000","last_undegraded":"2024-03-28T23:36:08.188603+0000","last_fullsized":"2024-03-28T23:36:08.188603+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:29.399434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.116944+0000","last_change":"2024-03-28T23:36:08.116485+0000","last_active":"2024-03-28T23:36:08.116944+0000","last_peered":"2024-03-28T23:36:08.116944+0000","last_clean":"2024-03-28T23:36:08.116944+0000","last_became_active":"2024-03-28T23:36:08.116068+0000","last_became_peered":"2024-03-28T23:36:08.116068+0000","last_unstale":"2024-03-28T23:36:08.116944+0000","last_undegraded":"2024-03-28T23:36:08.116944+0000","last_fullsized":"2024-03-28T23:36:08.116944+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:44.966490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.106595+0000","last_change":"2024-03-28T23:36:07.106269+0000","last_active":"2024-03-28T23:36:08.106595+0000","last_peered":"2024-03-28T23:36:08.106595+0000","last_clean":"2024-03-28T23:36:08.106595+0000","last_became_active":"2024-03-28T23:36:07.106026+0000","last_became_peered":"2024-03-28T23:36:07.106026+0000","last_unstale":"2024-03-28T23:36:08.106595+0000","last_undegraded":"2024-03-28T23:36:08.106595+0000","last_fullsized":"2024-03-28T23:36:08.106595+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:55.739448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109156+0000","last_change":"2024-03-28T23:35:59.083382+0000","last_active":"2024-03-28T23:36:09.109156+0000","last_peered":"2024-03-28T23:36:09.109156+0000","last_clean":"2024-03-28T23:36:09.109156+0000","last_became_active":"2024-03-28T23:35:59.083111+0000","last_became_peered":"2024-03-28T23:35:59.083111+0000","last_unstale":"2024-03-28T23:36:09.109156+0000","last_undegraded":"2024-03-28T23:36:09.109156+0000","last_fullsized":"2024-03-28T23:36:09.109156+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:36:35.247118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"151'4","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.105955+0000","last_change":"2024-03-28T23:35:58.071726+0000","last_active":"2024-03-28T23:36:08.105955+0000","last_peered":"2024-03-28T23:36:08.105955+0000","last_clean":"2024-03-28T23:36:08.105955+0000","last_became_active":"2024-03-28T23:35:58.071384+0000","last_became_peered":"2024-03-28T23:35:58.071384+0000","last_unstale":"2024-03-28T23:36:08.105955+0000","last_undegraded":"2024-03-28T23:36:08.105955+0000","last_fullsized":"2024-03-28T23:36:08.105955+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:31.860347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.130226+0000","last_change":"2024-03-28T23:36:08.129860+0000","last_active":"2024-03-28T23:36:08.130226+0000","last_peered":"2024-03-28T23:36:08.130226+0000","last_clean":"2024-03-28T23:36:08.130226+0000","last_became_active":"2024-03-28T23:36:08.129659+0000","last_became_peered":"2024-03-28T23:36:08.129659+0000","last_unstale":"2024-03-28T23:36:08.130226+0000","last_undegraded":"2024-03-28T23:36:08.130226+0000","last_fullsized":"2024-03-28T23:36:08.130226+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:26:20.822468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"155'8","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108307+0000","last_change":"2024-03-28T23:36:07.108715+0000","last_active":"2024-03-28T23:36:09.108307+0000","last_peered":"2024-03-28T23:36:09.108307+0000","last_clean":"2024-03-28T23:36:09.108307+0000","last_became_active":"2024-03-28T23:36:07.108286+0000","last_became_peered":"2024-03-28T23:36:07.108286+0000","last_unstale":"2024-03-28T23:36:09.108307+0000","last_undegraded":"2024-03-28T23:36:09.108307+0000","last_fullsized":"2024-03-28T23:36:09.108307+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:25.263616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108953+0000","last_change":"2024-03-28T23:35:59.082507+0000","last_active":"2024-03-28T23:36:09.108953+0000","last_peered":"2024-03-28T23:36:09.108953+0000","last_clean":"2024-03-28T23:36:09.108953+0000","last_became_active":"2024-03-28T23:35:59.082004+0000","last_became_peered":"2024-03-28T23:35:59.082004+0000","last_unstale":"2024-03-28T23:36:09.108953+0000","last_undegraded":"2024-03-28T23:36:09.108953+0000","last_fullsized":"2024-03-28T23:36:09.108953+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:54:41.513429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"151'1","reported_seq":21,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.109522+0000","last_change":"2024-03-28T23:35:58.070559+0000","last_active":"2024-03-28T23:36:08.109522+0000","last_peered":"2024-03-28T23:36:08.109522+0000","last_clean":"2024-03-28T23:36:08.109522+0000","last_became_active":"2024-03-28T23:35:58.070259+0000","last_became_peered":"2024-03-28T23:35:58.070259+0000","last_unstale":"2024-03-28T23:36:08.109522+0000","last_undegraded":"2024-03-28T23:36:08.109522+0000","last_fullsized":"2024-03-28T23:36:08.109522+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:23.832628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117349+0000","last_change":"2024-03-28T23:36:08.112610+0000","last_active":"2024-03-28T23:36:09.117349+0000","last_peered":"2024-03-28T23:36:09.117349+0000","last_clean":"2024-03-28T23:36:09.117349+0000","last_became_active":"2024-03-28T23:36:08.111833+0000","last_became_peered":"2024-03-28T23:36:08.111833+0000","last_unstale":"2024-03-28T23:36:09.117349+0000","last_undegraded":"2024-03-28T23:36:09.117349+0000","last_fullsized":"2024-03-28T23:36:09.117349+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:13:35.088394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.124175+0000","last_change":"2024-03-28T23:36:07.101366+0000","last_active":"2024-03-28T23:36:08.124175+0000","last_peered":"2024-03-28T23:36:08.124175+0000","last_clean":"2024-03-28T23:36:08.124175+0000","last_became_active":"2024-03-28T23:36:07.101016+0000","last_became_peered":"2024-03-28T23:36:07.101016+0000","last_unstale":"2024-03-28T23:36:08.124175+0000","last_undegraded":"2024-03-28T23:36:08.124175+0000","last_fullsized":"2024-03-28T23:36:08.124175+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:42:49.756856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.113087+0000","last_change":"2024-03-28T23:35:59.082185+0000","last_active":"2024-03-28T23:36:08.113087+0000","last_peered":"2024-03-28T23:36:08.113087+0000","last_clean":"2024-03-28T23:36:08.113087+0000","last_became_active":"2024-03-28T23:35:59.081892+0000","last_became_peered":"2024-03-28T23:35:59.081892+0000","last_unstale":"2024-03-28T23:36:08.113087+0000","last_undegraded":"2024-03-28T23:36:08.113087+0000","last_fullsized":"2024-03-28T23:36:08.113087+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:30:33.779993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"151'2","reported_seq":24,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109940+0000","last_change":"2024-03-28T23:35:58.069940+0000","last_active":"2024-03-28T23:36:09.109940+0000","last_peered":"2024-03-28T23:36:09.109940+0000","last_clean":"2024-03-28T23:36:09.109940+0000","last_became_active":"2024-03-28T23:35:58.069533+0000","last_became_peered":"2024-03-28T23:35:58.069533+0000","last_unstale":"2024-03-28T23:36:09.109940+0000","last_undegraded":"2024-03-28T23:36:09.109940+0000","last_fullsized":"2024-03-28T23:36:09.109940+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:02:14.868910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108903+0000","last_change":"2024-03-28T23:36:08.192478+0000","last_active":"2024-03-28T23:36:09.108903+0000","last_peered":"2024-03-28T23:36:09.108903+0000","last_clean":"2024-03-28T23:36:09.108903+0000","last_became_active":"2024-03-28T23:36:08.192173+0000","last_became_peered":"2024-03-28T23:36:08.192173+0000","last_unstale":"2024-03-28T23:36:09.108903+0000","last_undegraded":"2024-03-28T23:36:09.108903+0000","last_fullsized":"2024-03-28T23:36:09.108903+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:33.489134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120148+0000","last_change":"2024-03-28T23:36:07.111796+0000","last_active":"2024-03-28T23:36:09.120148+0000","last_peered":"2024-03-28T23:36:09.120148+0000","last_clean":"2024-03-28T23:36:09.120148+0000","last_became_active":"2024-03-28T23:36:07.111521+0000","last_became_peered":"2024-03-28T23:36:07.111521+0000","last_unstale":"2024-03-28T23:36:09.120148+0000","last_undegraded":"2024-03-28T23:36:09.120148+0000","last_fullsized":"2024-03-28T23:36:09.120148+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:49:35.024399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109457+0000","last_change":"2024-03-28T23:35:59.080903+0000","last_active":"2024-03-28T23:36:09.109457+0000","last_peered":"2024-03-28T23:36:09.109457+0000","last_clean":"2024-03-28T23:36:09.109457+0000","last_became_active":"2024-03-28T23:35:59.080632+0000","last_became_peered":"2024-03-28T23:35:59.080632+0000","last_unstale":"2024-03-28T23:36:09.109457+0000","last_undegraded":"2024-03-28T23:36:09.109457+0000","last_fullsized":"2024-03-28T23:36:09.109457+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:37:10.552720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"151'2","reported_seq":24,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117800+0000","last_change":"2024-03-28T23:35:58.064254+0000","last_active":"2024-03-28T23:36:09.117800+0000","last_peered":"2024-03-28T23:36:09.117800+0000","last_clean":"2024-03-28T23:36:09.117800+0000","last_became_active":"2024-03-28T23:35:58.063972+0000","last_became_peered":"2024-03-28T23:35:58.063972+0000","last_unstale":"2024-03-28T23:36:09.117800+0000","last_undegraded":"2024-03-28T23:36:09.117800+0000","last_fullsized":"2024-03-28T23:36:09.117800+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:36:14.911401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.110627+0000","last_change":"2024-03-28T23:36:08.109190+0000","last_active":"2024-03-28T23:36:08.110627+0000","last_peered":"2024-03-28T23:36:08.110627+0000","last_clean":"2024-03-28T23:36:08.110627+0000","last_became_active":"2024-03-28T23:36:08.108910+0000","last_became_peered":"2024-03-28T23:36:08.108910+0000","last_unstale":"2024-03-28T23:36:08.110627+0000","last_undegraded":"2024-03-28T23:36:08.110627+0000","last_fullsized":"2024-03-28T23:36:08.110627+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:26:10.148176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"155'3","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.129846+0000","last_change":"2024-03-28T23:36:07.115429+0000","last_active":"2024-03-28T23:36:08.129846+0000","last_peered":"2024-03-28T23:36:08.129846+0000","last_clean":"2024-03-28T23:36:08.129846+0000","last_became_active":"2024-03-28T23:36:07.115174+0000","last_became_peered":"2024-03-28T23:36:07.115174+0000","last_unstale":"2024-03-28T23:36:08.129846+0000","last_undegraded":"2024-03-28T23:36:08.129846+0000","last_fullsized":"2024-03-28T23:36:08.129846+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:21:22.235323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120195+0000","last_change":"2024-03-28T23:35:59.077647+0000","last_active":"2024-03-28T23:36:09.120195+0000","last_peered":"2024-03-28T23:36:09.120195+0000","last_clean":"2024-03-28T23:36:09.120195+0000","last_became_active":"2024-03-28T23:35:59.077374+0000","last_became_peered":"2024-03-28T23:35:59.077374+0000","last_unstale":"2024-03-28T23:36:09.120195+0000","last_undegraded":"2024-03-28T23:36:09.120195+0000","last_fullsized":"2024-03-28T23:36:09.120195+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:37:10.552720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"151'1","reported_seq":23,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119886+0000","last_change":"2024-03-28T23:35:58.074449+0000","last_active":"2024-03-28T23:36:09.119886+0000","last_peered":"2024-03-28T23:36:09.119886+0000","last_clean":"2024-03-28T23:36:09.119886+0000","last_became_active":"2024-03-28T23:35:58.074171+0000","last_became_peered":"2024-03-28T23:35:58.074171+0000","last_unstale":"2024-03-28T23:36:09.119886+0000","last_undegraded":"2024-03-28T23:36:09.119886+0000","last_fullsized":"2024-03-28T23:36:09.119886+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:41:31.356630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120382+0000","last_change":"2024-03-28T23:36:08.115737+0000","last_active":"2024-03-28T23:36:09.120382+0000","last_peered":"2024-03-28T23:36:09.120382+0000","last_clean":"2024-03-28T23:36:09.120382+0000","last_became_active":"2024-03-28T23:36:08.115468+0000","last_became_peered":"2024-03-28T23:36:08.115468+0000","last_unstale":"2024-03-28T23:36:09.120382+0000","last_undegraded":"2024-03-28T23:36:09.120382+0000","last_fullsized":"2024-03-28T23:36:09.120382+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:54.124466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"155'6","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.125866+0000","last_change":"2024-03-28T23:36:07.098719+0000","last_active":"2024-03-28T23:36:08.125866+0000","last_peered":"2024-03-28T23:36:08.125866+0000","last_clean":"2024-03-28T23:36:08.125866+0000","last_became_active":"2024-03-28T23:36:07.098395+0000","last_became_peered":"2024-03-28T23:36:07.098395+0000","last_unstale":"2024-03-28T23:36:08.125866+0000","last_undegraded":"2024-03-28T23:36:08.125866+0000","last_fullsized":"2024-03-28T23:36:08.125866+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:53:30.740785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117996+0000","last_change":"2024-03-28T23:35:59.086653+0000","last_active":"2024-03-28T23:36:09.117996+0000","last_peered":"2024-03-28T23:36:09.117996+0000","last_clean":"2024-03-28T23:36:09.117996+0000","last_became_active":"2024-03-28T23:35:59.086398+0000","last_became_peered":"2024-03-28T23:35:59.086398+0000","last_unstale":"2024-03-28T23:36:09.117996+0000","last_undegraded":"2024-03-28T23:36:09.117996+0000","last_fullsized":"2024-03-28T23:36:09.117996+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:18:46.035632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"151'1","reported_seq":21,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.112936+0000","last_change":"2024-03-28T23:35:58.073421+0000","last_active":"2024-03-28T23:36:08.112936+0000","last_peered":"2024-03-28T23:36:08.112936+0000","last_clean":"2024-03-28T23:36:08.112936+0000","last_became_active":"2024-03-28T23:35:58.072004+0000","last_became_peered":"2024-03-28T23:35:58.072004+0000","last_unstale":"2024-03-28T23:36:08.112936+0000","last_undegraded":"2024-03-28T23:36:08.112936+0000","last_fullsized":"2024-03-28T23:36:08.112936+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:10.808877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.110002+0000","last_change":"2024-03-28T23:36:08.110229+0000","last_active":"2024-03-28T23:36:09.110002+0000","last_peered":"2024-03-28T23:36:09.110002+0000","last_clean":"2024-03-28T23:36:09.110002+0000","last_became_active":"2024-03-28T23:36:08.109984+0000","last_became_peered":"2024-03-28T23:36:08.109984+0000","last_unstale":"2024-03-28T23:36:09.110002+0000","last_undegraded":"2024-03-28T23:36:09.110002+0000","last_fullsized":"2024-03-28T23:36:09.110002+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:58:10.521424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.110114+0000","last_change":"2024-03-28T23:36:07.111385+0000","last_active":"2024-03-28T23:36:09.110114+0000","last_peered":"2024-03-28T23:36:09.110114+0000","last_clean":"2024-03-28T23:36:09.110114+0000","last_became_active":"2024-03-28T23:36:07.111075+0000","last_became_peered":"2024-03-28T23:36:07.111075+0000","last_unstale":"2024-03-28T23:36:09.110114+0000","last_undegraded":"2024-03-28T23:36:09.110114+0000","last_fullsized":"2024-03-28T23:36:09.110114+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:13:32.250870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117969+0000","last_change":"2024-03-28T23:35:59.063776+0000","last_active":"2024-03-28T23:36:09.117969+0000","last_peered":"2024-03-28T23:36:09.117969+0000","last_clean":"2024-03-28T23:36:09.117969+0000","last_became_active":"2024-03-28T23:35:59.063214+0000","last_became_peered":"2024-03-28T23:35:59.063214+0000","last_unstale":"2024-03-28T23:36:09.117969+0000","last_undegraded":"2024-03-28T23:36:09.117969+0000","last_fullsized":"2024-03-28T23:36:09.117969+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:24:06.490491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117611+0000","last_change":"2024-03-28T23:35:58.072230+0000","last_active":"2024-03-28T23:36:09.117611+0000","last_peered":"2024-03-28T23:36:09.117611+0000","last_clean":"2024-03-28T23:36:09.117611+0000","last_became_active":"2024-03-28T23:35:58.071967+0000","last_became_peered":"2024-03-28T23:35:58.071967+0000","last_unstale":"2024-03-28T23:36:09.117611+0000","last_undegraded":"2024-03-28T23:36:09.117611+0000","last_fullsized":"2024-03-28T23:36:09.117611+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:07:28.431728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109881+0000","last_change":"2024-03-28T23:36:08.118205+0000","last_active":"2024-03-28T23:36:09.109881+0000","last_peered":"2024-03-28T23:36:09.109881+0000","last_clean":"2024-03-28T23:36:09.109881+0000","last_became_active":"2024-03-28T23:36:08.117900+0000","last_became_peered":"2024-03-28T23:36:08.117900+0000","last_unstale":"2024-03-28T23:36:09.109881+0000","last_undegraded":"2024-03-28T23:36:09.109881+0000","last_fullsized":"2024-03-28T23:36:09.109881+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:14.662449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120310+0000","last_change":"2024-03-28T23:36:07.115533+0000","last_active":"2024-03-28T23:36:09.120310+0000","last_peered":"2024-03-28T23:36:09.120310+0000","last_clean":"2024-03-28T23:36:09.120310+0000","last_became_active":"2024-03-28T23:36:07.115238+0000","last_became_peered":"2024-03-28T23:36:07.115238+0000","last_unstale":"2024-03-28T23:36:09.120310+0000","last_undegraded":"2024-03-28T23:36:09.120310+0000","last_fullsized":"2024-03-28T23:36:09.120310+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:44:31.556096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.105868+0000","last_change":"2024-03-28T23:35:59.070728+0000","last_active":"2024-03-28T23:36:08.105868+0000","last_peered":"2024-03-28T23:36:08.105868+0000","last_clean":"2024-03-28T23:36:08.105868+0000","last_became_active":"2024-03-28T23:35:59.070489+0000","last_became_peered":"2024-03-28T23:35:59.070489+0000","last_unstale":"2024-03-28T23:36:08.105868+0000","last_undegraded":"2024-03-28T23:36:08.105868+0000","last_fullsized":"2024-03-28T23:36:08.105868+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:24:07.077373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119635+0000","last_change":"2024-03-28T23:35:58.068807+0000","last_active":"2024-03-28T23:36:09.119635+0000","last_peered":"2024-03-28T23:36:09.119635+0000","last_clean":"2024-03-28T23:36:09.119635+0000","last_became_active":"2024-03-28T23:35:58.068530+0000","last_became_peered":"2024-03-28T23:35:58.068530+0000","last_unstale":"2024-03-28T23:36:09.119635+0000","last_undegraded":"2024-03-28T23:36:09.119635+0000","last_fullsized":"2024-03-28T23:36:09.119635+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:14:12.327316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119898+0000","last_change":"2024-03-28T23:36:08.113949+0000","last_active":"2024-03-28T23:36:09.119898+0000","last_peered":"2024-03-28T23:36:09.119898+0000","last_clean":"2024-03-28T23:36:09.119898+0000","last_became_active":"2024-03-28T23:36:08.113679+0000","last_became_peered":"2024-03-28T23:36:08.113679+0000","last_unstale":"2024-03-28T23:36:09.119898+0000","last_undegraded":"2024-03-28T23:36:09.119898+0000","last_fullsized":"2024-03-28T23:36:09.119898+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:39:57.434327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.104220+0000","last_change":"2024-03-28T23:36:07.101098+0000","last_active":"2024-03-28T23:36:08.104220+0000","last_peered":"2024-03-28T23:36:08.104220+0000","last_clean":"2024-03-28T23:36:08.104220+0000","last_became_active":"2024-03-28T23:36:07.100726+0000","last_became_peered":"2024-03-28T23:36:07.100726+0000","last_unstale":"2024-03-28T23:36:08.104220+0000","last_undegraded":"2024-03-28T23:36:08.104220+0000","last_fullsized":"2024-03-28T23:36:08.104220+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:06:45.247578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108910+0000","last_change":"2024-03-28T23:35:59.082657+0000","last_active":"2024-03-28T23:36:09.108910+0000","last_peered":"2024-03-28T23:36:09.108910+0000","last_clean":"2024-03-28T23:36:09.108910+0000","last_became_active":"2024-03-28T23:35:59.082390+0000","last_became_peered":"2024-03-28T23:35:59.082390+0000","last_unstale":"2024-03-28T23:36:09.108910+0000","last_undegraded":"2024-03-28T23:36:09.108910+0000","last_fullsized":"2024-03-28T23:36:09.108910+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:14:30.938161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.110165+0000","last_change":"2024-03-28T23:35:58.075215+0000","last_active":"2024-03-28T23:36:09.110165+0000","last_peered":"2024-03-28T23:36:09.110165+0000","last_clean":"2024-03-28T23:36:09.110165+0000","last_became_active":"2024-03-28T23:35:58.074891+0000","last_became_peered":"2024-03-28T23:35:58.074891+0000","last_unstale":"2024-03-28T23:36:09.110165+0000","last_undegraded":"2024-03-28T23:36:09.110165+0000","last_fullsized":"2024-03-28T23:36:09.110165+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:16.223515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.120777+0000","last_change":"2024-03-28T23:36:08.119235+0000","last_active":"2024-03-28T23:36:08.120777+0000","last_peered":"2024-03-28T23:36:08.120777+0000","last_clean":"2024-03-28T23:36:08.120777+0000","last_became_active":"2024-03-28T23:36:08.118875+0000","last_became_peered":"2024-03-28T23:36:08.118875+0000","last_unstale":"2024-03-28T23:36:08.120777+0000","last_undegraded":"2024-03-28T23:36:08.120777+0000","last_fullsized":"2024-03-28T23:36:08.120777+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:41.386690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109451+0000","last_change":"2024-03-28T23:36:07.110312+0000","last_active":"2024-03-28T23:36:09.109451+0000","last_peered":"2024-03-28T23:36:09.109451+0000","last_clean":"2024-03-28T23:36:09.109451+0000","last_became_active":"2024-03-28T23:36:07.109966+0000","last_became_peered":"2024-03-28T23:36:07.109966+0000","last_unstale":"2024-03-28T23:36:09.109451+0000","last_undegraded":"2024-03-28T23:36:09.109451+0000","last_fullsized":"2024-03-28T23:36:09.109451+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:12:19.745437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109057+0000","last_change":"2024-03-28T23:35:59.079653+0000","last_active":"2024-03-28T23:36:09.109057+0000","last_peered":"2024-03-28T23:36:09.109057+0000","last_clean":"2024-03-28T23:36:09.109057+0000","last_became_active":"2024-03-28T23:35:59.079390+0000","last_became_peered":"2024-03-28T23:35:59.079390+0000","last_unstale":"2024-03-28T23:36:09.109057+0000","last_undegraded":"2024-03-28T23:36:09.109057+0000","last_fullsized":"2024-03-28T23:36:09.109057+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:39:27.912295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"151'3","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.105309+0000","last_change":"2024-03-28T23:35:58.059191+0000","last_active":"2024-03-28T23:36:08.105309+0000","last_peered":"2024-03-28T23:36:08.105309+0000","last_clean":"2024-03-28T23:36:08.105309+0000","last_became_active":"2024-03-28T23:35:58.058860+0000","last_became_peered":"2024-03-28T23:35:58.058860+0000","last_unstale":"2024-03-28T23:36:08.105309+0000","last_undegraded":"2024-03-28T23:36:08.105309+0000","last_fullsized":"2024-03-28T23:36:08.105309+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:22.564724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119515+0000","last_change":"2024-03-28T23:36:08.121789+0000","last_active":"2024-03-28T23:36:09.119515+0000","last_peered":"2024-03-28T23:36:09.119515+0000","last_clean":"2024-03-28T23:36:09.119515+0000","last_became_active":"2024-03-28T23:36:08.121527+0000","last_became_peered":"2024-03-28T23:36:08.121527+0000","last_unstale":"2024-03-28T23:36:09.119515+0000","last_undegraded":"2024-03-28T23:36:09.119515+0000","last_fullsized":"2024-03-28T23:36:09.119515+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:56.743290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"155'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.129755+0000","last_change":"2024-03-28T23:36:07.111476+0000","last_active":"2024-03-28T23:36:08.129755+0000","last_peered":"2024-03-28T23:36:08.129755+0000","last_clean":"2024-03-28T23:36:08.129755+0000","last_became_active":"2024-03-28T23:36:07.111180+0000","last_became_peered":"2024-03-28T23:36:07.111180+0000","last_unstale":"2024-03-28T23:36:08.129755+0000","last_undegraded":"2024-03-28T23:36:08.129755+0000","last_fullsized":"2024-03-28T23:36:08.129755+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:33.319869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117521+0000","last_change":"2024-03-28T23:35:59.074941+0000","last_active":"2024-03-28T23:36:09.117521+0000","last_peered":"2024-03-28T23:36:09.117521+0000","last_clean":"2024-03-28T23:36:09.117521+0000","last_became_active":"2024-03-28T23:35:59.074636+0000","last_became_peered":"2024-03-28T23:35:59.074636+0000","last_unstale":"2024-03-28T23:36:09.117521+0000","last_undegraded":"2024-03-28T23:36:09.117521+0000","last_fullsized":"2024-03-28T23:36:09.117521+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:09.061218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"151'6","reported_seq":28,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109577+0000","last_change":"2024-03-28T23:35:58.075069+0000","last_active":"2024-03-28T23:36:09.109577+0000","last_peered":"2024-03-28T23:36:09.109577+0000","last_clean":"2024-03-28T23:36:09.109577+0000","last_became_active":"2024-03-28T23:35:58.074795+0000","last_became_peered":"2024-03-28T23:35:58.074795+0000","last_unstale":"2024-03-28T23:36:09.109577+0000","last_undegraded":"2024-03-28T23:36:09.109577+0000","last_fullsized":"2024-03-28T23:36:09.109577+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:49:07.831860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109150+0000","last_change":"2024-03-28T23:36:08.108153+0000","last_active":"2024-03-28T23:36:09.109150+0000","last_peered":"2024-03-28T23:36:09.109150+0000","last_clean":"2024-03-28T23:36:09.109150+0000","last_became_active":"2024-03-28T23:36:08.107857+0000","last_became_peered":"2024-03-28T23:36:08.107857+0000","last_unstale":"2024-03-28T23:36:09.109150+0000","last_undegraded":"2024-03-28T23:36:09.109150+0000","last_fullsized":"2024-03-28T23:36:09.109150+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:34.567279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109214+0000","last_change":"2024-03-28T23:36:07.099721+0000","last_active":"2024-03-28T23:36:09.109214+0000","last_peered":"2024-03-28T23:36:09.109214+0000","last_clean":"2024-03-28T23:36:09.109214+0000","last_became_active":"2024-03-28T23:36:07.099471+0000","last_became_peered":"2024-03-28T23:36:07.099471+0000","last_unstale":"2024-03-28T23:36:09.109214+0000","last_undegraded":"2024-03-28T23:36:09.109214+0000","last_fullsized":"2024-03-28T23:36:09.109214+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:37:18.587587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.099051+0000","last_change":"2024-03-28T23:35:59.076575+0000","last_active":"2024-03-28T23:36:08.099051+0000","last_peered":"2024-03-28T23:36:08.099051+0000","last_clean":"2024-03-28T23:36:08.099051+0000","last_became_active":"2024-03-28T23:35:59.076319+0000","last_became_peered":"2024-03-28T23:35:59.076319+0000","last_unstale":"2024-03-28T23:36:08.099051+0000","last_undegraded":"2024-03-28T23:36:08.099051+0000","last_fullsized":"2024-03-28T23:36:08.099051+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:28:45.099306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"151'2","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.108854+0000","last_change":"2024-03-28T23:35:58.073277+0000","last_active":"2024-03-28T23:36:08.108854+0000","last_peered":"2024-03-28T23:36:08.108854+0000","last_clean":"2024-03-28T23:36:08.108854+0000","last_became_active":"2024-03-28T23:35:58.071396+0000","last_became_peered":"2024-03-28T23:35:58.071396+0000","last_unstale":"2024-03-28T23:36:08.108854+0000","last_undegraded":"2024-03-28T23:36:08.108854+0000","last_fullsized":"2024-03-28T23:36:08.108854+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:36:25.234053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.190384+0000","last_change":"2024-03-28T23:36:08.188439+0000","last_active":"2024-03-28T23:36:08.190384+0000","last_peered":"2024-03-28T23:36:08.190384+0000","last_clean":"2024-03-28T23:36:08.190384+0000","last_became_active":"2024-03-28T23:36:08.187657+0000","last_became_peered":"2024-03-28T23:36:08.187657+0000","last_unstale":"2024-03-28T23:36:08.190384+0000","last_undegraded":"2024-03-28T23:36:08.190384+0000","last_fullsized":"2024-03-28T23:36:08.190384+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:22:52.579163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.123985+0000","last_change":"2024-03-28T23:36:07.097646+0000","last_active":"2024-03-28T23:36:08.123985+0000","last_peered":"2024-03-28T23:36:08.123985+0000","last_clean":"2024-03-28T23:36:08.123985+0000","last_became_active":"2024-03-28T23:36:07.097397+0000","last_became_peered":"2024-03-28T23:36:07.097397+0000","last_unstale":"2024-03-28T23:36:08.123985+0000","last_undegraded":"2024-03-28T23:36:08.123985+0000","last_fullsized":"2024-03-28T23:36:08.123985+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:20:17.759273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120104+0000","last_change":"2024-03-28T23:35:59.072656+0000","last_active":"2024-03-28T23:36:09.120104+0000","last_peered":"2024-03-28T23:36:09.120104+0000","last_clean":"2024-03-28T23:36:09.120104+0000","last_became_active":"2024-03-28T23:35:59.072400+0000","last_became_peered":"2024-03-28T23:35:59.072400+0000","last_unstale":"2024-03-28T23:36:09.120104+0000","last_undegraded":"2024-03-28T23:36:09.120104+0000","last_fullsized":"2024-03-28T23:36:09.120104+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:54:04.050429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109212+0000","last_change":"2024-03-28T23:35:58.071626+0000","last_active":"2024-03-28T23:36:09.109212+0000","last_peered":"2024-03-28T23:36:09.109212+0000","last_clean":"2024-03-28T23:36:09.109212+0000","last_became_active":"2024-03-28T23:35:58.071339+0000","last_became_peered":"2024-03-28T23:35:58.071339+0000","last_unstale":"2024-03-28T23:36:09.109212+0000","last_undegraded":"2024-03-28T23:36:09.109212+0000","last_fullsized":"2024-03-28T23:36:09.109212+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:02:10.808877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109855+0000","last_change":"2024-03-28T23:36:08.119583+0000","last_active":"2024-03-28T23:36:09.109855+0000","last_peered":"2024-03-28T23:36:09.109855+0000","last_clean":"2024-03-28T23:36:09.109855+0000","last_became_active":"2024-03-28T23:36:08.119334+0000","last_became_peered":"2024-03-28T23:36:08.119334+0000","last_unstale":"2024-03-28T23:36:09.109855+0000","last_undegraded":"2024-03-28T23:36:09.109855+0000","last_fullsized":"2024-03-28T23:36:09.109855+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:30:32.302845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"155'4","reported_seq":16,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.130565+0000","last_change":"2024-03-28T23:36:07.101042+0000","last_active":"2024-03-28T23:36:08.130565+0000","last_peered":"2024-03-28T23:36:08.130565+0000","last_clean":"2024-03-28T23:36:08.130565+0000","last_became_active":"2024-03-28T23:36:07.100702+0000","last_became_peered":"2024-03-28T23:36:07.100702+0000","last_unstale":"2024-03-28T23:36:08.130565+0000","last_undegraded":"2024-03-28T23:36:08.130565+0000","last_fullsized":"2024-03-28T23:36:08.130565+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:26:22.530062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109482+0000","last_change":"2024-03-28T23:35:59.076190+0000","last_active":"2024-03-28T23:36:09.109482+0000","last_peered":"2024-03-28T23:36:09.109482+0000","last_clean":"2024-03-28T23:36:09.109482+0000","last_became_active":"2024-03-28T23:35:59.075881+0000","last_became_peered":"2024-03-28T23:35:59.075881+0000","last_unstale":"2024-03-28T23:36:09.109482+0000","last_undegraded":"2024-03-28T23:36:09.109482+0000","last_fullsized":"2024-03-28T23:36:09.109482+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:50:09.386042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108479+0000","last_change":"2024-03-28T23:35:58.062542+0000","last_active":"2024-03-28T23:36:09.108479+0000","last_peered":"2024-03-28T23:36:09.108479+0000","last_clean":"2024-03-28T23:36:09.108479+0000","last_became_active":"2024-03-28T23:35:58.062074+0000","last_became_peered":"2024-03-28T23:35:58.062074+0000","last_unstale":"2024-03-28T23:36:09.108479+0000","last_undegraded":"2024-03-28T23:36:09.108479+0000","last_fullsized":"2024-03-28T23:36:09.108479+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:23:41.041246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117942+0000","last_change":"2024-03-28T23:36:08.111324+0000","last_active":"2024-03-28T23:36:09.117942+0000","last_peered":"2024-03-28T23:36:09.117942+0000","last_clean":"2024-03-28T23:36:09.117942+0000","last_became_active":"2024-03-28T23:36:08.111028+0000","last_became_peered":"2024-03-28T23:36:08.111028+0000","last_unstale":"2024-03-28T23:36:09.117942+0000","last_undegraded":"2024-03-28T23:36:09.117942+0000","last_fullsized":"2024-03-28T23:36:09.117942+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:25:43.622638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109968+0000","last_change":"2024-03-28T23:36:07.108343+0000","last_active":"2024-03-28T23:36:09.109968+0000","last_peered":"2024-03-28T23:36:09.109968+0000","last_clean":"2024-03-28T23:36:09.109968+0000","last_became_active":"2024-03-28T23:36:07.108078+0000","last_became_peered":"2024-03-28T23:36:07.108078+0000","last_unstale":"2024-03-28T23:36:09.109968+0000","last_undegraded":"2024-03-28T23:36:09.109968+0000","last_fullsized":"2024-03-28T23:36:09.109968+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:26:22.530062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.118076+0000","last_change":"2024-03-28T23:35:59.069831+0000","last_active":"2024-03-28T23:36:09.118076+0000","last_peered":"2024-03-28T23:36:09.118076+0000","last_clean":"2024-03-28T23:36:09.118076+0000","last_became_active":"2024-03-28T23:35:59.069272+0000","last_became_peered":"2024-03-28T23:35:59.069272+0000","last_unstale":"2024-03-28T23:36:09.118076+0000","last_undegraded":"2024-03-28T23:36:09.118076+0000","last_fullsized":"2024-03-28T23:36:09.118076+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:56:35.966156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"151'8","reported_seq":30,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119657+0000","last_change":"2024-03-28T23:35:58.071172+0000","last_active":"2024-03-28T23:36:09.119657+0000","last_peered":"2024-03-28T23:36:09.119657+0000","last_clean":"2024-03-28T23:36:09.119657+0000","last_became_active":"2024-03-28T23:35:58.070901+0000","last_became_peered":"2024-03-28T23:35:58.070901+0000","last_unstale":"2024-03-28T23:36:09.119657+0000","last_undegraded":"2024-03-28T23:36:09.119657+0000","last_fullsized":"2024-03-28T23:36:09.119657+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:16:53.124176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120049+0000","last_change":"2024-03-28T23:36:08.119458+0000","last_active":"2024-03-28T23:36:09.120049+0000","last_peered":"2024-03-28T23:36:09.120049+0000","last_clean":"2024-03-28T23:36:09.120049+0000","last_became_active":"2024-03-28T23:36:08.119179+0000","last_became_peered":"2024-03-28T23:36:08.119179+0000","last_unstale":"2024-03-28T23:36:09.120049+0000","last_undegraded":"2024-03-28T23:36:09.120049+0000","last_fullsized":"2024-03-28T23:36:09.120049+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:34.323711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117398+0000","last_change":"2024-03-28T23:36:07.098526+0000","last_active":"2024-03-28T23:36:09.117398+0000","last_peered":"2024-03-28T23:36:09.117398+0000","last_clean":"2024-03-28T23:36:09.117398+0000","last_became_active":"2024-03-28T23:36:07.098254+0000","last_became_peered":"2024-03-28T23:36:07.098254+0000","last_unstale":"2024-03-28T23:36:09.117398+0000","last_undegraded":"2024-03-28T23:36:09.117398+0000","last_fullsized":"2024-03-28T23:36:09.117398+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:28:53.134172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119537+0000","last_change":"2024-03-28T23:35:59.067559+0000","last_active":"2024-03-28T23:36:09.119537+0000","last_peered":"2024-03-28T23:36:09.119537+0000","last_clean":"2024-03-28T23:36:09.119537+0000","last_became_active":"2024-03-28T23:35:59.067265+0000","last_became_peered":"2024-03-28T23:35:59.067265+0000","last_unstale":"2024-03-28T23:36:09.119537+0000","last_undegraded":"2024-03-28T23:36:09.119537+0000","last_fullsized":"2024-03-28T23:36:09.119537+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:32.347982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"151'1","reported_seq":23,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.110247+0000","last_change":"2024-03-28T23:35:58.064795+0000","last_active":"2024-03-28T23:36:09.110247+0000","last_peered":"2024-03-28T23:36:09.110247+0000","last_clean":"2024-03-28T23:36:09.110247+0000","last_became_active":"2024-03-28T23:35:58.064506+0000","last_became_peered":"2024-03-28T23:35:58.064506+0000","last_unstale":"2024-03-28T23:36:09.110247+0000","last_undegraded":"2024-03-28T23:36:09.110247+0000","last_fullsized":"2024-03-28T23:36:09.110247+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:36:25.234053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108780+0000","last_change":"2024-03-28T23:36:08.114918+0000","last_active":"2024-03-28T23:36:09.108780+0000","last_peered":"2024-03-28T23:36:09.108780+0000","last_clean":"2024-03-28T23:36:09.108780+0000","last_became_active":"2024-03-28T23:36:08.114663+0000","last_became_peered":"2024-03-28T23:36:08.114663+0000","last_unstale":"2024-03-28T23:36:09.108780+0000","last_undegraded":"2024-03-28T23:36:09.108780+0000","last_fullsized":"2024-03-28T23:36:09.108780+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:56.120451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.187957+0000","last_change":"2024-03-28T23:36:07.113899+0000","last_active":"2024-03-28T23:36:08.187957+0000","last_peered":"2024-03-28T23:36:08.187957+0000","last_clean":"2024-03-28T23:36:08.187957+0000","last_became_active":"2024-03-28T23:36:07.113678+0000","last_became_peered":"2024-03-28T23:36:07.113678+0000","last_unstale":"2024-03-28T23:36:08.187957+0000","last_undegraded":"2024-03-28T23:36:08.187957+0000","last_fullsized":"2024-03-28T23:36:08.187957+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:27:04.119180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.187228+0000","last_change":"2024-03-28T23:35:59.075241+0000","last_active":"2024-03-28T23:36:08.187228+0000","last_peered":"2024-03-28T23:36:08.187228+0000","last_clean":"2024-03-28T23:36:08.187228+0000","last_became_active":"2024-03-28T23:35:59.074170+0000","last_became_peered":"2024-03-28T23:35:59.074170+0000","last_unstale":"2024-03-28T23:36:08.187228+0000","last_undegraded":"2024-03-28T23:36:08.187228+0000","last_fullsized":"2024-03-28T23:36:08.187228+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:16.910736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"151'2","reported_seq":24,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120382+0000","last_change":"2024-03-28T23:35:58.070169+0000","last_active":"2024-03-28T23:36:09.120382+0000","last_peered":"2024-03-28T23:36:09.120382+0000","last_clean":"2024-03-28T23:36:09.120382+0000","last_became_active":"2024-03-28T23:35:58.069913+0000","last_became_peered":"2024-03-28T23:35:58.069913+0000","last_unstale":"2024-03-28T23:36:09.120382+0000","last_undegraded":"2024-03-28T23:36:09.120382+0000","last_fullsized":"2024-03-28T23:36:09.120382+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:22:07.300796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.192034+0000","last_change":"2024-03-28T23:36:08.190757+0000","last_active":"2024-03-28T23:36:08.192034+0000","last_peered":"2024-03-28T23:36:08.192034+0000","last_clean":"2024-03-28T23:36:08.192034+0000","last_became_active":"2024-03-28T23:36:08.190491+0000","last_became_peered":"2024-03-28T23:36:08.190491+0000","last_unstale":"2024-03-28T23:36:08.192034+0000","last_undegraded":"2024-03-28T23:36:08.192034+0000","last_fullsized":"2024-03-28T23:36:08.192034+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:32.608667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108894+0000","last_change":"2024-03-28T23:36:07.103900+0000","last_active":"2024-03-28T23:36:09.108894+0000","last_peered":"2024-03-28T23:36:09.108894+0000","last_clean":"2024-03-28T23:36:09.108894+0000","last_became_active":"2024-03-28T23:36:07.103589+0000","last_became_peered":"2024-03-28T23:36:07.103589+0000","last_unstale":"2024-03-28T23:36:09.108894+0000","last_undegraded":"2024-03-28T23:36:09.108894+0000","last_fullsized":"2024-03-28T23:36:09.108894+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:08:38.942276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108337+0000","last_change":"2024-03-28T23:35:59.081763+0000","last_active":"2024-03-28T23:36:09.108337+0000","last_peered":"2024-03-28T23:36:09.108337+0000","last_clean":"2024-03-28T23:36:09.108337+0000","last_became_active":"2024-03-28T23:35:59.081465+0000","last_became_peered":"2024-03-28T23:35:59.081465+0000","last_unstale":"2024-03-28T23:36:09.108337+0000","last_undegraded":"2024-03-28T23:36:09.108337+0000","last_fullsized":"2024-03-28T23:36:09.108337+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:06:37.212712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"151'1","reported_seq":21,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.109458+0000","last_change":"2024-03-28T23:35:58.070308+0000","last_active":"2024-03-28T23:36:08.109458+0000","last_peered":"2024-03-28T23:36:08.109458+0000","last_clean":"2024-03-28T23:36:08.109458+0000","last_became_active":"2024-03-28T23:35:58.070024+0000","last_became_peered":"2024-03-28T23:35:58.070024+0000","last_unstale":"2024-03-28T23:36:08.109458+0000","last_undegraded":"2024-03-28T23:36:08.109458+0000","last_fullsized":"2024-03-28T23:36:08.109458+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:49:07.831860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.191661+0000","last_change":"2024-03-28T23:36:08.190948+0000","last_active":"2024-03-28T23:36:08.191661+0000","last_peered":"2024-03-28T23:36:08.191661+0000","last_clean":"2024-03-28T23:36:08.191661+0000","last_became_active":"2024-03-28T23:36:08.190659+0000","last_became_peered":"2024-03-28T23:36:08.190659+0000","last_unstale":"2024-03-28T23:36:08.191661+0000","last_undegraded":"2024-03-28T23:36:08.191661+0000","last_fullsized":"2024-03-28T23:36:08.191661+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:51:15.427349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108653+0000","last_change":"2024-03-28T23:36:07.110907+0000","last_active":"2024-03-28T23:36:09.108653+0000","last_peered":"2024-03-28T23:36:09.108653+0000","last_clean":"2024-03-28T23:36:09.108653+0000","last_became_active":"2024-03-28T23:36:07.110651+0000","last_became_peered":"2024-03-28T23:36:07.110651+0000","last_unstale":"2024-03-28T23:36:09.108653+0000","last_undegraded":"2024-03-28T23:36:09.108653+0000","last_fullsized":"2024-03-28T23:36:09.108653+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:06:16.270793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108755+0000","last_change":"2024-03-28T23:35:59.075326+0000","last_active":"2024-03-28T23:36:09.108755+0000","last_peered":"2024-03-28T23:36:09.108755+0000","last_clean":"2024-03-28T23:36:09.108755+0000","last_became_active":"2024-03-28T23:35:59.075063+0000","last_became_peered":"2024-03-28T23:35:59.075063+0000","last_unstale":"2024-03-28T23:36:09.108755+0000","last_undegraded":"2024-03-28T23:36:09.108755+0000","last_fullsized":"2024-03-28T23:36:09.108755+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:42:41.721990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120144+0000","last_change":"2024-03-28T23:35:58.073249+0000","last_active":"2024-03-28T23:36:09.120144+0000","last_peered":"2024-03-28T23:36:09.120144+0000","last_clean":"2024-03-28T23:36:09.120144+0000","last_became_active":"2024-03-28T23:35:58.072992+0000","last_became_peered":"2024-03-28T23:35:58.072992+0000","last_unstale":"2024-03-28T23:36:09.120144+0000","last_undegraded":"2024-03-28T23:36:09.120144+0000","last_fullsized":"2024-03-28T23:36:09.120144+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:58:36.456500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109797+0000","last_change":"2024-03-28T23:36:08.190992+0000","last_active":"2024-03-28T23:36:09.109797+0000","last_peered":"2024-03-28T23:36:09.109797+0000","last_clean":"2024-03-28T23:36:09.109797+0000","last_became_active":"2024-03-28T23:36:08.190725+0000","last_became_peered":"2024-03-28T23:36:08.190725+0000","last_unstale":"2024-03-28T23:36:09.109797+0000","last_undegraded":"2024-03-28T23:36:09.109797+0000","last_fullsized":"2024-03-28T23:36:09.109797+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:42:54.376192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109850+0000","last_change":"2024-03-28T23:36:07.110460+0000","last_active":"2024-03-28T23:36:09.109850+0000","last_peered":"2024-03-28T23:36:09.109850+0000","last_clean":"2024-03-28T23:36:09.109850+0000","last_became_active":"2024-03-28T23:36:07.110195+0000","last_became_peered":"2024-03-28T23:36:07.110195+0000","last_unstale":"2024-03-28T23:36:09.109850+0000","last_undegraded":"2024-03-28T23:36:09.109850+0000","last_fullsized":"2024-03-28T23:36:09.109850+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:39:56.430484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109471+0000","last_change":"2024-03-28T23:35:59.082001+0000","last_active":"2024-03-28T23:36:09.109471+0000","last_peered":"2024-03-28T23:36:09.109471+0000","last_clean":"2024-03-28T23:36:09.109471+0000","last_became_active":"2024-03-28T23:35:59.081706+0000","last_became_peered":"2024-03-28T23:35:59.081706+0000","last_unstale":"2024-03-28T23:36:09.109471+0000","last_undegraded":"2024-03-28T23:36:09.109471+0000","last_fullsized":"2024-03-28T23:36:09.109471+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:42:45.337484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"151'2","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.108957+0000","last_change":"2024-03-28T23:35:58.072448+0000","last_active":"2024-03-28T23:36:08.108957+0000","last_peered":"2024-03-28T23:36:08.108957+0000","last_clean":"2024-03-28T23:36:08.108957+0000","last_became_active":"2024-03-28T23:35:58.071494+0000","last_became_peered":"2024-03-28T23:35:58.071494+0000","last_unstale":"2024-03-28T23:36:08.108957+0000","last_undegraded":"2024-03-28T23:36:08.108957+0000","last_fullsized":"2024-03-28T23:36:08.108957+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:54:03.045194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.118018+0000","last_change":"2024-03-28T23:36:08.108038+0000","last_active":"2024-03-28T23:36:09.118018+0000","last_peered":"2024-03-28T23:36:09.118018+0000","last_clean":"2024-03-28T23:36:09.118018+0000","last_became_active":"2024-03-28T23:36:08.107752+0000","last_became_peered":"2024-03-28T23:36:08.107752+0000","last_unstale":"2024-03-28T23:36:09.118018+0000","last_undegraded":"2024-03-28T23:36:09.118018+0000","last_fullsized":"2024-03-28T23:36:09.118018+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:03:37.546835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109569+0000","last_change":"2024-03-28T23:36:07.114878+0000","last_active":"2024-03-28T23:36:09.109569+0000","last_peered":"2024-03-28T23:36:09.109569+0000","last_clean":"2024-03-28T23:36:09.109569+0000","last_became_active":"2024-03-28T23:36:07.114586+0000","last_became_peered":"2024-03-28T23:36:07.114586+0000","last_unstale":"2024-03-28T23:36:09.109569+0000","last_undegraded":"2024-03-28T23:36:09.109569+0000","last_fullsized":"2024-03-28T23:36:09.109569+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:21:50.427323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.109362+0000","last_change":"2024-03-28T23:35:59.080948+0000","last_active":"2024-03-28T23:36:08.109362+0000","last_peered":"2024-03-28T23:36:08.109362+0000","last_clean":"2024-03-28T23:36:08.109362+0000","last_became_active":"2024-03-28T23:35:59.080672+0000","last_became_peered":"2024-03-28T23:35:59.080672+0000","last_unstale":"2024-03-28T23:36:08.109362+0000","last_undegraded":"2024-03-28T23:36:08.109362+0000","last_fullsized":"2024-03-28T23:36:08.109362+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:39:27.912295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108487+0000","last_change":"2024-03-28T23:35:58.071966+0000","last_active":"2024-03-28T23:36:09.108487+0000","last_peered":"2024-03-28T23:36:09.108487+0000","last_clean":"2024-03-28T23:36:09.108487+0000","last_became_active":"2024-03-28T23:35:58.071656+0000","last_became_peered":"2024-03-28T23:35:58.071656+0000","last_unstale":"2024-03-28T23:36:09.108487+0000","last_undegraded":"2024-03-28T23:36:09.108487+0000","last_fullsized":"2024-03-28T23:36:09.108487+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:53:34.853174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117583+0000","last_change":"2024-03-28T23:36:08.108836+0000","last_active":"2024-03-28T23:36:09.117583+0000","last_peered":"2024-03-28T23:36:09.117583+0000","last_clean":"2024-03-28T23:36:09.117583+0000","last_became_active":"2024-03-28T23:36:08.108577+0000","last_became_peered":"2024-03-28T23:36:08.108577+0000","last_unstale":"2024-03-28T23:36:09.117583+0000","last_undegraded":"2024-03-28T23:36:09.117583+0000","last_fullsized":"2024-03-28T23:36:09.117583+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:32.608667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.127528+0000","last_change":"2024-03-28T23:36:07.103362+0000","last_active":"2024-03-28T23:36:08.127528+0000","last_peered":"2024-03-28T23:36:08.127528+0000","last_clean":"2024-03-28T23:36:08.127528+0000","last_became_active":"2024-03-28T23:36:07.103086+0000","last_became_peered":"2024-03-28T23:36:07.103086+0000","last_unstale":"2024-03-28T23:36:08.127528+0000","last_undegraded":"2024-03-28T23:36:08.127528+0000","last_fullsized":"2024-03-28T23:36:08.127528+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:13:32.250870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.109402+0000","last_change":"2024-03-28T23:35:59.074451+0000","last_active":"2024-03-28T23:36:08.109402+0000","last_peered":"2024-03-28T23:36:08.109402+0000","last_clean":"2024-03-28T23:36:08.109402+0000","last_became_active":"2024-03-28T23:35:59.074181+0000","last_became_peered":"2024-03-28T23:35:59.074181+0000","last_unstale":"2024-03-28T23:36:08.109402+0000","last_undegraded":"2024-03-28T23:36:08.109402+0000","last_fullsized":"2024-03-28T23:36:08.109402+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:54:41.513429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117696+0000","last_change":"2024-03-28T23:35:58.067234+0000","last_active":"2024-03-28T23:36:09.117696+0000","last_peered":"2024-03-28T23:36:09.117696+0000","last_clean":"2024-03-28T23:36:09.117696+0000","last_became_active":"2024-03-28T23:35:58.066942+0000","last_became_peered":"2024-03-28T23:35:58.066942+0000","last_unstale":"2024-03-28T23:36:09.117696+0000","last_undegraded":"2024-03-28T23:36:09.117696+0000","last_fullsized":"2024-03-28T23:36:09.117696+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:37:03.241869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120158+0000","last_change":"2024-03-28T23:36:08.121754+0000","last_active":"2024-03-28T23:36:09.120158+0000","last_peered":"2024-03-28T23:36:09.120158+0000","last_clean":"2024-03-28T23:36:09.120158+0000","last_became_active":"2024-03-28T23:36:08.121455+0000","last_became_peered":"2024-03-28T23:36:08.121455+0000","last_unstale":"2024-03-28T23:36:09.120158+0000","last_undegraded":"2024-03-28T23:36:09.120158+0000","last_fullsized":"2024-03-28T23:36:09.120158+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:39:47.021555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"156'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119016+0000","last_change":"2024-03-28T23:36:07.112395+0000","last_active":"2024-03-28T23:36:09.119016+0000","last_peered":"2024-03-28T23:36:09.119016+0000","last_clean":"2024-03-28T23:36:09.119016+0000","last_became_active":"2024-03-28T23:36:07.112088+0000","last_became_peered":"2024-03-28T23:36:07.112088+0000","last_unstale":"2024-03-28T23:36:09.119016+0000","last_undegraded":"2024-03-28T23:36:09.119016+0000","last_fullsized":"2024-03-28T23:36:09.119016+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:37.434300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109946+0000","last_change":"2024-03-28T23:35:59.084751+0000","last_active":"2024-03-28T23:36:09.109946+0000","last_peered":"2024-03-28T23:36:09.109946+0000","last_clean":"2024-03-28T23:36:09.109946+0000","last_became_active":"2024-03-28T23:35:59.084472+0000","last_became_peered":"2024-03-28T23:35:59.084472+0000","last_unstale":"2024-03-28T23:36:09.109946+0000","last_undegraded":"2024-03-28T23:36:09.109946+0000","last_fullsized":"2024-03-28T23:36:09.109946+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:17.228750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109252+0000","last_change":"2024-03-28T23:35:58.067542+0000","last_active":"2024-03-28T23:36:09.109252+0000","last_peered":"2024-03-28T23:36:09.109252+0000","last_clean":"2024-03-28T23:36:09.109252+0000","last_became_active":"2024-03-28T23:35:58.067013+0000","last_became_peered":"2024-03-28T23:35:58.067013+0000","last_unstale":"2024-03-28T23:36:09.109252+0000","last_undegraded":"2024-03-28T23:36:09.109252+0000","last_fullsized":"2024-03-28T23:36:09.109252+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:41:31.356630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.119756+0000","last_change":"2024-03-28T23:36:08.116214+0000","last_active":"2024-03-28T23:36:08.119756+0000","last_peered":"2024-03-28T23:36:08.119756+0000","last_clean":"2024-03-28T23:36:08.119756+0000","last_became_active":"2024-03-28T23:36:08.115064+0000","last_became_peered":"2024-03-28T23:36:08.115064+0000","last_unstale":"2024-03-28T23:36:08.119756+0000","last_undegraded":"2024-03-28T23:36:08.119756+0000","last_fullsized":"2024-03-28T23:36:08.119756+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:09:57.210214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117327+0000","last_change":"2024-03-28T23:36:07.109162+0000","last_active":"2024-03-28T23:36:09.117327+0000","last_peered":"2024-03-28T23:36:09.117327+0000","last_clean":"2024-03-28T23:36:09.117327+0000","last_became_active":"2024-03-28T23:36:07.108846+0000","last_became_peered":"2024-03-28T23:36:07.108846+0000","last_unstale":"2024-03-28T23:36:09.117327+0000","last_undegraded":"2024-03-28T23:36:09.117327+0000","last_fullsized":"2024-03-28T23:36:09.117327+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:24.945602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108490+0000","last_change":"2024-03-28T23:35:59.077874+0000","last_active":"2024-03-28T23:36:09.108490+0000","last_peered":"2024-03-28T23:36:09.108490+0000","last_clean":"2024-03-28T23:36:09.108490+0000","last_became_active":"2024-03-28T23:35:59.077613+0000","last_became_peered":"2024-03-28T23:35:59.077613+0000","last_unstale":"2024-03-28T23:36:09.108490+0000","last_undegraded":"2024-03-28T23:36:09.108490+0000","last_fullsized":"2024-03-28T23:36:09.108490+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:41:12.571144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"151'2","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.110233+0000","last_change":"2024-03-28T23:35:58.066436+0000","last_active":"2024-03-28T23:36:08.110233+0000","last_peered":"2024-03-28T23:36:08.110233+0000","last_clean":"2024-03-28T23:36:08.110233+0000","last_became_active":"2024-03-28T23:35:58.063834+0000","last_became_peered":"2024-03-28T23:35:58.063834+0000","last_unstale":"2024-03-28T23:36:08.110233+0000","last_undegraded":"2024-03-28T23:36:08.110233+0000","last_fullsized":"2024-03-28T23:36:08.110233+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:37:09.547486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109600+0000","last_change":"2024-03-28T23:36:08.122302+0000","last_active":"2024-03-28T23:36:09.109600+0000","last_peered":"2024-03-28T23:36:09.109600+0000","last_clean":"2024-03-28T23:36:09.109600+0000","last_became_active":"2024-03-28T23:36:08.122059+0000","last_became_peered":"2024-03-28T23:36:08.122059+0000","last_unstale":"2024-03-28T23:36:09.109600+0000","last_undegraded":"2024-03-28T23:36:09.109600+0000","last_fullsized":"2024-03-28T23:36:09.109600+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:24:51.260030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.188657+0000","last_change":"2024-03-28T23:36:07.099612+0000","last_active":"2024-03-28T23:36:08.188657+0000","last_peered":"2024-03-28T23:36:08.188657+0000","last_clean":"2024-03-28T23:36:08.188657+0000","last_became_active":"2024-03-28T23:36:07.099349+0000","last_became_peered":"2024-03-28T23:36:07.099349+0000","last_unstale":"2024-03-28T23:36:08.188657+0000","last_undegraded":"2024-03-28T23:36:08.188657+0000","last_fullsized":"2024-03-28T23:36:08.188657+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:13:34.084552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119706+0000","last_change":"2024-03-28T23:35:59.074166+0000","last_active":"2024-03-28T23:36:09.119706+0000","last_peered":"2024-03-28T23:36:09.119706+0000","last_clean":"2024-03-28T23:36:09.119706+0000","last_became_active":"2024-03-28T23:35:59.073916+0000","last_became_peered":"2024-03-28T23:35:59.073916+0000","last_unstale":"2024-03-28T23:36:09.119706+0000","last_undegraded":"2024-03-28T23:36:09.119706+0000","last_fullsized":"2024-03-28T23:36:09.119706+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:24.837863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"152'1","reported_seq":23,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.110078+0000","last_change":"2024-03-28T23:35:58.078010+0000","last_active":"2024-03-28T23:36:09.110078+0000","last_peered":"2024-03-28T23:36:09.110078+0000","last_clean":"2024-03-28T23:36:09.110078+0000","last_became_active":"2024-03-28T23:35:58.077736+0000","last_became_peered":"2024-03-28T23:35:58.077736+0000","last_unstale":"2024-03-28T23:36:09.110078+0000","last_undegraded":"2024-03-28T23:36:09.110078+0000","last_fullsized":"2024-03-28T23:36:09.110078+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:54:03.045194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119474+0000","last_change":"2024-03-28T23:36:08.120046+0000","last_active":"2024-03-28T23:36:09.119474+0000","last_peered":"2024-03-28T23:36:09.119474+0000","last_clean":"2024-03-28T23:36:09.119474+0000","last_became_active":"2024-03-28T23:36:08.119656+0000","last_became_peered":"2024-03-28T23:36:08.119656+0000","last_unstale":"2024-03-28T23:36:09.119474+0000","last_undegraded":"2024-03-28T23:36:09.119474+0000","last_fullsized":"2024-03-28T23:36:09.119474+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:21:23.239166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.126994+0000","last_change":"2024-03-28T23:36:07.111369+0000","last_active":"2024-03-28T23:36:08.126994+0000","last_peered":"2024-03-28T23:36:08.126994+0000","last_clean":"2024-03-28T23:36:08.126994+0000","last_became_active":"2024-03-28T23:36:07.111103+0000","last_became_peered":"2024-03-28T23:36:07.111103+0000","last_unstale":"2024-03-28T23:36:08.126994+0000","last_undegraded":"2024-03-28T23:36:08.126994+0000","last_fullsized":"2024-03-28T23:36:08.126994+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:50:17.420909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117513+0000","last_change":"2024-03-28T23:35:59.062460+0000","last_active":"2024-03-28T23:36:09.117513+0000","last_peered":"2024-03-28T23:36:09.117513+0000","last_clean":"2024-03-28T23:36:09.117513+0000","last_became_active":"2024-03-28T23:35:59.062185+0000","last_became_peered":"2024-03-28T23:35:59.062185+0000","last_unstale":"2024-03-28T23:36:09.117513+0000","last_undegraded":"2024-03-28T23:36:09.117513+0000","last_fullsized":"2024-03-28T23:36:09.117513+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:05:51.224827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109998+0000","last_change":"2024-03-28T23:35:58.072356+0000","last_active":"2024-03-28T23:36:09.109998+0000","last_peered":"2024-03-28T23:36:09.109998+0000","last_clean":"2024-03-28T23:36:09.109998+0000","last_became_active":"2024-03-28T23:35:58.072111+0000","last_became_peered":"2024-03-28T23:35:58.072111+0000","last_unstale":"2024-03-28T23:36:09.109998+0000","last_undegraded":"2024-03-28T23:36:09.109998+0000","last_fullsized":"2024-03-28T23:36:09.109998+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:23.832628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.119081+0000","last_change":"2024-03-28T23:36:08.115504+0000","last_active":"2024-03-28T23:36:08.119081+0000","last_peered":"2024-03-28T23:36:08.119081+0000","last_clean":"2024-03-28T23:36:08.119081+0000","last_became_active":"2024-03-28T23:36:08.115249+0000","last_became_peered":"2024-03-28T23:36:08.115249+0000","last_unstale":"2024-03-28T23:36:08.119081+0000","last_undegraded":"2024-03-28T23:36:08.119081+0000","last_fullsized":"2024-03-28T23:36:08.119081+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:24:21.264344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.110229+0000","last_change":"2024-03-28T23:36:07.105646+0000","last_active":"2024-03-28T23:36:08.110229+0000","last_peered":"2024-03-28T23:36:08.110229+0000","last_clean":"2024-03-28T23:36:08.110229+0000","last_became_active":"2024-03-28T23:36:07.105391+0000","last_became_peered":"2024-03-28T23:36:07.105391+0000","last_unstale":"2024-03-28T23:36:08.110229+0000","last_undegraded":"2024-03-28T23:36:08.110229+0000","last_fullsized":"2024-03-28T23:36:08.110229+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:39:56.430484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.119805+0000","last_change":"2024-03-28T23:35:59.079950+0000","last_active":"2024-03-28T23:36:09.119805+0000","last_peered":"2024-03-28T23:36:09.119805+0000","last_clean":"2024-03-28T23:36:09.119805+0000","last_became_active":"2024-03-28T23:35:59.079675+0000","last_became_peered":"2024-03-28T23:35:59.079675+0000","last_unstale":"2024-03-28T23:36:09.119805+0000","last_undegraded":"2024-03-28T23:36:09.119805+0000","last_fullsized":"2024-03-28T23:36:09.119805+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:36:26.239288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117843+0000","last_change":"2024-03-28T23:35:58.068737+0000","last_active":"2024-03-28T23:36:09.117843+0000","last_peered":"2024-03-28T23:36:09.117843+0000","last_clean":"2024-03-28T23:36:09.117843+0000","last_became_active":"2024-03-28T23:35:58.068481+0000","last_became_peered":"2024-03-28T23:35:58.068481+0000","last_unstale":"2024-03-28T23:36:09.117843+0000","last_undegraded":"2024-03-28T23:36:09.117843+0000","last_fullsized":"2024-03-28T23:36:09.117843+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:05:05.111519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109331+0000","last_change":"2024-03-28T23:36:08.117859+0000","last_active":"2024-03-28T23:36:09.109331+0000","last_peered":"2024-03-28T23:36:09.109331+0000","last_clean":"2024-03-28T23:36:09.109331+0000","last_became_active":"2024-03-28T23:36:08.117178+0000","last_became_peered":"2024-03-28T23:36:08.117178+0000","last_unstale":"2024-03-28T23:36:09.109331+0000","last_undegraded":"2024-03-28T23:36:09.109331+0000","last_fullsized":"2024-03-28T23:36:09.109331+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:30:42.818702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"155'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.191075+0000","last_change":"2024-03-28T23:36:07.115327+0000","last_active":"2024-03-28T23:36:08.191075+0000","last_peered":"2024-03-28T23:36:08.191075+0000","last_clean":"2024-03-28T23:36:08.191075+0000","last_became_active":"2024-03-28T23:36:07.115138+0000","last_became_peered":"2024-03-28T23:36:07.115138+0000","last_unstale":"2024-03-28T23:36:08.191075+0000","last_undegraded":"2024-03-28T23:36:08.191075+0000","last_fullsized":"2024-03-28T23:36:08.191075+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:02:47.373151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.106013+0000","last_change":"2024-03-28T23:35:59.077779+0000","last_active":"2024-03-28T23:36:08.106013+0000","last_peered":"2024-03-28T23:36:08.106013+0000","last_clean":"2024-03-28T23:36:08.106013+0000","last_became_active":"2024-03-28T23:35:59.077529+0000","last_became_peered":"2024-03-28T23:35:59.077529+0000","last_unstale":"2024-03-28T23:36:08.106013+0000","last_undegraded":"2024-03-28T23:36:08.106013+0000","last_fullsized":"2024-03-28T23:36:08.106013+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:23:42.046480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.104848+0000","last_change":"2024-03-28T23:35:58.072655+0000","last_active":"2024-03-28T23:36:08.104848+0000","last_peered":"2024-03-28T23:36:08.104848+0000","last_clean":"2024-03-28T23:36:08.104848+0000","last_became_active":"2024-03-28T23:35:58.072379+0000","last_became_peered":"2024-03-28T23:35:58.072379+0000","last_unstale":"2024-03-28T23:36:08.104848+0000","last_undegraded":"2024-03-28T23:36:08.104848+0000","last_fullsized":"2024-03-28T23:36:08.104848+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:57:05.274295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.120498+0000","last_change":"2024-03-28T23:36:08.125170+0000","last_active":"2024-03-28T23:36:09.120498+0000","last_peered":"2024-03-28T23:36:09.120498+0000","last_clean":"2024-03-28T23:36:09.120498+0000","last_became_active":"2024-03-28T23:36:08.124892+0000","last_became_peered":"2024-03-28T23:36:08.124892+0000","last_unstale":"2024-03-28T23:36:09.120498+0000","last_undegraded":"2024-03-28T23:36:09.120498+0000","last_fullsized":"2024-03-28T23:36:09.120498+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:35:36.142805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109742+0000","last_change":"2024-03-28T23:36:07.102970+0000","last_active":"2024-03-28T23:36:09.109742+0000","last_peered":"2024-03-28T23:36:09.109742+0000","last_clean":"2024-03-28T23:36:09.109742+0000","last_became_active":"2024-03-28T23:36:07.102708+0000","last_became_peered":"2024-03-28T23:36:07.102708+0000","last_unstale":"2024-03-28T23:36:09.109742+0000","last_undegraded":"2024-03-28T23:36:09.109742+0000","last_fullsized":"2024-03-28T23:36:09.109742+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:36:34.274154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.110032+0000","last_change":"2024-03-28T23:35:59.080115+0000","last_active":"2024-03-28T23:36:09.110032+0000","last_peered":"2024-03-28T23:36:09.110032+0000","last_clean":"2024-03-28T23:36:09.110032+0000","last_became_active":"2024-03-28T23:35:59.079809+0000","last_became_peered":"2024-03-28T23:35:59.079809+0000","last_unstale":"2024-03-28T23:36:09.110032+0000","last_undegraded":"2024-03-28T23:36:09.110032+0000","last_fullsized":"2024-03-28T23:36:09.110032+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:49:08.837095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109260+0000","last_change":"2024-03-28T23:35:58.071453+0000","last_active":"2024-03-28T23:36:09.109260+0000","last_peered":"2024-03-28T23:36:09.109260+0000","last_clean":"2024-03-28T23:36:09.109260+0000","last_became_active":"2024-03-28T23:35:58.071072+0000","last_became_peered":"2024-03-28T23:35:58.071072+0000","last_unstale":"2024-03-28T23:36:09.109260+0000","last_undegraded":"2024-03-28T23:36:09.109260+0000","last_fullsized":"2024-03-28T23:36:09.109260+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:20:20.872070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.117786+0000","last_change":"2024-03-28T23:36:08.127731+0000","last_active":"2024-03-28T23:36:09.117786+0000","last_peered":"2024-03-28T23:36:09.117786+0000","last_clean":"2024-03-28T23:36:09.117786+0000","last_became_active":"2024-03-28T23:36:08.127458+0000","last_became_peered":"2024-03-28T23:36:08.127458+0000","last_unstale":"2024-03-28T23:36:09.117786+0000","last_undegraded":"2024-03-28T23:36:09.117786+0000","last_fullsized":"2024-03-28T23:36:09.117786+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T11:27:08.495798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109055+0000","last_change":"2024-03-28T23:36:07.108521+0000","last_active":"2024-03-28T23:36:09.109055+0000","last_peered":"2024-03-28T23:36:09.109055+0000","last_clean":"2024-03-28T23:36:09.109055+0000","last_became_active":"2024-03-28T23:36:07.108225+0000","last_became_peered":"2024-03-28T23:36:07.108225+0000","last_unstale":"2024-03-28T23:36:09.109055+0000","last_undegraded":"2024-03-28T23:36:09.109055+0000","last_fullsized":"2024-03-28T23:36:09.109055+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:32.872729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.110002+0000","last_change":"2024-03-28T23:35:59.081608+0000","last_active":"2024-03-28T23:36:09.110002+0000","last_peered":"2024-03-28T23:36:09.110002+0000","last_clean":"2024-03-28T23:36:09.110002+0000","last_became_active":"2024-03-28T23:35:59.080839+0000","last_became_peered":"2024-03-28T23:35:59.080839+0000","last_unstale":"2024-03-28T23:36:09.110002+0000","last_undegraded":"2024-03-28T23:36:09.110002+0000","last_fullsized":"2024-03-28T23:36:09.110002+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:46:11.587017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.108567+0000","last_change":"2024-03-28T23:35:58.067740+0000","last_active":"2024-03-28T23:36:09.108567+0000","last_peered":"2024-03-28T23:36:09.108567+0000","last_clean":"2024-03-28T23:36:09.108567+0000","last_became_active":"2024-03-28T23:35:58.067418+0000","last_became_peered":"2024-03-28T23:35:58.067418+0000","last_unstale":"2024-03-28T23:36:09.108567+0000","last_undegraded":"2024-03-28T23:36:09.108567+0000","last_fullsized":"2024-03-28T23:36:09.108567+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:00:58.303444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109755+0000","last_change":"2024-03-28T23:36:08.117608+0000","last_active":"2024-03-28T23:36:09.109755+0000","last_peered":"2024-03-28T23:36:09.109755+0000","last_clean":"2024-03-28T23:36:09.109755+0000","last_became_active":"2024-03-28T23:36:08.116979+0000","last_became_peered":"2024-03-28T23:36:08.116979+0000","last_unstale":"2024-03-28T23:36:09.109755+0000","last_undegraded":"2024-03-28T23:36:09.109755+0000","last_fullsized":"2024-03-28T23:36:09.109755+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:36.635820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.128691+0000","last_change":"2024-03-28T23:36:07.107556+0000","last_active":"2024-03-28T23:36:08.128691+0000","last_peered":"2024-03-28T23:36:08.128691+0000","last_clean":"2024-03-28T23:36:08.128691+0000","last_became_active":"2024-03-28T23:36:07.107318+0000","last_became_peered":"2024-03-28T23:36:07.107318+0000","last_unstale":"2024-03-28T23:36:08.128691+0000","last_undegraded":"2024-03-28T23:36:08.128691+0000","last_fullsized":"2024-03-28T23:36:08.128691+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:21:50.427323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.118086+0000","last_change":"2024-03-28T23:35:59.079990+0000","last_active":"2024-03-28T23:36:09.118086+0000","last_peered":"2024-03-28T23:36:09.118086+0000","last_clean":"2024-03-28T23:36:09.118086+0000","last_became_active":"2024-03-28T23:35:59.079717+0000","last_became_peered":"2024-03-28T23:35:59.079717+0000","last_unstale":"2024-03-28T23:36:09.118086+0000","last_undegraded":"2024-03-28T23:36:09.118086+0000","last_fullsized":"2024-03-28T23:36:09.118086+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:21.402880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.106949+0000","last_change":"2024-03-28T23:35:58.071103+0000","last_active":"2024-03-28T23:36:08.106949+0000","last_peered":"2024-03-28T23:36:08.106949+0000","last_clean":"2024-03-28T23:36:08.106949+0000","last_became_active":"2024-03-28T23:35:58.070587+0000","last_became_peered":"2024-03-28T23:35:58.070587+0000","last_unstale":"2024-03-28T23:36:08.106949+0000","last_undegraded":"2024-03-28T23:36:08.106949+0000","last_fullsized":"2024-03-28T23:36:08.106949+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:16.223515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.190953+0000","last_change":"2024-03-28T23:36:08.190060+0000","last_active":"2024-03-28T23:36:08.190953+0000","last_peered":"2024-03-28T23:36:08.190953+0000","last_clean":"2024-03-28T23:36:08.190953+0000","last_became_active":"2024-03-28T23:36:08.189108+0000","last_became_peered":"2024-03-28T23:36:08.189108+0000","last_unstale":"2024-03-28T23:36:08.190953+0000","last_undegraded":"2024-03-28T23:36:08.190953+0000","last_fullsized":"2024-03-28T23:36:08.190953+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:07.085977+0000","last_clean_scrub_stamp":"2024-03-28T23:36:07.085977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:57:15.318239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-03-28T23:36:08.107131+0000","last_change":"2024-03-28T23:36:07.099839+0000","last_active":"2024-03-28T23:36:08.107131+0000","last_peered":"2024-03-28T23:36:08.107131+0000","last_clean":"2024-03-28T23:36:08.107131+0000","last_became_active":"2024-03-28T23:36:07.099575+0000","last_became_peered":"2024-03-28T23:36:07.099575+0000","last_unstale":"2024-03-28T23:36:08.107131+0000","last_undegraded":"2024-03-28T23:36:08.107131+0000","last_fullsized":"2024-03-28T23:36:08.107131+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:36:06.082134+0000","last_clean_scrub_stamp":"2024-03-28T23:36:06.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:12:19.745437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109162+0000","last_change":"2024-03-28T23:35:59.075847+0000","last_active":"2024-03-28T23:36:09.109162+0000","last_peered":"2024-03-28T23:36:09.109162+0000","last_clean":"2024-03-28T23:36:09.109162+0000","last_became_active":"2024-03-28T23:35:59.072889+0000","last_became_peered":"2024-03-28T23:35:59.072889+0000","last_unstale":"2024-03-28T23:36:09.109162+0000","last_undegraded":"2024-03-28T23:36:09.109162+0000","last_fullsized":"2024-03-28T23:36:09.109162+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:58.047268+0000","last_clean_scrub_stamp":"2024-03-28T23:35:58.047268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:35.927782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.110001+0000","last_change":"2024-03-28T23:35:58.078046+0000","last_active":"2024-03-28T23:36:09.110001+0000","last_peered":"2024-03-28T23:36:09.110001+0000","last_clean":"2024-03-28T23:36:09.110001+0000","last_became_active":"2024-03-28T23:35:58.077780+0000","last_became_peered":"2024-03-28T23:35:58.077780+0000","last_unstale":"2024-03-28T23:36:09.110001+0000","last_undegraded":"2024-03-28T23:36:09.110001+0000","last_fullsized":"2024-03-28T23:36:09.110001+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:35:57.042033+0000","last_clean_scrub_stamp":"2024-03-28T23:35:57.042033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:37:09.547486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":425,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-28T23:36:09.109339+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:36:09.109339+0000","last_peered":"2024-03-28T23:36:09.109339+0000","last_clean":"2024-03-28T23:36:09.109339+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:36:09.109339+0000","last_undegraded":"2024-03-28T23:36:09.109339+0000","last_fullsized":"2024-03-28T23:36:09.109339+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574928,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33148,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338692,"statfs":{"total":96636764160,"available":96602820608,"internally_reserved":0,"allocated":2015232,"data_stored":1538095,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574929,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33096,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338744,"statfs":{"total":96636764160,"available":96602873856,"internally_reserved":0,"allocated":1953792,"data_stored":947797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574929,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33116,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338724,"statfs":{"total":96636764160,"available":96602853376,"internally_reserved":0,"allocated":1974272,"data_stored":949485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574928,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33124,"kb_used_data":1936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338716,"statfs":{"total":96636764160,"available":96602845184,"internally_reserved":0,"allocated":1982464,"data_stored":950137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574928,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33124,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338716,"statfs":{"total":96636764160,"available":96602845184,"internally_reserved":0,"allocated":1990656,"data_stored":957652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574928,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33096,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338744,"statfs":{"total":96636764160,"available":96602873856,"internally_reserved":0,"allocated":1953792,"data_stored":948309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33104,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338736,"statfs":{"total":96636764160,"available":96602865664,"internally_reserved":0,"allocated":1961984,"data_stored":949339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574928,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33172,"kb_used_data":1992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338668,"statfs":{"total":96636764160,"available":96602796032,"internally_reserved":0,"allocated":2039808,"data_stored":1538927,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:36:12.589 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T23:36:12.589 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:36:12.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.724+0000 7fe705de9640 1 Processor -- start 2024-03-28T23:36:12.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.724+0000 7fe705de9640 1 -- start start 2024-03-28T23:36:12.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.724+0000 7fe705de9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe700152890 0x7fe7000c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:12.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.724+0000 7fe705de9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7000c5b20 0x7fe7001553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:12.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.724+0000 7fe705de9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe700158a30 con 0x7fe7000c8ed0 2024-03-28T23:36:12.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.724+0000 7fe705de9640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe7000c8480 con 0x7fe700152890 2024-03-28T23:36:12.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.724+0000 7fe705de9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe700159640 con 0x7fe7000c5b20 2024-03-28T23:36:12.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.725+0000 7fe6ff7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe700152890 0x7fe7000c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:12.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.725+0000 7fe6ff7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe700152890 0x7fe7000c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46634/0 (socket says 172.21.15.161:46634) 2024-03-28T23:36:12.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.725+0000 7fe6ff7fe640 1 -- 172.21.15.161:0/1014178712 learned_addr learned my addr 172.21.15.161:0/1014178712 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:12.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.725+0000 7fe6fffff640 1 --2- 172.21.15.161:0/1014178712 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7000c5b20 0x7fe7001553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:12.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.725+0000 7fe6ff7fe640 1 -- 172.21.15.161:0/1014178712 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7000c5b20 msgr2=0x7fe7001553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:12.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.725+0000 7fe6ff7fe640 1 --2- 172.21.15.161:0/1014178712 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7000c5b20 0x7fe7001553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.725+0000 7fe6ff7fe640 1 -- 172.21.15.161:0/1014178712 >> v1:172.21.15.170:6789/0 conn(0x7fe7000c8ed0 legacy=0x7fe7000c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:12.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.725+0000 7fe6ff7fe640 1 -- 172.21.15.161:0/1014178712 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe700158d10 con 0x7fe700152890 2024-03-28T23:36:12.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.725+0000 7fe6fe7fc640 1 -- 172.21.15.161:0/1014178712 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 651691685 0 0) 0x7fe700158a30 con 0x7fe7000c8ed0 2024-03-28T23:36:12.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.725+0000 7fe6fffff640 1 --2- 172.21.15.161:0/1014178712 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7000c5b20 0x7fe7001553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:12.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.726+0000 7fe6ff7fe640 1 --2- 172.21.15.161:0/1014178712 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe700152890 0x7fe7000c8990 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7fe7001597c0 tx=0x7fe6f002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f3f08af69ab04fe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:12.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.726+0000 7fe6fe7fc640 1 -- 172.21.15.161:0/1014178712 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f00317c0 con 0x7fe700152890 2024-03-28T23:36:12.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.726+0000 7fe6fe7fc640 1 -- 172.21.15.161:0/1014178712 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6f0037070 con 0x7fe700152890 2024-03-28T23:36:12.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.726+0000 7fe6fe7fc640 1 -- 172.21.15.161:0/1014178712 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0031b80 con 0x7fe700152890 2024-03-28T23:36:12.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.727+0000 7fe705de9640 1 -- 172.21.15.161:0/1014178712 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe700152890 msgr2=0x7fe7000c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:12.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.727+0000 7fe705de9640 1 --2- 172.21.15.161:0/1014178712 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe700152890 0x7fe7000c8990 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7fe7001597c0 tx=0x7fe6f002b860 comp rx=0 tx=0).stop 2024-03-28T23:36:12.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.727+0000 7fe705de9640 1 -- 172.21.15.161:0/1014178712 shutdown_connections 2024-03-28T23:36:12.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.727+0000 7fe705de9640 1 --2- 172.21.15.161:0/1014178712 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7000c5b20 0x7fe7001553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.727+0000 7fe705de9640 1 --2- 172.21.15.161:0/1014178712 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe700152890 0x7fe7000c8990 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.727+0000 7fe705de9640 1 -- 172.21.15.161:0/1014178712 >> 172.21.15.161:0/1014178712 conn(0x7fe7000ac620 msgr2=0x7fe7000aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:12.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.727+0000 7fe705de9640 1 -- 172.21.15.161:0/1014178712 shutdown_connections 2024-03-28T23:36:12.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.727+0000 7fe705de9640 1 -- 172.21.15.161:0/1014178712 wait complete. 2024-03-28T23:36:12.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.728+0000 7fe705de9640 1 Processor -- start 2024-03-28T23:36:12.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.728+0000 7fe705de9640 1 -- start start 2024-03-28T23:36:12.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.729+0000 7fe705de9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7000c5b20 0x7fe7001ebab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:12.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.729+0000 7fe705de9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7000c8ed0 0x7fe70014aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:12.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.729+0000 7fe705de9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe700152890 0x7fe70014afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:12.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.729+0000 7fe705de9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe700152c70 con 0x7fe700152890 2024-03-28T23:36:12.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.729+0000 7fe705de9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe7001592d0 con 0x7fe7000c5b20 2024-03-28T23:36:12.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.729+0000 7fe705de9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe700152df0 con 0x7fe7000c8ed0 2024-03-28T23:36:12.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.729+0000 7fe6fffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe700152890 0x7fe70014afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:12.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.729+0000 7fe6fffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe700152890 0x7fe70014afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46648/0 (socket says 172.21.15.161:46648) 2024-03-28T23:36:12.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.729+0000 7fe6fffff640 1 -- 172.21.15.161:0/1478539253 learned_addr learned my addr 172.21.15.161:0/1478539253 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:12.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.729+0000 7fe6ff7fe640 1 --2- 172.21.15.161:0/1478539253 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7000c5b20 0x7fe7001ebab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:12.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.729+0000 7fe6feffd640 1 --2- 172.21.15.161:0/1478539253 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7000c8ed0 0x7fe70014aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:12.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.729+0000 7fe6fffff640 1 -- 172.21.15.161:0/1478539253 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7000c8ed0 msgr2=0x7fe70014aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:12.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.729+0000 7fe6fffff640 1 --2- 172.21.15.161:0/1478539253 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7000c8ed0 0x7fe70014aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.729+0000 7fe6fffff640 1 -- 172.21.15.161:0/1478539253 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7000c5b20 msgr2=0x7fe7001ebab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:12.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.729+0000 7fe6fffff640 1 --2- 172.21.15.161:0/1478539253 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7000c5b20 0x7fe7001ebab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.729+0000 7fe6fffff640 1 -- 172.21.15.161:0/1478539253 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7000c2310 con 0x7fe700152890 2024-03-28T23:36:12.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.730+0000 7fe6fffff640 1 --2- 172.21.15.161:0/1478539253 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe700152890 0x7fe70014afc0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7fe6f800ef90 tx=0x7fe6f800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:12.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.730+0000 7fe6e7fff640 1 -- 172.21.15.161:0/1478539253 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f8018020 con 0x7fe700152890 2024-03-28T23:36:12.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.730+0000 7fe6e7fff640 1 -- 172.21.15.161:0/1478539253 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6f8014480 con 0x7fe700152890 2024-03-28T23:36:12.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.730+0000 7fe705de9640 1 -- 172.21.15.161:0/1478539253 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7000c4ea0 con 0x7fe700152890 2024-03-28T23:36:12.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.730+0000 7fe6e7fff640 1 -- 172.21.15.161:0/1478539253 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f8014850 con 0x7fe700152890 2024-03-28T23:36:12.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.730+0000 7fe705de9640 1 -- 172.21.15.161:0/1478539253 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7000c0150 con 0x7fe700152890 2024-03-28T23:36:12.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.731+0000 7fe705de9640 1 -- 172.21.15.161:0/1478539253 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe70004f4b0 con 0x7fe700152890 2024-03-28T23:36:12.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.731+0000 7fe6e7fff640 1 -- 172.21.15.161:0/1478539253 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe6f8005d10 con 0x7fe700152890 2024-03-28T23:36:12.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.732+0000 7fe6e7fff640 1 --2- 172.21.15.161:0/1478539253 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe6ec06b500 0x7fe6ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:12.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.732+0000 7fe6e7fff640 1 -- 172.21.15.161:0/1478539253 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 6957+0+0 (secure 0 0 0) 0x7fe6f808b190 con 0x7fe700152890 2024-03-28T23:36:12.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.732+0000 7fe6ff7fe640 1 --2- 172.21.15.161:0/1478539253 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe6ec06b500 0x7fe6ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:12.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.732+0000 7fe6ff7fe640 1 --2- 172.21.15.161:0/1478539253 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe6ec06b500 0x7fe6ec06d9c0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0036000 tx=0x7fe6f003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:12.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.736+0000 7fe6e7fff640 1 -- 172.21.15.161:0/1478539253 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe6f8022030 con 0x7fe700152890 2024-03-28T23:36:12.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.973+0000 7fe705de9640 1 -- 172.21.15.161:0/1478539253 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe70014c700 con 0x7fe700152890 2024-03-28T23:36:12.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.973+0000 7fe6e7fff640 1 -- 172.21.15.161:0/1478539253 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) ==== 95+0+6519 (secure 0 0 0) 0x7fe6f8010040 con 0x7fe700152890 2024-03-28T23:36:12.974 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:12.974 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":144,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7061,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4026332703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4026332703},{"type":"v1","addr":"172.21.15.161:6836","nonce":4026332703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":7070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/557522063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":557522063},{"type":"v1","addr":"172.21.15.170:6835","nonce":557522063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:58.406403+0000","modified":"2024-03-28T23:36:00.406209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8068},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8068":{"gid":8068,"name":"b","rank":0,"incarnation":138,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6834/1393137291","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1393137291},{"type":"v1","addr":"172.21.15.170:6834","nonce":1393137291}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8068,"qdb_cluster":[8068]},"id":11},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:36:07.448976+0000","modified":"2024-03-28T23:36:10.478828+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8056":{"gid":8056,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":5,"addr":"172.21.15.161:6837/2391019771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":2391019771},{"type":"v1","addr":"172.21.15.161:6837","nonce":2391019771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8056,"qdb_cluster":[8056]},"id":12}]} 2024-03-28T23:36:12.974 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 144 2024-03-28T23:36:12.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.976+0000 7fe705de9640 1 -- 172.21.15.161:0/1478539253 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe6ec06b500 msgr2=0x7fe6ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:12.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.976+0000 7fe705de9640 1 --2- 172.21.15.161:0/1478539253 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe6ec06b500 0x7fe6ec06d9c0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0036000 tx=0x7fe6f003d040 comp rx=0 tx=0).stop 2024-03-28T23:36:12.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.976+0000 7fe705de9640 1 -- 172.21.15.161:0/1478539253 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe700152890 msgr2=0x7fe70014afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:12.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.976+0000 7fe705de9640 1 --2- 172.21.15.161:0/1478539253 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe700152890 0x7fe70014afc0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7fe6f800ef90 tx=0x7fe6f800c490 comp rx=0 tx=0).stop 2024-03-28T23:36:12.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.976+0000 7fe705de9640 1 -- 172.21.15.161:0/1478539253 shutdown_connections 2024-03-28T23:36:12.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.976+0000 7fe705de9640 1 --2- 172.21.15.161:0/1478539253 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe6ec06b500 0x7fe6ec06d9c0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.976+0000 7fe705de9640 1 --2- 172.21.15.161:0/1478539253 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe700152890 0x7fe70014afc0 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.976+0000 7fe705de9640 1 --2- 172.21.15.161:0/1478539253 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7000c8ed0 0x7fe70014aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.976+0000 7fe705de9640 1 --2- 172.21.15.161:0/1478539253 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7000c5b20 0x7fe7001ebab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:12.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.976+0000 7fe705de9640 1 -- 172.21.15.161:0/1478539253 >> 172.21.15.161:0/1478539253 conn(0x7fe7000ac620 msgr2=0x7fe7000aa3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:12.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.976+0000 7fe705de9640 1 -- 172.21.15.161:0/1478539253 shutdown_connections 2024-03-28T23:36:12.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:12.976+0000 7fe705de9640 1 -- 172.21.15.161:0/1478539253 wait complete. 2024-03-28T23:36:12.988 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 11, "epoch": 144, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 11, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:35:58.406403+0000", "damaged": [], "data_pools": [ 24 ], "enabled": true, "epoch": 140, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8068": { "addr": "172.21.15.170:6834/1393137291", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 1393137291, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 1393137291, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8068, "incarnation": 138, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 23, "modified": "2024-03-28T23:36:00.406209+0000", "qdb_cluster": [ 8068 ], "qdb_leader": 8068, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8068 } } }, { "id": 12, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:36:07.448976+0000", "damaged": [], "data_pools": [ 26 ], "enabled": true, "epoch": 144, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs2", "in": [ 0 ], "info": { "gid_8056": { "addr": "172.21.15.161:6837/2391019771", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6835", "nonce": 2391019771, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 2391019771, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8056, "incarnation": 141, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 25, "modified": "2024-03-28T23:36:10.478828+0000", "qdb_cluster": [ 8056 ], "qdb_leader": 8056, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8056 } } } ], "standbys": [ { "addr": "172.21.15.161:6836/4026332703", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 4026332703, "type": "v2" }, { "addr": "172.21.15.161:6836", "nonce": 4026332703, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 137, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7061, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.170:6835/557522063", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 557522063, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 557522063, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 137, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7070, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:36:12.988 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-28T23:36:12.988 INFO:tasks.cephfs.test_failover:current = [{'gid': 8056, 'name': 'a', 'rank': 0, 'incarnation': 141, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.161:6837/2391019771', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 2391019771}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 2391019771}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 8068, 'name': 'b', 'rank': 0, 'incarnation': 138, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.170:6834/1393137291', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 1393137291}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 1393137291}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 7061, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6836/4026332703', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 4026332703}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 4026332703}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 137}, {'gid': 7070, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/557522063', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 557522063}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 557522063}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 137}] 2024-03-28T23:36:12.989 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:36:13.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.091+0000 7f8ac394f640 1 Processor -- start 2024-03-28T23:36:13.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.091+0000 7f8ac394f640 1 -- start start 2024-03-28T23:36:13.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.092+0000 7f8ac394f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8abc152890 0x7f8abc0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:13.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.092+0000 7f8ac394f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8abc0c8ed0 0x7f8abc0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:13.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.092+0000 7f8ac394f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8abc0c5950 0x7f8abc1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:13.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.092+0000 7f8ac394f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8abc0c8480 con 0x7f8abc0c8ed0 2024-03-28T23:36:13.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.092+0000 7f8ac394f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f8abc159640 con 0x7f8abc152890 2024-03-28T23:36:13.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.092+0000 7f8ac394f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8abc158a30 con 0x7f8abc0c5950 2024-03-28T23:36:13.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.092+0000 7f8ac16eb640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8abc152890 0x7f8abc0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:13.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.092+0000 7f8ac0eea640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8abc0c8ed0 0x7f8abc0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:13.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.092+0000 7f8ac16eb640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8abc152890 0x7f8abc0c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:53574/0 (socket says 172.21.15.161:53574) 2024-03-28T23:36:13.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.092+0000 7f8ac16eb640 1 -- 172.21.15.161:0/3389277068 learned_addr learned my addr 172.21.15.161:0/3389277068 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:13.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.092+0000 7f8ac1eec640 1 --2- 172.21.15.161:0/3389277068 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8abc0c5950 0x7f8abc1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:13.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.092+0000 7f8ac16eb640 1 -- 172.21.15.161:0/3389277068 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8abc0c5950 msgr2=0x7f8abc1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:13.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.092+0000 7f8ac16eb640 1 --2- 172.21.15.161:0/3389277068 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8abc0c5950 0x7f8abc1553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.092+0000 7f8ac16eb640 1 -- 172.21.15.161:0/3389277068 >> v2:172.21.15.161:3300/0 conn(0x7f8abc0c8ed0 msgr2=0x7f8abc0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:13.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.092+0000 7f8ac16eb640 1 --2- 172.21.15.161:0/3389277068 >> v2:172.21.15.161:3300/0 conn(0x7f8abc0c8ed0 0x7f8abc0c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.092+0000 7f8ac16eb640 1 -- 172.21.15.161:0/3389277068 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8abc158d10 con 0x7f8abc152890 2024-03-28T23:36:13.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.093+0000 7f8ac16eb640 1 --2- 172.21.15.161:0/3389277068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8abc152890 0x7f8abc0c8990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8abc08cc80 tx=0x7f8ab802b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67d3ffbc7f2714bc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:13.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.093+0000 7f8ab3fff640 1 -- 172.21.15.161:0/3389277068 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab8038070 con 0x7f8abc152890 2024-03-28T23:36:13.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.093+0000 7f8ab3fff640 1 -- 172.21.15.161:0/3389277068 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ab802f050 con 0x7f8abc152890 2024-03-28T23:36:13.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.093+0000 7f8ab3fff640 1 -- 172.21.15.161:0/3389277068 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab8031970 con 0x7f8abc152890 2024-03-28T23:36:13.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.094+0000 7f8ac394f640 1 -- 172.21.15.161:0/3389277068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8abc152890 msgr2=0x7f8abc0c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:13.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.094+0000 7f8ac394f640 1 --2- 172.21.15.161:0/3389277068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8abc152890 0x7f8abc0c8990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8abc08cc80 tx=0x7f8ab802b7f0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.094+0000 7f8ac394f640 1 -- 172.21.15.161:0/3389277068 shutdown_connections 2024-03-28T23:36:13.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.094+0000 7f8ac394f640 1 --2- 172.21.15.161:0/3389277068 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8abc0c5950 0x7f8abc1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.094+0000 7f8ac394f640 1 --2- 172.21.15.161:0/3389277068 >> v2:172.21.15.161:3300/0 conn(0x7f8abc0c8ed0 0x7f8abc0c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.094+0000 7f8ac394f640 1 --2- 172.21.15.161:0/3389277068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8abc152890 0x7f8abc0c8990 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.094+0000 7f8ac394f640 1 -- 172.21.15.161:0/3389277068 >> 172.21.15.161:0/3389277068 conn(0x7f8abc0ac620 msgr2=0x7f8abc0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:13.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.094+0000 7f8ac394f640 1 -- 172.21.15.161:0/3389277068 shutdown_connections 2024-03-28T23:36:13.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.094+0000 7f8ac394f640 1 -- 172.21.15.161:0/3389277068 wait complete. 2024-03-28T23:36:13.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.095+0000 7f8ac394f640 1 Processor -- start 2024-03-28T23:36:13.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.095+0000 7f8ac394f640 1 -- start start 2024-03-28T23:36:13.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.095+0000 7f8ac394f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8abc0c5950 0x7f8abc1eba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:13.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.095+0000 7f8ac394f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8abc0c8ed0 0x7f8abc149890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:13.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.095+0000 7f8ac394f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8abc152890 0x7f8abc14ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:13.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.095+0000 7f8ac394f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8abc152c70 con 0x7f8abc0c8ed0 2024-03-28T23:36:13.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.095+0000 7f8ac394f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8abc1592d0 con 0x7f8abc152890 2024-03-28T23:36:13.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.095+0000 7f8ac394f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8abc152df0 con 0x7f8abc0c5950 2024-03-28T23:36:13.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.096+0000 7f8ac0eea640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8abc0c8ed0 0x7f8abc149890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:13.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.096+0000 7f8ac0eea640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8abc0c8ed0 0x7f8abc149890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46674/0 (socket says 172.21.15.161:46674) 2024-03-28T23:36:13.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.096+0000 7f8ac16eb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8abc0c5950 0x7f8abc1eba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:13.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.096+0000 7f8ac0eea640 1 -- 172.21.15.161:0/4191794141 learned_addr learned my addr 172.21.15.161:0/4191794141 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:13.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.096+0000 7f8ac1eec640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8abc152890 0x7f8abc14ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:13.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.096+0000 7f8ac0eea640 1 -- 172.21.15.161:0/4191794141 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8abc0c5950 msgr2=0x7f8abc1eba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:13.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.096+0000 7f8ac0eea640 1 --2- 172.21.15.161:0/4191794141 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8abc0c5950 0x7f8abc1eba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.096+0000 7f8ac0eea640 1 -- 172.21.15.161:0/4191794141 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8abc152890 msgr2=0x7f8abc14ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:13.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.096+0000 7f8ac0eea640 1 --2- 172.21.15.161:0/4191794141 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8abc152890 0x7f8abc14ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.096+0000 7f8ac0eea640 1 -- 172.21.15.161:0/4191794141 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8abc149dd0 con 0x7f8abc0c8ed0 2024-03-28T23:36:13.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.096+0000 7f8ac0eea640 1 --2- 172.21.15.161:0/4191794141 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8abc0c8ed0 0x7f8abc149890 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f8aac00e840 tx=0x7f8aac00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:13.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.096+0000 7f8ab1ffb640 1 -- 172.21.15.161:0/4191794141 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aac017070 con 0x7f8abc0c8ed0 2024-03-28T23:36:13.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.096+0000 7f8ab1ffb640 1 -- 172.21.15.161:0/4191794141 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8aac00c940 con 0x7f8abc0c8ed0 2024-03-28T23:36:13.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.096+0000 7f8ac394f640 1 -- 172.21.15.161:0/4191794141 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8abc158290 con 0x7f8abc0c8ed0 2024-03-28T23:36:13.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.096+0000 7f8ab1ffb640 1 -- 172.21.15.161:0/4191794141 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aac00cd10 con 0x7f8abc0c8ed0 2024-03-28T23:36:13.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.096+0000 7f8ac394f640 1 -- 172.21.15.161:0/4191794141 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8abc156790 con 0x7f8abc0c8ed0 2024-03-28T23:36:13.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.098+0000 7f8ac394f640 1 -- 172.21.15.161:0/4191794141 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8abc04f4b0 con 0x7f8abc0c8ed0 2024-03-28T23:36:13.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.098+0000 7f8ab1ffb640 1 -- 172.21.15.161:0/4191794141 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8aac01a070 con 0x7f8abc0c8ed0 2024-03-28T23:36:13.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.098+0000 7f8ab1ffb640 1 --2- 172.21.15.161:0/4191794141 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8aa006b4d0 0x7f8aa006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:13.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.099+0000 7f8ac16eb640 1 --2- 172.21.15.161:0/4191794141 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8aa006b4d0 0x7f8aa006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:13.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.099+0000 7f8ab1ffb640 1 -- 172.21.15.161:0/4191794141 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 6957+0+0 (secure 0 0 0) 0x7f8aac08a960 con 0x7f8abc0c8ed0 2024-03-28T23:36:13.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.099+0000 7f8ac16eb640 1 --2- 172.21.15.161:0/4191794141 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8aa006b4d0 0x7f8aa006d990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f8ab8035000 tx=0x7f8ab8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:13.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.103+0000 7f8ab1ffb640 1 -- 172.21.15.161:0/4191794141 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8aac058cc0 con 0x7f8abc0c8ed0 2024-03-28T23:36:13.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.319+0000 7f8ac394f640 1 -- 172.21.15.161:0/4191794141 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8abc0c8140 con 0x7f8abc0c8ed0 2024-03-28T23:36:13.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.319+0000 7f8ab1ffb640 1 -- 172.21.15.161:0/4191794141 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) ==== 95+0+6519 (secure 0 0 0) 0x7f8aac05cd60 con 0x7f8abc0c8ed0 2024-03-28T23:36:13.320 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:13.320 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":144,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7061,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/4026332703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4026332703},{"type":"v1","addr":"172.21.15.161:6836","nonce":4026332703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":7070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/557522063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":557522063},{"type":"v1","addr":"172.21.15.170:6835","nonce":557522063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:58.406403+0000","modified":"2024-03-28T23:36:00.406209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8068},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8068":{"gid":8068,"name":"b","rank":0,"incarnation":138,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6834/1393137291","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1393137291},{"type":"v1","addr":"172.21.15.170:6834","nonce":1393137291}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8068,"qdb_cluster":[8068]},"id":11},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:36:07.448976+0000","modified":"2024-03-28T23:36:10.478828+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8056":{"gid":8056,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":5,"addr":"172.21.15.161:6837/2391019771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":2391019771},{"type":"v1","addr":"172.21.15.161:6837","nonce":2391019771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8056,"qdb_cluster":[8056]},"id":12}]} 2024-03-28T23:36:13.320 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 144 2024-03-28T23:36:13.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.321+0000 7f8ac394f640 1 -- 172.21.15.161:0/4191794141 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8aa006b4d0 msgr2=0x7f8aa006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:13.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.321+0000 7f8ac394f640 1 --2- 172.21.15.161:0/4191794141 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8aa006b4d0 0x7f8aa006d990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f8ab8035000 tx=0x7f8ab8008030 comp rx=0 tx=0).stop 2024-03-28T23:36:13.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.321+0000 7f8ac394f640 1 -- 172.21.15.161:0/4191794141 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8abc0c8ed0 msgr2=0x7f8abc149890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:13.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.321+0000 7f8ac394f640 1 --2- 172.21.15.161:0/4191794141 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8abc0c8ed0 0x7f8abc149890 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f8aac00e840 tx=0x7f8aac00ed00 comp rx=0 tx=0).stop 2024-03-28T23:36:13.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.321+0000 7f8ac394f640 1 -- 172.21.15.161:0/4191794141 shutdown_connections 2024-03-28T23:36:13.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.321+0000 7f8ac394f640 1 --2- 172.21.15.161:0/4191794141 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8aa006b4d0 0x7f8aa006d990 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.321+0000 7f8ac394f640 1 --2- 172.21.15.161:0/4191794141 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8abc0c8ed0 0x7f8abc149890 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.321+0000 7f8ac394f640 1 --2- 172.21.15.161:0/4191794141 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8abc152890 0x7f8abc14ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.321+0000 7f8ac394f640 1 --2- 172.21.15.161:0/4191794141 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8abc0c5950 0x7f8abc1eba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.321+0000 7f8ac394f640 1 -- 172.21.15.161:0/4191794141 >> 172.21.15.161:0/4191794141 conn(0x7f8abc0ac620 msgr2=0x7f8abc0c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:13.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.321+0000 7f8ac394f640 1 -- 172.21.15.161:0/4191794141 shutdown_connections 2024-03-28T23:36:13.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.321+0000 7f8ac394f640 1 -- 172.21.15.161:0/4191794141 wait complete. 2024-03-28T23:36:13.333 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 11, "epoch": 144, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 11, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:35:58.406403+0000", "damaged": [], "data_pools": [ 24 ], "enabled": true, "epoch": 140, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8068": { "addr": "172.21.15.170:6834/1393137291", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 1393137291, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 1393137291, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8068, "incarnation": 138, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 23, "modified": "2024-03-28T23:36:00.406209+0000", "qdb_cluster": [ 8068 ], "qdb_leader": 8068, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8068 } } }, { "id": 12, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:36:07.448976+0000", "damaged": [], "data_pools": [ 26 ], "enabled": true, "epoch": 144, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs2", "in": [ 0 ], "info": { "gid_8056": { "addr": "172.21.15.161:6837/2391019771", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6835", "nonce": 2391019771, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 2391019771, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8056, "incarnation": 141, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 25, "modified": "2024-03-28T23:36:10.478828+0000", "qdb_cluster": [ 8056 ], "qdb_leader": 8056, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8056 } } } ], "standbys": [ { "addr": "172.21.15.161:6836/4026332703", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 4026332703, "type": "v2" }, { "addr": "172.21.15.161:6836", "nonce": 4026332703, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 137, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7061, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.170:6835/557522063", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 557522063, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 557522063, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 137, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7070, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:36:13.334 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-28T23:36:13.334 INFO:tasks.cephfs.test_failover:current = [{'gid': 8056, 'name': 'a', 'rank': 0, 'incarnation': 141, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.161:6837/2391019771', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 2391019771}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 2391019771}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 8068, 'name': 'b', 'rank': 0, 'incarnation': 138, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.170:6834/1393137291', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 1393137291}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 1393137291}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 7061, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6836/4026332703', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 4026332703}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 4026332703}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 137}, {'gid': 7070, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/557522063', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 557522063}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 557522063}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 137}] 2024-03-28T23:36:13.334 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.c mds_join_fs cephfs2 2024-03-28T23:36:13.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.443+0000 7fa8e3ec0640 1 Processor -- start 2024-03-28T23:36:13.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.443+0000 7fa8e3ec0640 1 -- start start 2024-03-28T23:36:13.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.443+0000 7fa8e3ec0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8dc1589b0 0x7fa8dc158e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:13.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.443+0000 7fa8e3ec0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8dc0c6f60 con 0x7fa8dc0c0330 2024-03-28T23:36:13.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.443+0000 7fa8e3ec0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8dc0c6120 con 0x7fa8dc152bc0 2024-03-28T23:36:13.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.444+0000 7fa8e3ec0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa8dc0c6400 con 0x7fa8dc1589b0 2024-03-28T23:36:13.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.444+0000 7fa8e145b640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa8dc0c0330 0x7fa8dc0c9190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41720/0 (socket says 172.21.15.161:41720) 2024-03-28T23:36:13.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.444+0000 7fa8e145b640 1 -- 172.21.15.161:0/2339016933 learned_addr learned my addr 172.21.15.161:0/2339016933 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:13.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.444+0000 7fa8e245d640 1 --2- 172.21.15.161:0/2339016933 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8dc1589b0 0x7fa8dc158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:13.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.444+0000 7fa8e0c5a640 1 -- 172.21.15.161:0/2339016933 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 125096595 0 0) 0x7fa8dc0c6f60 con 0x7fa8dc0c0330 2024-03-28T23:36:13.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.444+0000 7fa8e0c5a640 1 -- 172.21.15.161:0/2339016933 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8c8003520 con 0x7fa8dc0c0330 2024-03-28T23:36:13.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.444+0000 7fa8e0c5a640 1 -- 172.21.15.161:0/2339016933 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4070008300 0 0) 0x7fa8dc0c6120 con 0x7fa8dc152bc0 2024-03-28T23:36:13.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.444+0000 7fa8e0c5a640 1 -- 172.21.15.161:0/2339016933 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8dc0c6f60 con 0x7fa8dc152bc0 2024-03-28T23:36:13.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.444+0000 7fa8e245d640 1 -- 172.21.15.161:0/2339016933 >> v1:172.21.15.170:6789/0 conn(0x7fa8dc152bc0 legacy=0x7fa8dc0c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:13.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.444+0000 7fa8e245d640 1 -- 172.21.15.161:0/2339016933 >> v1:172.21.15.161:6789/0 conn(0x7fa8dc0c0330 legacy=0x7fa8dc0c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:13.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.444+0000 7fa8e245d640 1 -- 172.21.15.161:0/2339016933 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8dc0c7240 con 0x7fa8dc1589b0 2024-03-28T23:36:13.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.445+0000 7fa8e245d640 1 --2- 172.21.15.161:0/2339016933 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8dc1589b0 0x7fa8dc158e40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4009dc0 tx=0x7fa8d402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf62355413b0796e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:13.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.445+0000 7fa8e0c5a640 1 -- 172.21.15.161:0/2339016933 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d4039070 con 0x7fa8dc1589b0 2024-03-28T23:36:13.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.445+0000 7fa8e0c5a640 1 -- 172.21.15.161:0/2339016933 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8d402cc40 con 0x7fa8dc1589b0 2024-03-28T23:36:13.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.445+0000 7fa8e0c5a640 1 -- 172.21.15.161:0/2339016933 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d4034440 con 0x7fa8dc1589b0 2024-03-28T23:36:13.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.446+0000 7fa8e3ec0640 1 -- 172.21.15.161:0/2339016933 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8dc1589b0 msgr2=0x7fa8dc158e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:13.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.446+0000 7fa8e3ec0640 1 --2- 172.21.15.161:0/2339016933 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8dc1589b0 0x7fa8dc158e40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4009dc0 tx=0x7fa8d402be90 comp rx=0 tx=0).stop 2024-03-28T23:36:13.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.446+0000 7fa8e3ec0640 1 -- 172.21.15.161:0/2339016933 shutdown_connections 2024-03-28T23:36:13.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.446+0000 7fa8e3ec0640 1 --2- 172.21.15.161:0/2339016933 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8dc1589b0 0x7fa8dc158e40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.446+0000 7fa8e3ec0640 1 -- 172.21.15.161:0/2339016933 >> 172.21.15.161:0/2339016933 conn(0x7fa8dc0caa80 msgr2=0x7fa8dc0cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:13.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.446+0000 7fa8e3ec0640 1 -- 172.21.15.161:0/2339016933 shutdown_connections 2024-03-28T23:36:13.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.446+0000 7fa8e3ec0640 1 -- 172.21.15.161:0/2339016933 wait complete. 2024-03-28T23:36:13.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.447+0000 7fa8e3ec0640 1 Processor -- start 2024-03-28T23:36:13.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.447+0000 7fa8e3ec0640 1 -- start start 2024-03-28T23:36:13.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.448+0000 7fa8e3ec0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8dc0c0330 0x7fa8dc155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:13.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.448+0000 7fa8e3ec0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8dc152bc0 0x7fa8dc0ab7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:13.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.448+0000 7fa8e3ec0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8dc1589b0 0x7fa8dc154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:13.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.448+0000 7fa8e3ec0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa8dc152fa0 con 0x7fa8dc0c0330 2024-03-28T23:36:13.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.448+0000 7fa8e3ec0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa8dc159380 con 0x7fa8dc1589b0 2024-03-28T23:36:13.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.448+0000 7fa8e3ec0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa8dc1eec80 con 0x7fa8dc152bc0 2024-03-28T23:36:13.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.448+0000 7fa8e1c5c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8dc0c0330 0x7fa8dc155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:13.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.448+0000 7fa8e1c5c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8dc0c0330 0x7fa8dc155fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46678/0 (socket says 172.21.15.161:46678) 2024-03-28T23:36:13.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.448+0000 7fa8e1c5c640 1 -- 172.21.15.161:0/420807118 learned_addr learned my addr 172.21.15.161:0/420807118 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:13.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.448+0000 7fa8e245d640 1 --2- 172.21.15.161:0/420807118 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8dc1589b0 0x7fa8dc154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:13.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.448+0000 7fa8e145b640 1 --2- 172.21.15.161:0/420807118 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8dc152bc0 0x7fa8dc0ab7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:13.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.448+0000 7fa8e1c5c640 1 -- 172.21.15.161:0/420807118 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8dc152bc0 msgr2=0x7fa8dc0ab7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:13.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.448+0000 7fa8e1c5c640 1 --2- 172.21.15.161:0/420807118 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8dc152bc0 0x7fa8dc0ab7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.448+0000 7fa8e1c5c640 1 -- 172.21.15.161:0/420807118 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8dc1589b0 msgr2=0x7fa8dc154f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:13.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.448+0000 7fa8e1c5c640 1 --2- 172.21.15.161:0/420807118 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8dc1589b0 0x7fa8dc154f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.448+0000 7fa8e1c5c640 1 -- 172.21.15.161:0/420807118 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8dc1eeed0 con 0x7fa8dc0c0330 2024-03-28T23:36:13.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.448+0000 7fa8e1c5c640 1 --2- 172.21.15.161:0/420807118 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8dc0c0330 0x7fa8dc155fc0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fa8d800c500 tx=0x7fa8d800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:13.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.448+0000 7fa8d27fc640 1 -- 172.21.15.161:0/420807118 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d80052d0 con 0x7fa8dc0c0330 2024-03-28T23:36:13.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.448+0000 7fa8e3ec0640 1 -- 172.21.15.161:0/420807118 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8dc1eaea0 con 0x7fa8dc0c0330 2024-03-28T23:36:13.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.448+0000 7fa8d27fc640 1 -- 172.21.15.161:0/420807118 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8d8005490 con 0x7fa8dc0c0330 2024-03-28T23:36:13.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.449+0000 7fa8d27fc640 1 -- 172.21.15.161:0/420807118 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8012870 con 0x7fa8dc0c0330 2024-03-28T23:36:13.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.449+0000 7fa8e3ec0640 1 -- 172.21.15.161:0/420807118 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8dc1eb980 con 0x7fa8dc0c0330 2024-03-28T23:36:13.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.450+0000 7fa8e3ec0640 1 -- 172.21.15.161:0/420807118 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8dc1ebc40 con 0x7fa8dc0c0330 2024-03-28T23:36:13.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.450+0000 7fa8d27fc640 1 -- 172.21.15.161:0/420807118 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa8d8012a10 con 0x7fa8dc0c0330 2024-03-28T23:36:13.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.450+0000 7fa8d27fc640 1 --2- 172.21.15.161:0/420807118 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa8c006b4d0 0x7fa8c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:13.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.451+0000 7fa8e145b640 1 --2- 172.21.15.161:0/420807118 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa8c006b4d0 0x7fa8c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:13.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.451+0000 7fa8d27fc640 1 -- 172.21.15.161:0/420807118 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 6957+0+0 (secure 0 0 0) 0x7fa8d808a2f0 con 0x7fa8dc0c0330 2024-03-28T23:36:13.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.451+0000 7fa8e145b640 1 --2- 172.21.15.161:0/420807118 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa8c006b4d0 0x7fa8c006d990 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc14aed0 tx=0x7fa8cc007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:13.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.455+0000 7fa8d27fc640 1 -- 172.21.15.161:0/420807118 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa8d8058650 con 0x7fa8dc0c0330 2024-03-28T23:36:13.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.722+0000 7fa8e3ec0640 1 -- 172.21.15.161:0/420807118 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) -- 0x7fa8dc067ac0 con 0x7fa8dc0c0330 2024-03-28T23:36:13.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.725+0000 7fa8d27fc640 1 -- 172.21.15.161:0/420807118 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v52)=0 v52) ==== 117+0+0 (secure 0 0 0) 0x7fa8dc067ac0 con 0x7fa8dc0c0330 2024-03-28T23:36:13.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.727+0000 7fa8e3ec0640 1 -- 172.21.15.161:0/420807118 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa8c006b4d0 msgr2=0x7fa8c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:13.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.727+0000 7fa8e3ec0640 1 --2- 172.21.15.161:0/420807118 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa8c006b4d0 0x7fa8c006d990 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc14aed0 tx=0x7fa8cc007d30 comp rx=0 tx=0).stop 2024-03-28T23:36:13.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.727+0000 7fa8e3ec0640 1 -- 172.21.15.161:0/420807118 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8dc0c0330 msgr2=0x7fa8dc155fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:13.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.727+0000 7fa8e3ec0640 1 --2- 172.21.15.161:0/420807118 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8dc0c0330 0x7fa8dc155fc0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fa8d800c500 tx=0x7fa8d800c9c0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.727+0000 7fa8e3ec0640 1 -- 172.21.15.161:0/420807118 shutdown_connections 2024-03-28T23:36:13.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.727+0000 7fa8e3ec0640 1 --2- 172.21.15.161:0/420807118 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa8c006b4d0 0x7fa8c006d990 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.727+0000 7fa8e3ec0640 1 --2- 172.21.15.161:0/420807118 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8dc1589b0 0x7fa8dc154f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.727+0000 7fa8e3ec0640 1 --2- 172.21.15.161:0/420807118 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8dc152bc0 0x7fa8dc0ab7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.727+0000 7fa8e3ec0640 1 --2- 172.21.15.161:0/420807118 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8dc0c0330 0x7fa8dc155fc0 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.727+0000 7fa8e3ec0640 1 -- 172.21.15.161:0/420807118 >> 172.21.15.161:0/420807118 conn(0x7fa8dc0caa80 msgr2=0x7fa8dc0cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:13.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.727+0000 7fa8e3ec0640 1 -- 172.21.15.161:0/420807118 shutdown_connections 2024-03-28T23:36:13.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.727+0000 7fa8e3ec0640 1 -- 172.21.15.161:0/420807118 wait complete. 2024-03-28T23:36:13.739 INFO:tasks.cephfs.test_failover:assigning affinity to cephfs2 for active mds (mds.a) 2024-03-28T23:36:13.739 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.a mds_join_fs cephfs2 2024-03-28T23:36:13.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.843+0000 7fa54fd2c640 1 Processor -- start 2024-03-28T23:36:13.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.843+0000 7fa54fd2c640 1 -- start start 2024-03-28T23:36:13.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.844+0000 7fa54fd2c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa548152bc0 0x7fa5480c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:13.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.844+0000 7fa54fd2c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa5480c8a80 0x7fa5480c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:13.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.844+0000 7fa54fd2c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5480c6f60 con 0x7fa5480c9420 2024-03-28T23:36:13.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.844+0000 7fa54fd2c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa5480c6120 con 0x7fa548152bc0 2024-03-28T23:36:13.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.844+0000 7fa54fd2c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fa5480c6400 con 0x7fa5480c8a80 2024-03-28T23:36:13.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.844+0000 7fa54dac8640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa548152bc0 0x7fa5480c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:13.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.844+0000 7fa54dac8640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa548152bc0 0x7fa5480c0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46682/0 (socket says 172.21.15.161:46682) 2024-03-28T23:36:13.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.844+0000 7fa54dac8640 1 -- 172.21.15.161:0/3638585726 learned_addr learned my addr 172.21.15.161:0/3638585726 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:13.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.844+0000 7fa54d2c7640 1 --2- 172.21.15.161:0/3638585726 >> v2:172.21.15.170:3300/0 conn(0x7fa5480c8a80 0x7fa5480c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:13.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.844+0000 7fa54dac8640 1 -- 172.21.15.161:0/3638585726 >> v2:172.21.15.170:3300/0 conn(0x7fa5480c8a80 msgr2=0x7fa5480c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:13.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.844+0000 7fa54dac8640 1 --2- 172.21.15.161:0/3638585726 >> v2:172.21.15.170:3300/0 conn(0x7fa5480c8a80 0x7fa5480c8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.844+0000 7fa54dac8640 1 -- 172.21.15.161:0/3638585726 >> v1:172.21.15.170:6789/0 conn(0x7fa5480c9420 legacy=0x7fa5481589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:13.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.844+0000 7fa54dac8640 1 -- 172.21.15.161:0/3638585726 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5480c7240 con 0x7fa548152bc0 2024-03-28T23:36:13.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.845+0000 7fa54dac8640 1 --2- 172.21.15.161:0/3638585726 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa548152bc0 0x7fa5480c0330 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fa5480b0c70 tx=0x7fa54402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f8912fc4279717 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:13.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.845+0000 7fa54cac6640 1 -- 172.21.15.161:0/3638585726 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa544034070 con 0x7fa548152bc0 2024-03-28T23:36:13.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.845+0000 7fa54cac6640 1 -- 172.21.15.161:0/3638585726 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa544030860 con 0x7fa548152bc0 2024-03-28T23:36:13.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.845+0000 7fa54cac6640 1 -- 172.21.15.161:0/3638585726 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa544030c30 con 0x7fa548152bc0 2024-03-28T23:36:13.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.846+0000 7fa54fd2c640 1 -- 172.21.15.161:0/3638585726 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa548152bc0 msgr2=0x7fa5480c0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:13.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.846+0000 7fa54fd2c640 1 --2- 172.21.15.161:0/3638585726 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa548152bc0 0x7fa5480c0330 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fa5480b0c70 tx=0x7fa54402b9a0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.846+0000 7fa54fd2c640 1 -- 172.21.15.161:0/3638585726 shutdown_connections 2024-03-28T23:36:13.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.846+0000 7fa54fd2c640 1 --2- 172.21.15.161:0/3638585726 >> v2:172.21.15.170:3300/0 conn(0x7fa5480c8a80 0x7fa5480c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.846+0000 7fa54fd2c640 1 --2- 172.21.15.161:0/3638585726 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa548152bc0 0x7fa5480c0330 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.846+0000 7fa54fd2c640 1 -- 172.21.15.161:0/3638585726 >> 172.21.15.161:0/3638585726 conn(0x7fa5480caa80 msgr2=0x7fa5480cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:13.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.846+0000 7fa54fd2c640 1 -- 172.21.15.161:0/3638585726 shutdown_connections 2024-03-28T23:36:13.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.846+0000 7fa54fd2c640 1 -- 172.21.15.161:0/3638585726 wait complete. 2024-03-28T23:36:13.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.847+0000 7fa54fd2c640 1 Processor -- start 2024-03-28T23:36:13.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.847+0000 7fa54fd2c640 1 -- start start 2024-03-28T23:36:13.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.847+0000 7fa54fd2c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa5480c8a80 0x7fa548155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:13.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.847+0000 7fa54fd2c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa5480c9420 0x7fa548157d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:13.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.847+0000 7fa54fd2c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa548152bc0 0x7fa548149630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:13.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.847+0000 7fa54fd2c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa548152fa0 con 0x7fa548152bc0 2024-03-28T23:36:13.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.847+0000 7fa54fd2c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa5481590c0 con 0x7fa5480c9420 2024-03-28T23:36:13.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.847+0000 7fa54fd2c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa5481551a0 con 0x7fa5480c8a80 2024-03-28T23:36:13.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.848+0000 7fa54e2c9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa548152bc0 0x7fa548149630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:13.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.848+0000 7fa54e2c9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa548152bc0 0x7fa548149630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46686/0 (socket says 172.21.15.161:46686) 2024-03-28T23:36:13.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.848+0000 7fa54e2c9640 1 -- 172.21.15.161:0/1206161941 learned_addr learned my addr 172.21.15.161:0/1206161941 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:13.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.848+0000 7fa54dac8640 1 --2- 172.21.15.161:0/1206161941 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa5480c8a80 0x7fa548155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:13.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.848+0000 7fa54d2c7640 1 --2- 172.21.15.161:0/1206161941 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa5480c9420 0x7fa548157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:13.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.848+0000 7fa54e2c9640 1 -- 172.21.15.161:0/1206161941 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa5480c8a80 msgr2=0x7fa548155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:13.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.848+0000 7fa54e2c9640 1 --2- 172.21.15.161:0/1206161941 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa5480c8a80 0x7fa548155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.848+0000 7fa54e2c9640 1 -- 172.21.15.161:0/1206161941 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa5480c9420 msgr2=0x7fa548157d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:13.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.848+0000 7fa54e2c9640 1 --2- 172.21.15.161:0/1206161941 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa5480c9420 0x7fa548157d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:13.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.848+0000 7fa54e2c9640 1 -- 172.21.15.161:0/1206161941 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa548154460 con 0x7fa548152bc0 2024-03-28T23:36:13.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.848+0000 7fa54e2c9640 1 --2- 172.21.15.161:0/1206161941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa548152bc0 0x7fa548149630 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fa54000c7d0 tx=0x7fa54000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:13.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.848+0000 7fa53e7fc640 1 -- 172.21.15.161:0/1206161941 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa540003b70 con 0x7fa548152bc0 2024-03-28T23:36:13.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.848+0000 7fa53e7fc640 1 -- 172.21.15.161:0/1206161941 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa540003d30 con 0x7fa548152bc0 2024-03-28T23:36:13.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.848+0000 7fa54fd2c640 1 -- 172.21.15.161:0/1206161941 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5481eeaa0 con 0x7fa548152bc0 2024-03-28T23:36:13.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.848+0000 7fa53e7fc640 1 -- 172.21.15.161:0/1206161941 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54000aa90 con 0x7fa548152bc0 2024-03-28T23:36:13.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.848+0000 7fa54fd2c640 1 -- 172.21.15.161:0/1206161941 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa548155410 con 0x7fa548152bc0 2024-03-28T23:36:13.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.849+0000 7fa54fd2c640 1 -- 172.21.15.161:0/1206161941 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa54804f4b0 con 0x7fa548152bc0 2024-03-28T23:36:13.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.850+0000 7fa53e7fc640 1 -- 172.21.15.161:0/1206161941 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa54000ac30 con 0x7fa548152bc0 2024-03-28T23:36:13.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.850+0000 7fa53e7fc640 1 --2- 172.21.15.161:0/1206161941 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa52c06b4d0 0x7fa52c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:13.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.850+0000 7fa54dac8640 1 --2- 172.21.15.161:0/1206161941 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa52c06b4d0 0x7fa52c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:13.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.850+0000 7fa53e7fc640 1 -- 172.21.15.161:0/1206161941 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 6957+0+0 (secure 0 0 0) 0x7fa54008a520 con 0x7fa548152bc0 2024-03-28T23:36:13.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.851+0000 7fa54dac8640 1 --2- 172.21.15.161:0/1206161941 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa52c06b4d0 0x7fa52c06d990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fa544036000 tx=0x7fa54403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:13.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:13.855+0000 7fa53e7fc640 1 -- 172.21.15.161:0/1206161941 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa5400587e0 con 0x7fa548152bc0 2024-03-28T23:36:13.972 INFO:tasks.ceph.mds.a.smithi161.stderr: -181> 2024-03-28T23:36:09.337+0000 7f4944c2e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:36:13.999 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-28T23:36:14.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.037+0000 7fa54fd2c640 1 -- 172.21.15.161:0/1206161941 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) -- 0x7fa5480abb70 con 0x7fa548152bc0 2024-03-28T23:36:14.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.039+0000 7fa53e7fc640 1 -- 172.21.15.161:0/1206161941 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v53)=0 v53) ==== 117+0+0 (secure 0 0 0) 0x7fa54005c880 con 0x7fa548152bc0 2024-03-28T23:36:14.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.042+0000 7fa54fd2c640 1 -- 172.21.15.161:0/1206161941 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa52c06b4d0 msgr2=0x7fa52c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:14.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.042+0000 7fa54fd2c640 1 --2- 172.21.15.161:0/1206161941 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa52c06b4d0 0x7fa52c06d990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fa544036000 tx=0x7fa54403b040 comp rx=0 tx=0).stop 2024-03-28T23:36:14.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.042+0000 7fa54fd2c640 1 -- 172.21.15.161:0/1206161941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa548152bc0 msgr2=0x7fa548149630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:14.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.042+0000 7fa54fd2c640 1 --2- 172.21.15.161:0/1206161941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa548152bc0 0x7fa548149630 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fa54000c7d0 tx=0x7fa54000cc90 comp rx=0 tx=0).stop 2024-03-28T23:36:14.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.042+0000 7fa54fd2c640 1 -- 172.21.15.161:0/1206161941 shutdown_connections 2024-03-28T23:36:14.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.042+0000 7fa54fd2c640 1 --2- 172.21.15.161:0/1206161941 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa52c06b4d0 0x7fa52c06d990 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:14.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.042+0000 7fa54fd2c640 1 --2- 172.21.15.161:0/1206161941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa548152bc0 0x7fa548149630 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:14.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.042+0000 7fa54fd2c640 1 --2- 172.21.15.161:0/1206161941 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa5480c9420 0x7fa548157d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:14.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.042+0000 7fa54fd2c640 1 --2- 172.21.15.161:0/1206161941 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa5480c8a80 0x7fa548155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:14.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.042+0000 7fa54fd2c640 1 -- 172.21.15.161:0/1206161941 >> 172.21.15.161:0/1206161941 conn(0x7fa5480caa80 msgr2=0x7fa54814d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:14.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.042+0000 7fa54fd2c640 1 -- 172.21.15.161:0/1206161941 shutdown_connections 2024-03-28T23:36:14.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.042+0000 7fa54fd2c640 1 -- 172.21.15.161:0/1206161941 wait complete. 2024-03-28T23:36:14.053 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail 11:0 2024-03-28T23:36:14.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.157+0000 7f1c8b09d640 1 Processor -- start 2024-03-28T23:36:14.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.157+0000 7f1c8b09d640 1 -- start start 2024-03-28T23:36:14.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.157+0000 7f1c8b09d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c840c5410 0x7f1c840c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:14.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.157+0000 7f1c8b09d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c84158a30 con 0x7f1c84152890 2024-03-28T23:36:14.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.157+0000 7f1c8b09d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c84158d10 con 0x7f1c841553a0 2024-03-28T23:36:14.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.157+0000 7f1c8b09d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1c840c8480 con 0x7f1c840c5410 2024-03-28T23:36:14.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.157+0000 7f1c88e39640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f1c84152890 0x7f1c840c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41742/0 (socket says 172.21.15.161:41742) 2024-03-28T23:36:14.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.158+0000 7f1c88e39640 1 -- 172.21.15.161:0/1490183619 learned_addr learned my addr 172.21.15.161:0/1490183619 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:14.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.158+0000 7f1c83fff640 1 --2- 172.21.15.161:0/1490183619 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c840c5410 0x7f1c840c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:14.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.158+0000 7f1c837fe640 1 -- 172.21.15.161:0/1490183619 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 832888267 0 0) 0x7f1c84158a30 con 0x7f1c84152890 2024-03-28T23:36:14.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.158+0000 7f1c837fe640 1 -- 172.21.15.161:0/1490183619 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c74003520 con 0x7f1c84152890 2024-03-28T23:36:14.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.158+0000 7f1c837fe640 1 -- 172.21.15.161:0/1490183619 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1314981739 0 0) 0x7f1c84158d10 con 0x7f1c841553a0 2024-03-28T23:36:14.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.158+0000 7f1c837fe640 1 -- 172.21.15.161:0/1490183619 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c84158a30 con 0x7f1c841553a0 2024-03-28T23:36:14.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.158+0000 7f1c83fff640 1 -- 172.21.15.161:0/1490183619 >> v1:172.21.15.170:6789/0 conn(0x7f1c841553a0 legacy=0x7f1c84155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:14.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.158+0000 7f1c83fff640 1 -- 172.21.15.161:0/1490183619 >> v1:172.21.15.161:6789/0 conn(0x7f1c84152890 legacy=0x7f1c840c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:14.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.158+0000 7f1c83fff640 1 -- 172.21.15.161:0/1490183619 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c84158ff0 con 0x7f1c840c5410 2024-03-28T23:36:14.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.159+0000 7f1c83fff640 1 --2- 172.21.15.161:0/1490183619 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c840c5410 0x7f1c840c5850 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1c700097c0 tx=0x7f1c7002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3490b6ad3f82516 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:14.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.159+0000 7f1c837fe640 1 -- 172.21.15.161:0/1490183619 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c70035020 con 0x7f1c840c5410 2024-03-28T23:36:14.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.159+0000 7f1c837fe640 1 -- 172.21.15.161:0/1490183619 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c70031800 con 0x7f1c840c5410 2024-03-28T23:36:14.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.159+0000 7f1c837fe640 1 -- 172.21.15.161:0/1490183619 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c70031bb0 con 0x7f1c840c5410 2024-03-28T23:36:14.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.159+0000 7f1c8b09d640 1 -- 172.21.15.161:0/1490183619 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c840c5410 msgr2=0x7f1c840c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:14.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.159+0000 7f1c8b09d640 1 --2- 172.21.15.161:0/1490183619 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c840c5410 0x7f1c840c5850 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1c700097c0 tx=0x7f1c7002b8a0 comp rx=0 tx=0).stop 2024-03-28T23:36:14.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.159+0000 7f1c8b09d640 1 -- 172.21.15.161:0/1490183619 shutdown_connections 2024-03-28T23:36:14.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.159+0000 7f1c8b09d640 1 --2- 172.21.15.161:0/1490183619 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c840c5410 0x7f1c840c5850 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:14.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.159+0000 7f1c8b09d640 1 -- 172.21.15.161:0/1490183619 >> 172.21.15.161:0/1490183619 conn(0x7f1c840ac620 msgr2=0x7f1c840aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:14.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.159+0000 7f1c8b09d640 1 -- 172.21.15.161:0/1490183619 shutdown_connections 2024-03-28T23:36:14.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.159+0000 7f1c8b09d640 1 -- 172.21.15.161:0/1490183619 wait complete. 2024-03-28T23:36:14.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.160+0000 7f1c8b09d640 1 Processor -- start 2024-03-28T23:36:14.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.160+0000 7f1c8b09d640 1 -- start start 2024-03-28T23:36:14.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.161+0000 7f1c8b09d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c840c5410 0x7f1c8414a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:14.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.161+0000 7f1c8b09d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c84152890 0x7f1c841ee6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:14.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.161+0000 7f1c8b09d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c841553a0 0x7f1c841eec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:14.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.161+0000 7f1c8b09d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1c84159640 con 0x7f1c841553a0 2024-03-28T23:36:14.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.161+0000 7f1c8b09d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f1c840c5d90 con 0x7f1c840c5410 2024-03-28T23:36:14.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.161+0000 7f1c8b09d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1c840c90a0 con 0x7f1c84152890 2024-03-28T23:36:14.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.161+0000 7f1c8963a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c841553a0 0x7f1c841eec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:14.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.161+0000 7f1c8963a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c841553a0 0x7f1c841eec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46708/0 (socket says 172.21.15.161:46708) 2024-03-28T23:36:14.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.161+0000 7f1c8963a640 1 -- 172.21.15.161:0/434066287 learned_addr learned my addr 172.21.15.161:0/434066287 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:14.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.161+0000 7f1c88e39640 1 --2- 172.21.15.161:0/434066287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c840c5410 0x7f1c8414a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:14.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.161+0000 7f1c83fff640 1 --2- 172.21.15.161:0/434066287 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c84152890 0x7f1c841ee6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:14.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.162+0000 7f1c8963a640 1 -- 172.21.15.161:0/434066287 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c84152890 msgr2=0x7f1c841ee6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:14.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.162+0000 7f1c8963a640 1 --2- 172.21.15.161:0/434066287 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c84152890 0x7f1c841ee6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:14.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.162+0000 7f1c8963a640 1 -- 172.21.15.161:0/434066287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c840c5410 msgr2=0x7f1c8414a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:14.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.162+0000 7f1c8963a640 1 --2- 172.21.15.161:0/434066287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c840c5410 0x7f1c8414a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:14.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.162+0000 7f1c8963a640 1 -- 172.21.15.161:0/434066287 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c8414ee70 con 0x7f1c841553a0 2024-03-28T23:36:14.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.162+0000 7f1c8963a640 1 --2- 172.21.15.161:0/434066287 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c841553a0 0x7f1c841eec20 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c00c710 tx=0x7f1c7c00cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:14.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.162+0000 7f1c817fa640 1 -- 172.21.15.161:0/434066287 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c003b70 con 0x7f1c841553a0 2024-03-28T23:36:14.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.162+0000 7f1c817fa640 1 -- 172.21.15.161:0/434066287 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c7c003d10 con 0x7f1c841553a0 2024-03-28T23:36:14.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.162+0000 7f1c817fa640 1 -- 172.21.15.161:0/434066287 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c00aa50 con 0x7f1c841553a0 2024-03-28T23:36:14.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.162+0000 7f1c8b09d640 1 -- 172.21.15.161:0/434066287 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c8406c980 con 0x7f1c841553a0 2024-03-28T23:36:14.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.162+0000 7f1c8b09d640 1 -- 172.21.15.161:0/434066287 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c8414f410 con 0x7f1c841553a0 2024-03-28T23:36:14.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.163+0000 7f1c8b09d640 1 -- 172.21.15.161:0/434066287 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c8404f4b0 con 0x7f1c841553a0 2024-03-28T23:36:14.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.164+0000 7f1c817fa640 1 -- 172.21.15.161:0/434066287 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1c7c00abf0 con 0x7f1c841553a0 2024-03-28T23:36:14.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.164+0000 7f1c817fa640 1 --2- 172.21.15.161:0/434066287 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1c6806b4d0 0x7f1c6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:14.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.164+0000 7f1c817fa640 1 -- 172.21.15.161:0/434066287 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 7043+0+0 (secure 0 0 0) 0x7f1c7c08a490 con 0x7f1c841553a0 2024-03-28T23:36:14.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.164+0000 7f1c88e39640 1 --2- 172.21.15.161:0/434066287 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1c6806b4d0 0x7f1c6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:14.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.164+0000 7f1c88e39640 1 --2- 172.21.15.161:0/434066287 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1c6806b4d0 0x7f1c6806d990 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f1c780011e0 tx=0x7f1c78007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:14.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.168+0000 7f1c817fa640 1 -- 172.21.15.161:0/434066287 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1c7c00ce20 con 0x7f1c841553a0 2024-03-28T23:36:14.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.411+0000 7f1c8b09d640 1 -- 172.21.15.161:0/434066287 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "11:0"} v 0) -- 0x7f1c840ac400 con 0x7f1c841553a0 2024-03-28T23:36:14.411 INFO:tasks.ceph.mon.a.smithi161.stderr:2024-03-28T23:36:14.411+0000 7fdc7b85b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T23:36:14.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.412+0000 7f1c817fa640 1 -- 172.21.15.161:0/434066287 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "11:0"}]=0 failed mds gid 8068 v146) ==== 98+0+0 (secure 0 0 0) 0x7f1c7c058790 con 0x7f1c841553a0 2024-03-28T23:36:14.413 INFO:teuthology.orchestra.run.smithi161.stderr:failed mds gid 8068 2024-03-28T23:36:14.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.416+0000 7f1c8b09d640 1 -- 172.21.15.161:0/434066287 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1c6806b4d0 msgr2=0x7f1c6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:14.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.416+0000 7f1c8b09d640 1 --2- 172.21.15.161:0/434066287 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1c6806b4d0 0x7f1c6806d990 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f1c780011e0 tx=0x7f1c78007d30 comp rx=0 tx=0).stop 2024-03-28T23:36:14.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.416+0000 7f1c8b09d640 1 -- 172.21.15.161:0/434066287 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c841553a0 msgr2=0x7f1c841eec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:14.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.416+0000 7f1c8b09d640 1 --2- 172.21.15.161:0/434066287 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c841553a0 0x7f1c841eec20 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c00c710 tx=0x7f1c7c00cbd0 comp rx=0 tx=0).stop 2024-03-28T23:36:14.417 INFO:tasks.ceph.mds.b.smithi170.stderr: -343> 2024-03-28T23:36:04.337+0000 7f8d22ab5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:36:14.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.417+0000 7f1c8b09d640 1 -- 172.21.15.161:0/434066287 shutdown_connections 2024-03-28T23:36:14.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.417+0000 7f1c8b09d640 1 --2- 172.21.15.161:0/434066287 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f1c6806b4d0 0x7f1c6806d990 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:14.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.417+0000 7f1c8b09d640 1 --2- 172.21.15.161:0/434066287 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c841553a0 0x7f1c841eec20 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:14.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.417+0000 7f1c8b09d640 1 --2- 172.21.15.161:0/434066287 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c84152890 0x7f1c841ee6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:14.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.417+0000 7f1c8b09d640 1 --2- 172.21.15.161:0/434066287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c840c5410 0x7f1c8414a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:14.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.417+0000 7f1c8b09d640 1 -- 172.21.15.161:0/434066287 >> 172.21.15.161:0/434066287 conn(0x7f1c840ac620 msgr2=0x7f1c8406be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:14.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.417+0000 7f1c8b09d640 1 -- 172.21.15.161:0/434066287 shutdown_connections 2024-03-28T23:36:14.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.417+0000 7f1c8b09d640 1 -- 172.21.15.161:0/434066287 wait complete. 2024-03-28T23:36:14.430 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:36:14.441 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2024-03-28T23:36:14.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.530+0000 7f37b089d640 1 Processor -- start 2024-03-28T23:36:14.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.530+0000 7f37b089d640 1 -- start start 2024-03-28T23:36:14.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.531+0000 7f37b089d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f37ac0c5850 0x7f37ac155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:14.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.531+0000 7f37b089d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37ac0c5d00 con 0x7f37ac152890 2024-03-28T23:36:14.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.531+0000 7f37b089d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37ac158a30 con 0x7f37ac0c5410 2024-03-28T23:36:14.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.531+0000 7f37b089d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f37ac0c8480 con 0x7f37ac0c5850 2024-03-28T23:36:14.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.531+0000 7f37aad9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f37ac0c5850 0x7f37ac155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:14.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.531+0000 7f37aad9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f37ac0c5850 0x7f37ac155ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46718/0 (socket says 172.21.15.161:46718) 2024-03-28T23:36:14.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.531+0000 7f37aad9d640 1 -- 172.21.15.161:0/726548469 learned_addr learned my addr 172.21.15.161:0/726548469 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:14.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.532+0000 7f37aad9d640 1 -- 172.21.15.161:0/726548469 >> v1:172.21.15.170:6789/0 conn(0x7f37ac0c5410 legacy=0x7f37ac1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:14.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.532+0000 7f37aad9d640 1 -- 172.21.15.161:0/726548469 >> v1:172.21.15.161:6789/0 conn(0x7f37ac152890 legacy=0x7f37ac0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:14.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.532+0000 7f37aad9d640 1 -- 172.21.15.161:0/726548469 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37ac158d10 con 0x7f37ac0c5850 2024-03-28T23:36:14.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.532+0000 7f37aad9d640 1 --2- 172.21.15.161:0/726548469 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37ac0c5850 0x7f37ac155ab0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f37a4009870 tx=0x7f37a402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edf90401370b7186 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:14.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.533+0000 7f37a959a640 1 -- 172.21.15.161:0/726548469 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a4034070 con 0x7f37ac0c5850 2024-03-28T23:36:14.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.533+0000 7f37a959a640 1 -- 172.21.15.161:0/726548469 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37a4030860 con 0x7f37ac0c5850 2024-03-28T23:36:14.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.533+0000 7f37a959a640 1 -- 172.21.15.161:0/726548469 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a4030c30 con 0x7f37ac0c5850 2024-03-28T23:36:14.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.533+0000 7f37b089d640 1 -- 172.21.15.161:0/726548469 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37ac0c5850 msgr2=0x7f37ac155ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:14.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.533+0000 7f37b089d640 1 --2- 172.21.15.161:0/726548469 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37ac0c5850 0x7f37ac155ab0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f37a4009870 tx=0x7f37a402b9a0 comp rx=0 tx=0).stop 2024-03-28T23:36:14.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.533+0000 7f37b089d640 1 -- 172.21.15.161:0/726548469 shutdown_connections 2024-03-28T23:36:14.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.533+0000 7f37b089d640 1 --2- 172.21.15.161:0/726548469 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37ac0c5850 0x7f37ac155ab0 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:14.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.533+0000 7f37b089d640 1 -- 172.21.15.161:0/726548469 >> 172.21.15.161:0/726548469 conn(0x7f37ac0ac620 msgr2=0x7f37ac0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:14.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.533+0000 7f37b089d640 1 -- 172.21.15.161:0/726548469 shutdown_connections 2024-03-28T23:36:14.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.533+0000 7f37b089d640 1 -- 172.21.15.161:0/726548469 wait complete. 2024-03-28T23:36:14.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.534+0000 7f37b089d640 1 Processor -- start 2024-03-28T23:36:14.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.534+0000 7f37b089d640 1 -- start start 2024-03-28T23:36:14.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.535+0000 7f37b089d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37ac0c5410 0x7f37ac1ee520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:14.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.535+0000 7f37b089d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37ac0c5850 0x7f37ac1490d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:14.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.535+0000 7f37b089d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37ac152890 0x7f37ac149610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:14.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.535+0000 7f37b089d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f37ac1592d0 con 0x7f37ac0c5850 2024-03-28T23:36:14.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.535+0000 7f37b089d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f37ac159640 con 0x7f37ac152890 2024-03-28T23:36:14.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.535+0000 7f37b089d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f37ac155ff0 con 0x7f37ac0c5410 2024-03-28T23:36:14.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.535+0000 7f37a9d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37ac0c5850 0x7f37ac1490d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:14.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.535+0000 7f37a9d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37ac0c5850 0x7f37ac1490d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46720/0 (socket says 172.21.15.161:46720) 2024-03-28T23:36:14.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.535+0000 7f37a9d9b640 1 -- 172.21.15.161:0/924660512 learned_addr learned my addr 172.21.15.161:0/924660512 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:14.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.535+0000 7f37aad9d640 1 --2- 172.21.15.161:0/924660512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37ac152890 0x7f37ac149610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:14.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.535+0000 7f37aa59c640 1 --2- 172.21.15.161:0/924660512 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37ac0c5410 0x7f37ac1ee520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:14.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.536+0000 7f37a9d9b640 1 -- 172.21.15.161:0/924660512 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37ac0c5410 msgr2=0x7f37ac1ee520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:14.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.536+0000 7f37a9d9b640 1 --2- 172.21.15.161:0/924660512 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37ac0c5410 0x7f37ac1ee520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:14.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.536+0000 7f37a9d9b640 1 -- 172.21.15.161:0/924660512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37ac152890 msgr2=0x7f37ac149610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:14.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.536+0000 7f37a9d9b640 1 --2- 172.21.15.161:0/924660512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37ac152890 0x7f37ac149610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:14.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.536+0000 7f37a9d9b640 1 -- 172.21.15.161:0/924660512 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37ac0c1400 con 0x7f37ac0c5850 2024-03-28T23:36:14.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.536+0000 7f37aa59c640 1 --2- 172.21.15.161:0/924660512 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37ac0c5410 0x7f37ac1ee520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:36:14.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.536+0000 7f37a9d9b640 1 --2- 172.21.15.161:0/924660512 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37ac0c5850 0x7f37ac1490d0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f37a000c3c0 tx=0x7f37a000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:14.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.536+0000 7f379affd640 1 -- 172.21.15.161:0/924660512 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a00052d0 con 0x7f37ac0c5850 2024-03-28T23:36:14.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.536+0000 7f379affd640 1 -- 172.21.15.161:0/924660512 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37a0005490 con 0x7f37ac0c5850 2024-03-28T23:36:14.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.536+0000 7f37b089d640 1 -- 172.21.15.161:0/924660512 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37ac1ef030 con 0x7f37ac0c5850 2024-03-28T23:36:14.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.536+0000 7f379affd640 1 -- 172.21.15.161:0/924660512 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a0006a20 con 0x7f37ac0c5850 2024-03-28T23:36:14.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.536+0000 7f37b089d640 1 -- 172.21.15.161:0/924660512 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37ac150e60 con 0x7f37ac0c5850 2024-03-28T23:36:14.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.537+0000 7f37b089d640 1 -- 172.21.15.161:0/924660512 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37ac04f4b0 con 0x7f37ac0c5850 2024-03-28T23:36:14.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.537+0000 7f379affd640 1 -- 172.21.15.161:0/924660512 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f37a0003640 con 0x7f37ac0c5850 2024-03-28T23:36:14.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.538+0000 7f379affd640 1 --2- 172.21.15.161:0/924660512 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f378c06b500 0x7f378c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:14.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.538+0000 7f379affd640 1 -- 172.21.15.161:0/924660512 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 7129+0+0 (secure 0 0 0) 0x7f37a0089a30 con 0x7f37ac0c5850 2024-03-28T23:36:14.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.538+0000 7f37aa59c640 1 --2- 172.21.15.161:0/924660512 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f378c06b500 0x7f378c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:14.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.538+0000 7f37aa59c640 1 --2- 172.21.15.161:0/924660512 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f378c06b500 0x7f378c06d9c0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f379c0012e0 tx=0x7f379c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:14.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.542+0000 7f379affd640 1 -- 172.21.15.161:0/924660512 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f37a0057d60 con 0x7f37ac0c5850 2024-03-28T23:36:14.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.751+0000 7f37b089d640 1 -- 172.21.15.161:0/924660512 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37ac0c8140 con 0x7f37ac0c5850 2024-03-28T23:36:14.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.752+0000 7f379affd640 1 -- 172.21.15.161:0/924660512 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) ==== 95+0+5683 (secure 0 0 0) 0x7f37a005be00 con 0x7f37ac0c5850 2024-03-28T23:36:14.753 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:14.753 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":147,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/557522063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":557522063},{"type":"v1","addr":"172.21.15.170:6835","nonce":557522063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:58.406403+0000","modified":"2024-03-28T23:36:14.417456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":158,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9183},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9183":{"gid":9183,"name":"a","rank":0,"incarnation":147,"state":"up:replay","state_seq":1,"addr":"172.21.15.161:6837/3236742821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3236742821},{"type":"v1","addr":"172.21.15.161:6837","nonce":3236742821}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":11},{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:36:07.448976+0000","modified":"2024-03-28T23:36:13.980327+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7061":{"gid":7061,"name":"c","rank":0,"incarnation":145,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.161:6836/4026332703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4026332703},{"type":"v1","addr":"172.21.15.161:6836","nonce":4026332703}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":12}]} 2024-03-28T23:36:14.753 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 147 2024-03-28T23:36:14.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.754+0000 7f37b089d640 1 -- 172.21.15.161:0/924660512 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f378c06b500 msgr2=0x7f378c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:14.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.754+0000 7f37b089d640 1 --2- 172.21.15.161:0/924660512 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f378c06b500 0x7f378c06d9c0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f379c0012e0 tx=0x7f379c009040 comp rx=0 tx=0).stop 2024-03-28T23:36:14.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.754+0000 7f37b089d640 1 -- 172.21.15.161:0/924660512 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37ac0c5850 msgr2=0x7f37ac1490d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:14.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.754+0000 7f37b089d640 1 --2- 172.21.15.161:0/924660512 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37ac0c5850 0x7f37ac1490d0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f37a000c3c0 tx=0x7f37a000c880 comp rx=0 tx=0).stop 2024-03-28T23:36:14.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.754+0000 7f37b089d640 1 -- 172.21.15.161:0/924660512 shutdown_connections 2024-03-28T23:36:14.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.754+0000 7f37b089d640 1 --2- 172.21.15.161:0/924660512 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f378c06b500 0x7f378c06d9c0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:14.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.754+0000 7f37b089d640 1 --2- 172.21.15.161:0/924660512 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37ac0c5850 0x7f37ac1490d0 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:14.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.754+0000 7f37b089d640 1 --2- 172.21.15.161:0/924660512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37ac152890 0x7f37ac149610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:14.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.754+0000 7f37b089d640 1 --2- 172.21.15.161:0/924660512 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37ac0c5410 0x7f37ac1ee520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:14.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.754+0000 7f37b089d640 1 -- 172.21.15.161:0/924660512 >> 172.21.15.161:0/924660512 conn(0x7f37ac0ac620 msgr2=0x7f37ac06cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:14.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.754+0000 7f37b089d640 1 -- 172.21.15.161:0/924660512 shutdown_connections 2024-03-28T23:36:14.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:14.754+0000 7f37b089d640 1 -- 172.21.15.161:0/924660512 wait complete. 2024-03-28T23:36:15.168 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 147, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:35:58.406403+0000', 'modified': '2024-03-28T23:36:14.417456+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 158, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9183}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9183': {'gid': 9183, 'name': 'a', 'rank': 0, 'incarnation': 147, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.161:6837/3236742821', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 3236742821}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 3236742821}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-28T23:36:15.168 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9183:up:replay 2024-03-28T23:36:16.169 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:36:16.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.280+0000 7f807d088640 1 Processor -- start 2024-03-28T23:36:16.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.280+0000 7f807d088640 1 -- start start 2024-03-28T23:36:16.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.281+0000 7f807d088640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8078152890 0x7f80780c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:16.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.281+0000 7f807d088640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80780c8ed0 0x7f80780c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:16.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.281+0000 7f807d088640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8078158a30 con 0x7f80780c5950 2024-03-28T23:36:16.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.281+0000 7f807d088640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f80780c8480 con 0x7f8078152890 2024-03-28T23:36:16.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.281+0000 7f807d088640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8078159640 con 0x7f80780c8ed0 2024-03-28T23:36:16.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.281+0000 7f8076d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8078152890 0x7f80780c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:16.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.281+0000 7f8076d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8078152890 0x7f80780c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46730/0 (socket says 172.21.15.161:46730) 2024-03-28T23:36:16.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.281+0000 7f8076d9d640 1 -- 172.21.15.161:0/570080976 learned_addr learned my addr 172.21.15.161:0/570080976 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:16.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.282+0000 7f807659c640 1 --2- 172.21.15.161:0/570080976 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80780c8ed0 0x7f80780c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:16.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.282+0000 7f8076d9d640 1 -- 172.21.15.161:0/570080976 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80780c8ed0 msgr2=0x7f80780c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:16.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.282+0000 7f8076d9d640 1 --2- 172.21.15.161:0/570080976 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80780c8ed0 0x7f80780c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:16.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.282+0000 7f8076d9d640 1 -- 172.21.15.161:0/570080976 >> v1:172.21.15.161:6789/0 conn(0x7f80780c5950 legacy=0x7f80781553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:16.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.282+0000 7f8076d9d640 1 -- 172.21.15.161:0/570080976 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8078158d10 con 0x7f8078152890 2024-03-28T23:36:16.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.282+0000 7f8076d9d640 1 --2- 172.21.15.161:0/570080976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8078152890 0x7f80780c8990 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f8070009e50 tx=0x7f807002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72a5148a9d3c8d60 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:16.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.283+0000 7f8075d9b640 1 -- 172.21.15.161:0/570080976 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f807002c9e0 con 0x7f8078152890 2024-03-28T23:36:16.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.283+0000 7f8075d9b640 1 -- 172.21.15.161:0/570080976 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f807002cb80 con 0x7f8078152890 2024-03-28T23:36:16.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.283+0000 7f8075d9b640 1 -- 172.21.15.161:0/570080976 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80700343f0 con 0x7f8078152890 2024-03-28T23:36:16.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.283+0000 7f807d088640 1 -- 172.21.15.161:0/570080976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8078152890 msgr2=0x7f80780c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:16.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.283+0000 7f807d088640 1 --2- 172.21.15.161:0/570080976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8078152890 0x7f80780c8990 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f8070009e50 tx=0x7f807002be40 comp rx=0 tx=0).stop 2024-03-28T23:36:16.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.283+0000 7f807d088640 1 -- 172.21.15.161:0/570080976 shutdown_connections 2024-03-28T23:36:16.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.283+0000 7f807d088640 1 --2- 172.21.15.161:0/570080976 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80780c8ed0 0x7f80780c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:16.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.283+0000 7f807d088640 1 --2- 172.21.15.161:0/570080976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8078152890 0x7f80780c8990 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:16.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.283+0000 7f807d088640 1 -- 172.21.15.161:0/570080976 >> 172.21.15.161:0/570080976 conn(0x7f80780ac620 msgr2=0x7f80780aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:16.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.283+0000 7f807d088640 1 -- 172.21.15.161:0/570080976 shutdown_connections 2024-03-28T23:36:16.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.283+0000 7f807d088640 1 -- 172.21.15.161:0/570080976 wait complete. 2024-03-28T23:36:16.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.285+0000 7f807d088640 1 Processor -- start 2024-03-28T23:36:16.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.285+0000 7f807d088640 1 -- start start 2024-03-28T23:36:16.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.285+0000 7f807d088640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80780c5950 0x7f80780c2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:16.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.285+0000 7f807d088640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80780c8ed0 0x7f807814bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:16.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.285+0000 7f807d088640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8078152890 0x7f80780c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:16.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.285+0000 7f807d088640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8078152c70 con 0x7f8078152890 2024-03-28T23:36:16.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.285+0000 7f807d088640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f80781592d0 con 0x7f80780c8ed0 2024-03-28T23:36:16.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.285+0000 7f807d088640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8078152df0 con 0x7f80780c5950 2024-03-28T23:36:16.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.285+0000 7f807759e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8078152890 0x7f80780c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:16.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.285+0000 7f807759e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8078152890 0x7f80780c3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46742/0 (socket says 172.21.15.161:46742) 2024-03-28T23:36:16.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.285+0000 7f807759e640 1 -- 172.21.15.161:0/78444065 learned_addr learned my addr 172.21.15.161:0/78444065 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:16.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.286+0000 7f8076d9d640 1 --2- 172.21.15.161:0/78444065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80780c5950 0x7f80780c2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:16.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.286+0000 7f807659c640 1 --2- 172.21.15.161:0/78444065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80780c8ed0 0x7f807814bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:16.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.286+0000 7f807759e640 1 -- 172.21.15.161:0/78444065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80780c5950 msgr2=0x7f80780c2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:16.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.286+0000 7f807759e640 1 --2- 172.21.15.161:0/78444065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80780c5950 0x7f80780c2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:16.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.286+0000 7f807759e640 1 -- 172.21.15.161:0/78444065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80780c8ed0 msgr2=0x7f807814bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:16.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.286+0000 7f807759e640 1 --2- 172.21.15.161:0/78444065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80780c8ed0 0x7f807814bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:16.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.286+0000 7f807759e640 1 -- 172.21.15.161:0/78444065 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80781eba00 con 0x7f8078152890 2024-03-28T23:36:16.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.286+0000 7f807759e640 1 --2- 172.21.15.161:0/78444065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8078152890 0x7f80780c3db0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f806800c390 tx=0x7f806800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:16.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.286+0000 7f806f7fe640 1 -- 172.21.15.161:0/78444065 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8068017070 con 0x7f8078152890 2024-03-28T23:36:16.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.286+0000 7f806f7fe640 1 -- 172.21.15.161:0/78444065 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80680052d0 con 0x7f8078152890 2024-03-28T23:36:16.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.286+0000 7f806f7fe640 1 -- 172.21.15.161:0/78444065 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8068003810 con 0x7f8078152890 2024-03-28T23:36:16.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.286+0000 7f807d088640 1 -- 172.21.15.161:0/78444065 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80781ebce0 con 0x7f8078152890 2024-03-28T23:36:16.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.286+0000 7f807d088640 1 -- 172.21.15.161:0/78444065 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80781e0100 con 0x7f8078152890 2024-03-28T23:36:16.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.287+0000 7f807d088640 1 -- 172.21.15.161:0/78444065 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f807806d310 con 0x7f8078152890 2024-03-28T23:36:16.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.288+0000 7f806f7fe640 1 -- 172.21.15.161:0/78444065 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f806801a070 con 0x7f8078152890 2024-03-28T23:36:16.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.288+0000 7f806f7fe640 1 --2- 172.21.15.161:0/78444065 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f805806b4d0 0x7f805806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:16.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.288+0000 7f806f7fe640 1 -- 172.21.15.161:0/78444065 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 7129+0+0 (secure 0 0 0) 0x7f8068089a40 con 0x7f8078152890 2024-03-28T23:36:16.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.288+0000 7f8076d9d640 1 --2- 172.21.15.161:0/78444065 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f805806b4d0 0x7f805806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:16.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.288+0000 7f8076d9d640 1 --2- 172.21.15.161:0/78444065 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f805806b4d0 0x7f805806d990 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f8070035000 tx=0x7f8070030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:16.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.292+0000 7f806f7fe640 1 -- 172.21.15.161:0/78444065 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f807806d310 con 0x7f8078152890 2024-03-28T23:36:16.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.505+0000 7f807d088640 1 -- 172.21.15.161:0/78444065 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80780c6650 con 0x7f8078152890 2024-03-28T23:36:16.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.506+0000 7f806f7fe640 1 -- 172.21.15.161:0/78444065 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) ==== 95+0+6516 (secure 0 0 0) 0x7f80780c6650 con 0x7f8078152890 2024-03-28T23:36:16.507 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:16.507 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":149,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/557522063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":557522063},{"type":"v1","addr":"172.21.15.170:6835","nonce":557522063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":7283,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/2996092857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":2996092857},{"type":"v1","addr":"172.21.15.170:6834","nonce":2996092857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":148}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:58.406403+0000","modified":"2024-03-28T23:36:15.422299+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":158,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9183},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9183":{"gid":9183,"name":"a","rank":0,"incarnation":147,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.161:6837/3236742821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3236742821},{"type":"v1","addr":"172.21.15.161:6837","nonce":3236742821}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":11},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:36:07.448976+0000","modified":"2024-03-28T23:36:16.420228+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7061":{"gid":7061,"name":"c","rank":0,"incarnation":145,"state":"up:active","state_seq":9,"addr":"172.21.15.161:6836/4026332703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4026332703},{"type":"v1","addr":"172.21.15.161:6836","nonce":4026332703}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7061,"qdb_cluster":[7061]},"id":12}]} 2024-03-28T23:36:16.507 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 149 2024-03-28T23:36:16.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.508+0000 7f807d088640 1 -- 172.21.15.161:0/78444065 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f805806b4d0 msgr2=0x7f805806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:16.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.508+0000 7f807d088640 1 --2- 172.21.15.161:0/78444065 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f805806b4d0 0x7f805806d990 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f8070035000 tx=0x7f8070030000 comp rx=0 tx=0).stop 2024-03-28T23:36:16.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.508+0000 7f807d088640 1 -- 172.21.15.161:0/78444065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8078152890 msgr2=0x7f80780c3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:16.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.508+0000 7f807d088640 1 --2- 172.21.15.161:0/78444065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8078152890 0x7f80780c3db0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f806800c390 tx=0x7f806800c850 comp rx=0 tx=0).stop 2024-03-28T23:36:16.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.508+0000 7f807d088640 1 -- 172.21.15.161:0/78444065 shutdown_connections 2024-03-28T23:36:16.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.508+0000 7f807d088640 1 --2- 172.21.15.161:0/78444065 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f805806b4d0 0x7f805806d990 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:16.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.508+0000 7f807d088640 1 --2- 172.21.15.161:0/78444065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8078152890 0x7f80780c3db0 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:16.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.508+0000 7f807d088640 1 --2- 172.21.15.161:0/78444065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80780c8ed0 0x7f807814bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:16.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.508+0000 7f807d088640 1 --2- 172.21.15.161:0/78444065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80780c5950 0x7f80780c2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:16.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.508+0000 7f807d088640 1 -- 172.21.15.161:0/78444065 >> 172.21.15.161:0/78444065 conn(0x7f80780ac620 msgr2=0x7f8078151190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:16.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.508+0000 7f807d088640 1 -- 172.21.15.161:0/78444065 shutdown_connections 2024-03-28T23:36:16.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:16.508+0000 7f807d088640 1 -- 172.21.15.161:0/78444065 wait complete. 2024-03-28T23:36:16.922 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 149, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:35:58.406403+0000', 'modified': '2024-03-28T23:36:15.422299+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 158, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9183}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9183': {'gid': 9183, 'name': 'a', 'rank': 0, 'incarnation': 147, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.161:6837/3236742821', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 3236742821}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 3236742821}]}, 'join_fscid': 12, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-28T23:36:16.923 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9183:up:rejoin 2024-03-28T23:36:17.462 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-28T23:36:17.924 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:36:18.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.028+0000 7fd2f8147640 1 Processor -- start 2024-03-28T23:36:18.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.028+0000 7fd2f8147640 1 -- start start 2024-03-28T23:36:18.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.029+0000 7fd2f8147640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd2f00c5410 0x7fd2f00c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:18.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.029+0000 7fd2f8147640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2f01553a0 0x7fd2f0155850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:18.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.029+0000 7fd2f8147640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2f0158a30 con 0x7fd2f0152890 2024-03-28T23:36:18.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.029+0000 7fd2f8147640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd2f00c8480 con 0x7fd2f00c5410 2024-03-28T23:36:18.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.029+0000 7fd2f8147640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd2f00c5d90 con 0x7fd2f01553a0 2024-03-28T23:36:18.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.029+0000 7fd2f5ee3640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd2f0152890 0x7fd2f00c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41772/0 (socket says 172.21.15.161:41772) 2024-03-28T23:36:18.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.029+0000 7fd2f5ee3640 1 -- 172.21.15.161:0/2385420780 learned_addr learned my addr 172.21.15.161:0/2385420780 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:18.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.029+0000 7fd2f4ee1640 1 -- 172.21.15.161:0/2385420780 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1793012468 0 0) 0x7fd2f0158a30 con 0x7fd2f0152890 2024-03-28T23:36:18.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.029+0000 7fd2f56e2640 1 --2- 172.21.15.161:0/2385420780 >> v2:172.21.15.170:3300/0 conn(0x7fd2f00c5410 0x7fd2f00c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:18.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.029+0000 7fd2f4ee1640 1 -- 172.21.15.161:0/2385420780 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2dc003520 con 0x7fd2f0152890 2024-03-28T23:36:18.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.029+0000 7fd2f66e4640 1 --2- 172.21.15.161:0/2385420780 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2f01553a0 0x7fd2f0155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:18.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.029+0000 7fd2f4ee1640 1 -- 172.21.15.161:0/2385420780 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 416867752 0 0) 0x7fd2dc003520 con 0x7fd2f0152890 2024-03-28T23:36:18.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.029+0000 7fd2f4ee1640 1 -- 172.21.15.161:0/2385420780 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2f01553a0 msgr2=0x7fd2f0155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:18.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.029+0000 7fd2f4ee1640 1 --2- 172.21.15.161:0/2385420780 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2f01553a0 0x7fd2f0155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:18.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.029+0000 7fd2f4ee1640 1 -- 172.21.15.161:0/2385420780 >> v2:172.21.15.170:3300/0 conn(0x7fd2f00c5410 msgr2=0x7fd2f00c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:18.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.029+0000 7fd2f4ee1640 1 --2- 172.21.15.161:0/2385420780 >> v2:172.21.15.170:3300/0 conn(0x7fd2f00c5410 0x7fd2f00c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:18.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.029+0000 7fd2f4ee1640 1 -- 172.21.15.161:0/2385420780 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2f0158d10 con 0x7fd2f0152890 2024-03-28T23:36:18.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.029+0000 7fd2f4ee1640 1 -- 172.21.15.161:0/2385420780 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fd2ec001670 con 0x7fd2f0152890 2024-03-28T23:36:18.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.030+0000 7fd2f4ee1640 1 -- 172.21.15.161:0/2385420780 >> v1:172.21.15.161:6789/0 conn(0x7fd2f0152890 legacy=0x7fd2f00c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:18.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.030+0000 7fd2f4ee1640 1 --2- 172.21.15.161:0/2385420780 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2dc003a90 0x7fd2dc003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:18.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.030+0000 7fd2f4ee1640 1 --2- 172.21.15.161:0/2385420780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2dc004800 0x7fd2dc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:18.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.030+0000 7fd2f4ee1640 1 --2- 172.21.15.161:0/2385420780 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2f01553a0 0x7fd2dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:18.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.030+0000 7fd2f4ee1640 1 -- 172.21.15.161:0/2385420780 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f00c5d90 con 0x7fd2dc004800 2024-03-28T23:36:18.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.030+0000 7fd2f4ee1640 1 -- 172.21.15.161:0/2385420780 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd2dc00ba70 con 0x7fd2dc003a90 2024-03-28T23:36:18.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.030+0000 7fd2f4ee1640 1 -- 172.21.15.161:0/2385420780 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd2dc00bc40 con 0x7fd2f01553a0 2024-03-28T23:36:18.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.030+0000 7fd2f56e2640 1 --2- 172.21.15.161:0/2385420780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2dc004800 0x7fd2dc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:18.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.030+0000 7fd2f5ee3640 1 --2- 172.21.15.161:0/2385420780 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2dc003a90 0x7fd2dc003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:18.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.030+0000 7fd2f66e4640 1 --2- 172.21.15.161:0/2385420780 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2f01553a0 0x7fd2dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:18.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.030+0000 7fd2f56e2640 1 -- 172.21.15.161:0/2385420780 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2f01553a0 msgr2=0x7fd2dc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:18.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.030+0000 7fd2f56e2640 1 --2- 172.21.15.161:0/2385420780 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2f01553a0 0x7fd2dc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:18.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.030+0000 7fd2f56e2640 1 -- 172.21.15.161:0/2385420780 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2dc003a90 msgr2=0x7fd2dc003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:18.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.030+0000 7fd2f56e2640 1 --2- 172.21.15.161:0/2385420780 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2dc003a90 0x7fd2dc003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:18.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.030+0000 7fd2f56e2640 1 -- 172.21.15.161:0/2385420780 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2dc003520 con 0x7fd2dc004800 2024-03-28T23:36:18.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.031+0000 7fd2f56e2640 1 --2- 172.21.15.161:0/2385420780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2dc004800 0x7fd2dc004c80 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fd2f00ad1f0 tx=0x7fd2dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edf5636cdad34abb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:18.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.031+0000 7fd2f4ee1640 1 -- 172.21.15.161:0/2385420780 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0009680 con 0x7fd2dc004800 2024-03-28T23:36:18.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.031+0000 7fd2f4ee1640 1 -- 172.21.15.161:0/2385420780 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2e0005990 con 0x7fd2dc004800 2024-03-28T23:36:18.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.031+0000 7fd2f4ee1640 1 -- 172.21.15.161:0/2385420780 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0005d40 con 0x7fd2dc004800 2024-03-28T23:36:18.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.031+0000 7fd2f8147640 1 -- 172.21.15.161:0/2385420780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2dc004800 msgr2=0x7fd2dc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:18.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.031+0000 7fd2f8147640 1 --2- 172.21.15.161:0/2385420780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2dc004800 0x7fd2dc004c80 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fd2f00ad1f0 tx=0x7fd2dc005830 comp rx=0 tx=0).stop 2024-03-28T23:36:18.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.031+0000 7fd2f66e4640 1 -- 172.21.15.161:0/2385420780 reap_dead start 2024-03-28T23:36:18.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.031+0000 7fd2f8147640 1 -- 172.21.15.161:0/2385420780 shutdown_connections 2024-03-28T23:36:18.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.031+0000 7fd2f8147640 1 -- 172.21.15.161:0/2385420780 >> 172.21.15.161:0/2385420780 conn(0x7fd2f00ac620 msgr2=0x7fd2f00aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:18.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.032+0000 7fd2f8147640 1 -- 172.21.15.161:0/2385420780 shutdown_connections 2024-03-28T23:36:18.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.032+0000 7fd2f8147640 1 -- 172.21.15.161:0/2385420780 wait complete. 2024-03-28T23:36:18.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.033+0000 7fd2f8147640 1 Processor -- start 2024-03-28T23:36:18.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.033+0000 7fd2f8147640 1 -- start start 2024-03-28T23:36:18.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.033+0000 7fd2f8147640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2dc004800 0x7fd2f00ab820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:18.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.033+0000 7fd2f8147640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2f014d870 0x7fd2f014f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:18.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.033+0000 7fd2f8147640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2f00c11e0 0x7fd2f014f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:18.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.033+0000 7fd2f8147640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f01592d0 con 0x7fd2dc004800 2024-03-28T23:36:18.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.033+0000 7fd2f8147640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f0159640 con 0x7fd2f014d870 2024-03-28T23:36:18.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.033+0000 7fd2f8147640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd2f00c90a0 con 0x7fd2f00c11e0 2024-03-28T23:36:18.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.033+0000 7fd2f5ee3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2dc004800 0x7fd2f00ab820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:18.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.033+0000 7fd2f5ee3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2dc004800 0x7fd2f00ab820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46790/0 (socket says 172.21.15.161:46790) 2024-03-28T23:36:18.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.033+0000 7fd2f5ee3640 1 -- 172.21.15.161:0/1093059603 learned_addr learned my addr 172.21.15.161:0/1093059603 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:18.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.034+0000 7fd2f5ee3640 1 -- 172.21.15.161:0/1093059603 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2f00c11e0 msgr2=0x7fd2f014f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:18.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.034+0000 7fd2f5ee3640 1 --2- 172.21.15.161:0/1093059603 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2f00c11e0 0x7fd2f014f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:18.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.034+0000 7fd2f5ee3640 1 -- 172.21.15.161:0/1093059603 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2f014d870 msgr2=0x7fd2f014f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:18.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.034+0000 7fd2f56e2640 1 --2- 172.21.15.161:0/1093059603 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2f014d870 0x7fd2f014f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:18.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.034+0000 7fd2f5ee3640 1 --2- 172.21.15.161:0/1093059603 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2f014d870 0x7fd2f014f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:18.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.034+0000 7fd2f5ee3640 1 -- 172.21.15.161:0/1093059603 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2f01eed90 con 0x7fd2dc004800 2024-03-28T23:36:18.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.034+0000 7fd2f56e2640 1 --2- 172.21.15.161:0/1093059603 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2f014d870 0x7fd2f014f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:36:18.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.034+0000 7fd2f5ee3640 1 --2- 172.21.15.161:0/1093059603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2dc004800 0x7fd2f00ab820 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec004e10 tx=0x7fd2ec007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:18.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.034+0000 7fd2e67fc640 1 -- 172.21.15.161:0/1093059603 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec00f7b0 con 0x7fd2dc004800 2024-03-28T23:36:18.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.034+0000 7fd2e67fc640 1 -- 172.21.15.161:0/1093059603 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2ec018070 con 0x7fd2dc004800 2024-03-28T23:36:18.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.034+0000 7fd2f8147640 1 -- 172.21.15.161:0/1093059603 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2f01eeab0 con 0x7fd2dc004800 2024-03-28T23:36:18.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.034+0000 7fd2e67fc640 1 -- 172.21.15.161:0/1093059603 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec00fb90 con 0x7fd2dc004800 2024-03-28T23:36:18.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.034+0000 7fd2f8147640 1 -- 172.21.15.161:0/1093059603 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2f014d0f0 con 0x7fd2dc004800 2024-03-28T23:36:18.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.035+0000 7fd2f8147640 1 -- 172.21.15.161:0/1093059603 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2f004f4b0 con 0x7fd2dc004800 2024-03-28T23:36:18.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.036+0000 7fd2e67fc640 1 -- 172.21.15.161:0/1093059603 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd2ec00fd30 con 0x7fd2dc004800 2024-03-28T23:36:18.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.036+0000 7fd2e67fc640 1 --2- 172.21.15.161:0/1093059603 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd2d406b500 0x7fd2d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:18.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.036+0000 7fd2e67fc640 1 -- 172.21.15.161:0/1093059603 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 7215+0+0 (secure 0 0 0) 0x7fd2ec08a610 con 0x7fd2dc004800 2024-03-28T23:36:18.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.036+0000 7fd2f56e2640 1 --2- 172.21.15.161:0/1093059603 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd2d406b500 0x7fd2d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:18.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.037+0000 7fd2f56e2640 1 --2- 172.21.15.161:0/1093059603 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd2d406b500 0x7fd2d406d9c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0035000 tx=0x7fd2e0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:18.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.041+0000 7fd2e67fc640 1 -- 172.21.15.161:0/1093059603 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd2ec058f40 con 0x7fd2dc004800 2024-03-28T23:36:18.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.251+0000 7fd2f8147640 1 -- 172.21.15.161:0/1093059603 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2f014fba0 con 0x7fd2dc004800 2024-03-28T23:36:18.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.252+0000 7fd2e67fc640 1 -- 172.21.15.161:0/1093059603 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) ==== 95+0+5687 (secure 0 0 0) 0x7fd2ec058840 con 0x7fd2dc004800 2024-03-28T23:36:18.253 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:18.253 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":151,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/557522063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":557522063},{"type":"v1","addr":"172.21.15.170:6835","nonce":557522063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:58.406403+0000","modified":"2024-03-28T23:36:17.431405+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":159,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7283":{"gid":7283,"name":"b","rank":0,"incarnation":151,"state":"up:replay","state_seq":1,"addr":"172.21.15.170:6834/2996092857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":2996092857},{"type":"v1","addr":"172.21.15.170:6834","nonce":2996092857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":11},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:36:07.448976+0000","modified":"2024-03-28T23:36:16.420228+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7061":{"gid":7061,"name":"c","rank":0,"incarnation":145,"state":"up:active","state_seq":9,"addr":"172.21.15.161:6836/4026332703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4026332703},{"type":"v1","addr":"172.21.15.161:6836","nonce":4026332703}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7061,"qdb_cluster":[7061]},"id":12}]} 2024-03-28T23:36:18.253 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 151 2024-03-28T23:36:18.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.254+0000 7fd2f8147640 1 -- 172.21.15.161:0/1093059603 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd2d406b500 msgr2=0x7fd2d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:18.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.254+0000 7fd2f8147640 1 --2- 172.21.15.161:0/1093059603 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd2d406b500 0x7fd2d406d9c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0035000 tx=0x7fd2e0030000 comp rx=0 tx=0).stop 2024-03-28T23:36:18.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.254+0000 7fd2f8147640 1 -- 172.21.15.161:0/1093059603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2dc004800 msgr2=0x7fd2f00ab820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:18.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.254+0000 7fd2f8147640 1 --2- 172.21.15.161:0/1093059603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2dc004800 0x7fd2f00ab820 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec004e10 tx=0x7fd2ec007430 comp rx=0 tx=0).stop 2024-03-28T23:36:18.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.254+0000 7fd2f8147640 1 -- 172.21.15.161:0/1093059603 shutdown_connections 2024-03-28T23:36:18.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.254+0000 7fd2f8147640 1 --2- 172.21.15.161:0/1093059603 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd2d406b500 0x7fd2d406d9c0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:18.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.254+0000 7fd2f8147640 1 --2- 172.21.15.161:0/1093059603 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2f00c11e0 0x7fd2f014f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:18.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.254+0000 7fd2f8147640 1 --2- 172.21.15.161:0/1093059603 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2f014d870 0x7fd2f014f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:18.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.254+0000 7fd2f8147640 1 --2- 172.21.15.161:0/1093059603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2dc004800 0x7fd2f00ab820 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:18.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.254+0000 7fd2f8147640 1 -- 172.21.15.161:0/1093059603 >> 172.21.15.161:0/1093059603 conn(0x7fd2f00ac620 msgr2=0x7fd2f00caa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:18.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.254+0000 7fd2f8147640 1 -- 172.21.15.161:0/1093059603 shutdown_connections 2024-03-28T23:36:18.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:18.254+0000 7fd2f8147640 1 -- 172.21.15.161:0/1093059603 wait complete. 2024-03-28T23:36:18.668 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 151, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:35:58.406403+0000', 'modified': '2024-03-28T23:36:17.431405+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 159, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7283}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7283': {'gid': 7283, 'name': 'b', 'rank': 0, 'incarnation': 151, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.170:6834/2996092857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 2996092857}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 2996092857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-28T23:36:18.668 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7283:up:replay 2024-03-28T23:36:19.337 INFO:tasks.ceph.mds.c.smithi161.stderr:2024-03-28T23:36:19.337+0000 7f7ea652d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:36:19.670 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:36:19.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.772+0000 7fdaa8180640 1 Processor -- start 2024-03-28T23:36:19.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.772+0000 7fdaa8180640 1 -- start start 2024-03-28T23:36:19.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.773+0000 7fdaa8180640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdaa00c5410 0x7fdaa00c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:19.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.773+0000 7fdaa8180640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdaa01553a0 0x7fdaa0155820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:19.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.773+0000 7fdaa8180640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdaa0158a30 con 0x7fdaa0152890 2024-03-28T23:36:19.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.773+0000 7fdaa8180640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fdaa00c8480 con 0x7fdaa01553a0 2024-03-28T23:36:19.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.773+0000 7fdaa8180640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdaa00c5d90 con 0x7fdaa00c5410 2024-03-28T23:36:19.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.773+0000 7fdaa671d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdaa01553a0 0x7fdaa0155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:19.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.773+0000 7fdaa671d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdaa01553a0 0x7fdaa0155820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46792/0 (socket says 172.21.15.161:46792) 2024-03-28T23:36:19.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.773+0000 7fdaa671d640 1 -- 172.21.15.161:0/768280437 learned_addr learned my addr 172.21.15.161:0/768280437 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:19.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.774+0000 7fdaa571b640 1 --2- 172.21.15.161:0/768280437 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdaa00c5410 0x7fdaa00c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:19.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.774+0000 7fdaa671d640 1 -- 172.21.15.161:0/768280437 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdaa00c5410 msgr2=0x7fdaa00c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:19.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.774+0000 7fdaa671d640 1 --2- 172.21.15.161:0/768280437 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdaa00c5410 0x7fdaa00c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:19.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.774+0000 7fdaa671d640 1 -- 172.21.15.161:0/768280437 >> v1:172.21.15.170:6789/0 conn(0x7fdaa0152890 legacy=0x7fdaa00c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:19.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.774+0000 7fdaa671d640 1 -- 172.21.15.161:0/768280437 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa0158d10 con 0x7fdaa01553a0 2024-03-28T23:36:19.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.774+0000 7fdaa571b640 1 --2- 172.21.15.161:0/768280437 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdaa00c5410 0x7fdaa00c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:36:19.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.774+0000 7fdaa671d640 1 --2- 172.21.15.161:0/768280437 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaa01553a0 0x7fdaa0155820 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fda98009e50 tx=0x7fda9802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5319533efaeca90c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:19.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.775+0000 7fdaa4f1a640 1 -- 172.21.15.161:0/768280437 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9802c9e0 con 0x7fdaa01553a0 2024-03-28T23:36:19.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.775+0000 7fdaa4f1a640 1 -- 172.21.15.161:0/768280437 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda9802cb80 con 0x7fdaa01553a0 2024-03-28T23:36:19.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.775+0000 7fdaa4f1a640 1 -- 172.21.15.161:0/768280437 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda980343f0 con 0x7fdaa01553a0 2024-03-28T23:36:19.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.775+0000 7fdaa8180640 1 -- 172.21.15.161:0/768280437 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaa01553a0 msgr2=0x7fdaa0155820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:19.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.775+0000 7fdaa8180640 1 --2- 172.21.15.161:0/768280437 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaa01553a0 0x7fdaa0155820 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fda98009e50 tx=0x7fda9802be40 comp rx=0 tx=0).stop 2024-03-28T23:36:19.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.775+0000 7fdaa8180640 1 -- 172.21.15.161:0/768280437 shutdown_connections 2024-03-28T23:36:19.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.775+0000 7fdaa8180640 1 --2- 172.21.15.161:0/768280437 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaa01553a0 0x7fdaa0155820 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:19.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.775+0000 7fdaa8180640 1 --2- 172.21.15.161:0/768280437 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdaa00c5410 0x7fdaa00c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:19.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.775+0000 7fdaa8180640 1 -- 172.21.15.161:0/768280437 >> 172.21.15.161:0/768280437 conn(0x7fdaa00ac620 msgr2=0x7fdaa00aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:19.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.775+0000 7fdaa8180640 1 -- 172.21.15.161:0/768280437 shutdown_connections 2024-03-28T23:36:19.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.775+0000 7fdaa8180640 1 -- 172.21.15.161:0/768280437 wait complete. 2024-03-28T23:36:19.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.776+0000 7fdaa8180640 1 Processor -- start 2024-03-28T23:36:19.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.777+0000 7fdaa8180640 1 -- start start 2024-03-28T23:36:19.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.777+0000 7fdaa8180640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaa00c5410 0x7fdaa014b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:19.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.777+0000 7fdaa8180640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdaa0152890 0x7fdaa0149040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:19.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.777+0000 7fdaa8180640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaa01553a0 0x7fdaa0149580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:19.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.777+0000 7fdaa8180640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa01592d0 con 0x7fdaa00c5410 2024-03-28T23:36:19.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.777+0000 7fdaa8180640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa0159640 con 0x7fdaa01553a0 2024-03-28T23:36:19.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.777+0000 7fdaa8180640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdaa00c90a0 con 0x7fdaa0152890 2024-03-28T23:36:19.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.777+0000 7fdaa5f1c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaa00c5410 0x7fdaa014b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:19.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.777+0000 7fdaa5f1c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaa00c5410 0x7fdaa014b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:46804/0 (socket says 172.21.15.161:46804) 2024-03-28T23:36:19.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.777+0000 7fdaa5f1c640 1 -- 172.21.15.161:0/393846323 learned_addr learned my addr 172.21.15.161:0/393846323 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:19.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.778+0000 7fdaa5f1c640 1 -- 172.21.15.161:0/393846323 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdaa0152890 msgr2=0x7fdaa0149040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:19.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.778+0000 7fdaa5f1c640 1 --2- 172.21.15.161:0/393846323 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdaa0152890 0x7fdaa0149040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:19.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.778+0000 7fdaa5f1c640 1 -- 172.21.15.161:0/393846323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaa01553a0 msgr2=0x7fdaa0149580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:19.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.778+0000 7fdaa5f1c640 1 --2- 172.21.15.161:0/393846323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaa01553a0 0x7fdaa0149580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:19.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.778+0000 7fdaa5f1c640 1 -- 172.21.15.161:0/393846323 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa01517c0 con 0x7fdaa00c5410 2024-03-28T23:36:19.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.778+0000 7fdaa5f1c640 1 --2- 172.21.15.161:0/393846323 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaa00c5410 0x7fdaa014b130 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fda9c00c430 tx=0x7fda9c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:19.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.778+0000 7fda967fc640 1 -- 172.21.15.161:0/393846323 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c011790 con 0x7fdaa00c5410 2024-03-28T23:36:19.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.778+0000 7fda967fc640 1 -- 172.21.15.161:0/393846323 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda9c017070 con 0x7fdaa00c5410 2024-03-28T23:36:19.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.778+0000 7fdaa8180640 1 -- 172.21.15.161:0/393846323 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaa01ee340 con 0x7fdaa00c5410 2024-03-28T23:36:19.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.778+0000 7fda967fc640 1 -- 172.21.15.161:0/393846323 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c011bb0 con 0x7fdaa00c5410 2024-03-28T23:36:19.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.778+0000 7fdaa8180640 1 -- 172.21.15.161:0/393846323 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaa01ee900 con 0x7fdaa00c5410 2024-03-28T23:36:19.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.779+0000 7fdaa8180640 1 -- 172.21.15.161:0/393846323 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaa004f4b0 con 0x7fdaa00c5410 2024-03-28T23:36:19.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.779+0000 7fda967fc640 1 -- 172.21.15.161:0/393846323 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fda9c01c460 con 0x7fdaa00c5410 2024-03-28T23:36:19.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.780+0000 7fda967fc640 1 --2- 172.21.15.161:0/393846323 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fda8406b4d0 0x7fda8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:19.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.780+0000 7fdaa571b640 1 --2- 172.21.15.161:0/393846323 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fda8406b4d0 0x7fda8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:19.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.780+0000 7fda967fc640 1 -- 172.21.15.161:0/393846323 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 7215+0+0 (secure 0 0 0) 0x7fda9c08a5b0 con 0x7fdaa00c5410 2024-03-28T23:36:19.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.780+0000 7fdaa571b640 1 --2- 172.21.15.161:0/393846323 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fda8406b4d0 0x7fda8406d990 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fdaa01588f0 tx=0x7fda90008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:19.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.785+0000 7fda967fc640 1 -- 172.21.15.161:0/393846323 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fda9c058ee0 con 0x7fdaa00c5410 2024-03-28T23:36:19.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.992+0000 7fdaa8180640 1 -- 172.21.15.161:0/393846323 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdaa00c4b90 con 0x7fdaa00c5410 2024-03-28T23:36:19.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.992+0000 7fda967fc640 1 -- 172.21.15.161:0/393846323 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) ==== 95+0+6513 (secure 0 0 0) 0x7fda9c0587e0 con 0x7fdaa00c5410 2024-03-28T23:36:19.993 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:19.993 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":153,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/557522063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":557522063},{"type":"v1","addr":"172.21.15.170:6835","nonce":557522063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":9216,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/789976100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":789976100},{"type":"v1","addr":"172.21.15.161:6837","nonce":789976100}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:58.406403+0000","modified":"2024-03-28T23:36:18.438226+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":159,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7283":{"gid":7283,"name":"b","rank":0,"incarnation":151,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.170:6834/2996092857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":2996092857},{"type":"v1","addr":"172.21.15.170:6834","nonce":2996092857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":11},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:36:07.448976+0000","modified":"2024-03-28T23:36:16.420228+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7061":{"gid":7061,"name":"c","rank":0,"incarnation":145,"state":"up:active","state_seq":9,"addr":"172.21.15.161:6836/4026332703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4026332703},{"type":"v1","addr":"172.21.15.161:6836","nonce":4026332703}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7061,"qdb_cluster":[7061]},"id":12}]} 2024-03-28T23:36:19.993 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 153 2024-03-28T23:36:19.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.994+0000 7fdaa8180640 1 -- 172.21.15.161:0/393846323 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fda8406b4d0 msgr2=0x7fda8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:19.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.994+0000 7fdaa8180640 1 --2- 172.21.15.161:0/393846323 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fda8406b4d0 0x7fda8406d990 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fdaa01588f0 tx=0x7fda90008040 comp rx=0 tx=0).stop 2024-03-28T23:36:19.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.994+0000 7fdaa8180640 1 -- 172.21.15.161:0/393846323 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaa00c5410 msgr2=0x7fdaa014b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:19.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.994+0000 7fdaa8180640 1 --2- 172.21.15.161:0/393846323 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaa00c5410 0x7fdaa014b130 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fda9c00c430 tx=0x7fda9c00c8f0 comp rx=0 tx=0).stop 2024-03-28T23:36:19.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.994+0000 7fdaa8180640 1 -- 172.21.15.161:0/393846323 shutdown_connections 2024-03-28T23:36:19.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.994+0000 7fdaa8180640 1 --2- 172.21.15.161:0/393846323 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fda8406b4d0 0x7fda8406d990 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:19.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.994+0000 7fdaa8180640 1 --2- 172.21.15.161:0/393846323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdaa01553a0 0x7fdaa0149580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:19.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.994+0000 7fdaa8180640 1 --2- 172.21.15.161:0/393846323 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdaa0152890 0x7fdaa0149040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:19.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.994+0000 7fdaa8180640 1 --2- 172.21.15.161:0/393846323 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaa00c5410 0x7fdaa014b130 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:19.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.994+0000 7fdaa8180640 1 -- 172.21.15.161:0/393846323 >> 172.21.15.161:0/393846323 conn(0x7fdaa00ac620 msgr2=0x7fdaa00bde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:19.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.994+0000 7fdaa8180640 1 -- 172.21.15.161:0/393846323 shutdown_connections 2024-03-28T23:36:19.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:19.994+0000 7fdaa8180640 1 -- 172.21.15.161:0/393846323 wait complete. 2024-03-28T23:36:20.409 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 153, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:35:58.406403+0000', 'modified': '2024-03-28T23:36:18.438226+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 159, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7283}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7283': {'gid': 7283, 'name': 'b', 'rank': 0, 'incarnation': 151, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.170:6834/2996092857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 2996092857}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 2996092857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-28T23:36:20.409 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7283:up:rejoin 2024-03-28T23:36:21.410 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:36:21.516 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.516+0000 7f11cc9e5640 1 Processor -- start 2024-03-28T23:36:21.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.516+0000 7f11cc9e5640 1 -- start start 2024-03-28T23:36:21.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.516+0000 7f11cc9e5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f11c8152890 0x7f11c80c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:21.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.516+0000 7f11cc9e5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f11c80c5b20 0x7f11c81553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:21.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.516+0000 7f11cc9e5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11c8158a30 con 0x7f11c80c8ed0 2024-03-28T23:36:21.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.516+0000 7f11cc9e5640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f11c80c8480 con 0x7f11c8152890 2024-03-28T23:36:21.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.516+0000 7f11cc9e5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f11c8159640 con 0x7f11c80c5b20 2024-03-28T23:36:21.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.517+0000 7f11c659c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f11c8152890 0x7f11c80c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:21.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.517+0000 7f11c5d9b640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f11c80c8ed0 0x7f11c80c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:39170/0 (socket says 172.21.15.161:39170) 2024-03-28T23:36:21.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.517+0000 7f11c5d9b640 1 -- 172.21.15.161:0/1501509081 learned_addr learned my addr 172.21.15.161:0/1501509081 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:21.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.517+0000 7f11c6d9d640 1 --2- 172.21.15.161:0/1501509081 >> v2:172.21.15.170:3300/0 conn(0x7f11c80c5b20 0x7f11c81553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:21.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.517+0000 7f11c659c640 1 -- 172.21.15.161:0/1501509081 >> v2:172.21.15.170:3300/0 conn(0x7f11c80c5b20 msgr2=0x7f11c81553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:21.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.517+0000 7f11c659c640 1 --2- 172.21.15.161:0/1501509081 >> v2:172.21.15.170:3300/0 conn(0x7f11c80c5b20 0x7f11c81553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:21.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.517+0000 7f11c659c640 1 -- 172.21.15.161:0/1501509081 >> v1:172.21.15.161:6789/0 conn(0x7f11c80c8ed0 legacy=0x7f11c80c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:21.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.517+0000 7f11c659c640 1 -- 172.21.15.161:0/1501509081 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11c8158d10 con 0x7f11c8152890 2024-03-28T23:36:21.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.517+0000 7f11c559a640 1 -- 172.21.15.161:0/1501509081 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3643922850 0 0) 0x7f11c8158a30 con 0x7f11c80c8ed0 2024-03-28T23:36:21.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.518+0000 7f11c659c640 1 --2- 172.21.15.161:0/1501509081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11c8152890 0x7f11c80c8990 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f11c80ad480 tx=0x7f11b802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b1e200fa11e86a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:21.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.518+0000 7f11c559a640 1 -- 172.21.15.161:0/1501509081 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b8038070 con 0x7f11c8152890 2024-03-28T23:36:21.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.518+0000 7f11c559a640 1 -- 172.21.15.161:0/1501509081 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11b8031710 con 0x7f11c8152890 2024-03-28T23:36:21.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.518+0000 7f11c559a640 1 -- 172.21.15.161:0/1501509081 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b8031ae0 con 0x7f11c8152890 2024-03-28T23:36:21.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.519+0000 7f11cc9e5640 1 -- 172.21.15.161:0/1501509081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11c8152890 msgr2=0x7f11c80c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:21.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.519+0000 7f11cc9e5640 1 --2- 172.21.15.161:0/1501509081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11c8152890 0x7f11c80c8990 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f11c80ad480 tx=0x7f11b802b7b0 comp rx=0 tx=0).stop 2024-03-28T23:36:21.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.519+0000 7f11cc9e5640 1 -- 172.21.15.161:0/1501509081 shutdown_connections 2024-03-28T23:36:21.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.519+0000 7f11cc9e5640 1 --2- 172.21.15.161:0/1501509081 >> v2:172.21.15.170:3300/0 conn(0x7f11c80c5b20 0x7f11c81553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:21.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.519+0000 7f11cc9e5640 1 --2- 172.21.15.161:0/1501509081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11c8152890 0x7f11c80c8990 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:21.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.519+0000 7f11cc9e5640 1 -- 172.21.15.161:0/1501509081 >> 172.21.15.161:0/1501509081 conn(0x7f11c80ac620 msgr2=0x7f11c80aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:21.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.519+0000 7f11cc9e5640 1 -- 172.21.15.161:0/1501509081 shutdown_connections 2024-03-28T23:36:21.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.519+0000 7f11cc9e5640 1 -- 172.21.15.161:0/1501509081 wait complete. 2024-03-28T23:36:21.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.520+0000 7f11cc9e5640 1 Processor -- start 2024-03-28T23:36:21.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.520+0000 7f11cc9e5640 1 -- start start 2024-03-28T23:36:21.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.521+0000 7f11cc9e5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11c80c5b20 0x7f11c8149c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:21.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.521+0000 7f11cc9e5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11c80c8ed0 0x7f11c814a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:21.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.521+0000 7f11cc9e5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11c8152890 0x7f11c81ee150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:21.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.521+0000 7f11cc9e5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f11c8152c70 con 0x7f11c8152890 2024-03-28T23:36:21.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.521+0000 7f11cc9e5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f11c81592d0 con 0x7f11c80c5b20 2024-03-28T23:36:21.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.521+0000 7f11cc9e5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f11c8152df0 con 0x7f11c80c8ed0 2024-03-28T23:36:21.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.521+0000 7f11c6d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11c8152890 0x7f11c81ee150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:21.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.521+0000 7f11c659c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11c80c5b20 0x7f11c8149c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:21.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.521+0000 7f11c6d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11c8152890 0x7f11c81ee150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42088/0 (socket says 172.21.15.161:42088) 2024-03-28T23:36:21.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.521+0000 7f11c6d9d640 1 -- 172.21.15.161:0/2378726703 learned_addr learned my addr 172.21.15.161:0/2378726703 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:21.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.521+0000 7f11c5d9b640 1 --2- 172.21.15.161:0/2378726703 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11c80c8ed0 0x7f11c814a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:21.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.522+0000 7f11c6d9d640 1 -- 172.21.15.161:0/2378726703 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11c80c8ed0 msgr2=0x7f11c814a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:21.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.522+0000 7f11c6d9d640 1 --2- 172.21.15.161:0/2378726703 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11c80c8ed0 0x7f11c814a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:21.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.522+0000 7f11c6d9d640 1 -- 172.21.15.161:0/2378726703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11c80c5b20 msgr2=0x7f11c8149c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:21.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.522+0000 7f11c6d9d640 1 --2- 172.21.15.161:0/2378726703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11c80c5b20 0x7f11c8149c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:21.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.522+0000 7f11c6d9d640 1 -- 172.21.15.161:0/2378726703 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11c8153730 con 0x7f11c8152890 2024-03-28T23:36:21.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.522+0000 7f11c5d9b640 1 --2- 172.21.15.161:0/2378726703 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11c80c8ed0 0x7f11c814a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:21.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.522+0000 7f11c659c640 1 --2- 172.21.15.161:0/2378726703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11c80c5b20 0x7f11c8149c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:21.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.522+0000 7f11c6d9d640 1 --2- 172.21.15.161:0/2378726703 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11c8152890 0x7f11c81ee150 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f11c000efc0 tx=0x7f11c000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:21.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.522+0000 7f11b6ffd640 1 -- 172.21.15.161:0/2378726703 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c0018020 con 0x7f11c8152890 2024-03-28T23:36:21.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.522+0000 7f11b6ffd640 1 -- 172.21.15.161:0/2378726703 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11c0014480 con 0x7f11c8152890 2024-03-28T23:36:21.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.522+0000 7f11cc9e5640 1 -- 172.21.15.161:0/2378726703 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11c8153a10 con 0x7f11c8152890 2024-03-28T23:36:21.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.522+0000 7f11b6ffd640 1 -- 172.21.15.161:0/2378726703 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c0014870 con 0x7f11c8152890 2024-03-28T23:36:21.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.522+0000 7f11cc9e5640 1 -- 172.21.15.161:0/2378726703 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11c80c0d30 con 0x7f11c8152890 2024-03-28T23:36:21.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.523+0000 7f11cc9e5640 1 -- 172.21.15.161:0/2378726703 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11c804f4b0 con 0x7f11c8152890 2024-03-28T23:36:21.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.524+0000 7f11b6ffd640 1 -- 172.21.15.161:0/2378726703 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f11c0005d10 con 0x7f11c8152890 2024-03-28T23:36:21.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.524+0000 7f11b6ffd640 1 --2- 172.21.15.161:0/2378726703 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f11a806b500 0x7f11a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:21.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.524+0000 7f11b6ffd640 1 -- 172.21.15.161:0/2378726703 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 7215+0+0 (secure 0 0 0) 0x7f11c008b4a0 con 0x7f11c8152890 2024-03-28T23:36:21.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.524+0000 7f11c659c640 1 --2- 172.21.15.161:0/2378726703 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f11a806b500 0x7f11a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:21.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.525+0000 7f11c659c640 1 --2- 172.21.15.161:0/2378726703 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f11a806b500 0x7f11a806d9c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f11b8035000 tx=0x7f11b8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:21.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.529+0000 7f11b6ffd640 1 -- 172.21.15.161:0/2378726703 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f11c0022030 con 0x7f11c8152890 2024-03-28T23:36:21.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.745+0000 7f11cc9e5640 1 -- 172.21.15.161:0/2378726703 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11c8156570 con 0x7f11c8152890 2024-03-28T23:36:21.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.746+0000 7f11b6ffd640 1 -- 172.21.15.161:0/2378726703 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+6520 (secure 0 0 0) 0x7f11c0010040 con 0x7f11c8152890 2024-03-28T23:36:21.746 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:21.747 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":154,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/557522063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":557522063},{"type":"v1","addr":"172.21.15.170:6835","nonce":557522063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":9216,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/789976100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":789976100},{"type":"v1","addr":"172.21.15.161:6837","nonce":789976100}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:58.406403+0000","modified":"2024-03-28T23:36:20.440473+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":159,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7283":{"gid":7283,"name":"b","rank":0,"incarnation":151,"state":"up:active","state_seq":4,"addr":"172.21.15.170:6834/2996092857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":2996092857},{"type":"v1","addr":"172.21.15.170:6834","nonce":2996092857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7283,"qdb_cluster":[7283]},"id":11},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:36:07.448976+0000","modified":"2024-03-28T23:36:16.420228+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7061":{"gid":7061,"name":"c","rank":0,"incarnation":145,"state":"up:active","state_seq":9,"addr":"172.21.15.161:6836/4026332703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4026332703},{"type":"v1","addr":"172.21.15.161:6836","nonce":4026332703}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7061,"qdb_cluster":[7061]},"id":12}]} 2024-03-28T23:36:21.747 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 154 2024-03-28T23:36:21.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.748+0000 7f11cc9e5640 1 -- 172.21.15.161:0/2378726703 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f11a806b500 msgr2=0x7f11a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:21.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.748+0000 7f11cc9e5640 1 --2- 172.21.15.161:0/2378726703 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f11a806b500 0x7f11a806d9c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f11b8035000 tx=0x7f11b8008030 comp rx=0 tx=0).stop 2024-03-28T23:36:21.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.748+0000 7f11cc9e5640 1 -- 172.21.15.161:0/2378726703 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11c8152890 msgr2=0x7f11c81ee150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:21.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.748+0000 7f11cc9e5640 1 --2- 172.21.15.161:0/2378726703 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11c8152890 0x7f11c81ee150 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f11c000efc0 tx=0x7f11c000c490 comp rx=0 tx=0).stop 2024-03-28T23:36:21.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.748+0000 7f11cc9e5640 1 -- 172.21.15.161:0/2378726703 shutdown_connections 2024-03-28T23:36:21.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.748+0000 7f11cc9e5640 1 --2- 172.21.15.161:0/2378726703 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f11a806b500 0x7f11a806d9c0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:21.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.748+0000 7f11cc9e5640 1 --2- 172.21.15.161:0/2378726703 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11c8152890 0x7f11c81ee150 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:21.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.748+0000 7f11cc9e5640 1 --2- 172.21.15.161:0/2378726703 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11c80c8ed0 0x7f11c814a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:21.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.748+0000 7f11cc9e5640 1 --2- 172.21.15.161:0/2378726703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11c80c5b20 0x7f11c8149c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:21.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.748+0000 7f11cc9e5640 1 -- 172.21.15.161:0/2378726703 >> 172.21.15.161:0/2378726703 conn(0x7f11c80ac620 msgr2=0x7f11c80740d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:21.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.748+0000 7f11cc9e5640 1 -- 172.21.15.161:0/2378726703 shutdown_connections 2024-03-28T23:36:21.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:21.748+0000 7f11cc9e5640 1 -- 172.21.15.161:0/2378726703 wait complete. 2024-03-28T23:36:22.162 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 154, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:35:58.406403+0000', 'modified': '2024-03-28T23:36:20.440473+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 159, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7283}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7283': {'gid': 7283, 'name': 'b', 'rank': 0, 'incarnation': 151, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.170:6834/2996092857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 2996092857}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 2996092857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7283, 'qdb_cluster': [7283]} 2024-03-28T23:36:22.162 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:36:22.162 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T23:36:22.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.265+0000 7f11534f4640 1 Processor -- start 2024-03-28T23:36:22.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.265+0000 7f11534f4640 1 -- start start 2024-03-28T23:36:22.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.266+0000 7f11534f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f114c152890 0x7f114c0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:22.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.266+0000 7f11534f4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f114c0c5b20 0x7f114c1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:22.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.266+0000 7f11534f4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f114c158a30 con 0x7f114c0c8ed0 2024-03-28T23:36:22.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.266+0000 7f11534f4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f114c0c8480 con 0x7f114c0c5b20 2024-03-28T23:36:22.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.266+0000 7f11534f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f114c159640 con 0x7f114c152890 2024-03-28T23:36:22.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.266+0000 7f1150a8f640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f114c0c8ed0 0x7f114c0c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:39182/0 (socket says 172.21.15.161:39182) 2024-03-28T23:36:22.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.266+0000 7f1150a8f640 1 -- 172.21.15.161:0/51561317 learned_addr learned my addr 172.21.15.161:0/51561317 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:22.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.266+0000 7f1151a91640 1 --2- 172.21.15.161:0/51561317 >> v2:172.21.15.170:3300/0 conn(0x7f114c0c5b20 0x7f114c1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:22.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.266+0000 7f1151290640 1 --2- 172.21.15.161:0/51561317 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f114c152890 0x7f114c0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:22.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.267+0000 7f1143fff640 1 -- 172.21.15.161:0/51561317 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2762084872 0 0) 0x7f114c158a30 con 0x7f114c0c8ed0 2024-03-28T23:36:22.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.267+0000 7f1143fff640 1 -- 172.21.15.161:0/51561317 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1138003520 con 0x7f114c0c8ed0 2024-03-28T23:36:22.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.267+0000 7f1151a91640 1 -- 172.21.15.161:0/51561317 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f114c152890 msgr2=0x7f114c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:22.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.267+0000 7f1151a91640 1 --2- 172.21.15.161:0/51561317 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f114c152890 0x7f114c0c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.267+0000 7f1151a91640 1 -- 172.21.15.161:0/51561317 >> v1:172.21.15.161:6789/0 conn(0x7f114c0c8ed0 legacy=0x7f114c0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:22.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.267+0000 7f1151a91640 1 -- 172.21.15.161:0/51561317 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f114c158d10 con 0x7f114c0c5b20 2024-03-28T23:36:22.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.267+0000 7f1151290640 1 --2- 172.21.15.161:0/51561317 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f114c152890 0x7f114c0c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:22.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.268+0000 7f1151a91640 1 --2- 172.21.15.161:0/51561317 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f114c0c5b20 0x7f114c1553a0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f114c1597c0 tx=0x7f114402bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61c44d93b67293aa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:22.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.268+0000 7f1143fff640 1 -- 172.21.15.161:0/51561317 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1144039070 con 0x7f114c0c5b20 2024-03-28T23:36:22.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.268+0000 7f1143fff640 1 -- 172.21.15.161:0/51561317 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f114402cb20 con 0x7f114c0c5b20 2024-03-28T23:36:22.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.268+0000 7f1143fff640 1 -- 172.21.15.161:0/51561317 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1144034440 con 0x7f114c0c5b20 2024-03-28T23:36:22.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.268+0000 7f11534f4640 1 -- 172.21.15.161:0/51561317 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f114c0c5b20 msgr2=0x7f114c1553a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:22.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.268+0000 7f11534f4640 1 --2- 172.21.15.161:0/51561317 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f114c0c5b20 0x7f114c1553a0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f114c1597c0 tx=0x7f114402bd70 comp rx=0 tx=0).stop 2024-03-28T23:36:22.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.268+0000 7f11534f4640 1 -- 172.21.15.161:0/51561317 shutdown_connections 2024-03-28T23:36:22.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.268+0000 7f11534f4640 1 --2- 172.21.15.161:0/51561317 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f114c0c5b20 0x7f114c1553a0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.268+0000 7f11534f4640 1 --2- 172.21.15.161:0/51561317 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f114c152890 0x7f114c0c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.268+0000 7f11534f4640 1 -- 172.21.15.161:0/51561317 >> 172.21.15.161:0/51561317 conn(0x7f114c0ac620 msgr2=0x7f114c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:22.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.268+0000 7f11534f4640 1 -- 172.21.15.161:0/51561317 shutdown_connections 2024-03-28T23:36:22.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.268+0000 7f11534f4640 1 -- 172.21.15.161:0/51561317 wait complete. 2024-03-28T23:36:22.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.269+0000 7f11534f4640 1 Processor -- start 2024-03-28T23:36:22.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.270+0000 7f11534f4640 1 -- start start 2024-03-28T23:36:22.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.270+0000 7f11534f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f114c0c5b20 0x7f114c1ea4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:22.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.270+0000 7f11534f4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f114c0c8ed0 0x7f114c1ee340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:22.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.270+0000 7f11534f4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f114c152890 0x7f114c144ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:22.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.270+0000 7f11534f4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f114c152c70 con 0x7f114c152890 2024-03-28T23:36:22.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.270+0000 7f11534f4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f114c1592d0 con 0x7f114c0c8ed0 2024-03-28T23:36:22.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.270+0000 7f11534f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f114c152df0 con 0x7f114c0c5b20 2024-03-28T23:36:22.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.270+0000 7f1151a91640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f114c152890 0x7f114c144ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:22.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.270+0000 7f1151a91640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f114c152890 0x7f114c144ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42100/0 (socket says 172.21.15.161:42100) 2024-03-28T23:36:22.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.270+0000 7f1151a91640 1 -- 172.21.15.161:0/3697357089 learned_addr learned my addr 172.21.15.161:0/3697357089 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:22.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.270+0000 7f1151290640 1 --2- 172.21.15.161:0/3697357089 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f114c0c5b20 0x7f114c1ea4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:22.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.270+0000 7f1150a8f640 1 --2- 172.21.15.161:0/3697357089 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f114c0c8ed0 0x7f114c1ee340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:22.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.271+0000 7f1151a91640 1 -- 172.21.15.161:0/3697357089 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f114c0c5b20 msgr2=0x7f114c1ea4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:22.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.271+0000 7f1151a91640 1 --2- 172.21.15.161:0/3697357089 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f114c0c5b20 0x7f114c1ea4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.271+0000 7f1151a91640 1 -- 172.21.15.161:0/3697357089 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f114c0c8ed0 msgr2=0x7f114c1ee340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:22.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.271+0000 7f1151a91640 1 --2- 172.21.15.161:0/3697357089 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f114c0c8ed0 0x7f114c1ee340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.271+0000 7f1151a91640 1 -- 172.21.15.161:0/3697357089 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f114c0c16f0 con 0x7f114c152890 2024-03-28T23:36:22.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.271+0000 7f1151290640 1 --2- 172.21.15.161:0/3697357089 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f114c0c5b20 0x7f114c1ea4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:22.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.271+0000 7f1151a91640 1 --2- 172.21.15.161:0/3697357089 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f114c152890 0x7f114c144ed0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f1144009e80 tx=0x7f1144004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:22.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.271+0000 7f1141ffb640 1 -- 172.21.15.161:0/3697357089 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1144039050 con 0x7f114c152890 2024-03-28T23:36:22.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.271+0000 7f1141ffb640 1 -- 172.21.15.161:0/3697357089 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1144045070 con 0x7f114c152890 2024-03-28T23:36:22.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.271+0000 7f11534f4640 1 -- 172.21.15.161:0/3697357089 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f114c1eaa00 con 0x7f114c152890 2024-03-28T23:36:22.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.271+0000 7f1141ffb640 1 -- 172.21.15.161:0/3697357089 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1144009450 con 0x7f114c152890 2024-03-28T23:36:22.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.271+0000 7f11534f4640 1 -- 172.21.15.161:0/3697357089 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f114c0c1130 con 0x7f114c152890 2024-03-28T23:36:22.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.273+0000 7f1141ffb640 1 -- 172.21.15.161:0/3697357089 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f114403e830 con 0x7f114c152890 2024-03-28T23:36:22.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.273+0000 7f1141ffb640 1 --2- 172.21.15.161:0/3697357089 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f113006b500 0x7f113006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:22.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.273+0000 7f1141ffb640 1 -- 172.21.15.161:0/3697357089 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 7215+0+0 (secure 0 0 0) 0x7f11440ad680 con 0x7f114c152890 2024-03-28T23:36:22.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.273+0000 7f1151290640 1 --2- 172.21.15.161:0/3697357089 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f113006b500 0x7f113006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:22.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.274+0000 7f1151290640 1 --2- 172.21.15.161:0/3697357089 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f113006b500 0x7f113006d9c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f113c009560 tx=0x7f113c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:22.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.274+0000 7f11561a8740 1 Processor -- start 2024-03-28T23:36:22.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.274+0000 7f11561a8740 1 -- start start 2024-03-28T23:36:22.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.275+0000 7f11561a8740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561224bd7db0 0x561224bd5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:22.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.275+0000 7f11561a8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561224bd8190 0x561224bd5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:22.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.275+0000 7f11561a8740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224bd6310 0x561224bd24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:22.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.275+0000 7f11561a8740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561224b3baf0 con 0x561224bd6310 2024-03-28T23:36:22.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.275+0000 7f11561a8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x561224ae49d0 con 0x561224bd8190 2024-03-28T23:36:22.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.275+0000 7f11561a8740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x561224b37b90 con 0x561224bd7db0 2024-03-28T23:36:22.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.275+0000 7f1151a91640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224bd6310 0x561224bd24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:22.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.275+0000 7f1151290640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561224bd7db0 0x561224bd5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:22.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.275+0000 7f1151a91640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224bd6310 0x561224bd24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42112/0 (socket says 172.21.15.161:42112) 2024-03-28T23:36:22.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.275+0000 7f1150a8f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561224bd8190 0x561224bd5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:22.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.275+0000 7f1151a91640 1 -- 172.21.15.161:0/2895001710 learned_addr learned my addr 172.21.15.161:0/2895001710 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:22.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.275+0000 7f1151290640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561224bd7db0 0x561224bd5890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:53926/0 (socket says 172.21.15.161:53926) 2024-03-28T23:36:22.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.275+0000 7f1151a91640 1 -- 172.21.15.161:0/2895001710 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561224bd7db0 msgr2=0x561224bd5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:22.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.275+0000 7f1151a91640 1 --2- 172.21.15.161:0/2895001710 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561224bd7db0 0x561224bd5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.275+0000 7f1151a91640 1 -- 172.21.15.161:0/2895001710 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561224bd8190 msgr2=0x561224bd5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:22.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.275+0000 7f1151a91640 1 --2- 172.21.15.161:0/2895001710 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561224bd8190 0x561224bd5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.275+0000 7f1151a91640 1 -- 172.21.15.161:0/2895001710 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612248502d0 con 0x561224bd6310 2024-03-28T23:36:22.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.275+0000 7f1151290640 1 --2- 172.21.15.161:0/2895001710 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561224bd7db0 0x561224bd5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:22.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.275+0000 7f1150a8f640 1 --2- 172.21.15.161:0/2895001710 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561224bd8190 0x561224bd5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:22.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.275+0000 7f1151a91640 1 --2- 172.21.15.161:0/2895001710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224bd6310 0x561224bd24e0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f1144077fa0 tx=0x7f1144009b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2101890d5393e9e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:22.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.275+0000 7f1136ffd640 1 -- 172.21.15.161:0/2895001710 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114407fbe0 con 0x561224bd6310 2024-03-28T23:36:22.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.275+0000 7f1136ffd640 1 -- 172.21.15.161:0/2895001710 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f114403a040 con 0x561224bd6310 2024-03-28T23:36:22.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.276+0000 7f1136ffd640 1 -- 172.21.15.161:0/2895001710 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11440841a0 con 0x561224bd6310 2024-03-28T23:36:22.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.276+0000 7f11561a8740 1 -- 172.21.15.161:0/2895001710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224bd6310 msgr2=0x561224bd24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:22.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.276+0000 7f11561a8740 1 --2- 172.21.15.161:0/2895001710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224bd6310 0x561224bd24e0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f1144077fa0 tx=0x7f1144009b90 comp rx=0 tx=0).stop 2024-03-28T23:36:22.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.276+0000 7f11561a8740 1 -- 172.21.15.161:0/2895001710 shutdown_connections 2024-03-28T23:36:22.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.276+0000 7f11561a8740 1 --2- 172.21.15.161:0/2895001710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224bd6310 0x561224bd24e0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.276+0000 7f11561a8740 1 --2- 172.21.15.161:0/2895001710 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561224bd8190 0x561224bd5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.276+0000 7f11561a8740 1 --2- 172.21.15.161:0/2895001710 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561224bd7db0 0x561224bd5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.276+0000 7f11561a8740 1 -- 172.21.15.161:0/2895001710 >> 172.21.15.161:0/2895001710 conn(0x561224bd4630 msgr2=0x561224bf9220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:22.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.276+0000 7f11561a8740 1 -- 172.21.15.161:0/2895001710 shutdown_connections 2024-03-28T23:36:22.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.276+0000 7f11561a8740 1 -- 172.21.15.161:0/2895001710 wait complete. 2024-03-28T23:36:22.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.277+0000 7f11561a8740 1 -- start start 2024-03-28T23:36:22.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.277+0000 7f11561a8740 1 Processor -- start 2024-03-28T23:36:22.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.278+0000 7f11561a8740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561224bd7db0 0x561224c24350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:22.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.278+0000 7f11561a8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561224bd8190 0x561224bd2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:22.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.278+0000 7f11561a8740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224c0ade0 0x561224bfbc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:22.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.278+0000 7f11561a8740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561224bfa090 con 0x561224c0ade0 2024-03-28T23:36:22.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.278+0000 7f11561a8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x561224af7f70 con 0x561224bd8190 2024-03-28T23:36:22.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.278+0000 7f11561a8740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x561224bf9ad0 con 0x561224bd7db0 2024-03-28T23:36:22.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.278+0000 7f11561a8740 10 client.0 _ll_register_callbacks cb 0x56122484fac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:36:22.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.279+0000 7f1151a91640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224c0ade0 0x561224bfbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:22.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.279+0000 7f1151a91640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224c0ade0 0x561224bfbc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42114/0 (socket says 172.21.15.161:42114) 2024-03-28T23:36:22.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.279+0000 7f1151a91640 1 -- 172.21.15.161:0/2174399124 learned_addr learned my addr 172.21.15.161:0/2174399124 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:22.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.279+0000 7f1151290640 1 --2- 172.21.15.161:0/2174399124 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561224bd7db0 0x561224c24350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:22.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.279+0000 7f1150a8f640 1 --2- 172.21.15.161:0/2174399124 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561224bd8190 0x561224bd2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:22.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.279+0000 7f1151a91640 1 -- 172.21.15.161:0/2174399124 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561224bd7db0 msgr2=0x561224c24350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:22.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.279+0000 7f1151a91640 1 --2- 172.21.15.161:0/2174399124 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561224bd7db0 0x561224c24350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.279+0000 7f1151a91640 1 -- 172.21.15.161:0/2174399124 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561224bd8190 msgr2=0x561224bd2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:22.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.279+0000 7f1151a91640 1 --2- 172.21.15.161:0/2174399124 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561224bd8190 0x561224bd2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.279+0000 7f1151a91640 1 -- 172.21.15.161:0/2174399124 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56122491a6b0 con 0x561224c0ade0 2024-03-28T23:36:22.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.279+0000 7f1151a91640 1 --2- 172.21.15.161:0/2174399124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224c0ade0 0x561224bfbc60 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f1144077fa0 tx=0x7f1144009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d771962f66d6cae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:22.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.279+0000 7f112bfff640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:36:22.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.279+0000 7f112bfff640 1 -- 172.21.15.161:0/2174399124 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1144088070 con 0x561224c0ade0 2024-03-28T23:36:22.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.279+0000 7f112bfff640 1 -- 172.21.15.161:0/2174399124 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(159..159 src has 1..159) ==== 7215+0+0 (secure 0 0 0) 0x7f1144079e40 con 0x561224c0ade0 2024-03-28T23:36:22.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.279+0000 7f11561a8740 1 -- 172.21.15.161:0/2174399124 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f114403a040 con 0x561224c0ade0 2024-03-28T23:36:22.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.279+0000 7f112bfff640 1 -- 172.21.15.161:0/2174399124 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) ==== 24+0+0 (secure 0 0 0) 0x7f11440ab260 con 0x561224c0ade0 2024-03-28T23:36:22.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.279+0000 7f11561a8740 10 client.9249 fetch_fsmap learned FSMap version 154 2024-03-28T23:36:22.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.279+0000 7f11561a8740 1 -- 172.21.15.161:0/2174399124 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) -- 0x561224bfa460 con 0x561224c0ade0 2024-03-28T23:36:22.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.279+0000 7f112bfff640 1 -- 172.21.15.161:0/2174399124 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f114403a040 con 0x561224c0ade0 2024-03-28T23:36:22.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.280+0000 7f112bfff640 1 -- 172.21.15.161:0/2174399124 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 154) ==== 4139+0+0 (secure 0 0 0) 0x7f1144083d10 con 0x561224c0ade0 2024-03-28T23:36:22.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.280+0000 7f11561a8740 10 client.9249 fetch_fsmap finished waiting for FSMap version 154 2024-03-28T23:36:22.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.280+0000 7f11561a8740 10 client.9249 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:36:22.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.280+0000 7f11561a8740 20 client.9249 populate_metadata read hostname 'smithi161' 2024-03-28T23:36:22.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.280+0000 7f11561a8740 1 --2- 172.21.15.161:0/2174399124 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x561224c278c0 0x561224c27ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:22.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.280+0000 7f11561a8740 4 client.9249 mds_command: new command op to 7283 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:36:22.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.280+0000 7f11561a8740 1 -- 172.21.15.161:0/2174399124 --> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561224b49030 con 0x561224c278c0 2024-03-28T23:36:22.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.280+0000 7f1151290640 1 --2- 172.21.15.161:0/2174399124 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x561224c278c0 0x561224c27ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:22.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.280+0000 7f112bfff640 10 client.9249 ms_handle_connect on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:22.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.280+0000 7f1151290640 1 --2- 172.21.15.161:0/2174399124 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x561224c278c0 0x561224c27ca0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbfce0f6be3a3a6b server_cookie=8445083b8f2d26eb in_seq=0 out_seq=0 2024-03-28T23:36:22.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.281+0000 7f112bfff640 1 -- 172.21.15.161:0/2174399124 <== mds.0 v2:172.21.15.170:6832/2996092857 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561224b49030 con 0x561224c278c0 2024-03-28T23:36:22.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.281+0000 7f112bfff640 10 client.9249 handle_command_reply: tid=0 2024-03-28T23:36:22.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.281+0000 7f11561a8740 1 client.9249 shutdown 2024-03-28T23:36:22.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.281+0000 7f11561a8740 1 -- 172.21.15.161:0/2174399124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224c0ade0 msgr2=0x561224bfbc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:22.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.281+0000 7f11561a8740 1 --2- 172.21.15.161:0/2174399124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224c0ade0 0x561224bfbc60 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f1144077fa0 tx=0x7f1144009750 comp rx=0 tx=0).stop 2024-03-28T23:36:22.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.281+0000 7f11561a8740 1 -- 172.21.15.161:0/2174399124 shutdown_connections 2024-03-28T23:36:22.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.281+0000 7f11561a8740 1 --2- 172.21.15.161:0/2174399124 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x561224c278c0 0x561224c27ca0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.281+0000 7f11561a8740 1 --2- 172.21.15.161:0/2174399124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224c0ade0 0x561224bfbc60 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.281+0000 7f11561a8740 1 --2- 172.21.15.161:0/2174399124 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561224bd8190 0x561224bd2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.281+0000 7f112bfff640 0 client.9249 ms_handle_reset on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:22.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.281+0000 7f11561a8740 1 --2- 172.21.15.161:0/2174399124 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561224bd7db0 0x561224c24350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.281+0000 7f11561a8740 1 -- 172.21.15.161:0/2174399124 >> 172.21.15.161:0/2174399124 conn(0x561224bd6310 msgr2=0x561224bd70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:22.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.281+0000 7f11561a8740 1 -- 172.21.15.161:0/2174399124 shutdown_connections 2024-03-28T23:36:22.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.281+0000 7f11561a8740 1 -- 172.21.15.161:0/2174399124 wait complete. 2024-03-28T23:36:22.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.281+0000 7f11561a8740 20 client.9249 trim_cache size 0 max 16384 2024-03-28T23:36:22.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.299+0000 7f11561a8740 1 Processor -- start 2024-03-28T23:36:22.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.299+0000 7f11561a8740 1 -- start start 2024-03-28T23:36:22.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.300+0000 7f11561a8740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224bd7db0 0x561224bfca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:22.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.300+0000 7f11561a8740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561224bd8190 0x561224bfcfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:22.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.300+0000 7f11561a8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561224c18420 0x561224bf2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:22.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.300+0000 7f11561a8740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f114c1fb120 con 0x561224bd7db0 2024-03-28T23:36:22.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.300+0000 7f11561a8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f114c1f8af0 con 0x561224c18420 2024-03-28T23:36:22.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.300+0000 7f11561a8740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f114c1f90b0 con 0x561224bd8190 2024-03-28T23:36:22.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.300+0000 7f1151290640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224bd7db0 0x561224bfca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:22.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.300+0000 7f1151290640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224bd7db0 0x561224bfca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42122/0 (socket says 172.21.15.161:42122) 2024-03-28T23:36:22.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.300+0000 7f1151290640 1 -- 172.21.15.161:0/867158283 learned_addr learned my addr 172.21.15.161:0/867158283 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:22.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.300+0000 7f1151a91640 1 --2- 172.21.15.161:0/867158283 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561224c18420 0x561224bf2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:22.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.300+0000 7f1150a8f640 1 --2- 172.21.15.161:0/867158283 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561224bd8190 0x561224bfcfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:22.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.300+0000 7f1151290640 1 -- 172.21.15.161:0/867158283 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561224bd8190 msgr2=0x561224bfcfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:22.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.300+0000 7f1151290640 1 --2- 172.21.15.161:0/867158283 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561224bd8190 0x561224bfcfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.300+0000 7f1151290640 1 -- 172.21.15.161:0/867158283 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561224c18420 msgr2=0x561224bf2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:22.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.300+0000 7f1151290640 1 --2- 172.21.15.161:0/867158283 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561224c18420 0x561224bf2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.300+0000 7f1151290640 1 -- 172.21.15.161:0/867158283 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561224c255f0 con 0x561224bd7db0 2024-03-28T23:36:22.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.300+0000 7f1151290640 1 --2- 172.21.15.161:0/867158283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224bd7db0 0x561224bfca80 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f113c016040 tx=0x7f113c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9830e475830c921e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:22.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.300+0000 7f11367fc640 1 -- 172.21.15.161:0/867158283 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f113c02f070 con 0x561224bd7db0 2024-03-28T23:36:22.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.300+0000 7f11367fc640 1 -- 172.21.15.161:0/867158283 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f113c01b040 con 0x561224bd7db0 2024-03-28T23:36:22.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.300+0000 7f11367fc640 1 -- 172.21.15.161:0/867158283 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f113c02baa0 con 0x561224bd7db0 2024-03-28T23:36:22.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.301+0000 7f11561a8740 1 -- 172.21.15.161:0/867158283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224bd7db0 msgr2=0x561224bfca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:22.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.301+0000 7f11561a8740 1 --2- 172.21.15.161:0/867158283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224bd7db0 0x561224bfca80 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f113c016040 tx=0x7f113c018000 comp rx=0 tx=0).stop 2024-03-28T23:36:22.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.301+0000 7f11561a8740 1 -- 172.21.15.161:0/867158283 shutdown_connections 2024-03-28T23:36:22.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.301+0000 7f11561a8740 1 --2- 172.21.15.161:0/867158283 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561224c18420 0x561224bf2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.301+0000 7f11561a8740 1 --2- 172.21.15.161:0/867158283 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561224bd8190 0x561224bfcfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.301+0000 7f11561a8740 1 --2- 172.21.15.161:0/867158283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224bd7db0 0x561224bfca80 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.301+0000 7f11561a8740 1 -- 172.21.15.161:0/867158283 >> 172.21.15.161:0/867158283 conn(0x561224bd6310 msgr2=0x561224bcd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:22.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.301+0000 7f11561a8740 1 -- 172.21.15.161:0/867158283 shutdown_connections 2024-03-28T23:36:22.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.301+0000 7f11561a8740 1 -- 172.21.15.161:0/867158283 wait complete. 2024-03-28T23:36:22.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.302+0000 7f11561a8740 1 -- start start 2024-03-28T23:36:22.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.302+0000 7f11561a8740 1 Processor -- start 2024-03-28T23:36:22.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.302+0000 7f11561a8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561224bd7db0 0x561224c20f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:22.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.302+0000 7f11561a8740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224bd8190 0x561224c0fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:22.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.302+0000 7f11561a8740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561224c18420 0x561224c8b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:22.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.302+0000 7f11561a8740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561224bf9ad0 con 0x561224bd8190 2024-03-28T23:36:22.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.302+0000 7f11561a8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x561224bf6780 con 0x561224bd7db0 2024-03-28T23:36:22.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.302+0000 7f11561a8740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x561224af7f70 con 0x561224c18420 2024-03-28T23:36:22.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.303+0000 7f11561a8740 10 client.0 _ll_register_callbacks cb 0x7f1148007e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:36:22.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.303+0000 7f1150a8f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224bd8190 0x561224c0fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:22.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.303+0000 7f1151290640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561224bd7db0 0x561224c20f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:22.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.303+0000 7f1150a8f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224bd8190 0x561224c0fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42132/0 (socket says 172.21.15.161:42132) 2024-03-28T23:36:22.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.303+0000 7f1151290640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561224bd7db0 0x561224c20f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:44616/0 (socket says 172.21.15.161:44616) 2024-03-28T23:36:22.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.303+0000 7f1150a8f640 1 -- 172.21.15.161:0/615629599 learned_addr learned my addr 172.21.15.161:0/615629599 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:22.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.303+0000 7f1151a91640 1 --2- 172.21.15.161:0/615629599 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561224c18420 0x561224c8b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:22.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.303+0000 7f1150a8f640 1 -- 172.21.15.161:0/615629599 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561224c18420 msgr2=0x561224c8b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:22.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.303+0000 7f1150a8f640 1 --2- 172.21.15.161:0/615629599 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561224c18420 0x561224c8b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.303+0000 7f1150a8f640 1 -- 172.21.15.161:0/615629599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561224bd7db0 msgr2=0x561224c20f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:22.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.303+0000 7f1150a8f640 1 --2- 172.21.15.161:0/615629599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561224bd7db0 0x561224c20f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.303+0000 7f1150a8f640 1 -- 172.21.15.161:0/615629599 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561224c0a710 con 0x561224bd8190 2024-03-28T23:36:22.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.303+0000 7f1151290640 1 --2- 172.21.15.161:0/615629599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561224bd7db0 0x561224c20f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:22.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.303+0000 7f1150a8f640 1 --2- 172.21.15.161:0/615629599 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224bd8190 0x561224c0fd80 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f1148016650 tx=0x7f1148016b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8574141efa173f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:22.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.303+0000 7f112bfff640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:36:22.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.303+0000 7f112bfff640 1 -- 172.21.15.161:0/615629599 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1148007320 con 0x561224bd8190 2024-03-28T23:36:22.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.303+0000 7f112bfff640 1 -- 172.21.15.161:0/615629599 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(159..159 src has 1..159) ==== 7215+0+0 (secure 0 0 0) 0x7f1148010040 con 0x561224bd8190 2024-03-28T23:36:22.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.303+0000 7f11561a8740 1 -- 172.21.15.161:0/615629599 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f113c01b040 con 0x561224bd8190 2024-03-28T23:36:22.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.303+0000 7f112bfff640 1 -- 172.21.15.161:0/615629599 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) ==== 24+0+0 (secure 0 0 0) 0x7f1148028050 con 0x561224bd8190 2024-03-28T23:36:22.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.303+0000 7f11561a8740 10 client.9255 fetch_fsmap learned FSMap version 154 2024-03-28T23:36:22.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.303+0000 7f11561a8740 1 -- 172.21.15.161:0/615629599 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) -- 0x561224c0a430 con 0x561224bd8190 2024-03-28T23:36:22.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.304+0000 7f112bfff640 1 -- 172.21.15.161:0/615629599 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f113c01b040 con 0x561224bd8190 2024-03-28T23:36:22.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.304+0000 7f112bfff640 1 -- 172.21.15.161:0/615629599 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 154) ==== 4139+0+0 (secure 0 0 0) 0x7f114802b020 con 0x561224bd8190 2024-03-28T23:36:22.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.304+0000 7f11561a8740 10 client.9255 fetch_fsmap finished waiting for FSMap version 154 2024-03-28T23:36:22.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.304+0000 7f11561a8740 10 client.9255 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:36:22.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.304+0000 7f11561a8740 20 client.9255 populate_metadata read hostname 'smithi161' 2024-03-28T23:36:22.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.304+0000 7f11561a8740 1 --2- 172.21.15.161:0/615629599 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x561224c0b3d0 0x561224c0b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:22.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.304+0000 7f11561a8740 4 client.9255 mds_command: new command op to 7283 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:36:22.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.304+0000 7f11561a8740 1 -- 172.21.15.161:0/615629599 --> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] -- command(tid 0: {"prefix": "status"}) -- 0x561224b43380 con 0x561224c0b3d0 2024-03-28T23:36:22.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.304+0000 7f1151290640 1 --2- 172.21.15.161:0/615629599 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x561224c0b3d0 0x561224c0b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:22.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.304+0000 7f112bfff640 10 client.9255 ms_handle_connect on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:22.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.304+0000 7f1151290640 1 --2- 172.21.15.161:0/615629599 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x561224c0b3d0 0x561224c0b7b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e971250de0aea1a3 server_cookie=6908e4e269b3c4f5 in_seq=0 out_seq=0 2024-03-28T23:36:22.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.304+0000 7f112bfff640 1 -- 172.21.15.161:0/615629599 <== mds.0 v2:172.21.15.170:6832/2996092857 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x561224b43380 con 0x561224c0b3d0 2024-03-28T23:36:22.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.304+0000 7f112bfff640 10 client.9255 handle_command_reply: tid=0 2024-03-28T23:36:22.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.304+0000 7f11561a8740 1 client.9255 shutdown 2024-03-28T23:36:22.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.304+0000 7f11561a8740 1 -- 172.21.15.161:0/615629599 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224bd8190 msgr2=0x561224c0fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:22.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.304+0000 7f11561a8740 1 --2- 172.21.15.161:0/615629599 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224bd8190 0x561224c0fd80 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f1148016650 tx=0x7f1148016b10 comp rx=0 tx=0).stop 2024-03-28T23:36:22.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.305+0000 7f11561a8740 1 -- 172.21.15.161:0/615629599 shutdown_connections 2024-03-28T23:36:22.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.305+0000 7f11561a8740 1 --2- 172.21.15.161:0/615629599 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x561224c0b3d0 0x561224c0b7b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.305+0000 7f11561a8740 1 --2- 172.21.15.161:0/615629599 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561224bd8190 0x561224c0fd80 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.305+0000 7f11561a8740 1 --2- 172.21.15.161:0/615629599 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561224c18420 0x561224c8b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.305+0000 7f11561a8740 1 --2- 172.21.15.161:0/615629599 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561224bd7db0 0x561224c20f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.305+0000 7f112bfff640 0 client.9255 ms_handle_reset on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:22.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.305+0000 7f11561a8740 1 -- 172.21.15.161:0/615629599 >> 172.21.15.161:0/615629599 conn(0x561224bd6310 msgr2=0x561224bfca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:22.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.305+0000 7f11561a8740 1 -- 172.21.15.161:0/615629599 shutdown_connections 2024-03-28T23:36:22.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.305+0000 7f11561a8740 1 -- 172.21.15.161:0/615629599 wait complete. 2024-03-28T23:36:22.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.305+0000 7f11561a8740 20 client.9255 trim_cache size 0 max 16384 2024-03-28T23:36:22.312 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:36:22.312 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:36:22.312 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:36:22.312 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 7283, 2024-03-28T23:36:22.312 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:36:22.313 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:36:22.313 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:36:22.313 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 4.8721883080000001, 2024-03-28T23:36:22.313 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 154, 2024-03-28T23:36:22.313 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 159, 2024-03-28T23:36:22.313 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 159, 2024-03-28T23:36:22.313 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 7.8620121059999999 2024-03-28T23:36:22.313 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:36:22.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.305+0000 7f11534f4640 1 -- 172.21.15.161:0/3697357089 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f113006b500 msgr2=0x7f113006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:22.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.305+0000 7f11534f4640 1 --2- 172.21.15.161:0/3697357089 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f113006b500 0x7f113006d9c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f113c009560 tx=0x7f113c009190 comp rx=0 tx=0).stop 2024-03-28T23:36:22.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.305+0000 7f11534f4640 1 -- 172.21.15.161:0/3697357089 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f114c152890 msgr2=0x7f114c144ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:22.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.305+0000 7f11534f4640 1 --2- 172.21.15.161:0/3697357089 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f114c152890 0x7f114c144ed0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f1144009e80 tx=0x7f1144004a40 comp rx=0 tx=0).stop 2024-03-28T23:36:22.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.305+0000 7f11534f4640 1 -- 172.21.15.161:0/3697357089 shutdown_connections 2024-03-28T23:36:22.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.305+0000 7f11534f4640 1 --2- 172.21.15.161:0/3697357089 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f113006b500 0x7f113006d9c0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.305+0000 7f11534f4640 1 --2- 172.21.15.161:0/3697357089 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f114c152890 0x7f114c144ed0 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.305+0000 7f11534f4640 1 --2- 172.21.15.161:0/3697357089 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f114c0c8ed0 0x7f114c1ee340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.305+0000 7f11534f4640 1 --2- 172.21.15.161:0/3697357089 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f114c0c5b20 0x7f114c1ea4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:22.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.305+0000 7f11534f4640 1 -- 172.21.15.161:0/3697357089 >> 172.21.15.161:0/3697357089 conn(0x7f114c0ac620 msgr2=0x7f114c1505f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:22.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.305+0000 7f11534f4640 1 -- 172.21.15.161:0/3697357089 shutdown_connections 2024-03-28T23:36:22.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:22.305+0000 7f11534f4640 1 -- 172.21.15.161:0/3697357089 wait complete. 2024-03-28T23:36:22.317 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 11, "epoch": 154, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 11, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:35:58.406403+0000", "damaged": [], "data_pools": [ 24 ], "enabled": true, "epoch": 154, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7283": { "addr": "172.21.15.170:6834/2996092857", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 2996092857, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 2996092857, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7283, "incarnation": 151, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 159, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 23, "modified": "2024-03-28T23:36:20.440473+0000", "qdb_cluster": [ 7283 ], "qdb_leader": 7283, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7283 } } }, { "id": 12, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:36:07.448976+0000", "damaged": [], "data_pools": [ 26 ], "enabled": true, "epoch": 149, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs2", "in": [ 0 ], "info": { "gid_7061": { "addr": "172.21.15.161:6836/4026332703", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 4026332703, "type": "v2" }, { "addr": "172.21.15.161:6836", "nonce": 4026332703, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7061, "incarnation": 145, "join_fscid": 12, "name": "c", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 157, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 25, "modified": "2024-03-28T23:36:16.420228+0000", "qdb_cluster": [ 7061 ], "qdb_leader": 7061, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7061 } } } ], "standbys": [ { "addr": "172.21.15.170:6835/557522063", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 557522063, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 557522063, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 137, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7070, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6837/789976100", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6835", "nonce": 789976100, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 789976100, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 152, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9216, "incarnation": 0, "join_fscid": 12, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:36:22.318 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': 12, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 12, 'name': 'c'}, {'join_fscid': -1, 'name': 'd', 'state': 'up:active'}] 2024-03-28T23:36:22.318 INFO:tasks.cephfs.test_failover:current = [{'gid': 9216, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6837/789976100', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 789976100}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 789976100}]}, 'join_fscid': 12, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 152}, {'gid': 7283, 'name': 'b', 'rank': 0, 'incarnation': 151, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.170:6834/2996092857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 2996092857}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 2996092857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 7061, 'name': 'c', 'rank': 0, 'incarnation': 145, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.161:6836/4026332703', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 4026332703}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 4026332703}]}, 'join_fscid': 12, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 7070, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/557522063', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 557522063}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 557522063}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 137}] 2024-03-28T23:36:22.319 DEBUG:tasks.cephfs.test_failover:'up:active' != 'up:standby' - up:active + up:standby 2024-03-28T23:36:22.319 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-28T23:36:25.425 INFO:tasks.ceph.mds.b.smithi170.stderr:2024-03-28T23:36:25.425+0000 7f5f303af640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:36:26.423 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:36:26.426 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:36:27.320 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:36:27.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.422+0000 7fd7f2855640 1 Processor -- start 2024-03-28T23:36:27.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.422+0000 7fd7f2855640 1 -- start start 2024-03-28T23:36:27.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.422+0000 7fd7f2855640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd7ec152890 0x7fd7ec0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:27.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.423+0000 7fd7f2855640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7ec158a30 con 0x7fd7ec0c5b20 2024-03-28T23:36:27.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.423+0000 7fd7f2855640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7ec158d10 con 0x7fd7ec0c8ed0 2024-03-28T23:36:27.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.423+0000 7fd7f2855640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd7ec0c8480 con 0x7fd7ec152890 2024-03-28T23:36:27.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.423+0000 7fd7f0df2640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd7ec0c5b20 0x7fd7ec1553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:39192/0 (socket says 172.21.15.161:39192) 2024-03-28T23:36:27.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.423+0000 7fd7f0df2640 1 -- 172.21.15.161:0/3780489981 learned_addr learned my addr 172.21.15.161:0/3780489981 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:27.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.423+0000 7fd7ebfff640 1 --2- 172.21.15.161:0/3780489981 >> v2:172.21.15.170:3300/0 conn(0x7fd7ec152890 0x7fd7ec0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:27.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.423+0000 7fd7eaffd640 1 -- 172.21.15.161:0/3780489981 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 183537620 0 0) 0x7fd7ec158d10 con 0x7fd7ec0c8ed0 2024-03-28T23:36:27.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.423+0000 7fd7eaffd640 1 -- 172.21.15.161:0/3780489981 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7d4003520 con 0x7fd7ec0c8ed0 2024-03-28T23:36:27.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.423+0000 7fd7eaffd640 1 -- 172.21.15.161:0/3780489981 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4292297477 0 0) 0x7fd7ec158a30 con 0x7fd7ec0c5b20 2024-03-28T23:36:27.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.423+0000 7fd7eaffd640 1 -- 172.21.15.161:0/3780489981 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7ec158d10 con 0x7fd7ec0c5b20 2024-03-28T23:36:27.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.423+0000 7fd7ebfff640 1 -- 172.21.15.161:0/3780489981 >> v1:172.21.15.170:6789/0 conn(0x7fd7ec0c8ed0 legacy=0x7fd7ec0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:27.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.423+0000 7fd7ebfff640 1 -- 172.21.15.161:0/3780489981 >> v1:172.21.15.161:6789/0 conn(0x7fd7ec0c5b20 legacy=0x7fd7ec1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:27.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.423+0000 7fd7ebfff640 1 -- 172.21.15.161:0/3780489981 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ec158ff0 con 0x7fd7ec152890 2024-03-28T23:36:27.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.424+0000 7fd7ebfff640 1 --2- 172.21.15.161:0/3780489981 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7ec152890 0x7fd7ec0c8990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc0097a0 tx=0x7fd7dc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60d9c06b6c6a0b84 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:27.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.424+0000 7fd7eaffd640 1 -- 172.21.15.161:0/3780489981 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc0315a0 con 0x7fd7ec152890 2024-03-28T23:36:27.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.424+0000 7fd7eaffd640 1 -- 172.21.15.161:0/3780489981 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7dc037070 con 0x7fd7ec152890 2024-03-28T23:36:27.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.424+0000 7fd7eaffd640 1 -- 172.21.15.161:0/3780489981 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc031960 con 0x7fd7ec152890 2024-03-28T23:36:27.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.425+0000 7fd7f2855640 1 -- 172.21.15.161:0/3780489981 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7ec152890 msgr2=0x7fd7ec0c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:27.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.425+0000 7fd7f2855640 1 --2- 172.21.15.161:0/3780489981 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7ec152890 0x7fd7ec0c8990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc0097a0 tx=0x7fd7dc02b840 comp rx=0 tx=0).stop 2024-03-28T23:36:27.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.425+0000 7fd7f2855640 1 -- 172.21.15.161:0/3780489981 shutdown_connections 2024-03-28T23:36:27.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.425+0000 7fd7f2855640 1 --2- 172.21.15.161:0/3780489981 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7ec152890 0x7fd7ec0c8990 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:27.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.425+0000 7fd7f2855640 1 -- 172.21.15.161:0/3780489981 >> 172.21.15.161:0/3780489981 conn(0x7fd7ec0ac620 msgr2=0x7fd7ec0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:27.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.425+0000 7fd7f2855640 1 -- 172.21.15.161:0/3780489981 shutdown_connections 2024-03-28T23:36:27.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.425+0000 7fd7f2855640 1 -- 172.21.15.161:0/3780489981 wait complete. 2024-03-28T23:36:27.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.426+0000 7fd7f2855640 1 Processor -- start 2024-03-28T23:36:27.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.426+0000 7fd7f2855640 1 -- start start 2024-03-28T23:36:27.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.427+0000 7fd7f2855640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7ec0c5b20 0x7fd7ec149520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:27.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.427+0000 7fd7f2855640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7ec0c8ed0 0x7fd7ec14f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:27.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.427+0000 7fd7f2855640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7ec152890 0x7fd7ec14f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:27.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.427+0000 7fd7f2855640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ec152c70 con 0x7fd7ec0c5b20 2024-03-28T23:36:27.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.427+0000 7fd7f2855640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ec159640 con 0x7fd7ec152890 2024-03-28T23:36:27.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.427+0000 7fd7f2855640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd7ec152df0 con 0x7fd7ec0c8ed0 2024-03-28T23:36:27.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.427+0000 7fd7ebfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7ec0c5b20 0x7fd7ec149520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:27.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.427+0000 7fd7ebfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7ec0c5b20 0x7fd7ec149520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42136/0 (socket says 172.21.15.161:42136) 2024-03-28T23:36:27.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.427+0000 7fd7ebfff640 1 -- 172.21.15.161:0/4041100534 learned_addr learned my addr 172.21.15.161:0/4041100534 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:27.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.427+0000 7fd7eb7fe640 1 --2- 172.21.15.161:0/4041100534 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7ec0c8ed0 0x7fd7ec14f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:27.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.427+0000 7fd7f0df2640 1 --2- 172.21.15.161:0/4041100534 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7ec152890 0x7fd7ec14f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:27.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.427+0000 7fd7ebfff640 1 -- 172.21.15.161:0/4041100534 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7ec0c8ed0 msgr2=0x7fd7ec14f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:27.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.427+0000 7fd7ebfff640 1 --2- 172.21.15.161:0/4041100534 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7ec0c8ed0 0x7fd7ec14f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:27.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.427+0000 7fd7ebfff640 1 -- 172.21.15.161:0/4041100534 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7ec152890 msgr2=0x7fd7ec14f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:27.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.427+0000 7fd7ebfff640 1 --2- 172.21.15.161:0/4041100534 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7ec152890 0x7fd7ec14f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:27.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.427+0000 7fd7ebfff640 1 -- 172.21.15.161:0/4041100534 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ec1edf20 con 0x7fd7ec0c5b20 2024-03-28T23:36:27.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.427+0000 7fd7ebfff640 1 --2- 172.21.15.161:0/4041100534 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7ec0c5b20 0x7fd7ec149520 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc000c00 tx=0x7fd7dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:27.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.428+0000 7fd7e8ff9640 1 -- 172.21.15.161:0/4041100534 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc032630 con 0x7fd7ec0c5b20 2024-03-28T23:36:27.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.428+0000 7fd7e8ff9640 1 -- 172.21.15.161:0/4041100534 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7dc0327d0 con 0x7fd7ec0c5b20 2024-03-28T23:36:27.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.428+0000 7fd7f2855640 1 -- 172.21.15.161:0/4041100534 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7ec1ee200 con 0x7fd7ec0c5b20 2024-03-28T23:36:27.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.428+0000 7fd7e8ff9640 1 -- 172.21.15.161:0/4041100534 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc032bc0 con 0x7fd7ec0c5b20 2024-03-28T23:36:27.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.428+0000 7fd7f2855640 1 -- 172.21.15.161:0/4041100534 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7ec1ee7c0 con 0x7fd7ec0c5b20 2024-03-28T23:36:27.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.429+0000 7fd7f2855640 1 -- 172.21.15.161:0/4041100534 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7ec1eeaf0 con 0x7fd7ec0c5b20 2024-03-28T23:36:27.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.430+0000 7fd7e8ff9640 1 -- 172.21.15.161:0/4041100534 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd7dc04d3f0 con 0x7fd7ec0c5b20 2024-03-28T23:36:27.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.430+0000 7fd7e8ff9640 1 --2- 172.21.15.161:0/4041100534 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd7d806b500 0x7fd7d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:27.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.430+0000 7fd7eb7fe640 1 --2- 172.21.15.161:0/4041100534 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd7d806b500 0x7fd7d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:27.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.430+0000 7fd7e8ff9640 1 -- 172.21.15.161:0/4041100534 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 7215+0+0 (secure 0 0 0) 0x7fd7dc02f030 con 0x7fd7ec0c5b20 2024-03-28T23:36:27.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.431+0000 7fd7eb7fe640 1 --2- 172.21.15.161:0/4041100534 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd7d806b500 0x7fd7d806d9c0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0007fd0 tx=0x7fd7e0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:27.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.435+0000 7fd7e8ff9640 1 -- 172.21.15.161:0/4041100534 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd7dc040070 con 0x7fd7ec0c5b20 2024-03-28T23:36:27.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.645+0000 7fd7f2855640 1 -- 172.21.15.161:0/4041100534 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7ec04f4b0 con 0x7fd7ec0c5b20 2024-03-28T23:36:27.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.646+0000 7fd7e8ff9640 1 -- 172.21.15.161:0/4041100534 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+6520 (secure 0 0 0) 0x7fd7dc07a360 con 0x7fd7ec0c5b20 2024-03-28T23:36:27.646 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:27.646 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":154,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/557522063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":557522063},{"type":"v1","addr":"172.21.15.170:6835","nonce":557522063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":9216,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/789976100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":789976100},{"type":"v1","addr":"172.21.15.161:6837","nonce":789976100}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:58.406403+0000","modified":"2024-03-28T23:36:20.440473+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":159,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7283":{"gid":7283,"name":"b","rank":0,"incarnation":151,"state":"up:active","state_seq":4,"addr":"172.21.15.170:6834/2996092857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":2996092857},{"type":"v1","addr":"172.21.15.170:6834","nonce":2996092857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7283,"qdb_cluster":[7283]},"id":11},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:36:07.448976+0000","modified":"2024-03-28T23:36:16.420228+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7061":{"gid":7061,"name":"c","rank":0,"incarnation":145,"state":"up:active","state_seq":9,"addr":"172.21.15.161:6836/4026332703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4026332703},{"type":"v1","addr":"172.21.15.161:6836","nonce":4026332703}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7061,"qdb_cluster":[7061]},"id":12}]} 2024-03-28T23:36:27.646 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 154 2024-03-28T23:36:27.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.648+0000 7fd7f2855640 1 -- 172.21.15.161:0/4041100534 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd7d806b500 msgr2=0x7fd7d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:27.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.648+0000 7fd7f2855640 1 --2- 172.21.15.161:0/4041100534 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd7d806b500 0x7fd7d806d9c0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0007fd0 tx=0x7fd7e0009040 comp rx=0 tx=0).stop 2024-03-28T23:36:27.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.648+0000 7fd7f2855640 1 -- 172.21.15.161:0/4041100534 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7ec0c5b20 msgr2=0x7fd7ec149520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:27.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.648+0000 7fd7f2855640 1 --2- 172.21.15.161:0/4041100534 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7ec0c5b20 0x7fd7ec149520 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc000c00 tx=0x7fd7dc0046c0 comp rx=0 tx=0).stop 2024-03-28T23:36:27.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.648+0000 7fd7f2855640 1 -- 172.21.15.161:0/4041100534 shutdown_connections 2024-03-28T23:36:27.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.648+0000 7fd7f2855640 1 --2- 172.21.15.161:0/4041100534 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd7d806b500 0x7fd7d806d9c0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:27.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.648+0000 7fd7f2855640 1 --2- 172.21.15.161:0/4041100534 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7ec152890 0x7fd7ec14f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:27.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.648+0000 7fd7f2855640 1 --2- 172.21.15.161:0/4041100534 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7ec0c8ed0 0x7fd7ec14f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:27.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.648+0000 7fd7f2855640 1 --2- 172.21.15.161:0/4041100534 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7ec0c5b20 0x7fd7ec149520 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:27.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.648+0000 7fd7f2855640 1 -- 172.21.15.161:0/4041100534 >> 172.21.15.161:0/4041100534 conn(0x7fd7ec0ac620 msgr2=0x7fd7ec14c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:27.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.648+0000 7fd7f2855640 1 -- 172.21.15.161:0/4041100534 shutdown_connections 2024-03-28T23:36:27.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:27.648+0000 7fd7f2855640 1 -- 172.21.15.161:0/4041100534 wait complete. 2024-03-28T23:36:28.062 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 154, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:35:58.406403+0000', 'modified': '2024-03-28T23:36:20.440473+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 159, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7283}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7283': {'gid': 7283, 'name': 'b', 'rank': 0, 'incarnation': 151, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.170:6834/2996092857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 2996092857}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 2996092857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7283, 'qdb_cluster': [7283]} 2024-03-28T23:36:28.062 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:36:28.062 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T23:36:28.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.165+0000 7f89ebdd0640 1 Processor -- start 2024-03-28T23:36:28.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.165+0000 7f89ebdd0640 1 -- start start 2024-03-28T23:36:28.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.166+0000 7f89ebdd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89e4152890 0x7f89e40c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:28.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.166+0000 7f89ebdd0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f89e40c8ed0 0x7f89e40c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:28.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.166+0000 7f89ebdd0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89e4158a30 con 0x7f89e40c5950 2024-03-28T23:36:28.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.166+0000 7f89ebdd0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f89e40c5db0 con 0x7f89e40c8ed0 2024-03-28T23:36:28.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.166+0000 7f89ebdd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f89e40c8480 con 0x7f89e4152890 2024-03-28T23:36:28.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.166+0000 7f89e936b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f89e40c8ed0 0x7f89e40c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:28.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.166+0000 7f89ea36d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f89e40c5950 0x7f89e41553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:39202/0 (socket says 172.21.15.161:39202) 2024-03-28T23:36:28.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.166+0000 7f89ea36d640 1 -- 172.21.15.161:0/1874125196 learned_addr learned my addr 172.21.15.161:0/1874125196 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:28.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.166+0000 7f89e9b6c640 1 --2- 172.21.15.161:0/1874125196 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89e4152890 0x7f89e40c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:28.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.166+0000 7f89e8b6a640 1 -- 172.21.15.161:0/1874125196 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2700458852 0 0) 0x7f89e4158a30 con 0x7f89e40c5950 2024-03-28T23:36:28.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.167+0000 7f89e8b6a640 1 -- 172.21.15.161:0/1874125196 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89cc003520 con 0x7f89e40c5950 2024-03-28T23:36:28.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.167+0000 7f89e936b640 1 -- 172.21.15.161:0/1874125196 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89e4152890 msgr2=0x7f89e40c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:28.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.167+0000 7f89e936b640 1 --2- 172.21.15.161:0/1874125196 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89e4152890 0x7f89e40c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.167+0000 7f89e936b640 1 -- 172.21.15.161:0/1874125196 >> v1:172.21.15.161:6789/0 conn(0x7f89e40c5950 legacy=0x7f89e41553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:28.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.167+0000 7f89e936b640 1 -- 172.21.15.161:0/1874125196 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e4158d10 con 0x7f89e40c8ed0 2024-03-28T23:36:28.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.167+0000 7f89e936b640 1 --2- 172.21.15.161:0/1874125196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89e40c8ed0 0x7f89e40c5410 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009790 tx=0x7f89d802b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cf0ce3bd72619bf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:28.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.167+0000 7f89e8b6a640 1 -- 172.21.15.161:0/1874125196 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d8039070 con 0x7f89e40c8ed0 2024-03-28T23:36:28.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.168+0000 7f89e8b6a640 1 -- 172.21.15.161:0/1874125196 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89d80316f0 con 0x7f89e40c8ed0 2024-03-28T23:36:28.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.168+0000 7f89e8b6a640 1 -- 172.21.15.161:0/1874125196 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d8031ac0 con 0x7f89e40c8ed0 2024-03-28T23:36:28.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.168+0000 7f89ebdd0640 1 -- 172.21.15.161:0/1874125196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89e40c8ed0 msgr2=0x7f89e40c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:28.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.168+0000 7f89ebdd0640 1 --2- 172.21.15.161:0/1874125196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89e40c8ed0 0x7f89e40c5410 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009790 tx=0x7f89d802b790 comp rx=0 tx=0).stop 2024-03-28T23:36:28.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.168+0000 7f89ebdd0640 1 -- 172.21.15.161:0/1874125196 shutdown_connections 2024-03-28T23:36:28.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.168+0000 7f89ebdd0640 1 --2- 172.21.15.161:0/1874125196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89e40c8ed0 0x7f89e40c5410 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.168+0000 7f89ebdd0640 1 --2- 172.21.15.161:0/1874125196 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89e4152890 0x7f89e40c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.168+0000 7f89ebdd0640 1 -- 172.21.15.161:0/1874125196 >> 172.21.15.161:0/1874125196 conn(0x7f89e40ac620 msgr2=0x7f89e40aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:28.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.168+0000 7f89ebdd0640 1 -- 172.21.15.161:0/1874125196 shutdown_connections 2024-03-28T23:36:28.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.168+0000 7f89ebdd0640 1 -- 172.21.15.161:0/1874125196 wait complete. 2024-03-28T23:36:28.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.169+0000 7f89ebdd0640 1 Processor -- start 2024-03-28T23:36:28.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.169+0000 7f89ebdd0640 1 -- start start 2024-03-28T23:36:28.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.170+0000 7f89ebdd0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89e40c5950 0x7f89e40c31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:28.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.170+0000 7f89ebdd0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89e40c8ed0 0x7f89e41ee840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:28.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.170+0000 7f89ebdd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89e4152890 0x7f89e4149b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:28.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.170+0000 7f89ebdd0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f89e41592d0 con 0x7f89e40c8ed0 2024-03-28T23:36:28.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.170+0000 7f89ebdd0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f89e4159640 con 0x7f89e40c5950 2024-03-28T23:36:28.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.170+0000 7f89ebdd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f89e4152c70 con 0x7f89e4152890 2024-03-28T23:36:28.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.170+0000 7f89e936b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89e40c8ed0 0x7f89e41ee840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:28.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.170+0000 7f89e936b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89e40c8ed0 0x7f89e41ee840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42160/0 (socket says 172.21.15.161:42160) 2024-03-28T23:36:28.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.170+0000 7f89e936b640 1 -- 172.21.15.161:0/479597455 learned_addr learned my addr 172.21.15.161:0/479597455 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:28.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.170+0000 7f89ea36d640 1 --2- 172.21.15.161:0/479597455 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89e4152890 0x7f89e4149b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:28.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.170+0000 7f89e9b6c640 1 --2- 172.21.15.161:0/479597455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89e40c5950 0x7f89e40c31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:28.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.170+0000 7f89e936b640 1 -- 172.21.15.161:0/479597455 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89e4152890 msgr2=0x7f89e4149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:28.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.170+0000 7f89e936b640 1 --2- 172.21.15.161:0/479597455 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89e4152890 0x7f89e4149b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.170+0000 7f89e936b640 1 -- 172.21.15.161:0/479597455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89e40c5950 msgr2=0x7f89e40c31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:28.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.170+0000 7f89e936b640 1 --2- 172.21.15.161:0/479597455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89e40c5950 0x7f89e40c31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.170+0000 7f89e936b640 1 -- 172.21.15.161:0/479597455 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e41eed80 con 0x7f89e40c8ed0 2024-03-28T23:36:28.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.171+0000 7f89e936b640 1 --2- 172.21.15.161:0/479597455 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89e40c8ed0 0x7f89e41ee840 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f89d8000c00 tx=0x7f89d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:28.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.171+0000 7f89e27fc640 1 -- 172.21.15.161:0/479597455 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d8043070 con 0x7f89e40c8ed0 2024-03-28T23:36:28.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.171+0000 7f89e27fc640 1 -- 172.21.15.161:0/479597455 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89d8034480 con 0x7f89e40c8ed0 2024-03-28T23:36:28.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.171+0000 7f89ebdd0640 1 -- 172.21.15.161:0/479597455 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89e40c1bd0 con 0x7f89e40c8ed0 2024-03-28T23:36:28.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.171+0000 7f89e27fc640 1 -- 172.21.15.161:0/479597455 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d8039040 con 0x7f89e40c8ed0 2024-03-28T23:36:28.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.171+0000 7f89ebdd0640 1 -- 172.21.15.161:0/479597455 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89e414d1d0 con 0x7f89e40c8ed0 2024-03-28T23:36:28.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.172+0000 7f89e27fc640 1 -- 172.21.15.161:0/479597455 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f89d8046070 con 0x7f89e40c8ed0 2024-03-28T23:36:28.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.173+0000 7f89e27fc640 1 --2- 172.21.15.161:0/479597455 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f89d006b4d0 0x7f89d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:28.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.173+0000 7f89e27fc640 1 -- 172.21.15.161:0/479597455 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 7215+0+0 (secure 0 0 0) 0x7f89d80ac200 con 0x7f89e40c8ed0 2024-03-28T23:36:28.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.173+0000 7f89e9b6c640 1 --2- 172.21.15.161:0/479597455 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f89d006b4d0 0x7f89d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:28.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.173+0000 7f89e9b6c640 1 --2- 172.21.15.161:0/479597455 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f89d006b4d0 0x7f89d006d990 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f89d4009b40 tx=0x7f89d4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:28.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.174+0000 7f89ee9b5740 1 Processor -- start 2024-03-28T23:36:28.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.174+0000 7f89ee9b5740 1 -- start start 2024-03-28T23:36:28.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.174+0000 7f89ee9b5740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563dc46dedb0 0x563dc46dc890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:28.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.175+0000 7f89ee9b5740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc46df190 0x563dc46dcdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:28.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.175+0000 7f89ee9b5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563dc46dd310 0x563dc46d94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:28.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.175+0000 7f89ee9b5740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x563dc4642af0 con 0x563dc46df190 2024-03-28T23:36:28.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.175+0000 7f89ee9b5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x563dc45eb9d0 con 0x563dc46dd310 2024-03-28T23:36:28.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.175+0000 7f89ee9b5740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x563dc463eb90 con 0x563dc46dedb0 2024-03-28T23:36:28.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.175+0000 7f89e936b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc46df190 0x563dc46dcdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:28.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.175+0000 7f89e936b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc46df190 0x563dc46dcdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42172/0 (socket says 172.21.15.161:42172) 2024-03-28T23:36:28.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.175+0000 7f89e9b6c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563dc46dedb0 0x563dc46dc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:28.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.175+0000 7f89e936b640 1 -- 172.21.15.161:0/3668988248 learned_addr learned my addr 172.21.15.161:0/3668988248 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:28.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.175+0000 7f89ea36d640 1 --2- 172.21.15.161:0/3668988248 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563dc46dd310 0x563dc46d94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:28.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.175+0000 7f89e936b640 1 -- 172.21.15.161:0/3668988248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563dc46dedb0 msgr2=0x563dc46dc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:28.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.175+0000 7f89e936b640 1 --2- 172.21.15.161:0/3668988248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563dc46dedb0 0x563dc46dc890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.175+0000 7f89e936b640 1 -- 172.21.15.161:0/3668988248 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563dc46dd310 msgr2=0x563dc46d94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:28.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.175+0000 7f89e936b640 1 --2- 172.21.15.161:0/3668988248 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563dc46dd310 0x563dc46d94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.175+0000 7f89e936b640 1 -- 172.21.15.161:0/3668988248 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563dc43572d0 con 0x563dc46df190 2024-03-28T23:36:28.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.175+0000 7f89e936b640 1 --2- 172.21.15.161:0/3668988248 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc46df190 0x563dc46dcdd0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f89d802d050 tx=0x7f89d807a0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b7f3033c8ad9bf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:28.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.175+0000 7f89cb7fe640 1 -- 172.21.15.161:0/3668988248 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d804d030 con 0x563dc46df190 2024-03-28T23:36:28.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.175+0000 7f89cb7fe640 1 -- 172.21.15.161:0/3668988248 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89d807a4e0 con 0x563dc46df190 2024-03-28T23:36:28.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.175+0000 7f89cb7fe640 1 -- 172.21.15.161:0/3668988248 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d807e580 con 0x563dc46df190 2024-03-28T23:36:28.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.176+0000 7f89ee9b5740 1 -- 172.21.15.161:0/3668988248 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc46df190 msgr2=0x563dc46dcdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:28.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.176+0000 7f89ee9b5740 1 --2- 172.21.15.161:0/3668988248 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc46df190 0x563dc46dcdd0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f89d802d050 tx=0x7f89d807a0e0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.176+0000 7f89ee9b5740 1 -- 172.21.15.161:0/3668988248 shutdown_connections 2024-03-28T23:36:28.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.176+0000 7f89ee9b5740 1 --2- 172.21.15.161:0/3668988248 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc46df190 0x563dc46dcdd0 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.176+0000 7f89ee9b5740 1 --2- 172.21.15.161:0/3668988248 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563dc46dd310 0x563dc46d94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.176+0000 7f89ee9b5740 1 --2- 172.21.15.161:0/3668988248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563dc46dedb0 0x563dc46dc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.176+0000 7f89ee9b5740 1 -- 172.21.15.161:0/3668988248 >> 172.21.15.161:0/3668988248 conn(0x563dc46db630 msgr2=0x563dc4700220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:28.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.176+0000 7f89ee9b5740 1 -- 172.21.15.161:0/3668988248 shutdown_connections 2024-03-28T23:36:28.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.176+0000 7f89ee9b5740 1 -- 172.21.15.161:0/3668988248 wait complete. 2024-03-28T23:36:28.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.176+0000 7f89ee9b5740 1 -- start start 2024-03-28T23:36:28.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.177+0000 7f89ee9b5740 1 Processor -- start 2024-03-28T23:36:28.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.178+0000 7f89ee9b5740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563dc46dedb0 0x563dc471b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:28.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.178+0000 7f89ee9b5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563dc46df190 0x563dc472eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:28.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.178+0000 7f89ee9b5740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc4703fd0 0x563dc47179d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:28.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.178+0000 7f89ee9b5740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x563dc4701090 con 0x563dc4703fd0 2024-03-28T23:36:28.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.178+0000 7f89ee9b5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x563dc45fef70 con 0x563dc46df190 2024-03-28T23:36:28.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.178+0000 7f89ee9b5740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x563dc4700ad0 con 0x563dc46dedb0 2024-03-28T23:36:28.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.178+0000 7f89ee9b5740 10 client.0 _ll_register_callbacks cb 0x563dc4356ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:36:28.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.178+0000 7f89ea36d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc4703fd0 0x563dc47179d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:28.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.178+0000 7f89ea36d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc4703fd0 0x563dc47179d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42176/0 (socket says 172.21.15.161:42176) 2024-03-28T23:36:28.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.178+0000 7f89ea36d640 1 -- 172.21.15.161:0/1607648854 learned_addr learned my addr 172.21.15.161:0/1607648854 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:28.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.178+0000 7f89e936b640 1 --2- 172.21.15.161:0/1607648854 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563dc46df190 0x563dc472eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:28.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.178+0000 7f89e9b6c640 1 --2- 172.21.15.161:0/1607648854 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563dc46dedb0 0x563dc471b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:28.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.178+0000 7f89e936b640 1 -- 172.21.15.161:0/1607648854 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563dc46dedb0 msgr2=0x563dc471b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:28.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.178+0000 7f89e936b640 1 --2- 172.21.15.161:0/1607648854 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563dc46dedb0 0x563dc471b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.178+0000 7f89e936b640 1 -- 172.21.15.161:0/1607648854 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc4703fd0 msgr2=0x563dc47179d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:28.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.178+0000 7f89e936b640 1 --2- 172.21.15.161:0/1607648854 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc4703fd0 0x563dc47179d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.178+0000 7f89e936b640 1 -- 172.21.15.161:0/1607648854 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563dc44216b0 con 0x563dc46df190 2024-03-28T23:36:28.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.178+0000 7f89ea36d640 1 --2- 172.21.15.161:0/1607648854 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc4703fd0 0x563dc47179d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:28.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.179+0000 7f89e936b640 1 --2- 172.21.15.161:0/1607648854 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563dc46df190 0x563dc472eb10 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009150 tx=0x7f89d80ac070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55464da6200fc20d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:28.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.179+0000 7f89c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-28T23:36:28.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.179+0000 7f89c8ff9640 1 -- 172.21.15.161:0/1607648854 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d804d070 con 0x563dc46df190 2024-03-28T23:36:28.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.179+0000 7f89c8ff9640 1 -- 172.21.15.161:0/1607648854 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(159..159 src has 1..159) ==== 7215+0+0 (secure 0 0 0) 0x7f89d8076b50 con 0x563dc46df190 2024-03-28T23:36:28.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.179+0000 7f89ee9b5740 1 -- 172.21.15.161:0/1607648854 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89d807a4e0 con 0x563dc46df190 2024-03-28T23:36:28.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.179+0000 7f89c8ff9640 1 -- 172.21.15.161:0/1607648854 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) ==== 24+0+0 (secure 0 0 0) 0x7f89d8082b40 con 0x563dc46df190 2024-03-28T23:36:28.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.179+0000 7f89ee9b5740 10 client.8398 fetch_fsmap learned FSMap version 154 2024-03-28T23:36:28.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.179+0000 7f89ee9b5740 1 -- 172.21.15.161:0/1607648854 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) -- 0x563dc47036a0 con 0x563dc46df190 2024-03-28T23:36:28.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.179+0000 7f89c8ff9640 1 -- 172.21.15.161:0/1607648854 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89d807a4e0 con 0x563dc46df190 2024-03-28T23:36:28.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.179+0000 7f89c8ff9640 1 -- 172.21.15.161:0/1607648854 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 154) ==== 4139+0+0 (secure 0 0 0) 0x7f89d807e170 con 0x563dc46df190 2024-03-28T23:36:28.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.179+0000 7f89ee9b5740 10 client.8398 fetch_fsmap finished waiting for FSMap version 154 2024-03-28T23:36:28.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.179+0000 7f89ee9b5740 10 client.8398 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:36:28.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.179+0000 7f89ee9b5740 20 client.8398 populate_metadata read hostname 'smithi161' 2024-03-28T23:36:28.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.179+0000 7f89ee9b5740 1 --2- 172.21.15.161:0/1607648854 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x563dc46fc8e0 0x563dc46fccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:28.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.179+0000 7f89ee9b5740 4 client.8398 mds_command: new command op to 7283 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:36:28.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.179+0000 7f89ee9b5740 1 -- 172.21.15.161:0/1607648854 --> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563dc4650030 con 0x563dc46fc8e0 2024-03-28T23:36:28.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.179+0000 7f89e9b6c640 1 --2- 172.21.15.161:0/1607648854 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x563dc46fc8e0 0x563dc46fccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:28.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.180+0000 7f89c8ff9640 10 client.8398 ms_handle_connect on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:28.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.180+0000 7f89e9b6c640 1 --2- 172.21.15.161:0/1607648854 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x563dc46fc8e0 0x563dc46fccc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae37499f26aa64da server_cookie=91644651d6faaa62 in_seq=0 out_seq=0 2024-03-28T23:36:28.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.180+0000 7f89c8ff9640 1 -- 172.21.15.161:0/1607648854 <== mds.0 v2:172.21.15.170:6832/2996092857 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563dc4650030 con 0x563dc46fc8e0 2024-03-28T23:36:28.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.180+0000 7f89c8ff9640 10 client.8398 handle_command_reply: tid=0 2024-03-28T23:36:28.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.180+0000 7f89ee9b5740 1 client.8398 shutdown 2024-03-28T23:36:28.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.181+0000 7f89ee9b5740 1 -- 172.21.15.161:0/1607648854 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563dc46df190 msgr2=0x563dc472eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:28.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.181+0000 7f89ee9b5740 1 --2- 172.21.15.161:0/1607648854 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563dc46df190 0x563dc472eb10 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009150 tx=0x7f89d80ac070 comp rx=0 tx=0).stop 2024-03-28T23:36:28.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.181+0000 7f89ee9b5740 1 -- 172.21.15.161:0/1607648854 shutdown_connections 2024-03-28T23:36:28.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.181+0000 7f89ee9b5740 1 --2- 172.21.15.161:0/1607648854 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x563dc46fc8e0 0x563dc46fccc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.181+0000 7f89ee9b5740 1 --2- 172.21.15.161:0/1607648854 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc4703fd0 0x563dc47179d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.181+0000 7f89ee9b5740 1 --2- 172.21.15.161:0/1607648854 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563dc46df190 0x563dc472eb10 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.181+0000 7f89c8ff9640 0 client.8398 ms_handle_reset on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:28.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.181+0000 7f89ee9b5740 1 --2- 172.21.15.161:0/1607648854 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563dc46dedb0 0x563dc471b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.181+0000 7f89ee9b5740 1 -- 172.21.15.161:0/1607648854 >> 172.21.15.161:0/1607648854 conn(0x563dc46dd310 msgr2=0x563dc4700120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:28.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.181+0000 7f89ee9b5740 1 -- 172.21.15.161:0/1607648854 shutdown_connections 2024-03-28T23:36:28.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.181+0000 7f89ee9b5740 1 -- 172.21.15.161:0/1607648854 wait complete. 2024-03-28T23:36:28.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.181+0000 7f89ee9b5740 20 client.8398 trim_cache size 0 max 16384 2024-03-28T23:36:28.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.215+0000 7f89ee9b5740 1 Processor -- start 2024-03-28T23:36:28.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.215+0000 7f89ee9b5740 1 -- start start 2024-03-28T23:36:28.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.216+0000 7f89ee9b5740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc46dedb0 0x563dc4717730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:28.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.216+0000 7f89ee9b5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563dc46df190 0x563dc46fccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:28.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.216+0000 7f89ee9b5740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563dc4703fd0 0x563dc46fd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:28.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.216+0000 7f89ee9b5740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f89e41faf20 con 0x563dc46dedb0 2024-03-28T23:36:28.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.216+0000 7f89ee9b5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f89e41f8980 con 0x563dc46df190 2024-03-28T23:36:28.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.216+0000 7f89ee9b5740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f89e41f8fd0 con 0x563dc4703fd0 2024-03-28T23:36:28.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.216+0000 7f89e9b6c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc46dedb0 0x563dc4717730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:28.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.216+0000 7f89e9b6c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc46dedb0 0x563dc4717730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42192/0 (socket says 172.21.15.161:42192) 2024-03-28T23:36:28.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.216+0000 7f89e9b6c640 1 -- 172.21.15.161:0/2129202954 learned_addr learned my addr 172.21.15.161:0/2129202954 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:28.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.216+0000 7f89e936b640 1 --2- 172.21.15.161:0/2129202954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563dc46df190 0x563dc46fccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:28.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.216+0000 7f89ea36d640 1 --2- 172.21.15.161:0/2129202954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563dc4703fd0 0x563dc46fd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:28.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.217+0000 7f89e9b6c640 1 -- 172.21.15.161:0/2129202954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563dc4703fd0 msgr2=0x563dc46fd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:28.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.217+0000 7f89e9b6c640 1 --2- 172.21.15.161:0/2129202954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563dc4703fd0 0x563dc46fd200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.217+0000 7f89e9b6c640 1 -- 172.21.15.161:0/2129202954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563dc46df190 msgr2=0x563dc46fccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:28.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.217+0000 7f89e9b6c640 1 --2- 172.21.15.161:0/2129202954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563dc46df190 0x563dc46fccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.217+0000 7f89e9b6c640 1 -- 172.21.15.161:0/2129202954 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563dc472beb0 con 0x563dc46dedb0 2024-03-28T23:36:28.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.217+0000 7f89e936b640 1 --2- 172.21.15.161:0/2129202954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563dc46df190 0x563dc46fccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:36:28.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.217+0000 7f89e9b6c640 1 --2- 172.21.15.161:0/2129202954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc46dedb0 0x563dc4717730 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f89d4011690 tx=0x7f89d4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=980a5529bc9383a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:28.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.217+0000 7f89caffd640 1 -- 172.21.15.161:0/2129202954 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d402db10 con 0x563dc46dedb0 2024-03-28T23:36:28.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.217+0000 7f89caffd640 1 -- 172.21.15.161:0/2129202954 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89d4032070 con 0x563dc46dedb0 2024-03-28T23:36:28.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.217+0000 7f89caffd640 1 -- 172.21.15.161:0/2129202954 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d402d4b0 con 0x563dc46dedb0 2024-03-28T23:36:28.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.217+0000 7f89ee9b5740 1 -- 172.21.15.161:0/2129202954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc46dedb0 msgr2=0x563dc4717730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:28.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.217+0000 7f89ee9b5740 1 --2- 172.21.15.161:0/2129202954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc46dedb0 0x563dc4717730 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f89d4011690 tx=0x7f89d4019000 comp rx=0 tx=0).stop 2024-03-28T23:36:28.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.217+0000 7f89ee9b5740 1 -- 172.21.15.161:0/2129202954 shutdown_connections 2024-03-28T23:36:28.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.217+0000 7f89ee9b5740 1 --2- 172.21.15.161:0/2129202954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563dc4703fd0 0x563dc46fd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.217+0000 7f89ee9b5740 1 --2- 172.21.15.161:0/2129202954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563dc46df190 0x563dc46fccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.217+0000 7f89ee9b5740 1 --2- 172.21.15.161:0/2129202954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc46dedb0 0x563dc4717730 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.217+0000 7f89ee9b5740 1 -- 172.21.15.161:0/2129202954 >> 172.21.15.161:0/2129202954 conn(0x563dc46dd310 msgr2=0x563dc471d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:28.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.217+0000 7f89ee9b5740 1 -- 172.21.15.161:0/2129202954 shutdown_connections 2024-03-28T23:36:28.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.218+0000 7f89ee9b5740 1 -- 172.21.15.161:0/2129202954 wait complete. 2024-03-28T23:36:28.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.218+0000 7f89ee9b5740 1 -- start start 2024-03-28T23:36:28.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.218+0000 7f89ee9b5740 1 Processor -- start 2024-03-28T23:36:28.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.220+0000 7f89ee9b5740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563dc46dedb0 0x563dc46fd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:28.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.220+0000 7f89ee9b5740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc46df190 0x563dc47108f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:28.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.220+0000 7f89ee9b5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563dc4703fd0 0x563dc4791d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:28.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.220+0000 7f89ee9b5740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x563dc4701090 con 0x563dc46df190 2024-03-28T23:36:28.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.220+0000 7f89ee9b5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x563dc47182e0 con 0x563dc4703fd0 2024-03-28T23:36:28.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.220+0000 7f89ee9b5740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x563dc472f5c0 con 0x563dc46dedb0 2024-03-28T23:36:28.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.220+0000 7f89ee9b5740 10 client.0 _ll_register_callbacks cb 0x7f89dc006cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:36:28.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.220+0000 7f89e936b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc46df190 0x563dc47108f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:28.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.220+0000 7f89e9b6c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563dc46dedb0 0x563dc46fd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:28.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.220+0000 7f89e936b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc46df190 0x563dc47108f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42204/0 (socket says 172.21.15.161:42204) 2024-03-28T23:36:28.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.220+0000 7f89ea36d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563dc4703fd0 0x563dc4791d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:28.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.220+0000 7f89e936b640 1 -- 172.21.15.161:0/1346204993 learned_addr learned my addr 172.21.15.161:0/1346204993 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:28.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.220+0000 7f89e9b6c640 1 -- 172.21.15.161:0/1346204993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563dc4703fd0 msgr2=0x563dc4791d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:28.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.220+0000 7f89e9b6c640 1 --2- 172.21.15.161:0/1346204993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563dc4703fd0 0x563dc4791d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.220+0000 7f89e9b6c640 1 -- 172.21.15.161:0/1346204993 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc46df190 msgr2=0x563dc47108f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:28.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.220+0000 7f89e9b6c640 1 --2- 172.21.15.161:0/1346204993 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc46df190 0x563dc47108f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.220+0000 7f89e9b6c640 1 -- 172.21.15.161:0/1346204993 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) -- 0x563dc4711880 con 0x563dc46dedb0 2024-03-28T23:36:28.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.220+0000 7f89e936b640 1 --2- 172.21.15.161:0/1346204993 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc46df190 0x563dc47108f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:28.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.220+0000 7f89e9b6c640 1 --2- 172.21.15.161:0/1346204993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563dc46dedb0 0x563dc46fd0c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f89d4031000 tx=0x7f89d4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ef6b185d0289ccd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:28.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.220+0000 7f89c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.170:3301/0 2024-03-28T23:36:28.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.220+0000 7f89c8ff9640 1 -- 172.21.15.161:0/1346204993 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d403a070 con 0x563dc46dedb0 2024-03-28T23:36:28.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.220+0000 7f89c8ff9640 1 -- 172.21.15.161:0/1346204993 <== mon.2 v2:172.21.15.170:3301/0 2 ==== osd_map(159..159 src has 1..159) ==== 7215+0+0 (secure 0 0 0) 0x7f89d4005990 con 0x563dc46dedb0 2024-03-28T23:36:28.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.221+0000 7f89ee9b5740 1 -- 172.21.15.161:0/1346204993 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89d4032070 con 0x563dc46dedb0 2024-03-28T23:36:28.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.221+0000 7f89c8ff9640 1 -- 172.21.15.161:0/1346204993 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_get_version_reply(handle=1 version=154) ==== 24+0+0 (secure 0 0 0) 0x7f89d4005b90 con 0x563dc46dedb0 2024-03-28T23:36:28.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.221+0000 7f89ee9b5740 10 client.7334 fetch_fsmap learned FSMap version 154 2024-03-28T23:36:28.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.221+0000 7f89ee9b5740 1 -- 172.21.15.161:0/1346204993 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) -- 0x563dc47004c0 con 0x563dc46dedb0 2024-03-28T23:36:28.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.221+0000 7f89c8ff9640 1 -- 172.21.15.161:0/1346204993 <== mon.2 v2:172.21.15.170:3301/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89d4032070 con 0x563dc46dedb0 2024-03-28T23:36:28.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.221+0000 7f89c8ff9640 1 -- 172.21.15.161:0/1346204993 <== mon.2 v2:172.21.15.170:3301/0 5 ==== fsmap(e 154) ==== 4139+0+0 (secure 0 0 0) 0x7f89d4047070 con 0x563dc46dedb0 2024-03-28T23:36:28.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.221+0000 7f89ee9b5740 10 client.7334 fetch_fsmap finished waiting for FSMap version 154 2024-03-28T23:36:28.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.221+0000 7f89ee9b5740 10 client.7334 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:36:28.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.221+0000 7f89ee9b5740 20 client.7334 populate_metadata read hostname 'smithi161' 2024-03-28T23:36:28.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.221+0000 7f89ee9b5740 1 --2- 172.21.15.161:0/1346204993 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x563dc471e3e0 0x563dc471e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:28.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.221+0000 7f89ee9b5740 4 client.7334 mds_command: new command op to 7283 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:36:28.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.221+0000 7f89ee9b5740 1 -- 172.21.15.161:0/1346204993 --> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] -- command(tid 0: {"prefix": "status"}) -- 0x563dc464a380 con 0x563dc471e3e0 2024-03-28T23:36:28.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.221+0000 7f89e936b640 1 --2- 172.21.15.161:0/1346204993 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x563dc471e3e0 0x563dc471e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:28.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.221+0000 7f89c8ff9640 10 client.7334 ms_handle_connect on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:28.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.222+0000 7f89e936b640 1 --2- 172.21.15.161:0/1346204993 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x563dc471e3e0 0x563dc471e7c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5a7c744dd7bb710 server_cookie=623b66e94ad6a75b in_seq=0 out_seq=0 2024-03-28T23:36:28.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.222+0000 7f89c8ff9640 1 -- 172.21.15.161:0/1346204993 <== mds.0 v2:172.21.15.170:6832/2996092857 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x563dc464a380 con 0x563dc471e3e0 2024-03-28T23:36:28.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.222+0000 7f89c8ff9640 10 client.7334 handle_command_reply: tid=0 2024-03-28T23:36:28.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.222+0000 7f89ee9b5740 1 client.7334 shutdown 2024-03-28T23:36:28.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.222+0000 7f89ee9b5740 1 -- 172.21.15.161:0/1346204993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563dc46dedb0 msgr2=0x563dc46fd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:28.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.222+0000 7f89ee9b5740 1 --2- 172.21.15.161:0/1346204993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563dc46dedb0 0x563dc46fd0c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f89d4031000 tx=0x7f89d4004730 comp rx=0 tx=0).stop 2024-03-28T23:36:28.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.222+0000 7f89ee9b5740 1 -- 172.21.15.161:0/1346204993 shutdown_connections 2024-03-28T23:36:28.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.222+0000 7f89ee9b5740 1 --2- 172.21.15.161:0/1346204993 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x563dc471e3e0 0x563dc471e7c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.222+0000 7f89ee9b5740 1 --2- 172.21.15.161:0/1346204993 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563dc46df190 0x563dc47108f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.222+0000 7f89ee9b5740 1 --2- 172.21.15.161:0/1346204993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563dc4703fd0 0x563dc4791d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.222+0000 7f89c8ff9640 0 client.7334 ms_handle_reset on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:28.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.222+0000 7f89ee9b5740 1 --2- 172.21.15.161:0/1346204993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563dc46dedb0 0x563dc46fd0c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.222+0000 7f89ee9b5740 1 -- 172.21.15.161:0/1346204993 >> 172.21.15.161:0/1346204993 conn(0x563dc46dd310 msgr2=0x563dc4702ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:28.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.222+0000 7f89ee9b5740 1 -- 172.21.15.161:0/1346204993 shutdown_connections 2024-03-28T23:36:28.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.222+0000 7f89ee9b5740 1 -- 172.21.15.161:0/1346204993 wait complete. 2024-03-28T23:36:28.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.222+0000 7f89ee9b5740 20 client.7334 trim_cache size 0 max 16384 2024-03-28T23:36:28.229 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:36:28.229 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:36:28.229 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:36:28.229 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 7283, 2024-03-28T23:36:28.229 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:36:28.230 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:36:28.230 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:36:28.230 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 10.789519879, 2024-03-28T23:36:28.230 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 154, 2024-03-28T23:36:28.230 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 159, 2024-03-28T23:36:28.230 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 159, 2024-03-28T23:36:28.230 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 13.779343533 2024-03-28T23:36:28.230 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:36:28.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.223+0000 7f89ebdd0640 1 -- 172.21.15.161:0/479597455 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f89d006b4d0 msgr2=0x7f89d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:28.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.223+0000 7f89ebdd0640 1 --2- 172.21.15.161:0/479597455 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f89d006b4d0 0x7f89d006d990 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f89d4009b40 tx=0x7f89d4009190 comp rx=0 tx=0).stop 2024-03-28T23:36:28.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.223+0000 7f89ebdd0640 1 -- 172.21.15.161:0/479597455 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89e40c8ed0 msgr2=0x7f89e41ee840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:28.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.223+0000 7f89ebdd0640 1 --2- 172.21.15.161:0/479597455 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89e40c8ed0 0x7f89e41ee840 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f89d8000c00 tx=0x7f89d80046c0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.223+0000 7f89ebdd0640 1 -- 172.21.15.161:0/479597455 shutdown_connections 2024-03-28T23:36:28.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.223+0000 7f89ebdd0640 1 --2- 172.21.15.161:0/479597455 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f89d006b4d0 0x7f89d006d990 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.223+0000 7f89ebdd0640 1 --2- 172.21.15.161:0/479597455 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89e40c8ed0 0x7f89e41ee840 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.223+0000 7f89ebdd0640 1 --2- 172.21.15.161:0/479597455 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89e4152890 0x7f89e4149b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.223+0000 7f89ebdd0640 1 --2- 172.21.15.161:0/479597455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89e40c5950 0x7f89e40c31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:28.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.223+0000 7f89ebdd0640 1 -- 172.21.15.161:0/479597455 >> 172.21.15.161:0/479597455 conn(0x7f89e40ac620 msgr2=0x7f89e4151a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:28.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.223+0000 7f89ebdd0640 1 -- 172.21.15.161:0/479597455 shutdown_connections 2024-03-28T23:36:28.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:28.223+0000 7f89ebdd0640 1 -- 172.21.15.161:0/479597455 wait complete. 2024-03-28T23:36:28.234 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 11, "epoch": 154, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 11, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:35:58.406403+0000", "damaged": [], "data_pools": [ 24 ], "enabled": true, "epoch": 154, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7283": { "addr": "172.21.15.170:6834/2996092857", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 2996092857, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 2996092857, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7283, "incarnation": 151, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 159, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 23, "modified": "2024-03-28T23:36:20.440473+0000", "qdb_cluster": [ 7283 ], "qdb_leader": 7283, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7283 } } }, { "id": 12, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:36:07.448976+0000", "damaged": [], "data_pools": [ 26 ], "enabled": true, "epoch": 149, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs2", "in": [ 0 ], "info": { "gid_7061": { "addr": "172.21.15.161:6836/4026332703", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 4026332703, "type": "v2" }, { "addr": "172.21.15.161:6836", "nonce": 4026332703, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7061, "incarnation": 145, "join_fscid": 12, "name": "c", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 157, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 25, "modified": "2024-03-28T23:36:16.420228+0000", "qdb_cluster": [ 7061 ], "qdb_leader": 7061, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7061 } } } ], "standbys": [ { "addr": "172.21.15.170:6835/557522063", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 557522063, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 557522063, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 137, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7070, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6837/789976100", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6835", "nonce": 789976100, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 789976100, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 152, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9216, "incarnation": 0, "join_fscid": 12, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:36:28.235 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': 12, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 12, 'name': 'c'}, {'join_fscid': -1, 'name': 'd', 'state': 'up:active'}] 2024-03-28T23:36:28.235 INFO:tasks.cephfs.test_failover:current = [{'gid': 9216, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6837/789976100', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 789976100}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 789976100}]}, 'join_fscid': 12, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 152}, {'gid': 7283, 'name': 'b', 'rank': 0, 'incarnation': 151, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.170:6834/2996092857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 2996092857}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 2996092857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 7061, 'name': 'c', 'rank': 0, 'incarnation': 145, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.161:6836/4026332703', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 4026332703}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 4026332703}]}, 'join_fscid': 12, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 7070, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/557522063', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 557522063}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 557522063}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 137}] 2024-03-28T23:36:28.235 DEBUG:tasks.cephfs.test_failover:'up:active' != 'up:standby' - up:active + up:standby 2024-03-28T23:36:28.236 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-28T23:36:33.236 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:36:33.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.338+0000 7f8cc275c640 1 Processor -- start 2024-03-28T23:36:33.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.339+0000 7f8cc275c640 1 -- start start 2024-03-28T23:36:33.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.339+0000 7f8cc275c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8cbc152890 0x7f8cbc0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:33.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.339+0000 7f8cc275c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cbc158a30 con 0x7f8cbc0c8ed0 2024-03-28T23:36:33.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.339+0000 7f8cc275c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cbc158d10 con 0x7f8cbc0c5b20 2024-03-28T23:36:33.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.339+0000 7f8cc275c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8cbc0c8480 con 0x7f8cbc152890 2024-03-28T23:36:33.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.339+0000 7f8cbbfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8cbc152890 0x7f8cbc0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:33.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.339+0000 7f8cbb7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f8cbc0c8ed0 0x7f8cbc0c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44102/0 (socket says 172.21.15.161:44102) 2024-03-28T23:36:33.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.339+0000 7f8cbb7fe640 1 -- 172.21.15.161:0/610094841 learned_addr learned my addr 172.21.15.161:0/610094841 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:33.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.340+0000 7f8cbaffd640 1 -- 172.21.15.161:0/610094841 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1077759797 0 0) 0x7f8cbc158d10 con 0x7f8cbc0c5b20 2024-03-28T23:36:33.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.340+0000 7f8cbaffd640 1 -- 172.21.15.161:0/610094841 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ca4003520 con 0x7f8cbc0c5b20 2024-03-28T23:36:33.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.340+0000 7f8cbbfff640 1 -- 172.21.15.161:0/610094841 >> v1:172.21.15.170:6789/0 conn(0x7f8cbc0c5b20 legacy=0x7f8cbc1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:33.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.340+0000 7f8cbbfff640 1 -- 172.21.15.161:0/610094841 >> v1:172.21.15.161:6789/0 conn(0x7f8cbc0c8ed0 legacy=0x7f8cbc0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:33.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.340+0000 7f8cbaffd640 1 -- 172.21.15.161:0/610094841 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3916464642 0 0) 0x7f8cbc158a30 con 0x7f8cbc0c8ed0 2024-03-28T23:36:33.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.340+0000 7f8cbbfff640 1 -- 172.21.15.161:0/610094841 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cbc158ff0 con 0x7f8cbc152890 2024-03-28T23:36:33.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.341+0000 7f8cbbfff640 1 --2- 172.21.15.161:0/610094841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cbc152890 0x7f8cbc0c8990 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f8cac0097a0 tx=0x7f8cac02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4ac04ad00214ba5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:33.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.341+0000 7f8cbaffd640 1 -- 172.21.15.161:0/610094841 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cac0315a0 con 0x7f8cbc152890 2024-03-28T23:36:33.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.341+0000 7f8cbaffd640 1 -- 172.21.15.161:0/610094841 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cac037070 con 0x7f8cbc152890 2024-03-28T23:36:33.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.341+0000 7f8cbaffd640 1 -- 172.21.15.161:0/610094841 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cac031940 con 0x7f8cbc152890 2024-03-28T23:36:33.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.342+0000 7f8cc275c640 1 -- 172.21.15.161:0/610094841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cbc152890 msgr2=0x7f8cbc0c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:33.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.342+0000 7f8cc275c640 1 --2- 172.21.15.161:0/610094841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cbc152890 0x7f8cbc0c8990 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f8cac0097a0 tx=0x7f8cac02b840 comp rx=0 tx=0).stop 2024-03-28T23:36:33.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.342+0000 7f8cc275c640 1 -- 172.21.15.161:0/610094841 shutdown_connections 2024-03-28T23:36:33.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.342+0000 7f8cc275c640 1 --2- 172.21.15.161:0/610094841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cbc152890 0x7f8cbc0c8990 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:33.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.342+0000 7f8cc275c640 1 -- 172.21.15.161:0/610094841 >> 172.21.15.161:0/610094841 conn(0x7f8cbc0ac620 msgr2=0x7f8cbc0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:33.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.342+0000 7f8cc275c640 1 -- 172.21.15.161:0/610094841 shutdown_connections 2024-03-28T23:36:33.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.342+0000 7f8cc275c640 1 -- 172.21.15.161:0/610094841 wait complete. 2024-03-28T23:36:33.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.343+0000 7f8cc275c640 1 Processor -- start 2024-03-28T23:36:33.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.344+0000 7f8cc275c640 1 -- start start 2024-03-28T23:36:33.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.345+0000 7f8cc275c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cbc0c5b20 0x7f8cbc14e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:33.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.345+0000 7f8cc275c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cbc0c8ed0 0x7f8cbc1edf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:33.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.345+0000 7f8cc275c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cbc152890 0x7f8cbc0c31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:33.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.345+0000 7f8cc275c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8cbc152c70 con 0x7f8cbc0c8ed0 2024-03-28T23:36:33.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.345+0000 7f8cc275c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8cbc159640 con 0x7f8cbc152890 2024-03-28T23:36:33.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.345+0000 7f8cc275c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8cbc152df0 con 0x7f8cbc0c5b20 2024-03-28T23:36:33.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.345+0000 7f8cbb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cbc0c8ed0 0x7f8cbc1edf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:33.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.345+0000 7f8cbb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cbc0c8ed0 0x7f8cbc1edf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43800/0 (socket says 172.21.15.161:43800) 2024-03-28T23:36:33.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.345+0000 7f8cbb7fe640 1 -- 172.21.15.161:0/3514392645 learned_addr learned my addr 172.21.15.161:0/3514392645 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:33.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.345+0000 7f8cbbfff640 1 --2- 172.21.15.161:0/3514392645 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cbc0c5b20 0x7f8cbc14e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:33.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.345+0000 7f8cc0cf9640 1 --2- 172.21.15.161:0/3514392645 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cbc152890 0x7f8cbc0c31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:33.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.345+0000 7f8cbb7fe640 1 -- 172.21.15.161:0/3514392645 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cbc0c5b20 msgr2=0x7f8cbc14e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:33.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.345+0000 7f8cbb7fe640 1 --2- 172.21.15.161:0/3514392645 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cbc0c5b20 0x7f8cbc14e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:33.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.345+0000 7f8cbb7fe640 1 -- 172.21.15.161:0/3514392645 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cbc152890 msgr2=0x7f8cbc0c31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:33.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.345+0000 7f8cbb7fe640 1 --2- 172.21.15.161:0/3514392645 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cbc152890 0x7f8cbc0c31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:33.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.345+0000 7f8cbb7fe640 1 -- 172.21.15.161:0/3514392645 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cbc14ddc0 con 0x7f8cbc0c8ed0 2024-03-28T23:36:33.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.345+0000 7f8cbb7fe640 1 --2- 172.21.15.161:0/3514392645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cbc0c8ed0 0x7f8cbc1edf40 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f8cb000c730 tx=0x7f8cb000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:33.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.345+0000 7f8cb8ff9640 1 -- 172.21.15.161:0/3514392645 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb0017070 con 0x7f8cbc0c8ed0 2024-03-28T23:36:33.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.346+0000 7f8cb8ff9640 1 -- 172.21.15.161:0/3514392645 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cb0007270 con 0x7f8cbc0c8ed0 2024-03-28T23:36:33.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.346+0000 7f8cc275c640 1 -- 172.21.15.161:0/3514392645 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cbc1ee880 con 0x7f8cbc0c8ed0 2024-03-28T23:36:33.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.346+0000 7f8cb8ff9640 1 -- 172.21.15.161:0/3514392645 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb0001d00 con 0x7f8cbc0c8ed0 2024-03-28T23:36:33.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.346+0000 7f8cc275c640 1 -- 172.21.15.161:0/3514392645 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cbc14e340 con 0x7f8cbc0c8ed0 2024-03-28T23:36:33.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.347+0000 7f8cc275c640 1 -- 172.21.15.161:0/3514392645 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cbc1eebc0 con 0x7f8cbc0c8ed0 2024-03-28T23:36:33.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.347+0000 7f8cb8ff9640 1 -- 172.21.15.161:0/3514392645 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8cb001a070 con 0x7f8cbc0c8ed0 2024-03-28T23:36:33.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.347+0000 7f8cb8ff9640 1 --2- 172.21.15.161:0/3514392645 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8ca806b4d0 0x7f8ca806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:33.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.348+0000 7f8cb8ff9640 1 -- 172.21.15.161:0/3514392645 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 7215+0+0 (secure 0 0 0) 0x7f8cb0089a60 con 0x7f8cbc0c8ed0 2024-03-28T23:36:33.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.348+0000 7f8cbbfff640 1 --2- 172.21.15.161:0/3514392645 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8ca806b4d0 0x7f8ca806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:33.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.348+0000 7f8cbbfff640 1 --2- 172.21.15.161:0/3514392645 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8ca806b4d0 0x7f8ca806d990 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f8cac036000 tx=0x7f8cac008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:33.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.352+0000 7f8cb8ff9640 1 -- 172.21.15.161:0/3514392645 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8cb0057cc0 con 0x7f8cbc0c8ed0 2024-03-28T23:36:33.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.589+0000 7f8cc275c640 1 -- 172.21.15.161:0/3514392645 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8cbc04f4b0 con 0x7f8cbc0c8ed0 2024-03-28T23:36:33.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.590+0000 7f8cb8ff9640 1 -- 172.21.15.161:0/3514392645 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+6520 (secure 0 0 0) 0x7f8cb005bd60 con 0x7f8cbc0c8ed0 2024-03-28T23:36:33.592 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:33.592 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":154,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/557522063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":557522063},{"type":"v1","addr":"172.21.15.170:6835","nonce":557522063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":9216,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/789976100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":789976100},{"type":"v1","addr":"172.21.15.161:6837","nonce":789976100}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:58.406403+0000","modified":"2024-03-28T23:36:20.440473+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":159,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7283":{"gid":7283,"name":"b","rank":0,"incarnation":151,"state":"up:active","state_seq":4,"addr":"172.21.15.170:6834/2996092857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":2996092857},{"type":"v1","addr":"172.21.15.170:6834","nonce":2996092857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7283,"qdb_cluster":[7283]},"id":11},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:36:07.448976+0000","modified":"2024-03-28T23:36:16.420228+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7061":{"gid":7061,"name":"c","rank":0,"incarnation":145,"state":"up:active","state_seq":9,"addr":"172.21.15.161:6836/4026332703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4026332703},{"type":"v1","addr":"172.21.15.161:6836","nonce":4026332703}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7061,"qdb_cluster":[7061]},"id":12}]} 2024-03-28T23:36:33.594 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 154 2024-03-28T23:36:33.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.592+0000 7f8cc275c640 1 -- 172.21.15.161:0/3514392645 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8ca806b4d0 msgr2=0x7f8ca806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:33.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.592+0000 7f8cc275c640 1 --2- 172.21.15.161:0/3514392645 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8ca806b4d0 0x7f8ca806d990 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f8cac036000 tx=0x7f8cac008030 comp rx=0 tx=0).stop 2024-03-28T23:36:33.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.592+0000 7f8cc275c640 1 -- 172.21.15.161:0/3514392645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cbc0c8ed0 msgr2=0x7f8cbc1edf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:33.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.592+0000 7f8cc275c640 1 --2- 172.21.15.161:0/3514392645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cbc0c8ed0 0x7f8cbc1edf40 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f8cb000c730 tx=0x7f8cb000cbf0 comp rx=0 tx=0).stop 2024-03-28T23:36:33.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.592+0000 7f8cc275c640 1 -- 172.21.15.161:0/3514392645 shutdown_connections 2024-03-28T23:36:33.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.592+0000 7f8cc275c640 1 --2- 172.21.15.161:0/3514392645 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8ca806b4d0 0x7f8ca806d990 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:33.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.592+0000 7f8cc275c640 1 --2- 172.21.15.161:0/3514392645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cbc0c8ed0 0x7f8cbc1edf40 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:33.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.592+0000 7f8cc275c640 1 --2- 172.21.15.161:0/3514392645 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cbc152890 0x7f8cbc0c31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:33.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.592+0000 7f8cc275c640 1 --2- 172.21.15.161:0/3514392645 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cbc0c5b20 0x7f8cbc14e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:33.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.592+0000 7f8cc275c640 1 -- 172.21.15.161:0/3514392645 >> 172.21.15.161:0/3514392645 conn(0x7f8cbc0ac620 msgr2=0x7f8cbc0c74a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:33.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.592+0000 7f8cc275c640 1 -- 172.21.15.161:0/3514392645 shutdown_connections 2024-03-28T23:36:33.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:33.592+0000 7f8cc275c640 1 -- 172.21.15.161:0/3514392645 wait complete. 2024-03-28T23:36:34.010 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 154, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:35:58.406403+0000', 'modified': '2024-03-28T23:36:20.440473+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 159, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7283}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7283': {'gid': 7283, 'name': 'b', 'rank': 0, 'incarnation': 151, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.170:6834/2996092857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 2996092857}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 2996092857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7283, 'qdb_cluster': [7283]} 2024-03-28T23:36:34.011 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:36:34.011 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T23:36:34.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.114+0000 7fc691040640 1 Processor -- start 2024-03-28T23:36:34.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.114+0000 7fc691040640 1 -- start start 2024-03-28T23:36:34.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.115+0000 7fc691040640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc68c0c5850 0x7fc68c155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:34.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.115+0000 7fc691040640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc68c0c5d00 con 0x7fc68c0c5410 2024-03-28T23:36:34.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.115+0000 7fc691040640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc68c158a30 con 0x7fc68c152890 2024-03-28T23:36:34.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.115+0000 7fc691040640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc68c0c8480 con 0x7fc68c0c5850 2024-03-28T23:36:34.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.115+0000 7fc68a59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc68c0c5410 0x7fc68c1553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44114/0 (socket says 172.21.15.161:44114) 2024-03-28T23:36:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.115+0000 7fc68a59c640 1 -- 172.21.15.161:0/4225825373 learned_addr learned my addr 172.21.15.161:0/4225825373 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.115+0000 7fc68b59e640 1 --2- 172.21.15.161:0/4225825373 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc68c0c5850 0x7fc68c155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.115+0000 7fc689d9b640 1 -- 172.21.15.161:0/4225825373 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 819509164 0 0) 0x7fc68c0c5d00 con 0x7fc68c0c5410 2024-03-28T23:36:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.115+0000 7fc689d9b640 1 -- 172.21.15.161:0/4225825373 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc674003520 con 0x7fc68c0c5410 2024-03-28T23:36:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.115+0000 7fc689d9b640 1 -- 172.21.15.161:0/4225825373 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3625600596 0 0) 0x7fc68c158a30 con 0x7fc68c152890 2024-03-28T23:36:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.115+0000 7fc689d9b640 1 -- 172.21.15.161:0/4225825373 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc68c0c5d00 con 0x7fc68c152890 2024-03-28T23:36:34.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.115+0000 7fc68b59e640 1 -- 172.21.15.161:0/4225825373 >> v1:172.21.15.170:6789/0 conn(0x7fc68c152890 legacy=0x7fc68c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:34.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.115+0000 7fc68b59e640 1 -- 172.21.15.161:0/4225825373 >> v1:172.21.15.161:6789/0 conn(0x7fc68c0c5410 legacy=0x7fc68c1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:34.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.115+0000 7fc68b59e640 1 -- 172.21.15.161:0/4225825373 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc68c158d10 con 0x7fc68c0c5850 2024-03-28T23:36:34.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.116+0000 7fc68b59e640 1 --2- 172.21.15.161:0/4225825373 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc68c0c5850 0x7fc68c155ab0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc684009dc0 tx=0x7fc68402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d436f5c977c3e8f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:34.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.116+0000 7fc689d9b640 1 -- 172.21.15.161:0/4225825373 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc684039070 con 0x7fc68c0c5850 2024-03-28T23:36:34.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.116+0000 7fc689d9b640 1 -- 172.21.15.161:0/4225825373 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc68402cc40 con 0x7fc68c0c5850 2024-03-28T23:36:34.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.116+0000 7fc689d9b640 1 -- 172.21.15.161:0/4225825373 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc684034440 con 0x7fc68c0c5850 2024-03-28T23:36:34.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.117+0000 7fc691040640 1 -- 172.21.15.161:0/4225825373 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc68c0c5850 msgr2=0x7fc68c155ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:34.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.117+0000 7fc691040640 1 --2- 172.21.15.161:0/4225825373 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc68c0c5850 0x7fc68c155ab0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc684009dc0 tx=0x7fc68402be90 comp rx=0 tx=0).stop 2024-03-28T23:36:34.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.117+0000 7fc691040640 1 -- 172.21.15.161:0/4225825373 shutdown_connections 2024-03-28T23:36:34.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.117+0000 7fc691040640 1 --2- 172.21.15.161:0/4225825373 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc68c0c5850 0x7fc68c155ab0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.117+0000 7fc691040640 1 -- 172.21.15.161:0/4225825373 >> 172.21.15.161:0/4225825373 conn(0x7fc68c0ac620 msgr2=0x7fc68c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:34.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.117+0000 7fc691040640 1 -- 172.21.15.161:0/4225825373 shutdown_connections 2024-03-28T23:36:34.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.117+0000 7fc691040640 1 -- 172.21.15.161:0/4225825373 wait complete. 2024-03-28T23:36:34.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.118+0000 7fc691040640 1 Processor -- start 2024-03-28T23:36:34.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.118+0000 7fc691040640 1 -- start start 2024-03-28T23:36:34.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.119+0000 7fc691040640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc68c0c5410 0x7fc68c157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:34.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.119+0000 7fc691040640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc68c0c5850 0x7fc68c0c3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.119+0000 7fc691040640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc68c152890 0x7fc68c0c4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.119+0000 7fc691040640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc68c1592d0 con 0x7fc68c0c5410 2024-03-28T23:36:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.119+0000 7fc691040640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc68c159640 con 0x7fc68c152890 2024-03-28T23:36:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.119+0000 7fc691040640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc68c155ff0 con 0x7fc68c0c5850 2024-03-28T23:36:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.119+0000 7fc68ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc68c0c5410 0x7fc68c157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:34.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.119+0000 7fc68ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc68c0c5410 0x7fc68c157f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43802/0 (socket says 172.21.15.161:43802) 2024-03-28T23:36:34.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.119+0000 7fc68b59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc68c152890 0x7fc68c0c4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:34.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.119+0000 7fc68ad9d640 1 -- 172.21.15.161:0/2652536916 learned_addr learned my addr 172.21.15.161:0/2652536916 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:34.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.119+0000 7fc68a59c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc68c0c5850 0x7fc68c0c3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:34.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.119+0000 7fc68ad9d640 1 -- 172.21.15.161:0/2652536916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc68c0c5850 msgr2=0x7fc68c0c3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:34.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.119+0000 7fc68ad9d640 1 --2- 172.21.15.161:0/2652536916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc68c0c5850 0x7fc68c0c3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.119+0000 7fc68ad9d640 1 -- 172.21.15.161:0/2652536916 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc68c152890 msgr2=0x7fc68c0c4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:34.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.119+0000 7fc68ad9d640 1 --2- 172.21.15.161:0/2652536916 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc68c152890 0x7fc68c0c4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.119+0000 7fc68ad9d640 1 -- 172.21.15.161:0/2652536916 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc68c154d90 con 0x7fc68c0c5410 2024-03-28T23:36:34.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.120+0000 7fc68ad9d640 1 --2- 172.21.15.161:0/2652536916 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc68c0c5410 0x7fc68c157f60 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fc67c00c430 tx=0x7fc67c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:34.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.120+0000 7fc6737fe640 1 -- 172.21.15.161:0/2652536916 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67c017070 con 0x7fc68c0c5410 2024-03-28T23:36:34.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.120+0000 7fc6737fe640 1 -- 172.21.15.161:0/2652536916 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc67c0052f0 con 0x7fc68c0c5410 2024-03-28T23:36:34.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.120+0000 7fc691040640 1 -- 172.21.15.161:0/2652536916 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc68c1eebc0 con 0x7fc68c0c5410 2024-03-28T23:36:34.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.120+0000 7fc6737fe640 1 -- 172.21.15.161:0/2652536916 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67c003830 con 0x7fc68c0c5410 2024-03-28T23:36:34.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.120+0000 7fc691040640 1 -- 172.21.15.161:0/2652536916 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc68c155310 con 0x7fc68c0c5410 2024-03-28T23:36:34.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.122+0000 7fc6737fe640 1 -- 172.21.15.161:0/2652536916 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc67c01a070 con 0x7fc68c0c5410 2024-03-28T23:36:34.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.122+0000 7fc6737fe640 1 --2- 172.21.15.161:0/2652536916 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc67806b4d0 0x7fc67806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:34.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.122+0000 7fc68a59c640 1 --2- 172.21.15.161:0/2652536916 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc67806b4d0 0x7fc67806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:34.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.122+0000 7fc6737fe640 1 -- 172.21.15.161:0/2652536916 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 7215+0+0 (secure 0 0 0) 0x7fc67c089a50 con 0x7fc68c0c5410 2024-03-28T23:36:34.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.123+0000 7fc68a59c640 1 --2- 172.21.15.161:0/2652536916 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc67806b4d0 0x7fc67806d990 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fc68c1514d0 tx=0x7fc680007be0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:34.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.123+0000 7fc693c24740 1 Processor -- start 2024-03-28T23:36:34.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.123+0000 7fc693c24740 1 -- start start 2024-03-28T23:36:34.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.124+0000 7fc693c24740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5603e42a6db0 0x5603e42a4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:34.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.124+0000 7fc693c24740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5603e42a7190 0x5603e42a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:34.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.124+0000 7fc693c24740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a5310 0x5603e42a14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:34.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.124+0000 7fc693c24740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5603e420aaf0 con 0x5603e42a5310 2024-03-28T23:36:34.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.124+0000 7fc693c24740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5603e41b39d0 con 0x5603e42a7190 2024-03-28T23:36:34.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.124+0000 7fc693c24740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x5603e4206b90 con 0x5603e42a6db0 2024-03-28T23:36:34.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.124+0000 7fc68b59e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a5310 0x5603e42a14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:34.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.124+0000 7fc68a59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5603e42a7190 0x5603e42a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:34.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.124+0000 7fc68a59c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5603e42a7190 0x5603e42a4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:40732/0 (socket says 172.21.15.161:40732) 2024-03-28T23:36:34.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.124+0000 7fc68ad9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5603e42a6db0 0x5603e42a4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:34.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.124+0000 7fc68a59c640 1 -- 172.21.15.161:0/4229485308 learned_addr learned my addr 172.21.15.161:0/4229485308 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:34.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.124+0000 7fc68b59e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a5310 0x5603e42a14e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43806/0 (socket says 172.21.15.161:43806) 2024-03-28T23:36:34.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.124+0000 7fc68a59c640 1 -- 172.21.15.161:0/4229485308 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5603e42a6db0 msgr2=0x5603e42a4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:34.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.124+0000 7fc68a59c640 1 --2- 172.21.15.161:0/4229485308 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5603e42a6db0 0x5603e42a4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.124+0000 7fc68a59c640 1 -- 172.21.15.161:0/4229485308 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a5310 msgr2=0x5603e42a14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:34.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.124+0000 7fc68a59c640 1 --2- 172.21.15.161:0/4229485308 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a5310 0x5603e42a14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.124+0000 7fc68a59c640 1 -- 172.21.15.161:0/4229485308 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603e3f1f2d0 con 0x5603e42a7190 2024-03-28T23:36:34.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.124+0000 7fc68ad9d640 1 --2- 172.21.15.161:0/4229485308 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5603e42a6db0 0x5603e42a4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:34.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.124+0000 7fc68a59c640 1 --2- 172.21.15.161:0/4229485308 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5603e42a7190 0x5603e42a4dd0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fc68c0ad1f0 tx=0x7fc680012000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4689e466f3f3e3f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:34.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.124+0000 7fc670ff9640 1 -- 172.21.15.161:0/4229485308 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc680025070 con 0x5603e42a7190 2024-03-28T23:36:34.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.124+0000 7fc670ff9640 1 -- 172.21.15.161:0/4229485308 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc68001b050 con 0x5603e42a7190 2024-03-28T23:36:34.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.124+0000 7fc670ff9640 1 -- 172.21.15.161:0/4229485308 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc680014dd0 con 0x5603e42a7190 2024-03-28T23:36:34.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.125+0000 7fc693c24740 1 -- 172.21.15.161:0/4229485308 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5603e42a7190 msgr2=0x5603e42a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:34.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.125+0000 7fc693c24740 1 --2- 172.21.15.161:0/4229485308 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5603e42a7190 0x5603e42a4dd0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fc68c0ad1f0 tx=0x7fc680012000 comp rx=0 tx=0).stop 2024-03-28T23:36:34.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.125+0000 7fc693c24740 1 -- 172.21.15.161:0/4229485308 shutdown_connections 2024-03-28T23:36:34.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.125+0000 7fc693c24740 1 --2- 172.21.15.161:0/4229485308 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a5310 0x5603e42a14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.125+0000 7fc693c24740 1 --2- 172.21.15.161:0/4229485308 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5603e42a7190 0x5603e42a4dd0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.125+0000 7fc693c24740 1 --2- 172.21.15.161:0/4229485308 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5603e42a6db0 0x5603e42a4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.125+0000 7fc693c24740 1 -- 172.21.15.161:0/4229485308 >> 172.21.15.161:0/4229485308 conn(0x5603e42a3630 msgr2=0x5603e42c8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:34.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.125+0000 7fc693c24740 1 -- 172.21.15.161:0/4229485308 shutdown_connections 2024-03-28T23:36:34.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.125+0000 7fc693c24740 1 -- 172.21.15.161:0/4229485308 wait complete. 2024-03-28T23:36:34.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.126+0000 7fc693c24740 1 -- start start 2024-03-28T23:36:34.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.126+0000 7fc693c24740 1 Processor -- start 2024-03-28T23:36:34.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc693c24740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a6db0 0x5603e42a1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:34.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc693c24740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5603e42a7190 0x5603e42ca890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:34.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc693c24740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5603e42f66b0 0x5603e42c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:34.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc693c24740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5603e42c9090 con 0x5603e42a6db0 2024-03-28T23:36:34.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc693c24740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5603e41c6f70 con 0x5603e42a7190 2024-03-28T23:36:34.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc693c24740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x5603e42c8ad0 con 0x5603e42f66b0 2024-03-28T23:36:34.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc693c24740 10 client.0 _ll_register_callbacks cb 0x5603e3f1eac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:36:34.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc68ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a6db0 0x5603e42a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:34.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc68ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a6db0 0x5603e42a1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43818/0 (socket says 172.21.15.161:43818) 2024-03-28T23:36:34.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc68ad9d640 1 -- 172.21.15.161:0/3197918827 learned_addr learned my addr 172.21.15.161:0/3197918827 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:34.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc68a59c640 1 --2- 172.21.15.161:0/3197918827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5603e42a7190 0x5603e42ca890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:34.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc68b59e640 1 --2- 172.21.15.161:0/3197918827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5603e42f66b0 0x5603e42c9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:34.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc68ad9d640 1 -- 172.21.15.161:0/3197918827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5603e42f66b0 msgr2=0x5603e42c9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:34.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc68ad9d640 1 --2- 172.21.15.161:0/3197918827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5603e42f66b0 0x5603e42c9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc68ad9d640 1 -- 172.21.15.161:0/3197918827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5603e42a7190 msgr2=0x5603e42ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:34.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc68ad9d640 1 --2- 172.21.15.161:0/3197918827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5603e42a7190 0x5603e42ca890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc68ad9d640 1 -- 172.21.15.161:0/3197918827 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603e3fe96b0 con 0x5603e42a6db0 2024-03-28T23:36:34.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc68ad9d640 1 --2- 172.21.15.161:0/3197918827 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a6db0 0x5603e42a1730 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fc67c057a30 tx=0x7fc67c05bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bbf0376e3fd9592 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:34.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc669ffb640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:36:34.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc669ffb640 1 -- 172.21.15.161:0/3197918827 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67c057380 con 0x5603e42a6db0 2024-03-28T23:36:34.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc669ffb640 1 -- 172.21.15.161:0/3197918827 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(159..159 src has 1..159) ==== 7215+0+0 (secure 0 0 0) 0x7fc67c053a80 con 0x5603e42a6db0 2024-03-28T23:36:34.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc693c24740 1 -- 172.21.15.161:0/3197918827 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc68001b050 con 0x5603e42a6db0 2024-03-28T23:36:34.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc669ffb640 1 -- 172.21.15.161:0/3197918827 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) ==== 24+0+0 (secure 0 0 0) 0x7fc67c053c80 con 0x5603e42a6db0 2024-03-28T23:36:34.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc693c24740 10 client.9303 fetch_fsmap learned FSMap version 154 2024-03-28T23:36:34.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.127+0000 7fc693c24740 1 -- 172.21.15.161:0/3197918827 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) -- 0x5603e42cd4d0 con 0x5603e42a6db0 2024-03-28T23:36:34.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.128+0000 7fc669ffb640 1 -- 172.21.15.161:0/3197918827 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc68001b050 con 0x5603e42a6db0 2024-03-28T23:36:34.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.128+0000 7fc669ffb640 1 -- 172.21.15.161:0/3197918827 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 154) ==== 4139+0+0 (secure 0 0 0) 0x7fc67c056bf0 con 0x5603e42a6db0 2024-03-28T23:36:34.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.128+0000 7fc693c24740 10 client.9303 fetch_fsmap finished waiting for FSMap version 154 2024-03-28T23:36:34.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.128+0000 7fc693c24740 10 client.9303 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:36:34.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.128+0000 7fc693c24740 20 client.9303 populate_metadata read hostname 'smithi161' 2024-03-28T23:36:34.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.128+0000 7fc693c24740 1 --2- 172.21.15.161:0/3197918827 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x5603e42cba60 0x5603e42cbe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:34.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.128+0000 7fc693c24740 4 client.9303 mds_command: new command op to 7283 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:36:34.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.128+0000 7fc693c24740 1 -- 172.21.15.161:0/3197918827 --> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603e4218030 con 0x5603e42cba60 2024-03-28T23:36:34.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.128+0000 7fc68a59c640 1 --2- 172.21.15.161:0/3197918827 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x5603e42cba60 0x5603e42cbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:34.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.128+0000 7fc68a59c640 1 --2- 172.21.15.161:0/3197918827 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x5603e42cba60 0x5603e42cbe60 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eecf388b11d94046 server_cookie=ce25d24ff1911f12 in_seq=0 out_seq=0 2024-03-28T23:36:34.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.128+0000 7fc669ffb640 10 client.9303 ms_handle_connect on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:34.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.129+0000 7fc669ffb640 1 -- 172.21.15.161:0/3197918827 <== mds.0 v2:172.21.15.170:6832/2996092857 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5603e4218030 con 0x5603e42cba60 2024-03-28T23:36:34.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.129+0000 7fc669ffb640 10 client.9303 handle_command_reply: tid=0 2024-03-28T23:36:34.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.129+0000 7fc693c24740 1 client.9303 shutdown 2024-03-28T23:36:34.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.129+0000 7fc693c24740 1 -- 172.21.15.161:0/3197918827 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a6db0 msgr2=0x5603e42a1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:34.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.129+0000 7fc693c24740 1 --2- 172.21.15.161:0/3197918827 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a6db0 0x5603e42a1730 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fc67c057a30 tx=0x7fc67c05bf10 comp rx=0 tx=0).stop 2024-03-28T23:36:34.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.129+0000 7fc693c24740 1 -- 172.21.15.161:0/3197918827 shutdown_connections 2024-03-28T23:36:34.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.129+0000 7fc693c24740 1 --2- 172.21.15.161:0/3197918827 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x5603e42cba60 0x5603e42cbe60 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.129+0000 7fc693c24740 1 --2- 172.21.15.161:0/3197918827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5603e42f66b0 0x5603e42c9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.129+0000 7fc669ffb640 0 client.9303 ms_handle_reset on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:34.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.129+0000 7fc693c24740 1 --2- 172.21.15.161:0/3197918827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5603e42a7190 0x5603e42ca890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.129+0000 7fc693c24740 1 --2- 172.21.15.161:0/3197918827 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a6db0 0x5603e42a1730 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.129+0000 7fc693c24740 1 -- 172.21.15.161:0/3197918827 >> 172.21.15.161:0/3197918827 conn(0x5603e42a5310 msgr2=0x5603e42a6250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:34.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.129+0000 7fc693c24740 1 -- 172.21.15.161:0/3197918827 shutdown_connections 2024-03-28T23:36:34.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.129+0000 7fc693c24740 1 -- 172.21.15.161:0/3197918827 wait complete. 2024-03-28T23:36:34.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.129+0000 7fc693c24740 20 client.9303 trim_cache size 0 max 16384 2024-03-28T23:36:34.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.164+0000 7fc693c24740 1 Processor -- start 2024-03-28T23:36:34.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.164+0000 7fc693c24740 1 -- start start 2024-03-28T23:36:34.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.164+0000 7fc693c24740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a6db0 0x5603e42fd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:34.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.164+0000 7fc693c24740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5603e42a7190 0x5603e42a7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:34.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.164+0000 7fc693c24740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5603e42f66b0 0x5603e42c7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:34.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.164+0000 7fc693c24740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc68c1fb1e0 con 0x5603e42a6db0 2024-03-28T23:36:34.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.164+0000 7fc693c24740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc68c14c540 con 0x5603e42f66b0 2024-03-28T23:36:34.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.164+0000 7fc693c24740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc68c1f92b0 con 0x5603e42a7190 2024-03-28T23:36:34.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.164+0000 7fc68ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a6db0 0x5603e42fd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:34.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.164+0000 7fc68ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a6db0 0x5603e42fd300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43834/0 (socket says 172.21.15.161:43834) 2024-03-28T23:36:34.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.164+0000 7fc68ad9d640 1 -- 172.21.15.161:0/777564662 learned_addr learned my addr 172.21.15.161:0/777564662 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:34.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.165+0000 7fc68b59e640 1 --2- 172.21.15.161:0/777564662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5603e42f66b0 0x5603e42c7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:34.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.165+0000 7fc68a59c640 1 --2- 172.21.15.161:0/777564662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5603e42a7190 0x5603e42a7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:34.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.165+0000 7fc68ad9d640 1 -- 172.21.15.161:0/777564662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5603e42a7190 msgr2=0x5603e42a7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:34.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.165+0000 7fc68ad9d640 1 --2- 172.21.15.161:0/777564662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5603e42a7190 0x5603e42a7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.165+0000 7fc68ad9d640 1 -- 172.21.15.161:0/777564662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5603e42f66b0 msgr2=0x5603e42c7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:34.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.165+0000 7fc68ad9d640 1 --2- 172.21.15.161:0/777564662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5603e42f66b0 0x5603e42c7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.165+0000 7fc68ad9d640 1 -- 172.21.15.161:0/777564662 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603e42c54f0 con 0x5603e42a6db0 2024-03-28T23:36:34.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.165+0000 7fc68ad9d640 1 --2- 172.21.15.161:0/777564662 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a6db0 0x5603e42fd300 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fc67c054190 tx=0x7fc67c014070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=949306372eb0967d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:34.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.165+0000 7fc66bfff640 1 -- 172.21.15.161:0/777564662 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67c056bf0 con 0x5603e42a6db0 2024-03-28T23:36:34.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.165+0000 7fc66bfff640 1 -- 172.21.15.161:0/777564662 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc67c076b50 con 0x5603e42a6db0 2024-03-28T23:36:34.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.165+0000 7fc66bfff640 1 -- 172.21.15.161:0/777564662 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67c057180 con 0x5603e42a6db0 2024-03-28T23:36:34.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.166+0000 7fc693c24740 1 -- 172.21.15.161:0/777564662 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a6db0 msgr2=0x5603e42fd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:34.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.166+0000 7fc693c24740 1 --2- 172.21.15.161:0/777564662 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a6db0 0x5603e42fd300 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fc67c054190 tx=0x7fc67c014070 comp rx=0 tx=0).stop 2024-03-28T23:36:34.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.166+0000 7fc693c24740 1 -- 172.21.15.161:0/777564662 shutdown_connections 2024-03-28T23:36:34.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.166+0000 7fc693c24740 1 --2- 172.21.15.161:0/777564662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5603e42f66b0 0x5603e42c7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.166+0000 7fc693c24740 1 --2- 172.21.15.161:0/777564662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5603e42a7190 0x5603e42a7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.166+0000 7fc693c24740 1 --2- 172.21.15.161:0/777564662 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a6db0 0x5603e42fd300 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.166+0000 7fc693c24740 1 -- 172.21.15.161:0/777564662 >> 172.21.15.161:0/777564662 conn(0x5603e42a5310 msgr2=0x5603e429c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:34.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.166+0000 7fc693c24740 1 -- 172.21.15.161:0/777564662 shutdown_connections 2024-03-28T23:36:34.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.166+0000 7fc693c24740 1 -- 172.21.15.161:0/777564662 wait complete. 2024-03-28T23:36:34.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.166+0000 7fc693c24740 1 -- start start 2024-03-28T23:36:34.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.166+0000 7fc693c24740 1 Processor -- start 2024-03-28T23:36:34.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.167+0000 7fc693c24740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a6db0 0x5603e42e68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:34.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.167+0000 7fc693c24740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5603e42a7190 0x5603e42a1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:34.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.167+0000 7fc693c24740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5603e42f66b0 0x5603e435a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:34.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.167+0000 7fc693c24740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5603e42c5000 con 0x5603e42a6db0 2024-03-28T23:36:34.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.167+0000 7fc693c24740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5603e42cb8e0 con 0x5603e42a7190 2024-03-28T23:36:34.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.167+0000 7fc693c24740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x5603e42c4550 con 0x5603e42f66b0 2024-03-28T23:36:34.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.167+0000 7fc693c24740 10 client.0 _ll_register_callbacks cb 0x7fc67c007f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:36:34.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.167+0000 7fc68ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a6db0 0x5603e42e68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:34.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.167+0000 7fc68ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a6db0 0x5603e42e68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43842/0 (socket says 172.21.15.161:43842) 2024-03-28T23:36:34.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.167+0000 7fc68ad9d640 1 -- 172.21.15.161:0/3535452937 learned_addr learned my addr 172.21.15.161:0/3535452937 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:34.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.167+0000 7fc68a59c640 1 --2- 172.21.15.161:0/3535452937 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5603e42a7190 0x5603e42a1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:34.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.167+0000 7fc68b59e640 1 --2- 172.21.15.161:0/3535452937 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5603e42f66b0 0x5603e435a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:34.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.167+0000 7fc68ad9d640 1 -- 172.21.15.161:0/3535452937 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5603e42f66b0 msgr2=0x5603e435a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:34.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.167+0000 7fc68ad9d640 1 --2- 172.21.15.161:0/3535452937 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5603e42f66b0 0x5603e435a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.167+0000 7fc68ad9d640 1 -- 172.21.15.161:0/3535452937 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5603e42a7190 msgr2=0x5603e42a1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:34.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.167+0000 7fc68ad9d640 1 --2- 172.21.15.161:0/3535452937 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5603e42a7190 0x5603e42a1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.167+0000 7fc68ad9d640 1 -- 172.21.15.161:0/3535452937 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603e42a8120 con 0x5603e42a6db0 2024-03-28T23:36:34.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.167+0000 7fc68ad9d640 1 --2- 172.21.15.161:0/3535452937 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a6db0 0x5603e42e68b0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fc67c05e230 tx=0x7fc67c05bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=debbe455882b77d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:34.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.167+0000 7fc669ffb640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:36:34.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.168+0000 7fc669ffb640 1 -- 172.21.15.161:0/3535452937 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67c057d30 con 0x5603e42a6db0 2024-03-28T23:36:34.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.168+0000 7fc669ffb640 1 -- 172.21.15.161:0/3535452937 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(159..159 src has 1..159) ==== 7215+0+0 (secure 0 0 0) 0x7fc67c07cde0 con 0x5603e42a6db0 2024-03-28T23:36:34.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.168+0000 7fc693c24740 1 -- 172.21.15.161:0/3535452937 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc67c076b50 con 0x5603e42a6db0 2024-03-28T23:36:34.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.168+0000 7fc669ffb640 1 -- 172.21.15.161:0/3535452937 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) ==== 24+0+0 (secure 0 0 0) 0x7fc67c056bf0 con 0x5603e42a6db0 2024-03-28T23:36:34.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.168+0000 7fc693c24740 10 client.9309 fetch_fsmap learned FSMap version 154 2024-03-28T23:36:34.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.168+0000 7fc693c24740 1 -- 172.21.15.161:0/3535452937 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) -- 0x5603e42a8120 con 0x5603e42a6db0 2024-03-28T23:36:34.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.168+0000 7fc669ffb640 1 -- 172.21.15.161:0/3535452937 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc67c076b50 con 0x5603e42a6db0 2024-03-28T23:36:34.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.168+0000 7fc669ffb640 1 -- 172.21.15.161:0/3535452937 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 154) ==== 4139+0+0 (secure 0 0 0) 0x7fc67c057180 con 0x5603e42a6db0 2024-03-28T23:36:34.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.168+0000 7fc693c24740 10 client.9309 fetch_fsmap finished waiting for FSMap version 154 2024-03-28T23:36:34.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.168+0000 7fc693c24740 10 client.9309 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:36:34.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.168+0000 7fc693c24740 20 client.9309 populate_metadata read hostname 'smithi161' 2024-03-28T23:36:34.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.168+0000 7fc693c24740 1 --2- 172.21.15.161:0/3535452937 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x5603e435a710 0x5603e4305d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:34.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.168+0000 7fc693c24740 4 client.9309 mds_command: new command op to 7283 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:36:34.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.168+0000 7fc693c24740 1 -- 172.21.15.161:0/3535452937 --> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] -- command(tid 0: {"prefix": "status"}) -- 0x5603e4212380 con 0x5603e435a710 2024-03-28T23:36:34.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.168+0000 7fc68a59c640 1 --2- 172.21.15.161:0/3535452937 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x5603e435a710 0x5603e4305d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:34.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.169+0000 7fc669ffb640 10 client.9309 ms_handle_connect on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:34.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.169+0000 7fc68a59c640 1 --2- 172.21.15.161:0/3535452937 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x5603e435a710 0x5603e4305d10 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5f11f20faa3aa5b server_cookie=bde7943b65ce1baf in_seq=0 out_seq=0 2024-03-28T23:36:34.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.169+0000 7fc669ffb640 1 -- 172.21.15.161:0/3535452937 <== mds.0 v2:172.21.15.170:6832/2996092857 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x5603e4212380 con 0x5603e435a710 2024-03-28T23:36:34.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.169+0000 7fc669ffb640 10 client.9309 handle_command_reply: tid=0 2024-03-28T23:36:34.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.169+0000 7fc693c24740 1 client.9309 shutdown 2024-03-28T23:36:34.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.169+0000 7fc693c24740 1 -- 172.21.15.161:0/3535452937 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a6db0 msgr2=0x5603e42e68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:34.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.169+0000 7fc693c24740 1 --2- 172.21.15.161:0/3535452937 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a6db0 0x5603e42e68b0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fc67c05e230 tx=0x7fc67c05bf10 comp rx=0 tx=0).stop 2024-03-28T23:36:34.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.169+0000 7fc693c24740 1 -- 172.21.15.161:0/3535452937 shutdown_connections 2024-03-28T23:36:34.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.169+0000 7fc693c24740 1 --2- 172.21.15.161:0/3535452937 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x5603e435a710 0x5603e4305d10 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.169+0000 7fc693c24740 1 --2- 172.21.15.161:0/3535452937 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5603e42f66b0 0x5603e435a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.169+0000 7fc693c24740 1 --2- 172.21.15.161:0/3535452937 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5603e42a7190 0x5603e42a1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.169+0000 7fc669ffb640 0 client.9309 ms_handle_reset on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:34.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.169+0000 7fc693c24740 1 --2- 172.21.15.161:0/3535452937 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5603e42a6db0 0x5603e42e68b0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.169+0000 7fc693c24740 1 -- 172.21.15.161:0/3535452937 >> 172.21.15.161:0/3535452937 conn(0x5603e42a5310 msgr2=0x5603e42a2610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:34.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.169+0000 7fc693c24740 1 -- 172.21.15.161:0/3535452937 shutdown_connections 2024-03-28T23:36:34.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.169+0000 7fc693c24740 1 -- 172.21.15.161:0/3535452937 wait complete. 2024-03-28T23:36:34.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.169+0000 7fc693c24740 20 client.9309 trim_cache size 0 max 16384 2024-03-28T23:36:34.179 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:36:34.179 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:36:34.179 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:36:34.179 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 7283, 2024-03-28T23:36:34.180 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:36:34.180 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:36:34.180 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:36:34.180 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 16.736575239, 2024-03-28T23:36:34.180 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 154, 2024-03-28T23:36:34.180 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 159, 2024-03-28T23:36:34.180 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 159, 2024-03-28T23:36:34.180 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 19.726398883000002 2024-03-28T23:36:34.180 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:36:34.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.169+0000 7fc691040640 1 -- 172.21.15.161:0/2652536916 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc67806b4d0 msgr2=0x7fc67806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:34.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.169+0000 7fc691040640 1 --2- 172.21.15.161:0/2652536916 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc67806b4d0 0x7fc67806d990 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fc68c1514d0 tx=0x7fc680007be0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.169+0000 7fc691040640 1 -- 172.21.15.161:0/2652536916 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc68c0c5410 msgr2=0x7fc68c157f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:34.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.169+0000 7fc691040640 1 --2- 172.21.15.161:0/2652536916 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc68c0c5410 0x7fc68c157f60 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fc67c00c430 tx=0x7fc67c00c8f0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.170+0000 7fc691040640 1 -- 172.21.15.161:0/2652536916 shutdown_connections 2024-03-28T23:36:34.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.170+0000 7fc691040640 1 --2- 172.21.15.161:0/2652536916 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc67806b4d0 0x7fc67806d990 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.170+0000 7fc691040640 1 --2- 172.21.15.161:0/2652536916 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc68c152890 0x7fc68c0c4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.170+0000 7fc691040640 1 --2- 172.21.15.161:0/2652536916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc68c0c5850 0x7fc68c0c3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.170+0000 7fc691040640 1 --2- 172.21.15.161:0/2652536916 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc68c0c5410 0x7fc68c157f60 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:34.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.170+0000 7fc691040640 1 -- 172.21.15.161:0/2652536916 >> 172.21.15.161:0/2652536916 conn(0x7fc68c0ac620 msgr2=0x7fc68c150670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:34.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.170+0000 7fc691040640 1 -- 172.21.15.161:0/2652536916 shutdown_connections 2024-03-28T23:36:34.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:34.170+0000 7fc691040640 1 -- 172.21.15.161:0/2652536916 wait complete. 2024-03-28T23:36:34.184 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 11, "epoch": 154, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 11, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:35:58.406403+0000", "damaged": [], "data_pools": [ 24 ], "enabled": true, "epoch": 154, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7283": { "addr": "172.21.15.170:6834/2996092857", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 2996092857, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 2996092857, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7283, "incarnation": 151, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 159, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 23, "modified": "2024-03-28T23:36:20.440473+0000", "qdb_cluster": [ 7283 ], "qdb_leader": 7283, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7283 } } }, { "id": 12, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:36:07.448976+0000", "damaged": [], "data_pools": [ 26 ], "enabled": true, "epoch": 149, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs2", "in": [ 0 ], "info": { "gid_7061": { "addr": "172.21.15.161:6836/4026332703", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 4026332703, "type": "v2" }, { "addr": "172.21.15.161:6836", "nonce": 4026332703, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7061, "incarnation": 145, "join_fscid": 12, "name": "c", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 157, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 25, "modified": "2024-03-28T23:36:16.420228+0000", "qdb_cluster": [ 7061 ], "qdb_leader": 7061, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7061 } } } ], "standbys": [ { "addr": "172.21.15.170:6835/557522063", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 557522063, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 557522063, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 137, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7070, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6837/789976100", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6835", "nonce": 789976100, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 789976100, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 152, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9216, "incarnation": 0, "join_fscid": 12, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:36:34.185 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': 12, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 12, 'name': 'c'}, {'join_fscid': -1, 'name': 'd', 'state': 'up:active'}] 2024-03-28T23:36:34.185 INFO:tasks.cephfs.test_failover:current = [{'gid': 9216, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6837/789976100', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 789976100}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 789976100}]}, 'join_fscid': 12, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 152}, {'gid': 7283, 'name': 'b', 'rank': 0, 'incarnation': 151, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.170:6834/2996092857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 2996092857}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 2996092857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 7061, 'name': 'c', 'rank': 0, 'incarnation': 145, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.161:6836/4026332703', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 4026332703}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 4026332703}]}, 'join_fscid': 12, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 7070, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/557522063', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 557522063}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 557522063}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 137}] 2024-03-28T23:36:34.186 DEBUG:tasks.cephfs.test_failover:'up:active' != 'up:standby' - up:active + up:standby 2024-03-28T23:36:34.186 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-28T23:36:39.187 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:36:39.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.288+0000 7faf9cd2c640 1 Processor -- start 2024-03-28T23:36:39.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.288+0000 7faf9cd2c640 1 -- start start 2024-03-28T23:36:39.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.288+0000 7faf9cd2c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7faf9814eca0 0x7faf980c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:39.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.288+0000 7faf9cd2c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf980c1dc0 0x7faf9806ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:39.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.288+0000 7faf9cd2c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7faf9806cf60 0x7faf980a98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:39.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.288+0000 7faf9cd2c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7faf98152460 con 0x7faf9814eca0 2024-03-28T23:36:39.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.288+0000 7faf9cd2c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7faf98151850 con 0x7faf9806cf60 2024-03-28T23:36:39.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.288+0000 7faf9cd2c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7faf98151b30 con 0x7faf980c1dc0 2024-03-28T23:36:39.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.289+0000 7faf9659c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7faf9814eca0 0x7faf980c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:39.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.289+0000 7faf9659c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7faf9814eca0 0x7faf980c1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43852/0 (socket says 172.21.15.161:43852) 2024-03-28T23:36:39.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.289+0000 7faf9659c640 1 -- 172.21.15.161:0/600580819 learned_addr learned my addr 172.21.15.161:0/600580819 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:39.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.289+0000 7faf96d9d640 1 --2- 172.21.15.161:0/600580819 >> v2:172.21.15.170:3300/0 conn(0x7faf9806cf60 0x7faf980a98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:39.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.289+0000 7faf95d9b640 1 --2- 172.21.15.161:0/600580819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf980c1dc0 0x7faf9806ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:39.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.289+0000 7faf9659c640 1 -- 172.21.15.161:0/600580819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf980c1dc0 msgr2=0x7faf9806ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:39.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.289+0000 7faf9659c640 1 --2- 172.21.15.161:0/600580819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf980c1dc0 0x7faf9806ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:39.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.289+0000 7faf9659c640 1 -- 172.21.15.161:0/600580819 >> v2:172.21.15.170:3300/0 conn(0x7faf9806cf60 msgr2=0x7faf980a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:39.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.289+0000 7faf9659c640 1 --2- 172.21.15.161:0/600580819 >> v2:172.21.15.170:3300/0 conn(0x7faf9806cf60 0x7faf980a98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:39.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.289+0000 7faf9659c640 1 -- 172.21.15.161:0/600580819 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf98151e10 con 0x7faf9814eca0 2024-03-28T23:36:39.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.290+0000 7faf9659c640 1 --2- 172.21.15.161:0/600580819 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf9814eca0 0x7faf980c1880 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7faf980c0e40 tx=0x7faf9002de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27745f8b7488ac98 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:39.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.290+0000 7faf9559a640 1 -- 172.21.15.161:0/600580819 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf90034070 con 0x7faf9814eca0 2024-03-28T23:36:39.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.290+0000 7faf9559a640 1 -- 172.21.15.161:0/600580819 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf9002ebc0 con 0x7faf9814eca0 2024-03-28T23:36:39.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.290+0000 7faf9559a640 1 -- 172.21.15.161:0/600580819 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf90038400 con 0x7faf9814eca0 2024-03-28T23:36:39.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.290+0000 7faf9cd2c640 1 -- 172.21.15.161:0/600580819 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf9814eca0 msgr2=0x7faf980c1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:39.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.290+0000 7faf9cd2c640 1 --2- 172.21.15.161:0/600580819 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf9814eca0 0x7faf980c1880 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7faf980c0e40 tx=0x7faf9002de10 comp rx=0 tx=0).stop 2024-03-28T23:36:39.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.290+0000 7faf9cd2c640 1 -- 172.21.15.161:0/600580819 shutdown_connections 2024-03-28T23:36:39.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.290+0000 7faf9cd2c640 1 --2- 172.21.15.161:0/600580819 >> v2:172.21.15.170:3300/0 conn(0x7faf9806cf60 0x7faf980a98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:39.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.290+0000 7faf9cd2c640 1 --2- 172.21.15.161:0/600580819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf980c1dc0 0x7faf9806ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:39.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.290+0000 7faf9cd2c640 1 --2- 172.21.15.161:0/600580819 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf9814eca0 0x7faf980c1880 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:39.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.291+0000 7faf9cd2c640 1 -- 172.21.15.161:0/600580819 >> 172.21.15.161:0/600580819 conn(0x7faf980c6670 msgr2=0x7faf980c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:39.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.291+0000 7faf9cd2c640 1 -- 172.21.15.161:0/600580819 shutdown_connections 2024-03-28T23:36:39.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.291+0000 7faf9cd2c640 1 -- 172.21.15.161:0/600580819 wait complete. 2024-03-28T23:36:39.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.292+0000 7faf9cd2c640 1 Processor -- start 2024-03-28T23:36:39.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.292+0000 7faf9cd2c640 1 -- start start 2024-03-28T23:36:39.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.292+0000 7faf9cd2c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf9806cf60 0x7faf981e6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:39.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.292+0000 7faf9cd2c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf980c1dc0 0x7faf9814b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:39.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.292+0000 7faf9cd2c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf9814eca0 0x7faf981e9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:39.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.292+0000 7faf9cd2c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7faf981514b0 con 0x7faf9806cf60 2024-03-28T23:36:39.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.292+0000 7faf9cd2c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7faf9814f080 con 0x7faf9814eca0 2024-03-28T23:36:39.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.292+0000 7faf9cd2c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7faf98151630 con 0x7faf980c1dc0 2024-03-28T23:36:39.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.293+0000 7faf9659c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf9806cf60 0x7faf981e6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:39.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.293+0000 7faf9659c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf9806cf60 0x7faf981e6130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43862/0 (socket says 172.21.15.161:43862) 2024-03-28T23:36:39.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.293+0000 7faf9659c640 1 -- 172.21.15.161:0/3722536837 learned_addr learned my addr 172.21.15.161:0/3722536837 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:39.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.293+0000 7faf96d9d640 1 --2- 172.21.15.161:0/3722536837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf9814eca0 0x7faf981e9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:39.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.293+0000 7faf95d9b640 1 --2- 172.21.15.161:0/3722536837 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf980c1dc0 0x7faf9814b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:39.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.293+0000 7faf9659c640 1 -- 172.21.15.161:0/3722536837 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf980c1dc0 msgr2=0x7faf9814b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:39.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.293+0000 7faf9659c640 1 --2- 172.21.15.161:0/3722536837 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf980c1dc0 0x7faf9814b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:39.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.293+0000 7faf9659c640 1 -- 172.21.15.161:0/3722536837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf9814eca0 msgr2=0x7faf981e9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:39.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.293+0000 7faf9659c640 1 --2- 172.21.15.161:0/3722536837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf9814eca0 0x7faf981e9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:39.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.293+0000 7faf9659c640 1 -- 172.21.15.161:0/3722536837 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf981e6670 con 0x7faf9806cf60 2024-03-28T23:36:39.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.293+0000 7faf9659c640 1 --2- 172.21.15.161:0/3722536837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf9806cf60 0x7faf981e6130 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7faf90000c00 tx=0x7faf900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:39.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.293+0000 7faf86ffd640 1 -- 172.21.15.161:0/3722536837 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf90047030 con 0x7faf9806cf60 2024-03-28T23:36:39.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.293+0000 7faf86ffd640 1 -- 172.21.15.161:0/3722536837 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf9002e9c0 con 0x7faf9806cf60 2024-03-28T23:36:39.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.293+0000 7faf9cd2c640 1 -- 172.21.15.161:0/3722536837 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf98071240 con 0x7faf9806cf60 2024-03-28T23:36:39.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.293+0000 7faf86ffd640 1 -- 172.21.15.161:0/3722536837 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf90006dd0 con 0x7faf9806cf60 2024-03-28T23:36:39.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.293+0000 7faf9cd2c640 1 -- 172.21.15.161:0/3722536837 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf981e97d0 con 0x7faf9806cf60 2024-03-28T23:36:39.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.294+0000 7faf9cd2c640 1 -- 172.21.15.161:0/3722536837 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf9804f4b0 con 0x7faf9806cf60 2024-03-28T23:36:39.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.295+0000 7faf86ffd640 1 -- 172.21.15.161:0/3722536837 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7faf9004a070 con 0x7faf9806cf60 2024-03-28T23:36:39.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.295+0000 7faf86ffd640 1 --2- 172.21.15.161:0/3722536837 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faf7806b4d0 0x7faf7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:39.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.295+0000 7faf86ffd640 1 -- 172.21.15.161:0/3722536837 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7faf900aed20 con 0x7faf9806cf60 2024-03-28T23:36:39.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.295+0000 7faf95d9b640 1 --2- 172.21.15.161:0/3722536837 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faf7806b4d0 0x7faf7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:39.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.296+0000 7faf95d9b640 1 --2- 172.21.15.161:0/3722536837 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faf7806b4d0 0x7faf7806d990 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7faf980c5f00 tx=0x7faf8c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:39.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.304+0000 7faf86ffd640 1 -- 172.21.15.161:0/3722536837 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7faf90051070 con 0x7faf9806cf60 2024-03-28T23:36:39.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.555+0000 7faf9cd2c640 1 -- 172.21.15.161:0/3722536837 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf980aa970 con 0x7faf9806cf60 2024-03-28T23:36:39.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.555+0000 7faf86ffd640 1 -- 172.21.15.161:0/3722536837 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+6520 (secure 0 0 0) 0x7faf9007cf80 con 0x7faf9806cf60 2024-03-28T23:36:39.557 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:39.557 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":154,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/557522063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":557522063},{"type":"v1","addr":"172.21.15.170:6835","nonce":557522063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":9216,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/789976100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":789976100},{"type":"v1","addr":"172.21.15.161:6837","nonce":789976100}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:58.406403+0000","modified":"2024-03-28T23:36:20.440473+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":159,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7283":{"gid":7283,"name":"b","rank":0,"incarnation":151,"state":"up:active","state_seq":4,"addr":"172.21.15.170:6834/2996092857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":2996092857},{"type":"v1","addr":"172.21.15.170:6834","nonce":2996092857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7283,"qdb_cluster":[7283]},"id":11},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:36:07.448976+0000","modified":"2024-03-28T23:36:16.420228+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7061":{"gid":7061,"name":"c","rank":0,"incarnation":145,"state":"up:active","state_seq":9,"addr":"172.21.15.161:6836/4026332703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4026332703},{"type":"v1","addr":"172.21.15.161:6836","nonce":4026332703}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7061,"qdb_cluster":[7061]},"id":12}]} 2024-03-28T23:36:39.557 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 154 2024-03-28T23:36:39.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.558+0000 7faf9cd2c640 1 -- 172.21.15.161:0/3722536837 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faf7806b4d0 msgr2=0x7faf7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:39.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.558+0000 7faf9cd2c640 1 --2- 172.21.15.161:0/3722536837 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faf7806b4d0 0x7faf7806d990 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7faf980c5f00 tx=0x7faf8c00b040 comp rx=0 tx=0).stop 2024-03-28T23:36:39.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.558+0000 7faf9cd2c640 1 -- 172.21.15.161:0/3722536837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf9806cf60 msgr2=0x7faf981e6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:39.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.558+0000 7faf9cd2c640 1 --2- 172.21.15.161:0/3722536837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf9806cf60 0x7faf981e6130 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7faf90000c00 tx=0x7faf900046c0 comp rx=0 tx=0).stop 2024-03-28T23:36:39.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.558+0000 7faf9cd2c640 1 -- 172.21.15.161:0/3722536837 shutdown_connections 2024-03-28T23:36:39.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.558+0000 7faf9cd2c640 1 --2- 172.21.15.161:0/3722536837 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7faf7806b4d0 0x7faf7806d990 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:39.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.558+0000 7faf9cd2c640 1 --2- 172.21.15.161:0/3722536837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf9814eca0 0x7faf981e9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:39.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.558+0000 7faf9cd2c640 1 --2- 172.21.15.161:0/3722536837 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf980c1dc0 0x7faf9814b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:39.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.558+0000 7faf9cd2c640 1 --2- 172.21.15.161:0/3722536837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf9806cf60 0x7faf981e6130 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:39.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.558+0000 7faf9cd2c640 1 -- 172.21.15.161:0/3722536837 >> 172.21.15.161:0/3722536837 conn(0x7faf980c6670 msgr2=0x7faf9814f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:39.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.558+0000 7faf9cd2c640 1 -- 172.21.15.161:0/3722536837 shutdown_connections 2024-03-28T23:36:39.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:39.558+0000 7faf9cd2c640 1 -- 172.21.15.161:0/3722536837 wait complete. 2024-03-28T23:36:39.974 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 154, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:35:58.406403+0000', 'modified': '2024-03-28T23:36:20.440473+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 159, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7283}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7283': {'gid': 7283, 'name': 'b', 'rank': 0, 'incarnation': 151, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.170:6834/2996092857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 2996092857}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 2996092857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7283, 'qdb_cluster': [7283]} 2024-03-28T23:36:39.974 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:36:39.974 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T23:36:40.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.077+0000 7efeca78a640 1 Processor -- start 2024-03-28T23:36:40.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.077+0000 7efeca78a640 1 -- start start 2024-03-28T23:36:40.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.077+0000 7efeca78a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efec4159380 0x7efec40bfed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:40.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.077+0000 7efeca78a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efec40c6840 con 0x7efec40c0410 2024-03-28T23:36:40.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.077+0000 7efeca78a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efec40c5da0 con 0x7efec40ab250 2024-03-28T23:36:40.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.077+0000 7efeca78a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7efec40c1170 con 0x7efec4159380 2024-03-28T23:36:40.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.078+0000 7efec3fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efec4159380 0x7efec40bfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:40.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.078+0000 7efec8d27640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7efec40c0410 0x7efec415fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44116/0 (socket says 172.21.15.161:44116) 2024-03-28T23:36:40.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.078+0000 7efec8d27640 1 -- 172.21.15.161:0/439529817 learned_addr learned my addr 172.21.15.161:0/439529817 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:40.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.078+0000 7efec2ffd640 1 -- 172.21.15.161:0/439529817 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3307625011 0 0) 0x7efec40c5da0 con 0x7efec40ab250 2024-03-28T23:36:40.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.078+0000 7efec2ffd640 1 -- 172.21.15.161:0/439529817 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efeac003520 con 0x7efec40ab250 2024-03-28T23:36:40.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.078+0000 7efec3fff640 1 -- 172.21.15.161:0/439529817 >> v1:172.21.15.170:6789/0 conn(0x7efec40ab250 legacy=0x7efec40c14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:40.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.078+0000 7efec3fff640 1 -- 172.21.15.161:0/439529817 >> v1:172.21.15.161:6789/0 conn(0x7efec40c0410 legacy=0x7efec415fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:40.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.078+0000 7efec3fff640 1 -- 172.21.15.161:0/439529817 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec40c6080 con 0x7efec4159380 2024-03-28T23:36:40.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.079+0000 7efec3fff640 1 --2- 172.21.15.161:0/439529817 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efec4159380 0x7efec40bfed0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7efeb40097a0 tx=0x7efeb402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9bd7e91288edcde server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:40.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.079+0000 7efec2ffd640 1 -- 172.21.15.161:0/439529817 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb40315a0 con 0x7efec4159380 2024-03-28T23:36:40.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.079+0000 7efec2ffd640 1 -- 172.21.15.161:0/439529817 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeb4037070 con 0x7efec4159380 2024-03-28T23:36:40.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.079+0000 7efec2ffd640 1 -- 172.21.15.161:0/439529817 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb4031960 con 0x7efec4159380 2024-03-28T23:36:40.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.079+0000 7efeca78a640 1 -- 172.21.15.161:0/439529817 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efec4159380 msgr2=0x7efec40bfed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:40.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.079+0000 7efeca78a640 1 --2- 172.21.15.161:0/439529817 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efec4159380 0x7efec40bfed0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7efeb40097a0 tx=0x7efeb402b840 comp rx=0 tx=0).stop 2024-03-28T23:36:40.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.079+0000 7efeca78a640 1 -- 172.21.15.161:0/439529817 shutdown_connections 2024-03-28T23:36:40.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.079+0000 7efeca78a640 1 --2- 172.21.15.161:0/439529817 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efec4159380 0x7efec40bfed0 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.079+0000 7efeca78a640 1 -- 172.21.15.161:0/439529817 >> 172.21.15.161:0/439529817 conn(0x7efec40c6b40 msgr2=0x7efec40c6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:40.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.079+0000 7efeca78a640 1 -- 172.21.15.161:0/439529817 shutdown_connections 2024-03-28T23:36:40.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.079+0000 7efeca78a640 1 -- 172.21.15.161:0/439529817 wait complete. 2024-03-28T23:36:40.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.080+0000 7efeca78a640 1 Processor -- start 2024-03-28T23:36:40.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.081+0000 7efeca78a640 1 -- start start 2024-03-28T23:36:40.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.081+0000 7efeca78a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efec40ab250 0x7efec4151990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:40.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.081+0000 7efeca78a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efec40c0410 0x7efec414bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:40.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.081+0000 7efeca78a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efec4159380 0x7efec414c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:40.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.081+0000 7efeca78a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efec40c64d0 con 0x7efec40ab250 2024-03-28T23:36:40.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.081+0000 7efeca78a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7efec40c0a00 con 0x7efec40c0410 2024-03-28T23:36:40.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.081+0000 7efeca78a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7efec4159760 con 0x7efec4159380 2024-03-28T23:36:40.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.081+0000 7efec3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efec40ab250 0x7efec4151990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:40.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.081+0000 7efec3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efec40ab250 0x7efec4151990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43884/0 (socket says 172.21.15.161:43884) 2024-03-28T23:36:40.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.081+0000 7efec3fff640 1 -- 172.21.15.161:0/500371575 learned_addr learned my addr 172.21.15.161:0/500371575 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:40.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.081+0000 7efec37fe640 1 --2- 172.21.15.161:0/500371575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efec40c0410 0x7efec414bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:40.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.081+0000 7efec8d27640 1 --2- 172.21.15.161:0/500371575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efec4159380 0x7efec414c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:40.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.082+0000 7efec3fff640 1 -- 172.21.15.161:0/500371575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efec4159380 msgr2=0x7efec414c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:40.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.082+0000 7efec3fff640 1 --2- 172.21.15.161:0/500371575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efec4159380 0x7efec414c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.082+0000 7efec3fff640 1 -- 172.21.15.161:0/500371575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efec40c0410 msgr2=0x7efec414bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:40.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.082+0000 7efec3fff640 1 --2- 172.21.15.161:0/500371575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efec40c0410 0x7efec414bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.082+0000 7efec3fff640 1 -- 172.21.15.161:0/500371575 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec40a7160 con 0x7efec40ab250 2024-03-28T23:36:40.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.082+0000 7efec3fff640 1 --2- 172.21.15.161:0/500371575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efec40ab250 0x7efec4151990 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7efeb40098c0 tx=0x7efeb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:40.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.082+0000 7efec0ff9640 1 -- 172.21.15.161:0/500371575 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb4032630 con 0x7efec40ab250 2024-03-28T23:36:40.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.082+0000 7efec0ff9640 1 -- 172.21.15.161:0/500371575 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeb40327f0 con 0x7efec40ab250 2024-03-28T23:36:40.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.082+0000 7efeca78a640 1 -- 172.21.15.161:0/500371575 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efec40a7440 con 0x7efec40ab250 2024-03-28T23:36:40.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.082+0000 7efec0ff9640 1 -- 172.21.15.161:0/500371575 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb4032be0 con 0x7efec40ab250 2024-03-28T23:36:40.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.082+0000 7efeca78a640 1 -- 172.21.15.161:0/500371575 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efec41e02d0 con 0x7efec40ab250 2024-03-28T23:36:40.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.084+0000 7efec0ff9640 1 -- 172.21.15.161:0/500371575 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efeb404e430 con 0x7efec40ab250 2024-03-28T23:36:40.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.085+0000 7efec0ff9640 1 --2- 172.21.15.161:0/500371575 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efeb006b4d0 0x7efeb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:40.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.085+0000 7efec0ff9640 1 -- 172.21.15.161:0/500371575 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7efeb402f030 con 0x7efec40ab250 2024-03-28T23:36:40.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.085+0000 7efec37fe640 1 --2- 172.21.15.161:0/500371575 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efeb006b4d0 0x7efeb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:40.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.085+0000 7efecd36e740 1 Processor -- start 2024-03-28T23:36:40.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.085+0000 7efecd36e740 1 -- start start 2024-03-28T23:36:40.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.085+0000 7efec37fe640 1 --2- 172.21.15.161:0/500371575 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efeb006b4d0 0x7efeb006d990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7efeb8007fd0 tx=0x7efeb8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:40.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.086+0000 7efecd36e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788adb0 0x561f27888890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:40.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.086+0000 7efecd36e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f2788b190 0x561f27888dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:40.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.086+0000 7efecd36e740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561f27889310 0x561f278854e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:40.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.086+0000 7efecd36e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561f277eeaf0 con 0x561f2788adb0 2024-03-28T23:36:40.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.086+0000 7efecd36e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x561f277979d0 con 0x561f2788b190 2024-03-28T23:36:40.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.086+0000 7efecd36e740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x561f277eab90 con 0x561f27889310 2024-03-28T23:36:40.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.086+0000 7efec3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788adb0 0x561f27888890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:40.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.086+0000 7efec3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788adb0 0x561f27888890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43900/0 (socket says 172.21.15.161:43900) 2024-03-28T23:36:40.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.086+0000 7efec3fff640 1 -- 172.21.15.161:0/3597165913 learned_addr learned my addr 172.21.15.161:0/3597165913 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:40.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.086+0000 7efec8d27640 1 --2- 172.21.15.161:0/3597165913 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561f27889310 0x561f278854e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:40.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.086+0000 7efec37fe640 1 --2- 172.21.15.161:0/3597165913 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f2788b190 0x561f27888dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:40.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.086+0000 7efec3fff640 1 -- 172.21.15.161:0/3597165913 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561f27889310 msgr2=0x561f278854e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:40.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.086+0000 7efec3fff640 1 --2- 172.21.15.161:0/3597165913 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561f27889310 0x561f278854e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.086+0000 7efec3fff640 1 -- 172.21.15.161:0/3597165913 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f2788b190 msgr2=0x561f27888dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:40.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.086+0000 7efec3fff640 1 --2- 172.21.15.161:0/3597165913 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f2788b190 0x561f27888dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.086+0000 7efec3fff640 1 -- 172.21.15.161:0/3597165913 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f275032d0 con 0x561f2788adb0 2024-03-28T23:36:40.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.086+0000 7efec3fff640 1 --2- 172.21.15.161:0/3597165913 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788adb0 0x561f27888890 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7efeb40aa410 tx=0x7efeb4006f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73d0a584bf04a67d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:40.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.086+0000 7efea9ffb640 1 -- 172.21.15.161:0/3597165913 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb4082dd0 con 0x561f2788adb0 2024-03-28T23:36:40.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.086+0000 7efea9ffb640 1 -- 172.21.15.161:0/3597165913 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeb4037050 con 0x561f2788adb0 2024-03-28T23:36:40.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.086+0000 7efea9ffb640 1 -- 172.21.15.161:0/3597165913 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb4078a70 con 0x561f2788adb0 2024-03-28T23:36:40.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.087+0000 7efecd36e740 1 -- 172.21.15.161:0/3597165913 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788adb0 msgr2=0x561f27888890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:40.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.087+0000 7efecd36e740 1 --2- 172.21.15.161:0/3597165913 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788adb0 0x561f27888890 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7efeb40aa410 tx=0x7efeb4006f40 comp rx=0 tx=0).stop 2024-03-28T23:36:40.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.087+0000 7efecd36e740 1 -- 172.21.15.161:0/3597165913 shutdown_connections 2024-03-28T23:36:40.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.087+0000 7efecd36e740 1 --2- 172.21.15.161:0/3597165913 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561f27889310 0x561f278854e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.087+0000 7efecd36e740 1 --2- 172.21.15.161:0/3597165913 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f2788b190 0x561f27888dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.087+0000 7efecd36e740 1 --2- 172.21.15.161:0/3597165913 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788adb0 0x561f27888890 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.087+0000 7efecd36e740 1 -- 172.21.15.161:0/3597165913 >> 172.21.15.161:0/3597165913 conn(0x561f27887630 msgr2=0x561f278ac220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:40.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.087+0000 7efecd36e740 1 -- 172.21.15.161:0/3597165913 shutdown_connections 2024-03-28T23:36:40.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.087+0000 7efecd36e740 1 -- 172.21.15.161:0/3597165913 wait complete. 2024-03-28T23:36:40.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.087+0000 7efecd36e740 1 -- start start 2024-03-28T23:36:40.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.088+0000 7efecd36e740 1 Processor -- start 2024-03-28T23:36:40.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.088+0000 7efecd36e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788adb0 0x561f278c7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:40.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.088+0000 7efecd36e740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561f2788b190 0x561f278a7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:40.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.088+0000 7efecd36e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f278d84b0 0x561f278a7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:40.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.088+0000 7efecd36e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561f277eeaf0 con 0x561f2788adb0 2024-03-28T23:36:40.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.088+0000 7efecd36e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x561f277aaf70 con 0x561f278d84b0 2024-03-28T23:36:40.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.088+0000 7efecd36e740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x561f278ad090 con 0x561f2788b190 2024-03-28T23:36:40.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.088+0000 7efecd36e740 10 client.0 _ll_register_callbacks cb 0x561f27502ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:36:40.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.088+0000 7efec3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788adb0 0x561f278c7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:40.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.089+0000 7efec3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788adb0 0x561f278c7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43906/0 (socket says 172.21.15.161:43906) 2024-03-28T23:36:40.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.089+0000 7efec3fff640 1 -- 172.21.15.161:0/1827861863 learned_addr learned my addr 172.21.15.161:0/1827861863 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:40.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.089+0000 7efec8d27640 1 --2- 172.21.15.161:0/1827861863 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f278d84b0 0x561f278a7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:40.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.089+0000 7efec37fe640 1 --2- 172.21.15.161:0/1827861863 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561f2788b190 0x561f278a7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:40.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.089+0000 7efec3fff640 1 -- 172.21.15.161:0/1827861863 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561f2788b190 msgr2=0x561f278a7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:40.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.089+0000 7efec3fff640 1 --2- 172.21.15.161:0/1827861863 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561f2788b190 0x561f278a7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.089+0000 7efec3fff640 1 -- 172.21.15.161:0/1827861863 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f278d84b0 msgr2=0x561f278a7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:40.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.089+0000 7efec3fff640 1 --2- 172.21.15.161:0/1827861863 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f278d84b0 0x561f278a7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.089+0000 7efec3fff640 1 -- 172.21.15.161:0/1827861863 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f275cd6b0 con 0x561f2788adb0 2024-03-28T23:36:40.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.089+0000 7efec3fff640 1 --2- 172.21.15.161:0/1827861863 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788adb0 0x561f278c7070 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7efeb40097a0 tx=0x7efeb402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f810452c0ac7b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:40.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.089+0000 7efea2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:36:40.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.089+0000 7efea2ffd640 1 -- 172.21.15.161:0/1827861863 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb40a7020 con 0x561f2788adb0 2024-03-28T23:36:40.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.089+0000 7efea2ffd640 1 -- 172.21.15.161:0/1827861863 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7efeb4078a70 con 0x561f2788adb0 2024-03-28T23:36:40.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.089+0000 7efecd36e740 1 -- 172.21.15.161:0/1827861863 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efeb4037050 con 0x561f2788adb0 2024-03-28T23:36:40.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.089+0000 7efea2ffd640 1 -- 172.21.15.161:0/1827861863 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) ==== 24+0+0 (secure 0 0 0) 0x7efeb4099050 con 0x561f2788adb0 2024-03-28T23:36:40.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.089+0000 7efecd36e740 10 client.9330 fetch_fsmap learned FSMap version 154 2024-03-28T23:36:40.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.089+0000 7efecd36e740 1 -- 172.21.15.161:0/1827861863 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) -- 0x561f278b0f30 con 0x561f2788adb0 2024-03-28T23:36:40.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.089+0000 7efea2ffd640 1 -- 172.21.15.161:0/1827861863 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeb4037050 con 0x561f2788adb0 2024-03-28T23:36:40.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.090+0000 7efea2ffd640 1 -- 172.21.15.161:0/1827861863 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 154) ==== 4139+0+0 (secure 0 0 0) 0x7efeb4031070 con 0x561f2788adb0 2024-03-28T23:36:40.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.090+0000 7efecd36e740 10 client.9330 fetch_fsmap finished waiting for FSMap version 154 2024-03-28T23:36:40.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.090+0000 7efecd36e740 10 client.9330 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:36:40.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.090+0000 7efecd36e740 20 client.9330 populate_metadata read hostname 'smithi161' 2024-03-28T23:36:40.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.090+0000 7efecd36e740 1 --2- 172.21.15.161:0/1827861863 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x561f278da730 0x561f278dab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:40.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.090+0000 7efecd36e740 4 client.9330 mds_command: new command op to 7283 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:36:40.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.090+0000 7efecd36e740 1 -- 172.21.15.161:0/1827861863 --> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f277fc030 con 0x561f278da730 2024-03-28T23:36:40.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.090+0000 7efec37fe640 1 --2- 172.21.15.161:0/1827861863 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x561f278da730 0x561f278dab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:40.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.090+0000 7efea2ffd640 10 client.9330 ms_handle_connect on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:40.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.090+0000 7efec37fe640 1 --2- 172.21.15.161:0/1827861863 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x561f278da730 0x561f278dab10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0ac18e6227fa6e5 server_cookie=5de6f66754db9756 in_seq=0 out_seq=0 2024-03-28T23:36:40.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.091+0000 7efea2ffd640 1 -- 172.21.15.161:0/1827861863 <== mds.0 v2:172.21.15.170:6832/2996092857 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561f277fc030 con 0x561f278da730 2024-03-28T23:36:40.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.091+0000 7efea2ffd640 10 client.9330 handle_command_reply: tid=0 2024-03-28T23:36:40.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.091+0000 7efecd36e740 1 client.9330 shutdown 2024-03-28T23:36:40.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.091+0000 7efecd36e740 1 -- 172.21.15.161:0/1827861863 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788adb0 msgr2=0x561f278c7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:40.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.091+0000 7efecd36e740 1 --2- 172.21.15.161:0/1827861863 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788adb0 0x561f278c7070 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7efeb40097a0 tx=0x7efeb402b840 comp rx=0 tx=0).stop 2024-03-28T23:36:40.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.091+0000 7efecd36e740 1 -- 172.21.15.161:0/1827861863 shutdown_connections 2024-03-28T23:36:40.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.091+0000 7efecd36e740 1 --2- 172.21.15.161:0/1827861863 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x561f278da730 0x561f278dab10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.091+0000 7efecd36e740 1 --2- 172.21.15.161:0/1827861863 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f278d84b0 0x561f278a7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.091+0000 7efea2ffd640 0 client.9330 ms_handle_reset on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:40.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.091+0000 7efecd36e740 1 --2- 172.21.15.161:0/1827861863 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561f2788b190 0x561f278a7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.091+0000 7efecd36e740 1 --2- 172.21.15.161:0/1827861863 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788adb0 0x561f278c7070 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.091+0000 7efecd36e740 1 -- 172.21.15.161:0/1827861863 >> 172.21.15.161:0/1827861863 conn(0x561f27889310 msgr2=0x561f27889740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:40.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.091+0000 7efecd36e740 1 -- 172.21.15.161:0/1827861863 shutdown_connections 2024-03-28T23:36:40.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.091+0000 7efecd36e740 1 -- 172.21.15.161:0/1827861863 wait complete. 2024-03-28T23:36:40.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.091+0000 7efecd36e740 20 client.9330 trim_cache size 0 max 16384 2024-03-28T23:36:40.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.109+0000 7efecd36e740 1 Processor -- start 2024-03-28T23:36:40.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.109+0000 7efecd36e740 1 -- start start 2024-03-28T23:36:40.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.110+0000 7efecd36e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f2788adb0 0x561f278abdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:40.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.110+0000 7efecd36e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788b190 0x561f27885200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:40.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.110+0000 7efecd36e740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561f278d84b0 0x561f27885740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:40.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.110+0000 7efecd36e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efec41fb4b0 con 0x561f2788b190 2024-03-28T23:36:40.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.110+0000 7efecd36e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7efec41f8f40 con 0x561f2788adb0 2024-03-28T23:36:40.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.110+0000 7efecd36e740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7efec41f9440 con 0x561f278d84b0 2024-03-28T23:36:40.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.110+0000 7efec37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788b190 0x561f27885200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:40.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.110+0000 7efec3fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f2788adb0 0x561f278abdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:40.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.110+0000 7efec8d27640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561f278d84b0 0x561f27885740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:40.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.110+0000 7efec37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788b190 0x561f27885200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43908/0 (socket says 172.21.15.161:43908) 2024-03-28T23:36:40.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.110+0000 7efec37fe640 1 -- 172.21.15.161:0/733696760 learned_addr learned my addr 172.21.15.161:0/733696760 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:40.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.110+0000 7efec37fe640 1 -- 172.21.15.161:0/733696760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561f278d84b0 msgr2=0x561f27885740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:40.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.110+0000 7efec37fe640 1 --2- 172.21.15.161:0/733696760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561f278d84b0 0x561f27885740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.110+0000 7efec37fe640 1 -- 172.21.15.161:0/733696760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f2788adb0 msgr2=0x561f278abdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:40.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.110+0000 7efec37fe640 1 --2- 172.21.15.161:0/733696760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f2788adb0 0x561f278abdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.110+0000 7efec37fe640 1 -- 172.21.15.161:0/733696760 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f278e1710 con 0x561f2788b190 2024-03-28T23:36:40.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.110+0000 7efec3fff640 1 --2- 172.21.15.161:0/733696760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f2788adb0 0x561f278abdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:40.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.110+0000 7efec37fe640 1 --2- 172.21.15.161:0/733696760 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788b190 0x561f27885200 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7efec40c5750 tx=0x7efeb800a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27d5d64c7079c3c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:40.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.111+0000 7efea97fa640 1 -- 172.21.15.161:0/733696760 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb802f070 con 0x561f2788b190 2024-03-28T23:36:40.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.111+0000 7efea97fa640 1 -- 172.21.15.161:0/733696760 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeb801b040 con 0x561f2788b190 2024-03-28T23:36:40.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.111+0000 7efea97fa640 1 -- 172.21.15.161:0/733696760 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb8016110 con 0x561f2788b190 2024-03-28T23:36:40.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.111+0000 7efecd36e740 1 -- 172.21.15.161:0/733696760 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788b190 msgr2=0x561f27885200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:40.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.111+0000 7efecd36e740 1 --2- 172.21.15.161:0/733696760 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788b190 0x561f27885200 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7efec40c5750 tx=0x7efeb800a000 comp rx=0 tx=0).stop 2024-03-28T23:36:40.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.111+0000 7efecd36e740 1 -- 172.21.15.161:0/733696760 shutdown_connections 2024-03-28T23:36:40.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.111+0000 7efecd36e740 1 --2- 172.21.15.161:0/733696760 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788b190 0x561f27885200 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.111+0000 7efecd36e740 1 --2- 172.21.15.161:0/733696760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561f278d84b0 0x561f27885740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.111+0000 7efecd36e740 1 --2- 172.21.15.161:0/733696760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f2788adb0 0x561f278abdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.111+0000 7efecd36e740 1 -- 172.21.15.161:0/733696760 >> 172.21.15.161:0/733696760 conn(0x561f27889310 msgr2=0x561f278d7040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:40.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.111+0000 7efecd36e740 1 -- 172.21.15.161:0/733696760 shutdown_connections 2024-03-28T23:36:40.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.111+0000 7efecd36e740 1 -- 172.21.15.161:0/733696760 wait complete. 2024-03-28T23:36:40.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.112+0000 7efecd36e740 1 -- start start 2024-03-28T23:36:40.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.112+0000 7efecd36e740 1 Processor -- start 2024-03-28T23:36:40.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.113+0000 7efecd36e740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561f2788adb0 0x561f2793a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:40.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.113+0000 7efecd36e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788b190 0x561f278a8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:40.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.113+0000 7efecd36e740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f278d84b0 0x561f278d8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:40.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.113+0000 7efecd36e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561f277aaf70 con 0x561f2788b190 2024-03-28T23:36:40.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.113+0000 7efecd36e740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x561f278da5b0 con 0x561f278d84b0 2024-03-28T23:36:40.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.113+0000 7efecd36e740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x561f278a8590 con 0x561f2788adb0 2024-03-28T23:36:40.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.113+0000 7efecd36e740 10 client.0 _ll_register_callbacks cb 0x7efeb8017040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:36:40.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.113+0000 7efec37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788b190 0x561f278a8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:40.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.113+0000 7efec37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788b190 0x561f278a8c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:43910/0 (socket says 172.21.15.161:43910) 2024-03-28T23:36:40.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.113+0000 7efec37fe640 1 -- 172.21.15.161:0/2528015057 learned_addr learned my addr 172.21.15.161:0/2528015057 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:40.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.113+0000 7efec3fff640 1 --2- 172.21.15.161:0/2528015057 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561f2788adb0 0x561f2793a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:40.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.113+0000 7efec8d27640 1 --2- 172.21.15.161:0/2528015057 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f278d84b0 0x561f278d8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:40.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.113+0000 7efec37fe640 1 -- 172.21.15.161:0/2528015057 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561f2788adb0 msgr2=0x561f2793a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:40.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.113+0000 7efec37fe640 1 --2- 172.21.15.161:0/2528015057 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561f2788adb0 0x561f2793a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.113+0000 7efec37fe640 1 -- 172.21.15.161:0/2528015057 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f278d84b0 msgr2=0x561f278d8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:40.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.113+0000 7efec37fe640 1 --2- 172.21.15.161:0/2528015057 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f278d84b0 0x561f278d8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.113+0000 7efec37fe640 1 -- 172.21.15.161:0/2528015057 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f27885f90 con 0x561f2788b190 2024-03-28T23:36:40.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.113+0000 7efec8d27640 1 --2- 172.21.15.161:0/2528015057 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f278d84b0 0x561f278d8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:36:40.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.114+0000 7efec37fe640 1 --2- 172.21.15.161:0/2528015057 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788b190 0x561f278a8c30 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7efeb8015000 tx=0x7efeb8000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5753e2272b36b63 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:40.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.114+0000 7efea2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:36:40.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.114+0000 7efea2ffd640 1 -- 172.21.15.161:0/2528015057 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb8038060 con 0x561f2788b190 2024-03-28T23:36:40.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.114+0000 7efecd36e740 1 -- 172.21.15.161:0/2528015057 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efeb801b040 con 0x561f2788b190 2024-03-28T23:36:40.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.114+0000 7efea2ffd640 1 -- 172.21.15.161:0/2528015057 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7efeb802aa30 con 0x561f2788b190 2024-03-28T23:36:40.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.114+0000 7efea2ffd640 1 -- 172.21.15.161:0/2528015057 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) ==== 24+0+0 (secure 0 0 0) 0x7efeb803e050 con 0x561f2788b190 2024-03-28T23:36:40.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.114+0000 7efecd36e740 10 client.9336 fetch_fsmap learned FSMap version 154 2024-03-28T23:36:40.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.114+0000 7efecd36e740 1 -- 172.21.15.161:0/2528015057 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) -- 0x561f27885ab0 con 0x561f2788b190 2024-03-28T23:36:40.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.114+0000 7efea2ffd640 1 -- 172.21.15.161:0/2528015057 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeb801b040 con 0x561f2788b190 2024-03-28T23:36:40.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.114+0000 7efea2ffd640 1 -- 172.21.15.161:0/2528015057 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 154) ==== 4139+0+0 (secure 0 0 0) 0x7efeb8041020 con 0x561f2788b190 2024-03-28T23:36:40.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.114+0000 7efecd36e740 10 client.9336 fetch_fsmap finished waiting for FSMap version 154 2024-03-28T23:36:40.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.114+0000 7efecd36e740 10 client.9336 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:36:40.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.114+0000 7efecd36e740 20 client.9336 populate_metadata read hostname 'smithi161' 2024-03-28T23:36:40.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.114+0000 7efecd36e740 1 --2- 172.21.15.161:0/2528015057 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x561f27939f00 0x561f278881d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:40.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.114+0000 7efecd36e740 4 client.9336 mds_command: new command op to 7283 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:36:40.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.114+0000 7efecd36e740 1 -- 172.21.15.161:0/2528015057 --> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] -- command(tid 0: {"prefix": "status"}) -- 0x561f277f6380 con 0x561f27939f00 2024-03-28T23:36:40.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.115+0000 7efec3fff640 1 --2- 172.21.15.161:0/2528015057 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x561f27939f00 0x561f278881d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:40.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.115+0000 7efea2ffd640 10 client.9336 ms_handle_connect on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:40.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.115+0000 7efec3fff640 1 --2- 172.21.15.161:0/2528015057 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x561f27939f00 0x561f278881d0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a0a998fd3a72b1e server_cookie=82b4393a8163cc06 in_seq=0 out_seq=0 2024-03-28T23:36:40.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.115+0000 7efea2ffd640 1 -- 172.21.15.161:0/2528015057 <== mds.0 v2:172.21.15.170:6832/2996092857 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x561f277f6380 con 0x561f27939f00 2024-03-28T23:36:40.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.115+0000 7efea2ffd640 10 client.9336 handle_command_reply: tid=0 2024-03-28T23:36:40.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.115+0000 7efecd36e740 1 client.9336 shutdown 2024-03-28T23:36:40.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.116+0000 7efecd36e740 1 -- 172.21.15.161:0/2528015057 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788b190 msgr2=0x561f278a8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:40.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.116+0000 7efecd36e740 1 --2- 172.21.15.161:0/2528015057 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788b190 0x561f278a8c30 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7efeb8015000 tx=0x7efeb8000ed0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.116+0000 7efecd36e740 1 -- 172.21.15.161:0/2528015057 shutdown_connections 2024-03-28T23:36:40.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.116+0000 7efecd36e740 1 --2- 172.21.15.161:0/2528015057 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x561f27939f00 0x561f278881d0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.116+0000 7efecd36e740 1 --2- 172.21.15.161:0/2528015057 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f2788b190 0x561f278a8c30 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.116+0000 7efecd36e740 1 --2- 172.21.15.161:0/2528015057 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x561f278d84b0 0x561f278d8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.116+0000 7efea2ffd640 0 client.9336 ms_handle_reset on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:40.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.116+0000 7efecd36e740 1 --2- 172.21.15.161:0/2528015057 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x561f2788adb0 0x561f2793a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.116+0000 7efecd36e740 1 -- 172.21.15.161:0/2528015057 >> 172.21.15.161:0/2528015057 conn(0x561f27889310 msgr2=0x561f278c6e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:40.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.116+0000 7efecd36e740 1 -- 172.21.15.161:0/2528015057 shutdown_connections 2024-03-28T23:36:40.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.116+0000 7efecd36e740 1 -- 172.21.15.161:0/2528015057 wait complete. 2024-03-28T23:36:40.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.116+0000 7efecd36e740 20 client.9336 trim_cache size 0 max 16384 2024-03-28T23:36:40.123 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:36:40.123 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:36:40.123 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:36:40.123 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 7283, 2024-03-28T23:36:40.123 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:36:40.124 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:36:40.124 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:36:40.124 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 22.684071656, 2024-03-28T23:36:40.124 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 154, 2024-03-28T23:36:40.124 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 161, 2024-03-28T23:36:40.124 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 159, 2024-03-28T23:36:40.124 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 25.673896346999999 2024-03-28T23:36:40.124 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:36:40.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.116+0000 7efeca78a640 1 -- 172.21.15.161:0/500371575 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efeb006b4d0 msgr2=0x7efeb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:40.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.116+0000 7efeca78a640 1 --2- 172.21.15.161:0/500371575 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efeb006b4d0 0x7efeb006d990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7efeb8007fd0 tx=0x7efeb8009040 comp rx=0 tx=0).stop 2024-03-28T23:36:40.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.116+0000 7efeca78a640 1 -- 172.21.15.161:0/500371575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efec40ab250 msgr2=0x7efec4151990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:40.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.116+0000 7efeca78a640 1 --2- 172.21.15.161:0/500371575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efec40ab250 0x7efec4151990 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7efeb40098c0 tx=0x7efeb40046c0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.116+0000 7efeca78a640 1 -- 172.21.15.161:0/500371575 shutdown_connections 2024-03-28T23:36:40.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.116+0000 7efeca78a640 1 --2- 172.21.15.161:0/500371575 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7efeb006b4d0 0x7efeb006d990 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.116+0000 7efeca78a640 1 --2- 172.21.15.161:0/500371575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efec4159380 0x7efec414c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.116+0000 7efeca78a640 1 --2- 172.21.15.161:0/500371575 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efec40c0410 0x7efec414bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.116+0000 7efeca78a640 1 --2- 172.21.15.161:0/500371575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efec40ab250 0x7efec4151990 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:40.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.116+0000 7efeca78a640 1 -- 172.21.15.161:0/500371575 >> 172.21.15.161:0/500371575 conn(0x7efec40c6b40 msgr2=0x7efec40c4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:40.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.117+0000 7efeca78a640 1 -- 172.21.15.161:0/500371575 shutdown_connections 2024-03-28T23:36:40.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:40.117+0000 7efeca78a640 1 -- 172.21.15.161:0/500371575 wait complete. 2024-03-28T23:36:40.130 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 11, "epoch": 154, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 11, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:35:58.406403+0000", "damaged": [], "data_pools": [ 24 ], "enabled": true, "epoch": 154, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7283": { "addr": "172.21.15.170:6834/2996092857", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 2996092857, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 2996092857, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7283, "incarnation": 151, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 159, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 23, "modified": "2024-03-28T23:36:20.440473+0000", "qdb_cluster": [ 7283 ], "qdb_leader": 7283, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7283 } } }, { "id": 12, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:36:07.448976+0000", "damaged": [], "data_pools": [ 26 ], "enabled": true, "epoch": 149, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs2", "in": [ 0 ], "info": { "gid_7061": { "addr": "172.21.15.161:6836/4026332703", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 4026332703, "type": "v2" }, { "addr": "172.21.15.161:6836", "nonce": 4026332703, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7061, "incarnation": 145, "join_fscid": 12, "name": "c", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 157, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 25, "modified": "2024-03-28T23:36:16.420228+0000", "qdb_cluster": [ 7061 ], "qdb_leader": 7061, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7061 } } } ], "standbys": [ { "addr": "172.21.15.170:6835/557522063", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 557522063, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 557522063, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 137, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7070, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6837/789976100", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6835", "nonce": 789976100, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 789976100, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 152, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9216, "incarnation": 0, "join_fscid": 12, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:36:40.131 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': 12, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 12, 'name': 'c'}, {'join_fscid': -1, 'name': 'd', 'state': 'up:active'}] 2024-03-28T23:36:40.131 INFO:tasks.cephfs.test_failover:current = [{'gid': 9216, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6837/789976100', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 789976100}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 789976100}]}, 'join_fscid': 12, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 152}, {'gid': 7283, 'name': 'b', 'rank': 0, 'incarnation': 151, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.170:6834/2996092857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 2996092857}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 2996092857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 7061, 'name': 'c', 'rank': 0, 'incarnation': 145, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.161:6836/4026332703', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 4026332703}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 4026332703}]}, 'join_fscid': 12, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 7070, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/557522063', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 557522063}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 557522063}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 137}] 2024-03-28T23:36:40.132 DEBUG:tasks.cephfs.test_failover:'up:active' != 'up:standby' - up:active + up:standby 2024-03-28T23:36:40.132 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-28T23:36:45.133 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:36:45.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.235+0000 7fa85e35c640 1 Processor -- start 2024-03-28T23:36:45.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.235+0000 7fa85e35c640 1 -- start start 2024-03-28T23:36:45.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.235+0000 7fa85e35c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa858152890 0x7fa8580c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:45.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.235+0000 7fa85e35c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa8580c8ed0 0x7fa8580c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:45.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.235+0000 7fa85e35c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa858158a30 con 0x7fa8580c5950 2024-03-28T23:36:45.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.235+0000 7fa85e35c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa8580c5db0 con 0x7fa8580c8ed0 2024-03-28T23:36:45.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.235+0000 7fa85e35c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa8580c8480 con 0x7fa858152890 2024-03-28T23:36:45.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.236+0000 7fa8577fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa8580c8ed0 0x7fa8580c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:45.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.236+0000 7fa8577fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa8580c8ed0 0x7fa8580c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34186/0 (socket says 172.21.15.161:34186) 2024-03-28T23:36:45.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.236+0000 7fa8577fe640 1 -- 172.21.15.161:0/1582982957 learned_addr learned my addr 172.21.15.161:0/1582982957 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:45.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.236+0000 7fa857fff640 1 --2- 172.21.15.161:0/1582982957 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa858152890 0x7fa8580c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:45.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.236+0000 7fa8577fe640 1 -- 172.21.15.161:0/1582982957 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa858152890 msgr2=0x7fa8580c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:45.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.236+0000 7fa8577fe640 1 --2- 172.21.15.161:0/1582982957 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa858152890 0x7fa8580c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:45.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.236+0000 7fa8577fe640 1 -- 172.21.15.161:0/1582982957 >> v1:172.21.15.170:6789/0 conn(0x7fa8580c5950 legacy=0x7fa8581553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:45.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.236+0000 7fa8577fe640 1 -- 172.21.15.161:0/1582982957 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa858158d10 con 0x7fa8580c8ed0 2024-03-28T23:36:45.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.237+0000 7fa8577fe640 1 --2- 172.21.15.161:0/1582982957 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8580c8ed0 0x7fa8580c5410 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fa84c009850 tx=0x7fa84c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c2210bad0694809 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:45.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.237+0000 7fa856ffd640 1 -- 172.21.15.161:0/1582982957 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84c037070 con 0x7fa8580c8ed0 2024-03-28T23:36:45.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.237+0000 7fa856ffd640 1 -- 172.21.15.161:0/1582982957 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa84c004030 con 0x7fa8580c8ed0 2024-03-28T23:36:45.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.237+0000 7fa856ffd640 1 -- 172.21.15.161:0/1582982957 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84c004460 con 0x7fa8580c8ed0 2024-03-28T23:36:45.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.237+0000 7fa85e35c640 1 -- 172.21.15.161:0/1582982957 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8580c8ed0 msgr2=0x7fa8580c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:45.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.237+0000 7fa85e35c640 1 --2- 172.21.15.161:0/1582982957 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8580c8ed0 0x7fa8580c5410 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fa84c009850 tx=0x7fa84c02b840 comp rx=0 tx=0).stop 2024-03-28T23:36:45.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.237+0000 7fa85e35c640 1 -- 172.21.15.161:0/1582982957 shutdown_connections 2024-03-28T23:36:45.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.237+0000 7fa85e35c640 1 --2- 172.21.15.161:0/1582982957 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8580c8ed0 0x7fa8580c5410 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:45.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.237+0000 7fa85e35c640 1 --2- 172.21.15.161:0/1582982957 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa858152890 0x7fa8580c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:45.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.238+0000 7fa85e35c640 1 -- 172.21.15.161:0/1582982957 >> 172.21.15.161:0/1582982957 conn(0x7fa8580ac620 msgr2=0x7fa8580aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:45.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.238+0000 7fa85e35c640 1 -- 172.21.15.161:0/1582982957 shutdown_connections 2024-03-28T23:36:45.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.238+0000 7fa85e35c640 1 -- 172.21.15.161:0/1582982957 wait complete. 2024-03-28T23:36:45.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.239+0000 7fa85e35c640 1 Processor -- start 2024-03-28T23:36:45.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.239+0000 7fa85e35c640 1 -- start start 2024-03-28T23:36:45.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.239+0000 7fa85e35c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8580c5950 0x7fa858141230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:45.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.239+0000 7fa85e35c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8580c8ed0 0x7fa8580ab430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:45.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.239+0000 7fa85e35c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa858152890 0x7fa85814cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:45.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.239+0000 7fa85e35c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa8581592d0 con 0x7fa8580c8ed0 2024-03-28T23:36:45.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.239+0000 7fa85e35c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa858159640 con 0x7fa8580c5950 2024-03-28T23:36:45.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.239+0000 7fa85e35c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa858152c70 con 0x7fa858152890 2024-03-28T23:36:45.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.240+0000 7fa8577fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8580c8ed0 0x7fa8580ab430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:45.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.240+0000 7fa8577fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8580c8ed0 0x7fa8580ab430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34202/0 (socket says 172.21.15.161:34202) 2024-03-28T23:36:45.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.240+0000 7fa8577fe640 1 -- 172.21.15.161:0/3216127587 learned_addr learned my addr 172.21.15.161:0/3216127587 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:45.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.240+0000 7fa85c8f9640 1 --2- 172.21.15.161:0/3216127587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa858152890 0x7fa85814cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:45.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.240+0000 7fa857fff640 1 --2- 172.21.15.161:0/3216127587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8580c5950 0x7fa858141230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:45.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.240+0000 7fa8577fe640 1 -- 172.21.15.161:0/3216127587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa858152890 msgr2=0x7fa85814cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:45.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.240+0000 7fa8577fe640 1 --2- 172.21.15.161:0/3216127587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa858152890 0x7fa85814cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:45.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.240+0000 7fa8577fe640 1 -- 172.21.15.161:0/3216127587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8580c5950 msgr2=0x7fa858141230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:45.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.240+0000 7fa8577fe640 1 --2- 172.21.15.161:0/3216127587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8580c5950 0x7fa858141230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:45.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.240+0000 7fa8577fe640 1 -- 172.21.15.161:0/3216127587 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa85814b720 con 0x7fa8580c8ed0 2024-03-28T23:36:45.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.240+0000 7fa8577fe640 1 --2- 172.21.15.161:0/3216127587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8580c8ed0 0x7fa8580ab430 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fa84c009970 tx=0x7fa84c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:45.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.240+0000 7fa854ff9640 1 -- 172.21.15.161:0/3216127587 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84c037050 con 0x7fa8580c8ed0 2024-03-28T23:36:45.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.240+0000 7fa854ff9640 1 -- 172.21.15.161:0/3216127587 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa84c038040 con 0x7fa8580c8ed0 2024-03-28T23:36:45.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.240+0000 7fa854ff9640 1 -- 172.21.15.161:0/3216127587 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84c003b80 con 0x7fa8580c8ed0 2024-03-28T23:36:45.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.240+0000 7fa85e35c640 1 -- 172.21.15.161:0/3216127587 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8580c1d00 con 0x7fa8580c8ed0 2024-03-28T23:36:45.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.240+0000 7fa85e35c640 1 -- 172.21.15.161:0/3216127587 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8580c1fa0 con 0x7fa8580c8ed0 2024-03-28T23:36:45.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.241+0000 7fa85e35c640 1 -- 172.21.15.161:0/3216127587 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa85804f4b0 con 0x7fa8580c8ed0 2024-03-28T23:36:45.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.242+0000 7fa854ff9640 1 -- 172.21.15.161:0/3216127587 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa84c045030 con 0x7fa8580c8ed0 2024-03-28T23:36:45.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.242+0000 7fa854ff9640 1 --2- 172.21.15.161:0/3216127587 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa84406b4d0 0x7fa84406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:45.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.243+0000 7fa854ff9640 1 -- 172.21.15.161:0/3216127587 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7fa84c02f020 con 0x7fa8580c8ed0 2024-03-28T23:36:45.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.243+0000 7fa857fff640 1 --2- 172.21.15.161:0/3216127587 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa84406b4d0 0x7fa84406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:45.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.243+0000 7fa857fff640 1 --2- 172.21.15.161:0/3216127587 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa84406b4d0 0x7fa84406d990 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fa848009b40 tx=0x7fa848009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:45.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.247+0000 7fa854ff9640 1 -- 172.21.15.161:0/3216127587 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa84c04c020 con 0x7fa8580c8ed0 2024-03-28T23:36:45.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.463+0000 7fa85e35c640 1 -- 172.21.15.161:0/3216127587 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa85814fad0 con 0x7fa8580c8ed0 2024-03-28T23:36:45.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.463+0000 7fa854ff9640 1 -- 172.21.15.161:0/3216127587 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+6520 (secure 0 0 0) 0x7fa84c079ef0 con 0x7fa8580c8ed0 2024-03-28T23:36:45.466 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:45.466 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":154,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/557522063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":557522063},{"type":"v1","addr":"172.21.15.170:6835","nonce":557522063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":9216,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/789976100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":789976100},{"type":"v1","addr":"172.21.15.161:6837","nonce":789976100}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:58.406403+0000","modified":"2024-03-28T23:36:20.440473+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":159,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7283":{"gid":7283,"name":"b","rank":0,"incarnation":151,"state":"up:active","state_seq":4,"addr":"172.21.15.170:6834/2996092857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":2996092857},{"type":"v1","addr":"172.21.15.170:6834","nonce":2996092857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7283,"qdb_cluster":[7283]},"id":11},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:36:07.448976+0000","modified":"2024-03-28T23:36:16.420228+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7061":{"gid":7061,"name":"c","rank":0,"incarnation":145,"state":"up:active","state_seq":9,"addr":"172.21.15.161:6836/4026332703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4026332703},{"type":"v1","addr":"172.21.15.161:6836","nonce":4026332703}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7061,"qdb_cluster":[7061]},"id":12}]} 2024-03-28T23:36:45.467 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 154 2024-03-28T23:36:45.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.465+0000 7fa85e35c640 1 -- 172.21.15.161:0/3216127587 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa84406b4d0 msgr2=0x7fa84406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:45.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.465+0000 7fa85e35c640 1 --2- 172.21.15.161:0/3216127587 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa84406b4d0 0x7fa84406d990 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fa848009b40 tx=0x7fa848009190 comp rx=0 tx=0).stop 2024-03-28T23:36:45.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.465+0000 7fa85e35c640 1 -- 172.21.15.161:0/3216127587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8580c8ed0 msgr2=0x7fa8580ab430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:45.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.465+0000 7fa85e35c640 1 --2- 172.21.15.161:0/3216127587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8580c8ed0 0x7fa8580ab430 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fa84c009970 tx=0x7fa84c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:36:45.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.465+0000 7fa85e35c640 1 -- 172.21.15.161:0/3216127587 shutdown_connections 2024-03-28T23:36:45.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.465+0000 7fa85e35c640 1 --2- 172.21.15.161:0/3216127587 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa84406b4d0 0x7fa84406d990 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:45.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.465+0000 7fa85e35c640 1 --2- 172.21.15.161:0/3216127587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8580c8ed0 0x7fa8580ab430 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:45.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.465+0000 7fa85e35c640 1 --2- 172.21.15.161:0/3216127587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa858152890 0x7fa85814cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:45.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.465+0000 7fa85e35c640 1 --2- 172.21.15.161:0/3216127587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8580c5950 0x7fa858141230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:45.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.465+0000 7fa85e35c640 1 -- 172.21.15.161:0/3216127587 >> 172.21.15.161:0/3216127587 conn(0x7fa8580ac620 msgr2=0x7fa858150460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:45.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.465+0000 7fa85e35c640 1 -- 172.21.15.161:0/3216127587 shutdown_connections 2024-03-28T23:36:45.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.466+0000 7fa85e35c640 1 -- 172.21.15.161:0/3216127587 wait complete. 2024-03-28T23:36:45.879 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 154, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:35:58.406403+0000', 'modified': '2024-03-28T23:36:20.440473+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 159, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7283}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7283': {'gid': 7283, 'name': 'b', 'rank': 0, 'incarnation': 151, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.170:6834/2996092857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 2996092857}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 2996092857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7283, 'qdb_cluster': [7283]} 2024-03-28T23:36:45.879 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:36:45.880 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T23:36:45.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.982+0000 7f58236eb640 1 Processor -- start 2024-03-28T23:36:45.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.982+0000 7f58236eb640 1 -- start start 2024-03-28T23:36:45.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.983+0000 7f58236eb640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f581c152a80 0x7f581c155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:45.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.983+0000 7f58236eb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f581c06c3c0 con 0x7f581c06d9f0 2024-03-28T23:36:45.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.983+0000 7f58236eb640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f581c0c5b90 con 0x7f581c06cb80 2024-03-28T23:36:45.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.983+0000 7f58236eb640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f581c156250 con 0x7f581c152a80 2024-03-28T23:36:45.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.983+0000 7f5820c86640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f581c06d9f0 0x7f581c0c93a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:53156/0 (socket says 172.21.15.161:53156) 2024-03-28T23:36:45.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.983+0000 7f5820c86640 1 -- 172.21.15.161:0/1762115957 learned_addr learned my addr 172.21.15.161:0/1762115957 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:45.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.983+0000 7f5821487640 1 --2- 172.21.15.161:0/1762115957 >> v2:172.21.15.170:3300/0 conn(0x7f581c152a80 0x7f581c155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:45.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.983+0000 7f5813fff640 1 -- 172.21.15.161:0/1762115957 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2192804898 0 0) 0x7f581c06c3c0 con 0x7f581c06d9f0 2024-03-28T23:36:45.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.984+0000 7f5813fff640 1 -- 172.21.15.161:0/1762115957 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5808003520 con 0x7f581c06d9f0 2024-03-28T23:36:45.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.984+0000 7f5813fff640 1 -- 172.21.15.161:0/1762115957 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1112268649 0 0) 0x7f581c0c5b90 con 0x7f581c06cb80 2024-03-28T23:36:45.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.984+0000 7f5813fff640 1 -- 172.21.15.161:0/1762115957 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f581c06c3c0 con 0x7f581c06cb80 2024-03-28T23:36:45.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.984+0000 7f5821487640 1 -- 172.21.15.161:0/1762115957 >> v1:172.21.15.170:6789/0 conn(0x7f581c06cb80 legacy=0x7f581c0c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:45.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.984+0000 7f5821487640 1 -- 172.21.15.161:0/1762115957 >> v1:172.21.15.161:6789/0 conn(0x7f581c06d9f0 legacy=0x7f581c0c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:45.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.984+0000 7f5821487640 1 -- 172.21.15.161:0/1762115957 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f581c0c55d0 con 0x7f581c152a80 2024-03-28T23:36:45.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.984+0000 7f5821487640 1 --2- 172.21.15.161:0/1762115957 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f581c152a80 0x7f581c155d10 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f58180097a0 tx=0x7f581802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5420d9a3a0def0a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:45.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.985+0000 7f5813fff640 1 -- 172.21.15.161:0/1762115957 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58180315a0 con 0x7f581c152a80 2024-03-28T23:36:45.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.985+0000 7f5813fff640 1 -- 172.21.15.161:0/1762115957 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5818037070 con 0x7f581c152a80 2024-03-28T23:36:45.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.985+0000 7f5813fff640 1 -- 172.21.15.161:0/1762115957 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5818031940 con 0x7f581c152a80 2024-03-28T23:36:45.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.985+0000 7f58236eb640 1 -- 172.21.15.161:0/1762115957 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f581c152a80 msgr2=0x7f581c155d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:45.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.985+0000 7f58236eb640 1 --2- 172.21.15.161:0/1762115957 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f581c152a80 0x7f581c155d10 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f58180097a0 tx=0x7f581802b840 comp rx=0 tx=0).stop 2024-03-28T23:36:45.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.985+0000 7f58236eb640 1 -- 172.21.15.161:0/1762115957 shutdown_connections 2024-03-28T23:36:45.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.985+0000 7f58236eb640 1 --2- 172.21.15.161:0/1762115957 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f581c152a80 0x7f581c155d10 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:45.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.985+0000 7f58236eb640 1 -- 172.21.15.161:0/1762115957 >> 172.21.15.161:0/1762115957 conn(0x7f581c0ac720 msgr2=0x7f581c0acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:45.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.985+0000 7f58236eb640 1 -- 172.21.15.161:0/1762115957 shutdown_connections 2024-03-28T23:36:45.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.985+0000 7f58236eb640 1 -- 172.21.15.161:0/1762115957 wait complete. 2024-03-28T23:36:45.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.987+0000 7f58236eb640 1 Processor -- start 2024-03-28T23:36:45.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.987+0000 7f58236eb640 1 -- start start 2024-03-28T23:36:45.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.988+0000 7f58236eb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f581c06cb80 0x7f581c0b0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:45.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.988+0000 7f58236eb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f581c06d9f0 0x7f581c141a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:45.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.988+0000 7f58236eb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f581c152a80 0x7f581c0ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:45.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.988+0000 7f58236eb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f581c0c7bd0 con 0x7f581c06d9f0 2024-03-28T23:36:45.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.988+0000 7f58236eb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f581c06c6f0 con 0x7f581c06cb80 2024-03-28T23:36:45.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.988+0000 7f58236eb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f581c0c7d50 con 0x7f581c152a80 2024-03-28T23:36:45.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.988+0000 7f5820c86640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f581c06d9f0 0x7f581c141a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:45.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.988+0000 7f5820c86640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f581c06d9f0 0x7f581c141a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34212/0 (socket says 172.21.15.161:34212) 2024-03-28T23:36:45.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.988+0000 7f5820c86640 1 -- 172.21.15.161:0/2506496425 learned_addr learned my addr 172.21.15.161:0/2506496425 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:45.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.988+0000 7f5821487640 1 --2- 172.21.15.161:0/2506496425 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f581c06cb80 0x7f581c0b0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:45.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.988+0000 7f5821c88640 1 --2- 172.21.15.161:0/2506496425 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f581c152a80 0x7f581c0ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:45.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.988+0000 7f5820c86640 1 -- 172.21.15.161:0/2506496425 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f581c152a80 msgr2=0x7f581c0ae3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:45.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.988+0000 7f5820c86640 1 --2- 172.21.15.161:0/2506496425 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f581c152a80 0x7f581c0ae3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:45.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.988+0000 7f5820c86640 1 -- 172.21.15.161:0/2506496425 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f581c06cb80 msgr2=0x7f581c0b0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:45.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.988+0000 7f5820c86640 1 --2- 172.21.15.161:0/2506496425 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f581c06cb80 0x7f581c0b0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:45.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.988+0000 7f5820c86640 1 -- 172.21.15.161:0/2506496425 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f581c0afa60 con 0x7f581c06d9f0 2024-03-28T23:36:45.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.988+0000 7f5820c86640 1 --2- 172.21.15.161:0/2506496425 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f581c06d9f0 0x7f581c141a70 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f580c00c500 tx=0x7f580c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:45.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.988+0000 7f5811ffb640 1 -- 172.21.15.161:0/2506496425 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580c017070 con 0x7f581c06d9f0 2024-03-28T23:36:45.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.989+0000 7f5811ffb640 1 -- 172.21.15.161:0/2506496425 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f580c005d00 con 0x7f581c06d9f0 2024-03-28T23:36:45.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.989+0000 7f58236eb640 1 -- 172.21.15.161:0/2506496425 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f581c142330 con 0x7f581c06d9f0 2024-03-28T23:36:45.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.989+0000 7f5811ffb640 1 -- 172.21.15.161:0/2506496425 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580c007330 con 0x7f581c06d9f0 2024-03-28T23:36:45.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.989+0000 7f58236eb640 1 -- 172.21.15.161:0/2506496425 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f581c0afd20 con 0x7f581c06d9f0 2024-03-28T23:36:45.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.990+0000 7f5811ffb640 1 -- 172.21.15.161:0/2506496425 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f580c01a070 con 0x7f581c06d9f0 2024-03-28T23:36:45.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.991+0000 7f5811ffb640 1 --2- 172.21.15.161:0/2506496425 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f580006b4d0 0x7f580006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:45.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.991+0000 7f5821487640 1 --2- 172.21.15.161:0/2506496425 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f580006b4d0 0x7f580006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:45.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.991+0000 7f5811ffb640 1 -- 172.21.15.161:0/2506496425 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7f580c089c20 con 0x7f581c06d9f0 2024-03-28T23:36:45.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.991+0000 7f5821487640 1 --2- 172.21.15.161:0/2506496425 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f580006b4d0 0x7f580006d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f5818036000 tx=0x7f581803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:45.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.992+0000 7f58263a5740 1 Processor -- start 2024-03-28T23:36:45.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.992+0000 7f58263a5740 1 -- start start 2024-03-28T23:36:45.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.993+0000 7f58263a5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ac535e1db0 0x55ac535df890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:45.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.993+0000 7f58263a5740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e2190 0x55ac535dfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:45.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.993+0000 7f58263a5740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ac535e0310 0x55ac535dc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:45.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.993+0000 7f58263a5740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55ac53545af0 con 0x55ac535e2190 2024-03-28T23:36:45.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.993+0000 7f58263a5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55ac534ee9d0 con 0x55ac535e1db0 2024-03-28T23:36:45.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.993+0000 7f58263a5740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55ac53541b90 con 0x55ac535e0310 2024-03-28T23:36:45.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.993+0000 7f5820c86640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e2190 0x55ac535dfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:45.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.993+0000 7f5820c86640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e2190 0x55ac535dfdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34218/0 (socket says 172.21.15.161:34218) 2024-03-28T23:36:45.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.993+0000 7f5820c86640 1 -- 172.21.15.161:0/638530909 learned_addr learned my addr 172.21.15.161:0/638530909 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:45.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.993+0000 7f5821487640 1 --2- 172.21.15.161:0/638530909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ac535e1db0 0x55ac535df890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:45.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.993+0000 7f5821c88640 1 --2- 172.21.15.161:0/638530909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ac535e0310 0x55ac535dc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:45.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.993+0000 7f5820c86640 1 -- 172.21.15.161:0/638530909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ac535e0310 msgr2=0x55ac535dc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:45.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.993+0000 7f5820c86640 1 --2- 172.21.15.161:0/638530909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ac535e0310 0x55ac535dc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:45.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.993+0000 7f5820c86640 1 -- 172.21.15.161:0/638530909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ac535e1db0 msgr2=0x55ac535df890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:45.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.993+0000 7f5820c86640 1 --2- 172.21.15.161:0/638530909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ac535e1db0 0x55ac535df890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:45.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.993+0000 7f5820c86640 1 -- 172.21.15.161:0/638530909 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac5325a2d0 con 0x55ac535e2190 2024-03-28T23:36:45.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.993+0000 7f5820c86640 1 --2- 172.21.15.161:0/638530909 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e2190 0x55ac535dfdd0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f580c057e50 tx=0x7f580c058000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88108d4d4d19965e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:45.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.994+0000 7f5806ffd640 1 -- 172.21.15.161:0/638530909 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580c074070 con 0x55ac535e2190 2024-03-28T23:36:45.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.994+0000 7f5806ffd640 1 -- 172.21.15.161:0/638530909 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f580c054530 con 0x55ac535e2190 2024-03-28T23:36:45.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.994+0000 7f5806ffd640 1 -- 172.21.15.161:0/638530909 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580c058120 con 0x55ac535e2190 2024-03-28T23:36:45.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.994+0000 7f58263a5740 1 -- 172.21.15.161:0/638530909 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e2190 msgr2=0x55ac535dfdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:45.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.994+0000 7f58263a5740 1 --2- 172.21.15.161:0/638530909 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e2190 0x55ac535dfdd0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f580c057e50 tx=0x7f580c058000 comp rx=0 tx=0).stop 2024-03-28T23:36:45.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.994+0000 7f58263a5740 1 -- 172.21.15.161:0/638530909 shutdown_connections 2024-03-28T23:36:45.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.994+0000 7f58263a5740 1 --2- 172.21.15.161:0/638530909 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e2190 0x55ac535dfdd0 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:45.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.994+0000 7f58263a5740 1 --2- 172.21.15.161:0/638530909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ac535e0310 0x55ac535dc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:45.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.994+0000 7f58263a5740 1 --2- 172.21.15.161:0/638530909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ac535e1db0 0x55ac535df890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:45.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.994+0000 7f58263a5740 1 -- 172.21.15.161:0/638530909 >> 172.21.15.161:0/638530909 conn(0x55ac535de630 msgr2=0x55ac53603220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:45.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.994+0000 7f58263a5740 1 -- 172.21.15.161:0/638530909 shutdown_connections 2024-03-28T23:36:45.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.994+0000 7f58263a5740 1 -- 172.21.15.161:0/638530909 wait complete. 2024-03-28T23:36:45.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.995+0000 7f58263a5740 1 -- start start 2024-03-28T23:36:45.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.995+0000 7f58263a5740 1 Processor -- start 2024-03-28T23:36:46.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.996+0000 7f58263a5740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ac535e1db0 0x55ac536022b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:46.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.996+0000 7f58263a5740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e2190 0x55ac53604be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:46.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.996+0000 7f58263a5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ac53605120 0x55ac5361a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:46.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.996+0000 7f58263a5740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55ac53545af0 con 0x55ac535e2190 2024-03-28T23:36:46.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.996+0000 7f58263a5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55ac53501f70 con 0x55ac53605120 2024-03-28T23:36:46.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.996+0000 7f58263a5740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55ac53604090 con 0x55ac535e1db0 2024-03-28T23:36:46.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.996+0000 7f58263a5740 10 client.0 _ll_register_callbacks cb 0x55ac53259ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:36:46.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.996+0000 7f5820c86640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e2190 0x55ac53604be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:46.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.996+0000 7f5820c86640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e2190 0x55ac53604be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34232/0 (socket says 172.21.15.161:34232) 2024-03-28T23:36:46.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.996+0000 7f5821487640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ac535e1db0 0x55ac536022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:46.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.996+0000 7f5820c86640 1 -- 172.21.15.161:0/914794840 learned_addr learned my addr 172.21.15.161:0/914794840 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:46.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.996+0000 7f5821c88640 1 --2- 172.21.15.161:0/914794840 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ac53605120 0x55ac5361a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:46.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.996+0000 7f5820c86640 1 -- 172.21.15.161:0/914794840 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ac535e1db0 msgr2=0x55ac536022b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:46.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.996+0000 7f5820c86640 1 --2- 172.21.15.161:0/914794840 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ac535e1db0 0x55ac536022b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:46.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.996+0000 7f5820c86640 1 -- 172.21.15.161:0/914794840 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ac53605120 msgr2=0x55ac5361a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:46.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.996+0000 7f5820c86640 1 --2- 172.21.15.161:0/914794840 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ac53605120 0x55ac5361a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:46.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.996+0000 7f5820c86640 1 -- 172.21.15.161:0/914794840 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac533246b0 con 0x55ac535e2190 2024-03-28T23:36:46.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.996+0000 7f5820c86640 1 --2- 172.21.15.161:0/914794840 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e2190 0x55ac53604be0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f580c056170 tx=0x7f580c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f2dc204cdab388 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:46.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.996+0000 7f57fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:36:46.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.996+0000 7f57fbfff640 1 -- 172.21.15.161:0/914794840 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580c074040 con 0x55ac535e2190 2024-03-28T23:36:46.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.996+0000 7f58263a5740 1 -- 172.21.15.161:0/914794840 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f580c054530 con 0x55ac535e2190 2024-03-28T23:36:46.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.996+0000 7f57fbfff640 1 -- 172.21.15.161:0/914794840 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7f580c058120 con 0x55ac535e2190 2024-03-28T23:36:46.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.997+0000 7f57fbfff640 1 -- 172.21.15.161:0/914794840 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) ==== 24+0+0 (secure 0 0 0) 0x7f580c07d050 con 0x55ac535e2190 2024-03-28T23:36:46.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.997+0000 7f58263a5740 10 client.9354 fetch_fsmap learned FSMap version 154 2024-03-28T23:36:46.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.997+0000 7f58263a5740 1 -- 172.21.15.161:0/914794840 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) -- 0x55ac53606200 con 0x55ac535e2190 2024-03-28T23:36:46.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.997+0000 7f57fbfff640 1 -- 172.21.15.161:0/914794840 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f580c054530 con 0x55ac535e2190 2024-03-28T23:36:46.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.997+0000 7f57fbfff640 1 -- 172.21.15.161:0/914794840 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 154) ==== 4139+0+0 (secure 0 0 0) 0x7f580c080020 con 0x55ac535e2190 2024-03-28T23:36:46.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.997+0000 7f58263a5740 10 client.9354 fetch_fsmap finished waiting for FSMap version 154 2024-03-28T23:36:46.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.997+0000 7f58263a5740 10 client.9354 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:36:46.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.997+0000 7f58263a5740 20 client.9354 populate_metadata read hostname 'smithi161' 2024-03-28T23:36:46.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.997+0000 7f58263a5740 1 --2- 172.21.15.161:0/914794840 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x55ac5362e2c0 0x55ac5362e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:46.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.997+0000 7f58263a5740 4 client.9354 mds_command: new command op to 7283 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:36:46.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.997+0000 7f58263a5740 1 -- 172.21.15.161:0/914794840 --> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac53553030 con 0x55ac5362e2c0 2024-03-28T23:36:46.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.997+0000 7f5821487640 1 --2- 172.21.15.161:0/914794840 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x55ac5362e2c0 0x55ac5362e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:46.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.998+0000 7f57fbfff640 10 client.9354 ms_handle_connect on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:46.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.998+0000 7f5821487640 1 --2- 172.21.15.161:0/914794840 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x55ac5362e2c0 0x55ac5362e6a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d0144eca8a3a353 server_cookie=ef4ee69f8cd61036 in_seq=0 out_seq=0 2024-03-28T23:36:46.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.998+0000 7f57fbfff640 1 -- 172.21.15.161:0/914794840 <== mds.0 v2:172.21.15.170:6832/2996092857 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ac53553030 con 0x55ac5362e2c0 2024-03-28T23:36:46.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.998+0000 7f57fbfff640 10 client.9354 handle_command_reply: tid=0 2024-03-28T23:36:46.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.998+0000 7f58263a5740 1 client.9354 shutdown 2024-03-28T23:36:46.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.998+0000 7f58263a5740 1 -- 172.21.15.161:0/914794840 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e2190 msgr2=0x55ac53604be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:46.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.998+0000 7f58263a5740 1 --2- 172.21.15.161:0/914794840 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e2190 0x55ac53604be0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f580c056170 tx=0x7f580c000ed0 comp rx=0 tx=0).stop 2024-03-28T23:36:46.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.998+0000 7f58263a5740 1 -- 172.21.15.161:0/914794840 shutdown_connections 2024-03-28T23:36:46.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.998+0000 7f58263a5740 1 --2- 172.21.15.161:0/914794840 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x55ac5362e2c0 0x55ac5362e6a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:46.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.998+0000 7f58263a5740 1 --2- 172.21.15.161:0/914794840 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e2190 0x55ac53604be0 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:46.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.998+0000 7f57fbfff640 0 client.9354 ms_handle_reset on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:46.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.998+0000 7f58263a5740 1 --2- 172.21.15.161:0/914794840 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ac53605120 0x55ac5361a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:46.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.998+0000 7f58263a5740 1 --2- 172.21.15.161:0/914794840 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ac535e1db0 0x55ac536022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:46.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.998+0000 7f58263a5740 1 -- 172.21.15.161:0/914794840 >> 172.21.15.161:0/914794840 conn(0x55ac535e0310 msgr2=0x55ac535db2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:46.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.999+0000 7f58263a5740 1 -- 172.21.15.161:0/914794840 shutdown_connections 2024-03-28T23:36:46.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.999+0000 7f58263a5740 1 -- 172.21.15.161:0/914794840 wait complete. 2024-03-28T23:36:46.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:45.999+0000 7f58263a5740 20 client.9354 trim_cache size 0 max 16384 2024-03-28T23:36:46.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.033+0000 7f58263a5740 1 Processor -- start 2024-03-28T23:36:46.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.033+0000 7f58263a5740 1 -- start start 2024-03-28T23:36:46.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.034+0000 7f58263a5740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e1db0 0x55ac535e0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:46.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.034+0000 7f58263a5740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ac535e2190 0x55ac53606870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:46.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.034+0000 7f58263a5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ac53605120 0x55ac53606db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:46.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.034+0000 7f58263a5740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f581c1fafa0 con 0x55ac535e1db0 2024-03-28T23:36:46.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.034+0000 7f58263a5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f581c1f8aa0 con 0x55ac53605120 2024-03-28T23:36:46.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.034+0000 7f58263a5740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f581c1f8fa0 con 0x55ac535e2190 2024-03-28T23:36:46.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.034+0000 7f5821487640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e1db0 0x55ac535e0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:46.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.034+0000 7f5821487640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e1db0 0x55ac535e0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34244/0 (socket says 172.21.15.161:34244) 2024-03-28T23:36:46.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.034+0000 7f5821487640 1 -- 172.21.15.161:0/2811793168 learned_addr learned my addr 172.21.15.161:0/2811793168 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:46.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.034+0000 7f5820c86640 1 --2- 172.21.15.161:0/2811793168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ac535e2190 0x55ac53606870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:46.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.034+0000 7f5821c88640 1 --2- 172.21.15.161:0/2811793168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ac53605120 0x55ac53606db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:46.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.035+0000 7f5820c86640 1 -- 172.21.15.161:0/2811793168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ac53605120 msgr2=0x55ac53606db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:46.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.035+0000 7f5820c86640 1 --2- 172.21.15.161:0/2811793168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ac53605120 0x55ac53606db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:46.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.035+0000 7f5820c86640 1 -- 172.21.15.161:0/2811793168 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e1db0 msgr2=0x55ac535e0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:46.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.035+0000 7f5820c86640 1 --2- 172.21.15.161:0/2811793168 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e1db0 0x55ac535e0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:46.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.035+0000 7f5820c86640 1 -- 172.21.15.161:0/2811793168 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac53617090 con 0x55ac535e2190 2024-03-28T23:36:46.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.035+0000 7f5821487640 1 --2- 172.21.15.161:0/2811793168 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e1db0 0x55ac535e0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:46.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.035+0000 7f5820c86640 1 --2- 172.21.15.161:0/2811793168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ac535e2190 0x55ac53606870 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f580c00cec0 tx=0x7f580c089000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=894862bf3976a092 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:46.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.035+0000 7f58067fc640 1 -- 172.21.15.161:0/2811793168 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580c074070 con 0x55ac535e2190 2024-03-28T23:36:46.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.035+0000 7f58067fc640 1 -- 172.21.15.161:0/2811793168 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f580c086070 con 0x55ac535e2190 2024-03-28T23:36:46.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.035+0000 7f58067fc640 1 -- 172.21.15.161:0/2811793168 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580c054530 con 0x55ac535e2190 2024-03-28T23:36:46.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.035+0000 7f58263a5740 1 -- 172.21.15.161:0/2811793168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ac535e2190 msgr2=0x55ac53606870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:46.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.035+0000 7f58263a5740 1 --2- 172.21.15.161:0/2811793168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ac535e2190 0x55ac53606870 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f580c00cec0 tx=0x7f580c089000 comp rx=0 tx=0).stop 2024-03-28T23:36:46.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.036+0000 7f58263a5740 1 -- 172.21.15.161:0/2811793168 shutdown_connections 2024-03-28T23:36:46.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.036+0000 7f58263a5740 1 --2- 172.21.15.161:0/2811793168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ac53605120 0x55ac53606db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:46.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.036+0000 7f58263a5740 1 --2- 172.21.15.161:0/2811793168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ac535e2190 0x55ac53606870 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:46.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.036+0000 7f58263a5740 1 --2- 172.21.15.161:0/2811793168 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e1db0 0x55ac535e0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:46.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.036+0000 7f58263a5740 1 -- 172.21.15.161:0/2811793168 >> 172.21.15.161:0/2811793168 conn(0x55ac535e0310 msgr2=0x55ac535e2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:46.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.036+0000 7f58263a5740 1 -- 172.21.15.161:0/2811793168 shutdown_connections 2024-03-28T23:36:46.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.036+0000 7f58263a5740 1 -- 172.21.15.161:0/2811793168 wait complete. 2024-03-28T23:36:46.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.036+0000 7f58263a5740 1 -- start start 2024-03-28T23:36:46.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.036+0000 7f58263a5740 1 Processor -- start 2024-03-28T23:36:46.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.037+0000 7f58263a5740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ac535e1db0 0x55ac5362a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:46.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.037+0000 7f58263a5740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e2190 0x55ac5362b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:46.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.037+0000 7f58263a5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ac53605120 0x55ac535ffe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:46.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.037+0000 7f58263a5740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55ac53501f70 con 0x55ac535e2190 2024-03-28T23:36:46.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.037+0000 7f58263a5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x55ac5362e140 con 0x55ac53605120 2024-03-28T23:36:46.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.037+0000 7f58263a5740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55ac5361b070 con 0x55ac535e1db0 2024-03-28T23:36:46.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.037+0000 7f58263a5740 10 client.0 _ll_register_callbacks cb 0x7f580c072070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:36:46.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.037+0000 7f5820c86640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e2190 0x55ac5362b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:46.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.037+0000 7f5820c86640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e2190 0x55ac5362b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:34260/0 (socket says 172.21.15.161:34260) 2024-03-28T23:36:46.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.037+0000 7f5820c86640 1 -- 172.21.15.161:0/2975590343 learned_addr learned my addr 172.21.15.161:0/2975590343 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:46.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.037+0000 7f5821c88640 1 --2- 172.21.15.161:0/2975590343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ac53605120 0x55ac535ffe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:46.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.037+0000 7f5821487640 1 --2- 172.21.15.161:0/2975590343 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ac535e1db0 0x55ac5362a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:46.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.037+0000 7f5820c86640 1 -- 172.21.15.161:0/2975590343 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ac535e1db0 msgr2=0x55ac5362a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:46.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.037+0000 7f5820c86640 1 --2- 172.21.15.161:0/2975590343 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ac535e1db0 0x55ac5362a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:46.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.037+0000 7f5820c86640 1 -- 172.21.15.161:0/2975590343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ac53605120 msgr2=0x55ac535ffe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:46.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.037+0000 7f5820c86640 1 --2- 172.21.15.161:0/2975590343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ac53605120 0x55ac535ffe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:46.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.037+0000 7f5820c86640 1 -- 172.21.15.161:0/2975590343 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac535e0df0 con 0x55ac535e2190 2024-03-28T23:36:46.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.037+0000 7f5820c86640 1 --2- 172.21.15.161:0/2975590343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e2190 0x55ac5362b1b0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f580c087bd0 tx=0x7f580c001150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5944f8969926412b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:46.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.037+0000 7f57fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:36:46.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.038+0000 7f57fbfff640 1 -- 172.21.15.161:0/2975590343 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580c074040 con 0x55ac535e2190 2024-03-28T23:36:46.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.038+0000 7f57fbfff640 1 -- 172.21.15.161:0/2975590343 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7f580c07bdc0 con 0x55ac535e2190 2024-03-28T23:36:46.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.038+0000 7f58263a5740 1 -- 172.21.15.161:0/2975590343 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f580c086070 con 0x55ac535e2190 2024-03-28T23:36:46.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.038+0000 7f57fbfff640 1 -- 172.21.15.161:0/2975590343 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) ==== 24+0+0 (secure 0 0 0) 0x7f580c054530 con 0x55ac535e2190 2024-03-28T23:36:46.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.038+0000 7f58263a5740 10 client.9360 fetch_fsmap learned FSMap version 154 2024-03-28T23:36:46.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.038+0000 7f58263a5740 1 -- 172.21.15.161:0/2975590343 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) -- 0x55ac535e0df0 con 0x55ac535e2190 2024-03-28T23:36:46.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.038+0000 7f57fbfff640 1 -- 172.21.15.161:0/2975590343 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f580c086070 con 0x55ac535e2190 2024-03-28T23:36:46.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.038+0000 7f57fbfff640 1 -- 172.21.15.161:0/2975590343 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 154) ==== 4139+0+0 (secure 0 0 0) 0x7f580c080070 con 0x55ac535e2190 2024-03-28T23:36:46.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.038+0000 7f58263a5740 10 client.9360 fetch_fsmap finished waiting for FSMap version 154 2024-03-28T23:36:46.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.038+0000 7f58263a5740 10 client.9360 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:36:46.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.038+0000 7f58263a5740 20 client.9360 populate_metadata read hostname 'smithi161' 2024-03-28T23:36:46.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.038+0000 7f58263a5740 1 --2- 172.21.15.161:0/2975590343 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x55ac53694aa0 0x55ac53694e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:46.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.038+0000 7f58263a5740 4 client.9360 mds_command: new command op to 7283 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:36:46.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.038+0000 7f58263a5740 1 -- 172.21.15.161:0/2975590343 --> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] -- command(tid 0: {"prefix": "status"}) -- 0x55ac5354d380 con 0x55ac53694aa0 2024-03-28T23:36:46.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.038+0000 7f5821487640 1 --2- 172.21.15.161:0/2975590343 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x55ac53694aa0 0x55ac53694e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:46.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.039+0000 7f57fbfff640 10 client.9360 ms_handle_connect on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:46.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.039+0000 7f5821487640 1 --2- 172.21.15.161:0/2975590343 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x55ac53694aa0 0x55ac53694e80 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17ff7ffe2e5a9556 server_cookie=d19df0c8e9e1dc9b in_seq=0 out_seq=0 2024-03-28T23:36:46.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.039+0000 7f57fbfff640 1 -- 172.21.15.161:0/2975590343 <== mds.0 v2:172.21.15.170:6832/2996092857 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55ac5354d380 con 0x55ac53694aa0 2024-03-28T23:36:46.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.039+0000 7f57fbfff640 10 client.9360 handle_command_reply: tid=0 2024-03-28T23:36:46.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.039+0000 7f58263a5740 1 client.9360 shutdown 2024-03-28T23:36:46.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.039+0000 7f58263a5740 1 -- 172.21.15.161:0/2975590343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e2190 msgr2=0x55ac5362b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:46.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.039+0000 7f58263a5740 1 --2- 172.21.15.161:0/2975590343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e2190 0x55ac5362b1b0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f580c087bd0 tx=0x7f580c001150 comp rx=0 tx=0).stop 2024-03-28T23:36:46.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.039+0000 7f58263a5740 1 -- 172.21.15.161:0/2975590343 shutdown_connections 2024-03-28T23:36:46.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.039+0000 7f58263a5740 1 --2- 172.21.15.161:0/2975590343 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x55ac53694aa0 0x55ac53694e80 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:46.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.039+0000 7f58263a5740 1 --2- 172.21.15.161:0/2975590343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ac535e2190 0x55ac5362b1b0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:46.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.039+0000 7f58263a5740 1 --2- 172.21.15.161:0/2975590343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ac53605120 0x55ac535ffe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:46.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.039+0000 7f57fbfff640 0 client.9360 ms_handle_reset on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:46.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.039+0000 7f58263a5740 1 --2- 172.21.15.161:0/2975590343 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ac535e1db0 0x55ac5362a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:46.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.039+0000 7f58263a5740 1 -- 172.21.15.161:0/2975590343 >> 172.21.15.161:0/2975590343 conn(0x55ac535e0310 msgr2=0x55ac535e2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:46.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.039+0000 7f58263a5740 1 -- 172.21.15.161:0/2975590343 shutdown_connections 2024-03-28T23:36:46.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.039+0000 7f58263a5740 1 -- 172.21.15.161:0/2975590343 wait complete. 2024-03-28T23:36:46.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.039+0000 7f58263a5740 20 client.9360 trim_cache size 0 max 16384 2024-03-28T23:36:46.046 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:36:46.047 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:36:46.047 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:36:46.047 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 7283, 2024-03-28T23:36:46.047 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:36:46.047 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:36:46.047 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:36:46.047 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 28.606690665999999, 2024-03-28T23:36:46.047 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 154, 2024-03-28T23:36:46.047 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 161, 2024-03-28T23:36:46.047 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 159, 2024-03-28T23:36:46.047 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 31.59651538 2024-03-28T23:36:46.047 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:36:46.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.040+0000 7f58236eb640 1 -- 172.21.15.161:0/2506496425 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f580006b4d0 msgr2=0x7f580006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:46.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.040+0000 7f58236eb640 1 --2- 172.21.15.161:0/2506496425 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f580006b4d0 0x7f580006d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f5818036000 tx=0x7f581803d040 comp rx=0 tx=0).stop 2024-03-28T23:36:46.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.040+0000 7f58236eb640 1 -- 172.21.15.161:0/2506496425 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f581c06d9f0 msgr2=0x7f581c141a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:46.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.040+0000 7f58236eb640 1 --2- 172.21.15.161:0/2506496425 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f581c06d9f0 0x7f581c141a70 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f580c00c500 tx=0x7f580c00c9c0 comp rx=0 tx=0).stop 2024-03-28T23:36:46.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.040+0000 7f58236eb640 1 -- 172.21.15.161:0/2506496425 shutdown_connections 2024-03-28T23:36:46.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.040+0000 7f58236eb640 1 --2- 172.21.15.161:0/2506496425 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f580006b4d0 0x7f580006d990 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:46.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.040+0000 7f58236eb640 1 --2- 172.21.15.161:0/2506496425 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f581c06d9f0 0x7f581c141a70 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:46.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.040+0000 7f58236eb640 1 --2- 172.21.15.161:0/2506496425 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f581c152a80 0x7f581c0ae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:46.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.040+0000 7f58236eb640 1 --2- 172.21.15.161:0/2506496425 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f581c06cb80 0x7f581c0b0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:46.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.040+0000 7f58236eb640 1 -- 172.21.15.161:0/2506496425 >> 172.21.15.161:0/2506496425 conn(0x7f581c0ac720 msgr2=0x7f581c0c9130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:46.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.040+0000 7f58236eb640 1 -- 172.21.15.161:0/2506496425 shutdown_connections 2024-03-28T23:36:46.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:46.040+0000 7f58236eb640 1 -- 172.21.15.161:0/2506496425 wait complete. 2024-03-28T23:36:46.051 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 11, "epoch": 154, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 11, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:35:58.406403+0000", "damaged": [], "data_pools": [ 24 ], "enabled": true, "epoch": 154, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7283": { "addr": "172.21.15.170:6834/2996092857", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 2996092857, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 2996092857, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7283, "incarnation": 151, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 159, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 23, "modified": "2024-03-28T23:36:20.440473+0000", "qdb_cluster": [ 7283 ], "qdb_leader": 7283, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7283 } } }, { "id": 12, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:36:07.448976+0000", "damaged": [], "data_pools": [ 26 ], "enabled": true, "epoch": 149, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs2", "in": [ 0 ], "info": { "gid_7061": { "addr": "172.21.15.161:6836/4026332703", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 4026332703, "type": "v2" }, { "addr": "172.21.15.161:6836", "nonce": 4026332703, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7061, "incarnation": 145, "join_fscid": 12, "name": "c", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 157, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 25, "modified": "2024-03-28T23:36:16.420228+0000", "qdb_cluster": [ 7061 ], "qdb_leader": 7061, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7061 } } } ], "standbys": [ { "addr": "172.21.15.170:6835/557522063", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 557522063, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 557522063, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 137, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7070, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6837/789976100", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6835", "nonce": 789976100, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 789976100, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 152, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9216, "incarnation": 0, "join_fscid": 12, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:36:46.052 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': 12, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 12, 'name': 'c'}, {'join_fscid': -1, 'name': 'd', 'state': 'up:active'}] 2024-03-28T23:36:46.053 INFO:tasks.cephfs.test_failover:current = [{'gid': 9216, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6837/789976100', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 789976100}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 789976100}]}, 'join_fscid': 12, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 152}, {'gid': 7283, 'name': 'b', 'rank': 0, 'incarnation': 151, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.170:6834/2996092857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 2996092857}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 2996092857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 7061, 'name': 'c', 'rank': 0, 'incarnation': 145, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.161:6836/4026332703', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 4026332703}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 4026332703}]}, 'join_fscid': 12, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 7070, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/557522063', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 557522063}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 557522063}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 137}] 2024-03-28T23:36:46.053 DEBUG:tasks.cephfs.test_failover:'up:active' != 'up:standby' - up:active + up:standby 2024-03-28T23:36:46.053 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-28T23:36:51.054 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:36:51.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.155+0000 7fbf6c7e9640 1 Processor -- start 2024-03-28T23:36:51.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.155+0000 7fbf6c7e9640 1 -- start start 2024-03-28T23:36:51.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.156+0000 7fbf6c7e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf640c57e0 0x7fbf6406c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:51.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.156+0000 7fbf6c7e9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf640c7ed0 con 0x7fbf641528b0 2024-03-28T23:36:51.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.156+0000 7fbf6c7e9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf640c5c90 con 0x7fbf640c53a0 2024-03-28T23:36:51.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.156+0000 7fbf6c7e9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbf64155f90 con 0x7fbf640c57e0 2024-03-28T23:36:51.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.156+0000 7fbf6a585640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fbf641528b0 0x7fbf640c8920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37404/0 (socket says 172.21.15.161:37404) 2024-03-28T23:36:51.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.156+0000 7fbf6a585640 1 -- 172.21.15.161:0/1585758644 learned_addr learned my addr 172.21.15.161:0/1585758644 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:51.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.156+0000 7fbf6ad86640 1 --2- 172.21.15.161:0/1585758644 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf640c57e0 0x7fbf6406c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:51.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.156+0000 7fbf69583640 1 -- 172.21.15.161:0/1585758644 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3439833634 0 0) 0x7fbf640c7ed0 con 0x7fbf641528b0 2024-03-28T23:36:51.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.156+0000 7fbf69583640 1 -- 172.21.15.161:0/1585758644 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf50003520 con 0x7fbf641528b0 2024-03-28T23:36:51.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.156+0000 7fbf69583640 1 -- 172.21.15.161:0/1585758644 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1807301453 0 0) 0x7fbf640c5c90 con 0x7fbf640c53a0 2024-03-28T23:36:51.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.156+0000 7fbf69583640 1 -- 172.21.15.161:0/1585758644 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf640c7ed0 con 0x7fbf640c53a0 2024-03-28T23:36:51.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.156+0000 7fbf6ad86640 1 -- 172.21.15.161:0/1585758644 >> v1:172.21.15.170:6789/0 conn(0x7fbf640c53a0 legacy=0x7fbf6406c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:51.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.156+0000 7fbf6ad86640 1 -- 172.21.15.161:0/1585758644 >> v1:172.21.15.161:6789/0 conn(0x7fbf641528b0 legacy=0x7fbf640c8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:51.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.157+0000 7fbf6ad86640 1 -- 172.21.15.161:0/1585758644 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf64152c90 con 0x7fbf640c57e0 2024-03-28T23:36:51.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.158+0000 7fbf6ad86640 1 --2- 172.21.15.161:0/1585758644 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf640c57e0 0x7fbf6406c850 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c009dc0 tx=0x7fbf5c02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d26b27d1a85d1f62 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:51.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.158+0000 7fbf69583640 1 -- 172.21.15.161:0/1585758644 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c039070 con 0x7fbf640c57e0 2024-03-28T23:36:51.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.158+0000 7fbf69583640 1 -- 172.21.15.161:0/1585758644 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf5c02cc40 con 0x7fbf640c57e0 2024-03-28T23:36:51.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.158+0000 7fbf69583640 1 -- 172.21.15.161:0/1585758644 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c034440 con 0x7fbf640c57e0 2024-03-28T23:36:51.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.158+0000 7fbf6c7e9640 1 -- 172.21.15.161:0/1585758644 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf640c57e0 msgr2=0x7fbf6406c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:51.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.158+0000 7fbf6c7e9640 1 --2- 172.21.15.161:0/1585758644 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf640c57e0 0x7fbf6406c850 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c009dc0 tx=0x7fbf5c02be90 comp rx=0 tx=0).stop 2024-03-28T23:36:51.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.159+0000 7fbf6c7e9640 1 -- 172.21.15.161:0/1585758644 shutdown_connections 2024-03-28T23:36:51.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.159+0000 7fbf6c7e9640 1 --2- 172.21.15.161:0/1585758644 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf640c57e0 0x7fbf6406c850 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:51.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.159+0000 7fbf6c7e9640 1 -- 172.21.15.161:0/1585758644 >> 172.21.15.161:0/1585758644 conn(0x7fbf640ac5b0 msgr2=0x7fbf640ac9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:51.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.159+0000 7fbf6c7e9640 1 -- 172.21.15.161:0/1585758644 shutdown_connections 2024-03-28T23:36:51.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.159+0000 7fbf6c7e9640 1 -- 172.21.15.161:0/1585758644 wait complete. 2024-03-28T23:36:51.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.160+0000 7fbf6c7e9640 1 Processor -- start 2024-03-28T23:36:51.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.160+0000 7fbf6c7e9640 1 -- start start 2024-03-28T23:36:51.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.160+0000 7fbf6c7e9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf640c53a0 0x7fbf641eb480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:51.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.160+0000 7fbf6c7e9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf640c57e0 0x7fbf641566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:51.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.160+0000 7fbf6c7e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf641528b0 0x7fbf641512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:51.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.160+0000 7fbf6c7e9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbf64155940 con 0x7fbf640c57e0 2024-03-28T23:36:51.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.160+0000 7fbf6c7e9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbf64155660 con 0x7fbf640c53a0 2024-03-28T23:36:51.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.160+0000 7fbf6c7e9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbf64155c20 con 0x7fbf641528b0 2024-03-28T23:36:51.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.160+0000 7fbf69d84640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf640c57e0 0x7fbf641566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:51.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.160+0000 7fbf69d84640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf640c57e0 0x7fbf641566d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53084/0 (socket says 172.21.15.161:53084) 2024-03-28T23:36:51.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.160+0000 7fbf69d84640 1 -- 172.21.15.161:0/3365376971 learned_addr learned my addr 172.21.15.161:0/3365376971 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:51.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.160+0000 7fbf6a585640 1 --2- 172.21.15.161:0/3365376971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf640c53a0 0x7fbf641eb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:51.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.160+0000 7fbf6ad86640 1 --2- 172.21.15.161:0/3365376971 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf641528b0 0x7fbf641512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:51.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.161+0000 7fbf69d84640 1 -- 172.21.15.161:0/3365376971 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf641528b0 msgr2=0x7fbf641512f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:51.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.161+0000 7fbf69d84640 1 --2- 172.21.15.161:0/3365376971 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf641528b0 0x7fbf641512f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:51.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.161+0000 7fbf69d84640 1 -- 172.21.15.161:0/3365376971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf640c53a0 msgr2=0x7fbf641eb480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:51.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.161+0000 7fbf69d84640 1 --2- 172.21.15.161:0/3365376971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf640c53a0 0x7fbf641eb480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:51.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.161+0000 7fbf69d84640 1 -- 172.21.15.161:0/3365376971 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf64156c10 con 0x7fbf640c57e0 2024-03-28T23:36:51.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.161+0000 7fbf69d84640 1 --2- 172.21.15.161:0/3365376971 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf640c57e0 0x7fbf641566d0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7fbf6000c390 tx=0x7fbf6000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:51.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.161+0000 7fbf5affd640 1 -- 172.21.15.161:0/3365376971 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf60017070 con 0x7fbf640c57e0 2024-03-28T23:36:51.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.161+0000 7fbf5affd640 1 -- 172.21.15.161:0/3365376971 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf600052d0 con 0x7fbf640c57e0 2024-03-28T23:36:51.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.161+0000 7fbf5affd640 1 -- 172.21.15.161:0/3365376971 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf60003810 con 0x7fbf640c57e0 2024-03-28T23:36:51.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.161+0000 7fbf6c7e9640 1 -- 172.21.15.161:0/3365376971 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf641542f0 con 0x7fbf640c57e0 2024-03-28T23:36:51.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.161+0000 7fbf6c7e9640 1 -- 172.21.15.161:0/3365376971 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf64153210 con 0x7fbf640c57e0 2024-03-28T23:36:51.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.162+0000 7fbf6c7e9640 1 -- 172.21.15.161:0/3365376971 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf6404f4b0 con 0x7fbf640c57e0 2024-03-28T23:36:51.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.163+0000 7fbf5affd640 1 -- 172.21.15.161:0/3365376971 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbf6001a070 con 0x7fbf640c57e0 2024-03-28T23:36:51.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.163+0000 7fbf5affd640 1 --2- 172.21.15.161:0/3365376971 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf4806b4d0 0x7fbf4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:51.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.163+0000 7fbf5affd640 1 -- 172.21.15.161:0/3365376971 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7fbf60089b60 con 0x7fbf640c57e0 2024-03-28T23:36:51.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.163+0000 7fbf6a585640 1 --2- 172.21.15.161:0/3365376971 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf4806b4d0 0x7fbf4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:51.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.164+0000 7fbf6a585640 1 --2- 172.21.15.161:0/3365376971 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf4806b4d0 0x7fbf4806d990 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fbf540011e0 tx=0x7fbf54007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:51.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.168+0000 7fbf5affd640 1 -- 172.21.15.161:0/3365376971 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbf60057d20 con 0x7fbf640c57e0 2024-03-28T23:36:51.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.522+0000 7fbf6c7e9640 1 -- 172.21.15.161:0/3365376971 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf640c80d0 con 0x7fbf640c57e0 2024-03-28T23:36:51.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.523+0000 7fbf5affd640 1 -- 172.21.15.161:0/3365376971 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+6520 (secure 0 0 0) 0x7fbf6005bdc0 con 0x7fbf640c57e0 2024-03-28T23:36:51.534 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:51.534 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":154,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/557522063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":557522063},{"type":"v1","addr":"172.21.15.170:6835","nonce":557522063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":9216,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/789976100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":789976100},{"type":"v1","addr":"172.21.15.161:6837","nonce":789976100}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:58.406403+0000","modified":"2024-03-28T23:36:20.440473+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":159,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7283":{"gid":7283,"name":"b","rank":0,"incarnation":151,"state":"up:active","state_seq":4,"addr":"172.21.15.170:6834/2996092857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":2996092857},{"type":"v1","addr":"172.21.15.170:6834","nonce":2996092857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7283,"qdb_cluster":[7283]},"id":11},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:36:07.448976+0000","modified":"2024-03-28T23:36:16.420228+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7061":{"gid":7061,"name":"c","rank":0,"incarnation":145,"state":"up:active","state_seq":9,"addr":"172.21.15.161:6836/4026332703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4026332703},{"type":"v1","addr":"172.21.15.161:6836","nonce":4026332703}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7061,"qdb_cluster":[7061]},"id":12}]} 2024-03-28T23:36:51.535 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 154 2024-03-28T23:36:51.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.525+0000 7fbf6c7e9640 1 -- 172.21.15.161:0/3365376971 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf4806b4d0 msgr2=0x7fbf4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:51.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.525+0000 7fbf6c7e9640 1 --2- 172.21.15.161:0/3365376971 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf4806b4d0 0x7fbf4806d990 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fbf540011e0 tx=0x7fbf54007d30 comp rx=0 tx=0).stop 2024-03-28T23:36:51.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.525+0000 7fbf6c7e9640 1 -- 172.21.15.161:0/3365376971 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf640c57e0 msgr2=0x7fbf641566d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:51.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.525+0000 7fbf6c7e9640 1 --2- 172.21.15.161:0/3365376971 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf640c57e0 0x7fbf641566d0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7fbf6000c390 tx=0x7fbf6000c850 comp rx=0 tx=0).stop 2024-03-28T23:36:51.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.525+0000 7fbf6c7e9640 1 -- 172.21.15.161:0/3365376971 shutdown_connections 2024-03-28T23:36:51.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.525+0000 7fbf6c7e9640 1 --2- 172.21.15.161:0/3365376971 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbf4806b4d0 0x7fbf4806d990 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:51.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.525+0000 7fbf6c7e9640 1 --2- 172.21.15.161:0/3365376971 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf640c57e0 0x7fbf641566d0 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:51.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.525+0000 7fbf6c7e9640 1 --2- 172.21.15.161:0/3365376971 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf641528b0 0x7fbf641512f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:51.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.525+0000 7fbf6c7e9640 1 --2- 172.21.15.161:0/3365376971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf640c53a0 0x7fbf641eb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:51.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.525+0000 7fbf6c7e9640 1 -- 172.21.15.161:0/3365376971 >> 172.21.15.161:0/3365376971 conn(0x7fbf640ac5b0 msgr2=0x7fbf640c8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:51.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.525+0000 7fbf6c7e9640 1 -- 172.21.15.161:0/3365376971 shutdown_connections 2024-03-28T23:36:51.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:51.525+0000 7fbf6c7e9640 1 -- 172.21.15.161:0/3365376971 wait complete. 2024-03-28T23:36:51.939 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 154, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:35:58.406403+0000', 'modified': '2024-03-28T23:36:20.440473+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 159, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7283}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7283': {'gid': 7283, 'name': 'b', 'rank': 0, 'incarnation': 151, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.170:6834/2996092857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 2996092857}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 2996092857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7283, 'qdb_cluster': [7283]} 2024-03-28T23:36:51.939 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:36:51.940 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T23:36:52.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.042+0000 7f2043eaa640 1 Processor -- start 2024-03-28T23:36:52.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.043+0000 7f2043eaa640 1 -- start start 2024-03-28T23:36:52.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.043+0000 7f2043eaa640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f203c152890 0x7f203c0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:52.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.043+0000 7f2043eaa640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f203c0c8ed0 0x7f203c0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:52.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.043+0000 7f2043eaa640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f203c158a30 con 0x7f203c0c5950 2024-03-28T23:36:52.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.043+0000 7f2043eaa640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f203c0c8480 con 0x7f203c0c8ed0 2024-03-28T23:36:52.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.043+0000 7f2043eaa640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f203c159640 con 0x7f203c152890 2024-03-28T23:36:52.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.043+0000 7f2041445640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f203c0c8ed0 0x7f203c0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:52.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.043+0000 7f2041445640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f203c0c8ed0 0x7f203c0c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53088/0 (socket says 172.21.15.161:53088) 2024-03-28T23:36:52.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.043+0000 7f2041445640 1 -- 172.21.15.161:0/1624954178 learned_addr learned my addr 172.21.15.161:0/1624954178 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:52.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.043+0000 7f2041c46640 1 --2- 172.21.15.161:0/1624954178 >> v2:172.21.15.170:3300/0 conn(0x7f203c152890 0x7f203c0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:52.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.044+0000 7f2041445640 1 -- 172.21.15.161:0/1624954178 >> v2:172.21.15.170:3300/0 conn(0x7f203c152890 msgr2=0x7f203c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:52.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.044+0000 7f2041445640 1 --2- 172.21.15.161:0/1624954178 >> v2:172.21.15.170:3300/0 conn(0x7f203c152890 0x7f203c0c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.044+0000 7f2041445640 1 -- 172.21.15.161:0/1624954178 >> v1:172.21.15.170:6789/0 conn(0x7f203c0c5950 legacy=0x7f203c1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:52.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.044+0000 7f2041445640 1 -- 172.21.15.161:0/1624954178 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f203c158d10 con 0x7f203c0c8ed0 2024-03-28T23:36:52.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.045+0000 7f2041445640 1 --2- 172.21.15.161:0/1624954178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f203c0c8ed0 0x7f203c0c5410 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f203c08cc80 tx=0x7f202c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f925ea536be24856 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:52.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.045+0000 7f2040c44640 1 -- 172.21.15.161:0/1624954178 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202c037070 con 0x7f203c0c8ed0 2024-03-28T23:36:52.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.045+0000 7f2040c44640 1 -- 172.21.15.161:0/1624954178 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f202c004230 con 0x7f203c0c8ed0 2024-03-28T23:36:52.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.045+0000 7f2040c44640 1 -- 172.21.15.161:0/1624954178 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202c003610 con 0x7f203c0c8ed0 2024-03-28T23:36:52.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.046+0000 7f2043eaa640 1 -- 172.21.15.161:0/1624954178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f203c0c8ed0 msgr2=0x7f203c0c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:52.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.046+0000 7f2043eaa640 1 --2- 172.21.15.161:0/1624954178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f203c0c8ed0 0x7f203c0c5410 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f203c08cc80 tx=0x7f202c02b860 comp rx=0 tx=0).stop 2024-03-28T23:36:52.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.046+0000 7f2043eaa640 1 -- 172.21.15.161:0/1624954178 shutdown_connections 2024-03-28T23:36:52.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.046+0000 7f2043eaa640 1 --2- 172.21.15.161:0/1624954178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f203c0c8ed0 0x7f203c0c5410 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.046+0000 7f2043eaa640 1 --2- 172.21.15.161:0/1624954178 >> v2:172.21.15.170:3300/0 conn(0x7f203c152890 0x7f203c0c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.046+0000 7f2043eaa640 1 -- 172.21.15.161:0/1624954178 >> 172.21.15.161:0/1624954178 conn(0x7f203c0ac620 msgr2=0x7f203c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:52.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.046+0000 7f2043eaa640 1 -- 172.21.15.161:0/1624954178 shutdown_connections 2024-03-28T23:36:52.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.046+0000 7f2043eaa640 1 -- 172.21.15.161:0/1624954178 wait complete. 2024-03-28T23:36:52.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.047+0000 7f2043eaa640 1 Processor -- start 2024-03-28T23:36:52.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.047+0000 7f2043eaa640 1 -- start start 2024-03-28T23:36:52.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.047+0000 7f2043eaa640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f203c0c5950 0x7f203c157960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:52.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.047+0000 7f2043eaa640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f203c0c8ed0 0x7f203c1ea830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:52.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.047+0000 7f2043eaa640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f203c152890 0x7f203c1ead70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:52.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.047+0000 7f2043eaa640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f203c152c70 con 0x7f203c0c8ed0 2024-03-28T23:36:52.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.047+0000 7f2043eaa640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f203c1592d0 con 0x7f203c152890 2024-03-28T23:36:52.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.047+0000 7f2043eaa640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f203c152df0 con 0x7f203c0c5950 2024-03-28T23:36:52.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.047+0000 7f2041445640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f203c0c8ed0 0x7f203c1ea830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:52.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.048+0000 7f2041445640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f203c0c8ed0 0x7f203c1ea830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53100/0 (socket says 172.21.15.161:53100) 2024-03-28T23:36:52.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.048+0000 7f2041445640 1 -- 172.21.15.161:0/3144453043 learned_addr learned my addr 172.21.15.161:0/3144453043 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:52.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.048+0000 7f2041c46640 1 --2- 172.21.15.161:0/3144453043 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f203c0c5950 0x7f203c157960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:52.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.048+0000 7f2042447640 1 --2- 172.21.15.161:0/3144453043 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f203c152890 0x7f203c1ead70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:52.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.048+0000 7f2041445640 1 -- 172.21.15.161:0/3144453043 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f203c0c5950 msgr2=0x7f203c157960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:52.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.048+0000 7f2041445640 1 --2- 172.21.15.161:0/3144453043 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f203c0c5950 0x7f203c157960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.048+0000 7f2041445640 1 -- 172.21.15.161:0/3144453043 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f203c152890 msgr2=0x7f203c1ead70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:52.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.048+0000 7f2041445640 1 --2- 172.21.15.161:0/3144453043 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f203c152890 0x7f203c1ead70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.048+0000 7f2041445640 1 -- 172.21.15.161:0/3144453043 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f203c14da00 con 0x7f203c0c8ed0 2024-03-28T23:36:52.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.048+0000 7f2041445640 1 --2- 172.21.15.161:0/3144453043 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f203c0c8ed0 0x7f203c1ea830 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f202c02b8a0 tx=0x7f202c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:52.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.048+0000 7f20327fc640 1 -- 172.21.15.161:0/3144453043 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202c037050 con 0x7f203c0c8ed0 2024-03-28T23:36:52.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.048+0000 7f20327fc640 1 -- 172.21.15.161:0/3144453043 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f202c043070 con 0x7f203c0c8ed0 2024-03-28T23:36:52.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.048+0000 7f2043eaa640 1 -- 172.21.15.161:0/3144453043 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f203c14dea0 con 0x7f203c0c8ed0 2024-03-28T23:36:52.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.048+0000 7f20327fc640 1 -- 172.21.15.161:0/3144453043 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202c003d80 con 0x7f203c0c8ed0 2024-03-28T23:36:52.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.048+0000 7f2043eaa640 1 -- 172.21.15.161:0/3144453043 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f203c1eb430 con 0x7f203c0c8ed0 2024-03-28T23:36:52.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.050+0000 7f20327fc640 1 -- 172.21.15.161:0/3144453043 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f202c006040 con 0x7f203c0c8ed0 2024-03-28T23:36:52.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.051+0000 7f20327fc640 1 --2- 172.21.15.161:0/3144453043 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f202006b4d0 0x7f202006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:52.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.051+0000 7f2041c46640 1 --2- 172.21.15.161:0/3144453043 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f202006b4d0 0x7f202006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:52.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.051+0000 7f20327fc640 1 -- 172.21.15.161:0/3144453043 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7f202c0acbe0 con 0x7f203c0c8ed0 2024-03-28T23:36:52.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.051+0000 7f2046a85740 1 Processor -- start 2024-03-28T23:36:52.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.051+0000 7f2041c46640 1 --2- 172.21.15.161:0/3144453043 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f202006b4d0 0x7f202006d990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f203c158600 tx=0x7f2038009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:52.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.051+0000 7f2046a85740 1 -- start start 2024-03-28T23:36:52.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.052+0000 7f2046a85740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563f9070fdb0 0x563f9070d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:52.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.052+0000 7f2046a85740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563f90710190 0x563f9070ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:52.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.052+0000 7f2046a85740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f9070e310 0x563f9070a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:52.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.052+0000 7f2046a85740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x563f90673af0 con 0x563f9070e310 2024-03-28T23:36:52.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.052+0000 7f2046a85740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x563f9061c9d0 con 0x563f9070fdb0 2024-03-28T23:36:52.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.052+0000 7f2046a85740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x563f9066fb90 con 0x563f90710190 2024-03-28T23:36:52.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.052+0000 7f2042447640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f9070e310 0x563f9070a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:52.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.052+0000 7f2041c46640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563f9070fdb0 0x563f9070d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:52.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.052+0000 7f2042447640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f9070e310 0x563f9070a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53114/0 (socket says 172.21.15.161:53114) 2024-03-28T23:36:52.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.052+0000 7f2042447640 1 -- 172.21.15.161:0/1828605087 learned_addr learned my addr 172.21.15.161:0/1828605087 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:52.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.052+0000 7f2041c46640 1 --2- 172.21.15.161:0/1828605087 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563f9070fdb0 0x563f9070d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:44082/0 (socket says 172.21.15.161:44082) 2024-03-28T23:36:52.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.052+0000 7f2041445640 1 --2- 172.21.15.161:0/1828605087 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563f90710190 0x563f9070ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:52.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.053+0000 7f2042447640 1 -- 172.21.15.161:0/1828605087 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563f90710190 msgr2=0x563f9070ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:52.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.053+0000 7f2042447640 1 --2- 172.21.15.161:0/1828605087 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563f90710190 0x563f9070ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.053+0000 7f2042447640 1 -- 172.21.15.161:0/1828605087 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563f9070fdb0 msgr2=0x563f9070d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:52.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.053+0000 7f2042447640 1 --2- 172.21.15.161:0/1828605087 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563f9070fdb0 0x563f9070d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.053+0000 7f2042447640 1 -- 172.21.15.161:0/1828605087 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f903882d0 con 0x563f9070e310 2024-03-28T23:36:52.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.053+0000 7f2041445640 1 --2- 172.21.15.161:0/1828605087 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563f90710190 0x563f9070ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:52.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.053+0000 7f2041c46640 1 --2- 172.21.15.161:0/1828605087 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563f9070fdb0 0x563f9070d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:52.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.053+0000 7f2042447640 1 --2- 172.21.15.161:0/1828605087 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f9070e310 0x563f9070a4e0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f2034001e90 tx=0x7f20340015d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97af9212b08259a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:52.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.053+0000 7f201f7fe640 1 -- 172.21.15.161:0/1828605087 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20340020a0 con 0x563f9070e310 2024-03-28T23:36:52.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.053+0000 7f201f7fe640 1 -- 172.21.15.161:0/1828605087 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2034018070 con 0x563f9070e310 2024-03-28T23:36:52.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.053+0000 7f201f7fe640 1 -- 172.21.15.161:0/1828605087 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2034003b70 con 0x563f9070e310 2024-03-28T23:36:52.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.053+0000 7f2046a85740 1 -- 172.21.15.161:0/1828605087 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f9070e310 msgr2=0x563f9070a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:52.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.053+0000 7f2046a85740 1 --2- 172.21.15.161:0/1828605087 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f9070e310 0x563f9070a4e0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f2034001e90 tx=0x7f20340015d0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.054+0000 7f2046a85740 1 -- 172.21.15.161:0/1828605087 shutdown_connections 2024-03-28T23:36:52.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.054+0000 7f2046a85740 1 --2- 172.21.15.161:0/1828605087 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f9070e310 0x563f9070a4e0 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.054+0000 7f2046a85740 1 --2- 172.21.15.161:0/1828605087 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563f90710190 0x563f9070ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.054+0000 7f2046a85740 1 --2- 172.21.15.161:0/1828605087 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563f9070fdb0 0x563f9070d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.054+0000 7f2046a85740 1 -- 172.21.15.161:0/1828605087 >> 172.21.15.161:0/1828605087 conn(0x563f9070c630 msgr2=0x563f90731220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:52.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.054+0000 7f2046a85740 1 -- 172.21.15.161:0/1828605087 shutdown_connections 2024-03-28T23:36:52.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.054+0000 7f2046a85740 1 -- 172.21.15.161:0/1828605087 wait complete. 2024-03-28T23:36:52.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.054+0000 7f2046a85740 1 -- start start 2024-03-28T23:36:52.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.054+0000 7f2046a85740 1 Processor -- start 2024-03-28T23:36:52.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.055+0000 7f2046a85740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563f9070fdb0 0x563f9075f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:52.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.055+0000 7f2046a85740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f90710190 0x563f907361c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:52.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.055+0000 7f2046a85740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563f90734300 0x563f9072cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:52.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.055+0000 7f2046a85740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x563f90732090 con 0x563f90710190 2024-03-28T23:36:52.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.055+0000 7f2046a85740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x563f9062ff70 con 0x563f90734300 2024-03-28T23:36:52.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.055+0000 7f2046a85740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x563f90731ad0 con 0x563f9070fdb0 2024-03-28T23:36:52.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.055+0000 7f2046a85740 10 client.0 _ll_register_callbacks cb 0x563f90387ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:36:52.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.055+0000 7f2041445640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f90710190 0x563f907361c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:52.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.055+0000 7f2041445640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f90710190 0x563f907361c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53120/0 (socket says 172.21.15.161:53120) 2024-03-28T23:36:52.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.055+0000 7f2041445640 1 -- 172.21.15.161:0/1328118193 learned_addr learned my addr 172.21.15.161:0/1328118193 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:52.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.055+0000 7f2042447640 1 --2- 172.21.15.161:0/1328118193 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563f90734300 0x563f9072cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:52.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.055+0000 7f2041c46640 1 --2- 172.21.15.161:0/1328118193 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563f9070fdb0 0x563f9075f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:52.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.055+0000 7f2041445640 1 -- 172.21.15.161:0/1328118193 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563f9070fdb0 msgr2=0x563f9075f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:52.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.055+0000 7f2041445640 1 --2- 172.21.15.161:0/1328118193 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563f9070fdb0 0x563f9075f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.055+0000 7f2041445640 1 -- 172.21.15.161:0/1328118193 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563f90734300 msgr2=0x563f9072cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:52.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.055+0000 7f2041445640 1 --2- 172.21.15.161:0/1328118193 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563f90734300 0x563f9072cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.055+0000 7f2041445640 1 -- 172.21.15.161:0/1328118193 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f904526b0 con 0x563f90710190 2024-03-28T23:36:52.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.056+0000 7f2041445640 1 --2- 172.21.15.161:0/1328118193 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f90710190 0x563f907361c0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f202c07add0 tx=0x7f202c031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=771b68a8f47c7e45 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:52.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.056+0000 7f201cff9640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:36:52.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.056+0000 7f201cff9640 1 -- 172.21.15.161:0/1328118193 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202c0834a0 con 0x563f90710190 2024-03-28T23:36:52.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.056+0000 7f201cff9640 1 -- 172.21.15.161:0/1328118193 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7f202c07fe00 con 0x563f90710190 2024-03-28T23:36:52.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.056+0000 7f2046a85740 1 -- 172.21.15.161:0/1328118193 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2034018070 con 0x563f90710190 2024-03-28T23:36:52.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.056+0000 7f201cff9640 1 -- 172.21.15.161:0/1328118193 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) ==== 24+0+0 (secure 0 0 0) 0x7f202c0982b0 con 0x563f90710190 2024-03-28T23:36:52.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.056+0000 7f2046a85740 10 client.9378 fetch_fsmap learned FSMap version 154 2024-03-28T23:36:52.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.056+0000 7f2046a85740 1 -- 172.21.15.161:0/1328118193 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) -- 0x563f90733820 con 0x563f90710190 2024-03-28T23:36:52.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.056+0000 7f201cff9640 1 -- 172.21.15.161:0/1328118193 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2034018070 con 0x563f90710190 2024-03-28T23:36:52.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.056+0000 7f201cff9640 1 -- 172.21.15.161:0/1328118193 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 154) ==== 4139+0+0 (secure 0 0 0) 0x7f202c07a290 con 0x563f90710190 2024-03-28T23:36:52.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.056+0000 7f2046a85740 10 client.9378 fetch_fsmap finished waiting for FSMap version 154 2024-03-28T23:36:52.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.056+0000 7f2046a85740 10 client.9378 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:36:52.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.056+0000 7f2046a85740 20 client.9378 populate_metadata read hostname 'smithi161' 2024-03-28T23:36:52.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.056+0000 7f2046a85740 1 --2- 172.21.15.161:0/1328118193 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x563f9075c520 0x563f9075c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:52.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.056+0000 7f2046a85740 4 client.9378 mds_command: new command op to 7283 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:36:52.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.056+0000 7f2046a85740 1 -- 172.21.15.161:0/1328118193 --> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f90681030 con 0x563f9075c520 2024-03-28T23:36:52.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.056+0000 7f2041c46640 1 --2- 172.21.15.161:0/1328118193 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x563f9075c520 0x563f9075c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:52.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.057+0000 7f201cff9640 10 client.9378 ms_handle_connect on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:52.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.057+0000 7f2041c46640 1 --2- 172.21.15.161:0/1328118193 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x563f9075c520 0x563f9075c940 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4437491a028eca7 server_cookie=8e50c612f8b75804 in_seq=0 out_seq=0 2024-03-28T23:36:52.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.057+0000 7f201cff9640 1 -- 172.21.15.161:0/1328118193 <== mds.0 v2:172.21.15.170:6832/2996092857 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563f90681030 con 0x563f9075c520 2024-03-28T23:36:52.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.057+0000 7f201cff9640 10 client.9378 handle_command_reply: tid=0 2024-03-28T23:36:52.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.057+0000 7f2046a85740 1 client.9378 shutdown 2024-03-28T23:36:52.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.058+0000 7f2046a85740 1 -- 172.21.15.161:0/1328118193 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f90710190 msgr2=0x563f907361c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:52.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.058+0000 7f2046a85740 1 --2- 172.21.15.161:0/1328118193 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f90710190 0x563f907361c0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f202c07add0 tx=0x7f202c031000 comp rx=0 tx=0).stop 2024-03-28T23:36:52.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.058+0000 7f2046a85740 1 -- 172.21.15.161:0/1328118193 shutdown_connections 2024-03-28T23:36:52.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.058+0000 7f2046a85740 1 --2- 172.21.15.161:0/1328118193 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x563f9075c520 0x563f9075c940 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.058+0000 7f2046a85740 1 --2- 172.21.15.161:0/1328118193 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f90710190 0x563f907361c0 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.058+0000 7f2046a85740 1 --2- 172.21.15.161:0/1328118193 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563f90734300 0x563f9072cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.058+0000 7f201cff9640 0 client.9378 ms_handle_reset on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:52.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.058+0000 7f2046a85740 1 --2- 172.21.15.161:0/1328118193 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563f9070fdb0 0x563f9075f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.058+0000 7f2046a85740 1 -- 172.21.15.161:0/1328118193 >> 172.21.15.161:0/1328118193 conn(0x563f9070e310 msgr2=0x563f9070e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:52.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.058+0000 7f2046a85740 1 -- 172.21.15.161:0/1328118193 shutdown_connections 2024-03-28T23:36:52.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.058+0000 7f2046a85740 1 -- 172.21.15.161:0/1328118193 wait complete. 2024-03-28T23:36:52.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.058+0000 7f2046a85740 20 client.9378 trim_cache size 0 max 16384 2024-03-28T23:36:52.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.076+0000 7f2046a85740 1 Processor -- start 2024-03-28T23:36:52.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.076+0000 7f2046a85740 1 -- start start 2024-03-28T23:36:52.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.076+0000 7f2046a85740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563f9070fdb0 0x563f9075c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:52.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.076+0000 7f2046a85740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f90710190 0x563f9074f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:52.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.076+0000 7f2046a85740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563f90734300 0x563f9074f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:52.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.076+0000 7f2046a85740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f203c1fb2e0 con 0x563f90710190 2024-03-28T23:36:52.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.076+0000 7f2046a85740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f203c1f8c20 con 0x563f9070fdb0 2024-03-28T23:36:52.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.076+0000 7f2046a85740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f203c1f9270 con 0x563f90734300 2024-03-28T23:36:52.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.076+0000 7f2041445640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f90710190 0x563f9074f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:52.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.076+0000 7f2041445640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f90710190 0x563f9074f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53134/0 (socket says 172.21.15.161:53134) 2024-03-28T23:36:52.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.076+0000 7f2041445640 1 -- 172.21.15.161:0/392368588 learned_addr learned my addr 172.21.15.161:0/392368588 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:52.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.076+0000 7f2042447640 1 --2- 172.21.15.161:0/392368588 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563f90734300 0x563f9074f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:52.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.076+0000 7f2041c46640 1 --2- 172.21.15.161:0/392368588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563f9070fdb0 0x563f9075c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:52.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.077+0000 7f2041445640 1 -- 172.21.15.161:0/392368588 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563f90734300 msgr2=0x563f9074f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:52.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.077+0000 7f2041445640 1 --2- 172.21.15.161:0/392368588 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563f90734300 0x563f9074f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.077+0000 7f2041445640 1 -- 172.21.15.161:0/392368588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563f9070fdb0 msgr2=0x563f9075c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:52.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.077+0000 7f2041445640 1 --2- 172.21.15.161:0/392368588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563f9070fdb0 0x563f9075c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.077+0000 7f2041445640 1 -- 172.21.15.161:0/392368588 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f9075f570 con 0x563f90710190 2024-03-28T23:36:52.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.077+0000 7f2041c46640 1 --2- 172.21.15.161:0/392368588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563f9070fdb0 0x563f9075c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:36:52.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.077+0000 7f2041445640 1 --2- 172.21.15.161:0/392368588 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f90710190 0x563f9074f4b0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f202c000c00 tx=0x7f202c07e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a530b7bfb20a45 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:52.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.077+0000 7f201effd640 1 -- 172.21.15.161:0/392368588 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202c0834a0 con 0x563f90710190 2024-03-28T23:36:52.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.077+0000 7f201effd640 1 -- 172.21.15.161:0/392368588 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f202c038040 con 0x563f90710190 2024-03-28T23:36:52.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.077+0000 7f201effd640 1 -- 172.21.15.161:0/392368588 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202c081100 con 0x563f90710190 2024-03-28T23:36:52.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.077+0000 7f2046a85740 1 -- 172.21.15.161:0/392368588 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f90710190 msgr2=0x563f9074f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:52.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.077+0000 7f2046a85740 1 --2- 172.21.15.161:0/392368588 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f90710190 0x563f9074f4b0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f202c000c00 tx=0x7f202c07e000 comp rx=0 tx=0).stop 2024-03-28T23:36:52.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.077+0000 7f2046a85740 1 -- 172.21.15.161:0/392368588 shutdown_connections 2024-03-28T23:36:52.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.077+0000 7f2046a85740 1 --2- 172.21.15.161:0/392368588 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f90710190 0x563f9074f4b0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.077+0000 7f2046a85740 1 --2- 172.21.15.161:0/392368588 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563f90734300 0x563f9074f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.077+0000 7f2046a85740 1 --2- 172.21.15.161:0/392368588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563f9070fdb0 0x563f9075c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.077+0000 7f2046a85740 1 -- 172.21.15.161:0/392368588 >> 172.21.15.161:0/392368588 conn(0x563f9070e310 msgr2=0x563f9072c890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:52.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.078+0000 7f2046a85740 1 -- 172.21.15.161:0/392368588 shutdown_connections 2024-03-28T23:36:52.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.078+0000 7f2046a85740 1 -- 172.21.15.161:0/392368588 wait complete. 2024-03-28T23:36:52.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.078+0000 7f2046a85740 1 -- start start 2024-03-28T23:36:52.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.078+0000 7f2046a85740 1 Processor -- start 2024-03-28T23:36:52.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.080+0000 7f2046a85740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563f9070fdb0 0x563f90709bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:52.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.080+0000 7f2046a85740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563f90710190 0x563f90729b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:52.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.080+0000 7f2046a85740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f90734300 0x563f9072a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:52.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.080+0000 7f2046a85740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x563f90733b00 con 0x563f90734300 2024-03-28T23:36:52.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.080+0000 7f2046a85740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x563f9075c3a0 con 0x563f90710190 2024-03-28T23:36:52.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.080+0000 7f2046a85740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x563f9072d5e0 con 0x563f9070fdb0 2024-03-28T23:36:52.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.080+0000 7f2046a85740 10 client.0 _ll_register_callbacks cb 0x7f202c0771b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:36:52.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.080+0000 7f2042447640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f90734300 0x563f9072a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:52.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.080+0000 7f2041c46640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563f9070fdb0 0x563f90709bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:52.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.080+0000 7f2042447640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f90734300 0x563f9072a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53140/0 (socket says 172.21.15.161:53140) 2024-03-28T23:36:52.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.080+0000 7f2042447640 1 -- 172.21.15.161:0/1190784734 learned_addr learned my addr 172.21.15.161:0/1190784734 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:52.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.080+0000 7f2041445640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563f90710190 0x563f90729b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:52.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.080+0000 7f2042447640 1 -- 172.21.15.161:0/1190784734 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563f9070fdb0 msgr2=0x563f90709bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:52.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.080+0000 7f2042447640 1 --2- 172.21.15.161:0/1190784734 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563f9070fdb0 0x563f90709bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.080+0000 7f2042447640 1 -- 172.21.15.161:0/1190784734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563f90710190 msgr2=0x563f90729b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:52.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.080+0000 7f2042447640 1 --2- 172.21.15.161:0/1190784734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563f90710190 0x563f90729b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.080+0000 7f2042447640 1 -- 172.21.15.161:0/1190784734 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f907c9ed0 con 0x563f90734300 2024-03-28T23:36:52.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.080+0000 7f2041c46640 1 --2- 172.21.15.161:0/1190784734 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563f9070fdb0 0x563f90709bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:52.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.080+0000 7f2041445640 1 --2- 172.21.15.161:0/1190784734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563f90710190 0x563f90729b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:52.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.080+0000 7f2042447640 1 --2- 172.21.15.161:0/1190784734 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f90734300 0x563f9072a040 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f2034009010 tx=0x7f2034001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a238885c5221322e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:52.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.080+0000 7f201cff9640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:36:52.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.080+0000 7f201cff9640 1 -- 172.21.15.161:0/1190784734 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20340020a0 con 0x563f90734300 2024-03-28T23:36:52.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.080+0000 7f201cff9640 1 -- 172.21.15.161:0/1190784734 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7f2034003b80 con 0x563f90734300 2024-03-28T23:36:52.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.080+0000 7f2046a85740 1 -- 172.21.15.161:0/1190784734 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f202c038040 con 0x563f90734300 2024-03-28T23:36:52.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.081+0000 7f201cff9640 1 -- 172.21.15.161:0/1190784734 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) ==== 24+0+0 (secure 0 0 0) 0x7f203402f050 con 0x563f90734300 2024-03-28T23:36:52.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.081+0000 7f2046a85740 10 client.9384 fetch_fsmap learned FSMap version 154 2024-03-28T23:36:52.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.081+0000 7f2046a85740 1 -- 172.21.15.161:0/1190784734 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) -- 0x563f907c31d0 con 0x563f90734300 2024-03-28T23:36:52.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.081+0000 7f201cff9640 1 -- 172.21.15.161:0/1190784734 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f202c038040 con 0x563f90734300 2024-03-28T23:36:52.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.081+0000 7f201cff9640 1 -- 172.21.15.161:0/1190784734 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 154) ==== 4139+0+0 (secure 0 0 0) 0x7f2034032020 con 0x563f90734300 2024-03-28T23:36:52.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.081+0000 7f2046a85740 10 client.9384 fetch_fsmap finished waiting for FSMap version 154 2024-03-28T23:36:52.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.081+0000 7f2046a85740 10 client.9384 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:36:52.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.081+0000 7f2046a85740 20 client.9384 populate_metadata read hostname 'smithi161' 2024-03-28T23:36:52.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.081+0000 7f2046a85740 1 --2- 172.21.15.161:0/1190784734 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x563f9075c520 0x563f907325a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:52.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.081+0000 7f2046a85740 4 client.9384 mds_command: new command op to 7283 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:36:52.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.081+0000 7f2046a85740 1 -- 172.21.15.161:0/1190784734 --> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] -- command(tid 0: {"prefix": "status"}) -- 0x563f9067b380 con 0x563f9075c520 2024-03-28T23:36:52.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.081+0000 7f2041c46640 1 --2- 172.21.15.161:0/1190784734 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x563f9075c520 0x563f907325a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:52.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.081+0000 7f201cff9640 10 client.9384 ms_handle_connect on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:52.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.081+0000 7f2041c46640 1 --2- 172.21.15.161:0/1190784734 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x563f9075c520 0x563f907325a0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2f3cc8c9c285e93 server_cookie=5d2d534a20ba9936 in_seq=0 out_seq=0 2024-03-28T23:36:52.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.082+0000 7f201cff9640 1 -- 172.21.15.161:0/1190784734 <== mds.0 v2:172.21.15.170:6832/2996092857 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x563f9067b380 con 0x563f9075c520 2024-03-28T23:36:52.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.082+0000 7f201cff9640 10 client.9384 handle_command_reply: tid=0 2024-03-28T23:36:52.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.082+0000 7f2046a85740 1 client.9384 shutdown 2024-03-28T23:36:52.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.082+0000 7f2046a85740 1 -- 172.21.15.161:0/1190784734 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f90734300 msgr2=0x563f9072a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:52.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.082+0000 7f2046a85740 1 --2- 172.21.15.161:0/1190784734 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f90734300 0x563f9072a040 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f2034009010 tx=0x7f2034001250 comp rx=0 tx=0).stop 2024-03-28T23:36:52.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.082+0000 7f2046a85740 1 -- 172.21.15.161:0/1190784734 shutdown_connections 2024-03-28T23:36:52.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.082+0000 7f2046a85740 1 --2- 172.21.15.161:0/1190784734 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x563f9075c520 0x563f907325a0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.082+0000 7f2046a85740 1 --2- 172.21.15.161:0/1190784734 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563f90734300 0x563f9072a040 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.082+0000 7f2046a85740 1 --2- 172.21.15.161:0/1190784734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x563f90710190 0x563f90729b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.082+0000 7f2046a85740 1 --2- 172.21.15.161:0/1190784734 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x563f9070fdb0 0x563f90709bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.082+0000 7f201cff9640 0 client.9384 ms_handle_reset on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:52.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.082+0000 7f2046a85740 1 -- 172.21.15.161:0/1190784734 >> 172.21.15.161:0/1190784734 conn(0x563f9070e310 msgr2=0x563f9075c900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:52.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.082+0000 7f2046a85740 1 -- 172.21.15.161:0/1190784734 shutdown_connections 2024-03-28T23:36:52.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.082+0000 7f2046a85740 1 -- 172.21.15.161:0/1190784734 wait complete. 2024-03-28T23:36:52.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.082+0000 7f2046a85740 20 client.9384 trim_cache size 0 max 16384 2024-03-28T23:36:52.089 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:36:52.089 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:36:52.089 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:36:52.089 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 7283, 2024-03-28T23:36:52.089 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:36:52.089 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:36:52.089 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:36:52.090 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 34.649496493999997, 2024-03-28T23:36:52.090 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 154, 2024-03-28T23:36:52.090 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 161, 2024-03-28T23:36:52.090 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 159, 2024-03-28T23:36:52.090 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 37.639321066999997 2024-03-28T23:36:52.090 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:36:52.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.083+0000 7f2043eaa640 1 -- 172.21.15.161:0/3144453043 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f202006b4d0 msgr2=0x7f202006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:52.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.083+0000 7f2043eaa640 1 --2- 172.21.15.161:0/3144453043 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f202006b4d0 0x7f202006d990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f203c158600 tx=0x7f2038009190 comp rx=0 tx=0).stop 2024-03-28T23:36:52.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.083+0000 7f2043eaa640 1 -- 172.21.15.161:0/3144453043 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f203c0c8ed0 msgr2=0x7f203c1ea830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:52.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.083+0000 7f2043eaa640 1 --2- 172.21.15.161:0/3144453043 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f203c0c8ed0 0x7f203c1ea830 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f202c02b8a0 tx=0x7f202c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.083+0000 7f2043eaa640 1 -- 172.21.15.161:0/3144453043 shutdown_connections 2024-03-28T23:36:52.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.083+0000 7f2043eaa640 1 --2- 172.21.15.161:0/3144453043 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f202006b4d0 0x7f202006d990 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.083+0000 7f2043eaa640 1 --2- 172.21.15.161:0/3144453043 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f203c0c8ed0 0x7f203c1ea830 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.083+0000 7f2043eaa640 1 --2- 172.21.15.161:0/3144453043 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f203c152890 0x7f203c1ead70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.083+0000 7f2043eaa640 1 --2- 172.21.15.161:0/3144453043 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f203c0c5950 0x7f203c157960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:52.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.083+0000 7f2043eaa640 1 -- 172.21.15.161:0/3144453043 >> 172.21.15.161:0/3144453043 conn(0x7f203c0ac620 msgr2=0x7f203c14aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:52.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.083+0000 7f2043eaa640 1 -- 172.21.15.161:0/3144453043 shutdown_connections 2024-03-28T23:36:52.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:52.083+0000 7f2043eaa640 1 -- 172.21.15.161:0/3144453043 wait complete. 2024-03-28T23:36:52.097 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 11, "epoch": 154, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 11, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:35:58.406403+0000", "damaged": [], "data_pools": [ 24 ], "enabled": true, "epoch": 154, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7283": { "addr": "172.21.15.170:6834/2996092857", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 2996092857, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 2996092857, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7283, "incarnation": 151, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 159, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 23, "modified": "2024-03-28T23:36:20.440473+0000", "qdb_cluster": [ 7283 ], "qdb_leader": 7283, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7283 } } }, { "id": 12, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:36:07.448976+0000", "damaged": [], "data_pools": [ 26 ], "enabled": true, "epoch": 149, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs2", "in": [ 0 ], "info": { "gid_7061": { "addr": "172.21.15.161:6836/4026332703", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 4026332703, "type": "v2" }, { "addr": "172.21.15.161:6836", "nonce": 4026332703, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7061, "incarnation": 145, "join_fscid": 12, "name": "c", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 157, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 25, "modified": "2024-03-28T23:36:16.420228+0000", "qdb_cluster": [ 7061 ], "qdb_leader": 7061, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7061 } } } ], "standbys": [ { "addr": "172.21.15.170:6835/557522063", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 557522063, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 557522063, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 137, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7070, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6837/789976100", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6835", "nonce": 789976100, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 789976100, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 152, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9216, "incarnation": 0, "join_fscid": 12, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:36:52.098 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': 12, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 12, 'name': 'c'}, {'join_fscid': -1, 'name': 'd', 'state': 'up:active'}] 2024-03-28T23:36:52.098 INFO:tasks.cephfs.test_failover:current = [{'gid': 9216, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6837/789976100', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 789976100}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 789976100}]}, 'join_fscid': 12, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 152}, {'gid': 7283, 'name': 'b', 'rank': 0, 'incarnation': 151, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.170:6834/2996092857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 2996092857}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 2996092857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 7061, 'name': 'c', 'rank': 0, 'incarnation': 145, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.161:6836/4026332703', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 4026332703}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 4026332703}]}, 'join_fscid': 12, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 7070, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/557522063', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 557522063}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 557522063}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 137}] 2024-03-28T23:36:52.099 DEBUG:tasks.cephfs.test_failover:'up:active' != 'up:standby' - up:active + up:standby 2024-03-28T23:36:52.099 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-28T23:36:56.544 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:36:56.546 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:36:57.100 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:36:57.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.203+0000 7f5dd3b5e640 1 Processor -- start 2024-03-28T23:36:57.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.203+0000 7f5dd3b5e640 1 -- start start 2024-03-28T23:36:57.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.204+0000 7f5dd3b5e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5dcc0c5410 0x7f5dcc0c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:57.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.204+0000 7f5dd3b5e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5dcc1553a0 0x7f5dcc155850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:57.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.204+0000 7f5dd3b5e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5dcc158a30 con 0x7f5dcc152890 2024-03-28T23:36:57.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.204+0000 7f5dd3b5e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5dcc0c8480 con 0x7f5dcc1553a0 2024-03-28T23:36:57.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.204+0000 7f5dd3b5e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f5dcc0c5d90 con 0x7f5dcc0c5410 2024-03-28T23:36:57.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.204+0000 7f5dd20fb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5dcc1553a0 0x7f5dcc155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:57.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.204+0000 7f5dd18fa640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f5dcc152890 0x7f5dcc0c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37410/0 (socket says 172.21.15.161:37410) 2024-03-28T23:36:57.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.204+0000 7f5dd20fb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5dcc1553a0 0x7f5dcc155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53146/0 (socket says 172.21.15.161:53146) 2024-03-28T23:36:57.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.204+0000 7f5dd18fa640 1 -- 172.21.15.161:0/2546380496 learned_addr learned my addr 172.21.15.161:0/2546380496 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:57.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.204+0000 7f5dd10f9640 1 --2- 172.21.15.161:0/2546380496 >> v2:172.21.15.170:3300/0 conn(0x7f5dcc0c5410 0x7f5dcc0c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:57.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.204+0000 7f5dd20fb640 1 -- 172.21.15.161:0/2546380496 >> v2:172.21.15.170:3300/0 conn(0x7f5dcc0c5410 msgr2=0x7f5dcc0c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:57.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.204+0000 7f5dd20fb640 1 --2- 172.21.15.161:0/2546380496 >> v2:172.21.15.170:3300/0 conn(0x7f5dcc0c5410 0x7f5dcc0c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:57.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.204+0000 7f5dd20fb640 1 -- 172.21.15.161:0/2546380496 >> v1:172.21.15.161:6789/0 conn(0x7f5dcc152890 legacy=0x7f5dcc0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:57.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.204+0000 7f5dd20fb640 1 -- 172.21.15.161:0/2546380496 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dcc158d10 con 0x7f5dcc1553a0 2024-03-28T23:36:57.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.204+0000 7f5dd08f8640 1 -- 172.21.15.161:0/2546380496 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2399406188 0 0) 0x7f5dcc158a30 con 0x7f5dcc152890 2024-03-28T23:36:57.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.205+0000 7f5dd20fb640 1 --2- 172.21.15.161:0/2546380496 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dcc1553a0 0x7f5dcc155850 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc0ad6e0 tx=0x7f5dc402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aecfb6e0d2a2c08b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:57.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.205+0000 7f5dd08f8640 1 -- 172.21.15.161:0/2546380496 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4039070 con 0x7f5dcc1553a0 2024-03-28T23:36:57.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.205+0000 7f5dd08f8640 1 -- 172.21.15.161:0/2546380496 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dc402cbf0 con 0x7f5dcc1553a0 2024-03-28T23:36:57.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.205+0000 7f5dd08f8640 1 -- 172.21.15.161:0/2546380496 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4034440 con 0x7f5dcc1553a0 2024-03-28T23:36:57.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.206+0000 7f5dd3b5e640 1 -- 172.21.15.161:0/2546380496 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dcc1553a0 msgr2=0x7f5dcc155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:57.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.206+0000 7f5dd3b5e640 1 --2- 172.21.15.161:0/2546380496 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dcc1553a0 0x7f5dcc155850 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc0ad6e0 tx=0x7f5dc402be40 comp rx=0 tx=0).stop 2024-03-28T23:36:57.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.206+0000 7f5dd3b5e640 1 -- 172.21.15.161:0/2546380496 shutdown_connections 2024-03-28T23:36:57.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.206+0000 7f5dd3b5e640 1 --2- 172.21.15.161:0/2546380496 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dcc1553a0 0x7f5dcc155850 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:57.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.206+0000 7f5dd3b5e640 1 --2- 172.21.15.161:0/2546380496 >> v2:172.21.15.170:3300/0 conn(0x7f5dcc0c5410 0x7f5dcc0c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:57.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.206+0000 7f5dd3b5e640 1 -- 172.21.15.161:0/2546380496 >> 172.21.15.161:0/2546380496 conn(0x7f5dcc0ac620 msgr2=0x7f5dcc0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:57.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.206+0000 7f5dd3b5e640 1 -- 172.21.15.161:0/2546380496 shutdown_connections 2024-03-28T23:36:57.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.206+0000 7f5dd3b5e640 1 -- 172.21.15.161:0/2546380496 wait complete. 2024-03-28T23:36:57.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.207+0000 7f5dd3b5e640 1 Processor -- start 2024-03-28T23:36:57.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.207+0000 7f5dd3b5e640 1 -- start start 2024-03-28T23:36:57.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.208+0000 7f5dd3b5e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dcc0c5410 0x7f5dcc14e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:57.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.208+0000 7f5dd3b5e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5dcc152890 0x7f5dcc14f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:57.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.208+0000 7f5dd3b5e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5dcc1553a0 0x7f5dcc14f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:57.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.208+0000 7f5dd3b5e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5dcc1592d0 con 0x7f5dcc0c5410 2024-03-28T23:36:57.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.208+0000 7f5dd3b5e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5dcc159640 con 0x7f5dcc1553a0 2024-03-28T23:36:57.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.208+0000 7f5dd3b5e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5dcc0c90a0 con 0x7f5dcc152890 2024-03-28T23:36:57.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.208+0000 7f5dd18fa640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dcc0c5410 0x7f5dcc14e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:57.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.208+0000 7f5dd18fa640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dcc0c5410 0x7f5dcc14e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53162/0 (socket says 172.21.15.161:53162) 2024-03-28T23:36:57.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.208+0000 7f5dd18fa640 1 -- 172.21.15.161:0/1813362159 learned_addr learned my addr 172.21.15.161:0/1813362159 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:57.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.208+0000 7f5dd20fb640 1 --2- 172.21.15.161:0/1813362159 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5dcc1553a0 0x7f5dcc14f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:57.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.208+0000 7f5dd10f9640 1 --2- 172.21.15.161:0/1813362159 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5dcc152890 0x7f5dcc14f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:57.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.208+0000 7f5dd18fa640 1 -- 172.21.15.161:0/1813362159 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5dcc152890 msgr2=0x7f5dcc14f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:57.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.208+0000 7f5dd18fa640 1 --2- 172.21.15.161:0/1813362159 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5dcc152890 0x7f5dcc14f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:57.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.208+0000 7f5dd18fa640 1 -- 172.21.15.161:0/1813362159 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5dcc1553a0 msgr2=0x7f5dcc14f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:57.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.208+0000 7f5dd18fa640 1 --2- 172.21.15.161:0/1813362159 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5dcc1553a0 0x7f5dcc14f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:57.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.208+0000 7f5dd18fa640 1 -- 172.21.15.161:0/1813362159 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dcc14d800 con 0x7f5dcc0c5410 2024-03-28T23:36:57.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.208+0000 7f5dd18fa640 1 --2- 172.21.15.161:0/1813362159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dcc0c5410 0x7f5dcc14e410 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f5dc000c4d0 tx=0x7f5dc000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:57.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.208+0000 7f5dbe7fc640 1 -- 172.21.15.161:0/1813362159 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0018070 con 0x7f5dcc0c5410 2024-03-28T23:36:57.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.208+0000 7f5dbe7fc640 1 -- 172.21.15.161:0/1813362159 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dc0007500 con 0x7f5dcc0c5410 2024-03-28T23:36:57.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.208+0000 7f5dd3b5e640 1 -- 172.21.15.161:0/1813362159 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dcc14d520 con 0x7f5dcc0c5410 2024-03-28T23:36:57.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.208+0000 7f5dbe7fc640 1 -- 172.21.15.161:0/1813362159 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0004a20 con 0x7f5dcc0c5410 2024-03-28T23:36:57.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.208+0000 7f5dd3b5e640 1 -- 172.21.15.161:0/1813362159 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dcc1eb560 con 0x7f5dcc0c5410 2024-03-28T23:36:57.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.210+0000 7f5dd3b5e640 1 -- 172.21.15.161:0/1813362159 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dcc04f4b0 con 0x7f5dcc0c5410 2024-03-28T23:36:57.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.210+0000 7f5dbe7fc640 1 -- 172.21.15.161:0/1813362159 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5dc001b070 con 0x7f5dcc0c5410 2024-03-28T23:36:57.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.210+0000 7f5dbe7fc640 1 --2- 172.21.15.161:0/1813362159 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5db006b4d0 0x7f5db006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:57.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.210+0000 7f5dd10f9640 1 --2- 172.21.15.161:0/1813362159 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5db006b4d0 0x7f5db006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:57.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.210+0000 7f5dbe7fc640 1 -- 172.21.15.161:0/1813362159 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7f5dc0089ab0 con 0x7f5dcc0c5410 2024-03-28T23:36:57.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.211+0000 7f5dd10f9640 1 --2- 172.21.15.161:0/1813362159 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5db006b4d0 0x7f5db006d990 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc14a950 tx=0x7f5dc8009110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:57.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.215+0000 7f5dbe7fc640 1 -- 172.21.15.161:0/1813362159 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5dc0057c80 con 0x7f5dcc0c5410 2024-03-28T23:36:57.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.471+0000 7f5dd3b5e640 1 -- 172.21.15.161:0/1813362159 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5dcc06bff0 con 0x7f5dcc0c5410 2024-03-28T23:36:57.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.472+0000 7f5dbe7fc640 1 -- 172.21.15.161:0/1813362159 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+6520 (secure 0 0 0) 0x7f5dc005bd20 con 0x7f5dcc0c5410 2024-03-28T23:36:57.472 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:57.472 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":154,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/557522063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":557522063},{"type":"v1","addr":"172.21.15.170:6835","nonce":557522063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":9216,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/789976100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":789976100},{"type":"v1","addr":"172.21.15.161:6837","nonce":789976100}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:58.406403+0000","modified":"2024-03-28T23:36:20.440473+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":159,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7283":{"gid":7283,"name":"b","rank":0,"incarnation":151,"state":"up:active","state_seq":4,"addr":"172.21.15.170:6834/2996092857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":2996092857},{"type":"v1","addr":"172.21.15.170:6834","nonce":2996092857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7283,"qdb_cluster":[7283]},"id":11},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:36:07.448976+0000","modified":"2024-03-28T23:36:16.420228+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7061":{"gid":7061,"name":"c","rank":0,"incarnation":145,"state":"up:active","state_seq":9,"addr":"172.21.15.161:6836/4026332703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4026332703},{"type":"v1","addr":"172.21.15.161:6836","nonce":4026332703}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7061,"qdb_cluster":[7061]},"id":12}]} 2024-03-28T23:36:57.473 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 154 2024-03-28T23:36:57.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.475+0000 7f5dd3b5e640 1 -- 172.21.15.161:0/1813362159 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5db006b4d0 msgr2=0x7f5db006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:57.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.475+0000 7f5dd3b5e640 1 --2- 172.21.15.161:0/1813362159 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5db006b4d0 0x7f5db006d990 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc14a950 tx=0x7f5dc8009110 comp rx=0 tx=0).stop 2024-03-28T23:36:57.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.475+0000 7f5dd3b5e640 1 -- 172.21.15.161:0/1813362159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dcc0c5410 msgr2=0x7f5dcc14e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:57.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.475+0000 7f5dd3b5e640 1 --2- 172.21.15.161:0/1813362159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dcc0c5410 0x7f5dcc14e410 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f5dc000c4d0 tx=0x7f5dc000c990 comp rx=0 tx=0).stop 2024-03-28T23:36:57.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.475+0000 7f5dd3b5e640 1 -- 172.21.15.161:0/1813362159 shutdown_connections 2024-03-28T23:36:57.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.475+0000 7f5dd3b5e640 1 --2- 172.21.15.161:0/1813362159 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5db006b4d0 0x7f5db006d990 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:57.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.475+0000 7f5dd3b5e640 1 --2- 172.21.15.161:0/1813362159 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5dcc1553a0 0x7f5dcc14f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:57.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.475+0000 7f5dd3b5e640 1 --2- 172.21.15.161:0/1813362159 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5dcc152890 0x7f5dcc14f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:57.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.475+0000 7f5dd3b5e640 1 --2- 172.21.15.161:0/1813362159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dcc0c5410 0x7f5dcc14e410 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:57.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.475+0000 7f5dd3b5e640 1 -- 172.21.15.161:0/1813362159 >> 172.21.15.161:0/1813362159 conn(0x7f5dcc0ac620 msgr2=0x7f5dcc06c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:57.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.475+0000 7f5dd3b5e640 1 -- 172.21.15.161:0/1813362159 shutdown_connections 2024-03-28T23:36:57.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.475+0000 7f5dd3b5e640 1 -- 172.21.15.161:0/1813362159 wait complete. 2024-03-28T23:36:57.888 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 154, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T23:35:58.406403+0000', 'modified': '2024-03-28T23:36:20.440473+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 159, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7283}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7283': {'gid': 7283, 'name': 'b', 'rank': 0, 'incarnation': 151, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.170:6834/2996092857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 2996092857}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 2996092857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7283, 'qdb_cluster': [7283]} 2024-03-28T23:36:57.888 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T23:36:57.888 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T23:36:57.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.992+0000 7fdfafbd0640 1 Processor -- start 2024-03-28T23:36:57.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.992+0000 7fdfafbd0640 1 -- start start 2024-03-28T23:36:57.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.993+0000 7fdfafbd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfa80c5410 0x7fdfa80c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:57.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.993+0000 7fdfafbd0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdfa81553a0 0x7fdfa8155820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:57.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.993+0000 7fdfafbd0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfa8158a30 con 0x7fdfa8152890 2024-03-28T23:36:57.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.993+0000 7fdfafbd0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fdfa80c8480 con 0x7fdfa81553a0 2024-03-28T23:36:57.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.993+0000 7fdfafbd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdfa80c5d90 con 0x7fdfa80c5410 2024-03-28T23:36:57.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.993+0000 7fdfae16d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdfa81553a0 0x7fdfa8155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:57.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.993+0000 7fdfae16d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdfa81553a0 0x7fdfa8155820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53170/0 (socket says 172.21.15.161:53170) 2024-03-28T23:36:57.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.993+0000 7fdfae16d640 1 -- 172.21.15.161:0/2773048123 learned_addr learned my addr 172.21.15.161:0/2773048123 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:57.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.993+0000 7fdfad16b640 1 --2- 172.21.15.161:0/2773048123 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfa80c5410 0x7fdfa80c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:57.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.993+0000 7fdfac96a640 1 -- 172.21.15.161:0/2773048123 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 139865627 0 0) 0x7fdfa8158a30 con 0x7fdfa8152890 2024-03-28T23:36:57.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.994+0000 7fdfae16d640 1 -- 172.21.15.161:0/2773048123 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfa80c5410 msgr2=0x7fdfa80c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:57.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.994+0000 7fdfae16d640 1 --2- 172.21.15.161:0/2773048123 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfa80c5410 0x7fdfa80c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:57.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.994+0000 7fdfae16d640 1 -- 172.21.15.161:0/2773048123 >> v1:172.21.15.170:6789/0 conn(0x7fdfa8152890 legacy=0x7fdfa80c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:57.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.994+0000 7fdfae16d640 1 -- 172.21.15.161:0/2773048123 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfa8158d10 con 0x7fdfa81553a0 2024-03-28T23:36:57.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.994+0000 7fdfae16d640 1 --2- 172.21.15.161:0/2773048123 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfa81553a0 0x7fdfa8155820 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0009e50 tx=0x7fdfa002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3108d648b235c00d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:57.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.994+0000 7fdfac96a640 1 -- 172.21.15.161:0/2773048123 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa002ca70 con 0x7fdfa81553a0 2024-03-28T23:36:57.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.994+0000 7fdfac96a640 1 -- 172.21.15.161:0/2773048123 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfa002cc10 con 0x7fdfa81553a0 2024-03-28T23:36:57.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.994+0000 7fdfac96a640 1 -- 172.21.15.161:0/2773048123 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa00343f0 con 0x7fdfa81553a0 2024-03-28T23:36:57.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.995+0000 7fdfafbd0640 1 -- 172.21.15.161:0/2773048123 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfa81553a0 msgr2=0x7fdfa8155820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:57.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.995+0000 7fdfafbd0640 1 --2- 172.21.15.161:0/2773048123 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfa81553a0 0x7fdfa8155820 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0009e50 tx=0x7fdfa002bed0 comp rx=0 tx=0).stop 2024-03-28T23:36:57.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.995+0000 7fdfafbd0640 1 -- 172.21.15.161:0/2773048123 shutdown_connections 2024-03-28T23:36:57.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.995+0000 7fdfafbd0640 1 --2- 172.21.15.161:0/2773048123 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfa81553a0 0x7fdfa8155820 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:57.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.995+0000 7fdfafbd0640 1 --2- 172.21.15.161:0/2773048123 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfa80c5410 0x7fdfa80c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:57.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.995+0000 7fdfafbd0640 1 -- 172.21.15.161:0/2773048123 >> 172.21.15.161:0/2773048123 conn(0x7fdfa80ac620 msgr2=0x7fdfa80aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:57.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.995+0000 7fdfafbd0640 1 -- 172.21.15.161:0/2773048123 shutdown_connections 2024-03-28T23:36:57.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.995+0000 7fdfafbd0640 1 -- 172.21.15.161:0/2773048123 wait complete. 2024-03-28T23:36:57.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.997+0000 7fdfafbd0640 1 Processor -- start 2024-03-28T23:36:57.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.997+0000 7fdfafbd0640 1 -- start start 2024-03-28T23:36:57.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.998+0000 7fdfafbd0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfa80c5410 0x7fdfa8150e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:57.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.998+0000 7fdfafbd0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfa8152890 0x7fdfa8151360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.998+0000 7fdfafbd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfa81553a0 0x7fdfa8147d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.998+0000 7fdfafbd0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa81592d0 con 0x7fdfa8152890 2024-03-28T23:36:58.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.998+0000 7fdfafbd0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa8159640 con 0x7fdfa80c5410 2024-03-28T23:36:58.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.998+0000 7fdfafbd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdfa80c90a0 con 0x7fdfa81553a0 2024-03-28T23:36:58.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.998+0000 7fdfad96c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfa80c5410 0x7fdfa8150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.998+0000 7fdfad16b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfa8152890 0x7fdfa8151360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.998+0000 7fdfad16b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfa8152890 0x7fdfa8151360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53178/0 (socket says 172.21.15.161:53178) 2024-03-28T23:36:58.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.998+0000 7fdfae16d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfa81553a0 0x7fdfa8147d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.998+0000 7fdfad96c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfa80c5410 0x7fdfa8150e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.161:44150/0 (socket says 172.21.15.161:44150) 2024-03-28T23:36:58.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.998+0000 7fdfad16b640 1 -- 172.21.15.161:0/521773911 learned_addr learned my addr 172.21.15.161:0/521773911 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:58.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.998+0000 7fdfad16b640 1 -- 172.21.15.161:0/521773911 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfa81553a0 msgr2=0x7fdfa8147d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:58.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.998+0000 7fdfad16b640 1 --2- 172.21.15.161:0/521773911 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfa81553a0 0x7fdfa8147d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.998+0000 7fdfad16b640 1 -- 172.21.15.161:0/521773911 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfa80c5410 msgr2=0x7fdfa8150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:58.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.998+0000 7fdfad16b640 1 --2- 172.21.15.161:0/521773911 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfa80c5410 0x7fdfa8150e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.998+0000 7fdfad16b640 1 -- 172.21.15.161:0/521773911 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfa81ebaa0 con 0x7fdfa8152890 2024-03-28T23:36:58.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.998+0000 7fdfad96c640 1 --2- 172.21.15.161:0/521773911 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfa80c5410 0x7fdfa8150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:58.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.999+0000 7fdfad16b640 1 --2- 172.21.15.161:0/521773911 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfa8152890 0x7fdfa8151360 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fdf9800e840 tx=0x7fdf9800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:58.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.999+0000 7fdf9e7fc640 1 -- 172.21.15.161:0/521773911 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf98017070 con 0x7fdfa8152890 2024-03-28T23:36:58.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.999+0000 7fdf9e7fc640 1 -- 172.21.15.161:0/521773911 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf9800c940 con 0x7fdfa8152890 2024-03-28T23:36:58.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.999+0000 7fdfafbd0640 1 -- 172.21.15.161:0/521773911 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfa814f6e0 con 0x7fdfa8152890 2024-03-28T23:36:58.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.999+0000 7fdf9e7fc640 1 -- 172.21.15.161:0/521773911 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9800cd10 con 0x7fdfa8152890 2024-03-28T23:36:58.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:57.999+0000 7fdfafbd0640 1 -- 172.21.15.161:0/521773911 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfa81eb7c0 con 0x7fdfa8152890 2024-03-28T23:36:58.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.000+0000 7fdf9e7fc640 1 -- 172.21.15.161:0/521773911 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdf9801a070 con 0x7fdfa8152890 2024-03-28T23:36:58.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.001+0000 7fdf9e7fc640 1 --2- 172.21.15.161:0/521773911 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdf8c06b4d0 0x7fdf8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.001+0000 7fdfad96c640 1 --2- 172.21.15.161:0/521773911 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdf8c06b4d0 0x7fdf8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.001+0000 7fdf9e7fc640 1 -- 172.21.15.161:0/521773911 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7fdf9808ab20 con 0x7fdfa8152890 2024-03-28T23:36:58.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.001+0000 7fdfad96c640 1 --2- 172.21.15.161:0/521773911 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdf8c06b4d0 0x7fdf8c06d990 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4000f60 tx=0x7fdfa4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:58.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.002+0000 7fdfb27b9740 1 Processor -- start 2024-03-28T23:36:58.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.002+0000 7fdfb27b9740 1 -- start start 2024-03-28T23:36:58.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.002+0000 7fdfb27b9740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558810259db0 0x558810257890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.002+0000 7fdfb27b9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55881025a190 0x558810257dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.002+0000 7fdfb27b9740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558810258310 0x5588102544e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.002+0000 7fdfb27b9740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5588101bdaf0 con 0x558810258310 2024-03-28T23:36:58.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.002+0000 7fdfb27b9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5588101669d0 con 0x55881025a190 2024-03-28T23:36:58.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.002+0000 7fdfb27b9740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x5588101b9b90 con 0x558810259db0 2024-03-28T23:36:58.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.003+0000 7fdfae16d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558810258310 0x5588102544e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.003+0000 7fdfad96c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558810259db0 0x558810257890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.003+0000 7fdfae16d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558810258310 0x5588102544e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53192/0 (socket says 172.21.15.161:53192) 2024-03-28T23:36:58.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.003+0000 7fdfae16d640 1 -- 172.21.15.161:0/3901105864 learned_addr learned my addr 172.21.15.161:0/3901105864 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:58.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.003+0000 7fdfad16b640 1 --2- 172.21.15.161:0/3901105864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55881025a190 0x558810257dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.003+0000 7fdfae16d640 1 -- 172.21.15.161:0/3901105864 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558810259db0 msgr2=0x558810257890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:58.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.003+0000 7fdfae16d640 1 --2- 172.21.15.161:0/3901105864 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558810259db0 0x558810257890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.003+0000 7fdfae16d640 1 -- 172.21.15.161:0/3901105864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55881025a190 msgr2=0x558810257dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:58.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.003+0000 7fdfae16d640 1 --2- 172.21.15.161:0/3901105864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55881025a190 0x558810257dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.003+0000 7fdfae16d640 1 -- 172.21.15.161:0/3901105864 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55880fed22d0 con 0x558810258310 2024-03-28T23:36:58.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.003+0000 7fdfae16d640 1 --2- 172.21.15.161:0/3901105864 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558810258310 0x5588102544e0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0009e50 tx=0x7fdfa002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b19e9a4b80d668f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:58.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.003+0000 7fdf8b7fe640 1 -- 172.21.15.161:0/3901105864 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa002c870 con 0x558810258310 2024-03-28T23:36:58.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.003+0000 7fdf8b7fe640 1 -- 172.21.15.161:0/3901105864 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfa002ca10 con 0x558810258310 2024-03-28T23:36:58.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.003+0000 7fdf8b7fe640 1 -- 172.21.15.161:0/3901105864 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa003f440 con 0x558810258310 2024-03-28T23:36:58.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.004+0000 7fdfb27b9740 1 -- 172.21.15.161:0/3901105864 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558810258310 msgr2=0x5588102544e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:58.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.004+0000 7fdfb27b9740 1 --2- 172.21.15.161:0/3901105864 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558810258310 0x5588102544e0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0009e50 tx=0x7fdfa002e000 comp rx=0 tx=0).stop 2024-03-28T23:36:58.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.004+0000 7fdfb27b9740 1 -- 172.21.15.161:0/3901105864 shutdown_connections 2024-03-28T23:36:58.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.004+0000 7fdfb27b9740 1 --2- 172.21.15.161:0/3901105864 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558810258310 0x5588102544e0 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.004+0000 7fdfb27b9740 1 --2- 172.21.15.161:0/3901105864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55881025a190 0x558810257dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.004+0000 7fdfb27b9740 1 --2- 172.21.15.161:0/3901105864 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558810259db0 0x558810257890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.004+0000 7fdfb27b9740 1 -- 172.21.15.161:0/3901105864 >> 172.21.15.161:0/3901105864 conn(0x558810256630 msgr2=0x55881027b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:58.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.004+0000 7fdfb27b9740 1 -- 172.21.15.161:0/3901105864 shutdown_connections 2024-03-28T23:36:58.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.004+0000 7fdfb27b9740 1 -- 172.21.15.161:0/3901105864 wait complete. 2024-03-28T23:36:58.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.005+0000 7fdfb27b9740 1 -- start start 2024-03-28T23:36:58.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.005+0000 7fdfb27b9740 1 Processor -- start 2024-03-28T23:36:58.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.006+0000 7fdfb27b9740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558810259db0 0x558810254b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.006+0000 7fdfb27b9740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55881025a190 0x5588102a7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.006+0000 7fdfb27b9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558810276c20 0x5588102770a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.006+0000 7fdfb27b9740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55881027c090 con 0x558810259db0 2024-03-28T23:36:58.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.006+0000 7fdfb27b9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x558810179f70 con 0x558810276c20 2024-03-28T23:36:58.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.006+0000 7fdfb27b9740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x55881027bad0 con 0x55881025a190 2024-03-28T23:36:58.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.006+0000 7fdfad96c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558810259db0 0x558810254b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.006+0000 7fdfb27b9740 10 client.0 _ll_register_callbacks cb 0x55880fed1ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:36:58.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.006+0000 7fdfad96c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558810259db0 0x558810254b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53200/0 (socket says 172.21.15.161:53200) 2024-03-28T23:36:58.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.006+0000 7fdfad96c640 1 -- 172.21.15.161:0/2185789409 learned_addr learned my addr 172.21.15.161:0/2185789409 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:58.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.006+0000 7fdfad16b640 1 --2- 172.21.15.161:0/2185789409 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55881025a190 0x5588102a7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.006+0000 7fdfae16d640 1 --2- 172.21.15.161:0/2185789409 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558810276c20 0x5588102770a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.007+0000 7fdfad96c640 1 -- 172.21.15.161:0/2185789409 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55881025a190 msgr2=0x5588102a7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:58.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.007+0000 7fdfad96c640 1 --2- 172.21.15.161:0/2185789409 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55881025a190 0x5588102a7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.007+0000 7fdfad96c640 1 -- 172.21.15.161:0/2185789409 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558810276c20 msgr2=0x5588102770a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:58.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.007+0000 7fdfad96c640 1 --2- 172.21.15.161:0/2185789409 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558810276c20 0x5588102770a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.007+0000 7fdfad96c640 1 -- 172.21.15.161:0/2185789409 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55880ff9c6b0 con 0x558810259db0 2024-03-28T23:36:58.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.007+0000 7fdfad96c640 1 --2- 172.21.15.161:0/2185789409 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558810259db0 0x558810254b00 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4014b70 tx=0x7fdfa4001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20f1d2cec4c74048 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:58.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.007+0000 7fdf88ff9640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:36:58.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.007+0000 7fdf88ff9640 1 -- 172.21.15.161:0/2185789409 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa4025020 con 0x558810259db0 2024-03-28T23:36:58.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.007+0000 7fdf88ff9640 1 -- 172.21.15.161:0/2185789409 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7fdfa4013c40 con 0x558810259db0 2024-03-28T23:36:58.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.007+0000 7fdfb27b9740 1 -- 172.21.15.161:0/2185789409 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfa002ca10 con 0x558810259db0 2024-03-28T23:36:58.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.007+0000 7fdf88ff9640 1 -- 172.21.15.161:0/2185789409 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) ==== 24+0+0 (secure 0 0 0) 0x7fdfa4021400 con 0x558810259db0 2024-03-28T23:36:58.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.007+0000 7fdfb27b9740 10 client.9405 fetch_fsmap learned FSMap version 154 2024-03-28T23:36:58.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.007+0000 7fdfb27b9740 1 -- 172.21.15.161:0/2185789409 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) -- 0x55881027f470 con 0x558810259db0 2024-03-28T23:36:58.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.007+0000 7fdf88ff9640 1 -- 172.21.15.161:0/2185789409 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfa002ca10 con 0x558810259db0 2024-03-28T23:36:58.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.007+0000 7fdf88ff9640 1 -- 172.21.15.161:0/2185789409 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 154) ==== 4139+0+0 (secure 0 0 0) 0x7fdfa4020040 con 0x558810259db0 2024-03-28T23:36:58.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.007+0000 7fdfb27b9740 10 client.9405 fetch_fsmap finished waiting for FSMap version 154 2024-03-28T23:36:58.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.007+0000 7fdfb27b9740 10 client.9405 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:36:58.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.007+0000 7fdfb27b9740 20 client.9405 populate_metadata read hostname 'smithi161' 2024-03-28T23:36:58.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.007+0000 7fdfb27b9740 1 --2- 172.21.15.161:0/2185789409 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x5588102a9c90 0x5588102aa0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.007+0000 7fdfb27b9740 4 client.9405 mds_command: new command op to 7283 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T23:36:58.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.007+0000 7fdfb27b9740 1 -- 172.21.15.161:0/2185789409 --> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588101cb030 con 0x5588102a9c90 2024-03-28T23:36:58.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.008+0000 7fdfad16b640 1 --2- 172.21.15.161:0/2185789409 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x5588102a9c90 0x5588102aa0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.008+0000 7fdf88ff9640 10 client.9405 ms_handle_connect on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:58.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.008+0000 7fdfad16b640 1 --2- 172.21.15.161:0/2185789409 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x5588102a9c90 0x5588102aa0f0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b06bef35ba9e481e server_cookie=f17a34ead66b56c3 in_seq=0 out_seq=0 2024-03-28T23:36:58.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.009+0000 7fdf88ff9640 1 -- 172.21.15.161:0/2185789409 <== mds.0 v2:172.21.15.170:6832/2996092857 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5588101cb030 con 0x5588102a9c90 2024-03-28T23:36:58.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.009+0000 7fdf88ff9640 10 client.9405 handle_command_reply: tid=0 2024-03-28T23:36:58.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.009+0000 7fdfb27b9740 1 client.9405 shutdown 2024-03-28T23:36:58.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.009+0000 7fdfb27b9740 1 -- 172.21.15.161:0/2185789409 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558810259db0 msgr2=0x558810254b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:58.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.009+0000 7fdfb27b9740 1 --2- 172.21.15.161:0/2185789409 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558810259db0 0x558810254b00 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4014b70 tx=0x7fdfa4001080 comp rx=0 tx=0).stop 2024-03-28T23:36:58.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.009+0000 7fdfb27b9740 1 -- 172.21.15.161:0/2185789409 shutdown_connections 2024-03-28T23:36:58.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.009+0000 7fdfb27b9740 1 --2- 172.21.15.161:0/2185789409 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x5588102a9c90 0x5588102aa0f0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.009+0000 7fdfb27b9740 1 --2- 172.21.15.161:0/2185789409 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558810276c20 0x5588102770a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.009+0000 7fdf88ff9640 0 client.9405 ms_handle_reset on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:58.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.009+0000 7fdfb27b9740 1 --2- 172.21.15.161:0/2185789409 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55881025a190 0x5588102a7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.009+0000 7fdfb27b9740 1 --2- 172.21.15.161:0/2185789409 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558810259db0 0x558810254b00 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.009+0000 7fdfb27b9740 1 -- 172.21.15.161:0/2185789409 >> 172.21.15.161:0/2185789409 conn(0x558810258310 msgr2=0x5588102532b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:58.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.009+0000 7fdfb27b9740 1 -- 172.21.15.161:0/2185789409 shutdown_connections 2024-03-28T23:36:58.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.009+0000 7fdfb27b9740 1 -- 172.21.15.161:0/2185789409 wait complete. 2024-03-28T23:36:58.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.009+0000 7fdfb27b9740 20 client.9405 trim_cache size 0 max 16384 2024-03-28T23:36:58.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.027+0000 7fdfb27b9740 1 Processor -- start 2024-03-28T23:36:58.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.027+0000 7fdfb27b9740 1 -- start start 2024-03-28T23:36:58.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.027+0000 7fdfb27b9740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558810259db0 0x55881025ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.027+0000 7fdfb27b9740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55881025a190 0x5588102534e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.027+0000 7fdfb27b9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558810276c20 0x558810253a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.027+0000 7fdfb27b9740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa81fb0a0 con 0x55881025a190 2024-03-28T23:36:58.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.027+0000 7fdfb27b9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa81f8a10 con 0x558810276c20 2024-03-28T23:36:58.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.027+0000 7fdfb27b9740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdfa81f9030 con 0x558810259db0 2024-03-28T23:36:58.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.028+0000 7fdfad16b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55881025a190 0x5588102534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.028+0000 7fdfad16b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55881025a190 0x5588102534e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53212/0 (socket says 172.21.15.161:53212) 2024-03-28T23:36:58.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.028+0000 7fdfad16b640 1 -- 172.21.15.161:0/223991839 learned_addr learned my addr 172.21.15.161:0/223991839 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:58.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.028+0000 7fdfad96c640 1 --2- 172.21.15.161:0/223991839 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558810259db0 0x55881025ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.028+0000 7fdfae16d640 1 --2- 172.21.15.161:0/223991839 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558810276c20 0x558810253a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.028+0000 7fdfad16b640 1 -- 172.21.15.161:0/223991839 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558810259db0 msgr2=0x55881025ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:58.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.028+0000 7fdfad16b640 1 --2- 172.21.15.161:0/223991839 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558810259db0 0x55881025ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.028+0000 7fdfad16b640 1 -- 172.21.15.161:0/223991839 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558810276c20 msgr2=0x558810253a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:58.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.028+0000 7fdfad16b640 1 --2- 172.21.15.161:0/223991839 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558810276c20 0x558810253a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.028+0000 7fdfad16b640 1 -- 172.21.15.161:0/223991839 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588102a78b0 con 0x55881025a190 2024-03-28T23:36:58.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.028+0000 7fdfad96c640 1 --2- 172.21.15.161:0/223991839 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558810259db0 0x55881025ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:58.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.028+0000 7fdfad16b640 1 --2- 172.21.15.161:0/223991839 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55881025a190 0x5588102534e0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fdf98000c00 tx=0x7fdf980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca03ead4e6504047 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:58.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.028+0000 7fdf8affd640 1 -- 172.21.15.161:0/223991839 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf98087070 con 0x55881025a190 2024-03-28T23:36:58.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.028+0000 7fdf8affd640 1 -- 172.21.15.161:0/223991839 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf98056d90 con 0x55881025a190 2024-03-28T23:36:58.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.028+0000 7fdf8affd640 1 -- 172.21.15.161:0/223991839 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf98021040 con 0x55881025a190 2024-03-28T23:36:58.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.029+0000 7fdfb27b9740 1 -- 172.21.15.161:0/223991839 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55881025a190 msgr2=0x5588102534e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:58.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.029+0000 7fdfb27b9740 1 --2- 172.21.15.161:0/223991839 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55881025a190 0x5588102534e0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fdf98000c00 tx=0x7fdf980046c0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.029+0000 7fdfb27b9740 1 -- 172.21.15.161:0/223991839 shutdown_connections 2024-03-28T23:36:58.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.029+0000 7fdfb27b9740 1 --2- 172.21.15.161:0/223991839 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55881025a190 0x5588102534e0 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.029+0000 7fdfb27b9740 1 --2- 172.21.15.161:0/223991839 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558810276c20 0x558810253a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.029+0000 7fdfb27b9740 1 --2- 172.21.15.161:0/223991839 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558810259db0 0x55881025ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.029+0000 7fdfb27b9740 1 -- 172.21.15.161:0/223991839 >> 172.21.15.161:0/223991839 conn(0x558810258310 msgr2=0x5588102b9ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:58.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.029+0000 7fdfb27b9740 1 -- 172.21.15.161:0/223991839 shutdown_connections 2024-03-28T23:36:58.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.029+0000 7fdfb27b9740 1 -- 172.21.15.161:0/223991839 wait complete. 2024-03-28T23:36:58.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.029+0000 7fdfb27b9740 1 -- start start 2024-03-28T23:36:58.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.029+0000 7fdfb27b9740 1 Processor -- start 2024-03-28T23:36:58.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.030+0000 7fdfb27b9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558810259db0 0x5588102b8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.030+0000 7fdfb27b9740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55881025a190 0x5588102536b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.030+0000 7fdfb27b9740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558810276c20 0x55881027a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.030+0000 7fdfb27b9740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55881027fa00 con 0x55881025a190 2024-03-28T23:36:58.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.030+0000 7fdfb27b9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x5588102a9b10 con 0x558810259db0 2024-03-28T23:36:58.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.030+0000 7fdfb27b9740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x5588102a7a80 con 0x558810276c20 2024-03-28T23:36:58.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.030+0000 7fdfb27b9740 10 client.0 _ll_register_callbacks cb 0x7fdfa40079f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T23:36:58.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.030+0000 7fdfad16b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55881025a190 0x5588102536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.030+0000 7fdfad16b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55881025a190 0x5588102536b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53222/0 (socket says 172.21.15.161:53222) 2024-03-28T23:36:58.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.030+0000 7fdfad16b640 1 -- 172.21.15.161:0/2502768432 learned_addr learned my addr 172.21.15.161:0/2502768432 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:58.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.030+0000 7fdfad96c640 1 --2- 172.21.15.161:0/2502768432 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558810259db0 0x5588102b8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.030+0000 7fdfae16d640 1 --2- 172.21.15.161:0/2502768432 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558810276c20 0x55881027a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.031+0000 7fdfad16b640 1 -- 172.21.15.161:0/2502768432 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558810276c20 msgr2=0x55881027a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:58.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.031+0000 7fdfad16b640 1 --2- 172.21.15.161:0/2502768432 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558810276c20 0x55881027a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.031+0000 7fdfad16b640 1 -- 172.21.15.161:0/2502768432 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558810259db0 msgr2=0x5588102b8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:58.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.031+0000 7fdfad16b640 1 --2- 172.21.15.161:0/2502768432 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558810259db0 0x5588102b8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.031+0000 7fdfad16b640 1 -- 172.21.15.161:0/2502768432 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588102932a0 con 0x55881025a190 2024-03-28T23:36:58.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.031+0000 7fdfad16b640 1 --2- 172.21.15.161:0/2502768432 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55881025a190 0x5588102536b0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fdf9800c520 tx=0x7fdf9801e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee64cb21fd18e47f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:58.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.031+0000 7fdf88ff9640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-28T23:36:58.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.031+0000 7fdf88ff9640 1 -- 172.21.15.161:0/2502768432 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf98021070 con 0x55881025a190 2024-03-28T23:36:58.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.031+0000 7fdf88ff9640 1 -- 172.21.15.161:0/2502768432 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7fdf98085d50 con 0x55881025a190 2024-03-28T23:36:58.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.031+0000 7fdfb27b9740 1 -- 172.21.15.161:0/2502768432 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf98056d90 con 0x55881025a190 2024-03-28T23:36:58.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.031+0000 7fdf88ff9640 1 -- 172.21.15.161:0/2502768432 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) ==== 24+0+0 (secure 0 0 0) 0x7fdf98087040 con 0x55881025a190 2024-03-28T23:36:58.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.031+0000 7fdfb27b9740 10 client.9411 fetch_fsmap learned FSMap version 154 2024-03-28T23:36:58.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.031+0000 7fdfb27b9740 1 -- 172.21.15.161:0/2502768432 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) -- 0x5588102977c0 con 0x55881025a190 2024-03-28T23:36:58.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.031+0000 7fdf88ff9640 1 -- 172.21.15.161:0/2502768432 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf98056d90 con 0x55881025a190 2024-03-28T23:36:58.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.031+0000 7fdf88ff9640 1 -- 172.21.15.161:0/2502768432 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 154) ==== 4139+0+0 (secure 0 0 0) 0x7fdf98058140 con 0x55881025a190 2024-03-28T23:36:58.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.031+0000 7fdfb27b9740 10 client.9411 fetch_fsmap finished waiting for FSMap version 154 2024-03-28T23:36:58.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.031+0000 7fdfb27b9740 10 client.9411 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T23:36:58.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.031+0000 7fdfb27b9740 20 client.9411 populate_metadata read hostname 'smithi161' 2024-03-28T23:36:58.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.031+0000 7fdfb27b9740 1 --2- 172.21.15.161:0/2502768432 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x55881030d500 0x5588102a81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.031+0000 7fdfb27b9740 4 client.9411 mds_command: new command op to 7283 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T23:36:58.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.031+0000 7fdfb27b9740 1 -- 172.21.15.161:0/2502768432 --> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] -- command(tid 0: {"prefix": "status"}) -- 0x5588101c5380 con 0x55881030d500 2024-03-28T23:36:58.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.032+0000 7fdfad96c640 1 --2- 172.21.15.161:0/2502768432 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x55881030d500 0x5588102a81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.032+0000 7fdf88ff9640 10 client.9411 ms_handle_connect on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:58.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.032+0000 7fdfad96c640 1 --2- 172.21.15.161:0/2502768432 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x55881030d500 0x5588102a81c0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b55d5aef32a07ee server_cookie=ca8a20593a7e44f8 in_seq=0 out_seq=0 2024-03-28T23:36:58.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.032+0000 7fdf88ff9640 1 -- 172.21.15.161:0/2502768432 <== mds.0 v2:172.21.15.170:6832/2996092857 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x5588101c5380 con 0x55881030d500 2024-03-28T23:36:58.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.032+0000 7fdf88ff9640 10 client.9411 handle_command_reply: tid=0 2024-03-28T23:36:58.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.032+0000 7fdfb27b9740 1 client.9411 shutdown 2024-03-28T23:36:58.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.032+0000 7fdfb27b9740 1 -- 172.21.15.161:0/2502768432 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55881025a190 msgr2=0x5588102536b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:58.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.032+0000 7fdfb27b9740 1 --2- 172.21.15.161:0/2502768432 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55881025a190 0x5588102536b0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fdf9800c520 tx=0x7fdf9801e040 comp rx=0 tx=0).stop 2024-03-28T23:36:58.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.032+0000 7fdfb27b9740 1 -- 172.21.15.161:0/2502768432 shutdown_connections 2024-03-28T23:36:58.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.032+0000 7fdfb27b9740 1 --2- 172.21.15.161:0/2502768432 >> [v2:172.21.15.170:6832/2996092857,v1:172.21.15.170:6834/2996092857] conn(0x55881030d500 0x5588102a81c0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.032+0000 7fdfb27b9740 1 --2- 172.21.15.161:0/2502768432 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55881025a190 0x5588102536b0 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.032+0000 7fdf88ff9640 0 client.9411 ms_handle_reset on v2:172.21.15.170:6832/2996092857 2024-03-28T23:36:58.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.032+0000 7fdfb27b9740 1 --2- 172.21.15.161:0/2502768432 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558810276c20 0x55881027a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.032+0000 7fdfb27b9740 1 --2- 172.21.15.161:0/2502768432 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558810259db0 0x5588102b8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.032+0000 7fdfb27b9740 1 -- 172.21.15.161:0/2502768432 >> 172.21.15.161:0/2502768432 conn(0x558810258310 msgr2=0x5588102b9ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:58.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.032+0000 7fdfb27b9740 1 -- 172.21.15.161:0/2502768432 shutdown_connections 2024-03-28T23:36:58.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.032+0000 7fdfb27b9740 1 -- 172.21.15.161:0/2502768432 wait complete. 2024-03-28T23:36:58.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.033+0000 7fdfb27b9740 20 client.9411 trim_cache size 0 max 16384 2024-03-28T23:36:58.040 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:36:58.040 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "cc975a39-1de3-40a7-9dc0-0e645d96eff8", 2024-03-28T23:36:58.040 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-28T23:36:58.040 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 7283, 2024-03-28T23:36:58.040 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-28T23:36:58.040 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-28T23:36:58.040 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-28T23:36:58.040 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 40.599938139000002, 2024-03-28T23:36:58.040 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 154, 2024-03-28T23:36:58.041 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 161, 2024-03-28T23:36:58.041 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 159, 2024-03-28T23:36:58.041 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 43.589763025000003 2024-03-28T23:36:58.041 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:36:58.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.033+0000 7fdfafbd0640 1 -- 172.21.15.161:0/521773911 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdf8c06b4d0 msgr2=0x7fdf8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:58.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.033+0000 7fdfafbd0640 1 --2- 172.21.15.161:0/521773911 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdf8c06b4d0 0x7fdf8c06d990 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4000f60 tx=0x7fdfa4009040 comp rx=0 tx=0).stop 2024-03-28T23:36:58.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.033+0000 7fdfafbd0640 1 -- 172.21.15.161:0/521773911 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfa8152890 msgr2=0x7fdfa8151360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:58.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.033+0000 7fdfafbd0640 1 --2- 172.21.15.161:0/521773911 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfa8152890 0x7fdfa8151360 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fdf9800e840 tx=0x7fdf9800ed00 comp rx=0 tx=0).stop 2024-03-28T23:36:58.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.033+0000 7fdfafbd0640 1 -- 172.21.15.161:0/521773911 shutdown_connections 2024-03-28T23:36:58.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.033+0000 7fdfafbd0640 1 --2- 172.21.15.161:0/521773911 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdf8c06b4d0 0x7fdf8c06d990 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.033+0000 7fdfafbd0640 1 --2- 172.21.15.161:0/521773911 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfa8152890 0x7fdfa8151360 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.033+0000 7fdfafbd0640 1 --2- 172.21.15.161:0/521773911 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfa81553a0 0x7fdfa8147d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.033+0000 7fdfafbd0640 1 --2- 172.21.15.161:0/521773911 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfa80c5410 0x7fdfa8150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.033+0000 7fdfafbd0640 1 -- 172.21.15.161:0/521773911 >> 172.21.15.161:0/521773911 conn(0x7fdfa80ac620 msgr2=0x7fdfa80c8660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:58.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.033+0000 7fdfafbd0640 1 -- 172.21.15.161:0/521773911 shutdown_connections 2024-03-28T23:36:58.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.033+0000 7fdfafbd0640 1 -- 172.21.15.161:0/521773911 wait complete. 2024-03-28T23:36:58.044 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 11, "epoch": 154, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 11, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:35:58.406403+0000", "damaged": [], "data_pools": [ 24 ], "enabled": true, "epoch": 154, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7283": { "addr": "172.21.15.170:6834/2996092857", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6832", "nonce": 2996092857, "type": "v2" }, { "addr": "172.21.15.170:6834", "nonce": 2996092857, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7283, "incarnation": 151, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 159, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 23, "modified": "2024-03-28T23:36:20.440473+0000", "qdb_cluster": [ 7283 ], "qdb_leader": 7283, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7283 } } }, { "id": 12, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T23:36:07.448976+0000", "damaged": [], "data_pools": [ 26 ], "enabled": true, "epoch": 149, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs2", "in": [ 0 ], "info": { "gid_7061": { "addr": "172.21.15.161:6836/4026332703", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6834", "nonce": 4026332703, "type": "v2" }, { "addr": "172.21.15.161:6836", "nonce": 4026332703, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7061, "incarnation": 145, "join_fscid": 12, "name": "c", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 157, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 25, "modified": "2024-03-28T23:36:16.420228+0000", "qdb_cluster": [ 7061 ], "qdb_leader": 7061, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7061 } } } ], "standbys": [ { "addr": "172.21.15.170:6835/557522063", "addrs": { "addrvec": [ { "addr": "172.21.15.170:6833", "nonce": 557522063, "type": "v2" }, { "addr": "172.21.15.170:6835", "nonce": 557522063, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 137, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7070, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.161:6837/789976100", "addrs": { "addrvec": [ { "addr": "172.21.15.161:6835", "nonce": 789976100, "type": "v2" }, { "addr": "172.21.15.161:6837", "nonce": 789976100, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 152, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9216, "incarnation": 0, "join_fscid": 12, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T23:36:58.045 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': 12, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 12, 'name': 'c'}, {'join_fscid': -1, 'name': 'd', 'state': 'up:active'}] 2024-03-28T23:36:58.046 INFO:tasks.cephfs.test_failover:current = [{'gid': 9216, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.161:6837/789976100', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 789976100}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 789976100}]}, 'join_fscid': 12, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 152}, {'gid': 7283, 'name': 'b', 'rank': 0, 'incarnation': 151, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.170:6834/2996092857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 2996092857}, {'type': 'v1', 'addr': '172.21.15.170:6834', 'nonce': 2996092857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 7061, 'name': 'c', 'rank': 0, 'incarnation': 145, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.161:6836/4026332703', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 4026332703}, {'type': 'v1', 'addr': '172.21.15.161:6836', 'nonce': 4026332703}]}, 'join_fscid': 12, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, {'gid': 7070, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.170:6835/557522063', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6833', 'nonce': 557522063}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 557522063}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'epoch': 137}] 2024-03-28T23:36:58.046 DEBUG:tasks.cephfs.test_failover:'up:active' != 'up:standby' - up:active + up:standby 2024-03-28T23:36:58.046 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-28T23:36:58.046 DEBUG:teuthology.orchestra.run.smithi161:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:36:58.091 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-28T23:36:58.144 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-28T23:36:58.145 INFO:tasks.cephfs.mount:Terminating background process 2024-03-28T23:36:58.145 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:36:58.145 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:36:58.160 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20793 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-28T23:36:58.160 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi161.front.sepia.ceph.com 2024-03-28T23:36:58.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:36:58.160 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-28T23:36:58.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:36:58.215 INFO:teuthology.orchestra.run.smithi161.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-28T23:36:58.216 INFO:tasks.cephfs.mount:Terminating background process 2024-03-28T23:36:58.216 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:36:58.216 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:36:58.233 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056374,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19690 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-28T23:36:58.233 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi170.front.sepia.ceph.com 2024-03-28T23:36:58.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:36:58.233 DEBUG:teuthology.orchestra.run.smithi170:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-28T23:36:58.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:36:58.288 INFO:teuthology.orchestra.run.smithi170.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-28T23:36:58.288 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:36:58.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.427+0000 7fc983a84640 1 Processor -- start 2024-03-28T23:36:58.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.427+0000 7fc983a84640 1 -- start start 2024-03-28T23:36:58.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.428+0000 7fc983a84640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc97c0a8fc0 0x7fc97c153150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.428+0000 7fc983a84640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc97c0aacb0 0x7fc97c153690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.428+0000 7fc983a84640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc97c1566f0 con 0x7fc97c150590 2024-03-28T23:36:58.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.428+0000 7fc983a84640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc97c1569d0 con 0x7fc97c0a8fc0 2024-03-28T23:36:58.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.428+0000 7fc983a84640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fc97c156cb0 con 0x7fc97c0aacb0 2024-03-28T23:36:58.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.428+0000 7fc981820640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc97c150590 0x7fc97c0a9f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37426/0 (socket says 172.21.15.161:37426) 2024-03-28T23:36:58.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.428+0000 7fc981820640 1 -- 172.21.15.161:0/1780717501 learned_addr learned my addr 172.21.15.161:0/1780717501 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:58.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.428+0000 7fc98101f640 1 --2- 172.21.15.161:0/1780717501 >> v2:172.21.15.161:3300/0 conn(0x7fc97c0a8fc0 0x7fc97c153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.428+0000 7fc982021640 1 --2- 172.21.15.161:0/1780717501 >> v2:172.21.15.170:3300/0 conn(0x7fc97c0aacb0 0x7fc97c153690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.428+0000 7fc98081e640 1 -- 172.21.15.161:0/1780717501 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2659584539 0 0) 0x7fc97c1566f0 con 0x7fc97c150590 2024-03-28T23:36:58.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.428+0000 7fc98081e640 1 -- 172.21.15.161:0/1780717501 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc968003520 con 0x7fc97c150590 2024-03-28T23:36:58.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.428+0000 7fc98101f640 1 -- 172.21.15.161:0/1780717501 >> v2:172.21.15.170:3300/0 conn(0x7fc97c0aacb0 msgr2=0x7fc97c153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:58.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.428+0000 7fc98101f640 1 --2- 172.21.15.161:0/1780717501 >> v2:172.21.15.170:3300/0 conn(0x7fc97c0aacb0 0x7fc97c153690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.428+0000 7fc98101f640 1 -- 172.21.15.161:0/1780717501 >> v1:172.21.15.161:6789/0 conn(0x7fc97c150590 legacy=0x7fc97c0a9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:58.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.428+0000 7fc98101f640 1 -- 172.21.15.161:0/1780717501 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc97c156f90 con 0x7fc97c0a8fc0 2024-03-28T23:36:58.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.429+0000 7fc98101f640 1 --2- 172.21.15.161:0/1780717501 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc97c0a8fc0 0x7fc97c153150 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0a98b0 tx=0x7fc96c02b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee329522bcf389f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:58.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.429+0000 7fc98081e640 1 -- 172.21.15.161:0/1780717501 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96c038070 con 0x7fc97c0a8fc0 2024-03-28T23:36:58.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.429+0000 7fc98081e640 1 -- 172.21.15.161:0/1780717501 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc96c031520 con 0x7fc97c0a8fc0 2024-03-28T23:36:58.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.429+0000 7fc98081e640 1 -- 172.21.15.161:0/1780717501 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96c0318f0 con 0x7fc97c0a8fc0 2024-03-28T23:36:58.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.430+0000 7fc983a84640 1 -- 172.21.15.161:0/1780717501 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc97c0a8fc0 msgr2=0x7fc97c153150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:58.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.430+0000 7fc983a84640 1 --2- 172.21.15.161:0/1780717501 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc97c0a8fc0 0x7fc97c153150 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0a98b0 tx=0x7fc96c02b7c0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.430+0000 7fc983a84640 1 -- 172.21.15.161:0/1780717501 shutdown_connections 2024-03-28T23:36:58.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.430+0000 7fc983a84640 1 --2- 172.21.15.161:0/1780717501 >> v2:172.21.15.170:3300/0 conn(0x7fc97c0aacb0 0x7fc97c153690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.430+0000 7fc983a84640 1 --2- 172.21.15.161:0/1780717501 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc97c0a8fc0 0x7fc97c153150 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.430+0000 7fc983a84640 1 -- 172.21.15.161:0/1780717501 >> 172.21.15.161:0/1780717501 conn(0x7fc97c0c6670 msgr2=0x7fc97c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:58.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.430+0000 7fc983a84640 1 -- 172.21.15.161:0/1780717501 shutdown_connections 2024-03-28T23:36:58.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.430+0000 7fc983a84640 1 -- 172.21.15.161:0/1780717501 wait complete. 2024-03-28T23:36:58.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.431+0000 7fc983a84640 1 Processor -- start 2024-03-28T23:36:58.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.431+0000 7fc983a84640 1 -- start start 2024-03-28T23:36:58.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.432+0000 7fc983a84640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc97c0a8fc0 0x7fc97c0c5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.432+0000 7fc983a84640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc97c0aacb0 0x7fc97c145400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.432+0000 7fc983a84640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc97c150590 0x7fc97c1e9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.432+0000 7fc983a84640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc97c0aa650 con 0x7fc97c0a8fc0 2024-03-28T23:36:58.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.432+0000 7fc983a84640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc97c150970 con 0x7fc97c150590 2024-03-28T23:36:58.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.432+0000 7fc983a84640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc97c0aa7d0 con 0x7fc97c0aacb0 2024-03-28T23:36:58.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.432+0000 7fc981820640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc97c0a8fc0 0x7fc97c0c5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.432+0000 7fc981820640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc97c0a8fc0 0x7fc97c0c5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53236/0 (socket says 172.21.15.161:53236) 2024-03-28T23:36:58.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.432+0000 7fc981820640 1 -- 172.21.15.161:0/3777063120 learned_addr learned my addr 172.21.15.161:0/3777063120 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:58.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.432+0000 7fc982021640 1 --2- 172.21.15.161:0/3777063120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc97c150590 0x7fc97c1e9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.432+0000 7fc98101f640 1 --2- 172.21.15.161:0/3777063120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc97c0aacb0 0x7fc97c145400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.432+0000 7fc981820640 1 -- 172.21.15.161:0/3777063120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc97c0aacb0 msgr2=0x7fc97c145400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:58.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.432+0000 7fc981820640 1 --2- 172.21.15.161:0/3777063120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc97c0aacb0 0x7fc97c145400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.432+0000 7fc981820640 1 -- 172.21.15.161:0/3777063120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc97c150590 msgr2=0x7fc97c1e9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:58.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.432+0000 7fc981820640 1 --2- 172.21.15.161:0/3777063120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc97c150590 0x7fc97c1e9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.432+0000 7fc981820640 1 -- 172.21.15.161:0/3777063120 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc97c0c1c80 con 0x7fc97c0a8fc0 2024-03-28T23:36:58.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.432+0000 7fc981820640 1 --2- 172.21.15.161:0/3777063120 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc97c0a8fc0 0x7fc97c0c5aa0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fc97400c3c0 tx=0x7fc97400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:58.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.432+0000 7fc9727fc640 1 -- 172.21.15.161:0/3777063120 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc974017070 con 0x7fc97c0a8fc0 2024-03-28T23:36:58.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.433+0000 7fc9727fc640 1 -- 172.21.15.161:0/3777063120 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc974005ce0 con 0x7fc97c0a8fc0 2024-03-28T23:36:58.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.433+0000 7fc983a84640 1 -- 172.21.15.161:0/3777063120 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc97c153d90 con 0x7fc97c0a8fc0 2024-03-28T23:36:58.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.433+0000 7fc9727fc640 1 -- 172.21.15.161:0/3777063120 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9740072f0 con 0x7fc97c0a8fc0 2024-03-28T23:36:58.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.433+0000 7fc983a84640 1 -- 172.21.15.161:0/3777063120 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc97c154310 con 0x7fc97c0a8fc0 2024-03-28T23:36:58.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.433+0000 7fc983a84640 1 -- 172.21.15.161:0/3777063120 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc97c04f4b0 con 0x7fc97c0a8fc0 2024-03-28T23:36:58.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.434+0000 7fc9727fc640 1 -- 172.21.15.161:0/3777063120 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc97401a070 con 0x7fc97c0a8fc0 2024-03-28T23:36:58.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.434+0000 7fc9727fc640 1 --2- 172.21.15.161:0/3777063120 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc96006b500 0x7fc96006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.434+0000 7fc9727fc640 1 -- 172.21.15.161:0/3777063120 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7fc974089a70 con 0x7fc97c0a8fc0 2024-03-28T23:36:58.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.434+0000 7fc98101f640 1 --2- 172.21.15.161:0/3777063120 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc96006b500 0x7fc96006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.435+0000 7fc98101f640 1 --2- 172.21.15.161:0/3777063120 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc96006b500 0x7fc96006d9c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fc96c035000 tx=0x7fc96c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:58.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.439+0000 7fc9727fc640 1 -- 172.21.15.161:0/3777063120 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc974057cd0 con 0x7fc97c0a8fc0 2024-03-28T23:36:58.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.648+0000 7fc983a84640 1 -- 172.21.15.161:0/3777063120 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc97c0a9d20 con 0x7fc97c0a8fc0 2024-03-28T23:36:58.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.649+0000 7fc9727fc640 1 -- 172.21.15.161:0/3777063120 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+6520 (secure 0 0 0) 0x7fc97405bd70 con 0x7fc97c0a8fc0 2024-03-28T23:36:58.649 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:58.650 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":154,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/557522063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":557522063},{"type":"v1","addr":"172.21.15.170:6835","nonce":557522063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":9216,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/789976100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":789976100},{"type":"v1","addr":"172.21.15.161:6837","nonce":789976100}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:58.406403+0000","modified":"2024-03-28T23:36:20.440473+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":159,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7283":{"gid":7283,"name":"b","rank":0,"incarnation":151,"state":"up:active","state_seq":4,"addr":"172.21.15.170:6834/2996092857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":2996092857},{"type":"v1","addr":"172.21.15.170:6834","nonce":2996092857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7283,"qdb_cluster":[7283]},"id":11},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:36:07.448976+0000","modified":"2024-03-28T23:36:16.420228+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7061":{"gid":7061,"name":"c","rank":0,"incarnation":145,"state":"up:active","state_seq":9,"addr":"172.21.15.161:6836/4026332703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4026332703},{"type":"v1","addr":"172.21.15.161:6836","nonce":4026332703}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7061,"qdb_cluster":[7061]},"id":12}]} 2024-03-28T23:36:58.650 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 154 2024-03-28T23:36:58.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.651+0000 7fc983a84640 1 -- 172.21.15.161:0/3777063120 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc96006b500 msgr2=0x7fc96006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:58.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.651+0000 7fc983a84640 1 --2- 172.21.15.161:0/3777063120 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc96006b500 0x7fc96006d9c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fc96c035000 tx=0x7fc96c030000 comp rx=0 tx=0).stop 2024-03-28T23:36:58.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.651+0000 7fc983a84640 1 -- 172.21.15.161:0/3777063120 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc97c0a8fc0 msgr2=0x7fc97c0c5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:58.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.651+0000 7fc983a84640 1 --2- 172.21.15.161:0/3777063120 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc97c0a8fc0 0x7fc97c0c5aa0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fc97400c3c0 tx=0x7fc97400c880 comp rx=0 tx=0).stop 2024-03-28T23:36:58.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.651+0000 7fc983a84640 1 -- 172.21.15.161:0/3777063120 shutdown_connections 2024-03-28T23:36:58.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.651+0000 7fc983a84640 1 --2- 172.21.15.161:0/3777063120 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc96006b500 0x7fc96006d9c0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.651+0000 7fc983a84640 1 --2- 172.21.15.161:0/3777063120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc97c150590 0x7fc97c1e9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.651+0000 7fc983a84640 1 --2- 172.21.15.161:0/3777063120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc97c0aacb0 0x7fc97c145400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.651+0000 7fc983a84640 1 --2- 172.21.15.161:0/3777063120 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc97c0a8fc0 0x7fc97c0c5aa0 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.651+0000 7fc983a84640 1 -- 172.21.15.161:0/3777063120 >> 172.21.15.161:0/3777063120 conn(0x7fc97c0c6670 msgr2=0x7fc97c149e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:58.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.651+0000 7fc983a84640 1 -- 172.21.15.161:0/3777063120 shutdown_connections 2024-03-28T23:36:58.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.651+0000 7fc983a84640 1 -- 172.21.15.161:0/3777063120 wait complete. 2024-03-28T23:36:58.662 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:36:58.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.763+0000 7f4d3c2b1640 1 Processor -- start 2024-03-28T23:36:58.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.763+0000 7f4d3c2b1640 1 -- start start 2024-03-28T23:36:58.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.764+0000 7f4d3c2b1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4d34152890 0x7f4d340c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.764+0000 7f4d3c2b1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d340c8ed0 0x7f4d340c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.764+0000 7f4d3c2b1640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d34158a30 con 0x7f4d340c5950 2024-03-28T23:36:58.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.764+0000 7f4d3c2b1640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f4d340c8480 con 0x7f4d34152890 2024-03-28T23:36:58.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.764+0000 7f4d3c2b1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4d34159640 con 0x7f4d340c8ed0 2024-03-28T23:36:58.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.764+0000 7f4d3a04d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4d34152890 0x7f4d340c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.764+0000 7f4d3a04d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4d34152890 0x7f4d340c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53246/0 (socket says 172.21.15.161:53246) 2024-03-28T23:36:58.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.764+0000 7f4d3a04d640 1 -- 172.21.15.161:0/941818635 learned_addr learned my addr 172.21.15.161:0/941818635 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:58.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.764+0000 7f4d3984c640 1 --2- 172.21.15.161:0/941818635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d340c8ed0 0x7f4d340c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.764+0000 7f4d3a04d640 1 -- 172.21.15.161:0/941818635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d340c8ed0 msgr2=0x7f4d340c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:58.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.764+0000 7f4d3a04d640 1 --2- 172.21.15.161:0/941818635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d340c8ed0 0x7f4d340c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.764+0000 7f4d3904b640 1 -- 172.21.15.161:0/941818635 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1918447026 0 0) 0x7f4d34158a30 con 0x7f4d340c5950 2024-03-28T23:36:58.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.764+0000 7f4d3a04d640 1 -- 172.21.15.161:0/941818635 >> v1:172.21.15.170:6789/0 conn(0x7f4d340c5950 legacy=0x7f4d341553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:58.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.764+0000 7f4d3a04d640 1 -- 172.21.15.161:0/941818635 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d34158d10 con 0x7f4d34152890 2024-03-28T23:36:58.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.765+0000 7f4d3a04d640 1 --2- 172.21.15.161:0/941818635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4d34152890 0x7f4d340c8990 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f4d30009870 tx=0x7f4d3002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb024f52dd6179c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:58.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.765+0000 7f4d3904b640 1 -- 172.21.15.161:0/941818635 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d300317c0 con 0x7f4d34152890 2024-03-28T23:36:58.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.765+0000 7f4d3904b640 1 -- 172.21.15.161:0/941818635 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d30037070 con 0x7f4d34152890 2024-03-28T23:36:58.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.765+0000 7f4d3904b640 1 -- 172.21.15.161:0/941818635 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d30031b80 con 0x7f4d34152890 2024-03-28T23:36:58.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.765+0000 7f4d3c2b1640 1 -- 172.21.15.161:0/941818635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4d34152890 msgr2=0x7f4d340c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:58.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.765+0000 7f4d3c2b1640 1 --2- 172.21.15.161:0/941818635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4d34152890 0x7f4d340c8990 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f4d30009870 tx=0x7f4d3002b860 comp rx=0 tx=0).stop 2024-03-28T23:36:58.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.766+0000 7f4d3c2b1640 1 -- 172.21.15.161:0/941818635 shutdown_connections 2024-03-28T23:36:58.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.766+0000 7f4d3c2b1640 1 --2- 172.21.15.161:0/941818635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d340c8ed0 0x7f4d340c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.766+0000 7f4d3c2b1640 1 --2- 172.21.15.161:0/941818635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4d34152890 0x7f4d340c8990 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.766+0000 7f4d3c2b1640 1 -- 172.21.15.161:0/941818635 >> 172.21.15.161:0/941818635 conn(0x7f4d340ac620 msgr2=0x7f4d340aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:58.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.766+0000 7f4d3c2b1640 1 -- 172.21.15.161:0/941818635 shutdown_connections 2024-03-28T23:36:58.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.766+0000 7f4d3c2b1640 1 -- 172.21.15.161:0/941818635 wait complete. 2024-03-28T23:36:58.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.767+0000 7f4d3c2b1640 1 Processor -- start 2024-03-28T23:36:58.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.767+0000 7f4d3c2b1640 1 -- start start 2024-03-28T23:36:58.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.767+0000 7f4d3c2b1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d340c5950 0x7f4d34146fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.767+0000 7f4d3c2b1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4d340c8ed0 0x7f4d341474f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.767+0000 7f4d3c2b1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d34152890 0x7f4d341edf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.767+0000 7f4d3c2b1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4d34152c70 con 0x7f4d340c8ed0 2024-03-28T23:36:58.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.767+0000 7f4d3c2b1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4d341592d0 con 0x7f4d34152890 2024-03-28T23:36:58.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.767+0000 7f4d3c2b1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4d34152df0 con 0x7f4d340c5950 2024-03-28T23:36:58.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.768+0000 7f4d3984c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4d340c8ed0 0x7f4d341474f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.768+0000 7f4d3984c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4d340c8ed0 0x7f4d341474f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53252/0 (socket says 172.21.15.161:53252) 2024-03-28T23:36:58.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.768+0000 7f4d3984c640 1 -- 172.21.15.161:0/843652004 learned_addr learned my addr 172.21.15.161:0/843652004 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:58.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.768+0000 7f4d3a04d640 1 --2- 172.21.15.161:0/843652004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d340c5950 0x7f4d34146fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.768+0000 7f4d3a84e640 1 --2- 172.21.15.161:0/843652004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d34152890 0x7f4d341edf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.768+0000 7f4d3984c640 1 -- 172.21.15.161:0/843652004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d340c5950 msgr2=0x7f4d34146fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:58.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.768+0000 7f4d3984c640 1 --2- 172.21.15.161:0/843652004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d340c5950 0x7f4d34146fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.768+0000 7f4d3984c640 1 -- 172.21.15.161:0/843652004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d34152890 msgr2=0x7f4d341edf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:58.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.768+0000 7f4d3984c640 1 --2- 172.21.15.161:0/843652004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d34152890 0x7f4d341edf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.768+0000 7f4d3984c640 1 -- 172.21.15.161:0/843652004 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d3414ca90 con 0x7f4d340c8ed0 2024-03-28T23:36:58.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.768+0000 7f4d3984c640 1 --2- 172.21.15.161:0/843652004 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4d340c8ed0 0x7f4d341474f0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f4d2400e930 tx=0x7f4d2400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:58.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.768+0000 7f4d2affd640 1 -- 172.21.15.161:0/843652004 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d24017070 con 0x7f4d340c8ed0 2024-03-28T23:36:58.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.768+0000 7f4d2affd640 1 -- 172.21.15.161:0/843652004 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d2400ca00 con 0x7f4d340c8ed0 2024-03-28T23:36:58.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.768+0000 7f4d3c2b1640 1 -- 172.21.15.161:0/843652004 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d3414d150 con 0x7f4d340c8ed0 2024-03-28T23:36:58.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.768+0000 7f4d2affd640 1 -- 172.21.15.161:0/843652004 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2400cdd0 con 0x7f4d340c8ed0 2024-03-28T23:36:58.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.768+0000 7f4d3c2b1640 1 -- 172.21.15.161:0/843652004 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d3414d710 con 0x7f4d340c8ed0 2024-03-28T23:36:58.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.769+0000 7f4d3c2b1640 1 -- 172.21.15.161:0/843652004 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d3404f4b0 con 0x7f4d340c8ed0 2024-03-28T23:36:58.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.770+0000 7f4d2affd640 1 -- 172.21.15.161:0/843652004 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4d2401a070 con 0x7f4d340c8ed0 2024-03-28T23:36:58.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.770+0000 7f4d2affd640 1 --2- 172.21.15.161:0/843652004 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4d1806b4d0 0x7f4d1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:58.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.770+0000 7f4d2affd640 1 -- 172.21.15.161:0/843652004 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7f4d2408ab30 con 0x7f4d340c8ed0 2024-03-28T23:36:58.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.770+0000 7f4d3a04d640 1 --2- 172.21.15.161:0/843652004 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4d1806b4d0 0x7f4d1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:58.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.771+0000 7f4d3a04d640 1 --2- 172.21.15.161:0/843652004 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4d1806b4d0 0x7f4d1806d990 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f4d30036000 tx=0x7f4d3003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:58.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.775+0000 7f4d2affd640 1 -- 172.21.15.161:0/843652004 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4d24058dc0 con 0x7f4d340c8ed0 2024-03-28T23:36:58.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.947+0000 7f4d3c2b1640 1 -- 172.21.15.161:0/843652004 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4d340c8140 con 0x7f4d340c8ed0 2024-03-28T23:36:58.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.948+0000 7f4d2affd640 1 -- 172.21.15.161:0/843652004 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v161) ==== 74+0+21282 (secure 0 0 0) 0x7f4d24088e20 con 0x7f4d340c8ed0 2024-03-28T23:36:58.949 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:36:58.949 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":161,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:36:36.466146+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:35:56.687416+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-28T23:35:57.379120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:36:05.484451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:36:06.423073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.c","mappings":[{"from":1,"to":0}]},{"pgid":"26.13","mappings":[{"from":1,"to":6}]},{"pgid":"26.28","mappings":[{"from":1,"to":0}]},{"pgid":"26.3b","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6837/3236742821":"2024-03-29T23:36:17.431214+0000","172.21.15.161:6835/3236742821":"2024-03-29T23:36:17.431214+0000","172.21.15.170:6834/1393137291":"2024-03-29T23:36:14.412210+0000","172.21.15.170:6832/1393137291":"2024-03-29T23:36:14.412210+0000","172.21.15.161:6837/2391019771":"2024-03-29T23:36:13.968386+0000","172.21.15.161:6835/2391019771":"2024-03-29T23:36:13.968386+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:36:58.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.951+0000 7f4d3c2b1640 1 -- 172.21.15.161:0/843652004 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4d1806b4d0 msgr2=0x7f4d1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:58.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.951+0000 7f4d3c2b1640 1 --2- 172.21.15.161:0/843652004 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4d1806b4d0 0x7f4d1806d990 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f4d30036000 tx=0x7f4d3003d040 comp rx=0 tx=0).stop 2024-03-28T23:36:58.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.951+0000 7f4d3c2b1640 1 -- 172.21.15.161:0/843652004 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4d340c8ed0 msgr2=0x7f4d341474f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:58.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.951+0000 7f4d3c2b1640 1 --2- 172.21.15.161:0/843652004 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4d340c8ed0 0x7f4d341474f0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f4d2400e930 tx=0x7f4d2400edf0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.951+0000 7f4d3c2b1640 1 -- 172.21.15.161:0/843652004 shutdown_connections 2024-03-28T23:36:58.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.951+0000 7f4d3c2b1640 1 --2- 172.21.15.161:0/843652004 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4d1806b4d0 0x7f4d1806d990 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.951+0000 7f4d3c2b1640 1 --2- 172.21.15.161:0/843652004 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4d340c8ed0 0x7f4d341474f0 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.951+0000 7f4d3c2b1640 1 --2- 172.21.15.161:0/843652004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d34152890 0x7f4d341edf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.951+0000 7f4d3c2b1640 1 --2- 172.21.15.161:0/843652004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d340c5950 0x7f4d34146fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:58.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.951+0000 7f4d3c2b1640 1 -- 172.21.15.161:0/843652004 >> 172.21.15.161:0/843652004 conn(0x7f4d340ac620 msgr2=0x7f4d340c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:58.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.951+0000 7f4d3c2b1640 1 -- 172.21.15.161:0/843652004 shutdown_connections 2024-03-28T23:36:58.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:58.951+0000 7f4d3c2b1640 1 -- 172.21.15.161:0/843652004 wait complete. 2024-03-28T23:36:58.963 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T23:35:56.687416+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '151', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T23:35:57.379120+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '152', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-03-28T23:36:05.484451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '155', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs2_data', 'create_time': '2024-03-28T23:36:06.423073+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '156', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:36:58.964 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:36:59.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.069+0000 7f4c5426f640 1 Processor -- start 2024-03-28T23:36:59.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.069+0000 7f4c5426f640 1 -- start start 2024-03-28T23:36:59.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.069+0000 7f4c5426f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4c4c1527c0 0x7f4c4c0c1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:59.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.069+0000 7f4c5426f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c4c0708e0 0x7f4c4c155840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:59.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.069+0000 7f4c5426f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c4c152ba0 con 0x7f4c4c0704c0 2024-03-28T23:36:59.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.069+0000 7f4c5426f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f4c4c0c3aa0 con 0x7f4c4c1527c0 2024-03-28T23:36:59.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.069+0000 7f4c5426f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4c4c070dc0 con 0x7f4c4c0708e0 2024-03-28T23:36:59.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.070+0000 7f4c5200b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4c4c1527c0 0x7f4c4c0c1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:59.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.070+0000 7f4c5180a640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f4c4c0704c0 0x7f4c4c155130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37438/0 (socket says 172.21.15.161:37438) 2024-03-28T23:36:59.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.070+0000 7f4c5180a640 1 -- 172.21.15.161:0/2938988020 learned_addr learned my addr 172.21.15.161:0/2938988020 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:59.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.070+0000 7f4c5280c640 1 --2- 172.21.15.161:0/2938988020 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c4c0708e0 0x7f4c4c155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:59.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.070+0000 7f4c5280c640 1 -- 172.21.15.161:0/2938988020 >> v2:172.21.15.161:3300/0 conn(0x7f4c4c1527c0 msgr2=0x7f4c4c0c1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:59.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.070+0000 7f4c5280c640 1 --2- 172.21.15.161:0/2938988020 >> v2:172.21.15.161:3300/0 conn(0x7f4c4c1527c0 0x7f4c4c0c1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:59.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.070+0000 7f4c5280c640 1 -- 172.21.15.161:0/2938988020 >> v1:172.21.15.161:6789/0 conn(0x7f4c4c0704c0 legacy=0x7f4c4c155130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:59.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.070+0000 7f4c5280c640 1 -- 172.21.15.161:0/2938988020 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c4c0c3cf0 con 0x7f4c4c0708e0 2024-03-28T23:36:59.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.070+0000 7f4c5200b640 1 --2- 172.21.15.161:0/2938988020 >> v2:172.21.15.161:3300/0 conn(0x7f4c4c1527c0 0x7f4c4c0c1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:59.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.070+0000 7f4c51009640 1 -- 172.21.15.161:0/2938988020 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2433957947 0 0) 0x7f4c4c152ba0 con 0x7f4c4c0704c0 2024-03-28T23:36:59.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.071+0000 7f4c5280c640 1 --2- 172.21.15.161:0/2938988020 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c4c0708e0 0x7f4c4c155840 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4c44009d90 tx=0x7f4c4402bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=721b99001b46f74a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:59.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.071+0000 7f4c51009640 1 -- 172.21.15.161:0/2938988020 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c44039070 con 0x7f4c4c0708e0 2024-03-28T23:36:59.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.071+0000 7f4c51009640 1 -- 172.21.15.161:0/2938988020 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c4402cb30 con 0x7f4c4c0708e0 2024-03-28T23:36:59.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.071+0000 7f4c51009640 1 -- 172.21.15.161:0/2938988020 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c44034440 con 0x7f4c4c0708e0 2024-03-28T23:36:59.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.071+0000 7f4c5426f640 1 -- 172.21.15.161:0/2938988020 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c4c0708e0 msgr2=0x7f4c4c155840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:59.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.071+0000 7f4c5426f640 1 --2- 172.21.15.161:0/2938988020 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c4c0708e0 0x7f4c4c155840 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4c44009d90 tx=0x7f4c4402bd80 comp rx=0 tx=0).stop 2024-03-28T23:36:59.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.072+0000 7f4c5426f640 1 -- 172.21.15.161:0/2938988020 shutdown_connections 2024-03-28T23:36:59.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.072+0000 7f4c5426f640 1 --2- 172.21.15.161:0/2938988020 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c4c0708e0 0x7f4c4c155840 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:59.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.072+0000 7f4c5426f640 1 --2- 172.21.15.161:0/2938988020 >> v2:172.21.15.161:3300/0 conn(0x7f4c4c1527c0 0x7f4c4c0c1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:59.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.072+0000 7f4c5426f640 1 -- 172.21.15.161:0/2938988020 >> 172.21.15.161:0/2938988020 conn(0x7f4c4c0c8810 msgr2=0x7f4c4c0c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:59.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.072+0000 7f4c5426f640 1 -- 172.21.15.161:0/2938988020 shutdown_connections 2024-03-28T23:36:59.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.072+0000 7f4c5426f640 1 -- 172.21.15.161:0/2938988020 wait complete. 2024-03-28T23:36:59.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.073+0000 7f4c5426f640 1 Processor -- start 2024-03-28T23:36:59.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.073+0000 7f4c5426f640 1 -- start start 2024-03-28T23:36:59.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.073+0000 7f4c5426f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c4c0704c0 0x7f4c4c1578f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:59.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.073+0000 7f4c5426f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c4c0708e0 0x7f4c4c0c7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:59.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.073+0000 7f4c5426f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c4c1527c0 0x7f4c4c1eaa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:59.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.073+0000 7f4c5426f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4c0c47d0 con 0x7f4c4c0704c0 2024-03-28T23:36:59.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.073+0000 7f4c5426f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4c0c42b0 con 0x7f4c4c1527c0 2024-03-28T23:36:59.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.073+0000 7f4c5426f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4c4c0c2060 con 0x7f4c4c0708e0 2024-03-28T23:36:59.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.073+0000 7f4c5200b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c4c0704c0 0x7f4c4c1578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:59.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.073+0000 7f4c5200b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c4c0704c0 0x7f4c4c1578f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53282/0 (socket says 172.21.15.161:53282) 2024-03-28T23:36:59.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.073+0000 7f4c5200b640 1 -- 172.21.15.161:0/431576035 learned_addr learned my addr 172.21.15.161:0/431576035 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:59.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.073+0000 7f4c5180a640 1 --2- 172.21.15.161:0/431576035 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c4c0708e0 0x7f4c4c0c7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:59.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.073+0000 7f4c5280c640 1 --2- 172.21.15.161:0/431576035 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c4c1527c0 0x7f4c4c1eaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:59.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.074+0000 7f4c5200b640 1 -- 172.21.15.161:0/431576035 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c4c0708e0 msgr2=0x7f4c4c0c7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:59.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.074+0000 7f4c5200b640 1 --2- 172.21.15.161:0/431576035 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c4c0708e0 0x7f4c4c0c7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:59.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.074+0000 7f4c5200b640 1 -- 172.21.15.161:0/431576035 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c4c1527c0 msgr2=0x7f4c4c1eaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:59.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.074+0000 7f4c5200b640 1 --2- 172.21.15.161:0/431576035 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c4c1527c0 0x7f4c4c1eaa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:59.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.074+0000 7f4c5200b640 1 -- 172.21.15.161:0/431576035 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c4c158bd0 con 0x7f4c4c0704c0 2024-03-28T23:36:59.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.074+0000 7f4c5200b640 1 --2- 172.21.15.161:0/431576035 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c4c0704c0 0x7f4c4c1578f0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c00e990 tx=0x7f4c3c00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:59.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.074+0000 7f4c42ffd640 1 -- 172.21.15.161:0/431576035 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c017070 con 0x7f4c4c0704c0 2024-03-28T23:36:59.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.074+0000 7f4c42ffd640 1 -- 172.21.15.161:0/431576035 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c3c00c9e0 con 0x7f4c4c0704c0 2024-03-28T23:36:59.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.074+0000 7f4c42ffd640 1 -- 172.21.15.161:0/431576035 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c00cdb0 con 0x7f4c4c0704c0 2024-03-28T23:36:59.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.074+0000 7f4c5426f640 1 -- 172.21.15.161:0/431576035 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c4c1eed20 con 0x7f4c4c0704c0 2024-03-28T23:36:59.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.074+0000 7f4c5426f640 1 -- 172.21.15.161:0/431576035 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c4c1550d0 con 0x7f4c4c0704c0 2024-03-28T23:36:59.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.075+0000 7f4c5426f640 1 -- 172.21.15.161:0/431576035 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c4c154be0 con 0x7f4c4c0704c0 2024-03-28T23:36:59.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.075+0000 7f4c42ffd640 1 -- 172.21.15.161:0/431576035 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4c3c01a070 con 0x7f4c4c0704c0 2024-03-28T23:36:59.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.076+0000 7f4c42ffd640 1 --2- 172.21.15.161:0/431576035 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4c3006b500 0x7f4c3006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:59.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.076+0000 7f4c5180a640 1 --2- 172.21.15.161:0/431576035 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4c3006b500 0x7f4c3006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:59.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.076+0000 7f4c42ffd640 1 -- 172.21.15.161:0/431576035 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7f4c3c08a9e0 con 0x7f4c4c0704c0 2024-03-28T23:36:59.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.076+0000 7f4c5180a640 1 --2- 172.21.15.161:0/431576035 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4c3006b500 0x7f4c3006d9c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c1ebba0 tx=0x7f4c48007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:59.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.084+0000 7f4c42ffd640 1 -- 172.21.15.161:0/431576035 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4c3c058bb0 con 0x7f4c4c0704c0 2024-03-28T23:36:59.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.354+0000 7f4c5426f640 1 -- 172.21.15.161:0/431576035 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4c4c04f4b0 con 0x7f4c4c0704c0 2024-03-28T23:36:59.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.355+0000 7f4c42ffd640 1 -- 172.21.15.161:0/431576035 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v161) ==== 93+0+10 (secure 0 0 0) 0x7f4c3c05cc50 con 0x7f4c4c0704c0 2024-03-28T23:36:59.355 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:36:59.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.357+0000 7f4c5426f640 1 -- 172.21.15.161:0/431576035 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4c3006b500 msgr2=0x7f4c3006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:59.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.357+0000 7f4c5426f640 1 --2- 172.21.15.161:0/431576035 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4c3006b500 0x7f4c3006d9c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c1ebba0 tx=0x7f4c48007d50 comp rx=0 tx=0).stop 2024-03-28T23:36:59.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.357+0000 7f4c5426f640 1 -- 172.21.15.161:0/431576035 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c4c0704c0 msgr2=0x7f4c4c1578f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:59.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.357+0000 7f4c5426f640 1 --2- 172.21.15.161:0/431576035 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c4c0704c0 0x7f4c4c1578f0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c00e990 tx=0x7f4c3c00ee50 comp rx=0 tx=0).stop 2024-03-28T23:36:59.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.357+0000 7f4c5426f640 1 -- 172.21.15.161:0/431576035 shutdown_connections 2024-03-28T23:36:59.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.357+0000 7f4c5426f640 1 --2- 172.21.15.161:0/431576035 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4c3006b500 0x7f4c3006d9c0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:59.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.357+0000 7f4c5426f640 1 --2- 172.21.15.161:0/431576035 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c4c1527c0 0x7f4c4c1eaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:59.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.357+0000 7f4c5426f640 1 --2- 172.21.15.161:0/431576035 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c4c0708e0 0x7f4c4c0c7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:59.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.357+0000 7f4c5426f640 1 --2- 172.21.15.161:0/431576035 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c4c0704c0 0x7f4c4c1578f0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:59.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.357+0000 7f4c5426f640 1 -- 172.21.15.161:0/431576035 >> 172.21.15.161:0/431576035 conn(0x7f4c4c0c8810 msgr2=0x7f4c4c1505c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:59.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.357+0000 7f4c5426f640 1 -- 172.21.15.161:0/431576035 shutdown_connections 2024-03-28T23:36:59.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.357+0000 7f4c5426f640 1 -- 172.21.15.161:0/431576035 wait complete. 2024-03-28T23:36:59.369 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T23:36:59.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.474+0000 7f83ad63b640 1 Processor -- start 2024-03-28T23:36:59.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.474+0000 7f83ad63b640 1 -- start start 2024-03-28T23:36:59.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.475+0000 7f83ad63b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83a8152bc0 0x7f83a80c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:59.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.475+0000 7f83ad63b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f83a80c8a80 0x7f83a80c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:59.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.475+0000 7f83ad63b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83a80c6f60 con 0x7f83a80c9420 2024-03-28T23:36:59.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.475+0000 7f83ad63b640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f83a80c6120 con 0x7f83a80c8a80 2024-03-28T23:36:59.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.475+0000 7f83ad63b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f83a80c6400 con 0x7f83a8152bc0 2024-03-28T23:36:59.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.475+0000 7f839ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f83a80c8a80 0x7f83a80c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:59.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.475+0000 7f839ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f83a80c8a80 0x7f83a80c8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53294/0 (socket says 172.21.15.161:53294) 2024-03-28T23:36:59.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.475+0000 7f839ffff640 1 -- 172.21.15.161:0/416031532 learned_addr learned my addr 172.21.15.161:0/416031532 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:59.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.475+0000 7f83a6ffd640 1 --2- 172.21.15.161:0/416031532 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83a8152bc0 0x7f83a80c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:59.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.476+0000 7f839ffff640 1 -- 172.21.15.161:0/416031532 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83a8152bc0 msgr2=0x7f83a80c0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:36:59.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.476+0000 7f839ffff640 1 --2- 172.21.15.161:0/416031532 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83a8152bc0 0x7f83a80c0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:59.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.476+0000 7f839ffff640 1 -- 172.21.15.161:0/416031532 >> v1:172.21.15.170:6789/0 conn(0x7f83a80c9420 legacy=0x7f83a81589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:59.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.476+0000 7f839ffff640 1 -- 172.21.15.161:0/416031532 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a80c7240 con 0x7f83a80c8a80 2024-03-28T23:36:59.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.476+0000 7f83a67fc640 1 -- 172.21.15.161:0/416031532 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1903505115 0 0) 0x7f83a80c6f60 con 0x7f83a80c9420 2024-03-28T23:36:59.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.476+0000 7f839ffff640 1 --2- 172.21.15.161:0/416031532 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83a80c8a80 0x7f83a80c8ee0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f8390009870 tx=0x7f839002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32c003b36d66a3be server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:59.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.477+0000 7f83a67fc640 1 -- 172.21.15.161:0/416031532 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8390034070 con 0x7f83a80c8a80 2024-03-28T23:36:59.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.477+0000 7f83a67fc640 1 -- 172.21.15.161:0/416031532 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8390030840 con 0x7f83a80c8a80 2024-03-28T23:36:59.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.477+0000 7f83a67fc640 1 -- 172.21.15.161:0/416031532 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8390030c10 con 0x7f83a80c8a80 2024-03-28T23:36:59.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.477+0000 7f83ad63b640 1 -- 172.21.15.161:0/416031532 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83a80c8a80 msgr2=0x7f83a80c8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:59.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.477+0000 7f83ad63b640 1 --2- 172.21.15.161:0/416031532 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83a80c8a80 0x7f83a80c8ee0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f8390009870 tx=0x7f839002b980 comp rx=0 tx=0).stop 2024-03-28T23:36:59.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.477+0000 7f83ad63b640 1 -- 172.21.15.161:0/416031532 shutdown_connections 2024-03-28T23:36:59.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.477+0000 7f83ad63b640 1 --2- 172.21.15.161:0/416031532 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83a80c8a80 0x7f83a80c8ee0 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:59.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.477+0000 7f83ad63b640 1 --2- 172.21.15.161:0/416031532 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83a8152bc0 0x7f83a80c0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:59.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.477+0000 7f83ad63b640 1 -- 172.21.15.161:0/416031532 >> 172.21.15.161:0/416031532 conn(0x7f83a80caa80 msgr2=0x7f83a80cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:59.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.477+0000 7f83ad63b640 1 -- 172.21.15.161:0/416031532 shutdown_connections 2024-03-28T23:36:59.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.477+0000 7f83ad63b640 1 -- 172.21.15.161:0/416031532 wait complete. 2024-03-28T23:36:59.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.479+0000 7f83ad63b640 1 Processor -- start 2024-03-28T23:36:59.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.479+0000 7f83ad63b640 1 -- start start 2024-03-28T23:36:59.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.479+0000 7f83ad63b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83a80c8a80 0x7f83a814b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:59.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.479+0000 7f83ad63b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83a80c9420 0x7f83a81ee270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:59.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.479+0000 7f83ad63b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83a8152bc0 0x7f83a81ee7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:59.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.479+0000 7f83ad63b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f83a8152fa0 con 0x7f83a80c9420 2024-03-28T23:36:59.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.479+0000 7f83ad63b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f83a81590c0 con 0x7f83a80c8a80 2024-03-28T23:36:59.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.479+0000 7f83ad63b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f83a8158f20 con 0x7f83a8152bc0 2024-03-28T23:36:59.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.479+0000 7f839ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83a80c9420 0x7f83a81ee270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:59.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.479+0000 7f839ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83a80c9420 0x7f83a81ee270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53308/0 (socket says 172.21.15.161:53308) 2024-03-28T23:36:59.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.479+0000 7f839ffff640 1 -- 172.21.15.161:0/4566989 learned_addr learned my addr 172.21.15.161:0/4566989 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:59.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.479+0000 7f83a6ffd640 1 --2- 172.21.15.161:0/4566989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83a80c8a80 0x7f83a814b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:59.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.479+0000 7f83a77fe640 1 --2- 172.21.15.161:0/4566989 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83a8152bc0 0x7f83a81ee7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:59.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.480+0000 7f839ffff640 1 -- 172.21.15.161:0/4566989 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83a8152bc0 msgr2=0x7f83a81ee7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:59.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.480+0000 7f839ffff640 1 --2- 172.21.15.161:0/4566989 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83a8152bc0 0x7f83a81ee7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:59.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.480+0000 7f839ffff640 1 -- 172.21.15.161:0/4566989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83a80c8a80 msgr2=0x7f83a814b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:59.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.480+0000 7f839ffff640 1 --2- 172.21.15.161:0/4566989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83a80c8a80 0x7f83a814b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:59.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.480+0000 7f839ffff640 1 -- 172.21.15.161:0/4566989 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a80ae2e0 con 0x7f83a80c9420 2024-03-28T23:36:59.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.480+0000 7f83a6ffd640 1 --2- 172.21.15.161:0/4566989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83a80c8a80 0x7f83a814b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:36:59.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.480+0000 7f839ffff640 1 --2- 172.21.15.161:0/4566989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83a80c9420 0x7f83a81ee270 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f8390000c00 tx=0x7f83900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:59.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.480+0000 7f839f7fe640 1 -- 172.21.15.161:0/4566989 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8390034040 con 0x7f83a80c9420 2024-03-28T23:36:59.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.480+0000 7f839f7fe640 1 -- 172.21.15.161:0/4566989 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8390030640 con 0x7f83a80c9420 2024-03-28T23:36:59.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.480+0000 7f839f7fe640 1 -- 172.21.15.161:0/4566989 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8390039d40 con 0x7f83a80c9420 2024-03-28T23:36:59.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.480+0000 7f83ad63b640 1 -- 172.21.15.161:0/4566989 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83a80ae5a0 con 0x7f83a80c9420 2024-03-28T23:36:59.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.480+0000 7f83ad63b640 1 -- 172.21.15.161:0/4566989 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83a8158920 con 0x7f83a80c9420 2024-03-28T23:36:59.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.481+0000 7f83ad63b640 1 -- 172.21.15.161:0/4566989 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83a804f4b0 con 0x7f83a80c9420 2024-03-28T23:36:59.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.482+0000 7f839f7fe640 1 -- 172.21.15.161:0/4566989 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f839004b060 con 0x7f83a80c9420 2024-03-28T23:36:59.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.482+0000 7f839f7fe640 1 --2- 172.21.15.161:0/4566989 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f838806b4d0 0x7f838806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:59.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.482+0000 7f839f7fe640 1 -- 172.21.15.161:0/4566989 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7f83900ac3e0 con 0x7f83a80c9420 2024-03-28T23:36:59.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.482+0000 7f83a6ffd640 1 --2- 172.21.15.161:0/4566989 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f838806b4d0 0x7f838806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:59.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.483+0000 7f83a6ffd640 1 --2- 172.21.15.161:0/4566989 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f838806b4d0 0x7f838806d990 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f8398009e90 tx=0x7f8398009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:59.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.487+0000 7f839f7fe640 1 -- 172.21.15.161:0/4566989 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8390030810 con 0x7f83a80c9420 2024-03-28T23:36:59.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.661+0000 7f83ad63b640 1 -- 172.21.15.161:0/4566989 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f83a80bfca0 con 0x7f83a80c9420 2024-03-28T23:36:59.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.661+0000 7f839f7fe640 1 -- 172.21.15.161:0/4566989 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v161) ==== 104+0+11 (secure 0 0 0) 0x7f839007a6e0 con 0x7f83a80c9420 2024-03-28T23:36:59.662 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:36:59.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.663+0000 7f83ad63b640 1 -- 172.21.15.161:0/4566989 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f838806b4d0 msgr2=0x7f838806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:59.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.663+0000 7f83ad63b640 1 --2- 172.21.15.161:0/4566989 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f838806b4d0 0x7f838806d990 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f8398009e90 tx=0x7f8398009190 comp rx=0 tx=0).stop 2024-03-28T23:36:59.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.663+0000 7f83ad63b640 1 -- 172.21.15.161:0/4566989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83a80c9420 msgr2=0x7f83a81ee270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:59.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.663+0000 7f83ad63b640 1 --2- 172.21.15.161:0/4566989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83a80c9420 0x7f83a81ee270 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f8390000c00 tx=0x7f83900046c0 comp rx=0 tx=0).stop 2024-03-28T23:36:59.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.663+0000 7f83ad63b640 1 -- 172.21.15.161:0/4566989 shutdown_connections 2024-03-28T23:36:59.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.663+0000 7f83ad63b640 1 --2- 172.21.15.161:0/4566989 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f838806b4d0 0x7f838806d990 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:59.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.663+0000 7f83ad63b640 1 --2- 172.21.15.161:0/4566989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83a80c9420 0x7f83a81ee270 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:59.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.663+0000 7f83ad63b640 1 --2- 172.21.15.161:0/4566989 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83a8152bc0 0x7f83a81ee7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:59.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.663+0000 7f83ad63b640 1 --2- 172.21.15.161:0/4566989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83a80c8a80 0x7f83a814b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:59.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.663+0000 7f83ad63b640 1 -- 172.21.15.161:0/4566989 >> 172.21.15.161:0/4566989 conn(0x7f83a80caa80 msgr2=0x7f83a80ab170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:59.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.663+0000 7f83ad63b640 1 -- 172.21.15.161:0/4566989 shutdown_connections 2024-03-28T23:36:59.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.663+0000 7f83ad63b640 1 -- 172.21.15.161:0/4566989 wait complete. 2024-03-28T23:36:59.675 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T23:36:59.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.781+0000 7fa7deec0640 1 Processor -- start 2024-03-28T23:36:59.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.782+0000 7fa7deec0640 1 -- start start 2024-03-28T23:36:59.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.782+0000 7fa7deec0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa7d80c6750 0x7fa7d806bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:59.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.782+0000 7fa7deec0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7d8152db0 con 0x7fa7d81529d0 2024-03-28T23:36:59.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.782+0000 7fa7deec0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7d80c8d40 con 0x7fa7d80c88b0 2024-03-28T23:36:59.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.782+0000 7fa7deec0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fa7d80c7210 con 0x7fa7d80c6750 2024-03-28T23:36:59.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.782+0000 7fa7dcc5c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa7d81529d0 0x7fa7d8155c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37448/0 (socket says 172.21.15.161:37448) 2024-03-28T23:36:59.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.782+0000 7fa7dcc5c640 1 -- 172.21.15.161:0/1006312604 learned_addr learned my addr 172.21.15.161:0/1006312604 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:36:59.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.782+0000 7fa7d7fff640 1 --2- 172.21.15.161:0/1006312604 >> v2:172.21.15.170:3300/0 conn(0x7fa7d80c6750 0x7fa7d806bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:59.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.783+0000 7fa7d77fe640 1 -- 172.21.15.161:0/1006312604 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1682547043 0 0) 0x7fa7d8152db0 con 0x7fa7d81529d0 2024-03-28T23:36:59.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.783+0000 7fa7d77fe640 1 -- 172.21.15.161:0/1006312604 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7c8003520 con 0x7fa7d81529d0 2024-03-28T23:36:59.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.783+0000 7fa7d7fff640 1 -- 172.21.15.161:0/1006312604 >> v1:172.21.15.170:6789/0 conn(0x7fa7d80c88b0 legacy=0x7fa7d80c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:59.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.783+0000 7fa7d7fff640 1 -- 172.21.15.161:0/1006312604 >> v1:172.21.15.161:6789/0 conn(0x7fa7d81529d0 legacy=0x7fa7d8155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:59.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.783+0000 7fa7d7fff640 1 -- 172.21.15.161:0/1006312604 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d80c6c50 con 0x7fa7d80c6750 2024-03-28T23:36:59.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.783+0000 7fa7d7fff640 1 --2- 172.21.15.161:0/1006312604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7d80c6750 0x7fa7d806bb60 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fa7c40097a0 tx=0x7fa7c402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c22f98347cb2b7f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:59.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.784+0000 7fa7d77fe640 1 -- 172.21.15.161:0/1006312604 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c4034070 con 0x7fa7d80c6750 2024-03-28T23:36:59.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.784+0000 7fa7d77fe640 1 -- 172.21.15.161:0/1006312604 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7c4030610 con 0x7fa7d80c6750 2024-03-28T23:36:59.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.784+0000 7fa7d77fe640 1 -- 172.21.15.161:0/1006312604 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c40309c0 con 0x7fa7d80c6750 2024-03-28T23:36:59.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.784+0000 7fa7deec0640 1 -- 172.21.15.161:0/1006312604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7d80c6750 msgr2=0x7fa7d806bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:59.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.784+0000 7fa7deec0640 1 --2- 172.21.15.161:0/1006312604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7d80c6750 0x7fa7d806bb60 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fa7c40097a0 tx=0x7fa7c402b960 comp rx=0 tx=0).stop 2024-03-28T23:36:59.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.784+0000 7fa7deec0640 1 -- 172.21.15.161:0/1006312604 shutdown_connections 2024-03-28T23:36:59.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.784+0000 7fa7deec0640 1 --2- 172.21.15.161:0/1006312604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7d80c6750 0x7fa7d806bb60 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:59.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.784+0000 7fa7deec0640 1 -- 172.21.15.161:0/1006312604 >> 172.21.15.161:0/1006312604 conn(0x7fa7d80c7430 msgr2=0x7fa7d80c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:36:59.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.784+0000 7fa7deec0640 1 -- 172.21.15.161:0/1006312604 shutdown_connections 2024-03-28T23:36:59.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.784+0000 7fa7deec0640 1 -- 172.21.15.161:0/1006312604 wait complete. 2024-03-28T23:36:59.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.785+0000 7fa7deec0640 1 Processor -- start 2024-03-28T23:36:59.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.785+0000 7fa7deec0640 1 -- start start 2024-03-28T23:36:59.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.786+0000 7fa7deec0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7d80c6750 0x7fa7d814d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:59.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.786+0000 7fa7deec0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7d80c88b0 0x7fa7d80ab660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:59.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.786+0000 7fa7deec0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7d81529d0 0x7fa7d80c1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:59.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.786+0000 7fa7deec0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d8155710 con 0x7fa7d80c6750 2024-03-28T23:36:59.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.786+0000 7fa7deec0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d8155430 con 0x7fa7d81529d0 2024-03-28T23:36:59.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.786+0000 7fa7deec0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa7d80aa160 con 0x7fa7d80c88b0 2024-03-28T23:36:59.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.786+0000 7fa7dcc5c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7d80c6750 0x7fa7d814d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:59.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.786+0000 7fa7dcc5c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7d80c6750 0x7fa7d814d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53312/0 (socket says 172.21.15.161:53312) 2024-03-28T23:36:59.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.786+0000 7fa7dcc5c640 1 -- 172.21.15.161:0/3410967526 learned_addr learned my addr 172.21.15.161:0/3410967526 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:36:59.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.786+0000 7fa7dd45d640 1 --2- 172.21.15.161:0/3410967526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7d81529d0 0x7fa7d80c1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:59.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.786+0000 7fa7d7fff640 1 --2- 172.21.15.161:0/3410967526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7d80c88b0 0x7fa7d80ab660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:59.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.787+0000 7fa7dcc5c640 1 -- 172.21.15.161:0/3410967526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7d80c88b0 msgr2=0x7fa7d80ab660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:59.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.787+0000 7fa7dcc5c640 1 --2- 172.21.15.161:0/3410967526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7d80c88b0 0x7fa7d80ab660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:59.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.787+0000 7fa7dcc5c640 1 -- 172.21.15.161:0/3410967526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7d81529d0 msgr2=0x7fa7d80c1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:36:59.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.787+0000 7fa7dcc5c640 1 --2- 172.21.15.161:0/3410967526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7d81529d0 0x7fa7d80c1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:36:59.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.787+0000 7fa7dcc5c640 1 -- 172.21.15.161:0/3410967526 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d8150580 con 0x7fa7d80c6750 2024-03-28T23:36:59.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.787+0000 7fa7dcc5c640 1 --2- 172.21.15.161:0/3410967526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7d80c6750 0x7fa7d814d8c0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc00c440 tx=0x7fa7cc00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:59.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.787+0000 7fa7d57fa640 1 -- 172.21.15.161:0/3410967526 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc017070 con 0x7fa7d80c6750 2024-03-28T23:36:59.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.787+0000 7fa7d57fa640 1 -- 172.21.15.161:0/3410967526 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7cc005ce0 con 0x7fa7d80c6750 2024-03-28T23:36:59.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.787+0000 7fa7d57fa640 1 -- 172.21.15.161:0/3410967526 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc0072b0 con 0x7fa7d80c6750 2024-03-28T23:36:59.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.787+0000 7fa7deec0640 1 -- 172.21.15.161:0/3410967526 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7d8154df0 con 0x7fa7d80c6750 2024-03-28T23:36:59.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.787+0000 7fa7deec0640 1 -- 172.21.15.161:0/3410967526 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7d80ac190 con 0x7fa7d80c6750 2024-03-28T23:36:59.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.788+0000 7fa7deec0640 1 -- 172.21.15.161:0/3410967526 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7d804f4b0 con 0x7fa7d80c6750 2024-03-28T23:36:59.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.789+0000 7fa7d57fa640 1 -- 172.21.15.161:0/3410967526 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa7cc01a070 con 0x7fa7d80c6750 2024-03-28T23:36:59.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.789+0000 7fa7d57fa640 1 --2- 172.21.15.161:0/3410967526 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa7bc06b500 0x7fa7bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:36:59.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.790+0000 7fa7d57fa640 1 -- 172.21.15.161:0/3410967526 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7fa7cc089aa0 con 0x7fa7d80c6750 2024-03-28T23:36:59.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.790+0000 7fa7d7fff640 1 --2- 172.21.15.161:0/3410967526 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa7bc06b500 0x7fa7bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:36:59.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.790+0000 7fa7d7fff640 1 --2- 172.21.15.161:0/3410967526 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa7bc06b500 0x7fa7bc06d9c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4036000 tx=0x7fa7c4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:36:59.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:36:59.794+0000 7fa7d57fa640 1 -- 172.21.15.161:0/3410967526 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa7cc057dc0 con 0x7fa7d80c6750 2024-03-28T23:37:00.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.023+0000 7fa7deec0640 1 -- 172.21.15.161:0/3410967526 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa7d806d800 con 0x7fa7d80c6750 2024-03-28T23:37:00.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.023+0000 7fa7d57fa640 1 -- 172.21.15.161:0/3410967526 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v161) ==== 100+0+11 (secure 0 0 0) 0x7fa7cc05be60 con 0x7fa7d80c6750 2024-03-28T23:37:00.024 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:37:00.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.025+0000 7fa7deec0640 1 -- 172.21.15.161:0/3410967526 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa7bc06b500 msgr2=0x7fa7bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:00.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.025+0000 7fa7deec0640 1 --2- 172.21.15.161:0/3410967526 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa7bc06b500 0x7fa7bc06d9c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4036000 tx=0x7fa7c4008030 comp rx=0 tx=0).stop 2024-03-28T23:37:00.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.025+0000 7fa7deec0640 1 -- 172.21.15.161:0/3410967526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7d80c6750 msgr2=0x7fa7d814d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:00.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.025+0000 7fa7deec0640 1 --2- 172.21.15.161:0/3410967526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7d80c6750 0x7fa7d814d8c0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc00c440 tx=0x7fa7cc00c900 comp rx=0 tx=0).stop 2024-03-28T23:37:00.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.025+0000 7fa7deec0640 1 -- 172.21.15.161:0/3410967526 shutdown_connections 2024-03-28T23:37:00.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.025+0000 7fa7deec0640 1 --2- 172.21.15.161:0/3410967526 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa7bc06b500 0x7fa7bc06d9c0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.025+0000 7fa7deec0640 1 --2- 172.21.15.161:0/3410967526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7d81529d0 0x7fa7d80c1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.025+0000 7fa7deec0640 1 --2- 172.21.15.161:0/3410967526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7d80c88b0 0x7fa7d80ab660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.025+0000 7fa7deec0640 1 --2- 172.21.15.161:0/3410967526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7d80c6750 0x7fa7d814d8c0 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.025+0000 7fa7deec0640 1 -- 172.21.15.161:0/3410967526 >> 172.21.15.161:0/3410967526 conn(0x7fa7d80c7430 msgr2=0x7fa7d80c5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:00.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.025+0000 7fa7deec0640 1 -- 172.21.15.161:0/3410967526 shutdown_connections 2024-03-28T23:37:00.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.025+0000 7fa7deec0640 1 -- 172.21.15.161:0/3410967526 wait complete. 2024-03-28T23:37:00.040 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-03-28T23:37:00.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.144+0000 7f5aeff3e640 1 Processor -- start 2024-03-28T23:37:00.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.145+0000 7f5aeff3e640 1 -- start start 2024-03-28T23:37:00.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.145+0000 7f5aeff3e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5ae8152bc0 0x7f5ae80c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:00.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.145+0000 7f5aeff3e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5ae81589b0 0x7f5ae8158e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:00.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.145+0000 7f5aeff3e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ae80c6f60 con 0x7f5ae80c8a80 2024-03-28T23:37:00.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.145+0000 7f5aeff3e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5ae80c6120 con 0x7f5ae81589b0 2024-03-28T23:37:00.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.145+0000 7f5aeff3e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f5ae80c6400 con 0x7f5ae8152bc0 2024-03-28T23:37:00.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.145+0000 7f5aee4db640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5ae81589b0 0x7f5ae8158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:00.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.145+0000 7f5aee4db640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5ae81589b0 0x7f5ae8158e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53324/0 (socket says 172.21.15.161:53324) 2024-03-28T23:37:00.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.145+0000 7f5aee4db640 1 -- 172.21.15.161:0/2461053284 learned_addr learned my addr 172.21.15.161:0/2461053284 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:00.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.146+0000 7f5aedcda640 1 --2- 172.21.15.161:0/2461053284 >> v2:172.21.15.170:3300/0 conn(0x7f5ae8152bc0 0x7f5ae80c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:00.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.146+0000 7f5aee4db640 1 -- 172.21.15.161:0/2461053284 >> v2:172.21.15.170:3300/0 conn(0x7f5ae8152bc0 msgr2=0x7f5ae80c0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:00.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.146+0000 7f5aee4db640 1 --2- 172.21.15.161:0/2461053284 >> v2:172.21.15.170:3300/0 conn(0x7f5ae8152bc0 0x7f5ae80c0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.146+0000 7f5aee4db640 1 -- 172.21.15.161:0/2461053284 >> v1:172.21.15.170:6789/0 conn(0x7f5ae80c8a80 legacy=0x7f5ae80c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:00.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.146+0000 7f5aee4db640 1 -- 172.21.15.161:0/2461053284 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae80c7240 con 0x7f5ae81589b0 2024-03-28T23:37:00.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.146+0000 7f5aedcda640 1 --2- 172.21.15.161:0/2461053284 >> v2:172.21.15.170:3300/0 conn(0x7f5ae8152bc0 0x7f5ae80c0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:37:00.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.146+0000 7f5aee4db640 1 --2- 172.21.15.161:0/2461053284 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ae81589b0 0x7f5ae8158e00 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f5ae80aa410 tx=0x7f5ae002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85f7a3d8729882d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:00.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.147+0000 7f5aeccd8640 1 -- 172.21.15.161:0/2461053284 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae002cb20 con 0x7f5ae81589b0 2024-03-28T23:37:00.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.147+0000 7f5aeccd8640 1 -- 172.21.15.161:0/2461053284 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae002ccc0 con 0x7f5ae81589b0 2024-03-28T23:37:00.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.147+0000 7f5aeccd8640 1 -- 172.21.15.161:0/2461053284 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae0034610 con 0x7f5ae81589b0 2024-03-28T23:37:00.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.147+0000 7f5aeff3e640 1 -- 172.21.15.161:0/2461053284 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ae81589b0 msgr2=0x7f5ae8158e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:00.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.147+0000 7f5aeff3e640 1 --2- 172.21.15.161:0/2461053284 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ae81589b0 0x7f5ae8158e00 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f5ae80aa410 tx=0x7f5ae002bf80 comp rx=0 tx=0).stop 2024-03-28T23:37:00.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.147+0000 7f5aeff3e640 1 -- 172.21.15.161:0/2461053284 shutdown_connections 2024-03-28T23:37:00.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.147+0000 7f5aeff3e640 1 --2- 172.21.15.161:0/2461053284 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ae81589b0 0x7f5ae8158e00 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.147+0000 7f5aeff3e640 1 --2- 172.21.15.161:0/2461053284 >> v2:172.21.15.170:3300/0 conn(0x7f5ae8152bc0 0x7f5ae80c0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.147+0000 7f5aeff3e640 1 -- 172.21.15.161:0/2461053284 >> 172.21.15.161:0/2461053284 conn(0x7f5ae80caa80 msgr2=0x7f5ae80cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:00.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.147+0000 7f5aeff3e640 1 -- 172.21.15.161:0/2461053284 shutdown_connections 2024-03-28T23:37:00.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.147+0000 7f5aeff3e640 1 -- 172.21.15.161:0/2461053284 wait complete. 2024-03-28T23:37:00.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.148+0000 7f5aeff3e640 1 Processor -- start 2024-03-28T23:37:00.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.148+0000 7f5aeff3e640 1 -- start start 2024-03-28T23:37:00.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.149+0000 7f5aeff3e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5ae80c8a80 0x7f5ae8147780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:00.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.149+0000 7f5aeff3e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ae8152bc0 0x7f5ae806bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:00.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.149+0000 7f5aeff3e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ae81589b0 0x7f5ae806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:00.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.149+0000 7f5aeff3e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae8152fa0 con 0x7f5ae8152bc0 2024-03-28T23:37:00.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.149+0000 7f5aeff3e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae80c95f0 con 0x7f5ae80c8a80 2024-03-28T23:37:00.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.149+0000 7f5aeff3e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f5ae80ab2a0 con 0x7f5ae81589b0 2024-03-28T23:37:00.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.149+0000 7f5aed4d9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ae8152bc0 0x7f5ae806bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:00.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.149+0000 7f5aed4d9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ae8152bc0 0x7f5ae806bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53340/0 (socket says 172.21.15.161:53340) 2024-03-28T23:37:00.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.149+0000 7f5aed4d9640 1 -- 172.21.15.161:0/290106785 learned_addr learned my addr 172.21.15.161:0/290106785 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:00.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.149+0000 7f5aedcda640 1 --2- 172.21.15.161:0/290106785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5ae80c8a80 0x7f5ae8147780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:00.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.149+0000 7f5aee4db640 1 --2- 172.21.15.161:0/290106785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ae81589b0 0x7f5ae806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:00.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.149+0000 7f5aed4d9640 1 -- 172.21.15.161:0/290106785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ae81589b0 msgr2=0x7f5ae806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:00.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.149+0000 7f5aed4d9640 1 --2- 172.21.15.161:0/290106785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ae81589b0 0x7f5ae806c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.149+0000 7f5aed4d9640 1 -- 172.21.15.161:0/290106785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5ae80c8a80 msgr2=0x7f5ae8147780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:00.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.149+0000 7f5aed4d9640 1 --2- 172.21.15.161:0/290106785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5ae80c8a80 0x7f5ae8147780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.150+0000 7f5aed4d9640 1 -- 172.21.15.161:0/290106785 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae8158780 con 0x7f5ae8152bc0 2024-03-28T23:37:00.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.150+0000 7f5aedcda640 1 --2- 172.21.15.161:0/290106785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5ae80c8a80 0x7f5ae8147780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:37:00.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.150+0000 7f5aed4d9640 1 --2- 172.21.15.161:0/290106785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ae8152bc0 0x7f5ae806bd60 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f5ad800c3f0 tx=0x7f5ad800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:00.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.150+0000 7f5ade7fc640 1 -- 172.21.15.161:0/290106785 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad80052d0 con 0x7f5ae8152bc0 2024-03-28T23:37:00.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.150+0000 7f5ade7fc640 1 -- 172.21.15.161:0/290106785 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ad8005490 con 0x7f5ae8152bc0 2024-03-28T23:37:00.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.150+0000 7f5aeff3e640 1 -- 172.21.15.161:0/290106785 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ae8158190 con 0x7f5ae8152bc0 2024-03-28T23:37:00.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.150+0000 7f5ade7fc640 1 -- 172.21.15.161:0/290106785 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad8007260 con 0x7f5ae8152bc0 2024-03-28T23:37:00.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.150+0000 7f5aeff3e640 1 -- 172.21.15.161:0/290106785 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ae80aaf30 con 0x7f5ae8152bc0 2024-03-28T23:37:00.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.151+0000 7f5aeff3e640 1 -- 172.21.15.161:0/290106785 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ae80ab710 con 0x7f5ae8152bc0 2024-03-28T23:37:00.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.151+0000 7f5ade7fc640 1 -- 172.21.15.161:0/290106785 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5ad8007400 con 0x7f5ae8152bc0 2024-03-28T23:37:00.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.152+0000 7f5ade7fc640 1 --2- 172.21.15.161:0/290106785 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5acc06b4d0 0x7f5acc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:00.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.152+0000 7f5ade7fc640 1 -- 172.21.15.161:0/290106785 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7f5ad8089c00 con 0x7f5ae8152bc0 2024-03-28T23:37:00.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.152+0000 7f5aedcda640 1 --2- 172.21.15.161:0/290106785 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5acc06b4d0 0x7f5acc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:00.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.152+0000 7f5aedcda640 1 --2- 172.21.15.161:0/290106785 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5acc06b4d0 0x7f5acc06d990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4009560 tx=0x7f5ae4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:00.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.156+0000 7f5ade7fc640 1 -- 172.21.15.161:0/290106785 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5ae80ab710 con 0x7f5ae8152bc0 2024-03-28T23:37:00.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.332+0000 7f5aeff3e640 1 -- 172.21.15.161:0/290106785 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) -- 0x7f5ae804f4b0 con 0x7f5ae8152bc0 2024-03-28T23:37:00.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.333+0000 7f5ade7fc640 1 -- 172.21.15.161:0/290106785 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v161) ==== 105+0+11 (secure 0 0 0) 0x7f5ad805ce20 con 0x7f5ae8152bc0 2024-03-28T23:37:00.333 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:37:00.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.335+0000 7f5aeff3e640 1 -- 172.21.15.161:0/290106785 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5acc06b4d0 msgr2=0x7f5acc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:00.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.335+0000 7f5aeff3e640 1 --2- 172.21.15.161:0/290106785 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5acc06b4d0 0x7f5acc06d990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4009560 tx=0x7f5ae4009190 comp rx=0 tx=0).stop 2024-03-28T23:37:00.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.335+0000 7f5aeff3e640 1 -- 172.21.15.161:0/290106785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ae8152bc0 msgr2=0x7f5ae806bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:00.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.335+0000 7f5aeff3e640 1 --2- 172.21.15.161:0/290106785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ae8152bc0 0x7f5ae806bd60 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f5ad800c3f0 tx=0x7f5ad800c8b0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.335+0000 7f5aeff3e640 1 -- 172.21.15.161:0/290106785 shutdown_connections 2024-03-28T23:37:00.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.335+0000 7f5aeff3e640 1 --2- 172.21.15.161:0/290106785 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f5acc06b4d0 0x7f5acc06d990 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.335+0000 7f5aeff3e640 1 --2- 172.21.15.161:0/290106785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ae8152bc0 0x7f5ae806bd60 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.335+0000 7f5aeff3e640 1 --2- 172.21.15.161:0/290106785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ae81589b0 0x7f5ae806c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.335+0000 7f5aeff3e640 1 --2- 172.21.15.161:0/290106785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5ae80c8a80 0x7f5ae8147780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.335+0000 7f5aeff3e640 1 -- 172.21.15.161:0/290106785 >> 172.21.15.161:0/290106785 conn(0x7f5ae80caa80 msgr2=0x7f5ae8148140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:00.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.335+0000 7f5aeff3e640 1 -- 172.21.15.161:0/290106785 shutdown_connections 2024-03-28T23:37:00.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.335+0000 7f5aeff3e640 1 -- 172.21.15.161:0/290106785 wait complete. 2024-03-28T23:37:00.346 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-03-28T23:37:00.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.449+0000 7fb8e2607640 1 Processor -- start 2024-03-28T23:37:00.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.449+0000 7fb8e2607640 1 -- start start 2024-03-28T23:37:00.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.450+0000 7fb8e2607640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb8dc156e70 0x7fb8dc153720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:00.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.450+0000 7fb8e2607640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8dc1572a0 con 0x7fb8dc1504f0 2024-03-28T23:37:00.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.450+0000 7fb8e2607640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8dc0c3400 con 0x7fb8dc070cd0 2024-03-28T23:37:00.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.450+0000 7fb8e2607640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fb8dc1508d0 con 0x7fb8dc156e70 2024-03-28T23:37:00.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.450+0000 7fb8dbfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb8dc1504f0 0x7fb8dc0725d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37454/0 (socket says 172.21.15.161:37454) 2024-03-28T23:37:00.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.450+0000 7fb8dbfff640 1 -- 172.21.15.161:0/933841308 learned_addr learned my addr 172.21.15.161:0/933841308 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:00.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.450+0000 7fb8e0ba4640 1 --2- 172.21.15.161:0/933841308 >> v2:172.21.15.170:3300/0 conn(0x7fb8dc156e70 0x7fb8dc153720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:00.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.450+0000 7fb8daffd640 1 -- 172.21.15.161:0/933841308 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2487660302 0 0) 0x7fb8dc1572a0 con 0x7fb8dc1504f0 2024-03-28T23:37:00.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.450+0000 7fb8daffd640 1 -- 172.21.15.161:0/933841308 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8c4003520 con 0x7fb8dc1504f0 2024-03-28T23:37:00.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.450+0000 7fb8e0ba4640 1 -- 172.21.15.161:0/933841308 >> v1:172.21.15.170:6789/0 conn(0x7fb8dc070cd0 legacy=0x7fb8dc153010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:00.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.450+0000 7fb8daffd640 1 -- 172.21.15.161:0/933841308 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2397701646 0 0) 0x7fb8dc0c3400 con 0x7fb8dc070cd0 2024-03-28T23:37:00.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.450+0000 7fb8e0ba4640 1 -- 172.21.15.161:0/933841308 >> v1:172.21.15.161:6789/0 conn(0x7fb8dc1504f0 legacy=0x7fb8dc0725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:00.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.450+0000 7fb8e0ba4640 1 -- 172.21.15.161:0/933841308 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8dc0a8120 con 0x7fb8dc156e70 2024-03-28T23:37:00.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.451+0000 7fb8e0ba4640 1 --2- 172.21.15.161:0/933841308 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8dc156e70 0x7fb8dc153720 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4009d90 tx=0x7fb8d402be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f50c702432fee58c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:00.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.451+0000 7fb8daffd640 1 -- 172.21.15.161:0/933841308 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4039070 con 0x7fb8dc156e70 2024-03-28T23:37:00.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.451+0000 7fb8daffd640 1 -- 172.21.15.161:0/933841308 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8d402cbd0 con 0x7fb8dc156e70 2024-03-28T23:37:00.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.451+0000 7fb8daffd640 1 -- 172.21.15.161:0/933841308 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4034440 con 0x7fb8dc156e70 2024-03-28T23:37:00.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.452+0000 7fb8e2607640 1 -- 172.21.15.161:0/933841308 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8dc156e70 msgr2=0x7fb8dc153720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:00.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.452+0000 7fb8e2607640 1 --2- 172.21.15.161:0/933841308 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8dc156e70 0x7fb8dc153720 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4009d90 tx=0x7fb8d402be20 comp rx=0 tx=0).stop 2024-03-28T23:37:00.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.452+0000 7fb8e2607640 1 -- 172.21.15.161:0/933841308 shutdown_connections 2024-03-28T23:37:00.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.452+0000 7fb8e2607640 1 --2- 172.21.15.161:0/933841308 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8dc156e70 0x7fb8dc153720 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.452+0000 7fb8e2607640 1 -- 172.21.15.161:0/933841308 >> 172.21.15.161:0/933841308 conn(0x7fb8dc0c8810 msgr2=0x7fb8dc0c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:00.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.452+0000 7fb8e2607640 1 -- 172.21.15.161:0/933841308 shutdown_connections 2024-03-28T23:37:00.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.452+0000 7fb8e2607640 1 -- 172.21.15.161:0/933841308 wait complete. 2024-03-28T23:37:00.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.453+0000 7fb8e2607640 1 Processor -- start 2024-03-28T23:37:00.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.453+0000 7fb8e2607640 1 -- start start 2024-03-28T23:37:00.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.454+0000 7fb8e2607640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8dc070cd0 0x7fb8dc1eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:00.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.454+0000 7fb8e2607640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8dc1504f0 0x7fb8dc0c7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:00.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.454+0000 7fb8e2607640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8dc156e70 0x7fb8dc06c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:00.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.454+0000 7fb8e2607640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc0a89c0 con 0x7fb8dc1504f0 2024-03-28T23:37:00.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.454+0000 7fb8e2607640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc0a86e0 con 0x7fb8dc070cd0 2024-03-28T23:37:00.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.454+0000 7fb8e2607640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb8dc153c60 con 0x7fb8dc156e70 2024-03-28T23:37:00.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.454+0000 7fb8db7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8dc1504f0 0x7fb8dc0c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:00.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.454+0000 7fb8db7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8dc1504f0 0x7fb8dc0c7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:53356/0 (socket says 172.21.15.161:53356) 2024-03-28T23:37:00.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.454+0000 7fb8db7fe640 1 -- 172.21.15.161:0/1834634027 learned_addr learned my addr 172.21.15.161:0/1834634027 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:00.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.454+0000 7fb8dbfff640 1 --2- 172.21.15.161:0/1834634027 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8dc070cd0 0x7fb8dc1eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:00.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.454+0000 7fb8e0ba4640 1 --2- 172.21.15.161:0/1834634027 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8dc156e70 0x7fb8dc06c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:00.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.454+0000 7fb8db7fe640 1 -- 172.21.15.161:0/1834634027 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8dc156e70 msgr2=0x7fb8dc06c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:00.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.454+0000 7fb8db7fe640 1 --2- 172.21.15.161:0/1834634027 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8dc156e70 0x7fb8dc06c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.454+0000 7fb8db7fe640 1 -- 172.21.15.161:0/1834634027 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8dc070cd0 msgr2=0x7fb8dc1eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:00.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.454+0000 7fb8db7fe640 1 --2- 172.21.15.161:0/1834634027 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8dc070cd0 0x7fb8dc1eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.454+0000 7fb8db7fe640 1 -- 172.21.15.161:0/1834634027 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8dc1546a0 con 0x7fb8dc1504f0 2024-03-28T23:37:00.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.454+0000 7fb8db7fe640 1 --2- 172.21.15.161:0/1834634027 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8dc1504f0 0x7fb8dc0c7430 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7fb8d000c440 tx=0x7fb8d000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:00.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.455+0000 7fb8d8ff9640 1 -- 172.21.15.161:0/1834634027 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d0017070 con 0x7fb8dc1504f0 2024-03-28T23:37:00.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.455+0000 7fb8d8ff9640 1 -- 172.21.15.161:0/1834634027 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8d0005480 con 0x7fb8dc1504f0 2024-03-28T23:37:00.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.455+0000 7fb8d8ff9640 1 -- 172.21.15.161:0/1834634027 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d0003830 con 0x7fb8dc1504f0 2024-03-28T23:37:00.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.455+0000 7fb8e2607640 1 -- 172.21.15.161:0/1834634027 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8dc0c8130 con 0x7fb8dc1504f0 2024-03-28T23:37:00.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.455+0000 7fb8e2607640 1 -- 172.21.15.161:0/1834634027 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8dc1eb350 con 0x7fb8dc1504f0 2024-03-28T23:37:00.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.456+0000 7fb8e2607640 1 -- 172.21.15.161:0/1834634027 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8dc0725d0 con 0x7fb8dc1504f0 2024-03-28T23:37:00.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.456+0000 7fb8d8ff9640 1 -- 172.21.15.161:0/1834634027 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb8d001a070 con 0x7fb8dc1504f0 2024-03-28T23:37:00.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.457+0000 7fb8d8ff9640 1 --2- 172.21.15.161:0/1834634027 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb8c806b4d0 0x7fb8c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:00.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.457+0000 7fb8d8ff9640 1 -- 172.21.15.161:0/1834634027 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7fb8d0089a00 con 0x7fb8dc1504f0 2024-03-28T23:37:00.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.457+0000 7fb8dbfff640 1 --2- 172.21.15.161:0/1834634027 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb8c806b4d0 0x7fb8c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:00.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.457+0000 7fb8dbfff640 1 --2- 172.21.15.161:0/1834634027 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb8c806b4d0 0x7fb8c806d990 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc005880 tx=0x7fb8cc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:00.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.461+0000 7fb8d8ff9640 1 -- 172.21.15.161:0/1834634027 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb8dc0725d0 con 0x7fb8dc1504f0 2024-03-28T23:37:00.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.639+0000 7fb8e2607640 1 -- 172.21.15.161:0/1834634027 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) -- 0x7fb8dc070ab0 con 0x7fb8dc1504f0 2024-03-28T23:37:00.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.639+0000 7fb8d8ff9640 1 -- 172.21.15.161:0/1834634027 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v161) ==== 101+0+11 (secure 0 0 0) 0x7fb8d0057c60 con 0x7fb8dc1504f0 2024-03-28T23:37:00.640 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:37:00.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.641+0000 7fb8e2607640 1 -- 172.21.15.161:0/1834634027 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb8c806b4d0 msgr2=0x7fb8c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:00.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.641+0000 7fb8e2607640 1 --2- 172.21.15.161:0/1834634027 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb8c806b4d0 0x7fb8c806d990 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc005880 tx=0x7fb8cc009040 comp rx=0 tx=0).stop 2024-03-28T23:37:00.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.641+0000 7fb8e2607640 1 -- 172.21.15.161:0/1834634027 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8dc1504f0 msgr2=0x7fb8dc0c7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:00.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.641+0000 7fb8e2607640 1 --2- 172.21.15.161:0/1834634027 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8dc1504f0 0x7fb8dc0c7430 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7fb8d000c440 tx=0x7fb8d000c900 comp rx=0 tx=0).stop 2024-03-28T23:37:00.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.641+0000 7fb8e2607640 1 -- 172.21.15.161:0/1834634027 shutdown_connections 2024-03-28T23:37:00.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.641+0000 7fb8e2607640 1 --2- 172.21.15.161:0/1834634027 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb8c806b4d0 0x7fb8c806d990 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.641+0000 7fb8e2607640 1 --2- 172.21.15.161:0/1834634027 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8dc1504f0 0x7fb8dc0c7430 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.641+0000 7fb8e2607640 1 --2- 172.21.15.161:0/1834634027 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8dc156e70 0x7fb8dc06c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.641+0000 7fb8e2607640 1 --2- 172.21.15.161:0/1834634027 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8dc070cd0 0x7fb8dc1eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.641+0000 7fb8e2607640 1 -- 172.21.15.161:0/1834634027 >> 172.21.15.161:0/1834634027 conn(0x7fb8dc0c8810 msgr2=0x7fb8dc0c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:00.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.642+0000 7fb8e2607640 1 -- 172.21.15.161:0/1834634027 shutdown_connections 2024-03-28T23:37:00.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.642+0000 7fb8e2607640 1 -- 172.21.15.161:0/1834634027 wait complete. 2024-03-28T23:37:00.653 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:37:00.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.755+0000 7f43f65e9640 1 Processor -- start 2024-03-28T23:37:00.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.755+0000 7f43f65e9640 1 -- start start 2024-03-28T23:37:00.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.755+0000 7f43f65e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43f01532e0 0x7f43f00c14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:00.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.755+0000 7f43f65e9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f43f0153b60 0x7f43f01502a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:00.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.755+0000 7f43f65e9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f43f01507e0 0x7f43f00c2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:00.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.755+0000 7f43f65e9640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f43f00736b0 con 0x7f43f01507e0 2024-03-28T23:37:00.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.755+0000 7f43f65e9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f43f0150c70 con 0x7f43f0153b60 2024-03-28T23:37:00.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.755+0000 7f43f65e9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f43f00730f0 con 0x7f43f01532e0 2024-03-28T23:37:00.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.756+0000 7f43f4b86640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f43f01507e0 0x7f43f00c2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:00.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.756+0000 7f43f4b86640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f43f01507e0 0x7f43f00c2080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37536/0 (socket says 172.21.15.161:37536) 2024-03-28T23:37:00.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.756+0000 7f43f4b86640 1 -- 172.21.15.161:0/959591099 learned_addr learned my addr 172.21.15.161:0/959591099 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:00.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.756+0000 7f43effff640 1 --2- 172.21.15.161:0/959591099 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43f01532e0 0x7f43f00c14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:00.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.756+0000 7f43ef7fe640 1 --2- 172.21.15.161:0/959591099 >> v2:172.21.15.170:3300/0 conn(0x7f43f0153b60 0x7f43f01502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:00.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.756+0000 7f43f4b86640 1 -- 172.21.15.161:0/959591099 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43f01532e0 msgr2=0x7f43f00c14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:00.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.756+0000 7f43f4b86640 1 --2- 172.21.15.161:0/959591099 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43f01532e0 0x7f43f00c14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.756+0000 7f43f4b86640 1 -- 172.21.15.161:0/959591099 >> v2:172.21.15.170:3300/0 conn(0x7f43f0153b60 msgr2=0x7f43f01502a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:00.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.756+0000 7f43f4b86640 1 --2- 172.21.15.161:0/959591099 >> v2:172.21.15.170:3300/0 conn(0x7f43f0153b60 0x7f43f01502a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.756+0000 7f43f4b86640 1 -- 172.21.15.161:0/959591099 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f00733d0 con 0x7f43f01507e0 2024-03-28T23:37:00.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.757+0000 7f43f4b86640 1 --2- 172.21.15.161:0/959591099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43f01507e0 0x7f43f00c2080 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f43f0070620 tx=0x7f43e802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42248938d0c8de46 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:00.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.757+0000 7f43eeffd640 1 -- 172.21.15.161:0/959591099 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e802c9e0 con 0x7f43f01507e0 2024-03-28T23:37:00.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.757+0000 7f43eeffd640 1 -- 172.21.15.161:0/959591099 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43e802cb80 con 0x7f43f01507e0 2024-03-28T23:37:00.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.757+0000 7f43eeffd640 1 -- 172.21.15.161:0/959591099 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e80343f0 con 0x7f43f01507e0 2024-03-28T23:37:00.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.757+0000 7f43f65e9640 1 -- 172.21.15.161:0/959591099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43f01507e0 msgr2=0x7f43f00c2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:00.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.757+0000 7f43f65e9640 1 --2- 172.21.15.161:0/959591099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43f01507e0 0x7f43f00c2080 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f43f0070620 tx=0x7f43e802be40 comp rx=0 tx=0).stop 2024-03-28T23:37:00.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.757+0000 7f43f65e9640 1 -- 172.21.15.161:0/959591099 shutdown_connections 2024-03-28T23:37:00.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.757+0000 7f43f65e9640 1 --2- 172.21.15.161:0/959591099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43f01507e0 0x7f43f00c2080 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.757+0000 7f43f65e9640 1 --2- 172.21.15.161:0/959591099 >> v2:172.21.15.170:3300/0 conn(0x7f43f0153b60 0x7f43f01502a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.757+0000 7f43f65e9640 1 --2- 172.21.15.161:0/959591099 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43f01532e0 0x7f43f00c14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.757+0000 7f43f65e9640 1 -- 172.21.15.161:0/959591099 >> 172.21.15.161:0/959591099 conn(0x7f43f00c6b40 msgr2=0x7f43f00c6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:00.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.757+0000 7f43f65e9640 1 -- 172.21.15.161:0/959591099 shutdown_connections 2024-03-28T23:37:00.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.757+0000 7f43f65e9640 1 -- 172.21.15.161:0/959591099 wait complete. 2024-03-28T23:37:00.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.758+0000 7f43f65e9640 1 Processor -- start 2024-03-28T23:37:00.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.759+0000 7f43f65e9640 1 -- start start 2024-03-28T23:37:00.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.759+0000 7f43f65e9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43f01507e0 0x7f43f0152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:00.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.759+0000 7f43f65e9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43f01532e0 0x7f43f00c58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:00.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.759+0000 7f43f65e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43f0153b60 0x7f43f0156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:00.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.759+0000 7f43f65e9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f43f00c1a00 con 0x7f43f01507e0 2024-03-28T23:37:00.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.759+0000 7f43f65e9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f43f0070d40 con 0x7f43f01532e0 2024-03-28T23:37:00.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.759+0000 7f43f65e9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f43f00c1b80 con 0x7f43f0153b60 2024-03-28T23:37:00.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.759+0000 7f43effff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43f01507e0 0x7f43f0152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:00.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.759+0000 7f43effff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43f01507e0 0x7f43f0152130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37550/0 (socket says 172.21.15.161:37550) 2024-03-28T23:37:00.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.759+0000 7f43effff640 1 -- 172.21.15.161:0/3725740525 learned_addr learned my addr 172.21.15.161:0/3725740525 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:00.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.759+0000 7f43f4b86640 1 --2- 172.21.15.161:0/3725740525 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43f0153b60 0x7f43f0156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:00.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.760+0000 7f43ef7fe640 1 --2- 172.21.15.161:0/3725740525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43f01532e0 0x7f43f00c58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:00.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.760+0000 7f43effff640 1 -- 172.21.15.161:0/3725740525 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43f0153b60 msgr2=0x7f43f0156a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:00.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.760+0000 7f43effff640 1 --2- 172.21.15.161:0/3725740525 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43f0153b60 0x7f43f0156a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.760+0000 7f43effff640 1 -- 172.21.15.161:0/3725740525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43f01532e0 msgr2=0x7f43f00c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:00.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.760+0000 7f43effff640 1 --2- 172.21.15.161:0/3725740525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43f01532e0 0x7f43f00c58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.760+0000 7f43effff640 1 -- 172.21.15.161:0/3725740525 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f014a390 con 0x7f43f01507e0 2024-03-28T23:37:00.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.760+0000 7f43effff640 1 --2- 172.21.15.161:0/3725740525 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43f01507e0 0x7f43f0152130 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f43e000e870 tx=0x7f43e000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:00.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.760+0000 7f43ecff9640 1 -- 172.21.15.161:0/3725740525 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e0017070 con 0x7f43f01507e0 2024-03-28T23:37:00.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.760+0000 7f43ecff9640 1 -- 172.21.15.161:0/3725740525 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43e000c8e0 con 0x7f43f01507e0 2024-03-28T23:37:00.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.760+0000 7f43f65e9640 1 -- 172.21.15.161:0/3725740525 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43f00c48b0 con 0x7f43f01507e0 2024-03-28T23:37:00.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.760+0000 7f43ecff9640 1 -- 172.21.15.161:0/3725740525 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e000ccd0 con 0x7f43f01507e0 2024-03-28T23:37:00.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.760+0000 7f43f65e9640 1 -- 172.21.15.161:0/3725740525 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43f00c4e70 con 0x7f43f01507e0 2024-03-28T23:37:00.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.761+0000 7f43f65e9640 1 -- 172.21.15.161:0/3725740525 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43f00a6400 con 0x7f43f01507e0 2024-03-28T23:37:00.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.762+0000 7f43ecff9640 1 -- 172.21.15.161:0/3725740525 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f43e001a070 con 0x7f43f01507e0 2024-03-28T23:37:00.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.762+0000 7f43ecff9640 1 --2- 172.21.15.161:0/3725740525 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f43dc06b500 0x7f43dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:00.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.762+0000 7f43ecff9640 1 -- 172.21.15.161:0/3725740525 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7f43e008a9c0 con 0x7f43f01507e0 2024-03-28T23:37:00.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.762+0000 7f43ef7fe640 1 --2- 172.21.15.161:0/3725740525 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f43dc06b500 0x7f43dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:00.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.762+0000 7f43ef7fe640 1 --2- 172.21.15.161:0/3725740525 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f43dc06b500 0x7f43dc06d9c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f43f0152960 tx=0x7f43e4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:00.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.766+0000 7f43ecff9640 1 -- 172.21.15.161:0/3725740525 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f43f00a6400 con 0x7f43f01507e0 2024-03-28T23:37:00.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.980+0000 7f43f65e9640 1 -- 172.21.15.161:0/3725740525 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43f00c0e40 con 0x7f43f01507e0 2024-03-28T23:37:00.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.981+0000 7f43ecff9640 1 -- 172.21.15.161:0/3725740525 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+6520 (secure 0 0 0) 0x7f43e008ce20 con 0x7f43f01507e0 2024-03-28T23:37:00.981 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:37:00.981 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":154,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/557522063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":557522063},{"type":"v1","addr":"172.21.15.170:6835","nonce":557522063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":9216,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/789976100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":789976100},{"type":"v1","addr":"172.21.15.161:6837","nonce":789976100}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:58.406403+0000","modified":"2024-03-28T23:36:20.440473+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":159,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7283":{"gid":7283,"name":"b","rank":0,"incarnation":151,"state":"up:active","state_seq":4,"addr":"172.21.15.170:6834/2996092857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":2996092857},{"type":"v1","addr":"172.21.15.170:6834","nonce":2996092857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7283,"qdb_cluster":[7283]},"id":11},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:36:07.448976+0000","modified":"2024-03-28T23:36:16.420228+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7061":{"gid":7061,"name":"c","rank":0,"incarnation":145,"state":"up:active","state_seq":9,"addr":"172.21.15.161:6836/4026332703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4026332703},{"type":"v1","addr":"172.21.15.161:6836","nonce":4026332703}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7061,"qdb_cluster":[7061]},"id":12}]} 2024-03-28T23:37:00.982 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 154 2024-03-28T23:37:00.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.983+0000 7f43f65e9640 1 -- 172.21.15.161:0/3725740525 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f43dc06b500 msgr2=0x7f43dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:00.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.983+0000 7f43f65e9640 1 --2- 172.21.15.161:0/3725740525 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f43dc06b500 0x7f43dc06d9c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f43f0152960 tx=0x7f43e4009190 comp rx=0 tx=0).stop 2024-03-28T23:37:00.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.983+0000 7f43f65e9640 1 -- 172.21.15.161:0/3725740525 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43f01507e0 msgr2=0x7f43f0152130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:00.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.983+0000 7f43f65e9640 1 --2- 172.21.15.161:0/3725740525 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43f01507e0 0x7f43f0152130 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f43e000e870 tx=0x7f43e000ed30 comp rx=0 tx=0).stop 2024-03-28T23:37:00.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.983+0000 7f43f65e9640 1 -- 172.21.15.161:0/3725740525 shutdown_connections 2024-03-28T23:37:00.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.983+0000 7f43f65e9640 1 --2- 172.21.15.161:0/3725740525 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f43dc06b500 0x7f43dc06d9c0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.983+0000 7f43f65e9640 1 --2- 172.21.15.161:0/3725740525 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43f0153b60 0x7f43f0156a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.983+0000 7f43f65e9640 1 --2- 172.21.15.161:0/3725740525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43f01532e0 0x7f43f00c58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.983+0000 7f43f65e9640 1 --2- 172.21.15.161:0/3725740525 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43f01507e0 0x7f43f0152130 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:00.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.983+0000 7f43f65e9640 1 -- 172.21.15.161:0/3725740525 >> 172.21.15.161:0/3725740525 conn(0x7f43f00c6b40 msgr2=0x7f43f00c2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:00.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.983+0000 7f43f65e9640 1 -- 172.21.15.161:0/3725740525 shutdown_connections 2024-03-28T23:37:00.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:00.983+0000 7f43f65e9640 1 -- 172.21.15.161:0/3725740525 wait complete. 2024-03-28T23:37:00.994 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:37:01.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.096+0000 7f2051f48640 1 Processor -- start 2024-03-28T23:37:01.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.097+0000 7f2051f48640 1 -- start start 2024-03-28T23:37:01.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.097+0000 7f2051f48640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f204c0c5850 0x7f204c155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:01.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.097+0000 7f2051f48640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f204c0c5d00 con 0x7f204c0c5410 2024-03-28T23:37:01.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.097+0000 7f2051f48640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f204c158a30 con 0x7f204c152890 2024-03-28T23:37:01.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.097+0000 7f2051f48640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f204c0c8480 con 0x7f204c0c5850 2024-03-28T23:37:01.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.097+0000 7f2043fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f204c0c5410 0x7f204c1553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48094/0 (socket says 172.21.15.161:48094) 2024-03-28T23:37:01.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.097+0000 7f2043fff640 1 -- 172.21.15.161:0/3112858492 learned_addr learned my addr 172.21.15.161:0/3112858492 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:01.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.097+0000 7f204bfff640 1 --2- 172.21.15.161:0/3112858492 >> v2:172.21.15.170:3300/0 conn(0x7f204c0c5850 0x7f204c155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:01.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.098+0000 7f204affd640 1 -- 172.21.15.161:0/3112858492 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3519417938 0 0) 0x7f204c0c5d00 con 0x7f204c0c5410 2024-03-28T23:37:01.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.098+0000 7f204affd640 1 -- 172.21.15.161:0/3112858492 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2038003520 con 0x7f204c0c5410 2024-03-28T23:37:01.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.098+0000 7f204affd640 1 -- 172.21.15.161:0/3112858492 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4170103281 0 0) 0x7f204c158a30 con 0x7f204c152890 2024-03-28T23:37:01.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.098+0000 7f204affd640 1 -- 172.21.15.161:0/3112858492 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f204c0c5d00 con 0x7f204c152890 2024-03-28T23:37:01.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.098+0000 7f204bfff640 1 -- 172.21.15.161:0/3112858492 >> v1:172.21.15.170:6789/0 conn(0x7f204c152890 legacy=0x7f204c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:01.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.098+0000 7f204bfff640 1 -- 172.21.15.161:0/3112858492 >> v1:172.21.15.161:6789/0 conn(0x7f204c0c5410 legacy=0x7f204c1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:01.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.098+0000 7f204bfff640 1 -- 172.21.15.161:0/3112858492 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f204c158d10 con 0x7f204c0c5850 2024-03-28T23:37:01.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.099+0000 7f204bfff640 1 --2- 172.21.15.161:0/3112858492 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f204c0c5850 0x7f204c155ab0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2044009d90 tx=0x7f204402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd6e23589def923a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:01.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.099+0000 7f204affd640 1 -- 172.21.15.161:0/3112858492 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2044039070 con 0x7f204c0c5850 2024-03-28T23:37:01.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.099+0000 7f204affd640 1 -- 172.21.15.161:0/3112858492 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f204402cc60 con 0x7f204c0c5850 2024-03-28T23:37:01.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.099+0000 7f204affd640 1 -- 172.21.15.161:0/3112858492 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2044034660 con 0x7f204c0c5850 2024-03-28T23:37:01.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.099+0000 7f2051f48640 1 -- 172.21.15.161:0/3112858492 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f204c0c5850 msgr2=0x7f204c155ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:01.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.099+0000 7f2051f48640 1 --2- 172.21.15.161:0/3112858492 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f204c0c5850 0x7f204c155ab0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2044009d90 tx=0x7f204402beb0 comp rx=0 tx=0).stop 2024-03-28T23:37:01.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.099+0000 7f2051f48640 1 -- 172.21.15.161:0/3112858492 shutdown_connections 2024-03-28T23:37:01.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.099+0000 7f2051f48640 1 --2- 172.21.15.161:0/3112858492 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f204c0c5850 0x7f204c155ab0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:01.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.099+0000 7f2051f48640 1 -- 172.21.15.161:0/3112858492 >> 172.21.15.161:0/3112858492 conn(0x7f204c0ac620 msgr2=0x7f204c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:01.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.099+0000 7f2051f48640 1 -- 172.21.15.161:0/3112858492 shutdown_connections 2024-03-28T23:37:01.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.099+0000 7f2051f48640 1 -- 172.21.15.161:0/3112858492 wait complete. 2024-03-28T23:37:01.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.100+0000 7f2051f48640 1 Processor -- start 2024-03-28T23:37:01.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.100+0000 7f2051f48640 1 -- start start 2024-03-28T23:37:01.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.101+0000 7f2051f48640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f204c0c5410 0x7f204c0abc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:01.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.101+0000 7f2051f48640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f204c0c5850 0x7f204c1eaf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:01.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.101+0000 7f2051f48640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f204c152890 0x7f204c1eb480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.101+0000 7f2051f48640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f204c1592d0 con 0x7f204c0c5410 2024-03-28T23:37:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.101+0000 7f2051f48640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f204c159640 con 0x7f204c152890 2024-03-28T23:37:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.101+0000 7f2051f48640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f204c155ff0 con 0x7f204c0c5850 2024-03-28T23:37:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.101+0000 7f204b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f204c0c5410 0x7f204c0abc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.101+0000 7f204b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f204c0c5410 0x7f204c0abc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37562/0 (socket says 172.21.15.161:37562) 2024-03-28T23:37:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.101+0000 7f204b7fe640 1 -- 172.21.15.161:0/2095328064 learned_addr learned my addr 172.21.15.161:0/2095328064 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.101+0000 7f204bfff640 1 --2- 172.21.15.161:0/2095328064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f204c152890 0x7f204c1eb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.101+0000 7f2043fff640 1 --2- 172.21.15.161:0/2095328064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f204c0c5850 0x7f204c1eaf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.102+0000 7f204b7fe640 1 -- 172.21.15.161:0/2095328064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f204c0c5850 msgr2=0x7f204c1eaf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.102+0000 7f204b7fe640 1 --2- 172.21.15.161:0/2095328064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f204c0c5850 0x7f204c1eaf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.102+0000 7f204b7fe640 1 -- 172.21.15.161:0/2095328064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f204c152890 msgr2=0x7f204c1eb480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.102+0000 7f204b7fe640 1 --2- 172.21.15.161:0/2095328064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f204c152890 0x7f204c1eb480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.102+0000 7f204b7fe640 1 -- 172.21.15.161:0/2095328064 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f204c1eedf0 con 0x7f204c0c5410 2024-03-28T23:37:01.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.102+0000 7f204b7fe640 1 --2- 172.21.15.161:0/2095328064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f204c0c5410 0x7f204c0abc90 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f203c00c530 tx=0x7f203c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:01.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.102+0000 7f2048ff9640 1 -- 172.21.15.161:0/2095328064 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f203c0052d0 con 0x7f204c0c5410 2024-03-28T23:37:01.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.102+0000 7f2048ff9640 1 -- 172.21.15.161:0/2095328064 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f203c005490 con 0x7f204c0c5410 2024-03-28T23:37:01.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.102+0000 7f2051f48640 1 -- 172.21.15.161:0/2095328064 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f204c1ef090 con 0x7f204c0c5410 2024-03-28T23:37:01.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.102+0000 7f2048ff9640 1 -- 172.21.15.161:0/2095328064 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f203c012870 con 0x7f204c0c5410 2024-03-28T23:37:01.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.102+0000 7f2051f48640 1 -- 172.21.15.161:0/2095328064 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f204c154e40 con 0x7f204c0c5410 2024-03-28T23:37:01.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.103+0000 7f2051f48640 1 -- 172.21.15.161:0/2095328064 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f204c155040 con 0x7f204c0c5410 2024-03-28T23:37:01.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.103+0000 7f2048ff9640 1 -- 172.21.15.161:0/2095328064 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f203c012a10 con 0x7f204c0c5410 2024-03-28T23:37:01.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.104+0000 7f2048ff9640 1 --2- 172.21.15.161:0/2095328064 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f202c06b500 0x7f202c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:01.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.104+0000 7f2043fff640 1 --2- 172.21.15.161:0/2095328064 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f202c06b500 0x7f202c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:01.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.104+0000 7f2048ff9640 1 -- 172.21.15.161:0/2095328064 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7f203c08a300 con 0x7f204c0c5410 2024-03-28T23:37:01.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.104+0000 7f2043fff640 1 --2- 172.21.15.161:0/2095328064 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f202c06b500 0x7f202c06d9c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f204c14cbe0 tx=0x7f2034007be0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:01.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.108+0000 7f2048ff9640 1 -- 172.21.15.161:0/2095328064 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f204c155040 con 0x7f204c0c5410 2024-03-28T23:37:01.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.447+0000 7f2051f48640 1 -- 172.21.15.161:0/2095328064 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f204c14f6d0 con 0x7f204c0c5410 2024-03-28T23:37:01.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.448+0000 7f2048ff9640 1 -- 172.21.15.161:0/2095328064 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v161) ==== 74+0+21282 (secure 0 0 0) 0x7f203c0584e0 con 0x7f204c0c5410 2024-03-28T23:37:01.449 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:37:01.449 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":161,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:36:36.466146+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:35:56.687416+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-28T23:35:57.379120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:36:05.484451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:36:06.423073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.c","mappings":[{"from":1,"to":0}]},{"pgid":"26.13","mappings":[{"from":1,"to":6}]},{"pgid":"26.28","mappings":[{"from":1,"to":0}]},{"pgid":"26.3b","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6837/3236742821":"2024-03-29T23:36:17.431214+0000","172.21.15.161:6835/3236742821":"2024-03-29T23:36:17.431214+0000","172.21.15.170:6834/1393137291":"2024-03-29T23:36:14.412210+0000","172.21.15.170:6832/1393137291":"2024-03-29T23:36:14.412210+0000","172.21.15.161:6837/2391019771":"2024-03-29T23:36:13.968386+0000","172.21.15.161:6835/2391019771":"2024-03-29T23:36:13.968386+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:37:01.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.450+0000 7f2051f48640 1 -- 172.21.15.161:0/2095328064 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f202c06b500 msgr2=0x7f202c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:01.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.450+0000 7f2051f48640 1 --2- 172.21.15.161:0/2095328064 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f202c06b500 0x7f202c06d9c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f204c14cbe0 tx=0x7f2034007be0 comp rx=0 tx=0).stop 2024-03-28T23:37:01.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.450+0000 7f2051f48640 1 -- 172.21.15.161:0/2095328064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f204c0c5410 msgr2=0x7f204c0abc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:01.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.450+0000 7f2051f48640 1 --2- 172.21.15.161:0/2095328064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f204c0c5410 0x7f204c0abc90 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f203c00c530 tx=0x7f203c00c9f0 comp rx=0 tx=0).stop 2024-03-28T23:37:01.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.450+0000 7f2051f48640 1 -- 172.21.15.161:0/2095328064 shutdown_connections 2024-03-28T23:37:01.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.450+0000 7f2051f48640 1 --2- 172.21.15.161:0/2095328064 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f202c06b500 0x7f202c06d9c0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:01.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.450+0000 7f2051f48640 1 --2- 172.21.15.161:0/2095328064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f204c152890 0x7f204c1eb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:01.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.450+0000 7f2051f48640 1 --2- 172.21.15.161:0/2095328064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f204c0c5850 0x7f204c1eaf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:01.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.450+0000 7f2051f48640 1 --2- 172.21.15.161:0/2095328064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f204c0c5410 0x7f204c0abc90 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:01.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.450+0000 7f2051f48640 1 -- 172.21.15.161:0/2095328064 >> 172.21.15.161:0/2095328064 conn(0x7f204c0ac620 msgr2=0x7f204c06be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:01.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.450+0000 7f2051f48640 1 -- 172.21.15.161:0/2095328064 shutdown_connections 2024-03-28T23:37:01.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.450+0000 7f2051f48640 1 -- 172.21.15.161:0/2095328064 wait complete. 2024-03-28T23:37:01.462 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-28T23:37:01.462 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:37:01.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.564+0000 7f263c8c0640 1 Processor -- start 2024-03-28T23:37:01.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.564+0000 7f263c8c0640 1 -- start start 2024-03-28T23:37:01.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.565+0000 7f263c8c0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2638152890 0x7f26380c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:01.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.565+0000 7f263c8c0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2638158a30 con 0x7f26380c5b20 2024-03-28T23:37:01.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.565+0000 7f263c8c0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2638158d10 con 0x7f26380c8ed0 2024-03-28T23:37:01.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.565+0000 7f263c8c0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f26380c8480 con 0x7f2638152890 2024-03-28T23:37:01.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.565+0000 7f2636d9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f26380c5b20 0x7f26381553a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48110/0 (socket says 172.21.15.161:48110) 2024-03-28T23:37:01.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.565+0000 7f2636d9d640 1 -- 172.21.15.161:0/2092572668 learned_addr learned my addr 172.21.15.161:0/2092572668 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:01.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.565+0000 7f263659c640 1 --2- 172.21.15.161:0/2092572668 >> v2:172.21.15.170:3300/0 conn(0x7f2638152890 0x7f26380c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:01.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.565+0000 7f263559a640 1 -- 172.21.15.161:0/2092572668 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4007553892 0 0) 0x7f2638158a30 con 0x7f26380c5b20 2024-03-28T23:37:01.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.565+0000 7f263559a640 1 -- 172.21.15.161:0/2092572668 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2624003520 con 0x7f26380c5b20 2024-03-28T23:37:01.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.565+0000 7f263559a640 1 -- 172.21.15.161:0/2092572668 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1484693730 0 0) 0x7f2638158d10 con 0x7f26380c8ed0 2024-03-28T23:37:01.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.565+0000 7f263559a640 1 -- 172.21.15.161:0/2092572668 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2638158a30 con 0x7f26380c8ed0 2024-03-28T23:37:01.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.565+0000 7f263659c640 1 -- 172.21.15.161:0/2092572668 >> v1:172.21.15.170:6789/0 conn(0x7f26380c8ed0 legacy=0x7f26380c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:01.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.565+0000 7f263659c640 1 -- 172.21.15.161:0/2092572668 >> v1:172.21.15.161:6789/0 conn(0x7f26380c5b20 legacy=0x7f26381553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:01.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.565+0000 7f263659c640 1 -- 172.21.15.161:0/2092572668 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2638158ff0 con 0x7f2638152890 2024-03-28T23:37:01.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.566+0000 7f263659c640 1 --2- 172.21.15.161:0/2092572668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2638152890 0x7f26380c8990 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f26200097a0 tx=0x7f262002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e8ccbe1c109b172 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:01.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.566+0000 7f263559a640 1 -- 172.21.15.161:0/2092572668 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26200315a0 con 0x7f2638152890 2024-03-28T23:37:01.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.566+0000 7f263559a640 1 -- 172.21.15.161:0/2092572668 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2620037070 con 0x7f2638152890 2024-03-28T23:37:01.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.566+0000 7f263559a640 1 -- 172.21.15.161:0/2092572668 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2620031960 con 0x7f2638152890 2024-03-28T23:37:01.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.567+0000 7f263c8c0640 1 -- 172.21.15.161:0/2092572668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2638152890 msgr2=0x7f26380c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:01.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.567+0000 7f263c8c0640 1 --2- 172.21.15.161:0/2092572668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2638152890 0x7f26380c8990 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f26200097a0 tx=0x7f262002b840 comp rx=0 tx=0).stop 2024-03-28T23:37:01.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.567+0000 7f263c8c0640 1 -- 172.21.15.161:0/2092572668 shutdown_connections 2024-03-28T23:37:01.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.567+0000 7f263c8c0640 1 --2- 172.21.15.161:0/2092572668 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2638152890 0x7f26380c8990 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:01.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.567+0000 7f263c8c0640 1 -- 172.21.15.161:0/2092572668 >> 172.21.15.161:0/2092572668 conn(0x7f26380ac620 msgr2=0x7f26380aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:01.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.567+0000 7f263c8c0640 1 -- 172.21.15.161:0/2092572668 shutdown_connections 2024-03-28T23:37:01.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.567+0000 7f263c8c0640 1 -- 172.21.15.161:0/2092572668 wait complete. 2024-03-28T23:37:01.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.568+0000 7f263c8c0640 1 Processor -- start 2024-03-28T23:37:01.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.568+0000 7f263c8c0640 1 -- start start 2024-03-28T23:37:01.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.569+0000 7f263c8c0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26380c5b20 0x7f2638149520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:01.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.569+0000 7f263c8c0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26380c8ed0 0x7f263814f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:01.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.569+0000 7f263c8c0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2638152890 0x7f263814f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:01.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.569+0000 7f263c8c0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2638152c70 con 0x7f2638152890 2024-03-28T23:37:01.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.569+0000 7f263c8c0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2638159640 con 0x7f26380c8ed0 2024-03-28T23:37:01.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.569+0000 7f263c8c0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2638152df0 con 0x7f26380c5b20 2024-03-28T23:37:01.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.569+0000 7f2636d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2638152890 0x7f263814f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:01.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.569+0000 7f2635d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26380c8ed0 0x7f263814f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:01.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.569+0000 7f2636d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2638152890 0x7f263814f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37568/0 (socket says 172.21.15.161:37568) 2024-03-28T23:37:01.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.569+0000 7f263659c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26380c5b20 0x7f2638149520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:01.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.569+0000 7f2636d9d640 1 -- 172.21.15.161:0/3145219749 learned_addr learned my addr 172.21.15.161:0/3145219749 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:01.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.569+0000 7f2635d9b640 1 -- 172.21.15.161:0/3145219749 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26380c5b20 msgr2=0x7f2638149520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:01.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.569+0000 7f2635d9b640 1 --2- 172.21.15.161:0/3145219749 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26380c5b20 0x7f2638149520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:01.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.569+0000 7f2635d9b640 1 -- 172.21.15.161:0/3145219749 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2638152890 msgr2=0x7f263814f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:01.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.569+0000 7f2635d9b640 1 --2- 172.21.15.161:0/3145219749 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2638152890 0x7f263814f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:01.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.569+0000 7f2635d9b640 1 -- 172.21.15.161:0/3145219749 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26381edf20 con 0x7f26380c8ed0 2024-03-28T23:37:01.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.569+0000 7f2636d9d640 1 --2- 172.21.15.161:0/3145219749 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2638152890 0x7f263814f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:37:01.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.570+0000 7f2635d9b640 1 --2- 172.21.15.161:0/3145219749 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26380c8ed0 0x7f263814f250 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2628007fd0 tx=0x7f262800c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:01.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.570+0000 7f262effd640 1 -- 172.21.15.161:0/3145219749 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2628017070 con 0x7f26380c8ed0 2024-03-28T23:37:01.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.570+0000 7f262effd640 1 -- 172.21.15.161:0/3145219749 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26280052f0 con 0x7f26380c8ed0 2024-03-28T23:37:01.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.570+0000 7f263c8c0640 1 -- 172.21.15.161:0/3145219749 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26381ee200 con 0x7f26380c8ed0 2024-03-28T23:37:01.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.570+0000 7f262effd640 1 -- 172.21.15.161:0/3145219749 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2628003640 con 0x7f26380c8ed0 2024-03-28T23:37:01.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.570+0000 7f263c8c0640 1 -- 172.21.15.161:0/3145219749 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26381ee7c0 con 0x7f26380c8ed0 2024-03-28T23:37:01.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.571+0000 7f263c8c0640 1 -- 172.21.15.161:0/3145219749 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26381eeaf0 con 0x7f26380c8ed0 2024-03-28T23:37:01.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.571+0000 7f262effd640 1 -- 172.21.15.161:0/3145219749 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f262801a070 con 0x7f26380c8ed0 2024-03-28T23:37:01.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.572+0000 7f262effd640 1 --2- 172.21.15.161:0/3145219749 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f261806b500 0x7f261806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:01.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.572+0000 7f262effd640 1 -- 172.21.15.161:0/3145219749 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7f2628089aa0 con 0x7f26380c8ed0 2024-03-28T23:37:01.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.572+0000 7f263659c640 1 --2- 172.21.15.161:0/3145219749 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f261806b500 0x7f261806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:01.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.572+0000 7f263659c640 1 --2- 172.21.15.161:0/3145219749 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f261806b500 0x7f261806d9c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f2620036000 tx=0x7f262003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:01.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.576+0000 7f262effd640 1 -- 172.21.15.161:0/3145219749 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2628057d00 con 0x7f26380c8ed0 2024-03-28T23:37:01.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.791+0000 7f263c8c0640 1 -- 172.21.15.161:0/3145219749 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f263804f4b0 con 0x7f26380c8ed0 2024-03-28T23:37:01.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.792+0000 7f262effd640 1 -- 172.21.15.161:0/3145219749 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+6520 (secure 0 0 0) 0x7f262805bda0 con 0x7f26380c8ed0 2024-03-28T23:37:01.793 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:37:01.793 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":154,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/557522063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":557522063},{"type":"v1","addr":"172.21.15.170:6835","nonce":557522063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":9216,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/789976100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":789976100},{"type":"v1","addr":"172.21.15.161:6837","nonce":789976100}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:58.406403+0000","modified":"2024-03-28T23:36:20.440473+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":159,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7283":{"gid":7283,"name":"b","rank":0,"incarnation":151,"state":"up:active","state_seq":4,"addr":"172.21.15.170:6834/2996092857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":2996092857},{"type":"v1","addr":"172.21.15.170:6834","nonce":2996092857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7283,"qdb_cluster":[7283]},"id":11},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:36:07.448976+0000","modified":"2024-03-28T23:36:16.420228+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7061":{"gid":7061,"name":"c","rank":0,"incarnation":145,"state":"up:active","state_seq":9,"addr":"172.21.15.161:6836/4026332703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4026332703},{"type":"v1","addr":"172.21.15.161:6836","nonce":4026332703}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7061,"qdb_cluster":[7061]},"id":12}]} 2024-03-28T23:37:01.793 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 154 2024-03-28T23:37:01.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.794+0000 7f263c8c0640 1 -- 172.21.15.161:0/3145219749 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f261806b500 msgr2=0x7f261806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:01.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.794+0000 7f263c8c0640 1 --2- 172.21.15.161:0/3145219749 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f261806b500 0x7f261806d9c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f2620036000 tx=0x7f262003d040 comp rx=0 tx=0).stop 2024-03-28T23:37:01.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.794+0000 7f263c8c0640 1 -- 172.21.15.161:0/3145219749 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26380c8ed0 msgr2=0x7f263814f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:01.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.794+0000 7f263c8c0640 1 --2- 172.21.15.161:0/3145219749 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26380c8ed0 0x7f263814f250 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2628007fd0 tx=0x7f262800c820 comp rx=0 tx=0).stop 2024-03-28T23:37:01.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.794+0000 7f263c8c0640 1 -- 172.21.15.161:0/3145219749 shutdown_connections 2024-03-28T23:37:01.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.794+0000 7f263c8c0640 1 --2- 172.21.15.161:0/3145219749 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f261806b500 0x7f261806d9c0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:01.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.794+0000 7f263c8c0640 1 --2- 172.21.15.161:0/3145219749 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2638152890 0x7f263814f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:01.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.794+0000 7f263c8c0640 1 --2- 172.21.15.161:0/3145219749 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26380c8ed0 0x7f263814f250 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:01.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.794+0000 7f263c8c0640 1 --2- 172.21.15.161:0/3145219749 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26380c5b20 0x7f2638149520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:01.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.794+0000 7f263c8c0640 1 -- 172.21.15.161:0/3145219749 >> 172.21.15.161:0/3145219749 conn(0x7f26380ac620 msgr2=0x7f263814c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:01.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.794+0000 7f263c8c0640 1 -- 172.21.15.161:0/3145219749 shutdown_connections 2024-03-28T23:37:01.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.794+0000 7f263c8c0640 1 -- 172.21.15.161:0/3145219749 wait complete. 2024-03-28T23:37:01.806 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:37:01.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.910+0000 7fa3e6f79640 1 Processor -- start 2024-03-28T23:37:01.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.910+0000 7fa3e6f79640 1 -- start start 2024-03-28T23:37:01.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.911+0000 7fa3e6f79640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa3e0152890 0x7fa3e00c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:01.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.911+0000 7fa3e6f79640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3e0158a30 con 0x7fa3e00c8ed0 2024-03-28T23:37:01.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.911+0000 7fa3e6f79640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3e0158d10 con 0x7fa3e00c5b20 2024-03-28T23:37:01.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.911+0000 7fa3e6f79640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fa3e00c8480 con 0x7fa3e0152890 2024-03-28T23:37:01.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.911+0000 7fa3d7fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa3e00c8ed0 0x7fa3e00c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48114/0 (socket says 172.21.15.161:48114) 2024-03-28T23:37:01.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.911+0000 7fa3d7fff640 1 -- 172.21.15.161:0/2086690188 learned_addr learned my addr 172.21.15.161:0/2086690188 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:01.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.911+0000 7fa3e4d15640 1 --2- 172.21.15.161:0/2086690188 >> v2:172.21.15.170:3300/0 conn(0x7fa3e0152890 0x7fa3e00c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:01.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.911+0000 7fa3d77fe640 1 -- 172.21.15.161:0/2086690188 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1587035531 0 0) 0x7fa3e0158a30 con 0x7fa3e00c8ed0 2024-03-28T23:37:01.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.911+0000 7fa3d77fe640 1 -- 172.21.15.161:0/2086690188 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3c8003520 con 0x7fa3e00c8ed0 2024-03-28T23:37:01.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.911+0000 7fa3e4d15640 1 -- 172.21.15.161:0/2086690188 >> v1:172.21.15.170:6789/0 conn(0x7fa3e00c5b20 legacy=0x7fa3e01553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:01.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.911+0000 7fa3d77fe640 1 -- 172.21.15.161:0/2086690188 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 464171429 0 0) 0x7fa3e0158d10 con 0x7fa3e00c5b20 2024-03-28T23:37:01.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.911+0000 7fa3e4d15640 1 -- 172.21.15.161:0/2086690188 >> v1:172.21.15.161:6789/0 conn(0x7fa3e00c8ed0 legacy=0x7fa3e00c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:01.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.911+0000 7fa3e4d15640 1 -- 172.21.15.161:0/2086690188 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3e0158ff0 con 0x7fa3e0152890 2024-03-28T23:37:01.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.912+0000 7fa3e4d15640 1 --2- 172.21.15.161:0/2086690188 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3e0152890 0x7fa3e00c8990 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc0097a0 tx=0x7fa3dc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bba31cf8b515ba3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:01.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.913+0000 7fa3d77fe640 1 -- 172.21.15.161:0/2086690188 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc0315a0 con 0x7fa3e0152890 2024-03-28T23:37:01.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.913+0000 7fa3d77fe640 1 -- 172.21.15.161:0/2086690188 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3dc037070 con 0x7fa3e0152890 2024-03-28T23:37:01.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.913+0000 7fa3d77fe640 1 -- 172.21.15.161:0/2086690188 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc031960 con 0x7fa3e0152890 2024-03-28T23:37:01.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.913+0000 7fa3e6f79640 1 -- 172.21.15.161:0/2086690188 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3e0152890 msgr2=0x7fa3e00c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:01.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.913+0000 7fa3e6f79640 1 --2- 172.21.15.161:0/2086690188 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3e0152890 0x7fa3e00c8990 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc0097a0 tx=0x7fa3dc02b840 comp rx=0 tx=0).stop 2024-03-28T23:37:01.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.913+0000 7fa3e6f79640 1 -- 172.21.15.161:0/2086690188 shutdown_connections 2024-03-28T23:37:01.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.913+0000 7fa3e6f79640 1 --2- 172.21.15.161:0/2086690188 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3e0152890 0x7fa3e00c8990 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:01.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.913+0000 7fa3e6f79640 1 -- 172.21.15.161:0/2086690188 >> 172.21.15.161:0/2086690188 conn(0x7fa3e00ac620 msgr2=0x7fa3e00aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:01.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.913+0000 7fa3e6f79640 1 -- 172.21.15.161:0/2086690188 shutdown_connections 2024-03-28T23:37:01.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.913+0000 7fa3e6f79640 1 -- 172.21.15.161:0/2086690188 wait complete. 2024-03-28T23:37:01.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.914+0000 7fa3e6f79640 1 Processor -- start 2024-03-28T23:37:01.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.914+0000 7fa3e6f79640 1 -- start start 2024-03-28T23:37:01.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.915+0000 7fa3e6f79640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3e00c5b20 0x7fa3e01eb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:01.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.915+0000 7fa3e6f79640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3e00c8ed0 0x7fa3e01ebdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:01.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.915+0000 7fa3e6f79640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa3e0152890 0x7fa3e00c9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:01.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.915+0000 7fa3e6f79640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e0152c70 con 0x7fa3e00c5b20 2024-03-28T23:37:01.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.915+0000 7fa3e6f79640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e0159640 con 0x7fa3e00c8ed0 2024-03-28T23:37:01.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.915+0000 7fa3e6f79640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa3e0152df0 con 0x7fa3e0152890 2024-03-28T23:37:01.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.915+0000 7fa3e4d15640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3e00c5b20 0x7fa3e01eb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:01.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.915+0000 7fa3e4d15640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3e00c5b20 0x7fa3e01eb3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37584/0 (socket says 172.21.15.161:37584) 2024-03-28T23:37:01.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.915+0000 7fa3e4d15640 1 -- 172.21.15.161:0/529311661 learned_addr learned my addr 172.21.15.161:0/529311661 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:01.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.915+0000 7fa3d7fff640 1 --2- 172.21.15.161:0/529311661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3e00c8ed0 0x7fa3e01ebdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:01.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.915+0000 7fa3e5516640 1 --2- 172.21.15.161:0/529311661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa3e0152890 0x7fa3e00c9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:01.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.916+0000 7fa3e4d15640 1 -- 172.21.15.161:0/529311661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa3e0152890 msgr2=0x7fa3e00c9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:01.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.916+0000 7fa3e4d15640 1 --2- 172.21.15.161:0/529311661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa3e0152890 0x7fa3e00c9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:01.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.916+0000 7fa3e4d15640 1 -- 172.21.15.161:0/529311661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3e00c8ed0 msgr2=0x7fa3e01ebdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:01.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.916+0000 7fa3e4d15640 1 --2- 172.21.15.161:0/529311661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3e00c8ed0 0x7fa3e01ebdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:01.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.916+0000 7fa3e4d15640 1 -- 172.21.15.161:0/529311661 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3e01ec2f0 con 0x7fa3e00c5b20 2024-03-28T23:37:01.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.916+0000 7fa3e4d15640 1 --2- 172.21.15.161:0/529311661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3e00c5b20 0x7fa3e01eb3a0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc0097a0 tx=0x7fa3dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:01.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.916+0000 7fa3d57fa640 1 -- 172.21.15.161:0/529311661 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc004020 con 0x7fa3e00c5b20 2024-03-28T23:37:01.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.916+0000 7fa3d57fa640 1 -- 172.21.15.161:0/529311661 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3dc0041e0 con 0x7fa3e00c5b20 2024-03-28T23:37:01.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.916+0000 7fa3d57fa640 1 -- 172.21.15.161:0/529311661 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc003610 con 0x7fa3e00c5b20 2024-03-28T23:37:01.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.916+0000 7fa3e6f79640 1 -- 172.21.15.161:0/529311661 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3e00c4d60 con 0x7fa3e00c5b20 2024-03-28T23:37:01.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.916+0000 7fa3e6f79640 1 -- 172.21.15.161:0/529311661 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3e00c1290 con 0x7fa3e00c5b20 2024-03-28T23:37:01.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.917+0000 7fa3e6f79640 1 -- 172.21.15.161:0/529311661 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3e004f4b0 con 0x7fa3e00c5b20 2024-03-28T23:37:01.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.918+0000 7fa3d57fa640 1 -- 172.21.15.161:0/529311661 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa3dc04c020 con 0x7fa3e00c5b20 2024-03-28T23:37:01.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.918+0000 7fa3d57fa640 1 --2- 172.21.15.161:0/529311661 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa3cc06b4d0 0x7fa3cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:01.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.918+0000 7fa3d57fa640 1 -- 172.21.15.161:0/529311661 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7fa3dc02f020 con 0x7fa3e00c5b20 2024-03-28T23:37:01.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.918+0000 7fa3d7fff640 1 --2- 172.21.15.161:0/529311661 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa3cc06b4d0 0x7fa3cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:01.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.918+0000 7fa3d7fff640 1 --2- 172.21.15.161:0/529311661 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa3cc06b4d0 0x7fa3cc06d990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa3e01e8760 tx=0x7fa3d0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:01.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:01.926+0000 7fa3d57fa640 1 -- 172.21.15.161:0/529311661 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa3dc040070 con 0x7fa3e00c5b20 2024-03-28T23:37:02.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.146+0000 7fa3e6f79640 1 -- 172.21.15.161:0/529311661 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3e00ac400 con 0x7fa3e00c5b20 2024-03-28T23:37:02.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.147+0000 7fa3d57fa640 1 -- 172.21.15.161:0/529311661 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+6520 (secure 0 0 0) 0x7fa3dc07a7e0 con 0x7fa3e00c5b20 2024-03-28T23:37:02.147 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:37:02.148 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":154,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/557522063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":557522063},{"type":"v1","addr":"172.21.15.170:6835","nonce":557522063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":9216,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/789976100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":789976100},{"type":"v1","addr":"172.21.15.161:6837","nonce":789976100}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:35:58.406403+0000","modified":"2024-03-28T23:36:20.440473+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":159,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7283":{"gid":7283,"name":"b","rank":0,"incarnation":151,"state":"up:active","state_seq":4,"addr":"172.21.15.170:6834/2996092857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":2996092857},{"type":"v1","addr":"172.21.15.170:6834","nonce":2996092857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7283,"qdb_cluster":[7283]},"id":11},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:36:07.448976+0000","modified":"2024-03-28T23:36:16.420228+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7061":{"gid":7061,"name":"c","rank":0,"incarnation":145,"state":"up:active","state_seq":9,"addr":"172.21.15.161:6836/4026332703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":4026332703},{"type":"v1","addr":"172.21.15.161:6836","nonce":4026332703}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7061,"qdb_cluster":[7061]},"id":12}]} 2024-03-28T23:37:02.148 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 154 2024-03-28T23:37:02.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.150+0000 7fa3e6f79640 1 -- 172.21.15.161:0/529311661 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa3cc06b4d0 msgr2=0x7fa3cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:02.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.150+0000 7fa3e6f79640 1 --2- 172.21.15.161:0/529311661 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa3cc06b4d0 0x7fa3cc06d990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa3e01e8760 tx=0x7fa3d0009040 comp rx=0 tx=0).stop 2024-03-28T23:37:02.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.150+0000 7fa3e6f79640 1 -- 172.21.15.161:0/529311661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3e00c5b20 msgr2=0x7fa3e01eb3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:02.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.150+0000 7fa3e6f79640 1 --2- 172.21.15.161:0/529311661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3e00c5b20 0x7fa3e01eb3a0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc0097a0 tx=0x7fa3dc0046c0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.150+0000 7fa3e6f79640 1 -- 172.21.15.161:0/529311661 shutdown_connections 2024-03-28T23:37:02.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.150+0000 7fa3e6f79640 1 --2- 172.21.15.161:0/529311661 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa3cc06b4d0 0x7fa3cc06d990 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.150+0000 7fa3e6f79640 1 --2- 172.21.15.161:0/529311661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa3e0152890 0x7fa3e00c9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.150+0000 7fa3e6f79640 1 --2- 172.21.15.161:0/529311661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3e00c8ed0 0x7fa3e01ebdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.150+0000 7fa3e6f79640 1 --2- 172.21.15.161:0/529311661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3e00c5b20 0x7fa3e01eb3a0 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.150+0000 7fa3e6f79640 1 -- 172.21.15.161:0/529311661 >> 172.21.15.161:0/529311661 conn(0x7fa3e00ac620 msgr2=0x7fa3e014ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:02.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.150+0000 7fa3e6f79640 1 -- 172.21.15.161:0/529311661 shutdown_connections 2024-03-28T23:37:02.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.150+0000 7fa3e6f79640 1 -- 172.21.15.161:0/529311661 wait complete. 2024-03-28T23:37:02.165 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:37:02.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.267+0000 7f853fc23640 1 Processor -- start 2024-03-28T23:37:02.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.267+0000 7f853fc23640 1 -- start start 2024-03-28T23:37:02.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.268+0000 7f853fc23640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8538152890 0x7f85380c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:02.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.268+0000 7f853fc23640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f85380c5b20 0x7f85381553a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:02.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.268+0000 7f853fc23640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8538158a30 con 0x7f85380c8ed0 2024-03-28T23:37:02.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.268+0000 7f853fc23640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f85380c8480 con 0x7f85380c5b20 2024-03-28T23:37:02.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.268+0000 7f853fc23640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f8538159640 con 0x7f8538152890 2024-03-28T23:37:02.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.268+0000 7f853e1c0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f85380c5b20 0x7f85381553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:02.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.268+0000 7f853d1be640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f85380c8ed0 0x7f85380c5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48128/0 (socket says 172.21.15.161:48128) 2024-03-28T23:37:02.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.269+0000 7f853d1be640 1 -- 172.21.15.161:0/2700397300 learned_addr learned my addr 172.21.15.161:0/2700397300 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:02.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.269+0000 7f853d9bf640 1 --2- 172.21.15.161:0/2700397300 >> v2:172.21.15.170:3300/0 conn(0x7f8538152890 0x7f85380c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:02.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.269+0000 7f853c9bd640 1 -- 172.21.15.161:0/2700397300 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1674372927 0 0) 0x7f8538158a30 con 0x7f85380c8ed0 2024-03-28T23:37:02.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.269+0000 7f853c9bd640 1 -- 172.21.15.161:0/2700397300 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8524003520 con 0x7f85380c8ed0 2024-03-28T23:37:02.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.269+0000 7f853e1c0640 1 -- 172.21.15.161:0/2700397300 >> v2:172.21.15.170:3300/0 conn(0x7f8538152890 msgr2=0x7f85380c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:02.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.269+0000 7f853e1c0640 1 --2- 172.21.15.161:0/2700397300 >> v2:172.21.15.170:3300/0 conn(0x7f8538152890 0x7f85380c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.269+0000 7f853e1c0640 1 -- 172.21.15.161:0/2700397300 >> v1:172.21.15.161:6789/0 conn(0x7f85380c8ed0 legacy=0x7f85380c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:02.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.269+0000 7f853e1c0640 1 -- 172.21.15.161:0/2700397300 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8538158d10 con 0x7f85380c5b20 2024-03-28T23:37:02.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.269+0000 7f853d9bf640 1 --2- 172.21.15.161:0/2700397300 >> v2:172.21.15.170:3300/0 conn(0x7f8538152890 0x7f85380c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:37:02.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.270+0000 7f853e1c0640 1 --2- 172.21.15.161:0/2700397300 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85380c5b20 0x7f85381553a0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f853808cc80 tx=0x7f853002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=174b7ff25e4efdb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:02.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.270+0000 7f853c9bd640 1 -- 172.21.15.161:0/2700397300 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8530039070 con 0x7f85380c5b20 2024-03-28T23:37:02.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.270+0000 7f853c9bd640 1 -- 172.21.15.161:0/2700397300 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f853002cb60 con 0x7f85380c5b20 2024-03-28T23:37:02.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.270+0000 7f853c9bd640 1 -- 172.21.15.161:0/2700397300 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8530034440 con 0x7f85380c5b20 2024-03-28T23:37:02.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.270+0000 7f853fc23640 1 -- 172.21.15.161:0/2700397300 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85380c5b20 msgr2=0x7f85381553a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:02.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.270+0000 7f853fc23640 1 --2- 172.21.15.161:0/2700397300 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85380c5b20 0x7f85381553a0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f853808cc80 tx=0x7f853002bdb0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.270+0000 7f853fc23640 1 -- 172.21.15.161:0/2700397300 shutdown_connections 2024-03-28T23:37:02.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.270+0000 7f853fc23640 1 --2- 172.21.15.161:0/2700397300 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85380c5b20 0x7f85381553a0 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.270+0000 7f853fc23640 1 --2- 172.21.15.161:0/2700397300 >> v2:172.21.15.170:3300/0 conn(0x7f8538152890 0x7f85380c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.270+0000 7f853fc23640 1 -- 172.21.15.161:0/2700397300 >> 172.21.15.161:0/2700397300 conn(0x7f85380ac620 msgr2=0x7f85380aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:02.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.270+0000 7f853fc23640 1 -- 172.21.15.161:0/2700397300 shutdown_connections 2024-03-28T23:37:02.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.270+0000 7f853fc23640 1 -- 172.21.15.161:0/2700397300 wait complete. 2024-03-28T23:37:02.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.271+0000 7f853fc23640 1 Processor -- start 2024-03-28T23:37:02.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.272+0000 7f853fc23640 1 -- start start 2024-03-28T23:37:02.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.272+0000 7f853fc23640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85380c5b20 0x7f8538153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:02.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.272+0000 7f853fc23640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85380c8ed0 0x7f85381e7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:02.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.272+0000 7f853fc23640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8538152890 0x7f85381ee350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:02.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.272+0000 7f853fc23640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8538152c70 con 0x7f85380c5b20 2024-03-28T23:37:02.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.272+0000 7f853fc23640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f85381592d0 con 0x7f85380c8ed0 2024-03-28T23:37:02.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.272+0000 7f853fc23640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8538152df0 con 0x7f8538152890 2024-03-28T23:37:02.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.272+0000 7f853d9bf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85380c5b20 0x7f8538153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:02.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.272+0000 7f853d9bf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85380c5b20 0x7f8538153fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37602/0 (socket says 172.21.15.161:37602) 2024-03-28T23:37:02.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.272+0000 7f853d9bf640 1 -- 172.21.15.161:0/3456244489 learned_addr learned my addr 172.21.15.161:0/3456244489 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:02.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.272+0000 7f853d1be640 1 --2- 172.21.15.161:0/3456244489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85380c8ed0 0x7f85381e7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:02.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.272+0000 7f853e1c0640 1 --2- 172.21.15.161:0/3456244489 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8538152890 0x7f85381ee350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:02.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.273+0000 7f853d9bf640 1 -- 172.21.15.161:0/3456244489 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8538152890 msgr2=0x7f85381ee350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:02.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.273+0000 7f853d9bf640 1 --2- 172.21.15.161:0/3456244489 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8538152890 0x7f85381ee350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.273+0000 7f853d9bf640 1 -- 172.21.15.161:0/3456244489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85380c8ed0 msgr2=0x7f85381e7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:02.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.273+0000 7f853d9bf640 1 --2- 172.21.15.161:0/3456244489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85380c8ed0 0x7f85381e7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.273+0000 7f853d9bf640 1 -- 172.21.15.161:0/3456244489 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8538153090 con 0x7f85380c5b20 2024-03-28T23:37:02.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.273+0000 7f853d9bf640 1 --2- 172.21.15.161:0/3456244489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85380c5b20 0x7f8538153fc0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f853400e990 tx=0x7f853400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:02.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.273+0000 7f852e7fc640 1 -- 172.21.15.161:0/3456244489 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8534017070 con 0x7f85380c5b20 2024-03-28T23:37:02.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.273+0000 7f852e7fc640 1 -- 172.21.15.161:0/3456244489 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f853400c9e0 con 0x7f85380c5b20 2024-03-28T23:37:02.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.273+0000 7f853fc23640 1 -- 172.21.15.161:0/3456244489 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8538153370 con 0x7f85380c5b20 2024-03-28T23:37:02.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.273+0000 7f852e7fc640 1 -- 172.21.15.161:0/3456244489 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853400cdb0 con 0x7f85380c5b20 2024-03-28T23:37:02.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.273+0000 7f853fc23640 1 -- 172.21.15.161:0/3456244489 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85380c2ab0 con 0x7f85380c5b20 2024-03-28T23:37:02.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.274+0000 7f853fc23640 1 -- 172.21.15.161:0/3456244489 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85380ac430 con 0x7f85380c5b20 2024-03-28T23:37:02.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.274+0000 7f852e7fc640 1 -- 172.21.15.161:0/3456244489 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f853401a070 con 0x7f85380c5b20 2024-03-28T23:37:02.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.275+0000 7f852e7fc640 1 --2- 172.21.15.161:0/3456244489 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f851c06b4d0 0x7f851c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:02.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.275+0000 7f852e7fc640 1 -- 172.21.15.161:0/3456244489 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7f853408a980 con 0x7f85380c5b20 2024-03-28T23:37:02.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.275+0000 7f853d1be640 1 --2- 172.21.15.161:0/3456244489 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f851c06b4d0 0x7f851c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:02.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.276+0000 7f853d1be640 1 --2- 172.21.15.161:0/3456244489 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f851c06b4d0 0x7f851c06d990 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f85381ebc60 tx=0x7f8528009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:02.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.279+0000 7f852e7fc640 1 -- 172.21.15.161:0/3456244489 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8534058be0 con 0x7f85380c5b20 2024-03-28T23:37:02.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.453+0000 7f853fc23640 1 -- 172.21.15.161:0/3456244489 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f853804f4b0 con 0x7f85380c5b20 2024-03-28T23:37:02.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.454+0000 7f852e7fc640 1 -- 172.21.15.161:0/3456244489 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v161) ==== 74+0+21282 (secure 0 0 0) 0x7f853405cc80 con 0x7f85380c5b20 2024-03-28T23:37:02.455 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:37:02.455 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":161,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:36:36.466146+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-28T23:35:56.687416+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-28T23:35:57.379120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:36:05.484451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:36:06.423073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.c","mappings":[{"from":1,"to":0}]},{"pgid":"26.13","mappings":[{"from":1,"to":6}]},{"pgid":"26.28","mappings":[{"from":1,"to":0}]},{"pgid":"26.3b","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6837/3236742821":"2024-03-29T23:36:17.431214+0000","172.21.15.161:6835/3236742821":"2024-03-29T23:36:17.431214+0000","172.21.15.170:6834/1393137291":"2024-03-29T23:36:14.412210+0000","172.21.15.170:6832/1393137291":"2024-03-29T23:36:14.412210+0000","172.21.15.161:6837/2391019771":"2024-03-29T23:36:13.968386+0000","172.21.15.161:6835/2391019771":"2024-03-29T23:36:13.968386+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:37:02.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.456+0000 7f853fc23640 1 -- 172.21.15.161:0/3456244489 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f851c06b4d0 msgr2=0x7f851c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:02.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.456+0000 7f853fc23640 1 --2- 172.21.15.161:0/3456244489 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f851c06b4d0 0x7f851c06d990 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f85381ebc60 tx=0x7f8528009040 comp rx=0 tx=0).stop 2024-03-28T23:37:02.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.456+0000 7f853fc23640 1 -- 172.21.15.161:0/3456244489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85380c5b20 msgr2=0x7f8538153fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:02.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.456+0000 7f853fc23640 1 --2- 172.21.15.161:0/3456244489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85380c5b20 0x7f8538153fc0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f853400e990 tx=0x7f853400ee50 comp rx=0 tx=0).stop 2024-03-28T23:37:02.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.456+0000 7f853fc23640 1 -- 172.21.15.161:0/3456244489 shutdown_connections 2024-03-28T23:37:02.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.456+0000 7f853fc23640 1 --2- 172.21.15.161:0/3456244489 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f851c06b4d0 0x7f851c06d990 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.456+0000 7f853fc23640 1 --2- 172.21.15.161:0/3456244489 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8538152890 0x7f85381ee350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.456+0000 7f853fc23640 1 --2- 172.21.15.161:0/3456244489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85380c8ed0 0x7f85381e7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.456+0000 7f853fc23640 1 --2- 172.21.15.161:0/3456244489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85380c5b20 0x7f8538153fc0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.456+0000 7f853fc23640 1 -- 172.21.15.161:0/3456244489 >> 172.21.15.161:0/3456244489 conn(0x7f85380ac620 msgr2=0x7f853806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:02.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.456+0000 7f853fc23640 1 -- 172.21.15.161:0/3456244489 shutdown_connections 2024-03-28T23:37:02.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.456+0000 7f853fc23640 1 -- 172.21.15.161:0/3456244489 wait complete. 2024-03-28T23:37:02.468 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-28T23:37:02.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.574+0000 7f85223c9640 1 Processor -- start 2024-03-28T23:37:02.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.574+0000 7f85223c9640 1 -- start start 2024-03-28T23:37:02.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.575+0000 7f85223c9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f851c152890 0x7f851c0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:02.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.575+0000 7f85223c9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f851c0c8ed0 0x7f851c0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:02.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.575+0000 7f85223c9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f851c158a30 con 0x7f851c0c5950 2024-03-28T23:37:02.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.575+0000 7f85223c9640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f851c0c8480 con 0x7f851c152890 2024-03-28T23:37:02.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.575+0000 7f85223c9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f851c159640 con 0x7f851c0c8ed0 2024-03-28T23:37:02.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.575+0000 7f851bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f851c152890 0x7f851c0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:02.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.575+0000 7f851bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f851c152890 0x7f851c0c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37618/0 (socket says 172.21.15.161:37618) 2024-03-28T23:37:02.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.575+0000 7f851bfff640 1 -- 172.21.15.161:0/1958108677 learned_addr learned my addr 172.21.15.161:0/1958108677 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:02.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.575+0000 7f851b7fe640 1 --2- 172.21.15.161:0/1958108677 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f851c0c8ed0 0x7f851c0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:02.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.576+0000 7f851affd640 1 -- 172.21.15.161:0/1958108677 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1898021368 0 0) 0x7f851c158a30 con 0x7f851c0c5950 2024-03-28T23:37:02.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.576+0000 7f851affd640 1 -- 172.21.15.161:0/1958108677 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8504003520 con 0x7f851c0c5950 2024-03-28T23:37:02.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.576+0000 7f851bfff640 1 -- 172.21.15.161:0/1958108677 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f851c0c8ed0 msgr2=0x7f851c0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:02.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.576+0000 7f851bfff640 1 --2- 172.21.15.161:0/1958108677 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f851c0c8ed0 0x7f851c0c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.576+0000 7f851bfff640 1 -- 172.21.15.161:0/1958108677 >> v1:172.21.15.170:6789/0 conn(0x7f851c0c5950 legacy=0x7f851c1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:02.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.576+0000 7f851bfff640 1 -- 172.21.15.161:0/1958108677 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f851c158d10 con 0x7f851c152890 2024-03-28T23:37:02.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.576+0000 7f851b7fe640 1 --2- 172.21.15.161:0/1958108677 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f851c0c8ed0 0x7f851c0c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:37:02.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.576+0000 7f851bfff640 1 --2- 172.21.15.161:0/1958108677 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f851c152890 0x7f851c0c8990 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f850c009870 tx=0x7f850c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=800bb60a818c4d6c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:02.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.576+0000 7f851affd640 1 -- 172.21.15.161:0/1958108677 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f850c0317c0 con 0x7f851c152890 2024-03-28T23:37:02.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.577+0000 7f851affd640 1 -- 172.21.15.161:0/1958108677 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f850c037070 con 0x7f851c152890 2024-03-28T23:37:02.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.577+0000 7f851affd640 1 -- 172.21.15.161:0/1958108677 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f850c031b80 con 0x7f851c152890 2024-03-28T23:37:02.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.577+0000 7f85223c9640 1 -- 172.21.15.161:0/1958108677 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f851c152890 msgr2=0x7f851c0c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:02.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.577+0000 7f85223c9640 1 --2- 172.21.15.161:0/1958108677 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f851c152890 0x7f851c0c8990 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f850c009870 tx=0x7f850c02b860 comp rx=0 tx=0).stop 2024-03-28T23:37:02.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.577+0000 7f85223c9640 1 -- 172.21.15.161:0/1958108677 shutdown_connections 2024-03-28T23:37:02.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.577+0000 7f85223c9640 1 --2- 172.21.15.161:0/1958108677 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f851c0c8ed0 0x7f851c0c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.577+0000 7f85223c9640 1 --2- 172.21.15.161:0/1958108677 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f851c152890 0x7f851c0c8990 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.577+0000 7f85223c9640 1 -- 172.21.15.161:0/1958108677 >> 172.21.15.161:0/1958108677 conn(0x7f851c0ac620 msgr2=0x7f851c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:02.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.577+0000 7f85223c9640 1 -- 172.21.15.161:0/1958108677 shutdown_connections 2024-03-28T23:37:02.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.577+0000 7f85223c9640 1 -- 172.21.15.161:0/1958108677 wait complete. 2024-03-28T23:37:02.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.578+0000 7f85223c9640 1 Processor -- start 2024-03-28T23:37:02.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.578+0000 7f85223c9640 1 -- start start 2024-03-28T23:37:02.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.579+0000 7f85223c9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f851c0c5950 0x7f851c1ea8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:02.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.579+0000 7f85223c9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f851c0c8ed0 0x7f851c0ad950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:02.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.579+0000 7f85223c9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f851c152890 0x7f851c0afc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:02.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.579+0000 7f85223c9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f851c152c70 con 0x7f851c152890 2024-03-28T23:37:02.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.579+0000 7f85223c9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f851c1592d0 con 0x7f851c0c5950 2024-03-28T23:37:02.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.579+0000 7f85223c9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f851c152df0 con 0x7f851c0c8ed0 2024-03-28T23:37:02.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.579+0000 7f8520966640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f851c152890 0x7f851c0afc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:02.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.579+0000 7f8520966640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f851c152890 0x7f851c0afc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37632/0 (socket says 172.21.15.161:37632) 2024-03-28T23:37:02.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.579+0000 7f8520966640 1 -- 172.21.15.161:0/1048483113 learned_addr learned my addr 172.21.15.161:0/1048483113 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:02.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.579+0000 7f851b7fe640 1 --2- 172.21.15.161:0/1048483113 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f851c0c8ed0 0x7f851c0ad950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:02.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.579+0000 7f851bfff640 1 --2- 172.21.15.161:0/1048483113 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f851c0c5950 0x7f851c1ea8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:02.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.579+0000 7f8520966640 1 -- 172.21.15.161:0/1048483113 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f851c0c8ed0 msgr2=0x7f851c0ad950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:02.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.579+0000 7f8520966640 1 --2- 172.21.15.161:0/1048483113 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f851c0c8ed0 0x7f851c0ad950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.579+0000 7f8520966640 1 -- 172.21.15.161:0/1048483113 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f851c0c5950 msgr2=0x7f851c1ea8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:02.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.579+0000 7f8520966640 1 --2- 172.21.15.161:0/1048483113 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f851c0c5950 0x7f851c1ea8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.579+0000 7f8520966640 1 -- 172.21.15.161:0/1048483113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f851c0ac000 con 0x7f851c152890 2024-03-28T23:37:02.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.580+0000 7f8520966640 1 --2- 172.21.15.161:0/1048483113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f851c152890 0x7f851c0afc10 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f851400c6e0 tx=0x7f851400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:02.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.580+0000 7f8518ff9640 1 -- 172.21.15.161:0/1048483113 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8514003b70 con 0x7f851c152890 2024-03-28T23:37:02.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.580+0000 7f8518ff9640 1 -- 172.21.15.161:0/1048483113 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8514003d10 con 0x7f851c152890 2024-03-28T23:37:02.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.580+0000 7f85223c9640 1 -- 172.21.15.161:0/1048483113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f851c0b0880 con 0x7f851c152890 2024-03-28T23:37:02.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.580+0000 7f8518ff9640 1 -- 172.21.15.161:0/1048483113 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85140052d0 con 0x7f851c152890 2024-03-28T23:37:02.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.580+0000 7f85223c9640 1 -- 172.21.15.161:0/1048483113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f851c0ade90 con 0x7f851c152890 2024-03-28T23:37:02.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.581+0000 7f85223c9640 1 -- 172.21.15.161:0/1048483113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f851c0af4e0 con 0x7f851c152890 2024-03-28T23:37:02.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.581+0000 7f8518ff9640 1 -- 172.21.15.161:0/1048483113 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8514005470 con 0x7f851c152890 2024-03-28T23:37:02.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.582+0000 7f8518ff9640 1 --2- 172.21.15.161:0/1048483113 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f850806b4d0 0x7f850806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:02.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.582+0000 7f8518ff9640 1 -- 172.21.15.161:0/1048483113 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 7331+0+0 (secure 0 0 0) 0x7f851408a580 con 0x7f851c152890 2024-03-28T23:37:02.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.582+0000 7f851bfff640 1 --2- 172.21.15.161:0/1048483113 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f850806b4d0 0x7f850806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:02.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.582+0000 7f851bfff640 1 --2- 172.21.15.161:0/1048483113 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f850806b4d0 0x7f850806d990 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f850c036000 tx=0x7f850c03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:02.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.586+0000 7f8518ff9640 1 -- 172.21.15.161:0/1048483113 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f85140587e0 con 0x7f851c152890 2024-03-28T23:37:02.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.854+0000 7f85223c9640 1 -- 172.21.15.161:0/1048483113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f851c04f4b0 con 0x7f851c152890 2024-03-28T23:37:02.854 INFO:tasks.ceph.mon.a.smithi161.stderr:2024-03-28T23:37:02.854+0000 7fdc7b85b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T23:37:02.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.857+0000 7f8518ff9640 1 -- 172.21.15.161:0/1048483113 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v155) ==== 161+0+0 (secure 0 0 0) 0x7f851405c880 con 0x7f851c152890 2024-03-28T23:37:02.858 INFO:teuthology.orchestra.run.smithi161.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-28T23:37:02.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.860+0000 7f85223c9640 1 -- 172.21.15.161:0/1048483113 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f850806b4d0 msgr2=0x7f850806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:02.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.860+0000 7f85223c9640 1 --2- 172.21.15.161:0/1048483113 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f850806b4d0 0x7f850806d990 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f850c036000 tx=0x7f850c03d040 comp rx=0 tx=0).stop 2024-03-28T23:37:02.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.860+0000 7f85223c9640 1 -- 172.21.15.161:0/1048483113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f851c152890 msgr2=0x7f851c0afc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:02.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.860+0000 7f85223c9640 1 --2- 172.21.15.161:0/1048483113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f851c152890 0x7f851c0afc10 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f851400c6e0 tx=0x7f851400cba0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.860+0000 7f85223c9640 1 -- 172.21.15.161:0/1048483113 shutdown_connections 2024-03-28T23:37:02.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.860+0000 7f85223c9640 1 --2- 172.21.15.161:0/1048483113 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f850806b4d0 0x7f850806d990 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.860+0000 7f85223c9640 1 --2- 172.21.15.161:0/1048483113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f851c152890 0x7f851c0afc10 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.860+0000 7f85223c9640 1 --2- 172.21.15.161:0/1048483113 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f851c0c8ed0 0x7f851c0ad950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.860+0000 7f85223c9640 1 --2- 172.21.15.161:0/1048483113 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f851c0c5950 0x7f851c1ea8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.860+0000 7f85223c9640 1 -- 172.21.15.161:0/1048483113 >> 172.21.15.161:0/1048483113 conn(0x7f851c0ac620 msgr2=0x7f851c0c9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:02.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.860+0000 7f85223c9640 1 -- 172.21.15.161:0/1048483113 shutdown_connections 2024-03-28T23:37:02.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.860+0000 7f85223c9640 1 -- 172.21.15.161:0/1048483113 wait complete. 2024-03-28T23:37:02.863 INFO:tasks.ceph.mds.b.smithi170.stderr: -1368> 2024-03-28T23:36:25.425+0000 7f5f303af640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:37:02.873 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-28T23:37:02.889 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2024-03-28T23:37:02.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.979+0000 7fe291993640 1 Processor -- start 2024-03-28T23:37:02.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.979+0000 7fe291993640 1 -- start start 2024-03-28T23:37:02.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.980+0000 7fe291993640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe28c0c0330 0x7fe28c0c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:02.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.980+0000 7fe291993640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe28c1589b0 0x7fe28c158e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:02.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.980+0000 7fe291993640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe28c0c6f60 con 0x7fe28c152bc0 2024-03-28T23:37:02.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.980+0000 7fe291993640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fe28c0c6120 con 0x7fe28c1589b0 2024-03-28T23:37:02.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.980+0000 7fe291993640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe28c0c6400 con 0x7fe28c0c0330 2024-03-28T23:37:02.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.980+0000 7fe28affd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe28c152bc0 0x7fe28c0c8a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48140/0 (socket says 172.21.15.161:48140) 2024-03-28T23:37:02.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.980+0000 7fe28affd640 1 -- 172.21.15.161:0/1022709349 learned_addr learned my addr 172.21.15.161:0/1022709349 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:02.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.980+0000 7fe28a7fc640 1 --2- 172.21.15.161:0/1022709349 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe28c0c0330 0x7fe28c0c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:02.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.980+0000 7fe28b7fe640 1 --2- 172.21.15.161:0/1022709349 >> v2:172.21.15.170:3300/0 conn(0x7fe28c1589b0 0x7fe28c158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:02.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.981+0000 7fe289ffb640 1 -- 172.21.15.161:0/1022709349 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 572857835 0 0) 0x7fe28c0c6f60 con 0x7fe28c152bc0 2024-03-28T23:37:02.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.981+0000 7fe289ffb640 1 -- 172.21.15.161:0/1022709349 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe278003520 con 0x7fe28c152bc0 2024-03-28T23:37:02.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.981+0000 7fe28a7fc640 1 -- 172.21.15.161:0/1022709349 >> v2:172.21.15.170:3300/0 conn(0x7fe28c1589b0 msgr2=0x7fe28c158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:02.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.981+0000 7fe28a7fc640 1 --2- 172.21.15.161:0/1022709349 >> v2:172.21.15.170:3300/0 conn(0x7fe28c1589b0 0x7fe28c158e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.981+0000 7fe28a7fc640 1 -- 172.21.15.161:0/1022709349 >> v1:172.21.15.161:6789/0 conn(0x7fe28c152bc0 legacy=0x7fe28c0c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:02.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.981+0000 7fe28a7fc640 1 -- 172.21.15.161:0/1022709349 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe28c0c96d0 con 0x7fe28c0c0330 2024-03-28T23:37:02.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.982+0000 7fe28a7fc640 1 --2- 172.21.15.161:0/1022709349 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe28c0c0330 0x7fe28c0c9190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe27c0097a0 tx=0x7fe27c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70f975c94f63e4e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:02.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.982+0000 7fe289ffb640 1 -- 172.21.15.161:0/1022709349 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c035020 con 0x7fe28c0c0330 2024-03-28T23:37:02.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.982+0000 7fe289ffb640 1 -- 172.21.15.161:0/1022709349 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe27c031830 con 0x7fe28c0c0330 2024-03-28T23:37:02.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.982+0000 7fe289ffb640 1 -- 172.21.15.161:0/1022709349 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c031c00 con 0x7fe28c0c0330 2024-03-28T23:37:02.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.982+0000 7fe291993640 1 -- 172.21.15.161:0/1022709349 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe28c0c0330 msgr2=0x7fe28c0c9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:02.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.982+0000 7fe291993640 1 --2- 172.21.15.161:0/1022709349 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe28c0c0330 0x7fe28c0c9190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe27c0097a0 tx=0x7fe27c02b8d0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.982+0000 7fe291993640 1 -- 172.21.15.161:0/1022709349 shutdown_connections 2024-03-28T23:37:02.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.982+0000 7fe291993640 1 --2- 172.21.15.161:0/1022709349 >> v2:172.21.15.170:3300/0 conn(0x7fe28c1589b0 0x7fe28c158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.982+0000 7fe291993640 1 --2- 172.21.15.161:0/1022709349 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe28c0c0330 0x7fe28c0c9190 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.982+0000 7fe291993640 1 -- 172.21.15.161:0/1022709349 >> 172.21.15.161:0/1022709349 conn(0x7fe28c0caa80 msgr2=0x7fe28c0cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:02.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.982+0000 7fe291993640 1 -- 172.21.15.161:0/1022709349 shutdown_connections 2024-03-28T23:37:02.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.982+0000 7fe291993640 1 -- 172.21.15.161:0/1022709349 wait complete. 2024-03-28T23:37:02.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.983+0000 7fe291993640 1 Processor -- start 2024-03-28T23:37:02.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.984+0000 7fe291993640 1 -- start start 2024-03-28T23:37:02.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.984+0000 7fe291993640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe28c0c0330 0x7fe28c1eaf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:02.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.984+0000 7fe291993640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe28c152bc0 0x7fe28c1516a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:02.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.984+0000 7fe291993640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe28c1589b0 0x7fe28c06ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:02.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.984+0000 7fe291993640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c152fa0 con 0x7fe28c1589b0 2024-03-28T23:37:02.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.984+0000 7fe291993640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c0c6bf0 con 0x7fe28c0c0330 2024-03-28T23:37:02.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.984+0000 7fe291993640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe28c06c7a0 con 0x7fe28c152bc0 2024-03-28T23:37:02.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.984+0000 7fe28b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe28c1589b0 0x7fe28c06ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:02.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.984+0000 7fe28b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe28c1589b0 0x7fe28c06ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37644/0 (socket says 172.21.15.161:37644) 2024-03-28T23:37:02.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.984+0000 7fe28b7fe640 1 -- 172.21.15.161:0/3450929917 learned_addr learned my addr 172.21.15.161:0/3450929917 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:02.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.985+0000 7fe28a7fc640 1 --2- 172.21.15.161:0/3450929917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe28c152bc0 0x7fe28c1516a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:02.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.985+0000 7fe28affd640 1 --2- 172.21.15.161:0/3450929917 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe28c0c0330 0x7fe28c1eaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:02.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.985+0000 7fe28b7fe640 1 -- 172.21.15.161:0/3450929917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe28c152bc0 msgr2=0x7fe28c1516a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:02.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.985+0000 7fe28b7fe640 1 --2- 172.21.15.161:0/3450929917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe28c152bc0 0x7fe28c1516a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.985+0000 7fe28b7fe640 1 -- 172.21.15.161:0/3450929917 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe28c0c0330 msgr2=0x7fe28c1eaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:02.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.985+0000 7fe28b7fe640 1 --2- 172.21.15.161:0/3450929917 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe28c0c0330 0x7fe28c1eaf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:02.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.985+0000 7fe28b7fe640 1 -- 172.21.15.161:0/3450929917 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe28c06ca60 con 0x7fe28c1589b0 2024-03-28T23:37:02.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.985+0000 7fe28affd640 1 --2- 172.21.15.161:0/3450929917 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe28c0c0330 0x7fe28c1eaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:37:02.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.985+0000 7fe28b7fe640 1 --2- 172.21.15.161:0/3450929917 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe28c1589b0 0x7fe28c06ba10 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fe28400efc0 tx=0x7fe28400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:02.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.985+0000 7fe283fff640 1 -- 172.21.15.161:0/3450929917 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe284018020 con 0x7fe28c1589b0 2024-03-28T23:37:02.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.985+0000 7fe283fff640 1 -- 172.21.15.161:0/3450929917 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe284014480 con 0x7fe28c1589b0 2024-03-28T23:37:02.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.985+0000 7fe283fff640 1 -- 172.21.15.161:0/3450929917 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe284014870 con 0x7fe28c1589b0 2024-03-28T23:37:02.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.985+0000 7fe291993640 1 -- 172.21.15.161:0/3450929917 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe28c06cd40 con 0x7fe28c1589b0 2024-03-28T23:37:02.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.985+0000 7fe291993640 1 -- 172.21.15.161:0/3450929917 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe28c151be0 con 0x7fe28c1589b0 2024-03-28T23:37:02.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.986+0000 7fe291993640 1 -- 172.21.15.161:0/3450929917 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe28c04f4b0 con 0x7fe28c1589b0 2024-03-28T23:37:02.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.987+0000 7fe283fff640 1 -- 172.21.15.161:0/3450929917 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe28401b020 con 0x7fe28c1589b0 2024-03-28T23:37:02.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.987+0000 7fe283fff640 1 --2- 172.21.15.161:0/3450929917 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe26c06b4d0 0x7fe26c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:02.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.987+0000 7fe283fff640 1 -- 172.21.15.161:0/3450929917 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(162..162 src has 1..162) ==== 7417+0+0 (secure 0 0 0) 0x7fe28408b310 con 0x7fe28c1589b0 2024-03-28T23:37:02.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.987+0000 7fe28affd640 1 --2- 172.21.15.161:0/3450929917 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe26c06b4d0 0x7fe26c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:02.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.987+0000 7fe28affd640 1 --2- 172.21.15.161:0/3450929917 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe26c06b4d0 0x7fe26c06d990 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fe274005880 tx=0x7fe274009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:02.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:02.991+0000 7fe283fff640 1 -- 172.21.15.161:0/3450929917 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe284022030 con 0x7fe28c1589b0 2024-03-28T23:37:03.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.220+0000 7fe291993640 1 -- 172.21.15.161:0/3450929917 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fe28c06cf10 con 0x7fe28c1589b0 2024-03-28T23:37:03.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.866+0000 7fe283fff640 1 -- 172.21.15.161:0/3450929917 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v156) ==== 104+0+0 (secure 0 0 0) 0x7fe284010040 con 0x7fe28c1589b0 2024-03-28T23:37:03.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.868+0000 7fe291993640 1 -- 172.21.15.161:0/3450929917 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe26c06b4d0 msgr2=0x7fe26c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:03.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.868+0000 7fe291993640 1 --2- 172.21.15.161:0/3450929917 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe26c06b4d0 0x7fe26c06d990 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fe274005880 tx=0x7fe274009040 comp rx=0 tx=0).stop 2024-03-28T23:37:03.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.868+0000 7fe291993640 1 -- 172.21.15.161:0/3450929917 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe28c1589b0 msgr2=0x7fe28c06ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:03.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.868+0000 7fe291993640 1 --2- 172.21.15.161:0/3450929917 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe28c1589b0 0x7fe28c06ba10 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fe28400efc0 tx=0x7fe28400c490 comp rx=0 tx=0).stop 2024-03-28T23:37:03.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.868+0000 7fe291993640 1 -- 172.21.15.161:0/3450929917 shutdown_connections 2024-03-28T23:37:03.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.868+0000 7fe291993640 1 --2- 172.21.15.161:0/3450929917 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe26c06b4d0 0x7fe26c06d990 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:03.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.868+0000 7fe291993640 1 --2- 172.21.15.161:0/3450929917 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe28c1589b0 0x7fe28c06ba10 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:03.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.868+0000 7fe291993640 1 --2- 172.21.15.161:0/3450929917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe28c152bc0 0x7fe28c1516a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:03.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.868+0000 7fe291993640 1 --2- 172.21.15.161:0/3450929917 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe28c0c0330 0x7fe28c1eaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:03.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.868+0000 7fe291993640 1 -- 172.21.15.161:0/3450929917 >> 172.21.15.161:0/3450929917 conn(0x7fe28c0caa80 msgr2=0x7fe28c14dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:03.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.868+0000 7fe291993640 1 -- 172.21.15.161:0/3450929917 shutdown_connections 2024-03-28T23:37:03.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.868+0000 7fe291993640 1 -- 172.21.15.161:0/3450929917 wait complete. 2024-03-28T23:37:03.874 INFO:tasks.ceph.mds.c.smithi161.stderr: -1357> 2024-03-28T23:36:19.337+0000 7f7ea652d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:37:03.885 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-28T23:37:03.903 INFO:tasks.ceph.mds.c.smithi161.stdout:starting mds.c at 2024-03-28T23:37:03.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.992+0000 7f21d6a55640 1 Processor -- start 2024-03-28T23:37:03.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.992+0000 7f21d6a55640 1 -- start start 2024-03-28T23:37:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.993+0000 7f21d6a55640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21d0152170 0x7f21d006cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.993+0000 7f21d6a55640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f21d00c5350 0x7f21d00c57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.993+0000 7f21d6a55640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21d006d8e0 con 0x7f21d00a5cf0 2024-03-28T23:37:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.993+0000 7f21d6a55640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f21d00c4900 con 0x7f21d00c5350 2024-03-28T23:37:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.993+0000 7f21d6a55640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f21d006dbc0 con 0x7f21d0152170 2024-03-28T23:37:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.993+0000 7f21d4ff2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f21d00c5350 0x7f21d00c57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.993+0000 7f21d4ff2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f21d00c5350 0x7f21d00c57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37672/0 (socket says 172.21.15.161:37672) 2024-03-28T23:37:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.993+0000 7f21d4ff2640 1 -- 172.21.15.161:0/635664362 learned_addr learned my addr 172.21.15.161:0/635664362 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.993+0000 7f21cffff640 1 --2- 172.21.15.161:0/635664362 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21d0152170 0x7f21d006cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.993+0000 7f21d4ff2640 1 -- 172.21.15.161:0/635664362 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21d0152170 msgr2=0x7f21d006cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:03.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.993+0000 7f21d4ff2640 1 --2- 172.21.15.161:0/635664362 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21d0152170 0x7f21d006cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:03.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.993+0000 7f21d4ff2640 1 -- 172.21.15.161:0/635664362 >> v1:172.21.15.170:6789/0 conn(0x7f21d00a5cf0 legacy=0x7f21d00c1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:03.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.993+0000 7f21d4ff2640 1 -- 172.21.15.161:0/635664362 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21d00c4b50 con 0x7f21d00c5350 2024-03-28T23:37:03.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.994+0000 7f21d4ff2640 1 --2- 172.21.15.161:0/635664362 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21d00c5350 0x7f21d00c57e0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f21d00c4a80 tx=0x7f21c802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e84f4060b4ddbdd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:03.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.994+0000 7f21ceffd640 1 -- 172.21.15.161:0/635664362 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c802ca50 con 0x7f21d00c5350 2024-03-28T23:37:03.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.994+0000 7f21ceffd640 1 -- 172.21.15.161:0/635664362 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21c802cbf0 con 0x7f21d00c5350 2024-03-28T23:37:03.996 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.994+0000 7f21ceffd640 1 -- 172.21.15.161:0/635664362 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c80343f0 con 0x7f21d00c5350 2024-03-28T23:37:03.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.995+0000 7f21d6a55640 1 -- 172.21.15.161:0/635664362 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21d00c5350 msgr2=0x7f21d00c57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:03.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.995+0000 7f21d6a55640 1 --2- 172.21.15.161:0/635664362 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21d00c5350 0x7f21d00c57e0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f21d00c4a80 tx=0x7f21c802beb0 comp rx=0 tx=0).stop 2024-03-28T23:37:04.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.995+0000 7f21d6a55640 1 -- 172.21.15.161:0/635664362 shutdown_connections 2024-03-28T23:37:04.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.995+0000 7f21d6a55640 1 --2- 172.21.15.161:0/635664362 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21d00c5350 0x7f21d00c57e0 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:04.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.995+0000 7f21d6a55640 1 --2- 172.21.15.161:0/635664362 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21d0152170 0x7f21d006cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:04.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.995+0000 7f21d6a55640 1 -- 172.21.15.161:0/635664362 >> 172.21.15.161:0/635664362 conn(0x7f21d00c6e20 msgr2=0x7f21d00c7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:04.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.995+0000 7f21d6a55640 1 -- 172.21.15.161:0/635664362 shutdown_connections 2024-03-28T23:37:04.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.995+0000 7f21d6a55640 1 -- 172.21.15.161:0/635664362 wait complete. 2024-03-28T23:37:04.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.996+0000 7f21d6a55640 1 Processor -- start 2024-03-28T23:37:04.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.996+0000 7f21d6a55640 1 -- start start 2024-03-28T23:37:04.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.997+0000 7f21d6a55640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21d00a5cf0 0x7f21d01ec540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:04.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.997+0000 7f21d6a55640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21d00c5350 0x7f21d0071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:04.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.997+0000 7f21d6a55640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21d0152170 0x7f21d0071570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:04.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.997+0000 7f21d6a55640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f21d00c22c0 con 0x7f21d0152170 2024-03-28T23:37:04.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.997+0000 7f21d6a55640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f21d00c2140 con 0x7f21d00c5350 2024-03-28T23:37:04.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.997+0000 7f21d6a55640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f21d006d4b0 con 0x7f21d00a5cf0 2024-03-28T23:37:04.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.997+0000 7f21d4ff2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21d0152170 0x7f21d0071570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:04.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.997+0000 7f21d4ff2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21d0152170 0x7f21d0071570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37684/0 (socket says 172.21.15.161:37684) 2024-03-28T23:37:04.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.997+0000 7f21d4ff2640 1 -- 172.21.15.161:0/1027432911 learned_addr learned my addr 172.21.15.161:0/1027432911 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:04.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.997+0000 7f21cf7fe640 1 --2- 172.21.15.161:0/1027432911 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21d00c5350 0x7f21d0071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:04.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.997+0000 7f21cffff640 1 --2- 172.21.15.161:0/1027432911 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21d00a5cf0 0x7f21d01ec540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:04.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.997+0000 7f21d4ff2640 1 -- 172.21.15.161:0/1027432911 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21d00a5cf0 msgr2=0x7f21d01ec540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:04.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.997+0000 7f21d4ff2640 1 --2- 172.21.15.161:0/1027432911 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21d00a5cf0 0x7f21d01ec540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:04.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.997+0000 7f21d4ff2640 1 -- 172.21.15.161:0/1027432911 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21d00c5350 msgr2=0x7f21d0071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:04.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.997+0000 7f21d4ff2640 1 --2- 172.21.15.161:0/1027432911 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21d00c5350 0x7f21d0071030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:04.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.997+0000 7f21d4ff2640 1 -- 172.21.15.161:0/1027432911 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21d01ece80 con 0x7f21d0152170 2024-03-28T23:37:04.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.997+0000 7f21d4ff2640 1 --2- 172.21.15.161:0/1027432911 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21d0152170 0x7f21d0071570 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f21c8009f40 tx=0x7f21c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:04.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.998+0000 7f21ccff9640 1 -- 172.21.15.161:0/1027432911 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c8043020 con 0x7f21d0152170 2024-03-28T23:37:04.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.998+0000 7f21ccff9640 1 -- 172.21.15.161:0/1027432911 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21c803e450 con 0x7f21d0152170 2024-03-28T23:37:04.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.998+0000 7f21d6a55640 1 -- 172.21.15.161:0/1027432911 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21d01ed120 con 0x7f21d0152170 2024-03-28T23:37:04.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.998+0000 7f21ccff9640 1 -- 172.21.15.161:0/1027432911 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c8009510 con 0x7f21d0152170 2024-03-28T23:37:04.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.998+0000 7f21d6a55640 1 -- 172.21.15.161:0/1027432911 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21d00abd40 con 0x7f21d0152170 2024-03-28T23:37:04.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.999+0000 7f21d6a55640 1 -- 172.21.15.161:0/1027432911 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21d0068f90 con 0x7f21d0152170 2024-03-28T23:37:04.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.999+0000 7f21ccff9640 1 -- 172.21.15.161:0/1027432911 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f21c8046020 con 0x7f21d0152170 2024-03-28T23:37:04.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:03.999+0000 7f21ccff9640 1 --2- 172.21.15.161:0/1027432911 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f21bc06b4d0 0x7f21bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:04.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:04.000+0000 7f21cffff640 1 --2- 172.21.15.161:0/1027432911 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f21bc06b4d0 0x7f21bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:04.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:04.000+0000 7f21ccff9640 1 -- 172.21.15.161:0/1027432911 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 7503+0+0 (secure 0 0 0) 0x7f21c80acf00 con 0x7f21d0152170 2024-03-28T23:37:04.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:04.000+0000 7f21cffff640 1 --2- 172.21.15.161:0/1027432911 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f21bc06b4d0 0x7f21bc06d990 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f21d0073910 tx=0x7f21c0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:04.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:04.004+0000 7f21ccff9640 1 -- 172.21.15.161:0/1027432911 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f21c804d030 con 0x7f21d0152170 2024-03-28T23:37:04.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:04.181+0000 7f21d6a55640 1 -- 172.21.15.161:0/1027432911 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f21d00c1db0 con 0x7f21d0152170 2024-03-28T23:37:04.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:04.871+0000 7f21ccff9640 1 -- 172.21.15.161:0/1027432911 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v164) ==== 181+0+0 (secure 0 0 0) 0x7f21c807b0c0 con 0x7f21d0152170 2024-03-28T23:37:04.871 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_metadata' removed 2024-03-28T23:37:04.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:04.875+0000 7f21d6a55640 1 -- 172.21.15.161:0/1027432911 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f21bc06b4d0 msgr2=0x7f21bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:04.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:04.875+0000 7f21d6a55640 1 --2- 172.21.15.161:0/1027432911 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f21bc06b4d0 0x7f21bc06d990 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f21d0073910 tx=0x7f21c0009190 comp rx=0 tx=0).stop 2024-03-28T23:37:04.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:04.875+0000 7f21d6a55640 1 -- 172.21.15.161:0/1027432911 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21d0152170 msgr2=0x7f21d0071570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:04.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:04.875+0000 7f21d6a55640 1 --2- 172.21.15.161:0/1027432911 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21d0152170 0x7f21d0071570 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f21c8009f40 tx=0x7f21c8004a40 comp rx=0 tx=0).stop 2024-03-28T23:37:04.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:04.875+0000 7f21d6a55640 1 -- 172.21.15.161:0/1027432911 shutdown_connections 2024-03-28T23:37:04.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:04.875+0000 7f21d6a55640 1 --2- 172.21.15.161:0/1027432911 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f21bc06b4d0 0x7f21bc06d990 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:04.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:04.875+0000 7f21d6a55640 1 --2- 172.21.15.161:0/1027432911 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21d0152170 0x7f21d0071570 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:04.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:04.875+0000 7f21d6a55640 1 --2- 172.21.15.161:0/1027432911 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21d00c5350 0x7f21d0071030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:04.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:04.875+0000 7f21d6a55640 1 --2- 172.21.15.161:0/1027432911 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21d00a5cf0 0x7f21d01ec540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:04.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:04.875+0000 7f21d6a55640 1 -- 172.21.15.161:0/1027432911 >> 172.21.15.161:0/1027432911 conn(0x7f21d00c6e20 msgr2=0x7f21d006cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:04.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:04.876+0000 7f21d6a55640 1 -- 172.21.15.161:0/1027432911 shutdown_connections 2024-03-28T23:37:04.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:04.876+0000 7f21d6a55640 1 -- 172.21.15.161:0/1027432911 wait complete. 2024-03-28T23:37:04.894 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-28T23:37:05.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.052+0000 7f4f3d6ed640 1 Processor -- start 2024-03-28T23:37:05.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.052+0000 7f4f3d6ed640 1 -- start start 2024-03-28T23:37:05.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.053+0000 7f4f3d6ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f38152740 0x7f4f3806b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:05.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.053+0000 7f4f3d6ed640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4f38155d70 0x7f4f380c9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:05.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.053+0000 7f4f3d6ed640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4f380c53e0 0x7f4f380c7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:05.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.053+0000 7f4f3d6ed640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f4f380c9980 con 0x7f4f380c53e0 2024-03-28T23:37:05.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.053+0000 7f4f3d6ed640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f4f381561d0 con 0x7f4f38155d70 2024-03-28T23:37:05.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.053+0000 7f4f3d6ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4f3806bb70 con 0x7f4f38152740 2024-03-28T23:37:05.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.053+0000 7f4f377fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4f380c53e0 0x7f4f380c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:05.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.053+0000 7f4f377fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4f380c53e0 0x7f4f380c7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37686/0 (socket says 172.21.15.161:37686) 2024-03-28T23:37:05.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.053+0000 7f4f377fe640 1 -- 172.21.15.161:0/184624265 learned_addr learned my addr 172.21.15.161:0/184624265 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:05.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.054+0000 7f4f377fe640 1 -- 172.21.15.161:0/184624265 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f38152740 msgr2=0x7f4f3806b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:05.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.054+0000 7f4f377fe640 1 --2- 172.21.15.161:0/184624265 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f38152740 0x7f4f3806b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:05.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.054+0000 7f4f377fe640 1 -- 172.21.15.161:0/184624265 >> v2:172.21.15.170:3300/0 conn(0x7f4f38155d70 msgr2=0x7f4f380c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:05.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.054+0000 7f4f2e7fc640 1 --2- 172.21.15.161:0/184624265 >> v2:172.21.15.170:3300/0 conn(0x7f4f38155d70 0x7f4f380c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:05.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.054+0000 7f4f377fe640 1 --2- 172.21.15.161:0/184624265 >> v2:172.21.15.170:3300/0 conn(0x7f4f38155d70 0x7f4f380c9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:05.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.054+0000 7f4f377fe640 1 -- 172.21.15.161:0/184624265 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f3806be50 con 0x7f4f380c53e0 2024-03-28T23:37:05.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.054+0000 7f4f377fe640 1 --2- 172.21.15.161:0/184624265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f380c53e0 0x7f4f380c7470 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f4f380c7ef0 tx=0x7f4f3002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9772584bca3ed0dc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:05.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.055+0000 7f4f367fc640 1 -- 172.21.15.161:0/184624265 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3002c9e0 con 0x7f4f380c53e0 2024-03-28T23:37:05.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.055+0000 7f4f367fc640 1 -- 172.21.15.161:0/184624265 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f3002cb80 con 0x7f4f380c53e0 2024-03-28T23:37:05.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.055+0000 7f4f367fc640 1 -- 172.21.15.161:0/184624265 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f300343f0 con 0x7f4f380c53e0 2024-03-28T23:37:05.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.055+0000 7f4f3d6ed640 1 -- 172.21.15.161:0/184624265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f380c53e0 msgr2=0x7f4f380c7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:05.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.055+0000 7f4f3d6ed640 1 --2- 172.21.15.161:0/184624265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f380c53e0 0x7f4f380c7470 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f4f380c7ef0 tx=0x7f4f3002be40 comp rx=0 tx=0).stop 2024-03-28T23:37:05.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.055+0000 7f4f3d6ed640 1 -- 172.21.15.161:0/184624265 shutdown_connections 2024-03-28T23:37:05.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.055+0000 7f4f3d6ed640 1 --2- 172.21.15.161:0/184624265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f380c53e0 0x7f4f380c7470 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:05.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.055+0000 7f4f3d6ed640 1 --2- 172.21.15.161:0/184624265 >> v2:172.21.15.170:3300/0 conn(0x7f4f38155d70 0x7f4f380c9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:05.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.055+0000 7f4f3d6ed640 1 --2- 172.21.15.161:0/184624265 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f38152740 0x7f4f3806b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:05.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.055+0000 7f4f3d6ed640 1 -- 172.21.15.161:0/184624265 >> 172.21.15.161:0/184624265 conn(0x7f4f380cac40 msgr2=0x7f4f380cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:05.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.055+0000 7f4f3d6ed640 1 -- 172.21.15.161:0/184624265 shutdown_connections 2024-03-28T23:37:05.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.055+0000 7f4f3d6ed640 1 -- 172.21.15.161:0/184624265 wait complete. 2024-03-28T23:37:05.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.056+0000 7f4f3d6ed640 1 Processor -- start 2024-03-28T23:37:05.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.057+0000 7f4f3d6ed640 1 -- start start 2024-03-28T23:37:05.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.057+0000 7f4f3d6ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f380c53e0 0x7f4f38154410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:05.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.057+0000 7f4f3d6ed640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f38152740 0x7f4f380ad710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:05.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.057+0000 7f4f3d6ed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f38155d70 0x7f4f380adc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:05.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.057+0000 7f4f3d6ed640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4f380c8f20 con 0x7f4f38152740 2024-03-28T23:37:05.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.057+0000 7f4f3d6ed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f4f380c8c40 con 0x7f4f38155d70 2024-03-28T23:37:05.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.057+0000 7f4f3d6ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f4f380c5870 con 0x7f4f380c53e0 2024-03-28T23:37:05.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.057+0000 7f4f2e7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f38152740 0x7f4f380ad710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:05.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.057+0000 7f4f2e7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f38152740 0x7f4f380ad710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37696/0 (socket says 172.21.15.161:37696) 2024-03-28T23:37:05.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.057+0000 7f4f2e7fc640 1 -- 172.21.15.161:0/992823763 learned_addr learned my addr 172.21.15.161:0/992823763 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:05.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.058+0000 7f4f36ffd640 1 --2- 172.21.15.161:0/992823763 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f380c53e0 0x7f4f38154410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:05.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.058+0000 7f4f377fe640 1 --2- 172.21.15.161:0/992823763 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f38155d70 0x7f4f380adc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:05.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.058+0000 7f4f2e7fc640 1 -- 172.21.15.161:0/992823763 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f380c53e0 msgr2=0x7f4f38154410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:05.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.058+0000 7f4f2e7fc640 1 --2- 172.21.15.161:0/992823763 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f380c53e0 0x7f4f38154410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:05.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.058+0000 7f4f2e7fc640 1 -- 172.21.15.161:0/992823763 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f38155d70 msgr2=0x7f4f380adc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:05.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.058+0000 7f4f2e7fc640 1 --2- 172.21.15.161:0/992823763 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f38155d70 0x7f4f380adc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:05.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.058+0000 7f4f2e7fc640 1 -- 172.21.15.161:0/992823763 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f381ea6e0 con 0x7f4f38152740 2024-03-28T23:37:05.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.058+0000 7f4f36ffd640 1 --2- 172.21.15.161:0/992823763 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f380c53e0 0x7f4f38154410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:37:05.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.058+0000 7f4f377fe640 1 --2- 172.21.15.161:0/992823763 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f38155d70 0x7f4f380adc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:37:05.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.058+0000 7f4f2e7fc640 1 --2- 172.21.15.161:0/992823763 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f38152740 0x7f4f380ad710 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f4f2000c7f0 tx=0x7f4f2000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:05.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.058+0000 7f4f2ffff640 1 -- 172.21.15.161:0/992823763 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f20015070 con 0x7f4f38152740 2024-03-28T23:37:05.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.058+0000 7f4f2ffff640 1 -- 172.21.15.161:0/992823763 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f20007820 con 0x7f4f38152740 2024-03-28T23:37:05.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.058+0000 7f4f3d6ed640 1 -- 172.21.15.161:0/992823763 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f381ea9c0 con 0x7f4f38152740 2024-03-28T23:37:05.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.058+0000 7f4f2ffff640 1 -- 172.21.15.161:0/992823763 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f20007bd0 con 0x7f4f38152740 2024-03-28T23:37:05.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.058+0000 7f4f3d6ed640 1 -- 172.21.15.161:0/992823763 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f381eaf80 con 0x7f4f38152740 2024-03-28T23:37:05.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.059+0000 7f4f3d6ed640 1 -- 172.21.15.161:0/992823763 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f38068f90 con 0x7f4f38152740 2024-03-28T23:37:05.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.060+0000 7f4f2ffff640 1 -- 172.21.15.161:0/992823763 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4f20018070 con 0x7f4f38152740 2024-03-28T23:37:05.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.060+0000 7f4f2ffff640 1 --2- 172.21.15.161:0/992823763 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4f1806b4d0 0x7f4f1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:05.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.060+0000 7f4f2ffff640 1 -- 172.21.15.161:0/992823763 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 7065+0+0 (secure 0 0 0) 0x7f4f2008a800 con 0x7f4f38152740 2024-03-28T23:37:05.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.060+0000 7f4f36ffd640 1 --2- 172.21.15.161:0/992823763 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4f1806b4d0 0x7f4f1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:05.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.060+0000 7f4f36ffd640 1 --2- 172.21.15.161:0/992823763 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4f1806b4d0 0x7f4f1806d990 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f4f2800a870 tx=0x7f4f28008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:05.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.065+0000 7f4f2ffff640 1 -- 172.21.15.161:0/992823763 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4f200591c0 con 0x7f4f38152740 2024-03-28T23:37:05.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.236+0000 7f4f3d6ed640 1 -- 172.21.15.161:0/992823763 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4f380ca4f0 con 0x7f4f38152740 2024-03-28T23:37:05.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.880+0000 7f4f2ffff640 1 -- 172.21.15.161:0/992823763 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v165) ==== 169+0+0 (secure 0 0 0) 0x7f4f20058ac0 con 0x7f4f38152740 2024-03-28T23:37:05.880 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data' removed 2024-03-28T23:37:05.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.882+0000 7f4f3d6ed640 1 -- 172.21.15.161:0/992823763 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4f1806b4d0 msgr2=0x7f4f1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:05.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.882+0000 7f4f3d6ed640 1 --2- 172.21.15.161:0/992823763 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4f1806b4d0 0x7f4f1806d990 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f4f2800a870 tx=0x7f4f28008040 comp rx=0 tx=0).stop 2024-03-28T23:37:05.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.882+0000 7f4f3d6ed640 1 -- 172.21.15.161:0/992823763 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f38152740 msgr2=0x7f4f380ad710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:05.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.882+0000 7f4f3d6ed640 1 --2- 172.21.15.161:0/992823763 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f38152740 0x7f4f380ad710 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f4f2000c7f0 tx=0x7f4f2000ccb0 comp rx=0 tx=0).stop 2024-03-28T23:37:05.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.882+0000 7f4f3d6ed640 1 -- 172.21.15.161:0/992823763 shutdown_connections 2024-03-28T23:37:05.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.882+0000 7f4f3d6ed640 1 --2- 172.21.15.161:0/992823763 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f4f1806b4d0 0x7f4f1806d990 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:05.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.882+0000 7f4f3d6ed640 1 --2- 172.21.15.161:0/992823763 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f38152740 0x7f4f380ad710 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:05.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.882+0000 7f4f3d6ed640 1 --2- 172.21.15.161:0/992823763 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f38155d70 0x7f4f380adc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:05.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.882+0000 7f4f3d6ed640 1 --2- 172.21.15.161:0/992823763 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f380c53e0 0x7f4f38154410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:05.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.882+0000 7f4f3d6ed640 1 -- 172.21.15.161:0/992823763 >> 172.21.15.161:0/992823763 conn(0x7f4f380cac40 msgr2=0x7f4f3814cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:05.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.882+0000 7f4f3d6ed640 1 -- 172.21.15.161:0/992823763 shutdown_connections 2024-03-28T23:37:05.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.883+0000 7f4f3d6ed640 1 -- 172.21.15.161:0/992823763 wait complete. 2024-03-28T23:37:05.895 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:37:05.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.999+0000 7f839d39a640 1 Processor -- start 2024-03-28T23:37:05.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:05.999+0000 7f839d39a640 1 -- start start 2024-03-28T23:37:06.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.000+0000 7f839d39a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8398150730 0x7f83980a7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.000+0000 7f839d39a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f83980a8240 0x7f83980726a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.000+0000 7f839d39a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8398072be0 0x7f8398156750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.000+0000 7f839d39a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8398153d80 con 0x7f8398150730 2024-03-28T23:37:06.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.000+0000 7f839d39a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f8398153170 con 0x7f83980a8240 2024-03-28T23:37:06.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.000+0000 7f839d39a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8398153450 con 0x7f8398072be0 2024-03-28T23:37:06.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.000+0000 7f8396ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8398150730 0x7f83980a7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.000+0000 7f8396ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8398150730 0x7f83980a7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37708/0 (socket says 172.21.15.161:37708) 2024-03-28T23:37:06.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.000+0000 7f8396ffd640 1 -- 172.21.15.161:0/2381896540 learned_addr learned my addr 172.21.15.161:0/2381896540 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:06.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.000+0000 7f83977fe640 1 --2- 172.21.15.161:0/2381896540 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8398072be0 0x7f8398156750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.000+0000 7f8396ffd640 1 -- 172.21.15.161:0/2381896540 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8398072be0 msgr2=0x7f8398156750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:06.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.000+0000 7f838e7fc640 1 --2- 172.21.15.161:0/2381896540 >> v2:172.21.15.170:3300/0 conn(0x7f83980a8240 0x7f83980726a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.000+0000 7f8396ffd640 1 --2- 172.21.15.161:0/2381896540 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8398072be0 0x7f8398156750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.000+0000 7f8396ffd640 1 -- 172.21.15.161:0/2381896540 >> v2:172.21.15.170:3300/0 conn(0x7f83980a8240 msgr2=0x7f83980726a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:06.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.000+0000 7f8396ffd640 1 --2- 172.21.15.161:0/2381896540 >> v2:172.21.15.170:3300/0 conn(0x7f83980a8240 0x7f83980726a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.000+0000 7f8396ffd640 1 -- 172.21.15.161:0/2381896540 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8398153730 con 0x7f8398150730 2024-03-28T23:37:06.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.000+0000 7f83977fe640 1 --2- 172.21.15.161:0/2381896540 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8398072be0 0x7f8398156750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:37:06.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.000+0000 7f838e7fc640 1 --2- 172.21.15.161:0/2381896540 >> v2:172.21.15.170:3300/0 conn(0x7f83980a8240 0x7f83980726a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:37:06.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.001+0000 7f8396ffd640 1 --2- 172.21.15.161:0/2381896540 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8398150730 0x7f83980a7d00 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f83980735f0 tx=0x7f838802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbbabd1712c1212c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:06.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.001+0000 7f83967fc640 1 -- 172.21.15.161:0/2381896540 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838802e070 con 0x7f8398150730 2024-03-28T23:37:06.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.001+0000 7f83967fc640 1 -- 172.21.15.161:0/2381896540 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8388037070 con 0x7f8398150730 2024-03-28T23:37:06.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.001+0000 7f83967fc640 1 -- 172.21.15.161:0/2381896540 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8388031a00 con 0x7f8398150730 2024-03-28T23:37:06.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.002+0000 7f839d39a640 1 -- 172.21.15.161:0/2381896540 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8398150730 msgr2=0x7f83980a7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:06.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.002+0000 7f839d39a640 1 --2- 172.21.15.161:0/2381896540 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8398150730 0x7f83980a7d00 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f83980735f0 tx=0x7f838802b880 comp rx=0 tx=0).stop 2024-03-28T23:37:06.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.002+0000 7f839d39a640 1 -- 172.21.15.161:0/2381896540 shutdown_connections 2024-03-28T23:37:06.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.002+0000 7f839d39a640 1 --2- 172.21.15.161:0/2381896540 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8398072be0 0x7f8398156750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.002+0000 7f839d39a640 1 --2- 172.21.15.161:0/2381896540 >> v2:172.21.15.170:3300/0 conn(0x7f83980a8240 0x7f83980726a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.002+0000 7f839d39a640 1 --2- 172.21.15.161:0/2381896540 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8398150730 0x7f83980a7d00 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.002+0000 7f839d39a640 1 -- 172.21.15.161:0/2381896540 >> 172.21.15.161:0/2381896540 conn(0x7f83980aa740 msgr2=0x7f83980aab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:06.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.002+0000 7f839d39a640 1 -- 172.21.15.161:0/2381896540 shutdown_connections 2024-03-28T23:37:06.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.002+0000 7f839d39a640 1 -- 172.21.15.161:0/2381896540 wait complete. 2024-03-28T23:37:06.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.003+0000 7f839d39a640 1 Processor -- start 2024-03-28T23:37:06.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.003+0000 7f839d39a640 1 -- start start 2024-03-28T23:37:06.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.004+0000 7f839d39a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8398072be0 0x7f83981e86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.004+0000 7f839d39a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83980a8240 0x7f83980ae150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.004+0000 7f839d39a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8398150730 0x7f839814ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.004+0000 7f839d39a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f83980c8680 con 0x7f8398072be0 2024-03-28T23:37:06.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.004+0000 7f839d39a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f83980c8500 con 0x7f8398150730 2024-03-28T23:37:06.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.004+0000 7f839d39a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8398156c90 con 0x7f83980a8240 2024-03-28T23:37:06.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.004+0000 7f8396ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8398072be0 0x7f83981e86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.004+0000 7f8396ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8398072be0 0x7f83981e86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37714/0 (socket says 172.21.15.161:37714) 2024-03-28T23:37:06.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.004+0000 7f8396ffd640 1 -- 172.21.15.161:0/2641998258 learned_addr learned my addr 172.21.15.161:0/2641998258 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:06.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.004+0000 7f838e7fc640 1 --2- 172.21.15.161:0/2641998258 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83980a8240 0x7f83980ae150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.004+0000 7f83977fe640 1 --2- 172.21.15.161:0/2641998258 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8398150730 0x7f839814ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.004+0000 7f8396ffd640 1 -- 172.21.15.161:0/2641998258 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83980a8240 msgr2=0x7f83980ae150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:06.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.004+0000 7f8396ffd640 1 --2- 172.21.15.161:0/2641998258 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83980a8240 0x7f83980ae150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.004+0000 7f8396ffd640 1 -- 172.21.15.161:0/2641998258 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8398150730 msgr2=0x7f839814ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:06.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.004+0000 7f8396ffd640 1 --2- 172.21.15.161:0/2641998258 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8398150730 0x7f839814ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.004+0000 7f8396ffd640 1 -- 172.21.15.161:0/2641998258 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f839814ab00 con 0x7f8398072be0 2024-03-28T23:37:06.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.004+0000 7f838e7fc640 1 --2- 172.21.15.161:0/2641998258 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83980a8240 0x7f83980ae150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:37:06.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.004+0000 7f8396ffd640 1 --2- 172.21.15.161:0/2641998258 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8398072be0 0x7f83981e86d0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f8388000c00 tx=0x7f83880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:06.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.004+0000 7f838ffff640 1 -- 172.21.15.161:0/2641998258 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8388032880 con 0x7f8398072be0 2024-03-28T23:37:06.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.004+0000 7f838ffff640 1 -- 172.21.15.161:0/2641998258 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8388032a20 con 0x7f8398072be0 2024-03-28T23:37:06.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.004+0000 7f839d39a640 1 -- 172.21.15.161:0/2641998258 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f839814ade0 con 0x7f8398072be0 2024-03-28T23:37:06.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.004+0000 7f838ffff640 1 -- 172.21.15.161:0/2641998258 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8388032e10 con 0x7f8398072be0 2024-03-28T23:37:06.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.004+0000 7f839d39a640 1 -- 172.21.15.161:0/2641998258 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f839814c870 con 0x7f8398072be0 2024-03-28T23:37:06.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.005+0000 7f839d39a640 1 -- 172.21.15.161:0/2641998258 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83980ada20 con 0x7f8398072be0 2024-03-28T23:37:06.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.006+0000 7f838ffff640 1 -- 172.21.15.161:0/2641998258 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f838804c060 con 0x7f8398072be0 2024-03-28T23:37:06.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.006+0000 7f838ffff640 1 --2- 172.21.15.161:0/2641998258 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f837806b4d0 0x7f837806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.006+0000 7f838ffff640 1 -- 172.21.15.161:0/2641998258 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 6651+0+0 (secure 0 0 0) 0x7f83880ac130 con 0x7f8398072be0 2024-03-28T23:37:06.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.006+0000 7f838e7fc640 1 --2- 172.21.15.161:0/2641998258 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f837806b4d0 0x7f837806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.007+0000 7f838e7fc640 1 --2- 172.21.15.161:0/2641998258 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f837806b4d0 0x7f837806d990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f83980a8680 tx=0x7f8380008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:06.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.010+0000 7f838ffff640 1 -- 172.21.15.161:0/2641998258 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8388040070 con 0x7f8398072be0 2024-03-28T23:37:06.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.181+0000 7f839d39a640 1 -- 172.21.15.161:0/2641998258 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f839804f4b0 con 0x7f8398072be0 2024-03-28T23:37:06.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.182+0000 7f838ffff640 1 -- 172.21.15.161:0/2641998258 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v165) ==== 74+0+17836 (secure 0 0 0) 0x7f838807ac90 con 0x7f8398072be0 2024-03-28T23:37:06.184 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:37:06.184 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":165,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:37:05.877513+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:36:05.484451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:36:06.423073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.c","mappings":[{"from":1,"to":0}]},{"pgid":"26.13","mappings":[{"from":1,"to":6}]},{"pgid":"26.28","mappings":[{"from":1,"to":0}]},{"pgid":"26.3b","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6834/4026332703":"2024-03-29T23:37:03.868933+0000","172.21.15.170:6834/2996092857":"2024-03-29T23:37:02.855183+0000","172.21.15.161:6836/4026332703":"2024-03-29T23:37:03.868933+0000","172.21.15.170:6832/2996092857":"2024-03-29T23:37:02.855183+0000","172.21.15.161:6837/3236742821":"2024-03-29T23:36:17.431214+0000","172.21.15.161:6835/3236742821":"2024-03-29T23:36:17.431214+0000","172.21.15.170:6834/1393137291":"2024-03-29T23:36:14.412210+0000","172.21.15.170:6832/1393137291":"2024-03-29T23:36:14.412210+0000","172.21.15.161:6837/2391019771":"2024-03-29T23:36:13.968386+0000","172.21.15.161:6835/2391019771":"2024-03-29T23:36:13.968386+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:37:06.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.184+0000 7f839d39a640 1 -- 172.21.15.161:0/2641998258 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f837806b4d0 msgr2=0x7f837806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:06.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.184+0000 7f839d39a640 1 --2- 172.21.15.161:0/2641998258 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f837806b4d0 0x7f837806d990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f83980a8680 tx=0x7f8380008040 comp rx=0 tx=0).stop 2024-03-28T23:37:06.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.184+0000 7f839d39a640 1 -- 172.21.15.161:0/2641998258 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8398072be0 msgr2=0x7f83981e86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:06.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.184+0000 7f839d39a640 1 --2- 172.21.15.161:0/2641998258 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8398072be0 0x7f83981e86d0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f8388000c00 tx=0x7f83880046c0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.184+0000 7f839d39a640 1 -- 172.21.15.161:0/2641998258 shutdown_connections 2024-03-28T23:37:06.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.184+0000 7f839d39a640 1 --2- 172.21.15.161:0/2641998258 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f837806b4d0 0x7f837806d990 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.184+0000 7f839d39a640 1 --2- 172.21.15.161:0/2641998258 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8398150730 0x7f839814ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.184+0000 7f839d39a640 1 --2- 172.21.15.161:0/2641998258 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83980a8240 0x7f83980ae150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.184+0000 7f839d39a640 1 --2- 172.21.15.161:0/2641998258 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8398072be0 0x7f83981e86d0 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.184+0000 7f839d39a640 1 -- 172.21.15.161:0/2641998258 >> 172.21.15.161:0/2641998258 conn(0x7f83980aa740 msgr2=0x7f8398150eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:06.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.184+0000 7f839d39a640 1 -- 172.21.15.161:0/2641998258 shutdown_connections 2024-03-28T23:37:06.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.184+0000 7f839d39a640 1 -- 172.21.15.161:0/2641998258 wait complete. 2024-03-28T23:37:06.196 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T23:29:39.343393+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-03-28T23:36:05.484451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '155', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs2_data', 'create_time': '2024-03-28T23:36:06.423073+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '156', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T23:37:06.196 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T23:37:06.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.304+0000 7fc495c40640 1 Processor -- start 2024-03-28T23:37:06.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.304+0000 7fc495c40640 1 -- start start 2024-03-28T23:37:06.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.305+0000 7fc495c40640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc49014eb50 0x7fc4900c4790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.305+0000 7fc495c40640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc4900c1580 0x7fc4900c2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.305+0000 7fc495c40640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4900aca60 con 0x7fc4900c4cd0 2024-03-28T23:37:06.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.305+0000 7fc495c40640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc4900c1ad0 con 0x7fc4900c1580 2024-03-28T23:37:06.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.305+0000 7fc495c40640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fc49006d270 con 0x7fc49014eb50 2024-03-28T23:37:06.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.305+0000 7fc487fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc4900c1580 0x7fc4900c2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.305+0000 7fc487fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc4900c1580 0x7fc4900c2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37718/0 (socket says 172.21.15.161:37718) 2024-03-28T23:37:06.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.305+0000 7fc487fff640 1 -- 172.21.15.161:0/3421004517 learned_addr learned my addr 172.21.15.161:0/3421004517 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:06.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.305+0000 7fc48f7fe640 1 --2- 172.21.15.161:0/3421004517 >> v2:172.21.15.170:3300/0 conn(0x7fc49014eb50 0x7fc4900c4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.306+0000 7fc487fff640 1 -- 172.21.15.161:0/3421004517 >> v2:172.21.15.170:3300/0 conn(0x7fc49014eb50 msgr2=0x7fc4900c4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:06.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.306+0000 7fc487fff640 1 --2- 172.21.15.161:0/3421004517 >> v2:172.21.15.170:3300/0 conn(0x7fc49014eb50 0x7fc4900c4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.306+0000 7fc487fff640 1 -- 172.21.15.161:0/3421004517 >> v1:172.21.15.170:6789/0 conn(0x7fc4900c4cd0 legacy=0x7fc4900c29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:06.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.306+0000 7fc487fff640 1 -- 172.21.15.161:0/3421004517 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc490151ea0 con 0x7fc4900c1580 2024-03-28T23:37:06.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.307+0000 7fc487fff640 1 --2- 172.21.15.161:0/3421004517 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4900c1580 0x7fc4900c2470 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fc49006fc20 tx=0x7fc48002c120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3457675ddb66bc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:06.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.307+0000 7fc48effd640 1 -- 172.21.15.161:0/3421004517 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4800078d0 con 0x7fc4900c1580 2024-03-28T23:37:06.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.308+0000 7fc48effd640 1 -- 172.21.15.161:0/3421004517 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc480007a70 con 0x7fc4900c1580 2024-03-28T23:37:06.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.308+0000 7fc48effd640 1 -- 172.21.15.161:0/3421004517 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc480007e40 con 0x7fc4900c1580 2024-03-28T23:37:06.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.309+0000 7fc495c40640 1 -- 172.21.15.161:0/3421004517 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4900c1580 msgr2=0x7fc4900c2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:06.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.310+0000 7fc495c40640 1 --2- 172.21.15.161:0/3421004517 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4900c1580 0x7fc4900c2470 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fc49006fc20 tx=0x7fc48002c120 comp rx=0 tx=0).stop 2024-03-28T23:37:06.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.310+0000 7fc495c40640 1 -- 172.21.15.161:0/3421004517 shutdown_connections 2024-03-28T23:37:06.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.310+0000 7fc495c40640 1 --2- 172.21.15.161:0/3421004517 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4900c1580 0x7fc4900c2470 secure :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fc49006fc20 tx=0x7fc48002c120 comp rx=0 tx=0).stop 2024-03-28T23:37:06.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.310+0000 7fc495c40640 1 --2- 172.21.15.161:0/3421004517 >> v2:172.21.15.170:3300/0 conn(0x7fc49014eb50 0x7fc4900c4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.310+0000 7fc495c40640 1 -- 172.21.15.161:0/3421004517 >> 172.21.15.161:0/3421004517 conn(0x7fc4900c3390 msgr2=0x7fc4900c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:06.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.311+0000 7fc495c40640 1 -- 172.21.15.161:0/3421004517 shutdown_connections 2024-03-28T23:37:06.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.311+0000 7fc495c40640 1 -- 172.21.15.161:0/3421004517 wait complete. 2024-03-28T23:37:06.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.313+0000 7fc495c40640 1 Processor -- start 2024-03-28T23:37:06.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.313+0000 7fc495c40640 1 -- start start 2024-03-28T23:37:06.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.314+0000 7fc495c40640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4900c4cd0 0x7fc490155e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.314+0000 7fc495c40640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc49014eb50 0x7fc490072fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.314+0000 7fc495c40640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4901eb320 0x7fc49014dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.314+0000 7fc495c40640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc49006d550 con 0x7fc4900c4cd0 2024-03-28T23:37:06.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.314+0000 7fc495c40640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc49006ce00 con 0x7fc49014eb50 2024-03-28T23:37:06.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.314+0000 7fc495c40640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc4900c30c0 con 0x7fc4901eb320 2024-03-28T23:37:06.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.314+0000 7fc48f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4900c4cd0 0x7fc490155e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.314+0000 7fc487fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc49014eb50 0x7fc490072fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.314+0000 7fc48ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4901eb320 0x7fc49014dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.314+0000 7fc48f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4900c4cd0 0x7fc490155e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37732/0 (socket says 172.21.15.161:37732) 2024-03-28T23:37:06.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.314+0000 7fc48f7fe640 1 -- 172.21.15.161:0/1594282848 learned_addr learned my addr 172.21.15.161:0/1594282848 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:06.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.315+0000 7fc48f7fe640 1 -- 172.21.15.161:0/1594282848 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4901eb320 msgr2=0x7fc49014dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.315+0000 7fc48f7fe640 1 --2- 172.21.15.161:0/1594282848 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4901eb320 0x7fc49014dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.315+0000 7fc48f7fe640 1 -- 172.21.15.161:0/1594282848 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc49014eb50 msgr2=0x7fc490072fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.315+0000 7fc48f7fe640 1 --2- 172.21.15.161:0/1594282848 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc49014eb50 0x7fc490072fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.315+0000 7fc48f7fe640 1 -- 172.21.15.161:0/1594282848 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc490153ca0 con 0x7fc4900c4cd0 2024-03-28T23:37:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.315+0000 7fc48ffff640 1 --2- 172.21.15.161:0/1594282848 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4901eb320 0x7fc49014dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:37:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.315+0000 7fc487fff640 1 --2- 172.21.15.161:0/1594282848 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc49014eb50 0x7fc490072fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:37:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.315+0000 7fc48f7fe640 1 --2- 172.21.15.161:0/1594282848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4900c4cd0 0x7fc490155e00 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fc478010bc0 tx=0x7fc47800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.316+0000 7fc48cff9640 1 -- 172.21.15.161:0/1594282848 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47801e070 con 0x7fc4900c4cd0 2024-03-28T23:37:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.316+0000 7fc48cff9640 1 -- 172.21.15.161:0/1594282848 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc47800cdd0 con 0x7fc4900c4cd0 2024-03-28T23:37:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.316+0000 7fc495c40640 1 -- 172.21.15.161:0/1594282848 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc490156340 con 0x7fc4900c4cd0 2024-03-28T23:37:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.316+0000 7fc48cff9640 1 -- 172.21.15.161:0/1594282848 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc478011690 con 0x7fc4900c4cd0 2024-03-28T23:37:06.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.316+0000 7fc495c40640 1 -- 172.21.15.161:0/1594282848 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4901568c0 con 0x7fc4900c4cd0 2024-03-28T23:37:06.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.317+0000 7fc495c40640 1 -- 172.21.15.161:0/1594282848 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc490153e70 con 0x7fc4900c4cd0 2024-03-28T23:37:06.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.317+0000 7fc48cff9640 1 -- 172.21.15.161:0/1594282848 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc478021070 con 0x7fc4900c4cd0 2024-03-28T23:37:06.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.318+0000 7fc48cff9640 1 --2- 172.21.15.161:0/1594282848 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc47006b550 0x7fc47006da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.318+0000 7fc487fff640 1 --2- 172.21.15.161:0/1594282848 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc47006b550 0x7fc47006da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.318+0000 7fc48cff9640 1 -- 172.21.15.161:0/1594282848 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 6651+0+0 (secure 0 0 0) 0x7fc47808c4b0 con 0x7fc4900c4cd0 2024-03-28T23:37:06.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.318+0000 7fc487fff640 1 --2- 172.21.15.161:0/1594282848 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc47006b550 0x7fc47006da10 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fc480035000 tx=0x7fc480030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:06.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.322+0000 7fc48cff9640 1 -- 172.21.15.161:0/1594282848 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc47805a9a0 con 0x7fc4900c4cd0 2024-03-28T23:37:06.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.492+0000 7fc495c40640 1 -- 172.21.15.161:0/1594282848 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc49004f4b0 con 0x7fc4900c4cd0 2024-03-28T23:37:06.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.492+0000 7fc48cff9640 1 -- 172.21.15.161:0/1594282848 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v165) ==== 93+0+10 (secure 0 0 0) 0x7fc47805ea40 con 0x7fc4900c4cd0 2024-03-28T23:37:06.493 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-28T23:37:06.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.494+0000 7fc495c40640 1 -- 172.21.15.161:0/1594282848 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc47006b550 msgr2=0x7fc47006da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:06.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.494+0000 7fc495c40640 1 --2- 172.21.15.161:0/1594282848 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc47006b550 0x7fc47006da10 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fc480035000 tx=0x7fc480030000 comp rx=0 tx=0).stop 2024-03-28T23:37:06.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.494+0000 7fc495c40640 1 -- 172.21.15.161:0/1594282848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4900c4cd0 msgr2=0x7fc490155e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:06.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.494+0000 7fc495c40640 1 --2- 172.21.15.161:0/1594282848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4900c4cd0 0x7fc490155e00 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fc478010bc0 tx=0x7fc47800c520 comp rx=0 tx=0).stop 2024-03-28T23:37:06.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.494+0000 7fc495c40640 1 -- 172.21.15.161:0/1594282848 shutdown_connections 2024-03-28T23:37:06.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.494+0000 7fc495c40640 1 --2- 172.21.15.161:0/1594282848 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc47006b550 0x7fc47006da10 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.494+0000 7fc495c40640 1 --2- 172.21.15.161:0/1594282848 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4901eb320 0x7fc49014dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.494+0000 7fc495c40640 1 --2- 172.21.15.161:0/1594282848 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc49014eb50 0x7fc490072fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.494+0000 7fc495c40640 1 --2- 172.21.15.161:0/1594282848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4900c4cd0 0x7fc490155e00 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.494+0000 7fc495c40640 1 -- 172.21.15.161:0/1594282848 >> 172.21.15.161:0/1594282848 conn(0x7fc4900c3390 msgr2=0x7fc49006c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:06.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.494+0000 7fc495c40640 1 -- 172.21.15.161:0/1594282848 shutdown_connections 2024-03-28T23:37:06.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.494+0000 7fc495c40640 1 -- 172.21.15.161:0/1594282848 wait complete. 2024-03-28T23:37:06.505 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-03-28T23:37:06.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.609+0000 7f29eab7a640 1 Processor -- start 2024-03-28T23:37:06.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.609+0000 7f29eab7a640 1 -- start start 2024-03-28T23:37:06.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.609+0000 7f29eab7a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f29e4152ae0 0x7f29e4159250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.609+0000 7f29eab7a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f29e40c56c0 0x7f29e40c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.610+0000 7f29eab7a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29e40c5b70 con 0x7f29e40afc80 2024-03-28T23:37:06.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.610+0000 7f29eab7a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f29e4159790 con 0x7f29e4152ae0 2024-03-28T23:37:06.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.610+0000 7f29eab7a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f29e40ae5d0 con 0x7f29e40c56c0 2024-03-28T23:37:06.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.610+0000 7f29e8916640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f29e4152ae0 0x7f29e4159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.610+0000 7f29e8916640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f29e4152ae0 0x7f29e4159250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37742/0 (socket says 172.21.15.161:37742) 2024-03-28T23:37:06.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.610+0000 7f29e8916640 1 -- 172.21.15.161:0/704426325 learned_addr learned my addr 172.21.15.161:0/704426325 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:06.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.610+0000 7f29e9117640 1 --2- 172.21.15.161:0/704426325 >> v2:172.21.15.170:3300/0 conn(0x7f29e40c56c0 0x7f29e40c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.610+0000 7f29e8916640 1 -- 172.21.15.161:0/704426325 >> v2:172.21.15.170:3300/0 conn(0x7f29e40c56c0 msgr2=0x7f29e40c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:06.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.610+0000 7f29e8916640 1 --2- 172.21.15.161:0/704426325 >> v2:172.21.15.170:3300/0 conn(0x7f29e40c56c0 0x7f29e40c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.610+0000 7f29e8916640 1 -- 172.21.15.161:0/704426325 >> v1:172.21.15.170:6789/0 conn(0x7f29e40afc80 legacy=0x7f29e40c89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:06.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.610+0000 7f29e8916640 1 -- 172.21.15.161:0/704426325 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29e40c5e10 con 0x7f29e4152ae0 2024-03-28T23:37:06.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.610+0000 7f29e9117640 1 --2- 172.21.15.161:0/704426325 >> v2:172.21.15.170:3300/0 conn(0x7f29e40c56c0 0x7f29e40c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:37:06.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.611+0000 7f29e8916640 1 --2- 172.21.15.161:0/704426325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29e4152ae0 0x7f29e4159250 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f29e40ada60 tx=0x7f29dc02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85e9813e32a46ad3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:06.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.611+0000 7f29e37fe640 1 -- 172.21.15.161:0/704426325 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29dc02c9f0 con 0x7f29e4152ae0 2024-03-28T23:37:06.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.611+0000 7f29e37fe640 1 -- 172.21.15.161:0/704426325 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29dc02cb90 con 0x7f29e4152ae0 2024-03-28T23:37:06.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.611+0000 7f29e37fe640 1 -- 172.21.15.161:0/704426325 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29dc0343f0 con 0x7f29e4152ae0 2024-03-28T23:37:06.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.611+0000 7f29eab7a640 1 -- 172.21.15.161:0/704426325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29e4152ae0 msgr2=0x7f29e4159250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:06.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.611+0000 7f29eab7a640 1 --2- 172.21.15.161:0/704426325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29e4152ae0 0x7f29e4159250 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f29e40ada60 tx=0x7f29dc02be50 comp rx=0 tx=0).stop 2024-03-28T23:37:06.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.611+0000 7f29eab7a640 1 -- 172.21.15.161:0/704426325 shutdown_connections 2024-03-28T23:37:06.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.612+0000 7f29eab7a640 1 --2- 172.21.15.161:0/704426325 >> v2:172.21.15.170:3300/0 conn(0x7f29e40c56c0 0x7f29e40c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.612+0000 7f29eab7a640 1 --2- 172.21.15.161:0/704426325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29e4152ae0 0x7f29e4159250 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.612+0000 7f29eab7a640 1 -- 172.21.15.161:0/704426325 >> 172.21.15.161:0/704426325 conn(0x7f29e40caaa0 msgr2=0x7f29e40caeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:06.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.612+0000 7f29eab7a640 1 -- 172.21.15.161:0/704426325 shutdown_connections 2024-03-28T23:37:06.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.612+0000 7f29eab7a640 1 -- 172.21.15.161:0/704426325 wait complete. 2024-03-28T23:37:06.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.613+0000 7f29eab7a640 1 Processor -- start 2024-03-28T23:37:06.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.613+0000 7f29eab7a640 1 -- start start 2024-03-28T23:37:06.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.613+0000 7f29eab7a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29e40afc80 0x7f29e406cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.614+0000 7f29eab7a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29e40c56c0 0x7f29e414a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.614+0000 7f29eab7a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29e4152ae0 0x7f29e40aae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.614+0000 7f29eab7a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f29e4152ec0 con 0x7f29e40afc80 2024-03-28T23:37:06.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.614+0000 7f29eab7a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f29e40c6150 con 0x7f29e40c56c0 2024-03-28T23:37:06.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.614+0000 7f29eab7a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f29e40adf70 con 0x7f29e4152ae0 2024-03-28T23:37:06.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.614+0000 7f29e8916640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29e40afc80 0x7f29e406cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.614+0000 7f29e8916640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29e40afc80 0x7f29e406cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37758/0 (socket says 172.21.15.161:37758) 2024-03-28T23:37:06.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.614+0000 7f29e8916640 1 -- 172.21.15.161:0/3143573087 learned_addr learned my addr 172.21.15.161:0/3143573087 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:06.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.614+0000 7f29e3fff640 1 --2- 172.21.15.161:0/3143573087 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29e40c56c0 0x7f29e414a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.614+0000 7f29e9117640 1 --2- 172.21.15.161:0/3143573087 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29e4152ae0 0x7f29e40aae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.614+0000 7f29e8916640 1 -- 172.21.15.161:0/3143573087 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29e4152ae0 msgr2=0x7f29e40aae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:06.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.614+0000 7f29e8916640 1 --2- 172.21.15.161:0/3143573087 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29e4152ae0 0x7f29e40aae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.614+0000 7f29e8916640 1 -- 172.21.15.161:0/3143573087 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29e40c56c0 msgr2=0x7f29e414a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:06.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.614+0000 7f29e8916640 1 --2- 172.21.15.161:0/3143573087 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29e40c56c0 0x7f29e414a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.614+0000 7f29e8916640 1 -- 172.21.15.161:0/3143573087 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29e4149ed0 con 0x7f29e40afc80 2024-03-28T23:37:06.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.614+0000 7f29e9117640 1 --2- 172.21.15.161:0/3143573087 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29e4152ae0 0x7f29e40aae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:37:06.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.614+0000 7f29e8916640 1 --2- 172.21.15.161:0/3143573087 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29e40afc80 0x7f29e406cc80 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f29dc000f80 tx=0x7f29dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:06.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.614+0000 7f29e17fa640 1 -- 172.21.15.161:0/3143573087 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29dc043020 con 0x7f29e40afc80 2024-03-28T23:37:06.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.614+0000 7f29e17fa640 1 -- 172.21.15.161:0/3143573087 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29dc03e450 con 0x7f29e40afc80 2024-03-28T23:37:06.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.614+0000 7f29eab7a640 1 -- 172.21.15.161:0/3143573087 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29e41532b0 con 0x7f29e40afc80 2024-03-28T23:37:06.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.614+0000 7f29e17fa640 1 -- 172.21.15.161:0/3143573087 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29dc03e820 con 0x7f29e40afc80 2024-03-28T23:37:06.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.614+0000 7f29eab7a640 1 -- 172.21.15.161:0/3143573087 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29e41556a0 con 0x7f29e40afc80 2024-03-28T23:37:06.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.615+0000 7f29eab7a640 1 -- 172.21.15.161:0/3143573087 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29e404f4b0 con 0x7f29e40afc80 2024-03-28T23:37:06.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.616+0000 7f29e17fa640 1 -- 172.21.15.161:0/3143573087 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f29dc005d10 con 0x7f29e40afc80 2024-03-28T23:37:06.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.616+0000 7f29e17fa640 1 --2- 172.21.15.161:0/3143573087 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f29c406b4d0 0x7f29c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.616+0000 7f29e17fa640 1 -- 172.21.15.161:0/3143573087 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 6651+0+0 (secure 0 0 0) 0x7f29dc0ad180 con 0x7f29e40afc80 2024-03-28T23:37:06.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.616+0000 7f29e3fff640 1 --2- 172.21.15.161:0/3143573087 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f29c406b4d0 0x7f29c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.617+0000 7f29e3fff640 1 --2- 172.21.15.161:0/3143573087 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f29c406b4d0 0x7f29c406d990 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f29cc006fd0 tx=0x7f29cc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:06.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.621+0000 7f29e17fa640 1 -- 172.21.15.161:0/3143573087 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f29dc07be00 con 0x7f29e40afc80 2024-03-28T23:37:06.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.792+0000 7f29eab7a640 1 -- 172.21.15.161:0/3143573087 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) -- 0x7f29e40caf20 con 0x7f29e40afc80 2024-03-28T23:37:06.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.793+0000 7f29e17fa640 1 -- 172.21.15.161:0/3143573087 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v165) ==== 105+0+11 (secure 0 0 0) 0x7f29dc07b700 con 0x7f29e40afc80 2024-03-28T23:37:06.793 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-28T23:37:06.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.794+0000 7f29eab7a640 1 -- 172.21.15.161:0/3143573087 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f29c406b4d0 msgr2=0x7f29c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:06.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.794+0000 7f29eab7a640 1 --2- 172.21.15.161:0/3143573087 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f29c406b4d0 0x7f29c406d990 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f29cc006fd0 tx=0x7f29cc008040 comp rx=0 tx=0).stop 2024-03-28T23:37:06.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.794+0000 7f29eab7a640 1 -- 172.21.15.161:0/3143573087 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29e40afc80 msgr2=0x7f29e406cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:06.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.794+0000 7f29eab7a640 1 --2- 172.21.15.161:0/3143573087 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29e40afc80 0x7f29e406cc80 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f29dc000f80 tx=0x7f29dc004a40 comp rx=0 tx=0).stop 2024-03-28T23:37:06.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.795+0000 7f29eab7a640 1 -- 172.21.15.161:0/3143573087 shutdown_connections 2024-03-28T23:37:06.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.795+0000 7f29eab7a640 1 --2- 172.21.15.161:0/3143573087 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f29c406b4d0 0x7f29c406d990 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.795+0000 7f29eab7a640 1 --2- 172.21.15.161:0/3143573087 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29e4152ae0 0x7f29e40aae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.795+0000 7f29eab7a640 1 --2- 172.21.15.161:0/3143573087 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29e40c56c0 0x7f29e414a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.795+0000 7f29eab7a640 1 --2- 172.21.15.161:0/3143573087 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29e40afc80 0x7f29e406cc80 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.795+0000 7f29eab7a640 1 -- 172.21.15.161:0/3143573087 >> 172.21.15.161:0/3143573087 conn(0x7f29e40caaa0 msgr2=0x7f29e40c6470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:06.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.795+0000 7f29eab7a640 1 -- 172.21.15.161:0/3143573087 shutdown_connections 2024-03-28T23:37:06.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.795+0000 7f29eab7a640 1 -- 172.21.15.161:0/3143573087 wait complete. 2024-03-28T23:37:06.807 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-03-28T23:37:06.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.914+0000 7f1373a23640 1 Processor -- start 2024-03-28T23:37:06.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.915+0000 7f1373a23640 1 -- start start 2024-03-28T23:37:06.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.916+0000 7f1373a23640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f136c06bb60 0x7f136c0c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.916+0000 7f1373a23640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f136c152db0 con 0x7f136c1529d0 2024-03-28T23:37:06.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.916+0000 7f1373a23640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f136c0c7210 con 0x7f136c0c6750 2024-03-28T23:37:06.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.916+0000 7f1373a23640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f136c0c6c50 con 0x7f136c06bb60 2024-03-28T23:37:06.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.916+0000 7f13717bf640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f136c1529d0 0x7f136c155c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48148/0 (socket says 172.21.15.161:48148) 2024-03-28T23:37:06.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.916+0000 7f13717bf640 1 -- 172.21.15.161:0/1944994275 learned_addr learned my addr 172.21.15.161:0/1944994275 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:06.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.916+0000 7f135ffff640 1 -- 172.21.15.161:0/1944994275 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4220638070 0 0) 0x7f136c152db0 con 0x7f136c1529d0 2024-03-28T23:37:06.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.916+0000 7f1371fc0640 1 --2- 172.21.15.161:0/1944994275 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f136c06bb60 0x7f136c0c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.916+0000 7f135ffff640 1 -- 172.21.15.161:0/1944994275 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1358003520 con 0x7f136c1529d0 2024-03-28T23:37:06.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.916+0000 7f135ffff640 1 -- 172.21.15.161:0/1944994275 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3545859002 0 0) 0x7f1358003520 con 0x7f136c1529d0 2024-03-28T23:37:06.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.916+0000 7f135ffff640 1 -- 172.21.15.161:0/1944994275 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f136c06bb60 msgr2=0x7f136c0c88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:06.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.916+0000 7f135ffff640 1 --2- 172.21.15.161:0/1944994275 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f136c06bb60 0x7f136c0c88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.916+0000 7f135ffff640 1 -- 172.21.15.161:0/1944994275 >> v1:172.21.15.170:6789/0 conn(0x7f136c0c6750 legacy=0x7f136c0c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:06.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.916+0000 7f135ffff640 1 -- 172.21.15.161:0/1944994275 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f136c0c6f30 con 0x7f136c1529d0 2024-03-28T23:37:06.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.916+0000 7f135ffff640 1 -- 172.21.15.161:0/1944994275 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f13640019f0 con 0x7f136c1529d0 2024-03-28T23:37:06.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.916+0000 7f135ffff640 1 -- 172.21.15.161:0/1944994275 >> v1:172.21.15.161:6789/0 conn(0x7f136c1529d0 legacy=0x7f136c155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:06.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.916+0000 7f135ffff640 1 --2- 172.21.15.161:0/1944994275 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13580039e0 0x7f1358003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.916+0000 7f135ffff640 1 --2- 172.21.15.161:0/1944994275 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1358004390 0x7f1358004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.916+0000 7f135ffff640 1 --2- 172.21.15.161:0/1944994275 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f136c06bb60 0x7f1358005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.916+0000 7f135ffff640 1 -- 172.21.15.161:0/1944994275 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f136c0c6c50 con 0x7f13580039e0 2024-03-28T23:37:06.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.916+0000 7f135ffff640 1 -- 172.21.15.161:0/1944994275 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f135800ba70 con 0x7f1358004390 2024-03-28T23:37:06.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.916+0000 7f135ffff640 1 -- 172.21.15.161:0/1944994275 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f135800bc70 con 0x7f136c06bb60 2024-03-28T23:37:06.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.917+0000 7f13717bf640 1 --2- 172.21.15.161:0/1944994275 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13580039e0 0x7f1358003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.917+0000 7f1370fbe640 1 --2- 172.21.15.161:0/1944994275 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1358004390 0x7f1358004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.917+0000 7f1371fc0640 1 --2- 172.21.15.161:0/1944994275 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f136c06bb60 0x7f1358005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.917+0000 7f13717bf640 1 -- 172.21.15.161:0/1944994275 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f136c06bb60 msgr2=0x7f1358005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:06.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.917+0000 7f13717bf640 1 --2- 172.21.15.161:0/1944994275 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f136c06bb60 0x7f1358005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.917+0000 7f13717bf640 1 -- 172.21.15.161:0/1944994275 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1358004390 msgr2=0x7f1358004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:06.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.917+0000 7f13717bf640 1 --2- 172.21.15.161:0/1944994275 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1358004390 0x7f1358004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.917+0000 7f13717bf640 1 -- 172.21.15.161:0/1944994275 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1358003520 con 0x7f13580039e0 2024-03-28T23:37:06.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.918+0000 7f13717bf640 1 --2- 172.21.15.161:0/1944994275 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13580039e0 0x7f1358003dc0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f136400a920 tx=0x7f1358005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54a9f0f2ef6d5caf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:06.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.918+0000 7f135ffff640 1 -- 172.21.15.161:0/1944994275 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f136400a4c0 con 0x7f13580039e0 2024-03-28T23:37:06.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.918+0000 7f135ffff640 1 -- 172.21.15.161:0/1944994275 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1364034650 con 0x7f13580039e0 2024-03-28T23:37:06.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.918+0000 7f135ffff640 1 -- 172.21.15.161:0/1944994275 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1364034a40 con 0x7f13580039e0 2024-03-28T23:37:06.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.918+0000 7f1373a23640 1 -- 172.21.15.161:0/1944994275 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13580039e0 msgr2=0x7f1358003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:06.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.918+0000 7f1373a23640 1 --2- 172.21.15.161:0/1944994275 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13580039e0 0x7f1358003dc0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f136400a920 tx=0x7f1358005830 comp rx=0 tx=0).stop 2024-03-28T23:37:06.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.918+0000 7f1371fc0640 1 -- 172.21.15.161:0/1944994275 reap_dead start 2024-03-28T23:37:06.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.918+0000 7f1373a23640 1 -- 172.21.15.161:0/1944994275 shutdown_connections 2024-03-28T23:37:06.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.918+0000 7f1373a23640 1 -- 172.21.15.161:0/1944994275 >> 172.21.15.161:0/1944994275 conn(0x7f136c0c7430 msgr2=0x7f136c0c7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:06.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.918+0000 7f1373a23640 1 -- 172.21.15.161:0/1944994275 shutdown_connections 2024-03-28T23:37:06.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.918+0000 7f1373a23640 1 -- 172.21.15.161:0/1944994275 wait complete. 2024-03-28T23:37:06.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.920+0000 7f1373a23640 1 Processor -- start 2024-03-28T23:37:06.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.920+0000 7f1373a23640 1 -- start start 2024-03-28T23:37:06.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.921+0000 7f1373a23640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13580039e0 0x7f136c1462c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.921+0000 7f1373a23640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f136c14f8f0 0x7f136c156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.921+0000 7f1373a23640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f136c154520 0x7f136c1eddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.921+0000 7f1373a23640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f136c0aa160 con 0x7f13580039e0 2024-03-28T23:37:06.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.921+0000 7f1373a23640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f136c155710 con 0x7f136c14f8f0 2024-03-28T23:37:06.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.921+0000 7f1373a23640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f136c0c5c00 con 0x7f136c154520 2024-03-28T23:37:06.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.921+0000 7f13717bf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13580039e0 0x7f136c1462c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.921+0000 7f13717bf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13580039e0 0x7f136c1462c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37766/0 (socket says 172.21.15.161:37766) 2024-03-28T23:37:06.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.921+0000 7f13717bf640 1 -- 172.21.15.161:0/1062553923 learned_addr learned my addr 172.21.15.161:0/1062553923 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:06.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.922+0000 7f1370fbe640 1 --2- 172.21.15.161:0/1062553923 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f136c14f8f0 0x7f136c156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.922+0000 7f1371fc0640 1 --2- 172.21.15.161:0/1062553923 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f136c154520 0x7f136c1eddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.922+0000 7f13717bf640 1 -- 172.21.15.161:0/1062553923 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f136c154520 msgr2=0x7f136c1eddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:06.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.922+0000 7f13717bf640 1 --2- 172.21.15.161:0/1062553923 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f136c154520 0x7f136c1eddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.922+0000 7f13717bf640 1 -- 172.21.15.161:0/1062553923 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f136c14f8f0 msgr2=0x7f136c156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:06.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.922+0000 7f13717bf640 1 --2- 172.21.15.161:0/1062553923 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f136c14f8f0 0x7f136c156900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:06.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.922+0000 7f13717bf640 1 -- 172.21.15.161:0/1062553923 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f136c1590d0 con 0x7f13580039e0 2024-03-28T23:37:06.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.922+0000 7f13717bf640 1 --2- 172.21.15.161:0/1062553923 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13580039e0 0x7f136c1462c0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f136402ccb0 tx=0x7f1364001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:06.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.922+0000 7f135dffb640 1 -- 172.21.15.161:0/1062553923 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1364004e30 con 0x7f13580039e0 2024-03-28T23:37:06.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.922+0000 7f135dffb640 1 -- 172.21.15.161:0/1062553923 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13640017b0 con 0x7f13580039e0 2024-03-28T23:37:06.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.922+0000 7f1373a23640 1 -- 172.21.15.161:0/1062553923 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f136c159340 con 0x7f13580039e0 2024-03-28T23:37:06.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.922+0000 7f135dffb640 1 -- 172.21.15.161:0/1062553923 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1364034800 con 0x7f13580039e0 2024-03-28T23:37:06.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.922+0000 7f1373a23640 1 -- 172.21.15.161:0/1062553923 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f136c153d20 con 0x7f13580039e0 2024-03-28T23:37:06.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.923+0000 7f1373a23640 1 -- 172.21.15.161:0/1062553923 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f136c04f4b0 con 0x7f13580039e0 2024-03-28T23:37:06.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.924+0000 7f135dffb640 1 -- 172.21.15.161:0/1062553923 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f13640349a0 con 0x7f13580039e0 2024-03-28T23:37:06.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.924+0000 7f135dffb640 1 --2- 172.21.15.161:0/1062553923 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f135006b500 0x7f135006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:06.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.924+0000 7f135dffb640 1 -- 172.21.15.161:0/1062553923 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 6651+0+0 (secure 0 0 0) 0x7f13640acbc0 con 0x7f13580039e0 2024-03-28T23:37:06.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.924+0000 7f1370fbe640 1 --2- 172.21.15.161:0/1062553923 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f135006b500 0x7f135006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:06.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.924+0000 7f1370fbe640 1 --2- 172.21.15.161:0/1062553923 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f135006b500 0x7f135006d9c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f136c148a90 tx=0x7f1360009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:06.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:06.928+0000 7f135dffb640 1 -- 172.21.15.161:0/1062553923 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f136407b720 con 0x7f13580039e0 2024-03-28T23:37:07.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.104+0000 7f1373a23640 1 -- 172.21.15.161:0/1062553923 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) -- 0x7f136c0c8180 con 0x7f13580039e0 2024-03-28T23:37:07.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.105+0000 7f135dffb640 1 -- 172.21.15.161:0/1062553923 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v165) ==== 101+0+11 (secure 0 0 0) 0x7f136407b020 con 0x7f13580039e0 2024-03-28T23:37:07.105 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-28T23:37:07.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.107+0000 7f1373a23640 1 -- 172.21.15.161:0/1062553923 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f135006b500 msgr2=0x7f135006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:07.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.107+0000 7f1373a23640 1 --2- 172.21.15.161:0/1062553923 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f135006b500 0x7f135006d9c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f136c148a90 tx=0x7f1360009040 comp rx=0 tx=0).stop 2024-03-28T23:37:07.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.107+0000 7f1373a23640 1 -- 172.21.15.161:0/1062553923 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13580039e0 msgr2=0x7f136c1462c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:07.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.107+0000 7f1373a23640 1 --2- 172.21.15.161:0/1062553923 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13580039e0 0x7f136c1462c0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f136402ccb0 tx=0x7f1364001dc0 comp rx=0 tx=0).stop 2024-03-28T23:37:07.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.107+0000 7f1373a23640 1 -- 172.21.15.161:0/1062553923 shutdown_connections 2024-03-28T23:37:07.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.107+0000 7f1373a23640 1 --2- 172.21.15.161:0/1062553923 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f135006b500 0x7f135006d9c0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:07.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.107+0000 7f1373a23640 1 --2- 172.21.15.161:0/1062553923 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f136c154520 0x7f136c1eddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:07.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.107+0000 7f1373a23640 1 --2- 172.21.15.161:0/1062553923 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f136c14f8f0 0x7f136c156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:07.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.107+0000 7f1373a23640 1 --2- 172.21.15.161:0/1062553923 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13580039e0 0x7f136c1462c0 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:07.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.107+0000 7f1373a23640 1 -- 172.21.15.161:0/1062553923 >> 172.21.15.161:0/1062553923 conn(0x7f136c0c7430 msgr2=0x7f136c151bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:07.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.107+0000 7f1373a23640 1 -- 172.21.15.161:0/1062553923 shutdown_connections 2024-03-28T23:37:07.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.107+0000 7f1373a23640 1 -- 172.21.15.161:0/1062553923 wait complete. 2024-03-28T23:37:07.118 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:37:07.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.220+0000 7f8076be2640 1 Processor -- start 2024-03-28T23:37:07.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.220+0000 7f8076be2640 1 -- start start 2024-03-28T23:37:07.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.221+0000 7f8076be2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f807006d9f0 0x7f8070155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:07.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.221+0000 7f8076be2640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f807006c3c0 con 0x7f8070152a80 2024-03-28T23:37:07.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.221+0000 7f8076be2640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80700c5b90 con 0x7f807006cb80 2024-03-28T23:37:07.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.221+0000 7f8076be2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8070156250 con 0x7f807006d9f0 2024-03-28T23:37:07.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.221+0000 7f807497e640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f8070152a80 0x7f80700c93a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48164/0 (socket says 172.21.15.161:48164) 2024-03-28T23:37:07.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.221+0000 7f807497e640 1 -- 172.21.15.161:0/129435933 learned_addr learned my addr 172.21.15.161:0/129435933 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:07.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.221+0000 7f806ffff640 1 --2- 172.21.15.161:0/129435933 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f807006d9f0 0x7f8070155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:07.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.221+0000 7f806f7fe640 1 -- 172.21.15.161:0/129435933 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4143452097 0 0) 0x7f807006c3c0 con 0x7f8070152a80 2024-03-28T23:37:07.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.221+0000 7f806f7fe640 1 -- 172.21.15.161:0/129435933 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8060003520 con 0x7f8070152a80 2024-03-28T23:37:07.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.222+0000 7f806f7fe640 1 -- 172.21.15.161:0/129435933 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 636630502 0 0) 0x7f80700c5b90 con 0x7f807006cb80 2024-03-28T23:37:07.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.222+0000 7f806f7fe640 1 -- 172.21.15.161:0/129435933 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f807006c3c0 con 0x7f807006cb80 2024-03-28T23:37:07.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.222+0000 7f806ffff640 1 -- 172.21.15.161:0/129435933 >> v1:172.21.15.170:6789/0 conn(0x7f807006cb80 legacy=0x7f80700c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:07.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.222+0000 7f806ffff640 1 -- 172.21.15.161:0/129435933 >> v1:172.21.15.161:6789/0 conn(0x7f8070152a80 legacy=0x7f80700c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:07.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.222+0000 7f806ffff640 1 -- 172.21.15.161:0/129435933 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80700c55d0 con 0x7f807006d9f0 2024-03-28T23:37:07.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.222+0000 7f806ffff640 1 --2- 172.21.15.161:0/129435933 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f807006d9f0 0x7f8070155d10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f805c0097d0 tx=0x7f805c02b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b50a871d3831edec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:07.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.223+0000 7f806f7fe640 1 -- 172.21.15.161:0/129435933 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805c031770 con 0x7f807006d9f0 2024-03-28T23:37:07.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.223+0000 7f806f7fe640 1 -- 172.21.15.161:0/129435933 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f805c037070 con 0x7f807006d9f0 2024-03-28T23:37:07.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.223+0000 7f806f7fe640 1 -- 172.21.15.161:0/129435933 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805c031b10 con 0x7f807006d9f0 2024-03-28T23:37:07.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.223+0000 7f8076be2640 1 -- 172.21.15.161:0/129435933 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f807006d9f0 msgr2=0x7f8070155d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:07.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.223+0000 7f8076be2640 1 --2- 172.21.15.161:0/129435933 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f807006d9f0 0x7f8070155d10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f805c0097d0 tx=0x7f805c02b810 comp rx=0 tx=0).stop 2024-03-28T23:37:07.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.223+0000 7f8076be2640 1 -- 172.21.15.161:0/129435933 shutdown_connections 2024-03-28T23:37:07.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.223+0000 7f8076be2640 1 --2- 172.21.15.161:0/129435933 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f807006d9f0 0x7f8070155d10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:07.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.223+0000 7f8076be2640 1 -- 172.21.15.161:0/129435933 >> 172.21.15.161:0/129435933 conn(0x7f80700ac720 msgr2=0x7f80700acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:07.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.223+0000 7f8076be2640 1 -- 172.21.15.161:0/129435933 shutdown_connections 2024-03-28T23:37:07.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.223+0000 7f8076be2640 1 -- 172.21.15.161:0/129435933 wait complete. 2024-03-28T23:37:07.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.224+0000 7f8076be2640 1 Processor -- start 2024-03-28T23:37:07.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.224+0000 7f8076be2640 1 -- start start 2024-03-28T23:37:07.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.225+0000 7f8076be2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f807006cb80 0x7f80701ee0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:07.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.225+0000 7f8076be2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f807006d9f0 0x7f80701578f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:07.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.225+0000 7f8076be2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8070152a80 0x7f8070157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:07.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.225+0000 7f8076be2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f80700c7bd0 con 0x7f8070152a80 2024-03-28T23:37:07.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.225+0000 7f8076be2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f807006c6f0 con 0x7f807006d9f0 2024-03-28T23:37:07.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.225+0000 7f8076be2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f80700c7d50 con 0x7f807006cb80 2024-03-28T23:37:07.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.225+0000 7f807517f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8070152a80 0x7f8070157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:07.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.225+0000 7f807517f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8070152a80 0x7f8070157e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37774/0 (socket says 172.21.15.161:37774) 2024-03-28T23:37:07.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.225+0000 7f807517f640 1 -- 172.21.15.161:0/2402184347 learned_addr learned my addr 172.21.15.161:0/2402184347 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:07.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.225+0000 7f807497e640 1 --2- 172.21.15.161:0/2402184347 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f807006cb80 0x7f80701ee0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:07.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.225+0000 7f806ffff640 1 --2- 172.21.15.161:0/2402184347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f807006d9f0 0x7f80701578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:07.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.226+0000 7f807517f640 1 -- 172.21.15.161:0/2402184347 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f807006cb80 msgr2=0x7f80701ee0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:07.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.226+0000 7f807517f640 1 --2- 172.21.15.161:0/2402184347 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f807006cb80 0x7f80701ee0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:07.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.226+0000 7f807517f640 1 -- 172.21.15.161:0/2402184347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f807006d9f0 msgr2=0x7f80701578f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:07.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.226+0000 7f807517f640 1 --2- 172.21.15.161:0/2402184347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f807006d9f0 0x7f80701578f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:07.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.226+0000 7f807517f640 1 -- 172.21.15.161:0/2402184347 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80700adb20 con 0x7f8070152a80 2024-03-28T23:37:07.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.226+0000 7f807517f640 1 --2- 172.21.15.161:0/2402184347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8070152a80 0x7f8070157e30 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f806800c790 tx=0x7f806800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:07.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.226+0000 7f806d7fa640 1 -- 172.21.15.161:0/2402184347 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8068003b70 con 0x7f8070152a80 2024-03-28T23:37:07.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.226+0000 7f806d7fa640 1 -- 172.21.15.161:0/2402184347 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8068003d10 con 0x7f8070152a80 2024-03-28T23:37:07.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.226+0000 7f8076be2640 1 -- 172.21.15.161:0/2402184347 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80700ad840 con 0x7f8070152a80 2024-03-28T23:37:07.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.226+0000 7f806d7fa640 1 -- 172.21.15.161:0/2402184347 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f806800aa50 con 0x7f8070152a80 2024-03-28T23:37:07.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.226+0000 7f8076be2640 1 -- 172.21.15.161:0/2402184347 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80701eec00 con 0x7f8070152a80 2024-03-28T23:37:07.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.227+0000 7f8076be2640 1 -- 172.21.15.161:0/2402184347 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f807004f4b0 con 0x7f8070152a80 2024-03-28T23:37:07.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.228+0000 7f806d7fa640 1 -- 172.21.15.161:0/2402184347 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f806800abf0 con 0x7f8070152a80 2024-03-28T23:37:07.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.228+0000 7f806d7fa640 1 --2- 172.21.15.161:0/2402184347 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f805406b4d0 0x7f805406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:07.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.228+0000 7f806d7fa640 1 -- 172.21.15.161:0/2402184347 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 6651+0+0 (secure 0 0 0) 0x7f806808a3a0 con 0x7f8070152a80 2024-03-28T23:37:07.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.228+0000 7f807497e640 1 --2- 172.21.15.161:0/2402184347 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f805406b4d0 0x7f805406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:07.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.229+0000 7f807497e640 1 --2- 172.21.15.161:0/2402184347 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f805406b4d0 0x7f805406d990 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f80640011e0 tx=0x7f8064007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:07.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.232+0000 7f806d7fa640 1 -- 172.21.15.161:0/2402184347 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f80680588b0 con 0x7f8070152a80 2024-03-28T23:37:07.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.492+0000 7f8076be2640 1 -- 172.21.15.161:0/2402184347 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80700c5b90 con 0x7f8070152a80 2024-03-28T23:37:07.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.492+0000 7f806d7fa640 1 -- 172.21.15.161:0/2402184347 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) ==== 95+0+5201 (secure 0 0 0) 0x7f806805c950 con 0x7f8070152a80 2024-03-28T23:37:07.493 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:37:07.493 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":160,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/557522063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":557522063},{"type":"v1","addr":"172.21.15.170:6835","nonce":557522063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":7499,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/3015739917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3015739917},{"type":"v1","addr":"172.21.15.170:6834","nonce":3015739917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":156},{"gid":9522,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/1423643374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1423643374},{"type":"v1","addr":"172.21.15.161:6836","nonce":1423643374}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":158}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:36:07.448976+0000","modified":"2024-03-28T23:37:06.880574+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":163,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9216},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9216":{"gid":9216,"name":"a","rank":0,"incarnation":157,"state":"up:active","state_seq":15,"addr":"172.21.15.161:6837/789976100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":789976100},{"type":"v1","addr":"172.21.15.161:6837","nonce":789976100}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9216,"qdb_cluster":[9216]},"id":12}]} 2024-03-28T23:37:07.493 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 160 2024-03-28T23:37:07.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.494+0000 7f8076be2640 1 -- 172.21.15.161:0/2402184347 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f805406b4d0 msgr2=0x7f805406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:07.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.494+0000 7f8076be2640 1 --2- 172.21.15.161:0/2402184347 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f805406b4d0 0x7f805406d990 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f80640011e0 tx=0x7f8064007d30 comp rx=0 tx=0).stop 2024-03-28T23:37:07.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.494+0000 7f8076be2640 1 -- 172.21.15.161:0/2402184347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8070152a80 msgr2=0x7f8070157e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:07.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.494+0000 7f8076be2640 1 --2- 172.21.15.161:0/2402184347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8070152a80 0x7f8070157e30 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f806800c790 tx=0x7f806800cc50 comp rx=0 tx=0).stop 2024-03-28T23:37:07.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.494+0000 7f8076be2640 1 -- 172.21.15.161:0/2402184347 shutdown_connections 2024-03-28T23:37:07.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.494+0000 7f8076be2640 1 --2- 172.21.15.161:0/2402184347 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f805406b4d0 0x7f805406d990 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:07.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.494+0000 7f8076be2640 1 --2- 172.21.15.161:0/2402184347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8070152a80 0x7f8070157e30 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:07.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.494+0000 7f8076be2640 1 --2- 172.21.15.161:0/2402184347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f807006d9f0 0x7f80701578f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:07.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.494+0000 7f8076be2640 1 --2- 172.21.15.161:0/2402184347 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f807006cb80 0x7f80701ee0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:07.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.494+0000 7f8076be2640 1 -- 172.21.15.161:0/2402184347 >> 172.21.15.161:0/2402184347 conn(0x7f80700ac720 msgr2=0x7f80700acb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:07.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.494+0000 7f8076be2640 1 -- 172.21.15.161:0/2402184347 shutdown_connections 2024-03-28T23:37:07.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.494+0000 7f8076be2640 1 -- 172.21.15.161:0/2402184347 wait complete. 2024-03-28T23:37:07.505 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:37:07.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.606+0000 7fc1a27bf640 1 Processor -- start 2024-03-28T23:37:07.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.606+0000 7fc1a27bf640 1 -- start start 2024-03-28T23:37:07.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.607+0000 7fc1a27bf640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc19c0c5410 0x7fc19c0c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:07.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.607+0000 7fc1a27bf640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc19c158a30 con 0x7fc19c152890 2024-03-28T23:37:07.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.607+0000 7fc1a27bf640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc19c158d10 con 0x7fc19c1553a0 2024-03-28T23:37:07.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.607+0000 7fc1a27bf640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fc19c0c8480 con 0x7fc19c0c5410 2024-03-28T23:37:07.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.607+0000 7fc19bfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc19c152890 0x7fc19c0c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48178/0 (socket says 172.21.15.161:48178) 2024-03-28T23:37:07.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.607+0000 7fc19bfff640 1 -- 172.21.15.161:0/2974070548 learned_addr learned my addr 172.21.15.161:0/2974070548 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:07.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.608+0000 7fc19b7fe640 1 --2- 172.21.15.161:0/2974070548 >> v2:172.21.15.170:3300/0 conn(0x7fc19c0c5410 0x7fc19c0c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:07.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.608+0000 7fc19affd640 1 -- 172.21.15.161:0/2974070548 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2153059498 0 0) 0x7fc19c158a30 con 0x7fc19c152890 2024-03-28T23:37:07.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.608+0000 7fc19affd640 1 -- 172.21.15.161:0/2974070548 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc184003520 con 0x7fc19c152890 2024-03-28T23:37:07.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.608+0000 7fc19b7fe640 1 -- 172.21.15.161:0/2974070548 >> v1:172.21.15.170:6789/0 conn(0x7fc19c1553a0 legacy=0x7fc19c155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:07.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.608+0000 7fc19affd640 1 -- 172.21.15.161:0/2974070548 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3900207892 0 0) 0x7fc184003520 con 0x7fc19c152890 2024-03-28T23:37:07.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.608+0000 7fc19b7fe640 1 -- 172.21.15.161:0/2974070548 >> v1:172.21.15.161:6789/0 conn(0x7fc19c152890 legacy=0x7fc19c0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:07.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.608+0000 7fc19b7fe640 1 -- 172.21.15.161:0/2974070548 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc19c158ff0 con 0x7fc19c0c5410 2024-03-28T23:37:07.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.609+0000 7fc19b7fe640 1 --2- 172.21.15.161:0/2974070548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc19c0c5410 0x7fc19c0c5850 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fc190009790 tx=0x7fc19002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbe465252fcb41db server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:07.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.609+0000 7fc19affd640 1 -- 172.21.15.161:0/2974070548 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc190037070 con 0x7fc19c0c5410 2024-03-28T23:37:07.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.609+0000 7fc19affd640 1 -- 172.21.15.161:0/2974070548 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc190030510 con 0x7fc19c0c5410 2024-03-28T23:37:07.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.609+0000 7fc19affd640 1 -- 172.21.15.161:0/2974070548 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18c001670 con 0x7fc19c0c5410 2024-03-28T23:37:07.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.610+0000 7fc1a27bf640 1 -- 172.21.15.161:0/2974070548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc19c0c5410 msgr2=0x7fc19c0c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:07.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.610+0000 7fc1a27bf640 1 --2- 172.21.15.161:0/2974070548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc19c0c5410 0x7fc19c0c5850 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fc190009790 tx=0x7fc19002b790 comp rx=0 tx=0).stop 2024-03-28T23:37:07.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.610+0000 7fc1a27bf640 1 -- 172.21.15.161:0/2974070548 shutdown_connections 2024-03-28T23:37:07.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.610+0000 7fc1a27bf640 1 --2- 172.21.15.161:0/2974070548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc19c0c5410 0x7fc19c0c5850 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:07.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.610+0000 7fc1a27bf640 1 -- 172.21.15.161:0/2974070548 >> 172.21.15.161:0/2974070548 conn(0x7fc19c0ac620 msgr2=0x7fc19c0aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:07.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.610+0000 7fc1a27bf640 1 -- 172.21.15.161:0/2974070548 shutdown_connections 2024-03-28T23:37:07.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.610+0000 7fc1a27bf640 1 -- 172.21.15.161:0/2974070548 wait complete. 2024-03-28T23:37:07.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.611+0000 7fc1a27bf640 1 Processor -- start 2024-03-28T23:37:07.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.611+0000 7fc1a27bf640 1 -- start start 2024-03-28T23:37:07.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.612+0000 7fc1a27bf640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc19c0c5410 0x7fc19c14d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:07.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.612+0000 7fc1a27bf640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc19c152890 0x7fc19c0c1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:07.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.612+0000 7fc1a27bf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc19c1553a0 0x7fc19c1eb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:07.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.612+0000 7fc1a27bf640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc19c159640 con 0x7fc19c1553a0 2024-03-28T23:37:07.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.612+0000 7fc1a27bf640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc19c0c5d90 con 0x7fc19c152890 2024-03-28T23:37:07.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.612+0000 7fc1a27bf640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc19c0c90a0 con 0x7fc19c0c5410 2024-03-28T23:37:07.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.612+0000 7fc1a0d5c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc19c1553a0 0x7fc19c1eb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:07.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.612+0000 7fc1a0d5c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc19c1553a0 0x7fc19c1eb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37784/0 (socket says 172.21.15.161:37784) 2024-03-28T23:37:07.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.612+0000 7fc1a0d5c640 1 -- 172.21.15.161:0/3567661279 learned_addr learned my addr 172.21.15.161:0/3567661279 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:07.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.612+0000 7fc19b7fe640 1 --2- 172.21.15.161:0/3567661279 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc19c152890 0x7fc19c0c1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:07.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.612+0000 7fc19bfff640 1 --2- 172.21.15.161:0/3567661279 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc19c0c5410 0x7fc19c14d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:07.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.612+0000 7fc1a0d5c640 1 -- 172.21.15.161:0/3567661279 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc19c0c5410 msgr2=0x7fc19c14d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:07.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.612+0000 7fc1a0d5c640 1 --2- 172.21.15.161:0/3567661279 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc19c0c5410 0x7fc19c14d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:07.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.612+0000 7fc1a0d5c640 1 -- 172.21.15.161:0/3567661279 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc19c152890 msgr2=0x7fc19c0c1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:07.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.612+0000 7fc1a0d5c640 1 --2- 172.21.15.161:0/3567661279 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc19c152890 0x7fc19c0c1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:07.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.612+0000 7fc1a0d5c640 1 -- 172.21.15.161:0/3567661279 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc19c1ebc60 con 0x7fc19c1553a0 2024-03-28T23:37:07.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.612+0000 7fc1a0d5c640 1 --2- 172.21.15.161:0/3567661279 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc19c1553a0 0x7fc19c1eb720 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fc19400c7a0 tx=0x7fc19400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:07.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.613+0000 7fc198ff9640 1 -- 172.21.15.161:0/3567661279 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc194003b70 con 0x7fc19c1553a0 2024-03-28T23:37:07.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.613+0000 7fc198ff9640 1 -- 172.21.15.161:0/3567661279 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc194003d30 con 0x7fc19c1553a0 2024-03-28T23:37:07.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.613+0000 7fc198ff9640 1 -- 172.21.15.161:0/3567661279 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19400aa30 con 0x7fc19c1553a0 2024-03-28T23:37:07.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.613+0000 7fc1a27bf640 1 -- 172.21.15.161:0/3567661279 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc19c14b730 con 0x7fc19c1553a0 2024-03-28T23:37:07.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.613+0000 7fc1a27bf640 1 -- 172.21.15.161:0/3567661279 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc19c149dc0 con 0x7fc19c1553a0 2024-03-28T23:37:07.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.614+0000 7fc1a27bf640 1 -- 172.21.15.161:0/3567661279 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc19c1499b0 con 0x7fc19c1553a0 2024-03-28T23:37:07.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.614+0000 7fc198ff9640 1 -- 172.21.15.161:0/3567661279 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc19400abd0 con 0x7fc19c1553a0 2024-03-28T23:37:07.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.615+0000 7fc198ff9640 1 --2- 172.21.15.161:0/3567661279 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc18806b4d0 0x7fc18806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:07.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.615+0000 7fc19bfff640 1 --2- 172.21.15.161:0/3567661279 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc18806b4d0 0x7fc18806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:07.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.615+0000 7fc198ff9640 1 -- 172.21.15.161:0/3567661279 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 6651+0+0 (secure 0 0 0) 0x7fc19408a2c0 con 0x7fc19c1553a0 2024-03-28T23:37:07.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.615+0000 7fc19bfff640 1 --2- 172.21.15.161:0/3567661279 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc18806b4d0 0x7fc18806d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fc18c005880 tx=0x7fc18c0057f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:07.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.619+0000 7fc198ff9640 1 -- 172.21.15.161:0/3567661279 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc19c1499b0 con 0x7fc19c1553a0 2024-03-28T23:37:07.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.797+0000 7fc1a27bf640 1 -- 172.21.15.161:0/3567661279 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc19c04f4b0 con 0x7fc19c1553a0 2024-03-28T23:37:07.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.798+0000 7fc198ff9640 1 -- 172.21.15.161:0/3567661279 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v165) ==== 74+0+17836 (secure 0 0 0) 0x7fc19408a0c0 con 0x7fc19c1553a0 2024-03-28T23:37:07.798 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:37:07.798 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":165,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:37:05.877513+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:36:05.484451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:36:06.423073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.c","mappings":[{"from":1,"to":0}]},{"pgid":"26.13","mappings":[{"from":1,"to":6}]},{"pgid":"26.28","mappings":[{"from":1,"to":0}]},{"pgid":"26.3b","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6834/4026332703":"2024-03-29T23:37:03.868933+0000","172.21.15.170:6834/2996092857":"2024-03-29T23:37:02.855183+0000","172.21.15.161:6836/4026332703":"2024-03-29T23:37:03.868933+0000","172.21.15.170:6832/2996092857":"2024-03-29T23:37:02.855183+0000","172.21.15.161:6837/3236742821":"2024-03-29T23:36:17.431214+0000","172.21.15.161:6835/3236742821":"2024-03-29T23:36:17.431214+0000","172.21.15.170:6834/1393137291":"2024-03-29T23:36:14.412210+0000","172.21.15.170:6832/1393137291":"2024-03-29T23:36:14.412210+0000","172.21.15.161:6837/2391019771":"2024-03-29T23:36:13.968386+0000","172.21.15.161:6835/2391019771":"2024-03-29T23:36:13.968386+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:37:07.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.800+0000 7fc1a27bf640 1 -- 172.21.15.161:0/3567661279 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc18806b4d0 msgr2=0x7fc18806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:07.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.800+0000 7fc1a27bf640 1 --2- 172.21.15.161:0/3567661279 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc18806b4d0 0x7fc18806d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fc18c005880 tx=0x7fc18c0057f0 comp rx=0 tx=0).stop 2024-03-28T23:37:07.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.800+0000 7fc1a27bf640 1 -- 172.21.15.161:0/3567661279 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc19c1553a0 msgr2=0x7fc19c1eb720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:07.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.800+0000 7fc1a27bf640 1 --2- 172.21.15.161:0/3567661279 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc19c1553a0 0x7fc19c1eb720 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fc19400c7a0 tx=0x7fc19400cc60 comp rx=0 tx=0).stop 2024-03-28T23:37:07.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.800+0000 7fc1a27bf640 1 -- 172.21.15.161:0/3567661279 shutdown_connections 2024-03-28T23:37:07.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.800+0000 7fc1a27bf640 1 --2- 172.21.15.161:0/3567661279 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc18806b4d0 0x7fc18806d990 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:07.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.800+0000 7fc1a27bf640 1 --2- 172.21.15.161:0/3567661279 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc19c1553a0 0x7fc19c1eb720 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:07.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.800+0000 7fc1a27bf640 1 --2- 172.21.15.161:0/3567661279 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc19c152890 0x7fc19c0c1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:07.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.800+0000 7fc1a27bf640 1 --2- 172.21.15.161:0/3567661279 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc19c0c5410 0x7fc19c14d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:07.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.800+0000 7fc1a27bf640 1 -- 172.21.15.161:0/3567661279 >> 172.21.15.161:0/3567661279 conn(0x7fc19c0ac620 msgr2=0x7fc19c06caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:07.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.800+0000 7fc1a27bf640 1 -- 172.21.15.161:0/3567661279 shutdown_connections 2024-03-28T23:37:07.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.800+0000 7fc1a27bf640 1 -- 172.21.15.161:0/3567661279 wait complete. 2024-03-28T23:37:07.811 INFO:tasks.cephfs.filesystem:Destroying file system cephfs2 and related pools 2024-03-28T23:37:07.812 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:37:07.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.920+0000 7fa42e047640 1 Processor -- start 2024-03-28T23:37:07.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.920+0000 7fa42e047640 1 -- start start 2024-03-28T23:37:07.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.920+0000 7fa42e047640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa42806ce80 0x7fa4280aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:07.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.920+0000 7fa42e047640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa42814c250 con 0x7fa4280a9680 2024-03-28T23:37:07.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.920+0000 7fa42e047640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4281529c0 con 0x7fa42814be70 2024-03-28T23:37:07.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.920+0000 7fa42e047640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa4280a9dc0 con 0x7fa42806ce80 2024-03-28T23:37:07.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.920+0000 7fa426ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa42806ce80 0x7fa4280aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:07.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.920+0000 7fa427fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa4280a9680 0x7fa42815c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48192/0 (socket says 172.21.15.161:48192) 2024-03-28T23:37:07.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.920+0000 7fa427fff640 1 -- 172.21.15.161:0/3172839083 learned_addr learned my addr 172.21.15.161:0/3172839083 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:07.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.921+0000 7fa426ffd640 1 -- 172.21.15.161:0/3172839083 >> v1:172.21.15.170:6789/0 conn(0x7fa42814be70 legacy=0x7fa4280ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:07.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.921+0000 7fa426ffd640 1 -- 172.21.15.161:0/3172839083 >> v1:172.21.15.161:6789/0 conn(0x7fa4280a9680 legacy=0x7fa42815c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:07.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.921+0000 7fa4267fc640 1 -- 172.21.15.161:0/3172839083 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1196975961 0 0) 0x7fa42814c250 con 0x7fa4280a9680 2024-03-28T23:37:07.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.921+0000 7fa426ffd640 1 -- 172.21.15.161:0/3172839083 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4280aa0a0 con 0x7fa42806ce80 2024-03-28T23:37:07.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.922+0000 7fa426ffd640 1 --2- 172.21.15.161:0/3172839083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa42806ce80 0x7fa4280aa5c0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0097a0 tx=0x7fa41c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22d8be1dfbf0f5b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:07.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.922+0000 7fa4267fc640 1 -- 172.21.15.161:0/3172839083 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41c035020 con 0x7fa42806ce80 2024-03-28T23:37:07.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.922+0000 7fa4267fc640 1 -- 172.21.15.161:0/3172839083 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa41c031830 con 0x7fa42806ce80 2024-03-28T23:37:07.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.922+0000 7fa4267fc640 1 -- 172.21.15.161:0/3172839083 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41c031c00 con 0x7fa42806ce80 2024-03-28T23:37:07.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.923+0000 7fa42e047640 1 -- 172.21.15.161:0/3172839083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa42806ce80 msgr2=0x7fa4280aa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:07.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.923+0000 7fa42e047640 1 --2- 172.21.15.161:0/3172839083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa42806ce80 0x7fa4280aa5c0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0097a0 tx=0x7fa41c02b8d0 comp rx=0 tx=0).stop 2024-03-28T23:37:07.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.923+0000 7fa42e047640 1 -- 172.21.15.161:0/3172839083 shutdown_connections 2024-03-28T23:37:07.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.923+0000 7fa42e047640 1 --2- 172.21.15.161:0/3172839083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa42806ce80 0x7fa4280aa5c0 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:07.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.923+0000 7fa42e047640 1 -- 172.21.15.161:0/3172839083 >> 172.21.15.161:0/3172839083 conn(0x7fa4280c6670 msgr2=0x7fa4280c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:07.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.923+0000 7fa42e047640 1 -- 172.21.15.161:0/3172839083 shutdown_connections 2024-03-28T23:37:07.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.923+0000 7fa42e047640 1 -- 172.21.15.161:0/3172839083 wait complete. 2024-03-28T23:37:07.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.924+0000 7fa42e047640 1 Processor -- start 2024-03-28T23:37:07.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.924+0000 7fa42e047640 1 -- start start 2024-03-28T23:37:07.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.925+0000 7fa42e047640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa42806ce80 0x7fa4280c54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:07.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.925+0000 7fa42e047640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4280a9680 0x7fa42814e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:07.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.925+0000 7fa42e047640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa42814be70 0x7fa42814edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:07.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.925+0000 7fa42e047640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa4280aba30 con 0x7fa42814be70 2024-03-28T23:37:07.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.925+0000 7fa42e047640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa4280ab8b0 con 0x7fa4280a9680 2024-03-28T23:37:07.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.925+0000 7fa42e047640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa42806b150 con 0x7fa42806ce80 2024-03-28T23:37:07.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.925+0000 7fa427fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa42814be70 0x7fa42814edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:07.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.925+0000 7fa427fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa42814be70 0x7fa42814edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37790/0 (socket says 172.21.15.161:37790) 2024-03-28T23:37:07.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.925+0000 7fa427fff640 1 -- 172.21.15.161:0/3255466898 learned_addr learned my addr 172.21.15.161:0/3255466898 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:07.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.925+0000 7fa426ffd640 1 --2- 172.21.15.161:0/3255466898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4280a9680 0x7fa42814e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:07.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.925+0000 7fa4277fe640 1 --2- 172.21.15.161:0/3255466898 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa42806ce80 0x7fa4280c54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:07.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.925+0000 7fa427fff640 1 -- 172.21.15.161:0/3255466898 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa42806ce80 msgr2=0x7fa4280c54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:07.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.925+0000 7fa427fff640 1 --2- 172.21.15.161:0/3255466898 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa42806ce80 0x7fa4280c54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:07.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.925+0000 7fa427fff640 1 -- 172.21.15.161:0/3255466898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4280a9680 msgr2=0x7fa42814e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:07.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.925+0000 7fa427fff640 1 --2- 172.21.15.161:0/3255466898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4280a9680 0x7fa42814e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:07.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.925+0000 7fa427fff640 1 -- 172.21.15.161:0/3255466898 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4281e76d0 con 0x7fa42814be70 2024-03-28T23:37:07.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.925+0000 7fa4277fe640 1 --2- 172.21.15.161:0/3255466898 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa42806ce80 0x7fa4280c54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:37:07.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.925+0000 7fa426ffd640 1 --2- 172.21.15.161:0/3255466898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4280a9680 0x7fa42814e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:37:07.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.925+0000 7fa427fff640 1 --2- 172.21.15.161:0/3255466898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa42814be70 0x7fa42814edc0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fa42000c730 tx=0x7fa42000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:07.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.925+0000 7fa40ffff640 1 -- 172.21.15.161:0/3255466898 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa420018070 con 0x7fa42814be70 2024-03-28T23:37:07.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.925+0000 7fa40ffff640 1 -- 172.21.15.161:0/3255466898 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa42000ce40 con 0x7fa42814be70 2024-03-28T23:37:07.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.925+0000 7fa42e047640 1 -- 172.21.15.161:0/3255466898 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4281ddcb0 con 0x7fa42814be70 2024-03-28T23:37:07.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.925+0000 7fa40ffff640 1 -- 172.21.15.161:0/3255466898 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42000a860 con 0x7fa42814be70 2024-03-28T23:37:07.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.925+0000 7fa42e047640 1 -- 172.21.15.161:0/3255466898 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4281db550 con 0x7fa42814be70 2024-03-28T23:37:07.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.926+0000 7fa42e047640 1 -- 172.21.15.161:0/3255466898 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa42804f4b0 con 0x7fa42814be70 2024-03-28T23:37:07.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.927+0000 7fa40ffff640 1 -- 172.21.15.161:0/3255466898 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa42001b070 con 0x7fa42814be70 2024-03-28T23:37:07.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.927+0000 7fa40ffff640 1 --2- 172.21.15.161:0/3255466898 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa41406b500 0x7fa41406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:07.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.927+0000 7fa40ffff640 1 -- 172.21.15.161:0/3255466898 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 6651+0+0 (secure 0 0 0) 0x7fa420089ad0 con 0x7fa42814be70 2024-03-28T23:37:07.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.927+0000 7fa4277fe640 1 --2- 172.21.15.161:0/3255466898 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa41406b500 0x7fa41406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:07.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.928+0000 7fa4277fe640 1 --2- 172.21.15.161:0/3255466898 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa41406b500 0x7fa41406d9c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fa418000f60 tx=0x7fa418009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:07.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:07.931+0000 7fa40ffff640 1 -- 172.21.15.161:0/3255466898 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa420058630 con 0x7fa42814be70 2024-03-28T23:37:08.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.147+0000 7fa42e047640 1 -- 172.21.15.161:0/3255466898 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa428156230 con 0x7fa42814be70 2024-03-28T23:37:08.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.147+0000 7fa40ffff640 1 -- 172.21.15.161:0/3255466898 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) ==== 95+0+5201 (secure 0 0 0) 0x7fa420057f30 con 0x7fa42814be70 2024-03-28T23:37:08.148 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:37:08.148 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":160,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/557522063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":557522063},{"type":"v1","addr":"172.21.15.170:6835","nonce":557522063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":7499,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/3015739917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3015739917},{"type":"v1","addr":"172.21.15.170:6834","nonce":3015739917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":156},{"gid":9522,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/1423643374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1423643374},{"type":"v1","addr":"172.21.15.161:6836","nonce":1423643374}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":158}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:36:07.448976+0000","modified":"2024-03-28T23:37:06.880574+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":163,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9216},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9216":{"gid":9216,"name":"a","rank":0,"incarnation":157,"state":"up:active","state_seq":15,"addr":"172.21.15.161:6837/789976100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":789976100},{"type":"v1","addr":"172.21.15.161:6837","nonce":789976100}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9216,"qdb_cluster":[9216]},"id":12}]} 2024-03-28T23:37:08.148 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 160 2024-03-28T23:37:08.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.149+0000 7fa42e047640 1 -- 172.21.15.161:0/3255466898 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa41406b500 msgr2=0x7fa41406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:08.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.149+0000 7fa42e047640 1 --2- 172.21.15.161:0/3255466898 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa41406b500 0x7fa41406d9c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fa418000f60 tx=0x7fa418009040 comp rx=0 tx=0).stop 2024-03-28T23:37:08.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.149+0000 7fa42e047640 1 -- 172.21.15.161:0/3255466898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa42814be70 msgr2=0x7fa42814edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:08.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.149+0000 7fa42e047640 1 --2- 172.21.15.161:0/3255466898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa42814be70 0x7fa42814edc0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fa42000c730 tx=0x7fa42000cbf0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.149+0000 7fa42e047640 1 -- 172.21.15.161:0/3255466898 shutdown_connections 2024-03-28T23:37:08.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.149+0000 7fa42e047640 1 --2- 172.21.15.161:0/3255466898 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa41406b500 0x7fa41406d9c0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.149+0000 7fa42e047640 1 --2- 172.21.15.161:0/3255466898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa42814be70 0x7fa42814edc0 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.149+0000 7fa42e047640 1 --2- 172.21.15.161:0/3255466898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4280a9680 0x7fa42814e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.149+0000 7fa42e047640 1 --2- 172.21.15.161:0/3255466898 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa42806ce80 0x7fa4280c54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.149+0000 7fa42e047640 1 -- 172.21.15.161:0/3255466898 >> 172.21.15.161:0/3255466898 conn(0x7fa4280c6670 msgr2=0x7fa4280ab1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:08.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.149+0000 7fa42e047640 1 -- 172.21.15.161:0/3255466898 shutdown_connections 2024-03-28T23:37:08.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.149+0000 7fa42e047640 1 -- 172.21.15.161:0/3255466898 wait complete. 2024-03-28T23:37:08.161 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:37:08.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.263+0000 7f65c14eb640 1 Processor -- start 2024-03-28T23:37:08.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.263+0000 7f65c14eb640 1 -- start start 2024-03-28T23:37:08.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.263+0000 7f65c14eb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65bc06ca20 0x7f65bc06ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:08.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.263+0000 7f65c14eb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f65bc0a98c0 0x7f65bc0a9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:08.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.263+0000 7f65c14eb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65bc151850 con 0x7f65bc14eca0 2024-03-28T23:37:08.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.263+0000 7f65c14eb640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f65bc152460 con 0x7f65bc0a98c0 2024-03-28T23:37:08.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.263+0000 7f65c14eb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f65bc151b30 con 0x7f65bc06ca20 2024-03-28T23:37:08.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.264+0000 7f65bb7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f65bc0a98c0 0x7f65bc0a9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:08.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.264+0000 7f65baffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f65bc14eca0 0x7f65bc0c1880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48196/0 (socket says 172.21.15.161:48196) 2024-03-28T23:37:08.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.264+0000 7f65bb7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f65bc0a98c0 0x7f65bc0a9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37796/0 (socket says 172.21.15.161:37796) 2024-03-28T23:37:08.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.264+0000 7f65baffd640 1 -- 172.21.15.161:0/2553773690 learned_addr learned my addr 172.21.15.161:0/2553773690 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:08.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.264+0000 7f65ba7fc640 1 --2- 172.21.15.161:0/2553773690 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65bc06ca20 0x7f65bc06ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:08.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.264+0000 7f65bb7fe640 1 -- 172.21.15.161:0/2553773690 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65bc06ca20 msgr2=0x7f65bc06ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:08.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.264+0000 7f65bb7fe640 1 --2- 172.21.15.161:0/2553773690 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65bc06ca20 0x7f65bc06ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.264+0000 7f65b9ffb640 1 -- 172.21.15.161:0/2553773690 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1215469281 0 0) 0x7f65bc151850 con 0x7f65bc14eca0 2024-03-28T23:37:08.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.264+0000 7f65bb7fe640 1 -- 172.21.15.161:0/2553773690 >> v1:172.21.15.161:6789/0 conn(0x7f65bc14eca0 legacy=0x7f65bc0c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:08.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.264+0000 7f65bb7fe640 1 -- 172.21.15.161:0/2553773690 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65bc151e10 con 0x7f65bc0a98c0 2024-03-28T23:37:08.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.265+0000 7f65bb7fe640 1 --2- 172.21.15.161:0/2553773690 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65bc0a98c0 0x7f65bc0a9d20 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f65b4009d90 tx=0x7f65b402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45d25cd65451a613 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:08.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.265+0000 7f65b9ffb640 1 -- 172.21.15.161:0/2553773690 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b4039070 con 0x7f65bc0a98c0 2024-03-28T23:37:08.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.265+0000 7f65b9ffb640 1 -- 172.21.15.161:0/2553773690 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65b402cbc0 con 0x7f65bc0a98c0 2024-03-28T23:37:08.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.265+0000 7f65b9ffb640 1 -- 172.21.15.161:0/2553773690 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b4034440 con 0x7f65bc0a98c0 2024-03-28T23:37:08.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.265+0000 7f65c14eb640 1 -- 172.21.15.161:0/2553773690 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65bc0a98c0 msgr2=0x7f65bc0a9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:08.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.265+0000 7f65c14eb640 1 --2- 172.21.15.161:0/2553773690 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65bc0a98c0 0x7f65bc0a9d20 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f65b4009d90 tx=0x7f65b402be10 comp rx=0 tx=0).stop 2024-03-28T23:37:08.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.265+0000 7f65c14eb640 1 -- 172.21.15.161:0/2553773690 shutdown_connections 2024-03-28T23:37:08.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.265+0000 7f65c14eb640 1 --2- 172.21.15.161:0/2553773690 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65bc0a98c0 0x7f65bc0a9d20 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.265+0000 7f65c14eb640 1 --2- 172.21.15.161:0/2553773690 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65bc06ca20 0x7f65bc06ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.265+0000 7f65c14eb640 1 -- 172.21.15.161:0/2553773690 >> 172.21.15.161:0/2553773690 conn(0x7f65bc0c6670 msgr2=0x7f65bc0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:08.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.265+0000 7f65c14eb640 1 -- 172.21.15.161:0/2553773690 shutdown_connections 2024-03-28T23:37:08.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.265+0000 7f65c14eb640 1 -- 172.21.15.161:0/2553773690 wait complete. 2024-03-28T23:37:08.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.267+0000 7f65c14eb640 1 Processor -- start 2024-03-28T23:37:08.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.267+0000 7f65c14eb640 1 -- start start 2024-03-28T23:37:08.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.267+0000 7f65c14eb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65bc06ca20 0x7f65bc06b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:08.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.267+0000 7f65c14eb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65bc0a98c0 0x7f65bc147ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:08.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.267+0000 7f65c14eb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65bc14eca0 0x7f65bc154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:08.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.267+0000 7f65c14eb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f65bc1514b0 con 0x7f65bc06ca20 2024-03-28T23:37:08.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.267+0000 7f65c14eb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f65bc14f080 con 0x7f65bc0a98c0 2024-03-28T23:37:08.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.267+0000 7f65c14eb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f65bc151630 con 0x7f65bc14eca0 2024-03-28T23:37:08.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.267+0000 7f65baffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65bc06ca20 0x7f65bc06b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:08.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.267+0000 7f65baffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65bc06ca20 0x7f65bc06b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37798/0 (socket says 172.21.15.161:37798) 2024-03-28T23:37:08.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.267+0000 7f65baffd640 1 -- 172.21.15.161:0/3390517922 learned_addr learned my addr 172.21.15.161:0/3390517922 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:08.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.268+0000 7f65bb7fe640 1 --2- 172.21.15.161:0/3390517922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65bc14eca0 0x7f65bc154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:08.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.268+0000 7f65ba7fc640 1 --2- 172.21.15.161:0/3390517922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65bc0a98c0 0x7f65bc147ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:08.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.268+0000 7f65baffd640 1 -- 172.21.15.161:0/3390517922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65bc14eca0 msgr2=0x7f65bc154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:08.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.268+0000 7f65baffd640 1 --2- 172.21.15.161:0/3390517922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65bc14eca0 0x7f65bc154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.268+0000 7f65baffd640 1 -- 172.21.15.161:0/3390517922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65bc0a98c0 msgr2=0x7f65bc147ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:08.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.268+0000 7f65baffd640 1 --2- 172.21.15.161:0/3390517922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65bc0a98c0 0x7f65bc147ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.268+0000 7f65baffd640 1 -- 172.21.15.161:0/3390517922 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65bc152af0 con 0x7f65bc06ca20 2024-03-28T23:37:08.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.268+0000 7f65bb7fe640 1 --2- 172.21.15.161:0/3390517922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65bc14eca0 0x7f65bc154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:37:08.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.268+0000 7f65baffd640 1 --2- 172.21.15.161:0/3390517922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65bc06ca20 0x7f65bc06b3c0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f65ac00c430 tx=0x7f65ac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:08.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.268+0000 7f65a37fe640 1 -- 172.21.15.161:0/3390517922 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ac0074e0 con 0x7f65bc06ca20 2024-03-28T23:37:08.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.268+0000 7f65a37fe640 1 -- 172.21.15.161:0/3390517922 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65ac0047e0 con 0x7f65bc06ca20 2024-03-28T23:37:08.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.268+0000 7f65c14eb640 1 -- 172.21.15.161:0/3390517922 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65bc152dd0 con 0x7f65bc06ca20 2024-03-28T23:37:08.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.268+0000 7f65a37fe640 1 -- 172.21.15.161:0/3390517922 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ac004bd0 con 0x7f65bc06ca20 2024-03-28T23:37:08.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.268+0000 7f65c14eb640 1 -- 172.21.15.161:0/3390517922 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65bc154810 con 0x7f65bc06ca20 2024-03-28T23:37:08.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.269+0000 7f65c14eb640 1 -- 172.21.15.161:0/3390517922 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65bc04f4b0 con 0x7f65bc06ca20 2024-03-28T23:37:08.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.270+0000 7f65a37fe640 1 -- 172.21.15.161:0/3390517922 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f65ac011070 con 0x7f65bc06ca20 2024-03-28T23:37:08.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.270+0000 7f65a37fe640 1 --2- 172.21.15.161:0/3390517922 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f65a806b4d0 0x7f65a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:08.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.270+0000 7f65a37fe640 1 -- 172.21.15.161:0/3390517922 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 6651+0+0 (secure 0 0 0) 0x7f65ac08a1e0 con 0x7f65bc06ca20 2024-03-28T23:37:08.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.270+0000 7f65ba7fc640 1 --2- 172.21.15.161:0/3390517922 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f65a806b4d0 0x7f65a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:08.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.271+0000 7f65ba7fc640 1 --2- 172.21.15.161:0/3390517922 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f65a806b4d0 0x7f65a806d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f65bc14fd90 tx=0x7f65b0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:08.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.277+0000 7f65a37fe640 1 -- 172.21.15.161:0/3390517922 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f65ac058d40 con 0x7f65bc06ca20 2024-03-28T23:37:08.441 INFO:tasks.ceph.mds.a.smithi161.stderr:2024-03-28T23:37:08.441+0000 7f76d68c5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:37:08.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.485+0000 7f65c14eb640 1 -- 172.21.15.161:0/3390517922 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65bc0693e0 con 0x7f65bc06ca20 2024-03-28T23:37:08.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.485+0000 7f65a37fe640 1 -- 172.21.15.161:0/3390517922 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) ==== 95+0+5201 (secure 0 0 0) 0x7f65ac058640 con 0x7f65bc06ca20 2024-03-28T23:37:08.486 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:37:08.486 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":160,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/557522063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":557522063},{"type":"v1","addr":"172.21.15.170:6835","nonce":557522063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":7499,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/3015739917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3015739917},{"type":"v1","addr":"172.21.15.170:6834","nonce":3015739917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":156},{"gid":9522,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/1423643374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1423643374},{"type":"v1","addr":"172.21.15.161:6836","nonce":1423643374}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":158}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T23:36:07.448976+0000","modified":"2024-03-28T23:37:06.880574+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":163,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9216},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9216":{"gid":9216,"name":"a","rank":0,"incarnation":157,"state":"up:active","state_seq":15,"addr":"172.21.15.161:6837/789976100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":789976100},{"type":"v1","addr":"172.21.15.161:6837","nonce":789976100}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9216,"qdb_cluster":[9216]},"id":12}]} 2024-03-28T23:37:08.486 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 160 2024-03-28T23:37:08.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.488+0000 7f65c14eb640 1 -- 172.21.15.161:0/3390517922 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f65a806b4d0 msgr2=0x7f65a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:08.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.488+0000 7f65c14eb640 1 --2- 172.21.15.161:0/3390517922 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f65a806b4d0 0x7f65a806d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f65bc14fd90 tx=0x7f65b0008040 comp rx=0 tx=0).stop 2024-03-28T23:37:08.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.488+0000 7f65c14eb640 1 -- 172.21.15.161:0/3390517922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65bc06ca20 msgr2=0x7f65bc06b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:08.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.488+0000 7f65c14eb640 1 --2- 172.21.15.161:0/3390517922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65bc06ca20 0x7f65bc06b3c0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f65ac00c430 tx=0x7f65ac00c8f0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.488+0000 7f65c14eb640 1 -- 172.21.15.161:0/3390517922 shutdown_connections 2024-03-28T23:37:08.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.488+0000 7f65c14eb640 1 --2- 172.21.15.161:0/3390517922 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f65a806b4d0 0x7f65a806d990 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.488+0000 7f65c14eb640 1 --2- 172.21.15.161:0/3390517922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65bc14eca0 0x7f65bc154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.488+0000 7f65c14eb640 1 --2- 172.21.15.161:0/3390517922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65bc0a98c0 0x7f65bc147ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.488+0000 7f65c14eb640 1 --2- 172.21.15.161:0/3390517922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65bc06ca20 0x7f65bc06b3c0 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.488+0000 7f65c14eb640 1 -- 172.21.15.161:0/3390517922 >> 172.21.15.161:0/3390517922 conn(0x7f65bc0c6670 msgr2=0x7f65bc0aa8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:08.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.488+0000 7f65c14eb640 1 -- 172.21.15.161:0/3390517922 shutdown_connections 2024-03-28T23:37:08.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.488+0000 7f65c14eb640 1 -- 172.21.15.161:0/3390517922 wait complete. 2024-03-28T23:37:08.499 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:37:08.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.604+0000 7f8aca1a0640 1 Processor -- start 2024-03-28T23:37:08.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.605+0000 7f8aca1a0640 1 -- start start 2024-03-28T23:37:08.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.605+0000 7f8aca1a0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8ac40c0bf0 0x7f8ac41523d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:08.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.605+0000 7f8aca1a0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ac4152910 0x7f8ac41551b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:08.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.605+0000 7f8aca1a0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ac41587d0 con 0x7f8ac406b3c0 2024-03-28T23:37:08.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.605+0000 7f8aca1a0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f8ac41593e0 con 0x7f8ac40c0bf0 2024-03-28T23:37:08.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.605+0000 7f8aca1a0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8ac4158ab0 con 0x7f8ac4152910 2024-03-28T23:37:08.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.606+0000 7f8ac999f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ac4152910 0x7f8ac41551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:08.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.606+0000 7f8ac899d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8ac40c0bf0 0x7f8ac41523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:08.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.606+0000 7f8ac919e640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f8ac406b3c0 0x7f8ac40c20e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:42358/0 (socket says 172.21.15.161:42358) 2024-03-28T23:37:08.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.606+0000 7f8ac999f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ac4152910 0x7f8ac41551b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:39678/0 (socket says 172.21.15.161:39678) 2024-03-28T23:37:08.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.606+0000 7f8ac919e640 1 -- 172.21.15.161:0/4223092064 learned_addr learned my addr 172.21.15.161:0/4223092064 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:08.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.606+0000 7f8ac999f640 1 -- 172.21.15.161:0/4223092064 >> v2:172.21.15.170:3300/0 conn(0x7f8ac40c0bf0 msgr2=0x7f8ac41523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:08.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.606+0000 7f8ac999f640 1 --2- 172.21.15.161:0/4223092064 >> v2:172.21.15.170:3300/0 conn(0x7f8ac40c0bf0 0x7f8ac41523d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.606+0000 7f8ac999f640 1 -- 172.21.15.161:0/4223092064 >> v1:172.21.15.170:6789/0 conn(0x7f8ac406b3c0 legacy=0x7f8ac40c20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:08.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.606+0000 7f8ac999f640 1 -- 172.21.15.161:0/4223092064 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ac4158d90 con 0x7f8ac4152910 2024-03-28T23:37:08.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.606+0000 7f8ac899d640 1 --2- 172.21.15.161:0/4223092064 >> v2:172.21.15.170:3300/0 conn(0x7f8ac40c0bf0 0x7f8ac41523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:37:08.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.607+0000 7f8ac999f640 1 --2- 172.21.15.161:0/4223092064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ac4152910 0x7f8ac41551b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8abc009d90 tx=0x7f8abc02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7437da84acf8b741 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:08.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.607+0000 7f8abbfff640 1 -- 172.21.15.161:0/4223092064 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8abc039070 con 0x7f8ac4152910 2024-03-28T23:37:08.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.607+0000 7f8abbfff640 1 -- 172.21.15.161:0/4223092064 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8abc02cbc0 con 0x7f8ac4152910 2024-03-28T23:37:08.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.607+0000 7f8abbfff640 1 -- 172.21.15.161:0/4223092064 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8abc034440 con 0x7f8ac4152910 2024-03-28T23:37:08.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.608+0000 7f8aca1a0640 1 -- 172.21.15.161:0/4223092064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ac4152910 msgr2=0x7f8ac41551b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:08.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.608+0000 7f8aca1a0640 1 --2- 172.21.15.161:0/4223092064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ac4152910 0x7f8ac41551b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8abc009d90 tx=0x7f8abc02be10 comp rx=0 tx=0).stop 2024-03-28T23:37:08.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.608+0000 7f8aca1a0640 1 -- 172.21.15.161:0/4223092064 shutdown_connections 2024-03-28T23:37:08.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.608+0000 7f8aca1a0640 1 --2- 172.21.15.161:0/4223092064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ac4152910 0x7f8ac41551b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.608+0000 7f8aca1a0640 1 --2- 172.21.15.161:0/4223092064 >> v2:172.21.15.170:3300/0 conn(0x7f8ac40c0bf0 0x7f8ac41523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.608+0000 7f8aca1a0640 1 -- 172.21.15.161:0/4223092064 >> 172.21.15.161:0/4223092064 conn(0x7f8ac40c41e0 msgr2=0x7f8ac40c45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:08.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.608+0000 7f8aca1a0640 1 -- 172.21.15.161:0/4223092064 shutdown_connections 2024-03-28T23:37:08.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.608+0000 7f8aca1a0640 1 -- 172.21.15.161:0/4223092064 wait complete. 2024-03-28T23:37:08.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.609+0000 7f8aca1a0640 1 Processor -- start 2024-03-28T23:37:08.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.609+0000 7f8aca1a0640 1 -- start start 2024-03-28T23:37:08.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.610+0000 7f8aca1a0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ac406b3c0 0x7f8ac4151770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:08.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.610+0000 7f8aca1a0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ac40c0bf0 0x7f8ac4149e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:08.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.610+0000 7f8aca1a0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ac4152910 0x7f8ac4152eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:08.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.610+0000 7f8aca1a0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac406b920 con 0x7f8ac40c0bf0 2024-03-28T23:37:08.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.610+0000 7f8aca1a0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac406b7a0 con 0x7f8ac406b3c0 2024-03-28T23:37:08.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.610+0000 7f8aca1a0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f8ac41556f0 con 0x7f8ac4152910 2024-03-28T23:37:08.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.610+0000 7f8ac899d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ac40c0bf0 0x7f8ac4149e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:08.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.610+0000 7f8ac899d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ac40c0bf0 0x7f8ac4149e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37806/0 (socket says 172.21.15.161:37806) 2024-03-28T23:37:08.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.610+0000 7f8ac899d640 1 -- 172.21.15.161:0/1263413449 learned_addr learned my addr 172.21.15.161:0/1263413449 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:08.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.610+0000 7f8ac919e640 1 --2- 172.21.15.161:0/1263413449 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ac406b3c0 0x7f8ac4151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:08.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.610+0000 7f8ac999f640 1 --2- 172.21.15.161:0/1263413449 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ac4152910 0x7f8ac4152eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:08.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.610+0000 7f8ac899d640 1 -- 172.21.15.161:0/1263413449 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ac4152910 msgr2=0x7f8ac4152eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:08.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.610+0000 7f8ac899d640 1 --2- 172.21.15.161:0/1263413449 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ac4152910 0x7f8ac4152eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.610+0000 7f8ac899d640 1 -- 172.21.15.161:0/1263413449 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ac406b3c0 msgr2=0x7f8ac4151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:08.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.610+0000 7f8ac899d640 1 --2- 172.21.15.161:0/1263413449 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ac406b3c0 0x7f8ac4151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.610+0000 7f8ac899d640 1 -- 172.21.15.161:0/1263413449 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ac4151d50 con 0x7f8ac40c0bf0 2024-03-28T23:37:08.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.610+0000 7f8ac899d640 1 --2- 172.21.15.161:0/1263413449 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ac40c0bf0 0x7f8ac4149e20 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f8ab400ebe0 tx=0x7f8ab400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:08.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.611+0000 7f8ab9ffb640 1 -- 172.21.15.161:0/1263413449 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4017070 con 0x7f8ac40c0bf0 2024-03-28T23:37:08.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.611+0000 7f8ab9ffb640 1 -- 172.21.15.161:0/1263413449 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ab400cc30 con 0x7f8ac40c0bf0 2024-03-28T23:37:08.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.611+0000 7f8aca1a0640 1 -- 172.21.15.161:0/1263413449 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ac41edbd0 con 0x7f8ac40c0bf0 2024-03-28T23:37:08.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.611+0000 7f8ab9ffb640 1 -- 172.21.15.161:0/1263413449 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4013470 con 0x7f8ac40c0bf0 2024-03-28T23:37:08.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.611+0000 7f8aca1a0640 1 -- 172.21.15.161:0/1263413449 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ac41ee190 con 0x7f8ac40c0bf0 2024-03-28T23:37:08.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.612+0000 7f8aca1a0640 1 -- 172.21.15.161:0/1263413449 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ac404f4b0 con 0x7f8ac40c0bf0 2024-03-28T23:37:08.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.612+0000 7f8ab9ffb640 1 -- 172.21.15.161:0/1263413449 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8ab401a070 con 0x7f8ac40c0bf0 2024-03-28T23:37:08.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.613+0000 7f8ab9ffb640 1 --2- 172.21.15.161:0/1263413449 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8aa806b4d0 0x7f8aa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:08.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.613+0000 7f8ab9ffb640 1 -- 172.21.15.161:0/1263413449 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 6651+0+0 (secure 0 0 0) 0x7f8ab408a790 con 0x7f8ac40c0bf0 2024-03-28T23:37:08.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.613+0000 7f8ac919e640 1 --2- 172.21.15.161:0/1263413449 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8aa806b4d0 0x7f8aa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:08.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.613+0000 7f8ac919e640 1 --2- 172.21.15.161:0/1263413449 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8aa806b4d0 0x7f8aa806d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0007fd0 tx=0x7f8ac0007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:08.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.617+0000 7f8ab9ffb640 1 -- 172.21.15.161:0/1263413449 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8ab4058ca0 con 0x7f8ac40c0bf0 2024-03-28T23:37:08.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.791+0000 7f8aca1a0640 1 -- 172.21.15.161:0/1263413449 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8ac414a360 con 0x7f8ac40c0bf0 2024-03-28T23:37:08.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.792+0000 7f8ab9ffb640 1 -- 172.21.15.161:0/1263413449 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v165) ==== 74+0+17836 (secure 0 0 0) 0x7f8ab405cd40 con 0x7f8ac40c0bf0 2024-03-28T23:37:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:37:08.793 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":165,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:37:05.877513+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-28T23:36:05.484451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-28T23:36:06.423073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.c","mappings":[{"from":1,"to":0}]},{"pgid":"26.13","mappings":[{"from":1,"to":6}]},{"pgid":"26.28","mappings":[{"from":1,"to":0}]},{"pgid":"26.3b","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6834/4026332703":"2024-03-29T23:37:03.868933+0000","172.21.15.170:6834/2996092857":"2024-03-29T23:37:02.855183+0000","172.21.15.161:6836/4026332703":"2024-03-29T23:37:03.868933+0000","172.21.15.170:6832/2996092857":"2024-03-29T23:37:02.855183+0000","172.21.15.161:6837/3236742821":"2024-03-29T23:36:17.431214+0000","172.21.15.161:6835/3236742821":"2024-03-29T23:36:17.431214+0000","172.21.15.170:6834/1393137291":"2024-03-29T23:36:14.412210+0000","172.21.15.170:6832/1393137291":"2024-03-29T23:36:14.412210+0000","172.21.15.161:6837/2391019771":"2024-03-29T23:36:13.968386+0000","172.21.15.161:6835/2391019771":"2024-03-29T23:36:13.968386+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:37:08.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.794+0000 7f8aca1a0640 1 -- 172.21.15.161:0/1263413449 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8aa806b4d0 msgr2=0x7f8aa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:08.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.794+0000 7f8aca1a0640 1 --2- 172.21.15.161:0/1263413449 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8aa806b4d0 0x7f8aa806d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0007fd0 tx=0x7f8ac0007e80 comp rx=0 tx=0).stop 2024-03-28T23:37:08.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.794+0000 7f8aca1a0640 1 -- 172.21.15.161:0/1263413449 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ac40c0bf0 msgr2=0x7f8ac4149e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:08.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.794+0000 7f8aca1a0640 1 --2- 172.21.15.161:0/1263413449 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ac40c0bf0 0x7f8ac4149e20 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f8ab400ebe0 tx=0x7f8ab400c520 comp rx=0 tx=0).stop 2024-03-28T23:37:08.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.795+0000 7f8aca1a0640 1 -- 172.21.15.161:0/1263413449 shutdown_connections 2024-03-28T23:37:08.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.795+0000 7f8aca1a0640 1 --2- 172.21.15.161:0/1263413449 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f8aa806b4d0 0x7f8aa806d990 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.795+0000 7f8aca1a0640 1 --2- 172.21.15.161:0/1263413449 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ac40c0bf0 0x7f8ac4149e20 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.795+0000 7f8aca1a0640 1 --2- 172.21.15.161:0/1263413449 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ac4152910 0x7f8ac4152eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.795+0000 7f8aca1a0640 1 --2- 172.21.15.161:0/1263413449 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ac406b3c0 0x7f8ac4151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.795+0000 7f8aca1a0640 1 -- 172.21.15.161:0/1263413449 >> 172.21.15.161:0/1263413449 conn(0x7f8ac40c41e0 msgr2=0x7f8ac4072a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:08.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.795+0000 7f8aca1a0640 1 -- 172.21.15.161:0/1263413449 shutdown_connections 2024-03-28T23:37:08.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.795+0000 7f8aca1a0640 1 -- 172.21.15.161:0/1263413449 wait complete. 2024-03-28T23:37:08.806 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs2 2024-03-28T23:37:08.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.909+0000 7f3c1e5c9640 1 Processor -- start 2024-03-28T23:37:08.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.909+0000 7f3c1e5c9640 1 -- start start 2024-03-28T23:37:08.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.910+0000 7f3c1e5c9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3c18072810 0x7f3c180c4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:08.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.910+0000 7f3c1e5c9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3c18070370 0x7f3c180c3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:08.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.910+0000 7f3c1e5c9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c180707e0 con 0x7f3c180c4d80 2024-03-28T23:37:08.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.910+0000 7f3c1e5c9640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f3c180a8520 con 0x7f3c18070370 2024-03-28T23:37:08.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.910+0000 7f3c1e5c9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f3c18153050 con 0x7f3c18072810 2024-03-28T23:37:08.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.910+0000 7f3c1cb66640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3c18070370 0x7f3c180c3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:08.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.910+0000 7f3c1cb66640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3c18070370 0x7f3c180c3390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37820/0 (socket says 172.21.15.161:37820) 2024-03-28T23:37:08.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.910+0000 7f3c1cb66640 1 -- 172.21.15.161:0/3671080348 learned_addr learned my addr 172.21.15.161:0/3671080348 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:08.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.910+0000 7f3c17fff640 1 --2- 172.21.15.161:0/3671080348 >> v2:172.21.15.170:3300/0 conn(0x7f3c18072810 0x7f3c180c4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:08.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.910+0000 7f3c1cb66640 1 -- 172.21.15.161:0/3671080348 >> v2:172.21.15.170:3300/0 conn(0x7f3c18072810 msgr2=0x7f3c180c4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:08.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.910+0000 7f3c1cb66640 1 --2- 172.21.15.161:0/3671080348 >> v2:172.21.15.170:3300/0 conn(0x7f3c18072810 0x7f3c180c4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.910+0000 7f3c1cb66640 1 -- 172.21.15.161:0/3671080348 >> v1:172.21.15.170:6789/0 conn(0x7f3c180c4d80 legacy=0x7f3c1806fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:08.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.910+0000 7f3c1cb66640 1 -- 172.21.15.161:0/3671080348 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c18153310 con 0x7f3c18070370 2024-03-28T23:37:08.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.910+0000 7f3c17fff640 1 --2- 172.21.15.161:0/3671080348 >> v2:172.21.15.170:3300/0 conn(0x7f3c18072810 0x7f3c180c4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:37:08.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.911+0000 7f3c1cb66640 1 --2- 172.21.15.161:0/3671080348 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c18070370 0x7f3c180c3390 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f3c180711a0 tx=0x7f3c1002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb29a9484bae8a2a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:08.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.911+0000 7f3c16ffd640 1 -- 172.21.15.161:0/3671080348 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1002c9f0 con 0x7f3c18070370 2024-03-28T23:37:08.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.911+0000 7f3c16ffd640 1 -- 172.21.15.161:0/3671080348 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c1002cb90 con 0x7f3c18070370 2024-03-28T23:37:08.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.911+0000 7f3c16ffd640 1 -- 172.21.15.161:0/3671080348 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c100343f0 con 0x7f3c18070370 2024-03-28T23:37:08.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.912+0000 7f3c1e5c9640 1 -- 172.21.15.161:0/3671080348 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c18070370 msgr2=0x7f3c180c3390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:08.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.912+0000 7f3c1e5c9640 1 --2- 172.21.15.161:0/3671080348 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c18070370 0x7f3c180c3390 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f3c180711a0 tx=0x7f3c1002be50 comp rx=0 tx=0).stop 2024-03-28T23:37:08.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.912+0000 7f3c1e5c9640 1 -- 172.21.15.161:0/3671080348 shutdown_connections 2024-03-28T23:37:08.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.912+0000 7f3c1e5c9640 1 --2- 172.21.15.161:0/3671080348 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c18070370 0x7f3c180c3390 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.912+0000 7f3c1e5c9640 1 --2- 172.21.15.161:0/3671080348 >> v2:172.21.15.170:3300/0 conn(0x7f3c18072810 0x7f3c180c4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.912+0000 7f3c1e5c9640 1 -- 172.21.15.161:0/3671080348 >> 172.21.15.161:0/3671080348 conn(0x7f3c180aa740 msgr2=0x7f3c180aab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:08.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.912+0000 7f3c1e5c9640 1 -- 172.21.15.161:0/3671080348 shutdown_connections 2024-03-28T23:37:08.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.912+0000 7f3c1e5c9640 1 -- 172.21.15.161:0/3671080348 wait complete. 2024-03-28T23:37:08.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.913+0000 7f3c1e5c9640 1 Processor -- start 2024-03-28T23:37:08.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.913+0000 7f3c1e5c9640 1 -- start start 2024-03-28T23:37:08.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.914+0000 7f3c1e5c9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c18070370 0x7f3c181e8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:08.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.914+0000 7f3c1e5c9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c18072810 0x7f3c1814b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:08.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.914+0000 7f3c1e5c9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c180c4d80 0x7f3c18155730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:08.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.914+0000 7f3c1e5c9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3c18153bb0 con 0x7f3c18070370 2024-03-28T23:37:08.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.914+0000 7f3c1e5c9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3c181538d0 con 0x7f3c180c4d80 2024-03-28T23:37:08.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.914+0000 7f3c1e5c9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3c18072bf0 con 0x7f3c18072810 2024-03-28T23:37:08.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.914+0000 7f3c17fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c18070370 0x7f3c181e8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:08.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.914+0000 7f3c17fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c18070370 0x7f3c181e8b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37828/0 (socket says 172.21.15.161:37828) 2024-03-28T23:37:08.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.914+0000 7f3c17fff640 1 -- 172.21.15.161:0/2286446358 learned_addr learned my addr 172.21.15.161:0/2286446358 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:08.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.914+0000 7f3c1cb66640 1 --2- 172.21.15.161:0/2286446358 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c180c4d80 0x7f3c18155730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:08.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.914+0000 7f3c177fe640 1 --2- 172.21.15.161:0/2286446358 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c18072810 0x7f3c1814b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:08.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.914+0000 7f3c17fff640 1 -- 172.21.15.161:0/2286446358 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c18072810 msgr2=0x7f3c1814b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:08.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.914+0000 7f3c17fff640 1 --2- 172.21.15.161:0/2286446358 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c18072810 0x7f3c1814b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.914+0000 7f3c17fff640 1 -- 172.21.15.161:0/2286446358 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c180c4d80 msgr2=0x7f3c18155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:08.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.914+0000 7f3c17fff640 1 --2- 172.21.15.161:0/2286446358 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c180c4d80 0x7f3c18155730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:08.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.914+0000 7f3c17fff640 1 -- 172.21.15.161:0/2286446358 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c181e90a0 con 0x7f3c18070370 2024-03-28T23:37:08.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.914+0000 7f3c17fff640 1 --2- 172.21.15.161:0/2286446358 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c18070370 0x7f3c181e8b60 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f3c0000c7f0 tx=0x7f3c0000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:08.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.915+0000 7f3c14ff9640 1 -- 172.21.15.161:0/2286446358 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c00015070 con 0x7f3c18070370 2024-03-28T23:37:08.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.915+0000 7f3c14ff9640 1 -- 172.21.15.161:0/2286446358 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c00007a30 con 0x7f3c18070370 2024-03-28T23:37:08.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.915+0000 7f3c1e5c9640 1 -- 172.21.15.161:0/2286446358 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c180abf00 con 0x7f3c18070370 2024-03-28T23:37:08.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.915+0000 7f3c14ff9640 1 -- 172.21.15.161:0/2286446358 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c00007e00 con 0x7f3c18070370 2024-03-28T23:37:08.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.915+0000 7f3c1e5c9640 1 -- 172.21.15.161:0/2286446358 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c181ebb50 con 0x7f3c18070370 2024-03-28T23:37:08.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.915+0000 7f3c1e5c9640 1 -- 172.21.15.161:0/2286446358 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c1804f4b0 con 0x7f3c18070370 2024-03-28T23:37:08.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.916+0000 7f3c14ff9640 1 -- 172.21.15.161:0/2286446358 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3c00018070 con 0x7f3c18070370 2024-03-28T23:37:08.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.917+0000 7f3c14ff9640 1 --2- 172.21.15.161:0/2286446358 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3bf806b4d0 0x7f3bf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:08.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.917+0000 7f3c14ff9640 1 -- 172.21.15.161:0/2286446358 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 6651+0+0 (secure 0 0 0) 0x7f3c0008a520 con 0x7f3c18070370 2024-03-28T23:37:08.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.917+0000 7f3c177fe640 1 --2- 172.21.15.161:0/2286446358 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3bf806b4d0 0x7f3bf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:08.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.917+0000 7f3c177fe640 1 --2- 172.21.15.161:0/2286446358 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3bf806b4d0 0x7f3bf806d990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f3c08006fd0 tx=0x7f3c08008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:08.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:08.921+0000 7f3c14ff9640 1 -- 172.21.15.161:0/2286446358 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3c181e90a0 con 0x7f3c18070370 2024-03-28T23:37:09.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.138+0000 7f3c1e5c9640 1 -- 172.21.15.161:0/2286446358 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs2"} v 0) -- 0x7f3c1814b8b0 con 0x7f3c18070370 2024-03-28T23:37:09.139 INFO:tasks.ceph.mon.a.smithi161.stderr:2024-03-28T23:37:09.139+0000 7fdc7b85b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T23:37:09.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.140+0000 7f3c14ff9640 1 -- 172.21.15.161:0/2286446358 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs2"}]=0 cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v161) ==== 163+0+0 (secure 0 0 0) 0x7f3c00058ff0 con 0x7f3c18070370 2024-03-28T23:37:09.141 INFO:teuthology.orchestra.run.smithi161.stderr:cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-28T23:37:09.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.143+0000 7f3c1e5c9640 1 -- 172.21.15.161:0/2286446358 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3bf806b4d0 msgr2=0x7f3bf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:09.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.143+0000 7f3c1e5c9640 1 --2- 172.21.15.161:0/2286446358 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3bf806b4d0 0x7f3bf806d990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f3c08006fd0 tx=0x7f3c08008040 comp rx=0 tx=0).stop 2024-03-28T23:37:09.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.143+0000 7f3c1e5c9640 1 -- 172.21.15.161:0/2286446358 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c18070370 msgr2=0x7f3c181e8b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:09.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.143+0000 7f3c1e5c9640 1 --2- 172.21.15.161:0/2286446358 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c18070370 0x7f3c181e8b60 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f3c0000c7f0 tx=0x7f3c0000cbf0 comp rx=0 tx=0).stop 2024-03-28T23:37:09.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.143+0000 7f3c1e5c9640 1 -- 172.21.15.161:0/2286446358 shutdown_connections 2024-03-28T23:37:09.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.143+0000 7f3c1e5c9640 1 --2- 172.21.15.161:0/2286446358 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3bf806b4d0 0x7f3bf806d990 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:09.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.143+0000 7f3c1e5c9640 1 --2- 172.21.15.161:0/2286446358 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c180c4d80 0x7f3c18155730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:09.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.143+0000 7f3c1e5c9640 1 --2- 172.21.15.161:0/2286446358 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c18072810 0x7f3c1814b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:09.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.143+0000 7f3c1e5c9640 1 --2- 172.21.15.161:0/2286446358 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c18070370 0x7f3c181e8b60 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:09.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.143+0000 7f3c1e5c9640 1 -- 172.21.15.161:0/2286446358 >> 172.21.15.161:0/2286446358 conn(0x7f3c180aa740 msgr2=0x7f3c180c8080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:09.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.144+0000 7f3c1e5c9640 1 -- 172.21.15.161:0/2286446358 shutdown_connections 2024-03-28T23:37:09.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.144+0000 7f3c1e5c9640 1 -- 172.21.15.161:0/2286446358 wait complete. 2024-03-28T23:37:09.146 INFO:tasks.ceph.mds.a.smithi161.stderr: -37> 2024-03-28T23:37:08.441+0000 7f76d68c5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T23:37:09.155 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs2 --yes-i-really-mean-it 2024-03-28T23:37:09.168 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-28T23:37:09.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.258+0000 7fc949948640 1 Processor -- start 2024-03-28T23:37:09.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.258+0000 7fc949948640 1 -- start start 2024-03-28T23:37:09.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.259+0000 7fc949948640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc9440c4e40 0x7fc9440c8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:09.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.259+0000 7fc949948640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc944158860 0x7fc944158d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:09.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.259+0000 7fc949948640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc944156060 con 0x7fc944072fe0 2024-03-28T23:37:09.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.259+0000 7fc949948640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc944072870 con 0x7fc9440c4e40 2024-03-28T23:37:09.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.259+0000 7fc949948640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc944072b50 con 0x7fc944158860 2024-03-28T23:37:09.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.259+0000 7fc942ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc9440c4e40 0x7fc9440c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:09.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.259+0000 7fc9427fc640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc944072fe0 0x7fc9440c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48214/0 (socket says 172.21.15.161:48214) 2024-03-28T23:37:09.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.259+0000 7fc942ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc9440c4e40 0x7fc9440c8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37866/0 (socket says 172.21.15.161:37866) 2024-03-28T23:37:09.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.259+0000 7fc9427fc640 1 -- 172.21.15.161:0/3696535185 learned_addr learned my addr 172.21.15.161:0/3696535185 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:09.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.259+0000 7fc9437fe640 1 --2- 172.21.15.161:0/3696535185 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc944158860 0x7fc944158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:09.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.259+0000 7fc942ffd640 1 -- 172.21.15.161:0/3696535185 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc944158860 msgr2=0x7fc944158d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:09.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.259+0000 7fc942ffd640 1 --2- 172.21.15.161:0/3696535185 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc944158860 0x7fc944158d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:09.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.259+0000 7fc942ffd640 1 -- 172.21.15.161:0/3696535185 >> v1:172.21.15.161:6789/0 conn(0x7fc944072fe0 legacy=0x7fc9440c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:09.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.259+0000 7fc942ffd640 1 -- 172.21.15.161:0/3696535185 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc944159250 con 0x7fc9440c4e40 2024-03-28T23:37:09.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.259+0000 7fc9437fe640 1 --2- 172.21.15.161:0/3696535185 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc944158860 0x7fc944158d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:37:09.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.260+0000 7fc942ffd640 1 --2- 172.21.15.161:0/3696535185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9440c4e40 0x7fc9440c8340 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fc9340097a0 tx=0x7fc93402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=118c090ebb1d9efd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:09.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.260+0000 7fc941ffb640 1 -- 172.21.15.161:0/3696535185 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc934035020 con 0x7fc9440c4e40 2024-03-28T23:37:09.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.260+0000 7fc941ffb640 1 -- 172.21.15.161:0/3696535185 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc934031830 con 0x7fc9440c4e40 2024-03-28T23:37:09.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.260+0000 7fc941ffb640 1 -- 172.21.15.161:0/3696535185 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc934031c00 con 0x7fc9440c4e40 2024-03-28T23:37:09.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.261+0000 7fc949948640 1 -- 172.21.15.161:0/3696535185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9440c4e40 msgr2=0x7fc9440c8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:09.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.261+0000 7fc949948640 1 --2- 172.21.15.161:0/3696535185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9440c4e40 0x7fc9440c8340 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fc9340097a0 tx=0x7fc93402b8d0 comp rx=0 tx=0).stop 2024-03-28T23:37:09.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.261+0000 7fc949948640 1 -- 172.21.15.161:0/3696535185 shutdown_connections 2024-03-28T23:37:09.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.261+0000 7fc949948640 1 --2- 172.21.15.161:0/3696535185 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc944158860 0x7fc944158d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:09.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.261+0000 7fc949948640 1 --2- 172.21.15.161:0/3696535185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9440c4e40 0x7fc9440c8340 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:09.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.261+0000 7fc949948640 1 -- 172.21.15.161:0/3696535185 >> 172.21.15.161:0/3696535185 conn(0x7fc9440c88b0 msgr2=0x7fc9440c8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:09.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.261+0000 7fc949948640 1 -- 172.21.15.161:0/3696535185 shutdown_connections 2024-03-28T23:37:09.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.261+0000 7fc949948640 1 -- 172.21.15.161:0/3696535185 wait complete. 2024-03-28T23:37:09.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.262+0000 7fc949948640 1 Processor -- start 2024-03-28T23:37:09.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.262+0000 7fc949948640 1 -- start start 2024-03-28T23:37:09.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.263+0000 7fc949948640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc944072fe0 0x7fc94414eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:09.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.263+0000 7fc949948640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9440c4e40 0x7fc944148d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:09.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.263+0000 7fc949948640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc944158860 0x7fc944149260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:09.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.263+0000 7fc949948640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc9440c5bd0 con 0x7fc944072fe0 2024-03-28T23:37:09.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.263+0000 7fc949948640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc9440c5a50 con 0x7fc9440c4e40 2024-03-28T23:37:09.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.263+0000 7fc949948640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc9440c6d30 con 0x7fc944158860 2024-03-28T23:37:09.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.263+0000 7fc942ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc944072fe0 0x7fc94414eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:09.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.263+0000 7fc942ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc944072fe0 0x7fc94414eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37880/0 (socket says 172.21.15.161:37880) 2024-03-28T23:37:09.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.263+0000 7fc942ffd640 1 -- 172.21.15.161:0/2577808870 learned_addr learned my addr 172.21.15.161:0/2577808870 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:09.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.263+0000 7fc942ffd640 1 -- 172.21.15.161:0/2577808870 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc944158860 msgr2=0x7fc944149260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:09.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.263+0000 7fc942ffd640 1 --2- 172.21.15.161:0/2577808870 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc944158860 0x7fc944149260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:09.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.263+0000 7fc942ffd640 1 -- 172.21.15.161:0/2577808870 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9440c4e40 msgr2=0x7fc944148d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:09.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.263+0000 7fc942ffd640 1 --2- 172.21.15.161:0/2577808870 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9440c4e40 0x7fc944148d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:09.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.263+0000 7fc942ffd640 1 -- 172.21.15.161:0/2577808870 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9441532b0 con 0x7fc944072fe0 2024-03-28T23:37:09.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.263+0000 7fc942ffd640 1 --2- 172.21.15.161:0/2577808870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc944072fe0 0x7fc94414eaf0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fc934034040 tx=0x7fc9340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:09.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.263+0000 7fc92bfff640 1 -- 172.21.15.161:0/2577808870 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc934040070 con 0x7fc944072fe0 2024-03-28T23:37:09.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.263+0000 7fc92bfff640 1 -- 172.21.15.161:0/2577808870 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc934039a00 con 0x7fc944072fe0 2024-03-28T23:37:09.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.263+0000 7fc949948640 1 -- 172.21.15.161:0/2577808870 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9440a9190 con 0x7fc944072fe0 2024-03-28T23:37:09.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.263+0000 7fc92bfff640 1 -- 172.21.15.161:0/2577808870 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc934035040 con 0x7fc944072fe0 2024-03-28T23:37:09.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.263+0000 7fc949948640 1 -- 172.21.15.161:0/2577808870 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9440a9750 con 0x7fc944072fe0 2024-03-28T23:37:09.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.264+0000 7fc949948640 1 -- 172.21.15.161:0/2577808870 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc94404f4b0 con 0x7fc944072fe0 2024-03-28T23:37:09.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.265+0000 7fc92bfff640 1 -- 172.21.15.161:0/2577808870 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc934038020 con 0x7fc944072fe0 2024-03-28T23:37:09.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.265+0000 7fc92bfff640 1 --2- 172.21.15.161:0/2577808870 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc93006b500 0x7fc93006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:09.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.265+0000 7fc9427fc640 1 --2- 172.21.15.161:0/2577808870 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc93006b500 0x7fc93006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:09.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.265+0000 7fc92bfff640 1 -- 172.21.15.161:0/2577808870 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(166..166 src has 1..166) ==== 6737+0+0 (secure 0 0 0) 0x7fc9340ac0d0 con 0x7fc944072fe0 2024-03-28T23:37:09.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.266+0000 7fc9427fc640 1 --2- 172.21.15.161:0/2577808870 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc93006b500 0x7fc93006d9c0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fc938000f60 tx=0x7fc938007a80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:09.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.270+0000 7fc92bfff640 1 -- 172.21.15.161:0/2577808870 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc9441532b0 con 0x7fc944072fe0 2024-03-28T23:37:09.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:09.481+0000 7fc949948640 1 -- 172.21.15.161:0/2577808870 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true} v 0) -- 0x7fc9440c8320 con 0x7fc944072fe0 2024-03-28T23:37:10.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.144+0000 7fc92bfff640 1 -- 172.21.15.161:0/2577808870 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true}]=0 v162) ==== 105+0+0 (secure 0 0 0) 0x7fc93407ab40 con 0x7fc944072fe0 2024-03-28T23:37:10.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.147+0000 7fc949948640 1 -- 172.21.15.161:0/2577808870 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc93006b500 msgr2=0x7fc93006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:10.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.147+0000 7fc949948640 1 --2- 172.21.15.161:0/2577808870 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc93006b500 0x7fc93006d9c0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fc938000f60 tx=0x7fc938007a80 comp rx=0 tx=0).stop 2024-03-28T23:37:10.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.147+0000 7fc949948640 1 -- 172.21.15.161:0/2577808870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc944072fe0 msgr2=0x7fc94414eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:10.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.147+0000 7fc949948640 1 --2- 172.21.15.161:0/2577808870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc944072fe0 0x7fc94414eaf0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fc934034040 tx=0x7fc9340046c0 comp rx=0 tx=0).stop 2024-03-28T23:37:10.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.147+0000 7fc949948640 1 -- 172.21.15.161:0/2577808870 shutdown_connections 2024-03-28T23:37:10.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.147+0000 7fc949948640 1 --2- 172.21.15.161:0/2577808870 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc93006b500 0x7fc93006d9c0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:10.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.147+0000 7fc949948640 1 --2- 172.21.15.161:0/2577808870 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc944158860 0x7fc944149260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:10.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.147+0000 7fc949948640 1 --2- 172.21.15.161:0/2577808870 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9440c4e40 0x7fc944148d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:10.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.147+0000 7fc949948640 1 --2- 172.21.15.161:0/2577808870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc944072fe0 0x7fc94414eaf0 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:10.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.147+0000 7fc949948640 1 -- 172.21.15.161:0/2577808870 >> 172.21.15.161:0/2577808870 conn(0x7fc9440c88b0 msgr2=0x7fc9440c5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:10.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.147+0000 7fc949948640 1 -- 172.21.15.161:0/2577808870 shutdown_connections 2024-03-28T23:37:10.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.147+0000 7fc949948640 1 -- 172.21.15.161:0/2577808870 wait complete. 2024-03-28T23:37:10.160 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_metadata cephfs2_metadata --yes-i-really-really-mean-it 2024-03-28T23:37:10.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.306+0000 7f1259e2a640 1 Processor -- start 2024-03-28T23:37:10.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.306+0000 7f1259e2a640 1 -- start start 2024-03-28T23:37:10.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.307+0000 7f1259e2a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f12540c1ad0 0x7f12540c4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:10.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.307+0000 7f1259e2a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12540c4e40 0x7f1254163ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:10.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.307+0000 7f1259e2a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1254159870 con 0x7f12540c3b90 2024-03-28T23:37:10.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.307+0000 7f1259e2a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f125415a1a0 con 0x7f12540c1ad0 2024-03-28T23:37:10.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.307+0000 7f1259e2a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f1254159b50 con 0x7f12540c4e40 2024-03-28T23:37:10.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.307+0000 7f1253fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12540c4e40 0x7f1254163ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:10.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.307+0000 7f1252ffd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f12540c1ad0 0x7f12540c4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:10.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.307+0000 7f1253fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12540c4e40 0x7f1254163ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:39740/0 (socket says 172.21.15.161:39740) 2024-03-28T23:37:10.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.307+0000 7f1253fff640 1 -- 172.21.15.161:0/820269057 learned_addr learned my addr 172.21.15.161:0/820269057 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:10.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.308+0000 7f1253fff640 1 -- 172.21.15.161:0/820269057 >> v2:172.21.15.170:3300/0 conn(0x7f12540c1ad0 msgr2=0x7f12540c4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:10.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.308+0000 7f1253fff640 1 --2- 172.21.15.161:0/820269057 >> v2:172.21.15.170:3300/0 conn(0x7f12540c1ad0 0x7f12540c4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:10.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.308+0000 7f1253fff640 1 -- 172.21.15.161:0/820269057 >> v1:172.21.15.170:6789/0 conn(0x7f12540c3b90 legacy=0x7f12541566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:10.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.308+0000 7f1253fff640 1 -- 172.21.15.161:0/820269057 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1254159e30 con 0x7f12540c4e40 2024-03-28T23:37:10.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.308+0000 7f1252ffd640 1 --2- 172.21.15.161:0/820269057 >> v2:172.21.15.170:3300/0 conn(0x7f12540c1ad0 0x7f12540c4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:37:10.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.308+0000 7f1253fff640 1 --2- 172.21.15.161:0/820269057 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12540c4e40 0x7f1254163ea0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f124c009e50 tx=0x7f124c02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72077c034aa160b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:10.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.309+0000 7f12527fc640 1 -- 172.21.15.161:0/820269057 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124c02ca70 con 0x7f12540c4e40 2024-03-28T23:37:10.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.309+0000 7f12527fc640 1 -- 172.21.15.161:0/820269057 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f124c02cc10 con 0x7f12540c4e40 2024-03-28T23:37:10.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.309+0000 7f12527fc640 1 -- 172.21.15.161:0/820269057 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124c0343f0 con 0x7f12540c4e40 2024-03-28T23:37:10.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.309+0000 7f1259e2a640 1 -- 172.21.15.161:0/820269057 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12540c4e40 msgr2=0x7f1254163ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:10.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.309+0000 7f1259e2a640 1 --2- 172.21.15.161:0/820269057 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12540c4e40 0x7f1254163ea0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f124c009e50 tx=0x7f124c02bed0 comp rx=0 tx=0).stop 2024-03-28T23:37:10.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.309+0000 7f1259e2a640 1 -- 172.21.15.161:0/820269057 shutdown_connections 2024-03-28T23:37:10.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.309+0000 7f1259e2a640 1 --2- 172.21.15.161:0/820269057 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12540c4e40 0x7f1254163ea0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:10.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.309+0000 7f1259e2a640 1 --2- 172.21.15.161:0/820269057 >> v2:172.21.15.170:3300/0 conn(0x7f12540c1ad0 0x7f12540c4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:10.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.309+0000 7f1259e2a640 1 -- 172.21.15.161:0/820269057 >> 172.21.15.161:0/820269057 conn(0x7f12540c6e20 msgr2=0x7f12540c7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:10.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.309+0000 7f1259e2a640 1 -- 172.21.15.161:0/820269057 shutdown_connections 2024-03-28T23:37:10.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.309+0000 7f1259e2a640 1 -- 172.21.15.161:0/820269057 wait complete. 2024-03-28T23:37:10.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.310+0000 7f1259e2a640 1 Processor -- start 2024-03-28T23:37:10.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.310+0000 7f1259e2a640 1 -- start start 2024-03-28T23:37:10.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.311+0000 7f1259e2a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12540c1ad0 0x7f12541f29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:10.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.311+0000 7f1259e2a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12540c3b90 0x7f12541f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:10.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.311+0000 7f1259e2a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12540c4e40 0x7f125415afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:10.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.311+0000 7f1259e2a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f12540c6830 con 0x7f12540c4e40 2024-03-28T23:37:10.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.311+0000 7f1259e2a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f12540c66b0 con 0x7f12540c3b90 2024-03-28T23:37:10.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.311+0000 7f1259e2a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f12540c1540 con 0x7f12540c1ad0 2024-03-28T23:37:10.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.311+0000 7f1253fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12540c4e40 0x7f125415afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:10.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.311+0000 7f1253fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12540c4e40 0x7f125415afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37884/0 (socket says 172.21.15.161:37884) 2024-03-28T23:37:10.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.311+0000 7f1253fff640 1 -- 172.21.15.161:0/3269815655 learned_addr learned my addr 172.21.15.161:0/3269815655 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:10.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.311+0000 7f12537fe640 1 --2- 172.21.15.161:0/3269815655 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12540c1ad0 0x7f12541f29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:10.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.311+0000 7f1252ffd640 1 --2- 172.21.15.161:0/3269815655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12540c3b90 0x7f12541f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:10.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.311+0000 7f1253fff640 1 -- 172.21.15.161:0/3269815655 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12540c1ad0 msgr2=0x7f12541f29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:10.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.311+0000 7f1253fff640 1 --2- 172.21.15.161:0/3269815655 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12540c1ad0 0x7f12541f29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:10.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.311+0000 7f1253fff640 1 -- 172.21.15.161:0/3269815655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12540c3b90 msgr2=0x7f12541f2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:10.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.311+0000 7f1253fff640 1 --2- 172.21.15.161:0/3269815655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12540c3b90 0x7f12541f2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:10.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.312+0000 7f1253fff640 1 -- 172.21.15.161:0/3269815655 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12541f3470 con 0x7f12540c4e40 2024-03-28T23:37:10.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.312+0000 7f1252ffd640 1 --2- 172.21.15.161:0/3269815655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12540c3b90 0x7f12541f2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:37:10.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.312+0000 7f1253fff640 1 --2- 172.21.15.161:0/3269815655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12540c4e40 0x7f125415afb0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f124c036040 tx=0x7f124c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:10.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.312+0000 7f123bfff640 1 -- 172.21.15.161:0/3269815655 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124c043020 con 0x7f12540c4e40 2024-03-28T23:37:10.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.312+0000 7f123bfff640 1 -- 172.21.15.161:0/3269815655 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f124c03e450 con 0x7f12540c4e40 2024-03-28T23:37:10.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.312+0000 7f123bfff640 1 -- 172.21.15.161:0/3269815655 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124c009510 con 0x7f12540c4e40 2024-03-28T23:37:10.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.312+0000 7f1259e2a640 1 -- 172.21.15.161:0/3269815655 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f125415d100 con 0x7f12540c4e40 2024-03-28T23:37:10.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.312+0000 7f1259e2a640 1 -- 172.21.15.161:0/3269815655 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f125415d6c0 con 0x7f12540c4e40 2024-03-28T23:37:10.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.313+0000 7f1259e2a640 1 -- 172.21.15.161:0/3269815655 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f125415d900 con 0x7f12540c4e40 2024-03-28T23:37:10.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.313+0000 7f123bfff640 1 -- 172.21.15.161:0/3269815655 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f124c046020 con 0x7f12540c4e40 2024-03-28T23:37:10.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.314+0000 7f123bfff640 1 --2- 172.21.15.161:0/3269815655 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f124006b4d0 0x7f124006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:10.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.314+0000 7f123bfff640 1 -- 172.21.15.161:0/3269815655 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(166..166 src has 1..166) ==== 6737+0+0 (secure 0 0 0) 0x7f124c0acc90 con 0x7f12540c4e40 2024-03-28T23:37:10.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.314+0000 7f12537fe640 1 --2- 172.21.15.161:0/3269815655 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f124006b4d0 0x7f124006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:10.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.314+0000 7f12537fe640 1 --2- 172.21.15.161:0/3269815655 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f124006b4d0 0x7f124006d990 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f1244007fd0 tx=0x7f1244009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:10.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.318+0000 7f123bfff640 1 -- 172.21.15.161:0/3269815655 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f125415d900 con 0x7f12540c4e40 2024-03-28T23:37:10.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:10.488+0000 7f1259e2a640 1 -- 172.21.15.161:0/3269815655 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1254068f90 con 0x7f12540c4e40 2024-03-28T23:37:11.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.146+0000 7f123bfff640 1 -- 172.21.15.161:0/3269815655 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_metadata' removed v167) ==== 184+0+0 (secure 0 0 0) 0x7f124c04d030 con 0x7f12540c4e40 2024-03-28T23:37:11.147 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs2_metadata' removed 2024-03-28T23:37:11.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.149+0000 7f1259e2a640 1 -- 172.21.15.161:0/3269815655 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f124006b4d0 msgr2=0x7f124006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:11.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.150+0000 7f1259e2a640 1 --2- 172.21.15.161:0/3269815655 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f124006b4d0 0x7f124006d990 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f1244007fd0 tx=0x7f1244009040 comp rx=0 tx=0).stop 2024-03-28T23:37:11.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.150+0000 7f1259e2a640 1 -- 172.21.15.161:0/3269815655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12540c4e40 msgr2=0x7f125415afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:11.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.150+0000 7f1259e2a640 1 --2- 172.21.15.161:0/3269815655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12540c4e40 0x7f125415afb0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f124c036040 tx=0x7f124c004a40 comp rx=0 tx=0).stop 2024-03-28T23:37:11.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.150+0000 7f1259e2a640 1 -- 172.21.15.161:0/3269815655 shutdown_connections 2024-03-28T23:37:11.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.150+0000 7f1259e2a640 1 --2- 172.21.15.161:0/3269815655 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f124006b4d0 0x7f124006d990 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:11.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.150+0000 7f1259e2a640 1 --2- 172.21.15.161:0/3269815655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12540c4e40 0x7f125415afb0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:11.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.150+0000 7f1259e2a640 1 --2- 172.21.15.161:0/3269815655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12540c3b90 0x7f12541f2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:11.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.150+0000 7f1259e2a640 1 --2- 172.21.15.161:0/3269815655 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12540c1ad0 0x7f12541f29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:11.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.150+0000 7f1259e2a640 1 -- 172.21.15.161:0/3269815655 >> 172.21.15.161:0/3269815655 conn(0x7f12540c6e20 msgr2=0x7f1254158740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:11.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.150+0000 7f1259e2a640 1 -- 172.21.15.161:0/3269815655 shutdown_connections 2024-03-28T23:37:11.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.150+0000 7f1259e2a640 1 -- 172.21.15.161:0/3269815655 wait complete. 2024-03-28T23:37:11.163 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_data cephfs2_data --yes-i-really-really-mean-it 2024-03-28T23:37:11.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.272+0000 7fd668bd0640 1 Processor -- start 2024-03-28T23:37:11.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.272+0000 7fd668bd0640 1 -- start start 2024-03-28T23:37:11.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.273+0000 7fd668bd0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd6640abb70 0x7fd6640abf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:11.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.273+0000 7fd668bd0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd66406d390 0x7fd66406dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:11.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.273+0000 7fd668bd0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd664149ec0 con 0x7fd6640c46b0 2024-03-28T23:37:11.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.273+0000 7fd668bd0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd664153ab0 con 0x7fd66406d390 2024-03-28T23:37:11.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.273+0000 7fd668bd0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fd664153180 con 0x7fd6640abb70 2024-03-28T23:37:11.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.273+0000 7fd661d9b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd66406d390 0x7fd66406dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:11.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.273+0000 7fd662d9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd6640c46b0 0x7fd6640c4b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41642/0 (socket says 172.21.15.161:41642) 2024-03-28T23:37:11.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.273+0000 7fd661d9b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd66406d390 0x7fd66406dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55684/0 (socket says 172.21.15.161:55684) 2024-03-28T23:37:11.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.273+0000 7fd662d9d640 1 -- 172.21.15.161:0/3611233148 learned_addr learned my addr 172.21.15.161:0/3611233148 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:11.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.273+0000 7fd66259c640 1 --2- 172.21.15.161:0/3611233148 >> v2:172.21.15.170:3300/0 conn(0x7fd6640abb70 0x7fd6640abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:11.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.273+0000 7fd661d9b640 1 -- 172.21.15.161:0/3611233148 >> v2:172.21.15.170:3300/0 conn(0x7fd6640abb70 msgr2=0x7fd6640abf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:11.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.273+0000 7fd661d9b640 1 --2- 172.21.15.161:0/3611233148 >> v2:172.21.15.170:3300/0 conn(0x7fd6640abb70 0x7fd6640abf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:11.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.273+0000 7fd661d9b640 1 -- 172.21.15.161:0/3611233148 >> v1:172.21.15.161:6789/0 conn(0x7fd6640c46b0 legacy=0x7fd6640c4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:11.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.273+0000 7fd661d9b640 1 -- 172.21.15.161:0/3611233148 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd664153460 con 0x7fd66406d390 2024-03-28T23:37:11.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.273+0000 7fd66159a640 1 -- 172.21.15.161:0/3611233148 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3074361800 0 0) 0x7fd664149ec0 con 0x7fd6640c46b0 2024-03-28T23:37:11.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.273+0000 7fd66259c640 1 --2- 172.21.15.161:0/3611233148 >> v2:172.21.15.170:3300/0 conn(0x7fd6640abb70 0x7fd6640abf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:37:11.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.274+0000 7fd661d9b640 1 --2- 172.21.15.161:0/3611233148 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd66406d390 0x7fd66406dcf0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7fd6640c42e0 tx=0x7fd65802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=898b7c8a2349b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:11.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.274+0000 7fd66159a640 1 -- 172.21.15.161:0/3611233148 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd658035020 con 0x7fd66406d390 2024-03-28T23:37:11.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.274+0000 7fd66159a640 1 -- 172.21.15.161:0/3611233148 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd658031830 con 0x7fd66406d390 2024-03-28T23:37:11.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.274+0000 7fd66159a640 1 -- 172.21.15.161:0/3611233148 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd658031c00 con 0x7fd66406d390 2024-03-28T23:37:11.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.275+0000 7fd668bd0640 1 -- 172.21.15.161:0/3611233148 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd66406d390 msgr2=0x7fd66406dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:11.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.275+0000 7fd668bd0640 1 --2- 172.21.15.161:0/3611233148 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd66406d390 0x7fd66406dcf0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7fd6640c42e0 tx=0x7fd65802b8d0 comp rx=0 tx=0).stop 2024-03-28T23:37:11.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.275+0000 7fd668bd0640 1 -- 172.21.15.161:0/3611233148 shutdown_connections 2024-03-28T23:37:11.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.275+0000 7fd668bd0640 1 --2- 172.21.15.161:0/3611233148 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd66406d390 0x7fd66406dcf0 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:11.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.275+0000 7fd668bd0640 1 --2- 172.21.15.161:0/3611233148 >> v2:172.21.15.170:3300/0 conn(0x7fd6640abb70 0x7fd6640abf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:11.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.275+0000 7fd668bd0640 1 -- 172.21.15.161:0/3611233148 >> 172.21.15.161:0/3611233148 conn(0x7fd66406b150 msgr2=0x7fd66406b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:11.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.275+0000 7fd668bd0640 1 -- 172.21.15.161:0/3611233148 shutdown_connections 2024-03-28T23:37:11.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.275+0000 7fd668bd0640 1 -- 172.21.15.161:0/3611233148 wait complete. 2024-03-28T23:37:11.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.276+0000 7fd668bd0640 1 Processor -- start 2024-03-28T23:37:11.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.276+0000 7fd668bd0640 1 -- start start 2024-03-28T23:37:11.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.277+0000 7fd668bd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd66406d390 0x7fd6640707f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:11.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.277+0000 7fd668bd0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6640abb70 0x7fd664146510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:11.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.277+0000 7fd668bd0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6640c46b0 0x7fd664146a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:11.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.277+0000 7fd668bd0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd66406e230 con 0x7fd6640c46b0 2024-03-28T23:37:11.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.277+0000 7fd668bd0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd6640c5210 con 0x7fd6640abb70 2024-03-28T23:37:11.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.277+0000 7fd668bd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd66406e3b0 con 0x7fd66406d390 2024-03-28T23:37:11.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.277+0000 7fd662d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6640c46b0 0x7fd664146a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:11.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.277+0000 7fd662d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6640c46b0 0x7fd664146a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55690/0 (socket says 172.21.15.161:55690) 2024-03-28T23:37:11.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.277+0000 7fd662d9d640 1 -- 172.21.15.161:0/671968070 learned_addr learned my addr 172.21.15.161:0/671968070 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:11.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.277+0000 7fd661d9b640 1 --2- 172.21.15.161:0/671968070 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6640abb70 0x7fd664146510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:11.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.277+0000 7fd66259c640 1 --2- 172.21.15.161:0/671968070 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd66406d390 0x7fd6640707f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:11.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.277+0000 7fd662d9d640 1 -- 172.21.15.161:0/671968070 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd66406d390 msgr2=0x7fd6640707f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:11.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.277+0000 7fd662d9d640 1 --2- 172.21.15.161:0/671968070 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd66406d390 0x7fd6640707f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:11.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.277+0000 7fd662d9d640 1 -- 172.21.15.161:0/671968070 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6640abb70 msgr2=0x7fd664146510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:11.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.277+0000 7fd662d9d640 1 --2- 172.21.15.161:0/671968070 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6640abb70 0x7fd664146510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:11.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.277+0000 7fd662d9d640 1 -- 172.21.15.161:0/671968070 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6641e9250 con 0x7fd6640c46b0 2024-03-28T23:37:11.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.277+0000 7fd662d9d640 1 --2- 172.21.15.161:0/671968070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6640c46b0 0x7fd664146a50 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fd65c00c730 tx=0x7fd65c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:11.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.277+0000 7fd652ffd640 1 -- 172.21.15.161:0/671968070 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c018070 con 0x7fd6640c46b0 2024-03-28T23:37:11.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.277+0000 7fd652ffd640 1 -- 172.21.15.161:0/671968070 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd65c00ce40 con 0x7fd6640c46b0 2024-03-28T23:37:11.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.277+0000 7fd668bd0640 1 -- 172.21.15.161:0/671968070 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6641e9530 con 0x7fd6640c46b0 2024-03-28T23:37:11.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.277+0000 7fd652ffd640 1 -- 172.21.15.161:0/671968070 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c00a860 con 0x7fd6640c46b0 2024-03-28T23:37:11.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.278+0000 7fd668bd0640 1 -- 172.21.15.161:0/671968070 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6641e9af0 con 0x7fd6640c46b0 2024-03-28T23:37:11.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.278+0000 7fd668bd0640 1 -- 172.21.15.161:0/671968070 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd664068f90 con 0x7fd6640c46b0 2024-03-28T23:37:11.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.279+0000 7fd652ffd640 1 -- 172.21.15.161:0/671968070 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd65c01b070 con 0x7fd6640c46b0 2024-03-28T23:37:11.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.279+0000 7fd652ffd640 1 --2- 172.21.15.161:0/671968070 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd64406b500 0x7fd64406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:11.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.279+0000 7fd652ffd640 1 -- 172.21.15.161:0/671968070 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 6297+0+0 (secure 0 0 0) 0x7fd65c0898f0 con 0x7fd6640c46b0 2024-03-28T23:37:11.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.279+0000 7fd66259c640 1 --2- 172.21.15.161:0/671968070 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd64406b500 0x7fd64406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:11.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.280+0000 7fd66259c640 1 --2- 172.21.15.161:0/671968070 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd64406b500 0x7fd64406d9c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fd65400a990 tx=0x7fd654008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:11.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.284+0000 7fd652ffd640 1 -- 172.21.15.161:0/671968070 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd65c0585b0 con 0x7fd6640c46b0 2024-03-28T23:37:11.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:11.461+0000 7fd668bd0640 1 -- 172.21.15.161:0/671968070 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd66406d7c0 con 0x7fd6640c46b0 2024-03-28T23:37:12.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.151+0000 7fd652ffd640 1 -- 172.21.15.161:0/671968070 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_data' removed v168) ==== 172+0+0 (secure 0 0 0) 0x7fd65c057eb0 con 0x7fd6640c46b0 2024-03-28T23:37:12.151 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs2_data' removed 2024-03-28T23:37:12.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.154+0000 7fd668bd0640 1 -- 172.21.15.161:0/671968070 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd64406b500 msgr2=0x7fd64406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:12.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.154+0000 7fd668bd0640 1 --2- 172.21.15.161:0/671968070 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd64406b500 0x7fd64406d9c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fd65400a990 tx=0x7fd654008040 comp rx=0 tx=0).stop 2024-03-28T23:37:12.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.154+0000 7fd668bd0640 1 -- 172.21.15.161:0/671968070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6640c46b0 msgr2=0x7fd664146a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:12.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.154+0000 7fd668bd0640 1 --2- 172.21.15.161:0/671968070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6640c46b0 0x7fd664146a50 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fd65c00c730 tx=0x7fd65c00cbf0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.154+0000 7fd668bd0640 1 -- 172.21.15.161:0/671968070 shutdown_connections 2024-03-28T23:37:12.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.154+0000 7fd668bd0640 1 --2- 172.21.15.161:0/671968070 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd64406b500 0x7fd64406d9c0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.154+0000 7fd668bd0640 1 --2- 172.21.15.161:0/671968070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6640c46b0 0x7fd664146a50 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.154+0000 7fd668bd0640 1 --2- 172.21.15.161:0/671968070 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6640abb70 0x7fd664146510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.154+0000 7fd668bd0640 1 --2- 172.21.15.161:0/671968070 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd66406d390 0x7fd6640707f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.154+0000 7fd668bd0640 1 -- 172.21.15.161:0/671968070 >> 172.21.15.161:0/671968070 conn(0x7fd66406b150 msgr2=0x7fd6641de390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:12.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.154+0000 7fd668bd0640 1 -- 172.21.15.161:0/671968070 shutdown_connections 2024-03-28T23:37:12.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.154+0000 7fd668bd0640 1 -- 172.21.15.161:0/671968070 wait complete. 2024-03-28T23:37:12.167 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-28T23:37:12.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.271+0000 7f6eb736c640 1 Processor -- start 2024-03-28T23:37:12.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.271+0000 7f6eb736c640 1 -- start start 2024-03-28T23:37:12.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.272+0000 7f6eb736c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6eb0152bc0 0x7f6eb00c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:12.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.272+0000 7f6eb736c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6eb01589b0 0x7f6eb0158e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:12.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.272+0000 7f6eb736c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6eb00c6f60 con 0x7f6eb00c8a80 2024-03-28T23:37:12.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.272+0000 7f6eb736c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f6eb00c6120 con 0x7f6eb0152bc0 2024-03-28T23:37:12.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.272+0000 7f6eb736c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6eb00c6400 con 0x7f6eb01589b0 2024-03-28T23:37:12.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.272+0000 7f6eb4907640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6eb00c8a80 0x7f6eb00c8ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41644/0 (socket says 172.21.15.161:41644) 2024-03-28T23:37:12.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.272+0000 7f6eb4907640 1 -- 172.21.15.161:0/837188142 learned_addr learned my addr 172.21.15.161:0/837188142 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:12.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6eb5108640 1 --2- 172.21.15.161:0/837188142 >> v2:172.21.15.170:3300/0 conn(0x7f6eb0152bc0 0x7f6eb00c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:12.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6eb5909640 1 --2- 172.21.15.161:0/837188142 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6eb01589b0 0x7f6eb0158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:12.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6ea7fff640 1 -- 172.21.15.161:0/837188142 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3087008976 0 0) 0x7f6eb00c6f60 con 0x7f6eb00c8a80 2024-03-28T23:37:12.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6ea7fff640 1 -- 172.21.15.161:0/837188142 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e9c003520 con 0x7f6eb00c8a80 2024-03-28T23:37:12.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6ea7fff640 1 -- 172.21.15.161:0/837188142 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4191362927 0 0) 0x7f6e9c003520 con 0x7f6eb00c8a80 2024-03-28T23:37:12.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6ea7fff640 1 -- 172.21.15.161:0/837188142 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6eb01589b0 msgr2=0x7f6eb0158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:12.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6ea7fff640 1 --2- 172.21.15.161:0/837188142 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6eb01589b0 0x7f6eb0158e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6ea7fff640 1 -- 172.21.15.161:0/837188142 >> v2:172.21.15.170:3300/0 conn(0x7f6eb0152bc0 msgr2=0x7f6eb00c0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:12.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6ea7fff640 1 --2- 172.21.15.161:0/837188142 >> v2:172.21.15.170:3300/0 conn(0x7f6eb0152bc0 0x7f6eb00c0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6ea7fff640 1 -- 172.21.15.161:0/837188142 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eb00c7240 con 0x7f6eb00c8a80 2024-03-28T23:37:12.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6ea7fff640 1 -- 172.21.15.161:0/837188142 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f6ea0001670 con 0x7f6eb00c8a80 2024-03-28T23:37:12.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6ea7fff640 1 -- 172.21.15.161:0/837188142 >> v1:172.21.15.161:6789/0 conn(0x7f6eb00c8a80 legacy=0x7f6eb00c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:12.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6ea7fff640 1 --2- 172.21.15.161:0/837188142 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e9c003a90 0x7f6e9c003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:12.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6ea7fff640 1 --2- 172.21.15.161:0/837188142 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6eb01589b0 0x7f6e9c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:12.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6ea7fff640 1 --2- 172.21.15.161:0/837188142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e9c004d40 0x7f6e9c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:12.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6ea7fff640 1 -- 172.21.15.161:0/837188142 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb00c6120 con 0x7f6e9c003a90 2024-03-28T23:37:12.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6ea7fff640 1 -- 172.21.15.161:0/837188142 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb00c6400 con 0x7f6e9c004d40 2024-03-28T23:37:12.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6ea7fff640 1 -- 172.21.15.161:0/837188142 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6e9c00b9f0 con 0x7f6eb01589b0 2024-03-28T23:37:12.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6eb5108640 1 --2- 172.21.15.161:0/837188142 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e9c003a90 0x7f6e9c003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:12.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6eb4907640 1 --2- 172.21.15.161:0/837188142 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6eb01589b0 0x7f6e9c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:12.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6eb5909640 1 --2- 172.21.15.161:0/837188142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e9c004d40 0x7f6e9c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:12.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6eb5108640 1 -- 172.21.15.161:0/837188142 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6eb01589b0 msgr2=0x7f6e9c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:12.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6eb5108640 1 --2- 172.21.15.161:0/837188142 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6eb01589b0 0x7f6e9c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6eb5108640 1 -- 172.21.15.161:0/837188142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e9c004d40 msgr2=0x7f6e9c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:12.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6eb5108640 1 --2- 172.21.15.161:0/837188142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e9c004d40 0x7f6e9c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.273+0000 7f6eb5108640 1 -- 172.21.15.161:0/837188142 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e9c003520 con 0x7f6e9c003a90 2024-03-28T23:37:12.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.274+0000 7f6eb5108640 1 --2- 172.21.15.161:0/837188142 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e9c003a90 0x7f6e9c003eb0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f6eac009ac0 tx=0x7f6e9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d8d362eed90c4b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:12.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.274+0000 7f6ea7fff640 1 -- 172.21.15.161:0/837188142 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eac039070 con 0x7f6e9c003a90 2024-03-28T23:37:12.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.274+0000 7f6ea7fff640 1 -- 172.21.15.161:0/837188142 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6eac02c710 con 0x7f6e9c003a90 2024-03-28T23:37:12.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.274+0000 7f6ea7fff640 1 -- 172.21.15.161:0/837188142 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eac009690 con 0x7f6e9c003a90 2024-03-28T23:37:12.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.275+0000 7f6eb736c640 1 -- 172.21.15.161:0/837188142 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e9c003a90 msgr2=0x7f6e9c003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:12.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.275+0000 7f6eb736c640 1 --2- 172.21.15.161:0/837188142 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e9c003a90 0x7f6e9c003eb0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f6eac009ac0 tx=0x7f6e9c005830 comp rx=0 tx=0).stop 2024-03-28T23:37:12.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.275+0000 7f6eb5909640 1 -- 172.21.15.161:0/837188142 reap_dead start 2024-03-28T23:37:12.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.275+0000 7f6eb736c640 1 -- 172.21.15.161:0/837188142 shutdown_connections 2024-03-28T23:37:12.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.275+0000 7f6eb736c640 1 -- 172.21.15.161:0/837188142 >> 172.21.15.161:0/837188142 conn(0x7f6eb00caa80 msgr2=0x7f6eb00cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:12.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.275+0000 7f6eb736c640 1 -- 172.21.15.161:0/837188142 shutdown_connections 2024-03-28T23:37:12.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.275+0000 7f6eb736c640 1 -- 172.21.15.161:0/837188142 wait complete. 2024-03-28T23:37:12.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.276+0000 7f6eb736c640 1 Processor -- start 2024-03-28T23:37:12.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.276+0000 7f6eb736c640 1 -- start start 2024-03-28T23:37:12.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.277+0000 7f6eb736c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e9c003a90 0x7f6eb0147e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:12.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.277+0000 7f6eb736c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6eb0148fb0 0x7f6eb01eab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:12.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.277+0000 7f6eb736c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6eb01eb0c0 0x7f6eb00aa4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:12.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.277+0000 7f6eb736c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb0152fa0 con 0x7f6eb0148fb0 2024-03-28T23:37:12.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.277+0000 7f6eb736c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb00c95f0 con 0x7f6eb01eb0c0 2024-03-28T23:37:12.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.277+0000 7f6eb736c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6eb00ac400 con 0x7f6e9c003a90 2024-03-28T23:37:12.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.277+0000 7f6eb4907640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6eb0148fb0 0x7f6eb01eab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:12.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.277+0000 7f6eb4907640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6eb0148fb0 0x7f6eb01eab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55702/0 (socket says 172.21.15.161:55702) 2024-03-28T23:37:12.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.277+0000 7f6eb4907640 1 -- 172.21.15.161:0/2711328211 learned_addr learned my addr 172.21.15.161:0/2711328211 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:12.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.277+0000 7f6eb5108640 1 --2- 172.21.15.161:0/2711328211 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e9c003a90 0x7f6eb0147e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:12.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.277+0000 7f6eb5909640 1 --2- 172.21.15.161:0/2711328211 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6eb01eb0c0 0x7f6eb00aa4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:12.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.277+0000 7f6eb4907640 1 -- 172.21.15.161:0/2711328211 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e9c003a90 msgr2=0x7f6eb0147e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:12.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.277+0000 7f6eb4907640 1 --2- 172.21.15.161:0/2711328211 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e9c003a90 0x7f6eb0147e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.277+0000 7f6eb4907640 1 -- 172.21.15.161:0/2711328211 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6eb01eb0c0 msgr2=0x7f6eb00aa4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:12.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.277+0000 7f6eb4907640 1 --2- 172.21.15.161:0/2711328211 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6eb01eb0c0 0x7f6eb00aa4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.277+0000 7f6eb4907640 1 -- 172.21.15.161:0/2711328211 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eb01ee090 con 0x7f6eb0148fb0 2024-03-28T23:37:12.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.277+0000 7f6eb5108640 1 --2- 172.21.15.161:0/2711328211 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e9c003a90 0x7f6eb0147e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:37:12.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.277+0000 7f6eb4907640 1 --2- 172.21.15.161:0/2711328211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6eb0148fb0 0x7f6eb01eab80 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0007fd0 tx=0x7f6ea0007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:12.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.277+0000 7f6ea5ffb640 1 -- 172.21.15.161:0/2711328211 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea000f750 con 0x7f6eb0148fb0 2024-03-28T23:37:12.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.277+0000 7f6ea5ffb640 1 -- 172.21.15.161:0/2711328211 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ea0018070 con 0x7f6eb0148fb0 2024-03-28T23:37:12.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.277+0000 7f6eb736c640 1 -- 172.21.15.161:0/2711328211 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eb01ee370 con 0x7f6eb0148fb0 2024-03-28T23:37:12.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.277+0000 7f6ea5ffb640 1 -- 172.21.15.161:0/2711328211 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea000fb30 con 0x7f6eb0148fb0 2024-03-28T23:37:12.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.278+0000 7f6eb736c640 1 -- 172.21.15.161:0/2711328211 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eb01ee930 con 0x7f6eb0148fb0 2024-03-28T23:37:12.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.278+0000 7f6eb736c640 1 -- 172.21.15.161:0/2711328211 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6eb004f4b0 con 0x7f6eb0148fb0 2024-03-28T23:37:12.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.279+0000 7f6ea5ffb640 1 -- 172.21.15.161:0/2711328211 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6ea000fcd0 con 0x7f6eb0148fb0 2024-03-28T23:37:12.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.279+0000 7f6ea5ffb640 1 --2- 172.21.15.161:0/2711328211 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6e9406b500 0x7f6e9406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:12.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.279+0000 7f6ea5ffb640 1 -- 172.21.15.161:0/2711328211 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7f6ea008a150 con 0x7f6eb0148fb0 2024-03-28T23:37:12.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.279+0000 7f6eb5108640 1 --2- 172.21.15.161:0/2711328211 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6e9406b500 0x7f6e9406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:12.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.280+0000 7f6eb5108640 1 --2- 172.21.15.161:0/2711328211 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6e9406b500 0x7f6e9406d9c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f6eac009ac0 tx=0x7f6eac030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:12.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.284+0000 7f6ea5ffb640 1 -- 172.21.15.161:0/2711328211 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6ea00589d0 con 0x7f6eb0148fb0 2024-03-28T23:37:12.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.456+0000 7f6eb736c640 1 -- 172.21.15.161:0/2711328211 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f6eb014bfe0 con 0x7f6eb0148fb0 2024-03-28T23:37:12.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.460+0000 7f6ea5ffb640 1 -- 172.21.15.161:0/2711328211 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v54) ==== 104+0+0 (secure 0 0 0) 0x7f6ea005ca70 con 0x7f6eb0148fb0 2024-03-28T23:37:12.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.462+0000 7f6eb736c640 1 -- 172.21.15.161:0/2711328211 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6e9406b500 msgr2=0x7f6e9406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:12.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.462+0000 7f6eb736c640 1 --2- 172.21.15.161:0/2711328211 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6e9406b500 0x7f6e9406d9c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f6eac009ac0 tx=0x7f6eac030000 comp rx=0 tx=0).stop 2024-03-28T23:37:12.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.462+0000 7f6eb736c640 1 -- 172.21.15.161:0/2711328211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6eb0148fb0 msgr2=0x7f6eb01eab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:12.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.462+0000 7f6eb736c640 1 --2- 172.21.15.161:0/2711328211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6eb0148fb0 0x7f6eb01eab80 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0007fd0 tx=0x7f6ea0007430 comp rx=0 tx=0).stop 2024-03-28T23:37:12.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.462+0000 7f6eb736c640 1 -- 172.21.15.161:0/2711328211 shutdown_connections 2024-03-28T23:37:12.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.462+0000 7f6eb736c640 1 --2- 172.21.15.161:0/2711328211 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6e9406b500 0x7f6e9406d9c0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.462+0000 7f6eb736c640 1 --2- 172.21.15.161:0/2711328211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6eb0148fb0 0x7f6eb01eab80 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.462+0000 7f6eb736c640 1 --2- 172.21.15.161:0/2711328211 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6eb01eb0c0 0x7f6eb00aa4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.462+0000 7f6eb736c640 1 --2- 172.21.15.161:0/2711328211 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e9c003a90 0x7f6eb0147e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.462+0000 7f6eb736c640 1 -- 172.21.15.161:0/2711328211 >> 172.21.15.161:0/2711328211 conn(0x7f6eb00caa80 msgr2=0x7f6eb014cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:12.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.462+0000 7f6eb736c640 1 -- 172.21.15.161:0/2711328211 shutdown_connections 2024-03-28T23:37:12.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.462+0000 7f6eb736c640 1 -- 172.21.15.161:0/2711328211 wait complete. 2024-03-28T23:37:12.473 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.a mds_join_fs 2024-03-28T23:37:12.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.580+0000 7fc31bcb5640 1 Processor -- start 2024-03-28T23:37:12.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.581+0000 7fc31bcb5640 1 -- start start 2024-03-28T23:37:12.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.581+0000 7fc31bcb5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc314152bc0 0x7fc3140c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:12.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.581+0000 7fc31bcb5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc3141589b0 0x7fc314158e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:12.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.581+0000 7fc31bcb5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3140c6f60 con 0x7fc3140c8a80 2024-03-28T23:37:12.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.582+0000 7fc31bcb5640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc3140c6120 con 0x7fc314152bc0 2024-03-28T23:37:12.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.582+0000 7fc31bcb5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fc3140c6400 con 0x7fc3141589b0 2024-03-28T23:37:12.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.582+0000 7fc319a51640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc314152bc0 0x7fc3140c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:12.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.582+0000 7fc319a51640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc314152bc0 0x7fc3140c0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55712/0 (socket says 172.21.15.161:55712) 2024-03-28T23:37:12.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.582+0000 7fc319a51640 1 -- 172.21.15.161:0/1835099835 learned_addr learned my addr 172.21.15.161:0/1835099835 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:12.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.582+0000 7fc31a252640 1 --2- 172.21.15.161:0/1835099835 >> v2:172.21.15.170:3300/0 conn(0x7fc3141589b0 0x7fc314158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:12.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.582+0000 7fc319a51640 1 -- 172.21.15.161:0/1835099835 >> v2:172.21.15.170:3300/0 conn(0x7fc3141589b0 msgr2=0x7fc314158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:12.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.582+0000 7fc319a51640 1 --2- 172.21.15.161:0/1835099835 >> v2:172.21.15.170:3300/0 conn(0x7fc3141589b0 0x7fc314158e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.582+0000 7fc319a51640 1 -- 172.21.15.161:0/1835099835 >> v1:172.21.15.170:6789/0 conn(0x7fc3140c8a80 legacy=0x7fc3140c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:12.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.582+0000 7fc319a51640 1 -- 172.21.15.161:0/1835099835 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3140c7240 con 0x7fc314152bc0 2024-03-28T23:37:12.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.583+0000 7fc319a51640 1 --2- 172.21.15.161:0/1835099835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc314152bc0 0x7fc3140c0330 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fc3140aa5d0 tx=0x7fc31002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f007724b31bc7be server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:12.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.583+0000 7fc318a4f640 1 -- 172.21.15.161:0/1835099835 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc310034070 con 0x7fc314152bc0 2024-03-28T23:37:12.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.583+0000 7fc318a4f640 1 -- 172.21.15.161:0/1835099835 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc310030860 con 0x7fc314152bc0 2024-03-28T23:37:12.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.583+0000 7fc318a4f640 1 -- 172.21.15.161:0/1835099835 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc310030c30 con 0x7fc314152bc0 2024-03-28T23:37:12.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.583+0000 7fc31bcb5640 1 -- 172.21.15.161:0/1835099835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc314152bc0 msgr2=0x7fc3140c0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:12.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.583+0000 7fc31bcb5640 1 --2- 172.21.15.161:0/1835099835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc314152bc0 0x7fc3140c0330 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fc3140aa5d0 tx=0x7fc31002b9a0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.583+0000 7fc31bcb5640 1 -- 172.21.15.161:0/1835099835 shutdown_connections 2024-03-28T23:37:12.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.583+0000 7fc31bcb5640 1 --2- 172.21.15.161:0/1835099835 >> v2:172.21.15.170:3300/0 conn(0x7fc3141589b0 0x7fc314158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.583+0000 7fc31bcb5640 1 --2- 172.21.15.161:0/1835099835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc314152bc0 0x7fc3140c0330 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.583+0000 7fc31bcb5640 1 -- 172.21.15.161:0/1835099835 >> 172.21.15.161:0/1835099835 conn(0x7fc3140caa80 msgr2=0x7fc3140cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:12.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.583+0000 7fc31bcb5640 1 -- 172.21.15.161:0/1835099835 shutdown_connections 2024-03-28T23:37:12.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.583+0000 7fc31bcb5640 1 -- 172.21.15.161:0/1835099835 wait complete. 2024-03-28T23:37:12.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.585+0000 7fc31bcb5640 1 Processor -- start 2024-03-28T23:37:12.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.585+0000 7fc31bcb5640 1 -- start start 2024-03-28T23:37:12.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.585+0000 7fc31bcb5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3140c8a80 0x7fc3140ab800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:12.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.585+0000 7fc31bcb5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc314152bc0 0x7fc314150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:12.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.585+0000 7fc31bcb5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3141589b0 0x7fc314151030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:12.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.585+0000 7fc31bcb5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc314152fa0 con 0x7fc314152bc0 2024-03-28T23:37:12.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.585+0000 7fc31bcb5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fc3140c95f0 con 0x7fc3140c8a80 2024-03-28T23:37:12.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.585+0000 7fc31bcb5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fc3141ede10 con 0x7fc3141589b0 2024-03-28T23:37:12.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.585+0000 7fc319250640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc314152bc0 0x7fc314150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:12.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.585+0000 7fc319250640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc314152bc0 0x7fc314150af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55716/0 (socket says 172.21.15.161:55716) 2024-03-28T23:37:12.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.585+0000 7fc319250640 1 -- 172.21.15.161:0/3332025708 learned_addr learned my addr 172.21.15.161:0/3332025708 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:12.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.585+0000 7fc319a51640 1 --2- 172.21.15.161:0/3332025708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3140c8a80 0x7fc3140ab800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:12.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.585+0000 7fc31a252640 1 --2- 172.21.15.161:0/3332025708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3141589b0 0x7fc314151030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:12.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.586+0000 7fc319250640 1 -- 172.21.15.161:0/3332025708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3141589b0 msgr2=0x7fc314151030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:12.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.586+0000 7fc319250640 1 --2- 172.21.15.161:0/3332025708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3141589b0 0x7fc314151030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.586+0000 7fc319250640 1 -- 172.21.15.161:0/3332025708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3140c8a80 msgr2=0x7fc3140ab800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:12.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.586+0000 7fc319250640 1 --2- 172.21.15.161:0/3332025708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3140c8a80 0x7fc3140ab800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.586+0000 7fc319250640 1 -- 172.21.15.161:0/3332025708 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc314147460 con 0x7fc314152bc0 2024-03-28T23:37:12.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.586+0000 7fc31a252640 1 --2- 172.21.15.161:0/3332025708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3141589b0 0x7fc314151030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:37:12.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.586+0000 7fc319250640 1 --2- 172.21.15.161:0/3332025708 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc314152bc0 0x7fc314150af0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7fc30400c400 tx=0x7fc30400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:12.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.586+0000 7fc30a7fc640 1 -- 172.21.15.161:0/3332025708 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3040052d0 con 0x7fc314152bc0 2024-03-28T23:37:12.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.586+0000 7fc30a7fc640 1 -- 172.21.15.161:0/3332025708 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc304005490 con 0x7fc314152bc0 2024-03-28T23:37:12.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.586+0000 7fc30a7fc640 1 -- 172.21.15.161:0/3332025708 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc304007260 con 0x7fc314152bc0 2024-03-28T23:37:12.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.586+0000 7fc31bcb5640 1 -- 172.21.15.161:0/3332025708 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc314151b80 con 0x7fc314152bc0 2024-03-28T23:37:12.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.586+0000 7fc31bcb5640 1 -- 172.21.15.161:0/3332025708 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc314151e60 con 0x7fc314152bc0 2024-03-28T23:37:12.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.587+0000 7fc31bcb5640 1 -- 172.21.15.161:0/3332025708 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3140cb7b0 con 0x7fc314152bc0 2024-03-28T23:37:12.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.587+0000 7fc30a7fc640 1 -- 172.21.15.161:0/3332025708 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc304007400 con 0x7fc314152bc0 2024-03-28T23:37:12.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.588+0000 7fc30a7fc640 1 --2- 172.21.15.161:0/3332025708 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc2f806b500 0x7fc2f806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:12.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.588+0000 7fc319a51640 1 --2- 172.21.15.161:0/3332025708 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc2f806b500 0x7fc2f806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:12.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.588+0000 7fc30a7fc640 1 -- 172.21.15.161:0/3332025708 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7fc3040895e0 con 0x7fc314152bc0 2024-03-28T23:37:12.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.588+0000 7fc319a51640 1 --2- 172.21.15.161:0/3332025708 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc2f806b500 0x7fc2f806d9c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc310036000 tx=0x7fc310008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:12.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.592+0000 7fc30a7fc640 1 -- 172.21.15.161:0/3332025708 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc3140cb7b0 con 0x7fc314152bc0 2024-03-28T23:37:12.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.764+0000 7fc31bcb5640 1 -- 172.21.15.161:0/3332025708 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.a", "name": "mds_join_fs"} v 0) -- 0x7fc31404f4b0 con 0x7fc314152bc0 2024-03-28T23:37:12.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.767+0000 7fc30a7fc640 1 -- 172.21.15.161:0/3332025708 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.a", "name": "mds_join_fs"}]=0 v55) ==== 96+0+0 (secure 0 0 0) 0x7fc30405ce20 con 0x7fc314152bc0 2024-03-28T23:37:12.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.769+0000 7fc31bcb5640 1 -- 172.21.15.161:0/3332025708 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc2f806b500 msgr2=0x7fc2f806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:12.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.769+0000 7fc31bcb5640 1 --2- 172.21.15.161:0/3332025708 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc2f806b500 0x7fc2f806d9c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc310036000 tx=0x7fc310008030 comp rx=0 tx=0).stop 2024-03-28T23:37:12.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.769+0000 7fc31bcb5640 1 -- 172.21.15.161:0/3332025708 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc314152bc0 msgr2=0x7fc314150af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:12.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.769+0000 7fc31bcb5640 1 --2- 172.21.15.161:0/3332025708 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc314152bc0 0x7fc314150af0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7fc30400c400 tx=0x7fc30400c8c0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.770+0000 7fc31bcb5640 1 -- 172.21.15.161:0/3332025708 shutdown_connections 2024-03-28T23:37:12.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.770+0000 7fc31bcb5640 1 --2- 172.21.15.161:0/3332025708 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fc2f806b500 0x7fc2f806d9c0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.770+0000 7fc31bcb5640 1 --2- 172.21.15.161:0/3332025708 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc314152bc0 0x7fc314150af0 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.770+0000 7fc31bcb5640 1 --2- 172.21.15.161:0/3332025708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3141589b0 0x7fc314151030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.770+0000 7fc31bcb5640 1 --2- 172.21.15.161:0/3332025708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3140c8a80 0x7fc3140ab800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.770+0000 7fc31bcb5640 1 -- 172.21.15.161:0/3332025708 >> 172.21.15.161:0/3332025708 conn(0x7fc3140caa80 msgr2=0x7fc31414f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:12.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.770+0000 7fc31bcb5640 1 -- 172.21.15.161:0/3332025708 shutdown_connections 2024-03-28T23:37:12.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.770+0000 7fc31bcb5640 1 -- 172.21.15.161:0/3332025708 wait complete. 2024-03-28T23:37:12.781 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.c mds_join_fs 2024-03-28T23:37:12.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.883+0000 7f0b4c38a640 1 Processor -- start 2024-03-28T23:37:12.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.883+0000 7f0b4c38a640 1 -- start start 2024-03-28T23:37:12.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.883+0000 7f0b4c38a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0b4414f5c0 0x7f0b440c2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:12.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.883+0000 7f0b4c38a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0b4406cdd0 0x7f0b441524c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:12.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.884+0000 7f0b4c38a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0b440c2690 0x7f0b44152a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:12.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.884+0000 7f0b4c38a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0b440c74d0 con 0x7f0b440c2690 2024-03-28T23:37:12.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.884+0000 7f0b4c38a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f0b440c6fa0 con 0x7f0b4414f5c0 2024-03-28T23:37:12.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.884+0000 7f0b4c38a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0b44152030 con 0x7f0b4406cdd0 2024-03-28T23:37:12.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.884+0000 7f0b4a927640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0b440c2690 0x7f0b44152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:12.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.884+0000 7f0b4a927640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0b440c2690 0x7f0b44152a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55722/0 (socket says 172.21.15.161:55722) 2024-03-28T23:37:12.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.884+0000 7f0b4a927640 1 -- 172.21.15.161:0/3361700833 learned_addr learned my addr 172.21.15.161:0/3361700833 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:12.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.884+0000 7f0b49925640 1 --2- 172.21.15.161:0/3361700833 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0b4406cdd0 0x7f0b441524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:12.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.884+0000 7f0b4a126640 1 --2- 172.21.15.161:0/3361700833 >> v2:172.21.15.170:3300/0 conn(0x7f0b4414f5c0 0x7f0b440c2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:12.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.884+0000 7f0b4a927640 1 -- 172.21.15.161:0/3361700833 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0b4406cdd0 msgr2=0x7f0b441524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:12.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.884+0000 7f0b4a927640 1 --2- 172.21.15.161:0/3361700833 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0b4406cdd0 0x7f0b441524c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.884+0000 7f0b4a927640 1 -- 172.21.15.161:0/3361700833 >> v2:172.21.15.170:3300/0 conn(0x7f0b4414f5c0 msgr2=0x7f0b440c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:12.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.884+0000 7f0b4a927640 1 --2- 172.21.15.161:0/3361700833 >> v2:172.21.15.170:3300/0 conn(0x7f0b4414f5c0 0x7f0b440c2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.884+0000 7f0b4a927640 1 -- 172.21.15.161:0/3361700833 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b440c2f10 con 0x7f0b440c2690 2024-03-28T23:37:12.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.885+0000 7f0b4a927640 1 --2- 172.21.15.161:0/3361700833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b440c2690 0x7f0b44152a00 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f0b440c76e0 tx=0x7f0b3c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb67e83b57e2611f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:12.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.885+0000 7f0b49124640 1 -- 172.21.15.161:0/3361700833 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c02ca70 con 0x7f0b440c2690 2024-03-28T23:37:12.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.885+0000 7f0b49124640 1 -- 172.21.15.161:0/3361700833 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b3c02cc10 con 0x7f0b440c2690 2024-03-28T23:37:12.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.885+0000 7f0b49124640 1 -- 172.21.15.161:0/3361700833 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c0343f0 con 0x7f0b440c2690 2024-03-28T23:37:12.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.885+0000 7f0b4c38a640 1 -- 172.21.15.161:0/3361700833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b440c2690 msgr2=0x7f0b44152a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:12.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.885+0000 7f0b4c38a640 1 --2- 172.21.15.161:0/3361700833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b440c2690 0x7f0b44152a00 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f0b440c76e0 tx=0x7f0b3c02bed0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.886+0000 7f0b4c38a640 1 -- 172.21.15.161:0/3361700833 shutdown_connections 2024-03-28T23:37:12.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.886+0000 7f0b4c38a640 1 --2- 172.21.15.161:0/3361700833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b440c2690 0x7f0b44152a00 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.886+0000 7f0b4c38a640 1 --2- 172.21.15.161:0/3361700833 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0b4406cdd0 0x7f0b441524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.886+0000 7f0b4c38a640 1 --2- 172.21.15.161:0/3361700833 >> v2:172.21.15.170:3300/0 conn(0x7f0b4414f5c0 0x7f0b440c2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.886+0000 7f0b4c38a640 1 -- 172.21.15.161:0/3361700833 >> 172.21.15.161:0/3361700833 conn(0x7f0b440a83f0 msgr2=0x7f0b440a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:12.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.886+0000 7f0b4c38a640 1 -- 172.21.15.161:0/3361700833 shutdown_connections 2024-03-28T23:37:12.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.886+0000 7f0b4c38a640 1 -- 172.21.15.161:0/3361700833 wait complete. 2024-03-28T23:37:12.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.887+0000 7f0b4c38a640 1 Processor -- start 2024-03-28T23:37:12.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.887+0000 7f0b4c38a640 1 -- start start 2024-03-28T23:37:12.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.887+0000 7f0b4c38a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b4406cdd0 0x7f0b44149cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:12.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.887+0000 7f0b4c38a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b440c2690 0x7f0b44154e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:12.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.887+0000 7f0b4c38a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0b4414f5c0 0x7f0b441553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:12.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.887+0000 7f0b4c38a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0b4406c140 con 0x7f0b440c2690 2024-03-28T23:37:12.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.887+0000 7f0b4c38a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0b4414f9a0 con 0x7f0b4406cdd0 2024-03-28T23:37:12.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.887+0000 7f0b4c38a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0b4406c3e0 con 0x7f0b4414f5c0 2024-03-28T23:37:12.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.887+0000 7f0b49925640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b440c2690 0x7f0b44154e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:12.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.887+0000 7f0b49925640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b440c2690 0x7f0b44154e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55734/0 (socket says 172.21.15.161:55734) 2024-03-28T23:37:12.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.887+0000 7f0b49925640 1 -- 172.21.15.161:0/3668324613 learned_addr learned my addr 172.21.15.161:0/3668324613 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:12.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.888+0000 7f0b4a126640 1 --2- 172.21.15.161:0/3668324613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b4406cdd0 0x7f0b44149cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:12.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.888+0000 7f0b4a927640 1 --2- 172.21.15.161:0/3668324613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0b4414f5c0 0x7f0b441553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:12.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.888+0000 7f0b49925640 1 -- 172.21.15.161:0/3668324613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0b4414f5c0 msgr2=0x7f0b441553c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:12.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.888+0000 7f0b49925640 1 --2- 172.21.15.161:0/3668324613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0b4414f5c0 0x7f0b441553c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.888+0000 7f0b49925640 1 -- 172.21.15.161:0/3668324613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b4406cdd0 msgr2=0x7f0b44149cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:12.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.888+0000 7f0b49925640 1 --2- 172.21.15.161:0/3668324613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b4406cdd0 0x7f0b44149cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:12.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.888+0000 7f0b49925640 1 -- 172.21.15.161:0/3668324613 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b4414bb90 con 0x7f0b440c2690 2024-03-28T23:37:12.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.888+0000 7f0b49925640 1 --2- 172.21.15.161:0/3668324613 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b440c2690 0x7f0b44154e80 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f0b3400d5f0 tx=0x7f0b3400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:12.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.888+0000 7f0b3affd640 1 -- 172.21.15.161:0/3668324613 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b34004240 con 0x7f0b440c2690 2024-03-28T23:37:12.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.888+0000 7f0b3affd640 1 -- 172.21.15.161:0/3668324613 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b340043e0 con 0x7f0b440c2690 2024-03-28T23:37:12.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.888+0000 7f0b3affd640 1 -- 172.21.15.161:0/3668324613 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b34003610 con 0x7f0b440c2690 2024-03-28T23:37:12.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.888+0000 7f0b4c38a640 1 -- 172.21.15.161:0/3668324613 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b4414be70 con 0x7f0b440c2690 2024-03-28T23:37:12.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.888+0000 7f0b4c38a640 1 -- 172.21.15.161:0/3668324613 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b441532f0 con 0x7f0b440c2690 2024-03-28T23:37:12.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.889+0000 7f0b4c38a640 1 -- 172.21.15.161:0/3668324613 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b4404f4b0 con 0x7f0b440c2690 2024-03-28T23:37:12.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.889+0000 7f0b3affd640 1 -- 172.21.15.161:0/3668324613 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0b3401b020 con 0x7f0b440c2690 2024-03-28T23:37:12.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.890+0000 7f0b3affd640 1 --2- 172.21.15.161:0/3668324613 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0b2806b500 0x7f0b2806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:12.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.890+0000 7f0b4a126640 1 --2- 172.21.15.161:0/3668324613 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0b2806b500 0x7f0b2806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:12.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.890+0000 7f0b3affd640 1 -- 172.21.15.161:0/3668324613 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7f0b34089ed0 con 0x7f0b440c2690 2024-03-28T23:37:12.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.890+0000 7f0b4a126640 1 --2- 172.21.15.161:0/3668324613 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0b2806b500 0x7f0b2806d9c0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f0b40006fd0 tx=0x7f0b40008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:12.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:12.894+0000 7f0b3affd640 1 -- 172.21.15.161:0/3668324613 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0b34058d00 con 0x7f0b440c2690 2024-03-28T23:37:13.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.065+0000 7f0b4c38a640 1 -- 172.21.15.161:0/3668324613 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"} v 0) -- 0x7f0b4406b830 con 0x7f0b440c2690 2024-03-28T23:37:13.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.069+0000 7f0b3affd640 1 -- 172.21.15.161:0/3668324613 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"}]=0 v56) ==== 96+0+0 (secure 0 0 0) 0x7f0b34058600 con 0x7f0b440c2690 2024-03-28T23:37:13.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.071+0000 7f0b4c38a640 1 -- 172.21.15.161:0/3668324613 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0b2806b500 msgr2=0x7f0b2806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:13.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.071+0000 7f0b4c38a640 1 --2- 172.21.15.161:0/3668324613 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0b2806b500 0x7f0b2806d9c0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f0b40006fd0 tx=0x7f0b40008040 comp rx=0 tx=0).stop 2024-03-28T23:37:13.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.071+0000 7f0b4c38a640 1 -- 172.21.15.161:0/3668324613 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b440c2690 msgr2=0x7f0b44154e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:13.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.071+0000 7f0b4c38a640 1 --2- 172.21.15.161:0/3668324613 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b440c2690 0x7f0b44154e80 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f0b3400d5f0 tx=0x7f0b3400dab0 comp rx=0 tx=0).stop 2024-03-28T23:37:13.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.071+0000 7f0b4c38a640 1 -- 172.21.15.161:0/3668324613 shutdown_connections 2024-03-28T23:37:13.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.071+0000 7f0b4c38a640 1 --2- 172.21.15.161:0/3668324613 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0b2806b500 0x7f0b2806d9c0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:13.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.071+0000 7f0b4c38a640 1 --2- 172.21.15.161:0/3668324613 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b440c2690 0x7f0b44154e80 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:13.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.071+0000 7f0b4c38a640 1 --2- 172.21.15.161:0/3668324613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0b4414f5c0 0x7f0b441553c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:13.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.071+0000 7f0b4c38a640 1 --2- 172.21.15.161:0/3668324613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0b4406cdd0 0x7f0b44149cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:13.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.071+0000 7f0b4c38a640 1 -- 172.21.15.161:0/3668324613 >> 172.21.15.161:0/3668324613 conn(0x7f0b440a83f0 msgr2=0x7f0b440a9620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:13.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.071+0000 7f0b4c38a640 1 -- 172.21.15.161:0/3668324613 shutdown_connections 2024-03-28T23:37:13.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.071+0000 7f0b4c38a640 1 -- 172.21.15.161:0/3668324613 wait complete. 2024-03-28T23:37:13.083 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla' 2024-03-28T23:37:13.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.184+0000 7f6fc13ac640 1 Processor -- start 2024-03-28T23:37:13.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.184+0000 7f6fc13ac640 1 -- start start 2024-03-28T23:37:13.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.185+0000 7f6fc13ac640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fbc152a80 0x7f6fbc155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:13.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.185+0000 7f6fc13ac640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6fbc06d9f0 0x7f6fbc0c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:13.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.185+0000 7f6fc13ac640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fbc0c98e0 con 0x7f6fbc06cb80 2024-03-28T23:37:13.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.185+0000 7f6fc13ac640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f6fbc156250 con 0x7f6fbc06d9f0 2024-03-28T23:37:13.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.185+0000 7f6fc13ac640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6fbc06c6f0 con 0x7f6fbc152a80 2024-03-28T23:37:13.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.185+0000 7f6fbb7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6fbc06cb80 0x7f6fbc0c74c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41652/0 (socket says 172.21.15.161:41652) 2024-03-28T23:37:13.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.185+0000 7f6fbb7fe640 1 -- 172.21.15.161:0/4106438727 learned_addr learned my addr 172.21.15.161:0/4106438727 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:13.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.185+0000 7f6fbaffd640 1 --2- 172.21.15.161:0/4106438727 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fbc152a80 0x7f6fbc155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:13.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.185+0000 7f6fb9ffb640 1 -- 172.21.15.161:0/4106438727 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 932609035 0 0) 0x7f6fbc0c98e0 con 0x7f6fbc06cb80 2024-03-28T23:37:13.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.185+0000 7f6fba7fc640 1 --2- 172.21.15.161:0/4106438727 >> v2:172.21.15.170:3300/0 conn(0x7f6fbc06d9f0 0x7f6fbc0c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:13.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.185+0000 7f6fb9ffb640 1 -- 172.21.15.161:0/4106438727 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fa4003520 con 0x7f6fbc06cb80 2024-03-28T23:37:13.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.185+0000 7f6fb9ffb640 1 -- 172.21.15.161:0/4106438727 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1901926492 0 0) 0x7f6fa4003520 con 0x7f6fbc06cb80 2024-03-28T23:37:13.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.185+0000 7f6fb9ffb640 1 -- 172.21.15.161:0/4106438727 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fbc152a80 msgr2=0x7f6fbc155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:13.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.185+0000 7f6fb9ffb640 1 --2- 172.21.15.161:0/4106438727 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fbc152a80 0x7f6fbc155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:13.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.185+0000 7f6fb9ffb640 1 -- 172.21.15.161:0/4106438727 >> v2:172.21.15.170:3300/0 conn(0x7f6fbc06d9f0 msgr2=0x7f6fbc0c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:13.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.185+0000 7f6fb9ffb640 1 --2- 172.21.15.161:0/4106438727 >> v2:172.21.15.170:3300/0 conn(0x7f6fbc06d9f0 0x7f6fbc0c93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:13.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.185+0000 7f6fb9ffb640 1 -- 172.21.15.161:0/4106438727 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fbc06c3c0 con 0x7f6fbc06cb80 2024-03-28T23:37:13.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.185+0000 7f6fb9ffb640 1 -- 172.21.15.161:0/4106438727 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f6fb4001670 con 0x7f6fbc06cb80 2024-03-28T23:37:13.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.185+0000 7f6fb9ffb640 1 -- 172.21.15.161:0/4106438727 >> v1:172.21.15.161:6789/0 conn(0x7f6fbc06cb80 legacy=0x7f6fbc0c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:13.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.185+0000 7f6fb9ffb640 1 --2- 172.21.15.161:0/4106438727 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fa4003a90 0x7f6fa4003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:13.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.185+0000 7f6fb9ffb640 1 --2- 172.21.15.161:0/4106438727 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fa4004800 0x7f6fa4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:13.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.185+0000 7f6fb9ffb640 1 --2- 172.21.15.161:0/4106438727 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fbc152a80 0x7f6fa4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:13.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.185+0000 7f6fb9ffb640 1 -- 172.21.15.161:0/4106438727 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6fbc156250 con 0x7f6fa4004800 2024-03-28T23:37:13.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.185+0000 7f6fb9ffb640 1 -- 172.21.15.161:0/4106438727 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6fbc06c6f0 con 0x7f6fa4003a90 2024-03-28T23:37:13.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.186+0000 7f6fb9ffb640 1 -- 172.21.15.161:0/4106438727 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6fa400bb70 con 0x7f6fbc152a80 2024-03-28T23:37:13.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.186+0000 7f6fba7fc640 1 --2- 172.21.15.161:0/4106438727 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fa4004800 0x7f6fa4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:13.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.186+0000 7f6fbb7fe640 1 --2- 172.21.15.161:0/4106438727 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fbc152a80 0x7f6fa4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:13.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.186+0000 7f6fbaffd640 1 --2- 172.21.15.161:0/4106438727 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fa4003a90 0x7f6fa4003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:13.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.186+0000 7f6fba7fc640 1 -- 172.21.15.161:0/4106438727 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fbc152a80 msgr2=0x7f6fa4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:13.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.186+0000 7f6fba7fc640 1 --2- 172.21.15.161:0/4106438727 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fbc152a80 0x7f6fa4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:13.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.186+0000 7f6fba7fc640 1 -- 172.21.15.161:0/4106438727 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fa4003a90 msgr2=0x7f6fa4003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:13.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.186+0000 7f6fba7fc640 1 --2- 172.21.15.161:0/4106438727 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fa4003a90 0x7f6fa4003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:13.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.186+0000 7f6fba7fc640 1 -- 172.21.15.161:0/4106438727 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fa4003520 con 0x7f6fa4004800 2024-03-28T23:37:13.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.187+0000 7f6fba7fc640 1 --2- 172.21.15.161:0/4106438727 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fa4004800 0x7f6fa4004c80 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0009ac0 tx=0x7f6fa4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f32526054b3599a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:13.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.187+0000 7f6fb9ffb640 1 -- 172.21.15.161:0/4106438727 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0039070 con 0x7f6fa4004800 2024-03-28T23:37:13.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.187+0000 7f6fb9ffb640 1 -- 172.21.15.161:0/4106438727 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fb002c710 con 0x7f6fa4004800 2024-03-28T23:37:13.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.187+0000 7f6fb9ffb640 1 -- 172.21.15.161:0/4106438727 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0009690 con 0x7f6fa4004800 2024-03-28T23:37:13.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.187+0000 7f6fc13ac640 1 -- 172.21.15.161:0/4106438727 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fa4004800 msgr2=0x7f6fa4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:13.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.187+0000 7f6fc13ac640 1 --2- 172.21.15.161:0/4106438727 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fa4004800 0x7f6fa4004c80 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0009ac0 tx=0x7f6fa4005830 comp rx=0 tx=0).stop 2024-03-28T23:37:13.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.187+0000 7f6fbb7fe640 1 -- 172.21.15.161:0/4106438727 reap_dead start 2024-03-28T23:37:13.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.187+0000 7f6fc13ac640 1 -- 172.21.15.161:0/4106438727 shutdown_connections 2024-03-28T23:37:13.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.187+0000 7f6fc13ac640 1 -- 172.21.15.161:0/4106438727 >> 172.21.15.161:0/4106438727 conn(0x7f6fbc0ac720 msgr2=0x7f6fbc0acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:13.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.187+0000 7f6fc13ac640 1 -- 172.21.15.161:0/4106438727 shutdown_connections 2024-03-28T23:37:13.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.187+0000 7f6fc13ac640 1 -- 172.21.15.161:0/4106438727 wait complete. 2024-03-28T23:37:13.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.189+0000 7f6fc13ac640 1 Processor -- start 2024-03-28T23:37:13.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.189+0000 7f6fc13ac640 1 -- start start 2024-03-28T23:37:13.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.189+0000 7f6fc13ac640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fa4004800 0x7f6fbc1565e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:13.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.189+0000 7f6fc13ac640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fbc158de0 0x7f6fbc0afca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:13.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.189+0000 7f6fc13ac640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fbc0b01e0 0x7f6fbc1edc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:13.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.189+0000 7f6fc13ac640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6fbc0c58b0 con 0x7f6fa4004800 2024-03-28T23:37:13.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.189+0000 7f6fc13ac640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6fbc0c55d0 con 0x7f6fbc0b01e0 2024-03-28T23:37:13.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.189+0000 7f6fc13ac640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f6fbc0c7bd0 con 0x7f6fbc158de0 2024-03-28T23:37:13.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.189+0000 7f6fbaffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fa4004800 0x7f6fbc1565e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:13.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.189+0000 7f6fbaffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fa4004800 0x7f6fbc1565e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55758/0 (socket says 172.21.15.161:55758) 2024-03-28T23:37:13.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.189+0000 7f6fbaffd640 1 -- 172.21.15.161:0/262381972 learned_addr learned my addr 172.21.15.161:0/262381972 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:13.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.190+0000 7f6fba7fc640 1 --2- 172.21.15.161:0/262381972 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fbc158de0 0x7f6fbc0afca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:13.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.190+0000 7f6fbb7fe640 1 --2- 172.21.15.161:0/262381972 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fbc0b01e0 0x7f6fbc1edc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:13.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.190+0000 7f6fbaffd640 1 -- 172.21.15.161:0/262381972 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fbc158de0 msgr2=0x7f6fbc0afca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:13.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.190+0000 7f6fbaffd640 1 --2- 172.21.15.161:0/262381972 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fbc158de0 0x7f6fbc0afca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:13.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.190+0000 7f6fbaffd640 1 -- 172.21.15.161:0/262381972 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fbc0b01e0 msgr2=0x7f6fbc1edc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:13.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.190+0000 7f6fbaffd640 1 --2- 172.21.15.161:0/262381972 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fbc0b01e0 0x7f6fbc1edc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:13.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.190+0000 7f6fbaffd640 1 -- 172.21.15.161:0/262381972 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fbc0ae340 con 0x7f6fa4004800 2024-03-28T23:37:13.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.190+0000 7f6fba7fc640 1 --2- 172.21.15.161:0/262381972 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fbc158de0 0x7f6fbc0afca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:37:13.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.190+0000 7f6fbb7fe640 1 --2- 172.21.15.161:0/262381972 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fbc0b01e0 0x7f6fbc1edc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:37:13.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.190+0000 7f6fbaffd640 1 --2- 172.21.15.161:0/262381972 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fa4004800 0x7f6fbc1565e0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f6fac009f90 tx=0x7f6fac00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:13.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.190+0000 7f6fa37fe640 1 -- 172.21.15.161:0/262381972 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fac015430 con 0x7f6fa4004800 2024-03-28T23:37:13.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.190+0000 7f6fa37fe640 1 -- 172.21.15.161:0/262381972 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fac0155d0 con 0x7f6fa4004800 2024-03-28T23:37:13.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.190+0000 7f6fc13ac640 1 -- 172.21.15.161:0/262381972 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fbc1ea850 con 0x7f6fa4004800 2024-03-28T23:37:13.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.190+0000 7f6fa37fe640 1 -- 172.21.15.161:0/262381972 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fac0159c0 con 0x7f6fa4004800 2024-03-28T23:37:13.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.190+0000 7f6fc13ac640 1 -- 172.21.15.161:0/262381972 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fbc1eae10 con 0x7f6fa4004800 2024-03-28T23:37:13.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.191+0000 7f6fc13ac640 1 -- 172.21.15.161:0/262381972 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fbc04f4b0 con 0x7f6fa4004800 2024-03-28T23:37:13.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.192+0000 7f6fa37fe640 1 -- 172.21.15.161:0/262381972 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6fac015b60 con 0x7f6fa4004800 2024-03-28T23:37:13.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.192+0000 7f6fa37fe640 1 --2- 172.21.15.161:0/262381972 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6fa806b500 0x7f6fa806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:13.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.192+0000 7f6fba7fc640 1 --2- 172.21.15.161:0/262381972 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6fa806b500 0x7f6fa806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:13.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.192+0000 7f6fa37fe640 1 -- 172.21.15.161:0/262381972 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7f6fac08b220 con 0x7f6fa4004800 2024-03-28T23:37:13.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.193+0000 7f6fba7fc640 1 --2- 172.21.15.161:0/262381972 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6fa806b500 0x7f6fa806d9c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0036000 tx=0x7f6fb0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:13.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.197+0000 7f6fa37fe640 1 -- 172.21.15.161:0/262381972 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6fac019030 con 0x7f6fa4004800 2024-03-28T23:37:13.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:13.435+0000 7f6fc13ac640 1 -- 172.21.15.161:0/262381972 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla"]} v 0) -- 0x7f6fbc06d7d0 con 0x7f6fa4004800 2024-03-28T23:37:14.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:14.157+0000 7f6fa37fe640 1 -- 172.21.15.161:0/262381972 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla"]}]=0 v417) ==== 146+0+0 (secure 0 0 0) 0x7f6fac010040 con 0x7f6fa4004800 2024-03-28T23:37:14.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:14.160+0000 7f6fc13ac640 1 -- 172.21.15.161:0/262381972 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6fa806b500 msgr2=0x7f6fa806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:14.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:14.160+0000 7f6fc13ac640 1 --2- 172.21.15.161:0/262381972 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6fa806b500 0x7f6fa806d9c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0036000 tx=0x7f6fb0030000 comp rx=0 tx=0).stop 2024-03-28T23:37:14.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:14.160+0000 7f6fc13ac640 1 -- 172.21.15.161:0/262381972 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fa4004800 msgr2=0x7f6fbc1565e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:14.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:14.160+0000 7f6fc13ac640 1 --2- 172.21.15.161:0/262381972 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fa4004800 0x7f6fbc1565e0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f6fac009f90 tx=0x7f6fac00f580 comp rx=0 tx=0).stop 2024-03-28T23:37:14.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:14.160+0000 7f6fc13ac640 1 -- 172.21.15.161:0/262381972 shutdown_connections 2024-03-28T23:37:14.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:14.160+0000 7f6fc13ac640 1 --2- 172.21.15.161:0/262381972 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f6fa806b500 0x7f6fa806d9c0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:14.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:14.160+0000 7f6fc13ac640 1 --2- 172.21.15.161:0/262381972 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fbc0b01e0 0x7f6fbc1edc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:14.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:14.160+0000 7f6fc13ac640 1 --2- 172.21.15.161:0/262381972 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fbc158de0 0x7f6fbc0afca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:14.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:14.160+0000 7f6fc13ac640 1 --2- 172.21.15.161:0/262381972 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fa4004800 0x7f6fbc1565e0 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:14.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:14.160+0000 7f6fc13ac640 1 -- 172.21.15.161:0/262381972 >> 172.21.15.161:0/262381972 conn(0x7f6fbc0ac720 msgr2=0x7f6fbc14ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:14.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:14.160+0000 7f6fc13ac640 1 -- 172.21.15.161:0/262381972 shutdown_connections 2024-03-28T23:37:14.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:14.160+0000 7f6fc13ac640 1 -- 172.21.15.161:0/262381972 wait complete. 2024-03-28T23:37:14.174 INFO:tasks.cephfs_test_runner:That a vanilla standby is preferred over others with mds_join_fs set to another fs. ... ERROR 2024-03-28T23:37:14.175 INFO:tasks.cephfs_test_runner: 2024-03-28T23:37:14.175 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-28T23:37:14.175 INFO:tasks.cephfs_test_runner:ERROR: test_join_fs_vanilla (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-28T23:37:14.175 INFO:tasks.cephfs_test_runner:That a vanilla standby is preferred over others with mds_join_fs set to another fs. 2024-03-28T23:37:14.175 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-28T23:37:14.175 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-28T23:37:14.175 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa/tasks/cephfs/test_failover.py", line 133, in test_join_fs_vanilla 2024-03-28T23:37:14.175 INFO:tasks.cephfs_test_runner: self._reach_target(target) 2024-03-28T23:37:14.175 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa/tasks/cephfs/test_failover.py", line 61, in _reach_target 2024-03-28T23:37:14.175 INFO:tasks.cephfs_test_runner: self.wait_until_true(takeover, 30) 2024-03-28T23:37:14.175 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa/tasks/ceph_test_case.py", line 347, in wait_until_true 2024-03-28T23:37:14.175 INFO:tasks.cephfs_test_runner: raise TestTimeoutError("Timed out after {0}s and {1} retries".format(elapsed, retry_count)) 2024-03-28T23:37:14.175 INFO:tasks.cephfs_test_runner:tasks.ceph_test_case.TestTimeoutError: Timed out after 30s and 0 retries 2024-03-28T23:37:14.176 INFO:tasks.cephfs_test_runner: 2024-03-28T23:37:14.176 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-28T23:37:14.176 INFO:tasks.cephfs_test_runner:Ran 8 tests in 402.516s 2024-03-28T23:37:14.176 INFO:tasks.cephfs_test_runner: 2024-03-28T23:37:14.176 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-03-28T23:37:14.176 INFO:tasks.cephfs_test_runner: 2024-03-28T23:37:14.176 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-28T23:37:14.176 INFO:tasks.cephfs_test_runner:ERROR: test_join_fs_vanilla (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-28T23:37:14.176 INFO:tasks.cephfs_test_runner:That a vanilla standby is preferred over others with mds_join_fs set to another fs. 2024-03-28T23:37:14.176 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-28T23:37:14.176 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-28T23:37:14.176 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa/tasks/cephfs/test_failover.py", line 133, in test_join_fs_vanilla 2024-03-28T23:37:14.176 INFO:tasks.cephfs_test_runner: self._reach_target(target) 2024-03-28T23:37:14.176 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa/tasks/cephfs/test_failover.py", line 61, in _reach_target 2024-03-28T23:37:14.176 INFO:tasks.cephfs_test_runner: self.wait_until_true(takeover, 30) 2024-03-28T23:37:14.177 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa/tasks/ceph_test_case.py", line 347, in wait_until_true 2024-03-28T23:37:14.177 INFO:tasks.cephfs_test_runner: raise TestTimeoutError("Timed out after {0}s and {1} retries".format(elapsed, retry_count)) 2024-03-28T23:37:14.177 INFO:tasks.cephfs_test_runner:tasks.ceph_test_case.TestTimeoutError: Timed out after 30s and 0 retries 2024-03-28T23:37:14.177 INFO:tasks.cephfs_test_runner: 2024-03-28T23:37:14.177 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_join_fs_vanilla (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-28T23:37:14.380 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1ad96a13450a4ffb9e9bd5c538854019 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_join_fs_vanilla (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-28T23:37:14.382 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-28T23:37:14.391 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-28T23:37:14.673 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-28T23:37:14.673 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-28T23:37:14.674 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:37:14.695 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20793 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-28T23:37:14.695 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-28T23:37:14.696 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T23:37:14.715 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056374,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19690 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-28T23:37:14.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:37:14.715 DEBUG:teuthology.orchestra.run.smithi170:> ip netns list 2024-03-28T23:37:14.773 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-28T23:37:14.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:37:14.774 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-28T23:37:14.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:37:14.845 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip link delete ceph-brx 2024-03-28T23:37:14.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:37:14.945 DEBUG:teuthology.orchestra.run.smithi161:> ip netns list 2024-03-28T23:37:14.964 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-28T23:37:14.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:37:14.965 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T23:37:15.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T23:37:15.035 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link delete ceph-brx 2024-03-28T23:37:15.133 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-28T23:37:15.143 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-28T23:37:15.143 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:37:15.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.247+0000 7f22341d0640 1 Processor -- start 2024-03-28T23:37:15.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.248+0000 7f22341d0640 1 -- start start 2024-03-28T23:37:15.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.248+0000 7f22341d0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f222c14be70 0x7f222c0ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:15.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.248+0000 7f22341d0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f222c0aacd0 0x7f222c15c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:15.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.248+0000 7f22341d0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f222c14c250 con 0x7f222c0ab6e0 2024-03-28T23:37:15.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.248+0000 7f22341d0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f222c1529c0 con 0x7f222c0aacd0 2024-03-28T23:37:15.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.248+0000 7f22341d0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f222c0a9dc0 con 0x7f222c14be70 2024-03-28T23:37:15.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.249+0000 7f223276d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f222c0aacd0 0x7f222c15c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:15.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.249+0000 7f223276d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f222c0aacd0 0x7f222c15c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55774/0 (socket says 172.21.15.161:55774) 2024-03-28T23:37:15.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.249+0000 7f223276d640 1 -- 172.21.15.161:0/3804178500 learned_addr learned my addr 172.21.15.161:0/3804178500 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:15.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.249+0000 7f2231f6c640 1 --2- 172.21.15.161:0/3804178500 >> v2:172.21.15.170:3300/0 conn(0x7f222c14be70 0x7f222c0ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:15.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.249+0000 7f223276d640 1 -- 172.21.15.161:0/3804178500 >> v2:172.21.15.170:3300/0 conn(0x7f222c14be70 msgr2=0x7f222c0ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:15.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.249+0000 7f223276d640 1 --2- 172.21.15.161:0/3804178500 >> v2:172.21.15.170:3300/0 conn(0x7f222c14be70 0x7f222c0ab1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:15.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.249+0000 7f223276d640 1 -- 172.21.15.161:0/3804178500 >> v1:172.21.15.170:6789/0 conn(0x7f222c0ab6e0 legacy=0x7f222c0aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:15.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.249+0000 7f223276d640 1 -- 172.21.15.161:0/3804178500 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f222c0aa0a0 con 0x7f222c0aacd0 2024-03-28T23:37:15.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.250+0000 7f223276d640 1 --2- 172.21.15.161:0/3804178500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f222c0aacd0 0x7f222c15c740 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f222c0c7520 tx=0x7f222402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7aef76aa131cde3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:15.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.250+0000 7f2230f6a640 1 -- 172.21.15.161:0/3804178500 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222402ca80 con 0x7f222c0aacd0 2024-03-28T23:37:15.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.250+0000 7f2230f6a640 1 -- 172.21.15.161:0/3804178500 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f222402cc20 con 0x7f222c0aacd0 2024-03-28T23:37:15.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.250+0000 7f2230f6a640 1 -- 172.21.15.161:0/3804178500 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22240343f0 con 0x7f222c0aacd0 2024-03-28T23:37:15.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.250+0000 7f22341d0640 1 -- 172.21.15.161:0/3804178500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f222c0aacd0 msgr2=0x7f222c15c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:15.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.250+0000 7f22341d0640 1 --2- 172.21.15.161:0/3804178500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f222c0aacd0 0x7f222c15c740 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f222c0c7520 tx=0x7f222402bee0 comp rx=0 tx=0).stop 2024-03-28T23:37:15.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.250+0000 7f22341d0640 1 -- 172.21.15.161:0/3804178500 shutdown_connections 2024-03-28T23:37:15.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.250+0000 7f22341d0640 1 --2- 172.21.15.161:0/3804178500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f222c0aacd0 0x7f222c15c740 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:15.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.250+0000 7f22341d0640 1 --2- 172.21.15.161:0/3804178500 >> v2:172.21.15.170:3300/0 conn(0x7f222c14be70 0x7f222c0ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:15.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.250+0000 7f22341d0640 1 -- 172.21.15.161:0/3804178500 >> 172.21.15.161:0/3804178500 conn(0x7f222c0c6670 msgr2=0x7f222c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:15.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.250+0000 7f22341d0640 1 -- 172.21.15.161:0/3804178500 shutdown_connections 2024-03-28T23:37:15.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.250+0000 7f22341d0640 1 -- 172.21.15.161:0/3804178500 wait complete. 2024-03-28T23:37:15.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.251+0000 7f22341d0640 1 Processor -- start 2024-03-28T23:37:15.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.252+0000 7f22341d0640 1 -- start start 2024-03-28T23:37:15.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.252+0000 7f22341d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f222c0aacd0 0x7f222c153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:15.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.252+0000 7f22341d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f222c0ab6e0 0x7f222c0c1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:15.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.252+0000 7f22341d0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f222c14be70 0x7f222c1483c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:15.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.252+0000 7f22341d0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f222c06b2d0 con 0x7f222c14be70 2024-03-28T23:37:15.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.252+0000 7f22341d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f222c06b150 con 0x7f222c0aacd0 2024-03-28T23:37:15.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.252+0000 7f22341d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f222c0c1520 con 0x7f222c0ab6e0 2024-03-28T23:37:15.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.252+0000 7f223276d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f222c14be70 0x7f222c1483c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:15.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.252+0000 7f223276d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f222c14be70 0x7f222c1483c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55780/0 (socket says 172.21.15.161:55780) 2024-03-28T23:37:15.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.252+0000 7f223276d640 1 -- 172.21.15.161:0/241923373 learned_addr learned my addr 172.21.15.161:0/241923373 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:15.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.252+0000 7f2231f6c640 1 --2- 172.21.15.161:0/241923373 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f222c0aacd0 0x7f222c153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:15.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.253+0000 7f223276d640 1 -- 172.21.15.161:0/241923373 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f222c0ab6e0 msgr2=0x7f222c0c1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:15.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.253+0000 7f223176b640 1 --2- 172.21.15.161:0/241923373 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f222c0ab6e0 0x7f222c0c1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:15.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.253+0000 7f223276d640 1 --2- 172.21.15.161:0/241923373 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f222c0ab6e0 0x7f222c0c1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:15.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.253+0000 7f223276d640 1 -- 172.21.15.161:0/241923373 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f222c0aacd0 msgr2=0x7f222c153970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:15.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.253+0000 7f223276d640 1 --2- 172.21.15.161:0/241923373 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f222c0aacd0 0x7f222c153970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:15.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.253+0000 7f223276d640 1 -- 172.21.15.161:0/241923373 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f222c14f370 con 0x7f222c14be70 2024-03-28T23:37:15.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.253+0000 7f2231f6c640 1 --2- 172.21.15.161:0/241923373 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f222c0aacd0 0x7f222c153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:37:15.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.253+0000 7f223176b640 1 --2- 172.21.15.161:0/241923373 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f222c0ab6e0 0x7f222c0c1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:37:15.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.253+0000 7f223276d640 1 --2- 172.21.15.161:0/241923373 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f222c14be70 0x7f222c1483c0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f2224009f40 tx=0x7f2224004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:15.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.253+0000 7f22227fc640 1 -- 172.21.15.161:0/241923373 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2224043020 con 0x7f222c14be70 2024-03-28T23:37:15.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.253+0000 7f22227fc640 1 -- 172.21.15.161:0/241923373 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f222403e450 con 0x7f222c14be70 2024-03-28T23:37:15.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.253+0000 7f22341d0640 1 -- 172.21.15.161:0/241923373 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f222c1490b0 con 0x7f222c14be70 2024-03-28T23:37:15.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.253+0000 7f22227fc640 1 -- 172.21.15.161:0/241923373 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2224009510 con 0x7f222c14be70 2024-03-28T23:37:15.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.253+0000 7f22341d0640 1 -- 172.21.15.161:0/241923373 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f222c1eeba0 con 0x7f222c14be70 2024-03-28T23:37:15.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.254+0000 7f22341d0640 1 -- 172.21.15.161:0/241923373 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f222c04f4b0 con 0x7f222c14be70 2024-03-28T23:37:15.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.254+0000 7f22227fc640 1 -- 172.21.15.161:0/241923373 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2224046020 con 0x7f222c14be70 2024-03-28T23:37:15.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.255+0000 7f22227fc640 1 --2- 172.21.15.161:0/241923373 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f221006b4d0 0x7f221006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:15.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.255+0000 7f2231f6c640 1 --2- 172.21.15.161:0/241923373 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f221006b4d0 0x7f221006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:15.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.255+0000 7f22227fc640 1 -- 172.21.15.161:0/241923373 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7f22240ac940 con 0x7f222c14be70 2024-03-28T23:37:15.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.255+0000 7f2231f6c640 1 --2- 172.21.15.161:0/241923373 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f221006b4d0 0x7f221006d990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f2228009ac0 tx=0x7f2228008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:15.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.259+0000 7f22227fc640 1 -- 172.21.15.161:0/241923373 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f222404d030 con 0x7f222c14be70 2024-03-28T23:37:15.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.431+0000 7f22341d0640 1 -- 172.21.15.161:0/241923373 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f222c0c0ff0 con 0x7f221006b4d0 2024-03-28T23:37:15.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.432+0000 7f22227fc640 1 -- 172.21.15.161:0/241923373 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13592 (secure 0 0 0) 0x7f222c0c0ff0 con 0x7f221006b4d0 2024-03-28T23:37:15.433 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:37:15.433 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:37:15.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.434+0000 7f22341d0640 1 -- 172.21.15.161:0/241923373 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f221006b4d0 msgr2=0x7f221006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:15.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.434+0000 7f22341d0640 1 --2- 172.21.15.161:0/241923373 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f221006b4d0 0x7f221006d990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f2228009ac0 tx=0x7f2228008040 comp rx=0 tx=0).stop 2024-03-28T23:37:15.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.434+0000 7f22341d0640 1 -- 172.21.15.161:0/241923373 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f222c14be70 msgr2=0x7f222c1483c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:15.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.434+0000 7f22341d0640 1 --2- 172.21.15.161:0/241923373 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f222c14be70 0x7f222c1483c0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f2224009f40 tx=0x7f2224004a40 comp rx=0 tx=0).stop 2024-03-28T23:37:15.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.434+0000 7f22341d0640 1 -- 172.21.15.161:0/241923373 shutdown_connections 2024-03-28T23:37:15.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.434+0000 7f22341d0640 1 --2- 172.21.15.161:0/241923373 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f221006b4d0 0x7f221006d990 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:15.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.434+0000 7f22341d0640 1 --2- 172.21.15.161:0/241923373 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f222c14be70 0x7f222c1483c0 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:15.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.434+0000 7f22341d0640 1 --2- 172.21.15.161:0/241923373 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f222c0ab6e0 0x7f222c0c1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:15.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.434+0000 7f22341d0640 1 --2- 172.21.15.161:0/241923373 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f222c0aacd0 0x7f222c153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:15.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.434+0000 7f22341d0640 1 -- 172.21.15.161:0/241923373 >> 172.21.15.161:0/241923373 conn(0x7f222c0c6670 msgr2=0x7f222c151630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:15.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.434+0000 7f22341d0640 1 -- 172.21.15.161:0/241923373 shutdown_connections 2024-03-28T23:37:15.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.434+0000 7f22341d0640 1 -- 172.21.15.161:0/241923373 wait complete. 2024-03-28T23:37:15.445 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":389,"stamp":"2024-03-28T23:37:15.399908+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":119,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":266184,"kb_used_data":16736,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754708536,"statfs":{"total":773094113280,"available":772821540864,"internally_reserved":0,"allocated":17137664,"data_stored":9529728,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2410,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":-92,"num_read_kb":-94,"num_write":-46,"num_write_kb":-14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.257728"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":449,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-28T23:37:12.155372+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:37:12.155372+0000","last_peered":"2024-03-28T23:37:12.155372+0000","last_clean":"2024-03-28T23:37:12.155372+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:37:12.155372+0000","last_undegraded":"2024-03-28T23:37:12.155372+0000","last_fullsized":"2024-03-28T23:37:12.155372+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33324,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338516,"statfs":{"total":96636764160,"available":96602640384,"internally_reserved":0,"allocated":2191360,"data_stored":1633992,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574942,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574942,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574941,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574941,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574941,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33324,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338516,"statfs":{"total":96636764160,"available":96602640384,"internally_reserved":0,"allocated":2191360,"data_stored":1633992,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:37:15.446 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:37:15.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.547+0000 7f89e3c9d640 1 Processor -- start 2024-03-28T23:37:15.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.547+0000 7f89e3c9d640 1 -- start start 2024-03-28T23:37:15.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.548+0000 7f89e3c9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f89dc14be70 0x7f89dc0ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:15.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.548+0000 7f89e3c9d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f89dc0ab6e0 0x7f89dc0aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:15.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.548+0000 7f89e3c9d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89dc14c250 con 0x7f89dc06ce80 2024-03-28T23:37:15.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.548+0000 7f89e3c9d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f89dc1529c0 con 0x7f89dc14be70 2024-03-28T23:37:15.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.548+0000 7f89e3c9d640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f89dc0a9dc0 con 0x7f89dc0ab6e0 2024-03-28T23:37:15.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.548+0000 7f89e1a39640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f89dc14be70 0x7f89dc0ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:15.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.548+0000 7f89e223a640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f89dc06ce80 0x7f89dc1553b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41660/0 (socket says 172.21.15.161:41660) 2024-03-28T23:37:15.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.548+0000 7f89e223a640 1 -- 172.21.15.161:0/2190989341 learned_addr learned my addr 172.21.15.161:0/2190989341 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:15.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.548+0000 7f89e1a39640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f89dc14be70 0x7f89dc0ab1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55788/0 (socket says 172.21.15.161:55788) 2024-03-28T23:37:15.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.548+0000 7f89e1238640 1 --2- 172.21.15.161:0/2190989341 >> v2:172.21.15.170:3300/0 conn(0x7f89dc0ab6e0 0x7f89dc0aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:15.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.548+0000 7f89e1a39640 1 -- 172.21.15.161:0/2190989341 >> v2:172.21.15.170:3300/0 conn(0x7f89dc0ab6e0 msgr2=0x7f89dc0aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:15.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.548+0000 7f89e1a39640 1 --2- 172.21.15.161:0/2190989341 >> v2:172.21.15.170:3300/0 conn(0x7f89dc0ab6e0 0x7f89dc0aa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:15.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.549+0000 7f89e1a39640 1 -- 172.21.15.161:0/2190989341 >> v1:172.21.15.161:6789/0 conn(0x7f89dc06ce80 legacy=0x7f89dc1553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:15.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.549+0000 7f89e1a39640 1 -- 172.21.15.161:0/2190989341 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89dc0aa0a0 con 0x7f89dc14be70 2024-03-28T23:37:15.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.549+0000 7f89e1a39640 1 --2- 172.21.15.161:0/2190989341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89dc14be70 0x7f89dc0ab1a0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f89dc06f540 tx=0x7f89d802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd73941b6835b1f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:15.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.549+0000 7f89e0a37640 1 -- 172.21.15.161:0/2190989341 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d8039070 con 0x7f89dc14be70 2024-03-28T23:37:15.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.549+0000 7f89e0a37640 1 -- 172.21.15.161:0/2190989341 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89d802cbf0 con 0x7f89dc14be70 2024-03-28T23:37:15.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.549+0000 7f89e0a37640 1 -- 172.21.15.161:0/2190989341 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d8034440 con 0x7f89dc14be70 2024-03-28T23:37:15.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.550+0000 7f89e3c9d640 1 -- 172.21.15.161:0/2190989341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89dc14be70 msgr2=0x7f89dc0ab1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:15.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.550+0000 7f89e3c9d640 1 --2- 172.21.15.161:0/2190989341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89dc14be70 0x7f89dc0ab1a0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f89dc06f540 tx=0x7f89d802be40 comp rx=0 tx=0).stop 2024-03-28T23:37:15.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.550+0000 7f89e3c9d640 1 -- 172.21.15.161:0/2190989341 shutdown_connections 2024-03-28T23:37:15.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.550+0000 7f89e3c9d640 1 --2- 172.21.15.161:0/2190989341 >> v2:172.21.15.170:3300/0 conn(0x7f89dc0ab6e0 0x7f89dc0aa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:15.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.550+0000 7f89e3c9d640 1 --2- 172.21.15.161:0/2190989341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89dc14be70 0x7f89dc0ab1a0 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:15.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.550+0000 7f89e3c9d640 1 -- 172.21.15.161:0/2190989341 >> 172.21.15.161:0/2190989341 conn(0x7f89dc0c6670 msgr2=0x7f89dc0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:15.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.550+0000 7f89e3c9d640 1 -- 172.21.15.161:0/2190989341 shutdown_connections 2024-03-28T23:37:15.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.550+0000 7f89e3c9d640 1 -- 172.21.15.161:0/2190989341 wait complete. 2024-03-28T23:37:15.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.551+0000 7f89e3c9d640 1 Processor -- start 2024-03-28T23:37:15.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.551+0000 7f89e3c9d640 1 -- start start 2024-03-28T23:37:15.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.552+0000 7f89e3c9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89dc06ce80 0x7f89dc14c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:15.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.552+0000 7f89e3c9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89dc0ab6e0 0x7f89dc1e92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:15.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.552+0000 7f89e3c9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89dc14be70 0x7f89dc14ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:15.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.552+0000 7f89e3c9d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc06b2d0 con 0x7f89dc14be70 2024-03-28T23:37:15.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.552+0000 7f89e3c9d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc06b150 con 0x7f89dc06ce80 2024-03-28T23:37:15.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.552+0000 7f89e3c9d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f89dc0c1520 con 0x7f89dc0ab6e0 2024-03-28T23:37:15.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.552+0000 7f89e223a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89dc14be70 0x7f89dc14ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:15.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.552+0000 7f89e223a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89dc14be70 0x7f89dc14ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55790/0 (socket says 172.21.15.161:55790) 2024-03-28T23:37:15.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.552+0000 7f89e223a640 1 -- 172.21.15.161:0/3431912282 learned_addr learned my addr 172.21.15.161:0/3431912282 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:15.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.552+0000 7f89e1a39640 1 --2- 172.21.15.161:0/3431912282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89dc06ce80 0x7f89dc14c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:15.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.552+0000 7f89e1238640 1 --2- 172.21.15.161:0/3431912282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89dc0ab6e0 0x7f89dc1e92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:15.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.552+0000 7f89e223a640 1 -- 172.21.15.161:0/3431912282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89dc0ab6e0 msgr2=0x7f89dc1e92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:15.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.552+0000 7f89e223a640 1 --2- 172.21.15.161:0/3431912282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89dc0ab6e0 0x7f89dc1e92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:15.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.552+0000 7f89e223a640 1 -- 172.21.15.161:0/3431912282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89dc06ce80 msgr2=0x7f89dc14c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:15.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.552+0000 7f89e223a640 1 --2- 172.21.15.161:0/3431912282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89dc06ce80 0x7f89dc14c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:15.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.552+0000 7f89e223a640 1 -- 172.21.15.161:0/3431912282 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89dc14c9a0 con 0x7f89dc14be70 2024-03-28T23:37:15.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.552+0000 7f89e223a640 1 --2- 172.21.15.161:0/3431912282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89dc14be70 0x7f89dc14ece0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f89d400c3a0 tx=0x7f89d400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:15.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.553+0000 7f89d27fc640 1 -- 172.21.15.161:0/3431912282 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d4017070 con 0x7f89dc14be70 2024-03-28T23:37:15.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.553+0000 7f89d27fc640 1 -- 172.21.15.161:0/3431912282 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89d4005ce0 con 0x7f89dc14be70 2024-03-28T23:37:15.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.553+0000 7f89e3c9d640 1 -- 172.21.15.161:0/3431912282 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89dc1e97f0 con 0x7f89dc14be70 2024-03-28T23:37:15.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.553+0000 7f89d27fc640 1 -- 172.21.15.161:0/3431912282 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d40072d0 con 0x7f89dc14be70 2024-03-28T23:37:15.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.553+0000 7f89e3c9d640 1 -- 172.21.15.161:0/3431912282 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89dc14e020 con 0x7f89dc14be70 2024-03-28T23:37:15.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.554+0000 7f89e3c9d640 1 -- 172.21.15.161:0/3431912282 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89dc071300 con 0x7f89dc14be70 2024-03-28T23:37:15.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.554+0000 7f89d27fc640 1 -- 172.21.15.161:0/3431912282 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f89d401a070 con 0x7f89dc14be70 2024-03-28T23:37:15.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.554+0000 7f89d27fc640 1 --2- 172.21.15.161:0/3431912282 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f89c006b4d0 0x7f89c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:15.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.554+0000 7f89d27fc640 1 -- 172.21.15.161:0/3431912282 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7f89d4089390 con 0x7f89dc14be70 2024-03-28T23:37:15.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.554+0000 7f89e1a39640 1 --2- 172.21.15.161:0/3431912282 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f89c006b4d0 0x7f89c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:15.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.555+0000 7f89e1a39640 1 --2- 172.21.15.161:0/3431912282 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f89c006b4d0 0x7f89c006d990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f89d8036000 tx=0x7f89d8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:15.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.559+0000 7f89d27fc640 1 -- 172.21.15.161:0/3431912282 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f89dc071300 con 0x7f89dc14be70 2024-03-28T23:37:15.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.725+0000 7f89e3c9d640 1 -- 172.21.15.161:0/3431912282 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f89dc04f4b0 con 0x7f89c006b4d0 2024-03-28T23:37:15.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.726+0000 7f89d27fc640 1 -- 172.21.15.161:0/3431912282 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13592 (secure 0 0 0) 0x7f89dc04f4b0 con 0x7f89c006b4d0 2024-03-28T23:37:15.727 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:37:15.727 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:37:15.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.728+0000 7f89e3c9d640 1 -- 172.21.15.161:0/3431912282 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f89c006b4d0 msgr2=0x7f89c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:15.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.728+0000 7f89e3c9d640 1 --2- 172.21.15.161:0/3431912282 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f89c006b4d0 0x7f89c006d990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f89d8036000 tx=0x7f89d8030000 comp rx=0 tx=0).stop 2024-03-28T23:37:15.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.728+0000 7f89e3c9d640 1 -- 172.21.15.161:0/3431912282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89dc14be70 msgr2=0x7f89dc14ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:15.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.728+0000 7f89e3c9d640 1 --2- 172.21.15.161:0/3431912282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89dc14be70 0x7f89dc14ece0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f89d400c3a0 tx=0x7f89d400c860 comp rx=0 tx=0).stop 2024-03-28T23:37:15.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.728+0000 7f89e3c9d640 1 -- 172.21.15.161:0/3431912282 shutdown_connections 2024-03-28T23:37:15.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.728+0000 7f89e3c9d640 1 --2- 172.21.15.161:0/3431912282 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f89c006b4d0 0x7f89c006d990 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:15.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.728+0000 7f89e3c9d640 1 --2- 172.21.15.161:0/3431912282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89dc14be70 0x7f89dc14ece0 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:15.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.728+0000 7f89e3c9d640 1 --2- 172.21.15.161:0/3431912282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89dc0ab6e0 0x7f89dc1e92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:15.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.728+0000 7f89e3c9d640 1 --2- 172.21.15.161:0/3431912282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89dc06ce80 0x7f89dc14c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:15.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.728+0000 7f89e3c9d640 1 -- 172.21.15.161:0/3431912282 >> 172.21.15.161:0/3431912282 conn(0x7f89dc0c6670 msgr2=0x7f89dc0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:15.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.728+0000 7f89e3c9d640 1 -- 172.21.15.161:0/3431912282 shutdown_connections 2024-03-28T23:37:15.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.728+0000 7f89e3c9d640 1 -- 172.21.15.161:0/3431912282 wait complete. 2024-03-28T23:37:15.739 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":389,"stamp":"2024-03-28T23:37:15.399908+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":119,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":266184,"kb_used_data":16736,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754708536,"statfs":{"total":773094113280,"available":772821540864,"internally_reserved":0,"allocated":17137664,"data_stored":9529728,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2410,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":-92,"num_read_kb":-94,"num_write":-46,"num_write_kb":-14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.257728"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":449,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-28T23:37:12.155372+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:37:12.155372+0000","last_peered":"2024-03-28T23:37:12.155372+0000","last_clean":"2024-03-28T23:37:12.155372+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:37:12.155372+0000","last_undegraded":"2024-03-28T23:37:12.155372+0000","last_fullsized":"2024-03-28T23:37:12.155372+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33324,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338516,"statfs":{"total":96636764160,"available":96602640384,"internally_reserved":0,"allocated":2191360,"data_stored":1633992,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574942,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574942,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574941,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574941,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574941,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33324,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338516,"statfs":{"total":96636764160,"available":96602640384,"internally_reserved":0,"allocated":2191360,"data_stored":1633992,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:37:15.739 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-28T23:37:15.740 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:37:15.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.841+0000 7fdd80e8f640 1 Processor -- start 2024-03-28T23:37:15.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.841+0000 7fdd80e8f640 1 -- start start 2024-03-28T23:37:15.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.842+0000 7fdd80e8f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd7c0c3770 0x7fdd7c0c1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:15.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.842+0000 7fdd80e8f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdd7c0c2300 0x7fdd7c15fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:15.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.842+0000 7fdd80e8f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd7c0c3bd0 con 0x7fdd7c0c3390 2024-03-28T23:37:15.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.842+0000 7fdd80e8f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fdd7c155e80 con 0x7fdd7c0c2300 2024-03-28T23:37:15.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.842+0000 7fdd80e8f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdd7c0a8410 con 0x7fdd7c0c3770 2024-03-28T23:37:15.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.842+0000 7fdd7ad9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdd7c0c2300 0x7fdd7c15fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:15.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.842+0000 7fdd7ad9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdd7c0c2300 0x7fdd7c15fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55802/0 (socket says 172.21.15.161:55802) 2024-03-28T23:37:15.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.842+0000 7fdd7ad9d640 1 -- 172.21.15.161:0/4278768040 learned_addr learned my addr 172.21.15.161:0/4278768040 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:15.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.842+0000 7fdd79d9b640 1 --2- 172.21.15.161:0/4278768040 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd7c0c3770 0x7fdd7c0c1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:15.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.843+0000 7fdd7ad9d640 1 -- 172.21.15.161:0/4278768040 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd7c0c3770 msgr2=0x7fdd7c0c1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:15.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.843+0000 7fdd7ad9d640 1 --2- 172.21.15.161:0/4278768040 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd7c0c3770 0x7fdd7c0c1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:15.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.843+0000 7fdd7ad9d640 1 -- 172.21.15.161:0/4278768040 >> v1:172.21.15.170:6789/0 conn(0x7fdd7c0c3390 legacy=0x7fdd7c0a7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:15.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.843+0000 7fdd7ad9d640 1 -- 172.21.15.161:0/4278768040 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd7c155270 con 0x7fdd7c0c2300 2024-03-28T23:37:15.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.843+0000 7fdd7ad9d640 1 --2- 172.21.15.161:0/4278768040 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd7c0c2300 0x7fdd7c15fae0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fdd74009870 tx=0x7fdd7402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c718f29cd15f9373 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:15.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.843+0000 7fdd7959a640 1 -- 172.21.15.161:0/4278768040 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd740317c0 con 0x7fdd7c0c2300 2024-03-28T23:37:15.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.843+0000 7fdd7959a640 1 -- 172.21.15.161:0/4278768040 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd74037070 con 0x7fdd7c0c2300 2024-03-28T23:37:15.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.843+0000 7fdd7959a640 1 -- 172.21.15.161:0/4278768040 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd74031b80 con 0x7fdd7c0c2300 2024-03-28T23:37:15.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.844+0000 7fdd80e8f640 1 -- 172.21.15.161:0/4278768040 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd7c0c2300 msgr2=0x7fdd7c15fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:15.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.844+0000 7fdd80e8f640 1 --2- 172.21.15.161:0/4278768040 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd7c0c2300 0x7fdd7c15fae0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fdd74009870 tx=0x7fdd7402b860 comp rx=0 tx=0).stop 2024-03-28T23:37:15.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.844+0000 7fdd80e8f640 1 -- 172.21.15.161:0/4278768040 shutdown_connections 2024-03-28T23:37:15.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.844+0000 7fdd80e8f640 1 --2- 172.21.15.161:0/4278768040 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd7c0c2300 0x7fdd7c15fae0 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:15.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.844+0000 7fdd80e8f640 1 --2- 172.21.15.161:0/4278768040 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd7c0c3770 0x7fdd7c0c1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:15.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.844+0000 7fdd80e8f640 1 -- 172.21.15.161:0/4278768040 >> 172.21.15.161:0/4278768040 conn(0x7fdd7c0aa740 msgr2=0x7fdd7c0aab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:15.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.844+0000 7fdd80e8f640 1 -- 172.21.15.161:0/4278768040 shutdown_connections 2024-03-28T23:37:15.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.844+0000 7fdd80e8f640 1 -- 172.21.15.161:0/4278768040 wait complete. 2024-03-28T23:37:15.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.845+0000 7fdd80e8f640 1 Processor -- start 2024-03-28T23:37:15.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.845+0000 7fdd80e8f640 1 -- start start 2024-03-28T23:37:15.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.846+0000 7fdd80e8f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd7c0c2300 0x7fdd7c0c7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:15.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.846+0000 7fdd80e8f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd7c0c3390 0x7fdd7c14cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:15.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.846+0000 7fdd80e8f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd7c0c3770 0x7fdd7c149090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:15.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.846+0000 7fdd80e8f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c155b10 con 0x7fdd7c0c2300 2024-03-28T23:37:15.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.846+0000 7fdd80e8f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c155830 con 0x7fdd7c0c3390 2024-03-28T23:37:15.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.846+0000 7fdd80e8f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fdd7c152a80 con 0x7fdd7c0c3770 2024-03-28T23:37:15.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.846+0000 7fdd7a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd7c0c2300 0x7fdd7c0c7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:15.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.846+0000 7fdd7a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd7c0c2300 0x7fdd7c0c7ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55812/0 (socket says 172.21.15.161:55812) 2024-03-28T23:37:15.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.846+0000 7fdd7a59c640 1 -- 172.21.15.161:0/2947046924 learned_addr learned my addr 172.21.15.161:0/2947046924 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:15.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.846+0000 7fdd7ad9d640 1 --2- 172.21.15.161:0/2947046924 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd7c0c3770 0x7fdd7c149090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:15.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.846+0000 7fdd79d9b640 1 --2- 172.21.15.161:0/2947046924 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd7c0c3390 0x7fdd7c14cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:15.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.846+0000 7fdd7a59c640 1 -- 172.21.15.161:0/2947046924 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd7c0c3770 msgr2=0x7fdd7c149090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:15.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.846+0000 7fdd7a59c640 1 --2- 172.21.15.161:0/2947046924 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd7c0c3770 0x7fdd7c149090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:15.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.846+0000 7fdd7a59c640 1 -- 172.21.15.161:0/2947046924 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd7c0c3390 msgr2=0x7fdd7c14cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:15.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.846+0000 7fdd7a59c640 1 --2- 172.21.15.161:0/2947046924 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd7c0c3390 0x7fdd7c14cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:15.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.846+0000 7fdd7a59c640 1 -- 172.21.15.161:0/2947046924 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd7c06fa20 con 0x7fdd7c0c2300 2024-03-28T23:37:15.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.846+0000 7fdd7ad9d640 1 --2- 172.21.15.161:0/2947046924 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd7c0c3770 0x7fdd7c149090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:37:15.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.847+0000 7fdd7a59c640 1 --2- 172.21.15.161:0/2947046924 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd7c0c2300 0x7fdd7c0c7ee0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c00c7c0 tx=0x7fdd6c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:15.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.847+0000 7fdd6affd640 1 -- 172.21.15.161:0/2947046924 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c016020 con 0x7fdd7c0c2300 2024-03-28T23:37:15.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.847+0000 7fdd6affd640 1 -- 172.21.15.161:0/2947046924 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd6c00aa50 con 0x7fdd7c0c2300 2024-03-28T23:37:15.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.847+0000 7fdd80e8f640 1 -- 172.21.15.161:0/2947046924 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd7c06f740 con 0x7fdd7c0c2300 2024-03-28T23:37:15.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.847+0000 7fdd6affd640 1 -- 172.21.15.161:0/2947046924 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c012420 con 0x7fdd7c0c2300 2024-03-28T23:37:15.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.847+0000 7fdd80e8f640 1 -- 172.21.15.161:0/2947046924 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd7c06d6c0 con 0x7fdd7c0c2300 2024-03-28T23:37:15.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.848+0000 7fdd80e8f640 1 -- 172.21.15.161:0/2947046924 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd7c04f4b0 con 0x7fdd7c0c2300 2024-03-28T23:37:15.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.848+0000 7fdd6affd640 1 -- 172.21.15.161:0/2947046924 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdd6c019020 con 0x7fdd7c0c2300 2024-03-28T23:37:15.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.848+0000 7fdd6affd640 1 --2- 172.21.15.161:0/2947046924 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdd5c06b4d0 0x7fdd5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:15.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.849+0000 7fdd6affd640 1 -- 172.21.15.161:0/2947046924 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7fdd6c08a2d0 con 0x7fdd7c0c2300 2024-03-28T23:37:15.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.849+0000 7fdd79d9b640 1 --2- 172.21.15.161:0/2947046924 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdd5c06b4d0 0x7fdd5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:15.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.849+0000 7fdd79d9b640 1 --2- 172.21.15.161:0/2947046924 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdd5c06b4d0 0x7fdd5c06d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fdd70006fd0 tx=0x7fdd70008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:15.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:15.853+0000 7fdd6affd640 1 -- 172.21.15.161:0/2947046924 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdd6c020030 con 0x7fdd7c0c2300 2024-03-28T23:37:16.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.021+0000 7fdd80e8f640 1 -- 172.21.15.161:0/2947046924 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdd7c073410 con 0x7fdd5c06b4d0 2024-03-28T23:37:16.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.022+0000 7fdd6affd640 1 -- 172.21.15.161:0/2947046924 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13592 (secure 0 0 0) 0x7fdd7c073410 con 0x7fdd5c06b4d0 2024-03-28T23:37:16.022 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:37:16.023 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:37:16.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.024+0000 7fdd80e8f640 1 -- 172.21.15.161:0/2947046924 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdd5c06b4d0 msgr2=0x7fdd5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:16.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.024+0000 7fdd80e8f640 1 --2- 172.21.15.161:0/2947046924 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdd5c06b4d0 0x7fdd5c06d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fdd70006fd0 tx=0x7fdd70008040 comp rx=0 tx=0).stop 2024-03-28T23:37:16.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.024+0000 7fdd80e8f640 1 -- 172.21.15.161:0/2947046924 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd7c0c2300 msgr2=0x7fdd7c0c7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:16.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.024+0000 7fdd80e8f640 1 --2- 172.21.15.161:0/2947046924 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd7c0c2300 0x7fdd7c0c7ee0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c00c7c0 tx=0x7fdd6c00cc80 comp rx=0 tx=0).stop 2024-03-28T23:37:16.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.024+0000 7fdd80e8f640 1 -- 172.21.15.161:0/2947046924 shutdown_connections 2024-03-28T23:37:16.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.024+0000 7fdd80e8f640 1 --2- 172.21.15.161:0/2947046924 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fdd5c06b4d0 0x7fdd5c06d990 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:16.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.024+0000 7fdd80e8f640 1 --2- 172.21.15.161:0/2947046924 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd7c0c3770 0x7fdd7c149090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:16.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.024+0000 7fdd80e8f640 1 --2- 172.21.15.161:0/2947046924 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd7c0c3390 0x7fdd7c14cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:16.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.024+0000 7fdd80e8f640 1 --2- 172.21.15.161:0/2947046924 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd7c0c2300 0x7fdd7c0c7ee0 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:16.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.024+0000 7fdd80e8f640 1 -- 172.21.15.161:0/2947046924 >> 172.21.15.161:0/2947046924 conn(0x7fdd7c0aa740 msgr2=0x7fdd7c1502a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:16.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.024+0000 7fdd80e8f640 1 -- 172.21.15.161:0/2947046924 shutdown_connections 2024-03-28T23:37:16.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.024+0000 7fdd80e8f640 1 -- 172.21.15.161:0/2947046924 wait complete. 2024-03-28T23:37:16.035 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":389,"stamp":"2024-03-28T23:37:15.399908+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":119,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":266184,"kb_used_data":16736,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754708536,"statfs":{"total":773094113280,"available":772821540864,"internally_reserved":0,"allocated":17137664,"data_stored":9529728,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2410,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":-92,"num_read_kb":-94,"num_write":-46,"num_write_kb":-14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.257728"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":449,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-28T23:37:12.155372+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:37:12.155372+0000","last_peered":"2024-03-28T23:37:12.155372+0000","last_clean":"2024-03-28T23:37:12.155372+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:37:12.155372+0000","last_undegraded":"2024-03-28T23:37:12.155372+0000","last_fullsized":"2024-03-28T23:37:12.155372+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33324,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338516,"statfs":{"total":96636764160,"available":96602640384,"internally_reserved":0,"allocated":2191360,"data_stored":1633992,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574942,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574942,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574941,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574941,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574941,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33324,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338516,"statfs":{"total":96636764160,"available":96602640384,"internally_reserved":0,"allocated":2191360,"data_stored":1633992,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:37:16.035 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T23:37:16.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.136+0000 7f28aecfb640 1 Processor -- start 2024-03-28T23:37:16.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.136+0000 7f28aecfb640 1 -- start start 2024-03-28T23:37:16.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.137+0000 7f28aecfb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f28a806e720 0x7f28a806c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:16.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.137+0000 7f28aecfb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28a806f650 con 0x7f28a806c160 2024-03-28T23:37:16.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.137+0000 7f28aecfb640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28a806f370 con 0x7f28a80c1cf0 2024-03-28T23:37:16.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.137+0000 7f28aecfb640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f28a806e290 con 0x7f28a806e720 2024-03-28T23:37:16.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.137+0000 7f28a7fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f28a806e720 0x7f28a806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:16.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.137+0000 7f28a7fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f28a806e720 0x7f28a806c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55818/0 (socket says 172.21.15.161:55818) 2024-03-28T23:37:16.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.137+0000 7f28aca97640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f28a806c160 0x7f28a8150560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41672/0 (socket says 172.21.15.161:41672) 2024-03-28T23:37:16.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.137+0000 7f28a7fff640 1 -- 172.21.15.161:0/2693885667 learned_addr learned my addr 172.21.15.161:0/2693885667 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:16.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.137+0000 7f28a7fff640 1 -- 172.21.15.161:0/2693885667 >> v1:172.21.15.170:6789/0 conn(0x7f28a80c1cf0 legacy=0x7f28a815d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:16.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.137+0000 7f28a7fff640 1 -- 172.21.15.161:0/2693885667 >> v1:172.21.15.161:6789/0 conn(0x7f28a806c160 legacy=0x7f28a8150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:16.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.137+0000 7f28a7fff640 1 -- 172.21.15.161:0/2693885667 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28a806dc40 con 0x7f28a806e720 2024-03-28T23:37:16.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.137+0000 7f28a77fe640 1 -- 172.21.15.161:0/2693885667 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2974402039 0 0) 0x7f28a806f650 con 0x7f28a806c160 2024-03-28T23:37:16.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.138+0000 7f28a7fff640 1 --2- 172.21.15.161:0/2693885667 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28a806e720 0x7f28a806c540 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f2894009870 tx=0x7f289402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b22a0efe85d75502 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:16.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.138+0000 7f28a77fe640 1 -- 172.21.15.161:0/2693885667 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2894034070 con 0x7f28a806e720 2024-03-28T23:37:16.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.138+0000 7f28a77fe640 1 -- 172.21.15.161:0/2693885667 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2894030860 con 0x7f28a806e720 2024-03-28T23:37:16.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.138+0000 7f28a77fe640 1 -- 172.21.15.161:0/2693885667 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2894030c30 con 0x7f28a806e720 2024-03-28T23:37:16.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.139+0000 7f28aecfb640 1 -- 172.21.15.161:0/2693885667 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28a806e720 msgr2=0x7f28a806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:16.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.139+0000 7f28aecfb640 1 --2- 172.21.15.161:0/2693885667 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28a806e720 0x7f28a806c540 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f2894009870 tx=0x7f289402b9a0 comp rx=0 tx=0).stop 2024-03-28T23:37:16.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.139+0000 7f28aecfb640 1 -- 172.21.15.161:0/2693885667 shutdown_connections 2024-03-28T23:37:16.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.139+0000 7f28aecfb640 1 --2- 172.21.15.161:0/2693885667 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28a806e720 0x7f28a806c540 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:16.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.139+0000 7f28aecfb640 1 -- 172.21.15.161:0/2693885667 >> 172.21.15.161:0/2693885667 conn(0x7f28a80c3be0 msgr2=0x7f28a80c3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:16.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.139+0000 7f28aecfb640 1 -- 172.21.15.161:0/2693885667 shutdown_connections 2024-03-28T23:37:16.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.139+0000 7f28aecfb640 1 -- 172.21.15.161:0/2693885667 wait complete. 2024-03-28T23:37:16.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.140+0000 7f28aecfb640 1 Processor -- start 2024-03-28T23:37:16.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.140+0000 7f28aecfb640 1 -- start start 2024-03-28T23:37:16.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.140+0000 7f28aecfb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28a806c160 0x7f28a80c0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:16.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.140+0000 7f28aecfb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28a806e720 0x7f28a8152990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:16.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.141+0000 7f28aecfb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28a80c1cf0 0x7f28a8152ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:16.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.141+0000 7f28aecfb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f28a80c2390 con 0x7f28a806c160 2024-03-28T23:37:16.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.141+0000 7f28aecfb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f28a80c2180 con 0x7f28a806e720 2024-03-28T23:37:16.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.141+0000 7f28aecfb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f28a806f000 con 0x7f28a80c1cf0 2024-03-28T23:37:16.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.141+0000 7f28aca97640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28a806c160 0x7f28a80c0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:16.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.141+0000 7f28aca97640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28a806c160 0x7f28a80c0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55834/0 (socket says 172.21.15.161:55834) 2024-03-28T23:37:16.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.141+0000 7f28aca97640 1 -- 172.21.15.161:0/2690984953 learned_addr learned my addr 172.21.15.161:0/2690984953 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:16.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.141+0000 7f28a7fff640 1 --2- 172.21.15.161:0/2690984953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28a806e720 0x7f28a8152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:16.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.141+0000 7f28ad298640 1 --2- 172.21.15.161:0/2690984953 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28a80c1cf0 0x7f28a8152ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:16.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.141+0000 7f28aca97640 1 -- 172.21.15.161:0/2690984953 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28a80c1cf0 msgr2=0x7f28a8152ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:16.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.141+0000 7f28aca97640 1 --2- 172.21.15.161:0/2690984953 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28a80c1cf0 0x7f28a8152ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:16.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.141+0000 7f28aca97640 1 -- 172.21.15.161:0/2690984953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28a806e720 msgr2=0x7f28a8152990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:16.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.141+0000 7f28aca97640 1 --2- 172.21.15.161:0/2690984953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28a806e720 0x7f28a8152990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:16.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.141+0000 7f28aca97640 1 -- 172.21.15.161:0/2690984953 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28a80ab350 con 0x7f28a806c160 2024-03-28T23:37:16.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.141+0000 7f28aca97640 1 --2- 172.21.15.161:0/2690984953 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28a806c160 0x7f28a80c0780 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f289c00c3f0 tx=0x7f289c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:16.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.141+0000 7f28a57fa640 1 -- 172.21.15.161:0/2690984953 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289c0074e0 con 0x7f28a806c160 2024-03-28T23:37:16.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.141+0000 7f28a57fa640 1 -- 172.21.15.161:0/2690984953 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f289c0047e0 con 0x7f28a806c160 2024-03-28T23:37:16.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.141+0000 7f28aecfb640 1 -- 172.21.15.161:0/2690984953 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28a80ab630 con 0x7f28a806c160 2024-03-28T23:37:16.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.141+0000 7f28a57fa640 1 -- 172.21.15.161:0/2690984953 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289c004bd0 con 0x7f28a806c160 2024-03-28T23:37:16.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.141+0000 7f28aecfb640 1 -- 172.21.15.161:0/2690984953 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28a8152190 con 0x7f28a806c160 2024-03-28T23:37:16.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.142+0000 7f28aecfb640 1 -- 172.21.15.161:0/2690984953 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28a804f4b0 con 0x7f28a806c160 2024-03-28T23:37:16.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.143+0000 7f28a57fa640 1 -- 172.21.15.161:0/2690984953 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f289c011070 con 0x7f28a806c160 2024-03-28T23:37:16.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.143+0000 7f28a57fa640 1 --2- 172.21.15.161:0/2690984953 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f288c06b4d0 0x7f288c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:16.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.143+0000 7f28a57fa640 1 -- 172.21.15.161:0/2690984953 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7f289c089d20 con 0x7f28a806c160 2024-03-28T23:37:16.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.143+0000 7f28a7fff640 1 --2- 172.21.15.161:0/2690984953 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f288c06b4d0 0x7f288c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:16.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.144+0000 7f28a7fff640 1 --2- 172.21.15.161:0/2690984953 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f288c06b4d0 0x7f288c06d990 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f2894036000 tx=0x7f2894008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:16.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.148+0000 7f28a57fa640 1 -- 172.21.15.161:0/2690984953 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f289c058510 con 0x7f28a806c160 2024-03-28T23:37:16.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.321+0000 7f28aecfb640 1 -- 172.21.15.161:0/2690984953 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f28a806f650 con 0x7f28a806c160 2024-03-28T23:37:16.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.322+0000 7f28a57fa640 1 -- 172.21.15.161:0/2690984953 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v168) ==== 74+0+13999 (secure 0 0 0) 0x7f289c05c5b0 con 0x7f28a806c160 2024-03-28T23:37:16.322 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:37:16.322 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":168,"fsid":"cc975a39-1de3-40a7-9dc0-0e645d96eff8","created":"2024-03-28T23:29:29.129460+0000","modified":"2024-03-28T23:37:12.147743+0000","last_up_change":"2024-03-28T23:29:39.231019+0000","last_in_change":"2024-03-28T23:29:31.460946+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T23:29:39.343393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b59decd-6f16-4eea-a2d3-1e0618870d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6817","nonce":4264782123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6819","nonce":4264782123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6823","nonce":4264782123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4264782123},{"type":"v1","addr":"172.21.15.161:6821","nonce":4264782123}]},"public_addr":"172.21.15.161:6817/4264782123","cluster_addr":"172.21.15.161:6819/4264782123","heartbeat_back_addr":"172.21.15.161:6823/4264782123","heartbeat_front_addr":"172.21.15.161:6821/4264782123","state":["exists","up"]},{"osd":1,"uuid":"ed6fe366-2832-4857-98f7-66f63a7b0f27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6801","nonce":3551918247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6803","nonce":3551918247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6807","nonce":3551918247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3551918247},{"type":"v1","addr":"172.21.15.161:6805","nonce":3551918247}]},"public_addr":"172.21.15.161:6801/3551918247","cluster_addr":"172.21.15.161:6803/3551918247","heartbeat_back_addr":"172.21.15.161:6807/3551918247","heartbeat_front_addr":"172.21.15.161:6805/3551918247","state":["exists","up"]},{"osd":2,"uuid":"91d1af65-9bf9-4271-98ac-b66ba5254e8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6809","nonce":3120803198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6811","nonce":3120803198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6815","nonce":3120803198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3120803198},{"type":"v1","addr":"172.21.15.161:6813","nonce":3120803198}]},"public_addr":"172.21.15.161:6809/3120803198","cluster_addr":"172.21.15.161:6811/3120803198","heartbeat_back_addr":"172.21.15.161:6815/3120803198","heartbeat_front_addr":"172.21.15.161:6813/3120803198","state":["exists","up"]},{"osd":3,"uuid":"0a6cc81f-c226-4929-9405-3efb9c0e0723","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6825","nonce":3880429893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6827","nonce":3880429893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6831","nonce":3880429893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3880429893},{"type":"v1","addr":"172.21.15.161:6829","nonce":3880429893}]},"public_addr":"172.21.15.161:6825/3880429893","cluster_addr":"172.21.15.161:6827/3880429893","heartbeat_back_addr":"172.21.15.161:6831/3880429893","heartbeat_front_addr":"172.21.15.161:6829/3880429893","state":["exists","up"]},{"osd":4,"uuid":"268e2360-1999-475b-b35f-4e7c443997d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6801","nonce":1557068995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6803","nonce":1557068995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6807","nonce":1557068995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1557068995},{"type":"v1","addr":"172.21.15.170:6805","nonce":1557068995}]},"public_addr":"172.21.15.170:6801/1557068995","cluster_addr":"172.21.15.170:6803/1557068995","heartbeat_back_addr":"172.21.15.170:6807/1557068995","heartbeat_front_addr":"172.21.15.170:6805/1557068995","state":["exists","up"]},{"osd":5,"uuid":"08c1ede6-ff85-4dc2-96b3-9a04170152ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6825","nonce":211299856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6827","nonce":211299856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6831","nonce":211299856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":211299856},{"type":"v1","addr":"172.21.15.170:6829","nonce":211299856}]},"public_addr":"172.21.15.170:6825/211299856","cluster_addr":"172.21.15.170:6827/211299856","heartbeat_back_addr":"172.21.15.170:6831/211299856","heartbeat_front_addr":"172.21.15.170:6829/211299856","state":["exists","up"]},{"osd":6,"uuid":"9e6e8b34-9805-4653-944f-e8b07dbcfadb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6809","nonce":844763145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6811","nonce":844763145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6815","nonce":844763145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":844763145},{"type":"v1","addr":"172.21.15.170:6813","nonce":844763145}]},"public_addr":"172.21.15.170:6809/844763145","cluster_addr":"172.21.15.170:6811/844763145","heartbeat_back_addr":"172.21.15.170:6815/844763145","heartbeat_front_addr":"172.21.15.170:6813/844763145","state":["exists","up"]},{"osd":7,"uuid":"fd44cf2b-243a-4e71-9d85-bf2fdb1adc39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6817","nonce":3081832568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6819","nonce":3081832568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6823","nonce":3081832568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":3081832568},{"type":"v1","addr":"172.21.15.170:6821","nonce":3081832568}]},"public_addr":"172.21.15.170:6817/3081832568","cluster_addr":"172.21.15.170:6819/3081832568","heartbeat_back_addr":"172.21.15.170:6823/3081832568","heartbeat_front_addr":"172.21.15.170:6821/3081832568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.610011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.552707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.595708+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.678536+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.485963+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.607752+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.616922+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T23:29:36.647241+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6837/789976100":"2024-03-29T23:37:09.139744+0000","172.21.15.161:6834/4026332703":"2024-03-29T23:37:03.868933+0000","172.21.15.170:6834/2996092857":"2024-03-29T23:37:02.855183+0000","172.21.15.161:6836/4026332703":"2024-03-29T23:37:03.868933+0000","172.21.15.170:6832/2996092857":"2024-03-29T23:37:02.855183+0000","172.21.15.161:6837/3236742821":"2024-03-29T23:36:17.431214+0000","172.21.15.161:6835/3236742821":"2024-03-29T23:36:17.431214+0000","172.21.15.170:6834/1393137291":"2024-03-29T23:36:14.412210+0000","172.21.15.170:6832/1393137291":"2024-03-29T23:36:14.412210+0000","172.21.15.161:6835/789976100":"2024-03-29T23:37:09.139744+0000","172.21.15.161:6837/2391019771":"2024-03-29T23:36:13.968386+0000","172.21.15.161:6835/2391019771":"2024-03-29T23:36:13.968386+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T23:37:16.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.324+0000 7f28aecfb640 1 -- 172.21.15.161:0/2690984953 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f288c06b4d0 msgr2=0x7f288c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:16.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.324+0000 7f28aecfb640 1 --2- 172.21.15.161:0/2690984953 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f288c06b4d0 0x7f288c06d990 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f2894036000 tx=0x7f2894008030 comp rx=0 tx=0).stop 2024-03-28T23:37:16.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.324+0000 7f28aecfb640 1 -- 172.21.15.161:0/2690984953 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28a806c160 msgr2=0x7f28a80c0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:16.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.324+0000 7f28aecfb640 1 --2- 172.21.15.161:0/2690984953 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28a806c160 0x7f28a80c0780 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f289c00c3f0 tx=0x7f289c00c8b0 comp rx=0 tx=0).stop 2024-03-28T23:37:16.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.324+0000 7f28aecfb640 1 -- 172.21.15.161:0/2690984953 shutdown_connections 2024-03-28T23:37:16.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.324+0000 7f28aecfb640 1 --2- 172.21.15.161:0/2690984953 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f288c06b4d0 0x7f288c06d990 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:16.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.324+0000 7f28aecfb640 1 --2- 172.21.15.161:0/2690984953 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28a80c1cf0 0x7f28a8152ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:16.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.324+0000 7f28aecfb640 1 --2- 172.21.15.161:0/2690984953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28a806e720 0x7f28a8152990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:16.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.324+0000 7f28aecfb640 1 --2- 172.21.15.161:0/2690984953 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28a806c160 0x7f28a80c0780 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:16.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.324+0000 7f28aecfb640 1 -- 172.21.15.161:0/2690984953 >> 172.21.15.161:0/2690984953 conn(0x7f28a80c3be0 msgr2=0x7f28a80c5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:16.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.324+0000 7f28aecfb640 1 -- 172.21.15.161:0/2690984953 shutdown_connections 2024-03-28T23:37:16.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:16.324+0000 7f28aecfb640 1 -- 172.21.15.161:0/2690984953 wait complete. 2024-03-28T23:37:17.335 INFO:tasks.ceph:Scrubbing osd.0 2024-03-28T23:37:17.336 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-28T23:37:17.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.453+0000 7ff7799e2640 1 Processor -- start 2024-03-28T23:37:17.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.453+0000 7ff7799e2640 1 -- start start 2024-03-28T23:37:17.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.454+0000 7ff7799e2640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff774152740 0x7ff77406b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:17.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.454+0000 7ff7799e2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff774155d70 0x7ff7740c9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:17.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.454+0000 7ff7799e2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7740c53e0 0x7ff7740c7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:17.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.454+0000 7ff7799e2640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff7740c9980 con 0x7ff774155d70 2024-03-28T23:37:17.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.454+0000 7ff7799e2640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7ff7741561d0 con 0x7ff774152740 2024-03-28T23:37:17.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.454+0000 7ff7799e2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff77406bb70 con 0x7ff7740c53e0 2024-03-28T23:37:17.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.454+0000 7ff7727fc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff774155d70 0x7ff7740c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:17.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.454+0000 7ff7727fc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff774155d70 0x7ff7740c9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55848/0 (socket says 172.21.15.161:55848) 2024-03-28T23:37:17.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.454+0000 7ff7727fc640 1 -- 172.21.15.161:0/1559051361 learned_addr learned my addr 172.21.15.161:0/1559051361 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:17.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.454+0000 7ff772ffd640 1 --2- 172.21.15.161:0/1559051361 >> v2:172.21.15.170:3300/0 conn(0x7ff774152740 0x7ff77406b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:17.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.454+0000 7ff7737fe640 1 --2- 172.21.15.161:0/1559051361 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7740c53e0 0x7ff7740c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:17.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.454+0000 7ff7727fc640 1 -- 172.21.15.161:0/1559051361 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7740c53e0 msgr2=0x7ff7740c7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:17.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.454+0000 7ff7727fc640 1 --2- 172.21.15.161:0/1559051361 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7740c53e0 0x7ff7740c7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:17.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.455+0000 7ff7727fc640 1 -- 172.21.15.161:0/1559051361 >> v2:172.21.15.170:3300/0 conn(0x7ff774152740 msgr2=0x7ff77406b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:17.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.455+0000 7ff7727fc640 1 --2- 172.21.15.161:0/1559051361 >> v2:172.21.15.170:3300/0 conn(0x7ff774152740 0x7ff77406b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:17.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.455+0000 7ff7727fc640 1 -- 172.21.15.161:0/1559051361 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff77406be50 con 0x7ff774155d70 2024-03-28T23:37:17.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.455+0000 7ff7727fc640 1 --2- 172.21.15.161:0/1559051361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff774155d70 0x7ff7740c9440 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7ff77408ccb0 tx=0x7ff76802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b8d465902115aa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:17.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.455+0000 7ff771ffb640 1 -- 172.21.15.161:0/1559051361 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff768037070 con 0x7ff774155d70 2024-03-28T23:37:17.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.455+0000 7ff771ffb640 1 -- 172.21.15.161:0/1559051361 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff768004230 con 0x7ff774155d70 2024-03-28T23:37:17.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.455+0000 7ff771ffb640 1 -- 172.21.15.161:0/1559051361 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff768003610 con 0x7ff774155d70 2024-03-28T23:37:17.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.456+0000 7ff7799e2640 1 -- 172.21.15.161:0/1559051361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff774155d70 msgr2=0x7ff7740c9440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:17.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.456+0000 7ff7799e2640 1 --2- 172.21.15.161:0/1559051361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff774155d70 0x7ff7740c9440 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7ff77408ccb0 tx=0x7ff76802b860 comp rx=0 tx=0).stop 2024-03-28T23:37:17.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.456+0000 7ff7799e2640 1 -- 172.21.15.161:0/1559051361 shutdown_connections 2024-03-28T23:37:17.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.456+0000 7ff7799e2640 1 --2- 172.21.15.161:0/1559051361 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7740c53e0 0x7ff7740c7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:17.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.456+0000 7ff7799e2640 1 --2- 172.21.15.161:0/1559051361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff774155d70 0x7ff7740c9440 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:17.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.456+0000 7ff7799e2640 1 --2- 172.21.15.161:0/1559051361 >> v2:172.21.15.170:3300/0 conn(0x7ff774152740 0x7ff77406b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:17.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.456+0000 7ff7799e2640 1 -- 172.21.15.161:0/1559051361 >> 172.21.15.161:0/1559051361 conn(0x7ff7740cac40 msgr2=0x7ff7740cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:17.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.456+0000 7ff7799e2640 1 -- 172.21.15.161:0/1559051361 shutdown_connections 2024-03-28T23:37:17.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.456+0000 7ff7799e2640 1 -- 172.21.15.161:0/1559051361 wait complete. 2024-03-28T23:37:17.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.457+0000 7ff7799e2640 1 Processor -- start 2024-03-28T23:37:17.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.457+0000 7ff7799e2640 1 -- start start 2024-03-28T23:37:17.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.458+0000 7ff7799e2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff7740c53e0 0x7ff774156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:17.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.458+0000 7ff7799e2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff774152740 0x7ff774154250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:17.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.458+0000 7ff7799e2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff774155d70 0x7ff7741e7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:17.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.458+0000 7ff7799e2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff7740c8f20 con 0x7ff774152740 2024-03-28T23:37:17.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.458+0000 7ff7799e2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff7740c8c40 con 0x7ff7740c53e0 2024-03-28T23:37:17.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.458+0000 7ff7799e2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff7740c5870 con 0x7ff774155d70 2024-03-28T23:37:17.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.458+0000 7ff7727fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff774152740 0x7ff774154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:17.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.458+0000 7ff7727fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff774152740 0x7ff774154250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55854/0 (socket says 172.21.15.161:55854) 2024-03-28T23:37:17.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.458+0000 7ff7727fc640 1 -- 172.21.15.161:0/3511890653 learned_addr learned my addr 172.21.15.161:0/3511890653 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:17.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.458+0000 7ff7737fe640 1 --2- 172.21.15.161:0/3511890653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff774155d70 0x7ff7741e7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:17.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.458+0000 7ff772ffd640 1 --2- 172.21.15.161:0/3511890653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff7740c53e0 0x7ff774156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:17.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.458+0000 7ff7727fc640 1 -- 172.21.15.161:0/3511890653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff774155d70 msgr2=0x7ff7741e7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:17.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.458+0000 7ff7727fc640 1 --2- 172.21.15.161:0/3511890653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff774155d70 0x7ff7741e7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:17.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.458+0000 7ff7727fc640 1 -- 172.21.15.161:0/3511890653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff7740c53e0 msgr2=0x7ff774156650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:17.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.458+0000 7ff7727fc640 1 --2- 172.21.15.161:0/3511890653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff7740c53e0 0x7ff774156650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:17.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.458+0000 7ff7727fc640 1 -- 172.21.15.161:0/3511890653 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7740c43d0 con 0x7ff774152740 2024-03-28T23:37:17.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.459+0000 7ff7727fc640 1 --2- 172.21.15.161:0/3511890653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff774152740 0x7ff774154250 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7ff768000c00 tx=0x7ff7680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:17.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.459+0000 7ff75bfff640 1 -- 172.21.15.161:0/3511890653 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff768037050 con 0x7ff774152740 2024-03-28T23:37:17.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.459+0000 7ff75bfff640 1 -- 172.21.15.161:0/3511890653 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff768043070 con 0x7ff774152740 2024-03-28T23:37:17.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.459+0000 7ff7799e2640 1 -- 172.21.15.161:0/3511890653 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7740c4680 con 0x7ff774152740 2024-03-28T23:37:17.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.459+0000 7ff75bfff640 1 -- 172.21.15.161:0/3511890653 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff768005e10 con 0x7ff774152740 2024-03-28T23:37:17.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.459+0000 7ff7799e2640 1 -- 172.21.15.161:0/3511890653 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7740c2d20 con 0x7ff774152740 2024-03-28T23:37:17.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.460+0000 7ff7799e2640 1 -- 172.21.15.161:0/3511890653 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff774068f90 con 0x7ff774152740 2024-03-28T23:37:17.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.461+0000 7ff75bfff640 1 -- 172.21.15.161:0/3511890653 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff768006040 con 0x7ff774152740 2024-03-28T23:37:17.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.461+0000 7ff75bfff640 1 --2- 172.21.15.161:0/3511890653 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff76006b4d0 0x7ff76006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:17.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.461+0000 7ff75bfff640 1 -- 172.21.15.161:0/3511890653 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7ff7680ac9d0 con 0x7ff774152740 2024-03-28T23:37:17.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.461+0000 7ff772ffd640 1 --2- 172.21.15.161:0/3511890653 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff76006b4d0 0x7ff76006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:17.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.461+0000 7ff75bfff640 1 --2- 172.21.15.161:0/3511890653 >> [v2:172.21.15.161:6816/4264782123,v1:172.21.15.161:6817/4264782123] conn(0x7ff760071c90 0x7ff7600740d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:17.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.461+0000 7ff75bfff640 1 -- 172.21.15.161:0/3511890653 --> [v2:172.21.15.161:6816/4264782123,v1:172.21.15.161:6817/4264782123] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff76800af30 con 0x7ff760071c90 2024-03-28T23:37:17.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.461+0000 7ff75bfff640 1 -- 172.21.15.161:0/3511890653 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=168) ==== 24+0+0 (secure 0 0 0) 0x7ff7680ace60 con 0x7ff774152740 2024-03-28T23:37:17.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.461+0000 7ff7737fe640 1 --2- 172.21.15.161:0/3511890653 >> [v2:172.21.15.161:6816/4264782123,v1:172.21.15.161:6817/4264782123] conn(0x7ff760071c90 0x7ff7600740d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:17.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.462+0000 7ff772ffd640 1 --2- 172.21.15.161:0/3511890653 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff76006b4d0 0x7ff76006d990 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7ff774152ec0 tx=0x7ff764009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:17.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.462+0000 7ff7737fe640 1 --2- 172.21.15.161:0/3511890653 >> [v2:172.21.15.161:6816/4264782123,v1:172.21.15.161:6817/4264782123] conn(0x7ff760071c90 0x7ff7600740d0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:17.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.462+0000 7ff75bfff640 1 -- 172.21.15.161:0/3511890653 <== osd.0 v2:172.21.15.161:6816/4264782123 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff76800af30 con 0x7ff760071c90 2024-03-28T23:37:17.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.475+0000 7ff7799e2640 1 -- 172.21.15.161:0/3511890653 --> [v2:172.21.15.161:6816/4264782123,v1:172.21.15.161:6817/4264782123] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff7740ca4f0 con 0x7ff760071c90 2024-03-28T23:37:17.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.476+0000 7ff75bfff640 1 -- 172.21.15.161:0/3511890653 <== osd.0 v2:172.21.15.161:6816/4264782123 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff7740ca4f0 con 0x7ff760071c90 2024-03-28T23:37:17.476 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:37:17.477 INFO:teuthology.orchestra.run.smithi161.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-28T23:37:17.477 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:37:17.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.476+0000 7ff7799e2640 1 -- 172.21.15.161:0/3511890653 >> [v2:172.21.15.161:6816/4264782123,v1:172.21.15.161:6817/4264782123] conn(0x7ff760071c90 msgr2=0x7ff7600740d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:17.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.476+0000 7ff7799e2640 1 --2- 172.21.15.161:0/3511890653 >> [v2:172.21.15.161:6816/4264782123,v1:172.21.15.161:6817/4264782123] conn(0x7ff760071c90 0x7ff7600740d0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:17.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.476+0000 7ff7799e2640 1 -- 172.21.15.161:0/3511890653 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff76006b4d0 msgr2=0x7ff76006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:17.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.476+0000 7ff7799e2640 1 --2- 172.21.15.161:0/3511890653 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff76006b4d0 0x7ff76006d990 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7ff774152ec0 tx=0x7ff764009190 comp rx=0 tx=0).stop 2024-03-28T23:37:17.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.476+0000 7ff7799e2640 1 -- 172.21.15.161:0/3511890653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff774152740 msgr2=0x7ff774154250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:17.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.476+0000 7ff7799e2640 1 --2- 172.21.15.161:0/3511890653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff774152740 0x7ff774154250 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7ff768000c00 tx=0x7ff7680046c0 comp rx=0 tx=0).stop 2024-03-28T23:37:17.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.476+0000 7ff7737fe640 1 -- 172.21.15.161:0/3511890653 reap_dead start 2024-03-28T23:37:17.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.476+0000 7ff7799e2640 1 -- 172.21.15.161:0/3511890653 shutdown_connections 2024-03-28T23:37:17.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.476+0000 7ff7799e2640 1 -- 172.21.15.161:0/3511890653 >> 172.21.15.161:0/3511890653 conn(0x7ff7740cac40 msgr2=0x7ff77414df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:17.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.476+0000 7ff7799e2640 1 -- 172.21.15.161:0/3511890653 shutdown_connections 2024-03-28T23:37:17.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.476+0000 7ff7799e2640 1 -- 172.21.15.161:0/3511890653 wait complete. 2024-03-28T23:37:17.487 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-28T23:37:17.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.589+0000 7fee48855640 1 Processor -- start 2024-03-28T23:37:17.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.589+0000 7fee48855640 1 -- start start 2024-03-28T23:37:17.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.590+0000 7fee48855640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fee440ac150 0x7fee44160af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:17.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.590+0000 7fee48855640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee44066fe0 con 0x7fee4406c3d0 2024-03-28T23:37:17.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.590+0000 7fee48855640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee440672c0 con 0x7fee44155b90 2024-03-28T23:37:17.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.590+0000 7fee48855640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fee44156070 con 0x7fee440ac150 2024-03-28T23:37:17.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.590+0000 7fee4259c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fee4406c3d0 0x7fee44152580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41684/0 (socket says 172.21.15.161:41684) 2024-03-28T23:37:17.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.590+0000 7fee4259c640 1 -- 172.21.15.161:0/1454534515 learned_addr learned my addr 172.21.15.161:0/1454534515 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:17.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.590+0000 7fee42d9d640 1 --2- 172.21.15.161:0/1454534515 >> v2:172.21.15.170:3300/0 conn(0x7fee440ac150 0x7fee44160af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:17.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.590+0000 7fee41d9b640 1 -- 172.21.15.161:0/1454534515 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4201715331 0 0) 0x7fee44066fe0 con 0x7fee4406c3d0 2024-03-28T23:37:17.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.590+0000 7fee41d9b640 1 -- 172.21.15.161:0/1454534515 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee30003520 con 0x7fee4406c3d0 2024-03-28T23:37:17.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.590+0000 7fee42d9d640 1 -- 172.21.15.161:0/1454534515 >> v1:172.21.15.170:6789/0 conn(0x7fee44155b90 legacy=0x7fee44158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:17.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.590+0000 7fee42d9d640 1 -- 172.21.15.161:0/1454534515 >> v1:172.21.15.161:6789/0 conn(0x7fee4406c3d0 legacy=0x7fee44152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:17.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.590+0000 7fee42d9d640 1 -- 172.21.15.161:0/1454534515 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee440c1c80 con 0x7fee440ac150 2024-03-28T23:37:17.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.591+0000 7fee42d9d640 1 --2- 172.21.15.161:0/1454534515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee440ac150 0x7fee44160af0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fee3c009d90 tx=0x7fee3c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61184541fc7676eb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:17.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.591+0000 7fee41d9b640 1 -- 172.21.15.161:0/1454534515 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c039070 con 0x7fee440ac150 2024-03-28T23:37:17.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.591+0000 7fee41d9b640 1 -- 172.21.15.161:0/1454534515 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee3c02cc60 con 0x7fee440ac150 2024-03-28T23:37:17.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.591+0000 7fee41d9b640 1 -- 172.21.15.161:0/1454534515 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c034640 con 0x7fee440ac150 2024-03-28T23:37:17.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.592+0000 7fee48855640 1 -- 172.21.15.161:0/1454534515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee440ac150 msgr2=0x7fee44160af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:17.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.592+0000 7fee48855640 1 --2- 172.21.15.161:0/1454534515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee440ac150 0x7fee44160af0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fee3c009d90 tx=0x7fee3c02beb0 comp rx=0 tx=0).stop 2024-03-28T23:37:17.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.592+0000 7fee48855640 1 -- 172.21.15.161:0/1454534515 shutdown_connections 2024-03-28T23:37:17.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.592+0000 7fee48855640 1 --2- 172.21.15.161:0/1454534515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee440ac150 0x7fee44160af0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:17.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.592+0000 7fee48855640 1 -- 172.21.15.161:0/1454534515 >> 172.21.15.161:0/1454534515 conn(0x7fee440c3be0 msgr2=0x7fee440c3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:17.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.592+0000 7fee48855640 1 -- 172.21.15.161:0/1454534515 shutdown_connections 2024-03-28T23:37:17.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.592+0000 7fee48855640 1 -- 172.21.15.161:0/1454534515 wait complete. 2024-03-28T23:37:17.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.593+0000 7fee48855640 1 Processor -- start 2024-03-28T23:37:17.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.593+0000 7fee48855640 1 -- start start 2024-03-28T23:37:17.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.594+0000 7fee48855640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee4406c3d0 0x7fee4414ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:17.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.594+0000 7fee48855640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee440ac150 0x7fee4414ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:17.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.594+0000 7fee48855640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee44155b90 0x7fee4414b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:17.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.594+0000 7fee48855640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fee440c2320 con 0x7fee44155b90 2024-03-28T23:37:17.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.594+0000 7fee48855640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fee440c21a0 con 0x7fee440ac150 2024-03-28T23:37:17.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.594+0000 7fee48855640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fee4406c7b0 con 0x7fee4406c3d0 2024-03-28T23:37:17.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.594+0000 7fee42d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee44155b90 0x7fee4414b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:17.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.594+0000 7fee42d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee44155b90 0x7fee4414b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55870/0 (socket says 172.21.15.161:55870) 2024-03-28T23:37:17.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.594+0000 7fee42d9d640 1 -- 172.21.15.161:0/680192812 learned_addr learned my addr 172.21.15.161:0/680192812 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:17.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.594+0000 7fee4259c640 1 --2- 172.21.15.161:0/680192812 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee4406c3d0 0x7fee4414ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:17.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.594+0000 7fee3bfff640 1 --2- 172.21.15.161:0/680192812 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee440ac150 0x7fee4414ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:17.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.594+0000 7fee42d9d640 1 -- 172.21.15.161:0/680192812 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee4406c3d0 msgr2=0x7fee4414ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:17.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.594+0000 7fee42d9d640 1 --2- 172.21.15.161:0/680192812 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee4406c3d0 0x7fee4414ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:17.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.594+0000 7fee42d9d640 1 -- 172.21.15.161:0/680192812 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee440ac150 msgr2=0x7fee4414ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:17.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.594+0000 7fee42d9d640 1 --2- 172.21.15.161:0/680192812 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee440ac150 0x7fee4414ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:17.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.594+0000 7fee42d9d640 1 -- 172.21.15.161:0/680192812 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee4414ca80 con 0x7fee44155b90 2024-03-28T23:37:17.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.595+0000 7fee42d9d640 1 --2- 172.21.15.161:0/680192812 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee44155b90 0x7fee4414b350 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fee3c009eb0 tx=0x7fee3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:17.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.595+0000 7fee3affd640 1 -- 172.21.15.161:0/680192812 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c042020 con 0x7fee44155b90 2024-03-28T23:37:17.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.595+0000 7fee3affd640 1 -- 172.21.15.161:0/680192812 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee3c03a040 con 0x7fee44155b90 2024-03-28T23:37:17.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.595+0000 7fee48855640 1 -- 172.21.15.161:0/680192812 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee4414d080 con 0x7fee44155b90 2024-03-28T23:37:17.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.595+0000 7fee3affd640 1 -- 172.21.15.161:0/680192812 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c039070 con 0x7fee44155b90 2024-03-28T23:37:17.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.595+0000 7fee48855640 1 -- 172.21.15.161:0/680192812 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee441504c0 con 0x7fee44155b90 2024-03-28T23:37:17.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.596+0000 7fee48855640 1 -- 172.21.15.161:0/680192812 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee4404f4b0 con 0x7fee44155b90 2024-03-28T23:37:17.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.596+0000 7fee3affd640 1 -- 172.21.15.161:0/680192812 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fee3c045020 con 0x7fee44155b90 2024-03-28T23:37:17.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.596+0000 7fee3affd640 1 --2- 172.21.15.161:0/680192812 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fee2406b500 0x7fee2406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:17.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.597+0000 7fee3affd640 1 -- 172.21.15.161:0/680192812 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7fee3c0ac3b0 con 0x7fee44155b90 2024-03-28T23:37:17.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.597+0000 7fee4259c640 1 --2- 172.21.15.161:0/680192812 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fee2406b500 0x7fee2406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:17.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.597+0000 7fee4259c640 1 --2- 172.21.15.161:0/680192812 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fee2406b500 0x7fee2406d9c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fee340011e0 tx=0x7fee34007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:17.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.601+0000 7fee3affd640 1 -- 172.21.15.161:0/680192812 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fee3c04c030 con 0x7fee44155b90 2024-03-28T23:37:17.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.771+0000 7fee48855640 1 -- 172.21.15.161:0/680192812 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fee4414c830 con 0x7fee2406b500 2024-03-28T23:37:17.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.772+0000 7fee3affd640 1 -- 172.21.15.161:0/680192812 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fee4414c830 con 0x7fee2406b500 2024-03-28T23:37:17.772 INFO:teuthology.orchestra.run.smithi161.stderr:instructed osd(s) 0 to deep-scrub 2024-03-28T23:37:17.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.773+0000 7fee48855640 1 -- 172.21.15.161:0/680192812 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fee2406b500 msgr2=0x7fee2406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:17.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.773+0000 7fee48855640 1 --2- 172.21.15.161:0/680192812 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fee2406b500 0x7fee2406d9c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fee340011e0 tx=0x7fee34007d30 comp rx=0 tx=0).stop 2024-03-28T23:37:17.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.774+0000 7fee48855640 1 -- 172.21.15.161:0/680192812 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee44155b90 msgr2=0x7fee4414b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:17.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.774+0000 7fee48855640 1 --2- 172.21.15.161:0/680192812 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee44155b90 0x7fee4414b350 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fee3c009eb0 tx=0x7fee3c004a40 comp rx=0 tx=0).stop 2024-03-28T23:37:17.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.774+0000 7fee48855640 1 -- 172.21.15.161:0/680192812 shutdown_connections 2024-03-28T23:37:17.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.774+0000 7fee48855640 1 --2- 172.21.15.161:0/680192812 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fee2406b500 0x7fee2406d9c0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:17.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.774+0000 7fee48855640 1 --2- 172.21.15.161:0/680192812 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee44155b90 0x7fee4414b350 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:17.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.774+0000 7fee48855640 1 --2- 172.21.15.161:0/680192812 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee440ac150 0x7fee4414ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:17.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.774+0000 7fee48855640 1 --2- 172.21.15.161:0/680192812 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee4406c3d0 0x7fee4414ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:17.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.774+0000 7fee48855640 1 -- 172.21.15.161:0/680192812 >> 172.21.15.161:0/680192812 conn(0x7fee440c3be0 msgr2=0x7fee440c0510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:17.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.774+0000 7fee48855640 1 -- 172.21.15.161:0/680192812 shutdown_connections 2024-03-28T23:37:17.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.774+0000 7fee48855640 1 -- 172.21.15.161:0/680192812 wait complete. 2024-03-28T23:37:17.785 INFO:tasks.ceph:Scrubbing osd.1 2024-03-28T23:37:17.785 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-28T23:37:17.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.892+0000 7f79f98ed640 1 Processor -- start 2024-03-28T23:37:17.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.892+0000 7f79f98ed640 1 -- start start 2024-03-28T23:37:17.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.893+0000 7f79f98ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79f40c5dc0 0x7f79f4152d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:17.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.893+0000 7f79f98ed640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79f40c5880 con 0x7f79f406bff0 2024-03-28T23:37:17.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.893+0000 7f79f98ed640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79f40c5b60 con 0x7f79f40c6ca0 2024-03-28T23:37:17.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.893+0000 7f79f98ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f79f406bb60 con 0x7f79f40c5dc0 2024-03-28T23:37:17.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.893+0000 7f79ea7fc640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f79f406bff0 0x7f79f4152600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41690/0 (socket says 172.21.15.161:41690) 2024-03-28T23:37:17.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.893+0000 7f79ea7fc640 1 -- 172.21.15.161:0/1991227248 learned_addr learned my addr 172.21.15.161:0/1991227248 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:17.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.893+0000 7f79f37fe640 1 --2- 172.21.15.161:0/1991227248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79f40c5dc0 0x7f79f4152d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:17.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.893+0000 7f79f27fc640 1 -- 172.21.15.161:0/1991227248 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2068404397 0 0) 0x7f79f40c5880 con 0x7f79f406bff0 2024-03-28T23:37:17.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.893+0000 7f79f27fc640 1 -- 172.21.15.161:0/1991227248 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79e0003520 con 0x7f79f406bff0 2024-03-28T23:37:17.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.893+0000 7f79f27fc640 1 -- 172.21.15.161:0/1991227248 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3348026737 0 0) 0x7f79f40c5b60 con 0x7f79f40c6ca0 2024-03-28T23:37:17.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.893+0000 7f79f37fe640 1 -- 172.21.15.161:0/1991227248 >> v1:172.21.15.170:6789/0 conn(0x7f79f40c6ca0 legacy=0x7f79f40c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:17.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.893+0000 7f79f37fe640 1 -- 172.21.15.161:0/1991227248 >> v1:172.21.15.161:6789/0 conn(0x7f79f406bff0 legacy=0x7f79f4152600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:17.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.894+0000 7f79f37fe640 1 -- 172.21.15.161:0/1991227248 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f4155f10 con 0x7f79f40c5dc0 2024-03-28T23:37:17.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.894+0000 7f79f37fe640 1 --2- 172.21.15.161:0/1991227248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79f40c5dc0 0x7f79f4152d10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f79ec009dc0 tx=0x7f79ec02bd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c0045dedbd5ffe5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:17.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.894+0000 7f79f27fc640 1 -- 172.21.15.161:0/1991227248 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ec039070 con 0x7f79f40c5dc0 2024-03-28T23:37:17.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.895+0000 7f79f27fc640 1 -- 172.21.15.161:0/1991227248 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79ec02c840 con 0x7f79f40c5dc0 2024-03-28T23:37:17.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.895+0000 7f79f27fc640 1 -- 172.21.15.161:0/1991227248 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79dc001670 con 0x7f79f40c5dc0 2024-03-28T23:37:17.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.895+0000 7f79f98ed640 1 -- 172.21.15.161:0/1991227248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79f40c5dc0 msgr2=0x7f79f4152d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:17.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.895+0000 7f79f98ed640 1 --2- 172.21.15.161:0/1991227248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79f40c5dc0 0x7f79f4152d10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f79ec009dc0 tx=0x7f79ec02bd10 comp rx=0 tx=0).stop 2024-03-28T23:37:17.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.895+0000 7f79f98ed640 1 -- 172.21.15.161:0/1991227248 shutdown_connections 2024-03-28T23:37:17.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.895+0000 7f79f98ed640 1 --2- 172.21.15.161:0/1991227248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79f40c5dc0 0x7f79f4152d10 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:17.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.895+0000 7f79f98ed640 1 -- 172.21.15.161:0/1991227248 >> 172.21.15.161:0/1991227248 conn(0x7f79f40ca9e0 msgr2=0x7f79f40cadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:17.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.895+0000 7f79f98ed640 1 -- 172.21.15.161:0/1991227248 shutdown_connections 2024-03-28T23:37:17.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.895+0000 7f79f98ed640 1 -- 172.21.15.161:0/1991227248 wait complete. 2024-03-28T23:37:17.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.896+0000 7f79f98ed640 1 Processor -- start 2024-03-28T23:37:17.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.897+0000 7f79f98ed640 1 -- start start 2024-03-28T23:37:17.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.897+0000 7f79f98ed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79f406bff0 0x7f79f41e7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:17.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.897+0000 7f79f98ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79f40c5dc0 0x7f79f40c2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:17.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.897+0000 7f79f98ed640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79f40c6ca0 0x7f79f40c2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:17.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.897+0000 7f79f98ed640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f79f40c9170 con 0x7f79f40c6ca0 2024-03-28T23:37:17.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.897+0000 7f79f98ed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f79f40c8ff0 con 0x7f79f406bff0 2024-03-28T23:37:17.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.897+0000 7f79f98ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f79f40c7080 con 0x7f79f40c5dc0 2024-03-28T23:37:17.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.897+0000 7f79f37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79f40c6ca0 0x7f79f40c2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:17.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.897+0000 7f79f37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79f40c6ca0 0x7f79f40c2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55876/0 (socket says 172.21.15.161:55876) 2024-03-28T23:37:17.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.897+0000 7f79f37fe640 1 -- 172.21.15.161:0/3816848732 learned_addr learned my addr 172.21.15.161:0/3816848732 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:17.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.898+0000 7f79f2ffd640 1 --2- 172.21.15.161:0/3816848732 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79f406bff0 0x7f79f41e7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:17.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.898+0000 7f79ea7fc640 1 --2- 172.21.15.161:0/3816848732 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79f40c5dc0 0x7f79f40c2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:17.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.898+0000 7f79f37fe640 1 -- 172.21.15.161:0/3816848732 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79f40c5dc0 msgr2=0x7f79f40c2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:17.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.898+0000 7f79f37fe640 1 --2- 172.21.15.161:0/3816848732 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79f40c5dc0 0x7f79f40c2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:17.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.898+0000 7f79f37fe640 1 -- 172.21.15.161:0/3816848732 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79f406bff0 msgr2=0x7f79f41e7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:17.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.898+0000 7f79f37fe640 1 --2- 172.21.15.161:0/3816848732 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79f406bff0 0x7f79f41e7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:17.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.898+0000 7f79f37fe640 1 -- 172.21.15.161:0/3816848732 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f41eb120 con 0x7f79f40c6ca0 2024-03-28T23:37:17.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.898+0000 7f79f37fe640 1 --2- 172.21.15.161:0/3816848732 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79f40c6ca0 0x7f79f40c2fd0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f79ec009ee0 tx=0x7f79ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:17.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.898+0000 7f79f0ff9640 1 -- 172.21.15.161:0/3816848732 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ec044020 con 0x7f79f40c6ca0 2024-03-28T23:37:17.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.898+0000 7f79f0ff9640 1 -- 172.21.15.161:0/3816848732 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79ec034d70 con 0x7f79f40c6ca0 2024-03-28T23:37:17.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.898+0000 7f79f0ff9640 1 -- 172.21.15.161:0/3816848732 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ec039040 con 0x7f79f40c6ca0 2024-03-28T23:37:17.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.898+0000 7f79f98ed640 1 -- 172.21.15.161:0/3816848732 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79f40c41e0 con 0x7f79f40c6ca0 2024-03-28T23:37:17.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.898+0000 7f79f98ed640 1 -- 172.21.15.161:0/3816848732 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79f40ae020 con 0x7f79f40c6ca0 2024-03-28T23:37:17.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.899+0000 7f79f98ed640 1 -- 172.21.15.161:0/3816848732 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f79f4068f90 con 0x7f79f40c6ca0 2024-03-28T23:37:17.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.900+0000 7f79f0ff9640 1 -- 172.21.15.161:0/3816848732 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f79ec047020 con 0x7f79f40c6ca0 2024-03-28T23:37:17.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.900+0000 7f79f0ff9640 1 --2- 172.21.15.161:0/3816848732 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f79d406b500 0x7f79d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:17.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.900+0000 7f79f0ff9640 1 -- 172.21.15.161:0/3816848732 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7f79ec0ac450 con 0x7f79f40c6ca0 2024-03-28T23:37:17.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.900+0000 7f79f2ffd640 1 --2- 172.21.15.161:0/3816848732 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f79d406b500 0x7f79d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:17.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.900+0000 7f79f0ff9640 1 --2- 172.21.15.161:0/3816848732 >> [v2:172.21.15.161:6800/3551918247,v1:172.21.15.161:6801/3551918247] conn(0x7f79d4071cc0 0x7f79d4074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:17.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.900+0000 7f79f0ff9640 1 -- 172.21.15.161:0/3816848732 --> [v2:172.21.15.161:6800/3551918247,v1:172.21.15.161:6801/3551918247] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f79d40747d0 con 0x7f79d4071cc0 2024-03-28T23:37:17.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.900+0000 7f79f0ff9640 1 -- 172.21.15.161:0/3816848732 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=168) ==== 24+0+0 (secure 0 0 0) 0x7f79ec0b3050 con 0x7f79f40c6ca0 2024-03-28T23:37:17.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.900+0000 7f79ea7fc640 1 --2- 172.21.15.161:0/3816848732 >> [v2:172.21.15.161:6800/3551918247,v1:172.21.15.161:6801/3551918247] conn(0x7f79d4071cc0 0x7f79d4074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:17.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.900+0000 7f79f2ffd640 1 --2- 172.21.15.161:0/3816848732 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f79d406b500 0x7f79d406d9c0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f79e4000f60 tx=0x7f79e4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:17.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.901+0000 7f79ea7fc640 1 --2- 172.21.15.161:0/3816848732 >> [v2:172.21.15.161:6800/3551918247,v1:172.21.15.161:6801/3551918247] conn(0x7f79d4071cc0 0x7f79d4074100 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:17.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.901+0000 7f79f0ff9640 1 -- 172.21.15.161:0/3816848732 <== osd.1 v2:172.21.15.161:6800/3551918247 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f79d40747d0 con 0x7f79d4071cc0 2024-03-28T23:37:17.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.914+0000 7f79f98ed640 1 -- 172.21.15.161:0/3816848732 --> [v2:172.21.15.161:6800/3551918247,v1:172.21.15.161:6801/3551918247] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f79f414f080 con 0x7f79d4071cc0 2024-03-28T23:37:17.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.914+0000 7f79f0ff9640 1 -- 172.21.15.161:0/3816848732 <== osd.1 v2:172.21.15.161:6800/3551918247 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f79f414f080 con 0x7f79d4071cc0 2024-03-28T23:37:17.915 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:37:17.915 INFO:teuthology.orchestra.run.smithi161.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-28T23:37:17.915 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:37:17.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.915+0000 7f79f98ed640 1 -- 172.21.15.161:0/3816848732 >> [v2:172.21.15.161:6800/3551918247,v1:172.21.15.161:6801/3551918247] conn(0x7f79d4071cc0 msgr2=0x7f79d4074100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:17.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.915+0000 7f79f98ed640 1 --2- 172.21.15.161:0/3816848732 >> [v2:172.21.15.161:6800/3551918247,v1:172.21.15.161:6801/3551918247] conn(0x7f79d4071cc0 0x7f79d4074100 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:17.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.915+0000 7f79f98ed640 1 -- 172.21.15.161:0/3816848732 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f79d406b500 msgr2=0x7f79d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:17.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.915+0000 7f79f98ed640 1 --2- 172.21.15.161:0/3816848732 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f79d406b500 0x7f79d406d9c0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f79e4000f60 tx=0x7f79e4009040 comp rx=0 tx=0).stop 2024-03-28T23:37:17.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.915+0000 7f79f98ed640 1 -- 172.21.15.161:0/3816848732 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79f40c6ca0 msgr2=0x7f79f40c2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:17.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.915+0000 7f79f98ed640 1 --2- 172.21.15.161:0/3816848732 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79f40c6ca0 0x7f79f40c2fd0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f79ec009ee0 tx=0x7f79ec004a40 comp rx=0 tx=0).stop 2024-03-28T23:37:17.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.915+0000 7f79f37fe640 1 -- 172.21.15.161:0/3816848732 reap_dead start 2024-03-28T23:37:17.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.915+0000 7f79f98ed640 1 -- 172.21.15.161:0/3816848732 shutdown_connections 2024-03-28T23:37:17.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.915+0000 7f79f98ed640 1 -- 172.21.15.161:0/3816848732 >> 172.21.15.161:0/3816848732 conn(0x7f79f40ca9e0 msgr2=0x7f79f40aa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:17.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.915+0000 7f79f98ed640 1 -- 172.21.15.161:0/3816848732 shutdown_connections 2024-03-28T23:37:17.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:17.915+0000 7f79f98ed640 1 -- 172.21.15.161:0/3816848732 wait complete. 2024-03-28T23:37:17.926 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-28T23:37:18.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.027+0000 7fd1ea9a0640 1 Processor -- start 2024-03-28T23:37:18.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.027+0000 7fd1ea9a0640 1 -- start start 2024-03-28T23:37:18.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.028+0000 7fd1ea9a0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1e4072750 0x7fd1e40c5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.028+0000 7fd1ea9a0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e4072be0 con 0x7fd1e40acc30 2024-03-28T23:37:18.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.028+0000 7fd1ea9a0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e4072ec0 con 0x7fd1e40c5180 2024-03-28T23:37:18.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.028+0000 7fd1ea9a0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd1e40731a0 con 0x7fd1e4072750 2024-03-28T23:37:18.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.028+0000 7fd1ea19f640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd1e40acc30 0x7fd1e4153590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41698/0 (socket says 172.21.15.161:41698) 2024-03-28T23:37:18.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.028+0000 7fd1ea19f640 1 -- 172.21.15.161:0/1802661508 learned_addr learned my addr 172.21.15.161:0/1802661508 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:18.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1e919d640 1 --2- 172.21.15.161:0/1802661508 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1e4072750 0x7fd1e40c5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1e899c640 1 -- 172.21.15.161:0/1802661508 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4160209775 0 0) 0x7fd1e4072be0 con 0x7fd1e40acc30 2024-03-28T23:37:18.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1e899c640 1 -- 172.21.15.161:0/1802661508 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1d0003520 con 0x7fd1e40acc30 2024-03-28T23:37:18.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1e899c640 1 -- 172.21.15.161:0/1802661508 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 884617610 0 0) 0x7fd1e4072ec0 con 0x7fd1e40c5180 2024-03-28T23:37:18.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1e899c640 1 -- 172.21.15.161:0/1802661508 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1e4072be0 con 0x7fd1e40c5180 2024-03-28T23:37:18.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1e899c640 1 -- 172.21.15.161:0/1802661508 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1371387194 0 0) 0x7fd1d0003520 con 0x7fd1e40acc30 2024-03-28T23:37:18.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1e899c640 1 -- 172.21.15.161:0/1802661508 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1e4072750 msgr2=0x7fd1e40c5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:18.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1e899c640 1 --2- 172.21.15.161:0/1802661508 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1e4072750 0x7fd1e40c5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1e899c640 1 -- 172.21.15.161:0/1802661508 >> v1:172.21.15.170:6789/0 conn(0x7fd1e40c5180 legacy=0x7fd1e4156e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1e899c640 1 -- 172.21.15.161:0/1802661508 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e4153070 con 0x7fd1e40acc30 2024-03-28T23:37:18.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1e899c640 1 -- 172.21.15.161:0/1802661508 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fd1dc001670 con 0x7fd1e40acc30 2024-03-28T23:37:18.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1e899c640 1 -- 172.21.15.161:0/1802661508 >> v1:172.21.15.161:6789/0 conn(0x7fd1e40acc30 legacy=0x7fd1e4153590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1e899c640 1 --2- 172.21.15.161:0/1802661508 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1d0003b70 0x7fd1d0003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1e899c640 1 --2- 172.21.15.161:0/1802661508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1d0004800 0x7fd1d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1e899c640 1 --2- 172.21.15.161:0/1802661508 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1d00051c0 0x7fd1d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1e899c640 1 -- 172.21.15.161:0/1802661508 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e40731a0 con 0x7fd1d00051c0 2024-03-28T23:37:18.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1e899c640 1 -- 172.21.15.161:0/1802661508 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d000bfb0 con 0x7fd1d0004800 2024-03-28T23:37:18.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1e899c640 1 -- 172.21.15.161:0/1802661508 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd1d000c1b0 con 0x7fd1d0003b70 2024-03-28T23:37:18.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1e919d640 1 --2- 172.21.15.161:0/1802661508 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1e4072750 0x7fd1e40c5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:37:18.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1ea19f640 1 --2- 172.21.15.161:0/1802661508 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1d00051c0 0x7fd1d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1e919d640 1 --2- 172.21.15.161:0/1802661508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1d0004800 0x7fd1d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1e999e640 1 --2- 172.21.15.161:0/1802661508 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1d0003b70 0x7fd1d0003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1ea19f640 1 -- 172.21.15.161:0/1802661508 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1d0003b70 msgr2=0x7fd1d0003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:18.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1ea19f640 1 --2- 172.21.15.161:0/1802661508 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1d0003b70 0x7fd1d0003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1ea19f640 1 -- 172.21.15.161:0/1802661508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1d0004800 msgr2=0x7fd1d0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:18.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1ea19f640 1 --2- 172.21.15.161:0/1802661508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1d0004800 0x7fd1d0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.029+0000 7fd1ea19f640 1 -- 172.21.15.161:0/1802661508 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d0003520 con 0x7fd1d00051c0 2024-03-28T23:37:18.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.030+0000 7fd1ea19f640 1 --2- 172.21.15.161:0/1802661508 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1d00051c0 0x7fd1d000b9c0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc00c190 tx=0x7fd1d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfe526f63b3038c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:18.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.030+0000 7fd1e899c640 1 -- 172.21.15.161:0/1802661508 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc007ce0 con 0x7fd1d00051c0 2024-03-28T23:37:18.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.030+0000 7fd1e899c640 1 -- 172.21.15.161:0/1802661508 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1dc030070 con 0x7fd1d00051c0 2024-03-28T23:37:18.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.030+0000 7fd1e899c640 1 -- 172.21.15.161:0/1802661508 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc005d90 con 0x7fd1d00051c0 2024-03-28T23:37:18.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.031+0000 7fd1ea9a0640 1 -- 172.21.15.161:0/1802661508 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1d00051c0 msgr2=0x7fd1d000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.031+0000 7fd1ea9a0640 1 --2- 172.21.15.161:0/1802661508 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1d00051c0 0x7fd1d000b9c0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc00c190 tx=0x7fd1d0005830 comp rx=0 tx=0).stop 2024-03-28T23:37:18.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.031+0000 7fd1ea19f640 1 -- 172.21.15.161:0/1802661508 reap_dead start 2024-03-28T23:37:18.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.031+0000 7fd1ea9a0640 1 -- 172.21.15.161:0/1802661508 shutdown_connections 2024-03-28T23:37:18.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.031+0000 7fd1ea9a0640 1 -- 172.21.15.161:0/1802661508 >> 172.21.15.161:0/1802661508 conn(0x7fd1e406ee80 msgr2=0x7fd1e406b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:18.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.031+0000 7fd1ea9a0640 1 -- 172.21.15.161:0/1802661508 shutdown_connections 2024-03-28T23:37:18.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.031+0000 7fd1ea9a0640 1 -- 172.21.15.161:0/1802661508 wait complete. 2024-03-28T23:37:18.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.032+0000 7fd1ea9a0640 1 Processor -- start 2024-03-28T23:37:18.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.032+0000 7fd1ea9a0640 1 -- start start 2024-03-28T23:37:18.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.033+0000 7fd1ea9a0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1d00051c0 0x7fd1e41eb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.033+0000 7fd1ea9a0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1e406c260 0x7fd1e40c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.033+0000 7fd1ea9a0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1e41e77f0 0x7fd1e41e7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.033+0000 7fd1ea9a0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e4153ca0 con 0x7fd1d00051c0 2024-03-28T23:37:18.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.033+0000 7fd1ea9a0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e4150800 con 0x7fd1e41e77f0 2024-03-28T23:37:18.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.033+0000 7fd1ea9a0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fd1e4153e20 con 0x7fd1e406c260 2024-03-28T23:37:18.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.033+0000 7fd1e999e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1d00051c0 0x7fd1e41eb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.033+0000 7fd1e999e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1d00051c0 0x7fd1e41eb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55888/0 (socket says 172.21.15.161:55888) 2024-03-28T23:37:18.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.033+0000 7fd1e999e640 1 -- 172.21.15.161:0/535294505 learned_addr learned my addr 172.21.15.161:0/535294505 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:18.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.033+0000 7fd1e919d640 1 --2- 172.21.15.161:0/535294505 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1e406c260 0x7fd1e40c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.033+0000 7fd1ea19f640 1 --2- 172.21.15.161:0/535294505 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1e41e77f0 0x7fd1e41e7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.033+0000 7fd1e999e640 1 -- 172.21.15.161:0/535294505 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1e406c260 msgr2=0x7fd1e40c6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.033+0000 7fd1e999e640 1 --2- 172.21.15.161:0/535294505 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1e406c260 0x7fd1e40c6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.033+0000 7fd1e999e640 1 -- 172.21.15.161:0/535294505 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1e41e77f0 msgr2=0x7fd1e41e7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.033+0000 7fd1e999e640 1 --2- 172.21.15.161:0/535294505 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1e41e77f0 0x7fd1e41e7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.033+0000 7fd1e999e640 1 -- 172.21.15.161:0/535294505 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e4157360 con 0x7fd1d00051c0 2024-03-28T23:37:18.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.033+0000 7fd1e999e640 1 --2- 172.21.15.161:0/535294505 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1d00051c0 0x7fd1e41eb020 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fd1e000fb10 tx=0x7fd1e000d150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:18.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.033+0000 7fd1da7fc640 1 -- 172.21.15.161:0/535294505 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e0003b80 con 0x7fd1d00051c0 2024-03-28T23:37:18.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.033+0000 7fd1da7fc640 1 -- 172.21.15.161:0/535294505 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1e0018070 con 0x7fd1d00051c0 2024-03-28T23:37:18.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.033+0000 7fd1ea9a0640 1 -- 172.21.15.161:0/535294505 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1e406dcc0 con 0x7fd1d00051c0 2024-03-28T23:37:18.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.033+0000 7fd1da7fc640 1 -- 172.21.15.161:0/535294505 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e000d8f0 con 0x7fd1d00051c0 2024-03-28T23:37:18.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.033+0000 7fd1ea9a0640 1 -- 172.21.15.161:0/535294505 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1e406cfa0 con 0x7fd1d00051c0 2024-03-28T23:37:18.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.034+0000 7fd1ea9a0640 1 -- 172.21.15.161:0/535294505 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1e406e1e0 con 0x7fd1d00051c0 2024-03-28T23:37:18.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.035+0000 7fd1da7fc640 1 -- 172.21.15.161:0/535294505 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd1e000da90 con 0x7fd1d00051c0 2024-03-28T23:37:18.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.035+0000 7fd1da7fc640 1 --2- 172.21.15.161:0/535294505 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd1c806b4d0 0x7fd1c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.035+0000 7fd1da7fc640 1 -- 172.21.15.161:0/535294505 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7fd1e008a3b0 con 0x7fd1d00051c0 2024-03-28T23:37:18.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.035+0000 7fd1e919d640 1 --2- 172.21.15.161:0/535294505 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd1c806b4d0 0x7fd1c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.036+0000 7fd1e919d640 1 --2- 172.21.15.161:0/535294505 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd1c806b4d0 0x7fd1c806d990 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fd1e406cbb0 tx=0x7fd1d4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:18.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.040+0000 7fd1da7fc640 1 -- 172.21.15.161:0/535294505 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd1e406e1e0 con 0x7fd1d00051c0 2024-03-28T23:37:18.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.212+0000 7fd1ea9a0640 1 -- 172.21.15.161:0/535294505 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fd1e404f4b0 con 0x7fd1c806b4d0 2024-03-28T23:37:18.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.212+0000 7fd1da7fc640 1 -- 172.21.15.161:0/535294505 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fd1e404f4b0 con 0x7fd1c806b4d0 2024-03-28T23:37:18.213 INFO:teuthology.orchestra.run.smithi161.stderr:instructed osd(s) 1 to deep-scrub 2024-03-28T23:37:18.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.214+0000 7fd1ea9a0640 1 -- 172.21.15.161:0/535294505 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd1c806b4d0 msgr2=0x7fd1c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.214+0000 7fd1ea9a0640 1 --2- 172.21.15.161:0/535294505 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd1c806b4d0 0x7fd1c806d990 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fd1e406cbb0 tx=0x7fd1d4009190 comp rx=0 tx=0).stop 2024-03-28T23:37:18.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.214+0000 7fd1ea9a0640 1 -- 172.21.15.161:0/535294505 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1d00051c0 msgr2=0x7fd1e41eb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.214+0000 7fd1ea9a0640 1 --2- 172.21.15.161:0/535294505 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1d00051c0 0x7fd1e41eb020 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fd1e000fb10 tx=0x7fd1e000d150 comp rx=0 tx=0).stop 2024-03-28T23:37:18.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.214+0000 7fd1ea9a0640 1 -- 172.21.15.161:0/535294505 shutdown_connections 2024-03-28T23:37:18.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.214+0000 7fd1ea9a0640 1 --2- 172.21.15.161:0/535294505 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fd1c806b4d0 0x7fd1c806d990 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.214+0000 7fd1ea9a0640 1 --2- 172.21.15.161:0/535294505 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1e41e77f0 0x7fd1e41e7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.214+0000 7fd1ea9a0640 1 --2- 172.21.15.161:0/535294505 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1e406c260 0x7fd1e40c6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.214+0000 7fd1ea9a0640 1 --2- 172.21.15.161:0/535294505 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1d00051c0 0x7fd1e41eb020 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.214+0000 7fd1ea9a0640 1 -- 172.21.15.161:0/535294505 >> 172.21.15.161:0/535294505 conn(0x7fd1e406ee80 msgr2=0x7fd1e414ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:18.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.214+0000 7fd1ea9a0640 1 -- 172.21.15.161:0/535294505 shutdown_connections 2024-03-28T23:37:18.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.214+0000 7fd1ea9a0640 1 -- 172.21.15.161:0/535294505 wait complete. 2024-03-28T23:37:18.225 INFO:tasks.ceph:Scrubbing osd.2 2024-03-28T23:37:18.225 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-28T23:37:18.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.332+0000 7f639dab5640 1 Processor -- start 2024-03-28T23:37:18.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.332+0000 7f639dab5640 1 -- start start 2024-03-28T23:37:18.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.333+0000 7f639dab5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63980abb70 0x7f63980abf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.333+0000 7f639dab5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f63980c46b0 0x7f63980c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.333+0000 7f639dab5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6398149ec0 con 0x7f639806d390 2024-03-28T23:37:18.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.333+0000 7f639dab5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f6398153ab0 con 0x7f63980c46b0 2024-03-28T23:37:18.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.333+0000 7f639dab5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f639806e400 con 0x7f63980abb70 2024-03-28T23:37:18.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.333+0000 7f6396ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f639806d390 0x7f639806dcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41704/0 (socket says 172.21.15.161:41704) 2024-03-28T23:37:18.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.333+0000 7f6396ffd640 1 -- 172.21.15.161:0/2212262798 learned_addr learned my addr 172.21.15.161:0/2212262798 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:18.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.333+0000 7f6397fff640 1 --2- 172.21.15.161:0/2212262798 >> v2:172.21.15.170:3300/0 conn(0x7f63980c46b0 0x7f63980c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.333+0000 7f63977fe640 1 --2- 172.21.15.161:0/2212262798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63980abb70 0x7f63980abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.333+0000 7f63967fc640 1 -- 172.21.15.161:0/2212262798 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4181902870 0 0) 0x7f6398149ec0 con 0x7f639806d390 2024-03-28T23:37:18.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.333+0000 7f63967fc640 1 -- 172.21.15.161:0/2212262798 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6380003520 con 0x7f639806d390 2024-03-28T23:37:18.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.333+0000 7f63967fc640 1 -- 172.21.15.161:0/2212262798 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1161237096 0 0) 0x7f6380003520 con 0x7f639806d390 2024-03-28T23:37:18.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.333+0000 7f63967fc640 1 -- 172.21.15.161:0/2212262798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63980abb70 msgr2=0x7f63980abf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:18.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.333+0000 7f63967fc640 1 --2- 172.21.15.161:0/2212262798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63980abb70 0x7f63980abf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.333+0000 7f63967fc640 1 -- 172.21.15.161:0/2212262798 >> v2:172.21.15.170:3300/0 conn(0x7f63980c46b0 msgr2=0x7f63980c4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:18.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.333+0000 7f63967fc640 1 --2- 172.21.15.161:0/2212262798 >> v2:172.21.15.170:3300/0 conn(0x7f63980c46b0 0x7f63980c4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.333+0000 7f63967fc640 1 -- 172.21.15.161:0/2212262798 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6398153180 con 0x7f639806d390 2024-03-28T23:37:18.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.333+0000 7f6397fff640 1 --2- 172.21.15.161:0/2212262798 >> v2:172.21.15.170:3300/0 conn(0x7f63980c46b0 0x7f63980c4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:37:18.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.333+0000 7f63967fc640 1 -- 172.21.15.161:0/2212262798 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f638c001670 con 0x7f639806d390 2024-03-28T23:37:18.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.333+0000 7f63977fe640 1 --2- 172.21.15.161:0/2212262798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63980abb70 0x7f63980abf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:37:18.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.333+0000 7f63967fc640 1 -- 172.21.15.161:0/2212262798 >> v1:172.21.15.161:6789/0 conn(0x7f639806d390 legacy=0x7f639806dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.333+0000 7f63967fc640 1 --2- 172.21.15.161:0/2212262798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63800039b0 0x7f6380003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.333+0000 7f63967fc640 1 --2- 172.21.15.161:0/2212262798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63980abb70 0x7f6380004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.333+0000 7f63967fc640 1 --2- 172.21.15.161:0/2212262798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6380004d40 0x7f63800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.334+0000 7f63967fc640 1 -- 172.21.15.161:0/2212262798 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6398153ab0 con 0x7f63800039b0 2024-03-28T23:37:18.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.334+0000 7f63967fc640 1 -- 172.21.15.161:0/2212262798 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f639806e400 con 0x7f6380004d40 2024-03-28T23:37:18.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.334+0000 7f63967fc640 1 -- 172.21.15.161:0/2212262798 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f638000b9c0 con 0x7f63980abb70 2024-03-28T23:37:18.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.334+0000 7f63977fe640 1 --2- 172.21.15.161:0/2212262798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63800039b0 0x7f6380003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.334+0000 7f6396ffd640 1 --2- 172.21.15.161:0/2212262798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63980abb70 0x7f6380004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.334+0000 7f6397fff640 1 --2- 172.21.15.161:0/2212262798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6380004d40 0x7f63800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.334+0000 7f63977fe640 1 -- 172.21.15.161:0/2212262798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63980abb70 msgr2=0x7f6380004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:18.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.334+0000 7f63977fe640 1 --2- 172.21.15.161:0/2212262798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63980abb70 0x7f6380004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.334+0000 7f63977fe640 1 -- 172.21.15.161:0/2212262798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6380004d40 msgr2=0x7f63800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:18.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.334+0000 7f63977fe640 1 --2- 172.21.15.161:0/2212262798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6380004d40 0x7f63800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.334+0000 7f63977fe640 1 -- 172.21.15.161:0/2212262798 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6380003520 con 0x7f63800039b0 2024-03-28T23:37:18.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.334+0000 7f6396ffd640 1 --2- 172.21.15.161:0/2212262798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63980abb70 0x7f6380004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:37:18.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.334+0000 7f6397fff640 1 --2- 172.21.15.161:0/2212262798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6380004d40 0x7f63800051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:37:18.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.335+0000 7f63977fe640 1 --2- 172.21.15.161:0/2212262798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63800039b0 0x7f6380003dd0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f6388009c90 tx=0x7f6380005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd31f41c805e991e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:18.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.335+0000 7f63967fc640 1 -- 172.21.15.161:0/2212262798 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6388039070 con 0x7f63800039b0 2024-03-28T23:37:18.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.335+0000 7f63967fc640 1 -- 172.21.15.161:0/2212262798 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f638802c8f0 con 0x7f63800039b0 2024-03-28T23:37:18.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.335+0000 7f63967fc640 1 -- 172.21.15.161:0/2212262798 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6388009860 con 0x7f63800039b0 2024-03-28T23:37:18.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.335+0000 7f639dab5640 1 -- 172.21.15.161:0/2212262798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63800039b0 msgr2=0x7f6380003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.335+0000 7f639dab5640 1 --2- 172.21.15.161:0/2212262798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63800039b0 0x7f6380003dd0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f6388009c90 tx=0x7f6380005830 comp rx=0 tx=0).stop 2024-03-28T23:37:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.335+0000 7f6397fff640 1 -- 172.21.15.161:0/2212262798 reap_dead start 2024-03-28T23:37:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.335+0000 7f639dab5640 1 -- 172.21.15.161:0/2212262798 shutdown_connections 2024-03-28T23:37:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.335+0000 7f639dab5640 1 -- 172.21.15.161:0/2212262798 >> 172.21.15.161:0/2212262798 conn(0x7f639806b150 msgr2=0x7f639806b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.335+0000 7f639dab5640 1 -- 172.21.15.161:0/2212262798 shutdown_connections 2024-03-28T23:37:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.335+0000 7f639dab5640 1 -- 172.21.15.161:0/2212262798 wait complete. 2024-03-28T23:37:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.336+0000 7f639dab5640 1 Processor -- start 2024-03-28T23:37:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.337+0000 7f639dab5640 1 -- start start 2024-03-28T23:37:18.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.337+0000 7f639dab5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63800039b0 0x7f6398070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.337+0000 7f639dab5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63981ed150 0x7f63981567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.337+0000 7f639dab5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63981e99b0 0x7f6398070fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.337+0000 7f639dab5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f63980c50a0 con 0x7f63981e99b0 2024-03-28T23:37:18.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.337+0000 7f639dab5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f6398153740 con 0x7f63800039b0 2024-03-28T23:37:18.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.337+0000 7f639dab5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f63980c5220 con 0x7f63981ed150 2024-03-28T23:37:18.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.337+0000 7f6397fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63981e99b0 0x7f6398070fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.337+0000 7f6397fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63981e99b0 0x7f6398070fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55914/0 (socket says 172.21.15.161:55914) 2024-03-28T23:37:18.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.337+0000 7f6397fff640 1 -- 172.21.15.161:0/4238356320 learned_addr learned my addr 172.21.15.161:0/4238356320 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:18.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.338+0000 7f63977fe640 1 --2- 172.21.15.161:0/4238356320 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63800039b0 0x7f6398070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.338+0000 7f6396ffd640 1 --2- 172.21.15.161:0/4238356320 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63981ed150 0x7f63981567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.338+0000 7f6397fff640 1 -- 172.21.15.161:0/4238356320 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63981ed150 msgr2=0x7f63981567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.338+0000 7f6397fff640 1 --2- 172.21.15.161:0/4238356320 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63981ed150 0x7f63981567a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.338+0000 7f6397fff640 1 -- 172.21.15.161:0/4238356320 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63800039b0 msgr2=0x7f6398070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.338+0000 7f6397fff640 1 --2- 172.21.15.161:0/4238356320 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63800039b0 0x7f6398070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.338+0000 7f6397fff640 1 -- 172.21.15.161:0/4238356320 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63981ed5d0 con 0x7f63981e99b0 2024-03-28T23:37:18.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.338+0000 7f6397fff640 1 --2- 172.21.15.161:0/4238356320 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63981e99b0 0x7f6398070fb0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f63980705c0 tx=0x7f63900115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:18.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.338+0000 7f637ffff640 1 -- 172.21.15.161:0/4238356320 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6390011de0 con 0x7f63981e99b0 2024-03-28T23:37:18.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.338+0000 7f637ffff640 1 -- 172.21.15.161:0/4238356320 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f639001b070 con 0x7f63981e99b0 2024-03-28T23:37:18.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.338+0000 7f637ffff640 1 -- 172.21.15.161:0/4238356320 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6390003990 con 0x7f63981e99b0 2024-03-28T23:37:18.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.338+0000 7f639dab5640 1 -- 172.21.15.161:0/4238356320 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63981e91d0 con 0x7f63981e99b0 2024-03-28T23:37:18.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.338+0000 7f639dab5640 1 -- 172.21.15.161:0/4238356320 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63980733a0 con 0x7f63981e99b0 2024-03-28T23:37:18.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.339+0000 7f639dab5640 1 -- 172.21.15.161:0/4238356320 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6398068f90 con 0x7f63981e99b0 2024-03-28T23:37:18.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.339+0000 7f637ffff640 1 -- 172.21.15.161:0/4238356320 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6390005d10 con 0x7f63981e99b0 2024-03-28T23:37:18.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.340+0000 7f637ffff640 1 --2- 172.21.15.161:0/4238356320 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f638406b4d0 0x7f638406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.340+0000 7f637ffff640 1 -- 172.21.15.161:0/4238356320 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7f639008a780 con 0x7f63981e99b0 2024-03-28T23:37:18.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.340+0000 7f63977fe640 1 --2- 172.21.15.161:0/4238356320 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f638406b4d0 0x7f638406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.340+0000 7f637ffff640 1 --2- 172.21.15.161:0/4238356320 >> [v2:172.21.15.161:6808/3120803198,v1:172.21.15.161:6809/3120803198] conn(0x7f6384071c90 0x7f63840740d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.340+0000 7f637ffff640 1 -- 172.21.15.161:0/4238356320 --> [v2:172.21.15.161:6808/3120803198,v1:172.21.15.161:6809/3120803198] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f63840747a0 con 0x7f6384071c90 2024-03-28T23:37:18.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.340+0000 7f637ffff640 1 -- 172.21.15.161:0/4238356320 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=168) ==== 24+0+0 (secure 0 0 0) 0x7f6390091050 con 0x7f63981e99b0 2024-03-28T23:37:18.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.340+0000 7f6396ffd640 1 --2- 172.21.15.161:0/4238356320 >> [v2:172.21.15.161:6808/3120803198,v1:172.21.15.161:6809/3120803198] conn(0x7f6384071c90 0x7f63840740d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.340+0000 7f63977fe640 1 --2- 172.21.15.161:0/4238356320 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f638406b4d0 0x7f638406d990 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f6388036000 tx=0x7f6388030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:18.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.340+0000 7f6396ffd640 1 --2- 172.21.15.161:0/4238356320 >> [v2:172.21.15.161:6808/3120803198,v1:172.21.15.161:6809/3120803198] conn(0x7f6384071c90 0x7f63840740d0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:18.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.341+0000 7f637ffff640 1 -- 172.21.15.161:0/4238356320 <== osd.2 v2:172.21.15.161:6808/3120803198 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63840747a0 con 0x7f6384071c90 2024-03-28T23:37:18.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.354+0000 7f639dab5640 1 -- 172.21.15.161:0/4238356320 --> [v2:172.21.15.161:6808/3120803198,v1:172.21.15.161:6809/3120803198] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f639806d7c0 con 0x7f6384071c90 2024-03-28T23:37:18.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.354+0000 7f637ffff640 1 -- 172.21.15.161:0/4238356320 <== osd.2 v2:172.21.15.161:6808/3120803198 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f639806d7c0 con 0x7f6384071c90 2024-03-28T23:37:18.355 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:37:18.355 INFO:teuthology.orchestra.run.smithi161.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-28T23:37:18.355 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:37:18.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.354+0000 7f639dab5640 1 -- 172.21.15.161:0/4238356320 >> [v2:172.21.15.161:6808/3120803198,v1:172.21.15.161:6809/3120803198] conn(0x7f6384071c90 msgr2=0x7f63840740d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.354+0000 7f639dab5640 1 --2- 172.21.15.161:0/4238356320 >> [v2:172.21.15.161:6808/3120803198,v1:172.21.15.161:6809/3120803198] conn(0x7f6384071c90 0x7f63840740d0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.355+0000 7f639dab5640 1 -- 172.21.15.161:0/4238356320 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f638406b4d0 msgr2=0x7f638406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.355+0000 7f639dab5640 1 --2- 172.21.15.161:0/4238356320 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f638406b4d0 0x7f638406d990 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f6388036000 tx=0x7f6388030000 comp rx=0 tx=0).stop 2024-03-28T23:37:18.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.355+0000 7f639dab5640 1 -- 172.21.15.161:0/4238356320 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63981e99b0 msgr2=0x7f6398070fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.355+0000 7f639dab5640 1 --2- 172.21.15.161:0/4238356320 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63981e99b0 0x7f6398070fb0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f63980705c0 tx=0x7f63900115a0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.355+0000 7f6397fff640 1 -- 172.21.15.161:0/4238356320 reap_dead start 2024-03-28T23:37:18.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.355+0000 7f639dab5640 1 -- 172.21.15.161:0/4238356320 shutdown_connections 2024-03-28T23:37:18.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.355+0000 7f639dab5640 1 -- 172.21.15.161:0/4238356320 >> 172.21.15.161:0/4238356320 conn(0x7f639806b150 msgr2=0x7f63980c3130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:18.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.355+0000 7f639dab5640 1 -- 172.21.15.161:0/4238356320 shutdown_connections 2024-03-28T23:37:18.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.355+0000 7f639dab5640 1 -- 172.21.15.161:0/4238356320 wait complete. 2024-03-28T23:37:18.365 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-28T23:37:18.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.467+0000 7f9304f25640 1 Processor -- start 2024-03-28T23:37:18.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.467+0000 7f9304f25640 1 -- start start 2024-03-28T23:37:18.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.468+0000 7f9304f25640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93000c5410 0x7f93000c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.468+0000 7f9304f25640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f93001553a0 0x7f9300155820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.468+0000 7f9304f25640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9300158a30 con 0x7f9300152890 2024-03-28T23:37:18.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.468+0000 7f9304f25640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f93000c8480 con 0x7f93001553a0 2024-03-28T23:37:18.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.468+0000 7f9304f25640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f93000c5d90 con 0x7f93000c5410 2024-03-28T23:37:18.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.468+0000 7f92fed9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f93001553a0 0x7f9300155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.468+0000 7f92fe59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f9300152890 0x7f93000c8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41710/0 (socket says 172.21.15.161:41710) 2024-03-28T23:37:18.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.468+0000 7f92fe59c640 1 -- 172.21.15.161:0/3124450819 learned_addr learned my addr 172.21.15.161:0/3124450819 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:18.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.468+0000 7f92fdd9b640 1 --2- 172.21.15.161:0/3124450819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93000c5410 0x7f93000c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.469+0000 7f92fed9d640 1 -- 172.21.15.161:0/3124450819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93000c5410 msgr2=0x7f93000c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:18.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.469+0000 7f92fed9d640 1 --2- 172.21.15.161:0/3124450819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93000c5410 0x7f93000c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.469+0000 7f92fed9d640 1 -- 172.21.15.161:0/3124450819 >> v1:172.21.15.161:6789/0 conn(0x7f9300152890 legacy=0x7f93000c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.469+0000 7f92fed9d640 1 -- 172.21.15.161:0/3124450819 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9300158d10 con 0x7f93001553a0 2024-03-28T23:37:18.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.469+0000 7f92fd59a640 1 -- 172.21.15.161:0/3124450819 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2588641459 0 0) 0x7f9300158a30 con 0x7f9300152890 2024-03-28T23:37:18.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.469+0000 7f92fed9d640 1 --2- 172.21.15.161:0/3124450819 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93001553a0 0x7f9300155820 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f92f80097a0 tx=0x7f92f802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac1385943b02167a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:18.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.469+0000 7f92fd59a640 1 -- 172.21.15.161:0/3124450819 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f80315a0 con 0x7f93001553a0 2024-03-28T23:37:18.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.470+0000 7f92fd59a640 1 -- 172.21.15.161:0/3124450819 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92f8037070 con 0x7f93001553a0 2024-03-28T23:37:18.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.470+0000 7f92fd59a640 1 -- 172.21.15.161:0/3124450819 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f8031960 con 0x7f93001553a0 2024-03-28T23:37:18.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.470+0000 7f9304f25640 1 -- 172.21.15.161:0/3124450819 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93001553a0 msgr2=0x7f9300155820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.470+0000 7f9304f25640 1 --2- 172.21.15.161:0/3124450819 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93001553a0 0x7f9300155820 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f92f80097a0 tx=0x7f92f802b840 comp rx=0 tx=0).stop 2024-03-28T23:37:18.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.470+0000 7f9304f25640 1 -- 172.21.15.161:0/3124450819 shutdown_connections 2024-03-28T23:37:18.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.470+0000 7f9304f25640 1 --2- 172.21.15.161:0/3124450819 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93001553a0 0x7f9300155820 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.470+0000 7f9304f25640 1 --2- 172.21.15.161:0/3124450819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93000c5410 0x7f93000c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.470+0000 7f9304f25640 1 -- 172.21.15.161:0/3124450819 >> 172.21.15.161:0/3124450819 conn(0x7f93000ac620 msgr2=0x7f93000aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:18.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.470+0000 7f9304f25640 1 -- 172.21.15.161:0/3124450819 shutdown_connections 2024-03-28T23:37:18.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.470+0000 7f9304f25640 1 -- 172.21.15.161:0/3124450819 wait complete. 2024-03-28T23:37:18.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.471+0000 7f9304f25640 1 Processor -- start 2024-03-28T23:37:18.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.471+0000 7f9304f25640 1 -- start start 2024-03-28T23:37:18.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.472+0000 7f9304f25640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93000c5410 0x7f930014cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.472+0000 7f9304f25640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9300152890 0x7f930014ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.472+0000 7f9304f25640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93001553a0 0x7f9300150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.472+0000 7f9304f25640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f93001592d0 con 0x7f93000c5410 2024-03-28T23:37:18.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.472+0000 7f9304f25640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f9300159640 con 0x7f9300152890 2024-03-28T23:37:18.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.472+0000 7f9304f25640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f93000c90a0 con 0x7f93001553a0 2024-03-28T23:37:18.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.472+0000 7f92fe59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93000c5410 0x7f930014cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.472+0000 7f92fe59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93000c5410 0x7f930014cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55936/0 (socket says 172.21.15.161:55936) 2024-03-28T23:37:18.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.472+0000 7f92fe59c640 1 -- 172.21.15.161:0/3663105835 learned_addr learned my addr 172.21.15.161:0/3663105835 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:18.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.472+0000 7f92fed9d640 1 --2- 172.21.15.161:0/3663105835 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93001553a0 0x7f9300150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.472+0000 7f92fe59c640 1 -- 172.21.15.161:0/3663105835 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93001553a0 msgr2=0x7f9300150c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.472+0000 7f92fdd9b640 1 --2- 172.21.15.161:0/3663105835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9300152890 0x7f930014ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.473+0000 7f92fe59c640 1 --2- 172.21.15.161:0/3663105835 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93001553a0 0x7f9300150c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.473+0000 7f92fe59c640 1 -- 172.21.15.161:0/3663105835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9300152890 msgr2=0x7f930014ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.473+0000 7f92fe59c640 1 --2- 172.21.15.161:0/3663105835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9300152890 0x7f930014ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.473+0000 7f92fe59c640 1 -- 172.21.15.161:0/3663105835 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93001ea5c0 con 0x7f93000c5410 2024-03-28T23:37:18.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.473+0000 7f92fdd9b640 1 --2- 172.21.15.161:0/3663105835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9300152890 0x7f930014ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:37:18.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.473+0000 7f92fe59c640 1 --2- 172.21.15.161:0/3663105835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93000c5410 0x7f930014cb30 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f92f000c890 tx=0x7f92f000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:18.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.473+0000 7f92eeffd640 1 -- 172.21.15.161:0/3663105835 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f0018070 con 0x7f93000c5410 2024-03-28T23:37:18.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.473+0000 7f92eeffd640 1 -- 172.21.15.161:0/3663105835 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92f000a5e0 con 0x7f93000c5410 2024-03-28T23:37:18.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.473+0000 7f9304f25640 1 -- 172.21.15.161:0/3663105835 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93001ea880 con 0x7f93000c5410 2024-03-28T23:37:18.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.473+0000 7f92eeffd640 1 -- 172.21.15.161:0/3663105835 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f000a9d0 con 0x7f93000c5410 2024-03-28T23:37:18.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.473+0000 7f9304f25640 1 -- 172.21.15.161:0/3663105835 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9300150430 con 0x7f93000c5410 2024-03-28T23:37:18.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.474+0000 7f9304f25640 1 -- 172.21.15.161:0/3663105835 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f930004f4b0 con 0x7f93000c5410 2024-03-28T23:37:18.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.474+0000 7f92eeffd640 1 -- 172.21.15.161:0/3663105835 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f92f000ab70 con 0x7f93000c5410 2024-03-28T23:37:18.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.475+0000 7f92eeffd640 1 --2- 172.21.15.161:0/3663105835 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f92e006b4d0 0x7f92e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.475+0000 7f92eeffd640 1 -- 172.21.15.161:0/3663105835 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7f92f00898c0 con 0x7f93000c5410 2024-03-28T23:37:18.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.475+0000 7f92fdd9b640 1 --2- 172.21.15.161:0/3663105835 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f92e006b4d0 0x7f92e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.475+0000 7f92fdd9b640 1 --2- 172.21.15.161:0/3663105835 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f92e006b4d0 0x7f92e006d990 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f93000746a0 tx=0x7f92f4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:18.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.479+0000 7f92eeffd640 1 -- 172.21.15.161:0/3663105835 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f92f0058680 con 0x7f93000c5410 2024-03-28T23:37:18.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.650+0000 7f9304f25640 1 -- 172.21.15.161:0/3663105835 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f93000c8120 con 0x7f92e006b4d0 2024-03-28T23:37:18.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.650+0000 7f92eeffd640 1 -- 172.21.15.161:0/3663105835 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f93000c8120 con 0x7f92e006b4d0 2024-03-28T23:37:18.651 INFO:teuthology.orchestra.run.smithi161.stderr:instructed osd(s) 2 to deep-scrub 2024-03-28T23:37:18.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.652+0000 7f9304f25640 1 -- 172.21.15.161:0/3663105835 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f92e006b4d0 msgr2=0x7f92e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.652+0000 7f9304f25640 1 --2- 172.21.15.161:0/3663105835 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f92e006b4d0 0x7f92e006d990 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f93000746a0 tx=0x7f92f4008040 comp rx=0 tx=0).stop 2024-03-28T23:37:18.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.652+0000 7f9304f25640 1 -- 172.21.15.161:0/3663105835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93000c5410 msgr2=0x7f930014cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.652+0000 7f9304f25640 1 --2- 172.21.15.161:0/3663105835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93000c5410 0x7f930014cb30 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f92f000c890 tx=0x7f92f000cd50 comp rx=0 tx=0).stop 2024-03-28T23:37:18.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.652+0000 7f9304f25640 1 -- 172.21.15.161:0/3663105835 shutdown_connections 2024-03-28T23:37:18.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.652+0000 7f9304f25640 1 --2- 172.21.15.161:0/3663105835 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f92e006b4d0 0x7f92e006d990 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.652+0000 7f9304f25640 1 --2- 172.21.15.161:0/3663105835 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93001553a0 0x7f9300150c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.652+0000 7f9304f25640 1 --2- 172.21.15.161:0/3663105835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9300152890 0x7f930014ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.652+0000 7f9304f25640 1 --2- 172.21.15.161:0/3663105835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93000c5410 0x7f930014cb30 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.652+0000 7f9304f25640 1 -- 172.21.15.161:0/3663105835 >> 172.21.15.161:0/3663105835 conn(0x7f93000ac620 msgr2=0x7f9300153a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:18.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.652+0000 7f9304f25640 1 -- 172.21.15.161:0/3663105835 shutdown_connections 2024-03-28T23:37:18.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.652+0000 7f9304f25640 1 -- 172.21.15.161:0/3663105835 wait complete. 2024-03-28T23:37:18.663 INFO:tasks.ceph:Scrubbing osd.3 2024-03-28T23:37:18.664 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-28T23:37:18.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.768+0000 7f3dcc20b640 1 Processor -- start 2024-03-28T23:37:18.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.769+0000 7f3dcc20b640 1 -- start start 2024-03-28T23:37:18.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.769+0000 7f3dcc20b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3dc406b440 0x7f3dc4155d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.769+0000 7f3dcc20b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dc406bb70 con 0x7f3dc40c53e0 2024-03-28T23:37:18.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.769+0000 7f3dcc20b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dc406be50 con 0x7f3dc4152740 2024-03-28T23:37:18.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.769+0000 7f3dcc20b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3dc40c8960 con 0x7f3dc406b440 2024-03-28T23:37:18.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.769+0000 7f3dca7a8640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f3dc40c53e0 0x7f3dc40c7470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41726/0 (socket says 172.21.15.161:41726) 2024-03-28T23:37:18.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.769+0000 7f3dca7a8640 1 -- 172.21.15.161:0/3125608761 learned_addr learned my addr 172.21.15.161:0/3125608761 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:18.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.770+0000 7f3dc97a6640 1 --2- 172.21.15.161:0/3125608761 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3dc406b440 0x7f3dc4155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.770+0000 7f3dc8fa5640 1 -- 172.21.15.161:0/3125608761 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 450606404 0 0) 0x7f3dc406bb70 con 0x7f3dc40c53e0 2024-03-28T23:37:18.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.770+0000 7f3dc8fa5640 1 -- 172.21.15.161:0/3125608761 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3db0003520 con 0x7f3dc40c53e0 2024-03-28T23:37:18.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.770+0000 7f3dc8fa5640 1 -- 172.21.15.161:0/3125608761 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3436112344 0 0) 0x7f3db0003520 con 0x7f3dc40c53e0 2024-03-28T23:37:18.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.770+0000 7f3dc8fa5640 1 -- 172.21.15.161:0/3125608761 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3dc406b440 msgr2=0x7f3dc4155d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:18.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.770+0000 7f3dc8fa5640 1 --2- 172.21.15.161:0/3125608761 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3dc406b440 0x7f3dc4155d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.770+0000 7f3dc8fa5640 1 -- 172.21.15.161:0/3125608761 >> v1:172.21.15.170:6789/0 conn(0x7f3dc4152740 legacy=0x7f3dc40c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.770+0000 7f3dc8fa5640 1 -- 172.21.15.161:0/3125608761 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc40c8c40 con 0x7f3dc40c53e0 2024-03-28T23:37:18.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.770+0000 7f3dc8fa5640 1 -- 172.21.15.161:0/3125608761 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7f3dbc0019f0 con 0x7f3dc40c53e0 2024-03-28T23:37:18.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.770+0000 7f3dc8fa5640 1 -- 172.21.15.161:0/3125608761 >> v1:172.21.15.161:6789/0 conn(0x7f3dc40c53e0 legacy=0x7f3dc40c7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.770+0000 7f3dc8fa5640 1 --2- 172.21.15.161:0/3125608761 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3db00039e0 0x7f3db0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.770+0000 7f3dc8fa5640 1 --2- 172.21.15.161:0/3125608761 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3db0004390 0x7f3db0004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.770+0000 7f3dc8fa5640 1 --2- 172.21.15.161:0/3125608761 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3dc406b440 0x7f3db0005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.770+0000 7f3dc8fa5640 1 -- 172.21.15.161:0/3125608761 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc40c8960 con 0x7f3db00039e0 2024-03-28T23:37:18.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.770+0000 7f3dc8fa5640 1 -- 172.21.15.161:0/3125608761 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3db000ba70 con 0x7f3db0004390 2024-03-28T23:37:18.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.770+0000 7f3dc8fa5640 1 -- 172.21.15.161:0/3125608761 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3db000bc70 con 0x7f3dc406b440 2024-03-28T23:37:18.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.770+0000 7f3dc9fa7640 1 --2- 172.21.15.161:0/3125608761 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3db00039e0 0x7f3db0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.770+0000 7f3dc97a6640 1 --2- 172.21.15.161:0/3125608761 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3db0004390 0x7f3db0004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.770+0000 7f3dca7a8640 1 --2- 172.21.15.161:0/3125608761 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3dc406b440 0x7f3db0005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.770+0000 7f3dc9fa7640 1 -- 172.21.15.161:0/3125608761 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3dc406b440 msgr2=0x7f3db0005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:18.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.770+0000 7f3dc9fa7640 1 --2- 172.21.15.161:0/3125608761 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3dc406b440 0x7f3db0005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.770+0000 7f3dc9fa7640 1 -- 172.21.15.161:0/3125608761 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3db0004390 msgr2=0x7f3db0004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:18.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.770+0000 7f3dc9fa7640 1 --2- 172.21.15.161:0/3125608761 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3db0004390 0x7f3db0004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.771+0000 7f3dc9fa7640 1 -- 172.21.15.161:0/3125608761 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db0003520 con 0x7f3db00039e0 2024-03-28T23:37:18.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.771+0000 7f3dc9fa7640 1 --2- 172.21.15.161:0/3125608761 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3db00039e0 0x7f3db0003dc0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f3db4001860 tx=0x7f3db0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8b87163c759bd93 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:18.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.771+0000 7f3dc8fa5640 1 -- 172.21.15.161:0/3125608761 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db402f050 con 0x7f3db00039e0 2024-03-28T23:37:18.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.771+0000 7f3dc8fa5640 1 -- 172.21.15.161:0/3125608761 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3db4033650 con 0x7f3db00039e0 2024-03-28T23:37:18.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.771+0000 7f3dc8fa5640 1 -- 172.21.15.161:0/3125608761 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db4001430 con 0x7f3db00039e0 2024-03-28T23:37:18.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.772+0000 7f3dcc20b640 1 -- 172.21.15.161:0/3125608761 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3db00039e0 msgr2=0x7f3db0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.772+0000 7f3dcc20b640 1 --2- 172.21.15.161:0/3125608761 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3db00039e0 0x7f3db0003dc0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f3db4001860 tx=0x7f3db0005830 comp rx=0 tx=0).stop 2024-03-28T23:37:18.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.772+0000 7f3dca7a8640 1 -- 172.21.15.161:0/3125608761 reap_dead start 2024-03-28T23:37:18.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.772+0000 7f3dcc20b640 1 -- 172.21.15.161:0/3125608761 shutdown_connections 2024-03-28T23:37:18.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.772+0000 7f3dcc20b640 1 -- 172.21.15.161:0/3125608761 >> 172.21.15.161:0/3125608761 conn(0x7f3dc40cac40 msgr2=0x7f3dc40cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:18.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.772+0000 7f3dcc20b640 1 -- 172.21.15.161:0/3125608761 shutdown_connections 2024-03-28T23:37:18.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.772+0000 7f3dcc20b640 1 -- 172.21.15.161:0/3125608761 wait complete. 2024-03-28T23:37:18.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.773+0000 7f3dcc20b640 1 Processor -- start 2024-03-28T23:37:18.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.773+0000 7f3dcc20b640 1 -- start start 2024-03-28T23:37:18.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.774+0000 7f3dcc20b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3db00039e0 0x7f3dc40c3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.774+0000 7f3dcc20b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3dc40c13f0 0x7f3dc40c4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.774+0000 7f3dcc20b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3dc40be9a0 0x7f3dc40bee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.774+0000 7f3dcc20b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc40c5870 con 0x7f3dc40be9a0 2024-03-28T23:37:18.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.774+0000 7f3dcc20b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc406b880 con 0x7f3dc40c13f0 2024-03-28T23:37:18.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.774+0000 7f3dcc20b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f3dc40c7b80 con 0x7f3db00039e0 2024-03-28T23:37:18.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.774+0000 7f3dca7a8640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3dc40be9a0 0x7f3dc40bee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.774+0000 7f3dca7a8640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3dc40be9a0 0x7f3dc40bee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55958/0 (socket says 172.21.15.161:55958) 2024-03-28T23:37:18.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.774+0000 7f3dca7a8640 1 -- 172.21.15.161:0/3530909028 learned_addr learned my addr 172.21.15.161:0/3530909028 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:18.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.774+0000 7f3dc97a6640 1 --2- 172.21.15.161:0/3530909028 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3dc40c13f0 0x7f3dc40c4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.774+0000 7f3dc9fa7640 1 --2- 172.21.15.161:0/3530909028 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3db00039e0 0x7f3dc40c3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.774+0000 7f3dca7a8640 1 -- 172.21.15.161:0/3530909028 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3db00039e0 msgr2=0x7f3dc40c3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.774+0000 7f3dca7a8640 1 --2- 172.21.15.161:0/3530909028 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3db00039e0 0x7f3dc40c3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.774+0000 7f3dca7a8640 1 -- 172.21.15.161:0/3530909028 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3dc40c13f0 msgr2=0x7f3dc40c4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.774+0000 7f3dca7a8640 1 --2- 172.21.15.161:0/3530909028 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3dc40c13f0 0x7f3dc40c4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.774+0000 7f3dca7a8640 1 -- 172.21.15.161:0/3530909028 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc40c3d20 con 0x7f3dc40be9a0 2024-03-28T23:37:18.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.774+0000 7f3dca7a8640 1 --2- 172.21.15.161:0/3530909028 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3dc40be9a0 0x7f3dc40bee20 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f3dc40ca1e0 tx=0x7f3dbc0079d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:18.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.774+0000 7f3dba7fc640 1 -- 172.21.15.161:0/3530909028 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc00aa60 con 0x7f3dc40be9a0 2024-03-28T23:37:18.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.774+0000 7f3dba7fc640 1 -- 172.21.15.161:0/3530909028 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dbc018070 con 0x7f3dc40be9a0 2024-03-28T23:37:18.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.774+0000 7f3dba7fc640 1 -- 172.21.15.161:0/3530909028 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc0133f0 con 0x7f3dc40be9a0 2024-03-28T23:37:18.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.774+0000 7f3dcc20b640 1 -- 172.21.15.161:0/3530909028 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dc40bf800 con 0x7f3dc40be9a0 2024-03-28T23:37:18.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.775+0000 7f3dcc20b640 1 -- 172.21.15.161:0/3530909028 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dc40c4ad0 con 0x7f3dc40be9a0 2024-03-28T23:37:18.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.775+0000 7f3dcc20b640 1 -- 172.21.15.161:0/3530909028 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3dc4068f90 con 0x7f3dc40be9a0 2024-03-28T23:37:18.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.776+0000 7f3dba7fc640 1 -- 172.21.15.161:0/3530909028 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3dbc013590 con 0x7f3dc40be9a0 2024-03-28T23:37:18.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.776+0000 7f3dba7fc640 1 --2- 172.21.15.161:0/3530909028 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3da806b4d0 0x7f3da806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.776+0000 7f3dba7fc640 1 -- 172.21.15.161:0/3530909028 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7f3dbc08a620 con 0x7f3dc40be9a0 2024-03-28T23:37:18.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.776+0000 7f3dc9fa7640 1 --2- 172.21.15.161:0/3530909028 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3da806b4d0 0x7f3da806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.777+0000 7f3dba7fc640 1 --2- 172.21.15.161:0/3530909028 >> [v2:172.21.15.161:6824/3880429893,v1:172.21.15.161:6825/3880429893] conn(0x7f3da8071c90 0x7f3da80740d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.777+0000 7f3dba7fc640 1 -- 172.21.15.161:0/3530909028 --> [v2:172.21.15.161:6824/3880429893,v1:172.21.15.161:6825/3880429893] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3dbc021ec0 con 0x7f3da8071c90 2024-03-28T23:37:18.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.777+0000 7f3dba7fc640 1 -- 172.21.15.161:0/3530909028 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=168) ==== 24+0+0 (secure 0 0 0) 0x7f3dbc091050 con 0x7f3dc40be9a0 2024-03-28T23:37:18.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.777+0000 7f3dc97a6640 1 --2- 172.21.15.161:0/3530909028 >> [v2:172.21.15.161:6824/3880429893,v1:172.21.15.161:6825/3880429893] conn(0x7f3da8071c90 0x7f3da80740d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.777+0000 7f3dc9fa7640 1 --2- 172.21.15.161:0/3530909028 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3da806b4d0 0x7f3da806d990 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f3db4036000 tx=0x7f3db4031000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:18.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.777+0000 7f3dc97a6640 1 --2- 172.21.15.161:0/3530909028 >> [v2:172.21.15.161:6824/3880429893,v1:172.21.15.161:6825/3880429893] conn(0x7f3da8071c90 0x7f3da80740d0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:18.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.778+0000 7f3dba7fc640 1 -- 172.21.15.161:0/3530909028 <== osd.3 v2:172.21.15.161:6824/3880429893 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3dbc021ec0 con 0x7f3da8071c90 2024-03-28T23:37:18.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.791+0000 7f3dcc20b640 1 -- 172.21.15.161:0/3530909028 --> [v2:172.21.15.161:6824/3880429893,v1:172.21.15.161:6825/3880429893] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f3dc40ca4f0 con 0x7f3da8071c90 2024-03-28T23:37:18.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.791+0000 7f3dba7fc640 1 -- 172.21.15.161:0/3530909028 <== osd.3 v2:172.21.15.161:6824/3880429893 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f3dc40ca4f0 con 0x7f3da8071c90 2024-03-28T23:37:18.791 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:37:18.792 INFO:teuthology.orchestra.run.smithi161.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-28T23:37:18.792 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:37:18.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.791+0000 7f3dcc20b640 1 -- 172.21.15.161:0/3530909028 >> [v2:172.21.15.161:6824/3880429893,v1:172.21.15.161:6825/3880429893] conn(0x7f3da8071c90 msgr2=0x7f3da80740d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.791+0000 7f3dcc20b640 1 --2- 172.21.15.161:0/3530909028 >> [v2:172.21.15.161:6824/3880429893,v1:172.21.15.161:6825/3880429893] conn(0x7f3da8071c90 0x7f3da80740d0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.791+0000 7f3dcc20b640 1 -- 172.21.15.161:0/3530909028 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3da806b4d0 msgr2=0x7f3da806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.791+0000 7f3dcc20b640 1 --2- 172.21.15.161:0/3530909028 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f3da806b4d0 0x7f3da806d990 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f3db4036000 tx=0x7f3db4031000 comp rx=0 tx=0).stop 2024-03-28T23:37:18.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.791+0000 7f3dcc20b640 1 -- 172.21.15.161:0/3530909028 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3dc40be9a0 msgr2=0x7f3dc40bee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.791+0000 7f3dcc20b640 1 --2- 172.21.15.161:0/3530909028 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3dc40be9a0 0x7f3dc40bee20 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f3dc40ca1e0 tx=0x7f3dbc0079d0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.791+0000 7f3dca7a8640 1 -- 172.21.15.161:0/3530909028 reap_dead start 2024-03-28T23:37:18.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.791+0000 7f3dcc20b640 1 -- 172.21.15.161:0/3530909028 shutdown_connections 2024-03-28T23:37:18.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.791+0000 7f3dcc20b640 1 -- 172.21.15.161:0/3530909028 >> 172.21.15.161:0/3530909028 conn(0x7f3dc40cac40 msgr2=0x7f3dc40c59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:18.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.791+0000 7f3dcc20b640 1 -- 172.21.15.161:0/3530909028 shutdown_connections 2024-03-28T23:37:18.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.791+0000 7f3dcc20b640 1 -- 172.21.15.161:0/3530909028 wait complete. 2024-03-28T23:37:18.802 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-28T23:37:18.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.904+0000 7f2cc13bc640 1 Processor -- start 2024-03-28T23:37:18.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.904+0000 7f2cc13bc640 1 -- start start 2024-03-28T23:37:18.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.905+0000 7f2cc13bc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2cbc152a80 0x7f2cbc155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.905+0000 7f2cc13bc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cbc06d9f0 0x7f2cbc0c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.905+0000 7f2cc13bc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2cbc06cb80 0x7f2cbc0c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.905+0000 7f2cc13bc640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2cbc156250 con 0x7f2cbc152a80 2024-03-28T23:37:18.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.905+0000 7f2cc13bc640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f2cbc06c6f0 con 0x7f2cbc06cb80 2024-03-28T23:37:18.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.905+0000 7f2cc13bc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2cbc0c98e0 con 0x7f2cbc06d9f0 2024-03-28T23:37:18.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.905+0000 7f2cbaffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2cbc152a80 0x7f2cbc155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.905+0000 7f2cbaffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2cbc152a80 0x7f2cbc155d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55964/0 (socket says 172.21.15.161:55964) 2024-03-28T23:37:18.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.905+0000 7f2cbaffd640 1 -- 172.21.15.161:0/901354427 learned_addr learned my addr 172.21.15.161:0/901354427 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:18.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.905+0000 7f2cbb7fe640 1 --2- 172.21.15.161:0/901354427 >> v2:172.21.15.170:3300/0 conn(0x7f2cbc06cb80 0x7f2cbc0c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.905+0000 7f2cb3fff640 1 --2- 172.21.15.161:0/901354427 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cbc06d9f0 0x7f2cbc0c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.905+0000 7f2cbaffd640 1 -- 172.21.15.161:0/901354427 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cbc06d9f0 msgr2=0x7f2cbc0c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:18.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.905+0000 7f2cbaffd640 1 --2- 172.21.15.161:0/901354427 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cbc06d9f0 0x7f2cbc0c93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.905+0000 7f2cbaffd640 1 -- 172.21.15.161:0/901354427 >> v2:172.21.15.170:3300/0 conn(0x7f2cbc06cb80 msgr2=0x7f2cbc0c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:18.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.905+0000 7f2cbaffd640 1 --2- 172.21.15.161:0/901354427 >> v2:172.21.15.170:3300/0 conn(0x7f2cbc06cb80 0x7f2cbc0c74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.905+0000 7f2cbaffd640 1 -- 172.21.15.161:0/901354427 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cbc06c3c0 con 0x7f2cbc152a80 2024-03-28T23:37:18.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.905+0000 7f2cbb7fe640 1 --2- 172.21.15.161:0/901354427 >> v2:172.21.15.170:3300/0 conn(0x7f2cbc06cb80 0x7f2cbc0c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:37:18.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.906+0000 7f2cbaffd640 1 --2- 172.21.15.161:0/901354427 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cbc152a80 0x7f2cbc155d10 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc0ad770 tx=0x7f2cac02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b11ceb81fee25c71 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:18.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.906+0000 7f2cba7fc640 1 -- 172.21.15.161:0/901354427 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cac038070 con 0x7f2cbc152a80 2024-03-28T23:37:18.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.906+0000 7f2cba7fc640 1 -- 172.21.15.161:0/901354427 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cac02f050 con 0x7f2cbc152a80 2024-03-28T23:37:18.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.906+0000 7f2cba7fc640 1 -- 172.21.15.161:0/901354427 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cac031970 con 0x7f2cbc152a80 2024-03-28T23:37:18.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.907+0000 7f2cc13bc640 1 -- 172.21.15.161:0/901354427 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cbc152a80 msgr2=0x7f2cbc155d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.907+0000 7f2cc13bc640 1 --2- 172.21.15.161:0/901354427 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cbc152a80 0x7f2cbc155d10 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc0ad770 tx=0x7f2cac02b7f0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.907+0000 7f2cc13bc640 1 -- 172.21.15.161:0/901354427 shutdown_connections 2024-03-28T23:37:18.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.907+0000 7f2cc13bc640 1 --2- 172.21.15.161:0/901354427 >> v2:172.21.15.170:3300/0 conn(0x7f2cbc06cb80 0x7f2cbc0c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.907+0000 7f2cc13bc640 1 --2- 172.21.15.161:0/901354427 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cbc06d9f0 0x7f2cbc0c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.907+0000 7f2cc13bc640 1 --2- 172.21.15.161:0/901354427 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cbc152a80 0x7f2cbc155d10 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.907+0000 7f2cc13bc640 1 -- 172.21.15.161:0/901354427 >> 172.21.15.161:0/901354427 conn(0x7f2cbc0ac720 msgr2=0x7f2cbc0acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:18.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.907+0000 7f2cc13bc640 1 -- 172.21.15.161:0/901354427 shutdown_connections 2024-03-28T23:37:18.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.907+0000 7f2cc13bc640 1 -- 172.21.15.161:0/901354427 wait complete. 2024-03-28T23:37:18.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.908+0000 7f2cc13bc640 1 Processor -- start 2024-03-28T23:37:18.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.908+0000 7f2cc13bc640 1 -- start start 2024-03-28T23:37:18.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.908+0000 7f2cc13bc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cbc06cb80 0x7f2cbc0b0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.908+0000 7f2cc13bc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cbc06d9f0 0x7f2cbc14b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.908+0000 7f2cc13bc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2cbc152a80 0x7f2cbc1ee9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.908+0000 7f2cc13bc640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2cbc0c58b0 con 0x7f2cbc06cb80 2024-03-28T23:37:18.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.908+0000 7f2cc13bc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f2cbc0c55d0 con 0x7f2cbc152a80 2024-03-28T23:37:18.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.908+0000 7f2cc13bc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f2cbc06d010 con 0x7f2cbc06d9f0 2024-03-28T23:37:18.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.909+0000 7f2cbaffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cbc06cb80 0x7f2cbc0b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.909+0000 7f2cbaffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cbc06cb80 0x7f2cbc0b0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55966/0 (socket says 172.21.15.161:55966) 2024-03-28T23:37:18.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.909+0000 7f2cbaffd640 1 -- 172.21.15.161:0/3680881283 learned_addr learned my addr 172.21.15.161:0/3680881283 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:18.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.909+0000 7f2cbb7fe640 1 --2- 172.21.15.161:0/3680881283 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2cbc152a80 0x7f2cbc1ee9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.909+0000 7f2cb3fff640 1 --2- 172.21.15.161:0/3680881283 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cbc06d9f0 0x7f2cbc14b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.909+0000 7f2cbaffd640 1 -- 172.21.15.161:0/3680881283 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cbc06d9f0 msgr2=0x7f2cbc14b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.909+0000 7f2cbaffd640 1 --2- 172.21.15.161:0/3680881283 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cbc06d9f0 0x7f2cbc14b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.909+0000 7f2cbaffd640 1 -- 172.21.15.161:0/3680881283 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2cbc152a80 msgr2=0x7f2cbc1ee9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:18.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.909+0000 7f2cbaffd640 1 --2- 172.21.15.161:0/3680881283 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2cbc152a80 0x7f2cbc1ee9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:18.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.909+0000 7f2cbaffd640 1 -- 172.21.15.161:0/3680881283 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cbc0af080 con 0x7f2cbc06cb80 2024-03-28T23:37:18.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.909+0000 7f2cbaffd640 1 --2- 172.21.15.161:0/3680881283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cbc06cb80 0x7f2cbc0b0140 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f2cac02b830 tx=0x7f2cac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:18.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.909+0000 7f2cb37fe640 1 -- 172.21.15.161:0/3680881283 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cac042070 con 0x7f2cbc06cb80 2024-03-28T23:37:18.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.909+0000 7f2cb37fe640 1 -- 172.21.15.161:0/3680881283 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cac03d890 con 0x7f2cbc06cb80 2024-03-28T23:37:18.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.909+0000 7f2cc13bc640 1 -- 172.21.15.161:0/3680881283 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cbc0ad9b0 con 0x7f2cbc06cb80 2024-03-28T23:37:18.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.909+0000 7f2cb37fe640 1 -- 172.21.15.161:0/3680881283 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cac038040 con 0x7f2cbc06cb80 2024-03-28T23:37:18.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.909+0000 7f2cc13bc640 1 -- 172.21.15.161:0/3680881283 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cbc0adf70 con 0x7f2cbc06cb80 2024-03-28T23:37:18.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.910+0000 7f2cc13bc640 1 -- 172.21.15.161:0/3680881283 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cbc0c1630 con 0x7f2cbc06cb80 2024-03-28T23:37:18.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.911+0000 7f2cb37fe640 1 -- 172.21.15.161:0/3680881283 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2cac045070 con 0x7f2cbc06cb80 2024-03-28T23:37:18.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.911+0000 7f2cb37fe640 1 --2- 172.21.15.161:0/3680881283 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2c9c06b4d0 0x7f2c9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:18.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.911+0000 7f2cb37fe640 1 -- 172.21.15.161:0/3680881283 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7f2cac0ac010 con 0x7f2cbc06cb80 2024-03-28T23:37:18.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.911+0000 7f2cb3fff640 1 --2- 172.21.15.161:0/3680881283 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2c9c06b4d0 0x7f2c9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:18.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.911+0000 7f2cb3fff640 1 --2- 172.21.15.161:0/3680881283 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2c9c06b4d0 0x7f2c9c06d990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc0c96f0 tx=0x7f2ca4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:18.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:18.915+0000 7f2cb37fe640 1 -- 172.21.15.161:0/3680881283 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2cbc0c1630 con 0x7f2cbc06cb80 2024-03-28T23:37:19.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.086+0000 7f2cc13bc640 1 -- 172.21.15.161:0/3680881283 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f2cbc0c8bd0 con 0x7f2c9c06b4d0 2024-03-28T23:37:19.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.086+0000 7f2cb37fe640 1 -- 172.21.15.161:0/3680881283 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f2cbc0c8bd0 con 0x7f2c9c06b4d0 2024-03-28T23:37:19.087 INFO:teuthology.orchestra.run.smithi161.stderr:instructed osd(s) 3 to deep-scrub 2024-03-28T23:37:19.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.088+0000 7f2cc13bc640 1 -- 172.21.15.161:0/3680881283 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2c9c06b4d0 msgr2=0x7f2c9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.088+0000 7f2cc13bc640 1 --2- 172.21.15.161:0/3680881283 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2c9c06b4d0 0x7f2c9c06d990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc0c96f0 tx=0x7f2ca4008040 comp rx=0 tx=0).stop 2024-03-28T23:37:19.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.088+0000 7f2cc13bc640 1 -- 172.21.15.161:0/3680881283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cbc06cb80 msgr2=0x7f2cbc0b0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.088+0000 7f2cc13bc640 1 --2- 172.21.15.161:0/3680881283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cbc06cb80 0x7f2cbc0b0140 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f2cac02b830 tx=0x7f2cac0046c0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.088+0000 7f2cc13bc640 1 -- 172.21.15.161:0/3680881283 shutdown_connections 2024-03-28T23:37:19.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.088+0000 7f2cc13bc640 1 --2- 172.21.15.161:0/3680881283 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f2c9c06b4d0 0x7f2c9c06d990 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.088+0000 7f2cc13bc640 1 --2- 172.21.15.161:0/3680881283 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2cbc152a80 0x7f2cbc1ee9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.088+0000 7f2cc13bc640 1 --2- 172.21.15.161:0/3680881283 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cbc06d9f0 0x7f2cbc14b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.088+0000 7f2cc13bc640 1 --2- 172.21.15.161:0/3680881283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cbc06cb80 0x7f2cbc0b0140 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.088+0000 7f2cc13bc640 1 -- 172.21.15.161:0/3680881283 >> 172.21.15.161:0/3680881283 conn(0x7f2cbc0ac720 msgr2=0x7f2cbc0acb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:19.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.088+0000 7f2cc13bc640 1 -- 172.21.15.161:0/3680881283 shutdown_connections 2024-03-28T23:37:19.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.088+0000 7f2cc13bc640 1 -- 172.21.15.161:0/3680881283 wait complete. 2024-03-28T23:37:19.100 INFO:tasks.ceph:Scrubbing osd.4 2024-03-28T23:37:19.100 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-28T23:37:19.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.206+0000 7fb6d1f3e640 1 Processor -- start 2024-03-28T23:37:19.206 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.206+0000 7fb6d1f3e640 1 -- start start 2024-03-28T23:37:19.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.206+0000 7fb6d1f3e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb6cc0ac430 0x7fb6cc152880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.206+0000 7fb6d1f3e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb6cc0c19b0 0x7fb6cc0c1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.206+0000 7fb6d1f3e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6cc0c3540 con 0x7fb6cc0c30c0 2024-03-28T23:37:19.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.206+0000 7fb6d1f3e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb6cc152dc0 con 0x7fb6cc0c19b0 2024-03-28T23:37:19.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.206+0000 7fb6d1f3e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fb6cc0c2300 con 0x7fb6cc0ac430 2024-03-28T23:37:19.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.207+0000 7fb6cbfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb6cc0c19b0 0x7fb6cc0c1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.207+0000 7fb6cbfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb6cc0c19b0 0x7fb6cc0c1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55982/0 (socket says 172.21.15.161:55982) 2024-03-28T23:37:19.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.207+0000 7fb6caffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb6cc0c30c0 0x7fb6cc0c24d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41728/0 (socket says 172.21.15.161:41728) 2024-03-28T23:37:19.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.207+0000 7fb6cbfff640 1 -- 172.21.15.161:0/2883566003 learned_addr learned my addr 172.21.15.161:0/2883566003 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:19.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.207+0000 7fb6cb7fe640 1 --2- 172.21.15.161:0/2883566003 >> v2:172.21.15.170:3300/0 conn(0x7fb6cc0ac430 0x7fb6cc152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.207+0000 7fb6cbfff640 1 -- 172.21.15.161:0/2883566003 >> v2:172.21.15.170:3300/0 conn(0x7fb6cc0ac430 msgr2=0x7fb6cc152880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:19.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.207+0000 7fb6cbfff640 1 --2- 172.21.15.161:0/2883566003 >> v2:172.21.15.170:3300/0 conn(0x7fb6cc0ac430 0x7fb6cc152880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.207+0000 7fb6cbfff640 1 -- 172.21.15.161:0/2883566003 >> v1:172.21.15.161:6789/0 conn(0x7fb6cc0c30c0 legacy=0x7fb6cc0c24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.207+0000 7fb6cbfff640 1 -- 172.21.15.161:0/2883566003 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6cc0c3820 con 0x7fb6cc0c19b0 2024-03-28T23:37:19.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.207+0000 7fb6ca7fc640 1 -- 172.21.15.161:0/2883566003 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2515994372 0 0) 0x7fb6cc0c3540 con 0x7fb6cc0c30c0 2024-03-28T23:37:19.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.208+0000 7fb6cbfff640 1 --2- 172.21.15.161:0/2883566003 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6cc0c19b0 0x7fb6cc0c1dc0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc0a95f0 tx=0x7fb6c402e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8985d4acbc8617f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:19.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.208+0000 7fb6ca7fc640 1 -- 172.21.15.161:0/2883566003 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c402ea90 con 0x7fb6cc0c19b0 2024-03-28T23:37:19.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.208+0000 7fb6ca7fc640 1 -- 172.21.15.161:0/2883566003 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6c402ec30 con 0x7fb6cc0c19b0 2024-03-28T23:37:19.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.208+0000 7fb6ca7fc640 1 -- 172.21.15.161:0/2883566003 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c4009600 con 0x7fb6cc0c19b0 2024-03-28T23:37:19.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.208+0000 7fb6d1f3e640 1 -- 172.21.15.161:0/2883566003 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6cc0c19b0 msgr2=0x7fb6cc0c1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.208+0000 7fb6d1f3e640 1 --2- 172.21.15.161:0/2883566003 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6cc0c19b0 0x7fb6cc0c1dc0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc0a95f0 tx=0x7fb6c402e0f0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.208+0000 7fb6d1f3e640 1 -- 172.21.15.161:0/2883566003 shutdown_connections 2024-03-28T23:37:19.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.208+0000 7fb6d1f3e640 1 --2- 172.21.15.161:0/2883566003 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6cc0c19b0 0x7fb6cc0c1dc0 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.208+0000 7fb6d1f3e640 1 --2- 172.21.15.161:0/2883566003 >> v2:172.21.15.170:3300/0 conn(0x7fb6cc0ac430 0x7fb6cc152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.208+0000 7fb6d1f3e640 1 -- 172.21.15.161:0/2883566003 >> 172.21.15.161:0/2883566003 conn(0x7fb6cc06b150 msgr2=0x7fb6cc06b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:19.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.208+0000 7fb6d1f3e640 1 -- 172.21.15.161:0/2883566003 shutdown_connections 2024-03-28T23:37:19.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.208+0000 7fb6d1f3e640 1 -- 172.21.15.161:0/2883566003 wait complete. 2024-03-28T23:37:19.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.209+0000 7fb6d1f3e640 1 Processor -- start 2024-03-28T23:37:19.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.210+0000 7fb6d1f3e640 1 -- start start 2024-03-28T23:37:19.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.210+0000 7fb6d1f3e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6cc0ac430 0x7fb6cc14aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.210+0000 7fb6d1f3e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6cc0c19b0 0x7fb6cc1edf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.210+0000 7fb6d1f3e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6cc0c30c0 0x7fb6cc147ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.210+0000 7fb6d1f3e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb6cc0c1490 con 0x7fb6cc0ac430 2024-03-28T23:37:19.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.210+0000 7fb6d1f3e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fb6cc0c11b0 con 0x7fb6cc0c19b0 2024-03-28T23:37:19.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.210+0000 7fb6d1f3e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fb6cc0c6440 con 0x7fb6cc0c30c0 2024-03-28T23:37:19.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.210+0000 7fb6cb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6cc0ac430 0x7fb6cc14aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.210+0000 7fb6cb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6cc0ac430 0x7fb6cc14aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55984/0 (socket says 172.21.15.161:55984) 2024-03-28T23:37:19.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.210+0000 7fb6cb7fe640 1 -- 172.21.15.161:0/3602992219 learned_addr learned my addr 172.21.15.161:0/3602992219 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:19.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.211+0000 7fb6caffd640 1 --2- 172.21.15.161:0/3602992219 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6cc0c19b0 0x7fb6cc1edf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.211+0000 7fb6cbfff640 1 --2- 172.21.15.161:0/3602992219 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6cc0c30c0 0x7fb6cc147ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.211+0000 7fb6cb7fe640 1 -- 172.21.15.161:0/3602992219 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6cc0c30c0 msgr2=0x7fb6cc147ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.211+0000 7fb6cb7fe640 1 --2- 172.21.15.161:0/3602992219 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6cc0c30c0 0x7fb6cc147ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.211+0000 7fb6cb7fe640 1 -- 172.21.15.161:0/3602992219 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6cc0c19b0 msgr2=0x7fb6cc1edf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.211+0000 7fb6cb7fe640 1 --2- 172.21.15.161:0/3602992219 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6cc0c19b0 0x7fb6cc1edf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.211+0000 7fb6cb7fe640 1 -- 172.21.15.161:0/3602992219 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6cc153440 con 0x7fb6cc0ac430 2024-03-28T23:37:19.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.211+0000 7fb6cb7fe640 1 --2- 172.21.15.161:0/3602992219 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6cc0ac430 0x7fb6cc14aed0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc00e870 tx=0x7fb6bc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:19.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.211+0000 7fb6b3fff640 1 -- 172.21.15.161:0/3602992219 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc017070 con 0x7fb6cc0ac430 2024-03-28T23:37:19.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.211+0000 7fb6b3fff640 1 -- 172.21.15.161:0/3602992219 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6bc00c940 con 0x7fb6cc0ac430 2024-03-28T23:37:19.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.211+0000 7fb6d1f3e640 1 -- 172.21.15.161:0/3602992219 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6cc153720 con 0x7fb6cc0ac430 2024-03-28T23:37:19.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.211+0000 7fb6b3fff640 1 -- 172.21.15.161:0/3602992219 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc00cd30 con 0x7fb6cc0ac430 2024-03-28T23:37:19.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.211+0000 7fb6d1f3e640 1 -- 172.21.15.161:0/3602992219 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6cc155430 con 0x7fb6cc0ac430 2024-03-28T23:37:19.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.212+0000 7fb6d1f3e640 1 -- 172.21.15.161:0/3602992219 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb6cc154b20 con 0x7fb6cc0ac430 2024-03-28T23:37:19.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.212+0000 7fb6b3fff640 1 -- 172.21.15.161:0/3602992219 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb6bc01a070 con 0x7fb6cc0ac430 2024-03-28T23:37:19.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.213+0000 7fb6b3fff640 1 --2- 172.21.15.161:0/3602992219 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb6b806b500 0x7fb6b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.213+0000 7fb6caffd640 1 --2- 172.21.15.161:0/3602992219 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb6b806b500 0x7fb6b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.213+0000 7fb6b3fff640 1 -- 172.21.15.161:0/3602992219 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7fb6bc08a460 con 0x7fb6cc0ac430 2024-03-28T23:37:19.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.213+0000 7fb6b3fff640 1 --2- 172.21.15.161:0/3602992219 >> [v2:172.21.15.170:6800/1557068995,v1:172.21.15.170:6801/1557068995] conn(0x7fb6b8071cc0 0x7fb6b8074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.213+0000 7fb6b3fff640 1 -- 172.21.15.161:0/3602992219 --> [v2:172.21.15.170:6800/1557068995,v1:172.21.15.170:6801/1557068995] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb6bc016ec0 con 0x7fb6b8071cc0 2024-03-28T23:37:19.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.213+0000 7fb6b3fff640 1 -- 172.21.15.161:0/3602992219 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=168) ==== 24+0+0 (secure 0 0 0) 0x7fb6bc091050 con 0x7fb6cc0ac430 2024-03-28T23:37:19.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.213+0000 7fb6caffd640 1 --2- 172.21.15.161:0/3602992219 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb6b806b500 0x7fb6b806d9c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0007fd0 tx=0x7fb6c0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:19.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.213+0000 7fb6cbfff640 1 --2- 172.21.15.161:0/3602992219 >> [v2:172.21.15.170:6800/1557068995,v1:172.21.15.170:6801/1557068995] conn(0x7fb6b8071cc0 0x7fb6b8074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.214+0000 7fb6cbfff640 1 --2- 172.21.15.161:0/3602992219 >> [v2:172.21.15.170:6800/1557068995,v1:172.21.15.170:6801/1557068995] conn(0x7fb6b8071cc0 0x7fb6b8074100 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:19.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.215+0000 7fb6b3fff640 1 -- 172.21.15.161:0/3602992219 <== osd.4 v2:172.21.15.170:6800/1557068995 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb6bc016ec0 con 0x7fb6b8071cc0 2024-03-28T23:37:19.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.228+0000 7fb6d1f3e640 1 -- 172.21.15.161:0/3602992219 --> [v2:172.21.15.170:6800/1557068995,v1:172.21.15.170:6801/1557068995] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb6cc1531f0 con 0x7fb6b8071cc0 2024-03-28T23:37:19.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.228+0000 7fb6b3fff640 1 -- 172.21.15.161:0/3602992219 <== osd.4 v2:172.21.15.170:6800/1557068995 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb6cc1531f0 con 0x7fb6b8071cc0 2024-03-28T23:37:19.228 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:37:19.229 INFO:teuthology.orchestra.run.smithi161.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-28T23:37:19.229 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:37:19.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.228+0000 7fb6d1f3e640 1 -- 172.21.15.161:0/3602992219 >> [v2:172.21.15.170:6800/1557068995,v1:172.21.15.170:6801/1557068995] conn(0x7fb6b8071cc0 msgr2=0x7fb6b8074100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.228+0000 7fb6d1f3e640 1 --2- 172.21.15.161:0/3602992219 >> [v2:172.21.15.170:6800/1557068995,v1:172.21.15.170:6801/1557068995] conn(0x7fb6b8071cc0 0x7fb6b8074100 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.228+0000 7fb6d1f3e640 1 -- 172.21.15.161:0/3602992219 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb6b806b500 msgr2=0x7fb6b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.228+0000 7fb6d1f3e640 1 --2- 172.21.15.161:0/3602992219 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fb6b806b500 0x7fb6b806d9c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0007fd0 tx=0x7fb6c0009040 comp rx=0 tx=0).stop 2024-03-28T23:37:19.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.228+0000 7fb6d1f3e640 1 -- 172.21.15.161:0/3602992219 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6cc0ac430 msgr2=0x7fb6cc14aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.228+0000 7fb6d1f3e640 1 --2- 172.21.15.161:0/3602992219 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6cc0ac430 0x7fb6cc14aed0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc00e870 tx=0x7fb6bc00ed30 comp rx=0 tx=0).stop 2024-03-28T23:37:19.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.228+0000 7fb6cbfff640 1 -- 172.21.15.161:0/3602992219 reap_dead start 2024-03-28T23:37:19.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.228+0000 7fb6d1f3e640 1 -- 172.21.15.161:0/3602992219 shutdown_connections 2024-03-28T23:37:19.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.228+0000 7fb6d1f3e640 1 -- 172.21.15.161:0/3602992219 >> 172.21.15.161:0/3602992219 conn(0x7fb6cc06b150 msgr2=0x7fb6cc14fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:19.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.229+0000 7fb6d1f3e640 1 -- 172.21.15.161:0/3602992219 shutdown_connections 2024-03-28T23:37:19.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.229+0000 7fb6d1f3e640 1 -- 172.21.15.161:0/3602992219 wait complete. 2024-03-28T23:37:19.239 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-28T23:37:19.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.341+0000 7f57d7b02640 1 Processor -- start 2024-03-28T23:37:19.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.341+0000 7f57d7b02640 1 -- start start 2024-03-28T23:37:19.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.342+0000 7f57d7b02640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f57d00aacd0 0x7f57d015d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.342+0000 7f57d7b02640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57d014c250 con 0x7f57d014be70 2024-03-28T23:37:19.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.342+0000 7f57d7b02640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57d01529c0 con 0x7f57d006ce80 2024-03-28T23:37:19.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.342+0000 7f57d7b02640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f57d00a9dc0 con 0x7f57d00aacd0 2024-03-28T23:37:19.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.342+0000 7f57d609f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f57d00aacd0 0x7f57d015d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.342+0000 7f57d589e640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f57d014be70 0x7f57d00ab1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41744/0 (socket says 172.21.15.161:41744) 2024-03-28T23:37:19.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.342+0000 7f57d589e640 1 -- 172.21.15.161:0/3472426248 learned_addr learned my addr 172.21.15.161:0/3472426248 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:19.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.342+0000 7f57d609f640 1 -- 172.21.15.161:0/3472426248 >> v1:172.21.15.170:6789/0 conn(0x7f57d006ce80 legacy=0x7f57d00aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.342+0000 7f57d609f640 1 -- 172.21.15.161:0/3472426248 >> v1:172.21.15.161:6789/0 conn(0x7f57d014be70 legacy=0x7f57d00ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.342+0000 7f57d609f640 1 -- 172.21.15.161:0/3472426248 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57d00aa0a0 con 0x7f57d00aacd0 2024-03-28T23:37:19.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.343+0000 7f57d609f640 1 --2- 172.21.15.161:0/3472426248 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57d00aacd0 0x7f57d015d750 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f57c80097a0 tx=0x7f57c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a751fa6350ecd6d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:19.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.343+0000 7f57d489c640 1 -- 172.21.15.161:0/3472426248 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c8035020 con 0x7f57d00aacd0 2024-03-28T23:37:19.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.343+0000 7f57d489c640 1 -- 172.21.15.161:0/3472426248 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57c8031830 con 0x7f57d00aacd0 2024-03-28T23:37:19.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.343+0000 7f57d489c640 1 -- 172.21.15.161:0/3472426248 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c8031c00 con 0x7f57d00aacd0 2024-03-28T23:37:19.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.344+0000 7f57d7b02640 1 -- 172.21.15.161:0/3472426248 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57d00aacd0 msgr2=0x7f57d015d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.344+0000 7f57d7b02640 1 --2- 172.21.15.161:0/3472426248 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57d00aacd0 0x7f57d015d750 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f57c80097a0 tx=0x7f57c802b8d0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.344+0000 7f57d7b02640 1 -- 172.21.15.161:0/3472426248 shutdown_connections 2024-03-28T23:37:19.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.344+0000 7f57d7b02640 1 --2- 172.21.15.161:0/3472426248 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57d00aacd0 0x7f57d015d750 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.344+0000 7f57d7b02640 1 -- 172.21.15.161:0/3472426248 >> 172.21.15.161:0/3472426248 conn(0x7f57d00c6670 msgr2=0x7f57d00c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:19.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.344+0000 7f57d7b02640 1 -- 172.21.15.161:0/3472426248 shutdown_connections 2024-03-28T23:37:19.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.344+0000 7f57d7b02640 1 -- 172.21.15.161:0/3472426248 wait complete. 2024-03-28T23:37:19.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.345+0000 7f57d7b02640 1 Processor -- start 2024-03-28T23:37:19.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.345+0000 7f57d7b02640 1 -- start start 2024-03-28T23:37:19.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.346+0000 7f57d7b02640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57d006ce80 0x7f57d00c1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.346+0000 7f57d7b02640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57d00aacd0 0x7f57d014d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.346+0000 7f57d7b02640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57d014be70 0x7f57d014d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.346+0000 7f57d7b02640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f57d00aba30 con 0x7f57d014be70 2024-03-28T23:37:19.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.346+0000 7f57d7b02640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f57d00ab8b0 con 0x7f57d006ce80 2024-03-28T23:37:19.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.346+0000 7f57d7b02640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f57d006b150 con 0x7f57d00aacd0 2024-03-28T23:37:19.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.346+0000 7f57d609f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57d014be70 0x7f57d014d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.346+0000 7f57d609f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57d014be70 0x7f57d014d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55996/0 (socket says 172.21.15.161:55996) 2024-03-28T23:37:19.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.346+0000 7f57d609f640 1 -- 172.21.15.161:0/3450730949 learned_addr learned my addr 172.21.15.161:0/3450730949 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:19.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.346+0000 7f57d589e640 1 --2- 172.21.15.161:0/3450730949 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57d006ce80 0x7f57d00c1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.346+0000 7f57d509d640 1 --2- 172.21.15.161:0/3450730949 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57d00aacd0 0x7f57d014d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.346+0000 7f57d609f640 1 -- 172.21.15.161:0/3450730949 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57d00aacd0 msgr2=0x7f57d014d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.346+0000 7f57d609f640 1 --2- 172.21.15.161:0/3450730949 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57d00aacd0 0x7f57d014d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.346+0000 7f57d609f640 1 -- 172.21.15.161:0/3450730949 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57d006ce80 msgr2=0x7f57d00c1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.346+0000 7f57d609f640 1 --2- 172.21.15.161:0/3450730949 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57d006ce80 0x7f57d00c1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.346+0000 7f57d609f640 1 -- 172.21.15.161:0/3450730949 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57d0071820 con 0x7f57d014be70 2024-03-28T23:37:19.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.346+0000 7f57d509d640 1 --2- 172.21.15.161:0/3450730949 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57d00aacd0 0x7f57d014d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:37:19.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.346+0000 7f57d609f640 1 --2- 172.21.15.161:0/3450730949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57d014be70 0x7f57d014d620 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f57c8034040 tx=0x7f57c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:19.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.347+0000 7f57c67fc640 1 -- 172.21.15.161:0/3450730949 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c8040070 con 0x7f57d014be70 2024-03-28T23:37:19.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.347+0000 7f57c67fc640 1 -- 172.21.15.161:0/3450730949 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57c80398c0 con 0x7f57d014be70 2024-03-28T23:37:19.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.347+0000 7f57d7b02640 1 -- 172.21.15.161:0/3450730949 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57d01e9970 con 0x7f57d014be70 2024-03-28T23:37:19.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.347+0000 7f57c67fc640 1 -- 172.21.15.161:0/3450730949 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c8035040 con 0x7f57d014be70 2024-03-28T23:37:19.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.347+0000 7f57d7b02640 1 -- 172.21.15.161:0/3450730949 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57d0071da0 con 0x7f57d014be70 2024-03-28T23:37:19.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.347+0000 7f57d7b02640 1 -- 172.21.15.161:0/3450730949 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57d004f4b0 con 0x7f57d014be70 2024-03-28T23:37:19.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.348+0000 7f57c67fc640 1 -- 172.21.15.161:0/3450730949 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f57c804b060 con 0x7f57d014be70 2024-03-28T23:37:19.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.348+0000 7f57c67fc640 1 --2- 172.21.15.161:0/3450730949 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f57b406b500 0x7f57b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.348+0000 7f57c67fc640 1 -- 172.21.15.161:0/3450730949 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7f57c80abb30 con 0x7f57d014be70 2024-03-28T23:37:19.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.348+0000 7f57d589e640 1 --2- 172.21.15.161:0/3450730949 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f57b406b500 0x7f57b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.349+0000 7f57d589e640 1 --2- 172.21.15.161:0/3450730949 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f57b406b500 0x7f57b406d9c0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f57cc005c00 tx=0x7f57cc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:19.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.353+0000 7f57c67fc640 1 -- 172.21.15.161:0/3450730949 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f57c807aa00 con 0x7f57d014be70 2024-03-28T23:37:19.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.527+0000 7f57d7b02640 1 -- 172.21.15.161:0/3450730949 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f57d00c0ff0 con 0x7f57b406b500 2024-03-28T23:37:19.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.527+0000 7f57c67fc640 1 -- 172.21.15.161:0/3450730949 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f57d00c0ff0 con 0x7f57b406b500 2024-03-28T23:37:19.528 INFO:teuthology.orchestra.run.smithi161.stderr:instructed osd(s) 4 to deep-scrub 2024-03-28T23:37:19.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.529+0000 7f57d7b02640 1 -- 172.21.15.161:0/3450730949 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f57b406b500 msgr2=0x7f57b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.529+0000 7f57d7b02640 1 --2- 172.21.15.161:0/3450730949 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f57b406b500 0x7f57b406d9c0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f57cc005c00 tx=0x7f57cc009040 comp rx=0 tx=0).stop 2024-03-28T23:37:19.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.529+0000 7f57d7b02640 1 -- 172.21.15.161:0/3450730949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57d014be70 msgr2=0x7f57d014d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.529+0000 7f57d7b02640 1 --2- 172.21.15.161:0/3450730949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57d014be70 0x7f57d014d620 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f57c8034040 tx=0x7f57c80046c0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.529+0000 7f57d7b02640 1 -- 172.21.15.161:0/3450730949 shutdown_connections 2024-03-28T23:37:19.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.529+0000 7f57d7b02640 1 --2- 172.21.15.161:0/3450730949 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f57b406b500 0x7f57b406d9c0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.529+0000 7f57d7b02640 1 --2- 172.21.15.161:0/3450730949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57d014be70 0x7f57d014d620 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.529+0000 7f57d7b02640 1 --2- 172.21.15.161:0/3450730949 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57d00aacd0 0x7f57d014d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.529+0000 7f57d7b02640 1 --2- 172.21.15.161:0/3450730949 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57d006ce80 0x7f57d00c1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.529+0000 7f57d7b02640 1 -- 172.21.15.161:0/3450730949 >> 172.21.15.161:0/3450730949 conn(0x7f57d00c6670 msgr2=0x7f57d014a410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:19.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.529+0000 7f57d7b02640 1 -- 172.21.15.161:0/3450730949 shutdown_connections 2024-03-28T23:37:19.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.529+0000 7f57d7b02640 1 -- 172.21.15.161:0/3450730949 wait complete. 2024-03-28T23:37:19.540 INFO:tasks.ceph:Scrubbing osd.5 2024-03-28T23:37:19.540 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-28T23:37:19.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.645+0000 7fed5c640640 1 Processor -- start 2024-03-28T23:37:19.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.645+0000 7fed5c640640 1 -- start start 2024-03-28T23:37:19.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.646+0000 7fed5c640640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fed540ac430 0x7fed54152880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.646+0000 7fed5c640640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed540c3540 con 0x7fed540c19b0 2024-03-28T23:37:19.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.646+0000 7fed5c640640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed540c3820 con 0x7fed540c30c0 2024-03-28T23:37:19.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.646+0000 7fed5c640640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fed54152dc0 con 0x7fed540ac430 2024-03-28T23:37:19.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.646+0000 7fed5abdd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fed540c19b0 0x7fed5415c780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41754/0 (socket says 172.21.15.161:41754) 2024-03-28T23:37:19.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.646+0000 7fed5abdd640 1 -- 172.21.15.161:0/4259018497 learned_addr learned my addr 172.21.15.161:0/4259018497 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:19.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.646+0000 7fed5a3dc640 1 --2- 172.21.15.161:0/4259018497 >> v2:172.21.15.170:3300/0 conn(0x7fed540ac430 0x7fed54152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.646+0000 7fed593da640 1 -- 172.21.15.161:0/4259018497 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1479831190 0 0) 0x7fed540c3540 con 0x7fed540c19b0 2024-03-28T23:37:19.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.646+0000 7fed593da640 1 -- 172.21.15.161:0/4259018497 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed40003520 con 0x7fed540c19b0 2024-03-28T23:37:19.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.646+0000 7fed593da640 1 -- 172.21.15.161:0/4259018497 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3099899811 0 0) 0x7fed40003520 con 0x7fed540c19b0 2024-03-28T23:37:19.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.647+0000 7fed593da640 1 -- 172.21.15.161:0/4259018497 >> v2:172.21.15.170:3300/0 conn(0x7fed540ac430 msgr2=0x7fed54152880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:19.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.647+0000 7fed593da640 1 --2- 172.21.15.161:0/4259018497 >> v2:172.21.15.170:3300/0 conn(0x7fed540ac430 0x7fed54152880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.647+0000 7fed593da640 1 -- 172.21.15.161:0/4259018497 >> v1:172.21.15.170:6789/0 conn(0x7fed540c30c0 legacy=0x7fed540c24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.647+0000 7fed593da640 1 -- 172.21.15.161:0/4259018497 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed540c3b00 con 0x7fed540c19b0 2024-03-28T23:37:19.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.647+0000 7fed593da640 1 -- 172.21.15.161:0/4259018497 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1141954955 0 0) 0x7fed4c0019f0 con 0x7fed540c19b0 2024-03-28T23:37:19.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.647+0000 7fed5a3dc640 1 --2- 172.21.15.161:0/4259018497 >> v2:172.21.15.170:3300/0 conn(0x7fed540ac430 0x7fed54152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:37:19.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.647+0000 7fed593da640 1 -- 172.21.15.161:0/4259018497 >> v1:172.21.15.161:6789/0 conn(0x7fed540c19b0 legacy=0x7fed5415c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.647+0000 7fed593da640 1 --2- 172.21.15.161:0/4259018497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed400039b0 0x7fed40003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.647+0000 7fed593da640 1 --2- 172.21.15.161:0/4259018497 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed40004800 0x7fed40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.647+0000 7fed593da640 1 --2- 172.21.15.161:0/4259018497 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed400051c0 0x7fed4000ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.647+0000 7fed593da640 1 -- 172.21.15.161:0/4259018497 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fed54152dc0 con 0x7fed400039b0 2024-03-28T23:37:19.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.647+0000 7fed593da640 1 -- 172.21.15.161:0/4259018497 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fed40010570 con 0x7fed40004800 2024-03-28T23:37:19.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.647+0000 7fed593da640 1 -- 172.21.15.161:0/4259018497 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fed40010770 con 0x7fed400051c0 2024-03-28T23:37:19.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.647+0000 7fed5a3dc640 1 --2- 172.21.15.161:0/4259018497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed400039b0 0x7fed40003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.647+0000 7fed59bdb640 1 --2- 172.21.15.161:0/4259018497 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed40004800 0x7fed40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.647+0000 7fed5abdd640 1 --2- 172.21.15.161:0/4259018497 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed400051c0 0x7fed4000ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.647+0000 7fed5a3dc640 1 -- 172.21.15.161:0/4259018497 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed400051c0 msgr2=0x7fed4000ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:19.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.647+0000 7fed5a3dc640 1 --2- 172.21.15.161:0/4259018497 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed400051c0 0x7fed4000ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.647+0000 7fed5a3dc640 1 -- 172.21.15.161:0/4259018497 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed40004800 msgr2=0x7fed40004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:19.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.647+0000 7fed5a3dc640 1 --2- 172.21.15.161:0/4259018497 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed40004800 0x7fed40004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.647+0000 7fed5a3dc640 1 -- 172.21.15.161:0/4259018497 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed40003520 con 0x7fed400039b0 2024-03-28T23:37:19.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.647+0000 7fed5abdd640 1 -- 172.21.15.161:0/4259018497 reap_dead start 2024-03-28T23:37:19.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.648+0000 7fed5a3dc640 1 --2- 172.21.15.161:0/4259018497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed400039b0 0x7fed40003dd0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fed50009c90 tx=0x7fed40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d526f7a08e203c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:19.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.648+0000 7fed593da640 1 -- 172.21.15.161:0/4259018497 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed50039070 con 0x7fed400039b0 2024-03-28T23:37:19.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.648+0000 7fed593da640 1 -- 172.21.15.161:0/4259018497 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed5002c8f0 con 0x7fed400039b0 2024-03-28T23:37:19.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.648+0000 7fed593da640 1 -- 172.21.15.161:0/4259018497 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed50009860 con 0x7fed400039b0 2024-03-28T23:37:19.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.648+0000 7fed5c640640 1 -- 172.21.15.161:0/4259018497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed400039b0 msgr2=0x7fed40003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.648+0000 7fed5c640640 1 --2- 172.21.15.161:0/4259018497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed400039b0 0x7fed40003dd0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fed50009c90 tx=0x7fed40005830 comp rx=0 tx=0).stop 2024-03-28T23:37:19.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.649+0000 7fed5c640640 1 -- 172.21.15.161:0/4259018497 shutdown_connections 2024-03-28T23:37:19.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.649+0000 7fed5c640640 1 --2- 172.21.15.161:0/4259018497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed400039b0 0x7fed40003dd0 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.649+0000 7fed5c640640 1 -- 172.21.15.161:0/4259018497 >> 172.21.15.161:0/4259018497 conn(0x7fed5406b150 msgr2=0x7fed5406b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:19.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.649+0000 7fed5c640640 1 -- 172.21.15.161:0/4259018497 shutdown_connections 2024-03-28T23:37:19.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.649+0000 7fed5c640640 1 -- 172.21.15.161:0/4259018497 wait complete. 2024-03-28T23:37:19.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.650+0000 7fed5c640640 1 Processor -- start 2024-03-28T23:37:19.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.650+0000 7fed5c640640 1 -- start start 2024-03-28T23:37:19.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.650+0000 7fed5c640640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed400039b0 0x7fed54149250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.650+0000 7fed5c640640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed541ea810 0x7fed54148210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.650+0000 7fed5c640640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed541dc630 0x7fed541dcab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.650+0000 7fed5c640640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fed540c6440 con 0x7fed400039b0 2024-03-28T23:37:19.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.650+0000 7fed5c640640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fed540c1490 con 0x7fed541dc630 2024-03-28T23:37:19.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.650+0000 7fed5c640640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fed540ac810 con 0x7fed541ea810 2024-03-28T23:37:19.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.651+0000 7fed5a3dc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed400039b0 0x7fed54149250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.651+0000 7fed5a3dc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed400039b0 0x7fed54149250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:56010/0 (socket says 172.21.15.161:56010) 2024-03-28T23:37:19.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.651+0000 7fed5a3dc640 1 -- 172.21.15.161:0/796832190 learned_addr learned my addr 172.21.15.161:0/796832190 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:19.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.651+0000 7fed5abdd640 1 --2- 172.21.15.161:0/796832190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed541dc630 0x7fed541dcab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.651+0000 7fed59bdb640 1 --2- 172.21.15.161:0/796832190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed541ea810 0x7fed54148210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.651+0000 7fed5a3dc640 1 -- 172.21.15.161:0/796832190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed541ea810 msgr2=0x7fed54148210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.651+0000 7fed5a3dc640 1 --2- 172.21.15.161:0/796832190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed541ea810 0x7fed54148210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.651+0000 7fed5a3dc640 1 -- 172.21.15.161:0/796832190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed541dc630 msgr2=0x7fed541dcab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.651+0000 7fed5a3dc640 1 --2- 172.21.15.161:0/796832190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed541dc630 0x7fed541dcab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.651+0000 7fed5a3dc640 1 -- 172.21.15.161:0/796832190 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed54149830 con 0x7fed400039b0 2024-03-28T23:37:19.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.651+0000 7fed5a3dc640 1 --2- 172.21.15.161:0/796832190 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed400039b0 0x7fed54149250 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fed50037040 tx=0x7fed50004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:19.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.651+0000 7fed4affd640 1 -- 172.21.15.161:0/796832190 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed50039050 con 0x7fed400039b0 2024-03-28T23:37:19.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.651+0000 7fed4affd640 1 -- 172.21.15.161:0/796832190 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed5003a040 con 0x7fed400039b0 2024-03-28T23:37:19.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.651+0000 7fed5c640640 1 -- 172.21.15.161:0/796832190 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed541dbb50 con 0x7fed400039b0 2024-03-28T23:37:19.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.651+0000 7fed4affd640 1 -- 172.21.15.161:0/796832190 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed50009860 con 0x7fed400039b0 2024-03-28T23:37:19.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.651+0000 7fed5c640640 1 -- 172.21.15.161:0/796832190 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed541dc110 con 0x7fed400039b0 2024-03-28T23:37:19.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.652+0000 7fed5c640640 1 -- 172.21.15.161:0/796832190 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fed54068f90 con 0x7fed400039b0 2024-03-28T23:37:19.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.652+0000 7fed4affd640 1 -- 172.21.15.161:0/796832190 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fed50047030 con 0x7fed400039b0 2024-03-28T23:37:19.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.653+0000 7fed4affd640 1 --2- 172.21.15.161:0/796832190 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fed3806b4d0 0x7fed3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.653+0000 7fed59bdb640 1 --2- 172.21.15.161:0/796832190 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fed3806b4d0 0x7fed3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.653+0000 7fed4affd640 1 -- 172.21.15.161:0/796832190 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7fed500ab5e0 con 0x7fed400039b0 2024-03-28T23:37:19.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.653+0000 7fed4affd640 1 --2- 172.21.15.161:0/796832190 >> [v2:172.21.15.170:6824/211299856,v1:172.21.15.170:6825/211299856] conn(0x7fed38071c90 0x7fed380740d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.653+0000 7fed4affd640 1 -- 172.21.15.161:0/796832190 --> [v2:172.21.15.170:6824/211299856,v1:172.21.15.170:6825/211299856] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fed5002c8f0 con 0x7fed38071c90 2024-03-28T23:37:19.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.653+0000 7fed4affd640 1 -- 172.21.15.161:0/796832190 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=168) ==== 24+0+0 (secure 0 0 0) 0x7fed500b2050 con 0x7fed400039b0 2024-03-28T23:37:19.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.653+0000 7fed5abdd640 1 --2- 172.21.15.161:0/796832190 >> [v2:172.21.15.170:6824/211299856,v1:172.21.15.170:6825/211299856] conn(0x7fed38071c90 0x7fed380740d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.653+0000 7fed59bdb640 1 --2- 172.21.15.161:0/796832190 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fed3806b4d0 0x7fed3806d990 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fed44000c00 tx=0x7fed44009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:19.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.654+0000 7fed5abdd640 1 --2- 172.21.15.161:0/796832190 >> [v2:172.21.15.170:6824/211299856,v1:172.21.15.170:6825/211299856] conn(0x7fed38071c90 0x7fed380740d0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:19.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.654+0000 7fed4affd640 1 -- 172.21.15.161:0/796832190 <== osd.5 v2:172.21.15.170:6824/211299856 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed5002c8f0 con 0x7fed38071c90 2024-03-28T23:37:19.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.667+0000 7fed5c640640 1 -- 172.21.15.161:0/796832190 --> [v2:172.21.15.170:6824/211299856,v1:172.21.15.170:6825/211299856] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fed5406ba20 con 0x7fed38071c90 2024-03-28T23:37:19.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.668+0000 7fed4affd640 1 -- 172.21.15.161:0/796832190 <== osd.5 v2:172.21.15.170:6824/211299856 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fed5406ba20 con 0x7fed38071c90 2024-03-28T23:37:19.668 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:37:19.669 INFO:teuthology.orchestra.run.smithi161.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-28T23:37:19.669 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:37:19.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.668+0000 7fed5c640640 1 -- 172.21.15.161:0/796832190 >> [v2:172.21.15.170:6824/211299856,v1:172.21.15.170:6825/211299856] conn(0x7fed38071c90 msgr2=0x7fed380740d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.668+0000 7fed5c640640 1 --2- 172.21.15.161:0/796832190 >> [v2:172.21.15.170:6824/211299856,v1:172.21.15.170:6825/211299856] conn(0x7fed38071c90 0x7fed380740d0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.668+0000 7fed5c640640 1 -- 172.21.15.161:0/796832190 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fed3806b4d0 msgr2=0x7fed3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.668+0000 7fed5c640640 1 --2- 172.21.15.161:0/796832190 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fed3806b4d0 0x7fed3806d990 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fed44000c00 tx=0x7fed44009040 comp rx=0 tx=0).stop 2024-03-28T23:37:19.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.668+0000 7fed5c640640 1 -- 172.21.15.161:0/796832190 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed400039b0 msgr2=0x7fed54149250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.668+0000 7fed5c640640 1 --2- 172.21.15.161:0/796832190 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed400039b0 0x7fed54149250 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fed50037040 tx=0x7fed50004730 comp rx=0 tx=0).stop 2024-03-28T23:37:19.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.668+0000 7fed5abdd640 1 -- 172.21.15.161:0/796832190 reap_dead start 2024-03-28T23:37:19.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.668+0000 7fed5c640640 1 -- 172.21.15.161:0/796832190 shutdown_connections 2024-03-28T23:37:19.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.668+0000 7fed5c640640 1 -- 172.21.15.161:0/796832190 >> 172.21.15.161:0/796832190 conn(0x7fed5406b150 msgr2=0x7fed540c5460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:19.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.668+0000 7fed5c640640 1 -- 172.21.15.161:0/796832190 shutdown_connections 2024-03-28T23:37:19.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.668+0000 7fed5c640640 1 -- 172.21.15.161:0/796832190 wait complete. 2024-03-28T23:37:19.679 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-28T23:37:19.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.780+0000 7f00cd63b640 1 Processor -- start 2024-03-28T23:37:19.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.780+0000 7f00cd63b640 1 -- start start 2024-03-28T23:37:19.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.781+0000 7f00cd63b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00c8152890 0x7f00c80c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.781+0000 7f00cd63b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f00c80c8ed0 0x7f00c80c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.781+0000 7f00cd63b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00c8158a30 con 0x7f00c80c5950 2024-03-28T23:37:19.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.781+0000 7f00cd63b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f00c80c5db0 con 0x7f00c80c8ed0 2024-03-28T23:37:19.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.781+0000 7f00cd63b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f00c80c8480 con 0x7f00c8152890 2024-03-28T23:37:19.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.781+0000 7f00c6ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00c8152890 0x7f00c80c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.781+0000 7f00c6ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00c8152890 0x7f00c80c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:58112/0 (socket says 172.21.15.161:58112) 2024-03-28T23:37:19.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.781+0000 7f00c6ffd640 1 -- 172.21.15.161:0/4060104988 learned_addr learned my addr 172.21.15.161:0/4060104988 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:19.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.782+0000 7f00c67fc640 1 --2- 172.21.15.161:0/4060104988 >> v2:172.21.15.170:3300/0 conn(0x7f00c80c8ed0 0x7f00c80c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.782+0000 7f00c6ffd640 1 -- 172.21.15.161:0/4060104988 >> v2:172.21.15.170:3300/0 conn(0x7f00c80c8ed0 msgr2=0x7f00c80c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:19.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.782+0000 7f00c6ffd640 1 --2- 172.21.15.161:0/4060104988 >> v2:172.21.15.170:3300/0 conn(0x7f00c80c8ed0 0x7f00c80c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.782+0000 7f00c6ffd640 1 -- 172.21.15.161:0/4060104988 >> v1:172.21.15.170:6789/0 conn(0x7f00c80c5950 legacy=0x7f00c81553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.782+0000 7f00c6ffd640 1 -- 172.21.15.161:0/4060104988 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c8158d10 con 0x7f00c8152890 2024-03-28T23:37:19.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.783+0000 7f00c6ffd640 1 --2- 172.21.15.161:0/4060104988 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00c8152890 0x7f00c80c8990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f00b8009e50 tx=0x7f00b802be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcb97cffb89894e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:19.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.783+0000 7f00c5ffb640 1 -- 172.21.15.161:0/4060104988 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b802c9e0 con 0x7f00c8152890 2024-03-28T23:37:19.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.783+0000 7f00c5ffb640 1 -- 172.21.15.161:0/4060104988 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00b802cb80 con 0x7f00c8152890 2024-03-28T23:37:19.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.783+0000 7f00c5ffb640 1 -- 172.21.15.161:0/4060104988 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b80343f0 con 0x7f00c8152890 2024-03-28T23:37:19.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.783+0000 7f00cd63b640 1 -- 172.21.15.161:0/4060104988 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00c8152890 msgr2=0x7f00c80c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.783+0000 7f00cd63b640 1 --2- 172.21.15.161:0/4060104988 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00c8152890 0x7f00c80c8990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f00b8009e50 tx=0x7f00b802be40 comp rx=0 tx=0).stop 2024-03-28T23:37:19.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.783+0000 7f00cd63b640 1 -- 172.21.15.161:0/4060104988 shutdown_connections 2024-03-28T23:37:19.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.783+0000 7f00cd63b640 1 --2- 172.21.15.161:0/4060104988 >> v2:172.21.15.170:3300/0 conn(0x7f00c80c8ed0 0x7f00c80c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.783+0000 7f00cd63b640 1 --2- 172.21.15.161:0/4060104988 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00c8152890 0x7f00c80c8990 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.783+0000 7f00cd63b640 1 -- 172.21.15.161:0/4060104988 >> 172.21.15.161:0/4060104988 conn(0x7f00c80ac620 msgr2=0x7f00c80aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:19.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.783+0000 7f00cd63b640 1 -- 172.21.15.161:0/4060104988 shutdown_connections 2024-03-28T23:37:19.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.783+0000 7f00cd63b640 1 -- 172.21.15.161:0/4060104988 wait complete. 2024-03-28T23:37:19.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.785+0000 7f00cd63b640 1 Processor -- start 2024-03-28T23:37:19.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.785+0000 7f00cd63b640 1 -- start start 2024-03-28T23:37:19.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.785+0000 7f00cd63b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00c80c5950 0x7f00c81ec3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.785+0000 7f00cd63b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00c80c8ed0 0x7f00c80c2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.785+0000 7f00cd63b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c8152890 0x7f00c81ee4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.785+0000 7f00cd63b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f00c81592d0 con 0x7f00c8152890 2024-03-28T23:37:19.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.785+0000 7f00cd63b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f00c8159640 con 0x7f00c80c8ed0 2024-03-28T23:37:19.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.785+0000 7f00cd63b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f00c8152c70 con 0x7f00c80c5950 2024-03-28T23:37:19.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.785+0000 7f00c77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c8152890 0x7f00c81ee4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.785+0000 7f00c77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c8152890 0x7f00c81ee4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:56014/0 (socket says 172.21.15.161:56014) 2024-03-28T23:37:19.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.785+0000 7f00c77fe640 1 -- 172.21.15.161:0/2725847568 learned_addr learned my addr 172.21.15.161:0/2725847568 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:19.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.785+0000 7f00c67fc640 1 --2- 172.21.15.161:0/2725847568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00c80c8ed0 0x7f00c80c2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.785+0000 7f00c6ffd640 1 --2- 172.21.15.161:0/2725847568 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00c80c5950 0x7f00c81ec3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.786+0000 7f00c77fe640 1 -- 172.21.15.161:0/2725847568 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00c80c5950 msgr2=0x7f00c81ec3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.786+0000 7f00c77fe640 1 --2- 172.21.15.161:0/2725847568 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00c80c5950 0x7f00c81ec3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.786+0000 7f00c77fe640 1 -- 172.21.15.161:0/2725847568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00c80c8ed0 msgr2=0x7f00c80c2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.786+0000 7f00c77fe640 1 --2- 172.21.15.161:0/2725847568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00c80c8ed0 0x7f00c80c2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.786+0000 7f00c77fe640 1 -- 172.21.15.161:0/2725847568 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c80c3350 con 0x7f00c8152890 2024-03-28T23:37:19.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.786+0000 7f00c77fe640 1 --2- 172.21.15.161:0/2725847568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c8152890 0x7f00c81ee4c0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f00c000c390 tx=0x7f00c000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:19.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.786+0000 7f00af7fe640 1 -- 172.21.15.161:0/2725847568 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c0017070 con 0x7f00c8152890 2024-03-28T23:37:19.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.786+0000 7f00af7fe640 1 -- 172.21.15.161:0/2725847568 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00c00052d0 con 0x7f00c8152890 2024-03-28T23:37:19.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.786+0000 7f00cd63b640 1 -- 172.21.15.161:0/2725847568 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00c81ef000 con 0x7f00c8152890 2024-03-28T23:37:19.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.786+0000 7f00af7fe640 1 -- 172.21.15.161:0/2725847568 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c0003810 con 0x7f00c8152890 2024-03-28T23:37:19.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.786+0000 7f00cd63b640 1 -- 172.21.15.161:0/2725847568 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00c80c28f0 con 0x7f00c8152890 2024-03-28T23:37:19.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.787+0000 7f00cd63b640 1 -- 172.21.15.161:0/2725847568 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00c804f4b0 con 0x7f00c8152890 2024-03-28T23:37:19.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.787+0000 7f00af7fe640 1 -- 172.21.15.161:0/2725847568 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f00c001a070 con 0x7f00c8152890 2024-03-28T23:37:19.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.788+0000 7f00af7fe640 1 --2- 172.21.15.161:0/2725847568 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f00b406b500 0x7f00b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:19.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.788+0000 7f00af7fe640 1 -- 172.21.15.161:0/2725847568 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7f00c0089480 con 0x7f00c8152890 2024-03-28T23:37:19.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.788+0000 7f00c6ffd640 1 --2- 172.21.15.161:0/2725847568 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f00b406b500 0x7f00b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:19.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.788+0000 7f00c6ffd640 1 --2- 172.21.15.161:0/2725847568 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f00b406b500 0x7f00b406d9c0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f00b8035000 tx=0x7f00b8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:19.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.792+0000 7f00af7fe640 1 -- 172.21.15.161:0/2725847568 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f00c0057c80 con 0x7f00c8152890 2024-03-28T23:37:19.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.963+0000 7f00cd63b640 1 -- 172.21.15.161:0/2725847568 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f00c80c4a10 con 0x7f00b406b500 2024-03-28T23:37:19.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.964+0000 7f00af7fe640 1 -- 172.21.15.161:0/2725847568 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f00c80c4a10 con 0x7f00b406b500 2024-03-28T23:37:19.964 INFO:teuthology.orchestra.run.smithi161.stderr:instructed osd(s) 5 to deep-scrub 2024-03-28T23:37:19.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.966+0000 7f00cd63b640 1 -- 172.21.15.161:0/2725847568 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f00b406b500 msgr2=0x7f00b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.966+0000 7f00cd63b640 1 --2- 172.21.15.161:0/2725847568 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f00b406b500 0x7f00b406d9c0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f00b8035000 tx=0x7f00b8030000 comp rx=0 tx=0).stop 2024-03-28T23:37:19.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.966+0000 7f00cd63b640 1 -- 172.21.15.161:0/2725847568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c8152890 msgr2=0x7f00c81ee4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:19.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.966+0000 7f00cd63b640 1 --2- 172.21.15.161:0/2725847568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c8152890 0x7f00c81ee4c0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f00c000c390 tx=0x7f00c000c850 comp rx=0 tx=0).stop 2024-03-28T23:37:19.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.966+0000 7f00cd63b640 1 -- 172.21.15.161:0/2725847568 shutdown_connections 2024-03-28T23:37:19.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.966+0000 7f00cd63b640 1 --2- 172.21.15.161:0/2725847568 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f00b406b500 0x7f00b406d9c0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.966+0000 7f00cd63b640 1 --2- 172.21.15.161:0/2725847568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c8152890 0x7f00c81ee4c0 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.966+0000 7f00cd63b640 1 --2- 172.21.15.161:0/2725847568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00c80c8ed0 0x7f00c80c2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.966+0000 7f00cd63b640 1 --2- 172.21.15.161:0/2725847568 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f00c80c5950 0x7f00c81ec3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:19.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.966+0000 7f00cd63b640 1 -- 172.21.15.161:0/2725847568 >> 172.21.15.161:0/2725847568 conn(0x7f00c80ac620 msgr2=0x7f00c80c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:19.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.966+0000 7f00cd63b640 1 -- 172.21.15.161:0/2725847568 shutdown_connections 2024-03-28T23:37:19.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:19.966+0000 7f00cd63b640 1 -- 172.21.15.161:0/2725847568 wait complete. 2024-03-28T23:37:19.977 INFO:tasks.ceph:Scrubbing osd.6 2024-03-28T23:37:19.977 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-28T23:37:20.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.081+0000 7f7fbe472640 1 Processor -- start 2024-03-28T23:37:20.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.081+0000 7f7fbe472640 1 -- start start 2024-03-28T23:37:20.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.082+0000 7f7fbe472640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7fb806b440 0x7f7fb8155d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.082+0000 7f7fbe472640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7fb80c53e0 0x7f7fb80c7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.082+0000 7f7fbe472640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fb806bb70 con 0x7f7fb8152740 2024-03-28T23:37:20.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.082+0000 7f7fbe472640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f7fb806be50 con 0x7f7fb806b440 2024-03-28T23:37:20.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.082+0000 7f7fbe472640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7fb80c8960 con 0x7f7fb80c53e0 2024-03-28T23:37:20.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.083+0000 7f7fbca0f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7fb80c53e0 0x7f7fb80c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.083+0000 7f7fb77fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7fb806b440 0x7f7fb8155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.083+0000 7f7fbca0f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7fb80c53e0 0x7f7fb80c7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:58132/0 (socket says 172.21.15.161:58132) 2024-03-28T23:37:20.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.083+0000 7f7fbca0f640 1 -- 172.21.15.161:0/2750416908 learned_addr learned my addr 172.21.15.161:0/2750416908 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:20.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.083+0000 7f7fbca0f640 1 -- 172.21.15.161:0/2750416908 >> v2:172.21.15.170:3300/0 conn(0x7f7fb806b440 msgr2=0x7f7fb8155d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:20.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.083+0000 7f7fbca0f640 1 --2- 172.21.15.161:0/2750416908 >> v2:172.21.15.170:3300/0 conn(0x7f7fb806b440 0x7f7fb8155d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.083+0000 7f7fbca0f640 1 -- 172.21.15.161:0/2750416908 >> v1:172.21.15.170:6789/0 conn(0x7f7fb8152740 legacy=0x7f7fb80c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.083+0000 7f7fbca0f640 1 -- 172.21.15.161:0/2750416908 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb80c8c40 con 0x7f7fb80c53e0 2024-03-28T23:37:20.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.083+0000 7f7fb77fe640 1 --2- 172.21.15.161:0/2750416908 >> v2:172.21.15.170:3300/0 conn(0x7f7fb806b440 0x7f7fb8155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:37:20.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.084+0000 7f7fbca0f640 1 --2- 172.21.15.161:0/2750416908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7fb80c53e0 0x7f7fb80c7470 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0009e50 tx=0x7f7fb002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc73dab4826b084b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:20.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.084+0000 7f7fb6ffd640 1 -- 172.21.15.161:0/2750416908 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb002ca70 con 0x7f7fb80c53e0 2024-03-28T23:37:20.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.084+0000 7f7fb6ffd640 1 -- 172.21.15.161:0/2750416908 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fb002cc10 con 0x7f7fb80c53e0 2024-03-28T23:37:20.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.084+0000 7f7fb6ffd640 1 -- 172.21.15.161:0/2750416908 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb00343f0 con 0x7f7fb80c53e0 2024-03-28T23:37:20.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.084+0000 7f7fbe472640 1 -- 172.21.15.161:0/2750416908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7fb80c53e0 msgr2=0x7f7fb80c7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.084+0000 7f7fbe472640 1 --2- 172.21.15.161:0/2750416908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7fb80c53e0 0x7f7fb80c7470 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0009e50 tx=0x7f7fb002bed0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.084+0000 7f7fbe472640 1 -- 172.21.15.161:0/2750416908 shutdown_connections 2024-03-28T23:37:20.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.084+0000 7f7fbe472640 1 --2- 172.21.15.161:0/2750416908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7fb80c53e0 0x7f7fb80c7470 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.084+0000 7f7fbe472640 1 --2- 172.21.15.161:0/2750416908 >> v2:172.21.15.170:3300/0 conn(0x7f7fb806b440 0x7f7fb8155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.084+0000 7f7fbe472640 1 -- 172.21.15.161:0/2750416908 >> 172.21.15.161:0/2750416908 conn(0x7f7fb80cac40 msgr2=0x7f7fb80cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:20.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.084+0000 7f7fbe472640 1 -- 172.21.15.161:0/2750416908 shutdown_connections 2024-03-28T23:37:20.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.085+0000 7f7fbe472640 1 -- 172.21.15.161:0/2750416908 wait complete. 2024-03-28T23:37:20.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.086+0000 7f7fbe472640 1 Processor -- start 2024-03-28T23:37:20.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.086+0000 7f7fbe472640 1 -- start start 2024-03-28T23:37:20.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.086+0000 7f7fbe472640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7fb806b440 0x7f7fb814bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.086+0000 7f7fbe472640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7fb80c53e0 0x7f7fb8158e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.086+0000 7f7fbe472640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7fb8152740 0x7f7fb80afe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.086+0000 7f7fbe472640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb80c5870 con 0x7f7fb806b440 2024-03-28T23:37:20.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.086+0000 7f7fbe472640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb806b880 con 0x7f7fb8152740 2024-03-28T23:37:20.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.086+0000 7f7fbe472640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f7fb8152b20 con 0x7f7fb80c53e0 2024-03-28T23:37:20.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.087+0000 7f7fb7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7fb806b440 0x7f7fb814bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.087+0000 7f7fb7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7fb806b440 0x7f7fb814bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:56020/0 (socket says 172.21.15.161:56020) 2024-03-28T23:37:20.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.087+0000 7f7fb7fff640 1 -- 172.21.15.161:0/1970500457 learned_addr learned my addr 172.21.15.161:0/1970500457 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:20.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.087+0000 7f7fbca0f640 1 --2- 172.21.15.161:0/1970500457 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7fb8152740 0x7f7fb80afe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.087+0000 7f7fb77fe640 1 --2- 172.21.15.161:0/1970500457 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7fb80c53e0 0x7f7fb8158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.087+0000 7f7fb7fff640 1 -- 172.21.15.161:0/1970500457 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7fb80c53e0 msgr2=0x7f7fb8158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.087+0000 7f7fb7fff640 1 --2- 172.21.15.161:0/1970500457 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7fb80c53e0 0x7f7fb8158e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.087+0000 7f7fb7fff640 1 -- 172.21.15.161:0/1970500457 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7fb8152740 msgr2=0x7f7fb80afe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.087+0000 7f7fb7fff640 1 --2- 172.21.15.161:0/1970500457 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7fb8152740 0x7f7fb80afe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.087+0000 7f7fb7fff640 1 -- 172.21.15.161:0/1970500457 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb80ae090 con 0x7f7fb806b440 2024-03-28T23:37:20.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.087+0000 7f7fb77fe640 1 --2- 172.21.15.161:0/1970500457 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7fb80c53e0 0x7f7fb8158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T23:37:20.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.087+0000 7f7fb7fff640 1 --2- 172.21.15.161:0/1970500457 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7fb806b440 0x7f7fb814bbf0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f7fa800c3f0 tx=0x7f7fa800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:20.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.087+0000 7f7fb4ff9640 1 -- 172.21.15.161:0/1970500457 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa80052d0 con 0x7f7fb806b440 2024-03-28T23:37:20.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.087+0000 7f7fb4ff9640 1 -- 172.21.15.161:0/1970500457 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fa8005490 con 0x7f7fb806b440 2024-03-28T23:37:20.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.087+0000 7f7fbe472640 1 -- 172.21.15.161:0/1970500457 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fb80ae370 con 0x7f7fb806b440 2024-03-28T23:37:20.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.087+0000 7f7fb4ff9640 1 -- 172.21.15.161:0/1970500457 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8007260 con 0x7f7fb806b440 2024-03-28T23:37:20.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.087+0000 7f7fbe472640 1 -- 172.21.15.161:0/1970500457 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fb80ae930 con 0x7f7fb806b440 2024-03-28T23:37:20.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.088+0000 7f7fbe472640 1 -- 172.21.15.161:0/1970500457 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7fb80aeff0 con 0x7f7fb806b440 2024-03-28T23:37:20.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.089+0000 7f7fb4ff9640 1 -- 172.21.15.161:0/1970500457 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7fa8007400 con 0x7f7fb806b440 2024-03-28T23:37:20.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.089+0000 7f7fb4ff9640 1 --2- 172.21.15.161:0/1970500457 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7fa406b500 0x7f7fa406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.089+0000 7f7fb4ff9640 1 -- 172.21.15.161:0/1970500457 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7f7fa8089710 con 0x7f7fb806b440 2024-03-28T23:37:20.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.089+0000 7f7fb77fe640 1 --2- 172.21.15.161:0/1970500457 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7fa406b500 0x7f7fa406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.089+0000 7f7fb4ff9640 1 --2- 172.21.15.161:0/1970500457 >> [v2:172.21.15.170:6808/844763145,v1:172.21.15.170:6809/844763145] conn(0x7f7fa4071cc0 0x7f7fa4074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.089+0000 7f7fb4ff9640 1 -- 172.21.15.161:0/1970500457 --> [v2:172.21.15.170:6808/844763145,v1:172.21.15.170:6809/844763145] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7fa8009e50 con 0x7f7fa4071cc0 2024-03-28T23:37:20.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.089+0000 7f7fb4ff9640 1 -- 172.21.15.161:0/1970500457 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=168) ==== 24+0+0 (secure 0 0 0) 0x7f7fa8090050 con 0x7f7fb806b440 2024-03-28T23:37:20.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.089+0000 7f7fbca0f640 1 --2- 172.21.15.161:0/1970500457 >> [v2:172.21.15.170:6808/844763145,v1:172.21.15.170:6809/844763145] conn(0x7f7fa4071cc0 0x7f7fa4074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.089+0000 7f7fb77fe640 1 --2- 172.21.15.161:0/1970500457 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7fa406b500 0x7f7fa406d9c0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8155320 tx=0x7f7fac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:20.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.090+0000 7f7fbca0f640 1 --2- 172.21.15.161:0/1970500457 >> [v2:172.21.15.170:6808/844763145,v1:172.21.15.170:6809/844763145] conn(0x7f7fa4071cc0 0x7f7fa4074100 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:20.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.091+0000 7f7fb4ff9640 1 -- 172.21.15.161:0/1970500457 <== osd.6 v2:172.21.15.170:6808/844763145 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7fa8009e50 con 0x7f7fa4071cc0 2024-03-28T23:37:20.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.104+0000 7f7fbe472640 1 -- 172.21.15.161:0/1970500457 --> [v2:172.21.15.170:6808/844763145,v1:172.21.15.170:6809/844763145] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f7fb8068f90 con 0x7f7fa4071cc0 2024-03-28T23:37:20.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.104+0000 7f7fb4ff9640 1 -- 172.21.15.161:0/1970500457 <== osd.6 v2:172.21.15.170:6808/844763145 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f7fb8068f90 con 0x7f7fa4071cc0 2024-03-28T23:37:20.105 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:37:20.105 INFO:teuthology.orchestra.run.smithi161.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-28T23:37:20.105 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:37:20.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.104+0000 7f7fbe472640 1 -- 172.21.15.161:0/1970500457 >> [v2:172.21.15.170:6808/844763145,v1:172.21.15.170:6809/844763145] conn(0x7f7fa4071cc0 msgr2=0x7f7fa4074100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.104+0000 7f7fbe472640 1 --2- 172.21.15.161:0/1970500457 >> [v2:172.21.15.170:6808/844763145,v1:172.21.15.170:6809/844763145] conn(0x7f7fa4071cc0 0x7f7fa4074100 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.104+0000 7f7fbe472640 1 -- 172.21.15.161:0/1970500457 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7fa406b500 msgr2=0x7f7fa406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.104+0000 7f7fbe472640 1 --2- 172.21.15.161:0/1970500457 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f7fa406b500 0x7f7fa406d9c0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8155320 tx=0x7f7fac008040 comp rx=0 tx=0).stop 2024-03-28T23:37:20.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.104+0000 7f7fbe472640 1 -- 172.21.15.161:0/1970500457 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7fb806b440 msgr2=0x7f7fb814bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.104+0000 7f7fbe472640 1 --2- 172.21.15.161:0/1970500457 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7fb806b440 0x7f7fb814bbf0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f7fa800c3f0 tx=0x7f7fa800c8b0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.104+0000 7f7fbca0f640 1 -- 172.21.15.161:0/1970500457 reap_dead start 2024-03-28T23:37:20.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.105+0000 7f7fbe472640 1 -- 172.21.15.161:0/1970500457 shutdown_connections 2024-03-28T23:37:20.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.105+0000 7f7fbe472640 1 -- 172.21.15.161:0/1970500457 >> 172.21.15.161:0/1970500457 conn(0x7f7fb80cac40 msgr2=0x7f7fb814ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:20.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.105+0000 7f7fbe472640 1 -- 172.21.15.161:0/1970500457 shutdown_connections 2024-03-28T23:37:20.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.105+0000 7f7fbe472640 1 -- 172.21.15.161:0/1970500457 wait complete. 2024-03-28T23:37:20.115 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-28T23:37:20.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.218+0000 7fbc280cd640 1 Processor -- start 2024-03-28T23:37:20.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.218+0000 7fbc280cd640 1 -- start start 2024-03-28T23:37:20.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.218+0000 7fbc280cd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fbc2014be70 0x7fbc200ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.218+0000 7fbc280cd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbc200aacd0 0x7fbc2015c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.219+0000 7fbc280cd640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc2014c250 con 0x7fbc200ab6e0 2024-03-28T23:37:20.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.219+0000 7fbc280cd640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fbc201529c0 con 0x7fbc200aacd0 2024-03-28T23:37:20.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.219+0000 7fbc280cd640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fbc200a9dc0 con 0x7fbc2014be70 2024-03-28T23:37:20.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.219+0000 7fbc2666a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbc200aacd0 0x7fbc2015c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.219+0000 7fbc25668640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fbc200ab6e0 0x7fbc200aa5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41756/0 (socket says 172.21.15.161:41756) 2024-03-28T23:37:20.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.219+0000 7fbc25668640 1 -- 172.21.15.161:0/2779100040 learned_addr learned my addr 172.21.15.161:0/2779100040 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:20.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.219+0000 7fbc25e69640 1 --2- 172.21.15.161:0/2779100040 >> v2:172.21.15.170:3300/0 conn(0x7fbc2014be70 0x7fbc200ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.219+0000 7fbc2666a640 1 -- 172.21.15.161:0/2779100040 >> v2:172.21.15.170:3300/0 conn(0x7fbc2014be70 msgr2=0x7fbc200ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:20.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.219+0000 7fbc2666a640 1 --2- 172.21.15.161:0/2779100040 >> v2:172.21.15.170:3300/0 conn(0x7fbc2014be70 0x7fbc200ab1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.219+0000 7fbc2666a640 1 -- 172.21.15.161:0/2779100040 >> v1:172.21.15.161:6789/0 conn(0x7fbc200ab6e0 legacy=0x7fbc200aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.219+0000 7fbc2666a640 1 -- 172.21.15.161:0/2779100040 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc200aa0a0 con 0x7fbc200aacd0 2024-03-28T23:37:20.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.219+0000 7fbc24e67640 1 -- 172.21.15.161:0/2779100040 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2109736503 0 0) 0x7fbc2014c250 con 0x7fbc200ab6e0 2024-03-28T23:37:20.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.219+0000 7fbc25e69640 1 --2- 172.21.15.161:0/2779100040 >> v2:172.21.15.170:3300/0 conn(0x7fbc2014be70 0x7fbc200ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:37:20.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.220+0000 7fbc2666a640 1 --2- 172.21.15.161:0/2779100040 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc200aacd0 0x7fbc2015c740 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7fbc200c7520 tx=0x7fbc1802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93302d86333b1d41 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:20.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.220+0000 7fbc24e67640 1 -- 172.21.15.161:0/2779100040 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc18039070 con 0x7fbc200aacd0 2024-03-28T23:37:20.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.220+0000 7fbc24e67640 1 -- 172.21.15.161:0/2779100040 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc1802cbf0 con 0x7fbc200aacd0 2024-03-28T23:37:20.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.220+0000 7fbc24e67640 1 -- 172.21.15.161:0/2779100040 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc18034440 con 0x7fbc200aacd0 2024-03-28T23:37:20.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.220+0000 7fbc280cd640 1 -- 172.21.15.161:0/2779100040 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc200aacd0 msgr2=0x7fbc2015c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.220+0000 7fbc280cd640 1 --2- 172.21.15.161:0/2779100040 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc200aacd0 0x7fbc2015c740 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7fbc200c7520 tx=0x7fbc1802be40 comp rx=0 tx=0).stop 2024-03-28T23:37:20.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.220+0000 7fbc280cd640 1 -- 172.21.15.161:0/2779100040 shutdown_connections 2024-03-28T23:37:20.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.220+0000 7fbc280cd640 1 --2- 172.21.15.161:0/2779100040 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc200aacd0 0x7fbc2015c740 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.220+0000 7fbc280cd640 1 --2- 172.21.15.161:0/2779100040 >> v2:172.21.15.170:3300/0 conn(0x7fbc2014be70 0x7fbc200ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.220+0000 7fbc280cd640 1 -- 172.21.15.161:0/2779100040 >> 172.21.15.161:0/2779100040 conn(0x7fbc200c6670 msgr2=0x7fbc200c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:20.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.220+0000 7fbc280cd640 1 -- 172.21.15.161:0/2779100040 shutdown_connections 2024-03-28T23:37:20.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.220+0000 7fbc280cd640 1 -- 172.21.15.161:0/2779100040 wait complete. 2024-03-28T23:37:20.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.221+0000 7fbc280cd640 1 Processor -- start 2024-03-28T23:37:20.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.222+0000 7fbc280cd640 1 -- start start 2024-03-28T23:37:20.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.222+0000 7fbc280cd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc200aacd0 0x7fbc201e9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.222+0000 7fbc280cd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc200ab6e0 0x7fbc200c1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.222+0000 7fbc280cd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc2014be70 0x7fbc201489a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.222+0000 7fbc280cd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2006b2d0 con 0x7fbc200aacd0 2024-03-28T23:37:20.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.222+0000 7fbc280cd640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2006b150 con 0x7fbc2014be70 2024-03-28T23:37:20.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.222+0000 7fbc280cd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fbc200c1520 con 0x7fbc200ab6e0 2024-03-28T23:37:20.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.222+0000 7fbc25e69640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc200aacd0 0x7fbc201e9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.222+0000 7fbc25e69640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc200aacd0 0x7fbc201e9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:56040/0 (socket says 172.21.15.161:56040) 2024-03-28T23:37:20.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.222+0000 7fbc25e69640 1 -- 172.21.15.161:0/4014964989 learned_addr learned my addr 172.21.15.161:0/4014964989 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:20.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.222+0000 7fbc25e69640 1 -- 172.21.15.161:0/4014964989 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc200ab6e0 msgr2=0x7fbc200c1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.222+0000 7fbc25e69640 1 --2- 172.21.15.161:0/4014964989 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc200ab6e0 0x7fbc200c1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.223+0000 7fbc25e69640 1 -- 172.21.15.161:0/4014964989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc2014be70 msgr2=0x7fbc201489a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.223+0000 7fbc2666a640 1 --2- 172.21.15.161:0/4014964989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc2014be70 0x7fbc201489a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.223+0000 7fbc25e69640 1 --2- 172.21.15.161:0/4014964989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc2014be70 0x7fbc201489a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.223+0000 7fbc25e69640 1 -- 172.21.15.161:0/4014964989 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc200bb910 con 0x7fbc200aacd0 2024-03-28T23:37:20.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.223+0000 7fbc25e69640 1 --2- 172.21.15.161:0/4014964989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc200aacd0 0x7fbc201e9030 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c00e7e0 tx=0x7fbc1c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:20.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.223+0000 7fbc167fc640 1 -- 172.21.15.161:0/4014964989 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c00f040 con 0x7fbc200aacd0 2024-03-28T23:37:20.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.223+0000 7fbc167fc640 1 -- 172.21.15.161:0/4014964989 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc1c004030 con 0x7fbc200aacd0 2024-03-28T23:37:20.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.223+0000 7fbc167fc640 1 -- 172.21.15.161:0/4014964989 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c004400 con 0x7fbc200aacd0 2024-03-28T23:37:20.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.223+0000 7fbc280cd640 1 -- 172.21.15.161:0/4014964989 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc200bbbf0 con 0x7fbc200aacd0 2024-03-28T23:37:20.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.223+0000 7fbc280cd640 1 -- 172.21.15.161:0/4014964989 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc200bc1b0 con 0x7fbc200aacd0 2024-03-28T23:37:20.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.224+0000 7fbc280cd640 1 -- 172.21.15.161:0/4014964989 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc200bc3f0 con 0x7fbc200aacd0 2024-03-28T23:37:20.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.224+0000 7fbc167fc640 1 -- 172.21.15.161:0/4014964989 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbc1c00c6d0 con 0x7fbc200aacd0 2024-03-28T23:37:20.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.225+0000 7fbc167fc640 1 --2- 172.21.15.161:0/4014964989 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbc0406b500 0x7fbc0406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.225+0000 7fbc167fc640 1 -- 172.21.15.161:0/4014964989 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7fbc1c089f60 con 0x7fbc200aacd0 2024-03-28T23:37:20.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.225+0000 7fbc25668640 1 --2- 172.21.15.161:0/4014964989 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbc0406b500 0x7fbc0406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.225+0000 7fbc25668640 1 --2- 172.21.15.161:0/4014964989 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbc0406b500 0x7fbc0406d9c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fbc10007fd0 tx=0x7fbc10007980 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:20.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.229+0000 7fbc167fc640 1 -- 172.21.15.161:0/4014964989 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbc200bc3f0 con 0x7fbc200aacd0 2024-03-28T23:37:20.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.400+0000 7fbc280cd640 1 -- 172.21.15.161:0/4014964989 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7fbc2004f4b0 con 0x7fbc0406b500 2024-03-28T23:37:20.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.401+0000 7fbc167fc640 1 -- 172.21.15.161:0/4014964989 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fbc2004f4b0 con 0x7fbc0406b500 2024-03-28T23:37:20.401 INFO:teuthology.orchestra.run.smithi161.stderr:instructed osd(s) 6 to deep-scrub 2024-03-28T23:37:20.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.403+0000 7fbc280cd640 1 -- 172.21.15.161:0/4014964989 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbc0406b500 msgr2=0x7fbc0406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.403+0000 7fbc280cd640 1 --2- 172.21.15.161:0/4014964989 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbc0406b500 0x7fbc0406d9c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fbc10007fd0 tx=0x7fbc10007980 comp rx=0 tx=0).stop 2024-03-28T23:37:20.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.403+0000 7fbc280cd640 1 -- 172.21.15.161:0/4014964989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc200aacd0 msgr2=0x7fbc201e9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.403+0000 7fbc280cd640 1 --2- 172.21.15.161:0/4014964989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc200aacd0 0x7fbc201e9030 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c00e7e0 tx=0x7fbc1c00eca0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.403+0000 7fbc280cd640 1 -- 172.21.15.161:0/4014964989 shutdown_connections 2024-03-28T23:37:20.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.403+0000 7fbc280cd640 1 --2- 172.21.15.161:0/4014964989 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fbc0406b500 0x7fbc0406d9c0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.403+0000 7fbc280cd640 1 --2- 172.21.15.161:0/4014964989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc2014be70 0x7fbc201489a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.403+0000 7fbc280cd640 1 --2- 172.21.15.161:0/4014964989 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc200ab6e0 0x7fbc200c1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.403+0000 7fbc280cd640 1 --2- 172.21.15.161:0/4014964989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc200aacd0 0x7fbc201e9030 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.403+0000 7fbc280cd640 1 -- 172.21.15.161:0/4014964989 >> 172.21.15.161:0/4014964989 conn(0x7fbc200c6670 msgr2=0x7fbc2014b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:20.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.403+0000 7fbc280cd640 1 -- 172.21.15.161:0/4014964989 shutdown_connections 2024-03-28T23:37:20.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.403+0000 7fbc280cd640 1 -- 172.21.15.161:0/4014964989 wait complete. 2024-03-28T23:37:20.414 INFO:tasks.ceph:Scrubbing osd.7 2024-03-28T23:37:20.414 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-28T23:37:20.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.519+0000 7f1571180640 1 Processor -- start 2024-03-28T23:37:20.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.519+0000 7f1571180640 1 -- start start 2024-03-28T23:37:20.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.520+0000 7f1571180640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f156c0ac430 0x7f156c152880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.520+0000 7f1571180640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f156c0c3540 con 0x7f156c0c30c0 2024-03-28T23:37:20.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.520+0000 7f1571180640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f156c0c3820 con 0x7f156c0c19b0 2024-03-28T23:37:20.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.520+0000 7f1571180640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f156c152dc0 con 0x7f156c0ac430 2024-03-28T23:37:20.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.520+0000 7f156ad9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f156c0ac430 0x7f156c152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.520+0000 7f156a59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f156c0c30c0 0x7f156c0c24d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:55854/0 (socket says 172.21.15.161:55854) 2024-03-28T23:37:20.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.520+0000 7f156a59c640 1 -- 172.21.15.161:0/1971611201 learned_addr learned my addr 172.21.15.161:0/1971611201 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:20.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.520+0000 7f156ad9d640 1 -- 172.21.15.161:0/1971611201 >> v1:172.21.15.170:6789/0 conn(0x7f156c0c19b0 legacy=0x7f156c15c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.520+0000 7f156ad9d640 1 -- 172.21.15.161:0/1971611201 >> v1:172.21.15.161:6789/0 conn(0x7f156c0c30c0 legacy=0x7f156c0c24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.520+0000 7f156ad9d640 1 -- 172.21.15.161:0/1971611201 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f156c0c3b00 con 0x7f156c0ac430 2024-03-28T23:37:20.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.520+0000 7f1569d9b640 1 -- 172.21.15.161:0/1971611201 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2354698003 0 0) 0x7f156c0c3540 con 0x7f156c0c30c0 2024-03-28T23:37:20.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.521+0000 7f156ad9d640 1 --2- 172.21.15.161:0/1971611201 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f156c0ac430 0x7f156c152880 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f155c0097a0 tx=0x7f155c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2008f9c4da686475 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:20.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.521+0000 7f1569d9b640 1 -- 172.21.15.161:0/1971611201 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c035020 con 0x7f156c0ac430 2024-03-28T23:37:20.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.521+0000 7f1569d9b640 1 -- 172.21.15.161:0/1971611201 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f155c031830 con 0x7f156c0ac430 2024-03-28T23:37:20.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.521+0000 7f1569d9b640 1 -- 172.21.15.161:0/1971611201 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c031c00 con 0x7f156c0ac430 2024-03-28T23:37:20.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.522+0000 7f1571180640 1 -- 172.21.15.161:0/1971611201 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f156c0ac430 msgr2=0x7f156c152880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.522+0000 7f1571180640 1 --2- 172.21.15.161:0/1971611201 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f156c0ac430 0x7f156c152880 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f155c0097a0 tx=0x7f155c02b8d0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.522+0000 7f1571180640 1 -- 172.21.15.161:0/1971611201 shutdown_connections 2024-03-28T23:37:20.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.522+0000 7f1571180640 1 --2- 172.21.15.161:0/1971611201 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f156c0ac430 0x7f156c152880 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.522+0000 7f1571180640 1 -- 172.21.15.161:0/1971611201 >> 172.21.15.161:0/1971611201 conn(0x7f156c06b150 msgr2=0x7f156c06b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:20.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.522+0000 7f1571180640 1 -- 172.21.15.161:0/1971611201 shutdown_connections 2024-03-28T23:37:20.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.522+0000 7f1571180640 1 -- 172.21.15.161:0/1971611201 wait complete. 2024-03-28T23:37:20.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.523+0000 7f1571180640 1 Processor -- start 2024-03-28T23:37:20.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.523+0000 7f1571180640 1 -- start start 2024-03-28T23:37:20.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.524+0000 7f1571180640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f156c0ac430 0x7f156c0a8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.524+0000 7f1571180640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f156c0c19b0 0x7f156c072a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.524+0000 7f1571180640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f156c0c30c0 0x7f156c149cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.524+0000 7f1571180640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f156c0c6440 con 0x7f156c0ac430 2024-03-28T23:37:20.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.524+0000 7f1571180640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f156c0c1490 con 0x7f156c0c19b0 2024-03-28T23:37:20.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.524+0000 7f1571180640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f156c0c2be0 con 0x7f156c0c30c0 2024-03-28T23:37:20.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.524+0000 7f156ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f156c0ac430 0x7f156c0a8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.524+0000 7f156ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f156c0ac430 0x7f156c0a8dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45114/0 (socket says 172.21.15.161:45114) 2024-03-28T23:37:20.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.524+0000 7f156ad9d640 1 -- 172.21.15.161:0/2665691829 learned_addr learned my addr 172.21.15.161:0/2665691829 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:20.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.524+0000 7f156a59c640 1 --2- 172.21.15.161:0/2665691829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f156c0c19b0 0x7f156c072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.524+0000 7f156b59e640 1 --2- 172.21.15.161:0/2665691829 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f156c0c30c0 0x7f156c149cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.524+0000 7f156ad9d640 1 -- 172.21.15.161:0/2665691829 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f156c0c30c0 msgr2=0x7f156c149cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.524+0000 7f156ad9d640 1 --2- 172.21.15.161:0/2665691829 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f156c0c30c0 0x7f156c149cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.524+0000 7f156ad9d640 1 -- 172.21.15.161:0/2665691829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f156c0c19b0 msgr2=0x7f156c072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.524+0000 7f156ad9d640 1 --2- 172.21.15.161:0/2665691829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f156c0c19b0 0x7f156c072a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.524+0000 7f156ad9d640 1 -- 172.21.15.161:0/2665691829 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f156c1ebff0 con 0x7f156c0ac430 2024-03-28T23:37:20.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.524+0000 7f156b59e640 1 --2- 172.21.15.161:0/2665691829 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f156c0c30c0 0x7f156c149cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:37:20.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.524+0000 7f156ad9d640 1 --2- 172.21.15.161:0/2665691829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f156c0ac430 0x7f156c0a8dd0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f155c0098c0 tx=0x7f155c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:20.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.524+0000 7f15537fe640 1 -- 172.21.15.161:0/2665691829 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c041070 con 0x7f156c0ac430 2024-03-28T23:37:20.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.525+0000 7f15537fe640 1 -- 172.21.15.161:0/2665691829 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f155c03aa30 con 0x7f156c0ac430 2024-03-28T23:37:20.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.525+0000 7f1571180640 1 -- 172.21.15.161:0/2665691829 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f156c1ec2d0 con 0x7f156c0ac430 2024-03-28T23:37:20.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.525+0000 7f15537fe640 1 -- 172.21.15.161:0/2665691829 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c035040 con 0x7f156c0ac430 2024-03-28T23:37:20.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.525+0000 7f1571180640 1 -- 172.21.15.161:0/2665691829 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f156c1ec890 con 0x7f156c0ac430 2024-03-28T23:37:20.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.525+0000 7f1571180640 1 -- 172.21.15.161:0/2665691829 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f156c1ebda0 con 0x7f156c0ac430 2024-03-28T23:37:20.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.526+0000 7f15537fe640 1 -- 172.21.15.161:0/2665691829 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f155c04c060 con 0x7f156c0ac430 2024-03-28T23:37:20.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.526+0000 7f15537fe640 1 --2- 172.21.15.161:0/2665691829 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f155806b500 0x7f155806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.526+0000 7f15537fe640 1 -- 172.21.15.161:0/2665691829 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7f155c0abc40 con 0x7f156c0ac430 2024-03-28T23:37:20.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.526+0000 7f156a59c640 1 --2- 172.21.15.161:0/2665691829 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f155806b500 0x7f155806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.526+0000 7f15537fe640 1 --2- 172.21.15.161:0/2665691829 >> [v2:172.21.15.170:6816/3081832568,v1:172.21.15.170:6817/3081832568] conn(0x7f1558071cc0 0x7f1558074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.526+0000 7f15537fe640 1 -- 172.21.15.161:0/2665691829 --> [v2:172.21.15.170:6816/3081832568,v1:172.21.15.170:6817/3081832568] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f155c02fe50 con 0x7f1558071cc0 2024-03-28T23:37:20.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.526+0000 7f15537fe640 1 -- 172.21.15.161:0/2665691829 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=168) ==== 24+0+0 (secure 0 0 0) 0x7f155c0ae340 con 0x7f156c0ac430 2024-03-28T23:37:20.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.527+0000 7f156b59e640 1 --2- 172.21.15.161:0/2665691829 >> [v2:172.21.15.170:6816/3081832568,v1:172.21.15.170:6817/3081832568] conn(0x7f1558071cc0 0x7f1558074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.527+0000 7f156a59c640 1 --2- 172.21.15.161:0/2665691829 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f155806b500 0x7f155806d9c0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f156c06e030 tx=0x7f1560007e40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:20.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.527+0000 7f156b59e640 1 --2- 172.21.15.161:0/2665691829 >> [v2:172.21.15.170:6816/3081832568,v1:172.21.15.170:6817/3081832568] conn(0x7f1558071cc0 0x7f1558074100 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:20.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.528+0000 7f15537fe640 1 -- 172.21.15.161:0/2665691829 <== osd.7 v2:172.21.15.170:6816/3081832568 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f155c02fe50 con 0x7f1558071cc0 2024-03-28T23:37:20.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.541+0000 7f1571180640 1 -- 172.21.15.161:0/2665691829 --> [v2:172.21.15.170:6816/3081832568,v1:172.21.15.170:6817/3081832568] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f156c068f90 con 0x7f1558071cc0 2024-03-28T23:37:20.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.541+0000 7f15537fe640 1 -- 172.21.15.161:0/2665691829 <== osd.7 v2:172.21.15.170:6816/3081832568 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f156c068f90 con 0x7f1558071cc0 2024-03-28T23:37:20.542 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-28T23:37:20.542 INFO:teuthology.orchestra.run.smithi161.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-28T23:37:20.542 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-28T23:37:20.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.542+0000 7f1571180640 1 -- 172.21.15.161:0/2665691829 >> [v2:172.21.15.170:6816/3081832568,v1:172.21.15.170:6817/3081832568] conn(0x7f1558071cc0 msgr2=0x7f1558074100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.542+0000 7f1571180640 1 --2- 172.21.15.161:0/2665691829 >> [v2:172.21.15.170:6816/3081832568,v1:172.21.15.170:6817/3081832568] conn(0x7f1558071cc0 0x7f1558074100 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.542+0000 7f1571180640 1 -- 172.21.15.161:0/2665691829 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f155806b500 msgr2=0x7f155806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.542+0000 7f1571180640 1 --2- 172.21.15.161:0/2665691829 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f155806b500 0x7f155806d9c0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f156c06e030 tx=0x7f1560007e40 comp rx=0 tx=0).stop 2024-03-28T23:37:20.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.542+0000 7f1571180640 1 -- 172.21.15.161:0/2665691829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f156c0ac430 msgr2=0x7f156c0a8dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.542+0000 7f1571180640 1 --2- 172.21.15.161:0/2665691829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f156c0ac430 0x7f156c0a8dd0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f155c0098c0 tx=0x7f155c0046c0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.542+0000 7f156b59e640 1 -- 172.21.15.161:0/2665691829 reap_dead start 2024-03-28T23:37:20.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.542+0000 7f1571180640 1 -- 172.21.15.161:0/2665691829 shutdown_connections 2024-03-28T23:37:20.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.542+0000 7f1571180640 1 -- 172.21.15.161:0/2665691829 >> 172.21.15.161:0/2665691829 conn(0x7f156c06b150 msgr2=0x7f156c06b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:20.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.542+0000 7f1571180640 1 -- 172.21.15.161:0/2665691829 shutdown_connections 2024-03-28T23:37:20.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.542+0000 7f1571180640 1 -- 172.21.15.161:0/2665691829 wait complete. 2024-03-28T23:37:20.553 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-28T23:37:20.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.654+0000 7fe8a8cb1640 1 Processor -- start 2024-03-28T23:37:20.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.654+0000 7fe8a8cb1640 1 -- start start 2024-03-28T23:37:20.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.655+0000 7fe8a8cb1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8a414eca0 0x7fe8a40c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.655+0000 7fe8a8cb1640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8a4151850 con 0x7fe8a40c1dc0 2024-03-28T23:37:20.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.655+0000 7fe8a8cb1640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8a4151b30 con 0x7fe8a40a98c0 2024-03-28T23:37:20.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.655+0000 7fe8a8cb1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe8a4152460 con 0x7fe8a414eca0 2024-03-28T23:37:20.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.655+0000 7fe8a1d9b640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe8a40c1dc0 0x7fe8a406ca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:55864/0 (socket says 172.21.15.161:55864) 2024-03-28T23:37:20.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.656+0000 7fe8a1d9b640 1 -- 172.21.15.161:0/3786558674 learned_addr learned my addr 172.21.15.161:0/3786558674 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:20.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.656+0000 7fe8a259c640 1 --2- 172.21.15.161:0/3786558674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8a414eca0 0x7fe8a40c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.656+0000 7fe8a159a640 1 -- 172.21.15.161:0/3786558674 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3433967006 0 0) 0x7fe8a4151850 con 0x7fe8a40c1dc0 2024-03-28T23:37:20.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.656+0000 7fe8a159a640 1 -- 172.21.15.161:0/3786558674 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe88c003520 con 0x7fe8a40c1dc0 2024-03-28T23:37:20.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.656+0000 7fe8a159a640 1 -- 172.21.15.161:0/3786558674 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2045373803 0 0) 0x7fe8a4151b30 con 0x7fe8a40a98c0 2024-03-28T23:37:20.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.656+0000 7fe8a159a640 1 -- 172.21.15.161:0/3786558674 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8a4151850 con 0x7fe8a40a98c0 2024-03-28T23:37:20.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.656+0000 7fe8a259c640 1 -- 172.21.15.161:0/3786558674 >> v1:172.21.15.170:6789/0 conn(0x7fe8a40a98c0 legacy=0x7fe8a40a9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.656+0000 7fe8a259c640 1 -- 172.21.15.161:0/3786558674 >> v1:172.21.15.161:6789/0 conn(0x7fe8a40c1dc0 legacy=0x7fe8a406ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.656+0000 7fe8a259c640 1 -- 172.21.15.161:0/3786558674 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a4151e10 con 0x7fe8a414eca0 2024-03-28T23:37:20.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.657+0000 7fe8a259c640 1 --2- 172.21.15.161:0/3786558674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8a414eca0 0x7fe8a40c1880 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe8980097d0 tx=0x7fe89802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ba4af93a37595eb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:20.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.657+0000 7fe8a159a640 1 -- 172.21.15.161:0/3786558674 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe898035020 con 0x7fe8a414eca0 2024-03-28T23:37:20.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.657+0000 7fe8a159a640 1 -- 172.21.15.161:0/3786558674 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe898031800 con 0x7fe8a414eca0 2024-03-28T23:37:20.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.657+0000 7fe8a159a640 1 -- 172.21.15.161:0/3786558674 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe898031bb0 con 0x7fe8a414eca0 2024-03-28T23:37:20.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.657+0000 7fe8a8cb1640 1 -- 172.21.15.161:0/3786558674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8a414eca0 msgr2=0x7fe8a40c1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.657+0000 7fe8a8cb1640 1 --2- 172.21.15.161:0/3786558674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8a414eca0 0x7fe8a40c1880 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe8980097d0 tx=0x7fe89802b8a0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.658+0000 7fe8a8cb1640 1 -- 172.21.15.161:0/3786558674 shutdown_connections 2024-03-28T23:37:20.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.658+0000 7fe8a8cb1640 1 --2- 172.21.15.161:0/3786558674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8a414eca0 0x7fe8a40c1880 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.658+0000 7fe8a8cb1640 1 -- 172.21.15.161:0/3786558674 >> 172.21.15.161:0/3786558674 conn(0x7fe8a40c6670 msgr2=0x7fe8a40c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:20.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.658+0000 7fe8a8cb1640 1 -- 172.21.15.161:0/3786558674 shutdown_connections 2024-03-28T23:37:20.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.658+0000 7fe8a8cb1640 1 -- 172.21.15.161:0/3786558674 wait complete. 2024-03-28T23:37:20.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.659+0000 7fe8a8cb1640 1 Processor -- start 2024-03-28T23:37:20.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.659+0000 7fe8a8cb1640 1 -- start start 2024-03-28T23:37:20.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.660+0000 7fe8a8cb1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8a40a98c0 0x7fe8a41eeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.660+0000 7fe8a8cb1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8a40c1dc0 0x7fe8a415f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.660+0000 7fe8a8cb1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8a414eca0 0x7fe8a414bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.660+0000 7fe8a8cb1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a41514b0 con 0x7fe8a40c1dc0 2024-03-28T23:37:20.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.660+0000 7fe8a8cb1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a414f080 con 0x7fe8a414eca0 2024-03-28T23:37:20.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.660+0000 7fe8a8cb1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fe8a4151630 con 0x7fe8a40a98c0 2024-03-28T23:37:20.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.660+0000 7fe8a1d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8a40c1dc0 0x7fe8a415f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.660+0000 7fe8a1d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8a40c1dc0 0x7fe8a415f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45116/0 (socket says 172.21.15.161:45116) 2024-03-28T23:37:20.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.660+0000 7fe8a1d9b640 1 -- 172.21.15.161:0/2546114739 learned_addr learned my addr 172.21.15.161:0/2546114739 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:20.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.660+0000 7fe8a259c640 1 --2- 172.21.15.161:0/2546114739 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8a40a98c0 0x7fe8a41eeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.660+0000 7fe8a2d9d640 1 --2- 172.21.15.161:0/2546114739 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8a414eca0 0x7fe8a414bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.660+0000 7fe8a1d9b640 1 -- 172.21.15.161:0/2546114739 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8a40a98c0 msgr2=0x7fe8a41eeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.660+0000 7fe8a1d9b640 1 --2- 172.21.15.161:0/2546114739 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8a40a98c0 0x7fe8a41eeb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.660+0000 7fe8a1d9b640 1 -- 172.21.15.161:0/2546114739 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8a414eca0 msgr2=0x7fe8a414bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.660+0000 7fe8a1d9b640 1 --2- 172.21.15.161:0/2546114739 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8a414eca0 0x7fe8a414bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.660+0000 7fe8a1d9b640 1 -- 172.21.15.161:0/2546114739 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a41ee080 con 0x7fe8a40c1dc0 2024-03-28T23:37:20.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.660+0000 7fe8a1d9b640 1 --2- 172.21.15.161:0/2546114739 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8a40c1dc0 0x7fe8a415f120 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fe89400c530 tx=0x7fe89400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:20.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.660+0000 7fe892ffd640 1 -- 172.21.15.161:0/2546114739 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe894017070 con 0x7fe8a40c1dc0 2024-03-28T23:37:20.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.660+0000 7fe892ffd640 1 -- 172.21.15.161:0/2546114739 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe894005ce0 con 0x7fe8a40c1dc0 2024-03-28T23:37:20.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.660+0000 7fe892ffd640 1 -- 172.21.15.161:0/2546114739 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe894007270 con 0x7fe8a40c1dc0 2024-03-28T23:37:20.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.660+0000 7fe8a8cb1640 1 -- 172.21.15.161:0/2546114739 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8a4155340 con 0x7fe8a40c1dc0 2024-03-28T23:37:20.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.660+0000 7fe8a8cb1640 1 -- 172.21.15.161:0/2546114739 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8a4155620 con 0x7fe8a40c1dc0 2024-03-28T23:37:20.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.661+0000 7fe8a8cb1640 1 -- 172.21.15.161:0/2546114739 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8a404f4b0 con 0x7fe8a40c1dc0 2024-03-28T23:37:20.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.662+0000 7fe892ffd640 1 -- 172.21.15.161:0/2546114739 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe89401a070 con 0x7fe8a40c1dc0 2024-03-28T23:37:20.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.662+0000 7fe892ffd640 1 --2- 172.21.15.161:0/2546114739 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe88406b4d0 0x7fe88406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.662+0000 7fe8a259c640 1 --2- 172.21.15.161:0/2546114739 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe88406b4d0 0x7fe88406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.662+0000 7fe892ffd640 1 -- 172.21.15.161:0/2546114739 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7fe894089490 con 0x7fe8a40c1dc0 2024-03-28T23:37:20.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.663+0000 7fe8a259c640 1 --2- 172.21.15.161:0/2546114739 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe88406b4d0 0x7fe88406d990 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fe898036000 tx=0x7fe89803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:20.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.667+0000 7fe892ffd640 1 -- 172.21.15.161:0/2546114739 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe894057d10 con 0x7fe8a40c1dc0 2024-03-28T23:37:20.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.836+0000 7fe8a8cb1640 1 -- 172.21.15.161:0/2546114739 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fe8a406c160 con 0x7fe88406b4d0 2024-03-28T23:37:20.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.837+0000 7fe892ffd640 1 -- 172.21.15.161:0/2546114739 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe8a406c160 con 0x7fe88406b4d0 2024-03-28T23:37:20.837 INFO:teuthology.orchestra.run.smithi161.stderr:instructed osd(s) 7 to deep-scrub 2024-03-28T23:37:20.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.839+0000 7fe8a8cb1640 1 -- 172.21.15.161:0/2546114739 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe88406b4d0 msgr2=0x7fe88406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.839+0000 7fe8a8cb1640 1 --2- 172.21.15.161:0/2546114739 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe88406b4d0 0x7fe88406d990 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fe898036000 tx=0x7fe89803c040 comp rx=0 tx=0).stop 2024-03-28T23:37:20.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.839+0000 7fe8a8cb1640 1 -- 172.21.15.161:0/2546114739 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8a40c1dc0 msgr2=0x7fe8a415f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.839+0000 7fe8a8cb1640 1 --2- 172.21.15.161:0/2546114739 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8a40c1dc0 0x7fe8a415f120 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fe89400c530 tx=0x7fe89400c9f0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.839+0000 7fe8a8cb1640 1 -- 172.21.15.161:0/2546114739 shutdown_connections 2024-03-28T23:37:20.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.839+0000 7fe8a8cb1640 1 --2- 172.21.15.161:0/2546114739 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fe88406b4d0 0x7fe88406d990 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.839+0000 7fe8a8cb1640 1 --2- 172.21.15.161:0/2546114739 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8a40c1dc0 0x7fe8a415f120 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.839+0000 7fe8a8cb1640 1 --2- 172.21.15.161:0/2546114739 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8a414eca0 0x7fe8a414bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.839+0000 7fe8a8cb1640 1 --2- 172.21.15.161:0/2546114739 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8a40a98c0 0x7fe8a41eeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.839+0000 7fe8a8cb1640 1 -- 172.21.15.161:0/2546114739 >> 172.21.15.161:0/2546114739 conn(0x7fe8a40c6670 msgr2=0x7fe8a4073000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:20.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.839+0000 7fe8a8cb1640 1 -- 172.21.15.161:0/2546114739 shutdown_connections 2024-03-28T23:37:20.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.839+0000 7fe8a8cb1640 1 -- 172.21.15.161:0/2546114739 wait complete. 2024-03-28T23:37:20.850 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:37:20.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.950+0000 7ff3a491e640 1 Processor -- start 2024-03-28T23:37:20.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.950+0000 7ff3a491e640 1 -- start start 2024-03-28T23:37:20.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.951+0000 7ff3a491e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff3a006ce80 0x7ff3a00aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.951+0000 7ff3a491e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3a014c250 con 0x7ff3a014be70 2024-03-28T23:37:20.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.951+0000 7ff3a491e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3a01529c0 con 0x7ff3a00a9680 2024-03-28T23:37:20.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.951+0000 7ff3a491e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7ff3a00a9dc0 con 0x7ff3a006ce80 2024-03-28T23:37:20.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.951+0000 7ff39e59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ff3a014be70 0x7ff3a00ab1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:55866/0 (socket says 172.21.15.161:55866) 2024-03-28T23:37:20.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.951+0000 7ff39e59c640 1 -- 172.21.15.161:0/2822206836 learned_addr learned my addr 172.21.15.161:0/2822206836 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:20.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.952+0000 7ff39dd9b640 1 --2- 172.21.15.161:0/2822206836 >> v2:172.21.15.170:3300/0 conn(0x7ff3a006ce80 0x7ff3a00aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.952+0000 7ff39d59a640 1 -- 172.21.15.161:0/2822206836 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2827019802 0 0) 0x7ff3a014c250 con 0x7ff3a014be70 2024-03-28T23:37:20.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.952+0000 7ff39d59a640 1 -- 172.21.15.161:0/2822206836 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff38c003520 con 0x7ff3a014be70 2024-03-28T23:37:20.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.952+0000 7ff39dd9b640 1 -- 172.21.15.161:0/2822206836 >> v1:172.21.15.170:6789/0 conn(0x7ff3a00a9680 legacy=0x7ff3a015c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.952+0000 7ff39dd9b640 1 -- 172.21.15.161:0/2822206836 >> v1:172.21.15.161:6789/0 conn(0x7ff3a014be70 legacy=0x7ff3a00ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.952+0000 7ff39d59a640 1 -- 172.21.15.161:0/2822206836 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4008959134 0 0) 0x7ff38c003520 con 0x7ff3a014be70 2024-03-28T23:37:20.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.952+0000 7ff39dd9b640 1 -- 172.21.15.161:0/2822206836 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3a00aa0a0 con 0x7ff3a006ce80 2024-03-28T23:37:20.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.953+0000 7ff39dd9b640 1 --2- 172.21.15.161:0/2822206836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3a006ce80 0x7ff3a00aa5c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff3880097a0 tx=0x7ff38802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d282e299d3e4cb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:20.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.953+0000 7ff39d59a640 1 -- 172.21.15.161:0/2822206836 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff388037070 con 0x7ff3a006ce80 2024-03-28T23:37:20.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.953+0000 7ff39d59a640 1 -- 172.21.15.161:0/2822206836 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff388030510 con 0x7ff3a006ce80 2024-03-28T23:37:20.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.953+0000 7ff39d59a640 1 -- 172.21.15.161:0/2822206836 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff398001670 con 0x7ff3a006ce80 2024-03-28T23:37:20.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.953+0000 7ff3a491e640 1 -- 172.21.15.161:0/2822206836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3a006ce80 msgr2=0x7ff3a00aa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.953+0000 7ff3a491e640 1 --2- 172.21.15.161:0/2822206836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3a006ce80 0x7ff3a00aa5c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff3880097a0 tx=0x7ff38802b790 comp rx=0 tx=0).stop 2024-03-28T23:37:20.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.954+0000 7ff3a491e640 1 -- 172.21.15.161:0/2822206836 shutdown_connections 2024-03-28T23:37:20.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.954+0000 7ff3a491e640 1 --2- 172.21.15.161:0/2822206836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3a006ce80 0x7ff3a00aa5c0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.954+0000 7ff3a491e640 1 -- 172.21.15.161:0/2822206836 >> 172.21.15.161:0/2822206836 conn(0x7ff3a00c6670 msgr2=0x7ff3a00c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:20.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.954+0000 7ff3a491e640 1 -- 172.21.15.161:0/2822206836 shutdown_connections 2024-03-28T23:37:20.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.954+0000 7ff3a491e640 1 -- 172.21.15.161:0/2822206836 wait complete. 2024-03-28T23:37:20.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.955+0000 7ff3a491e640 1 Processor -- start 2024-03-28T23:37:20.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.955+0000 7ff3a491e640 1 -- start start 2024-03-28T23:37:20.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.955+0000 7ff3a491e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3a006ce80 0x7ff3a0154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.955+0000 7ff3a491e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3a00a9680 0x7ff3a00c5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.955+0000 7ff3a491e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3a014be70 0x7ff3a01e6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.955+0000 7ff3a491e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a00aba30 con 0x7ff3a00a9680 2024-03-28T23:37:20.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.955+0000 7ff3a491e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a00ab8b0 con 0x7ff3a014be70 2024-03-28T23:37:20.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.955+0000 7ff3a491e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7ff3a006b150 con 0x7ff3a006ce80 2024-03-28T23:37:20.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.956+0000 7ff39dd9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3a00a9680 0x7ff3a00c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.956+0000 7ff39dd9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3a00a9680 0x7ff3a00c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45132/0 (socket says 172.21.15.161:45132) 2024-03-28T23:37:20.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.956+0000 7ff39dd9b640 1 -- 172.21.15.161:0/607148321 learned_addr learned my addr 172.21.15.161:0/607148321 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:20.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.956+0000 7ff39e59c640 1 --2- 172.21.15.161:0/607148321 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3a006ce80 0x7ff3a0154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.956+0000 7ff39ed9d640 1 --2- 172.21.15.161:0/607148321 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3a014be70 0x7ff3a01e6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.956+0000 7ff39dd9b640 1 -- 172.21.15.161:0/607148321 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3a006ce80 msgr2=0x7ff3a0154720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.956+0000 7ff39dd9b640 1 --2- 172.21.15.161:0/607148321 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3a006ce80 0x7ff3a0154720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.956+0000 7ff39dd9b640 1 -- 172.21.15.161:0/607148321 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3a014be70 msgr2=0x7ff3a01e6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:20.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.956+0000 7ff39dd9b640 1 --2- 172.21.15.161:0/607148321 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3a014be70 0x7ff3a01e6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:20.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.956+0000 7ff39dd9b640 1 -- 172.21.15.161:0/607148321 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3a0154c60 con 0x7ff3a00a9680 2024-03-28T23:37:20.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.956+0000 7ff39dd9b640 1 --2- 172.21.15.161:0/607148321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3a00a9680 0x7ff3a00c5610 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7ff38802bda0 tx=0x7ff3880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:20.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.956+0000 7ff396ffd640 1 -- 172.21.15.161:0/607148321 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff388042070 con 0x7ff3a00a9680 2024-03-28T23:37:20.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.956+0000 7ff396ffd640 1 -- 172.21.15.161:0/607148321 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff38803e870 con 0x7ff3a00a9680 2024-03-28T23:37:20.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.956+0000 7ff3a491e640 1 -- 172.21.15.161:0/607148321 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3a0154f00 con 0x7ff3a00a9680 2024-03-28T23:37:20.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.956+0000 7ff396ffd640 1 -- 172.21.15.161:0/607148321 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff388037040 con 0x7ff3a00a9680 2024-03-28T23:37:20.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.956+0000 7ff3a491e640 1 -- 172.21.15.161:0/607148321 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3a01ea550 con 0x7ff3a00a9680 2024-03-28T23:37:20.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.957+0000 7ff3a491e640 1 -- 172.21.15.161:0/607148321 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3a004f4b0 con 0x7ff3a00a9680 2024-03-28T23:37:20.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.958+0000 7ff396ffd640 1 -- 172.21.15.161:0/607148321 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff388045070 con 0x7ff3a00a9680 2024-03-28T23:37:20.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.958+0000 7ff396ffd640 1 --2- 172.21.15.161:0/607148321 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff38006b4d0 0x7ff38006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:20.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.958+0000 7ff396ffd640 1 -- 172.21.15.161:0/607148321 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7ff3880abe90 con 0x7ff3a00a9680 2024-03-28T23:37:20.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.958+0000 7ff39e59c640 1 --2- 172.21.15.161:0/607148321 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff38006b4d0 0x7ff38006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:20.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.958+0000 7ff39e59c640 1 --2- 172.21.15.161:0/607148321 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff38006b4d0 0x7ff38006d990 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7ff398005880 tx=0x7ff398009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:20.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:20.962+0000 7ff396ffd640 1 -- 172.21.15.161:0/607148321 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff38807ae80 con 0x7ff3a00a9680 2024-03-28T23:37:21.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:21.131+0000 7ff3a491e640 1 -- 172.21.15.161:0/607148321 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff3a00c0ff0 con 0x7ff38006b4d0 2024-03-28T23:37:21.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:21.132+0000 7ff396ffd640 1 -- 172.21.15.161:0/607148321 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13452 (secure 0 0 0) 0x7ff3a00c0ff0 con 0x7ff38006b4d0 2024-03-28T23:37:21.133 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:37:21.133 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:37:21.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:21.134+0000 7ff3a491e640 1 -- 172.21.15.161:0/607148321 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff38006b4d0 msgr2=0x7ff38006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:21.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:21.134+0000 7ff3a491e640 1 --2- 172.21.15.161:0/607148321 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff38006b4d0 0x7ff38006d990 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7ff398005880 tx=0x7ff398009040 comp rx=0 tx=0).stop 2024-03-28T23:37:21.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:21.134+0000 7ff3a491e640 1 -- 172.21.15.161:0/607148321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3a00a9680 msgr2=0x7ff3a00c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:21.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:21.134+0000 7ff3a491e640 1 --2- 172.21.15.161:0/607148321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3a00a9680 0x7ff3a00c5610 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7ff38802bda0 tx=0x7ff3880046c0 comp rx=0 tx=0).stop 2024-03-28T23:37:21.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:21.134+0000 7ff3a491e640 1 -- 172.21.15.161:0/607148321 shutdown_connections 2024-03-28T23:37:21.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:21.134+0000 7ff3a491e640 1 --2- 172.21.15.161:0/607148321 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7ff38006b4d0 0x7ff38006d990 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:21.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:21.134+0000 7ff3a491e640 1 --2- 172.21.15.161:0/607148321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3a00a9680 0x7ff3a00c5610 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:21.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:21.134+0000 7ff3a491e640 1 --2- 172.21.15.161:0/607148321 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3a014be70 0x7ff3a01e6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:21.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:21.134+0000 7ff3a491e640 1 --2- 172.21.15.161:0/607148321 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3a006ce80 0x7ff3a0154720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:21.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:21.134+0000 7ff3a491e640 1 -- 172.21.15.161:0/607148321 >> 172.21.15.161:0/607148321 conn(0x7ff3a00c6670 msgr2=0x7ff3a014b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:21.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:21.134+0000 7ff3a491e640 1 -- 172.21.15.161:0/607148321 shutdown_connections 2024-03-28T23:37:21.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:21.134+0000 7ff3a491e640 1 -- 172.21.15.161:0/607148321 wait complete. 2024-03-28T23:37:21.145 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":391,"stamp":"2024-03-28T23:37:19.400544+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":266184,"kb_used_data":16736,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754708536,"statfs":{"total":773094113280,"available":772821540864,"internally_reserved":0,"allocated":17137664,"data_stored":9529728,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.252592"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":449,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-28T23:37:12.155372+0000","last_change":"2024-03-28T23:29:41.302669+0000","last_active":"2024-03-28T23:37:12.155372+0000","last_peered":"2024-03-28T23:37:12.155372+0000","last_clean":"2024-03-28T23:37:12.155372+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:37:12.155372+0000","last_undegraded":"2024-03-28T23:37:12.155372+0000","last_fullsized":"2024-03-28T23:37:12.155372+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T23:29:40.232557+0000","last_clean_scrub_stamp":"2024-03-28T23:29:40.232557+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:09.290194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33324,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338516,"statfs":{"total":96636764160,"available":96602640384,"internally_reserved":0,"allocated":2191360,"data_stored":1633992,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574943,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574943,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574942,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574942,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574942,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574942,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33324,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338516,"statfs":{"total":96636764160,"available":96602640384,"internally_reserved":0,"allocated":2191360,"data_stored":1633992,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:37:21.148 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-28T23:29:40.232557+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=23, tm_min=29, tm_sec=40, tm_wday=3, tm_yday=88, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=28, tm_hour=23, tm_min=37, tm_sec=16, tm_wday=3, tm_yday=88, tm_isdst=0) 2024-03-28T23:37:21.148 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-28T23:37:26.616 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:37:26.619 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:37:41.149 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T23:37:41.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.264+0000 7fa030fd0640 1 Processor -- start 2024-03-28T23:37:41.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.264+0000 7fa030fd0640 1 -- start start 2024-03-28T23:37:41.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.265+0000 7fa030fd0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa02c14be70 0x7fa02c0ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:41.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.265+0000 7fa030fd0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa02c0ab6e0 0x7fa02c0aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:41.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.265+0000 7fa030fd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa02c06ce80 0x7fa02c0aab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:41.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.265+0000 7fa030fd0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa02c14c250 con 0x7fa02c14be70 2024-03-28T23:37:41.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.265+0000 7fa030fd0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7fa02c1529c0 con 0x7fa02c0ab6e0 2024-03-28T23:37:41.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.265+0000 7fa030fd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa02c0a9dc0 con 0x7fa02c06ce80 2024-03-28T23:37:41.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.265+0000 7fa02a59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa02c14be70 0x7fa02c0ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:41.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.265+0000 7fa02a59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa02c14be70 0x7fa02c0ab1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51868/0 (socket says 172.21.15.161:51868) 2024-03-28T23:37:41.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.265+0000 7fa02a59c640 1 -- 172.21.15.161:0/3136234389 learned_addr learned my addr 172.21.15.161:0/3136234389 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:41.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.266+0000 7fa02ad9d640 1 --2- 172.21.15.161:0/3136234389 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa02c06ce80 0x7fa02c0aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:41.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.266+0000 7fa023fff640 1 --2- 172.21.15.161:0/3136234389 >> v2:172.21.15.170:3300/0 conn(0x7fa02c0ab6e0 0x7fa02c0aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:41.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.266+0000 7fa02a59c640 1 -- 172.21.15.161:0/3136234389 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa02c06ce80 msgr2=0x7fa02c0aab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:41.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.266+0000 7fa02a59c640 1 --2- 172.21.15.161:0/3136234389 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa02c06ce80 0x7fa02c0aab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.266+0000 7fa02a59c640 1 -- 172.21.15.161:0/3136234389 >> v2:172.21.15.170:3300/0 conn(0x7fa02c0ab6e0 msgr2=0x7fa02c0aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:41.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.266+0000 7fa02a59c640 1 --2- 172.21.15.161:0/3136234389 >> v2:172.21.15.170:3300/0 conn(0x7fa02c0ab6e0 0x7fa02c0aa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.266+0000 7fa02a59c640 1 -- 172.21.15.161:0/3136234389 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa02c0aa0a0 con 0x7fa02c14be70 2024-03-28T23:37:41.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.266+0000 7fa02a59c640 1 --2- 172.21.15.161:0/3136234389 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa02c14be70 0x7fa02c0ab1a0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fa02c06f540 tx=0x7fa01c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52cf696e12e91f56 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:41.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.267+0000 7fa029d9b640 1 -- 172.21.15.161:0/3136234389 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01c038070 con 0x7fa02c14be70 2024-03-28T23:37:41.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.267+0000 7fa029d9b640 1 -- 172.21.15.161:0/3136234389 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa01c02f050 con 0x7fa02c14be70 2024-03-28T23:37:41.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.267+0000 7fa029d9b640 1 -- 172.21.15.161:0/3136234389 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01c031970 con 0x7fa02c14be70 2024-03-28T23:37:41.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.267+0000 7fa030fd0640 1 -- 172.21.15.161:0/3136234389 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa02c14be70 msgr2=0x7fa02c0ab1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:41.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.267+0000 7fa030fd0640 1 --2- 172.21.15.161:0/3136234389 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa02c14be70 0x7fa02c0ab1a0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fa02c06f540 tx=0x7fa01c02b7f0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.267+0000 7fa030fd0640 1 -- 172.21.15.161:0/3136234389 shutdown_connections 2024-03-28T23:37:41.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.267+0000 7fa030fd0640 1 --2- 172.21.15.161:0/3136234389 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa02c06ce80 0x7fa02c0aab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.267+0000 7fa030fd0640 1 --2- 172.21.15.161:0/3136234389 >> v2:172.21.15.170:3300/0 conn(0x7fa02c0ab6e0 0x7fa02c0aa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.267+0000 7fa030fd0640 1 --2- 172.21.15.161:0/3136234389 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa02c14be70 0x7fa02c0ab1a0 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.267+0000 7fa030fd0640 1 -- 172.21.15.161:0/3136234389 >> 172.21.15.161:0/3136234389 conn(0x7fa02c0c6670 msgr2=0x7fa02c0c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:41.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.267+0000 7fa030fd0640 1 -- 172.21.15.161:0/3136234389 shutdown_connections 2024-03-28T23:37:41.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.267+0000 7fa030fd0640 1 -- 172.21.15.161:0/3136234389 wait complete. 2024-03-28T23:37:41.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.269+0000 7fa030fd0640 1 Processor -- start 2024-03-28T23:37:41.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.269+0000 7fa030fd0640 1 -- start start 2024-03-28T23:37:41.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.269+0000 7fa030fd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa02c06ce80 0x7fa02c1e8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:41.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.269+0000 7fa030fd0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa02c0ab6e0 0x7fa02c0c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:41.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.269+0000 7fa030fd0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa02c14be70 0x7fa02c153420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:41.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.269+0000 7fa030fd0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c06b2d0 con 0x7fa02c0ab6e0 2024-03-28T23:37:41.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.269+0000 7fa030fd0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c06b150 con 0x7fa02c14be70 2024-03-28T23:37:41.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.269+0000 7fa030fd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7fa02c0c1520 con 0x7fa02c06ce80 2024-03-28T23:37:41.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.269+0000 7fa023fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa02c0ab6e0 0x7fa02c0c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:41.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.269+0000 7fa023fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa02c0ab6e0 0x7fa02c0c55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51878/0 (socket says 172.21.15.161:51878) 2024-03-28T23:37:41.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.269+0000 7fa023fff640 1 -- 172.21.15.161:0/861170283 learned_addr learned my addr 172.21.15.161:0/861170283 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:41.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.270+0000 7fa02a59c640 1 --2- 172.21.15.161:0/861170283 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa02c06ce80 0x7fa02c1e8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:41.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.270+0000 7fa02ad9d640 1 --2- 172.21.15.161:0/861170283 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa02c14be70 0x7fa02c153420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:41.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.270+0000 7fa023fff640 1 -- 172.21.15.161:0/861170283 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa02c06ce80 msgr2=0x7fa02c1e8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:41.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.270+0000 7fa023fff640 1 --2- 172.21.15.161:0/861170283 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa02c06ce80 0x7fa02c1e8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.270+0000 7fa023fff640 1 -- 172.21.15.161:0/861170283 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa02c14be70 msgr2=0x7fa02c153420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:41.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.270+0000 7fa023fff640 1 --2- 172.21.15.161:0/861170283 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa02c14be70 0x7fa02c153420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.270+0000 7fa023fff640 1 -- 172.21.15.161:0/861170283 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa02c0c4550 con 0x7fa02c0ab6e0 2024-03-28T23:37:41.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.270+0000 7fa02a59c640 1 --2- 172.21.15.161:0/861170283 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa02c06ce80 0x7fa02c1e8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:37:41.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.270+0000 7fa02ad9d640 1 --2- 172.21.15.161:0/861170283 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa02c14be70 0x7fa02c153420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:37:41.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.270+0000 7fa023fff640 1 --2- 172.21.15.161:0/861170283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa02c0ab6e0 0x7fa02c0c55f0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7fa01400d5f0 tx=0x7fa01400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:41.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.270+0000 7fa022ffd640 1 -- 172.21.15.161:0/861170283 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0140125f0 con 0x7fa02c0ab6e0 2024-03-28T23:37:41.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.270+0000 7fa022ffd640 1 -- 172.21.15.161:0/861170283 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa014019070 con 0x7fa02c0ab6e0 2024-03-28T23:37:41.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.270+0000 7fa030fd0640 1 -- 172.21.15.161:0/861170283 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa02c0c4830 con 0x7fa02c0ab6e0 2024-03-28T23:37:41.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.270+0000 7fa022ffd640 1 -- 172.21.15.161:0/861170283 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0140129d0 con 0x7fa02c0ab6e0 2024-03-28T23:37:41.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.270+0000 7fa030fd0640 1 -- 172.21.15.161:0/861170283 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa02c0c4df0 con 0x7fa02c0ab6e0 2024-03-28T23:37:41.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.271+0000 7fa030fd0640 1 -- 172.21.15.161:0/861170283 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa02c04f4b0 con 0x7fa02c0ab6e0 2024-03-28T23:37:41.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.272+0000 7fa022ffd640 1 -- 172.21.15.161:0/861170283 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa014012b70 con 0x7fa02c0ab6e0 2024-03-28T23:37:41.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.272+0000 7fa022ffd640 1 --2- 172.21.15.161:0/861170283 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa00c06b4d0 0x7fa00c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:41.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.272+0000 7fa022ffd640 1 -- 172.21.15.161:0/861170283 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7fa01408a5b0 con 0x7fa02c0ab6e0 2024-03-28T23:37:41.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.272+0000 7fa02a59c640 1 --2- 172.21.15.161:0/861170283 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa00c06b4d0 0x7fa00c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:41.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.273+0000 7fa02a59c640 1 --2- 172.21.15.161:0/861170283 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa00c06b4d0 0x7fa00c06d990 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fa01c035000 tx=0x7fa01c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:41.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.277+0000 7fa022ffd640 1 -- 172.21.15.161:0/861170283 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa014059480 con 0x7fa02c0ab6e0 2024-03-28T23:37:41.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.449+0000 7fa030fd0640 1 -- 172.21.15.161:0/861170283 --> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa02c0c0ff0 con 0x7fa00c06b4d0 2024-03-28T23:37:41.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.450+0000 7fa022ffd640 1 -- 172.21.15.161:0/861170283 <== mgr.4107 v2:172.21.15.161:6832/1776925385 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13462 (secure 0 0 0) 0x7fa02c0c0ff0 con 0x7fa00c06b4d0 2024-03-28T23:37:41.450 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:37:41.451 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-28T23:37:41.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.452+0000 7fa030fd0640 1 -- 172.21.15.161:0/861170283 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa00c06b4d0 msgr2=0x7fa00c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:41.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.452+0000 7fa030fd0640 1 --2- 172.21.15.161:0/861170283 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa00c06b4d0 0x7fa00c06d990 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fa01c035000 tx=0x7fa01c008030 comp rx=0 tx=0).stop 2024-03-28T23:37:41.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.452+0000 7fa030fd0640 1 -- 172.21.15.161:0/861170283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa02c0ab6e0 msgr2=0x7fa02c0c55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:41.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.452+0000 7fa030fd0640 1 --2- 172.21.15.161:0/861170283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa02c0ab6e0 0x7fa02c0c55f0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7fa01400d5f0 tx=0x7fa01400dab0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.452+0000 7fa030fd0640 1 -- 172.21.15.161:0/861170283 shutdown_connections 2024-03-28T23:37:41.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.452+0000 7fa030fd0640 1 --2- 172.21.15.161:0/861170283 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7fa00c06b4d0 0x7fa00c06d990 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.452+0000 7fa030fd0640 1 --2- 172.21.15.161:0/861170283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa02c0ab6e0 0x7fa02c0c55f0 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.452+0000 7fa030fd0640 1 --2- 172.21.15.161:0/861170283 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa02c14be70 0x7fa02c153420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.452+0000 7fa030fd0640 1 --2- 172.21.15.161:0/861170283 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa02c06ce80 0x7fa02c1e8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.452+0000 7fa030fd0640 1 -- 172.21.15.161:0/861170283 >> 172.21.15.161:0/861170283 conn(0x7fa02c0c6670 msgr2=0x7fa02c06c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:41.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.452+0000 7fa030fd0640 1 -- 172.21.15.161:0/861170283 shutdown_connections 2024-03-28T23:37:41.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.452+0000 7fa030fd0640 1 -- 172.21.15.161:0/861170283 wait complete. 2024-03-28T23:37:41.463 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":402,"stamp":"2024-03-28T23:37:41.404790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":266168,"kb_used_data":16736,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754708552,"statfs":{"total":773094113280,"available":772821557248,"internally_reserved":0,"allocated":17137664,"data_stored":9529728,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002442"},"pg_stats":[{"pgid":"1.0","version":"168'119","reported_seq":460,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-28T23:37:21.266722+0000","last_change":"2024-03-28T23:37:21.266760+0000","last_active":"2024-03-28T23:37:21.266722+0000","last_peered":"2024-03-28T23:37:21.266722+0000","last_clean":"2024-03-28T23:37:21.266722+0000","last_became_active":"2024-03-28T23:29:41.301838+0000","last_became_peered":"2024-03-28T23:29:41.301838+0000","last_unstale":"2024-03-28T23:37:21.266722+0000","last_undegraded":"2024-03-28T23:37:21.266722+0000","last_fullsized":"2024-03-28T23:37:21.266722+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'119","last_scrub_stamp":"2024-03-28T23:37:21.266625+0000","last_deep_scrub":"168'119","last_deep_scrub_stamp":"2024-03-28T23:37:21.266625+0000","last_clean_scrub_stamp":"2024-03-28T23:37:21.266625+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:03:47.669782+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65172,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33316,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338524,"statfs":{"total":96636764160,"available":96602648576,"internally_reserved":0,"allocated":2191360,"data_stored":1633992,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574947,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574947,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574946,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574946,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574946,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1043624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33316,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338524,"statfs":{"total":96636764160,"available":96602648576,"internally_reserved":0,"allocated":2191360,"data_stored":1633992,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32586,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T23:37:41.464 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-28T23:37:41.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.564+0000 7f0687f8c640 1 Processor -- start 2024-03-28T23:37:41.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.564+0000 7f0687f8c640 1 -- start start 2024-03-28T23:37:41.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.565+0000 7f0687f8c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0680152940 0x7f06800c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:41.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.565+0000 7f0687f8c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06800ca580 0x7f06800c8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:41.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.565+0000 7f0687f8c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06800c6df0 con 0x7f06800c9480 2024-03-28T23:37:41.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.565+0000 7f0687f8c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f06800c70d0 con 0x7f0680152940 2024-03-28T23:37:41.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.565+0000 7f0687f8c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f06800c5830 con 0x7f06800ca580 2024-03-28T23:37:41.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.566+0000 7f0685527640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06800ca580 0x7f06800c8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:41.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.566+0000 7f0685d28640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0680152940 0x7f06800c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:41.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.566+0000 7f0685527640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06800ca580 0x7f06800c8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.161:42596/0 (socket says 172.21.15.161:42596) 2024-03-28T23:37:41.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.566+0000 7f0685527640 1 -- 172.21.15.161:0/1554295769 learned_addr learned my addr 172.21.15.161:0/1554295769 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:41.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.566+0000 7f0686529640 1 --1- 172.21.15.161:0/1554295769 >> v1:172.21.15.170:6789/0 conn(0x7f06800c9480 0x7f0680155220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:49550/0 (socket says 172.21.15.161:49550) 2024-03-28T23:37:41.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.566+0000 7f0685d28640 1 -- 172.21.15.161:0/1554295769 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06800ca580 msgr2=0x7f06800c8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:41.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.566+0000 7f0685d28640 1 --2- 172.21.15.161:0/1554295769 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06800ca580 0x7f06800c8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.566+0000 7f0685d28640 1 -- 172.21.15.161:0/1554295769 >> v1:172.21.15.170:6789/0 conn(0x7f06800c9480 legacy=0x7f0680155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:41.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.566+0000 7f0685d28640 1 -- 172.21.15.161:0/1554295769 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06800c5b10 con 0x7f0680152940 2024-03-28T23:37:41.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.566+0000 7f0685527640 1 --2- 172.21.15.161:0/1554295769 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06800ca580 0x7f06800c8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:37:41.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.567+0000 7f0685d28640 1 --2- 172.21.15.161:0/1554295769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0680152940 0x7f06800c8a00 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f067c009d90 tx=0x7f067c02bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e02f64390dffdb92 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:41.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.567+0000 7f0684d26640 1 -- 172.21.15.161:0/1554295769 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067c039070 con 0x7f0680152940 2024-03-28T23:37:41.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.567+0000 7f0684d26640 1 -- 172.21.15.161:0/1554295769 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f067c02cb30 con 0x7f0680152940 2024-03-28T23:37:41.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.567+0000 7f0684d26640 1 -- 172.21.15.161:0/1554295769 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067c034440 con 0x7f0680152940 2024-03-28T23:37:41.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.567+0000 7f0687f8c640 1 -- 172.21.15.161:0/1554295769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0680152940 msgr2=0x7f06800c8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:41.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.567+0000 7f0687f8c640 1 --2- 172.21.15.161:0/1554295769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0680152940 0x7f06800c8a00 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f067c009d90 tx=0x7f067c02bd80 comp rx=0 tx=0).stop 2024-03-28T23:37:41.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.567+0000 7f0687f8c640 1 -- 172.21.15.161:0/1554295769 shutdown_connections 2024-03-28T23:37:41.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.567+0000 7f0687f8c640 1 --2- 172.21.15.161:0/1554295769 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06800ca580 0x7f06800c8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.567+0000 7f0687f8c640 1 --2- 172.21.15.161:0/1554295769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0680152940 0x7f06800c8a00 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.567+0000 7f0687f8c640 1 -- 172.21.15.161:0/1554295769 >> 172.21.15.161:0/1554295769 conn(0x7f06800caa00 msgr2=0x7f06800cae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:41.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.568+0000 7f0687f8c640 1 -- 172.21.15.161:0/1554295769 shutdown_connections 2024-03-28T23:37:41.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.568+0000 7f0687f8c640 1 -- 172.21.15.161:0/1554295769 wait complete. 2024-03-28T23:37:41.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.569+0000 7f0687f8c640 1 Processor -- start 2024-03-28T23:37:41.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.569+0000 7f0687f8c640 1 -- start start 2024-03-28T23:37:41.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.569+0000 7f0687f8c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06800c9480 0x7f068014f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:41.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.569+0000 7f0687f8c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06800ca580 0x7f068006b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:41.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.569+0000 7f0687f8c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0680152940 0x7f068014e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:41.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.569+0000 7f0687f8c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f06800c61c0 con 0x7f06800ca580 2024-03-28T23:37:41.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.569+0000 7f0687f8c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f06800c6040 con 0x7f0680152940 2024-03-28T23:37:41.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.569+0000 7f0687f8c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0680155930 con 0x7f06800c9480 2024-03-28T23:37:41.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.569+0000 7f0685527640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06800ca580 0x7f068006b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:41.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.569+0000 7f0685527640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06800ca580 0x7f068006b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51880/0 (socket says 172.21.15.161:51880) 2024-03-28T23:37:41.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.569+0000 7f0685527640 1 -- 172.21.15.161:0/1303720726 learned_addr learned my addr 172.21.15.161:0/1303720726 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:41.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.570+0000 7f0685d28640 1 --2- 172.21.15.161:0/1303720726 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06800c9480 0x7f068014f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:41.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.570+0000 7f0686529640 1 --2- 172.21.15.161:0/1303720726 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0680152940 0x7f068014e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:41.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.570+0000 7f0685527640 1 -- 172.21.15.161:0/1303720726 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06800c9480 msgr2=0x7f068014f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:41.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.570+0000 7f0685527640 1 --2- 172.21.15.161:0/1303720726 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06800c9480 0x7f068014f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.570+0000 7f0685527640 1 -- 172.21.15.161:0/1303720726 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0680152940 msgr2=0x7f068014e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:41.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.570+0000 7f0685527640 1 --2- 172.21.15.161:0/1303720726 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0680152940 0x7f068014e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.570+0000 7f0685527640 1 -- 172.21.15.161:0/1303720726 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06800c4dd0 con 0x7f06800ca580 2024-03-28T23:37:41.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.570+0000 7f0686529640 1 --2- 172.21.15.161:0/1303720726 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0680152940 0x7f068014e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T23:37:41.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.570+0000 7f0685527640 1 --2- 172.21.15.161:0/1303720726 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06800ca580 0x7f068006b9d0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f067000e9d0 tx=0x7f067000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:41.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.570+0000 7f06767fc640 1 -- 172.21.15.161:0/1303720726 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0670017070 con 0x7f06800ca580 2024-03-28T23:37:41.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.570+0000 7f06767fc640 1 -- 172.21.15.161:0/1303720726 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f067000ca60 con 0x7f06800ca580 2024-03-28T23:37:41.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.570+0000 7f0687f8c640 1 -- 172.21.15.161:0/1303720726 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f068014ee10 con 0x7f06800ca580 2024-03-28T23:37:41.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.570+0000 7f06767fc640 1 -- 172.21.15.161:0/1303720726 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067000ce10 con 0x7f06800ca580 2024-03-28T23:37:41.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.570+0000 7f0687f8c640 1 -- 172.21.15.161:0/1303720726 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06800c48b0 con 0x7f06800ca580 2024-03-28T23:37:41.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.571+0000 7f0687f8c640 1 -- 172.21.15.161:0/1303720726 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f068004f4b0 con 0x7f06800ca580 2024-03-28T23:37:41.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.572+0000 7f06767fc640 1 -- 172.21.15.161:0/1303720726 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f067001a070 con 0x7f06800ca580 2024-03-28T23:37:41.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.572+0000 7f06767fc640 1 --2- 172.21.15.161:0/1303720726 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f066406b4d0 0x7f066406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:41.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.572+0000 7f06767fc640 1 -- 172.21.15.161:0/1303720726 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7f067008a080 con 0x7f06800ca580 2024-03-28T23:37:41.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.572+0000 7f0685d28640 1 --2- 172.21.15.161:0/1303720726 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f066406b4d0 0x7f066406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:41.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.572+0000 7f0685d28640 1 --2- 172.21.15.161:0/1303720726 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f066406b4d0 0x7f066406d990 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f067c036000 tx=0x7f067c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:41.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.576+0000 7f06767fc640 1 -- 172.21.15.161:0/1303720726 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0670058900 con 0x7f06800ca580 2024-03-28T23:37:41.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.749+0000 7f0687f8c640 1 -- 172.21.15.161:0/1303720726 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f068006bf10 con 0x7f06800ca580 2024-03-28T23:37:41.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.752+0000 7f06767fc640 1 -- 172.21.15.161:0/1303720726 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v57)=0 v57) ==== 123+0+0 (secure 0 0 0) 0x7f067005c9a0 con 0x7f06800ca580 2024-03-28T23:37:41.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.754+0000 7f06767fc640 1 -- 172.21.15.161:0/1303720726 <== mon.0 v2:172.21.15.161:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f0670060f60 con 0x7f06800ca580 2024-03-28T23:37:41.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.755+0000 7f0687f8c640 1 -- 172.21.15.161:0/1303720726 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f066406b4d0 msgr2=0x7f066406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:41.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.755+0000 7f0687f8c640 1 --2- 172.21.15.161:0/1303720726 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f066406b4d0 0x7f066406d990 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f067c036000 tx=0x7f067c030000 comp rx=0 tx=0).stop 2024-03-28T23:37:41.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.755+0000 7f0687f8c640 1 -- 172.21.15.161:0/1303720726 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06800ca580 msgr2=0x7f068006b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:41.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.755+0000 7f0687f8c640 1 --2- 172.21.15.161:0/1303720726 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06800ca580 0x7f068006b9d0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f067000e9d0 tx=0x7f067000ee90 comp rx=0 tx=0).stop 2024-03-28T23:37:41.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.755+0000 7f0687f8c640 1 -- 172.21.15.161:0/1303720726 shutdown_connections 2024-03-28T23:37:41.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.755+0000 7f0687f8c640 1 --2- 172.21.15.161:0/1303720726 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f066406b4d0 0x7f066406d990 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.755+0000 7f0687f8c640 1 --2- 172.21.15.161:0/1303720726 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06800ca580 0x7f068006b9d0 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.755+0000 7f0687f8c640 1 --2- 172.21.15.161:0/1303720726 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0680152940 0x7f068014e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.755+0000 7f0687f8c640 1 --2- 172.21.15.161:0/1303720726 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06800c9480 0x7f068014f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.755+0000 7f0687f8c640 1 -- 172.21.15.161:0/1303720726 >> 172.21.15.161:0/1303720726 conn(0x7f06800caa00 msgr2=0x7f06801514e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:41.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.755+0000 7f0687f8c640 1 -- 172.21.15.161:0/1303720726 shutdown_connections 2024-03-28T23:37:41.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.755+0000 7f0687f8c640 1 -- 172.21.15.161:0/1303720726 wait complete. 2024-03-28T23:37:41.766 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-28T23:37:41.766 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T23:37:41.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.868+0000 7f0ae9d4b640 1 Processor -- start 2024-03-28T23:37:41.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.868+0000 7f0ae9d4b640 1 -- start start 2024-03-28T23:37:41.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.868+0000 7f0ae9d4b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0ae406ce80 0x7f0ae40aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:41.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.868+0000 7f0ae9d4b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ae40a9680 0x7f0ae40aab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:41.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.868+0000 7f0ae9d4b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ae414c250 con 0x7f0ae414be70 2024-03-28T23:37:41.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.868+0000 7f0ae9d4b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 -- 0x7f0ae41529c0 con 0x7f0ae406ce80 2024-03-28T23:37:41.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.868+0000 7f0ae9d4b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0ae40a9dc0 con 0x7f0ae40a9680 2024-03-28T23:37:41.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.869+0000 7f0ae3fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ae40a9680 0x7f0ae40aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:41.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.869+0000 7f0adbfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0ae406ce80 0x7f0ae40aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:41.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.869+0000 7f0ae37fe640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f0ae414be70 0x7f0ae40ab1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.161:49560/0 (socket says 172.21.15.161:49560) 2024-03-28T23:37:41.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.869+0000 7f0ae37fe640 1 -- 172.21.15.161:0/1939293674 learned_addr learned my addr 172.21.15.161:0/1939293674 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-28T23:37:41.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.869+0000 7f0adbfff640 1 -- 172.21.15.161:0/1939293674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ae40a9680 msgr2=0x7f0ae40aab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T23:37:41.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.869+0000 7f0adbfff640 1 --2- 172.21.15.161:0/1939293674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ae40a9680 0x7f0ae40aab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.869+0000 7f0adbfff640 1 -- 172.21.15.161:0/1939293674 >> v1:172.21.15.170:6789/0 conn(0x7f0ae414be70 legacy=0x7f0ae40ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:41.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.869+0000 7f0adbfff640 1 -- 172.21.15.161:0/1939293674 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae40aa0a0 con 0x7f0ae406ce80 2024-03-28T23:37:41.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.869+0000 7f0ae3fff640 1 --2- 172.21.15.161:0/1939293674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ae40a9680 0x7f0ae40aab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T23:37:41.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.870+0000 7f0adbfff640 1 --2- 172.21.15.161:0/1939293674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ae406ce80 0x7f0ae40aa5c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0097a0 tx=0x7f0acc02b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=804fdd66f563c249 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:41.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.870+0000 7f0ae2ffd640 1 -- 172.21.15.161:0/1939293674 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0acc038070 con 0x7f0ae406ce80 2024-03-28T23:37:41.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.870+0000 7f0ae2ffd640 1 -- 172.21.15.161:0/1939293674 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f0acc031700 con 0x7f0ae406ce80 2024-03-28T23:37:41.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.870+0000 7f0ae2ffd640 1 -- 172.21.15.161:0/1939293674 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0acc031af0 con 0x7f0ae406ce80 2024-03-28T23:37:41.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.871+0000 7f0ae9d4b640 1 -- 172.21.15.161:0/1939293674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ae406ce80 msgr2=0x7f0ae40aa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:41.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.871+0000 7f0ae9d4b640 1 --2- 172.21.15.161:0/1939293674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ae406ce80 0x7f0ae40aa5c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0097a0 tx=0x7f0acc02b7a0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.871+0000 7f0ae9d4b640 1 -- 172.21.15.161:0/1939293674 shutdown_connections 2024-03-28T23:37:41.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.871+0000 7f0ae9d4b640 1 --2- 172.21.15.161:0/1939293674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ae40a9680 0x7f0ae40aab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.871+0000 7f0ae9d4b640 1 --2- 172.21.15.161:0/1939293674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ae406ce80 0x7f0ae40aa5c0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.871+0000 7f0ae9d4b640 1 -- 172.21.15.161:0/1939293674 >> 172.21.15.161:0/1939293674 conn(0x7f0ae40c6670 msgr2=0x7f0ae40c6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:41.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.871+0000 7f0ae9d4b640 1 -- 172.21.15.161:0/1939293674 shutdown_connections 2024-03-28T23:37:41.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.871+0000 7f0ae9d4b640 1 -- 172.21.15.161:0/1939293674 wait complete. 2024-03-28T23:37:41.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.872+0000 7f0ae9d4b640 1 Processor -- start 2024-03-28T23:37:41.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.872+0000 7f0ae9d4b640 1 -- start start 2024-03-28T23:37:41.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.873+0000 7f0ae9d4b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ae406ce80 0x7f0ae4148c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:41.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.873+0000 7f0ae9d4b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ae40a9680 0x7f0ae4155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:41.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.873+0000 7f0ae9d4b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ae414be70 0x7f0ae41dea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:41.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.873+0000 7f0ae9d4b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae40aba30 con 0x7f0ae406ce80 2024-03-28T23:37:41.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.873+0000 7f0ae9d4b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae40ab8b0 con 0x7f0ae414be70 2024-03-28T23:37:41.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.873+0000 7f0ae9d4b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 -- 0x7f0ae406b150 con 0x7f0ae40a9680 2024-03-28T23:37:41.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.873+0000 7f0ae37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ae406ce80 0x7f0ae4148c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:41.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.873+0000 7f0ae37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ae406ce80 0x7f0ae4148c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51888/0 (socket says 172.21.15.161:51888) 2024-03-28T23:37:41.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.873+0000 7f0ae37fe640 1 -- 172.21.15.161:0/84782451 learned_addr learned my addr 172.21.15.161:0/84782451 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-28T23:37:41.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.873+0000 7f0ae3fff640 1 --2- 172.21.15.161:0/84782451 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ae414be70 0x7f0ae41dea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:41.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.873+0000 7f0adbfff640 1 --2- 172.21.15.161:0/84782451 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ae40a9680 0x7f0ae4155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:41.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.873+0000 7f0ae37fe640 1 -- 172.21.15.161:0/84782451 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ae40a9680 msgr2=0x7f0ae4155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:41.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.873+0000 7f0ae37fe640 1 --2- 172.21.15.161:0/84782451 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ae40a9680 0x7f0ae4155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.873+0000 7f0ae37fe640 1 -- 172.21.15.161:0/84782451 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ae414be70 msgr2=0x7f0ae41dea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:41.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.873+0000 7f0ae37fe640 1 --2- 172.21.15.161:0/84782451 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ae414be70 0x7f0ae41dea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:41.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.873+0000 7f0ae37fe640 1 -- 172.21.15.161:0/84782451 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae4155c60 con 0x7f0ae406ce80 2024-03-28T23:37:41.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.873+0000 7f0ae37fe640 1 --2- 172.21.15.161:0/84782451 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ae406ce80 0x7f0ae4148c40 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f0ad400c3f0 tx=0x7f0ad400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:41.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.873+0000 7f0ae0ff9640 1 -- 172.21.15.161:0/84782451 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4017070 con 0x7f0ae406ce80 2024-03-28T23:37:41.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.873+0000 7f0ae0ff9640 1 -- 172.21.15.161:0/84782451 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f0ad4005ce0 con 0x7f0ae406ce80 2024-03-28T23:37:41.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.873+0000 7f0ae9d4b640 1 -- 172.21.15.161:0/84782451 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ae4153c10 con 0x7f0ae406ce80 2024-03-28T23:37:41.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.873+0000 7f0ae0ff9640 1 -- 172.21.15.161:0/84782451 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4007330 con 0x7f0ae406ce80 2024-03-28T23:37:41.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.874+0000 7f0ae9d4b640 1 -- 172.21.15.161:0/84782451 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ae41ed050 con 0x7f0ae406ce80 2024-03-28T23:37:41.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.874+0000 7f0ae9d4b640 1 -- 172.21.15.161:0/84782451 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ae4153de0 con 0x7f0ae406ce80 2024-03-28T23:37:41.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.875+0000 7f0ae0ff9640 1 -- 172.21.15.161:0/84782451 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0ad401a070 con 0x7f0ae406ce80 2024-03-28T23:37:41.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.875+0000 7f0ae0ff9640 1 --2- 172.21.15.161:0/84782451 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0ac406b4d0 0x7f0ac406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T23:37:41.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.875+0000 7f0ae0ff9640 1 -- 172.21.15.161:0/84782451 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5765+0+0 (secure 0 0 0) 0x7f0ad40895f0 con 0x7f0ae406ce80 2024-03-28T23:37:41.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.875+0000 7f0adbfff640 1 --2- 172.21.15.161:0/84782451 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0ac406b4d0 0x7f0ac406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T23:37:41.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.876+0000 7f0adbfff640 1 --2- 172.21.15.161:0/84782451 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0ac406b4d0 0x7f0ac406d990 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0acc035000 tx=0x7f0acc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T23:37:41.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:41.880+0000 7f0ae0ff9640 1 -- 172.21.15.161:0/84782451 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0ad4057c30 con 0x7f0ae406ce80 2024-03-28T23:37:42.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:42.088+0000 7f0ae9d4b640 1 -- 172.21.15.161:0/84782451 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ae4152f90 con 0x7f0ae406ce80 2024-03-28T23:37:42.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:42.088+0000 7f0ae0ff9640 1 -- 172.21.15.161:0/84782451 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) ==== 95+0+3883 (secure 0 0 0) 0x7f0ad405bcd0 con 0x7f0ae406ce80 2024-03-28T23:37:42.089 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-28T23:37:42.089 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":162,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/557522063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6833","nonce":557522063},{"type":"v1","addr":"172.21.15.170:6835","nonce":557522063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":7499,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6834/3015739917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3015739917},{"type":"v1","addr":"172.21.15.170:6834","nonce":3015739917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":156},{"gid":9522,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/1423643374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1423643374},{"type":"v1","addr":"172.21.15.161:6836","nonce":1423643374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":162},{"gid":9603,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/4022047700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":4022047700},{"type":"v1","addr":"172.21.15.161:6837","nonce":4022047700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":162}],"filesystems":[]} 2024-03-28T23:37:42.090 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 162 2024-03-28T23:37:42.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:42.090+0000 7f0ae9d4b640 1 -- 172.21.15.161:0/84782451 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0ac406b4d0 msgr2=0x7f0ac406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:42.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:42.090+0000 7f0ae9d4b640 1 --2- 172.21.15.161:0/84782451 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0ac406b4d0 0x7f0ac406d990 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0acc035000 tx=0x7f0acc008030 comp rx=0 tx=0).stop 2024-03-28T23:37:42.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:42.090+0000 7f0ae9d4b640 1 -- 172.21.15.161:0/84782451 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ae406ce80 msgr2=0x7f0ae4148c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T23:37:42.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:42.090+0000 7f0ae9d4b640 1 --2- 172.21.15.161:0/84782451 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ae406ce80 0x7f0ae4148c40 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f0ad400c3f0 tx=0x7f0ad400c7f0 comp rx=0 tx=0).stop 2024-03-28T23:37:42.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:42.090+0000 7f0ae9d4b640 1 -- 172.21.15.161:0/84782451 shutdown_connections 2024-03-28T23:37:42.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:42.090+0000 7f0ae9d4b640 1 --2- 172.21.15.161:0/84782451 >> [v2:172.21.15.161:6832/1776925385,v1:172.21.15.161:6833/1776925385] conn(0x7f0ac406b4d0 0x7f0ac406d990 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:42.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:42.090+0000 7f0ae9d4b640 1 --2- 172.21.15.161:0/84782451 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ae414be70 0x7f0ae41dea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:42.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:42.090+0000 7f0ae9d4b640 1 --2- 172.21.15.161:0/84782451 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ae40a9680 0x7f0ae4155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:42.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:42.090+0000 7f0ae9d4b640 1 --2- 172.21.15.161:0/84782451 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ae406ce80 0x7f0ae4148c40 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T23:37:42.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:42.090+0000 7f0ae9d4b640 1 -- 172.21.15.161:0/84782451 >> 172.21.15.161:0/84782451 conn(0x7f0ae40c6670 msgr2=0x7f0ae40c21a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T23:37:42.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:42.090+0000 7f0ae9d4b640 1 -- 172.21.15.161:0/84782451 shutdown_connections 2024-03-28T23:37:42.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-28T23:37:42.090+0000 7f0ae9d4b640 1 -- 172.21.15.161:0/84782451 wait complete. 2024-03-28T23:37:42.101 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-28T23:37:42.102 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-28T23:37:42.102 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T23:37:42.102 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:37:42.110 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T23:37:42.111 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T23:37:42.111 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:37:42.160 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T23:37:42.161 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-28T23:37:42.161 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:37:42.172 INFO:tasks.ceph.mds.b:Stopped 2024-03-28T23:37:42.172 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-28T23:37:42.172 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:37:42.224 INFO:tasks.ceph.mds.d:Stopped 2024-03-28T23:37:42.224 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-28T23:37:42.224 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-28T23:37:42.224 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:37:42.250 INFO:tasks.ceph.osd.0:Stopped 2024-03-28T23:37:42.250 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-28T23:37:42.250 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:37:42.315 INFO:tasks.ceph.osd.1:Stopped 2024-03-28T23:37:42.315 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-28T23:37:42.315 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:37:42.379 INFO:tasks.ceph.osd.2:Stopped 2024-03-28T23:37:42.380 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-28T23:37:42.380 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:37:42.449 INFO:tasks.ceph.osd.3:Stopped 2024-03-28T23:37:42.449 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-28T23:37:42.449 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:37:42.486 INFO:tasks.ceph.osd.4:Stopped 2024-03-28T23:37:42.486 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-28T23:37:42.486 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:37:42.553 INFO:tasks.ceph.osd.5:Stopped 2024-03-28T23:37:42.553 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-28T23:37:42.554 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:37:42.622 INFO:tasks.ceph.osd.6:Stopped 2024-03-28T23:37:42.622 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-28T23:37:42.622 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:37:42.690 INFO:tasks.ceph.osd.7:Stopped 2024-03-28T23:37:42.690 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-28T23:37:42.691 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-28T23:37:42.691 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:37:42.716 INFO:tasks.ceph.mgr.y:Stopped 2024-03-28T23:37:42.717 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-28T23:37:42.717 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:37:42.758 INFO:tasks.ceph.mgr.x:Stopped 2024-03-28T23:37:42.759 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-28T23:37:42.759 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-28T23:37:42.759 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:37:42.770 INFO:tasks.ceph.mon.a:Stopped 2024-03-28T23:37:42.771 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-28T23:37:42.771 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:37:42.812 INFO:tasks.ceph.mon.b:Stopped 2024-03-28T23:37:42.812 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-28T23:37:42.812 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T23:37:42.865 INFO:tasks.ceph.mon.c:Stopped 2024-03-28T23:37:42.866 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-28T23:37:42.866 DEBUG:teuthology.orchestra.run.smithi161:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'Replacing daemon mds' | egrep -v '\(MDS_INSUFFICIENT_STANDBY\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(FS_DEGRADED\)' | head -n 1 2024-03-28T23:37:42.903 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi161.front.sepia.ceph.com 2024-03-28T23:37:42.903 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-28T23:37:43.108 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi161.front.sepia.ceph.com 2024-03-28T23:37:43.108 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-28T23:37:43.174 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi161.front.sepia.ceph.com 2024-03-28T23:37:43.174 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-28T23:37:43.229 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi161.front.sepia.ceph.com 2024-03-28T23:37:43.230 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-28T23:37:43.271 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi170.front.sepia.ceph.com 2024-03-28T23:37:43.271 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-28T23:37:43.401 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi170.front.sepia.ceph.com 2024-03-28T23:37:43.401 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-28T23:37:43.495 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi170.front.sepia.ceph.com 2024-03-28T23:37:43.495 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-28T23:37:43.553 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi170.front.sepia.ceph.com 2024-03-28T23:37:43.553 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-28T23:37:43.612 INFO:tasks.ceph:Archiving mon data... 2024-03-28T23:37:43.613 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627196/data/mon.a.tgz 2024-03-28T23:37:43.614 DEBUG:teuthology.orchestra.run.smithi161:> mktemp 2024-03-28T23:37:43.630 INFO:teuthology.orchestra.run.smithi161.stdout:/tmp/tmp.Gx2XJiMDjb 2024-03-28T23:37:43.630 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Gx2XJiMDjb 2024-03-28T23:37:43.853 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0666 /tmp/tmp.Gx2XJiMDjb 2024-03-28T23:37:43.938 DEBUG:teuthology.orchestra.remote:smithi161:/tmp/tmp.Gx2XJiMDjb is 594KB 2024-03-28T23:37:44.041 DEBUG:teuthology.orchestra.run.smithi161:> rm -fr /tmp/tmp.Gx2XJiMDjb 2024-03-28T23:37:44.057 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627196/data/mon.b.tgz 2024-03-28T23:37:44.057 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2024-03-28T23:37:44.075 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.olFHK3ts20 2024-03-28T23:37:44.076 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.olFHK3ts20 2024-03-28T23:37:44.286 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.olFHK3ts20 2024-03-28T23:37:44.371 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.olFHK3ts20 is 623KB 2024-03-28T23:37:44.471 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.olFHK3ts20 2024-03-28T23:37:44.485 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627196/data/mon.c.tgz 2024-03-28T23:37:44.486 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2024-03-28T23:37:44.540 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.hNru2zvkfY 2024-03-28T23:37:44.540 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.hNru2zvkfY 2024-03-28T23:37:44.732 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.hNru2zvkfY 2024-03-28T23:37:44.816 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.hNru2zvkfY is 623KB 2024-03-28T23:37:44.917 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.hNru2zvkfY 2024-03-28T23:37:44.931 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-28T23:37:44.932 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-28T23:37:44.934 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-28T23:37:45.021 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-28T23:37:45.052 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-28T23:37:45.053 INFO:tasks.ceph:Archiving crash dumps... 2024-03-28T23:37:45.055 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627196/remote/smithi161/crash 2024-03-28T23:37:45.056 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-28T23:37:45.091 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627196/remote/smithi170/crash 2024-03-28T23:37:45.092 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-28T23:37:45.118 INFO:tasks.ceph:Shutting down logrotate 2024-03-28T23:37:45.119 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:37:45.135 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T23:37:45.207 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-28T23:37:45.234 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-28T23:37:45.273 INFO:tasks.ceph:Compressing logs... 2024-03-28T23:37:45.274 DEBUG:teuthology.orchestra.run.smithi161:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T23:37:45.276 DEBUG:teuthology.orchestra.run.smithi170:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T23:37:45.366 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-28T23:37:45.367 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37796.log 2024-03-28T23:37:45.367 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37836.log 2024-03-28T23:37:45.367 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.37796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37676.log 2024-03-28T23:37:45.367 INFO:teuthology.orchestra.run.smithi170.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37796.log.gz/var/log/ceph/ceph-client.admin.37836.log: 2024-03-28T23:37:45.367 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-28T23:37:45.367 INFO:teuthology.orchestra.run.smithi170.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37836.log.gz 2024-03-28T23:37:45.368 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.37676.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-28T23:37:45.368 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph.audit.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37676.log.gz 2024-03-28T23:37:45.368 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-28T23:37:45.368 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37756.log 2024-03-28T23:37:45.368 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37876.log 2024-03-28T23:37:45.368 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.37756.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-28T23:37:45.368 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.37876.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37756.log.gz 2024-03-28T23:37:45.369 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-28T23:37:45.369 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37876.log.gz 2024-03-28T23:37:45.369 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37916.log 2024-03-28T23:37:45.369 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37716.log 2024-03-28T23:37:45.369 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.37916.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-28T23:37:45.369 INFO:teuthology.orchestra.run.smithi170.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37916.log.gz 2024-03-28T23:37:45.369 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.37716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42037.log 2024-03-28T23:37:45.370 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-mgr.x.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37716.log.gz 2024-03-28T23:37:45.370 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-28T23:37:45.370 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.1.42037.log: 89.7% 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-28T23:37:45.370 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/ceph-client.1.42037.log.gz 2024-03-28T23:37:45.370 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-28T23:37:45.370 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-28T23:37:45.370 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-mon.b.log: 92.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-28T23:37:45.370 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37636.log 2024-03-28T23:37:45.371 INFO:teuthology.orchestra.run.smithi170.stderr: 94.1%/var/log/ceph/ceph-mds.d.log: -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-28T23:37:45.371 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-28T23:37:45.371 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.37636.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37636.log.gz 2024-03-28T23:37:45.395 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-mon.c.log: 90.3% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-28T23:37:45.427 INFO:teuthology.orchestra.run.smithi170.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-28T23:37:45.586 INFO:teuthology.orchestra.run.smithi170.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-28T23:37:45.606 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62272.log 2024-03-28T23:37:45.606 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59135.log 2024-03-28T23:37:45.606 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.62272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43110.log 2024-03-28T23:37:45.606 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59135.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62272.log.gz 2024-03-28T23:37:45.606 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55315.log 2024-03-28T23:37:45.607 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59135.log.gz 2024-03-28T23:37:45.607 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71658.log 2024-03-28T23:37:45.607 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43110.log: /var/log/ceph/ceph-client.admin.55315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57043.log 2024-03-28T23:37:45.607 INFO:teuthology.orchestra.run.smithi161.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43110.log.gz 2024-03-28T23:37:45.607 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.71658.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55315.log.gz 2024-03-28T23:37:45.607 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43441.log 2024-03-28T23:37:45.607 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3%/var/log/ceph/ceph-client.admin.57043.log: -- replaced with /var/log/ceph/ceph-client.admin.71658.log.gz 2024-03-28T23:37:45.607 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52758.log 2024-03-28T23:37:45.608 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43441.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57043.log.gz 2024-03-28T23:37:45.608 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67549.log 2024-03-28T23:37:45.608 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43441.log.gz 2024-03-28T23:37:45.608 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45427.log 2024-03-28T23:37:45.608 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.67549.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52758.log.gz 2024-03-28T23:37:45.608 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51516.log 2024-03-28T23:37:45.608 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.45427.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67549.log.gz 2024-03-28T23:37:45.608 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-28T23:37:45.609 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54253.log 2024-03-28T23:37:45.609 INFO:teuthology.orchestra.run.smithi161.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.45427.log.gz 2024-03-28T23:37:45.609 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-mon.a.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51516.log.gz 2024-03-28T23:37:45.609 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69529.log 2024-03-28T23:37:45.609 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43399.log 2024-03-28T23:37:45.609 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54253.log: /var/log/ceph/ceph-client.admin.69529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70573.log 2024-03-28T23:37:45.609 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54253.log.gz 2024-03-28T23:37:45.609 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43399.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.69529.log.gz 2024-03-28T23:37:45.610 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68647.log 2024-03-28T23:37:45.610 INFO:teuthology.orchestra.run.smithi161.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43399.log.gz 2024-03-28T23:37:45.610 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.70573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70732.log 2024-03-28T23:37:45.610 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71424.log 2024-03-28T23:37:45.610 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.68647.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.70573.log.gz 2024-03-28T23:37:45.610 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.70732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58177.log 2024-03-28T23:37:45.610 INFO:teuthology.orchestra.run.smithi161.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.68647.log.gz 2024-03-28T23:37:45.610 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70732.log.gz 2024-03-28T23:37:45.611 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59265.log 2024-03-28T23:37:45.611 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.71424.log: /var/log/ceph/ceph-client.admin.58177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70195.log 2024-03-28T23:37:45.611 INFO:teuthology.orchestra.run.smithi161.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.71424.log.gz 2024-03-28T23:37:45.611 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59265.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58177.log.gz 2024-03-28T23:37:45.611 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46496.log 2024-03-28T23:37:45.611 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59265.log.gz 2024-03-28T23:37:45.611 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.70195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55273.log 2024-03-28T23:37:45.611 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54437.log 2024-03-28T23:37:45.612 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46496.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70195.log.gz 2024-03-28T23:37:45.612 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74682.log 2024-03-28T23:37:45.612 INFO:teuthology.orchestra.run.smithi161.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46496.log.gz 2024-03-28T23:37:45.612 INFO:teuthology.orchestra.run.smithi161.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.55273.log.gz 2024-03-28T23:37:45.612 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49313.log 2024-03-28T23:37:45.612 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54437.log: /var/log/ceph/ceph-client.admin.74682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42942.log 2024-03-28T23:37:45.612 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54437.log.gz 2024-03-28T23:37:45.612 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49313.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74682.log.gz 2024-03-28T23:37:45.612 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63123.log 2024-03-28T23:37:45.613 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.42942.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49313.log.gz 2024-03-28T23:37:45.613 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56246.log 2024-03-28T23:37:45.613 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60453.log 2024-03-28T23:37:45.613 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.63123.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42942.log.gz 2024-03-28T23:37:45.613 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58727.log 2024-03-28T23:37:45.613 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63123.log.gz 2024-03-28T23:37:45.613 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56246.log.gz 2024-03-28T23:37:45.613 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50339.log 2024-03-28T23:37:45.614 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60453.log: /var/log/ceph/ceph-client.admin.58727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44476.log 2024-03-28T23:37:45.614 INFO:teuthology.orchestra.run.smithi161.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60453.log.gz 2024-03-28T23:37:45.614 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50339.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58727.log.gz 2024-03-28T23:37:45.614 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59391.log 2024-03-28T23:37:45.614 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50339.log.gz 2024-03-28T23:37:45.614 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70153.log 2024-03-28T23:37:45.614 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70237.log 2024-03-28T23:37:45.614 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59391.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44476.log.gz 2024-03-28T23:37:45.615 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.70153.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59391.log.gz 2024-03-28T23:37:45.615 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48755.log 2024-03-28T23:37:45.615 INFO:teuthology.orchestra.run.smithi161.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.70153.log.gz 2024-03-28T23:37:45.615 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55399.log 2024-03-28T23:37:45.615 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.70237.log: /var/log/ceph/ceph-client.admin.48755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60537.log 2024-03-28T23:37:45.615 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70237.log.gz 2024-03-28T23:37:45.615 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49849.log 2024-03-28T23:37:45.615 INFO:teuthology.orchestra.run.smithi161.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48755.log.gz 2024-03-28T23:37:45.615 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55399.log.gz 2024-03-28T23:37:45.616 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64456.log 2024-03-28T23:37:45.616 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60537.log.gz 2024-03-28T23:37:45.616 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65740.log 2024-03-28T23:37:45.616 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49849.log: /var/log/ceph/ceph-client.admin.64456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51348.log 2024-03-28T23:37:45.616 INFO:teuthology.orchestra.run.smithi161.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49849.log.gz 2024-03-28T23:37:45.616 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64456.log.gz 2024-03-28T23:37:45.616 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.65740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41110.log 2024-03-28T23:37:45.616 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66514.log 2024-03-28T23:37:45.616 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51348.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65740.log.gz 2024-03-28T23:37:45.617 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.41110.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.51348.log.gz 2024-03-28T23:37:45.617 INFO:teuthology.orchestra.run.smithi161.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49110.log 2024-03-28T23:37:45.617 INFO:teuthology.orchestra.run.smithi161.stderr:gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41110.log.gz -5 2024-03-28T23:37:45.617 INFO:teuthology.orchestra.run.smithi161.stderr: --verbose --/var/log/ceph/ceph-client.admin.66514.log: /var/log/ceph/ceph-client.admin.74785.log 2024-03-28T23:37:45.617 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47783.log 2024-03-28T23:37:45.617 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66514.log.gz 2024-03-28T23:37:45.617 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.74785.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49110.log.gz 2024-03-28T23:37:45.617 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52027.log 2024-03-28T23:37:45.618 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74785.log.gz 2024-03-28T23:37:45.618 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65782.log 2024-03-28T23:37:45.618 INFO:teuthology.orchestra.run.smithi161.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47783.log.gz 2024-03-28T23:37:45.618 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71466.log 2024-03-28T23:37:45.618 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52027.log: /var/log/ceph/ceph-client.admin.65782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61521.log 2024-03-28T23:37:45.618 INFO:teuthology.orchestra.run.smithi161.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52027.log.gz 2024-03-28T23:37:45.618 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.71466.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65782.log.gz 2024-03-28T23:37:45.618 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70405.log 2024-03-28T23:37:45.619 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71466.log.gz 2024-03-28T23:37:45.619 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.61521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43315.log 2024-03-28T23:37:45.619 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61521.log.gz 2024-03-28T23:37:45.619 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.70405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59872.log 2024-03-28T23:37:45.619 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70405.log.gz 2024-03-28T23:37:45.619 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65950.log 2024-03-28T23:37:45.619 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43315.log.gz 2024-03-28T23:37:45.619 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66226.log 2024-03-28T23:37:45.619 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.65950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75499.log 2024-03-28T23:37:45.620 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59872.log.gz 2024-03-28T23:37:45.620 INFO:teuthology.orchestra.run.smithi161.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.65950.log.gz 2024-03-28T23:37:45.620 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.66226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52195.log 2024-03-28T23:37:45.620 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49710.log 2024-03-28T23:37:45.620 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.75499.log: /var/log/ceph/ceph-client.admin.52195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57929.log 2024-03-28T23:37:45.620 INFO:teuthology.orchestra.run.smithi161.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.75499.log.gz 2024-03-28T23:37:45.620 INFO:teuthology.orchestra.run.smithi161.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.66226.log.gz 2024-03-28T23:37:45.620 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52195.log.gz 2024-03-28T23:37:45.621 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58391.log 2024-03-28T23:37:45.621 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41904.log 2024-03-28T23:37:45.621 INFO:teuthology.orchestra.run.smithi161.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49710.log.gz 2024-03-28T23:37:45.621 INFO:teuthology.orchestra.run.smithi161.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57929.log.gz 2024-03-28T23:37:45.621 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69943.log 2024-03-28T23:37:45.621 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.41904.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58391.log.gz 2024-03-28T23:37:45.621 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62434.log 2024-03-28T23:37:45.621 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41904.log.gz 2024-03-28T23:37:45.621 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.69943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49891.log 2024-03-28T23:37:45.622 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69943.log.gz 2024-03-28T23:37:45.622 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.62434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47389.log 2024-03-28T23:37:45.622 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62434.log.gz 2024-03-28T23:37:45.622 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44713.log 2024-03-28T23:37:45.622 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49891.log: /var/log/ceph/ceph-client.admin.47389.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49891.log.gz 2024-03-28T23:37:45.622 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54395.log 2024-03-28T23:37:45.622 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41339.log 2024-03-28T23:37:45.622 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44713.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47389.log.gz 2024-03-28T23:37:45.622 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54395.log: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.44713.log.gz 2024-03-28T23:37:45.623 INFO:teuthology.orchestra.run.smithi161.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.61899.log 2024-03-28T23:37:45.623 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.54395.log.gz -5 2024-03-28T23:37:45.623 INFO:teuthology.orchestra.run.smithi161.stderr: --verbose -- /var/log/ceph/ceph-client.admin.56595.log 2024-03-28T23:37:45.623 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.41339.log: /var/log/ceph/ceph-client.admin.61899.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41339.log.gz 2024-03-28T23:37:45.623 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74102.log 2024-03-28T23:37:45.623 INFO:teuthology.orchestra.run.smithi161.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61899.log.gz 2024-03-28T23:37:45.623 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70363.log 2024-03-28T23:37:45.623 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56595.log: /var/log/ceph/ceph-client.admin.74102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67796.log 2024-03-28T23:37:45.623 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56595.log.gz 2024-03-28T23:37:45.623 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.70363.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.74102.log.gz 2024-03-28T23:37:45.624 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73807.log 2024-03-28T23:37:45.624 INFO:teuthology.orchestra.run.smithi161.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.70363.log.gz 2024-03-28T23:37:45.624 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.67796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66984.log 2024-03-28T23:37:45.624 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72502.log 2024-03-28T23:37:45.624 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.73807.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67796.log.gz 2024-03-28T23:37:45.624 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.66984.log: gzip -5 --verbose 88.7% -- -- replaced with /var/log/ceph/ceph-client.admin.73807.log.gz /var/log/ceph/ceph.tmp-client.admin.31117.log 2024-03-28T23:37:45.624 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:37:45.624 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66984.log.gz 2024-03-28T23:37:45.625 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61191.log 2024-03-28T23:37:45.625 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.72502.log: /var/log/ceph/ceph.tmp-client.admin.31117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41423.log 2024-03-28T23:37:45.625 INFO:teuthology.orchestra.run.smithi161.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31117.log.gz 2024-03-28T23:37:45.625 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72502.log.gz 2024-03-28T23:37:45.625 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.61191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65241.log 2024-03-28T23:37:45.625 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.41423.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61191.log.gz 2024-03-28T23:37:45.625 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49798.log 2024-03-28T23:37:45.625 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65387.log 2024-03-28T23:37:45.626 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.65241.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41423.log.gz 2024-03-28T23:37:45.626 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63736.log 2024-03-28T23:37:45.626 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65241.log.gz 2024-03-28T23:37:45.626 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49798.log.gz 2024-03-28T23:37:45.626 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74258.log 2024-03-28T23:37:45.626 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.65387.log: /var/log/ceph/ceph-client.admin.63736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66400.log 2024-03-28T23:37:45.626 INFO:teuthology.orchestra.run.smithi161.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.65387.log.gz 2024-03-28T23:37:45.627 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.74258.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63736.log.gz 2024-03-28T23:37:45.627 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65824.log 2024-03-28T23:37:45.627 INFO:teuthology.orchestra.run.smithi161.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.74258.log.gz 2024-03-28T23:37:45.627 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.66400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50381.log 2024-03-28T23:37:45.627 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65303.log 2024-03-28T23:37:45.627 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.65824.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66400.log.gz 2024-03-28T23:37:45.627 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43843.log 2024-03-28T23:37:45.627 INFO:teuthology.orchestra.run.smithi161.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.65824.log.gz 2024-03-28T23:37:45.628 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50381.log.gz 2024-03-28T23:37:45.628 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65471.log 2024-03-28T23:37:45.628 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.65303.log: /var/log/ceph/ceph-client.admin.43843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53328.log 2024-03-28T23:37:45.628 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65303.log.gz 2024-03-28T23:37:45.628 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.65471.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43843.log.gz 2024-03-28T23:37:45.628 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67717.log 2024-03-28T23:37:45.628 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65471.log.gz 2024-03-28T23:37:45.628 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75327.log 2024-03-28T23:37:45.628 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51264.log 2024-03-28T23:37:45.629 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53328.log.gz 2024-03-28T23:37:45.629 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.67717.log: /var/log/ceph/ceph-client.admin.75327.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67717.log.gz 2024-03-28T23:37:45.629 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69068.log 2024-03-28T23:37:45.629 INFO:teuthology.orchestra.run.smithi161.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.75327.log.gz 2024-03-28T23:37:45.629 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54647.log 2024-03-28T23:37:45.629 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51264.log: /var/log/ceph/ceph-client.admin.69068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73309.log 2024-03-28T23:37:45.629 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51264.log.gz 2024-03-28T23:37:45.629 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74556.log 2024-03-28T23:37:45.630 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54647.log.gz 2024-03-28T23:37:45.630 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.73309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73351.log 2024-03-28T23:37:45.630 INFO:teuthology.orchestra.run.smithi161.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.69068.log.gz 2024-03-28T23:37:45.630 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- 87.8% /var/log/ceph/ceph-client.admin.60999.log -- replaced with /var/log/ceph/ceph-client.admin.73309.log.gz 2024-03-28T23:37:45.630 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:37:45.630 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.74556.log: /var/log/ceph/ceph-client.admin.73351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68521.log 2024-03-28T23:37:45.630 INFO:teuthology.orchestra.run.smithi161.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.74556.log.gz 2024-03-28T23:37:45.630 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72221.log 2024-03-28T23:37:45.631 INFO:teuthology.orchestra.run.smithi161.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.73351.log.gz 2024-03-28T23:37:45.631 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.68521.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60999.log.gz 2024-03-28T23:37:45.631 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53909.log 2024-03-28T23:37:45.631 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.68521.log.gz 2024-03-28T23:37:45.631 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58643.log 2024-03-28T23:37:45.631 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.72221.log: gzip/var/log/ceph/ceph-client.admin.53909.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.49190.log 2024-03-28T23:37:45.631 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72221.log.gz 2024-03-28T23:37:45.631 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58643.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53909.log.gz 2024-03-28T23:37:45.632 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42884.log 2024-03-28T23:37:45.632 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44060.log 2024-03-28T23:37:45.632 INFO:teuthology.orchestra.run.smithi161.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58643.log.gz 2024-03-28T23:37:45.632 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49190.log: /var/log/ceph/ceph-client.admin.42884.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49190.log.gz 2024-03-28T23:37:45.632 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62955.log 2024-03-28T23:37:45.632 INFO:teuthology.orchestra.run.smithi161.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42884.log.gz 2024-03-28T23:37:45.632 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51744.log 2024-03-28T23:37:45.632 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44060.log: /var/log/ceph/ceph-client.admin.62955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60239.log 2024-03-28T23:37:45.632 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44060.log.gz 2024-03-28T23:37:45.633 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51744.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62955.log.gz 2024-03-28T23:37:45.633 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59620.log 2024-03-28T23:37:45.633 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51744.log.gz 2024-03-28T23:37:45.633 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53825.log 2024-03-28T23:37:45.633 INFO:teuthology.orchestra.run.smithi161.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60239.log.gz 2024-03-28T23:37:45.633 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55567.log 2024-03-28T23:37:45.633 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59620.log: /var/log/ceph/ceph-client.admin.53825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66292.log 2024-03-28T23:37:45.633 INFO:teuthology.orchestra.run.smithi161.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59620.log.gz 2024-03-28T23:37:45.634 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72007.log 2024-03-28T23:37:45.634 INFO:teuthology.orchestra.run.smithi161.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53825.log.gz 2024-03-28T23:37:45.634 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55567.log.gz 2024-03-28T23:37:45.634 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.66292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58967.log 2024-03-28T23:37:45.634 INFO:teuthology.orchestra.run.smithi161.stderr: 88.0%/var/log/ceph/ceph-client.admin.72007.log: -- replaced with /var/log/ceph/ceph-client.admin.66292.log.gz 2024-03-28T23:37:45.634 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72877.log 2024-03-28T23:37:45.634 INFO:teuthology.orchestra.run.smithi161.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.72007.log.gz 2024-03-28T23:37:45.634 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56893.log 2024-03-28T23:37:45.635 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59914.log 2024-03-28T23:37:45.635 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.72877.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58967.log.gz 2024-03-28T23:37:45.635 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41988.log 2024-03-28T23:37:45.635 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.72877.log.gz 2024-03-28T23:37:45.635 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56893.log.gz 2024-03-28T23:37:45.635 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50742.log 2024-03-28T23:37:45.635 INFO:teuthology.orchestra.run.smithi161.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.59914.log.gz 2024-03-28T23:37:45.635 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67026.log 2024-03-28T23:37:45.635 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.41988.log: /var/log/ceph/ceph-client.admin.50742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69327.log 2024-03-28T23:37:45.636 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.67026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67591.log 2024-03-28T23:37:45.636 INFO:teuthology.orchestra.run.smithi161.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.41988.log.gz 2024-03-28T23:37:45.636 INFO:teuthology.orchestra.run.smithi161.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.50742.log.gz 2024-03-28T23:37:45.636 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.69327.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67026.log.gz 2024-03-28T23:37:45.636 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61647.log 2024-03-28T23:37:45.636 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48671.log 2024-03-28T23:37:45.636 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.67591.log: /var/log/ceph/ceph-client.admin.61647.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.69327.log.gz 2024-03-28T23:37:45.636 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50550.log 2024-03-28T23:37:45.636 INFO:teuthology.orchestra.run.smithi161.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67591.log.gz 2024-03-28T23:37:45.637 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48671.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61647.log.gz 2024-03-28T23:37:45.637 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52674.log 2024-03-28T23:37:45.637 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48671.log.gz 2024-03-28T23:37:45.637 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44965.log 2024-03-28T23:37:45.637 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50550.log: /var/log/ceph/ceph-client.admin.52674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62690.log 2024-03-28T23:37:45.637 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50550.log.gz 2024-03-28T23:37:45.637 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44965.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52674.log.gz 2024-03-28T23:37:45.637 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61353.log 2024-03-28T23:37:45.637 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44965.log.gz 2024-03-28T23:37:45.638 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.62690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69008.log 2024-03-28T23:37:45.638 INFO:teuthology.orchestra.run.smithi161.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62690.log.gz 2024-03-28T23:37:45.638 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.61353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46412.log 2024-03-28T23:37:45.638 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61353.log.gz 2024-03-28T23:37:45.638 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.69008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50021.log 2024-03-28T23:37:45.638 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69008.log.gz 2024-03-28T23:37:45.638 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72598.log 2024-03-28T23:37:45.638 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46412.log.gz 2024-03-28T23:37:45.638 INFO:teuthology.orchestra.run.smithi161.stderr:gzip/var/log/ceph/ceph-client.admin.50021.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42674.log 2024-03-28T23:37:45.639 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.72598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53244.log 2024-03-28T23:37:45.639 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50021.log.gz 2024-03-28T23:37:45.639 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.42674.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72598.log.gz 2024-03-28T23:37:45.639 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60579.log 2024-03-28T23:37:45.639 INFO:teuthology.orchestra.run.smithi161.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42674.log.gz 2024-03-28T23:37:45.639 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61041.log 2024-03-28T23:37:45.639 INFO:teuthology.orchestra.run.smithi161.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.53244.log.gz 2024-03-28T23:37:45.639 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61689.log 2024-03-28T23:37:45.639 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.61041.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60579.log.gz 2024-03-28T23:37:45.640 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71044.log 2024-03-28T23:37:45.640 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61041.log.gz 2024-03-28T23:37:45.640 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.61689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66034.log 2024-03-28T23:37:45.640 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61689.log.gz 2024-03-28T23:37:45.640 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.71044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41650.log 2024-03-28T23:37:45.640 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.66034.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.71044.log.gz 2024-03-28T23:37:45.640 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51474.log 2024-03-28T23:37:45.640 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.41650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55963.log 2024-03-28T23:37:45.640 INFO:teuthology.orchestra.run.smithi161.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.66034.log.gz 2024-03-28T23:37:45.641 INFO:teuthology.orchestra.run.smithi161.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41650.log.gz 2024-03-28T23:37:45.641 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45385.log 2024-03-28T23:37:45.641 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43828.log 2024-03-28T23:37:45.641 INFO:teuthology.orchestra.run.smithi161.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51474.log.gz 2024-03-28T23:37:45.641 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.45385.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55963.log.gz 2024-03-28T23:37:45.641 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47252.log 2024-03-28T23:37:45.641 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45385.log.gz 2024-03-28T23:37:45.641 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58253.log 2024-03-28T23:37:45.641 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43828.log.gz 2024-03-28T23:37:45.642 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53456.log 2024-03-28T23:37:45.642 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47252.log: /var/log/ceph/ceph-client.admin.58253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41778.log 2024-03-28T23:37:45.642 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47252.log.gz 2024-03-28T23:37:45.642 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53456.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58253.log.gz 2024-03-28T23:37:45.642 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50231.log 2024-03-28T23:37:45.642 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41694.log 2024-03-28T23:37:45.642 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.41778.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53456.log.gz 2024-03-28T23:37:45.642 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50231.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41778.log.gz 2024-03-28T23:37:45.643 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58013.log 2024-03-28T23:37:45.643 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50231.log.gz 2024-03-28T23:37:45.643 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46328.log 2024-03-28T23:37:45.643 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.41694.log: /var/log/ceph/ceph-client.admin.58013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64000.log 2024-03-28T23:37:45.643 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41694.log.gz 2024-03-28T23:37:45.643 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46328.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58013.log.gz 2024-03-28T23:37:45.643 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60765.log 2024-03-28T23:37:45.643 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46328.log.gz 2024-03-28T23:37:45.643 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.64000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75541.log 2024-03-28T23:37:45.644 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61395.log 2024-03-28T23:37:45.644 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7%/var/log/ceph/ceph-client.admin.60765.log: -- replaced with /var/log/ceph/ceph-client.admin.64000.log.gz 2024-03-28T23:37:45.644 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.75541.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60765.log.gz 2024-03-28T23:37:45.644 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52418.log 2024-03-28T23:37:45.644 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75541.log.gz 2024-03-28T23:37:45.644 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58685.log 2024-03-28T23:37:45.644 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.61395.log: /var/log/ceph/ceph-client.admin.52418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59307.log 2024-03-28T23:37:45.644 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61395.log.gz 2024-03-28T23:37:45.645 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52418.log.gz 2024-03-28T23:37:45.645 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53783.log 2024-03-28T23:37:45.645 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58685.log.gz 2024-03-28T23:37:45.645 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76021.log 2024-03-28T23:37:45.645 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48377.log 2024-03-28T23:37:45.645 INFO:teuthology.orchestra.run.smithi161.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59307.log.gz 2024-03-28T23:37:45.645 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53783.log: /var/log/ceph/ceph-client.admin.76021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42482.log 2024-03-28T23:37:45.646 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53783.log.gz 2024-03-28T23:37:45.646 INFO:teuthology.orchestra.run.smithi161.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.76021.log.gz 2024-03-28T23:37:45.646 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75201.log/var/log/ceph/ceph-client.admin.48377.log: 2024-03-28T23:37:45.646 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.42482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52153.log 2024-03-28T23:37:45.646 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48377.log.gz 2024-03-28T23:37:45.646 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.75201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60035.log 2024-03-28T23:37:45.646 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75201.log.gz 2024-03-28T23:37:45.647 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68689.log 2024-03-28T23:37:45.647 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60035.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52153.log.gz 2024-03-28T23:37:45.647 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41862.log 2024-03-28T23:37:45.647 INFO:teuthology.orchestra.run.smithi161.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.60035.log.gz 2024-03-28T23:37:45.647 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42758.log 2024-03-28T23:37:45.647 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.68689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49566.log 2024-03-28T23:37:45.647 INFO:teuthology.orchestra.run.smithi161.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.42482.log.gz 2024-03-28T23:37:45.648 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.41862.log: /var/log/ceph/ceph-client.admin.42758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54605.log 2024-03-28T23:37:45.648 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41862.log.gz 91.1% 2024-03-28T23:37:45.648 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68689.log.gz 2024-03-28T23:37:45.648 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49566.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42758.log.gz 2024-03-28T23:37:45.648 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70898.log 2024-03-28T23:37:45.648 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49566.log.gz 2024-03-28T23:37:45.648 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61083.log 2024-03-28T23:37:45.649 INFO:teuthology.orchestra.run.smithi161.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.54605.log.gz 2024-03-28T23:37:45.649 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.70898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51432.log 2024-03-28T23:37:45.649 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.61083.log: gzip -5 --verbose -- 88.5% /var/log/ceph/ceph-client.admin.75159.log 2024-03-28T23:37:45.649 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70898.log.gz 2024-03-28T23:37:45.649 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51432.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61083.log.gz 2024-03-28T23:37:45.649 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53993.log 2024-03-28T23:37:45.649 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.75159.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51432.log.gz 2024-03-28T23:37:45.650 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67381.log 2024-03-28T23:37:45.650 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52325.log 88.3% 2024-03-28T23:37:45.650 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.75159.log.gz 2024-03-28T23:37:45.650 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.67381.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53993.log.gz 2024-03-28T23:37:45.650 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46916.log 2024-03-28T23:37:45.650 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67381.log.gz 2024-03-28T23:37:45.650 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62518.log 2024-03-28T23:37:45.651 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52325.log: /var/log/ceph/ceph-client.admin.46916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75961.log 2024-03-28T23:37:45.651 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52325.log.gz 2024-03-28T23:37:45.651 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.62518.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46916.log.gz 2024-03-28T23:37:45.651 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55777.log 2024-03-28T23:37:45.651 INFO:teuthology.orchestra.run.smithi161.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62518.log.gz 2024-03-28T23:37:45.651 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68605.log 2024-03-28T23:37:45.651 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.75961.log: /var/log/ceph/ceph-client.admin.55777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73681.log 2024-03-28T23:37:45.652 INFO:teuthology.orchestra.run.smithi161.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.75961.log.gz 2024-03-28T23:37:45.652 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.68605.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55777.log.gz 2024-03-28T23:37:45.652 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71000.log 2024-03-28T23:37:45.652 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68605.log.gz 2024-03-28T23:37:45.652 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58811.log 2024-03-28T23:37:45.652 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.73681.log: /var/log/ceph/ceph-client.admin.71000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69243.log 2024-03-28T23:37:45.652 INFO:teuthology.orchestra.run.smithi161.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73681.log.gz 2024-03-28T23:37:45.652 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58811.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71000.log.gz 2024-03-28T23:37:45.653 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55441.log 2024-03-28T23:37:45.653 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.69243.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58811.log.gz 2024-03-28T23:37:45.653 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51660.log 2024-03-28T23:37:45.653 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55441.log: gzip -5 89.3% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.69243.log.gz /var/log/ceph/ceph-client.admin.54479.log 2024-03-28T23:37:45.653 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:37:45.653 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51660.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55441.log.gz 2024-03-28T23:37:45.653 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73201.log 2024-03-28T23:37:45.653 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51660.log.gz 2024-03-28T23:37:45.653 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46664.log 2024-03-28T23:37:45.654 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.73201.log: gzip -5 --verbose -- 89.5% /var/log/ceph/ceph-client.admin.61857.log 2024-03-28T23:37:45.654 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54479.log.gz 2024-03-28T23:37:45.654 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46664.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73201.log.gz 2024-03-28T23:37:45.654 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67876.log 2024-03-28T23:37:45.654 INFO:teuthology.orchestra.run.smithi161.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46664.log.gz 2024-03-28T23:37:45.654 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.61857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66184.log 2024-03-28T23:37:45.654 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.67876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64918.log 2024-03-28T23:37:45.654 INFO:teuthology.orchestra.run.smithi161.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61857.log.gz 2024-03-28T23:37:45.655 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.66184.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67876.log.gz 2024-03-28T23:37:45.655 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58769.log 2024-03-28T23:37:45.655 INFO:teuthology.orchestra.run.smithi161.stderr: 89.0%/var/log/ceph/ceph-client.admin.64918.log: -- replaced with /var/log/ceph/ceph-client.admin.66184.log.gz 2024-03-28T23:37:45.655 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75370.log 2024-03-28T23:37:45.655 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47000.log 2024-03-28T23:37:45.655 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64918.log.gz 2024-03-28T23:37:45.655 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.75370.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58769.log.gz 2024-03-28T23:37:45.655 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46958.log 2024-03-28T23:37:45.656 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.75370.log.gz 2024-03-28T23:37:45.656 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67338.log 2024-03-28T23:37:45.656 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41192.log 2024-03-28T23:37:45.656 INFO:teuthology.orchestra.run.smithi161.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47000.log.gz 2024-03-28T23:37:45.656 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.67338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56372.log 2024-03-28T23:37:45.656 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46958.log.gz 2024-03-28T23:37:45.656 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.41192.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67338.log.gz 2024-03-28T23:37:45.656 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66100.log 2024-03-28T23:37:45.657 INFO:teuthology.orchestra.run.smithi161.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41192.log.gz 2024-03-28T23:37:45.657 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48797.log 2024-03-28T23:37:45.657 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.66100.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56372.log.gz 2024-03-28T23:37:45.657 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55021.log 2024-03-28T23:37:45.657 INFO:teuthology.orchestra.run.smithi161.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.66100.log.gz 2024-03-28T23:37:45.657 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61437.log 2024-03-28T23:37:45.657 INFO:teuthology.orchestra.run.smithi161.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48797.log.gz 2024-03-28T23:37:45.657 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55063.log 2024-03-28T23:37:45.658 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.61437.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.55021.log.gz 2024-03-28T23:37:45.658 INFO:teuthology.orchestra.run.smithi161.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.61731.log 2024-03-28T23:37:45.658 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55063.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61437.log.gz 2024-03-28T23:37:45.658 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56288.log 2024-03-28T23:37:45.658 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55063.log.gz 2024-03-28T23:37:45.658 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.61731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41553.log 2024-03-28T23:37:45.658 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61731.log.gz 2024-03-28T23:37:45.658 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53951.log 2024-03-28T23:37:45.659 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.41553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48202.log 2024-03-28T23:37:45.659 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56288.log.gz 2024-03-28T23:37:45.659 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53951.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41553.log.gz 2024-03-28T23:37:45.659 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63543.log 2024-03-28T23:37:45.659 INFO:teuthology.orchestra.run.smithi161.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53951.log.gz 2024-03-28T23:37:45.659 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54857.log 2024-03-28T23:37:45.659 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.63543.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48202.log.gz 2024-03-28T23:37:45.660 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46184.log 2024-03-28T23:37:45.660 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61125.log 2024-03-28T23:37:45.660 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46184.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54857.log.gz 2024-03-28T23:37:45.660 INFO:teuthology.orchestra.run.smithi161.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.63543.log.gz 2024-03-28T23:37:45.660 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53412.log 2024-03-28T23:37:45.660 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46184.log.gz 2024-03-28T23:37:45.660 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.61125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46244.log 2024-03-28T23:37:45.660 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46286.log 2024-03-28T23:37:45.660 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46244.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53412.log.gz 2024-03-28T23:37:45.661 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65002.log 2024-03-28T23:37:45.661 INFO:teuthology.orchestra.run.smithi161.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.61125.log.gz 2024-03-28T23:37:45.661 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46244.log.gz 2024-03-28T23:37:45.661 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49979.log 2024-03-28T23:37:45.661 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46286.log.gz 2024-03-28T23:37:45.661 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.65002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62602.log 2024-03-28T23:37:45.661 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59830.log 2024-03-28T23:37:45.661 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65002.log.gz 2024-03-28T23:37:45.661 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.62602.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49979.log.gz 2024-03-28T23:37:45.662 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55189.log 2024-03-28T23:37:45.662 INFO:teuthology.orchestra.run.smithi161.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62602.log.gz 2024-03-28T23:37:45.662 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58055.log 2024-03-28T23:37:45.662 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51012.log 2024-03-28T23:37:45.662 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59830.log.gz 2024-03-28T23:37:45.662 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58055.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55189.log.gz 2024-03-28T23:37:45.662 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44166.log 2024-03-28T23:37:45.662 INFO:teuthology.orchestra.run.smithi161.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58055.log.gz 2024-03-28T23:37:45.662 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59578.log 2024-03-28T23:37:45.663 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47557.log 2024-03-28T23:37:45.663 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51012.log.gz 2024-03-28T23:37:45.663 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59578.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44166.log.gz 2024-03-28T23:37:45.663 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74598.log 2024-03-28T23:37:45.663 INFO:teuthology.orchestra.run.smithi161.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59578.log.gz 2024-03-28T23:37:45.663 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71700.log 2024-03-28T23:37:45.663 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.74598.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-28T23:37:45.663 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47557.log.gz 2024-03-28T23:37:45.663 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.71700.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.74598.log.gz 2024-03-28T23:37:45.664 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69435.log 2024-03-28T23:37:45.664 INFO:teuthology.orchestra.run.smithi161.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.71700.log.gz 2024-03-28T23:37:45.664 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48713.log 2024-03-28T23:37:45.664 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.69435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47126.log 2024-03-28T23:37:45.664 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70069.log 2024-03-28T23:37:45.664 INFO:teuthology.orchestra.run.smithi161.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.69435.log.gz 2024-03-28T23:37:45.664 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47741.log 2024-03-28T23:37:45.664 INFO:teuthology.orchestra.run.smithi161.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48713.log.gz 2024-03-28T23:37:45.664 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.70069.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47126.log.gz 2024-03-28T23:37:45.665 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64624.log 2024-03-28T23:37:45.665 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70321.log 2024-03-28T23:37:45.665 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47741.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.70069.log.gz 2024-03-28T23:37:45.665 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.64624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67236.log 2024-03-28T23:37:45.665 INFO:teuthology.orchestra.run.smithi161.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47741.log.gz 2024-03-28T23:37:45.665 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64624.log.gz 2024-03-28T23:37:45.665 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5/var/log/ceph/ceph-client.admin.70321.log: --verbose -- /var/log/ceph/ceph-client.admin.51306.log 2024-03-28T23:37:45.665 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.67236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61773.log 2024-03-28T23:37:45.665 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70321.log.gz 2024-03-28T23:37:45.666 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51306.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67236.log.gz 2024-03-28T23:37:45.666 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51558.log 2024-03-28T23:37:45.666 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.61773.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51306.log.gz 2024-03-28T23:37:45.666 INFO:teuthology.orchestra.run.smithi161.stderr: 89.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-28T23:37:45.666 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54137.log 2024-03-28T23:37:45.666 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51558.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61773.log.gz 2024-03-28T23:37:45.666 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74869.log 2024-03-28T23:37:45.666 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51558.log.gz 2024-03-28T23:37:45.667 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54773.log 2024-03-28T23:37:45.667 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.74869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72305.log 2024-03-28T23:37:45.667 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54137.log.gz 2024-03-28T23:37:45.667 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66858.log 2024-03-28T23:37:45.667 INFO:teuthology.orchestra.run.smithi161.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.74869.log.gz 2024-03-28T23:37:45.667 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54773.log.gz 2024-03-28T23:37:45.667 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.72305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73849.log 2024-03-28T23:37:45.668 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47042.log 2024-03-28T23:37:45.668 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.66858.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72305.log.gz 2024-03-28T23:37:45.668 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68311.log 2024-03-28T23:37:45.668 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.73849.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.66858.log.gz 2024-03-28T23:37:45.668 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57423.log 2024-03-28T23:37:45.668 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.73849.log.gz 2024-03-28T23:37:45.668 INFO:teuthology.orchestra.run.smithi161.stderr: 89.0%/var/log/ceph/ceph-client.admin.68311.log: -- replaced with /var/log/ceph/ceph-client.admin.47042.log.gz 2024-03-28T23:37:45.669 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54731.log 2024-03-28T23:37:45.669 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6%/var/log/ceph/ceph-client.admin.57423.log: -- replaced with /var/log/ceph/ceph-client.admin.68311.log.gz 2024-03-28T23:37:45.669 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69691.log 2024-03-28T23:37:45.669 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70940.log 2024-03-28T23:37:45.669 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57423.log.gz 2024-03-28T23:37:45.669 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.69691.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54731.log.gz 2024-03-28T23:37:45.669 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44174.log 2024-03-28T23:37:45.669 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69691.log.gz 2024-03-28T23:37:45.670 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.70940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74215.log 2024-03-28T23:37:45.670 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44174.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70940.log.gz 2024-03-28T23:37:45.670 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67194.log 2024-03-28T23:37:45.670 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.74215.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44174.log.gz 2024-03-28T23:37:45.670 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73243.log 2024-03-28T23:37:45.670 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74215.log.gz 2024-03-28T23:37:45.670 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.67194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52632.log 2024-03-28T23:37:45.671 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.73243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68042.log 2024-03-28T23:37:45.671 INFO:teuthology.orchestra.run.smithi161.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.67194.log.gz 2024-03-28T23:37:45.671 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45946.log 2024-03-28T23:37:45.671 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.68042.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52632.log.gz 2024-03-28T23:37:45.671 INFO:teuthology.orchestra.run.smithi161.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.73243.log.gz 2024-03-28T23:37:45.671 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73975.log 2024-03-28T23:37:45.671 INFO:teuthology.orchestra.run.smithi161.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.68042.log.gz 2024-03-28T23:37:45.672 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.0.45946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51390.log 2024-03-28T23:37:45.672 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.73975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68214.log 2024-03-28T23:37:45.672 INFO:teuthology.orchestra.run.smithi161.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.0.45946.log.gz 2024-03-28T23:37:45.672 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51390.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.73975.log.gz 2024-03-28T23:37:45.672 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69571.log 2024-03-28T23:37:45.672 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51390.log.gz 2024-03-28T23:37:45.672 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.68214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51786.log 2024-03-28T23:37:45.672 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.69571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64162.log 2024-03-28T23:37:45.672 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68214.log.gz 2024-03-28T23:37:45.673 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68563.log 2024-03-28T23:37:45.673 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.64162.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.69571.log.gz 2024-03-28T23:37:45.673 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69487.log 88.7% 2024-03-28T23:37:45.673 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51786.log.gz 2024-03-28T23:37:45.673 INFO:teuthology.orchestra.run.smithi161.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.64162.log.gz 2024-03-28T23:37:45.673 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.68563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62025.log 2024-03-28T23:37:45.673 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.68563.log.gz 2024-03-28T23:37:45.673 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.69487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54521.log 2024-03-28T23:37:45.674 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.62025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44839.log 2024-03-28T23:37:45.674 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69487.log.gz 2024-03-28T23:37:45.674 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54521.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62025.log.gz 2024-03-28T23:37:45.674 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50465.log 2024-03-28T23:37:45.674 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44839.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54521.log.gz 2024-03-28T23:37:45.674 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42054.log 2024-03-28T23:37:45.674 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69285.log 2024-03-28T23:37:45.674 INFO:teuthology.orchestra.run.smithi161.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44839.log.gz 2024-03-28T23:37:45.675 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.42054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41820.log 2024-03-28T23:37:45.675 INFO:teuthology.orchestra.run.smithi161.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.50465.log.gz 2024-03-28T23:37:45.675 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.69285.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42054.log.gz 2024-03-28T23:37:45.675 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52590.log 2024-03-28T23:37:45.675 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52842.log 2024-03-28T23:37:45.675 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.41820.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69285.log.gz 2024-03-28T23:37:45.675 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71508.log 2024-03-28T23:37:45.675 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52590.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41820.log.gz 2024-03-28T23:37:45.676 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47294.log 2024-03-28T23:37:45.676 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52590.log.gz 2024-03-28T23:37:45.676 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52842.log.gz 2024-03-28T23:37:45.676 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.71508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48923.log 2024-03-28T23:37:45.676 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71508.log.gz 2024-03-28T23:37:45.676 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73477.log 2024-03-28T23:37:45.676 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51828.log 2024-03-28T23:37:45.676 INFO:teuthology.orchestra.run.smithi161.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47294.log.gz 2024-03-28T23:37:45.676 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.73477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61815.log 2024-03-28T23:37:45.677 INFO:teuthology.orchestra.run.smithi161.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48923.log.gz 2024-03-28T23:37:45.677 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56725.log 2024-03-28T23:37:45.677 INFO:teuthology.orchestra.run.smithi161.stderr: 91.2%/var/log/ceph/ceph-client.admin.61815.log: -- replaced with /var/log/ceph/ceph-client.admin.73477.log.gz 2024-03-28T23:37:45.677 INFO:teuthology.orchestra.run.smithi161.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51828.log.gz 2024-03-28T23:37:45.677 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73891.log 2024-03-28T23:37:45.677 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61815.log.gz 2024-03-28T23:37:45.677 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57467.log 2024-03-28T23:37:45.677 INFO:teuthology.orchestra.run.smithi161.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56725.log.gz 2024-03-28T23:37:45.677 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.73891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58349.log 2024-03-28T23:37:45.678 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57467.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.73891.log.gz 2024-03-28T23:37:45.678 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56553.log 2024-03-28T23:37:45.678 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71256.log 2024-03-28T23:37:45.678 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57467.log.gz 2024-03-28T23:37:45.678 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58349.log.gz 2024-03-28T23:37:45.678 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58517.log 2024-03-28T23:37:45.678 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.71256.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.56553.log.gz 2024-03-28T23:37:45.678 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43618.log 2024-03-28T23:37:45.679 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58517.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71256.log.gz 2024-03-28T23:37:45.679 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57593.log 2024-03-28T23:37:45.679 INFO:teuthology.orchestra.run.smithi161.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58517.log.gz 2024-03-28T23:37:45.679 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62476.log 2024-03-28T23:37:45.679 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70531.log 2024-03-28T23:37:45.679 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43618.log.gz 2024-03-28T23:37:45.679 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.62476.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57593.log.gz 2024-03-28T23:37:45.680 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51096.log 2024-03-28T23:37:45.680 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62476.log.gz 2024-03-28T23:37:45.680 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.70531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55525.log 2024-03-28T23:37:45.680 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50658.log 2024-03-28T23:37:45.680 INFO:teuthology.orchestra.run.smithi161.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70531.log.gz 2024-03-28T23:37:45.680 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55525.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51096.log.gz 2024-03-28T23:37:45.680 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50592.log 2024-03-28T23:37:45.681 INFO:teuthology.orchestra.run.smithi161.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55525.log.gz 2024-03-28T23:37:45.681 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69775.log 2024-03-28T23:37:45.681 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71826.log 2024-03-28T23:37:45.681 INFO:teuthology.orchestra.run.smithi161.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50658.log.gz 2024-03-28T23:37:45.681 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.69775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71742.log 2024-03-28T23:37:45.681 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.71826.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.69775.log.gz 2024-03-28T23:37:45.681 INFO:teuthology.orchestra.run.smithi161.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.50592.log.gz 2024-03-28T23:37:45.681 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50189.log 2024-03-28T23:37:45.681 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4%/var/log/ceph/ceph-client.admin.71742.log: -- replaced with /var/log/ceph/ceph-client.admin.71826.log.gz 2024-03-28T23:37:45.682 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74995.log 2024-03-28T23:37:45.682 INFO:teuthology.orchestra.run.smithi161.stderr: 87.5%/var/log/ceph/ceph-client.admin.50189.log: -- replaced with /var/log/ceph/ceph-client.admin.71742.log.gz 2024-03-28T23:37:45.682 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65559.log 2024-03-28T23:37:45.682 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.74995.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50189.log.gz 2024-03-28T23:37:45.682 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58853.log 2024-03-28T23:37:45.682 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.65559.log: gzip 89.1% -- replaced with /var/log/ceph/ceph-client.admin.74995.log.gz -5 2024-03-28T23:37:45.682 INFO:teuthology.orchestra.run.smithi161.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52376.log 2024-03-28T23:37:45.682 INFO:teuthology.orchestra.run.smithi161.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.65559.log.gz 2024-03-28T23:37:45.682 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50063.log 2024-03-28T23:37:45.683 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58853.log.gz 2024-03-28T23:37:45.683 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53160.log 2024-03-28T23:37:45.683 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50063.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.52376.log.gz -5 2024-03-28T23:37:45.683 INFO:teuthology.orchestra.run.smithi161.stderr: --verbose -- /var/log/ceph/ceph-client.admin.69859.log 2024-03-28T23:37:45.683 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50063.log.gz 2024-03-28T23:37:45.683 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62783.log 2024-03-28T23:37:45.683 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.69859.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.53160.log.gz 2024-03-28T23:37:45.683 INFO:teuthology.orchestra.run.smithi161.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.64582.log 2024-03-28T23:37:45.683 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2%/var/log/ceph/ceph-client.admin.62783.log: -- replaced with /var/log/ceph/ceph-client.admin.69859.log.gz 2024-03-28T23:37:45.684 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62193.log 2024-03-28T23:37:45.684 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.64582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42842.log 88.3% 2024-03-28T23:37:45.684 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62783.log.gz 2024-03-28T23:37:45.684 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.62193.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64582.log.gz 2024-03-28T23:37:45.684 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48461.log 2024-03-28T23:37:45.684 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62193.log.gz 2024-03-28T23:37:45.684 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.42842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52281.log 2024-03-28T23:37:45.684 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57211.log 2024-03-28T23:37:45.685 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42842.log.gz 2024-03-28T23:37:45.685 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52281.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48461.log.gz 2024-03-28T23:37:45.685 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65345.log 2024-03-28T23:37:45.685 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52281.log.gz 2024-03-28T23:37:45.685 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43842.log 2024-03-28T23:37:45.685 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.65345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63628.log 2024-03-28T23:37:45.685 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57211.log.gz 2024-03-28T23:37:45.685 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43842.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65345.log.gz 2024-03-28T23:37:45.686 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48071.log 2024-03-28T23:37:45.686 INFO:teuthology.orchestra.run.smithi161.stderr: 87.9%/var/log/ceph/ceph-client.admin.63628.log: -- replaced with /var/log/ceph/ceph-client.admin.43842.log.gz 2024-03-28T23:37:45.686 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48335.log 2024-03-28T23:37:45.686 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63628.log.gz 2024-03-28T23:37:45.686 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51180.log 2024-03-28T23:37:45.686 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48335.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48071.log.gz 2024-03-28T23:37:45.686 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44537.log 2024-03-28T23:37:45.686 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48335.log.gz 2024-03-28T23:37:45.687 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75415.log 2024-03-28T23:37:45.687 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46748.log 2024-03-28T23:37:45.687 INFO:teuthology.orchestra.run.smithi161.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51180.log.gz 2024-03-28T23:37:45.687 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.75415.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44537.log.gz 2024-03-28T23:37:45.687 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69177.log 2024-03-28T23:37:45.687 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75415.log.gz 2024-03-28T23:37:45.687 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47945.log 2024-03-28T23:37:45.687 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.69177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71298.log 2024-03-28T23:37:45.687 INFO:teuthology.orchestra.run.smithi161.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46748.log.gz 2024-03-28T23:37:45.688 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47699.log 2024-03-28T23:37:45.688 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.71298.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47945.log.gz 2024-03-28T23:37:45.688 INFO:teuthology.orchestra.run.smithi161.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.69177.log.gz 2024-03-28T23:37:45.688 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46068.log 2024-03-28T23:37:45.688 INFO:teuthology.orchestra.run.smithi161.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.71298.log.gz 2024-03-28T23:37:45.688 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44177.log 2024-03-28T23:37:45.688 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46068.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47699.log.gz 2024-03-28T23:37:45.688 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66632.log 2024-03-28T23:37:45.688 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44177.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46068.log.gz 2024-03-28T23:37:45.689 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47473.log 2024-03-28T23:37:45.689 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.66632.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44177.log.gz 2024-03-28T23:37:45.689 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45007.log 2024-03-28T23:37:45.689 INFO:teuthology.orchestra.run.smithi161.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.66632.log.gz 2024-03-28T23:37:45.689 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44881.log 2024-03-28T23:37:45.689 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.45007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63915.log 2024-03-28T23:37:45.689 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47473.log.gz 2024-03-28T23:37:45.689 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44881.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.45007.log.gz -5 2024-03-28T23:37:45.689 INFO:teuthology.orchestra.run.smithi161.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49937.log 2024-03-28T23:37:45.690 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.63915.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44881.log.gz 2024-03-28T23:37:45.690 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53076.log 2024-03-28T23:37:45.690 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51222.log 2024-03-28T23:37:45.690 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53076.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.63915.log.gz 2024-03-28T23:37:45.690 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49937.log.gz 2024-03-28T23:37:45.690 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45217.log 2024-03-28T23:37:45.690 INFO:teuthology.orchestra.run.smithi161.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53076.log.gz 2024-03-28T23:37:45.690 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62067.log 2024-03-28T23:37:45.691 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51222.log.gz 2024-03-28T23:37:45.691 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.45217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60891.log 2024-03-28T23:37:45.691 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.62067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66142.log 88.3% 2024-03-28T23:37:45.691 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45217.log.gz 2024-03-28T23:37:45.691 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60891.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.62067.log.gz 2024-03-28T23:37:45.691 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46580.log 2024-03-28T23:37:45.691 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60891.log.gz 2024-03-28T23:37:45.691 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.66142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53741.log 2024-03-28T23:37:45.692 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4%/var/log/ceph/ceph-client.admin.46580.log: -- replaced with /var/log/ceph/ceph-client.admin.66142.log.gz 2024-03-28T23:37:45.692 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62646.log 2024-03-28T23:37:45.692 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53741.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46580.log.gz 2024-03-28T23:37:45.692 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63873.log 2024-03-28T23:37:45.692 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53741.log.gz 2024-03-28T23:37:45.692 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.62646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50105.log 2024-03-28T23:37:45.692 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.63873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72985.log 2024-03-28T23:37:45.692 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62646.log.gz 2024-03-28T23:37:45.692 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50105.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.63873.log.gz 2024-03-28T23:37:45.693 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63165.log 2024-03-28T23:37:45.693 INFO:teuthology.orchestra.run.smithi161.stderr: 87.9%/var/log/ceph/ceph-client.admin.72985.log: -- replaced with /var/log/ceph/ceph-client.admin.50105.log.gz 2024-03-28T23:37:45.693 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46454.log 2024-03-28T23:37:45.693 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.63165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50970.log 2024-03-28T23:37:45.693 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72985.log.gz 2024-03-28T23:37:45.693 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46454.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63165.log.gz 2024-03-28T23:37:45.693 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75625.log 2024-03-28T23:37:45.693 INFO:teuthology.orchestra.run.smithi161.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46454.log.gz 2024-03-28T23:37:45.693 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72389.log 2024-03-28T23:37:45.694 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.75625.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50970.log.gz 2024-03-28T23:37:45.694 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75117.log 2024-03-28T23:37:45.694 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.72389.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75625.log.gz 2024-03-28T23:37:45.694 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-28T23:37:45.694 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72389.log.gz 2024-03-28T23:37:45.694 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.75117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51907.log 2024-03-28T23:37:45.694 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66900.log 2024-03-28T23:37:45.694 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75117.log.gz 2024-03-28T23:37:45.695 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53635.log 2024-03-28T23:37:45.695 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.66900.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51907.log.gz 2024-03-28T23:37:45.695 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67278.log 2024-03-28T23:37:45.695 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53635.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66900.log.gz 2024-03-28T23:37:45.695 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47599.log 2024-03-28T23:37:45.695 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53635.log.gz 2024-03-28T23:37:45.695 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.67278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65694.log 2024-03-28T23:37:45.696 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67278.log.gz 2024-03-28T23:37:45.696 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48250.log 2024-03-28T23:37:45.696 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47599.log.gz 2024-03-28T23:37:45.696 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.65694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45175.log 2024-03-28T23:37:45.696 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75835.log 2024-03-28T23:37:45.696 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7%/var/log/ceph/ceph-client.admin.48250.log: -- replaced with /var/log/ceph/ceph-client.admin.65694.log.gz 2024-03-28T23:37:45.696 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.45175.log: gzip 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48250.log.gz 2024-03-28T23:37:45.696 INFO:teuthology.orchestra.run.smithi161.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49355.log 2024-03-28T23:37:45.696 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.75835.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45175.log.gz 2024-03-28T23:37:45.697 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43231.log 2024-03-28T23:37:45.697 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.75835.log.gz 2024-03-28T23:37:45.697 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50273.log 2024-03-28T23:37:45.697 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53034.log 2024-03-28T23:37:45.697 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43231.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49355.log.gz 2024-03-28T23:37:45.697 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50273.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.43231.log.gz -5 2024-03-28T23:37:45.697 INFO:teuthology.orchestra.run.smithi161.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41297.log 2024-03-28T23:37:45.697 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73597.log 2024-03-28T23:37:45.698 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.41297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53118.log 2024-03-28T23:37:45.698 INFO:teuthology.orchestra.run.smithi161.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.50273.log.gz 2024-03-28T23:37:45.698 INFO:teuthology.orchestra.run.smithi161.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53034.log.gz 2024-03-28T23:37:45.698 INFO:teuthology.orchestra.run.smithi161.stderr:gzip/var/log/ceph/ceph-client.admin.73597.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53593.log 2024-03-28T23:37:45.698 INFO:teuthology.orchestra.run.smithi161.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41297.log.gz 2024-03-28T23:37:45.698 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53118.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73597.log.gz 2024-03-28T23:37:45.698 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70652.log 2024-03-28T23:37:45.699 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53118.log.gz 2024-03-28T23:37:45.699 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49068.log 2024-03-28T23:37:45.699 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53593.log: /var/log/ceph/ceph-client.admin.70652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68839.log 2024-03-28T23:37:45.699 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53593.log.gz 2024-03-28T23:37:45.699 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49068.log: gzip -5 --verbose 89.0% -- /var/log/ceph/ceph-client.admin.62352.log -- replaced with /var/log/ceph/ceph-client.admin.70652.log.gz 2024-03-28T23:37:45.699 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:37:45.699 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.68839.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49068.log.gz 2024-03-28T23:37:45.699 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69817.log 2024-03-28T23:37:45.700 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68839.log.gz 2024-03-28T23:37:45.700 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65992.log 2024-03-28T23:37:45.700 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.62352.log: /var/log/ceph/ceph-client.admin.69817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59223.log 2024-03-28T23:37:45.700 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62352.log.gz 2024-03-28T23:37:45.700 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.65992.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.69817.log.gz 2024-03-28T23:37:45.700 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53202.log 2024-03-28T23:37:45.700 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59223.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65992.log.gz 2024-03-28T23:37:45.701 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54815.log 2024-03-28T23:37:45.701 INFO:teuthology.orchestra.run.smithi161.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59223.log.gz 2024-03-28T23:37:45.701 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52884.log 2024-03-28T23:37:45.701 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60115.log 2024-03-28T23:37:45.701 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53202.log.gz 2024-03-28T23:37:45.701 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52884.log: gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54815.log.gz 2024-03-28T23:37:45.701 INFO:teuthology.orchestra.run.smithi161.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.64960.log 2024-03-28T23:37:45.701 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60115.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52884.log.gz 2024-03-28T23:37:45.701 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43845.log 2024-03-28T23:37:45.702 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60115.log.gz 2024-03-28T23:37:45.702 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64372.log 2024-03-28T23:37:45.702 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.64960.log: /var/log/ceph/ceph-client.admin.43845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53551.log 2024-03-28T23:37:45.702 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64960.log.gz 2024-03-28T23:37:45.702 INFO:teuthology.orchestra.run.smithi161.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43845.log.gz 2024-03-28T23:37:45.702 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.64372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57169.log 2024-03-28T23:37:45.702 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64372.log.gz 2024-03-28T23:37:45.702 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55231.log 2024-03-28T23:37:45.703 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53551.log: /var/log/ceph/ceph-client.admin.57169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45259.log 2024-03-28T23:37:45.703 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53551.log.gz 2024-03-28T23:37:45.703 INFO:teuthology.orchestra.run.smithi161.stderr:gzip 89.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.57169.log.gz/var/log/ceph/ceph-client.admin.55231.log: --verbose 2024-03-28T23:37:45.703 INFO:teuthology.orchestra.run.smithi161.stderr: -- /var/log/ceph/ceph-client.admin.60849.log 2024-03-28T23:37:45.703 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.45259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60281.log 2024-03-28T23:37:45.703 INFO:teuthology.orchestra.run.smithi161.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.55231.log.gz 2024-03-28T23:37:45.703 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60849.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45259.log.gz 2024-03-28T23:37:45.703 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60323.log 2024-03-28T23:37:45.704 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53867.log 2024-03-28T23:37:45.704 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60849.log.gz 2024-03-28T23:37:45.704 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60281.log: /var/log/ceph/ceph-client.admin.60323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55483.log 2024-03-28T23:37:45.704 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60281.log.gz 2024-03-28T23:37:45.704 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53867.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60323.log.gz 2024-03-28T23:37:45.704 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43273.log 2024-03-28T23:37:45.704 INFO:teuthology.orchestra.run.smithi161.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53867.log.gz 2024-03-28T23:37:45.704 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50700.log 2024-03-28T23:37:45.705 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61563.log 2024-03-28T23:37:45.705 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43273.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55483.log.gz 2024-03-28T23:37:45.705 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50700.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.43273.log.gz -5 2024-03-28T23:37:45.705 INFO:teuthology.orchestra.run.smithi161.stderr: --verbose -- /var/log/ceph/ceph-client.admin.68797.log 2024-03-28T23:37:45.705 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.61563.log: gzip -5 --verbose -- 87.7% /var/log/ceph/ceph-client.admin.63789.log -- replaced with /var/log/ceph/ceph-client.admin.50700.log.gz 2024-03-28T23:37:45.705 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:37:45.705 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.68797.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61563.log.gz 2024-03-28T23:37:45.705 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43068.log 2024-03-28T23:37:45.705 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.63789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42984.log 2024-03-28T23:37:45.706 INFO:teuthology.orchestra.run.smithi161.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.68797.log.gz 2024-03-28T23:37:45.706 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43068.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63789.log.gz 2024-03-28T23:37:45.706 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56330.log 2024-03-28T23:37:45.706 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5/var/log/ceph/ceph-client.admin.42984.log: --verbose -- /var/log/ceph/ceph-client.admin.71214.log 2024-03-28T23:37:45.706 INFO:teuthology.orchestra.run.smithi161.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43068.log.gz 2024-03-28T23:37:45.706 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63501.log 2024-03-28T23:37:45.706 INFO:teuthology.orchestra.run.smithi161.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42984.log.gz 2024-03-28T23:37:45.706 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.71214.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56330.log.gz 2024-03-28T23:37:45.707 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44397.log 2024-03-28T23:37:45.707 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.63501.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.71214.log.gz 2024-03-28T23:37:45.707 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48113.log 2024-03-28T23:37:45.707 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63501.log.gz 2024-03-28T23:37:45.707 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52548.log 2024-03-28T23:37:45.707 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44397.log: /var/log/ceph/ceph-client.admin.48113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44488.log 2024-03-28T23:37:45.707 INFO:teuthology.orchestra.run.smithi161.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44397.log.gz 2024-03-28T23:37:45.707 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52548.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48113.log.gz 2024-03-28T23:37:45.708 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72751.log 2024-03-28T23:37:45.708 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52548.log.gz 2024-03-28T23:37:45.708 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67423.log 2024-03-28T23:37:45.708 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44488.log.gz 2024-03-28T23:37:45.708 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51054.log 2024-03-28T23:37:45.708 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.72751.log: /var/log/ceph/ceph-client.admin.67423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57379.log 2024-03-28T23:37:45.708 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.72751.log.gz 2024-03-28T23:37:45.708 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51054.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67423.log.gz 2024-03-28T23:37:45.709 INFO:teuthology.orchestra.run.smithi161.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60933.log 2024-03-28T23:37:45.709 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57379.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51054.log.gz 2024-03-28T23:37:45.709 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65429.log 2024-03-28T23:37:45.709 INFO:teuthology.orchestra.run.smithi161.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57379.log.gz 2024-03-28T23:37:45.709 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68265.log 2024-03-28T23:37:45.709 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67068.log 2024-03-28T23:37:45.709 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.65429.log: /var/log/ceph/ceph-client.admin.68265.log: gzip -5 91.2% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.60933.log.gz 2024-03-28T23:37:45.709 INFO:teuthology.orchestra.run.smithi161.stderr: /var/log/ceph/ceph-client.admin.47210.log 2024-03-28T23:37:45.710 INFO:teuthology.orchestra.run.smithi161.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.65429.log.gz 2024-03-28T23:37:45.710 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68265.log.gz 2024-03-28T23:37:45.710 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68923.log 2024-03-28T23:37:45.710 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.67068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71784.log 2024-03-28T23:37:45.710 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47210.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67068.log.gz 2024-03-28T23:37:45.710 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.68923.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47210.log.gz 2024-03-28T23:37:45.710 INFO:teuthology.orchestra.run.smithi161.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.57845.log 2024-03-28T23:37:45.710 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.71784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62741.log 2024-03-28T23:37:45.711 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57845.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72666.log 2024-03-28T23:37:45.711 INFO:teuthology.orchestra.run.smithi161.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.68923.log.gz 2024-03-28T23:37:45.711 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71784.log.gz 2024-03-28T23:37:45.711 INFO:teuthology.orchestra.run.smithi161.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57845.log.gz 2024-03-28T23:37:45.711 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.62741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41381.log 2024-03-28T23:37:45.711 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.72666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67675.log 2024-03-28T23:37:45.711 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62741.log.gz 2024-03-28T23:37:45.711 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.41381.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.72666.log.gz 2024-03-28T23:37:45.711 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64498.log 2024-03-28T23:37:45.712 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74017.log 2024-03-28T23:37:45.712 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.67675.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41381.log.gz 2024-03-28T23:37:45.712 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.64498.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67675.log.gz 2024-03-28T23:37:45.712 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54689.log 2024-03-28T23:37:45.712 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57719.log 2024-03-28T23:37:45.712 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.74017.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64498.log.gz 2024-03-28T23:37:45.712 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54689.log: gzip -5 --verbose -- 87.6% /var/log/ceph/ceph-client.admin.43525.log 2024-03-28T23:37:45.712 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.74017.log.gz 2024-03-28T23:37:45.713 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54689.log.gz 2024-03-28T23:37:45.713 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55735.log 2024-03-28T23:37:45.713 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57719.log: /var/log/ceph/ceph-client.admin.43525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52800.log 2024-03-28T23:37:45.713 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57719.log.gz 2024-03-28T23:37:45.713 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55735.log: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.43525.log.gz 2024-03-28T23:37:45.713 INFO:teuthology.orchestra.run.smithi161.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44797.log 2024-03-28T23:37:45.713 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55735.log.gz 2024-03-28T23:37:45.713 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61233.log 2024-03-28T23:37:45.714 INFO:teuthology.orchestra.run.smithi161.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52800.log.gz 2024-03-28T23:37:45.714 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65515.log 2024-03-28T23:37:45.714 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44797.log: /var/log/ceph/ceph-client.admin.61233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64042.log 2024-03-28T23:37:45.714 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44797.log.gz 2024-03-28T23:37:45.714 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.65515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56122.log 2024-03-28T23:37:45.714 INFO:teuthology.orchestra.run.smithi161.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65515.log.gz 2024-03-28T23:37:45.714 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.64042.log: gzip -5 91.2% -- replaced with /var/log/ceph/ceph-client.admin.61233.log.gz 2024-03-28T23:37:45.714 INFO:teuthology.orchestra.run.smithi161.stderr: --verbose -- /var/log/ceph/ceph-client.admin.60197.log 2024-03-28T23:37:45.714 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74911.log 2024-03-28T23:37:45.715 INFO:teuthology.orchestra.run.smithi161.stderr: 91.1%/var/log/ceph/ceph-client.admin.60197.log: -- replaced with /var/log/ceph/ceph-client.admin.64042.log.gz 89.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.56122.log.gz -5 2024-03-28T23:37:45.715 INFO:teuthology.orchestra.run.smithi161.stderr: --verbose -- /var/log/ceph/ceph-client.admin.63670.log 2024-03-28T23:37:45.715 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:37:45.715 INFO:teuthology.orchestra.run.smithi161.stderr:gzip/var/log/ceph/ceph-client.admin.74911.log: -5 --verbose 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60197.log.gz 2024-03-28T23:37:45.715 INFO:teuthology.orchestra.run.smithi161.stderr: -- /var/log/ceph/ceph-client.admin.59349.log 2024-03-28T23:37:45.715 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.63670.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74911.log.gz 2024-03-28T23:37:45.715 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72137.log 2024-03-28T23:37:45.716 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57127.log 2024-03-28T23:37:45.716 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59349.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.63670.log.gz 2024-03-28T23:37:45.716 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.72137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60621.log 2024-03-28T23:37:45.716 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59349.log.gz 2024-03-28T23:37:45.716 INFO:teuthology.orchestra.run.smithi161.stderr:gzip/var/log/ceph/ceph-client.admin.57127.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.72137.log.gz 2024-03-28T23:37:45.716 INFO:teuthology.orchestra.run.smithi161.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60681.log 2024-03-28T23:37:45.716 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60621.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57127.log.gz 2024-03-28T23:37:45.716 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46538.log 2024-03-28T23:37:45.717 INFO:teuthology.orchestra.run.smithi161.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60621.log.gz 2024-03-28T23:37:45.717 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74827.log 2024-03-28T23:37:45.717 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60681.log.gz 2024-03-28T23:37:45.717 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58559.log 2024-03-28T23:37:45.717 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.74827.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46538.log.gz 2024-03-28T23:37:45.717 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55357.log 2024-03-28T23:37:45.717 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74827.log.gz 2024-03-28T23:37:45.717 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48156.log 2024-03-28T23:37:45.718 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58559.log: /var/log/ceph/ceph-client.admin.55357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70111.log 2024-03-28T23:37:45.718 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58559.log.gz 2024-03-28T23:37:45.718 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48156.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55357.log.gz 2024-03-28T23:37:45.718 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56511.log 2024-03-28T23:37:45.718 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.70111.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48156.log.gz 2024-03-28T23:37:45.718 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-28T23:37:45.718 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70111.log.gz 2024-03-28T23:37:45.718 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75709.log 2024-03-28T23:37:45.719 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56511.log.gz 2024-03-28T23:37:45.719 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72793.log 2024-03-28T23:37:45.719 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.75709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54563.log 2024-03-28T23:37:45.719 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.72793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54978.log 2024-03-28T23:37:45.719 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75709.log.gz 2024-03-28T23:37:45.719 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52111.log 2024-03-28T23:37:45.719 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62913.log 2024-03-28T23:37:45.719 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% 91.1% -- replaced with /var/log/ceph/ceph-client.admin.72793.log.gz 2024-03-28T23:37:45.720 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54563.log.gz 2024-03-28T23:37:45.720 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52111.log: gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54978.log.gz -5 2024-03-28T23:37:45.720 INFO:teuthology.orchestra.run.smithi161.stderr: --verbose -- /var/log/ceph/ceph-client.admin.75583.log 2024-03-28T23:37:45.720 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.62913.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52111.log.gz 2024-03-28T23:37:45.720 INFO:teuthology.orchestra.run.smithi161.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.72049.log 2024-03-28T23:37:45.720 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.75583.log: gzip -5 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62913.log.gz 2024-03-28T23:37:45.720 INFO:teuthology.orchestra.run.smithi161.stderr: --verbose -- /var/log/ceph/ceph-client.admin.37063.log 2024-03-28T23:37:45.720 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.72049.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.75583.log.gz 2024-03-28T23:37:45.721 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59051.log 2024-03-28T23:37:45.721 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72049.log.gz 2024-03-28T23:37:45.721 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.37063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63831.log 2024-03-28T23:37:45.721 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59051.log: gzip -5 --verbose -- 89.1% /var/log/ceph/ceph-client.admin.72431.log 2024-03-28T23:37:45.721 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37063.log.gz 2024-03-28T23:37:45.721 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7%/var/log/ceph/ceph-client.admin.63831.log: -- replaced with /var/log/ceph/ceph-client.admin.59051.log.gz 2024-03-28T23:37:45.721 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70447.log 2024-03-28T23:37:45.721 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63831.log.gz 2024-03-28T23:37:45.722 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.72431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57635.log 2024-03-28T23:37:45.722 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72431.log.gz 2024-03-28T23:37:45.722 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.70447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64666.log 2024-03-28T23:37:45.722 INFO:teuthology.orchestra.run.smithi161.stderr:gzip/var/log/ceph/ceph-client.admin.57635.log: -5 87.8% -- replaced with /var/log/ceph/ceph-client.admin.70447.log.gz 2024-03-28T23:37:45.722 INFO:teuthology.orchestra.run.smithi161.stderr: --verbose -- /var/log/ceph/ceph-client.admin.55693.log 2024-03-28T23:37:45.722 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.64666.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57635.log.gz 2024-03-28T23:37:45.722 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64666.log.gz 2024-03-28T23:37:45.722 INFO:teuthology.orchestra.run.smithi161.stderr: /var/log/ceph/ceph-client.admin.75793.log 2024-03-28T23:37:45.723 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50423.log 2024-03-28T23:37:45.723 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.75793.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55693.log.gz 2024-03-28T23:37:45.723 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54199.log 2024-03-28T23:37:45.723 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50423.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75793.log.gz 2024-03-28T23:37:45.723 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74342.log 2024-03-28T23:37:45.723 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50423.log.gz 2024-03-28T23:37:45.723 INFO:teuthology.orchestra.run.smithi161.stderr:gzip/var/log/ceph/ceph-client.admin.54199.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.57551.log 2024-03-28T23:37:45.723 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.74342.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54199.log.gz 2024-03-28T23:37:45.723 INFO:teuthology.orchestra.run.smithi161.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50808.log 2024-03-28T23:37:45.724 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74342.log.gz 2024-03-28T23:37:45.724 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55921.log 2024-03-28T23:37:45.724 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57551.log.gz 2024-03-28T23:37:45.724 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70489.log 2024-03-28T23:37:45.724 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55921.log: gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50808.log.gz 2024-03-28T23:37:45.724 INFO:teuthology.orchestra.run.smithi161.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.57295.log 2024-03-28T23:37:45.724 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55921.log.gz 2024-03-28T23:37:45.724 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.70489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56641.log 2024-03-28T23:37:45.725 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57295.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.70489.log.gz 2024-03-28T23:37:45.725 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47168.log 2024-03-28T23:37:45.725 INFO:teuthology.orchestra.run.smithi161.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57295.log.gz 2024-03-28T23:37:45.725 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72179.log 2024-03-28T23:37:45.725 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47168.log: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74059.log 2024-03-28T23:37:45.725 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56641.log.gz 2024-03-28T23:37:45.725 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.72179.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47168.log.gz 2024-03-28T23:37:45.725 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60495.log 2024-03-28T23:37:45.726 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73765.log 2024-03-28T23:37:45.726 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.72179.log.gz/var/log/ceph/ceph-client.admin.74059.log: 2024-03-28T23:37:45.726 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63225.log 2024-03-28T23:37:45.726 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74059.log.gz 2024-03-28T23:37:45.726 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.60495.log.gz 2024-03-28T23:37:45.726 INFO:teuthology.orchestra.run.smithi161.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.63459.log 2024-03-28T23:37:45.726 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.73765.log: /var/log/ceph/ceph-client.admin.63225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67152.log 2024-03-28T23:37:45.726 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73765.log.gz 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63225.log.gz 2024-03-28T23:37:45.727 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.63459.log: 2024-03-28T23:37:45.727 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66738.log 2024-03-28T23:37:45.727 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63459.log.gz 2024-03-28T23:37:45.727 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.67152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59433.log 2024-03-28T23:37:45.727 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67152.log.gz 2024-03-28T23:37:45.727 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.66738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41604.log 2024-03-28T23:37:45.727 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71616.log 2024-03-28T23:37:45.727 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4%/var/log/ceph/ceph-client.admin.41604.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.66738.log.gz 2024-03-28T23:37:45.728 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59433.log.gz 2024-03-28T23:37:45.728 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52506.log 2024-03-28T23:37:45.728 INFO:teuthology.orchestra.run.smithi161.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41604.log.gz 2024-03-28T23:37:45.728 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.71616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76061.log 2024-03-28T23:37:45.728 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52506.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71616.log.gz 2024-03-28T23:37:45.728 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52506.log.gz 2024-03-28T23:37:45.728 INFO:teuthology.orchestra.run.smithi161.stderr: /var/log/ceph/ceph-client.admin.41946.log 2024-03-28T23:37:45.728 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.76061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43662.log 2024-03-28T23:37:45.729 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76061.log.gz 2024-03-28T23:37:45.729 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.41946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58115.log 2024-03-28T23:37:45.729 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55879.log 2024-03-28T23:37:45.729 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41946.log.gz 2024-03-28T23:37:45.729 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60409.log 2024-03-28T23:37:45.729 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43662.log.gz 2024-03-28T23:37:45.729 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58115.log: /var/log/ceph/ceph-client.admin.55879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45555.log 2024-03-28T23:37:45.729 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58115.log.gz/var/log/ceph/ceph-client.admin.60409.log: 2024-03-28T23:37:45.730 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47657.log 2024-03-28T23:37:45.730 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55879.log.gz 2024-03-28T23:37:45.730 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60409.log.gz 2024-03-28T23:37:45.730 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41736.log 2024-03-28T23:37:45.730 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.45555.log: /var/log/ceph/ceph-client.admin.47657.log: gzip -5 --verbose 87.9% -- /var/log/ceph/ceph-client.admin.56809.log 2024-03-28T23:37:45.730 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45555.log.gz 2024-03-28T23:37:45.730 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.41736.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47657.log.gz 2024-03-28T23:37:45.730 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37316.log 2024-03-28T23:37:45.731 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41736.log.gz 2024-03-28T23:37:45.731 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56809.log: gzip -5 --verbose -- 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56809.log.gz 2024-03-28T23:37:45.731 INFO:teuthology.orchestra.run.smithi161.stderr: /var/log/ceph/ceph-client.admin.56767.log 2024-03-28T23:37:45.731 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.37316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48029.log 2024-03-28T23:37:45.731 INFO:teuthology.orchestra.run.smithi161.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37316.log.gz 2024-03-28T23:37:45.731 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49524.log 2024-03-28T23:37:45.731 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56767.log.gz 2024-03-28T23:37:45.731 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45091.log 2024-03-28T23:37:45.732 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48029.log.gz 2024-03-28T23:37:45.732 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55105.log 2024-03-28T23:37:45.732 INFO:teuthology.orchestra.run.smithi161.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49524.log.gz 2024-03-28T23:37:45.732 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.45091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66334.log 2024-03-28T23:37:45.732 INFO:teuthology.orchestra.run.smithi161.stderr:gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45091.log.gz/var/log/ceph/ceph-client.admin.55105.log: 2024-03-28T23:37:45.732 INFO:teuthology.orchestra.run.smithi161.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43831.log 2024-03-28T23:37:45.732 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.66334.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55105.log.gz 2024-03-28T23:37:45.732 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63417.log 2024-03-28T23:37:45.733 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43831.log: gzip -5 91.2% -- replaced with /var/log/ceph/ceph-client.admin.66334.log.gz 2024-03-28T23:37:45.733 INFO:teuthology.orchestra.run.smithi161.stderr: --verbose -- /var/log/ceph/ceph-client.admin.74472.log 2024-03-28T23:37:45.733 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43831.log.gz 2024-03-28T23:37:45.733 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.63417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71965.log 2024-03-28T23:37:45.733 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.74472.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63417.log.gz 2024-03-28T23:37:45.733 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44538.log 2024-03-28T23:37:45.733 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74472.log.gz 2024-03-28T23:37:45.733 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.71965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74144.log 2024-03-28T23:37:45.733 INFO:teuthology.orchestra.run.smithi161.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.71965.log.gz 2024-03-28T23:37:45.734 INFO:teuthology.orchestra.run.smithi161.stderr:gzip/var/log/ceph/ceph-client.admin.44538.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43026.log 2024-03-28T23:37:45.734 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.74144.log: gzip -5 --verbose 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44538.log.gz 2024-03-28T23:37:45.734 INFO:teuthology.orchestra.run.smithi161.stderr: -- /var/log/ceph/ceph-client.admin.52716.log 2024-03-28T23:37:45.734 INFO:teuthology.orchestra.run.smithi161.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.74144.log.gz 2024-03-28T23:37:45.734 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57803.log 2024-03-28T23:37:45.734 INFO:teuthology.orchestra.run.smithi161.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43026.log.gz 2024-03-28T23:37:45.734 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57085.log 2024-03-28T23:37:45.734 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52716.log.gz 2024-03-28T23:37:45.735 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64792.log 2024-03-28T23:37:45.735 INFO:teuthology.orchestra.run.smithi161.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57803.log.gz 2024-03-28T23:37:45.735 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66696.log 2024-03-28T23:37:45.735 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57085.log.gz 2024-03-28T23:37:45.735 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.64792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58475.log 2024-03-28T23:37:45.735 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64792.log.gz 2024-03-28T23:37:45.735 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.66696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73933.log 2024-03-28T23:37:45.735 INFO:teuthology.orchestra.run.smithi161.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.66696.log.gz 2024-03-28T23:37:45.736 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45049.log 2024-03-28T23:37:45.736 INFO:teuthology.orchestra.run.smithi161.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58475.log.gz 2024-03-28T23:37:45.736 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.73933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67633.log 2024-03-28T23:37:45.736 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.73933.log.gz 2024-03-28T23:37:45.736 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.45049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74300.log 2024-03-28T23:37:45.736 INFO:teuthology.orchestra.run.smithi161.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45049.log.gz 2024-03-28T23:37:45.736 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.67633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55837.log 2024-03-28T23:37:45.736 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.74300.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.67633.log.gz 2024-03-28T23:37:45.737 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64288.log 2024-03-28T23:37:45.737 INFO:teuthology.orchestra.run.smithi161.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.74300.log.gz 2024-03-28T23:37:45.737 INFO:teuthology.orchestra.run.smithi161.stderr:gzip/var/log/ceph/ceph-client.admin.55837.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.72263.log 2024-03-28T23:37:45.737 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.64288.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55837.log.gz 2024-03-28T23:37:45.737 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64288.log.gz 2024-03-28T23:37:45.737 INFO:teuthology.orchestra.run.smithi161.stderr: -- /var/log/ceph/ceph-client.admin.49026.log 2024-03-28T23:37:45.737 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.72263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69135.log 2024-03-28T23:37:45.738 INFO:teuthology.orchestra.run.smithi161.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.72263.log.gz 2024-03-28T23:37:45.738 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59009.log 2024-03-28T23:37:45.738 INFO:teuthology.orchestra.run.smithi161.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49026.log.gz 2024-03-28T23:37:45.738 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.69135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73639.log 2024-03-28T23:37:45.738 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59009.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69135.log.gz 2024-03-28T23:37:45.738 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53286.log 2024-03-28T23:37:45.738 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.73639.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59009.log.gz 2024-03-28T23:37:45.738 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73639.log.gz 2024-03-28T23:37:45.739 INFO:teuthology.orchestra.run.smithi161.stderr: /var/log/ceph/ceph-client.admin.64540.log 2024-03-28T23:37:45.739 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72095.log 2024-03-28T23:37:45.739 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.64540.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53286.log.gz 2024-03-28T23:37:45.739 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74953.log 2024-03-28T23:37:45.739 INFO:teuthology.orchestra.run.smithi161.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64540.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-28T23:37:45.739 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:37:45.739 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.72095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46622.log 2024-03-28T23:37:45.739 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.74953.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.72095.log.gz 2024-03-28T23:37:45.739 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46790.log 2024-03-28T23:37:45.740 INFO:teuthology.orchestra.run.smithi161.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.74953.log.gz 2024-03-28T23:37:45.740 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-mgr.y.log: /var/log/ceph/ceph-client.admin.46622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62151.log 2024-03-28T23:37:45.740 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7%/var/log/ceph/ceph-client.admin.46790.log: -- replaced with /var/log/ceph/ceph-client.admin.46622.log.gz 2024-03-28T23:37:45.740 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56042.log 2024-03-28T23:37:45.740 INFO:teuthology.orchestra.run.smithi161.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46790.log.gz 2024-03-28T23:37:45.740 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.62151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64750.log 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62151.log.gz 2024-03-28T23:37:45.740 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:37:45.740 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61941.log 2024-03-28T23:37:45.741 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.64750.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56042.log.gz 2024-03-28T23:37:45.741 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43152.log 2024-03-28T23:37:45.741 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64750.log.gz 2024-03-28T23:37:45.741 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56416.log 2024-03-28T23:37:45.741 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.61941.log: /var/log/ceph/ceph-client.admin.43152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43568.log 2024-03-28T23:37:45.741 INFO:teuthology.orchestra.run.smithi161.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.61941.log.gz 2024-03-28T23:37:45.741 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3%/var/log/ceph/ceph-client.admin.56416.log: -- replaced with /var/log/ceph/ceph-client.admin.43152.log.gz 2024-03-28T23:37:45.741 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56416.log.gz 2024-03-28T23:37:45.741 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44475.log 2024-03-28T23:37:45.742 INFO:teuthology.orchestra.run.smithi161.stderr:gzip/var/log/ceph/ceph-client.admin.43568.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.52069.log 2024-03-28T23:37:45.742 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44475.log: gzip -5 --verbose 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43568.log.gz 2024-03-28T23:37:45.742 INFO:teuthology.orchestra.run.smithi161.stderr: -- /var/log/ceph/ceph-client.admin.48419.log 2024-03-28T23:37:45.742 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44475.log.gz 2024-03-28T23:37:45.742 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.52069.log: -- /var/log/ceph/ceph-client.admin.61605.log 2024-03-28T23:37:45.742 INFO:teuthology.orchestra.run.smithi161.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52069.log.gz 2024-03-28T23:37:45.742 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57971.log 2024-03-28T23:37:45.742 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48419.log: /var/log/ceph/ceph-client.admin.61605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41509.log 2024-03-28T23:37:45.743 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57971.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61605.log.gz 2024-03-28T23:37:45.743 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48419.log.gz 2024-03-28T23:37:45.743 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57971.log.gz 2024-03-28T23:37:45.743 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63039.log 2024-03-28T23:37:45.743 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.41509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47515.log 2024-03-28T23:37:45.743 INFO:teuthology.orchestra.run.smithi161.stderr: 89.0%/var/log/ceph/ceph-client.admin.63039.log: -- replaced with /var/log/ceph/ceph-client.admin.41509.log.gz 2024-03-28T23:37:45.743 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48503.log 2024-03-28T23:37:45.743 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63039.log.gz 2024-03-28T23:37:45.743 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54917.log 2024-03-28T23:37:45.744 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47515.log: /var/log/ceph/ceph-client.admin.48503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65908.log 2024-03-28T23:37:45.744 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54917.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48503.log.gz 2024-03-28T23:37:45.744 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47515.log.gz 2024-03-28T23:37:45.744 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54917.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46832.log 2024-03-28T23:37:45.744 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:37:45.744 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.65908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48629.log 2024-03-28T23:37:45.744 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45343.log 2024-03-28T23:37:45.744 INFO:teuthology.orchestra.run.smithi161.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.65908.log.gz 2024-03-28T23:37:45.744 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46832.log: /var/log/ceph/ceph-client.admin.48629.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46832.log.gz 2024-03-28T23:37:45.745 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48629.log.gz 2024-03-28T23:37:45.745 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57001.log 2024-03-28T23:37:45.745 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.45343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62997.log 2024-03-28T23:37:45.745 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45343.log.gz 2024-03-28T23:37:45.745 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67110.log 2024-03-28T23:37:45.745 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57001.log: gzip -5 --verbose --/var/log/ceph/ceph-client.admin.62997.log: /var/log/ceph/ceph-client.admin.53370.log 2024-03-28T23:37:45.745 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.67110.log: 88.2% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62997.log.gz 2024-03-28T23:37:45.746 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57001.log.gz 2024-03-28T23:37:45.746 INFO:teuthology.orchestra.run.smithi161.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67110.log.gz 2024-03-28T23:37:45.746 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49271.log 2024-03-28T23:37:45.746 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68395.log 2024-03-28T23:37:45.746 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49271.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53370.log.gz 2024-03-28T23:37:45.746 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54077.log 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49271.log.gz 2024-03-28T23:37:45.746 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:37:45.746 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72709.log 2024-03-28T23:37:45.747 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.68395.log: gzip -5 --verbose/var/log/ceph/ceph-client.admin.54077.log: -- /var/log/ceph/ceph-client.admin.62109.log 2024-03-28T23:37:45.747 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68395.log.gz 2024-03-28T23:37:45.747 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54077.log.gz 2024-03-28T23:37:45.747 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.72709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62560.log 2024-03-28T23:37:45.747 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37423.log 2024-03-28T23:37:45.747 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.72709.log.gz 2024-03-28T23:37:45.747 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.62109.log: /var/log/ceph/ceph-client.admin.62560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71914.log 2024-03-28T23:37:45.748 INFO:teuthology.orchestra.run.smithi161.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62109.log.gz 2024-03-28T23:37:45.748 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.37423.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62560.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68755.log 2024-03-28T23:37:45.748 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:37:45.748 INFO:teuthology.orchestra.run.smithi161.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37423.log.gz 2024-03-28T23:37:45.748 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59493.log 2024-03-28T23:37:45.748 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.71914.log: /var/log/ceph/ceph-client.admin.68755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58433.log 2024-03-28T23:37:45.748 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59493.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.71914.log.gz 2024-03-28T23:37:45.748 INFO:teuthology.orchestra.run.smithi161.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.68755.log.gz 2024-03-28T23:37:45.749 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60365.log 2024-03-28T23:37:45.749 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59493.log.gz 2024-03-28T23:37:45.749 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46706.log 2024-03-28T23:37:45.749 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58433.log: /var/log/ceph/ceph-client.admin.60365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49398.log 2024-03-28T23:37:45.749 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46706.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.60365.log.gz 2024-03-28T23:37:45.749 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58433.log.gz 2024-03-28T23:37:45.749 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46706.log.gz 2024-03-28T23:37:45.750 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42716.log 2024-03-28T23:37:45.750 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59746.log 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49398.log.gz 2024-03-28T23:37:45.750 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:37:45.750 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.42716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50147.log 2024-03-28T23:37:45.750 INFO:teuthology.orchestra.run.smithi161.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42716.log.gz/var/log/ceph/ceph-client.admin.59746.log: 2024-03-28T23:37:45.750 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51138.log 2024-03-28T23:37:45.750 INFO:teuthology.orchestra.run.smithi161.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59746.log.gz 2024-03-28T23:37:45.750 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64414.log 2024-03-28T23:37:45.751 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50147.log: /var/log/ceph/ceph-client.admin.51138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70027.log 2024-03-28T23:37:45.751 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.64414.log: 88.7% 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51138.log.gz 2024-03-28T23:37:45.751 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50147.log.gz 2024-03-28T23:37:45.751 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47084.log 2024-03-28T23:37:45.751 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.64414.log.gz 2024-03-28T23:37:45.751 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.70027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74426.log 2024-03-28T23:37:45.751 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70027.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47825.log 2024-03-28T23:37:45.751 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:37:45.752 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47084.log: /var/log/ceph/ceph-client.admin.74426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55609.log 2024-03-28T23:37:45.752 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47825.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47084.log.gz 89.0% 2024-03-28T23:37:45.752 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.74426.log.gz 2024-03-28T23:37:45.752 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47825.log.gz 2024-03-28T23:37:45.752 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59704.log 2024-03-28T23:37:45.752 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.55609.log: /var/log/ceph/ceph-client.admin.73723.log 2024-03-28T23:37:45.752 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59704.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55609.log.gz 2024-03-28T23:37:45.753 INFO:teuthology.orchestra.run.smithi161.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.59704.log.gz 2024-03-28T23:37:45.753 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66942.log 2024-03-28T23:37:45.753 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.73723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63309.log 2024-03-28T23:37:45.753 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6%/var/log/ceph/ceph-client.admin.66942.log: -- replaced with /var/log/ceph/ceph-client.admin.73723.log.gz 2024-03-28T23:37:45.753 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57677.log 2024-03-28T23:37:45.753 INFO:teuthology.orchestra.run.smithi161.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.66942.log.gz 2024-03-28T23:37:45.753 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.63309.log: gzip -5 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63309.log.gz 2024-03-28T23:37:45.753 INFO:teuthology.orchestra.run.smithi161.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44923.log 2024-03-28T23:37:45.754 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57677.log: /var/log/ceph/ceph-client.admin.44923.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57677.log.gz 2024-03-28T23:37:45.754 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44923.log.gz 2024-03-28T23:37:45.754 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56851.log 2024-03-28T23:37:45.754 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69733.log 2024-03-28T23:37:45.754 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43830.log 2024-03-28T23:37:45.754 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69393.log 2024-03-28T23:37:45.754 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.69733.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56851.log.gz 2024-03-28T23:37:45.754 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43830.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69733.log.gz 2024-03-28T23:37:45.755 INFO:teuthology.orchestra.run.smithi161.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43830.log.gz 2024-03-28T23:37:45.755 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70279.log 2024-03-28T23:37:45.755 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.69393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52968.log 2024-03-28T23:37:45.755 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69393.log.gz 2024-03-28T23:37:45.755 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.70279.log: gzip -5 --verbose -- 89.0% -- replaced with /var/log/ceph/ceph-client.admin.70279.log.gz 2024-03-28T23:37:45.755 INFO:teuthology.orchestra.run.smithi161.stderr: /var/log/ceph/ceph-client.admin.62871.log 2024-03-28T23:37:45.755 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52968.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-28T23:37:45.755 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.62871.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.52968.log.gz 2024-03-28T23:37:45.756 INFO:teuthology.orchestra.run.smithi161.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62871.log.gz 2024-03-28T23:37:45.756 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70814.log 2024-03-28T23:37:45.756 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47431.log 2024-03-28T23:37:45.756 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75877.log 2024-03-28T23:37:45.756 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47431.log: /var/log/ceph/ceph-client.admin.70814.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47431.log.gz 2024-03-28T23:37:45.756 INFO:teuthology.orchestra.run.smithi161.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.70814.log.gz 2024-03-28T23:37:45.756 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71382.log 2024-03-28T23:37:45.756 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61983.log 2024-03-28T23:37:45.757 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.71382.log: /var/log/ceph/ceph-client.admin.75877.log: 87.6% 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71382.log.gz 2024-03-28T23:37:45.757 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.75877.log.gz 2024-03-28T23:37:45.757 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57337.log 2024-03-28T23:37:45.757 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.61983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63351.log 2024-03-28T23:37:45.757 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61983.log.gz 2024-03-28T23:37:45.757 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57337.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57337.log.gz 2024-03-28T23:37:45.757 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48545.log 2024-03-28T23:37:45.757 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51618.log 2024-03-28T23:37:45.758 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48545.log: /var/log/ceph/ceph-client.admin.63351.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48545.log.gz 2024-03-28T23:37:45.758 INFO:teuthology.orchestra.run.smithi161.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.63351.log.gz 2024-03-28T23:37:45.758 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69985.log 2024-03-28T23:37:45.758 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57761.log 2024-03-28T23:37:45.758 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51618.log.gz 2024-03-28T23:37:45.758 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.69985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43829.log 2024-03-28T23:37:45.758 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57761.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69985.log.gz 2024-03-28T23:37:45.759 INFO:teuthology.orchestra.run.smithi161.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57761.log.gz 2024-03-28T23:37:45.759 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49608.log 2024-03-28T23:37:45.759 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58913.log 2024-03-28T23:37:45.759 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49608.log: /var/log/ceph/ceph-client.admin.43829.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49608.log.gz 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43829.log.gz 2024-03-28T23:37:45.759 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:37:45.759 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52926.log 2024-03-28T23:37:45.759 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76103.log 2024-03-28T23:37:45.759 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52926.log: /var/log/ceph/ceph-client.admin.58913.log: 88.3% 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58913.log.gz 2024-03-28T23:37:45.760 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52926.log.gz 2024-03-28T23:37:45.760 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49440.log 2024-03-28T23:37:45.760 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43357.log 2024-03-28T23:37:45.760 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49440.log: /var/log/ceph/ceph-client.admin.76103.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49440.log.gz 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76103.log.gz 2024-03-28T23:37:45.760 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:37:45.760 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56683.log 2024-03-28T23:37:45.760 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72556.log 2024-03-28T23:37:45.761 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56683.log: /var/log/ceph/ceph-client.admin.43357.log: 87.8% 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56683.log.gz 2024-03-28T23:37:45.761 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43357.log.gz 2024-03-28T23:37:45.761 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59788.log 2024-03-28T23:37:45.761 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.72556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65652.log 2024-03-28T23:37:45.761 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72556.log.gz 2024-03-28T23:37:45.761 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52237.log 2024-03-28T23:37:45.761 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.65652.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59788.log.gz 2024-03-28T23:37:45.761 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65652.log.gz 2024-03-28T23:37:45.762 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44172.log 2024-03-28T23:37:45.762 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54353.log 2024-03-28T23:37:45.762 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44172.log: /var/log/ceph/ceph-client.admin.52237.log: 88.7% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44172.log.gz 2024-03-28T23:37:45.762 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52237.log.gz 2024-03-28T23:37:45.762 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-28T23:37:45.762 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74640.log 2024-03-28T23:37:45.762 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54353.log.gz 2024-03-28T23:37:45.762 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68170.log 2024-03-28T23:37:45.763 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.74640.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74640.log.gz 2024-03-28T23:37:45.763 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47987.log 2024-03-28T23:37:45.763 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.68170.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68170.log.gz 2024-03-28T23:37:45.763 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64330.log 2024-03-28T23:37:45.763 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47987.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47987.log.gz 2024-03-28T23:37:45.763 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74384.log 2024-03-28T23:37:45.763 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.64330.log: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-28T23:37:45.763 INFO:teuthology.orchestra.run.smithi161.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64330.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-28T23:37:45.764 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:37:45.764 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44061.log 2024-03-28T23:37:45.764 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.74384.log: /var/log/ceph/ceph-osd.2.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74384.log.gz 2024-03-28T23:37:45.764 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60807.log 2024-03-28T23:37:45.764 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44061.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44061.log.gz 2024-03-28T23:37:45.764 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71088.log 2024-03-28T23:37:45.764 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60807.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60807.log.gz 2024-03-28T23:37:45.765 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63267.log 2024-03-28T23:37:45.765 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.71088.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.71088.log.gz 2024-03-28T23:37:45.765 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64204.log 2024-03-28T23:37:45.765 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.63267.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63267.log.gz 2024-03-28T23:37:45.765 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68479.log 2024-03-28T23:37:45.765 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.64204.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64204.log.gz 2024-03-28T23:37:45.765 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60723.log 2024-03-28T23:37:45.765 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.68479.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68479.log.gz 2024-03-28T23:37:45.766 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47867.log 2024-03-28T23:37:45.766 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60723.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60723.log.gz 2024-03-28T23:37:45.766 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65123.log 2024-03-28T23:37:45.766 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47867.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47867.log.gz 2024-03-28T23:37:45.766 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61479.log 2024-03-28T23:37:45.766 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.65123.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.65123.log.gz 2024-03-28T23:37:45.766 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69901.log 2024-03-28T23:37:45.766 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.61479.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61479.log.gz 2024-03-28T23:37:45.767 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68881.log 2024-03-28T23:37:45.767 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.69901.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69901.log.gz 2024-03-28T23:37:45.767 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53500.log 2024-03-28T23:37:45.767 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.68881.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.68881.log.gz 2024-03-28T23:37:45.767 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44477.log 2024-03-28T23:37:45.767 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53500.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53500.log.gz 2024-03-28T23:37:45.767 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-28T23:37:45.767 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44477.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44477.log.gz 2024-03-28T23:37:45.768 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64834.log 2024-03-28T23:37:45.768 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54035.log 2024-03-28T23:37:45.768 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.64834.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64834.log.gz 2024-03-28T23:37:45.768 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73435.log 2024-03-28T23:37:45.768 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54035.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54035.log.gz 2024-03-28T23:37:45.768 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65866.log 2024-03-28T23:37:45.768 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.73435.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73435.log.gz 2024-03-28T23:37:45.768 INFO:teuthology.orchestra.run.smithi161.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-28T23:37:45.769 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44755.log 2024-03-28T23:37:45.769 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.65866.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.65866.log.gz 2024-03-28T23:37:45.769 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48587.log 2024-03-28T23:37:45.769 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44755.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44755.log.gz 2024-03-28T23:37:45.769 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41014.log 2024-03-28T23:37:45.769 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48839.log 2024-03-28T23:37:45.769 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.41014.log: /var/log/ceph/ceph-client.admin.48587.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41014.log.gz 2024-03-28T23:37:45.770 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48587.log.gz 2024-03-28T23:37:45.770 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44377.log 2024-03-28T23:37:45.770 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48839.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48839.log.gz 2024-03-28T23:37:45.770 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71550.log 2024-03-28T23:37:45.770 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44377.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44377.log.gz 2024-03-28T23:37:45.770 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56460.log 2024-03-28T23:37:45.770 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.71550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48881.log 2024-03-28T23:37:45.771 INFO:teuthology.orchestra.run.smithi170.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-28T23:37:45.829 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56460.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.71550.log.gz 2024-03-28T23:37:45.829 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56460.log.gz 2024-03-28T23:37:45.829 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65044.log 2024-03-28T23:37:45.830 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49668.log 2024-03-28T23:37:45.830 INFO:teuthology.orchestra.run.smithi161.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48881.log.gz 2024-03-28T23:37:45.830 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.65044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43840.log 2024-03-28T23:37:45.830 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49668.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65044.log.gz 2024-03-28T23:37:45.830 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62825.log 2024-03-28T23:37:45.830 INFO:teuthology.orchestra.run.smithi161.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49668.log.gz 2024-03-28T23:37:45.830 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64246.log 2024-03-28T23:37:45.830 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.62825.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43840.log.gz 2024-03-28T23:37:45.831 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62825.log.gz 2024-03-28T23:37:45.831 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45133.log 2024-03-28T23:37:45.831 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.64246.log: -- /var/log/ceph/ceph-client.admin.49754.log 2024-03-28T23:37:45.831 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64246.log.gz 2024-03-28T23:37:45.831 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.45133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59181.log 2024-03-28T23:37:45.831 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49754.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45133.log.gz 2024-03-28T23:37:45.831 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64708.log 2024-03-28T23:37:45.831 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49754.log.gz 2024-03-28T23:37:45.832 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68000.log 2024-03-28T23:37:45.832 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.64708.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59181.log.gz 2024-03-28T23:37:45.832 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48983.log 2024-03-28T23:37:45.832 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.68000.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68000.log.gz 2024-03-28T23:37:45.832 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66471.log 2024-03-28T23:37:45.832 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48983.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48983.log.gz 2024-03-28T23:37:45.832 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57253.log 2024-03-28T23:37:45.832 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.66471.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.66471.log.gz 2024-03-28T23:37:45.832 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71340.log 2024-03-28T23:37:45.833 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57253.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57253.log.gz 2024-03-28T23:37:45.833 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67507.log 2024-03-28T23:37:45.833 INFO:teuthology.orchestra.run.smithi161.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-28T23:37:45.833 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.71340.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64708.log.gz 2024-03-28T23:37:45.833 INFO:teuthology.orchestra.run.smithi161.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71340.log.gz 2024-03-28T23:37:45.833 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70856.log 2024-03-28T23:37:45.833 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.67507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75751.log 2024-03-28T23:37:45.833 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67507.log.gz 2024-03-28T23:37:45.833 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.70856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67958.log 2024-03-28T23:37:45.834 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.75751.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.70856.log.gz 2024-03-28T23:37:45.834 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.75751.log.gz 2024-03-28T23:37:45.834 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57509.log 2024-03-28T23:37:45.834 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72919.log 2024-03-28T23:37:45.834 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57509.log: /var/log/ceph/ceph-client.admin.67958.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.67958.log.gz 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57509.log.gz 2024-03-28T23:37:45.834 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:37:45.834 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68353.log 2024-03-28T23:37:45.835 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.72919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65610.log 2024-03-28T23:37:45.835 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.68353.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.72919.log.gz 2024-03-28T23:37:45.835 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68353.log.gz 2024-03-28T23:37:45.835 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55147.log 2024-03-28T23:37:45.835 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.65610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59536.log 2024-03-28T23:37:45.835 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65610.log.gz 2024-03-28T23:37:45.835 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75457.log 2024-03-28T23:37:45.836 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59536.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55147.log.gz 2024-03-28T23:37:45.836 INFO:teuthology.orchestra.run.smithi161.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59536.log.gz 2024-03-28T23:37:45.836 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45301.log 2024-03-28T23:37:45.836 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51702.log 2024-03-28T23:37:45.836 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.45301.log: /var/log/ceph/ceph-client.admin.75457.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45301.log.gz 2024-03-28T23:37:45.836 INFO:teuthology.orchestra.run.smithi161.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.75457.log.gz 2024-03-28T23:37:45.836 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59956.log 2024-03-28T23:37:45.836 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73135.log 2024-03-28T23:37:45.837 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59956.log: /var/log/ceph/ceph-client.admin.51702.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51702.log.gz 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59956.log.gz 2024-03-28T23:37:45.837 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:37:45.837 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53699.log 2024-03-28T23:37:45.837 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.73135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72347.log 2024-03-28T23:37:45.837 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53699.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.73135.log.gz 2024-03-28T23:37:45.837 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53699.log.gz 2024-03-28T23:37:45.838 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68437.log 2024-03-28T23:37:45.838 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.72347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73027.log 2024-03-28T23:37:45.838 INFO:teuthology.orchestra.run.smithi161.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.72347.log.gz 2024-03-28T23:37:45.838 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.68437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66589.log 2024-03-28T23:37:45.838 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.73027.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68437.log.gz 2024-03-28T23:37:45.838 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57887.log 2024-03-28T23:37:45.838 INFO:teuthology.orchestra.run.smithi161.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.73027.log.gz 2024-03-28T23:37:45.839 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71130.log 2024-03-28T23:37:45.839 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.66589.log: /var/log/ceph/ceph-client.admin.57887.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66589.log.gz 2024-03-28T23:37:45.839 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57887.log.gz 2024-03-28T23:37:45.839 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74514.log 2024-03-28T23:37:45.839 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63081.log 2024-03-28T23:37:45.839 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.74514.log: /var/log/ceph/ceph-client.admin.71130.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.74514.log.gz 89.2% -- replaced with /var/log/ceph/ceph-client.admin.71130.log.gz 2024-03-28T23:37:45.839 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:37:45.839 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67465.log 2024-03-28T23:37:45.840 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59093.log 2024-03-28T23:37:45.840 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.67465.log: /var/log/ceph/ceph-client.admin.63081.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63081.log.gz 2024-03-28T23:37:45.840 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67465.log.gz 2024-03-28T23:37:45.840 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44178.log 2024-03-28T23:37:45.840 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41465.log 2024-03-28T23:37:45.840 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59093.log.gz 2024-03-28T23:37:45.840 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48292.log 2024-03-28T23:37:45.841 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.41465.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44178.log.gz 2024-03-28T23:37:45.841 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41465.log.gz 2024-03-28T23:37:45.841 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46142.log 2024-03-28T23:37:45.841 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71172.log 2024-03-28T23:37:45.841 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46142.log: /var/log/ceph/ceph-client.admin.48292.log: 89.5% 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48292.log.gz 2024-03-28T23:37:45.841 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46142.log.gz 2024-03-28T23:37:45.841 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73093.log 2024-03-28T23:37:45.841 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46370.log 2024-03-28T23:37:45.842 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.73093.log: /var/log/ceph/ceph-client.admin.71172.log: 88.3% 89.5% -- replaced with /var/log/ceph/ceph-client.admin.73093.log.gz 2024-03-28T23:37:45.842 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71172.log.gz 2024-03-28T23:37:45.842 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75285.log 2024-03-28T23:37:45.842 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75919.log 2024-03-28T23:37:45.842 INFO:teuthology.orchestra.run.smithi161.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.46370.log.gz 2024-03-28T23:37:45.842 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.75285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44181.log 2024-03-28T23:37:45.842 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.75919.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.75285.log.gz 2024-03-28T23:37:45.842 INFO:teuthology.orchestra.run.smithi161.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75919.log.gz 2024-03-28T23:37:45.843 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64876.log 2024-03-28T23:37:45.843 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56204.log 2024-03-28T23:37:45.843 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.64876.log: /var/log/ceph/ceph-client.admin.44181.log: 89.4% 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64876.log.gz 2024-03-28T23:37:45.843 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44181.log.gz 2024-03-28T23:37:45.843 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49482.log 2024-03-28T23:37:45.843 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56935.log 2024-03-28T23:37:45.843 INFO:teuthology.orchestra.run.smithi161.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56204.log.gz 2024-03-28T23:37:45.843 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49482.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-28T23:37:45.844 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56935.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49482.log.gz 2024-03-28T23:37:45.844 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46874.log 91.2% 2024-03-28T23:37:45.844 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56935.log.gz 2024-03-28T23:37:45.844 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68084.log 2024-03-28T23:37:45.844 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.46874.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46874.log.gz 2024-03-28T23:37:45.844 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75243.log 2024-03-28T23:37:45.844 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.68084.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68084.log.gz 2024-03-28T23:37:45.844 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68126.log 2024-03-28T23:37:45.844 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.75243.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75243.log.gz 2024-03-28T23:37:45.845 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55651.log 2024-03-28T23:37:45.845 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.68126.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.68126.log.gz 2024-03-28T23:37:45.845 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75667.log 2024-03-28T23:37:45.845 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55651.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55651.log.gz 2024-03-28T23:37:45.845 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58601.log 2024-03-28T23:37:45.845 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.75667.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75667.log.gz 2024-03-28T23:37:45.845 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74742.log 2024-03-28T23:37:45.846 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58601.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58601.log.gz 2024-03-28T23:37:45.846 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43483.log 2024-03-28T23:37:45.846 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.74742.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.74742.log.gz 2024-03-28T23:37:45.846 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59662.log 2024-03-28T23:37:45.846 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43483.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43483.log.gz 2024-03-28T23:37:45.846 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42800.log 2024-03-28T23:37:45.846 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59662.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59662.log.gz 2024-03-28T23:37:45.847 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71870.log 2024-03-28T23:37:45.847 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.42800.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42800.log.gz 2024-03-28T23:37:45.847 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50850.log 2024-03-28T23:37:45.847 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.71870.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71870.log.gz 2024-03-28T23:37:45.847 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52460.log 2024-03-28T23:37:45.847 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50850.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.50850.log.gz 2024-03-28T23:37:45.847 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52460.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52460.log.gz 2024-03-28T23:37:45.847 INFO:teuthology.orchestra.run.smithi161.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-28T23:37:46.206 INFO:teuthology.orchestra.run.smithi161.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-28T23:37:46.534 INFO:teuthology.orchestra.run.smithi170.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-28T23:37:46.563 INFO:teuthology.orchestra.run.smithi170.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-28T23:37:46.566 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-28T23:37:46.625 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-28T23:37:46.627 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-28T23:37:46.627 INFO:teuthology.orchestra.run.smithi170.stderr:real 0m1.293s 2024-03-28T23:37:46.627 INFO:teuthology.orchestra.run.smithi170.stderr:user 0m5.413s 2024-03-28T23:37:46.627 INFO:teuthology.orchestra.run.smithi170.stderr:sys 0m0.201s 2024-03-28T23:37:46.652 INFO:teuthology.orchestra.run.smithi161.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-28T23:37:46.723 INFO:teuthology.orchestra.run.smithi161.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-28T23:37:46.795 INFO:teuthology.orchestra.run.smithi161.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-28T23:37:47.009 INFO:teuthology.orchestra.run.smithi161.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-28T23:37:47.010 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-28T23:37:47.011 INFO:teuthology.orchestra.run.smithi161.stderr:real 0m1.722s 2024-03-28T23:37:47.011 INFO:teuthology.orchestra.run.smithi161.stderr:user 0m6.619s 2024-03-28T23:37:47.011 INFO:teuthology.orchestra.run.smithi161.stderr:sys 0m0.625s 2024-03-28T23:37:47.011 INFO:tasks.ceph:Archiving logs... 2024-03-28T23:37:47.012 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627196/remote/smithi161/log 2024-03-28T23:37:47.013 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T23:37:48.525 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627196/remote/smithi170/log 2024-03-28T23:37:48.526 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T23:37:49.347 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-28T23:37:49.357 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-28T23:37:49.357 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T23:37:49.360 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T23:37:49.449 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-28T23:37:49.459 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-28T23:37:49.459 DEBUG:teuthology.orchestra.run.smithi161:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T23:37:49.462 DEBUG:teuthology.orchestra.run.smithi170:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T23:37:49.476 INFO:teuthology.orchestra.run.smithi161.stderr:bash: line 1: ntpq: command not found 2024-03-28T23:37:49.477 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2024-03-28T23:37:49.481 INFO:teuthology.orchestra.run.smithi161.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T23:37:49.481 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================== 2024-03-28T23:37:49.481 INFO:teuthology.orchestra.run.smithi161.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 60 -183us[ -188us] +/- 66ms 2024-03-28T23:37:49.482 INFO:teuthology.orchestra.run.smithi161.stdout:^* hv02.front.sepia.ceph.com 3 6 377 58 -276us[ -280us] +/- 60ms 2024-03-28T23:37:49.482 INFO:teuthology.orchestra.run.smithi161.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 +642us[ +642us] +/- 90ms 2024-03-28T23:37:49.482 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T23:37:49.483 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T23:37:49.483 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2024-03-28T23:37:49.484 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 59 -186us[ -186us] +/- 66ms 2024-03-28T23:37:49.484 INFO:teuthology.orchestra.run.smithi170.stdout:^* hv02.front.sepia.ceph.com 3 6 377 59 -281us[ -277us] +/- 60ms 2024-03-28T23:37:49.484 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 +635us[ +635us] +/- 90ms 2024-03-28T23:37:49.484 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T23:37:49.484 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-28T23:37:49.494 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-28T23:37:49.495 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-28T23:37:49.541 DEBUG:teuthology.orchestra.run.smithi161:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T23:37:49.632 DEBUG:teuthology.orchestra.run.smithi170:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T23:37:49.706 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T23:37:49.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:37:49.739 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T23:37:49.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:37:49.771 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-28T23:37:49.781 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-28T23:37:49.887 INFO:teuthology.task.internal:Duration was 909.549561 seconds 2024-03-28T23:37:49.887 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-28T23:37:49.896 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-28T23:37:49.897 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T23:37:49.899 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T23:37:49.943 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T23:37:49.945 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T23:37:50.329 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-28T23:37:50.329 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi161.front.sepia.ceph.com 2024-03-28T23:37:50.329 DEBUG:teuthology.orchestra.run.smithi161:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T23:37:50.351 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi170.front.sepia.ceph.com 2024-03-28T23:37:50.351 DEBUG:teuthology.orchestra.run.smithi170:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T23:37:50.373 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-28T23:37:50.373 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T23:37:50.393 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T23:37:50.443 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-28T23:37:50.443 DEBUG:teuthology.orchestra.run.smithi161:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T23:37:50.464 DEBUG:teuthology.orchestra.run.smithi170:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T23:37:50.592 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-28T23:37:50.604 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-28T23:37:50.604 DEBUG:teuthology.orchestra.run.smithi161:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T23:37:50.631 DEBUG:teuthology.orchestra.run.smithi170:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T23:37:50.658 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-28T23:37:50.669 DEBUG:teuthology.orchestra.run.smithi161:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T23:37:50.673 DEBUG:teuthology.orchestra.run.smithi170:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T23:37:50.696 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = core 2024-03-28T23:37:50.723 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = core 2024-03-28T23:37:50.739 DEBUG:teuthology.orchestra.run.smithi161:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T23:37:50.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:37:50.765 DEBUG:teuthology.orchestra.run.smithi170:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T23:37:50.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T23:37:50.793 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-28T23:37:50.804 INFO:teuthology.task.internal:Transferring archived files... 2024-03-28T23:37:50.805 DEBUG:teuthology.misc:Transferring archived files from smithi161:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627196/remote/smithi161 2024-03-28T23:37:50.805 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T23:37:50.842 DEBUG:teuthology.misc:Transferring archived files from smithi170:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627196/remote/smithi170 2024-03-28T23:37:50.842 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T23:37:50.879 INFO:teuthology.task.internal:Removing archive directory... 2024-03-28T23:37:50.879 DEBUG:teuthology.orchestra.run.smithi161:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T23:37:50.881 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T23:37:50.927 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-28T23:37:50.939 INFO:teuthology.task.internal:Not uploading archives. 2024-03-28T23:37:50.939 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-28T23:37:50.949 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-28T23:37:50.949 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T23:37:50.952 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T23:37:50.964 INFO:teuthology.orchestra.run.smithi161.stdout: 265489 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 23:37 /home/ubuntu/cephtest 2024-03-28T23:37:50.981 INFO:teuthology.orchestra.run.smithi170.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 23:37 /home/ubuntu/cephtest 2024-03-28T23:37:50.982 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-28T23:37:50.993 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-28T23:37:51.014 INFO:teuthology.nuke:Checking targets against current locks 2024-03-28T23:37:51.042 DEBUG:teuthology.nuke:shortname: smithi161 2024-03-28T23:37:51.042 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T23:37:51.064 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi161.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627196', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 23:13:52.513898', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBByTCof/EqEz1jIVz78GqGKb+p3QSZbaeon8+Zw2GmOCmoalhHdC1zWHiRPrTT1QZKvq2jFe7DSq61l/GXWj/YU='} 2024-03-28T23:37:51.069 DEBUG:teuthology.nuke:shortname: smithi170 2024-03-28T23:37:51.070 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T23:37:51.091 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627196', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 23:13:52.512897', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOcXNoCIPqw31oOW4OMimd9Wdo5l/Jf3E136yIF9WJYmGzcPjP9ayv8TR+bXAGJHUV+4rOjmqVjUj7KgskU0mUk='} 2024-03-28T23:37:51.111 INFO:teuthology.orchestra.console.smithi161:Power off 2024-03-28T23:37:51.111 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T23:37:51.134 DEBUG:teuthology.orchestra.console.smithi161:power off output: Chassis Power Control: Down/Off 2024-03-28T23:37:51.134 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T23:37:51.150 INFO:teuthology.orchestra.console.smithi170:Power off 2024-03-28T23:37:51.151 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T23:37:51.160 DEBUG:teuthology.orchestra.console.smithi161:check power output: Chassis Power is on 2024-03-28T23:37:51.173 DEBUG:teuthology.orchestra.console.smithi170:power off output: Chassis Power Control: Down/Off 2024-03-28T23:37:51.174 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T23:37:51.197 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2024-03-28T23:37:55.161 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T23:37:55.198 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T23:37:55.272 DEBUG:teuthology.orchestra.console.smithi161:check power output: Chassis Power is on 2024-03-28T23:37:55.309 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2024-03-28T23:37:59.276 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T23:37:59.310 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T23:37:59.388 DEBUG:teuthology.orchestra.console.smithi161:check power output: Chassis Power is on 2024-03-28T23:37:59.420 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is off 2024-03-28T23:37:59.521 INFO:teuthology.orchestra.console.smithi170:Power off completed 2024-03-28T23:38:03.390 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T23:38:03.501 DEBUG:teuthology.orchestra.console.smithi161:check power output: Chassis Power is off 2024-03-28T23:38:03.601 INFO:teuthology.orchestra.console.smithi161:Power off completed 2024-03-28T23:38:03.706 INFO:teuthology.run:Summary data: description: fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} duration: 909.549560546875 failure_reason: 'Test failure: test_join_fs_vanilla (tasks.cephfs.test_failover.TestClusterAffinity)' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=1ad96a13450a4ffb9e9bd5c538854019 status: fail success: false 2024-03-28T23:38:03.706 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T23:38:03.785 INFO:teuthology.run:FAIL